{"thread":{"id":"60768","subject":"[PATCH 0/5] for-each-ref: print all refs on empty string pattern","startedAt":"2024-01-19T14:27:10Z","lastAt":"2024-02-27T16:54:34Z","messageCount":94,"participants":["Karthik Nayak","Junio C Hamano","Phillip Wood","Patrick Steinhardt","phillip.wood123@gmail.com","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"487072","messageId":"20240119142705.139374-1-karthik.188@gmail.com","threadId":"60768","inReplyTo":null,"subject":"[PATCH 0/5] for-each-ref: print all refs on empty string pattern","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-19T14:27:00Z","receivedAt":"2024-01-19T14:27:10Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"With the upcoming introduction of the reftable backend, it becomes ever\nso important to provide the necessary tooling for printing all refs\nassociated with a repository.\n\nWhile regular refs stored within the \"refs/\" namespace are currently\nsupported by multiple commands like git-for-each-ref(1),\ngit-show-ref(1). Neither support printing all the operational refs\nwithin the repository.\n\nThis is crucial because with the reftable backend, all these refs will\nalso move to reftable. It would be necessary to identify all the refs\nthat are stored within the reftable since there is no easy way to do so\notherwise. This is because the reftable itself is a binary format and\nall access will be via git. Unlike the filesystem backend, which allows\naccess directly via the filesystem.\n\nThis patch series is a follow up to the RFC/discussion we had earlier on\nthe list [1].\n\nThe first 4 commits add the required functionality to ensure we can print\nall refs (regular, pseudo, HEAD). The 5th commit modifies the \ngit-for-each-ref(1) command to print all refs when an empty string pattern\nis used. This is a deviation from the current situation wherein the empty\nstring pattern currently matches and prints no refs.\n\n[1]: https://lore.kernel.org/git/20231221170715.110565-1-karthik.188@gmail.com/#t\n\nKarthik Nayak (5):\n refs: expose `is_pseudoref_syntax()`\n refs: make `is_pseudoref_syntax()` stricter\n refs: extract out `loose_fill_ref_dir_regular_file()`\n refs: introduce `refs_for_each_all_refs()`\n for-each-ref: avoid filtering on empty pattern\n\n Documentation/git-for-each-ref.txt | 3 +-\n builtin/for-each-ref.c | 21 ++++-\n ref-filter.c | 13 ++-\n ref-filter.h | 4 +-\n refs.c | 32 ++++++--\n refs.h | 12 +++\n refs/files-backend.c | 126 +++++++++++++++++++++--------\n refs/refs-internal.h | 7 ++\n t/t1407-worktree-ref-store.sh | 12 +--\n t/t6302-for-each-ref-filter.sh | 34 ++++++++\n 10 files changed, 214 insertions(+), 50 deletions(-)\n\n-- \n2.43.GIT\n\n"},{"id":"487073","messageId":"20240119142705.139374-2-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240119142705.139374-1-karthik.188@gmail.com","subject":"[PATCH 1/5] refs: expose `is_pseudoref_syntax()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-19T14:27:01Z","receivedAt":"2024-01-19T14:27:11Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `is_pseudoref_syntax()` function is static, since it is only used\nwithin `refs.c`. In the following commit, we will use this function to\nprovide an utility to add pseudorefs to the loose refs cache. So let's\nexpose this function via `refs.h`.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 2 +-\n refs.h | 6 ++++++\n 2 files changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex 2f58a3460a..5999605230 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -827,7 +827,7 @@ int is_per_worktree_ref(const char *refname)\n \t starts_with(refname, \"refs/rewritten/\");\n }\n \n-static int is_pseudoref_syntax(const char *refname)\n+int is_pseudoref_syntax(const char *refname)\n {\n \tconst char *c;\n \ndiff --git a/refs.h b/refs.h\nindex ff113bb12a..f1bbad83fb 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -846,6 +846,12 @@ const char **hidden_refs_to_excludes(const struct strvec *hide_refs);\n /* Is this a per-worktree ref living in the refs/ namespace? */\n int is_per_worktree_ref(const char *refname);\n \n+/*\n+ * Check whether a refname matches the pseudoref syntax. This is a surface\n+ * level check and can present false positives.\n+ */\n+int is_pseudoref_syntax(const char *refname);\n+\n /* Describes how a refname relates to worktrees */\n enum ref_worktree_type {\n \tREF_WORKTREE_CURRENT, /* implicitly per worktree, eg. HEAD or\n-- \n2.43.GIT\n\n"},{"id":"487075","messageId":"20240119142705.139374-3-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240119142705.139374-1-karthik.188@gmail.com","subject":"[PATCH 2/5] refs: make `is_pseudoref_syntax()` stricter","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-19T14:27:02Z","receivedAt":"2024-01-19T14:27:11Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `is_pseudoref_syntax()` function is used to determine if a\nparticular refname follows the pseudoref syntax. The pseudoref syntax is\nloosely defined at this instance as all refs which are in caps and use\nunderscores. Most of the pseudorefs also have the \"HEAD\" suffix.\n\nUsing this information we make the `is_pseudoref_syntax()` function\nstricter, by adding the check for \"HEAD\" suffix and for refs which don't\nend with the HEAD suffix, matching them with a predetermined list.\n\nThis requires fixing up t1407 to use the \"HEAD\" suffix for creation of\npseudorefs.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 21 ++++++++++++++++++---\n t/t1407-worktree-ref-store.sh | 12 ++++++------\n 2 files changed, 24 insertions(+), 9 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 5999605230..b84e173762 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -829,6 +829,14 @@ int is_per_worktree_ref(const char *refname)\n \n int is_pseudoref_syntax(const char *refname)\n {\n+\t/* TODO: move these pseudorefs to have _HEAD suffix */\n+\tstatic const char *const irregular_pseudorefs[] = {\n+\t\t\"BISECT_EXPECTED_REV\",\n+\t\t\"NOTES_MERGE_PARTIAL\",\n+\t\t\"NOTES_MERGE_REF\",\n+\t\t\"AUTO_MERGE\"\n+\t};\n+\tsize_t i;\n \tconst char *c;\n \n \tfor (c = refname; *c; c++) {\n@@ -837,10 +845,17 @@ int is_pseudoref_syntax(const char *refname)\n \t}\n \n \t/*\n-\t * HEAD is not a pseudoref, but it certainly uses the\n-\t * pseudoref syntax.\n+\t * Most pseudorefs end with _HEAD. HEAD itself is not a\n+\t * pseudoref, but it certainly uses the pseudoref syntax.\n \t */\n-\treturn 1;\n+\tif (ends_with(refname, \"HEAD\"))\n+\t\treturn 1;\n+\n+\tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n+\t\tif (!strcmp(refname, irregular_pseudorefs[i]))\n+\t\t\treturn 1;\n+\n+\treturn 0;\n }\n \n static int is_current_worktree_ref(const char *ref) {\ndiff --git a/t/t1407-worktree-ref-store.sh b/t/t1407-worktree-ref-store.sh\nindex 05b1881c59..53592c95f3 100755\n--- a/t/t1407-worktree-ref-store.sh\n+++ b/t/t1407-worktree-ref-store.sh\n@@ -61,18 +61,18 @@ test_expect_success 'create_symref(FOO, refs/heads/main)' '\n # PSEUDO-WT and refs/bisect/random do not create reflogs by default, so it is\n # not testing a realistic scenario.\n test_expect_success REFFILES 'for_each_reflog()' '\n-\techo $ZERO_OID > .git/logs/PSEUDO-MAIN &&\n+\techo $ZERO_OID >.git/logs/PSEUDO_MAIN_HEAD &&\n \tmkdir -p .git/logs/refs/bisect &&\n-\techo $ZERO_OID > .git/logs/refs/bisect/random &&\n+\techo $ZERO_OID >.git/logs/refs/bisect/random &&\n \n-\techo $ZERO_OID > .git/worktrees/wt/logs/PSEUDO-WT &&\n+\techo $ZERO_OID >.git/worktrees/wt/logs/PSEUDO_WT_HEAD &&\n \tmkdir -p .git/worktrees/wt/logs/refs/bisect &&\n-\techo $ZERO_OID > .git/worktrees/wt/logs/refs/bisect/wt-random &&\n+\techo $ZERO_OID >.git/worktrees/wt/logs/refs/bisect/wt-random &&\n \n \t$RWT for-each-reflog | cut -d\" \" -f 2- | sort >actual &&\n \tcat >expected <<-\\EOF &&\n \tHEAD 0x1\n-\tPSEUDO-WT 0x0\n+\tPSEUDO_WT_HEAD 0x0\n \trefs/bisect/wt-random 0x0\n \trefs/heads/main 0x0\n \trefs/heads/wt-main 0x0\n@@ -82,7 +82,7 @@ test_expect_success REFFILES 'for_each_reflog()' '\n \t$RMAIN for-each-reflog | cut -d\" \" -f 2- | sort >actual &&\n \tcat >expected <<-\\EOF &&\n \tHEAD 0x1\n-\tPSEUDO-MAIN 0x0\n+\tPSEUDO_MAIN_HEAD 0x0\n \trefs/bisect/random 0x0\n \trefs/heads/main 0x0\n \trefs/heads/wt-main 0x0\n-- \n2.43.GIT\n\n"},{"id":"487074","messageId":"20240119142705.139374-4-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240119142705.139374-1-karthik.188@gmail.com","subject":"[PATCH 3/5] refs: extract out `loose_fill_ref_dir_regular_file()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-19T14:27:03Z","receivedAt":"2024-01-19T14:27:12Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Extract out the code for adding a single file to the loose ref dir as\n`loose_fill_ref_dir_regular_file()` from `loose_fill_ref_dir()` in\n`refs/files-backend.c`.\n\nThis allows us to use this function independently in the following\ncommits where we add code to also add pseudorefs to the ref dir.\n\nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 62 +++++++++++++++++++++++---------------------\n 1 file changed, 33 insertions(+), 29 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 6734f2a309..a4884f557d 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -229,6 +229,38 @@ static void add_per_worktree_entries_to_dir(struct ref_dir *dir, const char *dir\n \t}\n }\n \n+static void loose_fill_ref_dir_regular_file(struct files_ref_store *refs,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t struct ref_dir *dir)\n+{\n+\tstruct object_id oid;\n+\tint flag;\n+\n+\tif (!refs_resolve_ref_unsafe(&refs->base, refname, RESOLVE_REF_READING,\n+\t\t\t\t &oid, &flag)) {\n+\t\toidclr(&oid);\n+\t\tflag |= REF_ISBROKEN;\n+\t} else if (is_null_oid(&oid)) {\n+\t\t/*\n+\t\t * It is so astronomically unlikely\n+\t\t * that null_oid is the OID of an\n+\t\t * actual object that we consider its\n+\t\t * appearance in a loose reference\n+\t\t * file to be repo corruption\n+\t\t * (probably due to a software bug).\n+\t\t */\n+\t\tflag |= REF_ISBROKEN;\n+\t}\n+\n+\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tif (!refname_is_safe(refname))\n+\t\t\tdie(\"loose refname is dangerous: %s\", refname);\n+\t\toidclr(&oid);\n+\t\tflag |= REF_BAD_NAME | REF_ISBROKEN;\n+\t}\n+\tadd_entry_to_dir(dir, create_ref_entry(refname, &oid, flag));\n+}\n+\n /*\n * Read the loose references from the namespace dirname into dir\n * (without recursing). dirname must end with '/'. dir must be the\n@@ -257,8 +289,6 @@ static void loose_fill_ref_dir(struct ref_store *ref_store,\n \tstrbuf_add(&refname, dirname, dirnamelen);\n \n \twhile ((de = readdir(d)) != NULL) {\n-\t\tstruct object_id oid;\n-\t\tint flag;\n \t\tunsigned char dtype;\n \n \t\tif (de->d_name[0] == '.')\n@@ -274,33 +304,7 @@ static void loose_fill_ref_dir(struct ref_store *ref_store,\n \t\t\t\t\t create_dir_entry(dir->cache, refname.buf,\n \t\t\t\t\t\t\t refname.len));\n \t\t} else if (dtype == DT_REG) {\n-\t\t\tif (!refs_resolve_ref_unsafe(&refs->base,\n-\t\t\t\t\t\t refname.buf,\n-\t\t\t\t\t\t RESOLVE_REF_READING,\n-\t\t\t\t\t\t &oid, &flag)) {\n-\t\t\t\toidclr(&oid);\n-\t\t\t\tflag |= REF_ISBROKEN;\n-\t\t\t} else if (is_null_oid(&oid)) {\n-\t\t\t\t/*\n-\t\t\t\t * It is so astronomically unlikely\n-\t\t\t\t * that null_oid is the OID of an\n-\t\t\t\t * actual object that we consider its\n-\t\t\t\t * appearance in a loose reference\n-\t\t\t\t * file to be repo corruption\n-\t\t\t\t * (probably due to a software bug).\n-\t\t\t\t */\n-\t\t\t\tflag |= REF_ISBROKEN;\n-\t\t\t}\n-\n-\t\t\tif (check_refname_format(refname.buf,\n-\t\t\t\t\t\t REFNAME_ALLOW_ONELEVEL)) {\n-\t\t\t\tif (!refname_is_safe(refname.buf))\n-\t\t\t\t\tdie(\"loose refname is dangerous: %s\", refname.buf);\n-\t\t\t\toidclr(&oid);\n-\t\t\t\tflag |= REF_BAD_NAME | REF_ISBROKEN;\n-\t\t\t}\n-\t\t\tadd_entry_to_dir(dir,\n-\t\t\t\t\t create_ref_entry(refname.buf, &oid, flag));\n+\t\t\tloose_fill_ref_dir_regular_file(refs, refname.buf, dir);\n \t\t}\n \t\tstrbuf_setlen(&refname, dirnamelen);\n \t}\n-- \n2.43.GIT\n\n"},{"id":"487076","messageId":"20240119142705.139374-5-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240119142705.139374-1-karthik.188@gmail.com","subject":"[PATCH 4/5] refs: introduce `refs_for_each_all_refs()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-19T14:27:04Z","receivedAt":"2024-01-19T14:27:13Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Introduce a new ref iteration flag `DO_FOR_EACH_INCLUDE_ALL_REFS`, which\nwill be used to iterate over all refs. In the files backend this is\nlimited to regular refs, pseudorefs and HEAD. For other backends like\nthe reftable this is the universal set of all refs stored in the\nbackend.\n\nRefs which fall outside the `refs/` and aren't either pseudorefs or HEAD\nare more of a grey area. This is because we don't block the users from\ncreating such refs but they are not officially supported. In the files\nbackend, we cannot isolate such files from other files.\n\nIntroduce `refs_for_each_all_refs()` which calls `do_for_each_ref()`\nwith this newly introduced flag.\n\nIn `refs/files-backend.c`, introduce a new function\n`add_pseudoref_like_entries()` to add pseudorefs and HEAD to the\n`ref_dir`. We then finally call `add_pseudoref_like_entries()` whenever\nthe `DO_FOR_EACH_INCLUDE_ALL_REFS` flag is set. Any new ref backend will\nalso have to implement similar changes on its end.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 7 +++++\n refs.h | 6 +++++\n refs/files-backend.c | 64 ++++++++++++++++++++++++++++++++++++++++----\n refs/refs-internal.h | 7 +++++\n 4 files changed, 79 insertions(+), 5 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex b84e173762..e527c6199d 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1722,6 +1722,13 @@ int for_each_rawref(each_ref_fn fn, void *cb_data)\n \treturn refs_for_each_rawref(get_main_ref_store(the_repository), fn, cb_data);\n }\n \n+int refs_for_each_all_refs(struct ref_store *refs, each_ref_fn fn,\n+\t\t\t void *cb_data)\n+{\n+\treturn do_for_each_ref(refs, \"\", NULL, fn, 0,\n+\t\t\t DO_FOR_EACH_INCLUDE_ALL_REFS, cb_data);\n+}\n+\n static int qsort_strcmp(const void *va, const void *vb)\n {\n \tconst char *a = *(const char **)va;\ndiff --git a/refs.h b/refs.h\nindex f1bbad83fb..fef70b6599 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -393,6 +393,12 @@ int for_each_namespaced_ref(const char **exclude_patterns,\n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data);\n int for_each_rawref(each_ref_fn fn, void *cb_data);\n \n+/*\n+ * Iterates over all ref types, regular, pseudorefs and HEAD.\n+ */\n+int refs_for_each_all_refs(struct ref_store *refs, each_ref_fn fn,\n+\t\t\t void *cb_data);\n+\n /*\n * Normalizes partial refs to their fully qualified form.\n * Will prepend to the if it doesn't start with 'refs/'.\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex a4884f557d..95a73b11bb 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -315,9 +315,58 @@ static void loose_fill_ref_dir(struct ref_store *ref_store,\n \tadd_per_worktree_entries_to_dir(dir, dirname);\n }\n \n-static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs)\n+/*\n+ * Add pseudorefs and HEAD to the ref dir by parsing the directory\n+ * for any files which follow the pseudoref syntax.\n+ */\n+static void add_pseudoref_like_entries(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_dir *dir,\n+\t\t\t\t\t const char *dirname)\n+{\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fill_ref_dir\");\n+\tstruct strbuf path = STRBUF_INIT, refname = STRBUF_INIT;\n+\tstruct dirent *de;\n+\tsize_t dirnamelen;\n+\tDIR *d;\n+\n+\tfiles_ref_path(refs, &path, dirname);\n+\n+\td = opendir(path.buf);\n+\tif (!d) {\n+\t\tstrbuf_release(&path);\n+\t\treturn;\n+\t}\n+\n+\tstrbuf_addstr(&refname, dirname);\n+\tdirnamelen = refname.len;\n+\n+\twhile ((de = readdir(d)) != NULL) {\n+\t\tunsigned char dtype;\n+\n+\t\tif (de->d_name[0] == '.')\n+\t\t\tcontinue;\n+\t\tif (ends_with(de->d_name, \".lock\"))\n+\t\t\tcontinue;\n+\t\tstrbuf_addstr(&refname, de->d_name);\n+\n+\t\tdtype = get_dtype(de, &path, 1);\n+\t\tif (dtype == DT_REG && is_pseudoref_syntax(de->d_name))\n+\t\t\tloose_fill_ref_dir_regular_file(refs, refname.buf, dir);\n+\n+\t\tstrbuf_setlen(&refname, dirnamelen);\n+\t}\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&path);\n+\tclosedir(d);\n+}\n+\n+static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs,\n+\t\t\t\t\t unsigned int flags)\n {\n \tif (!refs->loose) {\n+\t\tstruct ref_dir *dir;\n+\n \t\t/*\n \t\t * Mark the top-level directory complete because we\n \t\t * are about to read the only subdirectory that can\n@@ -328,12 +377,17 @@ static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs)\n \t\t/* We're going to fill the top level ourselves: */\n \t\trefs->loose->root->flag &= ~REF_INCOMPLETE;\n \n+\t\tdir = get_ref_dir(refs->loose->root);\n+\n+\t\tif (flags & DO_FOR_EACH_INCLUDE_ALL_REFS)\n+\t\t\tadd_pseudoref_like_entries(dir->cache->ref_store, dir,\n+\t\t\t\t\t\t refs->loose->root->name);\n+\n \t\t/*\n \t\t * Add an incomplete entry for \"refs/\" (to be filled\n \t\t * lazily):\n \t\t */\n-\t\tadd_entry_to_dir(get_ref_dir(refs->loose->root),\n-\t\t\t\t create_dir_entry(refs->loose, \"refs/\", 5));\n+\t\tadd_entry_to_dir(dir, create_dir_entry(refs->loose, \"refs/\", 5));\n \t}\n \treturn refs->loose;\n }\n@@ -861,7 +915,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \t * disk, and re-reads it if not.\n \t */\n \n-\tloose_iter = cache_ref_iterator_begin(get_loose_ref_cache(refs),\n+\tloose_iter = cache_ref_iterator_begin(get_loose_ref_cache(refs, flags),\n \t\t\t\t\t prefix, ref_store->repo, 1);\n \n \t/*\n@@ -1222,7 +1276,7 @@ static int files_pack_refs(struct ref_store *ref_store,\n \n \tpacked_refs_lock(refs->packed_ref_store, LOCK_DIE_ON_ERROR, &err);\n \n-\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs), NULL,\n+\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs, 0), NULL,\n \t\t\t\t\tthe_repository, 0);\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t/*\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 4af83bf9a5..981a91c4c6 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -260,6 +260,13 @@ enum do_for_each_ref_flags {\n \t * INCLUDE_BROKEN, since they are otherwise not included at all.\n \t */\n \tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n+\n+\t/*\n+\t * Include all refs in the $GIT_DIR in contrast to generally only listing\n+\t * references having the \"refs/\" prefix. In the files-backend this is\n+\t * limited to regular refs, pseudorefs and HEAD.\n+\t */\n+\tDO_FOR_EACH_INCLUDE_ALL_REFS = (1 << 3),\n };\n \n /*\n-- \n2.43.GIT\n\n"},{"id":"487077","messageId":"20240119142705.139374-6-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240119142705.139374-1-karthik.188@gmail.com","subject":"[PATCH 5/5] for-each-ref: avoid filtering on empty pattern","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-19T14:27:05Z","receivedAt":"2024-01-19T14:27:13Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"When the user uses an empty string pattern (\"\"), we don't match any refs\nin git-for-each-ref(1). This is because in git-for-each-ref(1), we use\npath based matching and an empty string doesn't match any path.\n\nIn this commit we change this behavior by making empty string pattern\nmatch all references. This is done by introducing a new flag\n`FILTER_REFS_NO_FILTER` in `ref-filter.c`, which uses the newly\nintroduced `refs_for_each_all_refs()` function to iterate over all the\nrefs in the repository.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-for-each-ref.txt | 3 ++-\n builtin/for-each-ref.c | 21 +++++++++++++++++-\n ref-filter.c | 13 ++++++++++--\n ref-filter.h | 4 +++-\n t/t6302-for-each-ref-filter.sh | 34 ++++++++++++++++++++++++++++++\n 5 files changed, 70 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex be9543f684..b1cb482bf5 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -32,7 +32,8 @@ OPTIONS\n \tIf one or more patterns are given, only refs are shown that\n \tmatch against at least one pattern, either using fnmatch(3) or\n \tliterally, in the latter case matching completely or from the\n-\tbeginning up to a slash.\n+\tbeginning up to a slash. If an empty string is provided all refs\n+\tare printed, including HEAD and pseudorefs.\n \n --stdin::\n \tIf `--stdin` is supplied, then the list of patterns is read from\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 3885a9c28e..5aa879e8be 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -25,6 +25,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstruct ref_format format = REF_FORMAT_INIT;\n \tint from_stdin = 0;\n \tstruct strvec vec = STRVEC_INIT;\n+\tunsigned int flags = FILTER_REFS_ALL;\n \n \tstruct option opts[] = {\n \t\tOPT_BIT('s', \"shell\", &format.quote_style,\n@@ -93,11 +94,29 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\t/* vec.v is NULL-terminated, just like 'argv'. */\n \t\tfilter.name_patterns = vec.v;\n \t} else {\n+\t\tsize_t i;\n+\n \t\tfilter.name_patterns = argv;\n+\n+\t\t/*\n+\t\t * Search for any empty string pattern, if it exists then we\n+\t\t * print all refs without any filtering.\n+\t\t */\n+\t\ti = 0;\n+\t\twhile (argv[i]) {\n+\t\t\tif (!argv[i][0]) {\n+\t\t\t\tflags = FILTER_REFS_NO_FILTER;\n+\t\t\t\t/* doing this removes any pattern from being matched */\n+\t\t\t\tfilter.name_patterns[0] = NULL;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\ti++;\n+\t\t}\n \t}\n \n \tfilter.match_as_path = 1;\n-\tfilter_and_format_refs(&filter, FILTER_REFS_ALL, sorting, &format);\n+\tfilter_and_format_refs(&filter, flags, sorting, &format);\n \n \tref_filter_clear(&filter);\n \tref_sorting_release(sorting);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 35b989e1df..6dac133b87 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2622,6 +2622,11 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t\t\t each_ref_fn cb,\n \t\t\t\t void *cb_data)\n {\n+\tif (filter->kind & FILTER_REFS_NO_FILTER) {\n+\t\treturn refs_for_each_all_refs(\n+\t\t\tget_main_ref_store(the_repository), cb, cb_data);\n+\t}\n+\n \tif (!filter->match_as_path) {\n \t\t/*\n \t\t * in this case, the patterns are applied after\n@@ -2775,8 +2780,12 @@ static struct ref_array_item *apply_ref_filter(const char *refname, const struct\n \n \t/* Obtain the current ref kind from filter_ref_kind() and ignore unwanted refs. */\n \tkind = filter_ref_kind(filter, refname);\n-\tif (!(kind & filter->kind))\n+\tif (filter->kind & FILTER_REFS_NO_FILTER) {\n+\t\tif (kind == FILTER_REFS_DETACHED_HEAD)\n+\t\t\tkind = FILTER_REFS_OTHERS;\n+\t} else if (!(kind & filter->kind)) {\n \t\treturn NULL;\n+\t}\n \n \tif (!filter_pattern_match(filter, refname))\n \t\treturn NULL;\n@@ -3041,7 +3050,7 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \t\t\tret = for_each_fullref_in(\"refs/remotes/\", fn, cb_data);\n \t\telse if (filter->kind == FILTER_REFS_TAGS)\n \t\t\tret = for_each_fullref_in(\"refs/tags/\", fn, cb_data);\n-\t\telse if (filter->kind & FILTER_REFS_ALL)\n+\t\telse if (filter->kind & FILTER_REFS_ALL || filter->kind & FILTER_REFS_NO_FILTER)\n \t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n \t\tif (!ret && (filter->kind & FILTER_REFS_DETACHED_HEAD))\n \t\t\thead_ref(fn, cb_data);\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 07cd6f6da3..1eab325ce0 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -22,7 +22,9 @@\n #define FILTER_REFS_ALL (FILTER_REFS_TAGS | FILTER_REFS_BRANCHES | \\\n \t\t\t\t FILTER_REFS_REMOTES | FILTER_REFS_OTHERS)\n #define FILTER_REFS_DETACHED_HEAD 0x0020\n-#define FILTER_REFS_KIND_MASK (FILTER_REFS_ALL | FILTER_REFS_DETACHED_HEAD)\n+#define FILTER_REFS_NO_FILTER 0x0040\n+#define FILTER_REFS_KIND_MASK (FILTER_REFS_ALL | FILTER_REFS_DETACHED_HEAD | \\\n+\t\t\t\t FILTER_REFS_NO_FILTER)\n \n struct atom_value;\n struct ref_sorting;\ndiff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh\nindex 82f3d1ea0f..3922326cab 100755\n--- a/t/t6302-for-each-ref-filter.sh\n+++ b/t/t6302-for-each-ref-filter.sh\n@@ -31,6 +31,40 @@ test_expect_success 'setup some history and refs' '\n \tgit update-ref refs/odd/spot main\n '\n \n+cat >expect <<-\\EOF\n+\tHEAD\n+\tORIG_HEAD\n+\trefs/heads/main\n+\trefs/heads/side\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+EOF\n+\n+test_expect_success 'empty pattern prints pseudorefs' '\n+\tgit update-ref ORIG_HEAD main &&\n+\tgit for-each-ref --format=\"%(refname)\" \"\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'empty pattern with other patterns' '\n+\tgit update-ref ORIG_HEAD main &&\n+\tgit for-each-ref --format=\"%(refname)\" \"\" \"refs/\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'empty pattern towards the end' '\n+\tgit update-ref ORIG_HEAD main &&\n+\tgit for-each-ref --format=\"%(refname)\" \"refs/\" \"\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'filtering with --points-at' '\n \tcat >expect <<-\\EOF &&\n \trefs/heads/main\n-- \n2.43.GIT\n\n"},{"id":"487104","messageId":"xmqq34utjbz9.fsf@gitster.g","threadId":"60768","inReplyTo":"20240119142705.139374-2-karthik.188@gmail.com","subject":"Re: [PATCH 1/5] refs: expose `is_pseudoref_syntax()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-19T20:37:46Z","receivedAt":"2024-01-19T20:37:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> The `is_pseudoref_syntax()` function is static, since it is only used\n> within `refs.c`. In the following commit, we will use this function to\n> provide an utility to add pseudorefs to the loose refs cache. So let's\n> expose this function via `refs.h`.\n>\n> Signed-off-by: Karthik Nayak \n> ---\n> refs.c | 2 +-\n> refs.h | 6 ++++++\n> 2 files changed, 7 insertions(+), 1 deletion(-)\n>\n> diff --git a/refs.c b/refs.c\n> index 2f58a3460a..5999605230 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -827,7 +827,7 @@ int is_per_worktree_ref(const char *refname)\n> \t starts_with(refname, \"refs/rewritten/\");\n> }\n> \n> -static int is_pseudoref_syntax(const char *refname)\n> +int is_pseudoref_syntax(const char *refname)\n> {\n> \tconst char *c;\n> \n> diff --git a/refs.h b/refs.h\n> index ff113bb12a..f1bbad83fb 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -846,6 +846,12 @@ const char **hidden_refs_to_excludes(const struct strvec *hide_refs);\n> /* Is this a per-worktree ref living in the refs/ namespace? */\n> int is_per_worktree_ref(const char *refname);\n> \n> +/*\n> + * Check whether a refname matches the pseudoref syntax. This is a surface\n> + * level check and can present false positives.\n> + */\n\nWhat does \"false positive\" mean in this context?\n\nis_pseudoref_syntax(\"FOO_HEAD\") says \"true\", and then if it is\n\"false positive\", that would mean \"FOO_HEAD\" is not a pseudo ref,\nright? What can a caller of this function do to deal with a false\npositive?\n\nOr do you mean \"FOO_HEAD\" is still a pseudo ref, but it may not\ncurrently exist? That is different from \"false positive\".\n\nAs the check is about \"does it match the pseudoref syntax?\", I would\nunderstand if what you wanted to say was something like: This only\nchecks the syntax, and such a pseudoref may not currently exist in\nthe repository---for that you'd need to call read_ref_full() or\nother ref API functions.\n\nPuzzled...\n\nThanks.\n"},{"id":"487105","messageId":"xmqqwms5hx3j.fsf@gitster.g","threadId":"60768","inReplyTo":"20240119142705.139374-3-karthik.188@gmail.com","subject":"Re: [PATCH 2/5] refs: make `is_pseudoref_syntax()` stricter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-19T20:44:32Z","receivedAt":"2024-01-19T20:44:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Using this information we make the `is_pseudoref_syntax()` function\n> stricter, by adding the check for \"HEAD\" suffix and for refs which don't\n> end with the HEAD suffix, matching them with a predetermined list.\n\nOK, so this partly answers my question on the previous step. Before\nmaking it more strict, the function worked only on the \"syntax\", so\na random string that can be a pseudo ref passed the check.\n\nBut stepping back a bit, if we call this function is_pseudoref_syntax(),\nwouldn't it be what we want to have anyway? You seem to want a\nseparate function called is_pseudoref() that rejects bogus uppercase\nstring \"FOO_BAR\" while accepting the known-good pseudoref you add\ntests for, plus the ${FOO}_HEAD for any value of ${FOO} we currently\nhave and we may want to add in the future.\n\n> int is_pseudoref_syntax(const char *refname)\n> {\n> +\t/* TODO: move these pseudorefs to have _HEAD suffix */\n> +\tstatic const char *const irregular_pseudorefs[] = {\n> +\t\t\"BISECT_EXPECTED_REV\",\n> +\t\t\"NOTES_MERGE_PARTIAL\",\n> +\t\t\"NOTES_MERGE_REF\",\n> +\t\t\"AUTO_MERGE\"\n> +\t};\n> +\tsize_t i;\n> \tconst char *c;\n> \n> \tfor (c = refname; *c; c++) {\n> @@ -837,10 +845,17 @@ int is_pseudoref_syntax(const char *refname)\n> \t}\n> \n> \t/*\n> -\t * HEAD is not a pseudoref, but it certainly uses the\n> -\t * pseudoref syntax.\n> +\t * Most pseudorefs end with _HEAD. HEAD itself is not a\n> +\t * pseudoref, but it certainly uses the pseudoref syntax.\n> \t */\n> -\treturn 1;\n> +\tif (ends_with(refname, \"HEAD\"))\n> +\t\treturn 1;\n\nI would imagine that at the final stage in which something like this\nwill be named is_pseudoref(), asking is_pseudoref(\"HEAD\") would\nreturn \"No\" (even though \"is_pseudoref_syntax()\", if the helper\nfunction remains, may say \"Yes\" to \"HEAD\"). And this ends_with()\nwill use \"_HEAD\", instead of \"HEAD\". But I am reading ahead of\nmyself, so let's keep going.\n\n> +\tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n> +\t\tif (!strcmp(refname, irregular_pseudorefs[i]))\n> +\t\t\treturn 1;\n> +\n> +\treturn 0;\n> }\n"},{"id":"487106","messageId":"xmqqsf2thwhj.fsf@gitster.g","threadId":"60768","inReplyTo":"20240119142705.139374-5-karthik.188@gmail.com","subject":"Re: [PATCH 4/5] refs: introduce `refs_for_each_all_refs()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-19T20:57:44Z","receivedAt":"2024-01-19T20:57:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> +static void add_pseudoref_like_entries(struct ref_store *ref_store,\n> +\t\t\t\t\t struct ref_dir *dir,\n> +\t\t\t\t\t const char *dirname)\n> +{\n> +\tstruct files_ref_store *refs =\n> +\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fill_ref_dir\");\n> +\tstruct strbuf path = STRBUF_INIT, refname = STRBUF_INIT;\n> +\tstruct dirent *de;\n> +\tsize_t dirnamelen;\n> +\tDIR *d;\n> +\n> +\tfiles_ref_path(refs, &path, dirname);\n> +\n> +\td = opendir(path.buf);\n> +\tif (!d) {\n> +\t\tstrbuf_release(&path);\n> +\t\treturn;\n> +\t}\n> +\n> +\tstrbuf_addstr(&refname, dirname);\n> +\tdirnamelen = refname.len;\n> +\n> +\twhile ((de = readdir(d)) != NULL) {\n> +\t\tunsigned char dtype;\n> +\n> +\t\tif (de->d_name[0] == '.')\n> +\t\t\tcontinue;\n> +\t\tif (ends_with(de->d_name, \".lock\"))\n> +\t\t\tcontinue;\n> +\t\tstrbuf_addstr(&refname, de->d_name);\n> +\n> +\t\tdtype = get_dtype(de, &path, 1);\n> +\t\tif (dtype == DT_REG && is_pseudoref_syntax(de->d_name))\n> +\t\t\tloose_fill_ref_dir_regular_file(refs, refname.buf, dir);\n\nThis looks more like add_pseudoref_entries() given that the general\ndirection is to have an \"allow\" list of pseudo refs (at this point\nafter the previous step of the series, is_pseudoref_syntax() is the\nis_pseudoref() function, and uses ends_with(\"_HEAD\") as a mere\noptimization to avoid listing all the possible pseudo refs that\nexists or will be added in the future whose name ends with \"_HEAD\").\n\nOther than the naming, I think these two steps make sense.\n\nThanks.\n"},{"id":"487205","messageId":"CAOLa=ZQNKCabBBx9OEmZzfEcpzmTSAh3do4EgoqMENiQW+Axgg@mail.gmail.com","threadId":"60768","inReplyTo":"xmqq34utjbz9.fsf@gitster.g","subject":"Re: [PATCH 1/5] refs: expose `is_pseudoref_syntax()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-22T15:40:11Z","receivedAt":"2024-01-22T15:40:38Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"On Fri, Jan 19, 2024 at 9:37 PM Junio C Hamano wrote:\n> >\n> > +/*\n> > + * Check whether a refname matches the pseudoref syntax. This is a surface\n> > + * level check and can present false positives.\n> > + */\n>\n> What does \"false positive\" mean in this context?\n>\n> is_pseudoref_syntax(\"FOO_HEAD\") says \"true\", and then if it is\n> \"false positive\", that would mean \"FOO_HEAD\" is not a pseudo ref,\n> right? What can a caller of this function do to deal with a false\n> positive?\n>\n> Or do you mean \"FOO_HEAD\" is still a pseudo ref, but it may not\n> currently exist? That is different from \"false positive\".\n>\n\nYes, I think this is what I wanted to say and what you say below is what I'll\nchange it to.\n\n> As the check is about \"does it match the pseudoref syntax?\", I would\n> understand if what you wanted to say was something like: This only\n> checks the syntax, and such a pseudoref may not currently exist in\n> the repository---for that you'd need to call read_ref_full() or\n> other ref API functions.\n>\n> Puzzled...\n>\n> Thanks.\n\nThanks!\n"},{"id":"487207","messageId":"CAOLa=ZTbtqvejpvNVY5MHU=Adx3tWQ=FqVJdRLG1gaxYu4BG7A@mail.gmail.com","threadId":"60768","inReplyTo":"xmqqsf2thwhj.fsf@gitster.g","subject":"Re: [PATCH 4/5] refs: introduce `refs_for_each_all_refs()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-22T15:48:51Z","receivedAt":"2024-01-22T15:49:18Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"On Fri, Jan 19, 2024 at 9:57 PM Junio C Hamano wrote:\n>\n> This looks more like add_pseudoref_entries() given that the general\n> direction is to have an \"allow\" list of pseudo refs (at this point\n> after the previous step of the series, is_pseudoref_syntax() is the\n> is_pseudoref() function, and uses ends_with(\"_HEAD\") as a mere\n> optimization to avoid listing all the possible pseudo refs that\n> exists or will be added in the future whose name ends with \"_HEAD\").\n>\n> Other than the naming, I think these two steps make sense.\n\nI think overall the naming is correct, I would change the comments in\n`is_pseudoref_syntax()`.\n\nBecause, apart from pseudorefs, we also want to print HEAD. This is also\nwhy the pattern matches \"HEAD\" instead of \"_HEAD\". I'll add some more\ncomments to clarify this.\n\nSo to summarize:\n1. `is_pseudoref_syntax()` checks for pseudoref like syntax, this also matches\nHEAD. Will add comments here to clarify that we do not actually check\nref contents.\n2. `add_pseudoref_like_entries()` adds pseudorefs and HEAD to the loose refs\ncache.\n"},{"id":"487210","messageId":"xmqqfrypuurw.fsf@gitster.g","threadId":"60768","inReplyTo":"CAOLa=ZTbtqvejpvNVY5MHU=Adx3tWQ=FqVJdRLG1gaxYu4BG7A@mail.gmail.com","subject":"Re: [PATCH 4/5] refs: introduce `refs_for_each_all_refs()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-22T17:45:23Z","receivedAt":"2024-01-22T17:45:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> On Fri, Jan 19, 2024 at 9:57 PM Junio C Hamano wrote:\n>>\n>> This looks more like add_pseudoref_entries() given that the general\n>> direction is to have an \"allow\" list of pseudo refs (at this point\n>> after the previous step of the series, is_pseudoref_syntax() is the\n>> is_pseudoref() function, and uses ends_with(\"_HEAD\") as a mere\n>> optimization to avoid listing all the possible pseudo refs that\n>> exists or will be added in the future whose name ends with \"_HEAD\").\n>>\n>> Other than the naming, I think these two steps make sense.\n>\n> I think overall the naming is correct, I would change the comments in\n> `is_pseudoref_syntax()`.\n>\n> Because, apart from pseudorefs, we also want to print HEAD. This is also\n> why the pattern matches \"HEAD\" instead of \"_HEAD\". I'll add some more\n> comments to clarify this.\n\nWith the hardcoded \"these are definitely pseudorefs\" list in the\nfunction, it no longer is is_pseudoref_SYNTAX() at all. I would\nrather prefer to see is_pseudoref() that says no to HEAD and have\nthe callers check\n\n\t-\tif (is_pseudoref_syntax(foo))\n\t+\tif (is_pseudoref(foo) || is_headref(foo))\n\nthan keeping the messy semantics we have. My second preference is\nto call it is_pseudoref_or_head() that says yes to \"HEAD\" and\npseudorefs, even though I like it much less.\n\nSimilarly, between giving the function under discussion a more\ndescriptive name add_pseudoref_and_head_entries(), or adding a new\nfunction add_head_entry() to make the callers call add_head_entry()\nand add_pseudoref_entries() separately, I have a slight preference\nfor the latter.\n\nThanks.\n"},{"id":"487216","messageId":"ee977173-bc6d-48f6-9bc8-e1d84fe3d95d@gmail.com","threadId":"60768","inReplyTo":"20240119142705.139374-3-karthik.188@gmail.com","subject":"Re: [PATCH 2/5] refs: make `is_pseudoref_syntax()` stricter","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2024-01-22T20:13:22Z","receivedAt":"2024-01-22T20:13:25Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Karthik\n\nOn 19/01/2024 14:27, Karthik Nayak wrote:\n> The `is_pseudoref_syntax()` function is used to determine if a\n> particular refname follows the pseudoref syntax. The pseudoref syntax is\n> loosely defined at this instance as all refs which are in caps and use\n> underscores. Most of the pseudorefs also have the \"HEAD\" suffix.\n> \n> Using this information we make the `is_pseudoref_syntax()` function\n> stricter, by adding the check for \"HEAD\" suffix and for refs which don't\n> end with the HEAD suffix, matching them with a predetermined list.\n> \n> This requires fixing up t1407 to use the \"HEAD\" suffix for creation of\n> pseudorefs.\n\nI'm concerned that this change is a regression. is_pseudoref_syntax() is \nused by is_current_worktree_ref() and so scripts that create pseudorefs \nthat do not conform to your new rules will break as git will no-longer \nconsider the pseudorefs they create to be worktree specific. The list of \nhard coded exceptions also looks quite short, I can see MERGE_AUTOSTASH \nand BISECT_START are missing and there are probably others I've not \nthought of.\n\nThe commit message would be a good place to discuss why you're making \nthis change, the implications of the change and any alternative \napproaches that you considered. As I understand it you're tying to get \nround the problem that the files backend stores pseudorefs mixed up with \nother non-ref files in $GIT_DIR. Another approach would be to read all \nthe files whose name matches the pseudoref syntax and see if its \ncontents looks like a valid ref skipping names like COMMIT_EDITMSG that \nwe know are not pseudorefs.\n\nBest Wishes\n\nPhillip\n\n> Signed-off-by: Karthik Nayak \n> ---\n> refs.c | 21 ++++++++++++++++++---\n> t/t1407-worktree-ref-store.sh | 12 ++++++------\n> 2 files changed, 24 insertions(+), 9 deletions(-)\n> \n> diff --git a/refs.c b/refs.c\n> index 5999605230..b84e173762 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -829,6 +829,14 @@ int is_per_worktree_ref(const char *refname)\n> \n> int is_pseudoref_syntax(const char *refname)\n> {\n> +\t/* TODO: move these pseudorefs to have _HEAD suffix */\n> +\tstatic const char *const irregular_pseudorefs[] = {\n> +\t\t\"BISECT_EXPECTED_REV\",\n> +\t\t\"NOTES_MERGE_PARTIAL\",\n> +\t\t\"NOTES_MERGE_REF\",\n> +\t\t\"AUTO_MERGE\"\n> +\t};\n> +\tsize_t i;\n> \tconst char *c;\n> \n> \tfor (c = refname; *c; c++) {\n> @@ -837,10 +845,17 @@ int is_pseudoref_syntax(const char *refname)\n> \t}\n> \n> \t/*\n> -\t * HEAD is not a pseudoref, but it certainly uses the\n> -\t * pseudoref syntax.\n> +\t * Most pseudorefs end with _HEAD. HEAD itself is not a\n> +\t * pseudoref, but it certainly uses the pseudoref syntax.\n> \t */\n> -\treturn 1;\n> +\tif (ends_with(refname, \"HEAD\"))\n> +\t\treturn 1;\n> +\n> +\tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n> +\t\tif (!strcmp(refname, irregular_pseudorefs[i]))\n> +\t\t\treturn 1;\n> +\n> +\treturn 0;\n> }\n> \n> static int is_current_worktree_ref(const char *ref) {\n> diff --git a/t/t1407-worktree-ref-store.sh b/t/t1407-worktree-ref-store.sh\n> index 05b1881c59..53592c95f3 100755\n> --- a/t/t1407-worktree-ref-store.sh\n> +++ b/t/t1407-worktree-ref-store.sh\n> @@ -61,18 +61,18 @@ test_expect_success 'create_symref(FOO, refs/heads/main)' '\n> # PSEUDO-WT and refs/bisect/random do not create reflogs by default, so it is\n> # not testing a realistic scenario.\n> test_expect_success REFFILES 'for_each_reflog()' '\n> -\techo $ZERO_OID > .git/logs/PSEUDO-MAIN &&\n> +\techo $ZERO_OID >.git/logs/PSEUDO_MAIN_HEAD &&\n> \tmkdir -p .git/logs/refs/bisect &&\n> -\techo $ZERO_OID > .git/logs/refs/bisect/random &&\n> +\techo $ZERO_OID >.git/logs/refs/bisect/random &&\n> \n> -\techo $ZERO_OID > .git/worktrees/wt/logs/PSEUDO-WT &&\n> +\techo $ZERO_OID >.git/worktrees/wt/logs/PSEUDO_WT_HEAD &&\n> \tmkdir -p .git/worktrees/wt/logs/refs/bisect &&\n> -\techo $ZERO_OID > .git/worktrees/wt/logs/refs/bisect/wt-random &&\n> +\techo $ZERO_OID >.git/worktrees/wt/logs/refs/bisect/wt-random &&\n> \n> \t$RWT for-each-reflog | cut -d\" \" -f 2- | sort >actual &&\n> \tcat >expected <<-\\EOF &&\n> \tHEAD 0x1\n> -\tPSEUDO-WT 0x0\n> +\tPSEUDO_WT_HEAD 0x0\n> \trefs/bisect/wt-random 0x0\n> \trefs/heads/main 0x0\n> \trefs/heads/wt-main 0x0\n> @@ -82,7 +82,7 @@ test_expect_success REFFILES 'for_each_reflog()' '\n> \t$RMAIN for-each-reflog | cut -d\" \" -f 2- | sort >actual &&\n> \tcat >expected <<-\\EOF &&\n> \tHEAD 0x1\n> -\tPSEUDO-MAIN 0x0\n> +\tPSEUDO_MAIN_HEAD 0x0\n> \trefs/bisect/random 0x0\n> \trefs/heads/main 0x0\n> \trefs/heads/wt-main 0x0\n"},{"id":"487219","messageId":"xmqqplxtrucm.fsf@gitster.g","threadId":"60768","inReplyTo":"ee977173-bc6d-48f6-9bc8-e1d84fe3d95d@gmail.com","subject":"Re: [PATCH 2/5] refs: make `is_pseudoref_syntax()` stricter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-22T20:22:49Z","receivedAt":"2024-01-22T20:22:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n> I'm concerned that this change is a regression. is_pseudoref_syntax()\n> is used by is_current_worktree_ref() and so scripts that create\n> pseudorefs that do not conform to your new rules will break as git\n> will no-longer consider the pseudorefs they create to be worktree\n> specific.\n\nIdeally, when the exception list in the function becomes more\ncomplete, those \"pseudorefs\" created by those scripts shouldn't\nprobably be created either as common or worktree specific thing\nif they are not \"pseudoref\".\n\n> The list of hard coded exceptions also looks quite short, I\n> can see MERGE_AUTOSTASH and BISECT_START are missing and there are\n> probably others I've not thought of.\n\nI agree that it is something we need to fix.\n\n> The commit message would be a good place to discuss why you're making\n> this change, the implications of the change and any alternative\n> approaches that you considered. As I understand it you're tying to get\n> round the problem that the files backend stores pseudorefs mixed up\n> with other non-ref files in $GIT_DIR.\n\nYup. The rationale may want to be explained better.\n\n> Another approach would be to\n> read all the files whose name matches the pseudoref syntax and see if\n> its contents looks like a valid ref skipping names like COMMIT_EDITMSG\n> that we know are not pseudorefs.\n\nIn the longer term, I'd prefer to see a simpler rule, like \"all-caps\nor underscore string, ending with _HEAD and nothing else are the\npseudorefs but we have these small number of exceptions that are\ngrandfathered\".\n\nThanks.\n"},{"id":"487250","messageId":"ba721840-7b67-4822-8046-c0da4d3b9bde@gmail.com","threadId":"60768","inReplyTo":"xmqqplxtrucm.fsf@gitster.g","subject":"Re: [PATCH 2/5] refs: make `is_pseudoref_syntax()` stricter","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2024-01-23T11:03:28Z","receivedAt":"2024-01-23T11:03:30Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Junio\n\nOn 22/01/2024 20:22, Junio C Hamano wrote:\n> Phillip Wood writes:\n> \n>> I'm concerned that this change is a regression. is_pseudoref_syntax()\n>> is used by is_current_worktree_ref() and so scripts that create\n>> pseudorefs that do not conform to your new rules will break as git\n>> will no-longer consider the pseudorefs they create to be worktree\n>> specific.\n> \n> Ideally, when the exception list in the function becomes more\n> complete, those \"pseudorefs\" created by those scripts shouldn't\n> probably be created either as common or worktree specific thing\n> if they are not \"pseudoref\".\n\nI not sure I quite understand what you mean here. Are you saying that \nscripts should stop using \"git update-ref\" and \"git rev-parse\" for \nanything that does not match the new pseudoref syntax?\n\n>> Another approach would be to\n>> read all the files whose name matches the pseudoref syntax and see if\n>> its contents looks like a valid ref skipping names like COMMIT_EDITMSG\n>> that we know are not pseudorefs.\n> \n> In the longer term, I'd prefer to see a simpler rule, like \"all-caps\n> or underscore string, ending with _HEAD and nothing else are the\n> pseudorefs but we have these small number of exceptions that are\n> grandfathered\".\n\nHopefully such a rule would stop us adding pseudorefs that are really \nprivate state like MERGE_AUTOSTASH. I think that is good in the long \nterm but isn't it is happening now with this patch without any warning \nto users? This patch changes the behavior of parse_worktree_ref() which \nthe files backend uses to figure out the path it should use when reading \nand writing a ref.\n\nBest Wishes\n\nPhillip\n\n"},{"id":"487251","messageId":"Za-gF_Hp_lXViGWw@tanuki","threadId":"60768","inReplyTo":"xmqqplxtrucm.fsf@gitster.g","subject":"Re: [PATCH 2/5] refs: make `is_pseudoref_syntax()` stricter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-23T11:16:39Z","receivedAt":"2024-01-23T11:16:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jan 22, 2024 at 12:22:49PM -0800, Junio C Hamano wrote:\n> Phillip Wood writes:\n> > The list of hard coded exceptions also looks quite short, I\n> > can see MERGE_AUTOSTASH and BISECT_START are missing and there are\n> > probably others I've not thought of.\n> \n> I agree that it is something we need to fix.\n\nI've taken a deeper look at BISECT_START because I previously missed it\nin my conversion to make special refs become normal pseudo refs. But as\nit turns out, BISECT_START is not a ref at all.\n\nDepending on how you execute git-bisect(1), it will either contain the\nobject ID of the detached HEAD or the branch you're starting the bisect\nfrom. This information is used to switch back to that state when you\nabort the bisect. So far this sounds like a proper ref indeed. But in\ncase you're starting from a branch it will not be a symref that points\nto this branch, but it will just contain the branch name. This is not a\nvalid format that could be read as a loose ref, and thus this file is\nnot a proper ref at all (except that sometimes it behaves like one when\nstarting from a detached HEAD).\n\nMy first hunch was to convert it so that it indeed always is a proper\nref. But thinking about it a bit more I'm less convinced that this is\nsensible as it is deeply tied to the behaviour of git-bisect(1) and only\nrepresents its internal state. I thus came to the conclusion that it is\nmore similar to the sequencer state that we have in \".git/rebase-merge\"\nand \".git/rebase-apply\" than anything else.\n\nSo if we wanted to rectify this, I think the most sensible way to\naddress this would be to introduce a new \".git/bisect-state\" directory\nthat contains all of git-bisect(1)'s state:\n\n - BISECT_TERMS -> bisect-state/terms\n - BISECT_LOG -> bisect-state/log\n - BISECT_START -> bisect-state/start\n - BISECT_RUN -> bisect-state/run\n - BISECT_FIRST_PARENT -> bisect-state/first-parent\n - BISECT_ANCESTORS_OK -> bisect-state/ancestors-ok\n\nI think this would make for a much cleaner solution overall as things\nare neatly contained. Cleaning up after a bisect would thus only require\na delete of \".git/bisect-state/\" and we're done.\n\nOf course, this would be a backwards-incompatible change. We could\ntransition to that newer schema by having newer Git versions recognize\nboth ways to store the state, but only ever write the new schema. But\nI'm not sure whether it would ultimately be worth it.\n\nPatrick\n"},{"id":"487252","messageId":"CAOLa=ZQOcqGBJOSehok4BYGUE8RKtnE9eiJYogeT8E6NWZ25xw@mail.gmail.com","threadId":"60768","inReplyTo":"ba721840-7b67-4822-8046-c0da4d3b9bde@gmail.com","subject":"Re: [PATCH 2/5] refs: make `is_pseudoref_syntax()` stricter","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-23T12:49:27Z","receivedAt":"2024-01-23T12:49:29Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello Phillip,\n\nPhillip Wood writes:\n>\n> Hopefully such a rule would stop us adding pseudorefs that are really\n> private state like MERGE_AUTOSTASH. I think that is good in the long\n> term but isn't it is happening now with this patch without any warning\n> to users? This patch changes the behavior of parse_worktree_ref() which\n> the files backend uses to figure out the path it should use when reading\n> and writing a ref.\n>\n\nI do agree with the problem you're outlining here. Changing\n`is_pseudoref_syntax()` does indeed break things since its also used by\n`parse_worktree_ref()`.\n\nI first thought I could get around this by adding the required missing\nrefs, but even that wouldn't work. Because BISECT_START has dual nature,\nit act as a ref and also as file storing a branch name as Patrick\nmentions in detail in his email [1]. Meaning if `is_pseudoref_syntax()`\nidentifies it as a pseudoref, it could be wrong and printing it as such\nmight not work. But we can't not match it because that is the current\nexpectation.\n\nSo there is no way to make `is_pseudoref_syntax()` stricter without\nbreaking backward compatibility. While we do want to reach that goal, we\nhave to go about in the other way around, that i.e.\n1. Fix all pseudorefs to have the '_HEAD' suffix.\n2. Move bisect files to '$GIT_DIR/bisect-state' (see [1] for more\ndetails).\nAfter this, we can safely make `is_pseudoref_syntax()` stricter.\n\nGiven this, I think for the next version, I'll do the following changes:\n1. keep `is_pseudoref_syntax()` as is.\n2. introduce `is_pseudoref()` which calls `is_pseudoref_syntax()` and\nalso checks the content of the file.\n3. replace use of `is_pseudoref_syntax()` with `is_pseudoref()` in this\npatch series.\n\n[1]: https://public-inbox.org/git/20240119142705.139374-1-karthik.188@gmail.com/T/#m6e3790e30613fd68349708faaf5f4d9c704ba677\n"},{"id":"487267","messageId":"9c800cd5-8d20-4df7-8834-f74ab000695e@gmail.com","threadId":"60768","inReplyTo":"Za-gF_Hp_lXViGWw@tanuki","subject":"Re: [PATCH 2/5] refs: make `is_pseudoref_syntax()` stricter","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2024-01-23T16:30:14Z","receivedAt":"2024-01-23T16:30:16Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Patrick\n\nOn 23/01/2024 11:16, Patrick Steinhardt wrote:\n> On Mon, Jan 22, 2024 at 12:22:49PM -0800, Junio C Hamano wrote:\n>> Phillip Wood writes:\n>>> The list of hard coded exceptions also looks quite short, I\n>>> can see MERGE_AUTOSTASH and BISECT_START are missing and there are\n>>> probably others I've not thought of.\n>>\n>> I agree that it is something we need to fix.\n> \n> I've taken a deeper look at BISECT_START because I previously missed it\n> in my conversion to make special refs become normal pseudo refs. But as\n> it turns out, BISECT_START is not a ref at all.\n> > Depending on how you execute git-bisect(1), it will either contain the\n> object ID of the detached HEAD or the branch you're starting the bisect\n> from. This information is used to switch back to that state when you\n> abort the bisect. So far this sounds like a proper ref indeed. But in\n> case you're starting from a branch it will not be a symref that points\n> to this branch, but it will just contain the branch name. This is not a\n> valid format that could be read as a loose ref, and thus this file is\n> not a proper ref at all (except that sometimes it behaves like one when\n> starting from a detached HEAD).\n\nThank you, I'd missed that\n\n> My first hunch was to convert it so that it indeed always is a proper\n> ref. But thinking about it a bit more I'm less convinced that this is\n> sensible as it is deeply tied to the behaviour of git-bisect(1) and only\n> represents its internal state. I thus came to the conclusion that it is\n> more similar to the sequencer state that we have in \".git/rebase-merge\"\n> and \".git/rebase-apply\" than anything else.\n> \n> So if we wanted to rectify this, I think the most sensible way to\n> address this would be to introduce a new \".git/bisect-state\" directory\n> that contains all of git-bisect(1)'s state:\n> \n> - BISECT_TERMS -> bisect-state/terms\n> - BISECT_LOG -> bisect-state/log\n> - BISECT_START -> bisect-state/start\n> - BISECT_RUN -> bisect-state/run\n> - BISECT_FIRST_PARENT -> bisect-state/first-parent\n> - BISECT_ANCESTORS_OK -> bisect-state/ancestors-ok\n> \n> I think this would make for a much cleaner solution overall as things\n> are neatly contained. Cleaning up after a bisect would thus only require\n> a delete of \".git/bisect-state/\" and we're done.\n> \n> Of course, this would be a backwards-incompatible change. We could\n> transition to that newer schema by having newer Git versions recognize\n> both ways to store the state, but only ever write the new schema. But\n> I'm not sure whether it would ultimately be worth it.\n\nI think that is a really good suggestion, it would bring bisect into \nline with am, rebase, cherry-pick etc. which keep their state in a \nsubdirectory rather than cluttering up .git.\n\nBest Wishes\n\nPhillip\n\n> Patrick\n"},{"id":"487268","messageId":"ffb41e0f-f787-439a-9afe-8dcda5da21a3@gmail.com","threadId":"60768","inReplyTo":"CAOLa=ZQOcqGBJOSehok4BYGUE8RKtnE9eiJYogeT8E6NWZ25xw@mail.gmail.com","subject":"Re: [PATCH 2/5] refs: make `is_pseudoref_syntax()` stricter","fromName":"","fromEmail":"phillip.wood123@gmail.com","sentAt":"2024-01-23T16:40:42Z","receivedAt":"2024-01-23T16:40:44Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Karthik\n\nOn 23/01/2024 12:49, Karthik Nayak wrote:\n> Hello Phillip,\n> \n> Phillip Wood writes:\n> [...]\n> So there is no way to make `is_pseudoref_syntax()` stricter without\n> breaking backward compatibility. While we do want to reach that goal, we\n> have to go about in the other way around, that i.e.\n> 1. Fix all pseudorefs to have the '_HEAD' suffix.\n\nI'm wary of renaming user facing pseudorefs like AUTO_MERGE. In the \nfuture we should try and avoid adding any without a \"_HEAD\" suffix\n\n> 2. Move bisect files to '$GIT_DIR/bisect-state' (see [1] for more\n> details).\n> After this, we can safely make `is_pseudoref_syntax()` stricter.\n> \n> Given this, I think for the next version, I'll do the following changes:\n> 1. keep `is_pseudoref_syntax()` as is.\n> 2. introduce `is_pseudoref()` which calls `is_pseudoref_syntax()` and\n> also checks the content of the file.\n\nWe could perhaps make is_pseudoref() stricter from the start as we're \nadding a new function, it could have a list of exceptions to the \"a \npseudoref must end with '_HEAD'\" rule like this patch. Being strict \nabout the pseudorefs we list with \"git for-each-ref\" might encourage \nusers to update any scripts they have that create \"pseudorefs\" that do \nnot match the new rules without us making any changes that actually \nbreak those scripts.\n\n> 3. replace use of `is_pseudoref_syntax()` with `is_pseudoref()` in this\n> patch series.\n\nThat sounds like a good way forward to me, lets see if Junio agrees.\n\nBest Wishes\n\nPhillip\n\n\n> [1]: https://public-inbox.org/git/20240119142705.139374-1-karthik.188@gmail.com/T/#m6e3790e30613fd68349708faaf5f4d9c704ba677\n"},{"id":"487271","messageId":"xmqq5xzkosqc.fsf@gitster.g","threadId":"60768","inReplyTo":"ba721840-7b67-4822-8046-c0da4d3b9bde@gmail.com","subject":"Re: [PATCH 2/5] refs: make `is_pseudoref_syntax()` stricter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-23T17:38:19Z","receivedAt":"2024-01-23T17:38:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n> I not sure I quite understand what you mean here. Are you saying that\n> scripts should stop using \"git update-ref\" and \"git rev-parse\" for\n> anything that does not match the new pseudoref syntax?\n\nYes, I am saying that, and also that we should have been stricter on\nwhat we accept and consider as pseudorefs---not just any file that\nsits directly under $GIT_DIR/., but ideally we should have limited\nto \"^[A-Z]*_HEAD$\" or something. The idea I am floating is to see if\nsuch a tightening can be done now without hearing too many screams.\n"},{"id":"487272","messageId":"xmqqwms0ndvu.fsf@gitster.g","threadId":"60768","inReplyTo":"Za-gF_Hp_lXViGWw@tanuki","subject":"Re: [PATCH 2/5] refs: make `is_pseudoref_syntax()` stricter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-23T17:44:21Z","receivedAt":"2024-01-23T17:44:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> My first hunch was to convert it so that it indeed always is a proper\n> ref. But thinking about it a bit more I'm less convinced that this is\n> sensible as it is deeply tied to the behaviour of git-bisect(1) and only\n> represents its internal state. I thus came to the conclusion that it is\n> more similar to the sequencer state that we have in \".git/rebase-merge\"\n> and \".git/rebase-apply\" than anything else.\n\nFair enough.\n\n> So if we wanted to rectify this, I think the most sensible way to\n> address this would be to introduce a new \".git/bisect-state\" directory\n> that contains all of git-bisect(1)'s state:\n>\n> - BISECT_TERMS -> bisect-state/terms\n> - BISECT_LOG -> bisect-state/log\n> - BISECT_START -> bisect-state/start\n> - BISECT_RUN -> bisect-state/run\n> - BISECT_FIRST_PARENT -> bisect-state/first-parent\n> - BISECT_ANCESTORS_OK -> bisect-state/ancestors-ok\n>\n> I think this would make for a much cleaner solution overall as things\n> are neatly contained. Cleaning up after a bisect would thus only require\n> a delete of \".git/bisect-state/\" and we're done.\n\nAnd bisect-state/ needs to be marked as per-worktree hierarchy, I suppose.\n\n> Of course, this would be a backwards-incompatible change.\n\nAs long as we ignore folks who switches versions of Git in the\nmiddle of their \"git bisect\" session, we should be OK.\n\nIf we really cared the backward compatibility, the new version of\nGit that knows and uses this new layout could notice these old-style\nfilenames and move them over to the new place under new names. From\nthere, everything should work (including things like \"git bisect log\").\n\nThanks.\n\n\n\n"},{"id":"487273","messageId":"xmqqsf2ondrx.fsf@gitster.g","threadId":"60768","inReplyTo":"CAOLa=ZQOcqGBJOSehok4BYGUE8RKtnE9eiJYogeT8E6NWZ25xw@mail.gmail.com","subject":"Re: [PATCH 2/5] refs: make `is_pseudoref_syntax()` stricter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-23T17:46:42Z","receivedAt":"2024-01-23T17:46:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Given this, I think for the next version, I'll do the following changes:\n> 1. keep `is_pseudoref_syntax()` as is.\n> 2. introduce `is_pseudoref()` which calls `is_pseudoref_syntax()` and\n> also checks the content of the file.\n> 3. replace use of `is_pseudoref_syntax()` with `is_pseudoref()` in this\n> patch series.\n\nThe content check in 2. was something that was mentioned in an\nearlier discussion, lack of which I completely missed during the\nreview of this current round. Sounds very good to add that.\n\nThanks.\n\n"},{"id":"487318","messageId":"ZbDPqOnyLw4yhu--@tanuki","threadId":"60768","inReplyTo":"xmqqwms0ndvu.fsf@gitster.g","subject":"Re: [PATCH 2/5] refs: make `is_pseudoref_syntax()` stricter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-24T08:51:52Z","receivedAt":"2024-01-24T08:51:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Jan 23, 2024 at 09:44:21AM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > My first hunch was to convert it so that it indeed always is a proper\n> > ref. But thinking about it a bit more I'm less convinced that this is\n> > sensible as it is deeply tied to the behaviour of git-bisect(1) and only\n> > represents its internal state. I thus came to the conclusion that it is\n> > more similar to the sequencer state that we have in \".git/rebase-merge\"\n> > and \".git/rebase-apply\" than anything else.\n> \n> Fair enough.\n> \n> > So if we wanted to rectify this, I think the most sensible way to\n> > address this would be to introduce a new \".git/bisect-state\" directory\n> > that contains all of git-bisect(1)'s state:\n> >\n> > - BISECT_TERMS -> bisect-state/terms\n> > - BISECT_LOG -> bisect-state/log\n> > - BISECT_START -> bisect-state/start\n> > - BISECT_RUN -> bisect-state/run\n> > - BISECT_FIRST_PARENT -> bisect-state/first-parent\n> > - BISECT_ANCESTORS_OK -> bisect-state/ancestors-ok\n> >\n> > I think this would make for a much cleaner solution overall as things\n> > are neatly contained. Cleaning up after a bisect would thus only require\n> > a delete of \".git/bisect-state/\" and we're done.\n> \n> And bisect-state/ needs to be marked as per-worktree hierarchy, I suppose.\n\nYes, \"bisect-state/\" would need to be stored in GIT_DIR, not COMMON_DIR.\n\n> > Of course, this would be a backwards-incompatible change.\n> \n> As long as we ignore folks who switches versions of Git in the\n> middle of their \"git bisect\" session, we should be OK.\n> \n> If we really cared the backward compatibility, the new version of\n> Git that knows and uses this new layout could notice these old-style\n> filenames and move them over to the new place under new names. From\n> there, everything should work (including things like \"git bisect log\").\n\nWe also have consider that there may be alternate implementations of Git\nthat would only know to handle the old layout. Those tools would be\nbroken in case we did such a migration, but they would be broken anyway\nif the bisect was started via Git and not via the tool.\n\nAnyway, I'll add this to our growing backlog of issues that we might\nwant to investigate once the reftable backend has been upstreamed. Which\nof course shouldn't preclude anybody else from picking up this topic in\ncase they are interested.\n\nPatrick\n"},{"id":"487332","messageId":"20240124152726.124873-1-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240119142705.139374-1-karthik.188@gmail.com","subject":"[PATCH v2 0/4] for-each-ref: print all refs on empty string pattern","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-24T15:27:22Z","receivedAt":"2024-01-24T15:27:34Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"This is the second version of my patch series to print refs\nwhen and empty string pattern is used with git-for-each-ref(1).\n\nWith the upcoming introduction of the reftable backend, it becomes ever\nso important to provide the necessary tooling for printing all refs\nassociated with a repository.\n\nWhile regular refs stored within the \"refs/\" namespace are currently\nsupported by multiple commands like git-for-each-ref(1),\ngit-show-ref(1). Neither support printing all the operational refs\nwithin the repository.\n\nThis is crucial because with the reftable backend, all these refs will\nalso move to reftable. It would be necessary to identify all the refs\nthat are stored within the reftable since there is no easy way to do so\notherwise. This is because the reftable itself is a binary format and\nall access will be via git. Unlike the filesystem backend, which allows\naccess directly via the filesystem.\n\nThis patch series is a follow up to the RFC/discussion we had earlier on\nthe list [1].\n\nThe first 4 commits add the required functionality to ensure we can print\nall refs (regular, pseudo, HEAD). The 5th commit modifies the\ngit-for-each-ref(1) command to print all refs when an empty string pattern\nis used. This is a deviation from the current situation wherein the empty\nstring pattern currently matches and prints no refs.\n\n[1]: https://lore.kernel.org/git/20231221170715.110565-1-karthik.188@gmail.com/#t\n\nChanges since v1:\n\n- Introduce `is_pseudoref()` and `is_headref()` and use them instead of\ndirectly using `is_pseudoref_syntax`.\n- Rename `add_pseudoref_like_entries()` to `add_pseudoref_and_head_entries()`\nsince it also adds the HEAD ref.\n- Also check for the pseudoref's contents to ensure it conforms to the ref\nformat. \n\nKarthik Nayak (4):\n refs: introduce `is_pseudoref()` and `is_headref()`\n refs: extract out `loose_fill_ref_dir_regular_file()`\n refs: introduce `refs_for_each_all_refs()`\n for-each-ref: avoid filtering on empty pattern\n\n Documentation/git-for-each-ref.txt | 3 +-\n builtin/for-each-ref.c | 21 ++++-\n ref-filter.c | 13 ++-\n ref-filter.h | 4 +-\n refs.c | 39 +++++++++\n refs.h | 9 ++\n refs/files-backend.c | 127 +++++++++++++++++++++--------\n refs/refs-internal.h | 7 ++\n t/t6302-for-each-ref-filter.sh | 34 ++++++++\n 9 files changed, 218 insertions(+), 39 deletions(-)\n\n-- \n2.43.GIT\n\n"},{"id":"487333","messageId":"20240124152726.124873-2-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240124152726.124873-1-karthik.188@gmail.com","subject":"[PATCH v2 1/4] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-24T15:27:23Z","receivedAt":"2024-01-24T15:27:34Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Introduce two new functions `is_pseudoref()` and `is_headref()`. This\nprovides the necessary functionality for us to add pseudorefs and HEAD\nto the loose ref cache in the files backend, allowing us to build\ntooling to print these refs.\n\nThe `is_pseudoref()` function internally calls `is_pseudoref_syntax()`\nbut adds onto it by also checking to ensure that the pseudoref either\nends with a \"_HEAD\" suffix or matches a list of exceptions. After which\nwe also parse the contents of the pseudoref to ensure that it conforms\nto the ref format.\n\nWe cannot directly add the new syntax checks to `is_pseudoref_syntax()`\nbecause the function is also used by `is_current_worktree_ref()` and\nmaking it stricter to match only known pseudorefs might have unintended\nconsequences due to files like 'BISECT_START' which isn't a pseudoref\nbut sometimes contains object ID.\n\nKeeping this in mind, we leave `is_pseudoref_syntax()` as is and create\n`is_pseudoref()` which is stricter. Ideally we'd want to move the new\nsyntax checks to `is_pseudoref_syntax()` but a prerequisite for this\nwould be to actually remove the exception list by converting those\npseudorefs to also contain a '_HEAD' suffix and perhaps move bisect\nrelated files like 'BISECT_START' to a new directory similar to the\n'rebase-merge' directory.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 32 ++++++++++++++++++++++++++++++++\n refs.h | 3 +++\n 2 files changed, 35 insertions(+)\n\ndiff --git a/refs.c b/refs.c\nindex 20e8f1ff1f..4b6bfc66fb 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -859,6 +859,38 @@ static int is_pseudoref_syntax(const char *refname)\n \treturn 1;\n }\n \n+int is_pseudoref(struct ref_store *refs, const char *refname)\n+{\n+\tstatic const char *const irregular_pseudorefs[] = {\n+\t\t\"AUTO_MERGE\",\n+\t\t\"BISECT_EXPECTED_REV\",\n+\t\t\"NOTES_MERGE_PARTIAL\",\n+\t\t\"NOTES_MERGE_REF\",\n+\t\t\"MERGE_AUTOSTASH\"\n+\t};\n+\tsize_t i;\n+\n+\tif (!is_pseudoref_syntax(refname))\n+\t\treturn 0;\n+\n+\tif (ends_with(refname, \"_HEAD\"))\n+\t\treturn refs_ref_exists(refs, refname);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n+\t\t if (!strcmp(refname, irregular_pseudorefs[i]))\n+\t\t\t return refs_ref_exists(refs, refname);\n+\n+\treturn 0;\n+}\n+\n+int is_headref(struct ref_store *refs, const char *refname)\n+{\n+\tif (!strcmp(refname, \"HEAD\"))\n+\t\treturn refs_ref_exists(refs, refname);\n+\n+\treturn 0;\n+}\n+\n static int is_current_worktree_ref(const char *ref) {\n \treturn is_pseudoref_syntax(ref) || is_per_worktree_ref(ref);\n }\ndiff --git a/refs.h b/refs.h\nindex 11b3b6ccea..46b8085d63 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1021,4 +1021,7 @@ extern struct ref_namespace_info ref_namespace[NAMESPACE__COUNT];\n */\n void update_ref_namespace(enum ref_namespace namespace, char *ref);\n \n+int is_pseudoref(struct ref_store *refs, const char *refname);\n+int is_headref(struct ref_store *refs, const char *refname);\n+\n #endif /* REFS_H */\n-- \n2.43.GIT\n\n"},{"id":"487334","messageId":"20240124152726.124873-3-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240124152726.124873-1-karthik.188@gmail.com","subject":"[PATCH v2 2/4] refs: extract out `loose_fill_ref_dir_regular_file()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-24T15:27:24Z","receivedAt":"2024-01-24T15:27:35Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Extract out the code for adding a single file to the loose ref dir as\n`loose_fill_ref_dir_regular_file()` from `loose_fill_ref_dir()` in\n`refs/files-backend.c`.\n\nThis allows us to use this function independently in the following\ncommits where we add code to also add pseudorefs to the ref dir.\n\nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 62 +++++++++++++++++++++++---------------------\n 1 file changed, 33 insertions(+), 29 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex b288fc97db..22495a4807 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -229,6 +229,38 @@ static void add_per_worktree_entries_to_dir(struct ref_dir *dir, const char *dir\n \t}\n }\n \n+static void loose_fill_ref_dir_regular_file(struct files_ref_store *refs,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t struct ref_dir *dir)\n+{\n+\tstruct object_id oid;\n+\tint flag;\n+\n+\tif (!refs_resolve_ref_unsafe(&refs->base, refname, RESOLVE_REF_READING,\n+\t\t\t\t &oid, &flag)) {\n+\t\toidclr(&oid);\n+\t\tflag |= REF_ISBROKEN;\n+\t} else if (is_null_oid(&oid)) {\n+\t\t/*\n+\t\t * It is so astronomically unlikely\n+\t\t * that null_oid is the OID of an\n+\t\t * actual object that we consider its\n+\t\t * appearance in a loose reference\n+\t\t * file to be repo corruption\n+\t\t * (probably due to a software bug).\n+\t\t */\n+\t\tflag |= REF_ISBROKEN;\n+\t}\n+\n+\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tif (!refname_is_safe(refname))\n+\t\t\tdie(\"loose refname is dangerous: %s\", refname);\n+\t\toidclr(&oid);\n+\t\tflag |= REF_BAD_NAME | REF_ISBROKEN;\n+\t}\n+\tadd_entry_to_dir(dir, create_ref_entry(refname, &oid, flag));\n+}\n+\n /*\n * Read the loose references from the namespace dirname into dir\n * (without recursing). dirname must end with '/'. dir must be the\n@@ -257,8 +289,6 @@ static void loose_fill_ref_dir(struct ref_store *ref_store,\n \tstrbuf_add(&refname, dirname, dirnamelen);\n \n \twhile ((de = readdir(d)) != NULL) {\n-\t\tstruct object_id oid;\n-\t\tint flag;\n \t\tunsigned char dtype;\n \n \t\tif (de->d_name[0] == '.')\n@@ -274,33 +304,7 @@ static void loose_fill_ref_dir(struct ref_store *ref_store,\n \t\t\t\t\t create_dir_entry(dir->cache, refname.buf,\n \t\t\t\t\t\t\t refname.len));\n \t\t} else if (dtype == DT_REG) {\n-\t\t\tif (!refs_resolve_ref_unsafe(&refs->base,\n-\t\t\t\t\t\t refname.buf,\n-\t\t\t\t\t\t RESOLVE_REF_READING,\n-\t\t\t\t\t\t &oid, &flag)) {\n-\t\t\t\toidclr(&oid);\n-\t\t\t\tflag |= REF_ISBROKEN;\n-\t\t\t} else if (is_null_oid(&oid)) {\n-\t\t\t\t/*\n-\t\t\t\t * It is so astronomically unlikely\n-\t\t\t\t * that null_oid is the OID of an\n-\t\t\t\t * actual object that we consider its\n-\t\t\t\t * appearance in a loose reference\n-\t\t\t\t * file to be repo corruption\n-\t\t\t\t * (probably due to a software bug).\n-\t\t\t\t */\n-\t\t\t\tflag |= REF_ISBROKEN;\n-\t\t\t}\n-\n-\t\t\tif (check_refname_format(refname.buf,\n-\t\t\t\t\t\t REFNAME_ALLOW_ONELEVEL)) {\n-\t\t\t\tif (!refname_is_safe(refname.buf))\n-\t\t\t\t\tdie(\"loose refname is dangerous: %s\", refname.buf);\n-\t\t\t\toidclr(&oid);\n-\t\t\t\tflag |= REF_BAD_NAME | REF_ISBROKEN;\n-\t\t\t}\n-\t\t\tadd_entry_to_dir(dir,\n-\t\t\t\t\t create_ref_entry(refname.buf, &oid, flag));\n+\t\t\tloose_fill_ref_dir_regular_file(refs, refname.buf, dir);\n \t\t}\n \t\tstrbuf_setlen(&refname, dirnamelen);\n \t}\n-- \n2.43.GIT\n\n"},{"id":"487335","messageId":"20240124152726.124873-4-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240124152726.124873-1-karthik.188@gmail.com","subject":"[PATCH v2 3/4] refs: introduce `refs_for_each_all_refs()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-24T15:27:25Z","receivedAt":"2024-01-24T15:27:36Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Introduce a new ref iteration flag `DO_FOR_EACH_INCLUDE_ALL_REFS`, which\nwill be used to iterate over all refs. In the files backend this is\nlimited to regular refs, pseudorefs and HEAD. For other backends like\nthe reftable this is the universal set of all refs stored in the\nbackend.\n\nRefs which fall outside the `refs/` and aren't either pseudorefs or HEAD\nare more of a grey area. This is because we don't block the users from\ncreating such refs but they are not officially supported. In the files\nbackend, we can isolate such files from other files.\n\nIntroduce `refs_for_each_all_refs()` which calls `do_for_each_ref()`\nwith this newly introduced flag.\n\nIn `refs/files-backend.c`, introduce a new function\n`add_pseudoref_and_head_entries()` to add pseudorefs and HEAD to the\n`ref_dir`. We then finally call `add_pseudoref_and_head_entries()`\nwhenever the `DO_FOR_EACH_INCLUDE_ALL_REFS` flag is set. Any new ref\nbackend will also have to implement similar changes on its end.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 7 +++++\n refs.h | 6 ++++\n refs/files-backend.c | 65 ++++++++++++++++++++++++++++++++++++++++----\n refs/refs-internal.h | 7 +++++\n 4 files changed, 80 insertions(+), 5 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 4b6bfc66fb..b5e63f133a 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1755,6 +1755,13 @@ int for_each_rawref(each_ref_fn fn, void *cb_data)\n \treturn refs_for_each_rawref(get_main_ref_store(the_repository), fn, cb_data);\n }\n \n+int refs_for_each_all_refs(struct ref_store *refs, each_ref_fn fn,\n+\t\t\t void *cb_data)\n+{\n+\treturn do_for_each_ref(refs, \"\", NULL, fn, 0,\n+\t\t\t DO_FOR_EACH_INCLUDE_ALL_REFS, cb_data);\n+}\n+\n static int qsort_strcmp(const void *va, const void *vb)\n {\n \tconst char *a = *(const char **)va;\ndiff --git a/refs.h b/refs.h\nindex 46b8085d63..77ecb820f9 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -396,6 +396,12 @@ int for_each_namespaced_ref(const char **exclude_patterns,\n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data);\n int for_each_rawref(each_ref_fn fn, void *cb_data);\n \n+/*\n+ * Iterates over all ref types, regular, pseudorefs and HEAD.\n+ */\n+int refs_for_each_all_refs(struct ref_store *refs, each_ref_fn fn,\n+\t\t\t void *cb_data);\n+\n /*\n * Normalizes partial refs to their fully qualified form.\n * Will prepend to the if it doesn't start with 'refs/'.\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 22495a4807..104f2e1ac7 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -315,9 +315,59 @@ static void loose_fill_ref_dir(struct ref_store *ref_store,\n \tadd_per_worktree_entries_to_dir(dir, dirname);\n }\n \n-static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs)\n+/*\n+ * Add pseudorefs to the ref dir by parsing the directory for any files\n+ * which follow the pseudoref syntax.\n+ */\n+static void add_pseudoref_and_head_entries(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_dir *dir,\n+\t\t\t\t\t const char *dirname)\n+{\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fill_ref_dir\");\n+\tstruct strbuf path = STRBUF_INIT, refname = STRBUF_INIT;\n+\tstruct dirent *de;\n+\tsize_t dirnamelen;\n+\tDIR *d;\n+\n+\tfiles_ref_path(refs, &path, dirname);\n+\n+\td = opendir(path.buf);\n+\tif (!d) {\n+\t\tstrbuf_release(&path);\n+\t\treturn;\n+\t}\n+\n+\tstrbuf_addstr(&refname, dirname);\n+\tdirnamelen = refname.len;\n+\n+\twhile ((de = readdir(d)) != NULL) {\n+\t\tunsigned char dtype;\n+\n+\t\tif (de->d_name[0] == '.')\n+\t\t\tcontinue;\n+\t\tif (ends_with(de->d_name, \".lock\"))\n+\t\t\tcontinue;\n+\t\tstrbuf_addstr(&refname, de->d_name);\n+\n+\t\tdtype = get_dtype(de, &path, 1);\n+\t\tif (dtype == DT_REG && (is_pseudoref(ref_store, de->d_name) ||\n+\t\t\t\t\t\t\t\tis_headref(ref_store, de->d_name)))\n+\t\t\tloose_fill_ref_dir_regular_file(refs, refname.buf, dir);\n+\n+\t\tstrbuf_setlen(&refname, dirnamelen);\n+\t}\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&path);\n+\tclosedir(d);\n+}\n+\n+static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs,\n+\t\t\t\t\t unsigned int flags)\n {\n \tif (!refs->loose) {\n+\t\tstruct ref_dir *dir;\n+\n \t\t/*\n \t\t * Mark the top-level directory complete because we\n \t\t * are about to read the only subdirectory that can\n@@ -328,12 +378,17 @@ static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs)\n \t\t/* We're going to fill the top level ourselves: */\n \t\trefs->loose->root->flag &= ~REF_INCOMPLETE;\n \n+\t\tdir = get_ref_dir(refs->loose->root);\n+\n+\t\tif (flags & DO_FOR_EACH_INCLUDE_ALL_REFS)\n+\t\t\tadd_pseudoref_and_head_entries(dir->cache->ref_store, dir,\n+\t\t\t\t\t\t\t\t\t\t refs->loose->root->name);\n+\n \t\t/*\n \t\t * Add an incomplete entry for \"refs/\" (to be filled\n \t\t * lazily):\n \t\t */\n-\t\tadd_entry_to_dir(get_ref_dir(refs->loose->root),\n-\t\t\t\t create_dir_entry(refs->loose, \"refs/\", 5));\n+\t\tadd_entry_to_dir(dir, create_dir_entry(refs->loose, \"refs/\", 5));\n \t}\n \treturn refs->loose;\n }\n@@ -861,7 +916,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \t * disk, and re-reads it if not.\n \t */\n \n-\tloose_iter = cache_ref_iterator_begin(get_loose_ref_cache(refs),\n+\tloose_iter = cache_ref_iterator_begin(get_loose_ref_cache(refs, flags),\n \t\t\t\t\t prefix, ref_store->repo, 1);\n \n \t/*\n@@ -1222,7 +1277,7 @@ static int files_pack_refs(struct ref_store *ref_store,\n \n \tpacked_refs_lock(refs->packed_ref_store, LOCK_DIE_ON_ERROR, &err);\n \n-\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs), NULL,\n+\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs, 0), NULL,\n \t\t\t\t\tthe_repository, 0);\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t/*\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 8e9f04cc67..1cf7506435 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -260,6 +260,13 @@ enum do_for_each_ref_flags {\n \t * INCLUDE_BROKEN, since they are otherwise not included at all.\n \t */\n \tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n+\n+\t/*\n+\t * Include all refs in the $GIT_DIR in contrast to generally only listing\n+\t * references having the \"refs/\" prefix. In the files-backend this is\n+\t * limited to regular refs, pseudorefs and HEAD.\n+\t */\n+\tDO_FOR_EACH_INCLUDE_ALL_REFS = (1 << 3),\n };\n \n /*\n-- \n2.43.GIT\n\n"},{"id":"487336","messageId":"20240124152726.124873-5-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240124152726.124873-1-karthik.188@gmail.com","subject":"[PATCH v2 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-24T15:27:26Z","receivedAt":"2024-01-24T15:27:37Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"When the user uses an empty string pattern (\"\"), we don't match any refs\nin git-for-each-ref(1). This is because in git-for-each-ref(1), we use\npath based matching and an empty string doesn't match any path.\n\nIn this commit we change this behavior by making empty string pattern\nmatch all references. This is done by introducing a new flag\n`FILTER_REFS_NO_FILTER` in `ref-filter.c`, which uses the newly\nintroduced `refs_for_each_all_refs()` function to iterate over all the\nrefs in the repository.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-for-each-ref.txt | 3 ++-\n builtin/for-each-ref.c | 21 +++++++++++++++++-\n ref-filter.c | 13 ++++++++++--\n ref-filter.h | 4 +++-\n t/t6302-for-each-ref-filter.sh | 34 ++++++++++++++++++++++++++++++\n 5 files changed, 70 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex be9543f684..b1cb482bf5 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -32,7 +32,8 @@ OPTIONS\n \tIf one or more patterns are given, only refs are shown that\n \tmatch against at least one pattern, either using fnmatch(3) or\n \tliterally, in the latter case matching completely or from the\n-\tbeginning up to a slash.\n+\tbeginning up to a slash. If an empty string is provided all refs\n+\tare printed, including HEAD and pseudorefs.\n \n --stdin::\n \tIf `--stdin` is supplied, then the list of patterns is read from\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 3885a9c28e..5aa879e8be 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -25,6 +25,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstruct ref_format format = REF_FORMAT_INIT;\n \tint from_stdin = 0;\n \tstruct strvec vec = STRVEC_INIT;\n+\tunsigned int flags = FILTER_REFS_ALL;\n \n \tstruct option opts[] = {\n \t\tOPT_BIT('s', \"shell\", &format.quote_style,\n@@ -93,11 +94,29 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\t/* vec.v is NULL-terminated, just like 'argv'. */\n \t\tfilter.name_patterns = vec.v;\n \t} else {\n+\t\tsize_t i;\n+\n \t\tfilter.name_patterns = argv;\n+\n+\t\t/*\n+\t\t * Search for any empty string pattern, if it exists then we\n+\t\t * print all refs without any filtering.\n+\t\t */\n+\t\ti = 0;\n+\t\twhile (argv[i]) {\n+\t\t\tif (!argv[i][0]) {\n+\t\t\t\tflags = FILTER_REFS_NO_FILTER;\n+\t\t\t\t/* doing this removes any pattern from being matched */\n+\t\t\t\tfilter.name_patterns[0] = NULL;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\ti++;\n+\t\t}\n \t}\n \n \tfilter.match_as_path = 1;\n-\tfilter_and_format_refs(&filter, FILTER_REFS_ALL, sorting, &format);\n+\tfilter_and_format_refs(&filter, flags, sorting, &format);\n \n \tref_filter_clear(&filter);\n \tref_sorting_release(sorting);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 35b989e1df..6dac133b87 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2622,6 +2622,11 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t\t\t each_ref_fn cb,\n \t\t\t\t void *cb_data)\n {\n+\tif (filter->kind & FILTER_REFS_NO_FILTER) {\n+\t\treturn refs_for_each_all_refs(\n+\t\t\tget_main_ref_store(the_repository), cb, cb_data);\n+\t}\n+\n \tif (!filter->match_as_path) {\n \t\t/*\n \t\t * in this case, the patterns are applied after\n@@ -2775,8 +2780,12 @@ static struct ref_array_item *apply_ref_filter(const char *refname, const struct\n \n \t/* Obtain the current ref kind from filter_ref_kind() and ignore unwanted refs. */\n \tkind = filter_ref_kind(filter, refname);\n-\tif (!(kind & filter->kind))\n+\tif (filter->kind & FILTER_REFS_NO_FILTER) {\n+\t\tif (kind == FILTER_REFS_DETACHED_HEAD)\n+\t\t\tkind = FILTER_REFS_OTHERS;\n+\t} else if (!(kind & filter->kind)) {\n \t\treturn NULL;\n+\t}\n \n \tif (!filter_pattern_match(filter, refname))\n \t\treturn NULL;\n@@ -3041,7 +3050,7 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \t\t\tret = for_each_fullref_in(\"refs/remotes/\", fn, cb_data);\n \t\telse if (filter->kind == FILTER_REFS_TAGS)\n \t\t\tret = for_each_fullref_in(\"refs/tags/\", fn, cb_data);\n-\t\telse if (filter->kind & FILTER_REFS_ALL)\n+\t\telse if (filter->kind & FILTER_REFS_ALL || filter->kind & FILTER_REFS_NO_FILTER)\n \t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n \t\tif (!ret && (filter->kind & FILTER_REFS_DETACHED_HEAD))\n \t\t\thead_ref(fn, cb_data);\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 07cd6f6da3..1eab325ce0 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -22,7 +22,9 @@\n #define FILTER_REFS_ALL (FILTER_REFS_TAGS | FILTER_REFS_BRANCHES | \\\n \t\t\t\t FILTER_REFS_REMOTES | FILTER_REFS_OTHERS)\n #define FILTER_REFS_DETACHED_HEAD 0x0020\n-#define FILTER_REFS_KIND_MASK (FILTER_REFS_ALL | FILTER_REFS_DETACHED_HEAD)\n+#define FILTER_REFS_NO_FILTER 0x0040\n+#define FILTER_REFS_KIND_MASK (FILTER_REFS_ALL | FILTER_REFS_DETACHED_HEAD | \\\n+\t\t\t\t FILTER_REFS_NO_FILTER)\n \n struct atom_value;\n struct ref_sorting;\ndiff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh\nindex 82f3d1ea0f..3922326cab 100755\n--- a/t/t6302-for-each-ref-filter.sh\n+++ b/t/t6302-for-each-ref-filter.sh\n@@ -31,6 +31,40 @@ test_expect_success 'setup some history and refs' '\n \tgit update-ref refs/odd/spot main\n '\n \n+cat >expect <<-\\EOF\n+\tHEAD\n+\tORIG_HEAD\n+\trefs/heads/main\n+\trefs/heads/side\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+EOF\n+\n+test_expect_success 'empty pattern prints pseudorefs' '\n+\tgit update-ref ORIG_HEAD main &&\n+\tgit for-each-ref --format=\"%(refname)\" \"\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'empty pattern with other patterns' '\n+\tgit update-ref ORIG_HEAD main &&\n+\tgit for-each-ref --format=\"%(refname)\" \"\" \"refs/\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'empty pattern towards the end' '\n+\tgit update-ref ORIG_HEAD main &&\n+\tgit for-each-ref --format=\"%(refname)\" \"refs/\" \"\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'filtering with --points-at' '\n \tcat >expect <<-\\EOF &&\n \trefs/heads/main\n-- \n2.43.GIT\n\n"},{"id":"487344","messageId":"xmqqfrymeega.fsf@gitster.g","threadId":"60768","inReplyTo":"20240124152726.124873-2-karthik.188@gmail.com","subject":"Re: [PATCH v2 1/4] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-24T19:09:09Z","receivedAt":"2024-01-24T19:09:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> We cannot directly add the new syntax checks to `is_pseudoref_syntax()`\n> because the function is also used by `is_current_worktree_ref()` and\n> making it stricter to match only known pseudorefs might have unintended\n> consequences due to files like 'BISECT_START' which isn't a pseudoref\n> but sometimes contains object ID.\n\nWell described.\n\n> diff --git a/refs.c b/refs.c\n> index 20e8f1ff1f..4b6bfc66fb 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -859,6 +859,38 @@ static int is_pseudoref_syntax(const char *refname)\n> \treturn 1;\n> }\n> \n> +int is_pseudoref(struct ref_store *refs, const char *refname)\n> +{\n> +\tstatic const char *const irregular_pseudorefs[] = {\n> +\t\t\"AUTO_MERGE\",\n> +\t\t\"BISECT_EXPECTED_REV\",\n> +\t\t\"NOTES_MERGE_PARTIAL\",\n> +\t\t\"NOTES_MERGE_REF\",\n> +\t\t\"MERGE_AUTOSTASH\"\n\nLet's end an array's initializer with a trailing comma, to help\nfuture patches to add entries to this array without unnecessary\npatch noise. \n\n> +\t};\n> +\tsize_t i;\n> +\n> +\tif (!is_pseudoref_syntax(refname))\n> +\t\treturn 0;\n> +\n> +\tif (ends_with(refname, \"_HEAD\"))\n> +\t\treturn refs_ref_exists(refs, refname);\n> +\n> +\tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n> +\t\t if (!strcmp(refname, irregular_pseudorefs[i]))\n> +\t\t\t return refs_ref_exists(refs, refname);\n> +\n> +\treturn 0;\n> +}\n\nThe above uses refs_ref_exists() because we want these to\nsuccessfully resolve for reading.\n\n> +int is_headref(struct ref_store *refs, const char *refname)\n> +{\n> +\tif (!strcmp(refname, \"HEAD\"))\n> +\t\treturn refs_ref_exists(refs, refname);\n\nGiven that \"git for-each-ref refs/remotes\" does not show\n\"refs/remotes/origin/HEAD\" in the output when we do not have the\nremote-tracking branch that symref points at, we probably do want\nto omit \"HEAD\" from the output when the HEAD symref points at an\nunborn branch. If refs_ref_exists() says \"no, it does not exist\"\nin such a case, we are perfectly fine with this code.\n\nWe do not have to worry about the unborn state for pseudorefs\nbecause they would never be symbolic. But that in turn makes me\nsuspect that the check done with refs_ref_exists() in the\nis_pseudoref() helper is a bit too lenient by allowing it to be a\nsymbolic ref. Shouldn't we be using a check based on\nread_ref_full(), like we did in another topic recently [*]?\n\n\n[Reference]\n\n * https://lore.kernel.org/git/xmqqzfxa9usx.fsf@gitster.g/\n"},{"id":"487365","messageId":"CAOLa=ZQxMZo2Y6x6GmVw=df_xS4tkF4D1_tZOeLb7jL5d5bKXA@mail.gmail.com","threadId":"60768","inReplyTo":"xmqqfrymeega.fsf@gitster.g","subject":"Re: [PATCH v2 1/4] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-25T16:20:15Z","receivedAt":"2024-01-25T16:20:17Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello Junio,\n\nJunio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> We cannot directly add the new syntax checks to `is_pseudoref_syntax()`\n>> because the function is also used by `is_current_worktree_ref()` and\n>> making it stricter to match only known pseudorefs might have unintended\n>> consequences due to files like 'BISECT_START' which isn't a pseudoref\n>> but sometimes contains object ID.\n>\n> Well described.\n>\n>> diff --git a/refs.c b/refs.c\n>> index 20e8f1ff1f..4b6bfc66fb 100644\n>> --- a/refs.c\n>> +++ b/refs.c\n>> @@ -859,6 +859,38 @@ static int is_pseudoref_syntax(const char *refname)\n>> \treturn 1;\n>> }\n>>\n>> +int is_pseudoref(struct ref_store *refs, const char *refname)\n>> +{\n>> +\tstatic const char *const irregular_pseudorefs[] = {\n>> +\t\t\"AUTO_MERGE\",\n>> +\t\t\"BISECT_EXPECTED_REV\",\n>> +\t\t\"NOTES_MERGE_PARTIAL\",\n>> +\t\t\"NOTES_MERGE_REF\",\n>> +\t\t\"MERGE_AUTOSTASH\"\n>\n> Let's end an array's initializer with a trailing comma, to help\n> future patches to add entries to this array without unnecessary\n> patch noise.\n\nSure, will add!\n\n>> +\t};\n>> +\tsize_t i;\n>> +\n>> +\tif (!is_pseudoref_syntax(refname))\n>> +\t\treturn 0;\n>> +\n>> +\tif (ends_with(refname, \"_HEAD\"))\n>> +\t\treturn refs_ref_exists(refs, refname);\n>> +\n>> +\tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n>> +\t\t if (!strcmp(refname, irregular_pseudorefs[i]))\n>> +\t\t\t return refs_ref_exists(refs, refname);\n>> +\n>> +\treturn 0;\n>> +}\n>\n> The above uses refs_ref_exists() because we want these to\n> successfully resolve for reading.\n>\n>> +int is_headref(struct ref_store *refs, const char *refname)\n>> +{\n>> +\tif (!strcmp(refname, \"HEAD\"))\n>> +\t\treturn refs_ref_exists(refs, refname);\n>\n> Given that \"git for-each-ref refs/remotes\" does not show\n> \"refs/remotes/origin/HEAD\" in the output when we do not have the\n> remote-tracking branch that symref points at, we probably do want\n> to omit \"HEAD\" from the output when the HEAD symref points at an\n> unborn branch. If refs_ref_exists() says \"no, it does not exist\"\n> in such a case, we are perfectly fine with this code.\n>\n> We do not have to worry about the unborn state for pseudorefs\n> because they would never be symbolic. But that in turn makes me\n> suspect that the check done with refs_ref_exists() in the\n> is_pseudoref() helper is a bit too lenient by allowing it to be a\n> symbolic ref. Shouldn't we be using a check based on\n> read_ref_full(), like we did in another topic recently [*]?\n>\n>\n> [Reference]\n>\n> * https://lore.kernel.org/git/xmqqzfxa9usx.fsf@gitster.g/\n>\n\nThanks, this makes sense and the link is helpful. I'll do something\nsimilar, but since HEAD can be a symref, I'll drop the\n`RESOLVE_REF_NO_RECURSE` flag and only use `RESOLVE_REF_READING`.\n\nI'll wait a day or two, before sending in the new version with the\nfixes. The current diff is\n\ndiff --git a/refs.c b/refs.c\nindex b5e63f133a..4a1fd30ef2 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -866,7 +866,7 @@ int is_pseudoref(struct ref_store *refs, const\nchar *refname)\n \t\t\"BISECT_EXPECTED_REV\",\n \t\t\"NOTES_MERGE_PARTIAL\",\n \t\t\"NOTES_MERGE_REF\",\n-\t\t\"MERGE_AUTOSTASH\"\n+\t\t\"MERGE_AUTOSTASH\",\n \t};\n \tsize_t i;\n\n@@ -885,10 +885,23 @@ int is_pseudoref(struct ref_store *refs, const\nchar *refname)\n\n int is_headref(struct ref_store *refs, const char *refname)\n {\n-\tif (!strcmp(refname, \"HEAD\"))\n-\t\treturn refs_ref_exists(refs, refname);\n+\tstruct object_id oid;\n+\tint flag;\n\n-\treturn 0;\n+\tif (strcmp(refname, \"HEAD\"))\n+\t\treturn 0;\n+\n+\t/*\n+\t * If HEAD doesn't exist, we don't have to die, but rather,\n+\t * we simply return 0.\n+\t */\n+\tif (read_ref_full(\"HEAD\", RESOLVE_REF_READING, &oid, &flag))\n+\t\treturn 0;\n+\n+\tif (is_null_oid(&oid))\n+\t\treturn 0;\n+\n+\treturn 1;\n }\n\n static int is_current_worktree_ref(const char *ref) {\n\n- Karthik\n"},{"id":"487367","messageId":"xmqqa5otxtqy.fsf@gitster.g","threadId":"60768","inReplyTo":"CAOLa=ZQxMZo2Y6x6GmVw=df_xS4tkF4D1_tZOeLb7jL5d5bKXA@mail.gmail.com","subject":"Re: [PATCH v2 1/4] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-25T16:28:21Z","receivedAt":"2024-01-25T16:28:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n>>> +int is_headref(struct ref_store *refs, const char *refname)\n>>> +{\n>>> +\tif (!strcmp(refname, \"HEAD\"))\n>>> +\t\treturn refs_ref_exists(refs, refname);\n>>\n>> Given that \"git for-each-ref refs/remotes\" does not show\n>> \"refs/remotes/origin/HEAD\" in the output when we do not have the\n>> remote-tracking branch that symref points at, we probably do want\n>> to omit \"HEAD\" from the output when the HEAD symref points at an\n>> unborn branch. If refs_ref_exists() says \"no, it does not exist\"\n>> in such a case, we are perfectly fine with this code.\n>>\n>> We do not have to worry about the unborn state for pseudorefs\n>> because they would never be symbolic. But that in turn makes me\n>> suspect that the check done with refs_ref_exists() in the\n>> is_pseudoref() helper is a bit too lenient by allowing it to be a\n>> symbolic ref. Shouldn't we be using a check based on\n>> read_ref_full(), like we did in another topic recently [*]?\n>>\n>>\n>> [Reference]\n>>\n>> * https://lore.kernel.org/git/xmqqzfxa9usx.fsf@gitster.g/\n>>\n>\n> Thanks, this makes sense and the link is helpful. I'll do something\n> similar, but since HEAD can be a symref, I'll drop the\n> `RESOLVE_REF_NO_RECURSE` flag and only use `RESOLVE_REF_READING`.\n\nJust to make sure there is no misunderstanding, I think how\nis_headref() does what it does in the patch is perfectly fine,\nincluding its use of refs_ref_exists(). The side I was referring to\nwith \"in turn makes me suspect\" is the other helper function that\nwill never have to deal with a symref. Use of refs_ref_exists() in\nthat function is too loose.\n\n"},{"id":"487387","messageId":"CAOLa=ZSTY+YQ1a5U6Q=RQNg5VMjP0wOLS4t5obua8_==BfbdhA@mail.gmail.com","threadId":"60768","inReplyTo":"xmqqa5otxtqy.fsf@gitster.g","subject":"Re: [PATCH v2 1/4] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-25T21:48:27Z","receivedAt":"2024-01-25T21:48:54Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"On Thu, Jan 25, 2024 at 5:28 PM Junio C Hamano wrote:\n>\n> Karthik Nayak writes:\n>\n> >>> +int is_headref(struct ref_store *refs, const char *refname)\n> >>> +{\n> >>> + if (!strcmp(refname, \"HEAD\"))\n> >>> + return refs_ref_exists(refs, refname);\n> >>\n> >> Given that \"git for-each-ref refs/remotes\" does not show\n> >> \"refs/remotes/origin/HEAD\" in the output when we do not have the\n> >> remote-tracking branch that symref points at, we probably do want\n> >> to omit \"HEAD\" from the output when the HEAD symref points at an\n> >> unborn branch. If refs_ref_exists() says \"no, it does not exist\"\n> >> in such a case, we are perfectly fine with this code.\n> >>\n> >> We do not have to worry about the unborn state for pseudorefs\n> >> because they would never be symbolic. But that in turn makes me\n> >> suspect that the check done with refs_ref_exists() in the\n> >> is_pseudoref() helper is a bit too lenient by allowing it to be a\n> >> symbolic ref. Shouldn't we be using a check based on\n> >> read_ref_full(), like we did in another topic recently [*]?\n> >>\n> >>\n> >> [Reference]\n> >>\n> >> * https://lore.kernel.org/git/xmqqzfxa9usx.fsf@gitster.g/\n> >>\n> >\n> > Thanks, this makes sense and the link is helpful. I'll do something\n> > similar, but since HEAD can be a symref, I'll drop the\n> > `RESOLVE_REF_NO_RECURSE` flag and only use `RESOLVE_REF_READING`.\n>\n> Just to make sure there is no misunderstanding, I think how\n> is_headref() does what it does in the patch is perfectly fine,\n> including its use of refs_ref_exists(). The side I was referring to\n> with \"in turn makes me suspect\" is the other helper function that\n> will never have to deal with a symref. Use of refs_ref_exists() in\n> that function is too loose.\n>\n\nAH! Totally misunderstood, thanks for reiterating.\n"},{"id":"487509","messageId":"20240129113527.607022-1-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240119142705.139374-1-karthik.188@gmail.com","subject":"[PATCH v3 0/4] for-each-ref: print all refs on empty string pattern","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-29T11:35:23Z","receivedAt":"2024-01-29T11:35:33Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"This is the second version of my patch series to print refs\nwhen and empty string pattern is used with git-for-each-ref(1).\n\nWith the upcoming introduction of the reftable backend, it becomes ever\nso important to provide the necessary tooling for printing all refs\nassociated with a repository.\n\nWhile regular refs stored within the \"refs/\" namespace are currently\nsupported by multiple commands like git-for-each-ref(1),\ngit-show-ref(1). Neither support printing all the operational refs\nwithin the repository.\n\nThis is crucial because with the reftable backend, all these refs will\nalso move to reftable. It would be necessary to identify all the refs\nthat are stored within the reftable since there is no easy way to do so\notherwise. This is because the reftable itself is a binary format and\nall access will be via git. Unlike the filesystem backend, which allows\naccess directly via the filesystem.\n\nThis patch series is a follow up to the RFC/discussion we had earlier on\nthe list [1].\n\nThe first 4 commits add the required functionality to ensure we can print\nall refs (regular, pseudo, HEAD). The 5th commit modifies the\ngit-for-each-ref(1) command to print all refs when an empty string pattern\nis used. This is a deviation from the current situation wherein the empty\nstring pattern currently matches and prints no refs.\n\n[1]: https://lore.kernel.org/git/20231221170715.110565-1-karthik.188@gmail.com/#t\n\nChanges since v1:\n- Added missing comma to the end of the `irregular_pseudorefs` array.\n- Modified `is_pseudoref` to only work with non symrefs. \n\nRange-diff against v2:\n\n1: 116d4c0e6d ! 1: 2141a2a62b refs: introduce `is_pseudoref()` and `is_headref()`\n @@ refs.c: static int is_pseudoref_syntax(const char *refname)\n +\t\t\"BISECT_EXPECTED_REV\",\n +\t\t\"NOTES_MERGE_PARTIAL\",\n +\t\t\"NOTES_MERGE_REF\",\n -+\t\t\"MERGE_AUTOSTASH\"\n ++\t\t\"MERGE_AUTOSTASH\",\n +\t};\n ++\tstruct object_id oid;\n +\tsize_t i;\n +\n +\tif (!is_pseudoref_syntax(refname))\n +\t\treturn 0;\n +\n -+\tif (ends_with(refname, \"_HEAD\"))\n -+\t\treturn refs_ref_exists(refs, refname);\n ++\tif (ends_with(refname, \"_HEAD\")) {\n ++\t\t read_ref_full(refname, RESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n ++\t\t &oid, NULL);\n ++\t\t return !is_null_oid(&oid);\n ++\t}\n +\n +\tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n -+\t\t if (!strcmp(refname, irregular_pseudorefs[i]))\n -+\t\t\t return refs_ref_exists(refs, refname);\n ++\t\tif (!strcmp(refname, irregular_pseudorefs[i])) {\n ++\t\t\tread_ref_full(refname, RESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n ++\t\t\t\t\t\t &oid, NULL);\n ++\t\t\treturn !is_null_oid(&oid);\n ++\t\t}\n +\n +\treturn 0;\n +}\n2: 4d4ca1cb26 = 2: c96f0a9c83 refs: extract out `loose_fill_ref_dir_regular_file()`\n3: a5c0c4bf31 = 3: d165358b83 refs: introduce `refs_for_each_all_refs()`\n4: a1c6537815 = 4: a17983d0ba for-each-ref: avoid filtering on empty pattern\n\n\nKarthik Nayak (4):\n refs: introduce `is_pseudoref()` and `is_headref()`\n refs: extract out `loose_fill_ref_dir_regular_file()`\n refs: introduce `refs_for_each_all_refs()`\n for-each-ref: avoid filtering on empty pattern\n\n Documentation/git-for-each-ref.txt | 3 +-\n builtin/for-each-ref.c | 21 ++++-\n ref-filter.c | 13 ++-\n ref-filter.h | 4 +-\n refs.c | 46 +++++++++++\n refs.h | 9 ++\n refs/files-backend.c | 127 +++++++++++++++++++++--------\n refs/refs-internal.h | 7 ++\n t/t6302-for-each-ref-filter.sh | 34 ++++++++\n 9 files changed, 225 insertions(+), 39 deletions(-)\n\n-- \n2.43.GIT\n\n"},{"id":"487510","messageId":"20240129113527.607022-2-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240129113527.607022-1-karthik.188@gmail.com","subject":"[PATCH v3 1/4] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-29T11:35:24Z","receivedAt":"2024-01-29T11:35:34Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Introduce two new functions `is_pseudoref()` and `is_headref()`. This\nprovides the necessary functionality for us to add pseudorefs and HEAD\nto the loose ref cache in the files backend, allowing us to build\ntooling to print these refs.\n\nThe `is_pseudoref()` function internally calls `is_pseudoref_syntax()`\nbut adds onto it by also checking to ensure that the pseudoref either\nends with a \"_HEAD\" suffix or matches a list of exceptions. After which\nwe also parse the contents of the pseudoref to ensure that it conforms\nto the ref format.\n\nWe cannot directly add the new syntax checks to `is_pseudoref_syntax()`\nbecause the function is also used by `is_current_worktree_ref()` and\nmaking it stricter to match only known pseudorefs might have unintended\nconsequences due to files like 'BISECT_START' which isn't a pseudoref\nbut sometimes contains object ID.\n\nKeeping this in mind, we leave `is_pseudoref_syntax()` as is and create\n`is_pseudoref()` which is stricter. Ideally we'd want to move the new\nsyntax checks to `is_pseudoref_syntax()` but a prerequisite for this\nwould be to actually remove the exception list by converting those\npseudorefs to also contain a '_HEAD' suffix and perhaps move bisect\nrelated files like 'BISECT_START' to a new directory similar to the\n'rebase-merge' directory.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 39 +++++++++++++++++++++++++++++++++++++++\n refs.h | 3 +++\n 2 files changed, 42 insertions(+)\n\ndiff --git a/refs.c b/refs.c\nindex 20e8f1ff1f..559f5aeea8 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -859,6 +859,45 @@ static int is_pseudoref_syntax(const char *refname)\n \treturn 1;\n }\n \n+int is_pseudoref(struct ref_store *refs, const char *refname)\n+{\n+\tstatic const char *const irregular_pseudorefs[] = {\n+\t\t\"AUTO_MERGE\",\n+\t\t\"BISECT_EXPECTED_REV\",\n+\t\t\"NOTES_MERGE_PARTIAL\",\n+\t\t\"NOTES_MERGE_REF\",\n+\t\t\"MERGE_AUTOSTASH\",\n+\t};\n+\tstruct object_id oid;\n+\tsize_t i;\n+\n+\tif (!is_pseudoref_syntax(refname))\n+\t\treturn 0;\n+\n+\tif (ends_with(refname, \"_HEAD\")) {\n+\t\t read_ref_full(refname, RESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n+\t\t &oid, NULL);\n+\t\t return !is_null_oid(&oid);\n+\t}\n+\n+\tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n+\t\tif (!strcmp(refname, irregular_pseudorefs[i])) {\n+\t\t\tread_ref_full(refname, RESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n+\t\t\t\t\t\t &oid, NULL);\n+\t\t\treturn !is_null_oid(&oid);\n+\t\t}\n+\n+\treturn 0;\n+}\n+\n+int is_headref(struct ref_store *refs, const char *refname)\n+{\n+\tif (!strcmp(refname, \"HEAD\"))\n+\t\treturn refs_ref_exists(refs, refname);\n+\n+\treturn 0;\n+}\n+\n static int is_current_worktree_ref(const char *ref) {\n \treturn is_pseudoref_syntax(ref) || is_per_worktree_ref(ref);\n }\ndiff --git a/refs.h b/refs.h\nindex 11b3b6ccea..46b8085d63 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1021,4 +1021,7 @@ extern struct ref_namespace_info ref_namespace[NAMESPACE__COUNT];\n */\n void update_ref_namespace(enum ref_namespace namespace, char *ref);\n \n+int is_pseudoref(struct ref_store *refs, const char *refname);\n+int is_headref(struct ref_store *refs, const char *refname);\n+\n #endif /* REFS_H */\n-- \n2.43.GIT\n\n"},{"id":"487511","messageId":"20240129113527.607022-3-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240129113527.607022-1-karthik.188@gmail.com","subject":"[PATCH v3 2/4] refs: extract out `loose_fill_ref_dir_regular_file()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-29T11:35:25Z","receivedAt":"2024-01-29T11:35:35Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Extract out the code for adding a single file to the loose ref dir as\n`loose_fill_ref_dir_regular_file()` from `loose_fill_ref_dir()` in\n`refs/files-backend.c`.\n\nThis allows us to use this function independently in the following\ncommits where we add code to also add pseudorefs to the ref dir.\n\nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 62 +++++++++++++++++++++++---------------------\n 1 file changed, 33 insertions(+), 29 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex b288fc97db..22495a4807 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -229,6 +229,38 @@ static void add_per_worktree_entries_to_dir(struct ref_dir *dir, const char *dir\n \t}\n }\n \n+static void loose_fill_ref_dir_regular_file(struct files_ref_store *refs,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t struct ref_dir *dir)\n+{\n+\tstruct object_id oid;\n+\tint flag;\n+\n+\tif (!refs_resolve_ref_unsafe(&refs->base, refname, RESOLVE_REF_READING,\n+\t\t\t\t &oid, &flag)) {\n+\t\toidclr(&oid);\n+\t\tflag |= REF_ISBROKEN;\n+\t} else if (is_null_oid(&oid)) {\n+\t\t/*\n+\t\t * It is so astronomically unlikely\n+\t\t * that null_oid is the OID of an\n+\t\t * actual object that we consider its\n+\t\t * appearance in a loose reference\n+\t\t * file to be repo corruption\n+\t\t * (probably due to a software bug).\n+\t\t */\n+\t\tflag |= REF_ISBROKEN;\n+\t}\n+\n+\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tif (!refname_is_safe(refname))\n+\t\t\tdie(\"loose refname is dangerous: %s\", refname);\n+\t\toidclr(&oid);\n+\t\tflag |= REF_BAD_NAME | REF_ISBROKEN;\n+\t}\n+\tadd_entry_to_dir(dir, create_ref_entry(refname, &oid, flag));\n+}\n+\n /*\n * Read the loose references from the namespace dirname into dir\n * (without recursing). dirname must end with '/'. dir must be the\n@@ -257,8 +289,6 @@ static void loose_fill_ref_dir(struct ref_store *ref_store,\n \tstrbuf_add(&refname, dirname, dirnamelen);\n \n \twhile ((de = readdir(d)) != NULL) {\n-\t\tstruct object_id oid;\n-\t\tint flag;\n \t\tunsigned char dtype;\n \n \t\tif (de->d_name[0] == '.')\n@@ -274,33 +304,7 @@ static void loose_fill_ref_dir(struct ref_store *ref_store,\n \t\t\t\t\t create_dir_entry(dir->cache, refname.buf,\n \t\t\t\t\t\t\t refname.len));\n \t\t} else if (dtype == DT_REG) {\n-\t\t\tif (!refs_resolve_ref_unsafe(&refs->base,\n-\t\t\t\t\t\t refname.buf,\n-\t\t\t\t\t\t RESOLVE_REF_READING,\n-\t\t\t\t\t\t &oid, &flag)) {\n-\t\t\t\toidclr(&oid);\n-\t\t\t\tflag |= REF_ISBROKEN;\n-\t\t\t} else if (is_null_oid(&oid)) {\n-\t\t\t\t/*\n-\t\t\t\t * It is so astronomically unlikely\n-\t\t\t\t * that null_oid is the OID of an\n-\t\t\t\t * actual object that we consider its\n-\t\t\t\t * appearance in a loose reference\n-\t\t\t\t * file to be repo corruption\n-\t\t\t\t * (probably due to a software bug).\n-\t\t\t\t */\n-\t\t\t\tflag |= REF_ISBROKEN;\n-\t\t\t}\n-\n-\t\t\tif (check_refname_format(refname.buf,\n-\t\t\t\t\t\t REFNAME_ALLOW_ONELEVEL)) {\n-\t\t\t\tif (!refname_is_safe(refname.buf))\n-\t\t\t\t\tdie(\"loose refname is dangerous: %s\", refname.buf);\n-\t\t\t\toidclr(&oid);\n-\t\t\t\tflag |= REF_BAD_NAME | REF_ISBROKEN;\n-\t\t\t}\n-\t\t\tadd_entry_to_dir(dir,\n-\t\t\t\t\t create_ref_entry(refname.buf, &oid, flag));\n+\t\t\tloose_fill_ref_dir_regular_file(refs, refname.buf, dir);\n \t\t}\n \t\tstrbuf_setlen(&refname, dirnamelen);\n \t}\n-- \n2.43.GIT\n\n"},{"id":"487512","messageId":"20240129113527.607022-4-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240129113527.607022-1-karthik.188@gmail.com","subject":"[PATCH v3 3/4] refs: introduce `refs_for_each_all_refs()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-29T11:35:26Z","receivedAt":"2024-01-29T11:35:36Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Introduce a new ref iteration flag `DO_FOR_EACH_INCLUDE_ALL_REFS`, which\nwill be used to iterate over all refs. In the files backend this is\nlimited to regular refs, pseudorefs and HEAD. For other backends like\nthe reftable this is the universal set of all refs stored in the\nbackend.\n\nRefs which fall outside the `refs/` and aren't either pseudorefs or HEAD\nare more of a grey area. This is because we don't block the users from\ncreating such refs but they are not officially supported. In the files\nbackend, we can isolate such files from other files.\n\nIntroduce `refs_for_each_all_refs()` which calls `do_for_each_ref()`\nwith this newly introduced flag.\n\nIn `refs/files-backend.c`, introduce a new function\n`add_pseudoref_and_head_entries()` to add pseudorefs and HEAD to the\n`ref_dir`. We then finally call `add_pseudoref_and_head_entries()`\nwhenever the `DO_FOR_EACH_INCLUDE_ALL_REFS` flag is set. Any new ref\nbackend will also have to implement similar changes on its end.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 7 +++++\n refs.h | 6 ++++\n refs/files-backend.c | 65 ++++++++++++++++++++++++++++++++++++++++----\n refs/refs-internal.h | 7 +++++\n 4 files changed, 80 insertions(+), 5 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 559f5aeea8..89b925719f 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1762,6 +1762,13 @@ int for_each_rawref(each_ref_fn fn, void *cb_data)\n \treturn refs_for_each_rawref(get_main_ref_store(the_repository), fn, cb_data);\n }\n \n+int refs_for_each_all_refs(struct ref_store *refs, each_ref_fn fn,\n+\t\t\t void *cb_data)\n+{\n+\treturn do_for_each_ref(refs, \"\", NULL, fn, 0,\n+\t\t\t DO_FOR_EACH_INCLUDE_ALL_REFS, cb_data);\n+}\n+\n static int qsort_strcmp(const void *va, const void *vb)\n {\n \tconst char *a = *(const char **)va;\ndiff --git a/refs.h b/refs.h\nindex 46b8085d63..77ecb820f9 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -396,6 +396,12 @@ int for_each_namespaced_ref(const char **exclude_patterns,\n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data);\n int for_each_rawref(each_ref_fn fn, void *cb_data);\n \n+/*\n+ * Iterates over all ref types, regular, pseudorefs and HEAD.\n+ */\n+int refs_for_each_all_refs(struct ref_store *refs, each_ref_fn fn,\n+\t\t\t void *cb_data);\n+\n /*\n * Normalizes partial refs to their fully qualified form.\n * Will prepend to the if it doesn't start with 'refs/'.\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 22495a4807..104f2e1ac7 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -315,9 +315,59 @@ static void loose_fill_ref_dir(struct ref_store *ref_store,\n \tadd_per_worktree_entries_to_dir(dir, dirname);\n }\n \n-static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs)\n+/*\n+ * Add pseudorefs to the ref dir by parsing the directory for any files\n+ * which follow the pseudoref syntax.\n+ */\n+static void add_pseudoref_and_head_entries(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_dir *dir,\n+\t\t\t\t\t const char *dirname)\n+{\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fill_ref_dir\");\n+\tstruct strbuf path = STRBUF_INIT, refname = STRBUF_INIT;\n+\tstruct dirent *de;\n+\tsize_t dirnamelen;\n+\tDIR *d;\n+\n+\tfiles_ref_path(refs, &path, dirname);\n+\n+\td = opendir(path.buf);\n+\tif (!d) {\n+\t\tstrbuf_release(&path);\n+\t\treturn;\n+\t}\n+\n+\tstrbuf_addstr(&refname, dirname);\n+\tdirnamelen = refname.len;\n+\n+\twhile ((de = readdir(d)) != NULL) {\n+\t\tunsigned char dtype;\n+\n+\t\tif (de->d_name[0] == '.')\n+\t\t\tcontinue;\n+\t\tif (ends_with(de->d_name, \".lock\"))\n+\t\t\tcontinue;\n+\t\tstrbuf_addstr(&refname, de->d_name);\n+\n+\t\tdtype = get_dtype(de, &path, 1);\n+\t\tif (dtype == DT_REG && (is_pseudoref(ref_store, de->d_name) ||\n+\t\t\t\t\t\t\t\tis_headref(ref_store, de->d_name)))\n+\t\t\tloose_fill_ref_dir_regular_file(refs, refname.buf, dir);\n+\n+\t\tstrbuf_setlen(&refname, dirnamelen);\n+\t}\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&path);\n+\tclosedir(d);\n+}\n+\n+static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs,\n+\t\t\t\t\t unsigned int flags)\n {\n \tif (!refs->loose) {\n+\t\tstruct ref_dir *dir;\n+\n \t\t/*\n \t\t * Mark the top-level directory complete because we\n \t\t * are about to read the only subdirectory that can\n@@ -328,12 +378,17 @@ static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs)\n \t\t/* We're going to fill the top level ourselves: */\n \t\trefs->loose->root->flag &= ~REF_INCOMPLETE;\n \n+\t\tdir = get_ref_dir(refs->loose->root);\n+\n+\t\tif (flags & DO_FOR_EACH_INCLUDE_ALL_REFS)\n+\t\t\tadd_pseudoref_and_head_entries(dir->cache->ref_store, dir,\n+\t\t\t\t\t\t\t\t\t\t refs->loose->root->name);\n+\n \t\t/*\n \t\t * Add an incomplete entry for \"refs/\" (to be filled\n \t\t * lazily):\n \t\t */\n-\t\tadd_entry_to_dir(get_ref_dir(refs->loose->root),\n-\t\t\t\t create_dir_entry(refs->loose, \"refs/\", 5));\n+\t\tadd_entry_to_dir(dir, create_dir_entry(refs->loose, \"refs/\", 5));\n \t}\n \treturn refs->loose;\n }\n@@ -861,7 +916,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \t * disk, and re-reads it if not.\n \t */\n \n-\tloose_iter = cache_ref_iterator_begin(get_loose_ref_cache(refs),\n+\tloose_iter = cache_ref_iterator_begin(get_loose_ref_cache(refs, flags),\n \t\t\t\t\t prefix, ref_store->repo, 1);\n \n \t/*\n@@ -1222,7 +1277,7 @@ static int files_pack_refs(struct ref_store *ref_store,\n \n \tpacked_refs_lock(refs->packed_ref_store, LOCK_DIE_ON_ERROR, &err);\n \n-\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs), NULL,\n+\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs, 0), NULL,\n \t\t\t\t\tthe_repository, 0);\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t/*\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 8e9f04cc67..1cf7506435 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -260,6 +260,13 @@ enum do_for_each_ref_flags {\n \t * INCLUDE_BROKEN, since they are otherwise not included at all.\n \t */\n \tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n+\n+\t/*\n+\t * Include all refs in the $GIT_DIR in contrast to generally only listing\n+\t * references having the \"refs/\" prefix. In the files-backend this is\n+\t * limited to regular refs, pseudorefs and HEAD.\n+\t */\n+\tDO_FOR_EACH_INCLUDE_ALL_REFS = (1 << 3),\n };\n \n /*\n-- \n2.43.GIT\n\n"},{"id":"487513","messageId":"20240129113527.607022-5-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240129113527.607022-1-karthik.188@gmail.com","subject":"[PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-29T11:35:27Z","receivedAt":"2024-01-29T11:35:37Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"When the user uses an empty string pattern (\"\"), we don't match any refs\nin git-for-each-ref(1). This is because in git-for-each-ref(1), we use\npath based matching and an empty string doesn't match any path.\n\nIn this commit we change this behavior by making empty string pattern\nmatch all references. This is done by introducing a new flag\n`FILTER_REFS_NO_FILTER` in `ref-filter.c`, which uses the newly\nintroduced `refs_for_each_all_refs()` function to iterate over all the\nrefs in the repository.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-for-each-ref.txt | 3 ++-\n builtin/for-each-ref.c | 21 +++++++++++++++++-\n ref-filter.c | 13 ++++++++++--\n ref-filter.h | 4 +++-\n t/t6302-for-each-ref-filter.sh | 34 ++++++++++++++++++++++++++++++\n 5 files changed, 70 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex be9543f684..b1cb482bf5 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -32,7 +32,8 @@ OPTIONS\n \tIf one or more patterns are given, only refs are shown that\n \tmatch against at least one pattern, either using fnmatch(3) or\n \tliterally, in the latter case matching completely or from the\n-\tbeginning up to a slash.\n+\tbeginning up to a slash. If an empty string is provided all refs\n+\tare printed, including HEAD and pseudorefs.\n \n --stdin::\n \tIf `--stdin` is supplied, then the list of patterns is read from\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 3885a9c28e..5aa879e8be 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -25,6 +25,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstruct ref_format format = REF_FORMAT_INIT;\n \tint from_stdin = 0;\n \tstruct strvec vec = STRVEC_INIT;\n+\tunsigned int flags = FILTER_REFS_ALL;\n \n \tstruct option opts[] = {\n \t\tOPT_BIT('s', \"shell\", &format.quote_style,\n@@ -93,11 +94,29 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\t/* vec.v is NULL-terminated, just like 'argv'. */\n \t\tfilter.name_patterns = vec.v;\n \t} else {\n+\t\tsize_t i;\n+\n \t\tfilter.name_patterns = argv;\n+\n+\t\t/*\n+\t\t * Search for any empty string pattern, if it exists then we\n+\t\t * print all refs without any filtering.\n+\t\t */\n+\t\ti = 0;\n+\t\twhile (argv[i]) {\n+\t\t\tif (!argv[i][0]) {\n+\t\t\t\tflags = FILTER_REFS_NO_FILTER;\n+\t\t\t\t/* doing this removes any pattern from being matched */\n+\t\t\t\tfilter.name_patterns[0] = NULL;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\ti++;\n+\t\t}\n \t}\n \n \tfilter.match_as_path = 1;\n-\tfilter_and_format_refs(&filter, FILTER_REFS_ALL, sorting, &format);\n+\tfilter_and_format_refs(&filter, flags, sorting, &format);\n \n \tref_filter_clear(&filter);\n \tref_sorting_release(sorting);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 35b989e1df..6dac133b87 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2622,6 +2622,11 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t\t\t each_ref_fn cb,\n \t\t\t\t void *cb_data)\n {\n+\tif (filter->kind & FILTER_REFS_NO_FILTER) {\n+\t\treturn refs_for_each_all_refs(\n+\t\t\tget_main_ref_store(the_repository), cb, cb_data);\n+\t}\n+\n \tif (!filter->match_as_path) {\n \t\t/*\n \t\t * in this case, the patterns are applied after\n@@ -2775,8 +2780,12 @@ static struct ref_array_item *apply_ref_filter(const char *refname, const struct\n \n \t/* Obtain the current ref kind from filter_ref_kind() and ignore unwanted refs. */\n \tkind = filter_ref_kind(filter, refname);\n-\tif (!(kind & filter->kind))\n+\tif (filter->kind & FILTER_REFS_NO_FILTER) {\n+\t\tif (kind == FILTER_REFS_DETACHED_HEAD)\n+\t\t\tkind = FILTER_REFS_OTHERS;\n+\t} else if (!(kind & filter->kind)) {\n \t\treturn NULL;\n+\t}\n \n \tif (!filter_pattern_match(filter, refname))\n \t\treturn NULL;\n@@ -3041,7 +3050,7 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \t\t\tret = for_each_fullref_in(\"refs/remotes/\", fn, cb_data);\n \t\telse if (filter->kind == FILTER_REFS_TAGS)\n \t\t\tret = for_each_fullref_in(\"refs/tags/\", fn, cb_data);\n-\t\telse if (filter->kind & FILTER_REFS_ALL)\n+\t\telse if (filter->kind & FILTER_REFS_ALL || filter->kind & FILTER_REFS_NO_FILTER)\n \t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n \t\tif (!ret && (filter->kind & FILTER_REFS_DETACHED_HEAD))\n \t\t\thead_ref(fn, cb_data);\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 07cd6f6da3..1eab325ce0 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -22,7 +22,9 @@\n #define FILTER_REFS_ALL (FILTER_REFS_TAGS | FILTER_REFS_BRANCHES | \\\n \t\t\t\t FILTER_REFS_REMOTES | FILTER_REFS_OTHERS)\n #define FILTER_REFS_DETACHED_HEAD 0x0020\n-#define FILTER_REFS_KIND_MASK (FILTER_REFS_ALL | FILTER_REFS_DETACHED_HEAD)\n+#define FILTER_REFS_NO_FILTER 0x0040\n+#define FILTER_REFS_KIND_MASK (FILTER_REFS_ALL | FILTER_REFS_DETACHED_HEAD | \\\n+\t\t\t\t FILTER_REFS_NO_FILTER)\n \n struct atom_value;\n struct ref_sorting;\ndiff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh\nindex 82f3d1ea0f..3922326cab 100755\n--- a/t/t6302-for-each-ref-filter.sh\n+++ b/t/t6302-for-each-ref-filter.sh\n@@ -31,6 +31,40 @@ test_expect_success 'setup some history and refs' '\n \tgit update-ref refs/odd/spot main\n '\n \n+cat >expect <<-\\EOF\n+\tHEAD\n+\tORIG_HEAD\n+\trefs/heads/main\n+\trefs/heads/side\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+EOF\n+\n+test_expect_success 'empty pattern prints pseudorefs' '\n+\tgit update-ref ORIG_HEAD main &&\n+\tgit for-each-ref --format=\"%(refname)\" \"\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'empty pattern with other patterns' '\n+\tgit update-ref ORIG_HEAD main &&\n+\tgit for-each-ref --format=\"%(refname)\" \"\" \"refs/\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'empty pattern towards the end' '\n+\tgit update-ref ORIG_HEAD main &&\n+\tgit for-each-ref --format=\"%(refname)\" \"refs/\" \"\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'filtering with --points-at' '\n \tcat >expect <<-\\EOF &&\n \trefs/heads/main\n-- \n2.43.GIT\n\n"},{"id":"487548","messageId":"xmqqy1c7ho4u.fsf@gitster.g","threadId":"60768","inReplyTo":"20240129113527.607022-1-karthik.188@gmail.com","subject":"Re: [PATCH v3 0/4] for-each-ref: print all refs on empty string pattern","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-29T20:37:37Z","receivedAt":"2024-01-29T20:37:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> This is the second version of my patch series to print refs\n> when and empty string pattern is used with git-for-each-ref(1).\n\nThanks. This is probably the third if I am counting correctly, and\nthe changes relative to the previous round match what I was\nexpecting to see.\n\nWill queue.\n"},{"id":"487944","messageId":"98d79d33-0d7e-4a9c-a6a3-ed9b58cd7445@gmail.com","threadId":"60768","inReplyTo":"20240129113527.607022-5-karthik.188@gmail.com","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2024-02-05T18:48:52Z","receivedAt":"2024-02-05T18:48:55Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Karthik\n\nOn 29/01/2024 11:35, Karthik Nayak wrote:\n> When the user uses an empty string pattern (\"\"), we don't match any refs\n> in git-for-each-ref(1). This is because in git-for-each-ref(1), we use\n> path based matching and an empty string doesn't match any path.\n>\n> In this commit we change this behavior by making empty string pattern\n> match all references. This is done by introducing a new flag\n> `FILTER_REFS_NO_FILTER` in `ref-filter.c`, which uses the newly\n> introduced `refs_for_each_all_refs()` function to iterate over all the\n> refs in the repository.\n\nIt actually iterates over all the refs in the current worktree, not all \nthe refs in the repository. I have to say that I find it extremely \nunintuitive that \"\" behaves differently to not giving a pattern. I \nwonder if we can find a better UI here - perhaps a command line option \nto include pseudorefs?\n\n> Signed-off-by: Karthik Nayak \n> ---\n> Documentation/git-for-each-ref.txt | 3 ++-\n> builtin/for-each-ref.c | 21 +++++++++++++++++-\n> ref-filter.c | 13 ++++++++++--\n> ref-filter.h | 4 +++-\n> t/t6302-for-each-ref-filter.sh | 34 ++++++++++++++++++++++++++++++\n> 5 files changed, 70 insertions(+), 5 deletions(-)\n> \n> diff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\n> index be9543f684..b1cb482bf5 100644\n> --- a/Documentation/git-for-each-ref.txt\n> +++ b/Documentation/git-for-each-ref.txt\n> @@ -32,7 +32,8 @@ OPTIONS\n> \tIf one or more patterns are given, only refs are shown that\n> \tmatch against at least one pattern, either using fnmatch(3) or\n> \tliterally, in the latter case matching completely or from the\n> -\tbeginning up to a slash.\n> +\tbeginning up to a slash. If an empty string is provided all refs\n> +\tare printed, including HEAD and pseudorefs.\n\nI think it would be helpful to clarify that it is all the refs for the \ncurrent worktree that are printed, not all the refs in the repository - \nwe still don't have a way to iterate over the per-worktree refs from \nother worktrees\n\nBest Wishes\n\nPhillip\n\n"},{"id":"488022","messageId":"ZcHEmyvvMR_b_Idl@tanuki","threadId":"60768","inReplyTo":"98d79d33-0d7e-4a9c-a6a3-ed9b58cd7445@gmail.com","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-06T05:33:15Z","receivedAt":"2024-02-06T05:33:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 05, 2024 at 06:48:52PM +0000, Phillip Wood wrote:\n> Hi Karthik\n> \n> On 29/01/2024 11:35, Karthik Nayak wrote:\n> > When the user uses an empty string pattern (\"\"), we don't match any refs\n> > in git-for-each-ref(1). This is because in git-for-each-ref(1), we use\n> > path based matching and an empty string doesn't match any path.\n> > \n> > In this commit we change this behavior by making empty string pattern\n> > match all references. This is done by introducing a new flag\n> > `FILTER_REFS_NO_FILTER` in `ref-filter.c`, which uses the newly\n> > introduced `refs_for_each_all_refs()` function to iterate over all the\n> > refs in the repository.\n> \n> It actually iterates over all the refs in the current worktree, not all the\n> refs in the repository. I have to say that I find it extremely unintuitive\n> that \"\" behaves differently to not giving a pattern. I wonder if we can find\n> a better UI here - perhaps a command line option to include pseudorefs?\n\nI tend to agree, and have argued for a flag in another thread, too [1].\nSomething like `--show-all-refs` feels a lot more intuitive to me and\nalso doesn't have the potential to break preexisting scripts which pass\nthe empty prefix (which would have returned the empty set of refs, but\nnow returns all refs).\n\n[1]: https://lore.kernel.org/git/ZZWCXFghtql4i4YE@tanuki/\n\nPatrick\n"},{"id":"488055","messageId":"CAOLa=ZR=_tt=ppphGMkxqj_YB5G+YkTMWGzRzcHTbrZz4ysb5w@mail.gmail.com","threadId":"60768","inReplyTo":"98d79d33-0d7e-4a9c-a6a3-ed9b58cd7445@gmail.com","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-06T08:52:42Z","receivedAt":"2024-02-06T08:53:09Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello,\n\nOn Mon, Feb 5, 2024 at 7:48 PM Phillip Wood wrote:\n>\n> Hi Karthik\n>\n> On 29/01/2024 11:35, Karthik Nayak wrote:\n> > When the user uses an empty string pattern (\"\"), we don't match any refs\n> > in git-for-each-ref(1). This is because in git-for-each-ref(1), we use\n> > path based matching and an empty string doesn't match any path.\n> >\n> > In this commit we change this behavior by making empty string pattern\n> > match all references. This is done by introducing a new flag\n> > `FILTER_REFS_NO_FILTER` in `ref-filter.c`, which uses the newly\n> > introduced `refs_for_each_all_refs()` function to iterate over all the\n> > refs in the repository.\n>\n> It actually iterates over all the refs in the current worktree, not all\n> the refs in the repository. I have to say that I find it extremely\n> unintuitive that \"\" behaves differently to not giving a pattern. I\n> wonder if we can find a better UI here - perhaps a command line option\n> to include pseudorefs?\n>\n\nAs Patrick mentioned, this was discussed a while ago and we decided to\nmove forward with the `git for-each-ref \"\"` syntax.\n\n> > Signed-off-by: Karthik Nayak \n> > ---\n> > Documentation/git-for-each-ref.txt | 3 ++-\n> > builtin/for-each-ref.c | 21 +++++++++++++++++-\n> > ref-filter.c | 13 ++++++++++--\n> > ref-filter.h | 4 +++-\n> > t/t6302-for-each-ref-filter.sh | 34 ++++++++++++++++++++++++++++++\n> > 5 files changed, 70 insertions(+), 5 deletions(-)\n> >\n> > diff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\n> > index be9543f684..b1cb482bf5 100644\n> > --- a/Documentation/git-for-each-ref.txt\n> > +++ b/Documentation/git-for-each-ref.txt\n> > @@ -32,7 +32,8 @@ OPTIONS\n> > If one or more patterns are given, only refs are shown that\n> > match against at least one pattern, either using fnmatch(3) or\n> > literally, in the latter case matching completely or from the\n> > - beginning up to a slash.\n> > + beginning up to a slash. If an empty string is provided all refs\n> > + are printed, including HEAD and pseudorefs.\n>\n> I think it would be helpful to clarify that it is all the refs for the\n> current worktree that are printed, not all the refs in the repository -\n> we still don't have a way to iterate over the per-worktree refs from\n> other worktrees\n>\n\nI agree, based on if we keep the current commits or remove them, I'll\nsend in a new patch or amend the current series.\n\nThanks!\n"},{"id":"488066","messageId":"c9d660a3-4254-4ac7-9476-b7817348aa60@gmail.com","threadId":"60768","inReplyTo":"ZcHEmyvvMR_b_Idl@tanuki","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2024-02-06T10:49:58Z","receivedAt":"2024-02-06T10:50:04Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Patrick\n\nOn 06/02/2024 05:33, Patrick Steinhardt wrote:\n> On Mon, Feb 05, 2024 at 06:48:52PM +0000, Phillip Wood wrote:\n>> Hi Karthik\n>>\n>> On 29/01/2024 11:35, Karthik Nayak wrote:\n>>> When the user uses an empty string pattern (\"\"), we don't match any refs\n>>> in git-for-each-ref(1). This is because in git-for-each-ref(1), we use\n>>> path based matching and an empty string doesn't match any path.\n>>>\n>>> In this commit we change this behavior by making empty string pattern\n>>> match all references. This is done by introducing a new flag\n>>> `FILTER_REFS_NO_FILTER` in `ref-filter.c`, which uses the newly\n>>> introduced `refs_for_each_all_refs()` function to iterate over all the\n>>> refs in the repository.\n>>\n>> It actually iterates over all the refs in the current worktree, not all the\n>> refs in the repository. I have to say that I find it extremely unintuitive\n>> that \"\" behaves differently to not giving a pattern. I wonder if we can find\n>> a better UI here - perhaps a command line option to include pseudorefs?\n> \n> I tend to agree, and have argued for a flag in another thread, too [1].\n\nThanks for that, I'd missed that discussion\n\n> Something like `--show-all-refs` feels a lot more intuitive to me and\n> also doesn't have the potential to break preexisting scripts which pass\n> the empty prefix (which would have returned the empty set of refs, but\n> now returns all refs).\n\nYes, and as you point out in that other thread flag would allow the \npseuderefs to be filtered and allow us to show the refs for all \nworktrees as well in the future.\n\nBest Wishes\n\nPhillip\n\n> [1]: https://lore.kernel.org/git/ZZWCXFghtql4i4YE@tanuki/\n> \n> Patrick\n"},{"id":"488070","messageId":"92ba680d-0b48-49f0-aafc-f503e5a5e0ea@gmail.com","threadId":"60768","inReplyTo":"CAOLa=ZR=_tt=ppphGMkxqj_YB5G+YkTMWGzRzcHTbrZz4ysb5w@mail.gmail.com","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2024-02-06T13:55:02Z","receivedAt":"2024-02-06T13:55:05Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Karthik\n\nOn 06/02/2024 08:52, Karthik Nayak wrote:\n> On Mon, Feb 5, 2024 at 7:48 PM Phillip Wood wrote:\n>> On 29/01/2024 11:35, Karthik Nayak wrote:\n>>> When the user uses an empty string pattern (\"\"), we don't match any refs\n>>> in git-for-each-ref(1). This is because in git-for-each-ref(1), we use\n>>> path based matching and an empty string doesn't match any path.\n>>>\n>>> In this commit we change this behavior by making empty string pattern\n>>> match all references. This is done by introducing a new flag\n>>> `FILTER_REFS_NO_FILTER` in `ref-filter.c`, which uses the newly\n>>> introduced `refs_for_each_all_refs()` function to iterate over all the\n>>> refs in the repository.\n>>\n>> It actually iterates over all the refs in the current worktree, not all\n>> the refs in the repository. I have to say that I find it extremely\n>> unintuitive that \"\" behaves differently to not giving a pattern. I\n>> wonder if we can find a better UI here - perhaps a command line option\n>> to include pseudorefs?\n>>\n> \n> As Patrick mentioned, this was discussed a while ago and we decided to\n> move forward with the `git for-each-ref \"\"` syntax.\n\nThanks I'd missed that discussion. I see that at the end of that \ndiscussion Junio was concerned that the proposed \"\" did not account for \n\"refs/worktrees/$worktree/*\" [1] - how has that been resolved?\n\nBest Wishes\n\nPhillip\n\n\n[1] https://lore.kernel.org/git/xmqq1qawr6p4.fsf@gitster.g/\n\n>>> Signed-off-by: Karthik Nayak \n>>> ---\n>>> Documentation/git-for-each-ref.txt | 3 ++-\n>>> builtin/for-each-ref.c | 21 +++++++++++++++++-\n>>> ref-filter.c | 13 ++++++++++--\n>>> ref-filter.h | 4 +++-\n>>> t/t6302-for-each-ref-filter.sh | 34 ++++++++++++++++++++++++++++++\n>>> 5 files changed, 70 insertions(+), 5 deletions(-)\n>>>\n>>> diff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\n>>> index be9543f684..b1cb482bf5 100644\n>>> --- a/Documentation/git-for-each-ref.txt\n>>> +++ b/Documentation/git-for-each-ref.txt\n>>> @@ -32,7 +32,8 @@ OPTIONS\n>>> If one or more patterns are given, only refs are shown that\n>>> match against at least one pattern, either using fnmatch(3) or\n>>> literally, in the latter case matching completely or from the\n>>> - beginning up to a slash.\n>>> + beginning up to a slash. If an empty string is provided all refs\n>>> + are printed, including HEAD and pseudorefs.\n>>\n>> I think it would be helpful to clarify that it is all the refs for the\n>> current worktree that are printed, not all the refs in the repository -\n>> we still don't have a way to iterate over the per-worktree refs from\n>> other worktrees\n>>\n> \n> I agree, based on if we keep the current commits or remove them, I'll\n> send in a new patch or amend the current series.\n> \n> Thanks!\n\n"},{"id":"488074","messageId":"CAOLa=ZRLuy2H-r3A0BE-mS4Zm=ebhLR+SemZ3nAghEnRB5dR_A@mail.gmail.com","threadId":"60768","inReplyTo":"92ba680d-0b48-49f0-aafc-f503e5a5e0ea@gmail.com","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-06T15:30:32Z","receivedAt":"2024-02-06T15:30:59Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello,\n\nOn Tue, Feb 6, 2024 at 2:55 PM Phillip Wood wrote:\n>\n> Hi Karthik\n>\n> On 06/02/2024 08:52, Karthik Nayak wrote:\n> > On Mon, Feb 5, 2024 at 7:48 PM Phillip Wood wrote:\n> >> On 29/01/2024 11:35, Karthik Nayak wrote:\n> >>> When the user uses an empty string pattern (\"\"), we don't match any refs\n> >>> in git-for-each-ref(1). This is because in git-for-each-ref(1), we use\n> >>> path based matching and an empty string doesn't match any path.\n> >>>\n> >>> In this commit we change this behavior by making empty string pattern\n> >>> match all references. This is done by introducing a new flag\n> >>> `FILTER_REFS_NO_FILTER` in `ref-filter.c`, which uses the newly\n> >>> introduced `refs_for_each_all_refs()` function to iterate over all the\n> >>> refs in the repository.\n> >>\n> >> It actually iterates over all the refs in the current worktree, not all\n> >> the refs in the repository. I have to say that I find it extremely\n> >> unintuitive that \"\" behaves differently to not giving a pattern. I\n> >> wonder if we can find a better UI here - perhaps a command line option\n> >> to include pseudorefs?\n> >>\n> >\n> > As Patrick mentioned, this was discussed a while ago and we decided to\n> > move forward with the `git for-each-ref \"\"` syntax.\n>\n> Thanks I'd missed that discussion. I see that at the end of that\n> discussion Junio was concerned that the proposed \"\" did not account for\n> \"refs/worktrees/$worktree/*\" [1] - how has that been resolved?\n>\n\nThe current implementation (merged to next) prints all the refs from the current\nworktree and doesn't support printing from other worktrees.\n\n $ git worktree add ../wt-files @~10\n Preparing worktree (detached HEAD 559d5138bc)\n HEAD is now at 559d5138bc Merge branch 'jc/make-libpath-template' into next\n\n $ cd ../wt-files/\n direnv: unloading\n\n $ git for-each-ref \"\" | grep -v \"refs\"\n 559d5138bc8b81354fd1bc3421ace614076e66de commit HEAD\n 559d5138bc8b81354fd1bc3421ace614076e66de commit ORIG_HEAD\n\n $ git rebase -i @~3\n Stopped at be65f9ef88... t/Makefile: get UNIT_TESTS list from C sources\n You can amend the commit now, with\n\n git commit --amend '-S'\n\n Once you are satisfied with your changes, run\n\n git rebase --continue\n\n $ git for-each-ref \"\" | grep -v \"refs\"\n be65f9ef88ff741454dcf10a0af6e384d0bf26cf commit HEAD\n 43f081b9c7dfb9c23e547ffee1778af0f30c5c4e commit ORIG_HEAD\n be65f9ef88ff741454dcf10a0af6e384d0bf26cf commit REBASE_HEAD\n"},{"id":"488077","messageId":"xmqqle7xjzic.fsf@gitster.g","threadId":"60768","inReplyTo":"92ba680d-0b48-49f0-aafc-f503e5a5e0ea@gmail.com","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-06T17:03:55Z","receivedAt":"2024-02-06T17:04:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n> Thanks I'd missed that discussion. I see that at the end of that\n> discussion Junio was concerned that the proposed \"\" did not account\n> for \"refs/worktrees/$worktree/*\" [1] - how has that been resolved?\n\nAh, that is an excellent point.\n\nIf we plan to never allow showing refs/worktrees/ hierarchy, then\nthe \"there is a default pattern, refs/, that gets used when there is\nno user-specified patterns\" model would be sufficient to allow\nshowing things that are directly beneath $GIT_DIR and are out of\nrefs/ hierarchy, but that does not explain why refs/worktrees/ is\nomitted. I'll envision a design for a longer term later, but an\neasy way out would be to add --include-worktree-refs option for\nthat, and at that point, adding --include-root-refs option for things\noutside the refs/ hierarchy may become a lot more natural solution.\n\nIn the longer term, I suspect that we would want something similar\nto the negative refspec magic (e.g., \"git log ':!Documentation/'\"\nthat shows things outside the named hierarchy) exposed to the API[*],\nso that we can say\n\n $ git for-each-ref --format=... \\\n\trefs/ !refs/heads/ !refs/tags/ !refs/remotes/\n\nto show things under refs/ excluding the commonly used hierarchies,\nand at that point, the current behaviour for \"no limit\" case can\nagain become explainable as having \"refs/ !refs/worktrees/\" as the\ndefault. It still does not explain why \"git for-each-ref refs/\"\nomits the refs/worktrees/ hierchy, unless the default limit pattern\nrule were something like \"unless you give a positive limit pattern\nrule, then we use 'refs/' by default, and unless you give a negative\nlimit pattern rule, then we use '!refs/worktrees/' by default\".\n\nIt then gives an easy explanation on the traditional behaviour, with\n\"\" used for \"including stuff outside refs/\", and is more flexible.\n\nThe use of dashed-options to include hierachies that are by default\nexcluded (e.g. \"--include-root-refs\" and \"--include-worktree-refs\")\nfeels limiting, but should be sufficient for our needs, both current\n(i.e. I want to see HEAD and FETCH_HEAD) and in the immediate future\n(i.e. I want to see worktree refs from that worktree), and I can buy\nthat as a good alternative solution, at least in the shorter term.\n\nI still worry that it may make introducing the negative ref patterns\nharder, though. How does --include-worktree-refs=another to include\nthe worktree refs from another worktree in refs/worktrees/another\ninteract with a negative pattern that was given from the command\nline that overlaps with it? Whatever interaction rules we define,\ncan we easily explain it in the documentation?\n\nJust like \"an empty string tells Git to include everything\" is a\nperfectly reasonable approach if we plan to never allow\nrefs/worktrees/ hierarchy, \"dashed-options for selected hierarchies\"\nis a perfectly reasonable approach if we plan to never allow\nnegative limit patterns, I suspect. We should stop complexity at\nsome point, and the decision to never support negative limit\npatterns might be the place to draw that line. I dunno.\n\n\n[Footnote]\n\n * Such an exclusion mechanism already exists and are used to hide\n certain refs from being seen over the network by \"git fetch\" and\n friends. I do not think it is plugged to the machinery used by\n for-each-ref and friends, but it smells like a reasonably easy\n thing to do.\n"},{"id":"488086","messageId":"xmqqr0hph1ku.fsf@gitster.g","threadId":"60768","inReplyTo":"xmqqle7xjzic.fsf@gitster.g","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-06T18:47:29Z","receivedAt":"2024-02-06T18:47:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Phillip Wood writes:\n>\n>> Thanks I'd missed that discussion. I see that at the end of that\n>> discussion Junio was concerned that the proposed \"\" did not account\n>> for \"refs/worktrees/$worktree/*\" [1] - how has that been resolved?\n>\n> Ah, that is an excellent point.\n> ...\n> The use of dashed-options to include hierachies that are by default\n> excluded (e.g. \"--include-root-refs\" and \"--include-worktree-refs\")\n> feels limiting, but should be sufficient for our needs, both current\n> (i.e. I want to see HEAD and FETCH_HEAD) and in the immediate future\n> (i.e. I want to see worktree refs from that worktree), and I can buy\n> that as a good alternative solution, at least in the shorter term.\n>\n> I still worry that it may make introducing the negative ref patterns\n> harder, though. How does --include-worktree-refs=another to include\n> the worktree refs from another worktree in refs/worktrees/another\n> interact with a negative pattern that was given from the command\n> line that overlaps with it? Whatever interaction rules we define,\n> can we easily explain it in the documentation?\n>\n> Just like \"an empty string tells Git to include everything\" is a\n> perfectly reasonable approach if we plan to never allow\n> refs/worktrees/ hierarchy, \"dashed-options for selected hierarchies\"\n> is a perfectly reasonable approach if we plan to never allow\n> negative limit patterns, I suspect. We should stop complexity at\n> some point, and the decision to never support negative limit\n> patterns might be the place to draw that line. I dunno.\n\nFor now, let's block the kn/for-all-refs topic until we figure out\nthe UI issue. Which means this (and the reftable integration that\nstarted to depend on it) will not be in the upcoming release.\n\nFWIW, I am leaning towards \"a set of narrowly targetted command line\noptions (like \"--include-root-refs\")\" approach over \"a empty string\ndefeats the built-in default of 'refs/' limit\".\n\nThanks.\n"},{"id":"488105","messageId":"CAOLa=ZSZJ=_VCppHXcJeE=Z61go4_040xyc1NBTu-o=xysLrdg@mail.gmail.com","threadId":"60768","inReplyTo":"xmqqr0hph1ku.fsf@gitster.g","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-06T22:10:41Z","receivedAt":"2024-02-06T22:10:43Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Junio C Hamano writes:\n>\n>> Phillip Wood writes:\n>>\n>>> Thanks I'd missed that discussion. I see that at the end of that\n>>> discussion Junio was concerned that the proposed \"\" did not account\n>>> for \"refs/worktrees/$worktree/*\" [1] - how has that been resolved?\n>>\n>> Ah, that is an excellent point.\n>> ...\n>> The use of dashed-options to include hierachies that are by default\n>> excluded (e.g. \"--include-root-refs\" and \"--include-worktree-refs\")\n>> feels limiting, but should be sufficient for our needs, both current\n>> (i.e. I want to see HEAD and FETCH_HEAD) and in the immediate future\n>> (i.e. I want to see worktree refs from that worktree), and I can buy\n>> that as a good alternative solution, at least in the shorter term.\n>>\n>> I still worry that it may make introducing the negative ref patterns\n>> harder, though. How does --include-worktree-refs=another to include\n>> the worktree refs from another worktree in refs/worktrees/another\n>> interact with a negative pattern that was given from the command\n>> line that overlaps with it? Whatever interaction rules we define,\n>> can we easily explain it in the documentation?\n>>\n>> Just like \"an empty string tells Git to include everything\" is a\n>> perfectly reasonable approach if we plan to never allow\n>> refs/worktrees/ hierarchy, \"dashed-options for selected hierarchies\"\n>> is a perfectly reasonable approach if we plan to never allow\n>> negative limit patterns, I suspect. We should stop complexity at\n>> some point, and the decision to never support negative limit\n>> patterns might be the place to draw that line. I dunno.\n>\n> For now, let's block the kn/for-all-refs topic until we figure out\n> the UI issue. Which means this (and the reftable integration that\n> started to depend on it) will not be in the upcoming release.\n>\n\nI think it makes sense to remove the kn/for-all-refs topic for now.\n\nI also think that the reftable integration branch can go forward though,\nsince the difference between v2 and v3 of that series was simply that\nv3 was rebased on top of kn/for-all-refs. So we can continue using v2.\n\n> FWIW, I am leaning towards \"a set of narrowly targetted command line\n> options (like \"--include-root-refs\")\" approach over \"a empty string\n> defeats the built-in default of 'refs/' limit\".\n\nI think this was what the earlier discussion in the RFC series was too,\nbut Phillip definitely helped consolidate the point.\n\nI'll send a new version of this patch series with `--include-root-refs`\noption and we can discuss on top of that.\n\nThanks all!\n"},{"id":"488106","messageId":"xmqqcyt9fdc7.fsf@gitster.g","threadId":"60768","inReplyTo":"CAOLa=ZSZJ=_VCppHXcJeE=Z61go4_040xyc1NBTu-o=xysLrdg@mail.gmail.com","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-06T22:16:24Z","receivedAt":"2024-02-06T22:16:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> I think this was what the earlier discussion in the RFC series was too,\n> but Phillip definitely helped consolidate the point.\n>\n> I'll send a new version of this patch series with `--include-root-refs`\n> option and we can discuss on top of that.\n\nThanks.\n\nBy the way, I am not married to the \"root refs\" name, but\n\n - we do not want to say \"all refs\", as I expect refs from other\n worktrees are not included, and possibly the option when\n combined with explicit patterns, like refs/tags/, may further be\n used to limit the output;\n\n - we do not want to say \"pseudo refs\", as I expect we would want to\n show HEAD that is (unfortunately) classified outside \"pseudoref\"\n class.\n\n"},{"id":"488116","messageId":"20240207014819.GA1399428@coredump.intra.peff.net","threadId":"60768","inReplyTo":"20240129113527.607022-2-karthik.188@gmail.com","subject":"Re: [PATCH v3 1/4] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-02-07T01:48:19Z","receivedAt":"2024-02-07T01:48:26Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 29, 2024 at 12:35:24PM +0100, Karthik Nayak wrote:\n\n> +int is_pseudoref(struct ref_store *refs, const char *refname)\n> [...]\n> +\tif (ends_with(refname, \"_HEAD\")) {\n> +\t\t read_ref_full(refname, RESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n> +\t\t &oid, NULL);\n> +\t\t return !is_null_oid(&oid);\n> +\t}\n\nI was going to prepare a patch on top, but since it looks like this was\nreverted out of 'next' to be revamped, I thought I'd mention it now:\n-Wunused-parameter notices that we never use the \"refs\" parameter to the\nfunction. And indeed it looks like a (possible) bug, since\nread_ref_full() is going to use the_repository to find the ref store.\n\nI think you'd want something like this squashed in (note that it also\nfixes a slight indent problem in the first block):\n\ndiff --git a/refs.c b/refs.c\nindex 3190df8d81..0d65c31117 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -875,15 +875,17 @@ int is_pseudoref(struct ref_store *refs, const char *refname)\n \t\treturn 0;\n \n \tif (ends_with(refname, \"_HEAD\")) {\n-\t\t read_ref_full(refname, RESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n-\t\t &oid, NULL);\n-\t\t return !is_null_oid(&oid);\n+\t\trefs_resolve_ref_unsafe(refs, refname,\n+\t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n+\t\t\t\t\t&oid, NULL);\n+\t\treturn !is_null_oid(&oid);\n \t}\n \n \tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n \t\tif (!strcmp(refname, irregular_pseudorefs[i])) {\n-\t\t\tread_ref_full(refname, RESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n-\t\t\t\t\t\t &oid, NULL);\n+\t\t\trefs_resolve_ref_unsafe(refs, refname,\n+\t\t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n+\t\t\t\t\t\t&oid, NULL);\n \t\t\treturn !is_null_oid(&oid);\n \t\t}\n \n\n-Peff\n"},{"id":"488126","messageId":"ZcM1wTrS3cqln8yF@tanuki","threadId":"60768","inReplyTo":"CAOLa=ZSZJ=_VCppHXcJeE=Z61go4_040xyc1NBTu-o=xysLrdg@mail.gmail.com","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-07T07:48:17Z","receivedAt":"2024-02-07T07:48:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 06, 2024 at 02:10:41PM -0800, Karthik Nayak wrote:\n> Junio C Hamano writes:\n> > Junio C Hamano writes:\n> >> Phillip Wood writes:\n[snip]\n> > For now, let's block the kn/for-all-refs topic until we figure out\n> > the UI issue. Which means this (and the reftable integration that\n> > started to depend on it) will not be in the upcoming release.\n> >\n> \n> I think it makes sense to remove the kn/for-all-refs topic for now.\n> \n> I also think that the reftable integration branch can go forward though,\n> since the difference between v2 and v3 of that series was simply that\n> v3 was rebased on top of kn/for-all-refs. So we can continue using v2.\n\nI've sent a rebased v4 that evicted the kn/for-all-refs dependency. This\nalso allowed me to adapt to some fixes for the reftable library which\nhave been merged down to `master` now so that the corresponding tests\nare now with `test_expect_success`.\n\nPatrick\n"},{"id":"488127","messageId":"CAOLa=ZRSmbadZ5uhk1PVOV=bjRadNakOwZP2HmmVwbc3VKGkyg@mail.gmail.com","threadId":"60768","inReplyTo":"20240207014819.GA1399428@coredump.intra.peff.net","subject":"Re: [PATCH v3 1/4] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-07T09:27:37Z","receivedAt":"2024-02-07T09:27:39Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Jeff King writes:\n\n> On Mon, Jan 29, 2024 at 12:35:24PM +0100, Karthik Nayak wrote:\n>\n>> +int is_pseudoref(struct ref_store *refs, const char *refname)\n>> [...]\n>> +\tif (ends_with(refname, \"_HEAD\")) {\n>> +\t\t read_ref_full(refname, RESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n>> +\t\t &oid, NULL);\n>> +\t\t return !is_null_oid(&oid);\n>> +\t}\n>\n> I was going to prepare a patch on top, but since it looks like this was\n> reverted out of 'next' to be revamped, I thought I'd mention it now:\n> -Wunused-parameter notices that we never use the \"refs\" parameter to the\n> function. And indeed it looks like a (possible) bug, since\n> read_ref_full() is going to use the_repository to find the ref store.\n>\n> I think you'd want something like this squashed in (note that it also\n> fixes a slight indent problem in the first block):\n>\n> diff --git a/refs.c b/refs.c\n> index 3190df8d81..0d65c31117 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -875,15 +875,17 @@ int is_pseudoref(struct ref_store *refs, const char *refname)\n> \t\treturn 0;\n>\n> \tif (ends_with(refname, \"_HEAD\")) {\n> -\t\t read_ref_full(refname, RESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n> -\t\t &oid, NULL);\n> -\t\t return !is_null_oid(&oid);\n> +\t\trefs_resolve_ref_unsafe(refs, refname,\n> +\t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n> +\t\t\t\t\t&oid, NULL);\n> +\t\treturn !is_null_oid(&oid);\n> \t}\n>\n> \tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n> \t\tif (!strcmp(refname, irregular_pseudorefs[i])) {\n> -\t\t\tread_ref_full(refname, RESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n> -\t\t\t\t\t\t &oid, NULL);\n> +\t\t\trefs_resolve_ref_unsafe(refs, refname,\n> +\t\t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n> +\t\t\t\t\t\t&oid, NULL);\n> \t\t\treturn !is_null_oid(&oid);\n> \t\t}\n>\n>\n> -Peff\n\nThanks Jeff, makes sense, will squash this in.\n"},{"id":"488137","messageId":"CAOLa=ZRcWYmEYnxh_=ykOerahZ61xaanLCj_JHHLvtrvN=Xs-w@mail.gmail.com","threadId":"60768","inReplyTo":"xmqqcyt9fdc7.fsf@gitster.g","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-07T14:10:27Z","receivedAt":"2024-02-07T14:10:55Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"On Tue, Feb 6, 2024 at 11:16 PM Junio C Hamano wrote:\n>\n> Karthik Nayak writes:\n>\n> > I think this was what the earlier discussion in the RFC series was too,\n> > but Phillip definitely helped consolidate the point.\n> >\n> > I'll send a new version of this patch series with `--include-root-refs`\n> > option and we can discuss on top of that.\n>\n> Thanks.\n>\n> By the way, I am not married to the \"root refs\" name, but\n>\n> - we do not want to say \"all refs\", as I expect refs from other\n> worktrees are not included, and possibly the option when\n> combined with explicit patterns, like refs/tags/, may further be\n> used to limit the output;\n>\n> - we do not want to say \"pseudo refs\", as I expect we would want to\n> show HEAD that is (unfortunately) classified outside \"pseudoref\"\n> class.\n>\n\nI'm thinking \"--all-ref-types\" might be a good alternative. Mostly because,\n\"--include-root-refs\" seems very specific to the files backend. Also, we don't\ninclude other refs which are not HEAD | pseudorefs, but in the $GIT_DIR.\n"},{"id":"488138","messageId":"xmqq1q9oe029.fsf@gitster.g","threadId":"60768","inReplyTo":"CAOLa=ZRcWYmEYnxh_=ykOerahZ61xaanLCj_JHHLvtrvN=Xs-w@mail.gmail.com","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-07T16:00:46Z","receivedAt":"2024-02-07T16:00:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> I'm thinking \"--all-ref-types\" might be a good alternative. Mostly because,\n> \"--include-root-refs\" seems very specific to the files backend. Also, we don't\n> include other refs which are not HEAD | pseudorefs, but in the $GIT_DIR.\n\nI strongly disagree wiht the \"files backend specific\" part of the\ncomment. No matter what backend you would use, refs and pseudorefs\nhave the full refname, which may look like \"HEAD\", \"FETCH_HEAD\",\n\"refs/heads/maint\", etc., and you can easily see these full refnames\nform a tree structure, with \"HEAD\", \"FETCH_HEAD\", \"refs/\" at the\nroot level.\n\nI do not understand your \"we don't include other refs\", either.\nThere may be \"things\" that are ignored by your implementation of\n\"for-each-ref ''\" even with the files backend in $GIT_DIR directory.\nThey are not refs, because the refs are by definition inside \"refs/\"\nhierarchy, unless they are ones that are specifically included from\noutside the hierarchy (\"pseudorefs\" is one class of specific\nexception, \"HEAD\" is another).\n"},{"id":"488139","messageId":"xmqqwmrgclh4.fsf@gitster.g","threadId":"60768","inReplyTo":"ZcM1wTrS3cqln8yF@tanuki","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-07T16:01:11Z","receivedAt":"2024-02-07T16:01:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> I also think that the reftable integration branch can go forward though,\n>> since the difference between v2 and v3 of that series was simply that\n>> v3 was rebased on top of kn/for-all-refs. So we can continue using v2.\n>\n> I've sent a rebased v4 that evicted the kn/for-all-refs dependency. This\n> also allowed me to adapt to some fixes for the reftable library which\n> have been merged down to `master` now so that the corresponding tests\n> are now with `test_expect_success`.\n\nThanks, will take a look.\n"},{"id":"488143","messageId":"CAOLa=ZQzz7_L_9cBmK+pgFwd_DFqfWDVRiaZMAxU+54kBq6Pcw@mail.gmail.com","threadId":"60768","inReplyTo":"xmqq1q9oe029.fsf@gitster.g","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-07T16:18:56Z","receivedAt":"2024-02-07T16:19:24Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"On Wed, Feb 7, 2024 at 5:00 PM Junio C Hamano wrote:\n>\n> Karthik Nayak writes:\n>\n> > I'm thinking \"--all-ref-types\" might be a good alternative. Mostly because,\n> > \"--include-root-refs\" seems very specific to the files backend. Also, we don't\n> > include other refs which are not HEAD | pseudorefs, but in the $GIT_DIR.\n>\n> I strongly disagree wiht the \"files backend specific\" part of the\n> comment. No matter what backend you would use, refs and pseudorefs\n> have the full refname, which may look like \"HEAD\", \"FETCH_HEAD\",\n> \"refs/heads/maint\", etc., and you can easily see these full refnames\n> form a tree structure, with \"HEAD\", \"FETCH_HEAD\", \"refs/\" at the\n> root level.\n\nI conceded to this point, I was thinking \"root\" here refers to $GIT_DIR\nand this structuring comes from the files backend. But I see the flaw there\nthat irrelevant of the backend, there is a tree hierarchy built up and for refs\nwithout prefixes, it can be considered as \"root\".\n\n> I do not understand your \"we don't include other refs\", either.\n> There may be \"things\" that are ignored by your implementation of\n> \"for-each-ref ''\" even with the files backend in $GIT_DIR directory.\n> They are not refs, because the refs are by definition inside \"refs/\"\n> hierarchy, unless they are ones that are specifically included from\n> outside the hierarchy (\"pseudorefs\" is one class of specific\n> exception, \"HEAD\" is another).\n\nThis is a bit of a grey area, what I mean is that we do allow users to create\nnon \"refs/\" prefixed refs:\n\n $ git update-ref foo @~1\n\n $ cat .git/foo\n 2b52187cd2930931c6d34436371f470bb26eef4f\n\nWhat I mean to say is that, by saying \"--include-root-refs\" it seems to imply\nthat any such refs should be included too, but this simply is not the case.\n"},{"id":"488148","messageId":"xmqq1q9ocje3.fsf@gitster.g","threadId":"60768","inReplyTo":"CAOLa=ZQzz7_L_9cBmK+pgFwd_DFqfWDVRiaZMAxU+54kBq6Pcw@mail.gmail.com","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-07T16:46:12Z","receivedAt":"2024-02-07T16:46:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> This is a bit of a grey area, what I mean is that we do allow users to create\n> non \"refs/\" prefixed refs:\n>\n> $ git update-ref foo @~1\n>\n> $ cat .git/foo\n> 2b52187cd2930931c6d34436371f470bb26eef4f\n>\n> What I mean to say is that, by saying \"--include-root-refs\" it seems to imply\n> that any such refs should be included too, but this simply is not the case.\n\nBut isn't that a quality of implementation issue? I'd consider it a\nbug once we have and enforce the definition of what pseudorefs are.\n"},{"id":"488161","messageId":"CAOLa=ZQaXxwrXmbmFvGR59EDo3Eqa-Xfc3OG9+6ES-veDU8Bhg@mail.gmail.com","threadId":"60768","inReplyTo":"xmqq1q9ocje3.fsf@gitster.g","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-07T17:02:34Z","receivedAt":"2024-02-07T17:03:03Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"On Wed, Feb 7, 2024 at 5:46 PM Junio C Hamano wrote:\n>\n> Karthik Nayak writes:\n>\n> > This is a bit of a grey area, what I mean is that we do allow users to create\n> > non \"refs/\" prefixed refs:\n> >\n> > $ git update-ref foo @~1\n> >\n> > $ cat .git/foo\n> > 2b52187cd2930931c6d34436371f470bb26eef4f\n> >\n> > What I mean to say is that, by saying \"--include-root-refs\" it seems to imply\n> > that any such refs should be included too, but this simply is not the case.\n>\n> But isn't that a quality of implementation issue? I'd consider it a\n> bug once we have and enforce the definition of what pseudorefs are.\n\nYeah, that makes sense. I'll use \"--include-root-refs\" then.\n\nThanks\n"},{"id":"488221","messageId":"ZcSVx4slikt4xB3D@tanuki","threadId":"60768","inReplyTo":"CAOLa=ZQaXxwrXmbmFvGR59EDo3Eqa-Xfc3OG9+6ES-veDU8Bhg@mail.gmail.com","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-08T08:50:15Z","receivedAt":"2024-02-08T08:50:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Feb 07, 2024 at 06:02:34PM +0100, Karthik Nayak wrote:\n> On Wed, Feb 7, 2024 at 5:46 PM Junio C Hamano wrote:\n> >\n> > Karthik Nayak writes:\n> >\n> > > This is a bit of a grey area, what I mean is that we do allow users to create\n> > > non \"refs/\" prefixed refs:\n> > >\n> > > $ git update-ref foo @~1\n> > >\n> > > $ cat .git/foo\n> > > 2b52187cd2930931c6d34436371f470bb26eef4f\n> > >\n> > > What I mean to say is that, by saying \"--include-root-refs\" it seems to imply\n> > > that any such refs should be included too, but this simply is not the case.\n> >\n> > But isn't that a quality of implementation issue? I'd consider it a\n> > bug once we have and enforce the definition of what pseudorefs are.\n> \n> Yeah, that makes sense. I'll use \"--include-root-refs\" then.\n\nI'd still argue that this is too specific to the files backend. The fact\nthat we end up only adding root refs to the files backend to me is an\nimplementation-specific limitation and not a feature. What I really want\nis to ask the backend to give me all refs that it knows of for the\ncurrent worktree.\n\nThe \"reftable\" backend has this snippet in its iterator:\n\n```\n/*\n * The files backend only lists references contained in\n * \"refs/\". We emulate the same behaviour here and thus skip\n * all references that don't start with this prefix.\n */\nif (!starts_with(iter->ref.refname, \"refs/\"))\n continue;\n```\n\nIf we add the new `--include-root-refs` flag then we would have to\nextend it to query whether the ref either starts with \"refs/\" or whether\nit might look like a pseudo-ref:\n\n```\nif (!starts_with(iter->ref.refname, \"refs/\") &&\n !(flags & INCLUDE_ROOT_REFS || is_pseudoref(iter->ref.refname)))\n continue;\n```\n\nThe problem I have is that it still wouldn't end up surfacing all refs\nwhich exist in the ref backend while being computationally more\nexpensive. So the original usecase I had in mind when pitching this\ntopic isn't actually addressed.\n\nI know that in theory, the reftable backend shouldn't contain refs other\nthan \"refs/\" or pseudo-refs anyway. But regardless of that, I think that\nformulating this in the form of \"root refs\" is too restrictive and too\nmuch focussed on the \"files\" backend. I wouldn't be surprised if there\nare ways to have git-update-ref(1) or other tools write refs with\nunexpected names, and not being able to learn about those is a\nlimitation.\n\nPutting this in the context of \"Give me all refs which you know of for\nthe current worktree\" is a lot simpler. It would still be equivalent to\n\"--include-root-refs\" for the \"files\" backend, because the files backend\ndoesn't have a way to properly track all refs it has ever created. And\nfor the \"reftable\" backend it's as simple as this:\n\n```\nif (!(iter->flags & DO_FOR_EACH_INCLUDE_ALL_REFS) &&\n !starts_with(iter->ref.refname, \"refs/\"))\n```\n\nI'm not sure about better terminology though. Something like\n`--include-all-worktree-refs` could easily lead to confusion because it\nmight make the user think that they also want to list refs from other\nworktrees. The best I can come up with is `--include-non-refs` -- pseudo\nrefs aren't real refs, and neither are refs which don't start with\n\"refs/\".\n\nPatrick\n"},{"id":"488222","messageId":"613b85a3-677d-40d8-84b5-69dd5c27cafe@gmail.com","threadId":"60768","inReplyTo":"xmqqcyt9fdc7.fsf@gitster.g","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2024-02-08T10:28:52Z","receivedAt":"2024-02-08T10:28:57Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Juino\n\nOn 06/02/2024 22:16, Junio C Hamano wrote:\n> - we do not want to say \"pseudo refs\", as I expect we would want to\n> show HEAD that is (unfortunately) classified outside \"pseudoref\"\n> class.\n\nThis is a bit of a tangent but I've been wondering what the practical \nbenefit of distinguishing between \"HEAD\" and \"pseudoref\" is. I don't \nknow the history so there may be a good reason but not classifying \n\"HEAD\" as a \"pseudoref\" seems to make discussions like this more \ncomplicated than they would be if \"HEAD\" were a \"pseudoref\". Would it be \nbeneficial to loosen the definition of \"psuedoref\" to remove the \nrestriction that they may not be symbolic refs or have a reflog?\n\nBest Wishes\n\nPhillip\n"},{"id":"488238","messageId":"xmqq7cjeq43t.fsf@gitster.g","threadId":"60768","inReplyTo":"ZcSVx4slikt4xB3D@tanuki","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-08T17:04:54Z","receivedAt":"2024-02-08T17:04:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> ```\n> if (!starts_with(iter->ref.refname, \"refs/\") &&\n> !(flags & INCLUDE_ROOT_REFS || is_pseudoref(iter->ref.refname)))\n> continue;\n> ```\n>\n> The problem I have is that it still wouldn't end up surfacing all refs\n> which exist in the ref backend while being computationally more\n> expensive. So the original usecase I had in mind when pitching this\n> topic isn't actually addressed.\n\nThe reftable format, as a database format, may be capable of having\n\"refs/heads/master\" and \"refs/heads/master/1\" at the same time, but\nto be used as a ref backend for Git, it must refrain from surfacing\nboth at the same time. I think it is the same deal that it should\nonly allow \"refs/*\", \"HEAD\", and so called pseudorefs to be stored.\nSo INCLUDE_ROOT_REFS should be sufficient as long as the \"ref\ncreation and update\" side is not letting random cruft (e.g.,\n\"config\") in. Isn't that sufficient?\n\n> I know that in theory, the reftable backend shouldn't contain refs other\n> than \"refs/\" or pseudo-refs anyway. But regardless of that, I think that\n> formulating this in the form of \"root refs\" is too restrictive and too\n> much focussed on the \"files\" backend.\n\nIt is not \"focused on\". The ref namespace of Git is tree-shaped,\nperiod. The shape may have originated from its first ref backend\nimplementation's limitation, but as we gain other backends, we are\nnot planning to lift such limitations, are we? So we may still say\n\"when there is a master branch, you cannot have master/1 branch (due\nto D/F conflict)\", even if there is no notion of directory or file\nin a backend implementation backed by a databasy file format. \"HEAD\"\nand \"CHERRY_PICK_HEAD\", unlike \"refs/tags/v1.0\", are at the \"root\nlevel\", not only when they are stored in a files backend, but always\nwhen they are presented to end-users, who can tell that they are not\ninside \"refs/\".\n\n"},{"id":"488239","messageId":"xmqq34u2q3zs.fsf@gitster.g","threadId":"60768","inReplyTo":"613b85a3-677d-40d8-84b5-69dd5c27cafe@gmail.com","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-08T17:07:19Z","receivedAt":"2024-02-08T17:07:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n> This is a bit of a tangent but I've been wondering what the practical\n> benefit of distinguishing between \"HEAD\" and \"pseudoref\" is.\n\nI hate that distinction too. The practical difference they gave me\nwas that we historically never made FOO_HEAD to be a symref, and we\ndo not want them to be in the future, but for HEAD, it is perfectly\nnatural to be a symref.\n\n"},{"id":"488241","messageId":"ZcUOP_rWUwymhe5c@ncase","threadId":"60768","inReplyTo":"xmqq7cjeq43t.fsf@gitster.g","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-08T17:24:15Z","receivedAt":"2024-02-08T17:24:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Feb 08, 2024 at 09:04:54AM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > ```\n> > if (!starts_with(iter->ref.refname, \"refs/\") &&\n> > !(flags & INCLUDE_ROOT_REFS || is_pseudoref(iter->ref.refname)))\n> > continue;\n> > ```\n> >\n> > The problem I have is that it still wouldn't end up surfacing all refs\n> > which exist in the ref backend while being computationally more\n> > expensive. So the original usecase I had in mind when pitching this\n> > topic isn't actually addressed.\n> \n> The reftable format, as a database format, may be capable of having\n> \"refs/heads/master\" and \"refs/heads/master/1\" at the same time, but\n> to be used as a ref backend for Git, it must refrain from surfacing\n> both at the same time. I think it is the same deal that it should\n> only allow \"refs/*\", \"HEAD\", and so called pseudorefs to be stored.\n> So INCLUDE_ROOT_REFS should be sufficient as long as the \"ref\n> creation and update\" side is not letting random cruft (e.g.,\n> \"config\") in. Isn't that sufficient?\n\nThat's a different problem from the one I have right now. Let's take the\nfollowing sequence of commands:\n\n $ git init repo\n Initialized empty Git repository in /tmp/repo/.git/\n $ git -C repo commit --allow-empty --message message\n [main (root-commit) aa5eec4] message\n $ git -C repo update-ref ref/head/foo HEAD\n $ ls repo/.git/ref/head/foo\n repo/.git/ref/head/foo\n\nNow the fact that you can create \"ref/head/foo\" is a bug that needs to\nbe fixed, no arguing there. The problem is that rectifying this problem\nwith the \"files\" backend is easy -- you look into the repo, notice that\nthere's a weird directory, and then \"rm -rf\" it.\n\nBut how do you learn about this ref existing with the \"reftable\" backend\nin the first place? You can't without looking at the binary format --\nthere doesn't exist a single command that would allow you to list all\nrefs unfiltered. But that is very much required in order to learn about\nmisbehaviour and fix it.\n\nAs I said -- this is a bug, and I agree that it shouldn't happen. But\nbugs happen, and especially with the new reftable format I expect them\nto happen. What I look for in this context is to create the tools to fix\nproblems like this, but `--include-root-refs` doesn't. A flag that\nunconditionally returns all refs, regardless of whether they have a bad\nname or not, does address the issue. Think of it of more of a debugging\ntool.\n\nSpelled out like that it brings me a different idea: maybe I'm just\ntrying to address this in the wrong tool. I plan to introduce ref\nbackend specific fsck checks, so that could be a better place to warn\nabout such refs with bad names. Like this we don't erode the tree-shaped\nnature by somehow accepting them in some tools, and we make clear that\nthis is indeed something that shouldn't happen.\n\n> > I know that in theory, the reftable backend shouldn't contain refs other\n> > than \"refs/\" or pseudo-refs anyway. But regardless of that, I think that\n> > formulating this in the form of \"root refs\" is too restrictive and too\n> > much focussed on the \"files\" backend.\n> \n> It is not \"focused on\". The ref namespace of Git is tree-shaped,\n> period. The shape may have originated from its first ref backend\n> implementation's limitation, but as we gain other backends, we are\n> not planning to lift such limitations, are we? So we may still say\n> \"when there is a master branch, you cannot have master/1 branch (due\n> to D/F conflict)\", even if there is no notion of directory or file\n> in a backend implementation backed by a databasy file format. \"HEAD\"\n> and \"CHERRY_PICK_HEAD\", unlike \"refs/tags/v1.0\", are at the \"root\n> level\", not only when they are stored in a files backend, but always\n> when they are presented to end-users, who can tell that they are not\n> inside \"refs/\".\n\nI agree, and I do not intend to change this.\n\nPatrick\n"},{"id":"488246","messageId":"xmqq34u2onaj.fsf@gitster.g","threadId":"60768","inReplyTo":"ZcUOP_rWUwymhe5c@ncase","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-08T17:53:24Z","receivedAt":"2024-02-08T17:53:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> That's a different problem from the one I have right now. Let's take the\n> following sequence of commands:\n>\n> $ git init repo\n> Initialized empty Git repository in /tmp/repo/.git/\n> $ git -C repo commit --allow-empty --message message\n> [main (root-commit) aa5eec4] message\n> $ git -C repo update-ref ref/head/foo HEAD\n> $ ls repo/.git/ref/head/foo\n> repo/.git/ref/head/foo\n>\n> Now the fact that you can create \"ref/head/foo\" is a bug that needs to\n> be fixed, no arguing there. The problem is that rectifying this problem\n> with the \"files\" backend is easy -- you look into the repo, notice that\n> there's a weird directory, and then \"rm -rf\" it.\n\nOK.\n\n> But how do you learn about this ref existing with the \"reftable\" backend\n> in the first place? You can't without looking at the binary format --\n> there doesn't exist a single command that would allow you to list all\n> refs unfiltered. But that is very much required in order to learn about\n> misbehaviour and fix it.\n\nI think I have been saying that it is perfectly OK if reftable\nbackend, being newer and backed by more experience using Git,\nrejected any attempt to create anything under \"ref/\" (to avoid\nconfusion to those who are reading from sidelines, it should allow\ncreating \"refs/mytool/\" for third-party tools to store their own\npointers).\n\n> As I said -- this is a bug, and I agree that it shouldn't happen. But\n> bugs happen, and especially with the new reftable format I expect them\n> to happen. What I look for in this context is to create the tools to fix\n> problems like this, but `--include-root-refs` doesn't. A flag that\n> unconditionally returns all refs, regardless of whether they have a bad\n> name or not, does address the issue. Think of it of more of a debugging\n> tool.\n\nOK, \"--include-all-refs\" would be fine. And without bugs there\nshould not be a difference.\n\nWhere does \"all refs in this worktree\" you mentioned fit in this\npicture? Should a bogus \"ref/foo/bar\" be considered to be worktree\nspecific, or is it an incorrect attempt to create a ref that is\nspecific to 'foo' worktree that is not the current one and be\nfiltered out?\n\n"},{"id":"488277","messageId":"ZcXddvQzlt6j7T7L@tanuki","threadId":"60768","inReplyTo":"xmqq34u2onaj.fsf@gitster.g","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-09T08:08:22Z","receivedAt":"2024-02-09T08:08:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Feb 08, 2024 at 09:53:24AM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n[snip]\n> > As I said -- this is a bug, and I agree that it shouldn't happen. But\n> > bugs happen, and especially with the new reftable format I expect them\n> > to happen. What I look for in this context is to create the tools to fix\n> > problems like this, but `--include-root-refs` doesn't. A flag that\n> > unconditionally returns all refs, regardless of whether they have a bad\n> > name or not, does address the issue. Think of it of more of a debugging\n> > tool.\n> \n> OK, \"--include-all-refs\" would be fine. And without bugs there\n> should not be a difference.\n> \n> Where does \"all refs in this worktree\" you mentioned fit in this\n> picture? Should a bogus \"ref/foo/bar\" be considered to be worktree\n> specific, or is it an incorrect attempt to create a ref that is\n> specific to 'foo' worktree that is not the current one and be\n> filtered out?\n\nGood question indeed. The reason I said \"all refs in this worktree\" is\nmostly that I don't think we should also be returning refs from other\nworktrees. I consider their refdbs to be separate refdbs, even though it\nis possible to access them via the special \"worktrees/$wt/refs\" syntax.\nSo I would say that such an option should return all refs of the current\nworktree's refdb, plus the common worktree refs.\n\nAnother important question here is how the \"files\" backend should behave\nwith such a flag. Should it try to read all loose files as refs and\nreturn those which just happen to look like one? That feels really wrong\nto me, as we would now have to special case those namespaces where we\nknow that it's not a proper ref, like \"worktrees/\", \"rebase-apply/\" and\nothers.\n\nThe alternative would be to make it behave like `--include-root-refs`,\nwhere we do a best effort and live with the fact that the \"files\"\nbackend cannot enumerate all refs it has created. This would mean that\nbehaviour between the \"files\" and \"reftable\" backend is diverging though\nbecause the latter can trivially figure out all refs it contains. The\nquestion is whether we are fine with that or not.\n\nDepending on the answer, I think we can go one of two ways:\n\n - Accept the diverging behaviour and add `--include-all-refs`. The\n \"files\" backend does a best effort and only includes root refs, the\n \"reftable\" backend lists all refs.\n\n - Double down on the fact that refs must either be pseudo refs or\n start with \"refs/\" and treat any ref that doesn't fit this bill as\n corrupted. The consequence here would be that we instead go with\n `--include-root-refs` that can be implemented the same for both\n backends. In addition, we add checks to git-fsck(1) to surface and\n flag refs with bogus names for the \"reftable\" backend.\n\nWhile I seem to have convinced you that `--include-all-refs` might not\nbe a bad idea after all, you have convinced me by now that the second\noption would be preferable. I'd be okay with either of these options as\nboth of them address the issue at hand.\n\nPatrick\n"},{"id":"488302","messageId":"xmqq8r3timnr.fsf@gitster.g","threadId":"60768","inReplyTo":"ZcXddvQzlt6j7T7L@tanuki","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-09T17:15:52Z","receivedAt":"2024-02-09T17:15:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Depending on the answer, I think we can go one of two ways:\n>\n> - Accept the diverging behaviour and add `--include-all-refs`. The\n> \"files\" backend does a best effort and only includes root refs, the\n> \"reftable\" backend lists all refs.\n>\n> - Double down on the fact that refs must either be pseudo refs or\n> start with \"refs/\" and treat any ref that doesn't fit this bill as\n> corrupted. The consequence here would be that we instead go with\n> `--include-root-refs` that can be implemented the same for both\n> backends. In addition, we add checks to git-fsck(1) to surface and\n> flag refs with bogus names for the \"reftable\" backend.\n>\n> While I seem to have convinced you that `--include-all-refs` might not\n> be a bad idea after all, you have convinced me by now that the second\n> option would be preferable. I'd be okay with either of these options as\n> both of them address the issue at hand.\n\nFor now my tentative preference is the latter. If ref/head/foo is\nan end-user mistake with one backend, it is cleaner if it is\nconsidered a mistake with other backends.\n\nDoesn't our ref enumeration/iteration API have \"include broken ones\nas well\" bit? I wonder if this issue becomes easier to solve by\n(re|ab)using that bit.\n\n"},{"id":"488313","messageId":"CAOLa=ZS3y=K6SCEoC7hZSi7vhAT1-W4fAzPb3rYaBbGcqO5Cyw@mail.gmail.com","threadId":"60768","inReplyTo":"xmqq8r3timnr.fsf@gitster.g","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-09T18:27:39Z","receivedAt":"2024-02-09T18:27:40Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Patrick Steinhardt writes:\n>\n>> Depending on the answer, I think we can go one of two ways:\n>>\n>> - Accept the diverging behaviour and add `--include-all-refs`. The\n>> \"files\" backend does a best effort and only includes root refs, the\n>> \"reftable\" backend lists all refs.\n>>\n>> - Double down on the fact that refs must either be pseudo refs or\n>> start with \"refs/\" and treat any ref that doesn't fit this bill as\n>> corrupted. The consequence here would be that we instead go with\n>> `--include-root-refs` that can be implemented the same for both\n>> backends. In addition, we add checks to git-fsck(1) to surface and\n>> flag refs with bogus names for the \"reftable\" backend.\n>>\n>> While I seem to have convinced you that `--include-all-refs` might not\n>> be a bad idea after all, you have convinced me by now that the second\n>> option would be preferable. I'd be okay with either of these options as\n>> both of them address the issue at hand.\n>\n> For now my tentative preference is the latter. If ref/head/foo is\n> an end-user mistake with one backend, it is cleaner if it is\n> considered a mistake with other backends.\n>\n> Doesn't our ref enumeration/iteration API have \"include broken ones\n> as well\" bit? I wonder if this issue becomes easier to solve by\n> (re|ab)using that bit.\n\nI'll then go ahead with point 2 then.\n\nI'll modify my patch series for now to fit in and will follow up \"checks\nto git-fsck(1) to surface and flag refs with bogus names for the\n\"reftable\" backend\" in a follow up series.\n\n- Karthik\n"},{"id":"488409","messageId":"20240211183923.131278-1-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240119142705.139374-1-karthik.188@gmail.com","subject":"[PATCH v4 0/5] for-each-ref: add '--include-root-refs' option","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-11T18:39:18Z","receivedAt":"2024-02-11T18:39:28Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"This is the forth version of my patch series to print root refs\nin git-for-each-ref(1).\n\nWith the introduction of the reftable backend, it becomes ever\nso important to provide the necessary tooling for printing all refs\nassociated with a worktree.\n\nWhile regular refs stored within the \"refs/\" namespace are currently\nsupported by multiple commands like git-for-each-ref(1),\ngit-show-ref(1). Neither support printing root refs within the worktree.\n\nThis patch series is a follow up to the RFC/discussion we had earlier on\nthe list [1].\n\nThe first 4 commits add the required functionality to ensure we can print\nall refs (regular, pseudo, HEAD). The 5th commit modifies the\ngit-for-each-ref(1) command to add the \"--include-root-refs\" command which\nwill include HEAD and pseudorefs with regular \"refs/\" refs.\n\n[1]: https://lore.kernel.org/git/20231221170715.110565-1-karthik.188@gmail.com/#t\n\nChanges from v3:\n1. Move from using 'git for-each-ref \"\"' to print root refs to adding\nthe '--include-root-refs' option for git-for-each-ref(1). This provides better\nUX for users.\n2. Modify `is_pseudoref()` to use `refs_resolve_ref_unsafe`.\n3. Includes reftable-backend changes and is now rebased on top of next (ed35d3359). \n\nRange-diff:\n\n1: 2141a2a62b ! 1: 98130a7ad7 refs: introduce `is_pseudoref()` and `is_headref()`\n @@ Commit message\n related files like 'BISECT_START' to a new directory similar to the\n 'rebase-merge' directory.\n \n + Helped-by: Jeff King \n Signed-off-by: Karthik Nayak \n \n ## refs.c ##\n @@ refs.c: static int is_pseudoref_syntax(const char *refname)\n +\t\treturn 0;\n +\n +\tif (ends_with(refname, \"_HEAD\")) {\n -+\t\t read_ref_full(refname, RESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n -+\t\t &oid, NULL);\n -+\t\t return !is_null_oid(&oid);\n ++\t\trefs_resolve_ref_unsafe(refs, refname,\n ++ \t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n ++ \t\t\t\t\t&oid, NULL);\n ++ \t\treturn !is_null_oid(&oid);\n +\t}\n +\n +\tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n +\t\tif (!strcmp(refname, irregular_pseudorefs[i])) {\n -+\t\t\tread_ref_full(refname, RESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n -+\t\t\t\t\t\t &oid, NULL);\n ++\t\t\trefs_resolve_ref_unsafe(refs, refname,\n ++ \t\t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n ++ \t\t\t\t\t\t&oid, NULL);\n +\t\t\treturn !is_null_oid(&oid);\n +\t\t}\n +\n2: c96f0a9c83 = 2: 060ab08af5 refs: extract out `loose_fill_ref_dir_regular_file()`\n3: d165358b83 ! 3: 40d2375ad9 refs: introduce `refs_for_each_all_refs()`\n @@ Metadata\n Author: Karthik Nayak \n \n ## Commit message ##\n - refs: introduce `refs_for_each_all_refs()`\n + refs: introduce `refs_for_each_include_root_refs()`\n \n - Introduce a new ref iteration flag `DO_FOR_EACH_INCLUDE_ALL_REFS`, which\n - will be used to iterate over all refs. In the files backend this is\n - limited to regular refs, pseudorefs and HEAD. For other backends like\n - the reftable this is the universal set of all refs stored in the\n - backend.\n + Introduce a new ref iteration flag `DO_FOR_EACH_INCLUDE_ROOT_REFS`,\n + which will be used to iterate over regular refs plus pseudorefs and\n + HEAD.\n \n Refs which fall outside the `refs/` and aren't either pseudorefs or HEAD\n are more of a grey area. This is because we don't block the users from\n - creating such refs but they are not officially supported. In the files\n - backend, we can isolate such files from other files.\n + creating such refs but they are not officially supported.\n \n - Introduce `refs_for_each_all_refs()` which calls `do_for_each_ref()`\n - with this newly introduced flag.\n + Introduce `refs_for_each_include_root_refs()` which calls\n + `do_for_each_ref()` with this newly introduced flag.\n \n In `refs/files-backend.c`, introduce a new function\n `add_pseudoref_and_head_entries()` to add pseudorefs and HEAD to the\n `ref_dir`. We then finally call `add_pseudoref_and_head_entries()`\n - whenever the `DO_FOR_EACH_INCLUDE_ALL_REFS` flag is set. Any new ref\n + whenever the `DO_FOR_EACH_INCLUDE_ROOT_REFS` flag is set. Any new ref\n backend will also have to implement similar changes on its end.\n \n Signed-off-by: Karthik Nayak \n @@ refs.c: int for_each_rawref(each_ref_fn fn, void *cb_data)\n \treturn refs_for_each_rawref(get_main_ref_store(the_repository), fn, cb_data);\n }\n \n -+int refs_for_each_all_refs(struct ref_store *refs, each_ref_fn fn,\n -+\t\t\t void *cb_data)\n ++int refs_for_each_include_root_refs(struct ref_store *refs, each_ref_fn fn,\n ++\t\t\t\t void *cb_data)\n +{\n +\treturn do_for_each_ref(refs, \"\", NULL, fn, 0,\n -+\t\t\t DO_FOR_EACH_INCLUDE_ALL_REFS, cb_data);\n ++\t\t\t DO_FOR_EACH_INCLUDE_ROOT_REFS, cb_data);\n +}\n +\n static int qsort_strcmp(const void *va, const void *vb)\n @@ refs.h: int for_each_namespaced_ref(const char **exclude_patterns,\n int for_each_rawref(each_ref_fn fn, void *cb_data);\n \n +/*\n -+ * Iterates over all ref types, regular, pseudorefs and HEAD.\n ++ * Iterates over all refs including root refs, i.e. pseudorefs and HEAD.\n + */\n -+int refs_for_each_all_refs(struct ref_store *refs, each_ref_fn fn,\n -+\t\t\t void *cb_data);\n ++int refs_for_each_include_root_refs(struct ref_store *refs, each_ref_fn fn,\n ++\t\t\t\t void *cb_data);\n +\n /*\n * Normalizes partial refs to their fully qualified form.\n @@ refs/files-backend.c: static struct ref_cache *get_loose_ref_cache(struct files_\n \n +\t\tdir = get_ref_dir(refs->loose->root);\n +\n -+\t\tif (flags & DO_FOR_EACH_INCLUDE_ALL_REFS)\n ++\t\tif (flags & DO_FOR_EACH_INCLUDE_ROOT_REFS)\n +\t\t\tadd_pseudoref_and_head_entries(dir->cache->ref_store, dir,\n -+\t\t\t\t\t\t\t\t\t\t refs->loose->root->name);\n ++\t\t\t\t\t\t refs->loose->root->name);\n +\n \t\t/*\n \t\t * Add an incomplete entry for \"refs/\" (to be filled\n @@ refs/refs-internal.h: enum do_for_each_ref_flags {\n \tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n +\n +\t/*\n -+\t * Include all refs in the $GIT_DIR in contrast to generally only listing\n -+\t * references having the \"refs/\" prefix. In the files-backend this is\n -+\t * limited to regular refs, pseudorefs and HEAD.\n ++\t * Include root refs i.e. HEAD and pseudorefs along with the regular\n ++\t * refs.\n +\t */\n -+\tDO_FOR_EACH_INCLUDE_ALL_REFS = (1 << 3),\n ++\tDO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),\n };\n \n /*\n4: a17983d0ba < -: ---------- for-each-ref: avoid filtering on empty pattern\n-: ---------- > 4: b4b9435505 ref-filter: rename 'FILTER_REFS_ALL' to 'FILTER_REFS_REGULAR'\n-: ---------- > 5: ee99ac41ae for-each-ref: add new option to include root refs\n\nKarthik Nayak (5):\n refs: introduce `is_pseudoref()` and `is_headref()`\n refs: extract out `loose_fill_ref_dir_regular_file()`\n refs: introduce `refs_for_each_include_root_refs()`\n ref-filter: rename 'FILTER_REFS_ALL' to 'FILTER_REFS_REGULAR'\n for-each-ref: add new option to include root refs\n\n Documentation/git-for-each-ref.txt | 5 +-\n builtin/for-each-ref.c | 11 ++-\n ref-filter.c | 29 ++++++-\n ref-filter.h | 7 +-\n refs.c | 48 +++++++++++\n refs.h | 9 ++\n refs/files-backend.c | 127 +++++++++++++++++++++--------\n refs/refs-internal.h | 6 ++\n refs/reftable-backend.c | 11 ++-\n t/t6302-for-each-ref-filter.sh | 31 +++++++\n 10 files changed, 238 insertions(+), 46 deletions(-)\n\n-- \n2.43.GIT\n\n"},{"id":"488410","messageId":"20240211183923.131278-2-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240211183923.131278-1-karthik.188@gmail.com","subject":"[PATCH v4 1/5] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-11T18:39:19Z","receivedAt":"2024-02-11T18:39:29Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Introduce two new functions `is_pseudoref()` and `is_headref()`. This\nprovides the necessary functionality for us to add pseudorefs and HEAD\nto the loose ref cache in the files backend, allowing us to build\ntooling to print these refs.\n\nThe `is_pseudoref()` function internally calls `is_pseudoref_syntax()`\nbut adds onto it by also checking to ensure that the pseudoref either\nends with a \"_HEAD\" suffix or matches a list of exceptions. After which\nwe also parse the contents of the pseudoref to ensure that it conforms\nto the ref format.\n\nWe cannot directly add the new syntax checks to `is_pseudoref_syntax()`\nbecause the function is also used by `is_current_worktree_ref()` and\nmaking it stricter to match only known pseudorefs might have unintended\nconsequences due to files like 'BISECT_START' which isn't a pseudoref\nbut sometimes contains object ID.\n\nKeeping this in mind, we leave `is_pseudoref_syntax()` as is and create\n`is_pseudoref()` which is stricter. Ideally we'd want to move the new\nsyntax checks to `is_pseudoref_syntax()` but a prerequisite for this\nwould be to actually remove the exception list by converting those\npseudorefs to also contain a '_HEAD' suffix and perhaps move bisect\nrelated files like 'BISECT_START' to a new directory similar to the\n'rebase-merge' directory.\n\nHelped-by: Jeff King \nSigned-off-by: Karthik Nayak \n---\n refs.c | 41 +++++++++++++++++++++++++++++++++++++++++\n refs.h | 3 +++\n 2 files changed, 44 insertions(+)\n\ndiff --git a/refs.c b/refs.c\nindex fff343c256..d8e4cf9a11 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -860,6 +860,47 @@ static int is_pseudoref_syntax(const char *refname)\n \treturn 1;\n }\n \n+int is_pseudoref(struct ref_store *refs, const char *refname)\n+{\n+\tstatic const char *const irregular_pseudorefs[] = {\n+\t\t\"AUTO_MERGE\",\n+\t\t\"BISECT_EXPECTED_REV\",\n+\t\t\"NOTES_MERGE_PARTIAL\",\n+\t\t\"NOTES_MERGE_REF\",\n+\t\t\"MERGE_AUTOSTASH\",\n+\t};\n+\tstruct object_id oid;\n+\tsize_t i;\n+\n+\tif (!is_pseudoref_syntax(refname))\n+\t\treturn 0;\n+\n+\tif (ends_with(refname, \"_HEAD\")) {\n+\t\trefs_resolve_ref_unsafe(refs, refname,\n+ \t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n+ \t\t\t\t\t&oid, NULL);\n+ \t\treturn !is_null_oid(&oid);\n+\t}\n+\n+\tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n+\t\tif (!strcmp(refname, irregular_pseudorefs[i])) {\n+\t\t\trefs_resolve_ref_unsafe(refs, refname,\n+ \t\t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n+ \t\t\t\t\t\t&oid, NULL);\n+\t\t\treturn !is_null_oid(&oid);\n+\t\t}\n+\n+\treturn 0;\n+}\n+\n+int is_headref(struct ref_store *refs, const char *refname)\n+{\n+\tif (!strcmp(refname, \"HEAD\"))\n+\t\treturn refs_ref_exists(refs, refname);\n+\n+\treturn 0;\n+}\n+\n static int is_current_worktree_ref(const char *ref) {\n \treturn is_pseudoref_syntax(ref) || is_per_worktree_ref(ref);\n }\ndiff --git a/refs.h b/refs.h\nindex 303c5fac4d..f66cdd731c 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1023,4 +1023,7 @@ extern struct ref_namespace_info ref_namespace[NAMESPACE__COUNT];\n */\n void update_ref_namespace(enum ref_namespace namespace, char *ref);\n \n+int is_pseudoref(struct ref_store *refs, const char *refname);\n+int is_headref(struct ref_store *refs, const char *refname);\n+\n #endif /* REFS_H */\n-- \n2.43.GIT\n\n"},{"id":"488411","messageId":"20240211183923.131278-3-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240211183923.131278-1-karthik.188@gmail.com","subject":"[PATCH v4 2/5] refs: extract out `loose_fill_ref_dir_regular_file()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-11T18:39:20Z","receivedAt":"2024-02-11T18:39:30Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Extract out the code for adding a single file to the loose ref dir as\n`loose_fill_ref_dir_regular_file()` from `loose_fill_ref_dir()` in\n`refs/files-backend.c`.\n\nThis allows us to use this function independently in the following\ncommits where we add code to also add pseudorefs to the ref dir.\n\nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 62 +++++++++++++++++++++++---------------------\n 1 file changed, 33 insertions(+), 29 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 75dcc21ecb..65128821a8 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -229,6 +229,38 @@ static void add_per_worktree_entries_to_dir(struct ref_dir *dir, const char *dir\n \t}\n }\n \n+static void loose_fill_ref_dir_regular_file(struct files_ref_store *refs,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t struct ref_dir *dir)\n+{\n+\tstruct object_id oid;\n+\tint flag;\n+\n+\tif (!refs_resolve_ref_unsafe(&refs->base, refname, RESOLVE_REF_READING,\n+\t\t\t\t &oid, &flag)) {\n+\t\toidclr(&oid);\n+\t\tflag |= REF_ISBROKEN;\n+\t} else if (is_null_oid(&oid)) {\n+\t\t/*\n+\t\t * It is so astronomically unlikely\n+\t\t * that null_oid is the OID of an\n+\t\t * actual object that we consider its\n+\t\t * appearance in a loose reference\n+\t\t * file to be repo corruption\n+\t\t * (probably due to a software bug).\n+\t\t */\n+\t\tflag |= REF_ISBROKEN;\n+\t}\n+\n+\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tif (!refname_is_safe(refname))\n+\t\t\tdie(\"loose refname is dangerous: %s\", refname);\n+\t\toidclr(&oid);\n+\t\tflag |= REF_BAD_NAME | REF_ISBROKEN;\n+\t}\n+\tadd_entry_to_dir(dir, create_ref_entry(refname, &oid, flag));\n+}\n+\n /*\n * Read the loose references from the namespace dirname into dir\n * (without recursing). dirname must end with '/'. dir must be the\n@@ -257,8 +289,6 @@ static void loose_fill_ref_dir(struct ref_store *ref_store,\n \tstrbuf_add(&refname, dirname, dirnamelen);\n \n \twhile ((de = readdir(d)) != NULL) {\n-\t\tstruct object_id oid;\n-\t\tint flag;\n \t\tunsigned char dtype;\n \n \t\tif (de->d_name[0] == '.')\n@@ -274,33 +304,7 @@ static void loose_fill_ref_dir(struct ref_store *ref_store,\n \t\t\t\t\t create_dir_entry(dir->cache, refname.buf,\n \t\t\t\t\t\t\t refname.len));\n \t\t} else if (dtype == DT_REG) {\n-\t\t\tif (!refs_resolve_ref_unsafe(&refs->base,\n-\t\t\t\t\t\t refname.buf,\n-\t\t\t\t\t\t RESOLVE_REF_READING,\n-\t\t\t\t\t\t &oid, &flag)) {\n-\t\t\t\toidclr(&oid);\n-\t\t\t\tflag |= REF_ISBROKEN;\n-\t\t\t} else if (is_null_oid(&oid)) {\n-\t\t\t\t/*\n-\t\t\t\t * It is so astronomically unlikely\n-\t\t\t\t * that null_oid is the OID of an\n-\t\t\t\t * actual object that we consider its\n-\t\t\t\t * appearance in a loose reference\n-\t\t\t\t * file to be repo corruption\n-\t\t\t\t * (probably due to a software bug).\n-\t\t\t\t */\n-\t\t\t\tflag |= REF_ISBROKEN;\n-\t\t\t}\n-\n-\t\t\tif (check_refname_format(refname.buf,\n-\t\t\t\t\t\t REFNAME_ALLOW_ONELEVEL)) {\n-\t\t\t\tif (!refname_is_safe(refname.buf))\n-\t\t\t\t\tdie(\"loose refname is dangerous: %s\", refname.buf);\n-\t\t\t\toidclr(&oid);\n-\t\t\t\tflag |= REF_BAD_NAME | REF_ISBROKEN;\n-\t\t\t}\n-\t\t\tadd_entry_to_dir(dir,\n-\t\t\t\t\t create_ref_entry(refname.buf, &oid, flag));\n+\t\t\tloose_fill_ref_dir_regular_file(refs, refname.buf, dir);\n \t\t}\n \t\tstrbuf_setlen(&refname, dirnamelen);\n \t}\n-- \n2.43.GIT\n\n"},{"id":"488412","messageId":"20240211183923.131278-4-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240211183923.131278-1-karthik.188@gmail.com","subject":"[PATCH v4 3/5] refs: introduce `refs_for_each_include_root_refs()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-11T18:39:21Z","receivedAt":"2024-02-11T18:39:31Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Introduce a new ref iteration flag `DO_FOR_EACH_INCLUDE_ROOT_REFS`,\nwhich will be used to iterate over regular refs plus pseudorefs and\nHEAD.\n\nRefs which fall outside the `refs/` and aren't either pseudorefs or HEAD\nare more of a grey area. This is because we don't block the users from\ncreating such refs but they are not officially supported.\n\nIntroduce `refs_for_each_include_root_refs()` which calls\n`do_for_each_ref()` with this newly introduced flag.\n\nIn `refs/files-backend.c`, introduce a new function\n`add_pseudoref_and_head_entries()` to add pseudorefs and HEAD to the\n`ref_dir`. We then finally call `add_pseudoref_and_head_entries()`\nwhenever the `DO_FOR_EACH_INCLUDE_ROOT_REFS` flag is set. Any new ref\nbackend will also have to implement similar changes on its end.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 7 +++++\n refs.h | 6 ++++\n refs/files-backend.c | 65 ++++++++++++++++++++++++++++++++++++++++----\n refs/refs-internal.h | 6 ++++\n 4 files changed, 79 insertions(+), 5 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex d8e4cf9a11..77f4c1e4c2 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1765,6 +1765,13 @@ int for_each_rawref(each_ref_fn fn, void *cb_data)\n \treturn refs_for_each_rawref(get_main_ref_store(the_repository), fn, cb_data);\n }\n \n+int refs_for_each_include_root_refs(struct ref_store *refs, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n+{\n+\treturn do_for_each_ref(refs, \"\", NULL, fn, 0,\n+\t\t\t DO_FOR_EACH_INCLUDE_ROOT_REFS, cb_data);\n+}\n+\n static int qsort_strcmp(const void *va, const void *vb)\n {\n \tconst char *a = *(const char **)va;\ndiff --git a/refs.h b/refs.h\nindex f66cdd731c..5cfaee6229 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -398,6 +398,12 @@ int for_each_namespaced_ref(const char **exclude_patterns,\n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data);\n int for_each_rawref(each_ref_fn fn, void *cb_data);\n \n+/*\n+ * Iterates over all refs including root refs, i.e. pseudorefs and HEAD.\n+ */\n+int refs_for_each_include_root_refs(struct ref_store *refs, each_ref_fn fn,\n+\t\t\t\t void *cb_data);\n+\n /*\n * Normalizes partial refs to their fully qualified form.\n * Will prepend to the if it doesn't start with 'refs/'.\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 65128821a8..9c1c42fe52 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -315,9 +315,59 @@ static void loose_fill_ref_dir(struct ref_store *ref_store,\n \tadd_per_worktree_entries_to_dir(dir, dirname);\n }\n \n-static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs)\n+/*\n+ * Add pseudorefs to the ref dir by parsing the directory for any files\n+ * which follow the pseudoref syntax.\n+ */\n+static void add_pseudoref_and_head_entries(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_dir *dir,\n+\t\t\t\t\t const char *dirname)\n+{\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fill_ref_dir\");\n+\tstruct strbuf path = STRBUF_INIT, refname = STRBUF_INIT;\n+\tstruct dirent *de;\n+\tsize_t dirnamelen;\n+\tDIR *d;\n+\n+\tfiles_ref_path(refs, &path, dirname);\n+\n+\td = opendir(path.buf);\n+\tif (!d) {\n+\t\tstrbuf_release(&path);\n+\t\treturn;\n+\t}\n+\n+\tstrbuf_addstr(&refname, dirname);\n+\tdirnamelen = refname.len;\n+\n+\twhile ((de = readdir(d)) != NULL) {\n+\t\tunsigned char dtype;\n+\n+\t\tif (de->d_name[0] == '.')\n+\t\t\tcontinue;\n+\t\tif (ends_with(de->d_name, \".lock\"))\n+\t\t\tcontinue;\n+\t\tstrbuf_addstr(&refname, de->d_name);\n+\n+\t\tdtype = get_dtype(de, &path, 1);\n+\t\tif (dtype == DT_REG && (is_pseudoref(ref_store, de->d_name) ||\n+\t\t\t\t\t\t\t\tis_headref(ref_store, de->d_name)))\n+\t\t\tloose_fill_ref_dir_regular_file(refs, refname.buf, dir);\n+\n+\t\tstrbuf_setlen(&refname, dirnamelen);\n+\t}\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&path);\n+\tclosedir(d);\n+}\n+\n+static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs,\n+\t\t\t\t\t unsigned int flags)\n {\n \tif (!refs->loose) {\n+\t\tstruct ref_dir *dir;\n+\n \t\t/*\n \t\t * Mark the top-level directory complete because we\n \t\t * are about to read the only subdirectory that can\n@@ -328,12 +378,17 @@ static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs)\n \t\t/* We're going to fill the top level ourselves: */\n \t\trefs->loose->root->flag &= ~REF_INCOMPLETE;\n \n+\t\tdir = get_ref_dir(refs->loose->root);\n+\n+\t\tif (flags & DO_FOR_EACH_INCLUDE_ROOT_REFS)\n+\t\t\tadd_pseudoref_and_head_entries(dir->cache->ref_store, dir,\n+\t\t\t\t\t\t refs->loose->root->name);\n+\n \t\t/*\n \t\t * Add an incomplete entry for \"refs/\" (to be filled\n \t\t * lazily):\n \t\t */\n-\t\tadd_entry_to_dir(get_ref_dir(refs->loose->root),\n-\t\t\t\t create_dir_entry(refs->loose, \"refs/\", 5));\n+\t\tadd_entry_to_dir(dir, create_dir_entry(refs->loose, \"refs/\", 5));\n \t}\n \treturn refs->loose;\n }\n@@ -861,7 +916,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \t * disk, and re-reads it if not.\n \t */\n \n-\tloose_iter = cache_ref_iterator_begin(get_loose_ref_cache(refs),\n+\tloose_iter = cache_ref_iterator_begin(get_loose_ref_cache(refs, flags),\n \t\t\t\t\t prefix, ref_store->repo, 1);\n \n \t/*\n@@ -1222,7 +1277,7 @@ static int files_pack_refs(struct ref_store *ref_store,\n \n \tpacked_refs_lock(refs->packed_ref_store, LOCK_DIE_ON_ERROR, &err);\n \n-\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs), NULL,\n+\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs, 0), NULL,\n \t\t\t\t\tthe_repository, 0);\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t/*\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 83e0f0bba3..73a8fa18ad 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -260,6 +260,12 @@ enum do_for_each_ref_flags {\n \t * INCLUDE_BROKEN, since they are otherwise not included at all.\n \t */\n \tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n+\n+\t/*\n+\t * Include root refs i.e. HEAD and pseudorefs along with the regular\n+\t * refs.\n+\t */\n+\tDO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),\n };\n \n /*\n-- \n2.43.GIT\n\n"},{"id":"488413","messageId":"20240211183923.131278-5-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240211183923.131278-1-karthik.188@gmail.com","subject":"[PATCH v4 4/5] ref-filter: rename 'FILTER_REFS_ALL' to 'FILTER_REFS_REGULAR'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-11T18:39:22Z","receivedAt":"2024-02-11T18:39:32Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The flag 'FILTER_REFS_ALL' is a bit ambiguous, where ALL doesn't specify\nif it means to contain refs from all worktrees or whether all types of\nrefs (regular, HEAD & pseudorefs) or all of the above.\n\nSince here it is actually referring to all refs with the \"refs/\" prefix,\nlet's rename it to 'FILTER_REFS_REGULAR' to indicate that this is\nspecifically for regular refs.\n\nSigned-off-by: Karthik Nayak \n---\n builtin/for-each-ref.c | 2 +-\n ref-filter.c | 2 +-\n ref-filter.h | 4 ++--\n 3 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 3885a9c28e..23d352e371 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -97,7 +97,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t}\n \n \tfilter.match_as_path = 1;\n-\tfilter_and_format_refs(&filter, FILTER_REFS_ALL, sorting, &format);\n+\tfilter_and_format_refs(&filter, FILTER_REFS_REGULAR, sorting, &format);\n \n \tref_filter_clear(&filter);\n \tref_sorting_release(sorting);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex be14b56e32..acb960e35c 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -3047,7 +3047,7 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \t\t\tret = for_each_fullref_in(\"refs/remotes/\", fn, cb_data);\n \t\telse if (filter->kind == FILTER_REFS_TAGS)\n \t\t\tret = for_each_fullref_in(\"refs/tags/\", fn, cb_data);\n-\t\telse if (filter->kind & FILTER_REFS_ALL)\n+\t\telse if (filter->kind & FILTER_REFS_REGULAR)\n \t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n \t\tif (!ret && (filter->kind & FILTER_REFS_DETACHED_HEAD))\n \t\t\thead_ref(fn, cb_data);\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 07cd6f6da3..5416936800 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -19,10 +19,10 @@\n #define FILTER_REFS_BRANCHES 0x0004\n #define FILTER_REFS_REMOTES 0x0008\n #define FILTER_REFS_OTHERS 0x0010\n-#define FILTER_REFS_ALL (FILTER_REFS_TAGS | FILTER_REFS_BRANCHES | \\\n+#define FILTER_REFS_REGULAR (FILTER_REFS_TAGS | FILTER_REFS_BRANCHES | \\\n \t\t\t\t FILTER_REFS_REMOTES | FILTER_REFS_OTHERS)\n #define FILTER_REFS_DETACHED_HEAD 0x0020\n-#define FILTER_REFS_KIND_MASK (FILTER_REFS_ALL | FILTER_REFS_DETACHED_HEAD)\n+#define FILTER_REFS_KIND_MASK (FILTER_REFS_REGULAR | FILTER_REFS_DETACHED_HEAD)\n \n struct atom_value;\n struct ref_sorting;\n-- \n2.43.GIT\n\n"},{"id":"488414","messageId":"20240211183923.131278-6-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240211183923.131278-1-karthik.188@gmail.com","subject":"[PATCH v4 5/5] for-each-ref: add new option to include root refs","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-11T18:39:23Z","receivedAt":"2024-02-11T18:39:33Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The git-for-each-ref(1) command doesn't provide a way to print root refs\ni.e pseudorefs and HEAD with the regular \"refs/\" prefixed refs.\n\nThis commit adds a new option \"--include-root-refs\" to\ngit-for-each-ref(1). When used this would also print pseudorefs and HEAD\nfor the current worktree.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-for-each-ref.txt | 5 ++++-\n builtin/for-each-ref.c | 11 ++++++++---\n ref-filter.c | 27 +++++++++++++++++++++++++-\n ref-filter.h | 5 ++++-\n refs/reftable-backend.c | 11 +++++++----\n t/t6302-for-each-ref-filter.sh | 31 ++++++++++++++++++++++++++++++\n 6 files changed, 80 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 3a9ad91b7a..c1dd12b93c 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n [verse]\n 'git for-each-ref' [--count=] [--shell|--perl|--python|--tcl]\n \t\t [(--sort=)...] [--format=]\n-\t\t [ --stdin | ... ]\n+\t\t [--include-root-refs] [ --stdin | ... ]\n \t\t [--points-at=]\n \t\t [--merged[=]] [--no-merged[=]]\n \t\t [--contains[=]] [--no-contains[=]]\n@@ -105,6 +105,9 @@ TAB %(refname)`.\n \tany excluded pattern(s) are shown. Matching is done using the\n \tsame rules as `` above.\n \n+--include-root-refs::\n+\tList root refs (HEAD and pseudorefs) apart from regular refs.\n+\n FIELD NAMES\n -----------\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 23d352e371..9ed146dad3 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -20,10 +20,10 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n {\n \tstruct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n-\tint icase = 0;\n+\tint icase = 0, include_root_refs = 0, from_stdin = 0;\n \tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_format format = REF_FORMAT_INIT;\n-\tint from_stdin = 0;\n+\tunsigned int flags = FILTER_REFS_REGULAR;\n \tstruct strvec vec = STRVEC_INIT;\n \n \tstruct option opts[] = {\n@@ -53,6 +53,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\tOPT_NO_CONTAINS(&filter.no_commit, N_(\"print only refs which don't contain the commit\")),\n \t\tOPT_BOOL(0, \"ignore-case\", &icase, N_(\"sorting and filtering are case insensitive\")),\n \t\tOPT_BOOL(0, \"stdin\", &from_stdin, N_(\"read reference patterns from stdin\")),\n+\t\tOPT_BOOL(0, \"include-root-refs\", &include_root_refs, N_(\"also include HEAD ref and pseudorefs\")),\n \t\tOPT_END(),\n \t};\n \n@@ -96,8 +97,12 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\tfilter.name_patterns = argv;\n \t}\n \n+\tif (include_root_refs) {\n+\t\tflags |= FILTER_REFS_ROOT_REFS;\n+\t}\n+\n \tfilter.match_as_path = 1;\n-\tfilter_and_format_refs(&filter, FILTER_REFS_REGULAR, sorting, &format);\n+\tfilter_and_format_refs(&filter, flags, sorting, &format);\n \n \tref_filter_clear(&filter);\n \tref_sorting_release(sorting);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex acb960e35c..0e83e29390 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2628,6 +2628,12 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t\t\t each_ref_fn cb,\n \t\t\t\t void *cb_data)\n {\n+\tif (filter->kind == FILTER_REFS_KIND_MASK) {\n+\t\t/* in this case, we want to print all refs including root refs. */\n+\t\treturn refs_for_each_include_root_refs(get_main_ref_store(the_repository),\n+\t\t\t\t\t\t cb, cb_data);\n+\t}\n+\n \tif (!filter->match_as_path) {\n \t\t/*\n \t\t * in this case, the patterns are applied after\n@@ -2750,6 +2756,9 @@ static int ref_kind_from_refname(const char *refname)\n \t\t\treturn ref_kind[i].kind;\n \t}\n \n+\tif (is_pseudoref(get_main_ref_store(the_repository), refname))\n+\t\treturn FILTER_REFS_PSEUDOREFS;\n+\n \treturn FILTER_REFS_OTHERS;\n }\n \n@@ -2781,6 +2790,16 @@ static struct ref_array_item *apply_ref_filter(const char *refname, const struct\n \n \t/* Obtain the current ref kind from filter_ref_kind() and ignore unwanted refs. */\n \tkind = filter_ref_kind(filter, refname);\n+\n+\t/*\n+\t * When printing HEAD with all other refs, we want to apply the same formatting\n+\t * rules as the other refs, so we simply ask it to be treated as a pseudoref.\n+\t */\n+\tif (filter->kind == FILTER_REFS_KIND_MASK && kind == FILTER_REFS_DETACHED_HEAD)\n+\t\tkind = FILTER_REFS_PSEUDOREFS;\n+\telse if (!(kind & filter->kind))\n+\t\treturn NULL;\n+\n \tif (!(kind & filter->kind))\n \t\treturn NULL;\n \n@@ -3049,7 +3068,13 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \t\t\tret = for_each_fullref_in(\"refs/tags/\", fn, cb_data);\n \t\telse if (filter->kind & FILTER_REFS_REGULAR)\n \t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n-\t\tif (!ret && (filter->kind & FILTER_REFS_DETACHED_HEAD))\n+\n+\t\t/*\n+\t\t * When printing all ref types, HEAD is already included,\n+\t\t * so we don't want to print HEAD again.\n+\t\t */\n+\t\tif (!ret && (filter->kind != FILTER_REFS_KIND_MASK) &&\n+\t\t (filter->kind & FILTER_REFS_DETACHED_HEAD))\n \t\t\thead_ref(fn, cb_data);\n \t}\n \ndiff --git a/ref-filter.h b/ref-filter.h\nindex 5416936800..0ca28d2bba 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -22,7 +22,10 @@\n #define FILTER_REFS_REGULAR (FILTER_REFS_TAGS | FILTER_REFS_BRANCHES | \\\n \t\t\t\t FILTER_REFS_REMOTES | FILTER_REFS_OTHERS)\n #define FILTER_REFS_DETACHED_HEAD 0x0020\n-#define FILTER_REFS_KIND_MASK (FILTER_REFS_REGULAR | FILTER_REFS_DETACHED_HEAD)\n+#define FILTER_REFS_PSEUDOREFS 0x0040\n+#define FILTER_REFS_ROOT_REFS (FILTER_REFS_DETACHED_HEAD | FILTER_REFS_PSEUDOREFS)\n+#define FILTER_REFS_KIND_MASK (FILTER_REFS_REGULAR | FILTER_REFS_DETACHED_HEAD | \\\n+\t\t\t\t FILTER_REFS_PSEUDOREFS)\n \n struct atom_value;\n struct ref_sorting;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex a14f2ad7f4..c23a516ac2 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -364,12 +364,15 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\tbreak;\n \n \t\t/*\n-\t\t * The files backend only lists references contained in\n-\t\t * \"refs/\". We emulate the same behaviour here and thus skip\n-\t\t * all references that don't start with this prefix.\n+\t\t * The files backend only lists references contained in \"refs/\" unless\n+\t\t * the root refs are to be included. We emulate the same behaviour here.\n \t\t */\n-\t\tif (!starts_with(iter->ref.refname, \"refs/\"))\n+\t\tif (!starts_with(iter->ref.refname, \"refs/\") &&\n+\t\t !(iter->flags & DO_FOR_EACH_INCLUDE_ROOT_REFS &&\n+\t\t (is_pseudoref(&iter->refs->base, iter->ref.refname) ||\n+\t\t is_headref(&iter->refs->base, iter->ref.refname)))) {\n \t\t\tcontinue;\n+\t\t}\n \n \t\tif (iter->prefix &&\n \t\t strncmp(iter->prefix, iter->ref.refname, strlen(iter->prefix))) {\ndiff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh\nindex 82f3d1ea0f..948f1bb5f4 100755\n--- a/t/t6302-for-each-ref-filter.sh\n+++ b/t/t6302-for-each-ref-filter.sh\n@@ -31,6 +31,37 @@ test_expect_success 'setup some history and refs' '\n \tgit update-ref refs/odd/spot main\n '\n \n+test_expect_success '--include-root-refs pattern prints pseudorefs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD\n+\tORIG_HEAD\n+\trefs/heads/main\n+\trefs/heads/side\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit update-ref ORIG_HEAD main &&\n+\tgit for-each-ref --format=\"%(refname)\" --include-root-refs >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--include-root-refs with other patterns' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD\n+\tORIG_HEAD\n+\tEOF\n+\tgit update-ref ORIG_HEAD main &&\n+\tgit for-each-ref --format=\"%(refname)\" --include-root-refs \"*HEAD\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'filtering with --points-at' '\n \tcat >expect <<-\\EOF &&\n \trefs/heads/main\n-- \n2.43.GIT\n\n"},{"id":"488427","messageId":"Zcm__DROE3hKsfRX@tanuki","threadId":"60768","inReplyTo":"CAOLa=ZS3y=K6SCEoC7hZSi7vhAT1-W4fAzPb3rYaBbGcqO5Cyw@mail.gmail.com","subject":"Re: [PATCH v3 4/4] for-each-ref: avoid filtering on empty pattern","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-12T06:51:40Z","receivedAt":"2024-02-12T06:51:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Feb 09, 2024 at 06:27:39PM +0000, Karthik Nayak wrote:\n> Junio C Hamano writes:\n> > Patrick Steinhardt writes:\n> >> Depending on the answer, I think we can go one of two ways:\n> >>\n> >> - Accept the diverging behaviour and add `--include-all-refs`. The\n> >> \"files\" backend does a best effort and only includes root refs, the\n> >> \"reftable\" backend lists all refs.\n> >>\n> >> - Double down on the fact that refs must either be pseudo refs or\n> >> start with \"refs/\" and treat any ref that doesn't fit this bill as\n> >> corrupted. The consequence here would be that we instead go with\n> >> `--include-root-refs` that can be implemented the same for both\n> >> backends. In addition, we add checks to git-fsck(1) to surface and\n> >> flag refs with bogus names for the \"reftable\" backend.\n> >>\n> >> While I seem to have convinced you that `--include-all-refs` might not\n> >> be a bad idea after all, you have convinced me by now that the second\n> >> option would be preferable. I'd be okay with either of these options as\n> >> both of them address the issue at hand.\n> >\n> > For now my tentative preference is the latter. If ref/head/foo is\n> > an end-user mistake with one backend, it is cleaner if it is\n> > considered a mistake with other backends.\n> >\n> > Doesn't our ref enumeration/iteration API have \"include broken ones\n> > as well\" bit? I wonder if this issue becomes easier to solve by\n> > (re|ab)using that bit.\n> \n> I'll then go ahead with point 2 then.\n> \n> I'll modify my patch series for now to fit in and will follow up \"checks\n> to git-fsck(1) to surface and flag refs with bogus names for the\n> \"reftable\" backend\" in a follow up series.\n\nThanks. Note that the fsck checks are also proposed as one of the GSoC\nprojects where you're listed as a mentor. Might be worth it to hold back\nuntil we know whether any student wants to work on it.\n\nPatrick\n"},{"id":"488442","messageId":"ZcoTbRxIaGmTd4fJ@tanuki","threadId":"60768","inReplyTo":"20240211183923.131278-2-karthik.188@gmail.com","subject":"Re: [PATCH v4 1/5] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-12T12:47:41Z","receivedAt":"2024-02-12T12:47:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sun, Feb 11, 2024 at 07:39:19PM +0100, Karthik Nayak wrote:\n> Introduce two new functions `is_pseudoref()` and `is_headref()`. This\n> provides the necessary functionality for us to add pseudorefs and HEAD\n> to the loose ref cache in the files backend, allowing us to build\n> tooling to print these refs.\n> \n> The `is_pseudoref()` function internally calls `is_pseudoref_syntax()`\n> but adds onto it by also checking to ensure that the pseudoref either\n> ends with a \"_HEAD\" suffix or matches a list of exceptions. After which\n> we also parse the contents of the pseudoref to ensure that it conforms\n> to the ref format.\n> \n> We cannot directly add the new syntax checks to `is_pseudoref_syntax()`\n> because the function is also used by `is_current_worktree_ref()` and\n> making it stricter to match only known pseudorefs might have unintended\n> consequences due to files like 'BISECT_START' which isn't a pseudoref\n> but sometimes contains object ID.\n> \n> Keeping this in mind, we leave `is_pseudoref_syntax()` as is and create\n> `is_pseudoref()` which is stricter. Ideally we'd want to move the new\n> syntax checks to `is_pseudoref_syntax()` but a prerequisite for this\n> would be to actually remove the exception list by converting those\n> pseudorefs to also contain a '_HEAD' suffix and perhaps move bisect\n> related files like 'BISECT_START' to a new directory similar to the\n> 'rebase-merge' directory.\n> \n> Helped-by: Jeff King \n> Signed-off-by: Karthik Nayak \n> ---\n> refs.c | 41 +++++++++++++++++++++++++++++++++++++++++\n> refs.h | 3 +++\n> 2 files changed, 44 insertions(+)\n> \n> diff --git a/refs.c b/refs.c\n> index fff343c256..d8e4cf9a11 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -860,6 +860,47 @@ static int is_pseudoref_syntax(const char *refname)\n> \treturn 1;\n> }\n> \n> +int is_pseudoref(struct ref_store *refs, const char *refname)\n> +{\n> +\tstatic const char *const irregular_pseudorefs[] = {\n> +\t\t\"AUTO_MERGE\",\n> +\t\t\"BISECT_EXPECTED_REV\",\n> +\t\t\"NOTES_MERGE_PARTIAL\",\n> +\t\t\"NOTES_MERGE_REF\",\n> +\t\t\"MERGE_AUTOSTASH\",\n> +\t};\n> +\tstruct object_id oid;\n> +\tsize_t i;\n> +\n> +\tif (!is_pseudoref_syntax(refname))\n> +\t\treturn 0;\n> +\n> +\tif (ends_with(refname, \"_HEAD\")) {\n> +\t\trefs_resolve_ref_unsafe(refs, refname,\n> + \t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n> + \t\t\t\t\t&oid, NULL);\n> + \t\treturn !is_null_oid(&oid);\n> +\t}\n\nI think it's quite confusing that `is_pseudoref()` not only checks\nwhether the refname may be a pseudoref, but also whether it actually\nexists. Furthermore, why is a pseudoref only considered to exist in case\nit's not a symbolic ref? That sounds overly restrictive to me.\n\nSo I think this at least needs to be renamed. But I find it really hard\nto come up with a proper name here because in my opinion the function\ndoes too much. `is_existing_pseudoref()` feels much too specific to me.\nAlso, the \"reftable\" backend wouldn't need to check whether the ref\nexists, but only whether a name that it encounters is a pseudoref name\nor not.\n\n> +\tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n> +\t\tif (!strcmp(refname, irregular_pseudorefs[i])) {\n> +\t\t\trefs_resolve_ref_unsafe(refs, refname,\n> + \t\t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n> + \t\t\t\t\t\t&oid, NULL);\n> +\t\t\treturn !is_null_oid(&oid);\n> +\t\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> +int is_headref(struct ref_store *refs, const char *refname)\n> +{\n> +\tif (!strcmp(refname, \"HEAD\"))\n> +\t\treturn refs_ref_exists(refs, refname);\n> +\n> +\treturn 0;\n> +}\n\nThe same comment applies here, as well.\n\nI also worry a bit about the API we have. It becomes really hard to\nfigure out which function to call now as the API surface seems to\nexplode. We have:\n\n - is_pseudoref_syntax\n - is_pseudoref\n - is_headref\n - check_refname_format\n - refname_is_safe\n\nI wonder whether we can maybe consolidate the interface into one or\nmaybe even two functions where the behaviour can be tweaked with a flag\nfield. Something like `refname_is_valid()` with a bunch of flags:\n\n - REFNAME_ACCEPT_HEAD to accept \"HEAD\"\n - REFNAME_ACCEPT_PSEUDOREF to accept all of the refs ending with\n \"_HEAD\" or being one of the irregular pseudorefs.\n - REFNAME_ACCEPT_INVALID_BUT_SAFE to accept refnames which aren't\n valid, but which would pass `refname_is_safe()`.\n\nAnother alternative could be something like `classify_refname()` that\naccepts a refname and returns an enum saying what kind of ref something\nis.\n\nGiven that this topic won't be included in Git v2.44 anymore, I think\nthat opening this can of worms would be sensible now.\n\nPatrick\n\n> static int is_current_worktree_ref(const char *ref) {\n> \treturn is_pseudoref_syntax(ref) || is_per_worktree_ref(ref);\n> }\n> diff --git a/refs.h b/refs.h\n> index 303c5fac4d..f66cdd731c 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -1023,4 +1023,7 @@ extern struct ref_namespace_info ref_namespace[NAMESPACE__COUNT];\n> */\n> void update_ref_namespace(enum ref_namespace namespace, char *ref);\n> \n> +int is_pseudoref(struct ref_store *refs, const char *refname);\n> +int is_headref(struct ref_store *refs, const char *refname);\n> +\n> #endif /* REFS_H */\n> -- \n> 2.43.GIT\n> \n"},{"id":"488455","messageId":"xmqqeddh7h2l.fsf@gitster.g","threadId":"60768","inReplyTo":"ZcoTbRxIaGmTd4fJ@tanuki","subject":"Re: [PATCH v4 1/5] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-12T17:01:06Z","receivedAt":"2024-02-12T17:01:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> I think it's quite confusing that `is_pseudoref()` not only checks\n> whether the refname may be a pseudoref, but also whether it actually\n> exists. Furthermore, why is a pseudoref only considered to exist in case\n> it's not a symbolic ref? That sounds overly restrictive to me.\n\nI am torn on this, but in favor of the proposed naming, primarily\nbecause is_pseudoref_syntax() was about \"does this string look like\nthe fullref a pseudoref would have?\", and the reason why we wanted\nto have this new function was we wanted to ask \"does this string\nname a valid pseudoref?\"\n\n Q: Is CHERRY_PICK_HEAD a pseudoref?\n A: It would have been if it existed, but I see only\n $GIT_DIR/CHERRY_PICK_HEAD that is a symbolic link, and it cannot\n be a pseudoref.\n\nI can certainly see a broken out set of helper functions to check\n\n - Does this string make a good fullref for a pseudoref?\n - Does a pseudoref with his string as its fullref exist?\n\nindependently. The first one would answer Yes and the second one\nwould answer No in such a context.\n\nThanks.\n\n"},{"id":"488460","messageId":"xmqqwmr95zig.fsf@gitster.g","threadId":"60768","inReplyTo":"20240211183923.131278-2-karthik.188@gmail.com","subject":"Re: [PATCH v4 1/5] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-12T18:05:43Z","receivedAt":"2024-02-12T18:05:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> +\tif (ends_with(refname, \"_HEAD\")) {\n> +\t\trefs_resolve_ref_unsafe(refs, refname,\n> + \t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n> + \t\t\t\t\t&oid, NULL);\n> + \t\treturn !is_null_oid(&oid);\n> +\t}\n\nFYI. I see\n\n.git/rebase-apply/patch:31: space before tab in indent.\n \t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n.git/rebase-apply/patch:32: space before tab in indent.\n \t\t\t\t\t&oid, NULL);\n.git/rebase-apply/patch:33: space before tab in indent.\n \t\treturn !is_null_oid(&oid);\n\naround here.\n"},{"id":"488531","messageId":"CAOLa=ZSQMqhBbOUtiqRbS+ftmB=C8wn8N7pcsC0izVXLsSCQsw@mail.gmail.com","threadId":"60768","inReplyTo":"ZcoTbRxIaGmTd4fJ@tanuki","subject":"Re: [PATCH v4 1/5] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-13T15:48:20Z","receivedAt":"2024-02-13T15:48:22Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello,\n\nPatrick Steinhardt writes:\n>> +\n>> +int is_headref(struct ref_store *refs, const char *refname)\n>> +{\n>> +\tif (!strcmp(refname, \"HEAD\"))\n>> +\t\treturn refs_ref_exists(refs, refname);\n>> +\n>> +\treturn 0;\n>> +}\n>\n> The same comment applies here, as well.\n>\n> I also worry a bit about the API we have. It becomes really hard to\n> figure out which function to call now as the API surface seems to\n> explode. We have:\n>\n> - is_pseudoref_syntax\n> - is_pseudoref\n> - is_headref\n> - check_refname_format\n> - refname_is_safe\n>\n\nI also found `is_head()` in 'reflog.c'.\n\n> I wonder whether we can maybe consolidate the interface into one or\n> maybe even two functions where the behaviour can be tweaked with a flag\n> field.\n>\n\nYou do bring up an interesting point about the APIs present and I agree\nthat it would be best to consolidate them to something much simpler and\nnicer.\n\n> Something like `refname_is_valid()` with a bunch of flags:\n>\n> - REFNAME_ACCEPT_HEAD to accept \"HEAD\"\n> - REFNAME_ACCEPT_PSEUDOREF to accept all of the refs ending with\n> \"_HEAD\" or being one of the irregular pseudorefs.\n> - REFNAME_ACCEPT_INVALID_BUT_SAFE to accept refnames which aren't\n> valid, but which would pass `refname_is_safe()`.\n>\n> Another alternative could be something like `classify_refname()` that\n> accepts a refname and returns an enum saying what kind of ref something\n> is.\n>\n> Given that this topic won't be included in Git v2.44 anymore, I think\n> that opening this can of worms would be sensible now.\n>\n\nOver the two I do prefer something like the former method of the callee\nusing flags to state the requirements, this way the function only does\nwhat is necessary between the listed operations.\n\nJunio C Hamano writes:\n> Patrick Steinhardt writes:\n>\n>> I think it's quite confusing that `is_pseudoref()` not only checks\n>> whether the refname may be a pseudoref, but also whether it actually\n>> exists. Furthermore, why is a pseudoref only considered to exist in case\n>> it's not a symbolic ref? That sounds overly restrictive to me.\n>\n> I am torn on this, but in favor of the proposed naming, primarily\n> because is_pseudoref_syntax() was about \"does this string look like\n> the fullref a pseudoref would have?\", and the reason why we wanted\n> to have this new function was we wanted to ask \"does this string\n> name a valid pseudoref?\"\n>\n> Q: Is CHERRY_PICK_HEAD a pseudoref?\n> A: It would have been if it existed, but I see only\n> $GIT_DIR/CHERRY_PICK_HEAD that is a symbolic link, and it cannot\n> be a pseudoref.\n>\n> I can certainly see a broken out set of helper functions to check\n>\n> - Does this string make a good fullref for a pseudoref?\n> - Does a pseudoref with his string as its fullref exist?\n>\n> independently. The first one would answer Yes and the second one\n> would answer No in such a context.\n>\n\nThis does work into the flags based mechanism that Patrick mentioned\ntoo. The users of this new function can then only request information as\nnecessary.\n\nJunio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> +\tif (ends_with(refname, \"_HEAD\")) {\n>> +\t\trefs_resolve_ref_unsafe(refs, refname,\n>> + \t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n>> + \t\t\t\t\t&oid, NULL);\n>> + \t\treturn !is_null_oid(&oid);\n>> +\t}\n>\n> FYI. I see\n>\n> .git/rebase-apply/patch:31: space before tab in indent.\n> \t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n> .git/rebase-apply/patch:32: space before tab in indent.\n> \t\t\t\t\t&oid, NULL);\n> .git/rebase-apply/patch:33: space before tab in indent.\n> \t\treturn !is_null_oid(&oid);\n>\n> around here.\n\nThanks for notifying, Jeff mentioned the same and I thought I fixed it.\nI'll have a look at why my editor did this and add some steps to avoid\nthis in the following patches.\n"},{"id":"488567","messageId":"xmqqh6ic5ex3.fsf@gitster.g","threadId":"60768","inReplyTo":"ZcoTbRxIaGmTd4fJ@tanuki","subject":"Re: [PATCH v4 1/5] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-13T19:42:48Z","receivedAt":"2024-02-13T19:42:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> I wonder whether we can maybe consolidate the interface into one or\n> maybe even two functions where the behaviour can be tweaked with a flag\n> field. Something like `refname_is_valid()` with a bunch of flags:\n>\n> - REFNAME_ACCEPT_HEAD to accept \"HEAD\"\n> - REFNAME_ACCEPT_PSEUDOREF to accept all of the refs ending with\n> \"_HEAD\" or being one of the irregular pseudorefs.\n> - REFNAME_ACCEPT_INVALID_BUT_SAFE to accept refnames which aren't\n> valid, but which would pass `refname_is_safe()`.\n\nI am certain we _can_, but it will take an actual patch to see if\nsuch a refactoring makes the callers easier to follow, which is the\nreal test. FWIW, I am much less skeptical than hopeful in this\nparticular case.\n\n\n\n"},{"id":"488632","messageId":"CAOLa=ZQN17Nyxo-uv7CytO1RkaPu9TPfeVHANvV=tycCGpy+Ng@mail.gmail.com","threadId":"60768","inReplyTo":"xmqqh6ic5ex3.fsf@gitster.g","subject":"Re: [PATCH v4 1/5] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-14T10:28:44Z","receivedAt":"2024-02-14T10:29:11Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello,\n\nOn Tue, Feb 13, 2024 at 8:42 PM Junio C Hamano wrote:\n>\n> Patrick Steinhardt writes:\n>\n> > I wonder whether we can maybe consolidate the interface into one or\n> > maybe even two functions where the behaviour can be tweaked with a flag\n> > field. Something like `refname_is_valid()` with a bunch of flags:\n> >\n> > - REFNAME_ACCEPT_HEAD to accept \"HEAD\"\n> > - REFNAME_ACCEPT_PSEUDOREF to accept all of the refs ending with\n> > \"_HEAD\" or being one of the irregular pseudorefs.\n> > - REFNAME_ACCEPT_INVALID_BUT_SAFE to accept refnames which aren't\n> > valid, but which would pass `refname_is_safe()`.\n>\n> I am certain we _can_, but it will take an actual patch to see if\n> such a refactoring makes the callers easier to follow, which is the\n> real test. FWIW, I am much less skeptical than hopeful in this\n> particular case.\n\nI was trying to implement this and realized that the changes sprawl\nmultiple files and\nand have a fair bit of complexity since `check_refname_format()`\nimplements its own\nflags. Overall, adding it to this patch series would overshadow what\nwe're trying to do here.\n\nI think it would be best to tackle this problem after this series has landed.\n\nJunio, let me know if you want me to reroll for the whitespace issues.\nOtherwise, I'll wait\nfor reviews here.\n"},{"id":"488653","messageId":"xmqqzfw3vv6o.fsf@gitster.g","threadId":"60768","inReplyTo":"CAOLa=ZQN17Nyxo-uv7CytO1RkaPu9TPfeVHANvV=tycCGpy+Ng@mail.gmail.com","subject":"Re: [PATCH v4 1/5] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-14T16:59:11Z","receivedAt":"2024-02-14T16:59:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Junio, let me know if you want me to reroll for the whitespace issues.\n\nI think I applied with \"am --whitespace=fix\", so we should be OK,\nbut if you can double check the result that would be appreciated.\n"},{"id":"488660","messageId":"CAOLa=ZTf16qje214mOhgm1XQcQCC=d0VgKem7mLDVb=o3k2vgA@mail.gmail.com","threadId":"60768","inReplyTo":"xmqqzfw3vv6o.fsf@gitster.g","subject":"Re: [PATCH v4 1/5] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-14T18:15:10Z","receivedAt":"2024-02-14T18:15:37Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"On Wed, Feb 14, 2024 at 5:59 PM Junio C Hamano wrote:\n>\n> Karthik Nayak writes:\n>\n> > Junio, let me know if you want me to reroll for the whitespace issues.\n>\n> I think I applied with \"am --whitespace=fix\", so we should be OK,\n> but if you can double check the result that would be appreciated.\n\nI did go through the patches and didn't find any other fixes needed. Thanks\n"},{"id":"489124","messageId":"ZdcJzs2vNkHJsjyN@tanuki","threadId":"60768","inReplyTo":"20240211183923.131278-6-karthik.188@gmail.com","subject":"Re: [PATCH v4 5/5] for-each-ref: add new option to include root refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-22T08:46:06Z","receivedAt":"2024-02-22T08:46:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sun, Feb 11, 2024 at 07:39:23PM +0100, Karthik Nayak wrote:\n> The git-for-each-ref(1) command doesn't provide a way to print root refs\n> i.e pseudorefs and HEAD with the regular \"refs/\" prefixed refs.\n> \n> This commit adds a new option \"--include-root-refs\" to\n> git-for-each-ref(1). When used this would also print pseudorefs and HEAD\n> for the current worktree.\n> \n> Signed-off-by: Karthik Nayak \n> ---\n> Documentation/git-for-each-ref.txt | 5 ++++-\n> builtin/for-each-ref.c | 11 ++++++++---\n> ref-filter.c | 27 +++++++++++++++++++++++++-\n> ref-filter.h | 5 ++++-\n> refs/reftable-backend.c | 11 +++++++----\n> t/t6302-for-each-ref-filter.sh | 31 ++++++++++++++++++++++++++++++\n> 6 files changed, 80 insertions(+), 10 deletions(-)\n> \n> diff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\n> index 3a9ad91b7a..c1dd12b93c 100644\n> --- a/Documentation/git-for-each-ref.txt\n> +++ b/Documentation/git-for-each-ref.txt\n> @@ -10,7 +10,7 @@ SYNOPSIS\n> [verse]\n> 'git for-each-ref' [--count=] [--shell|--perl|--python|--tcl]\n> \t\t [(--sort=)...] [--format=]\n> -\t\t [ --stdin | ... ]\n> +\t\t [--include-root-refs] [ --stdin | ... ]\n> \t\t [--points-at=]\n> \t\t [--merged[=]] [--no-merged[=]]\n> \t\t [--contains[=]] [--no-contains[=]]\n> @@ -105,6 +105,9 @@ TAB %(refname)`.\n> \tany excluded pattern(s) are shown. Matching is done using the\n> \tsame rules as `` above.\n> \n> +--include-root-refs::\n> +\tList root refs (HEAD and pseudorefs) apart from regular refs.\n> +\n> FIELD NAMES\n> -----------\n> \n> diff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\n> index 23d352e371..9ed146dad3 100644\n> --- a/builtin/for-each-ref.c\n> +++ b/builtin/for-each-ref.c\n> @@ -20,10 +20,10 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n> {\n> \tstruct ref_sorting *sorting;\n> \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n> -\tint icase = 0;\n> +\tint icase = 0, include_root_refs = 0, from_stdin = 0;\n> \tstruct ref_filter filter = REF_FILTER_INIT;\n> \tstruct ref_format format = REF_FORMAT_INIT;\n> -\tint from_stdin = 0;\n> +\tunsigned int flags = FILTER_REFS_REGULAR;\n> \tstruct strvec vec = STRVEC_INIT;\n> \n> \tstruct option opts[] = {\n> @@ -53,6 +53,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n> \t\tOPT_NO_CONTAINS(&filter.no_commit, N_(\"print only refs which don't contain the commit\")),\n> \t\tOPT_BOOL(0, \"ignore-case\", &icase, N_(\"sorting and filtering are case insensitive\")),\n> \t\tOPT_BOOL(0, \"stdin\", &from_stdin, N_(\"read reference patterns from stdin\")),\n> +\t\tOPT_BOOL(0, \"include-root-refs\", &include_root_refs, N_(\"also include HEAD ref and pseudorefs\")),\n> \t\tOPT_END(),\n> \t};\n> \n> @@ -96,8 +97,12 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n> \t\tfilter.name_patterns = argv;\n> \t}\n> \n> +\tif (include_root_refs) {\n> +\t\tflags |= FILTER_REFS_ROOT_REFS;\n> +\t}\n\nNit: we don't use braces for single-line blocks.\n\n> +\n> \tfilter.match_as_path = 1;\n> -\tfilter_and_format_refs(&filter, FILTER_REFS_REGULAR, sorting, &format);\n> +\tfilter_and_format_refs(&filter, flags, sorting, &format);\n> \n> \tref_filter_clear(&filter);\n> \tref_sorting_release(sorting);\n> diff --git a/ref-filter.c b/ref-filter.c\n> index acb960e35c..0e83e29390 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -2628,6 +2628,12 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n> \t\t\t\t each_ref_fn cb,\n> \t\t\t\t void *cb_data)\n> {\n> +\tif (filter->kind == FILTER_REFS_KIND_MASK) {\n> +\t\t/* in this case, we want to print all refs including root refs. */\n\nNit: s/in/In, as this is a full sentence.\n\n> +\t\treturn refs_for_each_include_root_refs(get_main_ref_store(the_repository),\n> +\t\t\t\t\t\t cb, cb_data);\n> +\t}\n> +\n> \tif (!filter->match_as_path) {\n> \t\t/*\n> \t\t * in this case, the patterns are applied after\n> @@ -2750,6 +2756,9 @@ static int ref_kind_from_refname(const char *refname)\n> \t\t\treturn ref_kind[i].kind;\n> \t}\n> \n> +\tif (is_pseudoref(get_main_ref_store(the_repository), refname))\n> +\t\treturn FILTER_REFS_PSEUDOREFS;\n> +\n> \treturn FILTER_REFS_OTHERS;\n> }\n> \n> @@ -2781,6 +2790,16 @@ static struct ref_array_item *apply_ref_filter(const char *refname, const struct\n> \n> \t/* Obtain the current ref kind from filter_ref_kind() and ignore unwanted refs. */\n> \tkind = filter_ref_kind(filter, refname);\n> +\n> +\t/*\n> +\t * When printing HEAD with all other refs, we want to apply the same formatting\n> +\t * rules as the other refs, so we simply ask it to be treated as a pseudoref.\n> +\t */\n> +\tif (filter->kind == FILTER_REFS_KIND_MASK && kind == FILTER_REFS_DETACHED_HEAD)\n> +\t\tkind = FILTER_REFS_PSEUDOREFS;\n\nI'm not sure why exactly we need to munge the kind here. Would be great\nif the comment explained what the actual difference would be.\n\n> +\telse if (!(kind & filter->kind))\n> +\t\treturn NULL;\n> +\n> \tif (!(kind & filter->kind))\n> \t\treturn NULL;\n\nThis condition here is duplicated now, isn't it?\n\nPatrick\n\n> @@ -3049,7 +3068,13 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n> \t\t\tret = for_each_fullref_in(\"refs/tags/\", fn, cb_data);\n> \t\telse if (filter->kind & FILTER_REFS_REGULAR)\n> \t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n> -\t\tif (!ret && (filter->kind & FILTER_REFS_DETACHED_HEAD))\n> +\n> +\t\t/*\n> +\t\t * When printing all ref types, HEAD is already included,\n> +\t\t * so we don't want to print HEAD again.\n> +\t\t */\n> +\t\tif (!ret && (filter->kind != FILTER_REFS_KIND_MASK) &&\n> +\t\t (filter->kind & FILTER_REFS_DETACHED_HEAD))\n> \t\t\thead_ref(fn, cb_data);\n> \t}\n> \n> diff --git a/ref-filter.h b/ref-filter.h\n> index 5416936800..0ca28d2bba 100644\n> --- a/ref-filter.h\n> +++ b/ref-filter.h\n> @@ -22,7 +22,10 @@\n> #define FILTER_REFS_REGULAR (FILTER_REFS_TAGS | FILTER_REFS_BRANCHES | \\\n> \t\t\t\t FILTER_REFS_REMOTES | FILTER_REFS_OTHERS)\n> #define FILTER_REFS_DETACHED_HEAD 0x0020\n> -#define FILTER_REFS_KIND_MASK (FILTER_REFS_REGULAR | FILTER_REFS_DETACHED_HEAD)\n> +#define FILTER_REFS_PSEUDOREFS 0x0040\n> +#define FILTER_REFS_ROOT_REFS (FILTER_REFS_DETACHED_HEAD | FILTER_REFS_PSEUDOREFS)\n> +#define FILTER_REFS_KIND_MASK (FILTER_REFS_REGULAR | FILTER_REFS_DETACHED_HEAD | \\\n> +\t\t\t\t FILTER_REFS_PSEUDOREFS)\n> \n> struct atom_value;\n> struct ref_sorting;\n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index a14f2ad7f4..c23a516ac2 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -364,12 +364,15 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> \t\t\tbreak;\n> \n> \t\t/*\n> -\t\t * The files backend only lists references contained in\n> -\t\t * \"refs/\". We emulate the same behaviour here and thus skip\n> -\t\t * all references that don't start with this prefix.\n> +\t\t * The files backend only lists references contained in \"refs/\" unless\n> +\t\t * the root refs are to be included. We emulate the same behaviour here.\n> \t\t */\n> -\t\tif (!starts_with(iter->ref.refname, \"refs/\"))\n> +\t\tif (!starts_with(iter->ref.refname, \"refs/\") &&\n> +\t\t !(iter->flags & DO_FOR_EACH_INCLUDE_ROOT_REFS &&\n> +\t\t (is_pseudoref(&iter->refs->base, iter->ref.refname) ||\n> +\t\t is_headref(&iter->refs->base, iter->ref.refname)))) {\n> \t\t\tcontinue;\n> +\t\t}\n> \n> \t\tif (iter->prefix &&\n> \t\t strncmp(iter->prefix, iter->ref.refname, strlen(iter->prefix))) {\n> diff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh\n> index 82f3d1ea0f..948f1bb5f4 100755\n> --- a/t/t6302-for-each-ref-filter.sh\n> +++ b/t/t6302-for-each-ref-filter.sh\n> @@ -31,6 +31,37 @@ test_expect_success 'setup some history and refs' '\n> \tgit update-ref refs/odd/spot main\n> '\n> \n> +test_expect_success '--include-root-refs pattern prints pseudorefs' '\n> +\tcat >expect <<-\\EOF &&\n> +\tHEAD\n> +\tORIG_HEAD\n> +\trefs/heads/main\n> +\trefs/heads/side\n> +\trefs/odd/spot\n> +\trefs/tags/annotated-tag\n> +\trefs/tags/doubly-annotated-tag\n> +\trefs/tags/doubly-signed-tag\n> +\trefs/tags/four\n> +\trefs/tags/one\n> +\trefs/tags/signed-tag\n> +\trefs/tags/three\n> +\trefs/tags/two\n> +\tEOF\n> +\tgit update-ref ORIG_HEAD main &&\n> +\tgit for-each-ref --format=\"%(refname)\" --include-root-refs >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success '--include-root-refs with other patterns' '\n> +\tcat >expect <<-\\EOF &&\n> +\tHEAD\n> +\tORIG_HEAD\n> +\tEOF\n> +\tgit update-ref ORIG_HEAD main &&\n> +\tgit for-each-ref --format=\"%(refname)\" --include-root-refs \"*HEAD\" >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> test_expect_success 'filtering with --points-at' '\n> \tcat >expect <<-\\EOF &&\n> \trefs/heads/main\n> -- \n> 2.43.GIT\n> \n"},{"id":"489132","messageId":"CAOLa=ZQkqC7ahWUn=9aZksQOietGEY9M9Chj=uLTPHoXm8MuiQ@mail.gmail.com","threadId":"60768","inReplyTo":"ZdcJzs2vNkHJsjyN@tanuki","subject":"Re: [PATCH v4 5/5] for-each-ref: add new option to include root refs","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-22T12:57:04Z","receivedAt":"2024-02-22T12:57:06Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n>> @@ -96,8 +97,12 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n>> \t\tfilter.name_patterns = argv;\n>> \t}\n>>\n>> +\tif (include_root_refs) {\n>> +\t\tflags |= FILTER_REFS_ROOT_REFS;\n>> +\t}\n>\n> Nit: we don't use braces for single-line blocks.\n>\n\nRight, thanks, I always trip on this.\n\n>> diff --git a/ref-filter.c b/ref-filter.c\n>> index acb960e35c..0e83e29390 100644\n>> --- a/ref-filter.c\n>> +++ b/ref-filter.c\n>> @@ -2628,6 +2628,12 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n>> \t\t\t\t each_ref_fn cb,\n>> \t\t\t\t void *cb_data)\n>> {\n>> +\tif (filter->kind == FILTER_REFS_KIND_MASK) {\n>> +\t\t/* in this case, we want to print all refs including root refs. */\n>\n> Nit: s/in/In, as this is a full sentence.\n>\n\nWill change\n\n\n>> @@ -2781,6 +2790,16 @@ static struct ref_array_item *apply_ref_filter(const char *refname, const struct\n>>\n>> \t/* Obtain the current ref kind from filter_ref_kind() and ignore unwanted refs. */\n>> \tkind = filter_ref_kind(filter, refname);\n>> +\n>> +\t/*\n>> +\t * When printing HEAD with all other refs, we want to apply the same formatting\n>> +\t * rules as the other refs, so we simply ask it to be treated as a pseudoref.\n>> +\t */\n>> +\tif (filter->kind == FILTER_REFS_KIND_MASK && kind == FILTER_REFS_DETACHED_HEAD)\n>> +\t\tkind = FILTER_REFS_PSEUDOREFS;\n>\n> I'm not sure why exactly we need to munge the kind here. Would be great\n> if the comment explained what the actual difference would be.\n>\n\nSo the difference is\n\n$ git for-each-ref --include-root-refs\n9eda75497d43f2f9c70c1e14afb865108f9b4b49 commit\tFETCH_HEAD\nee99ac41aeb4129866710fc5771f11e1c1742dee commit\tHEAD\n96c8a0712e569dd2812bf4fb5e72113caf326500 commit\tORIG_HEAD\n\nvs\n\n$ git for-each-ref --include-root-refs | grep -v refs/\nb4b94355057280749620c47999a4b45dc60f2681 commit\t(HEAD detached at b4b9435505)\n9eda75497d43f2f9c70c1e14afb865108f9b4b49 commit\tFETCH_HEAD\nee99ac41aeb4129866710fc5771f11e1c1742dee commit\tORIG_HEAD\n\nThis is because in `get_refname` we provide head description. But in\ngit-for-each-ref we don't want this, we want the format to be consistent.\n\n>> +\telse if (!(kind & filter->kind))\n>> +\t\treturn NULL;\n>> +\n>> \tif (!(kind & filter->kind))\n>> \t\treturn NULL;\n>\n> This condition here is duplicated now, isn't it?\n>\n> Patrick\n>\n\nSeems like a bad merge on my end. Will fix. Thanks for the review. Will\nsend a new version soon.\n"},{"id":"489133","messageId":"ZddJWWrSpxEdI6lS@tanuki","threadId":"60768","inReplyTo":"CAOLa=ZQkqC7ahWUn=9aZksQOietGEY9M9Chj=uLTPHoXm8MuiQ@mail.gmail.com","subject":"Re: [PATCH v4 5/5] for-each-ref: add new option to include root refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-22T13:17:13Z","receivedAt":"2024-02-22T13:17:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Feb 22, 2024 at 04:57:04AM -0800, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> >> @@ -2781,6 +2790,16 @@ static struct ref_array_item *apply_ref_filter(const char *refname, const struct\n> >>\n> >> \t/* Obtain the current ref kind from filter_ref_kind() and ignore unwanted refs. */\n> >> \tkind = filter_ref_kind(filter, refname);\n> >> +\n> >> +\t/*\n> >> +\t * When printing HEAD with all other refs, we want to apply the same formatting\n> >> +\t * rules as the other refs, so we simply ask it to be treated as a pseudoref.\n> >> +\t */\n> >> +\tif (filter->kind == FILTER_REFS_KIND_MASK && kind == FILTER_REFS_DETACHED_HEAD)\n> >> +\t\tkind = FILTER_REFS_PSEUDOREFS;\n> >\n> > I'm not sure why exactly we need to munge the kind here. Would be great\n> > if the comment explained what the actual difference would be.\n> >\n> \n> So the difference is\n> \n> $ git for-each-ref --include-root-refs\n> 9eda75497d43f2f9c70c1e14afb865108f9b4b49 commit\tFETCH_HEAD\n> ee99ac41aeb4129866710fc5771f11e1c1742dee commit\tHEAD\n> 96c8a0712e569dd2812bf4fb5e72113caf326500 commit\tORIG_HEAD\n> \n> vs\n> \n> $ git for-each-ref --include-root-refs | grep -v refs/\n> b4b94355057280749620c47999a4b45dc60f2681 commit\t(HEAD detached at b4b9435505)\n> 9eda75497d43f2f9c70c1e14afb865108f9b4b49 commit\tFETCH_HEAD\n> ee99ac41aeb4129866710fc5771f11e1c1742dee commit\tORIG_HEAD\n> \n> This is because in `get_refname` we provide head description. But in\n> git-for-each-ref we don't want this, we want the format to be consistent.\n\nAh, makes sense, thanks for the explanation! Briefly mentioning this in\nthe comment should help readers to understand this a lot faster.\n\nPatrick\n"},{"id":"489233","messageId":"20240223100112.44127-1-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240119142705.139374-1-karthik.188@gmail.com","subject":"[PATCH v5 0/5] for-each-ref: add '--include-root-refs' option","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-23T10:01:07Z","receivedAt":"2024-02-23T10:01:18Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"This is the fifth version of my patch series to print root refs\nin git-for-each-ref(1).\n\nWith the introduction of the reftable backend, it becomes ever\nso important to provide the necessary tooling for printing all refs\nassociated with a worktree.\n\nWhile regular refs stored within the \"refs/\" namespace are currently\nsupported by multiple commands like git-for-each-ref(1),\ngit-show-ref(1). Neither support printing root refs within the worktree.\n\nThis patch series is a follow up to the RFC/discussion we had earlier on\nthe list [1].\n\nThe first 4 commits add the required functionality to ensure we can print\nall refs (regular, pseudo, HEAD). The 5th commit modifies the\ngit-for-each-ref(1) command to add the \"--include-root-refs\" command which\nwill include HEAD and pseudorefs with regular \"refs/\" refs.\n\n[1]: https://lore.kernel.org/git/20231221170715.110565-1-karthik.188@gmail.com/#t\n\nChanges from v4:\n1. Fixed erratic whitespace\n2. Remove braces from single line block\n3. Starting the comments with a capital and also adding more context.\n4. Removed a duplicate check.\n\nThanks for the reviews.\n\nRange diff against v4:\n\n1: 98130a7ad7 ! 1: 6016042965 refs: introduce `is_pseudoref()` and `is_headref()`\n @@ refs.c: static int is_pseudoref_syntax(const char *refname)\n +\n +\tif (ends_with(refname, \"_HEAD\")) {\n +\t\trefs_resolve_ref_unsafe(refs, refname,\n -+ \t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n -+ \t\t\t\t\t&oid, NULL);\n -+ \t\treturn !is_null_oid(&oid);\n ++\t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n ++\t\t\t\t\t&oid, NULL);\n ++\t\treturn !is_null_oid(&oid);\n +\t}\n +\n +\tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n +\t\tif (!strcmp(refname, irregular_pseudorefs[i])) {\n +\t\t\trefs_resolve_ref_unsafe(refs, refname,\n -+ \t\t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n -+ \t\t\t\t\t\t&oid, NULL);\n ++\t\t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n ++\t\t\t\t\t\t&oid, NULL);\n +\t\t\treturn !is_null_oid(&oid);\n +\t\t}\n +\n2: 060ab08af5 = 2: acaa014841 refs: extract out `loose_fill_ref_dir_regular_file()`\n3: 40d2375ad9 = 3: f51c5bc307 refs: introduce `refs_for_each_include_root_refs()`\n4: b4b9435505 = 4: 7c004db6e7 ref-filter: rename 'FILTER_REFS_ALL' to 'FILTER_REFS_REGULAR'\n5: ee99ac41ae ! 5: 53f6c0a6db for-each-ref: add new option to include root refs\n @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const\n \t\tfilter.name_patterns = argv;\n \t}\n \n -+\tif (include_root_refs) {\n ++\tif (include_root_refs)\n +\t\tflags |= FILTER_REFS_ROOT_REFS;\n -+\t}\n +\n \tfilter.match_as_path = 1;\n -\tfilter_and_format_refs(&filter, FILTER_REFS_REGULAR, sorting, &format);\n @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t\t\t void *cb_data)\n {\n +\tif (filter->kind == FILTER_REFS_KIND_MASK) {\n -+\t\t/* in this case, we want to print all refs including root refs. */\n ++\t\t/* In this case, we want to print all refs including root refs. */\n +\t\treturn refs_for_each_include_root_refs(get_main_ref_store(the_repository),\n +\t\t\t\t\t\t cb, cb_data);\n +\t}\n @@ ref-filter.c: static struct ref_array_item *apply_ref_filter(const char *refname\n \n \t/* Obtain the current ref kind from filter_ref_kind() and ignore unwanted refs. */\n \tkind = filter_ref_kind(filter, refname);\n +-\tif (!(kind & filter->kind))\n +\n +\t/*\n -+\t * When printing HEAD with all other refs, we want to apply the same formatting\n -+\t * rules as the other refs, so we simply ask it to be treated as a pseudoref.\n ++\t * Generally HEAD refs are printed with special description denoting a rebase,\n ++\t * detached state and so forth. This is useful when only printing the HEAD ref\n ++\t * But when it is being printed along with other pseudorefs, it makes sense to\n ++\t * keep the formatting consistent. So we mask the type to act like a pseudoref.\n +\t */\n +\tif (filter->kind == FILTER_REFS_KIND_MASK && kind == FILTER_REFS_DETACHED_HEAD)\n +\t\tkind = FILTER_REFS_PSEUDOREFS;\n +\telse if (!(kind & filter->kind))\n -+\t\treturn NULL;\n -+\n - \tif (!(kind & filter->kind))\n \t\treturn NULL;\n \n + \tif (!filter_pattern_match(filter, refname))\n @@ ref-filter.c: static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \t\t\tret = for_each_fullref_in(\"refs/tags/\", fn, cb_data);\n \t\telse if (filter->kind & FILTER_REFS_REGULAR)\n\n\nKarthik Nayak (5):\n refs: introduce `is_pseudoref()` and `is_headref()`\n refs: extract out `loose_fill_ref_dir_regular_file()`\n refs: introduce `refs_for_each_include_root_refs()`\n ref-filter: rename 'FILTER_REFS_ALL' to 'FILTER_REFS_REGULAR'\n for-each-ref: add new option to include root refs\n\n Documentation/git-for-each-ref.txt | 5 +-\n builtin/for-each-ref.c | 10 ++-\n ref-filter.c | 30 ++++++-\n ref-filter.h | 7 +-\n refs.c | 48 +++++++++++\n refs.h | 9 ++\n refs/files-backend.c | 127 +++++++++++++++++++++--------\n refs/refs-internal.h | 6 ++\n refs/reftable-backend.c | 11 ++-\n t/t6302-for-each-ref-filter.sh | 31 +++++++\n 10 files changed, 237 insertions(+), 47 deletions(-)\n\n-- \n2.43.GIT\n\n"},{"id":"489234","messageId":"20240223100112.44127-2-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240223100112.44127-1-karthik.188@gmail.com","subject":"[PATCH v5 1/5] refs: introduce `is_pseudoref()` and `is_headref()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-23T10:01:08Z","receivedAt":"2024-02-23T10:01:18Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Introduce two new functions `is_pseudoref()` and `is_headref()`. This\nprovides the necessary functionality for us to add pseudorefs and HEAD\nto the loose ref cache in the files backend, allowing us to build\ntooling to print these refs.\n\nThe `is_pseudoref()` function internally calls `is_pseudoref_syntax()`\nbut adds onto it by also checking to ensure that the pseudoref either\nends with a \"_HEAD\" suffix or matches a list of exceptions. After which\nwe also parse the contents of the pseudoref to ensure that it conforms\nto the ref format.\n\nWe cannot directly add the new syntax checks to `is_pseudoref_syntax()`\nbecause the function is also used by `is_current_worktree_ref()` and\nmaking it stricter to match only known pseudorefs might have unintended\nconsequences due to files like 'BISECT_START' which isn't a pseudoref\nbut sometimes contains object ID.\n\nKeeping this in mind, we leave `is_pseudoref_syntax()` as is and create\n`is_pseudoref()` which is stricter. Ideally we'd want to move the new\nsyntax checks to `is_pseudoref_syntax()` but a prerequisite for this\nwould be to actually remove the exception list by converting those\npseudorefs to also contain a '_HEAD' suffix and perhaps move bisect\nrelated files like 'BISECT_START' to a new directory similar to the\n'rebase-merge' directory.\n\nHelped-by: Jeff King \nSigned-off-by: Karthik Nayak \n---\n refs.c | 41 +++++++++++++++++++++++++++++++++++++++++\n refs.h | 3 +++\n 2 files changed, 44 insertions(+)\n\ndiff --git a/refs.c b/refs.c\nindex fff343c256..3546d90831 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -860,6 +860,47 @@ static int is_pseudoref_syntax(const char *refname)\n \treturn 1;\n }\n \n+int is_pseudoref(struct ref_store *refs, const char *refname)\n+{\n+\tstatic const char *const irregular_pseudorefs[] = {\n+\t\t\"AUTO_MERGE\",\n+\t\t\"BISECT_EXPECTED_REV\",\n+\t\t\"NOTES_MERGE_PARTIAL\",\n+\t\t\"NOTES_MERGE_REF\",\n+\t\t\"MERGE_AUTOSTASH\",\n+\t};\n+\tstruct object_id oid;\n+\tsize_t i;\n+\n+\tif (!is_pseudoref_syntax(refname))\n+\t\treturn 0;\n+\n+\tif (ends_with(refname, \"_HEAD\")) {\n+\t\trefs_resolve_ref_unsafe(refs, refname,\n+\t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n+\t\t\t\t\t&oid, NULL);\n+\t\treturn !is_null_oid(&oid);\n+\t}\n+\n+\tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n+\t\tif (!strcmp(refname, irregular_pseudorefs[i])) {\n+\t\t\trefs_resolve_ref_unsafe(refs, refname,\n+\t\t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n+\t\t\t\t\t\t&oid, NULL);\n+\t\t\treturn !is_null_oid(&oid);\n+\t\t}\n+\n+\treturn 0;\n+}\n+\n+int is_headref(struct ref_store *refs, const char *refname)\n+{\n+\tif (!strcmp(refname, \"HEAD\"))\n+\t\treturn refs_ref_exists(refs, refname);\n+\n+\treturn 0;\n+}\n+\n static int is_current_worktree_ref(const char *ref) {\n \treturn is_pseudoref_syntax(ref) || is_per_worktree_ref(ref);\n }\ndiff --git a/refs.h b/refs.h\nindex 303c5fac4d..f66cdd731c 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1023,4 +1023,7 @@ extern struct ref_namespace_info ref_namespace[NAMESPACE__COUNT];\n */\n void update_ref_namespace(enum ref_namespace namespace, char *ref);\n \n+int is_pseudoref(struct ref_store *refs, const char *refname);\n+int is_headref(struct ref_store *refs, const char *refname);\n+\n #endif /* REFS_H */\n-- \n2.43.GIT\n\n"},{"id":"489235","messageId":"20240223100112.44127-3-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240223100112.44127-1-karthik.188@gmail.com","subject":"[PATCH v5 2/5] refs: extract out `loose_fill_ref_dir_regular_file()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-23T10:01:09Z","receivedAt":"2024-02-23T10:01:19Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Extract out the code for adding a single file to the loose ref dir as\n`loose_fill_ref_dir_regular_file()` from `loose_fill_ref_dir()` in\n`refs/files-backend.c`.\n\nThis allows us to use this function independently in the following\ncommits where we add code to also add pseudorefs to the ref dir.\n\nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 62 +++++++++++++++++++++++---------------------\n 1 file changed, 33 insertions(+), 29 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 75dcc21ecb..65128821a8 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -229,6 +229,38 @@ static void add_per_worktree_entries_to_dir(struct ref_dir *dir, const char *dir\n \t}\n }\n \n+static void loose_fill_ref_dir_regular_file(struct files_ref_store *refs,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t struct ref_dir *dir)\n+{\n+\tstruct object_id oid;\n+\tint flag;\n+\n+\tif (!refs_resolve_ref_unsafe(&refs->base, refname, RESOLVE_REF_READING,\n+\t\t\t\t &oid, &flag)) {\n+\t\toidclr(&oid);\n+\t\tflag |= REF_ISBROKEN;\n+\t} else if (is_null_oid(&oid)) {\n+\t\t/*\n+\t\t * It is so astronomically unlikely\n+\t\t * that null_oid is the OID of an\n+\t\t * actual object that we consider its\n+\t\t * appearance in a loose reference\n+\t\t * file to be repo corruption\n+\t\t * (probably due to a software bug).\n+\t\t */\n+\t\tflag |= REF_ISBROKEN;\n+\t}\n+\n+\tif (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tif (!refname_is_safe(refname))\n+\t\t\tdie(\"loose refname is dangerous: %s\", refname);\n+\t\toidclr(&oid);\n+\t\tflag |= REF_BAD_NAME | REF_ISBROKEN;\n+\t}\n+\tadd_entry_to_dir(dir, create_ref_entry(refname, &oid, flag));\n+}\n+\n /*\n * Read the loose references from the namespace dirname into dir\n * (without recursing). dirname must end with '/'. dir must be the\n@@ -257,8 +289,6 @@ static void loose_fill_ref_dir(struct ref_store *ref_store,\n \tstrbuf_add(&refname, dirname, dirnamelen);\n \n \twhile ((de = readdir(d)) != NULL) {\n-\t\tstruct object_id oid;\n-\t\tint flag;\n \t\tunsigned char dtype;\n \n \t\tif (de->d_name[0] == '.')\n@@ -274,33 +304,7 @@ static void loose_fill_ref_dir(struct ref_store *ref_store,\n \t\t\t\t\t create_dir_entry(dir->cache, refname.buf,\n \t\t\t\t\t\t\t refname.len));\n \t\t} else if (dtype == DT_REG) {\n-\t\t\tif (!refs_resolve_ref_unsafe(&refs->base,\n-\t\t\t\t\t\t refname.buf,\n-\t\t\t\t\t\t RESOLVE_REF_READING,\n-\t\t\t\t\t\t &oid, &flag)) {\n-\t\t\t\toidclr(&oid);\n-\t\t\t\tflag |= REF_ISBROKEN;\n-\t\t\t} else if (is_null_oid(&oid)) {\n-\t\t\t\t/*\n-\t\t\t\t * It is so astronomically unlikely\n-\t\t\t\t * that null_oid is the OID of an\n-\t\t\t\t * actual object that we consider its\n-\t\t\t\t * appearance in a loose reference\n-\t\t\t\t * file to be repo corruption\n-\t\t\t\t * (probably due to a software bug).\n-\t\t\t\t */\n-\t\t\t\tflag |= REF_ISBROKEN;\n-\t\t\t}\n-\n-\t\t\tif (check_refname_format(refname.buf,\n-\t\t\t\t\t\t REFNAME_ALLOW_ONELEVEL)) {\n-\t\t\t\tif (!refname_is_safe(refname.buf))\n-\t\t\t\t\tdie(\"loose refname is dangerous: %s\", refname.buf);\n-\t\t\t\toidclr(&oid);\n-\t\t\t\tflag |= REF_BAD_NAME | REF_ISBROKEN;\n-\t\t\t}\n-\t\t\tadd_entry_to_dir(dir,\n-\t\t\t\t\t create_ref_entry(refname.buf, &oid, flag));\n+\t\t\tloose_fill_ref_dir_regular_file(refs, refname.buf, dir);\n \t\t}\n \t\tstrbuf_setlen(&refname, dirnamelen);\n \t}\n-- \n2.43.GIT\n\n"},{"id":"489236","messageId":"20240223100112.44127-4-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240223100112.44127-1-karthik.188@gmail.com","subject":"[PATCH v5 3/5] refs: introduce `refs_for_each_include_root_refs()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-23T10:01:10Z","receivedAt":"2024-02-23T10:01:20Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Introduce a new ref iteration flag `DO_FOR_EACH_INCLUDE_ROOT_REFS`,\nwhich will be used to iterate over regular refs plus pseudorefs and\nHEAD.\n\nRefs which fall outside the `refs/` and aren't either pseudorefs or HEAD\nare more of a grey area. This is because we don't block the users from\ncreating such refs but they are not officially supported.\n\nIntroduce `refs_for_each_include_root_refs()` which calls\n`do_for_each_ref()` with this newly introduced flag.\n\nIn `refs/files-backend.c`, introduce a new function\n`add_pseudoref_and_head_entries()` to add pseudorefs and HEAD to the\n`ref_dir`. We then finally call `add_pseudoref_and_head_entries()`\nwhenever the `DO_FOR_EACH_INCLUDE_ROOT_REFS` flag is set. Any new ref\nbackend will also have to implement similar changes on its end.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 7 +++++\n refs.h | 6 ++++\n refs/files-backend.c | 65 ++++++++++++++++++++++++++++++++++++++++----\n refs/refs-internal.h | 6 ++++\n 4 files changed, 79 insertions(+), 5 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 3546d90831..7d58fe1e09 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1765,6 +1765,13 @@ int for_each_rawref(each_ref_fn fn, void *cb_data)\n \treturn refs_for_each_rawref(get_main_ref_store(the_repository), fn, cb_data);\n }\n \n+int refs_for_each_include_root_refs(struct ref_store *refs, each_ref_fn fn,\n+\t\t\t\t void *cb_data)\n+{\n+\treturn do_for_each_ref(refs, \"\", NULL, fn, 0,\n+\t\t\t DO_FOR_EACH_INCLUDE_ROOT_REFS, cb_data);\n+}\n+\n static int qsort_strcmp(const void *va, const void *vb)\n {\n \tconst char *a = *(const char **)va;\ndiff --git a/refs.h b/refs.h\nindex f66cdd731c..5cfaee6229 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -398,6 +398,12 @@ int for_each_namespaced_ref(const char **exclude_patterns,\n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data);\n int for_each_rawref(each_ref_fn fn, void *cb_data);\n \n+/*\n+ * Iterates over all refs including root refs, i.e. pseudorefs and HEAD.\n+ */\n+int refs_for_each_include_root_refs(struct ref_store *refs, each_ref_fn fn,\n+\t\t\t\t void *cb_data);\n+\n /*\n * Normalizes partial refs to their fully qualified form.\n * Will prepend to the if it doesn't start with 'refs/'.\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 65128821a8..9c1c42fe52 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -315,9 +315,59 @@ static void loose_fill_ref_dir(struct ref_store *ref_store,\n \tadd_per_worktree_entries_to_dir(dir, dirname);\n }\n \n-static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs)\n+/*\n+ * Add pseudorefs to the ref dir by parsing the directory for any files\n+ * which follow the pseudoref syntax.\n+ */\n+static void add_pseudoref_and_head_entries(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_dir *dir,\n+\t\t\t\t\t const char *dirname)\n+{\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fill_ref_dir\");\n+\tstruct strbuf path = STRBUF_INIT, refname = STRBUF_INIT;\n+\tstruct dirent *de;\n+\tsize_t dirnamelen;\n+\tDIR *d;\n+\n+\tfiles_ref_path(refs, &path, dirname);\n+\n+\td = opendir(path.buf);\n+\tif (!d) {\n+\t\tstrbuf_release(&path);\n+\t\treturn;\n+\t}\n+\n+\tstrbuf_addstr(&refname, dirname);\n+\tdirnamelen = refname.len;\n+\n+\twhile ((de = readdir(d)) != NULL) {\n+\t\tunsigned char dtype;\n+\n+\t\tif (de->d_name[0] == '.')\n+\t\t\tcontinue;\n+\t\tif (ends_with(de->d_name, \".lock\"))\n+\t\t\tcontinue;\n+\t\tstrbuf_addstr(&refname, de->d_name);\n+\n+\t\tdtype = get_dtype(de, &path, 1);\n+\t\tif (dtype == DT_REG && (is_pseudoref(ref_store, de->d_name) ||\n+\t\t\t\t\t\t\t\tis_headref(ref_store, de->d_name)))\n+\t\t\tloose_fill_ref_dir_regular_file(refs, refname.buf, dir);\n+\n+\t\tstrbuf_setlen(&refname, dirnamelen);\n+\t}\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&path);\n+\tclosedir(d);\n+}\n+\n+static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs,\n+\t\t\t\t\t unsigned int flags)\n {\n \tif (!refs->loose) {\n+\t\tstruct ref_dir *dir;\n+\n \t\t/*\n \t\t * Mark the top-level directory complete because we\n \t\t * are about to read the only subdirectory that can\n@@ -328,12 +378,17 @@ static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs)\n \t\t/* We're going to fill the top level ourselves: */\n \t\trefs->loose->root->flag &= ~REF_INCOMPLETE;\n \n+\t\tdir = get_ref_dir(refs->loose->root);\n+\n+\t\tif (flags & DO_FOR_EACH_INCLUDE_ROOT_REFS)\n+\t\t\tadd_pseudoref_and_head_entries(dir->cache->ref_store, dir,\n+\t\t\t\t\t\t refs->loose->root->name);\n+\n \t\t/*\n \t\t * Add an incomplete entry for \"refs/\" (to be filled\n \t\t * lazily):\n \t\t */\n-\t\tadd_entry_to_dir(get_ref_dir(refs->loose->root),\n-\t\t\t\t create_dir_entry(refs->loose, \"refs/\", 5));\n+\t\tadd_entry_to_dir(dir, create_dir_entry(refs->loose, \"refs/\", 5));\n \t}\n \treturn refs->loose;\n }\n@@ -861,7 +916,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \t * disk, and re-reads it if not.\n \t */\n \n-\tloose_iter = cache_ref_iterator_begin(get_loose_ref_cache(refs),\n+\tloose_iter = cache_ref_iterator_begin(get_loose_ref_cache(refs, flags),\n \t\t\t\t\t prefix, ref_store->repo, 1);\n \n \t/*\n@@ -1222,7 +1277,7 @@ static int files_pack_refs(struct ref_store *ref_store,\n \n \tpacked_refs_lock(refs->packed_ref_store, LOCK_DIE_ON_ERROR, &err);\n \n-\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs), NULL,\n+\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs, 0), NULL,\n \t\t\t\t\tthe_repository, 0);\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t/*\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 83e0f0bba3..73a8fa18ad 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -260,6 +260,12 @@ enum do_for_each_ref_flags {\n \t * INCLUDE_BROKEN, since they are otherwise not included at all.\n \t */\n \tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n+\n+\t/*\n+\t * Include root refs i.e. HEAD and pseudorefs along with the regular\n+\t * refs.\n+\t */\n+\tDO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),\n };\n \n /*\n-- \n2.43.GIT\n\n"},{"id":"489237","messageId":"20240223100112.44127-5-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240223100112.44127-1-karthik.188@gmail.com","subject":"[PATCH v5 4/5] ref-filter: rename 'FILTER_REFS_ALL' to 'FILTER_REFS_REGULAR'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-23T10:01:11Z","receivedAt":"2024-02-23T10:01:21Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The flag 'FILTER_REFS_ALL' is a bit ambiguous, where ALL doesn't specify\nif it means to contain refs from all worktrees or whether all types of\nrefs (regular, HEAD & pseudorefs) or all of the above.\n\nSince here it is actually referring to all refs with the \"refs/\" prefix,\nlet's rename it to 'FILTER_REFS_REGULAR' to indicate that this is\nspecifically for regular refs.\n\nSigned-off-by: Karthik Nayak \n---\n builtin/for-each-ref.c | 2 +-\n ref-filter.c | 2 +-\n ref-filter.h | 4 ++--\n 3 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 3885a9c28e..23d352e371 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -97,7 +97,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t}\n \n \tfilter.match_as_path = 1;\n-\tfilter_and_format_refs(&filter, FILTER_REFS_ALL, sorting, &format);\n+\tfilter_and_format_refs(&filter, FILTER_REFS_REGULAR, sorting, &format);\n \n \tref_filter_clear(&filter);\n \tref_sorting_release(sorting);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex be14b56e32..acb960e35c 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -3047,7 +3047,7 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \t\t\tret = for_each_fullref_in(\"refs/remotes/\", fn, cb_data);\n \t\telse if (filter->kind == FILTER_REFS_TAGS)\n \t\t\tret = for_each_fullref_in(\"refs/tags/\", fn, cb_data);\n-\t\telse if (filter->kind & FILTER_REFS_ALL)\n+\t\telse if (filter->kind & FILTER_REFS_REGULAR)\n \t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n \t\tif (!ret && (filter->kind & FILTER_REFS_DETACHED_HEAD))\n \t\t\thead_ref(fn, cb_data);\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 07cd6f6da3..5416936800 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -19,10 +19,10 @@\n #define FILTER_REFS_BRANCHES 0x0004\n #define FILTER_REFS_REMOTES 0x0008\n #define FILTER_REFS_OTHERS 0x0010\n-#define FILTER_REFS_ALL (FILTER_REFS_TAGS | FILTER_REFS_BRANCHES | \\\n+#define FILTER_REFS_REGULAR (FILTER_REFS_TAGS | FILTER_REFS_BRANCHES | \\\n \t\t\t\t FILTER_REFS_REMOTES | FILTER_REFS_OTHERS)\n #define FILTER_REFS_DETACHED_HEAD 0x0020\n-#define FILTER_REFS_KIND_MASK (FILTER_REFS_ALL | FILTER_REFS_DETACHED_HEAD)\n+#define FILTER_REFS_KIND_MASK (FILTER_REFS_REGULAR | FILTER_REFS_DETACHED_HEAD)\n \n struct atom_value;\n struct ref_sorting;\n-- \n2.43.GIT\n\n"},{"id":"489238","messageId":"20240223100112.44127-6-karthik.188@gmail.com","threadId":"60768","inReplyTo":"20240223100112.44127-1-karthik.188@gmail.com","subject":"[PATCH v5 5/5] for-each-ref: add new option to include root refs","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-23T10:01:12Z","receivedAt":"2024-02-23T10:01:22Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The git-for-each-ref(1) command doesn't provide a way to print root refs\ni.e pseudorefs and HEAD with the regular \"refs/\" prefixed refs.\n\nThis commit adds a new option \"--include-root-refs\" to\ngit-for-each-ref(1). When used this would also print pseudorefs and HEAD\nfor the current worktree.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-for-each-ref.txt | 5 ++++-\n builtin/for-each-ref.c | 10 +++++++---\n ref-filter.c | 28 +++++++++++++++++++++++++--\n ref-filter.h | 5 ++++-\n refs/reftable-backend.c | 11 +++++++----\n t/t6302-for-each-ref-filter.sh | 31 ++++++++++++++++++++++++++++++\n 6 files changed, 79 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 3a9ad91b7a..c1dd12b93c 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n [verse]\n 'git for-each-ref' [--count=] [--shell|--perl|--python|--tcl]\n \t\t [(--sort=)...] [--format=]\n-\t\t [ --stdin | ... ]\n+\t\t [--include-root-refs] [ --stdin | ... ]\n \t\t [--points-at=]\n \t\t [--merged[=]] [--no-merged[=]]\n \t\t [--contains[=]] [--no-contains[=]]\n@@ -105,6 +105,9 @@ TAB %(refname)`.\n \tany excluded pattern(s) are shown. Matching is done using the\n \tsame rules as `` above.\n \n+--include-root-refs::\n+\tList root refs (HEAD and pseudorefs) apart from regular refs.\n+\n FIELD NAMES\n -----------\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 23d352e371..919282e12a 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -20,10 +20,10 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n {\n \tstruct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n-\tint icase = 0;\n+\tint icase = 0, include_root_refs = 0, from_stdin = 0;\n \tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_format format = REF_FORMAT_INIT;\n-\tint from_stdin = 0;\n+\tunsigned int flags = FILTER_REFS_REGULAR;\n \tstruct strvec vec = STRVEC_INIT;\n \n \tstruct option opts[] = {\n@@ -53,6 +53,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\tOPT_NO_CONTAINS(&filter.no_commit, N_(\"print only refs which don't contain the commit\")),\n \t\tOPT_BOOL(0, \"ignore-case\", &icase, N_(\"sorting and filtering are case insensitive\")),\n \t\tOPT_BOOL(0, \"stdin\", &from_stdin, N_(\"read reference patterns from stdin\")),\n+\t\tOPT_BOOL(0, \"include-root-refs\", &include_root_refs, N_(\"also include HEAD ref and pseudorefs\")),\n \t\tOPT_END(),\n \t};\n \n@@ -96,8 +97,11 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\tfilter.name_patterns = argv;\n \t}\n \n+\tif (include_root_refs)\n+\t\tflags |= FILTER_REFS_ROOT_REFS;\n+\n \tfilter.match_as_path = 1;\n-\tfilter_and_format_refs(&filter, FILTER_REFS_REGULAR, sorting, &format);\n+\tfilter_and_format_refs(&filter, flags, sorting, &format);\n \n \tref_filter_clear(&filter);\n \tref_sorting_release(sorting);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex acb960e35c..0ec29f7385 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2628,6 +2628,12 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t\t\t each_ref_fn cb,\n \t\t\t\t void *cb_data)\n {\n+\tif (filter->kind == FILTER_REFS_KIND_MASK) {\n+\t\t/* In this case, we want to print all refs including root refs. */\n+\t\treturn refs_for_each_include_root_refs(get_main_ref_store(the_repository),\n+\t\t\t\t\t\t cb, cb_data);\n+\t}\n+\n \tif (!filter->match_as_path) {\n \t\t/*\n \t\t * in this case, the patterns are applied after\n@@ -2750,6 +2756,9 @@ static int ref_kind_from_refname(const char *refname)\n \t\t\treturn ref_kind[i].kind;\n \t}\n \n+\tif (is_pseudoref(get_main_ref_store(the_repository), refname))\n+\t\treturn FILTER_REFS_PSEUDOREFS;\n+\n \treturn FILTER_REFS_OTHERS;\n }\n \n@@ -2781,7 +2790,16 @@ static struct ref_array_item *apply_ref_filter(const char *refname, const struct\n \n \t/* Obtain the current ref kind from filter_ref_kind() and ignore unwanted refs. */\n \tkind = filter_ref_kind(filter, refname);\n-\tif (!(kind & filter->kind))\n+\n+\t/*\n+\t * Generally HEAD refs are printed with special description denoting a rebase,\n+\t * detached state and so forth. This is useful when only printing the HEAD ref\n+\t * But when it is being printed along with other pseudorefs, it makes sense to\n+\t * keep the formatting consistent. So we mask the type to act like a pseudoref.\n+\t */\n+\tif (filter->kind == FILTER_REFS_KIND_MASK && kind == FILTER_REFS_DETACHED_HEAD)\n+\t\tkind = FILTER_REFS_PSEUDOREFS;\n+\telse if (!(kind & filter->kind))\n \t\treturn NULL;\n \n \tif (!filter_pattern_match(filter, refname))\n@@ -3049,7 +3067,13 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \t\t\tret = for_each_fullref_in(\"refs/tags/\", fn, cb_data);\n \t\telse if (filter->kind & FILTER_REFS_REGULAR)\n \t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n-\t\tif (!ret && (filter->kind & FILTER_REFS_DETACHED_HEAD))\n+\n+\t\t/*\n+\t\t * When printing all ref types, HEAD is already included,\n+\t\t * so we don't want to print HEAD again.\n+\t\t */\n+\t\tif (!ret && (filter->kind != FILTER_REFS_KIND_MASK) &&\n+\t\t (filter->kind & FILTER_REFS_DETACHED_HEAD))\n \t\t\thead_ref(fn, cb_data);\n \t}\n \ndiff --git a/ref-filter.h b/ref-filter.h\nindex 5416936800..0ca28d2bba 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -22,7 +22,10 @@\n #define FILTER_REFS_REGULAR (FILTER_REFS_TAGS | FILTER_REFS_BRANCHES | \\\n \t\t\t\t FILTER_REFS_REMOTES | FILTER_REFS_OTHERS)\n #define FILTER_REFS_DETACHED_HEAD 0x0020\n-#define FILTER_REFS_KIND_MASK (FILTER_REFS_REGULAR | FILTER_REFS_DETACHED_HEAD)\n+#define FILTER_REFS_PSEUDOREFS 0x0040\n+#define FILTER_REFS_ROOT_REFS (FILTER_REFS_DETACHED_HEAD | FILTER_REFS_PSEUDOREFS)\n+#define FILTER_REFS_KIND_MASK (FILTER_REFS_REGULAR | FILTER_REFS_DETACHED_HEAD | \\\n+\t\t\t\t FILTER_REFS_PSEUDOREFS)\n \n struct atom_value;\n struct ref_sorting;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex a14f2ad7f4..c23a516ac2 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -364,12 +364,15 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\tbreak;\n \n \t\t/*\n-\t\t * The files backend only lists references contained in\n-\t\t * \"refs/\". We emulate the same behaviour here and thus skip\n-\t\t * all references that don't start with this prefix.\n+\t\t * The files backend only lists references contained in \"refs/\" unless\n+\t\t * the root refs are to be included. We emulate the same behaviour here.\n \t\t */\n-\t\tif (!starts_with(iter->ref.refname, \"refs/\"))\n+\t\tif (!starts_with(iter->ref.refname, \"refs/\") &&\n+\t\t !(iter->flags & DO_FOR_EACH_INCLUDE_ROOT_REFS &&\n+\t\t (is_pseudoref(&iter->refs->base, iter->ref.refname) ||\n+\t\t is_headref(&iter->refs->base, iter->ref.refname)))) {\n \t\t\tcontinue;\n+\t\t}\n \n \t\tif (iter->prefix &&\n \t\t strncmp(iter->prefix, iter->ref.refname, strlen(iter->prefix))) {\ndiff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh\nindex 82f3d1ea0f..948f1bb5f4 100755\n--- a/t/t6302-for-each-ref-filter.sh\n+++ b/t/t6302-for-each-ref-filter.sh\n@@ -31,6 +31,37 @@ test_expect_success 'setup some history and refs' '\n \tgit update-ref refs/odd/spot main\n '\n \n+test_expect_success '--include-root-refs pattern prints pseudorefs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD\n+\tORIG_HEAD\n+\trefs/heads/main\n+\trefs/heads/side\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit update-ref ORIG_HEAD main &&\n+\tgit for-each-ref --format=\"%(refname)\" --include-root-refs >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--include-root-refs with other patterns' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD\n+\tORIG_HEAD\n+\tEOF\n+\tgit update-ref ORIG_HEAD main &&\n+\tgit for-each-ref --format=\"%(refname)\" --include-root-refs \"*HEAD\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'filtering with --points-at' '\n \tcat >expect <<-\\EOF &&\n \trefs/heads/main\n-- \n2.43.GIT\n\n"},{"id":"489252","messageId":"xmqqzfvrrpju.fsf@gitster.g","threadId":"60768","inReplyTo":"20240223100112.44127-1-karthik.188@gmail.com","subject":"Re: [PATCH v5 0/5] for-each-ref: add '--include-root-refs' option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-23T18:41:41Z","receivedAt":"2024-02-23T18:41:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Changes from v4:\n> 1. Fixed erratic whitespace\n> 2. Remove braces from single line block\n> 3. Starting the comments with a capital and also adding more context.\n> 4. Removed a duplicate check.\n\nDoes #4 refer to this removal?\n\n \tif (filter->kind == FILTER_REFS_KIND_MASK && kind == FILTER_REFS_DETACHED_HEAD)\n \t\tkind = FILTER_REFS_PSEUDOREFS;\n \telse if (!(kind & filter->kind))\n \t\treturn NULL;\n \n-\tif (!(kind & filter->kind))\n-\t\treturn NULL;\n-\n \tif (!filter_pattern_match(filter, refname))\n \t\treturn NULL;\n \n\nIf filter->kind is MASK and kind is set to filter detached HEAD, we\nassign to and change the value of kind to FILTER_REFS_PSEUDOREFS,\nso it is unclear if the freestanding \"if kind and filter->kind does\nnot overlap, return NULL\" was redundant or outright buggy.\n\nThe hunk just stood out to me, but I haven't read other parts of the\nseries yet.\n\nThanks.\n"},{"id":"489255","messageId":"xmqqv86frlag.fsf@gitster.g","threadId":"60768","inReplyTo":"xmqqzfvrrpju.fsf@gitster.g","subject":"Re: [PATCH v5 0/5] for-each-ref: add '--include-root-refs' option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-23T20:13:43Z","receivedAt":"2024-02-23T20:13:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> Changes from v4:\n>> 1. Fixed erratic whitespace\n>> 2. Remove braces from single line block\n>> 3. Starting the comments with a capital and also adding more context.\n>> 4. Removed a duplicate check.\n>\n> Does #4 refer to this removal?\n>\n> \tif (filter->kind == FILTER_REFS_KIND_MASK && kind == FILTER_REFS_DETACHED_HEAD)\n> \t\tkind = FILTER_REFS_PSEUDOREFS;\n> \telse if (!(kind & filter->kind))\n> \t\treturn NULL;\n> \n> -\tif (!(kind & filter->kind))\n> -\t\treturn NULL;\n> -\n> \tif (!filter_pattern_match(filter, refname))\n> \t\treturn NULL;\n> \n>\n> If filter->kind is MASK and kind is set to filter detached HEAD, we\n> assign to and change the value of kind to FILTER_REFS_PSEUDOREFS,\n> so it is unclear if the freestanding \"if kind and filter->kind does\n> not overlap, return NULL\" was redundant or outright buggy.\n\nAh, of course this is simply redundant. The assignment to limit the\nkind to PSEUDOREFS happens only when filter->kind has all bits, and\nafter assigning a different non-zero value to kind, (kind & filter->kind)\nwill still overlap, so a freestanding \"if no overlap between kind and\nfilter-kind then return NULL\" will not trigger for such a case.\n\nThanks.\n\n"},{"id":"489464","messageId":"Zd2Ru7LWYyGprvcr@tanuki","threadId":"60768","inReplyTo":"20240223100112.44127-1-karthik.188@gmail.com","subject":"Re: [PATCH v5 0/5] for-each-ref: add '--include-root-refs' option","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-27T07:39:39Z","receivedAt":"2024-02-27T07:39:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Feb 23, 2024 at 11:01:07AM +0100, Karthik Nayak wrote:\n> This is the fifth version of my patch series to print root refs\n> in git-for-each-ref(1).\n> \n> With the introduction of the reftable backend, it becomes ever\n> so important to provide the necessary tooling for printing all refs\n> associated with a worktree.\n> \n> While regular refs stored within the \"refs/\" namespace are currently\n> supported by multiple commands like git-for-each-ref(1),\n> git-show-ref(1). Neither support printing root refs within the worktree.\n> \n> This patch series is a follow up to the RFC/discussion we had earlier on\n> the list [1].\n> \n> The first 4 commits add the required functionality to ensure we can print\n> all refs (regular, pseudo, HEAD). The 5th commit modifies the\n> git-for-each-ref(1) command to add the \"--include-root-refs\" command which\n> will include HEAD and pseudorefs with regular \"refs/\" refs.\n> \n> [1]: https://lore.kernel.org/git/20231221170715.110565-1-karthik.188@gmail.com/#t\n> \n> Changes from v4:\n> 1. Fixed erratic whitespace\n> 2. Remove braces from single line block\n> 3. Starting the comments with a capital and also adding more context.\n> 4. Removed a duplicate check.\n> \n> Thanks for the reviews.\n> \n> Range diff against v4:\n\nThe range-diff looks as expected, so this patch series looks good to me.\nThanks!\n\nPatrick\n\n> 1: 98130a7ad7 ! 1: 6016042965 refs: introduce `is_pseudoref()` and `is_headref()`\n> @@ refs.c: static int is_pseudoref_syntax(const char *refname)\n> +\n> +\tif (ends_with(refname, \"_HEAD\")) {\n> +\t\trefs_resolve_ref_unsafe(refs, refname,\n> -+ \t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n> -+ \t\t\t\t\t&oid, NULL);\n> -+ \t\treturn !is_null_oid(&oid);\n> ++\t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n> ++\t\t\t\t\t&oid, NULL);\n> ++\t\treturn !is_null_oid(&oid);\n> +\t}\n> +\n> +\tfor (i = 0; i < ARRAY_SIZE(irregular_pseudorefs); i++)\n> +\t\tif (!strcmp(refname, irregular_pseudorefs[i])) {\n> +\t\t\trefs_resolve_ref_unsafe(refs, refname,\n> -+ \t\t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n> -+ \t\t\t\t\t\t&oid, NULL);\n> ++\t\t\t\t\t\tRESOLVE_REF_READING | RESOLVE_REF_NO_RECURSE,\n> ++\t\t\t\t\t\t&oid, NULL);\n> +\t\t\treturn !is_null_oid(&oid);\n> +\t\t}\n> +\n> 2: 060ab08af5 = 2: acaa014841 refs: extract out `loose_fill_ref_dir_regular_file()`\n> 3: 40d2375ad9 = 3: f51c5bc307 refs: introduce `refs_for_each_include_root_refs()`\n> 4: b4b9435505 = 4: 7c004db6e7 ref-filter: rename 'FILTER_REFS_ALL' to 'FILTER_REFS_REGULAR'\n> 5: ee99ac41ae ! 5: 53f6c0a6db for-each-ref: add new option to include root refs\n> @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const\n> \t\tfilter.name_patterns = argv;\n> \t}\n> \n> -+\tif (include_root_refs) {\n> ++\tif (include_root_refs)\n> +\t\tflags |= FILTER_REFS_ROOT_REFS;\n> -+\t}\n> +\n> \tfilter.match_as_path = 1;\n> -\tfilter_and_format_refs(&filter, FILTER_REFS_REGULAR, sorting, &format);\n> @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,\n> \t\t\t\t void *cb_data)\n> {\n> +\tif (filter->kind == FILTER_REFS_KIND_MASK) {\n> -+\t\t/* in this case, we want to print all refs including root refs. */\n> ++\t\t/* In this case, we want to print all refs including root refs. */\n> +\t\treturn refs_for_each_include_root_refs(get_main_ref_store(the_repository),\n> +\t\t\t\t\t\t cb, cb_data);\n> +\t}\n> @@ ref-filter.c: static struct ref_array_item *apply_ref_filter(const char *refname\n> \n> \t/* Obtain the current ref kind from filter_ref_kind() and ignore unwanted refs. */\n> \tkind = filter_ref_kind(filter, refname);\n> +-\tif (!(kind & filter->kind))\n> +\n> +\t/*\n> -+\t * When printing HEAD with all other refs, we want to apply the same formatting\n> -+\t * rules as the other refs, so we simply ask it to be treated as a pseudoref.\n> ++\t * Generally HEAD refs are printed with special description denoting a rebase,\n> ++\t * detached state and so forth. This is useful when only printing the HEAD ref\n> ++\t * But when it is being printed along with other pseudorefs, it makes sense to\n> ++\t * keep the formatting consistent. So we mask the type to act like a pseudoref.\n> +\t */\n> +\tif (filter->kind == FILTER_REFS_KIND_MASK && kind == FILTER_REFS_DETACHED_HEAD)\n> +\t\tkind = FILTER_REFS_PSEUDOREFS;\n> +\telse if (!(kind & filter->kind))\n> -+\t\treturn NULL;\n> -+\n> - \tif (!(kind & filter->kind))\n> \t\treturn NULL;\n> \n> + \tif (!filter_pattern_match(filter, refname))\n> @@ ref-filter.c: static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n> \t\t\tret = for_each_fullref_in(\"refs/tags/\", fn, cb_data);\n> \t\telse if (filter->kind & FILTER_REFS_REGULAR)\n> \n> \n> Karthik Nayak (5):\n> refs: introduce `is_pseudoref()` and `is_headref()`\n> refs: extract out `loose_fill_ref_dir_regular_file()`\n> refs: introduce `refs_for_each_include_root_refs()`\n> ref-filter: rename 'FILTER_REFS_ALL' to 'FILTER_REFS_REGULAR'\n> for-each-ref: add new option to include root refs\n> \n> Documentation/git-for-each-ref.txt | 5 +-\n> builtin/for-each-ref.c | 10 ++-\n> ref-filter.c | 30 ++++++-\n> ref-filter.h | 7 +-\n> refs.c | 48 +++++++++++\n> refs.h | 9 ++\n> refs/files-backend.c | 127 +++++++++++++++++++++--------\n> refs/refs-internal.h | 6 ++\n> refs/reftable-backend.c | 11 ++-\n> t/t6302-for-each-ref-filter.sh | 31 +++++++\n> 10 files changed, 237 insertions(+), 47 deletions(-)\n> \n> -- \n> 2.43.GIT\n> \n"},{"id":"489547","messageId":"xmqqr0gxvoe6.fsf@gitster.g","threadId":"60768","inReplyTo":"Zd2Ru7LWYyGprvcr@tanuki","subject":"Re: [PATCH v5 0/5] for-each-ref: add '--include-root-refs' option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-27T16:54:25Z","receivedAt":"2024-02-27T16:54:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> Changes from v4:\n>> 1. Fixed erratic whitespace\n>> 2. Remove braces from single line block\n>> 3. Starting the comments with a capital and also adding more context.\n>> 4. Removed a duplicate check.\n>> \n>> Thanks for the reviews.\n>> \n>> Range diff against v4:\n>\n> The range-diff looks as expected, so this patch series looks good to me.\n> Thanks!\n>\n> Patrick\n\nThanks, let's mark the topic for 'next'.\n"}]}