{"thread":{"id":"61565","subject":"[GSoC][PATCH 0/2] ref consistency check infra setup","startedAt":"2024-05-30T12:28:06Z","lastAt":"2024-10-06T10:01:11Z","messageCount":282,"participants":["shejialuo","Junio C Hamano","Eric Sunshine","Patrick Steinhardt","Karthik Nayak","jialuo she","Justin Tobler","Teng Long"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"495932","messageId":"20240530122753.1114818-1-shejialuo@gmail.com","threadId":"61565","inReplyTo":null,"subject":"[GSoC][PATCH 0/2] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-05-30T12:27:51Z","receivedAt":"2024-05-30T12:28:06Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Hi,\n\nThis patch mainly focuses on two things:\n\n1. Setup the infrastructure of the ref consistency check.\n2. Implement ref name and content checks for files backend to verify the\ncorrectness about the infrastructure with corresponding unit test.\n\nSee https://github.com/shejialuo/git/pull/1 for CI.\n\nThanks,\nJialuo\n\nshejialuo (2):\n refs: setup ref consistency check infrastructure\n refs: add name and content check for file backend\n\n builtin/fsck.c | 5 ++\n refs.c | 5 ++\n refs.h | 5 ++\n refs/files-backend.c | 119 ++++++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 7 +++\n refs/refs-internal.h | 4 ++\n refs/reftable-backend.c | 7 +++\n t/t0602-reffiles-fsck.sh | 132 +++++++++++++++++++++++++++++++++++++++\n 8 files changed, 283 insertions(+), 1 deletion(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\n-- \n2.45.1\n\n"},{"id":"495933","messageId":"20240530122753.1114818-2-shejialuo@gmail.com","threadId":"61565","inReplyTo":"20240530122753.1114818-1-shejialuo@gmail.com","subject":"[GSoC][PATCH 1/2] refs: setup ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-05-30T12:27:52Z","receivedAt":"2024-05-30T12:28:19Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Add a new interface `fsck_refs_fn` for the `ref_storage_be`. Implement\ndummy method for files and reftable backends.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 5 +++++\n refs.c | 5 +++++\n refs.h | 5 +++++\n refs/files-backend.c | 9 ++++++++-\n refs/packed-backend.c | 7 +++++++\n refs/refs-internal.h | 4 ++++\n refs/reftable-backend.c | 7 +++++++\n 7 files changed, 41 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..65a26e2d1b 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -1065,6 +1065,11 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n+\tif (refs_fsck(get_main_ref_store(the_repository))) {\n+\t\terror(\"ref database is corrupt\");\n+\t\terrors_found |= ERROR_REFS;\n+\t}\n+\n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \ndiff --git a/refs.c b/refs.c\nindex 8260c27cde..5ac61d551c 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -316,6 +316,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs)\n+{\n+\treturn refs->be->fsck(refs);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex 34568ee1fb..2799820c40 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -544,6 +544,11 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Return 0 iff all refs in filesystem are consistent.\n+*/\n+int refs_fsck(struct ref_store *refs);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 5f3089d947..b6147c588b 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3299,6 +3299,11 @@ static int files_init_db(struct ref_store *ref_store,\n \treturn 0;\n }\n \n+static int files_fsck(struct ref_store *ref_store)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_create,\n@@ -3322,5 +3327,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a937e7dbfc..0617321634 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1704,6 +1704,11 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_create,\n@@ -1728,4 +1733,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 53a6c5d842..ef697bf3bf 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -675,6 +675,8 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -700,6 +702,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 1af86bbdec..f3f85cd2f0 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2167,6 +2167,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2191,4 +2196,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.45.1\n\n"},{"id":"495934","messageId":"20240530122753.1114818-3-shejialuo@gmail.com","threadId":"61565","inReplyTo":"20240530122753.1114818-1-shejialuo@gmail.com","subject":"[GSoC][PATCH 2/2] refs: add name and content check for file backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-05-30T12:27:53Z","receivedAt":"2024-05-30T12:28:27Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The existing git-fsck does not fully check bad format ref name such as\nstandalone '@' or name ending with \".lock\". And also `git-fsck` does not\nfully check ref content, the following contents will not be reported by\nthe original git-fsck command.\n\n1. \"c71dba5654404b9b0d378a6cbff1b743b9d31a34 garbage\": with trailing\n characters.\n2. \"c71dba5654404b9b0d378a6cbff1b743b9d31a34 \": with trailing empty\n characters.\n3. \"C71DBA5654404b9b0d378a6cbff1b743b9d31A34\": with uppercase hex.\n4. \"z71dba5654404b9b0d378a6cbff1b743b9d31a34\": with not hex character.\n\nProvide functionality to support such consistency checks for branch and\ntag refs and add a new unit test to verify this functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs/files-backend.c | 112 ++++++++++++++++++++++++++++++++-\n t/t0602-reffiles-fsck.sh | 132 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 243 insertions(+), 1 deletion(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex b6147c588b..3c811f5c03 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3299,11 +3299,121 @@ static int files_init_db(struct ref_store *ref_store,\n \treturn 0;\n }\n \n-static int files_fsck(struct ref_store *ref_store)\n+typedef int (*files_fsck_refs_fn)(const char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter);\n+\n+static int files_fsck_refs_name(const char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n {\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\terror(_(\"%s/%s: invalid refname format\"), refs_check_dir, iter->basename);\n+\t\treturn -1;\n+\t}\n+\n \treturn 0;\n }\n \n+static int files_fsck_refs_content(const char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf ref_content = STRBUF_INIT;\n+\n+\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n+\t\terror(_(\"%s/%s: unable to read the ref\"), refs_check_dir, iter->basename);\n+\t\tgoto clean;\n+\t}\n+\n+\t/*\n+\t * Case 1: check if the ref content length is valid and the last\n+\t * character is a newline.\n+\t */\n+\tif (ref_content.len != the_hash_algo->hexsz + 1 ||\n+\t\t\tref_content.buf[ref_content.len - 1] != '\\n') {\n+\t\tgoto failure;\n+\t}\n+\n+\t/*\n+\t * Case 2: the content should be range of [0-9a-f].\n+\t */\n+\tfor (size_t i = 0; i < the_hash_algo->hexsz; i++) {\n+\t\tif (!isdigit(ref_content.buf[i]) &&\n+\t\t\t\t(ref_content.buf[i] < 'a' || ref_content.buf[i] > 'f')) {\n+\t\t\tgoto failure;\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&ref_content);\n+\treturn 0;\n+\n+failure:\n+\terror(_(\"%s/%s: invalid ref content\"), refs_check_dir, iter->basename);\n+\n+clean:\n+\tstrbuf_release(&ref_content);\n+\treturn -1;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t\tconst char* refs_check_dir,\n+\t\t\t\tfiles_fsck_refs_fn *fsck_refs_fns)\n+{\n+\tstruct dir_iterator *iter;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\tint iter_status;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", ref_store->gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\n+\t/*\n+\t * The current implementation does not care about the worktree, the worktree\n+\t * may have no refs/heads or refs/tags directory. Simply return 0 now.\n+\t*/\n+\tif (!iter) {\n+\t\treturn 0;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode)) {\n+\t\t\tfor (files_fsck_refs_fn *fsck_refs_fn = fsck_refs_fns;\n+\t\t\t\t\t*fsck_refs_fn; fsck_refs_fn++) {\n+\t\t\t\tret |= (*fsck_refs_fn)(refs_check_dir, iter);\n+\t\t\t}\n+\t\t} else {\n+\t\t\terror(_(\"unexpected file type for '%s'\"), iter->basename);\n+\t\t\tret = -1;\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE) {\n+\t\tret = -1;\n+\t\terror(_(\"failed to iterate over '%s'\"), sb.buf);\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\n+\treturn ret;\n+}\n+\n+static int files_fsck(struct ref_store *ref_store)\n+{\n+\tint ret = 0;\n+\n+\tfiles_fsck_refs_fn fsck_refs_fns[] = {\n+\t\tfiles_fsck_refs_name,\n+\t\tfiles_fsck_refs_content,\n+\t\tNULL\n+\t};\n+\n+\tret = files_fsck_refs(ref_store, \"refs/heads\",fsck_refs_fns)\n+\t | files_fsck_refs(ref_store, \"refs/tags\", fsck_refs_fns);\n+\n+\treturn ret;\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_create,\ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..1c6c3804ff\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,132 @@\n+#!/bin/bash\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/.branch-1: invalid refname format\n+\t\terror: ref database is corrupt\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1.lock: invalid refname format\n+\t\terror: ref database is corrupt\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/@: invalid refname format\n+\t\terror: ref database is corrupt\n+\t\tEOF\n+\t\trm $branch_dir_prefix/@ &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse branch-1)\" > $branch_dir_prefix/branch-1-garbage &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-1-garbage: invalid ref content\n+\t\terror: ref database is corrupt\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse tag-1)\" > $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1-garbage: invalid ref content\n+\t\terror: ref database is corrupt\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s \" \"$(git rev-parse tag-2)\" > $tag_dir_prefix/tag-2-garbage &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-2-garbage: invalid ref content\n+\t\terror: ref database is corrupt\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\ttr -d \"\\n\" < $branch_dir_prefix/branch-1 > $branch_dir_prefix/branch-1-without-newline &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-1-without-newline: invalid ref content\n+\t\terror: ref database is corrupt\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-1-without-newline &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\ttr \"[:lower:]\" \"[:upper:]\" < $branch_dir_prefix/branch-2 > $branch_dir_prefix/branch-2-upper &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-upper: invalid ref content\n+\t\terror: ref database is corrupt\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-upper &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_done\n-- \n2.45.1\n\n"},{"id":"496007","messageId":"xmqqfrtyjc0w.fsf@gitster.g","threadId":"61565","inReplyTo":"20240530122753.1114818-2-shejialuo@gmail.com","subject":"Re: [GSoC][PATCH 1/2] refs: setup ref consistency check infrastructure","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-31T14:23:27Z","receivedAt":"2024-05-31T14:23:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"shejialuo writes:\n\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 5f3089d947..b6147c588b 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -3299,6 +3299,11 @@ static int files_init_db(struct ref_store *ref_store,\n> \treturn 0;\n> }\n> \n> +static int files_fsck(struct ref_store *ref_store)\n> +{\n> +\treturn 0;\n> +}\n> +\n> struct ref_storage_be refs_be_files = {\n> \t.name = \"files\",\n> \t.init = files_ref_store_create,\n> @@ -3322,5 +3327,7 @@ struct ref_storage_be refs_be_files = {\n> \t.reflog_exists = files_reflog_exists,\n> \t.create_reflog = files_create_reflog,\n> \t.delete_reflog = files_delete_reflog,\n> -\t.reflog_expire = files_reflog_expire\n> +\t.reflog_expire = files_reflog_expire,\n> +\n> +\t.fsck = files_fsck,\n\nWhat is the extra blank line doing there? It makes reader wonder\nwhy the .fsck member is somehow very special and different from\nothers. Is there a valid reason to single it out (and no, \"yes this\nis special because I invented it\" does not count as a valid reason)?\n\nThe same comment applies to a few other places in this patch.\n"},{"id":"496008","messageId":"xmqqa5k6jc0s.fsf@gitster.g","threadId":"61565","inReplyTo":"20240530122753.1114818-3-shejialuo@gmail.com","subject":"Re: [GSoC][PATCH 2/2] refs: add name and content check for file backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-31T14:23:31Z","receivedAt":"2024-05-31T14:23:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"shejialuo writes:\n\n> +static int files_fsck_refs_content(const char *refs_check_dir,\n> +\t\t\t\tstruct dir_iterator *iter)\n> +{\n> +\tstruct strbuf ref_content = STRBUF_INIT;\n\nThe caller makes sure that this gets called only on a regular file,\nso ...\n\n> +\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n\n... the use of strbuf_read_file() is fine (i.e. we do not have to\nworry about the iter->path to be pointing at a symbolic link) here.\n\n> +\t\terror(_(\"%s/%s: unable to read the ref\"), refs_check_dir, iter->basename);\n> +\t\tgoto clean;\n> +\t}\n> +\n> +\t/*\n> +\t * Case 1: check if the ref content length is valid and the last\n> +\t * character is a newline.\n> +\t */\n> +\tif (ref_content.len != the_hash_algo->hexsz + 1 ||\n> +\t\t\tref_content.buf[ref_content.len - 1] != '\\n') {\n\nFunny indentation.\n\n\tif (ref_content.len != the_hash_algo->hexsz + 1 ||\n\t ref_content.buf[ref_content.len - 1] != '\\n') {\n\nAlso, we do not want {braces} around a single statement block.\n\nIn any case, the two checks are good ONLY for regular refs and not\nfor symbolic refs. The users are free to create symbolic refs next\nto their branches, e.g. here is a way to say, \"among the maintenance\ntracks maint-2.30, maint-2.31, ... maint-2.44, maint-2.45, what I\nconsider the primary maintenance track is currently maint-2.45\":\n\n $ git symbolic-ref refs/heads/maint\n refs/heads/maint-2.45\n\nand if your directory walk encounters such a symbolic ref, your\nstrbuf_read_file() will read something like:\n\n $ cat .git/refs/heads/maint\n ref: refs/heads/maint-2.45\n\nAlso, the caller also needs to be prepared to find a real symbolic\nlink that is used as a symbolic ref, but for them you'd need to do a\nreadlink() and the expected contents would not have \"ref: \" prefix,\nso the code to implement actual check would have to be different.\nThe way how refs/files-backend.c:read_ref_internal() handles S_ISLNK\nwould be illuminating.\n\n> +\t\tgoto failure;\n> +\t}\n> +\t/*\n> +\t * Case 2: the content should be range of [0-9a-f].\n> +\t */\n> +\tfor (size_t i = 0; i < the_hash_algo->hexsz; i++) {\n> +\t\tif (!isdigit(ref_content.buf[i]) &&\n> +\t\t\t\t(ref_content.buf[i] < 'a' || ref_content.buf[i] > 'f')) {\n> +\t\t\tgoto failure;\n\nI do not think it is a good idea to suddenly redefine what a valid\nway to write object names in a loose ref file after ~20 years.\nGiven the popularity of Git, I would not be surprised at all if a\nthird-party tool or two have been writing their own refs with\nuppercase hexadecimal and we have been happily using them as\neverybody expects. It is a good idea to be strict when you are a\nproducer, and we do write object names always in lowercase hex, but\nwe are lenient when consuming what is possibly written by others.\nAs long as the hexadecimal string names an existing object\n(otherwise we have a dangling ref which would be another kind of\nerror, I presume), it should be at most FSCK_WARN when it does not\nlook like what _we_ wrote (e.g., if it uses uppercase hex, or if it\nhas extra bytes after the 40-hex (or 64-hex) other than the final\nLF). If it is shorter, of course that is an outright FSCK_ERROR.\n\nHow well does this interact with the fsck error levels (aka\nfsck_msg_type), by the way? It should be made to work well if the\ncurrent design does not.\n\n> +\t\t}\n> +\t}\n> +\n> +\tstrbuf_release(&ref_content);\n> +\treturn 0;\n> +\n> +failure:\n> +\terror(_(\"%s/%s: invalid ref content\"), refs_check_dir, iter->basename);\n\nIn addition, when we honor fsck error levels, such an unconditional\ncall to \"error\" may become an issue. If the user says a certain\nkind of anomaly is tolerated by setting a specific finding to\nFSCK_IGNORE, we should be silent about our finding.\n\n> +clean:\n> +\tstrbuf_release(&ref_content);\n> +\treturn -1;\n> +}\n> +\n> +static int files_fsck_refs(struct ref_store *ref_store,\n> +\t\t\t\tconst char* refs_check_dir,\n> +\t\t\t\tfiles_fsck_refs_fn *fsck_refs_fns)\n> +{\n> +\tstruct dir_iterator *iter;\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\tint ret = 0;\n> +\tint iter_status;\n> +\n> +\tstrbuf_addf(&sb, \"%s/%s\", ref_store->gitdir, refs_check_dir);\n> +\n> +\titer = dir_iterator_begin(sb.buf, 0);\n> +\n> +\t/*\n> +\t * The current implementation does not care about the worktree, the worktree\n> +\t * may have no refs/heads or refs/tags directory. Simply return 0 now.\n> +\t*/\n> +\tif (!iter) {\n> +\t\treturn 0;\n> +\t}\n> +\n> +\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n> +\t\tif (S_ISDIR(iter->st.st_mode)) {\n> +\t\t\tcontinue;\n> +\t\t} else if (S_ISREG(iter->st.st_mode)) {\n> +\t\t\tfor (files_fsck_refs_fn *fsck_refs_fn = fsck_refs_fns;\n> +\t\t\t\t\t*fsck_refs_fn; fsck_refs_fn++) {\n\nFunny indentation (again---the patch 2/2 has funny \"two extra HT\"\nindentation style all over the place). Write it like so:\n\n\t\t\tfor (files_fsck_refs_fn *fn = fsck_refs_fns;\n\t\t\t fn;\n fn++)\n\n> +\t\t\t\tret |= (*fsck_refs_fn)(refs_check_dir, iter);\n\nA pointer to a function does not need to be dereferenced explicitly\nlike so. Also, writing\n\n\t\t\t\tif (fn(refs_check_dir, iter))\n\t\t\t\t\tret = -1;\n\nwould be more consistent with the rest of the code around here,\nwhich does not OR int ret but explicitly set it to -1 when any\nhelper function detects an error.\n\n> +\t\t\t}\n> +\t\t} else {\n> +\t\t\terror(_(\"unexpected file type for '%s'\"), iter->basename);\n> +\t\t\tret = -1;\n\nThis is wrong. A symbolic link is a valid symbolic ref, even though\nwe no longer create such a symbolic ref by default.\n\n> +\t\t}\n> +\t}\n> +\n> +\tif (iter_status != ITER_DONE) {\n> +\t\tret = -1;\n> +\t\terror(_(\"failed to iterate over '%s'\"), sb.buf);\n> +\t}\n> +\n> +\tstrbuf_release(&sb);\n> +\n> +\treturn ret;\n> +}\n> +\n> +static int files_fsck(struct ref_store *ref_store)\n> +{\n> +\tint ret = 0;\n> +\n> +\tfiles_fsck_refs_fn fsck_refs_fns[] = {\n> +\t\tfiles_fsck_refs_name,\n> +\t\tfiles_fsck_refs_content,\n> +\t\tNULL\n> +\t};\n> +\n> +\tret = files_fsck_refs(ref_store, \"refs/heads\",fsck_refs_fns)\n\nMissing SP after a comma.\n\n> +\t | files_fsck_refs(ref_store, \"refs/tags\", fsck_refs_fns);\n\nWhy only these two hierarchies? Shouldn't it also be checking the\nremote tracking branches and notes?\n\n> +\treturn ret;\n> +}\n> +\n> struct ref_storage_be refs_be_files = {\n> \t.name = \"files\",\n> \t.init = files_ref_store_create,\n"},{"id":"496016","messageId":"ZlnqtRgBsGOZ_3Uy@ArchLinux","threadId":"61565","inReplyTo":"xmqqfrtyjc0w.fsf@gitster.g","subject":"Re: [GSoC][PATCH 1/2] refs: setup ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-05-31T15:20:21Z","receivedAt":"2024-05-31T15:20:25Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Fri, May 31, 2024 at 07:23:27AM -0700, Junio C Hamano wrote:\n> \n> What is the extra blank line doing there? It makes reader wonder\n> why the .fsck member is somehow very special and different from\n> others. Is there a valid reason to single it out (and no, \"yes this\n> is special because I invented it\" does not count as a valid reason)?\n> \n> The same comment applies to a few other places in this patch.\n\nThe interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It is organized as the five parts at now:\n\n 1. The name and the initialization interfaces.\n 2. The ref transaction interfaces.\n 3. The ref internal interfaces (pack, rename and copy).\n 4. The ref filesystem interfaces.\n 5. The reflog related interfaces.\n\nI firstly thought that we could just add a new function into the ref\ninternal interfaces part with the name `check_refs_fn`. However, the\nultimate goal is not only to achieve consistency check of refs but also\nconsistency check of reflogs. So it's not suitable to name this\ninterface to `check_refs_fn`.\n\nIn order to keep consistent with the git-fsck, we decide to name this\ninterface to `fsck_fn`. This semantic cannot be grouped into any above\nfive categories in my view. So I deliberately add a blank line to\nindicate its independence.\n"},{"id":"496019","messageId":"Zln0ErSydQMZSMhN@ArchLinux","threadId":"61565","inReplyTo":"xmqqa5k6jc0s.fsf@gitster.g","subject":"Re: [GSoC][PATCH 2/2] refs: add name and content check for file backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-05-31T16:00:18Z","receivedAt":"2024-05-31T16:00:24Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Fri, May 31, 2024 at 07:23:31AM -0700, Junio C Hamano wrote:\n> shejialuo writes:\n> \n> > +static int files_fsck_refs_content(const char *refs_check_dir,\n> > +\t\t\t\tstruct dir_iterator *iter)\n> > +{\n> > +\tstruct strbuf ref_content = STRBUF_INIT;\n> \n> The caller makes sure that this gets called only on a regular file,\n> so ...\n> \n> > +\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n> \n> ... the use of strbuf_read_file() is fine (i.e. we do not have to\n> worry about the iter->path to be pointing at a symbolic link) here.\n> \n> > +\t\terror(_(\"%s/%s: unable to read the ref\"), refs_check_dir, iter->basename);\n> > +\t\tgoto clean;\n> > +\t}\n> > +\n> > +\t/*\n> > +\t * Case 1: check if the ref content length is valid and the last\n> > +\t * character is a newline.\n> > +\t */\n> > +\tif (ref_content.len != the_hash_algo->hexsz + 1 ||\n> > +\t\t\tref_content.buf[ref_content.len - 1] != '\\n') {\n> \n> Funny indentation.\n> \n> \tif (ref_content.len != the_hash_algo->hexsz + 1 ||\n> \t ref_content.buf[ref_content.len - 1] != '\\n') {\n> \n> Also, we do not want {braces} around a single statement block.\n> \n\nI will clean the code. I am used to applying one extra HT.\n\n> In any case, the two checks are good ONLY for regular refs and not\n> for symbolic refs. The users are free to create symbolic refs next\n> to their branches, e.g. here is a way to say, \"among the maintenance\n> tracks maint-2.30, maint-2.31, ... maint-2.44, maint-2.45, what I\n> consider the primary maintenance track is currently maint-2.45\":\n> \n> $ git symbolic-ref refs/heads/maint\n> refs/heads/maint-2.45\n> \n> and if your directory walk encounters such a symbolic ref, your\n> strbuf_read_file() will read something like:\n> \n> $ cat .git/refs/heads/maint\n> ref: refs/heads/maint-2.45\n> \n\nYes, currently I intentionally ignore the symbolic ref. My original idea\nwas to make this patch as small as possible to set up the infrastructure\nof the ref consistence check.\n\nSo this is why I only choose \"refs/heads\" and \"refs/tags\" at current.\nThe \"refs/remote\" will contain \"HEAD\" which is a symbolic ref. So I\nignore \"refs/remote\".\n\nI just thought git would ONLY have one symbolic ref \"HEAD\". I will add\nmore patches in the next version to support the symbolic ref checks.\n\n> Also, the caller also needs to be prepared to find a real symbolic\n> link that is used as a symbolic ref, but for them you'd need to do a\n> readlink() and the expected contents would not have \"ref: \" prefix,\n> so the code to implement actual check would have to be different.\n> The way how refs/files-backend.c:read_ref_internal() handles S_ISLNK\n> would be illuminating.\n> \n\nI will handle this in the next version. However, I am a little curious\nabout the reason why people use a real symbolic in git. git has\nprovided the symbolic ref which should be more consistent and generic\nagainst the OS.\n\n> > +\t\tgoto failure;\n> > +\t}\n> > +\t/*\n> > +\t * Case 2: the content should be range of [0-9a-f].\n> > +\t */\n> > +\tfor (size_t i = 0; i < the_hash_algo->hexsz; i++) {\n> > +\t\tif (!isdigit(ref_content.buf[i]) &&\n> > +\t\t\t\t(ref_content.buf[i] < 'a' || ref_content.buf[i] > 'f')) {\n> > +\t\t\tgoto failure;\n> \n> I do not think it is a good idea to suddenly redefine what a valid\n> way to write object names in a loose ref file after ~20 years.\n> Given the popularity of Git, I would not be surprised at all if a\n> third-party tool or two have been writing their own refs with\n> uppercase hexadecimal and we have been happily using them as\n> everybody expects. It is a good idea to be strict when you are a\n> producer, and we do write object names always in lowercase hex, but\n> we are lenient when consuming what is possibly written by others.\n> As long as the hexadecimal string names an existing object\n> (otherwise we have a dangling ref which would be another kind of\n> error, I presume), it should be at most FSCK_WARN when it does not\n> look like what _we_ wrote (e.g., if it uses uppercase hex, or if it\n> has extra bytes after the 40-hex (or 64-hex) other than the final\n> LF). If it is shorter, of course that is an outright FSCK_ERROR.\n> \n\nYes, actually I have done some experiments using git-fsck. I am really\nsurprised by the fact that the few bytes will cause FSCK_ERROR. But the\nsituation with extra bytes are OK. I don't know this design is\nintentional.\n\n> How well does this interact with the fsck error levels (aka\n> fsck_msg_type), by the way? It should be made to work well if the\n> current design does not.\n> \n\nUnfortunately, I totally ignored this. I will try to implement the fsck\nerror levels in the next version.\n\n> > +\t\t}\n> > +\t}\n> > +\n> > +\tstrbuf_release(&ref_content);\n> > +\treturn 0;\n> > +\n> > +failure:\n> > +\terror(_(\"%s/%s: invalid ref content\"), refs_check_dir, iter->basename);\n> \n> In addition, when we honor fsck error levels, such an unconditional\n> call to \"error\" may become an issue. If the user says a certain\n> kind of anomaly is tolerated by setting a specific finding to\n> FSCK_IGNORE, we should be silent about our finding.\n> \n\nYes, I will add the code to support this.\n\n> > +clean:\n> > +\tstrbuf_release(&ref_content);\n> > +\treturn -1;\n> > +}\n> > +\n> > +static int files_fsck_refs(struct ref_store *ref_store,\n> > +\t\t\t\tconst char* refs_check_dir,\n> > +\t\t\t\tfiles_fsck_refs_fn *fsck_refs_fns)\n> > +{\n> > +\tstruct dir_iterator *iter;\n> > +\tstruct strbuf sb = STRBUF_INIT;\n> > +\tint ret = 0;\n> > +\tint iter_status;\n> > +\n> > +\tstrbuf_addf(&sb, \"%s/%s\", ref_store->gitdir, refs_check_dir);\n> > +\n> > +\titer = dir_iterator_begin(sb.buf, 0);\n> > +\n> > +\t/*\n> > +\t * The current implementation does not care about the worktree, the worktree\n> > +\t * may have no refs/heads or refs/tags directory. Simply return 0 now.\n> > +\t*/\n> > +\tif (!iter) {\n> > +\t\treturn 0;\n> > +\t}\n> > +\n> > +\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n> > +\t\tif (S_ISDIR(iter->st.st_mode)) {\n> > +\t\t\tcontinue;\n> > +\t\t} else if (S_ISREG(iter->st.st_mode)) {\n> > +\t\t\tfor (files_fsck_refs_fn *fsck_refs_fn = fsck_refs_fns;\n> > +\t\t\t\t\t*fsck_refs_fn; fsck_refs_fn++) {\n> \n> Funny indentation (again---the patch 2/2 has funny \"two extra HT\"\n> indentation style all over the place). Write it like so:\n> \n> \t\t\tfor (files_fsck_refs_fn *fn = fsck_refs_fns;\n> \t\t\t fn;\n> fn++)\n> \n> > +\t\t\t\tret |= (*fsck_refs_fn)(refs_check_dir, iter);\n> \n> A pointer to a function does not need to be dereferenced explicitly\n> like so. Also, writing\n> \n> \t\t\t\tif (fn(refs_check_dir, iter))\n> \t\t\t\t\tret = -1;\n> \n> would be more consistent with the rest of the code around here,\n> which does not OR int ret but explicitly set it to -1 when any\n> helper function detects an error.\n> \n\nI will handle this in the next version.\n\n> > +\t\t\t}\n> > +\t\t} else {\n> > +\t\t\terror(_(\"unexpected file type for '%s'\"), iter->basename);\n> > +\t\t\tret = -1;\n> \n> This is wrong. A symbolic link is a valid symbolic ref, even though\n> we no longer create such a symbolic ref by default.\n> \n\nAs above said, I have thought git should not handle this situation. I\nwill implement support for real symbolic file.\n\n> > +\t\t}\n> > +\t}\n> > +\n> > +\tif (iter_status != ITER_DONE) {\n> > +\t\tret = -1;\n> > +\t\terror(_(\"failed to iterate over '%s'\"), sb.buf);\n> > +\t}\n> > +\n> > +\tstrbuf_release(&sb);\n> > +\n> > +\treturn ret;\n> > +}\n> > +\n> > +static int files_fsck(struct ref_store *ref_store)\n> > +{\n> > +\tint ret = 0;\n> > +\n> > +\tfiles_fsck_refs_fn fsck_refs_fns[] = {\n> > +\t\tfiles_fsck_refs_name,\n> > +\t\tfiles_fsck_refs_content,\n> > +\t\tNULL\n> > +\t};\n> > +\n> > +\tret = files_fsck_refs(ref_store, \"refs/heads\",fsck_refs_fns)\n> \n> Missing SP after a comma.\n> \n\nI will fix it in the next version.\n\n> > +\t | files_fsck_refs(ref_store, \"refs/tags\", fsck_refs_fns);\n> \n> Why only these two hierarchies? Shouldn't it also be checking the\n> remote tracking branches and notes?\n> \n\nI have explained above, because I intentionally ignored the symbolic ref\nin this patch, so I don't handle \"refs/remote\" and notes. I guess I will\nhandle these situations in the next patch.\n\n> > +\treturn ret;\n> > +}\n> > +\n> > struct ref_storage_be refs_be_files = {\n> > \t.name = \"files\",\n> > \t.init = files_ref_store_create,\n\nThanks for your dedicated reviews. I have a more clean direction to\nsetup the consistence check for file backend.\n\nThanks,\nJialuo\n"},{"id":"496026","messageId":"CAPig+cRpwESDkZS3yFN8W3y8yugauecMbQC26rvYjVijH=2YLA@mail.gmail.com","threadId":"61565","inReplyTo":"xmqqa5k6jc0s.fsf@gitster.g","subject":"Re: [GSoC][PATCH 2/2] refs: add name and content check for file backend","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-05-31T18:31:42Z","receivedAt":"2024-05-31T18:31:55Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, May 31, 2024 at 10:24 AM Junio C Hamano wrote:\n> shejialuo writes:\n> > + error(_(\"unexpected file type for '%s'\"), iter->basename);\n> > + ret = -1;\n>\n> This is wrong. A symbolic link is a valid symbolic ref, even though\n> we no longer create such a symbolic ref by default.\n\nVery minor additional note: error() unconditionally returns -1 in\norder to support this pattern:\n\n ret = error(_(\"...\"));\n"},{"id":"496061","messageId":"Zl1vJNxOFh-Abj5Q@framework","threadId":"61565","inReplyTo":"20240530122753.1114818-3-shejialuo@gmail.com","subject":"Re: [GSoC][PATCH 2/2] refs: add name and content check for file backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-06-03T07:22:25Z","receivedAt":"2024-06-03T07:22:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, May 30, 2024 at 08:27:53PM +0800, shejialuo wrote:\n> The existing git-fsck does not fully check bad format ref name such as\n> standalone '@' or name ending with \".lock\". And also `git-fsck` does not\n> fully check ref content, the following contents will not be reported by\n> the original git-fsck command.\n> \n> 1. \"c71dba5654404b9b0d378a6cbff1b743b9d31a34 garbage\": with trailing\n> characters.\n> 2. \"c71dba5654404b9b0d378a6cbff1b743b9d31a34 \": with trailing empty\n> characters.\n> 3. \"C71DBA5654404b9b0d378a6cbff1b743b9d31A34\": with uppercase hex.\n> 4. \"z71dba5654404b9b0d378a6cbff1b743b9d31a34\": with not hex character.\n> \n> Provide functionality to support such consistency checks for branch and\n> tag refs and add a new unit test to verify this functionality.\n\nI would recommend to only introduce one check per commit. This commit\nintroduces multiple different checks to the files backend, which makes\nit harder to review.\n\n[snip]\n> +static int files_fsck_refs_content(const char *refs_check_dir,\n> +\t\t\t\tstruct dir_iterator *iter)\n> +{\n> +\tstruct strbuf ref_content = STRBUF_INIT;\n> +\n> +\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n> +\t\terror(_(\"%s/%s: unable to read the ref\"), refs_check_dir, iter->basename);\n> +\t\tgoto clean;\n> +\t}\n> +\n> +\t/*\n> +\t * Case 1: check if the ref content length is valid and the last\n> +\t * character is a newline.\n> +\t */\n> +\tif (ref_content.len != the_hash_algo->hexsz + 1 ||\n> +\t\t\tref_content.buf[ref_content.len - 1] != '\\n') {\n> +\t\tgoto failure;\n> +\t}\n> +\n> +\t/*\n> +\t * Case 2: the content should be range of [0-9a-f].\n> +\t */\n> +\tfor (size_t i = 0; i < the_hash_algo->hexsz; i++) {\n> +\t\tif (!isdigit(ref_content.buf[i]) &&\n> +\t\t\t\t(ref_content.buf[i] < 'a' || ref_content.buf[i] > 'f')) {\n> +\t\t\tgoto failure;\n> +\t\t}\n> +\t}\n> +\n> +\tstrbuf_release(&ref_content);\n> +\treturn 0;\n> +\n> +failure:\n> +\terror(_(\"%s/%s: invalid ref content\"), refs_check_dir, iter->basename);\n> +\n> +clean:\n> +\tstrbuf_release(&ref_content);\n> +\treturn -1;\n\nCan we reuse `parse_loose_ref_contents()` to get rid of much of the\nduplication here? That function would need to learn to indicate to the\ncaller that there is trailing data, but other than that it's already as\nstrict as we want it to be.\n\nOther than that, Junio has already added quite a bunch of comments, so\nI'll refrain from repeating them here.\n\nThanks!\n\nPatrick\n"},{"id":"496062","messageId":"Zl1vO7RPL0rUEvzX@framework","threadId":"61565","inReplyTo":"20240530122753.1114818-2-shejialuo@gmail.com","subject":"Re: [GSoC][PATCH 1/2] refs: setup ref consistency check infrastructure","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-06-03T07:22:35Z","receivedAt":"2024-06-03T07:22:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, May 30, 2024 at 08:27:52PM +0800, shejialuo wrote:\n> Add a new interface `fsck_refs_fn` for the `ref_storage_be`. Implement\n> dummy method for files and reftable backends.\n> \n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n> builtin/fsck.c | 5 +++++\n> refs.c | 5 +++++\n> refs.h | 5 +++++\n> refs/files-backend.c | 9 ++++++++-\n> refs/packed-backend.c | 7 +++++++\n> refs/refs-internal.h | 4 ++++\n> refs/reftable-backend.c | 7 +++++++\n> 7 files changed, 41 insertions(+), 1 deletion(-)\n> \n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index d13a226c2e..65a26e2d1b 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -1065,6 +1065,11 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n> \n> \tcheck_connectivity();\n> \n> +\tif (refs_fsck(get_main_ref_store(the_repository))) {\n> +\t\terror(\"ref database is corrupt\");\n> +\t\terrors_found |= ERROR_REFS;\n> +\t}\n\nOne thing I was wondering is whether we want to make the infrastructure\npart of the new git-refs(1) command, which is about to land via my\nseries that introduces ref backend migrations. We already make it a\nhabit that we spawn new processes here, as you can see in two lines\nbelow for example. It would also make it trivial to execute ref checks\nstandalone, by saying e.g. `git refs verify`.\n\n> \tif (the_repository->settings.core_commit_graph) {\n> \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n> \n[snip]\n> diff --git a/refs.h b/refs.h\n> index 34568ee1fb..2799820c40 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -544,6 +544,11 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n> */\n> int check_refname_format(const char *refname, int flags);\n> \n> +/*\n> + * Return 0 iff all refs in filesystem are consistent.\n> +*/\n> +int refs_fsck(struct ref_store *refs);\n\nWe should probably also mention that errors will be written to `stderr`?\n\nAlso, I noticed that you are missing the implementation for the debug\nbackend in \"refs/debug.c\".\n\n> /*\n> * Apply the rules from check_refname_format, but mutate the result until it\n> * is acceptable, and place the result in \"out\".\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 5f3089d947..b6147c588b 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -3299,6 +3299,11 @@ static int files_init_db(struct ref_store *ref_store,\n> \treturn 0;\n> }\n> \n> +static int files_fsck(struct ref_store *ref_store)\n> +{\n> +\treturn 0;\n> +}\n\nWe can already wire up the call to the embedded packed-refs ref store\nhere. We always want to call that from the files backend.\n\nPatrick\n"},{"id":"496065","messageId":"CAOLa=ZT895c-ag-8_r7SfJ+n92-S16JfCYgYhQf0caxg63cisg@mail.gmail.com","threadId":"61565","inReplyTo":"20240530122753.1114818-2-shejialuo@gmail.com","subject":"Re: [GSoC][PATCH 1/2] refs: setup ref consistency check infrastructure","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-06-03T08:56:10Z","receivedAt":"2024-06-03T08:56:12Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello,\n\nshejialuo writes:\n\n[snip]\n\n> diff --git a/refs.h b/refs.h\n> index 34568ee1fb..2799820c40 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -544,6 +544,11 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n> */\n> int check_refname_format(const char *refname, int flags);\n>\n> +/*\n> + * Return 0 iff all refs in filesystem are consistent.\n> +*/\n\ns/iff/if\n\nThe indentation seems to be wrong here. Also since we want to check refs\nand reflogs, it might be better to mention that the function checks the\nreference database for consistency.\n\n> +int refs_fsck(struct ref_store *refs);\n> +\n> /*\n> * Apply the rules from check_refname_format, but mutate the result until it\n> * is acceptable, and place the result in \"out\".\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 5f3089d947..b6147c588b 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -3299,6 +3299,11 @@ static int files_init_db(struct ref_store *ref_store,\n> \treturn 0;\n> }\n>\n> +static int files_fsck(struct ref_store *ref_store)\n> +{\n> +\treturn 0;\n> +}\n> +\n> struct ref_storage_be refs_be_files = {\n> \t.name = \"files\",\n> \t.init = files_ref_store_create,\n> @@ -3322,5 +3327,7 @@ struct ref_storage_be refs_be_files = {\n> \t.reflog_exists = files_reflog_exists,\n> \t.create_reflog = files_create_reflog,\n> \t.delete_reflog = files_delete_reflog,\n> -\t.reflog_expire = files_reflog_expire\n> +\t.reflog_expire = files_reflog_expire,\n> +\n> +\t.fsck = files_fsck,\n> };\n> diff --git a/refs/packed-backend.c b/refs/packed-backend.c\n> index a937e7dbfc..0617321634 100644\n> --- a/refs/packed-backend.c\n> +++ b/refs/packed-backend.c\n> @@ -1704,6 +1704,11 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n> \treturn empty_ref_iterator_begin();\n> }\n>\n> +static int packed_fsck(struct ref_store *ref_store)\n> +{\n> +\treturn 0;\n> +}\n> +\n> struct ref_storage_be refs_be_packed = {\n> \t.name = \"packed\",\n> \t.init = packed_ref_store_create,\n> @@ -1728,4 +1733,6 @@ struct ref_storage_be refs_be_packed = {\n> \t.create_reflog = NULL,\n> \t.delete_reflog = NULL,\n> \t.reflog_expire = NULL,\n> +\n> +\t.fsck = packed_fsck,\n> };\n> diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> index 53a6c5d842..ef697bf3bf 100644\n> --- a/refs/refs-internal.h\n> +++ b/refs/refs-internal.h\n> @@ -675,6 +675,8 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n> typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n> \t\t\t\t struct strbuf *referent);\n>\n> +typedef int fsck_fn(struct ref_store *ref_store);\n> +\n> struct ref_storage_be {\n> \tconst char *name;\n> \tref_store_init_fn *init;\n> @@ -700,6 +702,8 @@ struct ref_storage_be {\n> \tcreate_reflog_fn *create_reflog;\n> \tdelete_reflog_fn *delete_reflog;\n> \treflog_expire_fn *reflog_expire;\n> +\n> +\tfsck_fn *fsck;\n> };\n>\n> extern struct ref_storage_be refs_be_files;\n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index 1af86bbdec..f3f85cd2f0 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -2167,6 +2167,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n> \treturn ret;\n> }\n>\n> +static int reftable_be_fsck(struct ref_store *ref_store)\n> +{\n> +\treturn 0;\n> +}\n> +\n> struct ref_storage_be refs_be_reftable = {\n> \t.name = \"reftable\",\n> \t.init = reftable_be_init,\n> @@ -2191,4 +2196,6 @@ struct ref_storage_be refs_be_reftable = {\n> \t.create_reflog = reftable_be_create_reflog,\n> \t.delete_reflog = reftable_be_delete_reflog,\n> \t.reflog_expire = reftable_be_reflog_expire,\n> +\n> +\t.fsck = reftable_be_fsck,\n> };\n> --\n> 2.45.1\n"},{"id":"496192","messageId":"CAPig+cQxnNUGT_w_chPxgufQpie0cRccipifOz1ZQS49JsgByQ@mail.gmail.com","threadId":"61565","inReplyTo":"CAOLa=ZT895c-ag-8_r7SfJ+n92-S16JfCYgYhQf0caxg63cisg@mail.gmail.com","subject":"Re: [GSoC][PATCH 1/2] refs: setup ref consistency check infrastructure","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-06-03T22:03:00Z","receivedAt":"2024-06-03T22:03:12Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Jun 3, 2024 at 4:56 AM Karthik Nayak wrote:\n> shejialuo writes:\n> > +/*\n> > + * Return 0 iff all refs in filesystem are consistent.\n> > +*/\n>\n> s/iff/if\n\nI had spotted this, as well, but figured that the use of \"iff\" was\nintentional to mean \"if and only if\".\n"},{"id":"496997","messageId":"20240612085349.710785-1-shejialuo@gmail.com","threadId":"61565","inReplyTo":"20240530122753.1114818-1-shejialuo@gmail.com","subject":"[GSoC][PATCH v2 0/7] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-12T08:53:42Z","receivedAt":"2024-06-12T08:54:02Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"This patch sets up the ref consistency check infrastructure. We need to\nintegrate the fsck error messages. However, the git-fsck(1) is highly\ncoupled with the object database. And it uses `git_fsck_config` to\ninitialize the `fsck_options->fsck_msg_type`. For my implementation, I\ndo not add some new files into `fsck_options` due to the following\nreasons:\n\n1. The fileds in `fsck_options` are all related to objects. We only can\nreuse the `fsck_msg_type` and `strict` member, adding some ref-related\nfileds into `fsck_options` will break the semantics.\n2. We may laterc hange `fsck_options` and `fsck_objs_options`.\n\nThe code for integrating fsck error messages for refs may be a liitle\nredudant now, but if we change the `fsck_options`, it will be way\ncomplicated.\n\nAfter integrating the fsck error messages, setup the consistency check\ninfrastructure like patch v1. And introduce a new subcommand for builtin\ngit-refs and in git-fsck(1), use a child process to call this\nsubcommand.\n\nAt last, like patch v1, implement the consistency check for files\nbackend and corresponding unit tests.\n\nBecause I have added more commits compare to v1. So I provide the\ninnerdiff between the v1 and v2.\n\nThanks,\nJialuo\n\nshejialuo (7):\n fsck: add refs check interfaces to interface with fsck error levels\n refs: set up ref consistency check infrastructure\n builtin/refs: add verify subcommand\n builtin/fsck: add `git-refs verify` child process\n files-backend: add unified interface for refs scanning\n fsck: add ref name check for files backend\n fsck: add ref content check for files backend\n\n Documentation/fsck-msgids.txt | 12 ++\n Documentation/git-refs.txt | 11 ++\n builtin/fsck.c | 17 +++\n builtin/refs.c | 45 +++++++\n fsck.c | 69 ++++++++++\n fsck.h | 43 ++++++-\n refs.c | 7 +-\n refs.h | 8 ++\n refs/debug.c | 9 ++\n refs/files-backend.c | 234 +++++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 ++\n refs/refs-internal.h | 11 +-\n refs/reftable-backend.c | 8 ++\n t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++++\n 14 files changed, 685 insertions(+), 8 deletions(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\nInterdiff between patch v1 and patch v2:\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex 5edc06c658..69f86c5345 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,12 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefName`::\n+\t(ERROR) A ref has a bad name.\n+\n+`badRefSha`::\n+\t(ERROR) A ref has a bad sha.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \n@@ -34,6 +40,9 @@\n `badType`::\n \t(ERROR) Found an invalid object type.\n \n+`danglingSymref`::\n+\t(WARN) Found a dangling symref.\n+\n `duplicateEntries`::\n \t(ERROR) A tree contains duplicate file entries.\n \n@@ -176,6 +185,9 @@\n `symlinkTargetMissing`::\n \t(ERROR) Unable to read symbolic link target's blob.\n \n+`trailingRefContent`::\n+\t(WARN) A ref content has trailing contents.\n+\n `treeNotSorted`::\n \t(ERROR) A tree is not properly sorted.\n-\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..f9d36ea19d 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,13 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+--strict::\n+\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n+\tbe seen as ERROR.\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 65a26e2d1b..10d73f534f 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -896,6 +896,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \treturn res;\n }\n \n+static void fsck_refs(void)\n+{\n+\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n+\tchild_process_init(&refs_verify);\n+\trefs_verify.git_cmd = 1;\n+\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n+\tif (verbose)\n+\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n+\tif (check_strict)\n+\t\tstrvec_push(&refs_verify.args, \"--strict\");\n+\n+\tif (run_command(&refs_verify))\n+\t\terrors_found |= ERROR_REFS;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n@@ -1065,10 +1080,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n-\tif (refs_fsck(get_main_ref_store(the_repository))) {\n-\t\terror(\"ref database is corrupt\");\n-\t\terrors_found |= ERROR_REFS;\n-\t}\n+\tfsck_refs();\n \n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n-\ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..82ed0d57c1 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,55 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tint ret = 0;\n+\tunsigned int verbose = 0, strict = 0;\n+\tstruct fsck_refs_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n+\tstruct option options[] = {\n+\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"too many arguments\"));\n+\n+\tif (verbose)\n+\t\tfsck_refs_options.verbose = 1;\n+\tif (strict)\n+\t\tfsck_refs_options.strict = 1;\n+\n+\tgit_config(git_fsck_refs_config, &fsck_refs_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n+\n+\t/*\n+\t * Explicitly free the allocated array. This is necessary because\n+\t * this program is executed as child process of git-fsck(1) and the\n+\t * allocated array may not freed when git-fsck(1) aborts somewhere.\n+\t */\n+\tfree(fsck_refs_options.msg_type);\n+\treturn ret;\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/fsck.c b/fsck.c\nindex 8ef962199f..13528c646e 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1249,6 +1249,20 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n+int fsck_refs_error_function(struct fsck_refs_options *o UNUSED,\n+\t\t\t const char *name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tif (msg_type == FSCK_WARN) {\n+\t\twarning(\"%s: %s\", name, message);\n+\t\treturn 0;\n+\t}\n+\terror(\"%s: %s\", name, message);\n+\treturn 1;\n+}\n+\n int fsck_error_function(struct fsck_options *o,\n \t\t\tconst struct object_id *oid,\n \t\t\tenum object_type object_type UNUSED,\n@@ -1323,6 +1337,61 @@ int fsck_finish(struct fsck_options *options)\n \treturn ret;\n }\n \n+int fsck_refs_report(struct fsck_refs_options *o,\n+\t\t const char *name,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tenum fsck_msg_type msg_type =\n+\t\tfsck_msg_type(msg_id, (struct fsck_options*)o);\n+\tint ret = 0;\n+\n+\tif (msg_type == FSCK_IGNORE)\n+\t\treturn 0;\n+\n+\tif (msg_type == FSCK_FATAL)\n+\t\tmsg_type = FSCK_ERROR;\n+\telse if (msg_type == FSCK_INFO)\n+\t\tmsg_type = FSCK_WARN;\n+\n+\tprepare_msg_ids();\n+\tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n+\n+\tva_start(ap, fmt);\n+\tstrbuf_vaddf(&sb, fmt, ap);\n+\tret = o->error_func(o, name, msg_type, msg_id, sb.buf);\n+\tstrbuf_release(&sb);\n+\tva_end(ap);\n+\n+\treturn ret;\n+}\n+\n+int git_fsck_refs_config(const char *var, const char *value,\n+\t\t\t const struct config_context *ctx, void *cb)\n+{\n+\tstruct fsck_refs_options *options = cb;\n+\tconst char *msg_id;\n+\n+\t/*\n+\t * We don't check the value of fsck.skiplist here, because it\n+\t * is specific to object database, not reference database.\n+\t */\n+\tif (strcmp(var, \"fsck.skiplist\") == 0) {\n+\t\treturn 0;\n+\t}\n+\n+\tif (skip_prefix(var, \"fsck.\", &msg_id)) {\n+\t\tif (!value)\n+\t\t\treturn config_error_nonbool(var);\n+\t\tfsck_set_msg_type((struct fsck_options*)options, msg_id, value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, ctx, cb);\n+}\n+\n int git_fsck_config(const char *var, const char *value,\n \t\t const struct config_context *ctx, void *cb)\n {\ndiff --git a/fsck.h b/fsck.h\nindex 17fa2dda5d..5a55a567b0 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,8 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n+\tFUNC(BAD_REF_SHA, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n@@ -68,11 +70,13 @@ enum fsck_msg_type {\n \tFUNC(SYMLINK_TARGET_BLOB, ERROR) \\\n \t/* warnings */ \\\n \tFUNC(EMPTY_NAME, WARN) \\\n+\tFUNC(DANGLING_SYMREF, WARN) \\\n \tFUNC(FULL_PATHNAME, WARN) \\\n \tFUNC(HAS_DOT, WARN) \\\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n+\tFUNC(TRAILING_REF_CONTENT, WARN) \\\n \tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n \tFUNC(LARGE_PATHNAME, WARN) \\\n@@ -96,6 +100,7 @@ enum fsck_msg_id {\n };\n #undef MSG_ID\n \n+struct fsck_refs_options;\n struct fsck_options;\n struct object;\n \n@@ -107,6 +112,21 @@ void fsck_set_msg_type(struct fsck_options *options,\n void fsck_set_msg_types(struct fsck_options *options, const char *values);\n int is_valid_msg_type(const char *msg_id, const char *msg_type);\n \n+/*\n+ * callback function for fsck refs and reflogs.\n+ */\n+typedef int (*fsck_refs_error)(struct fsck_refs_options *o,\n+\t\t\t const char *name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n+\n+int fsck_refs_error_function(struct fsck_refs_options *o,\n+\t\t\t const char *name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n+\n /*\n * callback function for fsck_walk\n * type is the expected type of the object or OBJ_ANY\n@@ -135,11 +155,22 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message);\n \n+struct fsck_refs_options {\n+\tenum fsck_msg_type *msg_type;\n+\tunsigned strict:1;\n+\tfsck_refs_error error_func;\n+\tunsigned verbose:1;\n+};\n+\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n+\n struct fsck_options {\n+\tenum fsck_msg_type *msg_type;\n+\tunsigned strict:1;\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n-\tunsigned strict:1;\n-\tenum fsck_msg_type *msg_type;\n \tstruct oidset skiplist;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n@@ -221,6 +252,12 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+__attribute__((format (printf, 4, 5)))\n+int fsck_refs_report(struct fsck_refs_options *o,\n+\t\t const char *name,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\n@@ -247,6 +284,8 @@ const char *fsck_describe_object(struct fsck_options *options,\n \t\t\t\t const struct object_id *oid);\n \n struct key_value_info;\n+int git_fsck_refs_config(const char *var, const char *value,\n+\t\t\t const struct config_context *ctx, void *cb);\n /*\n * git_config() callback for use by fsck-y tools that want to support\n * fsck. fsck.skipList etc.\ndiff --git a/refs.c b/refs.c\nindex 06e191a316..1325f83269 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -316,9 +316,9 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n-int refs_fsck(struct ref_store *refs)\n+int refs_fsck(struct ref_store *refs, struct fsck_refs_options *o)\n {\n-\treturn refs->be->fsck(refs);\n+\treturn refs->be->fsck(refs, o);\n }\n \n void sanitize_refname_component(const char *refname, struct strbuf *out)\n@@ -1744,7 +1744,7 @@ static int refs_read_special_head(struct ref_store *ref_store,\n \t}\n \n \tresult = parse_loose_ref_contents(content.buf, oid, referent, type,\n-\t\t\t\t\t failure_errno);\n+\t\t\t\t\t failure_errno, NULL);\n \n done:\n \tstrbuf_release(&full_path);\ndiff --git a/refs.h b/refs.h\nindex 55160085b6..5a042695f5 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -3,6 +3,7 @@\n \n #include \"commit.h\"\n \n+struct fsck_refs_options;\n struct object_id;\n struct ref_store;\n struct repository;\n@@ -548,9 +549,11 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n int check_refname_format(const char *refname, int flags);\n \n /*\n- * Return 0 iff all refs in filesystem are consistent.\n-*/\n-int refs_fsck(struct ref_store *refs);\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_refs_options *o);\n \n /*\n * Apply the rules from check_refname_format, but mutate the result until it\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..110a264522 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,13 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o)\n+{\n+\ttrace_printf_key(&trace_refs, \"fsck\\n\");\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +458,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 5520a12a67..17d3e433f1 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -4,6 +4,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -548,7 +549,7 @@ static int read_ref_internal(struct ref_store *ref_store, const char *refname,\n \tstrbuf_rtrim(&sb_contents);\n \tbuf = sb_contents.buf;\n \n-\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr);\n+\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr, NULL);\n \n out:\n \tif (ret && !myerr)\n@@ -584,7 +585,7 @@ static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refn\n \n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno)\n+\t\t\t int *failure_errno, unsigned int *trailing)\n {\n \tconst char *p;\n \tif (skip_prefix(buf, \"ref:\", &buf)) {\n@@ -606,6 +607,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t*failure_errno = EINVAL;\n \t\treturn -1;\n \t}\n+\n+\tif (trailing && (*p != '\\0' && *p != '\\n'))\n+\t\t*trailing = 1;\n+\n \treturn 0;\n }\n \n@@ -829,10 +834,8 @@ static int lock_raw_ref(struct files_ref_store *refs,\n \t\t */\n \t\tif (refs_verify_refname_available(\n \t\t\t\t refs->packed_ref_store, refname,\n-\t\t\t\t extras, NULL, err)) {\n-\t\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\t\t\t extras, NULL, err))\n \t\t\tgoto error_return;\n-\t\t}\n \t}\n \n \tret = 0;\n@@ -3404,117 +3407,223 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n-typedef int (*files_fsck_refs_fn)(const char *refs_check_dir,\n-\t\t\t\tstruct dir_iterator *iter);\n-\n-static int files_fsck_refs_name(const char *refs_check_dir,\n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct fsck_refs_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_name(struct fsck_refs_options *o,\n+\t\t\t\tconst char *gitdir UNUSED,\n+\t\t\t\tconst char *refs_check_dir,\n \t\t\t\tstruct dir_iterator *iter)\n {\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n \tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n-\t\terror(_(\"%s/%s: invalid refname format\"), refs_check_dir, iter->basename);\n-\t\treturn -1;\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\tret = fsck_refs_report(o, sb.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n \t}\n \n-\treturn 0;\n+\tstrbuf_release(&sb);\n+\treturn ret;\n }\n \n-static int files_fsck_refs_content(const char *refs_check_dir,\n-\t\t\t\tstruct dir_iterator *iter)\n+static int files_fsck_symref(struct fsck_refs_options *o,\n+\t\t\t struct strbuf *refname,\n+\t\t\t struct strbuf *path)\n {\n-\tstruct strbuf ref_content = STRBUF_INIT;\n+\tstruct stat st;\n+\tint ret = 0;\n \n-\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n-\t\terror(_(\"%s/%s: unable to read the ref\"), refs_check_dir, iter->basename);\n-\t\tgoto clean;\n+\tif (lstat(path->buf, &st) < 0) {\n+\t\tret = fsck_refs_report(o, refname->buf,\n+\t\t\t\t FSCK_MSG_DANGLING_SYMREF,\n+\t\t\t\t \"point to non-existent ref\");\n+\t\tgoto out;\n \t}\n \n-\t/*\n-\t * Case 1: check if the ref content length is valid and the last\n-\t * character is a newline.\n-\t */\n-\tif (ref_content.len != the_hash_algo->hexsz + 1 ||\n-\t\t\tref_content.buf[ref_content.len - 1] != '\\n') {\n-\t\tgoto failure;\n+\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n+\t\tret = fsck_refs_report(o, refname->buf,\n+\t\t\t\t FSCK_MSG_DANGLING_SYMREF,\n+\t\t\t\t \"point to invalid object\");\n+\t\tgoto out;\n \t}\n+out:\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs_content(struct fsck_refs_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter)\n+{\n+\tstruct strbuf path = STRBUF_INIT,\n+\t\t refname = STRBUF_INIT,\n+\t\t ref_content = STRBUF_INIT,\n+\t\t referent = STRBUF_INIT;\n+\tunsigned int trailing = 0;\n+\tint failure_errno = 0;\n+\tunsigned int type = 0;\n+\tstruct object_id oid;\n+\tint ret = 0;\n+\n+\tstrbuf_addbuf(&path, &iter->path);\n+\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n \n \t/*\n-\t * Case 2: the content should be range of [0-9a-f].\n+\t * If the file is a symlink, we need to only check the connectivity\n+\t * of the destination object.\n \t */\n-\tfor (size_t i = 0; i < the_hash_algo->hexsz; i++) {\n-\t\tif (!isdigit(ref_content.buf[i]) &&\n-\t\t\t\t(ref_content.buf[i] < 'a' || ref_content.buf[i] > 'f')) {\n-\t\t\tgoto failure;\n+\tif (S_ISLNK(iter->st.st_mode)) {\n+\t\tstrbuf_strip_file_from_path(&path);\n+\t\tret = strbuf_readlink(&ref_content,\n+\t\t\t\t iter->path.buf, iter->st.st_size);\n+\t\tif (ret < 0) {\n+\t\t\tret = error_errno(_(\"could not read link '%s'\"),\n+\t\t\t\t\t iter->path.buf);\n+\t\t\tgoto clean;\n \t\t}\n+\t\tstrbuf_addbuf(&path, &ref_content);\n+\t\tstrbuf_reset(&ref_content);\n+\n+\t\tret = files_fsck_symref(o, &refname, &path);\n+\t\tgoto clean;\n \t}\n \n-\tstrbuf_release(&ref_content);\n-\treturn 0;\n+\tif (strbuf_read_file(&ref_content, path.buf, 0) < 0) {\n+\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n+\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n+\t\t\t\t &referent, &type,\n+\t\t\t\t &failure_errno, &trailing)) {\n+\t\tret = fsck_refs_report(o, refname.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_SHA,\n+\t\t\t\t \"invalid ref content\");\n+\t\tgoto clean;\n+\t}\n \n-failure:\n-\terror(_(\"%s/%s: invalid ref content\"), refs_check_dir, iter->basename);\n+\t/*\n+\t * If the ref is a symref, we need to check the destination name and\n+\t * connectivity.\n+\t */\n+\tif (referent.len && (type & REF_ISSYMREF)) {\n+\t\tstrbuf_reset(&path);\n+\t\tstrbuf_addf(&path, \"%s/%s\", gitdir, referent.buf);\n+\n+\t\tif (check_refname_format(referent.buf, 0)) {\n+\t\t\tret = fsck_refs_report(o, refname.buf,\n+\t\t\t\t\t FSCK_MSG_DANGLING_SYMREF,\n+\t\t\t\t\t \"point to invalid refname\");\n+\t\t\tgoto clean;\n+\t\t}\n+\n+\t\tret = files_fsck_symref(o, &refname, &path);\n+\t\tgoto clean;\n+\t} else {\n+\t\t/*\n+\t\t * Only regular refs could have a trailing garbage. Should\n+\t\t * be reported as a warning.\n+\t\t */\n+\t\tif (trailing) {\n+\t\t\tret = fsck_refs_report(o, refname.buf,\n+\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n+\t\t\t\t\t \"trailing garbage in ref\");\n+\t\t\tgoto clean;\n+\t\t}\n+\t}\n \n clean:\n+\tstrbuf_release(&path);\n+\tstrbuf_release(&refname);\n \tstrbuf_release(&ref_content);\n-\treturn -1;\n+\tstrbuf_release(&referent);\n+\treturn ret;\n }\n \n-static int files_fsck_refs(struct ref_store *ref_store,\n-\t\t\t\tconst char* refs_check_dir,\n-\t\t\t\tfiles_fsck_refs_fn *fsck_refs_fns)\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_refs_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n {\n-\tstruct dir_iterator *iter;\n+\tconst char *gitdir = ref_store->gitdir;\n \tstruct strbuf sb = STRBUF_INIT;\n-\tint ret = 0;\n+\tstruct dir_iterator *iter;\n \tint iter_status;\n+\tint ret = 0;\n \n-\tstrbuf_addf(&sb, \"%s/%s\", ref_store->gitdir, refs_check_dir);\n+\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n \n \titer = dir_iterator_begin(sb.buf, 0);\n \n-\t/*\n-\t * The current implementation does not care about the worktree, the worktree\n-\t * may have no refs/heads or refs/tags directory. Simply return 0 now.\n-\t*/\n \tif (!iter) {\n-\t\treturn 0;\n+\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n+\t\tgoto out;\n \t}\n \n \twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n \t\tif (S_ISDIR(iter->st.st_mode)) {\n \t\t\tcontinue;\n-\t\t} else if (S_ISREG(iter->st.st_mode)) {\n-\t\t\tfor (files_fsck_refs_fn *fsck_refs_fn = fsck_refs_fns;\n-\t\t\t\t\t*fsck_refs_fn; fsck_refs_fn++) {\n-\t\t\t\tret |= (*fsck_refs_fn)(refs_check_dir, iter);\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->verbose)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n+\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n \t\t\t}\n \t\t} else {\n-\t\t\terror(_(\"unexpected file type for '%s'\"), iter->basename);\n-\t\t\tret = -1;\n+\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n+\t\t\t\t iter->basename);\n \t\t}\n \t}\n \n-\tif (iter_status != ITER_DONE) {\n-\t\tret = -1;\n-\t\terror(_(\"failed to iterate over '%s'\"), sb.buf);\n-\t}\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n \n+out:\n \tstrbuf_release(&sb);\n-\n \treturn ret;\n }\n \n-static int files_fsck(struct ref_store *ref_store)\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_refs_options *o)\n {\n-\tint ret = 0;\n-\n-\tfiles_fsck_refs_fn fsck_refs_fns[] = {\n+\tint ret;\n+\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n \t\tfiles_fsck_refs_name,\n \t\tfiles_fsck_refs_content,\n \t\tNULL\n \t};\n \n-\tret = files_fsck_refs(ref_store, \"refs/heads\",fsck_refs_fns)\n-\t | files_fsck_refs(ref_store, \"refs/tags\", fsck_refs_fns);\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, \"Checking references consistency\");\n+\n+\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n+\n+\treturn ret;\n+}\n+\n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o)\n+{\n+\tint ret;\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\tret = files_fsck_refs(ref_store, o);\n \n \treturn ret;\n }\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 3de0fd59a7..db152053f8 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1733,7 +1733,8 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n-static int packed_fsck(struct ref_store *ref_store)\n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o)\n {\n \treturn 0;\n }\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 16d92b252e..eb3a7cdcc1 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_refs_options;\n struct ref_transaction;\n \n /*\n@@ -650,7 +651,8 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n-typedef int fsck_fn(struct ref_store *ref_store);\n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o);\n \n struct ref_storage_be {\n \tconst char *name;\n@@ -707,11 +709,12 @@ struct ref_store {\n \n /*\n * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for\n- * invalid contents.\n+ * invalid contents. Also trailing is set to 1 when there is any bytes after the\n+ * hex.\n */\n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno);\n+\t\t\t int *failure_errno, unsigned int *trailing);\n \n /*\n * Fill in the generic part of refs and add it to our collection of\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 8f0ce5aed0..7f606faa9e 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2242,7 +2242,8 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n-static int reftable_be_fsck(struct ref_store *ref_store)\n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_refs_options *o)\n {\n \treturn 0;\n }\ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nindex 1c6c3804ff..94cb93bf92 100755\n--- a/t/t0602-reffiles-fsck.sh\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -1,4 +1,4 @@\n-#!/bin/bash\n+#!/bin/sh\n \n test_description='Test reffiles backend consistency check'\n \n@@ -21,15 +21,15 @@ test_expect_success 'ref name should be checked' '\n \t\tgit tag tag-1 &&\n \t\tgit commit --allow-empty -m second &&\n \t\tgit checkout -b branch-2 &&\n-\t\tgit tag tag-2\n+\t\tgit tag tag-2 &&\n+\t\tgit tag multi_hierarchy/tag-2\n \t) &&\n \t(\n \t\tcd repo &&\n \t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n \t\ttest_must_fail git fsck 2>err &&\n \t\tcat >expect <<-EOF &&\n-\t\terror: refs/heads/.branch-1: invalid refname format\n-\t\terror: ref database is corrupt\n+\t\terror: refs/heads/.branch-1: badRefName: invalid refname format\n \t\tEOF\n \t\trm $branch_dir_prefix/.branch-1 &&\n \t\ttest_cmp expect err\n@@ -39,8 +39,7 @@ test_expect_success 'ref name should be checked' '\n \t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n \t\ttest_must_fail git fsck 2>err &&\n \t\tcat >expect <<-EOF &&\n-\t\terror: refs/tags/tag-1.lock: invalid refname format\n-\t\terror: ref database is corrupt\n+\t\terror: refs/tags/tag-1.lock: badRefName: invalid refname format\n \t\tEOF\n \t\trm $tag_dir_prefix/tag-1.lock &&\n \t\ttest_cmp expect err\n@@ -50,15 +49,24 @@ test_expect_success 'ref name should be checked' '\n \t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n \t\ttest_must_fail git fsck 2>err &&\n \t\tcat >expect <<-EOF &&\n-\t\terror: refs/heads/@: invalid refname format\n-\t\terror: ref database is corrupt\n+\t\terror: refs/heads/@: badRefName: invalid refname format\n \t\tEOF\n \t\trm $branch_dir_prefix/@ &&\n \t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_cmp expect err\n \t)\n '\n \n-test_expect_success 'ref content should be checked' '\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n \ttest_when_finished \"rm -rf repo\" &&\n \tgit init repo &&\n \tbranch_dir_prefix=.git/refs/heads &&\n@@ -72,13 +80,45 @@ test_expect_success 'ref content should be checked' '\n \t\tgit checkout -b branch-2 &&\n \t\tgit tag tag-2\n \t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\tgit -c fsck.badRefName=warn fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\tgit -c fsck.badRefName=ignore fsck 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+test_expect_success 'regular ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit checkout -b a/b/tag-2\n+\t) &&\n \t(\n \t\tcd repo &&\n \t\tprintf \"%s garbage\" \"$(git rev-parse branch-1)\" > $branch_dir_prefix/branch-1-garbage &&\n-\t\ttest_must_fail git fsck 2>err &&\n+\t\tgit fsck 2>err &&\n \t\tcat >expect <<-EOF &&\n-\t\terror: refs/heads/branch-1-garbage: invalid ref content\n-\t\terror: ref database is corrupt\n+\t\twarning: refs/heads/branch-1-garbage: trailingRefContent: trailing garbage in ref\n \t\tEOF\n \t\trm $branch_dir_prefix/branch-1-garbage &&\n \t\ttest_cmp expect err\n@@ -86,10 +126,9 @@ test_expect_success 'ref content should be checked' '\n \t(\n \t\tcd repo &&\n \t\tprintf \"%s garbage\" \"$(git rev-parse tag-1)\" > $tag_dir_prefix/tag-1-garbage &&\n-\t\ttest_must_fail git fsck 2>err &&\n+\t\ttest_must_fail git -c fsck.trailingRefContent=error fsck 2>err &&\n \t\tcat >expect <<-EOF &&\n-\t\terror: refs/tags/tag-1-garbage: invalid ref content\n-\t\terror: ref database is corrupt\n+\t\terror: refs/tags/tag-1-garbage: trailingRefContent: trailing garbage in ref\n \t\tEOF\n \t\trm $tag_dir_prefix/tag-1-garbage &&\n \t\ttest_cmp expect err\n@@ -97,34 +136,74 @@ test_expect_success 'ref content should be checked' '\n \t(\n \t\tcd repo &&\n \t\tprintf \"%s \" \"$(git rev-parse tag-2)\" > $tag_dir_prefix/tag-2-garbage &&\n-\t\ttest_must_fail git fsck 2>err &&\n+\t\tgit fsck 2>err &&\n \t\tcat >expect <<-EOF &&\n-\t\terror: refs/tags/tag-2-garbage: invalid ref content\n-\t\terror: ref database is corrupt\n+\t\twarning: refs/tags/tag-2-garbage: trailingRefContent: trailing garbage in ref\n \t\tEOF\n \t\trm $tag_dir_prefix/tag-2-garbage &&\n \t\ttest_cmp expect err\n \t) &&\n \t(\n \t\tcd repo &&\n-\t\ttr -d \"\\n\" < $branch_dir_prefix/branch-1 > $branch_dir_prefix/branch-1-without-newline &&\n-\t\ttest_must_fail git fsck 2>err &&\n+\t\tprintf \"xfsazqfxcadas\" > $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n \t\tcat >expect <<-EOF &&\n-\t\terror: refs/heads/branch-1-without-newline: invalid ref content\n-\t\terror: ref database is corrupt\n+\t\terror: refs/tags/tag-2-bad: badRefSha: invalid ref content\n \t\tEOF\n-\t\trm $branch_dir_prefix/branch-1-without-newline &&\n+\t\trm $tag_dir_prefix/tag-2-bad &&\n \t\ttest_cmp expect err\n \t) &&\n \t(\n \t\tcd repo &&\n-\t\ttr \"[:lower:]\" \"[:upper:]\" < $branch_dir_prefix/branch-2 > $branch_dir_prefix/branch-2-upper &&\n-\t\ttest_must_fail git fsck 2>err &&\n+\t\tprintf \"xfsazqfxcadas\" > $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/a/b/branch-2-bad: badRefSha: invalid ref content\n+\t\tEOF\n+\t\trm $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'symbolic ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads/branch-3\" > $branch_dir_prefix/branch-2-bad &&\n+\t\tgit refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-2-bad: danglingSymref: point to non-existent ref\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n+\t\tgit refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-2-bad: danglingSymref: point to invalid refname\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n+\t\tgit refs verify 2>err &&\n \t\tcat >expect <<-EOF &&\n-\t\terror: refs/heads/branch-2-upper: invalid ref content\n-\t\terror: ref database is corrupt\n+\t\twarning: refs/heads/branch-2-bad: danglingSymref: point to invalid object\n \t\tEOF\n-\t\trm $branch_dir_prefix/branch-2-upper &&\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n \t\ttest_cmp expect err\n \t)\n '\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex b06c46999d..cc5bbfd732 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -10,7 +10,6 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n GIT_TEST_DEFAULT_REF_FORMAT=reftable\n export GIT_TEST_DEFAULT_REF_FORMAT\n \n-TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n INVALID_OID=$(test_oid 001)\ndiff --git a/t/t1300-config.sh b/t/t1300-config.sh\nindex 9de2d95f06..f3c4d28e06 100755\n--- a/t/t1300-config.sh\n+++ b/t/t1300-config.sh\n@@ -596,8 +596,7 @@ test_expect_success 'get bool variable with empty value' '\n \n test_expect_success 'no arguments, but no crash' '\n \ttest_must_fail git config >output 2>&1 &&\n-\techo \"error: no action specified\" >expect &&\n-\ttest_cmp expect output\n+\ttest_grep usage output\n '\n \n cat > .git/config << EOF\n@@ -2835,12 +2834,6 @@ test_expect_success 'specifying multiple modes causes failure' '\n \ttest_cmp expect err\n '\n \n-test_expect_success 'writing to stdin is rejected' '\n-\techo \"fatal: writing to stdin is not supported\" >expect &&\n-\ttest_must_fail git config ${mode_set} --file - foo.bar baz 2>err &&\n-\ttest_cmp expect err\n-'\n-\n done\n \n test_done\n\n-- \n2.45.2\n\n"},{"id":"496998","messageId":"20240612085349.710785-2-shejialuo@gmail.com","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[GSoC][PATCH v2 1/7] fsck: add refs check interfaces to interface with fsck error levels","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-12T08:53:43Z","receivedAt":"2024-06-12T08:54:09Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) focuses on object database consistency check. It relies\non the \"fsck_options\" to interact with fsck error levels. However,\n\"fsck_options\" aims at checking the object database which makes it\nunsuitable to change the semantics of it. Instead, create\n\"fsck_refs_options\" structure to handle refs consistency check.\n\nThe \"git_fsck_config\" sets up the \"msg_type\" and \"skiplist\" member of\nthe \"fsck_options\". For refs, we just need the \"msg_type\". In order to\nallow setting up more refs-specific options easily later, add a separate\nfunction \"git_fsck_refs_config\" to initialize the refs-specific options.\n\nMove the \"msg_type\" and \"strict\" member to the top of the \"fsck_options\"\nwhich allows us to convert \"fsck_refs_options *\" to \"fsck_options *\" to\nreuse the interfaces provided by \"fsck.h\" without changing the original\ncode.\n\nThe static function \"report\" provided by \"fsck.c\" aims at reporting the\nproblems related to object database which cannot be reused for refs.\nProvide \"fsck_refs_report\" function to integrate the fsck error levels\ninto reference consistency check.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n fsck.h | 39 +++++++++++++++++++++++++++++++--\n 2 files changed, 106 insertions(+), 2 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex 8ef962199f..13528c646e 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1249,6 +1249,20 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n+int fsck_refs_error_function(struct fsck_refs_options *o UNUSED,\n+\t\t\t const char *name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tif (msg_type == FSCK_WARN) {\n+\t\twarning(\"%s: %s\", name, message);\n+\t\treturn 0;\n+\t}\n+\terror(\"%s: %s\", name, message);\n+\treturn 1;\n+}\n+\n int fsck_error_function(struct fsck_options *o,\n \t\t\tconst struct object_id *oid,\n \t\t\tenum object_type object_type UNUSED,\n@@ -1323,6 +1337,61 @@ int fsck_finish(struct fsck_options *options)\n \treturn ret;\n }\n \n+int fsck_refs_report(struct fsck_refs_options *o,\n+\t\t const char *name,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tenum fsck_msg_type msg_type =\n+\t\tfsck_msg_type(msg_id, (struct fsck_options*)o);\n+\tint ret = 0;\n+\n+\tif (msg_type == FSCK_IGNORE)\n+\t\treturn 0;\n+\n+\tif (msg_type == FSCK_FATAL)\n+\t\tmsg_type = FSCK_ERROR;\n+\telse if (msg_type == FSCK_INFO)\n+\t\tmsg_type = FSCK_WARN;\n+\n+\tprepare_msg_ids();\n+\tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n+\n+\tva_start(ap, fmt);\n+\tstrbuf_vaddf(&sb, fmt, ap);\n+\tret = o->error_func(o, name, msg_type, msg_id, sb.buf);\n+\tstrbuf_release(&sb);\n+\tva_end(ap);\n+\n+\treturn ret;\n+}\n+\n+int git_fsck_refs_config(const char *var, const char *value,\n+\t\t\t const struct config_context *ctx, void *cb)\n+{\n+\tstruct fsck_refs_options *options = cb;\n+\tconst char *msg_id;\n+\n+\t/*\n+\t * We don't check the value of fsck.skiplist here, because it\n+\t * is specific to object database, not reference database.\n+\t */\n+\tif (strcmp(var, \"fsck.skiplist\") == 0) {\n+\t\treturn 0;\n+\t}\n+\n+\tif (skip_prefix(var, \"fsck.\", &msg_id)) {\n+\t\tif (!value)\n+\t\t\treturn config_error_nonbool(var);\n+\t\tfsck_set_msg_type((struct fsck_options*)options, msg_id, value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, ctx, cb);\n+}\n+\n int git_fsck_config(const char *var, const char *value,\n \t\t const struct config_context *ctx, void *cb)\n {\ndiff --git a/fsck.h b/fsck.h\nindex 17fa2dda5d..7451b1f91b 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -96,6 +96,7 @@ enum fsck_msg_id {\n };\n #undef MSG_ID\n \n+struct fsck_refs_options;\n struct fsck_options;\n struct object;\n \n@@ -107,6 +108,21 @@ void fsck_set_msg_type(struct fsck_options *options,\n void fsck_set_msg_types(struct fsck_options *options, const char *values);\n int is_valid_msg_type(const char *msg_id, const char *msg_type);\n \n+/*\n+ * callback function for fsck refs and reflogs.\n+ */\n+typedef int (*fsck_refs_error)(struct fsck_refs_options *o,\n+\t\t\t const char *name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n+\n+int fsck_refs_error_function(struct fsck_refs_options *o,\n+\t\t\t const char *name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n+\n /*\n * callback function for fsck_walk\n * type is the expected type of the object or OBJ_ANY\n@@ -135,11 +151,22 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message);\n \n+struct fsck_refs_options {\n+\tenum fsck_msg_type *msg_type;\n+\tunsigned strict:1;\n+\tfsck_refs_error error_func;\n+\tunsigned verbose:1;\n+};\n+\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n+\n struct fsck_options {\n+\tenum fsck_msg_type *msg_type;\n+\tunsigned strict:1;\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n-\tunsigned strict:1;\n-\tenum fsck_msg_type *msg_type;\n \tstruct oidset skiplist;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n@@ -221,6 +248,12 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+__attribute__((format (printf, 4, 5)))\n+int fsck_refs_report(struct fsck_refs_options *o,\n+\t\t const char *name,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\n@@ -247,6 +280,8 @@ const char *fsck_describe_object(struct fsck_options *options,\n \t\t\t\t const struct object_id *oid);\n \n struct key_value_info;\n+int git_fsck_refs_config(const char *var, const char *value,\n+\t\t\t const struct config_context *ctx, void *cb);\n /*\n * git_config() callback for use by fsck-y tools that want to support\n * fsck. fsck.skipList etc.\n-- \n2.45.2\n\n"},{"id":"496999","messageId":"20240612085349.710785-3-shejialuo@gmail.com","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[GSoC][PATCH v2 2/7] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-12T08:53:44Z","receivedAt":"2024-06-12T08:54:13Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It's organized as the five parts:\n\n1. The name and the initialization interfaces.\n2. The ref transaction interfaces.\n3. The ref internal interfaces (pack, rename and copy).\n4. The ref filesystem interfaces.\n5. The reflog related interfaces.\n\nTo keep consistent with the git-fsck(1), add a new interface named\n\"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\ngrouped into any above five categories. Explicitly add blank line to\nmake it different from others.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs.c | 5 +++++\n refs.h | 8 ++++++++\n refs/debug.c | 9 +++++++++\n refs/files-backend.c | 15 ++++++++++++++-\n refs/packed-backend.c | 8 ++++++++\n refs/refs-internal.h | 6 ++++++\n refs/reftable-backend.c | 8 ++++++++\n 7 files changed, 58 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex f7c7765d23..0922439275 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -316,6 +316,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs, struct fsck_refs_options *o)\n+{\n+\treturn refs->be->fsck(refs, o);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex 76d25df4de..5a042695f5 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -3,6 +3,7 @@\n \n #include \"commit.h\"\n \n+struct fsck_refs_options;\n struct object_id;\n struct ref_store;\n struct repository;\n@@ -547,6 +548,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_refs_options *o);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..110a264522 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,13 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o)\n+{\n+\ttrace_printf_key(&trace_refs, \"fsck\\n\");\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +458,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex cb752d32b6..e965345ad8 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3402,6 +3402,17 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o)\n+{\n+\tint ret;\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\treturn ret;\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_init,\n@@ -3428,5 +3439,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex c4c1e36aa2..db152053f8 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1733,6 +1733,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_init,\n@@ -1760,4 +1766,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex cbcb6f9c36..8f42f21e77 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_refs_options;\n struct ref_transaction;\n \n /*\n@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -677,6 +681,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex e555be4671..7f606faa9e 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2242,6 +2242,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_refs_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2269,4 +2275,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.45.2\n\n"},{"id":"497000","messageId":"20240612085349.710785-4-shejialuo@gmail.com","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[GSoC][PATCH v2 3/7] builtin/refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-12T08:53:45Z","receivedAt":"2024-06-12T08:54:16Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\ncheck the reference database consistency.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/git-refs.txt | 11 ++++++++++\n builtin/refs.c | 45 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 56 insertions(+)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..f9d36ea19d 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,13 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+--strict::\n+\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n+\tbe seen as ERROR.\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..82ed0d57c1 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,55 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tint ret = 0;\n+\tunsigned int verbose = 0, strict = 0;\n+\tstruct fsck_refs_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n+\tstruct option options[] = {\n+\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"too many arguments\"));\n+\n+\tif (verbose)\n+\t\tfsck_refs_options.verbose = 1;\n+\tif (strict)\n+\t\tfsck_refs_options.strict = 1;\n+\n+\tgit_config(git_fsck_refs_config, &fsck_refs_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n+\n+\t/*\n+\t * Explicitly free the allocated array. This is necessary because\n+\t * this program is executed as child process of git-fsck(1) and the\n+\t * allocated array may not freed when git-fsck(1) aborts somewhere.\n+\t */\n+\tfree(fsck_refs_options.msg_type);\n+\treturn ret;\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \n-- \n2.45.2\n\n"},{"id":"497001","messageId":"20240612085349.710785-5-shejialuo@gmail.com","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[GSoC][PATCH v2 4/7] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-12T08:53:46Z","receivedAt":"2024-06-12T08:54:20Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new function \"fsck_refs\" that initializes and runs a child\nprocess to execute the \"git-refs verify\" command.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..10d73f534f 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -896,6 +896,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \treturn res;\n }\n \n+static void fsck_refs(void)\n+{\n+\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n+\tchild_process_init(&refs_verify);\n+\trefs_verify.git_cmd = 1;\n+\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n+\tif (verbose)\n+\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n+\tif (check_strict)\n+\t\tstrvec_push(&refs_verify.args, \"--strict\");\n+\n+\tif (run_command(&refs_verify))\n+\t\terrors_found |= ERROR_REFS;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n@@ -1065,6 +1080,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n+\tfsck_refs();\n+\n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \n-- \n2.45.2\n\n"},{"id":"497002","messageId":"20240612085349.710785-6-shejialuo@gmail.com","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[GSoC][PATCH v2 5/7] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-12T08:53:47Z","receivedAt":"2024-06-12T08:54:23Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"For refs and reflogs, we need to scan its corresponding directories to\ncheck every regular file or symbolic link which shares the same pattern.\nIntroduce a unified interface for scanning directories for\nfiles-backend.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs/files-backend.c | 75 ++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 75 insertions(+)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex e965345ad8..b26cfb8ba6 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -4,6 +4,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -3402,6 +3403,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct fsck_refs_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_refs_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n+{\n+\tconst char *gitdir = ref_store->gitdir;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\n+\tif (!iter) {\n+\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n+\t\tgoto out;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->verbose)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n+\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n+\t\t\t\t iter->basename);\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n+\n+out:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_refs_options *o)\n+{\n+\tint ret;\n+\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tNULL\n+\t};\n+\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, \"Checking references consistency\");\n+\n+\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n+\n+\treturn ret;\n+}\n+\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_refs_options *o)\n {\n@@ -3410,6 +3483,8 @@ static int files_fsck(struct ref_store *ref_store,\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n \tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\tret = files_fsck_refs(ref_store, o);\n+\n \treturn ret;\n }\n \n-- \n2.45.2\n\n"},{"id":"497003","messageId":"20240612085349.710785-7-shejialuo@gmail.com","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[GSoC][PATCH v2 6/7] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-12T08:53:48Z","receivedAt":"2024-06-12T08:54:27Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) only implicitly checks the reference, it does not fully\ncheck refs with bad format name such as standalone \"@\" and name ending\nwith \".lock\".\n\nIn order to provide such checks, add a new fsck message id \"badRefName\"\nwith default ERROR type. Use existing \"check_refname_format\" to explicit\ncheck the ref name. And add a new unit test to verify the functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 +\n fsck.h | 1 +\n refs/files-backend.c | 20 +++++++\n t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++\n 4 files changed, 125 insertions(+)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex 5edc06c658..cc85c897cc 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,9 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefName`::\n+\t(ERROR) A ref has a bad name.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex 7451b1f91b..1423a5e428 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex b26cfb8ba6..266f1ffe8a 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3413,6 +3413,25 @@ typedef int (*files_fsck_refs_fn)(struct fsck_refs_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n+static int files_fsck_refs_name(struct fsck_refs_options *o,\n+\t\t\t\tconst char *gitdir UNUSED,\n+\t\t\t\tconst char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\tret = fsck_refs_report(o, sb.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_refs_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3464,6 +3483,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n {\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tfiles_fsck_refs_name,\n \t\tNULL\n \t};\n \ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..b2db58d2c6\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit tag multi_hierarchy/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1.lock: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/@ &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\tgit -c fsck.badRefName=warn fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\tgit -c fsck.badRefName=ignore fsck 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+test_done\n-- \n2.45.2\n\n"},{"id":"497004","messageId":"20240612085349.710785-8-shejialuo@gmail.com","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[GSoC][PATCH v2 7/7] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-12T08:53:49Z","receivedAt":"2024-06-12T08:54:30Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Enhance the git-fsck(1) command by adding a check for reference content\nin the files backend. The new functionality ensures that symrefs, real\nsymbolic link and regular refs are validated correctly.\n\nAdd a new function \"files_fsck_symref\" to check whether the symrefs and\nsymbolic link points to a valid object and a new function\n\"files_fsck_refs_content\" handles both regular refs and symbolic refs.\n\nIn order to check the trailing content, add a new parameter\n\"trailing\" to \"parse_loose_ref_contents\" function.\n\nLast, add the following FSCK MESSAGEs:\n\n1. \"badRefSha(ERROR)\": A ref has a bad sha.\n2. \"danglingSymre(WARN)\": Found a dangling symref.\n3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 9 +++\n fsck.h | 3 +\n refs.c | 2 +-\n refs/files-backend.c | 124 +++++++++++++++++++++++++++++++++-\n refs/refs-internal.h | 5 +-\n t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++++++\n 6 files changed, 248 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex cc85c897cc..69f86c5345 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -22,6 +22,9 @@\n `badRefName`::\n \t(ERROR) A ref has a bad name.\n \n+`badRefSha`::\n+\t(ERROR) A ref has a bad sha.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \n@@ -37,6 +40,9 @@\n `badType`::\n \t(ERROR) Found an invalid object type.\n \n+`danglingSymref`::\n+\t(WARN) Found a dangling symref.\n+\n `duplicateEntries`::\n \t(ERROR) A tree contains duplicate file entries.\n \n@@ -179,6 +185,9 @@\n `symlinkTargetMissing`::\n \t(ERROR) Unable to read symbolic link target's blob.\n \n+`trailingRefContent`::\n+\t(WARN) A ref content has trailing contents.\n+\n `treeNotSorted`::\n \t(ERROR) A tree is not properly sorted.\n \ndiff --git a/fsck.h b/fsck.h\nindex 1423a5e428..5a55a567b0 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -32,6 +32,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n+\tFUNC(BAD_REF_SHA, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n@@ -69,11 +70,13 @@ enum fsck_msg_type {\n \tFUNC(SYMLINK_TARGET_BLOB, ERROR) \\\n \t/* warnings */ \\\n \tFUNC(EMPTY_NAME, WARN) \\\n+\tFUNC(DANGLING_SYMREF, WARN) \\\n \tFUNC(FULL_PATHNAME, WARN) \\\n \tFUNC(HAS_DOT, WARN) \\\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n+\tFUNC(TRAILING_REF_CONTENT, WARN) \\\n \tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n \tFUNC(LARGE_PATHNAME, WARN) \\\ndiff --git a/refs.c b/refs.c\nindex 0922439275..1325f83269 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1744,7 +1744,7 @@ static int refs_read_special_head(struct ref_store *ref_store,\n \t}\n \n \tresult = parse_loose_ref_contents(content.buf, oid, referent, type,\n-\t\t\t\t\t failure_errno);\n+\t\t\t\t\t failure_errno, NULL);\n \n done:\n \tstrbuf_release(&full_path);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 266f1ffe8a..17d3e433f1 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -549,7 +549,7 @@ static int read_ref_internal(struct ref_store *ref_store, const char *refname,\n \tstrbuf_rtrim(&sb_contents);\n \tbuf = sb_contents.buf;\n \n-\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr);\n+\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr, NULL);\n \n out:\n \tif (ret && !myerr)\n@@ -585,7 +585,7 @@ static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refn\n \n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno)\n+\t\t\t int *failure_errno, unsigned int *trailing)\n {\n \tconst char *p;\n \tif (skip_prefix(buf, \"ref:\", &buf)) {\n@@ -607,6 +607,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t*failure_errno = EINVAL;\n \t\treturn -1;\n \t}\n+\n+\tif (trailing && (*p != '\\0' && *p != '\\n'))\n+\t\t*trailing = 1;\n+\n \treturn 0;\n }\n \n@@ -3432,6 +3436,121 @@ static int files_fsck_refs_name(struct fsck_refs_options *o,\n \treturn ret;\n }\n \n+static int files_fsck_symref(struct fsck_refs_options *o,\n+\t\t\t struct strbuf *refname,\n+\t\t\t struct strbuf *path)\n+{\n+\tstruct stat st;\n+\tint ret = 0;\n+\n+\tif (lstat(path->buf, &st) < 0) {\n+\t\tret = fsck_refs_report(o, refname->buf,\n+\t\t\t\t FSCK_MSG_DANGLING_SYMREF,\n+\t\t\t\t \"point to non-existent ref\");\n+\t\tgoto out;\n+\t}\n+\n+\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n+\t\tret = fsck_refs_report(o, refname->buf,\n+\t\t\t\t FSCK_MSG_DANGLING_SYMREF,\n+\t\t\t\t \"point to invalid object\");\n+\t\tgoto out;\n+\t}\n+out:\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs_content(struct fsck_refs_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter)\n+{\n+\tstruct strbuf path = STRBUF_INIT,\n+\t\t refname = STRBUF_INIT,\n+\t\t ref_content = STRBUF_INIT,\n+\t\t referent = STRBUF_INIT;\n+\tunsigned int trailing = 0;\n+\tint failure_errno = 0;\n+\tunsigned int type = 0;\n+\tstruct object_id oid;\n+\tint ret = 0;\n+\n+\tstrbuf_addbuf(&path, &iter->path);\n+\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\n+\t/*\n+\t * If the file is a symlink, we need to only check the connectivity\n+\t * of the destination object.\n+\t */\n+\tif (S_ISLNK(iter->st.st_mode)) {\n+\t\tstrbuf_strip_file_from_path(&path);\n+\t\tret = strbuf_readlink(&ref_content,\n+\t\t\t\t iter->path.buf, iter->st.st_size);\n+\t\tif (ret < 0) {\n+\t\t\tret = error_errno(_(\"could not read link '%s'\"),\n+\t\t\t\t\t iter->path.buf);\n+\t\t\tgoto clean;\n+\t\t}\n+\t\tstrbuf_addbuf(&path, &ref_content);\n+\t\tstrbuf_reset(&ref_content);\n+\n+\t\tret = files_fsck_symref(o, &refname, &path);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (strbuf_read_file(&ref_content, path.buf, 0) < 0) {\n+\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n+\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n+\t\t\t\t &referent, &type,\n+\t\t\t\t &failure_errno, &trailing)) {\n+\t\tret = fsck_refs_report(o, refname.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_SHA,\n+\t\t\t\t \"invalid ref content\");\n+\t\tgoto clean;\n+\t}\n+\n+\t/*\n+\t * If the ref is a symref, we need to check the destination name and\n+\t * connectivity.\n+\t */\n+\tif (referent.len && (type & REF_ISSYMREF)) {\n+\t\tstrbuf_reset(&path);\n+\t\tstrbuf_addf(&path, \"%s/%s\", gitdir, referent.buf);\n+\n+\t\tif (check_refname_format(referent.buf, 0)) {\n+\t\t\tret = fsck_refs_report(o, refname.buf,\n+\t\t\t\t\t FSCK_MSG_DANGLING_SYMREF,\n+\t\t\t\t\t \"point to invalid refname\");\n+\t\t\tgoto clean;\n+\t\t}\n+\n+\t\tret = files_fsck_symref(o, &refname, &path);\n+\t\tgoto clean;\n+\t} else {\n+\t\t/*\n+\t\t * Only regular refs could have a trailing garbage. Should\n+\t\t * be reported as a warning.\n+\t\t */\n+\t\tif (trailing) {\n+\t\t\tret = fsck_refs_report(o, refname.buf,\n+\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n+\t\t\t\t\t \"trailing garbage in ref\");\n+\t\t\tgoto clean;\n+\t\t}\n+\t}\n+\n+clean:\n+\tstrbuf_release(&path);\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&ref_content);\n+\tstrbuf_release(&referent);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_refs_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3484,6 +3603,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n \t\tfiles_fsck_refs_name,\n+\t\tfiles_fsck_refs_content,\n \t\tNULL\n \t};\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 8f42f21e77..eb3a7cdcc1 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -709,11 +709,12 @@ struct ref_store {\n \n /*\n * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for\n- * invalid contents.\n+ * invalid contents. Also trailing is set to 1 when there is any bytes after the\n+ * hex.\n */\n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno);\n+\t\t\t int *failure_errno, unsigned int *trailing);\n \n /*\n * Fill in the generic part of refs and add it to our collection of\ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nindex b2db58d2c6..94cb93bf92 100755\n--- a/t/t0602-reffiles-fsck.sh\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -98,4 +98,114 @@ test_expect_success 'ref name check should be adapted into fsck messages' '\n \t)\n '\n \n+test_expect_success 'regular ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit checkout -b a/b/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse branch-1)\" > $branch_dir_prefix/branch-1-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse tag-1)\" > $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_must_fail git -c fsck.trailingRefContent=error fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s \" \"$(git rev-parse tag-2)\" > $tag_dir_prefix/tag-2-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/tags/tag-2-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-2-bad: badRefSha: invalid ref content\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/a/b/branch-2-bad: badRefSha: invalid ref content\n+\t\tEOF\n+\t\trm $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'symbolic ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads/branch-3\" > $branch_dir_prefix/branch-2-bad &&\n+\t\tgit refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-2-bad: danglingSymref: point to non-existent ref\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n+\t\tgit refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-2-bad: danglingSymref: point to invalid refname\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n+\t\tgit refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-2-bad: danglingSymref: point to invalid object\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.2\n\n"},{"id":"497030","messageId":"ZmmQUUXEeG6d8y2f@ArchLinux","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"Re: [GSoC][PATCH v2 0/7] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-12T12:10:57Z","receivedAt":"2024-06-12T12:11:00Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Sorry for the confusion, I wanna emphasis what this series of patches\nreplies on.\n\nIt is built upon on the this patch.\n\n\thttps://lore.kernel.org/git/cover.1716451672.git.ps@pks.im/T/#t\n\nThanks,\nJialuo\n"},{"id":"497101","messageId":"xmqqr0d0iqey.fsf@gitster.g","threadId":"61565","inReplyTo":"20240612085349.710785-8-shejialuo@gmail.com","subject":"Re: [GSoC][PATCH v2 7/7] fsck: add ref content check for files backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-06-13T19:38:45Z","receivedAt":"2024-06-13T19:38:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"shejialuo writes:\n\n> In order to check the trailing content, add a new parameter\n> \"trailing\" to \"parse_loose_ref_contents\" function.\n\nAbout this one.\n\n> int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n> \t\t\t struct strbuf *referent, unsigned int *type,\n> -\t\t\t int *failure_errno)\n> +\t\t\t int *failure_errno, unsigned int *trailing)\n> {\n> \tconst char *p;\n> \tif (skip_prefix(buf, \"ref:\", &buf)) {\n> @@ -607,6 +607,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n> \t\t*failure_errno = EINVAL;\n> \t\treturn -1;\n> \t}\n> +\n> +\tif (trailing && (*p != '\\0' && *p != '\\n'))\n> +\t\t*trailing = 1;\n> +\n> \treturn 0;\n> }\n\nWe know what the garbage looked like at this point. The caller owns\nthe \"buf\" pointer and we are pointing into that buffer with the\npointer p, and the garbage is right there.\n\nSo I am not sure if losing information by using \"uint *\" is a good\nidea. Wouldn't it make more sense to take \"const char **trailing\"\nas a parameter and tell the caller where the trailing junk begins?\n\n> +static int files_fsck_symref(struct fsck_refs_options *o,\n> +\t\t\t struct strbuf *refname,\n> +\t\t\t struct strbuf *path)\n\nThis does not take things like HEAD or refs/remotes/origin/HEAD to\nvalidate. Instead, the caller is responsible for either doing a\nreadlink on a symbolic link, or reading a textual symref and\nstripping \"ref: \" prefix from it, before calling this function.\nThe \"refname\" parameter is not HEAD or refs/remotes/origin/HEAD but\nthe pointee of the symref.\n\nSo I'd imagine that renaming it to fsck_symref_target or along that\nline to clarify that we are not checking the symref, but the target\nof a symref, would be a good idea.\n\n> +{\n> +\tstruct stat st;\n> +\tint ret = 0;\n> +\n> +\tif (lstat(path->buf, &st) < 0) {\n> +\t\tret = fsck_refs_report(o, refname->buf,\n> +\t\t\t\t FSCK_MSG_DANGLING_SYMREF,\n> +\t\t\t\t \"point to non-existent ref\");\n> +\t\tgoto out;\n> +\t}\n\nIs that an error? Just like being on an unborn branch is not an\nerror, it could be argued that a symref that points at a branch yet\nto be born wouldn't be an error, either, no?\n\n> +\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n> +\t\tret = fsck_refs_report(o, refname->buf,\n> +\t\t\t\t FSCK_MSG_DANGLING_SYMREF,\n> +\t\t\t\t \"point to invalid object\");\n> +\t\tgoto out;\n\nThe use of \"object\" here is highly misleading. Yes, you can call a\nfilesystem entity like \"a regular file\", \"a directory\", etc. \"an\nobject\", but the word can refer to many other kinds of \"object\". In\nfact, I originally read this to mean \"we are referring to an object\nin the object database that is corrupt\" or something, but of course\nthat is not what we are complaining about. We are complaining that\nthe symbolic link points at a file of wrong type (like a directory).\n\nSo, in short, missing is probably OK. Pointing at a wrong thing\n(like a directory or block device) is not. Where, if any, do we\ncatch a symbolic ref that tries to escape the refs/* hierarchy\n(e.g. \".git/refs/heads/my-crazy-ref\" that is a symbolic link that\npoints at \"../../../../else/where\" that is not even part of the\nrepository), by the way?\n\nThanks.\n"},{"id":"497122","messageId":"ZmvTI73P2fQ6AkOp@ArchLinux","threadId":"61565","inReplyTo":"xmqqr0d0iqey.fsf@gitster.g","subject":"Re: [GSoC][PATCH v2 7/7] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-14T05:20:35Z","receivedAt":"2024-06-14T05:20:38Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Thu, Jun 13, 2024 at 12:38:45PM -0700, Junio C Hamano wrote:\n> shejialuo writes:\n> \n> > In order to check the trailing content, add a new parameter\n> > \"trailing\" to \"parse_loose_ref_contents\" function.\n> \n> About this one.\n> \n> > int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n> > \t\t\t struct strbuf *referent, unsigned int *type,\n> > -\t\t\t int *failure_errno)\n> > +\t\t\t int *failure_errno, unsigned int *trailing)\n> > {\n> > \tconst char *p;\n> > \tif (skip_prefix(buf, \"ref:\", &buf)) {\n> > @@ -607,6 +607,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n> > \t\t*failure_errno = EINVAL;\n> > \t\treturn -1;\n> > \t}\n> > +\n> > +\tif (trailing && (*p != '\\0' && *p != '\\n'))\n> > +\t\t*trailing = 1;\n> > +\n> > \treturn 0;\n> > }\n> \n> We know what the garbage looked like at this point. The caller owns\n> the \"buf\" pointer and we are pointing into that buffer with the\n> pointer p, and the garbage is right there.\n> \n> So I am not sure if losing information by using \"uint *\" is a good\n> idea. Wouldn't it make more sense to take \"const char **trailing\"\n> as a parameter and tell the caller where the trailing junk begins?\n> \n\nYes, I totally agree that using \"uint *\" will lose a lot of information\nhere. Actually I have used the \"const char **trailing\", but I made a\nmistake to result the wild pointer. This is because when\n\"parse_loose_ref_contents\" handles symref, it will never handle `*p`.\nWhen the caller defines `const char *trailing`, it will be wild pointer.\nBut I think we could set it to `NULL` when handling symref.\n\nI will change the code in the next version.\n\n> > +static int files_fsck_symref(struct fsck_refs_options *o,\n> > +\t\t\t struct strbuf *refname,\n> > +\t\t\t struct strbuf *path)\n> \n> This does not take things like HEAD or refs/remotes/origin/HEAD to\n> validate. Instead, the caller is responsible for either doing a\n> readlink on a symbolic link, or reading a textual symref and\n> stripping \"ref: \" prefix from it, before calling this function.\n> The \"refname\" parameter is not HEAD or refs/remotes/origin/HEAD but\n> the pointee of the symref.\n> \n> So I'd imagine that renaming it to fsck_symref_target or along that\n> line to clarify that we are not checking the symref, but the target\n> of a symref, would be a good idea.\n> \n\nThat's not correct. The \"refname\" parameter is EXACTLY the symref\nitself. What we do here is to check the \"path\" paramteter, there are two\nsituations:\n\n1. For symref we will strip \"ref: \" prefix, and combine the girdir and\nthe stripped content to get the \"path\" parameter.\n2. For symbolic we will get its actual path (here I made a mistake, I\ntotally forget the situation when it points to absolute path, I will\nrevise the code for handling it).\n\nThe design here is just check whether the symref points to the correct\nthing. It does not care about the pointee. The code will traverse every\nregular file under the \"refs/\" directory, eventually we will check the\n\"pointee\" status. For example, a symref \"sym-branch\" and a regular ref\n\"branch\".\n\n sym-branch: \"ref: refs/heads/branch\".\n branch: \"xxxx\"\n\nThe design will not report any error for \"sym-branch\". I think we should\ndiscuss here whether this design is OK.\n\n> > +{\n> > +\tstruct stat st;\n> > +\tint ret = 0;\n> > +\n> > +\tif (lstat(path->buf, &st) < 0) {\n> > +\t\tret = fsck_refs_report(o, refname->buf,\n> > +\t\t\t\t FSCK_MSG_DANGLING_SYMREF,\n> > +\t\t\t\t \"point to non-existent ref\");\n> > +\t\tgoto out;\n> > +\t}\n> \n> Is that an error? Just like being on an unborn branch is not an\n> error, it could be argued that a symref that points at a branch yet\n> to be born wouldn't be an error, either, no?\n> \n\nThe reason why I choose \"danglingSymref\" and warn severity is that I let\nthe code be align with \"git checkout\". When we use \"git checkout\" for a\ndangling symref. It would produce the following output:\n\n $ git checkout branch-3\n warning: ignoring dangling symref refs/heads/branch-3\n error: pathspec 'branch-3' did not match any file(s) known to git\n\nSo I prefer to warn severity.\n\n> > +\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n> > +\t\tret = fsck_refs_report(o, refname->buf,\n> > +\t\t\t\t FSCK_MSG_DANGLING_SYMREF,\n> > +\t\t\t\t \"point to invalid object\");\n> > +\t\tgoto out;\n> \n> The use of \"object\" here is highly misleading. Yes, you can call a\n> filesystem entity like \"a regular file\", \"a directory\", etc. \"an\n> object\", but the word can refer to many other kinds of \"object\". In\n> fact, I originally read this to mean \"we are referring to an object\n> in the object database that is corrupt\" or something, but of course\n> that is not what we are complaining about. We are complaining that\n> the symbolic link points at a file of wrong type (like a directory).\n> \n\nYes, it brings a lot of misleading here. I will clean the code and\ncommit message (I also used object in commit message).\n\n> So, in short, missing is probably OK. Pointing at a wrong thing\n> (like a directory or block device) is not. Where, if any, do we\n> catch a symbolic ref that tries to escape the refs/* hierarchy\n> (e.g. \".git/refs/heads/my-crazy-ref\" that is a symbolic link that\n> points at \"../../../../else/where\" that is not even part of the\n> repository), by the way?\n> \n\nI intentionally ignored the \"escape\" situation. Actually, the path could\nbe either absolute or relative. It may be a little complicated. I will\nfind a way to support this in the next version.\n\n> Thanks.\n\nThanks,\nJialuo\n"},{"id":"497173","messageId":"xmqqo783im5k.fsf@gitster.g","threadId":"61565","inReplyTo":"ZmvTI73P2fQ6AkOp@ArchLinux","subject":"Re: [GSoC][PATCH v2 7/7] fsck: add ref content check for files backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-06-14T15:23:03Z","receivedAt":"2024-06-14T15:23:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"shejialuo writes:\n\n>> > +static int files_fsck_symref(struct fsck_refs_options *o,\n>> > +\t\t\t struct strbuf *refname,\n>> > +\t\t\t struct strbuf *path)\n>> \n>> This does not take things like HEAD or refs/remotes/origin/HEAD to\n>> validate. Instead, the caller is responsible for either doing a\n>> readlink on a symbolic link, or reading a textual symref and\n>> stripping \"ref: \" prefix from it, before calling this function.\n>> The \"refname\" parameter is not HEAD or refs/remotes/origin/HEAD but\n>> the pointee of the symref.\n>> \n>> So I'd imagine that renaming it to fsck_symref_target or along that\n>> line to clarify that we are not checking the symref, but the target\n>> of a symref, would be a good idea.\n>\n> That's not correct. The \"refname\" parameter is EXACTLY the symref\n> itself.\n\nYeah, but the story is the same. We are not really checking\nanything about the symref (i.e. the thing in \"refname\") being funny.\nWe are checking what is in \"path\" (like \"does it exist?\") and the\n\"refname\" is there only for reporting purposes (i.e. \"we have a\nsymbolic ref REFNAME, that points at PATH which is a wrong thing\").\n\n>> > +{\n>> > +\tstruct stat st;\n>> > +\tint ret = 0;\n>> > +\n>> > +\tif (lstat(path->buf, &st) < 0) {\n>> > +\t\tret = fsck_refs_report(o, refname->buf,\n>> > +\t\t\t\t FSCK_MSG_DANGLING_SYMREF,\n>> > +\t\t\t\t \"point to non-existent ref\");\n>> > +\t\tgoto out;\n>> > +\t}\n>> \n>> Is that an error? Just like being on an unborn branch is not an\n>> error, it could be argued that a symref that points at a branch yet\n>> to be born wouldn't be an error, either, no?\n>> \n>\n> The reason why I choose \"danglingSymref\" and warn severity is that I let\n> the code be align with \"git checkout\". When we use \"git checkout\" for a\n> dangling symref. It would produce the following output:\n>\n> $ git checkout branch-3\n> warning: ignoring dangling symref refs/heads/branch-3\n> error: pathspec 'branch-3' did not match any file(s) known to git\n>\n> So I prefer to warn severity.\n\nIf you do this from that situation, \n\n $ git branch branch-3 master\n\nwhat happens is that the pointee of branch-3 is created at the\ncommit pointed at by 'master'. No error. No warnings.\n\nIn a freshly created respository, HEAD is a dangling symbolic ref,\nand that is not an error. You can create a root commit from there\njust fine.\n\nIf there is anything that needs improvement in your example, it is\nthat \"checkout branch-3\" should be taught to either (1) not warn\nabout dangling symbolic link and just give the error, which is in\nline with how \"git checkout HEAD\" in a freshly created repository\nbehaves, or (2) just like unborn 'master' pointed at by 'HEAD' is\nperfectly happy to be checked out, allow the unborn 'branch-3' to be\npointed at by 'HEAD', and arrange the first commit (which will be a\nroot commit) you create in that state to be pointed by the ref\n'branch-3' points at.\n\nSo from all of these reasons, I do not think missing target should\nbe treated as any error worthy event. Not even warning.\n\nOn the other hand, the target of the symref in \"path\" must be\nchecked, even if it does not currently exist, for its validity, the\nsame way an existing ref gets checked (lives inside refs/, passes\ncheck-ref-format, etc.).\n\n> I intentionally ignored the \"escape\" situation. Actually, the path could\n> be either absolute or relative. It may be a little complicated. I will\n> find a way to support this in the next version.\n\nYes, if this wants to claim to be part of \"FSCK\", it should catch\nall the errors the regular runtime would complain about, and\n\"escape\" thing is one of the first things that you need to get\nright. Whatever refs.c:read_ref_internal() for S_ISLNK(st.st_mode)\ncase takes as legit should be considered legit, the \"fallback - read\nthrough the symlink as if it were a non-symlink\" case probably wants\nto be warned. refs.c:resolve_ref_unsafe(), which is used at the low\nlevel from object-name.c:get_oid_basic() via refs.c:repo_dwim_ref(),\nhas further checks to see if a refname that a symref resolves to is\nvalid and the runtime sanity relies on these checks.\n\nThanks.\n\n"},{"id":"497180","messageId":"CAH-kW5f0u1uFH7Y83e4Vp8avM_7MPLkDG7v9NPV1nL=wZ--wmw@mail.gmail.com","threadId":"61565","inReplyTo":"xmqqo783im5k.fsf@gitster.g","subject":"Re: [GSoC][PATCH v2 7/7] fsck: add ref content check for files backend","fromName":"jialuo she","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-14T17:05:43Z","receivedAt":"2024-06-14T17:05:57Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Junio C Hamano writes：\n>\n> shejialuo writes:\n>\n> >> > +static int files_fsck_symref(struct fsck_refs_options *o,\n> >> > + struct strbuf *refname,\n> >> > + struct strbuf *path)\n> >>\n> >> This does not take things like HEAD or refs/remotes/origin/HEAD to\n> >> validate. Instead, the caller is responsible for either doing a\n> >> readlink on a symbolic link, or reading a textual symref and\n> >> stripping \"ref: \" prefix from it, before calling this function.\n> >> The \"refname\" parameter is not HEAD or refs/remotes/origin/HEAD but\n> >> the pointee of the symref.\n> >>\n> >> So I'd imagine that renaming it to fsck_symref_target or along that\n> >> line to clarify that we are not checking the symref, but the target\n> >> of a symref, would be a good idea.\n> >\n> > That's not correct. The \"refname\" parameter is EXACTLY the symref\n> > itself.\n>\n> Yeah, but the story is the same. We are not really checking\n> anything about the symref (i.e. the thing in \"refname\") being funny.\n> We are checking what is in \"path\" (like \"does it exist?\") and the\n> \"refname\" is there only for reporting purposes (i.e. \"we have a\n> symbolic ref REFNAME, that points at PATH which is a wrong thing\").\n>\n\nYes, I agree, will rename the function in the next version.\n\n> >> > +{\n> >> > + struct stat st;\n> >> > + int ret = 0;\n> >> > +\n> >> > + if (lstat(path->buf, &st) < 0) {\n> >> > + ret = fsck_refs_report(o, refname->buf,\n> >> > + FSCK_MSG_DANGLING_SYMREF,\n> >> > + \"point to non-existent ref\");\n> >> > + goto out;\n> >> > + }\n> >>\n> >> Is that an error? Just like being on an unborn branch is not an\n> >> error, it could be argued that a symref that points at a branch yet\n> >> to be born wouldn't be an error, either, no?\n> >>\n> >\n> > The reason why I choose \"danglingSymref\" and warn severity is that I let\n> > the code be align with \"git checkout\". When we use \"git checkout\" for a\n> > dangling symref. It would produce the following output:\n> >\n> > $ git checkout branch-3\n> > warning: ignoring dangling symref refs/heads/branch-3\n> > error: pathspec 'branch-3' did not match any file(s) known to git\n> >\n> > So I prefer to warn severity.\n>\n> If you do this from that situation,\n>\n> $ git branch branch-3 master\n>\n> what happens is that the pointee of branch-3 is created at the\n> commit pointed at by 'master'. No error. No warnings.\n>\n> In a freshly created respository, HEAD is a dangling symbolic ref,\n> and that is not an error. You can create a root commit from there\n> just fine.\n>\n\nI am totally shocked by the fact that it will create the pointee file. I am\na little curious about the design here. I know when creating a new repo,\nHEAD is a dangling symbolic ref. When we do first commit, it will\ncreate the pointee file. But HEAD is a special file, it should be treated\ndifferently. Why we treat the other symrefs the same way.\n\n> If there is anything that needs improvement in your example, it is\n> that \"checkout branch-3\" should be taught to either (1) not warn\n> about dangling symbolic link and just give the error, which is in\n> line with how \"git checkout HEAD\" in a freshly created repository\n> behaves, or (2) just like unborn 'master' pointed at by 'HEAD' is\n> perfectly happy to be checked out, allow the unborn 'branch-3' to be\n> pointed at by 'HEAD', and arrange the first commit (which will be a\n> root commit) you create in that state to be pointed by the ref\n> 'branch-3' points at.\n>\n\nShould we really improve this? From my perspective, a user will get\nmore information when this warn message shows up. The\n\"error: pathspec 'branch-3' did not match any file(s) known to git\" is\nless informative.\n\nMaybe the pro git users do not use git command to write to the git file\nsystem. However, this is a bad way to do this. We always want to\nthe user to use the command to handle with git file system.\n\nInstead, we should warn about the user when using\n\n $ git branch branch-3 master\n\nIt will create a new pointee here, I suppose we should warn the user\nhere except HEAD ref. The user could get the dangling symref by\ndeleting the pointee ref. For example:\n\n $ git symbolic-ref refs/heads/main master\n $ git checkout -b master-copy\n $ git branch --delete master\n\nShould we warn this for user in git-fsck(1)?\n\n> So from all of these reasons, I do not think missing target should\n> be treated as any error worthy event. Not even warning.\n>\n\nYes, if git-branch implicitly creates the pointee ref file. We should not\nhandle such case. But I think giving a warning here except HEAD\nref is worthful.\n\n> On the other hand, the target of the symref in \"path\" must be\n> checked, even if it does not currently exist, for its validity, the\n> same way an existing ref gets checked (lives inside refs/, passes\n> check-ref-format, etc.).\n>\n\nYes!\n\n> > I intentionally ignored the \"escape\" situation. Actually, the path could\n> > be either absolute or relative. It may be a little complicated. I will\n> > find a way to support this in the next version.\n>\n> Yes, if this wants to claim to be part of \"FSCK\", it should catch\n> all the errors the regular runtime would complain about, and\n> \"escape\" thing is one of the first things that you need to get\n> right. Whatever refs.c:read_ref_internal() for S_ISLNK(st.st_mode)\n> case takes as legit should be considered legit, the \"fallback - read\n> through the symlink as if it were a non-symlink\" case probably wants\n> to be warned. refs.c:resolve_ref_unsafe(), which is used at the low\n> level from object-name.c:get_oid_basic() via refs.c:repo_dwim_ref(),\n> has further checks to see if a refname that a symref resolves to is\n> valid and the runtime sanity relies on these checks.\n>\n\nThanks for your help! I will look at these codes to implement the\n\"escape\" check.\n\n> Thanks.\n>\n"},{"id":"497187","messageId":"xmqq1q4zfj4r.fsf@gitster.g","threadId":"61565","inReplyTo":"CAH-kW5f0u1uFH7Y83e4Vp8avM_7MPLkDG7v9NPV1nL=wZ--wmw@mail.gmail.com","subject":"Re: [GSoC][PATCH v2 7/7] fsck: add ref content check for files backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-06-14T18:56:36Z","receivedAt":"2024-06-14T18:56:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"jialuo she writes:\n\n> Instead, we should warn about the user when using\n>\n> $ git branch branch-3 master\n>\n> It will create a new pointee here, I suppose we should warn the user\n> here except HEAD ref.\n\nThe user created branch-3 as a symlink, fully expecting that the\npointee will be updated.\n\nBy the way, a seemingly normal branch looking symref is very handy\nand that is what my \"maint\" branch looks like. Right now it is\npointing at maint-2.45 (because 2.45.0 was the last feature release)\n\n $ git symbolic-ref refs/heads/maint\n refs/heads/maint-2.45\n\nand I would imagine many other users are using the trick to maintain\nmultiple maintenance tracks. They would be upset if their next \"Now\n2.46.0 release is done, let's start the 2.46 maintenance track\"\nmantra, which would look like so:\n\n\t$ git symbolic-ref refs/heads/maint refs/heads/maint-2.46\n\t$ git branch maint v2.46.0\n\nstarts to give an unnecessary warning.\n"},{"id":"497211","messageId":"CAOLa=ZQtAOnD8kX4d5C0WKFOjqU9tgeWtHp-aXu+rDe3gmK4yA@mail.gmail.com","threadId":"61565","inReplyTo":"20240612085349.710785-3-shejialuo@gmail.com","subject":"Re: [GSoC][PATCH v2 2/7] refs: set up ref consistency check infrastructure","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-06-15T14:50:05Z","receivedAt":"2024-06-15T14:50:07Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n[snip]\n\n> diff --git a/refs/debug.c b/refs/debug.c\n> index 547d9245b9..110a264522 100644\n> --- a/refs/debug.c\n> +++ b/refs/debug.c\n> @@ -419,6 +419,13 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n> \treturn res;\n> }\n>\n> +static int debug_fsck(struct ref_store *ref_store,\n> +\t\t struct fsck_refs_options *o)\n> +{\n> +\ttrace_printf_key(&trace_refs, \"fsck\\n\");\n> +\treturn 0;\n> +}\n> +\n>\n\nThe debug interface acts as a wrapper around the actual refs backend\nbeing used, as such, you need to still call the underlying backend here.\n\n[snip]\n"},{"id":"497212","messageId":"CAOLa=ZS+3_=Q=KNdauWrU3javdDNneHPaS5KaoM6iCTLP8O8mA@mail.gmail.com","threadId":"61565","inReplyTo":"20240612085349.710785-4-shejialuo@gmail.com","subject":"Re: [GSoC][PATCH v2 3/7] builtin/refs: add verify subcommand","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-06-15T14:57:31Z","receivedAt":"2024-06-15T14:57:35Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\n> check the reference database consistency.\n>\n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n> Documentation/git-refs.txt | 11 ++++++++++\n> builtin/refs.c | 45 ++++++++++++++++++++++++++++++++++++++\n> 2 files changed, 56 insertions(+)\n>\n> diff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\n> index 5b99e04385..f9d36ea19d 100644\n> --- a/Documentation/git-refs.txt\n> +++ b/Documentation/git-refs.txt\n> @@ -10,6 +10,7 @@ SYNOPSIS\n> --------\n> [verse]\n> 'git refs migrate' --ref-format= [--dry-run]\n> +'git refs verify' [--strict] [--verbose]\n>\n> DESCRIPTION\n> -----------\n> @@ -22,6 +23,9 @@ COMMANDS\n> migrate::\n> \tMigrate ref store between different formats.\n>\n> +verify::\n> +\tVerify reference database consistency.\n> +\n> OPTIONS\n> -------\n>\n> @@ -39,6 +43,13 @@ include::ref-storage-format.txt[]\n> \tcan be used to double check that the migration works as expected before\n> \tperforming the actual migration.\n>\n\nAbove it states that the options mentioned are \"specific to 'git refs\nmigrate'\", shouldn't we do something similar here?\n\n> +--strict::\n> +\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n> +\tbe seen as ERROR.\n> +\n> +--verbose::\n> +\tWhen verifying the reference database consistency, be chatty.\n> +\n> KNOWN LIMITATIONS\n> -----------------\n>\n> diff --git a/builtin/refs.c b/builtin/refs.c\n> index 46dcd150d4..82ed0d57c1 100644\n> --- a/builtin/refs.c\n> +++ b/builtin/refs.c\n> @@ -1,4 +1,6 @@\n> #include \"builtin.h\"\n> +#include \"config.h\"\n> +#include \"fsck.h\"\n> #include \"parse-options.h\"\n> #include \"refs.h\"\n> #include \"repository.h\"\n> @@ -7,6 +9,9 @@\n> #define REFS_MIGRATE_USAGE \\\n> \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n>\n> +#define REFS_VERIFY_USAGE \\\n> +\tN_(\"git refs verify [--strict] [--verbose]\")\n> +\n> static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n> {\n> \tconst char * const migrate_usage[] = {\n> @@ -58,15 +63,55 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n> \treturn err;\n> }\n>\n> +static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n> +{\n> +\tconst char * const verify_usage[] = {\n> +\t\tREFS_VERIFY_USAGE,\n> +\t\tNULL,\n> +\t};\n> +\tint ret = 0;\n> +\tunsigned int verbose = 0, strict = 0;\n> +\tstruct fsck_refs_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n> +\tstruct option options[] = {\n> +\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n> +\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n> +\t\tOPT_END(),\n> +\t};\n> +\n> +\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n> +\tif (argc)\n> +\t\tusage(_(\"too many arguments\"));\n> +\n> +\tif (verbose)\n> +\t\tfsck_refs_options.verbose = 1;\n> +\tif (strict)\n> +\t\tfsck_refs_options.strict = 1;\n> +\n> +\tgit_config(git_fsck_refs_config, &fsck_refs_options);\n> +\tprepare_repo_settings(the_repository);\n> +\n> +\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n> +\n> +\t/*\n> +\t * Explicitly free the allocated array. This is necessary because\n> +\t * this program is executed as child process of git-fsck(1) and the\n> +\t * allocated array may not freed when git-fsck(1) aborts somewhere.\n> +\t */\n> +\tfree(fsck_refs_options.msg_type);\n\nNice, this is a good thing to do even if this isn't called as a child\nprocess.\n\n> +\treturn ret;\n> +}\n> +\n> int cmd_refs(int argc, const char **argv, const char *prefix)\n> {\n> \tconst char * const refs_usage[] = {\n> \t\tREFS_MIGRATE_USAGE,\n> +\t\tREFS_VERIFY_USAGE,\n> \t\tNULL,\n> \t};\n> \tparse_opt_subcommand_fn *fn = NULL;\n> \tstruct option opts[] = {\n> \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n> +\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n> \t\tOPT_END(),\n> \t};\n>\n> --\n> 2.45.2\n"},{"id":"497213","messageId":"CAOLa=ZRQX+7gYrrPZXQtTy=yq+oBGvyWGYXtwMZatpm+VSaUyw@mail.gmail.com","threadId":"61565","inReplyTo":"20240612085349.710785-5-shejialuo@gmail.com","subject":"Re: [GSoC][PATCH v2 4/7] builtin/fsck: add `git-refs verify` child process","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-06-15T15:02:12Z","receivedAt":"2024-06-15T15:02:18Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> Introduce a new function \"fsck_refs\" that initializes and runs a child\n> process to execute the \"git-refs verify\" command.\n>\n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n> builtin/fsck.c | 17 +++++++++++++++++\n> 1 file changed, 17 insertions(+)\n>\n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index d13a226c2e..10d73f534f 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -896,6 +896,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n> \treturn res;\n> }\n>\n> +static void fsck_refs(void)\n> +{\n> +\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n> +\tchild_process_init(&refs_verify);\n> +\trefs_verify.git_cmd = 1;\n> +\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n> +\tif (verbose)\n> +\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n> +\tif (check_strict)\n> +\t\tstrvec_push(&refs_verify.args, \"--strict\");\n> +\n> +\tif (run_command(&refs_verify))\n> +\t\terrors_found |= ERROR_REFS;\n> +}\n> +\n\nAt first I thought we need to call `child_process_clear()` here, but\nseems like `run_command` does that internally.\n\n> static char const * const fsck_usage[] = {\n> \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n> \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n> @@ -1065,6 +1080,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n>\n> \tcheck_connectivity();\n>\n> +\tfsck_refs();\n> +\n> \tif (the_repository->settings.core_commit_graph) {\n> \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n>\n> --\n> 2.45.2\n"},{"id":"497217","messageId":"CAOLa=ZQ5msKTHwDou2RmCrUCdfDuYdFwJQx3Q9gK6aAJvGw-pg@mail.gmail.com","threadId":"61565","inReplyTo":"20240612085349.710785-6-shejialuo@gmail.com","subject":"Re: [GSoC][PATCH v2 5/7] files-backend: add unified interface for refs scanning","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-06-15T20:51:14Z","receivedAt":"2024-06-15T20:51:16Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> For refs and reflogs, we need to scan its corresponding directories to\n> check every regular file or symbolic link which shares the same pattern.\n> Introduce a unified interface for scanning directories for\n> files-backend.\n>\n\nHere we talk about reflogs, but we only add a iterator for\n\"$GIT_DIR/refs\". Should we add something for reflogs too?\n\n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n> refs/files-backend.c | 75 ++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 75 insertions(+)\n>\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index e965345ad8..b26cfb8ba6 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -4,6 +4,7 @@\n> #include \"../gettext.h\"\n> #include \"../hash.h\"\n> #include \"../hex.h\"\n> +#include \"../fsck.h\"\n> #include \"../refs.h\"\n> #include \"refs-internal.h\"\n> #include \"ref-cache.h\"\n> @@ -3402,6 +3403,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n> \treturn ret;\n> }\n>\n> +/*\n> + * For refs and reflogs, they share a unified interface when scanning\n> + * the whole directory. This function is used as the callback for each\n> + * regular file or symlink in the directory.\n> + */\n> +typedef int (*files_fsck_refs_fn)(struct fsck_refs_options *o,\n> +\t\t\t\t const char *gitdir,\n> +\t\t\t\t const char *refs_check_dir,\n> +\t\t\t\t struct dir_iterator *iter);\n> +\n> +static int files_fsck_refs_dir(struct ref_store *ref_store,\n> +\t\t\t struct fsck_refs_options *o,\n> +\t\t\t const char *refs_check_dir,\n> +\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n> +{\n> +\tconst char *gitdir = ref_store->gitdir;\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\tstruct dir_iterator *iter;\n> +\tint iter_status;\n> +\tint ret = 0;\n> +\n> +\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n> +\n> +\titer = dir_iterator_begin(sb.buf, 0);\n> +\n> +\tif (!iter) {\n> +\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n> +\t\tgoto out;\n> +\t}\n> +\n> +\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n> +\t\tif (S_ISDIR(iter->st.st_mode)) {\n> +\t\t\tcontinue;\n> +\t\t} else if (S_ISREG(iter->st.st_mode) ||\n> +\t\t\t S_ISLNK(iter->st.st_mode)) {\n> +\t\t\tif (o->verbose)\n> +\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n> +\t\t\t\t\t refs_check_dir, iter->relative_path);\n> +\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n> +\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n> +\t\t\t\t\tret = -1;\n> +\t\t\t}\n> +\t\t} else {\n> +\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n> +\t\t\t\t iter->basename);\n> +\t\t}\n> +\t}\n> +\n> +\tif (iter_status != ITER_DONE)\n> +\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n> +\n> +out:\n> +\tstrbuf_release(&sb);\n> +\treturn ret;\n> +}\n> +\n> +static int files_fsck_refs(struct ref_store *ref_store,\n> +\t\t\t struct fsck_refs_options *o)\n> +{\n> +\tint ret;\n> +\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n> +\t\tNULL\n> +\t};\n> +\n> +\tif (o->verbose)\n> +\t\tfprintf_ln(stderr, \"Checking references consistency\");\n> +\n> +\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n> +\n> +\treturn ret;\n> +}\n\nI'm not fully sure why this needs to be separate from `files_fsck`. Is\nthat because we plan to also introduce `files_fsck_reflogs`?\n\n> +\n> static int files_fsck(struct ref_store *ref_store,\n> \t\t struct fsck_refs_options *o)\n> {\n> @@ -3410,6 +3483,8 @@ static int files_fsck(struct ref_store *ref_store,\n> \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n>\n> \tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n> +\tret = files_fsck_refs(ref_store, o);\n> +\n> \treturn ret;\n> }\n>\n> --\n> 2.45.2\n"},{"id":"497269","messageId":"ZnFByMI1lWUDSxLd@ArchLinux","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[GSoC][PATCH v3 0/7] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-18T08:14:00Z","receivedAt":"2024-06-18T08:14:06Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"This is the third version. Mainly focuses on the following problems:\n\n1. Add \"escape\" check for symrefs and symbolic links.\n2. Resolve the reviews.\n3. Fix some errors in version 2.\n\nshejialuo (7):\n fsck: add refs check interfaces to interface with fsck error levels\n refs: set up ref consistency check infrastructure\n builtin/refs: add verify subcommand\n builtin/fsck: add `git-refs verify` child process\n files-backend: add unified interface for refs scanning\n fsck: add ref name check for files backend\n fsck: add ref content check for files backend\n\n Documentation/fsck-msgids.txt | 12 ++\n Documentation/git-refs.txt | 13 ++\n builtin/fsck.c | 17 +++\n builtin/refs.c | 45 ++++++\n fsck.c | 69 +++++++++\n fsck.h | 49 ++++++-\n refs.c | 7 +-\n refs.h | 8 ++\n refs/debug.c | 11 ++\n refs/files-backend.c | 255 +++++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 ++\n refs/refs-internal.h | 11 +-\n refs/reftable-backend.c | 8 ++\n t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++\n 14 files changed, 716 insertions(+), 8 deletions(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\nRange-diff against v2:\n1: a65de15dc7 ! 1: b8d20c4be9 fsck: add refs check interfaces to interface with fsck error levels\n @@ fsck.h: int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n +struct fsck_refs_options {\n +\tenum fsck_msg_type *msg_type;\n +\tunsigned strict:1;\n ++\n +\tfsck_refs_error error_func;\n +\tunsigned verbose:1;\n +};\n @@ fsck.h: int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n +}\n +\n struct fsck_options {\n ++\t/*\n ++\t * Reorder the fields to allow `fsck_ref_options` to use\n ++\t * the interfaces using `struct fsck_options`.\n ++\t */\n +\tenum fsck_msg_type *msg_type;\n +\tunsigned strict:1;\n ++\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n -\tunsigned strict:1;\n2: a87149069a ! 2: c7ed95d6b4 refs: set up ref consistency check infrastructure\n @@ refs/debug.c: static int debug_reflog_expire(struct ref_store *ref_store, const\n +static int debug_fsck(struct ref_store *ref_store,\n +\t\t struct fsck_refs_options *o)\n +{\n -+\ttrace_printf_key(&trace_refs, \"fsck\\n\");\n -+\treturn 0;\n ++\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n ++\tint res = drefs->refs->be->fsck(drefs->refs, o);\n ++\ttrace_printf_key(&trace_refs, \"fsck: %d\\n\", res);\n ++\treturn res;\n +}\n +\n struct ref_storage_be refs_be_debug = {\n3: e1f1b77527 ! 3: 373f470fdc builtin/refs: add verify subcommand\n @@ Documentation/git-refs.txt: include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n ++The following options are specific to 'git refs verify':\n ++\n +--strict::\n +\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n +\tbe seen as ERROR.\n4: 2b0ee79e78 = 4: 4f8ebea573 builtin/fsck: add `git-refs verify` child process\n5: 6d23eaa1d1 ! 5: 6062b3b453 files-backend: add unified interface for refs scanning\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n \t\t struct fsck_refs_options *o)\n {\n @@ refs/files-backend.c: static int files_fsck(struct ref_store *ref_store,\n + \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n - \tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n -+\tret = files_fsck_refs(ref_store, o);\n +-\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n ++\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o)\n ++\t | files_fsck_refs(ref_store, o);\n +\n \treturn ret;\n }\n6: 8c55ecdd88 = 6: 87a22f4c7f fsck: add ref name check for files backend\n7: aba5f50940 ! 7: b945f61ef1 fsck: add ref content check for files backend\n @@ Commit message\n in the files backend. The new functionality ensures that symrefs, real\n symbolic link and regular refs are validated correctly.\n \n - Add a new function \"files_fsck_symref\" to check whether the symrefs and\n - symbolic link points to a valid object and a new function\n - \"files_fsck_refs_content\" handles both regular refs and symbolic refs.\n + In order to check the trailing content of the regular refs, add a new\n + parameter `trailing` to `parse_loose_ref_contents`.\n \n - In order to check the trailing content, add a new parameter\n - \"trailing\" to \"parse_loose_ref_contents\" function.\n + For symrefs, `parse_loose_ref_contents` will set the \"referent\".\n + However, symbolic link could be either absolute or relative. Use\n + \"strbuf_add_real_path\" to read the symbolic link and convert the\n + relative path to absolute path. Then use \"skip_prefix\" to make it align\n + with symref \"referent\".\n +\n + Thus, the symrefs and symbolic links could share the same interface. Add\n + a new function \"files_fsck_symref_target\" which aims at checking the\n + following things:\n +\n + 1. whether the pointee is under the `refs/` directory.\n + 2. whether the pointee name is correct.\n + 3. whether the pointee path is a wrong type in filesystem.\n \n Last, add the following FSCK MESSAGEs:\n \n - 1. \"badRefSha(ERROR)\": A ref has a bad sha.\n - 2. \"danglingSymre(WARN)\": Found a dangling symref.\n + 1. \"badRefContent(ERROR)\": A ref has a bad content\n + 2. \"badSymrefPointee(ERROR)\": The pointee of a symref is bad.\n 3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n \n Mentored-by: Patrick Steinhardt \n @@ Commit message\n \n ## Documentation/fsck-msgids.txt ##\n @@\n + `badParentSha1`::\n + \t(ERROR) A commit object has a bad parent sha1.\n + \n ++`badRefContent`::\n ++\t(ERROR) A ref has a bad content.\n ++\n `badRefName`::\n \t(ERROR) A ref has a bad name.\n \n -+`badRefSha`::\n -+\t(ERROR) A ref has a bad sha.\n ++`badSymrefPointee`::\n ++\t(ERROR) The pointee of a symref is bad.\n +\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \n -@@\n - `badType`::\n - \t(ERROR) Found an invalid object type.\n - \n -+`danglingSymref`::\n -+\t(WARN) Found a dangling symref.\n -+\n - `duplicateEntries`::\n - \t(ERROR) A tree contains duplicate file entries.\n - \n @@\n `symlinkTargetMissing`::\n \t(ERROR) Unable to read symbolic link target's blob.\n @@ fsck.h: enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n -+\tFUNC(BAD_REF_SHA, ERROR) \\\n ++\tFUNC(BAD_REF_CONTENT, ERROR) \\\n ++\tFUNC(BAD_SYMREF_POINTEE, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n @@ fsck.h: enum fsck_msg_type {\n - \tFUNC(SYMLINK_TARGET_BLOB, ERROR) \\\n - \t/* warnings */ \\\n - \tFUNC(EMPTY_NAME, WARN) \\\n -+\tFUNC(DANGLING_SYMREF, WARN) \\\n - \tFUNC(FULL_PATHNAME, WARN) \\\n - \tFUNC(HAS_DOT, WARN) \\\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n @@ refs.c: static int refs_read_special_head(struct ref_store *ref_store,\n \tstrbuf_release(&full_path);\n \n ## refs/files-backend.c ##\n +@@\n + #include \"../git-compat-util.h\"\n ++#include \"../abspath.h\"\n + #include \"../copy.h\"\n + #include \"../environment.h\"\n + #include \"../gettext.h\"\n @@ refs/files-backend.c: static int read_ref_internal(struct ref_store *ref_store, const char *refname,\n \tstrbuf_rtrim(&sb_contents);\n \tbuf = sb_contents.buf;\n @@ refs/files-backend.c: static int files_read_symbolic_ref(struct ref_store *ref_s\n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n -\t\t\t int *failure_errno)\n -+\t\t\t int *failure_errno, unsigned int *trailing)\n ++\t\t\t int *failure_errno, const char **trailing)\n {\n \tconst char *p;\n \tif (skip_prefix(buf, \"ref:\", &buf)) {\n @@ refs/files-backend.c: int parse_loose_ref_contents(const char *buf, struct objec\n \t\treturn -1;\n \t}\n +\n -+\tif (trailing && (*p != '\\0' && *p != '\\n'))\n -+\t\t*trailing = 1;\n ++\tif (trailing)\n ++\t\t*trailing = p;\n +\n \treturn 0;\n }\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_refs_options *\n \treturn ret;\n }\n \n -+static int files_fsck_symref(struct fsck_refs_options *o,\n -+\t\t\t struct strbuf *refname,\n -+\t\t\t struct strbuf *path)\n ++/*\n ++ * Check the symref \"pointee_name\" and \"pointee_path\". The caller should\n ++ * make sure that \"pointee_path\" is absolute. For symbolic ref, \"pointee_name\"\n ++ * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n ++ * be the relative path agaignst \"gitdir\".\n ++ */\n ++static int files_fsck_symref_target(struct fsck_refs_options *o,\n ++\t\t\t\t const char *refname,\n ++\t\t\t\t const char *pointee_name,\n ++\t\t\t\t const char *pointee_path)\n +{\n ++\tconst char *p = NULL;\n +\tstruct stat st;\n +\tint ret = 0;\n +\n -+\tif (lstat(path->buf, &st) < 0) {\n -+\t\tret = fsck_refs_report(o, refname->buf,\n -+\t\t\t\t FSCK_MSG_DANGLING_SYMREF,\n -+\t\t\t\t \"point to non-existent ref\");\n ++\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n ++\n ++\t\tret = fsck_refs_report(o, refname,\n ++\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n ++\t\t\t\t \"point to target out of refs hierarchy\");\n +\t\tgoto out;\n +\t}\n +\n ++\tif (check_refname_format(pointee_name, 0)) {\n ++\t\tret = fsck_refs_report(o, refname,\n ++\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n ++\t\t\t\t \"point to invalid refname\");\n ++\t}\n ++\n ++\tif (lstat(pointee_path, &st) < 0)\n ++\t\tgoto out;\n ++\n +\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n -+\t\tret = fsck_refs_report(o, refname->buf,\n -+\t\t\t\t FSCK_MSG_DANGLING_SYMREF,\n -+\t\t\t\t \"point to invalid object\");\n ++\t\tret = fsck_refs_report(o, refname,\n ++\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n ++\t\t\t\t \"point to invalid target\");\n +\t\tgoto out;\n +\t}\n +out:\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_refs_options *\n +\t\t\t\t const char *refs_check_dir,\n +\t\t\t\t struct dir_iterator *iter)\n +{\n -+\tstruct strbuf path = STRBUF_INIT,\n -+\t\t refname = STRBUF_INIT,\n ++\tstruct strbuf pointee_path = STRBUF_INIT,\n +\t\t ref_content = STRBUF_INIT,\n -+\t\t referent = STRBUF_INIT;\n -+\tunsigned int trailing = 0;\n ++\t\t abs_gitdir = STRBUF_INIT,\n ++\t\t referent = STRBUF_INIT,\n ++\t\t refname = STRBUF_INIT;\n ++\tconst char *trailing = NULL;\n +\tint failure_errno = 0;\n +\tunsigned int type = 0;\n +\tstruct object_id oid;\n +\tint ret = 0;\n +\n -+\tstrbuf_addbuf(&path, &iter->path);\n +\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n +\n +\t/*\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_refs_options *\n +\t * of the destination object.\n +\t */\n +\tif (S_ISLNK(iter->st.st_mode)) {\n -+\t\tstrbuf_strip_file_from_path(&path);\n -+\t\tret = strbuf_readlink(&ref_content,\n -+\t\t\t\t iter->path.buf, iter->st.st_size);\n -+\t\tif (ret < 0) {\n -+\t\t\tret = error_errno(_(\"could not read link '%s'\"),\n -+\t\t\t\t\t iter->path.buf);\n ++\t\tconst char *pointee_name = NULL;\n ++\n ++\t\tstrbuf_add_real_path(&pointee_path, iter->path.buf);\n ++\n ++\t\tstrbuf_add_absolute_path(&abs_gitdir, gitdir);\n ++\t\tstrbuf_normalize_path(&abs_gitdir);\n ++\t\tif (!is_dir_sep(abs_gitdir.buf[abs_gitdir.len - 1]))\n ++\t\t\tstrbuf_addch(&abs_gitdir, '/');\n ++\n ++\t\tif (!skip_prefix(pointee_path.buf,\n ++\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n ++\t\t\tret = fsck_refs_report(o, refname.buf,\n ++\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n ++\t\t\t\t\t \"point to target outside gitdir\");\n +\t\t\tgoto clean;\n +\t\t}\n -+\t\tstrbuf_addbuf(&path, &ref_content);\n -+\t\tstrbuf_reset(&ref_content);\n +\n -+\t\tret = files_fsck_symref(o, &refname, &path);\n ++\t\tret = files_fsck_symref_target(o, refname.buf, pointee_name,\n ++\t\t\t\t\t pointee_path.buf);\n +\t\tgoto clean;\n +\t}\n +\n -+\tif (strbuf_read_file(&ref_content, path.buf, 0) < 0) {\n ++\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n +\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n +\t\t\t\t refs_check_dir, iter->relative_path);\n +\t\tgoto clean;\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_refs_options *\n +\t\t\t\t &referent, &type,\n +\t\t\t\t &failure_errno, &trailing)) {\n +\t\tret = fsck_refs_report(o, refname.buf,\n -+\t\t\t\t FSCK_MSG_BAD_REF_SHA,\n ++\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n +\t\t\t\t \"invalid ref content\");\n +\t\tgoto clean;\n +\t}\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_refs_options *\n +\t * connectivity.\n +\t */\n +\tif (referent.len && (type & REF_ISSYMREF)) {\n -+\t\tstrbuf_reset(&path);\n -+\t\tstrbuf_addf(&path, \"%s/%s\", gitdir, referent.buf);\n -+\n -+\t\tif (check_refname_format(referent.buf, 0)) {\n -+\t\t\tret = fsck_refs_report(o, refname.buf,\n -+\t\t\t\t\t FSCK_MSG_DANGLING_SYMREF,\n -+\t\t\t\t\t \"point to invalid refname\");\n -+\t\t\tgoto clean;\n -+\t\t}\n ++\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n ++\t\tstrbuf_rtrim(&referent);\n +\n -+\t\tret = files_fsck_symref(o, &refname, &path);\n ++\t\tret = files_fsck_symref_target(o, refname.buf, referent.buf,\n ++\t\t\t\t\t pointee_path.buf);\n +\t\tgoto clean;\n +\t} else {\n +\t\t/*\n +\t\t * Only regular refs could have a trailing garbage. Should\n +\t\t * be reported as a warning.\n +\t\t */\n -+\t\tif (trailing) {\n ++\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n +\t\t\tret = fsck_refs_report(o, refname.buf,\n +\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n +\t\t\t\t\t \"trailing garbage in ref\");\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_refs_options *\n +\t}\n +\n +clean:\n -+\tstrbuf_release(&path);\n ++\tstrbuf_release(&abs_gitdir);\n ++\tstrbuf_release(&pointee_path);\n +\tstrbuf_release(&refname);\n +\tstrbuf_release(&ref_content);\n +\tstrbuf_release(&referent);\n @@ refs/refs-internal.h: struct ref_store {\n /*\n * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for\n - * invalid contents.\n -+ * invalid contents. Also trailing is set to 1 when there is any bytes after the\n -+ * hex.\n ++ * invalid contents. Also *trailing is set to the first character after the\n ++ * refname or NULL if the referent is not empty.\n */\n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n -\t\t\t int *failure_errno);\n -+\t\t\t int *failure_errno, unsigned int *trailing);\n ++\t\t\t int *failure_errno, const char **trailing);\n \n /*\n * Fill in the generic part of refs and add it to our collection of\n @@ t/t0602-reffiles-fsck.sh: test_expect_success 'ref name check should be adapted\n +\t\tprintf \"xfsazqfxcadas\" > $tag_dir_prefix/tag-2-bad &&\n +\t\ttest_must_fail git refs verify 2>err &&\n +\t\tcat >expect <<-EOF &&\n -+\t\terror: refs/tags/tag-2-bad: badRefSha: invalid ref content\n ++\t\terror: refs/tags/tag-2-bad: badRefContent: invalid ref content\n +\t\tEOF\n +\t\trm $tag_dir_prefix/tag-2-bad &&\n +\t\ttest_cmp expect err\n @@ t/t0602-reffiles-fsck.sh: test_expect_success 'ref name check should be adapted\n +\t\tprintf \"xfsazqfxcadas\" > $branch_dir_prefix/a/b/branch-2-bad &&\n +\t\ttest_must_fail git refs verify 2>err &&\n +\t\tcat >expect <<-EOF &&\n -+\t\terror: refs/heads/a/b/branch-2-bad: badRefSha: invalid ref content\n ++\t\terror: refs/heads/a/b/branch-2-bad: badRefContent: invalid ref content\n +\t\tEOF\n +\t\trm $branch_dir_prefix/a/b/branch-2-bad &&\n +\t\ttest_cmp expect err\n @@ t/t0602-reffiles-fsck.sh: test_expect_success 'ref name check should be adapted\n +\t) &&\n +\t(\n +\t\tcd repo &&\n -+\t\tprintf \"ref: refs/heads/branch-3\" > $branch_dir_prefix/branch-2-bad &&\n -+\t\tgit refs verify 2>err &&\n ++\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n ++\t\ttest_must_fail git refs verify 2>err &&\n +\t\tcat >expect <<-EOF &&\n -+\t\twarning: refs/heads/branch-2-bad: danglingSymref: point to non-existent ref\n ++\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid refname\n +\t\tEOF\n +\t\trm $branch_dir_prefix/branch-2-bad &&\n +\t\ttest_cmp expect err\n +\t) &&\n +\t(\n +\t\tcd repo &&\n -+\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n -+\t\tgit refs verify 2>err &&\n ++\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n ++\t\ttest_must_fail git refs verify 2>err &&\n +\t\tcat >expect <<-EOF &&\n -+\t\twarning: refs/heads/branch-2-bad: danglingSymref: point to invalid refname\n ++\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid target\n +\t\tEOF\n +\t\trm $branch_dir_prefix/branch-2-bad &&\n +\t\ttest_cmp expect err\n +\t) &&\n +\t(\n +\t\tcd repo &&\n -+\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n -+\t\tgit refs verify 2>err &&\n ++\t\tprintf \"ref: logs/maint-v2.45\" > $branch_dir_prefix/branch-2-bad &&\n ++\t\ttest_must_fail git refs verify 2>err &&\n +\t\tcat >expect <<-EOF &&\n -+\t\twarning: refs/heads/branch-2-bad: danglingSymref: point to invalid object\n ++\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to target out of refs hierarchy\n +\t\tEOF\n +\t\trm $branch_dir_prefix/branch-2-bad &&\n +\t\ttest_cmp expect err\n-- \n2.45.2\n\n"},{"id":"497270","messageId":"ZnFCEYypdAyXMMlg@ArchLinux","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[GSoC][PATCH v3 1/7] fsck: add refs check interfaces to interface with fsck error levels","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-18T08:15:13Z","receivedAt":"2024-06-18T08:15:18Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) focuses on object database consistency check. It relies\non the \"fsck_options\" to interact with fsck error levels. However,\n\"fsck_options\" aims at checking the object database which makes it\nunsuitable to change the semantics of it. Instead, create\n\"fsck_refs_options\" structure to handle refs consistency check.\n\nThe \"git_fsck_config\" sets up the \"msg_type\" and \"skiplist\" member of\nthe \"fsck_options\". For refs, we just need the \"msg_type\". In order to\nallow setting up more refs-specific options easily later, add a separate\nfunction \"git_fsck_refs_config\" to initialize the refs-specific options.\n\nMove the \"msg_type\" and \"strict\" member to the top of the \"fsck_options\"\nwhich allows us to convert \"fsck_refs_options *\" to \"fsck_options *\" to\nreuse the interfaces provided by \"fsck.h\" without changing the original\ncode.\n\nThe static function \"report\" provided by \"fsck.c\" aims at reporting the\nproblems related to object database which cannot be reused for refs.\nProvide \"fsck_refs_report\" function to integrate the fsck error levels\ninto reference consistency check.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n fsck.h | 45 ++++++++++++++++++++++++++++++++++++--\n 2 files changed, 112 insertions(+), 2 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex 8ef962199f..13528c646e 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1249,6 +1249,20 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n+int fsck_refs_error_function(struct fsck_refs_options *o UNUSED,\n+\t\t\t const char *name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tif (msg_type == FSCK_WARN) {\n+\t\twarning(\"%s: %s\", name, message);\n+\t\treturn 0;\n+\t}\n+\terror(\"%s: %s\", name, message);\n+\treturn 1;\n+}\n+\n int fsck_error_function(struct fsck_options *o,\n \t\t\tconst struct object_id *oid,\n \t\t\tenum object_type object_type UNUSED,\n@@ -1323,6 +1337,61 @@ int fsck_finish(struct fsck_options *options)\n \treturn ret;\n }\n \n+int fsck_refs_report(struct fsck_refs_options *o,\n+\t\t const char *name,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tenum fsck_msg_type msg_type =\n+\t\tfsck_msg_type(msg_id, (struct fsck_options*)o);\n+\tint ret = 0;\n+\n+\tif (msg_type == FSCK_IGNORE)\n+\t\treturn 0;\n+\n+\tif (msg_type == FSCK_FATAL)\n+\t\tmsg_type = FSCK_ERROR;\n+\telse if (msg_type == FSCK_INFO)\n+\t\tmsg_type = FSCK_WARN;\n+\n+\tprepare_msg_ids();\n+\tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n+\n+\tva_start(ap, fmt);\n+\tstrbuf_vaddf(&sb, fmt, ap);\n+\tret = o->error_func(o, name, msg_type, msg_id, sb.buf);\n+\tstrbuf_release(&sb);\n+\tva_end(ap);\n+\n+\treturn ret;\n+}\n+\n+int git_fsck_refs_config(const char *var, const char *value,\n+\t\t\t const struct config_context *ctx, void *cb)\n+{\n+\tstruct fsck_refs_options *options = cb;\n+\tconst char *msg_id;\n+\n+\t/*\n+\t * We don't check the value of fsck.skiplist here, because it\n+\t * is specific to object database, not reference database.\n+\t */\n+\tif (strcmp(var, \"fsck.skiplist\") == 0) {\n+\t\treturn 0;\n+\t}\n+\n+\tif (skip_prefix(var, \"fsck.\", &msg_id)) {\n+\t\tif (!value)\n+\t\t\treturn config_error_nonbool(var);\n+\t\tfsck_set_msg_type((struct fsck_options*)options, msg_id, value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, ctx, cb);\n+}\n+\n int git_fsck_config(const char *var, const char *value,\n \t\t const struct config_context *ctx, void *cb)\n {\ndiff --git a/fsck.h b/fsck.h\nindex 17fa2dda5d..6a38ac4a16 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -96,6 +96,7 @@ enum fsck_msg_id {\n };\n #undef MSG_ID\n \n+struct fsck_refs_options;\n struct fsck_options;\n struct object;\n \n@@ -107,6 +108,21 @@ void fsck_set_msg_type(struct fsck_options *options,\n void fsck_set_msg_types(struct fsck_options *options, const char *values);\n int is_valid_msg_type(const char *msg_id, const char *msg_type);\n \n+/*\n+ * callback function for fsck refs and reflogs.\n+ */\n+typedef int (*fsck_refs_error)(struct fsck_refs_options *o,\n+\t\t\t const char *name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n+\n+int fsck_refs_error_function(struct fsck_refs_options *o,\n+\t\t\t const char *name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n+\n /*\n * callback function for fsck_walk\n * type is the expected type of the object or OBJ_ANY\n@@ -135,11 +151,28 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message);\n \n+struct fsck_refs_options {\n+\tenum fsck_msg_type *msg_type;\n+\tunsigned strict:1;\n+\n+\tfsck_refs_error error_func;\n+\tunsigned verbose:1;\n+};\n+\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n+\n struct fsck_options {\n+\t/*\n+\t * Reorder the fields to allow `fsck_ref_options` to use\n+\t * the interfaces using `struct fsck_options`.\n+\t */\n+\tenum fsck_msg_type *msg_type;\n+\tunsigned strict:1;\n+\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n-\tunsigned strict:1;\n-\tenum fsck_msg_type *msg_type;\n \tstruct oidset skiplist;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n@@ -221,6 +254,12 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+__attribute__((format (printf, 4, 5)))\n+int fsck_refs_report(struct fsck_refs_options *o,\n+\t\t const char *name,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\n@@ -247,6 +286,8 @@ const char *fsck_describe_object(struct fsck_options *options,\n \t\t\t\t const struct object_id *oid);\n \n struct key_value_info;\n+int git_fsck_refs_config(const char *var, const char *value,\n+\t\t\t const struct config_context *ctx, void *cb);\n /*\n * git_config() callback for use by fsck-y tools that want to support\n * fsck. fsck.skipList etc.\n-- \n2.45.2\n\n"},{"id":"497271","messageId":"ZnFCiYrvAypGLtcC@ArchLinux","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[PATCH v3 2/7] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-18T08:17:13Z","receivedAt":"2024-06-18T08:17:17Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It's organized as the five parts:\n\n1. The name and the initialization interfaces.\n2. The ref transaction interfaces.\n3. The ref internal interfaces (pack, rename and copy).\n4. The ref filesystem interfaces.\n5. The reflog related interfaces.\n\nTo keep consistent with the git-fsck(1), add a new interface named\n\"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\ngrouped into any above five categories. Explicitly add blank line to\nmake it different from others.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs.c | 5 +++++\n refs.h | 8 ++++++++\n refs/debug.c | 11 +++++++++++\n refs/files-backend.c | 15 ++++++++++++++-\n refs/packed-backend.c | 8 ++++++++\n refs/refs-internal.h | 6 ++++++\n refs/reftable-backend.c | 8 ++++++++\n 7 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex f7c7765d23..0922439275 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -316,6 +316,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs, struct fsck_refs_options *o)\n+{\n+\treturn refs->be->fsck(refs, o);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex 76d25df4de..5a042695f5 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -3,6 +3,7 @@\n \n #include \"commit.h\"\n \n+struct fsck_refs_options;\n struct object_id;\n struct ref_store;\n struct repository;\n@@ -547,6 +548,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_refs_options *o);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..6306c25fd3 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tint res = drefs->refs->be->fsck(drefs->refs, o);\n+\ttrace_printf_key(&trace_refs, \"fsck: %d\\n\", res);\n+\treturn res;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex cb752d32b6..e965345ad8 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3402,6 +3402,17 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o)\n+{\n+\tint ret;\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\treturn ret;\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_init,\n@@ -3428,5 +3439,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex c4c1e36aa2..db152053f8 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1733,6 +1733,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_init,\n@@ -1760,4 +1766,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex cbcb6f9c36..8f42f21e77 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_refs_options;\n struct ref_transaction;\n \n /*\n@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -677,6 +681,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex e555be4671..7f606faa9e 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2242,6 +2242,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_refs_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2269,4 +2275,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.45.2\n\n"},{"id":"497272","messageId":"ZnFCu7kDjjoNr9dr@ArchLinux","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[PATCH v3 3/7] builtin/refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-18T08:18:03Z","receivedAt":"2024-06-18T08:18:07Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\ncheck the reference database consistency.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/git-refs.txt | 13 +++++++++++\n builtin/refs.c | 45 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 58 insertions(+)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..16b518ecce 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+The following options are specific to 'git refs verify':\n+\n+--strict::\n+\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n+\tbe seen as ERROR.\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..82ed0d57c1 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,55 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tint ret = 0;\n+\tunsigned int verbose = 0, strict = 0;\n+\tstruct fsck_refs_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n+\tstruct option options[] = {\n+\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"too many arguments\"));\n+\n+\tif (verbose)\n+\t\tfsck_refs_options.verbose = 1;\n+\tif (strict)\n+\t\tfsck_refs_options.strict = 1;\n+\n+\tgit_config(git_fsck_refs_config, &fsck_refs_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n+\n+\t/*\n+\t * Explicitly free the allocated array. This is necessary because\n+\t * this program is executed as child process of git-fsck(1) and the\n+\t * allocated array may not freed when git-fsck(1) aborts somewhere.\n+\t */\n+\tfree(fsck_refs_options.msg_type);\n+\treturn ret;\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \n-- \n2.45.2\n\n"},{"id":"497273","messageId":"ZnFC1rne3SBemacT@ArchLinux","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[PATCH v3 4/7] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-18T08:18:30Z","receivedAt":"2024-06-18T08:18:34Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new function \"fsck_refs\" that initializes and runs a child\nprocess to execute the \"git-refs verify\" command.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..10d73f534f 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -896,6 +896,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \treturn res;\n }\n \n+static void fsck_refs(void)\n+{\n+\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n+\tchild_process_init(&refs_verify);\n+\trefs_verify.git_cmd = 1;\n+\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n+\tif (verbose)\n+\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n+\tif (check_strict)\n+\t\tstrvec_push(&refs_verify.args, \"--strict\");\n+\n+\tif (run_command(&refs_verify))\n+\t\terrors_found |= ERROR_REFS;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n@@ -1065,6 +1080,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n+\tfsck_refs();\n+\n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \n-- \n2.45.2\n\n"},{"id":"497274","messageId":"ZnFC8YgcfYhhYb2V@ArchLinux","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[PATCH v3 5/7] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-18T08:18:57Z","receivedAt":"2024-06-18T08:19:01Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"For refs and reflogs, we need to scan its corresponding directories to\ncheck every regular file or symbolic link which shares the same pattern.\nIntroduce a unified interface for scanning directories for\nfiles-backend.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs/files-backend.c | 77 +++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 76 insertions(+), 1 deletion(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex e965345ad8..74242dbba4 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -4,6 +4,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -3402,6 +3403,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct fsck_refs_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_refs_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n+{\n+\tconst char *gitdir = ref_store->gitdir;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\n+\tif (!iter) {\n+\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n+\t\tgoto out;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->verbose)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n+\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n+\t\t\t\t iter->basename);\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n+\n+out:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_refs_options *o)\n+{\n+\tint ret;\n+\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tNULL\n+\t};\n+\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, \"Checking references consistency\");\n+\n+\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n+\n+\treturn ret;\n+}\n+\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_refs_options *o)\n {\n@@ -3409,7 +3482,9 @@ static int files_fsck(struct ref_store *ref_store,\n \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n-\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o)\n+\t | files_fsck_refs(ref_store, o);\n+\n \treturn ret;\n }\n \n-- \n2.45.2\n\n"},{"id":"497275","messageId":"ZnFDCjgywD_Oatmz@ArchLinux","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[PATCH v3 6/7] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-18T08:19:22Z","receivedAt":"2024-06-18T08:19:26Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) only implicitly checks the reference, it does not fully\ncheck refs with bad format name such as standalone \"@\" and name ending\nwith \".lock\".\n\nIn order to provide such checks, add a new fsck message id \"badRefName\"\nwith default ERROR type. Use existing \"check_refname_format\" to explicit\ncheck the ref name. And add a new unit test to verify the functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 +\n fsck.h | 1 +\n refs/files-backend.c | 20 +++++++\n t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++\n 4 files changed, 125 insertions(+)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex 5edc06c658..cc85c897cc 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,9 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefName`::\n+\t(ERROR) A ref has a bad name.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex 6a38ac4a16..6b3e68b873 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 74242dbba4..3f0b565a13 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3413,6 +3413,25 @@ typedef int (*files_fsck_refs_fn)(struct fsck_refs_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n+static int files_fsck_refs_name(struct fsck_refs_options *o,\n+\t\t\t\tconst char *gitdir UNUSED,\n+\t\t\t\tconst char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\tret = fsck_refs_report(o, sb.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_refs_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3464,6 +3483,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n {\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tfiles_fsck_refs_name,\n \t\tNULL\n \t};\n \ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..b2db58d2c6\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit tag multi_hierarchy/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1.lock: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/@ &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\tgit -c fsck.badRefName=warn fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\tgit -c fsck.badRefName=ignore fsck 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+test_done\n-- \n2.45.2\n\n"},{"id":"497276","messageId":"ZnFDSohT0_374NGS@ArchLinux","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[PATCH v3 7/7] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-18T08:20:26Z","receivedAt":"2024-06-18T08:20:30Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Enhance the git-fsck(1) command by adding a check for reference content\nin the files backend. The new functionality ensures that symrefs, real\nsymbolic link and regular refs are validated correctly.\n\nIn order to check the trailing content of the regular refs, add a new\nparameter `trailing` to `parse_loose_ref_contents`.\n\nFor symrefs, `parse_loose_ref_contents` will set the \"referent\".\nHowever, symbolic link could be either absolute or relative. Use\n\"strbuf_add_real_path\" to read the symbolic link and convert the\nrelative path to absolute path. Then use \"skip_prefix\" to make it align\nwith symref \"referent\".\n\nThus, the symrefs and symbolic links could share the same interface. Add\na new function \"files_fsck_symref_target\" which aims at checking the\nfollowing things:\n\n1. whether the pointee is under the `refs/` directory.\n2. whether the pointee name is correct.\n3. whether the pointee path is a wrong type in filesystem.\n\nLast, add the following FSCK MESSAGEs:\n\n1. \"badRefContent(ERROR)\": A ref has a bad content\n2. \"badSymrefPointee(ERROR)\": The pointee of a symref is bad.\n3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 9 +++\n fsck.h | 3 +\n refs.c | 2 +-\n refs/files-backend.c | 145 +++++++++++++++++++++++++++++++++-\n refs/refs-internal.h | 5 +-\n t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++\n 6 files changed, 269 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex cc85c897cc..ebe2de2bdc 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,9 +19,15 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefContent`::\n+\t(ERROR) A ref has a bad content.\n+\n `badRefName`::\n \t(ERROR) A ref has a bad name.\n \n+`badSymrefPointee`::\n+\t(ERROR) The pointee of a symref is bad.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \n@@ -179,6 +185,9 @@\n `symlinkTargetMissing`::\n \t(ERROR) Unable to read symbolic link target's blob.\n \n+`trailingRefContent`::\n+\t(WARN) A ref content has trailing contents.\n+\n `treeNotSorted`::\n \t(ERROR) A tree is not properly sorted.\n \ndiff --git a/fsck.h b/fsck.h\nindex 6b3e68b873..215b1466d2 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -32,6 +32,8 @@ enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n+\tFUNC(BAD_REF_CONTENT, ERROR) \\\n+\tFUNC(BAD_SYMREF_POINTEE, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n@@ -74,6 +76,7 @@ enum fsck_msg_type {\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n+\tFUNC(TRAILING_REF_CONTENT, WARN) \\\n \tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n \tFUNC(LARGE_PATHNAME, WARN) \\\ndiff --git a/refs.c b/refs.c\nindex 0922439275..1325f83269 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1744,7 +1744,7 @@ static int refs_read_special_head(struct ref_store *ref_store,\n \t}\n \n \tresult = parse_loose_ref_contents(content.buf, oid, referent, type,\n-\t\t\t\t\t failure_errno);\n+\t\t\t\t\t failure_errno, NULL);\n \n done:\n \tstrbuf_release(&full_path);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 3f0b565a13..56a64cb97c 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1,4 +1,5 @@\n #include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n #include \"../copy.h\"\n #include \"../environment.h\"\n #include \"../gettext.h\"\n@@ -549,7 +550,7 @@ static int read_ref_internal(struct ref_store *ref_store, const char *refname,\n \tstrbuf_rtrim(&sb_contents);\n \tbuf = sb_contents.buf;\n \n-\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr);\n+\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr, NULL);\n \n out:\n \tif (ret && !myerr)\n@@ -585,7 +586,7 @@ static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refn\n \n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno)\n+\t\t\t int *failure_errno, const char **trailing)\n {\n \tconst char *p;\n \tif (skip_prefix(buf, \"ref:\", &buf)) {\n@@ -607,6 +608,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t*failure_errno = EINVAL;\n \t\treturn -1;\n \t}\n+\n+\tif (trailing)\n+\t\t*trailing = p;\n+\n \treturn 0;\n }\n \n@@ -3432,6 +3437,141 @@ static int files_fsck_refs_name(struct fsck_refs_options *o,\n \treturn ret;\n }\n \n+/*\n+ * Check the symref \"pointee_name\" and \"pointee_path\". The caller should\n+ * make sure that \"pointee_path\" is absolute. For symbolic ref, \"pointee_name\"\n+ * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n+ * be the relative path agaignst \"gitdir\".\n+ */\n+static int files_fsck_symref_target(struct fsck_refs_options *o,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const char *pointee_name,\n+\t\t\t\t const char *pointee_path)\n+{\n+\tconst char *p = NULL;\n+\tstruct stat st;\n+\tint ret = 0;\n+\n+\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n+\n+\t\tret = fsck_refs_report(o, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to target out of refs hierarchy\");\n+\t\tgoto out;\n+\t}\n+\n+\tif (check_refname_format(pointee_name, 0)) {\n+\t\tret = fsck_refs_report(o, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid refname\");\n+\t}\n+\n+\tif (lstat(pointee_path, &st) < 0)\n+\t\tgoto out;\n+\n+\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n+\t\tret = fsck_refs_report(o, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid target\");\n+\t\tgoto out;\n+\t}\n+out:\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs_content(struct fsck_refs_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter)\n+{\n+\tstruct strbuf pointee_path = STRBUF_INIT,\n+\t\t ref_content = STRBUF_INIT,\n+\t\t abs_gitdir = STRBUF_INIT,\n+\t\t referent = STRBUF_INIT,\n+\t\t refname = STRBUF_INIT;\n+\tconst char *trailing = NULL;\n+\tint failure_errno = 0;\n+\tunsigned int type = 0;\n+\tstruct object_id oid;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\n+\t/*\n+\t * If the file is a symlink, we need to only check the connectivity\n+\t * of the destination object.\n+\t */\n+\tif (S_ISLNK(iter->st.st_mode)) {\n+\t\tconst char *pointee_name = NULL;\n+\n+\t\tstrbuf_add_real_path(&pointee_path, iter->path.buf);\n+\n+\t\tstrbuf_add_absolute_path(&abs_gitdir, gitdir);\n+\t\tstrbuf_normalize_path(&abs_gitdir);\n+\t\tif (!is_dir_sep(abs_gitdir.buf[abs_gitdir.len - 1]))\n+\t\t\tstrbuf_addch(&abs_gitdir, '/');\n+\n+\t\tif (!skip_prefix(pointee_path.buf,\n+\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n+\t\t\tret = fsck_refs_report(o, refname.buf,\n+\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t\t \"point to target outside gitdir\");\n+\t\t\tgoto clean;\n+\t\t}\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, pointee_name,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n+\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n+\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n+\t\t\t\t &referent, &type,\n+\t\t\t\t &failure_errno, &trailing)) {\n+\t\tret = fsck_refs_report(o, refname.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n+\t\t\t\t \"invalid ref content\");\n+\t\tgoto clean;\n+\t}\n+\n+\t/*\n+\t * If the ref is a symref, we need to check the destination name and\n+\t * connectivity.\n+\t */\n+\tif (referent.len && (type & REF_ISSYMREF)) {\n+\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n+\t\tstrbuf_rtrim(&referent);\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, referent.buf,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t} else {\n+\t\t/*\n+\t\t * Only regular refs could have a trailing garbage. Should\n+\t\t * be reported as a warning.\n+\t\t */\n+\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n+\t\t\tret = fsck_refs_report(o, refname.buf,\n+\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n+\t\t\t\t\t \"trailing garbage in ref\");\n+\t\t\tgoto clean;\n+\t\t}\n+\t}\n+\n+clean:\n+\tstrbuf_release(&abs_gitdir);\n+\tstrbuf_release(&pointee_path);\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&ref_content);\n+\tstrbuf_release(&referent);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_refs_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3484,6 +3624,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n \t\tfiles_fsck_refs_name,\n+\t\tfiles_fsck_refs_content,\n \t\tNULL\n \t};\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 8f42f21e77..72f0945ada 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -709,11 +709,12 @@ struct ref_store {\n \n /*\n * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for\n- * invalid contents.\n+ * invalid contents. Also *trailing is set to the first character after the\n+ * refname or NULL if the referent is not empty.\n */\n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno);\n+\t\t\t int *failure_errno, const char **trailing);\n \n /*\n * Fill in the generic part of refs and add it to our collection of\ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nindex b2db58d2c6..35bf40ee64 100755\n--- a/t/t0602-reffiles-fsck.sh\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -98,4 +98,114 @@ test_expect_success 'ref name check should be adapted into fsck messages' '\n \t)\n '\n \n+test_expect_success 'regular ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit checkout -b a/b/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse branch-1)\" > $branch_dir_prefix/branch-1-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse tag-1)\" > $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_must_fail git -c fsck.trailingRefContent=error fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s \" \"$(git rev-parse tag-2)\" > $tag_dir_prefix/tag-2-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/tags/tag-2-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/a/b/branch-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'symbolic ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid refname\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid target\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: logs/maint-v2.45\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to target out of refs hierarchy\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.2\n\n"},{"id":"497277","messageId":"CAOLa=ZS57EsZk+7xOW9imbA4oWANKf0b+HxBZ8hGFaQQkCsvcQ@mail.gmail.com","threadId":"61565","inReplyTo":"ZnFCEYypdAyXMMlg@ArchLinux","subject":"Re: [GSoC][PATCH v3 1/7] fsck: add refs check interfaces to interface with fsck error levels","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-06-18T08:38:07Z","receivedAt":"2024-06-18T08:38:09Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n[snip]\n\n> struct fsck_options {\n> +\t/*\n> +\t * Reorder the fields to allow `fsck_ref_options` to use\n> +\t * the interfaces using `struct fsck_options`.\n> +\t */\n\nWhy is this added? It makes sense to have it in the commit message\nbecause it talks about the change, but why make it persistent in the\ncode?\n\n[snip]\n"},{"id":"497278","messageId":"CAOLa=ZT=B=7Ec7Ui6JmOCFC7gh+G0D=GGGO0MzL_HBcGW9pbOQ@mail.gmail.com","threadId":"61565","inReplyTo":"ZnFCu7kDjjoNr9dr@ArchLinux","subject":"Re: [PATCH v3 3/7] builtin/refs: add verify subcommand","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-06-18T08:44:46Z","receivedAt":"2024-06-18T08:44:48Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\n> check the reference database consistency.\n>\n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n> Documentation/git-refs.txt | 13 +++++++++++\n> builtin/refs.c | 45 ++++++++++++++++++++++++++++++++++++++\n> 2 files changed, 58 insertions(+)\n>\n> diff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\n> index 5b99e04385..16b518ecce 100644\n> --- a/Documentation/git-refs.txt\n> +++ b/Documentation/git-refs.txt\n> @@ -10,6 +10,7 @@ SYNOPSIS\n> --------\n> [verse]\n> 'git refs migrate' --ref-format= [--dry-run]\n> +'git refs verify' [--strict] [--verbose]\n>\n> DESCRIPTION\n> -----------\n> @@ -22,6 +23,9 @@ COMMANDS\n> migrate::\n> \tMigrate ref store between different formats.\n>\n> +verify::\n> +\tVerify reference database consistency.\n> +\n> OPTIONS\n> -------\n>\n> @@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n> \tcan be used to double check that the migration works as expected before\n> \tperforming the actual migration.\n>\n> +The following options are specific to 'git refs verify':\n> +\n> +--strict::\n> +\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n> +\tbe seen as ERROR.\n> +\n\nNit: It would be nice to redirect users to the fsck documentation to\nunderstand severity levels.\n\n[snip]\n"},{"id":"497279","messageId":"ZnFJTdHQcHVagflt@ArchLinux","threadId":"61565","inReplyTo":"CAOLa=ZS57EsZk+7xOW9imbA4oWANKf0b+HxBZ8hGFaQQkCsvcQ@mail.gmail.com","subject":"Re: [GSoC][PATCH v3 1/7] fsck: add refs check interfaces to interface with fsck error levels","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-18T08:46:05Z","receivedAt":"2024-06-18T08:46:08Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jun 18, 2024 at 04:38:07AM -0400, Karthik Nayak wrote:\n> shejialuo writes:\n> \n> [snip]\n> \n> > struct fsck_options {\n> > +\t/*\n> > +\t * Reorder the fields to allow `fsck_ref_options` to use\n> > +\t * the interfaces using `struct fsck_options`.\n> > +\t */\n> \n> Why is this added? It makes sense to have it in the commit message\n> because it talks about the change, but why make it persistent in the\n> code?\n> \n\nI explicitly add this comments due to the following reasons:\n\n1. If someone needs to change the `fsck_options`, without this comment,\nhe might be just add some new fields at top. Although the change will\nfail the tests here, I think we should mention this in code.\n2. My later intention is that we should extract these common fields out\nof the `fsck_options` and `fsck_ref_options`.\n\n> [snip]\n\n\n"},{"id":"497280","messageId":"ZnFJzFVNjisQn6dU@ArchLinux","threadId":"61565","inReplyTo":"CAOLa=ZT=B=7Ec7Ui6JmOCFC7gh+G0D=GGGO0MzL_HBcGW9pbOQ@mail.gmail.com","subject":"Re: [PATCH v3 3/7] builtin/refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-18T08:48:12Z","receivedAt":"2024-06-18T08:48:16Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jun 18, 2024 at 04:44:46AM -0400, Karthik Nayak wrote:\n> shejialuo writes:\n> \n> > Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\n> > check the reference database consistency.\n> >\n> > Mentored-by: Patrick Steinhardt \n> > Mentored-by: Karthik Nayak \n> > Signed-off-by: shejialuo \n> > ---\n> > Documentation/git-refs.txt | 13 +++++++++++\n> > builtin/refs.c | 45 ++++++++++++++++++++++++++++++++++++++\n> > 2 files changed, 58 insertions(+)\n> >\n> > diff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\n> > index 5b99e04385..16b518ecce 100644\n> > --- a/Documentation/git-refs.txt\n> > +++ b/Documentation/git-refs.txt\n> > @@ -10,6 +10,7 @@ SYNOPSIS\n> > --------\n> > [verse]\n> > 'git refs migrate' --ref-format= [--dry-run]\n> > +'git refs verify' [--strict] [--verbose]\n> >\n> > DESCRIPTION\n> > -----------\n> > @@ -22,6 +23,9 @@ COMMANDS\n> > migrate::\n> > \tMigrate ref store between different formats.\n> >\n> > +verify::\n> > +\tVerify reference database consistency.\n> > +\n> > OPTIONS\n> > -------\n> >\n> > @@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n> > \tcan be used to double check that the migration works as expected before\n> > \tperforming the actual migration.\n> >\n> > +The following options are specific to 'git refs verify':\n> > +\n> > +--strict::\n> > +\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n> > +\tbe seen as ERROR.\n> > +\n> \n> Nit: It would be nice to redirect users to the fsck documentation to\n> understand severity levels.\n> \n\nI agree. Will improve this in the next version.\n\n> [snip]\n\n\n"},{"id":"497281","messageId":"ZnFOvagjx6fRxJbw@ArchLinux","threadId":"61565","inReplyTo":"CAOLa=ZQ5msKTHwDou2RmCrUCdfDuYdFwJQx3Q9gK6aAJvGw-pg@mail.gmail.com","subject":"Re: [GSoC][PATCH v2 5/7] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-18T09:09:17Z","receivedAt":"2024-06-18T09:09:21Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Sat, Jun 15, 2024 at 04:51:14PM -0400, Karthik Nayak wrote:\n> shejialuo writes:\n> \n> > For refs and reflogs, we need to scan its corresponding directories to\n> > check every regular file or symbolic link which shares the same pattern.\n> > Introduce a unified interface for scanning directories for\n> > files-backend.\n> >\n> \n> Here we talk about reflogs, but we only add a iterator for\n> \"$GIT_DIR/refs\". Should we add something for reflogs too?\n> \n\nHere, we simply set up the basic structure. For both refs and reflogs,\nwe will traverse the file system. What we concern about is the regular\nfile and symlink file. The \"refs_check_dir\" parameter is used to\nindicate what we will check. Actually, we do not add any actual check\noperations for refs in this commit.\n\n> > Mentored-by: Patrick Steinhardt \n> > Mentored-by: Karthik Nayak \n> > Signed-off-by: shejialuo \n> > ---\n> > refs/files-backend.c | 75 ++++++++++++++++++++++++++++++++++++++++++++\n> > 1 file changed, 75 insertions(+)\n> >\n> > diff --git a/refs/files-backend.c b/refs/files-backend.c\n> > index e965345ad8..b26cfb8ba6 100644\n> > --- a/refs/files-backend.c\n> > +++ b/refs/files-backend.c\n> > @@ -4,6 +4,7 @@\n> > #include \"../gettext.h\"\n> > #include \"../hash.h\"\n> > #include \"../hex.h\"\n> > +#include \"../fsck.h\"\n> > #include \"../refs.h\"\n> > #include \"refs-internal.h\"\n> > #include \"ref-cache.h\"\n> > @@ -3402,6 +3403,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n> > \treturn ret;\n> > }\n> >\n> > +/*\n> > + * For refs and reflogs, they share a unified interface when scanning\n> > + * the whole directory. This function is used as the callback for each\n> > + * regular file or symlink in the directory.\n> > + */\n> > +typedef int (*files_fsck_refs_fn)(struct fsck_refs_options *o,\n> > +\t\t\t\t const char *gitdir,\n> > +\t\t\t\t const char *refs_check_dir,\n> > +\t\t\t\t struct dir_iterator *iter);\n> > +\n> > +static int files_fsck_refs_dir(struct ref_store *ref_store,\n> > +\t\t\t struct fsck_refs_options *o,\n> > +\t\t\t const char *refs_check_dir,\n> > +\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n> > +{\n> > +\tconst char *gitdir = ref_store->gitdir;\n> > +\tstruct strbuf sb = STRBUF_INIT;\n> > +\tstruct dir_iterator *iter;\n> > +\tint iter_status;\n> > +\tint ret = 0;\n> > +\n> > +\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n> > +\n> > +\titer = dir_iterator_begin(sb.buf, 0);\n> > +\n> > +\tif (!iter) {\n> > +\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n> > +\t\tgoto out;\n> > +\t}\n> > +\n> > +\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n> > +\t\tif (S_ISDIR(iter->st.st_mode)) {\n> > +\t\t\tcontinue;\n> > +\t\t} else if (S_ISREG(iter->st.st_mode) ||\n> > +\t\t\t S_ISLNK(iter->st.st_mode)) {\n> > +\t\t\tif (o->verbose)\n> > +\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n> > +\t\t\t\t\t refs_check_dir, iter->relative_path);\n> > +\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n> > +\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n> > +\t\t\t\t\tret = -1;\n> > +\t\t\t}\n> > +\t\t} else {\n> > +\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n> > +\t\t\t\t iter->basename);\n> > +\t\t}\n> > +\t}\n> > +\n> > +\tif (iter_status != ITER_DONE)\n> > +\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n> > +\n> > +out:\n> > +\tstrbuf_release(&sb);\n> > +\treturn ret;\n> > +}\n> > +\n> > +static int files_fsck_refs(struct ref_store *ref_store,\n> > +\t\t\t struct fsck_refs_options *o)\n> > +{\n> > +\tint ret;\n> > +\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n> > +\t\tNULL\n> > +\t};\n> > +\n> > +\tif (o->verbose)\n> > +\t\tfprintf_ln(stderr, \"Checking references consistency\");\n> > +\n> > +\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n> > +\n> > +\treturn ret;\n> > +}\n> \n> I'm not fully sure why this needs to be separate from `files_fsck`. Is\n> that because we plan to also introduce `files_fsck_reflogs`?\n> \n\nYes, exactly. Later we need to introduce \"files_fsck_reflogs\" which\nshould define the different `files_fsck_refs_fn` here. So just introduce\na function here.\n\n> > +\n> > static int files_fsck(struct ref_store *ref_store,\n> > \t\t struct fsck_refs_options *o)\n> > {\n> > @@ -3410,6 +3483,8 @@ static int files_fsck(struct ref_store *ref_store,\n> > \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n> >\n> > \tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n> > +\tret = files_fsck_refs(ref_store, o);\n> > +\n> > \treturn ret;\n> > }\n> >\n> > --\n> > 2.45.2\n\n\n"},{"id":"497287","messageId":"xmqqfrta2ryp.fsf@gitster.g","threadId":"61565","inReplyTo":"ZnFJTdHQcHVagflt@ArchLinux","subject":"Re: [GSoC][PATCH v3 1/7] fsck: add refs check interfaces to interface with fsck error levels","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-06-18T15:25:34Z","receivedAt":"2024-06-18T15:25:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"shejialuo writes:\n\n> On Tue, Jun 18, 2024 at 04:38:07AM -0400, Karthik Nayak wrote:\n>> shejialuo writes:\n>> \n>> [snip]\n>> \n>> > struct fsck_options {\n>> > +\t/*\n>> > +\t * Reorder the fields to allow `fsck_ref_options` to use\n>> > +\t * the interfaces using `struct fsck_options`.\n>> > +\t */\n>> \n>> Why is this added? It makes sense to have it in the commit message\n>> because it talks about the change, but why make it persistent in the\n>> code?\n>> \n>\n> I explicitly add this comments due to the following reasons:\n>\n> 1. If someone needs to change the `fsck_options`, without this comment,\n> he might be just add some new fields at top. Although the change will\n> fail the tests here, I think we should mention this in code.\n\nDo you mean you plan to take advantage of the fact that early\nmembers of two structures are the same? IOW, if there is a function\nthat takes a pointer to smaller fsck_refs_options, you plan to pass\na pointer to fsck_options from some callers, e.g.\n\n extern void func(struct fsck_refs_options *);\n void a_caller(struct fsck_options *o)\n {\n\tfunc((struct fsck_options *)o);\n\t...\n\nIf that is the case, then ...\n\nDo not do that.\n\nYour data structure design is broken. Instead you would do this:\n\n\tstruct fsck_options {\n\t\tstruct fsck_refs_options refs;\n\t\t... other members ...\n\t};\n\tvoid a_caller(struct fsck_options *o)\n\t{\n\t\tfunc(&o->refs);\n\t\t...\n\n"},{"id":"497289","messageId":"xmqq8qz22r6r.fsf@gitster.g","threadId":"61565","inReplyTo":"ZnFByMI1lWUDSxLd@ArchLinux","subject":"Re: [GSoC][PATCH v3 0/7] ref consistency check infra setup","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-06-18T15:42:20Z","receivedAt":"2024-06-18T15:42:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"shejialuo writes:\n\n> This is the third version. Mainly focuses on the following problems:\n>\n> 1. Add \"escape\" check for symrefs and symbolic links.\n> 2. Resolve the reviews.\n> 3. Fix some errors in version 2.\n>\n> shejialuo (7):\n> fsck: add refs check interfaces to interface with fsck error levels\n> refs: set up ref consistency check infrastructure\n> builtin/refs: add verify subcommand\n> builtin/fsck: add `git-refs verify` child process\n> files-backend: add unified interface for refs scanning\n> fsck: add ref name check for files backend\n> fsck: add ref content check for files backend\n\nWhat commit is this series based on? It does not seem to apply\ncleanly on top of the same base as I queued an earlier iteration of\nthis series (f931bfb2 (Merge branch 'ps/ref-storage-migration' into\nsj/ref-fsck, 2024-06-13)), or the current tip of 'master' that\nalready merges that prerequisite topic. The early 6 applies on top\nof 'master' fine, but the last one. I wiggled the last one in (the\npatch to the fsck-msgids.txt documentation expects stale entries\nthat were already removed in 2.45.2), but if you are basing your\npatches on an unusual commit, please describe it clearly here in the\ncover letter so that others can locate (or recreate as needed) it.\n"},{"id":"497290","messageId":"CAH-kW5cxQQFpr30FV9uB3HcyuSgoCYRs8UxiStd9v5P1cXjK7w@mail.gmail.com","threadId":"61565","inReplyTo":"xmqqfrta2ryp.fsf@gitster.g","subject":"Re: [GSoC][PATCH v3 1/7] fsck: add refs check interfaces to interface with fsck error levels","fromName":"jialuo she","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-18T15:47:55Z","receivedAt":"2024-06-18T15:48:09Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Junio C Hamano writes:\n>\n> shejialuo writes:\n>\n> > On Tue, Jun 18, 2024 at 04:38:07AM -0400, Karthik Nayak wrote:\n> >> shejialuo writes:\n> >>\n> >> [snip]\n> >>\n> >> > struct fsck_options {\n> >> > + /*\n> >> > + * Reorder the fields to allow `fsck_ref_options` to use\n> >> > + * the interfaces using `struct fsck_options`.\n> >> > + */\n> >>\n> >> Why is this added? It makes sense to have it in the commit message\n> >> because it talks about the change, but why make it persistent in the\n> >> code?\n> >>\n> >\n> > I explicitly add this comments due to the following reasons:\n> >\n> > 1. If someone needs to change the `fsck_options`, without this comment,\n> > he might be just add some new fields at top. Although the change will\n> > fail the tests here, I think we should mention this in code.\n>\n> Do you mean you plan to take advantage of the fact that early\n> members of two structures are the same? IOW, if there is a function\n> that takes a pointer to smaller fsck_refs_options, you plan to pass\n> a pointer to fsck_options from some callers, e.g.\n>\n> extern void func(struct fsck_refs_options *);\n> void a_caller(struct fsck_options *o)\n> {\n> func((struct fsck_options *)o);\n> ...\n>\n\nI do not want to convert \"struct fsck_options*\" to \"struct fsck_refs_options*\".\nInstead, I want to convert \"struct fsck_refs_options*\" to \"struct fsck_options*\"\nto reuse the functions which use the \"struct fsck_options*\" parameter.\n\nLike the commit message said:\n\n Move the \"msg_type\" and \"strict\" member to the top of the \"fsck_options\"\n which allows us to convert \"fsck_refs_options *\" to \"fsck_options *\" to\n reuse the interfaces provided by \"fsck.h\" without changing the original\n code.\n\nIt may seem we should add some fields into `fsck_options`, but I don't think\nit's a good idea. I will elaborate my design here:\n\nThe git-fsck(1) is highly relevant with the object database. I don't want to add\nsome new fields into \"fsck_options\" due to the following reason:\n\nThe fields in fsck_options are all related to objects except \"fsck_msg_type\"\nand \"strict\". Adding some ref-related fields into \"fsck_options\" will break the\nsemantics.\n\nActually, it may be perfect that I may abstract some interfaces here, however,\nit would be too complicated for this patch, many functions use \"fsck_options *\"\nas the parameter. I think we may do this later.\n"},{"id":"497291","messageId":"CAH-kW5e079dkg99C3rroj+vh6C2L6roqVFYWg_Ykno3WVpHOhg@mail.gmail.com","threadId":"61565","inReplyTo":"xmqq8qz22r6r.fsf@gitster.g","subject":"Re: [GSoC][PATCH v3 0/7] ref consistency check infra setup","fromName":"jialuo she","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-18T15:56:54Z","receivedAt":"2024-06-18T15:57:07Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Junio C Hamano writes:\n>\n> shejialuo writes:\n>\n> > This is the third version. Mainly focuses on the following problems:\n> >\n> > 1. Add \"escape\" check for symrefs and symbolic links.\n> > 2. Resolve the reviews.\n> > 3. Fix some errors in version 2.\n> >\n> > shejialuo (7):\n> > fsck: add refs check interfaces to interface with fsck error levels\n> > refs: set up ref consistency check infrastructure\n> > builtin/refs: add verify subcommand\n> > builtin/fsck: add `git-refs verify` child process\n> > files-backend: add unified interface for refs scanning\n> > fsck: add ref name check for files backend\n> > fsck: add ref content check for files backend\n>\n> What commit is this series based on? It does not seem to apply\n> cleanly on top of the same base as I queued an earlier iteration of\n> this series (f931bfb2 (Merge branch 'ps/ref-storage-migration' into\n> sj/ref-fsck, 2024-06-13)), or the current tip of 'master' that\n> already merges that prerequisite topic. The early 6 applies on top\n> of 'master' fine, but the last one. I wiggled the last one in (the\n> patch to the fsck-msgids.txt documentation expects stale entries\n> that were already removed in 2.45.2), but if you are basing your\n> patches on an unusual commit, please describe it clearly here in the\n> cover letter so that others can locate (or recreate as needed) it.\n\nWell, I simply does \"git checkout -b js-refs-fsck-v3\" for the current branch\n\"js-refs-fsck\". And I use \"git rebase\" to change the code in this branch.\nSo the base commit should be unchanged. However, maybe Patrick\nhas changed the code and I forgot to update upstream.\n\nAnd I have set up the base using the following commands:\n\n $ git remote add junio https://github.com/gister/git.git\n $ git fetch junio\n $ git branch js-refs-fsck junio/ps/ref-storage-migration\n\nI am sorry for the confusion. Next version, I will merge the master code\nand send the patch. I am sorry to make you so inconvenient.\n"},{"id":"497314","messageId":"ZnJE-QKgUfM3WSp5@ArchLinux","threadId":"61565","inReplyTo":"xmqqfrta2ryp.fsf@gitster.g","subject":"Re: [GSoC][PATCH v3 1/7] fsck: add refs check interfaces to interface with fsck error levels","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-19T02:39:53Z","receivedAt":"2024-06-19T02:39:56Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jun 18, 2024 at 08:25:34AM -0700, Junio C Hamano wrote:\n> shejialuo writes:\n> \n> > On Tue, Jun 18, 2024 at 04:38:07AM -0400, Karthik Nayak wrote:\n> >> shejialuo writes:\n> >> \n> >> [snip]\n> >> \n> >> > struct fsck_options {\n> >> > +\t/*\n> >> > +\t * Reorder the fields to allow `fsck_ref_options` to use\n> >> > +\t * the interfaces using `struct fsck_options`.\n> >> > +\t */\n> >> \n> >> Why is this added? It makes sense to have it in the commit message\n> >> because it talks about the change, but why make it persistent in the\n> >> code?\n> >> \n> >\n> > I explicitly add this comments due to the following reasons:\n> >\n> > 1. If someone needs to change the `fsck_options`, without this comment,\n> > he might be just add some new fields at top. Although the change will\n> > fail the tests here, I think we should mention this in code.\n> \n> Do you mean you plan to take advantage of the fact that early\n> members of two structures are the same? IOW, if there is a function\n> that takes a pointer to smaller fsck_refs_options, you plan to pass\n> a pointer to fsck_options from some callers, e.g.\n> \n> extern void func(struct fsck_refs_options *);\n> void a_caller(struct fsck_options *o)\n> {\n> \tfunc((struct fsck_options *)o);\n> \t...\n> \n> If that is the case, then ...\n> \n> Do not do that.\n> \n> Your data structure design is broken. Instead you would do this:\n> \n> \tstruct fsck_options {\n> \t\tstruct fsck_refs_options refs;\n> \t\t... other members ...\n> \t};\n> \tvoid a_caller(struct fsck_options *o)\n> \t{\n> \t\tfunc(&o->refs);\n> \t\t...\n\nWell, I totally agree with this. It's bad to convert the pointer type. I\nwill change the code in this patch to make it OK.\n\nThanks for your advice.\n\n\n"},{"id":"497333","messageId":"ZnKKy52QFO2UhqM6@ArchLinux","threadId":"61565","inReplyTo":"20240612085349.710785-1-shejialuo@gmail.com","subject":"[GSoC][PATCH v4 0/7] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-19T07:37:47Z","receivedAt":"2024-06-19T07:37:49Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Hi All:\n\nThis version mainly focuses on solving following reviews:\n\n1. As Junio said, it's a bad idea to convert the \"fsck_refs_options *\"\nto \"fsck_options *\" or convert the \"fsck_options *\" to\n\"fsck_refs_options *\". So this version, I changed a lot for the first\npatch. I refactor the \"fsck_options *\" which contains refs-specific,\nobjs-specific and common settings. This designs allows us to use\ncombination to make the code much more clean.\n2. As Karthik said, I should enhance documents for \"git refs verify\".\nThis version does this.\n\nAnd Junio found that the commit the previous series based on is strange.\nIn order to make our life easier, I change the based on commit. Now it\nis based on the top master. And I will always keep the code align with\nthe master to avoid strange things.\n\n 66ac6e4bcd (The fourteenth batch, 2024-06-17)\n\nThanks\n\nshejialuo (7):\n fsck: add refs check interfaces to interact with fsck error levels\n refs: set up ref consistency check infrastructure\n builtin/refs: add verify subcommand\n builtin/fsck: add `git-refs verify` child process\n files-backend: add unified interface for refs scanning\n fsck: add ref name check for files backend\n fsck: add ref content check for files backend\n\n Documentation/fsck-msgids.txt | 12 ++\n Documentation/git-refs.txt | 13 ++\n builtin/fsck.c | 25 +++-\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 5 +-\n builtin/refs.c | 45 ++++++\n builtin/unpack-objects.c | 2 +-\n fetch-pack.c | 8 +-\n fsck.c | 113 +++++++++++----\n fsck.h | 83 ++++++++---\n object-file.c | 2 +-\n refs.c | 7 +-\n refs.h | 8 ++\n refs/debug.c | 11 ++\n refs/files-backend.c | 255 +++++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 ++\n refs/refs-internal.h | 11 +-\n refs/reftable-backend.c | 8 ++\n t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++\n 19 files changed, 765 insertions(+), 64 deletions(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\nRange-diff against v3:\n1: b8d20c4be9 < -: ---------- fsck: add refs check interfaces to interface with fsck error levels\n-: ---------- > 1: 2d2c1f2cd3 fsck: add refs check interfaces to interact with fsck error levels\n2: c7ed95d6b4 ! 2: 4bacffe44e refs: set up ref consistency check infrastructure\n @@ refs.c: int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n -+int refs_fsck(struct ref_store *refs, struct fsck_refs_options *o)\n ++int refs_fsck(struct ref_store *refs, struct fsck_options *o)\n +{\n +\treturn refs->be->fsck(refs, o);\n +}\n @@ refs.h\n \n #include \"commit.h\"\n \n -+struct fsck_refs_options;\n ++struct fsck_options;\n struct object_id;\n struct ref_store;\n struct repository;\n @@ refs.h: int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void\n + * reflogs are consistent, and non-zero otherwise. The errors will be\n + * written to stderr.\n + */\n -+int refs_fsck(struct ref_store *refs, struct fsck_refs_options *o);\n ++int refs_fsck(struct ref_store *refs, struct fsck_options *o);\n +\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n @@ refs/debug.c: static int debug_reflog_expire(struct ref_store *ref_store, const\n }\n \n +static int debug_fsck(struct ref_store *ref_store,\n -+\t\t struct fsck_refs_options *o)\n ++\t\t struct fsck_options *o)\n +{\n +\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n +\tint res = drefs->refs->be->fsck(drefs->refs, o);\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n }\n \n +static int files_fsck(struct ref_store *ref_store,\n -+\t\t struct fsck_refs_options *o)\n ++\t\t struct fsck_options *o)\n +{\n +\tint ret;\n +\tstruct files_ref_store *refs =\n @@ refs/packed-backend.c: static struct ref_iterator *packed_reflog_iterator_begin(\n }\n \n +static int packed_fsck(struct ref_store *ref_store,\n -+\t\t struct fsck_refs_options *o)\n ++\t\t struct fsck_options *o)\n +{\n +\treturn 0;\n +}\n @@ refs/refs-internal.h\n #include \"refs.h\"\n #include \"iterator.h\"\n \n -+struct fsck_refs_options;\n ++struct fsck_options;\n struct ref_transaction;\n \n /*\n @@ refs/refs-internal.h: typedef int read_raw_ref_fn(struct ref_store *ref_store, c\n \t\t\t\t struct strbuf *referent);\n \n +typedef int fsck_fn(struct ref_store *ref_store,\n -+\t\t struct fsck_refs_options *o);\n ++\t\t struct fsck_options *o);\n +\n struct ref_storage_be {\n \tconst char *name;\n @@ refs/reftable-backend.c: static int reftable_be_reflog_expire(struct ref_store *\n }\n \n +static int reftable_be_fsck(struct ref_store *ref_store,\n -+\t\t\t struct fsck_refs_options *o)\n ++\t\t\t struct fsck_options *o)\n +{\n +\treturn 0;\n +}\n3: 373f470fdc ! 3: f00acab4d1 builtin/refs: add verify subcommand\n @@ Documentation/git-refs.txt: include::ref-storage-format.txt[]\n +\n +--strict::\n +\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n -+\tbe seen as ERROR.\n ++\tbe seen as ERROR. See linkgit:git-fsck[1].\n +\n +--verbose::\n +\tWhen verifying the reference database consistency, be chatty.\n @@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c\n +\t};\n +\tint ret = 0;\n +\tunsigned int verbose = 0, strict = 0;\n -+\tstruct fsck_refs_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n ++\tstruct fsck_options fsck_options = FSCK_OPTIONS_DEFAULT;\n +\tstruct option options[] = {\n +\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n +\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n @@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c\n +\t\tusage(_(\"too many arguments\"));\n +\n +\tif (verbose)\n -+\t\tfsck_refs_options.verbose = 1;\n ++\t\tfsck_options.verbose = 1;\n +\tif (strict)\n -+\t\tfsck_refs_options.strict = 1;\n ++\t\tfsck_options.strict = 1;\n +\n -+\tgit_config(git_fsck_refs_config, &fsck_refs_options);\n ++\tgit_config(git_fsck_config, &fsck_options);\n +\tprepare_repo_settings(the_repository);\n +\n -+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n ++\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_options);\n +\n +\t/*\n -+\t * Explicitly free the allocated array. This is necessary because\n -+\t * this program is executed as child process of git-fsck(1) and the\n -+\t * allocated array may not freed when git-fsck(1) aborts somewhere.\n ++\t * Explicitly free the allocated array and object skiplist set. Because\n ++\t * we reuse `git_fsck_config` here. It will still set the skiplist.\n +\t */\n -+\tfree(fsck_refs_options.msg_type);\n ++\tfree(fsck_options.msg_type);\n ++\toidset_clear(&fsck_options.objs_options.skiplist);\n +\treturn ret;\n +}\n +\n4: 4f8ebea573 = 4: 6494f5b0c2 builtin/fsck: add `git-refs verify` child process\n5: 6062b3b453 ! 5: 2c064f4f71 files-backend: add unified interface for refs scanning\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n + * the whole directory. This function is used as the callback for each\n + * regular file or symlink in the directory.\n + */\n -+typedef int (*files_fsck_refs_fn)(struct fsck_refs_options *o,\n ++typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n +\t\t\t\t const char *gitdir,\n +\t\t\t\t const char *refs_check_dir,\n +\t\t\t\t struct dir_iterator *iter);\n +\n +static int files_fsck_refs_dir(struct ref_store *ref_store,\n -+\t\t\t struct fsck_refs_options *o,\n ++\t\t\t struct fsck_options *o,\n +\t\t\t const char *refs_check_dir,\n +\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n +{\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +}\n +\n +static int files_fsck_refs(struct ref_store *ref_store,\n -+\t\t\t struct fsck_refs_options *o)\n ++\t\t\t struct fsck_options *o)\n +{\n +\tint ret;\n +\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +}\n +\n static int files_fsck(struct ref_store *ref_store,\n - \t\t struct fsck_refs_options *o)\n + \t\t struct fsck_options *o)\n {\n @@ refs/files-backend.c: static int files_fsck(struct ref_store *ref_store,\n \tstruct files_ref_store *refs =\n6: 87a22f4c7f ! 6: 88b54b9e1a fsck: add ref name check for files backend\n @@ fsck.h: enum fsck_msg_type {\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n \n ## refs/files-backend.c ##\n -@@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_refs_options *o,\n +@@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n -+static int files_fsck_refs_name(struct fsck_refs_options *o,\n ++static int files_fsck_refs_name(struct fsck_options *o,\n +\t\t\t\tconst char *gitdir UNUSED,\n +\t\t\t\tconst char *refs_check_dir,\n +\t\t\t\tstruct dir_iterator *iter)\n @@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_refs_options\n +}\n +\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n - \t\t\t struct fsck_refs_options *o,\n + \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n @@ refs/files-backend.c: static int files_fsck_refs(struct ref_store *ref_store,\n {\n7: b945f61ef1 ! 7: 8309a4746a fsck: add ref content check for files backend\n @@ refs/files-backend.c: int parse_loose_ref_contents(const char *buf, struct objec\n \treturn 0;\n }\n \n -@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_refs_options *o,\n +@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n \treturn ret;\n }\n \n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_refs_options *\n + * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n + * be the relative path agaignst \"gitdir\".\n + */\n -+static int files_fsck_symref_target(struct fsck_refs_options *o,\n ++static int files_fsck_symref_target(struct fsck_options *o,\n +\t\t\t\t const char *refname,\n +\t\t\t\t const char *pointee_name,\n +\t\t\t\t const char *pointee_path)\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_refs_options *\n +\treturn ret;\n +}\n +\n -+static int files_fsck_refs_content(struct fsck_refs_options *o,\n ++static int files_fsck_refs_content(struct fsck_options *o,\n +\t\t\t\t const char *gitdir,\n +\t\t\t\t const char *refs_check_dir,\n +\t\t\t\t struct dir_iterator *iter)\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_refs_options *\n +}\n +\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n - \t\t\t struct fsck_refs_options *o,\n + \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n @@ refs/files-backend.c: static int files_fsck_refs(struct ref_store *ref_store,\n \tint ret;\n-- \n2.45.2\n\n"},{"id":"497334","messageId":"ZnKLja6bSNLukkTR@ArchLinux","threadId":"61565","inReplyTo":"ZnKKy52QFO2UhqM6@ArchLinux","subject":"[GSoC][PATCH v4 1/7] fsck: add refs check interfaces to interact with fsck error levels","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-19T07:41:01Z","receivedAt":"2024-06-19T07:41:03Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) focuses on object database consistency check. It relies\non the \"fsck_options\" to interact with fsck error levels. However,\n\"fsck_options\" aims at checking the object database which contains a lot\nof fields only related to object database.\n\nIn order to add ref operations, create a new struct named\n\"fsck_refs_options\" and a new struct named \"fsck_objs_options\". Remove\nobject-related fields from \"fsck_options\" to \"fsck_objs_options\". Change\nthe \"fsck_options\" with three parts of members:\n\n1. The \"fsck_refs_options\".\n2. The \"fsck_objs_options\".\n3. The common settings both for refs and objects. Because we leave\n common settings in \"fsck_options\". The setup process could be fully\n reused without any code changing.\n\nAlso add related macros to align with the current code. Because we\nremove some fields from \"fsck_options\" to \"fsck_objs_options\". Change\nthe influenced code to use the \"fsck_options.objs_options\" instead of\nusing \"fsck_options\" itself.\n\nThe static function \"report\" provided by \"fsck.c\" aims at reporting the\nproblems related to object database which cannot be reused for refs.\nProvide \"fsck_refs_report\" function to integrate the fsck error levels\ninto reference consistency check.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 8 +--\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 5 +-\n builtin/unpack-objects.c | 2 +-\n fetch-pack.c | 8 +--\n fsck.c | 113 +++++++++++++++++++++++++++++----------\n fsck.h | 79 ++++++++++++++++++++-------\n object-file.c | 2 +-\n 8 files changed, 161 insertions(+), 58 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..a5b82e228c 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -233,7 +233,7 @@ static void mark_unreachable_referents(const struct object_id *oid)\n \t\t\tobject_as_type(obj, type, 0);\n \t}\n \n-\toptions.walk = mark_used;\n+\toptions.objs_options.walk = mark_used;\n \tfsck_walk(obj, NULL, &options);\n \tif (obj->type == OBJ_TREE)\n \t\tfree_tree_buffer((struct tree *)obj);\n@@ -936,9 +936,9 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \targc = parse_options(argc, argv, prefix, fsck_opts, fsck_usage, 0);\n \n-\tfsck_walk_options.walk = mark_object;\n-\tfsck_obj_options.walk = mark_used;\n-\tfsck_obj_options.error_func = fsck_error_func;\n+\tfsck_walk_options.objs_options.walk = mark_object;\n+\tfsck_obj_options.objs_options.walk = mark_used;\n+\tfsck_obj_options.objs_options.error_func = fsck_error_func;\n \tif (check_strict)\n \t\tfsck_obj_options.strict = 1;\n \ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 856428fef9..8824d7b8a9 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1746,7 +1746,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\tusage(index_pack_usage);\n \n \tdisable_replace_refs();\n-\tfsck_options.walk = mark_link;\n+\tfsck_options.objs_options.walk = mark_link;\n \n \treset_pack_idx_option(&opts);\n \topts.flags |= WRITE_REV;\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex 4767f1a97e..e88475f009 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -91,8 +91,9 @@ int cmd_mktag(int argc, const char **argv, const char *prefix)\n \tif (strbuf_read(&buf, 0, 0) < 0)\n \t\tdie_errno(_(\"could not read from stdin\"));\n \n-\tfsck_options.error_func = mktag_fsck_error_func;\n-\tfsck_set_msg_type_from_ids(&fsck_options, FSCK_MSG_EXTRA_HEADER_ENTRY,\n+\tfsck_options.objs_options.error_func = mktag_fsck_error_func;\n+\tfsck_set_msg_type_from_ids(&fsck_options,\n+\t\t\t\t FSCK_MSG_EXTRA_HEADER_ENTRY,\n \t\t\t\t FSCK_WARN);\n \t/* config might set fsck.extraHeaderEntry=* again */\n \tgit_config(git_fsck_config, &fsck_options);\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex f1c85a00ae..368810a2a1 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -239,7 +239,7 @@ static int check_object(struct object *obj, enum object_type type,\n \t\tdie(\"Whoops! Cannot find object '%s'\", oid_to_hex(&obj->oid));\n \tif (fsck_object(obj, obj_buf->buffer, obj_buf->size, &fsck_options))\n \t\tdie(\"fsck error in packed object\");\n-\tfsck_options.walk = check_object;\n+\tfsck_options.objs_options.walk = check_object;\n \tif (fsck_walk(obj, NULL, &fsck_options))\n \t\tdie(\"Error on reachable objects of %s\", oid_to_hex(&obj->oid));\n \twrite_cached_object(obj, obj_buf);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex eba9e420ea..148f9bd371 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1220,7 +1220,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t} else\n \t\talternate_shallow_file = NULL;\n \tif (get_pack(args, fd, pack_lockfiles, NULL, sought, nr_sought,\n-\t\t &fsck_options.gitmodules_found))\n+\t\t &fsck_options.objs_options.gitmodules_found))\n \t\tdie(_(\"git fetch-pack: fetch failed.\"));\n \tif (fsck_finish(&fsck_options))\n \t\tdie(\"fsck failed\");\n@@ -1780,7 +1780,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \n \t\t\tif (get_pack(args, fd, pack_lockfiles,\n \t\t\t\t packfile_uris.nr ? &index_pack_args : NULL,\n-\t\t\t\t sought, nr_sought, &fsck_options.gitmodules_found))\n+\t\t\t\t sought, nr_sought,\n+\t\t\t\t &fsck_options.objs_options.gitmodules_found))\n \t\t\t\tdie(_(\"git fetch-pack: fetch failed.\"));\n \t\t\tdo_check_stateless_delimiter(args->stateless_rpc, &reader);\n \n@@ -1823,7 +1824,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \n \t\tpackname[the_hash_algo->hexsz] = '\\0';\n \n-\t\tparse_gitmodules_oids(cmd.out, &fsck_options.gitmodules_found);\n+\t\tparse_gitmodules_oids(cmd.out,\n+\t\t\t\t &fsck_options.objs_options.gitmodules_found);\n \n \t\tclose(cmd.out);\n \ndiff --git a/fsck.c b/fsck.c\nindex e193930ae7..4949d2e110 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -203,7 +203,8 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n \t\tif (!strcmp(buf, \"skiplist\")) {\n \t\t\tif (equal == len)\n \t\t\t\tdie(\"skiplist requires a path\");\n-\t\t\toidset_parse_file(&options->skiplist, buf + equal + 1);\n+\t\t\toidset_parse_file(&options->objs_options.skiplist,\n+\t\t\t\t\t buf + equal + 1);\n \t\t\tbuf += len + 1;\n \t\t\tcontinue;\n \t\t}\n@@ -220,7 +221,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n static int object_on_skiplist(struct fsck_options *opts,\n \t\t\t const struct object_id *oid)\n {\n-\treturn opts && oid && oidset_contains(&opts->skiplist, oid);\n+\treturn opts && oid && oidset_contains(&opts->objs_options.skiplist, oid);\n }\n \n __attribute__((format (printf, 5, 6)))\n@@ -249,8 +250,8 @@ static int report(struct fsck_options *options,\n \n \tva_start(ap, fmt);\n \tstrbuf_vaddf(&sb, fmt, ap);\n-\tresult = options->error_func(options, oid, object_type,\n-\t\t\t\t msg_type, msg_id, sb.buf);\n+\tresult = options->objs_options.error_func(options, oid, object_type,\n+\t\t\t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n \tva_end(ap);\n \n@@ -259,20 +260,20 @@ static int report(struct fsck_options *options,\n \n void fsck_enable_object_names(struct fsck_options *options)\n {\n-\tif (!options->object_names)\n-\t\toptions->object_names = kh_init_oid_map();\n+\tif (!options->objs_options.object_names)\n+\t\toptions->objs_options.object_names = kh_init_oid_map();\n }\n \n const char *fsck_get_object_name(struct fsck_options *options,\n \t\t\t\t const struct object_id *oid)\n {\n \tkhiter_t pos;\n-\tif (!options->object_names)\n+\tif (!options->objs_options.object_names)\n \t\treturn NULL;\n-\tpos = kh_get_oid_map(options->object_names, *oid);\n-\tif (pos >= kh_end(options->object_names))\n+\tpos = kh_get_oid_map(options->objs_options.object_names, *oid);\n+\tif (pos >= kh_end(options->objs_options.object_names))\n \t\treturn NULL;\n-\treturn kh_value(options->object_names, pos);\n+\treturn kh_value(options->objs_options.object_names, pos);\n }\n \n void fsck_put_object_name(struct fsck_options *options,\n@@ -284,15 +285,17 @@ void fsck_put_object_name(struct fsck_options *options,\n \tkhiter_t pos;\n \tint hashret;\n \n-\tif (!options->object_names)\n+\tif (!options->objs_options.object_names)\n \t\treturn;\n \n-\tpos = kh_put_oid_map(options->object_names, *oid, &hashret);\n+\tpos = kh_put_oid_map(options->objs_options.object_names,\n+\t\t\t *oid, &hashret);\n \tif (!hashret)\n \t\treturn;\n \tva_start(ap, fmt);\n \tstrbuf_vaddf(&buf, fmt, ap);\n-\tkh_value(options->object_names, pos) = strbuf_detach(&buf, NULL);\n+\tkh_value(options->objs_options.object_names, pos) =\n+\t\tstrbuf_detach(&buf, NULL);\n \tva_end(ap);\n }\n \n@@ -318,6 +321,7 @@ const char *fsck_describe_object(struct fsck_options *options,\n \n static int fsck_walk_tree(struct tree *tree, void *data, struct fsck_options *options)\n {\n+\tstruct fsck_objs_options *objs_options = &options->objs_options;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tint res = 0;\n@@ -342,14 +346,14 @@ static int fsck_walk_tree(struct tree *tree, void *data, struct fsck_options *op\n \t\t\tif (name && obj)\n \t\t\t\tfsck_put_object_name(options, &entry.oid, \"%s%s/\",\n \t\t\t\t\t\t name, entry.path);\n-\t\t\tresult = options->walk(obj, OBJ_TREE, data, options);\n+\t\t\tresult = objs_options->walk(obj, OBJ_TREE, data, options);\n \t\t}\n \t\telse if (S_ISREG(entry.mode) || S_ISLNK(entry.mode)) {\n \t\t\tobj = (struct object *)lookup_blob(the_repository, &entry.oid);\n \t\t\tif (name && obj)\n \t\t\t\tfsck_put_object_name(options, &entry.oid, \"%s%s\",\n \t\t\t\t\t\t name, entry.path);\n-\t\t\tresult = options->walk(obj, OBJ_BLOB, data, options);\n+\t\t\tresult = objs_options->walk(obj, OBJ_BLOB, data, options);\n \t\t}\n \t\telse {\n \t\t\tresult = error(\"in tree %s: entry %s has bad mode %.6o\",\n@@ -366,6 +370,7 @@ static int fsck_walk_tree(struct tree *tree, void *data, struct fsck_options *op\n \n static int fsck_walk_commit(struct commit *commit, void *data, struct fsck_options *options)\n {\n+\tstruct fsck_objs_options *objs_options = &options->objs_options;\n \tint counter = 0, generation = 0, name_prefix_len = 0;\n \tstruct commit_list *parents;\n \tint res;\n@@ -380,8 +385,8 @@ static int fsck_walk_commit(struct commit *commit, void *data, struct fsck_optio\n \t\tfsck_put_object_name(options, get_commit_tree_oid(commit),\n \t\t\t\t \"%s:\", name);\n \n-\tresult = options->walk((struct object *) repo_get_commit_tree(the_repository, commit),\n-\t\t\t OBJ_TREE, data, options);\n+\tresult = objs_options->walk((struct object *) repo_get_commit_tree(the_repository, commit),\n+\t\t\t\t OBJ_TREE, data, options);\n \tif (result < 0)\n \t\treturn result;\n \tres = result;\n@@ -423,7 +428,8 @@ static int fsck_walk_commit(struct commit *commit, void *data, struct fsck_optio\n \t\t\telse\n \t\t\t\tfsck_put_object_name(options, oid, \"%s^\", name);\n \t\t}\n-\t\tresult = options->walk((struct object *)parents->item, OBJ_COMMIT, data, options);\n+\t\tresult = objs_options->walk((struct object *)parents->item,\n+\t\t\t\t\t OBJ_COMMIT, data, options);\n \t\tif (result < 0)\n \t\t\treturn result;\n \t\tif (!res)\n@@ -436,12 +442,13 @@ static int fsck_walk_commit(struct commit *commit, void *data, struct fsck_optio\n static int fsck_walk_tag(struct tag *tag, void *data, struct fsck_options *options)\n {\n \tconst char *name = fsck_get_object_name(options, &tag->object.oid);\n+\tstruct fsck_objs_options *objs_options = &options->objs_options;\n \n \tif (parse_tag(tag))\n \t\treturn -1;\n \tif (name)\n \t\tfsck_put_object_name(options, &tag->tagged->oid, \"%s\", name);\n-\treturn options->walk(tag->tagged, OBJ_ANY, data, options);\n+\treturn objs_options->walk(tag->tagged, OBJ_ANY, data, options);\n }\n \n int fsck_walk(struct object *obj, void *data, struct fsck_options *options)\n@@ -598,6 +605,7 @@ static int fsck_tree(const struct object_id *tree_oid,\n \tunsigned o_mode;\n \tconst char *o_name;\n \tstruct name_stack df_dup_candidates = { NULL };\n+\tstruct fsck_objs_options *objs_options = &options->objs_options;\n \n \tif (init_tree_desc_gently(&desc, tree_oid, buffer, size,\n \t\t\t\t TREE_DESC_RAW_MODES)) {\n@@ -628,7 +636,7 @@ static int fsck_tree(const struct object_id *tree_oid,\n \n \t\tif (is_hfs_dotgitmodules(name) || is_ntfs_dotgitmodules(name)) {\n \t\t\tif (!S_ISLNK(mode))\n-\t\t\t\toidset_insert(&options->gitmodules_found,\n+\t\t\t\toidset_insert(&objs_options->gitmodules_found,\n \t\t\t\t\t entry_oid);\n \t\t\telse\n \t\t\t\tretval += report(options,\n@@ -639,7 +647,7 @@ static int fsck_tree(const struct object_id *tree_oid,\n \n \t\tif (is_hfs_dotgitattributes(name) || is_ntfs_dotgitattributes(name)) {\n \t\t\tif (!S_ISLNK(mode))\n-\t\t\t\toidset_insert(&options->gitattributes_found,\n+\t\t\t\toidset_insert(&objs_options->gitattributes_found,\n \t\t\t\t\t entry_oid);\n \t\t\telse\n \t\t\t\tretval += report(options, tree_oid, OBJ_TREE,\n@@ -666,7 +674,7 @@ static int fsck_tree(const struct object_id *tree_oid,\n \t\t\t\thas_dotgit |= is_ntfs_dotgit(backslash);\n \t\t\t\tif (is_ntfs_dotgitmodules(backslash)) {\n \t\t\t\t\tif (!S_ISLNK(mode))\n-\t\t\t\t\t\toidset_insert(&options->gitmodules_found,\n+\t\t\t\t\t\toidset_insert(&objs_options->gitmodules_found,\n \t\t\t\t\t\t\t entry_oid);\n \t\t\t\t\telse\n \t\t\t\t\t\tretval += report(options, tree_oid, OBJ_TREE,\n@@ -1102,16 +1110,17 @@ static int fsck_gitmodules_fn(const char *var, const char *value,\n static int fsck_blob(const struct object_id *oid, const char *buf,\n \t\t unsigned long size, struct fsck_options *options)\n {\n+\tstruct fsck_objs_options *objs_options = &options->objs_options;\n \tint ret = 0;\n \n \tif (object_on_skiplist(options, oid))\n \t\treturn 0;\n \n-\tif (oidset_contains(&options->gitmodules_found, oid)) {\n+\tif (oidset_contains(&objs_options->gitmodules_found, oid)) {\n \t\tstruct config_options config_opts = { 0 };\n \t\tstruct fsck_gitmodules_data data;\n \n-\t\toidset_insert(&options->gitmodules_done, oid);\n+\t\toidset_insert(&objs_options->gitmodules_done, oid);\n \n \t\tif (!buf) {\n \t\t\t/*\n@@ -1137,13 +1146,14 @@ static int fsck_blob(const struct object_id *oid, const char *buf,\n \t\tret |= data.ret;\n \t}\n \n-\tif (oidset_contains(&options->gitattributes_found, oid)) {\n+\tif (oidset_contains(&objs_options->gitattributes_found, oid)) {\n \t\tconst char *ptr;\n \n-\t\toidset_insert(&options->gitattributes_done, oid);\n+\t\toidset_insert(&objs_options->gitattributes_done, oid);\n \n \t\tif (!buf || size > ATTR_MAX_FILE_SIZE) {\n \t\t\t/*\n+\n \t\t\t * A missing buffer here is a sign that the caller found the\n \t\t\t * blob too gigantic to load into memory. Let's just consider\n \t\t\t * that an error.\n@@ -1197,6 +1207,20 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n+int fsck_refs_error_function(struct fsck_options *o UNUSED,\n+\t\t\t const char *name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tif (msg_type == FSCK_WARN) {\n+\t\twarning(\"%s: %s\", name, message);\n+\t\treturn 0;\n+\t}\n+\terror(\"%s: %s\", name, message);\n+\treturn 1;\n+}\n+\n int fsck_error_function(struct fsck_options *o,\n \t\t\tconst struct object_id *oid,\n \t\t\tenum object_type object_type UNUSED,\n@@ -1255,18 +1279,51 @@ static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \n int fsck_finish(struct fsck_options *options)\n {\n+\tstruct fsck_objs_options *objs_options = &options->objs_options;\n \tint ret = 0;\n \n-\tret |= fsck_blobs(&options->gitmodules_found, &options->gitmodules_done,\n+\tret |= fsck_blobs(&objs_options->gitmodules_found,\n+\t\t\t &objs_options->gitmodules_done,\n \t\t\t FSCK_MSG_GITMODULES_MISSING, FSCK_MSG_GITMODULES_BLOB,\n \t\t\t options, \".gitmodules\");\n-\tret |= fsck_blobs(&options->gitattributes_found, &options->gitattributes_done,\n+\tret |= fsck_blobs(&objs_options->gitattributes_found,\n+\t\t\t &objs_options->gitattributes_done,\n \t\t\t FSCK_MSG_GITATTRIBUTES_MISSING, FSCK_MSG_GITATTRIBUTES_BLOB,\n \t\t\t options, \".gitattributes\");\n \n \treturn ret;\n }\n \n+int fsck_refs_report(struct fsck_options *o,\n+\t\t const char *name,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, o);\n+\tint ret = 0;\n+\n+\tif (msg_type == FSCK_IGNORE)\n+\t\treturn 0;\n+\n+\tif (msg_type == FSCK_FATAL)\n+\t\tmsg_type = FSCK_ERROR;\n+\telse if (msg_type == FSCK_INFO)\n+\t\tmsg_type = FSCK_WARN;\n+\n+\tprepare_msg_ids();\n+\tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n+\n+\tva_start(ap, fmt);\n+\tstrbuf_vaddf(&sb, fmt, ap);\n+\tret = o->refs_options.error_func(o, name, msg_type, msg_id, sb.buf);\n+\tstrbuf_release(&sb);\n+\tva_end(ap);\n+\n+\treturn ret;\n+}\n+\n int git_fsck_config(const char *var, const char *value,\n \t\t const struct config_context *ctx, void *cb)\n {\ndiff --git a/fsck.h b/fsck.h\nindex 6085a384f6..0391dffbb0 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -103,6 +103,21 @@ void fsck_set_msg_type(struct fsck_options *options,\n void fsck_set_msg_types(struct fsck_options *options, const char *values);\n int is_valid_msg_type(const char *msg_id, const char *msg_type);\n \n+/*\n+ * callback function for fsck refs and reflogs.\n+ */\n+typedef int (*fsck_refs_error)(struct fsck_options *o,\n+\t\t\t const char *name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n+\n+int fsck_refs_error_function(struct fsck_options *o,\n+\t\t\t const char *name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n+\n /*\n * callback function for fsck_walk\n * type is the expected type of the object or OBJ_ANY\n@@ -115,10 +130,12 @@ typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n \t\t\t void *data, struct fsck_options *options);\n \n /* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n-typedef int (*fsck_error)(struct fsck_options *o,\n-\t\t\t const struct object_id *oid, enum object_type object_type,\n-\t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n-\t\t\t const char *message);\n+typedef int (*fsck_obj_error)(struct fsck_options *o,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n \n int fsck_error_function(struct fsck_options *o,\n \t\t\tconst struct object_id *oid, enum object_type object_type,\n@@ -131,11 +148,17 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message);\n \n-struct fsck_options {\n+struct fsck_refs_options {\n+\tfsck_refs_error error_func;\n+};\n+\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n+\n+struct fsck_objs_options {\n \tfsck_walk_func walk;\n-\tfsck_error error_func;\n-\tunsigned strict:1;\n-\tenum fsck_msg_type *msg_type;\n+\tfsck_obj_error error_func;\n \tstruct oidset skiplist;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n@@ -144,29 +167,43 @@ struct fsck_options {\n \tkh_oid_map_t *object_names;\n };\n \n-#define FSCK_OPTIONS_DEFAULT { \\\n+#define FSCK_OBJS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_error_function, \\\n \t.skiplist = OIDSET_INIT, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function \\\n }\n-#define FSCK_OPTIONS_STRICT { \\\n-\t.strict = 1, \\\n+#define FSCK_OBJS_OPTIONS_MISSING_GITMODULES { \\\n+\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function, \\\n+}\n+\n+struct fsck_options {\n+\tstruct fsck_refs_options refs_options;\n+\tstruct fsck_objs_options objs_options;\n+\tenum fsck_msg_type *msg_type;\n+\tunsigned strict:1,\n+\t\t verbose:1;\n+};\n+\n+#define FSCK_OPTIONS_DEFAULT { \\\n+\t.refs_options = FSCK_REFS_OPTIONS_DEFAULT, \\\n+\t.objs_options = FSCK_OBJS_OPTIONS_DEFAULT, \\\n+}\n+#define FSCK_OPTIONS_STRICT { \\\n+\t.refs_options = FSCK_REFS_OPTIONS_DEFAULT, \\\n+\t.objs_options = FSCK_OBJS_OPTIONS_DEFAULT, \\\n+\t.strict = 1, \\\n }\n #define FSCK_OPTIONS_MISSING_GITMODULES { \\\n+\t.refs_options = FSCK_REFS_OPTIONS_DEFAULT, \\\n+\t.objs_options = FSCK_OBJS_OPTIONS_MISSING_GITMODULES, \\\n \t.strict = 1, \\\n-\t.gitmodules_found = OIDSET_INIT, \\\n-\t.gitmodules_done = OIDSET_INIT, \\\n-\t.gitattributes_found = OIDSET_INIT, \\\n-\t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n }\n \n /* descend in all linked child objects\n@@ -209,6 +246,12 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+__attribute__((format (printf, 4, 5)))\n+int fsck_refs_report(struct fsck_options *o,\n+\t\t const char *name,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\ndiff --git a/object-file.c b/object-file.c\nindex d3cf4b8b2e..b027d70725 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2510,7 +2510,7 @@ static int index_mem(struct index_state *istate,\n \t\tstruct fsck_options opts = FSCK_OPTIONS_DEFAULT;\n \n \t\topts.strict = 1;\n-\t\topts.error_func = hash_format_check_report;\n+\t\topts.objs_options.error_func = hash_format_check_report;\n \t\tif (fsck_buffer(null_oid(), type, buf, size, &opts))\n \t\t\tdie(_(\"refusing to create malformed object\"));\n \t\tfsck_finish(&opts);\n-- \n2.45.2\n\n"},{"id":"497335","messageId":"ZnKLr-H_HI-I-7xo@ArchLinux","threadId":"61565","inReplyTo":"ZnKKy52QFO2UhqM6@ArchLinux","subject":"[GSoC][PATCH v4 2/7] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-19T07:41:35Z","receivedAt":"2024-06-19T07:41:38Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It's organized as the five parts:\n\n1. The name and the initialization interfaces.\n2. The ref transaction interfaces.\n3. The ref internal interfaces (pack, rename and copy).\n4. The ref filesystem interfaces.\n5. The reflog related interfaces.\n\nTo keep consistent with the git-fsck(1), add a new interface named\n\"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\ngrouped into any above five categories. Explicitly add blank line to\nmake it different from others.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs.c | 5 +++++\n refs.h | 8 ++++++++\n refs/debug.c | 11 +++++++++++\n refs/files-backend.c | 15 ++++++++++++++-\n refs/packed-backend.c | 8 ++++++++\n refs/refs-internal.h | 6 ++++++\n refs/reftable-backend.c | 8 ++++++++\n 7 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex 0c1e45f5a9..cb3b8ec36a 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -316,6 +316,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs, struct fsck_options *o)\n+{\n+\treturn refs->be->fsck(refs, o);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex 756039a051..93fa23c7ee 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -3,6 +3,7 @@\n \n #include \"commit.h\"\n \n+struct fsck_options;\n struct object_id;\n struct ref_store;\n struct repository;\n@@ -547,6 +548,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_options *o);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..45e2e784a0 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tint res = drefs->refs->be->fsck(drefs->refs, o);\n+\ttrace_printf_key(&trace_refs, \"fsck: %d\\n\", res);\n+\treturn res;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 4519b46171..ba9e57d1e9 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3405,6 +3405,17 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\treturn ret;\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_init,\n@@ -3431,5 +3442,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex c4c1e36aa2..ad3e8fb1d1 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1733,6 +1733,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_init,\n@@ -1760,4 +1766,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex cbcb6f9c36..280acb7f9e 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_options;\n struct ref_transaction;\n \n /*\n@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_options *o);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -677,6 +681,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 3a75d63eb5..2b2630d47a 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2281,6 +2281,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2308,4 +2314,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.45.2\n\n"},{"id":"497336","messageId":"ZnKL0-u_s19-Fdmt@ArchLinux","threadId":"61565","inReplyTo":"ZnKKy52QFO2UhqM6@ArchLinux","subject":"[GSoC][PATCH v4 3/7] builtin/refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-19T07:42:11Z","receivedAt":"2024-06-19T07:42:14Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\ncheck the reference database consistency.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/git-refs.txt | 13 +++++++++++\n builtin/refs.c | 45 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 58 insertions(+)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..1244a85b64 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+The following options are specific to 'git refs verify':\n+\n+--strict::\n+\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n+\tbe seen as ERROR. See linkgit:git-fsck[1].\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..231a051f6b 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,55 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tint ret = 0;\n+\tunsigned int verbose = 0, strict = 0;\n+\tstruct fsck_options fsck_options = FSCK_OPTIONS_DEFAULT;\n+\tstruct option options[] = {\n+\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"too many arguments\"));\n+\n+\tif (verbose)\n+\t\tfsck_options.verbose = 1;\n+\tif (strict)\n+\t\tfsck_options.strict = 1;\n+\n+\tgit_config(git_fsck_config, &fsck_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_options);\n+\n+\t/*\n+\t * Explicitly free the allocated array and object skiplist set. Because\n+\t * we reuse `git_fsck_config` here. It will still set the skiplist.\n+\t */\n+\tfree(fsck_options.msg_type);\n+\toidset_clear(&fsck_options.objs_options.skiplist);\n+\treturn ret;\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \n-- \n2.45.2\n\n"},{"id":"497337","messageId":"ZnKL7Evg-lLIzW4e@ArchLinux","threadId":"61565","inReplyTo":"ZnKKy52QFO2UhqM6@ArchLinux","subject":"[GSoC][PATCH v4 4/7] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-19T07:42:36Z","receivedAt":"2024-06-19T07:42:39Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new function \"fsck_refs\" that initializes and runs a child\nprocess to execute the \"git-refs verify\" command.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex a5b82e228c..cd988ff167 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -896,6 +896,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \treturn res;\n }\n \n+static void fsck_refs(void)\n+{\n+\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n+\tchild_process_init(&refs_verify);\n+\trefs_verify.git_cmd = 1;\n+\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n+\tif (verbose)\n+\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n+\tif (check_strict)\n+\t\tstrvec_push(&refs_verify.args, \"--strict\");\n+\n+\tif (run_command(&refs_verify))\n+\t\terrors_found |= ERROR_REFS;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n@@ -1065,6 +1080,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n+\tfsck_refs();\n+\n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \n-- \n2.45.2\n\n"},{"id":"497338","messageId":"ZnKMOVsHs-RlheS_@ArchLinux","threadId":"61565","inReplyTo":"ZnKKy52QFO2UhqM6@ArchLinux","subject":"[GSoC][PATCH v4 5/7] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-19T07:43:53Z","receivedAt":"2024-06-19T07:43:56Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"For refs and reflogs, we need to scan its corresponding directories to\ncheck every regular file or symbolic link which shares the same pattern.\nIntroduce a unified interface for scanning directories for\nfiles-backend.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs/files-backend.c | 77 +++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 76 insertions(+), 1 deletion(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex ba9e57d1e9..5156530774 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -4,6 +4,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -3405,6 +3406,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n+{\n+\tconst char *gitdir = ref_store->gitdir;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\n+\tif (!iter) {\n+\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n+\t\tgoto out;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->verbose)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n+\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n+\t\t\t\t iter->basename);\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n+\n+out:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tNULL\n+\t};\n+\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, \"Checking references consistency\");\n+\n+\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n+\n+\treturn ret;\n+}\n+\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_options *o)\n {\n@@ -3412,7 +3485,9 @@ static int files_fsck(struct ref_store *ref_store,\n \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n-\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o)\n+\t | files_fsck_refs(ref_store, o);\n+\n \treturn ret;\n }\n \n-- \n2.45.2\n\n"},{"id":"497339","messageId":"ZnKMUV_ozbcg7NG6@ArchLinux","threadId":"61565","inReplyTo":"ZnKKy52QFO2UhqM6@ArchLinux","subject":"[GSoC][PATCH v4 6/7] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-19T07:44:17Z","receivedAt":"2024-06-19T07:44:20Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) only implicitly checks the reference, it does not fully\ncheck refs with bad format name such as standalone \"@\" and name ending\nwith \".lock\".\n\nIn order to provide such checks, add a new fsck message id \"badRefName\"\nwith default ERROR type. Use existing \"check_refname_format\" to explicit\ncheck the ref name. And add a new unit test to verify the functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 +\n fsck.h | 1 +\n refs/files-backend.c | 20 +++++++\n t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++\n 4 files changed, 125 insertions(+)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex f643585a34..dab4012246 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,9 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefName`::\n+\t(ERROR) A ref has a bad name.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex 0391dffbb0..f26dec2ea4 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 5156530774..5bc233b524 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3416,6 +3416,25 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n+static int files_fsck_refs_name(struct fsck_options *o,\n+\t\t\t\tconst char *gitdir UNUSED,\n+\t\t\t\tconst char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\tret = fsck_refs_report(o, sb.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3467,6 +3486,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n {\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tfiles_fsck_refs_name,\n \t\tNULL\n \t};\n \ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..b2db58d2c6\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit tag multi_hierarchy/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1.lock: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/@ &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\tgit -c fsck.badRefName=warn fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\tgit -c fsck.badRefName=ignore fsck 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+test_done\n-- \n2.45.2\n\n"},{"id":"497340","messageId":"ZnKMY4VdHKHAMNY6@ArchLinux","threadId":"61565","inReplyTo":"ZnKKy52QFO2UhqM6@ArchLinux","subject":"[GSoC][PATCH v4 7/7] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-19T07:44:35Z","receivedAt":"2024-06-19T07:44:38Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Enhance the git-fsck(1) command by adding a check for reference content\nin the files backend. The new functionality ensures that symrefs, real\nsymbolic link and regular refs are validated correctly.\n\nIn order to check the trailing content of the regular refs, add a new\nparameter `trailing` to `parse_loose_ref_contents`.\n\nFor symrefs, `parse_loose_ref_contents` will set the \"referent\".\nHowever, symbolic link could be either absolute or relative. Use\n\"strbuf_add_real_path\" to read the symbolic link and convert the\nrelative path to absolute path. Then use \"skip_prefix\" to make it align\nwith symref \"referent\".\n\nThus, the symrefs and symbolic links could share the same interface. Add\na new function \"files_fsck_symref_target\" which aims at checking the\nfollowing things:\n\n1. whether the pointee is under the `refs/` directory.\n2. whether the pointee name is correct.\n3. whether the pointee path is a wrong type in filesystem.\n\nLast, add the following FSCK MESSAGEs:\n\n1. \"badRefContent(ERROR)\": A ref has a bad content\n2. \"badSymrefPointee(ERROR)\": The pointee of a symref is bad.\n3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 9 +++\n fsck.h | 3 +\n refs.c | 2 +-\n refs/files-backend.c | 145 +++++++++++++++++++++++++++++++++-\n refs/refs-internal.h | 5 +-\n t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++\n 6 files changed, 269 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex dab4012246..b1630a478b 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,9 +19,15 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefContent`::\n+\t(ERROR) A ref has a bad content.\n+\n `badRefName`::\n \t(ERROR) A ref has a bad name.\n \n+`badSymrefPointee`::\n+\t(ERROR) The pointee of a symref is bad.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \n@@ -167,6 +173,9 @@\n `nullSha1`::\n \t(WARN) Tree contains entries pointing to a null sha1.\n \n+`trailingRefContent`::\n+\t(WARN) A ref content has trailing contents.\n+\n `treeNotSorted`::\n \t(ERROR) A tree is not properly sorted.\n \ndiff --git a/fsck.h b/fsck.h\nindex f26dec2ea4..8afee05f20 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -32,6 +32,8 @@ enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n+\tFUNC(BAD_REF_CONTENT, ERROR) \\\n+\tFUNC(BAD_SYMREF_POINTEE, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n@@ -72,6 +74,7 @@ enum fsck_msg_type {\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n+\tFUNC(TRAILING_REF_CONTENT, WARN) \\\n \tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n \tFUNC(LARGE_PATHNAME, WARN) \\\ndiff --git a/refs.c b/refs.c\nindex cb3b8ec36a..5fc34d39b1 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1744,7 +1744,7 @@ static int refs_read_special_head(struct ref_store *ref_store,\n \t}\n \n \tresult = parse_loose_ref_contents(content.buf, oid, referent, type,\n-\t\t\t\t\t failure_errno);\n+\t\t\t\t\t failure_errno, NULL);\n \n done:\n \tstrbuf_release(&full_path);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 5bc233b524..3b8e76dab5 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1,4 +1,5 @@\n #include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n #include \"../copy.h\"\n #include \"../environment.h\"\n #include \"../gettext.h\"\n@@ -551,7 +552,7 @@ static int read_ref_internal(struct ref_store *ref_store, const char *refname,\n \tstrbuf_rtrim(&sb_contents);\n \tbuf = sb_contents.buf;\n \n-\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr);\n+\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr, NULL);\n \n out:\n \tif (ret && !myerr)\n@@ -587,7 +588,7 @@ static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refn\n \n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno)\n+\t\t\t int *failure_errno, const char **trailing)\n {\n \tconst char *p;\n \tif (skip_prefix(buf, \"ref:\", &buf)) {\n@@ -609,6 +610,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t*failure_errno = EINVAL;\n \t\treturn -1;\n \t}\n+\n+\tif (trailing)\n+\t\t*trailing = p;\n+\n \treturn 0;\n }\n \n@@ -3435,6 +3440,141 @@ static int files_fsck_refs_name(struct fsck_options *o,\n \treturn ret;\n }\n \n+/*\n+ * Check the symref \"pointee_name\" and \"pointee_path\". The caller should\n+ * make sure that \"pointee_path\" is absolute. For symbolic ref, \"pointee_name\"\n+ * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n+ * be the relative path agaignst \"gitdir\".\n+ */\n+static int files_fsck_symref_target(struct fsck_options *o,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const char *pointee_name,\n+\t\t\t\t const char *pointee_path)\n+{\n+\tconst char *p = NULL;\n+\tstruct stat st;\n+\tint ret = 0;\n+\n+\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n+\n+\t\tret = fsck_refs_report(o, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to target out of refs hierarchy\");\n+\t\tgoto out;\n+\t}\n+\n+\tif (check_refname_format(pointee_name, 0)) {\n+\t\tret = fsck_refs_report(o, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid refname\");\n+\t}\n+\n+\tif (lstat(pointee_path, &st) < 0)\n+\t\tgoto out;\n+\n+\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n+\t\tret = fsck_refs_report(o, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid target\");\n+\t\tgoto out;\n+\t}\n+out:\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs_content(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter)\n+{\n+\tstruct strbuf pointee_path = STRBUF_INIT,\n+\t\t ref_content = STRBUF_INIT,\n+\t\t abs_gitdir = STRBUF_INIT,\n+\t\t referent = STRBUF_INIT,\n+\t\t refname = STRBUF_INIT;\n+\tconst char *trailing = NULL;\n+\tint failure_errno = 0;\n+\tunsigned int type = 0;\n+\tstruct object_id oid;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\n+\t/*\n+\t * If the file is a symlink, we need to only check the connectivity\n+\t * of the destination object.\n+\t */\n+\tif (S_ISLNK(iter->st.st_mode)) {\n+\t\tconst char *pointee_name = NULL;\n+\n+\t\tstrbuf_add_real_path(&pointee_path, iter->path.buf);\n+\n+\t\tstrbuf_add_absolute_path(&abs_gitdir, gitdir);\n+\t\tstrbuf_normalize_path(&abs_gitdir);\n+\t\tif (!is_dir_sep(abs_gitdir.buf[abs_gitdir.len - 1]))\n+\t\t\tstrbuf_addch(&abs_gitdir, '/');\n+\n+\t\tif (!skip_prefix(pointee_path.buf,\n+\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n+\t\t\tret = fsck_refs_report(o, refname.buf,\n+\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t\t \"point to target outside gitdir\");\n+\t\t\tgoto clean;\n+\t\t}\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, pointee_name,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n+\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n+\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n+\t\t\t\t &referent, &type,\n+\t\t\t\t &failure_errno, &trailing)) {\n+\t\tret = fsck_refs_report(o, refname.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n+\t\t\t\t \"invalid ref content\");\n+\t\tgoto clean;\n+\t}\n+\n+\t/*\n+\t * If the ref is a symref, we need to check the destination name and\n+\t * connectivity.\n+\t */\n+\tif (referent.len && (type & REF_ISSYMREF)) {\n+\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n+\t\tstrbuf_rtrim(&referent);\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, referent.buf,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t} else {\n+\t\t/*\n+\t\t * Only regular refs could have a trailing garbage. Should\n+\t\t * be reported as a warning.\n+\t\t */\n+\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n+\t\t\tret = fsck_refs_report(o, refname.buf,\n+\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n+\t\t\t\t\t \"trailing garbage in ref\");\n+\t\t\tgoto clean;\n+\t\t}\n+\t}\n+\n+clean:\n+\tstrbuf_release(&abs_gitdir);\n+\tstrbuf_release(&pointee_path);\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&ref_content);\n+\tstrbuf_release(&referent);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3487,6 +3627,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n \t\tfiles_fsck_refs_name,\n+\t\tfiles_fsck_refs_content,\n \t\tNULL\n \t};\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 280acb7f9e..1126c6102a 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -709,11 +709,12 @@ struct ref_store {\n \n /*\n * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for\n- * invalid contents.\n+ * invalid contents. Also *trailing is set to the first character after the\n+ * refname or NULL if the referent is not empty.\n */\n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno);\n+\t\t\t int *failure_errno, const char **trailing);\n \n /*\n * Fill in the generic part of refs and add it to our collection of\ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nindex b2db58d2c6..35bf40ee64 100755\n--- a/t/t0602-reffiles-fsck.sh\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -98,4 +98,114 @@ test_expect_success 'ref name check should be adapted into fsck messages' '\n \t)\n '\n \n+test_expect_success 'regular ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit checkout -b a/b/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse branch-1)\" > $branch_dir_prefix/branch-1-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse tag-1)\" > $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_must_fail git -c fsck.trailingRefContent=error fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s \" \"$(git rev-parse tag-2)\" > $tag_dir_prefix/tag-2-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/tags/tag-2-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/a/b/branch-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'symbolic ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid refname\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid target\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: logs/maint-v2.45\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to target out of refs hierarchy\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.2\n\n"},{"id":"497415","messageId":"xmqqmsnfwmqy.fsf@gitster.g","threadId":"61565","inReplyTo":"ZnKLja6bSNLukkTR@ArchLinux","subject":"Re: [GSoC][PATCH v4 1/7] fsck: add refs check interfaces to interact with fsck error levels","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-06-20T17:24:37Z","receivedAt":"2024-06-20T17:24:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"shejialuo writes:\n\n> The git-fsck(1) focuses on object database consistency check. It relies\n> on the \"fsck_options\" to interact with fsck error levels. However,\n> \"fsck_options\" aims at checking the object database which contains a lot\n> of fields only related to object database.\n>\n> In order to add ref operations, create a new struct named\n> \"fsck_refs_options\" and a new struct named \"fsck_objs_options\". Remove\n> object-related fields from \"fsck_options\" to \"fsck_objs_options\". Change\n> the \"fsck_options\" with three parts of members:\n>\n> 1. The \"fsck_refs_options\".\n> 2. The \"fsck_objs_options\".\n> 3. The common settings both for refs and objects. Because we leave\n> common settings in \"fsck_options\". The setup process could be fully\n> reused without any code changing.\n>\n> Also add related macros to align with the current code. Because we\n> remove some fields from \"fsck_options\" to \"fsck_objs_options\". Change\n> the influenced code to use the \"fsck_options.objs_options\" instead of\n> using \"fsck_options\" itself.\n>\n> The static function \"report\" provided by \"fsck.c\" aims at reporting the\n> problems related to object database which cannot be reused for refs.\n> Provide \"fsck_refs_report\" function to integrate the fsck error levels\n> into reference consistency check.\n>\n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n\nThis seems to be doing too many things at once, making the result a\nlot harder to review than necessary. At this point, nobody checks\nrefs and reports problems with refs, so fsck_refs_report() has no\ncallers and it is impossible to tell if the function signature of\nit, iow, the set of parameters it receives, is sufficient, for\nexample.\n\nStepping back a bit, it is true that (1) all existing checks are\nabout \"objects\", and (2) all checks we want to implement around\n\"objects\" and \"refs\" can be split cleanly into these two categories?\n\nI am wondering if there are checks and reports that would benefit\nfrom having access to both objects and refs (e.g. when checking a\nref, you may want to see both what the name of the ref is and what\nobject the ref points at), in which case, being forced to implement\nsuch a check-and-report as \"object\" or \"ref\" that has access to only\ndifferent subset of information may turn out to be too limiting.\n\nYes, I am OK with having substructure in fsck_options, but I am\ndoubting if it is a good idea to have a separate fsck_refs_report()\nthat can only take \"name\" that is different from fsck.c::report().\n\nFor example, how would we ensure that refs/heads/foo is allowed to\npoint at a commit object and nothing else, and how would we report a\nviolation when we find that ref/heads/foo is pointing at a tag,\ni.e., \"refs/heads/foo points at\nf665776185ad074b236c00751d666da7d1977dbe which is a tag\". The\nfsck_refs_report() function is not equipped to do that; neither is\n.refs_options.error_func() that only takes \"name\".\n\n> +int fsck_refs_report(struct fsck_options *o,\n> +\t\t const char *name,\n> +\t\t enum fsck_msg_id msg_id,\n> +\t\t const char *fmt, ...)\n> ...\n> +\tva_start(ap, fmt);\n> +\tstrbuf_vaddf(&sb, fmt, ap);\n> +\tret = o->refs_options.error_func(o, name, msg_type, msg_id, sb.buf);\n> +\tstrbuf_release(&sb);\n> +\tva_end(ap);\n\nPerhaps the code and data structure of the entire series may be\ncapable of supporting such a check-and-report, but the primary point\nI am making is that among what [1/7] adds, we cannot sanely judge if\nthese \"refs\" related additions are sensible by looking at [1/7].\n\nThanks.\n"},{"id":"497446","messageId":"ZnTkatbhit7vAwwO@ArchLinux","threadId":"61565","inReplyTo":"xmqqmsnfwmqy.fsf@gitster.g","subject":"Re: [GSoC][PATCH v4 1/7] fsck: add refs check interfaces to interact with fsck error levels","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-21T02:24:42Z","receivedAt":"2024-06-21T02:24:45Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Thu, Jun 20, 2024 at 10:24:37AM -0700, Junio C Hamano wrote:\n> shejialuo writes:\n\n[snip]\n\n> This seems to be doing too many things at once, making the result a\n> lot harder to review than necessary. At this point, nobody checks\n> refs and reports problems with refs, so fsck_refs_report() has no\n> callers and it is impossible to tell if the function signature of\n> it, iow, the set of parameters it receives, is sufficient, for\n> example.\n> \n\nI will split this commit into multiple commits in the next version.\n\n> Stepping back a bit, it is true that (1) all existing checks are\n> about \"objects\", and (2) all checks we want to implement around\n> \"objects\" and \"refs\" can be split cleanly into these two categories?\n> \n\nI am sure that \"objects\" and \"refs\" checks cannot be split cleanly into\ntwo categories. Let me elaborate more about this. The git-fsck(1) has\nalready implicitly checked refs. When executing git-fsck(1) with no\narguments, it will call \"builtin/fsck.c::get_default_heads()\". It will\ntraverse every raw ref and use \"fsck_handle_ref\" to check the refs.\n\n\tstruct obj *obj;\n\tobj = parse_object(the_repository, oid);\n\tif (!obj) {\n\t\t...\n\t\terrors_found |= ERROR_REACHABLE;\n\t\treturn 0;\n\t}\n\n\tif (obj->type != OBJ_COMMIT && is_branch(refname)) {\n\t\t...\n\t\terrors_found |= ERROR_REFS;\n\t}\n\tobj->flags |= USED;\n\tmark_object_reachable(obj);\n\treturn 0;\n\ngit-fsck(1) firstly scans the object directory to check every loose\nobject. If everything is OK, it will set \"obj->flags\" like the\nfollowing:\n\n\tobj->flags &= ~(REACHABLE | SEEN);\n\tobj->flags |= HAS_OBJ;\n\nWhen an object is marked with \"UNREACHABLE\" and \"UNUSED\", it will be\ndangling.\n\nWhen checking objects, git-fsck(1) still checks the refs. But from my\nunderstanding, the main motivation here is mark the \"USED\" flag\nbecause we want to tell whether an object is dangling and BTY we check\nrefs. So, we cannot really classify into these two categories. But we can\nclassify into these two categories from action view. When checking refs,\nwe look at the refs options; when checking objects, we look at the\nobjects options. Although we may implicitly check some other things,\nthis should be OK.\n\nSo, I think \"fsck_options\" is only about options. Because the nature of\nthe connectivity between refs and commit objects, we cannot make the\nboundary so clear.\n\n> I am wondering if there are checks and reports that would benefit\n> from having access to both objects and refs (e.g. when checking a\n> ref, you may want to see both what the name of the ref is and what\n> object the ref points at), in which case, being forced to implement\n> such a check-and-report as \"object\" or \"ref\" that has access to only\n> different subset of information may turn out to be too limiting.\n> \n\nShould \"check-and-report\" does this? I am doubt. There are some cases we\ndo look at both object and ref. But \"report\" should not do this. The\nflow git-fsck(1) does is like the following:\n\n\tif (some_thing_bad) {\n\t\tret = report(..., message);\n\t}\n\nHere, \"check\" will provide dedicated message to report. So If we really\nboth look at both object and ref, the caller should be responsible for\nthat. Like the message \"refs/heads/foo points at\nf665776185ad074b236c00751d666da7d1977dbe which is a tag\". The caller\nshould produce such message. It will call some APIs to get the\ninformation and pass it to the \"report\" function.\n\n> Yes, I am OK with having substructure in fsck_options, but I am\n> doubting if it is a good idea to have a separate fsck_refs_report()\n> that can only take \"name\" that is different from fsck.c::report().\n> \n> For example, how would we ensure that refs/heads/foo is allowed to\n> point at a commit object and nothing else, and how would we report a\n> violation when we find that ref/heads/foo is pointing at a tag,\n> i.e., \"refs/heads/foo points at\n> f665776185ad074b236c00751d666da7d1977dbe which is a tag\". The\n> fsck_refs_report() function is not equipped to do that; neither is\n> .refs_options.error_func() that only takes \"name\".\n> \n\nYes, I am not satisfied with current design either. Actually, there are\nmany redundant code in new \"fsck_refs_report\" function. Let's look at\n\"fsck.c::report()\" closely to see what it does.\n\n\tstatic int report(struct fsck_options *options,\n\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n\t\t\t\tenum fsck_msg_id msg_id, const char *fmt, ...)\n\t{\n\t\tva_list ap;\n\t\tstruct strbuf sb = STRBUF_INIT;\n\t\tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n\t\tint result;\n\n\t\tif (msg_type == FSCK_IGNORE)\n\t\t\treturn 0;\n\n\t\tif (object_on_skiplist(options, oid))\n\t\t\treturn 0;\n\n\t\tif (msg_type == FSCK_FATAL)\n\t\t\tmsg_type = FSCK_ERROR;\n\t\telse if (msg_type == FSCK_INFO)\n\t\t\tmsg_type = FSCK_WARN;\n\n\t\tprepare_msg_ids();\n\t\tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n\n\t\tva_start(ap, fmt);\n\t\tstrbuf_vaddf(&sb, fmt, ap);\n\t\tresult = options->objs_options.error_func(options, oid, object_type,\n\t\t\t\t\t\t\t\tmsg_type, msg_id, sb.buf);\n\t\tstrbuf_release(&sb);\n\t\tva_end(ap);\n\n\t\treturn result;\n\t}\n\nIt will get the current message \"msg_type\" which would be configured and\nit will convert the \"FSCK_FATAL\" to \"FSCK_ERROR\", and \"FSCK_INFO\" to\n\"FSCK_WARN\". From above code, we can easily know what would the error\n(warn) message be the following:\n\n\t'object: fsck_describe_object(o, oid): FSCK_MESSAGE: message'.\n\nThe \"fsck_describe_object\" aims to provide meaningful message about the\noid. And \"FSCK_MESSAGE\" is \"msg_id_info[msg_id]\".The last field \"message\"\nis the caller-provided message.\n\nSo the hardest part to reuse this function is that it receives the\nparameters related to object such as `oid` and `object_type`. Actually,\nthe \"fsck_refs_report\" could generate such information \"refs/heads/foo\npoints at f665776185ad074b236c00751d666da7d1977dbe which is a tag\" like\nthe following (I use string literal for simplicity here):\n\n\tif (some_thing_bad) {\n\t\t...\n\t\tret = fsck_refs_report(o, \"refs/heads/foo\", FSCK_BAD_REF_CONTENT,\n\t\t\t\"points at f665776185ad074b236c00751d666da7d1977dbe which is a tag\")\n\t}\n\nIt will generate the following output in the current design:\n\n\terror: refs/heads/foo: badRefContent: points at\n\tf665776185ad074b236c00751d666da7d1977dbe which is a tag\n\nSo my idea is that we just reuse the \"report\" function here. The bad\npart for current \"report\" function is that it is highly coupled with the\nobject check. However, my idea is just like the \"fsck_refs_report\".\n\n\t: : \n\nThus, we may unify the report interface both for refs and objects. And\nwe don't define two differtn interface:\n\n\t1. \"fsck_refs_error\"\n\t2. \"fsck_objs_error\"\n\nAlso, we just need to use \"fsck_refs_report\" and \"fsck_objs_report\" to\nwrap the \"report\" function here to avoid redundance.\n\n> > +int fsck_refs_report(struct fsck_options *o,\n> > +\t\t const char *name,\n> > +\t\t enum fsck_msg_id msg_id,\n> > +\t\t const char *fmt, ...)\n> > ...\n> > +\tva_start(ap, fmt);\n> > +\tstrbuf_vaddf(&sb, fmt, ap);\n> > +\tret = o->refs_options.error_func(o, name, msg_type, msg_id, sb.buf);\n> > +\tstrbuf_release(&sb);\n> > +\tva_end(ap);\n> \n> Perhaps the code and data structure of the entire series may be\n> capable of supporting such a check-and-report, but the primary point\n> I am making is that among what [1/7] adds, we cannot sanely judge if\n> these \"refs\" related additions are sensible by looking at [1/7].\n> \n\nFrom my above reply, I guess the current design gives the caller\nextensibility to provide customize message. All we do is make code more\nclean without redundance.\n\n> Thanks.\n\nThanks,\nJialuo\n"},{"id":"497744","messageId":"Zn2Ah3WDhtOmzrzn@ArchLinux","threadId":"61565","inReplyTo":"ZnKKy52QFO2UhqM6@ArchLinux","subject":"[GSoC][PATCH v5 00/12] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-27T15:08:55Z","receivedAt":"2024-06-27T15:09:00Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Hi All, this version have been changed a lot compared with the v4. The\nmainly changed part is how to integrate the ref checks into the current\ngit-fsck(1) which concentrates on the object database check.\n\nIn the previous version, I design the following structures:\n\n struct fsck_options {\n struct fsck_refs_options refs_options;\n struct fsck_objs_options objs_options;\n enum fsck_msg_type *msg_type;\n unsigned strict:1,\n verbose:1;\n };\n\nAlthough Junio said It's OK that therea are sub-structures in\n\"fsck_options\", I decide not to use the above design. This is because in\nthe current implementation, we either check refs or objects. The above\ndesign is a combination. It will give us the feeling that \"fsck_options\"\nis used to check both refs and objects.\n\nSo I decide to create two new structs \"fsck_objects_options\" and\n\"fsck_refs_options\". So we need to think about how to design this. In my\nimplementation, I totally change the \"fsck_options\" to be a general\noptions which \"fsck_objects_options\" and \"fsck_refs_options\" will\nincorporate.\n\nHowever, there are some options related to configs such as\n\"enum fsck_msg_type *msg_type\" and \"skiplist\". I change them into the\n\"fskc.c\" as the static variable for singleton pattern.\n\nAnd then extract the common options into \"fsck_options\". Thus we could\nreuse some interfaces when chekcing refs or objects.\n\nAt last, solve the review from Junio. Junio hopes that we could use a\nunified interface to report fsck error messages. This version solves\nthis.\n\nBTY, this patch is based on the commit:\n\n 1e1586e4ed (The sixteenth batch, 2024-06-24)\n\nThanks all\n\nshejialuo (12):\n fsck: rename \"fsck_options\" to \"fsck_objects_options\"\n fsck: use \"fsck_configs\" to set up configs\n fsck: abstract common options for reusing\n fsck: add \"fsck_refs_options\" struct\n fsck: add a unified interface for reporting fsck messages\n fsck: add \"fsck_refs_options\" initialization macros\n refs: set up ref consistency check infrastructure\n builtin/refs: add verify subcommand\n builtin/fsck: add `git-refs verify` child process\n files-backend: add unified interface for refs scanning\n fsck: add ref name check for files backend\n fsck: add ref content check for files backend\n\n Documentation/fsck-msgids.txt | 12 ++\n Documentation/git-refs.txt | 13 ++\n builtin/fsck.c | 38 +++--\n builtin/index-pack.c | 18 +--\n builtin/mktag.c | 15 +-\n builtin/refs.c | 36 +++++\n builtin/unpack-objects.c | 14 +-\n fetch-pack.c | 12 +-\n fsck.c | 197 +++++++++++++++++++-------\n fsck.h | 122 ++++++++++++----\n object-file.c | 20 +--\n refs.c | 7 +-\n refs.h | 8 ++\n refs/debug.c | 11 ++\n refs/files-backend.c | 255 +++++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 ++\n refs/refs-internal.h | 11 +-\n refs/reftable-backend.c | 8 ++\n t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++\n 19 files changed, 889 insertions(+), 127 deletions(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\nRange-diff against v4:\n 1: 2d2c1f2cd3 < -: ---------- fsck: add refs check interfaces to interact with fsck error levels\n -: ---------- > 1: d83b5797cb fsck: rename \"fsck_options\" to \"fsck_objects_options\"\n -: ---------- > 2: 88000b092a fsck: use \"fsck_configs\" to set up configs\n -: ---------- > 3: 65093bae64 fsck: abstract common options for reusing\n -: ---------- > 4: 403750da30 fsck: add \"fsck_refs_options\" struct\n -: ---------- > 5: 8dea5654d5 fsck: add a unified interface for reporting fsck messages\n -: ---------- > 6: 8fb62b54c6 fsck: add \"fsck_refs_options\" initialization macros\n 2: 4bacffe44e ! 7: 2ecd1f5407 refs: set up ref consistency check infrastructure\n @@ refs.c: int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n -+int refs_fsck(struct ref_store *refs, struct fsck_options *o)\n ++int refs_fsck(struct ref_store *refs, struct fsck_refs_options *o)\n +{\n +\treturn refs->be->fsck(refs, o);\n +}\n @@ refs.h\n \n #include \"commit.h\"\n \n -+struct fsck_options;\n ++struct fsck_refs_options;\n struct object_id;\n struct ref_store;\n struct repository;\n @@ refs.h: int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void\n + * reflogs are consistent, and non-zero otherwise. The errors will be\n + * written to stderr.\n + */\n -+int refs_fsck(struct ref_store *refs, struct fsck_options *o);\n ++int refs_fsck(struct ref_store *refs, struct fsck_refs_options *o);\n +\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n @@ refs/debug.c: static int debug_reflog_expire(struct ref_store *ref_store, const\n }\n \n +static int debug_fsck(struct ref_store *ref_store,\n -+\t\t struct fsck_options *o)\n ++\t\t struct fsck_refs_options *o)\n +{\n +\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n +\tint res = drefs->refs->be->fsck(drefs->refs, o);\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n }\n \n +static int files_fsck(struct ref_store *ref_store,\n -+\t\t struct fsck_options *o)\n ++\t\t struct fsck_refs_options *o)\n +{\n +\tint ret;\n +\tstruct files_ref_store *refs =\n @@ refs/packed-backend.c: static struct ref_iterator *packed_reflog_iterator_begin(\n }\n \n +static int packed_fsck(struct ref_store *ref_store,\n -+\t\t struct fsck_options *o)\n ++\t\t struct fsck_refs_options *o)\n +{\n +\treturn 0;\n +}\n @@ refs/refs-internal.h: typedef int read_raw_ref_fn(struct ref_store *ref_store, c\n \t\t\t\t struct strbuf *referent);\n \n +typedef int fsck_fn(struct ref_store *ref_store,\n -+\t\t struct fsck_options *o);\n ++\t\t struct fsck_refs_options *o);\n +\n struct ref_storage_be {\n \tconst char *name;\n @@ refs/reftable-backend.c: static int reftable_be_reflog_expire(struct ref_store *\n }\n \n +static int reftable_be_fsck(struct ref_store *ref_store,\n -+\t\t\t struct fsck_options *o)\n ++\t\t\t struct fsck_refs_options *o)\n +{\n +\treturn 0;\n +}\n 3: f00acab4d1 ! 8: 534a3d00af builtin/refs: add verify subcommand\n @@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c\n \n +static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n +{\n ++\tstruct fsck_refs_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n +\tconst char * const verify_usage[] = {\n +\t\tREFS_VERIFY_USAGE,\n +\t\tNULL,\n +\t};\n -+\tint ret = 0;\n +\tunsigned int verbose = 0, strict = 0;\n -+\tstruct fsck_options fsck_options = FSCK_OPTIONS_DEFAULT;\n +\tstruct option options[] = {\n +\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n +\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n @@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c\n +\t\tusage(_(\"too many arguments\"));\n +\n +\tif (verbose)\n -+\t\tfsck_options.verbose = 1;\n ++\t\tfsck_refs_options.fsck_options.verbose = 1;\n +\tif (strict)\n -+\t\tfsck_options.strict = 1;\n ++\t\tfsck_refs_options.fsck_options.strict = 1;\n +\n -+\tgit_config(git_fsck_config, &fsck_options);\n ++\tgit_config(git_fsck_config, &fsck_refs_options.fsck_options);\n +\tprepare_repo_settings(the_repository);\n +\n -+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_options);\n -+\n -+\t/*\n -+\t * Explicitly free the allocated array and object skiplist set. Because\n -+\t * we reuse `git_fsck_config` here. It will still set the skiplist.\n -+\t */\n -+\tfree(fsck_options.msg_type);\n -+\toidset_clear(&fsck_options.objs_options.skiplist);\n -+\treturn ret;\n ++\treturn refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n +}\n +\n int cmd_refs(int argc, const char **argv, const char *prefix)\n 4: 6494f5b0c2 = 9: 054c8cea89 builtin/fsck: add `git-refs verify` child process\n 5: 2c064f4f71 ! 10: 7f87c6a26e files-backend: add unified interface for refs scanning\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n + * the whole directory. This function is used as the callback for each\n + * regular file or symlink in the directory.\n + */\n -+typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n ++typedef int (*files_fsck_refs_fn)(struct fsck_refs_options *o,\n +\t\t\t\t const char *gitdir,\n +\t\t\t\t const char *refs_check_dir,\n +\t\t\t\t struct dir_iterator *iter);\n +\n +static int files_fsck_refs_dir(struct ref_store *ref_store,\n -+\t\t\t struct fsck_options *o,\n ++\t\t\t struct fsck_refs_options *o,\n +\t\t\t const char *refs_check_dir,\n +\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n +{\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +\t\t\tcontinue;\n +\t\t} else if (S_ISREG(iter->st.st_mode) ||\n +\t\t\t S_ISLNK(iter->st.st_mode)) {\n -+\t\t\tif (o->verbose)\n ++\t\t\tif (o->fsck_options.verbose)\n +\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n +\t\t\t\t\t refs_check_dir, iter->relative_path);\n +\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +}\n +\n +static int files_fsck_refs(struct ref_store *ref_store,\n -+\t\t\t struct fsck_options *o)\n ++\t\t\t struct fsck_refs_options *o)\n +{\n +\tint ret;\n +\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n +\t\tNULL\n +\t};\n +\n -+\tif (o->verbose)\n ++\tif (o->fsck_options.verbose)\n +\t\tfprintf_ln(stderr, \"Checking references consistency\");\n +\n +\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +}\n +\n static int files_fsck(struct ref_store *ref_store,\n - \t\t struct fsck_options *o)\n + \t\t struct fsck_refs_options *o)\n {\n @@ refs/files-backend.c: static int files_fsck(struct ref_store *ref_store,\n \tstruct files_ref_store *refs =\n 6: 88b54b9e1a ! 11: 7d78014e5f fsck: add ref name check for files backend\n @@ fsck.h: enum fsck_msg_type {\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n \n ## refs/files-backend.c ##\n -@@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n +@@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_refs_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n -+static int files_fsck_refs_name(struct fsck_options *o,\n ++static int files_fsck_refs_name(struct fsck_refs_options *o,\n +\t\t\t\tconst char *gitdir UNUSED,\n +\t\t\t\tconst char *refs_check_dir,\n +\t\t\t\tstruct dir_iterator *iter)\n @@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n +\n +\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n +\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n -+\t\tret = fsck_refs_report(o, sb.buf,\n ++\t\tret = fsck_refs_report(o, NULL, sb.buf,\n +\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n +\t\t\t\t \"invalid refname format\");\n +\t}\n @@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n +}\n +\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n - \t\t\t struct fsck_options *o,\n + \t\t\t struct fsck_refs_options *o,\n \t\t\t const char *refs_check_dir,\n @@ refs/files-backend.c: static int files_fsck_refs(struct ref_store *ref_store,\n {\n 7: 8309a4746a ! 12: 16209a73ed fsck: add ref content check for files backend\n @@ refs/files-backend.c: int parse_loose_ref_contents(const char *buf, struct objec\n \treturn 0;\n }\n \n -@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_refs_options *o,\n \treturn ret;\n }\n \n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n + * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n + * be the relative path agaignst \"gitdir\".\n + */\n -+static int files_fsck_symref_target(struct fsck_options *o,\n ++static int files_fsck_symref_target(struct fsck_refs_options *o,\n +\t\t\t\t const char *refname,\n +\t\t\t\t const char *pointee_name,\n +\t\t\t\t const char *pointee_path)\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\n +\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n +\n -+\t\tret = fsck_refs_report(o, refname,\n ++\t\tret = fsck_refs_report(o, NULL, refname,\n +\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n +\t\t\t\t \"point to target out of refs hierarchy\");\n +\t\tgoto out;\n +\t}\n +\n +\tif (check_refname_format(pointee_name, 0)) {\n -+\t\tret = fsck_refs_report(o, refname,\n ++\t\tret = fsck_refs_report(o, NULL, refname,\n +\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n +\t\t\t\t \"point to invalid refname\");\n +\t}\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\t\tgoto out;\n +\n +\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n -+\t\tret = fsck_refs_report(o, refname,\n ++\t\tret = fsck_refs_report(o, NULL, refname,\n +\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n +\t\t\t\t \"point to invalid target\");\n +\t\tgoto out;\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\treturn ret;\n +}\n +\n -+static int files_fsck_refs_content(struct fsck_options *o,\n ++static int files_fsck_refs_content(struct fsck_refs_options *o,\n +\t\t\t\t const char *gitdir,\n +\t\t\t\t const char *refs_check_dir,\n +\t\t\t\t struct dir_iterator *iter)\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\n +\t\tif (!skip_prefix(pointee_path.buf,\n +\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n -+\t\t\tret = fsck_refs_report(o, refname.buf,\n ++\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n +\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n +\t\t\t\t\t \"point to target outside gitdir\");\n +\t\t\tgoto clean;\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n +\t\t\t\t &referent, &type,\n +\t\t\t\t &failure_errno, &trailing)) {\n -+\t\tret = fsck_refs_report(o, refname.buf,\n ++\t\tret = fsck_refs_report(o, NULL, refname.buf,\n +\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n +\t\t\t\t \"invalid ref content\");\n +\t\tgoto clean;\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\t\t * be reported as a warning.\n +\t\t */\n +\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n -+\t\t\tret = fsck_refs_report(o, refname.buf,\n ++\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n +\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n +\t\t\t\t\t \"trailing garbage in ref\");\n +\t\t\tgoto clean;\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +}\n +\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n - \t\t\t struct fsck_options *o,\n + \t\t\t struct fsck_refs_options *o,\n \t\t\t const char *refs_check_dir,\n @@ refs/files-backend.c: static int files_fsck_refs(struct ref_store *ref_store,\n \tint ret;\n-- \n2.45.2\n\n"},{"id":"497746","messageId":"Zn2BXRatUdX16PD3@ArchLinux","threadId":"61565","inReplyTo":"Zn2Ah3WDhtOmzrzn@ArchLinux","subject":"[GSoC][PATCH v5 01/12] fsck: rename \"fsck_options\" to \"fsck_objects_options\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-27T15:12:29Z","receivedAt":"2024-06-27T15:12:34Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) focuses on object database consistency check. It relies\non the \"fsck_options\" to interact with fsck error levels. However,\n\"fsck_options\" aims at checking the object database which contains a lot\nof fields only related to object database.\n\nIn order to provide git-fsck(1) with refs consistency check. Rename\n\"fsck_options\" to \"fsck_objects_options\" to explicitly indicate this\nstructure is used to check the object database.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 13 ++++----\n builtin/index-pack.c | 18 +++++------\n builtin/mktag.c | 13 ++++----\n builtin/unpack-objects.c | 14 ++++-----\n fetch-pack.c | 12 ++++----\n fsck.c | 64 ++++++++++++++++++++++------------------\n fsck.h | 43 ++++++++++++++-------------\n object-file.c | 4 +--\n 8 files changed, 95 insertions(+), 86 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..ec3220880d 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -42,8 +42,8 @@ static int check_full = 1;\n static int connectivity_only;\n static int check_strict;\n static int keep_cache_objects;\n-static struct fsck_options fsck_walk_options = FSCK_OPTIONS_DEFAULT;\n-static struct fsck_options fsck_obj_options = FSCK_OPTIONS_DEFAULT;\n+static struct fsck_objects_options fsck_walk_options = FSCK_OBJECTS_OPTIONS_DEFAULT;\n+static struct fsck_objects_options fsck_obj_options = FSCK_OBJECTS_OPTIONS_DEFAULT;\n static int errors_found;\n static int write_lost_and_found;\n static int verbose;\n@@ -89,7 +89,7 @@ static int objerror(struct object *obj, const char *err)\n \treturn -1;\n }\n \n-static int fsck_error_func(struct fsck_options *o UNUSED,\n+static int fsck_error_func(struct fsck_objects_options *o UNUSED,\n \t\t\t const struct object_id *oid,\n \t\t\t enum object_type object_type,\n \t\t\t enum fsck_msg_type msg_type,\n@@ -118,7 +118,7 @@ static int fsck_error_func(struct fsck_options *o UNUSED,\n static struct object_array pending;\n \n static int mark_object(struct object *obj, enum object_type type,\n-\t\t void *data, struct fsck_options *options UNUSED)\n+\t\t void *data, struct fsck_objects_options *options UNUSED)\n {\n \tstruct object *parent = data;\n \n@@ -204,7 +204,8 @@ static int traverse_reachable(void)\n }\n \n static int mark_used(struct object *obj, enum object_type type UNUSED,\n-\t\t void *data UNUSED, struct fsck_options *options UNUSED)\n+\t\t void *data UNUSED,\n+\t\t struct fsck_objects_options *options UNUSED)\n {\n \tif (!obj)\n \t\treturn 1;\n@@ -214,7 +215,7 @@ static int mark_used(struct object *obj, enum object_type type UNUSED,\n \n static void mark_unreachable_referents(const struct object_id *oid)\n {\n-\tstruct fsck_options options = FSCK_OPTIONS_DEFAULT;\n+\tstruct fsck_objects_options options = FSCK_OBJECTS_OPTIONS_DEFAULT;\n \tstruct object *obj = lookup_object(the_repository, oid);\n \n \tif (!obj || !(obj->flags & HAS_OBJ))\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 856428fef9..08ebeedfd3 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -127,7 +127,7 @@ static int nr_threads;\n static int from_stdin;\n static int strict;\n static int do_fsck_object;\n-static struct fsck_options fsck_options = FSCK_OPTIONS_MISSING_GITMODULES;\n+static struct fsck_objects_options fsck_objects_options = FSCK_OBJECTS_OPTIONS_MISSING_GITMODULES;\n static int verbose;\n static const char *progress_title;\n static int show_resolving_progress;\n@@ -220,7 +220,7 @@ static void cleanup_thread(void)\n \n static int mark_link(struct object *obj, enum object_type type,\n \t\t void *data UNUSED,\n-\t\t struct fsck_options *options UNUSED)\n+\t\t struct fsck_objects_options *options UNUSED)\n {\n \tif (!obj)\n \t\treturn -1;\n@@ -852,7 +852,7 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,\n \t\t\telse\n \t\t\t\tdie(_(\"invalid blob object %s\"), oid_to_hex(oid));\n \t\t\tif (do_fsck_object &&\n-\t\t\t fsck_object(&blob->object, (void *)data, size, &fsck_options))\n+\t\t\t fsck_object(&blob->object, (void *)data, size, &fsck_objects_options))\n \t\t\t\tdie(_(\"fsck error in packed object\"));\n \t\t} else {\n \t\t\tstruct object *obj;\n@@ -871,9 +871,9 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,\n \t\t\tif (!obj)\n \t\t\t\tdie(_(\"invalid %s\"), type_name(type));\n \t\t\tif (do_fsck_object &&\n-\t\t\t fsck_object(obj, buf, size, &fsck_options))\n+\t\t\t fsck_object(obj, buf, size, &fsck_objects_options))\n \t\t\t\tdie(_(\"fsck error in packed object\"));\n-\t\t\tif (strict && fsck_walk(obj, NULL, &fsck_options))\n+\t\t\tif (strict && fsck_walk(obj, NULL, &fsck_objects_options))\n \t\t\t\tdie(_(\"Not all child objects of %s are reachable\"), oid_to_hex(&obj->oid));\n \n \t\t\tif (obj->type == OBJ_TREE) {\n@@ -1746,7 +1746,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\tusage(index_pack_usage);\n \n \tdisable_replace_refs();\n-\tfsck_options.walk = mark_link;\n+\tfsck_objects_options.walk = mark_link;\n \n \treset_pack_idx_option(&opts);\n \topts.flags |= WRITE_REV;\n@@ -1770,13 +1770,13 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t} else if (skip_to_optional_arg(arg, \"--strict\", &arg)) {\n \t\t\t\tstrict = 1;\n \t\t\t\tdo_fsck_object = 1;\n-\t\t\t\tfsck_set_msg_types(&fsck_options, arg);\n+\t\t\t\tfsck_set_msg_types(&fsck_objects_options, arg);\n \t\t\t} else if (!strcmp(arg, \"--check-self-contained-and-connected\")) {\n \t\t\t\tstrict = 1;\n \t\t\t\tcheck_self_contained_and_connected = 1;\n \t\t\t} else if (skip_to_optional_arg(arg, \"--fsck-objects\", &arg)) {\n \t\t\t\tdo_fsck_object = 1;\n-\t\t\t\tfsck_set_msg_types(&fsck_options, arg);\n+\t\t\t\tfsck_set_msg_types(&fsck_objects_options, arg);\n \t\t\t} else if (!strcmp(arg, \"--verify\")) {\n \t\t\t\tverify = 1;\n \t\t\t} else if (!strcmp(arg, \"--verify-stat\")) {\n@@ -1943,7 +1943,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \telse\n \t\tclose(input_fd);\n \n-\tif (do_fsck_object && fsck_finish(&fsck_options))\n+\tif (do_fsck_object && fsck_finish(&fsck_objects_options))\n \t\tdie(_(\"fsck error in pack objects\"));\n \n \tfree(opts.anomaly);\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex 4767f1a97e..be2abc71d8 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -15,9 +15,9 @@ static char const * const builtin_mktag_usage[] = {\n };\n static int option_strict = 1;\n \n-static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n+static struct fsck_objects_options fsck_objects_options = FSCK_OBJECTS_OPTIONS_STRICT;\n \n-static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n+static int mktag_fsck_error_func(struct fsck_objects_options *o UNUSED,\n \t\t\t\t const struct object_id *oid UNUSED,\n \t\t\t\t enum object_type object_type UNUSED,\n \t\t\t\t enum fsck_msg_type msg_type,\n@@ -91,12 +91,13 @@ int cmd_mktag(int argc, const char **argv, const char *prefix)\n \tif (strbuf_read(&buf, 0, 0) < 0)\n \t\tdie_errno(_(\"could not read from stdin\"));\n \n-\tfsck_options.error_func = mktag_fsck_error_func;\n-\tfsck_set_msg_type_from_ids(&fsck_options, FSCK_MSG_EXTRA_HEADER_ENTRY,\n+\tfsck_objects_options.error_func = mktag_fsck_error_func;\n+\tfsck_set_msg_type_from_ids(&fsck_objects_options,\n+\t\t\t\t FSCK_MSG_EXTRA_HEADER_ENTRY,\n \t\t\t\t FSCK_WARN);\n \t/* config might set fsck.extraHeaderEntry=* again */\n-\tgit_config(git_fsck_config, &fsck_options);\n-\tif (fsck_tag_standalone(NULL, buf.buf, buf.len, &fsck_options,\n+\tgit_config(git_fsck_config, &fsck_objects_options);\n+\tif (fsck_tag_standalone(NULL, buf.buf, buf.len, &fsck_objects_options,\n \t\t\t\t&tagged_oid, &tagged_type))\n \t\tdie(_(\"tag on stdin did not pass our strict fsck check\"));\n \ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex f1c85a00ae..c59e330db9 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -25,7 +25,7 @@ static unsigned int offset, len;\n static off_t consumed_bytes;\n static off_t max_input_size;\n static git_hash_ctx ctx;\n-static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n+static struct fsck_objects_options fsck_objects_options = FSCK_OBJECTS_OPTIONS_STRICT;\n static struct progress *progress;\n \n /*\n@@ -212,7 +212,7 @@ static void write_cached_object(struct object *obj, struct obj_buffer *obj_buf)\n */\n static int check_object(struct object *obj, enum object_type type,\n \t\t\tvoid *data UNUSED,\n-\t\t\tstruct fsck_options *options UNUSED)\n+\t\t\tstruct fsck_objects_options *options UNUSED)\n {\n \tstruct obj_buffer *obj_buf;\n \n@@ -237,10 +237,10 @@ static int check_object(struct object *obj, enum object_type type,\n \tobj_buf = lookup_object_buffer(obj);\n \tif (!obj_buf)\n \t\tdie(\"Whoops! Cannot find object '%s'\", oid_to_hex(&obj->oid));\n-\tif (fsck_object(obj, obj_buf->buffer, obj_buf->size, &fsck_options))\n+\tif (fsck_object(obj, obj_buf->buffer, obj_buf->size, &fsck_objects_options))\n \t\tdie(\"fsck error in packed object\");\n-\tfsck_options.walk = check_object;\n-\tif (fsck_walk(obj, NULL, &fsck_options))\n+\tfsck_objects_options.walk = check_object;\n+\tif (fsck_walk(obj, NULL, &fsck_objects_options))\n \t\tdie(\"Error on reachable objects of %s\", oid_to_hex(&obj->oid));\n \twrite_cached_object(obj, obj_buf);\n \treturn 0;\n@@ -635,7 +635,7 @@ int cmd_unpack_objects(int argc, const char **argv, const char *prefix UNUSED)\n \t\t\t}\n \t\t\tif (skip_prefix(arg, \"--strict=\", &arg)) {\n \t\t\t\tstrict = 1;\n-\t\t\t\tfsck_set_msg_types(&fsck_options, arg);\n+\t\t\t\tfsck_set_msg_types(&fsck_objects_options, arg);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (starts_with(arg, \"--pack_header=\")) {\n@@ -671,7 +671,7 @@ int cmd_unpack_objects(int argc, const char **argv, const char *prefix UNUSED)\n \tthe_hash_algo->final_oid_fn(&oid, &tmp_ctx);\n \tif (strict) {\n \t\twrite_rest();\n-\t\tif (fsck_finish(&fsck_options))\n+\t\tif (fsck_finish(&fsck_objects_options))\n \t\t\tdie(_(\"fsck error in pack objects\"));\n \t}\n \tif (!hasheq(fill(the_hash_algo->rawsz), oid.hash))\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex eba9e420ea..7d1e4a5087 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -46,7 +46,7 @@ static int server_supports_filtering;\n static int advertise_sid;\n static struct shallow_lock shallow_lock;\n static const char *alternate_shallow_file;\n-static struct fsck_options fsck_options = FSCK_OPTIONS_MISSING_GITMODULES;\n+static struct fsck_objects_options fsck_objects_options = FSCK_OBJECTS_OPTIONS_MISSING_GITMODULES;\n static struct strbuf fsck_msg_types = STRBUF_INIT;\n static struct string_list uri_protocols = STRING_LIST_INIT_DUP;\n \n@@ -1220,9 +1220,9 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t} else\n \t\talternate_shallow_file = NULL;\n \tif (get_pack(args, fd, pack_lockfiles, NULL, sought, nr_sought,\n-\t\t &fsck_options.gitmodules_found))\n+\t\t &fsck_objects_options.gitmodules_found))\n \t\tdie(_(\"git fetch-pack: fetch failed.\"));\n-\tif (fsck_finish(&fsck_options))\n+\tif (fsck_finish(&fsck_objects_options))\n \t\tdie(\"fsck failed\");\n \n all_done:\n@@ -1780,7 +1780,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \n \t\t\tif (get_pack(args, fd, pack_lockfiles,\n \t\t\t\t packfile_uris.nr ? &index_pack_args : NULL,\n-\t\t\t\t sought, nr_sought, &fsck_options.gitmodules_found))\n+\t\t\t\t sought, nr_sought, &fsck_objects_options.gitmodules_found))\n \t\t\t\tdie(_(\"git fetch-pack: fetch failed.\"));\n \t\t\tdo_check_stateless_delimiter(args->stateless_rpc, &reader);\n \n@@ -1823,7 +1823,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \n \t\tpackname[the_hash_algo->hexsz] = '\\0';\n \n-\t\tparse_gitmodules_oids(cmd.out, &fsck_options.gitmodules_found);\n+\t\tparse_gitmodules_oids(cmd.out, &fsck_objects_options.gitmodules_found);\n \n \t\tclose(cmd.out);\n \n@@ -1844,7 +1844,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \tstring_list_clear(&packfile_uris, 0);\n \tstrvec_clear(&index_pack_args);\n \n-\tif (fsck_finish(&fsck_options))\n+\tif (fsck_finish(&fsck_objects_options))\n \t\tdie(\"fsck failed\");\n \n \tif (negotiator)\ndiff --git a/fsck.c b/fsck.c\nindex e193930ae7..c24a0f9fae 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -99,7 +99,7 @@ void list_config_fsck_msg_ids(struct string_list *list, const char *prefix)\n }\n \n static enum fsck_msg_type fsck_msg_type(enum fsck_msg_id msg_id,\n-\tstruct fsck_options *options)\n+\t\t\t\t\tstruct fsck_objects_options *options)\n {\n \tassert(msg_id >= 0 && msg_id < FSCK_MSG_MAX);\n \n@@ -134,7 +134,7 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type)\n \treturn 1;\n }\n \n-void fsck_set_msg_type_from_ids(struct fsck_options *options,\n+void fsck_set_msg_type_from_ids(struct fsck_objects_options *options,\n \t\t\t\tenum fsck_msg_id msg_id,\n \t\t\t\tenum fsck_msg_type msg_type)\n {\n@@ -150,7 +150,7 @@ void fsck_set_msg_type_from_ids(struct fsck_options *options,\n \toptions->msg_type[msg_id] = msg_type;\n }\n \n-void fsck_set_msg_type(struct fsck_options *options,\n+void fsck_set_msg_type(struct fsck_objects_options *options,\n \t\t const char *msg_id_str, const char *msg_type_str)\n {\n \tint msg_id = parse_msg_id(msg_id_str);\n@@ -179,7 +179,7 @@ void fsck_set_msg_type(struct fsck_options *options,\n \tfree(to_free);\n }\n \n-void fsck_set_msg_types(struct fsck_options *options, const char *values)\n+void fsck_set_msg_types(struct fsck_objects_options *options, const char *values)\n {\n \tchar *buf = xstrdup(values), *to_free = buf;\n \tint done = 0;\n@@ -217,14 +217,14 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n \tfree(to_free);\n }\n \n-static int object_on_skiplist(struct fsck_options *opts,\n+static int object_on_skiplist(struct fsck_objects_options *opts,\n \t\t\t const struct object_id *oid)\n {\n \treturn opts && oid && oidset_contains(&opts->skiplist, oid);\n }\n \n __attribute__((format (printf, 5, 6)))\n-static int report(struct fsck_options *options,\n+static int report(struct fsck_objects_options *options,\n \t\t const struct object_id *oid, enum object_type object_type,\n \t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n {\n@@ -257,13 +257,13 @@ static int report(struct fsck_options *options,\n \treturn result;\n }\n \n-void fsck_enable_object_names(struct fsck_options *options)\n+void fsck_enable_object_names(struct fsck_objects_options *options)\n {\n \tif (!options->object_names)\n \t\toptions->object_names = kh_init_oid_map();\n }\n \n-const char *fsck_get_object_name(struct fsck_options *options,\n+const char *fsck_get_object_name(struct fsck_objects_options *options,\n \t\t\t\t const struct object_id *oid)\n {\n \tkhiter_t pos;\n@@ -275,7 +275,7 @@ const char *fsck_get_object_name(struct fsck_options *options,\n \treturn kh_value(options->object_names, pos);\n }\n \n-void fsck_put_object_name(struct fsck_options *options,\n+void fsck_put_object_name(struct fsck_objects_options *options,\n \t\t\t const struct object_id *oid,\n \t\t\t const char *fmt, ...)\n {\n@@ -296,7 +296,7 @@ void fsck_put_object_name(struct fsck_options *options,\n \tva_end(ap);\n }\n \n-const char *fsck_describe_object(struct fsck_options *options,\n+const char *fsck_describe_object(struct fsck_objects_options *options,\n \t\t\t\t const struct object_id *oid)\n {\n \tstatic struct strbuf bufs[] = {\n@@ -316,7 +316,8 @@ const char *fsck_describe_object(struct fsck_options *options,\n \treturn buf->buf;\n }\n \n-static int fsck_walk_tree(struct tree *tree, void *data, struct fsck_options *options)\n+static int fsck_walk_tree(struct tree *tree, void *data,\n+\t\t\t struct fsck_objects_options *options)\n {\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n@@ -364,7 +365,8 @@ static int fsck_walk_tree(struct tree *tree, void *data, struct fsck_options *op\n \treturn res;\n }\n \n-static int fsck_walk_commit(struct commit *commit, void *data, struct fsck_options *options)\n+static int fsck_walk_commit(struct commit *commit, void *data,\n+\t\t\t struct fsck_objects_options *options)\n {\n \tint counter = 0, generation = 0, name_prefix_len = 0;\n \tstruct commit_list *parents;\n@@ -433,7 +435,8 @@ static int fsck_walk_commit(struct commit *commit, void *data, struct fsck_optio\n \treturn res;\n }\n \n-static int fsck_walk_tag(struct tag *tag, void *data, struct fsck_options *options)\n+static int fsck_walk_tag(struct tag *tag, void *data,\n+\t\t\t struct fsck_objects_options *options)\n {\n \tconst char *name = fsck_get_object_name(options, &tag->object.oid);\n \n@@ -444,7 +447,8 @@ static int fsck_walk_tag(struct tag *tag, void *data, struct fsck_options *optio\n \treturn options->walk(tag->tagged, OBJ_ANY, data, options);\n }\n \n-int fsck_walk(struct object *obj, void *data, struct fsck_options *options)\n+int fsck_walk(struct object *obj, void *data,\n+\t struct fsck_objects_options *options)\n {\n \tif (!obj)\n \t\treturn -1;\n@@ -580,7 +584,7 @@ static int verify_ordered(unsigned mode1, const char *name1,\n \n static int fsck_tree(const struct object_id *tree_oid,\n \t\t const char *buffer, unsigned long size,\n-\t\t struct fsck_options *options)\n+\t\t struct fsck_objects_options *options)\n {\n \tint retval = 0;\n \tint has_null_sha1 = 0;\n@@ -793,7 +797,7 @@ static int fsck_tree(const struct object_id *tree_oid,\n */\n static int verify_headers(const void *data, unsigned long size,\n \t\t\t const struct object_id *oid, enum object_type type,\n-\t\t\t struct fsck_options *options)\n+\t\t\t struct fsck_objects_options *options)\n {\n \tconst char *buffer = (const char *)data;\n \tunsigned long i;\n@@ -825,7 +829,7 @@ static int verify_headers(const void *data, unsigned long size,\n \n static int fsck_ident(const char **ident,\n \t\t const struct object_id *oid, enum object_type type,\n-\t\t struct fsck_options *options)\n+\t\t struct fsck_objects_options *options)\n {\n \tconst char *p = *ident;\n \tchar *end;\n@@ -885,7 +889,7 @@ static int fsck_ident(const char **ident,\n \n static int fsck_commit(const struct object_id *oid,\n \t\t const char *buffer, unsigned long size,\n-\t\t struct fsck_options *options)\n+\t\t struct fsck_objects_options *options)\n {\n \tstruct object_id tree_oid, parent_oid;\n \tunsigned author_count;\n@@ -946,7 +950,7 @@ static int fsck_commit(const struct object_id *oid,\n }\n \n static int fsck_tag(const struct object_id *oid, const char *buffer,\n-\t\t unsigned long size, struct fsck_options *options)\n+\t\t unsigned long size, struct fsck_objects_options *options)\n {\n \tstruct object_id tagged_oid;\n \tint tagged_type;\n@@ -955,7 +959,7 @@ static int fsck_tag(const struct object_id *oid, const char *buffer,\n }\n \n int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n-\t\t\tunsigned long size, struct fsck_options *options,\n+\t\t\tunsigned long size, struct fsck_objects_options *options,\n \t\t\tstruct object_id *tagged_oid,\n \t\t\tint *tagged_type)\n {\n@@ -1050,7 +1054,7 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n \n struct fsck_gitmodules_data {\n \tconst struct object_id *oid;\n-\tstruct fsck_options *options;\n+\tstruct fsck_objects_options *options;\n \tint ret;\n };\n \n@@ -1100,7 +1104,8 @@ static int fsck_gitmodules_fn(const char *var, const char *value,\n }\n \n static int fsck_blob(const struct object_id *oid, const char *buf,\n-\t\t unsigned long size, struct fsck_options *options)\n+\t\t unsigned long size,\n+\t\t struct fsck_objects_options *options)\n {\n \tint ret = 0;\n \n@@ -1170,7 +1175,7 @@ static int fsck_blob(const struct object_id *oid, const char *buf,\n }\n \n int fsck_object(struct object *obj, void *data, unsigned long size,\n-\tstruct fsck_options *options)\n+\t\tstruct fsck_objects_options *options)\n {\n \tif (!obj)\n \t\treturn report(options, NULL, OBJ_NONE, FSCK_MSG_BAD_OBJECT_SHA1, \"no valid object to fsck\");\n@@ -1180,7 +1185,7 @@ int fsck_object(struct object *obj, void *data, unsigned long size,\n \n int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\tconst void *data, unsigned long size,\n-\t\tstruct fsck_options *options)\n+\t\tstruct fsck_objects_options *options)\n {\n \tif (type == OBJ_BLOB)\n \t\treturn fsck_blob(oid, data, size, options);\n@@ -1197,7 +1202,7 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n-int fsck_error_function(struct fsck_options *o,\n+int fsck_error_function(struct fsck_objects_options *o,\n \t\t\tconst struct object_id *oid,\n \t\t\tenum object_type object_type UNUSED,\n \t\t\tenum fsck_msg_type msg_type,\n@@ -1214,7 +1219,8 @@ int fsck_error_function(struct fsck_options *o,\n \n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n-\t\t struct fsck_options *options, const char *blob_type)\n+\t\t struct fsck_objects_options *options,\n+\t\t const char *blob_type)\n {\n \tint ret = 0;\n \tstruct oidset_iter iter;\n@@ -1253,7 +1259,7 @@ static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \treturn ret;\n }\n \n-int fsck_finish(struct fsck_options *options)\n+int fsck_finish(struct fsck_objects_options *options)\n {\n \tint ret = 0;\n \n@@ -1270,7 +1276,7 @@ int fsck_finish(struct fsck_options *options)\n int git_fsck_config(const char *var, const char *value,\n \t\t const struct config_context *ctx, void *cb)\n {\n-\tstruct fsck_options *options = cb;\n+\tstruct fsck_objects_options *options = cb;\n \tconst char *msg_id;\n \n \tif (strcmp(var, \"fsck.skiplist\") == 0) {\n@@ -1300,7 +1306,7 @@ int git_fsck_config(const char *var, const char *value,\n * Custom error callbacks that are used in more than one place.\n */\n \n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+int fsck_error_cb_print_missing_gitmodules(struct fsck_objects_options *o,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t enum object_type object_type,\n \t\t\t\t\t enum fsck_msg_type msg_type,\ndiff --git a/fsck.h b/fsck.h\nindex 6085a384f6..b64164db17 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -92,15 +92,15 @@ enum fsck_msg_id {\n };\n #undef MSG_ID\n \n-struct fsck_options;\n+struct fsck_objects_options;\n struct object;\n \n-void fsck_set_msg_type_from_ids(struct fsck_options *options,\n+void fsck_set_msg_type_from_ids(struct fsck_objects_options *options,\n \t\t\t\tenum fsck_msg_id msg_id,\n \t\t\t\tenum fsck_msg_type msg_type);\n-void fsck_set_msg_type(struct fsck_options *options,\n+void fsck_set_msg_type(struct fsck_objects_options *options,\n \t\t const char *msg_id, const char *msg_type);\n-void fsck_set_msg_types(struct fsck_options *options, const char *values);\n+void fsck_set_msg_types(struct fsck_objects_options *options, const char *values);\n int is_valid_msg_type(const char *msg_id, const char *msg_type);\n \n /*\n@@ -112,26 +112,26 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n * >0\terror signaled and do not abort\n */\n typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n-\t\t\t void *data, struct fsck_options *options);\n+\t\t\t void *data, struct fsck_objects_options *options);\n \n /* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n-typedef int (*fsck_error)(struct fsck_options *o,\n+typedef int (*fsck_error)(struct fsck_objects_options *o,\n \t\t\t const struct object_id *oid, enum object_type object_type,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n-int fsck_error_function(struct fsck_options *o,\n+int fsck_error_function(struct fsck_objects_options *o,\n \t\t\tconst struct object_id *oid, enum object_type object_type,\n \t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\tconst char *message);\n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+int fsck_error_cb_print_missing_gitmodules(struct fsck_objects_options *o,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t enum object_type object_type,\n \t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message);\n \n-struct fsck_options {\n+struct fsck_objects_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n \tunsigned strict:1;\n@@ -144,7 +144,7 @@ struct fsck_options {\n \tkh_oid_map_t *object_names;\n };\n \n-#define FSCK_OPTIONS_DEFAULT { \\\n+#define FSCK_OBJECTS_OPTIONS_DEFAULT { \\\n \t.skiplist = OIDSET_INIT, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n@@ -152,7 +152,7 @@ struct fsck_options {\n \t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_error_function \\\n }\n-#define FSCK_OPTIONS_STRICT { \\\n+#define FSCK_OBJECTS_OPTIONS_STRICT { \\\n \t.strict = 1, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n@@ -160,7 +160,7 @@ struct fsck_options {\n \t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_error_function, \\\n }\n-#define FSCK_OPTIONS_MISSING_GITMODULES { \\\n+#define FSCK_OBJECTS_OPTIONS_MISSING_GITMODULES { \\\n \t.strict = 1, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n@@ -176,14 +176,15 @@ struct fsck_options {\n * >0\treturn value of the first signaled error >0 (in the case of no other errors)\n * 0\t\teverything OK\n */\n-int fsck_walk(struct object *obj, void *data, struct fsck_options *options);\n+int fsck_walk(struct object *obj, void *data,\n+\t struct fsck_objects_options *options);\n \n /*\n * Blob objects my pass a NULL data pointer, which indicates they are too large\n * to fit in memory. All other types must pass a real buffer.\n */\n int fsck_object(struct object *obj, void *data, unsigned long size,\n-\tstruct fsck_options *options);\n+\t\tstruct fsck_objects_options *options);\n \n /*\n * Same as fsck_object(), but for when the caller doesn't have an object\n@@ -191,14 +192,14 @@ int fsck_object(struct object *obj, void *data, unsigned long size,\n */\n int fsck_buffer(const struct object_id *oid, enum object_type,\n \t\tconst void *data, unsigned long size,\n-\t\tstruct fsck_options *options);\n+\t\tstruct fsck_objects_options *options);\n \n /*\n * fsck a tag, and pass info about it back to the caller. This is\n * exposed fsck_object() internals for git-mktag(1).\n */\n int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n-\t\t\tunsigned long size, struct fsck_options *options,\n+\t\t\tunsigned long size, struct fsck_objects_options *options,\n \t\t\tstruct object_id *tagged_oid,\n \t\t\tint *tag_type);\n \n@@ -207,7 +208,7 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n * after completing all fsck_object() calls in order to resolve any remaining\n * checks.\n */\n-int fsck_finish(struct fsck_options *options);\n+int fsck_finish(struct fsck_objects_options *options);\n \n /*\n * Subsystem for storing human-readable names for each object.\n@@ -224,14 +225,14 @@ int fsck_finish(struct fsck_options *options);\n * points to a rotating array of static buffers, and may be invalidated by a\n * subsequent call.\n */\n-void fsck_enable_object_names(struct fsck_options *options);\n-const char *fsck_get_object_name(struct fsck_options *options,\n+void fsck_enable_object_names(struct fsck_objects_options *options);\n+const char *fsck_get_object_name(struct fsck_objects_options *options,\n \t\t\t\t const struct object_id *oid);\n __attribute__((format (printf,3,4)))\n-void fsck_put_object_name(struct fsck_options *options,\n+void fsck_put_object_name(struct fsck_objects_options *options,\n \t\t\t const struct object_id *oid,\n \t\t\t const char *fmt, ...);\n-const char *fsck_describe_object(struct fsck_options *options,\n+const char *fsck_describe_object(struct fsck_objects_options *options,\n \t\t\t\t const struct object_id *oid);\n \n struct key_value_info;\ndiff --git a/object-file.c b/object-file.c\nindex d3cf4b8b2e..ec44ac3d82 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2472,7 +2472,7 @@ int repo_has_object_file(struct repository *r,\n * report the minimal fsck error here, and rely on the caller to\n * give more context.\n */\n-static int hash_format_check_report(struct fsck_options *opts UNUSED,\n+static int hash_format_check_report(struct fsck_objects_options *opts UNUSED,\n \t\t\t\t const struct object_id *oid UNUSED,\n \t\t\t\t enum object_type object_type UNUSED,\n \t\t\t\t enum fsck_msg_type msg_type UNUSED,\n@@ -2507,7 +2507,7 @@ static int index_mem(struct index_state *istate,\n \t\t}\n \t}\n \tif (flags & HASH_FORMAT_CHECK) {\n-\t\tstruct fsck_options opts = FSCK_OPTIONS_DEFAULT;\n+\t\tstruct fsck_objects_options opts = FSCK_OBJECTS_OPTIONS_DEFAULT;\n \n \t\topts.strict = 1;\n \t\topts.error_func = hash_format_check_report;\n-- \n2.45.2\n\n"},{"id":"497747","messageId":"Zn2BgEw0geZwvr3_@ArchLinux","threadId":"61565","inReplyTo":"Zn2Ah3WDhtOmzrzn@ArchLinux","subject":"[GSoC][PATCH v5 02/12] fsck: use \"fsck_configs\" to set up configs","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-27T15:13:04Z","receivedAt":"2024-06-27T15:13:08Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Some fields such as \"msg_type\" and \"skiplist\" in \"fsck_objects_options\"\nare not options, these fields are related to \"git-config(1)\" which are\ninitialized using \"git_fsck_config\" function. Create a static variable\nnamed \"fsck_configs\" in \"fsck.c\" which aims at handling configs. Thus we\ndon't need to reply on the \"fsck_objects_options\" to set up the fsck\nerror message severity.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 29 ++++++++++++++++++-----------\n fsck.h | 3 ---\n 2 files changed, 18 insertions(+), 14 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex c24a0f9fae..81b93f02fc 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -24,6 +24,14 @@\n \n static ssize_t max_tree_entry_len = 4096;\n \n+static struct {\n+\tenum fsck_msg_type *msg_type;\n+\tstruct oidset oid_skiplist;\n+} fsck_configs = {\n+\t.msg_type = NULL,\n+\t.oid_skiplist = OIDSET_INIT\n+};\n+\n #define STR(x) #x\n #define MSG_ID(id, msg_type) { STR(id), NULL, NULL, FSCK_##msg_type },\n static struct {\n@@ -103,7 +111,7 @@ static enum fsck_msg_type fsck_msg_type(enum fsck_msg_id msg_id,\n {\n \tassert(msg_id >= 0 && msg_id < FSCK_MSG_MAX);\n \n-\tif (!options->msg_type) {\n+\tif (!fsck_configs.msg_type) {\n \t\tenum fsck_msg_type msg_type = msg_id_info[msg_id].msg_type;\n \n \t\tif (options->strict && msg_type == FSCK_WARN)\n@@ -111,7 +119,7 @@ static enum fsck_msg_type fsck_msg_type(enum fsck_msg_id msg_id,\n \t\treturn msg_type;\n \t}\n \n-\treturn options->msg_type[msg_id];\n+\treturn fsck_configs.msg_type[msg_id];\n }\n \n static enum fsck_msg_type parse_msg_type(const char *str)\n@@ -138,16 +146,16 @@ void fsck_set_msg_type_from_ids(struct fsck_objects_options *options,\n \t\t\t\tenum fsck_msg_id msg_id,\n \t\t\t\tenum fsck_msg_type msg_type)\n {\n-\tif (!options->msg_type) {\n+\tif (!fsck_configs.msg_type) {\n \t\tint i;\n \t\tenum fsck_msg_type *severity;\n \t\tALLOC_ARRAY(severity, FSCK_MSG_MAX);\n \t\tfor (i = 0; i < FSCK_MSG_MAX; i++)\n \t\t\tseverity[i] = fsck_msg_type(i, options);\n-\t\toptions->msg_type = severity;\n+\t\tfsck_configs.msg_type = severity;\n \t}\n \n-\toptions->msg_type[msg_id] = msg_type;\n+\tfsck_configs.msg_type[msg_id] = msg_type;\n }\n \n void fsck_set_msg_type(struct fsck_objects_options *options,\n@@ -203,7 +211,7 @@ void fsck_set_msg_types(struct fsck_objects_options *options, const char *values\n \t\tif (!strcmp(buf, \"skiplist\")) {\n \t\t\tif (equal == len)\n \t\t\t\tdie(\"skiplist requires a path\");\n-\t\t\toidset_parse_file(&options->skiplist, buf + equal + 1);\n+\t\t\toidset_parse_file(&fsck_configs.oid_skiplist, buf + equal + 1);\n \t\t\tbuf += len + 1;\n \t\t\tcontinue;\n \t\t}\n@@ -217,10 +225,9 @@ void fsck_set_msg_types(struct fsck_objects_options *options, const char *values\n \tfree(to_free);\n }\n \n-static int object_on_skiplist(struct fsck_objects_options *opts,\n-\t\t\t const struct object_id *oid)\n+static int object_on_skiplist(const struct object_id *oid)\n {\n-\treturn opts && oid && oidset_contains(&opts->skiplist, oid);\n+\treturn oid && oidset_contains(&fsck_configs.oid_skiplist, oid);\n }\n \n __attribute__((format (printf, 5, 6)))\n@@ -236,7 +243,7 @@ static int report(struct fsck_objects_options *options,\n \tif (msg_type == FSCK_IGNORE)\n \t\treturn 0;\n \n-\tif (object_on_skiplist(options, oid))\n+\tif (object_on_skiplist(oid))\n \t\treturn 0;\n \n \tif (msg_type == FSCK_FATAL)\n@@ -1109,7 +1116,7 @@ static int fsck_blob(const struct object_id *oid, const char *buf,\n {\n \tint ret = 0;\n \n-\tif (object_on_skiplist(options, oid))\n+\tif (object_on_skiplist(oid))\n \t\treturn 0;\n \n \tif (oidset_contains(&options->gitmodules_found, oid)) {\ndiff --git a/fsck.h b/fsck.h\nindex b64164db17..37deadc4bd 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -135,8 +135,6 @@ struct fsck_objects_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n \tunsigned strict:1;\n-\tenum fsck_msg_type *msg_type;\n-\tstruct oidset skiplist;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n \tstruct oidset gitattributes_found;\n@@ -145,7 +143,6 @@ struct fsck_objects_options {\n };\n \n #define FSCK_OBJECTS_OPTIONS_DEFAULT { \\\n-\t.skiplist = OIDSET_INIT, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n-- \n2.45.2\n\n"},{"id":"497748","messageId":"Zn2BnsfvzfyMOZNb@ArchLinux","threadId":"61565","inReplyTo":"Zn2Ah3WDhtOmzrzn@ArchLinux","subject":"[GSoC][PATCH v5 03/12] fsck: abstract common options for reusing","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-27T15:13:34Z","receivedAt":"2024-06-27T15:13:39Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Create \"fsck_options\" struct to contain the general fields. Change\n\"fsck_objects_options\" to incorporate the general \"fsck_options\"\nand object-speicifc options.\n\nThe fsck message will use \"strict\" option to set the corresponding\n\"msg_type\". Rename the parameter name from \"struct fsck_objects_options\n*\" to \"struct fsck_options *\".\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 6 +++---\n builtin/index-pack.c | 4 ++--\n builtin/mktag.c | 6 +++---\n builtin/unpack-objects.c | 2 +-\n fsck.c | 25 ++++++++++++++++---------\n fsck.h | 32 ++++++++++++++++++++++----------\n object-file.c | 4 ++--\n 7 files changed, 49 insertions(+), 30 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex ec3220880d..c383125027 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -939,9 +939,9 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tfsck_walk_options.walk = mark_object;\n \tfsck_obj_options.walk = mark_used;\n-\tfsck_obj_options.error_func = fsck_error_func;\n+\tfsck_obj_options.fsck_options.error_func = fsck_error_func;\n \tif (check_strict)\n-\t\tfsck_obj_options.strict = 1;\n+\t\tfsck_obj_options.fsck_options.strict = 1;\n \n \tif (show_progress == -1)\n \t\tshow_progress = isatty(2);\n@@ -956,7 +956,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \tif (name_objects)\n \t\tfsck_enable_object_names(&fsck_walk_options);\n \n-\tgit_config(git_fsck_config, &fsck_obj_options);\n+\tgit_config(git_fsck_config, &fsck_obj_options.fsck_options);\n \tprepare_repo_settings(the_repository);\n \n \tif (connectivity_only) {\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 08ebeedfd3..360106b0c8 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1770,13 +1770,13 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t} else if (skip_to_optional_arg(arg, \"--strict\", &arg)) {\n \t\t\t\tstrict = 1;\n \t\t\t\tdo_fsck_object = 1;\n-\t\t\t\tfsck_set_msg_types(&fsck_objects_options, arg);\n+\t\t\t\tfsck_set_msg_types(&fsck_objects_options.fsck_options, arg);\n \t\t\t} else if (!strcmp(arg, \"--check-self-contained-and-connected\")) {\n \t\t\t\tstrict = 1;\n \t\t\t\tcheck_self_contained_and_connected = 1;\n \t\t\t} else if (skip_to_optional_arg(arg, \"--fsck-objects\", &arg)) {\n \t\t\t\tdo_fsck_object = 1;\n-\t\t\t\tfsck_set_msg_types(&fsck_objects_options, arg);\n+\t\t\t\tfsck_set_msg_types(&fsck_objects_options.fsck_options, arg);\n \t\t\t} else if (!strcmp(arg, \"--verify\")) {\n \t\t\t\tverify = 1;\n \t\t\t} else if (!strcmp(arg, \"--verify-stat\")) {\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex be2abc71d8..76860f4c7c 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -91,12 +91,12 @@ int cmd_mktag(int argc, const char **argv, const char *prefix)\n \tif (strbuf_read(&buf, 0, 0) < 0)\n \t\tdie_errno(_(\"could not read from stdin\"));\n \n-\tfsck_objects_options.error_func = mktag_fsck_error_func;\n-\tfsck_set_msg_type_from_ids(&fsck_objects_options,\n+\tfsck_objects_options.fsck_options.error_func = mktag_fsck_error_func;\n+\tfsck_set_msg_type_from_ids(&fsck_objects_options.fsck_options,\n \t\t\t\t FSCK_MSG_EXTRA_HEADER_ENTRY,\n \t\t\t\t FSCK_WARN);\n \t/* config might set fsck.extraHeaderEntry=* again */\n-\tgit_config(git_fsck_config, &fsck_objects_options);\n+\tgit_config(git_fsck_config, &fsck_objects_options.fsck_options);\n \tif (fsck_tag_standalone(NULL, buf.buf, buf.len, &fsck_objects_options,\n \t\t\t\t&tagged_oid, &tagged_type))\n \t\tdie(_(\"tag on stdin did not pass our strict fsck check\"));\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex c59e330db9..d8d0b14018 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -635,7 +635,7 @@ int cmd_unpack_objects(int argc, const char **argv, const char *prefix UNUSED)\n \t\t\t}\n \t\t\tif (skip_prefix(arg, \"--strict=\", &arg)) {\n \t\t\t\tstrict = 1;\n-\t\t\t\tfsck_set_msg_types(&fsck_objects_options, arg);\n+\t\t\t\tfsck_set_msg_types(&fsck_objects_options.fsck_options, arg);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (starts_with(arg, \"--pack_header=\")) {\ndiff --git a/fsck.c b/fsck.c\nindex 81b93f02fc..7ac6e4587c 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -107,7 +107,7 @@ void list_config_fsck_msg_ids(struct string_list *list, const char *prefix)\n }\n \n static enum fsck_msg_type fsck_msg_type(enum fsck_msg_id msg_id,\n-\t\t\t\t\tstruct fsck_objects_options *options)\n+\t\t\t\t\tstruct fsck_options *options)\n {\n \tassert(msg_id >= 0 && msg_id < FSCK_MSG_MAX);\n \n@@ -142,7 +142,7 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type)\n \treturn 1;\n }\n \n-void fsck_set_msg_type_from_ids(struct fsck_objects_options *options,\n+void fsck_set_msg_type_from_ids(struct fsck_options *options,\n \t\t\t\tenum fsck_msg_id msg_id,\n \t\t\t\tenum fsck_msg_type msg_type)\n {\n@@ -158,7 +158,7 @@ void fsck_set_msg_type_from_ids(struct fsck_objects_options *options,\n \tfsck_configs.msg_type[msg_id] = msg_type;\n }\n \n-void fsck_set_msg_type(struct fsck_objects_options *options,\n+void fsck_set_msg_type(struct fsck_options *options,\n \t\t const char *msg_id_str, const char *msg_type_str)\n {\n \tint msg_id = parse_msg_id(msg_id_str);\n@@ -187,7 +187,7 @@ void fsck_set_msg_type(struct fsck_objects_options *options,\n \tfree(to_free);\n }\n \n-void fsck_set_msg_types(struct fsck_objects_options *options, const char *values)\n+void fsck_set_msg_types(struct fsck_options *options, const char *values)\n {\n \tchar *buf = xstrdup(values), *to_free = buf;\n \tint done = 0;\n@@ -237,9 +237,16 @@ static int report(struct fsck_objects_options *options,\n {\n \tva_list ap;\n \tstruct strbuf sb = STRBUF_INIT;\n-\tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n+\tstruct fsck_options *fsck_options;\n+\tenum fsck_msg_type msg_type;\n \tint result;\n \n+\tif (options)\n+\t\tfsck_options = &options->fsck_options;\n+\telse\n+\t\tBUG(\"fsck_options is not set\");\n+\n+\tmsg_type = fsck_msg_type(msg_id, fsck_options);\n \tif (msg_type == FSCK_IGNORE)\n \t\treturn 0;\n \n@@ -256,8 +263,8 @@ static int report(struct fsck_objects_options *options,\n \n \tva_start(ap, fmt);\n \tstrbuf_vaddf(&sb, fmt, ap);\n-\tresult = options->error_func(options, oid, object_type,\n-\t\t\t\t msg_type, msg_id, sb.buf);\n+\tresult = fsck_options->error_func(options, oid, object_type,\n+\t\t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n \tva_end(ap);\n \n@@ -711,7 +718,7 @@ static int fsck_tree(const struct object_id *tree_oid,\n \t\t * bits..\n \t\t */\n \t\tcase S_IFREG | 0664:\n-\t\t\tif (!options->strict)\n+\t\t\tif (!options->fsck_options.strict)\n \t\t\t\tbreak;\n \t\t\t/* fallthrough */\n \t\tdefault:\n@@ -1283,7 +1290,7 @@ int fsck_finish(struct fsck_objects_options *options)\n int git_fsck_config(const char *var, const char *value,\n \t\t const struct config_context *ctx, void *cb)\n {\n-\tstruct fsck_objects_options *options = cb;\n+\tstruct fsck_options *options = cb;\n \tconst char *msg_id;\n \n \tif (strcmp(var, \"fsck.skiplist\") == 0) {\ndiff --git a/fsck.h b/fsck.h\nindex 37deadc4bd..e531b44a66 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -92,15 +92,16 @@ enum fsck_msg_id {\n };\n #undef MSG_ID\n \n+struct fsck_options;\n struct fsck_objects_options;\n struct object;\n \n-void fsck_set_msg_type_from_ids(struct fsck_objects_options *options,\n+void fsck_set_msg_type_from_ids(struct fsck_options *options,\n \t\t\t\tenum fsck_msg_id msg_id,\n \t\t\t\tenum fsck_msg_type msg_type);\n-void fsck_set_msg_type(struct fsck_objects_options *options,\n+void fsck_set_msg_type(struct fsck_options *options,\n \t\t const char *msg_id, const char *msg_type);\n-void fsck_set_msg_types(struct fsck_objects_options *options, const char *values);\n+void fsck_set_msg_types(struct fsck_options *options, const char *values);\n int is_valid_msg_type(const char *msg_id, const char *msg_type);\n \n /*\n@@ -131,10 +132,15 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_objects_options *o,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message);\n \n+struct fsck_options {\n+\tfsck_error error_func;\n+\tunsigned verbose:1,\n+\t\t strict:1;\n+};\n+\n struct fsck_objects_options {\n+\tstruct fsck_options fsck_options;\n \tfsck_walk_func walk;\n-\tfsck_error error_func;\n-\tunsigned strict:1;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n \tstruct oidset gitattributes_found;\n@@ -143,27 +149,33 @@ struct fsck_objects_options {\n };\n \n #define FSCK_OBJECTS_OPTIONS_DEFAULT { \\\n+\t.fsck_options = { \\\n+\t\t.error_func = fsck_error_function, \\\n+\t}, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function \\\n }\n #define FSCK_OBJECTS_OPTIONS_STRICT { \\\n-\t.strict = 1, \\\n+\t.fsck_options = { \\\n+\t\t.error_func = fsck_error_function, \\\n+\t\t.strict = 1, \\\n+\t}, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function, \\\n }\n #define FSCK_OBJECTS_OPTIONS_MISSING_GITMODULES { \\\n-\t.strict = 1, \\\n+\t.fsck_options = { \\\n+\t\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n+\t\t.strict = 1, \\\n+\t}, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n }\n \n /* descend in all linked child objects\ndiff --git a/object-file.c b/object-file.c\nindex ec44ac3d82..9eda05ee01 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2509,8 +2509,8 @@ static int index_mem(struct index_state *istate,\n \tif (flags & HASH_FORMAT_CHECK) {\n \t\tstruct fsck_objects_options opts = FSCK_OBJECTS_OPTIONS_DEFAULT;\n \n-\t\topts.strict = 1;\n-\t\topts.error_func = hash_format_check_report;\n+\t\topts.fsck_options.strict = 1;\n+\t\topts.fsck_options.error_func = hash_format_check_report;\n \t\tif (fsck_buffer(null_oid(), type, buf, size, &opts))\n \t\t\tdie(_(\"refusing to create malformed object\"));\n \t\tfsck_finish(&opts);\n-- \n2.45.2\n\n"},{"id":"497749","messageId":"Zn2BtjrhzsJlfH8U@ArchLinux","threadId":"61565","inReplyTo":"Zn2Ah3WDhtOmzrzn@ArchLinux","subject":"[GSoC][PATCH v5 04/12] fsck: add \"fsck_refs_options\" struct","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-27T15:13:58Z","receivedAt":"2024-06-27T15:14:02Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Add \"fsck_refs_options\" to support ref consistency checks.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.h | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/fsck.h b/fsck.h\nindex e531b44a66..a17fee30b4 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -138,6 +138,10 @@ struct fsck_options {\n \t\t strict:1;\n };\n \n+struct fsck_refs_options {\n+\tstruct fsck_options fsck_options;\n+};\n+\n struct fsck_objects_options {\n \tstruct fsck_options fsck_options;\n \tfsck_walk_func walk;\n-- \n2.45.2\n\n"},{"id":"497750","messageId":"Zn2BzdUa8tT6_WpN@ArchLinux","threadId":"61565","inReplyTo":"Zn2Ah3WDhtOmzrzn@ArchLinux","subject":"[GSoC][PATCH v5 05/12] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-27T15:14:21Z","receivedAt":"2024-06-27T15:14:26Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The static function \"report\" provided by \"fsck.c\" aims at reporting the\nproblems related to object database which cannot be reused for refs.\nIn order to provide a unified interface which can report either objects\nor refs, create a new function \"fsck_report\" by adding two parameters\n\"refs_options\" and \"checked_ref_name\" following the \"report\" prototype.\nHowever, instead of using \"...\", provide \"va_list\" to allow more\nflexibility.\n\nThe \"vfsck_report\" function will use \"error_func\" registered in\n\"fsck_options\" function to report customized messages. Change\n\"error_func\" prototype to align with the \"vfsck_report\".\n\nChange \"report\" function to make it use \"vfsck_report\" to report\nobjects-related messages. Add a new function called \"fsck_refs_report\"\nto use \"vfsck_report\" to report refs-related messages. Also, create a\ngeneral function \"fsck_report\" for future use.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 4 ++-\n builtin/mktag.c | 4 ++-\n fsck.c | 88 ++++++++++++++++++++++++++++++++++++++++---------\n fsck.h | 39 +++++++++++++++++++---\n object-file.c | 14 ++++----\n 5 files changed, 122 insertions(+), 27 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex c383125027..2a3b536c1b 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -89,9 +89,11 @@ static int objerror(struct object *obj, const char *err)\n \treturn -1;\n }\n \n-static int fsck_error_func(struct fsck_objects_options *o UNUSED,\n+static int fsck_error_func(struct fsck_objects_options *objects_options UNUSED,\n+\t\t\t struct fsck_refs_options *refs_options UNUSED,\n \t\t\t const struct object_id *oid,\n \t\t\t enum object_type object_type,\n+\t\t\t const char *checked_ref_name UNUSED,\n \t\t\t enum fsck_msg_type msg_type,\n \t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t const char *message)\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex 76860f4c7c..0779a778e9 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -17,9 +17,11 @@ static int option_strict = 1;\n \n static struct fsck_objects_options fsck_objects_options = FSCK_OBJECTS_OPTIONS_STRICT;\n \n-static int mktag_fsck_error_func(struct fsck_objects_options *o UNUSED,\n+static int mktag_fsck_error_func(struct fsck_objects_options *objects_options UNUSED,\n+\t\t\t\t struct fsck_refs_options *refs_options UNUSED,\n \t\t\t\t const struct object_id *oid UNUSED,\n \t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const char *checked_ref_name UNUSED,\n \t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t\t const char *message)\ndiff --git a/fsck.c b/fsck.c\nindex 7ac6e4587c..5184d17736 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -230,19 +230,23 @@ static int object_on_skiplist(const struct object_id *oid)\n \treturn oid && oidset_contains(&fsck_configs.oid_skiplist, oid);\n }\n \n-__attribute__((format (printf, 5, 6)))\n-static int report(struct fsck_objects_options *options,\n-\t\t const struct object_id *oid, enum object_type object_type,\n-\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+static int vfsck_report(struct fsck_objects_options *objects_options,\n+\t\t\tstruct fsck_refs_options *refs_options,\n+\t\t\tconst struct object_id *oid,\n+\t\t\tenum object_type object_type,\n+\t\t\tconst char *checked_ref_name,\n+\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n {\n-\tva_list ap;\n+\tva_list ap_copy;\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct fsck_options *fsck_options;\n \tenum fsck_msg_type msg_type;\n \tint result;\n \n-\tif (options)\n-\t\tfsck_options = &options->fsck_options;\n+\tif (objects_options)\n+\t\tfsck_options = &objects_options->fsck_options;\n+\telse if (refs_options)\n+\t\tfsck_options = &refs_options->fsck_options;\n \telse\n \t\tBUG(\"fsck_options is not set\");\n \n@@ -261,9 +265,10 @@ static int report(struct fsck_objects_options *options,\n \tprepare_msg_ids();\n \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n \n-\tva_start(ap, fmt);\n-\tstrbuf_vaddf(&sb, fmt, ap);\n-\tresult = fsck_options->error_func(options, oid, object_type,\n+\tva_copy(ap_copy, ap);\n+\tstrbuf_vaddf(&sb, fmt, ap_copy);\n+\tresult = fsck_options->error_func(objects_options, NULL,\n+\t\t\t\t\t oid, object_type, checked_ref_name,\n \t\t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n \tva_end(ap);\n@@ -271,6 +276,51 @@ static int report(struct fsck_objects_options *options,\n \treturn result;\n }\n \n+__attribute__((format (printf, 5, 6)))\n+static int report(struct fsck_objects_options *objects_options,\n+\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = vfsck_report(objects_options, NULL, oid, object_type, \"\",\n+\t\t\t msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n+int fsck_report(struct fsck_objects_options *objects_options,\n+\t\tstruct fsck_refs_options *refs_options,\n+\t\tconst struct object_id *oid,\n+\t\tenum object_type object_type,\n+\t\tconst char *checked_ref_name,\n+\t\tenum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = vfsck_report(objects_options, refs_options, oid, object_type,\n+\t\t\t checked_ref_name, msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+\n+}\n+\n+int fsck_refs_report(struct fsck_refs_options *refs_options,\n+\t\t const struct object_id *oid,\n+\t\t const char *checked_ref_name,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = fsck_report(NULL, refs_options, oid, OBJ_NONE,\n+\t\t\t checked_ref_name, msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n void fsck_enable_object_names(struct fsck_objects_options *options)\n {\n \tif (!options->object_names)\n@@ -1216,18 +1266,22 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n-int fsck_error_function(struct fsck_objects_options *o,\n+int fsck_error_function(struct fsck_objects_options *objects_options,\n+\t\t\tstruct fsck_refs_options *refs_options UNUSED,\n \t\t\tconst struct object_id *oid,\n \t\t\tenum object_type object_type UNUSED,\n+\t\t\tconst char *checked_ref_name UNUSED,\n \t\t\tenum fsck_msg_type msg_type,\n \t\t\tenum fsck_msg_id msg_id UNUSED,\n \t\t\tconst char *message)\n {\n \tif (msg_type == FSCK_WARN) {\n-\t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n+\t\twarning(\"object %s: %s\",\n+\t\t\tfsck_describe_object(objects_options, oid), message);\n \t\treturn 0;\n \t}\n-\terror(\"object %s: %s\", fsck_describe_object(o, oid), message);\n+\terror(\"object %s: %s\",\n+\t fsck_describe_object(objects_options, oid), message);\n \treturn 1;\n }\n \n@@ -1320,9 +1374,11 @@ int git_fsck_config(const char *var, const char *value,\n * Custom error callbacks that are used in more than one place.\n */\n \n-int fsck_error_cb_print_missing_gitmodules(struct fsck_objects_options *o,\n+int fsck_error_cb_print_missing_gitmodules(struct fsck_objects_options *objects_options,\n+\t\t\t\t\t struct fsck_refs_options *refs_options,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t const char *checked_ref_name,\n \t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message)\n@@ -1331,5 +1387,7 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_objects_options *o,\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n+\treturn fsck_error_function(objects_options, refs_options,\n+\t\t\t\t oid, object_type, checked_ref_name,\n+\t\t\t\t msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex a17fee30b4..70d5e78ae6 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -93,6 +93,7 @@ enum fsck_msg_id {\n #undef MSG_ID\n \n struct fsck_options;\n+struct fsck_refs_options;\n struct fsck_objects_options;\n struct object;\n \n@@ -115,19 +116,27 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n \t\t\t void *data, struct fsck_objects_options *options);\n \n-/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n-typedef int (*fsck_error)(struct fsck_objects_options *o,\n+/*\n+ * callback function for reporting errors when checking either objects or refs\n+ */\n+typedef int (*fsck_error)(struct fsck_objects_options *objects_options,\n+\t\t\t struct fsck_refs_options *refs_options,\n \t\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t\t const char *checked_ref_name,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n-int fsck_error_function(struct fsck_objects_options *o,\n+int fsck_error_function(struct fsck_objects_options *objects_options,\n+\t\t\tstruct fsck_refs_options *refs_options,\n \t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\tconst char *checked_ref_name,\n \t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\tconst char *message);\n-int fsck_error_cb_print_missing_gitmodules(struct fsck_objects_options *o,\n+int fsck_error_cb_print_missing_gitmodules(struct fsck_objects_options *objects_options,\n+\t\t\t\t\t struct fsck_refs_options *refs_options,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t const char *checked_ref_name,\n \t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message);\n@@ -223,6 +232,28 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_objects_options *options);\n \n+/*\n+ * Provide a unified interface for either fscking refs or objects.\n+ * It will get the current msg error type and call the error_func callback\n+ * which is registered in the \"fsck_options\" struct. For refs, the caller\n+ * should pass NULL for \"objs_options\". For objects, the caller should pass\n+ * NULL for \"refs_options\".\n+ */\n+__attribute__((format (printf, 7, 8)))\n+int fsck_report(struct fsck_objects_options *objects_options,\n+\t\tstruct fsck_refs_options *refs_options,\n+\t\tconst struct object_id *oid,\n+\t\tenum object_type object_type,\n+\t\tconst char *checked_ref_name,\n+\t\tenum fsck_msg_id msg_id, const char *fmt, ...);\n+\n+__attribute__((format (printf, 5, 6)))\n+int fsck_refs_report(struct fsck_refs_options *refs_options,\n+\t\t const struct object_id *oid,\n+\t\t const char *checked_ref_name,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\ndiff --git a/object-file.c b/object-file.c\nindex 9eda05ee01..5cb9117fc4 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2472,12 +2472,14 @@ int repo_has_object_file(struct repository *r,\n * report the minimal fsck error here, and rely on the caller to\n * give more context.\n */\n-static int hash_format_check_report(struct fsck_objects_options *opts UNUSED,\n-\t\t\t\t const struct object_id *oid UNUSED,\n-\t\t\t\t enum object_type object_type UNUSED,\n-\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n-\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t\t const char *message)\n+static int hash_format_check_report(struct fsck_objects_options *objects_options UNUSED,\n+\t\t\t\t struct fsck_refs_options *refs_options UNUSED,\n+\t\t\t\t const struct object_id *oid UNUSED,\n+\t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const char *chekced_ref_name UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \terror(_(\"object fails fsck: %s\"), message);\n \treturn 1;\n-- \n2.45.2\n\n"},{"id":"497751","messageId":"Zn2B7q0Ai1c81lEf@ArchLinux","threadId":"61565","inReplyTo":"Zn2Ah3WDhtOmzrzn@ArchLinux","subject":"[GSoC][PATCH v5 06/12] fsck: add \"fsck_refs_options\" initialization macros","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-27T15:14:54Z","receivedAt":"2024-06-27T15:14:58Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\" macros to\ncreate the \"fsck_refs_options\" easily. Add refs-specific \"error_func\"\ncallback \"fsck_refs_error_function\".\n\n\"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\npassed the oid, it will use \"oid_to_hex\" to get the corresponding hex\nvalue to report to the user.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 23 +++++++++++++++++++++++\n fsck.h | 19 +++++++++++++++++++\n 2 files changed, 42 insertions(+)\n\ndiff --git a/fsck.c b/fsck.c\nindex 5184d17736..4869566d19 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1285,6 +1285,29 @@ int fsck_error_function(struct fsck_objects_options *objects_options,\n \treturn 1;\n }\n \n+int fsck_refs_error_function(struct fsck_objects_options *objects_options UNUSED,\n+\t\t\t struct fsck_refs_options *refs_options,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type UNUSED,\n+\t\t\t const char *checked_ref_name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tstatic struct strbuf sb = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&sb, checked_ref_name);\n+\tif (oid)\n+\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n+\n+\tif (msg_type == FSCK_WARN) {\n+\t\twarning(\"%s: %s\", sb.buf, message);\n+\t\treturn 0;\n+\t}\n+\terror(\"%s: %s\", sb.buf, message);\n+\treturn 1;\n+}\n+\n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n \t\t struct fsck_objects_options *options,\ndiff --git a/fsck.h b/fsck.h\nindex 70d5e78ae6..e903845690 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -140,6 +140,14 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_objects_options *objects_\n \t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message);\n+int fsck_refs_error_function(struct fsck_objects_options *objects_options,\n+\t\t\t struct fsck_refs_options *refs_options,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type,\n+\t\t\t const char *checked_ref_name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n \n struct fsck_options {\n \tfsck_error error_func;\n@@ -150,6 +158,17 @@ struct fsck_options {\n struct fsck_refs_options {\n \tstruct fsck_options fsck_options;\n };\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.fsck_options = { \\\n+\t\t.error_func = fsck_refs_error_function, \\\n+\t}, \\\n+}\n+#define FSCK_REFS_OPTIONS_STRICT { \\\n+\t.fsck_options = { \\\n+\t\t.error_func = fsck_refs_error_function, \\\n+\t\t.strict = 1, \\\n+\t}, \\\n+}\n \n struct fsck_objects_options {\n \tstruct fsck_options fsck_options;\n-- \n2.45.2\n\n"},{"id":"497752","messageId":"Zn2CAgdg65SlJvRu@ArchLinux","threadId":"61565","inReplyTo":"Zn2Ah3WDhtOmzrzn@ArchLinux","subject":"[GSoC][PATCH v5 07/12] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-27T15:15:14Z","receivedAt":"2024-06-27T15:15:18Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It's organized as the five parts:\n\n1. The name and the initialization interfaces.\n2. The ref transaction interfaces.\n3. The ref internal interfaces (pack, rename and copy).\n4. The ref filesystem interfaces.\n5. The reflog related interfaces.\n\nTo keep consistent with the git-fsck(1), add a new interface named\n\"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\ngrouped into any above five categories. Explicitly add blank line to\nmake it different from others.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs.c | 5 +++++\n refs.h | 8 ++++++++\n refs/debug.c | 11 +++++++++++\n refs/files-backend.c | 15 ++++++++++++++-\n refs/packed-backend.c | 8 ++++++++\n refs/refs-internal.h | 6 ++++++\n refs/reftable-backend.c | 8 ++++++++\n 7 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex dd0d9c360f..646fdfaf22 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -316,6 +316,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs, struct fsck_refs_options *o)\n+{\n+\treturn refs->be->fsck(refs, o);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex df8cbf6124..a8f4deb108 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -3,6 +3,7 @@\n \n #include \"commit.h\"\n \n+struct fsck_refs_options;\n struct object_id;\n struct ref_store;\n struct repository;\n@@ -547,6 +548,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_refs_options *o);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..6306c25fd3 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tint res = drefs->refs->be->fsck(drefs->refs, o);\n+\ttrace_printf_key(&trace_refs, \"fsck: %d\\n\", res);\n+\treturn res;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 44c5c3b201..7257fffb77 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3406,6 +3406,17 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o)\n+{\n+\tint ret;\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\treturn ret;\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_init,\n@@ -3432,5 +3443,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex c4c1e36aa2..db152053f8 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1733,6 +1733,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_init,\n@@ -1760,4 +1766,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex fa975d69aa..ecc082baf8 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_options;\n struct ref_transaction;\n \n /*\n@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_refs_options *o);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -677,6 +681,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 7d872a32ac..2a358b40ca 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2292,6 +2292,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_refs_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2319,4 +2325,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.45.2\n\n"},{"id":"497753","messageId":"Zn2CHJzhLntjqPuG@ArchLinux","threadId":"61565","inReplyTo":"Zn2Ah3WDhtOmzrzn@ArchLinux","subject":"[GSoC][PATCH v5 08/12] builtin/refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-27T15:15:40Z","receivedAt":"2024-06-27T15:15:44Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\ncheck the reference database consistency.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/git-refs.txt | 13 +++++++++++++\n builtin/refs.c | 36 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 49 insertions(+)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..1244a85b64 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+The following options are specific to 'git refs verify':\n+\n+--strict::\n+\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n+\tbe seen as ERROR. See linkgit:git-fsck[1].\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..511b0d0f8b 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,46 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct fsck_refs_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tunsigned int verbose = 0, strict = 0;\n+\tstruct option options[] = {\n+\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"too many arguments\"));\n+\n+\tif (verbose)\n+\t\tfsck_refs_options.fsck_options.verbose = 1;\n+\tif (strict)\n+\t\tfsck_refs_options.fsck_options.strict = 1;\n+\n+\tgit_config(git_fsck_config, &fsck_refs_options.fsck_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\treturn refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \n-- \n2.45.2\n\n"},{"id":"497754","messageId":"Zn2CMab7155p41Mk@ArchLinux","threadId":"61565","inReplyTo":"Zn2Ah3WDhtOmzrzn@ArchLinux","subject":"[GSoC][PATCH v5 09/12] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-27T15:16:01Z","receivedAt":"2024-06-27T15:16:04Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new function \"fsck_refs\" that initializes and runs a child\nprocess to execute the \"git-refs verify\" command.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 2a3b536c1b..a5654da62b 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -899,6 +899,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \treturn res;\n }\n \n+static void fsck_refs(void)\n+{\n+\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n+\tchild_process_init(&refs_verify);\n+\trefs_verify.git_cmd = 1;\n+\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n+\tif (verbose)\n+\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n+\tif (check_strict)\n+\t\tstrvec_push(&refs_verify.args, \"--strict\");\n+\n+\tif (run_command(&refs_verify))\n+\t\terrors_found |= ERROR_REFS;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n@@ -1068,6 +1083,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n+\tfsck_refs();\n+\n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \n-- \n2.45.2\n\n"},{"id":"497755","messageId":"Zn2CYu_xTKg9snbJ@ArchLinux","threadId":"61565","inReplyTo":"Zn2Ah3WDhtOmzrzn@ArchLinux","subject":"[GSoC][PATCH v5 10/12] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-27T15:16:50Z","receivedAt":"2024-06-27T15:16:54Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"For refs and reflogs, we need to scan its corresponding directories to\ncheck every regular file or symbolic link which shares the same pattern.\nIntroduce a unified interface for scanning directories for\nfiles-backend.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs/files-backend.c | 77 +++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 76 insertions(+), 1 deletion(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 7257fffb77..aefc947fc3 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -4,6 +4,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -3406,6 +3407,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct fsck_refs_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_refs_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n+{\n+\tconst char *gitdir = ref_store->gitdir;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\n+\tif (!iter) {\n+\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n+\t\tgoto out;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->fsck_options.verbose)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n+\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n+\t\t\t\t iter->basename);\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n+\n+out:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_refs_options *o)\n+{\n+\tint ret;\n+\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tNULL\n+\t};\n+\n+\tif (o->fsck_options.verbose)\n+\t\tfprintf_ln(stderr, \"Checking references consistency\");\n+\n+\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n+\n+\treturn ret;\n+}\n+\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_refs_options *o)\n {\n@@ -3413,7 +3486,9 @@ static int files_fsck(struct ref_store *ref_store,\n \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n-\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o)\n+\t | files_fsck_refs(ref_store, o);\n+\n \treturn ret;\n }\n \n-- \n2.45.2\n\n"},{"id":"497756","messageId":"Zn2Coy8iTWcfCu-g@ArchLinux","threadId":"61565","inReplyTo":"Zn2Ah3WDhtOmzrzn@ArchLinux","subject":"[GSoC][PATCH v5 11/12] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-27T15:17:55Z","receivedAt":"2024-06-27T15:17:59Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) only implicitly checks the reference, it does not fully\ncheck refs with bad format name such as standalone \"@\" and name ending\nwith \".lock\".\n\nIn order to provide such checks, add a new fsck message id \"badRefName\"\nwith default ERROR type. Use existing \"check_refname_format\" to explicit\ncheck the ref name. And add a new unit test to verify the functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 +\n fsck.h | 1 +\n refs/files-backend.c | 20 +++++++\n t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++\n 4 files changed, 125 insertions(+)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex f643585a34..dab4012246 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,9 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefName`::\n+\t(ERROR) A ref has a bad name.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex e903845690..aad2211bff 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex aefc947fc3..e840a768b4 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3417,6 +3417,25 @@ typedef int (*files_fsck_refs_fn)(struct fsck_refs_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n+static int files_fsck_refs_name(struct fsck_refs_options *o,\n+\t\t\t\tconst char *gitdir UNUSED,\n+\t\t\t\tconst char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\tret = fsck_refs_report(o, NULL, sb.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_refs_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3468,6 +3487,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n {\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tfiles_fsck_refs_name,\n \t\tNULL\n \t};\n \ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..b2db58d2c6\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit tag multi_hierarchy/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1.lock: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/@ &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\tgit -c fsck.badRefName=warn fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\tgit -c fsck.badRefName=ignore fsck 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+test_done\n-- \n2.45.2\n\n"},{"id":"497757","messageId":"Zn2CtZ8_ai5Oy224@ArchLinux","threadId":"61565","inReplyTo":"Zn2Ah3WDhtOmzrzn@ArchLinux","subject":"[GSoC][PATCH v5 12/12] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-27T15:18:13Z","receivedAt":"2024-06-27T15:18:17Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Enhance the git-fsck(1) command by adding a check for reference content\nin the files backend. The new functionality ensures that symrefs, real\nsymbolic link and regular refs are validated correctly.\n\nIn order to check the trailing content of the regular refs, add a new\nparameter `trailing` to `parse_loose_ref_contents`.\n\nFor symrefs, `parse_loose_ref_contents` will set the \"referent\".\nHowever, symbolic link could be either absolute or relative. Use\n\"strbuf_add_real_path\" to read the symbolic link and convert the\nrelative path to absolute path. Then use \"skip_prefix\" to make it align\nwith symref \"referent\".\n\nThus, the symrefs and symbolic links could share the same interface. Add\na new function \"files_fsck_symref_target\" which aims at checking the\nfollowing things:\n\n1. whether the pointee is under the `refs/` directory.\n2. whether the pointee name is correct.\n3. whether the pointee path is a wrong type in filesystem.\n\nLast, add the following FSCK MESSAGEs:\n\n1. \"badRefContent(ERROR)\": A ref has a bad content\n2. \"badSymrefPointee(ERROR)\": The pointee of a symref is bad.\n3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 9 +++\n fsck.h | 3 +\n refs.c | 2 +-\n refs/files-backend.c | 145 +++++++++++++++++++++++++++++++++-\n refs/refs-internal.h | 5 +-\n t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++\n 6 files changed, 269 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex dab4012246..b1630a478b 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,9 +19,15 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefContent`::\n+\t(ERROR) A ref has a bad content.\n+\n `badRefName`::\n \t(ERROR) A ref has a bad name.\n \n+`badSymrefPointee`::\n+\t(ERROR) The pointee of a symref is bad.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \n@@ -167,6 +173,9 @@\n `nullSha1`::\n \t(WARN) Tree contains entries pointing to a null sha1.\n \n+`trailingRefContent`::\n+\t(WARN) A ref content has trailing contents.\n+\n `treeNotSorted`::\n \t(ERROR) A tree is not properly sorted.\n \ndiff --git a/fsck.h b/fsck.h\nindex aad2211bff..5c1b113674 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -32,6 +32,8 @@ enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n+\tFUNC(BAD_REF_CONTENT, ERROR) \\\n+\tFUNC(BAD_SYMREF_POINTEE, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n@@ -72,6 +74,7 @@ enum fsck_msg_type {\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n+\tFUNC(TRAILING_REF_CONTENT, WARN) \\\n \tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n \tFUNC(LARGE_PATHNAME, WARN) \\\ndiff --git a/refs.c b/refs.c\nindex 646fdfaf22..55d581ab12 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1758,7 +1758,7 @@ static int refs_read_special_head(struct ref_store *ref_store,\n \t}\n \n \tresult = parse_loose_ref_contents(content.buf, oid, referent, type,\n-\t\t\t\t\t failure_errno);\n+\t\t\t\t\t failure_errno, NULL);\n \n done:\n \tstrbuf_release(&full_path);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex e840a768b4..0e6b6d3f6a 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1,4 +1,5 @@\n #include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n #include \"../copy.h\"\n #include \"../environment.h\"\n #include \"../gettext.h\"\n@@ -551,7 +552,7 @@ static int read_ref_internal(struct ref_store *ref_store, const char *refname,\n \tstrbuf_rtrim(&sb_contents);\n \tbuf = sb_contents.buf;\n \n-\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr);\n+\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr, NULL);\n \n out:\n \tif (ret && !myerr)\n@@ -587,7 +588,7 @@ static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refn\n \n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno)\n+\t\t\t int *failure_errno, const char **trailing)\n {\n \tconst char *p;\n \tif (skip_prefix(buf, \"ref:\", &buf)) {\n@@ -609,6 +610,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t*failure_errno = EINVAL;\n \t\treturn -1;\n \t}\n+\n+\tif (trailing)\n+\t\t*trailing = p;\n+\n \treturn 0;\n }\n \n@@ -3436,6 +3441,141 @@ static int files_fsck_refs_name(struct fsck_refs_options *o,\n \treturn ret;\n }\n \n+/*\n+ * Check the symref \"pointee_name\" and \"pointee_path\". The caller should\n+ * make sure that \"pointee_path\" is absolute. For symbolic ref, \"pointee_name\"\n+ * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n+ * be the relative path agaignst \"gitdir\".\n+ */\n+static int files_fsck_symref_target(struct fsck_refs_options *o,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const char *pointee_name,\n+\t\t\t\t const char *pointee_path)\n+{\n+\tconst char *p = NULL;\n+\tstruct stat st;\n+\tint ret = 0;\n+\n+\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n+\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to target out of refs hierarchy\");\n+\t\tgoto out;\n+\t}\n+\n+\tif (check_refname_format(pointee_name, 0)) {\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid refname\");\n+\t}\n+\n+\tif (lstat(pointee_path, &st) < 0)\n+\t\tgoto out;\n+\n+\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid target\");\n+\t\tgoto out;\n+\t}\n+out:\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs_content(struct fsck_refs_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter)\n+{\n+\tstruct strbuf pointee_path = STRBUF_INIT,\n+\t\t ref_content = STRBUF_INIT,\n+\t\t abs_gitdir = STRBUF_INIT,\n+\t\t referent = STRBUF_INIT,\n+\t\t refname = STRBUF_INIT;\n+\tconst char *trailing = NULL;\n+\tint failure_errno = 0;\n+\tunsigned int type = 0;\n+\tstruct object_id oid;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\n+\t/*\n+\t * If the file is a symlink, we need to only check the connectivity\n+\t * of the destination object.\n+\t */\n+\tif (S_ISLNK(iter->st.st_mode)) {\n+\t\tconst char *pointee_name = NULL;\n+\n+\t\tstrbuf_add_real_path(&pointee_path, iter->path.buf);\n+\n+\t\tstrbuf_add_absolute_path(&abs_gitdir, gitdir);\n+\t\tstrbuf_normalize_path(&abs_gitdir);\n+\t\tif (!is_dir_sep(abs_gitdir.buf[abs_gitdir.len - 1]))\n+\t\t\tstrbuf_addch(&abs_gitdir, '/');\n+\n+\t\tif (!skip_prefix(pointee_path.buf,\n+\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n+\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t\t \"point to target outside gitdir\");\n+\t\t\tgoto clean;\n+\t\t}\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, pointee_name,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n+\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n+\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n+\t\t\t\t &referent, &type,\n+\t\t\t\t &failure_errno, &trailing)) {\n+\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n+\t\t\t\t \"invalid ref content\");\n+\t\tgoto clean;\n+\t}\n+\n+\t/*\n+\t * If the ref is a symref, we need to check the destination name and\n+\t * connectivity.\n+\t */\n+\tif (referent.len && (type & REF_ISSYMREF)) {\n+\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n+\t\tstrbuf_rtrim(&referent);\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, referent.buf,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t} else {\n+\t\t/*\n+\t\t * Only regular refs could have a trailing garbage. Should\n+\t\t * be reported as a warning.\n+\t\t */\n+\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n+\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n+\t\t\t\t\t \"trailing garbage in ref\");\n+\t\t\tgoto clean;\n+\t\t}\n+\t}\n+\n+clean:\n+\tstrbuf_release(&abs_gitdir);\n+\tstrbuf_release(&pointee_path);\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&ref_content);\n+\tstrbuf_release(&referent);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_refs_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3488,6 +3628,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n \t\tfiles_fsck_refs_name,\n+\t\tfiles_fsck_refs_content,\n \t\tNULL\n \t};\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex ecc082baf8..915d0d1f20 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -709,11 +709,12 @@ struct ref_store {\n \n /*\n * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for\n- * invalid contents.\n+ * invalid contents. Also *trailing is set to the first character after the\n+ * refname or NULL if the referent is not empty.\n */\n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno);\n+\t\t\t int *failure_errno, const char **trailing);\n \n /*\n * Fill in the generic part of refs and add it to our collection of\ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nindex b2db58d2c6..35bf40ee64 100755\n--- a/t/t0602-reffiles-fsck.sh\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -98,4 +98,114 @@ test_expect_success 'ref name check should be adapted into fsck messages' '\n \t)\n '\n \n+test_expect_success 'regular ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit checkout -b a/b/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse branch-1)\" > $branch_dir_prefix/branch-1-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse tag-1)\" > $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_must_fail git -c fsck.trailingRefContent=error fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s \" \"$(git rev-parse tag-2)\" > $tag_dir_prefix/tag-2-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/tags/tag-2-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/a/b/branch-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'symbolic ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid refname\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid target\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: logs/maint-v2.45\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to target out of refs hierarchy\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.2\n\n"},{"id":"497765","messageId":"xmqqa5j6rs03.fsf@gitster.g","threadId":"61565","inReplyTo":"Zn2BXRatUdX16PD3@ArchLinux","subject":"Re: [GSoC][PATCH v5 01/12] fsck: rename \"fsck_options\" to \"fsck_objects_options\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-06-27T21:32:44Z","receivedAt":"2024-06-27T21:32:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"shejialuo writes:\n\n> -static int fsck_error_func(struct fsck_options *o UNUSED,\n> +static int fsck_error_func(struct fsck_objects_options *o UNUSED,\n> \t\t\t const struct object_id *oid,\n> \t\t\t enum object_type object_type,\n> \t\t\t enum fsck_msg_type msg_type,\n\nIt is curious that the addition/renaming of fsck_objects_options is\npresumably to allow fsck_${xyzzy}_options to be added for different\n$xyzzy (the first one being \"refs\"), and this function is only about\nfsck_objects_options. What name would the corresponding error\nfunction, called by checkers that take fsck_${xyzzy}_options, be\ngiven? fsck_${xyzzy}_error_func()? Shouldn't this be then become\nfsck_objects_error_func() or something?\n\nHaving said that.\n\nDo we really need such a parallel system between \"objects\" and other\nkinds of things being checked that you are introducing with this\nstep? What benefit are we getting from this additional complexity?\n\nI would have expected that adding ref-related new members that\nobject consistency checkers has no interest in to the fsck_options\nstructure would be sufficient for the purpose of this series. Or if\nwe really wanted to prepare for more complex future, use of the\n\"union of variants, switched with a tag\" pattern to arrange the data\nthis way:\n\n\tstruct fsck_options {\n\t\tenum fsck_type {\n\t\t\tFSCK_OBJECTS,\n\t\t\tFSCK_REFS,\n\t\t\t...\n\t\t} t;\n\t\tunion {\n\t\t\tstruct fsck_objects_options objects;\n\t\t\tstruct fsck_refs_options refs;\n\t\t} u;\n\t};\n\nwould still allow functions like fsck_error_func(), and\nfsck_set_msg_types(), etc. to work on the common \"fsck_options\".\n\nI dunno.\n"},{"id":"497766","messageId":"xmqqzfr6qcx8.fsf@gitster.g","threadId":"61565","inReplyTo":"Zn2BgEw0geZwvr3_@ArchLinux","subject":"Re: [GSoC][PATCH v5 02/12] fsck: use \"fsck_configs\" to set up configs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-06-27T21:43:47Z","receivedAt":"2024-06-27T21:43:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"shejialuo writes:\n\n> Some fields such as \"msg_type\" and \"skiplist\" in \"fsck_objects_options\"\n> are not options, these fields are related to \"git-config(1)\" which are\n> initialized using \"git_fsck_config\" function. Create a static variable\n> named \"fsck_configs\" in \"fsck.c\" which aims at handling configs. Thus we\n> don't need to reply on the \"fsck_objects_options\" to set up the fsck\n> error message severity.\n\nreply???\n\nAs configuration often is used to prepopulate options, I need a lot\nstonger justification to split these into a different structure than\n\"'config' is a noun that is different from a noun 'option'\". \n\nIf we intend to have many \"option\" instances but what these two\nmembers store will be the same across these \"option\" instances, for\nexample, that would be a lot better reason why we may want to\nseparate these two members out of it, but I have a suspicion that if\nwe were to use the \"union with tags\" approach, these two would\nbecome members of the common part shared between \"objects' and\n\"refs\", i.e. the overall data structure might look more like this:\n\n\tstruct fsck_options {\n\t\tenum fsck_msg_type *msg_type;\n\t\tstruct oidset oid_skiplist;\n\t\tenum fsck_what_check { O, R } tag;\n\t\tunion {\n\t\t\tstruct fsck_object_options o;\n\t\t\tstruct fsck_ref_options r;\n\t\t} u;\n\t};\n\nby moving these two members out of fsck_object_options and moving\nthem to the shared part.\n\nI dunno. It is unclear what the real reason is for these two things\nto be extracted out and made to appear totally independent from the\n\"options\" thing to begin with, and I am not sure if I agree with the\nreason when it is known.\n"},{"id":"497784","messageId":"Zn4xeoqhCeBCSNvg@ArchLinux","threadId":"61565","inReplyTo":"xmqqa5j6rs03.fsf@gitster.g","subject":"Re: [GSoC][PATCH v5 01/12] fsck: rename \"fsck_options\" to \"fsck_objects_options\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-28T03:43:54Z","receivedAt":"2024-06-28T03:43:57Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Thu, Jun 27, 2024 at 02:32:44PM -0700, Junio C Hamano wrote:\n> shejialuo writes:\n> \n> > -static int fsck_error_func(struct fsck_options *o UNUSED,\n> > +static int fsck_error_func(struct fsck_objects_options *o UNUSED,\n> > \t\t\t const struct object_id *oid,\n> > \t\t\t enum object_type object_type,\n> > \t\t\t enum fsck_msg_type msg_type,\n> \n> It is curious that the addition/renaming of fsck_objects_options is\n> presumably to allow fsck_${xyzzy}_options to be added for different\n> $xyzzy (the first one being \"refs\"), and this function is only about\n> fsck_objects_options. What name would the corresponding error\n> function, called by checkers that take fsck_${xyzzy}_options, be\n> given? fsck_${xyzzy}_error_func()? Shouldn't this be then become\n> fsck_objects_error_func() or something?\n> \n\nYes, it should be definitely changed here. Will improve in the next\nversion.\n\n> Having said that.\n> \n> Do we really need such a parallel system between \"objects\" and other\n> kinds of things being checked that you are introducing with this\n> step? What benefit are we getting from this additional complexity?\n> \n\nI am agree that the most simple way to handle for this series is add\nsome ref-related new members. Thus, we can reuse existing code. However,\nit makes me feel so weird when implementing the code using this idea.\nFor example,\n\n struct fsck_options {\n struct fsck_refs_options;\n ...\n }\n\nWhen we create a new \"fsck_options\", it will be so misleading that the\ncaller may think we will handle both refs and objects checks by using\n\"fsck_options\". So I just introduce this parallel system. When checking\nobjects, caller should explicitly create \"fsck_objects_options\", when\nchecking refs, caller should explicitly create \"fsck_refs_options\".\n\nBecause in semantics, we introduce a new check here. Combination means\nwe will check the both. Although it is simple, but it will cause a lot\nof trouble in the future.\n\n> I would have expected that adding ref-related new members that\n> object consistency checkers has no interest in to the fsck_options\n> structure would be sufficient for the purpose of this series. Or if\n> we really wanted to prepare for more complex future, use of the\n> \"union of variants, switched with a tag\" pattern to arrange the data\n> this way:\n> \n> \tstruct fsck_options {\n> \t\tenum fsck_type {\n> \t\t\tFSCK_OBJECTS,\n> \t\t\tFSCK_REFS,\n> \t\t\t...\n> \t\t} t;\n> \t\tunion {\n> \t\t\tstruct fsck_objects_options objects;\n> \t\t\tstruct fsck_refs_options refs;\n> \t\t} u;\n> \t};\n> \n> would still allow functions like fsck_error_func(), and\n> fsck_set_msg_types(), etc. to work on the common \"fsck_options\".\n> \n\nI agree that we could use this pattern, using union will make the\nsemantics more clear.\n\n> I dunno.\n"},{"id":"497786","messageId":"Zn46hGUqz1ZtUpGz@ArchLinux","threadId":"61565","inReplyTo":"xmqqzfr6qcx8.fsf@gitster.g","subject":"Re: [GSoC][PATCH v5 02/12] fsck: use \"fsck_configs\" to set up configs","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-06-28T04:22:28Z","receivedAt":"2024-06-28T04:22:31Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Thu, Jun 27, 2024 at 02:43:47PM -0700, Junio C Hamano wrote:\n> shejialuo writes:\n> \n> > Some fields such as \"msg_type\" and \"skiplist\" in \"fsck_objects_options\"\n> > are not options, these fields are related to \"git-config(1)\" which are\n> > initialized using \"git_fsck_config\" function. Create a static variable\n> > named \"fsck_configs\" in \"fsck.c\" which aims at handling configs. Thus we\n> > don't need to reply on the \"fsck_objects_options\" to set up the fsck\n> > error message severity.\n> \n> reply???\n> \n\nSorry, I often make mistake to type \"rely\" as \"reply\".\n\n> As configuration often is used to prepopulate options, I need a lot\n> stonger justification to split these into a different structure than\n> \"'config' is a noun that is different from a noun 'option'\". \n> \n> If we intend to have many \"option\" instances but what these two\n> members store will be the same across these \"option\" instances, for\n> example, that would be a lot better reason why we may want to\n> separate these two members out of it, but I have a suspicion that if\n> we were to use the \"union with tags\" approach, these two would\n> become members of the common part shared between \"objects' and\n> \"refs\", i.e. the overall data structure might look more like this:\n> \n\nActually, I feel really wired for this part. Let me elaborate on this.\n\"fsck.c::git_fsck_config()\" is used to set up the configs. It will\neventually call the \"fsck.c::fsck_set_msg_type_from_ids\" like the\nfollowing:\n\n void fsck_set_msg_type_from_ids(struct fsck_options *options,\n enum fsck_msg_id msg_id,\n enum fsck_msg_type msg_type)\n {\n if (!options->msg_type) {\n int i;\n enum fsck_msg_type *severity.\n ALLOC_ARRAY(severity, FSCK_MSG_MAX);\n for (i = 0; i < FSCK_MSG_MAX; i++)\n severity[i] = fsck_msg_type(i, options);\n options->msg_type = severity;\n }\n\n options->msg_type[msg_id] = msg_type;\n }\n\nIn the current codebase, the caller will simply create a \"fsck_options\"\nand setup the fsck error message severity. However, let's see\n\"builtin/fskc.c\", it creates the following two \"fsck_options\" and it\nonly uses\n\n static struct fsck_options fsck_walk_options = FSCK_OPTIONS_DEFAULT;\n static struct fsck_options fsck_obj_options = FSCK_OPTIONS_DEFAULT;\n\nHowever, the code only uses \"fsck_obj_options\" to setup the configs. So\nit makes me feel so strange. So I just want to make it separation. Maybe\na little wrong here.\n\n> \tstruct fsck_options {\n> \t\tenum fsck_msg_type *msg_type;\n> \t\tstruct oidset oid_skiplist;\n> \t\tenum fsck_what_check { O, R } tag;\n> \t\tunion {\n> \t\t\tstruct fsck_object_options o;\n> \t\t\tstruct fsck_ref_options r;\n> \t\t} u;\n> \t};\n> \n> by moving these two members out of fsck_object_options and moving\n> them to the shared part.\n> \n> I dunno. It is unclear what the real reason is for these two things\n> to be extracted out and made to appear totally independent from the\n> \"options\" thing to begin with, and I am not sure if I agree with the\n> reason when it is known.\n"},{"id":"497887","messageId":"ZoLHtmOKTfxMSxvw@ArchLinux","threadId":"61565","inReplyTo":"Zn2Ah3WDhtOmzrzn@ArchLinux","subject":"[GSoC][PATCH v6 00/11] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-01T15:13:58Z","receivedAt":"2024-07-01T15:14:04Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Hi All:\n\nThis version follows the Junio's advice. Instead of creating the\nfollowing data structure:\n\n\tstruct fsck_options {\n\t\tenum fsck_type {\n\t\t\tFSCK_OBJECTS,\n\t\t\tFSCK_REFS,\n\t\t\t...\n\t\t} t;\n\t\tunion {\n\t\t\tstruct fsck_objects_options objects;\n\t\t\tstruct fsck_refs_options refs;\n\t\t} u;\n\t};\n\nI simply use the combination idea where \"fsck_options\" will incorporate\n\"fsck_objects_options\" and \"fsck_refs_options\". Karthik has told me that\nI should balance the job I should does and the extensibility for future.\nSo I use the most clear way to do this. Also Junio has said:\n\n> I would have expected that adding ref-related new members that\n> object consistency checkers has no interest in to the fsck_options\n> structure would be sufficient for the purpose of this series.\n\nSo this patch makes the design simple here.\n\nThanks for every reviewer.\n\nJialuo.\n\nshejialuo (11):\n fsck: add \"fsck_objects_options\" to hold objects-related options\n fsck: rename \"skiplist\" to \"oid_skiplist\"\n fsck: add \"fsck_refs_options\" into \"fsck_options\"\n fsck: add a unified interface for reporting fsck messages\n fsck: add \"fsck_refs_options\" initialization macros\n refs: set up ref consistency check infrastructure\n builtin/refs: add verify subcommand\n builtin/fsck: add `git-refs verify` child process\n files-backend: add unified interface for refs scanning\n fsck: add ref name check for files backend\n fsck: add ref content check for files backend\n\n Documentation/fsck-msgids.txt | 12 ++\n Documentation/git-refs.txt | 13 ++\n builtin/fsck.c | 44 ++++--\n builtin/index-pack.c | 4 +-\n builtin/mktag.c | 3 +-\n builtin/refs.c | 44 ++++++\n builtin/unpack-objects.c | 4 +-\n fetch-pack.c | 10 +-\n fsck.c | 158 +++++++++++++++------\n fsck.h | 115 ++++++++++-----\n object-file.c | 13 +-\n refs.c | 7 +-\n refs.h | 8 ++\n refs/debug.c | 11 ++\n refs/files-backend.c | 255 +++++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 ++\n refs/refs-internal.h | 11 +-\n refs/reftable-backend.c | 8 ++\n t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++\n 19 files changed, 824 insertions(+), 115 deletions(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\nRange-diff against v5:\n 1: d83b5797cb < -: ---------- fsck: rename \"fsck_options\" to \"fsck_objects_options\"\n 2: 88000b092a < -: ---------- fsck: use \"fsck_configs\" to set up configs\n 3: 65093bae64 < -: ---------- fsck: abstract common options for reusing\n 4: 403750da30 < -: ---------- fsck: add \"fsck_refs_options\" struct\n 5: 8dea5654d5 < -: ---------- fsck: add a unified interface for reporting fsck messages\n -: ---------- > 1: d32ae41a4d fsck: add \"fsck_objects_options\" to hold objects-related options\n -: ---------- > 2: 589a7a6fac fsck: rename \"skiplist\" to \"oid_skiplist\"\n -: ---------- > 3: bab97d7c82 fsck: add \"fsck_refs_options\" into \"fsck_options\"\n -: ---------- > 4: 276da52a6b fsck: add a unified interface for reporting fsck messages\n 6: 8fb62b54c6 ! 5: e93940c50c fsck: add \"fsck_refs_options\" initialization macros\n @@ Commit message\n fsck: add \"fsck_refs_options\" initialization macros\n \n Add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\" macros to\n - create the \"fsck_refs_options\" easily. Add refs-specific \"error_func\"\n - callback \"fsck_refs_error_function\".\n + create the refs options easily. Add refs-specific \"error_func\" callback\n + \"fsck_refs_error_function\".\n \n \"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\n passed the oid, it will use \"oid_to_hex\" to get the corresponding hex\n @@ Commit message\n Signed-off-by: shejialuo \n \n ## fsck.c ##\n -@@ fsck.c: int fsck_error_function(struct fsck_objects_options *objects_options,\n +@@ fsck.c: int fsck_objects_error_function(struct fsck_options *o,\n \treturn 1;\n }\n \n -+int fsck_refs_error_function(struct fsck_objects_options *objects_options UNUSED,\n -+\t\t\t struct fsck_refs_options *refs_options,\n ++int fsck_refs_error_function(struct fsck_options *options UNUSED,\n +\t\t\t const struct object_id *oid,\n +\t\t\t enum object_type object_type UNUSED,\n +\t\t\t const char *checked_ref_name,\n @@ fsck.c: int fsck_error_function(struct fsck_objects_options *objects_options,\n +\n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n - \t\t struct fsck_objects_options *options,\n + \t\t struct fsck_options *options, const char *blob_type)\n \n ## fsck.h ##\n -@@ fsck.h: int fsck_error_cb_print_missing_gitmodules(struct fsck_objects_options *objects_\n - \t\t\t\t\t enum fsck_msg_type msg_type,\n - \t\t\t\t\t enum fsck_msg_id msg_id,\n - \t\t\t\t\t const char *message);\n -+int fsck_refs_error_function(struct fsck_objects_options *objects_options,\n -+\t\t\t struct fsck_refs_options *refs_options,\n +@@ fsck.h: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n + \t\t\t\t\t\t enum fsck_msg_type msg_type,\n + \t\t\t\t\t\t enum fsck_msg_id msg_id,\n + \t\t\t\t\t\t const char *message);\n ++int fsck_refs_error_function(struct fsck_options *options,\n +\t\t\t const struct object_id *oid,\n +\t\t\t enum object_type object_type,\n +\t\t\t const char *checked_ref_name,\n @@ fsck.h: int fsck_error_cb_print_missing_gitmodules(struct fsck_objects_options *\n +\t\t\t enum fsck_msg_id msg_id,\n +\t\t\t const char *message);\n \n - struct fsck_options {\n - \tfsck_error error_func;\n -@@ fsck.h: struct fsck_options {\n struct fsck_refs_options {\n - \tstruct fsck_options fsck_options;\n - };\n + \tunsigned verbose:1;\n +@@ fsck.h: struct fsck_options {\n + \t\t.gitattributes_done = OIDSET_INIT, \\\n + \t} \\\n + }\n +#define FSCK_REFS_OPTIONS_DEFAULT { \\\n -+\t.fsck_options = { \\\n -+\t\t.error_func = fsck_refs_error_function, \\\n -+\t}, \\\n ++\t.error_func = fsck_refs_error_function, \\\n +}\n +#define FSCK_REFS_OPTIONS_STRICT { \\\n -+\t.fsck_options = { \\\n -+\t\t.error_func = fsck_refs_error_function, \\\n -+\t\t.strict = 1, \\\n -+\t}, \\\n ++\t.strict = 1, \\\n ++\t.error_func = fsck_refs_error_function, \\\n +}\n \n - struct fsck_objects_options {\n - \tstruct fsck_options fsck_options;\n + /* descend in all linked child objects\n + * the return value is:\n 7: 2ecd1f5407 ! 6: 85aa953f6d refs: set up ref consistency check infrastructure\n @@ refs.c: int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n -+int refs_fsck(struct ref_store *refs, struct fsck_refs_options *o)\n ++int refs_fsck(struct ref_store *refs, struct fsck_options *o)\n +{\n +\treturn refs->be->fsck(refs, o);\n +}\n @@ refs.h\n \n #include \"commit.h\"\n \n -+struct fsck_refs_options;\n ++struct fsck_options;\n struct object_id;\n struct ref_store;\n struct repository;\n @@ refs.h: int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void\n + * reflogs are consistent, and non-zero otherwise. The errors will be\n + * written to stderr.\n + */\n -+int refs_fsck(struct ref_store *refs, struct fsck_refs_options *o);\n ++int refs_fsck(struct ref_store *refs, struct fsck_options *o);\n +\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n @@ refs/debug.c: static int debug_reflog_expire(struct ref_store *ref_store, const\n }\n \n +static int debug_fsck(struct ref_store *ref_store,\n -+\t\t struct fsck_refs_options *o)\n ++\t\t struct fsck_options *o)\n +{\n +\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n +\tint res = drefs->refs->be->fsck(drefs->refs, o);\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n }\n \n +static int files_fsck(struct ref_store *ref_store,\n -+\t\t struct fsck_refs_options *o)\n ++\t\t struct fsck_options *o)\n +{\n +\tint ret;\n +\tstruct files_ref_store *refs =\n @@ refs/packed-backend.c: static struct ref_iterator *packed_reflog_iterator_begin(\n }\n \n +static int packed_fsck(struct ref_store *ref_store,\n -+\t\t struct fsck_refs_options *o)\n ++\t\t struct fsck_options *o)\n +{\n +\treturn 0;\n +}\n @@ refs/refs-internal.h: typedef int read_raw_ref_fn(struct ref_store *ref_store, c\n \t\t\t\t struct strbuf *referent);\n \n +typedef int fsck_fn(struct ref_store *ref_store,\n -+\t\t struct fsck_refs_options *o);\n ++\t\t struct fsck_options *o);\n +\n struct ref_storage_be {\n \tconst char *name;\n @@ refs/reftable-backend.c: static int reftable_be_reflog_expire(struct ref_store *\n }\n \n +static int reftable_be_fsck(struct ref_store *ref_store,\n -+\t\t\t struct fsck_refs_options *o)\n ++\t\t\t struct fsck_options *o)\n +{\n +\treturn 0;\n +}\n 8: 534a3d00af ! 7: ce7adc7372 builtin/refs: add verify subcommand\n @@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c\n \n +static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n +{\n -+\tstruct fsck_refs_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n ++\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n +\tconst char * const verify_usage[] = {\n +\t\tREFS_VERIFY_USAGE,\n +\t\tNULL,\n @@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c\n +\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n +\t\tOPT_END(),\n +\t};\n ++\tint ret = 0;\n +\n +\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n +\tif (argc)\n +\t\tusage(_(\"too many arguments\"));\n +\n +\tif (verbose)\n -+\t\tfsck_refs_options.fsck_options.verbose = 1;\n ++\t\tfsck_refs_options.refs_options.verbose = 1;\n +\tif (strict)\n -+\t\tfsck_refs_options.fsck_options.strict = 1;\n ++\t\tfsck_refs_options.strict = 1;\n +\n -+\tgit_config(git_fsck_config, &fsck_refs_options.fsck_options);\n ++\tgit_config(git_fsck_config, &fsck_refs_options);\n +\tprepare_repo_settings(the_repository);\n +\n -+\treturn refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n ++\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n ++\n ++\t/*\n ++\t * Explicitly free the allocated array and \"oid_skiplist\"\n ++\t */\n ++\tfree(fsck_refs_options.msg_type);\n ++\toidset_clear(&fsck_refs_options.oid_skiplist);\n ++\treturn ret;\n +}\n +\n int cmd_refs(int argc, const char **argv, const char *prefix)\n 9: 054c8cea89 = 8: 035eafe10b builtin/fsck: add `git-refs verify` child process\n10: 7f87c6a26e ! 9: 9398bf3f0d files-backend: add unified interface for refs scanning\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n + * the whole directory. This function is used as the callback for each\n + * regular file or symlink in the directory.\n + */\n -+typedef int (*files_fsck_refs_fn)(struct fsck_refs_options *o,\n ++typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n +\t\t\t\t const char *gitdir,\n +\t\t\t\t const char *refs_check_dir,\n +\t\t\t\t struct dir_iterator *iter);\n +\n +static int files_fsck_refs_dir(struct ref_store *ref_store,\n -+\t\t\t struct fsck_refs_options *o,\n ++\t\t\t struct fsck_options *o,\n +\t\t\t const char *refs_check_dir,\n +\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n +{\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +\t\t\tcontinue;\n +\t\t} else if (S_ISREG(iter->st.st_mode) ||\n +\t\t\t S_ISLNK(iter->st.st_mode)) {\n -+\t\t\tif (o->fsck_options.verbose)\n ++\t\t\tif (o->refs_options.verbose)\n +\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n +\t\t\t\t\t refs_check_dir, iter->relative_path);\n +\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +}\n +\n +static int files_fsck_refs(struct ref_store *ref_store,\n -+\t\t\t struct fsck_refs_options *o)\n ++\t\t\t struct fsck_options *o)\n +{\n +\tint ret;\n +\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n +\t\tNULL\n +\t};\n +\n -+\tif (o->fsck_options.verbose)\n ++\tif (o->refs_options.verbose)\n +\t\tfprintf_ln(stderr, \"Checking references consistency\");\n +\n +\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +}\n +\n static int files_fsck(struct ref_store *ref_store,\n - \t\t struct fsck_refs_options *o)\n + \t\t struct fsck_options *o)\n {\n @@ refs/files-backend.c: static int files_fsck(struct ref_store *ref_store,\n \tstruct files_ref_store *refs =\n11: 7d78014e5f ! 10: 046773e35c fsck: add ref name check for files backend\n @@ fsck.h: enum fsck_msg_type {\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n \n ## refs/files-backend.c ##\n -@@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_refs_options *o,\n +@@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n -+static int files_fsck_refs_name(struct fsck_refs_options *o,\n ++static int files_fsck_refs_name(struct fsck_options *o,\n +\t\t\t\tconst char *gitdir UNUSED,\n +\t\t\t\tconst char *refs_check_dir,\n +\t\t\t\tstruct dir_iterator *iter)\n @@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_refs_options\n +}\n +\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n - \t\t\t struct fsck_refs_options *o,\n + \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n @@ refs/files-backend.c: static int files_fsck_refs(struct ref_store *ref_store,\n {\n12: 16209a73ed ! 11: c9b9599589 fsck: add ref content check for files backend\n @@ refs/files-backend.c: int parse_loose_ref_contents(const char *buf, struct objec\n \treturn 0;\n }\n \n -@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_refs_options *o,\n +@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n \treturn ret;\n }\n \n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_refs_options *\n + * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n + * be the relative path agaignst \"gitdir\".\n + */\n -+static int files_fsck_symref_target(struct fsck_refs_options *o,\n ++static int files_fsck_symref_target(struct fsck_options *o,\n +\t\t\t\t const char *refname,\n +\t\t\t\t const char *pointee_name,\n +\t\t\t\t const char *pointee_path)\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_refs_options *\n +\treturn ret;\n +}\n +\n -+static int files_fsck_refs_content(struct fsck_refs_options *o,\n ++static int files_fsck_refs_content(struct fsck_options *o,\n +\t\t\t\t const char *gitdir,\n +\t\t\t\t const char *refs_check_dir,\n +\t\t\t\t struct dir_iterator *iter)\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_refs_options *\n +}\n +\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n - \t\t\t struct fsck_refs_options *o,\n + \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n @@ refs/files-backend.c: static int files_fsck_refs(struct ref_store *ref_store,\n \tint ret;\n-- \n2.45.2\n\n"},{"id":"497888","messageId":"ZoLIqOB_X01zdTVi@ArchLinux","threadId":"61565","inReplyTo":"ZoLHtmOKTfxMSxvw@ArchLinux","subject":"[PATCH v6 01/11] fsck: add \"fsck_objects_options\" to hold objects-related options","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-01T15:18:00Z","receivedAt":"2024-07-01T15:18:05Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) focuses on object database consistency check. It relies\non the \"fsck_options\" to interact with fsck error levels. However,\n\"fsck_options\" aims at checking the object database which contains a lot\nof fields only related to object database.\n\nIn order to add ref operations, remove the options related to objects in\n\"fsck_options\" to \"fsck_objects_options\" and let the \"fsck_options\"\nincorporate \"fsck_objects_options\". Change the macros and influenced\ncode to adapt into the above design.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 12 +++++-----\n builtin/index-pack.c | 4 ++--\n builtin/mktag.c | 2 +-\n builtin/unpack-objects.c | 4 ++--\n fetch-pack.c | 10 ++++----\n fsck.c | 51 +++++++++++++++++++++------------------\n fsck.h | 52 ++++++++++++++++++++++++----------------\n object-file.c | 2 +-\n 8 files changed, 77 insertions(+), 60 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..13b64f723f 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -42,8 +42,8 @@ static int check_full = 1;\n static int connectivity_only;\n static int check_strict;\n static int keep_cache_objects;\n-static struct fsck_options fsck_walk_options = FSCK_OPTIONS_DEFAULT;\n-static struct fsck_options fsck_obj_options = FSCK_OPTIONS_DEFAULT;\n+static struct fsck_options fsck_walk_options = FSCK_OBJECTS_OPTIONS_DEFAULT;\n+static struct fsck_options fsck_obj_options = FSCK_OBJECTS_OPTIONS_DEFAULT;\n static int errors_found;\n static int write_lost_and_found;\n static int verbose;\n@@ -214,7 +214,7 @@ static int mark_used(struct object *obj, enum object_type type UNUSED,\n \n static void mark_unreachable_referents(const struct object_id *oid)\n {\n-\tstruct fsck_options options = FSCK_OPTIONS_DEFAULT;\n+\tstruct fsck_options options = FSCK_OBJECTS_OPTIONS_DEFAULT;\n \tstruct object *obj = lookup_object(the_repository, oid);\n \n \tif (!obj || !(obj->flags & HAS_OBJ))\n@@ -233,7 +233,7 @@ static void mark_unreachable_referents(const struct object_id *oid)\n \t\t\tobject_as_type(obj, type, 0);\n \t}\n \n-\toptions.walk = mark_used;\n+\toptions.objects_options.walk = mark_used;\n \tfsck_walk(obj, NULL, &options);\n \tif (obj->type == OBJ_TREE)\n \t\tfree_tree_buffer((struct tree *)obj);\n@@ -936,8 +936,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \targc = parse_options(argc, argv, prefix, fsck_opts, fsck_usage, 0);\n \n-\tfsck_walk_options.walk = mark_object;\n-\tfsck_obj_options.walk = mark_used;\n+\tfsck_walk_options.objects_options.walk = mark_object;\n+\tfsck_obj_options.objects_options.walk = mark_used;\n \tfsck_obj_options.error_func = fsck_error_func;\n \tif (check_strict)\n \t\tfsck_obj_options.strict = 1;\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 856428fef9..59eb8c0355 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -127,7 +127,7 @@ static int nr_threads;\n static int from_stdin;\n static int strict;\n static int do_fsck_object;\n-static struct fsck_options fsck_options = FSCK_OPTIONS_MISSING_GITMODULES;\n+static struct fsck_options fsck_options = FSCK_OBJECTS_OPTIONS_MISSING_GITMODULES;\n static int verbose;\n static const char *progress_title;\n static int show_resolving_progress;\n@@ -1746,7 +1746,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\tusage(index_pack_usage);\n \n \tdisable_replace_refs();\n-\tfsck_options.walk = mark_link;\n+\tfsck_options.objects_options.walk = mark_link;\n \n \treset_pack_idx_option(&opts);\n \topts.flags |= WRITE_REV;\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex 4767f1a97e..c6fbeb58d4 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -15,7 +15,7 @@ static char const * const builtin_mktag_usage[] = {\n };\n static int option_strict = 1;\n \n-static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n+static struct fsck_options fsck_options = FSCK_OBJECTS_OPTIONS_STRICT;\n \n static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n \t\t\t\t const struct object_id *oid UNUSED,\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex f1c85a00ae..53cff0a91c 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -25,7 +25,7 @@ static unsigned int offset, len;\n static off_t consumed_bytes;\n static off_t max_input_size;\n static git_hash_ctx ctx;\n-static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n+static struct fsck_options fsck_options = FSCK_OBJECTS_OPTIONS_STRICT;\n static struct progress *progress;\n \n /*\n@@ -239,7 +239,7 @@ static int check_object(struct object *obj, enum object_type type,\n \t\tdie(\"Whoops! Cannot find object '%s'\", oid_to_hex(&obj->oid));\n \tif (fsck_object(obj, obj_buf->buffer, obj_buf->size, &fsck_options))\n \t\tdie(\"fsck error in packed object\");\n-\tfsck_options.walk = check_object;\n+\tfsck_options.objects_options.walk = check_object;\n \tif (fsck_walk(obj, NULL, &fsck_options))\n \t\tdie(\"Error on reachable objects of %s\", oid_to_hex(&obj->oid));\n \twrite_cached_object(obj, obj_buf);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 42f48fbc31..adce10fb4c 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -46,7 +46,7 @@ static int server_supports_filtering;\n static int advertise_sid;\n static struct shallow_lock shallow_lock;\n static const char *alternate_shallow_file;\n-static struct fsck_options fsck_options = FSCK_OPTIONS_MISSING_GITMODULES;\n+static struct fsck_options fsck_options = FSCK_OBJECTS_OPTIONS_MISSING_GITMODULES;\n static struct strbuf fsck_msg_types = STRBUF_INIT;\n static struct string_list uri_protocols = STRING_LIST_INIT_DUP;\n \n@@ -1222,7 +1222,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t} else\n \t\talternate_shallow_file = NULL;\n \tif (get_pack(args, fd, pack_lockfiles, NULL, sought, nr_sought,\n-\t\t &fsck_options.gitmodules_found))\n+\t\t &fsck_options.objects_options.gitmodules_found))\n \t\tdie(_(\"git fetch-pack: fetch failed.\"));\n \tif (fsck_finish(&fsck_options))\n \t\tdie(\"fsck failed\");\n@@ -1782,7 +1782,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \n \t\t\tif (get_pack(args, fd, pack_lockfiles,\n \t\t\t\t packfile_uris.nr ? &index_pack_args : NULL,\n-\t\t\t\t sought, nr_sought, &fsck_options.gitmodules_found))\n+\t\t\t\t sought, nr_sought,\n+\t\t\t\t &fsck_options.objects_options.gitmodules_found))\n \t\t\t\tdie(_(\"git fetch-pack: fetch failed.\"));\n \t\t\tdo_check_stateless_delimiter(args->stateless_rpc, &reader);\n \n@@ -1825,7 +1826,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \n \t\tpackname[the_hash_algo->hexsz] = '\\0';\n \n-\t\tparse_gitmodules_oids(cmd.out, &fsck_options.gitmodules_found);\n+\t\tparse_gitmodules_oids(cmd.out,\n+\t\t\t\t &fsck_options.objects_options.gitmodules_found);\n \n \t\tclose(cmd.out);\n \ndiff --git a/fsck.c b/fsck.c\nindex e193930ae7..e9848f2678 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -259,20 +259,20 @@ static int report(struct fsck_options *options,\n \n void fsck_enable_object_names(struct fsck_options *options)\n {\n-\tif (!options->object_names)\n-\t\toptions->object_names = kh_init_oid_map();\n+\tif (!options->objects_options.object_names)\n+\t\toptions->objects_options.object_names = kh_init_oid_map();\n }\n \n const char *fsck_get_object_name(struct fsck_options *options,\n \t\t\t\t const struct object_id *oid)\n {\n \tkhiter_t pos;\n-\tif (!options->object_names)\n+\tif (!options->objects_options.object_names)\n \t\treturn NULL;\n-\tpos = kh_get_oid_map(options->object_names, *oid);\n-\tif (pos >= kh_end(options->object_names))\n+\tpos = kh_get_oid_map(options->objects_options.object_names, *oid);\n+\tif (pos >= kh_end(options->objects_options.object_names))\n \t\treturn NULL;\n-\treturn kh_value(options->object_names, pos);\n+\treturn kh_value(options->objects_options.object_names, pos);\n }\n \n void fsck_put_object_name(struct fsck_options *options,\n@@ -284,15 +284,16 @@ void fsck_put_object_name(struct fsck_options *options,\n \tkhiter_t pos;\n \tint hashret;\n \n-\tif (!options->object_names)\n+\tif (!options->objects_options.object_names)\n \t\treturn;\n \n-\tpos = kh_put_oid_map(options->object_names, *oid, &hashret);\n+\tpos = kh_put_oid_map(options->objects_options.object_names,\n+\t\t\t *oid, &hashret);\n \tif (!hashret)\n \t\treturn;\n \tva_start(ap, fmt);\n \tstrbuf_vaddf(&buf, fmt, ap);\n-\tkh_value(options->object_names, pos) = strbuf_detach(&buf, NULL);\n+\tkh_value(options->objects_options.object_names, pos) = strbuf_detach(&buf, NULL);\n \tva_end(ap);\n }\n \n@@ -342,14 +343,14 @@ static int fsck_walk_tree(struct tree *tree, void *data, struct fsck_options *op\n \t\t\tif (name && obj)\n \t\t\t\tfsck_put_object_name(options, &entry.oid, \"%s%s/\",\n \t\t\t\t\t\t name, entry.path);\n-\t\t\tresult = options->walk(obj, OBJ_TREE, data, options);\n+\t\t\tresult = options->objects_options.walk(obj, OBJ_TREE, data, options);\n \t\t}\n \t\telse if (S_ISREG(entry.mode) || S_ISLNK(entry.mode)) {\n \t\t\tobj = (struct object *)lookup_blob(the_repository, &entry.oid);\n \t\t\tif (name && obj)\n \t\t\t\tfsck_put_object_name(options, &entry.oid, \"%s%s\",\n \t\t\t\t\t\t name, entry.path);\n-\t\t\tresult = options->walk(obj, OBJ_BLOB, data, options);\n+\t\t\tresult = options->objects_options.walk(obj, OBJ_BLOB, data, options);\n \t\t}\n \t\telse {\n \t\t\tresult = error(\"in tree %s: entry %s has bad mode %.6o\",\n@@ -380,7 +381,7 @@ static int fsck_walk_commit(struct commit *commit, void *data, struct fsck_optio\n \t\tfsck_put_object_name(options, get_commit_tree_oid(commit),\n \t\t\t\t \"%s:\", name);\n \n-\tresult = options->walk((struct object *) repo_get_commit_tree(the_repository, commit),\n+\tresult = options->objects_options.walk((struct object *) repo_get_commit_tree(the_repository, commit),\n \t\t\t OBJ_TREE, data, options);\n \tif (result < 0)\n \t\treturn result;\n@@ -423,7 +424,7 @@ static int fsck_walk_commit(struct commit *commit, void *data, struct fsck_optio\n \t\t\telse\n \t\t\t\tfsck_put_object_name(options, oid, \"%s^\", name);\n \t\t}\n-\t\tresult = options->walk((struct object *)parents->item, OBJ_COMMIT, data, options);\n+\t\tresult = options->objects_options.walk((struct object *)parents->item, OBJ_COMMIT, data, options);\n \t\tif (result < 0)\n \t\t\treturn result;\n \t\tif (!res)\n@@ -441,7 +442,7 @@ static int fsck_walk_tag(struct tag *tag, void *data, struct fsck_options *optio\n \t\treturn -1;\n \tif (name)\n \t\tfsck_put_object_name(options, &tag->tagged->oid, \"%s\", name);\n-\treturn options->walk(tag->tagged, OBJ_ANY, data, options);\n+\treturn options->objects_options.walk(tag->tagged, OBJ_ANY, data, options);\n }\n \n int fsck_walk(struct object *obj, void *data, struct fsck_options *options)\n@@ -598,6 +599,7 @@ static int fsck_tree(const struct object_id *tree_oid,\n \tunsigned o_mode;\n \tconst char *o_name;\n \tstruct name_stack df_dup_candidates = { NULL };\n+\tstruct fsck_objects_options *objects_options = &options->objects_options;\n \n \tif (init_tree_desc_gently(&desc, tree_oid, buffer, size,\n \t\t\t\t TREE_DESC_RAW_MODES)) {\n@@ -628,7 +630,7 @@ static int fsck_tree(const struct object_id *tree_oid,\n \n \t\tif (is_hfs_dotgitmodules(name) || is_ntfs_dotgitmodules(name)) {\n \t\t\tif (!S_ISLNK(mode))\n-\t\t\t\toidset_insert(&options->gitmodules_found,\n+\t\t\t\toidset_insert(&objects_options->gitmodules_found,\n \t\t\t\t\t entry_oid);\n \t\t\telse\n \t\t\t\tretval += report(options,\n@@ -639,7 +641,7 @@ static int fsck_tree(const struct object_id *tree_oid,\n \n \t\tif (is_hfs_dotgitattributes(name) || is_ntfs_dotgitattributes(name)) {\n \t\t\tif (!S_ISLNK(mode))\n-\t\t\t\toidset_insert(&options->gitattributes_found,\n+\t\t\t\toidset_insert(&objects_options->gitattributes_found,\n \t\t\t\t\t entry_oid);\n \t\t\telse\n \t\t\t\tretval += report(options, tree_oid, OBJ_TREE,\n@@ -666,7 +668,7 @@ static int fsck_tree(const struct object_id *tree_oid,\n \t\t\t\thas_dotgit |= is_ntfs_dotgit(backslash);\n \t\t\t\tif (is_ntfs_dotgitmodules(backslash)) {\n \t\t\t\t\tif (!S_ISLNK(mode))\n-\t\t\t\t\t\toidset_insert(&options->gitmodules_found,\n+\t\t\t\t\t\toidset_insert(&objects_options->gitmodules_found,\n \t\t\t\t\t\t\t entry_oid);\n \t\t\t\t\telse\n \t\t\t\t\t\tretval += report(options, tree_oid, OBJ_TREE,\n@@ -1107,11 +1109,11 @@ static int fsck_blob(const struct object_id *oid, const char *buf,\n \tif (object_on_skiplist(options, oid))\n \t\treturn 0;\n \n-\tif (oidset_contains(&options->gitmodules_found, oid)) {\n+\tif (oidset_contains(&options->objects_options.gitmodules_found, oid)) {\n \t\tstruct config_options config_opts = { 0 };\n \t\tstruct fsck_gitmodules_data data;\n \n-\t\toidset_insert(&options->gitmodules_done, oid);\n+\t\toidset_insert(&options->objects_options.gitmodules_done, oid);\n \n \t\tif (!buf) {\n \t\t\t/*\n@@ -1137,10 +1139,10 @@ static int fsck_blob(const struct object_id *oid, const char *buf,\n \t\tret |= data.ret;\n \t}\n \n-\tif (oidset_contains(&options->gitattributes_found, oid)) {\n+\tif (oidset_contains(&options->objects_options.gitattributes_found, oid)) {\n \t\tconst char *ptr;\n \n-\t\toidset_insert(&options->gitattributes_done, oid);\n+\t\toidset_insert(&options->objects_options.gitattributes_done, oid);\n \n \t\tif (!buf || size > ATTR_MAX_FILE_SIZE) {\n \t\t\t/*\n@@ -1255,12 +1257,15 @@ static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \n int fsck_finish(struct fsck_options *options)\n {\n+\tstruct fsck_objects_options *objects_options = &options->objects_options;\n \tint ret = 0;\n \n-\tret |= fsck_blobs(&options->gitmodules_found, &options->gitmodules_done,\n+\tret |= fsck_blobs(&objects_options->gitmodules_found,\n+\t\t\t &objects_options->gitmodules_done,\n \t\t\t FSCK_MSG_GITMODULES_MISSING, FSCK_MSG_GITMODULES_BLOB,\n \t\t\t options, \".gitmodules\");\n-\tret |= fsck_blobs(&options->gitattributes_found, &options->gitattributes_done,\n+\tret |= fsck_blobs(&objects_options->gitattributes_found,\n+\t\t\t &objects_options->gitattributes_done,\n \t\t\t FSCK_MSG_GITATTRIBUTES_MISSING, FSCK_MSG_GITATTRIBUTES_BLOB,\n \t\t\t options, \".gitattributes\");\n \ndiff --git a/fsck.h b/fsck.h\nindex 6085a384f6..37b6f6676f 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -131,12 +131,8 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message);\n \n-struct fsck_options {\n+struct fsck_objects_options {\n \tfsck_walk_func walk;\n-\tfsck_error error_func;\n-\tunsigned strict:1;\n-\tenum fsck_msg_type *msg_type;\n-\tstruct oidset skiplist;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n \tstruct oidset gitattributes_found;\n@@ -144,29 +140,43 @@ struct fsck_options {\n \tkh_oid_map_t *object_names;\n };\n \n-#define FSCK_OPTIONS_DEFAULT { \\\n+struct fsck_options {\n+\tunsigned strict:1;\n+\tenum fsck_msg_type *msg_type;\n+\tstruct oidset skiplist;\n+\tfsck_error error_func;\n+\tstruct fsck_objects_options objects_options;\n+};\n+\n+#define FSCK_OBJECTS_OPTIONS_DEFAULT { \\\n \t.skiplist = OIDSET_INIT, \\\n-\t.gitmodules_found = OIDSET_INIT, \\\n-\t.gitmodules_done = OIDSET_INIT, \\\n-\t.gitattributes_found = OIDSET_INIT, \\\n-\t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function \\\n+\t.error_func = fsck_error_function, \\\n+\t.objects_options = { \\\n+\t\t.gitmodules_found = OIDSET_INIT, \\\n+\t\t.gitmodules_done = OIDSET_INIT, \\\n+\t\t.gitattributes_found = OIDSET_INIT, \\\n+\t\t.gitattributes_done = OIDSET_INIT, \\\n+\t} \\\n }\n-#define FSCK_OPTIONS_STRICT { \\\n+#define FSCK_OBJECTS_OPTIONS_STRICT { \\\n \t.strict = 1, \\\n-\t.gitmodules_found = OIDSET_INIT, \\\n-\t.gitmodules_done = OIDSET_INIT, \\\n-\t.gitattributes_found = OIDSET_INIT, \\\n-\t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_error_function, \\\n+\t.objects_options = { \\\n+\t\t.gitmodules_found = OIDSET_INIT, \\\n+\t\t.gitmodules_done = OIDSET_INIT, \\\n+\t\t.gitattributes_found = OIDSET_INIT, \\\n+\t\t.gitattributes_done = OIDSET_INIT, \\\n+\t} \\\n }\n-#define FSCK_OPTIONS_MISSING_GITMODULES { \\\n+#define FSCK_OBJECTS_OPTIONS_MISSING_GITMODULES { \\\n \t.strict = 1, \\\n-\t.gitmodules_found = OIDSET_INIT, \\\n-\t.gitmodules_done = OIDSET_INIT, \\\n-\t.gitattributes_found = OIDSET_INIT, \\\n-\t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n+\t.objects_options = { \\\n+\t\t.gitmodules_found = OIDSET_INIT, \\\n+\t\t.gitmodules_done = OIDSET_INIT, \\\n+\t\t.gitattributes_found = OIDSET_INIT, \\\n+\t\t.gitattributes_done = OIDSET_INIT, \\\n+\t} \\\n }\n \n /* descend in all linked child objects\ndiff --git a/object-file.c b/object-file.c\nindex d3cf4b8b2e..4e10dd9804 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2507,7 +2507,7 @@ static int index_mem(struct index_state *istate,\n \t\t}\n \t}\n \tif (flags & HASH_FORMAT_CHECK) {\n-\t\tstruct fsck_options opts = FSCK_OPTIONS_DEFAULT;\n+\t\tstruct fsck_options opts = FSCK_OBJECTS_OPTIONS_DEFAULT;\n \n \t\topts.strict = 1;\n \t\topts.error_func = hash_format_check_report;\n-- \n2.45.2\n\n"},{"id":"497889","messageId":"ZoLI6UU5utHX9CV5@ArchLinux","threadId":"61565","inReplyTo":"ZoLHtmOKTfxMSxvw@ArchLinux","subject":"[GSoC][PATCH v6 02/11] fsck: rename \"skiplist\" to \"oid_skiplist\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-01T15:19:05Z","receivedAt":"2024-07-01T15:19:09Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Because we introduce ref consistency check. The original \"skiplist\" is a\ncommon option which is set up during handling user configs. To avoid\ncausing ambiguity, rename \"skiplist\" to \"oid_skiplist\".\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 4 ++--\n fsck.h | 4 ++--\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex e9848f2678..04d0aa1766 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -203,7 +203,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n \t\tif (!strcmp(buf, \"skiplist\")) {\n \t\t\tif (equal == len)\n \t\t\t\tdie(\"skiplist requires a path\");\n-\t\t\toidset_parse_file(&options->skiplist, buf + equal + 1);\n+\t\t\toidset_parse_file(&options->oid_skiplist, buf + equal + 1);\n \t\t\tbuf += len + 1;\n \t\t\tcontinue;\n \t\t}\n@@ -220,7 +220,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n static int object_on_skiplist(struct fsck_options *opts,\n \t\t\t const struct object_id *oid)\n {\n-\treturn opts && oid && oidset_contains(&opts->skiplist, oid);\n+\treturn opts && oid && oidset_contains(&opts->oid_skiplist, oid);\n }\n \n __attribute__((format (printf, 5, 6)))\ndiff --git a/fsck.h b/fsck.h\nindex 37b6f6676f..0c581ef082 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -143,13 +143,13 @@ struct fsck_objects_options {\n struct fsck_options {\n \tunsigned strict:1;\n \tenum fsck_msg_type *msg_type;\n-\tstruct oidset skiplist;\n+\tstruct oidset oid_skiplist;\n \tfsck_error error_func;\n \tstruct fsck_objects_options objects_options;\n };\n \n #define FSCK_OBJECTS_OPTIONS_DEFAULT { \\\n-\t.skiplist = OIDSET_INIT, \\\n+\t.oid_skiplist = OIDSET_INIT, \\\n \t.error_func = fsck_error_function, \\\n \t.objects_options = { \\\n \t\t.gitmodules_found = OIDSET_INIT, \\\n-- \n2.45.2\n\n"},{"id":"497890","messageId":"ZoLI_ip6a6fVXFMz@ArchLinux","threadId":"61565","inReplyTo":"ZoLHtmOKTfxMSxvw@ArchLinux","subject":"[GSoC][PATCH v6 03/11] fsck: add \"fsck_refs_options\" into \"fsck_options\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-01T15:19:26Z","receivedAt":"2024-07-01T15:19:31Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce \"fsck_refs_options\" to represent the refs-related options.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.h | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/fsck.h b/fsck.h\nindex 0c581ef082..ff7281b410 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -131,6 +131,10 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message);\n \n+struct fsck_refs_options {\n+\tunsigned verbose:1;\n+};\n+\n struct fsck_objects_options {\n \tfsck_walk_func walk;\n \tstruct oidset gitmodules_found;\n@@ -146,6 +150,7 @@ struct fsck_options {\n \tstruct oidset oid_skiplist;\n \tfsck_error error_func;\n \tstruct fsck_objects_options objects_options;\n+\tstruct fsck_refs_options refs_options;\n };\n \n #define FSCK_OBJECTS_OPTIONS_DEFAULT { \\\n-- \n2.45.2\n\n"},{"id":"497891","messageId":"ZoLJDoY6BiHW6ul1@ArchLinux","threadId":"61565","inReplyTo":"ZoLHtmOKTfxMSxvw@ArchLinux","subject":"[GSoC][PATCH v6 04/11] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-01T15:19:42Z","receivedAt":"2024-07-01T15:19:48Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The static function \"report\" provided by \"fsck.c\" aims at reporting the\nproblems related to object database which cannot be reused for refs.\nIn order to provide a unified interface which can report either objects\nor refs, create a new function \"vfsck_report\" by adding\n\"checked_ref_name\" parameter following the \"report\" prototype. However,\ninstead of using \"...\", provide \"va_list\" to allow more flexibility.\n\nThe \"vfsck_report\" function will use \"error_func\" registered in\n\"fsck_options\" function to report customized messages. Change\n\"error_func\" prototype to align with the \"vfsck_report\".\n\nChange \"report\" function to make it use \"vfsck_report\" to report\nobjects-related messages. Add a new function called \"fsck_refs_report\"\nto use \"vfsck_report\" to report refs-related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 15 ++++-----\n builtin/mktag.c | 1 +\n fsck.c | 81 ++++++++++++++++++++++++++++++++++++-------------\n fsck.h | 40 +++++++++++++++---------\n object-file.c | 11 ++++---\n 5 files changed, 101 insertions(+), 47 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 13b64f723f..cdf2218dfd 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -89,12 +89,13 @@ static int objerror(struct object *obj, const char *err)\n \treturn -1;\n }\n \n-static int fsck_error_func(struct fsck_options *o UNUSED,\n-\t\t\t const struct object_id *oid,\n-\t\t\t enum object_type object_type,\n-\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t const char *message)\n+static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t enum object_type object_type,\n+\t\t\t\t const char *checked_ref_name UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \tswitch (msg_type) {\n \tcase FSCK_WARN:\n@@ -938,7 +939,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tfsck_walk_options.objects_options.walk = mark_object;\n \tfsck_obj_options.objects_options.walk = mark_used;\n-\tfsck_obj_options.error_func = fsck_error_func;\n+\tfsck_obj_options.error_func = fsck_objects_error_func;\n \tif (check_strict)\n \t\tfsck_obj_options.strict = 1;\n \ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex c6fbeb58d4..d8e4b91134 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -20,6 +20,7 @@ static struct fsck_options fsck_options = FSCK_OBJECTS_OPTIONS_STRICT;\n static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n \t\t\t\t const struct object_id *oid UNUSED,\n \t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const char *checked_ref_name UNUSED,\n \t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t\t const char *message)\ndiff --git a/fsck.c b/fsck.c\nindex 04d0aa1766..31ae8d57ce 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -223,12 +223,18 @@ static int object_on_skiplist(struct fsck_options *opts,\n \treturn opts && oid && oidset_contains(&opts->oid_skiplist, oid);\n }\n \n-__attribute__((format (printf, 5, 6)))\n-static int report(struct fsck_options *options,\n-\t\t const struct object_id *oid, enum object_type object_type,\n-\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+/*\n+ * Provide a unified interface for either fscking refs or objects.\n+ * It will get the current msg error type and call the error_func callback\n+ * which is registered in the \"fsck_options\" struct.\n+ */\n+static int vfsck_report(struct fsck_options *options,\n+\t\t\tconst struct object_id *oid,\n+\t\t\tenum object_type object_type,\n+\t\t\tconst char *checked_ref_name,\n+\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n {\n-\tva_list ap;\n+\tva_list ap_copy;\n \tstruct strbuf sb = STRBUF_INIT;\n \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n \tint result;\n@@ -247,9 +253,9 @@ static int report(struct fsck_options *options,\n \tprepare_msg_ids();\n \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n \n-\tva_start(ap, fmt);\n-\tstrbuf_vaddf(&sb, fmt, ap);\n-\tresult = options->error_func(options, oid, object_type,\n+\tva_copy(ap_copy, ap);\n+\tstrbuf_vaddf(&sb, fmt, ap_copy);\n+\tresult = options->error_func(options, oid, object_type, checked_ref_name,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n \tva_end(ap);\n@@ -257,6 +263,36 @@ static int report(struct fsck_options *options,\n \treturn result;\n }\n \n+__attribute__((format (printf, 5, 6)))\n+static int report(struct fsck_options *options,\n+\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = vfsck_report(options, oid, object_type, NULL,\n+\t\t\t msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n+\n+\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t const char *checked_ref_name,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = vfsck_report(options, oid, OBJ_NONE,\n+\t\t\t checked_ref_name, msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n void fsck_enable_object_names(struct fsck_options *options)\n {\n \tif (!options->objects_options.object_names)\n@@ -1199,12 +1235,13 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid,\n-\t\t\tenum object_type object_type UNUSED,\n-\t\t\tenum fsck_msg_type msg_type,\n-\t\t\tenum fsck_msg_id msg_id UNUSED,\n-\t\t\tconst char *message)\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tenum object_type object_type UNUSED,\n+\t\t\t\tconst char *checked_ref_name UNUSED,\n+\t\t\t\tenum fsck_msg_type msg_type,\n+\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n+\t\t\t\tconst char *message)\n {\n \tif (msg_type == FSCK_WARN) {\n \t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n@@ -1305,16 +1342,18 @@ int git_fsck_config(const char *var, const char *value,\n * Custom error callbacks that are used in more than one place.\n */\n \n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message)\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t const char *checked_ref_name,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message)\n {\n \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n+\treturn fsck_objects_error_function(o, oid, object_type, checked_ref_name,\n+\t\t\t\t\t msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex ff7281b410..e611b3b65b 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -114,22 +114,27 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n \t\t\t void *data, struct fsck_options *options);\n \n-/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n+/*\n+ * callback function for reporting errors when checking either objects or refs\n+ */\n typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t\t const char *checked_ref_name,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid, enum object_type object_type,\n-\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n-\t\t\tconst char *message);\n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message);\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\t\tconst char *checked_ref_name,\n+\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n+\t\t\t\tconst char *message);\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t const char *checked_ref_name,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message);\n \n struct fsck_refs_options {\n \tunsigned verbose:1;\n@@ -155,7 +160,7 @@ struct fsck_options {\n \n #define FSCK_OBJECTS_OPTIONS_DEFAULT { \\\n \t.oid_skiplist = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function, \\\n+\t.error_func = fsck_objects_error_function, \\\n \t.objects_options = { \\\n \t\t.gitmodules_found = OIDSET_INIT, \\\n \t\t.gitmodules_done = OIDSET_INIT, \\\n@@ -165,7 +170,7 @@ struct fsck_options {\n }\n #define FSCK_OBJECTS_OPTIONS_STRICT { \\\n \t.strict = 1, \\\n-\t.error_func = fsck_error_function, \\\n+\t.error_func = fsck_objects_error_function, \\\n \t.objects_options = { \\\n \t\t.gitmodules_found = OIDSET_INIT, \\\n \t\t.gitmodules_done = OIDSET_INIT, \\\n@@ -175,7 +180,7 @@ struct fsck_options {\n }\n #define FSCK_OBJECTS_OPTIONS_MISSING_GITMODULES { \\\n \t.strict = 1, \\\n-\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n+\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n \t.objects_options = { \\\n \t\t.gitmodules_found = OIDSET_INIT, \\\n \t\t.gitmodules_done = OIDSET_INIT, \\\n@@ -224,6 +229,13 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+__attribute__((format (printf, 5, 6)))\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t const char *checked_ref_name,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\ndiff --git a/object-file.c b/object-file.c\nindex 4e10dd9804..a681087625 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2473,11 +2473,12 @@ int repo_has_object_file(struct repository *r,\n * give more context.\n */\n static int hash_format_check_report(struct fsck_options *opts UNUSED,\n-\t\t\t\t const struct object_id *oid UNUSED,\n-\t\t\t\t enum object_type object_type UNUSED,\n-\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n-\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t\t const char *message)\n+\t\t\t\t const struct object_id *oid UNUSED,\n+\t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const char *ref_checked_name UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \terror(_(\"object fails fsck: %s\"), message);\n \treturn 1;\n-- \n2.45.2\n\n"},{"id":"497892","messageId":"ZoLJH2yILjSQzuE6@ArchLinux","threadId":"61565","inReplyTo":"ZoLHtmOKTfxMSxvw@ArchLinux","subject":"[GSoC][PATCH v6 05/11] fsck: add \"fsck_refs_options\" initialization macros","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-01T15:19:59Z","receivedAt":"2024-07-01T15:20:04Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\" macros to\ncreate the refs options easily. Add refs-specific \"error_func\" callback\n\"fsck_refs_error_function\".\n\n\"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\npassed the oid, it will use \"oid_to_hex\" to get the corresponding hex\nvalue to report to the user.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 22 ++++++++++++++++++++++\n fsck.h | 14 ++++++++++++++\n 2 files changed, 36 insertions(+)\n\ndiff --git a/fsck.c b/fsck.c\nindex 31ae8d57ce..ea00fb4790 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1251,6 +1251,28 @@ int fsck_objects_error_function(struct fsck_options *o,\n \treturn 1;\n }\n \n+int fsck_refs_error_function(struct fsck_options *options UNUSED,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type UNUSED,\n+\t\t\t const char *checked_ref_name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tstatic struct strbuf sb = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&sb, checked_ref_name);\n+\tif (oid)\n+\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n+\n+\tif (msg_type == FSCK_WARN) {\n+\t\twarning(\"%s: %s\", sb.buf, message);\n+\t\treturn 0;\n+\t}\n+\terror(\"%s: %s\", sb.buf, message);\n+\treturn 1;\n+}\n+\n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n \t\t struct fsck_options *options, const char *blob_type)\ndiff --git a/fsck.h b/fsck.h\nindex e611b3b65b..887fe2969f 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -135,6 +135,13 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message);\n+int fsck_refs_error_function(struct fsck_options *options,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type,\n+\t\t\t const char *checked_ref_name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n \n struct fsck_refs_options {\n \tunsigned verbose:1;\n@@ -188,6 +195,13 @@ struct fsck_options {\n \t\t.gitattributes_done = OIDSET_INIT, \\\n \t} \\\n }\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n+#define FSCK_REFS_OPTIONS_STRICT { \\\n+\t.strict = 1, \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n \n /* descend in all linked child objects\n * the return value is:\n-- \n2.45.2\n\n"},{"id":"497893","messageId":"ZoLJOmBjzKSq9L56@ArchLinux","threadId":"61565","inReplyTo":"ZoLHtmOKTfxMSxvw@ArchLinux","subject":"[GSoC][PATCH v6 06/11] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-01T15:20:26Z","receivedAt":"2024-07-01T15:20:30Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It's organized as the five parts:\n\n1. The name and the initialization interfaces.\n2. The ref transaction interfaces.\n3. The ref internal interfaces (pack, rename and copy).\n4. The ref filesystem interfaces.\n5. The reflog related interfaces.\n\nTo keep consistent with the git-fsck(1), add a new interface named\n\"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\ngrouped into any above five categories. Explicitly add blank line to\nmake it different from others.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs.c | 5 +++++\n refs.h | 8 ++++++++\n refs/debug.c | 11 +++++++++++\n refs/files-backend.c | 15 ++++++++++++++-\n refs/packed-backend.c | 8 ++++++++\n refs/refs-internal.h | 6 ++++++\n refs/reftable-backend.c | 8 ++++++++\n 7 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex dd0d9c360f..5f17de38e2 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -316,6 +316,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs, struct fsck_options *o)\n+{\n+\treturn refs->be->fsck(refs, o);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex df8cbf6124..946e929af9 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -3,6 +3,7 @@\n \n #include \"commit.h\"\n \n+struct fsck_options;\n struct object_id;\n struct ref_store;\n struct repository;\n@@ -547,6 +548,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_options *o);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..45e2e784a0 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tint res = drefs->refs->be->fsck(drefs->refs, o);\n+\ttrace_printf_key(&trace_refs, \"fsck: %d\\n\", res);\n+\treturn res;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 44c5c3b201..817813e723 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3406,6 +3406,17 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\treturn ret;\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_init,\n@@ -3432,5 +3443,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex c4c1e36aa2..ad3e8fb1d1 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1733,6 +1733,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_init,\n@@ -1760,4 +1766,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex fa975d69aa..a905e187cd 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_options;\n struct ref_transaction;\n \n /*\n@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_options *o);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -677,6 +681,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 7d872a32ac..7f07de8d45 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2292,6 +2292,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2319,4 +2325,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.45.2\n\n"},{"id":"497894","messageId":"ZoLJUOy7LLbWj3k0@ArchLinux","threadId":"61565","inReplyTo":"ZoLHtmOKTfxMSxvw@ArchLinux","subject":"[GSoC][PATCH v6 07/11] builtin/refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-01T15:20:48Z","receivedAt":"2024-07-01T15:20:52Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\ncheck the reference database consistency.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/git-refs.txt | 13 +++++++++++\n builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 57 insertions(+)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..1244a85b64 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+The following options are specific to 'git refs verify':\n+\n+--strict::\n+\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n+\tbe seen as ERROR. See linkgit:git-fsck[1].\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..34371573f7 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,54 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tunsigned int verbose = 0, strict = 0;\n+\tstruct option options[] = {\n+\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\tint ret = 0;\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"too many arguments\"));\n+\n+\tif (verbose)\n+\t\tfsck_refs_options.refs_options.verbose = 1;\n+\tif (strict)\n+\t\tfsck_refs_options.strict = 1;\n+\n+\tgit_config(git_fsck_config, &fsck_refs_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n+\n+\t/*\n+\t * Explicitly free the allocated array and \"oid_skiplist\"\n+\t */\n+\tfree(fsck_refs_options.msg_type);\n+\toidset_clear(&fsck_refs_options.oid_skiplist);\n+\treturn ret;\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \n-- \n2.45.2\n\n"},{"id":"497895","messageId":"ZoLJZ0HGOofz3-cl@ArchLinux","threadId":"61565","inReplyTo":"ZoLHtmOKTfxMSxvw@ArchLinux","subject":"[GSoC][PATCH v6 08/11] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-01T15:21:11Z","receivedAt":"2024-07-01T15:21:15Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new function \"fsck_refs\" that initializes and runs a child\nprocess to execute the \"git-refs verify\" command.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex cdf2218dfd..aea635ea00 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -897,6 +897,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \treturn res;\n }\n \n+static void fsck_refs(void)\n+{\n+\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n+\tchild_process_init(&refs_verify);\n+\trefs_verify.git_cmd = 1;\n+\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n+\tif (verbose)\n+\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n+\tif (check_strict)\n+\t\tstrvec_push(&refs_verify.args, \"--strict\");\n+\n+\tif (run_command(&refs_verify))\n+\t\terrors_found |= ERROR_REFS;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n@@ -1066,6 +1081,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n+\tfsck_refs();\n+\n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \n-- \n2.45.2\n\n"},{"id":"497896","messageId":"ZoLJeIhdxHx1O7uA@ArchLinux","threadId":"61565","inReplyTo":"ZoLHtmOKTfxMSxvw@ArchLinux","subject":"[GSoC][PATCH v6 09/11] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-01T15:21:28Z","receivedAt":"2024-07-01T15:21:32Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"For refs and reflogs, we need to scan its corresponding directories to\ncheck every regular file or symbolic link which shares the same pattern.\nIntroduce a unified interface for scanning directories for\nfiles-backend.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs/files-backend.c | 77 +++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 76 insertions(+), 1 deletion(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 817813e723..744ee5c5db 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -4,6 +4,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -3406,6 +3407,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n+{\n+\tconst char *gitdir = ref_store->gitdir;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\n+\tif (!iter) {\n+\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n+\t\tgoto out;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->refs_options.verbose)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n+\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n+\t\t\t\t iter->basename);\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n+\n+out:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tNULL\n+\t};\n+\n+\tif (o->refs_options.verbose)\n+\t\tfprintf_ln(stderr, \"Checking references consistency\");\n+\n+\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n+\n+\treturn ret;\n+}\n+\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_options *o)\n {\n@@ -3413,7 +3486,9 @@ static int files_fsck(struct ref_store *ref_store,\n \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n-\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o)\n+\t | files_fsck_refs(ref_store, o);\n+\n \treturn ret;\n }\n \n-- \n2.45.2\n\n"},{"id":"497897","messageId":"ZoLJiWPK58YDRrQf@ArchLinux","threadId":"61565","inReplyTo":"ZoLHtmOKTfxMSxvw@ArchLinux","subject":"[GSoC][PATCH v6 10/11] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-01T15:21:45Z","receivedAt":"2024-07-01T15:21:50Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) only implicitly checks the reference, it does not fully\ncheck refs with bad format name such as standalone \"@\" and name ending\nwith \".lock\".\n\nIn order to provide such checks, add a new fsck message id \"badRefName\"\nwith default ERROR type. Use existing \"check_refname_format\" to explicit\ncheck the ref name. And add a new unit test to verify the functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 +\n fsck.h | 1 +\n refs/files-backend.c | 20 +++++++\n t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++\n 4 files changed, 125 insertions(+)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex f643585a34..dab4012246 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,9 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefName`::\n+\t(ERROR) A ref has a bad name.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex 887fe2969f..fb035d5457 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 744ee5c5db..3badec62ce 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3417,6 +3417,25 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n+static int files_fsck_refs_name(struct fsck_options *o,\n+\t\t\t\tconst char *gitdir UNUSED,\n+\t\t\t\tconst char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\tret = fsck_refs_report(o, NULL, sb.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3468,6 +3487,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n {\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tfiles_fsck_refs_name,\n \t\tNULL\n \t};\n \ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..b2db58d2c6\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit tag multi_hierarchy/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1.lock: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/@ &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\tgit -c fsck.badRefName=warn fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\tgit -c fsck.badRefName=ignore fsck 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+test_done\n-- \n2.45.2\n\n"},{"id":"497898","messageId":"ZoLJmNZhxB6LsE8B@ArchLinux","threadId":"61565","inReplyTo":"ZoLHtmOKTfxMSxvw@ArchLinux","subject":"[GSoC][PATCH v6 11/11] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-01T15:22:00Z","receivedAt":"2024-07-01T15:22:04Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Enhance the git-fsck(1) command by adding a check for reference content\nin the files backend. The new functionality ensures that symrefs, real\nsymbolic link and regular refs are validated correctly.\n\nIn order to check the trailing content of the regular refs, add a new\nparameter `trailing` to `parse_loose_ref_contents`.\n\nFor symrefs, `parse_loose_ref_contents` will set the \"referent\".\nHowever, symbolic link could be either absolute or relative. Use\n\"strbuf_add_real_path\" to read the symbolic link and convert the\nrelative path to absolute path. Then use \"skip_prefix\" to make it align\nwith symref \"referent\".\n\nThus, the symrefs and symbolic links could share the same interface. Add\na new function \"files_fsck_symref_target\" which aims at checking the\nfollowing things:\n\n1. whether the pointee is under the `refs/` directory.\n2. whether the pointee name is correct.\n3. whether the pointee path is a wrong type in filesystem.\n\nLast, add the following FSCK MESSAGEs:\n\n1. \"badRefContent(ERROR)\": A ref has a bad content\n2. \"badSymrefPointee(ERROR)\": The pointee of a symref is bad.\n3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 9 +++\n fsck.h | 3 +\n refs.c | 2 +-\n refs/files-backend.c | 145 +++++++++++++++++++++++++++++++++-\n refs/refs-internal.h | 5 +-\n t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++\n 6 files changed, 269 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex dab4012246..b1630a478b 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,9 +19,15 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefContent`::\n+\t(ERROR) A ref has a bad content.\n+\n `badRefName`::\n \t(ERROR) A ref has a bad name.\n \n+`badSymrefPointee`::\n+\t(ERROR) The pointee of a symref is bad.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \n@@ -167,6 +173,9 @@\n `nullSha1`::\n \t(WARN) Tree contains entries pointing to a null sha1.\n \n+`trailingRefContent`::\n+\t(WARN) A ref content has trailing contents.\n+\n `treeNotSorted`::\n \t(ERROR) A tree is not properly sorted.\n \ndiff --git a/fsck.h b/fsck.h\nindex fb035d5457..e917b221f6 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -32,6 +32,8 @@ enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n+\tFUNC(BAD_REF_CONTENT, ERROR) \\\n+\tFUNC(BAD_SYMREF_POINTEE, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n@@ -72,6 +74,7 @@ enum fsck_msg_type {\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n+\tFUNC(TRAILING_REF_CONTENT, WARN) \\\n \tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n \tFUNC(LARGE_PATHNAME, WARN) \\\ndiff --git a/refs.c b/refs.c\nindex 5f17de38e2..4e94f92997 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1758,7 +1758,7 @@ static int refs_read_special_head(struct ref_store *ref_store,\n \t}\n \n \tresult = parse_loose_ref_contents(content.buf, oid, referent, type,\n-\t\t\t\t\t failure_errno);\n+\t\t\t\t\t failure_errno, NULL);\n \n done:\n \tstrbuf_release(&full_path);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 3badec62ce..21a9a80d39 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1,4 +1,5 @@\n #include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n #include \"../copy.h\"\n #include \"../environment.h\"\n #include \"../gettext.h\"\n@@ -551,7 +552,7 @@ static int read_ref_internal(struct ref_store *ref_store, const char *refname,\n \tstrbuf_rtrim(&sb_contents);\n \tbuf = sb_contents.buf;\n \n-\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr);\n+\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr, NULL);\n \n out:\n \tif (ret && !myerr)\n@@ -587,7 +588,7 @@ static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refn\n \n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno)\n+\t\t\t int *failure_errno, const char **trailing)\n {\n \tconst char *p;\n \tif (skip_prefix(buf, \"ref:\", &buf)) {\n@@ -609,6 +610,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t*failure_errno = EINVAL;\n \t\treturn -1;\n \t}\n+\n+\tif (trailing)\n+\t\t*trailing = p;\n+\n \treturn 0;\n }\n \n@@ -3436,6 +3441,141 @@ static int files_fsck_refs_name(struct fsck_options *o,\n \treturn ret;\n }\n \n+/*\n+ * Check the symref \"pointee_name\" and \"pointee_path\". The caller should\n+ * make sure that \"pointee_path\" is absolute. For symbolic ref, \"pointee_name\"\n+ * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n+ * be the relative path agaignst \"gitdir\".\n+ */\n+static int files_fsck_symref_target(struct fsck_options *o,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const char *pointee_name,\n+\t\t\t\t const char *pointee_path)\n+{\n+\tconst char *p = NULL;\n+\tstruct stat st;\n+\tint ret = 0;\n+\n+\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n+\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to target out of refs hierarchy\");\n+\t\tgoto out;\n+\t}\n+\n+\tif (check_refname_format(pointee_name, 0)) {\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid refname\");\n+\t}\n+\n+\tif (lstat(pointee_path, &st) < 0)\n+\t\tgoto out;\n+\n+\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid target\");\n+\t\tgoto out;\n+\t}\n+out:\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs_content(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter)\n+{\n+\tstruct strbuf pointee_path = STRBUF_INIT,\n+\t\t ref_content = STRBUF_INIT,\n+\t\t abs_gitdir = STRBUF_INIT,\n+\t\t referent = STRBUF_INIT,\n+\t\t refname = STRBUF_INIT;\n+\tconst char *trailing = NULL;\n+\tint failure_errno = 0;\n+\tunsigned int type = 0;\n+\tstruct object_id oid;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\n+\t/*\n+\t * If the file is a symlink, we need to only check the connectivity\n+\t * of the destination object.\n+\t */\n+\tif (S_ISLNK(iter->st.st_mode)) {\n+\t\tconst char *pointee_name = NULL;\n+\n+\t\tstrbuf_add_real_path(&pointee_path, iter->path.buf);\n+\n+\t\tstrbuf_add_absolute_path(&abs_gitdir, gitdir);\n+\t\tstrbuf_normalize_path(&abs_gitdir);\n+\t\tif (!is_dir_sep(abs_gitdir.buf[abs_gitdir.len - 1]))\n+\t\t\tstrbuf_addch(&abs_gitdir, '/');\n+\n+\t\tif (!skip_prefix(pointee_path.buf,\n+\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n+\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t\t \"point to target outside gitdir\");\n+\t\t\tgoto clean;\n+\t\t}\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, pointee_name,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n+\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n+\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n+\t\t\t\t &referent, &type,\n+\t\t\t\t &failure_errno, &trailing)) {\n+\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n+\t\t\t\t \"invalid ref content\");\n+\t\tgoto clean;\n+\t}\n+\n+\t/*\n+\t * If the ref is a symref, we need to check the destination name and\n+\t * connectivity.\n+\t */\n+\tif (referent.len && (type & REF_ISSYMREF)) {\n+\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n+\t\tstrbuf_rtrim(&referent);\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, referent.buf,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t} else {\n+\t\t/*\n+\t\t * Only regular refs could have a trailing garbage. Should\n+\t\t * be reported as a warning.\n+\t\t */\n+\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n+\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n+\t\t\t\t\t \"trailing garbage in ref\");\n+\t\t\tgoto clean;\n+\t\t}\n+\t}\n+\n+clean:\n+\tstrbuf_release(&abs_gitdir);\n+\tstrbuf_release(&pointee_path);\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&ref_content);\n+\tstrbuf_release(&referent);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3488,6 +3628,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n \t\tfiles_fsck_refs_name,\n+\t\tfiles_fsck_refs_content,\n \t\tNULL\n \t};\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex a905e187cd..2fabf41d14 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -709,11 +709,12 @@ struct ref_store {\n \n /*\n * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for\n- * invalid contents.\n+ * invalid contents. Also *trailing is set to the first character after the\n+ * refname or NULL if the referent is not empty.\n */\n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno);\n+\t\t\t int *failure_errno, const char **trailing);\n \n /*\n * Fill in the generic part of refs and add it to our collection of\ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nindex b2db58d2c6..35bf40ee64 100755\n--- a/t/t0602-reffiles-fsck.sh\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -98,4 +98,114 @@ test_expect_success 'ref name check should be adapted into fsck messages' '\n \t)\n '\n \n+test_expect_success 'regular ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit checkout -b a/b/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse branch-1)\" > $branch_dir_prefix/branch-1-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse tag-1)\" > $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_must_fail git -c fsck.trailingRefContent=error fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s \" \"$(git rev-parse tag-2)\" > $tag_dir_prefix/tag-2-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/tags/tag-2-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/a/b/branch-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'symbolic ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid refname\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid target\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: logs/maint-v2.45\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to target out of refs hierarchy\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.2\n\n"},{"id":"497968","messageId":"CAOLa=ZSFTcMn+CsEcE1G5hnW37rw81zcUaXBywVY7xbBN7mhkA@mail.gmail.com","threadId":"61565","inReplyTo":"ZoLHtmOKTfxMSxvw@ArchLinux","subject":"Re: [GSoC][PATCH v6 00/11] ref consistency check infra setup","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-07-02T10:33:36Z","receivedAt":"2024-07-02T10:33:38Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello,\n\nshejialuo writes:\n\n> Hi All:\n>\n> This version follows the Junio's advice. Instead of creating the\n> following data structure:\n>\n> \tstruct fsck_options {\n> \t\tenum fsck_type {\n> \t\t\tFSCK_OBJECTS,\n> \t\t\tFSCK_REFS,\n> \t\t\t...\n> \t\t} t;\n> \t\tunion {\n> \t\t\tstruct fsck_objects_options objects;\n> \t\t\tstruct fsck_refs_options refs;\n> \t\t} u;\n> \t};\n>\n> I simply use the combination idea where \"fsck_options\" will incorporate\n> \"fsck_objects_options\" and \"fsck_refs_options\". Karthik has told me that\n> I should balance the job I should does and the extensibility for future.\n> So I use the most clear way to do this. Also Junio has said:\n>\n\nIf I understood Junio's comments correctly, he was drawing out the point\nabout if we even need the separation of options for refs. Since the only\noption we're adding is a verbose:\n\n struct fsck_refs_options {\n \tunsigned verbose:1;\n };\n\nwouldn't it be better if we simply amended `fsck_options` as so:\n\n diff --git a/fsck.h b/fsck.h\n index 6085a384f6..ea97f48acc 100644\n --- a/fsck.h\n +++ b/fsck.h\n @@ -135,6 +135,7 @@ struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n \tunsigned strict:1;\n +\tunsigned verbose_refs:1;\n \tenum fsck_msg_type *msg_type;\n \tstruct oidset skiplist;\n \tstruct oidset gitmodules_found;\n\nYour approach seems to take a different path though, where we create a\nnew route of creating two new structs, one for refs and another for\nobjects and adding both to fsck_objects. If we're doing this, wouldn't\nit be better to use the enum+union idea, like Junio mentioned? That way\nwe would have clarity around which type it represents.\n\n[snip]\n"},{"id":"497970","messageId":"ZoPvTKU4A73PXqf6@ArchLinux","threadId":"61565","inReplyTo":"CAOLa=ZSFTcMn+CsEcE1G5hnW37rw81zcUaXBywVY7xbBN7mhkA@mail.gmail.com","subject":"Re: [GSoC][PATCH v6 00/11] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-02T12:15:08Z","receivedAt":"2024-07-02T12:15:10Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jul 02, 2024 at 10:33:36AM +0000, Karthik Nayak wrote:\n> Hello,\n> \n> shejialuo writes:\n> \n> > Hi All:\n> >\n> > This version follows the Junio's advice. Instead of creating the\n> > following data structure:\n> >\n> > \tstruct fsck_options {\n> > \t\tenum fsck_type {\n> > \t\t\tFSCK_OBJECTS,\n> > \t\t\tFSCK_REFS,\n> > \t\t\t...\n> > \t\t} t;\n> > \t\tunion {\n> > \t\t\tstruct fsck_objects_options objects;\n> > \t\t\tstruct fsck_refs_options refs;\n> > \t\t} u;\n> > \t};\n> >\n> > I simply use the combination idea where \"fsck_options\" will incorporate\n> > \"fsck_objects_options\" and \"fsck_refs_options\". Karthik has told me that\n> > I should balance the job I should does and the extensibility for future.\n> > So I use the most clear way to do this. Also Junio has said:\n> >\n> \n> If I understood Junio's comments correctly, he was drawing out the point\n> about if we even need the separation of options for refs. Since the only\n> option we're adding is a verbose:\n> \n> struct fsck_refs_options {\n> \tunsigned verbose:1;\n> };\n> \n> wouldn't it be better if we simply amended `fsck_options` as so:\n> \n> diff --git a/fsck.h b/fsck.h\n> index 6085a384f6..ea97f48acc 100644\n> --- a/fsck.h\n> +++ b/fsck.h\n> @@ -135,6 +135,7 @@ struct fsck_options {\n> \tfsck_walk_func walk;\n> \tfsck_error error_func;\n> \tunsigned strict:1;\n> +\tunsigned verbose_refs:1;\n> \tenum fsck_msg_type *msg_type;\n> \tstruct oidset skiplist;\n> \tstruct oidset gitmodules_found;\n> \n> Your approach seems to take a different path though, where we create a\n> new route of creating two new structs, one for refs and another for\n> objects and adding both to fsck_objects. If we're doing this, wouldn't\n> it be better to use the enum+union idea, like Junio mentioned? That way\n> we would have clarity around which type it represents.\n> \n\nI agree. Let's give up breaking the structs. I will send a new version\nimmediately.\n\nThanks.\n\n> [snip]\n\n\n"},{"id":"497980","messageId":"xmqqplrvst02.fsf@gitster.g","threadId":"61565","inReplyTo":"CAOLa=ZSFTcMn+CsEcE1G5hnW37rw81zcUaXBywVY7xbBN7mhkA@mail.gmail.com","subject":"Re: [GSoC][PATCH v6 00/11] ref consistency check infra setup","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-02T15:39:25Z","receivedAt":"2024-07-02T15:39:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> If I understood Junio's comments correctly, he was drawing out the point\n> about if we even need the separation of options for refs. Since the only\n> option we're adding is a verbose:\n> ...\n> Your approach seems to take a different path though, where we create a\n> new route of creating two new structs, one for refs and another for\n> objects and adding both to fsck_objects. If we're doing this, wouldn't\n> it be better to use the enum+union idea, like Junio mentioned? That way\n> we would have clarity around which type it represents.\n\nYup. If we are going to over-engineer this, enum+union would be a\nreasonable way to do so, but we should ask if we need to split (and\nmore importantly, if we know the problem space well enough to make\nthe right split) in the first place. Just like premature optimization\nis bad, premature factoring and over-modularization is bad.\n\nThanks.\n\n"},{"id":"498019","messageId":"ZoVX6sn2C9VIeZ38@ArchLinux","threadId":"61565","inReplyTo":"ZoLHtmOKTfxMSxvw@ArchLinux","subject":"[GSoC][PATCH v7 0/9] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-03T13:53:46Z","receivedAt":"2024-07-03T13:53:48Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Hi All:\n\nWe have come to an agreement that there is no need to add complexity in\n\"fsck_options\". So this version just adds a new field into \"fsck_option\"\nfor refs. The other parts are just the same as the previous version.\n\nCI: https://github.com/shejialuo/git/pull/7\n\nThis patch is made upon the latest commit:\n\n 06e570c0df (Sync with 'maint', 2024-07-02)\n\n> Just like premature optimization is bad, premature factoring and\n> over-modularization is bad.\n\nI have learned a lot from yesterday's email. I always think we should\nmodularize the code which made a lot of trouble in this series. However,\nthe range-diff below may contain some commits from others. (Because this\npatch is built upon the latest, sorry for the inconvenience)\n\nThanks for every reviewer.\n\nshejialuo (9):\n fsck: rename \"skiplist\" to \"oid_skiplist\"\n fsck: add a unified interface for reporting fsck messages\n fsck: add refs-related options and error report function\n refs: set up ref consistency check infrastructure\n builtin/refs: add verify subcommand\n builtin/fsck: add `git-refs verify` child process\n files-backend: add unified interface for refs scanning\n fsck: add ref name check for files backend\n fsck: add ref content check for files backend\n\n Documentation/fsck-msgids.txt | 12 ++\n Documentation/git-refs.txt | 13 ++\n builtin/fsck.c | 32 ++++-\n builtin/mktag.c | 1 +\n builtin/refs.c | 44 ++++++\n fsck.c | 107 +++++++++++---\n fsck.h | 63 ++++++---\n object-file.c | 11 +-\n refs.c | 7 +-\n refs.h | 8 ++\n refs/debug.c | 11 ++\n refs/files-backend.c | 255 +++++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 ++\n refs/refs-internal.h | 11 +-\n refs/reftable-backend.c | 8 ++\n t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++\n 16 files changed, 745 insertions(+), 57 deletions(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\nRange-diff against v6:\n 1: d32ae41a4d < -: ---------- fsck: add \"fsck_objects_options\" to hold objects-related options\n 2: 589a7a6fac < -: ---------- fsck: rename \"skiplist\" to \"oid_skiplist\"\n 3: bab97d7c82 < -: ---------- fsck: add \"fsck_refs_options\" into \"fsck_options\"\n -: ---------- > 1: f5fcf36132 fsck: rename \"skiplist\" to \"oid_skiplist\"\n 4: 276da52a6b ! 2: c81b5b6fba fsck: add a unified interface for reporting fsck messages\n @@ builtin/fsck.c: static int objerror(struct object *obj, const char *err)\n \tcase FSCK_WARN:\n @@ builtin/fsck.c: int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n - \tfsck_walk_options.objects_options.walk = mark_object;\n - \tfsck_obj_options.objects_options.walk = mark_used;\n + \tfsck_walk_options.walk = mark_object;\n + \tfsck_obj_options.walk = mark_used;\n -\tfsck_obj_options.error_func = fsck_error_func;\n +\tfsck_obj_options.error_func = fsck_objects_error_func;\n \tif (check_strict)\n @@ builtin/fsck.c: int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \n ## builtin/mktag.c ##\n -@@ builtin/mktag.c: static struct fsck_options fsck_options = FSCK_OBJECTS_OPTIONS_STRICT;\n +@@ builtin/mktag.c: static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n \t\t\t\t const struct object_id *oid UNUSED,\n \t\t\t\t enum object_type object_type UNUSED,\n @@ fsck.c: static int report(struct fsck_options *options,\n +\n void fsck_enable_object_names(struct fsck_options *options)\n {\n - \tif (!options->objects_options.object_names)\n + \tif (!options->object_names)\n @@ fsck.c: int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n @@ fsck.h: int is_valid_msg_type(const char *msg_id, const char *msg_type);\n +\t\t\t\t\t\t enum fsck_msg_id msg_id,\n +\t\t\t\t\t\t const char *message);\n \n - struct fsck_refs_options {\n - \tunsigned verbose:1;\n + struct fsck_options {\n + \tfsck_walk_func walk;\n @@ fsck.h: struct fsck_options {\n + };\n \n - #define FSCK_OBJECTS_OPTIONS_DEFAULT { \\\n - \t.oid_skiplist = OIDSET_INIT, \\\n --\t.error_func = fsck_error_function, \\\n -+\t.error_func = fsck_objects_error_function, \\\n - \t.objects_options = { \\\n - \t\t.gitmodules_found = OIDSET_INIT, \\\n - \t\t.gitmodules_done = OIDSET_INIT, \\\n -@@ fsck.h: struct fsck_options {\n + #define FSCK_OPTIONS_DEFAULT { \\\n +-\t.skiplist = OIDSET_INIT, \\\n ++\t.oid_skiplist = OIDSET_INIT, \\\n + \t.gitmodules_found = OIDSET_INIT, \\\n + \t.gitmodules_done = OIDSET_INIT, \\\n + \t.gitattributes_found = OIDSET_INIT, \\\n + \t.gitattributes_done = OIDSET_INIT, \\\n +-\t.error_func = fsck_error_function \\\n ++\t.error_func = fsck_objects_error_function \\\n }\n - #define FSCK_OBJECTS_OPTIONS_STRICT { \\\n + #define FSCK_OPTIONS_STRICT { \\\n \t.strict = 1, \\\n +@@ fsck.h: struct fsck_options {\n + \t.gitmodules_done = OIDSET_INIT, \\\n + \t.gitattributes_found = OIDSET_INIT, \\\n + \t.gitattributes_done = OIDSET_INIT, \\\n -\t.error_func = fsck_error_function, \\\n +\t.error_func = fsck_objects_error_function, \\\n - \t.objects_options = { \\\n - \t\t.gitmodules_found = OIDSET_INIT, \\\n - \t\t.gitmodules_done = OIDSET_INIT, \\\n -@@ fsck.h: struct fsck_options {\n }\n - #define FSCK_OBJECTS_OPTIONS_MISSING_GITMODULES { \\\n + #define FSCK_OPTIONS_MISSING_GITMODULES { \\\n \t.strict = 1, \\\n +@@ fsck.h: struct fsck_options {\n + \t.gitmodules_done = OIDSET_INIT, \\\n + \t.gitattributes_found = OIDSET_INIT, \\\n + \t.gitattributes_done = OIDSET_INIT, \\\n -\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n +\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n - \t.objects_options = { \\\n - \t\t.gitmodules_found = OIDSET_INIT, \\\n - \t\t.gitmodules_done = OIDSET_INIT, \\\n + }\n + \n + /* descend in all linked child objects\n @@ fsck.h: int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n 5: e93940c50c ! 3: 53156dc847 fsck: add \"fsck_refs_options\" initialization macros\n @@ Metadata\n Author: shejialuo \n \n ## Commit message ##\n - fsck: add \"fsck_refs_options\" initialization macros\n + fsck: add refs-related options and error report function\n \n - Add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\" macros to\n - create the refs options easily. Add refs-specific \"error_func\" callback\n - \"fsck_refs_error_function\".\n + Add refs-related options to the \"fsck_options\", create refs-specific\n + \"error_func\" callback \"fsck_refs_error_function\".\n \n \"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\n - passed the oid, it will use \"oid_to_hex\" to get the corresponding hex\n - value to report to the user.\n + passes the oid, it will use \"oid_to_hex\" to get the corresponding hex\n + value to report to the caller.\n +\n + Last, add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\"\n + macros to create refs options easily.\n \n Mentored-by: Patrick Steinhardt \n Mentored-by: Karthik Nayak \n @@ fsck.h: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *\n +\t\t\t enum fsck_msg_id msg_id,\n +\t\t\t const char *message);\n \n - struct fsck_refs_options {\n - \tunsigned verbose:1;\n + struct fsck_options {\n + \tfsck_walk_func walk;\n + \tfsck_error error_func;\n + \tunsigned strict:1;\n ++\tunsigned verbose_refs:1;\n + \tenum fsck_msg_type *msg_type;\n + \tstruct oidset oid_skiplist;\n + \tstruct oidset gitmodules_found;\n @@ fsck.h: struct fsck_options {\n - \t\t.gitattributes_done = OIDSET_INIT, \\\n - \t} \\\n + \t.gitattributes_done = OIDSET_INIT, \\\n + \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n +#define FSCK_REFS_OPTIONS_DEFAULT { \\\n +\t.error_func = fsck_refs_error_function, \\\n 6: 85aa953f6d ! 4: 358f4a1be9 refs: set up ref consistency check infrastructure\n @@ refs.c: int check_refname_format(const char *refname, int flags)\n \n ## refs.h ##\n @@\n - \n #include \"commit.h\"\n + #include \"repository.h\"\n \n +struct fsck_options;\n struct object_id;\n struct ref_store;\n - struct repository;\n + struct strbuf;\n @@ refs.h: int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n 7: ce7adc7372 ! 5: e632859df6 builtin/refs: add verify subcommand\n @@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c\n +\t\tusage(_(\"too many arguments\"));\n +\n +\tif (verbose)\n -+\t\tfsck_refs_options.refs_options.verbose = 1;\n ++\t\tfsck_refs_options.verbose_refs = 1;\n +\tif (strict)\n +\t\tfsck_refs_options.strict = 1;\n +\n 8: 035eafe10b = 6: 5d7a6261ae builtin/fsck: add `git-refs verify` child process\n 9: 9398bf3f0d ! 7: 33311af1b1 files-backend: add unified interface for refs scanning\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +\t\t\tcontinue;\n +\t\t} else if (S_ISREG(iter->st.st_mode) ||\n +\t\t\t S_ISLNK(iter->st.st_mode)) {\n -+\t\t\tif (o->refs_options.verbose)\n ++\t\t\tif (o->verbose_refs)\n +\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n +\t\t\t\t\t refs_check_dir, iter->relative_path);\n +\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +\t\tNULL\n +\t};\n +\n -+\tif (o->refs_options.verbose)\n ++\tif (o->verbose_refs)\n +\t\tfprintf_ln(stderr, \"Checking references consistency\");\n +\n +\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n10: 046773e35c = 8: 02bd3ac952 fsck: add ref name check for files backend\n11: c9b9599589 ! 9: 94d123471d fsck: add ref content check for files backend\n @@ refs.c: static int refs_read_special_head(struct ref_store *ref_store,\n \n ## refs/files-backend.c ##\n @@\n + #define USE_THE_REPOSITORY_VARIABLE\n + \n #include \"../git-compat-util.h\"\n +#include \"../abspath.h\"\n #include \"../copy.h\"\n-- \n2.45.2\n\n"},{"id":"498020","messageId":"ZoVYjrfCFO0_K4Ry@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v7 1/9] fsck: rename \"skiplist\" to \"oid_skiplist\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-03T13:56:30Z","receivedAt":"2024-07-03T13:56:32Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Because we introduce ref consistency check. The original \"skiplist\" is a\ncommon option which is set up during handling user configs. To avoid\ncausing ambiguity, rename \"skiplist\" to \"oid_skiplist\".\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 4 ++--\n fsck.h | 2 +-\n 2 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex eea7145470..1960bfeba9 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n \t\tif (!strcmp(buf, \"skiplist\")) {\n \t\t\tif (equal == len)\n \t\t\t\tdie(\"skiplist requires a path\");\n-\t\t\toidset_parse_file(&options->skiplist, buf + equal + 1,\n+\t\t\toidset_parse_file(&options->oid_skiplist, buf + equal + 1,\n \t\t\t\t\t the_repository->hash_algo);\n \t\t\tbuf += len + 1;\n \t\t\tcontinue;\n@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n static int object_on_skiplist(struct fsck_options *opts,\n \t\t\t const struct object_id *oid)\n {\n-\treturn opts && oid && oidset_contains(&opts->skiplist, oid);\n+\treturn opts && oid && oidset_contains(&opts->oid_skiplist, oid);\n }\n \n __attribute__((format (printf, 5, 6)))\ndiff --git a/fsck.h b/fsck.h\nindex 6085a384f6..1ee3dd85ba 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -136,7 +136,7 @@ struct fsck_options {\n \tfsck_error error_func;\n \tunsigned strict:1;\n \tenum fsck_msg_type *msg_type;\n-\tstruct oidset skiplist;\n+\tstruct oidset oid_skiplist;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n \tstruct oidset gitattributes_found;\n-- \n2.45.2\n\n"},{"id":"498021","messageId":"ZoVYn6C-QOuHM2sC@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v7 2/9] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-03T13:56:47Z","receivedAt":"2024-07-03T13:56:48Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The static function \"report\" provided by \"fsck.c\" aims at reporting the\nproblems related to object database which cannot be reused for refs.\nIn order to provide a unified interface which can report either objects\nor refs, create a new function \"vfsck_report\" by adding\n\"checked_ref_name\" parameter following the \"report\" prototype. However,\ninstead of using \"...\", provide \"va_list\" to allow more flexibility.\n\nThe \"vfsck_report\" function will use \"error_func\" registered in\n\"fsck_options\" function to report customized messages. Change\n\"error_func\" prototype to align with the \"vfsck_report\".\n\nChange \"report\" function to make it use \"vfsck_report\" to report\nobjects-related messages. Add a new function called \"fsck_refs_report\"\nto use \"vfsck_report\" to report refs-related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 15 ++++-----\n builtin/mktag.c | 1 +\n fsck.c | 81 ++++++++++++++++++++++++++++++++++++-------------\n fsck.h | 42 ++++++++++++++++---------\n object-file.c | 11 ++++---\n 5 files changed, 102 insertions(+), 48 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..de34538c4f 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -89,12 +89,13 @@ static int objerror(struct object *obj, const char *err)\n \treturn -1;\n }\n \n-static int fsck_error_func(struct fsck_options *o UNUSED,\n-\t\t\t const struct object_id *oid,\n-\t\t\t enum object_type object_type,\n-\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t const char *message)\n+static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t enum object_type object_type,\n+\t\t\t\t const char *checked_ref_name UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \tswitch (msg_type) {\n \tcase FSCK_WARN:\n@@ -938,7 +939,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tfsck_walk_options.walk = mark_object;\n \tfsck_obj_options.walk = mark_used;\n-\tfsck_obj_options.error_func = fsck_error_func;\n+\tfsck_obj_options.error_func = fsck_objects_error_func;\n \tif (check_strict)\n \t\tfsck_obj_options.strict = 1;\n \ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex 4767f1a97e..42f945c584 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -20,6 +20,7 @@ static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n \t\t\t\t const struct object_id *oid UNUSED,\n \t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const char *checked_ref_name UNUSED,\n \t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t\t const char *message)\ndiff --git a/fsck.c b/fsck.c\nindex 1960bfeba9..7182ce8e80 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -226,12 +226,18 @@ static int object_on_skiplist(struct fsck_options *opts,\n \treturn opts && oid && oidset_contains(&opts->oid_skiplist, oid);\n }\n \n-__attribute__((format (printf, 5, 6)))\n-static int report(struct fsck_options *options,\n-\t\t const struct object_id *oid, enum object_type object_type,\n-\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+/*\n+ * Provide a unified interface for either fscking refs or objects.\n+ * It will get the current msg error type and call the error_func callback\n+ * which is registered in the \"fsck_options\" struct.\n+ */\n+static int vfsck_report(struct fsck_options *options,\n+\t\t\tconst struct object_id *oid,\n+\t\t\tenum object_type object_type,\n+\t\t\tconst char *checked_ref_name,\n+\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n {\n-\tva_list ap;\n+\tva_list ap_copy;\n \tstruct strbuf sb = STRBUF_INIT;\n \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n \tint result;\n@@ -250,9 +256,9 @@ static int report(struct fsck_options *options,\n \tprepare_msg_ids();\n \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n \n-\tva_start(ap, fmt);\n-\tstrbuf_vaddf(&sb, fmt, ap);\n-\tresult = options->error_func(options, oid, object_type,\n+\tva_copy(ap_copy, ap);\n+\tstrbuf_vaddf(&sb, fmt, ap_copy);\n+\tresult = options->error_func(options, oid, object_type, checked_ref_name,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n \tva_end(ap);\n@@ -260,6 +266,36 @@ static int report(struct fsck_options *options,\n \treturn result;\n }\n \n+__attribute__((format (printf, 5, 6)))\n+static int report(struct fsck_options *options,\n+\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = vfsck_report(options, oid, object_type, NULL,\n+\t\t\t msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n+\n+\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t const char *checked_ref_name,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = vfsck_report(options, oid, OBJ_NONE,\n+\t\t\t checked_ref_name, msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n void fsck_enable_object_names(struct fsck_options *options)\n {\n \tif (!options->object_names)\n@@ -1200,12 +1236,13 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid,\n-\t\t\tenum object_type object_type UNUSED,\n-\t\t\tenum fsck_msg_type msg_type,\n-\t\t\tenum fsck_msg_id msg_id UNUSED,\n-\t\t\tconst char *message)\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tenum object_type object_type UNUSED,\n+\t\t\t\tconst char *checked_ref_name UNUSED,\n+\t\t\t\tenum fsck_msg_type msg_type,\n+\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n+\t\t\t\tconst char *message)\n {\n \tif (msg_type == FSCK_WARN) {\n \t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n@@ -1303,16 +1340,18 @@ int git_fsck_config(const char *var, const char *value,\n * Custom error callbacks that are used in more than one place.\n */\n \n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message)\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t const char *checked_ref_name,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message)\n {\n \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n+\treturn fsck_objects_error_function(o, oid, object_type, checked_ref_name,\n+\t\t\t\t\t msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex 1ee3dd85ba..f703dfb5e8 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -114,22 +114,27 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n \t\t\t void *data, struct fsck_options *options);\n \n-/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n+/*\n+ * callback function for reporting errors when checking either objects or refs\n+ */\n typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t\t const char *checked_ref_name,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid, enum object_type object_type,\n-\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n-\t\t\tconst char *message);\n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message);\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\t\tconst char *checked_ref_name,\n+\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n+\t\t\t\tconst char *message);\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t const char *checked_ref_name,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message);\n \n struct fsck_options {\n \tfsck_walk_func walk;\n@@ -145,12 +150,12 @@ struct fsck_options {\n };\n \n #define FSCK_OPTIONS_DEFAULT { \\\n-\t.skiplist = OIDSET_INIT, \\\n+\t.oid_skiplist = OIDSET_INIT, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function \\\n+\t.error_func = fsck_objects_error_function \\\n }\n #define FSCK_OPTIONS_STRICT { \\\n \t.strict = 1, \\\n@@ -158,7 +163,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function, \\\n+\t.error_func = fsck_objects_error_function, \\\n }\n #define FSCK_OPTIONS_MISSING_GITMODULES { \\\n \t.strict = 1, \\\n@@ -166,7 +171,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n+\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n \n /* descend in all linked child objects\n@@ -209,6 +214,13 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+__attribute__((format (printf, 5, 6)))\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t const char *checked_ref_name,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\ndiff --git a/object-file.c b/object-file.c\nindex 065103be3e..d2c6427935 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2470,11 +2470,12 @@ int repo_has_object_file(struct repository *r,\n * give more context.\n */\n static int hash_format_check_report(struct fsck_options *opts UNUSED,\n-\t\t\t\t const struct object_id *oid UNUSED,\n-\t\t\t\t enum object_type object_type UNUSED,\n-\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n-\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t\t const char *message)\n+\t\t\t\t const struct object_id *oid UNUSED,\n+\t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const char *ref_checked_name UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \terror(_(\"object fails fsck: %s\"), message);\n \treturn 1;\n-- \n2.45.2\n\n"},{"id":"498022","messageId":"ZoVYro3NYjpz7Cmd@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v7 3/9] fsck: add refs-related options and error report function","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-03T13:57:02Z","receivedAt":"2024-07-03T13:57:03Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Add refs-related options to the \"fsck_options\", create refs-specific\n\"error_func\" callback \"fsck_refs_error_function\".\n\n\"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\npasses the oid, it will use \"oid_to_hex\" to get the corresponding hex\nvalue to report to the caller.\n\nLast, add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\"\nmacros to create refs options easily.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 22 ++++++++++++++++++++++\n fsck.h | 15 +++++++++++++++\n 2 files changed, 37 insertions(+)\n\ndiff --git a/fsck.c b/fsck.c\nindex 7182ce8e80..d1dcbdcac2 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1252,6 +1252,28 @@ int fsck_objects_error_function(struct fsck_options *o,\n \treturn 1;\n }\n \n+int fsck_refs_error_function(struct fsck_options *options UNUSED,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type UNUSED,\n+\t\t\t const char *checked_ref_name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tstatic struct strbuf sb = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&sb, checked_ref_name);\n+\tif (oid)\n+\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n+\n+\tif (msg_type == FSCK_WARN) {\n+\t\twarning(\"%s: %s\", sb.buf, message);\n+\t\treturn 0;\n+\t}\n+\terror(\"%s: %s\", sb.buf, message);\n+\treturn 1;\n+}\n+\n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n \t\t struct fsck_options *options, const char *blob_type)\ndiff --git a/fsck.h b/fsck.h\nindex f703dfb5e8..246055c0f9 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -135,11 +135,19 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message);\n+int fsck_refs_error_function(struct fsck_options *options,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type,\n+\t\t\t const char *checked_ref_name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n \n struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n \tunsigned strict:1;\n+\tunsigned verbose_refs:1;\n \tenum fsck_msg_type *msg_type;\n \tstruct oidset oid_skiplist;\n \tstruct oidset gitmodules_found;\n@@ -173,6 +181,13 @@ struct fsck_options {\n \t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n+#define FSCK_REFS_OPTIONS_STRICT { \\\n+\t.strict = 1, \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n \n /* descend in all linked child objects\n * the return value is:\n-- \n2.45.2\n\n"},{"id":"498023","messageId":"ZoVY31KvLduy25iX@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v7 4/9] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-03T13:57:51Z","receivedAt":"2024-07-03T13:57:52Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It's organized as the five parts:\n\n1. The name and the initialization interfaces.\n2. The ref transaction interfaces.\n3. The ref internal interfaces (pack, rename and copy).\n4. The ref filesystem interfaces.\n5. The reflog related interfaces.\n\nTo keep consistent with the git-fsck(1), add a new interface named\n\"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\ngrouped into any above five categories. Explicitly add blank line to\nmake it different from others.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs.c | 5 +++++\n refs.h | 8 ++++++++\n refs/debug.c | 11 +++++++++++\n refs/files-backend.c | 15 ++++++++++++++-\n refs/packed-backend.c | 8 ++++++++\n refs/refs-internal.h | 6 ++++++\n refs/reftable-backend.c | 8 ++++++++\n 7 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex bb90a18875..410919246b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs, struct fsck_options *o)\n+{\n+\treturn refs->be->fsck(refs, o);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex 0ecba21b4a..804d6a7fce 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -4,6 +4,7 @@\n #include \"commit.h\"\n #include \"repository.h\"\n \n+struct fsck_options;\n struct object_id;\n struct ref_store;\n struct strbuf;\n@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_options *o);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..45e2e784a0 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tint res = drefs->refs->be->fsck(drefs->refs, o);\n+\ttrace_printf_key(&trace_refs, \"fsck: %d\\n\", res);\n+\treturn res;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex aa52d9be7c..d89eeda8ef 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3408,6 +3408,17 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\treturn ret;\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_init,\n@@ -3434,5 +3445,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a0666407cd..5209b0b212 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_init,\n@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex fa975d69aa..a905e187cd 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_options;\n struct ref_transaction;\n \n /*\n@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_options *o);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -677,6 +681,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex fbe74c239d..b5a1a526df 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.45.2\n\n"},{"id":"498024","messageId":"ZoVY7lbtMQZzjm7x@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v7 5/9] builtin/refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-03T13:58:06Z","receivedAt":"2024-07-03T13:58:07Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\ncheck the reference database consistency.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/git-refs.txt | 13 +++++++++++\n builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 57 insertions(+)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..1244a85b64 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+The following options are specific to 'git refs verify':\n+\n+--strict::\n+\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n+\tbe seen as ERROR. See linkgit:git-fsck[1].\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..74720f5e0d 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,54 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tunsigned int verbose = 0, strict = 0;\n+\tstruct option options[] = {\n+\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\tint ret = 0;\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"too many arguments\"));\n+\n+\tif (verbose)\n+\t\tfsck_refs_options.verbose_refs = 1;\n+\tif (strict)\n+\t\tfsck_refs_options.strict = 1;\n+\n+\tgit_config(git_fsck_config, &fsck_refs_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n+\n+\t/*\n+\t * Explicitly free the allocated array and \"oid_skiplist\"\n+\t */\n+\tfree(fsck_refs_options.msg_type);\n+\toidset_clear(&fsck_refs_options.oid_skiplist);\n+\treturn ret;\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \n-- \n2.45.2\n\n"},{"id":"498025","messageId":"ZoVY-0NDEoD6A1HU@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v7 6/9] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-03T13:58:19Z","receivedAt":"2024-07-03T13:58:20Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new function \"fsck_refs\" that initializes and runs a child\nprocess to execute the \"git-refs verify\" command.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex de34538c4f..ec3357722c 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -897,6 +897,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \treturn res;\n }\n \n+static void fsck_refs(void)\n+{\n+\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n+\tchild_process_init(&refs_verify);\n+\trefs_verify.git_cmd = 1;\n+\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n+\tif (verbose)\n+\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n+\tif (check_strict)\n+\t\tstrvec_push(&refs_verify.args, \"--strict\");\n+\n+\tif (run_command(&refs_verify))\n+\t\terrors_found |= ERROR_REFS;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n@@ -1066,6 +1081,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n+\tfsck_refs();\n+\n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \n-- \n2.45.2\n\n"},{"id":"498026","messageId":"ZoVZF0h_5VGVyuNI@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v7 7/9] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-03T13:58:47Z","receivedAt":"2024-07-03T13:58:49Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"For refs and reflogs, we need to scan its corresponding directories to\ncheck every regular file or symbolic link which shares the same pattern.\nIntroduce a unified interface for scanning directories for\nfiles-backend.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs/files-backend.c | 77 +++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 76 insertions(+), 1 deletion(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex d89eeda8ef..84acb58782 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -6,6 +6,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -3408,6 +3409,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n+{\n+\tconst char *gitdir = ref_store->gitdir;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\n+\tif (!iter) {\n+\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n+\t\tgoto out;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->verbose_refs)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n+\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n+\t\t\t\t iter->basename);\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n+\n+out:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tNULL\n+\t};\n+\n+\tif (o->verbose_refs)\n+\t\tfprintf_ln(stderr, \"Checking references consistency\");\n+\n+\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n+\n+\treturn ret;\n+}\n+\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_options *o)\n {\n@@ -3415,7 +3488,9 @@ static int files_fsck(struct ref_store *ref_store,\n \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n-\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o)\n+\t | files_fsck_refs(ref_store, o);\n+\n \treturn ret;\n }\n \n-- \n2.45.2\n\n"},{"id":"498027","messageId":"ZoVZJrU-pZZq96L7@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v7 8/9] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-03T13:59:02Z","receivedAt":"2024-07-03T13:59:03Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) only implicitly checks the reference, it does not fully\ncheck refs with bad format name such as standalone \"@\" and name ending\nwith \".lock\".\n\nIn order to provide such checks, add a new fsck message id \"badRefName\"\nwith default ERROR type. Use existing \"check_refname_format\" to explicit\ncheck the ref name. And add a new unit test to verify the functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 +\n fsck.h | 1 +\n refs/files-backend.c | 20 +++++++\n t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++\n 4 files changed, 125 insertions(+)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex f643585a34..dab4012246 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,9 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefName`::\n+\t(ERROR) A ref has a bad name.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex 246055c0f9..90457d1a1f 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 84acb58782..69a76048d3 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3419,6 +3419,25 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n+static int files_fsck_refs_name(struct fsck_options *o,\n+\t\t\t\tconst char *gitdir UNUSED,\n+\t\t\t\tconst char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\tret = fsck_refs_report(o, NULL, sb.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3470,6 +3489,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n {\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tfiles_fsck_refs_name,\n \t\tNULL\n \t};\n \ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..b2db58d2c6\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit tag multi_hierarchy/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1.lock: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/@ &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\tgit -c fsck.badRefName=warn fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\tgit -c fsck.badRefName=ignore fsck 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+test_done\n-- \n2.45.2\n\n"},{"id":"498028","messageId":"ZoVZNVyfscQYL3ua@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v7 9/9] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-03T13:59:17Z","receivedAt":"2024-07-03T13:59:19Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Enhance the git-fsck(1) command by adding a check for reference content\nin the files backend. The new functionality ensures that symrefs, real\nsymbolic link and regular refs are validated correctly.\n\nIn order to check the trailing content of the regular refs, add a new\nparameter `trailing` to `parse_loose_ref_contents`.\n\nFor symrefs, `parse_loose_ref_contents` will set the \"referent\".\nHowever, symbolic link could be either absolute or relative. Use\n\"strbuf_add_real_path\" to read the symbolic link and convert the\nrelative path to absolute path. Then use \"skip_prefix\" to make it align\nwith symref \"referent\".\n\nThus, the symrefs and symbolic links could share the same interface. Add\na new function \"files_fsck_symref_target\" which aims at checking the\nfollowing things:\n\n1. whether the pointee is under the `refs/` directory.\n2. whether the pointee name is correct.\n3. whether the pointee path is a wrong type in filesystem.\n\nLast, add the following FSCK MESSAGEs:\n\n1. \"badRefContent(ERROR)\": A ref has a bad content\n2. \"badSymrefPointee(ERROR)\": The pointee of a symref is bad.\n3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 9 +++\n fsck.h | 3 +\n refs.c | 2 +-\n refs/files-backend.c | 145 +++++++++++++++++++++++++++++++++-\n refs/refs-internal.h | 5 +-\n t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++\n 6 files changed, 269 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex dab4012246..b1630a478b 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,9 +19,15 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefContent`::\n+\t(ERROR) A ref has a bad content.\n+\n `badRefName`::\n \t(ERROR) A ref has a bad name.\n \n+`badSymrefPointee`::\n+\t(ERROR) The pointee of a symref is bad.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \n@@ -167,6 +173,9 @@\n `nullSha1`::\n \t(WARN) Tree contains entries pointing to a null sha1.\n \n+`trailingRefContent`::\n+\t(WARN) A ref content has trailing contents.\n+\n `treeNotSorted`::\n \t(ERROR) A tree is not properly sorted.\n \ndiff --git a/fsck.h b/fsck.h\nindex 90457d1a1f..637f596930 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -32,6 +32,8 @@ enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n+\tFUNC(BAD_REF_CONTENT, ERROR) \\\n+\tFUNC(BAD_SYMREF_POINTEE, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n@@ -72,6 +74,7 @@ enum fsck_msg_type {\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n+\tFUNC(TRAILING_REF_CONTENT, WARN) \\\n \tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n \tFUNC(LARGE_PATHNAME, WARN) \\\ndiff --git a/refs.c b/refs.c\nindex 410919246b..eb82fb7d4e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1760,7 +1760,7 @@ static int refs_read_special_head(struct ref_store *ref_store,\n \t}\n \n \tresult = parse_loose_ref_contents(content.buf, oid, referent, type,\n-\t\t\t\t\t failure_errno);\n+\t\t\t\t\t failure_errno, NULL);\n \n done:\n \tstrbuf_release(&full_path);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 69a76048d3..d98ef45403 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1,6 +1,7 @@\n #define USE_THE_REPOSITORY_VARIABLE\n \n #include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n #include \"../copy.h\"\n #include \"../environment.h\"\n #include \"../gettext.h\"\n@@ -553,7 +554,7 @@ static int read_ref_internal(struct ref_store *ref_store, const char *refname,\n \tstrbuf_rtrim(&sb_contents);\n \tbuf = sb_contents.buf;\n \n-\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr);\n+\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr, NULL);\n \n out:\n \tif (ret && !myerr)\n@@ -589,7 +590,7 @@ static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refn\n \n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno)\n+\t\t\t int *failure_errno, const char **trailing)\n {\n \tconst char *p;\n \tif (skip_prefix(buf, \"ref:\", &buf)) {\n@@ -611,6 +612,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t*failure_errno = EINVAL;\n \t\treturn -1;\n \t}\n+\n+\tif (trailing)\n+\t\t*trailing = p;\n+\n \treturn 0;\n }\n \n@@ -3438,6 +3443,141 @@ static int files_fsck_refs_name(struct fsck_options *o,\n \treturn ret;\n }\n \n+/*\n+ * Check the symref \"pointee_name\" and \"pointee_path\". The caller should\n+ * make sure that \"pointee_path\" is absolute. For symbolic ref, \"pointee_name\"\n+ * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n+ * be the relative path agaignst \"gitdir\".\n+ */\n+static int files_fsck_symref_target(struct fsck_options *o,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const char *pointee_name,\n+\t\t\t\t const char *pointee_path)\n+{\n+\tconst char *p = NULL;\n+\tstruct stat st;\n+\tint ret = 0;\n+\n+\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n+\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to target out of refs hierarchy\");\n+\t\tgoto out;\n+\t}\n+\n+\tif (check_refname_format(pointee_name, 0)) {\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid refname\");\n+\t}\n+\n+\tif (lstat(pointee_path, &st) < 0)\n+\t\tgoto out;\n+\n+\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid target\");\n+\t\tgoto out;\n+\t}\n+out:\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs_content(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter)\n+{\n+\tstruct strbuf pointee_path = STRBUF_INIT,\n+\t\t ref_content = STRBUF_INIT,\n+\t\t abs_gitdir = STRBUF_INIT,\n+\t\t referent = STRBUF_INIT,\n+\t\t refname = STRBUF_INIT;\n+\tconst char *trailing = NULL;\n+\tint failure_errno = 0;\n+\tunsigned int type = 0;\n+\tstruct object_id oid;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\n+\t/*\n+\t * If the file is a symlink, we need to only check the connectivity\n+\t * of the destination object.\n+\t */\n+\tif (S_ISLNK(iter->st.st_mode)) {\n+\t\tconst char *pointee_name = NULL;\n+\n+\t\tstrbuf_add_real_path(&pointee_path, iter->path.buf);\n+\n+\t\tstrbuf_add_absolute_path(&abs_gitdir, gitdir);\n+\t\tstrbuf_normalize_path(&abs_gitdir);\n+\t\tif (!is_dir_sep(abs_gitdir.buf[abs_gitdir.len - 1]))\n+\t\t\tstrbuf_addch(&abs_gitdir, '/');\n+\n+\t\tif (!skip_prefix(pointee_path.buf,\n+\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n+\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t\t \"point to target outside gitdir\");\n+\t\t\tgoto clean;\n+\t\t}\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, pointee_name,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n+\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n+\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n+\t\t\t\t &referent, &type,\n+\t\t\t\t &failure_errno, &trailing)) {\n+\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n+\t\t\t\t \"invalid ref content\");\n+\t\tgoto clean;\n+\t}\n+\n+\t/*\n+\t * If the ref is a symref, we need to check the destination name and\n+\t * connectivity.\n+\t */\n+\tif (referent.len && (type & REF_ISSYMREF)) {\n+\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n+\t\tstrbuf_rtrim(&referent);\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, referent.buf,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t} else {\n+\t\t/*\n+\t\t * Only regular refs could have a trailing garbage. Should\n+\t\t * be reported as a warning.\n+\t\t */\n+\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n+\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n+\t\t\t\t\t \"trailing garbage in ref\");\n+\t\t\tgoto clean;\n+\t\t}\n+\t}\n+\n+clean:\n+\tstrbuf_release(&abs_gitdir);\n+\tstrbuf_release(&pointee_path);\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&ref_content);\n+\tstrbuf_release(&referent);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3490,6 +3630,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n \t\tfiles_fsck_refs_name,\n+\t\tfiles_fsck_refs_content,\n \t\tNULL\n \t};\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex a905e187cd..2fabf41d14 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -709,11 +709,12 @@ struct ref_store {\n \n /*\n * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for\n- * invalid contents.\n+ * invalid contents. Also *trailing is set to the first character after the\n+ * refname or NULL if the referent is not empty.\n */\n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno);\n+\t\t\t int *failure_errno, const char **trailing);\n \n /*\n * Fill in the generic part of refs and add it to our collection of\ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nindex b2db58d2c6..35bf40ee64 100755\n--- a/t/t0602-reffiles-fsck.sh\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -98,4 +98,114 @@ test_expect_success 'ref name check should be adapted into fsck messages' '\n \t)\n '\n \n+test_expect_success 'regular ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit checkout -b a/b/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse branch-1)\" > $branch_dir_prefix/branch-1-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse tag-1)\" > $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_must_fail git -c fsck.trailingRefContent=error fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s \" \"$(git rev-parse tag-2)\" > $tag_dir_prefix/tag-2-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/tags/tag-2-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/a/b/branch-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'symbolic ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid refname\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid target\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: logs/maint-v2.45\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to target out of refs hierarchy\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.2\n\n"},{"id":"498130","messageId":"ysrjjyksfwto35ug7o6xbgbu53vvjso6nov3u2q6vvdprolqwh@vjhcc4x45j7t","threadId":"61565","inReplyTo":"ZoVYjrfCFO0_K4Ry@ArchLinux","subject":"Re: [GSoC][PATCH v7 1/9] fsck: rename \"skiplist\" to \"oid_skiplist\"","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-07-05T22:07:29Z","receivedAt":"2024-07-05T22:07:56Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/07/03 09:56PM, shejialuo wrote:\n> Because we introduce ref consistency check. The original \"skiplist\" is a\n> common option which is set up during handling user configs. To avoid\n> causing ambiguity, rename \"skiplist\" to \"oid_skiplist\".\n\nI think the commit message could be expanded on to provide additional\ncontext and reasoning for the change. From reading this alone, it sounds\nlike we have already introduced the ref consistency check and are now\nlooking to rename a variable. When really this is a preparatory change.\nWithout reading ahead, I'm also left wondering why this name change\nreduces ambiguity.\n\n-Justin\n"},{"id":"498131","messageId":"adijctr2gmmyydpnnrmnu5p5n7mambfrd2ylqm6jtmvv2jdlxo@fbmhti7b4tks","threadId":"61565","inReplyTo":"ZoVYn6C-QOuHM2sC@ArchLinux","subject":"Re: [GSoC][PATCH v7 2/9] fsck: add a unified interface for reporting fsck messages","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-07-05T22:43:37Z","receivedAt":"2024-07-05T22:44:04Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/07/03 09:56PM, shejialuo wrote:\n> The static function \"report\" provided by \"fsck.c\" aims at reporting the\n> problems related to object database which cannot be reused for refs.\n> In order to provide a unified interface which can report either objects\n> or refs, create a new function \"vfsck_report\" by adding\n> \"checked_ref_name\" parameter following the \"report\" prototype. However,\n> instead of using \"...\", provide \"va_list\" to allow more flexibility.\n> \n> The \"vfsck_report\" function will use \"error_func\" registered in\n> \"fsck_options\" function to report customized messages. Change\n> \"error_func\" prototype to align with the \"vfsck_report\".\n> \n> Change \"report\" function to make it use \"vfsck_report\" to report\n> objects-related messages. Add a new function called \"fsck_refs_report\"\n> to use \"vfsck_report\" to report refs-related messages.\n\nTo restate in my own words, the existing \"report\" function is checked if\nan fsck message should be returned and also formats the message. It is\ncurrently setup to only handle object database related fsck problems.\nThe `fsck_error` function type is generalized to also accept a reference\nname enabling fsck problems to be reported for references.\n\n> \n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n> builtin/fsck.c | 15 ++++-----\n> builtin/mktag.c | 1 +\n> fsck.c | 81 ++++++++++++++++++++++++++++++++++++-------------\n> fsck.h | 42 ++++++++++++++++---------\n> object-file.c | 11 ++++---\n> 5 files changed, 102 insertions(+), 48 deletions(-)\n> \n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index d13a226c2e..de34538c4f 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -89,12 +89,13 @@ static int objerror(struct object *obj, const char *err)\n> \treturn -1;\n> }\n> \n> -static int fsck_error_func(struct fsck_options *o UNUSED,\n> -\t\t\t const struct object_id *oid,\n> -\t\t\t enum object_type object_type,\n> -\t\t\t enum fsck_msg_type msg_type,\n> -\t\t\t enum fsck_msg_id msg_id UNUSED,\n> -\t\t\t const char *message)\n> +static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n> +\t\t\t\t const struct object_id *oid,\n> +\t\t\t\t enum object_type object_type,\n> +\t\t\t\t const char *checked_ref_name UNUSED,\n> +\t\t\t\t enum fsck_msg_type msg_type,\n> +\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> +\t\t\t\t const char *message)\n> {\n> \tswitch (msg_type) {\n> \tcase FSCK_WARN:\n> @@ -938,7 +939,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n> \n> \tfsck_walk_options.walk = mark_object;\n> \tfsck_obj_options.walk = mark_used;\n> -\tfsck_obj_options.error_func = fsck_error_func;\n> +\tfsck_obj_options.error_func = fsck_objects_error_func;\n> \tif (check_strict)\n> \t\tfsck_obj_options.strict = 1;\n> \n> diff --git a/builtin/mktag.c b/builtin/mktag.c\n> index 4767f1a97e..42f945c584 100644\n> --- a/builtin/mktag.c\n> +++ b/builtin/mktag.c\n> @@ -20,6 +20,7 @@ static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n> static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n> \t\t\t\t const struct object_id *oid UNUSED,\n> \t\t\t\t enum object_type object_type UNUSED,\n> +\t\t\t\t const char *checked_ref_name UNUSED,\n> \t\t\t\t enum fsck_msg_type msg_type,\n> \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> \t\t\t\t const char *message)\n> diff --git a/fsck.c b/fsck.c\n> index 1960bfeba9..7182ce8e80 100644\n> --- a/fsck.c\n> +++ b/fsck.c\n> @@ -226,12 +226,18 @@ static int object_on_skiplist(struct fsck_options *opts,\n> \treturn opts && oid && oidset_contains(&opts->oid_skiplist, oid);\n> }\n> \n> -__attribute__((format (printf, 5, 6)))\n> -static int report(struct fsck_options *options,\n> -\t\t const struct object_id *oid, enum object_type object_type,\n> -\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n> +/*\n> + * Provide a unified interface for either fscking refs or objects.\n> + * It will get the current msg error type and call the error_func callback\n> + * which is registered in the \"fsck_options\" struct.\n> + */\n> +static int vfsck_report(struct fsck_options *options,\n> +\t\t\tconst struct object_id *oid,\n> +\t\t\tenum object_type object_type,\n> +\t\t\tconst char *checked_ref_name,\n> +\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n> {\n> -\tva_list ap;\n> +\tva_list ap_copy;\n> \tstruct strbuf sb = STRBUF_INIT;\n> \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n> \tint result;\n> @@ -250,9 +256,9 @@ static int report(struct fsck_options *options,\n> \tprepare_msg_ids();\n> \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n> \n> -\tva_start(ap, fmt);\n> -\tstrbuf_vaddf(&sb, fmt, ap);\n> -\tresult = options->error_func(options, oid, object_type,\n> +\tva_copy(ap_copy, ap);\n> +\tstrbuf_vaddf(&sb, fmt, ap_copy);\n> +\tresult = options->error_func(options, oid, object_type, checked_ref_name,\n> \t\t\t\t msg_type, msg_id, sb.buf);\n> \tstrbuf_release(&sb);\n> \tva_end(ap);\n> @@ -260,6 +266,36 @@ static int report(struct fsck_options *options,\n> \treturn result;\n> }\n> \n> +__attribute__((format (printf, 5, 6)))\n> +static int report(struct fsck_options *options,\n> +\t\t const struct object_id *oid, enum object_type object_type,\n> +\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n> +{\n> +\tva_list ap;\n> +\tint result;\n> +\tva_start(ap, fmt);\n> +\tresult = vfsck_report(options, oid, object_type, NULL,\n> +\t\t\t msg_id, fmt, ap);\n> +\tva_end(ap);\n> +\treturn result;\n> +}\n> +\n> +\n\nLooks like there are some extra new lines here.\n\n> +\n> +int fsck_refs_report(struct fsck_options *options,\n> +\t\t const struct object_id *oid,\n> +\t\t const char *checked_ref_name,\n> +\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n> +{\n> +\tva_list ap;\n> +\tint result;\n> +\tva_start(ap, fmt);\n> +\tresult = vfsck_report(options, oid, OBJ_NONE,\n> +\t\t\t checked_ref_name, msg_id, fmt, ap);\n\nDo we expect that reference related fsck problems may also be skipped\nvia the oid_skiplist?\n\n> +\tva_end(ap);\n> +\treturn result;\n> +}\n> +\n> void fsck_enable_object_names(struct fsck_options *options)\n> {\n> \tif (!options->object_names)\n> @@ -1200,12 +1236,13 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n> \t\t type);\n> }\n> \n> -int fsck_error_function(struct fsck_options *o,\n> -\t\t\tconst struct object_id *oid,\n> -\t\t\tenum object_type object_type UNUSED,\n> -\t\t\tenum fsck_msg_type msg_type,\n> -\t\t\tenum fsck_msg_id msg_id UNUSED,\n> -\t\t\tconst char *message)\n> +int fsck_objects_error_function(struct fsck_options *o,\n> +\t\t\t\tconst struct object_id *oid,\n> +\t\t\t\tenum object_type object_type UNUSED,\n> +\t\t\t\tconst char *checked_ref_name UNUSED,\n> +\t\t\t\tenum fsck_msg_type msg_type,\n> +\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n> +\t\t\t\tconst char *message)\n> {\n> \tif (msg_type == FSCK_WARN) {\n> \t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n> @@ -1303,16 +1340,18 @@ int git_fsck_config(const char *var, const char *value,\n> * Custom error callbacks that are used in more than one place.\n> */\n> \n> -int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n> -\t\t\t\t\t const struct object_id *oid,\n> -\t\t\t\t\t enum object_type object_type,\n> -\t\t\t\t\t enum fsck_msg_type msg_type,\n> -\t\t\t\t\t enum fsck_msg_id msg_id,\n> -\t\t\t\t\t const char *message)\n> +int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n> +\t\t\t\t\t\t const struct object_id *oid,\n> +\t\t\t\t\t\t enum object_type object_type,\n> +\t\t\t\t\t\t const char *checked_ref_name,\n> +\t\t\t\t\t\t enum fsck_msg_type msg_type,\n> +\t\t\t\t\t\t enum fsck_msg_id msg_id,\n> +\t\t\t\t\t\t const char *message)\n> {\n> \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n> \t\tputs(oid_to_hex(oid));\n> \t\treturn 0;\n> \t}\n> -\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n> +\treturn fsck_objects_error_function(o, oid, object_type, checked_ref_name,\n> +\t\t\t\t\t msg_type, msg_id, message);\n> }\n> diff --git a/fsck.h b/fsck.h\n> index 1ee3dd85ba..f703dfb5e8 100644\n> --- a/fsck.h\n> +++ b/fsck.h\n> @@ -114,22 +114,27 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n> typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n> \t\t\t void *data, struct fsck_options *options);\n> \n> -/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n> +/*\n> + * callback function for reporting errors when checking either objects or refs\n> + */\n> typedef int (*fsck_error)(struct fsck_options *o,\n> \t\t\t const struct object_id *oid, enum object_type object_type,\n> +\t\t\t const char *checked_ref_name,\n> \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n> \t\t\t const char *message);\n> \n> -int fsck_error_function(struct fsck_options *o,\n> -\t\t\tconst struct object_id *oid, enum object_type object_type,\n> -\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n> -\t\t\tconst char *message);\n> -int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n> -\t\t\t\t\t const struct object_id *oid,\n> -\t\t\t\t\t enum object_type object_type,\n> -\t\t\t\t\t enum fsck_msg_type msg_type,\n> -\t\t\t\t\t enum fsck_msg_id msg_id,\n> -\t\t\t\t\t const char *message);\n> +int fsck_objects_error_function(struct fsck_options *o,\n> +\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n> +\t\t\t\tconst char *checked_ref_name,\n> +\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n> +\t\t\t\tconst char *message);\n> +int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n> +\t\t\t\t\t\t const struct object_id *oid,\n> +\t\t\t\t\t\t enum object_type object_type,\n> +\t\t\t\t\t\t const char *checked_ref_name,\n> +\t\t\t\t\t\t enum fsck_msg_type msg_type,\n> +\t\t\t\t\t\t enum fsck_msg_id msg_id,\n> +\t\t\t\t\t\t const char *message);\n> \n> struct fsck_options {\n> \tfsck_walk_func walk;\n> @@ -145,12 +150,12 @@ struct fsck_options {\n> };\n> \n> #define FSCK_OPTIONS_DEFAULT { \\\n> -\t.skiplist = OIDSET_INIT, \\\n> +\t.oid_skiplist = OIDSET_INIT, \\\n\nSince we renamed skiplist -> oid_skiplist in the previous patch,\nshouldn't we update this there?\n\n> \t.gitmodules_found = OIDSET_INIT, \\\n> \t.gitmodules_done = OIDSET_INIT, \\\n> \t.gitattributes_found = OIDSET_INIT, \\\n> \t.gitattributes_done = OIDSET_INIT, \\\n> -\t.error_func = fsck_error_function \\\n> +\t.error_func = fsck_objects_error_function \\\n> }\n> #define FSCK_OPTIONS_STRICT { \\\n> \t.strict = 1, \\\n> @@ -158,7 +163,7 @@ struct fsck_options {\n> \t.gitmodules_done = OIDSET_INIT, \\\n> \t.gitattributes_found = OIDSET_INIT, \\\n> \t.gitattributes_done = OIDSET_INIT, \\\n> -\t.error_func = fsck_error_function, \\\n> +\t.error_func = fsck_objects_error_function, \\\n> }\n> #define FSCK_OPTIONS_MISSING_GITMODULES { \\\n> \t.strict = 1, \\\n> @@ -166,7 +171,7 @@ struct fsck_options {\n> \t.gitmodules_done = OIDSET_INIT, \\\n> \t.gitattributes_found = OIDSET_INIT, \\\n> \t.gitattributes_done = OIDSET_INIT, \\\n> -\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n> +\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n> }\n> \n> /* descend in all linked child objects\n> @@ -209,6 +214,13 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n> */\n> int fsck_finish(struct fsck_options *options);\n> \n\nIn my opinion it would be nice to add a comment to document\n`fsck_refs_report()` here.\n\n> +__attribute__((format (printf, 5, 6)))\n> +int fsck_refs_report(struct fsck_options *options,\n> +\t\t const struct object_id *oid,\n> +\t\t const char *checked_ref_name,\n> +\t\t enum fsck_msg_id msg_id,\n> +\t\t const char *fmt, ...);\n> +\n> /*\n> * Subsystem for storing human-readable names for each object.\n> *\n> diff --git a/object-file.c b/object-file.c\n> index 065103be3e..d2c6427935 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -2470,11 +2470,12 @@ int repo_has_object_file(struct repository *r,\n> * give more context.\n> */\n> static int hash_format_check_report(struct fsck_options *opts UNUSED,\n> -\t\t\t\t const struct object_id *oid UNUSED,\n> -\t\t\t\t enum object_type object_type UNUSED,\n> -\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n> -\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> -\t\t\t\t const char *message)\n> +\t\t\t\t const struct object_id *oid UNUSED,\n> +\t\t\t\t enum object_type object_type UNUSED,\n> +\t\t\t\t const char *ref_checked_name UNUSED,\n> +\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n> +\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> +\t\t\t\t const char *message)\n> {\n> \terror(_(\"object fails fsck: %s\"), message);\n> \treturn 1;\n> -- \n> 2.45.2\n> \n> \n"},{"id":"498238","messageId":"ZovWUa8JvzXooqwL@ArchLinux","threadId":"61565","inReplyTo":"ysrjjyksfwto35ug7o6xbgbu53vvjso6nov3u2q6vvdprolqwh@vjhcc4x45j7t","subject":"Re: [GSoC][PATCH v7 1/9] fsck: rename \"skiplist\" to \"oid_skiplist\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-08T12:06:41Z","receivedAt":"2024-07-08T12:06:43Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Fri, Jul 05, 2024 at 05:07:29PM -0500, Justin Tobler wrote:\n> On 24/07/03 09:56PM, shejialuo wrote:\n> > Because we introduce ref consistency check. The original \"skiplist\" is a\n> > common option which is set up during handling user configs. To avoid\n> > causing ambiguity, rename \"skiplist\" to \"oid_skiplist\".\n> \n> I think the commit message could be expanded on to provide additional\n> context and reasoning for the change. From reading this alone, it sounds\n> like we have already introduced the ref consistency check and are now\n> looking to rename a variable. When really this is a preparatory change.\n> Without reading ahead, I'm also left wondering why this name change\n> reduces ambiguity.\n> \n\nYes, I will add more information to show the context. \"skiplist\" is\ninitialized using \"git_fsck_config\" to parse the user-specific config.\nIn this series, we introduce ref-specific check. \"skiplist\" is a general\nname which may make the caller think \"skiplist\" is related to both the\nrefs and objects.\n\nFor later implementation, we may also introduce \"skiplist\" for refs, but\nfor refs, we concern about the name not the \"oidset\". I will add more\ninformation to explain how this change reduces ambiguity.\n\nThanks,\nJialuo\n"},{"id":"498239","messageId":"ZovXrXfDfBJHa_Tg@ArchLinux","threadId":"61565","inReplyTo":"adijctr2gmmyydpnnrmnu5p5n7mambfrd2ylqm6jtmvv2jdlxo@fbmhti7b4tks","subject":"Re: [GSoC][PATCH v7 2/9] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-08T12:12:29Z","receivedAt":"2024-07-08T12:12:31Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Fri, Jul 05, 2024 at 05:43:37PM -0500, Justin Tobler wrote:\n> On 24/07/03 09:56PM, shejialuo wrote:\n> > The static function \"report\" provided by \"fsck.c\" aims at reporting the\n> > problems related to object database which cannot be reused for refs.\n> > In order to provide a unified interface which can report either objects\n> > or refs, create a new function \"vfsck_report\" by adding\n> > \"checked_ref_name\" parameter following the \"report\" prototype. However,\n> > instead of using \"...\", provide \"va_list\" to allow more flexibility.\n> > \n> > The \"vfsck_report\" function will use \"error_func\" registered in\n> > \"fsck_options\" function to report customized messages. Change\n> > \"error_func\" prototype to align with the \"vfsck_report\".\n> > \n> > Change \"report\" function to make it use \"vfsck_report\" to report\n> > objects-related messages. Add a new function called \"fsck_refs_report\"\n> > to use \"vfsck_report\" to report refs-related messages.\n> \n> To restate in my own words, the existing \"report\" function is checked if\n> an fsck message should be returned and also formats the message. It is\n> currently setup to only handle object database related fsck problems.\n> The `fsck_error` function type is generalized to also accept a reference\n> name enabling fsck problems to be reported for references.\n> \n\nExactly, I will add more information in the commit message.\n\nThanks,\nJialuo\n"},{"id":"498249","messageId":"ZovqY4vQnQBAs7PH@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v8 0/9] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-08T13:32:19Z","receivedAt":"2024-07-08T13:32:22Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Hi All:\n\nThis version enhances the patch 1 and patch 2 commit messages according\nto the review of Justin.\n\nThanks,\nJialuo\n\nshejialuo (9):\n fsck: rename \"skiplist\" to \"oid_skiplist\"\n fsck: add a unified interface for reporting fsck messages\n fsck: add refs-related options and error report function\n refs: set up ref consistency check infrastructure\n builtin/refs: add verify subcommand\n builtin/fsck: add `git-refs verify` child process\n files-backend: add unified interface for refs scanning\n fsck: add ref name check for files backend\n fsck: add ref content check for files backend\n\n Documentation/fsck-msgids.txt | 12 ++\n Documentation/git-refs.txt | 13 ++\n builtin/fsck.c | 32 ++++-\n builtin/mktag.c | 1 +\n builtin/refs.c | 44 ++++++\n fsck.c | 107 +++++++++++---\n fsck.h | 63 ++++++---\n object-file.c | 11 +-\n refs.c | 7 +-\n refs.h | 8 ++\n refs/debug.c | 11 ++\n refs/files-backend.c | 255 +++++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 ++\n refs/refs-internal.h | 11 +-\n refs/reftable-backend.c | 8 ++\n t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++\n 16 files changed, 745 insertions(+), 57 deletions(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\nRange-diff against v7:\n 1: f5fcf36132 ! 1: 61e475840f fsck: rename \"skiplist\" to \"oid_skiplist\"\n @@ Metadata\n ## Commit message ##\n fsck: rename \"skiplist\" to \"oid_skiplist\"\n \n - Because we introduce ref consistency check. The original \"skiplist\" is a\n - common option which is set up during handling user configs. To avoid\n - causing ambiguity, rename \"skiplist\" to \"oid_skiplist\".\n + The \"skiplist\" field in \"fsck_options\" is related to objects. Because we\n + are going to introduce ref consistency check, the \"skiplist\" name is too\n + general which will make the caller think \"skiplist\" is related to both\n + the refs and objects.\n +\n + It may seem that for both refs and objects, we should provide a general\n + \"skiplist\" here. However, the type for \"skiplist\" is `struct oidset`\n + which is totally unsuitable for refs.\n +\n + To avoid above ambiguity, rename \"skiplist\" to \"oid_skiplist\".\n \n Mentored-by: Patrick Steinhardt \n Mentored-by: Karthik Nayak \n 2: c81b5b6fba ! 2: f2576d88a9 fsck: add a unified interface for reporting fsck messages\n @@ Metadata\n ## Commit message ##\n fsck: add a unified interface for reporting fsck messages\n \n - The static function \"report\" provided by \"fsck.c\" aims at reporting the\n - problems related to object database which cannot be reused for refs.\n - In order to provide a unified interface which can report either objects\n - or refs, create a new function \"vfsck_report\" by adding\n - \"checked_ref_name\" parameter following the \"report\" prototype. However,\n - instead of using \"...\", provide \"va_list\" to allow more flexibility.\n + The static function \"report\" provided by \"fsck.c\" aims at checking fsck\n + error type and calling the callback \"error_func\" to report the message.\n + However, \"report\" function is only related to object database which\n + cannot be reused for refs. In order to provide a unified interface which\n + can report either objects or refs, create a new function \"vfsck_report\"\n + by adding \"checked_ref_name\" parameter following the \"report\" prototype.\n + Instead of using \"...\", provide \"va_list\" to allow more flexibility.\n \n - The \"vfsck_report\" function will use \"error_func\" registered in\n - \"fsck_options\" function to report customized messages. Change\n - \"error_func\" prototype to align with the \"vfsck_report\".\n + Like \"report\", the \"vfsck_report\" function will use \"error_func\"\n + registered in \"fsck_options\" to report customized messages. Change\n + \"error_func\" prototype to align with the new \"vfsck_report\".\n \n - Change \"report\" function to make it use \"vfsck_report\" to report\n - objects-related messages. Add a new function called \"fsck_refs_report\"\n - to use \"vfsck_report\" to report refs-related messages.\n + Then, change \"report\" function to use \"vfsck_report\" to report objects\n + related messages. Add a new function called \"fsck_refs_report\" to use\n + \"vfsck_report\" to report refs related messages.\n \n Mentored-by: Patrick Steinhardt \n Mentored-by: Karthik Nayak \n 3: 53156dc847 = 3: c3c2dda50c fsck: add refs-related options and error report function\n 4: 358f4a1be9 = 4: e826dc17ec refs: set up ref consistency check infrastructure\n 5: e632859df6 = 5: 33cac4882b builtin/refs: add verify subcommand\n 6: 5d7a6261ae = 6: 32668e3543 builtin/fsck: add `git-refs verify` child process\n 7: 33311af1b1 = 7: df83b2a990 files-backend: add unified interface for refs scanning\n 8: 02bd3ac952 = 8: c696c15651 fsck: add ref name check for files backend\n 9: 94d123471d = 9: 8b0f3aeb9c fsck: add ref content check for files backend\n-- \n2.45.2\n\n"},{"id":"498250","messageId":"ZovrAe_P0vf9dbPZ@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v8 1/9] fsck: rename \"skiplist\" to \"oid_skiplist\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-08T13:34:57Z","receivedAt":"2024-07-08T13:35:00Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The \"skiplist\" field in \"fsck_options\" is related to objects. Because we\nare going to introduce ref consistency check, the \"skiplist\" name is too\ngeneral which will make the caller think \"skiplist\" is related to both\nthe refs and objects.\n\nIt may seem that for both refs and objects, we should provide a general\n\"skiplist\" here. However, the type for \"skiplist\" is `struct oidset`\nwhich is totally unsuitable for refs.\n\nTo avoid above ambiguity, rename \"skiplist\" to \"oid_skiplist\".\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 4 ++--\n fsck.h | 2 +-\n 2 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex eea7145470..1960bfeba9 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n \t\tif (!strcmp(buf, \"skiplist\")) {\n \t\t\tif (equal == len)\n \t\t\t\tdie(\"skiplist requires a path\");\n-\t\t\toidset_parse_file(&options->skiplist, buf + equal + 1,\n+\t\t\toidset_parse_file(&options->oid_skiplist, buf + equal + 1,\n \t\t\t\t\t the_repository->hash_algo);\n \t\t\tbuf += len + 1;\n \t\t\tcontinue;\n@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n static int object_on_skiplist(struct fsck_options *opts,\n \t\t\t const struct object_id *oid)\n {\n-\treturn opts && oid && oidset_contains(&opts->skiplist, oid);\n+\treturn opts && oid && oidset_contains(&opts->oid_skiplist, oid);\n }\n \n __attribute__((format (printf, 5, 6)))\ndiff --git a/fsck.h b/fsck.h\nindex 6085a384f6..1ee3dd85ba 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -136,7 +136,7 @@ struct fsck_options {\n \tfsck_error error_func;\n \tunsigned strict:1;\n \tenum fsck_msg_type *msg_type;\n-\tstruct oidset skiplist;\n+\tstruct oidset oid_skiplist;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n \tstruct oidset gitattributes_found;\n-- \n2.45.2\n\n"},{"id":"498251","messageId":"ZovrFCzRg06pq5eI@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v8 2/9] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-08T13:35:16Z","receivedAt":"2024-07-08T13:35:19Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The static function \"report\" provided by \"fsck.c\" aims at checking fsck\nerror type and calling the callback \"error_func\" to report the message.\nHowever, \"report\" function is only related to object database which\ncannot be reused for refs. In order to provide a unified interface which\ncan report either objects or refs, create a new function \"vfsck_report\"\nby adding \"checked_ref_name\" parameter following the \"report\" prototype.\nInstead of using \"...\", provide \"va_list\" to allow more flexibility.\n\nLike \"report\", the \"vfsck_report\" function will use \"error_func\"\nregistered in \"fsck_options\" to report customized messages. Change\n\"error_func\" prototype to align with the new \"vfsck_report\".\n\nThen, change \"report\" function to use \"vfsck_report\" to report objects\nrelated messages. Add a new function called \"fsck_refs_report\" to use\n\"vfsck_report\" to report refs related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 15 ++++-----\n builtin/mktag.c | 1 +\n fsck.c | 81 ++++++++++++++++++++++++++++++++++++-------------\n fsck.h | 42 ++++++++++++++++---------\n object-file.c | 11 ++++---\n 5 files changed, 102 insertions(+), 48 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..de34538c4f 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -89,12 +89,13 @@ static int objerror(struct object *obj, const char *err)\n \treturn -1;\n }\n \n-static int fsck_error_func(struct fsck_options *o UNUSED,\n-\t\t\t const struct object_id *oid,\n-\t\t\t enum object_type object_type,\n-\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t const char *message)\n+static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t enum object_type object_type,\n+\t\t\t\t const char *checked_ref_name UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \tswitch (msg_type) {\n \tcase FSCK_WARN:\n@@ -938,7 +939,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tfsck_walk_options.walk = mark_object;\n \tfsck_obj_options.walk = mark_used;\n-\tfsck_obj_options.error_func = fsck_error_func;\n+\tfsck_obj_options.error_func = fsck_objects_error_func;\n \tif (check_strict)\n \t\tfsck_obj_options.strict = 1;\n \ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex 4767f1a97e..42f945c584 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -20,6 +20,7 @@ static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n \t\t\t\t const struct object_id *oid UNUSED,\n \t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const char *checked_ref_name UNUSED,\n \t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t\t const char *message)\ndiff --git a/fsck.c b/fsck.c\nindex 1960bfeba9..7182ce8e80 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -226,12 +226,18 @@ static int object_on_skiplist(struct fsck_options *opts,\n \treturn opts && oid && oidset_contains(&opts->oid_skiplist, oid);\n }\n \n-__attribute__((format (printf, 5, 6)))\n-static int report(struct fsck_options *options,\n-\t\t const struct object_id *oid, enum object_type object_type,\n-\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+/*\n+ * Provide a unified interface for either fscking refs or objects.\n+ * It will get the current msg error type and call the error_func callback\n+ * which is registered in the \"fsck_options\" struct.\n+ */\n+static int vfsck_report(struct fsck_options *options,\n+\t\t\tconst struct object_id *oid,\n+\t\t\tenum object_type object_type,\n+\t\t\tconst char *checked_ref_name,\n+\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n {\n-\tva_list ap;\n+\tva_list ap_copy;\n \tstruct strbuf sb = STRBUF_INIT;\n \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n \tint result;\n@@ -250,9 +256,9 @@ static int report(struct fsck_options *options,\n \tprepare_msg_ids();\n \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n \n-\tva_start(ap, fmt);\n-\tstrbuf_vaddf(&sb, fmt, ap);\n-\tresult = options->error_func(options, oid, object_type,\n+\tva_copy(ap_copy, ap);\n+\tstrbuf_vaddf(&sb, fmt, ap_copy);\n+\tresult = options->error_func(options, oid, object_type, checked_ref_name,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n \tva_end(ap);\n@@ -260,6 +266,36 @@ static int report(struct fsck_options *options,\n \treturn result;\n }\n \n+__attribute__((format (printf, 5, 6)))\n+static int report(struct fsck_options *options,\n+\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = vfsck_report(options, oid, object_type, NULL,\n+\t\t\t msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n+\n+\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t const char *checked_ref_name,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = vfsck_report(options, oid, OBJ_NONE,\n+\t\t\t checked_ref_name, msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n void fsck_enable_object_names(struct fsck_options *options)\n {\n \tif (!options->object_names)\n@@ -1200,12 +1236,13 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid,\n-\t\t\tenum object_type object_type UNUSED,\n-\t\t\tenum fsck_msg_type msg_type,\n-\t\t\tenum fsck_msg_id msg_id UNUSED,\n-\t\t\tconst char *message)\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tenum object_type object_type UNUSED,\n+\t\t\t\tconst char *checked_ref_name UNUSED,\n+\t\t\t\tenum fsck_msg_type msg_type,\n+\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n+\t\t\t\tconst char *message)\n {\n \tif (msg_type == FSCK_WARN) {\n \t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n@@ -1303,16 +1340,18 @@ int git_fsck_config(const char *var, const char *value,\n * Custom error callbacks that are used in more than one place.\n */\n \n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message)\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t const char *checked_ref_name,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message)\n {\n \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n+\treturn fsck_objects_error_function(o, oid, object_type, checked_ref_name,\n+\t\t\t\t\t msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex 1ee3dd85ba..f703dfb5e8 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -114,22 +114,27 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n \t\t\t void *data, struct fsck_options *options);\n \n-/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n+/*\n+ * callback function for reporting errors when checking either objects or refs\n+ */\n typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t\t const char *checked_ref_name,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid, enum object_type object_type,\n-\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n-\t\t\tconst char *message);\n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message);\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\t\tconst char *checked_ref_name,\n+\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n+\t\t\t\tconst char *message);\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t const char *checked_ref_name,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message);\n \n struct fsck_options {\n \tfsck_walk_func walk;\n@@ -145,12 +150,12 @@ struct fsck_options {\n };\n \n #define FSCK_OPTIONS_DEFAULT { \\\n-\t.skiplist = OIDSET_INIT, \\\n+\t.oid_skiplist = OIDSET_INIT, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function \\\n+\t.error_func = fsck_objects_error_function \\\n }\n #define FSCK_OPTIONS_STRICT { \\\n \t.strict = 1, \\\n@@ -158,7 +163,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function, \\\n+\t.error_func = fsck_objects_error_function, \\\n }\n #define FSCK_OPTIONS_MISSING_GITMODULES { \\\n \t.strict = 1, \\\n@@ -166,7 +171,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n+\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n \n /* descend in all linked child objects\n@@ -209,6 +214,13 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+__attribute__((format (printf, 5, 6)))\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t const char *checked_ref_name,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\ndiff --git a/object-file.c b/object-file.c\nindex 065103be3e..d2c6427935 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2470,11 +2470,12 @@ int repo_has_object_file(struct repository *r,\n * give more context.\n */\n static int hash_format_check_report(struct fsck_options *opts UNUSED,\n-\t\t\t\t const struct object_id *oid UNUSED,\n-\t\t\t\t enum object_type object_type UNUSED,\n-\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n-\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t\t const char *message)\n+\t\t\t\t const struct object_id *oid UNUSED,\n+\t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const char *ref_checked_name UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \terror(_(\"object fails fsck: %s\"), message);\n \treturn 1;\n-- \n2.45.2\n\n"},{"id":"498252","messageId":"ZovrI1yRmS0X2BHu@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v8 3/9] fsck: add refs-related options and error report function","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-08T13:35:31Z","receivedAt":"2024-07-08T13:35:33Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Add refs-related options to the \"fsck_options\", create refs-specific\n\"error_func\" callback \"fsck_refs_error_function\".\n\n\"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\npasses the oid, it will use \"oid_to_hex\" to get the corresponding hex\nvalue to report to the caller.\n\nLast, add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\"\nmacros to create refs options easily.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 22 ++++++++++++++++++++++\n fsck.h | 15 +++++++++++++++\n 2 files changed, 37 insertions(+)\n\ndiff --git a/fsck.c b/fsck.c\nindex 7182ce8e80..d1dcbdcac2 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1252,6 +1252,28 @@ int fsck_objects_error_function(struct fsck_options *o,\n \treturn 1;\n }\n \n+int fsck_refs_error_function(struct fsck_options *options UNUSED,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type UNUSED,\n+\t\t\t const char *checked_ref_name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tstatic struct strbuf sb = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&sb, checked_ref_name);\n+\tif (oid)\n+\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n+\n+\tif (msg_type == FSCK_WARN) {\n+\t\twarning(\"%s: %s\", sb.buf, message);\n+\t\treturn 0;\n+\t}\n+\terror(\"%s: %s\", sb.buf, message);\n+\treturn 1;\n+}\n+\n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n \t\t struct fsck_options *options, const char *blob_type)\ndiff --git a/fsck.h b/fsck.h\nindex f703dfb5e8..246055c0f9 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -135,11 +135,19 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message);\n+int fsck_refs_error_function(struct fsck_options *options,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type,\n+\t\t\t const char *checked_ref_name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n \n struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n \tunsigned strict:1;\n+\tunsigned verbose_refs:1;\n \tenum fsck_msg_type *msg_type;\n \tstruct oidset oid_skiplist;\n \tstruct oidset gitmodules_found;\n@@ -173,6 +181,13 @@ struct fsck_options {\n \t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n+#define FSCK_REFS_OPTIONS_STRICT { \\\n+\t.strict = 1, \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n \n /* descend in all linked child objects\n * the return value is:\n-- \n2.45.2\n\n"},{"id":"498253","messageId":"ZovrMn9cjNrsOCEb@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v8 4/9] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-08T13:35:46Z","receivedAt":"2024-07-08T13:35:49Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It's organized as the five parts:\n\n1. The name and the initialization interfaces.\n2. The ref transaction interfaces.\n3. The ref internal interfaces (pack, rename and copy).\n4. The ref filesystem interfaces.\n5. The reflog related interfaces.\n\nTo keep consistent with the git-fsck(1), add a new interface named\n\"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\ngrouped into any above five categories. Explicitly add blank line to\nmake it different from others.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs.c | 5 +++++\n refs.h | 8 ++++++++\n refs/debug.c | 11 +++++++++++\n refs/files-backend.c | 15 ++++++++++++++-\n refs/packed-backend.c | 8 ++++++++\n refs/refs-internal.h | 6 ++++++\n refs/reftable-backend.c | 8 ++++++++\n 7 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex bb90a18875..410919246b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs, struct fsck_options *o)\n+{\n+\treturn refs->be->fsck(refs, o);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex 0ecba21b4a..804d6a7fce 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -4,6 +4,7 @@\n #include \"commit.h\"\n #include \"repository.h\"\n \n+struct fsck_options;\n struct object_id;\n struct ref_store;\n struct strbuf;\n@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_options *o);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..45e2e784a0 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tint res = drefs->refs->be->fsck(drefs->refs, o);\n+\ttrace_printf_key(&trace_refs, \"fsck: %d\\n\", res);\n+\treturn res;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex aa52d9be7c..d89eeda8ef 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3408,6 +3408,17 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\treturn ret;\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_init,\n@@ -3434,5 +3445,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a0666407cd..5209b0b212 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_init,\n@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex fa975d69aa..a905e187cd 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_options;\n struct ref_transaction;\n \n /*\n@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_options *o);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -677,6 +681,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex fbe74c239d..b5a1a526df 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.45.2\n\n"},{"id":"498254","messageId":"ZovrQVtf3lXO86oE@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v8 5/9] builtin/refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-08T13:36:01Z","receivedAt":"2024-07-08T13:36:03Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\ncheck the reference database consistency.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/git-refs.txt | 13 +++++++++++\n builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 57 insertions(+)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..1244a85b64 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+The following options are specific to 'git refs verify':\n+\n+--strict::\n+\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n+\tbe seen as ERROR. See linkgit:git-fsck[1].\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..74720f5e0d 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,54 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tunsigned int verbose = 0, strict = 0;\n+\tstruct option options[] = {\n+\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\tint ret = 0;\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"too many arguments\"));\n+\n+\tif (verbose)\n+\t\tfsck_refs_options.verbose_refs = 1;\n+\tif (strict)\n+\t\tfsck_refs_options.strict = 1;\n+\n+\tgit_config(git_fsck_config, &fsck_refs_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n+\n+\t/*\n+\t * Explicitly free the allocated array and \"oid_skiplist\"\n+\t */\n+\tfree(fsck_refs_options.msg_type);\n+\toidset_clear(&fsck_refs_options.oid_skiplist);\n+\treturn ret;\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \n-- \n2.45.2\n\n"},{"id":"498255","messageId":"ZovrUO0mU0WTUCQO@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v8 6/9] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-08T13:36:16Z","receivedAt":"2024-07-08T13:36:18Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new function \"fsck_refs\" that initializes and runs a child\nprocess to execute the \"git-refs verify\" command.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex de34538c4f..ec3357722c 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -897,6 +897,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \treturn res;\n }\n \n+static void fsck_refs(void)\n+{\n+\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n+\tchild_process_init(&refs_verify);\n+\trefs_verify.git_cmd = 1;\n+\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n+\tif (verbose)\n+\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n+\tif (check_strict)\n+\t\tstrvec_push(&refs_verify.args, \"--strict\");\n+\n+\tif (run_command(&refs_verify))\n+\t\terrors_found |= ERROR_REFS;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n@@ -1066,6 +1081,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n+\tfsck_refs();\n+\n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \n-- \n2.45.2\n\n"},{"id":"498256","messageId":"ZovrX-YWae1Iy7D2@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v8 7/9] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-08T13:36:31Z","receivedAt":"2024-07-08T13:36:33Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"For refs and reflogs, we need to scan its corresponding directories to\ncheck every regular file or symbolic link which shares the same pattern.\nIntroduce a unified interface for scanning directories for\nfiles-backend.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs/files-backend.c | 77 +++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 76 insertions(+), 1 deletion(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex d89eeda8ef..84acb58782 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -6,6 +6,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -3408,6 +3409,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n+{\n+\tconst char *gitdir = ref_store->gitdir;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\n+\tif (!iter) {\n+\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n+\t\tgoto out;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->verbose_refs)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n+\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n+\t\t\t\t iter->basename);\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n+\n+out:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tNULL\n+\t};\n+\n+\tif (o->verbose_refs)\n+\t\tfprintf_ln(stderr, \"Checking references consistency\");\n+\n+\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n+\n+\treturn ret;\n+}\n+\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_options *o)\n {\n@@ -3415,7 +3488,9 @@ static int files_fsck(struct ref_store *ref_store,\n \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n-\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o)\n+\t | files_fsck_refs(ref_store, o);\n+\n \treturn ret;\n }\n \n-- \n2.45.2\n\n"},{"id":"498257","messageId":"Zovrbbu_ahbe4mNr@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v8 8/9] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-08T13:36:45Z","receivedAt":"2024-07-08T13:36:48Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) only implicitly checks the reference, it does not fully\ncheck refs with bad format name such as standalone \"@\" and name ending\nwith \".lock\".\n\nIn order to provide such checks, add a new fsck message id \"badRefName\"\nwith default ERROR type. Use existing \"check_refname_format\" to explicit\ncheck the ref name. And add a new unit test to verify the functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 +\n fsck.h | 1 +\n refs/files-backend.c | 20 +++++++\n t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++\n 4 files changed, 125 insertions(+)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex f643585a34..dab4012246 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,9 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefName`::\n+\t(ERROR) A ref has a bad name.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex 246055c0f9..90457d1a1f 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 84acb58782..69a76048d3 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3419,6 +3419,25 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n+static int files_fsck_refs_name(struct fsck_options *o,\n+\t\t\t\tconst char *gitdir UNUSED,\n+\t\t\t\tconst char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\tret = fsck_refs_report(o, NULL, sb.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3470,6 +3489,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n {\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tfiles_fsck_refs_name,\n \t\tNULL\n \t};\n \ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..b2db58d2c6\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit tag multi_hierarchy/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1.lock: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/@ &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\tgit -c fsck.badRefName=warn fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\tgit -c fsck.badRefName=ignore fsck 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+test_done\n-- \n2.45.2\n\n"},{"id":"498258","messageId":"ZovrfD-lvHI2t9NN@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v8 9/9] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-08T13:37:00Z","receivedAt":"2024-07-08T13:37:02Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Enhance the git-fsck(1) command by adding a check for reference content\nin the files backend. The new functionality ensures that symrefs, real\nsymbolic link and regular refs are validated correctly.\n\nIn order to check the trailing content of the regular refs, add a new\nparameter `trailing` to `parse_loose_ref_contents`.\n\nFor symrefs, `parse_loose_ref_contents` will set the \"referent\".\nHowever, symbolic link could be either absolute or relative. Use\n\"strbuf_add_real_path\" to read the symbolic link and convert the\nrelative path to absolute path. Then use \"skip_prefix\" to make it align\nwith symref \"referent\".\n\nThus, the symrefs and symbolic links could share the same interface. Add\na new function \"files_fsck_symref_target\" which aims at checking the\nfollowing things:\n\n1. whether the pointee is under the `refs/` directory.\n2. whether the pointee name is correct.\n3. whether the pointee path is a wrong type in filesystem.\n\nLast, add the following FSCK MESSAGEs:\n\n1. \"badRefContent(ERROR)\": A ref has a bad content\n2. \"badSymrefPointee(ERROR)\": The pointee of a symref is bad.\n3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 9 +++\n fsck.h | 3 +\n refs.c | 2 +-\n refs/files-backend.c | 145 +++++++++++++++++++++++++++++++++-\n refs/refs-internal.h | 5 +-\n t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++\n 6 files changed, 269 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex dab4012246..b1630a478b 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,9 +19,15 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefContent`::\n+\t(ERROR) A ref has a bad content.\n+\n `badRefName`::\n \t(ERROR) A ref has a bad name.\n \n+`badSymrefPointee`::\n+\t(ERROR) The pointee of a symref is bad.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \n@@ -167,6 +173,9 @@\n `nullSha1`::\n \t(WARN) Tree contains entries pointing to a null sha1.\n \n+`trailingRefContent`::\n+\t(WARN) A ref content has trailing contents.\n+\n `treeNotSorted`::\n \t(ERROR) A tree is not properly sorted.\n \ndiff --git a/fsck.h b/fsck.h\nindex 90457d1a1f..637f596930 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -32,6 +32,8 @@ enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n+\tFUNC(BAD_REF_CONTENT, ERROR) \\\n+\tFUNC(BAD_SYMREF_POINTEE, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n@@ -72,6 +74,7 @@ enum fsck_msg_type {\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n+\tFUNC(TRAILING_REF_CONTENT, WARN) \\\n \tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n \tFUNC(LARGE_PATHNAME, WARN) \\\ndiff --git a/refs.c b/refs.c\nindex 410919246b..eb82fb7d4e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1760,7 +1760,7 @@ static int refs_read_special_head(struct ref_store *ref_store,\n \t}\n \n \tresult = parse_loose_ref_contents(content.buf, oid, referent, type,\n-\t\t\t\t\t failure_errno);\n+\t\t\t\t\t failure_errno, NULL);\n \n done:\n \tstrbuf_release(&full_path);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 69a76048d3..d98ef45403 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1,6 +1,7 @@\n #define USE_THE_REPOSITORY_VARIABLE\n \n #include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n #include \"../copy.h\"\n #include \"../environment.h\"\n #include \"../gettext.h\"\n@@ -553,7 +554,7 @@ static int read_ref_internal(struct ref_store *ref_store, const char *refname,\n \tstrbuf_rtrim(&sb_contents);\n \tbuf = sb_contents.buf;\n \n-\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr);\n+\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr, NULL);\n \n out:\n \tif (ret && !myerr)\n@@ -589,7 +590,7 @@ static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refn\n \n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno)\n+\t\t\t int *failure_errno, const char **trailing)\n {\n \tconst char *p;\n \tif (skip_prefix(buf, \"ref:\", &buf)) {\n@@ -611,6 +612,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t*failure_errno = EINVAL;\n \t\treturn -1;\n \t}\n+\n+\tif (trailing)\n+\t\t*trailing = p;\n+\n \treturn 0;\n }\n \n@@ -3438,6 +3443,141 @@ static int files_fsck_refs_name(struct fsck_options *o,\n \treturn ret;\n }\n \n+/*\n+ * Check the symref \"pointee_name\" and \"pointee_path\". The caller should\n+ * make sure that \"pointee_path\" is absolute. For symbolic ref, \"pointee_name\"\n+ * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n+ * be the relative path agaignst \"gitdir\".\n+ */\n+static int files_fsck_symref_target(struct fsck_options *o,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const char *pointee_name,\n+\t\t\t\t const char *pointee_path)\n+{\n+\tconst char *p = NULL;\n+\tstruct stat st;\n+\tint ret = 0;\n+\n+\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n+\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to target out of refs hierarchy\");\n+\t\tgoto out;\n+\t}\n+\n+\tif (check_refname_format(pointee_name, 0)) {\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid refname\");\n+\t}\n+\n+\tif (lstat(pointee_path, &st) < 0)\n+\t\tgoto out;\n+\n+\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid target\");\n+\t\tgoto out;\n+\t}\n+out:\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs_content(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter)\n+{\n+\tstruct strbuf pointee_path = STRBUF_INIT,\n+\t\t ref_content = STRBUF_INIT,\n+\t\t abs_gitdir = STRBUF_INIT,\n+\t\t referent = STRBUF_INIT,\n+\t\t refname = STRBUF_INIT;\n+\tconst char *trailing = NULL;\n+\tint failure_errno = 0;\n+\tunsigned int type = 0;\n+\tstruct object_id oid;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\n+\t/*\n+\t * If the file is a symlink, we need to only check the connectivity\n+\t * of the destination object.\n+\t */\n+\tif (S_ISLNK(iter->st.st_mode)) {\n+\t\tconst char *pointee_name = NULL;\n+\n+\t\tstrbuf_add_real_path(&pointee_path, iter->path.buf);\n+\n+\t\tstrbuf_add_absolute_path(&abs_gitdir, gitdir);\n+\t\tstrbuf_normalize_path(&abs_gitdir);\n+\t\tif (!is_dir_sep(abs_gitdir.buf[abs_gitdir.len - 1]))\n+\t\t\tstrbuf_addch(&abs_gitdir, '/');\n+\n+\t\tif (!skip_prefix(pointee_path.buf,\n+\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n+\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t\t \"point to target outside gitdir\");\n+\t\t\tgoto clean;\n+\t\t}\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, pointee_name,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n+\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n+\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n+\t\t\t\t &referent, &type,\n+\t\t\t\t &failure_errno, &trailing)) {\n+\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n+\t\t\t\t \"invalid ref content\");\n+\t\tgoto clean;\n+\t}\n+\n+\t/*\n+\t * If the ref is a symref, we need to check the destination name and\n+\t * connectivity.\n+\t */\n+\tif (referent.len && (type & REF_ISSYMREF)) {\n+\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n+\t\tstrbuf_rtrim(&referent);\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, referent.buf,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t} else {\n+\t\t/*\n+\t\t * Only regular refs could have a trailing garbage. Should\n+\t\t * be reported as a warning.\n+\t\t */\n+\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n+\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n+\t\t\t\t\t \"trailing garbage in ref\");\n+\t\t\tgoto clean;\n+\t\t}\n+\t}\n+\n+clean:\n+\tstrbuf_release(&abs_gitdir);\n+\tstrbuf_release(&pointee_path);\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&ref_content);\n+\tstrbuf_release(&referent);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3490,6 +3630,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n \t\tfiles_fsck_refs_name,\n+\t\tfiles_fsck_refs_content,\n \t\tNULL\n \t};\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex a905e187cd..2fabf41d14 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -709,11 +709,12 @@ struct ref_store {\n \n /*\n * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for\n- * invalid contents.\n+ * invalid contents. Also *trailing is set to the first character after the\n+ * refname or NULL if the referent is not empty.\n */\n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno);\n+\t\t\t int *failure_errno, const char **trailing);\n \n /*\n * Fill in the generic part of refs and add it to our collection of\ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nindex b2db58d2c6..35bf40ee64 100755\n--- a/t/t0602-reffiles-fsck.sh\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -98,4 +98,114 @@ test_expect_success 'ref name check should be adapted into fsck messages' '\n \t)\n '\n \n+test_expect_success 'regular ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit checkout -b a/b/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse branch-1)\" > $branch_dir_prefix/branch-1-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse tag-1)\" > $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_must_fail git -c fsck.trailingRefContent=error fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s \" \"$(git rev-parse tag-2)\" > $tag_dir_prefix/tag-2-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/tags/tag-2-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/a/b/branch-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'symbolic ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid refname\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid target\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: logs/maint-v2.45\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to target out of refs hierarchy\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.2\n\n"},{"id":"498262","messageId":"CAOLa=ZRPaiHK+NezigXEyWceriHZ7rb6msR51zcit9X=8rHFyQ@mail.gmail.com","threadId":"61565","inReplyTo":"ZovrFCzRg06pq5eI@ArchLinux","subject":"Re: [GSoC][PATCH v8 2/9] fsck: add a unified interface for reporting fsck messages","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-07-08T14:36:38Z","receivedAt":"2024-07-08T14:36:39Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> The static function \"report\" provided by \"fsck.c\" aims at checking fsck\n> error type and calling the callback \"error_func\" to report the message.\n> However, \"report\" function is only related to object database which\n> cannot be reused for refs. In order to provide a unified interface which\n> can report either objects or refs, create a new function \"vfsck_report\"\n> by adding \"checked_ref_name\" parameter following the \"report\" prototype.\n> Instead of using \"...\", provide \"va_list\" to allow more flexibility.\n>\n> Like \"report\", the \"vfsck_report\" function will use \"error_func\"\n> registered in \"fsck_options\" to report customized messages. Change\n> \"error_func\" prototype to align with the new \"vfsck_report\".\n>\n> Then, change \"report\" function to use \"vfsck_report\" to report objects\n> related messages. Add a new function called \"fsck_refs_report\" to use\n> \"vfsck_report\" to report refs related messages.\n>\n\nNot sure I really understand why we need to do this. Why can't we simply\nadd `const char *checked_ref_name` to the existing 'report' and\npropagate this also to 'error_func'. Why do we need all this parallel\nflows?\n\nApart from that, what does 'v' in 'vfsck_report' signify?\n\nPerhaps it is also because this commit is doing a lot of things and we\ncould have simplified it into smaller commits?\n\n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n> builtin/fsck.c | 15 ++++-----\n> builtin/mktag.c | 1 +\n> fsck.c | 81 ++++++++++++++++++++++++++++++++++++-------------\n> fsck.h | 42 ++++++++++++++++---------\n> object-file.c | 11 ++++---\n> 5 files changed, 102 insertions(+), 48 deletions(-)\n>\n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index d13a226c2e..de34538c4f 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -89,12 +89,13 @@ static int objerror(struct object *obj, const char *err)\n> \treturn -1;\n> }\n>\n> -static int fsck_error_func(struct fsck_options *o UNUSED,\n> -\t\t\t const struct object_id *oid,\n> -\t\t\t enum object_type object_type,\n> -\t\t\t enum fsck_msg_type msg_type,\n> -\t\t\t enum fsck_msg_id msg_id UNUSED,\n> -\t\t\t const char *message)\n> +static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n> +\t\t\t\t const struct object_id *oid,\n> +\t\t\t\t enum object_type object_type,\n> +\t\t\t\t const char *checked_ref_name UNUSED,\n> +\t\t\t\t enum fsck_msg_type msg_type,\n> +\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> +\t\t\t\t const char *message)\n> {\n> \tswitch (msg_type) {\n> \tcase FSCK_WARN:\n> @@ -938,7 +939,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n>\n> \tfsck_walk_options.walk = mark_object;\n> \tfsck_obj_options.walk = mark_used;\n> -\tfsck_obj_options.error_func = fsck_error_func;\n> +\tfsck_obj_options.error_func = fsck_objects_error_func;\n> \tif (check_strict)\n> \t\tfsck_obj_options.strict = 1;\n>\n> diff --git a/builtin/mktag.c b/builtin/mktag.c\n> index 4767f1a97e..42f945c584 100644\n> --- a/builtin/mktag.c\n> +++ b/builtin/mktag.c\n> @@ -20,6 +20,7 @@ static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n> static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n> \t\t\t\t const struct object_id *oid UNUSED,\n> \t\t\t\t enum object_type object_type UNUSED,\n> +\t\t\t\t const char *checked_ref_name UNUSED,\n> \t\t\t\t enum fsck_msg_type msg_type,\n> \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> \t\t\t\t const char *message)\n> diff --git a/fsck.c b/fsck.c\n> index 1960bfeba9..7182ce8e80 100644\n> --- a/fsck.c\n> +++ b/fsck.c\n> @@ -226,12 +226,18 @@ static int object_on_skiplist(struct fsck_options *opts,\n> \treturn opts && oid && oidset_contains(&opts->oid_skiplist, oid);\n> }\n>\n> -__attribute__((format (printf, 5, 6)))\n> -static int report(struct fsck_options *options,\n> -\t\t const struct object_id *oid, enum object_type object_type,\n> -\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n> +/*\n> + * Provide a unified interface for either fscking refs or objects.\n> + * It will get the current msg error type and call the error_func callback\n> + * which is registered in the \"fsck_options\" struct.\n> + */\n> +static int vfsck_report(struct fsck_options *options,\n> +\t\t\tconst struct object_id *oid,\n> +\t\t\tenum object_type object_type,\n> +\t\t\tconst char *checked_ref_name,\n> +\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n> {\n> -\tva_list ap;\n> +\tva_list ap_copy;\n> \tstruct strbuf sb = STRBUF_INIT;\n> \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n> \tint result;\n> @@ -250,9 +256,9 @@ static int report(struct fsck_options *options,\n> \tprepare_msg_ids();\n> \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n>\n> -\tva_start(ap, fmt);\n> -\tstrbuf_vaddf(&sb, fmt, ap);\n> -\tresult = options->error_func(options, oid, object_type,\n> +\tva_copy(ap_copy, ap);\n> +\tstrbuf_vaddf(&sb, fmt, ap_copy);\n> +\tresult = options->error_func(options, oid, object_type, checked_ref_name,\n> \t\t\t\t msg_type, msg_id, sb.buf);\n> \tstrbuf_release(&sb);\n> \tva_end(ap);\n> @@ -260,6 +266,36 @@ static int report(struct fsck_options *options,\n> \treturn result;\n> }\n>\n> +__attribute__((format (printf, 5, 6)))\n>\n\nShouldn't this be moved to the header file too?\n\n> +static int report(struct fsck_options *options,\n> +\t\t const struct object_id *oid, enum object_type object_type,\n> +\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n> +{\n> +\tva_list ap;\n> +\tint result;\n> +\tva_start(ap, fmt);\n> +\tresult = vfsck_report(options, oid, object_type, NULL,\n> +\t\t\t msg_id, fmt, ap);\n> +\tva_end(ap);\n> +\treturn result;\n> +}\n> +\n> +\n> +\n\nThere is an extra newline here.\n\n> +int fsck_refs_report(struct fsck_options *options,\n> +\t\t const struct object_id *oid,\n> +\t\t const char *checked_ref_name,\n> +\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n> +{\n> +\tva_list ap;\n> +\tint result;\n> +\tva_start(ap, fmt);\n> +\tresult = vfsck_report(options, oid, OBJ_NONE,\n> +\t\t\t checked_ref_name, msg_id, fmt, ap);\n> +\tva_end(ap);\n> +\treturn result;\n> +}\n> +\n> void fsck_enable_object_names(struct fsck_options *options)\n> {\n> \tif (!options->object_names)\n> @@ -1200,12 +1236,13 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n> \t\t type);\n> }\n>\n> -int fsck_error_function(struct fsck_options *o,\n> -\t\t\tconst struct object_id *oid,\n> -\t\t\tenum object_type object_type UNUSED,\n> -\t\t\tenum fsck_msg_type msg_type,\n> -\t\t\tenum fsck_msg_id msg_id UNUSED,\n> -\t\t\tconst char *message)\n> +int fsck_objects_error_function(struct fsck_options *o,\n> +\t\t\t\tconst struct object_id *oid,\n> +\t\t\t\tenum object_type object_type UNUSED,\n> +\t\t\t\tconst char *checked_ref_name UNUSED,\n> +\t\t\t\tenum fsck_msg_type msg_type,\n> +\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n> +\t\t\t\tconst char *message)\n> {\n> \tif (msg_type == FSCK_WARN) {\n> \t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n> @@ -1303,16 +1340,18 @@ int git_fsck_config(const char *var, const char *value,\n> * Custom error callbacks that are used in more than one place.\n> */\n>\n> -int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n> -\t\t\t\t\t const struct object_id *oid,\n> -\t\t\t\t\t enum object_type object_type,\n> -\t\t\t\t\t enum fsck_msg_type msg_type,\n> -\t\t\t\t\t enum fsck_msg_id msg_id,\n> -\t\t\t\t\t const char *message)\n> +int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n> +\t\t\t\t\t\t const struct object_id *oid,\n> +\t\t\t\t\t\t enum object_type object_type,\n> +\t\t\t\t\t\t const char *checked_ref_name,\n> +\t\t\t\t\t\t enum fsck_msg_type msg_type,\n> +\t\t\t\t\t\t enum fsck_msg_id msg_id,\n> +\t\t\t\t\t\t const char *message)\n> {\n> \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n> \t\tputs(oid_to_hex(oid));\n> \t\treturn 0;\n> \t}\n> -\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n> +\treturn fsck_objects_error_function(o, oid, object_type, checked_ref_name,\n> +\t\t\t\t\t msg_type, msg_id, message);\n> }\n> diff --git a/fsck.h b/fsck.h\n> index 1ee3dd85ba..f703dfb5e8 100644\n> --- a/fsck.h\n> +++ b/fsck.h\n> @@ -114,22 +114,27 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n> typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n> \t\t\t void *data, struct fsck_options *options);\n>\n> -/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n> +/*\n> + * callback function for reporting errors when checking either objects or refs\n> + */\n> typedef int (*fsck_error)(struct fsck_options *o,\n> \t\t\t const struct object_id *oid, enum object_type object_type,\n> +\t\t\t const char *checked_ref_name,\n> \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n> \t\t\t const char *message);\n>\n> -int fsck_error_function(struct fsck_options *o,\n> -\t\t\tconst struct object_id *oid, enum object_type object_type,\n> -\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n> -\t\t\tconst char *message);\n> -int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n> -\t\t\t\t\t const struct object_id *oid,\n> -\t\t\t\t\t enum object_type object_type,\n> -\t\t\t\t\t enum fsck_msg_type msg_type,\n> -\t\t\t\t\t enum fsck_msg_id msg_id,\n> -\t\t\t\t\t const char *message);\n> +int fsck_objects_error_function(struct fsck_options *o,\n> +\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n> +\t\t\t\tconst char *checked_ref_name,\n> +\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n> +\t\t\t\tconst char *message);\n> +int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n> +\t\t\t\t\t\t const struct object_id *oid,\n> +\t\t\t\t\t\t enum object_type object_type,\n> +\t\t\t\t\t\t const char *checked_ref_name,\n> +\t\t\t\t\t\t enum fsck_msg_type msg_type,\n> +\t\t\t\t\t\t enum fsck_msg_id msg_id,\n> +\t\t\t\t\t\t const char *message);\n>\n> struct fsck_options {\n> \tfsck_walk_func walk;\n> @@ -145,12 +150,12 @@ struct fsck_options {\n> };\n>\n> #define FSCK_OPTIONS_DEFAULT { \\\n> -\t.skiplist = OIDSET_INIT, \\\n> +\t.oid_skiplist = OIDSET_INIT, \\\n> \t.gitmodules_found = OIDSET_INIT, \\\n> \t.gitmodules_done = OIDSET_INIT, \\\n> \t.gitattributes_found = OIDSET_INIT, \\\n> \t.gitattributes_done = OIDSET_INIT, \\\n> -\t.error_func = fsck_error_function \\\n> +\t.error_func = fsck_objects_error_function \\\n> }\n> #define FSCK_OPTIONS_STRICT { \\\n> \t.strict = 1, \\\n> @@ -158,7 +163,7 @@ struct fsck_options {\n> \t.gitmodules_done = OIDSET_INIT, \\\n> \t.gitattributes_found = OIDSET_INIT, \\\n> \t.gitattributes_done = OIDSET_INIT, \\\n> -\t.error_func = fsck_error_function, \\\n> +\t.error_func = fsck_objects_error_function, \\\n> }\n> #define FSCK_OPTIONS_MISSING_GITMODULES { \\\n> \t.strict = 1, \\\n> @@ -166,7 +171,7 @@ struct fsck_options {\n> \t.gitmodules_done = OIDSET_INIT, \\\n> \t.gitattributes_found = OIDSET_INIT, \\\n> \t.gitattributes_done = OIDSET_INIT, \\\n> -\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n> +\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n> }\n>\n> /* descend in all linked child objects\n> @@ -209,6 +214,13 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n> */\n> int fsck_finish(struct fsck_options *options);\n>\n> +__attribute__((format (printf, 5, 6)))\n> +int fsck_refs_report(struct fsck_options *options,\n> +\t\t const struct object_id *oid,\n> +\t\t const char *checked_ref_name,\n> +\t\t enum fsck_msg_id msg_id,\n> +\t\t const char *fmt, ...);\n> +\n> /*\n> * Subsystem for storing human-readable names for each object.\n> *\n> diff --git a/object-file.c b/object-file.c\n> index 065103be3e..d2c6427935 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -2470,11 +2470,12 @@ int repo_has_object_file(struct repository *r,\n> * give more context.\n> */\n> static int hash_format_check_report(struct fsck_options *opts UNUSED,\n> -\t\t\t\t const struct object_id *oid UNUSED,\n> -\t\t\t\t enum object_type object_type UNUSED,\n> -\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n> -\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> -\t\t\t\t const char *message)\n> +\t\t\t\t const struct object_id *oid UNUSED,\n> +\t\t\t\t enum object_type object_type UNUSED,\n> +\t\t\t\t const char *ref_checked_name UNUSED,\n> +\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n> +\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> +\t\t\t\t const char *message)\n> {\n> \terror(_(\"object fails fsck: %s\"), message);\n> \treturn 1;\n> --\n> 2.45.2\n"},{"id":"498263","messageId":"CAOLa=ZSgZMmfJWcPdKYyLrO-Mo0G-TKj3pYRREjvLt_ypuEk6g@mail.gmail.com","threadId":"61565","inReplyTo":"ZovrI1yRmS0X2BHu@ArchLinux","subject":"Re: [GSoC][PATCH v8 3/9] fsck: add refs-related options and error report function","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-07-08T14:49:43Z","receivedAt":"2024-07-08T14:49:45Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> Add refs-related options to the \"fsck_options\", create refs-specific\n> \"error_func\" callback \"fsck_refs_error_function\".\n>\n> \"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\n> passes the oid, it will use \"oid_to_hex\" to get the corresponding hex\n> value to report to the caller.\n>\n> Last, add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\"\n> macros to create refs options easily.\n>\n\nCarrying over from the previous commit, couldn't we simply do something\nlike:\n\n diff --git a/fsck.c b/fsck.c\n index eea7145470..32ae36a4fc 100644\n --- a/fsck.c\n +++ b/fsck.c\n @@ -1202,17 +1203,33 @@ int fsck_buffer(const struct object_id\n*oid, enum object_type type,\n\n int fsck_error_function(struct fsck_options *o,\n \t\t\tconst struct object_id *oid,\n -\t\t\tenum object_type object_type UNUSED,\n +\t\t\tenum object_type object_type,\n +\t\t\tconst char *checked_ref_name,\n \t\t\tenum fsck_msg_type msg_type,\n \t\t\tenum fsck_msg_id msg_id UNUSED,\n \t\t\tconst char *message)\n {\n +\tstatic struct strbuf sb = STRBUF_INIT;\n +\tint ret = 0;\n +\n +\tif (checked_ref_name) {\n +\t\tstrbuf_addstr(\"ref %s\", checked_ref_name);\n +\t\tif (oid)\n +\t\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n +\t} else {\n +\t\tstrbuf_addstr(\"object %s\", fsck_describe_object(o, oid));\n +\t}\n +\n \tif (msg_type == FSCK_WARN) {\n -\t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n -\t\treturn 0;\n +\t\twarning(\"%s: %s\", sb.buf, message);\n +\t\tret = 0;\n +\t\tgoto cleanup;\n \t}\n -\terror(\"object %s: %s\", fsck_describe_object(o, oid), message);\n -\treturn 1;\n +\terror(\"%s: %s\", sb.buf, message);\n +\n +cleanup:\n +\tstrbuf_release(&sb);\n +\treturn ret;\n }\n\n\n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n> fsck.c | 22 ++++++++++++++++++++++\n> fsck.h | 15 +++++++++++++++\n> 2 files changed, 37 insertions(+)\n>\n> diff --git a/fsck.c b/fsck.c\n> index 7182ce8e80..d1dcbdcac2 100644\n> --- a/fsck.c\n> +++ b/fsck.c\n> @@ -1252,6 +1252,28 @@ int fsck_objects_error_function(struct fsck_options *o,\n> \treturn 1;\n> }\n>\n> +int fsck_refs_error_function(struct fsck_options *options UNUSED,\n> +\t\t\t const struct object_id *oid,\n> +\t\t\t enum object_type object_type UNUSED,\n> +\t\t\t const char *checked_ref_name,\n> +\t\t\t enum fsck_msg_type msg_type,\n> +\t\t\t enum fsck_msg_id msg_id UNUSED,\n> +\t\t\t const char *message)\n> +{\n> +\tstatic struct strbuf sb = STRBUF_INIT;\n> +\n> +\tstrbuf_addstr(&sb, checked_ref_name);\n> +\tif (oid)\n> +\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n> +\n> +\tif (msg_type == FSCK_WARN) {\n> +\t\twarning(\"%s: %s\", sb.buf, message);\n> +\t\treturn 0;\n> +\t}\n> +\terror(\"%s: %s\", sb.buf, message);\n> +\treturn 1;\n> +}\n> +\n\nWe don't free strbuf here.\n\n> static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n> \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n> \t\t struct fsck_options *options, const char *blob_type)\n> diff --git a/fsck.h b/fsck.h\n> index f703dfb5e8..246055c0f9 100644\n> --- a/fsck.h\n> +++ b/fsck.h\n> @@ -135,11 +135,19 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n> \t\t\t\t\t\t enum fsck_msg_type msg_type,\n> \t\t\t\t\t\t enum fsck_msg_id msg_id,\n> \t\t\t\t\t\t const char *message);\n> +int fsck_refs_error_function(struct fsck_options *options,\n> +\t\t\t const struct object_id *oid,\n> +\t\t\t enum object_type object_type,\n> +\t\t\t const char *checked_ref_name,\n> +\t\t\t enum fsck_msg_type msg_type,\n> +\t\t\t enum fsck_msg_id msg_id,\n> +\t\t\t const char *message);\n>\n> struct fsck_options {\n> \tfsck_walk_func walk;\n> \tfsck_error error_func;\n> \tunsigned strict:1;\n> +\tunsigned verbose_refs:1;\n\nNit: Here we have the subject 'refs' towards the end of the name.\n\n> \tenum fsck_msg_type *msg_type;\n> \tstruct oidset oid_skiplist;\n\nbut here we have the subject 'oid' at the start of the name. Perhaps we\ncan rename this to 'skip_oids'?\n\n> \tstruct oidset gitmodules_found;\n> @@ -173,6 +181,13 @@ struct fsck_options {\n> \t.gitattributes_done = OIDSET_INIT, \\\n> \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n> }\n> +#define FSCK_REFS_OPTIONS_DEFAULT { \\\n> +\t.error_func = fsck_refs_error_function, \\\n> +}\n> +#define FSCK_REFS_OPTIONS_STRICT { \\\n> +\t.strict = 1, \\\n> +\t.error_func = fsck_refs_error_function, \\\n> +}\n>\n> /* descend in all linked child objects\n> * the return value is:\n> --\n> 2.45.2\n"},{"id":"498264","messageId":"Zov_PaaxF1Mbp_dP@ArchLinux","threadId":"61565","inReplyTo":"CAOLa=ZRPaiHK+NezigXEyWceriHZ7rb6msR51zcit9X=8rHFyQ@mail.gmail.com","subject":"Re: [GSoC][PATCH v8 2/9] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-08T15:01:17Z","receivedAt":"2024-07-08T15:01:19Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Mon, Jul 08, 2024 at 10:36:38AM -0400, Karthik Nayak wrote:\n> shejialuo writes:\n> \n> > The static function \"report\" provided by \"fsck.c\" aims at checking fsck\n> > error type and calling the callback \"error_func\" to report the message.\n> > However, \"report\" function is only related to object database which\n> > cannot be reused for refs. In order to provide a unified interface which\n> > can report either objects or refs, create a new function \"vfsck_report\"\n> > by adding \"checked_ref_name\" parameter following the \"report\" prototype.\n> > Instead of using \"...\", provide \"va_list\" to allow more flexibility.\n> >\n> > Like \"report\", the \"vfsck_report\" function will use \"error_func\"\n> > registered in \"fsck_options\" to report customized messages. Change\n> > \"error_func\" prototype to align with the new \"vfsck_report\".\n> >\n> > Then, change \"report\" function to use \"vfsck_report\" to report objects\n> > related messages. Add a new function called \"fsck_refs_report\" to use\n> > \"vfsck_report\" to report refs related messages.\n> >\n> \n> Not sure I really understand why we need to do this. Why can't we simply\n> add `const char *checked_ref_name` to the existing 'report' and\n> propagate this also to 'error_func'. Why do we need all this parallel\n> flows?\n> \n\nYes, we could just add a parameter \"const char *checked_ref_name\" to the\nexisting \"report\". This may seem the simplest way to do. However, it\nwill also introduce some trouble below:\n\n1. \"report\" function should be exported to the outside, we need to\nrename it to \"fsck_report\". Well, we need to change a lot of code here.\nAnd we MUST do this, because \"report\" is a general name. When exporting\nto the outside, it's not proper.\n2. When we add a new parameter in \"report\", for all the \"report\" calls,\nwe need to pass this new parameter with NULL.\n\nUse this way, we could do not change \"report\" function prototype and the\ncorresponding calls. Most importantly, we could let the caller feel\ntransparent. Using \"report\", caller can just ignore \"checked_ref_name\".\nAlso for \"fsck_refs_report\", we could ignore some UNUSED parameters.\n\nSo I think this design is more elegant than just adding a new parameter\nin the existing \"report\" function.\n\n> Apart from that, what does 'v' in 'vfsck_report' signify?\n> \n\nBecause I use \"va_list\" parameter, I want to emphasis on this. And this\nprovides flexibility that we could add a \"fsck_report\" function later.\nThere are many codes in git code base using this way. I just followed\nthis.\n\n> Perhaps it is also because this commit is doing a lot of things and we\n> could have simplified it into smaller commits?\n> \n\nActually, this commit is very clear. I just want to provide a unified\nfunction \"vfsck_report\" here. And let the \"report\" use this function and\n\"fsck_refs_report\" function use this.\n\nSo I don't know whether we should split this commit into multiple\ncommits. They are just tied together.\n"},{"id":"498273","messageId":"ZowGjztkmQnUkD5w@ArchLinux","threadId":"61565","inReplyTo":"CAOLa=ZSgZMmfJWcPdKYyLrO-Mo0G-TKj3pYRREjvLt_ypuEk6g@mail.gmail.com","subject":"Re: [GSoC][PATCH v8 3/9] fsck: add refs-related options and error report function","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-08T15:32:31Z","receivedAt":"2024-07-08T15:32:33Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Mon, Jul 08, 2024 at 07:49:43AM -0700, Karthik Nayak wrote:\n> shejialuo writes:\n> \n> > Add refs-related options to the \"fsck_options\", create refs-specific\n> > \"error_func\" callback \"fsck_refs_error_function\".\n> >\n> > \"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\n> > passes the oid, it will use \"oid_to_hex\" to get the corresponding hex\n> > value to report to the caller.\n> >\n> > Last, add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\"\n> > macros to create refs options easily.\n> >\n> \n> Carrying over from the previous commit, couldn't we simply do something\n> like:\n> \n> diff --git a/fsck.c b/fsck.c\n> index eea7145470..32ae36a4fc 100644\n> --- a/fsck.c\n> +++ b/fsck.c\n> @@ -1202,17 +1203,33 @@ int fsck_buffer(const struct object_id\n> *oid, enum object_type type,\n> \n> int fsck_error_function(struct fsck_options *o,\n> \t\t\tconst struct object_id *oid,\n> -\t\t\tenum object_type object_type UNUSED,\n> +\t\t\tenum object_type object_type,\n> +\t\t\tconst char *checked_ref_name,\n> \t\t\tenum fsck_msg_type msg_type,\n> \t\t\tenum fsck_msg_id msg_id UNUSED,\n> \t\t\tconst char *message)\n> {\n> +\tstatic struct strbuf sb = STRBUF_INIT;\n> +\tint ret = 0;\n> +\n> +\tif (checked_ref_name) {\n> +\t\tstrbuf_addstr(\"ref %s\", checked_ref_name);\n> +\t\tif (oid)\n> +\t\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n> +\t} else {\n> +\t\tstrbuf_addstr(\"object %s\", fsck_describe_object(o, oid));\n> +\t}\n> +\n> \tif (msg_type == FSCK_WARN) {\n> -\t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n> -\t\treturn 0;\n> +\t\twarning(\"%s: %s\", sb.buf, message);\n> +\t\tret = 0;\n> +\t\tgoto cleanup;\n> \t}\n> -\terror(\"object %s: %s\", fsck_describe_object(o, oid), message);\n> -\treturn 1;\n> +\terror(\"%s: %s\", sb.buf, message);\n> +\n> +cleanup:\n> +\tstrbuf_release(&sb);\n> +\treturn ret;\n> }\n> \n> \n\nHowever, \"fsck_error_function\" will be used as the callback for\nobject-related checks. I don't think we should use one function to\nincorporate all the situations.\n\nFor example, if we pass both \"checked_ref_name\" and \"oid\" here, how does\nthis one function knows whether we handle refs or objects. Although we\nmay use some flags here to choose the different situations, it's bad\nidea from my perspective. Adding a new callback here will be cleaner.\n\n> > Mentored-by: Patrick Steinhardt \n> > Mentored-by: Karthik Nayak \n> > Signed-off-by: shejialuo \n> > ---\n> > fsck.c | 22 ++++++++++++++++++++++\n> > fsck.h | 15 +++++++++++++++\n> > 2 files changed, 37 insertions(+)\n> >\n> > diff --git a/fsck.c b/fsck.c\n> > index 7182ce8e80..d1dcbdcac2 100644\n> > --- a/fsck.c\n> > +++ b/fsck.c\n> > @@ -1252,6 +1252,28 @@ int fsck_objects_error_function(struct fsck_options *o,\n> > \treturn 1;\n> > }\n> >\n> > +int fsck_refs_error_function(struct fsck_options *options UNUSED,\n> > +\t\t\t const struct object_id *oid,\n> > +\t\t\t enum object_type object_type UNUSED,\n> > +\t\t\t const char *checked_ref_name,\n> > +\t\t\t enum fsck_msg_type msg_type,\n> > +\t\t\t enum fsck_msg_id msg_id UNUSED,\n> > +\t\t\t const char *message)\n> > +{\n> > +\tstatic struct strbuf sb = STRBUF_INIT;\n> > +\n> > +\tstrbuf_addstr(&sb, checked_ref_name);\n> > +\tif (oid)\n> > +\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n> > +\n> > +\tif (msg_type == FSCK_WARN) {\n> > +\t\twarning(\"%s: %s\", sb.buf, message);\n> > +\t\treturn 0;\n> > +\t}\n> > +\terror(\"%s: %s\", sb.buf, message);\n> > +\treturn 1;\n> > +}\n> > +\n> \n> We don't free strbuf here.\n> \n\nYes, I made a mistake here. I should not define the \"sb\" static, will\nfix it in the next version.\n\n> > static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n> > \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n> > \t\t struct fsck_options *options, const char *blob_type)\n> > diff --git a/fsck.h b/fsck.h\n> > index f703dfb5e8..246055c0f9 100644\n> > --- a/fsck.h\n> > +++ b/fsck.h\n> > @@ -135,11 +135,19 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n> > \t\t\t\t\t\t enum fsck_msg_type msg_type,\n> > \t\t\t\t\t\t enum fsck_msg_id msg_id,\n> > \t\t\t\t\t\t const char *message);\n> > +int fsck_refs_error_function(struct fsck_options *options,\n> > +\t\t\t const struct object_id *oid,\n> > +\t\t\t enum object_type object_type,\n> > +\t\t\t const char *checked_ref_name,\n> > +\t\t\t enum fsck_msg_type msg_type,\n> > +\t\t\t enum fsck_msg_id msg_id,\n> > +\t\t\t const char *message);\n> >\n> > struct fsck_options {\n> > \tfsck_walk_func walk;\n> > \tfsck_error error_func;\n> > \tunsigned strict:1;\n> > +\tunsigned verbose_refs:1;\n> \n> Nit: Here we have the subject 'refs' towards the end of the name.\n> \n> > \tenum fsck_msg_type *msg_type;\n> > \tstruct oidset oid_skiplist;\n> \n> but here we have the subject 'oid' at the start of the name. Perhaps we\n> can rename this to 'skip_oids'?\n> \n\nI agree. \"skip_oids\" will be more meaningful. I will fix it in the next\nversion.\n\n> > \tstruct oidset gitmodules_found;\n> > @@ -173,6 +181,13 @@ struct fsck_options {\n> > \t.gitattributes_done = OIDSET_INIT, \\\n> > \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n> > }\n> > +#define FSCK_REFS_OPTIONS_DEFAULT { \\\n> > +\t.error_func = fsck_refs_error_function, \\\n> > +}\n> > +#define FSCK_REFS_OPTIONS_STRICT { \\\n> > +\t.strict = 1, \\\n> > +\t.error_func = fsck_refs_error_function, \\\n> > +}\n> >\n> > /* descend in all linked child objects\n> > * the return value is:\n> > --\n> > 2.45.2\n\n"},{"id":"498292","messageId":"CAOLa=ZR5mGOVsf8RSKxvQkT-9Ndm40CNuSEOY3Pbeu-2F0uqew@mail.gmail.com","threadId":"61565","inReplyTo":"Zov_PaaxF1Mbp_dP@ArchLinux","subject":"Re: [GSoC][PATCH v8 2/9] fsck: add a unified interface for reporting fsck messages","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-07-08T17:11:43Z","receivedAt":"2024-07-08T17:11:45Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> On Mon, Jul 08, 2024 at 10:36:38AM -0400, Karthik Nayak wrote:\n>> shejialuo writes:\n>>\n>> > The static function \"report\" provided by \"fsck.c\" aims at checking fsck\n>> > error type and calling the callback \"error_func\" to report the message.\n>> > However, \"report\" function is only related to object database which\n>> > cannot be reused for refs. In order to provide a unified interface which\n>> > can report either objects or refs, create a new function \"vfsck_report\"\n>> > by adding \"checked_ref_name\" parameter following the \"report\" prototype.\n>> > Instead of using \"...\", provide \"va_list\" to allow more flexibility.\n>> >\n>> > Like \"report\", the \"vfsck_report\" function will use \"error_func\"\n>> > registered in \"fsck_options\" to report customized messages. Change\n>> > \"error_func\" prototype to align with the new \"vfsck_report\".\n>> >\n>> > Then, change \"report\" function to use \"vfsck_report\" to report objects\n>> > related messages. Add a new function called \"fsck_refs_report\" to use\n>> > \"vfsck_report\" to report refs related messages.\n>> >\n>>\n>> Not sure I really understand why we need to do this. Why can't we simply\n>> add `const char *checked_ref_name` to the existing 'report' and\n>> propagate this also to 'error_func'. Why do we need all this parallel\n>> flows?\n>>\n>\n> Yes, we could just add a parameter \"const char *checked_ref_name\" to the\n> existing \"report\". This may seem the simplest way to do. However, it\n> will also introduce some trouble below:\n>\n> 1. \"report\" function should be exported to the outside, we need to\n> rename it to \"fsck_report\". Well, we need to change a lot of code here.\n> And we MUST do this, because \"report\" is a general name. When exporting\n> to the outside, it's not proper.\n>\n\nagreed.\n\n> 2. When we add a new parameter in \"report\", for all the \"report\" calls,\n> we need to pass this new parameter with NULL.\n>\n\nagreed too.\n\n> Use this way, we could do not change \"report\" function prototype and the\n> corresponding calls. Most importantly, we could let the caller feel\n> transparent. Using \"report\", caller can just ignore \"checked_ref_name\".\n> Also for \"fsck_refs_report\", we could ignore some UNUSED parameters.\n>\n> So I think this design is more elegant than just adding a new parameter\n> in the existing \"report\" function.\n>\n\nI understand what you're saying, I also checked and can see that there\nare 60 references to the `report()` function. So perhaps there is some\nmerit in keeping it as is and adding a new 'report_refs()'.\n\n>> Apart from that, what does 'v' in 'vfsck_report' signify?\n>>\n>\n> Because I use \"va_list\" parameter, I want to emphasis on this. And this\n> provides flexibility that we could add a \"fsck_report\" function later.\n> There are many codes in git code base using this way. I just followed\n> this.\n>\n\nI see. Makes sense.\n\n>> Perhaps it is also because this commit is doing a lot of things and we\n>> could have simplified it into smaller commits?\n>>\n>\n> Actually, this commit is very clear. I just want to provide a unified\n> function \"vfsck_report\" here. And let the \"report\" use this function and\n> \"fsck_refs_report\" function use this.\n>\n> So I don't know whether we should split this commit into multiple\n> commits. They are just tied together.\n"},{"id":"498355","messageId":"Zo0sQCBqyxX8dJ-f@ArchLinux","threadId":"61565","inReplyTo":"ZoVX6sn2C9VIeZ38@ArchLinux","subject":"[GSoC][PATCH v9 0/9] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-09T12:32:54Z","receivedAt":"2024-07-09T12:32:58Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Hi All:\n\nThis version changes the following things:\n\n1. Instead of using \"oid_skiplist\", use \"skip_oids\".\n2. In \"fsck_refs_error_function\", I define a static struct strbuf here,\nthe reason why I define it to be `static` is that we may call this\nfunction many times. It's a bad idea to allocate memory too frequently.\nAnd we should never free its memory. I made a mistake here, I should\ncall `strbuf_reset` function every time. So this version, add\n\"strbuf_reset\" to make the behavior correct.\n\nshejialuo (9):\n fsck: rename \"skiplist\" to \"skip_oids\"\n fsck: add a unified interface for reporting fsck messages\n fsck: add refs-related options and error report function\n refs: set up ref consistency check infrastructure\n builtin/refs: add verify subcommand\n builtin/fsck: add `git-refs verify` child process\n files-backend: add unified interface for refs scanning\n fsck: add ref name check for files backend\n fsck: add ref content check for files backend\n\n Documentation/fsck-msgids.txt | 12 ++\n Documentation/git-refs.txt | 13 ++\n builtin/fsck.c | 32 ++++-\n builtin/mktag.c | 1 +\n builtin/refs.c | 44 ++++++\n fsck.c | 108 +++++++++++---\n fsck.h | 63 ++++++---\n object-file.c | 11 +-\n refs.c | 7 +-\n refs.h | 8 ++\n refs/debug.c | 11 ++\n refs/files-backend.c | 255 +++++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 ++\n refs/refs-internal.h | 11 +-\n refs/reftable-backend.c | 8 ++\n t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++\n 16 files changed, 746 insertions(+), 57 deletions(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\nRange-diff against v8:\n 1: 61e475840f ! 1: e044f933de fsck: rename \"skiplist\" to \"oid_skiplist\"\n @@ Metadata\n Author: shejialuo \n \n ## Commit message ##\n - fsck: rename \"skiplist\" to \"oid_skiplist\"\n + fsck: rename \"skiplist\" to \"skip_oids\"\n \n The \"skiplist\" field in \"fsck_options\" is related to objects. Because we\n are going to introduce ref consistency check, the \"skiplist\" name is too\n @@ Commit message\n \"skiplist\" here. However, the type for \"skiplist\" is `struct oidset`\n which is totally unsuitable for refs.\n \n - To avoid above ambiguity, rename \"skiplist\" to \"oid_skiplist\".\n + To avoid above ambiguity, rename \"skiplist\" to \"skip_oids\".\n \n Mentored-by: Patrick Steinhardt \n Mentored-by: Karthik Nayak \n @@ fsck.c: void fsck_set_msg_types(struct fsck_options *options, const char *values\n \t\t\tif (equal == len)\n \t\t\t\tdie(\"skiplist requires a path\");\n -\t\t\toidset_parse_file(&options->skiplist, buf + equal + 1,\n -+\t\t\toidset_parse_file(&options->oid_skiplist, buf + equal + 1,\n ++\t\t\toidset_parse_file(&options->skip_oids, buf + equal + 1,\n \t\t\t\t\t the_repository->hash_algo);\n \t\t\tbuf += len + 1;\n \t\t\tcontinue;\n @@ fsck.c: void fsck_set_msg_types(struct fsck_options *options, const char *values\n \t\t\t const struct object_id *oid)\n {\n -\treturn opts && oid && oidset_contains(&opts->skiplist, oid);\n -+\treturn opts && oid && oidset_contains(&opts->oid_skiplist, oid);\n ++\treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n __attribute__((format (printf, 5, 6)))\n @@ fsck.h: struct fsck_options {\n \tunsigned strict:1;\n \tenum fsck_msg_type *msg_type;\n -\tstruct oidset skiplist;\n -+\tstruct oidset oid_skiplist;\n ++\tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n \tstruct oidset gitattributes_found;\n +@@ fsck.h: struct fsck_options {\n + };\n + \n + #define FSCK_OPTIONS_DEFAULT { \\\n +-\t.skiplist = OIDSET_INIT, \\\n ++\t.skip_oids = OIDSET_INIT, \\\n + \t.gitmodules_found = OIDSET_INIT, \\\n + \t.gitmodules_done = OIDSET_INIT, \\\n + \t.gitattributes_found = OIDSET_INIT, \\\n 2: f2576d88a9 ! 2: daaf3d0ffe fsck: add a unified interface for reporting fsck messages\n @@ builtin/mktag.c: static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n \n ## fsck.c ##\n @@ fsck.c: static int object_on_skiplist(struct fsck_options *opts,\n - \treturn opts && oid && oidset_contains(&opts->oid_skiplist, oid);\n + \treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n -__attribute__((format (printf, 5, 6)))\n @@ fsck.h: int is_valid_msg_type(const char *msg_id, const char *msg_type);\n struct fsck_options {\n \tfsck_walk_func walk;\n @@ fsck.h: struct fsck_options {\n - };\n - \n - #define FSCK_OPTIONS_DEFAULT { \\\n --\t.skiplist = OIDSET_INIT, \\\n -+\t.oid_skiplist = OIDSET_INIT, \\\n - \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n 3: c3c2dda50c ! 3: 40da85ae30 fsck: add refs-related options and error report function\n @@ fsck.c: int fsck_objects_error_function(struct fsck_options *o,\n +{\n +\tstatic struct strbuf sb = STRBUF_INIT;\n +\n ++\tstrbuf_reset(&sb);\n +\tstrbuf_addstr(&sb, checked_ref_name);\n +\tif (oid)\n +\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n @@ fsck.h: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *\n \tunsigned strict:1;\n +\tunsigned verbose_refs:1;\n \tenum fsck_msg_type *msg_type;\n - \tstruct oidset oid_skiplist;\n + \tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n @@ fsck.h: struct fsck_options {\n \t.gitattributes_done = OIDSET_INIT, \\\n 4: e826dc17ec = 4: a38ea1b117 refs: set up ref consistency check infrastructure\n 5: 33cac4882b ! 5: 8320f56e0b builtin/refs: add verify subcommand\n @@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c\n +\t * Explicitly free the allocated array and \"oid_skiplist\"\n +\t */\n +\tfree(fsck_refs_options.msg_type);\n -+\toidset_clear(&fsck_refs_options.oid_skiplist);\n ++\toidset_clear(&fsck_refs_options.skip_oids);\n +\treturn ret;\n +}\n +\n 6: 32668e3543 = 6: 6614a06ef5 builtin/fsck: add `git-refs verify` child process\n 7: df83b2a990 = 7: 928cc96396 files-backend: add unified interface for refs scanning\n 8: c696c15651 = 8: 4d50d4932f fsck: add ref name check for files backend\n 9: 8b0f3aeb9c = 9: 7edb810819 fsck: add ref content check for files backend\n-- \n2.45.2\n\n"},{"id":"498356","messageId":"Zo0ubAe__yLH4O3r@ArchLinux","threadId":"61565","inReplyTo":"Zo0sQCBqyxX8dJ-f@ArchLinux","subject":"[GSoC][PATCH v9 1/9] fsck: rename \"skiplist\" to \"skip_oids\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-09T12:34:52Z","receivedAt":"2024-07-09T12:34:54Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The \"skiplist\" field in \"fsck_options\" is related to objects. Because we\nare going to introduce ref consistency check, the \"skiplist\" name is too\ngeneral which will make the caller think \"skiplist\" is related to both\nthe refs and objects.\n\nIt may seem that for both refs and objects, we should provide a general\n\"skiplist\" here. However, the type for \"skiplist\" is `struct oidset`\nwhich is totally unsuitable for refs.\n\nTo avoid above ambiguity, rename \"skiplist\" to \"skip_oids\".\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 4 ++--\n fsck.h | 4 ++--\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex eea7145470..3f32441492 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n \t\tif (!strcmp(buf, \"skiplist\")) {\n \t\t\tif (equal == len)\n \t\t\t\tdie(\"skiplist requires a path\");\n-\t\t\toidset_parse_file(&options->skiplist, buf + equal + 1,\n+\t\t\toidset_parse_file(&options->skip_oids, buf + equal + 1,\n \t\t\t\t\t the_repository->hash_algo);\n \t\t\tbuf += len + 1;\n \t\t\tcontinue;\n@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n static int object_on_skiplist(struct fsck_options *opts,\n \t\t\t const struct object_id *oid)\n {\n-\treturn opts && oid && oidset_contains(&opts->skiplist, oid);\n+\treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n __attribute__((format (printf, 5, 6)))\ndiff --git a/fsck.h b/fsck.h\nindex 6085a384f6..bcfb2e34cd 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -136,7 +136,7 @@ struct fsck_options {\n \tfsck_error error_func;\n \tunsigned strict:1;\n \tenum fsck_msg_type *msg_type;\n-\tstruct oidset skiplist;\n+\tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n \tstruct oidset gitattributes_found;\n@@ -145,7 +145,7 @@ struct fsck_options {\n };\n \n #define FSCK_OPTIONS_DEFAULT { \\\n-\t.skiplist = OIDSET_INIT, \\\n+\t.skip_oids = OIDSET_INIT, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n-- \n2.45.2\n\n"},{"id":"498357","messageId":"Zo0ufgFhREKlmBFG@ArchLinux","threadId":"61565","inReplyTo":"Zo0sQCBqyxX8dJ-f@ArchLinux","subject":"[GSoC][PATCH v9 2/9] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-09T12:35:10Z","receivedAt":"2024-07-09T12:35:12Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The static function \"report\" provided by \"fsck.c\" aims at checking fsck\nerror type and calling the callback \"error_func\" to report the message.\nHowever, \"report\" function is only related to object database which\ncannot be reused for refs. In order to provide a unified interface which\ncan report either objects or refs, create a new function \"vfsck_report\"\nby adding \"checked_ref_name\" parameter following the \"report\" prototype.\nInstead of using \"...\", provide \"va_list\" to allow more flexibility.\n\nLike \"report\", the \"vfsck_report\" function will use \"error_func\"\nregistered in \"fsck_options\" to report customized messages. Change\n\"error_func\" prototype to align with the new \"vfsck_report\".\n\nThen, change \"report\" function to use \"vfsck_report\" to report objects\nrelated messages. Add a new function called \"fsck_refs_report\" to use\n\"vfsck_report\" to report refs related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 15 ++++-----\n builtin/mktag.c | 1 +\n fsck.c | 81 ++++++++++++++++++++++++++++++++++++-------------\n fsck.h | 40 +++++++++++++++---------\n object-file.c | 11 ++++---\n 5 files changed, 101 insertions(+), 47 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..de34538c4f 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -89,12 +89,13 @@ static int objerror(struct object *obj, const char *err)\n \treturn -1;\n }\n \n-static int fsck_error_func(struct fsck_options *o UNUSED,\n-\t\t\t const struct object_id *oid,\n-\t\t\t enum object_type object_type,\n-\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t const char *message)\n+static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t enum object_type object_type,\n+\t\t\t\t const char *checked_ref_name UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \tswitch (msg_type) {\n \tcase FSCK_WARN:\n@@ -938,7 +939,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tfsck_walk_options.walk = mark_object;\n \tfsck_obj_options.walk = mark_used;\n-\tfsck_obj_options.error_func = fsck_error_func;\n+\tfsck_obj_options.error_func = fsck_objects_error_func;\n \tif (check_strict)\n \t\tfsck_obj_options.strict = 1;\n \ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex 4767f1a97e..42f945c584 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -20,6 +20,7 @@ static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n \t\t\t\t const struct object_id *oid UNUSED,\n \t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const char *checked_ref_name UNUSED,\n \t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t\t const char *message)\ndiff --git a/fsck.c b/fsck.c\nindex 3f32441492..e1819964e3 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -226,12 +226,18 @@ static int object_on_skiplist(struct fsck_options *opts,\n \treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n-__attribute__((format (printf, 5, 6)))\n-static int report(struct fsck_options *options,\n-\t\t const struct object_id *oid, enum object_type object_type,\n-\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+/*\n+ * Provide a unified interface for either fscking refs or objects.\n+ * It will get the current msg error type and call the error_func callback\n+ * which is registered in the \"fsck_options\" struct.\n+ */\n+static int vfsck_report(struct fsck_options *options,\n+\t\t\tconst struct object_id *oid,\n+\t\t\tenum object_type object_type,\n+\t\t\tconst char *checked_ref_name,\n+\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n {\n-\tva_list ap;\n+\tva_list ap_copy;\n \tstruct strbuf sb = STRBUF_INIT;\n \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n \tint result;\n@@ -250,9 +256,9 @@ static int report(struct fsck_options *options,\n \tprepare_msg_ids();\n \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n \n-\tva_start(ap, fmt);\n-\tstrbuf_vaddf(&sb, fmt, ap);\n-\tresult = options->error_func(options, oid, object_type,\n+\tva_copy(ap_copy, ap);\n+\tstrbuf_vaddf(&sb, fmt, ap_copy);\n+\tresult = options->error_func(options, oid, object_type, checked_ref_name,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n \tva_end(ap);\n@@ -260,6 +266,36 @@ static int report(struct fsck_options *options,\n \treturn result;\n }\n \n+__attribute__((format (printf, 5, 6)))\n+static int report(struct fsck_options *options,\n+\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = vfsck_report(options, oid, object_type, NULL,\n+\t\t\t msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n+\n+\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t const char *checked_ref_name,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = vfsck_report(options, oid, OBJ_NONE,\n+\t\t\t checked_ref_name, msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n void fsck_enable_object_names(struct fsck_options *options)\n {\n \tif (!options->object_names)\n@@ -1200,12 +1236,13 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid,\n-\t\t\tenum object_type object_type UNUSED,\n-\t\t\tenum fsck_msg_type msg_type,\n-\t\t\tenum fsck_msg_id msg_id UNUSED,\n-\t\t\tconst char *message)\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tenum object_type object_type UNUSED,\n+\t\t\t\tconst char *checked_ref_name UNUSED,\n+\t\t\t\tenum fsck_msg_type msg_type,\n+\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n+\t\t\t\tconst char *message)\n {\n \tif (msg_type == FSCK_WARN) {\n \t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n@@ -1303,16 +1340,18 @@ int git_fsck_config(const char *var, const char *value,\n * Custom error callbacks that are used in more than one place.\n */\n \n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message)\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t const char *checked_ref_name,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message)\n {\n \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n+\treturn fsck_objects_error_function(o, oid, object_type, checked_ref_name,\n+\t\t\t\t\t msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex bcfb2e34cd..8ce48395f6 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -114,22 +114,27 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n \t\t\t void *data, struct fsck_options *options);\n \n-/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n+/*\n+ * callback function for reporting errors when checking either objects or refs\n+ */\n typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t\t const char *checked_ref_name,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid, enum object_type object_type,\n-\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n-\t\t\tconst char *message);\n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message);\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\t\tconst char *checked_ref_name,\n+\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n+\t\t\t\tconst char *message);\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t const char *checked_ref_name,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message);\n \n struct fsck_options {\n \tfsck_walk_func walk;\n@@ -150,7 +155,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function \\\n+\t.error_func = fsck_objects_error_function \\\n }\n #define FSCK_OPTIONS_STRICT { \\\n \t.strict = 1, \\\n@@ -158,7 +163,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function, \\\n+\t.error_func = fsck_objects_error_function, \\\n }\n #define FSCK_OPTIONS_MISSING_GITMODULES { \\\n \t.strict = 1, \\\n@@ -166,7 +171,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n+\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n \n /* descend in all linked child objects\n@@ -209,6 +214,13 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+__attribute__((format (printf, 5, 6)))\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t const char *checked_ref_name,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\ndiff --git a/object-file.c b/object-file.c\nindex 065103be3e..d2c6427935 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2470,11 +2470,12 @@ int repo_has_object_file(struct repository *r,\n * give more context.\n */\n static int hash_format_check_report(struct fsck_options *opts UNUSED,\n-\t\t\t\t const struct object_id *oid UNUSED,\n-\t\t\t\t enum object_type object_type UNUSED,\n-\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n-\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t\t const char *message)\n+\t\t\t\t const struct object_id *oid UNUSED,\n+\t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const char *ref_checked_name UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \terror(_(\"object fails fsck: %s\"), message);\n \treturn 1;\n-- \n2.45.2\n\n"},{"id":"498358","messageId":"Zo0uiz1y6hJld2Rv@ArchLinux","threadId":"61565","inReplyTo":"Zo0sQCBqyxX8dJ-f@ArchLinux","subject":"[GSoC][PATCH v9 3/9] fsck: add refs-related options and error report function","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-09T12:35:23Z","receivedAt":"2024-07-09T12:35:25Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Add refs-related options to the \"fsck_options\", create refs-specific\n\"error_func\" callback \"fsck_refs_error_function\".\n\n\"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\npasses the oid, it will use \"oid_to_hex\" to get the corresponding hex\nvalue to report to the caller.\n\nLast, add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\"\nmacros to create refs options easily.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 23 +++++++++++++++++++++++\n fsck.h | 15 +++++++++++++++\n 2 files changed, 38 insertions(+)\n\ndiff --git a/fsck.c b/fsck.c\nindex e1819964e3..c5c7e8454f 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1252,6 +1252,29 @@ int fsck_objects_error_function(struct fsck_options *o,\n \treturn 1;\n }\n \n+int fsck_refs_error_function(struct fsck_options *options UNUSED,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type UNUSED,\n+\t\t\t const char *checked_ref_name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tstatic struct strbuf sb = STRBUF_INIT;\n+\n+\tstrbuf_reset(&sb);\n+\tstrbuf_addstr(&sb, checked_ref_name);\n+\tif (oid)\n+\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n+\n+\tif (msg_type == FSCK_WARN) {\n+\t\twarning(\"%s: %s\", sb.buf, message);\n+\t\treturn 0;\n+\t}\n+\terror(\"%s: %s\", sb.buf, message);\n+\treturn 1;\n+}\n+\n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n \t\t struct fsck_options *options, const char *blob_type)\ndiff --git a/fsck.h b/fsck.h\nindex 8ce48395f6..ff52913494 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -135,11 +135,19 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message);\n+int fsck_refs_error_function(struct fsck_options *options,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type,\n+\t\t\t const char *checked_ref_name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n \n struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n \tunsigned strict:1;\n+\tunsigned verbose_refs:1;\n \tenum fsck_msg_type *msg_type;\n \tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n@@ -173,6 +181,13 @@ struct fsck_options {\n \t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n+#define FSCK_REFS_OPTIONS_STRICT { \\\n+\t.strict = 1, \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n \n /* descend in all linked child objects\n * the return value is:\n-- \n2.45.2\n\n"},{"id":"498359","messageId":"Zo0umVPABD1wkxQG@ArchLinux","threadId":"61565","inReplyTo":"Zo0sQCBqyxX8dJ-f@ArchLinux","subject":"[GSoC][PATCH v9 4/9] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-09T12:35:37Z","receivedAt":"2024-07-09T12:35:41Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It's organized as the five parts:\n\n1. The name and the initialization interfaces.\n2. The ref transaction interfaces.\n3. The ref internal interfaces (pack, rename and copy).\n4. The ref filesystem interfaces.\n5. The reflog related interfaces.\n\nTo keep consistent with the git-fsck(1), add a new interface named\n\"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\ngrouped into any above five categories. Explicitly add blank line to\nmake it different from others.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs.c | 5 +++++\n refs.h | 8 ++++++++\n refs/debug.c | 11 +++++++++++\n refs/files-backend.c | 15 ++++++++++++++-\n refs/packed-backend.c | 8 ++++++++\n refs/refs-internal.h | 6 ++++++\n refs/reftable-backend.c | 8 ++++++++\n 7 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex bb90a18875..410919246b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs, struct fsck_options *o)\n+{\n+\treturn refs->be->fsck(refs, o);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex 0ecba21b4a..804d6a7fce 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -4,6 +4,7 @@\n #include \"commit.h\"\n #include \"repository.h\"\n \n+struct fsck_options;\n struct object_id;\n struct ref_store;\n struct strbuf;\n@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_options *o);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..45e2e784a0 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tint res = drefs->refs->be->fsck(drefs->refs, o);\n+\ttrace_printf_key(&trace_refs, \"fsck: %d\\n\", res);\n+\treturn res;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex aa52d9be7c..d89eeda8ef 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3408,6 +3408,17 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\treturn ret;\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_init,\n@@ -3434,5 +3445,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a0666407cd..5209b0b212 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_init,\n@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex fa975d69aa..a905e187cd 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_options;\n struct ref_transaction;\n \n /*\n@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_options *o);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -677,6 +681,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex fbe74c239d..b5a1a526df 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.45.2\n\n"},{"id":"498360","messageId":"Zo0up3jKBHy-6GRh@ArchLinux","threadId":"61565","inReplyTo":"Zo0sQCBqyxX8dJ-f@ArchLinux","subject":"[GSoC][PATCH v9 5/9] builtin/refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-09T12:35:51Z","receivedAt":"2024-07-09T12:35:55Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\ncheck the reference database consistency.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/git-refs.txt | 13 +++++++++++\n builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 57 insertions(+)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..1244a85b64 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+The following options are specific to 'git refs verify':\n+\n+--strict::\n+\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n+\tbe seen as ERROR. See linkgit:git-fsck[1].\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..baa96f5b3f 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,54 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tunsigned int verbose = 0, strict = 0;\n+\tstruct option options[] = {\n+\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\tint ret = 0;\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"too many arguments\"));\n+\n+\tif (verbose)\n+\t\tfsck_refs_options.verbose_refs = 1;\n+\tif (strict)\n+\t\tfsck_refs_options.strict = 1;\n+\n+\tgit_config(git_fsck_config, &fsck_refs_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n+\n+\t/*\n+\t * Explicitly free the allocated array and \"oid_skiplist\"\n+\t */\n+\tfree(fsck_refs_options.msg_type);\n+\toidset_clear(&fsck_refs_options.skip_oids);\n+\treturn ret;\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \n-- \n2.45.2\n\n"},{"id":"498361","messageId":"Zo0us3MQE0VYNrCu@ArchLinux","threadId":"61565","inReplyTo":"Zo0sQCBqyxX8dJ-f@ArchLinux","subject":"[GSoC][PATCH v9 6/9] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-09T12:36:03Z","receivedAt":"2024-07-09T12:36:07Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new function \"fsck_refs\" that initializes and runs a child\nprocess to execute the \"git-refs verify\" command.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex de34538c4f..ec3357722c 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -897,6 +897,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \treturn res;\n }\n \n+static void fsck_refs(void)\n+{\n+\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n+\tchild_process_init(&refs_verify);\n+\trefs_verify.git_cmd = 1;\n+\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n+\tif (verbose)\n+\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n+\tif (check_strict)\n+\t\tstrvec_push(&refs_verify.args, \"--strict\");\n+\n+\tif (run_command(&refs_verify))\n+\t\terrors_found |= ERROR_REFS;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n@@ -1066,6 +1081,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n+\tfsck_refs();\n+\n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \n-- \n2.45.2\n\n"},{"id":"498362","messageId":"Zo0uv4MKz2T6sOt7@ArchLinux","threadId":"61565","inReplyTo":"Zo0sQCBqyxX8dJ-f@ArchLinux","subject":"[GSoC][PATCH v9 7/9] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-09T12:36:15Z","receivedAt":"2024-07-09T12:36:18Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"For refs and reflogs, we need to scan its corresponding directories to\ncheck every regular file or symbolic link which shares the same pattern.\nIntroduce a unified interface for scanning directories for\nfiles-backend.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs/files-backend.c | 77 +++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 76 insertions(+), 1 deletion(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex d89eeda8ef..84acb58782 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -6,6 +6,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -3408,6 +3409,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n+{\n+\tconst char *gitdir = ref_store->gitdir;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\n+\tif (!iter) {\n+\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n+\t\tgoto out;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->verbose_refs)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n+\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n+\t\t\t\t iter->basename);\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n+\n+out:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tNULL\n+\t};\n+\n+\tif (o->verbose_refs)\n+\t\tfprintf_ln(stderr, \"Checking references consistency\");\n+\n+\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n+\n+\treturn ret;\n+}\n+\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_options *o)\n {\n@@ -3415,7 +3488,9 @@ static int files_fsck(struct ref_store *ref_store,\n \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n-\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o)\n+\t | files_fsck_refs(ref_store, o);\n+\n \treturn ret;\n }\n \n-- \n2.45.2\n\n"},{"id":"498363","messageId":"Zo0uzGvLSiU74Ode@ArchLinux","threadId":"61565","inReplyTo":"Zo0sQCBqyxX8dJ-f@ArchLinux","subject":"[GSoC][PATCH v9 8/9] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-09T12:36:28Z","receivedAt":"2024-07-09T12:36:31Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) only implicitly checks the reference, it does not fully\ncheck refs with bad format name such as standalone \"@\" and name ending\nwith \".lock\".\n\nIn order to provide such checks, add a new fsck message id \"badRefName\"\nwith default ERROR type. Use existing \"check_refname_format\" to explicit\ncheck the ref name. And add a new unit test to verify the functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 +\n fsck.h | 1 +\n refs/files-backend.c | 20 +++++++\n t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++\n 4 files changed, 125 insertions(+)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex f643585a34..dab4012246 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,9 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefName`::\n+\t(ERROR) A ref has a bad name.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex ff52913494..03825e86b1 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 84acb58782..69a76048d3 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3419,6 +3419,25 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n+static int files_fsck_refs_name(struct fsck_options *o,\n+\t\t\t\tconst char *gitdir UNUSED,\n+\t\t\t\tconst char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\tret = fsck_refs_report(o, NULL, sb.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3470,6 +3489,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n {\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tfiles_fsck_refs_name,\n \t\tNULL\n \t};\n \ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..b2db58d2c6\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit tag multi_hierarchy/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1.lock: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/@ &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\tgit -c fsck.badRefName=warn fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\tgit -c fsck.badRefName=ignore fsck 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+test_done\n-- \n2.45.2\n\n"},{"id":"498364","messageId":"Zo0u2egtB0UU22v8@ArchLinux","threadId":"61565","inReplyTo":"Zo0sQCBqyxX8dJ-f@ArchLinux","subject":"[GSoC][PATCH v9 9/9] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-09T12:36:41Z","receivedAt":"2024-07-09T12:36:43Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Enhance the git-fsck(1) command by adding a check for reference content\nin the files backend. The new functionality ensures that symrefs, real\nsymbolic link and regular refs are validated correctly.\n\nIn order to check the trailing content of the regular refs, add a new\nparameter `trailing` to `parse_loose_ref_contents`.\n\nFor symrefs, `parse_loose_ref_contents` will set the \"referent\".\nHowever, symbolic link could be either absolute or relative. Use\n\"strbuf_add_real_path\" to read the symbolic link and convert the\nrelative path to absolute path. Then use \"skip_prefix\" to make it align\nwith symref \"referent\".\n\nThus, the symrefs and symbolic links could share the same interface. Add\na new function \"files_fsck_symref_target\" which aims at checking the\nfollowing things:\n\n1. whether the pointee is under the `refs/` directory.\n2. whether the pointee name is correct.\n3. whether the pointee path is a wrong type in filesystem.\n\nLast, add the following FSCK MESSAGEs:\n\n1. \"badRefContent(ERROR)\": A ref has a bad content\n2. \"badSymrefPointee(ERROR)\": The pointee of a symref is bad.\n3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 9 +++\n fsck.h | 3 +\n refs.c | 2 +-\n refs/files-backend.c | 145 +++++++++++++++++++++++++++++++++-\n refs/refs-internal.h | 5 +-\n t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++\n 6 files changed, 269 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex dab4012246..b1630a478b 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,9 +19,15 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefContent`::\n+\t(ERROR) A ref has a bad content.\n+\n `badRefName`::\n \t(ERROR) A ref has a bad name.\n \n+`badSymrefPointee`::\n+\t(ERROR) The pointee of a symref is bad.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \n@@ -167,6 +173,9 @@\n `nullSha1`::\n \t(WARN) Tree contains entries pointing to a null sha1.\n \n+`trailingRefContent`::\n+\t(WARN) A ref content has trailing contents.\n+\n `treeNotSorted`::\n \t(ERROR) A tree is not properly sorted.\n \ndiff --git a/fsck.h b/fsck.h\nindex 03825e86b1..73592c9d2a 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -32,6 +32,8 @@ enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n+\tFUNC(BAD_REF_CONTENT, ERROR) \\\n+\tFUNC(BAD_SYMREF_POINTEE, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n@@ -72,6 +74,7 @@ enum fsck_msg_type {\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n+\tFUNC(TRAILING_REF_CONTENT, WARN) \\\n \tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n \tFUNC(LARGE_PATHNAME, WARN) \\\ndiff --git a/refs.c b/refs.c\nindex 410919246b..eb82fb7d4e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1760,7 +1760,7 @@ static int refs_read_special_head(struct ref_store *ref_store,\n \t}\n \n \tresult = parse_loose_ref_contents(content.buf, oid, referent, type,\n-\t\t\t\t\t failure_errno);\n+\t\t\t\t\t failure_errno, NULL);\n \n done:\n \tstrbuf_release(&full_path);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 69a76048d3..d98ef45403 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1,6 +1,7 @@\n #define USE_THE_REPOSITORY_VARIABLE\n \n #include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n #include \"../copy.h\"\n #include \"../environment.h\"\n #include \"../gettext.h\"\n@@ -553,7 +554,7 @@ static int read_ref_internal(struct ref_store *ref_store, const char *refname,\n \tstrbuf_rtrim(&sb_contents);\n \tbuf = sb_contents.buf;\n \n-\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr);\n+\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr, NULL);\n \n out:\n \tif (ret && !myerr)\n@@ -589,7 +590,7 @@ static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refn\n \n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno)\n+\t\t\t int *failure_errno, const char **trailing)\n {\n \tconst char *p;\n \tif (skip_prefix(buf, \"ref:\", &buf)) {\n@@ -611,6 +612,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t*failure_errno = EINVAL;\n \t\treturn -1;\n \t}\n+\n+\tif (trailing)\n+\t\t*trailing = p;\n+\n \treturn 0;\n }\n \n@@ -3438,6 +3443,141 @@ static int files_fsck_refs_name(struct fsck_options *o,\n \treturn ret;\n }\n \n+/*\n+ * Check the symref \"pointee_name\" and \"pointee_path\". The caller should\n+ * make sure that \"pointee_path\" is absolute. For symbolic ref, \"pointee_name\"\n+ * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n+ * be the relative path agaignst \"gitdir\".\n+ */\n+static int files_fsck_symref_target(struct fsck_options *o,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const char *pointee_name,\n+\t\t\t\t const char *pointee_path)\n+{\n+\tconst char *p = NULL;\n+\tstruct stat st;\n+\tint ret = 0;\n+\n+\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n+\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to target out of refs hierarchy\");\n+\t\tgoto out;\n+\t}\n+\n+\tif (check_refname_format(pointee_name, 0)) {\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid refname\");\n+\t}\n+\n+\tif (lstat(pointee_path, &st) < 0)\n+\t\tgoto out;\n+\n+\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid target\");\n+\t\tgoto out;\n+\t}\n+out:\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs_content(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter)\n+{\n+\tstruct strbuf pointee_path = STRBUF_INIT,\n+\t\t ref_content = STRBUF_INIT,\n+\t\t abs_gitdir = STRBUF_INIT,\n+\t\t referent = STRBUF_INIT,\n+\t\t refname = STRBUF_INIT;\n+\tconst char *trailing = NULL;\n+\tint failure_errno = 0;\n+\tunsigned int type = 0;\n+\tstruct object_id oid;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\n+\t/*\n+\t * If the file is a symlink, we need to only check the connectivity\n+\t * of the destination object.\n+\t */\n+\tif (S_ISLNK(iter->st.st_mode)) {\n+\t\tconst char *pointee_name = NULL;\n+\n+\t\tstrbuf_add_real_path(&pointee_path, iter->path.buf);\n+\n+\t\tstrbuf_add_absolute_path(&abs_gitdir, gitdir);\n+\t\tstrbuf_normalize_path(&abs_gitdir);\n+\t\tif (!is_dir_sep(abs_gitdir.buf[abs_gitdir.len - 1]))\n+\t\t\tstrbuf_addch(&abs_gitdir, '/');\n+\n+\t\tif (!skip_prefix(pointee_path.buf,\n+\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n+\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t\t \"point to target outside gitdir\");\n+\t\t\tgoto clean;\n+\t\t}\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, pointee_name,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n+\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n+\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n+\t\t\t\t &referent, &type,\n+\t\t\t\t &failure_errno, &trailing)) {\n+\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n+\t\t\t\t \"invalid ref content\");\n+\t\tgoto clean;\n+\t}\n+\n+\t/*\n+\t * If the ref is a symref, we need to check the destination name and\n+\t * connectivity.\n+\t */\n+\tif (referent.len && (type & REF_ISSYMREF)) {\n+\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n+\t\tstrbuf_rtrim(&referent);\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, referent.buf,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t} else {\n+\t\t/*\n+\t\t * Only regular refs could have a trailing garbage. Should\n+\t\t * be reported as a warning.\n+\t\t */\n+\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n+\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n+\t\t\t\t\t \"trailing garbage in ref\");\n+\t\t\tgoto clean;\n+\t\t}\n+\t}\n+\n+clean:\n+\tstrbuf_release(&abs_gitdir);\n+\tstrbuf_release(&pointee_path);\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&ref_content);\n+\tstrbuf_release(&referent);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3490,6 +3630,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n \t\tfiles_fsck_refs_name,\n+\t\tfiles_fsck_refs_content,\n \t\tNULL\n \t};\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex a905e187cd..2fabf41d14 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -709,11 +709,12 @@ struct ref_store {\n \n /*\n * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for\n- * invalid contents.\n+ * invalid contents. Also *trailing is set to the first character after the\n+ * refname or NULL if the referent is not empty.\n */\n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno);\n+\t\t\t int *failure_errno, const char **trailing);\n \n /*\n * Fill in the generic part of refs and add it to our collection of\ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nindex b2db58d2c6..35bf40ee64 100755\n--- a/t/t0602-reffiles-fsck.sh\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -98,4 +98,114 @@ test_expect_success 'ref name check should be adapted into fsck messages' '\n \t)\n '\n \n+test_expect_success 'regular ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit checkout -b a/b/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse branch-1)\" > $branch_dir_prefix/branch-1-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse tag-1)\" > $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_must_fail git -c fsck.trailingRefContent=error fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s \" \"$(git rev-parse tag-2)\" > $tag_dir_prefix/tag-2-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/tags/tag-2-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/a/b/branch-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'symbolic ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid refname\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid target\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: logs/maint-v2.45\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to target out of refs hierarchy\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.2\n\n"},{"id":"498376","messageId":"cu3ypjpzpdkazg7w7ho5njcrzvqklzycba3oipwhztlh6lvu6t@wtjew4mj6ku4","threadId":"61565","inReplyTo":"Zo0ufgFhREKlmBFG@ArchLinux","subject":"Re: [GSoC][PATCH v9 2/9] fsck: add a unified interface for reporting fsck messages","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-07-09T20:24:50Z","receivedAt":"2024-07-09T20:25:21Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/07/09 08:35PM, shejialuo wrote:\n> The static function \"report\" provided by \"fsck.c\" aims at checking fsck\n> error type and calling the callback \"error_func\" to report the message.\n> However, \"report\" function is only related to object database which\n> cannot be reused for refs. In order to provide a unified interface which\n> can report either objects or refs, create a new function \"vfsck_report\"\n> by adding \"checked_ref_name\" parameter following the \"report\" prototype.\n> Instead of using \"...\", provide \"va_list\" to allow more flexibility.\n> \n> Like \"report\", the \"vfsck_report\" function will use \"error_func\"\n> registered in \"fsck_options\" to report customized messages. Change\n> \"error_func\" prototype to align with the new \"vfsck_report\".\n> \n> Then, change \"report\" function to use \"vfsck_report\" to report objects\n> related messages. Add a new function called \"fsck_refs_report\" to use\n> \"vfsck_report\" to report refs related messages.\n> \n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n> builtin/fsck.c | 15 ++++-----\n> builtin/mktag.c | 1 +\n> fsck.c | 81 ++++++++++++++++++++++++++++++++++++-------------\n> fsck.h | 40 +++++++++++++++---------\n> object-file.c | 11 ++++---\n> 5 files changed, 101 insertions(+), 47 deletions(-)\n> \n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index d13a226c2e..de34538c4f 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -89,12 +89,13 @@ static int objerror(struct object *obj, const char *err)\n> \treturn -1;\n> }\n> \n> -static int fsck_error_func(struct fsck_options *o UNUSED,\n> -\t\t\t const struct object_id *oid,\n> -\t\t\t enum object_type object_type,\n> -\t\t\t enum fsck_msg_type msg_type,\n> -\t\t\t enum fsck_msg_id msg_id UNUSED,\n> -\t\t\t const char *message)\n> +static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n> +\t\t\t\t const struct object_id *oid,\n> +\t\t\t\t enum object_type object_type,\n> +\t\t\t\t const char *checked_ref_name UNUSED,\n> +\t\t\t\t enum fsck_msg_type msg_type,\n> +\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> +\t\t\t\t const char *message)\n\nThis is just a suggestion, but I think it would be slightly easier to\nreview if the `*_error_func()` renames were done in a separate preceding\npatch. That way the purpose of the renames can also be clearly\nexplained.\n\n> {\n> \tswitch (msg_type) {\n> \tcase FSCK_WARN:\n> @@ -938,7 +939,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n> \n> \tfsck_walk_options.walk = mark_object;\n> \tfsck_obj_options.walk = mark_used;\n> -\tfsck_obj_options.error_func = fsck_error_func;\n> +\tfsck_obj_options.error_func = fsck_objects_error_func;\n> \tif (check_strict)\n> \t\tfsck_obj_options.strict = 1;\n> \n[snip]\n> @@ -166,7 +171,7 @@ struct fsck_options {\n> \t.gitmodules_done = OIDSET_INIT, \\\n> \t.gitattributes_found = OIDSET_INIT, \\\n> \t.gitattributes_done = OIDSET_INIT, \\\n> -\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n> +\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n> }\n> \n> /* descend in all linked child objects\n> @@ -209,6 +214,13 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n> */\n> int fsck_finish(struct fsck_options *options);\n> \n> +__attribute__((format (printf, 5, 6)))\n> +int fsck_refs_report(struct fsck_options *options,\n> +\t\t const struct object_id *oid,\n> +\t\t const char *checked_ref_name,\n> +\t\t enum fsck_msg_id msg_id,\n> +\t\t const char *fmt, ...);\n> +\n\nI think I mentioned this in a previous reply, but it was missed. Not a\nbig deal, but it might be nice to document `int fsck_refs_report()`\nhere.\n\n-Justin\n"},{"id":"498378","messageId":"oan7pcamqpi2z4jzfshff2sfziqgisd3jcbpzfoiuyks5g57mk@7isgeztkcgmy","threadId":"61565","inReplyTo":"Zo0uiz1y6hJld2Rv@ArchLinux","subject":"Re: [GSoC][PATCH v9 3/9] fsck: add refs-related options and error report function","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-07-09T21:29:24Z","receivedAt":"2024-07-09T21:29:55Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/07/09 08:35PM, shejialuo wrote:\n> Add refs-related options to the \"fsck_options\", create refs-specific\n> \"error_func\" callback \"fsck_refs_error_function\".\n> \n> \"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\n> passes the oid, it will use \"oid_to_hex\" to get the corresponding hex\n> value to report to the caller.\n\nOut of curiousity, under what circumstances would the caller want to\nalso pass the oid? Would it simply be to optionally provide additional\ncontext?\n\n> \n> Last, add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\"\n> macros to create refs options easily.\n> \n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n> fsck.c | 23 +++++++++++++++++++++++\n> fsck.h | 15 +++++++++++++++\n> 2 files changed, 38 insertions(+)\n> \n> diff --git a/fsck.c b/fsck.c\n> index e1819964e3..c5c7e8454f 100644\n> --- a/fsck.c\n> +++ b/fsck.c\n> @@ -1252,6 +1252,29 @@ int fsck_objects_error_function(struct fsck_options *o,\n> \treturn 1;\n> }\n> \n> +int fsck_refs_error_function(struct fsck_options *options UNUSED,\n> +\t\t\t const struct object_id *oid,\n> +\t\t\t enum object_type object_type UNUSED,\n> +\t\t\t const char *checked_ref_name,\n> +\t\t\t enum fsck_msg_type msg_type,\n> +\t\t\t enum fsck_msg_id msg_id UNUSED,\n> +\t\t\t const char *message)\n> +{\n> +\tstatic struct strbuf sb = STRBUF_INIT;\n> +\n> +\tstrbuf_reset(&sb);\n\nNaive question, is there reason to reset `sb` immediately after\n`STRBUF_INIT`? My understanding is that because we initialize the\nbuffer, the other fields should also be zeroed. If so, resetting the\nbuffer here seems redundant.\n\n> +\tstrbuf_addstr(&sb, checked_ref_name);\n> +\tif (oid)\n> +\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n> +\n> +\tif (msg_type == FSCK_WARN) {\n> +\t\twarning(\"%s: %s\", sb.buf, message);\n> +\t\treturn 0;\n> +\t}\n> +\terror(\"%s: %s\", sb.buf, message);\n> +\treturn 1;\n> +}\n> +\n> static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n> \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n> \t\t struct fsck_options *options, const char *blob_type)\n> diff --git a/fsck.h b/fsck.h\n> index 8ce48395f6..ff52913494 100644\n> --- a/fsck.h\n> +++ b/fsck.h\n> @@ -135,11 +135,19 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n> \t\t\t\t\t\t enum fsck_msg_type msg_type,\n> \t\t\t\t\t\t enum fsck_msg_id msg_id,\n> \t\t\t\t\t\t const char *message);\n> +int fsck_refs_error_function(struct fsck_options *options,\n> +\t\t\t const struct object_id *oid,\n> +\t\t\t enum object_type object_type,\n> +\t\t\t const char *checked_ref_name,\n> +\t\t\t enum fsck_msg_type msg_type,\n> +\t\t\t enum fsck_msg_id msg_id,\n> +\t\t\t const char *message);\n> \n> struct fsck_options {\n> \tfsck_walk_func walk;\n> \tfsck_error error_func;\n> \tunsigned strict:1;\n> +\tunsigned verbose_refs:1;\n\nWhat is the purpose of adding `verbose_refs` in this patch? At this\npoint, I'm not seeing it used. If there is a reason to be included in\nthis patch, it might be nice to mention in the commit message.\n\n> \tenum fsck_msg_type *msg_type;\n> \tstruct oidset skip_oids;\n> \tstruct oidset gitmodules_found;\n> @@ -173,6 +181,13 @@ struct fsck_options {\n> \t.gitattributes_done = OIDSET_INIT, \\\n> \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n> }\n> +#define FSCK_REFS_OPTIONS_DEFAULT { \\\n> +\t.error_func = fsck_refs_error_function, \\\n> +}\n> +#define FSCK_REFS_OPTIONS_STRICT { \\\n> +\t.strict = 1, \\\n> +\t.error_func = fsck_refs_error_function, \\\n> +}\n> \n> /* descend in all linked child objects\n> * the return value is:\n> -- \n> 2.45.2\n> \n"},{"id":"498380","messageId":"CAPig+cR=RgMeaAy1PRGgHu6_Ak+7=_-5tGvBZRekKRxi7GtdHw@mail.gmail.com","threadId":"61565","inReplyTo":"oan7pcamqpi2z4jzfshff2sfziqgisd3jcbpzfoiuyks5g57mk@7isgeztkcgmy","subject":"Re: [GSoC][PATCH v9 3/9] fsck: add refs-related options and error report function","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-07-09T21:40:08Z","receivedAt":"2024-07-09T21:40:20Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Jul 9, 2024 at 5:30 PM Justin Tobler wrote:\n> On 24/07/09 08:35PM, shejialuo wrote:\n> > +int fsck_refs_error_function(struct fsck_options *options UNUSED,\n> > + const struct object_id *oid,\n> > + enum object_type object_type UNUSED,\n> > + const char *checked_ref_name,\n> > + enum fsck_msg_type msg_type,\n> > + enum fsck_msg_id msg_id UNUSED,\n> > + const char *message)\n> > +{\n> > + static struct strbuf sb = STRBUF_INIT;\n> > +\n> > + strbuf_reset(&sb);\n>\n> Naive question, is there reason to reset `sb` immediately after\n> `STRBUF_INIT`? My understanding is that because we initialize the\n> buffer, the other fields should also be zeroed. If so, resetting the\n> buffer here seems redundant.\n\nThis particular strbuf is static, so it needs to be cleared each time\nthe function is called.\n\nThe cover letter provides an argument for making it static: that this\nwill be called often, and we don't want to make a lot of repeated\nallocations. Personally, I find that argument rather weak. Why would\nan error function be called frequently? Is this really a hot path that\nneeds to worry about a few extra allocations? Also, importantly, every\nstatic added makes the code harder to \"libify\", so making it static\nrequires a very strong reason, but there doesn't seem to be such a\nreason in this case.\n"},{"id":"498382","messageId":"b7apetyejg5ycvvqkc5motpx45mbval5sixvlgwfrh4q5lzc5d@b32kkwo4qeer","threadId":"61565","inReplyTo":"Zo0umVPABD1wkxQG@ArchLinux","subject":"Re: [GSoC][PATCH v9 4/9] refs: set up ref consistency check infrastructure","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-07-09T22:11:36Z","receivedAt":"2024-07-09T22:12:09Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/07/09 08:35PM, shejialuo wrote:\n> The interfaces defined in the `ref_storage_be` are carefully structured\n> in semantic. It's organized as the five parts:\n> \n> 1. The name and the initialization interfaces.\n> 2. The ref transaction interfaces.\n> 3. The ref internal interfaces (pack, rename and copy).\n> 4. The ref filesystem interfaces.\n> 5. The reflog related interfaces.\n> \n> To keep consistent with the git-fsck(1), add a new interface named\n> \"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\n> grouped into any above five categories. Explicitly add blank line to\n> make it different from others.\n\nIn this patch we are not only adding the `fsck` infrastructure to\n`ref_storage_be`, but also wiring what are essentially no-op functions \nto the various implementations. It might be good to mention this in the\ncommit message so it is better understood that future patches will \nactually provide proper function implementations.\n\n-Justin\n"},{"id":"498384","messageId":"eekcci7aeucp2srdotwdghfitinp7s6quc545ktlaif7kpucah@3vpegal5lsdd","threadId":"61565","inReplyTo":"Zo0up3jKBHy-6GRh@ArchLinux","subject":"Re: [GSoC][PATCH v9 5/9] builtin/refs: add verify subcommand","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-07-09T22:30:42Z","receivedAt":"2024-07-09T22:31:13Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/07/09 08:35PM, shejialuo wrote:\n> Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\n> check the reference database consistency.\n\nThe next patch in the series uses `git-refs verify` to when running\ngit-fsck(1). It might be worth mentioning here that this is also\nintended as the entry point for fscking refs.\n\n> \n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n[snip]\n> @@ -58,15 +63,54 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n> \treturn err;\n> }\n> \n> +static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n> +{\n> +\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n> +\tconst char * const verify_usage[] = {\n> +\t\tREFS_VERIFY_USAGE,\n> +\t\tNULL,\n> +\t};\n> +\tunsigned int verbose = 0, strict = 0;\n> +\tstruct option options[] = {\n> +\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n> +\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n> +\t\tOPT_END(),\n> +\t};\n> +\tint ret = 0;\n\nnit: Being that we always assign a value to `ret`, defaulting here to \nzero is redundant.\n\n> +\n> +\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n> +\tif (argc)\n> +\t\tusage(_(\"too many arguments\"));\n> +\n> +\tif (verbose)\n> +\t\tfsck_refs_options.verbose_refs = 1;\n> +\tif (strict)\n> +\t\tfsck_refs_options.strict = 1;\n> +\n> +\tgit_config(git_fsck_config, &fsck_refs_options);\n> +\tprepare_repo_settings(the_repository);\n> +\n> +\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n> +\n> +\t/*\n> +\t * Explicitly free the allocated array and \"oid_skiplist\"\n\ns/oid_skiplist/skip_oids\n\nI think we forgot to update this comment after the variable was renamed\nin a previous patch version.\n\n> +\t */\n> +\tfree(fsck_refs_options.msg_type);\n> +\toidset_clear(&fsck_refs_options.skip_oids);\n> +\treturn ret;\n> +}\n> +\n> int cmd_refs(int argc, const char **argv, const char *prefix)\n> {\n> \tconst char * const refs_usage[] = {\n> \t\tREFS_MIGRATE_USAGE,\n> +\t\tREFS_VERIFY_USAGE,\n> \t\tNULL,\n> \t};\n> \tparse_opt_subcommand_fn *fn = NULL;\n> \tstruct option opts[] = {\n> \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n> +\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n> \t\tOPT_END(),\n> \t};\n> \n> -- \n> 2.45.2\n> \n"},{"id":"498439","messageId":"Zo56CEdLdQywXgBI@ArchLinux","threadId":"61565","inReplyTo":"cu3ypjpzpdkazg7w7ho5njcrzvqklzycba3oipwhztlh6lvu6t@wtjew4mj6ku4","subject":"Re: [GSoC][PATCH v9 2/9] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T12:09:44Z","receivedAt":"2024-07-10T12:09:45Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jul 09, 2024 at 03:24:50PM -0500, Justin Tobler wrote:\n> On 24/07/09 08:35PM, shejialuo wrote:\n> > The static function \"report\" provided by \"fsck.c\" aims at checking fsck\n> > error type and calling the callback \"error_func\" to report the message.\n> > However, \"report\" function is only related to object database which\n> > cannot be reused for refs. In order to provide a unified interface which\n> > can report either objects or refs, create a new function \"vfsck_report\"\n> > by adding \"checked_ref_name\" parameter following the \"report\" prototype.\n> > Instead of using \"...\", provide \"va_list\" to allow more flexibility.\n> > \n> > Like \"report\", the \"vfsck_report\" function will use \"error_func\"\n> > registered in \"fsck_options\" to report customized messages. Change\n> > \"error_func\" prototype to align with the new \"vfsck_report\".\n> > \n> > Then, change \"report\" function to use \"vfsck_report\" to report objects\n> > related messages. Add a new function called \"fsck_refs_report\" to use\n> > \"vfsck_report\" to report refs related messages.\n> > \n> > Mentored-by: Patrick Steinhardt \n> > Mentored-by: Karthik Nayak \n> > Signed-off-by: shejialuo \n> > ---\n> > builtin/fsck.c | 15 ++++-----\n> > builtin/mktag.c | 1 +\n> > fsck.c | 81 ++++++++++++++++++++++++++++++++++++-------------\n> > fsck.h | 40 +++++++++++++++---------\n> > object-file.c | 11 ++++---\n> > 5 files changed, 101 insertions(+), 47 deletions(-)\n> > \n> > diff --git a/builtin/fsck.c b/builtin/fsck.c\n> > index d13a226c2e..de34538c4f 100644\n> > --- a/builtin/fsck.c\n> > +++ b/builtin/fsck.c\n> > @@ -89,12 +89,13 @@ static int objerror(struct object *obj, const char *err)\n> > \treturn -1;\n> > }\n> > \n> > -static int fsck_error_func(struct fsck_options *o UNUSED,\n> > -\t\t\t const struct object_id *oid,\n> > -\t\t\t enum object_type object_type,\n> > -\t\t\t enum fsck_msg_type msg_type,\n> > -\t\t\t enum fsck_msg_id msg_id UNUSED,\n> > -\t\t\t const char *message)\n> > +static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n> > +\t\t\t\t const struct object_id *oid,\n> > +\t\t\t\t enum object_type object_type,\n> > +\t\t\t\t const char *checked_ref_name UNUSED,\n> > +\t\t\t\t enum fsck_msg_type msg_type,\n> > +\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> > +\t\t\t\t const char *message)\n> \n> This is just a suggestion, but I think it would be slightly easier to\n> review if the `*_error_func()` renames were done in a separate preceding\n> patch. That way the purpose of the renames can also be clearly\n> explained.\n> \n\nI agree with this, will change in the next version.\n\n> > {\n> > \tswitch (msg_type) {\n> > \tcase FSCK_WARN:\n> > @@ -938,7 +939,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n> > \n> > \tfsck_walk_options.walk = mark_object;\n> > \tfsck_obj_options.walk = mark_used;\n> > -\tfsck_obj_options.error_func = fsck_error_func;\n> > +\tfsck_obj_options.error_func = fsck_objects_error_func;\n> > \tif (check_strict)\n> > \t\tfsck_obj_options.strict = 1;\n> > \n> [snip]\n> > @@ -166,7 +171,7 @@ struct fsck_options {\n> > \t.gitmodules_done = OIDSET_INIT, \\\n> > \t.gitattributes_found = OIDSET_INIT, \\\n> > \t.gitattributes_done = OIDSET_INIT, \\\n> > -\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n> > +\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n> > }\n> > \n> > /* descend in all linked child objects\n> > @@ -209,6 +214,13 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n> > */\n> > int fsck_finish(struct fsck_options *options);\n> > \n> > +__attribute__((format (printf, 5, 6)))\n> > +int fsck_refs_report(struct fsck_options *options,\n> > +\t\t const struct object_id *oid,\n> > +\t\t const char *checked_ref_name,\n> > +\t\t enum fsck_msg_id msg_id,\n> > +\t\t const char *fmt, ...);\n> > +\n> \n> I think I mentioned this in a previous reply, but it was missed. Not a\n> big deal, but it might be nice to document `int fsck_refs_report()`\n> here.\n> \n\nI will improve this in the next version.\n\n> -Justin\n\nThanks,\nJialuo\n"},{"id":"498440","messageId":"Zo5695Ig1q6ronkg@ArchLinux","threadId":"61565","inReplyTo":"CAPig+cR=RgMeaAy1PRGgHu6_Ak+7=_-5tGvBZRekKRxi7GtdHw@mail.gmail.com","subject":"Re: [GSoC][PATCH v9 3/9] fsck: add refs-related options and error report function","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T12:13:43Z","receivedAt":"2024-07-10T12:13:42Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jul 09, 2024 at 05:40:08PM -0400, Eric Sunshine wrote:\n> On Tue, Jul 9, 2024 at 5:30 PM Justin Tobler wrote:\n> > On 24/07/09 08:35PM, shejialuo wrote:\n> > > +int fsck_refs_error_function(struct fsck_options *options UNUSED,\n> > > + const struct object_id *oid,\n> > > + enum object_type object_type UNUSED,\n> > > + const char *checked_ref_name,\n> > > + enum fsck_msg_type msg_type,\n> > > + enum fsck_msg_id msg_id UNUSED,\n> > > + const char *message)\n> > > +{\n> > > + static struct strbuf sb = STRBUF_INIT;\n> > > +\n> > > + strbuf_reset(&sb);\n> >\n> > Naive question, is there reason to reset `sb` immediately after\n> > `STRBUF_INIT`? My understanding is that because we initialize the\n> > buffer, the other fields should also be zeroed. If so, resetting the\n> > buffer here seems redundant.\n> \n> This particular strbuf is static, so it needs to be cleared each time\n> the function is called.\n> \n> The cover letter provides an argument for making it static: that this\n> will be called often, and we don't want to make a lot of repeated\n> allocations. Personally, I find that argument rather weak. Why would\n> an error function be called frequently? Is this really a hot path that\n> needs to worry about a few extra allocations? Also, importantly, every\n> static added makes the code harder to \"libify\", so making it static\n> requires a very strong reason, but there doesn't seem to be such a\n> reason in this case.\n\nI didn't consider the issue of libify. I just want to reduce some memory\nallocations. I will change this in the next version.\n\nThanks,\nJialuo\n"},{"id":"498441","messageId":"Zo5-XAzqt3YxWJ-v@ArchLinux","threadId":"61565","inReplyTo":"oan7pcamqpi2z4jzfshff2sfziqgisd3jcbpzfoiuyks5g57mk@7isgeztkcgmy","subject":"Re: [GSoC][PATCH v9 3/9] fsck: add refs-related options and error report function","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T12:28:12Z","receivedAt":"2024-07-10T12:28:12Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jul 09, 2024 at 04:29:24PM -0500, Justin Tobler wrote:\n> On 24/07/09 08:35PM, shejialuo wrote:\n> > Add refs-related options to the \"fsck_options\", create refs-specific\n> > \"error_func\" callback \"fsck_refs_error_function\".\n> > \n> > \"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\n> > passes the oid, it will use \"oid_to_hex\" to get the corresponding hex\n> > value to report to the caller.\n> \n> Out of curiousity, under what circumstances would the caller want to\n> also pass the oid? Would it simply be to optionally provide additional\n> context?\n> \n\nBecause when we check the refs, we will use \"parse_loose_ref_contents\"\nhere to check the ref contents. Below is the prototype:\n\n int parse_loose_ref_contents(const char *buf,\n struct object_id *oid,\n ...)\n\nSo we could get a oid here. However, we don't know the type of the oid.\nIt may not be commit object but rather a tag object. And I want to\nprovide more flexible operations for caller. When caller passes the oid.\nThe message could be the following:\n\n ref_name -> (oid) : fsck_error_type: user-passed message.\n\nSo, actually we have provided additional context for the caller. From\nanother perspective, the object check needs the \"oid\" parameter, we\ncannot remove it from the callback \"error_func\" prototype. So why not\njust reuse this parameter? It truly provides the caller more flexibility\nwithout big changes.\n\n> > struct fsck_options {\n> > \tfsck_walk_func walk;\n> > \tfsck_error error_func;\n> > \tunsigned strict:1;\n> > +\tunsigned verbose_refs:1;\n> \n> What is the purpose of adding `verbose_refs` in this patch? At this\n> point, I'm not seeing it used. If there is a reason to be included in\n> this patch, it might be nice to mention in the commit message.\n> \n\nThe reason is that fsck builtin handles the object check but we want to\nuse \"git refs verify\" command to handle ref check and we put all the\nreal functionality into each file backend. And there is only one entry\npoint in the \"git refs verify\" command. So we need to use \"fsck_options\"\nas the parameter to maintain this state across the ref checks.\n\nActually, \"git-fsck(1)\" maintains \"verbose\" global variable. I think I\nshould not add this option in this patch which may cause a lot of\nconfusion here. I will improve this in the next version.\n\n> > \tenum fsck_msg_type *msg_type;\n> > \tstruct oidset skip_oids;\n> > \tstruct oidset gitmodules_found;\n> > @@ -173,6 +181,13 @@ struct fsck_options {\n> > \t.gitattributes_done = OIDSET_INIT, \\\n> > \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n> > }\n> > +#define FSCK_REFS_OPTIONS_DEFAULT { \\\n> > +\t.error_func = fsck_refs_error_function, \\\n> > +}\n> > +#define FSCK_REFS_OPTIONS_STRICT { \\\n> > +\t.strict = 1, \\\n> > +\t.error_func = fsck_refs_error_function, \\\n> > +}\n> > \n> > /* descend in all linked child objects\n> > * the return value is:\n> > -- \n> > 2.45.2\n> > \n"},{"id":"498442","messageId":"Zo5-wOYnVMcOPQc6@ArchLinux","threadId":"61565","inReplyTo":"b7apetyejg5ycvvqkc5motpx45mbval5sixvlgwfrh4q5lzc5d@b32kkwo4qeer","subject":"Re: [GSoC][PATCH v9 4/9] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T12:29:52Z","receivedAt":"2024-07-10T12:29:52Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jul 09, 2024 at 05:11:36PM -0500, Justin Tobler wrote:\n> On 24/07/09 08:35PM, shejialuo wrote:\n> > The interfaces defined in the `ref_storage_be` are carefully structured\n> > in semantic. It's organized as the five parts:\n> > \n> > 1. The name and the initialization interfaces.\n> > 2. The ref transaction interfaces.\n> > 3. The ref internal interfaces (pack, rename and copy).\n> > 4. The ref filesystem interfaces.\n> > 5. The reflog related interfaces.\n> > \n> > To keep consistent with the git-fsck(1), add a new interface named\n> > \"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\n> > grouped into any above five categories. Explicitly add blank line to\n> > make it different from others.\n> \n> In this patch we are not only adding the `fsck` infrastructure to\n> `ref_storage_be`, but also wiring what are essentially no-op functions \n> to the various implementations. It might be good to mention this in the\n> commit message so it is better understood that future patches will \n> actually provide proper function implementations.\n> \n\nYes, I agree. I will improve this in the next version.\n\n> -Justin\n"},{"id":"498443","messageId":"Zo5_TzJuc6VBdZar@ArchLinux","threadId":"61565","inReplyTo":"eekcci7aeucp2srdotwdghfitinp7s6quc545ktlaif7kpucah@3vpegal5lsdd","subject":"Re: [GSoC][PATCH v9 5/9] builtin/refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T12:32:15Z","receivedAt":"2024-07-10T12:32:15Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jul 09, 2024 at 05:30:42PM -0500, Justin Tobler wrote:\n> On 24/07/09 08:35PM, shejialuo wrote:\n> > Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\n> > check the reference database consistency.\n> \n> The next patch in the series uses `git-refs verify` to when running\n> git-fsck(1). It might be worth mentioning here that this is also\n> intended as the entry point for fscking refs.\n> \n\nYes, I will improve the commit message, in the next patch.\n\n> > \n> > Mentored-by: Patrick Steinhardt \n> > Mentored-by: Karthik Nayak \n> > Signed-off-by: shejialuo \n> [snip]\n> > @@ -58,15 +63,54 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n> > \treturn err;\n> > }\n> > \n> > +static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n> > +{\n> > +\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n> > +\tconst char * const verify_usage[] = {\n> > +\t\tREFS_VERIFY_USAGE,\n> > +\t\tNULL,\n> > +\t};\n> > +\tunsigned int verbose = 0, strict = 0;\n> > +\tstruct option options[] = {\n> > +\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n> > +\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n> > +\t\tOPT_END(),\n> > +\t};\n> > +\tint ret = 0;\n> \n> nit: Being that we always assign a value to `ret`, defaulting here to \n> zero is redundant.\n> \n\nI agree.\n\n> > +\n> > +\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n> > +\tif (argc)\n> > +\t\tusage(_(\"too many arguments\"));\n> > +\n> > +\tif (verbose)\n> > +\t\tfsck_refs_options.verbose_refs = 1;\n> > +\tif (strict)\n> > +\t\tfsck_refs_options.strict = 1;\n> > +\n> > +\tgit_config(git_fsck_config, &fsck_refs_options);\n> > +\tprepare_repo_settings(the_repository);\n> > +\n> > +\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n> > +\n> > +\t/*\n> > +\t * Explicitly free the allocated array and \"oid_skiplist\"\n> \n> s/oid_skiplist/skip_oids\n> \n> I think we forgot to update this comment after the variable was renamed\n> in a previous patch version.\n> \n\nYes, thanks for you remind. I have just forgotten here.\n\n"},{"id":"498452","messageId":"Zo6eJi8BePrQxTQV@ArchLinux","threadId":"61565","inReplyTo":"Zo0sQCBqyxX8dJ-f@ArchLinux","subject":"[GSoC][PATCH v10 00/10] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T14:43:50Z","receivedAt":"2024-07-10T14:43:51Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Hi All:\n\nThis version handles the following problems:\n\n1. Following the Justin's advice. Add a new commit to rename the\nobjects-related fsck error functions to make the series more clean.\n2. Add a simple documentation for \"fsck_refs_report\".\n3. The previous implementation of \"fsck_refs_error_function\" used\n\"static struct strbuf\", as Eric said, it's a bad idea to use \"static\"\nbecause of infrequent usage of \"fsck_refs_error_function\". And it will\nmake the code arder to \"libify\". I didn't consider this. So In this\nversion, use \"struct strbuf\" instead and add the corresponding\nmemory-free operation.\n4. I should not add \"verbose_refs\" option in \"fsck: add refs-related\nerror report function\" commit. I add this option in \"builtin/refs\" to\nmake no confusion.\n5. Enchance the commit message for \"refs: set up ref consistency check\ninfrastructure\" to mention that we have added placeholder functions.\n6. Change the \"oid_skiplist\" from comments which are ignored by the\nprevious version.\n\nThanks for every reviwer.\n\nCI: https://github.com/shejialuo/git/pull/8\n\nThanks,\nJialuo\n\nshejialuo (10):\n fsck: rename \"skiplist\" to \"skip_oids\"\n fsck: rename objects-related fsck error functions\n fsck: add a unified interface for reporting fsck messages\n fsck: add refs-related error report function\n refs: set up ref consistency check infrastructure\n builtin/refs: add verify subcommand and verbose_refs for\n \"fsck_options\"\n builtin/fsck: add `git-refs verify` child process\n files-backend: add unified interface for refs scanning\n fsck: add ref name check for files backend\n fsck: add ref content check for files backend\n\n Documentation/fsck-msgids.txt | 12 ++\n Documentation/git-refs.txt | 13 ++\n builtin/fsck.c | 32 ++++-\n builtin/mktag.c | 1 +\n builtin/refs.c | 44 ++++++\n fsck.c | 109 ++++++++++++---\n fsck.h | 66 ++++++---\n object-file.c | 11 +-\n refs.c | 7 +-\n refs.h | 8 ++\n refs/debug.c | 11 ++\n refs/files-backend.c | 255 +++++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 ++\n refs/refs-internal.h | 11 +-\n refs/reftable-backend.c | 8 ++\n t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++\n 16 files changed, 750 insertions(+), 57 deletions(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\nRange-diff against v9:\n 1: e044f933de = 1: e044f933de fsck: rename \"skiplist\" to \"skip_oids\"\n -: ---------- > 2: 73a7c53a23 fsck: rename objects-related fsck error functions\n 2: daaf3d0ffe ! 3: df4837e960 fsck: add a unified interface for reporting fsck messages\n @@ Commit message\n \n ## builtin/fsck.c ##\n @@ builtin/fsck.c: static int objerror(struct object *obj, const char *err)\n - \treturn -1;\n - }\n - \n --static int fsck_error_func(struct fsck_options *o UNUSED,\n --\t\t\t const struct object_id *oid,\n --\t\t\t enum object_type object_type,\n --\t\t\t enum fsck_msg_type msg_type,\n --\t\t\t enum fsck_msg_id msg_id UNUSED,\n --\t\t\t const char *message)\n -+static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n -+\t\t\t\t const struct object_id *oid,\n -+\t\t\t\t enum object_type object_type,\n + static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n + \t\t\t\t const struct object_id *oid,\n + \t\t\t\t enum object_type object_type,\n +\t\t\t\t const char *checked_ref_name UNUSED,\n -+\t\t\t\t enum fsck_msg_type msg_type,\n -+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n -+\t\t\t\t const char *message)\n - {\n - \tswitch (msg_type) {\n - \tcase FSCK_WARN:\n -@@ builtin/fsck.c: int cmd_fsck(int argc, const char **argv, const char *prefix)\n - \n - \tfsck_walk_options.walk = mark_object;\n - \tfsck_obj_options.walk = mark_used;\n --\tfsck_obj_options.error_func = fsck_error_func;\n -+\tfsck_obj_options.error_func = fsck_objects_error_func;\n - \tif (check_strict)\n - \t\tfsck_obj_options.strict = 1;\n - \n + \t\t\t\t enum fsck_msg_type msg_type,\n + \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n + \t\t\t\t const char *message)\n \n ## builtin/mktag.c ##\n @@ builtin/mktag.c: static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n @@ fsck.c: static int report(struct fsck_options *options,\n {\n \tif (!options->object_names)\n @@ fsck.c: int fsck_buffer(const struct object_id *oid, enum object_type type,\n - \t\t type);\n - }\n - \n --int fsck_error_function(struct fsck_options *o,\n --\t\t\tconst struct object_id *oid,\n --\t\t\tenum object_type object_type UNUSED,\n --\t\t\tenum fsck_msg_type msg_type,\n --\t\t\tenum fsck_msg_id msg_id UNUSED,\n --\t\t\tconst char *message)\n -+int fsck_objects_error_function(struct fsck_options *o,\n -+\t\t\t\tconst struct object_id *oid,\n -+\t\t\t\tenum object_type object_type UNUSED,\n + int fsck_objects_error_function(struct fsck_options *o,\n + \t\t\t\tconst struct object_id *oid,\n + \t\t\t\tenum object_type object_type UNUSED,\n +\t\t\t\tconst char *checked_ref_name UNUSED,\n -+\t\t\t\tenum fsck_msg_type msg_type,\n -+\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n -+\t\t\t\tconst char *message)\n - {\n - \tif (msg_type == FSCK_WARN) {\n - \t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n + \t\t\t\tenum fsck_msg_type msg_type,\n + \t\t\t\tenum fsck_msg_id msg_id UNUSED,\n + \t\t\t\tconst char *message)\n @@ fsck.c: int git_fsck_config(const char *var, const char *value,\n - * Custom error callbacks that are used in more than one place.\n - */\n - \n --int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n --\t\t\t\t\t const struct object_id *oid,\n --\t\t\t\t\t enum object_type object_type,\n --\t\t\t\t\t enum fsck_msg_type msg_type,\n --\t\t\t\t\t enum fsck_msg_id msg_id,\n --\t\t\t\t\t const char *message)\n -+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n -+\t\t\t\t\t\t const struct object_id *oid,\n -+\t\t\t\t\t\t enum object_type object_type,\n + int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n + \t\t\t\t\t\t const struct object_id *oid,\n + \t\t\t\t\t\t enum object_type object_type,\n +\t\t\t\t\t\t const char *checked_ref_name,\n -+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n -+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n -+\t\t\t\t\t\t const char *message)\n - {\n - \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n + \t\t\t\t\t\t enum fsck_msg_type msg_type,\n + \t\t\t\t\t\t enum fsck_msg_id msg_id,\n + \t\t\t\t\t\t const char *message)\n +@@ fsck.c: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n --\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n +-\treturn fsck_objects_error_function(o, oid, object_type,\n +\treturn fsck_objects_error_function(o, oid, object_type, checked_ref_name,\n -+\t\t\t\t\t msg_type, msg_id, message);\n + \t\t\t\t\t msg_type, msg_id, message);\n }\n \n ## fsck.h ##\n @@ fsck.h: int is_valid_msg_type(const char *msg_id, const char *msg_type);\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n --int fsck_error_function(struct fsck_options *o,\n --\t\t\tconst struct object_id *oid, enum object_type object_type,\n --\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n --\t\t\tconst char *message);\n --int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n --\t\t\t\t\t const struct object_id *oid,\n --\t\t\t\t\t enum object_type object_type,\n --\t\t\t\t\t enum fsck_msg_type msg_type,\n --\t\t\t\t\t enum fsck_msg_id msg_id,\n --\t\t\t\t\t const char *message);\n -+int fsck_objects_error_function(struct fsck_options *o,\n -+\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n + int fsck_objects_error_function(struct fsck_options *o,\n + \t\t\t\tconst struct object_id *oid, enum object_type object_type,\n +\t\t\t\tconst char *checked_ref_name,\n -+\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n -+\t\t\t\tconst char *message);\n -+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n -+\t\t\t\t\t\t const struct object_id *oid,\n -+\t\t\t\t\t\t enum object_type object_type,\n + \t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n + \t\t\t\tconst char *message);\n + int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n + \t\t\t\t\t\t const struct object_id *oid,\n + \t\t\t\t\t\t enum object_type object_type,\n +\t\t\t\t\t\t const char *checked_ref_name,\n -+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n -+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n -+\t\t\t\t\t\t const char *message);\n - \n - struct fsck_options {\n - \tfsck_walk_func walk;\n -@@ fsck.h: struct fsck_options {\n - \t.gitmodules_done = OIDSET_INIT, \\\n - \t.gitattributes_found = OIDSET_INIT, \\\n - \t.gitattributes_done = OIDSET_INIT, \\\n --\t.error_func = fsck_error_function \\\n -+\t.error_func = fsck_objects_error_function \\\n - }\n - #define FSCK_OPTIONS_STRICT { \\\n - \t.strict = 1, \\\n -@@ fsck.h: struct fsck_options {\n - \t.gitmodules_done = OIDSET_INIT, \\\n - \t.gitattributes_found = OIDSET_INIT, \\\n - \t.gitattributes_done = OIDSET_INIT, \\\n --\t.error_func = fsck_error_function, \\\n -+\t.error_func = fsck_objects_error_function, \\\n - }\n - #define FSCK_OPTIONS_MISSING_GITMODULES { \\\n - \t.strict = 1, \\\n -@@ fsck.h: struct fsck_options {\n - \t.gitmodules_done = OIDSET_INIT, \\\n - \t.gitattributes_found = OIDSET_INIT, \\\n - \t.gitattributes_done = OIDSET_INIT, \\\n --\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n -+\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n - }\n - \n - /* descend in all linked child objects\n + \t\t\t\t\t\t enum fsck_msg_type msg_type,\n + \t\t\t\t\t\t enum fsck_msg_id msg_id,\n + \t\t\t\t\t\t const char *message);\n @@ fsck.h: int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n ++/*\n ++ * Report an error or warning for refs.\n ++ */\n +__attribute__((format (printf, 5, 6)))\n +int fsck_refs_report(struct fsck_options *options,\n +\t\t const struct object_id *oid,\n 3: 40da85ae30 ! 4: c8eb01c987 fsck: add refs-related options and error report function\n @@ Metadata\n Author: shejialuo \n \n ## Commit message ##\n - fsck: add refs-related options and error report function\n + fsck: add refs-related error report function\n \n Add refs-related options to the \"fsck_options\", create refs-specific\n \"error_func\" callback \"fsck_refs_error_function\".\n @@ fsck.c: int fsck_objects_error_function(struct fsck_options *o,\n +\t\t\t enum fsck_msg_id msg_id UNUSED,\n +\t\t\t const char *message)\n +{\n -+\tstatic struct strbuf sb = STRBUF_INIT;\n ++\tstruct strbuf sb = STRBUF_INIT;\n ++\tint ret = 0;\n +\n -+\tstrbuf_reset(&sb);\n +\tstrbuf_addstr(&sb, checked_ref_name);\n +\tif (oid)\n +\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n +\n -+\tif (msg_type == FSCK_WARN) {\n ++\tif (msg_type == FSCK_WARN)\n +\t\twarning(\"%s: %s\", sb.buf, message);\n -+\t\treturn 0;\n -+\t}\n -+\terror(\"%s: %s\", sb.buf, message);\n -+\treturn 1;\n ++\telse\n ++\t\tret = error(\"%s: %s\", sb.buf, message);\n ++\n ++\tstrbuf_release(&sb);\n ++\treturn ret;\n +}\n +\n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n @@ fsck.h: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *\n \n struct fsck_options {\n \tfsck_walk_func walk;\n - \tfsck_error error_func;\n - \tunsigned strict:1;\n -+\tunsigned verbose_refs:1;\n - \tenum fsck_msg_type *msg_type;\n - \tstruct oidset skip_oids;\n - \tstruct oidset gitmodules_found;\n @@ fsck.h: struct fsck_options {\n \t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n 4: a38ea1b117 ! 5: e4085df496 refs: set up ref consistency check infrastructure\n @@ Commit message\n grouped into any above five categories. Explicitly add blank line to\n make it different from others.\n \n + Last, implement placeholder functions for each ref backends.\n +\n Mentored-by: Patrick Steinhardt \n Mentored-by: Karthik Nayak \n Signed-off-by: shejialuo \n 5: 8320f56e0b ! 6: 497f224bed builtin/refs: add verify subcommand\n @@ Metadata\n Author: shejialuo \n \n ## Commit message ##\n - builtin/refs: add verify subcommand\n + builtin/refs: add verify subcommand and verbose_refs for \"fsck_options\"\n \n Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\n - check the reference database consistency.\n + check the reference database consistency and also this subcommand will\n + be used as the entry point of checking refs for \"git-fsck(1)\". Last, add\n + \"verbose_refs\" field into \"fsck_options\" to indicate whether we should\n + print verbose messages when checking refs consistency.\n \n Mentored-by: Patrick Steinhardt \n Mentored-by: Karthik Nayak \n @@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c\n +\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n +\t\tOPT_END(),\n +\t};\n -+\tint ret = 0;\n ++\tint ret;\n +\n +\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n +\tif (argc)\n @@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c\n +\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n +\n +\t/*\n -+\t * Explicitly free the allocated array and \"oid_skiplist\"\n ++\t * Explicitly free the allocated array and \"skip_oids\" set\n +\t */\n +\tfree(fsck_refs_options.msg_type);\n +\toidset_clear(&fsck_refs_options.skip_oids);\n @@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c\n \t\tOPT_END(),\n \t};\n \n +\n + ## fsck.h ##\n +@@ fsck.h: struct fsck_options {\n + \tfsck_walk_func walk;\n + \tfsck_error error_func;\n + \tunsigned strict:1;\n ++\tunsigned verbose_refs:1;\n + \tenum fsck_msg_type *msg_type;\n + \tstruct oidset skip_oids;\n + \tstruct oidset gitmodules_found;\n 6: 6614a06ef5 = 7: 86a14c7b43 builtin/fsck: add `git-refs verify` child process\n 7: 928cc96396 = 8: daedb80b47 files-backend: add unified interface for refs scanning\n 8: 4d50d4932f = 9: c36d588e4f fsck: add ref name check for files backend\n 9: 7edb810819 = 10: 521e0d9ca3 fsck: add ref content check for files backend\n-- \n2.45.2\n\n"},{"id":"498453","messageId":"Zo6ezSp2Lx2G9N7N@ArchLinux","threadId":"61565","inReplyTo":"Zo6eJi8BePrQxTQV@ArchLinux","subject":"[GSoC][PATCH v10 01/10] fsck: rename \"skiplist\" to \"skip_oids\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T14:46:37Z","receivedAt":"2024-07-10T14:46:38Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The \"skiplist\" field in \"fsck_options\" is related to objects. Because we\nare going to introduce ref consistency check, the \"skiplist\" name is too\ngeneral which will make the caller think \"skiplist\" is related to both\nthe refs and objects.\n\nIt may seem that for both refs and objects, we should provide a general\n\"skiplist\" here. However, the type for \"skiplist\" is `struct oidset`\nwhich is totally unsuitable for refs.\n\nTo avoid above ambiguity, rename \"skiplist\" to \"skip_oids\".\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 4 ++--\n fsck.h | 4 ++--\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex eea7145470..3f32441492 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n \t\tif (!strcmp(buf, \"skiplist\")) {\n \t\t\tif (equal == len)\n \t\t\t\tdie(\"skiplist requires a path\");\n-\t\t\toidset_parse_file(&options->skiplist, buf + equal + 1,\n+\t\t\toidset_parse_file(&options->skip_oids, buf + equal + 1,\n \t\t\t\t\t the_repository->hash_algo);\n \t\t\tbuf += len + 1;\n \t\t\tcontinue;\n@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n static int object_on_skiplist(struct fsck_options *opts,\n \t\t\t const struct object_id *oid)\n {\n-\treturn opts && oid && oidset_contains(&opts->skiplist, oid);\n+\treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n __attribute__((format (printf, 5, 6)))\ndiff --git a/fsck.h b/fsck.h\nindex 6085a384f6..bcfb2e34cd 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -136,7 +136,7 @@ struct fsck_options {\n \tfsck_error error_func;\n \tunsigned strict:1;\n \tenum fsck_msg_type *msg_type;\n-\tstruct oidset skiplist;\n+\tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n \tstruct oidset gitattributes_found;\n@@ -145,7 +145,7 @@ struct fsck_options {\n };\n \n #define FSCK_OPTIONS_DEFAULT { \\\n-\t.skiplist = OIDSET_INIT, \\\n+\t.skip_oids = OIDSET_INIT, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n-- \n2.45.2\n\n"},{"id":"498454","messageId":"Zo6e3WIe7yJSs1_l@ArchLinux","threadId":"61565","inReplyTo":"Zo6eJi8BePrQxTQV@ArchLinux","subject":"[GSoC][PATCH v10 02/10] fsck: rename objects-related fsck error functions","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T14:46:53Z","receivedAt":"2024-07-10T14:46:54Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The names of objects-related fsck error functions are general. It's OK\nwhen there is only object database check. However, we are going to\nintroduce refs database check. In order to avoid ambiguity, rename\nobjects-related fsck error functions to explicitly indicate these\nfunctions are used to report objects-related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 14 +++++++-------\n fsck.c | 27 ++++++++++++++-------------\n fsck.h | 26 +++++++++++++-------------\n 3 files changed, 34 insertions(+), 33 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..6d86bbe1e9 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -89,12 +89,12 @@ static int objerror(struct object *obj, const char *err)\n \treturn -1;\n }\n \n-static int fsck_error_func(struct fsck_options *o UNUSED,\n-\t\t\t const struct object_id *oid,\n-\t\t\t enum object_type object_type,\n-\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t const char *message)\n+static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t enum object_type object_type,\n+\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \tswitch (msg_type) {\n \tcase FSCK_WARN:\n@@ -938,7 +938,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tfsck_walk_options.walk = mark_object;\n \tfsck_obj_options.walk = mark_used;\n-\tfsck_obj_options.error_func = fsck_error_func;\n+\tfsck_obj_options.error_func = fsck_objects_error_func;\n \tif (check_strict)\n \t\tfsck_obj_options.strict = 1;\n \ndiff --git a/fsck.c b/fsck.c\nindex 3f32441492..0aaff7f635 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1200,12 +1200,12 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid,\n-\t\t\tenum object_type object_type UNUSED,\n-\t\t\tenum fsck_msg_type msg_type,\n-\t\t\tenum fsck_msg_id msg_id UNUSED,\n-\t\t\tconst char *message)\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tenum object_type object_type UNUSED,\n+\t\t\t\tenum fsck_msg_type msg_type,\n+\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n+\t\t\t\tconst char *message)\n {\n \tif (msg_type == FSCK_WARN) {\n \t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n@@ -1303,16 +1303,17 @@ int git_fsck_config(const char *var, const char *value,\n * Custom error callbacks that are used in more than one place.\n */\n \n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message)\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message)\n {\n \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n+\treturn fsck_objects_error_function(o, oid, object_type,\n+\t\t\t\t\t msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex bcfb2e34cd..41ebebbb59 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -120,16 +120,16 @@ typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid, enum object_type object_type,\n-\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n-\t\t\tconst char *message);\n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message);\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n+\t\t\t\tconst char *message);\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message);\n \n struct fsck_options {\n \tfsck_walk_func walk;\n@@ -150,7 +150,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function \\\n+\t.error_func = fsck_objects_error_function \\\n }\n #define FSCK_OPTIONS_STRICT { \\\n \t.strict = 1, \\\n@@ -158,7 +158,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function, \\\n+\t.error_func = fsck_objects_error_function, \\\n }\n #define FSCK_OPTIONS_MISSING_GITMODULES { \\\n \t.strict = 1, \\\n@@ -166,7 +166,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n+\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n \n /* descend in all linked child objects\n-- \n2.45.2\n\n"},{"id":"498455","messageId":"Zo6e61kSAvuvBvBS@ArchLinux","threadId":"61565","inReplyTo":"Zo6eJi8BePrQxTQV@ArchLinux","subject":"[GSoC][PATCH v10 03/10] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T14:47:07Z","receivedAt":"2024-07-10T14:47:07Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The static function \"report\" provided by \"fsck.c\" aims at checking fsck\nerror type and calling the callback \"error_func\" to report the message.\nHowever, \"report\" function is only related to object database which\ncannot be reused for refs. In order to provide a unified interface which\ncan report either objects or refs, create a new function \"vfsck_report\"\nby adding \"checked_ref_name\" parameter following the \"report\" prototype.\nInstead of using \"...\", provide \"va_list\" to allow more flexibility.\n\nLike \"report\", the \"vfsck_report\" function will use \"error_func\"\nregistered in \"fsck_options\" to report customized messages. Change\n\"error_func\" prototype to align with the new \"vfsck_report\".\n\nThen, change \"report\" function to use \"vfsck_report\" to report objects\nrelated messages. Add a new function called \"fsck_refs_report\" to use\n\"vfsck_report\" to report refs related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 1 +\n builtin/mktag.c | 1 +\n fsck.c | 56 +++++++++++++++++++++++++++++++++++++++++--------\n fsck.h | 17 ++++++++++++++-\n object-file.c | 11 +++++-----\n 5 files changed, 71 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 6d86bbe1e9..de34538c4f 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -92,6 +92,7 @@ static int objerror(struct object *obj, const char *err)\n static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n \t\t\t\t const struct object_id *oid,\n \t\t\t\t enum object_type object_type,\n+\t\t\t\t const char *checked_ref_name UNUSED,\n \t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t\t const char *message)\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex 4767f1a97e..42f945c584 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -20,6 +20,7 @@ static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n \t\t\t\t const struct object_id *oid UNUSED,\n \t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const char *checked_ref_name UNUSED,\n \t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t\t const char *message)\ndiff --git a/fsck.c b/fsck.c\nindex 0aaff7f635..e1819964e3 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -226,12 +226,18 @@ static int object_on_skiplist(struct fsck_options *opts,\n \treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n-__attribute__((format (printf, 5, 6)))\n-static int report(struct fsck_options *options,\n-\t\t const struct object_id *oid, enum object_type object_type,\n-\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+/*\n+ * Provide a unified interface for either fscking refs or objects.\n+ * It will get the current msg error type and call the error_func callback\n+ * which is registered in the \"fsck_options\" struct.\n+ */\n+static int vfsck_report(struct fsck_options *options,\n+\t\t\tconst struct object_id *oid,\n+\t\t\tenum object_type object_type,\n+\t\t\tconst char *checked_ref_name,\n+\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n {\n-\tva_list ap;\n+\tva_list ap_copy;\n \tstruct strbuf sb = STRBUF_INIT;\n \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n \tint result;\n@@ -250,9 +256,9 @@ static int report(struct fsck_options *options,\n \tprepare_msg_ids();\n \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n \n-\tva_start(ap, fmt);\n-\tstrbuf_vaddf(&sb, fmt, ap);\n-\tresult = options->error_func(options, oid, object_type,\n+\tva_copy(ap_copy, ap);\n+\tstrbuf_vaddf(&sb, fmt, ap_copy);\n+\tresult = options->error_func(options, oid, object_type, checked_ref_name,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n \tva_end(ap);\n@@ -260,6 +266,36 @@ static int report(struct fsck_options *options,\n \treturn result;\n }\n \n+__attribute__((format (printf, 5, 6)))\n+static int report(struct fsck_options *options,\n+\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = vfsck_report(options, oid, object_type, NULL,\n+\t\t\t msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n+\n+\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t const char *checked_ref_name,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = vfsck_report(options, oid, OBJ_NONE,\n+\t\t\t checked_ref_name, msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n void fsck_enable_object_names(struct fsck_options *options)\n {\n \tif (!options->object_names)\n@@ -1203,6 +1239,7 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n int fsck_objects_error_function(struct fsck_options *o,\n \t\t\t\tconst struct object_id *oid,\n \t\t\t\tenum object_type object_type UNUSED,\n+\t\t\t\tconst char *checked_ref_name UNUSED,\n \t\t\t\tenum fsck_msg_type msg_type,\n \t\t\t\tenum fsck_msg_id msg_id UNUSED,\n \t\t\t\tconst char *message)\n@@ -1306,6 +1343,7 @@ int git_fsck_config(const char *var, const char *value,\n int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t const char *checked_ref_name,\n \t\t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message)\n@@ -1314,6 +1352,6 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_objects_error_function(o, oid, object_type,\n+\treturn fsck_objects_error_function(o, oid, object_type, checked_ref_name,\n \t\t\t\t\t msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex 41ebebbb59..f88e5faa94 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -114,19 +114,24 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n \t\t\t void *data, struct fsck_options *options);\n \n-/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n+/*\n+ * callback function for reporting errors when checking either objects or refs\n+ */\n typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t\t const char *checked_ref_name,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n int fsck_objects_error_function(struct fsck_options *o,\n \t\t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\t\tconst char *checked_ref_name,\n \t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t\tconst char *message);\n int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t const char *checked_ref_name,\n \t\t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message);\n@@ -209,6 +214,16 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+/*\n+ * Report an error or warning for refs.\n+ */\n+__attribute__((format (printf, 5, 6)))\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t const char *checked_ref_name,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\ndiff --git a/object-file.c b/object-file.c\nindex 065103be3e..d2c6427935 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2470,11 +2470,12 @@ int repo_has_object_file(struct repository *r,\n * give more context.\n */\n static int hash_format_check_report(struct fsck_options *opts UNUSED,\n-\t\t\t\t const struct object_id *oid UNUSED,\n-\t\t\t\t enum object_type object_type UNUSED,\n-\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n-\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t\t const char *message)\n+\t\t\t\t const struct object_id *oid UNUSED,\n+\t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const char *ref_checked_name UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \terror(_(\"object fails fsck: %s\"), message);\n \treturn 1;\n-- \n2.45.2\n\n"},{"id":"498456","messageId":"Zo6e-WdPpSjMhPPV@ArchLinux","threadId":"61565","inReplyTo":"Zo6eJi8BePrQxTQV@ArchLinux","subject":"[GSoC][PATCH v10 04/10] fsck: add refs-related error report function","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T14:47:21Z","receivedAt":"2024-07-10T14:47:21Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Add refs-related options to the \"fsck_options\", create refs-specific\n\"error_func\" callback \"fsck_refs_error_function\".\n\n\"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\npasses the oid, it will use \"oid_to_hex\" to get the corresponding hex\nvalue to report to the caller.\n\nLast, add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\"\nmacros to create refs options easily.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 24 ++++++++++++++++++++++++\n fsck.h | 14 ++++++++++++++\n 2 files changed, 38 insertions(+)\n\ndiff --git a/fsck.c b/fsck.c\nindex e1819964e3..f14ac533b0 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1252,6 +1252,30 @@ int fsck_objects_error_function(struct fsck_options *o,\n \treturn 1;\n }\n \n+int fsck_refs_error_function(struct fsck_options *options UNUSED,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type UNUSED,\n+\t\t\t const char *checked_ref_name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tstrbuf_addstr(&sb, checked_ref_name);\n+\tif (oid)\n+\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n+\n+\tif (msg_type == FSCK_WARN)\n+\t\twarning(\"%s: %s\", sb.buf, message);\n+\telse\n+\t\tret = error(\"%s: %s\", sb.buf, message);\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n \t\t struct fsck_options *options, const char *blob_type)\ndiff --git a/fsck.h b/fsck.h\nindex f88e5faa94..fe5d4d2ad9 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -135,6 +135,13 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message);\n+int fsck_refs_error_function(struct fsck_options *options,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type,\n+\t\t\t const char *checked_ref_name,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n \n struct fsck_options {\n \tfsck_walk_func walk;\n@@ -173,6 +180,13 @@ struct fsck_options {\n \t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n+#define FSCK_REFS_OPTIONS_STRICT { \\\n+\t.strict = 1, \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n \n /* descend in all linked child objects\n * the return value is:\n-- \n2.45.2\n\n"},{"id":"498457","messageId":"Zo6fB9uMZDSCQ2HD@ArchLinux","threadId":"61565","inReplyTo":"Zo6eJi8BePrQxTQV@ArchLinux","subject":"[GSoC][PATCH v10 05/10] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T14:47:35Z","receivedAt":"2024-07-10T14:47:35Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It's organized as the five parts:\n\n1. The name and the initialization interfaces.\n2. The ref transaction interfaces.\n3. The ref internal interfaces (pack, rename and copy).\n4. The ref filesystem interfaces.\n5. The reflog related interfaces.\n\nTo keep consistent with the git-fsck(1), add a new interface named\n\"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\ngrouped into any above five categories. Explicitly add blank line to\nmake it different from others.\n\nLast, implement placeholder functions for each ref backends.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs.c | 5 +++++\n refs.h | 8 ++++++++\n refs/debug.c | 11 +++++++++++\n refs/files-backend.c | 15 ++++++++++++++-\n refs/packed-backend.c | 8 ++++++++\n refs/refs-internal.h | 6 ++++++\n refs/reftable-backend.c | 8 ++++++++\n 7 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex bb90a18875..410919246b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs, struct fsck_options *o)\n+{\n+\treturn refs->be->fsck(refs, o);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex 0ecba21b4a..804d6a7fce 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -4,6 +4,7 @@\n #include \"commit.h\"\n #include \"repository.h\"\n \n+struct fsck_options;\n struct object_id;\n struct ref_store;\n struct strbuf;\n@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_options *o);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..45e2e784a0 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tint res = drefs->refs->be->fsck(drefs->refs, o);\n+\ttrace_printf_key(&trace_refs, \"fsck: %d\\n\", res);\n+\treturn res;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex aa52d9be7c..d89eeda8ef 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3408,6 +3408,17 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\treturn ret;\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_init,\n@@ -3434,5 +3445,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a0666407cd..5209b0b212 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_init,\n@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex fa975d69aa..a905e187cd 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_options;\n struct ref_transaction;\n \n /*\n@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_options *o);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -677,6 +681,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex fbe74c239d..b5a1a526df 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.45.2\n\n"},{"id":"498458","messageId":"Zo6fFS7xzFwWKrEW@ArchLinux","threadId":"61565","inReplyTo":"Zo6eJi8BePrQxTQV@ArchLinux","subject":"[GSoC][PATCH v10 06/10] builtin/refs: add verify subcommand and verbose_refs for \"fsck_options\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T14:47:49Z","receivedAt":"2024-07-10T14:47:49Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\ncheck the reference database consistency and also this subcommand will\nbe used as the entry point of checking refs for \"git-fsck(1)\". Last, add\n\"verbose_refs\" field into \"fsck_options\" to indicate whether we should\nprint verbose messages when checking refs consistency.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/git-refs.txt | 13 +++++++++++\n builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++\n fsck.h | 1 +\n 3 files changed, 58 insertions(+)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..1244a85b64 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+The following options are specific to 'git refs verify':\n+\n+--strict::\n+\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n+\tbe seen as ERROR. See linkgit:git-fsck[1].\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..599b526786 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,54 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tunsigned int verbose = 0, strict = 0;\n+\tstruct option options[] = {\n+\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\tint ret;\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"too many arguments\"));\n+\n+\tif (verbose)\n+\t\tfsck_refs_options.verbose_refs = 1;\n+\tif (strict)\n+\t\tfsck_refs_options.strict = 1;\n+\n+\tgit_config(git_fsck_config, &fsck_refs_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n+\n+\t/*\n+\t * Explicitly free the allocated array and \"skip_oids\" set\n+\t */\n+\tfree(fsck_refs_options.msg_type);\n+\toidset_clear(&fsck_refs_options.skip_oids);\n+\treturn ret;\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/fsck.h b/fsck.h\nindex fe5d4d2ad9..ef1f1ed15e 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -147,6 +147,7 @@ struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n \tunsigned strict:1;\n+\tunsigned verbose_refs:1;\n \tenum fsck_msg_type *msg_type;\n \tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n-- \n2.45.2\n\n"},{"id":"498459","messageId":"Zo6fJCt-isAMDveB@ArchLinux","threadId":"61565","inReplyTo":"Zo6eJi8BePrQxTQV@ArchLinux","subject":"[GSoC][PATCH v10 07/10] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T14:48:04Z","receivedAt":"2024-07-10T14:48:04Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new function \"fsck_refs\" that initializes and runs a child\nprocess to execute the \"git-refs verify\" command.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex de34538c4f..ec3357722c 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -897,6 +897,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \treturn res;\n }\n \n+static void fsck_refs(void)\n+{\n+\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n+\tchild_process_init(&refs_verify);\n+\trefs_verify.git_cmd = 1;\n+\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n+\tif (verbose)\n+\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n+\tif (check_strict)\n+\t\tstrvec_push(&refs_verify.args, \"--strict\");\n+\n+\tif (run_command(&refs_verify))\n+\t\terrors_found |= ERROR_REFS;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n@@ -1066,6 +1081,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n+\tfsck_refs();\n+\n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \n-- \n2.45.2\n\n"},{"id":"498460","messageId":"Zo6fMt-ikLAYxMH3@ArchLinux","threadId":"61565","inReplyTo":"Zo6eJi8BePrQxTQV@ArchLinux","subject":"[GSoC][PATCH v10 08/10] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T14:48:18Z","receivedAt":"2024-07-10T14:48:18Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"For refs and reflogs, we need to scan its corresponding directories to\ncheck every regular file or symbolic link which shares the same pattern.\nIntroduce a unified interface for scanning directories for\nfiles-backend.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs/files-backend.c | 77 +++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 76 insertions(+), 1 deletion(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex d89eeda8ef..84acb58782 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -6,6 +6,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -3408,6 +3409,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n+{\n+\tconst char *gitdir = ref_store->gitdir;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\n+\tif (!iter) {\n+\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n+\t\tgoto out;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->verbose_refs)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n+\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n+\t\t\t\t iter->basename);\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n+\n+out:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tNULL\n+\t};\n+\n+\tif (o->verbose_refs)\n+\t\tfprintf_ln(stderr, \"Checking references consistency\");\n+\n+\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n+\n+\treturn ret;\n+}\n+\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_options *o)\n {\n@@ -3415,7 +3488,9 @@ static int files_fsck(struct ref_store *ref_store,\n \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n-\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o)\n+\t | files_fsck_refs(ref_store, o);\n+\n \treturn ret;\n }\n \n-- \n2.45.2\n\n"},{"id":"498461","messageId":"Zo6fQJIzH8DRfgKg@ArchLinux","threadId":"61565","inReplyTo":"Zo6eJi8BePrQxTQV@ArchLinux","subject":"[GSoC][PATCH v10 09/10] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T14:48:32Z","receivedAt":"2024-07-10T14:48:32Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) only implicitly checks the reference, it does not fully\ncheck refs with bad format name such as standalone \"@\" and name ending\nwith \".lock\".\n\nIn order to provide such checks, add a new fsck message id \"badRefName\"\nwith default ERROR type. Use existing \"check_refname_format\" to explicit\ncheck the ref name. And add a new unit test to verify the functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 +\n fsck.h | 1 +\n refs/files-backend.c | 20 +++++++\n t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++\n 4 files changed, 125 insertions(+)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex f643585a34..dab4012246 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,9 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefName`::\n+\t(ERROR) A ref has a bad name.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex ef1f1ed15e..5fcb249735 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 84acb58782..69a76048d3 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3419,6 +3419,25 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n+static int files_fsck_refs_name(struct fsck_options *o,\n+\t\t\t\tconst char *gitdir UNUSED,\n+\t\t\t\tconst char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\tret = fsck_refs_report(o, NULL, sb.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3470,6 +3489,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n {\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tfiles_fsck_refs_name,\n \t\tNULL\n \t};\n \ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..b2db58d2c6\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit tag multi_hierarchy/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1.lock: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/@ &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\tgit -c fsck.badRefName=warn fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\tgit -c fsck.badRefName=ignore fsck 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+test_done\n-- \n2.45.2\n\n"},{"id":"498462","messageId":"Zo6fTrrcAyagcg_W@ArchLinux","threadId":"61565","inReplyTo":"Zo6eJi8BePrQxTQV@ArchLinux","subject":"[GSoC][PATCH v10 10/10] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-10T14:48:46Z","receivedAt":"2024-07-10T14:48:46Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Enhance the git-fsck(1) command by adding a check for reference content\nin the files backend. The new functionality ensures that symrefs, real\nsymbolic link and regular refs are validated correctly.\n\nIn order to check the trailing content of the regular refs, add a new\nparameter `trailing` to `parse_loose_ref_contents`.\n\nFor symrefs, `parse_loose_ref_contents` will set the \"referent\".\nHowever, symbolic link could be either absolute or relative. Use\n\"strbuf_add_real_path\" to read the symbolic link and convert the\nrelative path to absolute path. Then use \"skip_prefix\" to make it align\nwith symref \"referent\".\n\nThus, the symrefs and symbolic links could share the same interface. Add\na new function \"files_fsck_symref_target\" which aims at checking the\nfollowing things:\n\n1. whether the pointee is under the `refs/` directory.\n2. whether the pointee name is correct.\n3. whether the pointee path is a wrong type in filesystem.\n\nLast, add the following FSCK MESSAGEs:\n\n1. \"badRefContent(ERROR)\": A ref has a bad content\n2. \"badSymrefPointee(ERROR)\": The pointee of a symref is bad.\n3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 9 +++\n fsck.h | 3 +\n refs.c | 2 +-\n refs/files-backend.c | 145 +++++++++++++++++++++++++++++++++-\n refs/refs-internal.h | 5 +-\n t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++\n 6 files changed, 269 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex dab4012246..b1630a478b 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,9 +19,15 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefContent`::\n+\t(ERROR) A ref has a bad content.\n+\n `badRefName`::\n \t(ERROR) A ref has a bad name.\n \n+`badSymrefPointee`::\n+\t(ERROR) The pointee of a symref is bad.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \n@@ -167,6 +173,9 @@\n `nullSha1`::\n \t(WARN) Tree contains entries pointing to a null sha1.\n \n+`trailingRefContent`::\n+\t(WARN) A ref content has trailing contents.\n+\n `treeNotSorted`::\n \t(ERROR) A tree is not properly sorted.\n \ndiff --git a/fsck.h b/fsck.h\nindex 5fcb249735..3ad2cc86c9 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -32,6 +32,8 @@ enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n+\tFUNC(BAD_REF_CONTENT, ERROR) \\\n+\tFUNC(BAD_SYMREF_POINTEE, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n@@ -72,6 +74,7 @@ enum fsck_msg_type {\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n+\tFUNC(TRAILING_REF_CONTENT, WARN) \\\n \tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n \tFUNC(LARGE_PATHNAME, WARN) \\\ndiff --git a/refs.c b/refs.c\nindex 410919246b..eb82fb7d4e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1760,7 +1760,7 @@ static int refs_read_special_head(struct ref_store *ref_store,\n \t}\n \n \tresult = parse_loose_ref_contents(content.buf, oid, referent, type,\n-\t\t\t\t\t failure_errno);\n+\t\t\t\t\t failure_errno, NULL);\n \n done:\n \tstrbuf_release(&full_path);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 69a76048d3..d98ef45403 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1,6 +1,7 @@\n #define USE_THE_REPOSITORY_VARIABLE\n \n #include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n #include \"../copy.h\"\n #include \"../environment.h\"\n #include \"../gettext.h\"\n@@ -553,7 +554,7 @@ static int read_ref_internal(struct ref_store *ref_store, const char *refname,\n \tstrbuf_rtrim(&sb_contents);\n \tbuf = sb_contents.buf;\n \n-\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr);\n+\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr, NULL);\n \n out:\n \tif (ret && !myerr)\n@@ -589,7 +590,7 @@ static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refn\n \n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno)\n+\t\t\t int *failure_errno, const char **trailing)\n {\n \tconst char *p;\n \tif (skip_prefix(buf, \"ref:\", &buf)) {\n@@ -611,6 +612,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t*failure_errno = EINVAL;\n \t\treturn -1;\n \t}\n+\n+\tif (trailing)\n+\t\t*trailing = p;\n+\n \treturn 0;\n }\n \n@@ -3438,6 +3443,141 @@ static int files_fsck_refs_name(struct fsck_options *o,\n \treturn ret;\n }\n \n+/*\n+ * Check the symref \"pointee_name\" and \"pointee_path\". The caller should\n+ * make sure that \"pointee_path\" is absolute. For symbolic ref, \"pointee_name\"\n+ * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n+ * be the relative path agaignst \"gitdir\".\n+ */\n+static int files_fsck_symref_target(struct fsck_options *o,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const char *pointee_name,\n+\t\t\t\t const char *pointee_path)\n+{\n+\tconst char *p = NULL;\n+\tstruct stat st;\n+\tint ret = 0;\n+\n+\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n+\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to target out of refs hierarchy\");\n+\t\tgoto out;\n+\t}\n+\n+\tif (check_refname_format(pointee_name, 0)) {\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid refname\");\n+\t}\n+\n+\tif (lstat(pointee_path, &st) < 0)\n+\t\tgoto out;\n+\n+\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n+\t\tret = fsck_refs_report(o, NULL, refname,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid target\");\n+\t\tgoto out;\n+\t}\n+out:\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs_content(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter)\n+{\n+\tstruct strbuf pointee_path = STRBUF_INIT,\n+\t\t ref_content = STRBUF_INIT,\n+\t\t abs_gitdir = STRBUF_INIT,\n+\t\t referent = STRBUF_INIT,\n+\t\t refname = STRBUF_INIT;\n+\tconst char *trailing = NULL;\n+\tint failure_errno = 0;\n+\tunsigned int type = 0;\n+\tstruct object_id oid;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\n+\t/*\n+\t * If the file is a symlink, we need to only check the connectivity\n+\t * of the destination object.\n+\t */\n+\tif (S_ISLNK(iter->st.st_mode)) {\n+\t\tconst char *pointee_name = NULL;\n+\n+\t\tstrbuf_add_real_path(&pointee_path, iter->path.buf);\n+\n+\t\tstrbuf_add_absolute_path(&abs_gitdir, gitdir);\n+\t\tstrbuf_normalize_path(&abs_gitdir);\n+\t\tif (!is_dir_sep(abs_gitdir.buf[abs_gitdir.len - 1]))\n+\t\t\tstrbuf_addch(&abs_gitdir, '/');\n+\n+\t\tif (!skip_prefix(pointee_path.buf,\n+\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n+\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t\t \"point to target outside gitdir\");\n+\t\t\tgoto clean;\n+\t\t}\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, pointee_name,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n+\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n+\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n+\t\t\t\t &referent, &type,\n+\t\t\t\t &failure_errno, &trailing)) {\n+\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n+\t\t\t\t \"invalid ref content\");\n+\t\tgoto clean;\n+\t}\n+\n+\t/*\n+\t * If the ref is a symref, we need to check the destination name and\n+\t * connectivity.\n+\t */\n+\tif (referent.len && (type & REF_ISSYMREF)) {\n+\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n+\t\tstrbuf_rtrim(&referent);\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, referent.buf,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t} else {\n+\t\t/*\n+\t\t * Only regular refs could have a trailing garbage. Should\n+\t\t * be reported as a warning.\n+\t\t */\n+\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n+\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n+\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n+\t\t\t\t\t \"trailing garbage in ref\");\n+\t\t\tgoto clean;\n+\t\t}\n+\t}\n+\n+clean:\n+\tstrbuf_release(&abs_gitdir);\n+\tstrbuf_release(&pointee_path);\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&ref_content);\n+\tstrbuf_release(&referent);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3490,6 +3630,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n \t\tfiles_fsck_refs_name,\n+\t\tfiles_fsck_refs_content,\n \t\tNULL\n \t};\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex a905e187cd..2fabf41d14 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -709,11 +709,12 @@ struct ref_store {\n \n /*\n * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for\n- * invalid contents.\n+ * invalid contents. Also *trailing is set to the first character after the\n+ * refname or NULL if the referent is not empty.\n */\n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno);\n+\t\t\t int *failure_errno, const char **trailing);\n \n /*\n * Fill in the generic part of refs and add it to our collection of\ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nindex b2db58d2c6..35bf40ee64 100755\n--- a/t/t0602-reffiles-fsck.sh\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -98,4 +98,114 @@ test_expect_success 'ref name check should be adapted into fsck messages' '\n \t)\n '\n \n+test_expect_success 'regular ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit checkout -b a/b/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse branch-1)\" > $branch_dir_prefix/branch-1-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse tag-1)\" > $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_must_fail git -c fsck.trailingRefContent=error fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s \" \"$(git rev-parse tag-2)\" > $tag_dir_prefix/tag-2-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/tags/tag-2-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/a/b/branch-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'symbolic ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid refname\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid target\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: logs/maint-v2.45\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to target out of refs hierarchy\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.2\n\n"},{"id":"498479","messageId":"xmqqv81dufg0.fsf@gitster.g","threadId":"61565","inReplyTo":"Zo6e61kSAvuvBvBS@ArchLinux","subject":"Re: [GSoC][PATCH v10 03/10] fsck: add a unified interface for reporting fsck messages","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-10T21:04:15Z","receivedAt":"2024-07-10T21:04:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"shejialuo writes:\n\n> The static function \"report\" provided by \"fsck.c\" aims at checking fsck\n> error type and calling the callback \"error_func\" to report the message.\n> However, \"report\" function is only related to object database which\n> cannot be reused for refs. In order to provide a unified interface which\n> can report either objects or refs, create a new function \"vfsck_report\"\n> by adding \"checked_ref_name\" parameter following the \"report\" prototype.\n> Instead of using \"...\", provide \"va_list\" to allow more flexibility.\n\nLike strbuf_vinsertf(), it is a good idea to have \"v\" in the name of\na function that takes va_list, but fsck_vreport() would probably be\na better name here. Arguably, the original report() is misnamed (as\na printf-like function that takes format string, it probably would\nhave wanted to be reportf() instead), but unless we are fixing that\nat the same time, calling this fsck_vreportf() would probably be too\nmuch. Consistently misnaming it by omitting the final \"f\" would be\nfine.\n\nAt this step it is still not clear if the previous step was really\nneeded; you have this \"v\" thing that is designed to be usable by\nboth reporting issues around objects and issues around refs, but we\nwill hopefully see why when we read later patches.\n> diff --git a/object-file.c b/object-file.c\n> index 065103be3e..d2c6427935 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -2470,11 +2470,12 @@ int repo_has_object_file(struct repository *r,\n> * give more context.\n> */\n> static int hash_format_check_report(struct fsck_options *opts UNUSED,\n> -\t\t\t\t const struct object_id *oid UNUSED,\n> -\t\t\t\t enum object_type object_type UNUSED,\n> -\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n> -\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> -\t\t\t\t const char *message)\n> +\t\t\t\t const struct object_id *oid UNUSED,\n> +\t\t\t\t enum object_type object_type UNUSED,\n> +\t\t\t\t const char *ref_checked_name UNUSED,\n> +\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n> +\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> +\t\t\t\t const char *message)\n\nThat is somewhat annoying reindentation. What happened here?\n"},{"id":"498480","messageId":"xmqq7cdtue7c.fsf@gitster.g","threadId":"61565","inReplyTo":"Zo6fFS7xzFwWKrEW@ArchLinux","subject":"Re: [GSoC][PATCH v10 06/10] builtin/refs: add verify subcommand and verbose_refs for \"fsck_options\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-10T21:31:03Z","receivedAt":"2024-07-10T21:31:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"shejialuo writes:\n\n> Subject: Re: [GSoC][PATCH v10 06/10] builtin/refs: add verify subcommand and verbose_refs for \"fsck_options\"\n\nJust saying\n\n\tgit refs: add verify subcommand\n\nwould be clearer. If you really want to talk about two modes, you\ncould say\n\n\tgit refs: add \"verify [--strict|--verbose]\" subcommand\n\nbut that may be too much.\n\n> Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\n> check the reference database consistency and also this subcommand will\n> be used as the entry point of checking refs for \"git-fsck(1)\". Last, add\n> \"verbose_refs\" field into \"fsck_options\" to indicate whether we should\n> print verbose messages when checking refs consistency.\n\nIs there a reason why this has to be verbose_refs and not a simple\nverbose bit? When people see how it is useful to ask for the\nverbose output while checking refs, wouldn't people wish to add the\nsame \"--verbose\" support while checking objects, and at that point,\nwouldn't it be awkward to add verbose_objs member to the struct and\nhaving to flip both bits on?\n\n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n> Documentation/git-refs.txt | 13 +++++++++++\n> builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++\n> fsck.h | 1 +\n> 3 files changed, 58 insertions(+)\n>\n> diff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\n> index 5b99e04385..1244a85b64 100644\n> --- a/Documentation/git-refs.txt\n> +++ b/Documentation/git-refs.txt\n> @@ -10,6 +10,7 @@ SYNOPSIS\n> --------\n> [verse]\n> 'git refs migrate' --ref-format= [--dry-run]\n> +'git refs verify' [--strict] [--verbose]\n> \n> DESCRIPTION\n> -----------\n> @@ -22,6 +23,9 @@ COMMANDS\n> migrate::\n> \tMigrate ref store between different formats.\n> \n> +verify::\n> +\tVerify reference database consistency.\n> +\n\nThe error reporting function for refs consistency check was still\nabout reporting a problem for a single ref. I am wondering how\nconsistency violations that are not about a single ref should be\nhandled. For example, if refs/packed-backend.c:packed_fsck() finds\nthat the file is not sorted properly or has some unparseable garbage\nin it, it is not something you can report as \"refs/heads/main is\nbroken\", but those who are interested in seeing the \"reference\ndatabase consistency\" verified, it is very much what they want the\ntool to notice. How would detection of such a breakage that is not\nattributed to a single ref fit in this \"ref consistency check\ninfrastructure\" that was introduced by [05/10]?\n\n> +\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n> +\tif (argc)\n> +\t\tusage(_(\"too many arguments\"));\n\nI do not think we want to change this line in this topic, but\nbecause I noticed that the issue is widespread, let me make a note\nhere that we may want to clean up all the commands that give this\nmessage as a #leftoverbit item:\n\n $ git cmd foo baz\n usage: too many arguments\n\nis very unfriendly in that it is not immediately obvious to users\nwhich arguments are excess. Should they have given \"git cmd\"?\nIs it \"git cmd foo\" that does not take any argument?\n\nIf you said something like\n\n $ git refs verify baz\n error: 'git refs verify' takes no arguments\n\nor even\n\n $ git refs verify baz\n error: unknown argument 'baz' given to 'git refs verify'\n\nit would be much more helpful.\n\n\nThanks.\n"},{"id":"498507","messageId":"Zo_JKyCHrN_lJFAx@ArchLinux","threadId":"61565","inReplyTo":"xmqqv81dufg0.fsf@gitster.g","subject":"Re: [GSoC][PATCH v10 03/10] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-11T11:59:39Z","receivedAt":"2024-07-11T11:59:38Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Wed, Jul 10, 2024 at 02:04:15PM -0700, Junio C Hamano wrote:\n> shejialuo writes:\n> \n> > The static function \"report\" provided by \"fsck.c\" aims at checking fsck\n> > error type and calling the callback \"error_func\" to report the message.\n> > However, \"report\" function is only related to object database which\n> > cannot be reused for refs. In order to provide a unified interface which\n> > can report either objects or refs, create a new function \"vfsck_report\"\n> > by adding \"checked_ref_name\" parameter following the \"report\" prototype.\n> > Instead of using \"...\", provide \"va_list\" to allow more flexibility.\n> \n> Like strbuf_vinsertf(), it is a good idea to have \"v\" in the name of\n> a function that takes va_list, but fsck_vreport() would probably be\n> a better name here. Arguably, the original report() is misnamed (as\n> a printf-like function that takes format string, it probably would\n> have wanted to be reportf() instead), but unless we are fixing that\n> at the same time, calling this fsck_vreportf() would probably be too\n> much. Consistently misnaming it by omitting the final \"f\" would be\n> fine.\n> \n\nYes，I will rename it to \"fsck_vreport\".\n\n> At this step it is still not clear if the previous step was really\n> needed; you have this \"v\" thing that is designed to be usable by\n> both reporting issues around objects and issues around refs, but we\n> will hopefully see why when we read later patches.\n\nFrom my perspective, I think we should put the previous commit after\nthis commit. I agree with you that if we put it later, it will be\nmuch clearer and eaiser to understand.\n\n> > diff --git a/object-file.c b/object-file.c\n> > index 065103be3e..d2c6427935 100644\n> > --- a/object-file.c\n> > +++ b/object-file.c\n> > @@ -2470,11 +2470,12 @@ int repo_has_object_file(struct repository *r,\n> > * give more context.\n> > */\n> > static int hash_format_check_report(struct fsck_options *opts UNUSED,\n> > -\t\t\t\t const struct object_id *oid UNUSED,\n> > -\t\t\t\t enum object_type object_type UNUSED,\n> > -\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n> > -\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> > -\t\t\t\t const char *message)\n> > +\t\t\t\t const struct object_id *oid UNUSED,\n> > +\t\t\t\t enum object_type object_type UNUSED,\n> > +\t\t\t\t const char *ref_checked_name UNUSED,\n> > +\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n> > +\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> > +\t\t\t\t const char *message)\n> \n> That is somewhat annoying reindentation. What happened here?\n\nThe original code's indentation breaks. There are one more space in each\nline like the following:\n\n static int hash_format_check_report(struct fsck_options *opts UNUSED,\n const struct object_id *oid UNUSED)\n ...\n\nI think I could fix this by the way.\n\n\n\n"},{"id":"498509","messageId":"Zo_SmLOg8znMJm52@ArchLinux","threadId":"61565","inReplyTo":"xmqq7cdtue7c.fsf@gitster.g","subject":"Re: [GSoC][PATCH v10 06/10] builtin/refs: add verify subcommand and verbose_refs for \"fsck_options\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-11T12:39:52Z","receivedAt":"2024-07-11T12:39:53Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Wed, Jul 10, 2024 at 02:31:03PM -0700, Junio C Hamano wrote:\n> shejialuo writes:\n> \n> > Subject: Re: [GSoC][PATCH v10 06/10] builtin/refs: add verify subcommand and verbose_refs for \"fsck_options\"\n> \n> Just saying\n> \n> \tgit refs: add verify subcommand\n> \n> would be clearer. If you really want to talk about two modes, you\n> could say\n> \n> \tgit refs: add \"verify [--strict|--verbose]\" subcommand\n> \n> but that may be too much.\n> \n\nThanks, I will change in the next version.\n\n> > Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\n> > check the reference database consistency and also this subcommand will\n> > be used as the entry point of checking refs for \"git-fsck(1)\". Last, add\n> > \"verbose_refs\" field into \"fsck_options\" to indicate whether we should\n> > print verbose messages when checking refs consistency.\n> \n> Is there a reason why this has to be verbose_refs and not a simple\n> verbose bit? When people see how it is useful to ask for the\n> verbose output while checking refs, wouldn't people wish to add the\n> same \"--verbose\" support while checking objects, and at that point,\n> wouldn't it be awkward to add verbose_objs member to the struct and\n> having to flip both bits on?\n> \n\nActually, this is really what I thought. I just want to provide more\nfind-grained control here. However, when I implemented the code, I also\nfelt awkward about this. I can't find the balance here.\n\nI will improve this in the next version.\n\n> > Mentored-by: Patrick Steinhardt \n> > Mentored-by: Karthik Nayak \n> > Signed-off-by: shejialuo \n> > ---\n> > Documentation/git-refs.txt | 13 +++++++++++\n> > builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++\n> > fsck.h | 1 +\n> > 3 files changed, 58 insertions(+)\n> >\n> > diff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\n> > index 5b99e04385..1244a85b64 100644\n> > --- a/Documentation/git-refs.txt\n> > +++ b/Documentation/git-refs.txt\n> > @@ -10,6 +10,7 @@ SYNOPSIS\n> > --------\n> > [verse]\n> > 'git refs migrate' --ref-format= [--dry-run]\n> > +'git refs verify' [--strict] [--verbose]\n> > \n> > DESCRIPTION\n> > -----------\n> > @@ -22,6 +23,9 @@ COMMANDS\n> > migrate::\n> > \tMigrate ref store between different formats.\n> > \n> > +verify::\n> > +\tVerify reference database consistency.\n> > +\n> \n> The error reporting function for refs consistency check was still\n> about reporting a problem for a single ref. I am wondering how\n> consistency violations that are not about a single ref should be\n> handled. For example, if refs/packed-backend.c:packed_fsck() finds\n> that the file is not sorted properly or has some unparseable garbage\n> in it, it is not something you can report as \"refs/heads/main is\n> broken\", but those who are interested in seeing the \"reference\n> database consistency\" verified, it is very much what they want the\n> tool to notice. How would detection of such a breakage that is not\n> attributed to a single ref fit in this \"ref consistency check\n> infrastructure\" that was introduced by [05/10]?\n> \n\nYes, I didn't consider other cases. Although I have said in the subject\nthat this series is to set up the infrastructure of fscking refs. It's\na little hard for me to set up a perfect \"fsck_refs_report\" at the\nmoment.\n\nAs you said, I think currently I should consider about the packed-refs\nin this series. I will find a way to achieve this in the next version.\nWell, I could say I intentionally ignored this problem. But we should\nface the problem directly.\n\nReally thanks.\n\n> > +\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n> > +\tif (argc)\n> > +\t\tusage(_(\"too many arguments\"));\n> \n> I do not think we want to change this line in this topic, but\n> because I noticed that the issue is widespread, let me make a note\n> here that we may want to clean up all the commands that give this\n> message as a #leftoverbit item:\n> \n> $ git cmd foo baz\n> usage: too many arguments\n> \n> is very unfriendly in that it is not immediately obvious to users\n> which arguments are excess. Should they have given \"git cmd\"?\n> Is it \"git cmd foo\" that does not take any argument?\n> \n> If you said something like\n> \n> $ git refs verify baz\n> error: 'git refs verify' takes no arguments\n> \n> or even\n> \n> $ git refs verify baz\n> error: unknown argument 'baz' given to 'git refs verify'\n> \n> it would be much more helpful.\n> \n\nI will improve this in the next version.\n\n> \n> Thanks.\n"},{"id":"498663","messageId":"ZpPEdmUN1Z5tqbK3@ArchLinux","threadId":"61565","inReplyTo":"Zo6eJi8BePrQxTQV@ArchLinux","subject":"[GSoC][PATCH v11 00/10] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-14T12:28:38Z","receivedAt":"2024-07-14T12:28:35Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Hi All:\n\nThis version handles the following problems:\n\n1. Reorder the commit message to let \"fsck: rename objects-related fsck\nerror functions\" behind the \"fsck: add a unified interface for reporting\nfsck messages\".\n2. Rename \"vfsck_report\" to \"fsck_vreport\" to be align with the\ncodebase.\n3. Make the \"git-refs verify\" commit message clean, and add\nuser-friendly error message here.\n4. Rename \"verbose_refs\" to \"verbose\" field in \"fsck_options\".\n\nAnd the most important change in this version is to enhance the\n\"fsck_refs_report\" function. After some investigations, there are the\nfollowing situations where we should handle when checking ref\nconsistency.\n\n1. When checking loose refs and reflofs, we only need the checkee\ninformation, because they are standalone files.\n2. When checking packed-refs, we should check the packed-refs itself,\nfor example whether it is sorted or there are some garbage trailing\ncontents. However, we should also check each ref (sub_checkee) in the\nfile.\n3. When checking reftable refs, we need to check the binary file,\nhowever, I does not truly understand the principle of reftable refs. But\nwe could still use the idea like 2 case.\n\nBy the above statements, I change the \"fsck_refs_report\" shown as below:\n\nint fsck_refs_error_function(struct fsck_options *options UNUSED,\n\t\t\t const struct object_id *oid,\n\t\t\t enum object_type object_type UNUSED,\n\t\t\t const char *ref_checkee,\n\t\t\t const char *sub_ref_checkee,\n\t\t\t enum fsck_msg_type msg_type,\n\t\t\t enum fsck_msg_id msg_id UNUSED,\n\t\t\t const char *message)\n{\n\tstruct strbuf sb = STRBUF_INIT;\n\tint ret = 0;\n\n\tif (sub_ref_checkee)\n\t\tstrbuf_addf(&sb, \"%s.%s\", ref_checkee, sub_ref_checkee);\n\telse\n\t\tstrbuf_addstr(&sb, ref_checkee);\n\n\tif (oid)\n\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n\n\tif (msg_type == FSCK_WARN)\n\t\twarning(\"%s: %s\", sb.buf, message);\n\telse\n\t\tret = error(\"%s: %s\", sb.buf, message);\n\n\tstrbuf_release(&sb);\n\treturn ret;\n}\n\nIt could provide the following report messages:\n\n1. \"ref_checkee\": \"fsck error name\": \"user message\".\n2. \"ref_checkee.sub_ref_checkee\": \"fsck error name\": \"user message\".\n3. \"ref_checkee -> (oid hex)\": \"fsck error name\": \"user message\".\n4. \"ref_checkee.sub_ref_checkee -> (oid hex)\": \"fsck error name\": \"user\nmessage\".\n\nshejialuo (10):\n fsck: rename \"skiplist\" to \"skip_oids\"\n fsck: add a unified interface for reporting fsck messages\n fsck: rename objects-related fsck error functions\n fsck: add refs-related error report function\n refs: set up ref consistency check infrastructure\n git refs: add verify subcommand\n builtin/fsck: add `git-refs verify` child process\n files-backend: add unified interface for refs scanning\n fsck: add ref name check for files backend\n fsck: add ref content check for files backend\n\n Documentation/fsck-msgids.txt | 12 ++\n Documentation/git-refs.txt | 13 ++\n builtin/fsck.c | 33 ++++-\n builtin/mktag.c | 2 +\n builtin/refs.c | 44 ++++++\n fsck.c | 118 +++++++++++++---\n fsck.h | 69 ++++++---\n object-file.c | 12 +-\n refs.c | 7 +-\n refs.h | 8 ++\n refs/debug.c | 11 ++\n refs/files-backend.c | 255 +++++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 ++\n refs/refs-internal.h | 11 +-\n refs/reftable-backend.c | 8 ++\n t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++\n 16 files changed, 766 insertions(+), 56 deletions(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\nRange-diff against v10:\n 1: e044f933de = 1: a69705b777 fsck: rename \"skiplist\" to \"skip_oids\"\n 3: df4837e960 ! 2: 1ef1036348 fsck: add a unified interface for reporting fsck messages\n @@ Commit message\n error type and calling the callback \"error_func\" to report the message.\n However, \"report\" function is only related to object database which\n cannot be reused for refs. In order to provide a unified interface which\n - can report either objects or refs, create a new function \"vfsck_report\"\n - by adding \"checked_ref_name\" parameter following the \"report\" prototype.\n - Instead of using \"...\", provide \"va_list\" to allow more flexibility.\n + can report either objects or refs, create a new function \"fsck_vreport\"\n + following the \"report\" prototype. Instead of using \"...\", provide\n + \"va_list\" to allow more flexibility.\n \n - Like \"report\", the \"vfsck_report\" function will use \"error_func\"\n + When checking loose refs and reflogs, we only need to pass the checked\n + name to the fsck error report function. However, for packed-refs and\n + reftable refs, we need to check both the consistency of the file itself\n + and the refs or reflogs contained in the file. In order to provide above\n + checks, add two parameters \"ref_checkee\" and \"sub_ref_checkee\" in\n + \"fsck_vreport\" function.\n +\n + Like \"report\", the \"fsck_vreport\" function will use \"error_func\"\n registered in \"fsck_options\" to report customized messages. Change\n - \"error_func\" prototype to align with the new \"vfsck_report\".\n + \"error_func\" prototype to align with the new \"fsck_vreport\".\n \n - Then, change \"report\" function to use \"vfsck_report\" to report objects\n + Then, change \"report\" function to use \"fsck_vreport\" to report objects\n related messages. Add a new function called \"fsck_refs_report\" to use\n - \"vfsck_report\" to report refs related messages.\n + \"fsck_vreport\" to report refs related messages.\n \n Mentored-by: Patrick Steinhardt \n Mentored-by: Karthik Nayak \n @@ Commit message\n \n ## builtin/fsck.c ##\n @@ builtin/fsck.c: static int objerror(struct object *obj, const char *err)\n - static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n - \t\t\t\t const struct object_id *oid,\n - \t\t\t\t enum object_type object_type,\n -+\t\t\t\t const char *checked_ref_name UNUSED,\n - \t\t\t\t enum fsck_msg_type msg_type,\n - \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n - \t\t\t\t const char *message)\n + static int fsck_error_func(struct fsck_options *o UNUSED,\n + \t\t\t const struct object_id *oid,\n + \t\t\t enum object_type object_type,\n ++\t\t\t const char *ref_checkee UNUSED,\n ++\t\t\t const char *sub_ref_checkee UNUSED,\n + \t\t\t enum fsck_msg_type msg_type,\n + \t\t\t enum fsck_msg_id msg_id UNUSED,\n + \t\t\t const char *message)\n \n ## builtin/mktag.c ##\n @@ builtin/mktag.c: static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n \t\t\t\t const struct object_id *oid UNUSED,\n \t\t\t\t enum object_type object_type UNUSED,\n -+\t\t\t\t const char *checked_ref_name UNUSED,\n ++\t\t\t\t const char *ref_checkee UNUSED,\n ++\t\t\t\t const char *sub_ref_checkee UNUSED,\n \t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t\t const char *message)\n @@ fsck.c: static int object_on_skiplist(struct fsck_options *opts,\n + * It will get the current msg error type and call the error_func callback\n + * which is registered in the \"fsck_options\" struct.\n + */\n -+static int vfsck_report(struct fsck_options *options,\n ++static int fsck_vreport(struct fsck_options *options,\n +\t\t\tconst struct object_id *oid,\n +\t\t\tenum object_type object_type,\n -+\t\t\tconst char *checked_ref_name,\n ++\t\t\tconst char *ref_checkee,\n ++\t\t\tconst char *sub_ref_checkee,\n +\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n {\n -\tva_list ap;\n @@ fsck.c: static int report(struct fsck_options *options,\n \n -\tva_start(ap, fmt);\n -\tstrbuf_vaddf(&sb, fmt, ap);\n --\tresult = options->error_func(options, oid, object_type,\n +\tva_copy(ap_copy, ap);\n +\tstrbuf_vaddf(&sb, fmt, ap_copy);\n -+\tresult = options->error_func(options, oid, object_type, checked_ref_name,\n + \tresult = options->error_func(options, oid, object_type,\n ++\t\t\t\t ref_checkee, sub_ref_checkee,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n \tva_end(ap);\n @@ fsck.c: static int report(struct fsck_options *options,\n +{\n +\tva_list ap;\n +\tint result;\n ++\n +\tva_start(ap, fmt);\n -+\tresult = vfsck_report(options, oid, object_type, NULL,\n ++\tresult = fsck_vreport(options, oid, object_type, NULL, NULL,\n +\t\t\t msg_id, fmt, ap);\n +\tva_end(ap);\n ++\n +\treturn result;\n +}\n +\n -+\n -+\n +int fsck_refs_report(struct fsck_options *options,\n +\t\t const struct object_id *oid,\n -+\t\t const char *checked_ref_name,\n ++\t\t const char *ref_checkee,\n ++\t\t const char *sub_ref_checkee,\n +\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n +{\n +\tva_list ap;\n +\tint result;\n +\tva_start(ap, fmt);\n -+\tresult = vfsck_report(options, oid, OBJ_NONE,\n -+\t\t\t checked_ref_name, msg_id, fmt, ap);\n ++\tresult = fsck_vreport(options, oid, OBJ_NONE, ref_checkee, sub_ref_checkee,\n ++\t\t\t msg_id, fmt, ap);\n +\tva_end(ap);\n +\treturn result;\n +}\n @@ fsck.c: static int report(struct fsck_options *options,\n {\n \tif (!options->object_names)\n @@ fsck.c: int fsck_buffer(const struct object_id *oid, enum object_type type,\n - int fsck_objects_error_function(struct fsck_options *o,\n - \t\t\t\tconst struct object_id *oid,\n - \t\t\t\tenum object_type object_type UNUSED,\n -+\t\t\t\tconst char *checked_ref_name UNUSED,\n - \t\t\t\tenum fsck_msg_type msg_type,\n - \t\t\t\tenum fsck_msg_id msg_id UNUSED,\n - \t\t\t\tconst char *message)\n + int fsck_error_function(struct fsck_options *o,\n + \t\t\tconst struct object_id *oid,\n + \t\t\tenum object_type object_type UNUSED,\n ++\t\t\tconst char *ref_checkee UNUSED,\n ++\t\t\tconst char *sub_ref_checkee UNUSED,\n + \t\t\tenum fsck_msg_type msg_type,\n + \t\t\tenum fsck_msg_id msg_id UNUSED,\n + \t\t\tconst char *message)\n @@ fsck.c: int git_fsck_config(const char *var, const char *value,\n - int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n - \t\t\t\t\t\t const struct object_id *oid,\n - \t\t\t\t\t\t enum object_type object_type,\n -+\t\t\t\t\t\t const char *checked_ref_name,\n - \t\t\t\t\t\t enum fsck_msg_type msg_type,\n - \t\t\t\t\t\t enum fsck_msg_id msg_id,\n - \t\t\t\t\t\t const char *message)\n -@@ fsck.c: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n + int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n + \t\t\t\t\t const struct object_id *oid,\n + \t\t\t\t\t enum object_type object_type,\n ++\t\t\t\t\t const char *ref_checkee,\n ++\t\t\t\t\t const char *sub_ref_checkee,\n + \t\t\t\t\t enum fsck_msg_type msg_type,\n + \t\t\t\t\t enum fsck_msg_id msg_id,\n + \t\t\t\t\t const char *message)\n +@@ fsck.c: int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n --\treturn fsck_objects_error_function(o, oid, object_type,\n -+\treturn fsck_objects_error_function(o, oid, object_type, checked_ref_name,\n - \t\t\t\t\t msg_type, msg_id, message);\n +-\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n ++\treturn fsck_error_function(o, oid, object_type, ref_checkee,\n ++\t\t\t\t sub_ref_checkee, msg_type, msg_id, message);\n }\n \n ## fsck.h ##\n @@ fsck.h: int is_valid_msg_type(const char *msg_id, const char *msg_type);\n + */\n typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t const struct object_id *oid, enum object_type object_type,\n -+\t\t\t const char *checked_ref_name,\n ++\t\t\t const char *ref_checkee, const char *sub_ref_checkee,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n - int fsck_objects_error_function(struct fsck_options *o,\n - \t\t\t\tconst struct object_id *oid, enum object_type object_type,\n -+\t\t\t\tconst char *checked_ref_name,\n - \t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n - \t\t\t\tconst char *message);\n - int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n - \t\t\t\t\t\t const struct object_id *oid,\n - \t\t\t\t\t\t enum object_type object_type,\n -+\t\t\t\t\t\t const char *checked_ref_name,\n - \t\t\t\t\t\t enum fsck_msg_type msg_type,\n - \t\t\t\t\t\t enum fsck_msg_id msg_id,\n - \t\t\t\t\t\t const char *message);\n + int fsck_error_function(struct fsck_options *o,\n + \t\t\tconst struct object_id *oid, enum object_type object_type,\n ++\t\t\tconst char *ref_checkee, const char *sub_ref_checkee,\n + \t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n + \t\t\tconst char *message);\n + int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n + \t\t\t\t\t const struct object_id *oid,\n + \t\t\t\t\t enum object_type object_type,\n ++\t\t\t\t\t const char *ref_checkee,\n ++\t\t\t\t\t const char *sub_ref_checkee,\n + \t\t\t\t\t enum fsck_msg_type msg_type,\n + \t\t\t\t\t enum fsck_msg_id msg_id,\n + \t\t\t\t\t const char *message);\n @@ fsck.h: int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n @@ fsck.h: int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n +/*\n + * Report an error or warning for refs.\n + */\n -+__attribute__((format (printf, 5, 6)))\n ++__attribute__((format (printf, 6, 7)))\n +int fsck_refs_report(struct fsck_options *options,\n +\t\t const struct object_id *oid,\n -+\t\t const char *checked_ref_name,\n ++\t\t const char *ref_checkee,\n ++\t\t const char *sub_ref_checkee,\n +\t\t enum fsck_msg_id msg_id,\n +\t\t const char *fmt, ...);\n +\n @@ object-file.c: int repo_has_object_file(struct repository *r,\n -\t\t\t\t const char *message)\n +\t\t\t\t const struct object_id *oid UNUSED,\n +\t\t\t\t enum object_type object_type UNUSED,\n -+\t\t\t\t const char *ref_checked_name UNUSED,\n ++\t\t\t\t const char *ref_checkee UNUSED,\n ++\t\t\t\t const char *sub_ref_checkee UNUSED,\n +\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n +\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n +\t\t\t\t const char *message)\n 2: 73a7c53a23 ! 3: d17cf6166e fsck: rename objects-related fsck error functions\n @@ Commit message\n fsck: rename objects-related fsck error functions\n \n The names of objects-related fsck error functions are general. It's OK\n - when there is only object database check. However, we are going to\n - introduce refs database check. In order to avoid ambiguity, rename\n - objects-related fsck error functions to explicitly indicate these\n + when there is only object database check. However, we have introduced\n + refs database check report function. To avoid ambiguity, rename\n + object-related fsck error functions to explicitly indicate these\n functions are used to report objects-related messages.\n \n Mentored-by: Patrick Steinhardt \n @@ builtin/fsck.c: static int objerror(struct object *obj, const char *err)\n -static int fsck_error_func(struct fsck_options *o UNUSED,\n -\t\t\t const struct object_id *oid,\n -\t\t\t enum object_type object_type,\n +-\t\t\t const char *ref_checkee UNUSED,\n +-\t\t\t const char *sub_ref_checkee UNUSED,\n -\t\t\t enum fsck_msg_type msg_type,\n -\t\t\t enum fsck_msg_id msg_id UNUSED,\n -\t\t\t const char *message)\n +static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n +\t\t\t\t const struct object_id *oid,\n +\t\t\t\t enum object_type object_type,\n ++\t\t\t\t const char *ref_checkee UNUSED,\n ++\t\t\t\t const char *sub_ref_checkee UNUSED,\n +\t\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n +\t\t\t\t const char *message)\n @@ fsck.c: int fsck_buffer(const struct object_id *oid, enum object_type type,\n -int fsck_error_function(struct fsck_options *o,\n -\t\t\tconst struct object_id *oid,\n -\t\t\tenum object_type object_type UNUSED,\n +-\t\t\tconst char *ref_checkee UNUSED,\n +-\t\t\tconst char *sub_ref_checkee UNUSED,\n -\t\t\tenum fsck_msg_type msg_type,\n -\t\t\tenum fsck_msg_id msg_id UNUSED,\n -\t\t\tconst char *message)\n +int fsck_objects_error_function(struct fsck_options *o,\n +\t\t\t\tconst struct object_id *oid,\n +\t\t\t\tenum object_type object_type UNUSED,\n ++\t\t\t\tconst char *ref_checkee UNUSED,\n ++\t\t\t\tconst char *sub_ref_checkee UNUSED,\n +\t\t\t\tenum fsck_msg_type msg_type,\n +\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n +\t\t\t\tconst char *message)\n @@ fsck.c: int git_fsck_config(const char *var, const char *value,\n -int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n -\t\t\t\t\t const struct object_id *oid,\n -\t\t\t\t\t enum object_type object_type,\n +-\t\t\t\t\t const char *ref_checkee,\n +-\t\t\t\t\t const char *sub_ref_checkee,\n -\t\t\t\t\t enum fsck_msg_type msg_type,\n -\t\t\t\t\t enum fsck_msg_id msg_id,\n -\t\t\t\t\t const char *message)\n +int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n +\t\t\t\t\t\t const struct object_id *oid,\n +\t\t\t\t\t\t enum object_type object_type,\n ++\t\t\t\t\t\t const char *ref_checkee,\n ++\t\t\t\t\t\t const char *sub_ref_checkee,\n +\t\t\t\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t\t\t\t enum fsck_msg_id msg_id,\n +\t\t\t\t\t\t const char *message)\n @@ fsck.c: int git_fsck_config(const char *var, const char *value,\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n --\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n -+\treturn fsck_objects_error_function(o, oid, object_type,\n -+\t\t\t\t\t msg_type, msg_id, message);\n +-\treturn fsck_error_function(o, oid, object_type, ref_checkee,\n +-\t\t\t\t sub_ref_checkee, msg_type, msg_id, message);\n ++\treturn fsck_objects_error_function(o, oid, object_type, ref_checkee,\n ++\t\t\t\t\t sub_ref_checkee, msg_type, msg_id,\n ++\t\t\t\t\t message);\n }\n \n ## fsck.h ##\n @@ fsck.h: typedef int (*fsck_error)(struct fsck_options *o,\n \n -int fsck_error_function(struct fsck_options *o,\n -\t\t\tconst struct object_id *oid, enum object_type object_type,\n +-\t\t\tconst char *ref_checkee, const char *sub_ref_checkee,\n -\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n -\t\t\tconst char *message);\n -int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n -\t\t\t\t\t const struct object_id *oid,\n -\t\t\t\t\t enum object_type object_type,\n +-\t\t\t\t\t const char *ref_checkee,\n +-\t\t\t\t\t const char *sub_ref_checkee,\n -\t\t\t\t\t enum fsck_msg_type msg_type,\n -\t\t\t\t\t enum fsck_msg_id msg_id,\n -\t\t\t\t\t const char *message);\n +int fsck_objects_error_function(struct fsck_options *o,\n +\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n ++\t\t\t\tconst char *ref_checkee, const char *sub_ref_checkee,\n +\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n +\t\t\t\tconst char *message);\n +int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n +\t\t\t\t\t\t const struct object_id *oid,\n +\t\t\t\t\t\t enum object_type object_type,\n ++\t\t\t\t\t\t const char *ref_checkee,\n ++\t\t\t\t\t\t const char *sub_ref_checkee,\n +\t\t\t\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t\t\t\t enum fsck_msg_id msg_id,\n +\t\t\t\t\t\t const char *message);\n 4: c8eb01c987 ! 4: ee17c0835b fsck: add refs-related error report function\n @@ Metadata\n ## Commit message ##\n fsck: add refs-related error report function\n \n - Add refs-related options to the \"fsck_options\", create refs-specific\n - \"error_func\" callback \"fsck_refs_error_function\".\n + Create refs-specific \"error_func\" callback \"fsck_refs_error_function\"\n + which could provide the following report messages.\n \n - \"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\n + 1. \"ref_checkee\": \"fsck error name\": \"user message\".\n + 2. \"ref_checkee.sub_ref_checkee\": \"fsck error name\": \"user message\".\n + 3. \"ref_checkee -> (oid hex)\": \"fsck error name\": \"user message\".\n + 4. \"ref_checkee.sub_ref_checkee -> (pid hex)\": \"fsck error name\": \"user\n + message\".\n +\n + \"fsck_refs_error_function\" uses the \"ref_checkee\" and \"sub_ref_checkee\"\n + parameters to indicate the information of the checked refs. For loose\n + ref and reflog, it only uses the \"ref_checkee\" parameter. For packed\n + refs and reftable refs, when checking the consistency of the file\n + itself, it still only uses \"ref_checkee\" parameter. However, when\n + checking the consistency of the ref or reflog contained in the file, it\n + will use \"sub_ref_checkee\" parameter to indicate that we are not\n + checking the file but the incorporated ref or reflog.\n +\n + \"fsck_refs_error_function\" will use the \"oid\" parameter if the caller\n passes the oid, it will use \"oid_to_hex\" to get the corresponding hex\n value to report to the caller.\n \n @@ fsck.c: int fsck_objects_error_function(struct fsck_options *o,\n +int fsck_refs_error_function(struct fsck_options *options UNUSED,\n +\t\t\t const struct object_id *oid,\n +\t\t\t enum object_type object_type UNUSED,\n -+\t\t\t const char *checked_ref_name,\n ++\t\t\t const char *ref_checkee,\n ++\t\t\t const char *sub_ref_checkee,\n +\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t enum fsck_msg_id msg_id UNUSED,\n +\t\t\t const char *message)\n @@ fsck.c: int fsck_objects_error_function(struct fsck_options *o,\n +\tstruct strbuf sb = STRBUF_INIT;\n +\tint ret = 0;\n +\n -+\tstrbuf_addstr(&sb, checked_ref_name);\n ++\tif (sub_ref_checkee)\n ++\t\tstrbuf_addf(&sb, \"%s.%s\", ref_checkee, sub_ref_checkee);\n ++\telse\n ++\t\tstrbuf_addstr(&sb, ref_checkee);\n ++\n +\tif (oid)\n +\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n +\n @@ fsck.h: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *\n +int fsck_refs_error_function(struct fsck_options *options,\n +\t\t\t const struct object_id *oid,\n +\t\t\t enum object_type object_type,\n -+\t\t\t const char *checked_ref_name,\n ++\t\t\t const char *ref_checkee,\n ++\t\t\t const char *sub_ref_checkee,\n +\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t enum fsck_msg_id msg_id,\n +\t\t\t const char *message);\n 5: e4085df496 = 5: 4718ba7ddc refs: set up ref consistency check infrastructure\n 6: 497f224bed ! 6: 76163fb5d9 builtin/refs: add verify subcommand and verbose_refs for \"fsck_options\"\n @@ Metadata\n Author: shejialuo \n \n ## Commit message ##\n - builtin/refs: add verify subcommand and verbose_refs for \"fsck_options\"\n + git refs: add verify subcommand\n \n Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\n check the reference database consistency and also this subcommand will\n be used as the entry point of checking refs for \"git-fsck(1)\". Last, add\n - \"verbose_refs\" field into \"fsck_options\" to indicate whether we should\n - print verbose messages when checking refs consistency.\n + \"verbose\" field into \"fsck_options\" to indicate whether we should print\n + verbose messages when checking refs and objects consistency.\n \n Mentored-by: Patrick Steinhardt \n Mentored-by: Karthik Nayak \n @@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c\n +\n +\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n +\tif (argc)\n -+\t\tusage(_(\"too many arguments\"));\n ++\t\tusage(_(\"'git refs verify' takes no arguments\"));\n +\n +\tif (verbose)\n -+\t\tfsck_refs_options.verbose_refs = 1;\n ++\t\tfsck_refs_options.verbose = 1;\n +\tif (strict)\n +\t\tfsck_refs_options.strict = 1;\n +\n @@ fsck.h: struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n \tunsigned strict:1;\n -+\tunsigned verbose_refs:1;\n ++\tunsigned verbose:1;\n \tenum fsck_msg_type *msg_type;\n \tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n 7: 86a14c7b43 = 7: 27f766fb8e builtin/fsck: add `git-refs verify` child process\n 8: daedb80b47 ! 8: e2ab45ec9f files-backend: add unified interface for refs scanning\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +\t\t\tcontinue;\n +\t\t} else if (S_ISREG(iter->st.st_mode) ||\n +\t\t\t S_ISLNK(iter->st.st_mode)) {\n -+\t\t\tif (o->verbose_refs)\n ++\t\t\tif (o->verbose)\n +\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n +\t\t\t\t\t refs_check_dir, iter->relative_path);\n +\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +\t\tNULL\n +\t};\n +\n -+\tif (o->verbose_refs)\n ++\tif (o->verbose)\n +\t\tfprintf_ln(stderr, \"Checking references consistency\");\n +\n +\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n 9: c36d588e4f ! 9: cd438fb56d fsck: add ref name check for files backend\n @@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n +\n +\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n +\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n -+\t\tret = fsck_refs_report(o, NULL, sb.buf,\n ++\t\tret = fsck_refs_report(o, NULL, sb.buf, NULL,\n +\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n +\t\t\t\t \"invalid refname format\");\n +\t}\n10: 521e0d9ca3 ! 10: 15662c6934 fsck: add ref content check for files backend\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\n +\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n +\n -+\t\tret = fsck_refs_report(o, NULL, refname,\n ++\t\tret = fsck_refs_report(o, NULL, refname, NULL,\n +\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n +\t\t\t\t \"point to target out of refs hierarchy\");\n +\t\tgoto out;\n +\t}\n +\n +\tif (check_refname_format(pointee_name, 0)) {\n -+\t\tret = fsck_refs_report(o, NULL, refname,\n ++\t\tret = fsck_refs_report(o, NULL, refname, NULL,\n +\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n +\t\t\t\t \"point to invalid refname\");\n +\t}\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\t\tgoto out;\n +\n +\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n -+\t\tret = fsck_refs_report(o, NULL, refname,\n ++\t\tret = fsck_refs_report(o, NULL, refname, NULL,\n +\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n +\t\t\t\t \"point to invalid target\");\n +\t\tgoto out;\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\n +\t\tif (!skip_prefix(pointee_path.buf,\n +\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n -+\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n ++\t\t\tret = fsck_refs_report(o, NULL, refname.buf, NULL,\n +\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n +\t\t\t\t\t \"point to target outside gitdir\");\n +\t\t\tgoto clean;\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n +\t\t\t\t &referent, &type,\n +\t\t\t\t &failure_errno, &trailing)) {\n -+\t\tret = fsck_refs_report(o, NULL, refname.buf,\n ++\t\tret = fsck_refs_report(o, NULL, refname.buf, NULL,\n +\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n +\t\t\t\t \"invalid ref content\");\n +\t\tgoto clean;\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\t\t * be reported as a warning.\n +\t\t */\n +\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n -+\t\t\tret = fsck_refs_report(o, NULL, refname.buf,\n ++\t\t\tret = fsck_refs_report(o, NULL, refname.buf, NULL,\n +\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n +\t\t\t\t\t \"trailing garbage in ref\");\n +\t\t\tgoto clean;\n-- \n2.45.2\n\n"},{"id":"498664","messageId":"ZpPFA0jvHvBeolsC@ArchLinux","threadId":"61565","inReplyTo":"ZpPEdmUN1Z5tqbK3@ArchLinux","subject":"[GSoC][PATCH v11 01/10] fsck: rename \"skiplist\" to \"skip_oids\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-14T12:30:59Z","receivedAt":"2024-07-14T12:30:57Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The \"skiplist\" field in \"fsck_options\" is related to objects. Because we\nare going to introduce ref consistency check, the \"skiplist\" name is too\ngeneral which will make the caller think \"skiplist\" is related to both\nthe refs and objects.\n\nIt may seem that for both refs and objects, we should provide a general\n\"skiplist\" here. However, the type for \"skiplist\" is `struct oidset`\nwhich is totally unsuitable for refs.\n\nTo avoid above ambiguity, rename \"skiplist\" to \"skip_oids\".\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 4 ++--\n fsck.h | 4 ++--\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex eea7145470..3f32441492 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n \t\tif (!strcmp(buf, \"skiplist\")) {\n \t\t\tif (equal == len)\n \t\t\t\tdie(\"skiplist requires a path\");\n-\t\t\toidset_parse_file(&options->skiplist, buf + equal + 1,\n+\t\t\toidset_parse_file(&options->skip_oids, buf + equal + 1,\n \t\t\t\t\t the_repository->hash_algo);\n \t\t\tbuf += len + 1;\n \t\t\tcontinue;\n@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n static int object_on_skiplist(struct fsck_options *opts,\n \t\t\t const struct object_id *oid)\n {\n-\treturn opts && oid && oidset_contains(&opts->skiplist, oid);\n+\treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n __attribute__((format (printf, 5, 6)))\ndiff --git a/fsck.h b/fsck.h\nindex 6085a384f6..bcfb2e34cd 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -136,7 +136,7 @@ struct fsck_options {\n \tfsck_error error_func;\n \tunsigned strict:1;\n \tenum fsck_msg_type *msg_type;\n-\tstruct oidset skiplist;\n+\tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n \tstruct oidset gitattributes_found;\n@@ -145,7 +145,7 @@ struct fsck_options {\n };\n \n #define FSCK_OPTIONS_DEFAULT { \\\n-\t.skiplist = OIDSET_INIT, \\\n+\t.skip_oids = OIDSET_INIT, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n-- \n2.45.2\n\n"},{"id":"498665","messageId":"ZpPFElAYLVtRlq-e@ArchLinux","threadId":"61565","inReplyTo":"ZpPEdmUN1Z5tqbK3@ArchLinux","subject":"[GSoC][PATCH v11 02/10] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-14T12:31:14Z","receivedAt":"2024-07-14T12:31:11Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The static function \"report\" provided by \"fsck.c\" aims at checking fsck\nerror type and calling the callback \"error_func\" to report the message.\nHowever, \"report\" function is only related to object database which\ncannot be reused for refs. In order to provide a unified interface which\ncan report either objects or refs, create a new function \"fsck_vreport\"\nfollowing the \"report\" prototype. Instead of using \"...\", provide\n\"va_list\" to allow more flexibility.\n\nWhen checking loose refs and reflogs, we only need to pass the checked\nname to the fsck error report function. However, for packed-refs and\nreftable refs, we need to check both the consistency of the file itself\nand the refs or reflogs contained in the file. In order to provide above\nchecks, add two parameters \"ref_checkee\" and \"sub_ref_checkee\" in\n\"fsck_vreport\" function.\n\nLike \"report\", the \"fsck_vreport\" function will use \"error_func\"\nregistered in \"fsck_options\" to report customized messages. Change\n\"error_func\" prototype to align with the new \"fsck_vreport\".\n\nThen, change \"report\" function to use \"fsck_vreport\" to report objects\nrelated messages. Add a new function called \"fsck_refs_report\" to use\n\"fsck_vreport\" to report refs related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 2 ++\n builtin/mktag.c | 2 ++\n fsck.c | 60 ++++++++++++++++++++++++++++++++++++++++++-------\n fsck.h | 19 +++++++++++++++-\n object-file.c | 12 +++++-----\n 5 files changed, 81 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..8aeb8b17e2 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -92,6 +92,8 @@ static int objerror(struct object *obj, const char *err)\n static int fsck_error_func(struct fsck_options *o UNUSED,\n \t\t\t const struct object_id *oid,\n \t\t\t enum object_type object_type,\n+\t\t\t const char *ref_checkee UNUSED,\n+\t\t\t const char *sub_ref_checkee UNUSED,\n \t\t\t enum fsck_msg_type msg_type,\n \t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t const char *message)\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex 4767f1a97e..b5f9e108e5 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -20,6 +20,8 @@ static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n \t\t\t\t const struct object_id *oid UNUSED,\n \t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const char *ref_checkee UNUSED,\n+\t\t\t\t const char *sub_ref_checkee UNUSED,\n \t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t\t const char *message)\ndiff --git a/fsck.c b/fsck.c\nindex 3f32441492..7fceecdfae 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -226,12 +226,19 @@ static int object_on_skiplist(struct fsck_options *opts,\n \treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n-__attribute__((format (printf, 5, 6)))\n-static int report(struct fsck_options *options,\n-\t\t const struct object_id *oid, enum object_type object_type,\n-\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+/*\n+ * Provide a unified interface for either fscking refs or objects.\n+ * It will get the current msg error type and call the error_func callback\n+ * which is registered in the \"fsck_options\" struct.\n+ */\n+static int fsck_vreport(struct fsck_options *options,\n+\t\t\tconst struct object_id *oid,\n+\t\t\tenum object_type object_type,\n+\t\t\tconst char *ref_checkee,\n+\t\t\tconst char *sub_ref_checkee,\n+\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n {\n-\tva_list ap;\n+\tva_list ap_copy;\n \tstruct strbuf sb = STRBUF_INIT;\n \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n \tint result;\n@@ -250,9 +257,10 @@ static int report(struct fsck_options *options,\n \tprepare_msg_ids();\n \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n \n-\tva_start(ap, fmt);\n-\tstrbuf_vaddf(&sb, fmt, ap);\n+\tva_copy(ap_copy, ap);\n+\tstrbuf_vaddf(&sb, fmt, ap_copy);\n \tresult = options->error_func(options, oid, object_type,\n+\t\t\t\t ref_checkee, sub_ref_checkee,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n \tva_end(ap);\n@@ -260,6 +268,37 @@ static int report(struct fsck_options *options,\n \treturn result;\n }\n \n+__attribute__((format (printf, 5, 6)))\n+static int report(struct fsck_options *options,\n+\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\n+\tva_start(ap, fmt);\n+\tresult = fsck_vreport(options, oid, object_type, NULL, NULL,\n+\t\t\t msg_id, fmt, ap);\n+\tva_end(ap);\n+\n+\treturn result;\n+}\n+\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t const char *ref_checkee,\n+\t\t const char *sub_ref_checkee,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = fsck_vreport(options, oid, OBJ_NONE, ref_checkee, sub_ref_checkee,\n+\t\t\t msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n void fsck_enable_object_names(struct fsck_options *options)\n {\n \tif (!options->object_names)\n@@ -1203,6 +1242,8 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n int fsck_error_function(struct fsck_options *o,\n \t\t\tconst struct object_id *oid,\n \t\t\tenum object_type object_type UNUSED,\n+\t\t\tconst char *ref_checkee UNUSED,\n+\t\t\tconst char *sub_ref_checkee UNUSED,\n \t\t\tenum fsck_msg_type msg_type,\n \t\t\tenum fsck_msg_id msg_id UNUSED,\n \t\t\tconst char *message)\n@@ -1306,6 +1347,8 @@ int git_fsck_config(const char *var, const char *value,\n int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t const char *ref_checkee,\n+\t\t\t\t\t const char *sub_ref_checkee,\n \t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message)\n@@ -1314,5 +1357,6 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n+\treturn fsck_error_function(o, oid, object_type, ref_checkee,\n+\t\t\t\t sub_ref_checkee, msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex bcfb2e34cd..61ca38afd6 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -114,19 +114,25 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n \t\t\t void *data, struct fsck_options *options);\n \n-/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n+/*\n+ * callback function for reporting errors when checking either objects or refs\n+ */\n typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t\t const char *ref_checkee, const char *sub_ref_checkee,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n int fsck_error_function(struct fsck_options *o,\n \t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\tconst char *ref_checkee, const char *sub_ref_checkee,\n \t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\tconst char *message);\n int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t const char *ref_checkee,\n+\t\t\t\t\t const char *sub_ref_checkee,\n \t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message);\n@@ -209,6 +215,17 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+/*\n+ * Report an error or warning for refs.\n+ */\n+__attribute__((format (printf, 6, 7)))\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t const char *ref_checkee,\n+\t\t const char *sub_ref_checkee,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\ndiff --git a/object-file.c b/object-file.c\nindex 065103be3e..bc63b80c48 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2470,11 +2470,13 @@ int repo_has_object_file(struct repository *r,\n * give more context.\n */\n static int hash_format_check_report(struct fsck_options *opts UNUSED,\n-\t\t\t\t const struct object_id *oid UNUSED,\n-\t\t\t\t enum object_type object_type UNUSED,\n-\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n-\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t\t const char *message)\n+\t\t\t\t const struct object_id *oid UNUSED,\n+\t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const char *ref_checkee UNUSED,\n+\t\t\t\t const char *sub_ref_checkee UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \terror(_(\"object fails fsck: %s\"), message);\n \treturn 1;\n-- \n2.45.2\n\n"},{"id":"498666","messageId":"ZpPFHX_bpyAwfGri@ArchLinux","threadId":"61565","inReplyTo":"ZpPEdmUN1Z5tqbK3@ArchLinux","subject":"[GSoC][PATCH v11 03/10] fsck: rename objects-related fsck error functions","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-14T12:31:25Z","receivedAt":"2024-07-14T12:31:22Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The names of objects-related fsck error functions are general. It's OK\nwhen there is only object database check. However, we have introduced\nrefs database check report function. To avoid ambiguity, rename\nobject-related fsck error functions to explicitly indicate these\nfunctions are used to report objects-related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 18 +++++++++---------\n fsck.c | 37 +++++++++++++++++++------------------\n fsck.h | 32 ++++++++++++++++----------------\n 3 files changed, 44 insertions(+), 43 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 8aeb8b17e2..d22488c5d0 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -89,14 +89,14 @@ static int objerror(struct object *obj, const char *err)\n \treturn -1;\n }\n \n-static int fsck_error_func(struct fsck_options *o UNUSED,\n-\t\t\t const struct object_id *oid,\n-\t\t\t enum object_type object_type,\n-\t\t\t const char *ref_checkee UNUSED,\n-\t\t\t const char *sub_ref_checkee UNUSED,\n-\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t const char *message)\n+static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t enum object_type object_type,\n+\t\t\t\t const char *ref_checkee UNUSED,\n+\t\t\t\t const char *sub_ref_checkee UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \tswitch (msg_type) {\n \tcase FSCK_WARN:\n@@ -940,7 +940,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tfsck_walk_options.walk = mark_object;\n \tfsck_obj_options.walk = mark_used;\n-\tfsck_obj_options.error_func = fsck_error_func;\n+\tfsck_obj_options.error_func = fsck_objects_error_func;\n \tif (check_strict)\n \t\tfsck_obj_options.strict = 1;\n \ndiff --git a/fsck.c b/fsck.c\nindex 7fceecdfae..d66ea4ef44 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1239,14 +1239,14 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid,\n-\t\t\tenum object_type object_type UNUSED,\n-\t\t\tconst char *ref_checkee UNUSED,\n-\t\t\tconst char *sub_ref_checkee UNUSED,\n-\t\t\tenum fsck_msg_type msg_type,\n-\t\t\tenum fsck_msg_id msg_id UNUSED,\n-\t\t\tconst char *message)\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tenum object_type object_type UNUSED,\n+\t\t\t\tconst char *ref_checkee UNUSED,\n+\t\t\t\tconst char *sub_ref_checkee UNUSED,\n+\t\t\t\tenum fsck_msg_type msg_type,\n+\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n+\t\t\t\tconst char *message)\n {\n \tif (msg_type == FSCK_WARN) {\n \t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n@@ -1344,19 +1344,20 @@ int git_fsck_config(const char *var, const char *value,\n * Custom error callbacks that are used in more than one place.\n */\n \n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t const char *ref_checkee,\n-\t\t\t\t\t const char *sub_ref_checkee,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message)\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t const char *ref_checkee,\n+\t\t\t\t\t\t const char *sub_ref_checkee,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message)\n {\n \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_error_function(o, oid, object_type, ref_checkee,\n-\t\t\t\t sub_ref_checkee, msg_type, msg_id, message);\n+\treturn fsck_objects_error_function(o, oid, object_type, ref_checkee,\n+\t\t\t\t\t sub_ref_checkee, msg_type, msg_id,\n+\t\t\t\t\t message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex 61ca38afd6..0b40d9ec28 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -123,19 +123,19 @@ typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid, enum object_type object_type,\n-\t\t\tconst char *ref_checkee, const char *sub_ref_checkee,\n-\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n-\t\t\tconst char *message);\n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t const char *ref_checkee,\n-\t\t\t\t\t const char *sub_ref_checkee,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message);\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\t\tconst char *ref_checkee, const char *sub_ref_checkee,\n+\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n+\t\t\t\tconst char *message);\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t const char *ref_checkee,\n+\t\t\t\t\t\t const char *sub_ref_checkee,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message);\n \n struct fsck_options {\n \tfsck_walk_func walk;\n@@ -156,7 +156,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function \\\n+\t.error_func = fsck_objects_error_function \\\n }\n #define FSCK_OPTIONS_STRICT { \\\n \t.strict = 1, \\\n@@ -164,7 +164,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function, \\\n+\t.error_func = fsck_objects_error_function, \\\n }\n #define FSCK_OPTIONS_MISSING_GITMODULES { \\\n \t.strict = 1, \\\n@@ -172,7 +172,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n+\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n \n /* descend in all linked child objects\n-- \n2.45.2\n\n"},{"id":"498667","messageId":"ZpPFKPJ4cw1V9U7M@ArchLinux","threadId":"61565","inReplyTo":"ZpPEdmUN1Z5tqbK3@ArchLinux","subject":"[GSoC][PATCH v11 04/10] fsck: add refs-related error report function","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-14T12:31:36Z","receivedAt":"2024-07-14T12:31:32Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Create refs-specific \"error_func\" callback \"fsck_refs_error_function\"\nwhich could provide the following report messages.\n\n1. \"ref_checkee\": \"fsck error name\": \"user message\".\n2. \"ref_checkee.sub_ref_checkee\": \"fsck error name\": \"user message\".\n3. \"ref_checkee -> (oid hex)\": \"fsck error name\": \"user message\".\n4. \"ref_checkee.sub_ref_checkee -> (oid hex)\": \"fsck error name\": \"user\n message\".\n\n\"fsck_refs_error_function\" uses the \"ref_checkee\" and \"sub_ref_checkee\"\nparameters to indicate the information of the checked refs. For loose\nref and reflog, it only uses the \"ref_checkee\" parameter. For packed\nrefs and reftable refs, when checking the consistency of the file\nitself, it still only uses \"ref_checkee\" parameter. However, when\nchecking the consistency of the ref or reflog contained in the file, it\nwill use \"sub_ref_checkee\" parameter to indicate that we are not\nchecking the file but the incorporated ref or reflog.\n\n\"fsck_refs_error_function\" will use the \"oid\" parameter if the caller\npasses the oid, it will use \"oid_to_hex\" to get the corresponding hex\nvalue to report to the caller.\n\nLast, add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\"\nmacros to create refs options easily.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 29 +++++++++++++++++++++++++++++\n fsck.h | 15 +++++++++++++++\n 2 files changed, 44 insertions(+)\n\ndiff --git a/fsck.c b/fsck.c\nindex d66ea4ef44..4d18f20b09 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1256,6 +1256,35 @@ int fsck_objects_error_function(struct fsck_options *o,\n \treturn 1;\n }\n \n+int fsck_refs_error_function(struct fsck_options *options UNUSED,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type UNUSED,\n+\t\t\t const char *ref_checkee,\n+\t\t\t const char *sub_ref_checkee,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tif (sub_ref_checkee)\n+\t\tstrbuf_addf(&sb, \"%s.%s\", ref_checkee, sub_ref_checkee);\n+\telse\n+\t\tstrbuf_addstr(&sb, ref_checkee);\n+\n+\tif (oid)\n+\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n+\n+\tif (msg_type == FSCK_WARN)\n+\t\twarning(\"%s: %s\", sb.buf, message);\n+\telse\n+\t\tret = error(\"%s: %s\", sb.buf, message);\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n \t\t struct fsck_options *options, const char *blob_type)\ndiff --git a/fsck.h b/fsck.h\nindex 0b40d9ec28..7ae640ac6c 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -136,6 +136,14 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message);\n+int fsck_refs_error_function(struct fsck_options *options,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type,\n+\t\t\t const char *ref_checkee,\n+\t\t\t const char *sub_ref_checkee,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n \n struct fsck_options {\n \tfsck_walk_func walk;\n@@ -174,6 +182,13 @@ struct fsck_options {\n \t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n+#define FSCK_REFS_OPTIONS_STRICT { \\\n+\t.strict = 1, \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n \n /* descend in all linked child objects\n * the return value is:\n-- \n2.45.2\n\n"},{"id":"498668","messageId":"ZpPFM8nqzHUxYkCe@ArchLinux","threadId":"61565","inReplyTo":"ZpPEdmUN1Z5tqbK3@ArchLinux","subject":"[GSoC][PATCH v11 05/10] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-14T12:31:47Z","receivedAt":"2024-07-14T12:31:43Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It's organized as the five parts:\n\n1. The name and the initialization interfaces.\n2. The ref transaction interfaces.\n3. The ref internal interfaces (pack, rename and copy).\n4. The ref filesystem interfaces.\n5. The reflog related interfaces.\n\nTo keep consistent with the git-fsck(1), add a new interface named\n\"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\ngrouped into any above five categories. Explicitly add blank line to\nmake it different from others.\n\nLast, implement placeholder functions for each ref backends.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs.c | 5 +++++\n refs.h | 8 ++++++++\n refs/debug.c | 11 +++++++++++\n refs/files-backend.c | 15 ++++++++++++++-\n refs/packed-backend.c | 8 ++++++++\n refs/refs-internal.h | 6 ++++++\n refs/reftable-backend.c | 8 ++++++++\n 7 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex bb90a18875..410919246b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs, struct fsck_options *o)\n+{\n+\treturn refs->be->fsck(refs, o);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex 0ecba21b4a..804d6a7fce 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -4,6 +4,7 @@\n #include \"commit.h\"\n #include \"repository.h\"\n \n+struct fsck_options;\n struct object_id;\n struct ref_store;\n struct strbuf;\n@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_options *o);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..45e2e784a0 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tint res = drefs->refs->be->fsck(drefs->refs, o);\n+\ttrace_printf_key(&trace_refs, \"fsck: %d\\n\", res);\n+\treturn res;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex aa52d9be7c..d89eeda8ef 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3408,6 +3408,17 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\treturn ret;\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_init,\n@@ -3434,5 +3445,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a0666407cd..5209b0b212 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_init,\n@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex fa975d69aa..a905e187cd 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_options;\n struct ref_transaction;\n \n /*\n@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_options *o);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -677,6 +681,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex fbe74c239d..b5a1a526df 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.45.2\n\n"},{"id":"498669","messageId":"ZpPFPzSwB3-FmZx8@ArchLinux","threadId":"61565","inReplyTo":"ZpPEdmUN1Z5tqbK3@ArchLinux","subject":"[GSoC][PATCH v11 06/10] git refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-14T12:31:59Z","receivedAt":"2024-07-14T12:31:56Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\ncheck the reference database consistency and also this subcommand will\nbe used as the entry point of checking refs for \"git-fsck(1)\". Last, add\n\"verbose\" field into \"fsck_options\" to indicate whether we should print\nverbose messages when checking refs and objects consistency.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/git-refs.txt | 13 +++++++++++\n builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++\n fsck.h | 1 +\n 3 files changed, 58 insertions(+)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..1244a85b64 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+The following options are specific to 'git refs verify':\n+\n+--strict::\n+\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n+\tbe seen as ERROR. See linkgit:git-fsck[1].\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..4831c9e28e 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,54 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tunsigned int verbose = 0, strict = 0;\n+\tstruct option options[] = {\n+\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\tint ret;\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"'git refs verify' takes no arguments\"));\n+\n+\tif (verbose)\n+\t\tfsck_refs_options.verbose = 1;\n+\tif (strict)\n+\t\tfsck_refs_options.strict = 1;\n+\n+\tgit_config(git_fsck_config, &fsck_refs_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n+\n+\t/*\n+\t * Explicitly free the allocated array and \"skip_oids\" set\n+\t */\n+\tfree(fsck_refs_options.msg_type);\n+\toidset_clear(&fsck_refs_options.skip_oids);\n+\treturn ret;\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/fsck.h b/fsck.h\nindex 7ae640ac6c..6803696f68 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -149,6 +149,7 @@ struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n \tunsigned strict:1;\n+\tunsigned verbose:1;\n \tenum fsck_msg_type *msg_type;\n \tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n-- \n2.45.2\n\n"},{"id":"498670","messageId":"ZpPFSe1nIDRGtsH9@ArchLinux","threadId":"61565","inReplyTo":"ZpPEdmUN1Z5tqbK3@ArchLinux","subject":"[GSoC][PATCH v11 07/10] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-14T12:32:09Z","receivedAt":"2024-07-14T12:32:05Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new function \"fsck_refs\" that initializes and runs a child\nprocess to execute the \"git-refs verify\" command.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d22488c5d0..d3b466b84e 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -898,6 +898,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \treturn res;\n }\n \n+static void fsck_refs(void)\n+{\n+\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n+\tchild_process_init(&refs_verify);\n+\trefs_verify.git_cmd = 1;\n+\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n+\tif (verbose)\n+\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n+\tif (check_strict)\n+\t\tstrvec_push(&refs_verify.args, \"--strict\");\n+\n+\tif (run_command(&refs_verify))\n+\t\terrors_found |= ERROR_REFS;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n@@ -1067,6 +1082,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n+\tfsck_refs();\n+\n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \n-- \n2.45.2\n\n"},{"id":"498671","messageId":"ZpPFWB4XtLqoHIx9@ArchLinux","threadId":"61565","inReplyTo":"ZpPEdmUN1Z5tqbK3@ArchLinux","subject":"[GSoC][PATCH v11 08/10] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-14T12:32:24Z","receivedAt":"2024-07-14T12:32:21Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"For refs and reflogs, we need to scan its corresponding directories to\ncheck every regular file or symbolic link which shares the same pattern.\nIntroduce a unified interface for scanning directories for\nfiles-backend.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs/files-backend.c | 77 +++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 76 insertions(+), 1 deletion(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex d89eeda8ef..794e9f3f2e 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -6,6 +6,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -3408,6 +3409,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n+{\n+\tconst char *gitdir = ref_store->gitdir;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\n+\tif (!iter) {\n+\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n+\t\tgoto out;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->verbose)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n+\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n+\t\t\t\t iter->basename);\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n+\n+out:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tNULL\n+\t};\n+\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, \"Checking references consistency\");\n+\n+\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n+\n+\treturn ret;\n+}\n+\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_options *o)\n {\n@@ -3415,7 +3488,9 @@ static int files_fsck(struct ref_store *ref_store,\n \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n-\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o)\n+\t | files_fsck_refs(ref_store, o);\n+\n \treturn ret;\n }\n \n-- \n2.45.2\n\n"},{"id":"498672","messageId":"ZpPFYp-yIkH7hzWt@ArchLinux","threadId":"61565","inReplyTo":"ZpPEdmUN1Z5tqbK3@ArchLinux","subject":"[GSoC][PATCH v11 09/10] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-14T12:32:34Z","receivedAt":"2024-07-14T12:32:31Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) only implicitly checks the reference, it does not fully\ncheck refs with bad format name such as standalone \"@\" and name ending\nwith \".lock\".\n\nIn order to provide such checks, add a new fsck message id \"badRefName\"\nwith default ERROR type. Use existing \"check_refname_format\" to explicit\ncheck the ref name. And add a new unit test to verify the functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 +\n fsck.h | 1 +\n refs/files-backend.c | 20 +++++++\n t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++\n 4 files changed, 125 insertions(+)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex f643585a34..dab4012246 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,9 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefName`::\n+\t(ERROR) A ref has a bad name.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex 6803696f68..2a2441e147 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 794e9f3f2e..d20e149214 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3419,6 +3419,25 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n+static int files_fsck_refs_name(struct fsck_options *o,\n+\t\t\t\tconst char *gitdir UNUSED,\n+\t\t\t\tconst char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\tret = fsck_refs_report(o, NULL, sb.buf, NULL,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3470,6 +3489,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n {\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tfiles_fsck_refs_name,\n \t\tNULL\n \t};\n \ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..b2db58d2c6\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit tag multi_hierarchy/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1.lock: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/@ &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\tgit -c fsck.badRefName=warn fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\tgit -c fsck.badRefName=ignore fsck 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+test_done\n-- \n2.45.2\n\n"},{"id":"498673","messageId":"ZpPFbYOqDVWKz0ic@ArchLinux","threadId":"61565","inReplyTo":"ZpPEdmUN1Z5tqbK3@ArchLinux","subject":"[GSoC][PATCH v11 10/10] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-14T12:32:45Z","receivedAt":"2024-07-14T12:32:41Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Enhance the git-fsck(1) command by adding a check for reference content\nin the files backend. The new functionality ensures that symrefs, real\nsymbolic link and regular refs are validated correctly.\n\nIn order to check the trailing content of the regular refs, add a new\nparameter `trailing` to `parse_loose_ref_contents`.\n\nFor symrefs, `parse_loose_ref_contents` will set the \"referent\".\nHowever, symbolic link could be either absolute or relative. Use\n\"strbuf_add_real_path\" to read the symbolic link and convert the\nrelative path to absolute path. Then use \"skip_prefix\" to make it align\nwith symref \"referent\".\n\nThus, the symrefs and symbolic links could share the same interface. Add\na new function \"files_fsck_symref_target\" which aims at checking the\nfollowing things:\n\n1. whether the pointee is under the `refs/` directory.\n2. whether the pointee name is correct.\n3. whether the pointee path is a wrong type in filesystem.\n\nLast, add the following FSCK MESSAGEs:\n\n1. \"badRefContent(ERROR)\": A ref has a bad content\n2. \"badSymrefPointee(ERROR)\": The pointee of a symref is bad.\n3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 9 +++\n fsck.h | 3 +\n refs.c | 2 +-\n refs/files-backend.c | 145 +++++++++++++++++++++++++++++++++-\n refs/refs-internal.h | 5 +-\n t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++\n 6 files changed, 269 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex dab4012246..b1630a478b 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,9 +19,15 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefContent`::\n+\t(ERROR) A ref has a bad content.\n+\n `badRefName`::\n \t(ERROR) A ref has a bad name.\n \n+`badSymrefPointee`::\n+\t(ERROR) The pointee of a symref is bad.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \n@@ -167,6 +173,9 @@\n `nullSha1`::\n \t(WARN) Tree contains entries pointing to a null sha1.\n \n+`trailingRefContent`::\n+\t(WARN) A ref content has trailing contents.\n+\n `treeNotSorted`::\n \t(ERROR) A tree is not properly sorted.\n \ndiff --git a/fsck.h b/fsck.h\nindex 2a2441e147..e92a5844ae 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -32,6 +32,8 @@ enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n+\tFUNC(BAD_REF_CONTENT, ERROR) \\\n+\tFUNC(BAD_SYMREF_POINTEE, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n@@ -72,6 +74,7 @@ enum fsck_msg_type {\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n+\tFUNC(TRAILING_REF_CONTENT, WARN) \\\n \tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n \tFUNC(LARGE_PATHNAME, WARN) \\\ndiff --git a/refs.c b/refs.c\nindex 410919246b..eb82fb7d4e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1760,7 +1760,7 @@ static int refs_read_special_head(struct ref_store *ref_store,\n \t}\n \n \tresult = parse_loose_ref_contents(content.buf, oid, referent, type,\n-\t\t\t\t\t failure_errno);\n+\t\t\t\t\t failure_errno, NULL);\n \n done:\n \tstrbuf_release(&full_path);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex d20e149214..42d2f676b9 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1,6 +1,7 @@\n #define USE_THE_REPOSITORY_VARIABLE\n \n #include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n #include \"../copy.h\"\n #include \"../environment.h\"\n #include \"../gettext.h\"\n@@ -553,7 +554,7 @@ static int read_ref_internal(struct ref_store *ref_store, const char *refname,\n \tstrbuf_rtrim(&sb_contents);\n \tbuf = sb_contents.buf;\n \n-\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr);\n+\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr, NULL);\n \n out:\n \tif (ret && !myerr)\n@@ -589,7 +590,7 @@ static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refn\n \n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno)\n+\t\t\t int *failure_errno, const char **trailing)\n {\n \tconst char *p;\n \tif (skip_prefix(buf, \"ref:\", &buf)) {\n@@ -611,6 +612,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t*failure_errno = EINVAL;\n \t\treturn -1;\n \t}\n+\n+\tif (trailing)\n+\t\t*trailing = p;\n+\n \treturn 0;\n }\n \n@@ -3438,6 +3443,141 @@ static int files_fsck_refs_name(struct fsck_options *o,\n \treturn ret;\n }\n \n+/*\n+ * Check the symref \"pointee_name\" and \"pointee_path\". The caller should\n+ * make sure that \"pointee_path\" is absolute. For symbolic ref, \"pointee_name\"\n+ * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n+ * be the relative path agaignst \"gitdir\".\n+ */\n+static int files_fsck_symref_target(struct fsck_options *o,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const char *pointee_name,\n+\t\t\t\t const char *pointee_path)\n+{\n+\tconst char *p = NULL;\n+\tstruct stat st;\n+\tint ret = 0;\n+\n+\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n+\n+\t\tret = fsck_refs_report(o, NULL, refname, NULL,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to target out of refs hierarchy\");\n+\t\tgoto out;\n+\t}\n+\n+\tif (check_refname_format(pointee_name, 0)) {\n+\t\tret = fsck_refs_report(o, NULL, refname, NULL,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid refname\");\n+\t}\n+\n+\tif (lstat(pointee_path, &st) < 0)\n+\t\tgoto out;\n+\n+\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n+\t\tret = fsck_refs_report(o, NULL, refname, NULL,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid target\");\n+\t\tgoto out;\n+\t}\n+out:\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs_content(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter)\n+{\n+\tstruct strbuf pointee_path = STRBUF_INIT,\n+\t\t ref_content = STRBUF_INIT,\n+\t\t abs_gitdir = STRBUF_INIT,\n+\t\t referent = STRBUF_INIT,\n+\t\t refname = STRBUF_INIT;\n+\tconst char *trailing = NULL;\n+\tint failure_errno = 0;\n+\tunsigned int type = 0;\n+\tstruct object_id oid;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\n+\t/*\n+\t * If the file is a symlink, we need to only check the connectivity\n+\t * of the destination object.\n+\t */\n+\tif (S_ISLNK(iter->st.st_mode)) {\n+\t\tconst char *pointee_name = NULL;\n+\n+\t\tstrbuf_add_real_path(&pointee_path, iter->path.buf);\n+\n+\t\tstrbuf_add_absolute_path(&abs_gitdir, gitdir);\n+\t\tstrbuf_normalize_path(&abs_gitdir);\n+\t\tif (!is_dir_sep(abs_gitdir.buf[abs_gitdir.len - 1]))\n+\t\t\tstrbuf_addch(&abs_gitdir, '/');\n+\n+\t\tif (!skip_prefix(pointee_path.buf,\n+\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n+\t\t\tret = fsck_refs_report(o, NULL, refname.buf, NULL,\n+\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t\t \"point to target outside gitdir\");\n+\t\t\tgoto clean;\n+\t\t}\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, pointee_name,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n+\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n+\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n+\t\t\t\t &referent, &type,\n+\t\t\t\t &failure_errno, &trailing)) {\n+\t\tret = fsck_refs_report(o, NULL, refname.buf, NULL,\n+\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n+\t\t\t\t \"invalid ref content\");\n+\t\tgoto clean;\n+\t}\n+\n+\t/*\n+\t * If the ref is a symref, we need to check the destination name and\n+\t * connectivity.\n+\t */\n+\tif (referent.len && (type & REF_ISSYMREF)) {\n+\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n+\t\tstrbuf_rtrim(&referent);\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, referent.buf,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t} else {\n+\t\t/*\n+\t\t * Only regular refs could have a trailing garbage. Should\n+\t\t * be reported as a warning.\n+\t\t */\n+\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n+\t\t\tret = fsck_refs_report(o, NULL, refname.buf, NULL,\n+\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n+\t\t\t\t\t \"trailing garbage in ref\");\n+\t\t\tgoto clean;\n+\t\t}\n+\t}\n+\n+clean:\n+\tstrbuf_release(&abs_gitdir);\n+\tstrbuf_release(&pointee_path);\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&ref_content);\n+\tstrbuf_release(&referent);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3490,6 +3630,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n \t\tfiles_fsck_refs_name,\n+\t\tfiles_fsck_refs_content,\n \t\tNULL\n \t};\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex a905e187cd..2fabf41d14 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -709,11 +709,12 @@ struct ref_store {\n \n /*\n * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for\n- * invalid contents.\n+ * invalid contents. Also *trailing is set to the first character after the\n+ * refname or NULL if the referent is not empty.\n */\n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno);\n+\t\t\t int *failure_errno, const char **trailing);\n \n /*\n * Fill in the generic part of refs and add it to our collection of\ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nindex b2db58d2c6..35bf40ee64 100755\n--- a/t/t0602-reffiles-fsck.sh\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -98,4 +98,114 @@ test_expect_success 'ref name check should be adapted into fsck messages' '\n \t)\n '\n \n+test_expect_success 'regular ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit checkout -b a/b/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse branch-1)\" > $branch_dir_prefix/branch-1-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse tag-1)\" > $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_must_fail git -c fsck.trailingRefContent=error fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s \" \"$(git rev-parse tag-2)\" > $tag_dir_prefix/tag-2-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/tags/tag-2-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/a/b/branch-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'symbolic ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid refname\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid target\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: logs/maint-v2.45\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to target out of refs hierarchy\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.2\n\n"},{"id":"498916","messageId":"CAOLa=ZS07xF-MYLoS8O2ynE5DdPJPEEwaPsUR893fYUC+U1ZwQ@mail.gmail.com","threadId":"61565","inReplyTo":"ZpPFElAYLVtRlq-e@ArchLinux","subject":"Re: [GSoC][PATCH v11 02/10] fsck: add a unified interface for reporting fsck messages","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-07-18T13:26:30Z","receivedAt":"2024-07-18T13:26:31Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> The static function \"report\" provided by \"fsck.c\" aims at checking fsck\n> error type and calling the callback \"error_func\" to report the message.\n> However, \"report\" function is only related to object database which\n> cannot be reused for refs. In order to provide a unified interface which\n> can report either objects or refs, create a new function \"fsck_vreport\"\n> following the \"report\" prototype. Instead of using \"...\", provide\n> \"va_list\" to allow more flexibility.\n>\n> When checking loose refs and reflogs, we only need to pass the checked\n> name to the fsck error report function. However, for packed-refs and\n> reftable refs, we need to check both the consistency of the file itself\n> and the refs or reflogs contained in the file. In order to provide above\n> checks, add two parameters \"ref_checkee\" and \"sub_ref_checkee\" in\n> \"fsck_vreport\" function.\n\nNit: It would be nice, if you described here, what is the expected usage\nof \"ref_checkee\" and \"sub_ref_checkee\".\n\n[snip]\n\n> diff --git a/fsck.h b/fsck.h\n> index bcfb2e34cd..61ca38afd6 100644\n> --- a/fsck.h\n> +++ b/fsck.h\n> @@ -114,19 +114,25 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n> typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n> \t\t\t void *data, struct fsck_options *options);\n>\n> -/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n> +/*\n> + * callback function for reporting errors when checking either objects or refs\n> + */\n> typedef int (*fsck_error)(struct fsck_options *o,\n> \t\t\t const struct object_id *oid, enum object_type object_type,\n> +\t\t\t const char *ref_checkee, const char *sub_ref_checkee,\n\nThis makes me really wonder if this is the best way we can do this? This\nseems to solve for the current situation, but what happens if you want\nto also adding the reftable size or packed-refs size here? Would you\nintroduce another field?\n\nwould it be better to add a single `const struct *fsck_refs_info`\ninstead?\n\nPerhaps something like:\n\nstruct fsck_refs_info {\n char *refname;\n union {\n struct {\n ...\n } reftable;\n struct {\n ...\n } files;\n } u;\n}\n\nOf course we can fill in the details as we need them.\n\n> \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n> \t\t\t const char *message);\n>\n> int fsck_error_function(struct fsck_options *o,\n> \t\t\tconst struct object_id *oid, enum object_type object_type,\n> +\t\t\tconst char *ref_checkee, const char *sub_ref_checkee,\n> \t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n> \t\t\tconst char *message);\n> int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n> \t\t\t\t\t const struct object_id *oid,\n> \t\t\t\t\t enum object_type object_type,\n> +\t\t\t\t\t const char *ref_checkee,\n> +\t\t\t\t\t const char *sub_ref_checkee,\n> \t\t\t\t\t enum fsck_msg_type msg_type,\n> \t\t\t\t\t enum fsck_msg_id msg_id,\n> \t\t\t\t\t const char *message);\n> @@ -209,6 +215,17 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n> */\n> int fsck_finish(struct fsck_options *options);\n>\n> +/*\n> + * Report an error or warning for refs.\n> + */\n> +__attribute__((format (printf, 6, 7)))\n> +int fsck_refs_report(struct fsck_options *options,\n> +\t\t const struct object_id *oid,\n> +\t\t const char *ref_checkee,\n> +\t\t const char *sub_ref_checkee,\n> +\t\t enum fsck_msg_id msg_id,\n> +\t\t const char *fmt, ...);\n> +\n> /*\n> * Subsystem for storing human-readable names for each object.\n> *\n> diff --git a/object-file.c b/object-file.c\n> index 065103be3e..bc63b80c48 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -2470,11 +2470,13 @@ int repo_has_object_file(struct repository *r,\n> * give more context.\n> */\n> static int hash_format_check_report(struct fsck_options *opts UNUSED,\n> -\t\t\t\t const struct object_id *oid UNUSED,\n> -\t\t\t\t enum object_type object_type UNUSED,\n> -\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n> -\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> -\t\t\t\t const char *message)\n> +\t\t\t\t const struct object_id *oid UNUSED,\n> +\t\t\t\t enum object_type object_type UNUSED,\n> +\t\t\t\t const char *ref_checkee UNUSED,\n> +\t\t\t\t const char *sub_ref_checkee UNUSED,\n> +\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n> +\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n> +\t\t\t\t const char *message)\n> {\n> \terror(_(\"object fails fsck: %s\"), message);\n> \treturn 1;\n> --\n> 2.45.2\n"},{"id":"498922","messageId":"CAOLa=ZRYoYfY-P-dzpYbz0Yq44_TYGyJamf1hufJ0fxGZb3Z0Q@mail.gmail.com","threadId":"61565","inReplyTo":"ZpPFbYOqDVWKz0ic@ArchLinux","subject":"Re: [GSoC][PATCH v11 10/10] fsck: add ref content check for files backend","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-07-18T14:31:38Z","receivedAt":"2024-07-18T14:31:41Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> Enhance the git-fsck(1) command by adding a check for reference content\n> in the files backend. The new functionality ensures that symrefs, real\n> symbolic link and regular refs are validated correctly.\n>\n> In order to check the trailing content of the regular refs, add a new\n> parameter `trailing` to `parse_loose_ref_contents`.\n>\n> For symrefs, `parse_loose_ref_contents` will set the \"referent\".\n> However, symbolic link could be either absolute or relative. Use\n> \"strbuf_add_real_path\" to read the symbolic link and convert the\n> relative path to absolute path. Then use \"skip_prefix\" to make it align\n> with symref \"referent\".\n>\n> Thus, the symrefs and symbolic links could share the same interface. Add\n> a new function \"files_fsck_symref_target\" which aims at checking the\n> following things:\n>\n> 1. whether the pointee is under the `refs/` directory.\n> 2. whether the pointee name is correct.\n> 3. whether the pointee path is a wrong type in filesystem.\n>\n> Last, add the following FSCK MESSAGEs:\n>\n> 1. \"badRefContent(ERROR)\": A ref has a bad content\n> 2. \"badSymrefPointee(ERROR)\": The pointee of a symref is bad.\n> 3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n>\n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n> Documentation/fsck-msgids.txt | 9 +++\n> fsck.h | 3 +\n> refs.c | 2 +-\n> refs/files-backend.c | 145 +++++++++++++++++++++++++++++++++-\n> refs/refs-internal.h | 5 +-\n> t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++\n> 6 files changed, 269 insertions(+), 5 deletions(-)\n>\n> diff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\n> index dab4012246..b1630a478b 100644\n> --- a/Documentation/fsck-msgids.txt\n> +++ b/Documentation/fsck-msgids.txt\n> @@ -19,9 +19,15 @@\n> `badParentSha1`::\n> \t(ERROR) A commit object has a bad parent sha1.\n>\n> +`badRefContent`::\n> +\t(ERROR) A ref has a bad content.\n> +\n> `badRefName`::\n> \t(ERROR) A ref has a bad name.\n>\n> +`badSymrefPointee`::\n> +\t(ERROR) The pointee of a symref is bad.\n> +\n> `badTagName`::\n> \t(INFO) A tag has an invalid format.\n>\n> @@ -167,6 +173,9 @@\n> `nullSha1`::\n> \t(WARN) Tree contains entries pointing to a null sha1.\n>\n> +`trailingRefContent`::\n> +\t(WARN) A ref content has trailing contents.\n> +\n> `treeNotSorted`::\n> \t(ERROR) A tree is not properly sorted.\n>\n> diff --git a/fsck.h b/fsck.h\n> index 2a2441e147..e92a5844ae 100644\n> --- a/fsck.h\n> +++ b/fsck.h\n> @@ -32,6 +32,8 @@ enum fsck_msg_type {\n> \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n> \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n> \tFUNC(BAD_REF_NAME, ERROR) \\\n> +\tFUNC(BAD_REF_CONTENT, ERROR) \\\n> +\tFUNC(BAD_SYMREF_POINTEE, ERROR) \\\n> \tFUNC(BAD_TIMEZONE, ERROR) \\\n> \tFUNC(BAD_TREE, ERROR) \\\n> \tFUNC(BAD_TREE_SHA1, ERROR) \\\n> @@ -72,6 +74,7 @@ enum fsck_msg_type {\n> \tFUNC(HAS_DOTDOT, WARN) \\\n> \tFUNC(HAS_DOTGIT, WARN) \\\n> \tFUNC(NULL_SHA1, WARN) \\\n> +\tFUNC(TRAILING_REF_CONTENT, WARN) \\\n> \tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n> \tFUNC(NUL_IN_COMMIT, WARN) \\\n> \tFUNC(LARGE_PATHNAME, WARN) \\\n> diff --git a/refs.c b/refs.c\n> index 410919246b..eb82fb7d4e 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -1760,7 +1760,7 @@ static int refs_read_special_head(struct ref_store *ref_store,\n> \t}\n>\n> \tresult = parse_loose_ref_contents(content.buf, oid, referent, type,\n> -\t\t\t\t\t failure_errno);\n> +\t\t\t\t\t failure_errno, NULL);\n>\n> done:\n> \tstrbuf_release(&full_path);\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index d20e149214..42d2f676b9 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -1,6 +1,7 @@\n> #define USE_THE_REPOSITORY_VARIABLE\n>\n> #include \"../git-compat-util.h\"\n> +#include \"../abspath.h\"\n> #include \"../copy.h\"\n> #include \"../environment.h\"\n> #include \"../gettext.h\"\n> @@ -553,7 +554,7 @@ static int read_ref_internal(struct ref_store *ref_store, const char *refname,\n> \tstrbuf_rtrim(&sb_contents);\n> \tbuf = sb_contents.buf;\n>\n> -\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr);\n> +\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr, NULL);\n>\n> out:\n> \tif (ret && !myerr)\n> @@ -589,7 +590,7 @@ static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refn\n>\n> int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n> \t\t\t struct strbuf *referent, unsigned int *type,\n> -\t\t\t int *failure_errno)\n> +\t\t\t int *failure_errno, const char **trailing)\n> {\n> \tconst char *p;\n> \tif (skip_prefix(buf, \"ref:\", &buf)) {\n> @@ -611,6 +612,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n> \t\t*failure_errno = EINVAL;\n> \t\treturn -1;\n> \t}\n> +\n> +\tif (trailing)\n> +\t\t*trailing = p;\n> +\n> \treturn 0;\n> }\n>\n> @@ -3438,6 +3443,141 @@ static int files_fsck_refs_name(struct fsck_options *o,\n> \treturn ret;\n> }\n>\n> +/*\n> + * Check the symref \"pointee_name\" and \"pointee_path\". The caller should\n> + * make sure that \"pointee_path\" is absolute. For symbolic ref, \"pointee_name\"\n> + * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n> + * be the relative path agaignst \"gitdir\".\n> + */\n> +static int files_fsck_symref_target(struct fsck_options *o,\n> +\t\t\t\t const char *refname,\n> +\t\t\t\t const char *pointee_name,\n> +\t\t\t\t const char *pointee_path)\n> +{\n> +\tconst char *p = NULL;\n> +\tstruct stat st;\n> +\tint ret = 0;\n> +\n> +\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n> +\n> +\t\tret = fsck_refs_report(o, NULL, refname, NULL,\n> +\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n> +\t\t\t\t \"point to target out of refs hierarchy\");\n> +\t\tgoto out;\n> +\t}\n> +\n> +\tif (check_refname_format(pointee_name, 0)) {\n> +\t\tret = fsck_refs_report(o, NULL, refname, NULL,\n> +\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n> +\t\t\t\t \"point to invalid refname\");\n> +\t}\n> +\n> +\tif (lstat(pointee_path, &st) < 0)\n> +\t\tgoto out;\n> +\n> +\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n> +\t\tret = fsck_refs_report(o, NULL, refname, NULL,\n> +\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n> +\t\t\t\t \"point to invalid target\");\n> +\t\tgoto out;\n> +\t}\n> +out:\n> +\treturn ret;\n> +}\n> +\n> +static int files_fsck_refs_content(struct fsck_options *o,\n> +\t\t\t\t const char *gitdir,\n> +\t\t\t\t const char *refs_check_dir,\n> +\t\t\t\t struct dir_iterator *iter)\n> +{\n> +\tstruct strbuf pointee_path = STRBUF_INIT,\n> +\t\t ref_content = STRBUF_INIT,\n> +\t\t abs_gitdir = STRBUF_INIT,\n> +\t\t referent = STRBUF_INIT,\n> +\t\t refname = STRBUF_INIT;\n> +\tconst char *trailing = NULL;\n> +\tint failure_errno = 0;\n> +\tunsigned int type = 0;\n> +\tstruct object_id oid;\n> +\tint ret = 0;\n> +\n> +\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n> +\n> +\t/*\n> +\t * If the file is a symlink, we need to only check the connectivity\n> +\t * of the destination object.\n> +\t */\n> +\tif (S_ISLNK(iter->st.st_mode)) {\n> +\t\tconst char *pointee_name = NULL;\n> +\n> +\t\tstrbuf_add_real_path(&pointee_path, iter->path.buf);\n> +\n> +\t\tstrbuf_add_absolute_path(&abs_gitdir, gitdir);\n> +\t\tstrbuf_normalize_path(&abs_gitdir);\n> +\t\tif (!is_dir_sep(abs_gitdir.buf[abs_gitdir.len - 1]))\n> +\t\t\tstrbuf_addch(&abs_gitdir, '/');\n> +\n> +\t\tif (!skip_prefix(pointee_path.buf,\n> +\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n> +\t\t\tret = fsck_refs_report(o, NULL, refname.buf, NULL,\n> +\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n> +\t\t\t\t\t \"point to target outside gitdir\");\n> +\t\t\tgoto clean;\n> +\t\t}\n> +\n> +\t\tret = files_fsck_symref_target(o, refname.buf, pointee_name,\n> +\t\t\t\t\t pointee_path.buf);\n> +\t\tgoto clean;\n> +\t}\n> +\n> +\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n> +\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n> +\t\t\t\t refs_check_dir, iter->relative_path);\n> +\t\tgoto clean;\n> +\t}\n> +\n> +\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n> +\t\t\t\t &referent, &type,\n> +\t\t\t\t &failure_errno, &trailing)) {\n> +\t\tret = fsck_refs_report(o, NULL, refname.buf, NULL,\n> +\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n> +\t\t\t\t \"invalid ref content\");\n> +\t\tgoto clean;\n> +\t}\n> +\n> +\t/*\n> +\t * If the ref is a symref, we need to check the destination name and\n> +\t * connectivity.\n> +\t */\n> +\tif (referent.len && (type & REF_ISSYMREF)) {\n> +\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n> +\t\tstrbuf_rtrim(&referent);\n> +\n> +\t\tret = files_fsck_symref_target(o, refname.buf, referent.buf,\n> +\t\t\t\t\t pointee_path.buf);\n> +\t\tgoto clean;\n> +\t} else {\n> +\t\t/*\n> +\t\t * Only regular refs could have a trailing garbage. Should\n> +\t\t * be reported as a warning.\n> +\t\t */\n\nWhat happens if a symbolic reference has trailing garbage ?\n\n> +\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n> +\t\t\tret = fsck_refs_report(o, NULL, refname.buf, NULL,\n> +\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n> +\t\t\t\t\t \"trailing garbage in ref\");\n> +\t\t\tgoto clean;\n> +\t\t}\n> +\t}\n> +\n> +clean:\n> +\tstrbuf_release(&abs_gitdir);\n> +\tstrbuf_release(&pointee_path);\n> +\tstrbuf_release(&refname);\n> +\tstrbuf_release(&ref_content);\n> +\tstrbuf_release(&referent);\n> +\treturn ret;\n> +}\n> +\n> static int files_fsck_refs_dir(struct ref_store *ref_store,\n> \t\t\t struct fsck_options *o,\n> \t\t\t const char *refs_check_dir,\n> @@ -3490,6 +3630,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n> \tint ret;\n> \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n> \t\tfiles_fsck_refs_name,\n> +\t\tfiles_fsck_refs_content,\n> \t\tNULL\n> \t};\n>\n> diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> index a905e187cd..2fabf41d14 100644\n> --- a/refs/refs-internal.h\n> +++ b/refs/refs-internal.h\n> @@ -709,11 +709,12 @@ struct ref_store {\n>\n> /*\n> * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for\n> - * invalid contents.\n> + * invalid contents. Also *trailing is set to the first character after the\n> + * refname or NULL if the referent is not empty.\n> */\n> int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n> \t\t\t struct strbuf *referent, unsigned int *type,\n> -\t\t\t int *failure_errno);\n> +\t\t\t int *failure_errno, const char **trailing);\n>\n> /*\n> * Fill in the generic part of refs and add it to our collection of\n> diff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\n> index b2db58d2c6..35bf40ee64 100755\n> --- a/t/t0602-reffiles-fsck.sh\n> +++ b/t/t0602-reffiles-fsck.sh\n> @@ -98,4 +98,114 @@ test_expect_success 'ref name check should be adapted into fsck messages' '\n> \t)\n> '\n>\n> +test_expect_success 'regular ref content should be checked' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\tbranch_dir_prefix=.git/refs/heads &&\n> +\ttag_dir_prefix=.git/refs/tags &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit commit --allow-empty -m initial &&\n> +\t\tgit checkout -b branch-1 &&\n> +\t\tgit tag tag-1 &&\n> +\t\tgit commit --allow-empty -m second &&\n> +\t\tgit checkout -b branch-2 &&\n> +\t\tgit tag tag-2 &&\n> +\t\tgit checkout -b a/b/tag-2\n> +\t) &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tprintf \"%s garbage\" \"$(git rev-parse branch-1)\" > $branch_dir_prefix/branch-1-garbage &&\n> +\t\tgit fsck 2>err &&\n> +\t\tcat >expect <<-EOF &&\n> +\t\twarning: refs/heads/branch-1-garbage: trailingRefContent: trailing garbage in ref\n> +\t\tEOF\n> +\t\trm $branch_dir_prefix/branch-1-garbage &&\n> +\t\ttest_cmp expect err\n> +\t) &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tprintf \"%s garbage\" \"$(git rev-parse tag-1)\" > $tag_dir_prefix/tag-1-garbage &&\n> +\t\ttest_must_fail git -c fsck.trailingRefContent=error fsck 2>err &&\n> +\t\tcat >expect <<-EOF &&\n> +\t\terror: refs/tags/tag-1-garbage: trailingRefContent: trailing garbage in ref\n> +\t\tEOF\n> +\t\trm $tag_dir_prefix/tag-1-garbage &&\n> +\t\ttest_cmp expect err\n> +\t) &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tprintf \"%s \" \"$(git rev-parse tag-2)\" > $tag_dir_prefix/tag-2-garbage &&\n> +\t\tgit fsck 2>err &&\n> +\t\tcat >expect <<-EOF &&\n> +\t\twarning: refs/tags/tag-2-garbage: trailingRefContent: trailing garbage in ref\n> +\t\tEOF\n> +\t\trm $tag_dir_prefix/tag-2-garbage &&\n> +\t\ttest_cmp expect err\n> +\t) &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tprintf \"xfsazqfxcadas\" > $tag_dir_prefix/tag-2-bad &&\n> +\t\ttest_must_fail git refs verify 2>err &&\n> +\t\tcat >expect <<-EOF &&\n> +\t\terror: refs/tags/tag-2-bad: badRefContent: invalid ref content\n> +\t\tEOF\n> +\t\trm $tag_dir_prefix/tag-2-bad &&\n> +\t\ttest_cmp expect err\n> +\t) &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tprintf \"xfsazqfxcadas\" > $branch_dir_prefix/a/b/branch-2-bad &&\n> +\t\ttest_must_fail git refs verify 2>err &&\n> +\t\tcat >expect <<-EOF &&\n> +\t\terror: refs/heads/a/b/branch-2-bad: badRefContent: invalid ref content\n> +\t\tEOF\n> +\t\trm $branch_dir_prefix/a/b/branch-2-bad &&\n> +\t\ttest_cmp expect err\n> +\t)\n> +'\n> +\n> +test_expect_success 'symbolic ref content should be checked' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\tbranch_dir_prefix=.git/refs/heads &&\n> +\ttag_dir_prefix=.git/refs/tags &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit commit --allow-empty -m initial &&\n> +\t\tgit checkout -b branch-1 &&\n> +\t\tgit tag tag-1\n> +\t) &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n> +\t\ttest_must_fail git refs verify 2>err &&\n> +\t\tcat >expect <<-EOF &&\n> +\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid refname\n> +\t\tEOF\n> +\t\trm $branch_dir_prefix/branch-2-bad &&\n> +\t\ttest_cmp expect err\n> +\t) &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n> +\t\ttest_must_fail git refs verify 2>err &&\n> +\t\tcat >expect <<-EOF &&\n> +\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid target\n> +\t\tEOF\n> +\t\trm $branch_dir_prefix/branch-2-bad &&\n> +\t\ttest_cmp expect err\n> +\t) &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tprintf \"ref: logs/maint-v2.45\" > $branch_dir_prefix/branch-2-bad &&\n> +\t\ttest_must_fail git refs verify 2>err &&\n> +\t\tcat >expect <<-EOF &&\n> +\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to target out of refs hierarchy\n> +\t\tEOF\n> +\t\trm $branch_dir_prefix/branch-2-bad &&\n> +\t\ttest_cmp expect err\n> +\t)\n> +'\n> +\n> test_done\n> --\n> 2.45.2\n"},{"id":"498928","messageId":"xmqqsew6wuu9.fsf@gitster.g","threadId":"61565","inReplyTo":"CAOLa=ZRYoYfY-P-dzpYbz0Yq44_TYGyJamf1hufJ0fxGZb3Z0Q@mail.gmail.com","subject":"Re: [GSoC][PATCH v11 10/10] fsck: add ref content check for files backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-18T16:03:42Z","receivedAt":"2024-07-18T16:03:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> shejialuo writes:\n> ... 260+ lines of the original removed ...\n>> +\t\t */\n>\n> What happens if a symbolic reference has trailing garbage ?\n>\n> ... 160+ lines of the original removed ...\n\nIt is a pain to have to look for only a single line with a new piece\nof information in a 400+ line response, more than 99% of which are\nquoted original. Can you trim your quote a bit better?\n\nThanks.\n"},{"id":"498957","messageId":"CAOLa=ZSx70nT1RoLxwipz5tO=Kkw25QWn_wL4TJRprNSi0cvvg@mail.gmail.com","threadId":"61565","inReplyTo":"xmqqsew6wuu9.fsf@gitster.g","subject":"Re: [GSoC][PATCH v11 10/10] fsck: add ref content check for files backend","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-07-19T08:33:56Z","receivedAt":"2024-07-19T08:33:58Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> shejialuo writes:\n>> ... 260+ lines of the original removed ...\n>>> +\t\t */\n>>\n>> What happens if a symbolic reference has trailing garbage ?\n>>\n>> ... 160+ lines of the original removed ...\n>\n> It is a pain to have to look for only a single line with a new piece\n> of information in a 400+ line response, more than 99% of which are\n> quoted original. Can you trim your quote a bit better?\n>\n> Thanks.\n\nI usually do, I missed out here! Sorry about that.\n"},{"id":"498990","messageId":"CAH-kW5dAup6aJHbC9HGGg8288+eSMuMub52QXGZkOrHgGRuNOQ@mail.gmail.com","threadId":"61565","inReplyTo":"CAOLa=ZRYoYfY-P-dzpYbz0Yq44_TYGyJamf1hufJ0fxGZb3Z0Q@mail.gmail.com","subject":"Re: [GSoC][PATCH v11 10/10] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-20T07:16:20Z","receivedAt":"2024-07-20T07:16:19Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Karthik Nayak writes:\n>\n> shejialuo writes:\n> > + /*\n> > + * If the ref is a symref, we need to check the destination name and\n> > + * connectivity.\n> > + */\n> > + if (referent.len && (type & REF_ISSYMREF)) {\n> > + strbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n> > + strbuf_rtrim(&referent);\n> > +\n> > + ret = files_fsck_symref_target(o, refname.buf, referent.buf,\n> > + pointee_path.buf);\n> > + goto clean;\n> > + } else {\n> > + /*\n> > + * Only regular refs could have a trailing garbage. Should\n> > + * be reported as a warning.\n> > + */\n>\n> What happens if a symbolic reference has trailing garbage ?\n>\n\nThe \"parse_loose_ref_contents\" will return the referent. In this function,\nit will skip the prefix \"ref:\" to get the pointee. If there are some trailing\ngarbage, it will be reported by the \"files_fsck_symref_target\".\n\n\"files_fsck_symref_target\" will use \"check_refname_format\" function\nto check the pointee. For example, if the content is \"ref: refs/heads/\nmaster garbage\". The \"refs/heads/master garbage\" is a bad name.\n\nHowever, in my design, the trailing spaces or newline will be ignored,\nI thought we may not report this problem. And I use \"strbuf_rtrim\" here\nto ignore spaces and newlines.\n\nAnd I think there are some differences between symbolic refs and\nregular refs when parsing. For regular refs, git will ignore any trailing\ngarbage, however for symbolic refs, git will only ignore the newlines\nand spaces garbage. And git will not parse \"refs/heads/master gar\",\nit's an error here. But for regular refs, for example \"edaca... garbage\",\ngit will parse it normally without any warnings.\n\nSo question comes here, should we warn the user about the trailing\nnewlines or spaces. When using \"git symbolic-ref refs/heads/maint\nrefs/heads/main\", the \"refs/heads/maint\" will contain the newline\n'\\n' here and git also accepts content without newline '\\n'. And I\nthink we should not warn the user about one newline or no newline.\n\nIn my opinion, we should do this. It's not hard to do that. We only\nwarn the user for the following two situations:\n\n1. two or more newlines.\n2. one or more spaces.\n\nI will improve code in the next version.\n\nThanks,\nJialuo\n"},{"id":"498991","messageId":"ZptmNkRAgQAqgT03@ArchLinux","threadId":"61565","inReplyTo":"CAOLa=ZS07xF-MYLoS8O2ynE5DdPJPEEwaPsUR893fYUC+U1ZwQ@mail.gmail.com","subject":"Re: [GSoC][PATCH v11 02/10] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-20T07:24:38Z","receivedAt":"2024-07-20T07:24:28Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Thu, Jul 18, 2024 at 06:26:30AM -0700, Karthik Nayak wrote:\n> shejialuo writes:\n>\n> > diff --git a/fsck.h b/fsck.h\n> > index bcfb2e34cd..61ca38afd6 100644\n> > --- a/fsck.h\n> > +++ b/fsck.h\n> > @@ -114,19 +114,25 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n> > typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n> > \t\t\t void *data, struct fsck_options *options);\n> >\n> > -/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n> > +/*\n> > + * callback function for reporting errors when checking either objects or refs\n> > + */\n> > typedef int (*fsck_error)(struct fsck_options *o,\n> > \t\t\t const struct object_id *oid, enum object_type object_type,\n> > +\t\t\t const char *ref_checkee, const char *sub_ref_checkee,\n> \n> This makes me really wonder if this is the best way we can do this? This\n> seems to solve for the current situation, but what happens if you want\n> to also adding the reftable size or packed-refs size here? Would you\n> introduce another field?\n> \n> would it be better to add a single `const struct *fsck_refs_info`\n> instead?\n> \n> Perhaps something like:\n> \n> struct fsck_refs_info {\n> char *refname;\n> union {\n> struct {\n> ...\n> } reftable;\n> struct {\n> ...\n> } files;\n> } u;\n> }\n> \n> Of course we can fill in the details as we need them.\n> \n\nI agree, we should design an extensible data structure here. I will use\nthis idea. Because we don't know what we will do in the current time.\nHowever, I think \"refname\" is not good, instead I decide to use\n\"ref_checkee\", \"refname\" may let caller think we only check the refname.\nHowever, we need to also check reflog.\n"},{"id":"498992","messageId":"CAH-kW5cdTZJdFa-iZQCwFBZtUBAv_yH_87NbpNHb9yoGhJQkcg@mail.gmail.com","threadId":"61565","inReplyTo":"CAH-kW5dAup6aJHbC9HGGg8288+eSMuMub52QXGZkOrHgGRuNOQ@mail.gmail.com","subject":"Re: [GSoC][PATCH v11 10/10] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-20T08:43:37Z","receivedAt":"2024-07-20T08:43:37Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":">\n> The \"parse_loose_ref_contents\" will return the referent. In this function,\n> it will skip the prefix \"ref:\" to get the pointee. If there are some trailing\n> garbage, it will be reported by the \"files_fsck_symref_target\".\n>\n> \"files_fsck_symref_target\" will use \"check_refname_format\" function\n> to check the pointee. For example, if the content is \"ref: refs/heads/\n> master garbage\". The \"refs/heads/master garbage\" is a bad name.\n>\n> However, in my design, the trailing spaces or newline will be ignored,\n> I thought we may not report this problem. And I use \"strbuf_rtrim\" here\n> to ignore spaces and newlines.\n>\n> And I think there are some differences between symbolic refs and\n> regular refs when parsing. For regular refs, git will ignore any trailing\n> garbage, however for symbolic refs, git will only ignore the newlines\n> and spaces garbage. And git will not parse \"refs/heads/master gar\",\n> it's an error here. But for regular refs, for example \"edaca... garbage\",\n> git will parse it normally without any warnings.\n>\n> So question comes here, should we warn the user about the trailing\n> newlines or spaces. When using \"git symbolic-ref refs/heads/maint\n> refs/heads/main\", the \"refs/heads/maint\" will contain the newline\n> '\\n' here and git also accepts content without newline '\\n'. And I\n> think we should not warn the user about one newline or no newline.\n>\n> In my opinion, we should do this. It's not hard to do that. We only\n> warn the user for the following two situations:\n>\n> 1. two or more newlines.\n> 2. one or more spaces.\n>\n> I will improve code in the next version.\n>\n\nWell, I have changed my idea here. Sorry for that. If we report this as\nwarning, what is the difference between the following two cases:\n\n1. \"ref: refs/heads/master garbage \"\n2. \"ref: refs/heads/master garbage\"\n\nFor 1, if we do this, we will first report it has the extra spaces and then\nreport it's a bad refname. For 2, we will report it's a bad refname. I think\nwe should not report extra spaces or newlines here even with the error\ntype \"warning\".\n"},{"id":"498993","messageId":"ZpuCg1GL1YE_sJBP@ArchLinux","threadId":"61565","inReplyTo":"ZpPEdmUN1Z5tqbK3@ArchLinux","subject":"[GSoC][PATCH v12 00/10] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-20T09:25:23Z","receivedAt":"2024-07-20T09:25:15Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Hi All:\n\nThis version handles the following problem:\n\nIn order to provide more extensibility, I follow the Karthik's advice\nusing the following data structure:\n\n struct fsck_refs_info {\n char *ref_checkee;\n union {\n struct {\n char *sub_ref_checkee;\n } files;\n };\n };\n\nBecause we use a `struct` here, we could add more fields when we want to\nimplement morec checks without changing the existing functions. And I\nmove this structure into the \"fsck_options\". Thus there is no need for\nus to change the \"error_func\" prototype which makes this series more\nclearer.\n\nAnd Karthik asked me why I does not handle the trailing garbage for\nsymbolic refs. And I cited my understanding here:\n\n> The \"parse_loose_ref_contents\" will return the referent. In this function,\n> it will skip the prefix \"ref:\" to get the pointee. If there are some trailing\n> garbage, it will be reported by the \"files_fsck_symref_target\".\n\n> \"files_fsck_symref_target\" will use \"check_refname_format\" function\n> to check the pointee. For example, if the content is \"ref: refs/heads/\n> master garbage\". The \"refs/heads/master garbage\" is a bad name.\n\n> However, in my design, the trailing spaces or newline will be ignored,\n> I thought we may not report this problem. And I use \"strbuf_rtrim\" here\n> to ignore spaces and newlines.\n\nWe should not report this even with warning in my perspecitve.\n\nCI: https://github.com/shejialuo/git/pull/10\n\nshejialuo (10):\n fsck: rename \"skiplist\" to \"skip_oids\"\n fsck: add a unified interface for reporting fsck messages\n fsck: rename objects-related fsck error functions\n fsck: add refs-related error report function\n refs: set up ref consistency check infrastructure\n git refs: add verify subcommand\n builtin/fsck: add `git-refs verify` child process\n files-backend: add unified interface for refs scanning\n fsck: add ref name check for files backend\n fsck: add ref content check for files backend\n\n Documentation/fsck-msgids.txt | 12 ++\n Documentation/git-refs.txt | 13 ++\n builtin/fsck.c | 31 ++++-\n builtin/refs.c | 44 ++++++\n fsck.c | 106 +++++++++++---\n fsck.h | 74 +++++++---\n refs.c | 7 +-\n refs.h | 8 ++\n refs/debug.c | 11 ++\n refs/files-backend.c | 253 +++++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 ++\n refs/refs-internal.h | 11 +-\n refs/reftable-backend.c | 8 ++\n t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++\n 14 files changed, 746 insertions(+), 51 deletions(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\nRange-diff against v11:\n 1: a69705b777 = 1: a69705b777 fsck: rename \"skiplist\" to \"skip_oids\"\n 2: 1ef1036348 < -: ---------- fsck: add a unified interface for reporting fsck messages\n -: ---------- > 2: a4bfccd938 fsck: add a unified interface for reporting fsck messages\n 3: d17cf6166e ! 3: 9bc8892761 fsck: rename objects-related fsck error functions\n @@ builtin/fsck.c: static int objerror(struct object *obj, const char *err)\n -static int fsck_error_func(struct fsck_options *o UNUSED,\n -\t\t\t const struct object_id *oid,\n -\t\t\t enum object_type object_type,\n --\t\t\t const char *ref_checkee UNUSED,\n --\t\t\t const char *sub_ref_checkee UNUSED,\n -\t\t\t enum fsck_msg_type msg_type,\n -\t\t\t enum fsck_msg_id msg_id UNUSED,\n -\t\t\t const char *message)\n +static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n +\t\t\t\t const struct object_id *oid,\n +\t\t\t\t enum object_type object_type,\n -+\t\t\t\t const char *ref_checkee UNUSED,\n -+\t\t\t\t const char *sub_ref_checkee UNUSED,\n +\t\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n +\t\t\t\t const char *message)\n @@ fsck.c: int fsck_buffer(const struct object_id *oid, enum object_type type,\n -int fsck_error_function(struct fsck_options *o,\n -\t\t\tconst struct object_id *oid,\n -\t\t\tenum object_type object_type UNUSED,\n --\t\t\tconst char *ref_checkee UNUSED,\n --\t\t\tconst char *sub_ref_checkee UNUSED,\n -\t\t\tenum fsck_msg_type msg_type,\n -\t\t\tenum fsck_msg_id msg_id UNUSED,\n -\t\t\tconst char *message)\n +int fsck_objects_error_function(struct fsck_options *o,\n +\t\t\t\tconst struct object_id *oid,\n +\t\t\t\tenum object_type object_type UNUSED,\n -+\t\t\t\tconst char *ref_checkee UNUSED,\n -+\t\t\t\tconst char *sub_ref_checkee UNUSED,\n +\t\t\t\tenum fsck_msg_type msg_type,\n +\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n +\t\t\t\tconst char *message)\n @@ fsck.c: int git_fsck_config(const char *var, const char *value,\n -int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n -\t\t\t\t\t const struct object_id *oid,\n -\t\t\t\t\t enum object_type object_type,\n --\t\t\t\t\t const char *ref_checkee,\n --\t\t\t\t\t const char *sub_ref_checkee,\n -\t\t\t\t\t enum fsck_msg_type msg_type,\n -\t\t\t\t\t enum fsck_msg_id msg_id,\n -\t\t\t\t\t const char *message)\n +int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n +\t\t\t\t\t\t const struct object_id *oid,\n +\t\t\t\t\t\t enum object_type object_type,\n -+\t\t\t\t\t\t const char *ref_checkee,\n -+\t\t\t\t\t\t const char *sub_ref_checkee,\n +\t\t\t\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t\t\t\t enum fsck_msg_id msg_id,\n +\t\t\t\t\t\t const char *message)\n @@ fsck.c: int git_fsck_config(const char *var, const char *value,\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n --\treturn fsck_error_function(o, oid, object_type, ref_checkee,\n --\t\t\t\t sub_ref_checkee, msg_type, msg_id, message);\n -+\treturn fsck_objects_error_function(o, oid, object_type, ref_checkee,\n -+\t\t\t\t\t sub_ref_checkee, msg_type, msg_id,\n -+\t\t\t\t\t message);\n +-\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n ++\treturn fsck_objects_error_function(o, oid, object_type,\n ++\t\t\t\t\t msg_type, msg_id, message);\n }\n \n ## fsck.h ##\n @@ fsck.h: typedef int (*fsck_error)(struct fsck_options *o,\n \n -int fsck_error_function(struct fsck_options *o,\n -\t\t\tconst struct object_id *oid, enum object_type object_type,\n --\t\t\tconst char *ref_checkee, const char *sub_ref_checkee,\n -\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n -\t\t\tconst char *message);\n -int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n -\t\t\t\t\t const struct object_id *oid,\n -\t\t\t\t\t enum object_type object_type,\n --\t\t\t\t\t const char *ref_checkee,\n --\t\t\t\t\t const char *sub_ref_checkee,\n -\t\t\t\t\t enum fsck_msg_type msg_type,\n -\t\t\t\t\t enum fsck_msg_id msg_id,\n -\t\t\t\t\t const char *message);\n +int fsck_objects_error_function(struct fsck_options *o,\n +\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n -+\t\t\t\tconst char *ref_checkee, const char *sub_ref_checkee,\n +\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n +\t\t\t\tconst char *message);\n +int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n +\t\t\t\t\t\t const struct object_id *oid,\n +\t\t\t\t\t\t enum object_type object_type,\n -+\t\t\t\t\t\t const char *ref_checkee,\n -+\t\t\t\t\t\t const char *sub_ref_checkee,\n +\t\t\t\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t\t\t\t enum fsck_msg_id msg_id,\n +\t\t\t\t\t\t const char *message);\n \n - struct fsck_options {\n - \tfsck_walk_func walk;\n + /*\n + * The information for reporting refs-related error message\n @@ fsck.h: struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n 4: 19e049ee15 ! 4: 82296dc2b9 fsck: add refs-related error report function\n @@ Commit message\n fsck: add refs-related error report function\n \n Create refs-specific \"error_func\" callback \"fsck_refs_error_function\"\n - which could provide the following report messages.\n + which could provide the following report messages for files backend\n \n 1. \"ref_checkee\": \"fsck error name\": \"user message\".\n 2. \"ref_checkee.sub_ref_checkee\": \"fsck error name\": \"user message\".\n @@ Commit message\n message\".\n \n \"fsck_refs_error_function\" uses the \"ref_checkee\" and \"sub_ref_checkee\"\n - parameters to indicate the information of the checked refs. For loose\n - ref and reflog, it only uses the \"ref_checkee\" parameter. For packed\n + in the \"fsck_refs_info\" to indicate the information of the checked refs.\n + For loose ref and reflog, it only uses the \"ref_checkee\". For packed\n refs and reftable refs, when checking the consistency of the file\n - itself, it still only uses \"ref_checkee\" parameter. However, when\n - checking the consistency of the ref or reflog contained in the file, it\n - will use \"sub_ref_checkee\" parameter to indicate that we are not\n - checking the file but the incorporated ref or reflog.\n + itself, it still only uses \"ref_checkee\". However, when checking the\n + consistency of the ref or reflog contained in the file ,it will use the\n + \"sub_ref_checkee\" to indicate that we are not checking the file but the\n + incorporated ref or reflog.\n \n \"fsck_refs_error_function\" will use the \"oid\" parameter if the caller\n passes the oid, it will use \"oid_to_hex\" to get the corresponding hex\n @@ fsck.c: int fsck_objects_error_function(struct fsck_options *o,\n \treturn 1;\n }\n \n -+int fsck_refs_error_function(struct fsck_options *options UNUSED,\n ++int fsck_refs_error_function(struct fsck_options *options,\n +\t\t\t const struct object_id *oid,\n +\t\t\t enum object_type object_type UNUSED,\n -+\t\t\t const char *ref_checkee,\n -+\t\t\t const char *sub_ref_checkee,\n +\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t enum fsck_msg_id msg_id UNUSED,\n +\t\t\t const char *message)\n +{\n +\tstruct strbuf sb = STRBUF_INIT;\n ++\tstruct fsck_refs_info *refs_info = &options->refs_info;\n +\tint ret = 0;\n +\n -+\tif (sub_ref_checkee)\n -+\t\tstrbuf_addf(&sb, \"%s.%s\", ref_checkee, sub_ref_checkee);\n -+\telse\n -+\t\tstrbuf_addstr(&sb, ref_checkee);\n ++\tif (the_repository->ref_storage_format == REF_STORAGE_FORMAT_FILES) {\n ++\t\tstrbuf_addstr(&sb, refs_info->ref_checkee);\n ++\t\tif (refs_info->u.files.sub_ref_checkee)\n ++\t\t\tstrbuf_addf(&sb, \".%s\", refs_info->u.files.sub_ref_checkee);\n +\n -+\tif (oid)\n -+\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n ++\t\tif (oid)\n ++\t\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n ++\t}\n +\n +\tif (msg_type == FSCK_WARN)\n +\t\twarning(\"%s: %s\", sb.buf, message);\n @@ fsck.h: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *\n +int fsck_refs_error_function(struct fsck_options *options,\n +\t\t\t const struct object_id *oid,\n +\t\t\t enum object_type object_type,\n -+\t\t\t const char *ref_checkee,\n -+\t\t\t const char *sub_ref_checkee,\n +\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t enum fsck_msg_id msg_id,\n +\t\t\t const char *message);\n \n - struct fsck_options {\n - \tfsck_walk_func walk;\n + /*\n + * The information for reporting refs-related error message\n @@ fsck.h: struct fsck_options {\n \t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n 5: f175afc37c = 5: c5cac2e318 refs: set up ref consistency check infrastructure\n 6: e177157faa ! 6: 84d840506e git refs: add verify subcommand\n @@ fsck.h: struct fsck_options {\n \tunsigned strict:1;\n +\tunsigned verbose:1;\n \tenum fsck_msg_type *msg_type;\n + \tstruct fsck_refs_info refs_info;\n \tstruct oidset skip_oids;\n - \tstruct oidset gitmodules_found;\n 7: ee0e322f2b = 7: 3fc77ec329 builtin/fsck: add `git-refs verify` child process\n 8: 6a04fb0170 = 8: 44a75141fa files-backend: add unified interface for refs scanning\n 9: 7d11836deb ! 9: 4a0d58b07d fsck: add ref name check for files backend\n @@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n +\n +\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n +\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n -+\t\tret = fsck_refs_report(o, NULL, sb.buf, NULL,\n ++\t\to->refs_info.ref_checkee = sb.buf;\n ++\t\tret = fsck_refs_report(o, NULL,\n +\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n +\t\t\t\t \"invalid refname format\");\n +\t}\n10: ad696852ba ! 10: c529670e54 fsck: add ref content check for files backend\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\n +\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n +\n -+\t\tret = fsck_refs_report(o, NULL, refname, NULL,\n ++\t\tret = fsck_refs_report(o, NULL,\n +\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n +\t\t\t\t \"point to target out of refs hierarchy\");\n +\t\tgoto out;\n +\t}\n +\n +\tif (check_refname_format(pointee_name, 0)) {\n -+\t\tret = fsck_refs_report(o, NULL, refname, NULL,\n ++\t\tret = fsck_refs_report(o, NULL,\n +\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n +\t\t\t\t \"point to invalid refname\");\n +\t}\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\t\tgoto out;\n +\n +\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n -+\t\tret = fsck_refs_report(o, NULL, refname, NULL,\n ++\t\tret = fsck_refs_report(o, NULL,\n +\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n +\t\t\t\t \"point to invalid target\");\n +\t\tgoto out;\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\tint ret = 0;\n +\n +\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n ++\to->refs_info.ref_checkee = refname.buf;\n +\n +\t/*\n +\t * If the file is a symlink, we need to only check the connectivity\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\n +\t\tif (!skip_prefix(pointee_path.buf,\n +\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n -+\t\t\tret = fsck_refs_report(o, NULL, refname.buf, NULL,\n ++\t\t\tret = fsck_refs_report(o, NULL,\n +\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n +\t\t\t\t\t \"point to target outside gitdir\");\n +\t\t\tgoto clean;\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n +\t\t\t\t &referent, &type,\n +\t\t\t\t &failure_errno, &trailing)) {\n -+\t\tret = fsck_refs_report(o, NULL, refname.buf, NULL,\n ++\t\tret = fsck_refs_report(o, NULL,\n +\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n +\t\t\t\t \"invalid ref content\");\n +\t\tgoto clean;\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\t\t\t\t\t pointee_path.buf);\n +\t\tgoto clean;\n +\t} else {\n -+\t\t/*\n -+\t\t * Only regular refs could have a trailing garbage. Should\n -+\t\t * be reported as a warning.\n -+\t\t */\n +\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n -+\t\t\tret = fsck_refs_report(o, NULL, refname.buf, NULL,\n ++\t\t\tret = fsck_refs_report(o, NULL,\n +\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n +\t\t\t\t\t \"trailing garbage in ref\");\n +\t\t\tgoto clean;\n-- \n2.45.2\n\n"},{"id":"498994","messageId":"ZpuC_vtbiqWWhK_B@ArchLinux","threadId":"61565","inReplyTo":"ZpuCg1GL1YE_sJBP@ArchLinux","subject":"[GSoC][PATCH v12 01/10] fsck: rename \"skiplist\" to \"skip_oids\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-20T09:27:26Z","receivedAt":"2024-07-20T09:27:17Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The \"skiplist\" field in \"fsck_options\" is related to objects. Because we\nare going to introduce ref consistency check, the \"skiplist\" name is too\ngeneral which will make the caller think \"skiplist\" is related to both\nthe refs and objects.\n\nIt may seem that for both refs and objects, we should provide a general\n\"skiplist\" here. However, the type for \"skiplist\" is `struct oidset`\nwhich is totally unsuitable for refs.\n\nTo avoid above ambiguity, rename \"skiplist\" to \"skip_oids\".\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 4 ++--\n fsck.h | 4 ++--\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex eea7145470..3f32441492 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n \t\tif (!strcmp(buf, \"skiplist\")) {\n \t\t\tif (equal == len)\n \t\t\t\tdie(\"skiplist requires a path\");\n-\t\t\toidset_parse_file(&options->skiplist, buf + equal + 1,\n+\t\t\toidset_parse_file(&options->skip_oids, buf + equal + 1,\n \t\t\t\t\t the_repository->hash_algo);\n \t\t\tbuf += len + 1;\n \t\t\tcontinue;\n@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n static int object_on_skiplist(struct fsck_options *opts,\n \t\t\t const struct object_id *oid)\n {\n-\treturn opts && oid && oidset_contains(&opts->skiplist, oid);\n+\treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n __attribute__((format (printf, 5, 6)))\ndiff --git a/fsck.h b/fsck.h\nindex 6085a384f6..bcfb2e34cd 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -136,7 +136,7 @@ struct fsck_options {\n \tfsck_error error_func;\n \tunsigned strict:1;\n \tenum fsck_msg_type *msg_type;\n-\tstruct oidset skiplist;\n+\tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n \tstruct oidset gitattributes_found;\n@@ -145,7 +145,7 @@ struct fsck_options {\n };\n \n #define FSCK_OPTIONS_DEFAULT { \\\n-\t.skiplist = OIDSET_INIT, \\\n+\t.skip_oids = OIDSET_INIT, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n-- \n2.45.2\n\n"},{"id":"498995","messageId":"ZpuDCc8SZx8e4mLP@ArchLinux","threadId":"61565","inReplyTo":"ZpuCg1GL1YE_sJBP@ArchLinux","subject":"[GSoC][PATCH v12 02/10] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-20T09:27:37Z","receivedAt":"2024-07-20T09:27:28Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The static function \"report\" provided by \"fsck.c\" aims at checking fsck\nerror type and calling the callback \"error_func\" to report the message.\nHowever, \"report\" function is only related to object database which\ncannot be reused for refs. In order to provide a unified interface which\ncan report either objects or refs, create a new function \"fsck_vreport\"\nfollowing the \"report\" prototype. Instead of using \"...\", provide\n\"va_list\" to allow more flexibility.\n\nIn order to provide an extensible error report for refs, add a new\n\"fsck_refs_info\" structure to incorporate an union for supporting\ndifferent checks for files backend and reftable backend. Then\nincorporate this structure into the \"fsck_options\" to reuse the\n\"error_func\" callback.\n\nThen, change \"report\" function to use \"fsck_vreport\" to report objects\nrelated messages. Add a new function called \"fsck_refs_report\" to use\n\"fsck_vreport\" to report refs related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 46 +++++++++++++++++++++++++++++++++++++++-------\n fsck.h | 26 +++++++++++++++++++++++++-\n 2 files changed, 64 insertions(+), 8 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex 3f32441492..f2417e65fd 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -226,12 +226,17 @@ static int object_on_skiplist(struct fsck_options *opts,\n \treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n-__attribute__((format (printf, 5, 6)))\n-static int report(struct fsck_options *options,\n-\t\t const struct object_id *oid, enum object_type object_type,\n-\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+/*\n+ * Provide a unified interface for either fscking refs or objects.\n+ * It will get the current msg error type and call the error_func callback\n+ * which is registered in the \"fsck_options\" struct.\n+ */\n+static int fsck_vreport(struct fsck_options *options,\n+\t\t\tconst struct object_id *oid,\n+\t\t\tenum object_type object_type,\n+\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n {\n-\tva_list ap;\n+\tva_list ap_copy;\n \tstruct strbuf sb = STRBUF_INIT;\n \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n \tint result;\n@@ -250,8 +255,8 @@ static int report(struct fsck_options *options,\n \tprepare_msg_ids();\n \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n \n-\tva_start(ap, fmt);\n-\tstrbuf_vaddf(&sb, fmt, ap);\n+\tva_copy(ap_copy, ap);\n+\tstrbuf_vaddf(&sb, fmt, ap_copy);\n \tresult = options->error_func(options, oid, object_type,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n@@ -260,6 +265,33 @@ static int report(struct fsck_options *options,\n \treturn result;\n }\n \n+__attribute__((format (printf, 5, 6)))\n+static int report(struct fsck_options *options,\n+\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\n+\tva_start(ap, fmt);\n+\tresult = fsck_vreport(options, oid, object_type, msg_id, fmt, ap);\n+\tva_end(ap);\n+\n+\treturn result;\n+}\n+\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = fsck_vreport(options, oid, OBJ_NONE, msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n void fsck_enable_object_names(struct fsck_options *options)\n {\n \tif (!options->object_names)\ndiff --git a/fsck.h b/fsck.h\nindex bcfb2e34cd..a2ecb39b51 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -114,7 +114,9 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n \t\t\t void *data, struct fsck_options *options);\n \n-/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n+/*\n+ * callback function for reporting errors when checking either objects or refs\n+ */\n typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t const struct object_id *oid, enum object_type object_type,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n@@ -131,11 +133,24 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message);\n \n+/*\n+ * The information for reporting refs-related error message\n+ */\n+struct fsck_refs_info {\n+\tchar *ref_checkee;\n+\tunion {\n+\t\tstruct {\n+\t\t\tchar *sub_ref_checkee;\n+\t\t} files;\n+\t} u;\n+};\n+\n struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n \tunsigned strict:1;\n \tenum fsck_msg_type *msg_type;\n+\tstruct fsck_refs_info refs_info;\n \tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n@@ -209,6 +224,15 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+/*\n+ * Report an error or warning for refs.\n+ */\n+__attribute__((format (printf, 4, 5)))\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\n-- \n2.45.2\n\n"},{"id":"498996","messageId":"ZpuDFUL7D3GQV8EO@ArchLinux","threadId":"61565","inReplyTo":"ZpuCg1GL1YE_sJBP@ArchLinux","subject":"[GSoC][PATCH v12 03/10] fsck: rename objects-related fsck error functions","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-20T09:27:49Z","receivedAt":"2024-07-20T09:27:40Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The names of objects-related fsck error functions are general. It's OK\nwhen there is only object database check. However, we have introduced\nrefs database check report function. To avoid ambiguity, rename\nobject-related fsck error functions to explicitly indicate these\nfunctions are used to report objects-related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 14 +++++++-------\n fsck.c | 27 ++++++++++++++-------------\n fsck.h | 26 +++++++++++++-------------\n 3 files changed, 34 insertions(+), 33 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..6d86bbe1e9 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -89,12 +89,12 @@ static int objerror(struct object *obj, const char *err)\n \treturn -1;\n }\n \n-static int fsck_error_func(struct fsck_options *o UNUSED,\n-\t\t\t const struct object_id *oid,\n-\t\t\t enum object_type object_type,\n-\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t const char *message)\n+static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t enum object_type object_type,\n+\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \tswitch (msg_type) {\n \tcase FSCK_WARN:\n@@ -938,7 +938,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tfsck_walk_options.walk = mark_object;\n \tfsck_obj_options.walk = mark_used;\n-\tfsck_obj_options.error_func = fsck_error_func;\n+\tfsck_obj_options.error_func = fsck_objects_error_func;\n \tif (check_strict)\n \t\tfsck_obj_options.strict = 1;\n \ndiff --git a/fsck.c b/fsck.c\nindex f2417e65fd..9a7e3d8679 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1232,12 +1232,12 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid,\n-\t\t\tenum object_type object_type UNUSED,\n-\t\t\tenum fsck_msg_type msg_type,\n-\t\t\tenum fsck_msg_id msg_id UNUSED,\n-\t\t\tconst char *message)\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tenum object_type object_type UNUSED,\n+\t\t\t\tenum fsck_msg_type msg_type,\n+\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n+\t\t\t\tconst char *message)\n {\n \tif (msg_type == FSCK_WARN) {\n \t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n@@ -1335,16 +1335,17 @@ int git_fsck_config(const char *var, const char *value,\n * Custom error callbacks that are used in more than one place.\n */\n \n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message)\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message)\n {\n \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n+\treturn fsck_objects_error_function(o, oid, object_type,\n+\t\t\t\t\t msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex a2ecb39b51..6411437334 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -122,16 +122,16 @@ typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid, enum object_type object_type,\n-\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n-\t\t\tconst char *message);\n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message);\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n+\t\t\t\tconst char *message);\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message);\n \n /*\n * The information for reporting refs-related error message\n@@ -165,7 +165,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function \\\n+\t.error_func = fsck_objects_error_function \\\n }\n #define FSCK_OPTIONS_STRICT { \\\n \t.strict = 1, \\\n@@ -173,7 +173,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function, \\\n+\t.error_func = fsck_objects_error_function, \\\n }\n #define FSCK_OPTIONS_MISSING_GITMODULES { \\\n \t.strict = 1, \\\n@@ -181,7 +181,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n+\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n \n /* descend in all linked child objects\n-- \n2.45.2\n\n"},{"id":"498997","messageId":"ZpuDIKyxF5z8kbAK@ArchLinux","threadId":"61565","inReplyTo":"ZpuCg1GL1YE_sJBP@ArchLinux","subject":"[GSoC][PATCH v12 04/10] fsck: add refs-related error report function","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-20T09:28:00Z","receivedAt":"2024-07-20T09:27:50Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Create refs-specific \"error_func\" callback \"fsck_refs_error_function\"\nwhich could provide the following report messages for files backend\n\n1. \"ref_checkee\": \"fsck error name\": \"user message\".\n2. \"ref_checkee.sub_ref_checkee\": \"fsck error name\": \"user message\".\n3. \"ref_checkee -> (oid hex)\": \"fsck error name\": \"user message\".\n4. \"ref_checkee.sub_ref_checkee -> (oid hex)\": \"fsck error name\": \"user\n message\".\n\n\"fsck_refs_error_function\" uses the \"ref_checkee\" and \"sub_ref_checkee\"\nin the \"fsck_refs_info\" to indicate the information of the checked refs.\nFor loose ref and reflog, it only uses the \"ref_checkee\". For packed\nrefs and reftable refs, when checking the consistency of the file\nitself, it still only uses \"ref_checkee\". However, when checking the\nconsistency of the ref or reflog contained in the file ,it will use the\n\"sub_ref_checkee\" to indicate that we are not checking the file but the\nincorporated ref or reflog.\n\n\"fsck_refs_error_function\" will use the \"oid\" parameter if the caller\npasses the oid, it will use \"oid_to_hex\" to get the corresponding hex\nvalue to report to the caller.\n\nLast, add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\"\nmacros to create refs options easily.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 29 +++++++++++++++++++++++++++++\n fsck.h | 13 +++++++++++++\n 2 files changed, 42 insertions(+)\n\ndiff --git a/fsck.c b/fsck.c\nindex 9a7e3d8679..e87b13fdc3 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1247,6 +1247,35 @@ int fsck_objects_error_function(struct fsck_options *o,\n \treturn 1;\n }\n \n+int fsck_refs_error_function(struct fsck_options *options,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type UNUSED,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct fsck_refs_info *refs_info = &options->refs_info;\n+\tint ret = 0;\n+\n+\tif (the_repository->ref_storage_format == REF_STORAGE_FORMAT_FILES) {\n+\t\tstrbuf_addstr(&sb, refs_info->ref_checkee);\n+\t\tif (refs_info->u.files.sub_ref_checkee)\n+\t\t\tstrbuf_addf(&sb, \".%s\", refs_info->u.files.sub_ref_checkee);\n+\n+\t\tif (oid)\n+\t\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n+\t}\n+\n+\tif (msg_type == FSCK_WARN)\n+\t\twarning(\"%s: %s\", sb.buf, message);\n+\telse\n+\t\tret = error(\"%s: %s\", sb.buf, message);\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n \t\t struct fsck_options *options, const char *blob_type)\ndiff --git a/fsck.h b/fsck.h\nindex 6411437334..a3870ffe2b 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -132,6 +132,12 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message);\n+int fsck_refs_error_function(struct fsck_options *options,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n \n /*\n * The information for reporting refs-related error message\n@@ -183,6 +189,13 @@ struct fsck_options {\n \t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n+#define FSCK_REFS_OPTIONS_STRICT { \\\n+\t.strict = 1, \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n \n /* descend in all linked child objects\n * the return value is:\n-- \n2.45.2\n\n"},{"id":"498998","messageId":"ZpuDKnxcYQ7fAatI@ArchLinux","threadId":"61565","inReplyTo":"ZpuCg1GL1YE_sJBP@ArchLinux","subject":"[GSoC][PATCH v12 05/10] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-20T09:28:10Z","receivedAt":"2024-07-20T09:28:01Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It's organized as the five parts:\n\n1. The name and the initialization interfaces.\n2. The ref transaction interfaces.\n3. The ref internal interfaces (pack, rename and copy).\n4. The ref filesystem interfaces.\n5. The reflog related interfaces.\n\nTo keep consistent with the git-fsck(1), add a new interface named\n\"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\ngrouped into any above five categories. Explicitly add blank line to\nmake it different from others.\n\nLast, implement placeholder functions for each ref backends.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs.c | 5 +++++\n refs.h | 8 ++++++++\n refs/debug.c | 11 +++++++++++\n refs/files-backend.c | 15 ++++++++++++++-\n refs/packed-backend.c | 8 ++++++++\n refs/refs-internal.h | 6 ++++++\n refs/reftable-backend.c | 8 ++++++++\n 7 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex bb90a18875..410919246b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs, struct fsck_options *o)\n+{\n+\treturn refs->be->fsck(refs, o);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex 0ecba21b4a..804d6a7fce 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -4,6 +4,7 @@\n #include \"commit.h\"\n #include \"repository.h\"\n \n+struct fsck_options;\n struct object_id;\n struct ref_store;\n struct strbuf;\n@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_options *o);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..45e2e784a0 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tint res = drefs->refs->be->fsck(drefs->refs, o);\n+\ttrace_printf_key(&trace_refs, \"fsck: %d\\n\", res);\n+\treturn res;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex aa52d9be7c..d89eeda8ef 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3408,6 +3408,17 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\treturn ret;\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_init,\n@@ -3434,5 +3445,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a0666407cd..5209b0b212 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_init,\n@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex fa975d69aa..a905e187cd 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_options;\n struct ref_transaction;\n \n /*\n@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_options *o);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -677,6 +681,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex fbe74c239d..b5a1a526df 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.45.2\n\n"},{"id":"498999","messageId":"ZpuDQDbt9hazvn7D@ArchLinux","threadId":"61565","inReplyTo":"ZpuCg1GL1YE_sJBP@ArchLinux","subject":"[GSoC][PATCH v12 06/10] git refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-20T09:28:32Z","receivedAt":"2024-07-20T09:28:23Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\ncheck the reference database consistency and also this subcommand will\nbe used as the entry point of checking refs for \"git-fsck(1)\". Last, add\n\"verbose\" field into \"fsck_options\" to indicate whether we should print\nverbose messages when checking refs and objects consistency.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/git-refs.txt | 13 +++++++++++\n builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++\n fsck.h | 1 +\n 3 files changed, 58 insertions(+)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..1244a85b64 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+The following options are specific to 'git refs verify':\n+\n+--strict::\n+\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n+\tbe seen as ERROR. See linkgit:git-fsck[1].\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..4831c9e28e 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,54 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tunsigned int verbose = 0, strict = 0;\n+\tstruct option options[] = {\n+\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\tint ret;\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"'git refs verify' takes no arguments\"));\n+\n+\tif (verbose)\n+\t\tfsck_refs_options.verbose = 1;\n+\tif (strict)\n+\t\tfsck_refs_options.strict = 1;\n+\n+\tgit_config(git_fsck_config, &fsck_refs_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n+\n+\t/*\n+\t * Explicitly free the allocated array and \"skip_oids\" set\n+\t */\n+\tfree(fsck_refs_options.msg_type);\n+\toidset_clear(&fsck_refs_options.skip_oids);\n+\treturn ret;\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/fsck.h b/fsck.h\nindex a3870ffe2b..98e2225593 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -155,6 +155,7 @@ struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n \tunsigned strict:1;\n+\tunsigned verbose:1;\n \tenum fsck_msg_type *msg_type;\n \tstruct fsck_refs_info refs_info;\n \tstruct oidset skip_oids;\n-- \n2.45.2\n\n"},{"id":"499000","messageId":"ZpuDS6AjYj2gelWm@ArchLinux","threadId":"61565","inReplyTo":"ZpuCg1GL1YE_sJBP@ArchLinux","subject":"[GSoC][PATCH v12 07/10] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-20T09:28:43Z","receivedAt":"2024-07-20T09:28:33Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new function \"fsck_refs\" that initializes and runs a child\nprocess to execute the \"git-refs verify\" command.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 6d86bbe1e9..6cdff55fd6 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -896,6 +896,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \treturn res;\n }\n \n+static void fsck_refs(void)\n+{\n+\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n+\tchild_process_init(&refs_verify);\n+\trefs_verify.git_cmd = 1;\n+\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n+\tif (verbose)\n+\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n+\tif (check_strict)\n+\t\tstrvec_push(&refs_verify.args, \"--strict\");\n+\n+\tif (run_command(&refs_verify))\n+\t\terrors_found |= ERROR_REFS;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n@@ -1065,6 +1080,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n+\tfsck_refs();\n+\n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \n-- \n2.45.2\n\n"},{"id":"499001","messageId":"ZpuDVc_DAM0DRMFQ@ArchLinux","threadId":"61565","inReplyTo":"ZpuCg1GL1YE_sJBP@ArchLinux","subject":"[GSoC][PATCH v12 08/10] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-20T09:28:53Z","receivedAt":"2024-07-20T09:28:44Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"For refs and reflogs, we need to scan its corresponding directories to\ncheck every regular file or symbolic link which shares the same pattern.\nIntroduce a unified interface for scanning directories for\nfiles-backend.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs/files-backend.c | 77 +++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 76 insertions(+), 1 deletion(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex d89eeda8ef..794e9f3f2e 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -6,6 +6,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -3408,6 +3409,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n+{\n+\tconst char *gitdir = ref_store->gitdir;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\n+\tif (!iter) {\n+\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n+\t\tgoto out;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->verbose)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n+\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n+\t\t\t\t iter->basename);\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n+\n+out:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\tint ret;\n+\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tNULL\n+\t};\n+\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, \"Checking references consistency\");\n+\n+\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n+\n+\treturn ret;\n+}\n+\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_options *o)\n {\n@@ -3415,7 +3488,9 @@ static int files_fsck(struct ref_store *ref_store,\n \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n-\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o)\n+\t | files_fsck_refs(ref_store, o);\n+\n \treturn ret;\n }\n \n-- \n2.45.2\n\n"},{"id":"499002","messageId":"ZpuDXoNxXV8U5p4S@ArchLinux","threadId":"61565","inReplyTo":"ZpuCg1GL1YE_sJBP@ArchLinux","subject":"[GSoC][PATCH v12 09/10] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-20T09:29:02Z","receivedAt":"2024-07-20T09:28:52Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) only implicitly checks the reference, it does not fully\ncheck refs with bad format name such as standalone \"@\" and name ending\nwith \".lock\".\n\nIn order to provide such checks, add a new fsck message id \"badRefName\"\nwith default ERROR type. Use existing \"check_refname_format\" to explicit\ncheck the ref name. And add a new unit test to verify the functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 +\n fsck.h | 1 +\n refs/files-backend.c | 21 +++++++\n t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++\n 4 files changed, 126 insertions(+)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex f643585a34..dab4012246 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,9 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefName`::\n+\t(ERROR) A ref has a bad name.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex 98e2225593..421ec62b8f 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 794e9f3f2e..f93c44c4d1 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3419,6 +3419,26 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n+static int files_fsck_refs_name(struct fsck_options *o,\n+\t\t\t\tconst char *gitdir UNUSED,\n+\t\t\t\tconst char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\to->refs_info.ref_checkee = sb.buf;\n+\t\tret = fsck_refs_report(o, NULL,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3470,6 +3490,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n {\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tfiles_fsck_refs_name,\n \t\tNULL\n \t};\n \ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..b2db58d2c6\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit tag multi_hierarchy/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1.lock: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/@ &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\tgit -c fsck.badRefName=warn fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\tgit -c fsck.badRefName=ignore fsck 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+test_done\n-- \n2.45.2\n\n"},{"id":"499003","messageId":"ZpuDaE-wevt_mEj7@ArchLinux","threadId":"61565","inReplyTo":"ZpuCg1GL1YE_sJBP@ArchLinux","subject":"[GSoC][PATCH v12 10/10] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-20T09:29:12Z","receivedAt":"2024-07-20T09:29:02Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Enhance the git-fsck(1) command by adding a check for reference content\nin the files backend. The new functionality ensures that symrefs, real\nsymbolic link and regular refs are validated correctly.\n\nIn order to check the trailing content of the regular refs, add a new\nparameter `trailing` to `parse_loose_ref_contents`.\n\nFor symrefs, `parse_loose_ref_contents` will set the \"referent\".\nHowever, symbolic link could be either absolute or relative. Use\n\"strbuf_add_real_path\" to read the symbolic link and convert the\nrelative path to absolute path. Then use \"skip_prefix\" to make it align\nwith symref \"referent\".\n\nThus, the symrefs and symbolic links could share the same interface. Add\na new function \"files_fsck_symref_target\" which aims at checking the\nfollowing things:\n\n1. whether the pointee is under the `refs/` directory.\n2. whether the pointee name is correct.\n3. whether the pointee path is a wrong type in filesystem.\n\nLast, add the following FSCK MESSAGEs:\n\n1. \"badRefContent(ERROR)\": A ref has a bad content\n2. \"badSymrefPointee(ERROR)\": The pointee of a symref is bad.\n3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 9 +++\n fsck.h | 3 +\n refs.c | 2 +-\n refs/files-backend.c | 142 +++++++++++++++++++++++++++++++++-\n refs/refs-internal.h | 5 +-\n t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++\n 6 files changed, 266 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex dab4012246..b1630a478b 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,9 +19,15 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefContent`::\n+\t(ERROR) A ref has a bad content.\n+\n `badRefName`::\n \t(ERROR) A ref has a bad name.\n \n+`badSymrefPointee`::\n+\t(ERROR) The pointee of a symref is bad.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \n@@ -167,6 +173,9 @@\n `nullSha1`::\n \t(WARN) Tree contains entries pointing to a null sha1.\n \n+`trailingRefContent`::\n+\t(WARN) A ref content has trailing contents.\n+\n `treeNotSorted`::\n \t(ERROR) A tree is not properly sorted.\n \ndiff --git a/fsck.h b/fsck.h\nindex 421ec62b8f..adc88e01c7 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -32,6 +32,8 @@ enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n+\tFUNC(BAD_REF_CONTENT, ERROR) \\\n+\tFUNC(BAD_SYMREF_POINTEE, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n@@ -72,6 +74,7 @@ enum fsck_msg_type {\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n+\tFUNC(TRAILING_REF_CONTENT, WARN) \\\n \tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n \tFUNC(LARGE_PATHNAME, WARN) \\\ndiff --git a/refs.c b/refs.c\nindex 410919246b..eb82fb7d4e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1760,7 +1760,7 @@ static int refs_read_special_head(struct ref_store *ref_store,\n \t}\n \n \tresult = parse_loose_ref_contents(content.buf, oid, referent, type,\n-\t\t\t\t\t failure_errno);\n+\t\t\t\t\t failure_errno, NULL);\n \n done:\n \tstrbuf_release(&full_path);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex f93c44c4d1..d8712269f1 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1,6 +1,7 @@\n #define USE_THE_REPOSITORY_VARIABLE\n \n #include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n #include \"../copy.h\"\n #include \"../environment.h\"\n #include \"../gettext.h\"\n@@ -553,7 +554,7 @@ static int read_ref_internal(struct ref_store *ref_store, const char *refname,\n \tstrbuf_rtrim(&sb_contents);\n \tbuf = sb_contents.buf;\n \n-\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr);\n+\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr, NULL);\n \n out:\n \tif (ret && !myerr)\n@@ -589,7 +590,7 @@ static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refn\n \n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno)\n+\t\t\t int *failure_errno, const char **trailing)\n {\n \tconst char *p;\n \tif (skip_prefix(buf, \"ref:\", &buf)) {\n@@ -611,6 +612,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t*failure_errno = EINVAL;\n \t\treturn -1;\n \t}\n+\n+\tif (trailing)\n+\t\t*trailing = p;\n+\n \treturn 0;\n }\n \n@@ -3439,6 +3444,138 @@ static int files_fsck_refs_name(struct fsck_options *o,\n \treturn ret;\n }\n \n+/*\n+ * Check the symref \"pointee_name\" and \"pointee_path\". The caller should\n+ * make sure that \"pointee_path\" is absolute. For symbolic ref, \"pointee_name\"\n+ * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n+ * be the relative path agaignst \"gitdir\".\n+ */\n+static int files_fsck_symref_target(struct fsck_options *o,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const char *pointee_name,\n+\t\t\t\t const char *pointee_path)\n+{\n+\tconst char *p = NULL;\n+\tstruct stat st;\n+\tint ret = 0;\n+\n+\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n+\n+\t\tret = fsck_refs_report(o, NULL,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to target out of refs hierarchy\");\n+\t\tgoto out;\n+\t}\n+\n+\tif (check_refname_format(pointee_name, 0)) {\n+\t\tret = fsck_refs_report(o, NULL,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid refname\");\n+\t}\n+\n+\tif (lstat(pointee_path, &st) < 0)\n+\t\tgoto out;\n+\n+\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n+\t\tret = fsck_refs_report(o, NULL,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"point to invalid target\");\n+\t\tgoto out;\n+\t}\n+out:\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs_content(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter)\n+{\n+\tstruct strbuf pointee_path = STRBUF_INIT,\n+\t\t ref_content = STRBUF_INIT,\n+\t\t abs_gitdir = STRBUF_INIT,\n+\t\t referent = STRBUF_INIT,\n+\t\t refname = STRBUF_INIT;\n+\tconst char *trailing = NULL;\n+\tint failure_errno = 0;\n+\tunsigned int type = 0;\n+\tstruct object_id oid;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\to->refs_info.ref_checkee = refname.buf;\n+\n+\t/*\n+\t * If the file is a symlink, we need to only check the connectivity\n+\t * of the destination object.\n+\t */\n+\tif (S_ISLNK(iter->st.st_mode)) {\n+\t\tconst char *pointee_name = NULL;\n+\n+\t\tstrbuf_add_real_path(&pointee_path, iter->path.buf);\n+\n+\t\tstrbuf_add_absolute_path(&abs_gitdir, gitdir);\n+\t\tstrbuf_normalize_path(&abs_gitdir);\n+\t\tif (!is_dir_sep(abs_gitdir.buf[abs_gitdir.len - 1]))\n+\t\t\tstrbuf_addch(&abs_gitdir, '/');\n+\n+\t\tif (!skip_prefix(pointee_path.buf,\n+\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n+\t\t\tret = fsck_refs_report(o, NULL,\n+\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t\t \"point to target outside gitdir\");\n+\t\t\tgoto clean;\n+\t\t}\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, pointee_name,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n+\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n+\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n+\t\t\t\t &referent, &type,\n+\t\t\t\t &failure_errno, &trailing)) {\n+\t\tret = fsck_refs_report(o, NULL,\n+\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n+\t\t\t\t \"invalid ref content\");\n+\t\tgoto clean;\n+\t}\n+\n+\t/*\n+\t * If the ref is a symref, we need to check the destination name and\n+\t * connectivity.\n+\t */\n+\tif (referent.len && (type & REF_ISSYMREF)) {\n+\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n+\t\tstrbuf_rtrim(&referent);\n+\n+\t\tret = files_fsck_symref_target(o, refname.buf, referent.buf,\n+\t\t\t\t\t pointee_path.buf);\n+\t\tgoto clean;\n+\t} else {\n+\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n+\t\t\tret = fsck_refs_report(o, NULL,\n+\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n+\t\t\t\t\t \"trailing garbage in ref\");\n+\t\t\tgoto clean;\n+\t\t}\n+\t}\n+\n+clean:\n+\tstrbuf_release(&abs_gitdir);\n+\tstrbuf_release(&pointee_path);\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&ref_content);\n+\tstrbuf_release(&referent);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3491,6 +3628,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n \t\tfiles_fsck_refs_name,\n+\t\tfiles_fsck_refs_content,\n \t\tNULL\n \t};\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex a905e187cd..2fabf41d14 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -709,11 +709,12 @@ struct ref_store {\n \n /*\n * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for\n- * invalid contents.\n+ * invalid contents. Also *trailing is set to the first character after the\n+ * refname or NULL if the referent is not empty.\n */\n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno);\n+\t\t\t int *failure_errno, const char **trailing);\n \n /*\n * Fill in the generic part of refs and add it to our collection of\ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nindex b2db58d2c6..35bf40ee64 100755\n--- a/t/t0602-reffiles-fsck.sh\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -98,4 +98,114 @@ test_expect_success 'ref name check should be adapted into fsck messages' '\n \t)\n '\n \n+test_expect_success 'regular ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit checkout -b a/b/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse branch-1)\" > $branch_dir_prefix/branch-1-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse tag-1)\" > $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_must_fail git -c fsck.trailingRefContent=error fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s \" \"$(git rev-parse tag-2)\" > $tag_dir_prefix/tag-2-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/tags/tag-2-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/a/b/branch-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'symbolic ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid refname\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid target\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: logs/maint-v2.45\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to target out of refs hierarchy\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.2\n\n"},{"id":"499137","messageId":"CAOLa=ZQKvEEKYHj5WACjfAT4DUb3=p2g=WaWHr2zG0FE1UJO_A@mail.gmail.com","threadId":"61565","inReplyTo":"ZpuDCc8SZx8e4mLP@ArchLinux","subject":"Re: [GSoC][PATCH v12 02/10] fsck: add a unified interface for reporting fsck messages","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-07-23T08:35:47Z","receivedAt":"2024-07-23T08:35:48Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n[snip]\n\n> diff --git a/fsck.h b/fsck.h\n> index bcfb2e34cd..a2ecb39b51 100644\n> --- a/fsck.h\n> +++ b/fsck.h\n> @@ -114,7 +114,9 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n> typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n> \t\t\t void *data, struct fsck_options *options);\n>\n> -/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n> +/*\n> + * callback function for reporting errors when checking either objects or refs\n> + */\n> typedef int (*fsck_error)(struct fsck_options *o,\n> \t\t\t const struct object_id *oid, enum object_type object_type,\n> \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n> @@ -131,11 +133,24 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n> \t\t\t\t\t enum fsck_msg_id msg_id,\n> \t\t\t\t\t const char *message);\n>\n> +/*\n> + * The information for reporting refs-related error message\n> + */\n> +struct fsck_refs_info {\n> +\tchar *ref_checkee;\n> +\tunion {\n> +\t\tstruct {\n> +\t\t\tchar *sub_ref_checkee;\n> +\t\t} files;\n> +\t} u;\n> +};\n> +\n\nWhen I suggested using a struct, it was to replace sending both\n'ref_checkee' and 'sub_ref_checkee' to the error functions. I.e to send\na single struct instead of two different fields. But now, we've included\nit in the 'fsck_options' struct, I don't think it should be part of the\noptions. Because the fsck_options is used beyond the error function\nwhile 'fsck_refs_info' is only needed in the error functions, right?\n"},{"id":"499510","messageId":"ZqeXrPROpEg_pRS2@ArchLinux","threadId":"61565","inReplyTo":"ZpuCg1GL1YE_sJBP@ArchLinux","subject":"[GSoC][PATCH v13 00/10] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-29T13:22:52Z","receivedAt":"2024-07-29T13:22:34Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Hi All:\n\nThis version mainly focuses on the problem about how should we provide\nextensibility for reporting the problem of refs. In patch v10, Junio\nasked a question here:\n\n> The error reporting function for refs consistency check was still\n> about reporting a problem for a single ref. I am wondering how\n> consistency violations that are not about a single ref should be\n> handled. For example, if refs/packed-backend.c:packed_fsck() finds\n> that the file is not sorted properly or has some unparseable garbage\n> in it, it is not something you can report as \"refs/heads/main is\n> broken\", but those who are interested in seeing the \"reference\n> database consistency\" verified, it is very much what they want the\n> tool to notice. How would detection of such a breakage that is not\n> attributed to a single ref fit in this \"ref consistency check\n> infrastructure\" that was introduced by [05/10]?\n\nActually, I think that the original parameter \"checked_ref_name\" is a\nbad name which makes the reader think that \"we only handle refs\". And\nthis is my fault for this design. However, I misunderstood the Junio's\nword and made things complicated.\n\nThe patch v11 and v12 wants to solve a problem that we should provide\nextensibility for reporting refs problem. However, these two versions\nstill made things complicated. After an offline meeting with Patrick and\nKarthik, we design the following simple flat data structure:\n\n struct fsck_refs_info {\n const char *path;\n };\n\nIt is simple and provides extensibility.\n\nThanks,\nJialuo\n\nshejialuo (10):\n fsck: rename \"skiplist\" to \"skip_oids\"\n fsck: add a unified interface for reporting fsck messages\n fsck: rename objects-related fsck error functions\n fsck: add refs-related error report function\n refs: set up ref consistency check infrastructure\n git refs: add verify subcommand\n builtin/fsck: add `git-refs verify` child process\n files-backend: add unified interface for refs scanning\n fsck: add ref name check for files backend\n fsck: add ref content check for files backend\n\n Documentation/fsck-msgids.txt | 12 ++\n Documentation/git-refs.txt | 13 ++\n builtin/fsck.c | 32 ++++-\n builtin/mktag.c | 1 +\n builtin/refs.c | 44 ++++++\n fsck.c | 109 +++++++++++----\n fsck.h | 74 +++++++---\n object-file.c | 11 +-\n refs.c | 7 +-\n refs.h | 8 ++\n refs/debug.c | 11 ++\n refs/files-backend.c | 251 +++++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 ++\n refs/refs-internal.h | 11 +-\n refs/reftable-backend.c | 8 ++\n t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++\n 16 files changed, 754 insertions(+), 57 deletions(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\nRange-diff against v12:\n 1: a69705b777 = 1: a69705b777 fsck: rename \"skiplist\" to \"skip_oids\"\n 2: a4bfccd938 < -: ---------- fsck: add a unified interface for reporting fsck messages\n -: ---------- > 2: 178329d085 fsck: add a unified interface for reporting fsck messages\n 3: 9bc8892761 ! 3: 341cb841bf fsck: rename objects-related fsck error functions\n @@ builtin/fsck.c: static int objerror(struct object *obj, const char *err)\n -static int fsck_error_func(struct fsck_options *o UNUSED,\n -\t\t\t const struct object_id *oid,\n -\t\t\t enum object_type object_type,\n +-\t\t\t const struct fsck_refs_info *refs_info UNUSED,\n -\t\t\t enum fsck_msg_type msg_type,\n -\t\t\t enum fsck_msg_id msg_id UNUSED,\n -\t\t\t const char *message)\n +static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n +\t\t\t\t const struct object_id *oid,\n +\t\t\t\t enum object_type object_type,\n ++\t\t\t\t const struct fsck_refs_info *refs_info UNUSED,\n +\t\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n +\t\t\t\t const char *message)\n @@ fsck.c: int fsck_buffer(const struct object_id *oid, enum object_type type,\n -int fsck_error_function(struct fsck_options *o,\n -\t\t\tconst struct object_id *oid,\n -\t\t\tenum object_type object_type UNUSED,\n +-\t\t\tconst struct fsck_refs_info *refs_info UNUSED,\n -\t\t\tenum fsck_msg_type msg_type,\n -\t\t\tenum fsck_msg_id msg_id UNUSED,\n -\t\t\tconst char *message)\n +int fsck_objects_error_function(struct fsck_options *o,\n +\t\t\t\tconst struct object_id *oid,\n +\t\t\t\tenum object_type object_type UNUSED,\n ++\t\t\t\tconst struct fsck_refs_info *refs_info UNUSED,\n +\t\t\t\tenum fsck_msg_type msg_type,\n +\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n +\t\t\t\tconst char *message)\n @@ fsck.c: int git_fsck_config(const char *var, const char *value,\n -int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n -\t\t\t\t\t const struct object_id *oid,\n -\t\t\t\t\t enum object_type object_type,\n +-\t\t\t\t\t const struct fsck_refs_info *refs_info,\n -\t\t\t\t\t enum fsck_msg_type msg_type,\n -\t\t\t\t\t enum fsck_msg_id msg_id,\n -\t\t\t\t\t const char *message)\n +int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n +\t\t\t\t\t\t const struct object_id *oid,\n +\t\t\t\t\t\t enum object_type object_type,\n ++\t\t\t\t\t\t const struct fsck_refs_info *refs_info,\n +\t\t\t\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t\t\t\t enum fsck_msg_id msg_id,\n +\t\t\t\t\t\t const char *message)\n @@ fsck.c: int git_fsck_config(const char *var, const char *value,\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n --\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n -+\treturn fsck_objects_error_function(o, oid, object_type,\n +-\treturn fsck_error_function(o, oid, object_type, refs_info,\n +-\t\t\t\t msg_type, msg_id, message);\n ++\treturn fsck_objects_error_function(o, oid, object_type, refs_info,\n +\t\t\t\t\t msg_type, msg_id, message);\n }\n \n @@ fsck.h: typedef int (*fsck_error)(struct fsck_options *o,\n \n -int fsck_error_function(struct fsck_options *o,\n -\t\t\tconst struct object_id *oid, enum object_type object_type,\n +-\t\t\tconst struct fsck_refs_info *refs_info,\n -\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n -\t\t\tconst char *message);\n -int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n -\t\t\t\t\t const struct object_id *oid,\n -\t\t\t\t\t enum object_type object_type,\n +-\t\t\t\t\t const struct fsck_refs_info *refs_info,\n -\t\t\t\t\t enum fsck_msg_type msg_type,\n -\t\t\t\t\t enum fsck_msg_id msg_id,\n -\t\t\t\t\t const char *message);\n +int fsck_objects_error_function(struct fsck_options *o,\n +\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n ++\t\t\t\tconst struct fsck_refs_info *refs_info,\n +\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n +\t\t\t\tconst char *message);\n +int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n +\t\t\t\t\t\t const struct object_id *oid,\n +\t\t\t\t\t\t enum object_type object_type,\n ++\t\t\t\t\t\t const struct fsck_refs_info *refs_info,\n +\t\t\t\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t\t\t\t enum fsck_msg_id msg_id,\n +\t\t\t\t\t\t const char *message);\n 4: 82296dc2b9 ! 4: b328f3b218 fsck: add refs-related error report function\n @@ Metadata\n ## Commit message ##\n fsck: add refs-related error report function\n \n - Create refs-specific \"error_func\" callback \"fsck_refs_error_function\"\n - which could provide the following report messages for files backend\n + Add refs-related options to the \"fsck_options\", create refs-specific\n + \"error_func\" callback \"fsck_refs_error_function\".\n \n - 1. \"ref_checkee\": \"fsck error name\": \"user message\".\n - 2. \"ref_checkee.sub_ref_checkee\": \"fsck error name\": \"user message\".\n - 3. \"ref_checkee -> (oid hex)\": \"fsck error name\": \"user message\".\n - 4. \"ref_checkee.sub_ref_checkee -> (oid hex)\": \"fsck error name\": \"user\n - message\".\n -\n - \"fsck_refs_error_function\" uses the \"ref_checkee\" and \"sub_ref_checkee\"\n - in the \"fsck_refs_info\" to indicate the information of the checked refs.\n - For loose ref and reflog, it only uses the \"ref_checkee\". For packed\n - refs and reftable refs, when checking the consistency of the file\n - itself, it still only uses \"ref_checkee\". However, when checking the\n - consistency of the ref or reflog contained in the file ,it will use the\n - \"sub_ref_checkee\" to indicate that we are not checking the file but the\n - incorporated ref or reflog.\n -\n - \"fsck_refs_error_function\" will use the \"oid\" parameter if the caller\n + \"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\n passes the oid, it will use \"oid_to_hex\" to get the corresponding hex\n value to report to the caller.\n \n @@ fsck.c: int fsck_objects_error_function(struct fsck_options *o,\n +int fsck_refs_error_function(struct fsck_options *options,\n +\t\t\t const struct object_id *oid,\n +\t\t\t enum object_type object_type UNUSED,\n ++\t\t\t const struct fsck_refs_info *refs_info,\n +\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t enum fsck_msg_id msg_id UNUSED,\n +\t\t\t const char *message)\n +{\n +\tstruct strbuf sb = STRBUF_INIT;\n -+\tstruct fsck_refs_info *refs_info = &options->refs_info;\n +\tint ret = 0;\n +\n -+\tif (the_repository->ref_storage_format == REF_STORAGE_FORMAT_FILES) {\n -+\t\tstrbuf_addstr(&sb, refs_info->ref_checkee);\n -+\t\tif (refs_info->u.files.sub_ref_checkee)\n -+\t\t\tstrbuf_addf(&sb, \".%s\", refs_info->u.files.sub_ref_checkee);\n ++\tstrbuf_addstr(&sb, refs_info->path);\n +\n -+\t\tif (oid)\n -+\t\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n -+\t}\n ++\tif (oid)\n ++\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n +\n +\tif (msg_type == FSCK_WARN)\n +\t\twarning(\"%s: %s\", sb.buf, message);\n @@ fsck.h: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *\n +int fsck_refs_error_function(struct fsck_options *options,\n +\t\t\t const struct object_id *oid,\n +\t\t\t enum object_type object_type,\n ++\t\t\t const struct fsck_refs_info *refs_info,\n +\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t enum fsck_msg_id msg_id,\n +\t\t\t const char *message);\n 5: c5cac2e318 ! 5: 481bffac6e refs: set up ref consistency check infrastructure\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +static int files_fsck(struct ref_store *ref_store,\n +\t\t struct fsck_options *o)\n +{\n -+\tint ret;\n +\tstruct files_ref_store *refs =\n +\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n +\n -+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n -+\treturn ret;\n ++\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n +}\n +\n struct ref_storage_be refs_be_files = {\n 6: 84d840506e ! 6: cea8908338 git refs: add verify subcommand\n @@ fsck.h: struct fsck_options {\n \tunsigned strict:1;\n +\tunsigned verbose:1;\n \tenum fsck_msg_type *msg_type;\n - \tstruct fsck_refs_info refs_info;\n \tstruct oidset skip_oids;\n + \tstruct oidset gitmodules_found;\n 7: 3fc77ec329 = 7: 2200167bf6 builtin/fsck: add `git-refs verify` child process\n 8: 44a75141fa ! 8: c69883be9a files-backend: add unified interface for refs scanning\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +static int files_fsck_refs(struct ref_store *ref_store,\n +\t\t\t struct fsck_options *o)\n +{\n -+\tint ret;\n +\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n +\t\tNULL\n +\t};\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +\tif (o->verbose)\n +\t\tfprintf_ln(stderr, \"Checking references consistency\");\n +\n -+\tret = files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n ++\treturn files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n +\n -+\treturn ret;\n +}\n +\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_options *o)\n {\n -@@ refs/files-backend.c: static int files_fsck(struct ref_store *ref_store,\n \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n --\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n -+\tret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o)\n -+\t | files_fsck_refs(ref_store, o);\n -+\n - \treturn ret;\n +-\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n ++\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o) |\n ++\t files_fsck_refs(ref_store, o);\n }\n \n + struct ref_storage_be refs_be_files = {\n 9: 4a0d58b07d ! 9: 5ce0e7367e fsck: add ref name check for files backend\n @@ Documentation/fsck-msgids.txt\n \t(ERROR) A commit object has a bad parent sha1.\n \n +`badRefName`::\n -+\t(ERROR) A ref has a bad name.\n ++\t(ERROR) A ref has an invalid format.\n +\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n @@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n +\t\t\t\tstruct dir_iterator *iter)\n +{\n +\tstruct strbuf sb = STRBUF_INIT;\n ++\tstruct fsck_refs_info info;\n +\tint ret = 0;\n +\n +\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n +\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n -+\t\to->refs_info.ref_checkee = sb.buf;\n -+\t\tret = fsck_refs_report(o, NULL,\n ++\t\tinfo.path = sb.buf;\n ++\t\tret = fsck_refs_report(o, NULL, &info,\n +\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n +\t\t\t\t \"invalid refname format\");\n +\t}\n @@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n @@ refs/files-backend.c: static int files_fsck_refs(struct ref_store *ref_store,\n + \t\t\t struct fsck_options *o)\n {\n - \tint ret;\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n +\t\tfiles_fsck_refs_name,\n \t\tNULL\n10: c529670e54 ! 10: f77ca18c68 fsck: add ref content check for files backend\n @@ Documentation/fsck-msgids.txt\n +\t(ERROR) A ref has a bad content.\n +\n `badRefName`::\n - \t(ERROR) A ref has a bad name.\n + \t(ERROR) A ref has an invalid format.\n \n +`badSymrefPointee`::\n +\t(ERROR) The pointee of a symref is bad.\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n + * be the relative path agaignst \"gitdir\".\n + */\n +static int files_fsck_symref_target(struct fsck_options *o,\n ++\t\t\t\t struct fsck_refs_info *info,\n +\t\t\t\t const char *refname,\n +\t\t\t\t const char *pointee_name,\n +\t\t\t\t const char *pointee_path)\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\n +\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n +\n -+\t\tret = fsck_refs_report(o, NULL,\n ++\t\tret = fsck_refs_report(o, NULL, info,\n +\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n -+\t\t\t\t \"point to target out of refs hierarchy\");\n ++\t\t\t\t \"points to ref outside the refs directory\");\n +\t\tgoto out;\n +\t}\n +\n +\tif (check_refname_format(pointee_name, 0)) {\n -+\t\tret = fsck_refs_report(o, NULL,\n ++\t\tret = fsck_refs_report(o, NULL, info,\n +\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n -+\t\t\t\t \"point to invalid refname\");\n ++\t\t\t\t \"points to refname with invalid format\");\n +\t}\n +\n +\tif (lstat(pointee_path, &st) < 0)\n +\t\tgoto out;\n +\n +\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n -+\t\tret = fsck_refs_report(o, NULL,\n ++\t\tret = fsck_refs_report(o, NULL, info,\n +\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n -+\t\t\t\t \"point to invalid target\");\n ++\t\t\t\t \"points to an invalid file type\");\n +\t\tgoto out;\n +\t}\n +out:\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\t\t referent = STRBUF_INIT,\n +\t\t refname = STRBUF_INIT;\n +\tconst char *trailing = NULL;\n ++\tstruct fsck_refs_info info;\n +\tint failure_errno = 0;\n +\tunsigned int type = 0;\n +\tstruct object_id oid;\n +\tint ret = 0;\n +\n +\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n -+\to->refs_info.ref_checkee = refname.buf;\n ++\tinfo.path = refname.buf;\n +\n +\t/*\n +\t * If the file is a symlink, we need to only check the connectivity\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\n +\t\tif (!skip_prefix(pointee_path.buf,\n +\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n -+\t\t\tret = fsck_refs_report(o, NULL,\n ++\t\t\tret = fsck_refs_report(o, NULL, &info,\n +\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n +\t\t\t\t\t \"point to target outside gitdir\");\n +\t\t\tgoto clean;\n +\t\t}\n +\n -+\t\tret = files_fsck_symref_target(o, refname.buf, pointee_name,\n -+\t\t\t\t\t pointee_path.buf);\n ++\t\tret = files_fsck_symref_target(o, &info, refname.buf,\n ++\t\t\t\t\t pointee_name, pointee_path.buf);\n +\t\tgoto clean;\n +\t}\n +\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n +\t\t\t\t &referent, &type,\n +\t\t\t\t &failure_errno, &trailing)) {\n -+\t\tret = fsck_refs_report(o, NULL,\n ++\t\tret = fsck_refs_report(o, NULL, &info,\n +\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n +\t\t\t\t \"invalid ref content\");\n +\t\tgoto clean;\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n +\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n +\t\tstrbuf_rtrim(&referent);\n +\n -+\t\tret = files_fsck_symref_target(o, refname.buf, referent.buf,\n -+\t\t\t\t\t pointee_path.buf);\n ++\t\tret = files_fsck_symref_target(o, &info, refname.buf,\n ++\t\t\t\t\t referent.buf, pointee_path.buf);\n +\t\tgoto clean;\n +\t} else {\n +\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n -+\t\t\tret = fsck_refs_report(o, NULL,\n ++\t\t\tret = fsck_refs_report(o, NULL, &info,\n +\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n +\t\t\t\t\t \"trailing garbage in ref\");\n +\t\t\tgoto clean;\n @@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n @@ refs/files-backend.c: static int files_fsck_refs(struct ref_store *ref_store,\n - \tint ret;\n + {\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n \t\tfiles_fsck_refs_name,\n +\t\tfiles_fsck_refs_content,\n @@ t/t0602-reffiles-fsck.sh: test_expect_success 'ref name check should be adapted\n +\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n +\t\ttest_must_fail git refs verify 2>err &&\n +\t\tcat >expect <<-EOF &&\n -+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid refname\n ++\t\terror: refs/heads/branch-2-bad: badSymrefPointee: points to refname with invalid format\n +\t\tEOF\n +\t\trm $branch_dir_prefix/branch-2-bad &&\n +\t\ttest_cmp expect err\n @@ t/t0602-reffiles-fsck.sh: test_expect_success 'ref name check should be adapted\n +\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n +\t\ttest_must_fail git refs verify 2>err &&\n +\t\tcat >expect <<-EOF &&\n -+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to invalid target\n ++\t\terror: refs/heads/branch-2-bad: badSymrefPointee: points to an invalid file type\n +\t\tEOF\n +\t\trm $branch_dir_prefix/branch-2-bad &&\n +\t\ttest_cmp expect err\n @@ t/t0602-reffiles-fsck.sh: test_expect_success 'ref name check should be adapted\n +\t\tprintf \"ref: logs/maint-v2.45\" > $branch_dir_prefix/branch-2-bad &&\n +\t\ttest_must_fail git refs verify 2>err &&\n +\t\tcat >expect <<-EOF &&\n -+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: point to target out of refs hierarchy\n ++\t\terror: refs/heads/branch-2-bad: badSymrefPointee: points to ref outside the refs directory\n +\t\tEOF\n +\t\trm $branch_dir_prefix/branch-2-bad &&\n +\t\ttest_cmp expect err\n-- \n2.45.2\n\n"},{"id":"499511","messageId":"ZqeYfASSt-sdXuKv@ArchLinux","threadId":"61565","inReplyTo":"ZqeXrPROpEg_pRS2@ArchLinux","subject":"[GSoC][PATCH v13 01/10] fsck: rename \"skiplist\" to \"skip_oids\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-29T13:26:20Z","receivedAt":"2024-07-29T13:26:01Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The \"skiplist\" field in \"fsck_options\" is related to objects. Because we\nare going to introduce ref consistency check, the \"skiplist\" name is too\ngeneral which will make the caller think \"skiplist\" is related to both\nthe refs and objects.\n\nIt may seem that for both refs and objects, we should provide a general\n\"skiplist\" here. However, the type for \"skiplist\" is `struct oidset`\nwhich is totally unsuitable for refs.\n\nTo avoid above ambiguity, rename \"skiplist\" to \"skip_oids\".\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 4 ++--\n fsck.h | 4 ++--\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex eea7145470..3f32441492 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n \t\tif (!strcmp(buf, \"skiplist\")) {\n \t\t\tif (equal == len)\n \t\t\t\tdie(\"skiplist requires a path\");\n-\t\t\toidset_parse_file(&options->skiplist, buf + equal + 1,\n+\t\t\toidset_parse_file(&options->skip_oids, buf + equal + 1,\n \t\t\t\t\t the_repository->hash_algo);\n \t\t\tbuf += len + 1;\n \t\t\tcontinue;\n@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n static int object_on_skiplist(struct fsck_options *opts,\n \t\t\t const struct object_id *oid)\n {\n-\treturn opts && oid && oidset_contains(&opts->skiplist, oid);\n+\treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n __attribute__((format (printf, 5, 6)))\ndiff --git a/fsck.h b/fsck.h\nindex 6085a384f6..bcfb2e34cd 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -136,7 +136,7 @@ struct fsck_options {\n \tfsck_error error_func;\n \tunsigned strict:1;\n \tenum fsck_msg_type *msg_type;\n-\tstruct oidset skiplist;\n+\tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n \tstruct oidset gitattributes_found;\n@@ -145,7 +145,7 @@ struct fsck_options {\n };\n \n #define FSCK_OPTIONS_DEFAULT { \\\n-\t.skiplist = OIDSET_INIT, \\\n+\t.skip_oids = OIDSET_INIT, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n-- \n2.45.2\n\n"},{"id":"499512","messageId":"ZqeYhkaArVmMdrnK@ArchLinux","threadId":"61565","inReplyTo":"ZqeXrPROpEg_pRS2@ArchLinux","subject":"[GSoC][PATCH v13 02/10] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-29T13:26:30Z","receivedAt":"2024-07-29T13:26:11Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The static function \"report\" provided by \"fsck.c\" aims at checking fsck\nerror type and calling the callback \"error_func\" to report the message.\nHowever, \"report\" function is only related to object database which\ncannot be reused for refs. In order to provide a unified interface which\ncan report either objects or refs, create a new function \"fsck_vreport\"\nfollowing the \"report\" prototype. Instead of using \"...\", provide\n\"va_list\" to allow more flexibility.\n\nIn order to provide an extensible error report for refs, add a new\n\"fsck_refs_info\" structure and add parameter \"const struct\n*fsck_refs_info\" into \"fsck_vreport\" function.\n\nLike \"report\", the \"fsck_vreport\" function will use \"error_func\"\nregistered in \"fsck_options\" to report customized messages. Change\n\"error_func\" prototype to align with the new \"fsck_vreport\".\n\nThen, change \"report\" function to use \"fsck_vreport\" to report objects\nrelated messages. Add a new function called \"fsck_refs_report\" to use\n\"fsck_vreport\" to report refs related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 1 +\n builtin/mktag.c | 1 +\n fsck.c | 56 +++++++++++++++++++++++++++++++++++++++++--------\n fsck.h | 25 +++++++++++++++++++++-\n object-file.c | 11 +++++-----\n 5 files changed, 79 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..6abad60e7e 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -92,6 +92,7 @@ static int objerror(struct object *obj, const char *err)\n static int fsck_error_func(struct fsck_options *o UNUSED,\n \t\t\t const struct object_id *oid,\n \t\t\t enum object_type object_type,\n+\t\t\t const struct fsck_refs_info *refs_info UNUSED,\n \t\t\t enum fsck_msg_type msg_type,\n \t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t const char *message)\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex 4767f1a97e..6496deca0a 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -20,6 +20,7 @@ static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n \t\t\t\t const struct object_id *oid UNUSED,\n \t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const struct fsck_refs_info *refs_info UNUSED,\n \t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t\t const char *message)\ndiff --git a/fsck.c b/fsck.c\nindex 3f32441492..1185e9a8ad 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -226,12 +226,18 @@ static int object_on_skiplist(struct fsck_options *opts,\n \treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n-__attribute__((format (printf, 5, 6)))\n-static int report(struct fsck_options *options,\n-\t\t const struct object_id *oid, enum object_type object_type,\n-\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+/*\n+ * Provide a unified interface for either fscking refs or objects.\n+ * It will get the current msg error type and call the error_func callback\n+ * which is registered in the \"fsck_options\" struct.\n+ */\n+static int fsck_vreport(struct fsck_options *options,\n+\t\t\tconst struct object_id *oid,\n+\t\t\tenum object_type object_type,\n+\t\t\tconst struct fsck_refs_info *refs_info,\n+\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n {\n-\tva_list ap;\n+\tva_list ap_copy;\n \tstruct strbuf sb = STRBUF_INIT;\n \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n \tint result;\n@@ -250,9 +256,9 @@ static int report(struct fsck_options *options,\n \tprepare_msg_ids();\n \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n \n-\tva_start(ap, fmt);\n-\tstrbuf_vaddf(&sb, fmt, ap);\n-\tresult = options->error_func(options, oid, object_type,\n+\tva_copy(ap_copy, ap);\n+\tstrbuf_vaddf(&sb, fmt, ap_copy);\n+\tresult = options->error_func(options, oid, object_type, refs_info,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n \tva_end(ap);\n@@ -260,6 +266,35 @@ static int report(struct fsck_options *options,\n \treturn result;\n }\n \n+__attribute__((format (printf, 5, 6)))\n+static int report(struct fsck_options *options,\n+\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\n+\tva_start(ap, fmt);\n+\tresult = fsck_vreport(options, oid, object_type, NULL, msg_id, fmt, ap);\n+\tva_end(ap);\n+\n+\treturn result;\n+}\n+\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t const struct fsck_refs_info *refs_info,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = fsck_vreport(options, oid, OBJ_NONE, refs_info,\n+\t\t\t msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n void fsck_enable_object_names(struct fsck_options *options)\n {\n \tif (!options->object_names)\n@@ -1203,6 +1238,7 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n int fsck_error_function(struct fsck_options *o,\n \t\t\tconst struct object_id *oid,\n \t\t\tenum object_type object_type UNUSED,\n+\t\t\tconst struct fsck_refs_info *refs_info UNUSED,\n \t\t\tenum fsck_msg_type msg_type,\n \t\t\tenum fsck_msg_id msg_id UNUSED,\n \t\t\tconst char *message)\n@@ -1306,6 +1342,7 @@ int git_fsck_config(const char *var, const char *value,\n int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t const struct fsck_refs_info *refs_info,\n \t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message)\n@@ -1314,5 +1351,6 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n+\treturn fsck_error_function(o, oid, object_type, refs_info,\n+\t\t\t\t msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex bcfb2e34cd..4f01a46cc7 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -92,6 +92,7 @@ enum fsck_msg_id {\n };\n #undef MSG_ID\n \n+struct fsck_refs_info;\n struct fsck_options;\n struct object;\n \n@@ -114,23 +115,35 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n \t\t\t void *data, struct fsck_options *options);\n \n-/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n+/*\n+ * callback function for reporting errors when checking either objects or refs\n+ */\n typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t\t const struct fsck_refs_info *refs_info,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n int fsck_error_function(struct fsck_options *o,\n \t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\tconst struct fsck_refs_info *refs_info,\n \t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\tconst char *message);\n int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t const struct fsck_refs_info *refs_info,\n \t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message);\n \n+/*\n+ * The information for reporting refs-related error message\n+ */\n+struct fsck_refs_info {\n+\tconst char *path;\n+};\n+\n struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n@@ -209,6 +222,16 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+/*\n+ * Report an error or warning for refs.\n+ */\n+__attribute__((format (printf, 5, 6)))\n+int fsck_refs_report(struct fsck_options *options,\n+\t\t const struct object_id *oid,\n+\t\t const struct fsck_refs_info *refs_info,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\ndiff --git a/object-file.c b/object-file.c\nindex 065103be3e..91ddab2696 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2470,11 +2470,12 @@ int repo_has_object_file(struct repository *r,\n * give more context.\n */\n static int hash_format_check_report(struct fsck_options *opts UNUSED,\n-\t\t\t\t const struct object_id *oid UNUSED,\n-\t\t\t\t enum object_type object_type UNUSED,\n-\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n-\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t\t const char *message)\n+\t\t\t\t const struct object_id *oid UNUSED,\n+\t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t const struct fsck_refs_info *refs_info UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \terror(_(\"object fails fsck: %s\"), message);\n \treturn 1;\n-- \n2.45.2\n\n"},{"id":"499513","messageId":"ZqeYkP5YiISbMrrN@ArchLinux","threadId":"61565","inReplyTo":"ZqeXrPROpEg_pRS2@ArchLinux","subject":"[GSoC][PATCH v13 03/10] fsck: rename objects-related fsck error functions","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-29T13:26:40Z","receivedAt":"2024-07-29T13:26:21Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The names of objects-related fsck error functions are general. It's OK\nwhen there is only object database check. However, we have introduced\nrefs database check report function. To avoid ambiguity, rename\nobject-related fsck error functions to explicitly indicate these\nfunctions are used to report objects-related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 16 ++++++++--------\n fsck.c | 32 ++++++++++++++++----------------\n fsck.h | 30 +++++++++++++++---------------\n 3 files changed, 39 insertions(+), 39 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 6abad60e7e..8553e2bc1b 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -89,13 +89,13 @@ static int objerror(struct object *obj, const char *err)\n \treturn -1;\n }\n \n-static int fsck_error_func(struct fsck_options *o UNUSED,\n-\t\t\t const struct object_id *oid,\n-\t\t\t enum object_type object_type,\n-\t\t\t const struct fsck_refs_info *refs_info UNUSED,\n-\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t const char *message)\n+static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t enum object_type object_type,\n+\t\t\t\t const struct fsck_refs_info *refs_info UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \tswitch (msg_type) {\n \tcase FSCK_WARN:\n@@ -939,7 +939,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tfsck_walk_options.walk = mark_object;\n \tfsck_obj_options.walk = mark_used;\n-\tfsck_obj_options.error_func = fsck_error_func;\n+\tfsck_obj_options.error_func = fsck_objects_error_func;\n \tif (check_strict)\n \t\tfsck_obj_options.strict = 1;\n \ndiff --git a/fsck.c b/fsck.c\nindex 1185e9a8ad..af61fa90ba 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1235,13 +1235,13 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid,\n-\t\t\tenum object_type object_type UNUSED,\n-\t\t\tconst struct fsck_refs_info *refs_info UNUSED,\n-\t\t\tenum fsck_msg_type msg_type,\n-\t\t\tenum fsck_msg_id msg_id UNUSED,\n-\t\t\tconst char *message)\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tenum object_type object_type UNUSED,\n+\t\t\t\tconst struct fsck_refs_info *refs_info UNUSED,\n+\t\t\t\tenum fsck_msg_type msg_type,\n+\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n+\t\t\t\tconst char *message)\n {\n \tif (msg_type == FSCK_WARN) {\n \t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n@@ -1339,18 +1339,18 @@ int git_fsck_config(const char *var, const char *value,\n * Custom error callbacks that are used in more than one place.\n */\n \n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t const struct fsck_refs_info *refs_info,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message)\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t const struct fsck_refs_info *refs_info,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message)\n {\n \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_error_function(o, oid, object_type, refs_info,\n-\t\t\t\t msg_type, msg_id, message);\n+\treturn fsck_objects_error_function(o, oid, object_type, refs_info,\n+\t\t\t\t\t msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex 4f01a46cc7..f53ac339d2 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -124,18 +124,18 @@ typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid, enum object_type object_type,\n-\t\t\tconst struct fsck_refs_info *refs_info,\n-\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n-\t\t\tconst char *message);\n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t const struct fsck_refs_info *refs_info,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message);\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\t\tconst struct fsck_refs_info *refs_info,\n+\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n+\t\t\t\tconst char *message);\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t const struct fsck_refs_info *refs_info,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message);\n \n /*\n * The information for reporting refs-related error message\n@@ -163,7 +163,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function \\\n+\t.error_func = fsck_objects_error_function \\\n }\n #define FSCK_OPTIONS_STRICT { \\\n \t.strict = 1, \\\n@@ -171,7 +171,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function, \\\n+\t.error_func = fsck_objects_error_function, \\\n }\n #define FSCK_OPTIONS_MISSING_GITMODULES { \\\n \t.strict = 1, \\\n@@ -179,7 +179,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n+\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n \n /* descend in all linked child objects\n-- \n2.45.2\n\n"},{"id":"499514","messageId":"ZqeYm2J1LeXn_1-4@ArchLinux","threadId":"61565","inReplyTo":"ZqeXrPROpEg_pRS2@ArchLinux","subject":"[GSoC][PATCH v13 04/10] fsck: add refs-related error report function","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-29T13:26:51Z","receivedAt":"2024-07-29T13:26:32Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Add refs-related options to the \"fsck_options\", create refs-specific\n\"error_func\" callback \"fsck_refs_error_function\".\n\n\"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\npasses the oid, it will use \"oid_to_hex\" to get the corresponding hex\nvalue to report to the caller.\n\nLast, add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\"\nmacros to create refs options easily.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 25 +++++++++++++++++++++++++\n fsck.h | 14 ++++++++++++++\n 2 files changed, 39 insertions(+)\n\ndiff --git a/fsck.c b/fsck.c\nindex af61fa90ba..56de29b4c0 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1251,6 +1251,31 @@ int fsck_objects_error_function(struct fsck_options *o,\n \treturn 1;\n }\n \n+int fsck_refs_error_function(struct fsck_options *options,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type UNUSED,\n+\t\t\t const struct fsck_refs_info *refs_info,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tstrbuf_addstr(&sb, refs_info->path);\n+\n+\tif (oid)\n+\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n+\n+\tif (msg_type == FSCK_WARN)\n+\t\twarning(\"%s: %s\", sb.buf, message);\n+\telse\n+\t\tret = error(\"%s: %s\", sb.buf, message);\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n \t\t struct fsck_options *options, const char *blob_type)\ndiff --git a/fsck.h b/fsck.h\nindex f53ac339d2..a4a4ba88ee 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -136,6 +136,13 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message);\n+int fsck_refs_error_function(struct fsck_options *options,\n+\t\t\t const struct object_id *oid,\n+\t\t\t enum object_type object_type,\n+\t\t\t const struct fsck_refs_info *refs_info,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n \n /*\n * The information for reporting refs-related error message\n@@ -181,6 +188,13 @@ struct fsck_options {\n \t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n+#define FSCK_REFS_OPTIONS_STRICT { \\\n+\t.strict = 1, \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n \n /* descend in all linked child objects\n * the return value is:\n-- \n2.45.2\n\n"},{"id":"499515","messageId":"ZqeYptLyOxAfKwDi@ArchLinux","threadId":"61565","inReplyTo":"ZqeXrPROpEg_pRS2@ArchLinux","subject":"[GSoC][PATCH v13 05/10] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-29T13:27:02Z","receivedAt":"2024-07-29T13:26:43Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It's organized as the five parts:\n\n1. The name and the initialization interfaces.\n2. The ref transaction interfaces.\n3. The ref internal interfaces (pack, rename and copy).\n4. The ref filesystem interfaces.\n5. The reflog related interfaces.\n\nTo keep consistent with the git-fsck(1), add a new interface named\n\"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\ngrouped into any above five categories. Explicitly add blank line to\nmake it different from others.\n\nLast, implement placeholder functions for each ref backends.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs.c | 5 +++++\n refs.h | 8 ++++++++\n refs/debug.c | 11 +++++++++++\n refs/files-backend.c | 13 ++++++++++++-\n refs/packed-backend.c | 8 ++++++++\n refs/refs-internal.h | 6 ++++++\n refs/reftable-backend.c | 8 ++++++++\n 7 files changed, 58 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex bb90a18875..410919246b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs, struct fsck_options *o)\n+{\n+\treturn refs->be->fsck(refs, o);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex 0ecba21b4a..804d6a7fce 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -4,6 +4,7 @@\n #include \"commit.h\"\n #include \"repository.h\"\n \n+struct fsck_options;\n struct object_id;\n struct ref_store;\n struct strbuf;\n@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_options *o);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..45e2e784a0 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tint res = drefs->refs->be->fsck(drefs->refs, o);\n+\ttrace_printf_key(&trace_refs, \"fsck: %d\\n\", res);\n+\treturn res;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex aa52d9be7c..4630eb1f80 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3408,6 +3408,15 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_init,\n@@ -3434,5 +3443,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a0666407cd..5209b0b212 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_init,\n@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex fa975d69aa..a905e187cd 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_options;\n struct ref_transaction;\n \n /*\n@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_options *o);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -677,6 +681,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex fbe74c239d..b5a1a526df 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.45.2\n\n"},{"id":"499516","messageId":"ZqeYsNtl90N1fVDy@ArchLinux","threadId":"61565","inReplyTo":"ZqeXrPROpEg_pRS2@ArchLinux","subject":"[GSoC][PATCH v13 06/10] git refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-29T13:27:12Z","receivedAt":"2024-07-29T13:26:53Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\ncheck the reference database consistency and also this subcommand will\nbe used as the entry point of checking refs for \"git-fsck(1)\". Last, add\n\"verbose\" field into \"fsck_options\" to indicate whether we should print\nverbose messages when checking refs and objects consistency.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/git-refs.txt | 13 +++++++++++\n builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++\n fsck.h | 1 +\n 3 files changed, 58 insertions(+)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..1244a85b64 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+The following options are specific to 'git refs verify':\n+\n+--strict::\n+\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n+\tbe seen as ERROR. See linkgit:git-fsck[1].\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..4831c9e28e 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,54 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tunsigned int verbose = 0, strict = 0;\n+\tstruct option options[] = {\n+\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\tint ret;\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"'git refs verify' takes no arguments\"));\n+\n+\tif (verbose)\n+\t\tfsck_refs_options.verbose = 1;\n+\tif (strict)\n+\t\tfsck_refs_options.strict = 1;\n+\n+\tgit_config(git_fsck_config, &fsck_refs_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n+\n+\t/*\n+\t * Explicitly free the allocated array and \"skip_oids\" set\n+\t */\n+\tfree(fsck_refs_options.msg_type);\n+\toidset_clear(&fsck_refs_options.skip_oids);\n+\treturn ret;\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/fsck.h b/fsck.h\nindex a4a4ba88ee..b03dba442e 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -155,6 +155,7 @@ struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n \tunsigned strict:1;\n+\tunsigned verbose:1;\n \tenum fsck_msg_type *msg_type;\n \tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n-- \n2.45.2\n\n"},{"id":"499517","messageId":"ZqeYuWCQxOwW6msl@ArchLinux","threadId":"61565","inReplyTo":"ZqeXrPROpEg_pRS2@ArchLinux","subject":"[GSoC][PATCH v13 07/10] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-29T13:27:21Z","receivedAt":"2024-07-29T13:27:02Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new function \"fsck_refs\" that initializes and runs a child\nprocess to execute the \"git-refs verify\" command.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 8553e2bc1b..7ca1628ba5 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -897,6 +897,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \treturn res;\n }\n \n+static void fsck_refs(void)\n+{\n+\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n+\tchild_process_init(&refs_verify);\n+\trefs_verify.git_cmd = 1;\n+\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n+\tif (verbose)\n+\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n+\tif (check_strict)\n+\t\tstrvec_push(&refs_verify.args, \"--strict\");\n+\n+\tif (run_command(&refs_verify))\n+\t\terrors_found |= ERROR_REFS;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n@@ -1066,6 +1081,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n+\tfsck_refs();\n+\n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \n-- \n2.45.2\n\n"},{"id":"499518","messageId":"ZqeYw-k-MzhPTNRf@ArchLinux","threadId":"61565","inReplyTo":"ZqeXrPROpEg_pRS2@ArchLinux","subject":"[GSoC][PATCH v13 08/10] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-29T13:27:31Z","receivedAt":"2024-07-29T13:27:13Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"For refs and reflogs, we need to scan its corresponding directories to\ncheck every regular file or symbolic link which shares the same pattern.\nIntroduce a unified interface for scanning directories for\nfiles-backend.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs/files-backend.c | 74 +++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 73 insertions(+), 1 deletion(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 4630eb1f80..cb184953c1 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -6,6 +6,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -3408,13 +3409,84 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n+{\n+\tconst char *gitdir = ref_store->gitdir;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\n+\tif (!iter) {\n+\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n+\t\tgoto out;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->verbose)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n+\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n+\t\t\t\t iter->basename);\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n+\n+out:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tNULL\n+\t};\n+\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, \"Checking references consistency\");\n+\n+\treturn files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n+\n+}\n+\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_options *o)\n {\n \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n-\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o) |\n+\t files_fsck_refs(ref_store, o);\n }\n \n struct ref_storage_be refs_be_files = {\n-- \n2.45.2\n\n"},{"id":"499519","messageId":"ZqeY0eHNZjKhNvIH@ArchLinux","threadId":"61565","inReplyTo":"ZqeXrPROpEg_pRS2@ArchLinux","subject":"[GSoC][PATCH v13 09/10] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-29T13:27:45Z","receivedAt":"2024-07-29T13:27:27Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) only implicitly checks the reference, it does not fully\ncheck refs with bad format name such as standalone \"@\" and name ending\nwith \".lock\".\n\nIn order to provide such checks, add a new fsck message id \"badRefName\"\nwith default ERROR type. Use existing \"check_refname_format\" to explicit\ncheck the ref name. And add a new unit test to verify the functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 +\n fsck.h | 1 +\n refs/files-backend.c | 22 ++++++++\n t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++\n 4 files changed, 127 insertions(+)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex f643585a34..d8e437a043 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,9 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefName`::\n+\t(ERROR) A ref has an invalid format.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex b03dba442e..ce56ce4bef 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex cb184953c1..0d4fc27768 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3419,6 +3419,27 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n+static int files_fsck_refs_name(struct fsck_options *o,\n+\t\t\t\tconst char *gitdir UNUSED,\n+\t\t\t\tconst char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct fsck_refs_info info;\n+\tint ret = 0;\n+\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\tinfo.path = sb.buf;\n+\t\tret = fsck_refs_report(o, NULL, &info,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3469,6 +3490,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o)\n {\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n+\t\tfiles_fsck_refs_name,\n \t\tNULL\n \t};\n \ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..b2db58d2c6\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit tag multi_hierarchy/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1.lock: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1.lock &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/@ &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_must_fail git fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\t\tgit -c fsck.badRefName=warn fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/.branch-1 &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\t\tgit -c fsck.badRefName=ignore fsck 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+test_done\n-- \n2.45.2\n\n"},{"id":"499520","messageId":"ZqeY3Dhj-Fo-bZ2k@ArchLinux","threadId":"61565","inReplyTo":"ZqeXrPROpEg_pRS2@ArchLinux","subject":"[GSoC][PATCH v13 10/10] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-29T13:27:56Z","receivedAt":"2024-07-29T13:27:37Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Enhance the git-fsck(1) command by adding a check for reference content\nin the files backend. The new functionality ensures that symrefs, real\nsymbolic link and regular refs are validated correctly.\n\nIn order to check the trailing content of the regular refs, add a new\nparameter `trailing` to `parse_loose_ref_contents`.\n\nFor symrefs, `parse_loose_ref_contents` will set the \"referent\".\nHowever, symbolic link could be either absolute or relative. Use\n\"strbuf_add_real_path\" to read the symbolic link and convert the\nrelative path to absolute path. Then use \"skip_prefix\" to make it align\nwith symref \"referent\".\n\nThus, the symrefs and symbolic links could share the same interface. Add\na new function \"files_fsck_symref_target\" which aims at checking the\nfollowing things:\n\n1. whether the pointee is under the `refs/` directory.\n2. whether the pointee name is correct.\n3. whether the pointee path is a wrong type in filesystem.\n\nLast, add the following FSCK MESSAGEs:\n\n1. \"badRefContent(ERROR)\": A ref has a bad content\n2. \"badSymrefPointee(ERROR)\": The pointee of a symref is bad.\n3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 9 +++\n fsck.h | 3 +\n refs.c | 2 +-\n refs/files-backend.c | 144 +++++++++++++++++++++++++++++++++-\n refs/refs-internal.h | 5 +-\n t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++\n 6 files changed, 268 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex d8e437a043..8fe24a960e 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,9 +19,15 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefContent`::\n+\t(ERROR) A ref has a bad content.\n+\n `badRefName`::\n \t(ERROR) A ref has an invalid format.\n \n+`badSymrefPointee`::\n+\t(ERROR) The pointee of a symref is bad.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \n@@ -167,6 +173,9 @@\n `nullSha1`::\n \t(WARN) Tree contains entries pointing to a null sha1.\n \n+`trailingRefContent`::\n+\t(WARN) A ref content has trailing contents.\n+\n `treeNotSorted`::\n \t(ERROR) A tree is not properly sorted.\n \ndiff --git a/fsck.h b/fsck.h\nindex ce56ce4bef..710b3513d0 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -32,6 +32,8 @@ enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n+\tFUNC(BAD_REF_CONTENT, ERROR) \\\n+\tFUNC(BAD_SYMREF_POINTEE, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n@@ -72,6 +74,7 @@ enum fsck_msg_type {\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n+\tFUNC(TRAILING_REF_CONTENT, WARN) \\\n \tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n \tFUNC(LARGE_PATHNAME, WARN) \\\ndiff --git a/refs.c b/refs.c\nindex 410919246b..eb82fb7d4e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1760,7 +1760,7 @@ static int refs_read_special_head(struct ref_store *ref_store,\n \t}\n \n \tresult = parse_loose_ref_contents(content.buf, oid, referent, type,\n-\t\t\t\t\t failure_errno);\n+\t\t\t\t\t failure_errno, NULL);\n \n done:\n \tstrbuf_release(&full_path);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 0d4fc27768..131eec7307 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1,6 +1,7 @@\n #define USE_THE_REPOSITORY_VARIABLE\n \n #include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n #include \"../copy.h\"\n #include \"../environment.h\"\n #include \"../gettext.h\"\n@@ -553,7 +554,7 @@ static int read_ref_internal(struct ref_store *ref_store, const char *refname,\n \tstrbuf_rtrim(&sb_contents);\n \tbuf = sb_contents.buf;\n \n-\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr);\n+\tret = parse_loose_ref_contents(buf, oid, referent, type, &myerr, NULL);\n \n out:\n \tif (ret && !myerr)\n@@ -589,7 +590,7 @@ static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refn\n \n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno)\n+\t\t\t int *failure_errno, const char **trailing)\n {\n \tconst char *p;\n \tif (skip_prefix(buf, \"ref:\", &buf)) {\n@@ -611,6 +612,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t*failure_errno = EINVAL;\n \t\treturn -1;\n \t}\n+\n+\tif (trailing)\n+\t\t*trailing = p;\n+\n \treturn 0;\n }\n \n@@ -3440,6 +3445,140 @@ static int files_fsck_refs_name(struct fsck_options *o,\n \treturn ret;\n }\n \n+/*\n+ * Check the symref \"pointee_name\" and \"pointee_path\". The caller should\n+ * make sure that \"pointee_path\" is absolute. For symbolic ref, \"pointee_name\"\n+ * would be the content after \"refs:\". For symblic link, \"pointee_name\" would\n+ * be the relative path agaignst \"gitdir\".\n+ */\n+static int files_fsck_symref_target(struct fsck_options *o,\n+\t\t\t\t struct fsck_refs_info *info,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const char *pointee_name,\n+\t\t\t\t const char *pointee_path)\n+{\n+\tconst char *p = NULL;\n+\tstruct stat st;\n+\tint ret = 0;\n+\n+\tif (!skip_prefix(pointee_name, \"refs/\", &p)) {\n+\n+\t\tret = fsck_refs_report(o, NULL, info,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"points to ref outside the refs directory\");\n+\t\tgoto out;\n+\t}\n+\n+\tif (check_refname_format(pointee_name, 0)) {\n+\t\tret = fsck_refs_report(o, NULL, info,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"points to refname with invalid format\");\n+\t}\n+\n+\tif (lstat(pointee_path, &st) < 0)\n+\t\tgoto out;\n+\n+\tif (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {\n+\t\tret = fsck_refs_report(o, NULL, info,\n+\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t \"points to an invalid file type\");\n+\t\tgoto out;\n+\t}\n+out:\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs_content(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter)\n+{\n+\tstruct strbuf pointee_path = STRBUF_INIT,\n+\t\t ref_content = STRBUF_INIT,\n+\t\t abs_gitdir = STRBUF_INIT,\n+\t\t referent = STRBUF_INIT,\n+\t\t refname = STRBUF_INIT;\n+\tconst char *trailing = NULL;\n+\tstruct fsck_refs_info info;\n+\tint failure_errno = 0;\n+\tunsigned int type = 0;\n+\tstruct object_id oid;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\tinfo.path = refname.buf;\n+\n+\t/*\n+\t * If the file is a symlink, we need to only check the connectivity\n+\t * of the destination object.\n+\t */\n+\tif (S_ISLNK(iter->st.st_mode)) {\n+\t\tconst char *pointee_name = NULL;\n+\n+\t\tstrbuf_add_real_path(&pointee_path, iter->path.buf);\n+\n+\t\tstrbuf_add_absolute_path(&abs_gitdir, gitdir);\n+\t\tstrbuf_normalize_path(&abs_gitdir);\n+\t\tif (!is_dir_sep(abs_gitdir.buf[abs_gitdir.len - 1]))\n+\t\t\tstrbuf_addch(&abs_gitdir, '/');\n+\n+\t\tif (!skip_prefix(pointee_path.buf,\n+\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n+\t\t\tret = fsck_refs_report(o, NULL, &info,\n+\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n+\t\t\t\t\t \"point to target outside gitdir\");\n+\t\t\tgoto clean;\n+\t\t}\n+\n+\t\tret = files_fsck_symref_target(o, &info, refname.buf,\n+\t\t\t\t\t pointee_name, pointee_path.buf);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n+\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n+\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\tgoto clean;\n+\t}\n+\n+\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n+\t\t\t\t &referent, &type,\n+\t\t\t\t &failure_errno, &trailing)) {\n+\t\tret = fsck_refs_report(o, NULL, &info,\n+\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n+\t\t\t\t \"invalid ref content\");\n+\t\tgoto clean;\n+\t}\n+\n+\t/*\n+\t * If the ref is a symref, we need to check the destination name and\n+\t * connectivity.\n+\t */\n+\tif (referent.len && (type & REF_ISSYMREF)) {\n+\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n+\t\tstrbuf_rtrim(&referent);\n+\n+\t\tret = files_fsck_symref_target(o, &info, refname.buf,\n+\t\t\t\t\t referent.buf, pointee_path.buf);\n+\t\tgoto clean;\n+\t} else {\n+\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n+\t\t\tret = fsck_refs_report(o, NULL, &info,\n+\t\t\t\t\t FSCK_MSG_TRAILING_REF_CONTENT,\n+\t\t\t\t\t \"trailing garbage in ref\");\n+\t\t\tgoto clean;\n+\t\t}\n+\t}\n+\n+clean:\n+\tstrbuf_release(&abs_gitdir);\n+\tstrbuf_release(&pointee_path);\n+\tstrbuf_release(&refname);\n+\tstrbuf_release(&ref_content);\n+\tstrbuf_release(&referent);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3491,6 +3630,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n {\n \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n \t\tfiles_fsck_refs_name,\n+\t\tfiles_fsck_refs_content,\n \t\tNULL\n \t};\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex a905e187cd..2fabf41d14 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -709,11 +709,12 @@ struct ref_store {\n \n /*\n * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for\n- * invalid contents.\n+ * invalid contents. Also *trailing is set to the first character after the\n+ * refname or NULL if the referent is not empty.\n */\n int parse_loose_ref_contents(const char *buf, struct object_id *oid,\n \t\t\t struct strbuf *referent, unsigned int *type,\n-\t\t\t int *failure_errno);\n+\t\t\t int *failure_errno, const char **trailing);\n \n /*\n * Fill in the generic part of refs and add it to our collection of\ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nindex b2db58d2c6..29cd824224 100755\n--- a/t/t0602-reffiles-fsck.sh\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -98,4 +98,114 @@ test_expect_success 'ref name check should be adapted into fsck messages' '\n \t)\n '\n \n+test_expect_success 'regular ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1 &&\n+\t\tgit commit --allow-empty -m second &&\n+\t\tgit checkout -b branch-2 &&\n+\t\tgit tag tag-2 &&\n+\t\tgit checkout -b a/b/tag-2\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse branch-1)\" > $branch_dir_prefix/branch-1-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/heads/branch-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s garbage\" \"$(git rev-parse tag-1)\" > $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_must_fail git -c fsck.trailingRefContent=error fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-1-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-1-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"%s \" \"$(git rev-parse tag-2)\" > $tag_dir_prefix/tag-2-garbage &&\n+\t\tgit fsck 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\twarning: refs/tags/tag-2-garbage: trailingRefContent: trailing garbage in ref\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-garbage &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/tags/tag-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $tag_dir_prefix/tag-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"xfsazqfxcadas\" > $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/a/b/branch-2-bad: badRefContent: invalid ref content\n+\t\tEOF\n+\t\trm $branch_dir_prefix/a/b/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'symbolic ref content should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\t\tgit checkout -b branch-1 &&\n+\t\tgit tag tag-1\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads/.branch\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: points to refname with invalid format\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: refs/heads\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: points to an invalid file type\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t) &&\n+\t(\n+\t\tcd repo &&\n+\t\tprintf \"ref: logs/maint-v2.45\" > $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: refs/heads/branch-2-bad: badSymrefPointee: points to ref outside the refs directory\n+\t\tEOF\n+\t\trm $branch_dir_prefix/branch-2-bad &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.2\n\n"},{"id":"499579","messageId":"Zqik1JFs-RYCwNnz@tanuki","threadId":"61565","inReplyTo":"ZqeYhkaArVmMdrnK@ArchLinux","subject":"Re: [GSoC][PATCH v13 02/10] fsck: add a unified interface for reporting fsck messages","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-07-30T08:31:16Z","receivedAt":"2024-07-30T08:31:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jul 29, 2024 at 09:26:30PM +0800, shejialuo wrote:\n> The static function \"report\" provided by \"fsck.c\" aims at checking fsck\n> error type and calling the callback \"error_func\" to report the message.\n> However, \"report\" function is only related to object database which\n> cannot be reused for refs.\n\nNit: it would be nice to mention _why_ it cannot be reused for refs.\n\n> diff --git a/fsck.c b/fsck.c\n> index 3f32441492..1185e9a8ad 100644\n> --- a/fsck.c\n> +++ b/fsck.c\n> @@ -226,12 +226,18 @@ static int object_on_skiplist(struct fsck_options *opts,\n> \treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n> }\n> \n> -__attribute__((format (printf, 5, 6)))\n> -static int report(struct fsck_options *options,\n> -\t\t const struct object_id *oid, enum object_type object_type,\n> -\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n> +/*\n> + * Provide a unified interface for either fscking refs or objects.\n> + * It will get the current msg error type and call the error_func callback\n> + * which is registered in the \"fsck_options\" struct.\n> + */\n> +static int fsck_vreport(struct fsck_options *options,\n> +\t\t\tconst struct object_id *oid,\n> +\t\t\tenum object_type object_type,\n> +\t\t\tconst struct fsck_refs_info *refs_info,\n> +\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n> {\n> -\tva_list ap;\n> +\tva_list ap_copy;\n> \tstruct strbuf sb = STRBUF_INIT;\n> \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n> \tint result;\n\nIt is a bit weird that this new generic function receives non-generic\ninputs which are specific to the respective subsystems (objects or refs)\nthat we are checking.\n\nA better design would likely be to make `error_func()` receive a void\npointer such that `error_func()` and then have the respective subsystems\nprovide a function that knows to format the message while receiving\neither a `struct fsck_object_report *` or a `struct fsck_ref_report *`.\n\nI don't think this is particularly worriesome though as it is still\nmanageable right now. So I'm fine if we want to leave this as-is, and\nthen we can iterate on this in a future patch series as required.\n\n> @@ -250,9 +256,9 @@ static int report(struct fsck_options *options,\n> \tprepare_msg_ids();\n> \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n> \n> -\tva_start(ap, fmt);\n> -\tstrbuf_vaddf(&sb, fmt, ap);\n> -\tresult = options->error_func(options, oid, object_type,\n> +\tva_copy(ap_copy, ap);\n> +\tstrbuf_vaddf(&sb, fmt, ap_copy);\n> +\tresult = options->error_func(options, oid, object_type, refs_info,\n> \t\t\t\t msg_type, msg_id, sb.buf);\n> \tstrbuf_release(&sb);\n> \tva_end(ap);\n> @@ -260,6 +266,35 @@ static int report(struct fsck_options *options,\n> \treturn result;\n> }\n> \n> +__attribute__((format (printf, 5, 6)))\n> +static int report(struct fsck_options *options,\n> +\t\t const struct object_id *oid, enum object_type object_type,\n> +\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n> +{\n> +\tva_list ap;\n> +\tint result;\n> +\n> +\tva_start(ap, fmt);\n> +\tresult = fsck_vreport(options, oid, object_type, NULL, msg_id, fmt, ap);\n> +\tva_end(ap);\n> +\n> +\treturn result;\n> +}\n\nAs far as I can see, `report()` is now specific to reporting errors with\nobjects while `fsck_vreport()` is the generic part. Do we want to rename\nthe function to `fsck_report_object()` to clarify, or would that cause\ntoo much churn?\n\nHm. Seeing that we have 62 callsites of that function it may be too much\nchurn indeed.\n\n> +int fsck_refs_report(struct fsck_options *options,\n> +\t\t const struct object_id *oid,\n> +\t\t const struct fsck_refs_info *refs_info,\n> +\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n\nWould `fsck_report_ref()` be a better name?\n\nWhat is the intent of the `oid` field? Would it be set to the object ID\nthat a reference points to? What if the reference is a non-resolving\nsymbolic reference? I wonder whether we can just remove it.\n\nPatrick\n"},{"id":"499580","messageId":"Zqik2RCfA6MAzzXx@tanuki","threadId":"61565","inReplyTo":"ZqeYkP5YiISbMrrN@ArchLinux","subject":"Re: [GSoC][PATCH v13 03/10] fsck: rename objects-related fsck error functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-07-30T08:31:21Z","receivedAt":"2024-07-30T08:31:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jul 29, 2024 at 09:26:40PM +0800, shejialuo wrote:\n> The names of objects-related fsck error functions are general. It's OK\n\ns/general/generic, I guess.\n\n> when there is only object database check. However, we have introduced\n> refs database check report function. To avoid ambiguity, rename\n> object-related fsck error functions to explicitly indicate these\n> functions are used to report objects-related messages.\n\nI agree that it is sensible to rename the functions to make it more\nexplicit that they are only used by the object-checking infrastructure.\n\nPatrick\n"},{"id":"499581","messageId":"Zqik3kQivsfetIQj@tanuki","threadId":"61565","inReplyTo":"ZqeYm2J1LeXn_1-4@ArchLinux","subject":"Re: [GSoC][PATCH v13 04/10] fsck: add refs-related error report function","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-07-30T08:31:26Z","receivedAt":"2024-07-30T08:31:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jul 29, 2024 at 09:26:51PM +0800, shejialuo wrote:\n> Add refs-related options to the \"fsck_options\", create refs-specific\n> \"error_func\" callback \"fsck_refs_error_function\".\n\nWe should have an explanation _why_ we are adding these functions in the\ncommit message.\n\n> \"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\n> passes the oid, it will use \"oid_to_hex\" to get the corresponding hex\n> value to report to the caller.\n> \n> Last, add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\"\n> macros to create refs options easily.\n\nIt is a bit unclear to me what you mean with \"create refs options\neasily\". Do you mean to say that `git refs check` (or whatever this will\nbe called) will have flags like \"--strict\"?\n\n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n> fsck.c | 25 +++++++++++++++++++++++++\n> fsck.h | 14 ++++++++++++++\n> 2 files changed, 39 insertions(+)\n> \n> diff --git a/fsck.c b/fsck.c\n> index af61fa90ba..56de29b4c0 100644\n> --- a/fsck.c\n> +++ b/fsck.c\n> @@ -1251,6 +1251,31 @@ int fsck_objects_error_function(struct fsck_options *o,\n> \treturn 1;\n> }\n> \n> +int fsck_refs_error_function(struct fsck_options *options,\n> +\t\t\t const struct object_id *oid,\n> +\t\t\t enum object_type object_type UNUSED,\n> +\t\t\t const struct fsck_refs_info *refs_info,\n> +\t\t\t enum fsck_msg_type msg_type,\n> +\t\t\t enum fsck_msg_id msg_id UNUSED,\n> +\t\t\t const char *message)\n> +{\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\tint ret = 0;\n> +\n> +\tstrbuf_addstr(&sb, refs_info->path);\n> +\n> +\tif (oid)\n> +\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n\nOkay, so we do end up printing the object ID indeed. But wouldn't we\nwant to potentially do the same with symbolic refs?\n\nAlso, would it make more sense to put the `oid` (and potentially the\n`referent` when we also handle symbolic refs) into `struct\nfsck_refs_info`? Like this, the whole state would be self-contained in\nthat structure, which would also make my proposal from a preceding\ncommit more feasible where the subsystem-specific error functions only\nget a void pointer to this structure. It would require another\nrefactoring on top to move the object type and OID into a `struct\nfsck_objects_info`, too, but that shouldn't be too involved, I guess.\n\nPatrick\n"},{"id":"499582","messageId":"Zqik5Lwya6bvxxl1@tanuki","threadId":"61565","inReplyTo":"ZqeYptLyOxAfKwDi@ArchLinux","subject":"Re: [GSoC][PATCH v13 05/10] refs: set up ref consistency check infrastructure","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-07-30T08:31:32Z","receivedAt":"2024-07-30T08:31:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jul 29, 2024 at 09:27:02PM +0800, shejialuo wrote:\n> The interfaces defined in the `ref_storage_be` are carefully structured\n> in semantic. It's organized as the five parts:\n> \n> 1. The name and the initialization interfaces.\n> 2. The ref transaction interfaces.\n> 3. The ref internal interfaces (pack, rename and copy).\n> 4. The ref filesystem interfaces.\n> 5. The reflog related interfaces.\n> \n> To keep consistent with the git-fsck(1), add a new interface named\n> \"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\n> grouped into any above five categories. Explicitly add blank line to\n> make it different from others.\n> \n> Last, implement placeholder functions for each ref backends.\n\nYou're carefully explaining what you are doing and where you are placing\nthe new callback functions. But you never explain why you add those\nfunctions in the first place, which I would think is much more important\nthan explaining the placement of the new callbacks.\n\nOther than that this patch looks good to me.\n\nPatrick\n"},{"id":"499583","messageId":"Zqik6cgm_6HYVUKy@tanuki","threadId":"61565","inReplyTo":"ZqeYsNtl90N1fVDy@ArchLinux","subject":"Re: [GSoC][PATCH v13 06/10] git refs: add verify subcommand","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-07-30T08:31:37Z","receivedAt":"2024-07-30T08:31:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jul 29, 2024 at 09:27:12PM +0800, shejialuo wrote:\n\nThe subject should probably start with \"builtin/refs\", not \"git refs\".\n\n> Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\n> check the reference database consistency and also this subcommand will\n> be used as the entry point of checking refs for \"git-fsck(1)\". Last, add\n> \"verbose\" field into \"fsck_options\" to indicate whether we should print\n> verbose messages when checking refs and objects consistency.\n\nNice. I very much like that we now have a common home for such low-level\nref-related commands. Also, \"verify\" is neatly in line with e.g. `git\ncommit-graph verify\".\n\n> @@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n> \tcan be used to double check that the migration works as expected before\n> \tperforming the actual migration.\n> \n> +The following options are specific to 'git refs verify':\n> +\n> +--strict::\n> +\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n> +\tbe seen as ERROR. See linkgit:git-fsck[1].\n\nHow about:\n\n \"Enable stricter error checking. This will cause warnings to be\n reported as errors. See linkgit:git-fsck[1].\"\n\n> +--verbose::\n> +\tWhen verifying the reference database consistency, be chatty.\n\nI wonder whether this really helps all that much. It doesn't really say\nwhat it adds on top of the default mode. So unless we document what\nexactly this changes, I rather think we can just leave it aways as\nbasically everyone knows what a \"--verbose\" flag does.\n\n> KNOWN LIMITATIONS\n> -----------------\n> \n> diff --git a/builtin/refs.c b/builtin/refs.c\n> index 46dcd150d4..4831c9e28e 100644\n> --- a/builtin/refs.c\n> +++ b/builtin/refs.c\n> @@ -1,4 +1,6 @@\n> #include \"builtin.h\"\n> +#include \"config.h\"\n> +#include \"fsck.h\"\n> #include \"parse-options.h\"\n> #include \"refs.h\"\n> #include \"repository.h\"\n> @@ -7,6 +9,9 @@\n> #define REFS_MIGRATE_USAGE \\\n> \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n> \n> +#define REFS_VERIFY_USAGE \\\n> +\tN_(\"git refs verify [--strict] [--verbose]\")\n> +\n> static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n> {\n> \tconst char * const migrate_usage[] = {\n> @@ -58,15 +63,54 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n> \treturn err;\n> }\n> \n> +static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n> +{\n> +\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n\nSo we don't ever end up using `FSCK_REFS_OPTIONS_STRICT`? If so, I think\nwe should just drop that declaration in the preceding patch.\n\n> +\tconst char * const verify_usage[] = {\n> +\t\tREFS_VERIFY_USAGE,\n> +\t\tNULL,\n> +\t};\n> +\tunsigned int verbose = 0, strict = 0;\n> +\tstruct option options[] = {\n> +\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n> +\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n> +\t\tOPT_END(),\n> +\t};\n> +\tint ret;\n> +\n> +\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n> +\tif (argc)\n> +\t\tusage(_(\"'git refs verify' takes no arguments\"));\n> +\n> +\tif (verbose)\n> +\t\tfsck_refs_options.verbose = 1;\n> +\tif (strict)\n> +\t\tfsck_refs_options.strict = 1;\n\nInstead of manually setting those variables, we can pass pointers to\nthose member variables in the `struct option`s directly.\n\n> +\tgit_config(git_fsck_config, &fsck_refs_options);\n> +\tprepare_repo_settings(the_repository);\n> +\n> +\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n> +\n> +\t/*\n> +\t * Explicitly free the allocated array and \"skip_oids\" set\n> +\t */\n> +\tfree(fsck_refs_options.msg_type);\n> +\toidset_clear(&fsck_refs_options.skip_oids);\n\nShould we provide a `fsck_options_clear()` function that does this for\nus? Otherwise we'll have to adapt callsites of `refs_fsck` whenever\ninternal implementation details of the subsystem add newly allocated\nmembers.\n\n> +\treturn ret;\n> +}\n> +\n> int cmd_refs(int argc, const char **argv, const char *prefix)\n> {\n> \tconst char * const refs_usage[] = {\n> \t\tREFS_MIGRATE_USAGE,\n> +\t\tREFS_VERIFY_USAGE,\n> \t\tNULL,\n> \t};\n> \tparse_opt_subcommand_fn *fn = NULL;\n> \tstruct option opts[] = {\n> \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n> +\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n> \t\tOPT_END(),\n> \t};\n> \n> diff --git a/fsck.h b/fsck.h\n> index a4a4ba88ee..b03dba442e 100644\n> --- a/fsck.h\n> +++ b/fsck.h\n> @@ -155,6 +155,7 @@ struct fsck_options {\n> \tfsck_walk_func walk;\n> \tfsck_error error_func;\n> \tunsigned strict:1;\n> +\tunsigned verbose:1;\n\nOkay. Let's see whether this field will be used in a subsequent patch.\nIf not, we should drop it and get rid of the option altogether, I guess.\n\nPatrick\n"},{"id":"499584","messageId":"Zqik73qynJvOLctw@tanuki","threadId":"61565","inReplyTo":"ZqeYw-k-MzhPTNRf@ArchLinux","subject":"Re: [GSoC][PATCH v13 08/10] files-backend: add unified interface for refs scanning","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-07-30T08:31:43Z","receivedAt":"2024-07-30T08:31:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jul 29, 2024 at 09:27:31PM +0800, shejialuo wrote:\n> For refs and reflogs, we need to scan its corresponding directories to\n> check every regular file or symbolic link which shares the same pattern.\n> Introduce a unified interface for scanning directories for\n> files-backend.\n> \n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n> refs/files-backend.c | 74 +++++++++++++++++++++++++++++++++++++++++++-\n> 1 file changed, 73 insertions(+), 1 deletion(-)\n> \n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 4630eb1f80..cb184953c1 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -6,6 +6,7 @@\n> #include \"../gettext.h\"\n> #include \"../hash.h\"\n> #include \"../hex.h\"\n> +#include \"../fsck.h\"\n> #include \"../refs.h\"\n> #include \"refs-internal.h\"\n> #include \"ref-cache.h\"\n> @@ -3408,13 +3409,84 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n> \treturn ret;\n> }\n> \n> +/*\n> + * For refs and reflogs, they share a unified interface when scanning\n> + * the whole directory. This function is used as the callback for each\n> + * regular file or symlink in the directory.\n> + */\n> +typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n> +\t\t\t\t const char *gitdir,\n> +\t\t\t\t const char *refs_check_dir,\n> +\t\t\t\t struct dir_iterator *iter);\n> +\n> +static int files_fsck_refs_dir(struct ref_store *ref_store,\n> +\t\t\t struct fsck_options *o,\n> +\t\t\t const char *refs_check_dir,\n> +\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n> +{\n> +\tconst char *gitdir = ref_store->gitdir;\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\tstruct dir_iterator *iter;\n> +\tint iter_status;\n> +\tint ret = 0;\n> +\n> +\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n> +\n> +\titer = dir_iterator_begin(sb.buf, 0);\n> +\n> +\tif (!iter) {\n> +\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n> +\t\tgoto out;\n> +\t}\n\nThe error message should probably be marked as translatable. Also, I'd\npersonally remove the newline between `iter = ...` and the error check\nas those are a logical unit.\n\n> +\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n> +\t\tif (S_ISDIR(iter->st.st_mode)) {\n> +\t\t\tcontinue;\n> +\t\t} else if (S_ISREG(iter->st.st_mode) ||\n> +\t\t\t S_ISLNK(iter->st.st_mode)) {\n> +\t\t\tif (o->verbose)\n> +\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n> +\t\t\t\t\t refs_check_dir, iter->relative_path);\n\nOkay, we do end up using the `verbose` flag :)\n\n> +\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n> +\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n> +\t\t\t\t\tret = -1;\n> +\t\t\t}\n> +\t\t} else {\n> +\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n> +\t\t\t\t iter->basename);\n\nInstead of printing this as an error directly, shouldn't we report it\nvia the `fsck_refs_report` interface?\n\n> +\t\t}\n> +\t}\n\nOkay. It does make sense to do our own directory walk as that will allow\nus to check files which would otherwise not be reported by the normal\nrefs interfaces.\n\n> +\tif (iter_status != ITER_DONE)\n> +\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n\nReporting this as an error feels sensible though as we have no ref to\ntie this error to, and it feels like a generic error.\n\n> +out:\n> +\tstrbuf_release(&sb);\n> +\treturn ret;\n> +}\n> +\n> +static int files_fsck_refs(struct ref_store *ref_store,\n> +\t\t\t struct fsck_options *o)\n> +{\n> +\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n> +\t\tNULL\n\nThe last member should also end with a comma.\n\n> +\t};\n> +\n> +\tif (o->verbose)\n> +\t\tfprintf_ln(stderr, \"Checking references consistency\");\n> +\n> +\treturn files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n> +\n\nThis newline should be removed.\n\n> +}\n> +\n> static int files_fsck(struct ref_store *ref_store,\n> \t\t struct fsck_options *o)\n> {\n> \tstruct files_ref_store *refs =\n> \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n> \n> -\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n> +\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o) |\n> +\t files_fsck_refs(ref_store, o);\n\nI'd think we should first check loose files and then continue to check\nthe packed ref store. That's really only a gut feeling though, and I\ncannot exactly say why that feels more natural.\n\nPatrick\n"},{"id":"499585","messageId":"Zqik9EOKd43WxGXp@tanuki","threadId":"61565","inReplyTo":"ZqeY0eHNZjKhNvIH@ArchLinux","subject":"Re: [GSoC][PATCH v13 09/10] fsck: add ref name check for files backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-07-30T08:31:48Z","receivedAt":"2024-07-30T08:31:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jul 29, 2024 at 09:27:45PM +0800, shejialuo wrote:\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index cb184953c1..0d4fc27768 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -3419,6 +3419,27 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n> \t\t\t\t const char *refs_check_dir,\n> \t\t\t\t struct dir_iterator *iter);\n> \n> +static int files_fsck_refs_name(struct fsck_options *o,\n> +\t\t\t\tconst char *gitdir UNUSED,\n> +\t\t\t\tconst char *refs_check_dir,\n> +\t\t\t\tstruct dir_iterator *iter)\n> +{\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\tstruct fsck_refs_info info;\n> +\tint ret = 0;\n> +\n> +\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n> +\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n> +\t\tinfo.path = sb.buf;\n> +\t\tret = fsck_refs_report(o, NULL, &info,\n> +\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n> +\t\t\t\t \"invalid refname format\");\n> +\t}\n> +\n> +\tstrbuf_release(&sb);\n> +\treturn ret;\n> +}\n> +\n> static int files_fsck_refs_dir(struct ref_store *ref_store,\n> \t\t\t struct fsck_options *o,\n> \t\t\t const char *refs_check_dir,\n> @@ -3469,6 +3490,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n> \t\t\t struct fsck_options *o)\n> {\n> \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n> +\t\tfiles_fsck_refs_name,\n> \t\tNULL\n\nNeat. I very much like that we can simply add new checks to this\nfunction and the rest is handled for us already. Makes this whole thing\nnicely extensible.\n\n> \t};\n> \n> diff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\n> new file mode 100755\n> index 0000000000..b2db58d2c6\n> --- /dev/null\n> +++ b/t/t0602-reffiles-fsck.sh\n> @@ -0,0 +1,101 @@\n> +#!/bin/sh\n> +\n> +test_description='Test reffiles backend consistency check'\n> +\n> +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n> +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n> +GIT_TEST_DEFAULT_REF_FORMAT=files\n> +export GIT_TEST_DEFAULT_REF_FORMAT\n> +\n> +. ./test-lib.sh\n\nIs this test suite intentionally not marked with\n`TEST_PASSES_SANITIZE_LEAK=true`?\n\n> +\n> +test_expect_success 'ref name should be checked' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\tbranch_dir_prefix=.git/refs/heads &&\n> +\ttag_dir_prefix=.git/refs/tags &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit commit --allow-empty -m initial &&\n> +\t\tgit checkout -b branch-1 &&\n> +\t\tgit tag tag-1 &&\n> +\t\tgit commit --allow-empty -m second &&\n> +\t\tgit checkout -b branch-2 &&\n> +\t\tgit tag tag-2 &&\n> +\t\tgit tag multi_hierarchy/tag-2\n> +\t) &&\n\nI don't quite get why you create several subshells only to cd into\n`repo` in each of them. Isn't a single subshell sufficient for all of\nthose tests? If you want to delimit blocks, then you can simply add an\nempty newline between them.\n\n> +\t(\n> +\t\tcd repo &&\n> +\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n> +\t\ttest_must_fail git fsck 2>err &&\n> +\t\tcat >expect <<-EOF &&\n> +\t\terror: refs/heads/.branch-1: badRefName: invalid refname format\n> +\t\tEOF\n> +\t\trm $branch_dir_prefix/.branch-1 &&\n> +\t\ttest_cmp expect err\n> +\t) &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n> +\t\ttest_must_fail git fsck 2>err &&\n> +\t\tcat >expect <<-EOF &&\n> +\t\terror: refs/tags/tag-1.lock: badRefName: invalid refname format\n> +\t\tEOF\n> +\t\trm $tag_dir_prefix/tag-1.lock &&\n> +\t\ttest_cmp expect err\n> +\t) &&\n\nThe other cases all make sense, but I don't think that a file ending\nwith \".lock\" should be marked as having a \"badRefName\". It is expected\nthat concurrent writers may have such lock files.\n\nWhat could make sense is to eventually mark stale lock files older than\nX amount of time as errors or warnings. But I'd think that this is\noutside of the scope of this patch series.\n\nPatrick\n"},{"id":"499586","messageId":"Zqik-iqO4-yNuIn7@tanuki","threadId":"61565","inReplyTo":"ZqeY3Dhj-Fo-bZ2k@ArchLinux","subject":"Re: [GSoC][PATCH v13 10/10] fsck: add ref content check for files backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-07-30T08:31:54Z","receivedAt":"2024-07-30T08:31:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jul 29, 2024 at 09:27:56PM +0800, shejialuo wrote:\n> Enhance the git-fsck(1) command by adding a check for reference content\n> in the files backend. The new functionality ensures that symrefs, real\n> symbolic link and regular refs are validated correctly.\n> \n> In order to check the trailing content of the regular refs, add a new\n> parameter `trailing` to `parse_loose_ref_contents`.\n> \n> For symrefs, `parse_loose_ref_contents` will set the \"referent\".\n> However, symbolic link could be either absolute or relative. Use\n> \"strbuf_add_real_path\" to read the symbolic link and convert the\n> relative path to absolute path. Then use \"skip_prefix\" to make it align\n> with symref \"referent\".\n> \n> Thus, the symrefs and symbolic links could share the same interface. Add\n> a new function \"files_fsck_symref_target\" which aims at checking the\n> following things:\n> \n> 1. whether the pointee is under the `refs/` directory.\n> 2. whether the pointee name is correct.\n> 3. whether the pointee path is a wrong type in filesystem.\n> \n> Last, add the following FSCK MESSAGEs:\n> \n> 1. \"badRefContent(ERROR)\": A ref has a bad content\n> 2. \"badSymrefPointee(ERROR)\": The pointee of a symref is bad.\n> 3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n\nI think it would have been fine to stop at the preceding commit as it\nclearly demonstrates how the whole infrastructure is supposed to work.\nAdditional checks like those you add here would then be a good candidate\nfor a separate patch series. This would help you get the first patch\nseries landed faster as you really only have to focus on setting up the\nbaseline infrastructure.\n\nFeel free to keep or drop this patch as you prefer though, I don't want\nto discourage you aiming higher. Just keep in mind that the more you add\non top the longer it takes to land a patch series :)\n\n> diff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\n> index d8e437a043..8fe24a960e 100644\n> --- a/Documentation/fsck-msgids.txt\n> +++ b/Documentation/fsck-msgids.txt\n> @@ -19,9 +19,15 @@\n> `badParentSha1`::\n> \t(ERROR) A commit object has a bad parent sha1.\n> \n> +`badRefContent`::\n> +\t(ERROR) A ref has a bad content.\n> +\n\ns/a bad/bad\n\n> +static int files_fsck_refs_content(struct fsck_options *o,\n> +\t\t\t\t const char *gitdir,\n> +\t\t\t\t const char *refs_check_dir,\n> +\t\t\t\t struct dir_iterator *iter)\n> +{\n> +\tstruct strbuf pointee_path = STRBUF_INIT,\n> +\t\t ref_content = STRBUF_INIT,\n> +\t\t abs_gitdir = STRBUF_INIT,\n> +\t\t referent = STRBUF_INIT,\n> +\t\t refname = STRBUF_INIT;\n\nNit: I think it's more customary to start each of the lines with `struct\nstrbuf`. Not a 100% certain on this one, though.\n\n> +\tconst char *trailing = NULL;\n> +\tstruct fsck_refs_info info;\n> +\tint failure_errno = 0;\n> +\tunsigned int type = 0;\n> +\tstruct object_id oid;\n> +\tint ret = 0;\n> +\n> +\tstrbuf_addf(&refname, \"%s/%s\", refs_check_dir, iter->relative_path);\n> +\tinfo.path = refname.buf;\n> +\n> +\t/*\n> +\t * If the file is a symlink, we need to only check the connectivity\n> +\t * of the destination object.\n> +\t */\n> +\tif (S_ISLNK(iter->st.st_mode)) {\n> +\t\tconst char *pointee_name = NULL;\n> +\n> +\t\tstrbuf_add_real_path(&pointee_path, iter->path.buf);\n> +\n> +\t\tstrbuf_add_absolute_path(&abs_gitdir, gitdir);\n> +\t\tstrbuf_normalize_path(&abs_gitdir);\n> +\t\tif (!is_dir_sep(abs_gitdir.buf[abs_gitdir.len - 1]))\n> +\t\t\tstrbuf_addch(&abs_gitdir, '/');\n> +\n> +\t\tif (!skip_prefix(pointee_path.buf,\n> +\t\t\t\t abs_gitdir.buf, &pointee_name)) {\n> +\t\t\tret = fsck_refs_report(o, NULL, &info,\n> +\t\t\t\t\t FSCK_MSG_BAD_SYMREF_POINTEE,\n> +\t\t\t\t\t \"point to target outside gitdir\");\n> +\t\t\tgoto clean;\n> +\t\t}\n> +\n> +\t\tret = files_fsck_symref_target(o, &info, refname.buf,\n> +\t\t\t\t\t pointee_name, pointee_path.buf);\n> +\t\tgoto clean;\n> +\t}\n> +\n> +\tif (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) {\n> +\t\tret = error_errno(_(\"%s/%s: unable to read the ref\"),\n> +\t\t\t\t refs_check_dir, iter->relative_path);\n> +\t\tgoto clean;\n> +\t}\n> +\n> +\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n> +\t\t\t\t &referent, &type,\n> +\t\t\t\t &failure_errno, &trailing)) {\n> +\t\tret = fsck_refs_report(o, NULL, &info,\n> +\t\t\t\t FSCK_MSG_BAD_REF_CONTENT,\n> +\t\t\t\t \"invalid ref content\");\n> +\t\tgoto clean;\n> +\t}\n> +\n> +\t/*\n> +\t * If the ref is a symref, we need to check the destination name and\n> +\t * connectivity.\n> +\t */\n> +\tif (referent.len && (type & REF_ISSYMREF)) {\n> +\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n> +\t\tstrbuf_rtrim(&referent);\n> +\n> +\t\tret = files_fsck_symref_target(o, &info, refname.buf,\n> +\t\t\t\t\t referent.buf, pointee_path.buf);\n> +\t\tgoto clean;\n> +\t} else {\n> +\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n\nIn case the ref ends with a newline, should we check that the next\ncharacter is `\\0`? Otherwise, it may contain multiple lines, which is\nnot allowed for a normal ref.\n\nAlso, shouldn't the ref always end with a newline?\n\nPatrick\n"},{"id":"499587","messageId":"Zqik_yal_hPTlotg@tanuki","threadId":"61565","inReplyTo":"ZqeXrPROpEg_pRS2@ArchLinux","subject":"Re: [GSoC][PATCH v13 00/10] ref consistency check infra setup","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-07-30T08:31:59Z","receivedAt":"2024-07-30T08:32:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jul 29, 2024 at 09:22:52PM +0800, shejialuo wrote:\n> Hi All:\n> \n> This version mainly focuses on the problem about how should we provide\n> extensibility for reporting the problem of refs. In patch v10, Junio\n> asked a question here:\n> \n> > The error reporting function for refs consistency check was still\n> > about reporting a problem for a single ref. I am wondering how\n> > consistency violations that are not about a single ref should be\n> > handled. For example, if refs/packed-backend.c:packed_fsck() finds\n> > that the file is not sorted properly or has some unparseable garbage\n> > in it, it is not something you can report as \"refs/heads/main is\n> > broken\", but those who are interested in seeing the \"reference\n> > database consistency\" verified, it is very much what they want the\n> > tool to notice. How would detection of such a breakage that is not\n> > attributed to a single ref fit in this \"ref consistency check\n> > infrastructure\" that was introduced by [05/10]?\n> \n> Actually, I think that the original parameter \"checked_ref_name\" is a\n> bad name which makes the reader think that \"we only handle refs\". And\n> this is my fault for this design. However, I misunderstood the Junio's\n> word and made things complicated.\n> \n> The patch v11 and v12 wants to solve a problem that we should provide\n> extensibility for reporting refs problem. However, these two versions\n> still made things complicated. After an offline meeting with Patrick and\n> Karthik, we design the following simple flat data structure:\n> \n> struct fsck_refs_info {\n> const char *path;\n> };\n> \n> It is simple and provides extensibility.\n\nI've got a bunch of comments, mostly because I've been out for quite a\nwhile and thus didn't follow the progression of this series. But don't\nlet yourself be discouraged by the volume, I quite like the shape of\nthis patch series and think that it is very close to what we actually\nwant to have.\n\nI'd say that this series is likely to become ready soonish.\n\nThanks!\n\nPatrick\n"},{"id":"499612","messageId":"Zqj_FmazX6SW98Eq@ArchLinux","threadId":"61565","inReplyTo":"Zqik1JFs-RYCwNnz@tanuki","subject":"Re: [GSoC][PATCH v13 02/10] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-30T14:56:22Z","receivedAt":"2024-07-30T14:56:03Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jul 30, 2024 at 10:31:16AM +0200, Patrick Steinhardt wrote:\n> On Mon, Jul 29, 2024 at 09:26:30PM +0800, shejialuo wrote:\n> > The static function \"report\" provided by \"fsck.c\" aims at checking fsck\n> > error type and calling the callback \"error_func\" to report the message.\n> > However, \"report\" function is only related to object database which\n> > cannot be reused for refs.\n> \n> Nit: it would be nice to mention _why_ it cannot be reused for refs.\n> \n> > diff --git a/fsck.c b/fsck.c\n> > index 3f32441492..1185e9a8ad 100644\n> > --- a/fsck.c\n> > +++ b/fsck.c\n> > @@ -226,12 +226,18 @@ static int object_on_skiplist(struct fsck_options *opts,\n> > \treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n> > }\n> > \n> > -__attribute__((format (printf, 5, 6)))\n> > -static int report(struct fsck_options *options,\n> > -\t\t const struct object_id *oid, enum object_type object_type,\n> > -\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n> > +/*\n> > + * Provide a unified interface for either fscking refs or objects.\n> > + * It will get the current msg error type and call the error_func callback\n> > + * which is registered in the \"fsck_options\" struct.\n> > + */\n> > +static int fsck_vreport(struct fsck_options *options,\n> > +\t\t\tconst struct object_id *oid,\n> > +\t\t\tenum object_type object_type,\n> > +\t\t\tconst struct fsck_refs_info *refs_info,\n> > +\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n> > {\n> > -\tva_list ap;\n> > +\tva_list ap_copy;\n> > \tstruct strbuf sb = STRBUF_INIT;\n> > \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n> > \tint result;\n> \n> It is a bit weird that this new generic function receives non-generic\n> inputs which are specific to the respective subsystems (objects or refs)\n> that we are checking.\n> \n\nActually, this is one of the biggest problem when implementing the\ninfrastructure. The original function \"report\" only cares about\nreporting the problem of objects. So the callback \"error_func\" uses the\nsimilar prototype.\n\nProblem comes when we want to add ref-related report. In my very former\nimplementation, I just created a new function \"fsck_refs_report\" to just\ncopy some codes from \"report\" and defines refs-related callback.\n\nHowever, this is a bad way because we make duplication. If we want to\nreuse the \"report\" function, we should add new parameters into \"report\"\nand \"error_func\". This is the idea of this patch. However, as you can\nsee, there are so many \"report\" function calls in the codebase, it's bad\nto change them. So I define a more common function called \"fsck_vreport\"\nfunction and wrap \"report\" to eventually call this function.\n\n> A better design would likely be to make `error_func()` receive a void\n> pointer such that `error_func()` and then have the respective subsystems\n> provide a function that knows to format the message while receiving\n> either a `struct fsck_object_report *` or a `struct fsck_ref_report *`.\n> \n\nYes, I agree with this idea. And I think we should use only one function\ncalled \"fsck_reportf\" to report any fsck-related messages. We could\ndesign the following callback \"prototype\".\n\n typedef int (*fsck_error)(struct fsck_options *o,\n void *info,\n enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n const char *message);\n\nThus, we could make \"fsck_reportf\" generic. It will handle the common\n\"fsck_options\" and \"enum fsck_msg_id\" and then it will call \"fsck_error\"\ncallback. The user could pass either refs information or objects\ninformation.\n\n> I don't think this is particularly worriesome though as it is still\n> manageable right now. So I'm fine if we want to leave this as-is, and\n> then we can iterate on this in a future patch series as required.\n> \n\nI strongly suggest that we should use the above design for the following\nreasons:\n\n1. We only expose one interface called \"fsck_reportf\" which will make\nthe code clear. Actually, there is no different between reporting refs\nand reporting objects.\n2. We provide more extensibility here, because we will never change\n\"fsck_reportf\" and \"fsck_error\" prototype when we want to add more info\nfor either refs or objects.\n\nBut do we really need this? Junio, could you please give some advice\nhere. How do you think about this design. In my perspective, the only\noverhead here is that there are too many \"report\" function we should\nrefactor.\n\n> > @@ -250,9 +256,9 @@ static int report(struct fsck_options *options,\n> > \tprepare_msg_ids();\n> > \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n> > \n> > -\tva_start(ap, fmt);\n> > -\tstrbuf_vaddf(&sb, fmt, ap);\n> > -\tresult = options->error_func(options, oid, object_type,\n> > +\tva_copy(ap_copy, ap);\n> > +\tstrbuf_vaddf(&sb, fmt, ap_copy);\n> > +\tresult = options->error_func(options, oid, object_type, refs_info,\n> > \t\t\t\t msg_type, msg_id, sb.buf);\n> > \tstrbuf_release(&sb);\n> > \tva_end(ap);\n> > @@ -260,6 +266,35 @@ static int report(struct fsck_options *options,\n> > \treturn result;\n> > }\n> > \n> > +__attribute__((format (printf, 5, 6)))\n> > +static int report(struct fsck_options *options,\n> > +\t\t const struct object_id *oid, enum object_type object_type,\n> > +\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n> > +{\n> > +\tva_list ap;\n> > +\tint result;\n> > +\n> > +\tva_start(ap, fmt);\n> > +\tresult = fsck_vreport(options, oid, object_type, NULL, msg_id, fmt, ap);\n> > +\tva_end(ap);\n> > +\n> > +\treturn result;\n> > +}\n> \n> As far as I can see, `report()` is now specific to reporting errors with\n> objects while `fsck_vreport()` is the generic part. Do we want to rename\n> the function to `fsck_report_object()` to clarify, or would that cause\n> too much churn?\n> \n> Hm. Seeing that we have 62 callsites of that function it may be too much\n> churn indeed.\n> \n\nYes, there are too many references for \"report\" function. That's why I\nwrap the \"report\" using \"fsck_vreport\".\n\n> > +int fsck_refs_report(struct fsck_options *options,\n> > +\t\t const struct object_id *oid,\n> > +\t\t const struct fsck_refs_info *refs_info,\n> > +\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n> \n> Would `fsck_report_ref()` be a better name?\n> \n\nI agree. However, if we use the above design, we will just use\n\"fsck_reportf\" here both for refs and objects.\n\n> What is the intent of the `oid` field? Would it be set to the object ID\n> that a reference points to? What if the reference is a non-resolving\n> symbolic reference? I wonder whether we can just remove it.\n> \n\n`oid` is used to be the object ID that a reference points to. If the\nreference is a symbolic link or symref, we do not care about it. The\ncaller should just pass `NULL`. Actually, we may not use this field.\nI just suppose that we may provide the user more information. Because\nwhen using \"file-backend.c::parse_loose_ref_contents()\" we will\nautomatically get the `oid` if the ref is a regular reference. So I just\nprovide `oid` here.\n\n> Patrick\n"},{"id":"499613","messageId":"Zqj_zITuG-gsNM2R@ArchLinux","threadId":"61565","inReplyTo":"Zqik2RCfA6MAzzXx@tanuki","subject":"Re: [GSoC][PATCH v13 03/10] fsck: rename objects-related fsck error functions","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-30T14:59:24Z","receivedAt":"2024-07-30T14:59:05Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jul 30, 2024 at 10:31:21AM +0200, Patrick Steinhardt wrote:\n> On Mon, Jul 29, 2024 at 09:26:40PM +0800, shejialuo wrote:\n> > The names of objects-related fsck error functions are general. It's OK\n> \n> s/general/generic, I guess.\n> \n\nI will improve this in the next version.\n\n"},{"id":"499614","messageId":"ZqkCH8BEE1mX9zfP@ArchLinux","threadId":"61565","inReplyTo":"Zqik3kQivsfetIQj@tanuki","subject":"Re: [GSoC][PATCH v13 04/10] fsck: add refs-related error report function","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-30T15:09:19Z","receivedAt":"2024-07-30T15:08:59Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jul 30, 2024 at 10:31:26AM +0200, Patrick Steinhardt wrote:\n> On Mon, Jul 29, 2024 at 09:26:51PM +0800, shejialuo wrote:\n> > Add refs-related options to the \"fsck_options\", create refs-specific\n> > \"error_func\" callback \"fsck_refs_error_function\".\n> \n> We should have an explanation _why_ we are adding these functions in the\n> commit message.\n> \n\nYes, I will improve this in the next version.\n\n> > \"fsck_refs_error_function\" will use the \"oid\" parameter. When the caller\n> > passes the oid, it will use \"oid_to_hex\" to get the corresponding hex\n> > value to report to the caller.\n> > \n> > Last, add \"FSCK_REFS_OPTIONS_DEFAULT\" and \"FSCK_REFS_OPTIONS_STRICT\"\n> > macros to create refs options easily.\n> \n> It is a bit unclear to me what you mean with \"create refs options\n> easily\". Do you mean to say that `git refs check` (or whatever this will\n> be called) will have flags like \"--strict\"?\n> \n\nYes, when the user passes `--strict`, all the warn type will be seen as\nthe error type. So I create \"FSCK_REFS_OPTIONS_STRICT\". However, I\ndidn't think too much here. I just followed the way the codebase does\nfor the objects.\n\n> > Mentored-by: Patrick Steinhardt \n> > Mentored-by: Karthik Nayak \n> > Signed-off-by: shejialuo \n> > ---\n> > fsck.c | 25 +++++++++++++++++++++++++\n> > fsck.h | 14 ++++++++++++++\n> > 2 files changed, 39 insertions(+)\n> > \n> > diff --git a/fsck.c b/fsck.c\n> > index af61fa90ba..56de29b4c0 100644\n> > --- a/fsck.c\n> > +++ b/fsck.c\n> > @@ -1251,6 +1251,31 @@ int fsck_objects_error_function(struct fsck_options *o,\n> > \treturn 1;\n> > }\n> > \n> > +int fsck_refs_error_function(struct fsck_options *options,\n> > +\t\t\t const struct object_id *oid,\n> > +\t\t\t enum object_type object_type UNUSED,\n> > +\t\t\t const struct fsck_refs_info *refs_info,\n> > +\t\t\t enum fsck_msg_type msg_type,\n> > +\t\t\t enum fsck_msg_id msg_id UNUSED,\n> > +\t\t\t const char *message)\n> > +{\n> > +\tstruct strbuf sb = STRBUF_INIT;\n> > +\tint ret = 0;\n> > +\n> > +\tstrbuf_addstr(&sb, refs_info->path);\n> > +\n> > +\tif (oid)\n> > +\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(oid));\n> \n> Okay, so we do end up printing the object ID indeed. But wouldn't we\n> want to potentially do the same with symbolic refs?\n> \n> Also, would it make more sense to put the `oid` (and potentially the\n> `referent` when we also handle symbolic refs) into `struct\n> fsck_refs_info`? Like this, the whole state would be self-contained in\n> that structure, which would also make my proposal from a preceding\n> commit more feasible where the subsystem-specific error functions only\n> get a void pointer to this structure. It would require another\n> refactoring on top to move the object type and OID into a `struct\n> fsck_objects_info`, too, but that shouldn't be too involved, I guess.\n> \n\nYes, I totally agree here. I didn't consider symrefs here, actually we\nshould provide \"symref -> referent\" message if we have providen the\n\"regular ref -> oid\" message.\n\nI think we should do this. And I have commented on \"[Patch v13][2/10]\",\nit is a necessity we should refactor this part.\n\n> Patrick\n"},{"id":"499615","messageId":"ZqkCds6C8adxrrIX@ArchLinux","threadId":"61565","inReplyTo":"Zqik5Lwya6bvxxl1@tanuki","subject":"Re: [GSoC][PATCH v13 05/10] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-30T15:10:46Z","receivedAt":"2024-07-30T15:10:27Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jul 30, 2024 at 10:31:32AM +0200, Patrick Steinhardt wrote:\n> On Mon, Jul 29, 2024 at 09:27:02PM +0800, shejialuo wrote:\n> > The interfaces defined in the `ref_storage_be` are carefully structured\n> > in semantic. It's organized as the five parts:\n> > \n> > 1. The name and the initialization interfaces.\n> > 2. The ref transaction interfaces.\n> > 3. The ref internal interfaces (pack, rename and copy).\n> > 4. The ref filesystem interfaces.\n> > 5. The reflog related interfaces.\n> > \n> > To keep consistent with the git-fsck(1), add a new interface named\n> > \"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\n> > grouped into any above five categories. Explicitly add blank line to\n> > make it different from others.\n> > \n> > Last, implement placeholder functions for each ref backends.\n> \n> You're carefully explaining what you are doing and where you are placing\n> the new callback functions. But you never explain why you add those\n> functions in the first place, which I would think is much more important\n> than explaining the placement of the new callbacks.\n> \n\nThanks, I will improve this in the next version.\n\n> Other than that this patch looks good to me.\n> \n> Patrick\n\n\n"},{"id":"499620","messageId":"ZqkN4wNK0PIRyNky@ArchLinux","threadId":"61565","inReplyTo":"Zqik6cgm_6HYVUKy@tanuki","subject":"Re: [GSoC][PATCH v13 06/10] git refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-30T15:59:31Z","receivedAt":"2024-07-30T15:59:11Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jul 30, 2024 at 10:31:37AM +0200, Patrick Steinhardt wrote:\n> On Mon, Jul 29, 2024 at 09:27:12PM +0800, shejialuo wrote:\n> \n> The subject should probably start with \"builtin/refs\", not \"git refs\".\n> \n\nYes, I will improve this in the next version.\n\n> > @@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n> > \tcan be used to double check that the migration works as expected before\n> > \tperforming the actual migration.\n> > \n> > +The following options are specific to 'git refs verify':\n> > +\n> > +--strict::\n> > +\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n> > +\tbe seen as ERROR. See linkgit:git-fsck[1].\n> \n> How about:\n> \n> \"Enable stricter error checking. This will cause warnings to be\n> reported as errors. See linkgit:git-fsck[1].\"\n> \n\nYes, it is much more clear. Actually, I really feel hard to write a good\ndocument.\n\n> > +--verbose::\n> > +\tWhen verifying the reference database consistency, be chatty.\n> \n> I wonder whether this really helps all that much. It doesn't really say\n> what it adds on top of the default mode. So unless we document what\n> exactly this changes, I rather think we can just leave it aways as\n> basically everyone knows what a \"--verbose\" flag does.\n> \n\nYes, I think so. `--verbose` is a common flag. However, we have already\nadded this, so we may just leave it here. It's not bad to add more\ninformation.\n\n> > +static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n> > +{\n> > +\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n> \n> So we don't ever end up using `FSCK_REFS_OPTIONS_STRICT`? If so, I think\n> we should just drop that declaration in the preceding patch.\n> \n\nI agree here. I will delete `FSCK_REFS_OPTIONS_STRICT`.\n\n> > +\tconst char * const verify_usage[] = {\n> > +\t\tREFS_VERIFY_USAGE,\n> > +\t\tNULL,\n> > +\t};\n> > +\tunsigned int verbose = 0, strict = 0;\n> > +\tstruct option options[] = {\n> > +\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n> > +\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n> > +\t\tOPT_END(),\n> > +\t};\n> > +\tint ret;\n> > +\n> > +\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n> > +\tif (argc)\n> > +\t\tusage(_(\"'git refs verify' takes no arguments\"));\n> > +\n> > +\tif (verbose)\n> > +\t\tfsck_refs_options.verbose = 1;\n> > +\tif (strict)\n> > +\t\tfsck_refs_options.strict = 1;\n> \n> Instead of manually setting those variables, we can pass pointers to\n> those member variables in the `struct option`s directly.\n> \n\nYes, but I have tried but found that the types are mismatching, I will\nfind a way to do this.\n\n> > +\tgit_config(git_fsck_config, &fsck_refs_options);\n> > +\tprepare_repo_settings(the_repository);\n> > +\n> > +\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n> > +\n> > +\t/*\n> > +\t * Explicitly free the allocated array and \"skip_oids\" set\n> > +\t */\n> > +\tfree(fsck_refs_options.msg_type);\n> > +\toidset_clear(&fsck_refs_options.skip_oids);\n> \n> Should we provide a `fsck_options_clear()` function that does this for\n> us? Otherwise we'll have to adapt callsites of `refs_fsck` whenever\n> internal implementation details of the subsystem add newly allocated\n> members.\n> \n\nYes, I agree with this. I wanna talk more on this. In the first time, I\ndid not call `oidset_clear` and I failed the CI tests. It made me\nconfused. Because we never use \"skip_oids\" in the ref check, why the\ntests said that \"fsck_refs.options.skip_oids\" was not freed.\n\nThis is because when executing the command \"git -c fsck.skipList=.. fsck\",\nin the subprocess `git refs verify`, the code will still setup the\n\"skip_oids\" by the config. So we should explicitly free the \"skip_oids\".\n\nBut how does \"fsck.c\" free \"skip_oids\", actually \"fsck.c\" never frees\n\"skip_oids\". This is because \"git-fsck(1)\" defines the following:\n\n static struct fsck_options fsck_walk_options = FSCK_OPTIONS_DEFAULT;\n static struct fsck_options fsck_obj_options = FSCK_OPTIONS_DEFAULT;\n\nBecause these two options are \"static\", so there is no memory leak. We\nleave it to the operating system. So maybe a more simple way is just to\nadd \"static\" identifier in \"cmd_refs_verify\" which means:\n\n - struct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n + static struct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n\nBut I don't think we should use `static`, because Eric has told me that\nmaking a variable \"static\" will make the code harder to \"libfy\". So\nlet's use \"fsck_options_clear\" function instead.\n\n"},{"id":"499622","messageId":"ZqkQbjtDFawIwlvf@ArchLinux","threadId":"61565","inReplyTo":"Zqik73qynJvOLctw@tanuki","subject":"Re: [GSoC][PATCH v13 08/10] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-30T16:10:22Z","receivedAt":"2024-07-30T16:10:02Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"> > +\titer = dir_iterator_begin(sb.buf, 0);\n> > +\n> > +\tif (!iter) {\n> > +\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n> > +\t\tgoto out;\n> > +\t}\n> \n> The error message should probably be marked as translatable. Also, I'd\n> personally remove the newline between `iter = ...` and the error check\n> as those are a logical unit.\n> \n\nYes, I will improve this in the next version.\n\n> > +\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n> > +\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n> > +\t\t\t\t\tret = -1;\n> > +\t\t\t}\n> > +\t\t} else {\n> > +\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n> > +\t\t\t\t iter->basename);\n> \n> Instead of printing this as an error directly, shouldn't we report it\n> via the `fsck_refs_report` interface?\n> \n\nYes, exactly we should use this interface. I accidentally ignored this.\nThanks.\n\n> > +out:\n> > +\tstrbuf_release(&sb);\n> > +\treturn ret;\n> > +}\n> > +\n> > +static int files_fsck_refs(struct ref_store *ref_store,\n> > +\t\t\t struct fsck_options *o)\n> > +{\n> > +\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n> > +\t\tNULL\n> \n> The last member should also end with a comma.\n> \n\nI will improve this in the next version.\n\n> > +\t};\n> > +\n> > +\tif (o->verbose)\n> > +\t\tfprintf_ln(stderr, \"Checking references consistency\");\n> > +\n> > +\treturn files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n> > +\n> \n> This newline should be removed.\n> \n\nOK.\n\n> > +}\n> > +\n> > static int files_fsck(struct ref_store *ref_store,\n> > \t\t struct fsck_options *o)\n> > {\n> > \tstruct files_ref_store *refs =\n> > \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n> > \n> > -\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n> > +\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o) |\n> > +\t files_fsck_refs(ref_store, o);\n> \n> I'd think we should first check loose files and then continue to check\n> the packed ref store. That's really only a gut feeling though, and I\n> cannot exactly say why that feels more natural.\n> \n\nIt would feel more natural. Because packed-ref will point to the\nloose ref. So we should first check the loose refs. For example. If a\nregular ref is bad, we will first report the problem. And suppose the\npacked-ref have recorded this regular ref. When checking packed-ref, we\ncould not check the regular ref itself. We only need to check one thing.\n\n Whether the pointee exists under the \"refs/\" directory.\n\nAnd we do not need to check regular ref again because we have checked in\nthe loose refs part.\n\n> Patrick\n"},{"id":"499623","messageId":"ZqkRYLlqtE9tgyDF@ArchLinux","threadId":"61565","inReplyTo":"Zqik9EOKd43WxGXp@tanuki","subject":"Re: [GSoC][PATCH v13 09/10] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-30T16:14:24Z","receivedAt":"2024-07-30T16:14:03Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"> > diff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\n> > new file mode 100755\n> > index 0000000000..b2db58d2c6\n> > --- /dev/null\n> > +++ b/t/t0602-reffiles-fsck.sh\n> > @@ -0,0 +1,101 @@\n> > +#!/bin/sh\n> > +\n> > +test_description='Test reffiles backend consistency check'\n> > +\n> > +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n> > +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n> > +GIT_TEST_DEFAULT_REF_FORMAT=files\n> > +export GIT_TEST_DEFAULT_REF_FORMAT\n> > +\n> > +. ./test-lib.sh\n> \n> Is this test suite intentionally not marked with\n> `TEST_PASSES_SANITIZE_LEAK=true`?\n> \n\nNo, I don't know this. I will add `TEST_PASSES_SANITIZE_LEAK=true` and\nexport this environment variable.\n\n> > +\n> > +test_expect_success 'ref name should be checked' '\n> > +\ttest_when_finished \"rm -rf repo\" &&\n> > +\tgit init repo &&\n> > +\tbranch_dir_prefix=.git/refs/heads &&\n> > +\ttag_dir_prefix=.git/refs/tags &&\n> > +\t(\n> > +\t\tcd repo &&\n> > +\t\tgit commit --allow-empty -m initial &&\n> > +\t\tgit checkout -b branch-1 &&\n> > +\t\tgit tag tag-1 &&\n> > +\t\tgit commit --allow-empty -m second &&\n> > +\t\tgit checkout -b branch-2 &&\n> > +\t\tgit tag tag-2 &&\n> > +\t\tgit tag multi_hierarchy/tag-2\n> > +\t) &&\n> \n> I don't quite get why you create several subshells only to cd into\n> `repo` in each of them. Isn't a single subshell sufficient for all of\n> those tests? If you want to delimit blocks, then you can simply add an\n> empty newline between them.\n> \n\nI just want to delimit, I will use newline in the next version.\n\n> > +\t(\n> > +\t\tcd repo &&\n> > +\t\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n> > +\t\ttest_must_fail git fsck 2>err &&\n> > +\t\tcat >expect <<-EOF &&\n> > +\t\terror: refs/heads/.branch-1: badRefName: invalid refname format\n> > +\t\tEOF\n> > +\t\trm $branch_dir_prefix/.branch-1 &&\n> > +\t\ttest_cmp expect err\n> > +\t) &&\n> > +\t(\n> > +\t\tcd repo &&\n> > +\t\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n> > +\t\ttest_must_fail git fsck 2>err &&\n> > +\t\tcat >expect <<-EOF &&\n> > +\t\terror: refs/tags/tag-1.lock: badRefName: invalid refname format\n> > +\t\tEOF\n> > +\t\trm $tag_dir_prefix/tag-1.lock &&\n> > +\t\ttest_cmp expect err\n> > +\t) &&\n> \n> The other cases all make sense, but I don't think that a file ending\n> with \".lock\" should be marked as having a \"badRefName\". It is expected\n> that concurrent writers may have such lock files.\n> \n> What could make sense is to eventually mark stale lock files older than\n> X amount of time as errors or warnings. But I'd think that this is\n> outside of the scope of this patch series.\n> \n\nIf so, let us just ignore \".lock\" situation at the moment.\n\n> Patrick\n"},{"id":"499625","messageId":"ZqkT-3aAi6msd9K4@ArchLinux","threadId":"61565","inReplyTo":"Zqik-iqO4-yNuIn7@tanuki","subject":"Re: [GSoC][PATCH v13 10/10] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-30T16:25:31Z","receivedAt":"2024-07-30T16:25:11Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jul 30, 2024 at 10:31:54AM +0200, Patrick Steinhardt wrote:\n> On Mon, Jul 29, 2024 at 09:27:56PM +0800, shejialuo wrote:\n> > Enhance the git-fsck(1) command by adding a check for reference content\n> > in the files backend. The new functionality ensures that symrefs, real\n> > symbolic link and regular refs are validated correctly.\n> > \n> > In order to check the trailing content of the regular refs, add a new\n> > parameter `trailing` to `parse_loose_ref_contents`.\n> > \n> > For symrefs, `parse_loose_ref_contents` will set the \"referent\".\n> > However, symbolic link could be either absolute or relative. Use\n> > \"strbuf_add_real_path\" to read the symbolic link and convert the\n> > relative path to absolute path. Then use \"skip_prefix\" to make it align\n> > with symref \"referent\".\n> > \n> > Thus, the symrefs and symbolic links could share the same interface. Add\n> > a new function \"files_fsck_symref_target\" which aims at checking the\n> > following things:\n> > \n> > 1. whether the pointee is under the `refs/` directory.\n> > 2. whether the pointee name is correct.\n> > 3. whether the pointee path is a wrong type in filesystem.\n> > \n> > Last, add the following FSCK MESSAGEs:\n> > \n> > 1. \"badRefContent(ERROR)\": A ref has a bad content\n> > 2. \"badSymrefPointee(ERROR)\": The pointee of a symref is bad.\n> > 3. \"trailingRefContent(WARN)\": A ref content has trailing contents.\n> \n> I think it would have been fine to stop at the preceding commit as it\n> clearly demonstrates how the whole infrastructure is supposed to work.\n> Additional checks like those you add here would then be a good candidate\n> for a separate patch series. This would help you get the first patch\n> series landed faster as you really only have to focus on setting up the\n> baseline infrastructure.\n> \n> Feel free to keep or drop this patch as you prefer though, I don't want\n> to discourage you aiming higher. Just keep in mind that the more you add\n> on top the longer it takes to land a patch series :)\n> \n\nI will drop this patch in the next version. Actually, in the very former\nversion, I didn't realise that the effort to set up the infra is so\nmuch.\n\n> > +\t/*\n> > +\t * If the ref is a symref, we need to check the destination name and\n> > +\t * connectivity.\n> > +\t */\n> > +\tif (referent.len && (type & REF_ISSYMREF)) {\n> > +\t\tstrbuf_addf(&pointee_path, \"%s/%s\", gitdir, referent.buf);\n> > +\t\tstrbuf_rtrim(&referent);\n> > +\n> > +\t\tret = files_fsck_symref_target(o, &info, refname.buf,\n> > +\t\t\t\t\t referent.buf, pointee_path.buf);\n> > +\t\tgoto clean;\n> > +\t} else {\n> > +\t\tif (trailing && (*trailing != '\\0' && *trailing != '\\n')) {\n> \n> In case the ref ends with a newline, should we check that the next\n> character is `\\0`? Otherwise, it may contain multiple lines, which is\n> not allowed for a normal ref.\n> \n> Also, shouldn't the ref always end with a newline?\n> \n\nThis is a very interesting question here. Based on my experiments, I\nhave found the following things:\n\nIt's OK that regular refs contain multiple newlines. And git totally\nallows such case. The current code does not handle multiple newlines.\n\nFor symrefs, it allows spaces and newlines, for example:\n\n ref: refs/heads/master \n ref: refs/heads/master \\n\\n\\n\n\nBut for such case, git will report error:\n\n ref: refs/heads/master garbage\n\nAnd ref can be end with a newline or not. Both will be accepted. Junio\nhave told me that there is no spec really. So, I ignore multiple\nnewlines for regular refs and also ignore multiple newlines and trailing\nspaces for symref.\n\n> Patrick\n\n\n"},{"id":"499626","messageId":"ZqkU8whn1LfTRdfk@ArchLinux","threadId":"61565","inReplyTo":"Zqik_yal_hPTlotg@tanuki","subject":"Re: [GSoC][PATCH v13 00/10] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-30T16:29:39Z","receivedAt":"2024-07-30T16:29:18Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"> I've got a bunch of comments, mostly because I've been out for quite a\n> while and thus didn't follow the progression of this series. But don't\n> let yourself be discouraged by the volume, I quite like the shape of\n> this patch series and think that it is very close to what we actually\n> want to have.\n> \n> I'd say that this series is likely to become ready soonish.\n> \n> Thanks!\n> \n> Patrick\n\nThat's OK, Patrick. It's important for me to receive reviews thus we can\nmake this series better. And I will wait for the advice from the Junio\nabout the design of the \"fsck_error\" here. If Junio were satisfied with\nthis design. I will clean the code and submit a new version until\neverything is OK.\n\nThanks,\nJialuo\n\n"},{"id":"499631","messageId":"CAPig+cRD3F-QHneC+0bEUcWt4ep8hXgjg4OT_MvfLjQKNE54iA@mail.gmail.com","threadId":"61565","inReplyTo":"ZqkN4wNK0PIRyNky@ArchLinux","subject":"Re: [GSoC][PATCH v13 06/10] git refs: add verify subcommand","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-07-30T17:56:58Z","receivedAt":"2024-07-30T17:57:10Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Jul 30, 2024 at 11:59 AM shejialuo wrote:\n> On Tue, Jul 30, 2024 at 10:31:37AM +0200, Patrick Steinhardt wrote:\n> > On Mon, Jul 29, 2024 at 09:27:12PM +0800, shejialuo wrote:\n> > > + /*\n> > > + * Explicitly free the allocated array and \"skip_oids\" set\n> > > + */\n> > > + free(fsck_refs_options.msg_type);\n> > > + oidset_clear(&fsck_refs_options.skip_oids);\n> >\n> > Should we provide a `fsck_options_clear()` function that does this for\n> > us? Otherwise we'll have to adapt callsites of `refs_fsck` whenever\n> > internal implementation details of the subsystem add newly allocated\n> > members.\n> [...]\n> But how does \"fsck.c\" free \"skip_oids\", actually \"fsck.c\" never frees\n> \"skip_oids\". This is because \"git-fsck(1)\" defines the following:\n>\n> static struct fsck_options fsck_walk_options = FSCK_OPTIONS_DEFAULT;\n> static struct fsck_options fsck_obj_options = FSCK_OPTIONS_DEFAULT;\n>\n> Because these two options are \"static\", so there is no memory leak. We\n> leave it to the operating system. So maybe a more simple way is just to\n> add \"static\" identifier in \"cmd_refs_verify\" which means:\n>\n> - struct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n> + static struct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n>\n> But I don't think we should use `static`, because Eric has told me that\n> making a variable \"static\" will make the code harder to \"libfy\". So\n> let's use \"fsck_options_clear\" function instead.\n\nI haven't been following this topic closely and I'm not familiar with\nthis code (and don't have much time now to dig into it), but I suspect\nthe context here is rather different from the one[*] in which I was\nhighly skeptical of the use of `static`. The `static` in that earlier\ncase was suspicious/questionable for two reasons. First, it was a case\nof premature optimization (which, by definition, is frowned upon).\nSecond, it was in a \"library\" function (namely, top-level\nfsck.c:fsck_refs_error_function()) which may someday become a linkable\nlibrary which other programs (aside from `git` itself) may utilize.\nHaving a static strbuf in the library function makes the function\nnon-reentrant and takes memory management out of the hands of the\nclient.\n\nIn the case under discussion here (namely `builtin/fsck.c`), it is a\nGit-specific command, not library code. As such, \"libification\" is\nmuch less of an issue since Git-specific command code is less likely\nto be reused by some other project. (However, that's not to say that\nwe shouldn't worry about unnecessary use of `static` even in builtin\ncommands; code from those commands does periodically migrate from\n`builtin/*.c` to top-level library oriented `*.c`.)\n\nSo, considering that the variable under discussion:\n\n struct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n\nis part of a builtin command rather than library code, we don't have\nto worry about \"libification\" so much, thus making it `static` would\nbe a workable approach. However, doing so merely to avoid complaint by\nthe leak-checker does not seem like good justification. Hence, keeping\nthis variable non-static and freeing it explicitly seems a better idea\n(which is what this code does presently).\n\nI do agree with Patrick that adding fsck_options_clear() to top-level\n`fsck.h` would be sensible since it frees callers from having to know\nimplementation details of `fsck_options`.\n\nBy the way, regarding the static `fsck_walk_options` and\n`fsck_obj_options` those are probably global static for convenience\nrather than out of necessity. It might very well be possible to make\nthose local variables in builtin/fsck.c:cmd_fsck() and then plumb them\nthrough to called functions so that they don't have to be static, and\nthen they would be freed manually by cmd_fsck(), as well. However,\nthat sort of change is well outside the scope of this topic.\n\n[*] https://lore.kernel.org/git/CAPig+cR=RgMeaAy1PRGgHu6_Ak+7=_-5tGvBZRekKRxi7GtdHw@mail.gmail.com/\n"},{"id":"499647","messageId":"xmqqh6c6a61e.fsf@gitster.g","threadId":"61565","inReplyTo":"ZqeY3Dhj-Fo-bZ2k@ArchLinux","subject":"Re: [GSoC][PATCH v13 10/10] fsck: add ref content check for files backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-30T22:06:37Z","receivedAt":"2024-07-30T22:06:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"shejialuo writes:\n\n> +static int files_fsck_refs_content(struct fsck_options *o,\n> +\t\t\t\t const char *gitdir,\n> +\t\t\t\t const char *refs_check_dir,\n> +\t\t\t\t struct dir_iterator *iter)\n> +{\n> + ...\n> +\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n> +\t\t\t\t &referent, &type,\n> +\t\t\t\t &failure_errno, &trailing)) {\n\nThe function parse_loose_ref_contents() needs to know what the hash\nalgorithm is, and it used to implicitly assume that the_repository's\nhash algorithm was an OK thing to use. Patrick's recent clean-up\nseries instead passes \"struct ref_store *refs\" throughout the call\nchain so that \"ref->repo->hash_algo\" can be used. This needs some\nmatching change, which means ...\n\n> \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n> \t\tfiles_fsck_refs_name,\n> +\t\tfiles_fsck_refs_content,\n> \t\tNULL\n> \t};\n\n... the function signature for files_fsck_refs_fn must change to\nhave something that lets you access repo->hash_algo.\n\n\nBy the way, unless the most common use of an array is to pass it\naround as a collection of items and operate on the collection, it is\na better practice to name an array with a singular noun. Name the\narray as fsck_refs_fn[] not _fns[]. This is so that you can refer\nto a single element in a more grammatical way. E.g. with\n\n struct dog dog[] = { { .breed=\"shiba\" }, { .breed=\"beagle\" } };\n\nyou can say \"dog[0] has brown fur\" instead of \"dogs[0] has ...\".\n\nIn this case, you do not treat the collection of functions as a one\nthing and do something to the collection. Instead you'd repeat over\nthe functions in a loop and individually call them, perhaps like so:\n\n\tfor (i = 0; fsck_fn[i] != NULL; i++)\n\t\tfsck_fn[i](...);\n\nso it is very much more appropriate to name the array itself as\nsingular to allow you to say \"first fsck_fn\", \"next fsck_fn\", etc.\n\n\n"},{"id":"499714","messageId":"ZqpkAakuNkm8qJMM@ArchLinux","threadId":"61565","inReplyTo":"xmqqh6c6a61e.fsf@gitster.g","subject":"Re: [GSoC][PATCH v13 10/10] fsck: add ref content check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-07-31T16:19:13Z","receivedAt":"2024-07-31T16:18:53Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Jul 30, 2024 at 03:06:37PM -0700, Junio C Hamano wrote:\n> shejialuo writes:\n> \n> > +static int files_fsck_refs_content(struct fsck_options *o,\n> > +\t\t\t\t const char *gitdir,\n> > +\t\t\t\t const char *refs_check_dir,\n> > +\t\t\t\t struct dir_iterator *iter)\n> > +{\n> > + ...\n> > +\tif (parse_loose_ref_contents(ref_content.buf, &oid,\n> > +\t\t\t\t &referent, &type,\n> > +\t\t\t\t &failure_errno, &trailing)) {\n> \n> The function parse_loose_ref_contents() needs to know what the hash\n> algorithm is, and it used to implicitly assume that the_repository's\n> hash algorithm was an OK thing to use. Patrick's recent clean-up\n> series instead passes \"struct ref_store *refs\" throughout the call\n> chain so that \"ref->repo->hash_algo\" can be used. This needs some\n> matching change, which means ...\n> \n> > \tfiles_fsck_refs_fn fsck_refs_fns[]= {\n> > \t\tfiles_fsck_refs_name,\n> > +\t\tfiles_fsck_refs_content,\n> > \t\tNULL\n> > \t};\n> \n> ... the function signature for files_fsck_refs_fn must change to\n> have something that lets you access repo->hash_algo.\n> \n\nThanks for your remind, I have scanned this patch:\n\n\thttps://lore.kernel.org/git/fe0e2c3617c8040c632dbc3de613a1d22e8070f7.1722316795.git.ps@pks.im/\n\nI guess I will handle this later. It seems that this series has not come\ninto the cooking tree. I will update this part until Patrick's patch\ngets merged into \"next\".\n\n> \n> By the way, unless the most common use of an array is to pass it\n> around as a collection of items and operate on the collection, it is\n> a better practice to name an array with a singular noun. Name the\n> array as fsck_refs_fn[] not _fns[]. This is so that you can refer\n> to a single element in a more grammatical way. E.g. with\n> \n> struct dog dog[] = { { .breed=\"shiba\" }, { .breed=\"beagle\" } };\n> \n> you can say \"dog[0] has brown fur\" instead of \"dogs[0] has ...\".\n> \n> In this case, you do not treat the collection of functions as a one\n> thing and do something to the collection. Instead you'd repeat over\n> the functions in a loop and individually call them, perhaps like so:\n> \n> \tfor (i = 0; fsck_fn[i] != NULL; i++)\n> \t\tfsck_fn[i](...);\n> \n> so it is very much more appropriate to name the array itself as\n> singular to allow you to say \"first fsck_fn\", \"next fsck_fn\", etc.\n> \n\nThanks, I have learned a lot here. I will improve this in the next\nversion.\n"},{"id":"499857","messageId":"ZqulmWVBaeyP4blf@ArchLinux","threadId":"61565","inReplyTo":"ZqeXrPROpEg_pRS2@ArchLinux","subject":"[GSoC][PATCH v14 00/11] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-01T15:11:21Z","receivedAt":"2024-08-01T15:11:00Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Hi All,\n\nThis version does the following things:\n\n1. By following the advice from Patrick, we should make the callback\nfunction be generic by adding only one \"void * fsck_report\" parameter.\nThus the commit sequence will be much more clearer. And it wll be much\neasier for reviewers to review. And I have split the commit into more\ncommits in this version.\n2. Enhance the commit messages to provide more context about why we\nshould do this.\n3. Patrick advices that we should initialize the \"fsck_options\" member\nwhen parsing the options. However, because the original \"strict\" and\n\"verbose\" field are defined as the bit field, we cannot take the address\nof them. So I simply remove the bit field.\n4. As Patrick said, \".lock\" should not be reported as error. At current,\nignore files ending with \".lock\".\n5. Add a fsck msg type called \"badRefFiletype\" which indicates that a\nref has a bad file type when scanning the directory.\n6. Junio advices instead of using \"fsck_refs_fns\", we should use the\nsingular version \"fsck_refs_fn\", fix this.\n7. Drop the last patch because in this series, we mainly focus on the\ninfra, I will add a series later to add ref content check.\n\nHowever, there is one thing holding. Junio advices that I should\nfollow the Patrick's change to change the prototype of \"files_fsck_refs_fn\"\n\n https://lore.kernel.org/git/fe0e2c3617c8040c632dbc3de613a1d22e8070f7.1722316795.git.ps@pks.im/\n\nHowever, at current, this patch is not in the cooking tree. I will\nhandle this later.\n\nshejialuo (11):\n fsck: rename \"skiplist\" to \"skip_oids\"\n fsck: make \"fsck_error\" callback generic\n fsck: add a unified interface for reporting fsck messages\n fsck: add refs report function\n fsck: add refs-related error callback\n fsck: rename objects-related fsck error functions\n refs: set up ref consistency check infrastructure\n builtin/refs: add verify subcommand\n builtin/fsck: add `git-refs verify` child process\n files-backend: add unified interface for refs scanning\n fsck: add ref name check for files backend\n\n Documentation/fsck-msgids.txt | 6 ++\n Documentation/git-refs.txt | 13 ++++\n builtin/fsck.c | 34 +++++++--\n builtin/mktag.c | 3 +-\n builtin/refs.c | 34 +++++++++\n fsck.c | 131 +++++++++++++++++++++++++++-------\n fsck.h | 76 +++++++++++++++-----\n object-file.c | 9 ++-\n refs.c | 5 ++\n refs.h | 8 +++\n refs/debug.c | 11 +++\n refs/files-backend.c | 118 +++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 +++\n refs/refs-internal.h | 6 ++\n refs/reftable-backend.c | 8 +++\n t/t0602-reffiles-fsck.sh | 94 ++++++++++++++++++++++++\n 16 files changed, 504 insertions(+), 60 deletions(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\nRange-diff against v13:\n 1: 772cad5b92 = 1: 75b64a219d fsck: rename \"skiplist\" to \"skip_oids\"\n 2: e76449977d < -: ---------- fsck: add a unified interface for reporting fsck messages\n -: ---------- > 2: 81433d1628 fsck: make \"fsck_error\" callback generic\n -: ---------- > 3: 0792c51e6d fsck: add a unified interface for reporting fsck messages\n -: ---------- > 4: ecd144af15 fsck: add refs report function\n -: ---------- > 5: e80dba0cab fsck: add refs-related error callback\n 3: 26bde4283f ! 6: a61db42bf2 fsck: rename objects-related fsck error functions\n @@ Metadata\n ## Commit message ##\n fsck: rename objects-related fsck error functions\n \n - The names of objects-related fsck error functions are general. It's OK\n + The names of objects-related fsck error functions are generic. It's OK\n when there is only object database check. However, we have introduced\n refs database check report function. To avoid ambiguity, rename\n object-related fsck error functions to explicitly indicate these\n @@ builtin/fsck.c: static int objerror(struct object *obj, const char *err)\n }\n \n -static int fsck_error_func(struct fsck_options *o UNUSED,\n --\t\t\t const struct object_id *oid,\n --\t\t\t enum object_type object_type,\n --\t\t\t const struct fsck_refs_info *refs_info UNUSED,\n +-\t\t\t void *fsck_report,\n -\t\t\t enum fsck_msg_type msg_type,\n -\t\t\t enum fsck_msg_id msg_id UNUSED,\n -\t\t\t const char *message)\n +static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n -+\t\t\t\t const struct object_id *oid,\n -+\t\t\t\t enum object_type object_type,\n -+\t\t\t\t const struct fsck_refs_info *refs_info UNUSED,\n ++\t\t\t\t void *fsck_report,\n +\t\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n +\t\t\t\t const char *message)\n {\n - \tswitch (msg_type) {\n - \tcase FSCK_WARN:\n + \tstruct fsck_object_report *report = fsck_report;\n + \tconst struct object_id *oid = report->oid;\n @@ builtin/fsck.c: int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tfsck_walk_options.walk = mark_object;\n @@ fsck.c: int fsck_buffer(const struct object_id *oid, enum object_type type,\n }\n \n -int fsck_error_function(struct fsck_options *o,\n --\t\t\tconst struct object_id *oid,\n --\t\t\tenum object_type object_type UNUSED,\n --\t\t\tconst struct fsck_refs_info *refs_info UNUSED,\n +-\t\t\tvoid *fsck_report,\n -\t\t\tenum fsck_msg_type msg_type,\n -\t\t\tenum fsck_msg_id msg_id UNUSED,\n -\t\t\tconst char *message)\n +int fsck_objects_error_function(struct fsck_options *o,\n -+\t\t\t\tconst struct object_id *oid,\n -+\t\t\t\tenum object_type object_type UNUSED,\n -+\t\t\t\tconst struct fsck_refs_info *refs_info UNUSED,\n ++\t\t\t\tvoid *fsck_report,\n +\t\t\t\tenum fsck_msg_type msg_type,\n +\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n +\t\t\t\tconst char *message)\n {\n - \tif (msg_type == FSCK_WARN) {\n - \t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n + \tstruct fsck_object_report *report = fsck_report;\n + \tconst struct object_id *oid = report->oid;\n @@ fsck.c: int git_fsck_config(const char *var, const char *value,\n * Custom error callbacks that are used in more than one place.\n */\n \n -int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n --\t\t\t\t\t const struct object_id *oid,\n --\t\t\t\t\t enum object_type object_type,\n --\t\t\t\t\t const struct fsck_refs_info *refs_info,\n +-\t\t\t\t\t void *fsck_report,\n -\t\t\t\t\t enum fsck_msg_type msg_type,\n -\t\t\t\t\t enum fsck_msg_id msg_id,\n -\t\t\t\t\t const char *message)\n +int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n -+\t\t\t\t\t\t const struct object_id *oid,\n -+\t\t\t\t\t\t enum object_type object_type,\n -+\t\t\t\t\t\t const struct fsck_refs_info *refs_info,\n ++\t\t\t\t\t\t void *fsck_report,\n +\t\t\t\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t\t\t\t enum fsck_msg_id msg_id,\n +\t\t\t\t\t\t const char *message)\n {\n \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n - \t\tputs(oid_to_hex(oid));\n + \t\tstruct fsck_object_report *report = fsck_report;\n + \t\tputs(oid_to_hex(report->oid));\n \t\treturn 0;\n \t}\n --\treturn fsck_error_function(o, oid, object_type, refs_info,\n --\t\t\t\t msg_type, msg_id, message);\n -+\treturn fsck_objects_error_function(o, oid, object_type, refs_info,\n -+\t\t\t\t\t msg_type, msg_id, message);\n +-\treturn fsck_error_function(o, fsck_report, msg_type, msg_id, message);\n ++\treturn fsck_objects_error_function(o, fsck_report, msg_type,msg_id, message);\n }\n \n ## fsck.h ##\n @@ fsck.h: typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t const char *message);\n \n -int fsck_error_function(struct fsck_options *o,\n --\t\t\tconst struct object_id *oid, enum object_type object_type,\n --\t\t\tconst struct fsck_refs_info *refs_info,\n +-\t\t\tvoid *fsck_report,\n -\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n -\t\t\tconst char *message);\n -int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n --\t\t\t\t\t const struct object_id *oid,\n --\t\t\t\t\t enum object_type object_type,\n --\t\t\t\t\t const struct fsck_refs_info *refs_info,\n +-\t\t\t\t\t void *fsck_report,\n -\t\t\t\t\t enum fsck_msg_type msg_type,\n -\t\t\t\t\t enum fsck_msg_id msg_id,\n -\t\t\t\t\t const char *message);\n +int fsck_objects_error_function(struct fsck_options *o,\n -+\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n -+\t\t\t\tconst struct fsck_refs_info *refs_info,\n ++\t\t\t\tvoid *fsck_report,\n +\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n +\t\t\t\tconst char *message);\n +int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n -+\t\t\t\t\t\t const struct object_id *oid,\n -+\t\t\t\t\t\t enum object_type object_type,\n -+\t\t\t\t\t\t const struct fsck_refs_info *refs_info,\n ++\t\t\t\t\t\t void *fsck_report,\n +\t\t\t\t\t\t enum fsck_msg_type msg_type,\n +\t\t\t\t\t\t enum fsck_msg_id msg_id,\n +\t\t\t\t\t\t const char *message);\n \n - /*\n - * The information for reporting refs-related error message\n + int fsck_refs_error_function(struct fsck_options *options,\n + \t\t\t void *fsck_report,\n @@ fsck.h: struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n @@ fsck.h: struct fsck_options {\n -\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n +\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n - \n - /* descend in all linked child objects\n + #define FSCK_REFS_OPTIONS_DEFAULT { \\\n + \t.error_func = fsck_refs_error_function, \\\n 4: 56fc833a54 < -: ---------- fsck: add refs-related error report function\n 5: 90f992ecfb ! 7: f2248bc52d refs: set up ref consistency check infrastructure\n @@ Metadata\n ## Commit message ##\n refs: set up ref consistency check infrastructure\n \n + The \"struct ref_store\" is the base class which contains the \"be\" pointer\n + which provides backend-specific functions whose interfaces are defined\n + in the \"ref_storage_be\". We could reuse this polymorphism to define only\n + one interface. For every backend, we need to provide its own function\n + pointer.\n +\n The interfaces defined in the `ref_storage_be` are carefully structured\n in semantic. It's organized as the five parts:\n \n 6: b2277f5ef4 ! 8: 0c5463d757 git refs: add verify subcommand\n @@ Metadata\n Author: shejialuo \n \n ## Commit message ##\n - git refs: add verify subcommand\n + builtin/refs: add verify subcommand\n \n Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\n check the reference database consistency and also this subcommand will\n - be used as the entry point of checking refs for \"git-fsck(1)\". Last, add\n - \"verbose\" field into \"fsck_options\" to indicate whether we should print\n - verbose messages when checking refs and objects consistency.\n + be used as the entry point of checking refs for \"git-fsck(1)\".\n +\n + Add \"verbose\" field into \"fsck_options\" to indicate whether we should\n + print verbose messages when checking refs and objects consistency.\n +\n + Remove bit-field for \"strict\" field, this is because we cannot take\n + address of a bit-field which makes it unhandy to set member variables\n + when parsing the command line options.\n +\n + The \"git-fsck(1)\" declares \"fsck_options\" variable with \"static\"\n + identifier which avoids complaint by the leak-checker. However, in\n + \"git-refs verify\", we need to do memory clean manually. Thus add\n + \"fsck_options_clear\" function in \"fsck.c\" to provide memory clean\n + operation.\n \n Mentored-by: Patrick Steinhardt \n Mentored-by: Karthik Nayak \n @@ Documentation/git-refs.txt: include::ref-storage-format.txt[]\n +The following options are specific to 'git refs verify':\n +\n +--strict::\n -+\tEnable more strict checking, every WARN severity for the `Fsck Messages`\n -+\tbe seen as ERROR. See linkgit:git-fsck[1].\n ++\tEnable stricter error checking. This will cause warnings to be\n ++\treported as errors. See linkgit:git-fsck[1].\n +\n +--verbose::\n +\tWhen verifying the reference database consistency, be chatty.\n @@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c\n +\t\tREFS_VERIFY_USAGE,\n +\t\tNULL,\n +\t};\n -+\tunsigned int verbose = 0, strict = 0;\n +\tstruct option options[] = {\n -+\t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n -+\t\tOPT_BOOL(0, \"strict\", &strict, N_(\"enable strict checking\")),\n ++\t\tOPT_BOOL(0, \"verbose\", &fsck_refs_options.verbose, N_(\"be verbose\")),\n ++\t\tOPT_BOOL(0, \"strict\", &fsck_refs_options.strict, N_(\"enable strict checking\")),\n +\t\tOPT_END(),\n +\t};\n +\tint ret;\n @@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c\n +\tif (argc)\n +\t\tusage(_(\"'git refs verify' takes no arguments\"));\n +\n -+\tif (verbose)\n -+\t\tfsck_refs_options.verbose = 1;\n -+\tif (strict)\n -+\t\tfsck_refs_options.strict = 1;\n -+\n +\tgit_config(git_fsck_config, &fsck_refs_options);\n +\tprepare_repo_settings(the_repository);\n +\n +\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n +\n -+\t/*\n -+\t * Explicitly free the allocated array and \"skip_oids\" set\n -+\t */\n -+\tfree(fsck_refs_options.msg_type);\n -+\toidset_clear(&fsck_refs_options.skip_oids);\n ++\tfsck_options_clear(&fsck_refs_options);\n +\treturn ret;\n +}\n +\n @@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c\n \t};\n \n \n + ## fsck.c ##\n +@@ fsck.c: int fsck_finish(struct fsck_options *options)\n + \treturn ret;\n + }\n + \n ++void fsck_options_clear(struct fsck_options *options)\n ++{\n ++\tfree(options->msg_type);\n ++\toidset_clear(&options->skip_oids);\n ++\toidset_clear(&options->gitmodules_found);\n ++\toidset_clear(&options->gitmodules_done);\n ++\toidset_clear(&options->gitattributes_found);\n ++\toidset_clear(&options->gitattributes_done);\n ++\tkh_clear_oid_map(options->object_names);\n ++}\n ++\n + int git_fsck_config(const char *var, const char *value,\n + \t\t const struct config_context *ctx, void *cb)\n + {\n +\n ## fsck.h ##\n -@@ fsck.h: struct fsck_options {\n +@@ fsck.h: struct fsck_ref_report {\n + struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n - \tunsigned strict:1;\n -+\tunsigned verbose:1;\n +-\tunsigned strict:1;\n ++\tunsigned strict;\n ++\tunsigned verbose;\n \tenum fsck_msg_type *msg_type;\n \tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n +@@ fsck.h: int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n + */\n + int fsck_finish(struct fsck_options *options);\n + \n ++/*\n ++ * Clear the fsck_options struct, freeing any allocated memory.\n ++ */\n ++void fsck_options_clear(struct fsck_options *options);\n ++\n + /*\n + * Report an error or warning for refs.\n + */\n 7: f96d0f200d = 9: 0ac25c2f6e builtin/fsck: add `git-refs verify` child process\n 8: f9a0c16bef ! 10: 2b6f5e6c46 files-backend: add unified interface for refs scanning\n @@ Commit message\n Mentored-by: Karthik Nayak \n Signed-off-by: shejialuo \n \n + ## Documentation/fsck-msgids.txt ##\n +@@\n + `badParentSha1`::\n + \t(ERROR) A commit object has a bad parent sha1.\n + \n ++`badRefFiletype`::\n ++\t(ERROR) A ref has a bad file type.\n ++\n + `badTagName`::\n + \t(INFO) A tag has an invalid format.\n + \n +\n + ## fsck.h ##\n +@@ fsck.h: enum fsck_msg_type {\n + \tFUNC(BAD_NAME, ERROR) \\\n + \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n + \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n ++\tFUNC(BAD_REF_FILETYPE, ERROR) \\\n + \tFUNC(BAD_TIMEZONE, ERROR) \\\n + \tFUNC(BAD_TREE, ERROR) \\\n + \tFUNC(BAD_TREE_SHA1, ERROR) \\\n +\n ## refs/files-backend.c ##\n @@\n #include \"../gettext.h\"\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +static int files_fsck_refs_dir(struct ref_store *ref_store,\n +\t\t\t struct fsck_options *o,\n +\t\t\t const char *refs_check_dir,\n -+\t\t\t files_fsck_refs_fn *fsck_refs_fns)\n ++\t\t\t files_fsck_refs_fn *fsck_refs_fn)\n +{\n +\tconst char *gitdir = ref_store->gitdir;\n +\tstruct strbuf sb = STRBUF_INIT;\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n +\n +\titer = dir_iterator_begin(sb.buf, 0);\n -+\n +\tif (!iter) {\n -+\t\tret = error_errno(\"cannot open directory %s\", sb.buf);\n ++\t\tret = error_errno(_(\"cannot open directory %s\"), sb.buf);\n +\t\tgoto out;\n +\t}\n +\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +\t\t\tif (o->verbose)\n +\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n +\t\t\t\t\t refs_check_dir, iter->relative_path);\n -+\t\t\tfor (size_t i = 0; fsck_refs_fns[i]; i++) {\n -+\t\t\t\tif (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))\n ++\t\t\tfor (size_t i = 0; fsck_refs_fn[i]; i++) {\n ++\t\t\t\tif (fsck_refs_fn[i](o, gitdir, refs_check_dir, iter))\n +\t\t\t\t\tret = -1;\n +\t\t\t}\n +\t\t} else {\n -+\t\t\tret = error(_(\"unexpected file type for '%s'\"),\n -+\t\t\t\t iter->basename);\n ++\t\t\tstruct fsck_ref_report report = { .path = iter->basename };\n ++\t\t\tif (fsck_report_ref(o, &report,\n ++\t\t\t\t\t FSCK_MSG_BAD_REF_FILETYPE,\n ++\t\t\t\t\t \"unexpected file type\"))\n ++\t\t\t\tret = -1;\n +\t\t}\n +\t}\n +\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +static int files_fsck_refs(struct ref_store *ref_store,\n +\t\t\t struct fsck_options *o)\n +{\n -+\tfiles_fsck_refs_fn fsck_refs_fns[]= {\n -+\t\tNULL\n ++\tfiles_fsck_refs_fn fsck_refs_fn[]= {\n ++\t\tNULL,\n +\t};\n +\n +\tif (o->verbose)\n +\t\tfprintf_ln(stderr, \"Checking references consistency\");\n -+\n -+\treturn files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fns);\n ++\treturn files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fn);\n +\n +}\n +\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n -\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n -+\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o) |\n -+\t files_fsck_refs(ref_store, o);\n ++\treturn files_fsck_refs(ref_store, o) |\n ++\t refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n }\n \n struct ref_storage_be refs_be_files = {\n 9: ee55ee5787 < -: ---------- fsck: add ref name check for files backend\n10: 9256328cbb < -: ---------- fsck: add ref content check for files backend\n -: ---------- > 11: 14f2739bd7 fsck: add ref name check for files backend\n-- \n2.45.2\n\n"},{"id":"499858","messageId":"ZqumHEP4OGR4Rr0E@ArchLinux","threadId":"61565","inReplyTo":"ZqulmWVBaeyP4blf@ArchLinux","subject":"[GSoC][PATCH v14 01/11] fsck: rename \"skiplist\" to \"skip_oids\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-01T15:13:32Z","receivedAt":"2024-08-01T15:13:10Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The \"skiplist\" field in \"fsck_options\" is related to objects. Because we\nare going to introduce ref consistency check, the \"skiplist\" name is too\ngeneral which will make the caller think \"skiplist\" is related to both\nthe refs and objects.\n\nIt may seem that for both refs and objects, we should provide a general\n\"skiplist\" here. However, the type for \"skiplist\" is `struct oidset`\nwhich is totally unsuitable for refs.\n\nTo avoid above ambiguity, rename \"skiplist\" to \"skip_oids\".\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 4 ++--\n fsck.h | 4 ++--\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex eea7145470..3f32441492 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n \t\tif (!strcmp(buf, \"skiplist\")) {\n \t\t\tif (equal == len)\n \t\t\t\tdie(\"skiplist requires a path\");\n-\t\t\toidset_parse_file(&options->skiplist, buf + equal + 1,\n+\t\t\toidset_parse_file(&options->skip_oids, buf + equal + 1,\n \t\t\t\t\t the_repository->hash_algo);\n \t\t\tbuf += len + 1;\n \t\t\tcontinue;\n@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n static int object_on_skiplist(struct fsck_options *opts,\n \t\t\t const struct object_id *oid)\n {\n-\treturn opts && oid && oidset_contains(&opts->skiplist, oid);\n+\treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n __attribute__((format (printf, 5, 6)))\ndiff --git a/fsck.h b/fsck.h\nindex 6085a384f6..bcfb2e34cd 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -136,7 +136,7 @@ struct fsck_options {\n \tfsck_error error_func;\n \tunsigned strict:1;\n \tenum fsck_msg_type *msg_type;\n-\tstruct oidset skiplist;\n+\tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n \tstruct oidset gitattributes_found;\n@@ -145,7 +145,7 @@ struct fsck_options {\n };\n \n #define FSCK_OPTIONS_DEFAULT { \\\n-\t.skiplist = OIDSET_INIT, \\\n+\t.skip_oids = OIDSET_INIT, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n-- \n2.45.2\n\n"},{"id":"499859","messageId":"ZqumLSxEVtkxzzEV@ArchLinux","threadId":"61565","inReplyTo":"ZqulmWVBaeyP4blf@ArchLinux","subject":"[GSoC][PATCH v14 02/11] fsck: make \"fsck_error\" callback generic","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-01T15:13:49Z","receivedAt":"2024-08-01T15:13:27Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The \"fsck_error\" callback is designed to report the objects-related\nerror messages. It accepts two parameter \"oid\" and \"object_type\" which\nis not generic. In order to provide a unified callback which can report\neither objects or refs, remove the objects-related parameters and add\nthe generic parameter \"void *fsck_report\".\n\nCreate a new \"fsck_object_report\" structure which incorporates the\nremoved parameters \"oid\" and \"object_type\". Then change the\ncorresponding references to adapt to new \"fsck_error\" callback.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 7 +++++--\n builtin/mktag.c | 3 +--\n fsck.c | 20 +++++++++++++-------\n fsck.h | 17 ++++++++++++-----\n object-file.c | 9 ++++-----\n 5 files changed, 35 insertions(+), 21 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..9673a08286 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -90,12 +90,15 @@ static int objerror(struct object *obj, const char *err)\n }\n \n static int fsck_error_func(struct fsck_options *o UNUSED,\n-\t\t\t const struct object_id *oid,\n-\t\t\t enum object_type object_type,\n+\t\t\t void *fsck_report,\n \t\t\t enum fsck_msg_type msg_type,\n \t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t const char *message)\n {\n+\tstruct fsck_object_report *report = fsck_report;\n+\tconst struct object_id *oid = report->oid;\n+\tenum object_type object_type = report->object_type;\n+\n \tswitch (msg_type) {\n \tcase FSCK_WARN:\n \t\t/* TRANSLATORS: e.g. warning in tree 01bfda: */\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex 4767f1a97e..c6b644219f 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -18,8 +18,7 @@ static int option_strict = 1;\n static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n \n static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n-\t\t\t\t const struct object_id *oid UNUSED,\n-\t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t void *fsck_report UNUSED,\n \t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t\t const char *message)\ndiff --git a/fsck.c b/fsck.c\nindex 3f32441492..4c1f8bc44a 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -232,6 +232,10 @@ static int report(struct fsck_options *options,\n \t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n {\n \tva_list ap;\n+\tstruct fsck_object_report report = {\n+\t\t.oid = oid,\n+\t\t.object_type = object_type\n+\t};\n \tstruct strbuf sb = STRBUF_INIT;\n \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n \tint result;\n@@ -252,7 +256,7 @@ static int report(struct fsck_options *options,\n \n \tva_start(ap, fmt);\n \tstrbuf_vaddf(&sb, fmt, ap);\n-\tresult = options->error_func(options, oid, object_type,\n+\tresult = options->error_func(options, &report,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n \tva_end(ap);\n@@ -1201,12 +1205,14 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n }\n \n int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid,\n-\t\t\tenum object_type object_type UNUSED,\n+\t\t\tvoid *fsck_report,\n \t\t\tenum fsck_msg_type msg_type,\n \t\t\tenum fsck_msg_id msg_id UNUSED,\n \t\t\tconst char *message)\n {\n+\tstruct fsck_object_report *report = fsck_report;\n+\tconst struct object_id *oid = report->oid;\n+\n \tif (msg_type == FSCK_WARN) {\n \t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n \t\treturn 0;\n@@ -1304,15 +1310,15 @@ int git_fsck_config(const char *var, const char *value,\n */\n \n int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t void *fsck_report,\n \t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message)\n {\n \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n-\t\tputs(oid_to_hex(oid));\n+\t\tstruct fsck_object_report *report = fsck_report;\n+\t\tputs(oid_to_hex(report->oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n+\treturn fsck_error_function(o, fsck_report, msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex bcfb2e34cd..303174a5d8 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -114,23 +114,30 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n \t\t\t void *data, struct fsck_options *options);\n \n-/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n+/*\n+ * Callback for reporting errors either for objects or refs. The \"fsck_report\"\n+ * is a generic pointer that can be used to pass any information.\n+ */\n typedef int (*fsck_error)(struct fsck_options *o,\n-\t\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t\t void *fsck_report,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\tvoid *fsck_report,\n \t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\tconst char *message);\n int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t void *fsck_report,\n \t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message);\n \n+struct fsck_object_report {\n+\tconst struct object_id *oid;\n+\tenum object_type object_type;\n+};\n+\n struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\ndiff --git a/object-file.c b/object-file.c\nindex 065103be3e..05ac6ebed6 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2470,11 +2470,10 @@ int repo_has_object_file(struct repository *r,\n * give more context.\n */\n static int hash_format_check_report(struct fsck_options *opts UNUSED,\n-\t\t\t\t const struct object_id *oid UNUSED,\n-\t\t\t\t enum object_type object_type UNUSED,\n-\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n-\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t\t const char *message)\n+\t\t\t\t void *fsck_report UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \terror(_(\"object fails fsck: %s\"), message);\n \treturn 1;\n-- \n2.45.2\n\n"},{"id":"499860","messageId":"ZqumN5cfsQYHlU5X@ArchLinux","threadId":"61565","inReplyTo":"ZqulmWVBaeyP4blf@ArchLinux","subject":"[GSoC][PATCH v14 03/11] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-01T15:13:59Z","receivedAt":"2024-08-01T15:13:36Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The static function \"report\" provided by \"fsck.c\" aims at checking error\ntype and calling the callback \"error_func\" to report the message. Both\nrefs and objects need to check the error type of the current fsck\nmessage. In order to extract this common behavior, create a new function\n\"fsck_vreport\". Instead of using \"...\", provide \"va_list\" to allow more\nflexibility.\n\nInstead of changing \"report\" prototype to be algin with the\n\"fsck_vreport\" function, we leave the \"report\" prototype unchanged due\nto the reason that there are nearly 62 references about \"report\"\nfunction. Simply change \"report\" function to use \"fsck_vreport\" to\nreport objects related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 49 ++++++++++++++++++++++++++++++++++---------------\n 1 file changed, 34 insertions(+), 15 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex 4c1f8bc44a..b394a9e397 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -226,16 +226,16 @@ static int object_on_skiplist(struct fsck_options *opts,\n \treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n-__attribute__((format (printf, 5, 6)))\n-static int report(struct fsck_options *options,\n-\t\t const struct object_id *oid, enum object_type object_type,\n-\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+/*\n+ * Provide the common functionality for either fscking refs or objects.\n+ * It will get the current msg error type and call the error_func callback\n+ * which is registered in the \"fsck_options\" struct.\n+ */\n+static int fsck_vreport(struct fsck_options *options,\n+\t\t\tvoid *fsck_report,\n+\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n {\n-\tva_list ap;\n-\tstruct fsck_object_report report = {\n-\t\t.oid = oid,\n-\t\t.object_type = object_type\n-\t};\n+\tva_list ap_copy;\n \tstruct strbuf sb = STRBUF_INIT;\n \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n \tint result;\n@@ -243,9 +243,6 @@ static int report(struct fsck_options *options,\n \tif (msg_type == FSCK_IGNORE)\n \t\treturn 0;\n \n-\tif (object_on_skiplist(options, oid))\n-\t\treturn 0;\n-\n \tif (msg_type == FSCK_FATAL)\n \t\tmsg_type = FSCK_ERROR;\n \telse if (msg_type == FSCK_INFO)\n@@ -254,9 +251,9 @@ static int report(struct fsck_options *options,\n \tprepare_msg_ids();\n \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n \n-\tva_start(ap, fmt);\n-\tstrbuf_vaddf(&sb, fmt, ap);\n-\tresult = options->error_func(options, &report,\n+\tva_copy(ap_copy, ap);\n+\tstrbuf_vaddf(&sb, fmt, ap_copy);\n+\tresult = options->error_func(options, fsck_report,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n \tva_end(ap);\n@@ -264,6 +261,28 @@ static int report(struct fsck_options *options,\n \treturn result;\n }\n \n+__attribute__((format (printf, 5, 6)))\n+static int report(struct fsck_options *options,\n+\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tstruct fsck_object_report report = {\n+\t\t.oid = oid,\n+\t\t.object_type = object_type\n+\t};\n+\tint result;\n+\n+\tif (object_on_skiplist(options, oid))\n+\t\treturn 0;\n+\n+\tva_start(ap, fmt);\n+\tresult = fsck_vreport(options, &report, msg_id, fmt, ap);\n+\tva_end(ap);\n+\n+\treturn result;\n+}\n+\n void fsck_enable_object_names(struct fsck_options *options)\n {\n \tif (!options->object_names)\n-- \n2.45.2\n\n"},{"id":"499861","messageId":"ZqumQthEwDjoDs6u@ArchLinux","threadId":"61565","inReplyTo":"ZqulmWVBaeyP4blf@ArchLinux","subject":"[GSoC][PATCH v14 04/11] fsck: add refs report function","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-01T15:14:10Z","receivedAt":"2024-08-01T15:13:48Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new struct \"fsck_ref_report\" to contain the information we\nneed when reporting refs-related messages.\n\nWith the new \"fsck_vreport\" function, add a new function\n\"fsck_report_ref\" to report refs-related fsck error message. Unlike\n\"report\" function uses the exact parameters, we simply pass \"struct\nfsck_ref_report *report\" as the parameter. This is because at current we\ndon't know exactly how many fields we need. By passing this parameter,\nwe don't need to change this function prototype when we want to add more\ninformation into \"fsck_ref_report\".\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 13 +++++++++++++\n fsck.h | 16 ++++++++++++++++\n 2 files changed, 29 insertions(+)\n\ndiff --git a/fsck.c b/fsck.c\nindex b394a9e397..ee888c1417 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -283,6 +283,19 @@ static int report(struct fsck_options *options,\n \treturn result;\n }\n \n+int fsck_report_ref(struct fsck_options *options,\n+\t\t struct fsck_ref_report *report,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = fsck_vreport(options, report, msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n void fsck_enable_object_names(struct fsck_options *options)\n {\n \tif (!options->object_names)\ndiff --git a/fsck.h b/fsck.h\nindex 303174a5d8..0918c28d0f 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -138,6 +138,12 @@ struct fsck_object_report {\n \tenum object_type object_type;\n };\n \n+struct fsck_ref_report {\n+\tconst char *path;\n+\tconst struct object_id *oid;\n+\tconst char *referent;\n+};\n+\n struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n@@ -216,6 +222,16 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+/*\n+ * Report an error or warning for refs.\n+ */\n+__attribute__((format (printf, 4, 5)))\n+int fsck_report_ref(struct fsck_options *options,\n+\t\t struct fsck_ref_report *report,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\n-- \n2.45.2\n\n"},{"id":"499862","messageId":"ZqumS7l6FjFz50DG@ArchLinux","threadId":"61565","inReplyTo":"ZqulmWVBaeyP4blf@ArchLinux","subject":"[GSoC][PATCH v14 05/11] fsck: add refs-related error callback","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-01T15:14:19Z","receivedAt":"2024-08-01T15:13:58Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"We have introduced \"fsck_report_ref\" function to report the error\nmessage for refs. We still need to add the corresponding callback\nfunction. Create refs-specific \"error_func\" callback\n\"fsck_refs_error_function\".\n\nLast, add \"FSCK_REFS_OPTIONS_DEFAULT\" macro to create default options\nwhen checking ref consistency.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 28 ++++++++++++++++++++++++++++\n fsck.h | 9 +++++++++\n 2 files changed, 37 insertions(+)\n\ndiff --git a/fsck.c b/fsck.c\nindex ee888c1417..f0ae760c86 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1253,6 +1253,34 @@ int fsck_error_function(struct fsck_options *o,\n \treturn 1;\n }\n \n+int fsck_refs_error_function(struct fsck_options *options UNUSED,\n+\t\t\t void *fsck_report,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tstruct fsck_ref_report *report = fsck_report;\n+\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tstrbuf_addstr(&sb, report->path);\n+\n+\tif (report->oid)\n+\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(report->oid));\n+\telse if (report->referent)\n+\t\tstrbuf_addf(&sb, \" -> (%s)\", report->referent);\n+\n+\tif (msg_type == FSCK_WARN)\n+\t\twarning(\"%s: %s\", sb.buf, message);\n+\telse\n+\t\tret = error(\"%s: %s\", sb.buf, message);\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+\n+}\n+\n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n \t\t struct fsck_options *options, const char *blob_type)\ndiff --git a/fsck.h b/fsck.h\nindex 0918c28d0f..2276ea7e34 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -133,6 +133,12 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t const char *message);\n \n+int fsck_refs_error_function(struct fsck_options *options,\n+\t\t\t void *fsck_report,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n+\n struct fsck_object_report {\n \tconst struct object_id *oid;\n \tenum object_type object_type;\n@@ -181,6 +187,9 @@ struct fsck_options {\n \t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n }\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n \n /* descend in all linked child objects\n * the return value is:\n-- \n2.45.2\n\n"},{"id":"499863","messageId":"ZqumVWxLZpBYkrGB@ArchLinux","threadId":"61565","inReplyTo":"ZqulmWVBaeyP4blf@ArchLinux","subject":"[GSoC][PATCH v14 06/11] fsck: rename objects-related fsck error functions","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-01T15:14:29Z","receivedAt":"2024-08-01T15:14:07Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The names of objects-related fsck error functions are generic. It's OK\nwhen there is only object database check. However, we have introduced\nrefs database check report function. To avoid ambiguity, rename\nobject-related fsck error functions to explicitly indicate these\nfunctions are used to report objects-related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 12 ++++++------\n fsck.c | 22 +++++++++++-----------\n fsck.h | 24 ++++++++++++------------\n 3 files changed, 29 insertions(+), 29 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 9673a08286..766bbd014d 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -89,11 +89,11 @@ static int objerror(struct object *obj, const char *err)\n \treturn -1;\n }\n \n-static int fsck_error_func(struct fsck_options *o UNUSED,\n-\t\t\t void *fsck_report,\n-\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t const char *message)\n+static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n+\t\t\t\t void *fsck_report,\n+\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \tstruct fsck_object_report *report = fsck_report;\n \tconst struct object_id *oid = report->oid;\n@@ -941,7 +941,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tfsck_walk_options.walk = mark_object;\n \tfsck_obj_options.walk = mark_used;\n-\tfsck_obj_options.error_func = fsck_error_func;\n+\tfsck_obj_options.error_func = fsck_objects_error_func;\n \tif (check_strict)\n \t\tfsck_obj_options.strict = 1;\n \ndiff --git a/fsck.c b/fsck.c\nindex f0ae760c86..0a870fcec0 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1236,11 +1236,11 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tvoid *fsck_report,\n-\t\t\tenum fsck_msg_type msg_type,\n-\t\t\tenum fsck_msg_id msg_id UNUSED,\n-\t\t\tconst char *message)\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tvoid *fsck_report,\n+\t\t\t\tenum fsck_msg_type msg_type,\n+\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n+\t\t\t\tconst char *message)\n {\n \tstruct fsck_object_report *report = fsck_report;\n \tconst struct object_id *oid = report->oid;\n@@ -1369,16 +1369,16 @@ int git_fsck_config(const char *var, const char *value,\n * Custom error callbacks that are used in more than one place.\n */\n \n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t void *fsck_report,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message)\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t void *fsck_report,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message)\n {\n \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n \t\tstruct fsck_object_report *report = fsck_report;\n \t\tputs(oid_to_hex(report->oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_error_function(o, fsck_report, msg_type, msg_id, message);\n+\treturn fsck_objects_error_function(o, fsck_report, msg_type,msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex 2276ea7e34..2002590f60 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -123,15 +123,15 @@ typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tvoid *fsck_report,\n-\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n-\t\t\tconst char *message);\n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t void *fsck_report,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message);\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tvoid *fsck_report,\n+\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n+\t\t\t\tconst char *message);\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t void *fsck_report,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message);\n \n int fsck_refs_error_function(struct fsck_options *options,\n \t\t\t void *fsck_report,\n@@ -169,7 +169,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function \\\n+\t.error_func = fsck_objects_error_function \\\n }\n #define FSCK_OPTIONS_STRICT { \\\n \t.strict = 1, \\\n@@ -177,7 +177,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function, \\\n+\t.error_func = fsck_objects_error_function, \\\n }\n #define FSCK_OPTIONS_MISSING_GITMODULES { \\\n \t.strict = 1, \\\n@@ -185,7 +185,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n+\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n #define FSCK_REFS_OPTIONS_DEFAULT { \\\n \t.error_func = fsck_refs_error_function, \\\n-- \n2.45.2\n\n"},{"id":"499864","messageId":"ZqumXiEAc_oqdfAp@ArchLinux","threadId":"61565","inReplyTo":"ZqulmWVBaeyP4blf@ArchLinux","subject":"[GSoC][PATCH v14 07/11] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-01T15:14:38Z","receivedAt":"2024-08-01T15:14:19Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The \"struct ref_store\" is the base class which contains the \"be\" pointer\nwhich provides backend-specific functions whose interfaces are defined\nin the \"ref_storage_be\". We could reuse this polymorphism to define only\none interface. For every backend, we need to provide its own function\npointer.\n\nThe interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It's organized as the five parts:\n\n1. The name and the initialization interfaces.\n2. The ref transaction interfaces.\n3. The ref internal interfaces (pack, rename and copy).\n4. The ref filesystem interfaces.\n5. The reflog related interfaces.\n\nTo keep consistent with the git-fsck(1), add a new interface named\n\"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\ngrouped into any above five categories. Explicitly add blank line to\nmake it different from others.\n\nLast, implement placeholder functions for each ref backends.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs.c | 5 +++++\n refs.h | 8 ++++++++\n refs/debug.c | 11 +++++++++++\n refs/files-backend.c | 13 ++++++++++++-\n refs/packed-backend.c | 8 ++++++++\n refs/refs-internal.h | 6 ++++++\n refs/reftable-backend.c | 8 ++++++++\n 7 files changed, 58 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex 915aeb4d1d..6f642dc681 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs, struct fsck_options *o)\n+{\n+\treturn refs->be->fsck(refs, o);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex b3e39bc257..405073621a 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -4,6 +4,7 @@\n #include \"commit.h\"\n #include \"repository.h\"\n \n+struct fsck_options;\n struct object_id;\n struct ref_store;\n struct strbuf;\n@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_options *o);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..45e2e784a0 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tint res = drefs->refs->be->fsck(drefs->refs, o);\n+\ttrace_printf_key(&trace_refs, \"fsck: %d\\n\", res);\n+\treturn res;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex aa52d9be7c..4630eb1f80 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3408,6 +3408,15 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_init,\n@@ -3434,5 +3443,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a0666407cd..5209b0b212 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_init,\n@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex fa975d69aa..a905e187cd 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_options;\n struct ref_transaction;\n \n /*\n@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_options *o);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -677,6 +681,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex fbe74c239d..b5a1a526df 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.45.2\n\n"},{"id":"499865","messageId":"Zqumaq_iwFkwMU6z@ArchLinux","threadId":"61565","inReplyTo":"ZqulmWVBaeyP4blf@ArchLinux","subject":"[GSoC][PATCH v14 08/11] builtin/refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-01T15:14:50Z","receivedAt":"2024-08-01T15:14:29Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\ncheck the reference database consistency and also this subcommand will\nbe used as the entry point of checking refs for \"git-fsck(1)\".\n\nAdd \"verbose\" field into \"fsck_options\" to indicate whether we should\nprint verbose messages when checking refs and objects consistency.\n\nRemove bit-field for \"strict\" field, this is because we cannot take\naddress of a bit-field which makes it unhandy to set member variables\nwhen parsing the command line options.\n\nThe \"git-fsck(1)\" declares \"fsck_options\" variable with \"static\"\nidentifier which avoids complaint by the leak-checker. However, in\n\"git-refs verify\", we need to do memory clean manually. Thus add\n\"fsck_options_clear\" function in \"fsck.c\" to provide memory clean\noperation.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/git-refs.txt | 13 +++++++++++++\n builtin/refs.c | 34 ++++++++++++++++++++++++++++++++++\n fsck.c | 11 +++++++++++\n fsck.h | 8 +++++++-\n 4 files changed, 65 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..ce31f93061 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+The following options are specific to 'git refs verify':\n+\n+--strict::\n+\tEnable stricter error checking. This will cause warnings to be\n+\treported as errors. See linkgit:git-fsck[1].\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..131f98be98 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,44 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"verbose\", &fsck_refs_options.verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &fsck_refs_options.strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\tint ret;\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"'git refs verify' takes no arguments\"));\n+\n+\tgit_config(git_fsck_config, &fsck_refs_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n+\n+\tfsck_options_clear(&fsck_refs_options);\n+\treturn ret;\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/fsck.c b/fsck.c\nindex 0a870fcec0..d5e7c88eab 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1336,6 +1336,17 @@ int fsck_finish(struct fsck_options *options)\n \treturn ret;\n }\n \n+void fsck_options_clear(struct fsck_options *options)\n+{\n+\tfree(options->msg_type);\n+\toidset_clear(&options->skip_oids);\n+\toidset_clear(&options->gitmodules_found);\n+\toidset_clear(&options->gitmodules_done);\n+\toidset_clear(&options->gitattributes_found);\n+\toidset_clear(&options->gitattributes_done);\n+\tkh_clear_oid_map(options->object_names);\n+}\n+\n int git_fsck_config(const char *var, const char *value,\n \t\t const struct config_context *ctx, void *cb)\n {\ndiff --git a/fsck.h b/fsck.h\nindex 2002590f60..d551a9fe86 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -153,7 +153,8 @@ struct fsck_ref_report {\n struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n-\tunsigned strict:1;\n+\tunsigned strict;\n+\tunsigned verbose;\n \tenum fsck_msg_type *msg_type;\n \tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n@@ -231,6 +232,11 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+/*\n+ * Clear the fsck_options struct, freeing any allocated memory.\n+ */\n+void fsck_options_clear(struct fsck_options *options);\n+\n /*\n * Report an error or warning for refs.\n */\n-- \n2.45.2\n\n"},{"id":"499866","messageId":"ZqumdJz3-0mqh6Rc@ArchLinux","threadId":"61565","inReplyTo":"ZqulmWVBaeyP4blf@ArchLinux","subject":"[GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-01T15:15:00Z","receivedAt":"2024-08-01T15:14:38Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new function \"fsck_refs\" that initializes and runs a child\nprocess to execute the \"git-refs verify\" command.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 766bbd014d..b6ac878270 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -899,6 +899,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \treturn res;\n }\n \n+static void fsck_refs(void)\n+{\n+\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n+\tchild_process_init(&refs_verify);\n+\trefs_verify.git_cmd = 1;\n+\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n+\tif (verbose)\n+\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n+\tif (check_strict)\n+\t\tstrvec_push(&refs_verify.args, \"--strict\");\n+\n+\tif (run_command(&refs_verify))\n+\t\terrors_found |= ERROR_REFS;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n@@ -1068,6 +1083,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n+\tfsck_refs();\n+\n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \n-- \n2.45.2\n\n"},{"id":"499867","messageId":"ZqumgrIo7pJ--reA@ArchLinux","threadId":"61565","inReplyTo":"ZqulmWVBaeyP4blf@ArchLinux","subject":"[GSoC][PATCH v14 10/11] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-01T15:15:14Z","receivedAt":"2024-08-01T15:14:53Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"For refs and reflogs, we need to scan its corresponding directories to\ncheck every regular file or symbolic link which shares the same pattern.\nIntroduce a unified interface for scanning directories for\nfiles-backend.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 ++\n fsck.h | 1 +\n refs/files-backend.c | 75 ++++++++++++++++++++++++++++++++++-\n 3 files changed, 78 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex f643585a34..7c809fddf1 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,9 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefFiletype`::\n+\t(ERROR) A ref has a bad file type.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex d551a9fe86..af02174973 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_FILETYPE, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 4630eb1f80..5574e78656 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -6,6 +6,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -3408,13 +3409,85 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n+\t\t\t\t const char *gitdir,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fn)\n+{\n+\tconst char *gitdir = ref_store->gitdir;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\tif (!iter) {\n+\t\tret = error_errno(_(\"cannot open directory %s\"), sb.buf);\n+\t\tgoto out;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->verbose)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fn[i]; i++) {\n+\t\t\t\tif (fsck_refs_fn[i](o, gitdir, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tstruct fsck_ref_report report = { .path = iter->basename };\n+\t\t\tif (fsck_report_ref(o, &report,\n+\t\t\t\t\t FSCK_MSG_BAD_REF_FILETYPE,\n+\t\t\t\t\t \"unexpected file type\"))\n+\t\t\t\tret = -1;\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n+\n+out:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\tfiles_fsck_refs_fn fsck_refs_fn[]= {\n+\t\tNULL,\n+\t};\n+\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, \"Checking references consistency\");\n+\treturn files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fn);\n+\n+}\n+\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_options *o)\n {\n \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n-\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\treturn files_fsck_refs(ref_store, o) |\n+\t refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n }\n \n struct ref_storage_be refs_be_files = {\n-- \n2.45.2\n\n"},{"id":"499868","messageId":"ZqumjDzfeaLZQgr5@ArchLinux","threadId":"61565","inReplyTo":"ZqulmWVBaeyP4blf@ArchLinux","subject":"[GSoC][PATCH v14 11/11] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-01T15:15:24Z","receivedAt":"2024-08-01T15:15:02Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) only implicitly checks the reference, it does not fully\ncheck refs with bad format name such as standalone \"@\".\n\nHowever, a file ending with \".lock\" should not be marked as having a bad\nref name. It is expected that concurrent writers may have such lock files.\nWe currently ignore this situation. But for bare \".lock\" file, we will\nreport it as error.\n\nIn order to provide such checks, add a new fsck message id \"badRefName\"\nwith default ERROR type. Use existing \"check_refname_format\" to explicit\ncheck the ref name. And add a new unit test to verify the functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 ++\n fsck.h | 1 +\n refs/files-backend.c | 32 ++++++++++++\n t/t0602-reffiles-fsck.sh | 94 +++++++++++++++++++++++++++++++++++\n 4 files changed, 130 insertions(+)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex 7c809fddf1..68a2801f15 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -22,6 +22,9 @@\n `badRefFiletype`::\n \t(ERROR) A ref has a bad file type.\n \n+`badRefName`::\n+\t(ERROR) A ref has an invalid format.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex af02174973..500b4c04d2 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -32,6 +32,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_FILETYPE, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 5574e78656..1186b6cbb1 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3419,6 +3419,37 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n+static int files_fsck_refs_name(struct fsck_options *o,\n+\t\t\t\tconst char *gitdir UNUSED,\n+\t\t\t\tconst char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tsize_t len = 0;\n+\tint ret = 0;\n+\n+\t/*\n+\t * Ignore the files ending with \".lock\" as they may be lock files\n+\t * However, do not allow bare \".lock\" files.\n+\t */\n+\tif (strip_suffix(iter->basename, \".lock\", &len) && (len != 0))\n+\t\tgoto clean;\n+\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstruct fsck_ref_report report = { .path = NULL };\n+\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\treport.path = sb.buf;\n+\t\tret = fsck_report_ref(o, &report,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n+\t}\n+\n+clean:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3471,6 +3502,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o)\n {\n \tfiles_fsck_refs_fn fsck_refs_fn[]= {\n+\t\tfiles_fsck_refs_name,\n \t\tNULL,\n \t};\n \ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..2be28427ab\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,94 @@\n+#!/bin/sh\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+TEST_PASSES_SANITIZE_LEAK=true\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\tcd repo &&\n+\n+\tgit commit --allow-empty -m initial &&\n+\tgit checkout -b branch-1 &&\n+\tgit tag tag-1 &&\n+\tgit commit --allow-empty -m second &&\n+\tgit checkout -b branch-2 &&\n+\tgit tag tag-2 &&\n+\tgit tag multi_hierarchy/tag-2 &&\n+\n+\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\ttest_must_fail git fsck 2>err &&\n+\tcat >expect <<-EOF &&\n+\terror: refs/heads/.branch-1: badRefName: invalid refname format\n+\tEOF\n+\trm $branch_dir_prefix/.branch-1 &&\n+\ttest_cmp expect err &&\n+\n+\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\ttest_must_fail git fsck 2>err &&\n+\tcat >expect <<-EOF &&\n+\terror: refs/heads/@: badRefName: invalid refname format\n+\tEOF\n+\trm $branch_dir_prefix/@ &&\n+\ttest_cmp expect err &&\n+\n+\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\ttest_must_fail git fsck 2>err &&\n+\tcat >expect <<-EOF &&\n+\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\tEOF\n+\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\ttest_cmp expect err &&\n+\n+\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\tgit fsck 2>err &&\n+\trm $tag_dir_prefix/tag-1.lock &&\n+\ttest_must_be_empty err &&\n+\n+\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/.lock &&\n+\ttest_must_fail git fsck 2>err &&\n+\tcat >expect <<-EOF &&\n+\terror: refs/tags/.lock: badRefName: invalid refname format\n+\tEOF\n+\trm $tag_dir_prefix/.lock &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\tcd repo &&\n+\tgit commit --allow-empty -m initial &&\n+\tgit checkout -b branch-1 &&\n+\tgit tag tag-1 &&\n+\tgit commit --allow-empty -m second &&\n+\tgit checkout -b branch-2 &&\n+\tgit tag tag-2 &&\n+\n+\n+\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\tgit -c fsck.badRefName=warn fsck 2>err &&\n+\tcat >expect <<-EOF &&\n+\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\tEOF\n+\trm $branch_dir_prefix/.branch-1 &&\n+\ttest_cmp expect err &&\n+\n+\n+\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\tgit -c fsck.badRefName=ignore fsck 2>err &&\n+\ttest_must_be_empty err\n+'\n+\n+test_done\n-- \n2.45.2\n\n"},{"id":"500065","messageId":"ZrDMXgEuE8yeSzkw@tanuki","threadId":"61565","inReplyTo":"ZqumN5cfsQYHlU5X@ArchLinux","subject":"Re: [GSoC][PATCH v14 03/11] fsck: add a unified interface for reporting fsck messages","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-05T12:58:06Z","receivedAt":"2024-08-05T12:58:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 01, 2024 at 11:13:59PM +0800, shejialuo wrote:\n> @@ -254,9 +251,9 @@ static int report(struct fsck_options *options,\n> \tprepare_msg_ids();\n> \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n> \n> -\tva_start(ap, fmt);\n> -\tstrbuf_vaddf(&sb, fmt, ap);\n> -\tresult = options->error_func(options, &report,\n> +\tva_copy(ap_copy, ap);\n\nCan't we use `ap` directly instead of copying it? We'd have to get rid\nof the call to `va_end` as our caller already does that, but other than\nthat I don't see any reason to copy the argument list here.\n\n> +\tstrbuf_vaddf(&sb, fmt, ap_copy);\n> +\tresult = options->error_func(options, fsck_report,\n> \t\t\t\t msg_type, msg_id, sb.buf);\n> \tstrbuf_release(&sb);\n> \tva_end(ap);\n\nPatrick\n"},{"id":"500066","messageId":"ZrDMZN-73ZxF95Gb@tanuki","threadId":"61565","inReplyTo":"ZqumQthEwDjoDs6u@ArchLinux","subject":"Re: [GSoC][PATCH v14 04/11] fsck: add refs report function","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-05T12:58:12Z","receivedAt":"2024-08-05T12:58:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 01, 2024 at 11:14:10PM +0800, shejialuo wrote:\n> @@ -216,6 +222,16 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n> */\n> int fsck_finish(struct fsck_options *options);\n> \n> +/*\n> + * Report an error or warning for refs.\n> + */\n> +__attribute__((format (printf, 4, 5)))\n> +int fsck_report_ref(struct fsck_options *options,\n> +\t\t struct fsck_ref_report *report,\n> +\t\t enum fsck_msg_id msg_id,\n> +\t\t const char *fmt, ...);\n\nNice that we got rid of the parts that don't matter for reporting refs\nnow.\n\nPatrick\n"},{"id":"500067","messageId":"ZrDMaegRZSjP9twB@tanuki","threadId":"61565","inReplyTo":"ZqumS7l6FjFz50DG@ArchLinux","subject":"Re: [GSoC][PATCH v14 05/11] fsck: add refs-related error callback","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-05T12:58:17Z","receivedAt":"2024-08-05T12:58:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 01, 2024 at 11:14:19PM +0800, shejialuo wrote:\n> We have introduced \"fsck_report_ref\" function to report the error\n> message for refs. We still need to add the corresponding callback\n> function. Create refs-specific \"error_func\" callback\n> \"fsck_refs_error_function\".\n> \n> Last, add \"FSCK_REFS_OPTIONS_DEFAULT\" macro to create default options\n> when checking ref consistency.\n\nNit: personally, I'd squash this commit into the preceding one. They\nbelong together logically, as the preceding step doesn't make any sense\nwithout the error reporting function and this step here doesn't make any\nsense without introducing `struct fsck_ref_report`.\n\nPatrick\n"},{"id":"500068","messageId":"ZrDMbnIgGNK8F99N@tanuki","threadId":"61565","inReplyTo":"ZqumVWxLZpBYkrGB@ArchLinux","subject":"Re: [GSoC][PATCH v14 06/11] fsck: rename objects-related fsck error functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-05T12:58:22Z","receivedAt":"2024-08-05T12:58:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 01, 2024 at 11:14:29PM +0800, shejialuo wrote:\n> The names of objects-related fsck error functions are generic. It's OK\n> when there is only object database check. However, we have introduced\n> refs database check report function. To avoid ambiguity, rename\n> object-related fsck error functions to explicitly indicate these\n> functions are used to report objects-related messages.\n\nNit: I think it would be a bit easier to follow if you moved this step\na bit earlier, namely before you introduce the ref-related infra.\n\nPatrick\n"},{"id":"500069","messageId":"ZrDMdEJR6DV5HyLD@tanuki","threadId":"61565","inReplyTo":"ZqumdJz3-0mqh6Rc@ArchLinux","subject":"Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-05T12:58:28Z","receivedAt":"2024-08-05T12:58:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 01, 2024 at 11:15:00PM +0800, shejialuo wrote:\n> Introduce a new function \"fsck_refs\" that initializes and runs a child\n> process to execute the \"git-refs verify\" command.\n\nIt's `git refs verify`, not `git-refs verify` both in the commit body\nand subject.\n\n> Mentored-by: Patrick Steinhardt \n> Mentored-by: Karthik Nayak \n> Signed-off-by: shejialuo \n> ---\n> builtin/fsck.c | 17 +++++++++++++++++\n> 1 file changed, 17 insertions(+)\n> \n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index 766bbd014d..b6ac878270 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -899,6 +899,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n> \treturn res;\n> }\n> \n> +static void fsck_refs(void)\n> +{\n> +\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n> +\tchild_process_init(&refs_verify);\n> +\trefs_verify.git_cmd = 1;\n> +\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n> +\tif (verbose)\n> +\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n> +\tif (check_strict)\n> +\t\tstrvec_push(&refs_verify.args, \"--strict\");\n> +\n> +\tif (run_command(&refs_verify))\n> +\t\terrors_found |= ERROR_REFS;\n> +}\n\nOkay. I think that it's sensible to execute this as part of git-fsck(1).\nBut do we want to provide an option to disable this new check, as well?\n\nIt does feel a bit like opening a can of worms, though. None of the\nother checks have trivial ways to disable them, and git-fsck(1) is\ngaining more and more checks. So if we can disable ref checks, we may\nalso want to have options to disable checks for objects, connectivity,\nreverse indices, indices, commit graphs and whatnot. In other words, in\nmy opinion we need to think a bit bigger and design a proper UI around\nthis.\n\nBut I don't think that should happen as part of this commit series, as\nit is already big enough. So either we just accept this patch as-is. Or\nwe evict it from this series and handle it in the future together with\nall the other taks that one may potentially want to disable.\n\nI'd rather pick option two.\n\nPatrick\n"},{"id":"500070","messageId":"ZrDMeumizT-xKLrF@tanuki","threadId":"61565","inReplyTo":"ZqumgrIo7pJ--reA@ArchLinux","subject":"Re: [GSoC][PATCH v14 10/11] files-backend: add unified interface for refs scanning","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-05T12:58:34Z","receivedAt":"2024-08-05T12:58:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 01, 2024 at 11:15:14PM +0800, shejialuo wrote:\n> +static int files_fsck_refs(struct ref_store *ref_store,\n> +\t\t\t struct fsck_options *o)\n> +{\n> +\tfiles_fsck_refs_fn fsck_refs_fn[]= {\n> +\t\tNULL,\n> +\t};\n> +\n> +\tif (o->verbose)\n> +\t\tfprintf_ln(stderr, \"Checking references consistency\");\n\nThis string should be marked for translation.\n\nPatrick\n"},{"id":"500071","messageId":"ZrDMghZ3Vsuor-rO@tanuki","threadId":"61565","inReplyTo":"ZqumjDzfeaLZQgr5@ArchLinux","subject":"Re: [GSoC][PATCH v14 11/11] fsck: add ref name check for files backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-05T12:58:42Z","receivedAt":"2024-08-05T12:58:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 01, 2024 at 11:15:24PM +0800, shejialuo wrote:\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 5574e78656..1186b6cbb1 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -3419,6 +3419,37 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n> \t\t\t\t const char *refs_check_dir,\n> \t\t\t\t struct dir_iterator *iter);\n> \n> +static int files_fsck_refs_name(struct fsck_options *o,\n> +\t\t\t\tconst char *gitdir UNUSED,\n> +\t\t\t\tconst char *refs_check_dir,\n> +\t\t\t\tstruct dir_iterator *iter)\n> +{\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\tsize_t len = 0;\n> +\tint ret = 0;\n> +\n> +\t/*\n> +\t * Ignore the files ending with \".lock\" as they may be lock files\n> +\t * However, do not allow bare \".lock\" files.\n> +\t */\n> +\tif (strip_suffix(iter->basename, \".lock\", &len) && (len != 0))\n> +\t\tgoto clean;\n\nBetter:\n\n if (ends_with(iter->basename, \".lock))\n goto cleanup;\n\nPatrick\n"},{"id":"500072","messageId":"ZrDMiXX5KhA9WVNI@tanuki","threadId":"61565","inReplyTo":"ZqulmWVBaeyP4blf@ArchLinux","subject":"Re: [GSoC][PATCH v14 00/11] ref consistency check infra setup","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-05T12:58:49Z","receivedAt":"2024-08-05T12:58:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 01, 2024 at 11:11:21PM +0800, shejialuo wrote:\n> Hi All,\n> \n> This version does the following things:\n> \n> 1. By following the advice from Patrick, we should make the callback\n> function be generic by adding only one \"void * fsck_report\" parameter.\n> Thus the commit sequence will be much more clearer. And it wll be much\n> easier for reviewers to review. And I have split the commit into more\n> commits in this version.\n> 2. Enhance the commit messages to provide more context about why we\n> should do this.\n> 3. Patrick advices that we should initialize the \"fsck_options\" member\n> when parsing the options. However, because the original \"strict\" and\n> \"verbose\" field are defined as the bit field, we cannot take the address\n> of them. So I simply remove the bit field.\n> 4. As Patrick said, \".lock\" should not be reported as error. At current,\n> ignore files ending with \".lock\".\n> 5. Add a fsck msg type called \"badRefFiletype\" which indicates that a\n> ref has a bad file type when scanning the directory.\n> 6. Junio advices instead of using \"fsck_refs_fns\", we should use the\n> singular version \"fsck_refs_fn\", fix this.\n> 7. Drop the last patch because in this series, we mainly focus on the\n> infra, I will add a series later to add ref content check.\n> \n> However, there is one thing holding. Junio advices that I should\n> follow the Patrick's change to change the prototype of \"files_fsck_refs_fn\"\n> \n> https://lore.kernel.org/git/fe0e2c3617c8040c632dbc3de613a1d22e8070f7.1722316795.git.ps@pks.im/\n> \n> However, at current, this patch is not in the cooking tree. I will\n> handle this later.\n\nThis looks very good to me. I've got another set of smallish nits, but\nall of those should be quick to address, I think. I think that the next\nversion should likely be ready to go.\n\nThanks!\n\nPatrick\n"},{"id":"500085","messageId":"ZrDrUp0jeeIOnn0D@ArchLinux","threadId":"61565","inReplyTo":"ZrDMXgEuE8yeSzkw@tanuki","subject":"Re: [GSoC][PATCH v14 03/11] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-05T15:10:10Z","receivedAt":"2024-08-05T15:09:43Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Mon, Aug 05, 2024 at 02:58:06PM +0200, Patrick Steinhardt wrote:\n> On Thu, Aug 01, 2024 at 11:13:59PM +0800, shejialuo wrote:\n> > @@ -254,9 +251,9 @@ static int report(struct fsck_options *options,\n> > \tprepare_msg_ids();\n> > \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n> > \n> > -\tva_start(ap, fmt);\n> > -\tstrbuf_vaddf(&sb, fmt, ap);\n> > -\tresult = options->error_func(options, &report,\n> > +\tva_copy(ap_copy, ap);\n> \n> Can't we use `ap` directly instead of copying it? We'd have to get rid\n> of the call to `va_end` as our caller already does that, but other than\n> that I don't see any reason to copy the argument list here.\n> \n\nThanks, Patrick! This is right. I will fix this in the next version.\n\n> > +\tstrbuf_vaddf(&sb, fmt, ap_copy);\n> > +\tresult = options->error_func(options, fsck_report,\n> > \t\t\t\t msg_type, msg_id, sb.buf);\n> > \tstrbuf_release(&sb);\n> > \tva_end(ap);\n> \n> Patrick\n\n\n"},{"id":"500086","messageId":"ZrDtVJYoJJZDesB4@ArchLinux","threadId":"61565","inReplyTo":"ZrDMdEJR6DV5HyLD@tanuki","subject":"Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-05T15:18:44Z","receivedAt":"2024-08-05T15:18:18Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"> > +static void fsck_refs(void)\n> > +{\n> > +\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n> > +\tchild_process_init(&refs_verify);\n> > +\trefs_verify.git_cmd = 1;\n> > +\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n> > +\tif (verbose)\n> > +\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n> > +\tif (check_strict)\n> > +\t\tstrvec_push(&refs_verify.args, \"--strict\");\n> > +\n> > +\tif (run_command(&refs_verify))\n> > +\t\terrors_found |= ERROR_REFS;\n> > +}\n> \n> Okay. I think that it's sensible to execute this as part of git-fsck(1).\n> But do we want to provide an option to disable this new check, as well?\n> \n> It does feel a bit like opening a can of worms, though. None of the\n> other checks have trivial ways to disable them, and git-fsck(1) is\n> gaining more and more checks. So if we can disable ref checks, we may\n> also want to have options to disable checks for objects, connectivity,\n> reverse indices, indices, commit graphs and whatnot. In other words, in\n> my opinion we need to think a bit bigger and design a proper UI around\n> this.\n> \n> But I don't think that should happen as part of this commit series, as\n> it is already big enough. So either we just accept this patch as-is. Or\n> we evict it from this series and handle it in the future together with\n> all the other taks that one may potentially want to disable.\n> \n> I'd rather pick option two.\n> \n\nAfter talking with Patrick offline, we decide to drop this patch. At\ncurrent, we should put this change slowly for the user. Because many\npeople use \"git-fsck(1)\", currently we don't have a way to disable ref\nchecks by default. It's a little beyond this series.\n\nWe may consider later.\n\n> Patrick\n"},{"id":"500099","messageId":"ZrEBKjzbyxtMdCCx@ArchLinux","threadId":"61565","inReplyTo":"ZqulmWVBaeyP4blf@ArchLinux","subject":"[GSoC][PATCH v15 0/9] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-05T16:43:22Z","receivedAt":"2024-08-05T16:42:56Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Hi All:\n\nThis version handles the following problems:\n\n1. Patrick advices that I should not use `va_copy` in the changed\n`report` function. Actually this is a mistake, this version avoids\nredundant `ap` copy.\n2. Patrick advices I should rebase [v14 05/11] into [v14 04/11]. I\nfollow this advice in this version.\n3. Patrick advices that we should put [v14 06/11] before we introduce\nref-related operations. This version reorders the commit sequence. It's\na minor change.\n4. Patrick suggests at current we should not add `git refs verify`\ncommand into \"git-fsck(1)\". This is because we should disable this new\ncheck by default for the users. Many users use \"git-fsck(1)\" in their daily\nworkflow. We should not be aggressive. However, if we provide this\nmechanism in this series, we will again make more complexity. So this\nversion drop patch [v14 09/11]. Also because of dropping, change the\ntest file to use \"git refs verify\" command instead of \"git fsck\"\ncommand.\n5. Patrick suggests that we should use `ends_with` instead of\n`strip_suffix`, fix.\n\nThere is another important problem this patch solves:\n\nAt v13, Junio has suggested that the `files_fsck_refs_fn` should be\nadapted to Patrick's change. Actually, I made a bad design before. I\nshould always pass the `ref_store` structure. So I change it to\n\n -typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n -\t\t\t\t const char *gitdir,\n +typedef int (*files_fsck_refs_fn)(struct ref_store *ref_store,\n +\t\t\t\t struct fsck_options *o,\n const char *refs_check_dir,\n struct dir_iterator *iter);\n\n`gitdir` could be got by using `ref_store` parameter. By using\n`ref_store` parameter, we provide extensibility here. If something else\nchange, we merely need to change \"files_fsck_refs_fn\" prototype.\n\nBecause I drop one patch and rebase one patch. I provide the `interdiff`\nfor reviewers to make the life easier.\n\nDue to the deadline of the GSoC, I will speed up the review feedback\nprocess.\n\nThanks,\nJialuo\n\nshejialuo (9):\n fsck: rename \"skiplist\" to \"skip_oids\"\n fsck: rename objects-related fsck error functions\n fsck: make \"fsck_error\" callback generic\n fsck: add a unified interface for reporting fsck messages\n fsck: add refs report function\n refs: set up ref consistency check infrastructure\n builtin/refs: add verify subcommand\n files-backend: add unified interface for refs scanning\n fsck: add ref name check for files backend\n\n Documentation/fsck-msgids.txt | 6 ++\n Documentation/git-refs.txt | 13 ++++\n builtin/fsck.c | 17 +++--\n builtin/mktag.c | 3 +-\n builtin/refs.c | 34 +++++++++\n fsck.c | 127 +++++++++++++++++++++++++++-------\n fsck.h | 76 +++++++++++++++-----\n object-file.c | 9 ++-\n refs.c | 5 ++\n refs.h | 8 +++\n refs/debug.c | 11 +++\n refs/files-backend.c | 116 ++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 +++\n refs/refs-internal.h | 6 ++\n refs/reftable-backend.c | 8 +++\n t/t0602-reffiles-fsck.sh | 92 ++++++++++++++++++++++++\n 16 files changed, 480 insertions(+), 59 deletions(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\nInterdiff against v14:\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex b6ac878270..766bbd014d 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -899,21 +899,6 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \treturn res;\n }\n \n-static void fsck_refs(void)\n-{\n-\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n-\tchild_process_init(&refs_verify);\n-\trefs_verify.git_cmd = 1;\n-\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n-\tif (verbose)\n-\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n-\tif (check_strict)\n-\t\tstrvec_push(&refs_verify.args, \"--strict\");\n-\n-\tif (run_command(&refs_verify))\n-\t\terrors_found |= ERROR_REFS;\n-}\n-\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\\n\"\n \t \" [--[no-]full] [--strict] [--verbose] [--lost-found]\\n\"\n@@ -1083,8 +1068,6 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tcheck_connectivity();\n \n-\tfsck_refs();\n-\n \tif (the_repository->settings.core_commit_graph) {\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \ndiff --git a/fsck.c b/fsck.c\nindex d5e7c88eab..7eb5cdefdd 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -235,7 +235,6 @@ static int fsck_vreport(struct fsck_options *options,\n \t\t\tvoid *fsck_report,\n \t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n {\n-\tva_list ap_copy;\n \tstruct strbuf sb = STRBUF_INIT;\n \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n \tint result;\n@@ -251,12 +250,10 @@ static int fsck_vreport(struct fsck_options *options,\n \tprepare_msg_ids();\n \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n \n-\tva_copy(ap_copy, ap);\n-\tstrbuf_vaddf(&sb, fmt, ap_copy);\n+\tstrbuf_vaddf(&sb, fmt, ap);\n \tresult = options->error_func(options, fsck_report,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n-\tva_end(ap);\n \n \treturn result;\n }\n@@ -1391,5 +1388,6 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\tputs(oid_to_hex(report->oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_objects_error_function(o, fsck_report, msg_type,msg_id, message);\n+\treturn fsck_objects_error_function(o, fsck_report,\n+\t\t\t\t\t msg_type, msg_id, message);\n }\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 1186b6cbb1..6e6b47251d 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3414,26 +3414,25 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n * the whole directory. This function is used as the callback for each\n * regular file or symlink in the directory.\n */\n-typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n-\t\t\t\t const char *gitdir,\n+typedef int (*files_fsck_refs_fn)(struct ref_store *ref_store,\n+\t\t\t\t struct fsck_options *o,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n-static int files_fsck_refs_name(struct fsck_options *o,\n-\t\t\t\tconst char *gitdir UNUSED,\n+static int files_fsck_refs_name(struct ref_store *ref_store UNUSED,\n+\t\t\t\tstruct fsck_options *o,\n \t\t\t\tconst char *refs_check_dir,\n \t\t\t\tstruct dir_iterator *iter)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n-\tsize_t len = 0;\n \tint ret = 0;\n \n \t/*\n \t * Ignore the files ending with \".lock\" as they may be lock files\n \t * However, do not allow bare \".lock\" files.\n \t */\n-\tif (strip_suffix(iter->basename, \".lock\", &len) && (len != 0))\n-\t\tgoto clean;\n+\tif (iter->basename[0] != '.' && ends_with(iter->basename, \".lock\"))\n+\t\tgoto cleanup;\n \n \tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n \t\tstruct fsck_ref_report report = { .path = NULL };\n@@ -3445,7 +3444,7 @@ static int files_fsck_refs_name(struct fsck_options *o,\n \t\t\t\t \"invalid refname format\");\n \t}\n \n-clean:\n+cleanup:\n \tstrbuf_release(&sb);\n \treturn ret;\n }\n@@ -3455,13 +3454,12 @@ static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t const char *refs_check_dir,\n \t\t\t files_fsck_refs_fn *fsck_refs_fn)\n {\n-\tconst char *gitdir = ref_store->gitdir;\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct dir_iterator *iter;\n \tint iter_status;\n \tint ret = 0;\n \n-\tstrbuf_addf(&sb, \"%s/%s\", gitdir, refs_check_dir);\n+\tstrbuf_addf(&sb, \"%s/%s\", ref_store->gitdir, refs_check_dir);\n \n \titer = dir_iterator_begin(sb.buf, 0);\n \tif (!iter) {\n@@ -3478,7 +3476,7 @@ static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n \t\t\t\t\t refs_check_dir, iter->relative_path);\n \t\t\tfor (size_t i = 0; fsck_refs_fn[i]; i++) {\n-\t\t\t\tif (fsck_refs_fn[i](o, gitdir, refs_check_dir, iter))\n+\t\t\t\tif (fsck_refs_fn[i](ref_store, o, refs_check_dir, iter))\n \t\t\t\t\tret = -1;\n \t\t\t}\n \t\t} else {\n@@ -3507,7 +3505,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \t};\n \n \tif (o->verbose)\n-\t\tfprintf_ln(stderr, \"Checking references consistency\");\n+\t\tfprintf_ln(stderr, _(\"Checking references consistency\"));\n \treturn files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fn);\n \n }\ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nindex 2be28427ab..71a4d1a5ae 100755\n--- a/t/t0602-reffiles-fsck.sh\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -26,7 +26,7 @@ test_expect_success 'ref name should be checked' '\n \tgit tag multi_hierarchy/tag-2 &&\n \n \tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n-\ttest_must_fail git fsck 2>err &&\n+\ttest_must_fail git refs verify 2>err &&\n \tcat >expect <<-EOF &&\n \terror: refs/heads/.branch-1: badRefName: invalid refname format\n \tEOF\n@@ -34,7 +34,7 @@ test_expect_success 'ref name should be checked' '\n \ttest_cmp expect err &&\n \n \tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n-\ttest_must_fail git fsck 2>err &&\n+\ttest_must_fail git refs verify 2>err &&\n \tcat >expect <<-EOF &&\n \terror: refs/heads/@: badRefName: invalid refname format\n \tEOF\n@@ -42,7 +42,7 @@ test_expect_success 'ref name should be checked' '\n \ttest_cmp expect err &&\n \n \tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n-\ttest_must_fail git fsck 2>err &&\n+\ttest_must_fail git refs verify 2>err &&\n \tcat >expect <<-EOF &&\n \terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n \tEOF\n@@ -50,12 +50,12 @@ test_expect_success 'ref name should be checked' '\n \ttest_cmp expect err &&\n \n \tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n-\tgit fsck 2>err &&\n+\tgit refs verify 2>err &&\n \trm $tag_dir_prefix/tag-1.lock &&\n \ttest_must_be_empty err &&\n \n \tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/.lock &&\n-\ttest_must_fail git fsck 2>err &&\n+\ttest_must_fail git refs verify 2>err &&\n \tcat >expect <<-EOF &&\n \terror: refs/tags/.lock: badRefName: invalid refname format\n \tEOF\n@@ -76,18 +76,16 @@ test_expect_success 'ref name check should be adapted into fsck messages' '\n \tgit checkout -b branch-2 &&\n \tgit tag tag-2 &&\n \n-\n \tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n-\tgit -c fsck.badRefName=warn fsck 2>err &&\n+\tgit -c fsck.badRefName=warn refs verify 2>err &&\n \tcat >expect <<-EOF &&\n \twarning: refs/heads/.branch-1: badRefName: invalid refname format\n \tEOF\n \trm $branch_dir_prefix/.branch-1 &&\n \ttest_cmp expect err &&\n \n-\n \tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n-\tgit -c fsck.badRefName=ignore fsck 2>err &&\n+\tgit -c fsck.badRefName=ignore refs verify 2>err &&\n \ttest_must_be_empty err\n '\n \n-- \n2.46.0\n\n"},{"id":"500100","messageId":"ZrEBncAbTZIpwczk@ArchLinux","threadId":"61565","inReplyTo":"ZrEBKjzbyxtMdCCx@ArchLinux","subject":"[GSoC][PATCH v15 1/9] fsck: rename \"skiplist\" to \"skip_oids\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-05T16:45:17Z","receivedAt":"2024-08-05T16:44:50Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The \"skiplist\" field in \"fsck_options\" is related to objects. Because we\nare going to introduce ref consistency check, the \"skiplist\" name is too\ngeneral which will make the caller think \"skiplist\" is related to both\nthe refs and objects.\n\nIt may seem that for both refs and objects, we should provide a general\n\"skiplist\" here. However, the type for \"skiplist\" is `struct oidset`\nwhich is totally unsuitable for refs.\n\nTo avoid above ambiguity, rename \"skiplist\" to \"skip_oids\".\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 4 ++--\n fsck.h | 4 ++--\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex eea7145470..3f32441492 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n \t\tif (!strcmp(buf, \"skiplist\")) {\n \t\t\tif (equal == len)\n \t\t\t\tdie(\"skiplist requires a path\");\n-\t\t\toidset_parse_file(&options->skiplist, buf + equal + 1,\n+\t\t\toidset_parse_file(&options->skip_oids, buf + equal + 1,\n \t\t\t\t\t the_repository->hash_algo);\n \t\t\tbuf += len + 1;\n \t\t\tcontinue;\n@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n static int object_on_skiplist(struct fsck_options *opts,\n \t\t\t const struct object_id *oid)\n {\n-\treturn opts && oid && oidset_contains(&opts->skiplist, oid);\n+\treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n __attribute__((format (printf, 5, 6)))\ndiff --git a/fsck.h b/fsck.h\nindex 6085a384f6..bcfb2e34cd 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -136,7 +136,7 @@ struct fsck_options {\n \tfsck_error error_func;\n \tunsigned strict:1;\n \tenum fsck_msg_type *msg_type;\n-\tstruct oidset skiplist;\n+\tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n \tstruct oidset gitattributes_found;\n@@ -145,7 +145,7 @@ struct fsck_options {\n };\n \n #define FSCK_OPTIONS_DEFAULT { \\\n-\t.skiplist = OIDSET_INIT, \\\n+\t.skip_oids = OIDSET_INIT, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n-- \n2.46.0\n\n"},{"id":"500101","messageId":"ZrEBpkTgbWf0Hrr4@ArchLinux","threadId":"61565","inReplyTo":"ZrEBKjzbyxtMdCCx@ArchLinux","subject":"[GSoC][PATCH v15 2/9] fsck: rename objects-related fsck error functions","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-05T16:45:26Z","receivedAt":"2024-08-05T16:44:59Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The names of objects-related fsck error functions are generic. It's OK\nwhen there is only object database check. However, we are going to\nintroduce refs database check report function. To avoid ambiguity,\nrename object-related fsck error functions to explicitly indicate these\nfunctions are used to report objects-related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 14 +++++++-------\n fsck.c | 17 +++++++++--------\n fsck.h | 26 +++++++++++++-------------\n 3 files changed, 29 insertions(+), 28 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..6d86bbe1e9 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -89,12 +89,12 @@ static int objerror(struct object *obj, const char *err)\n \treturn -1;\n }\n \n-static int fsck_error_func(struct fsck_options *o UNUSED,\n-\t\t\t const struct object_id *oid,\n-\t\t\t enum object_type object_type,\n-\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t const char *message)\n+static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t enum object_type object_type,\n+\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \tswitch (msg_type) {\n \tcase FSCK_WARN:\n@@ -938,7 +938,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tfsck_walk_options.walk = mark_object;\n \tfsck_obj_options.walk = mark_used;\n-\tfsck_obj_options.error_func = fsck_error_func;\n+\tfsck_obj_options.error_func = fsck_objects_error_func;\n \tif (check_strict)\n \t\tfsck_obj_options.strict = 1;\n \ndiff --git a/fsck.c b/fsck.c\nindex 3f32441492..8347842cfb 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1200,7 +1200,7 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n-int fsck_error_function(struct fsck_options *o,\n+int fsck_objects_error_function(struct fsck_options *o,\n \t\t\tconst struct object_id *oid,\n \t\t\tenum object_type object_type UNUSED,\n \t\t\tenum fsck_msg_type msg_type,\n@@ -1303,16 +1303,17 @@ int git_fsck_config(const char *var, const char *value,\n * Custom error callbacks that are used in more than one place.\n */\n \n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message)\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message)\n {\n \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n+\treturn fsck_objects_error_function(o, oid, object_type,\n+\t\t\t\t\t msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex bcfb2e34cd..41ebebbb59 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -120,16 +120,16 @@ typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid, enum object_type object_type,\n-\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n-\t\t\tconst char *message);\n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message);\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n+\t\t\t\tconst char *message);\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message);\n \n struct fsck_options {\n \tfsck_walk_func walk;\n@@ -150,7 +150,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function \\\n+\t.error_func = fsck_objects_error_function \\\n }\n #define FSCK_OPTIONS_STRICT { \\\n \t.strict = 1, \\\n@@ -158,7 +158,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function, \\\n+\t.error_func = fsck_objects_error_function, \\\n }\n #define FSCK_OPTIONS_MISSING_GITMODULES { \\\n \t.strict = 1, \\\n@@ -166,7 +166,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n+\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n \n /* descend in all linked child objects\n-- \n2.46.0\n\n"},{"id":"500102","messageId":"ZrEBsym854FtguPI@ArchLinux","threadId":"61565","inReplyTo":"ZrEBKjzbyxtMdCCx@ArchLinux","subject":"[GSoC][PATCH v15 3/9] fsck: make \"fsck_error\" callback generic","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-05T16:45:39Z","receivedAt":"2024-08-05T16:45:13Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The \"fsck_error\" callback is designed to report the objects-related\nerror messages. It accepts two parameter \"oid\" and \"object_type\" which\nis not generic. In order to provide a unified callback which can report\neither objects or refs, remove the objects-related parameters and add\nthe generic parameter \"void *fsck_report\".\n\nCreate a new \"fsck_object_report\" structure which incorporates the\nremoved parameters \"oid\" and \"object_type\". Then change the\ncorresponding references to adapt to new \"fsck_error\" callback.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 7 +++++--\n builtin/mktag.c | 3 +--\n fsck.c | 26 ++++++++++++++++----------\n fsck.h | 17 ++++++++++++-----\n object-file.c | 9 ++++-----\n 5 files changed, 38 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 6d86bbe1e9..766bbd014d 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -90,12 +90,15 @@ static int objerror(struct object *obj, const char *err)\n }\n \n static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n-\t\t\t\t const struct object_id *oid,\n-\t\t\t\t enum object_type object_type,\n+\t\t\t\t void *fsck_report,\n \t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t\t const char *message)\n {\n+\tstruct fsck_object_report *report = fsck_report;\n+\tconst struct object_id *oid = report->oid;\n+\tenum object_type object_type = report->object_type;\n+\n \tswitch (msg_type) {\n \tcase FSCK_WARN:\n \t\t/* TRANSLATORS: e.g. warning in tree 01bfda: */\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex 4767f1a97e..c6b644219f 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -18,8 +18,7 @@ static int option_strict = 1;\n static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n \n static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n-\t\t\t\t const struct object_id *oid UNUSED,\n-\t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t void *fsck_report UNUSED,\n \t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t\t const char *message)\ndiff --git a/fsck.c b/fsck.c\nindex 8347842cfb..cca6ae144f 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -232,6 +232,10 @@ static int report(struct fsck_options *options,\n \t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n {\n \tva_list ap;\n+\tstruct fsck_object_report report = {\n+\t\t.oid = oid,\n+\t\t.object_type = object_type\n+\t};\n \tstruct strbuf sb = STRBUF_INIT;\n \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n \tint result;\n@@ -252,7 +256,7 @@ static int report(struct fsck_options *options,\n \n \tva_start(ap, fmt);\n \tstrbuf_vaddf(&sb, fmt, ap);\n-\tresult = options->error_func(options, oid, object_type,\n+\tresult = options->error_func(options, &report,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n \tva_end(ap);\n@@ -1201,12 +1205,14 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n }\n \n int fsck_objects_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid,\n-\t\t\tenum object_type object_type UNUSED,\n-\t\t\tenum fsck_msg_type msg_type,\n-\t\t\tenum fsck_msg_id msg_id UNUSED,\n-\t\t\tconst char *message)\n+\t\t\t\tvoid *fsck_report,\n+\t\t\t\tenum fsck_msg_type msg_type,\n+\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n+\t\t\t\tconst char *message)\n {\n+\tstruct fsck_object_report *report = fsck_report;\n+\tconst struct object_id *oid = report->oid;\n+\n \tif (msg_type == FSCK_WARN) {\n \t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n \t\treturn 0;\n@@ -1304,16 +1310,16 @@ int git_fsck_config(const char *var, const char *value,\n */\n \n int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t void *fsck_report,\n \t\t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message)\n {\n \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n-\t\tputs(oid_to_hex(oid));\n+\t\tstruct fsck_object_report *report = fsck_report;\n+\t\tputs(oid_to_hex(report->oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_objects_error_function(o, oid, object_type,\n+\treturn fsck_objects_error_function(o, fsck_report,\n \t\t\t\t\t msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex 41ebebbb59..3b80d02506 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -114,23 +114,30 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n \t\t\t void *data, struct fsck_options *options);\n \n-/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n+/*\n+ * Callback for reporting errors either for objects or refs. The \"fsck_report\"\n+ * is a generic pointer that can be used to pass any information.\n+ */\n typedef int (*fsck_error)(struct fsck_options *o,\n-\t\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t\t void *fsck_report,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n int fsck_objects_error_function(struct fsck_options *o,\n-\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\t\tvoid *fsck_report,\n \t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t\tconst char *message);\n int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t void *fsck_report,\n \t\t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message);\n \n+struct fsck_object_report {\n+\tconst struct object_id *oid;\n+\tenum object_type object_type;\n+};\n+\n struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\ndiff --git a/object-file.c b/object-file.c\nindex 065103be3e..05ac6ebed6 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2470,11 +2470,10 @@ int repo_has_object_file(struct repository *r,\n * give more context.\n */\n static int hash_format_check_report(struct fsck_options *opts UNUSED,\n-\t\t\t\t const struct object_id *oid UNUSED,\n-\t\t\t\t enum object_type object_type UNUSED,\n-\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n-\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t\t const char *message)\n+\t\t\t\t void *fsck_report UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \terror(_(\"object fails fsck: %s\"), message);\n \treturn 1;\n-- \n2.46.0\n\n"},{"id":"500103","messageId":"ZrEBvB9Urj8mG4i-@ArchLinux","threadId":"61565","inReplyTo":"ZrEBKjzbyxtMdCCx@ArchLinux","subject":"[GSoC][PATCH v15 4/9] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-05T16:45:48Z","receivedAt":"2024-08-05T16:45:21Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The static function \"report\" provided by \"fsck.c\" aims at checking error\ntype and calling the callback \"error_func\" to report the message. Both\nrefs and objects need to check the error type of the current fsck\nmessage. In order to extract this common behavior, create a new function\n\"fsck_vreport\". Instead of using \"...\", provide \"va_list\" to allow more\nflexibility.\n\nInstead of changing \"report\" prototype to be algin with the\n\"fsck_vreport\" function, we leave the \"report\" prototype unchanged due\nto the reason that there are nearly 62 references about \"report\"\nfunction. Simply change \"report\" function to use \"fsck_vreport\" to\nreport objects related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 44 ++++++++++++++++++++++++++++++--------------\n 1 file changed, 30 insertions(+), 14 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex cca6ae144f..3614aa56a3 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -226,16 +226,15 @@ static int object_on_skiplist(struct fsck_options *opts,\n \treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n-__attribute__((format (printf, 5, 6)))\n-static int report(struct fsck_options *options,\n-\t\t const struct object_id *oid, enum object_type object_type,\n-\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+/*\n+ * Provide the common functionality for either fscking refs or objects.\n+ * It will get the current msg error type and call the error_func callback\n+ * which is registered in the \"fsck_options\" struct.\n+ */\n+static int fsck_vreport(struct fsck_options *options,\n+\t\t\tvoid *fsck_report,\n+\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n {\n-\tva_list ap;\n-\tstruct fsck_object_report report = {\n-\t\t.oid = oid,\n-\t\t.object_type = object_type\n-\t};\n \tstruct strbuf sb = STRBUF_INIT;\n \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n \tint result;\n@@ -243,9 +242,6 @@ static int report(struct fsck_options *options,\n \tif (msg_type == FSCK_IGNORE)\n \t\treturn 0;\n \n-\tif (object_on_skiplist(options, oid))\n-\t\treturn 0;\n-\n \tif (msg_type == FSCK_FATAL)\n \t\tmsg_type = FSCK_ERROR;\n \telse if (msg_type == FSCK_INFO)\n@@ -254,11 +250,31 @@ static int report(struct fsck_options *options,\n \tprepare_msg_ids();\n \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n \n-\tva_start(ap, fmt);\n \tstrbuf_vaddf(&sb, fmt, ap);\n-\tresult = options->error_func(options, &report,\n+\tresult = options->error_func(options, fsck_report,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n+\n+\treturn result;\n+}\n+\n+__attribute__((format (printf, 5, 6)))\n+static int report(struct fsck_options *options,\n+\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tstruct fsck_object_report report = {\n+\t\t.oid = oid,\n+\t\t.object_type = object_type\n+\t};\n+\tint result;\n+\n+\tif (object_on_skiplist(options, oid))\n+\t\treturn 0;\n+\n+\tva_start(ap, fmt);\n+\tresult = fsck_vreport(options, &report, msg_id, fmt, ap);\n \tva_end(ap);\n \n \treturn result;\n-- \n2.46.0\n\n"},{"id":"500104","messageId":"ZrEBxYagfzAfZKVJ@ArchLinux","threadId":"61565","inReplyTo":"ZrEBKjzbyxtMdCCx@ArchLinux","subject":"[GSoC][PATCH v15 5/9] fsck: add refs report function","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-05T16:45:57Z","receivedAt":"2024-08-05T16:45:30Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new struct \"fsck_ref_report\" to contain the information we\nneed when reporting refs-related messages.\n\nWith the new \"fsck_vreport\" function, add a new function\n\"fsck_report_ref\" to report refs-related fsck error message. Unlike\n\"report\" function uses the exact parameters, we simply pass \"struct\nfsck_ref_report *report\" as the parameter. This is because at current we\ndon't know exactly how many fields we need. By passing this parameter,\nwe don't need to change this function prototype when we want to add more\ninformation into \"fsck_ref_report\".\n\nWe have introduced \"fsck_report_ref\" function to report the error\nmessage for refs. We still need to add the corresponding callback\nfunction. Create refs-specific \"error_func\" callback\n\"fsck_refs_error_function\".\n\nLast, add \"FSCK_REFS_OPTIONS_DEFAULT\" macro to create default options\nwhen checking ref consistency.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 41 +++++++++++++++++++++++++++++++++++++++++\n fsck.h | 25 +++++++++++++++++++++++++\n 2 files changed, 66 insertions(+)\n\ndiff --git a/fsck.c b/fsck.c\nindex 3614aa56a3..38554b626e 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -280,6 +280,19 @@ static int report(struct fsck_options *options,\n \treturn result;\n }\n \n+int fsck_report_ref(struct fsck_options *options,\n+\t\t struct fsck_ref_report *report,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = fsck_vreport(options, report, msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n void fsck_enable_object_names(struct fsck_options *options)\n {\n \tif (!options->object_names)\n@@ -1237,6 +1250,34 @@ int fsck_objects_error_function(struct fsck_options *o,\n \treturn 1;\n }\n \n+int fsck_refs_error_function(struct fsck_options *options UNUSED,\n+\t\t\t void *fsck_report,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tstruct fsck_ref_report *report = fsck_report;\n+\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tstrbuf_addstr(&sb, report->path);\n+\n+\tif (report->oid)\n+\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(report->oid));\n+\telse if (report->referent)\n+\t\tstrbuf_addf(&sb, \" -> (%s)\", report->referent);\n+\n+\tif (msg_type == FSCK_WARN)\n+\t\twarning(\"%s: %s\", sb.buf, message);\n+\telse\n+\t\tret = error(\"%s: %s\", sb.buf, message);\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+\n+}\n+\n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n \t\t struct fsck_options *options, const char *blob_type)\ndiff --git a/fsck.h b/fsck.h\nindex 3b80d02506..2002590f60 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -133,11 +133,23 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message);\n \n+int fsck_refs_error_function(struct fsck_options *options,\n+\t\t\t void *fsck_report,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n+\n struct fsck_object_report {\n \tconst struct object_id *oid;\n \tenum object_type object_type;\n };\n \n+struct fsck_ref_report {\n+\tconst char *path;\n+\tconst struct object_id *oid;\n+\tconst char *referent;\n+};\n+\n struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n@@ -175,6 +187,9 @@ struct fsck_options {\n \t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n \n /* descend in all linked child objects\n * the return value is:\n@@ -216,6 +231,16 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+/*\n+ * Report an error or warning for refs.\n+ */\n+__attribute__((format (printf, 4, 5)))\n+int fsck_report_ref(struct fsck_options *options,\n+\t\t struct fsck_ref_report *report,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\n-- \n2.46.0\n\n"},{"id":"500105","messageId":"ZrEBzq8nkLnnl_vT@ArchLinux","threadId":"61565","inReplyTo":"ZrEBKjzbyxtMdCCx@ArchLinux","subject":"[GSoC][PATCH v15 6/9] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-05T16:46:06Z","receivedAt":"2024-08-05T16:45:39Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The \"struct ref_store\" is the base class which contains the \"be\" pointer\nwhich provides backend-specific functions whose interfaces are defined\nin the \"ref_storage_be\". We could reuse this polymorphism to define only\none interface. For every backend, we need to provide its own function\npointer.\n\nThe interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It's organized as the five parts:\n\n1. The name and the initialization interfaces.\n2. The ref transaction interfaces.\n3. The ref internal interfaces (pack, rename and copy).\n4. The ref filesystem interfaces.\n5. The reflog related interfaces.\n\nTo keep consistent with the git-fsck(1), add a new interface named\n\"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\ngrouped into any above five categories. Explicitly add blank line to\nmake it different from others.\n\nLast, implement placeholder functions for each ref backends.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs.c | 5 +++++\n refs.h | 8 ++++++++\n refs/debug.c | 11 +++++++++++\n refs/files-backend.c | 13 ++++++++++++-\n refs/packed-backend.c | 8 ++++++++\n refs/refs-internal.h | 6 ++++++\n refs/reftable-backend.c | 8 ++++++++\n 7 files changed, 58 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex 915aeb4d1d..6f642dc681 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs, struct fsck_options *o)\n+{\n+\treturn refs->be->fsck(refs, o);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex b3e39bc257..405073621a 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -4,6 +4,7 @@\n #include \"commit.h\"\n #include \"repository.h\"\n \n+struct fsck_options;\n struct object_id;\n struct ref_store;\n struct strbuf;\n@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_options *o);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..45e2e784a0 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tint res = drefs->refs->be->fsck(drefs->refs, o);\n+\ttrace_printf_key(&trace_refs, \"fsck: %d\\n\", res);\n+\treturn res;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex aa52d9be7c..4630eb1f80 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3408,6 +3408,15 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_init,\n@@ -3434,5 +3443,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a0666407cd..5209b0b212 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_init,\n@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex fa975d69aa..a905e187cd 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_options;\n struct ref_transaction;\n \n /*\n@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_options *o);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -677,6 +681,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex fbe74c239d..b5a1a526df 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.46.0\n\n"},{"id":"500106","messageId":"ZrEB16WGGc28dxu2@ArchLinux","threadId":"61565","inReplyTo":"ZrEBKjzbyxtMdCCx@ArchLinux","subject":"[GSoC][PATCH v15 7/9] builtin/refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-05T16:46:15Z","receivedAt":"2024-08-05T16:45:48Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\ncheck the reference database consistency and also this subcommand will\nbe used as the entry point of checking refs for \"git-fsck(1)\".\n\nAdd \"verbose\" field into \"fsck_options\" to indicate whether we should\nprint verbose messages when checking refs and objects consistency.\n\nRemove bit-field for \"strict\" field, this is because we cannot take\naddress of a bit-field which makes it unhandy to set member variables\nwhen parsing the command line options.\n\nThe \"git-fsck(1)\" declares \"fsck_options\" variable with \"static\"\nidentifier which avoids complaint by the leak-checker. However, in\n\"git-refs verify\", we need to do memory clean manually. Thus add\n\"fsck_options_clear\" function in \"fsck.c\" to provide memory clean\noperation.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/git-refs.txt | 13 +++++++++++++\n builtin/refs.c | 34 ++++++++++++++++++++++++++++++++++\n fsck.c | 11 +++++++++++\n fsck.h | 8 +++++++-\n 4 files changed, 65 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..ce31f93061 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+The following options are specific to 'git refs verify':\n+\n+--strict::\n+\tEnable stricter error checking. This will cause warnings to be\n+\treported as errors. See linkgit:git-fsck[1].\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..131f98be98 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,44 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"verbose\", &fsck_refs_options.verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &fsck_refs_options.strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\tint ret;\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"'git refs verify' takes no arguments\"));\n+\n+\tgit_config(git_fsck_config, &fsck_refs_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n+\n+\tfsck_options_clear(&fsck_refs_options);\n+\treturn ret;\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/fsck.c b/fsck.c\nindex 38554b626e..7eb5cdefdd 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1333,6 +1333,17 @@ int fsck_finish(struct fsck_options *options)\n \treturn ret;\n }\n \n+void fsck_options_clear(struct fsck_options *options)\n+{\n+\tfree(options->msg_type);\n+\toidset_clear(&options->skip_oids);\n+\toidset_clear(&options->gitmodules_found);\n+\toidset_clear(&options->gitmodules_done);\n+\toidset_clear(&options->gitattributes_found);\n+\toidset_clear(&options->gitattributes_done);\n+\tkh_clear_oid_map(options->object_names);\n+}\n+\n int git_fsck_config(const char *var, const char *value,\n \t\t const struct config_context *ctx, void *cb)\n {\ndiff --git a/fsck.h b/fsck.h\nindex 2002590f60..d551a9fe86 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -153,7 +153,8 @@ struct fsck_ref_report {\n struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n-\tunsigned strict:1;\n+\tunsigned strict;\n+\tunsigned verbose;\n \tenum fsck_msg_type *msg_type;\n \tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n@@ -231,6 +232,11 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+/*\n+ * Clear the fsck_options struct, freeing any allocated memory.\n+ */\n+void fsck_options_clear(struct fsck_options *options);\n+\n /*\n * Report an error or warning for refs.\n */\n-- \n2.46.0\n\n"},{"id":"500107","messageId":"ZrEB4PjNPM6SZspS@ArchLinux","threadId":"61565","inReplyTo":"ZrEBKjzbyxtMdCCx@ArchLinux","subject":"[GSoC][PATCH v15 8/9] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-05T16:46:24Z","receivedAt":"2024-08-05T16:45:57Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"For refs and reflogs, we need to scan its corresponding directories to\ncheck every regular file or symbolic link which shares the same pattern.\nIntroduce a unified interface for scanning directories for\nfiles-backend.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 ++\n fsck.h | 1 +\n refs/files-backend.c | 74 ++++++++++++++++++++++++++++++++++-\n 3 files changed, 77 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex f643585a34..7c809fddf1 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,9 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefFiletype`::\n+\t(ERROR) A ref has a bad file type.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex d551a9fe86..af02174973 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_FILETYPE, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 4630eb1f80..f337356860 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -6,6 +6,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -3408,13 +3409,84 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct ref_store *ref_store,\n+\t\t\t\t struct fsck_options *o,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fn)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", ref_store->gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\tif (!iter) {\n+\t\tret = error_errno(_(\"cannot open directory %s\"), sb.buf);\n+\t\tgoto out;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->verbose)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fn[i]; i++) {\n+\t\t\t\tif (fsck_refs_fn[i](ref_store, o, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tstruct fsck_ref_report report = { .path = iter->basename };\n+\t\t\tif (fsck_report_ref(o, &report,\n+\t\t\t\t\t FSCK_MSG_BAD_REF_FILETYPE,\n+\t\t\t\t\t \"unexpected file type\"))\n+\t\t\t\tret = -1;\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n+\n+out:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\tfiles_fsck_refs_fn fsck_refs_fn[]= {\n+\t\tNULL,\n+\t};\n+\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, _(\"Checking references consistency\"));\n+\treturn files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fn);\n+\n+}\n+\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_options *o)\n {\n \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n-\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\treturn files_fsck_refs(ref_store, o) |\n+\t refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n }\n \n struct ref_storage_be refs_be_files = {\n-- \n2.46.0\n\n"},{"id":"500108","messageId":"ZrEB6YWB82CyH0DI@ArchLinux","threadId":"61565","inReplyTo":"ZrEBKjzbyxtMdCCx@ArchLinux","subject":"[GSoC][PATCH v15 9/9] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-05T16:46:33Z","receivedAt":"2024-08-05T16:46:06Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) only implicitly checks the reference, it does not fully\ncheck refs with bad format name such as standalone \"@\".\n\nHowever, a file ending with \".lock\" should not be marked as having a bad\nref name. It is expected that concurrent writers may have such lock files.\nWe currently ignore this situation. But for bare \".lock\" file, we will\nreport it as error.\n\nIn order to provide such checks, add a new fsck message id \"badRefName\"\nwith default ERROR type. Use existing \"check_refname_format\" to explicit\ncheck the ref name. And add a new unit test to verify the functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 ++\n fsck.h | 1 +\n refs/files-backend.c | 31 ++++++++++++\n t/t0602-reffiles-fsck.sh | 92 +++++++++++++++++++++++++++++++++++\n 4 files changed, 127 insertions(+)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex 7c809fddf1..68a2801f15 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -22,6 +22,9 @@\n `badRefFiletype`::\n \t(ERROR) A ref has a bad file type.\n \n+`badRefName`::\n+\t(ERROR) A ref has an invalid format.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex af02174973..500b4c04d2 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -32,6 +32,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_FILETYPE, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex f337356860..6e6b47251d 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3419,6 +3419,36 @@ typedef int (*files_fsck_refs_fn)(struct ref_store *ref_store,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n+static int files_fsck_refs_name(struct ref_store *ref_store UNUSED,\n+\t\t\t\tstruct fsck_options *o,\n+\t\t\t\tconst char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\t/*\n+\t * Ignore the files ending with \".lock\" as they may be lock files\n+\t * However, do not allow bare \".lock\" files.\n+\t */\n+\tif (iter->basename[0] != '.' && ends_with(iter->basename, \".lock\"))\n+\t\tgoto cleanup;\n+\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstruct fsck_ref_report report = { .path = NULL };\n+\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\treport.path = sb.buf;\n+\t\tret = fsck_report_ref(o, &report,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n+\t}\n+\n+cleanup:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3470,6 +3500,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o)\n {\n \tfiles_fsck_refs_fn fsck_refs_fn[]= {\n+\t\tfiles_fsck_refs_name,\n \t\tNULL,\n \t};\n \ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..71a4d1a5ae\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,92 @@\n+#!/bin/sh\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+TEST_PASSES_SANITIZE_LEAK=true\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\tcd repo &&\n+\n+\tgit commit --allow-empty -m initial &&\n+\tgit checkout -b branch-1 &&\n+\tgit tag tag-1 &&\n+\tgit commit --allow-empty -m second &&\n+\tgit checkout -b branch-2 &&\n+\tgit tag tag-2 &&\n+\tgit tag multi_hierarchy/tag-2 &&\n+\n+\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\ttest_must_fail git refs verify 2>err &&\n+\tcat >expect <<-EOF &&\n+\terror: refs/heads/.branch-1: badRefName: invalid refname format\n+\tEOF\n+\trm $branch_dir_prefix/.branch-1 &&\n+\ttest_cmp expect err &&\n+\n+\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\ttest_must_fail git refs verify 2>err &&\n+\tcat >expect <<-EOF &&\n+\terror: refs/heads/@: badRefName: invalid refname format\n+\tEOF\n+\trm $branch_dir_prefix/@ &&\n+\ttest_cmp expect err &&\n+\n+\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\ttest_must_fail git refs verify 2>err &&\n+\tcat >expect <<-EOF &&\n+\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\tEOF\n+\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\ttest_cmp expect err &&\n+\n+\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\tgit refs verify 2>err &&\n+\trm $tag_dir_prefix/tag-1.lock &&\n+\ttest_must_be_empty err &&\n+\n+\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/.lock &&\n+\ttest_must_fail git refs verify 2>err &&\n+\tcat >expect <<-EOF &&\n+\terror: refs/tags/.lock: badRefName: invalid refname format\n+\tEOF\n+\trm $tag_dir_prefix/.lock &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\tcd repo &&\n+\tgit commit --allow-empty -m initial &&\n+\tgit checkout -b branch-1 &&\n+\tgit tag tag-1 &&\n+\tgit commit --allow-empty -m second &&\n+\tgit checkout -b branch-2 &&\n+\tgit tag tag-2 &&\n+\n+\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\tgit -c fsck.badRefName=warn refs verify 2>err &&\n+\tcat >expect <<-EOF &&\n+\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\tEOF\n+\trm $branch_dir_prefix/.branch-1 &&\n+\ttest_cmp expect err &&\n+\n+\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\tgit -c fsck.badRefName=ignore refs verify 2>err &&\n+\ttest_must_be_empty err\n+'\n+\n+test_done\n-- \n2.46.0\n\n"},{"id":"500116","messageId":"xmqq4j7y3kmt.fsf@gitster.g","threadId":"61565","inReplyTo":"ZrDtVJYoJJZDesB4@ArchLinux","subject":"Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-05T18:11:22Z","receivedAt":"2024-08-05T18:11:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"shejialuo writes:\n\n>> > +static void fsck_refs(void)\n>> > +{\n>> > +\tstruct child_process refs_verify = CHILD_PROCESS_INIT;\n>> > +\tchild_process_init(&refs_verify);\n>> > +\trefs_verify.git_cmd = 1;\n>> > +\tstrvec_pushl(&refs_verify.args, \"refs\", \"verify\", NULL);\n>> > +\tif (verbose)\n>> > +\t\tstrvec_push(&refs_verify.args, \"--verbose\");\n>> > +\tif (check_strict)\n>> > +\t\tstrvec_push(&refs_verify.args, \"--strict\");\n>> > +\n>> > +\tif (run_command(&refs_verify))\n>> > +\t\terrors_found |= ERROR_REFS;\n>> > +}\n>> \n>> Okay. I think that it's sensible to execute this as part of git-fsck(1).\n>> But do we want to provide an option to disable this new check, as well?\n>> \n>> It does feel a bit like opening a can of worms, though. None of the\n>> other checks have trivial ways to disable them, and git-fsck(1) is\n>> gaining more and more checks. So if we can disable ref checks, we may\n>> also want to have options to disable checks for objects, connectivity,\n>> reverse indices, indices, commit graphs and whatnot. In other words, in\n>> my opinion we need to think a bit bigger and design a proper UI around\n>> this.\n>> \n>> But I don't think that should happen as part of this commit series, as\n>> it is already big enough. So either we just accept this patch as-is. Or\n>> we evict it from this series and handle it in the future together with\n>> all the other taks that one may potentially want to disable.\n>> \n>> I'd rather pick option two.\n>> \n>\n> After talking with Patrick offline, we decide to drop this patch. At\n> current, we should put this change slowly for the user. Because many\n> people use \"git-fsck(1)\", currently we don't have a way to disable ref\n> checks by default. It's a little beyond this series.\n>\n> We may consider later.\n\nHmph, I am fine with the approach to take it slower.\n\nBUT.\n\nHere is what the diffstat for the whole thing in the updated round\nv15 looked like. Do most of these changes outside refs/ still\nneeded if we do not give any way to even optionally enable it via\n\"fsck\" for those who feel adventurous? Should we still be touching\nfsck.[ch] and other fsck related infrastructure in the series?\n\n Documentation/fsck-msgids.txt | 6 ++\n Documentation/git-refs.txt | 13 +++++\n builtin/fsck.c | 17 +++---\n builtin/mktag.c | 3 +-\n builtin/refs.c | 34 +++++++++++\n fsck.c | 127 +++++++++++++++++++++++++++++++++---------\n fsck.h | 76 +++++++++++++++++++------\n object-file.c | 9 ++-\n refs.c | 5 ++\n refs.h | 8 +++\n refs/debug.c | 11 ++++\n refs/files-backend.c | 116 +++++++++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 +++\n refs/refs-internal.h | 6 ++\n refs/reftable-backend.c | 8 +++\n t/t0602-reffiles-fsck.sh | 92 ++++++++++++++++++++++++++++++\n 16 files changed, 480 insertions(+), 59 deletions(-)\n"},{"id":"500117","messageId":"ZrEbllB6WjLfWqNZ@ArchLinux","threadId":"61565","inReplyTo":"xmqq4j7y3kmt.fsf@gitster.g","subject":"Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-05T18:36:06Z","receivedAt":"2024-08-05T18:35:39Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"> \n> Hmph, I am fine with the approach to take it slower.\n> \n> BUT.\n> \n> Here is what the diffstat for the whole thing in the updated round\n> v15 looked like. Do most of these changes outside refs/ still\n> needed if we do not give any way to even optionally enable it via\n> \"fsck\" for those who feel adventurous? Should we still be touching\n> fsck.[ch] and other fsck related infrastructure in the series?\n> \n\nFrom my current standing, regardless of whether we run the \"git refs\nverify\" subprocess inside \"git-fsck(1)\", we must change the fsck\ninfrastructure illustrated by the following:\n\nFrom the development of this series, we can know the main problem is\nthat fsck error message is highly coupled with the object checks. Even\nif we don't perform \"git refs verify\" in \"git-fsck(1)\", we still need to\nfollow the fsck msg framework when executing \"git refs verify\". We\ncannot avoid this.\n\nThe content we change for \"fsck.[ch]\" is mainly the fsck msg part. We do\nnot change anything about the objects.\n\nI agree with you that it would be strange if we do not expose any\ninterfaces for user who are adventurous. Actually we may simply add an\noption \"--refs-experimental\" or simply \"--refs\" to allow the users check\nref consistency by using \"git-fsck(1)\".\n\nI guess the concern that Patrick cares about is that we ONLY make refs\noptional here, but do not provide options for other checks. It will be\nstrange from this perspective.\n\n> Documentation/fsck-msgids.txt | 6 ++\n> Documentation/git-refs.txt | 13 +++++\n> builtin/fsck.c | 17 +++---\n> builtin/mktag.c | 3 +-\n> builtin/refs.c | 34 +++++++++++\n> fsck.c | 127 +++++++++++++++++++++++++++++++++---------\n> fsck.h | 76 +++++++++++++++++++------\n> object-file.c | 9 ++-\n> refs.c | 5 ++\n> refs.h | 8 +++\n> refs/debug.c | 11 ++++\n> refs/files-backend.c | 116 +++++++++++++++++++++++++++++++++++++-\n> refs/packed-backend.c | 8 +++\n> refs/refs-internal.h | 6 ++\n> refs/reftable-backend.c | 8 +++\n> t/t0602-reffiles-fsck.sh | 92 ++++++++++++++++++++++++++++++\n> 16 files changed, 480 insertions(+), 59 deletions(-)\n"},{"id":"500121","messageId":"xmqqsevi220s.fsf@gitster.g","threadId":"61565","inReplyTo":"ZrEbllB6WjLfWqNZ@ArchLinux","subject":"Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-05T19:38:43Z","receivedAt":"2024-08-05T19:38:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"shejialuo writes:\n\n> I agree with you that it would be strange if we do not expose any\n> interfaces for user who are adventurous. Actually we may simply add an\n> option \"--refs-experimental\" or simply \"--refs\" to allow the users check\n> ref consistency by using \"git-fsck(1)\".\n>\n> I guess the concern that Patrick cares about is that we ONLY make refs\n> optional here, but do not provide options for other checks. It will be\n> strange from this perspective.\n\nI do not care about strange all that much. I however care about new\ncomplexity in the code, complexity that is not taken advantage of\nand is not exercised. \n\nYou said\n\n> From the development of this series, we can know the main problem is\n> that fsck error message is highly coupled with the object checks.\n\nand even if it is true and we have problem in fsck code paths, we\ncannot see if _your_ solution to that problem is a good one without\nhaving the code that exercises your additional code.\n\nBut if \"git refs verify\" does exercise all the new code paths (and\nthe refactored code that existed before this series, sitting now in\ndifferent places), then I do not have to worry about it. My question\nwas primarily to extract \"even though we do not wire this up to fsck,\nwe already have another code paths that uses all these changes\" out\nof you.\n\nThanks.\n"},{"id":"500142","messageId":"ZrFxb1FNRCzu-NuW@ArchLinux","threadId":"61565","inReplyTo":"xmqqsevi220s.fsf@gitster.g","subject":"Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-06T00:42:23Z","receivedAt":"2024-08-06T00:41:56Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Mon, Aug 05, 2024 at 12:38:43PM -0700, Junio C Hamano wrote:\n> shejialuo writes:\n> \n> > I agree with you that it would be strange if we do not expose any\n> > interfaces for user who are adventurous. Actually we may simply add an\n> > option \"--refs-experimental\" or simply \"--refs\" to allow the users check\n> > ref consistency by using \"git-fsck(1)\".\n> >\n> > I guess the concern that Patrick cares about is that we ONLY make refs\n> > optional here, but do not provide options for other checks. It will be\n> > strange from this perspective.\n> \n> I do not care about strange all that much. I however care about new\n> complexity in the code, complexity that is not taken advantage of\n> and is not exercised. \n> \n> You said\n> \n> > From the development of this series, we can know the main problem is\n> > that fsck error message is highly coupled with the object checks.\n> \n> and even if it is true and we have problem in fsck code paths, we\n> cannot see if _your_ solution to that problem is a good one without\n> having the code that exercises your additional code.\n> \n> But if \"git refs verify\" does exercise all the new code paths (and\n> the refactored code that existed before this series, sitting now in\n> different places), then I do not have to worry about it. My question\n> was primarily to extract \"even though we do not wire this up to fsck,\n> we already have another code paths that uses all these changes\" out\n> of you.\n> \n\nI understand what you mean here. I can say that \"git refs verify\" only\nexercises a part of the new code paths. The main reason why this series\nchanges a lot of \"fsck.[ch]\" is that I want to expose the\n\"fsck_report_ref\" interface to report refs-related errors. So I guess\nthis part should be covered.\n\nAt the current implementation, we change the \"fsck.[ch]\" for the\nfollowing three things:\n\n1. Refactor \"report\" to use \"fsck_vreport\"\n2. Create \"fsck_report_ref\" for refs check.\n3. Do some simple renames to distinguish between refs and objects.\n\nWe do cover the second case in \"git refs verify\". But sadly, the first\ncase and third case are covered in \"git-fsck(1)\". So, \"git refs verify\"\ndoes not exercise the refactored code.\n\nHowever, I am a little confused. Actually, in the implementation, refs\ncheck and objects check are independent.\n\nI think we should wire up \"git refs verify\" to \"git-fsck(1)\". Because we\nhave no way to exercise the above case 1 and 3. If we do not, we will\nbring a lot of complexity here.\n\n> Thanks.\n"},{"id":"500150","messageId":"ZrG0zr3Wh44a8GvF@tanuki","threadId":"61565","inReplyTo":"xmqqsevi220s.fsf@gitster.g","subject":"Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-06T05:29:50Z","receivedAt":"2024-08-06T05:29:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 05, 2024 at 12:38:43PM -0700, Junio C Hamano wrote:\n> shejialuo writes:\n> \n> > I agree with you that it would be strange if we do not expose any\n> > interfaces for user who are adventurous. Actually we may simply add an\n> > option \"--refs-experimental\" or simply \"--refs\" to allow the users check\n> > ref consistency by using \"git-fsck(1)\".\n> >\n> > I guess the concern that Patrick cares about is that we ONLY make refs\n> > optional here, but do not provide options for other checks. It will be\n> > strange from this perspective.\n> \n> I do not care about strange all that much. I however care about new\n> complexity in the code, complexity that is not taken advantage of\n> and is not exercised. \n> \n> You said\n> \n> > From the development of this series, we can know the main problem is\n> > that fsck error message is highly coupled with the object checks.\n> \n> and even if it is true and we have problem in fsck code paths, we\n> cannot see if _your_ solution to that problem is a good one without\n> having the code that exercises your additional code.\n> \n> But if \"git refs verify\" does exercise all the new code paths (and\n> the refactored code that existed before this series, sitting now in\n> different places), then I do not have to worry about it. My question\n> was primarily to extract \"even though we do not wire this up to fsck,\n> we already have another code paths that uses all these changes\" out\n> of you.\n\nYeah, there is quite some complexity introduced in fsck code, where most\nof the complexity comes from making the reportin functions more generic.\nAnd we do end up using that in this series even if we don't integrate\n`git refs verify` with `git fsck`. It brings us the ability to configure\nthe checks performed by `git refs verify` in the same way like we can\nconfigure all the other checks in `git fsck`, such that we can enable,\ndisable or change severity levels for each of the messages reported by\nit.\n\nI think that this is a sensible way to go about it, and it leaves us\nwith a more flexible and consistent error reporting infrastructure that\nwe can eventually also use for other commands that git-fsck(1) shells\nout to.\n\nPatrick\n"},{"id":"500152","messageId":"ZrG89e6IVMEuq0IK@tanuki","threadId":"61565","inReplyTo":"ZrFxb1FNRCzu-NuW@ArchLinux","subject":"Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-06T06:04:37Z","receivedAt":"2024-08-06T06:04:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 06, 2024 at 08:42:23AM +0800, shejialuo wrote:\n> On Mon, Aug 05, 2024 at 12:38:43PM -0700, Junio C Hamano wrote:\n> > shejialuo writes:\n> > \n> > > I agree with you that it would be strange if we do not expose any\n> > > interfaces for user who are adventurous. Actually we may simply add an\n> > > option \"--refs-experimental\" or simply \"--refs\" to allow the users check\n> > > ref consistency by using \"git-fsck(1)\".\n> > >\n> > > I guess the concern that Patrick cares about is that we ONLY make refs\n> > > optional here, but do not provide options for other checks. It will be\n> > > strange from this perspective.\n> > \n> > I do not care about strange all that much. I however care about new\n> > complexity in the code, complexity that is not taken advantage of\n> > and is not exercised. \n> > \n> > You said\n> > \n> > > From the development of this series, we can know the main problem is\n> > > that fsck error message is highly coupled with the object checks.\n> > \n> > and even if it is true and we have problem in fsck code paths, we\n> > cannot see if _your_ solution to that problem is a good one without\n> > having the code that exercises your additional code.\n> > \n> > But if \"git refs verify\" does exercise all the new code paths (and\n> > the refactored code that existed before this series, sitting now in\n> > different places), then I do not have to worry about it. My question\n> > was primarily to extract \"even though we do not wire this up to fsck,\n> > we already have another code paths that uses all these changes\" out\n> > of you.\n> > \n> \n> I understand what you mean here. I can say that \"git refs verify\" only\n> exercises a part of the new code paths. The main reason why this series\n> changes a lot of \"fsck.[ch]\" is that I want to expose the\n> \"fsck_report_ref\" interface to report refs-related errors. So I guess\n> this part should be covered.\n> \n> At the current implementation, we change the \"fsck.[ch]\" for the\n> following three things:\n> \n> 1. Refactor \"report\" to use \"fsck_vreport\"\n> 2. Create \"fsck_report_ref\" for refs check.\n> 3. Do some simple renames to distinguish between refs and objects.\n> \n> We do cover the second case in \"git refs verify\". But sadly, the first\n> case and third case are covered in \"git-fsck(1)\". So, \"git refs verify\"\n> does not exercise the refactored code.\n> \n> However, I am a little confused. Actually, in the implementation, refs\n> check and objects check are independent.\n> \n> I think we should wire up \"git refs verify\" to \"git-fsck(1)\". Because we\n> have no way to exercise the above case 1 and 3. If we do not, we will\n> bring a lot of complexity here.\n\nI don't think that is necessary. Basically, what you are saying is that\nwe seem to be testing only the new refs-related reporting that you have\nintroduced via your refactorings, but not the preexisting objects\nrelated functionality.\n\nThat isn't true though. The object-specific reporting functions that you\nhave refactored already had callers before, and it still has callers now\nbecause you adapted those accordingly. You can assume that those callers\nalready had tests before your refactorings, and as no tests broken you\ncan be reasonably sure that your refactorings are sound for the object\nrelated code.\n\nFurthermore, you do exercise the new ref-related parts via a couple of\ntests that exercise `git refs verify`. Consequently, all parts of the\nrefactoring are covered by either old or new tests, and we should be\ngood here.\n\nSo even though we do not exercise the ref-related parts via git-fsck(1),\nit is still subjected to tests. Eventually, when we start calling `git\nrefs verify` in git-fsck(1), we'd introduce more tests that verify that\nthe integration of those two commands works as expected, as well.\n\nSo, to summarize: the refactored functionality is both used and tested\nand I think it's sensible to defer the integration of git-fsck(1) and\ngit-refs(1).\n\nThanks!\n\nPatrick\n"},{"id":"500161","messageId":"ZrHRm-qaZu6BKPoZ@tanuki","threadId":"61565","inReplyTo":"ZrEBKjzbyxtMdCCx@ArchLinux","subject":"Re: [GSoC][PATCH v15 0/9] ref consistency check infra setup","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-06T07:32:43Z","receivedAt":"2024-08-06T07:32:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 06, 2024 at 12:43:22AM +0800, shejialuo wrote:\n> Hi All:\n> \n> This version handles the following problems:\n> \n> 1. Patrick advices that I should not use `va_copy` in the changed\n> `report` function. Actually this is a mistake, this version avoids\n> redundant `ap` copy.\n> 2. Patrick advices I should rebase [v14 05/11] into [v14 04/11]. I\n> follow this advice in this version.\n> 3. Patrick advices that we should put [v14 06/11] before we introduce\n> ref-related operations. This version reorders the commit sequence. It's\n> a minor change.\n> 4. Patrick suggests at current we should not add `git refs verify`\n> command into \"git-fsck(1)\". This is because we should disable this new\n> check by default for the users. Many users use \"git-fsck(1)\" in their daily\n> workflow. We should not be aggressive. However, if we provide this\n> mechanism in this series, we will again make more complexity. So this\n> version drop patch [v14 09/11]. Also because of dropping, change the\n> test file to use \"git refs verify\" command instead of \"git fsck\"\n> command.\n> 5. Patrick suggests that we should use `ends_with` instead of\n> `strip_suffix`, fix.\n> \n> There is another important problem this patch solves:\n> \n> At v13, Junio has suggested that the `files_fsck_refs_fn` should be\n> adapted to Patrick's change. Actually, I made a bad design before. I\n> should always pass the `ref_store` structure. So I change it to\n> \n> -typedef int (*files_fsck_refs_fn)(struct fsck_options *o,\n> -\t\t\t\t const char *gitdir,\n> +typedef int (*files_fsck_refs_fn)(struct ref_store *ref_store,\n> +\t\t\t\t struct fsck_options *o,\n> const char *refs_check_dir,\n> struct dir_iterator *iter);\n> \n> `gitdir` could be got by using `ref_store` parameter. By using\n> `ref_store` parameter, we provide extensibility here. If something else\n> change, we merely need to change \"files_fsck_refs_fn\" prototype.\n> \n> Because I drop one patch and rebase one patch. I provide the `interdiff`\n> for reviewers to make the life easier.\n> \n> Due to the deadline of the GSoC, I will speed up the review feedback\n> process.\n\nI've got another small set of nits, almost not worth addressing. I was a\nbit torn whether to send them or not as the series is in a good shape\nalready, in my opinion. But let's maybe wait one or two more days for\nadditional feedback, and then (hopefully) reroll this a final time.\n\nThanks for all your work!\n\nPatrick\n"},{"id":"500162","messageId":"ZrHRoCka8yCAwxzz@tanuki","threadId":"61565","inReplyTo":"ZrEBxYagfzAfZKVJ@ArchLinux","subject":"Re: [GSoC][PATCH v15 5/9] fsck: add refs report function","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-06T07:32:48Z","receivedAt":"2024-08-06T07:32:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 06, 2024 at 12:45:57AM +0800, shejialuo wrote:\n> @@ -1237,6 +1250,34 @@ int fsck_objects_error_function(struct fsck_options *o,\n> \treturn 1;\n> }\n> \n> +int fsck_refs_error_function(struct fsck_options *options UNUSED,\n> +\t\t\t void *fsck_report,\n> +\t\t\t enum fsck_msg_type msg_type,\n> +\t\t\t enum fsck_msg_id msg_id UNUSED,\n> +\t\t\t const char *message)\n> +{\n> +\tstruct fsck_ref_report *report = fsck_report;\n> +\n\nNit: there's an unneeded empty newline here.\n\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\tint ret = 0;\n> +\n> +\tstrbuf_addstr(&sb, report->path);\n> +\n> +\tif (report->oid)\n> +\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(report->oid));\n> +\telse if (report->referent)\n> +\t\tstrbuf_addf(&sb, \" -> (%s)\", report->referent);\n> +\n> +\tif (msg_type == FSCK_WARN)\n> +\t\twarning(\"%s: %s\", sb.buf, message);\n> +\telse\n> +\t\tret = error(\"%s: %s\", sb.buf, message);\n> +\n> +\tstrbuf_release(&sb);\n> +\treturn ret;\n> +\n\nHere, as well.\n\nPatrick\n"},{"id":"500163","messageId":"ZrHRpnpGkH19godh@tanuki","threadId":"61565","inReplyTo":"ZrEB16WGGc28dxu2@ArchLinux","subject":"Re: [GSoC][PATCH v15 7/9] builtin/refs: add verify subcommand","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-06T07:32:54Z","receivedAt":"2024-08-06T07:32:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 06, 2024 at 12:46:15AM +0800, shejialuo wrote:\n> @@ -58,15 +63,44 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n> \treturn err;\n> }\n> \n> +static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n> +{\n> +\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n> +\tconst char * const verify_usage[] = {\n> +\t\tREFS_VERIFY_USAGE,\n> +\t\tNULL,\n> +\t};\n> +\tstruct option options[] = {\n> +\t\tOPT_BOOL(0, \"verbose\", &fsck_refs_options.verbose, N_(\"be verbose\")),\n> +\t\tOPT_BOOL(0, \"strict\", &fsck_refs_options.strict, N_(\"enable strict checking\")),\n> +\t\tOPT_END(),\n> +\t};\n> +\tint ret;\n> +\n> +\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n> +\tif (argc)\n> +\t\tusage(_(\"'git refs verify' takes no arguments\"));\n\nJunio has posted a patch series [1] where he wants to get rid of\nmessages that simply say \"no arguments\" or \"too many arguments\". I guess\nwe can play nice and also move into the same direction here, where we\ninstead tell the user which argument we didn't expect.\n\nSo I'd propose to make this:\n\n argc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n if (argc)\n usage(_(\"unknown argument: '%s'\", argv[0]));\n\n[1]: <20240806003539.3292562-1-gitster@pobox.com>\n\nPatrick\n"},{"id":"500164","messageId":"ZrHRrEJmCdyL8l1i@tanuki","threadId":"61565","inReplyTo":"ZrEB4PjNPM6SZspS@ArchLinux","subject":"Re: [GSoC][PATCH v15 8/9] files-backend: add unified interface for refs scanning","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-06T07:33:00Z","receivedAt":"2024-08-06T07:33:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 06, 2024 at 12:46:24AM +0800, shejialuo wrote:\n[snip]\n> +static int files_fsck_refs(struct ref_store *ref_store,\n> +\t\t\t struct fsck_options *o)\n> +{\n> +\tfiles_fsck_refs_fn fsck_refs_fn[]= {\n> +\t\tNULL,\n> +\t};\n> +\n> +\tif (o->verbose)\n> +\t\tfprintf_ln(stderr, _(\"Checking references consistency\"));\n> +\treturn files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fn);\n> +\n\nNit: another empty newline.\n> +}\n> +\n\nPatrick\n"},{"id":"500231","messageId":"xmqq7cctzlxh.fsf@gitster.g","threadId":"61565","inReplyTo":"ZrG89e6IVMEuq0IK@tanuki","subject":"Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-06T15:54:34Z","receivedAt":"2024-08-06T15:54:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> > But if \"git refs verify\" does exercise all the new code paths (and\n>> > the refactored code that existed before this series, sitting now in\n>> > different places), then I do not have to worry about it. My question\n>> > was primarily to extract \"even though we do not wire this up to fsck,\n>> > we already have another code paths that uses all these changes\" out\n>> > of you.\n>> ...\n> So, to summarize: the refactored functionality is both used and tested\n> and I think it's sensible to defer the integration of git-fsck(1) and\n> git-refs(1).\n\nAfter refactoring, existing functionality about objects are used, of\ncourse, (there is no other code that does so), the refactoring lets\nthe code to learn to perform checks on references, and these new\nchecks are exercised by \"git refs verify\".\n\nI took what shejialuo said that way, and that is fine by me when I\nsaid the above ;-). So I think we all are on the same page?\n\nThanks.\n\n\n\n\n"},{"id":"500232","messageId":"xmqqsevhy6e7.fsf@gitster.g","threadId":"61565","inReplyTo":"ZrHRpnpGkH19godh@tanuki","subject":"Re: [GSoC][PATCH v15 7/9] builtin/refs: add verify subcommand","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-06T16:15:28Z","receivedAt":"2024-08-06T16:15:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> +\tif (argc)\n>> +\t\tusage(_(\"'git refs verify' takes no arguments\"));\n>\n> Junio has posted a patch series [1] where he wants to get rid of\n> messages that simply say \"no arguments\" or \"too many arguments\".\n> ...\n> So I'd propose to make this:\n>\n> argc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n> if (argc)\n> usage(_(\"unknown argument: '%s'\", argv[0]));\n\nI probably should have said that I am fully behind the intent\nagainst \"too many arguments\", but I am not 100% behind the\nparticular messaging used in the patch series I sent out.\n\nOne potential complaint I expected to hear, for example, was that \"a\nis unknown\" given when you said \"git cmd a a a a a\" is not all that\nclear ;-). To alleviate, you would have to say \"git cmd takes only\n2 arguments\" if 'a' you are complaining about is the third one.\n\nAlso, many people would consider that \"unexpected argument\" is\nbetter than \"unknown argument\".\n\nI personally think the message above is absolutely clear and good.\n\nYou say that 'git refs verify' takes no arguments, and for somebody\nwho said \"git refs verify a b c d e\", there is no doubt that all of\nthese a b c d e are unwanted. And there is no room to misinterpret\nthe message as \"'git refs' is ok but 'git refs verify' is already\nunwelcome with extra argument\", either [*].\n\nIn short, I think the message in the patch here is good, and it is\nthe other \"war on 'too many arguments'\" series whose messages need\nto be thought further.\n\n\n[Foornote]\n\n * ... which was the problem I was trying to address in the current\n message \"too many arguments\" that does not even say which early\n part of the command line we consider is \"command\" that was given\n \"arguments\"---to uninitiated who said \"git refs verify foo\", it\n is unclera if that's \"git refs\" command whose first argument is\n \"verify\", \"git\" command whose first two arguments are \"refs\n verify\", etc.\n\n"},{"id":"500259","messageId":"ZrL81Q6hp8kDjWVL@tanuki","threadId":"61565","inReplyTo":"xmqq7cctzlxh.fsf@gitster.g","subject":"Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-07T04:49:25Z","receivedAt":"2024-08-07T04:49:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 06, 2024 at 08:54:34AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> >> > But if \"git refs verify\" does exercise all the new code paths (and\n> >> > the refactored code that existed before this series, sitting now in\n> >> > different places), then I do not have to worry about it. My question\n> >> > was primarily to extract \"even though we do not wire this up to fsck,\n> >> > we already have another code paths that uses all these changes\" out\n> >> > of you.\n> >> ...\n> > So, to summarize: the refactored functionality is both used and tested\n> > and I think it's sensible to defer the integration of git-fsck(1) and\n> > git-refs(1).\n> \n> After refactoring, existing functionality about objects are used, of\n> course, (there is no other code that does so), the refactoring lets\n> the code to learn to perform checks on references, and these new\n> checks are exercised by \"git refs verify\".\n> \n> I took what shejialuo said that way, and that is fine by me when I\n> said the above ;-). So I think we all are on the same page?\n\nYup, we are. I was mostly aiming to reassure Jialuo, who was a bit\nuncertain whether his answers had been sufficient or not.\n\nThanks!\n\nPatrick\n"},{"id":"500268","messageId":"ZrMMaubE_V7wUZ3H@tanuki","threadId":"61565","inReplyTo":"xmqqsevhy6e7.fsf@gitster.g","subject":"Re: [GSoC][PATCH v15 7/9] builtin/refs: add verify subcommand","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-07T05:55:54Z","receivedAt":"2024-08-07T05:55:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 06, 2024 at 09:15:28AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> >> +\tif (argc)\n> >> +\t\tusage(_(\"'git refs verify' takes no arguments\"));\n> >\n> > Junio has posted a patch series [1] where he wants to get rid of\n> > messages that simply say \"no arguments\" or \"too many arguments\".\n> > ...\n> > So I'd propose to make this:\n> >\n> > argc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n> > if (argc)\n> > usage(_(\"unknown argument: '%s'\", argv[0]));\n> \n> I probably should have said that I am fully behind the intent\n> against \"too many arguments\", but I am not 100% behind the\n> particular messaging used in the patch series I sent out.\n> \n> One potential complaint I expected to hear, for example, was that \"a\n> is unknown\" given when you said \"git cmd a a a a a\" is not all that\n> clear ;-). To alleviate, you would have to say \"git cmd takes only\n> 2 arguments\" if 'a' you are complaining about is the third one.\n> \n> Also, many people would consider that \"unexpected argument\" is\n> better than \"unknown argument\".\n> \n> I personally think the message above is absolutely clear and good.\n> \n> You say that 'git refs verify' takes no arguments, and for somebody\n> who said \"git refs verify a b c d e\", there is no doubt that all of\n> these a b c d e are unwanted. And there is no room to misinterpret\n> the message as \"'git refs' is ok but 'git refs verify' is already\n> unwelcome with extra argument\", either [*].\n> \n> In short, I think the message in the patch here is good, and it is\n> the other \"war on 'too many arguments'\" series whose messages need\n> to be thought further.\n\nJust to clarify: with \"the patch\" you probably refer to the current\nversion that Jialuo has, right? In other words, keep the current version\nthat he has and adapt the message in the future, when we have decided\nwhat to do about those \"too many arguments\" messages?\n\nIf so, then the only two I had were some spurious newlines. I'm not sure\nwhether these really would be worth rerolling the whole patch series.\n\nPatrick\n"},{"id":"500295","messageId":"CAOLa=ZRV4A2CyugxghHEE6qoaZ1b8aubyDOTC9PuDo1DJLa30A@mail.gmail.com","threadId":"61565","inReplyTo":"ZrEBsym854FtguPI@ArchLinux","subject":"Re: [GSoC][PATCH v15 3/9] fsck: make \"fsck_error\" callback generic","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-08-07T08:03:00Z","receivedAt":"2024-08-07T08:03:02Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> The \"fsck_error\" callback is designed to report the objects-related\n> error messages. It accepts two parameter \"oid\" and \"object_type\" which\n> is not generic. In order to provide a unified callback which can report\n> either objects or refs, remove the objects-related parameters and add\n> the generic parameter \"void *fsck_report\".\n>\n> Create a new \"fsck_object_report\" structure which incorporates the\n> removed parameters \"oid\" and \"object_type\". Then change the\n> corresponding references to adapt to new \"fsck_error\" callback.\n>\n\nSo since my last review, we've now changed the function to accept a\n'void *' parameter, this makes sense, this way the refs error function\nand the objects error function can now only receive the data they care\nabout.\n\n[snip]\n"},{"id":"500296","messageId":"CAOLa=ZQdmC+EQvxpqSjkXUo_PB2apkU2NteC49v6mvyxsByk8Q@mail.gmail.com","threadId":"61565","inReplyTo":"ZrEBvB9Urj8mG4i-@ArchLinux","subject":"Re: [GSoC][PATCH v15 4/9] fsck: add a unified interface for reporting fsck messages","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-08-07T08:05:39Z","receivedAt":"2024-08-07T08:05:41Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> The static function \"report\" provided by \"fsck.c\" aims at checking error\n> type and calling the callback \"error_func\" to report the message. Both\n> refs and objects need to check the error type of the current fsck\n> message. In order to extract this common behavior, create a new function\n> \"fsck_vreport\". Instead of using \"...\", provide \"va_list\" to allow more\n> flexibility.\n>\n> Instead of changing \"report\" prototype to be algin with the\n\nNit: s/be algin/align/\n\nRest of the patch looks great!\n\n[snip]\n"},{"id":"500301","messageId":"CAOLa=ZTuJC8HhSZo88PANAC4sntKWPFTprjbF5bks0tMg414pg@mail.gmail.com","threadId":"61565","inReplyTo":"ZrEBKjzbyxtMdCCx@ArchLinux","subject":"Re: [GSoC][PATCH v15 0/9] ref consistency check infra setup","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-08-07T09:29:31Z","receivedAt":"2024-08-07T09:29:32Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> Hi All:\n>\n> This version handles the following problems:\n>\n> 1. Patrick advices that I should not use `va_copy` in the changed\n> `report` function. Actually this is a mistake, this version avoids\n> redundant `ap` copy.\n> 2. Patrick advices I should rebase [v14 05/11] into [v14 04/11]. I\n> follow this advice in this version.\n> 3. Patrick advices that we should put [v14 06/11] before we introduce\n> ref-related operations. This version reorders the commit sequence. It's\n> a minor change.\n> 4. Patrick suggests at current we should not add `git refs verify`\n> command into \"git-fsck(1)\". This is because we should disable this new\n> check by default for the users. Many users use \"git-fsck(1)\" in their daily\n> workflow. We should not be aggressive. However, if we provide this\n> mechanism in this series, we will again make more complexity. So this\n> version drop patch [v14 09/11]. Also because of dropping, change the\n> test file to use \"git refs verify\" command instead of \"git fsck\"\n> command.\n\nThis is the biggest change in this version and it makes sense. It can\nstill be added later on, but for now users can use this via `git refs\nverify`.\n\n> 5. Patrick suggests that we should use `ends_with` instead of\n> `strip_suffix`, fix.\n>\n\nApart from the minor nits, I think this version looks good.\nThanks!\n"},{"id":"500310","messageId":"ZrNtjeV5_gyZXBgB@ArchLinux.localdomain","threadId":"61565","inReplyTo":"ZrMMaubE_V7wUZ3H@tanuki","subject":"Re: [GSoC][PATCH v15 7/9] builtin/refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-07T12:50:21Z","receivedAt":"2024-08-07T12:49:55Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"> If so, then the only two I had were some spurious newlines. I'm not sure\n> whether these really would be worth rerolling the whole patch series.\n> \n\nKarthik has given some reviews. I guess I need to reroll because there\nis one typo error in commit message. It's important to make this fixed.\n\n> Patrick\n\n"},{"id":"500333","messageId":"xmqqjzgspbti.fsf@gitster.g","threadId":"61565","inReplyTo":"ZrMMaubE_V7wUZ3H@tanuki","subject":"Re: [GSoC][PATCH v15 7/9] builtin/refs: add verify subcommand","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-07T15:55:21Z","receivedAt":"2024-08-07T15:55:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Tue, Aug 06, 2024 at 09:15:28AM -0700, Junio C Hamano wrote:\n>> Patrick Steinhardt writes:\n>> \n>> >> +\tif (argc)\n>> >> +\t\tusage(_(\"'git refs verify' takes no arguments\"));\n>> > \n>> > Junio has posted a patch series [1] where he wants to get rid of\n>> > messages that simply say \"no arguments\" or \"too many arguments\".\n>> > ...\n>> > So I'd propose to make this:\n>> >\n>> > argc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n>> > if (argc)\n>> > usage(_(\"unknown argument: '%s'\", argv[0]));\n>> \n>> I probably should have said that I am fully behind the intent\n>> against \"too many arguments\", but I am not 100% behind the\n>> particular messaging used in the patch series I sent out.\n>> \n>> One potential complaint I expected to hear, for example, was that \"a\n>> is unknown\" given when you said \"git cmd a a a a a\" is not all that\n>> clear ;-). To alleviate, you would have to say \"git cmd takes only\n>> 2 arguments\" if 'a' you are complaining about is the third one.\n>> \n>> Also, many people would consider that \"unexpected argument\" is\n>> better than \"unknown argument\".\n>> \n>> I personally think the message above is absolutely clear and good.\n>> \n>> You say that 'git refs verify' takes no arguments, and for somebody\n>> who said \"git refs verify a b c d e\", there is no doubt that all of\n>> these a b c d e are unwanted. And there is no room to misinterpret\n>> the message as \"'git refs' is ok but 'git refs verify' is already\n>> unwelcome with extra argument\", either [*].\n>> \n>> In short, I think the message in the patch here is good, and it is\n>> the other \"war on 'too many arguments'\" series whose messages need\n>> to be thought further.\n>\n> Just to clarify: with \"the patch\" you probably refer to the current\n> version that Jialuo has, right? In other words, keep the current version\n> that he has and adapt the message in the future, when we have decided\n> what to do about those \"too many arguments\" messages?\n\nI meant that I think (1) that \"'git refs verify' takes no arguments\"\nis a good message, and (2) that there is no further change needed to\nthe patch that started this review thread, regardless of how we want\nto deal with \"too many arguments\" messages.\n\n> If so, then the only two I had were some spurious newlines. I'm not sure\n> whether these really would be worth rerolling the whole patch series.\n\nYup, those blank lines were annoying while scanning the patches, but\nthey alone would not be something that makes a reroll _required_. A\nreroll that clearly shows that the incremental change since the last\nround is only these blank line changes is not too much to process,\nso \"not worth the reviewer time\" is not a huge reason to avoid it,\neither. I'd see that it is up to how perfectionist the patch\nsubmitter wants to be ;-)\n\nThanks.\n\n"},{"id":"500420","messageId":"CAOLa=ZTjVgVk_OodC+L+MC014vHbh6jztKNHyQjcXwKzmkqhbg@mail.gmail.com","threadId":"61565","inReplyTo":"ZrNtjeV5_gyZXBgB@ArchLinux.localdomain","subject":"Re: [GSoC][PATCH v15 7/9] builtin/refs: add verify subcommand","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-08-08T10:22:02Z","receivedAt":"2024-08-08T10:22:04Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n>> If so, then the only two I had were some spurious newlines. I'm not sure\n>> whether these really would be worth rerolling the whole patch series.\n>>\n>\n> Karthik has given some reviews. I guess I need to reroll because there\n> is one typo error in commit message. It's important to make this fixed.\n>\n>> Patrick\n\nI would say, my nits by themselves don't require a re-roll, but if\nyou're re-rolling, it'd be nice to resolve them too :)\n"},{"id":"500422","messageId":"ZrSqMmD-quQ18a9F@ArchLinux.localdomain","threadId":"61565","inReplyTo":"ZrEBKjzbyxtMdCCx@ArchLinux","subject":"[GSoC][PATCH v16 0/9] ref consistency check infra setup","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-08T11:21:22Z","receivedAt":"2024-08-08T11:20:55Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Hi All, this version handles some minor changes:\n\n1. Remove redundant newlines.\n2. Fix typo in commit message.\n\nAt last, I wanna thank every reviewer. As we can see, this series starts\nfrom 5.27, it's a very long journey. I have learned a lot.\n\nThanks\nJialuo\n\nshejialuo (9):\n fsck: rename \"skiplist\" to \"skip_oids\"\n fsck: rename objects-related fsck error functions\n fsck: make \"fsck_error\" callback generic\n fsck: add a unified interface for reporting fsck messages\n fsck: add refs report function\n refs: set up ref consistency check infrastructure\n builtin/refs: add verify subcommand\n files-backend: add unified interface for refs scanning\n fsck: add ref name check for files backend\n\n Documentation/fsck-msgids.txt | 6 ++\n Documentation/git-refs.txt | 13 ++++\n builtin/fsck.c | 17 +++--\n builtin/mktag.c | 3 +-\n builtin/refs.c | 34 +++++++++\n fsck.c | 125 +++++++++++++++++++++++++++-------\n fsck.h | 76 ++++++++++++++++-----\n object-file.c | 9 ++-\n refs.c | 5 ++\n refs.h | 8 +++\n refs/debug.c | 11 +++\n refs/files-backend.c | 115 ++++++++++++++++++++++++++++++-\n refs/packed-backend.c | 8 +++\n refs/refs-internal.h | 6 ++\n refs/reftable-backend.c | 8 +++\n t/t0602-reffiles-fsck.sh | 92 +++++++++++++++++++++++++\n 16 files changed, 477 insertions(+), 59 deletions(-)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\nRange-diff against v15:\n 1: 9aeaa3211c = 1: 9aeaa3211c fsck: rename \"skiplist\" to \"skip_oids\"\n 2: 7511340a21 = 2: 7511340a21 fsck: rename objects-related fsck error functions\n 3: ee971d17f4 = 3: ee971d17f4 fsck: make \"fsck_error\" callback generic\n 4: 59ccdab54d ! 4: f80fa00538 fsck: add a unified interface for reporting fsck messages\n @@ Commit message\n \"fsck_vreport\". Instead of using \"...\", provide \"va_list\" to allow more\n flexibility.\n \n - Instead of changing \"report\" prototype to be algin with the\n + Instead of changing \"report\" prototype to be align with the\n \"fsck_vreport\" function, we leave the \"report\" prototype unchanged due\n to the reason that there are nearly 62 references about \"report\"\n function. Simply change \"report\" function to use \"fsck_vreport\" to\n 5: b5607ac61c ! 5: 8c0376cfd5 fsck: add refs report function\n @@ fsck.c: int fsck_objects_error_function(struct fsck_options *o,\n +\t\t\t const char *message)\n +{\n +\tstruct fsck_ref_report *report = fsck_report;\n -+\n +\tstruct strbuf sb = STRBUF_INIT;\n +\tint ret = 0;\n +\n @@ fsck.c: int fsck_objects_error_function(struct fsck_options *o,\n +\n +\tstrbuf_release(&sb);\n +\treturn ret;\n -+\n +}\n +\n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n 6: 5819406b40 = 6: 552fe62ff4 refs: set up ref consistency check infrastructure\n 7: 6a9c194a05 = 7: 3b357fa89e builtin/refs: add verify subcommand\n 8: 76406b1303 ! 8: 8d2c7b3aa5 files-backend: add unified interface for refs scanning\n @@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store\n +\tif (o->verbose)\n +\t\tfprintf_ln(stderr, _(\"Checking references consistency\"));\n +\treturn files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fn);\n -+\n +}\n +\n static int files_fsck(struct ref_store *ref_store,\n 9: ed02380516 = 9: b39533ae56 fsck: add ref name check for files backend\n-- \n2.46.0\n\n"},{"id":"500423","messageId":"ZrSq3Z8tYrGwBOqC@ArchLinux","threadId":"61565","inReplyTo":"ZrSqMmD-quQ18a9F@ArchLinux.localdomain","subject":"[GSoC][PATCH v16 1/9] fsck: rename \"skiplist\" to \"skip_oids\"","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-08T11:24:13Z","receivedAt":"2024-08-08T11:23:45Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The \"skiplist\" field in \"fsck_options\" is related to objects. Because we\nare going to introduce ref consistency check, the \"skiplist\" name is too\ngeneral which will make the caller think \"skiplist\" is related to both\nthe refs and objects.\n\nIt may seem that for both refs and objects, we should provide a general\n\"skiplist\" here. However, the type for \"skiplist\" is `struct oidset`\nwhich is totally unsuitable for refs.\n\nTo avoid above ambiguity, rename \"skiplist\" to \"skip_oids\".\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 4 ++--\n fsck.h | 4 ++--\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex eea7145470..3f32441492 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n \t\tif (!strcmp(buf, \"skiplist\")) {\n \t\t\tif (equal == len)\n \t\t\t\tdie(\"skiplist requires a path\");\n-\t\t\toidset_parse_file(&options->skiplist, buf + equal + 1,\n+\t\t\toidset_parse_file(&options->skip_oids, buf + equal + 1,\n \t\t\t\t\t the_repository->hash_algo);\n \t\t\tbuf += len + 1;\n \t\t\tcontinue;\n@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)\n static int object_on_skiplist(struct fsck_options *opts,\n \t\t\t const struct object_id *oid)\n {\n-\treturn opts && oid && oidset_contains(&opts->skiplist, oid);\n+\treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n __attribute__((format (printf, 5, 6)))\ndiff --git a/fsck.h b/fsck.h\nindex 6085a384f6..bcfb2e34cd 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -136,7 +136,7 @@ struct fsck_options {\n \tfsck_error error_func;\n \tunsigned strict:1;\n \tenum fsck_msg_type *msg_type;\n-\tstruct oidset skiplist;\n+\tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n \tstruct oidset gitmodules_done;\n \tstruct oidset gitattributes_found;\n@@ -145,7 +145,7 @@ struct fsck_options {\n };\n \n #define FSCK_OPTIONS_DEFAULT { \\\n-\t.skiplist = OIDSET_INIT, \\\n+\t.skip_oids = OIDSET_INIT, \\\n \t.gitmodules_found = OIDSET_INIT, \\\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n-- \n2.46.0\n\n"},{"id":"500424","messageId":"ZrSq6XV8hKRZMrnt@ArchLinux","threadId":"61565","inReplyTo":"ZrSqMmD-quQ18a9F@ArchLinux.localdomain","subject":"[GSoC][PATCH v16 2/9] fsck: rename objects-related fsck error functions","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-08T11:24:25Z","receivedAt":"2024-08-08T11:23:56Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The names of objects-related fsck error functions are generic. It's OK\nwhen there is only object database check. However, we are going to\nintroduce refs database check report function. To avoid ambiguity,\nrename object-related fsck error functions to explicitly indicate these\nfunctions are used to report objects-related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 14 +++++++-------\n fsck.c | 17 +++++++++--------\n fsck.h | 26 +++++++++++++-------------\n 3 files changed, 29 insertions(+), 28 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d13a226c2e..6d86bbe1e9 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -89,12 +89,12 @@ static int objerror(struct object *obj, const char *err)\n \treturn -1;\n }\n \n-static int fsck_error_func(struct fsck_options *o UNUSED,\n-\t\t\t const struct object_id *oid,\n-\t\t\t enum object_type object_type,\n-\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t const char *message)\n+static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t enum object_type object_type,\n+\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \tswitch (msg_type) {\n \tcase FSCK_WARN:\n@@ -938,7 +938,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tfsck_walk_options.walk = mark_object;\n \tfsck_obj_options.walk = mark_used;\n-\tfsck_obj_options.error_func = fsck_error_func;\n+\tfsck_obj_options.error_func = fsck_objects_error_func;\n \tif (check_strict)\n \t\tfsck_obj_options.strict = 1;\n \ndiff --git a/fsck.c b/fsck.c\nindex 3f32441492..8347842cfb 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1200,7 +1200,7 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n \t\t type);\n }\n \n-int fsck_error_function(struct fsck_options *o,\n+int fsck_objects_error_function(struct fsck_options *o,\n \t\t\tconst struct object_id *oid,\n \t\t\tenum object_type object_type UNUSED,\n \t\t\tenum fsck_msg_type msg_type,\n@@ -1303,16 +1303,17 @@ int git_fsck_config(const char *var, const char *value,\n * Custom error callbacks that are used in more than one place.\n */\n \n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message)\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message)\n {\n \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n \t\tputs(oid_to_hex(oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_error_function(o, oid, object_type, msg_type, msg_id, message);\n+\treturn fsck_objects_error_function(o, oid, object_type,\n+\t\t\t\t\t msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex bcfb2e34cd..41ebebbb59 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -120,16 +120,16 @@ typedef int (*fsck_error)(struct fsck_options *o,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n-int fsck_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid, enum object_type object_type,\n-\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n-\t\t\tconst char *message);\n-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t enum object_type object_type,\n-\t\t\t\t\t enum fsck_msg_type msg_type,\n-\t\t\t\t\t enum fsck_msg_id msg_id,\n-\t\t\t\t\t const char *message);\n+int fsck_objects_error_function(struct fsck_options *o,\n+\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n+\t\t\t\tconst char *message);\n+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n+\t\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t\t\t\t const char *message);\n \n struct fsck_options {\n \tfsck_walk_func walk;\n@@ -150,7 +150,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function \\\n+\t.error_func = fsck_objects_error_function \\\n }\n #define FSCK_OPTIONS_STRICT { \\\n \t.strict = 1, \\\n@@ -158,7 +158,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_function, \\\n+\t.error_func = fsck_objects_error_function, \\\n }\n #define FSCK_OPTIONS_MISSING_GITMODULES { \\\n \t.strict = 1, \\\n@@ -166,7 +166,7 @@ struct fsck_options {\n \t.gitmodules_done = OIDSET_INIT, \\\n \t.gitattributes_found = OIDSET_INIT, \\\n \t.gitattributes_done = OIDSET_INIT, \\\n-\t.error_func = fsck_error_cb_print_missing_gitmodules, \\\n+\t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n \n /* descend in all linked child objects\n-- \n2.46.0\n\n"},{"id":"500425","messageId":"ZrSrdwOCWrXpMYIA@ArchLinux","threadId":"61565","inReplyTo":"ZrSqMmD-quQ18a9F@ArchLinux.localdomain","subject":"[GSoC][PATCH v16 3/9] fsck: make \"fsck_error\" callback generic","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-08T11:26:47Z","receivedAt":"2024-08-08T11:26:19Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The \"fsck_error\" callback is designed to report the objects-related\nerror messages. It accepts two parameter \"oid\" and \"object_type\" which\nis not generic. In order to provide a unified callback which can report\neither objects or refs, remove the objects-related parameters and add\nthe generic parameter \"void *fsck_report\".\n\nCreate a new \"fsck_object_report\" structure which incorporates the\nremoved parameters \"oid\" and \"object_type\". Then change the\ncorresponding references to adapt to new \"fsck_error\" callback.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n builtin/fsck.c | 7 +++++--\n builtin/mktag.c | 3 +--\n fsck.c | 26 ++++++++++++++++----------\n fsck.h | 17 ++++++++++++-----\n object-file.c | 9 ++++-----\n 5 files changed, 38 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 6d86bbe1e9..766bbd014d 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -90,12 +90,15 @@ static int objerror(struct object *obj, const char *err)\n }\n \n static int fsck_objects_error_func(struct fsck_options *o UNUSED,\n-\t\t\t\t const struct object_id *oid,\n-\t\t\t\t enum object_type object_type,\n+\t\t\t\t void *fsck_report,\n \t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t\t const char *message)\n {\n+\tstruct fsck_object_report *report = fsck_report;\n+\tconst struct object_id *oid = report->oid;\n+\tenum object_type object_type = report->object_type;\n+\n \tswitch (msg_type) {\n \tcase FSCK_WARN:\n \t\t/* TRANSLATORS: e.g. warning in tree 01bfda: */\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex 4767f1a97e..c6b644219f 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -18,8 +18,7 @@ static int option_strict = 1;\n static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;\n \n static int mktag_fsck_error_func(struct fsck_options *o UNUSED,\n-\t\t\t\t const struct object_id *oid UNUSED,\n-\t\t\t\t enum object_type object_type UNUSED,\n+\t\t\t\t void *fsck_report UNUSED,\n \t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t enum fsck_msg_id msg_id UNUSED,\n \t\t\t\t const char *message)\ndiff --git a/fsck.c b/fsck.c\nindex 8347842cfb..cca6ae144f 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -232,6 +232,10 @@ static int report(struct fsck_options *options,\n \t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n {\n \tva_list ap;\n+\tstruct fsck_object_report report = {\n+\t\t.oid = oid,\n+\t\t.object_type = object_type\n+\t};\n \tstruct strbuf sb = STRBUF_INIT;\n \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n \tint result;\n@@ -252,7 +256,7 @@ static int report(struct fsck_options *options,\n \n \tva_start(ap, fmt);\n \tstrbuf_vaddf(&sb, fmt, ap);\n-\tresult = options->error_func(options, oid, object_type,\n+\tresult = options->error_func(options, &report,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n \tva_end(ap);\n@@ -1201,12 +1205,14 @@ int fsck_buffer(const struct object_id *oid, enum object_type type,\n }\n \n int fsck_objects_error_function(struct fsck_options *o,\n-\t\t\tconst struct object_id *oid,\n-\t\t\tenum object_type object_type UNUSED,\n-\t\t\tenum fsck_msg_type msg_type,\n-\t\t\tenum fsck_msg_id msg_id UNUSED,\n-\t\t\tconst char *message)\n+\t\t\t\tvoid *fsck_report,\n+\t\t\t\tenum fsck_msg_type msg_type,\n+\t\t\t\tenum fsck_msg_id msg_id UNUSED,\n+\t\t\t\tconst char *message)\n {\n+\tstruct fsck_object_report *report = fsck_report;\n+\tconst struct object_id *oid = report->oid;\n+\n \tif (msg_type == FSCK_WARN) {\n \t\twarning(\"object %s: %s\", fsck_describe_object(o, oid), message);\n \t\treturn 0;\n@@ -1304,16 +1310,16 @@ int git_fsck_config(const char *var, const char *value,\n */\n \n int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t void *fsck_report,\n \t\t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message)\n {\n \tif (msg_id == FSCK_MSG_GITMODULES_MISSING) {\n-\t\tputs(oid_to_hex(oid));\n+\t\tstruct fsck_object_report *report = fsck_report;\n+\t\tputs(oid_to_hex(report->oid));\n \t\treturn 0;\n \t}\n-\treturn fsck_objects_error_function(o, oid, object_type,\n+\treturn fsck_objects_error_function(o, fsck_report,\n \t\t\t\t\t msg_type, msg_id, message);\n }\ndiff --git a/fsck.h b/fsck.h\nindex 41ebebbb59..3b80d02506 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -114,23 +114,30 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);\n typedef int (*fsck_walk_func)(struct object *obj, enum object_type object_type,\n \t\t\t void *data, struct fsck_options *options);\n \n-/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n+/*\n+ * Callback for reporting errors either for objects or refs. The \"fsck_report\"\n+ * is a generic pointer that can be used to pass any information.\n+ */\n typedef int (*fsck_error)(struct fsck_options *o,\n-\t\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t\t void *fsck_report,\n \t\t\t enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t const char *message);\n \n int fsck_objects_error_function(struct fsck_options *o,\n-\t\t\t\tconst struct object_id *oid, enum object_type object_type,\n+\t\t\t\tvoid *fsck_report,\n \t\t\t\tenum fsck_msg_type msg_type, enum fsck_msg_id msg_id,\n \t\t\t\tconst char *message);\n int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n-\t\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t\t enum object_type object_type,\n+\t\t\t\t\t\t void *fsck_report,\n \t\t\t\t\t\t enum fsck_msg_type msg_type,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message);\n \n+struct fsck_object_report {\n+\tconst struct object_id *oid;\n+\tenum object_type object_type;\n+};\n+\n struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\ndiff --git a/object-file.c b/object-file.c\nindex 065103be3e..05ac6ebed6 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2470,11 +2470,10 @@ int repo_has_object_file(struct repository *r,\n * give more context.\n */\n static int hash_format_check_report(struct fsck_options *opts UNUSED,\n-\t\t\t\t const struct object_id *oid UNUSED,\n-\t\t\t\t enum object_type object_type UNUSED,\n-\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n-\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n-\t\t\t\t const char *message)\n+\t\t\t\t void *fsck_report UNUSED,\n+\t\t\t\t enum fsck_msg_type msg_type UNUSED,\n+\t\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t\t const char *message)\n {\n \terror(_(\"object fails fsck: %s\"), message);\n \treturn 1;\n-- \n2.46.0\n\n"},{"id":"500426","messageId":"ZrSrgRGwI_jldprn@ArchLinux","threadId":"61565","inReplyTo":"ZrSqMmD-quQ18a9F@ArchLinux.localdomain","subject":"[GSoC][PATCH v16 4/9] fsck: add a unified interface for reporting fsck messages","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-08T11:26:57Z","receivedAt":"2024-08-08T11:26:29Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The static function \"report\" provided by \"fsck.c\" aims at checking error\ntype and calling the callback \"error_func\" to report the message. Both\nrefs and objects need to check the error type of the current fsck\nmessage. In order to extract this common behavior, create a new function\n\"fsck_vreport\". Instead of using \"...\", provide \"va_list\" to allow more\nflexibility.\n\nInstead of changing \"report\" prototype to be align with the\n\"fsck_vreport\" function, we leave the \"report\" prototype unchanged due\nto the reason that there are nearly 62 references about \"report\"\nfunction. Simply change \"report\" function to use \"fsck_vreport\" to\nreport objects related messages.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 44 ++++++++++++++++++++++++++++++--------------\n 1 file changed, 30 insertions(+), 14 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex cca6ae144f..3614aa56a3 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -226,16 +226,15 @@ static int object_on_skiplist(struct fsck_options *opts,\n \treturn opts && oid && oidset_contains(&opts->skip_oids, oid);\n }\n \n-__attribute__((format (printf, 5, 6)))\n-static int report(struct fsck_options *options,\n-\t\t const struct object_id *oid, enum object_type object_type,\n-\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+/*\n+ * Provide the common functionality for either fscking refs or objects.\n+ * It will get the current msg error type and call the error_func callback\n+ * which is registered in the \"fsck_options\" struct.\n+ */\n+static int fsck_vreport(struct fsck_options *options,\n+\t\t\tvoid *fsck_report,\n+\t\t\tenum fsck_msg_id msg_id, const char *fmt, va_list ap)\n {\n-\tva_list ap;\n-\tstruct fsck_object_report report = {\n-\t\t.oid = oid,\n-\t\t.object_type = object_type\n-\t};\n \tstruct strbuf sb = STRBUF_INIT;\n \tenum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);\n \tint result;\n@@ -243,9 +242,6 @@ static int report(struct fsck_options *options,\n \tif (msg_type == FSCK_IGNORE)\n \t\treturn 0;\n \n-\tif (object_on_skiplist(options, oid))\n-\t\treturn 0;\n-\n \tif (msg_type == FSCK_FATAL)\n \t\tmsg_type = FSCK_ERROR;\n \telse if (msg_type == FSCK_INFO)\n@@ -254,11 +250,31 @@ static int report(struct fsck_options *options,\n \tprepare_msg_ids();\n \tstrbuf_addf(&sb, \"%s: \", msg_id_info[msg_id].camelcased);\n \n-\tva_start(ap, fmt);\n \tstrbuf_vaddf(&sb, fmt, ap);\n-\tresult = options->error_func(options, &report,\n+\tresult = options->error_func(options, fsck_report,\n \t\t\t\t msg_type, msg_id, sb.buf);\n \tstrbuf_release(&sb);\n+\n+\treturn result;\n+}\n+\n+__attribute__((format (printf, 5, 6)))\n+static int report(struct fsck_options *options,\n+\t\t const struct object_id *oid, enum object_type object_type,\n+\t\t enum fsck_msg_id msg_id, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tstruct fsck_object_report report = {\n+\t\t.oid = oid,\n+\t\t.object_type = object_type\n+\t};\n+\tint result;\n+\n+\tif (object_on_skiplist(options, oid))\n+\t\treturn 0;\n+\n+\tva_start(ap, fmt);\n+\tresult = fsck_vreport(options, &report, msg_id, fmt, ap);\n \tva_end(ap);\n \n \treturn result;\n-- \n2.46.0\n\n"},{"id":"500427","messageId":"ZrSrjO2ltoJuppKA@ArchLinux","threadId":"61565","inReplyTo":"ZrSqMmD-quQ18a9F@ArchLinux.localdomain","subject":"[GSoC][PATCH v16 5/9] fsck: add refs report function","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-08T11:27:08Z","receivedAt":"2024-08-08T11:26:39Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new struct \"fsck_ref_report\" to contain the information we\nneed when reporting refs-related messages.\n\nWith the new \"fsck_vreport\" function, add a new function\n\"fsck_report_ref\" to report refs-related fsck error message. Unlike\n\"report\" function uses the exact parameters, we simply pass \"struct\nfsck_ref_report *report\" as the parameter. This is because at current we\ndon't know exactly how many fields we need. By passing this parameter,\nwe don't need to change this function prototype when we want to add more\ninformation into \"fsck_ref_report\".\n\nWe have introduced \"fsck_report_ref\" function to report the error\nmessage for refs. We still need to add the corresponding callback\nfunction. Create refs-specific \"error_func\" callback\n\"fsck_refs_error_function\".\n\nLast, add \"FSCK_REFS_OPTIONS_DEFAULT\" macro to create default options\nwhen checking ref consistency.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n fsck.c | 39 +++++++++++++++++++++++++++++++++++++++\n fsck.h | 25 +++++++++++++++++++++++++\n 2 files changed, 64 insertions(+)\n\ndiff --git a/fsck.c b/fsck.c\nindex 3614aa56a3..e16c892f6a 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -280,6 +280,19 @@ static int report(struct fsck_options *options,\n \treturn result;\n }\n \n+int fsck_report_ref(struct fsck_options *options,\n+\t\t struct fsck_ref_report *report,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tint result;\n+\tva_start(ap, fmt);\n+\tresult = fsck_vreport(options, report, msg_id, fmt, ap);\n+\tva_end(ap);\n+\treturn result;\n+}\n+\n void fsck_enable_object_names(struct fsck_options *options)\n {\n \tif (!options->object_names)\n@@ -1237,6 +1250,32 @@ int fsck_objects_error_function(struct fsck_options *o,\n \treturn 1;\n }\n \n+int fsck_refs_error_function(struct fsck_options *options UNUSED,\n+\t\t\t void *fsck_report,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id UNUSED,\n+\t\t\t const char *message)\n+{\n+\tstruct fsck_ref_report *report = fsck_report;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tstrbuf_addstr(&sb, report->path);\n+\n+\tif (report->oid)\n+\t\tstrbuf_addf(&sb, \" -> (%s)\", oid_to_hex(report->oid));\n+\telse if (report->referent)\n+\t\tstrbuf_addf(&sb, \" -> (%s)\", report->referent);\n+\n+\tif (msg_type == FSCK_WARN)\n+\t\twarning(\"%s: %s\", sb.buf, message);\n+\telse\n+\t\tret = error(\"%s: %s\", sb.buf, message);\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,\n \t\t enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,\n \t\t struct fsck_options *options, const char *blob_type)\ndiff --git a/fsck.h b/fsck.h\nindex 3b80d02506..2002590f60 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -133,11 +133,23 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,\n \t\t\t\t\t\t enum fsck_msg_id msg_id,\n \t\t\t\t\t\t const char *message);\n \n+int fsck_refs_error_function(struct fsck_options *options,\n+\t\t\t void *fsck_report,\n+\t\t\t enum fsck_msg_type msg_type,\n+\t\t\t enum fsck_msg_id msg_id,\n+\t\t\t const char *message);\n+\n struct fsck_object_report {\n \tconst struct object_id *oid;\n \tenum object_type object_type;\n };\n \n+struct fsck_ref_report {\n+\tconst char *path;\n+\tconst struct object_id *oid;\n+\tconst char *referent;\n+};\n+\n struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n@@ -175,6 +187,9 @@ struct fsck_options {\n \t.gitattributes_done = OIDSET_INIT, \\\n \t.error_func = fsck_objects_error_cb_print_missing_gitmodules, \\\n }\n+#define FSCK_REFS_OPTIONS_DEFAULT { \\\n+\t.error_func = fsck_refs_error_function, \\\n+}\n \n /* descend in all linked child objects\n * the return value is:\n@@ -216,6 +231,16 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+/*\n+ * Report an error or warning for refs.\n+ */\n+__attribute__((format (printf, 4, 5)))\n+int fsck_report_ref(struct fsck_options *options,\n+\t\t struct fsck_ref_report *report,\n+\t\t enum fsck_msg_id msg_id,\n+\t\t const char *fmt, ...);\n+\n+\n /*\n * Subsystem for storing human-readable names for each object.\n *\n-- \n2.46.0\n\n"},{"id":"500428","messageId":"ZrSrlU8TesYsTb2C@ArchLinux","threadId":"61565","inReplyTo":"ZrSqMmD-quQ18a9F@ArchLinux.localdomain","subject":"[GSoC][PATCH v16 6/9] refs: set up ref consistency check infrastructure","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-08T11:27:17Z","receivedAt":"2024-08-08T11:26:49Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The \"struct ref_store\" is the base class which contains the \"be\" pointer\nwhich provides backend-specific functions whose interfaces are defined\nin the \"ref_storage_be\". We could reuse this polymorphism to define only\none interface. For every backend, we need to provide its own function\npointer.\n\nThe interfaces defined in the `ref_storage_be` are carefully structured\nin semantic. It's organized as the five parts:\n\n1. The name and the initialization interfaces.\n2. The ref transaction interfaces.\n3. The ref internal interfaces (pack, rename and copy).\n4. The ref filesystem interfaces.\n5. The reflog related interfaces.\n\nTo keep consistent with the git-fsck(1), add a new interface named\n\"fsck_refs_fn\" to the end of \"ref_storage_be\". This semantic cannot be\ngrouped into any above five categories. Explicitly add blank line to\nmake it different from others.\n\nLast, implement placeholder functions for each ref backends.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n refs.c | 5 +++++\n refs.h | 8 ++++++++\n refs/debug.c | 11 +++++++++++\n refs/files-backend.c | 13 ++++++++++++-\n refs/packed-backend.c | 8 ++++++++\n refs/refs-internal.h | 6 ++++++\n refs/reftable-backend.c | 8 ++++++++\n 7 files changed, 58 insertions(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex 915aeb4d1d..6f642dc681 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)\n \treturn check_or_sanitize_refname(refname, flags, NULL);\n }\n \n+int refs_fsck(struct ref_store *refs, struct fsck_options *o)\n+{\n+\treturn refs->be->fsck(refs, o);\n+}\n+\n void sanitize_refname_component(const char *refname, struct strbuf *out)\n {\n \tif (check_or_sanitize_refname(refname, REFNAME_ALLOW_ONELEVEL, out))\ndiff --git a/refs.h b/refs.h\nindex b3e39bc257..405073621a 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -4,6 +4,7 @@\n #include \"commit.h\"\n #include \"repository.h\"\n \n+struct fsck_options;\n struct object_id;\n struct ref_store;\n struct strbuf;\n@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat\n */\n int check_refname_format(const char *refname, int flags);\n \n+/*\n+ * Check the reference database for consistency. Return 0 if refs and\n+ * reflogs are consistent, and non-zero otherwise. The errors will be\n+ * written to stderr.\n+ */\n+int refs_fsck(struct ref_store *refs, struct fsck_options *o);\n+\n /*\n * Apply the rules from check_refname_format, but mutate the result until it\n * is acceptable, and place the result in \"out\".\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 547d9245b9..45e2e784a0 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \treturn res;\n }\n \n+static int debug_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tint res = drefs->refs->be->fsck(drefs->refs, o);\n+\ttrace_printf_key(&trace_refs, \"fsck: %d\\n\", res);\n+\treturn res;\n+}\n+\n struct ref_storage_be refs_be_debug = {\n \t.name = \"debug\",\n \t.init = NULL,\n@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = {\n \t.create_reflog = debug_create_reflog,\n \t.delete_reflog = debug_delete_reflog,\n \t.reflog_expire = debug_reflog_expire,\n+\n+\t.fsck = debug_fsck,\n };\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex aa52d9be7c..4630eb1f80 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3408,6 +3408,15 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int files_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\tstruct files_ref_store *refs =\n+\t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+}\n+\n struct ref_storage_be refs_be_files = {\n \t.name = \"files\",\n \t.init = files_ref_store_init,\n@@ -3434,5 +3443,7 @@ struct ref_storage_be refs_be_files = {\n \t.reflog_exists = files_reflog_exists,\n \t.create_reflog = files_create_reflog,\n \t.delete_reflog = files_delete_reflog,\n-\t.reflog_expire = files_reflog_expire\n+\t.reflog_expire = files_reflog_expire,\n+\n+\t.fsck = files_fsck,\n };\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a0666407cd..5209b0b212 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_s\n \treturn empty_ref_iterator_begin();\n }\n \n+static int packed_fsck(struct ref_store *ref_store,\n+\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_packed = {\n \t.name = \"packed\",\n \t.init = packed_ref_store_init,\n@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = {\n \t.create_reflog = NULL,\n \t.delete_reflog = NULL,\n \t.reflog_expire = NULL,\n+\n+\t.fsck = packed_fsck,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex fa975d69aa..a905e187cd 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -4,6 +4,7 @@\n #include \"refs.h\"\n #include \"iterator.h\"\n \n+struct fsck_options;\n struct ref_transaction;\n \n /*\n@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,\n typedef int read_symbolic_ref_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t\t struct strbuf *referent);\n \n+typedef int fsck_fn(struct ref_store *ref_store,\n+\t\t struct fsck_options *o);\n+\n struct ref_storage_be {\n \tconst char *name;\n \tref_store_init_fn *init;\n@@ -677,6 +681,8 @@ struct ref_storage_be {\n \tcreate_reflog_fn *create_reflog;\n \tdelete_reflog_fn *delete_reflog;\n \treflog_expire_fn *reflog_expire;\n+\n+\tfsck_fn *fsck;\n };\n \n extern struct ref_storage_be refs_be_files;\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex fbe74c239d..b5a1a526df 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+static int reftable_be_fsck(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_reftable = {\n \t.name = \"reftable\",\n \t.init = reftable_be_init,\n@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = {\n \t.create_reflog = reftable_be_create_reflog,\n \t.delete_reflog = reftable_be_delete_reflog,\n \t.reflog_expire = reftable_be_reflog_expire,\n+\n+\t.fsck = reftable_be_fsck,\n };\n-- \n2.46.0\n\n"},{"id":"500429","messageId":"ZrSroE8vLlZCK2jp@ArchLinux","threadId":"61565","inReplyTo":"ZrSqMmD-quQ18a9F@ArchLinux.localdomain","subject":"[GSoC][PATCH v16 7/9] builtin/refs: add verify subcommand","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-08T11:27:28Z","receivedAt":"2024-08-08T11:27:00Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"Introduce a new subcommand \"verify\" in git-refs(1) to allow the user to\ncheck the reference database consistency and also this subcommand will\nbe used as the entry point of checking refs for \"git-fsck(1)\".\n\nAdd \"verbose\" field into \"fsck_options\" to indicate whether we should\nprint verbose messages when checking refs and objects consistency.\n\nRemove bit-field for \"strict\" field, this is because we cannot take\naddress of a bit-field which makes it unhandy to set member variables\nwhen parsing the command line options.\n\nThe \"git-fsck(1)\" declares \"fsck_options\" variable with \"static\"\nidentifier which avoids complaint by the leak-checker. However, in\n\"git-refs verify\", we need to do memory clean manually. Thus add\n\"fsck_options_clear\" function in \"fsck.c\" to provide memory clean\noperation.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/git-refs.txt | 13 +++++++++++++\n builtin/refs.c | 34 ++++++++++++++++++++++++++++++++++\n fsck.c | 11 +++++++++++\n fsck.h | 8 +++++++-\n 4 files changed, 65 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-refs.txt b/Documentation/git-refs.txt\nindex 5b99e04385..ce31f93061 100644\n--- a/Documentation/git-refs.txt\n+++ b/Documentation/git-refs.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git refs migrate' --ref-format= [--dry-run]\n+'git refs verify' [--strict] [--verbose]\n \n DESCRIPTION\n -----------\n@@ -22,6 +23,9 @@ COMMANDS\n migrate::\n \tMigrate ref store between different formats.\n \n+verify::\n+\tVerify reference database consistency.\n+\n OPTIONS\n -------\n \n@@ -39,6 +43,15 @@ include::ref-storage-format.txt[]\n \tcan be used to double check that the migration works as expected before\n \tperforming the actual migration.\n \n+The following options are specific to 'git refs verify':\n+\n+--strict::\n+\tEnable stricter error checking. This will cause warnings to be\n+\treported as errors. See linkgit:git-fsck[1].\n+\n+--verbose::\n+\tWhen verifying the reference database consistency, be chatty.\n+\n KNOWN LIMITATIONS\n -----------------\n \ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 46dcd150d4..131f98be98 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -1,4 +1,6 @@\n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n #include \"refs.h\"\n #include \"repository.h\"\n@@ -7,6 +9,9 @@\n #define REFS_MIGRATE_USAGE \\\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n+#define REFS_VERIFY_USAGE \\\n+\tN_(\"git refs verify [--strict] [--verbose]\")\n+\n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const migrate_usage[] = {\n@@ -58,15 +63,44 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n \treturn err;\n }\n \n+static int cmd_refs_verify(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;\n+\tconst char * const verify_usage[] = {\n+\t\tREFS_VERIFY_USAGE,\n+\t\tNULL,\n+\t};\n+\tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"verbose\", &fsck_refs_options.verbose, N_(\"be verbose\")),\n+\t\tOPT_BOOL(0, \"strict\", &fsck_refs_options.strict, N_(\"enable strict checking\")),\n+\t\tOPT_END(),\n+\t};\n+\tint ret;\n+\n+\targc = parse_options(argc, argv, prefix, options, verify_usage, 0);\n+\tif (argc)\n+\t\tusage(_(\"'git refs verify' takes no arguments\"));\n+\n+\tgit_config(git_fsck_config, &fsck_refs_options);\n+\tprepare_repo_settings(the_repository);\n+\n+\tret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);\n+\n+\tfsck_options_clear(&fsck_refs_options);\n+\treturn ret;\n+}\n+\n int cmd_refs(int argc, const char **argv, const char *prefix)\n {\n \tconst char * const refs_usage[] = {\n \t\tREFS_MIGRATE_USAGE,\n+\t\tREFS_VERIFY_USAGE,\n \t\tNULL,\n \t};\n \tparse_opt_subcommand_fn *fn = NULL;\n \tstruct option opts[] = {\n \t\tOPT_SUBCOMMAND(\"migrate\", &fn, cmd_refs_migrate),\n+\t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_refs_verify),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/fsck.c b/fsck.c\nindex e16c892f6a..3756f52459 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -1331,6 +1331,17 @@ int fsck_finish(struct fsck_options *options)\n \treturn ret;\n }\n \n+void fsck_options_clear(struct fsck_options *options)\n+{\n+\tfree(options->msg_type);\n+\toidset_clear(&options->skip_oids);\n+\toidset_clear(&options->gitmodules_found);\n+\toidset_clear(&options->gitmodules_done);\n+\toidset_clear(&options->gitattributes_found);\n+\toidset_clear(&options->gitattributes_done);\n+\tkh_clear_oid_map(options->object_names);\n+}\n+\n int git_fsck_config(const char *var, const char *value,\n \t\t const struct config_context *ctx, void *cb)\n {\ndiff --git a/fsck.h b/fsck.h\nindex 2002590f60..d551a9fe86 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -153,7 +153,8 @@ struct fsck_ref_report {\n struct fsck_options {\n \tfsck_walk_func walk;\n \tfsck_error error_func;\n-\tunsigned strict:1;\n+\tunsigned strict;\n+\tunsigned verbose;\n \tenum fsck_msg_type *msg_type;\n \tstruct oidset skip_oids;\n \tstruct oidset gitmodules_found;\n@@ -231,6 +232,11 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,\n */\n int fsck_finish(struct fsck_options *options);\n \n+/*\n+ * Clear the fsck_options struct, freeing any allocated memory.\n+ */\n+void fsck_options_clear(struct fsck_options *options);\n+\n /*\n * Report an error or warning for refs.\n */\n-- \n2.46.0\n\n"},{"id":"500431","messageId":"ZrSsk27zqOqSXTpH@ArchLinux","threadId":"61565","inReplyTo":"ZrSqMmD-quQ18a9F@ArchLinux.localdomain","subject":"[GSoC][PATCH v16 8/9] files-backend: add unified interface for refs scanning","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-08T11:31:31Z","receivedAt":"2024-08-08T11:31:02Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"For refs and reflogs, we need to scan its corresponding directories to\ncheck every regular file or symbolic link which shares the same pattern.\nIntroduce a unified interface for scanning directories for\nfiles-backend.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 ++\n fsck.h | 1 +\n refs/files-backend.c | 73 ++++++++++++++++++++++++++++++++++-\n 3 files changed, 76 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex f643585a34..7c809fddf1 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -19,6 +19,9 @@\n `badParentSha1`::\n \t(ERROR) A commit object has a bad parent sha1.\n \n+`badRefFiletype`::\n+\t(ERROR) A ref has a bad file type.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex d551a9fe86..af02174973 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -31,6 +31,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_NAME, ERROR) \\\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REF_FILETYPE, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 4630eb1f80..e511e1dcce 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -6,6 +6,7 @@\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n+#include \"../fsck.h\"\n #include \"../refs.h\"\n #include \"refs-internal.h\"\n #include \"ref-cache.h\"\n@@ -3408,13 +3409,83 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,\n \treturn ret;\n }\n \n+/*\n+ * For refs and reflogs, they share a unified interface when scanning\n+ * the whole directory. This function is used as the callback for each\n+ * regular file or symlink in the directory.\n+ */\n+typedef int (*files_fsck_refs_fn)(struct ref_store *ref_store,\n+\t\t\t\t struct fsck_options *o,\n+\t\t\t\t const char *refs_check_dir,\n+\t\t\t\t struct dir_iterator *iter);\n+\n+static int files_fsck_refs_dir(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o,\n+\t\t\t const char *refs_check_dir,\n+\t\t\t files_fsck_refs_fn *fsck_refs_fn)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct dir_iterator *iter;\n+\tint iter_status;\n+\tint ret = 0;\n+\n+\tstrbuf_addf(&sb, \"%s/%s\", ref_store->gitdir, refs_check_dir);\n+\n+\titer = dir_iterator_begin(sb.buf, 0);\n+\tif (!iter) {\n+\t\tret = error_errno(_(\"cannot open directory %s\"), sb.buf);\n+\t\tgoto out;\n+\t}\n+\n+\twhile ((iter_status = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tif (S_ISDIR(iter->st.st_mode)) {\n+\t\t\tcontinue;\n+\t\t} else if (S_ISREG(iter->st.st_mode) ||\n+\t\t\t S_ISLNK(iter->st.st_mode)) {\n+\t\t\tif (o->verbose)\n+\t\t\t\tfprintf_ln(stderr, \"Checking %s/%s\",\n+\t\t\t\t\t refs_check_dir, iter->relative_path);\n+\t\t\tfor (size_t i = 0; fsck_refs_fn[i]; i++) {\n+\t\t\t\tif (fsck_refs_fn[i](ref_store, o, refs_check_dir, iter))\n+\t\t\t\t\tret = -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tstruct fsck_ref_report report = { .path = iter->basename };\n+\t\t\tif (fsck_report_ref(o, &report,\n+\t\t\t\t\t FSCK_MSG_BAD_REF_FILETYPE,\n+\t\t\t\t\t \"unexpected file type\"))\n+\t\t\t\tret = -1;\n+\t\t}\n+\t}\n+\n+\tif (iter_status != ITER_DONE)\n+\t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n+\n+out:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int files_fsck_refs(struct ref_store *ref_store,\n+\t\t\t struct fsck_options *o)\n+{\n+\tfiles_fsck_refs_fn fsck_refs_fn[]= {\n+\t\tNULL,\n+\t};\n+\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, _(\"Checking references consistency\"));\n+\treturn files_fsck_refs_dir(ref_store, o, \"refs\", fsck_refs_fn);\n+}\n+\n static int files_fsck(struct ref_store *ref_store,\n \t\t struct fsck_options *o)\n {\n \tstruct files_ref_store *refs =\n \t\tfiles_downcast(ref_store, REF_STORE_READ, \"fsck\");\n \n-\treturn refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n+\treturn files_fsck_refs(ref_store, o) |\n+\t refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);\n }\n \n struct ref_storage_be refs_be_files = {\n-- \n2.46.0\n\n"},{"id":"500432","messageId":"ZrSsngBqfyTPDg7g@ArchLinux","threadId":"61565","inReplyTo":"ZrSqMmD-quQ18a9F@ArchLinux.localdomain","subject":"[GSoC][PATCH v16 9/9] fsck: add ref name check for files backend","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-08-08T11:31:42Z","receivedAt":"2024-08-08T11:31:14Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"The git-fsck(1) only implicitly checks the reference, it does not fully\ncheck refs with bad format name such as standalone \"@\".\n\nHowever, a file ending with \".lock\" should not be marked as having a bad\nref name. It is expected that concurrent writers may have such lock files.\nWe currently ignore this situation. But for bare \".lock\" file, we will\nreport it as error.\n\nIn order to provide such checks, add a new fsck message id \"badRefName\"\nwith default ERROR type. Use existing \"check_refname_format\" to explicit\ncheck the ref name. And add a new unit test to verify the functionality.\n\nMentored-by: Patrick Steinhardt \nMentored-by: Karthik Nayak \nSigned-off-by: shejialuo \n---\n Documentation/fsck-msgids.txt | 3 ++\n fsck.h | 1 +\n refs/files-backend.c | 31 ++++++++++++\n t/t0602-reffiles-fsck.sh | 92 +++++++++++++++++++++++++++++++++++\n 4 files changed, 127 insertions(+)\n create mode 100755 t/t0602-reffiles-fsck.sh\n\ndiff --git a/Documentation/fsck-msgids.txt b/Documentation/fsck-msgids.txt\nindex 7c809fddf1..68a2801f15 100644\n--- a/Documentation/fsck-msgids.txt\n+++ b/Documentation/fsck-msgids.txt\n@@ -22,6 +22,9 @@\n `badRefFiletype`::\n \t(ERROR) A ref has a bad file type.\n \n+`badRefName`::\n+\t(ERROR) A ref has an invalid format.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex af02174973..500b4c04d2 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -32,6 +32,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REF_FILETYPE, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex e511e1dcce..7f6eefa960 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3419,6 +3419,36 @@ typedef int (*files_fsck_refs_fn)(struct ref_store *ref_store,\n \t\t\t\t const char *refs_check_dir,\n \t\t\t\t struct dir_iterator *iter);\n \n+static int files_fsck_refs_name(struct ref_store *ref_store UNUSED,\n+\t\t\t\tstruct fsck_options *o,\n+\t\t\t\tconst char *refs_check_dir,\n+\t\t\t\tstruct dir_iterator *iter)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\t/*\n+\t * Ignore the files ending with \".lock\" as they may be lock files\n+\t * However, do not allow bare \".lock\" files.\n+\t */\n+\tif (iter->basename[0] != '.' && ends_with(iter->basename, \".lock\"))\n+\t\tgoto cleanup;\n+\n+\tif (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {\n+\t\tstruct fsck_ref_report report = { .path = NULL };\n+\n+\t\tstrbuf_addf(&sb, \"%s/%s\", refs_check_dir, iter->relative_path);\n+\t\treport.path = sb.buf;\n+\t\tret = fsck_report_ref(o, &report,\n+\t\t\t\t FSCK_MSG_BAD_REF_NAME,\n+\t\t\t\t \"invalid refname format\");\n+\t}\n+\n+cleanup:\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o,\n \t\t\t const char *refs_check_dir,\n@@ -3470,6 +3500,7 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \t\t\t struct fsck_options *o)\n {\n \tfiles_fsck_refs_fn fsck_refs_fn[]= {\n+\t\tfiles_fsck_refs_name,\n \t\tNULL,\n \t};\n \ndiff --git a/t/t0602-reffiles-fsck.sh b/t/t0602-reffiles-fsck.sh\nnew file mode 100755\nindex 0000000000..71a4d1a5ae\n--- /dev/null\n+++ b/t/t0602-reffiles-fsck.sh\n@@ -0,0 +1,92 @@\n+#!/bin/sh\n+\n+test_description='Test reffiles backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+TEST_PASSES_SANITIZE_LEAK=true\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'ref name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\tcd repo &&\n+\n+\tgit commit --allow-empty -m initial &&\n+\tgit checkout -b branch-1 &&\n+\tgit tag tag-1 &&\n+\tgit commit --allow-empty -m second &&\n+\tgit checkout -b branch-2 &&\n+\tgit tag tag-2 &&\n+\tgit tag multi_hierarchy/tag-2 &&\n+\n+\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\ttest_must_fail git refs verify 2>err &&\n+\tcat >expect <<-EOF &&\n+\terror: refs/heads/.branch-1: badRefName: invalid refname format\n+\tEOF\n+\trm $branch_dir_prefix/.branch-1 &&\n+\ttest_cmp expect err &&\n+\n+\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\ttest_must_fail git refs verify 2>err &&\n+\tcat >expect <<-EOF &&\n+\terror: refs/heads/@: badRefName: invalid refname format\n+\tEOF\n+\trm $branch_dir_prefix/@ &&\n+\ttest_cmp expect err &&\n+\n+\tcp $tag_dir_prefix/multi_hierarchy/tag-2 $tag_dir_prefix/multi_hierarchy/@ &&\n+\ttest_must_fail git refs verify 2>err &&\n+\tcat >expect <<-EOF &&\n+\terror: refs/tags/multi_hierarchy/@: badRefName: invalid refname format\n+\tEOF\n+\trm $tag_dir_prefix/multi_hierarchy/@ &&\n+\ttest_cmp expect err &&\n+\n+\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock &&\n+\tgit refs verify 2>err &&\n+\trm $tag_dir_prefix/tag-1.lock &&\n+\ttest_must_be_empty err &&\n+\n+\tcp $tag_dir_prefix/tag-1 $tag_dir_prefix/.lock &&\n+\ttest_must_fail git refs verify 2>err &&\n+\tcat >expect <<-EOF &&\n+\terror: refs/tags/.lock: badRefName: invalid refname format\n+\tEOF\n+\trm $tag_dir_prefix/.lock &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'ref name check should be adapted into fsck messages' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tbranch_dir_prefix=.git/refs/heads &&\n+\ttag_dir_prefix=.git/refs/tags &&\n+\tcd repo &&\n+\tgit commit --allow-empty -m initial &&\n+\tgit checkout -b branch-1 &&\n+\tgit tag tag-1 &&\n+\tgit commit --allow-empty -m second &&\n+\tgit checkout -b branch-2 &&\n+\tgit tag tag-2 &&\n+\n+\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 &&\n+\tgit -c fsck.badRefName=warn refs verify 2>err &&\n+\tcat >expect <<-EOF &&\n+\twarning: refs/heads/.branch-1: badRefName: invalid refname format\n+\tEOF\n+\trm $branch_dir_prefix/.branch-1 &&\n+\ttest_cmp expect err &&\n+\n+\tcp $branch_dir_prefix/branch-1 $branch_dir_prefix/@ &&\n+\tgit -c fsck.badRefName=ignore refs verify 2>err &&\n+\ttest_must_be_empty err\n+'\n+\n+test_done\n-- \n2.46.0\n\n"},{"id":"500434","messageId":"ZrSyDi9tD1wtZm-F@tanuki","threadId":"61565","inReplyTo":"ZrSqMmD-quQ18a9F@ArchLinux.localdomain","subject":"Re: [GSoC][PATCH v16 0/9] ref consistency check infra setup","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-08T11:54:54Z","receivedAt":"2024-08-08T11:55:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 08, 2024 at 07:21:22PM +0800, shejialuo wrote:\n> Hi All, this version handles some minor changes:\n> \n> 1. Remove redundant newlines.\n> 2. Fix typo in commit message.\n> \n> At last, I wanna thank every reviewer. As we can see, this series starts\n> from 5.27, it's a very long journey. I have learned a lot.\n> \n> Thanks\n> Jialuo\n\nThe range diff looks as expected to me, so this series is ready\nto be merged from my point of view. Thanks!\n\nPatrick\n"},{"id":"500502","messageId":"xmqqa5hnhsxi.fsf@gitster.g","threadId":"61565","inReplyTo":"ZrSyDi9tD1wtZm-F@tanuki","subject":"Re: [GSoC][PATCH v16 0/9] ref consistency check infra setup","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-08T16:37:29Z","receivedAt":"2024-08-08T16:37:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Thu, Aug 08, 2024 at 07:21:22PM +0800, shejialuo wrote:\n>> Hi All, this version handles some minor changes:\n>> \n>> 1. Remove redundant newlines.\n>> 2. Fix typo in commit message.\n>> \n>> At last, I wanna thank every reviewer. As we can see, this series starts\n>> from 5.27, it's a very long journey. I have learned a lot.\n>> \n>> Thanks\n>> Jialuo\n>\n> The range diff looks as expected to me, so this series is ready\n> to be merged from my point of view. Thanks!\n>\n> Patrick\n\nYeah, looking good here, too.\n\nWill replace. Let me mark this topic for 'next'.\n\nThanks.\n"},{"id":"504108","messageId":"20241004191433.64416-1-tenglong.tl@alibaba-inc.com","threadId":"61565","inReplyTo":"ZrSroE8vLlZCK2jp@ArchLinux","subject":"[PATCH 1/2] refs.c: remove redundant translation markers","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2024-10-04T19:14:33Z","receivedAt":"2024-10-04T19:14:49Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"From: Teng Long \n\nshejialuo wrote:\n\n>> --- a/builtin/refs.c\n>> +++ b/builtin/refs.c\n>> @@ -1,4 +1,6 @@\n>> #include \"builtin.h\"\n>> +#include \"config.h\"\n>> +#include \"fsck.h\"\n>> #include \"parse-options.h\"\n>> #include \"refs.h\"\n>> #include \"repository.h\"\n>> @@ -7,6 +9,9 @@\n>> #define REFS_MIGRATE_USAGE \\\n>> \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n>> \n>> +#define REFS_VERIFY_USAGE \\\n>> +\tN_(\"git refs verify [--strict] [--verbose]\")\n\nThis define may not require translation, then maybe it should\nlook like:\n\nSigned-off-by: Teng Long \n---\n builtin/refs.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/refs.c b/builtin/refs.c\nindex 24978a7b..3a799aa6 100644\n--- a/builtin/refs.c\n+++ b/builtin/refs.c\n@@ -10,7 +10,7 @@\n \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n \n #define REFS_VERIFY_USAGE \\\n-\tN_(\"git refs verify [--strict] [--verbose]\")\n+\t\"git refs verify [--strict] [--verbose]\"\n \n static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)\n {\n-- \n2.43.0.rc1.4.g79da3975.dirty\n\n\nThanks.\n"},{"id":"504176","messageId":"ZwJf7A9vnRWQgF41@ArchLinux","threadId":"61565","inReplyTo":"20241004191433.64416-1-tenglong.tl@alibaba-inc.com","subject":"Re: [PATCH 1/2] refs.c: remove redundant translation markers","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2024-10-06T10:01:16Z","receivedAt":"2024-10-06T10:01:11Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Sat, Oct 05, 2024 at 03:14:33AM +0800, Teng Long wrote:\n> From: Teng Long \n\n[snip]\n\n> >> @@ -7,6 +9,9 @@\n> >> #define REFS_MIGRATE_USAGE \\\n> >> \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n> >> \n> >> +#define REFS_VERIFY_USAGE \\\n> >> +\tN_(\"git refs verify [--strict] [--verbose]\")\n> \n> This define may not require translation, then maybe it should\n> look like:\n> \n> Signed-off-by: Teng Long \n> ---\n> builtin/refs.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n> \n> diff --git a/builtin/refs.c b/builtin/refs.c\n> index 24978a7b..3a799aa6 100644\n> --- a/builtin/refs.c\n> +++ b/builtin/refs.c\n> @@ -10,7 +10,7 @@\n> \tN_(\"git refs migrate --ref-format= [--dry-run]\")\n> \n> #define REFS_VERIFY_USAGE \\\n> -\tN_(\"git refs verify [--strict] [--verbose]\")\n> +\t\"git refs verify [--strict] [--verbose]\"\n\nI am OK with this change, actually I just followed the way what\nPatrick does for \"git refs migrate\" subcommand. So at first glance, I\nwonder why the \"git refs migrate\" subcommand usage should not be\nchanged.\n\nAfter searching the mailing list for information, I found that Junio has\nexplained why in\n\n\thttps://lore.kernel.org/git/xmqqbk5y3j8a.fsf@gitster.g/\n\nBecause `--ref-format` could be used as the placeholder, it will be much\nmore helpful when translating in a different locale.\n\nThanks,\nJialuo\n"}]}