{"thread":{"id":"63987","subject":"[PATCH 0/5] refs/reftable: add fsck checks","startedAt":"2025-08-19T12:21:10Z","lastAt":"2025-10-07T16:25:36Z","messageCount":64,"participants":["Karthik Nayak","shejialuo","Junio C Hamano","Patrick Steinhardt","Kristoffer Haugsbakk","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"524438","messageId":"20250819-228-reftable-introduce-consistency-checks-v1-0-8b8f6879fa9e@gmail.com","threadId":"63987","inReplyTo":null,"subject":"[PATCH 0/5] refs/reftable: add fsck checks","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-19T12:20:59Z","receivedAt":"2025-08-19T12:21:10Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"This series adds the required infrastructure and also some fsck checks\nfor the reftable backend.\n\nSince the reftable backend is treated as a library within the Git\ncodebase, we don't want to spillover our internal fsck implementation\ninto the library. At the same time, the fsck checks need to access\ninternal structures of the reftable library which aren't exposed outside\nthe library.\n\nSo we solve this by adding a 'reftable/fsck.[ch]' which implements and\nexposes a checker for the reftable library and returns specific errors\nas defined by the library. We then add glue code within\n'refs/reftable-backend.c' to map these errors to errors which Git's fsck\nimplementation would understand. This allows us to separate concerns.\n\nThis series then adds some checks on the stack ('reftable/tables.list')\nlevel of reftable, namely:\n1. The table name is as per the spec\n2. The number of tables are consistent\n3. The tables.list has a newline at the end of file\n4. The table names follow correct index sequences\n\nI also plan to send in follow up series's which will implement further\nchecks and go into deeper layers (tables, block, references).\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/fsck-msgids.adoc | 15 +++-\n Makefile | 1 +\n fsck.h | 154 +++++++++++++++++++++--------------------\n meson.build | 1 +\n refs/reftable-backend.c | 70 +++++++++++++++++--\n reftable/fsck.c | 132 +++++++++++++++++++++++++++++++++++\n reftable/reftable-fsck.h | 44 ++++++++++++\n t/meson.build | 3 +-\n t/t0614-reftable-fsck.sh | 138 ++++++++++++++++++++++++++++++++++++\n 9 files changed, 473 insertions(+), 85 deletions(-)\n\nKarthik Nayak (5):\n fsck: order 'fsck_msg_type' alphabetically\n refs/reftable: add fsck check for checking the table name\n refs/reftable: add fsck check for number of tables\n refs/reftable: add fsck check for trailing newline\n refs/reftable: add fsck check for incorrect update index\n\n\n\nbase-commit: c44beea485f0f2feaf460e2ac87fdd5608d63cf0\nchange-id: 20250714-228-reftable-introduce-consistency-checks-379ded93c544\n\nThanks\n- Karthik\n\n"},{"id":"524439","messageId":"20250819-228-reftable-introduce-consistency-checks-v1-1-8b8f6879fa9e@gmail.com","threadId":"63987","inReplyTo":"20250819-228-reftable-introduce-consistency-checks-v1-0-8b8f6879fa9e@gmail.com","subject":"[PATCH 1/5] fsck: order 'fsck_msg_type' alphabetically","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-19T12:21:00Z","receivedAt":"2025-08-19T12:21:11Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The list of 'fsck_msg_type' seem to be alphabetically ordered, but there\nare a few small misses. Fix this by sorting the sub-sections of the\nlist to maintain alphabetical ordering. Also fix a clang-format issue\nwhere the escaped newlines are not aligned.\n\nWhile here, remove a duplicate instance of 'gitmodulesLarge' in the\n'fsck-msgids' documentation.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/fsck-msgids.adoc | 3 -\n fsck.h | 150 ++++++++++++++++++++---------------------\n 2 files changed, 75 insertions(+), 78 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.adoc b/Documentation/fsck-msgids.adoc\nindex 0ba4f9a27e..1c912615f9 100644\n--- a/Documentation/fsck-msgids.adoc\n+++ b/Documentation/fsck-msgids.adoc\n@@ -104,9 +104,6 @@\n `gitmodulesParse`::\n \t(INFO) Could not parse `.gitmodules` blob.\n \n-`gitmodulesLarge`;\n-\t(ERROR) `.gitmodules` blob is too large to parse.\n-\n `gitmodulesPath`::\n \t(ERROR) `.gitmodules` path is invalid.\n \ndiff --git a/fsck.h b/fsck.h\nindex dd7df3d5b3..559ad57807 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -20,82 +20,82 @@ enum fsck_msg_type {\n * two in sync.\n */\n \n-#define FOREACH_FSCK_MSG_ID(FUNC) \\\n-\t/* fatal errors */ \\\n-\tFUNC(NUL_IN_HEADER, FATAL) \\\n-\tFUNC(UNTERMINATED_HEADER, FATAL) \\\n-\t/* errors */ \\\n-\tFUNC(BAD_DATE, ERROR) \\\n-\tFUNC(BAD_DATE_OVERFLOW, ERROR) \\\n-\tFUNC(BAD_EMAIL, ERROR) \\\n-\tFUNC(BAD_NAME, ERROR) \\\n-\tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n-\tFUNC(BAD_PACKED_REF_ENTRY, ERROR) \\\n-\tFUNC(BAD_PACKED_REF_HEADER, ERROR) \\\n-\tFUNC(BAD_PARENT_SHA1, ERROR) \\\n-\tFUNC(BAD_REF_CONTENT, ERROR) \\\n-\tFUNC(BAD_REF_FILETYPE, ERROR) \\\n-\tFUNC(BAD_REF_NAME, ERROR) \\\n-\tFUNC(BAD_REFERENT_NAME, ERROR) \\\n-\tFUNC(BAD_TIMEZONE, ERROR) \\\n-\tFUNC(BAD_TREE, ERROR) \\\n-\tFUNC(BAD_TREE_SHA1, ERROR) \\\n-\tFUNC(BAD_TYPE, ERROR) \\\n-\tFUNC(DUPLICATE_ENTRIES, ERROR) \\\n-\tFUNC(MISSING_AUTHOR, ERROR) \\\n-\tFUNC(MISSING_COMMITTER, ERROR) \\\n-\tFUNC(MISSING_EMAIL, ERROR) \\\n-\tFUNC(MISSING_NAME_BEFORE_EMAIL, ERROR) \\\n-\tFUNC(MISSING_OBJECT, ERROR) \\\n-\tFUNC(MISSING_SPACE_BEFORE_DATE, ERROR) \\\n-\tFUNC(MISSING_SPACE_BEFORE_EMAIL, ERROR) \\\n-\tFUNC(MISSING_TAG, ERROR) \\\n-\tFUNC(MISSING_TAG_ENTRY, ERROR) \\\n-\tFUNC(MISSING_TREE, ERROR) \\\n-\tFUNC(MISSING_TYPE, ERROR) \\\n-\tFUNC(MISSING_TYPE_ENTRY, ERROR) \\\n-\tFUNC(MULTIPLE_AUTHORS, ERROR) \\\n-\tFUNC(PACKED_REF_ENTRY_NOT_TERMINATED, ERROR) \\\n-\tFUNC(PACKED_REF_UNSORTED, ERROR) \\\n-\tFUNC(TREE_NOT_SORTED, ERROR) \\\n-\tFUNC(UNKNOWN_TYPE, ERROR) \\\n-\tFUNC(ZERO_PADDED_DATE, ERROR) \\\n-\tFUNC(GITMODULES_MISSING, ERROR) \\\n-\tFUNC(GITMODULES_BLOB, ERROR) \\\n-\tFUNC(GITMODULES_LARGE, ERROR) \\\n-\tFUNC(GITMODULES_NAME, ERROR) \\\n-\tFUNC(GITMODULES_SYMLINK, ERROR) \\\n-\tFUNC(GITMODULES_URL, ERROR) \\\n-\tFUNC(GITMODULES_PATH, ERROR) \\\n-\tFUNC(GITMODULES_UPDATE, ERROR) \\\n-\tFUNC(GITATTRIBUTES_MISSING, ERROR) \\\n-\tFUNC(GITATTRIBUTES_LARGE, ERROR) \\\n-\tFUNC(GITATTRIBUTES_LINE_LENGTH, ERROR) \\\n-\tFUNC(GITATTRIBUTES_BLOB, ERROR) \\\n-\t/* warnings */ \\\n-\tFUNC(EMPTY_NAME, WARN) \\\n-\tFUNC(FULL_PATHNAME, WARN) \\\n-\tFUNC(HAS_DOT, WARN) \\\n-\tFUNC(HAS_DOTDOT, WARN) \\\n-\tFUNC(HAS_DOTGIT, WARN) \\\n-\tFUNC(NULL_SHA1, WARN) \\\n-\tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n-\tFUNC(NUL_IN_COMMIT, WARN) \\\n-\tFUNC(LARGE_PATHNAME, WARN) \\\n+#define FOREACH_FSCK_MSG_ID(FUNC) \\\n+\t/* fatal errors */ \\\n+\tFUNC(NUL_IN_HEADER, FATAL) \\\n+\tFUNC(UNTERMINATED_HEADER, FATAL) \\\n+\t/* errors */ \\\n+\tFUNC(BAD_DATE, ERROR) \\\n+\tFUNC(BAD_DATE_OVERFLOW, ERROR) \\\n+\tFUNC(BAD_EMAIL, ERROR) \\\n+\tFUNC(BAD_NAME, ERROR) \\\n+\tFUNC(BAD_OBJECT_SHA1, ERROR) \\\n+\tFUNC(BAD_PACKED_REF_ENTRY, ERROR) \\\n+\tFUNC(BAD_PACKED_REF_HEADER, ERROR) \\\n+\tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REFERENT_NAME, ERROR) \\\n+\tFUNC(BAD_REF_CONTENT, ERROR) \\\n+\tFUNC(BAD_REF_FILETYPE, ERROR) \\\n+\tFUNC(BAD_REF_NAME, ERROR) \\\n+\tFUNC(BAD_TIMEZONE, ERROR) \\\n+\tFUNC(BAD_TREE, ERROR) \\\n+\tFUNC(BAD_TREE_SHA1, ERROR) \\\n+\tFUNC(BAD_TYPE, ERROR) \\\n+\tFUNC(DUPLICATE_ENTRIES, ERROR) \\\n+\tFUNC(GITATTRIBUTES_BLOB, ERROR) \\\n+\tFUNC(GITATTRIBUTES_LARGE, ERROR) \\\n+\tFUNC(GITATTRIBUTES_LINE_LENGTH, ERROR) \\\n+\tFUNC(GITATTRIBUTES_MISSING, ERROR) \\\n+\tFUNC(GITMODULES_BLOB, ERROR) \\\n+\tFUNC(GITMODULES_LARGE, ERROR) \\\n+\tFUNC(GITMODULES_MISSING, ERROR) \\\n+\tFUNC(GITMODULES_NAME, ERROR) \\\n+\tFUNC(GITMODULES_PATH, ERROR) \\\n+\tFUNC(GITMODULES_SYMLINK, ERROR) \\\n+\tFUNC(GITMODULES_UPDATE, ERROR) \\\n+\tFUNC(GITMODULES_URL, ERROR) \\\n+\tFUNC(MISSING_AUTHOR, ERROR) \\\n+\tFUNC(MISSING_COMMITTER, ERROR) \\\n+\tFUNC(MISSING_EMAIL, ERROR) \\\n+\tFUNC(MISSING_NAME_BEFORE_EMAIL, ERROR) \\\n+\tFUNC(MISSING_OBJECT, ERROR) \\\n+\tFUNC(MISSING_SPACE_BEFORE_DATE, ERROR) \\\n+\tFUNC(MISSING_SPACE_BEFORE_EMAIL, ERROR) \\\n+\tFUNC(MISSING_TAG, ERROR) \\\n+\tFUNC(MISSING_TAG_ENTRY, ERROR) \\\n+\tFUNC(MISSING_TREE, ERROR) \\\n+\tFUNC(MISSING_TYPE, ERROR) \\\n+\tFUNC(MISSING_TYPE_ENTRY, ERROR) \\\n+\tFUNC(MULTIPLE_AUTHORS, ERROR) \\\n+\tFUNC(PACKED_REF_ENTRY_NOT_TERMINATED, ERROR) \\\n+\tFUNC(PACKED_REF_UNSORTED, ERROR) \\\n+\tFUNC(TREE_NOT_SORTED, ERROR) \\\n+\tFUNC(UNKNOWN_TYPE, ERROR) \\\n+\tFUNC(ZERO_PADDED_DATE, ERROR) \\\n+\t/* warnings */ \\\n+\tFUNC(EMPTY_NAME, WARN) \\\n+\tFUNC(FULL_PATHNAME, WARN) \\\n+\tFUNC(HAS_DOT, WARN) \\\n+\tFUNC(HAS_DOTDOT, WARN) \\\n+\tFUNC(HAS_DOTGIT, WARN) \\\n+\tFUNC(LARGE_PATHNAME, WARN) \\\n+\tFUNC(NULL_SHA1, WARN) \\\n+\tFUNC(NUL_IN_COMMIT, WARN) \\\n+\tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \t/* infos (reported as warnings, but ignored by default) */ \\\n-\tFUNC(BAD_FILEMODE, INFO) \\\n-\tFUNC(EMPTY_PACKED_REFS_FILE, INFO) \\\n-\tFUNC(GITMODULES_PARSE, INFO) \\\n-\tFUNC(GITIGNORE_SYMLINK, INFO) \\\n-\tFUNC(GITATTRIBUTES_SYMLINK, INFO) \\\n-\tFUNC(MAILMAP_SYMLINK, INFO) \\\n-\tFUNC(BAD_TAG_NAME, INFO) \\\n-\tFUNC(MISSING_TAGGER_ENTRY, INFO) \\\n-\tFUNC(SYMLINK_REF, INFO) \\\n-\tFUNC(REF_MISSING_NEWLINE, INFO) \\\n-\tFUNC(SYMREF_TARGET_IS_NOT_A_REF, INFO) \\\n-\tFUNC(TRAILING_REF_CONTENT, INFO) \\\n-\t/* ignored (elevated when requested) */ \\\n+\tFUNC(BAD_FILEMODE, INFO) \\\n+\tFUNC(BAD_TAG_NAME, INFO) \\\n+\tFUNC(EMPTY_PACKED_REFS_FILE, INFO) \\\n+\tFUNC(GITATTRIBUTES_SYMLINK, INFO) \\\n+\tFUNC(GITIGNORE_SYMLINK, INFO) \\\n+\tFUNC(GITMODULES_PARSE, INFO) \\\n+\tFUNC(MAILMAP_SYMLINK, INFO) \\\n+\tFUNC(MISSING_TAGGER_ENTRY, INFO) \\\n+\tFUNC(REF_MISSING_NEWLINE, INFO) \\\n+\tFUNC(SYMLINK_REF, INFO) \\\n+\tFUNC(SYMREF_TARGET_IS_NOT_A_REF, INFO) \\\n+\tFUNC(TRAILING_REF_CONTENT, INFO) \\\n+\t/* ignored (elevated when requested) */ \\\n \tFUNC(EXTRA_HEADER_ENTRY, IGNORE)\n \n #define MSG_ID(id, msg_type) FSCK_MSG_##id,\n\n-- \n2.50.1\n\n"},{"id":"524440","messageId":"20250819-228-reftable-introduce-consistency-checks-v1-2-8b8f6879fa9e@gmail.com","threadId":"63987","inReplyTo":"20250819-228-reftable-introduce-consistency-checks-v1-0-8b8f6879fa9e@gmail.com","subject":"[PATCH 2/5] refs/reftable: add fsck check for checking the table name","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-19T12:21:01Z","receivedAt":"2025-08-19T12:21:12Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git refs verify` command is used to run fsck checks on the\nreference backends. This command is also invoked when users run 'git\nfsck'. While the files-backend has some fsck checks added, the reftable\nbackend lacks such checks. Let's add the required infrastructure and a\ncheck to test for the table names in the 'tables.list' of reftables.\n\nFor the infrastructure, since the reftable library is treated as an\nindependent library we should ensure that the library code works\nindependently without knowledge about Git's internals. To do this,\nadd both 'reftable/fsck.c' and 'reftable/reftable-fsck.h'. Which\nprovide an entry point 'reftable_fsck_check' for running fsck checks\nover a provided reftable stack. The callee provides the function with\ncallbacks to handle issue and information reporting.\n\nAdd glue code in 'refs/reftable-backend.c' which calls the reftable\nlibrary to perform the fsck checks. Here we also map the reftable errors\nto Git' fsck errors.\n\nIntroduce a check to validate table names for a given reftable stack.\nAlso add 'badReftableTableName' as a corresponding error within Git. Add\na test to check for this behavior.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/fsck-msgids.adoc | 3 +++\n Makefile | 1 +\n fsck.h | 1 +\n meson.build | 1 +\n refs/reftable-backend.c | 61 +++++++++++++++++++++++++++++++++++++-----\n reftable/fsck.c | 50 ++++++++++++++++++++++++++++++++++\n reftable/reftable-fsck.h | 38 ++++++++++++++++++++++++++\n t/meson.build | 3 ++-\n t/t0614-reftable-fsck.sh | 35 ++++++++++++++++++++++++\n 9 files changed, 186 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.adoc b/Documentation/fsck-msgids.adoc\nindex 1c912615f9..784ddc0df5 100644\n--- a/Documentation/fsck-msgids.adoc\n+++ b/Documentation/fsck-msgids.adoc\n@@ -38,6 +38,9 @@\n `badReferentName`::\n \t(ERROR) The referent name of a symref is invalid.\n \n+`badReftableTableName`::\n+\t(ERROR) A reftable table has an invalid name.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/Makefile b/Makefile\nindex e11340c1ae..f2ddcc8d7c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2733,6 +2733,7 @@ REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n+REFTABLE_OBJS += reftable/fsck.o\n REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/record.o\ndiff --git a/fsck.h b/fsck.h\nindex 559ad57807..5901f944a1 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -34,6 +34,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_PACKED_REF_HEADER, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REFERENT_NAME, ERROR) \\\n+\tFUNC(BAD_REFTABLE_TABLE_NAME, ERROR) \\\n \tFUNC(BAD_REF_CONTENT, ERROR) \\\n \tFUNC(BAD_REF_FILETYPE, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\ndiff --git a/meson.build b/meson.build\nindex 5dd299b496..82879fbfaa 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -452,6 +452,7 @@ libgit_sources = [\n 'reftable/error.c',\n 'reftable/block.c',\n 'reftable/blocksource.c',\n+ 'reftable/fsck.c',\n 'reftable/iter.c',\n 'reftable/merged.c',\n 'reftable/pq.c',\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 8dae1e1112..ccd12052f2 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -6,20 +6,21 @@\n #include \"../config.h\"\n #include \"../dir.h\"\n #include \"../environment.h\"\n+#include \"../fsck.h\"\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n #include \"../iterator.h\"\n #include \"../ident.h\"\n-#include \"../lockfile.h\"\n #include \"../object.h\"\n #include \"../path.h\"\n #include \"../refs.h\"\n #include \"../reftable/reftable-basics.h\"\n-#include \"../reftable/reftable-stack.h\"\n-#include \"../reftable/reftable-record.h\"\n #include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-fsck.h\"\n #include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-stack.h\"\n #include \"../repo-settings.h\"\n #include \"../setup.h\"\n #include \"../strmap.h\"\n@@ -2675,11 +2676,59 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n-static int reftable_be_fsck(struct ref_store *ref_store UNUSED,\n-\t\t\t struct fsck_options *o UNUSED,\n+static void reftable_fsck_verbose_handler(const char *msg, void *cb_data)\n+{\n+\tstruct fsck_options *o = cb_data;\n+\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, \"%s\", _(msg));\n+}\n+\n+static int reftable_fsck_error_handler(struct reftable_fsck_info info,\n+\t\t\t\t void *cb_data)\n+{\n+\tstruct fsck_options *o = cb_data;\n+\tstruct fsck_ref_report report = { .path = info.path };\n+\tenum fsck_msg_id msg_id;\n+\n+\tswitch (info.error) {\n+\tcase REFTABLE_FSCK_ERROR_TABLE_NAME:\n+\t\tmsg_id = FSCK_MSG_BAD_REFTABLE_TABLE_NAME;\n+\t\tbreak;\n+\tdefault:\n+\t\tBUG(\"unknown fsck error: %d\", info.error);\n+\t}\n+\n+\treturn fsck_report_ref(o, &report, msg_id, \"%s\", info.msg);\n+}\n+\n+static int reftable_be_fsck(struct ref_store *ref_store, struct fsck_options *o,\n \t\t\t struct worktree *wt UNUSED)\n {\n-\treturn 0;\n+\tstruct reftable_ref_store *refs;\n+\tstruct strmap_entry *entry;\n+\tstruct hashmap_iter iter;\n+\tint ret = 0;\n+\n+\trefs = reftable_be_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, _(\"Checking references consistency\"));\n+\n+\tret = reftable_fsck_check(refs->main_backend.stack, reftable_fsck_error_handler,\n+\t\t\t\t reftable_fsck_verbose_handler, o);\n+\tif (!ret)\n+\t\treturn ret;\n+\n+\tstrmap_for_each_entry(&refs->worktree_backends, &iter, entry) {\n+\t\tstruct reftable_backend *b = (struct reftable_backend *)entry->value;\n+\t\tret = reftable_fsck_check(b->stack, reftable_fsck_error_handler,\n+\t\t\t\t\t reftable_fsck_verbose_handler, o);\n+\t\tif (!ret)\n+\t\t\treturn ret;\n+\t}\n+\n+\treturn ret;\n }\n \n struct ref_storage_be refs_be_reftable = {\ndiff --git a/reftable/fsck.c b/reftable/fsck.c\nnew file mode 100644\nindex 0000000000..22ec3c26e9\n--- /dev/null\n+++ b/reftable/fsck.c\n@@ -0,0 +1,50 @@\n+#include \"basics.h\"\n+#include \"reftable-fsck.h\"\n+#include \"stack.h\"\n+\n+int reftable_fsck_check(struct reftable_stack *stack,\n+\t\t\treftable_fsck_report_fn report_fn,\n+\t\t\treftable_fsck_verbose_fn verbose_fn,\n+\t\t\tvoid *cb_data)\n+{\n+\tchar **names = NULL;\n+\tuint64_t min, max;\n+\tint err = 0;\n+\n+\tif (stack == NULL)\n+\t\tgoto out;\n+\n+\terr = read_lines(stack->list_file, &names);\n+\tif (err < 0)\n+\t\tgoto out;\n+\n+\tverbose_fn(\"Checking reftable table names\", cb_data);\n+\n+\tfor (size_t i = 0; names[i]; i++) {\n+\t\tstruct reftable_fsck_info info = {\n+\t\t\t.error = REFTABLE_FSCK_ERROR_TABLE_NAME,\n+\t\t\t.path = names[i],\n+\t\t\t.msg = \"invalid reftable name\"\n+\t\t};\n+\t\tuint32_t rnd;\n+\t\t/*\n+\t\t * We want to match the tail '.ref'. One extra byte to ensure\n+\t\t * that there is no unexpected extra character and one byte for\n+\t\t * the null terminator added by sscanf.\n+\t\t */\n+\t\tchar tail[6];\n+\n+\t\tif (sscanf(names[i], \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x%5s\",\n+\t\t\t &min, &max, &rnd, tail) != 4) {\n+\t\t\terr = report_fn(info, cb_data);\n+\t\t}\n+\n+\t\tif (strcmp(tail, \".ref\")) {\n+\t\t\terr = report_fn(info, cb_data);\n+\t\t}\n+\t}\n+\n+out:\n+\tfree_names(names);\n+\treturn err;\n+}\ndiff --git a/reftable/reftable-fsck.h b/reftable/reftable-fsck.h\nnew file mode 100644\nindex 0000000000..087430d979\n--- /dev/null\n+++ b/reftable/reftable-fsck.h\n@@ -0,0 +1,38 @@\n+#ifndef REFTABLE_FSCK_H\n+#define REFTABLE_FSCK_H\n+\n+#include \"reftable-stack.h\"\n+\n+enum reftable_fsck_error {\n+\t/* Invalid table name */\n+\tREFTABLE_FSCK_ERROR_TABLE_NAME = -1,\n+};\n+\n+/* Represents an individual error encounctered during the FSCK checks. */\n+struct reftable_fsck_info {\n+\tenum reftable_fsck_error error;\n+\tconst char *msg;\n+\tconst char *path;\n+};\n+\n+typedef int reftable_fsck_report_fn(struct reftable_fsck_info info,\n+\t\t\t\t void *cb_data);\n+typedef void reftable_fsck_verbose_fn(const char *msg, void *cb_data);\n+\n+/*\n+ * Given a reftable stack, perform FSCK check on the stack.\n+ *\n+ * If an issue is encountered, the issue is reported to the callee via the\n+ * provided 'report_fn'. If the issue is non-recoverable the flow will not\n+ * conitnue. If it is recoverable, the flow will continue and further issues\n+ * will be reported as identified.\n+ *\n+ * The 'verbose_fn' will be invoked to provide verbose information about\n+ * the progress and state of the FSCK checks.\n+ */\n+int reftable_fsck_check(struct reftable_stack *stack,\n+\t\t\treftable_fsck_report_fn report_fn,\n+\t\t\treftable_fsck_verbose_fn verbose_fn,\n+\t\t\tvoid *cb_data);\n+\n+#endif /* REFTABLE_FSCK_H */\ndiff --git a/t/meson.build b/t/meson.build\nindex bbeba1a8d5..a8eb44eb30 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -145,6 +145,7 @@ integration_tests = [\n 't0611-reftable-httpd.sh',\n 't0612-reftable-jgit-compatibility.sh',\n 't0613-reftable-write-options.sh',\n+ 't0614-reftable-fsck.sh',\n 't1000-read-tree-m-3way.sh',\n 't1001-read-tree-m-2way.sh',\n 't1002-read-tree-m-u-2way.sh',\n@@ -1214,4 +1215,4 @@ if perl.found() and time.found()\n timeout: 0,\n )\n endforeach\n-endif\n\\ No newline at end of file\n+endif\ndiff --git a/t/t0614-reftable-fsck.sh b/t/t0614-reftable-fsck.sh\nnew file mode 100755\nindex 0000000000..0d11871b1c\n--- /dev/null\n+++ b/t/t0614-reftable-fsck.sh\n@@ -0,0 +1,35 @@\n+#!/bin/sh\n+\n+test_description='Test reftable backend consistency check'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'table name should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\n+\t\tgit refs verify 2>err &&\n+\t\ttest_must_be_empty err &&\n+\n+\t\tTABLE_NAME=$(cat .git/reftable/tables.list | head -n1) &&\n+\t\tsed \"1s/$/extra/\" .git/reftable/tables.list >.git/reftable/tables.list.tmp &&\n+\t\tmv .git/reftable/tables.list.tmp .git/reftable/tables.list &&\n+\t\tmv .git/reftable/${TABLE_NAME} .git/reftable/${TABLE_NAME}extra &&\n+\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: ${TABLE_NAME}extra: badReftableTableName: invalid reftable name\n+\t\tEOF\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_done\n\n-- \n2.50.1\n\n"},{"id":"524441","messageId":"20250819-228-reftable-introduce-consistency-checks-v1-3-8b8f6879fa9e@gmail.com","threadId":"63987","inReplyTo":"20250819-228-reftable-introduce-consistency-checks-v1-0-8b8f6879fa9e@gmail.com","subject":"[PATCH 3/5] refs/reftable: add fsck check for number of tables","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-19T12:21:02Z","receivedAt":"2025-08-19T12:21:12Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Introduce a reftable fsck check to check that the number of files in the\nreftable directory matches the number of files listed in 'tables.list'.\nWe do this by iterating over the files in the reftable directory and\ncounting all the files present excluding the 'tables.list'. This is also\nexposed over Git's fsck checks as a 'badReftableStackCount' error.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/fsck-msgids.adoc | 3 +++\n fsck.h | 1 +\n refs/reftable-backend.c | 3 +++\n reftable/fsck.c | 34 ++++++++++++++++++++++++++++++++++\n reftable/reftable-fsck.h | 2 ++\n t/t0614-reftable-fsck.sh | 20 ++++++++++++++++++++\n 6 files changed, 63 insertions(+)\n\ndiff --git a/Documentation/fsck-msgids.adoc b/Documentation/fsck-msgids.adoc\nindex 784ddc0df5..707e2fc50a 100644\n--- a/Documentation/fsck-msgids.adoc\n+++ b/Documentation/fsck-msgids.adoc\n@@ -38,6 +38,9 @@\n `badReferentName`::\n \t(ERROR) The referent name of a symref is invalid.\n \n+`badReftableStackCount`::\n+\t(ERROR) Mismatch in number of tables.\n+\n `badReftableTableName`::\n \t(ERROR) A reftable table has an invalid name.\n \ndiff --git a/fsck.h b/fsck.h\nindex 5901f944a1..256effc4f8 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -34,6 +34,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_PACKED_REF_HEADER, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REFERENT_NAME, ERROR) \\\n+\tFUNC(BAD_REFTABLE_STACK_COUNT, ERROR) \\\n \tFUNC(BAD_REFTABLE_TABLE_NAME, ERROR) \\\n \tFUNC(BAD_REF_CONTENT, ERROR) \\\n \tFUNC(BAD_REF_FILETYPE, ERROR) \\\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex ccd12052f2..616f4ee0f3 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2695,6 +2695,9 @@ static int reftable_fsck_error_handler(struct reftable_fsck_info info,\n \tcase REFTABLE_FSCK_ERROR_TABLE_NAME:\n \t\tmsg_id = FSCK_MSG_BAD_REFTABLE_TABLE_NAME;\n \t\tbreak;\n+\tcase REFTABLE_FSCK_ERROR_STACK_COUNT:\n+\t\tmsg_id = FSCK_MSG_BAD_REFTABLE_STACK_COUNT;\n+\t\tbreak;\n \tdefault:\n \t\tBUG(\"unknown fsck error: %d\", info.error);\n \t}\ndiff --git a/reftable/fsck.c b/reftable/fsck.c\nindex 22ec3c26e9..e92a630276 100644\n--- a/reftable/fsck.c\n+++ b/reftable/fsck.c\n@@ -2,6 +2,28 @@\n #include \"reftable-fsck.h\"\n #include \"stack.h\"\n \n+static int reftable_fsck_valid_stack_count(struct reftable_stack *st)\n+{\n+\tDIR *dir = opendir(st->reftable_dir);\n+\tstruct dirent *d = NULL;\n+\tunsigned int count = 0;\n+\n+\tif (!dir)\n+\t\treturn 0;\n+\n+\twhile ((d = readdir(dir))) {\n+\t\tif (!strcmp(d->d_name, \"tables.list\"))\n+\t\t\tcontinue;\n+\n+\t\tif (d->d_type == DT_REG)\n+\t\t\tcount++;\n+\t}\n+\n+\tclosedir(dir);\n+\n+\treturn count == st->tables_len;\n+}\n+\n int reftable_fsck_check(struct reftable_stack *stack,\n \t\t\treftable_fsck_report_fn report_fn,\n \t\t\treftable_fsck_verbose_fn verbose_fn,\n@@ -44,6 +66,18 @@ int reftable_fsck_check(struct reftable_stack *stack,\n \t\t}\n \t}\n \n+\tverbose_fn(\"Checking reftable tables count\", cb_data);\n+\n+\tif (!reftable_fsck_valid_stack_count(stack)) {\n+\t\tstruct reftable_fsck_info info = {\n+\t\t\t.error = REFTABLE_FSCK_ERROR_STACK_COUNT,\n+\t\t\t.path = stack->list_file,\n+\t\t\t.msg = \"mismatch in number of tables\"\n+\t\t};\n+\n+\t\terr = report_fn(info, cb_data);\n+\t}\n+\n out:\n \tfree_names(names);\n \treturn err;\ndiff --git a/reftable/reftable-fsck.h b/reftable/reftable-fsck.h\nindex 087430d979..888c3968b7 100644\n--- a/reftable/reftable-fsck.h\n+++ b/reftable/reftable-fsck.h\n@@ -6,6 +6,8 @@\n enum reftable_fsck_error {\n \t/* Invalid table name */\n \tREFTABLE_FSCK_ERROR_TABLE_NAME = -1,\n+\t/* Incorrect number of tables present */\n+\tREFTABLE_FSCK_ERROR_STACK_COUNT = -2,\n };\n \n /* Represents an individual error encounctered during the FSCK checks. */\ndiff --git a/t/t0614-reftable-fsck.sh b/t/t0614-reftable-fsck.sh\nindex 0d11871b1c..a351fed562 100755\n--- a/t/t0614-reftable-fsck.sh\n+++ b/t/t0614-reftable-fsck.sh\n@@ -32,4 +32,24 @@ test_expect_success 'table name should be checked' '\n \t)\n '\n \n+test_expect_success 'table count should be checked' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\n+\t\tgit refs verify 2>err &&\n+\t\ttest_must_be_empty err &&\n+\n+\t\ttouch .git/reftable/0x000000002812-0x000000002813-c830a596.ref &&\n+\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: $(pwd)/.git/reftable/tables.list: badReftableStackCount: mismatch in number of tables\n+\t\tEOF\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n\n-- \n2.50.1\n\n"},{"id":"524442","messageId":"20250819-228-reftable-introduce-consistency-checks-v1-4-8b8f6879fa9e@gmail.com","threadId":"63987","inReplyTo":"20250819-228-reftable-introduce-consistency-checks-v1-0-8b8f6879fa9e@gmail.com","subject":"[PATCH 4/5] refs/reftable: add fsck check for trailing newline","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-19T12:21:03Z","receivedAt":"2025-08-19T12:21:13Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Introduce a fsck check for the reftable backend, which checks if the\n'tables.list' contains a newline. The reftable backend writes a trailing\nnewline when writing the 'tables.list', but it doesn't check for it when\nreading the file. A missing newline however indicates that the file was\nmanually tampered with, so let's raise this as an error to the user.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/fsck-msgids.adoc | 3 +++\n fsck.h | 1 +\n refs/reftable-backend.c | 3 +++\n reftable/fsck.c | 36 ++++++++++++++++++++++++++++++++++++\n reftable/reftable-fsck.h | 2 ++\n t/t0614-reftable-fsck.sh | 21 +++++++++++++++++++++\n 6 files changed, 66 insertions(+)\n\ndiff --git a/Documentation/fsck-msgids.adoc b/Documentation/fsck-msgids.adoc\nindex 707e2fc50a..1432b1de06 100644\n--- a/Documentation/fsck-msgids.adoc\n+++ b/Documentation/fsck-msgids.adoc\n@@ -41,6 +41,9 @@\n `badReftableStackCount`::\n \t(ERROR) Mismatch in number of tables.\n \n+`badReftableStackListNewline`::\n+\t(ERROR) Reftable stack list missing trailing newline.\n+\n `badReftableTableName`::\n \t(ERROR) A reftable table has an invalid name.\n \ndiff --git a/fsck.h b/fsck.h\nindex 256effc4f8..33432bae79 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -35,6 +35,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REFERENT_NAME, ERROR) \\\n \tFUNC(BAD_REFTABLE_STACK_COUNT, ERROR) \\\n+\tFUNC(BAD_REFTABLE_STACK_LIST_NEWLINE, ERROR) \\\n \tFUNC(BAD_REFTABLE_TABLE_NAME, ERROR) \\\n \tFUNC(BAD_REF_CONTENT, ERROR) \\\n \tFUNC(BAD_REF_FILETYPE, ERROR) \\\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 616f4ee0f3..0087afa3ac 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2698,6 +2698,9 @@ static int reftable_fsck_error_handler(struct reftable_fsck_info info,\n \tcase REFTABLE_FSCK_ERROR_STACK_COUNT:\n \t\tmsg_id = FSCK_MSG_BAD_REFTABLE_STACK_COUNT;\n \t\tbreak;\n+\tcase REFTABLE_FSCK_ERROR_STACK_LIST_MISSING_NEWLINE:\n+\t\tmsg_id = FSCK_MSG_BAD_REFTABLE_STACK_LIST_NEWLINE;\n+\t\tbreak;\n \tdefault:\n \t\tBUG(\"unknown fsck error: %d\", info.error);\n \t}\ndiff --git a/reftable/fsck.c b/reftable/fsck.c\nindex e92a630276..b4898fd2cd 100644\n--- a/reftable/fsck.c\n+++ b/reftable/fsck.c\n@@ -1,7 +1,31 @@\n #include \"basics.h\"\n+#include \"reftable-error.h\"\n #include \"reftable-fsck.h\"\n #include \"stack.h\"\n \n+static int reftable_fsck_stack_contains_newline(const char *list_file)\n+{\n+\tFILE *f = fopen(list_file, \"r\");\n+\tint c = 0;\n+\n+\tif (f == NULL) {\n+\t\tif (errno == ENOENT)\n+\t\t\treturn 0;\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\n+\tif (fseek(f, 0, SEEK_END) == 0) {\n+\t\tlong size = ftell(f);\n+\t\tif (size <= 0)\n+\t\t\treturn REFTABLE_IO_ERROR;\n+\t\tfseek(f, -1, SEEK_END);\n+\t\tc = fgetc(f);\n+\t}\n+\tfclose(f);\n+\n+\treturn c == '\\n';\n+}\n+\n static int reftable_fsck_valid_stack_count(struct reftable_stack *st)\n {\n \tDIR *dir = opendir(st->reftable_dir);\n@@ -66,6 +90,18 @@ int reftable_fsck_check(struct reftable_stack *stack,\n \t\t}\n \t}\n \n+\tverbose_fn(\"Checking trailing newline in stack list\", cb_data);\n+\n+\tif (!reftable_fsck_stack_contains_newline(stack->list_file)) {\n+\t\tstruct reftable_fsck_info info = {\n+\t\t\t.error = REFTABLE_FSCK_ERROR_STACK_LIST_MISSING_NEWLINE,\n+\t\t\t.path = stack->list_file,\n+\t\t\t.msg = \"trailing newline missing in stack list\"\n+\t\t};\n+\n+\t\terr = report_fn(info, cb_data);\n+\t}\n+\n \tverbose_fn(\"Checking reftable tables count\", cb_data);\n \n \tif (!reftable_fsck_valid_stack_count(stack)) {\ndiff --git a/reftable/reftable-fsck.h b/reftable/reftable-fsck.h\nindex 888c3968b7..8e6cb6c7d2 100644\n--- a/reftable/reftable-fsck.h\n+++ b/reftable/reftable-fsck.h\n@@ -8,6 +8,8 @@ enum reftable_fsck_error {\n \tREFTABLE_FSCK_ERROR_TABLE_NAME = -1,\n \t/* Incorrect number of tables present */\n \tREFTABLE_FSCK_ERROR_STACK_COUNT = -2,\n+\t/* Newline missing at the end of the stack list */\n+\tREFTABLE_FSCK_ERROR_STACK_LIST_MISSING_NEWLINE = -3,\n };\n \n /* Represents an individual error encounctered during the FSCK checks. */\ndiff --git a/t/t0614-reftable-fsck.sh b/t/t0614-reftable-fsck.sh\nindex a351fed562..937c5dd37a 100755\n--- a/t/t0614-reftable-fsck.sh\n+++ b/t/t0614-reftable-fsck.sh\n@@ -52,4 +52,25 @@ test_expect_success 'table count should be checked' '\n \t)\n '\n \n+test_expect_success 'stack list must contain trailing newline' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\n+\t\tgit refs verify 2>err &&\n+\t\ttest_must_be_empty err &&\n+\n+\t\tprintf \"%s\" \"$(cat .git/reftable/tables.list)\" >.git/reftable/tables.list.tmp &&\n+\t\tmv .git/reftable/tables.list.tmp .git/reftable/tables.list &&\n+\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: $(pwd)/.git/reftable/tables.list: badReftableStackListNewline: trailing newline missing in stack list\n+\t\tEOF\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n\n-- \n2.50.1\n\n"},{"id":"524443","messageId":"20250819-228-reftable-introduce-consistency-checks-v1-5-8b8f6879fa9e@gmail.com","threadId":"63987","inReplyTo":"20250819-228-reftable-introduce-consistency-checks-v1-0-8b8f6879fa9e@gmail.com","subject":"[PATCH 5/5] refs/reftable: add fsck check for incorrect update index","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-19T12:21:04Z","receivedAt":"2025-08-19T12:21:14Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Introduce a fsck check for the reftable backend, which checks if the\ntables in 'tables.list' contain sequential update index. The tables in\nthe reftable backend should contain sequential update index. This fsck\ncheck ensures that.\n\nWe must note that the reftable backend itself doesn't check to ensure\nthis and it also doesn't check to ensure that the index in the table\nname matches the index in the header or the table. The latter is not\nimplemented in this fsck check either and will be added in a future\npatch where we add fsck checks for internals of a table.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/fsck-msgids.adoc | 3 ++\n fsck.h | 1 +\n refs/reftable-backend.c | 3 ++\n reftable/fsck.c | 14 +++++++++-\n reftable/reftable-fsck.h | 2 ++\n t/t0614-reftable-fsck.sh | 62 ++++++++++++++++++++++++++++++++++++++++++\n 6 files changed, 84 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/fsck-msgids.adoc b/Documentation/fsck-msgids.adoc\nindex 1432b1de06..982d51876c 100644\n--- a/Documentation/fsck-msgids.adoc\n+++ b/Documentation/fsck-msgids.adoc\n@@ -47,6 +47,9 @@\n `badReftableTableName`::\n \t(ERROR) A reftable table has an invalid name.\n \n+`badReftableUpdateIndex`::\n+\t(ERROR) Incorrect update index found for table.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex 33432bae79..60e9b84183 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -37,6 +37,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_REFTABLE_STACK_COUNT, ERROR) \\\n \tFUNC(BAD_REFTABLE_STACK_LIST_NEWLINE, ERROR) \\\n \tFUNC(BAD_REFTABLE_TABLE_NAME, ERROR) \\\n+\tFUNC(BAD_REFTABLE_UPDATE_INDEX, ERROR) \\\n \tFUNC(BAD_REF_CONTENT, ERROR) \\\n \tFUNC(BAD_REF_FILETYPE, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 0087afa3ac..d5993238db 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2701,6 +2701,9 @@ static int reftable_fsck_error_handler(struct reftable_fsck_info info,\n \tcase REFTABLE_FSCK_ERROR_STACK_LIST_MISSING_NEWLINE:\n \t\tmsg_id = FSCK_MSG_BAD_REFTABLE_STACK_LIST_NEWLINE;\n \t\tbreak;\n+\tcase REFTABLE_FSCK_ERROR_UPDATE_INDEX:\n+\t\tmsg_id = FSCK_MSG_BAD_REFTABLE_UPDATE_INDEX;\n+\t\tbreak;\n \tdefault:\n \t\tBUG(\"unknown fsck error: %d\", info.error);\n \t}\ndiff --git a/reftable/fsck.c b/reftable/fsck.c\nindex b4898fd2cd..a6551b9a3c 100644\n--- a/reftable/fsck.c\n+++ b/reftable/fsck.c\n@@ -53,8 +53,8 @@ int reftable_fsck_check(struct reftable_stack *stack,\n \t\t\treftable_fsck_verbose_fn verbose_fn,\n \t\t\tvoid *cb_data)\n {\n+\tuint64_t min, max, prev_max = 0;\n \tchar **names = NULL;\n-\tuint64_t min, max;\n \tint err = 0;\n \n \tif (stack == NULL)\n@@ -85,9 +85,21 @@ int reftable_fsck_check(struct reftable_stack *stack,\n \t\t\terr = report_fn(info, cb_data);\n \t\t}\n \n+\t\tif (min != (prev_max + 1) || max < min) {\n+\t\t\tstruct reftable_fsck_info info = {\n+\t\t\t\t.error = REFTABLE_FSCK_ERROR_UPDATE_INDEX,\n+\t\t\t\t.path = names[i],\n+\t\t\t\t.msg = \"incorrect update index in table name\"\n+\t\t\t};\n+\n+\t\t\terr = report_fn(info, cb_data);\n+\t\t}\n+\n \t\tif (strcmp(tail, \".ref\")) {\n \t\t\terr = report_fn(info, cb_data);\n \t\t}\n+\n+\t\tprev_max = max;\n \t}\n \n \tverbose_fn(\"Checking trailing newline in stack list\", cb_data);\ndiff --git a/reftable/reftable-fsck.h b/reftable/reftable-fsck.h\nindex 8e6cb6c7d2..49437280bb 100644\n--- a/reftable/reftable-fsck.h\n+++ b/reftable/reftable-fsck.h\n@@ -10,6 +10,8 @@ enum reftable_fsck_error {\n \tREFTABLE_FSCK_ERROR_STACK_COUNT = -2,\n \t/* Newline missing at the end of the stack list */\n \tREFTABLE_FSCK_ERROR_STACK_LIST_MISSING_NEWLINE = -3,\n+\t/* Incorrect update index for table */\n+\tREFTABLE_FSCK_ERROR_UPDATE_INDEX = -4,\n };\n \n /* Represents an individual error encounctered during the FSCK checks. */\ndiff --git a/t/t0614-reftable-fsck.sh b/t/t0614-reftable-fsck.sh\nindex 937c5dd37a..bdcbd65a9f 100755\n--- a/t/t0614-reftable-fsck.sh\n+++ b/t/t0614-reftable-fsck.sh\n@@ -73,4 +73,66 @@ test_expect_success 'stack list must contain trailing newline' '\n \t)\n '\n \n+test_expect_success 'table update index should be sequential between tables' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\n+\t\t# Lock the existing table to disable auto-compaction\n+\t\tCUR_TABLE=$(cat .git/reftable/tables.list | tail -n1) &&\n+\t\ttouch .git/reftable/${CUR_TABLE}.lock &&\n+\t\tgit update-ref refs/heads/sample @ &&\n+\t\trm .git/reftable/${CUR_TABLE}.lock &&\n+\n+\t\tgit refs verify 2>err &&\n+\t\ttest_must_be_empty err &&\n+\n+\t\tTABLE_NAME=$(cat .git/reftable/tables.list | tail -n1) &&\n+\t\tNEW_TABLE_NAME=$(echo ${TABLE_NAME} | sed \"s/0003/0009/g\") &&\n+\n+\t\tsed \"2s/.*/${NEW_TABLE_NAME}/\" .git/reftable/tables.list >.git/reftable/tables.list.tmp &&\n+\t\tmv .git/reftable/tables.list.tmp .git/reftable/tables.list &&\n+\t\tmv .git/reftable/${TABLE_NAME} .git/reftable/${NEW_TABLE_NAME} &&\n+\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: ${NEW_TABLE_NAME}: badReftableUpdateIndex: incorrect update index in table name\n+\t\tEOF\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'table update index should be sequential within a table' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\n+\t\t# Lock the existing table to disable auto-compaction\n+\t\tCUR_TABLE=$(cat .git/reftable/tables.list | tail -n1) &&\n+\t\ttouch .git/reftable/${CUR_TABLE}.lock &&\n+\t\tgit update-ref refs/heads/sample @ &&\n+\t\trm .git/reftable/${CUR_TABLE}.lock &&\n+\n+\t\tgit refs verify 2>err &&\n+\t\ttest_must_be_empty err &&\n+\n+\t\tTABLE_NAME=$(cat .git/reftable/tables.list | tail -n1) &&\n+\t\tNEW_TABLE_NAME=$(echo ${TABLE_NAME} | sed \"s/\\(.*\\)0003/\\10002/\") &&\n+\n+\t\tsed \"2s/.*/${NEW_TABLE_NAME}/\" .git/reftable/tables.list >.git/reftable/tables.list.tmp &&\n+\t\tmv .git/reftable/tables.list.tmp .git/reftable/tables.list &&\n+\t\tmv .git/reftable/${TABLE_NAME} .git/reftable/${NEW_TABLE_NAME} &&\n+\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: ${NEW_TABLE_NAME}: badReftableUpdateIndex: incorrect update index in table name\n+\t\tEOF\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n\n-- \n2.50.1\n\n"},{"id":"524973","messageId":"aK3fHRMFiRBYNiJE@ArchLinux","threadId":"63987","inReplyTo":"20250819-228-reftable-introduce-consistency-checks-v1-2-8b8f6879fa9e@gmail.com","subject":"Re: [PATCH 2/5] refs/reftable: add fsck check for checking the table name","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-08-26T16:21:49Z","receivedAt":"2025-08-26T16:21:53Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Aug 19, 2025 at 02:21:01PM +0200, Karthik Nayak wrote:\n> The `git refs verify` command is used to run fsck checks on the\n> reference backends. This command is also invoked when users run 'git\n> fsck'. While the files-backend has some fsck checks added, the reftable\n> backend lacks such checks. Let's add the required infrastructure and a\n> check to test for the table names in the 'tables.list' of reftables.\n> \n> For the infrastructure, since the reftable library is treated as an\n> independent library we should ensure that the library code works\n> independently without knowledge about Git's internals. To do this,\n> add both 'reftable/fsck.c' and 'reftable/reftable-fsck.h'. Which\n\nA design question here, we name the \"fsck.c\" for the source code but for\nthe header, we use \"reftable-fsck.h\", it is a little strange. Why not\njust \"fsck.h\" instead of \"reftable-fsck.h\".\n\n> provide an entry point 'reftable_fsck_check' for running fsck checks\n> over a provided reftable stack. The callee provides the function with\n> callbacks to handle issue and information reporting.\n> \n> Add glue code in 'refs/reftable-backend.c' which calls the reftable\n> library to perform the fsck checks. Here we also map the reftable errors\n> to Git' fsck errors.\n> \n> Introduce a check to validate table names for a given reftable stack.\n> Also add 'badReftableTableName' as a corresponding error within Git. Add\n> a test to check for this behavior.\n> \n> Signed-off-by: Karthik Nayak \n> ---\n> Documentation/fsck-msgids.adoc | 3 +++\n> Makefile | 1 +\n> fsck.h | 1 +\n> meson.build | 1 +\n> refs/reftable-backend.c | 61 +++++++++++++++++++++++++++++++++++++-----\n> reftable/fsck.c | 50 ++++++++++++++++++++++++++++++++++\n> reftable/reftable-fsck.h | 38 ++++++++++++++++++++++++++\n> t/meson.build | 3 ++-\n> t/t0614-reftable-fsck.sh | 35 ++++++++++++++++++++++++\n> 9 files changed, 186 insertions(+), 7 deletions(-)\n> \n> diff --git a/Documentation/fsck-msgids.adoc b/Documentation/fsck-msgids.adoc\n> index 1c912615f9..784ddc0df5 100644\n> --- a/Documentation/fsck-msgids.adoc\n> +++ b/Documentation/fsck-msgids.adoc\n> @@ -38,6 +38,9 @@\n> `badReferentName`::\n> \t(ERROR) The referent name of a symref is invalid.\n> \n> +`badReftableTableName`::\n> +\t(ERROR) A reftable table has an invalid name.\n> +\n\nWhen reading this, I feel a little strange. `Reftable` already indicates\nit is a table. Should we simply say like the following:\n\n A reftable has an invalid table name\n\n> `badTagName`::\n> \t(INFO) A tag has an invalid format.\n> \n> diff --git a/Makefile b/Makefile\n> index e11340c1ae..f2ddcc8d7c 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -2733,6 +2733,7 @@ REFTABLE_OBJS += reftable/error.o\n> REFTABLE_OBJS += reftable/block.o\n> REFTABLE_OBJS += reftable/blocksource.o\n> REFTABLE_OBJS += reftable/iter.o\n> +REFTABLE_OBJS += reftable/fsck.o\n> REFTABLE_OBJS += reftable/merged.o\n> REFTABLE_OBJS += reftable/pq.o\n> REFTABLE_OBJS += reftable/record.o\n> diff --git a/fsck.h b/fsck.h\n> index 559ad57807..5901f944a1 100644\n> --- a/fsck.h\n> +++ b/fsck.h\n> @@ -34,6 +34,7 @@ enum fsck_msg_type {\n> \tFUNC(BAD_PACKED_REF_HEADER, ERROR) \\\n> \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n> \tFUNC(BAD_REFERENT_NAME, ERROR) \\\n> +\tFUNC(BAD_REFTABLE_TABLE_NAME, ERROR) \\\n> \tFUNC(BAD_REF_CONTENT, ERROR) \\\n> \tFUNC(BAD_REF_FILETYPE, ERROR) \\\n> \tFUNC(BAD_REF_NAME, ERROR) \\\n> diff --git a/meson.build b/meson.build\n> index 5dd299b496..82879fbfaa 100644\n> --- a/meson.build\n> +++ b/meson.build\n> @@ -452,6 +452,7 @@ libgit_sources = [\n> 'reftable/error.c',\n> 'reftable/block.c',\n> 'reftable/blocksource.c',\n> + 'reftable/fsck.c',\n> 'reftable/iter.c',\n> 'reftable/merged.c',\n> 'reftable/pq.c',\n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index 8dae1e1112..ccd12052f2 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -6,20 +6,21 @@\n> #include \"../config.h\"\n> #include \"../dir.h\"\n> #include \"../environment.h\"\n> +#include \"../fsck.h\"\n> #include \"../gettext.h\"\n> #include \"../hash.h\"\n> #include \"../hex.h\"\n> #include \"../iterator.h\"\n> #include \"../ident.h\"\n> -#include \"../lockfile.h\"\n\nHere, we delete this header file. Is the reason that we don't need this\nheader file anymore?\n\n> #include \"../object.h\"\n> #include \"../path.h\"\n> #include \"../refs.h\"\n> #include \"../reftable/reftable-basics.h\"\n> -#include \"../reftable/reftable-stack.h\"\n> -#include \"../reftable/reftable-record.h\"\n> #include \"../reftable/reftable-error.h\"\n> +#include \"../reftable/reftable-fsck.h\"\n> #include \"../reftable/reftable-iterator.h\"\n> +#include \"../reftable/reftable-record.h\"\n> +#include \"../reftable/reftable-stack.h\"\n> #include \"../repo-settings.h\"\n> #include \"../setup.h\"\n> #include \"../strmap.h\"\n> @@ -2675,11 +2676,59 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n> \treturn ret;\n> }\n> \n> -static int reftable_be_fsck(struct ref_store *ref_store UNUSED,\n> -\t\t\t struct fsck_options *o UNUSED,\n> +static void reftable_fsck_verbose_handler(const char *msg, void *cb_data)\n> +{\n> +\tstruct fsck_options *o = cb_data;\n> +\n> +\tif (o->verbose)\n> +\t\tfprintf_ln(stderr, \"%s\", _(msg));\n> +}\n> +\n> +static int reftable_fsck_error_handler(struct reftable_fsck_info info,\n\nA design question: why do we need to pass the value \"info\" instead of\npointer?\n\n> +\t\t\t\t void *cb_data)\n> +{\n> +\tstruct fsck_options *o = cb_data;\n> +\tstruct fsck_ref_report report = { .path = info.path };\n\nLet's make it reverse-christmas-tree ordering.\n\n> +\tenum fsck_msg_id msg_id;\n> +\n> +\tswitch (info.error) {\n> +\tcase REFTABLE_FSCK_ERROR_TABLE_NAME:\n> +\t\tmsg_id = FSCK_MSG_BAD_REFTABLE_TABLE_NAME;\n> +\t\tbreak;\n> +\tdefault:\n> +\t\tBUG(\"unknown fsck error: %d\", info.error);\n> +\t}\n> +\n> +\treturn fsck_report_ref(o, &report, msg_id, \"%s\", info.msg);\n> +}\n> +\n> +static int reftable_be_fsck(struct ref_store *ref_store, struct fsck_options *o,\n> \t\t\t struct worktree *wt UNUSED)\n> {\n> -\treturn 0;\n> +\tstruct reftable_ref_store *refs;\n> +\tstruct strmap_entry *entry;\n> +\tstruct hashmap_iter iter;\n> +\tint ret = 0;\n> +\n> +\trefs = reftable_be_downcast(ref_store, REF_STORE_READ, \"fsck\");\n> +\n> +\tif (o->verbose)\n> +\t\tfprintf_ln(stderr, _(\"Checking references consistency\"));\n> +\n> +\tret = reftable_fsck_check(refs->main_backend.stack, reftable_fsck_error_handler,\n> +\t\t\t\t reftable_fsck_verbose_handler, o);\n> +\tif (!ret)\n> +\t\treturn ret;\n> +\n\nFrom my understanding, if we find that there is any trouble in the main\nworktree reftable backend, we would just abort the check. Should we\ncontinue to check the linked worktrees?\n\n> +\tstrmap_for_each_entry(&refs->worktree_backends, &iter, entry) {\n> +\t\tstruct reftable_backend *b = (struct reftable_backend *)entry->value;\n> +\t\tret = reftable_fsck_check(b->stack, reftable_fsck_error_handler,\n> +\t\t\t\t\t reftable_fsck_verbose_handler, o);\n> +\t\tif (!ret)\n> +\t\t\treturn ret;\n> +\t}\n> +\n> +\treturn ret;\n> }\n> \n> struct ref_storage_be refs_be_reftable = {\n> diff --git a/reftable/fsck.c b/reftable/fsck.c\n> new file mode 100644\n> index 0000000000..22ec3c26e9\n> --- /dev/null\n> +++ b/reftable/fsck.c\n> @@ -0,0 +1,50 @@\n> +#include \"basics.h\"\n> +#include \"reftable-fsck.h\"\n> +#include \"stack.h\"\n> +\n> +int reftable_fsck_check(struct reftable_stack *stack,\n> +\t\t\treftable_fsck_report_fn report_fn,\n> +\t\t\treftable_fsck_verbose_fn verbose_fn,\n> +\t\t\tvoid *cb_data)\n> +{\n> +\tchar **names = NULL;\n> +\tuint64_t min, max;\n> +\tint err = 0;\n> +\n> +\tif (stack == NULL)\n> +\t\tgoto out;\n> +\n> +\terr = read_lines(stack->list_file, &names);\n> +\tif (err < 0)\n> +\t\tgoto out;\n> +\n> +\tverbose_fn(\"Checking reftable table names\", cb_data);\n> +\n> +\tfor (size_t i = 0; names[i]; i++) {\n> +\t\tstruct reftable_fsck_info info = {\n> +\t\t\t.error = REFTABLE_FSCK_ERROR_TABLE_NAME,\n> +\t\t\t.path = names[i],\n> +\t\t\t.msg = \"invalid reftable name\"\n> +\t\t};\n\nShould we define this data structure outside of the loop? It's\nunnecessary here as we could change \".path\" and \".msg\" dynamically in\nthe loop.\n\n> +\t\tuint32_t rnd;\n> +\t\t/*\n> +\t\t * We want to match the tail '.ref'. One extra byte to ensure\n> +\t\t * that there is no unexpected extra character and one byte for\n> +\t\t * the null terminator added by sscanf.\n> +\t\t */\n> +\t\tchar tail[6];\n> +\n> +\t\tif (sscanf(names[i], \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x%5s\",\n> +\t\t\t &min, &max, &rnd, tail) != 4) {\n> +\t\t\terr = report_fn(info, cb_data);\n\nI think we could just pass pointer to avoid unnecessary copy operations.\nBesides that, I think here we report two different kinds of problem. But\nwe would give report the user always the same message `invalid reftable\nname`. This is too vague.\n\nI think we'd better set different messages for different problems.\n\n> +\t\t}\n> +\n> +\t\tif (strcmp(tail, \".ref\")) {\n> +\t\t\terr = report_fn(info, cb_data);\n> +\t\t}\n> +\t}\n> +\n> +out:\n> +\tfree_names(names);\n> +\treturn err;\n> +}\n> diff --git a/reftable/reftable-fsck.h b/reftable/reftable-fsck.h\n> new file mode 100644\n> index 0000000000..087430d979\n> --- /dev/null\n> +++ b/reftable/reftable-fsck.h\n> @@ -0,0 +1,38 @@\n> +#ifndef REFTABLE_FSCK_H\n> +#define REFTABLE_FSCK_H\n> +\n> +#include \"reftable-stack.h\"\n> +\n> +enum reftable_fsck_error {\n> +\t/* Invalid table name */\n> +\tREFTABLE_FSCK_ERROR_TABLE_NAME = -1,\n> +};\n> +\n> +/* Represents an individual error encounctered during the FSCK checks. */\n> +struct reftable_fsck_info {\n> +\tenum reftable_fsck_error error;\n> +\tconst char *msg;\n> +\tconst char *path;\n> +};\n> +\n> +typedef int reftable_fsck_report_fn(struct reftable_fsck_info info,\n> +\t\t\t\t void *cb_data);\n\nAs I have explained above, we should use `struct reftable_fsck_info\n*info` instead of `struct reftable_fsck_info info`.\n\n> +typedef void reftable_fsck_verbose_fn(const char *msg, void *cb_data);\n> +\n> +/*\n> + * Given a reftable stack, perform FSCK check on the stack.\n> + *\n> + * If an issue is encountered, the issue is reported to the callee via the\n> + * provided 'report_fn'. If the issue is non-recoverable the flow will not\n> + * conitnue. If it is recoverable, the flow will continue and further issues\n> + * will be reported as identified.\n> + *\n> + * The 'verbose_fn' will be invoked to provide verbose information about\n> + * the progress and state of the FSCK checks.\n> + */\n> +int reftable_fsck_check(struct reftable_stack *stack,\n> +\t\t\treftable_fsck_report_fn report_fn,\n> +\t\t\treftable_fsck_verbose_fn verbose_fn,\n> +\t\t\tvoid *cb_data);\n> +\n> +#endif /* REFTABLE_FSCK_H */\n> diff --git a/t/meson.build b/t/meson.build\n> index bbeba1a8d5..a8eb44eb30 100644\n> --- a/t/meson.build\n> +++ b/t/meson.build\n> @@ -145,6 +145,7 @@ integration_tests = [\n> 't0611-reftable-httpd.sh',\n> 't0612-reftable-jgit-compatibility.sh',\n> 't0613-reftable-write-options.sh',\n> + 't0614-reftable-fsck.sh',\n> 't1000-read-tree-m-3way.sh',\n> 't1001-read-tree-m-2way.sh',\n> 't1002-read-tree-m-u-2way.sh',\n> @@ -1214,4 +1215,4 @@ if perl.found() and time.found()\n> timeout: 0,\n> )\n> endforeach\n> -endif\n> \\ No newline at end of file\n> +endif\n> diff --git a/t/t0614-reftable-fsck.sh b/t/t0614-reftable-fsck.sh\n> new file mode 100755\n> index 0000000000..0d11871b1c\n> --- /dev/null\n> +++ b/t/t0614-reftable-fsck.sh\n> @@ -0,0 +1,35 @@\n> +#!/bin/sh\n> +\n> +test_description='Test reftable backend consistency check'\n> +\n> +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n> +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n> +GIT_TEST_DEFAULT_REF_FORMAT=reftable\n> +export GIT_TEST_DEFAULT_REF_FORMAT\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'table name should be checked' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit commit --allow-empty -m initial &&\n> +\n> +\t\tgit refs verify 2>err &&\n> +\t\ttest_must_be_empty err &&\n> +\n> +\t\tTABLE_NAME=$(cat .git/reftable/tables.list | head -n1) &&\n> +\t\tsed \"1s/$/extra/\" .git/reftable/tables.list >.git/reftable/tables.list.tmp &&\n> +\t\tmv .git/reftable/tables.list.tmp .git/reftable/tables.list &&\n> +\t\tmv .git/reftable/${TABLE_NAME} .git/reftable/${TABLE_NAME}extra &&\n> +\n> +\t\ttest_must_fail git refs verify 2>err &&\n> +\t\tcat >expect <<-EOF &&\n> +\t\terror: ${TABLE_NAME}extra: badReftableTableName: invalid reftable name\n> +\t\tEOF\n> +\t\ttest_cmp expect err\n> +\t)\n> +'\n\nWe would check two kinds of errors, should we add two tests instead of\nonly this one.\n\n> +\n> +test_done\n> \n> -- \n> 2.50.1\n> \n\nThanks,\nJialuo\n"},{"id":"524974","messageId":"aK3hwQbO3YwdXa3q@ArchLinux","threadId":"63987","inReplyTo":"20250819-228-reftable-introduce-consistency-checks-v1-3-8b8f6879fa9e@gmail.com","subject":"Re: [PATCH 3/5] refs/reftable: add fsck check for number of tables","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-08-26T16:33:40Z","receivedAt":"2025-08-26T16:33:44Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Aug 19, 2025 at 02:21:02PM +0200, Karthik Nayak wrote:\n> diff --git a/reftable/fsck.c b/reftable/fsck.c\n> index 22ec3c26e9..e92a630276 100644\n> --- a/reftable/fsck.c\n> +++ b/reftable/fsck.c\n> @@ -2,6 +2,28 @@\n> #include \"reftable-fsck.h\"\n> #include \"stack.h\"\n> \n> +static int reftable_fsck_valid_stack_count(struct reftable_stack *st)\n> +{\n> +\tDIR *dir = opendir(st->reftable_dir);\n> +\tstruct dirent *d = NULL;\n> +\tunsigned int count = 0;\n> +\n> +\tif (!dir)\n> +\t\treturn 0;\n> +\n> +\twhile ((d = readdir(dir))) {\n> +\t\tif (!strcmp(d->d_name, \"tables.list\"))\n> +\t\t\tcontinue;\n> +\n> +\t\tif (d->d_type == DT_REG)\n> +\t\t\tcount++;\n> +\t}\n> +\n> +\tclosedir(dir);\n> +\n> +\treturn count == st->tables_len;\n> +}\n> +\n\nThe above logic is clear to understand but I think we should our\ninternal interface in \"dir-iterator.h\" to implement above logic.\n\n> int reftable_fsck_check(struct reftable_stack *stack,\n> \t\t\treftable_fsck_report_fn report_fn,\n> \t\t\treftable_fsck_verbose_fn verbose_fn,\n> @@ -44,6 +66,18 @@ int reftable_fsck_check(struct reftable_stack *stack,\n> \t\t}\n> \t}\n> \n> +\tverbose_fn(\"Checking reftable tables count\", cb_data);\n> +\n> +\tif (!reftable_fsck_valid_stack_count(stack)) {\n> +\t\tstruct reftable_fsck_info info = {\n> +\t\t\t.error = REFTABLE_FSCK_ERROR_STACK_COUNT,\n> +\t\t\t.path = stack->list_file,\n> +\t\t\t.msg = \"mismatch in number of tables\"\n> +\t\t};\n> +\n\nWhen reading here, I somehow understand the reason why you define this\ndata structure in the loop. But I still think we could just define only\none `info`.\n\nBTY, I wonder whether we should define some auxiliary functions for each\ncheck instead of adding logic directly in `reftable_fsck_check`\nfunction?\n\n> +\t\terr = report_fn(info, cb_data);\n> +\t}\n> +\n> out:\n> \tfree_names(names);\n> \treturn err;\n\nThanks,\nJialuo\n"},{"id":"524976","messageId":"aK3jNK82FILr2GuT@ArchLinux","threadId":"63987","inReplyTo":"20250819-228-reftable-introduce-consistency-checks-v1-0-8b8f6879fa9e@gmail.com","subject":"Re: [PATCH 0/5] refs/reftable: add fsck checks","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-08-26T16:39:16Z","receivedAt":"2025-08-26T16:39:19Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Aug 19, 2025 at 02:20:59PM +0200, Karthik Nayak wrote:\n> This series adds the required infrastructure and also some fsck checks\n> for the reftable backend.\n> \n> Since the reftable backend is treated as a library within the Git\n> codebase, we don't want to spillover our internal fsck implementation\n> into the library. At the same time, the fsck checks need to access\n> internal structures of the reftable library which aren't exposed outside\n> the library.\n> \n> So we solve this by adding a 'reftable/fsck.[ch]' which implements and\n> exposes a checker for the reftable library and returns specific errors\n> as defined by the library. We then add glue code within\n> 'refs/reftable-backend.c' to map these errors to errors which Git's fsck\n> implementation would understand. This allows us to separate concerns.\n> \n> This series then adds some checks on the stack ('reftable/tables.list')\n> level of reftable, namely:\n> 1. The table name is as per the spec\n> 2. The number of tables are consistent\n> 3. The tables.list has a newline at the end of file\n> 4. The table names follow correct index sequences\n> \n> I also plan to send in follow up series's which will implement further\n> checks and go into deeper layers (tables, block, references).\n> \n\nThanks for your patches, it's very nice to see that we begin to\nimplement the consistency checks for reftable backend. And I have left\nsome comments.\n\nThanks,\nJialuo\n"},{"id":"524978","messageId":"aK3kYZA1eq-sCs9b@ArchLinux","threadId":"63987","inReplyTo":"20250819-228-reftable-introduce-consistency-checks-v1-3-8b8f6879fa9e@gmail.com","subject":"Re: [PATCH 3/5] refs/reftable: add fsck check for number of tables","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-08-26T16:44:17Z","receivedAt":"2025-08-26T16:44:20Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Aug 19, 2025 at 02:21:02PM +0200, Karthik Nayak wrote:\n> +test_expect_success 'table count should be checked' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit commit --allow-empty -m initial &&\n> +\n> +\t\tgit refs verify 2>err &&\n> +\t\ttest_must_be_empty err &&\n> +\n> +\t\ttouch .git/reftable/0x000000002812-0x000000002813-c830a596.ref &&\n> +\n> +\t\ttest_must_fail git refs verify 2>err &&\n> +\t\tcat >expect <<-EOF &&\n> +\t\terror: $(pwd)/.git/reftable/tables.list: badReftableStackCount: mismatch in number of tables\n\nThis is a bad usage, we should just use `reftable/tables.list`. And this\nis a common pattern. We would print the relative path against the \".git\"\ndirectory.\n\nThanks,\nJialuo\n"},{"id":"525273","messageId":"CAOLa=ZR43JYu1ky_HF7nC4xkVe6B+fMWTNK+sczaar_8YNcd8A@mail.gmail.com","threadId":"63987","inReplyTo":"aK3fHRMFiRBYNiJE@ArchLinux","subject":"Re: [PATCH 2/5] refs/reftable: add fsck check for checking the table name","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-01T13:33:24Z","receivedAt":"2025-09-01T13:33:27Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> On Tue, Aug 19, 2025 at 02:21:01PM +0200, Karthik Nayak wrote:\n>> The `git refs verify` command is used to run fsck checks on the\n>> reference backends. This command is also invoked when users run 'git\n>> fsck'. While the files-backend has some fsck checks added, the reftable\n>> backend lacks such checks. Let's add the required infrastructure and a\n>> check to test for the table names in the 'tables.list' of reftables.\n>>\n>> For the infrastructure, since the reftable library is treated as an\n>> independent library we should ensure that the library code works\n>> independently without knowledge about Git's internals. To do this,\n>> add both 'reftable/fsck.c' and 'reftable/reftable-fsck.h'. Which\n>\n> A design question here, we name the \"fsck.c\" for the source code but for\n> the header, we use \"reftable-fsck.h\", it is a little strange. Why not\n> just \"fsck.h\" instead of \"reftable-fsck.h\".\n>\n\nSince the reftable code is treated as an external library, all\n'reftable-.*.h' headers are treated as headers which expose APIs for the\nlibraries users. We would have defined 'reftable/fsck.h' if there were\ninternal users of the 'fsck.c' code. But there are none.\n\n\n>> diff --git a/Documentation/fsck-msgids.adoc b/Documentation/fsck-msgids.adoc\n>> index 1c912615f9..784ddc0df5 100644\n>> --- a/Documentation/fsck-msgids.adoc\n>> +++ b/Documentation/fsck-msgids.adoc\n>> @@ -38,6 +38,9 @@\n>> `badReferentName`::\n>> \t(ERROR) The referent name of a symref is invalid.\n>>\n>> +`badReftableTableName`::\n>> +\t(ERROR) A reftable table has an invalid name.\n>> +\n>\n> When reading this, I feel a little strange. `Reftable` already indicates\n> it is a table. Should we simply say like the following:\n>\n> A reftable has an invalid table name\n>\n\nI'm not sure about this, since 'reftable' refers to the reference\nbackend and the 'table' refers to an individual table within the\n'reftable' format. I would say both are important.\n\nCC'ing Patrick here for a second opinion.\n\n>> `badTagName`::\n>> \t(INFO) A tag has an invalid format.\n>>\n>> diff --git a/Makefile b/Makefile\n>> index e11340c1ae..f2ddcc8d7c 100644\n>> --- a/Makefile\n>> +++ b/Makefile\n>> @@ -2733,6 +2733,7 @@ REFTABLE_OBJS += reftable/error.o\n>> REFTABLE_OBJS += reftable/block.o\n>> REFTABLE_OBJS += reftable/blocksource.o\n>> REFTABLE_OBJS += reftable/iter.o\n>> +REFTABLE_OBJS += reftable/fsck.o\n>> REFTABLE_OBJS += reftable/merged.o\n>> REFTABLE_OBJS += reftable/pq.o\n>> REFTABLE_OBJS += reftable/record.o\n>> diff --git a/fsck.h b/fsck.h\n>> index 559ad57807..5901f944a1 100644\n>> --- a/fsck.h\n>> +++ b/fsck.h\n>> @@ -34,6 +34,7 @@ enum fsck_msg_type {\n>> \tFUNC(BAD_PACKED_REF_HEADER, ERROR) \\\n>> \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n>> \tFUNC(BAD_REFERENT_NAME, ERROR) \\\n>> +\tFUNC(BAD_REFTABLE_TABLE_NAME, ERROR) \\\n>> \tFUNC(BAD_REF_CONTENT, ERROR) \\\n>> \tFUNC(BAD_REF_FILETYPE, ERROR) \\\n>> \tFUNC(BAD_REF_NAME, ERROR) \\\n>> diff --git a/meson.build b/meson.build\n>> index 5dd299b496..82879fbfaa 100644\n>> --- a/meson.build\n>> +++ b/meson.build\n>> @@ -452,6 +452,7 @@ libgit_sources = [\n>> 'reftable/error.c',\n>> 'reftable/block.c',\n>> 'reftable/blocksource.c',\n>> + 'reftable/fsck.c',\n>> 'reftable/iter.c',\n>> 'reftable/merged.c',\n>> 'reftable/pq.c',\n>> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n>> index 8dae1e1112..ccd12052f2 100644\n>> --- a/refs/reftable-backend.c\n>> +++ b/refs/reftable-backend.c\n>> @@ -6,20 +6,21 @@\n>> #include \"../config.h\"\n>> #include \"../dir.h\"\n>> #include \"../environment.h\"\n>> +#include \"../fsck.h\"\n>> #include \"../gettext.h\"\n>> #include \"../hash.h\"\n>> #include \"../hex.h\"\n>> #include \"../iterator.h\"\n>> #include \"../ident.h\"\n>> -#include \"../lockfile.h\"\n>\n> Here, we delete this header file. Is the reason that we don't need this\n> header file anymore?\n>\n\nYes, it wasn't needed in the first place, let me add a comment in the\ncommit message.\n\n>> #include \"../object.h\"\n>> #include \"../path.h\"\n>> #include \"../refs.h\"\n>> #include \"../reftable/reftable-basics.h\"\n>> -#include \"../reftable/reftable-stack.h\"\n>> -#include \"../reftable/reftable-record.h\"\n>> #include \"../reftable/reftable-error.h\"\n>> +#include \"../reftable/reftable-fsck.h\"\n>> #include \"../reftable/reftable-iterator.h\"\n>> +#include \"../reftable/reftable-record.h\"\n>> +#include \"../reftable/reftable-stack.h\"\n>> #include \"../repo-settings.h\"\n>> #include \"../setup.h\"\n>> #include \"../strmap.h\"\n>> @@ -2675,11 +2676,59 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n>> \treturn ret;\n>> }\n>>\n>> -static int reftable_be_fsck(struct ref_store *ref_store UNUSED,\n>> -\t\t\t struct fsck_options *o UNUSED,\n>> +static void reftable_fsck_verbose_handler(const char *msg, void *cb_data)\n>> +{\n>> +\tstruct fsck_options *o = cb_data;\n>> +\n>> +\tif (o->verbose)\n>> +\t\tfprintf_ln(stderr, \"%s\", _(msg));\n>> +}\n>> +\n>> +static int reftable_fsck_error_handler(struct reftable_fsck_info info,\n>\n> A design question: why do we need to pass the value \"info\" instead of\n> pointer?\n>\n\nI didn't see a reason to make it a pointer. But it does make it more\nefficient when the struct size increases. Let me change it!\n\n>\n>> +\t\t\t\t void *cb_data)\n>> +{\n>> +\tstruct fsck_options *o = cb_data;\n>> +\tstruct fsck_ref_report report = { .path = info.path };\n>\n> Let's make it reverse-christmas-tree ordering.\n>\n\nWill change!\n\n>> +static int reftable_be_fsck(struct ref_store *ref_store, struct fsck_options *o,\n>> \t\t\t struct worktree *wt UNUSED)\n>> {\n>> -\treturn 0;\n>> +\tstruct reftable_ref_store *refs;\n>> +\tstruct strmap_entry *entry;\n>> +\tstruct hashmap_iter iter;\n>> +\tint ret = 0;\n>> +\n>> +\trefs = reftable_be_downcast(ref_store, REF_STORE_READ, \"fsck\");\n>> +\n>> +\tif (o->verbose)\n>> +\t\tfprintf_ln(stderr, _(\"Checking references consistency\"));\n>> +\n>> +\tret = reftable_fsck_check(refs->main_backend.stack, reftable_fsck_error_handler,\n>> +\t\t\t\t reftable_fsck_verbose_handler, o);\n>> +\tif (!ret)\n>> +\t\treturn ret;\n>> +\n>\n> From my understanding, if we find that there is any trouble in the main\n> worktree reftable backend, we would just abort the check. Should we\n> continue to check the linked worktrees?\n>\n\nI think that makes sense. Let me make that change.\n\n>> diff --git a/reftable/fsck.c b/reftable/fsck.c\n>> new file mode 100644\n>> index 0000000000..22ec3c26e9\n>> --- /dev/null\n>> +++ b/reftable/fsck.c\n>> @@ -0,0 +1,50 @@\n>> +#include \"basics.h\"\n>> +#include \"reftable-fsck.h\"\n>> +#include \"stack.h\"\n>> +\n>> +int reftable_fsck_check(struct reftable_stack *stack,\n>> +\t\t\treftable_fsck_report_fn report_fn,\n>> +\t\t\treftable_fsck_verbose_fn verbose_fn,\n>> +\t\t\tvoid *cb_data)\n>> +{\n>> +\tchar **names = NULL;\n>> +\tuint64_t min, max;\n>> +\tint err = 0;\n>> +\n>> +\tif (stack == NULL)\n>> +\t\tgoto out;\n>> +\n>> +\terr = read_lines(stack->list_file, &names);\n>> +\tif (err < 0)\n>> +\t\tgoto out;\n>> +\n>> +\tverbose_fn(\"Checking reftable table names\", cb_data);\n>> +\n>> +\tfor (size_t i = 0; names[i]; i++) {\n>> +\t\tstruct reftable_fsck_info info = {\n>> +\t\t\t.error = REFTABLE_FSCK_ERROR_TABLE_NAME,\n>> +\t\t\t.path = names[i],\n>> +\t\t\t.msg = \"invalid reftable name\"\n>> +\t\t};\n>\n> Should we define this data structure outside of the loop? It's\n> unnecessary here as we could change \".path\" and \".msg\" dynamically in\n> the loop.\n>\n\nI don't think it'd make much difference for reftables, since tables are\ngeometrically packed. But I don't feel strongly, so I'll make the\nchange.\n\n>> +\t\tuint32_t rnd;\n>> +\t\t/*\n>> +\t\t * We want to match the tail '.ref'. One extra byte to ensure\n>> +\t\t * that there is no unexpected extra character and one byte for\n>> +\t\t * the null terminator added by sscanf.\n>> +\t\t */\n>> +\t\tchar tail[6];\n>> +\n>> +\t\tif (sscanf(names[i], \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x%5s\",\n>> +\t\t\t &min, &max, &rnd, tail) != 4) {\n>> +\t\t\terr = report_fn(info, cb_data);\n>\n> I think we could just pass pointer to avoid unnecessary copy operations.\n> Besides that, I think here we report two different kinds of problem. But\n> we would give report the user always the same message `invalid reftable\n> name`. This is too vague.\n>\n\nNot sure what you mean by 'unnecessary copy operations', could you\nelaborate?\n\n> I think we'd better set different messages for different problems.\n>\n\nFair enough, let me modify that.\n\n[snip]\n\n>> diff --git a/t/t0614-reftable-fsck.sh b/t/t0614-reftable-fsck.sh\n>> new file mode 100755\n>> index 0000000000..0d11871b1c\n>> --- /dev/null\n>> +++ b/t/t0614-reftable-fsck.sh\n>> @@ -0,0 +1,35 @@\n>> +#!/bin/sh\n>> +\n>> +test_description='Test reftable backend consistency check'\n>> +\n>> +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n>> +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n>> +GIT_TEST_DEFAULT_REF_FORMAT=reftable\n>> +export GIT_TEST_DEFAULT_REF_FORMAT\n>> +\n>> +. ./test-lib.sh\n>> +\n>> +test_expect_success 'table name should be checked' '\n>> +\ttest_when_finished \"rm -rf repo\" &&\n>> +\tgit init repo &&\n>> +\t(\n>> +\t\tcd repo &&\n>> +\t\tgit commit --allow-empty -m initial &&\n>> +\n>> +\t\tgit refs verify 2>err &&\n>> +\t\ttest_must_be_empty err &&\n>> +\n>> +\t\tTABLE_NAME=$(cat .git/reftable/tables.list | head -n1) &&\n>> +\t\tsed \"1s/$/extra/\" .git/reftable/tables.list >.git/reftable/tables.list.tmp &&\n>> +\t\tmv .git/reftable/tables.list.tmp .git/reftable/tables.list &&\n>> +\t\tmv .git/reftable/${TABLE_NAME} .git/reftable/${TABLE_NAME}extra &&\n>> +\n>> +\t\ttest_must_fail git refs verify 2>err &&\n>> +\t\tcat >expect <<-EOF &&\n>> +\t\terror: ${TABLE_NAME}extra: badReftableTableName: invalid reftable name\n>> +\t\tEOF\n>> +\t\ttest_cmp expect err\n>> +\t)\n>> +'\n>\n> We would check two kinds of errors, should we add two tests instead of\n> only this one.\n>\n\nYeah, makes sense, will add!\n\n>> +\n>> +test_done\n>>\n>> --\n>> 2.50.1\n>>\n>\n> Thanks,\n> Jialuo\n\nThanks for the review.\n"},{"id":"525274","messageId":"CAOLa=ZTuDtWC9bCQ4h+tvwGXoL7THxV5fLPxi6a1Z73B=7y+4w@mail.gmail.com","threadId":"63987","inReplyTo":"aK3hwQbO3YwdXa3q@ArchLinux","subject":"Re: [PATCH 3/5] refs/reftable: add fsck check for number of tables","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-01T13:40:04Z","receivedAt":"2025-09-01T13:40:07Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> On Tue, Aug 19, 2025 at 02:21:02PM +0200, Karthik Nayak wrote:\n>> diff --git a/reftable/fsck.c b/reftable/fsck.c\n>> index 22ec3c26e9..e92a630276 100644\n>> --- a/reftable/fsck.c\n>> +++ b/reftable/fsck.c\n>> @@ -2,6 +2,28 @@\n>> #include \"reftable-fsck.h\"\n>> #include \"stack.h\"\n>>\n>> +static int reftable_fsck_valid_stack_count(struct reftable_stack *st)\n>> +{\n>> +\tDIR *dir = opendir(st->reftable_dir);\n>> +\tstruct dirent *d = NULL;\n>> +\tunsigned int count = 0;\n>> +\n>> +\tif (!dir)\n>> +\t\treturn 0;\n>> +\n>> +\twhile ((d = readdir(dir))) {\n>> +\t\tif (!strcmp(d->d_name, \"tables.list\"))\n>> +\t\t\tcontinue;\n>> +\n>> +\t\tif (d->d_type == DT_REG)\n>> +\t\t\tcount++;\n>> +\t}\n>> +\n>> +\tclosedir(dir);\n>> +\n>> +\treturn count == st->tables_len;\n>> +}\n>> +\n>\n> The above logic is clear to understand but I think we should our\n> internal interface in \"dir-iterator.h\" to implement above logic.\n>\n\nSince the reftable library is treated as external one. We can't add and\nrely on code outside of the library. That's why you'll see some\nduplication here and there.\n\n>> int reftable_fsck_check(struct reftable_stack *stack,\n>> \t\t\treftable_fsck_report_fn report_fn,\n>> \t\t\treftable_fsck_verbose_fn verbose_fn,\n>> @@ -44,6 +66,18 @@ int reftable_fsck_check(struct reftable_stack *stack,\n>> \t\t}\n>> \t}\n>>\n>> +\tverbose_fn(\"Checking reftable tables count\", cb_data);\n>> +\n>> +\tif (!reftable_fsck_valid_stack_count(stack)) {\n>> +\t\tstruct reftable_fsck_info info = {\n>> +\t\t\t.error = REFTABLE_FSCK_ERROR_STACK_COUNT,\n>> +\t\t\t.path = stack->list_file,\n>> +\t\t\t.msg = \"mismatch in number of tables\"\n>> +\t\t};\n>> +\n>\n> When reading here, I somehow understand the reason why you define this\n> data structure in the loop. But I still think we could just define only\n> one `info`.\n>\n\nI tried to rewrite it like you suggested, but I think it still makes\nsense to keep the error definitions separate. They help provide\nlocalized context. Otherwise, we'd define the error at the start, then\nset individual fields later on. This causes some confusion.\n\n>\n> BTY, I wonder whether we should define some auxiliary functions for each\n> check instead of adding logic directly in `reftable_fsck_check`\n> function?\n>\n\nPost this patch series we'll dive into block and reference checks, which\nwill be isolated into individual functions.\n\n>> +\t\terr = report_fn(info, cb_data);\n>> +\t}\n>> +\n>> out:\n>> \tfree_names(names);\n>> \treturn err;\n>\n> Thanks,\n> Jialuo\n"},{"id":"525276","messageId":"CAOLa=ZSVu9Y9MFE8S0xV9YysE53aD3bK2Wx9Q3Cr3UEwGS2JGg@mail.gmail.com","threadId":"63987","inReplyTo":"aK3kYZA1eq-sCs9b@ArchLinux","subject":"Re: [PATCH 3/5] refs/reftable: add fsck check for number of tables","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-01T13:52:19Z","receivedAt":"2025-09-01T13:52:22Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> On Tue, Aug 19, 2025 at 02:21:02PM +0200, Karthik Nayak wrote:\n>> +test_expect_success 'table count should be checked' '\n>> +\ttest_when_finished \"rm -rf repo\" &&\n>> +\tgit init repo &&\n>> +\t(\n>> +\t\tcd repo &&\n>> +\t\tgit commit --allow-empty -m initial &&\n>> +\n>> +\t\tgit refs verify 2>err &&\n>> +\t\ttest_must_be_empty err &&\n>> +\n>> +\t\ttouch .git/reftable/0x000000002812-0x000000002813-c830a596.ref &&\n>> +\n>> +\t\ttest_must_fail git refs verify 2>err &&\n>> +\t\tcat >expect <<-EOF &&\n>> +\t\terror: $(pwd)/.git/reftable/tables.list: badReftableStackCount: mismatch in number of tables\n>\n> This is a bad usage, we should just use `reftable/tables.list`. And this\n> is a common pattern. We would print the relative path against the \".git\"\n> directory.\n>\n\nGood point, this can be fixed to 'reftable/tables.list', we don't need\nto obtain it from the stack.\n\n> Thanks,\n> Jialuo\n\nThanks\n"},{"id":"525277","messageId":"CAOLa=ZRiG_6TD7ff=F+C2De1WqBSYHLc+Ev7NDbCtF3hFreP=Q@mail.gmail.com","threadId":"63987","inReplyTo":"aK3jNK82FILr2GuT@ArchLinux","subject":"Re: [PATCH 0/5] refs/reftable: add fsck checks","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-01T13:52:46Z","receivedAt":"2025-09-01T13:52:47Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"shejialuo writes:\n\n> On Tue, Aug 19, 2025 at 02:20:59PM +0200, Karthik Nayak wrote:\n>> This series adds the required infrastructure and also some fsck checks\n>> for the reftable backend.\n>>\n>> Since the reftable backend is treated as a library within the Git\n>> codebase, we don't want to spillover our internal fsck implementation\n>> into the library. At the same time, the fsck checks need to access\n>> internal structures of the reftable library which aren't exposed outside\n>> the library.\n>>\n>> So we solve this by adding a 'reftable/fsck.[ch]' which implements and\n>> exposes a checker for the reftable library and returns specific errors\n>> as defined by the library. We then add glue code within\n>> 'refs/reftable-backend.c' to map these errors to errors which Git's fsck\n>> implementation would understand. This allows us to separate concerns.\n>>\n>> This series then adds some checks on the stack ('reftable/tables.list')\n>> level of reftable, namely:\n>> 1. The table name is as per the spec\n>> 2. The number of tables are consistent\n>> 3. The tables.list has a newline at the end of file\n>> 4. The table names follow correct index sequences\n>>\n>> I also plan to send in follow up series's which will implement further\n>> checks and go into deeper layers (tables, block, references).\n>>\n>\n> Thanks for your patches, it's very nice to see that we begin to\n> implement the consistency checks for reftable backend. And I have left\n> some comments.\n>\n\nThanks for your comments and the review. I'll send in a new version soon.\n\n> Thanks,\n> Jialuo\n"},{"id":"525429","messageId":"aLhFFFKOo4CtVJJy@ArchLinux","threadId":"63987","inReplyTo":"CAOLa=ZR43JYu1ky_HF7nC4xkVe6B+fMWTNK+sczaar_8YNcd8A@mail.gmail.com","subject":"Re: [PATCH 2/5] refs/reftable: add fsck check for checking the table name","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-09-03T13:39:32Z","receivedAt":"2025-09-03T13:39:27Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Mon, Sep 01, 2025 at 06:33:24AM -0700, Karthik Nayak wrote:\n> shejialuo writes:\n> \n> > On Tue, Aug 19, 2025 at 02:21:01PM +0200, Karthik Nayak wrote:\n> >> The `git refs verify` command is used to run fsck checks on the\n> >> reference backends. This command is also invoked when users run 'git\n> >> fsck'. While the files-backend has some fsck checks added, the reftable\n> >> backend lacks such checks. Let's add the required infrastructure and a\n> >> check to test for the table names in the 'tables.list' of reftables.\n> >>\n> >> For the infrastructure, since the reftable library is treated as an\n> >> independent library we should ensure that the library code works\n> >> independently without knowledge about Git's internals. To do this,\n> >> add both 'reftable/fsck.c' and 'reftable/reftable-fsck.h'. Which\n> >\n> > A design question here, we name the \"fsck.c\" for the source code but for\n> > the header, we use \"reftable-fsck.h\", it is a little strange. Why not\n> > just \"fsck.h\" instead of \"reftable-fsck.h\".\n> >\n> \n> Since the reftable code is treated as an external library, all\n> 'reftable-.*.h' headers are treated as headers which expose APIs for the\n> libraries users. We would have defined 'reftable/fsck.h' if there were\n> internal users of the 'fsck.c' code. But there are none.\n> \n\nI understand the design. Thanks for the explanation.\n\n[snip]\n\n> >> +\t\tuint32_t rnd;\n> >> +\t\t/*\n> >> +\t\t * We want to match the tail '.ref'. One extra byte to ensure\n> >> +\t\t * that there is no unexpected extra character and one byte for\n> >> +\t\t * the null terminator added by sscanf.\n> >> +\t\t */\n> >> +\t\tchar tail[6];\n> >> +\n> >> +\t\tif (sscanf(names[i], \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x%5s\",\n> >> +\t\t\t &min, &max, &rnd, tail) != 4) {\n> >> +\t\t\terr = report_fn(info, cb_data);\n> >\n> > I think we could just pass pointer to avoid unnecessary copy operations.\n> > Besides that, I think here we report two different kinds of problem. But\n> > we would give report the user always the same message `invalid reftable\n> > name`. This is too vague.\n> >\n> \n> Not sure what you mean by 'unnecessary copy operations', could you\n> elaborate?\n> \n\nIn `report_fn`, we would copy the `info` value for each call. That's my\nmeaning.\n\nThanks,\nJialuo\n"},{"id":"526684","messageId":"20250918-228-reftable-introduce-consistency-checks-v3-0-271af03eb34d@gmail.com","threadId":"63987","inReplyTo":"20250819-228-reftable-introduce-consistency-checks-v1-0-8b8f6879fa9e@gmail.com","subject":"[PATCH v3 0/8] refs/reftable: add consistency checks","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-18T08:11:41Z","receivedAt":"2025-09-18T08:11:50Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The reference subsystems allows for adding backend specific consistency\nchecks. These checks are run as part of 'git refs verify'.\n\nWhile the files backend has some consistency checks added, the reftable\nbackend currently has none. This series first tightens the reftable\nbackend to make it a little more strict and then also adds the required\ninfrastructure and some simple consistency checks.\n\nSince the reftable backend is treated as a library within the Git\ncodebase, we don't want to spillover our internal fsck implementation\ninto the library. At the same time, the fsck checks need to access\ninternal structures of the reftable library which aren't exposed outside\nthe library.\n\nSo we solve this by adding a 'reftable/fsck.[ch]' which implements and\nexposes a checker for the reftable library and returns specific errors\nas defined by the library. We then add glue code within\n'refs/reftable-backend.c' to map these errors to errors which Git's fsck\nimplementation would understand. This allows us to separate concerns.\n\nWe add the following consistency checks:\n\n 1. Check for validating the reftable table name. This is treated as a\n warning since the reftable specification only suggests a table name\n but doesn't enforce it. Also there is a difference in the table name\n used in Git vs that in jGit.\n\n 2. Check for checking additional files present in the reftable\n directory.\n\nWe tighten the reftable backend by raising a REFTABLE_FORMAT_ERROR error\nwhen:\n\n1. The 'tables.list' file doesn't have a trailing newline.\n 2. Tables added to a reftable stack are not sequential.\n\n---\nChanges in v3:\n- I took a long hiatus from this topic, mostly due to other priorities.\n This has been rebased on top of '92c87bdc40 (The eighth batch,\n 2025-09-12)' since there were conflicts.\n- Junio suggested that two of the consistency checks (trailing newlines,\n sequential update indices for tables in stack) should actually be\n checked during runtime. I have made that change in this version.\n- I've cleaned up the code and modularized the 'reftable/fsck.c' code.\n- Invalid table name emits a warning, since the reftable spec doesn't\n enforce it but only makes a suggestion.\n- Broken down the commits to make it easier to review.\n- Link to v2: https://lore.kernel.org/r/20250902-228-reftable-introduce-consistency-checks-v2-0-4f96b3834779@gmail.com\n\nChanges in v2:\n- Ensured that 'struct reftable_fsck_info' is passed around as a\n pointer, this provides a smaller footprint (pointer size vs struct\n size).\n- Run FSCK checks for other worktrees too, even if one of them fails.\n- Separate messaging for table name vs table check and add additional\n test.\n- Use the relative path in messages used.\n- Small style and typo fixes.\n- Link to v1: https://lore.kernel.org/r/20250819-228-reftable-introduce-consistency-checks-v1-0-8b8f6879fa9e@gmail.com\n\n---\n Documentation/fsck-msgids.adoc | 9 ++--\n Makefile | 3 +-\n fsck.h | 40 +++++++-------\n meson.build | 1 +\n refs.c | 4 ++\n refs/debug.c | 1 -\n refs/files-backend.c | 3 --\n refs/reftable-backend.c | 59 ++++++++++++++++++---\n reftable/basics.c | 28 ++++++----\n reftable/basics.h | 7 +--\n reftable/fsck.c | 112 +++++++++++++++++++++++++++++++++++++++\n reftable/reftable-fsck.h | 42 +++++++++++++++\n reftable/stack.c | 15 ++++--\n t/meson.build | 1 +\n t/t0614-reftable-fsck.sh | 55 +++++++++++++++++++\n t/unit-tests/u-reftable-basics.c | 23 ++++++--\n t/unit-tests/u-reftable-stack.c | 28 ++++++++++\n 17 files changed, 378 insertions(+), 53 deletions(-)\n\nKarthik Nayak (8):\n refs: remove unused headers\n refs: move consistency check msg to generic layer\n reftable: check for trailing newline in 'tables.list'\n reftable: ensure tables in a stack use sequential update indices\n Documentation/fsck-msgids: remove duplicate msg id\n fsck: order 'fsck_msg_type' alphabetically\n reftable: add code to facilitate consistency checks\n refs/reftable: add fsck check for checking the table name\n\nRange-diff versus v2:\n\n1: eea34c56f0 < -: ---------- fsck: order 'fsck_msg_type' alphabetically\n2: dafcf618e9 < -: ---------- refs/reftable: add fsck check for checking the table name\n3: 20294ade9b < -: ---------- refs/reftable: add fsck check for number of tables\n4: 03c7979528 < -: ---------- refs/reftable: add fsck check for trailing newline\n5: eb74502cd3 < -: ---------- refs/reftable: add fsck check for incorrect update index\n-: ---------- > 1: c9f39a04ca refs: remove unused headers\n-: ---------- > 2: e1baf61a8a refs: move consistency check msg to generic layer\n-: ---------- > 3: 88a2ae1171 reftable: check for trailing newline in 'tables.list'\n-: ---------- > 4: 2dd1750a9d reftable: ensure tables in a stack use sequential update indices\n-: ---------- > 5: a7f6c52385 Documentation/fsck-msgids: remove duplicate msg id\n-: ---------- > 6: 873c21c73f fsck: order 'fsck_msg_type' alphabetically\n-: ---------- > 7: cbaac94328 reftable: add code to facilitate consistency checks\n-: ---------- > 8: e7fcc15608 refs/reftable: add fsck check for checking the table name\n\n\nbase-commit: a483264b01b977f3e65a4419103c21e6af7412a2\nchange-id: 20250714-228-reftable-introduce-consistency-checks-379ded93c544\n\nThanks\n- Karthik\n\n"},{"id":"526685","messageId":"20250918-228-reftable-introduce-consistency-checks-v3-1-271af03eb34d@gmail.com","threadId":"63987","inReplyTo":"20250918-228-reftable-introduce-consistency-checks-v3-0-271af03eb34d@gmail.com","subject":"[PATCH v3 1/8] refs: remove unused headers","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-18T08:11:42Z","receivedAt":"2025-09-18T08:11:50Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"In the 'refs/' namespace, some of the included header files are not\nneeded, let's remove them.\n\nSigned-off-by: Karthik Nayak \n---\n refs/debug.c | 1 -\n refs/files-backend.c | 1 -\n refs/reftable-backend.c | 1 -\n 3 files changed, 3 deletions(-)\n\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 1cb955961e..697adbd0dc 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -1,7 +1,6 @@\n #include \"git-compat-util.h\"\n #include \"hex.h\"\n #include \"refs-internal.h\"\n-#include \"string-list.h\"\n #include \"trace.h\"\n \n static struct trace_key trace_refs = TRACE_KEY_INIT(REFS);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 1b3bf26add..d4fb033417 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -20,7 +20,6 @@\n #include \"../dir-iterator.h\"\n #include \"../lockfile.h\"\n #include \"../object.h\"\n-#include \"../object-file.h\"\n #include \"../path.h\"\n #include \"../dir.h\"\n #include \"../chdir-notify.h\"\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 9e889da2ff..2152349cb9 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -11,7 +11,6 @@\n #include \"../hex.h\"\n #include \"../iterator.h\"\n #include \"../ident.h\"\n-#include \"../lockfile.h\"\n #include \"../object.h\"\n #include \"../path.h\"\n #include \"../refs.h\"\n\n-- \n2.51.0\n\n"},{"id":"526686","messageId":"20250918-228-reftable-introduce-consistency-checks-v3-2-271af03eb34d@gmail.com","threadId":"63987","inReplyTo":"20250918-228-reftable-introduce-consistency-checks-v3-0-271af03eb34d@gmail.com","subject":"[PATCH v3 2/8] refs: move consistency check msg to generic layer","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-18T08:11:43Z","receivedAt":"2025-09-18T08:11:51Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The files-backend prints a message before the consistency checks run.\nMove this to the generic layer so both the files and reftable backend\ncan benefit from this message.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 4 ++++\n refs/files-backend.c | 2 --\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 4ff55cf24f..4a7c394226 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -32,6 +32,7 @@\n #include \"commit.h\"\n #include \"wildmatch.h\"\n #include \"ident.h\"\n+#include \"fsck.h\"\n \n /*\n * List of all available backends\n@@ -323,6 +324,9 @@ int check_refname_format(const char *refname, int flags)\n int refs_fsck(struct ref_store *refs, struct fsck_options *o,\n \t struct worktree *wt)\n {\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, _(\"Checking references consistency\"));\n+\n \treturn refs->be->fsck(refs, o, wt);\n }\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex d4fb033417..603b1343d8 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3906,8 +3906,6 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \t\tNULL,\n \t};\n \n-\tif (o->verbose)\n-\t\tfprintf_ln(stderr, _(\"Checking references consistency\"));\n \treturn files_fsck_refs_dir(ref_store, o, \"refs\", wt, fsck_refs_fn);\n }\n \n\n-- \n2.51.0\n\n"},{"id":"526687","messageId":"20250918-228-reftable-introduce-consistency-checks-v3-3-271af03eb34d@gmail.com","threadId":"63987","inReplyTo":"20250918-228-reftable-introduce-consistency-checks-v3-0-271af03eb34d@gmail.com","subject":"[PATCH v3 3/8] reftable: check for trailing newline in 'tables.list'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-18T08:11:44Z","receivedAt":"2025-09-18T08:11:52Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"In the reftable format, the 'tables.list' file contains a newline\nseparated list of tables. While we parse this file, we do not check or\ncare about trailing newlines. Tighten the parser in `parse_names()` to\nreturn an appropriate error if there is no trailing newline.\n\nThis requires modification to `parse_names()` to accept a third argument\nwhich will hold the error value.\n\nSigned-off-by: Karthik Nayak \n---\n reftable/basics.c | 28 +++++++++++++++++++---------\n reftable/basics.h | 7 ++++---\n reftable/stack.c | 6 ++----\n t/unit-tests/u-reftable-basics.c | 23 +++++++++++++++++++----\n 4 files changed, 44 insertions(+), 20 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 9988ebd635..75d4086769 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -195,7 +195,7 @@ size_t names_length(const char **names)\n \treturn p - names;\n }\n \n-char **parse_names(char *buf, int size)\n+char **parse_names(char *buf, int size, int *err)\n {\n \tchar **names = NULL;\n \tsize_t names_cap = 0;\n@@ -205,30 +205,40 @@ char **parse_names(char *buf, int size)\n \n \twhile (p < end) {\n \t\tchar *next = strchr(p, '\\n');\n-\t\tif (next && next < end) {\n+\t\tif (!next) {\n+\t\t\t*err = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n+\t\t} else if (next < end) {\n \t\t\t*next = 0;\n \t\t} else {\n \t\t\tnext = end;\n \t\t}\n+\n \t\tif (p < next) {\n \t\t\tif (REFTABLE_ALLOC_GROW(names, names_len + 1,\n-\t\t\t\t\t\tnames_cap))\n-\t\t\t\tgoto err;\n+\t\t\t\t\t\tnames_cap)) {\n+\t\t\t\t*err = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n \n \t\t\tnames[names_len] = reftable_strdup(p);\n-\t\t\tif (!names[names_len++])\n-\t\t\t\tgoto err;\n+\t\t\tif (!names[names_len++]) {\n+\t\t\t\t*err = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n \t\t}\n \t\tp = next + 1;\n \t}\n \n-\tif (REFTABLE_ALLOC_GROW(names, names_len + 1, names_cap))\n-\t\tgoto err;\n+\tif (REFTABLE_ALLOC_GROW(names, names_len + 1, names_cap)) {\n+\t\t*err = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tnames[names_len] = NULL;\n \n \treturn names;\n \n-err:\n+done:\n \tfor (size_t i = 0; i < names_len; i++)\n \t\treftable_free(names[i]);\n \treftable_free(names);\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 7d22f96261..019dfe6d7e 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -167,10 +167,11 @@ void free_names(char **a);\n \n /*\n * Parse a newline separated list of names. `size` is the length of the buffer,\n- * without terminating '\\0'. Empty names are discarded. Returns a `NULL`\n- * pointer when allocations fail.\n+ * without terminating '\\0'. Empty names are discarded.\n+ *\n+ * Errors are assigned to the `err` variable.\n */\n-char **parse_names(char *buf, int size);\n+char **parse_names(char *buf, int size, int *err);\n \n /* compares two NULL-terminated arrays of strings. */\n int names_equal(const char **a, const char **b);\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex f91ce50bcd..955be1edb6 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -109,11 +109,9 @@ static int fd_read_lines(int fd, char ***namesp)\n \t}\n \tbuf[size] = 0;\n \n-\t*namesp = parse_names(buf, size);\n-\tif (!*namesp) {\n-\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t*namesp = parse_names(buf, size, &err);\n+\tif (!*namesp)\n \t\tgoto done;\n-\t}\n \n done:\n \treftable_free(buf);\ndiff --git a/t/unit-tests/u-reftable-basics.c b/t/unit-tests/u-reftable-basics.c\nindex a0471083e7..f77ec96429 100644\n--- a/t/unit-tests/u-reftable-basics.c\n+++ b/t/unit-tests/u-reftable-basics.c\n@@ -9,6 +9,7 @@ license that can be found in the LICENSE file or at\n #include \"unit-test.h\"\n #include \"lib-reftable.h\"\n #include \"reftable/basics.h\"\n+#include \"reftable/reftable-error.h\"\n \n struct integer_needle_lesseq_args {\n \tint needle;\n@@ -79,14 +80,17 @@ void test_reftable_basics__names_equal(void)\n void test_reftable_basics__parse_names(void)\n {\n \tchar in1[] = \"line\\n\";\n-\tchar in2[] = \"a\\nb\\nc\";\n-\tchar **out = parse_names(in1, strlen(in1));\n+\tchar in2[] = \"a\\nb\\nc\\n\";\n+\tint err = 0;\n+\tchar **out = parse_names(in1, strlen(in1), &err);\n+\tcl_assert(err == 0);\n \tcl_assert(out != NULL);\n \tcl_assert_equal_s(out[0], \"line\");\n \tcl_assert(!out[1]);\n \tfree_names(out);\n \n-\tout = parse_names(in2, strlen(in2));\n+\tout = parse_names(in2, strlen(in2), &err);\n+\tcl_assert(err == 0);\n \tcl_assert(out != NULL);\n \tcl_assert_equal_s(out[0], \"a\");\n \tcl_assert_equal_s(out[1], \"b\");\n@@ -95,10 +99,21 @@ void test_reftable_basics__parse_names(void)\n \tfree_names(out);\n }\n \n+void test_reftable_basics__parse_names_missing_newline(void)\n+{\n+\tchar in1[] = \"line\\nline2\";\n+\tint err = 0;\n+\tchar **out = parse_names(in1, strlen(in1), &err);\n+\tcl_assert(err == REFTABLE_FORMAT_ERROR);\n+\tcl_assert(out == NULL);\n+}\n+\n void test_reftable_basics__parse_names_drop_empty_string(void)\n {\n \tchar in[] = \"a\\n\\nb\\n\";\n-\tchar **out = parse_names(in, strlen(in));\n+\tint err = 0;\n+\tchar **out = parse_names(in, strlen(in), &err);\n+\tcl_assert(err == 0);\n \tcl_assert(out != NULL);\n \tcl_assert_equal_s(out[0], \"a\");\n \t/* simply '\\n' should be dropped as empty string */\n\n-- \n2.51.0\n\n"},{"id":"526688","messageId":"20250918-228-reftable-introduce-consistency-checks-v3-5-271af03eb34d@gmail.com","threadId":"63987","inReplyTo":"20250918-228-reftable-introduce-consistency-checks-v3-0-271af03eb34d@gmail.com","subject":"[PATCH v3 5/8] Documentation/fsck-msgids: remove duplicate msg id","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-18T08:11:46Z","receivedAt":"2025-09-18T08:11:54Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `gitmodulesLarge` is repeated twice. Remove the second duplicate.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/fsck-msgids.adoc | 3 ---\n 1 file changed, 3 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.adoc b/Documentation/fsck-msgids.adoc\nindex 0ba4f9a27e..1c912615f9 100644\n--- a/Documentation/fsck-msgids.adoc\n+++ b/Documentation/fsck-msgids.adoc\n@@ -104,9 +104,6 @@\n `gitmodulesParse`::\n \t(INFO) Could not parse `.gitmodules` blob.\n \n-`gitmodulesLarge`;\n-\t(ERROR) `.gitmodules` blob is too large to parse.\n-\n `gitmodulesPath`::\n \t(ERROR) `.gitmodules` path is invalid.\n \n\n-- \n2.51.0\n\n"},{"id":"526689","messageId":"20250918-228-reftable-introduce-consistency-checks-v3-4-271af03eb34d@gmail.com","threadId":"63987","inReplyTo":"20250918-228-reftable-introduce-consistency-checks-v3-0-271af03eb34d@gmail.com","subject":"[PATCH v3 4/8] reftable: ensure tables in a stack use sequential update indices","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-18T08:11:45Z","receivedAt":"2025-09-18T08:11:54Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"When tables are loaded into a stack, we expect that the tables are\nsequentially ordered by their update indices. But there is no validation\ndone for this. Add validation to ensure that tables loaded are\nsequential.\n\nRaise a 'REFTABLE_FORMAT_ERROR' when this validation fails.\n\nSigned-off-by: Karthik Nayak \n---\n reftable/stack.c | 9 +++++++++\n t/unit-tests/u-reftable-stack.c | 28 ++++++++++++++++++++++++++++\n 2 files changed, 37 insertions(+)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 955be1edb6..a458f5a4c5 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -238,6 +238,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st,\n \t\t\t\t int reuse_open)\n {\n \tsize_t cur_len = !st->merged ? 0 : st->merged->tables_len;\n+\tconst struct reftable_table *prev_table = NULL;\n \tstruct reftable_table **cur = NULL;\n \tstruct reftable_table **reused = NULL;\n \tstruct reftable_table **new_tables = NULL;\n@@ -317,6 +318,14 @@ static int reftable_stack_reload_once(struct reftable_stack *st,\n \n \t\tnew_tables[new_tables_len] = table;\n \t\tnew_tables_len++;\n+\n+\t\t/* table's update indices must be sequential */\n+\t\tif (prev_table && (prev_table->max_update_index != table->min_update_index - 1)) {\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tprev_table = table;\n \t}\n \n \t/* success! */\ndiff --git a/t/unit-tests/u-reftable-stack.c b/t/unit-tests/u-reftable-stack.c\nindex a8b91812e8..465f4a2689 100644\n--- a/t/unit-tests/u-reftable-stack.c\n+++ b/t/unit-tests/u-reftable-stack.c\n@@ -1330,3 +1330,31 @@ void test_reftable_stack__invalid_limit_updates(void)\n \treftable_stack_destroy(st);\n \tclear_dir(dir);\n }\n+\n+void test_reftable_stack__non_seq_update_indices(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct reftable_stack *st1 = NULL;\n+\tchar *dir = get_tmp_dir(__LINE__);\n+\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = (char *)\"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)\"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = (char *)\"branch2\",\n+\t\t.update_index = 3,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)\"master\",\n+\t};\n+\n+\tcl_assert_equal_i(reftable_new_stack(&st1, dir, &opts), 0);\n+\tcl_assert_equal_i(reftable_stack_add(st1, write_test_ref, &ref1, 0), 0);\n+\tcl_assert_equal_i(reftable_stack_add(st1, write_test_ref, &ref2, 0),\n+\t\t\t REFTABLE_FORMAT_ERROR);\n+\n+\treftable_stack_destroy(st1);\n+\tclear_dir(dir);\n+}\n\n-- \n2.51.0\n\n"},{"id":"526690","messageId":"20250918-228-reftable-introduce-consistency-checks-v3-6-271af03eb34d@gmail.com","threadId":"63987","inReplyTo":"20250918-228-reftable-introduce-consistency-checks-v3-0-271af03eb34d@gmail.com","subject":"[PATCH v3 6/8] fsck: order 'fsck_msg_type' alphabetically","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-18T08:11:47Z","receivedAt":"2025-09-18T08:11:55Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The list of 'fsck_msg_type' seem to be alphabetically ordered, but there\nare a few small misses. Fix this by sorting the sub-sections of the\nlist to maintain alphabetical ordering.\n\nSigned-off-by: Karthik Nayak \n---\n fsck.h | 38 +++++++++++++++++++-------------------\n 1 file changed, 19 insertions(+), 19 deletions(-)\n\ndiff --git a/fsck.h b/fsck.h\nindex dd7df3d5b3..6b0db235e0 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -33,15 +33,27 @@ enum fsck_msg_type {\n \tFUNC(BAD_PACKED_REF_ENTRY, ERROR) \\\n \tFUNC(BAD_PACKED_REF_HEADER, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REFERENT_NAME, ERROR) \\\n \tFUNC(BAD_REF_CONTENT, ERROR) \\\n \tFUNC(BAD_REF_FILETYPE, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n-\tFUNC(BAD_REFERENT_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n \tFUNC(BAD_TYPE, ERROR) \\\n \tFUNC(DUPLICATE_ENTRIES, ERROR) \\\n+\tFUNC(GITATTRIBUTES_BLOB, ERROR) \\\n+\tFUNC(GITATTRIBUTES_LARGE, ERROR) \\\n+\tFUNC(GITATTRIBUTES_LINE_LENGTH, ERROR) \\\n+\tFUNC(GITATTRIBUTES_MISSING, ERROR) \\\n+\tFUNC(GITMODULES_BLOB, ERROR) \\\n+\tFUNC(GITMODULES_LARGE, ERROR) \\\n+\tFUNC(GITMODULES_MISSING, ERROR) \\\n+\tFUNC(GITMODULES_NAME, ERROR) \\\n+\tFUNC(GITMODULES_PATH, ERROR) \\\n+\tFUNC(GITMODULES_SYMLINK, ERROR) \\\n+\tFUNC(GITMODULES_UPDATE, ERROR) \\\n+\tFUNC(GITMODULES_URL, ERROR) \\\n \tFUNC(MISSING_AUTHOR, ERROR) \\\n \tFUNC(MISSING_COMMITTER, ERROR) \\\n \tFUNC(MISSING_EMAIL, ERROR) \\\n@@ -60,39 +72,27 @@ enum fsck_msg_type {\n \tFUNC(TREE_NOT_SORTED, ERROR) \\\n \tFUNC(UNKNOWN_TYPE, ERROR) \\\n \tFUNC(ZERO_PADDED_DATE, ERROR) \\\n-\tFUNC(GITMODULES_MISSING, ERROR) \\\n-\tFUNC(GITMODULES_BLOB, ERROR) \\\n-\tFUNC(GITMODULES_LARGE, ERROR) \\\n-\tFUNC(GITMODULES_NAME, ERROR) \\\n-\tFUNC(GITMODULES_SYMLINK, ERROR) \\\n-\tFUNC(GITMODULES_URL, ERROR) \\\n-\tFUNC(GITMODULES_PATH, ERROR) \\\n-\tFUNC(GITMODULES_UPDATE, ERROR) \\\n-\tFUNC(GITATTRIBUTES_MISSING, ERROR) \\\n-\tFUNC(GITATTRIBUTES_LARGE, ERROR) \\\n-\tFUNC(GITATTRIBUTES_LINE_LENGTH, ERROR) \\\n-\tFUNC(GITATTRIBUTES_BLOB, ERROR) \\\n \t/* warnings */ \\\n \tFUNC(EMPTY_NAME, WARN) \\\n \tFUNC(FULL_PATHNAME, WARN) \\\n \tFUNC(HAS_DOT, WARN) \\\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n+\tFUNC(LARGE_PATHNAME, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n-\tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n-\tFUNC(LARGE_PATHNAME, WARN) \\\n+\tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \t/* infos (reported as warnings, but ignored by default) */ \\\n \tFUNC(BAD_FILEMODE, INFO) \\\n+\tFUNC(BAD_TAG_NAME, INFO) \\\n \tFUNC(EMPTY_PACKED_REFS_FILE, INFO) \\\n-\tFUNC(GITMODULES_PARSE, INFO) \\\n-\tFUNC(GITIGNORE_SYMLINK, INFO) \\\n \tFUNC(GITATTRIBUTES_SYMLINK, INFO) \\\n+\tFUNC(GITIGNORE_SYMLINK, INFO) \\\n+\tFUNC(GITMODULES_PARSE, INFO) \\\n \tFUNC(MAILMAP_SYMLINK, INFO) \\\n-\tFUNC(BAD_TAG_NAME, INFO) \\\n \tFUNC(MISSING_TAGGER_ENTRY, INFO) \\\n-\tFUNC(SYMLINK_REF, INFO) \\\n \tFUNC(REF_MISSING_NEWLINE, INFO) \\\n+\tFUNC(SYMLINK_REF, INFO) \\\n \tFUNC(SYMREF_TARGET_IS_NOT_A_REF, INFO) \\\n \tFUNC(TRAILING_REF_CONTENT, INFO) \\\n \t/* ignored (elevated when requested) */ \\\n\n-- \n2.51.0\n\n"},{"id":"526691","messageId":"20250918-228-reftable-introduce-consistency-checks-v3-7-271af03eb34d@gmail.com","threadId":"63987","inReplyTo":"20250918-228-reftable-introduce-consistency-checks-v3-0-271af03eb34d@gmail.com","subject":"[PATCH v3 7/8] reftable: add code to facilitate consistency checks","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-18T08:11:48Z","receivedAt":"2025-09-18T08:11:56Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git refs verify` command is used to run consistency checks on the\nreference backends. This command is also invoked when users run 'git\nfsck'. While the files-backend has some fsck checks added, the reftable\nbackend lacks such checks. Let's add the required infrastructure and a\ncheck to test for the files present in the reftable directory.\n\nSince the reftable library is treated as an independent library we\nshould ensure that the library code works independently without\nknowledge about Git's internals. To do this, add both 'reftable/fsck.c'\nand 'reftable/reftable-fsck.h'. Which provide an entry point\n'reftable_fsck_check' for running fsck checks over a provided reftable\nstack. The callee provides the function with callbacks to handle issue\nand information reporting.\n\nThe added check, goes over all files in the reftable directory and\nvalidates that they have the expected file type and a valid name. It\nraises specific errors for both.\n\nWhile here, move 'reftable/error.o' in the Makefile to retain\nlexicographic ordering.\n\nSigned-off-by: Karthik Nayak \n---\n Makefile | 3 +-\n meson.build | 1 +\n reftable/fsck.c | 112 +++++++++++++++++++++++++++++++++++++++++++++++\n reftable/reftable-fsck.h | 42 ++++++++++++++++++\n 4 files changed, 157 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex 4c95affadb..03fbaf2b21 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2732,9 +2732,10 @@ XDIFF_OBJS += xdiff/xutils.o\n xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n-REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n+REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/fsck.o\n REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\ndiff --git a/meson.build b/meson.build\nindex b3dfcc0497..8914252910 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -452,6 +452,7 @@ libgit_sources = [\n 'reftable/error.c',\n 'reftable/block.c',\n 'reftable/blocksource.c',\n+ 'reftable/fsck.c',\n 'reftable/iter.c',\n 'reftable/merged.c',\n 'reftable/pq.c',\ndiff --git a/reftable/fsck.c b/reftable/fsck.c\nnew file mode 100644\nindex 0000000000..785e4b43e8\n--- /dev/null\n+++ b/reftable/fsck.c\n@@ -0,0 +1,112 @@\n+#include \"basics.h\"\n+#include \"reftable-fsck.h\"\n+#include \"stack.h\"\n+\n+static bool valid_table_name(const char *name, uint64_t *min_update_index,\n+\t\t\t uint64_t *max_update_index)\n+{\n+\tconst char *ptr = name;\n+\tchar *endptr;\n+\n+\t/* strtoull doesn't set errno on success */\n+\terrno = 0;\n+\n+\t*min_update_index = strtoull(ptr, &endptr, 16);\n+\tif (errno == EINVAL)\n+\t\treturn false;\n+\tptr = endptr;\n+\n+\tif (strncmp(ptr, \"-\", 1))\n+\t\treturn false;\n+\tptr++;\n+\n+\t*max_update_index = strtoull(ptr, &endptr, 16);\n+\tif (errno == EINVAL)\n+\t\treturn false;\n+\tptr = endptr;\n+\n+\tif (*ptr != '-')\n+\t\treturn false;\n+\tptr++;\n+\n+\tstrtoul(ptr, &endptr, 16);\n+\tif (errno == EINVAL)\n+\t\treturn false;\n+\tptr = endptr;\n+\n+\tif (strcmp(ptr, \".ref\") && strcmp(ptr, \".log\"))\n+\t\treturn false;\n+\n+\treturn true;\n+}\n+\n+static int stack_check_all_files_in_dir(struct reftable_stack *stack,\n+\t\t\t\t\treftable_fsck_report_fn report_fn,\n+\t\t\t\t\tvoid *cb_data)\n+{\n+\tDIR *dir = opendir(stack->reftable_dir);\n+\tstruct reftable_fsck_info info;\n+\tstruct dirent *d = NULL;\n+\tuint64_t min, max;\n+\tint err = 0;\n+\n+\tif (!dir)\n+\t\treturn 0;\n+\n+\twhile ((d = readdir(dir))) {\n+\t\tif (!strcmp(d->d_name, \"tables.list\"))\n+\t\t\tcontinue;\n+\n+\t\tif ((d->d_name[0] == '.' &&\n+\t\t (d->d_name[1] == '\\0' ||\n+\t\t (d->d_name[1] == '.' && d->d_name[2] == '\\0'))))\n+\t\t\tcontinue;\n+\n+\t\tif (d->d_type == DT_REG) {\n+\t\t\tif (!valid_table_name(d->d_name, &min, &max)) {\n+\t\t\t\tinfo.error = REFTABLE_FSCK_ERROR_TABLE_NAME;\n+\t\t\t\tinfo.msg = \"file with invalid table name\";\n+\t\t\t\tinfo.path = d->d_name;\n+\n+\t\t\t\terr |= report_fn(&info, cb_data);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tinfo.error = REFTABLE_FSCK_ERROR_INVALID_FILE_TYPE;\n+\t\t\tinfo.msg = \"file with unexpected type\";\n+\t\t\tinfo.path = d->d_name;\n+\n+\t\t\terr |= report_fn(&info, cb_data);\n+\t\t}\n+\t}\n+\n+\tclosedir(dir);\n+\treturn err;\n+}\n+\n+static int stack_checks(struct reftable_stack *stack,\n+\t\t\treftable_fsck_report_fn report_fn,\n+\t\t\tvoid *cb_data)\n+{\n+\tstruct reftable_buf msg = REFTABLE_BUF_INIT;\n+\tchar **names = NULL;\n+\tint err = 0;\n+\n+\tif (stack == NULL)\n+\t\tgoto out;\n+\n+\terr |= stack_check_all_files_in_dir(stack, report_fn, cb_data);\n+\n+out:\n+\tfree_names(names);\n+\treftable_buf_release(&msg);\n+\treturn err;\n+}\n+\n+int reftable_fsck_check(struct reftable_stack *stack,\n+\t\t\treftable_fsck_report_fn report_fn,\n+\t\t\treftable_fsck_verbose_fn verbose_fn,\n+\t\t\tvoid *cb_data)\n+{\n+\tverbose_fn(\"Checking reftable: stack checks\", cb_data);\n+\treturn stack_checks(stack, report_fn, cb_data);\n+}\ndiff --git a/reftable/reftable-fsck.h b/reftable/reftable-fsck.h\nnew file mode 100644\nindex 0000000000..5e13ac9f02\n--- /dev/null\n+++ b/reftable/reftable-fsck.h\n@@ -0,0 +1,42 @@\n+#ifndef REFTABLE_FSCK_H\n+#define REFTABLE_FSCK_H\n+\n+#include \"reftable-stack.h\"\n+\n+enum reftable_fsck_error {\n+\t/* Non regular file in the reftable directory */\n+\tREFTABLE_FSCK_ERROR_INVALID_FILE_TYPE = 0,\n+\t/* Invalid table name */\n+\tREFTABLE_FSCK_ERROR_TABLE_NAME,\n+\t/* Used for bounds checking, must be last */\n+\tREFTABLE_FSCK_MAX_VALUE\n+};\n+\n+/* Represents an individual error encountered during the FSCK checks. */\n+struct reftable_fsck_info {\n+\tenum reftable_fsck_error error;\n+\tconst char *msg;\n+\tconst char *path;\n+};\n+\n+typedef int reftable_fsck_report_fn(struct reftable_fsck_info *info,\n+\t\t\t\t void *cb_data);\n+typedef void reftable_fsck_verbose_fn(const char *msg, void *cb_data);\n+\n+/*\n+ * Given a reftable stack, perform consistency checks on the stack.\n+ *\n+ * If an issue is encountered, the issue is reported to the callee via the\n+ * provided 'report_fn'. If the issue is non-recoverable the flow will not\n+ * continue. If it is recoverable, the flow will continue and further issues\n+ * will be reported as identified.\n+ *\n+ * The 'verbose_fn' will be invoked to provide verbose information about\n+ * the progress and state of the consistency checks.\n+ */\n+int reftable_fsck_check(struct reftable_stack *stack,\n+\t\t\treftable_fsck_report_fn report_fn,\n+\t\t\treftable_fsck_verbose_fn verbose_fn,\n+\t\t\tvoid *cb_data);\n+\n+#endif /* REFTABLE_FSCK_H */\n\n-- \n2.51.0\n\n"},{"id":"526692","messageId":"20250918-228-reftable-introduce-consistency-checks-v3-8-271af03eb34d@gmail.com","threadId":"63987","inReplyTo":"20250918-228-reftable-introduce-consistency-checks-v3-0-271af03eb34d@gmail.com","subject":"[PATCH v3 8/8] refs/reftable: add fsck check for checking the table name","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-18T08:11:49Z","receivedAt":"2025-09-18T08:11:56Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Add glue code in 'refs/reftable-backend.c' which calls the reftable\nlibrary to perform the fsck checks. Here we also map the reftable errors\nto Git' fsck errors.\n\nIntroduce a check to validate table names for a given reftable stack.\nAlso add 'badReftableTableName' as a corresponding error within Git. The\nreftable specification mentions:\n\n It suggested to use\n ${min_update_index}-${max_update_index}-${random}.ref as a naming\n convention.\n\nSo treat non-conformant file names as warnings. Introduce another check\nto check for file types, non-expected filetypes will be treated as\nerrors.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/fsck-msgids.adoc | 6 +++++\n fsck.h | 2 ++\n refs/reftable-backend.c | 58 ++++++++++++++++++++++++++++++++++++++----\n t/meson.build | 1 +\n t/t0614-reftable-fsck.sh | 55 +++++++++++++++++++++++++++++++++++++++\n 5 files changed, 117 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.adoc b/Documentation/fsck-msgids.adoc\nindex 1c912615f9..d10fe9bb35 100644\n--- a/Documentation/fsck-msgids.adoc\n+++ b/Documentation/fsck-msgids.adoc\n@@ -38,6 +38,12 @@\n `badReferentName`::\n \t(ERROR) The referent name of a symref is invalid.\n \n+`badReftableFiletype`::\n+\t(ERROR) File with unexpected type in reftable directory.\n+\n+`badReftableTableName`::\n+\t(WARN) A reftable table has an invalid name.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex 6b0db235e0..c857fcdd7c 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -34,6 +34,7 @@ enum fsck_msg_type {\n \tFUNC(BAD_PACKED_REF_HEADER, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n \tFUNC(BAD_REFERENT_NAME, ERROR) \\\n+\tFUNC(BAD_REFTABLE_FILETYPE, ERROR) \\\n \tFUNC(BAD_REF_CONTENT, ERROR) \\\n \tFUNC(BAD_REF_FILETYPE, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n@@ -73,6 +74,7 @@ enum fsck_msg_type {\n \tFUNC(UNKNOWN_TYPE, ERROR) \\\n \tFUNC(ZERO_PADDED_DATE, ERROR) \\\n \t/* warnings */ \\\n+\tFUNC(BAD_REFTABLE_TABLE_NAME, WARN) \\\n \tFUNC(EMPTY_NAME, WARN) \\\n \tFUNC(FULL_PATHNAME, WARN) \\\n \tFUNC(HAS_DOT, WARN) \\\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 2152349cb9..1a18f4bf92 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -6,6 +6,7 @@\n #include \"../config.h\"\n #include \"../dir.h\"\n #include \"../environment.h\"\n+#include \"../fsck.h\"\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n@@ -15,10 +16,11 @@\n #include \"../path.h\"\n #include \"../refs.h\"\n #include \"../reftable/reftable-basics.h\"\n-#include \"../reftable/reftable-stack.h\"\n-#include \"../reftable/reftable-record.h\"\n #include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-fsck.h\"\n #include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-stack.h\"\n #include \"../repo-settings.h\"\n #include \"../setup.h\"\n #include \"../strmap.h\"\n@@ -2707,11 +2709,57 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n-static int reftable_be_fsck(struct ref_store *ref_store UNUSED,\n-\t\t\t struct fsck_options *o UNUSED,\n+static void reftable_fsck_verbose_handler(const char *msg, void *cb_data)\n+{\n+\tstruct fsck_options *o = cb_data;\n+\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, \"%s\", msg);\n+}\n+\n+static const enum fsck_msg_id fsck_msg_id_map[] = {\n+\t[REFTABLE_FSCK_ERROR_INVALID_FILE_TYPE] = FSCK_MSG_BAD_REFTABLE_FILETYPE,\n+\t[REFTABLE_FSCK_ERROR_TABLE_NAME] = FSCK_MSG_BAD_REFTABLE_TABLE_NAME,\n+};\n+\n+static int reftable_fsck_error_handler(struct reftable_fsck_info *info,\n+\t\t\t\t void *cb_data)\n+{\n+\tstruct fsck_ref_report report = { .path = info->path };\n+\tstruct fsck_options *o = cb_data;\n+\tenum fsck_msg_id msg_id;\n+\n+\tif (info->error < 0 || info->error >= REFTABLE_FSCK_MAX_VALUE)\n+\t\tBUG(\"unknown fsck error: %d\", info->error);\n+\n+\tmsg_id = fsck_msg_id_map[info->error];\n+\n+\tif (!msg_id)\n+\t\tBUG(\"fsck_msg_id value missing for reftable error: %d\", info->error);\n+\n+\treturn fsck_report_ref(o, &report, msg_id, \"%s\", info->msg);\n+}\n+\n+static int reftable_be_fsck(struct ref_store *ref_store, struct fsck_options *o,\n \t\t\t struct worktree *wt UNUSED)\n {\n-\treturn 0;\n+\tstruct reftable_ref_store *refs;\n+\tstruct strmap_entry *entry;\n+\tstruct hashmap_iter iter;\n+\tint ret = 0;\n+\n+\trefs = reftable_be_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tret |= reftable_fsck_check(refs->main_backend.stack, reftable_fsck_error_handler,\n+\t\t\t\t reftable_fsck_verbose_handler, o);\n+\n+\tstrmap_for_each_entry(&refs->worktree_backends, &iter, entry) {\n+\t\tstruct reftable_backend *b = (struct reftable_backend *)entry->value;\n+\t\tret |= reftable_fsck_check(b->stack, reftable_fsck_error_handler,\n+\t\t\t\t\t reftable_fsck_verbose_handler, o);\n+\t}\n+\n+\treturn ret;\n }\n \n struct ref_storage_be refs_be_reftable = {\ndiff --git a/t/meson.build b/t/meson.build\nindex 7974795fe4..ec1fc0b2a1 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -146,6 +146,7 @@ integration_tests = [\n 't0611-reftable-httpd.sh',\n 't0612-reftable-jgit-compatibility.sh',\n 't0613-reftable-write-options.sh',\n+ 't0614-reftable-fsck.sh',\n 't1000-read-tree-m-3way.sh',\n 't1001-read-tree-m-2way.sh',\n 't1002-read-tree-m-u-2way.sh',\ndiff --git a/t/t0614-reftable-fsck.sh b/t/t0614-reftable-fsck.sh\nnew file mode 100755\nindex 0000000000..d4e6765b6b\n--- /dev/null\n+++ b/t/t0614-reftable-fsck.sh\n@@ -0,0 +1,55 @@\n+#!/bin/sh\n+\n+test_description='Test reftable backend consistency check'\n+\n+GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+for TABLE_NAME in \"foo-bar-e4d12d59.ref\" \\\n+\t\"0x00000000zzzz-0x00000000zzzz-e4d12d59.ref\" \\\n+\t\"0x000000000001-0x000000000002-e4d12d59.abc\" \\\n+\t\"0x000000000001-0x000000000002-e4d12d59.refabc\"; do\n+\ttest_expect_success \"table name $TABLE_NAME should be checked\" '\n+\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\tgit init repo &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit commit --allow-empty -m initial &&\n+\n+\t\t\tgit refs verify 2>err &&\n+\t\t\ttest_must_be_empty err &&\n+\n+\t\t\ttouch \".git/reftable/$TABLE_NAME\" &&\n+\n+\t\t\tgit refs verify 2>err &&\n+\t\t\tcat >expect <<-EOF &&\n+\t\t\twarning: ${TABLE_NAME}: badReftableTableName: file with invalid table name\n+\t\t\tEOF\n+\t\t\ttest_cmp expect err\n+\t\t)\n+\t'\n+done\n+\n+test_expect_success \"invalid file type should be checked\" '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\n+\t\tgit refs verify 2>err &&\n+\t\ttest_must_be_empty err &&\n+\n+\t\tmkdir \".git/reftable/foo\" &&\n+\n+\t\ttest_must_fail git refs verify 2>err &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: foo: badReftableFiletype: file with unexpected type\n+\t\tEOF\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_done\n\n-- \n2.51.0\n\n"},{"id":"526708","messageId":"xmqqo6r793iw.fsf@gitster.g","threadId":"63987","inReplyTo":"20250918-228-reftable-introduce-consistency-checks-v3-3-271af03eb34d@gmail.com","subject":"Re: [PATCH v3 3/8] reftable: check for trailing newline in 'tables.list'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-18T15:36:07Z","receivedAt":"2025-09-18T15:36:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> diff --git a/reftable/basics.h b/reftable/basics.h\n> index 7d22f96261..019dfe6d7e 100644\n> --- a/reftable/basics.h\n> +++ b/reftable/basics.h\n> @@ -167,10 +167,11 @@ void free_names(char **a);\n> \n> /*\n> * Parse a newline separated list of names. `size` is the length of the buffer,\n> - * without terminating '\\0'. Empty names are discarded. Returns a `NULL`\n> - * pointer when allocations fail.\n> + * without terminating '\\0'. Empty names are discarded.\n> + *\n> + * Errors are assigned to the `err` variable.\n> */\n> -char **parse_names(char *buf, int size);\n> +char **parse_names(char *buf, int size, int *err);\n> \n> /* compares two NULL-terminated arrays of strings. */\n> int names_equal(const char **a, const char **b);\n\nMakes sense.\n\n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index f91ce50bcd..955be1edb6 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -109,11 +109,9 @@ static int fd_read_lines(int fd, char ***namesp)\n> \t}\n> \tbuf[size] = 0;\n> \n> -\t*namesp = parse_names(buf, size);\n> -\tif (!*namesp) {\n> -\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n> +\t*namesp = parse_names(buf, size, &err);\n> +\tif (!*namesp)\n> \t\tgoto done;\n\nNice.\n\n> diff --git a/t/unit-tests/u-reftable-basics.c b/t/unit-tests/u-reftable-basics.c\n> index a0471083e7..f77ec96429 100644\n> --- a/t/unit-tests/u-reftable-basics.c\n> +++ b/t/unit-tests/u-reftable-basics.c\n> @@ -9,6 +9,7 @@ license that can be found in the LICENSE file or at\n> #include \"unit-test.h\"\n> #include \"lib-reftable.h\"\n> #include \"reftable/basics.h\"\n> +#include \"reftable/reftable-error.h\"\n> \n> struct integer_needle_lesseq_args {\n> \tint needle;\n> @@ -79,14 +80,17 @@ void test_reftable_basics__names_equal(void)\n> void test_reftable_basics__parse_names(void)\n> {\n> \tchar in1[] = \"line\\n\";\n> -\tchar in2[] = \"a\\nb\\nc\";\n> -\tchar **out = parse_names(in1, strlen(in1));\n> +\tchar in2[] = \"a\\nb\\nc\\n\";\n> +\tint err = 0;\n> +\tchar **out = parse_names(in1, strlen(in1), &err);\n> +\tcl_assert(err == 0);\n> \tcl_assert(out != NULL);\n> \tcl_assert_equal_s(out[0], \"line\");\n> \tcl_assert(!out[1]);\n> \tfree_names(out);\n> \n> -\tout = parse_names(in2, strlen(in2));\n> +\tout = parse_names(in2, strlen(in2), &err);\n> +\tcl_assert(err == 0);\n> \tcl_assert(out != NULL);\n> \tcl_assert_equal_s(out[0], \"a\");\n> \tcl_assert_equal_s(out[1], \"b\");\n\nSensible.\n\n> @@ -95,10 +99,21 @@ void test_reftable_basics__parse_names(void)\n> \tfree_names(out);\n> }\n> \n> +void test_reftable_basics__parse_names_missing_newline(void)\n> +{\n> +\tchar in1[] = \"line\\nline2\";\n> +\tint err = 0;\n> +\tchar **out = parse_names(in1, strlen(in1), &err);\n> +\tcl_assert(err == REFTABLE_FORMAT_ERROR);\n> +\tcl_assert(out == NULL);\n> +}\n\nOK.\n\n> void test_reftable_basics__parse_names_drop_empty_string(void)\n> {\n> \tchar in[] = \"a\\n\\nb\\n\";\n> -\tchar **out = parse_names(in, strlen(in));\n> +\tint err = 0;\n> +\tchar **out = parse_names(in, strlen(in), &err);\n> +\tcl_assert(err == 0);\n\nI'll drop an extra SP after == here (no need to resend only to fix\nthis).\n\n> \tcl_assert(out != NULL);\n> \tcl_assert_equal_s(out[0], \"a\");\n> \t/* simply '\\n' should be dropped as empty string */\n"},{"id":"527095","messageId":"CAOLa=ZTaLF9X+gqR1rhymfY5L=z8h0wsEQ-DW1Vi3LQe1Zd86w@mail.gmail.com","threadId":"63987","inReplyTo":"xmqqo6r793iw.fsf@gitster.g","subject":"Re: [PATCH v3 3/8] reftable: check for trailing newline in 'tables.list'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-23T15:42:39Z","receivedAt":"2025-09-23T15:42:42Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> diff --git a/reftable/basics.h b/reftable/basics.h\n>> index 7d22f96261..019dfe6d7e 100644\n>> --- a/reftable/basics.h\n>> +++ b/reftable/basics.h\n>> @@ -167,10 +167,11 @@ void free_names(char **a);\n>>\n>> /*\n>> * Parse a newline separated list of names. `size` is the length of the buffer,\n>> - * without terminating '\\0'. Empty names are discarded. Returns a `NULL`\n>> - * pointer when allocations fail.\n>> + * without terminating '\\0'. Empty names are discarded.\n>> + *\n>> + * Errors are assigned to the `err` variable.\n>> */\n>> -char **parse_names(char *buf, int size);\n>> +char **parse_names(char *buf, int size, int *err);\n>>\n>> /* compares two NULL-terminated arrays of strings. */\n>> int names_equal(const char **a, const char **b);\n>\n> Makes sense.\n>\n>> diff --git a/reftable/stack.c b/reftable/stack.c\n>> index f91ce50bcd..955be1edb6 100644\n>> --- a/reftable/stack.c\n>> +++ b/reftable/stack.c\n>> @@ -109,11 +109,9 @@ static int fd_read_lines(int fd, char ***namesp)\n>> \t}\n>> \tbuf[size] = 0;\n>>\n>> -\t*namesp = parse_names(buf, size);\n>> -\tif (!*namesp) {\n>> -\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n>> +\t*namesp = parse_names(buf, size, &err);\n>> +\tif (!*namesp)\n>> \t\tgoto done;\n>\n> Nice.\n>\n>> diff --git a/t/unit-tests/u-reftable-basics.c b/t/unit-tests/u-reftable-basics.c\n>> index a0471083e7..f77ec96429 100644\n>> --- a/t/unit-tests/u-reftable-basics.c\n>> +++ b/t/unit-tests/u-reftable-basics.c\n>> @@ -9,6 +9,7 @@ license that can be found in the LICENSE file or at\n>> #include \"unit-test.h\"\n>> #include \"lib-reftable.h\"\n>> #include \"reftable/basics.h\"\n>> +#include \"reftable/reftable-error.h\"\n>>\n>> struct integer_needle_lesseq_args {\n>> \tint needle;\n>> @@ -79,14 +80,17 @@ void test_reftable_basics__names_equal(void)\n>> void test_reftable_basics__parse_names(void)\n>> {\n>> \tchar in1[] = \"line\\n\";\n>> -\tchar in2[] = \"a\\nb\\nc\";\n>> -\tchar **out = parse_names(in1, strlen(in1));\n>> +\tchar in2[] = \"a\\nb\\nc\\n\";\n>> +\tint err = 0;\n>> +\tchar **out = parse_names(in1, strlen(in1), &err);\n>> +\tcl_assert(err == 0);\n>> \tcl_assert(out != NULL);\n>> \tcl_assert_equal_s(out[0], \"line\");\n>> \tcl_assert(!out[1]);\n>> \tfree_names(out);\n>>\n>> -\tout = parse_names(in2, strlen(in2));\n>> +\tout = parse_names(in2, strlen(in2), &err);\n>> +\tcl_assert(err == 0);\n>> \tcl_assert(out != NULL);\n>> \tcl_assert_equal_s(out[0], \"a\");\n>> \tcl_assert_equal_s(out[1], \"b\");\n>\n> Sensible.\n>\n>> @@ -95,10 +99,21 @@ void test_reftable_basics__parse_names(void)\n>> \tfree_names(out);\n>> }\n>>\n>> +void test_reftable_basics__parse_names_missing_newline(void)\n>> +{\n>> +\tchar in1[] = \"line\\nline2\";\n>> +\tint err = 0;\n>> +\tchar **out = parse_names(in1, strlen(in1), &err);\n>> +\tcl_assert(err == REFTABLE_FORMAT_ERROR);\n>> +\tcl_assert(out == NULL);\n>> +}\n>\n> OK.\n>\n>> void test_reftable_basics__parse_names_drop_empty_string(void)\n>> {\n>> \tchar in[] = \"a\\n\\nb\\n\";\n>> -\tchar **out = parse_names(in, strlen(in));\n>> +\tint err = 0;\n>> +\tchar **out = parse_names(in, strlen(in), &err);\n>> +\tcl_assert(err == 0);\n>\n> I'll drop an extra SP after == here (no need to resend only to fix\n> this).\n>\n\nAh! thanks for doing that. I'll patch it locally incase I need to\nreroll!\n\nKarthik\n"},{"id":"527175","messageId":"aNOHjdVEbCufSCPw@pks.im","threadId":"63987","inReplyTo":"20250918-228-reftable-introduce-consistency-checks-v3-3-271af03eb34d@gmail.com","subject":"Re: [PATCH v3 3/8] reftable: check for trailing newline in 'tables.list'","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-24T05:54:21Z","receivedAt":"2025-09-24T05:54:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Sep 18, 2025 at 10:11:44AM +0200, Karthik Nayak wrote:\n> diff --git a/reftable/basics.c b/reftable/basics.c\n> index 9988ebd635..75d4086769 100644\n> --- a/reftable/basics.c\n> +++ b/reftable/basics.c\n> @@ -195,7 +195,7 @@ size_t names_length(const char **names)\n> \treturn p - names;\n> }\n> \n> -char **parse_names(char *buf, int size)\n> +char **parse_names(char *buf, int size, int *err)\n> {\n> \tchar **names = NULL;\n> \tsize_t names_cap = 0;\n\nNit: Wouldn't it be more natural to return an `int` and assign the\nresult to an out-pointer?\n\n> @@ -205,30 +205,40 @@ char **parse_names(char *buf, int size)\n> \n> \twhile (p < end) {\n> \t\tchar *next = strchr(p, '\\n');\n\nNot a new issue, but it's kind of broken that we use strchr(3p) here. We\nreally should be using `memchr(p, '\\n', size - (end - p))` as the user\nprovides the size to us. And the provided size should be `size_t`.\n\n> -\t\tif (next && next < end) {\n> +\t\tif (!next) {\n> +\t\t\t*err = REFTABLE_FORMAT_ERROR;\n> +\t\t\tgoto done;\n> +\t\t} else if (next < end) {\n> \t\t\t*next = 0;\n\nCan we maybe convert this line to `*next = '\\0'` while at it? It made my\nreading hiccup a bit.\n\nPatrick\n"},{"id":"527176","messageId":"aNOHl65jYyoNXou_@pks.im","threadId":"63987","inReplyTo":"20250918-228-reftable-introduce-consistency-checks-v3-4-271af03eb34d@gmail.com","subject":"Re: [PATCH v3 4/8] reftable: ensure tables in a stack use sequential update indices","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-24T05:54:31Z","receivedAt":"2025-09-24T05:54:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Sep 18, 2025 at 10:11:45AM +0200, Karthik Nayak wrote:\n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index 955be1edb6..a458f5a4c5 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -317,6 +318,14 @@ static int reftable_stack_reload_once(struct reftable_stack *st,\n> \n> \t\tnew_tables[new_tables_len] = table;\n> \t\tnew_tables_len++;\n> +\n> +\t\t/* table's update indices must be sequential */\n\nLet's make this a full sentence starting with an upper-case letter and a\nperiod.\n\n> +\t\tif (prev_table && (prev_table->max_update_index != table->min_update_index - 1)) {\n\nI wonder whether this check is too strict. It _must_ be true that the\nnew table's minimum update index is greater than the previous table's\nmaximum update index. But in theory, there is no reason why there cannot\nbe a gap between those.\n\nThe reason why this makes me a bit uneasy is stack compaction. Say we\nhave three different tables:\n\n - A base table with record r1 with update index 1.\n - A second table with record r2 with update index 2.\n - A third table with a deletion record d(r2) and a new record r3 with\n update index 3.\n\nNow if we compact the second and the third table, the compaction will\nrealize that r2 is deleted and thus no longer needs to be part of the\ncompacted table. So the new state is:\n\n - A base table with record r1 and update index r1.\n - The compacted table with record r3 with update index 3.\n\nI'm not too certain how the minimum update index of that second table\nwould be encoded in the header. In theory, both minimum and maximum\nupdate index of that table could truthfully be 3, and the result would\nstill be both valid and sensible. The new check you introduce would\ntrigger though, as there now is a gap between those two tables.\n\nSo I think we should loosen that condition to ensure that we have proper\nordering of update indices, but not a gapless order.\n\nPatrick\n"},{"id":"527177","messageId":"aNOHqEq5qxXrOCX7@pks.im","threadId":"63987","inReplyTo":"20250918-228-reftable-introduce-consistency-checks-v3-7-271af03eb34d@gmail.com","subject":"Re: [PATCH v3 7/8] reftable: add code to facilitate consistency checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-24T05:54:48Z","receivedAt":"2025-09-24T05:54:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Sep 18, 2025 at 10:11:48AM +0200, Karthik Nayak wrote:\n> diff --git a/reftable/fsck.c b/reftable/fsck.c\n> new file mode 100644\n> index 0000000000..785e4b43e8\n> --- /dev/null\n> +++ b/reftable/fsck.c\n> @@ -0,0 +1,112 @@\n> +#include \"basics.h\"\n> +#include \"reftable-fsck.h\"\n> +#include \"stack.h\"\n> +\n> +static bool valid_table_name(const char *name, uint64_t *min_update_index,\n> +\t\t\t uint64_t *max_update_index)\n> +{\n> +\tconst char *ptr = name;\n> +\tchar *endptr;\n> +\n> +\t/* strtoull doesn't set errno on success */\n> +\terrno = 0;\n> +\n> +\t*min_update_index = strtoull(ptr, &endptr, 16);\n> +\tif (errno == EINVAL)\n> +\t\treturn false;\n\nstrtoull may also return ERANGE. In general, shouldn't we abort whenever\nerrno is non-zero here?\n\n> +\tptr = endptr;\n> +\n> +\tif (strncmp(ptr, \"-\", 1))\n> +\t\treturn false;\n\nBetter:\n\n if (*ptr != '-')\n return false;\n\n> +\tptr++;\n> +\n> +\t*max_update_index = strtoull(ptr, &endptr, 16);\n> +\tif (errno == EINVAL)\n> +\t\treturn false;\n> +\tptr = endptr;\n> +\n> +\tif (*ptr != '-')\n> +\t\treturn false;\n> +\tptr++;\n> +\n> +\tstrtoul(ptr, &endptr, 16);\n> +\tif (errno == EINVAL)\n> +\t\treturn false;\n> +\tptr = endptr;\n> +\n> +\tif (strcmp(ptr, \".ref\") && strcmp(ptr, \".log\"))\n> +\t\treturn false;\n\nYup, makes sense. We don't do so ourselves, but in theory it is possible\nfor tables to have a \".log\" suffix. If so, they are expected to only\ncontain reflog records.\n\n> +\treturn true;\n> +}\n> +\n> +static int stack_check_all_files_in_dir(struct reftable_stack *stack,\n> +\t\t\t\t\treftable_fsck_report_fn report_fn,\n> +\t\t\t\t\tvoid *cb_data)\n> +{\n> +\tDIR *dir = opendir(stack->reftable_dir);\n\nI think it would make sense to move this function call close to the\nconditional.\n\n> +\tstruct reftable_fsck_info info;\n> +\tstruct dirent *d = NULL;\n> +\tuint64_t min, max;\n> +\tint err = 0;\n> +\n> +\tif (!dir)\n> +\t\treturn 0;\n> +\n> +\twhile ((d = readdir(dir))) {\n> +\t\tif (!strcmp(d->d_name, \"tables.list\"))\n> +\t\t\tcontinue;\n> +\n> +\t\tif ((d->d_name[0] == '.' &&\n> +\t\t (d->d_name[1] == '\\0' ||\n> +\t\t (d->d_name[1] == '.' && d->d_name[2] == '\\0'))))\n> +\t\t\tcontinue;\n> +\n> +\t\tif (d->d_type == DT_REG) {\n> +\t\t\tif (!valid_table_name(d->d_name, &min, &max)) {\n> +\t\t\t\tinfo.error = REFTABLE_FSCK_ERROR_TABLE_NAME;\n> +\t\t\t\tinfo.msg = \"file with invalid table name\";\n> +\t\t\t\tinfo.path = d->d_name;\n> +\n> +\t\t\t\terr |= report_fn(&info, cb_data);\n> +\t\t\t}\n\nOne problem with this is that this is racy with concurrent writers. We\ndon't recognize the \"tables.list.lock\" file, and neither do we recognize\n\"0x*-0x*.{ref,log}.temp.XXXXXX\"-style files.\n\nWould it be a better approach be to instead go through table names as\nloaded by the stack? The reftable code already knows to prune unknown\nfiles anyway, so I don't think we should scan for any other files.\n\n> +\t\t} else {\n> +\t\t\tinfo.error = REFTABLE_FSCK_ERROR_INVALID_FILE_TYPE;\n> +\t\t\tinfo.msg = \"file with unexpected type\";\n> +\t\t\tinfo.path = d->d_name;\n> +\n> +\t\t\terr |= report_fn(&info, cb_data);\n> +\t\t}\n> +\t}\n> +\n> +\tclosedir(dir);\n> +\treturn err;\n> +}\n> +\n> +static int stack_checks(struct reftable_stack *stack,\n> +\t\t\treftable_fsck_report_fn report_fn,\n> +\t\t\tvoid *cb_data)\n> +{\n> +\tstruct reftable_buf msg = REFTABLE_BUF_INIT;\n> +\tchar **names = NULL;\n\nThis variable is unused.\n\n> +\tint err = 0;\n> +\n> +\tif (stack == NULL)\n> +\t\tgoto out;\n\nWhy should someone ever pass a `NULL` stack?\n\n> +\terr |= stack_check_all_files_in_dir(stack, report_fn, cb_data);\n> +\n> +out:\n> +\tfree_names(names);\n> +\treftable_buf_release(&msg);\n> +\treturn err;\n> +}\n> +\n> +int reftable_fsck_check(struct reftable_stack *stack,\n> +\t\t\treftable_fsck_report_fn report_fn,\n> +\t\t\treftable_fsck_verbose_fn verbose_fn,\n> +\t\t\tvoid *cb_data)\n> +{\n> +\tverbose_fn(\"Checking reftable: stack checks\", cb_data);\n> +\treturn stack_checks(stack, report_fn, cb_data);\n\nNit: having this extra function call to `stack_checks()` feels a bit\nweird as it could just as well be inlined. Is this preparing for a\nfuture change?\n\n> +}\n> diff --git a/reftable/reftable-fsck.h b/reftable/reftable-fsck.h\n> new file mode 100644\n> index 0000000000..5e13ac9f02\n> --- /dev/null\n> +++ b/reftable/reftable-fsck.h\n> @@ -0,0 +1,42 @@\n> +#ifndef REFTABLE_FSCK_H\n> +#define REFTABLE_FSCK_H\n> +\n> +#include \"reftable-stack.h\"\n> +\n> +enum reftable_fsck_error {\n> +\t/* Non regular file in the reftable directory */\n> +\tREFTABLE_FSCK_ERROR_INVALID_FILE_TYPE = 0,\n> +\t/* Invalid table name */\n> +\tREFTABLE_FSCK_ERROR_TABLE_NAME,\n> +\t/* Used for bounds checking, must be last */\n> +\tREFTABLE_FSCK_MAX_VALUE\n\nLet's add a trailing comma here.\n\nPatrick\n"},{"id":"527178","messageId":"aNOHr7lm4WXUyHa5@pks.im","threadId":"63987","inReplyTo":"20250918-228-reftable-introduce-consistency-checks-v3-8-271af03eb34d@gmail.com","subject":"Re: [PATCH v3 8/8] refs/reftable: add fsck check for checking the table name","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-24T05:54:55Z","receivedAt":"2025-09-24T05:55:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Sep 18, 2025 at 10:11:49AM +0200, Karthik Nayak wrote:\n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index 2152349cb9..1a18f4bf92 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -2707,11 +2709,57 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n> \treturn ret;\n> }\n> \n> -static int reftable_be_fsck(struct ref_store *ref_store UNUSED,\n> -\t\t\t struct fsck_options *o UNUSED,\n> +static void reftable_fsck_verbose_handler(const char *msg, void *cb_data)\n> +{\n> +\tstruct fsck_options *o = cb_data;\n> +\n> +\tif (o->verbose)\n> +\t\tfprintf_ln(stderr, \"%s\", msg);\n> +}\n> +\n> +static const enum fsck_msg_id fsck_msg_id_map[] = {\n> +\t[REFTABLE_FSCK_ERROR_INVALID_FILE_TYPE] = FSCK_MSG_BAD_REFTABLE_FILETYPE,\n> +\t[REFTABLE_FSCK_ERROR_TABLE_NAME] = FSCK_MSG_BAD_REFTABLE_TABLE_NAME,\n> +};\n> +\n> +static int reftable_fsck_error_handler(struct reftable_fsck_info *info,\n> +\t\t\t\t void *cb_data)\n> +{\n> +\tstruct fsck_ref_report report = { .path = info->path };\n> +\tstruct fsck_options *o = cb_data;\n> +\tenum fsck_msg_id msg_id;\n> +\n> +\tif (info->error < 0 || info->error >= REFTABLE_FSCK_MAX_VALUE)\n> +\t\tBUG(\"unknown fsck error: %d\", info->error);\n\n`info->error` is an enum, and whether or not it is signed is an\nimplementation detail of the platform. But I wonder whether this check\nmay cause some platforms to warn about an impossible condition.\n\n> +\n> +\tmsg_id = fsck_msg_id_map[info->error];\n> +\n> +\tif (!msg_id)\n> +\t\tBUG(\"fsck_msg_id value missing for reftable error: %d\", info->error);\n\nYup, makes sense.\n\nPatrick\n"},{"id":"527186","messageId":"b6d03748-8171-4df3-83fd-7f4025d9eb6c@app.fastmail.com","threadId":"63987","inReplyTo":"20250918-228-reftable-introduce-consistency-checks-v3-3-271af03eb34d@gmail.com","subject":"Re: [PATCH v3 3/8] reftable: check for trailing newline in 'tables.list'","fromName":"Kristoffer Haugsbakk","fromEmail":"kristofferhaugsbakk@fastmail.com","sentAt":"2025-09-24T07:24:57Z","receivedAt":"2025-09-24T07:25:18Z","isPatch":true,"sender":{"key":"kristofferhaugsbakk@fastmail.com","avatar":null},"body":"On Thu, Sep 18, 2025, at 10:11, Karthik Nayak wrote:\n> In the reftable format, the 'tables.list' file contains a newline\n> separated list of tables. While we parse this file, we do not check or\n> care about trailing newlines. Tighten the parser in `parse_names()` to\n> return an appropriate error if there is no trailing newline.\n\nNit:[1] newline-separated + requiring a trailing newline sounds like it\nreally equals: newline-terminated list. Is this moving from\neffectively using newline-separated to a newline-terminated format?\n\n† 1: Since others have commented anyway\n\n>\n> This requires modification to `parse_names()` to accept a third argument\n> which will hold the error value.\n>\n> Signed-off-by: Karthik Nayak \n> ---\n"},{"id":"527198","messageId":"CAOLa=ZQMDjpMLeyHxeePY3VQjD1GhotXA6-GDhTNY_BDu4zSVQ@mail.gmail.com","threadId":"63987","inReplyTo":"aNOHjdVEbCufSCPw@pks.im","subject":"Re: [PATCH v3 3/8] reftable: check for trailing newline in 'tables.list'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-24T10:02:01Z","receivedAt":"2025-09-24T10:02:03Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Thu, Sep 18, 2025 at 10:11:44AM +0200, Karthik Nayak wrote:\n>> diff --git a/reftable/basics.c b/reftable/basics.c\n>> index 9988ebd635..75d4086769 100644\n>> --- a/reftable/basics.c\n>> +++ b/reftable/basics.c\n>> @@ -195,7 +195,7 @@ size_t names_length(const char **names)\n>> \treturn p - names;\n>> }\n>>\n>> -char **parse_names(char *buf, int size)\n>> +char **parse_names(char *buf, int size, int *err)\n>> {\n>> \tchar **names = NULL;\n>> \tsize_t names_cap = 0;\n>\n> Nit: Wouldn't it be more natural to return an `int` and assign the\n> result to an out-pointer?\n>\n\nI thought about that too, I couldn't find enough consistency or reason to\nwarrant one over the other. So I picked the one with the least change.\nLet me change it.\n\n>> @@ -205,30 +205,40 @@ char **parse_names(char *buf, int size)\n>>\n>> \twhile (p < end) {\n>> \t\tchar *next = strchr(p, '\\n');\n>\n> Not a new issue, but it's kind of broken that we use strchr(3p) here. We\n> really should be using `memchr(p, '\\n', size - (end - p))` as the user\n> provides the size to us. And the provided size should be `size_t`.\n>\n\nI think that's fair. But I'll avoid making this change now, I've already\nadded a few commits which are mostly tangential.\n\n>> -\t\tif (next && next < end) {\n>> +\t\tif (!next) {\n>> +\t\t\t*err = REFTABLE_FORMAT_ERROR;\n>> +\t\t\tgoto done;\n>> +\t\t} else if (next < end) {\n>> \t\t\t*next = 0;\n>\n> Can we maybe convert this line to `*next = '\\0'` while at it? It made my\n> reading hiccup a bit.\n>\n\nYeah, I could definitely add this in.\n\n> Patrick\n"},{"id":"527204","messageId":"CAOLa=ZSkx--7A6zs3RK3Noa=bBZ8mSLMVWBYr7tUFXQkrstgMw@mail.gmail.com","threadId":"63987","inReplyTo":"b6d03748-8171-4df3-83fd-7f4025d9eb6c@app.fastmail.com","subject":"Re: [PATCH v3 3/8] reftable: check for trailing newline in 'tables.list'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-24T11:06:49Z","receivedAt":"2025-09-24T11:06:51Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"\"Kristoffer Haugsbakk\" writes:\n\n> On Thu, Sep 18, 2025, at 10:11, Karthik Nayak wrote:\n>> In the reftable format, the 'tables.list' file contains a newline\n>> separated list of tables. While we parse this file, we do not check or\n>> care about trailing newlines. Tighten the parser in `parse_names()` to\n>> return an appropriate error if there is no trailing newline.\n>\n> Nit:[1] newline-separated + requiring a trailing newline sounds like it\n> really equals: newline-terminated list. Is this moving from\n> effectively using newline-separated to a newline-terminated format?\n>\n> † 1: Since others have commented anyway\n>\n\nI see the confusion, it is a newline-separated list, but we don't\ncheck/care for the last newline. We don't require a separate terminating\nnewline. Let me amend the commit message to make this clearer.\n\n>>\n>> This requires modification to `parse_names()` to accept a third argument\n>> which will hold the error value.\n>>\n>> Signed-off-by: Karthik Nayak \n>> ---\n"},{"id":"527206","messageId":"CAOLa=ZTf7KL23+=Fggfg=4LXt1Dsd6nRCFg3q_Dhuom2Bk+L7A@mail.gmail.com","threadId":"63987","inReplyTo":"aNOHl65jYyoNXou_@pks.im","subject":"Re: [PATCH v3 4/8] reftable: ensure tables in a stack use sequential update indices","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-24T11:20:07Z","receivedAt":"2025-09-24T11:20:10Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Thu, Sep 18, 2025 at 10:11:45AM +0200, Karthik Nayak wrote:\n>> diff --git a/reftable/stack.c b/reftable/stack.c\n>> index 955be1edb6..a458f5a4c5 100644\n>> --- a/reftable/stack.c\n>> +++ b/reftable/stack.c\n>> @@ -317,6 +318,14 @@ static int reftable_stack_reload_once(struct reftable_stack *st,\n>>\n>> \t\tnew_tables[new_tables_len] = table;\n>> \t\tnew_tables_len++;\n>> +\n>> +\t\t/* table's update indices must be sequential */\n>\n> Let's make this a full sentence starting with an upper-case letter and a\n> period.\n>\n>> +\t\tif (prev_table && (prev_table->max_update_index != table->min_update_index - 1)) {\n>\n> I wonder whether this check is too strict. It _must_ be true that the\n> new table's minimum update index is greater than the previous table's\n> maximum update index. But in theory, there is no reason why there cannot\n> be a gap between those.\n>\n> The reason why this makes me a bit uneasy is stack compaction. Say we\n> have three different tables:\n>\n> - A base table with record r1 with update index 1.\n> - A second table with record r2 with update index 2.\n> - A third table with a deletion record d(r2) and a new record r3 with\n> update index 3.\n>\n> Now if we compact the second and the third table, the compaction will\n> realize that r2 is deleted and thus no longer needs to be part of the\n> compacted table. So the new state is:\n>\n> - A base table with record r1 and update index r1.\n> - The compacted table with record r3 with update index 3.\n>\n\nThat's a good counter example. I didn't know this was possible with the\nreftable format. From 'reftable/stack.c: stack_compact_locked()', we\nuse the min,max index from the first, last table being compacted for the\ntable name.\n\n err = format_name(&next_name,\nreftable_table_min_update_index(st->tables[first]),\n\t\t\t reftable_table_max_update_index(st->tables[last]));\n\nwe also set the writer's limit in 'reftable/stack.c:\nstack_write_compact()' similarly, which sets the min,max index for the\nwriter:\n\n err = reftable_writer_set_limits(wr, st->tables[first]->min_update_index,\n\t\t\t\t\t st->tables[last]->max_update_index);\n\n\n\n> I'm not too certain how the minimum update index of that second table\n> would be encoded in the header. In theory, both minimum and maximum\n> update index of that table could truthfully be 3, and the result would\n> still be both valid and sensible. The new check you introduce would\n> trigger though, as there now is a gap between those two tables.\n>\n> So I think we should loosen that condition to ensure that we have proper\n> ordering of update indices, but not a gapless order.\n>\n> Patrick\n\nSo currently it does seem like our implementation, still uses the first\nand last table's indices to set the min,max index of the new table.\n\nHowever, I think your point holds. I do think eventually we could\noptimize this to ensure that we do something like you described.\n\nI will make changes accordingly.\n"},{"id":"527243","messageId":"xmqqikh7lob7.fsf@gitster.g","threadId":"63987","inReplyTo":"CAOLa=ZTf7KL23+=Fggfg=4LXt1Dsd6nRCFg3q_Dhuom2Bk+L7A@mail.gmail.com","subject":"Re: [PATCH v3 4/8] reftable: ensure tables in a stack use sequential update indices","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-24T18:04:28Z","receivedAt":"2025-09-24T18:04:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Patrick Steinhardt writes:\n>\n>> On Thu, Sep 18, 2025 at 10:11:45AM +0200, Karthik Nayak wrote:\n>>> diff --git a/reftable/stack.c b/reftable/stack.c\n>>> index 955be1edb6..a458f5a4c5 100644\n>>> --- a/reftable/stack.c\n>>> +++ b/reftable/stack.c\n>>> @@ -317,6 +318,14 @@ static int reftable_stack_reload_once(struct reftable_stack *st,\n>>>\n>>> \t\tnew_tables[new_tables_len] = table;\n>>> \t\tnew_tables_len++;\n>>> +\n>>> +\t\t/* table's update indices must be sequential */\n>>\n>> Let's make this a full sentence starting with an upper-case letter and a\n>> period.\n>>\n>>> +\t\tif (prev_table && (prev_table->max_update_index != table->min_update_index - 1)) {\n>>\n>> I wonder whether this check is too strict. It _must_ be true that the\n>> new table's minimum update index is greater than the previous table's\n>> maximum update index. But in theory, there is no reason why there cannot\n>> be a gap between those.\n>>\n>> The reason why this makes me a bit uneasy is stack compaction. Say we\n>> have three different tables:\n>>\n>> - A base table with record r1 with update index 1.\n>> - A second table with record r2 with update index 2.\n>> - A third table with a deletion record d(r2) and a new record r3 with\n>> update index 3.\n>>\n>> Now if we compact the second and the third table, the compaction will\n>> realize that r2 is deleted and thus no longer needs to be part of the\n>> compacted table. So the new state is:\n>>\n>> - A base table with record r1 and update index r1.\n>> - The compacted table with record r3 with update index 3.\n> ...\n> However, I think your point holds. I do think eventually we could\n> optimize this to ensure that we do something like you described.\n>\n> I will make changes accordingly.\n\nIf you allow gaps in the indices, it is a bit confusing to call them\n\"sequential\"; \"monotonically increasing\" is less confusing and it\nconveys the author's intention to allow gaps clear (otherwise the\nauthor wouldn't be using such an awkward two-word phrase instead of\n\"sequencial\").\n\n\n\n"},{"id":"527247","messageId":"CAOLa=ZQ641MncC9ACm9jfjx0WtQ+nK2shtyucQOxd08LDXDzAw@mail.gmail.com","threadId":"63987","inReplyTo":"aNOHqEq5qxXrOCX7@pks.im","subject":"Re: [PATCH v3 7/8] reftable: add code to facilitate consistency checks","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-24T18:40:31Z","receivedAt":"2025-09-24T18:40:34Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Thu, Sep 18, 2025 at 10:11:48AM +0200, Karthik Nayak wrote:\n>> diff --git a/reftable/fsck.c b/reftable/fsck.c\n>> new file mode 100644\n>> index 0000000000..785e4b43e8\n>> --- /dev/null\n>> +++ b/reftable/fsck.c\n>> @@ -0,0 +1,112 @@\n>> +#include \"basics.h\"\n>> +#include \"reftable-fsck.h\"\n>> +#include \"stack.h\"\n>> +\n>> +static bool valid_table_name(const char *name, uint64_t *min_update_index,\n>> +\t\t\t uint64_t *max_update_index)\n>> +{\n>> +\tconst char *ptr = name;\n>> +\tchar *endptr;\n>> +\n>> +\t/* strtoull doesn't set errno on success */\n>> +\terrno = 0;\n>> +\n>> +\t*min_update_index = strtoull(ptr, &endptr, 16);\n>> +\tif (errno == EINVAL)\n>> +\t\treturn false;\n>\n> strtoull may also return ERANGE. In general, shouldn't we abort whenever\n> errno is non-zero here?\n>\n\nYeah, that would be much better. will change.\n\n>> +\tptr = endptr;\n>> +\n>> +\tif (strncmp(ptr, \"-\", 1))\n>> +\t\treturn false;\n>\n> Better:\n>\n> if (*ptr != '-')\n> return false;\n>\n\nI did use that below. I think I missed changing this, will do.\n\n>> +\tptr++;\n>> +\n>> +\t*max_update_index = strtoull(ptr, &endptr, 16);\n>> +\tif (errno == EINVAL)\n>> +\t\treturn false;\n>> +\tptr = endptr;\n>> +\n>> +\tif (*ptr != '-')\n>> +\t\treturn false;\n>> +\tptr++;\n>> +\n>> +\tstrtoul(ptr, &endptr, 16);\n>> +\tif (errno == EINVAL)\n>> +\t\treturn false;\n>> +\tptr = endptr;\n>> +\n>> +\tif (strcmp(ptr, \".ref\") && strcmp(ptr, \".log\"))\n>> +\t\treturn false;\n>\n> Yup, makes sense. We don't do so ourselves, but in theory it is possible\n> for tables to have a \".log\" suffix. If so, they are expected to only\n> contain reflog records.\n>\n\nYeah, I missed this in the previous iteration, but realized while\nreading the spec that this could be possible.\n\n>> +\treturn true;\n>> +}\n>> +\n>> +static int stack_check_all_files_in_dir(struct reftable_stack *stack,\n>> +\t\t\t\t\treftable_fsck_report_fn report_fn,\n>> +\t\t\t\t\tvoid *cb_data)\n>> +{\n>> +\tDIR *dir = opendir(stack->reftable_dir);\n>\n> I think it would make sense to move this function call close to the\n> conditional.\n>\n\nFair enough, will move.\n\n>> +\tstruct reftable_fsck_info info;\n>> +\tstruct dirent *d = NULL;\n>> +\tuint64_t min, max;\n>> +\tint err = 0;\n>> +\n>> +\tif (!dir)\n>> +\t\treturn 0;\n>> +\n>> +\twhile ((d = readdir(dir))) {\n>> +\t\tif (!strcmp(d->d_name, \"tables.list\"))\n>> +\t\t\tcontinue;\n>> +\n>> +\t\tif ((d->d_name[0] == '.' &&\n>> +\t\t (d->d_name[1] == '\\0' ||\n>> +\t\t (d->d_name[1] == '.' && d->d_name[2] == '\\0'))))\n>> +\t\t\tcontinue;\n>> +\n>> +\t\tif (d->d_type == DT_REG) {\n>> +\t\t\tif (!valid_table_name(d->d_name, &min, &max)) {\n>> +\t\t\t\tinfo.error = REFTABLE_FSCK_ERROR_TABLE_NAME;\n>> +\t\t\t\tinfo.msg = \"file with invalid table name\";\n>> +\t\t\t\tinfo.path = d->d_name;\n>> +\n>> +\t\t\t\terr |= report_fn(&info, cb_data);\n>> +\t\t\t}\n>\n> One problem with this is that this is racy with concurrent writers. We\n> don't recognize the \"tables.list.lock\" file, and neither do we recognize\n> \"0x*-0x*.{ref,log}.temp.XXXXXX\"-style files.\n>\n> Would it be a better approach be to instead go through table names as\n> loaded by the stack? The reftable code already knows to prune unknown\n> files anyway, so I don't think we should scan for any other files.\n>\n\nI actually had a more structured code here, where the idea was:\n\n- For each stack\n - Run stack level checks\n - For each table in stack\n - Run table level checks\n - For each block in table\n - Run block level checks\n - For each ref / log\n - Run ref / log level checks\n\nBut we move some of my tests to be runtime checks, leaving this as the\nonly check remaining. We could still do the first level of what I\nmentioned above. The only reason I didn't was because we wanted to check\nall files in the stack dir. But I think this is much better, having\nunknown files in the reftable directory doesn't affect the repository in\nany way. So I would argue perhaps that we shouldn't even care about it.\n\n>> +\t\t} else {\n>> +\t\t\tinfo.error = REFTABLE_FSCK_ERROR_INVALID_FILE_TYPE;\n>> +\t\t\tinfo.msg = \"file with unexpected type\";\n>> +\t\t\tinfo.path = d->d_name;\n>> +\n>> +\t\t\terr |= report_fn(&info, cb_data);\n>> +\t\t}\n>> +\t}\n>> +\n>> +\tclosedir(dir);\n>> +\treturn err;\n>> +}\n>> +\n>> +static int stack_checks(struct reftable_stack *stack,\n>> +\t\t\treftable_fsck_report_fn report_fn,\n>> +\t\t\tvoid *cb_data)\n>> +{\n>> +\tstruct reftable_buf msg = REFTABLE_BUF_INIT;\n>> +\tchar **names = NULL;\n>\n> This variable is unused.\n>\n\nLeftover code, will cleanup.\n\n>> +\tint err = 0;\n>> +\n>> +\tif (stack == NULL)\n>> +\t\tgoto out;\n>\n> Why should someone ever pass a `NULL` stack?\n>\n\nThis should be safe to remove.\n\n>> +\terr |= stack_check_all_files_in_dir(stack, report_fn, cb_data);\n>> +\n>> +out:\n>> +\tfree_names(names);\n>> +\treftable_buf_release(&msg);\n>> +\treturn err;\n>> +}\n>> +\n>> +int reftable_fsck_check(struct reftable_stack *stack,\n>> +\t\t\treftable_fsck_report_fn report_fn,\n>> +\t\t\treftable_fsck_verbose_fn verbose_fn,\n>> +\t\t\tvoid *cb_data)\n>> +{\n>> +\tverbose_fn(\"Checking reftable: stack checks\", cb_data);\n>> +\treturn stack_checks(stack, report_fn, cb_data);\n>\n> Nit: having this extra function call to `stack_checks()` feels a bit\n> weird as it could just as well be inlined. Is this preparing for a\n> future change?\n\nYeah, mostly the idea was to break things up into layers as I mentioned\nabove. Let's make it simpler for now and we can make it nicer when we\nget around adding more checks.\n\n>\n>> +}\n>> diff --git a/reftable/reftable-fsck.h b/reftable/reftable-fsck.h\n>> new file mode 100644\n>> index 0000000000..5e13ac9f02\n>> --- /dev/null\n>> +++ b/reftable/reftable-fsck.h\n>> @@ -0,0 +1,42 @@\n>> +#ifndef REFTABLE_FSCK_H\n>> +#define REFTABLE_FSCK_H\n>> +\n>> +#include \"reftable-stack.h\"\n>> +\n>> +enum reftable_fsck_error {\n>> +\t/* Non regular file in the reftable directory */\n>> +\tREFTABLE_FSCK_ERROR_INVALID_FILE_TYPE = 0,\n>> +\t/* Invalid table name */\n>> +\tREFTABLE_FSCK_ERROR_TABLE_NAME,\n>> +\t/* Used for bounds checking, must be last */\n>> +\tREFTABLE_FSCK_MAX_VALUE\n>\n> Let's add a trailing comma here.\n>\n> Patrick\n\nWill do.\n"},{"id":"527248","messageId":"CAOLa=ZS+CGh6kuT87xoR_GnxJkBABk3g0CtOqU9wfnCVS=AcBw@mail.gmail.com","threadId":"63987","inReplyTo":"aNOHr7lm4WXUyHa5@pks.im","subject":"Re: [PATCH v3 8/8] refs/reftable: add fsck check for checking the table name","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-24T18:44:06Z","receivedAt":"2025-09-24T18:44:09Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n>> +static int reftable_fsck_error_handler(struct reftable_fsck_info *info,\n>> +\t\t\t\t void *cb_data)\n>> +{\n>> +\tstruct fsck_ref_report report = { .path = info->path };\n>> +\tstruct fsck_options *o = cb_data;\n>> +\tenum fsck_msg_id msg_id;\n>> +\n>> +\tif (info->error < 0 || info->error >= REFTABLE_FSCK_MAX_VALUE)\n>> +\t\tBUG(\"unknown fsck error: %d\", info->error);\n>\n> `info->error` is an enum, and whether or not it is signed is an\n> implementation detail of the platform. But I wonder whether this check\n> may cause some platforms to warn about an impossible condition.\n>\n\nI didn't really think of that. I guess typecasting it to an int would be\nthe best way forward here.\n\n>> +\n>> +\tmsg_id = fsck_msg_id_map[info->error];\n>> +\n>> +\tif (!msg_id)\n>> +\t\tBUG(\"fsck_msg_id value missing for reftable error: %d\", info->error);\n>\n> Yup, makes sense.\n>\n> Patrick\n\nThanks for the review.\n"},{"id":"527254","messageId":"CAOLa=ZQxM_iRMLoKt2ZoTSngkQ3EzXiJJ9hiJxRqHSaq2o0Nfw@mail.gmail.com","threadId":"63987","inReplyTo":"xmqqikh7lob7.fsf@gitster.g","subject":"Re: [PATCH v3 4/8] reftable: ensure tables in a stack use sequential update indices","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-24T20:13:51Z","receivedAt":"2025-09-24T20:13:53Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n\n>>> Now if we compact the second and the third table, the compaction will\n>>> realize that r2 is deleted and thus no longer needs to be part of the\n>>> compacted table. So the new state is:\n>>>\n>>> - A base table with record r1 and update index r1.\n>>> - The compacted table with record r3 with update index 3.\n>> ...\n>> However, I think your point holds. I do think eventually we could\n>> optimize this to ensure that we do something like you described.\n>>\n>> I will make changes accordingly.\n>\n> If you allow gaps in the indices, it is a bit confusing to call them\n> \"sequential\"; \"monotonically increasing\" is less confusing and it\n> conveys the author's intention to allow gaps clear (otherwise the\n> author wouldn't be using such an awkward two-word phrase instead of\n> \"sequencial\").\n\nWouldn't 'monotonically increasing' suggest that\nprev_table.max_update_index can be equal to cur_table.min_update_index?\nI have locally changed it to 'ascending order' for similar reasons.\n"},{"id":"527285","messageId":"aNTdTu08kKugIr7j@pks.im","threadId":"63987","inReplyTo":"CAOLa=ZQxM_iRMLoKt2ZoTSngkQ3EzXiJJ9hiJxRqHSaq2o0Nfw@mail.gmail.com","subject":"Re: [PATCH v3 4/8] reftable: ensure tables in a stack use sequential update indices","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-25T06:12:30Z","receivedAt":"2025-09-25T06:12:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Sep 24, 2025 at 01:13:51PM -0700, Karthik Nayak wrote:\n> Junio C Hamano writes:\n> \n> \n> >>> Now if we compact the second and the third table, the compaction will\n> >>> realize that r2 is deleted and thus no longer needs to be part of the\n> >>> compacted table. So the new state is:\n> >>>\n> >>> - A base table with record r1 and update index r1.\n> >>> - The compacted table with record r3 with update index 3.\n> >> ...\n> >> However, I think your point holds. I do think eventually we could\n> >> optimize this to ensure that we do something like you described.\n> >>\n> >> I will make changes accordingly.\n> >\n> > If you allow gaps in the indices, it is a bit confusing to call them\n> > \"sequential\"; \"monotonically increasing\" is less confusing and it\n> > conveys the author's intention to allow gaps clear (otherwise the\n> > author wouldn't be using such an awkward two-word phrase instead of\n> > \"sequencial\").\n> \n> Wouldn't 'monotonically increasing' suggest that\n> prev_table.max_update_index can be equal to cur_table.min_update_index?\n> I have locally changed it to 'ascending order' for similar reasons.\n\nI guess the correct phrase here is \"strictly monotonically increasing\".\n\nPatrick\n"},{"id":"527286","messageId":"aNTdwzUMlubjcppb@pks.im","threadId":"63987","inReplyTo":"CAOLa=ZQ641MncC9ACm9jfjx0WtQ+nK2shtyucQOxd08LDXDzAw@mail.gmail.com","subject":"Re: [PATCH v3 7/8] reftable: add code to facilitate consistency checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-25T06:14:27Z","receivedAt":"2025-09-25T06:14:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Sep 24, 2025 at 11:40:31AM -0700, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> > On Thu, Sep 18, 2025 at 10:11:48AM +0200, Karthik Nayak wrote:\n> >> diff --git a/reftable/fsck.c b/reftable/fsck.c\n> >> new file mode 100644\n> >> index 0000000000..785e4b43e8\n> >> --- /dev/null\n> >> +++ b/reftable/fsck.c\n[snip]\n> >> +\tstruct reftable_fsck_info info;\n> >> +\tstruct dirent *d = NULL;\n> >> +\tuint64_t min, max;\n> >> +\tint err = 0;\n> >> +\n> >> +\tif (!dir)\n> >> +\t\treturn 0;\n> >> +\n> >> +\twhile ((d = readdir(dir))) {\n> >> +\t\tif (!strcmp(d->d_name, \"tables.list\"))\n> >> +\t\t\tcontinue;\n> >> +\n> >> +\t\tif ((d->d_name[0] == '.' &&\n> >> +\t\t (d->d_name[1] == '\\0' ||\n> >> +\t\t (d->d_name[1] == '.' && d->d_name[2] == '\\0'))))\n> >> +\t\t\tcontinue;\n> >> +\n> >> +\t\tif (d->d_type == DT_REG) {\n> >> +\t\t\tif (!valid_table_name(d->d_name, &min, &max)) {\n> >> +\t\t\t\tinfo.error = REFTABLE_FSCK_ERROR_TABLE_NAME;\n> >> +\t\t\t\tinfo.msg = \"file with invalid table name\";\n> >> +\t\t\t\tinfo.path = d->d_name;\n> >> +\n> >> +\t\t\t\terr |= report_fn(&info, cb_data);\n> >> +\t\t\t}\n> >\n> > One problem with this is that this is racy with concurrent writers. We\n> > don't recognize the \"tables.list.lock\" file, and neither do we recognize\n> > \"0x*-0x*.{ref,log}.temp.XXXXXX\"-style files.\n> >\n> > Would it be a better approach be to instead go through table names as\n> > loaded by the stack? The reftable code already knows to prune unknown\n> > files anyway, so I don't think we should scan for any other files.\n> >\n> \n> I actually had a more structured code here, where the idea was:\n> \n> - For each stack\n> - Run stack level checks\n> - For each table in stack\n> - Run table level checks\n> - For each block in table\n> - Run block level checks\n> - For each ref / log\n> - Run ref / log level checks\n> \n> But we move some of my tests to be runtime checks, leaving this as the\n> only check remaining. We could still do the first level of what I\n> mentioned above. The only reason I didn't was because we wanted to check\n> all files in the stack dir. But I think this is much better, having\n> unknown files in the reftable directory doesn't affect the repository in\n> any way. So I would argue perhaps that we shouldn't even care about it.\n\nYeah, agreed. As long as we don't know about any edge cases where this\ndoes or did create problems I agree.\n\nPatrick\n"},{"id":"527334","messageId":"xmqqo6qyh57i.fsf@gitster.g","threadId":"63987","inReplyTo":"aNTdTu08kKugIr7j@pks.im","subject":"Re: [PATCH v3 4/8] reftable: ensure tables in a stack use sequential update indices","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-25T16:22:57Z","receivedAt":"2025-09-25T16:22:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> Wouldn't 'monotonically increasing' suggest that\n>> prev_table.max_update_index can be equal to cur_table.min_update_index?\n>> I have locally changed it to 'ascending order' for similar reasons.\n>\n> I guess the correct phrase here is \"strictly monotonically increasing\".\n\nBoth of you are right and I was wrong. Thanks for a clarification.\n\n"},{"id":"527980","messageId":"20251006-228-reftable-introduce-consistency-checks-v5-0-f196d386214f@gmail.com","threadId":"63987","inReplyTo":"20250819-228-reftable-introduce-consistency-checks-v1-0-8b8f6879fa9e@gmail.com","subject":"[PATCH v5 0/7] refs/reftable: add consistency checks","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-06T14:22:58Z","receivedAt":"2025-10-06T14:23:06Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The reference subsystems allows for adding backend specific consistency\nchecks. These checks are run as part of 'git refs verify'.\n\nWhile the files backend has some consistency checks added, the reftable\nbackend currently has none. This series first tightens the reftable\nbackend to make it a little more strict and then also adds the required\ninfrastructure and some simple consistency checks.\n\nSince the reftable backend is treated as a library within the Git\ncodebase, we don't want to spillover our internal fsck implementation\ninto the library. At the same time, the fsck checks need to access\ninternal structures of the reftable library which aren't exposed outside\nthe library.\n\nSo we solve this by adding a 'reftable/fsck.[ch]' which implements and\nexposes a checker for the reftable library and returns specific errors\nas defined by the library. We then add glue code within\n'refs/reftable-backend.c' to map these errors to errors which Git's fsck\nimplementation would understand. This allows us to separate concerns.\n\nWe add the following consistency checks:\n\n 1. Check for validating the reftable table name. This is treated as a\n warning since the reftable specification only suggests a table name\n but doesn't enforce it. Also there is a difference in the table name\n used in Git vs that in jGit.\n\nWe tighten the reftable backend by raising a REFTABLE_FORMAT_ERROR error\nwhen:\n\n1. The 'tables.list' file doesn't have a trailing newline.\n\n---\nChanges in v5:\n- Added documentation around the return value of 'parse_names()'.\n- Added a test to validate that 'git refs verify' doesn't barf against\n a clean working repository with multiple reftable tables.\n- Link to v4: https://lore.kernel.org/all/20250926-228-reftable-introduce-consistency-checks-v4-0-c96fd8551c0d@gmail.com\n\nChanges in v4:\n- The biggest change is to iterate over the tables in a reftable stack\n for consistency checks instead of all files inside the REFTABLE_DIR.\n This avoids all race conditions. Also, since we only check the tables\n in a stack, it no longer makes sense to check file type.\n- The discussion about update indices was concluded that tables indices\n in a stack must be strictly monotonically increasing. While modifying\n the code to do the same. I realized that we already have this check in\n 'reftable_addition_add()' where we check while adding a new table to\n the stack: `wr->min_update_index < add->next_update_index`. So I've\n dropped this patch from the series.\n- Change parse_names() to accept the output string array as an argument\n and return an error instead. This makes the flow a little easier to\n understand.\n- Link to v3: https://lore.kernel.org/r/20250918-228-reftable-introduce-consistency-checks-v3-0-271af03eb34d@gmail.com\n\nChanges in v3:\n- I took a long hiatus from this topic, mostly due to other priorities.\n This has been rebased on top of '92c87bdc40 (The eighth batch,\n 2025-09-12)' since there were conflicts.\n- Junio suggested that two of the consistency checks (trailing newlines,\n sequential update indices for tables in stack) should actually be\n checked during runtime. I have made that change in this version.\n- I've cleaned up the code and modularized the 'reftable/fsck.c' code.\n- Invalid table name emits a warning, since the reftable spec doesn't\n enforce it but only makes a suggestion.\n- Broken down the commits to make it easier to review.\n- Link to v2: https://lore.kernel.org/r/20250902-228-reftable-introduce-consistency-checks-v2-0-4f96b3834779@gmail.com\n\nChanges in v2:\n- Ensured that 'struct reftable_fsck_info' is passed around as a\n pointer, this provides a smaller footprint (pointer size vs struct\n size).\n- Run FSCK checks for other worktrees too, even if one of them fails.\n- Separate messaging for table name vs table check and add additional\n test.\n- Use the relative path in messages used.\n- Small style and typo fixes.\n- Link to v1: https://lore.kernel.org/r/20250819-228-reftable-introduce-consistency-checks-v1-0-8b8f6879fa9e@gmail.com\n\n---\n Documentation/fsck-msgids.adoc | 6 +--\n Makefile | 3 +-\n fsck.h | 39 +++++++--------\n meson.build | 1 +\n refs.c | 4 ++\n refs/debug.c | 1 -\n refs/files-backend.c | 3 --\n refs/reftable-backend.c | 58 ++++++++++++++++++++---\n reftable/basics.c | 37 ++++++++++-----\n reftable/basics.h | 7 +--\n reftable/fsck.c | 100 +++++++++++++++++++++++++++++++++++++++\n reftable/reftable-fsck.h | 40 ++++++++++++++++\n reftable/stack.c | 7 +--\n t/meson.build | 1 +\n t/t0614-reftable-fsck.sh | 58 +++++++++++++++++++++++\n t/unit-tests/u-reftable-basics.c | 24 ++++++++--\n 16 files changed, 330 insertions(+), 59 deletions(-)\n\nKarthik Nayak (7):\n refs: remove unused headers\n refs: move consistency check msg to generic layer\n reftable: check for trailing newline in 'tables.list'\n Documentation/fsck-msgids: remove duplicate msg id\n fsck: order 'fsck_msg_type' alphabetically\n reftable: add code to facilitate consistency checks\n refs/reftable: add fsck check for checking the table name\n\nRange-diff versus v4:\n\n1: 4e40ab1ff7 < -: ---------- refs/reftable: add consistency checks\n2: b91194e060 = 1: 6e3766330b refs: remove unused headers\n3: d48afbf588 = 2: e93c0deaf7 refs: move consistency check msg to generic layer\n4: cd7ca2a585 ! 3: 7a282473a1 reftable: check for trailing newline in 'tables.list'\n @@ reftable/basics.h: void free_names(char **a);\n - * without terminating '\\0'. Empty names are discarded. Returns a `NULL`\n - * pointer when allocations fail.\n + * without terminating '\\0'. Empty names are discarded.\n ++ *\n ++ * Returns 0 on success, a reftable error code on error.\n */\n -char **parse_names(char *buf, int size);\n +int parse_names(char *buf, int size, char ***out);\n5: e3e0c0b4ae = 4: 4b47088232 Documentation/fsck-msgids: remove duplicate msg id\n6: 24a8d93adc = 5: 112ae21321 fsck: order 'fsck_msg_type' alphabetically\n7: d83d763be1 = 6: 3d1fc18260 reftable: add code to facilitate consistency checks\n8: d86ecd5bed ! 7: 2b628e3623 refs/reftable: add fsck check for checking the table name\n @@ Commit message\n \n So treat non-conformant file names as warnings.\n \n - While adding the fsck header to 'refs/reftable-backend.c', order the\n - list of headers.\n + While adding the fsck header to 'refs/reftable-backend.c', modify the\n + list to maintain lexicographical ordering.\n \n Signed-off-by: Karthik Nayak \n \n @@ t/t0614-reftable-fsck.sh (new)\n +\n +. ./test-lib.sh\n +\n ++test_expect_success \"no errors reported on a well formed repository\" '\n ++\ttest_when_finished \"rm -rf repo\" &&\n ++\tgit init repo &&\n ++\t(\n ++\t\tcd repo &&\n ++\t\tgit commit --allow-empty -m initial &&\n ++\n ++\t\tfor i in $(test_seq 20)\n ++\t\tdo\n ++\t\t\tgit update-ref branch-$i HEAD || return 1\n ++\t\tdone &&\n ++\n ++\t\t# The repository should end up with multiple tables.\n ++\t\ttest_line_count \">\" 1 .git/reftable/tables.list &&\n ++\n ++\t\tgit refs verify 2>err &&\n ++\t\ttest_must_be_empty err\n ++\t)\n ++'\n ++\n +for TABLE_NAME in \"foo-bar-e4d12d59.ref\" \\\n +\t\"0x00000000zzzz-0x00000000zzzz-e4d12d59.ref\" \\\n +\t\"0x000000000001-0x000000000002-e4d12d59.abc\" \\\n\n\nbase-commit: a483264b01b977f3e65a4419103c21e6af7412a2\nchange-id: 20250714-228-reftable-introduce-consistency-checks-379ded93c544\n\nThanks\n- Karthik\n\n"},{"id":"527981","messageId":"20251006-228-reftable-introduce-consistency-checks-v5-1-f196d386214f@gmail.com","threadId":"63987","inReplyTo":"20251006-228-reftable-introduce-consistency-checks-v5-0-f196d386214f@gmail.com","subject":"[PATCH v5 1/7] refs: remove unused headers","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-06T14:22:59Z","receivedAt":"2025-10-06T14:23:06Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"In the 'refs/' namespace, some of the included header files are not\nneeded, let's remove them.\n\nSigned-off-by: Karthik Nayak \n---\n refs/debug.c | 1 -\n refs/files-backend.c | 1 -\n refs/reftable-backend.c | 1 -\n 3 files changed, 3 deletions(-)\n\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 1cb955961e..697adbd0dc 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -1,7 +1,6 @@\n #include \"git-compat-util.h\"\n #include \"hex.h\"\n #include \"refs-internal.h\"\n-#include \"string-list.h\"\n #include \"trace.h\"\n \n static struct trace_key trace_refs = TRACE_KEY_INIT(REFS);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 1b3bf26add..d4fb033417 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -20,7 +20,6 @@\n #include \"../dir-iterator.h\"\n #include \"../lockfile.h\"\n #include \"../object.h\"\n-#include \"../object-file.h\"\n #include \"../path.h\"\n #include \"../dir.h\"\n #include \"../chdir-notify.h\"\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 9e889da2ff..2152349cb9 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -11,7 +11,6 @@\n #include \"../hex.h\"\n #include \"../iterator.h\"\n #include \"../ident.h\"\n-#include \"../lockfile.h\"\n #include \"../object.h\"\n #include \"../path.h\"\n #include \"../refs.h\"\n\n-- \n2.51.0\n\n"},{"id":"527982","messageId":"20251006-228-reftable-introduce-consistency-checks-v5-2-f196d386214f@gmail.com","threadId":"63987","inReplyTo":"20251006-228-reftable-introduce-consistency-checks-v5-0-f196d386214f@gmail.com","subject":"[PATCH v5 2/7] refs: move consistency check msg to generic layer","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-06T14:23:00Z","receivedAt":"2025-10-06T14:23:07Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The files-backend prints a message before the consistency checks run.\nMove this to the generic layer so both the files and reftable backend\ncan benefit from this message.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 4 ++++\n refs/files-backend.c | 2 --\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 4ff55cf24f..4a7c394226 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -32,6 +32,7 @@\n #include \"commit.h\"\n #include \"wildmatch.h\"\n #include \"ident.h\"\n+#include \"fsck.h\"\n \n /*\n * List of all available backends\n@@ -323,6 +324,9 @@ int check_refname_format(const char *refname, int flags)\n int refs_fsck(struct ref_store *refs, struct fsck_options *o,\n \t struct worktree *wt)\n {\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, _(\"Checking references consistency\"));\n+\n \treturn refs->be->fsck(refs, o, wt);\n }\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex d4fb033417..603b1343d8 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3906,8 +3906,6 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \t\tNULL,\n \t};\n \n-\tif (o->verbose)\n-\t\tfprintf_ln(stderr, _(\"Checking references consistency\"));\n \treturn files_fsck_refs_dir(ref_store, o, \"refs\", wt, fsck_refs_fn);\n }\n \n\n-- \n2.51.0\n\n"},{"id":"527983","messageId":"20251006-228-reftable-introduce-consistency-checks-v5-3-f196d386214f@gmail.com","threadId":"63987","inReplyTo":"20251006-228-reftable-introduce-consistency-checks-v5-0-f196d386214f@gmail.com","subject":"[PATCH v5 3/7] reftable: check for trailing newline in 'tables.list'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-06T14:23:01Z","receivedAt":"2025-10-06T14:23:08Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"In the reftable format, the 'tables.list' file contains a\nnewline separated list of tables. While we parse this file, we do not\ncheck or care about the last newline. Tighten the parser in\n`parse_names()` to return an appropriate error if the last newline is\nmissing.\n\nThis requires modification to `parse_names()` to now return the error\nwhile accepting the output as a third argument.\n\nSigned-off-by: Karthik Nayak \n---\n reftable/basics.c | 37 ++++++++++++++++++++++++-------------\n reftable/basics.h | 7 ++++---\n reftable/stack.c | 7 +------\n t/unit-tests/u-reftable-basics.c | 24 ++++++++++++++++++++----\n 4 files changed, 49 insertions(+), 26 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 9988ebd635..e969927b61 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -195,44 +195,55 @@ size_t names_length(const char **names)\n \treturn p - names;\n }\n \n-char **parse_names(char *buf, int size)\n+int parse_names(char *buf, int size, char ***out)\n {\n \tchar **names = NULL;\n \tsize_t names_cap = 0;\n \tsize_t names_len = 0;\n \tchar *p = buf;\n \tchar *end = buf + size;\n+\tint err = 0;\n \n \twhile (p < end) {\n \t\tchar *next = strchr(p, '\\n');\n-\t\tif (next && next < end) {\n-\t\t\t*next = 0;\n+\t\tif (!next) {\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n+\t\t} else if (next < end) {\n+\t\t\t*next = '\\0';\n \t\t} else {\n \t\t\tnext = end;\n \t\t}\n+\n \t\tif (p < next) {\n \t\t\tif (REFTABLE_ALLOC_GROW(names, names_len + 1,\n-\t\t\t\t\t\tnames_cap))\n-\t\t\t\tgoto err;\n+\t\t\t\t\t\tnames_cap)) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n \n \t\t\tnames[names_len] = reftable_strdup(p);\n-\t\t\tif (!names[names_len++])\n-\t\t\t\tgoto err;\n+\t\t\tif (!names[names_len++]) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n \t\t}\n \t\tp = next + 1;\n \t}\n \n-\tif (REFTABLE_ALLOC_GROW(names, names_len + 1, names_cap))\n-\t\tgoto err;\n+\tif (REFTABLE_ALLOC_GROW(names, names_len + 1, names_cap)) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tnames[names_len] = NULL;\n \n-\treturn names;\n-\n-err:\n+\t*out = names;\n+\treturn 0;\n+done:\n \tfor (size_t i = 0; i < names_len; i++)\n \t\treftable_free(names[i]);\n \treftable_free(names);\n-\treturn NULL;\n+\treturn err;\n }\n \n int names_equal(const char **a, const char **b)\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 7d22f96261..e4b83b2b03 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -167,10 +167,11 @@ void free_names(char **a);\n \n /*\n * Parse a newline separated list of names. `size` is the length of the buffer,\n- * without terminating '\\0'. Empty names are discarded. Returns a `NULL`\n- * pointer when allocations fail.\n+ * without terminating '\\0'. Empty names are discarded.\n+ *\n+ * Returns 0 on success, a reftable error code on error.\n */\n-char **parse_names(char *buf, int size);\n+int parse_names(char *buf, int size, char ***out);\n \n /* compares two NULL-terminated arrays of strings. */\n int names_equal(const char **a, const char **b);\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex f91ce50bcd..65d89820bd 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -109,12 +109,7 @@ static int fd_read_lines(int fd, char ***namesp)\n \t}\n \tbuf[size] = 0;\n \n-\t*namesp = parse_names(buf, size);\n-\tif (!*namesp) {\n-\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n-\t\tgoto done;\n-\t}\n-\n+\terr = parse_names(buf, size, namesp);\n done:\n \treftable_free(buf);\n \treturn err;\ndiff --git a/t/unit-tests/u-reftable-basics.c b/t/unit-tests/u-reftable-basics.c\nindex a0471083e7..73566ed0eb 100644\n--- a/t/unit-tests/u-reftable-basics.c\n+++ b/t/unit-tests/u-reftable-basics.c\n@@ -9,6 +9,7 @@ license that can be found in the LICENSE file or at\n #include \"unit-test.h\"\n #include \"lib-reftable.h\"\n #include \"reftable/basics.h\"\n+#include \"reftable/reftable-error.h\"\n \n struct integer_needle_lesseq_args {\n \tint needle;\n@@ -79,14 +80,18 @@ void test_reftable_basics__names_equal(void)\n void test_reftable_basics__parse_names(void)\n {\n \tchar in1[] = \"line\\n\";\n-\tchar in2[] = \"a\\nb\\nc\";\n-\tchar **out = parse_names(in1, strlen(in1));\n+\tchar in2[] = \"a\\nb\\nc\\n\";\n+\tchar **out = NULL;\n+\tint err = parse_names(in1, strlen(in1), &out);\n+\tcl_assert(err == 0);\n \tcl_assert(out != NULL);\n \tcl_assert_equal_s(out[0], \"line\");\n \tcl_assert(!out[1]);\n \tfree_names(out);\n \n-\tout = parse_names(in2, strlen(in2));\n+\tout = NULL;\n+\terr = parse_names(in2, strlen(in2), &out);\n+\tcl_assert(err == 0);\n \tcl_assert(out != NULL);\n \tcl_assert_equal_s(out[0], \"a\");\n \tcl_assert_equal_s(out[1], \"b\");\n@@ -95,10 +100,21 @@ void test_reftable_basics__parse_names(void)\n \tfree_names(out);\n }\n \n+void test_reftable_basics__parse_names_missing_newline(void)\n+{\n+\tchar in1[] = \"line\\nline2\";\n+\tchar **out = NULL;\n+\tint err = parse_names(in1, strlen(in1), &out);\n+\tcl_assert(err == REFTABLE_FORMAT_ERROR);\n+\tcl_assert(out == NULL);\n+}\n+\n void test_reftable_basics__parse_names_drop_empty_string(void)\n {\n \tchar in[] = \"a\\n\\nb\\n\";\n-\tchar **out = parse_names(in, strlen(in));\n+\tchar **out = NULL;\n+\tint err = parse_names(in, strlen(in), &out);\n+\tcl_assert(err == 0);\n \tcl_assert(out != NULL);\n \tcl_assert_equal_s(out[0], \"a\");\n \t/* simply '\\n' should be dropped as empty string */\n\n-- \n2.51.0\n\n"},{"id":"527984","messageId":"20251006-228-reftable-introduce-consistency-checks-v5-4-f196d386214f@gmail.com","threadId":"63987","inReplyTo":"20251006-228-reftable-introduce-consistency-checks-v5-0-f196d386214f@gmail.com","subject":"[PATCH v5 4/7] Documentation/fsck-msgids: remove duplicate msg id","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-06T14:23:02Z","receivedAt":"2025-10-06T14:23:09Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `gitmodulesLarge` is repeated twice. Remove the second duplicate.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/fsck-msgids.adoc | 3 ---\n 1 file changed, 3 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.adoc b/Documentation/fsck-msgids.adoc\nindex 0ba4f9a27e..1c912615f9 100644\n--- a/Documentation/fsck-msgids.adoc\n+++ b/Documentation/fsck-msgids.adoc\n@@ -104,9 +104,6 @@\n `gitmodulesParse`::\n \t(INFO) Could not parse `.gitmodules` blob.\n \n-`gitmodulesLarge`;\n-\t(ERROR) `.gitmodules` blob is too large to parse.\n-\n `gitmodulesPath`::\n \t(ERROR) `.gitmodules` path is invalid.\n \n\n-- \n2.51.0\n\n"},{"id":"527985","messageId":"20251006-228-reftable-introduce-consistency-checks-v5-5-f196d386214f@gmail.com","threadId":"63987","inReplyTo":"20251006-228-reftable-introduce-consistency-checks-v5-0-f196d386214f@gmail.com","subject":"[PATCH v5 5/7] fsck: order 'fsck_msg_type' alphabetically","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-06T14:23:03Z","receivedAt":"2025-10-06T14:23:10Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The list of 'fsck_msg_type' seem to be alphabetically ordered, but there\nare a few small misses. Fix this by sorting the sub-sections of the\nlist to maintain alphabetical ordering.\n\nSigned-off-by: Karthik Nayak \n---\n fsck.h | 38 +++++++++++++++++++-------------------\n 1 file changed, 19 insertions(+), 19 deletions(-)\n\ndiff --git a/fsck.h b/fsck.h\nindex dd7df3d5b3..6b0db235e0 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -33,15 +33,27 @@ enum fsck_msg_type {\n \tFUNC(BAD_PACKED_REF_ENTRY, ERROR) \\\n \tFUNC(BAD_PACKED_REF_HEADER, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REFERENT_NAME, ERROR) \\\n \tFUNC(BAD_REF_CONTENT, ERROR) \\\n \tFUNC(BAD_REF_FILETYPE, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n-\tFUNC(BAD_REFERENT_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n \tFUNC(BAD_TYPE, ERROR) \\\n \tFUNC(DUPLICATE_ENTRIES, ERROR) \\\n+\tFUNC(GITATTRIBUTES_BLOB, ERROR) \\\n+\tFUNC(GITATTRIBUTES_LARGE, ERROR) \\\n+\tFUNC(GITATTRIBUTES_LINE_LENGTH, ERROR) \\\n+\tFUNC(GITATTRIBUTES_MISSING, ERROR) \\\n+\tFUNC(GITMODULES_BLOB, ERROR) \\\n+\tFUNC(GITMODULES_LARGE, ERROR) \\\n+\tFUNC(GITMODULES_MISSING, ERROR) \\\n+\tFUNC(GITMODULES_NAME, ERROR) \\\n+\tFUNC(GITMODULES_PATH, ERROR) \\\n+\tFUNC(GITMODULES_SYMLINK, ERROR) \\\n+\tFUNC(GITMODULES_UPDATE, ERROR) \\\n+\tFUNC(GITMODULES_URL, ERROR) \\\n \tFUNC(MISSING_AUTHOR, ERROR) \\\n \tFUNC(MISSING_COMMITTER, ERROR) \\\n \tFUNC(MISSING_EMAIL, ERROR) \\\n@@ -60,39 +72,27 @@ enum fsck_msg_type {\n \tFUNC(TREE_NOT_SORTED, ERROR) \\\n \tFUNC(UNKNOWN_TYPE, ERROR) \\\n \tFUNC(ZERO_PADDED_DATE, ERROR) \\\n-\tFUNC(GITMODULES_MISSING, ERROR) \\\n-\tFUNC(GITMODULES_BLOB, ERROR) \\\n-\tFUNC(GITMODULES_LARGE, ERROR) \\\n-\tFUNC(GITMODULES_NAME, ERROR) \\\n-\tFUNC(GITMODULES_SYMLINK, ERROR) \\\n-\tFUNC(GITMODULES_URL, ERROR) \\\n-\tFUNC(GITMODULES_PATH, ERROR) \\\n-\tFUNC(GITMODULES_UPDATE, ERROR) \\\n-\tFUNC(GITATTRIBUTES_MISSING, ERROR) \\\n-\tFUNC(GITATTRIBUTES_LARGE, ERROR) \\\n-\tFUNC(GITATTRIBUTES_LINE_LENGTH, ERROR) \\\n-\tFUNC(GITATTRIBUTES_BLOB, ERROR) \\\n \t/* warnings */ \\\n \tFUNC(EMPTY_NAME, WARN) \\\n \tFUNC(FULL_PATHNAME, WARN) \\\n \tFUNC(HAS_DOT, WARN) \\\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n+\tFUNC(LARGE_PATHNAME, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n-\tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n-\tFUNC(LARGE_PATHNAME, WARN) \\\n+\tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \t/* infos (reported as warnings, but ignored by default) */ \\\n \tFUNC(BAD_FILEMODE, INFO) \\\n+\tFUNC(BAD_TAG_NAME, INFO) \\\n \tFUNC(EMPTY_PACKED_REFS_FILE, INFO) \\\n-\tFUNC(GITMODULES_PARSE, INFO) \\\n-\tFUNC(GITIGNORE_SYMLINK, INFO) \\\n \tFUNC(GITATTRIBUTES_SYMLINK, INFO) \\\n+\tFUNC(GITIGNORE_SYMLINK, INFO) \\\n+\tFUNC(GITMODULES_PARSE, INFO) \\\n \tFUNC(MAILMAP_SYMLINK, INFO) \\\n-\tFUNC(BAD_TAG_NAME, INFO) \\\n \tFUNC(MISSING_TAGGER_ENTRY, INFO) \\\n-\tFUNC(SYMLINK_REF, INFO) \\\n \tFUNC(REF_MISSING_NEWLINE, INFO) \\\n+\tFUNC(SYMLINK_REF, INFO) \\\n \tFUNC(SYMREF_TARGET_IS_NOT_A_REF, INFO) \\\n \tFUNC(TRAILING_REF_CONTENT, INFO) \\\n \t/* ignored (elevated when requested) */ \\\n\n-- \n2.51.0\n\n"},{"id":"527986","messageId":"20251006-228-reftable-introduce-consistency-checks-v5-6-f196d386214f@gmail.com","threadId":"63987","inReplyTo":"20251006-228-reftable-introduce-consistency-checks-v5-0-f196d386214f@gmail.com","subject":"[PATCH v5 6/7] reftable: add code to facilitate consistency checks","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-06T14:23:04Z","receivedAt":"2025-10-06T14:23:11Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git refs verify` command is used to run consistency checks on the\nreference backends. This command is also invoked when users run 'git\nfsck'. While the files-backend has some fsck checks added, the reftable\nbackend lacks such checks. Let's add the required infrastructure and a\ncheck to test for the files present in the reftable directory.\n\nSince the reftable library is treated as an independent library we\nshould ensure that the library code works independently without\nknowledge about Git's internals. To do this, add both 'reftable/fsck.c'\nand 'reftable/reftable-fsck.h'. Which provide an entry point\n'reftable_fsck_check' for running fsck checks over a provided reftable\nstack. The callee provides the function with callbacks to handle issue\nand information reporting.\n\nThe added check, goes over all tables in the reftable stack validates\nthat they have a valid name. It not, it raises an error.\n\nWhile here, move 'reftable/error.o' in the Makefile to retain\nlexicographic ordering.\n\nSigned-off-by: Karthik Nayak \n---\n Makefile | 3 +-\n meson.build | 1 +\n reftable/fsck.c | 100 +++++++++++++++++++++++++++++++++++++++++++++++\n reftable/reftable-fsck.h | 40 +++++++++++++++++++\n 4 files changed, 143 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex 4c95affadb..03fbaf2b21 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2732,9 +2732,10 @@ XDIFF_OBJS += xdiff/xutils.o\n xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n-REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n+REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/fsck.o\n REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\ndiff --git a/meson.build b/meson.build\nindex b3dfcc0497..8914252910 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -452,6 +452,7 @@ libgit_sources = [\n 'reftable/error.c',\n 'reftable/block.c',\n 'reftable/blocksource.c',\n+ 'reftable/fsck.c',\n 'reftable/iter.c',\n 'reftable/merged.c',\n 'reftable/pq.c',\ndiff --git a/reftable/fsck.c b/reftable/fsck.c\nnew file mode 100644\nindex 0000000000..26b9115b14\n--- /dev/null\n+++ b/reftable/fsck.c\n@@ -0,0 +1,100 @@\n+#include \"basics.h\"\n+#include \"reftable-fsck.h\"\n+#include \"reftable-table.h\"\n+#include \"stack.h\"\n+\n+static bool table_has_valid_name(const char *name)\n+{\n+\tconst char *ptr = name;\n+\tchar *endptr;\n+\n+\t/* strtoull doesn't set errno on success */\n+\terrno = 0;\n+\n+\tstrtoull(ptr, &endptr, 16);\n+\tif (errno)\n+\t\treturn false;\n+\tptr = endptr;\n+\n+\tif (*ptr != '-')\n+\t\treturn false;\n+\tptr++;\n+\n+\tstrtoull(ptr, &endptr, 16);\n+\tif (errno)\n+\t\treturn false;\n+\tptr = endptr;\n+\n+\tif (*ptr != '-')\n+\t\treturn false;\n+\tptr++;\n+\n+\tstrtoul(ptr, &endptr, 16);\n+\tif (errno)\n+\t\treturn false;\n+\tptr = endptr;\n+\n+\tif (strcmp(ptr, \".ref\") && strcmp(ptr, \".log\"))\n+\t\treturn false;\n+\n+\treturn true;\n+}\n+\n+typedef int (*table_check_fn)(struct reftable_table *table,\n+\t\t\t reftable_fsck_report_fn report_fn,\n+\t\t\t void *cb_data);\n+\n+static int table_check_name(struct reftable_table *table,\n+\t\t\t reftable_fsck_report_fn report_fn,\n+\t\t\t void *cb_data)\n+{\n+\tif (!table_has_valid_name(table->name)) {\n+\t\tstruct reftable_fsck_info info;\n+\n+\t\tinfo.error = REFTABLE_FSCK_ERROR_TABLE_NAME;\n+\t\tinfo.msg = \"invalid reftable table name\";\n+\t\tinfo.path = table->name;\n+\n+\t\treturn report_fn(&info, cb_data);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int table_checks(struct reftable_table *table,\n+\t\t\treftable_fsck_report_fn report_fn,\n+\t\t\treftable_fsck_verbose_fn verbose_fn UNUSED,\n+\t\t\tvoid *cb_data)\n+{\n+\ttable_check_fn table_check_fns[] = {\n+\t\ttable_check_name,\n+\t\tNULL,\n+\t};\n+\tint err = 0;\n+\n+\tfor (size_t i = 0; table_check_fns[i]; i++)\n+\t\terr |= table_check_fns[i](table, report_fn, cb_data);\n+\n+\treturn err;\n+}\n+\n+int reftable_fsck_check(struct reftable_stack *stack,\n+\t\t\treftable_fsck_report_fn report_fn,\n+\t\t\treftable_fsck_verbose_fn verbose_fn,\n+\t\t\tvoid *cb_data)\n+{\n+\tstruct reftable_buf msg = REFTABLE_BUF_INIT;\n+\tint err = 0;\n+\n+\tfor (size_t i = 0; i < stack->tables_len; i++) {\n+\t\treftable_buf_reset(&msg);\n+\t\treftable_buf_addstr(&msg, \"Checking table: \");\n+\t\treftable_buf_addstr(&msg, stack->tables[i]->name);\n+\t\tverbose_fn(msg.buf, cb_data);\n+\n+\t\terr |= table_checks(stack->tables[i], report_fn, verbose_fn, cb_data);\n+\t}\n+\n+\treftable_buf_release(&msg);\n+\treturn err;\n+}\ndiff --git a/reftable/reftable-fsck.h b/reftable/reftable-fsck.h\nnew file mode 100644\nindex 0000000000..007a392cf9\n--- /dev/null\n+++ b/reftable/reftable-fsck.h\n@@ -0,0 +1,40 @@\n+#ifndef REFTABLE_FSCK_H\n+#define REFTABLE_FSCK_H\n+\n+#include \"reftable-stack.h\"\n+\n+enum reftable_fsck_error {\n+\t/* Invalid table name */\n+\tREFTABLE_FSCK_ERROR_TABLE_NAME = 0,\n+\t/* Used for bounds checking, must be last */\n+\tREFTABLE_FSCK_MAX_VALUE,\n+};\n+\n+/* Represents an individual error encountered during the FSCK checks. */\n+struct reftable_fsck_info {\n+\tenum reftable_fsck_error error;\n+\tconst char *msg;\n+\tconst char *path;\n+};\n+\n+typedef int reftable_fsck_report_fn(struct reftable_fsck_info *info,\n+\t\t\t\t void *cb_data);\n+typedef void reftable_fsck_verbose_fn(const char *msg, void *cb_data);\n+\n+/*\n+ * Given a reftable stack, perform consistency checks on the stack.\n+ *\n+ * If an issue is encountered, the issue is reported to the callee via the\n+ * provided 'report_fn'. If the issue is non-recoverable the flow will not\n+ * continue. If it is recoverable, the flow will continue and further issues\n+ * will be reported as identified.\n+ *\n+ * The 'verbose_fn' will be invoked to provide verbose information about\n+ * the progress and state of the consistency checks.\n+ */\n+int reftable_fsck_check(struct reftable_stack *stack,\n+\t\t\treftable_fsck_report_fn report_fn,\n+\t\t\treftable_fsck_verbose_fn verbose_fn,\n+\t\t\tvoid *cb_data);\n+\n+#endif /* REFTABLE_FSCK_H */\n\n-- \n2.51.0\n\n"},{"id":"527987","messageId":"20251006-228-reftable-introduce-consistency-checks-v5-7-f196d386214f@gmail.com","threadId":"63987","inReplyTo":"20251006-228-reftable-introduce-consistency-checks-v5-0-f196d386214f@gmail.com","subject":"[PATCH v5 7/7] refs/reftable: add fsck check for checking the table name","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-06T14:23:05Z","receivedAt":"2025-10-06T14:23:11Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Add glue code in 'refs/reftable-backend.c' which calls the reftable\nlibrary to perform the fsck checks. Here we also map the reftable errors\nto Git' fsck errors.\n\nIntroduce a check to validate table names for a given reftable stack.\nAlso add 'badReftableTableName' as a corresponding error within Git. The\nreftable specification mentions:\n\n It suggested to use\n ${min_update_index}-${max_update_index}-${random}.ref as a naming\n convention.\n\nSo treat non-conformant file names as warnings.\n\nWhile adding the fsck header to 'refs/reftable-backend.c', modify the\nlist to maintain lexicographical ordering.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/fsck-msgids.adoc | 3 +++\n fsck.h | 1 +\n refs/reftable-backend.c | 57 +++++++++++++++++++++++++++++++++++++----\n t/meson.build | 1 +\n t/t0614-reftable-fsck.sh | 58 ++++++++++++++++++++++++++++++++++++++++++\n 5 files changed, 115 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.adoc b/Documentation/fsck-msgids.adoc\nindex 1c912615f9..81f11ba125 100644\n--- a/Documentation/fsck-msgids.adoc\n+++ b/Documentation/fsck-msgids.adoc\n@@ -38,6 +38,9 @@\n `badReferentName`::\n \t(ERROR) The referent name of a symref is invalid.\n \n+`badReftableTableName`::\n+\t(WARN) A reftable table has an invalid name.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex 6b0db235e0..759df97655 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -73,6 +73,7 @@ enum fsck_msg_type {\n \tFUNC(UNKNOWN_TYPE, ERROR) \\\n \tFUNC(ZERO_PADDED_DATE, ERROR) \\\n \t/* warnings */ \\\n+\tFUNC(BAD_REFTABLE_TABLE_NAME, WARN) \\\n \tFUNC(EMPTY_NAME, WARN) \\\n \tFUNC(FULL_PATHNAME, WARN) \\\n \tFUNC(HAS_DOT, WARN) \\\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 2152349cb9..b106fd8b53 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -6,6 +6,7 @@\n #include \"../config.h\"\n #include \"../dir.h\"\n #include \"../environment.h\"\n+#include \"../fsck.h\"\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n@@ -15,10 +16,11 @@\n #include \"../path.h\"\n #include \"../refs.h\"\n #include \"../reftable/reftable-basics.h\"\n-#include \"../reftable/reftable-stack.h\"\n-#include \"../reftable/reftable-record.h\"\n #include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-fsck.h\"\n #include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-stack.h\"\n #include \"../repo-settings.h\"\n #include \"../setup.h\"\n #include \"../strmap.h\"\n@@ -2707,11 +2709,56 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n-static int reftable_be_fsck(struct ref_store *ref_store UNUSED,\n-\t\t\t struct fsck_options *o UNUSED,\n+static void reftable_fsck_verbose_handler(const char *msg, void *cb_data)\n+{\n+\tstruct fsck_options *o = cb_data;\n+\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, \"%s\", msg);\n+}\n+\n+static const enum fsck_msg_id fsck_msg_id_map[] = {\n+\t[REFTABLE_FSCK_ERROR_TABLE_NAME] = FSCK_MSG_BAD_REFTABLE_TABLE_NAME,\n+};\n+\n+static int reftable_fsck_error_handler(struct reftable_fsck_info *info,\n+\t\t\t\t void *cb_data)\n+{\n+\tstruct fsck_ref_report report = { .path = info->path };\n+\tstruct fsck_options *o = cb_data;\n+\tenum fsck_msg_id msg_id;\n+\n+\tif (info->error < 0 || info->error >= REFTABLE_FSCK_MAX_VALUE)\n+\t\tBUG(\"unknown fsck error: %d\", (int)info->error);\n+\n+\tmsg_id = fsck_msg_id_map[info->error];\n+\n+\tif (!msg_id)\n+\t\tBUG(\"fsck_msg_id value missing for reftable error: %d\", (int)info->error);\n+\n+\treturn fsck_report_ref(o, &report, msg_id, \"%s\", info->msg);\n+}\n+\n+static int reftable_be_fsck(struct ref_store *ref_store, struct fsck_options *o,\n \t\t\t struct worktree *wt UNUSED)\n {\n-\treturn 0;\n+\tstruct reftable_ref_store *refs;\n+\tstruct strmap_entry *entry;\n+\tstruct hashmap_iter iter;\n+\tint ret = 0;\n+\n+\trefs = reftable_be_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tret |= reftable_fsck_check(refs->main_backend.stack, reftable_fsck_error_handler,\n+\t\t\t\t reftable_fsck_verbose_handler, o);\n+\n+\tstrmap_for_each_entry(&refs->worktree_backends, &iter, entry) {\n+\t\tstruct reftable_backend *b = (struct reftable_backend *)entry->value;\n+\t\tret |= reftable_fsck_check(b->stack, reftable_fsck_error_handler,\n+\t\t\t\t\t reftable_fsck_verbose_handler, o);\n+\t}\n+\n+\treturn ret;\n }\n \n struct ref_storage_be refs_be_reftable = {\ndiff --git a/t/meson.build b/t/meson.build\nindex 7974795fe4..ec1fc0b2a1 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -146,6 +146,7 @@ integration_tests = [\n 't0611-reftable-httpd.sh',\n 't0612-reftable-jgit-compatibility.sh',\n 't0613-reftable-write-options.sh',\n+ 't0614-reftable-fsck.sh',\n 't1000-read-tree-m-3way.sh',\n 't1001-read-tree-m-2way.sh',\n 't1002-read-tree-m-u-2way.sh',\ndiff --git a/t/t0614-reftable-fsck.sh b/t/t0614-reftable-fsck.sh\nnew file mode 100755\nindex 0000000000..a5be279ab3\n--- /dev/null\n+++ b/t/t0614-reftable-fsck.sh\n@@ -0,0 +1,58 @@\n+#!/bin/sh\n+\n+test_description='Test reftable backend consistency check'\n+\n+GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success \"no errors reported on a well formed repository\" '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\n+\t\tfor i in $(test_seq 20)\n+\t\tdo\n+\t\t\tgit update-ref branch-$i HEAD || return 1\n+\t\tdone &&\n+\n+\t\t# The repository should end up with multiple tables.\n+\t\ttest_line_count \">\" 1 .git/reftable/tables.list &&\n+\n+\t\tgit refs verify 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+for TABLE_NAME in \"foo-bar-e4d12d59.ref\" \\\n+\t\"0x00000000zzzz-0x00000000zzzz-e4d12d59.ref\" \\\n+\t\"0x000000000001-0x000000000002-e4d12d59.abc\" \\\n+\t\"0x000000000001-0x000000000002-e4d12d59.refabc\"; do\n+\ttest_expect_success \"table name $TABLE_NAME should be checked\" '\n+\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\tgit init repo &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit commit --allow-empty -m initial &&\n+\n+\t\t\tgit refs verify 2>err &&\n+\t\t\ttest_must_be_empty err &&\n+\n+\t\t\tEXISTING_TABLE=$(head -n1 .git/reftable/tables.list) &&\n+\t\t\tmv \".git/reftable/$EXISTING_TABLE\" \".git/reftable/$TABLE_NAME\" &&\n+\t\t\tsed \"s/${EXISTING_TABLE}/${TABLE_NAME}/g\" .git/reftable/tables.list > tables.list &&\n+\t\t\tmv tables.list .git/reftable/tables.list &&\n+\n+\t\t\tgit refs verify 2>err &&\n+\t\t\tcat >expect <<-EOF &&\n+\t\t\twarning: ${TABLE_NAME}: badReftableTableName: invalid reftable table name\n+\t\t\tEOF\n+\t\t\ttest_cmp expect err\n+\t\t)\n+\t'\n+done\n+\n+test_done\n\n-- \n2.51.0\n\n"},{"id":"528048","messageId":"xmqq5xcrof92.fsf@gitster.g","threadId":"63987","inReplyTo":"20251006-228-reftable-introduce-consistency-checks-v5-0-f196d386214f@gmail.com","subject":"Re: [PATCH v5 0/7] refs/reftable: add consistency checks","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-10-06T22:08:09Z","receivedAt":"2025-10-06T22:08:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> The reference subsystems allows for adding backend specific consistency\n> checks. These checks are run as part of 'git refs verify'.\n>\n> While the files backend has some consistency checks added, the reftable\n> backend currently has none. This series first tightens the reftable\n> backend to make it a little more strict and then also adds the required\n> infrastructure and some simple consistency checks.\n>\n> Since the reftable backend is treated as a library within the Git\n> codebase, we don't want to spillover our internal fsck implementation\n> into the library. At the same time, the fsck checks need to access\n> internal structures of the reftable library which aren't exposed outside\n> the library.\n>\n> So we solve this by adding a 'reftable/fsck.[ch]' which implements and\n> exposes a checker for the reftable library and returns specific errors\n> as defined by the library. We then add glue code within\n> 'refs/reftable-backend.c' to map these errors to errors which Git's fsck\n> implementation would understand. This allows us to separate concerns.\n>\n> We add the following consistency checks:\n>\n> 1. Check for validating the reftable table name. This is treated as a\n> warning since the reftable specification only suggests a table name\n> but doesn't enforce it. Also there is a difference in the table name\n> used in Git vs that in jGit.\n>\n> We tighten the reftable backend by raising a REFTABLE_FORMAT_ERROR error\n> when:\n>\n> 1. The 'tables.list' file doesn't have a trailing newline.\n>\n> ---\n> Changes in v5:\n> - Added documentation around the return value of 'parse_names()'.\n> - Added a test to validate that 'git refs verify' doesn't barf against\n> a clean working repository with multiple reftable tables.\n> - Link to v4: https://lore.kernel.org/all/20250926-228-reftable-introduce-consistency-checks-v4-0-c96fd8551c0d@gmail.com\n\nLooking good. Shall we declare victory and mark the topic for\n'next' now?\n\nThanks. \n"},{"id":"528051","messageId":"20251007023242.GA2747748@coredump.intra.peff.net","threadId":"63987","inReplyTo":"20251006-228-reftable-introduce-consistency-checks-v5-7-f196d386214f@gmail.com","subject":"Re: [PATCH v5 7/7] refs/reftable: add fsck check for checking the table name","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-10-07T02:32:42Z","receivedAt":"2025-10-07T02:32:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 06, 2025 at 04:23:05PM +0200, Karthik Nayak wrote:\n\n> +test_expect_success \"no errors reported on a well formed repository\" '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tgit commit --allow-empty -m initial &&\n> +\n> +\t\tfor i in $(test_seq 20)\n> +\t\tdo\n> +\t\t\tgit update-ref branch-$i HEAD || return 1\n> +\t\tdone &&\n\nDid you mean refs/heads/branch-$i here? As it is written, it creates a\nroot ref, and the name does not conform to the usual rules (all-caps,\nand ending in _HEAD). There are some holes in our checks, which is why\nit doesn't barf yet, but I have a series to fix that which I hope to\nsend out later this week.\n\n> +\t\t# The repository should end up with multiple tables.\n> +\t\ttest_line_count \">\" 1 .git/reftable/tables.list &&\n> +\n> +\t\tgit refs verify 2>err &&\n> +\t\ttest_must_be_empty err\n> +\t)\n\nArguably this verify command should be complaining about the broken\nnames, too.\n\n-Peff\n"},{"id":"528054","messageId":"CAOLa=ZSGsfhUM+cn0XGDJnFHLswxYqSOePPk+LXK0g3cYjaXfA@mail.gmail.com","threadId":"63987","inReplyTo":"20251007023242.GA2747748@coredump.intra.peff.net","subject":"Re: [PATCH v5 7/7] refs/reftable: add fsck check for checking the table name","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-07T08:45:21Z","receivedAt":"2025-10-07T08:45:24Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Jeff King writes:\n\n> On Mon, Oct 06, 2025 at 04:23:05PM +0200, Karthik Nayak wrote:\n>\n>> +test_expect_success \"no errors reported on a well formed repository\" '\n>> +\ttest_when_finished \"rm -rf repo\" &&\n>> +\tgit init repo &&\n>> +\t(\n>> +\t\tcd repo &&\n>> +\t\tgit commit --allow-empty -m initial &&\n>> +\n>> +\t\tfor i in $(test_seq 20)\n>> +\t\tdo\n>> +\t\t\tgit update-ref branch-$i HEAD || return 1\n>> +\t\tdone &&\n>\n> Did you mean refs/heads/branch-$i here? As it is written, it creates a\n> root ref, and the name does not conform to the usual rules (all-caps,\n> and ending in _HEAD). There are some holes in our checks, which is why\n> it doesn't barf yet, but I have a series to fix that which I hope to\n> send out later this week.\n>\n\nYeah, this was definitely a miss on my side. It works because currently\nwe haven't yet added reference level checks to reftables.\n\nThis series only adds stack/table level checks.\n\n>> +\t\t# The repository should end up with multiple tables.\n>> +\t\ttest_line_count \">\" 1 .git/reftable/tables.list &&\n>> +\n>> +\t\tgit refs verify 2>err &&\n>> +\t\ttest_must_be_empty err\n>> +\t)\n>\n> Arguably this verify command should be complaining about the broken\n> names, too.\n>\n\nYes, eventually it will when we implement reference level checks. Since\nthat's missing, it currently doesn't barf.\n\nIt does work as-is and we could leave it at that, until we actually\nimplement the reference level checks. But I think a quick re-roll will\navoid future confusion.\n\n> -Peff\n\nThanks for the review. Looking forward to your series.\n"},{"id":"528055","messageId":"CAOLa=ZRToJOUd_Devs54NfLCJuMZSBEwNcM8J0sbvy7x5Rb=PA@mail.gmail.com","threadId":"63987","inReplyTo":"xmqq5xcrof92.fsf@gitster.g","subject":"Re: [PATCH v5 0/7] refs/reftable: add consistency checks","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-07T08:47:14Z","receivedAt":"2025-10-07T08:47:16Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> The reference subsystems allows for adding backend specific consistency\n>> checks. These checks are run as part of 'git refs verify'.\n>>\n>> While the files backend has some consistency checks added, the reftable\n>> backend currently has none. This series first tightens the reftable\n>> backend to make it a little more strict and then also adds the required\n>> infrastructure and some simple consistency checks.\n>>\n>> Since the reftable backend is treated as a library within the Git\n>> codebase, we don't want to spillover our internal fsck implementation\n>> into the library. At the same time, the fsck checks need to access\n>> internal structures of the reftable library which aren't exposed outside\n>> the library.\n>>\n>> So we solve this by adding a 'reftable/fsck.[ch]' which implements and\n>> exposes a checker for the reftable library and returns specific errors\n>> as defined by the library. We then add glue code within\n>> 'refs/reftable-backend.c' to map these errors to errors which Git's fsck\n>> implementation would understand. This allows us to separate concerns.\n>>\n>> We add the following consistency checks:\n>>\n>> 1. Check for validating the reftable table name. This is treated as a\n>> warning since the reftable specification only suggests a table name\n>> but doesn't enforce it. Also there is a difference in the table name\n>> used in Git vs that in jGit.\n>>\n>> We tighten the reftable backend by raising a REFTABLE_FORMAT_ERROR error\n>> when:\n>>\n>> 1. The 'tables.list' file doesn't have a trailing newline.\n>>\n>> ---\n>> Changes in v5:\n>> - Added documentation around the return value of 'parse_names()'.\n>> - Added a test to validate that 'git refs verify' doesn't barf against\n>> a clean working repository with multiple reftable tables.\n>> - Link to v4: https://lore.kernel.org/all/20250926-228-reftable-introduce-consistency-checks-v4-0-c96fd8551c0d@gmail.com\n>\n> Looking good. Shall we declare victory and mark the topic for\n> 'next' now?\n>\n> Thanks.\n\nPeff pointed out a mistake in my test, where I create root refs instead\nof branches. This works without issues as we don't yet have reference\nlevel checks on reftables. While it is good as is, I do think it is\nconfusing, so will send in a new version with a fix. Let's hold out for\nthat and we can merge that to 'next'.\n\nThanks,\nKarthik\n"},{"id":"528080","messageId":"20251007-228-reftable-introduce-consistency-checks-v6-0-638cff42f0b0@gmail.com","threadId":"63987","inReplyTo":"20250819-228-reftable-introduce-consistency-checks-v1-0-8b8f6879fa9e@gmail.com","subject":"[PATCH v6 0/7] refs/reftable: add consistency checks","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-07T12:11:24Z","receivedAt":"2025-10-07T12:11:36Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The reference subsystems allows for adding backend specific consistency\nchecks. These checks are run as part of 'git refs verify'.\n\nWhile the files backend has some consistency checks added, the reftable\nbackend currently has none. This series first tightens the reftable\nbackend to make it a little more strict and then also adds the required\ninfrastructure and some simple consistency checks.\n\nSince the reftable backend is treated as a library within the Git\ncodebase, we don't want to spillover our internal fsck implementation\ninto the library. At the same time, the fsck checks need to access\ninternal structures of the reftable library which aren't exposed outside\nthe library.\n\nSo we solve this by adding a 'reftable/fsck.[ch]' which implements and\nexposes a checker for the reftable library and returns specific errors\nas defined by the library. We then add glue code within\n'refs/reftable-backend.c' to map these errors to errors which Git's fsck\nimplementation would understand. This allows us to separate concerns.\n\nWe add the following consistency checks:\n\n 1. Check for validating the reftable table name. This is treated as a\n warning since the reftable specification only suggests a table name\n but doesn't enforce it. Also there is a difference in the table name\n used in Git vs that in jGit.\n\nWe tighten the reftable backend by raising a REFTABLE_FORMAT_ERROR error\nwhen:\n\n1. The 'tables.list' file doesn't have a trailing newline.\n\n---\nChanges in v6:\n- In t/t0614-reftable-fsck.sh, create branches instead of root refs.\n This worked becuase we don't have reference level checks still\n implemented for reftables. Let's avoid confusion of a breaking test\n when we add reference level checks. \n- Link to v5: https://lore.kernel.org/r/20251006-228-reftable-introduce-consistency-checks-v5-0-f196d386214f@gmail.com\n\nChanges in v5:\n- Added documentation around the return value of 'parse_names()'.\n- Added a test to validate that 'git refs verify' doesn't barf against\n a clean working repository with multiple reftable tables.\n- Link to v4: https://lore.kernel.org/all/20250926-228-reftable-introduce-consistency-checks-v4-0-c96fd8551c0d@gmail.com\n\nChanges in v4:\n- The biggest change is to iterate over the tables in a reftable stack\n for consistency checks instead of all files inside the REFTABLE_DIR.\n This avoids all race conditions. Also, since we only check the tables\n in a stack, it no longer makes sense to check file type.\n- The discussion about update indices was concluded that tables indices\n in a stack must be strictly monotonically increasing. While modifying\n the code to do the same. I realized that we already have this check in\n 'reftable_addition_add()' where we check while adding a new table to\n the stack: `wr->min_update_index < add->next_update_index`. So I've\n dropped this patch from the series.\n- Change parse_names() to accept the output string array as an argument\n and return an error instead. This makes the flow a little easier to\n understand.\n- Link to v3: https://lore.kernel.org/r/20250918-228-reftable-introduce-consistency-checks-v3-0-271af03eb34d@gmail.com\n\nChanges in v3:\n- I took a long hiatus from this topic, mostly due to other priorities.\n This has been rebased on top of '92c87bdc40 (The eighth batch,\n 2025-09-12)' since there were conflicts.\n- Junio suggested that two of the consistency checks (trailing newlines,\n sequential update indices for tables in stack) should actually be\n checked during runtime. I have made that change in this version.\n- I've cleaned up the code and modularized the 'reftable/fsck.c' code.\n- Invalid table name emits a warning, since the reftable spec doesn't\n enforce it but only makes a suggestion.\n- Broken down the commits to make it easier to review.\n- Link to v2: https://lore.kernel.org/r/20250902-228-reftable-introduce-consistency-checks-v2-0-4f96b3834779@gmail.com\n\nChanges in v2:\n- Ensured that 'struct reftable_fsck_info' is passed around as a\n pointer, this provides a smaller footprint (pointer size vs struct\n size).\n- Run FSCK checks for other worktrees too, even if one of them fails.\n- Separate messaging for table name vs table check and add additional\n test.\n- Use the relative path in messages used.\n- Small style and typo fixes.\n- Link to v1: https://lore.kernel.org/r/20250819-228-reftable-introduce-consistency-checks-v1-0-8b8f6879fa9e@gmail.com\n\n---\n Documentation/fsck-msgids.adoc | 6 +--\n Makefile | 3 +-\n fsck.h | 39 +++++++--------\n meson.build | 1 +\n refs.c | 4 ++\n refs/debug.c | 1 -\n refs/files-backend.c | 3 --\n refs/reftable-backend.c | 58 ++++++++++++++++++++---\n reftable/basics.c | 37 ++++++++++-----\n reftable/basics.h | 7 +--\n reftable/fsck.c | 100 +++++++++++++++++++++++++++++++++++++++\n reftable/reftable-fsck.h | 40 ++++++++++++++++\n reftable/stack.c | 7 +--\n t/meson.build | 1 +\n t/t0614-reftable-fsck.sh | 58 +++++++++++++++++++++++\n t/unit-tests/u-reftable-basics.c | 24 ++++++++--\n 16 files changed, 330 insertions(+), 59 deletions(-)\n\nKarthik Nayak (7):\n refs: remove unused headers\n refs: move consistency check msg to generic layer\n reftable: check for trailing newline in 'tables.list'\n Documentation/fsck-msgids: remove duplicate msg id\n fsck: order 'fsck_msg_type' alphabetically\n reftable: add code to facilitate consistency checks\n refs/reftable: add fsck check for checking the table name\n\nRange-diff versus v5:\n\n1: 85480cbb60 = 1: 671a79a3af refs: remove unused headers\n2: b8fdad314a = 2: dbf9df8d3c refs: move consistency check msg to generic layer\n3: 4ce029ed8e = 3: dbc478dbe6 reftable: check for trailing newline in 'tables.list'\n4: 50655b2272 = 4: 062d66f7ed Documentation/fsck-msgids: remove duplicate msg id\n5: 0b4c2295d9 = 5: a70974a39c fsck: order 'fsck_msg_type' alphabetically\n6: 2abcaa9b23 = 6: a1dea4335e reftable: add code to facilitate consistency checks\n7: 1f59191f22 ! 7: dcd172827b refs/reftable: add fsck check for checking the table name\n @@ t/t0614-reftable-fsck.sh (new)\n +\n +\t\tfor i in $(test_seq 20)\n +\t\tdo\n -+\t\t\tgit update-ref branch-$i HEAD || return 1\n ++\t\t\tgit update-ref refs/heads/branch-$i HEAD || return 1\n +\t\tdone &&\n +\n +\t\t# The repository should end up with multiple tables.\n\n\nbase-commit: a483264b01b977f3e65a4419103c21e6af7412a2\nchange-id: 20250714-228-reftable-introduce-consistency-checks-379ded93c544\n\nThanks\n- Karthik\n\n"},{"id":"528081","messageId":"20251007-228-reftable-introduce-consistency-checks-v6-1-638cff42f0b0@gmail.com","threadId":"63987","inReplyTo":"20251007-228-reftable-introduce-consistency-checks-v6-0-638cff42f0b0@gmail.com","subject":"[PATCH v6 1/7] refs: remove unused headers","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-07T12:11:25Z","receivedAt":"2025-10-07T12:11:36Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"In the 'refs/' namespace, some of the included header files are not\nneeded, let's remove them.\n\nSigned-off-by: Karthik Nayak \n---\n refs/debug.c | 1 -\n refs/files-backend.c | 1 -\n refs/reftable-backend.c | 1 -\n 3 files changed, 3 deletions(-)\n\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 1cb955961e..697adbd0dc 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -1,7 +1,6 @@\n #include \"git-compat-util.h\"\n #include \"hex.h\"\n #include \"refs-internal.h\"\n-#include \"string-list.h\"\n #include \"trace.h\"\n \n static struct trace_key trace_refs = TRACE_KEY_INIT(REFS);\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 1b3bf26add..d4fb033417 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -20,7 +20,6 @@\n #include \"../dir-iterator.h\"\n #include \"../lockfile.h\"\n #include \"../object.h\"\n-#include \"../object-file.h\"\n #include \"../path.h\"\n #include \"../dir.h\"\n #include \"../chdir-notify.h\"\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 9e889da2ff..2152349cb9 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -11,7 +11,6 @@\n #include \"../hex.h\"\n #include \"../iterator.h\"\n #include \"../ident.h\"\n-#include \"../lockfile.h\"\n #include \"../object.h\"\n #include \"../path.h\"\n #include \"../refs.h\"\n\n-- \n2.51.0\n\n"},{"id":"528082","messageId":"20251007-228-reftable-introduce-consistency-checks-v6-2-638cff42f0b0@gmail.com","threadId":"63987","inReplyTo":"20251007-228-reftable-introduce-consistency-checks-v6-0-638cff42f0b0@gmail.com","subject":"[PATCH v6 2/7] refs: move consistency check msg to generic layer","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-07T12:11:26Z","receivedAt":"2025-10-07T12:11:37Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The files-backend prints a message before the consistency checks run.\nMove this to the generic layer so both the files and reftable backend\ncan benefit from this message.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 4 ++++\n refs/files-backend.c | 2 --\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 4ff55cf24f..4a7c394226 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -32,6 +32,7 @@\n #include \"commit.h\"\n #include \"wildmatch.h\"\n #include \"ident.h\"\n+#include \"fsck.h\"\n \n /*\n * List of all available backends\n@@ -323,6 +324,9 @@ int check_refname_format(const char *refname, int flags)\n int refs_fsck(struct ref_store *refs, struct fsck_options *o,\n \t struct worktree *wt)\n {\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, _(\"Checking references consistency\"));\n+\n \treturn refs->be->fsck(refs, o, wt);\n }\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex d4fb033417..603b1343d8 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3906,8 +3906,6 @@ static int files_fsck_refs(struct ref_store *ref_store,\n \t\tNULL,\n \t};\n \n-\tif (o->verbose)\n-\t\tfprintf_ln(stderr, _(\"Checking references consistency\"));\n \treturn files_fsck_refs_dir(ref_store, o, \"refs\", wt, fsck_refs_fn);\n }\n \n\n-- \n2.51.0\n\n"},{"id":"528084","messageId":"20251007-228-reftable-introduce-consistency-checks-v6-3-638cff42f0b0@gmail.com","threadId":"63987","inReplyTo":"20251007-228-reftable-introduce-consistency-checks-v6-0-638cff42f0b0@gmail.com","subject":"[PATCH v6 3/7] reftable: check for trailing newline in 'tables.list'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-07T12:11:27Z","receivedAt":"2025-10-07T12:11:38Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"In the reftable format, the 'tables.list' file contains a\nnewline separated list of tables. While we parse this file, we do not\ncheck or care about the last newline. Tighten the parser in\n`parse_names()` to return an appropriate error if the last newline is\nmissing.\n\nThis requires modification to `parse_names()` to now return the error\nwhile accepting the output as a third argument.\n\nSigned-off-by: Karthik Nayak \n---\n reftable/basics.c | 37 ++++++++++++++++++++++++-------------\n reftable/basics.h | 7 ++++---\n reftable/stack.c | 7 +------\n t/unit-tests/u-reftable-basics.c | 24 ++++++++++++++++++++----\n 4 files changed, 49 insertions(+), 26 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 9988ebd635..e969927b61 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -195,44 +195,55 @@ size_t names_length(const char **names)\n \treturn p - names;\n }\n \n-char **parse_names(char *buf, int size)\n+int parse_names(char *buf, int size, char ***out)\n {\n \tchar **names = NULL;\n \tsize_t names_cap = 0;\n \tsize_t names_len = 0;\n \tchar *p = buf;\n \tchar *end = buf + size;\n+\tint err = 0;\n \n \twhile (p < end) {\n \t\tchar *next = strchr(p, '\\n');\n-\t\tif (next && next < end) {\n-\t\t\t*next = 0;\n+\t\tif (!next) {\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n+\t\t} else if (next < end) {\n+\t\t\t*next = '\\0';\n \t\t} else {\n \t\t\tnext = end;\n \t\t}\n+\n \t\tif (p < next) {\n \t\t\tif (REFTABLE_ALLOC_GROW(names, names_len + 1,\n-\t\t\t\t\t\tnames_cap))\n-\t\t\t\tgoto err;\n+\t\t\t\t\t\tnames_cap)) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n \n \t\t\tnames[names_len] = reftable_strdup(p);\n-\t\t\tif (!names[names_len++])\n-\t\t\t\tgoto err;\n+\t\t\tif (!names[names_len++]) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n \t\t}\n \t\tp = next + 1;\n \t}\n \n-\tif (REFTABLE_ALLOC_GROW(names, names_len + 1, names_cap))\n-\t\tgoto err;\n+\tif (REFTABLE_ALLOC_GROW(names, names_len + 1, names_cap)) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tnames[names_len] = NULL;\n \n-\treturn names;\n-\n-err:\n+\t*out = names;\n+\treturn 0;\n+done:\n \tfor (size_t i = 0; i < names_len; i++)\n \t\treftable_free(names[i]);\n \treftable_free(names);\n-\treturn NULL;\n+\treturn err;\n }\n \n int names_equal(const char **a, const char **b)\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 7d22f96261..e4b83b2b03 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -167,10 +167,11 @@ void free_names(char **a);\n \n /*\n * Parse a newline separated list of names. `size` is the length of the buffer,\n- * without terminating '\\0'. Empty names are discarded. Returns a `NULL`\n- * pointer when allocations fail.\n+ * without terminating '\\0'. Empty names are discarded.\n+ *\n+ * Returns 0 on success, a reftable error code on error.\n */\n-char **parse_names(char *buf, int size);\n+int parse_names(char *buf, int size, char ***out);\n \n /* compares two NULL-terminated arrays of strings. */\n int names_equal(const char **a, const char **b);\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex f91ce50bcd..65d89820bd 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -109,12 +109,7 @@ static int fd_read_lines(int fd, char ***namesp)\n \t}\n \tbuf[size] = 0;\n \n-\t*namesp = parse_names(buf, size);\n-\tif (!*namesp) {\n-\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n-\t\tgoto done;\n-\t}\n-\n+\terr = parse_names(buf, size, namesp);\n done:\n \treftable_free(buf);\n \treturn err;\ndiff --git a/t/unit-tests/u-reftable-basics.c b/t/unit-tests/u-reftable-basics.c\nindex a0471083e7..73566ed0eb 100644\n--- a/t/unit-tests/u-reftable-basics.c\n+++ b/t/unit-tests/u-reftable-basics.c\n@@ -9,6 +9,7 @@ license that can be found in the LICENSE file or at\n #include \"unit-test.h\"\n #include \"lib-reftable.h\"\n #include \"reftable/basics.h\"\n+#include \"reftable/reftable-error.h\"\n \n struct integer_needle_lesseq_args {\n \tint needle;\n@@ -79,14 +80,18 @@ void test_reftable_basics__names_equal(void)\n void test_reftable_basics__parse_names(void)\n {\n \tchar in1[] = \"line\\n\";\n-\tchar in2[] = \"a\\nb\\nc\";\n-\tchar **out = parse_names(in1, strlen(in1));\n+\tchar in2[] = \"a\\nb\\nc\\n\";\n+\tchar **out = NULL;\n+\tint err = parse_names(in1, strlen(in1), &out);\n+\tcl_assert(err == 0);\n \tcl_assert(out != NULL);\n \tcl_assert_equal_s(out[0], \"line\");\n \tcl_assert(!out[1]);\n \tfree_names(out);\n \n-\tout = parse_names(in2, strlen(in2));\n+\tout = NULL;\n+\terr = parse_names(in2, strlen(in2), &out);\n+\tcl_assert(err == 0);\n \tcl_assert(out != NULL);\n \tcl_assert_equal_s(out[0], \"a\");\n \tcl_assert_equal_s(out[1], \"b\");\n@@ -95,10 +100,21 @@ void test_reftable_basics__parse_names(void)\n \tfree_names(out);\n }\n \n+void test_reftable_basics__parse_names_missing_newline(void)\n+{\n+\tchar in1[] = \"line\\nline2\";\n+\tchar **out = NULL;\n+\tint err = parse_names(in1, strlen(in1), &out);\n+\tcl_assert(err == REFTABLE_FORMAT_ERROR);\n+\tcl_assert(out == NULL);\n+}\n+\n void test_reftable_basics__parse_names_drop_empty_string(void)\n {\n \tchar in[] = \"a\\n\\nb\\n\";\n-\tchar **out = parse_names(in, strlen(in));\n+\tchar **out = NULL;\n+\tint err = parse_names(in, strlen(in), &out);\n+\tcl_assert(err == 0);\n \tcl_assert(out != NULL);\n \tcl_assert_equal_s(out[0], \"a\");\n \t/* simply '\\n' should be dropped as empty string */\n\n-- \n2.51.0\n\n"},{"id":"528083","messageId":"20251007-228-reftable-introduce-consistency-checks-v6-4-638cff42f0b0@gmail.com","threadId":"63987","inReplyTo":"20251007-228-reftable-introduce-consistency-checks-v6-0-638cff42f0b0@gmail.com","subject":"[PATCH v6 4/7] Documentation/fsck-msgids: remove duplicate msg id","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-07T12:11:28Z","receivedAt":"2025-10-07T12:11:39Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `gitmodulesLarge` is repeated twice. Remove the second duplicate.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/fsck-msgids.adoc | 3 ---\n 1 file changed, 3 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.adoc b/Documentation/fsck-msgids.adoc\nindex 0ba4f9a27e..1c912615f9 100644\n--- a/Documentation/fsck-msgids.adoc\n+++ b/Documentation/fsck-msgids.adoc\n@@ -104,9 +104,6 @@\n `gitmodulesParse`::\n \t(INFO) Could not parse `.gitmodules` blob.\n \n-`gitmodulesLarge`;\n-\t(ERROR) `.gitmodules` blob is too large to parse.\n-\n `gitmodulesPath`::\n \t(ERROR) `.gitmodules` path is invalid.\n \n\n-- \n2.51.0\n\n"},{"id":"528085","messageId":"20251007-228-reftable-introduce-consistency-checks-v6-5-638cff42f0b0@gmail.com","threadId":"63987","inReplyTo":"20251007-228-reftable-introduce-consistency-checks-v6-0-638cff42f0b0@gmail.com","subject":"[PATCH v6 5/7] fsck: order 'fsck_msg_type' alphabetically","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-07T12:11:29Z","receivedAt":"2025-10-07T12:11:40Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The list of 'fsck_msg_type' seem to be alphabetically ordered, but there\nare a few small misses. Fix this by sorting the sub-sections of the\nlist to maintain alphabetical ordering.\n\nSigned-off-by: Karthik Nayak \n---\n fsck.h | 38 +++++++++++++++++++-------------------\n 1 file changed, 19 insertions(+), 19 deletions(-)\n\ndiff --git a/fsck.h b/fsck.h\nindex dd7df3d5b3..6b0db235e0 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -33,15 +33,27 @@ enum fsck_msg_type {\n \tFUNC(BAD_PACKED_REF_ENTRY, ERROR) \\\n \tFUNC(BAD_PACKED_REF_HEADER, ERROR) \\\n \tFUNC(BAD_PARENT_SHA1, ERROR) \\\n+\tFUNC(BAD_REFERENT_NAME, ERROR) \\\n \tFUNC(BAD_REF_CONTENT, ERROR) \\\n \tFUNC(BAD_REF_FILETYPE, ERROR) \\\n \tFUNC(BAD_REF_NAME, ERROR) \\\n-\tFUNC(BAD_REFERENT_NAME, ERROR) \\\n \tFUNC(BAD_TIMEZONE, ERROR) \\\n \tFUNC(BAD_TREE, ERROR) \\\n \tFUNC(BAD_TREE_SHA1, ERROR) \\\n \tFUNC(BAD_TYPE, ERROR) \\\n \tFUNC(DUPLICATE_ENTRIES, ERROR) \\\n+\tFUNC(GITATTRIBUTES_BLOB, ERROR) \\\n+\tFUNC(GITATTRIBUTES_LARGE, ERROR) \\\n+\tFUNC(GITATTRIBUTES_LINE_LENGTH, ERROR) \\\n+\tFUNC(GITATTRIBUTES_MISSING, ERROR) \\\n+\tFUNC(GITMODULES_BLOB, ERROR) \\\n+\tFUNC(GITMODULES_LARGE, ERROR) \\\n+\tFUNC(GITMODULES_MISSING, ERROR) \\\n+\tFUNC(GITMODULES_NAME, ERROR) \\\n+\tFUNC(GITMODULES_PATH, ERROR) \\\n+\tFUNC(GITMODULES_SYMLINK, ERROR) \\\n+\tFUNC(GITMODULES_UPDATE, ERROR) \\\n+\tFUNC(GITMODULES_URL, ERROR) \\\n \tFUNC(MISSING_AUTHOR, ERROR) \\\n \tFUNC(MISSING_COMMITTER, ERROR) \\\n \tFUNC(MISSING_EMAIL, ERROR) \\\n@@ -60,39 +72,27 @@ enum fsck_msg_type {\n \tFUNC(TREE_NOT_SORTED, ERROR) \\\n \tFUNC(UNKNOWN_TYPE, ERROR) \\\n \tFUNC(ZERO_PADDED_DATE, ERROR) \\\n-\tFUNC(GITMODULES_MISSING, ERROR) \\\n-\tFUNC(GITMODULES_BLOB, ERROR) \\\n-\tFUNC(GITMODULES_LARGE, ERROR) \\\n-\tFUNC(GITMODULES_NAME, ERROR) \\\n-\tFUNC(GITMODULES_SYMLINK, ERROR) \\\n-\tFUNC(GITMODULES_URL, ERROR) \\\n-\tFUNC(GITMODULES_PATH, ERROR) \\\n-\tFUNC(GITMODULES_UPDATE, ERROR) \\\n-\tFUNC(GITATTRIBUTES_MISSING, ERROR) \\\n-\tFUNC(GITATTRIBUTES_LARGE, ERROR) \\\n-\tFUNC(GITATTRIBUTES_LINE_LENGTH, ERROR) \\\n-\tFUNC(GITATTRIBUTES_BLOB, ERROR) \\\n \t/* warnings */ \\\n \tFUNC(EMPTY_NAME, WARN) \\\n \tFUNC(FULL_PATHNAME, WARN) \\\n \tFUNC(HAS_DOT, WARN) \\\n \tFUNC(HAS_DOTDOT, WARN) \\\n \tFUNC(HAS_DOTGIT, WARN) \\\n+\tFUNC(LARGE_PATHNAME, WARN) \\\n \tFUNC(NULL_SHA1, WARN) \\\n-\tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \tFUNC(NUL_IN_COMMIT, WARN) \\\n-\tFUNC(LARGE_PATHNAME, WARN) \\\n+\tFUNC(ZERO_PADDED_FILEMODE, WARN) \\\n \t/* infos (reported as warnings, but ignored by default) */ \\\n \tFUNC(BAD_FILEMODE, INFO) \\\n+\tFUNC(BAD_TAG_NAME, INFO) \\\n \tFUNC(EMPTY_PACKED_REFS_FILE, INFO) \\\n-\tFUNC(GITMODULES_PARSE, INFO) \\\n-\tFUNC(GITIGNORE_SYMLINK, INFO) \\\n \tFUNC(GITATTRIBUTES_SYMLINK, INFO) \\\n+\tFUNC(GITIGNORE_SYMLINK, INFO) \\\n+\tFUNC(GITMODULES_PARSE, INFO) \\\n \tFUNC(MAILMAP_SYMLINK, INFO) \\\n-\tFUNC(BAD_TAG_NAME, INFO) \\\n \tFUNC(MISSING_TAGGER_ENTRY, INFO) \\\n-\tFUNC(SYMLINK_REF, INFO) \\\n \tFUNC(REF_MISSING_NEWLINE, INFO) \\\n+\tFUNC(SYMLINK_REF, INFO) \\\n \tFUNC(SYMREF_TARGET_IS_NOT_A_REF, INFO) \\\n \tFUNC(TRAILING_REF_CONTENT, INFO) \\\n \t/* ignored (elevated when requested) */ \\\n\n-- \n2.51.0\n\n"},{"id":"528086","messageId":"20251007-228-reftable-introduce-consistency-checks-v6-6-638cff42f0b0@gmail.com","threadId":"63987","inReplyTo":"20251007-228-reftable-introduce-consistency-checks-v6-0-638cff42f0b0@gmail.com","subject":"[PATCH v6 6/7] reftable: add code to facilitate consistency checks","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-07T12:11:30Z","receivedAt":"2025-10-07T12:11:41Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git refs verify` command is used to run consistency checks on the\nreference backends. This command is also invoked when users run 'git\nfsck'. While the files-backend has some fsck checks added, the reftable\nbackend lacks such checks. Let's add the required infrastructure and a\ncheck to test for the files present in the reftable directory.\n\nSince the reftable library is treated as an independent library we\nshould ensure that the library code works independently without\nknowledge about Git's internals. To do this, add both 'reftable/fsck.c'\nand 'reftable/reftable-fsck.h'. Which provide an entry point\n'reftable_fsck_check' for running fsck checks over a provided reftable\nstack. The callee provides the function with callbacks to handle issue\nand information reporting.\n\nThe added check, goes over all tables in the reftable stack validates\nthat they have a valid name. It not, it raises an error.\n\nWhile here, move 'reftable/error.o' in the Makefile to retain\nlexicographic ordering.\n\nSigned-off-by: Karthik Nayak \n---\n Makefile | 3 +-\n meson.build | 1 +\n reftable/fsck.c | 100 +++++++++++++++++++++++++++++++++++++++++++++++\n reftable/reftable-fsck.h | 40 +++++++++++++++++++\n 4 files changed, 143 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex 4c95affadb..03fbaf2b21 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2732,9 +2732,10 @@ XDIFF_OBJS += xdiff/xutils.o\n xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n-REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n+REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/fsck.o\n REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\ndiff --git a/meson.build b/meson.build\nindex b3dfcc0497..8914252910 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -452,6 +452,7 @@ libgit_sources = [\n 'reftable/error.c',\n 'reftable/block.c',\n 'reftable/blocksource.c',\n+ 'reftable/fsck.c',\n 'reftable/iter.c',\n 'reftable/merged.c',\n 'reftable/pq.c',\ndiff --git a/reftable/fsck.c b/reftable/fsck.c\nnew file mode 100644\nindex 0000000000..26b9115b14\n--- /dev/null\n+++ b/reftable/fsck.c\n@@ -0,0 +1,100 @@\n+#include \"basics.h\"\n+#include \"reftable-fsck.h\"\n+#include \"reftable-table.h\"\n+#include \"stack.h\"\n+\n+static bool table_has_valid_name(const char *name)\n+{\n+\tconst char *ptr = name;\n+\tchar *endptr;\n+\n+\t/* strtoull doesn't set errno on success */\n+\terrno = 0;\n+\n+\tstrtoull(ptr, &endptr, 16);\n+\tif (errno)\n+\t\treturn false;\n+\tptr = endptr;\n+\n+\tif (*ptr != '-')\n+\t\treturn false;\n+\tptr++;\n+\n+\tstrtoull(ptr, &endptr, 16);\n+\tif (errno)\n+\t\treturn false;\n+\tptr = endptr;\n+\n+\tif (*ptr != '-')\n+\t\treturn false;\n+\tptr++;\n+\n+\tstrtoul(ptr, &endptr, 16);\n+\tif (errno)\n+\t\treturn false;\n+\tptr = endptr;\n+\n+\tif (strcmp(ptr, \".ref\") && strcmp(ptr, \".log\"))\n+\t\treturn false;\n+\n+\treturn true;\n+}\n+\n+typedef int (*table_check_fn)(struct reftable_table *table,\n+\t\t\t reftable_fsck_report_fn report_fn,\n+\t\t\t void *cb_data);\n+\n+static int table_check_name(struct reftable_table *table,\n+\t\t\t reftable_fsck_report_fn report_fn,\n+\t\t\t void *cb_data)\n+{\n+\tif (!table_has_valid_name(table->name)) {\n+\t\tstruct reftable_fsck_info info;\n+\n+\t\tinfo.error = REFTABLE_FSCK_ERROR_TABLE_NAME;\n+\t\tinfo.msg = \"invalid reftable table name\";\n+\t\tinfo.path = table->name;\n+\n+\t\treturn report_fn(&info, cb_data);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int table_checks(struct reftable_table *table,\n+\t\t\treftable_fsck_report_fn report_fn,\n+\t\t\treftable_fsck_verbose_fn verbose_fn UNUSED,\n+\t\t\tvoid *cb_data)\n+{\n+\ttable_check_fn table_check_fns[] = {\n+\t\ttable_check_name,\n+\t\tNULL,\n+\t};\n+\tint err = 0;\n+\n+\tfor (size_t i = 0; table_check_fns[i]; i++)\n+\t\terr |= table_check_fns[i](table, report_fn, cb_data);\n+\n+\treturn err;\n+}\n+\n+int reftable_fsck_check(struct reftable_stack *stack,\n+\t\t\treftable_fsck_report_fn report_fn,\n+\t\t\treftable_fsck_verbose_fn verbose_fn,\n+\t\t\tvoid *cb_data)\n+{\n+\tstruct reftable_buf msg = REFTABLE_BUF_INIT;\n+\tint err = 0;\n+\n+\tfor (size_t i = 0; i < stack->tables_len; i++) {\n+\t\treftable_buf_reset(&msg);\n+\t\treftable_buf_addstr(&msg, \"Checking table: \");\n+\t\treftable_buf_addstr(&msg, stack->tables[i]->name);\n+\t\tverbose_fn(msg.buf, cb_data);\n+\n+\t\terr |= table_checks(stack->tables[i], report_fn, verbose_fn, cb_data);\n+\t}\n+\n+\treftable_buf_release(&msg);\n+\treturn err;\n+}\ndiff --git a/reftable/reftable-fsck.h b/reftable/reftable-fsck.h\nnew file mode 100644\nindex 0000000000..007a392cf9\n--- /dev/null\n+++ b/reftable/reftable-fsck.h\n@@ -0,0 +1,40 @@\n+#ifndef REFTABLE_FSCK_H\n+#define REFTABLE_FSCK_H\n+\n+#include \"reftable-stack.h\"\n+\n+enum reftable_fsck_error {\n+\t/* Invalid table name */\n+\tREFTABLE_FSCK_ERROR_TABLE_NAME = 0,\n+\t/* Used for bounds checking, must be last */\n+\tREFTABLE_FSCK_MAX_VALUE,\n+};\n+\n+/* Represents an individual error encountered during the FSCK checks. */\n+struct reftable_fsck_info {\n+\tenum reftable_fsck_error error;\n+\tconst char *msg;\n+\tconst char *path;\n+};\n+\n+typedef int reftable_fsck_report_fn(struct reftable_fsck_info *info,\n+\t\t\t\t void *cb_data);\n+typedef void reftable_fsck_verbose_fn(const char *msg, void *cb_data);\n+\n+/*\n+ * Given a reftable stack, perform consistency checks on the stack.\n+ *\n+ * If an issue is encountered, the issue is reported to the callee via the\n+ * provided 'report_fn'. If the issue is non-recoverable the flow will not\n+ * continue. If it is recoverable, the flow will continue and further issues\n+ * will be reported as identified.\n+ *\n+ * The 'verbose_fn' will be invoked to provide verbose information about\n+ * the progress and state of the consistency checks.\n+ */\n+int reftable_fsck_check(struct reftable_stack *stack,\n+\t\t\treftable_fsck_report_fn report_fn,\n+\t\t\treftable_fsck_verbose_fn verbose_fn,\n+\t\t\tvoid *cb_data);\n+\n+#endif /* REFTABLE_FSCK_H */\n\n-- \n2.51.0\n\n"},{"id":"528087","messageId":"20251007-228-reftable-introduce-consistency-checks-v6-7-638cff42f0b0@gmail.com","threadId":"63987","inReplyTo":"20251007-228-reftable-introduce-consistency-checks-v6-0-638cff42f0b0@gmail.com","subject":"[PATCH v6 7/7] refs/reftable: add fsck check for checking the table name","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-07T12:11:31Z","receivedAt":"2025-10-07T12:11:42Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Add glue code in 'refs/reftable-backend.c' which calls the reftable\nlibrary to perform the fsck checks. Here we also map the reftable errors\nto Git' fsck errors.\n\nIntroduce a check to validate table names for a given reftable stack.\nAlso add 'badReftableTableName' as a corresponding error within Git. The\nreftable specification mentions:\n\n It suggested to use\n ${min_update_index}-${max_update_index}-${random}.ref as a naming\n convention.\n\nSo treat non-conformant file names as warnings.\n\nWhile adding the fsck header to 'refs/reftable-backend.c', modify the\nlist to maintain lexicographical ordering.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/fsck-msgids.adoc | 3 +++\n fsck.h | 1 +\n refs/reftable-backend.c | 57 +++++++++++++++++++++++++++++++++++++----\n t/meson.build | 1 +\n t/t0614-reftable-fsck.sh | 58 ++++++++++++++++++++++++++++++++++++++++++\n 5 files changed, 115 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/fsck-msgids.adoc b/Documentation/fsck-msgids.adoc\nindex 1c912615f9..81f11ba125 100644\n--- a/Documentation/fsck-msgids.adoc\n+++ b/Documentation/fsck-msgids.adoc\n@@ -38,6 +38,9 @@\n `badReferentName`::\n \t(ERROR) The referent name of a symref is invalid.\n \n+`badReftableTableName`::\n+\t(WARN) A reftable table has an invalid name.\n+\n `badTagName`::\n \t(INFO) A tag has an invalid format.\n \ndiff --git a/fsck.h b/fsck.h\nindex 6b0db235e0..759df97655 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -73,6 +73,7 @@ enum fsck_msg_type {\n \tFUNC(UNKNOWN_TYPE, ERROR) \\\n \tFUNC(ZERO_PADDED_DATE, ERROR) \\\n \t/* warnings */ \\\n+\tFUNC(BAD_REFTABLE_TABLE_NAME, WARN) \\\n \tFUNC(EMPTY_NAME, WARN) \\\n \tFUNC(FULL_PATHNAME, WARN) \\\n \tFUNC(HAS_DOT, WARN) \\\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 2152349cb9..b106fd8b53 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -6,6 +6,7 @@\n #include \"../config.h\"\n #include \"../dir.h\"\n #include \"../environment.h\"\n+#include \"../fsck.h\"\n #include \"../gettext.h\"\n #include \"../hash.h\"\n #include \"../hex.h\"\n@@ -15,10 +16,11 @@\n #include \"../path.h\"\n #include \"../refs.h\"\n #include \"../reftable/reftable-basics.h\"\n-#include \"../reftable/reftable-stack.h\"\n-#include \"../reftable/reftable-record.h\"\n #include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-fsck.h\"\n #include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-stack.h\"\n #include \"../repo-settings.h\"\n #include \"../setup.h\"\n #include \"../strmap.h\"\n@@ -2707,11 +2709,56 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \treturn ret;\n }\n \n-static int reftable_be_fsck(struct ref_store *ref_store UNUSED,\n-\t\t\t struct fsck_options *o UNUSED,\n+static void reftable_fsck_verbose_handler(const char *msg, void *cb_data)\n+{\n+\tstruct fsck_options *o = cb_data;\n+\n+\tif (o->verbose)\n+\t\tfprintf_ln(stderr, \"%s\", msg);\n+}\n+\n+static const enum fsck_msg_id fsck_msg_id_map[] = {\n+\t[REFTABLE_FSCK_ERROR_TABLE_NAME] = FSCK_MSG_BAD_REFTABLE_TABLE_NAME,\n+};\n+\n+static int reftable_fsck_error_handler(struct reftable_fsck_info *info,\n+\t\t\t\t void *cb_data)\n+{\n+\tstruct fsck_ref_report report = { .path = info->path };\n+\tstruct fsck_options *o = cb_data;\n+\tenum fsck_msg_id msg_id;\n+\n+\tif (info->error < 0 || info->error >= REFTABLE_FSCK_MAX_VALUE)\n+\t\tBUG(\"unknown fsck error: %d\", (int)info->error);\n+\n+\tmsg_id = fsck_msg_id_map[info->error];\n+\n+\tif (!msg_id)\n+\t\tBUG(\"fsck_msg_id value missing for reftable error: %d\", (int)info->error);\n+\n+\treturn fsck_report_ref(o, &report, msg_id, \"%s\", info->msg);\n+}\n+\n+static int reftable_be_fsck(struct ref_store *ref_store, struct fsck_options *o,\n \t\t\t struct worktree *wt UNUSED)\n {\n-\treturn 0;\n+\tstruct reftable_ref_store *refs;\n+\tstruct strmap_entry *entry;\n+\tstruct hashmap_iter iter;\n+\tint ret = 0;\n+\n+\trefs = reftable_be_downcast(ref_store, REF_STORE_READ, \"fsck\");\n+\n+\tret |= reftable_fsck_check(refs->main_backend.stack, reftable_fsck_error_handler,\n+\t\t\t\t reftable_fsck_verbose_handler, o);\n+\n+\tstrmap_for_each_entry(&refs->worktree_backends, &iter, entry) {\n+\t\tstruct reftable_backend *b = (struct reftable_backend *)entry->value;\n+\t\tret |= reftable_fsck_check(b->stack, reftable_fsck_error_handler,\n+\t\t\t\t\t reftable_fsck_verbose_handler, o);\n+\t}\n+\n+\treturn ret;\n }\n \n struct ref_storage_be refs_be_reftable = {\ndiff --git a/t/meson.build b/t/meson.build\nindex 7974795fe4..ec1fc0b2a1 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -146,6 +146,7 @@ integration_tests = [\n 't0611-reftable-httpd.sh',\n 't0612-reftable-jgit-compatibility.sh',\n 't0613-reftable-write-options.sh',\n+ 't0614-reftable-fsck.sh',\n 't1000-read-tree-m-3way.sh',\n 't1001-read-tree-m-2way.sh',\n 't1002-read-tree-m-u-2way.sh',\ndiff --git a/t/t0614-reftable-fsck.sh b/t/t0614-reftable-fsck.sh\nnew file mode 100755\nindex 0000000000..85cc47d67e\n--- /dev/null\n+++ b/t/t0614-reftable-fsck.sh\n@@ -0,0 +1,58 @@\n+#!/bin/sh\n+\n+test_description='Test reftable backend consistency check'\n+\n+GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+. ./test-lib.sh\n+\n+test_expect_success \"no errors reported on a well formed repository\" '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit commit --allow-empty -m initial &&\n+\n+\t\tfor i in $(test_seq 20)\n+\t\tdo\n+\t\t\tgit update-ref refs/heads/branch-$i HEAD || return 1\n+\t\tdone &&\n+\n+\t\t# The repository should end up with multiple tables.\n+\t\ttest_line_count \">\" 1 .git/reftable/tables.list &&\n+\n+\t\tgit refs verify 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+for TABLE_NAME in \"foo-bar-e4d12d59.ref\" \\\n+\t\"0x00000000zzzz-0x00000000zzzz-e4d12d59.ref\" \\\n+\t\"0x000000000001-0x000000000002-e4d12d59.abc\" \\\n+\t\"0x000000000001-0x000000000002-e4d12d59.refabc\"; do\n+\ttest_expect_success \"table name $TABLE_NAME should be checked\" '\n+\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\tgit init repo &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\tgit commit --allow-empty -m initial &&\n+\n+\t\t\tgit refs verify 2>err &&\n+\t\t\ttest_must_be_empty err &&\n+\n+\t\t\tEXISTING_TABLE=$(head -n1 .git/reftable/tables.list) &&\n+\t\t\tmv \".git/reftable/$EXISTING_TABLE\" \".git/reftable/$TABLE_NAME\" &&\n+\t\t\tsed \"s/${EXISTING_TABLE}/${TABLE_NAME}/g\" .git/reftable/tables.list > tables.list &&\n+\t\t\tmv tables.list .git/reftable/tables.list &&\n+\n+\t\t\tgit refs verify 2>err &&\n+\t\t\tcat >expect <<-EOF &&\n+\t\t\twarning: ${TABLE_NAME}: badReftableTableName: invalid reftable table name\n+\t\t\tEOF\n+\t\t\ttest_cmp expect err\n+\t\t)\n+\t'\n+done\n+\n+test_done\n\n-- \n2.51.0\n\n"},{"id":"528115","messageId":"aOUVB8hfnYiBm1V2@pks.im","threadId":"63987","inReplyTo":"20251007-228-reftable-introduce-consistency-checks-v6-0-638cff42f0b0@gmail.com","subject":"Re: [PATCH v6 0/7] refs/reftable: add consistency checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-07T13:26:31Z","receivedAt":"2025-10-07T13:26:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 07, 2025 at 02:11:24PM +0200, Karthik Nayak wrote:\n> Changes in v6:\n> - In t/t0614-reftable-fsck.sh, create branches instead of root refs.\n> This worked becuase we don't have reference level checks still\n> implemented for reftables. Let's avoid confusion of a breaking test\n> when we add reference level checks. \n> - Link to v5: https://lore.kernel.org/r/20251006-228-reftable-introduce-consistency-checks-v5-0-f196d386214f@gmail.com\n\nThanks, this version looks good to me!\n\nPatrick\n"},{"id":"528125","messageId":"xmqqecren3va.fsf@gitster.g","threadId":"63987","inReplyTo":"CAOLa=ZRToJOUd_Devs54NfLCJuMZSBEwNcM8J0sbvy7x5Rb=PA@mail.gmail.com","subject":"Re: [PATCH v5 0/7] refs/reftable: add consistency checks","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-10-07T15:11:37Z","receivedAt":"2025-10-07T15:11:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Peff pointed out a mistake in my test, where I create root refs instead\n> of branches. This works without issues as we don't yet have reference\n> level checks on reftables. While it is good as is, I do think it is\n> confusing, so will send in a new version with a fix. Let's hold out for\n> that and we can merge that to 'next'.\n\nYup, I saw that exchange on the \"oops you missed refs/heads/!\",\nwhich I also missed. Thanks, let me mark it to expect an update.\n"},{"id":"528131","messageId":"xmqqfrbullvm.fsf@gitster.g","threadId":"63987","inReplyTo":"aOUVB8hfnYiBm1V2@pks.im","subject":"Re: [PATCH v6 0/7] refs/reftable: add consistency checks","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-10-07T16:25:33Z","receivedAt":"2025-10-07T16:25:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Tue, Oct 07, 2025 at 02:11:24PM +0200, Karthik Nayak wrote:\n>> Changes in v6:\n>> - In t/t0614-reftable-fsck.sh, create branches instead of root refs.\n>> This worked becuase we don't have reference level checks still\n>> implemented for reftables. Let's avoid confusion of a breaking test\n>> when we add reference level checks. \n>> - Link to v5: https://lore.kernel.org/r/20251006-228-reftable-introduce-consistency-checks-v5-0-f196d386214f@gmail.com\n>\n> Thanks, this version looks good to me!\n\nThanks, all.\n"}]}