{"thread":{"id":"66784","subject":"[PATCH 0/5] Improvements for storage format handling","startedAt":"2026-11-05T15:02:01Z","lastAt":"2026-11-09T07:31:26Z","messageCount":13,"participants":["Patrick Steinhardt","Karthik Nayak","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"557557","messageId":"20261105-pks-storage-format-improvements-v1-0-870bb4212518@pks.im","threadId":"66784","inReplyTo":null,"subject":"[PATCH 0/5] Improvements for storage format handling","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-05T15:02:01Z","receivedAt":"2026-11-05T15:02:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series includes a couple of improvements for how we handle\nref and object storage formats.\n\nMost importantly, this series allows us to run tests with a different\nobject storage backend via `GIT_TEST_DEFAULT_OBJECT_STORAGE_FORMAT=`. As\nwe don't have an alternative backend, I also introduce the ability to\nspecify a default object (and ref) storage format that carries a payload\nso that you can run all tests with \"files://objdir\", for example. That\nsurfaces a bunch of tests that assume objects exist in \".git/objects\".\n\nOverall it's a surprisingly small number of test failures, only 1105 out\nof 35000 tests fail. Fixing those is not part of this patch series, but\nwe'll start upstreaming fixes once this lands. Such tests will either\nhave to be adapted to use plumbing commands, or they'll need to be\ntagged with a prerequisite. It's going to be a bunch of work, but it's\nbasically preparing for the introduction of alternative backends\nalready.\n\nThe series is built on top of 0ffa3e35c7c (The 15th batch, 2026-11-04)\nwith ps/odb-object-storage-extension at 87a55d790ca (builtin/submodule:\npropagate \"--object-storage-format=\", 2026-10-27) merged into it.\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (5):\n      refs: create leading directories\n      odb/files: create leading directories\n      setup: allow speciyfing payloads with default storage formats\n      builtin/rev-parse: print payload for the ref storage format\n      t: allow testing with a non-default object storage foramt\n\n builtin/rev-parse.c     |  6 +++++-\n odb/source-files.c      |  1 +\n refs/files-backend.c    |  1 +\n refs/reftable-backend.c |  1 +\n setup.c                 | 24 ++++++++++++++++-----\n t/t0001-init.sh         | 57 ++++++++++++++++++++++++++++++++++++++++++++++---\n t/t1423-ref-backend.sh  | 24 +++++++++++++++++++++\n t/t1500-rev-parse.sh    |  9 ++++++++\n t/test-lib.sh           |  4 ++++\n 9 files changed, 118 insertions(+), 9 deletions(-)\n\n\n---\nbase-commit: df904e7114cda6ca4ebf514115835a2dcd2f2b64\nchange-id: 20261105-pks-storage-format-improvements-7b716599bada\n\n\n"},{"id":"557558","messageId":"20261105-pks-storage-format-improvements-v1-1-870bb4212518@pks.im","threadId":"66784","inReplyTo":"20261105-pks-storage-format-improvements-v1-0-870bb4212518@pks.im","subject":"[PATCH 1/5] refs: create leading directories","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-05T15:02:02Z","receivedAt":"2026-11-05T15:02:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Both the \"files\" and \"reftable\" backends accept a payload that allows\nthe user to use a non-default location for their files. So e.g. if the\nuser specifies \"files://refdir\" as the ref storage format, then we would\nput files into \".git/refdir/refs\" instead of \".git/refs\".\n\nUsing this is somewhat unwieldy though, as the directory must exist\nbefore we create the ref storage as we'd otherwise die. This isn't that\nbad in the case where the user specifies an absolute path that is\noutside of the repository. But it becomes quite weird when asked to\ncreate the directory relative to commondir, as the user would first have\nto \"mkdir -p repo/.git/refdir\".\n\nAdapt the reference backends to create leading directories. Note that we\nuse the `noshare()` variant for this because it's very much possible\nthat it will exist outside of the repository, and if so we shouldn't\nadapt permissions on such leading directories.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/files-backend.c    |  1 +\n refs/reftable-backend.c |  1 +\n t/t1423-ref-backend.sh  | 24 ++++++++++++++++++++++++\n 3 files changed, 26 insertions(+)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 87d04557a53..1bdb16c0e7b 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3699,6 +3699,7 @@ static int files_ref_store_create_on_disk(struct ref_store *ref_store,\n \t *   they do not understand the reference format extension.\n \t */\n \tstrbuf_addf(&sb, \"%s/refs\", ref_store->gitdir);\n+\tsafe_create_leading_directories_no_share(sb.buf);\n \tsafe_create_dir(refs->base.repo, sb.buf, 1);\n \tadjust_shared_perm(refs->base.repo, sb.buf);\n \ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex d0de0663552..3ba6a9ac0db 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -504,6 +504,7 @@ static int reftable_be_create_on_disk(struct ref_store *ref_store,\n \tstruct strbuf sb = STRBUF_INIT;\n \n \tstrbuf_addf(&sb, \"%s/reftable\", refs->base.gitdir);\n+\tsafe_create_leading_directories_no_share(sb.buf);\n \tsafe_create_dir(ref_store->repo, sb.buf, 1);\n \tstrbuf_reset(&sb);\n \ndiff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh\nindex ee2bb66b99c..85d1e876b82 100755\n--- a/t/t1423-ref-backend.sh\n+++ b/t/t1423-ref-backend.sh\n@@ -284,6 +284,30 @@ test_expect_success 'initializing repository with --ref-storage-format and paylo\n \ttest_cmp expect actual\n '\n \n+for format in files reftable\n+do\n+\ttest_expect_success \"default ref storage format with $format backend and payload\" '\n+\t\ttest_when_finished \"rm -rf repo refdir\" &&\n+\t\tBACKEND=\"$format://$(pwd)/refdir\" &&\n+\t\techo \"$BACKEND\" >expect &&\n+\t\tgit init --ref-storage-format=\"$BACKEND\" repo &&\n+\t\ttest_path_is_dir refdir &&\n+\t\tcase \"$format\" in\n+\t\tfiles)\n+\t\t\ttest_path_is_file refdir/HEAD &&\n+\t\t\ttest_path_is_dir refdir/refs\n+\t\t\t;;\n+\t\treftable)\n+\t\t\ttest_path_is_dir refdir/reftable &&\n+\t\t\ttest_path_is_file refdir/reftable/tables.list\n+\t\t\t;;\n+\t\t*)\n+\t\t\tBUG \"unknown ref storage format: $format\"\n+\t\t\t;;\n+\t\tesac\n+\t'\n+done\n+\n test_expect_success 'cloning repository with alt ref directory' '\n \ttest_when_finished \"rm -rf source repo refdir\" &&\n \tmkdir refdir &&\n\n-- \n2.56.0.500.g5d06dc02cd.dirty\n\n\n"},{"id":"557559","messageId":"20261105-pks-storage-format-improvements-v1-2-870bb4212518@pks.im","threadId":"66784","inReplyTo":"20261105-pks-storage-format-improvements-v1-0-870bb4212518@pks.im","subject":"[PATCH 2/5] odb/files: create leading directories","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-05T15:02:03Z","receivedAt":"2026-11-05T15:02:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"With the same reasoning as in the preceding commit, create leading\ndirectories when creating the \"files\" object backend on disk.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 1 +\n t/t0001-init.sh    | 6 +++---\n 2 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 6b423499fe5..e7503548e38 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -97,6 +97,7 @@ static int odb_source_files_create_on_disk(struct odb_source *source,\n \tstruct strbuf line = STRBUF_INIT;\n \tint ret;\n \n+\tsafe_create_leading_directories_no_share(source->path);\n \tsafe_create_dir(source->odb->repo, source->path, 1);\n \n \tstrbuf_addf(&path, \"%s/pack\", source->path);\ndiff --git a/t/t0001-init.sh b/t/t0001-init.sh\nindex 23317e0733a..735ab9d8475 100755\n--- a/t/t0001-init.sh\n+++ b/t/t0001-init.sh\n@@ -944,13 +944,13 @@ test_expect_success 'init with --object-storage-format=files://<relative-path>'\n \n test_expect_success 'init with --object-storage-format=files://<absolute-path>' '\n \ttest_when_finished \"rm -rf objformat objdir\" &&\n-\tgit init --object-storage-format=files://\"$(pwd)/objdir\" objformat &&\n-\techo \"files://$(pwd)/objdir\" >expect &&\n+\tgit init --object-storage-format=files://\"$(pwd)/absolute/objdir\" objformat &&\n+\techo \"files://$(pwd)/absolute/objdir\" >expect &&\n \tgit -C objformat config get extensions.objectstorage >actual &&\n \ttest_cmp expect actual &&\n \ttest_commit -C objformat A &&\n \tgit -C objformat rev-parse --verify HEAD &&\n-\ttest_path_is_dir objdir/pack\n+\ttest_path_is_dir absolute/objdir/pack\n '\n \n test_expect_success 'GIT_OBJECT_DIRECTORY takes precedence over extensions.objectStorage payload' '\n\n-- \n2.56.0.500.g5d06dc02cd.dirty\n\n\n"},{"id":"557560","messageId":"20261105-pks-storage-format-improvements-v1-3-870bb4212518@pks.im","threadId":"66784","inReplyTo":"20261105-pks-storage-format-improvements-v1-0-870bb4212518@pks.im","subject":"[PATCH 3/5] setup: allow speciyfing payloads with default storage formats","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-05T15:02:04Z","receivedAt":"2026-11-05T15:02:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While the \"git init --ref-storage-format=\" switch allows the user to\nspecify a payload that gets passed to the backend, this isn't possible\nwhen specifying a default via \"init.defaultRefStorageFormat\" or via\n`GIT_DEFAULT_REF_STORAGE_FORMAT`. The same restriction also applies to\nthe object storage format, which needlessly restricts the flexibility\nand usefulness of these mechanism.\n\nExtend the logic to allow specifying a payload.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n setup.c         | 24 +++++++++++++++++++-----\n t/t0001-init.sh | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 70 insertions(+), 5 deletions(-)\n\ndiff --git a/setup.c b/setup.c\nindex 186815c7801..cedc031c54f 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -2797,7 +2797,9 @@ static void separate_git_dir(struct repository *repo,\n struct default_format_config {\n \tint hash;\n \tenum ref_storage_format ref_storage_format;\n+\tchar *ref_storage_payload;\n \tenum object_storage_format object_storage_format;\n+\tchar *object_storage_payload;\n };\n \n static int read_default_format_config(const char *key, const char *value,\n@@ -2823,9 +2825,12 @@ static int read_default_format_config(const char *key, const char *value,\n \t\tret = git_config_string(&str, key, value);\n \t\tif (ret)\n \t\t\tgoto out;\n-\t\tcfg->ref_storage_format = ref_storage_format_by_name(str);\n+\n+\t\tFREE_AND_NULL(cfg->ref_storage_payload);\n+\t\tcfg->ref_storage_format = ref_storage_format_by_uri(str, &cfg->ref_storage_payload);\n \t\tif (cfg->ref_storage_format == REF_STORAGE_FORMAT_UNKNOWN)\n \t\t\twarning(_(\"unknown ref storage format '%s'\"), str);\n+\n \t\tgoto out;\n \t}\n \n@@ -2833,9 +2838,12 @@ static int read_default_format_config(const char *key, const char *value,\n \t\tret = git_config_string(&str, key, value);\n \t\tif (ret)\n \t\t\tgoto out;\n-\t\tcfg->object_storage_format = object_storage_format_by_name(str);\n+\n+\t\tFREE_AND_NULL(cfg->object_storage_payload);\n+\t\tcfg->object_storage_format = object_storage_format_by_uri(str, &cfg->object_storage_payload);\n \t\tif (cfg->object_storage_format == OBJECT_STORAGE_FORMAT_UNKNOWN)\n \t\t\twarning(_(\"unknown object storage format '%s'\"), str);\n+\n \t\tgoto out;\n \t}\n \n@@ -2947,17 +2955,19 @@ static void repository_format_configure(struct repository_format *repo_fmt,\n \t\tref_storage_format = repo_fmt->ref_storage_format;\n \t\tref_storage_payload = xstrdup_or_null(repo_fmt->ref_storage_payload);\n \t} else if ((env = getenv(\"GIT_DEFAULT_REF_STORAGE_FORMAT\"))) {\n-\t\tref_storage_format = ref_storage_format_by_name(env);\n+\t\tref_storage_format = ref_storage_format_by_uri(env, &ref_storage_payload);\n \t\tif (ref_storage_format == REF_STORAGE_FORMAT_UNKNOWN)\n \t\t\tdie(_(\"unknown ref storage format specified via %s: '%s'\"),\n \t\t\t    \"GIT_DEFAULT_REF_STORAGE_FORMAT\", env);\n \t} else if ((env = getenv(\"GIT_DEFAULT_REF_FORMAT\"))) {\n-\t\tref_storage_format = ref_storage_format_by_name(env);\n+\t\tref_storage_format = ref_storage_format_by_uri(env, &ref_storage_payload);\n \t\tif (ref_storage_format == REF_STORAGE_FORMAT_UNKNOWN)\n \t\t\tdie(_(\"unknown ref storage format specified via %s: '%s'\"),\n \t\t\t    \"GIT_DEFAULT_REF_FORMAT\", env);\n \t} else if (cfg.ref_storage_format != REF_STORAGE_FORMAT_UNKNOWN) {\n \t\tref_storage_format = cfg.ref_storage_format;\n+\t\tref_storage_payload = cfg.ref_storage_payload;\n+\t\tcfg.ref_storage_payload = NULL;\n \t} else {\n \t\tref_storage_format = REF_STORAGE_FORMAT_DEFAULT;\n \t}\n@@ -2995,12 +3005,14 @@ static void repository_format_configure(struct repository_format *repo_fmt,\n \t\tobject_storage_format = repo_fmt->object_storage_format;\n \t\tobject_storage_payload = xstrdup_or_null(repo_fmt->object_storage_payload);\n \t} else if ((env = getenv(\"GIT_DEFAULT_OBJECT_STORAGE_FORMAT\"))) {\n-\t\tobject_storage_format = object_storage_format_by_name(env);\n+\t\tobject_storage_format = object_storage_format_by_uri(env, &object_storage_payload);\n \t\tif (object_storage_format == OBJECT_STORAGE_FORMAT_UNKNOWN)\n \t\t\tdie(_(\"unknown object storage format specified via %s: '%s'\"),\n \t\t\t    \"GIT_DEFAULT_OBJECT_STORAGE_FORMAT\", env);\n \t} else if (cfg.object_storage_format != OBJECT_STORAGE_FORMAT_UNKNOWN) {\n \t\tobject_storage_format = cfg.object_storage_format;\n+\t\tobject_storage_payload = cfg.object_storage_payload;\n+\t\tcfg.object_storage_payload = NULL;\n \t} else {\n \t\tobject_storage_format = OBJECT_STORAGE_FORMAT_FILES;\n \t}\n@@ -3024,10 +3036,12 @@ static void repository_format_configure(struct repository_format *repo_fmt,\n \t\t\tdie(_(\"attempt to reinitialize repository with different object storage payload\"));\n \t}\n \n+\tfree(cfg.ref_storage_payload);\n \tfree(repo_fmt->ref_storage_payload);\n \trepo_fmt->ref_storage_format = ref_storage_format;\n \trepo_fmt->ref_storage_payload = ref_storage_payload;\n \n+\tfree(cfg.object_storage_payload);\n \tfree(repo_fmt->object_storage_payload);\n \trepo_fmt->object_storage_format = object_storage_format;\n \trepo_fmt->object_storage_payload = object_storage_payload;\ndiff --git a/t/t0001-init.sh b/t/t0001-init.sh\nindex 735ab9d8475..09b7258bf10 100755\n--- a/t/t0001-init.sh\n+++ b/t/t0001-init.sh\n@@ -745,6 +745,33 @@ do\n \t'\n done\n \n+test_expect_success 'initializing repository with init.defaultRefStorageFormat and payload' '\n+\ttest_when_finished \"rm -rf repo refdir\" &&\n+\n+\tBACKEND=\"files://$(pwd)/refdir\" &&\n+\ttest_config_global init.defaultRefStorageFormat \"$BACKEND\" &&\n+\t(\n+\t\tsane_unset GIT_DEFAULT_REF_STORAGE_FORMAT &&\n+\t\tgit init repo\n+\t) &&\n+\ttest_path_is_file refdir/HEAD &&\n+\ttest_path_is_dir refdir/refs &&\n+\tgit -C repo config get extensions.refstorage >actual &&\n+\techo \"$BACKEND\" >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'initializing repository with GIT_DEFAULT_REF_STORAGE_FORMAT and payload' '\n+\ttest_when_finished \"rm -rf repo refdir\" &&\n+\tBACKEND=\"files://$(pwd)/refdir\" &&\n+\tGIT_DEFAULT_REF_STORAGE_FORMAT=\"$BACKEND\" git init repo &&\n+\ttest_path_is_file refdir/HEAD &&\n+\ttest_path_is_dir refdir/refs &&\n+\tgit -C repo config get extensions.refstorage >actual &&\n+\techo \"$BACKEND\" >expect &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success \"--ref-storage-format= overrides GIT_DEFAULT_REF_STORAGE_FOMAT\" '\n \ttest_when_finished \"rm -rf refformat\" &&\n \tGIT_DEFAULT_REF_STORAGE_FORMAT=files git init --ref-storage-format=reftable refformat &&\n@@ -1064,6 +1091,30 @@ test_expect_success 'init with init.defaultObjectStorageFormat' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'init with init.defaultObjectStorageFormat and payload' '\n+\ttest_when_finished \"rm -rf repo objdir\" &&\n+\tBACKEND=\"files://$(pwd)/objdir\" &&\n+\ttest_config_global init.defaultObjectStorageFormat \"$BACKEND\" &&\n+\t(\n+\t\tsane_unset GIT_DEFAULT_OBJECT_STORAGE_FORMAT &&\n+\t\tgit init repo\n+\t) &&\n+\ttest_path_is_dir objdir/pack &&\n+\tgit -C repo config get extensions.objectstorage >actual &&\n+\techo \"$BACKEND\" >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init with GIT_DEFAULT_OBJECT_STORAGE_FORMAT and payload' '\n+\ttest_when_finished \"rm -rf repo objdir\" &&\n+\tBACKEND=\"files://$(pwd)/objdir\" &&\n+\tGIT_DEFAULT_OBJECT_STORAGE_FORMAT=\"$BACKEND\" git init repo &&\n+\ttest_path_is_dir objdir/pack &&\n+\tgit -C repo config get extensions.objectstorage >actual &&\n+\techo \"$BACKEND\" >expect &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success MINGW 'core.hidedotfiles = false' '\n \tgit config --global core.hidedotfiles false &&\n \trm -rf newdir &&\n\n-- \n2.56.0.500.g5d06dc02cd.dirty\n\n\n"},{"id":"557561","messageId":"20261105-pks-storage-format-improvements-v1-4-870bb4212518@pks.im","threadId":"66784","inReplyTo":"20261105-pks-storage-format-improvements-v1-0-870bb4212518@pks.im","subject":"[PATCH 4/5] builtin/rev-parse: print payload for the ref storage format","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-05T15:02:05Z","receivedAt":"2026-11-05T15:02:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While we print both the object storage format and its payload for the\n\"--show-object-storage-format\" switch, we only print the ref storage\nformat with \"--show-ref-storage-format\". Adapt the latter to also show\nthe payload.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/rev-parse.c  | 6 +++++-\n t/t1500-rev-parse.sh | 9 +++++++++\n 2 files changed, 14 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/rev-parse.c b/builtin/rev-parse.c\nindex d2154eac8d2..fbc5ef87276 100644\n--- a/builtin/rev-parse.c\n+++ b/builtin/rev-parse.c\n@@ -1136,7 +1136,11 @@ int cmd_rev_parse(int argc,\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (!strcmp(arg, \"--show-ref-format\") || !strcmp(arg, \"--show-ref-storage-format\")) {\n-\t\t\t\tputs(ref_storage_format_to_name(the_repository->ref_storage_format));\n+\t\t\t\tconst char *format = ref_storage_format_to_name(the_repository->ref_storage_format);\n+\t\t\t\tif (the_repository->ref_storage_payload)\n+\t\t\t\t\tprintf(\"%s://%s\\n\", format, the_repository->ref_storage_payload);\n+\t\t\t\telse\n+\t\t\t\t\tputs(format);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (!strcmp(arg, \"--show-object-storage-format\")) {\ndiff --git a/t/t1500-rev-parse.sh b/t/t1500-rev-parse.sh\nindex 30bf8a4d0a0..c4e6bc1dd89 100755\n--- a/t/t1500-rev-parse.sh\n+++ b/t/t1500-rev-parse.sh\n@@ -209,6 +209,7 @@ test_expect_success 'rev-parse --show-object-format in repo' '\n \n \n test_expect_success RUST 'rev-parse --show-object-format in repo with compat mode' '\n+\ttest_when_finished \"rm -rf repo\" &&\n \tmkdir repo &&\n \t(\n \t\tsane_unset GIT_DEFAULT_HASH &&\n@@ -247,6 +248,14 @@ test_expect_success 'rev-parse --show-ref-storage-format' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'rev-parse --show-ref-storage-format with payload' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo --ref-storage-format=\"files://$(pwd)/refdir\" &&\n+\tgit -C repo rev-parse --show-ref-storage-format >actual &&\n+\techo \"files://$(pwd)/refdir\" >expect &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'rev-parse --show-ref-storage-format with invalid storage' '\n \ttest_when_finished \"rm -rf repo\" &&\n \tgit init repo &&\n\n-- \n2.56.0.500.g5d06dc02cd.dirty\n\n\n"},{"id":"557562","messageId":"20261105-pks-storage-format-improvements-v1-5-870bb4212518@pks.im","threadId":"66784","inReplyTo":"20261105-pks-storage-format-improvements-v1-0-870bb4212518@pks.im","subject":"[PATCH 5/5] t: allow testing with a non-default object storage foramt","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-05T15:02:06Z","receivedAt":"2026-11-05T15:02:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"It is already possible to test Git with a non-default ref storage format\nby specifying `GIT_TEST_DEFAULT_REF_STORAGE_FORMAT`. We don't provide\nthe same mechanism yet for testing the object storage format, and will\ninstead use the default (\"files\"). Even worse though, we may inherit\nwhatever the user has configured in `GIT_DEFAULT_OBJECT_STORAGE_FORMAT`.\n\nIntroduce `GIT_TEST_DEFAULT_OBJECT_STORAGE_FORMAT` and have it default\nto the \"files\" backend.\n\nNote that this environment variable isn't all that useful, as we do not\nhave an alternative backend. But what it does allow is to figure out\nwhich tests will definitely break once we do have an alternative backend\nbecause they assume that objects exist in \".git/objects\", as we can now\nmove objects into a different location by specifying a payload.\n\nSure enough, running with \"files://objdir\" shows that around 1105 or our\ntests are failing. This is a surprisingly small number of failures, but\nwe'll have to fix them if we ever want to introduce an alternative\nbackend.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n t/test-lib.sh | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex a893963f86c..7fdd8272162 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -576,9 +576,13 @@ export EDITOR\n GIT_TEST_BUILTIN_HASH=$(\"$GIT_BINARY\" version --build-options | sed -ne 's/^default-hash: //p')\n GIT_DEFAULT_HASH=\"${GIT_TEST_DEFAULT_HASH:-$GIT_TEST_BUILTIN_HASH}\"\n export GIT_DEFAULT_HASH\n+\n GIT_DEFAULT_REF_STORAGE_FORMAT=\"${GIT_TEST_DEFAULT_REF_STORAGE_FORMAT:-files}\"\n export GIT_DEFAULT_REF_STORAGE_FORMAT\n \n+GIT_DEFAULT_OBJECT_STORAGE_FORMAT=\"${GIT_TEST_DEFAULT_OBJECT_STORAGE_FORMAT:-files}\"\n+export GIT_DEFAULT_OBJECT_STORAGE_FORMAT\n+\n # Tests using GIT_TRACE typically don't want <timestamp> <file>:<line> output\n GIT_TRACE_BARE=1\n export GIT_TRACE_BARE\n\n-- \n2.56.0.500.g5d06dc02cd.dirty\n\n\n"},{"id":"557656","messageId":"CAOLa=ZR3jxonz2o+JKe=pyzSACwJH=uu3an141PEG1Ssq0pBWg@mail.gmail.com","threadId":"66784","inReplyTo":"20261105-pks-storage-format-improvements-v1-1-870bb4212518@pks.im","subject":"Re: [PATCH 1/5] refs: create leading directories","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-11-06T10:37:23Z","receivedAt":"2026-11-06T10:37:23Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\nThe patch looks good.\n\n> diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh\n> index ee2bb66b99c..85d1e876b82 100755\n> --- a/t/t1423-ref-backend.sh\n> +++ b/t/t1423-ref-backend.sh\n> @@ -284,6 +284,30 @@ test_expect_success 'initializing repository with --ref-storage-format and paylo\n>  \ttest_cmp expect actual\n>  '\n>\n> +for format in files reftable\n> +do\n\nThis would run the loop for both the files and reftable CI jobs.\n\n> +\ttest_expect_success \"default ref storage format with $format backend and payload\" '\n> +\t\ttest_when_finished \"rm -rf repo refdir\" &&\n> +\t\tBACKEND=\"$format://$(pwd)/refdir\" &&\n> +\t\techo \"$BACKEND\" >expect &&\n> +\t\tgit init --ref-storage-format=\"$BACKEND\" repo &&\n> +\t\ttest_path_is_dir refdir &&\n> +\t\tcase \"$format\" in\n> +\t\tfiles)\n> +\t\t\ttest_path_is_file refdir/HEAD &&\n> +\t\t\ttest_path_is_dir refdir/refs\n> +\t\t\t;;\n> +\t\treftable)\n> +\t\t\ttest_path_is_dir refdir/reftable &&\n> +\t\t\ttest_path_is_file refdir/reftable/tables.list\n> +\t\t\t;;\n> +\t\t*)\n> +\t\t\tBUG \"unknown ref storage format: $format\"\n> +\t\t\t;;\n> +\t\tesac\n> +\t'\n> +done\n> +\n\nShould we instead do (untested):\n\ntest_expect_success \"default ref storage format with $format backend\nand payload\" '\n\ttest_when_finished \"rm -rf repo refdir\" &&\n    format=\"$(test_detect_ref_format)\" &&\n\tBACKEND=\"$format://$(pwd)/refdir\" &&\n\techo \"$BACKEND\" >expect &&\n\tgit init --ref-storage-format=\"$BACKEND\" repo &&\n\ttest_path_is_dir refdir &&\n\tcase \"$format\" in\n\tfiles)\n\t\ttest_path_is_file refdir/HEAD &&\n\t\ttest_path_is_dir refdir/refs\n\t\t;;\n\treftable)\n\t\ttest_path_is_dir refdir/reftable &&\n\t\ttest_path_is_file refdir/reftable/tables.list\n\t\t;;\n\t*)\n\t\tBUG \"unknown ref storage format: $format\"\n\t\t;;\n\tesac\n'\n\nThis eliminates the loop, while still testing both backends on their\nrespective CI jobs.\n\n>  test_expect_success 'cloning repository with alt ref directory' '\n>  \ttest_when_finished \"rm -rf source repo refdir\" &&\n>  \tmkdir refdir &&\n>\n> --\n> 2.56.0.500.g5d06dc02cd.dirty\n"},{"id":"557657","messageId":"CAOLa=ZQfSA7FzpNPry9tRgF5RAqBPaof7=Q-XR+8nA2koh5gsQ@mail.gmail.com","threadId":"66784","inReplyTo":"20261105-pks-storage-format-improvements-v1-4-870bb4212518@pks.im","subject":"Re: [PATCH 4/5] builtin/rev-parse: print payload for the ref storage format","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-11-06T10:40:23Z","receivedAt":"2026-11-06T10:40:23Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> While we print both the object storage format and its payload for the\n> \"--show-object-storage-format\" switch, we only print the ref storage\n> format with \"--show-ref-storage-format\". Adapt the latter to also show\n> the payload.\n>\n\nNice, this makes sense and was clearly a miss. Thanks for fixing.\n\n[snip]\n"},{"id":"557658","messageId":"CAOLa=ZROTekhsC5g4vQKgCKg-akb4X0DMFf3bcfbO2k0uHtxsg@mail.gmail.com","threadId":"66784","inReplyTo":"20261105-pks-storage-format-improvements-v1-0-870bb4212518@pks.im","subject":"Re: [PATCH 0/5] Improvements for storage format handling","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-11-06T10:41:41Z","receivedAt":"2026-11-06T10:41:41Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> Hi,\n>\n> this patch series includes a couple of improvements for how we handle\n> ref and object storage formats.\n>\n> Most importantly, this series allows us to run tests with a different\n> object storage backend via `GIT_TEST_DEFAULT_OBJECT_STORAGE_FORMAT=`. As\n> we don't have an alternative backend, I also introduce the ability to\n> specify a default object (and ref) storage format that carries a payload\n> so that you can run all tests with \"files://objdir\", for example. That\n> surfaces a bunch of tests that assume objects exist in \".git/objects\".\n>\n> Overall it's a surprisingly small number of test failures, only 1105 out\n> of 35000 tests fail. Fixing those is not part of this patch series, but\n> we'll start upstreaming fixes once this lands. Such tests will either\n> have to be adapted to use plumbing commands, or they'll need to be\n> tagged with a prerequisite. It's going to be a bunch of work, but it's\n> basically preparing for the introduction of alternative backends\n> already.\n>\n> The series is built on top of 0ffa3e35c7c (The 15th batch, 2026-11-04)\n> with ps/odb-object-storage-extension at 87a55d790ca (builtin/submodule:\n> propagate \"--object-storage-format=\", 2026-10-27) merged into it.\n>\n> Thanks!\n>\n> Patrick\n>\n\nThis was a very easy read. I don't have much to add apart from a comment\non 1/5. But you can also leave it as is! Thanks\n"},{"id":"557661","messageId":"au2zpr5qla3VG04h@pks.im","threadId":"66784","inReplyTo":"CAOLa=ZR3jxonz2o+JKe=pyzSACwJH=uu3an141PEG1Ssq0pBWg@mail.gmail.com","subject":"Re: [PATCH 1/5] refs: create leading directories","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-06T11:01:58Z","receivedAt":"2026-11-06T11:01:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Nov 06, 2026 at 02:37:23AM -0800, Karthik Nayak wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> > diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh\n> > index ee2bb66b99c..85d1e876b82 100755\n> > --- a/t/t1423-ref-backend.sh\n> > +++ b/t/t1423-ref-backend.sh\n> > @@ -284,6 +284,30 @@ test_expect_success 'initializing repository with --ref-storage-format and paylo\n> >  \ttest_cmp expect actual\n> >  '\n> >\n> > +for format in files reftable\n> > +do\n> \n> This would run the loop for both the files and reftable CI jobs.\n> \n> > +\ttest_expect_success \"default ref storage format with $format backend and payload\" '\n> > +\t\ttest_when_finished \"rm -rf repo refdir\" &&\n> > +\t\tBACKEND=\"$format://$(pwd)/refdir\" &&\n> > +\t\techo \"$BACKEND\" >expect &&\n> > +\t\tgit init --ref-storage-format=\"$BACKEND\" repo &&\n> > +\t\ttest_path_is_dir refdir &&\n> > +\t\tcase \"$format\" in\n> > +\t\tfiles)\n> > +\t\t\ttest_path_is_file refdir/HEAD &&\n> > +\t\t\ttest_path_is_dir refdir/refs\n> > +\t\t\t;;\n> > +\t\treftable)\n> > +\t\t\ttest_path_is_dir refdir/reftable &&\n> > +\t\t\ttest_path_is_file refdir/reftable/tables.list\n> > +\t\t\t;;\n> > +\t\t*)\n> > +\t\t\tBUG \"unknown ref storage format: $format\"\n> > +\t\t\t;;\n> > +\t\tesac\n> > +\t'\n> > +done\n> > +\n> \n> Should we instead do (untested):\n> \n> test_expect_success \"default ref storage format with $format backend\n> and payload\" '\n> \ttest_when_finished \"rm -rf repo refdir\" &&\n>     format=\"$(test_detect_ref_format)\" &&\n> \tBACKEND=\"$format://$(pwd)/refdir\" &&\n> \techo \"$BACKEND\" >expect &&\n> \tgit init --ref-storage-format=\"$BACKEND\" repo &&\n> \ttest_path_is_dir refdir &&\n> \tcase \"$format\" in\n> \tfiles)\n> \t\ttest_path_is_file refdir/HEAD &&\n> \t\ttest_path_is_dir refdir/refs\n> \t\t;;\n> \treftable)\n> \t\ttest_path_is_dir refdir/reftable &&\n> \t\ttest_path_is_file refdir/reftable/tables.list\n> \t\t;;\n> \t*)\n> \t\tBUG \"unknown ref storage format: $format\"\n> \t\t;;\n> \tesac\n> '\n> \n> This eliminates the loop, while still testing both backends on their\n> respective CI jobs.\n\nWell, in this case the system under test is explicitly the ref storage\nformat itself and how it behaves together with the extension. The\nGIT_TEST_DEFAULT_REF_STORAGE format on the other hand is design to\nexplicitly test that everything else works as-is with an alternate ref\nstore.\n\nSo I think it makes sense to keep this explicit test.\n\nPatrick\n\n"},{"id":"557662","messageId":"au2zyG0TY6IjmK39@pks.im","threadId":"66784","inReplyTo":"CAOLa=ZROTekhsC5g4vQKgCKg-akb4X0DMFf3bcfbO2k0uHtxsg@mail.gmail.com","subject":"Re: [PATCH 0/5] Improvements for storage format handling","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-06T11:02:32Z","receivedAt":"2026-11-06T11:02:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Nov 06, 2026 at 02:41:41AM -0800, Karthik Nayak wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> > Hi,\n> >\n> > this patch series includes a couple of improvements for how we handle\n> > ref and object storage formats.\n> >\n> > Most importantly, this series allows us to run tests with a different\n> > object storage backend via `GIT_TEST_DEFAULT_OBJECT_STORAGE_FORMAT=`. As\n> > we don't have an alternative backend, I also introduce the ability to\n> > specify a default object (and ref) storage format that carries a payload\n> > so that you can run all tests with \"files://objdir\", for example. That\n> > surfaces a bunch of tests that assume objects exist in \".git/objects\".\n> >\n> > Overall it's a surprisingly small number of test failures, only 1105 out\n> > of 35000 tests fail. Fixing those is not part of this patch series, but\n> > we'll start upstreaming fixes once this lands. Such tests will either\n> > have to be adapted to use plumbing commands, or they'll need to be\n> > tagged with a prerequisite. It's going to be a bunch of work, but it's\n> > basically preparing for the introduction of alternative backends\n> > already.\n> >\n> > The series is built on top of 0ffa3e35c7c (The 15th batch, 2026-11-04)\n> > with ps/odb-object-storage-extension at 87a55d790ca (builtin/submodule:\n> > propagate \"--object-storage-format=\", 2026-10-27) merged into it.\n> >\n> > Thanks!\n> >\n> > Patrick\n> >\n> \n> This was a very easy read. I don't have much to add apart from a comment\n> on 1/5. But you can also leave it as is! Thanks\n\nThanks for your review! I'd prefer to leave 1/5 as-is as explained in my\nreply there.\n\nPatrick\n\n"},{"id":"557727","messageId":"xmqqzevliy6j.fsf@gitster.g","threadId":"66784","inReplyTo":"20261105-pks-storage-format-improvements-v1-5-870bb4212518@pks.im","subject":"Re: [PATCH 5/5] t: allow testing with a non-default object storage foramt","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-06T18:59:00Z","receivedAt":"2026-11-06T18:59:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> Subject: Re: [PATCH 5/5] t: allow testing with a non-default object storage foramt\n\n\"foramt\" -> \"format\".\n\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index a893963f86c..7fdd8272162 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -576,9 +576,13 @@ export EDITOR\n>  GIT_TEST_BUILTIN_HASH=$(\"$GIT_BINARY\" version --build-options | sed -ne 's/^default-hash: //p')\n>  GIT_DEFAULT_HASH=\"${GIT_TEST_DEFAULT_HASH:-$GIT_TEST_BUILTIN_HASH}\"\n>  export GIT_DEFAULT_HASH\n> +\n>  GIT_DEFAULT_REF_STORAGE_FORMAT=\"${GIT_TEST_DEFAULT_REF_STORAGE_FORMAT:-files}\"\n>  export GIT_DEFAULT_REF_STORAGE_FORMAT\n\nThis unrelated change created unnecessary conflicts.\n\nNo need to resend.\n\n>  \n> +GIT_DEFAULT_OBJECT_STORAGE_FORMAT=\"${GIT_TEST_DEFAULT_OBJECT_STORAGE_FORMAT:-files}\"\n> +export GIT_DEFAULT_OBJECT_STORAGE_FORMAT\n> +\n>  # Tests using GIT_TRACE typically don't want <timestamp> <file>:<line> output\n>  GIT_TRACE_BARE=1\n>  export GIT_TRACE_BARE\n\n"},{"id":"557880","messageId":"avF2xjwjcyU6egIf@pks.im","threadId":"66784","inReplyTo":"xmqqzevliy6j.fsf@gitster.g","subject":"Re: [PATCH 5/5] t: allow testing with a non-default object storage foramt","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-09T07:31:26Z","receivedAt":"2026-11-09T07:31:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Nov 06, 2026 at 10:59:00AM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> > Subject: Re: [PATCH 5/5] t: allow testing with a non-default object storage foramt\n> \n> \"foramt\" -> \"format\".\n> \n> > diff --git a/t/test-lib.sh b/t/test-lib.sh\n> > index a893963f86c..7fdd8272162 100644\n> > --- a/t/test-lib.sh\n> > +++ b/t/test-lib.sh\n> > @@ -576,9 +576,13 @@ export EDITOR\n> >  GIT_TEST_BUILTIN_HASH=$(\"$GIT_BINARY\" version --build-options | sed -ne 's/^default-hash: //p')\n> >  GIT_DEFAULT_HASH=\"${GIT_TEST_DEFAULT_HASH:-$GIT_TEST_BUILTIN_HASH}\"\n> >  export GIT_DEFAULT_HASH\n> > +\n> >  GIT_DEFAULT_REF_STORAGE_FORMAT=\"${GIT_TEST_DEFAULT_REF_STORAGE_FORMAT:-files}\"\n> >  export GIT_DEFAULT_REF_STORAGE_FORMAT\n> \n> This unrelated change created unnecessary conflicts.\n> \n> No need to resend.\n\nOkay, I've adapted these locally so that I have them included in case I\nneed to reroll. Thanks!\n\nPatrick\n\n"}]}