git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v5 26/26] introduce "extensions" form of core.repositoryformatversion

From
David Turner <dturner@twopensource.com>
Date
Oct 28, 2015, 02:14 UTC
Message-ID
<1445998467-11511-27-git-send-email-dturner@twopensource.com>
In-Reply-To
<1445998467-11511-1-git-send-email-dturner@twopensource.com>
From: Jeff King <peff@peff.net>

Normally we try to avoid bumps of the whole-repository core.repositoryformatversion field. However, it is unavoidable if we want to safely change certain aspects of git in a backwards-incompatible way (e.g., modifying the set of ref tips that we must traverse to generate a list of unreachable, safe-to-prune objects).

If we were to bump the repository version for every such change, then any implementation understanding version `X` would also have to understand `X-1`, `X-2`, and so forth, even though the incompatibilities may be in orthogonal parts of the system, and there is otherwise no reason we cannot implement one without the other (or more importantly, that the user cannot choose to use one feature without the other, weighing the tradeoff in compatibility only for that particular feature).

This patch documents the existing repositoryformatversion strategy and introduces a new format, "1", which lets a repository specify that it must run with an arbitrary set of extensions. This can be used, for example:

 - to inform git that the objects should not be pruned based
   only on the reachability of the ref tips (e.g, because it
   has "clone --shared" children)
 - that the refs are stored in a format besides the usual
   "refs" and "packed-refs" directories

Because we bump to format "1", and because format "1" requires that a running git knows about any extensions mentioned, we know that older versions of the code will not do something dangerous when confronted with these new formats.

For example, if the user chooses to use database storage for refs, they may set the "extensions.refbackend" config to "db". Older versions of git will not understand format "1" and bail. Versions of git which understand "1" but do not know about "refbackend", or which know about "refbackend" but not about the "db" backend, will refuse to run. This is annoying, of course, but much better than the alternative of claiming that there are no refs in the repository, or writing to a location that other implementations will not read.

Note that we are only defining the rules for format 1 here. We do not ever write format 1 ourselves; it is a tool that is meant to be used by users and future extensions to provide safety with older implementations.

Signed-off-by: Jeff King <peff@peff.net>
---
 Documentation/technical/repository-version.txt | 81 ++++++++++++++++++++++++++
 cache.h                                        |  6 ++
 setup.c                                        | 37 +++++++++++-
 t/t1302-repo-version.sh                        | 38 ++++++++++++
 4 files changed, 159 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/technical/repository-version.txt
diff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt
new file mode 100644
index 0000000..3d7106d
--- /dev/null
+++ b/Documentation/technical/repository-version.txt
@@ -0,0 +1,81 @@
+Git Repository Format Versions
+==============================
+
+Every git repository is marked with a numeric version in the
+`core.repositoryformatversion` key of its `config` file. This version
+specifies the rules for operating on the on-disk repository data. An
+implementation of git which does not understand a particular version
+advertised by an on-disk repository MUST NOT operate on that repository;
+doing so risks not only producing wrong results, but actually losing
+data.
+
+Because of this rule, version bumps should be kept to an absolute
+minimum. Instead, we generally prefer these strategies:
+
+  - bumping format version numbers of individual data files (e.g.,
+    index, packfiles, etc). This restricts the incompatibilities only to
+    those files.
+
+  - introducing new data that gracefully degrades when used by older
+    clients (e.g., pack bitmap files are ignored by older clients, which
+    simply do not take advantage of the optimization they provide).
+
+A whole-repository format version bump should only be part of a change
+that cannot be independently versioned. For instance, if one were to
+change the reachability rules for objects, or the rules for locking
+refs, that would require a bump of the repository format version.
+
+Note that this applies only to accessing the repository's disk contents
+directly. An older client which understands only format `0` may still
+connect via `git://` to a repository using format `1`, as long as the
+server process understands format `1`.
+
+The preferred strategy for rolling out a version bump (whether whole
+repository or for a single file) is to teach git to read the new format,
+and allow writing the new format with a config switch or command line
+option (for experimentation or for those who do not care about backwards
+compatibility with older gits). Then after a long period to allow the
+reading capability to become common, we may switch to writing the new
+format by default.
+
+The currently defined format versions are:
+
+Version `0`
+-----------
+
+This is the format defined by the initial version of git, including but
+not limited to the format of the repository directory, the repository
+configuration file, and the object and ref storage. Specifying the
+complete behavior of git is beyond the scope of this document.
+
+Version `1`
+-----------
+
+This format is identical to version `0`, with the following exceptions:
+
+  1. When reading the `core.repositoryformatversion` variable, a git
+     implementation which supports version 1 MUST also read any
+     configuration keys found in the `extensions` section of the
+     configuration file.
+
+  2. If a version-1 repository specifies any `extensions.*` keys that
+     the running git has not implemented, the operation MUST NOT
+     proceed. Similarly, if the value of any known key is not understood
+     by the implementation, the operation MUST NOT proceed.
+
+Note that if no extensions are specified in the config file, then
+`core.repositoryformatversion` SHOULD be set to `0` (setting it to `1`
+provides no benefit, and makes the repository incompatible with older
+implementations of git).
+
+This document will serve as the master list for extensions. Any
+implementation wishing to define a new extension should make a note of
+it here, in order to claim the name.
+
+The defined extensions are:
+
+`noop`
+~~~~~~
+
+This extension does not change git's behavior at all. It is useful only
+for testing format-1 compatibility.
diff --git a/cache.h b/cache.h
index 28b9896..90ec48e 100644
--- a/cache.h
+++ b/cache.h
@@ -696,7 +696,13 @@ extern char *notes_ref_name;
 
 extern int grafts_replace_parents;
 
+/*
+ * GIT_REPO_VERSION is the version we write by default. The
+ * _READ variant is the highest number we know how to
+ * handle.
+ */
 #define GIT_REPO_VERSION 0
+#define GIT_REPO_VERSION_READ 1
 extern int repository_format_version;
 extern int check_repository_format(void);
 
diff --git a/setup.c b/setup.c
index 2b64cbb..0c29469 100644
--- a/setup.c
+++ b/setup.c
@@ -5,6 +5,7 @@
 static int inside_git_dir = -1;
 static int inside_work_tree = -1;
 static int work_tree_config_is_bogus;
+static struct string_list unknown_extensions = STRING_LIST_INIT_DUP;
 
 /*
  * The input parameter must contain an absolute path, and it must already be
@@ -349,10 +350,23 @@ void setup_work_tree(void)
 
 static int check_repo_format(const char *var, const char *value, void *cb)
 {
+	const char *ext;
+
 	if (strcmp(var, "core.repositoryformatversion") == 0)
 		repository_format_version = git_config_int(var, value);
 	else if (strcmp(var, "core.sharedrepository") == 0)
 		shared_repository = git_config_perm(var, value);
+	else if (skip_prefix(var, "extensions.", &ext)) {
+		/*
+		 * record any known extensions here; otherwise,
+		 * we fall through to recording it as unknown, and
+		 * check_repository_format will complain
+		 */
+		if (!strcmp(ext, "noop"))
+			;
+		else
+			string_list_append(&unknown_extensions, ext);
+	}
 	return 0;
 }
 
@@ -363,6 +377,8 @@ static int check_repository_format_gently(const char *gitdir, int *nongit_ok)
 	config_fn_t fn;
 	int ret = 0;
 
+	string_list_clear(&unknown_extensions, 0);
+
 	if (get_common_dir(&sb, gitdir))
 		fn = check_repo_format;
 	else
@@ -380,16 +396,31 @@ static int check_repository_format_gently(const char *gitdir, int *nongit_ok)
 	 * is a good one.
 	 */
 	git_config_early(fn, NULL, repo_config);
-	if (GIT_REPO_VERSION < repository_format_version) {
+	if (GIT_REPO_VERSION_READ < repository_format_version) {
 		if (!nongit_ok)
 			die ("Expected git repo version <= %d, found %d",
-			     GIT_REPO_VERSION, repository_format_version);
+			     GIT_REPO_VERSION_READ, repository_format_version);
 		warning("Expected git repo version <= %d, found %d",
-			GIT_REPO_VERSION, repository_format_version);
+			GIT_REPO_VERSION_READ, repository_format_version);
 		warning("Please upgrade Git");
 		*nongit_ok = -1;
 		ret = -1;
 	}
+
+	if (repository_format_version >= 1 && unknown_extensions.nr) {
+		int i;
+
+		if (!nongit_ok)
+			die("unknown repository extension: %s",
+			    unknown_extensions.items[0].string);
+
+		for (i = 0; i < unknown_extensions.nr; i++)
+			warning("unknown repository extension: %s",
+				unknown_extensions.items[i].string);
+		*nongit_ok = -1;
+		ret = -1;
+	}
+
 	strbuf_release(&sb);
 	return ret;
 }
diff --git a/t/t1302-repo-version.sh b/t/t1302-repo-version.sh
index 0d9388a..8dd6fd7 100755
--- a/t/t1302-repo-version.sh
+++ b/t/t1302-repo-version.sh
@@ -67,4 +67,42 @@ test_expect_success 'gitdir required mode' '
 	)
 '
 
+check_allow () {
+	git rev-parse --git-dir >actual &&
+	echo .git >expect &&
+	test_cmp expect actual
+}
+
+check_abort () {
+	test_must_fail git rev-parse --git-dir
+}
+
+# avoid git-config, since it cannot be trusted to run
+# in a repository with a broken version
+mkconfig () {
+	echo '[core]' &&
+	echo "repositoryformatversion = $1" &&
+	shift &&
+
+	if test $# -gt 0; then
+		echo '[extensions]' &&
+		for i in "$@"; do
+			echo "$i"
+		done
+	fi
+}
+
+while read outcome version extensions; do
+	test_expect_success "$outcome version=$version $extensions" "
+		mkconfig $version $extensions >.git/config &&
+		check_${outcome}
+	"
+done <<\EOF
+allow 0
+allow 1
+allow 1 noop
+abort 1 no-such-extension
+allow 0 no-such-extension
+EOF
+
 test_done

-- 
2.4.2.658.g6d8523e-twtrsrc
Previous: Michael HaggertyNext: Michael Haggerty
Message 31 of 67 in “refs backend pre-vtable”
  1. 00/26 refs backend pre-vtableDavid Turner, Oct 28, 2015
  2. 01/26 refs.c: create a public version of verify_refname_availableDavid Turner, Oct 28, 2015
  3. 02/26 refs: make is_branch publicDavid Turner, Oct 28, 2015
  4. 03/26 refs-be-files.c: rename refs to refs-be-filesDavid Turner, Oct 28, 2015
  5. 04/26 refs.c: add a new refs.c file to hold all common refs codeDavid Turner, Oct 28, 2015
  6. 05/26 refs.c: move update_ref to refs.cDavid Turner, Oct 28, 2015
  7. 06/26 refs.c: move delete_pseudoref and delete_ref to the common codeDavid Turner, Oct 28, 2015
  8. 07/26 refs.c: move read_ref_at to the common refs fileDavid Turner, Oct 28, 2015
  9. 08/26 refs.c: move the hidden refs functions to the common codeDavid Turner, Oct 28, 2015
  10. 09/26 refs.c: move dwim and friend functions to the common refs codeDavid Turner, Oct 28, 2015
  11. 10/26 refs.c: move warn_if_dangling_symref* to the common codeDavid Turner, Oct 28, 2015
  12. 11/26 refs.c: move read_ref, read_ref_full and ref_exists to the common codeDavid Turner, Oct 28, 2015
  13. 12/26 refs.c: move resolve_refdup to commonDavid Turner, Oct 28, 2015
  14. 13/26 refs.c: move check_refname_format to the common codeDavid Turner, Oct 28, 2015
  15. 14/26 refs.c: move is_branch to the common codeDavid Turner, Oct 28, 2015
  16. 15/26 refs.c: move prettify_refname to the common codeDavid Turner, Oct 28, 2015
  17. 16/26 refs.c: move ref iterators to the common codeDavid Turner, Oct 28, 2015
  18. Michael HaggertyNov 1, 2015
  19. 17/26 refs.c: move head_ref_namespaced to the common codeDavid Turner, Oct 28, 2015
  20. 18/26 refs: move transaction functions into common codeDavid Turner, Oct 28, 2015
  21. Michael HaggertyNov 1, 2015
  22. David TurnerNov 2, 2015
  23. 19/26 refs.c: move refname_is_safe to the common codeDavid Turner, Oct 28, 2015
  24. 20/26 refs.c: move copy_msg to the common codeDavid Turner, Oct 28, 2015
  25. 21/26 refs.c: move peel_object to the common codeDavid Turner, Oct 28, 2015
  26. 22/26 refs.c: move should_autocreate_reflog to common codeDavid Turner, Oct 28, 2015
  27. 23/26 initdb: move safe_create_dir into common codeDavid Turner, Oct 28, 2015
  28. 24/26 refs: make files_log_ref_write functions publicDavid Turner, Oct 28, 2015
  29. 25/26 refs: break out ref conflict checksDavid Turner, Oct 28, 2015
  30. Michael HaggertyNov 2, 2015
  31. 26/26 introduce "extensions" form of core.repositoryformatversionDavid Turner, Oct 28, 2015
  32. Michael HaggertyNov 3, 2015
  33. 00/25 refs backend pre-vtableMichael Haggerty, Nov 3, 2015
  34. 01/25 refs: make is_branch publicMichael Haggerty, Nov 3, 2015
  35. 02/25 refs/files-backend.c: new file, renamed from refs.cMichael Haggerty, Nov 3, 2015
  36. 03/25 refs: add a new file, refs/refs.c, to hold common refs codeMichael Haggerty, Nov 3, 2015
  37. 04/25 refs: move update_ref to refs/refs.cMichael Haggerty, Nov 3, 2015
  38. 05/25 refs: move delete_pseudoref and delete_ref to the common codeMichael Haggerty, Nov 3, 2015
  39. 06/25 refs: move read_ref_at to the common refs fileMichael Haggerty, Nov 3, 2015
  40. 07/25 refs: move the hidden refs functions to the common codeMichael Haggerty, Nov 3, 2015
  41. 08/25 refs: move dwim and friend functions to the common refs codeMichael Haggerty, Nov 3, 2015
  42. 09/25 refs: move warn_if_dangling_symref* to the common codeMichael Haggerty, Nov 3, 2015
  43. 10/25 refs: move read_ref, read_ref_full and ref_exists to the common codeMichael Haggerty, Nov 3, 2015
  44. 11/25 refs: move resolve_refdup to commonMichael Haggerty, Nov 3, 2015
  45. 12/25 refs: move check_refname_format to the common codeMichael Haggerty, Nov 3, 2015
  46. 13/25 refs: move is_branch to the common codeMichael Haggerty, Nov 3, 2015
  47. 14/25 refs: move prettify_refname to the common codeMichael Haggerty, Nov 3, 2015
  48. 15/25 refs: move ref iterators to the common codeMichael Haggerty, Nov 3, 2015
  49. 16/25 refs: move head_ref_namespaced to the common codeMichael Haggerty, Nov 3, 2015
  50. 17/25 refs: move transaction functions to the common codeMichael Haggerty, Nov 3, 2015
  51. 18/25 refs: move refname_is_safe to the common codeMichael Haggerty, Nov 3, 2015
  52. 19/25 refs: move copy_msg to the common codeMichael Haggerty, Nov 3, 2015
  53. 20/25 refs: move peel_object to the common codeMichael Haggerty, Nov 3, 2015
  54. 21/25 refs: move should_autocreate_reflog to common codeMichael Haggerty, Nov 3, 2015
  55. 22/25 initdb: make safe_create_dir publicMichael Haggerty, Nov 3, 2015
  56. 23/25 files_log_ref_write: new functionMichael Haggerty, Nov 3, 2015
  57. 24/25 refs: create a shared version of verify_refname_availableMichael Haggerty, Nov 3, 2015
  58. 25/25 refs: break out ref conflict checksMichael Haggerty, Nov 3, 2015
  59. David TurnerNov 4, 2015
  60. Michael HaggertyNov 5, 2015
  61. David TurnerNov 5, 2015
  62. Michael HaggertyNov 6, 2015
  63. Junio C HamanoNov 6, 2015
  64. Junio C HamanoNov 6, 2015
  65. Michael HaggertyNov 8, 2015
  66. Michael HaggertyNov 8, 2015
  67. Junio C HamanoNov 8, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.