git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v5 03/12] bundle-uri: create base key-value pair parsing

From
Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Date
Oct 12, 2022, 12:52 UTC
Message-ID
<430e01cd2a4127a17c4bfc1b0db7209964736de8.1665579160.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.1333.v5.git.1665579160.gitgitgadget@gmail.com>
From: Derrick Stolee <derrickstolee@github.com>

There will be two primary ways to advertise a bundle list: as a list of packet lines in Git's protocol v2 and as a config file served from a bundle URI. Both of these fundamentally use a list of key-value pairs. We will use the same set of key-value pairs across these formats.

Create a new bundle_list_update() method that is currently unusued, but will be used in the next change. It inspects each key to see if it is understood and then applies it to the given bundle_list. Here are the keys that we teach Git to understand:

* bundle.version: This value should be an integer. Git currently
  understands only version 1 and will ignore the list if the version is
  any other value. This version can be increased in the future if we
  need to add new keys that Git should not ignore. We can add new
  "heuristic" keys without incrementing the version.
* bundle.mode: This value should be one of "all" or "any". If this
  mode is not understood, then Git will ignore the list. This mode
  indicates whether Git needs all of the bundle list items to make a
  complete view of the content or if any single item is sufficient.

The rest of the keys use a bundle identifier "<id>" as part of the key name. Keys using the same "<id>" describe a single bundle list item.

* bundle.<id>.uri: This stores the URI of the bundle item. This
  currently is expected to be an absolute URI, but will be relaxed to be
  a relative URI in the future.

While parsing, return an error if a URI key is repeated, since we can make that restriction with bundle lists.

Make the git_parse_int() method global so we can parse the integer version value carefully.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 Documentation/config.txt        |  2 +
 Documentation/config/bundle.txt | 24 +++++++++++
 bundle-uri.c                    | 76 +++++++++++++++++++++++++++++++++
 config.c                        |  2 +-
 config.h                        |  1 +
 5 files changed, 104 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/config/bundle.txt
diff --git a/Documentation/config.txt b/Documentation/config.txt
index e376d547ce0..4280af6992e 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -387,6 +387,8 @@ include::config/branch.txt[]
 
 include::config/browser.txt[]
 
+include::config/bundle.txt[]
+
 include::config/checkout.txt[]
 
 include::config/clean.txt[]
diff --git a/Documentation/config/bundle.txt b/Documentation/config/bundle.txt
new file mode 100644
index 00000000000..daa21eb674a
--- /dev/null
+++ b/Documentation/config/bundle.txt
@@ -0,0 +1,24 @@
+bundle.*::
+	The `bundle.*` keys may appear in a bundle list file found via the
+	`git clone --bundle-uri` option. These keys currently have no effect
+	if placed in a repository config file, though this will change in the
+	future. See link:technical/bundle-uri.html[the bundle URI design
+	document] for more details.
+
+bundle.version::
+	This integer value advertises the version of the bundle list format
+	used by the bundle list. Currently, the only accepted value is `1`.
+
+bundle.mode::
+	This string value should be either `all` or `any`. This value describes
+	whether all of the advertised bundles are required to unbundle a
+	complete understanding of the bundled information (`all`) or if any one
+	of the listed bundle URIs is sufficient (`any`).
+
+bundle.<id>.*::
+	The `bundle.<id>.*` keys are used to describe a single item in the
+	bundle list, grouped under `<id>` for identification purposes.
+
+bundle.<id>.uri::
+	This string value defines the URI by which Git can reach the contents
+	of this `<id>`. This URI may be a bundle file or another bundle list.
diff --git a/bundle-uri.c b/bundle-uri.c
index f9a8db221bc..0bc59dd9c34 100644
--- a/bundle-uri.c
+++ b/bundle-uri.c
@@ -6,6 +6,7 @@
 #include "run-command.h"
 #include "hashmap.h"
 #include "pkt-line.h"
+#include "config.h"
 
 static int compare_bundles(const void *hashmap_cmp_fn_data,
 			   const struct hashmap_entry *he1,
@@ -65,6 +66,81 @@ int for_all_bundles_in_list(struct bundle_list *list,
 	return 0;
 }
 
+/**
+ * Given a key-value pair, update the state of the given bundle list.
+ * Returns 0 if the key-value pair is understood. Returns -1 if the key
+ * is not understood or the value is malformed.
+ */
+MAYBE_UNUSED
+static int bundle_list_update(const char *key, const char *value,
+			      struct bundle_list *list)
+{
+	struct strbuf id = STRBUF_INIT;
+	struct remote_bundle_info lookup = REMOTE_BUNDLE_INFO_INIT;
+	struct remote_bundle_info *bundle;
+	const char *subsection, *subkey;
+	size_t subsection_len;
+
+	if (parse_config_key(key, "bundle", &subsection, &subsection_len, &subkey))
+		return -1;
+
+	if (!subsection_len) {
+		if (!strcmp(subkey, "version")) {
+			int version;
+			if (!git_parse_int(value, &version))
+				return -1;
+			if (version != 1)
+				return -1;
+
+			list->version = version;
+			return 0;
+		}
+
+		if (!strcmp(subkey, "mode")) {
+			if (!strcmp(value, "all"))
+				list->mode = BUNDLE_MODE_ALL;
+			else if (!strcmp(value, "any"))
+				list->mode = BUNDLE_MODE_ANY;
+			else
+				return -1;
+			return 0;
+		}
+
+		/* Ignore other unknown global keys. */
+		return 0;
+	}
+
+	strbuf_add(&id, subsection, subsection_len);
+
+	/*
+	 * Check for an existing bundle with this <id>, or create one
+	 * if necessary.
+	 */
+	lookup.id = id.buf;
+	hashmap_entry_init(&lookup.ent, strhash(lookup.id));
+	if (!(bundle = hashmap_get_entry(&list->bundles, &lookup, ent, NULL))) {
+		CALLOC_ARRAY(bundle, 1);
+		bundle->id = strbuf_detach(&id, NULL);
+		hashmap_entry_init(&bundle->ent, strhash(bundle->id));
+		hashmap_add(&list->bundles, &bundle->ent);
+	}
+	strbuf_release(&id);
+
+	if (!strcmp(subkey, "uri")) {
+		if (bundle->uri)
+			return -1;
+		bundle->uri = xstrdup(value);
+		return 0;
+	}
+
+	/*
+	 * At this point, we ignore any information that we don't
+	 * understand, assuming it to be hints for a heuristic the client
+	 * does not currently understand.
+	 */
+	return 0;
+}
+
 static char *find_temp_filename(void)
 {
 	int fd;
diff --git a/config.c b/config.c
index 015bec360f5..e93101249f6 100644
--- a/config.c
+++ b/config.c
@@ -1214,7 +1214,7 @@ static int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max)
 	return 0;
 }
 
-static int git_parse_int(const char *value, int *ret)
+int git_parse_int(const char *value, int *ret)
 {
 	intmax_t tmp;
 	if (!git_parse_signed(value, &tmp, maximum_signed_value_of_type(int)))
diff --git a/config.h b/config.h
index ca994d77147..ef9eade6414 100644
--- a/config.h
+++ b/config.h
@@ -206,6 +206,7 @@ int config_with_options(config_fn_t fn, void *,
 
 int git_parse_ssize_t(const char *, ssize_t *);
 int git_parse_ulong(const char *, unsigned long *);
+int git_parse_int(const char *value, int *ret);
 
 /**
  * Same as `git_config_bool`, except that it returns -1 on error rather
-- 
gitgitgadget
Previous: Derrick Stolee via GitGitGadgetNext: Ævar Arnfjörð Bjarmason via GitGitGadget
Message 79 of 94 in “Bundle URIs III: Parse and download from bundle lists”
  1. 0/7 Bundle URIs III: Parse and download from bundle listsDerrick Stolee via GitGitGadget, Aug 22, 2022
  2. 1/7 bundle-uri: create bundle_list struct and helpersDerrick Stolee via GitGitGadget, Aug 22, 2022
  3. Junio C HamanoAug 22, 2022
  4. 3/7 bundle-uri: create "key=value" line parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Aug 22, 2022
  5. Junio C HamanoAug 22, 2022
  6. Derrick StoleeAug 23, 2022
  7. Josh SteadmonSep 2, 2022
  8. 6/7 bundle-uri: limit recursion depth for bundle listsDerrick Stolee via GitGitGadget, Aug 22, 2022
  9. 4/7 bundle-uri: unit test "key=value" parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Aug 22, 2022
  10. 4/7 bundle-uri: unit test "key=value" parsingTeng Long, Sep 1, 2022
  11. 5/7 bundle-uri: parse bundle list in config formatDerrick Stolee via GitGitGadget, Aug 22, 2022
  12. Junio C HamanoAug 22, 2022
  13. Derrick StoleeAug 23, 2022
  14. Jonathan TanAug 31, 2022
  15. 5/7 bundle-uri: parse bundle list in config formatTeng Long, Sep 1, 2022
  16. 2/7 bundle-uri: create base key-value pair parsingDerrick Stolee via GitGitGadget, Aug 22, 2022
  17. Junio C HamanoAug 22, 2022
  18. Derrick StoleeAug 23, 2022
  19. Jonathan TanAug 31, 2022
  20. Glen ChooAug 31, 2022
  21. 4/7 bundle-uri: unit test "key=value" parsingTeng Long, Sep 1, 2022
  22. 7/7 bundle-uri: fetch a list of bundlesDerrick Stolee via GitGitGadget, Aug 22, 2022
  23. Josh SteadmonSep 2, 2022
  24. 7/7 bundle-uri: fetch a list of bundlesTeng Long, Sep 5, 2022
  25. Derrick StoleeSep 8, 2022
  26. 0/9 Bundle URIs III: Parse and download from bundle listsDerrick Stolee via GitGitGadget, Sep 9, 2022
  27. 1/9 bundle-uri: short-circuit capability parsingDerrick Stolee via GitGitGadget, Sep 9, 2022
  28. Junio C HamanoSep 9, 2022
  29. Derrick StoleeSep 19, 2022
  30. 2/9 bundle-uri: use plain string in find_temp_filename()Derrick Stolee via GitGitGadget, Sep 9, 2022
  31. Junio C HamanoSep 9, 2022
  32. Derrick StoleeSep 19, 2022
  33. Junio C HamanoSep 19, 2022
  34. 4/9 bundle-uri: create base key-value pair parsingDerrick Stolee via GitGitGadget, Sep 9, 2022
  35. Jonathan TanSep 29, 2022
  36. 3/9 bundle-uri: create bundle_list struct and helpersDerrick Stolee via GitGitGadget, Sep 9, 2022
  37. 5/9 bundle-uri: create "key=value" line parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Sep 9, 2022
  38. 6/9 bundle-uri: unit test "key=value" parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Sep 9, 2022
  39. 7/9 bundle-uri: parse bundle list in config formatDerrick Stolee via GitGitGadget, Sep 9, 2022
  40. 8/9 bundle-uri: limit recursion depth for bundle listsDerrick Stolee via GitGitGadget, Sep 9, 2022
  41. 9/9 bundle-uri: fetch a list of bundlesDerrick Stolee via GitGitGadget, Sep 9, 2022
  42. Jonathan TanSep 29, 2022
  43. Derrick StoleeSep 30, 2022
  44. Derrick StoleeSep 26, 2022
  45. Junio C HamanoSep 26, 2022
  46. Jonathan TanSep 29, 2022
  47. Derrick StoleeSep 30, 2022
  48. 0/9 Bundle URIs III: Parse and download from bundle listsDerrick Stolee via GitGitGadget, Oct 4, 2022
  49. 1/9 bundle-uri: use plain string in find_temp_filename()Derrick Stolee via GitGitGadget, Oct 4, 2022
  50. 2/9 bundle-uri: create bundle_list struct and helpersDerrick Stolee via GitGitGadget, Oct 4, 2022
  51. 4/9 bundle-uri: create "key=value" line parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Oct 4, 2022
  52. 3/9 bundle-uri: create base key-value pair parsingDerrick Stolee via GitGitGadget, Oct 4, 2022
  53. 5/9 bundle-uri: unit test "key=value" parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Oct 4, 2022
  54. 6/9 bundle-uri: parse bundle list in config formatDerrick Stolee via GitGitGadget, Oct 4, 2022
  55. 9/9 bundle-uri: suppress stderr from remote-httpsDerrick Stolee via GitGitGadget, Oct 4, 2022
  56. 7/9 bundle-uri: limit recursion depth for bundle listsDerrick Stolee via GitGitGadget, Oct 4, 2022
  57. 8/9 bundle-uri: fetch a list of bundlesDerrick Stolee via GitGitGadget, Oct 4, 2022
  58. Jonathan TanOct 4, 2022
  59. Derrick StoleeOct 7, 2022
  60. 00/11 Bundle URIs III: Parse and download from bundle listsDerrick Stolee via GitGitGadget, Oct 10, 2022
  61. 02/11 bundle-uri: create bundle_list struct and helpersDerrick Stolee via GitGitGadget, Oct 10, 2022
  62. 01/11 bundle-uri: use plain string in find_temp_filename()Derrick Stolee via GitGitGadget, Oct 10, 2022
  63. 03/11 bundle-uri: create base key-value pair parsingDerrick Stolee via GitGitGadget, Oct 10, 2022
  64. 04/11 bundle-uri: create "key=value" line parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Oct 10, 2022
  65. 05/11 bundle-uri: unit test "key=value" parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Oct 10, 2022
  66. 06/11 bundle-uri: parse bundle list in config formatDerrick Stolee via GitGitGadget, Oct 10, 2022
  67. 07/11 bundle-uri: limit recursion depth for bundle listsDerrick Stolee via GitGitGadget, Oct 10, 2022
  68. 08/11 bundle: add flags to verify_bundle(), skip walkDerrick Stolee via GitGitGadget, Oct 10, 2022
  69. Junio C HamanoOct 10, 2022
  70. Derrick StoleeOct 10, 2022
  71. Junio C HamanoOct 10, 2022
  72. Derrick StoleeOct 11, 2022
  73. 09/11 bundle-uri: fetch a list of bundlesDerrick Stolee via GitGitGadget, Oct 10, 2022
  74. 10/11 bundle-uri: quiet failed unbundlingsDerrick Stolee via GitGitGadget, Oct 10, 2022
  75. 11/11 bundle-uri: suppress stderr from remote-httpsDerrick Stolee via GitGitGadget, Oct 10, 2022
  76. 00/12 Bundle URIs III: Parse and download from bundle listsDerrick Stolee via GitGitGadget, Oct 12, 2022
  77. 01/12 bundle-uri: use plain string in find_temp_filename()Derrick Stolee via GitGitGadget, Oct 12, 2022
  78. 02/12 bundle-uri: create bundle_list struct and helpersDerrick Stolee via GitGitGadget, Oct 12, 2022
  79. 03/12 bundle-uri: create base key-value pair parsingDerrick Stolee via GitGitGadget, Oct 12, 2022
  80. 05/12 bundle-uri: unit test "key=value" parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Oct 12, 2022
  81. 04/12 bundle-uri: create "key=value" line parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Oct 12, 2022
  82. 06/12 bundle-uri: parse bundle list in config formatDerrick Stolee via GitGitGadget, Oct 12, 2022
  83. 07/12 bundle-uri: limit recursion depth for bundle listsDerrick Stolee via GitGitGadget, Oct 12, 2022
  84. 10/12 bundle: add flags to verify_bundle()Derrick Stolee via GitGitGadget, Oct 12, 2022
  85. 08/12 bundle: properly clear all revision flagsDerrick Stolee via GitGitGadget, Oct 12, 2022
  86. Junio C HamanoOct 12, 2022
  87. 09/12 bundle-uri: fetch a list of bundlesDerrick Stolee via GitGitGadget, Oct 12, 2022
  88. Junio C HamanoOct 26, 2022
  89. 12/12 bundle-uri: suppress stderr from remote-httpsDerrick Stolee via GitGitGadget, Oct 12, 2022
  90. Junio C HamanoOct 26, 2022
  91. 11/12 bundle-uri: quiet failed unbundlingsDerrick Stolee via GitGitGadget, Oct 12, 2022
  92. Junio C HamanoOct 12, 2022
  93. Derrick StoleeOct 26, 2022
  94. Junio C HamanoOct 26, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.