{"thread":{"id":"58342","subject":"[PATCH 0/7] Bundle URIs III: Parse and download from bundle lists","startedAt":"2022-08-22T15:19:06Z","lastAt":"2022-10-26T19:09:49Z","messageCount":94,"participants":["Derrick Stolee via GitGitGadget","Ævar Arnfjörð Bjarmason via GitGitGadget","Junio C Hamano","Derrick Stolee","Glen Choo","Jonathan Tan","Teng Long","Josh Steadmon"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"461760","messageId":"pull.1333.git.1661181174.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":null,"subject":"[PATCH 0/7] Bundle URIs III: Parse and download from bundle lists","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-22T15:12:47Z","receivedAt":"2022-08-22T15:19:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the third series building the bundle URI feature. It is built on top\nof ds/bundle-uri-clone, which introduced 'git clone --bundle-uri=' where is\na URI to a bundle file. This series adds the capability of downloading and\nparsing a bundle list and then downloading the URIs in that list.\n\nThe core functionality of bundle lists is implemented by creating data\nstructures from a list of key-value pairs. These pairs can come from a\nplain-text file in Git config format, but in the future, we will support the\nlist being supplied by packet lines over Git's protocol v2 in the\n'bundle-uri' command (reserved for the next series).\n\nThe patches are organized in this way:\n\n 1. Patches 1-2 create the bundle list data structures and the logic for\n populating the list from key-value pairs.\n\n 2. Patches 3-4 teach Git to parse \"key=value\" lines to construct a bundle\n list. Add unit tests that ensure this logic constructs lists correctly.\n These patches are adapted from Ævar's RFC [1] and were previously seen\n in my combined RFC [2].\n\n 3. Patch 5 teaches Git to parse Git config files into bundle lists.\n\n 4. Patches 6-7 implement the ability to download a bundle list and\n recursively download the contained bundles (and possibly the bundle\n lists within). This is limited by a constant depth to avoid issues with\n cycles or otherwise incorrectly configured bundle lists.\n\n[1]\nhttps://lore.kernel.org/git/RFC-cover-v2-00.36-00000000000-20220418T165545Z-avarab@gmail.com/\n\n[2]\nhttps://lore.kernel.org/git/pull.1234.git.1653072042.gitgitgadget@gmail.com/\n\nAt the end of this series, users can bootstrap clones using 'git clone\n--bundle-uri= ' where points to a bundle list instead of a single bundle\nfile.\n\nAs outlined in the design document [1], the next steps after this are:\n\n 1. Implement the protocol v2 verb, re-using the bundle list logic from (2).\n Use this to auto-discover bundle URIs during 'git clone' (behind a\n config option). [2]\n 2. Implement the 'creationToken' heuristic, allowing incremental 'git\n fetch' commands to download a bundle list from a configured URI, and\n only download bundles that are new based on the creation token values.\n [3]\n\nI have prepared some of this work as pull requests on my personal fork so\ncurious readers can look ahead to where we are going:\n\n[3]\nhttps://lore.kernel.org/git/pull.1248.v3.git.1658757188.gitgitgadget@gmail.com\n\n[4] https://github.com/derrickstolee/git/pull/21\n\n[5] https://github.com/derrickstolee/git/pull/22\n\nThanks,\n\n * Stolee\n\nDerrick Stolee (5):\n bundle-uri: create bundle_list struct and helpers\n bundle-uri: create base key-value pair parsing\n bundle-uri: parse bundle list in config format\n bundle-uri: limit recursion depth for bundle lists\n bundle-uri: fetch a list of bundles\n\nÆvar Arnfjörð Bjarmason (2):\n bundle-uri: create \"key=value\" line parsing\n bundle-uri: unit test \"key=value\" parsing\n\n Documentation/config.txt | 2 +\n Documentation/config/bundle.txt | 22 ++\n Makefile | 1 +\n bundle-uri.c | 442 +++++++++++++++++++++++++++++++-\n bundle-uri.h | 98 ++++++-\n t/helper/test-bundle-uri.c | 90 +++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5558-clone-bundle-uri.sh | 93 +++++++\n t/t5750-bundle-uri-parse.sh | 141 ++++++++++\n t/test-lib-functions.sh | 11 +\n 11 files changed, 889 insertions(+), 13 deletions(-)\n create mode 100644 Documentation/config/bundle.txt\n create mode 100644 t/helper/test-bundle-uri.c\n create mode 100755 t/t5750-bundle-uri-parse.sh\n\n\nbase-commit: e21e663cd1942df29979d3e01f7eacb532727bb7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1333%2Fderrickstolee%2Fbundle-redo%2Flist-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1333/derrickstolee/bundle-redo/list-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1333\n-- \ngitgitgadget\n"},{"id":"461761","messageId":"c39438886586f30b020e8ac473c7facaa757f822.1661181174.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.git.1661181174.gitgitgadget@gmail.com","subject":"[PATCH 1/7] bundle-uri: create bundle_list struct and helpers","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-22T15:12:48Z","receivedAt":"2022-08-22T15:19:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt will likely be rare where a user uses a single bundle URI and expects\nthat URI to point to a bundle. Instead, that URI will likely be a list\nof bundles provided in some format. Alternatively, the Git server could\nadvertise a list of bundles.\n\nIn anticipation of these two ways of advertising multiple bundles,\ncreate a data structure that represents such a list. This will be\npopulated using a common API, but for now focus on what data can be\nrepresented.\n\nEach list contains a number of remote_bundle_info structs. These contain\nan 'id' that is used to uniquely identify them in the list, and also a\n'uri' that contains the location of its data. Finally, there is a strbuf\ncontaining the filename used when Git downloads the contents to disk.\n\nThe list itself stores these remote_bundle_info structs in a hashtable\nusing 'id' as the key. The order of the structs in the input is\nconsidered unimportant, but future modifications to the format and these\ndata structures will place ordering possibilities on the set. The list\nalso has a few \"global\" properties, including the version (used when\nparsing the list) and the mode. The mode is one of these two options:\n\n1. BUNDLE_MODE_ALL: all listed URIs are intended to be combined\n together. The client should download all of the advertised data to\n have a complete copy of the data.\n\n2. BUNDLE_MODE_ANY: any one listed item is sufficient to have a complete\n copy of the data. The client can choose arbitrarily from these\n options. In the future, the client may use pings to find the closest\n URI among geodistributed replicas, or use some other heuristic\n information added to the format.\n\nThis API is currently unused, but will soon be expanded with parsing\nlogic and then be consumed by the bundle URI download logic.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 61 ++++++++++++++++++++++++++++++++++++++++++++++++\n bundle-uri.h | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 126 insertions(+)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 4a8cc74ed05..ceeef0b6641 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -4,6 +4,67 @@\n #include \"object-store.h\"\n #include \"refs.h\"\n #include \"run-command.h\"\n+#include \"hashmap.h\"\n+#include \"pkt-line.h\"\n+\n+static int compare_bundles(const void *hashmap_cmp_fn_data,\n+\t\t\t const struct hashmap_entry *he1,\n+\t\t\t const struct hashmap_entry *he2,\n+\t\t\t const void *id)\n+{\n+\tconst struct remote_bundle_info *e1 =\n+\t\tcontainer_of(he1, const struct remote_bundle_info, ent);\n+\tconst struct remote_bundle_info *e2 =\n+\t\tcontainer_of(he2, const struct remote_bundle_info, ent);\n+\n+\treturn strcmp(e1->id, id ? (const char *)id : e2->id);\n+}\n+\n+void init_bundle_list(struct bundle_list *list)\n+{\n+\tmemset(list, 0, sizeof(*list));\n+\n+\t/* Implied defaults. */\n+\tlist->mode = BUNDLE_MODE_ALL;\n+\tlist->version = 1;\n+\n+\thashmap_init(&list->bundles, compare_bundles, NULL, 0);\n+}\n+\n+static int clear_remote_bundle_info(struct remote_bundle_info *bundle,\n+\t\t\t\t void *data)\n+{\n+\tfree(bundle->id);\n+\tfree(bundle->uri);\n+\tstrbuf_release(&bundle->file);\n+\treturn 0;\n+}\n+\n+void clear_bundle_list(struct bundle_list *list)\n+{\n+\tif (!list)\n+\t\treturn;\n+\n+\tfor_all_bundles_in_list(list, clear_remote_bundle_info, NULL);\n+\thashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent);\n+}\n+\n+int for_all_bundles_in_list(struct bundle_list *list,\n+\t\t\t bundle_iterator iter,\n+\t\t\t void *data)\n+{\n+\tstruct remote_bundle_info *info;\n+\tstruct hashmap_iter i;\n+\n+\thashmap_for_each_entry(&list->bundles, &i, info, ent) {\n+\t\tint result = iter(info, data);\n+\n+\t\tif (result)\n+\t\t\treturn result;\n+\t}\n+\n+\treturn 0;\n+}\n \n static int find_temp_filename(struct strbuf *name)\n {\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 8a152f1ef14..6692aa4b170 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -1,7 +1,72 @@\n #ifndef BUNDLE_URI_H\n #define BUNDLE_URI_H\n \n+#include \"hashmap.h\"\n+#include \"strbuf.h\"\n+\n struct repository;\n+struct string_list;\n+\n+/**\n+ * The remote_bundle_info struct contains information for a single bundle\n+ * URI. This may be initialized simply by a given URI or might have\n+ * additional metadata associated with it if the bundle was advertised by\n+ * a bundle list.\n+ */\n+struct remote_bundle_info {\n+\tstruct hashmap_entry ent;\n+\n+\t/**\n+\t * The 'id' is a name given to the bundle for reference\n+\t * by other bundle infos.\n+\t */\n+\tchar *id;\n+\n+\t/**\n+\t * The 'uri' is the location of the remote bundle so\n+\t * it can be downloaded on-demand. This will be NULL\n+\t * if there was no table of contents.\n+\t */\n+\tchar *uri;\n+\n+\t/**\n+\t * If the bundle has been downloaded, then 'file' is a\n+\t * filename storing its contents. Otherwise, 'file' is\n+\t * an empty string.\n+\t */\n+\tstruct strbuf file;\n+};\n+\n+#define REMOTE_BUNDLE_INFO_INIT { \\\n+\t.file = STRBUF_INIT, \\\n+}\n+\n+enum bundle_list_mode {\n+\tBUNDLE_MODE_NONE = 0,\n+\tBUNDLE_MODE_ALL,\n+\tBUNDLE_MODE_ANY\n+};\n+\n+/**\n+ * A bundle_list contains an unordered set of remote_bundle_info structs,\n+ * as well as information about the bundle listing, such as version and\n+ * mode.\n+ */\n+struct bundle_list {\n+\tint version;\n+\tenum bundle_list_mode mode;\n+\tstruct hashmap bundles;\n+};\n+\n+void init_bundle_list(struct bundle_list *list);\n+void clear_bundle_list(struct bundle_list *list);\n+\n+typedef int (*bundle_iterator)(struct remote_bundle_info *bundle,\n+\t\t\t void *data);\n+\n+int for_all_bundles_in_list(struct bundle_list *list,\n+\t\t\t bundle_iterator iter,\n+\t\t\t void *data);\n \n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n-- \ngitgitgadget\n\n"},{"id":"461762","messageId":"49c4f88b6fd804f0bd5c62d523b45431846f4cee.1661181174.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.git.1661181174.gitgitgadget@gmail.com","subject":"[PATCH 3/7] bundle-uri: create \"key=value\" line parsing","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-22T15:12:50Z","receivedAt":"2022-08-22T15:19:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nWhen advertising a bundle list over Git's protocol v2, we will use\npacket lines. Each line will be of the form \"key=value\" representing a\nbundle list. Connect the API necessary for Git's transport to the\nkey-value pair parsing created in the previous change.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 27 ++++++++++++++++++++++++++-\n bundle-uri.h | 14 +++++++++++++-\n 2 files changed, 39 insertions(+), 2 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex ade7eccce39..9a7d09349fe 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -71,7 +71,6 @@ int for_all_bundles_in_list(struct bundle_list *list,\n * Returns 0 if the key-value pair is understood. Returns 1 if the key\n * is not understood or the value is malformed.\n */\n-MAYBE_UNUSED\n static int bundle_list_update(const char *key, const char *value,\n \t\t\t struct bundle_list *list)\n {\n@@ -301,3 +300,29 @@ cleanup:\n \tstrbuf_release(&filename);\n \treturn result;\n }\n+\n+/**\n+ * General API for {transport,connect}.c etc.\n+ */\n+int bundle_uri_parse_line(struct bundle_list *list, const char *line)\n+{\n+\tint result;\n+\tconst char *equals;\n+\tstruct strbuf key = STRBUF_INIT;\n+\n+\tif (!strlen(line))\n+\t\treturn error(_(\"bundle-uri: got an empty line\"));\n+\n+\tequals = strchr(line, '=');\n+\n+\tif (!equals)\n+\t\treturn error(_(\"bundle-uri: line is not of the form 'key=value'\"));\n+\tif (line == equals || !*(equals + 1))\n+\t\treturn error(_(\"bundle-uri: line has empty key or value\"));\n+\n+\tstrbuf_add(&key, line, equals - line);\n+\tresult = bundle_list_update(key.buf, equals + 1, list);\n+\tstrbuf_release(&key);\n+\n+\treturn result;\n+}\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 6692aa4b170..f725c9796f7 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -76,4 +76,16 @@ int for_all_bundles_in_list(struct bundle_list *list,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n-#endif\n+/**\n+ * General API for {transport,connect}.c etc.\n+ */\n+\n+/**\n+ * Parse a \"key=value\" packet line from the bundle-uri verb.\n+ *\n+ * Returns 0 on success and non-zero on error.\n+ */\n+int bundle_uri_parse_line(struct bundle_list *list,\n+\t\t\t const char *line);\n+\n+#endif /* BUNDLE_URI_H */\n-- \ngitgitgadget\n\n"},{"id":"461763","messageId":"039e172849c6b028df1abf258666c77bd42b23fe.1661181174.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.git.1661181174.gitgitgadget@gmail.com","subject":"[PATCH 6/7] bundle-uri: limit recursion depth for bundle lists","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-22T15:12:53Z","receivedAt":"2022-08-22T15:19:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe next change will start allowing us to parse bundle lists that are\ndownloaded from a provided bundle URI. Those lists might point to other\nlists, which could proceed to an arbitrary depth (and even create\ncycles). Restructure fetch_bundle_uri() to have an internal version that\nhas a recursion depth. Compare that to a new max_bundle_uri_depth\nconstant that is twice as high as we expect this depth to be for any\nlegitimate use of bundle list linking.\n\nWe can consider making max_bundle_uri_depth a configurable value if\nthere is demonstrated value in the future.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 21 ++++++++++++++++++++-\n 1 file changed, 20 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex dca88ed1e89..c9f3df28b2f 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -334,11 +334,25 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \treturn result;\n }\n \n-int fetch_bundle_uri(struct repository *r, const char *uri)\n+/**\n+ * This limits the recursion on fetch_bundle_uri_internal() when following\n+ * bundle lists.\n+ */\n+static int max_bundle_uri_depth = 4;\n+\n+static int fetch_bundle_uri_internal(struct repository *r,\n+\t\t\t\t const char *uri,\n+\t\t\t\t int depth)\n {\n \tint result = 0;\n \tstruct strbuf filename = STRBUF_INIT;\n \n+\tif (depth >= max_bundle_uri_depth) {\n+\t\twarning(_(\"exceeded bundle URI recursion limit (%d)\"),\n+\t\t\tmax_bundle_uri_depth);\n+\t\treturn -1;\n+\t}\n+\n \tif ((result = find_temp_filename(&filename)))\n \t\tgoto cleanup;\n \n@@ -363,6 +377,11 @@ cleanup:\n \treturn result;\n }\n \n+int fetch_bundle_uri(struct repository *r, const char *uri)\n+{\n+\treturn fetch_bundle_uri_internal(r, uri, 0);\n+}\n+\n /**\n * General API for {transport,connect}.c etc.\n */\n-- \ngitgitgadget\n\n"},{"id":"461764","messageId":"7580e1f09aff2acdf7a7cb86bf8dc7e294cffd33.1661181174.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.git.1661181174.gitgitgadget@gmail.com","subject":"[PATCH 4/7] bundle-uri: unit test \"key=value\" parsing","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-22T15:12:51Z","receivedAt":"2022-08-22T15:19:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nCreate a new 'test-tool bundle-uri' test helper. This helper will assist\nin testing logic deep in the bundle URI feature.\n\nThis change introduces the 'parse-key-values' subcommand, which parses\nstdin as a list of lines. These are fed into bundle_uri_parse_line() to\ntest how we construct a 'struct bundle_list' from that data. The list is\nthen output to stdout as if the key-value pairs were a Git config file.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n bundle-uri.c | 33 ++++++++++++++\n bundle-uri.h | 3 ++\n t/helper/test-bundle-uri.c | 63 +++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5750-bundle-uri-parse.sh | 91 +++++++++++++++++++++++++++++++++++++\n t/test-lib-functions.sh | 11 +++++\n 8 files changed, 204 insertions(+)\n create mode 100644 t/helper/test-bundle-uri.c\n create mode 100755 t/t5750-bundle-uri-parse.sh\n\ndiff --git a/Makefile b/Makefile\nindex 7d5f48069ea..7dee0329c49 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -722,6 +722,7 @@ PROGRAMS += $(patsubst %.o,git-%$X,$(PROGRAM_OBJS))\n TEST_BUILTINS_OBJS += test-advise.o\n TEST_BUILTINS_OBJS += test-bitmap.o\n TEST_BUILTINS_OBJS += test-bloom.o\n+TEST_BUILTINS_OBJS += test-bundle-uri.o\n TEST_BUILTINS_OBJS += test-chmtime.o\n TEST_BUILTINS_OBJS += test-config.o\n TEST_BUILTINS_OBJS += test-crontab.o\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 9a7d09349fe..d56c5e33d5f 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -66,6 +66,39 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \treturn 0;\n }\n \n+static int summarize_bundle(struct remote_bundle_info *info, void *data)\n+{\n+\tFILE *fp = data;\n+\tfprintf(fp, \"[bundle \\\"%s\\\"]\\n\", info->id);\n+\tfprintf(fp, \"\\turi = %s\\n\", info->uri);\n+\treturn 0;\n+}\n+\n+void print_bundle_list(FILE *fp, struct bundle_list *list)\n+{\n+\tconst char *mode;\n+\n+\tswitch (list->mode) {\n+\tcase BUNDLE_MODE_ALL:\n+\t\tmode = \"all\";\n+\t\tbreak;\n+\n+\tcase BUNDLE_MODE_ANY:\n+\t\tmode = \"any\";\n+\t\tbreak;\n+\n+\tcase BUNDLE_MODE_NONE:\n+\tdefault:\n+\t\tmode = \"\";\n+\t}\n+\n+\tprintf(\"[bundle]\\n\");\n+\tprintf(\"\\tversion = %d\\n\", list->version);\n+\tprintf(\"\\tmode = %s\\n\", mode);\n+\n+\tfor_all_bundles_in_list(list, summarize_bundle, fp);\n+}\n+\n /**\n * Given a key-value pair, update the state of the given bundle list.\n * Returns 0 if the key-value pair is understood. Returns 1 if the key\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex f725c9796f7..41a1510a4ac 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -68,6 +68,9 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \t\t\t bundle_iterator iter,\n \t\t\t void *data);\n \n+struct FILE;\n+void print_bundle_list(FILE *fp, struct bundle_list *list);\n+\n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n * based on that information.\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nnew file mode 100644\nindex 00000000000..5cb0c9196fa\n--- /dev/null\n+++ b/t/helper/test-bundle-uri.c\n@@ -0,0 +1,63 @@\n+#include \"test-tool.h\"\n+#include \"parse-options.h\"\n+#include \"bundle-uri.h\"\n+#include \"strbuf.h\"\n+#include \"string-list.h\"\n+\n+static int cmd__bundle_uri_parse_key_values(int argc, const char **argv)\n+{\n+\tconst char *usage[] = {\n+\t\t\"test-tool bundle-uri parse-key-values []\",\n+\t\tNULL\n+\t};\n+\tstruct option options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL, options, usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION |\n+\t\t\t PARSE_OPT_KEEP_ARGV0);\n+\tif (argc == 1)\n+\t\tgoto usage;\n+\n+\tif (!strcmp(argv[1], \"parse-key-values\"))\n+\t\treturn cmd__bundle_uri_parse_key_values(argc - 1, argv + 1);\n+\terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n+\n+usage:\n+\tusage_with_options(usage, options);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 318fdbab0c3..fbe2d9d8108 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -17,6 +17,7 @@ static struct test_cmd cmds[] = {\n \t{ \"advise\", cmd__advise_if_enabled },\n \t{ \"bitmap\", cmd__bitmap },\n \t{ \"bloom\", cmd__bloom },\n+\t{ \"bundle-uri\", cmd__bundle_uri },\n \t{ \"chmtime\", cmd__chmtime },\n \t{ \"config\", cmd__config },\n \t{ \"crontab\", cmd__crontab },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex bb799271631..b2aa1f39a8f 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -7,6 +7,7 @@\n int cmd__advise_if_enabled(int argc, const char **argv);\n int cmd__bitmap(int argc, const char **argv);\n int cmd__bloom(int argc, const char **argv);\n+int cmd__bundle_uri(int argc, const char **argv);\n int cmd__chmtime(int argc, const char **argv);\n int cmd__config(int argc, const char **argv);\n int cmd__crontab(int argc, const char **argv);\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nnew file mode 100755\nindex 00000000000..675c1f1d2f4\n--- /dev/null\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -0,0 +1,91 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri bundle_uri_parse_line()\"\n+\n+TEST_NO_CREATE_REPO=1\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success 'bundle_uri_parse_line() just URIs' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-key-values actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' '\n+\tcat >in <<-\\EOF &&\n+\t=bogus-value\n+\tbogus-key=\n+\tEOF\n+\n+\tcat >err.expect <<-EOF &&\n+\terror: bundle-uri: line has empty key or value\n+\terror: bad line: '\\''=bogus-value'\\''\n+\terror: bundle-uri: line has empty key or value\n+\terror: bad line: '\\''bogus-key='\\''\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty lines' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\terror: bundle-uri: got an empty line\n+\terror: bad line: '\\'''\\''\n+\terror: bundle-uri: got an empty line\n+\terror: bad line: '\\'''\\''\n+\tEOF\n+\n+\t# We fail, but try to continue parsing regardless\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_done\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 6da7273f1d5..3175d665add 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1956,3 +1956,14 @@ test_is_magic_mtime () {\n \trm -f .git/test-mtime-actual\n \treturn $ret\n }\n+\n+# Given two filenames, parse both using 'git config --list --file'\n+# and compare the sorted output of those commands. Useful when\n+# wanting to ignore whitespace differences and sorting concerns.\n+test_cmp_config_output () {\n+\tgit config --list --file=\"$1\" >config-expect &&\n+\tgit config --list --file=\"$2\" >config-actual &&\n+\tsort config-expect >sorted-expect &&\n+\tsort config-actual >sorted-actual &&\n+\ttest_cmp sorted-expect sorted-actual\n+}\n-- \ngitgitgadget\n\n"},{"id":"461765","messageId":"1d1bd9c710327b4d705cfede017771da7fb6ec52.1661181174.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.git.1661181174.gitgitgadget@gmail.com","subject":"[PATCH 5/7] bundle-uri: parse bundle list in config format","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-22T15:12:52Z","receivedAt":"2022-08-22T15:19:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a bundle provider wants to operate independently from a Git remote,\nthey want to provide a single, consistent URI that users can use in\ntheir 'git clone --bundle-uri' commands. At this point, the Git client\nexpects that URI to be a single bundle that can be unbundled and used to\nbootstrap the rest of the clone from the Git server. This single bundle\ncannot be re-used to assist with future incremental fetches.\n\nTo allow for the incremental fetch case, teach Git to understand a\nbundle list that could be advertised at an independent bundle URI. Such\na bundle list is likely to be inspected by human readers, even if only\nby the bundle provider creating the list. For this reason, we can take\nour expected \"key=value\" pairs and instead format them using Git config\nformat.\n\nCreate parse_bundle_list_in_config_format() to parse a file in config\nformat and convert that into a 'struct bundle_list' filled with its\nunderstanding of the contents.\n\nBe careful to call git_config_from_file_with_options() because the\ndefault action for git_config_from_file() is to die() on a parsing\nerror. The current warning isn't particularly helpful if it arises to a\nuser, but it will be made more verbose at a higher layer later.\n\nUpdate 'test-tool bundle-uri' to take this config file format as input.\nIt uses a filename instead of stdin because there is no existing way to\nparse a FILE pointer in the config machinery. Using\ngit_config_from_mem() is overly complicated and more likely to introduce\nbugs than this simpler version. I would rather have a slightly confusing\ntest helper than complicated product code.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 29 +++++++++++++++++++++\n bundle-uri.h | 10 ++++++++\n t/helper/test-bundle-uri.c | 45 ++++++++++++++++++++++++++-------\n t/t5750-bundle-uri-parse.sh | 50 +++++++++++++++++++++++++++++++++++++\n 4 files changed, 125 insertions(+), 9 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex d56c5e33d5f..dca88ed1e89 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -6,6 +6,7 @@\n #include \"run-command.h\"\n #include \"hashmap.h\"\n #include \"pkt-line.h\"\n+#include \"config.h\"\n \n static int compare_bundles(const void *hashmap_cmp_fn_data,\n \t\t\t const struct hashmap_entry *he1,\n@@ -172,6 +173,34 @@ static int bundle_list_update(const char *key, const char *value,\n \treturn 0;\n }\n \n+static int config_to_bundle_list(const char *key, const char *value, void *data)\n+{\n+\tstruct bundle_list *list = data;\n+\treturn bundle_list_update(key, value, list);\n+}\n+\n+int parse_bundle_list_in_config_format(const char *uri,\n+\t\t\t\t const char *filename,\n+\t\t\t\t struct bundle_list *list)\n+{\n+\tint result;\n+\tstruct config_options opts = {\n+\t\t.error_action = CONFIG_ERROR_ERROR,\n+\t};\n+\n+\tlist->mode = BUNDLE_MODE_NONE;\n+\tresult = git_config_from_file_with_options(config_to_bundle_list,\n+\t\t\t\t\t\t filename, list,\n+\t\t\t\t\t\t &opts);\n+\n+\tif (!result && list->mode == BUNDLE_MODE_NONE) {\n+\t\twarning(_(\"bundle list at '%s' has no mode\"), uri);\n+\t\tresult = 1;\n+\t}\n+\n+\treturn result;\n+}\n+\n static int find_temp_filename(struct strbuf *name)\n {\n \tint fd;\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 41a1510a4ac..294ac804140 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -71,6 +71,16 @@ int for_all_bundles_in_list(struct bundle_list *list,\n struct FILE;\n void print_bundle_list(FILE *fp, struct bundle_list *list);\n \n+/**\n+ * A bundle URI may point to a bundle list where the key=value\n+ * pairs are provided in config file format. This method is\n+ * exposed publicly for testing purposes.\n+ */\n+\n+int parse_bundle_list_in_config_format(const char *uri,\n+\t\t\t\t const char *filename,\n+\t\t\t\t struct bundle_list *list);\n+\n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n * based on that information.\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex 5cb0c9196fa..23ce0eebca3 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -4,27 +4,52 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n \n-static int cmd__bundle_uri_parse_key_values(int argc, const char **argv)\n+enum input_mode {\n+\tKEY_VALUE_PAIRS,\n+\tCONFIG_FILE,\n+};\n+\n+static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mode)\n {\n-\tconst char *usage[] = {\n+\tconst char *key_value_usage[] = {\n \t\t\"test-tool bundle-uri parse-key-values \",\n+\t\tNULL\n+\t};\n \tstruct option options[] = {\n \t\tOPT_END(),\n \t};\n+\tconst char **usage = key_value_usage;\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct bundle_list list;\n \tint err = 0;\n \n-\targc = parse_options(argc, argv, NULL, options, usage, 0);\n-\tif (argc)\n-\t\tgoto usage;\n+\tif (mode == CONFIG_FILE)\n+\t\tusage = config_usage;\n+\n+\targc = parse_options(argc, argv, NULL, options, usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tinit_bundle_list(&list);\n-\twhile (strbuf_getline(&sb, stdin) != EOF) {\n-\t\tif (bundle_uri_parse_line(&list, sb.buf) < 0)\n-\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\n+\tswitch (mode) {\n+\tcase KEY_VALUE_PAIRS:\n+\t\tif (argc)\n+\t\t\tgoto usage;\n+\t\twhile (strbuf_getline(&sb, stdin) != EOF) {\n+\t\t\tif (bundle_uri_parse_line(&list, sb.buf) < 0)\n+\t\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\t\t}\n+\t\tbreak;\n+\n+\tcase CONFIG_FILE:\n+\t\tif (argc != 1)\n+\t\t\tgoto usage;\n+\t\terr = parse_bundle_list_in_config_format(\"\", argv[0], &list);\n+\t\tbreak;\n \t}\n \tstrbuf_release(&sb);\n \n@@ -55,7 +80,9 @@ int cmd__bundle_uri(int argc, const char **argv)\n \t\tgoto usage;\n \n \tif (!strcmp(argv[1], \"parse-key-values\"))\n-\t\treturn cmd__bundle_uri_parse_key_values(argc - 1, argv + 1);\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS);\n+\tif (!strcmp(argv[1], \"parse-config\"))\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE);\n \terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n \n usage:\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex 675c1f1d2f4..dd9dc36bfd7 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -88,4 +88,54 @@ test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty lines' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format: just URIs' '\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config expect >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'parse config format edge cases: empty key or value' '\n+\tcat >in1 <<-\\EOF &&\n+\t= bogus-value\n+\tEOF\n+\n+\tcat >err1 <<-EOF &&\n+\terror: bad config line 1 in file in1\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = \n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-config in1 >actual 2>err &&\n+\ttest_cmp err1 err &&\n+\ttest_cmp_config_output expect actual &&\n+\n+\tcat >in2 <<-\\EOF &&\n+\tbogus-key =\n+\tEOF\n+\n+\tcat >err2 <<-EOF &&\n+\twarning: bundle list at '\\'''\\'' has no mode\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-config in2 >actual 2>err &&\n+\ttest_cmp err2 err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"461766","messageId":"7e4e4656e530395d055abac2a59e93866c9a0de2.1661181174.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.git.1661181174.gitgitgadget@gmail.com","subject":"[PATCH 2/7] bundle-uri: create base key-value pair parsing","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-22T15:12:49Z","receivedAt":"2022-08-22T15:19:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThere will be two primary ways to advertise a bundle list: as a list of\npacket lines in Git's protocol v2 and as a config file served from a\nbundle URI. Both of these fundamentally use a list of key-value pairs.\nWe will use the same set of key-value pairs across these formats.\n\nCreate a new bundle_list_update() method that is currently unusued, but\nwill be used in the next change. It inspects each key to see if it is\nunderstood and then applies it to the given bundle_list. Here are the\nkeys that we teach Git to understand:\n\n* bundle.version: This value should be an integer. Git currently\n understands only version 1 and will ignore the list if the version is\n any other value. This version can be increased in the future if we\n need to add new keys that Git should not ignore. We can add new\n \"heuristic\" keys without incrementing the version.\n\n* bundle.mode: This value should be one of \"all\" or \"any\". If this\n mode is not understood, then Git will ignore the list. This mode\n indicates whether Git needs all of the bundle list items to make a\n complete view of the content or if any single item is sufficient.\n\nThe rest of the keys use a bundle identifier \"\" as part of the key\nname. Keys using the same \"\" describe a single bundle list item.\n\n* bundle..uri: This stores the URI of the bundle item. This\n currently is expected to be an absolute URI, but will be relaxed to be\n a relative URI in the future.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 2 +\n Documentation/config/bundle.txt | 22 ++++++++++\n bundle-uri.c | 74 +++++++++++++++++++++++++++++++++\n 3 files changed, 98 insertions(+)\n create mode 100644 Documentation/config/bundle.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex e376d547ce0..4280af6992e 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -387,6 +387,8 @@ include::config/branch.txt[]\n \n include::config/browser.txt[]\n \n+include::config/bundle.txt[]\n+\n include::config/checkout.txt[]\n \n include::config/clean.txt[]\ndiff --git a/Documentation/config/bundle.txt b/Documentation/config/bundle.txt\nnew file mode 100644\nindex 00000000000..3515bfe38d1\n--- /dev/null\n+++ b/Documentation/config/bundle.txt\n@@ -0,0 +1,22 @@\n+bundle.*::\n+\tThe `bundle.*` keys are used when communicating a list of bundle URIs\n+\tSee link:technical/bundle-uri.html[the bundle URI design document] for\n+\tmore details.\n+\n+bundle.version::\n+\tThis integer value advertises the version of the bundle list format\n+\tused by the bundle list. Currently, the only accepted value is `1`.\n+\n+bundle.mode::\n+\tThis string value should be either `all` or `any`. This value describes\n+\twhether all of the advertised bundles are required to unbundle a\n+\tcomplete understanding of the bundled information (`all`) or if any one\n+\tof the listed bundle URIs is sufficient (`any`).\n+\n+bundle..*::\n+\tThe `bundle..*` keys are used to describe a single item in the\n+\tbundle list, grouped under `` for identification purposes.\n+\n+bundle..uri::\n+\tThis string value defines the URI by which Git can reach the contents\n+\tof this ``. This URI may be a bundle file or another bundle list.\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex ceeef0b6641..ade7eccce39 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -66,6 +66,80 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \treturn 0;\n }\n \n+/**\n+ * Given a key-value pair, update the state of the given bundle list.\n+ * Returns 0 if the key-value pair is understood. Returns 1 if the key\n+ * is not understood or the value is malformed.\n+ */\n+MAYBE_UNUSED\n+static int bundle_list_update(const char *key, const char *value,\n+\t\t\t struct bundle_list *list)\n+{\n+\tconst char *pkey, *dot;\n+\tstruct strbuf id = STRBUF_INIT;\n+\tstruct remote_bundle_info lookup = REMOTE_BUNDLE_INFO_INIT;\n+\tstruct remote_bundle_info *bundle;\n+\n+\tif (!skip_prefix(key, \"bundle.\", &pkey))\n+\t\treturn 1;\n+\n+\tdot = strchr(pkey, '.');\n+\tif (!dot) {\n+\t\tif (!strcmp(pkey, \"version\")) {\n+\t\t\tint version = atoi(value);\n+\t\t\tif (version != 1)\n+\t\t\t\treturn 1;\n+\n+\t\t\tlist->version = version;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\tif (!strcmp(pkey, \"mode\")) {\n+\t\t\tif (!strcmp(value, \"all\"))\n+\t\t\t\tlist->mode = BUNDLE_MODE_ALL;\n+\t\t\telse if (!strcmp(value, \"any\"))\n+\t\t\t\tlist->mode = BUNDLE_MODE_ANY;\n+\t\t\telse\n+\t\t\t\treturn 1;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\t/* Ignore other unknown global keys. */\n+\t\treturn 0;\n+\t}\n+\n+\tstrbuf_add(&id, pkey, dot - pkey);\n+\tdot++;\n+\n+\t/*\n+\t * Check for an existing bundle with this , or create one\n+\t * if necessary.\n+\t */\n+\tlookup.id = id.buf;\n+\thashmap_entry_init(&lookup.ent, strhash(lookup.id));\n+\tif (!(bundle = hashmap_get_entry(&list->bundles, &lookup, ent, NULL))) {\n+\t\tCALLOC_ARRAY(bundle, 1);\n+\t\tbundle->id = strbuf_detach(&id, NULL);\n+\t\tstrbuf_init(&bundle->file, 0);\n+\t\thashmap_entry_init(&bundle->ent, strhash(bundle->id));\n+\t\thashmap_add(&list->bundles, &bundle->ent);\n+\t}\n+\tstrbuf_release(&id);\n+\n+\tif (!strcmp(dot, \"uri\")) {\n+\t\tfree(bundle->uri);\n+\t\tbundle->uri = xstrdup(value);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * At this point, we ignore any information that we don't\n+\t * understand, assuming it to be hints for a heuristic the client\n+\t * does not currently understand.\n+\t */\n+\treturn 0;\n+}\n+\n static int find_temp_filename(struct strbuf *name)\n {\n \tint fd;\n-- \ngitgitgadget\n\n"},{"id":"461767","messageId":"7b45c06cc9e0294311d9f00d40eb1fa4f8f146f9.1661181174.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.git.1661181174.gitgitgadget@gmail.com","subject":"[PATCH 7/7] bundle-uri: fetch a list of bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-08-22T15:12:54Z","receivedAt":"2022-08-22T15:19:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen the content at a given bundle URI is not understood as a bundle\n(based on inspecting the initial content), then Git currently gives up\nand ignores that content. Independent bundle providers may want to split\nup the bundle content into multiple bundles, but still make them\navailable from a single URI.\n\nTeach Git to attempt parsing the bundle URI content as a Git config file\nproviding the key=value pairs for a bundle list. Git then looks at the\nmode of the list to see if ANY single bundle is sufficient or if ALL\nbundles are required. The content at the selected URIs are downloaded\nand the content is inspected again, creating a recursive process.\n\nTo guard the recursion against malformed or malicious content, limit the\nrecursion depth to a reasonable four for now. This can be converted to a\nconfigured value in the future if necessary. The value of four is twice\nas high as expected to be useful (a bundle list is unlikely to point to\nmore bundle lists).\n\nTo test this scenario, create an interesting bundle topology where three\nincremental bundles are built on top of a single full bundle. By using a\nmerge commit, the two middle bundles are \"independent\" in that they do\nnot require each other in order to unbundle themselves. They each only\nneed the base bundle. The bundle containing the merge commit requires\nboth of the middle bundles, though. This leads to some interesting\ndecisions when unbundling, especially when we later implement heuristics\nthat promote downloading bundles until the prerequisite commits are\nsatisfied.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 211 +++++++++++++++++++++++++++++++++---\n bundle-uri.h | 6 +\n t/t5558-clone-bundle-uri.sh | 93 ++++++++++++++++\n 3 files changed, 293 insertions(+), 17 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex c9f3df28b2f..37867afca27 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -35,9 +35,10 @@ void init_bundle_list(struct bundle_list *list)\n static int clear_remote_bundle_info(struct remote_bundle_info *bundle,\n \t\t\t\t void *data)\n {\n-\tfree(bundle->id);\n-\tfree(bundle->uri);\n+\tFREE_AND_NULL(bundle->id);\n+\tFREE_AND_NULL(bundle->uri);\n \tstrbuf_release(&bundle->file);\n+\tbundle->unbundled = 0;\n \treturn 0;\n }\n \n@@ -334,18 +335,102 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \treturn result;\n }\n \n+struct bundle_list_context {\n+\tstruct repository *r;\n+\tstruct bundle_list *list;\n+\tenum bundle_list_mode mode;\n+\tint count;\n+\tint depth;\n+};\n+\n+/*\n+ * This early definition is necessary because we use indirect recursion:\n+ *\n+ * While iterating through a bundle list that was downloaded as part\n+ * of fetch_bundle_uri_internal(), iterator methods eventually call it\n+ * again, but with depth + 1.\n+ */\n+static int fetch_bundle_uri_internal(struct repository *r,\n+\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t int depth,\n+\t\t\t\t struct bundle_list *list);\n+\n+static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data)\n+{\n+\tstruct bundle_list_context *ctx = data;\n+\n+\tif (ctx->mode == BUNDLE_MODE_ANY && ctx->count)\n+\t\treturn 0;\n+\n+\tctx->count++;\n+\treturn fetch_bundle_uri_internal(ctx->r, bundle, ctx->depth + 1, ctx->list);\n+}\n+\n+static int download_bundle_list(struct repository *r,\n+\t\t\t\tstruct bundle_list *local_list,\n+\t\t\t\tstruct bundle_list *global_list,\n+\t\t\t\tint depth)\n+{\n+\tstruct bundle_list_context ctx = {\n+\t\t.r = r,\n+\t\t.list = global_list,\n+\t\t.depth = depth + 1,\n+\t\t.mode = local_list->mode,\n+\t};\n+\n+\treturn for_all_bundles_in_list(local_list, download_bundle_to_file, &ctx);\n+}\n+\n+static int fetch_bundle_list_in_config_format(struct repository *r,\n+\t\t\t\t\t struct bundle_list *global_list,\n+\t\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t\t int depth)\n+{\n+\tint result;\n+\tstruct bundle_list list_from_bundle;\n+\n+\tinit_bundle_list(&list_from_bundle);\n+\n+\tif ((result = parse_bundle_list_in_config_format(bundle->uri,\n+\t\t\t\t\t\t\t bundle->file.buf,\n+\t\t\t\t\t\t\t &list_from_bundle)))\n+\t\tgoto cleanup;\n+\n+\tif (list_from_bundle.mode == BUNDLE_MODE_NONE) {\n+\t\twarning(_(\"unrecognized bundle mode from URI '%s'\"),\n+\t\t\tbundle->uri);\n+\t\tresult = -1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif ((result = download_bundle_list(r, &list_from_bundle,\n+\t\t\t\t\t global_list, depth)))\n+\t\tgoto cleanup;\n+\n+cleanup:\n+\tclear_bundle_list(&list_from_bundle);\n+\treturn result;\n+}\n+\n /**\n * This limits the recursion on fetch_bundle_uri_internal() when following\n * bundle lists.\n */\n static int max_bundle_uri_depth = 4;\n \n+/**\n+ * Recursively download all bundles advertised at the given URI\n+ * to files. If the file is a bundle, then add it to the given\n+ * 'list'. Otherwise, expect a bundle list and recurse on the\n+ * URIs in that list according to the list mode (ANY or ALL).\n+ */\n static int fetch_bundle_uri_internal(struct repository *r,\n-\t\t\t\t const char *uri,\n-\t\t\t\t int depth)\n+\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t int depth,\n+\t\t\t\t struct bundle_list *list)\n {\n \tint result = 0;\n-\tstruct strbuf filename = STRBUF_INIT;\n+\tstruct remote_bundle_info *bcopy;\n \n \tif (depth >= max_bundle_uri_depth) {\n \t\twarning(_(\"exceeded bundle URI recursion limit (%d)\"),\n@@ -353,33 +438,125 @@ static int fetch_bundle_uri_internal(struct repository *r,\n \t\treturn -1;\n \t}\n \n-\tif ((result = find_temp_filename(&filename)))\n+\tif (!bundle->file.len &&\n+\t (result = find_temp_filename(&bundle->file)))\n \t\tgoto cleanup;\n \n-\tif ((result = copy_uri_to_file(filename.buf, uri))) {\n-\t\twarning(_(\"failed to download bundle from URI '%s'\"), uri);\n+\tif ((result = copy_uri_to_file(bundle->file.buf, bundle->uri))) {\n+\t\twarning(_(\"failed to download bundle from URI '%s'\"), bundle->uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = !is_bundle(filename.buf, 0))) {\n-\t\twarning(_(\"file at URI '%s' is not a bundle\"), uri);\n+\tif ((result = !is_bundle(bundle->file.buf, 1))) {\n+\t\tresult = fetch_bundle_list_in_config_format(\n+\t\t\t\tr, list, bundle, depth);\n+\t\tif (result)\n+\t\t\twarning(_(\"file at URI '%s' is not a bundle or bundle list\"),\n+\t\t\t\tbundle->uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = unbundle_from_file(r, filename.buf))) {\n-\t\twarning(_(\"failed to unbundle bundle from URI '%s'\"), uri);\n-\t\tgoto cleanup;\n-\t}\n+\t/* Copy the bundle and insert it into the global list. */\n+\tCALLOC_ARRAY(bcopy, 1);\n+\tbcopy->id = xstrdup(bundle->id);\n+\tstrbuf_init(&bcopy->file, 0);\n+\tstrbuf_add(&bcopy->file, bundle->file.buf, bundle->file.len);\n+\thashmap_entry_init(&bcopy->ent, strhash(bcopy->id));\n+\thashmap_add(&list->bundles, &bcopy->ent);\n \n cleanup:\n-\tunlink(filename.buf);\n-\tstrbuf_release(&filename);\n+\tif (result)\n+\t\tunlink(bundle->file.buf);\n \treturn result;\n }\n \n+struct attempt_unbundle_context {\n+\tstruct repository *r;\n+\tint success_count;\n+\tint failure_count;\n+};\n+\n+static int attempt_unbundle(struct remote_bundle_info *info, void *data)\n+{\n+\tstruct attempt_unbundle_context *ctx = data;\n+\n+\tif (info->unbundled || !unbundle_from_file(ctx->r, info->file.buf)) {\n+\t\tctx->success_count++;\n+\t\tinfo->unbundled = 1;\n+\t} else {\n+\t\tctx->failure_count++;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int unbundle_all_bundles(struct repository *r,\n+\t\t\t\tstruct bundle_list *list)\n+{\n+\tint last_success_count = -1;\n+\tstruct attempt_unbundle_context ctx = {\n+\t\t.r = r,\n+\t};\n+\n+\t/*\n+\t * Iterate through all bundles looking for ones that can\n+\t * successfully unbundle. If any succeed, then perhaps another\n+\t * will succeed in the next attempt.\n+\t */\n+\twhile (last_success_count < ctx.success_count) {\n+\t\tlast_success_count = ctx.success_count;\n+\n+\t\tctx.success_count = 0;\n+\t\tctx.failure_count = 0;\n+\t\tfor_all_bundles_in_list(list, attempt_unbundle, &ctx);\n+\t}\n+\n+\tif (ctx.success_count)\n+\t\tgit_config_set_multivar_gently(\"log.excludedecoration\",\n+\t\t\t\t\t\t\"refs/bundle/\",\n+\t\t\t\t\t\t\"refs/bundle/\",\n+\t\t\t\t\t\tCONFIG_FLAGS_FIXED_VALUE |\n+\t\t\t\t\t\tCONFIG_FLAGS_MULTI_REPLACE);\n+\n+\tif (ctx.failure_count)\n+\t\twarning(_(\"failed to unbundle %d bundles\"),\n+\t\t\tctx.failure_count);\n+\n+\treturn 0;\n+}\n+\n+static int unlink_bundle(struct remote_bundle_info *info, void *data)\n+{\n+\tif (info->file.buf)\n+\t\tunlink_or_warn(info->file.buf);\n+\treturn 0;\n+}\n+\n int fetch_bundle_uri(struct repository *r, const char *uri)\n {\n-\treturn fetch_bundle_uri_internal(r, uri, 0);\n+\tint result;\n+\tstruct bundle_list list;\n+\tstruct remote_bundle_info bundle = {\n+\t\t.uri = xstrdup(uri),\n+\t\t.id = xstrdup(\"\"),\n+\t\t.file = STRBUF_INIT,\n+\t};\n+\n+\tinit_bundle_list(&list);\n+\n+\t/* If a bundle is added to this global list, then it is required. */\n+\tlist.mode = BUNDLE_MODE_ALL;\n+\n+\tif ((result = fetch_bundle_uri_internal(r, &bundle, 0, &list)))\n+\t\tgoto cleanup;\n+\n+\tresult = unbundle_all_bundles(r, &list);\n+\n+cleanup:\n+\tfor_all_bundles_in_list(&list, unlink_bundle, NULL);\n+\tclear_bundle_list(&list);\n+\tclear_remote_bundle_info(&bundle, NULL);\n+\treturn result;\n }\n \n /**\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 294ac804140..e9d85a6ecfb 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -35,6 +35,12 @@ struct remote_bundle_info {\n \t * an empty string.\n \t */\n \tstruct strbuf file;\n+\n+\t/**\n+\t * If the bundle has been unbundled successfully, then\n+\t * this boolean is true.\n+\t */\n+\tunsigned unbundled:1;\n };\n \n #define REMOTE_BUNDLE_INFO_INIT { \\\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex ad666a2d28a..592790b49f0 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -41,6 +41,72 @@ test_expect_success 'clone with file:// bundle' '\n \ttest_cmp expect actual\n '\n \n+# To get interesting tests for bundle lists, we need to construct a\n+# somewhat-interesting commit history.\n+#\n+# ---------------- bundle-4\n+#\n+# 4\n+# / \\\n+# ----|---|------- bundle-3\n+# | |\n+# | 3\n+# | |\n+# ----|---|------- bundle-2\n+# | |\n+# 2 |\n+# | |\n+# ----|---|------- bundle-1\n+# \\ /\n+# 1\n+# |\n+# (previous commits)\n+test_expect_success 'construct incremental bundle list' '\n+\t(\n+\t\tcd clone-from &&\n+\t\tgit checkout -b base &&\n+\t\ttest_commit 1 &&\n+\t\tgit checkout -b left &&\n+\t\ttest_commit 2 &&\n+\t\tgit checkout -b right base &&\n+\t\ttest_commit 3 &&\n+\t\tgit checkout -b merge left &&\n+\t\tgit merge right -m \"4\" &&\n+\n+\t\tgit bundle create bundle-1.bundle base &&\n+\t\tgit bundle create bundle-2.bundle base..left &&\n+\t\tgit bundle create bundle-3.bundle base..right &&\n+\t\tgit bundle create bundle-4.bundle merge --not left right\n+\t)\n+'\n+\n+test_expect_success 'clone bundle list (file, no heuristic)' '\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-3.bundle\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-4.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" . clone-list-file &&\n+\tfor oid in $(git -C clone-from for-each-ref --format=\"%(objectname)\")\n+\tdo\n+\t\tgit -C clone-list-file rev-parse $oid || return 1\n+\tdone\n+'\n+\n+\n #########################################################################\n # HTTP tests begin here\n \n@@ -75,6 +141,33 @@ test_expect_success 'clone HTTP bundle' '\n \ttest_config -C clone-http log.excludedecoration refs/bundle/\n '\n \n+test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n+\tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = $HTTPD_URL/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = $HTTPD_URL/bundle-2.bundle\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = $HTTPD_URL/bundle-3.bundle\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = $HTTPD_URL/bundle-4.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" . clone-list-http &&\n+\tfor oid in $(git -C clone-from for-each-ref --format=\"%(objectname)\")\n+\tdo\n+\t\tgit -C clone-list-http rev-parse $oid || return 1\n+\tdone\n+'\n+\n # Do not add tests here unless they use the HTTP server, as they will\n # not run unless the HTTP dependencies exist.\n \n-- \ngitgitgadget\n"},{"id":"461781","messageId":"xmqqfshoduub.fsf@gitster.g","threadId":"58342","inReplyTo":"c39438886586f30b020e8ac473c7facaa757f822.1661181174.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/7] bundle-uri: create bundle_list struct and helpers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-22T17:57:48Z","receivedAt":"2022-08-22T17:57:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +/**\n> + * The remote_bundle_info struct contains information for a single bundle\n> + * URI. This may be initialized simply by a given URI or might have\n> + * additional metadata associated with it if the bundle was advertised by\n> + * a bundle list.\n> + */\n> +struct remote_bundle_info {\n> +\tstruct hashmap_entry ent;\n> +\n> +\t/**\n> +\t * The 'id' is a name given to the bundle for reference\n> +\t * by other bundle infos.\n> +\t */\n> +\tchar *id;\n> +\n> +\t/**\n> +\t * The 'uri' is the location of the remote bundle so\n> +\t * it can be downloaded on-demand. This will be NULL\n> +\t * if there was no table of contents.\n> +\t */\n> +\tchar *uri;\n> +\n> +\t/**\n> +\t * If the bundle has been downloaded, then 'file' is a\n> +\t * filename storing its contents. Otherwise, 'file' is\n> +\t * an empty string.\n> +\t */\n> +\tstruct strbuf file;\n> +};\n\nPresumably the sequence of events are that first a bundle list is\nobtained, with their .file member set to empty, then http worker(s)\ndownload and deposit the contents to files at which time the .file\nmember is set to the resulting file. The file downloader presumably\nuses the usual \"create a temporary file, download to it, and then\ncommit it by closing and then renaming\" dance, and the downloading\nhttp worker may want to have two strbufs somewhere it can access to\ncome up with the name of the temporary and the name of the final\nfile. But once the result becomes a committed file, its name will\nnot change, or will it?\n\nAt this step without the code that actually uses the data, use of\nstrbuf, instead of \"char *\" like id and uri members do, smells like\na premature optimization, and it is unclear if the optimization is\neven effective.\n\nOther than that, looks good to me.\n"},{"id":"461785","messageId":"xmqqzgfwcf79.fsf@gitster.g","threadId":"58342","inReplyTo":"7e4e4656e530395d055abac2a59e93866c9a0de2.1661181174.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/7] bundle-uri: create base key-value pair parsing","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-22T18:20:58Z","receivedAt":"2022-08-22T18:21:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index e376d547ce0..4280af6992e 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -387,6 +387,8 @@ include::config/branch.txt[]\n> \n> include::config/browser.txt[]\n> \n> +include::config/bundle.txt[]\n> +\n\nThe file that records a list of bundles may borrow the format of git\nconfig files, but will we store their contents in configuration\nfiles in the receiving (or originating) repository? With the\npresence of fields like \"bundle.version\", I somehow doubt it.\n\nShould \"git config --help\" list them?\n\n> diff --git a/Documentation/config/bundle.txt b/Documentation/config/bundle.txt\n> new file mode 100644\n> index 00000000000..3515bfe38d1\n> --- /dev/null\n> +++ b/Documentation/config/bundle.txt\n\nIf the answer is \"no\", then this file looks out of place.\n\n> diff --git a/bundle-uri.c b/bundle-uri.c\n> index ceeef0b6641..ade7eccce39 100644\n> --- a/bundle-uri.c\n> +++ b/bundle-uri.c\n> @@ -66,6 +66,80 @@ int for_all_bundles_in_list(struct bundle_list *list,\n> \treturn 0;\n> }\n> \n> +/**\n> + * Given a key-value pair, update the state of the given bundle list.\n> + * Returns 0 if the key-value pair is understood. Returns 1 if the key\n> + * is not understood or the value is malformed.\n\nLet's stick to the \"error is negative\" if we do not have a strong\nreason not to.\n\n> + */\n> +MAYBE_UNUSED\n> +static int bundle_list_update(const char *key, const char *value,\n> +\t\t\t struct bundle_list *list)\n> +{\n> +\tconst char *pkey, *dot;\n> +\tstruct strbuf id = STRBUF_INIT;\n> +\tstruct remote_bundle_info lookup = REMOTE_BUNDLE_INFO_INIT;\n> +\tstruct remote_bundle_info *bundle;\n> +\n> +\tif (!skip_prefix(key, \"bundle.\", &pkey))\n> +\t\treturn 1;\n> +\tdot = strchr(pkey, '.');\n> +\tif (!dot) {\n> +\t\tif (!strcmp(pkey, \"version\")) {\n> +\t\t\tint version = atoi(value);\n\nCan atoi() safely fail? Are we happy of pkey that says \"1A\" and we\nparse it as \"1\"?\n\n> +\t\t\tif (version != 1)\n> +\t\t\t\treturn 1;\n> +\n> +\t\t\tlist->version = version;\n> +\t\t\treturn 0;\n> +\t\t}\n\nIs it OK for a bundle list described in the config-file format to\nhave \"bundle.version\" twice, giving different values? It feels\ncounter-intuitive to apply the \"last one wins\" rule that is usual\nfor configuration files.\n\n> +\t\tif (!strcmp(pkey, \"mode\")) {\n> +\t\t\tif (!strcmp(value, \"all\"))\n> +\t\t\t\tlist->mode = BUNDLE_MODE_ALL;\n> +\t\t\telse if (!strcmp(value, \"any\"))\n> +\t\t\t\tlist->mode = BUNDLE_MODE_ANY;\n> +\t\t\telse\n> +\t\t\t\treturn 1;\n> +\t\t\treturn 0;\n> +\t\t}\n\nLikewise for bundle.mode\n\n> +\t\t/* Ignore other unknown global keys. */\n> +\t\treturn 0;\n> +\t}\n> +\n> +\tstrbuf_add(&id, pkey, dot - pkey);\n> +\tdot++;\n> +\n> +\t/*\n> +\t * Check for an existing bundle with this , or create one\n> +\t * if necessary.\n> +\t */\n> +\tlookup.id = id.buf;\n> +\thashmap_entry_init(&lookup.ent, strhash(lookup.id));\n> +\tif (!(bundle = hashmap_get_entry(&list->bundles, &lookup, ent, NULL))) {\n> +\t\tCALLOC_ARRAY(bundle, 1);\n> +\t\tbundle->id = strbuf_detach(&id, NULL);\n> +\t\tstrbuf_init(&bundle->file, 0);\n> +\t\thashmap_entry_init(&bundle->ent, strhash(bundle->id));\n> +\t\thashmap_add(&list->bundles, &bundle->ent);\n> +\t}\n> +\tstrbuf_release(&id);\n> +\n> +\tif (!strcmp(dot, \"uri\")) {\n> +\t\tfree(bundle->uri);\n> +\t\tbundle->uri = xstrdup(value);\n> +\t\treturn 0;\n> +\t}\n\nThis explicitly implements \"the last one wins\". Would it really\nmake sense for a server to serve a bundle list that says redundant\nand wasteful pieces of information, i.e.\n\n [bundle \"1\"]\n\turl = one\n\turl = two\n\nIt is not like doing so would allow us to reuse an otherwise mostly\ngood file by appending new information and that would be a performance\nor storage win. So I am not quite sure why we want \"the last one wins\"\nrule here. It instead looks like something we want to sanity check\nand complain about.\n\n> +\t/*\n> +\t * At this point, we ignore any information that we don't\n> +\t * understand, assuming it to be hints for a heuristic the client\n> +\t * does not currently understand.\n> +\t */\n\nThis is sensible.\n\n> +\treturn 0;\n> +}\n> +\n> static int find_temp_filename(struct strbuf *name)\n> {\n> \tint fd;\n"},{"id":"461793","messageId":"xmqqo7wccckm.fsf@gitster.g","threadId":"58342","inReplyTo":"49c4f88b6fd804f0bd5c62d523b45431846f4cee.1661181174.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/7] bundle-uri: create \"key=value\" line parsing","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-22T19:17:45Z","receivedAt":"2022-08-22T19:19:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Ævar Arnfjörð Bjarmason via GitGitGadget\" \nwrites:\n\n> +/**\n> + * General API for {transport,connect}.c etc.\n> + */\n> +int bundle_uri_parse_line(struct bundle_list *list, const char *line)\n> +{\n> +\tint result;\n> +\tconst char *equals;\n> +\tstruct strbuf key = STRBUF_INIT;\n> +\n> +\tif (!strlen(line))\n> +\t\treturn error(_(\"bundle-uri: got an empty line\"));\n> +\n> +\tequals = strchr(line, '=');\n> +\n> +\tif (!equals)\n> +\t\treturn error(_(\"bundle-uri: line is not of the form 'key=value'\"));\n> +\tif (line == equals || !*(equals + 1))\n> +\t\treturn error(_(\"bundle-uri: line has empty key or value\"));\n\nThe suggestions implied by my asking fall strictly into the \"it does\nnot have to exist here at this step and we can later extend it\", but\nfor something whose equivalent can be stored in our configuration\nfile, it is curious why we _insist_ to refuse an empty string as the\nvalue.\n\nI do not miss the \"key alone without even '=' means 'true'\"\nconvention, personally, so insisting to have '=' is OK, but the\ninability to have an empty string as a value looks a bit disturbing.\n\nThis depends on how the helper gets called, but most likely the\ncaller has a single line of pkt-line that it GAVE us to process, so\nit sounds a bit wasteful to insist that \"line\" to be const to us and\nforce us to use a separate strbuf, instead of just stuffing NUL at\nwhere we found '=' and pass the two halves to bundle_list_update().\n\nNot a huge deal, it is just something I found funny in the \"back in\nthe days we coded together, Linus would never have written like\nthis\" way.\n\nOther than that small detail, the code looks OK to me.\n\n> +\tstrbuf_add(&key, line, equals - line);\n> +\tresult = bundle_list_update(key.buf, equals + 1, list);\n> +\tstrbuf_release(&key);\n> +\n> +\treturn result;\n> +}\n> diff --git a/bundle-uri.h b/bundle-uri.h\n> index 6692aa4b170..f725c9796f7 100644\n> --- a/bundle-uri.h\n> +++ b/bundle-uri.h\n> @@ -76,4 +76,16 @@ int for_all_bundles_in_list(struct bundle_list *list,\n> */\n> int fetch_bundle_uri(struct repository *r, const char *uri);\n> \n> -#endif\n> +/**\n> + * General API for {transport,connect}.c etc.\n> + */\n> +\n> +/**\n> + * Parse a \"key=value\" packet line from the bundle-uri verb.\n> + *\n> + * Returns 0 on success and non-zero on error.\n> + */\n> +int bundle_uri_parse_line(struct bundle_list *list,\n> +\t\t\t const char *line);\n> +\n> +#endif /* BUNDLE_URI_H */\n"},{"id":"461794","messageId":"xmqqilmkcc7i.fsf@gitster.g","threadId":"58342","inReplyTo":"1d1bd9c710327b4d705cfede017771da7fb6ec52.1661181174.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/7] bundle-uri: parse bundle list in config format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-22T19:25:37Z","receivedAt":"2022-08-22T19:32:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> To allow for the incremental fetch case, teach Git to understand a\n> bundle list that could be advertised at an independent bundle URI. Such\n> a bundle list is likely to be inspected by human readers, even if only\n> by the bundle provider creating the list. For this reason, we can take\n> our expected \"key=value\" pairs and instead format them using Git config\n> format.\n\n\"can\" does not explain why it is a good idea. \"As a sequence of\nkey=value pairs is a lot more dense and harder to read than the\nconfiguration file format, let's declare that it is the format we\nuse in a file that holds a bundle-list\" would be.\n\nI do not personally buy it, though. As I hinted in an earlier step,\nsome trait we associate with our configuration fioe format, like the\n\"last one wins\" semantics, are undesirable ones, so even if we reuse\nthe appearance of the text, the semantics would have to become\ndifferent (including \"syntax errors lead to die()\" mentioned\nelsewhere in the proposed log message).\n\n> Update 'test-tool bundle-uri' to take this config file format as input.\n> It uses a filename instead of stdin because there is no existing way to\n> parse a FILE pointer in the config machinery. Using\n> git_config_from_mem() is overly complicated and more likely to introduce\n> bugs than this simpler version. I would rather have a slightly confusing\n> test helper than complicated product code.\n\nAll the troubles described above seem to come from the initial\nmistake to try reusing the configuration file parser or reusing the\nconfiguration file format, at least to me.\n\n"},{"id":"461861","messageId":"e5c1d197-fdee-e03d-42e0-a6aff37e595b@github.com","threadId":"58342","inReplyTo":"xmqqzgfwcf79.fsf@gitster.g","subject":"Re: [PATCH 2/7] bundle-uri: create base key-value pair parsing","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-23T16:29:04Z","receivedAt":"2022-08-23T18:15:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/22/2022 2:20 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> diff --git a/Documentation/config.txt b/Documentation/config.txt\n>> index e376d547ce0..4280af6992e 100644\n>> --- a/Documentation/config.txt\n>> +++ b/Documentation/config.txt\n>> @@ -387,6 +387,8 @@ include::config/branch.txt[]\n>> \n>> include::config/browser.txt[]\n>> \n>> +include::config/bundle.txt[]\n>> +\n> \n> The file that records a list of bundles may borrow the format of git\n> config files, but will we store their contents in configuration\n> files in the receiving (or originating) repository? With the\n> presence of fields like \"bundle.version\", I somehow doubt it.\n> \n> Should \"git config --help\" list them?\n\nI suppose that at this point, they should be left out, since\nwriting them to your Git config does nothing.\n\nIn the future, having these config values present will advertise\nthe bundle list during the 'bundle-uri' protocol v2 command. That\ncould use some clarification in the documentation, too, perhaps\nwith a \"bundle.*\" item discussing how all of the other items are\nrelated to that advertisement.\n\n>> +/**\n>> + * Given a key-value pair, update the state of the given bundle list.\n>> + * Returns 0 if the key-value pair is understood. Returns 1 if the key\n>> + * is not understood or the value is malformed.\n> \n> Let's stick to the \"error is negative\" if we do not have a strong\n> reason not to.\n\nRight. Can do.\n \n>> + */\n>> +MAYBE_UNUSED\n>> +static int bundle_list_update(const char *key, const char *value,\n>> +\t\t\t struct bundle_list *list)\n>> +{\n>> +\tconst char *pkey, *dot;\n>> +\tstruct strbuf id = STRBUF_INIT;\n>> +\tstruct remote_bundle_info lookup = REMOTE_BUNDLE_INFO_INIT;\n>> +\tstruct remote_bundle_info *bundle;\n>> +\n>> +\tif (!skip_prefix(key, \"bundle.\", &pkey))\n>> +\t\treturn 1;\n>> +\tdot = strchr(pkey, '.');\n>> +\tif (!dot) {\n>> +\t\tif (!strcmp(pkey, \"version\")) {\n>> +\t\t\tint version = atoi(value);\n> \n> Can atoi() safely fail? Are we happy of pkey that says \"1A\" and we\n> parse it as \"1\"?\n> \n>> +\t\t\tif (version != 1)\n>> +\t\t\t\treturn 1;\n>> +\n>> +\t\t\tlist->version = version;\n>> +\t\t\treturn 0;\n>> +\t\t}\n> \n> Is it OK for a bundle list described in the config-file format to\n> have \"bundle.version\" twice, giving different values? It feels\n> counter-intuitive to apply the \"last one wins\" rule that is usual\n> for configuration files.\n\n...\n> This explicitly implements \"the last one wins\". Would it really\n> make sense for a server to serve a bundle list that says redundant\n> and wasteful pieces of information, i.e.\n> \n> [bundle \"1\"]\n> \turl = one\n> \turl = two\n> \n> It is not like doing so would allow us to reuse an otherwise mostly\n> good file by appending new information and that would be a performance\n> or storage win. So I am not quite sure why we want \"the last one wins\"\n> rule here. It instead looks like something we want to sanity check\n> and complain about.\n\nI could switch this to \"expect at most one value\" and add warnings for\nduplicate keys. Should duplicate keys then mean \"the bundle list is\nmalformed, abort downloading bundles\"? That seems reasonable to me.\n\nThanks,\n-Stolee\n"},{"id":"461862","messageId":"c6731744-143a-5007-5186-f64852ad1123@github.com","threadId":"58342","inReplyTo":"xmqqo7wccckm.fsf@gitster.g","subject":"Re: [PATCH 3/7] bundle-uri: create \"key=value\" line parsing","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-23T16:31:19Z","receivedAt":"2022-08-23T18:16:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/22/2022 3:17 PM, Junio C Hamano wrote:\n> \"Ævar Arnfjörð Bjarmason via GitGitGadget\" \n> writes:\n> \n>> +/**\n>> + * General API for {transport,connect}.c etc.\n>> + */\n>> +int bundle_uri_parse_line(struct bundle_list *list, const char *line)\n>> +{\n>> +\tint result;\n>> +\tconst char *equals;\n>> +\tstruct strbuf key = STRBUF_INIT;\n>> +\n>> +\tif (!strlen(line))\n>> +\t\treturn error(_(\"bundle-uri: got an empty line\"));\n>> +\n>> +\tequals = strchr(line, '=');\n>> +\n>> +\tif (!equals)\n>> +\t\treturn error(_(\"bundle-uri: line is not of the form 'key=value'\"));\n>> +\tif (line == equals || !*(equals + 1))\n>> +\t\treturn error(_(\"bundle-uri: line has empty key or value\"));\n> \n> The suggestions implied by my asking fall strictly into the \"it does\n> not have to exist here at this step and we can later extend it\", but\n> for something whose equivalent can be stored in our configuration\n> file, it is curious why we _insist_ to refuse an empty string as the\n> value.\n> \n> I do not miss the \"key alone without even '=' means 'true'\"\n> convention, personally, so insisting to have '=' is OK, but the\n> inability to have an empty string as a value looks a bit disturbing.\n\nI'd be happy to switch this to allow an empty value.\n \n> This depends on how the helper gets called, but most likely the\n> caller has a single line of pkt-line that it GAVE us to process, so\n> it sounds a bit wasteful to insist that \"line\" to be const to us and\n> force us to use a separate strbuf, instead of just stuffing NUL at\n> where we found '=' and pass the two halves to bundle_list_update().\n\nI can look into using a non-const buffer.\n\nThanks,\n-Stolee\n\n"},{"id":"461863","messageId":"8d8e5256-8b87-7154-5865-675c2355b915@github.com","threadId":"58342","inReplyTo":"xmqqilmkcc7i.fsf@gitster.g","subject":"Re: [PATCH 5/7] bundle-uri: parse bundle list in config format","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-23T16:43:32Z","receivedAt":"2022-08-23T18:26:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/22/2022 3:25 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> To allow for the incremental fetch case, teach Git to understand a\n>> bundle list that could be advertised at an independent bundle URI. Such\n>> a bundle list is likely to be inspected by human readers, even if only\n>> by the bundle provider creating the list. For this reason, we can take\n>> our expected \"key=value\" pairs and instead format them using Git config\n>> format.\n> \n> \"can\" does not explain why it is a good idea. \"As a sequence of\n> key=value pairs is a lot more dense and harder to read than the\n> configuration file format, let's declare that it is the format we\n> use in a file that holds a bundle-list\" would be.\n\nThis \"more dense and harder to read\" was definitely my intention for\nwanting a different format. \n\n> I do not personally buy it, though. As I hinted in an earlier step,\n> some trait we associate with our configuration fioe format, like the\n> \"last one wins\" semantics, are undesirable ones, so even if we reuse\n> the appearance of the text, the semantics would have to become\n> different (including \"syntax errors lead to die()\" mentioned\n> elsewhere in the proposed log message).\n\nThe points you made earlier about \"last one wins\" semantics are the\nbiggest road-blocks to using the config file format, from what I've read\nso far. We could change those semantics to be different from my current\nimplementation which respects the \"last one wins\" rule, and then that\nmakes the config format match not as closely. That burden of avoiding\nmultiple key values is not on the end-user but the bundle provider to\nmatch the new expectations. (There might be something we should be careful\nabout when advertising the bundle list from our Git config in the\n'bundle-uri' command in the next series.)\n\nThe \"syntax errors lead to die()\" is mitigated by using\nCONFIG_ERROR_ERROR, which is what I meant by \"Be careful to call...\" I\nshould have been more clear that we are _not_ going to die() based on the\nremote data. We might write an error message and then abort the bundle\ndownload.\n\nWith all of these points in mind, I'd still prefer to use the config file\nformat as described in the design document. If you still don't agree, then\nI'll change the format to be key=value pairs split with newlines, and\nupdate the design document accordingly.\n\nThanks,\n-Stolee\n"},{"id":"462331","messageId":"kl6l5yi8cbs3.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"58342","inReplyTo":"7e4e4656e530395d055abac2a59e93866c9a0de2.1661181174.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/7] bundle-uri: create base key-value pair parsing","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-08-31T22:02:04Z","receivedAt":"2022-08-31T22:02:20Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +/**\n> + * Given a key-value pair, update the state of the given bundle list.\n> + * Returns 0 if the key-value pair is understood. Returns 1 if the key\n> + * is not understood or the value is malformed.\n> + */\n> +MAYBE_UNUSED\n> +static int bundle_list_update(const char *key, const char *value,\n> +\t\t\t struct bundle_list *list)\n> +{\n> +\tconst char *pkey, *dot;\n> +\tstruct strbuf id = STRBUF_INIT;\n> +\tstruct remote_bundle_info lookup = REMOTE_BUNDLE_INFO_INIT;\n> +\tstruct remote_bundle_info *bundle;\n> +\n> +\tif (!skip_prefix(key, \"bundle.\", &pkey))\n> +\t\treturn 1;\n> +\n> +\tdot = strchr(pkey, '.');\n> +\tif (!dot) {\n> +\t\tif (!strcmp(pkey, \"version\")) {\n> +\t\t\tint version = atoi(value);\n> +\t\t\tif (version != 1)\n> +\t\t\t\treturn 1;\n> +\n> +\t\t\tlist->version = version;\n> +\t\t\treturn 0;\n> +\t\t}\n> +\n> +\t\tif (!strcmp(pkey, \"mode\")) {\n> +\t\t\tif (!strcmp(value, \"all\"))\n> +\t\t\t\tlist->mode = BUNDLE_MODE_ALL;\n> +\t\t\telse if (!strcmp(value, \"any\"))\n> +\t\t\t\tlist->mode = BUNDLE_MODE_ANY;\n> +\t\t\telse\n> +\t\t\t\treturn 1;\n> +\t\t\treturn 0;\n> +\t\t}\n\nDrive-by comment from Review Club: we could simplify\n\"section.[subsection.]key\" parsing using parse_config_key(). There are\nother places in the code that do custom parsing like this, but maybe\nthey should use parse_config_key() too.\n\n> +\n> +\t\t/* Ignore other unknown global keys. */\n> +\t\treturn 0;\n> +\t}\n> +\n> +\tstrbuf_add(&id, pkey, dot - pkey);\n> +\tdot++;\n> +\n> +\t/*\n> +\t * Check for an existing bundle with this , or create one\n> +\t * if necessary.\n> +\t */\n> +\tlookup.id = id.buf;\n> +\thashmap_entry_init(&lookup.ent, strhash(lookup.id));\n> +\tif (!(bundle = hashmap_get_entry(&list->bundles, &lookup, ent, NULL))) {\n> +\t\tCALLOC_ARRAY(bundle, 1);\n> +\t\tbundle->id = strbuf_detach(&id, NULL);\n> +\t\tstrbuf_init(&bundle->file, 0);\n> +\t\thashmap_entry_init(&bundle->ent, strhash(bundle->id));\n> +\t\thashmap_add(&list->bundles, &bundle->ent);\n> +\t}\n> +\tstrbuf_release(&id);\n> +\n> +\tif (!strcmp(dot, \"uri\")) {\n> +\t\tfree(bundle->uri);\n> +\t\tbundle->uri = xstrdup(value);\n> +\t\treturn 0;\n> +\t}\n> +\n> +\t/*\n> +\t * At this point, we ignore any information that we don't\n> +\t * understand, assuming it to be hints for a heuristic the client\n> +\t * does not currently understand.\n> +\t */\n> +\treturn 0;\n> +}\n> +\n> static int find_temp_filename(struct strbuf *name)\n> {\n> \tint fd;\n> -- \n> gitgitgadget\n"},{"id":"462332","messageId":"20220831221003.373578-1-jonathantanmy@google.com","threadId":"58342","inReplyTo":"e5c1d197-fdee-e03d-42e0-a6aff37e595b@github.com","subject":"Re: [PATCH 2/7] bundle-uri: create base key-value pair parsing","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-08-31T22:10:02Z","receivedAt":"2022-08-31T22:10:14Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Derrick Stolee writes:\n> On 8/22/2022 2:20 PM, Junio C Hamano wrote:\n> > \"Derrick Stolee via GitGitGadget\" writes:\n> > \n> >> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> >> index e376d547ce0..4280af6992e 100644\n> >> --- a/Documentation/config.txt\n> >> +++ b/Documentation/config.txt\n> >> @@ -387,6 +387,8 @@ include::config/branch.txt[]\n> >> \n> >> include::config/browser.txt[]\n> >> \n> >> +include::config/bundle.txt[]\n> >> +\n> > \n> > The file that records a list of bundles may borrow the format of git\n> > config files, but will we store their contents in configuration\n> > files in the receiving (or originating) repository? With the\n> > presence of fields like \"bundle.version\", I somehow doubt it.\n> > \n> > Should \"git config --help\" list them?\n> \n> I suppose that at this point, they should be left out, since\n> writing them to your Git config does nothing.\n> \n> In the future, having these config values present will advertise\n> the bundle list during the 'bundle-uri' protocol v2 command. That\n> could use some clarification in the documentation, too, perhaps\n> with a \"bundle.*\" item discussing how all of the other items are\n> related to that advertisement.\n\nI think the main point of confusion is that these config variables\ncurrently do nothing when in a repo config, but they will be\nsubsequently used once we implement advertising them, and it is\nconvenient that these configs delegate to other files that have the same\nformat (and that we can specify, at the CLI, a file of the same format).\nMaybe documentation like this would clear up the confusion:\n\n bundle.*::\n \tThe `bundle.*` keys may appear in a repo's config, in a file\n \tlinked by bundle..uri, or in a file passed to \"clone\n \t--bundle-uri\".\n +\n NEEDSWORK: Currently, only the latter 2 situations work. `bundle.*` keys\n appearing in a repo's config will take effect once support for\n advertising bundles in fetch protocol v2 is implemented.\n +\n See link:technical/bundle-uri.html[the bundle URI design document] for\n more details.\n"},{"id":"462333","messageId":"20220831221822.374872-1-jonathantanmy@google.com","threadId":"58342","inReplyTo":"xmqqilmkcc7i.fsf@gitster.g","subject":"Re: [PATCH 5/7] bundle-uri: parse bundle list in config format","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-08-31T22:18:22Z","receivedAt":"2022-08-31T22:18:31Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Junio C Hamano writes:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n> > To allow for the incremental fetch case, teach Git to understand a\n> > bundle list that could be advertised at an independent bundle URI. Such\n> > a bundle list is likely to be inspected by human readers, even if only\n> > by the bundle provider creating the list. For this reason, we can take\n> > our expected \"key=value\" pairs and instead format them using Git config\n> > format.\n> \n> \"can\" does not explain why it is a good idea. \"As a sequence of\n> key=value pairs is a lot more dense and harder to read than the\n> configuration file format, let's declare that it is the format we\n> use in a file that holds a bundle-list\" would be.\n> \n> I do not personally buy it, though. As I hinted in an earlier step,\n> some trait we associate with our configuration fioe format, like the\n> \"last one wins\" semantics, are undesirable ones, so even if we reuse\n> the appearance of the text, the semantics would have to become\n> different (including \"syntax errors lead to die()\" mentioned\n> elsewhere in the proposed log message).\n\nOne reason for using the configuration file format (which perhaps could\nhave been better explained in the commit message) is that we plan to\nhave a way for a repo to advertise a list of bundles during fetch.\nI think that config is a natural place to put that, even with its \"last\none wins\" semantics.\n\nIt could be argued that we can just put a single URI in config and only\nallow advertising of a single URI (and then use a different format for\nthe bundle lists with semantics that are stricter than \"last one wins\"),\nbut that seems unnecessarily restrictive (and would make the client make\none more network request). And if we're advertising multiple bundles, it\nseems reasonable to make all bundle lists have the same format (whether\nthey are in config or in a separate file).\n"},{"id":"462409","messageId":"20220901023838.19263-1-tenglong.tl@alibaba-inc.com","threadId":"58342","inReplyTo":"7e4e4656e530395d055abac2a59e93866c9a0de2.1661181174.git.gitgitgadget@gmail.com","subject":"[PATCH 4/7] bundle-uri: unit test \"key=value\" parsing","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2022-09-01T02:38:38Z","receivedAt":"2022-09-01T02:38:54Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> +> +\tinit_bundle_list(&list);\n> +\twhile (strbuf_getline(&sb, stdin) != EOF) {\n> +\t\tif (bundle_uri_parse_line(&list, sb.buf) < 0)\n> +\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n> +\t}\n\nThe command to write such a test is useful for people who\nwant to experiment about the feature, Thanks. On top of that,\nI have a little question about the condition:\n\n if (bundle_uri_parse_line(&list, sb.buf) < 0)\n\n\"bundle_uri_parse_line\" will call \"bundle_list_update\" inside, and\ncould get the result of it as \"bundle_uri_parse_line\"'s return, then\nactually \"bundle_list_update\" could return \"1\", so I'm not sure but maybe\nthe line could modified to:\n\n if (bundle_uri_parse_line(&list, sb.buf))\n\nat here.\n\nThanks.\n"},{"id":"462410","messageId":"20220901025622.19901-1-tenglong.tl@alibaba-inc.com","threadId":"58342","inReplyTo":"7580e1f09aff2acdf7a7cb86bf8dc7e294cffd33.1661181174.git.gitgitgadget@gmail.com","subject":"[PATCH 4/7] bundle-uri: unit test \"key=value\" parsing","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2022-09-01T02:56:22Z","receivedAt":"2022-09-01T02:56:31Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Ævar Arnfjörð Bjarmason writes\n> +void print_bundle_list(FILE *fp, struct bundle_list *list)\n> +{\n> +\tconst char *mode;\n> +\n> +\tswitch (list->mode) {\n> +\tcase BUNDLE_MODE_ALL:\n> +\t\tmode = \"all\";\n> +\t\tbreak;\n> +\n> +\tcase BUNDLE_MODE_ANY:\n> +\t\tmode = \"any\";\n> +\t\tbreak;\n> +\n> +\tcase BUNDLE_MODE_NONE:\n> +\tdefault:\n> +\t\tmode = \"\";\n> +\t}\n> +\n> +\tprintf(\"[bundle]\\n\");\n> +\tprintf(\"\\tversion = %d\\n\", list->version);\n> +\tprintf(\"\\tmode = %s\\n\", mode);\n> +\n> +\tfor_all_bundles_in_list(list, summarize_bundle, fp);\n> +}\n\n\"print_bundle_list\" use to print the git config formatting lines of\n\"bundle_list\", it's supported to use a \"FILE *fp\" as it's output. The\n\"for_all_bundles_in_list\" use it, but other places seems not, I'm not\nsure, maybe we should change to:\n\n fprintf(fp, \"[bundle]\\n\");\n fprintf(fp, \"\\tversion = %d\\n\", list->version);\n fprintf(fp, \"\\tmode = %s\\n\", mode);\n\nhere?\n\nThanks.\n"},{"id":"462424","messageId":"20220901080509.35003-1-tenglong.tl@alibaba-inc.com","threadId":"58342","inReplyTo":"1d1bd9c710327b4d705cfede017771da7fb6ec52.1661181174.git.gitgitgadget@gmail.com","subject":"[PATCH 5/7] bundle-uri: parse bundle list in config format","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2022-09-01T08:05:09Z","receivedAt":"2022-09-01T08:05:56Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Derrick Stolee writes:\n\n> diff --git a/bundle-uri.h b/bundle-uri.h\n> index 41a1510a4ac..294ac804140 100644\n> --- a/bundle-uri.h\n> +++ b/bundle-uri.h\n> @@ -71,6 +71,16 @@ int for_all_bundles_in_list(struct bundle_list *list,\n> struct FILE;\n> void print_bundle_list(FILE *fp, struct bundle_list *list);\n>\n> +/**\n> + * A bundle URI may point to a bundle list where the key=value\n> + * pairs are provided in config file format. This method is\n> + * exposed publicly for testing purposes.\n> + */\n> +\n> +int parse_bundle_list_in_config_format(const char *uri,\n> +\t\t\t\t const char *filename,\n> +\t\t\t\t struct bundle_list *list);\n> +\n\nAlthough the comment clarifies the purpose of why to introduce\n\"parse_bundle_list_in_config_format\", but I think this API is useful if finally\nconfig format is supported. So far, we have a API names \"bundle_uri_parse_line\"\nwhich is used to parsing key-value pairs and package into bundle list, I think\nmaybe we should rename the API name from \"parse_bundle_list_in_config_format\" to\n\"bundle_uri_parse_config_format\", maybe better in my opinion for more consistent\nnaming. I think it doesnt break anything, feel free to accept or remain.\n\nThanks.\n"},{"id":"462562","messageId":"YxKUpyaejATs/3SQ@google.com","threadId":"58342","inReplyTo":"49c4f88b6fd804f0bd5c62d523b45431846f4cee.1661181174.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/7] bundle-uri: create \"key=value\" line parsing","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2022-09-02T23:41:27Z","receivedAt":"2022-09-02T23:41:39Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2022.08.22 15:12, Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n> From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n> \n> \n> When advertising a bundle list over Git's protocol v2, we will use\n> packet lines. Each line will be of the form \"key=value\" representing a\n> bundle list. Connect the API necessary for Git's transport to the\n> key-value pair parsing created in the previous change.\n\nSince we're not actually implementing advertisement via proto v2 in this\nseries, could we add an additional paragraph noting that this is useful\nnow for implementing the test helper in the next patch?\n"},{"id":"462564","messageId":"YxKXHKUe05C3m6ZZ@google.com","threadId":"58342","inReplyTo":"7b45c06cc9e0294311d9f00d40eb1fa4f8f146f9.1661181174.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 7/7] bundle-uri: fetch a list of bundles","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2022-09-02T23:51:56Z","receivedAt":"2022-09-02T23:52:07Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2022.08.22 15:12, Derrick Stolee via GitGitGadget wrote:\n[snip]\n> +static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data)\n> +{\n> +\tstruct bundle_list_context *ctx = data;\n> +\n> +\tif (ctx->mode == BUNDLE_MODE_ANY && ctx->count)\n> +\t\treturn 0;\n> +\n> +\tctx->count++;\n> +\treturn fetch_bundle_uri_internal(ctx->r, bundle, ctx->depth + 1, ctx->list);\n> +}\n\nWe should check whether fetch_bundle_uri_internal() actually succeeds\nbefore we increment ctx->count here. Otherwise, if we're in\nBUNDLE_MODE_ANY and the client gets unlucky that one of the servers\nhosting a bundle file is offline, it won't retry any of the other\nservers.\n"},{"id":"462630","messageId":"20220905125050.83312-1-tenglong.tl@alibaba-inc.com","threadId":"58342","inReplyTo":"7b45c06cc9e0294311d9f00d40eb1fa4f8f146f9.1661181174.git.gitgitgadget@gmail.com","subject":"[PATCH 7/7] bundle-uri: fetch a list of bundles","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2022-09-05T12:50:50Z","receivedAt":"2022-09-05T12:51:10Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"\nDerrick Stolee writes:\n\n> int fetch_bundle_uri(struct repository *r, const char *uri)\n> {\n> -\treturn fetch_bundle_uri_internal(r, uri, 0);\n> +\tint result;\n> +\tstruct bundle_list list;\n> +\tstruct remote_bundle_info bundle = {\n> +\t\t.uri = xstrdup(uri),\n> +\t\t.id = xstrdup(\"\"),\n\nVery readable code, thank you very much.\n\nI'm a little curious why we use the \"\" as the init value of\n\".id\"?\n\nThanks.\n"},{"id":"462815","messageId":"c6e4fd5f-b85e-2108-3212-21604e6db986@github.com","threadId":"58342","inReplyTo":"20220905125050.83312-1-tenglong.tl@alibaba-inc.com","subject":"Re: [PATCH 7/7] bundle-uri: fetch a list of bundles","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-08T17:10:36Z","receivedAt":"2022-09-08T17:10:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/5/2022 8:50 AM, Teng Long wrote:\n> \n> Derrick Stolee writes:\n> \n>> int fetch_bundle_uri(struct repository *r, const char *uri)\n>> {\n>> -\treturn fetch_bundle_uri_internal(r, uri, 0);\n>> +\tint result;\n>> +\tstruct bundle_list list;\n>> +\tstruct remote_bundle_info bundle = {\n>> +\t\t.uri = xstrdup(uri),\n>> +\t\t.id = xstrdup(\"\"),\n> \n> Very readable code, thank you very much.\n> \n> I'm a little curious why we use the \"\" as the init value of\n> \".id\"?\n\nIn this case, we need a valid ID to initialize the bundle list (since\nit will add the remote_bundle_info to the hash set), but it is\nconsidered the info \"above\" all other lists. We could also specify an\nempty string here, but we just can't use NULL.\n\nThanks,\n-Stolee\n"},{"id":"462852","messageId":"2ca431e6c377f42d70ea9ce0ac74058fc983ea2c.1662734015.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v2.git.1662734015.gitgitgadget@gmail.com","subject":"[PATCH v2 1/9] bundle-uri: short-circuit capability parsing","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-09T14:33:27Z","receivedAt":"2022-09-09T14:33:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen parsing the capability lines from the 'git remote-https' process,\nwe can stop reading the lines once we notice the 'get' capability.\n\nReported-by: Teng Long \nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 4a8cc74ed05..7173ed065e9 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -56,8 +56,10 @@ static int download_https_uri_to_file(const char *file, const char *uri)\n \twhile (!strbuf_getline(&line, child_out)) {\n \t\tif (!line.len)\n \t\t\tbreak;\n-\t\tif (!strcmp(line.buf, \"get\"))\n+\t\tif (!strcmp(line.buf, \"get\")) {\n \t\t\tfound_get = 1;\n+\t\t\tbreak;\n+\t\t}\n \t}\n \tstrbuf_release(&line);\n \n-- \ngitgitgadget\n\n"},{"id":"462853","messageId":"pull.1333.v2.git.1662734015.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.git.1661181174.gitgitgadget@gmail.com","subject":"[PATCH v2 0/9] Bundle URIs III: Parse and download from bundle lists","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-09T14:33:26Z","receivedAt":"2022-09-09T14:33:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the third series building the bundle URI feature. It is built on top\nof ds/bundle-uri-clone, which introduced 'git clone --bundle-uri=' where is\na URI to a bundle file. This series adds the capability of downloading and\nparsing a bundle list and then downloading the URIs in that list.\n\nThe core functionality of bundle lists is implemented by creating data\nstructures from a list of key-value pairs. These pairs can come from a\nplain-text file in Git config format, but in the future, we will support the\nlist being supplied by packet lines over Git's protocol v2 in the\n'bundle-uri' command (reserved for the next series).\n\nThe patches are organized in this way:\n\n 1. Patches 1-2 are cleanups from the previous part. The first was\n recommended by Teng Long and the second allows us to simplify our bundle\n list data structure slightly.\n\n 2. Patches 3-4 create the bundle list data structures and the logic for\n populating the list from key-value pairs.\n\n 3. Patches 5-6 teach Git to parse \"key=value\" lines to construct a bundle\n list. Add unit tests that ensure this logic constructs lists correctly.\n These patches are adapted from Ævar's RFC [1] and were previously seen\n in my combined RFC [2].\n\n 4. Patch 7 teaches Git to parse Git config files into bundle lists.\n\n 5. Patches 8-9 implement the ability to download a bundle list and\n recursively download the contained bundles (and possibly the bundle\n lists within). This is limited by a constant depth to avoid issues with\n cycles or otherwise incorrectly configured bundle lists.\n\n[1]\nhttps://lore.kernel.org/git/RFC-cover-v2-00.36-00000000000-20220418T165545Z-avarab@gmail.com/\n\n[2]\nhttps://lore.kernel.org/git/pull.1234.git.1653072042.gitgitgadget@gmail.com/\n\nAt the end of this series, users can bootstrap clones using 'git clone\n--bundle-uri= ' where points to a bundle list instead of a single bundle\nfile.\n\nAs outlined in the design document [1], the next steps after this are:\n\n 1. Implement the protocol v2 verb, re-using the bundle list logic from (2).\n Use this to auto-discover bundle URIs during 'git clone' (behind a\n config option). [2]\n 2. Implement the 'creationToken' heuristic, allowing incremental 'git\n fetch' commands to download a bundle list from a configured URI, and\n only download bundles that are new based on the creation token values.\n [3]\n\nI have prepared some of this work as pull requests on my personal fork so\ncurious readers can look ahead to where we are going:\n\n[3]\nhttps://lore.kernel.org/git/pull.1248.v3.git.1658757188.gitgitgadget@gmail.com\n\n[4] https://github.com/derrickstolee/git/pull/21\n\n[5] https://github.com/derrickstolee/git/pull/22\n\n\nUpdates in v2\n=============\n\nThank you to all of the voices who chimed in on the previous version. I'm\nsorry it took so long for me to get a new version.\n\n * I've done a rather thorough overhaul to minimize how often later patches\n rewrite portions of earlier patches.\n\n * We no longer use a strbuf in struct remote_bundle_info. Instead, use a\n 'char *' and only in the patch where it is first used.\n\n * The config documentation is more clearly indicating that the bundle.*\n section has no effect in the repository config (at the moment, which will\n change in the next series).\n\n * The bundle.version value is now parsed using git_parse_int().\n\n * The config key is now parsed using parse_config_key().\n\n * Commit messages clarify more about the context of the change in the\n bigger picture of the bundle URI effort.\n\n * Some printf()s are correctly changed to fprintf()s.\n\n * The test helper CLI is unified across the two modes. They both take a\n filename now.\n\n * The count of downloaded bundles is now only updated after a successful\n download, allowing the \"any\" mode to keep trying after a failure.\n\nThanks,\n\n * Stolee\n\nDerrick Stolee (7):\n bundle-uri: short-circuit capability parsing\n bundle-uri: use plain string in find_temp_filename()\n bundle-uri: create bundle_list struct and helpers\n bundle-uri: create base key-value pair parsing\n bundle-uri: parse bundle list in config format\n bundle-uri: limit recursion depth for bundle lists\n bundle-uri: fetch a list of bundles\n\nÆvar Arnfjörð Bjarmason (2):\n bundle-uri: create \"key=value\" line parsing\n bundle-uri: unit test \"key=value\" parsing\n\n Documentation/config.txt | 2 +\n Documentation/config/bundle.txt | 24 ++\n Makefile | 1 +\n bundle-uri.c | 466 ++++++++++++++++++++++++++++++--\n bundle-uri.h | 93 +++++++\n config.c | 2 +-\n config.h | 1 +\n t/helper/test-bundle-uri.c | 95 +++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5558-clone-bundle-uri.sh | 93 +++++++\n t/t5750-bundle-uri-parse.sh | 171 ++++++++++++\n t/test-lib-functions.sh | 11 +\n 13 files changed, 942 insertions(+), 19 deletions(-)\n create mode 100644 Documentation/config/bundle.txt\n create mode 100644 t/helper/test-bundle-uri.c\n create mode 100755 t/t5750-bundle-uri-parse.sh\n\n\nbase-commit: e21e663cd1942df29979d3e01f7eacb532727bb7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1333%2Fderrickstolee%2Fbundle-redo%2Flist-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1333/derrickstolee/bundle-redo/list-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1333\n\nRange-diff vs v1:\n\n -: ----------- > 1: 2ca431e6c37 bundle-uri: short-circuit capability parsing\n -: ----------- > 2: ee6c4b824c2 bundle-uri: use plain string in find_temp_filename()\n 1: c3943888658 ! 3: d9812440594 bundle-uri: create bundle_list struct and helpers\n @@ bundle-uri.c\n +static int clear_remote_bundle_info(struct remote_bundle_info *bundle,\n +\t\t\t\t void *data)\n +{\n -+\tfree(bundle->id);\n -+\tfree(bundle->uri);\n -+\tstrbuf_release(&bundle->file);\n ++\tFREE_AND_NULL(bundle->id);\n ++\tFREE_AND_NULL(bundle->uri);\n +\treturn 0;\n +}\n +\n @@ bundle-uri.c\n +\treturn 0;\n +}\n \n - static int find_temp_filename(struct strbuf *name)\n + static char *find_temp_filename(void)\n {\n \n ## bundle-uri.h ##\n @@ bundle-uri.h\n +\t * if there was no table of contents.\n +\t */\n +\tchar *uri;\n -+\n -+\t/**\n -+\t * If the bundle has been downloaded, then 'file' is a\n -+\t * filename storing its contents. Otherwise, 'file' is\n -+\t * an empty string.\n -+\t */\n -+\tstruct strbuf file;\n +};\n +\n -+#define REMOTE_BUNDLE_INFO_INIT { \\\n -+\t.file = STRBUF_INIT, \\\n -+}\n ++#define REMOTE_BUNDLE_INFO_INIT { 0 }\n +\n +enum bundle_list_mode {\n +\tBUNDLE_MODE_NONE = 0,\n 2: 7e4e4656e53 ! 4: 70daef66833 bundle-uri: create base key-value pair parsing\n @@ Commit message\n currently is expected to be an absolute URI, but will be relaxed to be\n a relative URI in the future.\n \n + While parsing, return an error if a URI key is repeated, since we can\n + make that restriction with bundle lists.\n +\n + Make the git_parse_int() method global so we can parse the integer\n + version value carefully.\n +\n Signed-off-by: Derrick Stolee \n \n ## Documentation/config.txt ##\n @@ Documentation/config.txt: include::config/branch.txt[]\n ## Documentation/config/bundle.txt (new) ##\n @@\n +bundle.*::\n -+\tThe `bundle.*` keys are used when communicating a list of bundle URIs\n -+\tSee link:technical/bundle-uri.html[the bundle URI design document] for\n -+\tmore details.\n ++\tThe `bundle.*` keys may appear in a bundle list file found via the\n ++\t`git clone --bundle-uri` option. These keys currently have no effect\n ++\tif placed in a repository config file, though this will change in the\n ++\tfuture. See link:technical/bundle-uri.html[the bundle URI design\n ++\tdocument] for more details.\n +\n +bundle.version::\n +\tThis integer value advertises the version of the bundle list format\n @@ Documentation/config/bundle.txt (new)\n +\tof this ``. This URI may be a bundle file or another bundle list.\n \n ## bundle-uri.c ##\n +@@\n + #include \"run-command.h\"\n + #include \"hashmap.h\"\n + #include \"pkt-line.h\"\n ++#include \"config.h\"\n + \n + static int compare_bundles(const void *hashmap_cmp_fn_data,\n + \t\t\t const struct hashmap_entry *he1,\n @@ bundle-uri.c: int for_all_bundles_in_list(struct bundle_list *list,\n \treturn 0;\n }\n @@ bundle-uri.c: int for_all_bundles_in_list(struct bundle_list *list,\n +static int bundle_list_update(const char *key, const char *value,\n +\t\t\t struct bundle_list *list)\n +{\n -+\tconst char *pkey, *dot;\n +\tstruct strbuf id = STRBUF_INIT;\n +\tstruct remote_bundle_info lookup = REMOTE_BUNDLE_INFO_INIT;\n +\tstruct remote_bundle_info *bundle;\n ++\tconst char *subsection, *subkey;\n ++\tsize_t subsection_len;\n +\n -+\tif (!skip_prefix(key, \"bundle.\", &pkey))\n -+\t\treturn 1;\n ++\tif (parse_config_key(key, \"bundle\", &subsection, &subsection_len, &subkey))\n ++\t\treturn -1;\n +\n -+\tdot = strchr(pkey, '.');\n -+\tif (!dot) {\n -+\t\tif (!strcmp(pkey, \"version\")) {\n -+\t\t\tint version = atoi(value);\n ++\tif (!subsection_len) {\n ++\t\tif (!strcmp(subkey, \"version\")) {\n ++\t\t\tint version;\n ++\t\t\tif (!git_parse_int(value, &version))\n ++\t\t\t\treturn -1;\n +\t\t\tif (version != 1)\n -+\t\t\t\treturn 1;\n ++\t\t\t\treturn -1;\n +\n +\t\t\tlist->version = version;\n +\t\t\treturn 0;\n +\t\t}\n +\n -+\t\tif (!strcmp(pkey, \"mode\")) {\n ++\t\tif (!strcmp(subkey, \"mode\")) {\n +\t\t\tif (!strcmp(value, \"all\"))\n +\t\t\t\tlist->mode = BUNDLE_MODE_ALL;\n +\t\t\telse if (!strcmp(value, \"any\"))\n +\t\t\t\tlist->mode = BUNDLE_MODE_ANY;\n +\t\t\telse\n -+\t\t\t\treturn 1;\n ++\t\t\t\treturn -1;\n +\t\t\treturn 0;\n +\t\t}\n +\n @@ bundle-uri.c: int for_all_bundles_in_list(struct bundle_list *list,\n +\t\treturn 0;\n +\t}\n +\n -+\tstrbuf_add(&id, pkey, dot - pkey);\n -+\tdot++;\n ++\tstrbuf_add(&id, subsection, subsection_len);\n +\n +\t/*\n +\t * Check for an existing bundle with this , or create one\n @@ bundle-uri.c: int for_all_bundles_in_list(struct bundle_list *list,\n +\tif (!(bundle = hashmap_get_entry(&list->bundles, &lookup, ent, NULL))) {\n +\t\tCALLOC_ARRAY(bundle, 1);\n +\t\tbundle->id = strbuf_detach(&id, NULL);\n -+\t\tstrbuf_init(&bundle->file, 0);\n +\t\thashmap_entry_init(&bundle->ent, strhash(bundle->id));\n +\t\thashmap_add(&list->bundles, &bundle->ent);\n +\t}\n +\tstrbuf_release(&id);\n +\n -+\tif (!strcmp(dot, \"uri\")) {\n -+\t\tfree(bundle->uri);\n ++\tif (!strcmp(subkey, \"uri\")) {\n ++\t\tif (bundle->uri)\n ++\t\t\treturn -1;\n +\t\tbundle->uri = xstrdup(value);\n +\t\treturn 0;\n +\t}\n @@ bundle-uri.c: int for_all_bundles_in_list(struct bundle_list *list,\n +\treturn 0;\n +}\n +\n - static int find_temp_filename(struct strbuf *name)\n + static char *find_temp_filename(void)\n {\n \tint fd;\n +\n + ## config.c ##\n +@@ config.c: static int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max)\n + \treturn 0;\n + }\n + \n +-static int git_parse_int(const char *value, int *ret)\n ++int git_parse_int(const char *value, int *ret)\n + {\n + \tintmax_t tmp;\n + \tif (!git_parse_signed(value, &tmp, maximum_signed_value_of_type(int)))\n +\n + ## config.h ##\n +@@ config.h: int config_with_options(config_fn_t fn, void *,\n + \n + int git_parse_ssize_t(const char *, ssize_t *);\n + int git_parse_ulong(const char *, unsigned long *);\n ++int git_parse_int(const char *value, int *ret);\n + \n + /**\n + * Same as `git_config_bool`, except that it returns -1 on error rather\n 3: 49c4f88b6fd ! 5: 4df3f834029 bundle-uri: create \"key=value\" line parsing\n @@ Commit message\n bundle list. Connect the API necessary for Git's transport to the\n key-value pair parsing created in the previous change.\n \n + We are not currently implementing this protocol v2 functionality, but\n + instead preparing to expose this parsing to be unit-testable.\n +\n Co-authored-by: Derrick Stolee \n Signed-off-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: Derrick Stolee \n @@ bundle-uri.c: int for_all_bundles_in_list(struct bundle_list *list,\n \t\t\t struct bundle_list *list)\n {\n @@ bundle-uri.c: cleanup:\n - \tstrbuf_release(&filename);\n + \tfree(filename);\n \treturn result;\n }\n +\n @@ bundle-uri.h: int for_all_bundles_in_list(struct bundle_list *list,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n --#endif\n +/**\n + * General API for {transport,connect}.c etc.\n + */\n @@ bundle-uri.h: int for_all_bundles_in_list(struct bundle_list *list,\n +int bundle_uri_parse_line(struct bundle_list *list,\n +\t\t\t const char *line);\n +\n -+#endif /* BUNDLE_URI_H */\n + #endif\n 4: 7580e1f09af ! 6: 91c5b58f011 bundle-uri: unit test \"key=value\" parsing\n @@ Commit message\n in testing logic deep in the bundle URI feature.\n \n This change introduces the 'parse-key-values' subcommand, which parses\n - stdin as a list of lines. These are fed into bundle_uri_parse_line() to\n - test how we construct a 'struct bundle_list' from that data. The list is\n - then output to stdout as if the key-value pairs were a Git config file.\n + an input file as a list of lines. These are fed into\n + bundle_uri_parse_line() to test how we construct a 'struct bundle_list'\n + from that data. The list is then output to stdout as if the key-value\n + pairs were a Git config file.\n +\n + We use an input file instead of stdin because of a future change to\n + parse in config-file format that works better as an input file.\n \n Co-authored-by: Derrick Stolee \n Signed-off-by: Ævar Arnfjörð Bjarmason \n @@ bundle-uri.c: int for_all_bundles_in_list(struct bundle_list *list,\n +\t\tmode = \"\";\n +\t}\n +\n -+\tprintf(\"[bundle]\\n\");\n -+\tprintf(\"\\tversion = %d\\n\", list->version);\n -+\tprintf(\"\\tmode = %s\\n\", mode);\n ++\tfprintf(fp, \"[bundle]\\n\");\n ++\tfprintf(fp, \"\\tversion = %d\\n\", list->version);\n ++\tfprintf(fp, \"\\tmode = %s\\n\", mode);\n +\n +\tfor_all_bundles_in_list(list, summarize_bundle, fp);\n +}\n @@ t/helper/test-bundle-uri.c (new)\n +#include \"strbuf.h\"\n +#include \"string-list.h\"\n +\n -+static int cmd__bundle_uri_parse_key_values(int argc, const char **argv)\n ++static int cmd__bundle_uri_parse(int argc, const char **argv)\n +{\n -+\tconst char *usage[] = {\n -+\t\t\"test-tool bundle-uri parse-key-values \",\n +\t\tNULL\n +\t};\n ++\tconst char **usage = key_value_usage;\n +\tstruct option options[] = {\n +\t\tOPT_END(),\n +\t};\n +\tstruct strbuf sb = STRBUF_INIT;\n +\tstruct bundle_list list;\n +\tint err = 0;\n ++\tFILE *fp;\n +\n +\targc = parse_options(argc, argv, NULL, options, usage, 0);\n -+\tif (argc)\n ++\tif (argc != 1)\n +\t\tgoto usage;\n +\n +\tinit_bundle_list(&list);\n -+\twhile (strbuf_getline(&sb, stdin) != EOF) {\n -+\t\tif (bundle_uri_parse_line(&list, sb.buf) < 0)\n ++\tfp = fopen(argv[0], \"r\");\n ++\tif (!fp)\n ++\t\tdie(\"failed to open '%s'\", argv[0]);\n ++\n ++\twhile (strbuf_getline(&sb, fp) != EOF) {\n ++\t\tif (bundle_uri_parse_line(&list, sb.buf))\n +\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n +\t}\n +\tstrbuf_release(&sb);\n ++\tfclose(fp);\n +\n +\tprint_bundle_list(stdout, &list);\n +\n @@ t/helper/test-bundle-uri.c (new)\n +\t\tgoto usage;\n +\n +\tif (!strcmp(argv[1], \"parse-key-values\"))\n -+\t\treturn cmd__bundle_uri_parse_key_values(argc - 1, argv + 1);\n ++\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1);\n +\terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n +\n +usage:\n @@ t/t5750-bundle-uri-parse.sh (new)\n +\t\turi = file:///usr/share/git/bundle.bdl\n +\tEOF\n +\n -+\ttest-tool bundle-uri parse-key-values actual 2>err &&\n ++\ttest-tool bundle-uri parse-key-values in >actual 2>err &&\n +\ttest_must_be_empty err &&\n +\ttest_cmp_config_output expect actual\n +'\n @@ t/t5750-bundle-uri-parse.sh (new)\n +\t\tmode = all\n +\tEOF\n +\n -+\ttest_must_fail test-tool bundle-uri parse-key-values actual 2>err &&\n ++\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n +\ttest_cmp err.expect err &&\n +\ttest_cmp_config_output expect actual\n +'\n @@ t/t5750-bundle-uri-parse.sh (new)\n +\t\turi = file:///usr/share/git/bundle.bdl\n +\tEOF\n +\n -+\ttest_must_fail test-tool bundle-uri parse-key-values actual 2>err &&\n ++\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n ++\ttest_cmp err.expect err &&\n ++\ttest_cmp_config_output expect actual\n ++'\n ++\n ++test_expect_success 'bundle_uri_parse_line() parsing edge cases: duplicate lines' '\n ++\tcat >in <<-\\EOF &&\n ++\tbundle.one.uri=http://example.com/bundle.bdl\n ++\tbundle.two.uri=https://example.com/bundle.bdl\n ++\tbundle.one.uri=https://example.com/bundle-2.bdl\n ++\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n ++\tEOF\n ++\n ++\tcat >err.expect <<-\\EOF &&\n ++\terror: bad line: '\\''bundle.one.uri=https://example.com/bundle-2.bdl'\\''\n ++\tEOF\n ++\n ++\t# We fail, but try to continue parsing regardless\n ++\tcat >expect <<-\\EOF &&\n ++\t[bundle]\n ++\t\tversion = 1\n ++\t\tmode = all\n ++\t[bundle \"one\"]\n ++\t\turi = http://example.com/bundle.bdl\n ++\t[bundle \"two\"]\n ++\t\turi = https://example.com/bundle.bdl\n ++\t[bundle \"three\"]\n ++\t\turi = file:///usr/share/git/bundle.bdl\n ++\tEOF\n ++\n ++\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n +\ttest_cmp err.expect err &&\n +\ttest_cmp_config_output expect actual\n +'\n 5: 1d1bd9c7103 ! 7: 1492b8f5ef0 bundle-uri: parse bundle list in config format\n @@ Commit message\n our expected \"key=value\" pairs and instead format them using Git config\n format.\n \n - Create parse_bundle_list_in_config_format() to parse a file in config\n - format and convert that into a 'struct bundle_list' filled with its\n + Create bundle_uri_parse_config_format() to parse a file in config format\n + and convert that into a 'struct bundle_list' filled with its\n understanding of the contents.\n \n - Be careful to call git_config_from_file_with_options() because the\n - default action for git_config_from_file() is to die() on a parsing\n - error. The current warning isn't particularly helpful if it arises to a\n - user, but it will be made more verbose at a higher layer later.\n + Be careful to use error_action CONFIG_ERROR_ERROR when calling\n + git_config_from_file_with_options() because the default action for\n + git_config_from_file() is to die() on a parsing error. The current\n + warning isn't particularly helpful if it arises to a user, but it will\n + be made more verbose at a higher layer later.\n \n Update 'test-tool bundle-uri' to take this config file format as input.\n It uses a filename instead of stdin because there is no existing way to\n parse a FILE pointer in the config machinery. Using\n git_config_from_mem() is overly complicated and more likely to introduce\n - bugs than this simpler version. I would rather have a slightly confusing\n - test helper than complicated product code.\n + bugs than this simpler version.\n \n Signed-off-by: Derrick Stolee \n \n ## bundle-uri.c ##\n -@@\n - #include \"run-command.h\"\n - #include \"hashmap.h\"\n - #include \"pkt-line.h\"\n -+#include \"config.h\"\n - \n - static int compare_bundles(const void *hashmap_cmp_fn_data,\n - \t\t\t const struct hashmap_entry *he1,\n @@ bundle-uri.c: static int bundle_list_update(const char *key, const char *value,\n \treturn 0;\n }\n @@ bundle-uri.c: static int bundle_list_update(const char *key, const char *value,\n +\treturn bundle_list_update(key, value, list);\n +}\n +\n -+int parse_bundle_list_in_config_format(const char *uri,\n -+\t\t\t\t const char *filename,\n -+\t\t\t\t struct bundle_list *list)\n ++int bundle_uri_parse_config_format(const char *uri,\n ++\t\t\t\t const char *filename,\n ++\t\t\t\t struct bundle_list *list)\n +{\n +\tint result;\n +\tstruct config_options opts = {\n +\t\t.error_action = CONFIG_ERROR_ERROR,\n +\t};\n +\n -+\tlist->mode = BUNDLE_MODE_NONE;\n +\tresult = git_config_from_file_with_options(config_to_bundle_list,\n +\t\t\t\t\t\t filename, list,\n +\t\t\t\t\t\t &opts);\n @@ bundle-uri.c: static int bundle_list_update(const char *key, const char *value,\n +\treturn result;\n +}\n +\n - static int find_temp_filename(struct strbuf *name)\n + static char *find_temp_filename(void)\n {\n \tint fd;\n \n @@ bundle-uri.h: int for_all_bundles_in_list(struct bundle_list *list,\n + * pairs are provided in config file format. This method is\n + * exposed publicly for testing purposes.\n + */\n -+\n -+int parse_bundle_list_in_config_format(const char *uri,\n -+\t\t\t\t const char *filename,\n -+\t\t\t\t struct bundle_list *list);\n ++int bundle_uri_parse_config_format(const char *uri,\n ++\t\t\t\t const char *filename,\n ++\t\t\t\t struct bundle_list *list);\n +\n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n @@ t/helper/test-bundle-uri.c\n #include \"strbuf.h\"\n #include \"string-list.h\"\n \n --static int cmd__bundle_uri_parse_key_values(int argc, const char **argv)\n +-static int cmd__bundle_uri_parse(int argc, const char **argv)\n +enum input_mode {\n +\tKEY_VALUE_PAIRS,\n +\tCONFIG_FILE,\n @@ t/helper/test-bundle-uri.c\n +\n +static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mode)\n {\n --\tconst char *usage[] = {\n -+\tconst char *key_value_usage[] = {\n - \t\t\"test-tool bundle-uri parse-key-values \",\n \t\tNULL\n \t};\n +\tconst char *config_usage[] = {\n +\t\t\"test-tool bundle-uri parse-config \",\n +\t\tNULL\n +\t};\n + \tconst char **usage = key_value_usage;\n \tstruct option options[] = {\n \t\tOPT_END(),\n - \t};\n -+\tconst char **usage = key_value_usage;\n - \tstruct strbuf sb = STRBUF_INIT;\n - \tstruct bundle_list list;\n +@@ t/helper/test-bundle-uri.c: static int cmd__bundle_uri_parse(int argc, const char **argv)\n \tint err = 0;\n + \tFILE *fp;\n \n -\targc = parse_options(argc, argv, NULL, options, usage, 0);\n --\tif (argc)\n +-\tif (argc != 1)\n -\t\tgoto usage;\n +\tif (mode == CONFIG_FILE)\n +\t\tusage = config_usage;\n @@ t/helper/test-bundle-uri.c\n +\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tinit_bundle_list(&list);\n --\twhile (strbuf_getline(&sb, stdin) != EOF) {\n --\t\tif (bundle_uri_parse_line(&list, sb.buf) < 0)\n +-\tfp = fopen(argv[0], \"r\");\n +-\tif (!fp)\n +-\t\tdie(\"failed to open '%s'\", argv[0]);\n + \n +-\twhile (strbuf_getline(&sb, fp) != EOF) {\n +-\t\tif (bundle_uri_parse_line(&list, sb.buf))\n -\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n -+\n +\tswitch (mode) {\n +\tcase KEY_VALUE_PAIRS:\n -+\t\tif (argc)\n ++\t\tif (argc != 1)\n +\t\t\tgoto usage;\n -+\t\twhile (strbuf_getline(&sb, stdin) != EOF) {\n -+\t\t\tif (bundle_uri_parse_line(&list, sb.buf) < 0)\n ++\t\tfp = fopen(argv[0], \"r\");\n ++\t\tif (!fp)\n ++\t\t\tdie(\"failed to open '%s'\", argv[0]);\n ++\t\twhile (strbuf_getline(&sb, fp) != EOF) {\n ++\t\t\tif (bundle_uri_parse_line(&list, sb.buf))\n +\t\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n +\t\t}\n ++\t\tfclose(fp);\n +\t\tbreak;\n +\n +\tcase CONFIG_FILE:\n +\t\tif (argc != 1)\n +\t\t\tgoto usage;\n -+\t\terr = parse_bundle_list_in_config_format(\"\", argv[0], &list);\n ++\t\terr = bundle_uri_parse_config_format(\"\", argv[0], &list);\n +\t\tbreak;\n \t}\n \tstrbuf_release(&sb);\n +-\tfclose(fp);\n + \n + \tprint_bundle_list(stdout, &list);\n \n @@ t/helper/test-bundle-uri.c: int cmd__bundle_uri(int argc, const char **argv)\n \t\tgoto usage;\n \n \tif (!strcmp(argv[1], \"parse-key-values\"))\n --\t\treturn cmd__bundle_uri_parse_key_values(argc - 1, argv + 1);\n +-\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1);\n +\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS);\n +\tif (!strcmp(argv[1], \"parse-config\"))\n +\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE);\n @@ t/helper/test-bundle-uri.c: int cmd__bundle_uri(int argc, const char **argv)\n usage:\n \n ## t/t5750-bundle-uri-parse.sh ##\n -@@ t/t5750-bundle-uri-parse.sh: test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty lines' '\n +@@ t/t5750-bundle-uri-parse.sh: test_expect_success 'bundle_uri_parse_line() parsing edge cases: duplicate lines\n \ttest_cmp_config_output expect actual\n '\n \n @@ t/t5750-bundle-uri-parse.sh: test_expect_success 'bundle_uri_parse_line() parsin\n +\tcat >expect <<-\\EOF &&\n +\t[bundle]\n +\t\tversion = 1\n -+\t\tmode = \n ++\t\tmode = all\n +\tEOF\n +\n +\ttest_must_fail test-tool bundle-uri parse-config in1 >actual 2>err &&\n @@ t/t5750-bundle-uri-parse.sh: test_expect_success 'bundle_uri_parse_line() parsin\n +\tEOF\n +\n +\tcat >err2 <<-EOF &&\n -+\twarning: bundle list at '\\'''\\'' has no mode\n ++\terror: bad config line 1 in file in2\n +\tEOF\n +\n +\ttest_must_fail test-tool bundle-uri parse-config in2 >actual 2>err &&\n 6: 039e172849c ! 8: b5d570082fa bundle-uri: limit recursion depth for bundle lists\n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *fi\n +\t\t\t\t int depth)\n {\n \tint result = 0;\n - \tstruct strbuf filename = STRBUF_INIT;\n + \tchar *filename;\n \n +\tif (depth >= max_bundle_uri_depth) {\n +\t\twarning(_(\"exceeded bundle URI recursion limit (%d)\"),\n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *fi\n +\t\treturn -1;\n +\t}\n +\n - \tif ((result = find_temp_filename(&filename)))\n + \tif (!(filename = find_temp_filename())) {\n + \t\tresult = -1;\n \t\tgoto cleanup;\n - \n @@ bundle-uri.c: cleanup:\n \treturn result;\n }\n 7: 7b45c06cc9e ! 9: a6ab8f7c699 bundle-uri: fetch a list of bundles\n @@ Commit message\n Signed-off-by: Derrick Stolee \n \n ## bundle-uri.c ##\n -@@ bundle-uri.c: void init_bundle_list(struct bundle_list *list)\n - static int clear_remote_bundle_info(struct remote_bundle_info *bundle,\n - \t\t\t\t void *data)\n +@@ bundle-uri.c: static int clear_remote_bundle_info(struct remote_bundle_info *bundle,\n {\n --\tfree(bundle->id);\n --\tfree(bundle->uri);\n -+\tFREE_AND_NULL(bundle->id);\n -+\tFREE_AND_NULL(bundle->uri);\n - \tstrbuf_release(&bundle->file);\n + \tFREE_AND_NULL(bundle->id);\n + \tFREE_AND_NULL(bundle->uri);\n ++\tFREE_AND_NULL(bundle->file);\n +\tbundle->unbundled = 0;\n \treturn 0;\n }\n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *fi\n +\n +static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data)\n +{\n ++\tint res;\n +\tstruct bundle_list_context *ctx = data;\n +\n +\tif (ctx->mode == BUNDLE_MODE_ANY && ctx->count)\n +\t\treturn 0;\n +\n -+\tctx->count++;\n -+\treturn fetch_bundle_uri_internal(ctx->r, bundle, ctx->depth + 1, ctx->list);\n ++\tres = fetch_bundle_uri_internal(ctx->r, bundle, ctx->depth + 1, ctx->list);\n ++\n ++\t/*\n ++\t * Only increment count if the download succeeded. If our mode is\n ++\t * BUNDLE_MODE_ANY, then we will want to try other URIs in the\n ++\t * list in case they work instead.\n ++\t */\n ++\tif (!res)\n ++\t\tctx->count++;\n ++\treturn res;\n +}\n +\n +static int download_bundle_list(struct repository *r,\n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *fi\n +\n +\tinit_bundle_list(&list_from_bundle);\n +\n -+\tif ((result = parse_bundle_list_in_config_format(bundle->uri,\n -+\t\t\t\t\t\t\t bundle->file.buf,\n -+\t\t\t\t\t\t\t &list_from_bundle)))\n ++\tif ((result = bundle_uri_parse_config_format(bundle->uri,\n ++\t\t\t\t\t\t bundle->file,\n ++\t\t\t\t\t\t &list_from_bundle)))\n +\t\tgoto cleanup;\n +\n +\tif (list_from_bundle.mode == BUNDLE_MODE_NONE) {\n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *fi\n +\t\t\t\t struct bundle_list *list)\n {\n \tint result = 0;\n --\tstruct strbuf filename = STRBUF_INIT;\n +-\tchar *filename;\n +\tstruct remote_bundle_info *bcopy;\n \n \tif (depth >= max_bundle_uri_depth) {\n @@ bundle-uri.c: static int fetch_bundle_uri_internal(struct repository *r,\n \t\treturn -1;\n \t}\n \n --\tif ((result = find_temp_filename(&filename)))\n -+\tif (!bundle->file.len &&\n -+\t (result = find_temp_filename(&bundle->file)))\n +-\tif (!(filename = find_temp_filename())) {\n ++\tif (!bundle->file &&\n ++\t !(bundle->file = find_temp_filename())) {\n + \t\tresult = -1;\n \t\tgoto cleanup;\n + \t}\n \n --\tif ((result = copy_uri_to_file(filename.buf, uri))) {\n +-\tif ((result = copy_uri_to_file(filename, uri))) {\n -\t\twarning(_(\"failed to download bundle from URI '%s'\"), uri);\n -+\tif ((result = copy_uri_to_file(bundle->file.buf, bundle->uri))) {\n ++\tif ((result = copy_uri_to_file(bundle->file, bundle->uri))) {\n +\t\twarning(_(\"failed to download bundle from URI '%s'\"), bundle->uri);\n \t\tgoto cleanup;\n \t}\n \n --\tif ((result = !is_bundle(filename.buf, 0))) {\n +-\tif ((result = !is_bundle(filename, 0))) {\n -\t\twarning(_(\"file at URI '%s' is not a bundle\"), uri);\n -+\tif ((result = !is_bundle(bundle->file.buf, 1))) {\n ++\tif ((result = !is_bundle(bundle->file, 1))) {\n +\t\tresult = fetch_bundle_list_in_config_format(\n +\t\t\t\tr, list, bundle, depth);\n +\t\tif (result)\n @@ bundle-uri.c: static int fetch_bundle_uri_internal(struct repository *r,\n \t\tgoto cleanup;\n \t}\n \n --\tif ((result = unbundle_from_file(r, filename.buf))) {\n +-\tif ((result = unbundle_from_file(r, filename))) {\n -\t\twarning(_(\"failed to unbundle bundle from URI '%s'\"), uri);\n -\t\tgoto cleanup;\n -\t}\n +\t/* Copy the bundle and insert it into the global list. */\n +\tCALLOC_ARRAY(bcopy, 1);\n +\tbcopy->id = xstrdup(bundle->id);\n -+\tstrbuf_init(&bcopy->file, 0);\n -+\tstrbuf_add(&bcopy->file, bundle->file.buf, bundle->file.len);\n ++\tbcopy->file = xstrdup(bundle->file);\n +\thashmap_entry_init(&bcopy->ent, strhash(bcopy->id));\n +\thashmap_add(&list->bundles, &bcopy->ent);\n \n cleanup:\n --\tunlink(filename.buf);\n --\tstrbuf_release(&filename);\n -+\tif (result)\n -+\t\tunlink(bundle->file.buf);\n +-\tif (filename)\n +-\t\tunlink(filename);\n +-\tfree(filename);\n ++\tif (result && bundle->file)\n ++\t\tunlink(bundle->file);\n \treturn result;\n }\n \n @@ bundle-uri.c: static int fetch_bundle_uri_internal(struct repository *r,\n +{\n +\tstruct attempt_unbundle_context *ctx = data;\n +\n -+\tif (info->unbundled || !unbundle_from_file(ctx->r, info->file.buf)) {\n ++\tif (info->unbundled || !unbundle_from_file(ctx->r, info->file)) {\n +\t\tctx->success_count++;\n +\t\tinfo->unbundled = 1;\n +\t} else {\n @@ bundle-uri.c: static int fetch_bundle_uri_internal(struct repository *r,\n +\n +static int unlink_bundle(struct remote_bundle_info *info, void *data)\n +{\n -+\tif (info->file.buf)\n -+\t\tunlink_or_warn(info->file.buf);\n ++\tif (info->file)\n ++\t\tunlink_or_warn(info->file);\n +\treturn 0;\n +}\n +\n @@ bundle-uri.c: static int fetch_bundle_uri_internal(struct repository *r,\n +\tstruct bundle_list list;\n +\tstruct remote_bundle_info bundle = {\n +\t\t.uri = xstrdup(uri),\n -+\t\t.id = xstrdup(\"\"),\n -+\t\t.file = STRBUF_INIT,\n ++\t\t.id = xstrdup(\"\"),\n +\t};\n +\n +\tinit_bundle_list(&list);\n @@ bundle-uri.c: static int fetch_bundle_uri_internal(struct repository *r,\n \n ## bundle-uri.h ##\n @@ bundle-uri.h: struct remote_bundle_info {\n - \t * an empty string.\n + \t * if there was no table of contents.\n \t */\n - \tstruct strbuf file;\n + \tchar *uri;\n ++\n ++\t/**\n ++\t * If the bundle has been downloaded, then 'file' is a\n ++\t * filename storing its contents. Otherwise, 'file' is\n ++\t * NULL.\n ++\t */\n ++\tchar *file;\n +\n +\t/**\n +\t * If the bundle has been unbundled successfully, then\n @@ bundle-uri.h: struct remote_bundle_info {\n +\tunsigned unbundled:1;\n };\n \n - #define REMOTE_BUNDLE_INFO_INIT { \\\n + #define REMOTE_BUNDLE_INFO_INIT { 0 }\n \n ## t/t5558-clone-bundle-uri.sh ##\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone with file:// bundle' '\n\n-- \ngitgitgadget\n"},{"id":"462854","messageId":"ee6c4b824c276364b77de0b85b54f428c236de00.1662734015.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v2.git.1662734015.gitgitgadget@gmail.com","subject":"[PATCH v2 2/9] bundle-uri: use plain string in find_temp_filename()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-09T14:33:28Z","receivedAt":"2022-09-09T14:33:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe find_temp_filename() method was created in 53a50892be2 (bundle-uri:\ncreate basic file-copy logic, 2022-08-09) and uses odb_mkstemp() to\ncreate a temporary filename. The odb_mkstemp() method uses a strbuf in\nits interface, but we do not need to continue carrying a strbuf\nthroughout the bundle URI code.\n\nConvert the find_temp_filename() method to use a 'char *' and modify its\nonly caller. This makes sense that we don't actually need to modify this\nfilename directly later, so using a strbuf is overkill.\n\nThis change will simplify the data structure for tracking a bundle list\nto use plain strings instead of strbufs.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 28 ++++++++++++++++------------\n 1 file changed, 16 insertions(+), 12 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 7173ed065e9..c52b2a2a64a 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -5,22 +5,23 @@\n #include \"refs.h\"\n #include \"run-command.h\"\n \n-static int find_temp_filename(struct strbuf *name)\n+static char *find_temp_filename(void)\n {\n \tint fd;\n+\tstruct strbuf name = STRBUF_INIT;\n \t/*\n \t * Find a temporary filename that is available. This is briefly\n \t * racy, but unlikely to collide.\n \t */\n-\tfd = odb_mkstemp(name, \"bundles/tmp_uri_XXXXXX\");\n+\tfd = odb_mkstemp(&name, \"bundles/tmp_uri_XXXXXX\");\n \tif (fd < 0) {\n \t\twarning(_(\"failed to create temporary file\"));\n-\t\treturn -1;\n+\t\treturn NULL;\n \t}\n \n \tclose(fd);\n-\tunlink(name->buf);\n-\treturn 0;\n+\tunlink(name.buf);\n+\treturn strbuf_detach(&name, NULL);\n }\n \n static int download_https_uri_to_file(const char *file, const char *uri)\n@@ -143,28 +144,31 @@ static int unbundle_from_file(struct repository *r, const char *file)\n int fetch_bundle_uri(struct repository *r, const char *uri)\n {\n \tint result = 0;\n-\tstruct strbuf filename = STRBUF_INIT;\n+\tchar *filename;\n \n-\tif ((result = find_temp_filename(&filename)))\n+\tif (!(filename = find_temp_filename())) {\n+\t\tresult = -1;\n \t\tgoto cleanup;\n+\t}\n \n-\tif ((result = copy_uri_to_file(filename.buf, uri))) {\n+\tif ((result = copy_uri_to_file(filename, uri))) {\n \t\twarning(_(\"failed to download bundle from URI '%s'\"), uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = !is_bundle(filename.buf, 0))) {\n+\tif ((result = !is_bundle(filename, 0))) {\n \t\twarning(_(\"file at URI '%s' is not a bundle\"), uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = unbundle_from_file(r, filename.buf))) {\n+\tif ((result = unbundle_from_file(r, filename))) {\n \t\twarning(_(\"failed to unbundle bundle from URI '%s'\"), uri);\n \t\tgoto cleanup;\n \t}\n \n cleanup:\n-\tunlink(filename.buf);\n-\tstrbuf_release(&filename);\n+\tif (filename)\n+\t\tunlink(filename);\n+\tfree(filename);\n \treturn result;\n }\n-- \ngitgitgadget\n\n"},{"id":"462855","messageId":"70daef66833744fb8b870f431af8201222c7a2ba.1662734015.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v2.git.1662734015.gitgitgadget@gmail.com","subject":"[PATCH v2 4/9] bundle-uri: create base key-value pair parsing","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-09T14:33:30Z","receivedAt":"2022-09-09T14:33:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThere will be two primary ways to advertise a bundle list: as a list of\npacket lines in Git's protocol v2 and as a config file served from a\nbundle URI. Both of these fundamentally use a list of key-value pairs.\nWe will use the same set of key-value pairs across these formats.\n\nCreate a new bundle_list_update() method that is currently unusued, but\nwill be used in the next change. It inspects each key to see if it is\nunderstood and then applies it to the given bundle_list. Here are the\nkeys that we teach Git to understand:\n\n* bundle.version: This value should be an integer. Git currently\n understands only version 1 and will ignore the list if the version is\n any other value. This version can be increased in the future if we\n need to add new keys that Git should not ignore. We can add new\n \"heuristic\" keys without incrementing the version.\n\n* bundle.mode: This value should be one of \"all\" or \"any\". If this\n mode is not understood, then Git will ignore the list. This mode\n indicates whether Git needs all of the bundle list items to make a\n complete view of the content or if any single item is sufficient.\n\nThe rest of the keys use a bundle identifier \"\" as part of the key\nname. Keys using the same \"\" describe a single bundle list item.\n\n* bundle..uri: This stores the URI of the bundle item. This\n currently is expected to be an absolute URI, but will be relaxed to be\n a relative URI in the future.\n\nWhile parsing, return an error if a URI key is repeated, since we can\nmake that restriction with bundle lists.\n\nMake the git_parse_int() method global so we can parse the integer\nversion value carefully.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 2 +\n Documentation/config/bundle.txt | 24 +++++++++++\n bundle-uri.c | 76 +++++++++++++++++++++++++++++++++\n config.c | 2 +-\n config.h | 1 +\n 5 files changed, 104 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/config/bundle.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex e376d547ce0..4280af6992e 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -387,6 +387,8 @@ include::config/branch.txt[]\n \n include::config/browser.txt[]\n \n+include::config/bundle.txt[]\n+\n include::config/checkout.txt[]\n \n include::config/clean.txt[]\ndiff --git a/Documentation/config/bundle.txt b/Documentation/config/bundle.txt\nnew file mode 100644\nindex 00000000000..daa21eb674a\n--- /dev/null\n+++ b/Documentation/config/bundle.txt\n@@ -0,0 +1,24 @@\n+bundle.*::\n+\tThe `bundle.*` keys may appear in a bundle list file found via the\n+\t`git clone --bundle-uri` option. These keys currently have no effect\n+\tif placed in a repository config file, though this will change in the\n+\tfuture. See link:technical/bundle-uri.html[the bundle URI design\n+\tdocument] for more details.\n+\n+bundle.version::\n+\tThis integer value advertises the version of the bundle list format\n+\tused by the bundle list. Currently, the only accepted value is `1`.\n+\n+bundle.mode::\n+\tThis string value should be either `all` or `any`. This value describes\n+\twhether all of the advertised bundles are required to unbundle a\n+\tcomplete understanding of the bundled information (`all`) or if any one\n+\tof the listed bundle URIs is sufficient (`any`).\n+\n+bundle..*::\n+\tThe `bundle..*` keys are used to describe a single item in the\n+\tbundle list, grouped under `` for identification purposes.\n+\n+bundle..uri::\n+\tThis string value defines the URI by which Git can reach the contents\n+\tof this ``. This URI may be a bundle file or another bundle list.\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 7a0bada6eda..4ccd14c8936 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -6,6 +6,7 @@\n #include \"run-command.h\"\n #include \"hashmap.h\"\n #include \"pkt-line.h\"\n+#include \"config.h\"\n \n static int compare_bundles(const void *hashmap_cmp_fn_data,\n \t\t\t const struct hashmap_entry *he1,\n@@ -65,6 +66,81 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \treturn 0;\n }\n \n+/**\n+ * Given a key-value pair, update the state of the given bundle list.\n+ * Returns 0 if the key-value pair is understood. Returns 1 if the key\n+ * is not understood or the value is malformed.\n+ */\n+MAYBE_UNUSED\n+static int bundle_list_update(const char *key, const char *value,\n+\t\t\t struct bundle_list *list)\n+{\n+\tstruct strbuf id = STRBUF_INIT;\n+\tstruct remote_bundle_info lookup = REMOTE_BUNDLE_INFO_INIT;\n+\tstruct remote_bundle_info *bundle;\n+\tconst char *subsection, *subkey;\n+\tsize_t subsection_len;\n+\n+\tif (parse_config_key(key, \"bundle\", &subsection, &subsection_len, &subkey))\n+\t\treturn -1;\n+\n+\tif (!subsection_len) {\n+\t\tif (!strcmp(subkey, \"version\")) {\n+\t\t\tint version;\n+\t\t\tif (!git_parse_int(value, &version))\n+\t\t\t\treturn -1;\n+\t\t\tif (version != 1)\n+\t\t\t\treturn -1;\n+\n+\t\t\tlist->version = version;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\tif (!strcmp(subkey, \"mode\")) {\n+\t\t\tif (!strcmp(value, \"all\"))\n+\t\t\t\tlist->mode = BUNDLE_MODE_ALL;\n+\t\t\telse if (!strcmp(value, \"any\"))\n+\t\t\t\tlist->mode = BUNDLE_MODE_ANY;\n+\t\t\telse\n+\t\t\t\treturn -1;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\t/* Ignore other unknown global keys. */\n+\t\treturn 0;\n+\t}\n+\n+\tstrbuf_add(&id, subsection, subsection_len);\n+\n+\t/*\n+\t * Check for an existing bundle with this , or create one\n+\t * if necessary.\n+\t */\n+\tlookup.id = id.buf;\n+\thashmap_entry_init(&lookup.ent, strhash(lookup.id));\n+\tif (!(bundle = hashmap_get_entry(&list->bundles, &lookup, ent, NULL))) {\n+\t\tCALLOC_ARRAY(bundle, 1);\n+\t\tbundle->id = strbuf_detach(&id, NULL);\n+\t\thashmap_entry_init(&bundle->ent, strhash(bundle->id));\n+\t\thashmap_add(&list->bundles, &bundle->ent);\n+\t}\n+\tstrbuf_release(&id);\n+\n+\tif (!strcmp(subkey, \"uri\")) {\n+\t\tif (bundle->uri)\n+\t\t\treturn -1;\n+\t\tbundle->uri = xstrdup(value);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * At this point, we ignore any information that we don't\n+\t * understand, assuming it to be hints for a heuristic the client\n+\t * does not currently understand.\n+\t */\n+\treturn 0;\n+}\n+\n static char *find_temp_filename(void)\n {\n \tint fd;\ndiff --git a/config.c b/config.c\nindex 015bec360f5..e93101249f6 100644\n--- a/config.c\n+++ b/config.c\n@@ -1214,7 +1214,7 @@ static int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max)\n \treturn 0;\n }\n \n-static int git_parse_int(const char *value, int *ret)\n+int git_parse_int(const char *value, int *ret)\n {\n \tintmax_t tmp;\n \tif (!git_parse_signed(value, &tmp, maximum_signed_value_of_type(int)))\ndiff --git a/config.h b/config.h\nindex ca994d77147..ef9eade6414 100644\n--- a/config.h\n+++ b/config.h\n@@ -206,6 +206,7 @@ int config_with_options(config_fn_t fn, void *,\n \n int git_parse_ssize_t(const char *, ssize_t *);\n int git_parse_ulong(const char *, unsigned long *);\n+int git_parse_int(const char *value, int *ret);\n \n /**\n * Same as `git_config_bool`, except that it returns -1 on error rather\n-- \ngitgitgadget\n\n"},{"id":"462856","messageId":"d981244059439372349c6c48c161fa989df2cce5.1662734015.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v2.git.1662734015.gitgitgadget@gmail.com","subject":"[PATCH v2 3/9] bundle-uri: create bundle_list struct and helpers","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-09T14:33:29Z","receivedAt":"2022-09-09T14:33:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt will likely be rare where a user uses a single bundle URI and expects\nthat URI to point to a bundle. Instead, that URI will likely be a list\nof bundles provided in some format. Alternatively, the Git server could\nadvertise a list of bundles.\n\nIn anticipation of these two ways of advertising multiple bundles,\ncreate a data structure that represents such a list. This will be\npopulated using a common API, but for now focus on what data can be\nrepresented.\n\nEach list contains a number of remote_bundle_info structs. These contain\nan 'id' that is used to uniquely identify them in the list, and also a\n'uri' that contains the location of its data. Finally, there is a strbuf\ncontaining the filename used when Git downloads the contents to disk.\n\nThe list itself stores these remote_bundle_info structs in a hashtable\nusing 'id' as the key. The order of the structs in the input is\nconsidered unimportant, but future modifications to the format and these\ndata structures will place ordering possibilities on the set. The list\nalso has a few \"global\" properties, including the version (used when\nparsing the list) and the mode. The mode is one of these two options:\n\n1. BUNDLE_MODE_ALL: all listed URIs are intended to be combined\n together. The client should download all of the advertised data to\n have a complete copy of the data.\n\n2. BUNDLE_MODE_ANY: any one listed item is sufficient to have a complete\n copy of the data. The client can choose arbitrarily from these\n options. In the future, the client may use pings to find the closest\n URI among geodistributed replicas, or use some other heuristic\n information added to the format.\n\nThis API is currently unused, but will soon be expanded with parsing\nlogic and then be consumed by the bundle URI download logic.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n bundle-uri.h | 56 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 116 insertions(+)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex c52b2a2a64a..7a0bada6eda 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -4,6 +4,66 @@\n #include \"object-store.h\"\n #include \"refs.h\"\n #include \"run-command.h\"\n+#include \"hashmap.h\"\n+#include \"pkt-line.h\"\n+\n+static int compare_bundles(const void *hashmap_cmp_fn_data,\n+\t\t\t const struct hashmap_entry *he1,\n+\t\t\t const struct hashmap_entry *he2,\n+\t\t\t const void *id)\n+{\n+\tconst struct remote_bundle_info *e1 =\n+\t\tcontainer_of(he1, const struct remote_bundle_info, ent);\n+\tconst struct remote_bundle_info *e2 =\n+\t\tcontainer_of(he2, const struct remote_bundle_info, ent);\n+\n+\treturn strcmp(e1->id, id ? (const char *)id : e2->id);\n+}\n+\n+void init_bundle_list(struct bundle_list *list)\n+{\n+\tmemset(list, 0, sizeof(*list));\n+\n+\t/* Implied defaults. */\n+\tlist->mode = BUNDLE_MODE_ALL;\n+\tlist->version = 1;\n+\n+\thashmap_init(&list->bundles, compare_bundles, NULL, 0);\n+}\n+\n+static int clear_remote_bundle_info(struct remote_bundle_info *bundle,\n+\t\t\t\t void *data)\n+{\n+\tFREE_AND_NULL(bundle->id);\n+\tFREE_AND_NULL(bundle->uri);\n+\treturn 0;\n+}\n+\n+void clear_bundle_list(struct bundle_list *list)\n+{\n+\tif (!list)\n+\t\treturn;\n+\n+\tfor_all_bundles_in_list(list, clear_remote_bundle_info, NULL);\n+\thashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent);\n+}\n+\n+int for_all_bundles_in_list(struct bundle_list *list,\n+\t\t\t bundle_iterator iter,\n+\t\t\t void *data)\n+{\n+\tstruct remote_bundle_info *info;\n+\tstruct hashmap_iter i;\n+\n+\thashmap_for_each_entry(&list->bundles, &i, info, ent) {\n+\t\tint result = iter(info, data);\n+\n+\t\tif (result)\n+\t\t\treturn result;\n+\t}\n+\n+\treturn 0;\n+}\n \n static char *find_temp_filename(void)\n {\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 8a152f1ef14..ff7e3fd3fb2 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -1,7 +1,63 @@\n #ifndef BUNDLE_URI_H\n #define BUNDLE_URI_H\n \n+#include \"hashmap.h\"\n+#include \"strbuf.h\"\n+\n struct repository;\n+struct string_list;\n+\n+/**\n+ * The remote_bundle_info struct contains information for a single bundle\n+ * URI. This may be initialized simply by a given URI or might have\n+ * additional metadata associated with it if the bundle was advertised by\n+ * a bundle list.\n+ */\n+struct remote_bundle_info {\n+\tstruct hashmap_entry ent;\n+\n+\t/**\n+\t * The 'id' is a name given to the bundle for reference\n+\t * by other bundle infos.\n+\t */\n+\tchar *id;\n+\n+\t/**\n+\t * The 'uri' is the location of the remote bundle so\n+\t * it can be downloaded on-demand. This will be NULL\n+\t * if there was no table of contents.\n+\t */\n+\tchar *uri;\n+};\n+\n+#define REMOTE_BUNDLE_INFO_INIT { 0 }\n+\n+enum bundle_list_mode {\n+\tBUNDLE_MODE_NONE = 0,\n+\tBUNDLE_MODE_ALL,\n+\tBUNDLE_MODE_ANY\n+};\n+\n+/**\n+ * A bundle_list contains an unordered set of remote_bundle_info structs,\n+ * as well as information about the bundle listing, such as version and\n+ * mode.\n+ */\n+struct bundle_list {\n+\tint version;\n+\tenum bundle_list_mode mode;\n+\tstruct hashmap bundles;\n+};\n+\n+void init_bundle_list(struct bundle_list *list);\n+void clear_bundle_list(struct bundle_list *list);\n+\n+typedef int (*bundle_iterator)(struct remote_bundle_info *bundle,\n+\t\t\t void *data);\n+\n+int for_all_bundles_in_list(struct bundle_list *list,\n+\t\t\t bundle_iterator iter,\n+\t\t\t void *data);\n \n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n-- \ngitgitgadget\n\n"},{"id":"462857","messageId":"4df3f83402908d88ef0bf491c1cc2a745f6660e5.1662734015.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v2.git.1662734015.gitgitgadget@gmail.com","subject":"[PATCH v2 5/9] bundle-uri: create \"key=value\" line parsing","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-09T14:33:31Z","receivedAt":"2022-09-09T14:34:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nWhen advertising a bundle list over Git's protocol v2, we will use\npacket lines. Each line will be of the form \"key=value\" representing a\nbundle list. Connect the API necessary for Git's transport to the\nkey-value pair parsing created in the previous change.\n\nWe are not currently implementing this protocol v2 functionality, but\ninstead preparing to expose this parsing to be unit-testable.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 27 ++++++++++++++++++++++++++-\n bundle-uri.h | 12 ++++++++++++\n 2 files changed, 38 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 4ccd14c8936..d4eb1ec7d4d 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -71,7 +71,6 @@ int for_all_bundles_in_list(struct bundle_list *list,\n * Returns 0 if the key-value pair is understood. Returns 1 if the key\n * is not understood or the value is malformed.\n */\n-MAYBE_UNUSED\n static int bundle_list_update(const char *key, const char *value,\n \t\t\t struct bundle_list *list)\n {\n@@ -308,3 +307,29 @@ cleanup:\n \tfree(filename);\n \treturn result;\n }\n+\n+/**\n+ * General API for {transport,connect}.c etc.\n+ */\n+int bundle_uri_parse_line(struct bundle_list *list, const char *line)\n+{\n+\tint result;\n+\tconst char *equals;\n+\tstruct strbuf key = STRBUF_INIT;\n+\n+\tif (!strlen(line))\n+\t\treturn error(_(\"bundle-uri: got an empty line\"));\n+\n+\tequals = strchr(line, '=');\n+\n+\tif (!equals)\n+\t\treturn error(_(\"bundle-uri: line is not of the form 'key=value'\"));\n+\tif (line == equals || !*(equals + 1))\n+\t\treturn error(_(\"bundle-uri: line has empty key or value\"));\n+\n+\tstrbuf_add(&key, line, equals - line);\n+\tresult = bundle_list_update(key.buf, equals + 1, list);\n+\tstrbuf_release(&key);\n+\n+\treturn result;\n+}\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex ff7e3fd3fb2..90583461929 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -67,4 +67,16 @@ int for_all_bundles_in_list(struct bundle_list *list,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n+/**\n+ * General API for {transport,connect}.c etc.\n+ */\n+\n+/**\n+ * Parse a \"key=value\" packet line from the bundle-uri verb.\n+ *\n+ * Returns 0 on success and non-zero on error.\n+ */\n+int bundle_uri_parse_line(struct bundle_list *list,\n+\t\t\t const char *line);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"462858","messageId":"91c5b58f011ad711c301eddc2fecf79ab1ba726a.1662734015.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v2.git.1662734015.gitgitgadget@gmail.com","subject":"[PATCH v2 6/9] bundle-uri: unit test \"key=value\" parsing","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-09T14:33:32Z","receivedAt":"2022-09-09T14:34:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nCreate a new 'test-tool bundle-uri' test helper. This helper will assist\nin testing logic deep in the bundle URI feature.\n\nThis change introduces the 'parse-key-values' subcommand, which parses\nan input file as a list of lines. These are fed into\nbundle_uri_parse_line() to test how we construct a 'struct bundle_list'\nfrom that data. The list is then output to stdout as if the key-value\npairs were a Git config file.\n\nWe use an input file instead of stdin because of a future change to\nparse in config-file format that works better as an input file.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n bundle-uri.c | 33 ++++++++++\n bundle-uri.h | 3 +\n t/helper/test-bundle-uri.c | 70 +++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5750-bundle-uri-parse.sh | 121 ++++++++++++++++++++++++++++++++++++\n t/test-lib-functions.sh | 11 ++++\n 8 files changed, 241 insertions(+)\n create mode 100644 t/helper/test-bundle-uri.c\n create mode 100755 t/t5750-bundle-uri-parse.sh\n\ndiff --git a/Makefile b/Makefile\nindex 7d5f48069ea..7dee0329c49 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -722,6 +722,7 @@ PROGRAMS += $(patsubst %.o,git-%$X,$(PROGRAM_OBJS))\n TEST_BUILTINS_OBJS += test-advise.o\n TEST_BUILTINS_OBJS += test-bitmap.o\n TEST_BUILTINS_OBJS += test-bloom.o\n+TEST_BUILTINS_OBJS += test-bundle-uri.o\n TEST_BUILTINS_OBJS += test-chmtime.o\n TEST_BUILTINS_OBJS += test-config.o\n TEST_BUILTINS_OBJS += test-crontab.o\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex d4eb1ec7d4d..74d5695e99e 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -66,6 +66,39 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \treturn 0;\n }\n \n+static int summarize_bundle(struct remote_bundle_info *info, void *data)\n+{\n+\tFILE *fp = data;\n+\tfprintf(fp, \"[bundle \\\"%s\\\"]\\n\", info->id);\n+\tfprintf(fp, \"\\turi = %s\\n\", info->uri);\n+\treturn 0;\n+}\n+\n+void print_bundle_list(FILE *fp, struct bundle_list *list)\n+{\n+\tconst char *mode;\n+\n+\tswitch (list->mode) {\n+\tcase BUNDLE_MODE_ALL:\n+\t\tmode = \"all\";\n+\t\tbreak;\n+\n+\tcase BUNDLE_MODE_ANY:\n+\t\tmode = \"any\";\n+\t\tbreak;\n+\n+\tcase BUNDLE_MODE_NONE:\n+\tdefault:\n+\t\tmode = \"\";\n+\t}\n+\n+\tfprintf(fp, \"[bundle]\\n\");\n+\tfprintf(fp, \"\\tversion = %d\\n\", list->version);\n+\tfprintf(fp, \"\\tmode = %s\\n\", mode);\n+\n+\tfor_all_bundles_in_list(list, summarize_bundle, fp);\n+}\n+\n /**\n * Given a key-value pair, update the state of the given bundle list.\n * Returns 0 if the key-value pair is understood. Returns 1 if the key\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 90583461929..0e56ab2ae5a 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -59,6 +59,9 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \t\t\t bundle_iterator iter,\n \t\t\t void *data);\n \n+struct FILE;\n+void print_bundle_list(FILE *fp, struct bundle_list *list);\n+\n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n * based on that information.\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nnew file mode 100644\nindex 00000000000..0329c56544f\n--- /dev/null\n+++ b/t/helper/test-bundle-uri.c\n@@ -0,0 +1,70 @@\n+#include \"test-tool.h\"\n+#include \"parse-options.h\"\n+#include \"bundle-uri.h\"\n+#include \"strbuf.h\"\n+#include \"string-list.h\"\n+\n+static int cmd__bundle_uri_parse(int argc, const char **argv)\n+{\n+\tconst char *key_value_usage[] = {\n+\t\t\"test-tool bundle-uri parse-key-values \",\n+\t\tNULL\n+\t};\n+\tconst char **usage = key_value_usage;\n+\tstruct option options[] = {\n+\t\tOPT_END(),\n+\t};\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct bundle_list list;\n+\tint err = 0;\n+\tFILE *fp;\n+\n+\targc = parse_options(argc, argv, NULL, options, usage, 0);\n+\tif (argc != 1)\n+\t\tgoto usage;\n+\n+\tinit_bundle_list(&list);\n+\tfp = fopen(argv[0], \"r\");\n+\tif (!fp)\n+\t\tdie(\"failed to open '%s'\", argv[0]);\n+\n+\twhile (strbuf_getline(&sb, fp) != EOF) {\n+\t\tif (bundle_uri_parse_line(&list, sb.buf))\n+\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\t}\n+\tstrbuf_release(&sb);\n+\tfclose(fp);\n+\n+\tprint_bundle_list(stdout, &list);\n+\n+\tclear_bundle_list(&list);\n+\n+\treturn !!err;\n+\n+usage:\n+\tusage_with_options(usage, options);\n+}\n+\n+int cmd__bundle_uri(int argc, const char **argv)\n+{\n+\tconst char *usage[] = {\n+\t\t\"test-tool bundle-uri []\",\n+\t\tNULL\n+\t};\n+\tstruct option options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL, options, usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION |\n+\t\t\t PARSE_OPT_KEEP_ARGV0);\n+\tif (argc == 1)\n+\t\tgoto usage;\n+\n+\tif (!strcmp(argv[1], \"parse-key-values\"))\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1);\n+\terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n+\n+usage:\n+\tusage_with_options(usage, options);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 318fdbab0c3..fbe2d9d8108 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -17,6 +17,7 @@ static struct test_cmd cmds[] = {\n \t{ \"advise\", cmd__advise_if_enabled },\n \t{ \"bitmap\", cmd__bitmap },\n \t{ \"bloom\", cmd__bloom },\n+\t{ \"bundle-uri\", cmd__bundle_uri },\n \t{ \"chmtime\", cmd__chmtime },\n \t{ \"config\", cmd__config },\n \t{ \"crontab\", cmd__crontab },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex bb799271631..b2aa1f39a8f 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -7,6 +7,7 @@\n int cmd__advise_if_enabled(int argc, const char **argv);\n int cmd__bitmap(int argc, const char **argv);\n int cmd__bloom(int argc, const char **argv);\n+int cmd__bundle_uri(int argc, const char **argv);\n int cmd__chmtime(int argc, const char **argv);\n int cmd__config(int argc, const char **argv);\n int cmd__crontab(int argc, const char **argv);\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nnew file mode 100755\nindex 00000000000..fd142a66ad5\n--- /dev/null\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -0,0 +1,121 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri bundle_uri_parse_line()\"\n+\n+TEST_NO_CREATE_REPO=1\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success 'bundle_uri_parse_line() just URIs' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' '\n+\tcat >in <<-\\EOF &&\n+\t=bogus-value\n+\tbogus-key=\n+\tEOF\n+\n+\tcat >err.expect <<-EOF &&\n+\terror: bundle-uri: line has empty key or value\n+\terror: bad line: '\\''=bogus-value'\\''\n+\terror: bundle-uri: line has empty key or value\n+\terror: bad line: '\\''bogus-key='\\''\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty lines' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\terror: bundle-uri: got an empty line\n+\terror: bad line: '\\'''\\''\n+\terror: bundle-uri: got an empty line\n+\terror: bad line: '\\'''\\''\n+\tEOF\n+\n+\t# We fail, but try to continue parsing regardless\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: duplicate lines' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\tbundle.one.uri=https://example.com/bundle-2.bdl\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\terror: bad line: '\\''bundle.one.uri=https://example.com/bundle-2.bdl'\\''\n+\tEOF\n+\n+\t# We fail, but try to continue parsing regardless\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_done\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 6da7273f1d5..3175d665add 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1956,3 +1956,14 @@ test_is_magic_mtime () {\n \trm -f .git/test-mtime-actual\n \treturn $ret\n }\n+\n+# Given two filenames, parse both using 'git config --list --file'\n+# and compare the sorted output of those commands. Useful when\n+# wanting to ignore whitespace differences and sorting concerns.\n+test_cmp_config_output () {\n+\tgit config --list --file=\"$1\" >config-expect &&\n+\tgit config --list --file=\"$2\" >config-actual &&\n+\tsort config-expect >sorted-expect &&\n+\tsort config-actual >sorted-actual &&\n+\ttest_cmp sorted-expect sorted-actual\n+}\n-- \ngitgitgadget\n\n"},{"id":"462859","messageId":"1492b8f5ef031f30374a626b4f6e772427531e23.1662734015.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v2.git.1662734015.gitgitgadget@gmail.com","subject":"[PATCH v2 7/9] bundle-uri: parse bundle list in config format","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-09T14:33:33Z","receivedAt":"2022-09-09T14:34:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a bundle provider wants to operate independently from a Git remote,\nthey want to provide a single, consistent URI that users can use in\ntheir 'git clone --bundle-uri' commands. At this point, the Git client\nexpects that URI to be a single bundle that can be unbundled and used to\nbootstrap the rest of the clone from the Git server. This single bundle\ncannot be re-used to assist with future incremental fetches.\n\nTo allow for the incremental fetch case, teach Git to understand a\nbundle list that could be advertised at an independent bundle URI. Such\na bundle list is likely to be inspected by human readers, even if only\nby the bundle provider creating the list. For this reason, we can take\nour expected \"key=value\" pairs and instead format them using Git config\nformat.\n\nCreate bundle_uri_parse_config_format() to parse a file in config format\nand convert that into a 'struct bundle_list' filled with its\nunderstanding of the contents.\n\nBe careful to use error_action CONFIG_ERROR_ERROR when calling\ngit_config_from_file_with_options() because the default action for\ngit_config_from_file() is to die() on a parsing error. The current\nwarning isn't particularly helpful if it arises to a user, but it will\nbe made more verbose at a higher layer later.\n\nUpdate 'test-tool bundle-uri' to take this config file format as input.\nIt uses a filename instead of stdin because there is no existing way to\nparse a FILE pointer in the config machinery. Using\ngit_config_from_mem() is overly complicated and more likely to introduce\nbugs than this simpler version.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 27 ++++++++++++++++++++\n bundle-uri.h | 9 +++++++\n t/helper/test-bundle-uri.c | 49 +++++++++++++++++++++++++++---------\n t/t5750-bundle-uri-parse.sh | 50 +++++++++++++++++++++++++++++++++++++\n 4 files changed, 123 insertions(+), 12 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 74d5695e99e..92354aa3bbd 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -173,6 +173,33 @@ static int bundle_list_update(const char *key, const char *value,\n \treturn 0;\n }\n \n+static int config_to_bundle_list(const char *key, const char *value, void *data)\n+{\n+\tstruct bundle_list *list = data;\n+\treturn bundle_list_update(key, value, list);\n+}\n+\n+int bundle_uri_parse_config_format(const char *uri,\n+\t\t\t\t const char *filename,\n+\t\t\t\t struct bundle_list *list)\n+{\n+\tint result;\n+\tstruct config_options opts = {\n+\t\t.error_action = CONFIG_ERROR_ERROR,\n+\t};\n+\n+\tresult = git_config_from_file_with_options(config_to_bundle_list,\n+\t\t\t\t\t\t filename, list,\n+\t\t\t\t\t\t &opts);\n+\n+\tif (!result && list->mode == BUNDLE_MODE_NONE) {\n+\t\twarning(_(\"bundle list at '%s' has no mode\"), uri);\n+\t\tresult = 1;\n+\t}\n+\n+\treturn result;\n+}\n+\n static char *find_temp_filename(void)\n {\n \tint fd;\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 0e56ab2ae5a..bc13d4c9929 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -62,6 +62,15 @@ int for_all_bundles_in_list(struct bundle_list *list,\n struct FILE;\n void print_bundle_list(FILE *fp, struct bundle_list *list);\n \n+/**\n+ * A bundle URI may point to a bundle list where the key=value\n+ * pairs are provided in config file format. This method is\n+ * exposed publicly for testing purposes.\n+ */\n+int bundle_uri_parse_config_format(const char *uri,\n+\t\t\t\t const char *filename,\n+\t\t\t\t struct bundle_list *list);\n+\n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n * based on that information.\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex 0329c56544f..25afd393428 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -4,12 +4,21 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n \n-static int cmd__bundle_uri_parse(int argc, const char **argv)\n+enum input_mode {\n+\tKEY_VALUE_PAIRS,\n+\tCONFIG_FILE,\n+};\n+\n+static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mode)\n {\n \tconst char *key_value_usage[] = {\n \t\t\"test-tool bundle-uri parse-key-values \",\n \t\tNULL\n \t};\n+\tconst char *config_usage[] = {\n+\t\t\"test-tool bundle-uri parse-config \",\n+\t\tNULL\n+\t};\n \tconst char **usage = key_value_usage;\n \tstruct option options[] = {\n \t\tOPT_END(),\n@@ -19,21 +28,35 @@ static int cmd__bundle_uri_parse(int argc, const char **argv)\n \tint err = 0;\n \tFILE *fp;\n \n-\targc = parse_options(argc, argv, NULL, options, usage, 0);\n-\tif (argc != 1)\n-\t\tgoto usage;\n+\tif (mode == CONFIG_FILE)\n+\t\tusage = config_usage;\n+\n+\targc = parse_options(argc, argv, NULL, options, usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tinit_bundle_list(&list);\n-\tfp = fopen(argv[0], \"r\");\n-\tif (!fp)\n-\t\tdie(\"failed to open '%s'\", argv[0]);\n \n-\twhile (strbuf_getline(&sb, fp) != EOF) {\n-\t\tif (bundle_uri_parse_line(&list, sb.buf))\n-\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\tswitch (mode) {\n+\tcase KEY_VALUE_PAIRS:\n+\t\tif (argc != 1)\n+\t\t\tgoto usage;\n+\t\tfp = fopen(argv[0], \"r\");\n+\t\tif (!fp)\n+\t\t\tdie(\"failed to open '%s'\", argv[0]);\n+\t\twhile (strbuf_getline(&sb, fp) != EOF) {\n+\t\t\tif (bundle_uri_parse_line(&list, sb.buf))\n+\t\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\t\t}\n+\t\tfclose(fp);\n+\t\tbreak;\n+\n+\tcase CONFIG_FILE:\n+\t\tif (argc != 1)\n+\t\t\tgoto usage;\n+\t\terr = bundle_uri_parse_config_format(\"\", argv[0], &list);\n+\t\tbreak;\n \t}\n \tstrbuf_release(&sb);\n-\tfclose(fp);\n \n \tprint_bundle_list(stdout, &list);\n \n@@ -62,7 +85,9 @@ int cmd__bundle_uri(int argc, const char **argv)\n \t\tgoto usage;\n \n \tif (!strcmp(argv[1], \"parse-key-values\"))\n-\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1);\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS);\n+\tif (!strcmp(argv[1], \"parse-config\"))\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE);\n \terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n \n usage:\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex fd142a66ad5..c2fe3f9c5a5 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -118,4 +118,54 @@ test_expect_success 'bundle_uri_parse_line() parsing edge cases: duplicate lines\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format: just URIs' '\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config expect >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'parse config format edge cases: empty key or value' '\n+\tcat >in1 <<-\\EOF &&\n+\t= bogus-value\n+\tEOF\n+\n+\tcat >err1 <<-EOF &&\n+\terror: bad config line 1 in file in1\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-config in1 >actual 2>err &&\n+\ttest_cmp err1 err &&\n+\ttest_cmp_config_output expect actual &&\n+\n+\tcat >in2 <<-\\EOF &&\n+\tbogus-key =\n+\tEOF\n+\n+\tcat >err2 <<-EOF &&\n+\terror: bad config line 1 in file in2\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-config in2 >actual 2>err &&\n+\ttest_cmp err2 err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"462860","messageId":"b5d570082fa7d63d97d3e96a48001e305700288d.1662734015.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v2.git.1662734015.gitgitgadget@gmail.com","subject":"[PATCH v2 8/9] bundle-uri: limit recursion depth for bundle lists","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-09T14:33:34Z","receivedAt":"2022-09-09T14:34:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe next change will start allowing us to parse bundle lists that are\ndownloaded from a provided bundle URI. Those lists might point to other\nlists, which could proceed to an arbitrary depth (and even create\ncycles). Restructure fetch_bundle_uri() to have an internal version that\nhas a recursion depth. Compare that to a new max_bundle_uri_depth\nconstant that is twice as high as we expect this depth to be for any\nlegitimate use of bundle list linking.\n\nWe can consider making max_bundle_uri_depth a configurable value if\nthere is demonstrated value in the future.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 21 ++++++++++++++++++++-\n 1 file changed, 20 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 92354aa3bbd..b8ca6cd9493 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -336,11 +336,25 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \treturn result;\n }\n \n-int fetch_bundle_uri(struct repository *r, const char *uri)\n+/**\n+ * This limits the recursion on fetch_bundle_uri_internal() when following\n+ * bundle lists.\n+ */\n+static int max_bundle_uri_depth = 4;\n+\n+static int fetch_bundle_uri_internal(struct repository *r,\n+\t\t\t\t const char *uri,\n+\t\t\t\t int depth)\n {\n \tint result = 0;\n \tchar *filename;\n \n+\tif (depth >= max_bundle_uri_depth) {\n+\t\twarning(_(\"exceeded bundle URI recursion limit (%d)\"),\n+\t\t\tmax_bundle_uri_depth);\n+\t\treturn -1;\n+\t}\n+\n \tif (!(filename = find_temp_filename())) {\n \t\tresult = -1;\n \t\tgoto cleanup;\n@@ -368,6 +382,11 @@ cleanup:\n \treturn result;\n }\n \n+int fetch_bundle_uri(struct repository *r, const char *uri)\n+{\n+\treturn fetch_bundle_uri_internal(r, uri, 0);\n+}\n+\n /**\n * General API for {transport,connect}.c etc.\n */\n-- \ngitgitgadget\n\n"},{"id":"462861","messageId":"a6ab8f7c69925b5dfd95f3e4c83c6ccd4d2dfee3.1662734015.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v2.git.1662734015.gitgitgadget@gmail.com","subject":"[PATCH v2 9/9] bundle-uri: fetch a list of bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-09T14:33:35Z","receivedAt":"2022-09-09T14:34:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen the content at a given bundle URI is not understood as a bundle\n(based on inspecting the initial content), then Git currently gives up\nand ignores that content. Independent bundle providers may want to split\nup the bundle content into multiple bundles, but still make them\navailable from a single URI.\n\nTeach Git to attempt parsing the bundle URI content as a Git config file\nproviding the key=value pairs for a bundle list. Git then looks at the\nmode of the list to see if ANY single bundle is sufficient or if ALL\nbundles are required. The content at the selected URIs are downloaded\nand the content is inspected again, creating a recursive process.\n\nTo guard the recursion against malformed or malicious content, limit the\nrecursion depth to a reasonable four for now. This can be converted to a\nconfigured value in the future if necessary. The value of four is twice\nas high as expected to be useful (a bundle list is unlikely to point to\nmore bundle lists).\n\nTo test this scenario, create an interesting bundle topology where three\nincremental bundles are built on top of a single full bundle. By using a\nmerge commit, the two middle bundles are \"independent\" in that they do\nnot require each other in order to unbundle themselves. They each only\nneed the base bundle. The bundle containing the merge commit requires\nboth of the middle bundles, though. This leads to some interesting\ndecisions when unbundling, especially when we later implement heuristics\nthat promote downloading bundles until the prerequisite commits are\nsatisfied.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 216 +++++++++++++++++++++++++++++++++---\n bundle-uri.h | 13 +++\n t/t5558-clone-bundle-uri.sh | 93 ++++++++++++++++\n 3 files changed, 306 insertions(+), 16 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex b8ca6cd9493..6a2fea26a94 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -37,6 +37,8 @@ static int clear_remote_bundle_info(struct remote_bundle_info *bundle,\n {\n \tFREE_AND_NULL(bundle->id);\n \tFREE_AND_NULL(bundle->uri);\n+\tFREE_AND_NULL(bundle->file);\n+\tbundle->unbundled = 0;\n \treturn 0;\n }\n \n@@ -336,18 +338,111 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \treturn result;\n }\n \n+struct bundle_list_context {\n+\tstruct repository *r;\n+\tstruct bundle_list *list;\n+\tenum bundle_list_mode mode;\n+\tint count;\n+\tint depth;\n+};\n+\n+/*\n+ * This early definition is necessary because we use indirect recursion:\n+ *\n+ * While iterating through a bundle list that was downloaded as part\n+ * of fetch_bundle_uri_internal(), iterator methods eventually call it\n+ * again, but with depth + 1.\n+ */\n+static int fetch_bundle_uri_internal(struct repository *r,\n+\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t int depth,\n+\t\t\t\t struct bundle_list *list);\n+\n+static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data)\n+{\n+\tint res;\n+\tstruct bundle_list_context *ctx = data;\n+\n+\tif (ctx->mode == BUNDLE_MODE_ANY && ctx->count)\n+\t\treturn 0;\n+\n+\tres = fetch_bundle_uri_internal(ctx->r, bundle, ctx->depth + 1, ctx->list);\n+\n+\t/*\n+\t * Only increment count if the download succeeded. If our mode is\n+\t * BUNDLE_MODE_ANY, then we will want to try other URIs in the\n+\t * list in case they work instead.\n+\t */\n+\tif (!res)\n+\t\tctx->count++;\n+\treturn res;\n+}\n+\n+static int download_bundle_list(struct repository *r,\n+\t\t\t\tstruct bundle_list *local_list,\n+\t\t\t\tstruct bundle_list *global_list,\n+\t\t\t\tint depth)\n+{\n+\tstruct bundle_list_context ctx = {\n+\t\t.r = r,\n+\t\t.list = global_list,\n+\t\t.depth = depth + 1,\n+\t\t.mode = local_list->mode,\n+\t};\n+\n+\treturn for_all_bundles_in_list(local_list, download_bundle_to_file, &ctx);\n+}\n+\n+static int fetch_bundle_list_in_config_format(struct repository *r,\n+\t\t\t\t\t struct bundle_list *global_list,\n+\t\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t\t int depth)\n+{\n+\tint result;\n+\tstruct bundle_list list_from_bundle;\n+\n+\tinit_bundle_list(&list_from_bundle);\n+\n+\tif ((result = bundle_uri_parse_config_format(bundle->uri,\n+\t\t\t\t\t\t bundle->file,\n+\t\t\t\t\t\t &list_from_bundle)))\n+\t\tgoto cleanup;\n+\n+\tif (list_from_bundle.mode == BUNDLE_MODE_NONE) {\n+\t\twarning(_(\"unrecognized bundle mode from URI '%s'\"),\n+\t\t\tbundle->uri);\n+\t\tresult = -1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif ((result = download_bundle_list(r, &list_from_bundle,\n+\t\t\t\t\t global_list, depth)))\n+\t\tgoto cleanup;\n+\n+cleanup:\n+\tclear_bundle_list(&list_from_bundle);\n+\treturn result;\n+}\n+\n /**\n * This limits the recursion on fetch_bundle_uri_internal() when following\n * bundle lists.\n */\n static int max_bundle_uri_depth = 4;\n \n+/**\n+ * Recursively download all bundles advertised at the given URI\n+ * to files. If the file is a bundle, then add it to the given\n+ * 'list'. Otherwise, expect a bundle list and recurse on the\n+ * URIs in that list according to the list mode (ANY or ALL).\n+ */\n static int fetch_bundle_uri_internal(struct repository *r,\n-\t\t\t\t const char *uri,\n-\t\t\t\t int depth)\n+\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t int depth,\n+\t\t\t\t struct bundle_list *list)\n {\n \tint result = 0;\n-\tchar *filename;\n+\tstruct remote_bundle_info *bcopy;\n \n \tif (depth >= max_bundle_uri_depth) {\n \t\twarning(_(\"exceeded bundle URI recursion limit (%d)\"),\n@@ -355,36 +450,125 @@ static int fetch_bundle_uri_internal(struct repository *r,\n \t\treturn -1;\n \t}\n \n-\tif (!(filename = find_temp_filename())) {\n+\tif (!bundle->file &&\n+\t !(bundle->file = find_temp_filename())) {\n \t\tresult = -1;\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = copy_uri_to_file(filename, uri))) {\n-\t\twarning(_(\"failed to download bundle from URI '%s'\"), uri);\n+\tif ((result = copy_uri_to_file(bundle->file, bundle->uri))) {\n+\t\twarning(_(\"failed to download bundle from URI '%s'\"), bundle->uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = !is_bundle(filename, 0))) {\n-\t\twarning(_(\"file at URI '%s' is not a bundle\"), uri);\n+\tif ((result = !is_bundle(bundle->file, 1))) {\n+\t\tresult = fetch_bundle_list_in_config_format(\n+\t\t\t\tr, list, bundle, depth);\n+\t\tif (result)\n+\t\t\twarning(_(\"file at URI '%s' is not a bundle or bundle list\"),\n+\t\t\t\tbundle->uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = unbundle_from_file(r, filename))) {\n-\t\twarning(_(\"failed to unbundle bundle from URI '%s'\"), uri);\n-\t\tgoto cleanup;\n-\t}\n+\t/* Copy the bundle and insert it into the global list. */\n+\tCALLOC_ARRAY(bcopy, 1);\n+\tbcopy->id = xstrdup(bundle->id);\n+\tbcopy->file = xstrdup(bundle->file);\n+\thashmap_entry_init(&bcopy->ent, strhash(bcopy->id));\n+\thashmap_add(&list->bundles, &bcopy->ent);\n \n cleanup:\n-\tif (filename)\n-\t\tunlink(filename);\n-\tfree(filename);\n+\tif (result && bundle->file)\n+\t\tunlink(bundle->file);\n \treturn result;\n }\n \n+struct attempt_unbundle_context {\n+\tstruct repository *r;\n+\tint success_count;\n+\tint failure_count;\n+};\n+\n+static int attempt_unbundle(struct remote_bundle_info *info, void *data)\n+{\n+\tstruct attempt_unbundle_context *ctx = data;\n+\n+\tif (info->unbundled || !unbundle_from_file(ctx->r, info->file)) {\n+\t\tctx->success_count++;\n+\t\tinfo->unbundled = 1;\n+\t} else {\n+\t\tctx->failure_count++;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int unbundle_all_bundles(struct repository *r,\n+\t\t\t\tstruct bundle_list *list)\n+{\n+\tint last_success_count = -1;\n+\tstruct attempt_unbundle_context ctx = {\n+\t\t.r = r,\n+\t};\n+\n+\t/*\n+\t * Iterate through all bundles looking for ones that can\n+\t * successfully unbundle. If any succeed, then perhaps another\n+\t * will succeed in the next attempt.\n+\t */\n+\twhile (last_success_count < ctx.success_count) {\n+\t\tlast_success_count = ctx.success_count;\n+\n+\t\tctx.success_count = 0;\n+\t\tctx.failure_count = 0;\n+\t\tfor_all_bundles_in_list(list, attempt_unbundle, &ctx);\n+\t}\n+\n+\tif (ctx.success_count)\n+\t\tgit_config_set_multivar_gently(\"log.excludedecoration\",\n+\t\t\t\t\t\t\"refs/bundle/\",\n+\t\t\t\t\t\t\"refs/bundle/\",\n+\t\t\t\t\t\tCONFIG_FLAGS_FIXED_VALUE |\n+\t\t\t\t\t\tCONFIG_FLAGS_MULTI_REPLACE);\n+\n+\tif (ctx.failure_count)\n+\t\twarning(_(\"failed to unbundle %d bundles\"),\n+\t\t\tctx.failure_count);\n+\n+\treturn 0;\n+}\n+\n+static int unlink_bundle(struct remote_bundle_info *info, void *data)\n+{\n+\tif (info->file)\n+\t\tunlink_or_warn(info->file);\n+\treturn 0;\n+}\n+\n int fetch_bundle_uri(struct repository *r, const char *uri)\n {\n-\treturn fetch_bundle_uri_internal(r, uri, 0);\n+\tint result;\n+\tstruct bundle_list list;\n+\tstruct remote_bundle_info bundle = {\n+\t\t.uri = xstrdup(uri),\n+\t\t.id = xstrdup(\"\"),\n+\t};\n+\n+\tinit_bundle_list(&list);\n+\n+\t/* If a bundle is added to this global list, then it is required. */\n+\tlist.mode = BUNDLE_MODE_ALL;\n+\n+\tif ((result = fetch_bundle_uri_internal(r, &bundle, 0, &list)))\n+\t\tgoto cleanup;\n+\n+\tresult = unbundle_all_bundles(r, &list);\n+\n+cleanup:\n+\tfor_all_bundles_in_list(&list, unlink_bundle, NULL);\n+\tclear_bundle_list(&list);\n+\tclear_remote_bundle_info(&bundle, NULL);\n+\treturn result;\n }\n \n /**\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex bc13d4c9929..4dbc269823c 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -28,6 +28,19 @@ struct remote_bundle_info {\n \t * if there was no table of contents.\n \t */\n \tchar *uri;\n+\n+\t/**\n+\t * If the bundle has been downloaded, then 'file' is a\n+\t * filename storing its contents. Otherwise, 'file' is\n+\t * NULL.\n+\t */\n+\tchar *file;\n+\n+\t/**\n+\t * If the bundle has been unbundled successfully, then\n+\t * this boolean is true.\n+\t */\n+\tunsigned unbundled:1;\n };\n \n #define REMOTE_BUNDLE_INFO_INIT { 0 }\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex ad666a2d28a..592790b49f0 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -41,6 +41,72 @@ test_expect_success 'clone with file:// bundle' '\n \ttest_cmp expect actual\n '\n \n+# To get interesting tests for bundle lists, we need to construct a\n+# somewhat-interesting commit history.\n+#\n+# ---------------- bundle-4\n+#\n+# 4\n+# / \\\n+# ----|---|------- bundle-3\n+# | |\n+# | 3\n+# | |\n+# ----|---|------- bundle-2\n+# | |\n+# 2 |\n+# | |\n+# ----|---|------- bundle-1\n+# \\ /\n+# 1\n+# |\n+# (previous commits)\n+test_expect_success 'construct incremental bundle list' '\n+\t(\n+\t\tcd clone-from &&\n+\t\tgit checkout -b base &&\n+\t\ttest_commit 1 &&\n+\t\tgit checkout -b left &&\n+\t\ttest_commit 2 &&\n+\t\tgit checkout -b right base &&\n+\t\ttest_commit 3 &&\n+\t\tgit checkout -b merge left &&\n+\t\tgit merge right -m \"4\" &&\n+\n+\t\tgit bundle create bundle-1.bundle base &&\n+\t\tgit bundle create bundle-2.bundle base..left &&\n+\t\tgit bundle create bundle-3.bundle base..right &&\n+\t\tgit bundle create bundle-4.bundle merge --not left right\n+\t)\n+'\n+\n+test_expect_success 'clone bundle list (file, no heuristic)' '\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-3.bundle\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-4.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" . clone-list-file &&\n+\tfor oid in $(git -C clone-from for-each-ref --format=\"%(objectname)\")\n+\tdo\n+\t\tgit -C clone-list-file rev-parse $oid || return 1\n+\tdone\n+'\n+\n+\n #########################################################################\n # HTTP tests begin here\n \n@@ -75,6 +141,33 @@ test_expect_success 'clone HTTP bundle' '\n \ttest_config -C clone-http log.excludedecoration refs/bundle/\n '\n \n+test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n+\tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = $HTTPD_URL/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = $HTTPD_URL/bundle-2.bundle\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = $HTTPD_URL/bundle-3.bundle\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = $HTTPD_URL/bundle-4.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" . clone-list-http &&\n+\tfor oid in $(git -C clone-from for-each-ref --format=\"%(objectname)\")\n+\tdo\n+\t\tgit -C clone-list-http rev-parse $oid || return 1\n+\tdone\n+'\n+\n # Do not add tests here unless they use the HTTP server, as they will\n # not run unless the HTTP dependencies exist.\n \n-- \ngitgitgadget\n"},{"id":"462875","messageId":"xmqqleqslauq.fsf@gitster.g","threadId":"58342","inReplyTo":"2ca431e6c377f42d70ea9ce0ac74058fc983ea2c.1662734015.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/9] bundle-uri: short-circuit capability parsing","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-09T17:24:29Z","receivedAt":"2022-09-09T17:24:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> When parsing the capability lines from the 'git remote-https' process,\n> we can stop reading the lines once we notice the 'get' capability.\n>\n> Reported-by: Teng Long \n> Signed-off-by: Derrick Stolee \n> ---\n> bundle-uri.c | 4 +++-\n> 1 file changed, 3 insertions(+), 1 deletion(-)\n>\n> diff --git a/bundle-uri.c b/bundle-uri.c\n> index 4a8cc74ed05..7173ed065e9 100644\n> --- a/bundle-uri.c\n> +++ b/bundle-uri.c\n> @@ -56,8 +56,10 @@ static int download_https_uri_to_file(const char *file, const char *uri)\n> \twhile (!strbuf_getline(&line, child_out)) {\n> \t\tif (!line.len)\n> \t\t\tbreak;\n> -\t\tif (!strcmp(line.buf, \"get\"))\n> +\t\tif (!strcmp(line.buf, \"get\")) {\n> \t\t\tfound_get = 1;\n> +\t\t\tbreak;\n> +\t\t}\n> \t}\n\nHmph, is this safe to do? Who is feeding child_out? Aren't they\nget upset if we do not slurp what they write to us? Are we\nexpecting to read more from them after this part? Aren't we get\nupset if we leave some other stuff when we read from child_out after\nwe saw \"get\"? If we respond to child_in without reading all from\nthem, do we not get into a deadlock?\n\nPerhaps these are all silly questions, but the description above\ndoes not quite answer them.\n\n> \tstrbuf_release(&line);\n"},{"id":"462878","messageId":"xmqqczc4l9di.fsf@gitster.g","threadId":"58342","inReplyTo":"ee6c4b824c276364b77de0b85b54f428c236de00.1662734015.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/9] bundle-uri: use plain string in find_temp_filename()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-09T17:56:25Z","receivedAt":"2022-09-09T17:58:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> The find_temp_filename() method was created in 53a50892be2 (bundle-uri:\n> create basic file-copy logic, 2022-08-09) and uses odb_mkstemp() to\n> create a temporary filename. The odb_mkstemp() method uses a strbuf in\n> its interface, but we do not need to continue carrying a strbuf\n> throughout the bundle URI code.\n\nWhat the patch does is not wrong per-se, but it is unfortunate that,\neven though we accepted a known-to-be-racy approach for expediency\nearlier, the first update to that is not to replace it with a\nnon-racy and safe approach, but make it easier to use, encouraging\nuse of the racy approach and give it an appearance of a clean code\nX-<.\n\n"},{"id":"463213","messageId":"a492902d-7227-3172-9f73-3ddea51cf2c5@github.com","threadId":"58342","inReplyTo":"xmqqczc4l9di.fsf@gitster.g","subject":"Re: [PATCH v2 2/9] bundle-uri: use plain string in find_temp_filename()","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-19T17:54:35Z","receivedAt":"2022-09-19T17:55:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/9/2022 1:56 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> The find_temp_filename() method was created in 53a50892be2 (bundle-uri:\n>> create basic file-copy logic, 2022-08-09) and uses odb_mkstemp() to\n>> create a temporary filename. The odb_mkstemp() method uses a strbuf in\n>> its interface, but we do not need to continue carrying a strbuf\n>> throughout the bundle URI code.\n> \n> What the patch does is not wrong per-se, but it is unfortunate that,\n> even though we accepted a known-to-be-racy approach for expediency\n> earlier, the first update to that is not to replace it with a\n> non-racy and safe approach, but make it easier to use, encouraging\n> use of the racy approach and give it an appearance of a clean code\n> X-<.\n \nHopefully you would be encouraged by future efforts to replace this\ntemporary file name with something deterministic based on the URI so\nwe can restart downloads that were halted, even if the process needs\nto restart. But for now, this change helps us to remove the strbuf\nfrom the remote_bundle_info struct.\n\nThanks,\n-Stolee\n"},{"id":"463214","messageId":"c40247c6-d925-af1a-4ce2-6dcec7fb748d@github.com","threadId":"58342","inReplyTo":"xmqqleqslauq.fsf@gitster.g","subject":"Re: [PATCH v2 1/9] bundle-uri: short-circuit capability parsing","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-19T17:55:32Z","receivedAt":"2022-09-19T17:55:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/9/2022 1:24 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> When parsing the capability lines from the 'git remote-https' process,\n>> we can stop reading the lines once we notice the 'get' capability.\n>>\n>> Reported-by: Teng Long \n>> Signed-off-by: Derrick Stolee \n>> ---\n>> bundle-uri.c | 4 +++-\n>> 1 file changed, 3 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/bundle-uri.c b/bundle-uri.c\n>> index 4a8cc74ed05..7173ed065e9 100644\n>> --- a/bundle-uri.c\n>> +++ b/bundle-uri.c\n>> @@ -56,8 +56,10 @@ static int download_https_uri_to_file(const char *file, const char *uri)\n>> \twhile (!strbuf_getline(&line, child_out)) {\n>> \t\tif (!line.len)\n>> \t\t\tbreak;\n>> -\t\tif (!strcmp(line.buf, \"get\"))\n>> +\t\tif (!strcmp(line.buf, \"get\")) {\n>> \t\t\tfound_get = 1;\n>> +\t\t\tbreak;\n>> +\t\t}\n>> \t}\n> \n> Hmph, is this safe to do? Who is feeding child_out? Aren't they\n> get upset if we do not slurp what they write to us? Are we\n> expecting to read more from them after this part? Aren't we get\n> upset if we leave some other stuff when we read from child_out after\n> we saw \"get\"? If we respond to child_in without reading all from\n> them, do we not get into a deadlock?\n> \n> Perhaps these are all silly questions, but the description above\n> does not quite answer them.\n\nIn my testing, this has not been a problem, but that does not mean\nthat it is safe to do. I'll drop this patch in v3.\n\nThanks,\n-Stolee\n"},{"id":"463216","messageId":"xmqqmtav9qm9.fsf@gitster.g","threadId":"58342","inReplyTo":"a492902d-7227-3172-9f73-3ddea51cf2c5@github.com","subject":"Re: [PATCH v2 2/9] bundle-uri: use plain string in find_temp_filename()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-19T18:16:30Z","receivedAt":"2022-09-19T18:16:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> ... something deterministic based on the URI so\n> we can restart downloads that were halted, even if the process needs\n> to restart. But for now, this change helps us to remove the strbuf\n> from the remote_bundle_info struct.\n\nYay for a bright future ;-)\n\nThanks.\n"},{"id":"463624","messageId":"f61e5940-41d0-39dd-39fa-048acccfe221@github.com","threadId":"58342","inReplyTo":"pull.1333.v2.git.1662734015.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/9] Bundle URIs III: Parse and download from bundle lists","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-26T13:19:57Z","receivedAt":"2022-09-26T14:54:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/9/2022 10:33 AM, Derrick Stolee via GitGitGadget wrote:\n\n> Updates in v2\n> =============\n> \n> Thank you to all of the voices who chimed in on the previous version. I'm\n> sorry it took so long for me to get a new version.\n> \n> * I've done a rather thorough overhaul to minimize how often later patches\n> rewrite portions of earlier patches.\n> \n> * We no longer use a strbuf in struct remote_bundle_info. Instead, use a\n> 'char *' and only in the patch where it is first used.\n> \n> * The config documentation is more clearly indicating that the bundle.*\n> section has no effect in the repository config (at the moment, which will\n> change in the next series).\n> \n> * The bundle.version value is now parsed using git_parse_int().\n> \n> * The config key is now parsed using parse_config_key().\n> \n> * Commit messages clarify more about the context of the change in the\n> bigger picture of the bundle URI effort.\n> \n> * Some printf()s are correctly changed to fprintf()s.\n> \n> * The test helper CLI is unified across the two modes. They both take a\n> filename now.\n> \n> * The count of downloaded bundles is now only updated after a successful\n> download, allowing the \"any\" mode to keep trying after a failure.\n\nIf some of the reviewers from v1 could check that I responded to their\ncomments, then that would be a big help to getting this series moving\nagain.\n\nThanks,\n-Stolee\n"},{"id":"463665","messageId":"xmqqk05qne8h.fsf@gitster.g","threadId":"58342","inReplyTo":"f61e5940-41d0-39dd-39fa-048acccfe221@github.com","subject":"Re: [PATCH v2 0/9] Bundle URIs III: Parse and download from bundle lists","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-26T19:10:38Z","receivedAt":"2022-09-26T19:10:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> If some of the reviewers from v1 could check that I responded to their\n> comments, then that would be a big help to getting this series moving\n> again.\n\nThanks for a ping. Also, if reviewers who missed v1 can take a look\nand give fresh insights, that would also help polishing the series\nfurther.\n\n"},{"id":"463920","messageId":"20220929214906.1437133-1-jonathantanmy@google.com","threadId":"58342","inReplyTo":"70daef66833744fb8b870f431af8201222c7a2ba.1662734015.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/9] bundle-uri: create base key-value pair parsing","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-09-29T21:49:06Z","receivedAt":"2022-09-29T21:49:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n> @@ -65,6 +66,81 @@ int for_all_bundles_in_list(struct bundle_list *list,\n> \treturn 0;\n> }\n> \n> +/**\n> + * Given a key-value pair, update the state of the given bundle list.\n> + * Returns 0 if the key-value pair is understood. Returns 1 if the key\n> + * is not understood or the value is malformed.\n> + */\n> +MAYBE_UNUSED\n> +static int bundle_list_update(const char *key, const char *value,\n> +\t\t\t struct bundle_list *list)\n> +{\n[snip]\n> +\tif (parse_config_key(key, \"bundle\", &subsection, &subsection_len, &subkey))\n> +\t\treturn -1;\n\nThe comment at the top should say -1 instead of 1.\n"},{"id":"463922","messageId":"20220929215808.1438418-1-jonathantanmy@google.com","threadId":"58342","inReplyTo":"a6ab8f7c69925b5dfd95f3e4c83c6ccd4d2dfee3.1662734015.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 9/9] bundle-uri: fetch a list of bundles","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-09-29T21:58:08Z","receivedAt":"2022-09-29T21:58:14Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n> +static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data)\n> +{\n> +\tint res;\n> +\tstruct bundle_list_context *ctx = data;\n> +\n> +\tif (ctx->mode == BUNDLE_MODE_ANY && ctx->count)\n> +\t\treturn 0;\n> +\n> +\tres = fetch_bundle_uri_internal(ctx->r, bundle, ctx->depth + 1, ctx->list);\n> +\n> +\t/*\n> +\t * Only increment count if the download succeeded. If our mode is\n> +\t * BUNDLE_MODE_ANY, then we will want to try other URIs in the\n> +\t * list in case they work instead.\n> +\t */\n> +\tif (!res)\n> +\t\tctx->count++;\n> +\treturn res;\n> +}\n\nSo this returns nonzero if a download fails...\n\n> +static int download_bundle_list(struct repository *r,\n> +\t\t\t\tstruct bundle_list *local_list,\n> +\t\t\t\tstruct bundle_list *global_list,\n> +\t\t\t\tint depth)\n> +{\n> +\tstruct bundle_list_context ctx = {\n> +\t\t.r = r,\n> +\t\t.list = global_list,\n> +\t\t.depth = depth + 1,\n> +\t\t.mode = local_list->mode,\n> +\t};\n> +\n> +\treturn for_all_bundles_in_list(local_list, download_bundle_to_file, &ctx);\n> +}\n\n...and for_all_bundles_in_list does not proceed with the rest of the\nloop if any callback invocation returns nonzero. Don't we need to\ncontinue retrying the others if the mode is ANY?\n\n> +static int attempt_unbundle(struct remote_bundle_info *info, void *data)\n> +{\n> +\tstruct attempt_unbundle_context *ctx = data;\n> +\n> +\tif (info->unbundled || !unbundle_from_file(ctx->r, info->file)) {\n> +\t\tctx->success_count++;\n> +\t\tinfo->unbundled = 1;\n> +\t} else {\n> +\t\tctx->failure_count++;\n> +\t}\n> +\n> +\treturn 0;\n> +}\n\nDo we need to handle the case in which a file is missing but it's\nexpected because the mode is ANY and another file was successfully\ndownloaded?\n\n> +static int unbundle_all_bundles(struct repository *r,\n> +\t\t\t\tstruct bundle_list *list)\n> +{\n> +\tint last_success_count = -1;\n> +\tstruct attempt_unbundle_context ctx = {\n> +\t\t.r = r,\n> +\t};\n> +\n> +\t/*\n> +\t * Iterate through all bundles looking for ones that can\n> +\t * successfully unbundle. If any succeed, then perhaps another\n> +\t * will succeed in the next attempt.\n> +\t */\n> +\twhile (last_success_count < ctx.success_count) {\n> +\t\tlast_success_count = ctx.success_count;\n> +\n> +\t\tctx.success_count = 0;\n> +\t\tctx.failure_count = 0;\n> +\t\tfor_all_bundles_in_list(list, attempt_unbundle, &ctx);\n\nI think it would have been clearer if the invocation to\nfor_all_bundles_in_list were to stop early if a bundle has been\nsuccessfully unbundled, and then you can just run this loop n times,\ninstead of needing to reset the success count each time in order to\ncheck that the latest count is more than the prior one. But this works\ntoo.\n\n[snip tests]\n\nI see that there are ALL tests, but could we have an ANY test as well?\n"},{"id":"463923","messageId":"20220929220031.1438958-1-jonathantanmy@google.com","threadId":"58342","inReplyTo":"xmqqk05qne8h.fsf@gitster.g","subject":"Re: [PATCH v2 0/9] Bundle URIs III: Parse and download from bundle lists","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-09-29T22:00:30Z","receivedAt":"2022-09-29T22:00:49Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Junio C Hamano writes:\n> Derrick Stolee writes:\n> \n> > If some of the reviewers from v1 could check that I responded to their\n> > comments, then that would be a big help to getting this series moving\n> > again.\n\nYes, all my comments from v1 were indeed addressed, thanks.\n\n> Thanks for a ping. Also, if reviewers who missed v1 can take a look\n> and give fresh insights, that would also help polishing the series\n> further.\n\nI didn't miss v1 but I gave some new insights. :-) The patch set looks\ngood except for some commands I had on the last one.\n"},{"id":"463950","messageId":"a91f2868-d65b-3e24-cf18-b483b7807c29@github.com","threadId":"58342","inReplyTo":"20220929215808.1438418-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 9/9] bundle-uri: fetch a list of bundles","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-30T12:49:20Z","receivedAt":"2022-09-30T12:50:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/29/2022 5:58 PM, Jonathan Tan wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n>> +static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data)\n>> +{\n>> +\tint res;\n>> +\tstruct bundle_list_context *ctx = data;\n>> +\n>> +\tif (ctx->mode == BUNDLE_MODE_ANY && ctx->count)\n>> +\t\treturn 0;\n>> +\n>> +\tres = fetch_bundle_uri_internal(ctx->r, bundle, ctx->depth + 1, ctx->list);\n>> +\n>> +\t/*\n>> +\t * Only increment count if the download succeeded. If our mode is\n>> +\t * BUNDLE_MODE_ANY, then we will want to try other URIs in the\n>> +\t * list in case they work instead.\n>> +\t */\n>> +\tif (!res)\n>> +\t\tctx->count++;\n>> +\treturn res;\n>> +}\n> \n> So this returns nonzero if a download fails...\n> \n>> +static int download_bundle_list(struct repository *r,\n>> +\t\t\t\tstruct bundle_list *local_list,\n>> +\t\t\t\tstruct bundle_list *global_list,\n>> +\t\t\t\tint depth)\n>> +{\n>> +\tstruct bundle_list_context ctx = {\n>> +\t\t.r = r,\n>> +\t\t.list = global_list,\n>> +\t\t.depth = depth + 1,\n>> +\t\t.mode = local_list->mode,\n>> +\t};\n>> +\n>> +\treturn for_all_bundles_in_list(local_list, download_bundle_to_file, &ctx);\n>> +}\n> \n> ...and for_all_bundles_in_list does not proceed with the rest of the\n> loop if any callback invocation returns nonzero. Don't we need to\n> continue retrying the others if the mode is ANY?\n\nYou are right! Thanks.\n \n>> +static int attempt_unbundle(struct remote_bundle_info *info, void *data)\n>> +{\n>> +\tstruct attempt_unbundle_context *ctx = data;\n>> +\n>> +\tif (info->unbundled || !unbundle_from_file(ctx->r, info->file)) {\n>> +\t\tctx->success_count++;\n>> +\t\tinfo->unbundled = 1;\n>> +\t} else {\n>> +\t\tctx->failure_count++;\n>> +\t}\n>> +\n>> +\treturn 0;\n>> +}\n> \n> Do we need to handle the case in which a file is missing but it's\n> expected because the mode is ANY and another file was successfully\n> downloaded?\n\nBy \"file is missing\" I think you mean \"we never successfully downloaded\nthat file\" and I agree that we should skip those bundles. I'll add more\ntests for ANY mode to hopefully catch these issues.\n\n>> +static int unbundle_all_bundles(struct repository *r,\n>> +\t\t\t\tstruct bundle_list *list)\n>> +{\n>> +\tint last_success_count = -1;\n>> +\tstruct attempt_unbundle_context ctx = {\n>> +\t\t.r = r,\n>> +\t};\n>> +\n>> +\t/*\n>> +\t * Iterate through all bundles looking for ones that can\n>> +\t * successfully unbundle. If any succeed, then perhaps another\n>> +\t * will succeed in the next attempt.\n>> +\t */\n>> +\twhile (last_success_count < ctx.success_count) {\n>> +\t\tlast_success_count = ctx.success_count;\n>> +\n>> +\t\tctx.success_count = 0;\n>> +\t\tctx.failure_count = 0;\n>> +\t\tfor_all_bundles_in_list(list, attempt_unbundle, &ctx);\n> \n> I think it would have been clearer if the invocation to\n> for_all_bundles_in_list were to stop early if a bundle has been\n> successfully unbundled, and then you can just run this loop n times,\n> instead of needing to reset the success count each time in order to\n> check that the latest count is more than the prior one. But this works\n> too.\n\nIt's a little bit backwards to have the \"terminate early with nonzero\nvalue\" signal \"success\", but it would work. With careful commenting, I\nthink it's doable.\n\n> I see that there are ALL tests, but could we have an ANY test as well?\n\nYes, excellent point. They are absolutely necessary.\n\nThanks,\n-Stolee\n"},{"id":"463951","messageId":"1c3ec9f6-7733-8815-2122-300512628712@github.com","threadId":"58342","inReplyTo":"20220929220031.1438958-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 0/9] Bundle URIs III: Parse and download from bundle lists","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-30T13:21:15Z","receivedAt":"2022-09-30T13:21:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/29/2022 6:00 PM, Jonathan Tan wrote:\n> Junio C Hamano writes:\n>> Derrick Stolee writes:\n>>\n>>> If some of the reviewers from v1 could check that I responded to their\n>>> comments, then that would be a big help to getting this series moving\n>>> again.\n> \n> Yes, all my comments from v1 were indeed addressed, thanks.\n> \n>> Thanks for a ping. Also, if reviewers who missed v1 can take a look\n>> and give fresh insights, that would also help polishing the series\n>> further.\n> \n> I didn't miss v1 but I gave some new insights. :-) The patch set looks\n> good except for some commands I had on the last one.\n\nThanks for taking a detailed look. I've added extra \"any\" mode tests to\nmy local branch in addition to the code changes you recommended. I'll\nplan to send a v3 early next week, giving time for any other review\ncomments to trickle in.\n\nThanks,\n-Stolee\n"},{"id":"464121","messageId":"pull.1333.v3.git.1664886860.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v2.git.1662734015.gitgitgadget@gmail.com","subject":"[PATCH v3 0/9] Bundle URIs III: Parse and download from bundle lists","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T12:34:11Z","receivedAt":"2022-10-04T12:34:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the third series building the bundle URI feature. It is built on top\nof ds/bundle-uri-clone, which introduced 'git clone --bundle-uri=' where is\na URI to a bundle file. This series adds the capability of downloading and\nparsing a bundle list and then downloading the URIs in that list.\n\nThe core functionality of bundle lists is implemented by creating data\nstructures from a list of key-value pairs. These pairs can come from a\nplain-text file in Git config format, but in the future, we will support the\nlist being supplied by packet lines over Git's protocol v2 in the\n'bundle-uri' command (reserved for the next series).\n\nThe patches are organized in this way:\n\n 1. Patches 1-2 are cleanups from the previous part. The first was\n recommended by Teng Long and the second allows us to simplify our bundle\n list data structure slightly.\n\n 2. Patches 3-4 create the bundle list data structures and the logic for\n populating the list from key-value pairs.\n\n 3. Patches 5-6 teach Git to parse \"key=value\" lines to construct a bundle\n list. Add unit tests that ensure this logic constructs lists correctly.\n These patches are adapted from Ævar's RFC [1] and were previously seen\n in my combined RFC [2].\n\n 4. Patch 7 teaches Git to parse Git config files into bundle lists.\n\n 5. Patches 8-9 implement the ability to download a bundle list and\n recursively download the contained bundles (and possibly the bundle\n lists within). This is limited by a constant depth to avoid issues with\n cycles or otherwise incorrectly configured bundle lists.\n\n[1]\nhttps://lore.kernel.org/git/RFC-cover-v2-00.36-00000000000-20220418T165545Z-avarab@gmail.com/\n\n[2]\nhttps://lore.kernel.org/git/pull.1234.git.1653072042.gitgitgadget@gmail.com/\n\nAt the end of this series, users can bootstrap clones using 'git clone\n--bundle-uri= ' where points to a bundle list instead of a single bundle\nfile.\n\nAs outlined in the design document [1], the next steps after this are:\n\n 1. Implement the protocol v2 verb, re-using the bundle list logic from (2).\n Use this to auto-discover bundle URIs during 'git clone' (behind a\n config option). [2]\n 2. Implement the 'creationToken' heuristic, allowing incremental 'git\n fetch' commands to download a bundle list from a configured URI, and\n only download bundles that are new based on the creation token values.\n [3]\n\nI have prepared some of this work as pull requests on my personal fork so\ncurious readers can look ahead to where we are going:\n\n[3]\nhttps://lore.kernel.org/git/pull.1248.v3.git.1658757188.gitgitgadget@gmail.com\n\n[4] https://github.com/derrickstolee/git/pull/21\n\n[5] https://github.com/derrickstolee/git/pull/22\n\n\nUpdates in v3\n=============\n\n * Fixed a comment about a return value of -1.\n * Fixed and tested scenario where early URIs fail in \"any\" mode and Git\n should try the rest of the list.\n * Instead of using 'success_count' and 'failure_count', use the iterator\n return value to terminate the \"all\" mode loop early.\n\n\nUpdates in v2\n=============\n\nThank you to all of the voices who chimed in on the previous version. I'm\nsorry it took so long for me to get a new version.\n\n * I've done a rather thorough overhaul to minimize how often later patches\n rewrite portions of earlier patches.\n\n * We no longer use a strbuf in struct remote_bundle_info. Instead, use a\n 'char *' and only in the patch where it is first used.\n\n * The config documentation is more clearly indicating that the bundle.*\n section has no effect in the repository config (at the moment, which will\n change in the next series).\n\n * The bundle.version value is now parsed using git_parse_int().\n\n * The config key is now parsed using parse_config_key().\n\n * Commit messages clarify more about the context of the change in the\n bigger picture of the bundle URI effort.\n\n * Some printf()s are correctly changed to fprintf()s.\n\n * The test helper CLI is unified across the two modes. They both take a\n filename now.\n\n * The count of downloaded bundles is now only updated after a successful\n download, allowing the \"any\" mode to keep trying after a failure.\n\nThanks,\n\n * Stolee\n\nDerrick Stolee (7):\n bundle-uri: use plain string in find_temp_filename()\n bundle-uri: create bundle_list struct and helpers\n bundle-uri: create base key-value pair parsing\n bundle-uri: parse bundle list in config format\n bundle-uri: limit recursion depth for bundle lists\n bundle-uri: fetch a list of bundles\n bundle-uri: suppress stderr from remote-https\n\nÆvar Arnfjörð Bjarmason (2):\n bundle-uri: create \"key=value\" line parsing\n bundle-uri: unit test \"key=value\" parsing\n\n Documentation/config.txt | 2 +\n Documentation/config/bundle.txt | 24 ++\n Makefile | 1 +\n bundle-uri.c | 449 ++++++++++++++++++++++++++++++--\n bundle-uri.h | 93 +++++++\n config.c | 2 +-\n config.h | 1 +\n t/helper/test-bundle-uri.c | 95 +++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5558-clone-bundle-uri.sh | 143 ++++++++++\n t/t5750-bundle-uri-parse.sh | 171 ++++++++++++\n t/test-lib-functions.sh | 11 +\n 13 files changed, 976 insertions(+), 18 deletions(-)\n create mode 100644 Documentation/config/bundle.txt\n create mode 100644 t/helper/test-bundle-uri.c\n create mode 100755 t/t5750-bundle-uri-parse.sh\n\n\nbase-commit: e21e663cd1942df29979d3e01f7eacb532727bb7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1333%2Fderrickstolee%2Fbundle-redo%2Flist-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1333/derrickstolee/bundle-redo/list-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1333\n\nRange-diff vs v2:\n\n 1: 2ca431e6c37 < -: ----------- bundle-uri: short-circuit capability parsing\n 2: ee6c4b824c2 = 1: 48beccb0f5e bundle-uri: use plain string in find_temp_filename()\n 3: d9812440594 = 2: f0c4457951c bundle-uri: create bundle_list struct and helpers\n 4: 70daef66833 ! 3: 430e01cd2a4 bundle-uri: create base key-value pair parsing\n @@ bundle-uri.c: int for_all_bundles_in_list(struct bundle_list *list,\n \n +/**\n + * Given a key-value pair, update the state of the given bundle list.\n -+ * Returns 0 if the key-value pair is understood. Returns 1 if the key\n ++ * Returns 0 if the key-value pair is understood. Returns -1 if the key\n + * is not understood or the value is malformed.\n + */\n +MAYBE_UNUSED\n 5: 4df3f834029 ! 4: cd915d57f3b bundle-uri: create \"key=value\" line parsing\n @@ Commit message\n \n ## bundle-uri.c ##\n @@ bundle-uri.c: int for_all_bundles_in_list(struct bundle_list *list,\n - * Returns 0 if the key-value pair is understood. Returns 1 if the key\n + * Returns 0 if the key-value pair is understood. Returns -1 if the key\n * is not understood or the value is malformed.\n */\n -MAYBE_UNUSED\n 6: 91c5b58f011 ! 5: 4d8cac67f66 bundle-uri: unit test \"key=value\" parsing\n @@ bundle-uri.c: int for_all_bundles_in_list(struct bundle_list *list,\n +\n /**\n * Given a key-value pair, update the state of the given bundle list.\n - * Returns 0 if the key-value pair is understood. Returns 1 if the key\n + * Returns 0 if the key-value pair is understood. Returns -1 if the key\n \n ## bundle-uri.h ##\n @@ bundle-uri.h: int for_all_bundles_in_list(struct bundle_list *list,\n 7: 1492b8f5ef0 = 6: 0ecae3a44b3 bundle-uri: parse bundle list in config format\n 8: b5d570082fa = 7: 7e6b32313b0 bundle-uri: limit recursion depth for bundle lists\n 9: a6ab8f7c699 ! 8: 46799648b4c bundle-uri: fetch a list of bundles\n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *fi\n +\t */\n +\tif (!res)\n +\t\tctx->count++;\n -+\treturn res;\n ++\n ++\t/*\n ++\t * In BUNDLE_MODE_ANY, we need to continue iterating until we find\n ++\t * a bundle that works, so do not signal a failure here.\n ++\t */\n ++\treturn ctx->mode == BUNDLE_MODE_ANY ? 0 : res;\n +}\n +\n +static int download_bundle_list(struct repository *r,\n @@ bundle-uri.c: static int fetch_bundle_uri_internal(struct repository *r,\n \treturn result;\n }\n \n -+struct attempt_unbundle_context {\n -+\tstruct repository *r;\n -+\tint success_count;\n -+\tint failure_count;\n -+};\n -+\n ++/**\n ++ * This loop iterator breaks the loop with nonzero return code on the\n ++ * first successful unbundling of a bundle.\n ++ */\n +static int attempt_unbundle(struct remote_bundle_info *info, void *data)\n +{\n -+\tstruct attempt_unbundle_context *ctx = data;\n ++\tstruct repository *r = data;\n +\n -+\tif (info->unbundled || !unbundle_from_file(ctx->r, info->file)) {\n -+\t\tctx->success_count++;\n ++\tif (!info->file || info->unbundled)\n ++\t\treturn 0;\n ++\n ++\tif (!unbundle_from_file(r, info->file)) {\n +\t\tinfo->unbundled = 1;\n -+\t} else {\n -+\t\tctx->failure_count++;\n ++\t\treturn 1;\n +\t}\n +\n +\treturn 0;\n @@ bundle-uri.c: static int fetch_bundle_uri_internal(struct repository *r,\n +static int unbundle_all_bundles(struct repository *r,\n +\t\t\t\tstruct bundle_list *list)\n +{\n -+\tint last_success_count = -1;\n -+\tstruct attempt_unbundle_context ctx = {\n -+\t\t.r = r,\n -+\t};\n -+\n +\t/*\n +\t * Iterate through all bundles looking for ones that can\n +\t * successfully unbundle. If any succeed, then perhaps another\n +\t * will succeed in the next attempt.\n ++\t *\n ++\t * Keep in mind that a non-zero result for the loop here means\n ++\t * the loop terminated early on a successful unbundling, which\n ++\t * signals that we can try again.\n +\t */\n -+\twhile (last_success_count < ctx.success_count) {\n -+\t\tlast_success_count = ctx.success_count;\n -+\n -+\t\tctx.success_count = 0;\n -+\t\tctx.failure_count = 0;\n -+\t\tfor_all_bundles_in_list(list, attempt_unbundle, &ctx);\n -+\t}\n -+\n -+\tif (ctx.success_count)\n -+\t\tgit_config_set_multivar_gently(\"log.excludedecoration\",\n -+\t\t\t\t\t\t\"refs/bundle/\",\n -+\t\t\t\t\t\t\"refs/bundle/\",\n -+\t\t\t\t\t\tCONFIG_FLAGS_FIXED_VALUE |\n -+\t\t\t\t\t\tCONFIG_FLAGS_MULTI_REPLACE);\n -+\n -+\tif (ctx.failure_count)\n -+\t\twarning(_(\"failed to unbundle %d bundles\"),\n -+\t\t\tctx.failure_count);\n ++\twhile (for_all_bundles_in_list(list, attempt_unbundle, r)) ;\n +\n +\treturn 0;\n +}\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone with file:// bundle' '\n +\t\turi = file://$(pwd)/clone-from/bundle-4.bundle\n +\tEOF\n +\n -+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" . clone-list-file &&\n -+\tfor oid in $(git -C clone-from for-each-ref --format=\"%(objectname)\")\n -+\tdo\n -+\t\tgit -C clone-list-file rev-parse $oid || return 1\n -+\tdone\n ++\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-list-file &&\n ++\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n ++\tgit -C clone-list-file cat-file --batch-check bundle-list <<-EOF &&\n ++\t[bundle]\n ++\t\tversion = 1\n ++\t\tmode = any\n ++\n ++\t# Does not exist. Should be skipped.\n ++\t[bundle \"bundle-0\"]\n ++\t\turi = $HTTPD_URL/bundle-0.bundle\n ++\n ++\t[bundle \"bundle-1\"]\n ++\t\turi = $HTTPD_URL/bundle-1.bundle\n ++\n ++\t# Does not exist. Should be skipped.\n ++\t[bundle \"bundle-5\"]\n ++\t\turi = $HTTPD_URL/bundle-5.bundle\n ++\tEOF\n ++\n ++\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-any-file &&\n ++\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n ++\tgit -C clone-any-file cat-file --batch-check oids &&\n ++\tgit -C clone-list-http cat-file --batch-check \"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n ++\t[bundle]\n ++\t\tversion = 1\n ++\t\tmode = any\n ++\n ++\t# Does not exist. Should be skipped.\n ++\t[bundle \"bundle-0\"]\n ++\t\turi = $HTTPD_URL/bundle-0.bundle\n ++\n ++\t[bundle \"bundle-1\"]\n ++\t\turi = $HTTPD_URL/bundle-1.bundle\n ++\n ++\t# Does not exist. Should be skipped.\n ++\t[bundle \"bundle-5\"]\n ++\t\turi = $HTTPD_URL/bundle-5.bundle\n ++\tEOF\n ++\n ++\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" clone-from clone-any-http &&\n ++\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n ++\tgit -C clone-any-http cat-file --batch-check 9: d84544859e4 bundle-uri: suppress stderr from remote-https\n\n-- \ngitgitgadget\n"},{"id":"464122","messageId":"48beccb0f5efe6f9247968cd0d4c455c23a24c53.1664886860.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v3.git.1664886860.gitgitgadget@gmail.com","subject":"[PATCH v3 1/9] bundle-uri: use plain string in find_temp_filename()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T12:34:12Z","receivedAt":"2022-10-04T12:34:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe find_temp_filename() method was created in 53a50892be2 (bundle-uri:\ncreate basic file-copy logic, 2022-08-09) and uses odb_mkstemp() to\ncreate a temporary filename. The odb_mkstemp() method uses a strbuf in\nits interface, but we do not need to continue carrying a strbuf\nthroughout the bundle URI code.\n\nConvert the find_temp_filename() method to use a 'char *' and modify its\nonly caller. This makes sense that we don't actually need to modify this\nfilename directly later, so using a strbuf is overkill.\n\nThis change will simplify the data structure for tracking a bundle list\nto use plain strings instead of strbufs.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 28 ++++++++++++++++------------\n 1 file changed, 16 insertions(+), 12 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 4a8cc74ed05..8b2f4e08c9c 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -5,22 +5,23 @@\n #include \"refs.h\"\n #include \"run-command.h\"\n \n-static int find_temp_filename(struct strbuf *name)\n+static char *find_temp_filename(void)\n {\n \tint fd;\n+\tstruct strbuf name = STRBUF_INIT;\n \t/*\n \t * Find a temporary filename that is available. This is briefly\n \t * racy, but unlikely to collide.\n \t */\n-\tfd = odb_mkstemp(name, \"bundles/tmp_uri_XXXXXX\");\n+\tfd = odb_mkstemp(&name, \"bundles/tmp_uri_XXXXXX\");\n \tif (fd < 0) {\n \t\twarning(_(\"failed to create temporary file\"));\n-\t\treturn -1;\n+\t\treturn NULL;\n \t}\n \n \tclose(fd);\n-\tunlink(name->buf);\n-\treturn 0;\n+\tunlink(name.buf);\n+\treturn strbuf_detach(&name, NULL);\n }\n \n static int download_https_uri_to_file(const char *file, const char *uri)\n@@ -141,28 +142,31 @@ static int unbundle_from_file(struct repository *r, const char *file)\n int fetch_bundle_uri(struct repository *r, const char *uri)\n {\n \tint result = 0;\n-\tstruct strbuf filename = STRBUF_INIT;\n+\tchar *filename;\n \n-\tif ((result = find_temp_filename(&filename)))\n+\tif (!(filename = find_temp_filename())) {\n+\t\tresult = -1;\n \t\tgoto cleanup;\n+\t}\n \n-\tif ((result = copy_uri_to_file(filename.buf, uri))) {\n+\tif ((result = copy_uri_to_file(filename, uri))) {\n \t\twarning(_(\"failed to download bundle from URI '%s'\"), uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = !is_bundle(filename.buf, 0))) {\n+\tif ((result = !is_bundle(filename, 0))) {\n \t\twarning(_(\"file at URI '%s' is not a bundle\"), uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = unbundle_from_file(r, filename.buf))) {\n+\tif ((result = unbundle_from_file(r, filename))) {\n \t\twarning(_(\"failed to unbundle bundle from URI '%s'\"), uri);\n \t\tgoto cleanup;\n \t}\n \n cleanup:\n-\tunlink(filename.buf);\n-\tstrbuf_release(&filename);\n+\tif (filename)\n+\t\tunlink(filename);\n+\tfree(filename);\n \treturn result;\n }\n-- \ngitgitgadget\n\n"},{"id":"464123","messageId":"f0c4457951ca5d052d10f401497917cddcbe2aa4.1664886860.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v3.git.1664886860.gitgitgadget@gmail.com","subject":"[PATCH v3 2/9] bundle-uri: create bundle_list struct and helpers","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T12:34:13Z","receivedAt":"2022-10-04T12:34:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt will likely be rare where a user uses a single bundle URI and expects\nthat URI to point to a bundle. Instead, that URI will likely be a list\nof bundles provided in some format. Alternatively, the Git server could\nadvertise a list of bundles.\n\nIn anticipation of these two ways of advertising multiple bundles,\ncreate a data structure that represents such a list. This will be\npopulated using a common API, but for now focus on what data can be\nrepresented.\n\nEach list contains a number of remote_bundle_info structs. These contain\nan 'id' that is used to uniquely identify them in the list, and also a\n'uri' that contains the location of its data. Finally, there is a strbuf\ncontaining the filename used when Git downloads the contents to disk.\n\nThe list itself stores these remote_bundle_info structs in a hashtable\nusing 'id' as the key. The order of the structs in the input is\nconsidered unimportant, but future modifications to the format and these\ndata structures will place ordering possibilities on the set. The list\nalso has a few \"global\" properties, including the version (used when\nparsing the list) and the mode. The mode is one of these two options:\n\n1. BUNDLE_MODE_ALL: all listed URIs are intended to be combined\n together. The client should download all of the advertised data to\n have a complete copy of the data.\n\n2. BUNDLE_MODE_ANY: any one listed item is sufficient to have a complete\n copy of the data. The client can choose arbitrarily from these\n options. In the future, the client may use pings to find the closest\n URI among geodistributed replicas, or use some other heuristic\n information added to the format.\n\nThis API is currently unused, but will soon be expanded with parsing\nlogic and then be consumed by the bundle URI download logic.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n bundle-uri.h | 56 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 116 insertions(+)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 8b2f4e08c9c..f9a8db221bc 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -4,6 +4,66 @@\n #include \"object-store.h\"\n #include \"refs.h\"\n #include \"run-command.h\"\n+#include \"hashmap.h\"\n+#include \"pkt-line.h\"\n+\n+static int compare_bundles(const void *hashmap_cmp_fn_data,\n+\t\t\t const struct hashmap_entry *he1,\n+\t\t\t const struct hashmap_entry *he2,\n+\t\t\t const void *id)\n+{\n+\tconst struct remote_bundle_info *e1 =\n+\t\tcontainer_of(he1, const struct remote_bundle_info, ent);\n+\tconst struct remote_bundle_info *e2 =\n+\t\tcontainer_of(he2, const struct remote_bundle_info, ent);\n+\n+\treturn strcmp(e1->id, id ? (const char *)id : e2->id);\n+}\n+\n+void init_bundle_list(struct bundle_list *list)\n+{\n+\tmemset(list, 0, sizeof(*list));\n+\n+\t/* Implied defaults. */\n+\tlist->mode = BUNDLE_MODE_ALL;\n+\tlist->version = 1;\n+\n+\thashmap_init(&list->bundles, compare_bundles, NULL, 0);\n+}\n+\n+static int clear_remote_bundle_info(struct remote_bundle_info *bundle,\n+\t\t\t\t void *data)\n+{\n+\tFREE_AND_NULL(bundle->id);\n+\tFREE_AND_NULL(bundle->uri);\n+\treturn 0;\n+}\n+\n+void clear_bundle_list(struct bundle_list *list)\n+{\n+\tif (!list)\n+\t\treturn;\n+\n+\tfor_all_bundles_in_list(list, clear_remote_bundle_info, NULL);\n+\thashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent);\n+}\n+\n+int for_all_bundles_in_list(struct bundle_list *list,\n+\t\t\t bundle_iterator iter,\n+\t\t\t void *data)\n+{\n+\tstruct remote_bundle_info *info;\n+\tstruct hashmap_iter i;\n+\n+\thashmap_for_each_entry(&list->bundles, &i, info, ent) {\n+\t\tint result = iter(info, data);\n+\n+\t\tif (result)\n+\t\t\treturn result;\n+\t}\n+\n+\treturn 0;\n+}\n \n static char *find_temp_filename(void)\n {\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 8a152f1ef14..ff7e3fd3fb2 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -1,7 +1,63 @@\n #ifndef BUNDLE_URI_H\n #define BUNDLE_URI_H\n \n+#include \"hashmap.h\"\n+#include \"strbuf.h\"\n+\n struct repository;\n+struct string_list;\n+\n+/**\n+ * The remote_bundle_info struct contains information for a single bundle\n+ * URI. This may be initialized simply by a given URI or might have\n+ * additional metadata associated with it if the bundle was advertised by\n+ * a bundle list.\n+ */\n+struct remote_bundle_info {\n+\tstruct hashmap_entry ent;\n+\n+\t/**\n+\t * The 'id' is a name given to the bundle for reference\n+\t * by other bundle infos.\n+\t */\n+\tchar *id;\n+\n+\t/**\n+\t * The 'uri' is the location of the remote bundle so\n+\t * it can be downloaded on-demand. This will be NULL\n+\t * if there was no table of contents.\n+\t */\n+\tchar *uri;\n+};\n+\n+#define REMOTE_BUNDLE_INFO_INIT { 0 }\n+\n+enum bundle_list_mode {\n+\tBUNDLE_MODE_NONE = 0,\n+\tBUNDLE_MODE_ALL,\n+\tBUNDLE_MODE_ANY\n+};\n+\n+/**\n+ * A bundle_list contains an unordered set of remote_bundle_info structs,\n+ * as well as information about the bundle listing, such as version and\n+ * mode.\n+ */\n+struct bundle_list {\n+\tint version;\n+\tenum bundle_list_mode mode;\n+\tstruct hashmap bundles;\n+};\n+\n+void init_bundle_list(struct bundle_list *list);\n+void clear_bundle_list(struct bundle_list *list);\n+\n+typedef int (*bundle_iterator)(struct remote_bundle_info *bundle,\n+\t\t\t void *data);\n+\n+int for_all_bundles_in_list(struct bundle_list *list,\n+\t\t\t bundle_iterator iter,\n+\t\t\t void *data);\n \n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n-- \ngitgitgadget\n\n"},{"id":"464124","messageId":"cd915d57f3bced015dc154572612d4dd221d1618.1664886860.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v3.git.1664886860.gitgitgadget@gmail.com","subject":"[PATCH v3 4/9] bundle-uri: create \"key=value\" line parsing","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T12:34:15Z","receivedAt":"2022-10-04T12:34:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nWhen advertising a bundle list over Git's protocol v2, we will use\npacket lines. Each line will be of the form \"key=value\" representing a\nbundle list. Connect the API necessary for Git's transport to the\nkey-value pair parsing created in the previous change.\n\nWe are not currently implementing this protocol v2 functionality, but\ninstead preparing to expose this parsing to be unit-testable.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 27 ++++++++++++++++++++++++++-\n bundle-uri.h | 12 ++++++++++++\n 2 files changed, 38 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 0bc59dd9c34..372e6fac5cf 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -71,7 +71,6 @@ int for_all_bundles_in_list(struct bundle_list *list,\n * Returns 0 if the key-value pair is understood. Returns -1 if the key\n * is not understood or the value is malformed.\n */\n-MAYBE_UNUSED\n static int bundle_list_update(const char *key, const char *value,\n \t\t\t struct bundle_list *list)\n {\n@@ -306,3 +305,29 @@ cleanup:\n \tfree(filename);\n \treturn result;\n }\n+\n+/**\n+ * General API for {transport,connect}.c etc.\n+ */\n+int bundle_uri_parse_line(struct bundle_list *list, const char *line)\n+{\n+\tint result;\n+\tconst char *equals;\n+\tstruct strbuf key = STRBUF_INIT;\n+\n+\tif (!strlen(line))\n+\t\treturn error(_(\"bundle-uri: got an empty line\"));\n+\n+\tequals = strchr(line, '=');\n+\n+\tif (!equals)\n+\t\treturn error(_(\"bundle-uri: line is not of the form 'key=value'\"));\n+\tif (line == equals || !*(equals + 1))\n+\t\treturn error(_(\"bundle-uri: line has empty key or value\"));\n+\n+\tstrbuf_add(&key, line, equals - line);\n+\tresult = bundle_list_update(key.buf, equals + 1, list);\n+\tstrbuf_release(&key);\n+\n+\treturn result;\n+}\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex ff7e3fd3fb2..90583461929 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -67,4 +67,16 @@ int for_all_bundles_in_list(struct bundle_list *list,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n+/**\n+ * General API for {transport,connect}.c etc.\n+ */\n+\n+/**\n+ * Parse a \"key=value\" packet line from the bundle-uri verb.\n+ *\n+ * Returns 0 on success and non-zero on error.\n+ */\n+int bundle_uri_parse_line(struct bundle_list *list,\n+\t\t\t const char *line);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"464125","messageId":"430e01cd2a4127a17c4bfc1b0db7209964736de8.1664886860.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v3.git.1664886860.gitgitgadget@gmail.com","subject":"[PATCH v3 3/9] bundle-uri: create base key-value pair parsing","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T12:34:14Z","receivedAt":"2022-10-04T12:34:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThere will be two primary ways to advertise a bundle list: as a list of\npacket lines in Git's protocol v2 and as a config file served from a\nbundle URI. Both of these fundamentally use a list of key-value pairs.\nWe will use the same set of key-value pairs across these formats.\n\nCreate a new bundle_list_update() method that is currently unusued, but\nwill be used in the next change. It inspects each key to see if it is\nunderstood and then applies it to the given bundle_list. Here are the\nkeys that we teach Git to understand:\n\n* bundle.version: This value should be an integer. Git currently\n understands only version 1 and will ignore the list if the version is\n any other value. This version can be increased in the future if we\n need to add new keys that Git should not ignore. We can add new\n \"heuristic\" keys without incrementing the version.\n\n* bundle.mode: This value should be one of \"all\" or \"any\". If this\n mode is not understood, then Git will ignore the list. This mode\n indicates whether Git needs all of the bundle list items to make a\n complete view of the content or if any single item is sufficient.\n\nThe rest of the keys use a bundle identifier \"\" as part of the key\nname. Keys using the same \"\" describe a single bundle list item.\n\n* bundle..uri: This stores the URI of the bundle item. This\n currently is expected to be an absolute URI, but will be relaxed to be\n a relative URI in the future.\n\nWhile parsing, return an error if a URI key is repeated, since we can\nmake that restriction with bundle lists.\n\nMake the git_parse_int() method global so we can parse the integer\nversion value carefully.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 2 +\n Documentation/config/bundle.txt | 24 +++++++++++\n bundle-uri.c | 76 +++++++++++++++++++++++++++++++++\n config.c | 2 +-\n config.h | 1 +\n 5 files changed, 104 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/config/bundle.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex e376d547ce0..4280af6992e 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -387,6 +387,8 @@ include::config/branch.txt[]\n \n include::config/browser.txt[]\n \n+include::config/bundle.txt[]\n+\n include::config/checkout.txt[]\n \n include::config/clean.txt[]\ndiff --git a/Documentation/config/bundle.txt b/Documentation/config/bundle.txt\nnew file mode 100644\nindex 00000000000..daa21eb674a\n--- /dev/null\n+++ b/Documentation/config/bundle.txt\n@@ -0,0 +1,24 @@\n+bundle.*::\n+\tThe `bundle.*` keys may appear in a bundle list file found via the\n+\t`git clone --bundle-uri` option. These keys currently have no effect\n+\tif placed in a repository config file, though this will change in the\n+\tfuture. See link:technical/bundle-uri.html[the bundle URI design\n+\tdocument] for more details.\n+\n+bundle.version::\n+\tThis integer value advertises the version of the bundle list format\n+\tused by the bundle list. Currently, the only accepted value is `1`.\n+\n+bundle.mode::\n+\tThis string value should be either `all` or `any`. This value describes\n+\twhether all of the advertised bundles are required to unbundle a\n+\tcomplete understanding of the bundled information (`all`) or if any one\n+\tof the listed bundle URIs is sufficient (`any`).\n+\n+bundle..*::\n+\tThe `bundle..*` keys are used to describe a single item in the\n+\tbundle list, grouped under `` for identification purposes.\n+\n+bundle..uri::\n+\tThis string value defines the URI by which Git can reach the contents\n+\tof this ``. This URI may be a bundle file or another bundle list.\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex f9a8db221bc..0bc59dd9c34 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -6,6 +6,7 @@\n #include \"run-command.h\"\n #include \"hashmap.h\"\n #include \"pkt-line.h\"\n+#include \"config.h\"\n \n static int compare_bundles(const void *hashmap_cmp_fn_data,\n \t\t\t const struct hashmap_entry *he1,\n@@ -65,6 +66,81 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \treturn 0;\n }\n \n+/**\n+ * Given a key-value pair, update the state of the given bundle list.\n+ * Returns 0 if the key-value pair is understood. Returns -1 if the key\n+ * is not understood or the value is malformed.\n+ */\n+MAYBE_UNUSED\n+static int bundle_list_update(const char *key, const char *value,\n+\t\t\t struct bundle_list *list)\n+{\n+\tstruct strbuf id = STRBUF_INIT;\n+\tstruct remote_bundle_info lookup = REMOTE_BUNDLE_INFO_INIT;\n+\tstruct remote_bundle_info *bundle;\n+\tconst char *subsection, *subkey;\n+\tsize_t subsection_len;\n+\n+\tif (parse_config_key(key, \"bundle\", &subsection, &subsection_len, &subkey))\n+\t\treturn -1;\n+\n+\tif (!subsection_len) {\n+\t\tif (!strcmp(subkey, \"version\")) {\n+\t\t\tint version;\n+\t\t\tif (!git_parse_int(value, &version))\n+\t\t\t\treturn -1;\n+\t\t\tif (version != 1)\n+\t\t\t\treturn -1;\n+\n+\t\t\tlist->version = version;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\tif (!strcmp(subkey, \"mode\")) {\n+\t\t\tif (!strcmp(value, \"all\"))\n+\t\t\t\tlist->mode = BUNDLE_MODE_ALL;\n+\t\t\telse if (!strcmp(value, \"any\"))\n+\t\t\t\tlist->mode = BUNDLE_MODE_ANY;\n+\t\t\telse\n+\t\t\t\treturn -1;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\t/* Ignore other unknown global keys. */\n+\t\treturn 0;\n+\t}\n+\n+\tstrbuf_add(&id, subsection, subsection_len);\n+\n+\t/*\n+\t * Check for an existing bundle with this , or create one\n+\t * if necessary.\n+\t */\n+\tlookup.id = id.buf;\n+\thashmap_entry_init(&lookup.ent, strhash(lookup.id));\n+\tif (!(bundle = hashmap_get_entry(&list->bundles, &lookup, ent, NULL))) {\n+\t\tCALLOC_ARRAY(bundle, 1);\n+\t\tbundle->id = strbuf_detach(&id, NULL);\n+\t\thashmap_entry_init(&bundle->ent, strhash(bundle->id));\n+\t\thashmap_add(&list->bundles, &bundle->ent);\n+\t}\n+\tstrbuf_release(&id);\n+\n+\tif (!strcmp(subkey, \"uri\")) {\n+\t\tif (bundle->uri)\n+\t\t\treturn -1;\n+\t\tbundle->uri = xstrdup(value);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * At this point, we ignore any information that we don't\n+\t * understand, assuming it to be hints for a heuristic the client\n+\t * does not currently understand.\n+\t */\n+\treturn 0;\n+}\n+\n static char *find_temp_filename(void)\n {\n \tint fd;\ndiff --git a/config.c b/config.c\nindex 015bec360f5..e93101249f6 100644\n--- a/config.c\n+++ b/config.c\n@@ -1214,7 +1214,7 @@ static int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max)\n \treturn 0;\n }\n \n-static int git_parse_int(const char *value, int *ret)\n+int git_parse_int(const char *value, int *ret)\n {\n \tintmax_t tmp;\n \tif (!git_parse_signed(value, &tmp, maximum_signed_value_of_type(int)))\ndiff --git a/config.h b/config.h\nindex ca994d77147..ef9eade6414 100644\n--- a/config.h\n+++ b/config.h\n@@ -206,6 +206,7 @@ int config_with_options(config_fn_t fn, void *,\n \n int git_parse_ssize_t(const char *, ssize_t *);\n int git_parse_ulong(const char *, unsigned long *);\n+int git_parse_int(const char *value, int *ret);\n \n /**\n * Same as `git_config_bool`, except that it returns -1 on error rather\n-- \ngitgitgadget\n\n"},{"id":"464126","messageId":"4d8cac67f66fc9c6477efabde795da95fafc51ec.1664886860.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v3.git.1664886860.gitgitgadget@gmail.com","subject":"[PATCH v3 5/9] bundle-uri: unit test \"key=value\" parsing","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T12:34:16Z","receivedAt":"2022-10-04T12:34:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nCreate a new 'test-tool bundle-uri' test helper. This helper will assist\nin testing logic deep in the bundle URI feature.\n\nThis change introduces the 'parse-key-values' subcommand, which parses\nan input file as a list of lines. These are fed into\nbundle_uri_parse_line() to test how we construct a 'struct bundle_list'\nfrom that data. The list is then output to stdout as if the key-value\npairs were a Git config file.\n\nWe use an input file instead of stdin because of a future change to\nparse in config-file format that works better as an input file.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n bundle-uri.c | 33 ++++++++++\n bundle-uri.h | 3 +\n t/helper/test-bundle-uri.c | 70 +++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5750-bundle-uri-parse.sh | 121 ++++++++++++++++++++++++++++++++++++\n t/test-lib-functions.sh | 11 ++++\n 8 files changed, 241 insertions(+)\n create mode 100644 t/helper/test-bundle-uri.c\n create mode 100755 t/t5750-bundle-uri-parse.sh\n\ndiff --git a/Makefile b/Makefile\nindex 7d5f48069ea..7dee0329c49 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -722,6 +722,7 @@ PROGRAMS += $(patsubst %.o,git-%$X,$(PROGRAM_OBJS))\n TEST_BUILTINS_OBJS += test-advise.o\n TEST_BUILTINS_OBJS += test-bitmap.o\n TEST_BUILTINS_OBJS += test-bloom.o\n+TEST_BUILTINS_OBJS += test-bundle-uri.o\n TEST_BUILTINS_OBJS += test-chmtime.o\n TEST_BUILTINS_OBJS += test-config.o\n TEST_BUILTINS_OBJS += test-crontab.o\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 372e6fac5cf..c02e7f62eb1 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -66,6 +66,39 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \treturn 0;\n }\n \n+static int summarize_bundle(struct remote_bundle_info *info, void *data)\n+{\n+\tFILE *fp = data;\n+\tfprintf(fp, \"[bundle \\\"%s\\\"]\\n\", info->id);\n+\tfprintf(fp, \"\\turi = %s\\n\", info->uri);\n+\treturn 0;\n+}\n+\n+void print_bundle_list(FILE *fp, struct bundle_list *list)\n+{\n+\tconst char *mode;\n+\n+\tswitch (list->mode) {\n+\tcase BUNDLE_MODE_ALL:\n+\t\tmode = \"all\";\n+\t\tbreak;\n+\n+\tcase BUNDLE_MODE_ANY:\n+\t\tmode = \"any\";\n+\t\tbreak;\n+\n+\tcase BUNDLE_MODE_NONE:\n+\tdefault:\n+\t\tmode = \"\";\n+\t}\n+\n+\tfprintf(fp, \"[bundle]\\n\");\n+\tfprintf(fp, \"\\tversion = %d\\n\", list->version);\n+\tfprintf(fp, \"\\tmode = %s\\n\", mode);\n+\n+\tfor_all_bundles_in_list(list, summarize_bundle, fp);\n+}\n+\n /**\n * Given a key-value pair, update the state of the given bundle list.\n * Returns 0 if the key-value pair is understood. Returns -1 if the key\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 90583461929..0e56ab2ae5a 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -59,6 +59,9 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \t\t\t bundle_iterator iter,\n \t\t\t void *data);\n \n+struct FILE;\n+void print_bundle_list(FILE *fp, struct bundle_list *list);\n+\n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n * based on that information.\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nnew file mode 100644\nindex 00000000000..0329c56544f\n--- /dev/null\n+++ b/t/helper/test-bundle-uri.c\n@@ -0,0 +1,70 @@\n+#include \"test-tool.h\"\n+#include \"parse-options.h\"\n+#include \"bundle-uri.h\"\n+#include \"strbuf.h\"\n+#include \"string-list.h\"\n+\n+static int cmd__bundle_uri_parse(int argc, const char **argv)\n+{\n+\tconst char *key_value_usage[] = {\n+\t\t\"test-tool bundle-uri parse-key-values \",\n+\t\tNULL\n+\t};\n+\tconst char **usage = key_value_usage;\n+\tstruct option options[] = {\n+\t\tOPT_END(),\n+\t};\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct bundle_list list;\n+\tint err = 0;\n+\tFILE *fp;\n+\n+\targc = parse_options(argc, argv, NULL, options, usage, 0);\n+\tif (argc != 1)\n+\t\tgoto usage;\n+\n+\tinit_bundle_list(&list);\n+\tfp = fopen(argv[0], \"r\");\n+\tif (!fp)\n+\t\tdie(\"failed to open '%s'\", argv[0]);\n+\n+\twhile (strbuf_getline(&sb, fp) != EOF) {\n+\t\tif (bundle_uri_parse_line(&list, sb.buf))\n+\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\t}\n+\tstrbuf_release(&sb);\n+\tfclose(fp);\n+\n+\tprint_bundle_list(stdout, &list);\n+\n+\tclear_bundle_list(&list);\n+\n+\treturn !!err;\n+\n+usage:\n+\tusage_with_options(usage, options);\n+}\n+\n+int cmd__bundle_uri(int argc, const char **argv)\n+{\n+\tconst char *usage[] = {\n+\t\t\"test-tool bundle-uri []\",\n+\t\tNULL\n+\t};\n+\tstruct option options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL, options, usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION |\n+\t\t\t PARSE_OPT_KEEP_ARGV0);\n+\tif (argc == 1)\n+\t\tgoto usage;\n+\n+\tif (!strcmp(argv[1], \"parse-key-values\"))\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1);\n+\terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n+\n+usage:\n+\tusage_with_options(usage, options);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 318fdbab0c3..fbe2d9d8108 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -17,6 +17,7 @@ static struct test_cmd cmds[] = {\n \t{ \"advise\", cmd__advise_if_enabled },\n \t{ \"bitmap\", cmd__bitmap },\n \t{ \"bloom\", cmd__bloom },\n+\t{ \"bundle-uri\", cmd__bundle_uri },\n \t{ \"chmtime\", cmd__chmtime },\n \t{ \"config\", cmd__config },\n \t{ \"crontab\", cmd__crontab },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex bb799271631..b2aa1f39a8f 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -7,6 +7,7 @@\n int cmd__advise_if_enabled(int argc, const char **argv);\n int cmd__bitmap(int argc, const char **argv);\n int cmd__bloom(int argc, const char **argv);\n+int cmd__bundle_uri(int argc, const char **argv);\n int cmd__chmtime(int argc, const char **argv);\n int cmd__config(int argc, const char **argv);\n int cmd__crontab(int argc, const char **argv);\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nnew file mode 100755\nindex 00000000000..fd142a66ad5\n--- /dev/null\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -0,0 +1,121 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri bundle_uri_parse_line()\"\n+\n+TEST_NO_CREATE_REPO=1\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success 'bundle_uri_parse_line() just URIs' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' '\n+\tcat >in <<-\\EOF &&\n+\t=bogus-value\n+\tbogus-key=\n+\tEOF\n+\n+\tcat >err.expect <<-EOF &&\n+\terror: bundle-uri: line has empty key or value\n+\terror: bad line: '\\''=bogus-value'\\''\n+\terror: bundle-uri: line has empty key or value\n+\terror: bad line: '\\''bogus-key='\\''\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty lines' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\terror: bundle-uri: got an empty line\n+\terror: bad line: '\\'''\\''\n+\terror: bundle-uri: got an empty line\n+\terror: bad line: '\\'''\\''\n+\tEOF\n+\n+\t# We fail, but try to continue parsing regardless\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: duplicate lines' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\tbundle.one.uri=https://example.com/bundle-2.bdl\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\terror: bad line: '\\''bundle.one.uri=https://example.com/bundle-2.bdl'\\''\n+\tEOF\n+\n+\t# We fail, but try to continue parsing regardless\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_done\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 6da7273f1d5..3175d665add 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1956,3 +1956,14 @@ test_is_magic_mtime () {\n \trm -f .git/test-mtime-actual\n \treturn $ret\n }\n+\n+# Given two filenames, parse both using 'git config --list --file'\n+# and compare the sorted output of those commands. Useful when\n+# wanting to ignore whitespace differences and sorting concerns.\n+test_cmp_config_output () {\n+\tgit config --list --file=\"$1\" >config-expect &&\n+\tgit config --list --file=\"$2\" >config-actual &&\n+\tsort config-expect >sorted-expect &&\n+\tsort config-actual >sorted-actual &&\n+\ttest_cmp sorted-expect sorted-actual\n+}\n-- \ngitgitgadget\n\n"},{"id":"464127","messageId":"0ecae3a44b360fbb03a52b73536f83c457a6668d.1664886860.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v3.git.1664886860.gitgitgadget@gmail.com","subject":"[PATCH v3 6/9] bundle-uri: parse bundle list in config format","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T12:34:17Z","receivedAt":"2022-10-04T12:34:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a bundle provider wants to operate independently from a Git remote,\nthey want to provide a single, consistent URI that users can use in\ntheir 'git clone --bundle-uri' commands. At this point, the Git client\nexpects that URI to be a single bundle that can be unbundled and used to\nbootstrap the rest of the clone from the Git server. This single bundle\ncannot be re-used to assist with future incremental fetches.\n\nTo allow for the incremental fetch case, teach Git to understand a\nbundle list that could be advertised at an independent bundle URI. Such\na bundle list is likely to be inspected by human readers, even if only\nby the bundle provider creating the list. For this reason, we can take\nour expected \"key=value\" pairs and instead format them using Git config\nformat.\n\nCreate bundle_uri_parse_config_format() to parse a file in config format\nand convert that into a 'struct bundle_list' filled with its\nunderstanding of the contents.\n\nBe careful to use error_action CONFIG_ERROR_ERROR when calling\ngit_config_from_file_with_options() because the default action for\ngit_config_from_file() is to die() on a parsing error. The current\nwarning isn't particularly helpful if it arises to a user, but it will\nbe made more verbose at a higher layer later.\n\nUpdate 'test-tool bundle-uri' to take this config file format as input.\nIt uses a filename instead of stdin because there is no existing way to\nparse a FILE pointer in the config machinery. Using\ngit_config_from_mem() is overly complicated and more likely to introduce\nbugs than this simpler version.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 27 ++++++++++++++++++++\n bundle-uri.h | 9 +++++++\n t/helper/test-bundle-uri.c | 49 +++++++++++++++++++++++++++---------\n t/t5750-bundle-uri-parse.sh | 50 +++++++++++++++++++++++++++++++++++++\n 4 files changed, 123 insertions(+), 12 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex c02e7f62eb1..3d44ec2b1e6 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -173,6 +173,33 @@ static int bundle_list_update(const char *key, const char *value,\n \treturn 0;\n }\n \n+static int config_to_bundle_list(const char *key, const char *value, void *data)\n+{\n+\tstruct bundle_list *list = data;\n+\treturn bundle_list_update(key, value, list);\n+}\n+\n+int bundle_uri_parse_config_format(const char *uri,\n+\t\t\t\t const char *filename,\n+\t\t\t\t struct bundle_list *list)\n+{\n+\tint result;\n+\tstruct config_options opts = {\n+\t\t.error_action = CONFIG_ERROR_ERROR,\n+\t};\n+\n+\tresult = git_config_from_file_with_options(config_to_bundle_list,\n+\t\t\t\t\t\t filename, list,\n+\t\t\t\t\t\t &opts);\n+\n+\tif (!result && list->mode == BUNDLE_MODE_NONE) {\n+\t\twarning(_(\"bundle list at '%s' has no mode\"), uri);\n+\t\tresult = 1;\n+\t}\n+\n+\treturn result;\n+}\n+\n static char *find_temp_filename(void)\n {\n \tint fd;\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 0e56ab2ae5a..bc13d4c9929 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -62,6 +62,15 @@ int for_all_bundles_in_list(struct bundle_list *list,\n struct FILE;\n void print_bundle_list(FILE *fp, struct bundle_list *list);\n \n+/**\n+ * A bundle URI may point to a bundle list where the key=value\n+ * pairs are provided in config file format. This method is\n+ * exposed publicly for testing purposes.\n+ */\n+int bundle_uri_parse_config_format(const char *uri,\n+\t\t\t\t const char *filename,\n+\t\t\t\t struct bundle_list *list);\n+\n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n * based on that information.\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex 0329c56544f..25afd393428 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -4,12 +4,21 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n \n-static int cmd__bundle_uri_parse(int argc, const char **argv)\n+enum input_mode {\n+\tKEY_VALUE_PAIRS,\n+\tCONFIG_FILE,\n+};\n+\n+static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mode)\n {\n \tconst char *key_value_usage[] = {\n \t\t\"test-tool bundle-uri parse-key-values \",\n \t\tNULL\n \t};\n+\tconst char *config_usage[] = {\n+\t\t\"test-tool bundle-uri parse-config \",\n+\t\tNULL\n+\t};\n \tconst char **usage = key_value_usage;\n \tstruct option options[] = {\n \t\tOPT_END(),\n@@ -19,21 +28,35 @@ static int cmd__bundle_uri_parse(int argc, const char **argv)\n \tint err = 0;\n \tFILE *fp;\n \n-\targc = parse_options(argc, argv, NULL, options, usage, 0);\n-\tif (argc != 1)\n-\t\tgoto usage;\n+\tif (mode == CONFIG_FILE)\n+\t\tusage = config_usage;\n+\n+\targc = parse_options(argc, argv, NULL, options, usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tinit_bundle_list(&list);\n-\tfp = fopen(argv[0], \"r\");\n-\tif (!fp)\n-\t\tdie(\"failed to open '%s'\", argv[0]);\n \n-\twhile (strbuf_getline(&sb, fp) != EOF) {\n-\t\tif (bundle_uri_parse_line(&list, sb.buf))\n-\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\tswitch (mode) {\n+\tcase KEY_VALUE_PAIRS:\n+\t\tif (argc != 1)\n+\t\t\tgoto usage;\n+\t\tfp = fopen(argv[0], \"r\");\n+\t\tif (!fp)\n+\t\t\tdie(\"failed to open '%s'\", argv[0]);\n+\t\twhile (strbuf_getline(&sb, fp) != EOF) {\n+\t\t\tif (bundle_uri_parse_line(&list, sb.buf))\n+\t\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\t\t}\n+\t\tfclose(fp);\n+\t\tbreak;\n+\n+\tcase CONFIG_FILE:\n+\t\tif (argc != 1)\n+\t\t\tgoto usage;\n+\t\terr = bundle_uri_parse_config_format(\"\", argv[0], &list);\n+\t\tbreak;\n \t}\n \tstrbuf_release(&sb);\n-\tfclose(fp);\n \n \tprint_bundle_list(stdout, &list);\n \n@@ -62,7 +85,9 @@ int cmd__bundle_uri(int argc, const char **argv)\n \t\tgoto usage;\n \n \tif (!strcmp(argv[1], \"parse-key-values\"))\n-\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1);\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS);\n+\tif (!strcmp(argv[1], \"parse-config\"))\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE);\n \terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n \n usage:\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex fd142a66ad5..c2fe3f9c5a5 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -118,4 +118,54 @@ test_expect_success 'bundle_uri_parse_line() parsing edge cases: duplicate lines\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format: just URIs' '\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config expect >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'parse config format edge cases: empty key or value' '\n+\tcat >in1 <<-\\EOF &&\n+\t= bogus-value\n+\tEOF\n+\n+\tcat >err1 <<-EOF &&\n+\terror: bad config line 1 in file in1\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-config in1 >actual 2>err &&\n+\ttest_cmp err1 err &&\n+\ttest_cmp_config_output expect actual &&\n+\n+\tcat >in2 <<-\\EOF &&\n+\tbogus-key =\n+\tEOF\n+\n+\tcat >err2 <<-EOF &&\n+\terror: bad config line 1 in file in2\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-config in2 >actual 2>err &&\n+\ttest_cmp err2 err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"464128","messageId":"d84544859e4a0e40b3692cf2e838f38734589114.1664886861.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v3.git.1664886860.gitgitgadget@gmail.com","subject":"[PATCH v3 9/9] bundle-uri: suppress stderr from remote-https","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T12:34:20Z","receivedAt":"2022-10-04T12:34:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen downloading bundles from a git-remote-https subprocess, the bundle\nURI logic wants to be opportunistic and download as much as possible and\nwork with what did succeed. This is particularly important in the \"any\"\nmode, where any single bundle success will work.\n\nIf the URI is not available, the git-remote-https process will die()\nwith a \"fatal:\" error message, even though that error is not actually\nfatal to the super process. Since stderr is passed through, it looks\nlike a fatal error to the user.\n\nSuppress stderr to avoid these errors from bubbling to the surface. The\nbundle URI API adds its own warning() messages on these failures.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 1 +\n t/t5558-clone-bundle-uri.sh | 12 ++++++++++--\n 2 files changed, 11 insertions(+), 2 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex aaa1848044a..92af0eae224 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -230,6 +230,7 @@ static int download_https_uri_to_file(const char *file, const char *uri)\n \tint found_get = 0;\n \n \tstrvec_pushl(&cp.args, \"git-remote-https\", uri, NULL);\n+\tcp.err = -1;\n \tcp.in = -1;\n \tcp.out = -1;\n \ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 9690f19386f..a0ef0588e21 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -122,7 +122,11 @@ test_expect_success 'clone bundle list (file, any mode)' '\n \t\turi = $HTTPD_URL/bundle-5.bundle\n \tEOF\n \n-\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-any-file &&\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from clone-any-file 2>err &&\n+\t! grep \"fatal\" err &&\n+\tgrep \"warning: failed to download bundle from URI\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-any-file cat-file --batch-check err &&\n+\t! grep \"fatal\" err &&\n+\tgrep \"warning: failed to download bundle from URI\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-any-http cat-file --batch-check \n\nThe next change will start allowing us to parse bundle lists that are\ndownloaded from a provided bundle URI. Those lists might point to other\nlists, which could proceed to an arbitrary depth (and even create\ncycles). Restructure fetch_bundle_uri() to have an internal version that\nhas a recursion depth. Compare that to a new max_bundle_uri_depth\nconstant that is twice as high as we expect this depth to be for any\nlegitimate use of bundle list linking.\n\nWe can consider making max_bundle_uri_depth a configurable value if\nthere is demonstrated value in the future.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 21 ++++++++++++++++++++-\n 1 file changed, 20 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 3d44ec2b1e6..8a7c11c6393 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -334,11 +334,25 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \treturn result;\n }\n \n-int fetch_bundle_uri(struct repository *r, const char *uri)\n+/**\n+ * This limits the recursion on fetch_bundle_uri_internal() when following\n+ * bundle lists.\n+ */\n+static int max_bundle_uri_depth = 4;\n+\n+static int fetch_bundle_uri_internal(struct repository *r,\n+\t\t\t\t const char *uri,\n+\t\t\t\t int depth)\n {\n \tint result = 0;\n \tchar *filename;\n \n+\tif (depth >= max_bundle_uri_depth) {\n+\t\twarning(_(\"exceeded bundle URI recursion limit (%d)\"),\n+\t\t\tmax_bundle_uri_depth);\n+\t\treturn -1;\n+\t}\n+\n \tif (!(filename = find_temp_filename())) {\n \t\tresult = -1;\n \t\tgoto cleanup;\n@@ -366,6 +380,11 @@ cleanup:\n \treturn result;\n }\n \n+int fetch_bundle_uri(struct repository *r, const char *uri)\n+{\n+\treturn fetch_bundle_uri_internal(r, uri, 0);\n+}\n+\n /**\n * General API for {transport,connect}.c etc.\n */\n-- \ngitgitgadget\n\n"},{"id":"464130","messageId":"46799648b4ce16b0cbacc307443a343994ed064f.1664886861.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v3.git.1664886860.gitgitgadget@gmail.com","subject":"[PATCH v3 8/9] bundle-uri: fetch a list of bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T12:34:19Z","receivedAt":"2022-10-04T12:34:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen the content at a given bundle URI is not understood as a bundle\n(based on inspecting the initial content), then Git currently gives up\nand ignores that content. Independent bundle providers may want to split\nup the bundle content into multiple bundles, but still make them\navailable from a single URI.\n\nTeach Git to attempt parsing the bundle URI content as a Git config file\nproviding the key=value pairs for a bundle list. Git then looks at the\nmode of the list to see if ANY single bundle is sufficient or if ALL\nbundles are required. The content at the selected URIs are downloaded\nand the content is inspected again, creating a recursive process.\n\nTo guard the recursion against malformed or malicious content, limit the\nrecursion depth to a reasonable four for now. This can be converted to a\nconfigured value in the future if necessary. The value of four is twice\nas high as expected to be useful (a bundle list is unlikely to point to\nmore bundle lists).\n\nTo test this scenario, create an interesting bundle topology where three\nincremental bundles are built on top of a single full bundle. By using a\nmerge commit, the two middle bundles are \"independent\" in that they do\nnot require each other in order to unbundle themselves. They each only\nneed the base bundle. The bundle containing the merge commit requires\nboth of the middle bundles, though. This leads to some interesting\ndecisions when unbundling, especially when we later implement heuristics\nthat promote downloading bundles until the prerequisite commits are\nsatisfied.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 202 +++++++++++++++++++++++++++++++++---\n bundle-uri.h | 13 +++\n t/t5558-clone-bundle-uri.sh | 135 ++++++++++++++++++++++++\n 3 files changed, 334 insertions(+), 16 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 8a7c11c6393..aaa1848044a 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -37,6 +37,8 @@ static int clear_remote_bundle_info(struct remote_bundle_info *bundle,\n {\n \tFREE_AND_NULL(bundle->id);\n \tFREE_AND_NULL(bundle->uri);\n+\tFREE_AND_NULL(bundle->file);\n+\tbundle->unbundled = 0;\n \treturn 0;\n }\n \n@@ -334,18 +336,116 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \treturn result;\n }\n \n+struct bundle_list_context {\n+\tstruct repository *r;\n+\tstruct bundle_list *list;\n+\tenum bundle_list_mode mode;\n+\tint count;\n+\tint depth;\n+};\n+\n+/*\n+ * This early definition is necessary because we use indirect recursion:\n+ *\n+ * While iterating through a bundle list that was downloaded as part\n+ * of fetch_bundle_uri_internal(), iterator methods eventually call it\n+ * again, but with depth + 1.\n+ */\n+static int fetch_bundle_uri_internal(struct repository *r,\n+\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t int depth,\n+\t\t\t\t struct bundle_list *list);\n+\n+static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data)\n+{\n+\tint res;\n+\tstruct bundle_list_context *ctx = data;\n+\n+\tif (ctx->mode == BUNDLE_MODE_ANY && ctx->count)\n+\t\treturn 0;\n+\n+\tres = fetch_bundle_uri_internal(ctx->r, bundle, ctx->depth + 1, ctx->list);\n+\n+\t/*\n+\t * Only increment count if the download succeeded. If our mode is\n+\t * BUNDLE_MODE_ANY, then we will want to try other URIs in the\n+\t * list in case they work instead.\n+\t */\n+\tif (!res)\n+\t\tctx->count++;\n+\n+\t/*\n+\t * In BUNDLE_MODE_ANY, we need to continue iterating until we find\n+\t * a bundle that works, so do not signal a failure here.\n+\t */\n+\treturn ctx->mode == BUNDLE_MODE_ANY ? 0 : res;\n+}\n+\n+static int download_bundle_list(struct repository *r,\n+\t\t\t\tstruct bundle_list *local_list,\n+\t\t\t\tstruct bundle_list *global_list,\n+\t\t\t\tint depth)\n+{\n+\tstruct bundle_list_context ctx = {\n+\t\t.r = r,\n+\t\t.list = global_list,\n+\t\t.depth = depth + 1,\n+\t\t.mode = local_list->mode,\n+\t};\n+\n+\treturn for_all_bundles_in_list(local_list, download_bundle_to_file, &ctx);\n+}\n+\n+static int fetch_bundle_list_in_config_format(struct repository *r,\n+\t\t\t\t\t struct bundle_list *global_list,\n+\t\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t\t int depth)\n+{\n+\tint result;\n+\tstruct bundle_list list_from_bundle;\n+\n+\tinit_bundle_list(&list_from_bundle);\n+\n+\tif ((result = bundle_uri_parse_config_format(bundle->uri,\n+\t\t\t\t\t\t bundle->file,\n+\t\t\t\t\t\t &list_from_bundle)))\n+\t\tgoto cleanup;\n+\n+\tif (list_from_bundle.mode == BUNDLE_MODE_NONE) {\n+\t\twarning(_(\"unrecognized bundle mode from URI '%s'\"),\n+\t\t\tbundle->uri);\n+\t\tresult = -1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif ((result = download_bundle_list(r, &list_from_bundle,\n+\t\t\t\t\t global_list, depth)))\n+\t\tgoto cleanup;\n+\n+cleanup:\n+\tclear_bundle_list(&list_from_bundle);\n+\treturn result;\n+}\n+\n /**\n * This limits the recursion on fetch_bundle_uri_internal() when following\n * bundle lists.\n */\n static int max_bundle_uri_depth = 4;\n \n+/**\n+ * Recursively download all bundles advertised at the given URI\n+ * to files. If the file is a bundle, then add it to the given\n+ * 'list'. Otherwise, expect a bundle list and recurse on the\n+ * URIs in that list according to the list mode (ANY or ALL).\n+ */\n static int fetch_bundle_uri_internal(struct repository *r,\n-\t\t\t\t const char *uri,\n-\t\t\t\t int depth)\n+\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t int depth,\n+\t\t\t\t struct bundle_list *list)\n {\n \tint result = 0;\n-\tchar *filename;\n+\tstruct remote_bundle_info *bcopy;\n \n \tif (depth >= max_bundle_uri_depth) {\n \t\twarning(_(\"exceeded bundle URI recursion limit (%d)\"),\n@@ -353,36 +453,106 @@ static int fetch_bundle_uri_internal(struct repository *r,\n \t\treturn -1;\n \t}\n \n-\tif (!(filename = find_temp_filename())) {\n+\tif (!bundle->file &&\n+\t !(bundle->file = find_temp_filename())) {\n \t\tresult = -1;\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = copy_uri_to_file(filename, uri))) {\n-\t\twarning(_(\"failed to download bundle from URI '%s'\"), uri);\n+\tif ((result = copy_uri_to_file(bundle->file, bundle->uri))) {\n+\t\twarning(_(\"failed to download bundle from URI '%s'\"), bundle->uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = !is_bundle(filename, 0))) {\n-\t\twarning(_(\"file at URI '%s' is not a bundle\"), uri);\n+\tif ((result = !is_bundle(bundle->file, 1))) {\n+\t\tresult = fetch_bundle_list_in_config_format(\n+\t\t\t\tr, list, bundle, depth);\n+\t\tif (result)\n+\t\t\twarning(_(\"file at URI '%s' is not a bundle or bundle list\"),\n+\t\t\t\tbundle->uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = unbundle_from_file(r, filename))) {\n-\t\twarning(_(\"failed to unbundle bundle from URI '%s'\"), uri);\n-\t\tgoto cleanup;\n-\t}\n+\t/* Copy the bundle and insert it into the global list. */\n+\tCALLOC_ARRAY(bcopy, 1);\n+\tbcopy->id = xstrdup(bundle->id);\n+\tbcopy->file = xstrdup(bundle->file);\n+\thashmap_entry_init(&bcopy->ent, strhash(bcopy->id));\n+\thashmap_add(&list->bundles, &bcopy->ent);\n \n cleanup:\n-\tif (filename)\n-\t\tunlink(filename);\n-\tfree(filename);\n+\tif (result && bundle->file)\n+\t\tunlink(bundle->file);\n \treturn result;\n }\n \n+/**\n+ * This loop iterator breaks the loop with nonzero return code on the\n+ * first successful unbundling of a bundle.\n+ */\n+static int attempt_unbundle(struct remote_bundle_info *info, void *data)\n+{\n+\tstruct repository *r = data;\n+\n+\tif (!info->file || info->unbundled)\n+\t\treturn 0;\n+\n+\tif (!unbundle_from_file(r, info->file)) {\n+\t\tinfo->unbundled = 1;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int unbundle_all_bundles(struct repository *r,\n+\t\t\t\tstruct bundle_list *list)\n+{\n+\t/*\n+\t * Iterate through all bundles looking for ones that can\n+\t * successfully unbundle. If any succeed, then perhaps another\n+\t * will succeed in the next attempt.\n+\t *\n+\t * Keep in mind that a non-zero result for the loop here means\n+\t * the loop terminated early on a successful unbundling, which\n+\t * signals that we can try again.\n+\t */\n+\twhile (for_all_bundles_in_list(list, attempt_unbundle, r)) ;\n+\n+\treturn 0;\n+}\n+\n+static int unlink_bundle(struct remote_bundle_info *info, void *data)\n+{\n+\tif (info->file)\n+\t\tunlink_or_warn(info->file);\n+\treturn 0;\n+}\n+\n int fetch_bundle_uri(struct repository *r, const char *uri)\n {\n-\treturn fetch_bundle_uri_internal(r, uri, 0);\n+\tint result;\n+\tstruct bundle_list list;\n+\tstruct remote_bundle_info bundle = {\n+\t\t.uri = xstrdup(uri),\n+\t\t.id = xstrdup(\"\"),\n+\t};\n+\n+\tinit_bundle_list(&list);\n+\n+\t/* If a bundle is added to this global list, then it is required. */\n+\tlist.mode = BUNDLE_MODE_ALL;\n+\n+\tif ((result = fetch_bundle_uri_internal(r, &bundle, 0, &list)))\n+\t\tgoto cleanup;\n+\n+\tresult = unbundle_all_bundles(r, &list);\n+\n+cleanup:\n+\tfor_all_bundles_in_list(&list, unlink_bundle, NULL);\n+\tclear_bundle_list(&list);\n+\tclear_remote_bundle_info(&bundle, NULL);\n+\treturn result;\n }\n \n /**\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex bc13d4c9929..4dbc269823c 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -28,6 +28,19 @@ struct remote_bundle_info {\n \t * if there was no table of contents.\n \t */\n \tchar *uri;\n+\n+\t/**\n+\t * If the bundle has been downloaded, then 'file' is a\n+\t * filename storing its contents. Otherwise, 'file' is\n+\t * NULL.\n+\t */\n+\tchar *file;\n+\n+\t/**\n+\t * If the bundle has been unbundled successfully, then\n+\t * this boolean is true.\n+\t */\n+\tunsigned unbundled:1;\n };\n \n #define REMOTE_BUNDLE_INFO_INIT { 0 }\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex ad666a2d28a..9690f19386f 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -41,6 +41,92 @@ test_expect_success 'clone with file:// bundle' '\n \ttest_cmp expect actual\n '\n \n+# To get interesting tests for bundle lists, we need to construct a\n+# somewhat-interesting commit history.\n+#\n+# ---------------- bundle-4\n+#\n+# 4\n+# / \\\n+# ----|---|------- bundle-3\n+# | |\n+# | 3\n+# | |\n+# ----|---|------- bundle-2\n+# | |\n+# 2 |\n+# | |\n+# ----|---|------- bundle-1\n+# \\ /\n+# 1\n+# |\n+# (previous commits)\n+test_expect_success 'construct incremental bundle list' '\n+\t(\n+\t\tcd clone-from &&\n+\t\tgit checkout -b base &&\n+\t\ttest_commit 1 &&\n+\t\tgit checkout -b left &&\n+\t\ttest_commit 2 &&\n+\t\tgit checkout -b right base &&\n+\t\ttest_commit 3 &&\n+\t\tgit checkout -b merge left &&\n+\t\tgit merge right -m \"4\" &&\n+\n+\t\tgit bundle create bundle-1.bundle base &&\n+\t\tgit bundle create bundle-2.bundle base..left &&\n+\t\tgit bundle create bundle-3.bundle base..right &&\n+\t\tgit bundle create bundle-4.bundle merge --not left right\n+\t)\n+'\n+\n+test_expect_success 'clone bundle list (file, no heuristic)' '\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-3.bundle\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-4.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-list-file &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-list-file cat-file --batch-check bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = any\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-0\"]\n+\t\turi = $HTTPD_URL/bundle-0.bundle\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = $HTTPD_URL/bundle-1.bundle\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-5\"]\n+\t\turi = $HTTPD_URL/bundle-5.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-any-file &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-any-file cat-file --batch-check \"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = $HTTPD_URL/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = $HTTPD_URL/bundle-2.bundle\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = $HTTPD_URL/bundle-3.bundle\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = $HTTPD_URL/bundle-4.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" clone-from clone-list-http &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-list-http cat-file --batch-check \"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = any\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-0\"]\n+\t\turi = $HTTPD_URL/bundle-0.bundle\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = $HTTPD_URL/bundle-1.bundle\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-5\"]\n+\t\turi = $HTTPD_URL/bundle-5.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" clone-from clone-any-http &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-any-http cat-file --batch-check writes:\n> +static int unbundle_all_bundles(struct repository *r,\n> +\t\t\t\tstruct bundle_list *list)\n> +{\n> +\t/*\n> +\t * Iterate through all bundles looking for ones that can\n> +\t * successfully unbundle. If any succeed, then perhaps another\n> +\t * will succeed in the next attempt.\n> +\t *\n> +\t * Keep in mind that a non-zero result for the loop here means\n> +\t * the loop terminated early on a successful unbundling, which\n> +\t * signals that we can try again.\n> +\t */\n> +\twhile (for_all_bundles_in_list(list, attempt_unbundle, r)) ;\n> +\n> +\treturn 0;\n> +}\n\nThis function always returns 0 regardless of how many successful \niterations there were: we would need the number to be equal to the \nnumber of bundles in the list if ALL, and 1 if ANY. \n\nWhich brings up the question...we probably need a test for when the \nunbundling is unsuccessful. \n\nOther than that, everything looks good, including the removal of one \npatch and the addition of the \"bundle-uri: suppress stderr from \nremote-https\" patch.\n"},{"id":"464373","messageId":"baf088d7-72f7-a811-888e-81526957ca4b@github.com","threadId":"58342","inReplyTo":"20221004214455.3383714-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 8/9] bundle-uri: fetch a list of bundles","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-10-07T13:29:20Z","receivedAt":"2022-10-07T13:29:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/4/22 5:44 PM, Jonathan Tan wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n>> +static int unbundle_all_bundles(struct repository *r,\n>> +\t\t\t\tstruct bundle_list *list)\n>> +{\n>> +\t/*\n>> +\t * Iterate through all bundles looking for ones that can\n>> +\t * successfully unbundle. If any succeed, then perhaps another\n>> +\t * will succeed in the next attempt.\n>> +\t *\n>> +\t * Keep in mind that a non-zero result for the loop here means\n>> +\t * the loop terminated early on a successful unbundling, which\n>> +\t * signals that we can try again.\n>> +\t */\n>> +\twhile (for_all_bundles_in_list(list, attempt_unbundle, r)) ;\n>> +\n>> +\treturn 0;\n>> +}\n> \n> This function always returns 0 regardless of how many successful \n> iterations there were: we would need the number to be equal to the \n> number of bundles in the list if ALL, and 1 if ANY. \n\nThe ALL mode is a bit more permissive than requiring literally\nevery bundle: if some fail to download or apply, then we continue\nwith whatever we were able to unbundle. The ALL mode indicates that\nthe bundles build on each other, so the client should download as\nmany as possible. By contrast, ANY indicates that they are independent\nso the client should stop after the first successful download.\n\nWe could still find a way to indicate how many bundles were downloaded\nin the return of this method, but we don't want to have additional\nwarnings based on that return value.\n\n> Which brings up the question...we probably need a test for when the \n> unbundling is unsuccessful. \n\nI will add more failure scenarios, including no successful downloads\nor only a partial success in ALL mode.\n \n> Other than that, everything looks good, including the removal of one \n> patch and the addition of the \"bundle-uri: suppress stderr from \n> remote-https\" patch.\n\nThanks!\n\n-Stolee\n"},{"id":"464482","messageId":"pull.1333.v4.git.1665417859.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v3.git.1664886860.gitgitgadget@gmail.com","subject":"[PATCH v4 00/11] Bundle URIs III: Parse and download from bundle lists","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-10T16:04:08Z","receivedAt":"2022-10-10T16:04:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the third series building the bundle URI feature. It is built on top\nof ds/bundle-uri-clone, which introduced 'git clone --bundle-uri=' where is\na URI to a bundle file. This series adds the capability of downloading and\nparsing a bundle list and then downloading the URIs in that list.\n\nThe core functionality of bundle lists is implemented by creating data\nstructures from a list of key-value pairs. These pairs can come from a\nplain-text file in Git config format, but in the future, we will support the\nlist being supplied by packet lines over Git's protocol v2 in the\n'bundle-uri' command (reserved for the next series).\n\nThe patches are organized in this way (updated for v4):\n\n 1. Patch 1 is a cleanup from the previous part. This allows us to simplify\n our bundle list data structure slightly.\n\n 2. Patches 2-3 create the bundle list data structures and the logic for\n populating the list from key-value pairs.\n\n 3. Patches 4-5 teach Git to parse \"key=value\" lines to construct a bundle\n list. Add unit tests that ensure this logic constructs lists correctly.\n These patches are adapted from Ævar's RFC [1] and were previously seen\n in my combined RFC [2].\n\n 4. Patch 6 teaches Git to parse Git config files into bundle lists.\n\n 5. Patches 7-9 implement the ability to download a bundle list and\n recursively download the contained bundles (and possibly the bundle\n lists within). This is limited by a constant depth to avoid issues with\n cycles or otherwise incorrectly configured bundle lists. We also need to\n be careful when verifying the bundles due to ref caches, so some flags\n are added to unbundle() and verify_bundle().\n\n 6. Patches 10-11 suppress unhelpful warnings from user visibility.\n\n[1]\nhttps://lore.kernel.org/git/RFC-cover-v2-00.36-00000000000-20220418T165545Z-avarab@gmail.com/\n\n[2]\nhttps://lore.kernel.org/git/pull.1234.git.1653072042.gitgitgadget@gmail.com/\n\nAt the end of this series, users can bootstrap clones using 'git clone\n--bundle-uri= ' where points to a bundle list instead of a single bundle\nfile.\n\nAs outlined in the design document [1], the next steps after this are:\n\n 1. Implement the protocol v2 verb, re-using the bundle list logic from (2).\n Use this to auto-discover bundle URIs during 'git clone' (behind a\n config option). [2]\n 2. Implement the 'creationToken' heuristic, allowing incremental 'git\n fetch' commands to download a bundle list from a configured URI, and\n only download bundles that are new based on the creation token values.\n [3]\n\nI have prepared some of this work as pull requests on my personal fork so\ncurious readers can look ahead to where we are going:\n\n[3]\nhttps://lore.kernel.org/git/pull.1248.v3.git.1658757188.gitgitgadget@gmail.com\n\n[4] https://github.com/derrickstolee/git/pull/21\n\n[5] https://github.com/derrickstolee/git/pull/22\n\n\nUpdates in v4\n=============\n\n * Properly updated the patch outline.\n\n * Jonathan Tan asked for more tests, and this revealed some interesting\n behaviors which I have now either fixed or made explicit:\n \n 1. In \"all\" mode, we try to download and apply all bundles. Do not fail\n if a single bundle download fails.\n 2. Previously, not all bundles were being applied, and this was noticed\n by the added checks for the refs/bundles/* refs at the end of the\n tests. This revealed the need for removing the reachability walk from\n verify_bundle() since the written refs/bundles/* refs were not being\n picked up by the loose ref cache. Since removing the reachability walk\n seemed like the faster (for users) option, I went that direction.\n 3. While running those tests and examining the output carefully, I\n noticed several error messages related to missing prerequisites due to\n attempting unbundling in a random order. This doesn't appear in the\n later creationToken version, so I hadn't noticed it at the tip of my\n local work. These messages are removed with a new quiet mode for\n verify_bundle().\n\n\nUpdates in v3\n=============\n\n * Fixed a comment about a return value of -1.\n * Fixed and tested scenario where early URIs fail in \"any\" mode and Git\n should try the rest of the list.\n * Instead of using 'success_count' and 'failure_count', use the iterator\n return value to terminate the \"all\" mode loop early.\n\n\nUpdates in v2\n=============\n\nThank you to all of the voices who chimed in on the previous version. I'm\nsorry it took so long for me to get a new version.\n\n * I've done a rather thorough overhaul to minimize how often later patches\n rewrite portions of earlier patches.\n\n * We no longer use a strbuf in struct remote_bundle_info. Instead, use a\n 'char *' and only in the patch where it is first used.\n\n * The config documentation is more clearly indicating that the bundle.*\n section has no effect in the repository config (at the moment, which will\n change in the next series).\n\n * The bundle.version value is now parsed using git_parse_int().\n\n * The config key is now parsed using parse_config_key().\n\n * Commit messages clarify more about the context of the change in the\n bigger picture of the bundle URI effort.\n\n * Some printf()s are correctly changed to fprintf()s.\n\n * The test helper CLI is unified across the two modes. They both take a\n filename now.\n\n * The count of downloaded bundles is now only updated after a successful\n download, allowing the \"any\" mode to keep trying after a failure.\n\nThanks,\n\n * Stolee\n\nDerrick Stolee (9):\n bundle-uri: use plain string in find_temp_filename()\n bundle-uri: create bundle_list struct and helpers\n bundle-uri: create base key-value pair parsing\n bundle-uri: parse bundle list in config format\n bundle-uri: limit recursion depth for bundle lists\n bundle: add flags to verify_bundle(), skip walk\n bundle-uri: fetch a list of bundles\n bundle-uri: quiet failed unbundlings\n bundle-uri: suppress stderr from remote-https\n\nÆvar Arnfjörð Bjarmason (2):\n bundle-uri: create \"key=value\" line parsing\n bundle-uri: unit test \"key=value\" parsing\n\n Documentation/config.txt | 2 +\n Documentation/config/bundle.txt | 24 ++\n Makefile | 1 +\n builtin/bundle.c | 5 +-\n bundle-uri.c | 458 ++++++++++++++++++++++++++++++--\n bundle-uri.h | 93 +++++++\n bundle.c | 22 +-\n bundle.h | 16 +-\n config.c | 2 +-\n config.h | 1 +\n t/helper/test-bundle-uri.c | 95 +++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5558-clone-bundle-uri.sh | 275 +++++++++++++++++++\n t/t5750-bundle-uri-parse.sh | 171 ++++++++++++\n t/test-lib-functions.sh | 11 +\n transport.c | 2 +-\n 17 files changed, 1149 insertions(+), 31 deletions(-)\n create mode 100644 Documentation/config/bundle.txt\n create mode 100644 t/helper/test-bundle-uri.c\n create mode 100755 t/t5750-bundle-uri-parse.sh\n\n\nbase-commit: e21e663cd1942df29979d3e01f7eacb532727bb7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1333%2Fderrickstolee%2Fbundle-redo%2Flist-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1333/derrickstolee/bundle-redo/list-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1333\n\nRange-diff vs v3:\n\n 1: 48beccb0f5e = 1: 48beccb0f5e bundle-uri: use plain string in find_temp_filename()\n 2: f0c4457951c = 2: f0c4457951c bundle-uri: create bundle_list struct and helpers\n 3: 430e01cd2a4 = 3: 430e01cd2a4 bundle-uri: create base key-value pair parsing\n 4: cd915d57f3b = 4: cd915d57f3b bundle-uri: create \"key=value\" line parsing\n 5: 4d8cac67f66 = 5: 4d8cac67f66 bundle-uri: unit test \"key=value\" parsing\n 6: 0ecae3a44b3 = 6: 0ecae3a44b3 bundle-uri: parse bundle list in config format\n 7: 7e6b32313b0 = 7: 7e6b32313b0 bundle-uri: limit recursion depth for bundle lists\n -: ----------- > 8: 83f2cd893a4 bundle: add flags to verify_bundle(), skip walk\n 8: 46799648b4c ! 9: 6b9c764c6b3 bundle-uri: fetch a list of bundles\n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *fi\n +\t\tctx->count++;\n +\n +\t/*\n -+\t * In BUNDLE_MODE_ANY, we need to continue iterating until we find\n -+\t * a bundle that works, so do not signal a failure here.\n ++\t * To be opportunistic as possible, we continue iterating and\n ++\t * download as many bundles as we can, so we can apply the ones\n ++\t * that work, even in BUNDLE_MODE_ALL mode.\n +\t */\n -+\treturn ctx->mode == BUNDLE_MODE_ANY ? 0 : res;\n ++\treturn 0;\n +}\n +\n +static int download_bundle_list(struct repository *r,\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone with file:// bundle' '\n +\n +\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-list-file &&\n +\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n -+\tgit -C clone-list-file cat-file --batch-check refs &&\n ++\tgrep \"refs/bundles/\" refs >actual &&\n ++\tcat >expect <<-\\EOF &&\n ++\trefs/bundles/base\n ++\trefs/bundles/left\n ++\trefs/bundles/merge\n ++\trefs/bundles/right\n ++\tEOF\n ++\ttest_cmp expect actual\n ++'\n ++\n ++test_expect_success 'clone bundle list (file, all mode, some failures)' '\n ++\tcat >bundle-list <<-EOF &&\n ++\t[bundle]\n ++\t\tversion = 1\n ++\t\tmode = all\n ++\n ++\t# Does not exist. Should be skipped.\n ++\t[bundle \"bundle-0\"]\n ++\t\turi = file://$(pwd)/clone-from/bundle-0.bundle\n ++\n ++\t[bundle \"bundle-1\"]\n ++\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n ++\n ++\t[bundle \"bundle-2\"]\n ++\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n ++\n ++\t# No bundle-3 means bundle-4 will not apply.\n ++\n ++\t[bundle \"bundle-4\"]\n ++\t\turi = file://$(pwd)/clone-from/bundle-4.bundle\n ++\n ++\t# Does not exist. Should be skipped.\n ++\t[bundle \"bundle-5\"]\n ++\t\turi = file://$(pwd)/clone-from/bundle-5.bundle\n ++\tEOF\n ++\n ++\tGIT_TRACE2_PERF=1 \\\n ++\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-all-some &&\n ++\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n ++\tgit -C clone-all-some cat-file --batch-check refs &&\n ++\tgrep \"refs/bundles/\" refs >actual &&\n ++\tcat >expect <<-\\EOF &&\n ++\trefs/bundles/base\n ++\trefs/bundles/left\n ++\tEOF\n ++\ttest_cmp expect actual\n ++'\n ++\n ++test_expect_success 'clone bundle list (file, all mode, all failures)' '\n ++\tcat >bundle-list <<-EOF &&\n ++\t[bundle]\n ++\t\tversion = 1\n ++\t\tmode = all\n ++\n ++\t# Does not exist. Should be skipped.\n ++\t[bundle \"bundle-0\"]\n ++\t\turi = file://$(pwd)/clone-from/bundle-0.bundle\n ++\n ++\t# Does not exist. Should be skipped.\n ++\t[bundle \"bundle-5\"]\n ++\t\turi = file://$(pwd)/clone-from/bundle-5.bundle\n ++\tEOF\n ++\n ++\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-all-fail &&\n ++\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n ++\tgit -C clone-all-fail cat-file --batch-check refs &&\n ++\t! grep \"refs/bundles/\" refs\n +'\n +\n +test_expect_success 'clone bundle list (file, any mode)' '\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone with file:// bundle' '\n +\n +\t# Does not exist. Should be skipped.\n +\t[bundle \"bundle-0\"]\n -+\t\turi = $HTTPD_URL/bundle-0.bundle\n ++\t\turi = file://$(pwd)/clone-from/bundle-0.bundle\n +\n +\t[bundle \"bundle-1\"]\n -+\t\turi = $HTTPD_URL/bundle-1.bundle\n ++\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n +\n +\t# Does not exist. Should be skipped.\n +\t[bundle \"bundle-5\"]\n -+\t\turi = $HTTPD_URL/bundle-5.bundle\n ++\t\turi = file://$(pwd)/clone-from/bundle-5.bundle\n +\tEOF\n +\n +\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-any-file &&\n +\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n -+\tgit -C clone-any-file cat-file --batch-check refs &&\n ++\tgrep \"refs/bundles/\" refs >actual &&\n ++\tcat >expect <<-\\EOF &&\n ++\trefs/bundles/base\n ++\tEOF\n ++\ttest_cmp expect actual\n ++'\n ++\n ++test_expect_success 'clone bundle list (file, any mode, all failures)' '\n ++\tcat >bundle-list <<-EOF &&\n ++\t[bundle]\n ++\t\tversion = 1\n ++\t\tmode = any\n ++\n ++\t# Does not exist. Should be skipped.\n ++\t[bundle \"bundle-0\"]\n ++\t\turi = $HTTPD_URL/bundle-0.bundle\n ++\n ++\t# Does not exist. Should be skipped.\n ++\t[bundle \"bundle-5\"]\n ++\t\turi = $HTTPD_URL/bundle-5.bundle\n ++\tEOF\n ++\n ++\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-any-fail &&\n ++\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n ++\tgit -C clone-any-fail cat-file --batch-check refs &&\n ++\t! grep \"refs/bundles/\" refs\n +'\n +\n #########################################################################\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone HTTP bundle' '\n +\n +\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" clone-from clone-any-http &&\n +\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n -+\tgit -C clone-any-http cat-file --batch-check refs &&\n ++\tgrep \"refs/bundles/\" refs >actual &&\n ++\tcat >expect <<-\\EOF &&\n ++\trefs/bundles/base\n ++\trefs/bundles/left\n ++\trefs/bundles/merge\n ++\trefs/bundles/right\n ++\tEOF\n ++\ttest_cmp expect actual\n +'\n +\n # Do not add tests here unless they use the HTTP server, as they will\n -: ----------- > 10: 1cae3096624 bundle-uri: quiet failed unbundlings\n 9: d84544859e4 ! 11: 52a575f8a69 bundle-uri: suppress stderr from remote-https\n @@ bundle-uri.c: static int download_https_uri_to_file(const char *file, const char\n \n \n ## t/t5558-clone-bundle-uri.sh ##\n -@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (file, any mode)' '\n +@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (file, all mode, some failures)' '\n + \tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n + \t\tclone-from clone-all-some 2>err &&\n + \t! grep \"Repository lacks these prerequisite commits\" err &&\n ++\t! grep \"fatal\" err &&\n ++\tgrep \"warning: failed to download bundle from URI\" err &&\n + \n + \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n + \tgit -C clone-all-some cat-file --batch-check err &&\n + \t! grep \"Repository lacks these prerequisite commits\" err &&\n ++\t! grep \"fatal\" err &&\n ++\tgrep \"warning: failed to download bundle from URI\" err &&\n + \n + \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n + \tgit -C clone-all-fail cat-file --batch-check err &&\n ++\t\tclone-from clone-any-fail 2>err &&\n +\t! grep \"fatal\" err &&\n +\tgrep \"warning: failed to download bundle from URI\" err &&\n +\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n - \tgit -C clone-any-file cat-file --batch-check oids &&\n - \tgit -C clone-any-http cat-file --batch-check \n\nIt will likely be rare where a user uses a single bundle URI and expects\nthat URI to point to a bundle. Instead, that URI will likely be a list\nof bundles provided in some format. Alternatively, the Git server could\nadvertise a list of bundles.\n\nIn anticipation of these two ways of advertising multiple bundles,\ncreate a data structure that represents such a list. This will be\npopulated using a common API, but for now focus on what data can be\nrepresented.\n\nEach list contains a number of remote_bundle_info structs. These contain\nan 'id' that is used to uniquely identify them in the list, and also a\n'uri' that contains the location of its data. Finally, there is a strbuf\ncontaining the filename used when Git downloads the contents to disk.\n\nThe list itself stores these remote_bundle_info structs in a hashtable\nusing 'id' as the key. The order of the structs in the input is\nconsidered unimportant, but future modifications to the format and these\ndata structures will place ordering possibilities on the set. The list\nalso has a few \"global\" properties, including the version (used when\nparsing the list) and the mode. The mode is one of these two options:\n\n1. BUNDLE_MODE_ALL: all listed URIs are intended to be combined\n together. The client should download all of the advertised data to\n have a complete copy of the data.\n\n2. BUNDLE_MODE_ANY: any one listed item is sufficient to have a complete\n copy of the data. The client can choose arbitrarily from these\n options. In the future, the client may use pings to find the closest\n URI among geodistributed replicas, or use some other heuristic\n information added to the format.\n\nThis API is currently unused, but will soon be expanded with parsing\nlogic and then be consumed by the bundle URI download logic.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n bundle-uri.h | 56 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 116 insertions(+)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 8b2f4e08c9c..f9a8db221bc 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -4,6 +4,66 @@\n #include \"object-store.h\"\n #include \"refs.h\"\n #include \"run-command.h\"\n+#include \"hashmap.h\"\n+#include \"pkt-line.h\"\n+\n+static int compare_bundles(const void *hashmap_cmp_fn_data,\n+\t\t\t const struct hashmap_entry *he1,\n+\t\t\t const struct hashmap_entry *he2,\n+\t\t\t const void *id)\n+{\n+\tconst struct remote_bundle_info *e1 =\n+\t\tcontainer_of(he1, const struct remote_bundle_info, ent);\n+\tconst struct remote_bundle_info *e2 =\n+\t\tcontainer_of(he2, const struct remote_bundle_info, ent);\n+\n+\treturn strcmp(e1->id, id ? (const char *)id : e2->id);\n+}\n+\n+void init_bundle_list(struct bundle_list *list)\n+{\n+\tmemset(list, 0, sizeof(*list));\n+\n+\t/* Implied defaults. */\n+\tlist->mode = BUNDLE_MODE_ALL;\n+\tlist->version = 1;\n+\n+\thashmap_init(&list->bundles, compare_bundles, NULL, 0);\n+}\n+\n+static int clear_remote_bundle_info(struct remote_bundle_info *bundle,\n+\t\t\t\t void *data)\n+{\n+\tFREE_AND_NULL(bundle->id);\n+\tFREE_AND_NULL(bundle->uri);\n+\treturn 0;\n+}\n+\n+void clear_bundle_list(struct bundle_list *list)\n+{\n+\tif (!list)\n+\t\treturn;\n+\n+\tfor_all_bundles_in_list(list, clear_remote_bundle_info, NULL);\n+\thashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent);\n+}\n+\n+int for_all_bundles_in_list(struct bundle_list *list,\n+\t\t\t bundle_iterator iter,\n+\t\t\t void *data)\n+{\n+\tstruct remote_bundle_info *info;\n+\tstruct hashmap_iter i;\n+\n+\thashmap_for_each_entry(&list->bundles, &i, info, ent) {\n+\t\tint result = iter(info, data);\n+\n+\t\tif (result)\n+\t\t\treturn result;\n+\t}\n+\n+\treturn 0;\n+}\n \n static char *find_temp_filename(void)\n {\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 8a152f1ef14..ff7e3fd3fb2 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -1,7 +1,63 @@\n #ifndef BUNDLE_URI_H\n #define BUNDLE_URI_H\n \n+#include \"hashmap.h\"\n+#include \"strbuf.h\"\n+\n struct repository;\n+struct string_list;\n+\n+/**\n+ * The remote_bundle_info struct contains information for a single bundle\n+ * URI. This may be initialized simply by a given URI or might have\n+ * additional metadata associated with it if the bundle was advertised by\n+ * a bundle list.\n+ */\n+struct remote_bundle_info {\n+\tstruct hashmap_entry ent;\n+\n+\t/**\n+\t * The 'id' is a name given to the bundle for reference\n+\t * by other bundle infos.\n+\t */\n+\tchar *id;\n+\n+\t/**\n+\t * The 'uri' is the location of the remote bundle so\n+\t * it can be downloaded on-demand. This will be NULL\n+\t * if there was no table of contents.\n+\t */\n+\tchar *uri;\n+};\n+\n+#define REMOTE_BUNDLE_INFO_INIT { 0 }\n+\n+enum bundle_list_mode {\n+\tBUNDLE_MODE_NONE = 0,\n+\tBUNDLE_MODE_ALL,\n+\tBUNDLE_MODE_ANY\n+};\n+\n+/**\n+ * A bundle_list contains an unordered set of remote_bundle_info structs,\n+ * as well as information about the bundle listing, such as version and\n+ * mode.\n+ */\n+struct bundle_list {\n+\tint version;\n+\tenum bundle_list_mode mode;\n+\tstruct hashmap bundles;\n+};\n+\n+void init_bundle_list(struct bundle_list *list);\n+void clear_bundle_list(struct bundle_list *list);\n+\n+typedef int (*bundle_iterator)(struct remote_bundle_info *bundle,\n+\t\t\t void *data);\n+\n+int for_all_bundles_in_list(struct bundle_list *list,\n+\t\t\t bundle_iterator iter,\n+\t\t\t void *data);\n \n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n-- \ngitgitgadget\n\n"},{"id":"464484","messageId":"48beccb0f5efe6f9247968cd0d4c455c23a24c53.1665417859.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v4.git.1665417859.gitgitgadget@gmail.com","subject":"[PATCH v4 01/11] bundle-uri: use plain string in find_temp_filename()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-10T16:04:09Z","receivedAt":"2022-10-10T16:04:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe find_temp_filename() method was created in 53a50892be2 (bundle-uri:\ncreate basic file-copy logic, 2022-08-09) and uses odb_mkstemp() to\ncreate a temporary filename. The odb_mkstemp() method uses a strbuf in\nits interface, but we do not need to continue carrying a strbuf\nthroughout the bundle URI code.\n\nConvert the find_temp_filename() method to use a 'char *' and modify its\nonly caller. This makes sense that we don't actually need to modify this\nfilename directly later, so using a strbuf is overkill.\n\nThis change will simplify the data structure for tracking a bundle list\nto use plain strings instead of strbufs.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 28 ++++++++++++++++------------\n 1 file changed, 16 insertions(+), 12 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 4a8cc74ed05..8b2f4e08c9c 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -5,22 +5,23 @@\n #include \"refs.h\"\n #include \"run-command.h\"\n \n-static int find_temp_filename(struct strbuf *name)\n+static char *find_temp_filename(void)\n {\n \tint fd;\n+\tstruct strbuf name = STRBUF_INIT;\n \t/*\n \t * Find a temporary filename that is available. This is briefly\n \t * racy, but unlikely to collide.\n \t */\n-\tfd = odb_mkstemp(name, \"bundles/tmp_uri_XXXXXX\");\n+\tfd = odb_mkstemp(&name, \"bundles/tmp_uri_XXXXXX\");\n \tif (fd < 0) {\n \t\twarning(_(\"failed to create temporary file\"));\n-\t\treturn -1;\n+\t\treturn NULL;\n \t}\n \n \tclose(fd);\n-\tunlink(name->buf);\n-\treturn 0;\n+\tunlink(name.buf);\n+\treturn strbuf_detach(&name, NULL);\n }\n \n static int download_https_uri_to_file(const char *file, const char *uri)\n@@ -141,28 +142,31 @@ static int unbundle_from_file(struct repository *r, const char *file)\n int fetch_bundle_uri(struct repository *r, const char *uri)\n {\n \tint result = 0;\n-\tstruct strbuf filename = STRBUF_INIT;\n+\tchar *filename;\n \n-\tif ((result = find_temp_filename(&filename)))\n+\tif (!(filename = find_temp_filename())) {\n+\t\tresult = -1;\n \t\tgoto cleanup;\n+\t}\n \n-\tif ((result = copy_uri_to_file(filename.buf, uri))) {\n+\tif ((result = copy_uri_to_file(filename, uri))) {\n \t\twarning(_(\"failed to download bundle from URI '%s'\"), uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = !is_bundle(filename.buf, 0))) {\n+\tif ((result = !is_bundle(filename, 0))) {\n \t\twarning(_(\"file at URI '%s' is not a bundle\"), uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = unbundle_from_file(r, filename.buf))) {\n+\tif ((result = unbundle_from_file(r, filename))) {\n \t\twarning(_(\"failed to unbundle bundle from URI '%s'\"), uri);\n \t\tgoto cleanup;\n \t}\n \n cleanup:\n-\tunlink(filename.buf);\n-\tstrbuf_release(&filename);\n+\tif (filename)\n+\t\tunlink(filename);\n+\tfree(filename);\n \treturn result;\n }\n-- \ngitgitgadget\n\n"},{"id":"464485","messageId":"430e01cd2a4127a17c4bfc1b0db7209964736de8.1665417859.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v4.git.1665417859.gitgitgadget@gmail.com","subject":"[PATCH v4 03/11] bundle-uri: create base key-value pair parsing","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-10T16:04:11Z","receivedAt":"2022-10-10T16:04:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThere will be two primary ways to advertise a bundle list: as a list of\npacket lines in Git's protocol v2 and as a config file served from a\nbundle URI. Both of these fundamentally use a list of key-value pairs.\nWe will use the same set of key-value pairs across these formats.\n\nCreate a new bundle_list_update() method that is currently unusued, but\nwill be used in the next change. It inspects each key to see if it is\nunderstood and then applies it to the given bundle_list. Here are the\nkeys that we teach Git to understand:\n\n* bundle.version: This value should be an integer. Git currently\n understands only version 1 and will ignore the list if the version is\n any other value. This version can be increased in the future if we\n need to add new keys that Git should not ignore. We can add new\n \"heuristic\" keys without incrementing the version.\n\n* bundle.mode: This value should be one of \"all\" or \"any\". If this\n mode is not understood, then Git will ignore the list. This mode\n indicates whether Git needs all of the bundle list items to make a\n complete view of the content or if any single item is sufficient.\n\nThe rest of the keys use a bundle identifier \"\" as part of the key\nname. Keys using the same \"\" describe a single bundle list item.\n\n* bundle..uri: This stores the URI of the bundle item. This\n currently is expected to be an absolute URI, but will be relaxed to be\n a relative URI in the future.\n\nWhile parsing, return an error if a URI key is repeated, since we can\nmake that restriction with bundle lists.\n\nMake the git_parse_int() method global so we can parse the integer\nversion value carefully.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 2 +\n Documentation/config/bundle.txt | 24 +++++++++++\n bundle-uri.c | 76 +++++++++++++++++++++++++++++++++\n config.c | 2 +-\n config.h | 1 +\n 5 files changed, 104 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/config/bundle.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex e376d547ce0..4280af6992e 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -387,6 +387,8 @@ include::config/branch.txt[]\n \n include::config/browser.txt[]\n \n+include::config/bundle.txt[]\n+\n include::config/checkout.txt[]\n \n include::config/clean.txt[]\ndiff --git a/Documentation/config/bundle.txt b/Documentation/config/bundle.txt\nnew file mode 100644\nindex 00000000000..daa21eb674a\n--- /dev/null\n+++ b/Documentation/config/bundle.txt\n@@ -0,0 +1,24 @@\n+bundle.*::\n+\tThe `bundle.*` keys may appear in a bundle list file found via the\n+\t`git clone --bundle-uri` option. These keys currently have no effect\n+\tif placed in a repository config file, though this will change in the\n+\tfuture. See link:technical/bundle-uri.html[the bundle URI design\n+\tdocument] for more details.\n+\n+bundle.version::\n+\tThis integer value advertises the version of the bundle list format\n+\tused by the bundle list. Currently, the only accepted value is `1`.\n+\n+bundle.mode::\n+\tThis string value should be either `all` or `any`. This value describes\n+\twhether all of the advertised bundles are required to unbundle a\n+\tcomplete understanding of the bundled information (`all`) or if any one\n+\tof the listed bundle URIs is sufficient (`any`).\n+\n+bundle..*::\n+\tThe `bundle..*` keys are used to describe a single item in the\n+\tbundle list, grouped under `` for identification purposes.\n+\n+bundle..uri::\n+\tThis string value defines the URI by which Git can reach the contents\n+\tof this ``. This URI may be a bundle file or another bundle list.\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex f9a8db221bc..0bc59dd9c34 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -6,6 +6,7 @@\n #include \"run-command.h\"\n #include \"hashmap.h\"\n #include \"pkt-line.h\"\n+#include \"config.h\"\n \n static int compare_bundles(const void *hashmap_cmp_fn_data,\n \t\t\t const struct hashmap_entry *he1,\n@@ -65,6 +66,81 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \treturn 0;\n }\n \n+/**\n+ * Given a key-value pair, update the state of the given bundle list.\n+ * Returns 0 if the key-value pair is understood. Returns -1 if the key\n+ * is not understood or the value is malformed.\n+ */\n+MAYBE_UNUSED\n+static int bundle_list_update(const char *key, const char *value,\n+\t\t\t struct bundle_list *list)\n+{\n+\tstruct strbuf id = STRBUF_INIT;\n+\tstruct remote_bundle_info lookup = REMOTE_BUNDLE_INFO_INIT;\n+\tstruct remote_bundle_info *bundle;\n+\tconst char *subsection, *subkey;\n+\tsize_t subsection_len;\n+\n+\tif (parse_config_key(key, \"bundle\", &subsection, &subsection_len, &subkey))\n+\t\treturn -1;\n+\n+\tif (!subsection_len) {\n+\t\tif (!strcmp(subkey, \"version\")) {\n+\t\t\tint version;\n+\t\t\tif (!git_parse_int(value, &version))\n+\t\t\t\treturn -1;\n+\t\t\tif (version != 1)\n+\t\t\t\treturn -1;\n+\n+\t\t\tlist->version = version;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\tif (!strcmp(subkey, \"mode\")) {\n+\t\t\tif (!strcmp(value, \"all\"))\n+\t\t\t\tlist->mode = BUNDLE_MODE_ALL;\n+\t\t\telse if (!strcmp(value, \"any\"))\n+\t\t\t\tlist->mode = BUNDLE_MODE_ANY;\n+\t\t\telse\n+\t\t\t\treturn -1;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\t/* Ignore other unknown global keys. */\n+\t\treturn 0;\n+\t}\n+\n+\tstrbuf_add(&id, subsection, subsection_len);\n+\n+\t/*\n+\t * Check for an existing bundle with this , or create one\n+\t * if necessary.\n+\t */\n+\tlookup.id = id.buf;\n+\thashmap_entry_init(&lookup.ent, strhash(lookup.id));\n+\tif (!(bundle = hashmap_get_entry(&list->bundles, &lookup, ent, NULL))) {\n+\t\tCALLOC_ARRAY(bundle, 1);\n+\t\tbundle->id = strbuf_detach(&id, NULL);\n+\t\thashmap_entry_init(&bundle->ent, strhash(bundle->id));\n+\t\thashmap_add(&list->bundles, &bundle->ent);\n+\t}\n+\tstrbuf_release(&id);\n+\n+\tif (!strcmp(subkey, \"uri\")) {\n+\t\tif (bundle->uri)\n+\t\t\treturn -1;\n+\t\tbundle->uri = xstrdup(value);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * At this point, we ignore any information that we don't\n+\t * understand, assuming it to be hints for a heuristic the client\n+\t * does not currently understand.\n+\t */\n+\treturn 0;\n+}\n+\n static char *find_temp_filename(void)\n {\n \tint fd;\ndiff --git a/config.c b/config.c\nindex 015bec360f5..e93101249f6 100644\n--- a/config.c\n+++ b/config.c\n@@ -1214,7 +1214,7 @@ static int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max)\n \treturn 0;\n }\n \n-static int git_parse_int(const char *value, int *ret)\n+int git_parse_int(const char *value, int *ret)\n {\n \tintmax_t tmp;\n \tif (!git_parse_signed(value, &tmp, maximum_signed_value_of_type(int)))\ndiff --git a/config.h b/config.h\nindex ca994d77147..ef9eade6414 100644\n--- a/config.h\n+++ b/config.h\n@@ -206,6 +206,7 @@ int config_with_options(config_fn_t fn, void *,\n \n int git_parse_ssize_t(const char *, ssize_t *);\n int git_parse_ulong(const char *, unsigned long *);\n+int git_parse_int(const char *value, int *ret);\n \n /**\n * Same as `git_config_bool`, except that it returns -1 on error rather\n-- \ngitgitgadget\n\n"},{"id":"464486","messageId":"cd915d57f3bced015dc154572612d4dd221d1618.1665417859.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v4.git.1665417859.gitgitgadget@gmail.com","subject":"[PATCH v4 04/11] bundle-uri: create \"key=value\" line parsing","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-10T16:04:12Z","receivedAt":"2022-10-10T16:04:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nWhen advertising a bundle list over Git's protocol v2, we will use\npacket lines. Each line will be of the form \"key=value\" representing a\nbundle list. Connect the API necessary for Git's transport to the\nkey-value pair parsing created in the previous change.\n\nWe are not currently implementing this protocol v2 functionality, but\ninstead preparing to expose this parsing to be unit-testable.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 27 ++++++++++++++++++++++++++-\n bundle-uri.h | 12 ++++++++++++\n 2 files changed, 38 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 0bc59dd9c34..372e6fac5cf 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -71,7 +71,6 @@ int for_all_bundles_in_list(struct bundle_list *list,\n * Returns 0 if the key-value pair is understood. Returns -1 if the key\n * is not understood or the value is malformed.\n */\n-MAYBE_UNUSED\n static int bundle_list_update(const char *key, const char *value,\n \t\t\t struct bundle_list *list)\n {\n@@ -306,3 +305,29 @@ cleanup:\n \tfree(filename);\n \treturn result;\n }\n+\n+/**\n+ * General API for {transport,connect}.c etc.\n+ */\n+int bundle_uri_parse_line(struct bundle_list *list, const char *line)\n+{\n+\tint result;\n+\tconst char *equals;\n+\tstruct strbuf key = STRBUF_INIT;\n+\n+\tif (!strlen(line))\n+\t\treturn error(_(\"bundle-uri: got an empty line\"));\n+\n+\tequals = strchr(line, '=');\n+\n+\tif (!equals)\n+\t\treturn error(_(\"bundle-uri: line is not of the form 'key=value'\"));\n+\tif (line == equals || !*(equals + 1))\n+\t\treturn error(_(\"bundle-uri: line has empty key or value\"));\n+\n+\tstrbuf_add(&key, line, equals - line);\n+\tresult = bundle_list_update(key.buf, equals + 1, list);\n+\tstrbuf_release(&key);\n+\n+\treturn result;\n+}\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex ff7e3fd3fb2..90583461929 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -67,4 +67,16 @@ int for_all_bundles_in_list(struct bundle_list *list,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n+/**\n+ * General API for {transport,connect}.c etc.\n+ */\n+\n+/**\n+ * Parse a \"key=value\" packet line from the bundle-uri verb.\n+ *\n+ * Returns 0 on success and non-zero on error.\n+ */\n+int bundle_uri_parse_line(struct bundle_list *list,\n+\t\t\t const char *line);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"464487","messageId":"4d8cac67f66fc9c6477efabde795da95fafc51ec.1665417859.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v4.git.1665417859.gitgitgadget@gmail.com","subject":"[PATCH v4 05/11] bundle-uri: unit test \"key=value\" parsing","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-10T16:04:13Z","receivedAt":"2022-10-10T16:04:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nCreate a new 'test-tool bundle-uri' test helper. This helper will assist\nin testing logic deep in the bundle URI feature.\n\nThis change introduces the 'parse-key-values' subcommand, which parses\nan input file as a list of lines. These are fed into\nbundle_uri_parse_line() to test how we construct a 'struct bundle_list'\nfrom that data. The list is then output to stdout as if the key-value\npairs were a Git config file.\n\nWe use an input file instead of stdin because of a future change to\nparse in config-file format that works better as an input file.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n bundle-uri.c | 33 ++++++++++\n bundle-uri.h | 3 +\n t/helper/test-bundle-uri.c | 70 +++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5750-bundle-uri-parse.sh | 121 ++++++++++++++++++++++++++++++++++++\n t/test-lib-functions.sh | 11 ++++\n 8 files changed, 241 insertions(+)\n create mode 100644 t/helper/test-bundle-uri.c\n create mode 100755 t/t5750-bundle-uri-parse.sh\n\ndiff --git a/Makefile b/Makefile\nindex 7d5f48069ea..7dee0329c49 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -722,6 +722,7 @@ PROGRAMS += $(patsubst %.o,git-%$X,$(PROGRAM_OBJS))\n TEST_BUILTINS_OBJS += test-advise.o\n TEST_BUILTINS_OBJS += test-bitmap.o\n TEST_BUILTINS_OBJS += test-bloom.o\n+TEST_BUILTINS_OBJS += test-bundle-uri.o\n TEST_BUILTINS_OBJS += test-chmtime.o\n TEST_BUILTINS_OBJS += test-config.o\n TEST_BUILTINS_OBJS += test-crontab.o\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 372e6fac5cf..c02e7f62eb1 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -66,6 +66,39 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \treturn 0;\n }\n \n+static int summarize_bundle(struct remote_bundle_info *info, void *data)\n+{\n+\tFILE *fp = data;\n+\tfprintf(fp, \"[bundle \\\"%s\\\"]\\n\", info->id);\n+\tfprintf(fp, \"\\turi = %s\\n\", info->uri);\n+\treturn 0;\n+}\n+\n+void print_bundle_list(FILE *fp, struct bundle_list *list)\n+{\n+\tconst char *mode;\n+\n+\tswitch (list->mode) {\n+\tcase BUNDLE_MODE_ALL:\n+\t\tmode = \"all\";\n+\t\tbreak;\n+\n+\tcase BUNDLE_MODE_ANY:\n+\t\tmode = \"any\";\n+\t\tbreak;\n+\n+\tcase BUNDLE_MODE_NONE:\n+\tdefault:\n+\t\tmode = \"\";\n+\t}\n+\n+\tfprintf(fp, \"[bundle]\\n\");\n+\tfprintf(fp, \"\\tversion = %d\\n\", list->version);\n+\tfprintf(fp, \"\\tmode = %s\\n\", mode);\n+\n+\tfor_all_bundles_in_list(list, summarize_bundle, fp);\n+}\n+\n /**\n * Given a key-value pair, update the state of the given bundle list.\n * Returns 0 if the key-value pair is understood. Returns -1 if the key\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 90583461929..0e56ab2ae5a 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -59,6 +59,9 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \t\t\t bundle_iterator iter,\n \t\t\t void *data);\n \n+struct FILE;\n+void print_bundle_list(FILE *fp, struct bundle_list *list);\n+\n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n * based on that information.\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nnew file mode 100644\nindex 00000000000..0329c56544f\n--- /dev/null\n+++ b/t/helper/test-bundle-uri.c\n@@ -0,0 +1,70 @@\n+#include \"test-tool.h\"\n+#include \"parse-options.h\"\n+#include \"bundle-uri.h\"\n+#include \"strbuf.h\"\n+#include \"string-list.h\"\n+\n+static int cmd__bundle_uri_parse(int argc, const char **argv)\n+{\n+\tconst char *key_value_usage[] = {\n+\t\t\"test-tool bundle-uri parse-key-values \",\n+\t\tNULL\n+\t};\n+\tconst char **usage = key_value_usage;\n+\tstruct option options[] = {\n+\t\tOPT_END(),\n+\t};\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct bundle_list list;\n+\tint err = 0;\n+\tFILE *fp;\n+\n+\targc = parse_options(argc, argv, NULL, options, usage, 0);\n+\tif (argc != 1)\n+\t\tgoto usage;\n+\n+\tinit_bundle_list(&list);\n+\tfp = fopen(argv[0], \"r\");\n+\tif (!fp)\n+\t\tdie(\"failed to open '%s'\", argv[0]);\n+\n+\twhile (strbuf_getline(&sb, fp) != EOF) {\n+\t\tif (bundle_uri_parse_line(&list, sb.buf))\n+\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\t}\n+\tstrbuf_release(&sb);\n+\tfclose(fp);\n+\n+\tprint_bundle_list(stdout, &list);\n+\n+\tclear_bundle_list(&list);\n+\n+\treturn !!err;\n+\n+usage:\n+\tusage_with_options(usage, options);\n+}\n+\n+int cmd__bundle_uri(int argc, const char **argv)\n+{\n+\tconst char *usage[] = {\n+\t\t\"test-tool bundle-uri []\",\n+\t\tNULL\n+\t};\n+\tstruct option options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL, options, usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION |\n+\t\t\t PARSE_OPT_KEEP_ARGV0);\n+\tif (argc == 1)\n+\t\tgoto usage;\n+\n+\tif (!strcmp(argv[1], \"parse-key-values\"))\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1);\n+\terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n+\n+usage:\n+\tusage_with_options(usage, options);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 318fdbab0c3..fbe2d9d8108 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -17,6 +17,7 @@ static struct test_cmd cmds[] = {\n \t{ \"advise\", cmd__advise_if_enabled },\n \t{ \"bitmap\", cmd__bitmap },\n \t{ \"bloom\", cmd__bloom },\n+\t{ \"bundle-uri\", cmd__bundle_uri },\n \t{ \"chmtime\", cmd__chmtime },\n \t{ \"config\", cmd__config },\n \t{ \"crontab\", cmd__crontab },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex bb799271631..b2aa1f39a8f 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -7,6 +7,7 @@\n int cmd__advise_if_enabled(int argc, const char **argv);\n int cmd__bitmap(int argc, const char **argv);\n int cmd__bloom(int argc, const char **argv);\n+int cmd__bundle_uri(int argc, const char **argv);\n int cmd__chmtime(int argc, const char **argv);\n int cmd__config(int argc, const char **argv);\n int cmd__crontab(int argc, const char **argv);\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nnew file mode 100755\nindex 00000000000..fd142a66ad5\n--- /dev/null\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -0,0 +1,121 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri bundle_uri_parse_line()\"\n+\n+TEST_NO_CREATE_REPO=1\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success 'bundle_uri_parse_line() just URIs' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' '\n+\tcat >in <<-\\EOF &&\n+\t=bogus-value\n+\tbogus-key=\n+\tEOF\n+\n+\tcat >err.expect <<-EOF &&\n+\terror: bundle-uri: line has empty key or value\n+\terror: bad line: '\\''=bogus-value'\\''\n+\terror: bundle-uri: line has empty key or value\n+\terror: bad line: '\\''bogus-key='\\''\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty lines' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\terror: bundle-uri: got an empty line\n+\terror: bad line: '\\'''\\''\n+\terror: bundle-uri: got an empty line\n+\terror: bad line: '\\'''\\''\n+\tEOF\n+\n+\t# We fail, but try to continue parsing regardless\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: duplicate lines' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\tbundle.one.uri=https://example.com/bundle-2.bdl\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\terror: bad line: '\\''bundle.one.uri=https://example.com/bundle-2.bdl'\\''\n+\tEOF\n+\n+\t# We fail, but try to continue parsing regardless\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_done\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 6da7273f1d5..3175d665add 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1956,3 +1956,14 @@ test_is_magic_mtime () {\n \trm -f .git/test-mtime-actual\n \treturn $ret\n }\n+\n+# Given two filenames, parse both using 'git config --list --file'\n+# and compare the sorted output of those commands. Useful when\n+# wanting to ignore whitespace differences and sorting concerns.\n+test_cmp_config_output () {\n+\tgit config --list --file=\"$1\" >config-expect &&\n+\tgit config --list --file=\"$2\" >config-actual &&\n+\tsort config-expect >sorted-expect &&\n+\tsort config-actual >sorted-actual &&\n+\ttest_cmp sorted-expect sorted-actual\n+}\n-- \ngitgitgadget\n\n"},{"id":"464488","messageId":"0ecae3a44b360fbb03a52b73536f83c457a6668d.1665417859.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v4.git.1665417859.gitgitgadget@gmail.com","subject":"[PATCH v4 06/11] bundle-uri: parse bundle list in config format","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-10T16:04:14Z","receivedAt":"2022-10-10T16:04:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a bundle provider wants to operate independently from a Git remote,\nthey want to provide a single, consistent URI that users can use in\ntheir 'git clone --bundle-uri' commands. At this point, the Git client\nexpects that URI to be a single bundle that can be unbundled and used to\nbootstrap the rest of the clone from the Git server. This single bundle\ncannot be re-used to assist with future incremental fetches.\n\nTo allow for the incremental fetch case, teach Git to understand a\nbundle list that could be advertised at an independent bundle URI. Such\na bundle list is likely to be inspected by human readers, even if only\nby the bundle provider creating the list. For this reason, we can take\nour expected \"key=value\" pairs and instead format them using Git config\nformat.\n\nCreate bundle_uri_parse_config_format() to parse a file in config format\nand convert that into a 'struct bundle_list' filled with its\nunderstanding of the contents.\n\nBe careful to use error_action CONFIG_ERROR_ERROR when calling\ngit_config_from_file_with_options() because the default action for\ngit_config_from_file() is to die() on a parsing error. The current\nwarning isn't particularly helpful if it arises to a user, but it will\nbe made more verbose at a higher layer later.\n\nUpdate 'test-tool bundle-uri' to take this config file format as input.\nIt uses a filename instead of stdin because there is no existing way to\nparse a FILE pointer in the config machinery. Using\ngit_config_from_mem() is overly complicated and more likely to introduce\nbugs than this simpler version.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 27 ++++++++++++++++++++\n bundle-uri.h | 9 +++++++\n t/helper/test-bundle-uri.c | 49 +++++++++++++++++++++++++++---------\n t/t5750-bundle-uri-parse.sh | 50 +++++++++++++++++++++++++++++++++++++\n 4 files changed, 123 insertions(+), 12 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex c02e7f62eb1..3d44ec2b1e6 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -173,6 +173,33 @@ static int bundle_list_update(const char *key, const char *value,\n \treturn 0;\n }\n \n+static int config_to_bundle_list(const char *key, const char *value, void *data)\n+{\n+\tstruct bundle_list *list = data;\n+\treturn bundle_list_update(key, value, list);\n+}\n+\n+int bundle_uri_parse_config_format(const char *uri,\n+\t\t\t\t const char *filename,\n+\t\t\t\t struct bundle_list *list)\n+{\n+\tint result;\n+\tstruct config_options opts = {\n+\t\t.error_action = CONFIG_ERROR_ERROR,\n+\t};\n+\n+\tresult = git_config_from_file_with_options(config_to_bundle_list,\n+\t\t\t\t\t\t filename, list,\n+\t\t\t\t\t\t &opts);\n+\n+\tif (!result && list->mode == BUNDLE_MODE_NONE) {\n+\t\twarning(_(\"bundle list at '%s' has no mode\"), uri);\n+\t\tresult = 1;\n+\t}\n+\n+\treturn result;\n+}\n+\n static char *find_temp_filename(void)\n {\n \tint fd;\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 0e56ab2ae5a..bc13d4c9929 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -62,6 +62,15 @@ int for_all_bundles_in_list(struct bundle_list *list,\n struct FILE;\n void print_bundle_list(FILE *fp, struct bundle_list *list);\n \n+/**\n+ * A bundle URI may point to a bundle list where the key=value\n+ * pairs are provided in config file format. This method is\n+ * exposed publicly for testing purposes.\n+ */\n+int bundle_uri_parse_config_format(const char *uri,\n+\t\t\t\t const char *filename,\n+\t\t\t\t struct bundle_list *list);\n+\n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n * based on that information.\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex 0329c56544f..25afd393428 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -4,12 +4,21 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n \n-static int cmd__bundle_uri_parse(int argc, const char **argv)\n+enum input_mode {\n+\tKEY_VALUE_PAIRS,\n+\tCONFIG_FILE,\n+};\n+\n+static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mode)\n {\n \tconst char *key_value_usage[] = {\n \t\t\"test-tool bundle-uri parse-key-values \",\n \t\tNULL\n \t};\n+\tconst char *config_usage[] = {\n+\t\t\"test-tool bundle-uri parse-config \",\n+\t\tNULL\n+\t};\n \tconst char **usage = key_value_usage;\n \tstruct option options[] = {\n \t\tOPT_END(),\n@@ -19,21 +28,35 @@ static int cmd__bundle_uri_parse(int argc, const char **argv)\n \tint err = 0;\n \tFILE *fp;\n \n-\targc = parse_options(argc, argv, NULL, options, usage, 0);\n-\tif (argc != 1)\n-\t\tgoto usage;\n+\tif (mode == CONFIG_FILE)\n+\t\tusage = config_usage;\n+\n+\targc = parse_options(argc, argv, NULL, options, usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tinit_bundle_list(&list);\n-\tfp = fopen(argv[0], \"r\");\n-\tif (!fp)\n-\t\tdie(\"failed to open '%s'\", argv[0]);\n \n-\twhile (strbuf_getline(&sb, fp) != EOF) {\n-\t\tif (bundle_uri_parse_line(&list, sb.buf))\n-\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\tswitch (mode) {\n+\tcase KEY_VALUE_PAIRS:\n+\t\tif (argc != 1)\n+\t\t\tgoto usage;\n+\t\tfp = fopen(argv[0], \"r\");\n+\t\tif (!fp)\n+\t\t\tdie(\"failed to open '%s'\", argv[0]);\n+\t\twhile (strbuf_getline(&sb, fp) != EOF) {\n+\t\t\tif (bundle_uri_parse_line(&list, sb.buf))\n+\t\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\t\t}\n+\t\tfclose(fp);\n+\t\tbreak;\n+\n+\tcase CONFIG_FILE:\n+\t\tif (argc != 1)\n+\t\t\tgoto usage;\n+\t\terr = bundle_uri_parse_config_format(\"\", argv[0], &list);\n+\t\tbreak;\n \t}\n \tstrbuf_release(&sb);\n-\tfclose(fp);\n \n \tprint_bundle_list(stdout, &list);\n \n@@ -62,7 +85,9 @@ int cmd__bundle_uri(int argc, const char **argv)\n \t\tgoto usage;\n \n \tif (!strcmp(argv[1], \"parse-key-values\"))\n-\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1);\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS);\n+\tif (!strcmp(argv[1], \"parse-config\"))\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE);\n \terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n \n usage:\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex fd142a66ad5..c2fe3f9c5a5 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -118,4 +118,54 @@ test_expect_success 'bundle_uri_parse_line() parsing edge cases: duplicate lines\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format: just URIs' '\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config expect >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'parse config format edge cases: empty key or value' '\n+\tcat >in1 <<-\\EOF &&\n+\t= bogus-value\n+\tEOF\n+\n+\tcat >err1 <<-EOF &&\n+\terror: bad config line 1 in file in1\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-config in1 >actual 2>err &&\n+\ttest_cmp err1 err &&\n+\ttest_cmp_config_output expect actual &&\n+\n+\tcat >in2 <<-\\EOF &&\n+\tbogus-key =\n+\tEOF\n+\n+\tcat >err2 <<-EOF &&\n+\terror: bad config line 1 in file in2\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-config in2 >actual 2>err &&\n+\ttest_cmp err2 err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"464489","messageId":"7e6b32313b0f1922c0d0bfd104a288b8606306ec.1665417859.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v4.git.1665417859.gitgitgadget@gmail.com","subject":"[PATCH v4 07/11] bundle-uri: limit recursion depth for bundle lists","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-10T16:04:15Z","receivedAt":"2022-10-10T16:04:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe next change will start allowing us to parse bundle lists that are\ndownloaded from a provided bundle URI. Those lists might point to other\nlists, which could proceed to an arbitrary depth (and even create\ncycles). Restructure fetch_bundle_uri() to have an internal version that\nhas a recursion depth. Compare that to a new max_bundle_uri_depth\nconstant that is twice as high as we expect this depth to be for any\nlegitimate use of bundle list linking.\n\nWe can consider making max_bundle_uri_depth a configurable value if\nthere is demonstrated value in the future.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 21 ++++++++++++++++++++-\n 1 file changed, 20 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 3d44ec2b1e6..8a7c11c6393 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -334,11 +334,25 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \treturn result;\n }\n \n-int fetch_bundle_uri(struct repository *r, const char *uri)\n+/**\n+ * This limits the recursion on fetch_bundle_uri_internal() when following\n+ * bundle lists.\n+ */\n+static int max_bundle_uri_depth = 4;\n+\n+static int fetch_bundle_uri_internal(struct repository *r,\n+\t\t\t\t const char *uri,\n+\t\t\t\t int depth)\n {\n \tint result = 0;\n \tchar *filename;\n \n+\tif (depth >= max_bundle_uri_depth) {\n+\t\twarning(_(\"exceeded bundle URI recursion limit (%d)\"),\n+\t\t\tmax_bundle_uri_depth);\n+\t\treturn -1;\n+\t}\n+\n \tif (!(filename = find_temp_filename())) {\n \t\tresult = -1;\n \t\tgoto cleanup;\n@@ -366,6 +380,11 @@ cleanup:\n \treturn result;\n }\n \n+int fetch_bundle_uri(struct repository *r, const char *uri)\n+{\n+\treturn fetch_bundle_uri_internal(r, uri, 0);\n+}\n+\n /**\n * General API for {transport,connect}.c etc.\n */\n-- \ngitgitgadget\n\n"},{"id":"464490","messageId":"83f2cd893a4c47c947a93fe99d202d67f540e963.1665417859.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v4.git.1665417859.gitgitgadget@gmail.com","subject":"[PATCH v4 08/11] bundle: add flags to verify_bundle(), skip walk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-10T16:04:16Z","receivedAt":"2022-10-10T16:05:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe verify_bundle() method checks if a bundle can be applied to a given\nrepository. This not only verifies that certain commits exist in the\nrepository, but Git also checks that these commits are reachable.\n\nThis behavior dates back to the original git-bundle builtin written in\n2e0afafebd8 (Add git-bundle: move objects and references by archive,\n2007-02-22), but the message does not go into detail why the\nreachability check is important.\n\nSince verify_bundle() is called from unbundle(), we need to add an\noption to pipe the flags through that method.\n\nWhen unbundling from a list of bundles, Git will create refs that point\nto the tips of the latest bundle, which makes this reachability walk\nsucceed, in theory. However, the loose refs cache does not get\ninvalidated and hence the reachability walk fails. By disabling the\nreachability walk in the bundle URI code, we can get around this\nreachability check.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/bundle.c | 5 +++--\n bundle-uri.c | 8 +++++++-\n bundle.c | 12 +++++++-----\n bundle.h | 15 +++++++++++++--\n transport.c | 2 +-\n 5 files changed, 31 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex 2adad545a2e..7d983a238f0 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -119,7 +119,8 @@ static int cmd_bundle_verify(int argc, const char **argv, const char *prefix) {\n \t\tgoto cleanup;\n \t}\n \tclose(bundle_fd);\n-\tif (verify_bundle(the_repository, &header, !quiet)) {\n+\tif (verify_bundle(the_repository, &header,\n+\t\t\t quiet ? 0 : VERIFY_BUNDLE_VERBOSE)) {\n \t\tret = 1;\n \t\tgoto cleanup;\n \t}\n@@ -185,7 +186,7 @@ static int cmd_bundle_unbundle(int argc, const char **argv, const char *prefix)\n \t\tstrvec_pushl(&extra_index_pack_args, \"-v\", \"--progress-title\",\n \t\t\t _(\"Unbundling objects\"), NULL);\n \tret = !!unbundle(the_repository, &header, bundle_fd,\n-\t\t\t &extra_index_pack_args) ||\n+\t\t\t &extra_index_pack_args, 0) ||\n \t\tlist_bundle_refs(&header, argc, argv);\n \tbundle_header_release(&header);\n cleanup:\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 8a7c11c6393..ad5baabdd94 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -301,7 +301,13 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \tif ((bundle_fd = read_bundle_header(file, &header)) < 0)\n \t\treturn 1;\n \n-\tif ((result = unbundle(r, &header, bundle_fd, NULL)))\n+\t/*\n+\t * Skip the reachability walk here, since we will be adding\n+\t * a reachable ref pointing to the new tips, which will reach\n+\t * the prerequisite commits.\n+\t */\n+\tif ((result = unbundle(r, &header, bundle_fd, NULL,\n+\t\t\t VERIFY_BUNDLE_SKIP_REACHABLE)))\n \t\treturn 1;\n \n \t/*\ndiff --git a/bundle.c b/bundle.c\nindex 0208e6d90d3..36ffeb1e0eb 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -189,7 +189,7 @@ static int list_refs(struct string_list *r, int argc, const char **argv)\n \n int verify_bundle(struct repository *r,\n \t\t struct bundle_header *header,\n-\t\t int verbose)\n+\t\t enum verify_bundle_flags flags)\n {\n \t/*\n \t * Do fast check, then if any prereqs are missing then go line by line\n@@ -222,7 +222,8 @@ int verify_bundle(struct repository *r,\n \t\t\terror(\"%s\", message);\n \t\terror(\"%s %s\", oid_to_hex(oid), name);\n \t}\n-\tif (revs.pending.nr != p->nr)\n+\tif (revs.pending.nr != p->nr ||\n+\t (flags & VERIFY_BUNDLE_SKIP_REACHABLE))\n \t\tgoto cleanup;\n \treq_nr = revs.pending.nr;\n \tsetup_revisions(2, argv, &revs, NULL);\n@@ -259,7 +260,7 @@ int verify_bundle(struct repository *r,\n \t\t\tclear_commit_marks(commit, ALL_REV_FLAGS);\n \t}\n \n-\tif (verbose) {\n+\tif (flags & VERIFY_BUNDLE_VERBOSE) {\n \t\tstruct string_list *r;\n \n \t\tr = &header->references;\n@@ -620,7 +621,8 @@ err:\n }\n \n int unbundle(struct repository *r, struct bundle_header *header,\n-\t int bundle_fd, struct strvec *extra_index_pack_args)\n+\t int bundle_fd, struct strvec *extra_index_pack_args,\n+\t enum verify_bundle_flags flags)\n {\n \tstruct child_process ip = CHILD_PROCESS_INIT;\n \tstrvec_pushl(&ip.args, \"index-pack\", \"--fix-thin\", \"--stdin\", NULL);\n@@ -634,7 +636,7 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \t\tstrvec_clear(extra_index_pack_args);\n \t}\n \n-\tif (verify_bundle(r, header, 0))\n+\tif (verify_bundle(r, header, flags))\n \t\treturn -1;\n \tip.in = bundle_fd;\n \tip.no_stdout = 1;\ndiff --git a/bundle.h b/bundle.h\nindex 0c052f54964..9f798c00d93 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -29,7 +29,14 @@ int read_bundle_header_fd(int fd, struct bundle_header *header,\n int create_bundle(struct repository *r, const char *path,\n \t\t int argc, const char **argv, struct strvec *pack_options,\n \t\t int version);\n-int verify_bundle(struct repository *r, struct bundle_header *header, int verbose);\n+\n+enum verify_bundle_flags {\n+\tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n+\tVERIFY_BUNDLE_SKIP_REACHABLE = (1 << 1)\n+};\n+\n+int verify_bundle(struct repository *r, struct bundle_header *header,\n+\t\t enum verify_bundle_flags flags);\n \n /**\n * Unbundle after reading the header with read_bundle_header().\n@@ -40,9 +47,13 @@ int verify_bundle(struct repository *r, struct bundle_header *header, int verbos\n * Provide \"extra_index_pack_args\" to pass any extra arguments\n * (e.g. \"-v\" for verbose/progress), NULL otherwise. The provided\n * \"extra_index_pack_args\" (if any) will be strvec_clear()'d for you.\n+ *\n+ * Before unbundling, this method will call verify_bundle() with the\n+ * given 'flags'.\n */\n int unbundle(struct repository *r, struct bundle_header *header,\n-\t int bundle_fd, struct strvec *extra_index_pack_args);\n+\t int bundle_fd, struct strvec *extra_index_pack_args,\n+\t enum verify_bundle_flags flags);\n int list_bundle_refs(struct bundle_header *header,\n \t\tint argc, const char **argv);\n \ndiff --git a/transport.c b/transport.c\nindex 52db7a3cb09..c5d3042731a 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -178,7 +178,7 @@ static int fetch_refs_from_bundle(struct transport *transport,\n \tif (!data->get_refs_from_bundle_called)\n \t\tget_refs_from_bundle_inner(transport);\n \tret = unbundle(the_repository, &data->header, data->fd,\n-\t\t &extra_index_pack_args);\n+\t\t &extra_index_pack_args, 0);\n \ttransport->hash_algo = data->header.hash_algo;\n \treturn ret;\n }\n-- \ngitgitgadget\n\n"},{"id":"464491","messageId":"6b9c764c6b3b6e69fef47716bdbd8d705e4d479a.1665417859.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v4.git.1665417859.gitgitgadget@gmail.com","subject":"[PATCH v4 09/11] bundle-uri: fetch a list of bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-10T16:04:17Z","receivedAt":"2022-10-10T16:05:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen the content at a given bundle URI is not understood as a bundle\n(based on inspecting the initial content), then Git currently gives up\nand ignores that content. Independent bundle providers may want to split\nup the bundle content into multiple bundles, but still make them\navailable from a single URI.\n\nTeach Git to attempt parsing the bundle URI content as a Git config file\nproviding the key=value pairs for a bundle list. Git then looks at the\nmode of the list to see if ANY single bundle is sufficient or if ALL\nbundles are required. The content at the selected URIs are downloaded\nand the content is inspected again, creating a recursive process.\n\nTo guard the recursion against malformed or malicious content, limit the\nrecursion depth to a reasonable four for now. This can be converted to a\nconfigured value in the future if necessary. The value of four is twice\nas high as expected to be useful (a bundle list is unlikely to point to\nmore bundle lists).\n\nTo test this scenario, create an interesting bundle topology where three\nincremental bundles are built on top of a single full bundle. By using a\nmerge commit, the two middle bundles are \"independent\" in that they do\nnot require each other in order to unbundle themselves. They each only\nneed the base bundle. The bundle containing the merge commit requires\nboth of the middle bundles, though. This leads to some interesting\ndecisions when unbundling, especially when we later implement heuristics\nthat promote downloading bundles until the prerequisite commits are\nsatisfied.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 203 ++++++++++++++++++++++++++---\n bundle-uri.h | 13 ++\n t/t5558-clone-bundle-uri.sh | 248 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 448 insertions(+), 16 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex ad5baabdd94..c0a6fb05fad 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -37,6 +37,8 @@ static int clear_remote_bundle_info(struct remote_bundle_info *bundle,\n {\n \tFREE_AND_NULL(bundle->id);\n \tFREE_AND_NULL(bundle->uri);\n+\tFREE_AND_NULL(bundle->file);\n+\tbundle->unbundled = 0;\n \treturn 0;\n }\n \n@@ -340,18 +342,117 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \treturn result;\n }\n \n+struct bundle_list_context {\n+\tstruct repository *r;\n+\tstruct bundle_list *list;\n+\tenum bundle_list_mode mode;\n+\tint count;\n+\tint depth;\n+};\n+\n+/*\n+ * This early definition is necessary because we use indirect recursion:\n+ *\n+ * While iterating through a bundle list that was downloaded as part\n+ * of fetch_bundle_uri_internal(), iterator methods eventually call it\n+ * again, but with depth + 1.\n+ */\n+static int fetch_bundle_uri_internal(struct repository *r,\n+\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t int depth,\n+\t\t\t\t struct bundle_list *list);\n+\n+static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data)\n+{\n+\tint res;\n+\tstruct bundle_list_context *ctx = data;\n+\n+\tif (ctx->mode == BUNDLE_MODE_ANY && ctx->count)\n+\t\treturn 0;\n+\n+\tres = fetch_bundle_uri_internal(ctx->r, bundle, ctx->depth + 1, ctx->list);\n+\n+\t/*\n+\t * Only increment count if the download succeeded. If our mode is\n+\t * BUNDLE_MODE_ANY, then we will want to try other URIs in the\n+\t * list in case they work instead.\n+\t */\n+\tif (!res)\n+\t\tctx->count++;\n+\n+\t/*\n+\t * To be opportunistic as possible, we continue iterating and\n+\t * download as many bundles as we can, so we can apply the ones\n+\t * that work, even in BUNDLE_MODE_ALL mode.\n+\t */\n+\treturn 0;\n+}\n+\n+static int download_bundle_list(struct repository *r,\n+\t\t\t\tstruct bundle_list *local_list,\n+\t\t\t\tstruct bundle_list *global_list,\n+\t\t\t\tint depth)\n+{\n+\tstruct bundle_list_context ctx = {\n+\t\t.r = r,\n+\t\t.list = global_list,\n+\t\t.depth = depth + 1,\n+\t\t.mode = local_list->mode,\n+\t};\n+\n+\treturn for_all_bundles_in_list(local_list, download_bundle_to_file, &ctx);\n+}\n+\n+static int fetch_bundle_list_in_config_format(struct repository *r,\n+\t\t\t\t\t struct bundle_list *global_list,\n+\t\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t\t int depth)\n+{\n+\tint result;\n+\tstruct bundle_list list_from_bundle;\n+\n+\tinit_bundle_list(&list_from_bundle);\n+\n+\tif ((result = bundle_uri_parse_config_format(bundle->uri,\n+\t\t\t\t\t\t bundle->file,\n+\t\t\t\t\t\t &list_from_bundle)))\n+\t\tgoto cleanup;\n+\n+\tif (list_from_bundle.mode == BUNDLE_MODE_NONE) {\n+\t\twarning(_(\"unrecognized bundle mode from URI '%s'\"),\n+\t\t\tbundle->uri);\n+\t\tresult = -1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif ((result = download_bundle_list(r, &list_from_bundle,\n+\t\t\t\t\t global_list, depth)))\n+\t\tgoto cleanup;\n+\n+cleanup:\n+\tclear_bundle_list(&list_from_bundle);\n+\treturn result;\n+}\n+\n /**\n * This limits the recursion on fetch_bundle_uri_internal() when following\n * bundle lists.\n */\n static int max_bundle_uri_depth = 4;\n \n+/**\n+ * Recursively download all bundles advertised at the given URI\n+ * to files. If the file is a bundle, then add it to the given\n+ * 'list'. Otherwise, expect a bundle list and recurse on the\n+ * URIs in that list according to the list mode (ANY or ALL).\n+ */\n static int fetch_bundle_uri_internal(struct repository *r,\n-\t\t\t\t const char *uri,\n-\t\t\t\t int depth)\n+\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t int depth,\n+\t\t\t\t struct bundle_list *list)\n {\n \tint result = 0;\n-\tchar *filename;\n+\tstruct remote_bundle_info *bcopy;\n \n \tif (depth >= max_bundle_uri_depth) {\n \t\twarning(_(\"exceeded bundle URI recursion limit (%d)\"),\n@@ -359,36 +460,106 @@ static int fetch_bundle_uri_internal(struct repository *r,\n \t\treturn -1;\n \t}\n \n-\tif (!(filename = find_temp_filename())) {\n+\tif (!bundle->file &&\n+\t !(bundle->file = find_temp_filename())) {\n \t\tresult = -1;\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = copy_uri_to_file(filename, uri))) {\n-\t\twarning(_(\"failed to download bundle from URI '%s'\"), uri);\n+\tif ((result = copy_uri_to_file(bundle->file, bundle->uri))) {\n+\t\twarning(_(\"failed to download bundle from URI '%s'\"), bundle->uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = !is_bundle(filename, 0))) {\n-\t\twarning(_(\"file at URI '%s' is not a bundle\"), uri);\n+\tif ((result = !is_bundle(bundle->file, 1))) {\n+\t\tresult = fetch_bundle_list_in_config_format(\n+\t\t\t\tr, list, bundle, depth);\n+\t\tif (result)\n+\t\t\twarning(_(\"file at URI '%s' is not a bundle or bundle list\"),\n+\t\t\t\tbundle->uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = unbundle_from_file(r, filename))) {\n-\t\twarning(_(\"failed to unbundle bundle from URI '%s'\"), uri);\n-\t\tgoto cleanup;\n-\t}\n+\t/* Copy the bundle and insert it into the global list. */\n+\tCALLOC_ARRAY(bcopy, 1);\n+\tbcopy->id = xstrdup(bundle->id);\n+\tbcopy->file = xstrdup(bundle->file);\n+\thashmap_entry_init(&bcopy->ent, strhash(bcopy->id));\n+\thashmap_add(&list->bundles, &bcopy->ent);\n \n cleanup:\n-\tif (filename)\n-\t\tunlink(filename);\n-\tfree(filename);\n+\tif (result && bundle->file)\n+\t\tunlink(bundle->file);\n \treturn result;\n }\n \n+/**\n+ * This loop iterator breaks the loop with nonzero return code on the\n+ * first successful unbundling of a bundle.\n+ */\n+static int attempt_unbundle(struct remote_bundle_info *info, void *data)\n+{\n+\tstruct repository *r = data;\n+\n+\tif (!info->file || info->unbundled)\n+\t\treturn 0;\n+\n+\tif (!unbundle_from_file(r, info->file)) {\n+\t\tinfo->unbundled = 1;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int unbundle_all_bundles(struct repository *r,\n+\t\t\t\tstruct bundle_list *list)\n+{\n+\t/*\n+\t * Iterate through all bundles looking for ones that can\n+\t * successfully unbundle. If any succeed, then perhaps another\n+\t * will succeed in the next attempt.\n+\t *\n+\t * Keep in mind that a non-zero result for the loop here means\n+\t * the loop terminated early on a successful unbundling, which\n+\t * signals that we can try again.\n+\t */\n+\twhile (for_all_bundles_in_list(list, attempt_unbundle, r)) ;\n+\n+\treturn 0;\n+}\n+\n+static int unlink_bundle(struct remote_bundle_info *info, void *data)\n+{\n+\tif (info->file)\n+\t\tunlink_or_warn(info->file);\n+\treturn 0;\n+}\n+\n int fetch_bundle_uri(struct repository *r, const char *uri)\n {\n-\treturn fetch_bundle_uri_internal(r, uri, 0);\n+\tint result;\n+\tstruct bundle_list list;\n+\tstruct remote_bundle_info bundle = {\n+\t\t.uri = xstrdup(uri),\n+\t\t.id = xstrdup(\"\"),\n+\t};\n+\n+\tinit_bundle_list(&list);\n+\n+\t/* If a bundle is added to this global list, then it is required. */\n+\tlist.mode = BUNDLE_MODE_ALL;\n+\n+\tif ((result = fetch_bundle_uri_internal(r, &bundle, 0, &list)))\n+\t\tgoto cleanup;\n+\n+\tresult = unbundle_all_bundles(r, &list);\n+\n+cleanup:\n+\tfor_all_bundles_in_list(&list, unlink_bundle, NULL);\n+\tclear_bundle_list(&list);\n+\tclear_remote_bundle_info(&bundle, NULL);\n+\treturn result;\n }\n \n /**\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex bc13d4c9929..4dbc269823c 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -28,6 +28,19 @@ struct remote_bundle_info {\n \t * if there was no table of contents.\n \t */\n \tchar *uri;\n+\n+\t/**\n+\t * If the bundle has been downloaded, then 'file' is a\n+\t * filename storing its contents. Otherwise, 'file' is\n+\t * NULL.\n+\t */\n+\tchar *file;\n+\n+\t/**\n+\t * If the bundle has been unbundled successfully, then\n+\t * this boolean is true.\n+\t */\n+\tunsigned unbundled:1;\n };\n \n #define REMOTE_BUNDLE_INFO_INIT { 0 }\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex ad666a2d28a..a86dc04f528 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -41,6 +41,195 @@ test_expect_success 'clone with file:// bundle' '\n \ttest_cmp expect actual\n '\n \n+# To get interesting tests for bundle lists, we need to construct a\n+# somewhat-interesting commit history.\n+#\n+# ---------------- bundle-4\n+#\n+# 4\n+# / \\\n+# ----|---|------- bundle-3\n+# | |\n+# | 3\n+# | |\n+# ----|---|------- bundle-2\n+# | |\n+# 2 |\n+# | |\n+# ----|---|------- bundle-1\n+# \\ /\n+# 1\n+# |\n+# (previous commits)\n+test_expect_success 'construct incremental bundle list' '\n+\t(\n+\t\tcd clone-from &&\n+\t\tgit checkout -b base &&\n+\t\ttest_commit 1 &&\n+\t\tgit checkout -b left &&\n+\t\ttest_commit 2 &&\n+\t\tgit checkout -b right base &&\n+\t\ttest_commit 3 &&\n+\t\tgit checkout -b merge left &&\n+\t\tgit merge right -m \"4\" &&\n+\n+\t\tgit bundle create bundle-1.bundle base &&\n+\t\tgit bundle create bundle-2.bundle base..left &&\n+\t\tgit bundle create bundle-3.bundle base..right &&\n+\t\tgit bundle create bundle-4.bundle merge --not left right\n+\t)\n+'\n+\n+test_expect_success 'clone bundle list (file, no heuristic)' '\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-3.bundle\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-4.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-list-file &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-list-file cat-file --batch-check refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/merge\n+\trefs/bundles/right\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone bundle list (file, all mode, some failures)' '\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-0\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-0.bundle\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\n+\t# No bundle-3 means bundle-4 will not apply.\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-4.bundle\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-5\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-5.bundle\n+\tEOF\n+\n+\tGIT_TRACE2_PERF=1 \\\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-all-some &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-all-some cat-file --batch-check refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone bundle list (file, all mode, all failures)' '\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-0\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-0.bundle\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-5\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-5.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-all-fail &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-all-fail cat-file --batch-check refs &&\n+\t! grep \"refs/bundles/\" refs\n+'\n+\n+test_expect_success 'clone bundle list (file, any mode)' '\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = any\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-0\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-0.bundle\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-5\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-5.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-any-file &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-any-file cat-file --batch-check refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone bundle list (file, any mode, all failures)' '\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = any\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-0\"]\n+\t\turi = $HTTPD_URL/bundle-0.bundle\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-5\"]\n+\t\turi = $HTTPD_URL/bundle-5.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-any-fail &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-any-fail cat-file --batch-check refs &&\n+\t! grep \"refs/bundles/\" refs\n+'\n+\n #########################################################################\n # HTTP tests begin here\n \n@@ -75,6 +264,65 @@ test_expect_success 'clone HTTP bundle' '\n \ttest_config -C clone-http log.excludedecoration refs/bundle/\n '\n \n+test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n+\tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = $HTTPD_URL/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = $HTTPD_URL/bundle-2.bundle\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = $HTTPD_URL/bundle-3.bundle\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = $HTTPD_URL/bundle-4.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" clone-from clone-list-http &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-list-http cat-file --batch-check \"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = any\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-0\"]\n+\t\turi = $HTTPD_URL/bundle-0.bundle\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = $HTTPD_URL/bundle-1.bundle\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-5\"]\n+\t\turi = $HTTPD_URL/bundle-5.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" clone-from clone-any-http &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-any-http cat-file --batch-check refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/merge\n+\trefs/bundles/right\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n # Do not add tests here unless they use the HTTP server, as they will\n # not run unless the HTTP dependencies exist.\n \n-- \ngitgitgadget\n\n"},{"id":"464492","messageId":"1cae309662457be8a006de3c317a4f5f09438930.1665417859.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v4.git.1665417859.gitgitgadget@gmail.com","subject":"[PATCH v4 10/11] bundle-uri: quiet failed unbundlings","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-10T16:04:18Z","receivedAt":"2022-10-10T16:05:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen downloading a list of bundles in \"all\" mode, Git has no\nunderstanding of the dependencies between the bundles. Git attempts to\nunbundle the bundles in some order, but some may not pass the\nverify_bundle() step because of missing prerequisites. This is passed as\nerror messages to the user, even when they eventually succeed in later\nattempts after their dependent bundles are unbundled.\n\nAdd a new VERIFY_BUNDLE_QUIET flag to verify_bundle() that avoids the\nerror messages from the missing prerequisite commits. The method still\nreturns the number of missing prerequisit commits, allowing callers to\nunbundle() to notice that the bundle failed to apply.\n\nUse this flag in bundle-uri.c and test that the messages go away for\n'git clone --bundle-uri' commands.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 2 +-\n bundle.c | 10 ++++++++--\n bundle.h | 3 ++-\n t/t5558-clone-bundle-uri.sh | 25 ++++++++++++++++++++-----\n 4 files changed, 31 insertions(+), 9 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex c0a6fb05fad..18b993c207f 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -309,7 +309,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t * the prerequisite commits.\n \t */\n \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n-\t\t\t VERIFY_BUNDLE_SKIP_REACHABLE)))\n+\t\t\t VERIFY_BUNDLE_SKIP_REACHABLE | VERIFY_BUNDLE_QUIET)))\n \t\treturn 1;\n \n \t/*\ndiff --git a/bundle.c b/bundle.c\nindex 36ffeb1e0eb..143e7c4508f 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -218,7 +218,10 @@ int verify_bundle(struct repository *r,\n \t\t\tadd_pending_object(&revs, o, name);\n \t\t\tcontinue;\n \t\t}\n-\t\tif (++ret == 1)\n+\t\tret++;\n+\t\tif (flags & VERIFY_BUNDLE_QUIET)\n+\t\t\tcontinue;\n+\t\tif (ret == 1)\n \t\t\terror(\"%s\", message);\n \t\terror(\"%s %s\", oid_to_hex(oid), name);\n \t}\n@@ -246,7 +249,10 @@ int verify_bundle(struct repository *r,\n \t\tassert(o); /* otherwise we'd have returned early */\n \t\tif (o->flags & SHOWN)\n \t\t\tcontinue;\n-\t\tif (++ret == 1)\n+\t\tret++;\n+\t\tif (flags & VERIFY_BUNDLE_QUIET)\n+\t\t\tcontinue;\n+\t\tif (ret == 1)\n \t\t\terror(\"%s\", message);\n \t\terror(\"%s %s\", oid_to_hex(oid), name);\n \t}\ndiff --git a/bundle.h b/bundle.h\nindex 9f798c00d93..ba453404163 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -32,7 +32,8 @@ int create_bundle(struct repository *r, const char *path,\n \n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n-\tVERIFY_BUNDLE_SKIP_REACHABLE = (1 << 1)\n+\tVERIFY_BUNDLE_SKIP_REACHABLE = (1 << 1),\n+\tVERIFY_BUNDLE_QUIET = (1 << 2),\n };\n \n int verify_bundle(struct repository *r, struct bundle_header *header,\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex a86dc04f528..9b159078386 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -99,7 +99,10 @@ test_expect_success 'clone bundle list (file, no heuristic)' '\n \t\turi = file://$(pwd)/clone-from/bundle-4.bundle\n \tEOF\n \n-\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-list-file &&\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from clone-list-file 2>err &&\n+\t! grep \"Repository lacks these prerequisite commits\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-list-file cat-file --batch-check err &&\n+\t! grep \"Repository lacks these prerequisite commits\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-all-some cat-file --batch-check err &&\n+\t! grep \"Repository lacks these prerequisite commits\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-all-fail cat-file --batch-check err &&\n+\t! grep \"Repository lacks these prerequisite commits\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-any-file cat-file --batch-check err &&\n+\t! grep \"Repository lacks these prerequisite commits\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-list-http cat-file --batch-check \n\nWhen downloading bundles from a git-remote-https subprocess, the bundle\nURI logic wants to be opportunistic and download as much as possible and\nwork with what did succeed. This is particularly important in the \"any\"\nmode, where any single bundle success will work.\n\nIf the URI is not available, the git-remote-https process will die()\nwith a \"fatal:\" error message, even though that error is not actually\nfatal to the super process. Since stderr is passed through, it looks\nlike a fatal error to the user.\n\nSuppress stderr to avoid these errors from bubbling to the surface. The\nbundle URI API adds its own warning() messages on these failures.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 1 +\n t/t5558-clone-bundle-uri.sh | 16 ++++++++++++++--\n 2 files changed, 15 insertions(+), 2 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 18b993c207f..6bfba95f872 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -230,6 +230,7 @@ static int download_https_uri_to_file(const char *file, const char *uri)\n \tint found_get = 0;\n \n \tstrvec_pushl(&cp.args, \"git-remote-https\", uri, NULL);\n+\tcp.err = -1;\n \tcp.in = -1;\n \tcp.out = -1;\n \ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 9b159078386..9155f31fa2c 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -147,6 +147,8 @@ test_expect_success 'clone bundle list (file, all mode, some failures)' '\n \tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n \t\tclone-from clone-all-some 2>err &&\n \t! grep \"Repository lacks these prerequisite commits\" err &&\n+\t! grep \"fatal\" err &&\n+\tgrep \"warning: failed to download bundle from URI\" err &&\n \n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-all-some cat-file --batch-check err &&\n \t! grep \"Repository lacks these prerequisite commits\" err &&\n+\t! grep \"fatal\" err &&\n+\tgrep \"warning: failed to download bundle from URI\" err &&\n \n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-all-fail cat-file --batch-check err &&\n+\t! grep \"fatal\" err &&\n+\tgrep \"warning: failed to download bundle from URI\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-any-fail cat-file --batch-check err &&\n+\t! grep \"fatal\" err &&\n+\tgrep \"warning: failed to download bundle from URI\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-any-http cat-file --batch-check writes:\n\n> From: Derrick Stolee \n>\n> The verify_bundle() method checks if a bundle can be applied to a given\n> repository. This not only verifies that certain commits exist in the\n> repository, but Git also checks that these commits are reachable.\n>\n> This behavior dates back to the original git-bundle builtin written in\n> 2e0afafebd8 (Add git-bundle: move objects and references by archive,\n> 2007-02-22), but the message does not go into detail why the\n> reachability check is important.\n>\n> Since verify_bundle() is called from unbundle(), we need to add an\n> option to pipe the flags through that method.\n\nAll makes sense.\n\n> When unbundling from a list of bundles, Git will create refs that point\n> to the tips of the latest bundle, which makes this reachability walk\n> succeed, in theory. However, the loose refs cache does not get\n> invalidated and hence the reachability walk fails. By disabling the\n> reachability walk in the bundle URI code, we can get around this\n> reachability check.\n\nThe above makes it sound like the real culprit is that cache goes\nout of sync and the presented solution is a workaround; readers are\nleft in suspense if the \"real\" solution (as opposed to a workaround)\nwould come in a later step or in a future series.\n\n> diff --git a/bundle-uri.c b/bundle-uri.c\n> index 8a7c11c6393..ad5baabdd94 100644\n> --- a/bundle-uri.c\n> +++ b/bundle-uri.c\n> @@ -301,7 +301,13 @@ static int unbundle_from_file(struct repository *r, const char *file)\n> \tif ((bundle_fd = read_bundle_header(file, &header)) < 0)\n> \t\treturn 1;\n> \n> -\tif ((result = unbundle(r, &header, bundle_fd, NULL)))\n> +\t/*\n> +\t * Skip the reachability walk here, since we will be adding\n> +\t * a reachable ref pointing to the new tips, which will reach\n> +\t * the prerequisite commits.\n> +\t */\n> +\tif ((result = unbundle(r, &header, bundle_fd, NULL,\n> +\t\t\t VERIFY_BUNDLE_SKIP_REACHABLE)))\n> \t\treturn 1;\n\nThis is not a new problem introduced in this new round, but if we\nare updating this, can we fix it to omit assignment inside if\ncondition?\n\n * result is initialized to 0.\n\n * when unbundle returns non-zero, it is assigned to result and the\n function returns immediately, discarding whatever was assigned to\n the variable.\n\n * if unbundle returns zero, it is assigned to result and the\n control continues from here. We know result is set to 0, but\n then that is what it was initialized earlier.\n\n"},{"id":"464514","messageId":"08877824-b454-df37-1819-edee34919f95@github.com","threadId":"58342","inReplyTo":"xmqqbkqj4mj7.fsf@gitster.g","subject":"Re: [PATCH v4 08/11] bundle: add flags to verify_bundle(), skip walk","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-10-10T18:13:10Z","receivedAt":"2022-10-10T18:13:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/10/2022 1:27 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n\n>> When unbundling from a list of bundles, Git will create refs that point\n>> to the tips of the latest bundle, which makes this reachability walk\n>> succeed, in theory. However, the loose refs cache does not get\n>> invalidated and hence the reachability walk fails. By disabling the\n>> reachability walk in the bundle URI code, we can get around this\n>> reachability check.\n> \n> The above makes it sound like the real culprit is that cache goes\n> out of sync and the presented solution is a workaround; readers are\n> left in suspense if the \"real\" solution (as opposed to a workaround)\n> would come in a later step or in a future series.\n\nI've been going over the refs code multiple times today trying to\nfix this \"real\" culprit, with no luck. I can share this interesting\npoint:\n\n * The initial loop over the bundles tries to apply each, but the\n prerequisite objects are not present so we never reach the revision\n walk. A refs/bundle/* ref is added via update_ref().\n\n * The second loop over the bundles tries to apply each, but the only\n bundle with its prerequisites present also finds the commits as\n reachable (this must be where the loose ref cache is populated).\n Then, a refs/bundle/* ref is added via update_ref().\n\n * The third loop over the bundles finds a bundle whose prerequisites\n are present, but verify_bundle() rejected it because those commits\n were not seen from any ref.\n\nOther than identifying that issue, I was unable to track down exactly\nwhat is happening here or offer a fix. I had considered inserting\nmore cache frees deep in the refs code, but I wasn't sure what effect\nthat would have across the wider system.\n\n>> diff --git a/bundle-uri.c b/bundle-uri.c\n>> index 8a7c11c6393..ad5baabdd94 100644\n>> --- a/bundle-uri.c\n>> +++ b/bundle-uri.c\n>> @@ -301,7 +301,13 @@ static int unbundle_from_file(struct repository *r, const char *file)\n>> \tif ((bundle_fd = read_bundle_header(file, &header)) < 0)\n>> \t\treturn 1;\n>> \n>> -\tif ((result = unbundle(r, &header, bundle_fd, NULL)))\n>> +\t/*\n>> +\t * Skip the reachability walk here, since we will be adding\n>> +\t * a reachable ref pointing to the new tips, which will reach\n>> +\t * the prerequisite commits.\n>> +\t */\n>> +\tif ((result = unbundle(r, &header, bundle_fd, NULL,\n>> +\t\t\t VERIFY_BUNDLE_SKIP_REACHABLE)))\n>> \t\treturn 1;\n> \n> This is not a new problem introduced in this new round, but if we\n> are updating this, can we fix it to omit assignment inside if\n> condition?\n> \n> * result is initialized to 0.\n> \n> * when unbundle returns non-zero, it is assigned to result and the\n> function returns immediately, discarding whatever was assigned to\n> the variable.\n> \n> * if unbundle returns zero, it is assigned to result and the\n> control continues from here. We know result is set to 0, but\n> then that is what it was initialized earlier.\n \nSince we are not \"trusting\" the integer result of unbundle, we\ncan definitely stop this assignment in the if.\n\nThanks,\n-Stolee\n"},{"id":"464521","messageId":"xmqqk05734kr.fsf@gitster.g","threadId":"58342","inReplyTo":"08877824-b454-df37-1819-edee34919f95@github.com","subject":"Re: [PATCH v4 08/11] bundle: add flags to verify_bundle(), skip walk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-10T18:40:20Z","receivedAt":"2022-10-10T18:40:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> I've been going over the refs code multiple times today trying to\n> fix this \"real\" culprit, with no luck. I can share this interesting\n> point:\n>\n> * The initial loop over the bundles tries to apply each, but the\n> prerequisite objects are not present so we never reach the revision\n> walk. A refs/bundle/* ref is added via update_ref().\n>\n> * The second loop over the bundles tries to apply each, but the only\n> bundle with its prerequisites present also finds the commits as\n> reachable (this must be where the loose ref cache is populated).\n> Then, a refs/bundle/* ref is added via update_ref().\n>\n> * The third loop over the bundles finds a bundle whose prerequisites\n> are present, but verify_bundle() rejected it because those commits\n> were not seen from any ref.\n>\n> Other than identifying that issue, I was unable to track down exactly\n> what is happening here or offer a fix. I had considered inserting\n> more cache frees deep in the refs code, but I wasn't sure what effect\n> that would have across the wider system.\n\nOK. That certainly is understandable.\n\nAs a comment in the proposed log message that BUNDLE_SKIP_REACHABLE\nbit is a band aid papering over a problem we punted in this series,\nto guide future developers, I think what you wrote is sufficient.\nWe do not want them to think that skipping the check is our\npreferred longer term solution and add their own hack to keep\nskipping the check when they resolve \"the real culprit\".\n\nThanks.\n"},{"id":"464629","messageId":"d85704ef-97e8-57aa-9711-01d00b74d36e@github.com","threadId":"58342","inReplyTo":"xmqqk05734kr.fsf@gitster.g","subject":"Re: [PATCH v4 08/11] bundle: add flags to verify_bundle(), skip walk","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-10-11T19:04:26Z","receivedAt":"2022-10-11T19:04:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/10/2022 2:40 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> I've been going over the refs code multiple times today trying to\n>> fix this \"real\" culprit, with no luck. I can share this interesting\n>> point:\n>>\n>> * The initial loop over the bundles tries to apply each, but the\n>> prerequisite objects are not present so we never reach the revision\n>> walk. A refs/bundle/* ref is added via update_ref().\n>>\n>> * The second loop over the bundles tries to apply each, but the only\n>> bundle with its prerequisites present also finds the commits as\n>> reachable (this must be where the loose ref cache is populated).\n>> Then, a refs/bundle/* ref is added via update_ref().\n>>\n>> * The third loop over the bundles finds a bundle whose prerequisites\n>> are present, but verify_bundle() rejected it because those commits\n>> were not seen from any ref.\n>>\n>> Other than identifying that issue, I was unable to track down exactly\n>> what is happening here or offer a fix. I had considered inserting\n>> more cache frees deep in the refs code, but I wasn't sure what effect\n>> that would have across the wider system.\n> \n> OK. That certainly is understandable.\n> \n> As a comment in the proposed log message that BUNDLE_SKIP_REACHABLE\n> bit is a band aid papering over a problem we punted in this series,\n> to guide future developers, I think what you wrote is sufficient.\n> We do not want them to think that skipping the check is our\n> preferred longer term solution and add their own hack to keep\n> skipping the check when they resolve \"the real culprit\".\n\nI have discovered the real culprit, and my expectation was incorrect\nabout the loose ref cache. The key issue was that I was looking at\nthis loop:\n\n\ti = req_nr;\n\twhile (i && (commit = get_revision(&revs)))\n\t\tif (commit->object.flags & PREREQ_MARK)\n\t\t\ti--;\n\nand noticing that only one commit was being visited. I was not \nseeing the actually-important commit. But it wasn't the revision\nwalk's fault. The loop was terminating because \"i\" was reaching\nzero!\n\nIt turns out that verify_bundles() is not clearing the PREREQ_MARK\nflag, so multiple runs would incorrectly hit this short-circuit\nand terminate the walk early.\n\nI'll replace this patch with the correct fix soon.\n\nThanks,\n-Stolee\n"},{"id":"464695","messageId":"pull.1333.v5.git.1665579160.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v4.git.1665417859.gitgitgadget@gmail.com","subject":"[PATCH v5 00/12] Bundle URIs III: Parse and download from bundle lists","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T12:52:27Z","receivedAt":"2022-10-12T12:52:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the third series building the bundle URI feature. It is built on top\nof ds/bundle-uri-clone, which introduced 'git clone --bundle-uri=' where is\na URI to a bundle file. This series adds the capability of downloading and\nparsing a bundle list and then downloading the URIs in that list.\n\nThe core functionality of bundle lists is implemented by creating data\nstructures from a list of key-value pairs. These pairs can come from a\nplain-text file in Git config format, but in the future, we will support the\nlist being supplied by packet lines over Git's protocol v2 in the\n'bundle-uri' command (reserved for the next series).\n\nThe patches are organized in this way (updated for v4):\n\n 1. Patch 1 is a cleanup from the previous part. This allows us to simplify\n our bundle list data structure slightly.\n\n 2. Patches 2-3 create the bundle list data structures and the logic for\n populating the list from key-value pairs.\n\n 3. Patches 4-5 teach Git to parse \"key=value\" lines to construct a bundle\n list. Add unit tests that ensure this logic constructs lists correctly.\n These patches are adapted from Ævar's RFC [1] and were previously seen\n in my combined RFC [2].\n\n 4. Patch 6 teaches Git to parse Git config files into bundle lists.\n\n 5. Patches 7-9 implement the ability to download a bundle list and\n recursively download the contained bundles (and possibly the bundle\n lists within). This is limited by a constant depth to avoid issues with\n cycles or otherwise incorrectly configured bundle lists. This also fixes\n a previous bug when running verify_bundle() multiple times in the same\n process, as it did not clear the PREREQ_MARK flag upon leaving (see\n patch 8).\n\n 6. Patches 10-12 suppress unhelpful warnings from user visibility.\n\n[1]\nhttps://lore.kernel.org/git/RFC-cover-v2-00.36-00000000000-20220418T165545Z-avarab@gmail.com/\n\n[2]\nhttps://lore.kernel.org/git/pull.1234.git.1653072042.gitgitgadget@gmail.com/\n\nAt the end of this series, users can bootstrap clones using 'git clone\n--bundle-uri= ' where points to a bundle list instead of a single bundle\nfile.\n\nAs outlined in the design document [1], the next steps after this are:\n\n 1. Implement the protocol v2 verb, re-using the bundle list logic from (2).\n Use this to auto-discover bundle URIs during 'git clone' (behind a\n config option). [2]\n 2. Implement the 'creationToken' heuristic, allowing incremental 'git\n fetch' commands to download a bundle list from a configured URI, and\n only download bundles that are new based on the creation token values.\n [3]\n\nI have prepared some of this work as pull requests on my personal fork so\ncurious readers can look ahead to where we are going:\n\n[3]\nhttps://lore.kernel.org/git/pull.1248.v3.git.1658757188.gitgitgadget@gmail.com\n\n[4] https://github.com/derrickstolee/git/pull/21\n\n[5] https://github.com/derrickstolee/git/pull/22\n\n\nUpdates in v5\n=============\n\n * The bug about verify_bundle() not working multile times in the same\n process is fixed without removing the revision walk. Instead, more flags\n needed to be removed as the method cleaned up after itself.\n\n\nUpdates in v4\n=============\n\n * Properly updated the patch outline.\n\n * Jonathan Tan asked for more tests, and this revealed some interesting\n behaviors which I have now either fixed or made explicit:\n \n 1. In \"all\" mode, we try to download and apply all bundles. Do not fail\n if a single bundle download fails.\n 2. Previously, not all bundles were being applied, and this was noticed\n by the added checks for the refs/bundles/* refs at the end of the\n tests. This revealed the need for removing the reachability walk from\n verify_bundle() since the written refs/bundles/* refs were not being\n picked up by the loose ref cache. Since removing the reachability walk\n seemed like the faster (for users) option, I went that direction.\n 3. While running those tests and examining the output carefully, I\n noticed several error messages related to missing prerequisites due to\n attempting unbundling in a random order. This doesn't appear in the\n later creationToken version, so I hadn't noticed it at the tip of my\n local work. These messages are removed with a new quiet mode for\n verify_bundle().\n\n\nUpdates in v3\n=============\n\n * Fixed a comment about a return value of -1.\n * Fixed and tested scenario where early URIs fail in \"any\" mode and Git\n should try the rest of the list.\n * Instead of using 'success_count' and 'failure_count', use the iterator\n return value to terminate the \"all\" mode loop early.\n\n\nUpdates in v2\n=============\n\nThank you to all of the voices who chimed in on the previous version. I'm\nsorry it took so long for me to get a new version.\n\n * I've done a rather thorough overhaul to minimize how often later patches\n rewrite portions of earlier patches.\n\n * We no longer use a strbuf in struct remote_bundle_info. Instead, use a\n 'char *' and only in the patch where it is first used.\n\n * The config documentation is more clearly indicating that the bundle.*\n section has no effect in the repository config (at the moment, which will\n change in the next series).\n\n * The bundle.version value is now parsed using git_parse_int().\n\n * The config key is now parsed using parse_config_key().\n\n * Commit messages clarify more about the context of the change in the\n bigger picture of the bundle URI effort.\n\n * Some printf()s are correctly changed to fprintf()s.\n\n * The test helper CLI is unified across the two modes. They both take a\n filename now.\n\n * The count of downloaded bundles is now only updated after a successful\n download, allowing the \"any\" mode to keep trying after a failure.\n\nThanks,\n\n * Stolee\n\nDerrick Stolee (10):\n bundle-uri: use plain string in find_temp_filename()\n bundle-uri: create bundle_list struct and helpers\n bundle-uri: create base key-value pair parsing\n bundle-uri: parse bundle list in config format\n bundle-uri: limit recursion depth for bundle lists\n bundle: properly clear all revision flags\n bundle-uri: fetch a list of bundles\n bundle: add flags to verify_bundle()\n bundle-uri: quiet failed unbundlings\n bundle-uri: suppress stderr from remote-https\n\nÆvar Arnfjörð Bjarmason (2):\n bundle-uri: create \"key=value\" line parsing\n bundle-uri: unit test \"key=value\" parsing\n\n Documentation/config.txt | 2 +\n Documentation/config/bundle.txt | 24 ++\n Makefile | 1 +\n builtin/bundle.c | 5 +-\n bundle-uri.c | 458 ++++++++++++++++++++++++++++++--\n bundle-uri.h | 93 +++++++\n bundle.c | 42 +--\n bundle.h | 15 +-\n config.c | 2 +-\n config.h | 1 +\n t/helper/test-bundle-uri.c | 95 +++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5558-clone-bundle-uri.sh | 275 +++++++++++++++++++\n t/t5750-bundle-uri-parse.sh | 171 ++++++++++++\n t/test-lib-functions.sh | 11 +\n transport.c | 2 +-\n 17 files changed, 1156 insertions(+), 43 deletions(-)\n create mode 100644 Documentation/config/bundle.txt\n create mode 100644 t/helper/test-bundle-uri.c\n create mode 100755 t/t5750-bundle-uri-parse.sh\n\n\nbase-commit: e21e663cd1942df29979d3e01f7eacb532727bb7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1333%2Fderrickstolee%2Fbundle-redo%2Flist-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1333/derrickstolee/bundle-redo/list-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/1333\n\nRange-diff vs v4:\n\n 1: 48beccb0f5e = 1: 48beccb0f5e bundle-uri: use plain string in find_temp_filename()\n 2: f0c4457951c = 2: f0c4457951c bundle-uri: create bundle_list struct and helpers\n 3: 430e01cd2a4 = 3: 430e01cd2a4 bundle-uri: create base key-value pair parsing\n 4: cd915d57f3b = 4: cd915d57f3b bundle-uri: create \"key=value\" line parsing\n 5: 4d8cac67f66 = 5: 4d8cac67f66 bundle-uri: unit test \"key=value\" parsing\n 6: 0ecae3a44b3 = 6: 0ecae3a44b3 bundle-uri: parse bundle list in config format\n 7: 7e6b32313b0 = 7: 7e6b32313b0 bundle-uri: limit recursion depth for bundle lists\n -: ----------- > 8: 8dc5a8e4e63 bundle: properly clear all revision flags\n 9: 6b9c764c6b3 = 9: 51e9b8474fb bundle-uri: fetch a list of bundles\n 8: 83f2cd893a4 ! 10: fba3a4a117e bundle: add flags to verify_bundle(), skip walk\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - bundle: add flags to verify_bundle(), skip walk\n + bundle: add flags to verify_bundle()\n \n - The verify_bundle() method checks if a bundle can be applied to a given\n - repository. This not only verifies that certain commits exist in the\n - repository, but Git also checks that these commits are reachable.\n -\n - This behavior dates back to the original git-bundle builtin written in\n - 2e0afafebd8 (Add git-bundle: move objects and references by archive,\n - 2007-02-22), but the message does not go into detail why the\n - reachability check is important.\n -\n - Since verify_bundle() is called from unbundle(), we need to add an\n - option to pipe the flags through that method.\n -\n - When unbundling from a list of bundles, Git will create refs that point\n - to the tips of the latest bundle, which makes this reachability walk\n - succeed, in theory. However, the loose refs cache does not get\n - invalidated and hence the reachability walk fails. By disabling the\n - reachability walk in the bundle URI code, we can get around this\n - reachability check.\n + The verify_bundle() method has a 'verbose' option, but we will want to\n + extend this method to have more granular control over its output. First,\n + replace this 'verbose' option with a new 'flags' option with a single\n + possible value: VERIFY_BUNDLE_VERBOSE.\n \n Signed-off-by: Derrick Stolee \n \n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *fi\n +\t * a reachable ref pointing to the new tips, which will reach\n +\t * the prerequisite commits.\n +\t */\n -+\tif ((result = unbundle(r, &header, bundle_fd, NULL,\n -+\t\t\t VERIFY_BUNDLE_SKIP_REACHABLE)))\n ++\tif ((result = unbundle(r, &header, bundle_fd, NULL, 0)))\n \t\treturn 1;\n \n \t/*\n @@ bundle.c: static int list_refs(struct string_list *r, int argc, const char **arg\n \t/*\n \t * Do fast check, then if any prereqs are missing then go line by line\n @@ bundle.c: int verify_bundle(struct repository *r,\n - \t\t\terror(\"%s\", message);\n \t\terror(\"%s %s\", oid_to_hex(oid), name);\n \t}\n --\tif (revs.pending.nr != p->nr)\n -+\tif (revs.pending.nr != p->nr ||\n -+\t (flags & VERIFY_BUNDLE_SKIP_REACHABLE))\n - \t\tgoto cleanup;\n - \treq_nr = revs.pending.nr;\n - \tsetup_revisions(2, argv, &revs, NULL);\n -@@ bundle.c: int verify_bundle(struct repository *r,\n - \t\t\tclear_commit_marks(commit, ALL_REV_FLAGS);\n - \t}\n \n -\tif (verbose) {\n +\tif (flags & VERIFY_BUNDLE_VERBOSE) {\n @@ bundle.h: int read_bundle_header_fd(int fd, struct bundle_header *header,\n +\n +enum verify_bundle_flags {\n +\tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n -+\tVERIFY_BUNDLE_SKIP_REACHABLE = (1 << 1)\n +};\n +\n +int verify_bundle(struct repository *r, struct bundle_header *header,\n 10: 1cae3096624 ! 11: 2e0bfa834f1 bundle-uri: quiet failed unbundlings\n @@ Commit message\n \n Signed-off-by: Derrick Stolee \n \n + ## builtin/bundle.c ##\n +@@ builtin/bundle.c: static int cmd_bundle_verify(int argc, const char **argv, const char *prefix) {\n + \t}\n + \tclose(bundle_fd);\n + \tif (verify_bundle(the_repository, &header,\n +-\t\t\t quiet ? 0 : VERIFY_BUNDLE_VERBOSE)) {\n ++\t\t\t quiet ? VERIFY_BUNDLE_QUIET : VERIFY_BUNDLE_VERBOSE)) {\n + \t\tret = 1;\n + \t\tgoto cleanup;\n + \t}\n +\n ## bundle-uri.c ##\n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *file)\n + \t * a reachable ref pointing to the new tips, which will reach\n \t * the prerequisite commits.\n \t */\n - \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n --\t\t\t VERIFY_BUNDLE_SKIP_REACHABLE)))\n -+\t\t\t VERIFY_BUNDLE_SKIP_REACHABLE | VERIFY_BUNDLE_QUIET)))\n +-\tif ((result = unbundle(r, &header, bundle_fd, NULL, 0)))\n ++\tif ((result = unbundle(r, &header, bundle_fd, NULL,\n ++\t\t\t VERIFY_BUNDLE_QUIET)))\n \t\treturn 1;\n \n \t/*\n @@ bundle.h: int create_bundle(struct repository *r, const char *path,\n \n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n --\tVERIFY_BUNDLE_SKIP_REACHABLE = (1 << 1)\n -+\tVERIFY_BUNDLE_SKIP_REACHABLE = (1 << 1),\n -+\tVERIFY_BUNDLE_QUIET = (1 << 2),\n ++\tVERIFY_BUNDLE_QUIET = (1 << 1),\n };\n \n int verify_bundle(struct repository *r, struct bundle_header *header,\n 11: 52a575f8a69 = 12: 5729ff2af4b bundle-uri: suppress stderr from remote-https\n\n-- \ngitgitgadget\n"},{"id":"464696","messageId":"48beccb0f5efe6f9247968cd0d4c455c23a24c53.1665579160.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v5.git.1665579160.gitgitgadget@gmail.com","subject":"[PATCH v5 01/12] bundle-uri: use plain string in find_temp_filename()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T12:52:28Z","receivedAt":"2022-10-12T12:52:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe find_temp_filename() method was created in 53a50892be2 (bundle-uri:\ncreate basic file-copy logic, 2022-08-09) and uses odb_mkstemp() to\ncreate a temporary filename. The odb_mkstemp() method uses a strbuf in\nits interface, but we do not need to continue carrying a strbuf\nthroughout the bundle URI code.\n\nConvert the find_temp_filename() method to use a 'char *' and modify its\nonly caller. This makes sense that we don't actually need to modify this\nfilename directly later, so using a strbuf is overkill.\n\nThis change will simplify the data structure for tracking a bundle list\nto use plain strings instead of strbufs.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 28 ++++++++++++++++------------\n 1 file changed, 16 insertions(+), 12 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 4a8cc74ed05..8b2f4e08c9c 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -5,22 +5,23 @@\n #include \"refs.h\"\n #include \"run-command.h\"\n \n-static int find_temp_filename(struct strbuf *name)\n+static char *find_temp_filename(void)\n {\n \tint fd;\n+\tstruct strbuf name = STRBUF_INIT;\n \t/*\n \t * Find a temporary filename that is available. This is briefly\n \t * racy, but unlikely to collide.\n \t */\n-\tfd = odb_mkstemp(name, \"bundles/tmp_uri_XXXXXX\");\n+\tfd = odb_mkstemp(&name, \"bundles/tmp_uri_XXXXXX\");\n \tif (fd < 0) {\n \t\twarning(_(\"failed to create temporary file\"));\n-\t\treturn -1;\n+\t\treturn NULL;\n \t}\n \n \tclose(fd);\n-\tunlink(name->buf);\n-\treturn 0;\n+\tunlink(name.buf);\n+\treturn strbuf_detach(&name, NULL);\n }\n \n static int download_https_uri_to_file(const char *file, const char *uri)\n@@ -141,28 +142,31 @@ static int unbundle_from_file(struct repository *r, const char *file)\n int fetch_bundle_uri(struct repository *r, const char *uri)\n {\n \tint result = 0;\n-\tstruct strbuf filename = STRBUF_INIT;\n+\tchar *filename;\n \n-\tif ((result = find_temp_filename(&filename)))\n+\tif (!(filename = find_temp_filename())) {\n+\t\tresult = -1;\n \t\tgoto cleanup;\n+\t}\n \n-\tif ((result = copy_uri_to_file(filename.buf, uri))) {\n+\tif ((result = copy_uri_to_file(filename, uri))) {\n \t\twarning(_(\"failed to download bundle from URI '%s'\"), uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = !is_bundle(filename.buf, 0))) {\n+\tif ((result = !is_bundle(filename, 0))) {\n \t\twarning(_(\"file at URI '%s' is not a bundle\"), uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = unbundle_from_file(r, filename.buf))) {\n+\tif ((result = unbundle_from_file(r, filename))) {\n \t\twarning(_(\"failed to unbundle bundle from URI '%s'\"), uri);\n \t\tgoto cleanup;\n \t}\n \n cleanup:\n-\tunlink(filename.buf);\n-\tstrbuf_release(&filename);\n+\tif (filename)\n+\t\tunlink(filename);\n+\tfree(filename);\n \treturn result;\n }\n-- \ngitgitgadget\n\n"},{"id":"464697","messageId":"f0c4457951ca5d052d10f401497917cddcbe2aa4.1665579160.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v5.git.1665579160.gitgitgadget@gmail.com","subject":"[PATCH v5 02/12] bundle-uri: create bundle_list struct and helpers","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T12:52:29Z","receivedAt":"2022-10-12T12:52:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt will likely be rare where a user uses a single bundle URI and expects\nthat URI to point to a bundle. Instead, that URI will likely be a list\nof bundles provided in some format. Alternatively, the Git server could\nadvertise a list of bundles.\n\nIn anticipation of these two ways of advertising multiple bundles,\ncreate a data structure that represents such a list. This will be\npopulated using a common API, but for now focus on what data can be\nrepresented.\n\nEach list contains a number of remote_bundle_info structs. These contain\nan 'id' that is used to uniquely identify them in the list, and also a\n'uri' that contains the location of its data. Finally, there is a strbuf\ncontaining the filename used when Git downloads the contents to disk.\n\nThe list itself stores these remote_bundle_info structs in a hashtable\nusing 'id' as the key. The order of the structs in the input is\nconsidered unimportant, but future modifications to the format and these\ndata structures will place ordering possibilities on the set. The list\nalso has a few \"global\" properties, including the version (used when\nparsing the list) and the mode. The mode is one of these two options:\n\n1. BUNDLE_MODE_ALL: all listed URIs are intended to be combined\n together. The client should download all of the advertised data to\n have a complete copy of the data.\n\n2. BUNDLE_MODE_ANY: any one listed item is sufficient to have a complete\n copy of the data. The client can choose arbitrarily from these\n options. In the future, the client may use pings to find the closest\n URI among geodistributed replicas, or use some other heuristic\n information added to the format.\n\nThis API is currently unused, but will soon be expanded with parsing\nlogic and then be consumed by the bundle URI download logic.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n bundle-uri.h | 56 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 116 insertions(+)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 8b2f4e08c9c..f9a8db221bc 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -4,6 +4,66 @@\n #include \"object-store.h\"\n #include \"refs.h\"\n #include \"run-command.h\"\n+#include \"hashmap.h\"\n+#include \"pkt-line.h\"\n+\n+static int compare_bundles(const void *hashmap_cmp_fn_data,\n+\t\t\t const struct hashmap_entry *he1,\n+\t\t\t const struct hashmap_entry *he2,\n+\t\t\t const void *id)\n+{\n+\tconst struct remote_bundle_info *e1 =\n+\t\tcontainer_of(he1, const struct remote_bundle_info, ent);\n+\tconst struct remote_bundle_info *e2 =\n+\t\tcontainer_of(he2, const struct remote_bundle_info, ent);\n+\n+\treturn strcmp(e1->id, id ? (const char *)id : e2->id);\n+}\n+\n+void init_bundle_list(struct bundle_list *list)\n+{\n+\tmemset(list, 0, sizeof(*list));\n+\n+\t/* Implied defaults. */\n+\tlist->mode = BUNDLE_MODE_ALL;\n+\tlist->version = 1;\n+\n+\thashmap_init(&list->bundles, compare_bundles, NULL, 0);\n+}\n+\n+static int clear_remote_bundle_info(struct remote_bundle_info *bundle,\n+\t\t\t\t void *data)\n+{\n+\tFREE_AND_NULL(bundle->id);\n+\tFREE_AND_NULL(bundle->uri);\n+\treturn 0;\n+}\n+\n+void clear_bundle_list(struct bundle_list *list)\n+{\n+\tif (!list)\n+\t\treturn;\n+\n+\tfor_all_bundles_in_list(list, clear_remote_bundle_info, NULL);\n+\thashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent);\n+}\n+\n+int for_all_bundles_in_list(struct bundle_list *list,\n+\t\t\t bundle_iterator iter,\n+\t\t\t void *data)\n+{\n+\tstruct remote_bundle_info *info;\n+\tstruct hashmap_iter i;\n+\n+\thashmap_for_each_entry(&list->bundles, &i, info, ent) {\n+\t\tint result = iter(info, data);\n+\n+\t\tif (result)\n+\t\t\treturn result;\n+\t}\n+\n+\treturn 0;\n+}\n \n static char *find_temp_filename(void)\n {\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 8a152f1ef14..ff7e3fd3fb2 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -1,7 +1,63 @@\n #ifndef BUNDLE_URI_H\n #define BUNDLE_URI_H\n \n+#include \"hashmap.h\"\n+#include \"strbuf.h\"\n+\n struct repository;\n+struct string_list;\n+\n+/**\n+ * The remote_bundle_info struct contains information for a single bundle\n+ * URI. This may be initialized simply by a given URI or might have\n+ * additional metadata associated with it if the bundle was advertised by\n+ * a bundle list.\n+ */\n+struct remote_bundle_info {\n+\tstruct hashmap_entry ent;\n+\n+\t/**\n+\t * The 'id' is a name given to the bundle for reference\n+\t * by other bundle infos.\n+\t */\n+\tchar *id;\n+\n+\t/**\n+\t * The 'uri' is the location of the remote bundle so\n+\t * it can be downloaded on-demand. This will be NULL\n+\t * if there was no table of contents.\n+\t */\n+\tchar *uri;\n+};\n+\n+#define REMOTE_BUNDLE_INFO_INIT { 0 }\n+\n+enum bundle_list_mode {\n+\tBUNDLE_MODE_NONE = 0,\n+\tBUNDLE_MODE_ALL,\n+\tBUNDLE_MODE_ANY\n+};\n+\n+/**\n+ * A bundle_list contains an unordered set of remote_bundle_info structs,\n+ * as well as information about the bundle listing, such as version and\n+ * mode.\n+ */\n+struct bundle_list {\n+\tint version;\n+\tenum bundle_list_mode mode;\n+\tstruct hashmap bundles;\n+};\n+\n+void init_bundle_list(struct bundle_list *list);\n+void clear_bundle_list(struct bundle_list *list);\n+\n+typedef int (*bundle_iterator)(struct remote_bundle_info *bundle,\n+\t\t\t void *data);\n+\n+int for_all_bundles_in_list(struct bundle_list *list,\n+\t\t\t bundle_iterator iter,\n+\t\t\t void *data);\n \n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n-- \ngitgitgadget\n\n"},{"id":"464698","messageId":"430e01cd2a4127a17c4bfc1b0db7209964736de8.1665579160.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v5.git.1665579160.gitgitgadget@gmail.com","subject":"[PATCH v5 03/12] bundle-uri: create base key-value pair parsing","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T12:52:30Z","receivedAt":"2022-10-12T12:53:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThere will be two primary ways to advertise a bundle list: as a list of\npacket lines in Git's protocol v2 and as a config file served from a\nbundle URI. Both of these fundamentally use a list of key-value pairs.\nWe will use the same set of key-value pairs across these formats.\n\nCreate a new bundle_list_update() method that is currently unusued, but\nwill be used in the next change. It inspects each key to see if it is\nunderstood and then applies it to the given bundle_list. Here are the\nkeys that we teach Git to understand:\n\n* bundle.version: This value should be an integer. Git currently\n understands only version 1 and will ignore the list if the version is\n any other value. This version can be increased in the future if we\n need to add new keys that Git should not ignore. We can add new\n \"heuristic\" keys without incrementing the version.\n\n* bundle.mode: This value should be one of \"all\" or \"any\". If this\n mode is not understood, then Git will ignore the list. This mode\n indicates whether Git needs all of the bundle list items to make a\n complete view of the content or if any single item is sufficient.\n\nThe rest of the keys use a bundle identifier \"\" as part of the key\nname. Keys using the same \"\" describe a single bundle list item.\n\n* bundle..uri: This stores the URI of the bundle item. This\n currently is expected to be an absolute URI, but will be relaxed to be\n a relative URI in the future.\n\nWhile parsing, return an error if a URI key is repeated, since we can\nmake that restriction with bundle lists.\n\nMake the git_parse_int() method global so we can parse the integer\nversion value carefully.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 2 +\n Documentation/config/bundle.txt | 24 +++++++++++\n bundle-uri.c | 76 +++++++++++++++++++++++++++++++++\n config.c | 2 +-\n config.h | 1 +\n 5 files changed, 104 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/config/bundle.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex e376d547ce0..4280af6992e 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -387,6 +387,8 @@ include::config/branch.txt[]\n \n include::config/browser.txt[]\n \n+include::config/bundle.txt[]\n+\n include::config/checkout.txt[]\n \n include::config/clean.txt[]\ndiff --git a/Documentation/config/bundle.txt b/Documentation/config/bundle.txt\nnew file mode 100644\nindex 00000000000..daa21eb674a\n--- /dev/null\n+++ b/Documentation/config/bundle.txt\n@@ -0,0 +1,24 @@\n+bundle.*::\n+\tThe `bundle.*` keys may appear in a bundle list file found via the\n+\t`git clone --bundle-uri` option. These keys currently have no effect\n+\tif placed in a repository config file, though this will change in the\n+\tfuture. See link:technical/bundle-uri.html[the bundle URI design\n+\tdocument] for more details.\n+\n+bundle.version::\n+\tThis integer value advertises the version of the bundle list format\n+\tused by the bundle list. Currently, the only accepted value is `1`.\n+\n+bundle.mode::\n+\tThis string value should be either `all` or `any`. This value describes\n+\twhether all of the advertised bundles are required to unbundle a\n+\tcomplete understanding of the bundled information (`all`) or if any one\n+\tof the listed bundle URIs is sufficient (`any`).\n+\n+bundle..*::\n+\tThe `bundle..*` keys are used to describe a single item in the\n+\tbundle list, grouped under `` for identification purposes.\n+\n+bundle..uri::\n+\tThis string value defines the URI by which Git can reach the contents\n+\tof this ``. This URI may be a bundle file or another bundle list.\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex f9a8db221bc..0bc59dd9c34 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -6,6 +6,7 @@\n #include \"run-command.h\"\n #include \"hashmap.h\"\n #include \"pkt-line.h\"\n+#include \"config.h\"\n \n static int compare_bundles(const void *hashmap_cmp_fn_data,\n \t\t\t const struct hashmap_entry *he1,\n@@ -65,6 +66,81 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \treturn 0;\n }\n \n+/**\n+ * Given a key-value pair, update the state of the given bundle list.\n+ * Returns 0 if the key-value pair is understood. Returns -1 if the key\n+ * is not understood or the value is malformed.\n+ */\n+MAYBE_UNUSED\n+static int bundle_list_update(const char *key, const char *value,\n+\t\t\t struct bundle_list *list)\n+{\n+\tstruct strbuf id = STRBUF_INIT;\n+\tstruct remote_bundle_info lookup = REMOTE_BUNDLE_INFO_INIT;\n+\tstruct remote_bundle_info *bundle;\n+\tconst char *subsection, *subkey;\n+\tsize_t subsection_len;\n+\n+\tif (parse_config_key(key, \"bundle\", &subsection, &subsection_len, &subkey))\n+\t\treturn -1;\n+\n+\tif (!subsection_len) {\n+\t\tif (!strcmp(subkey, \"version\")) {\n+\t\t\tint version;\n+\t\t\tif (!git_parse_int(value, &version))\n+\t\t\t\treturn -1;\n+\t\t\tif (version != 1)\n+\t\t\t\treturn -1;\n+\n+\t\t\tlist->version = version;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\tif (!strcmp(subkey, \"mode\")) {\n+\t\t\tif (!strcmp(value, \"all\"))\n+\t\t\t\tlist->mode = BUNDLE_MODE_ALL;\n+\t\t\telse if (!strcmp(value, \"any\"))\n+\t\t\t\tlist->mode = BUNDLE_MODE_ANY;\n+\t\t\telse\n+\t\t\t\treturn -1;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\t/* Ignore other unknown global keys. */\n+\t\treturn 0;\n+\t}\n+\n+\tstrbuf_add(&id, subsection, subsection_len);\n+\n+\t/*\n+\t * Check for an existing bundle with this , or create one\n+\t * if necessary.\n+\t */\n+\tlookup.id = id.buf;\n+\thashmap_entry_init(&lookup.ent, strhash(lookup.id));\n+\tif (!(bundle = hashmap_get_entry(&list->bundles, &lookup, ent, NULL))) {\n+\t\tCALLOC_ARRAY(bundle, 1);\n+\t\tbundle->id = strbuf_detach(&id, NULL);\n+\t\thashmap_entry_init(&bundle->ent, strhash(bundle->id));\n+\t\thashmap_add(&list->bundles, &bundle->ent);\n+\t}\n+\tstrbuf_release(&id);\n+\n+\tif (!strcmp(subkey, \"uri\")) {\n+\t\tif (bundle->uri)\n+\t\t\treturn -1;\n+\t\tbundle->uri = xstrdup(value);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * At this point, we ignore any information that we don't\n+\t * understand, assuming it to be hints for a heuristic the client\n+\t * does not currently understand.\n+\t */\n+\treturn 0;\n+}\n+\n static char *find_temp_filename(void)\n {\n \tint fd;\ndiff --git a/config.c b/config.c\nindex 015bec360f5..e93101249f6 100644\n--- a/config.c\n+++ b/config.c\n@@ -1214,7 +1214,7 @@ static int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max)\n \treturn 0;\n }\n \n-static int git_parse_int(const char *value, int *ret)\n+int git_parse_int(const char *value, int *ret)\n {\n \tintmax_t tmp;\n \tif (!git_parse_signed(value, &tmp, maximum_signed_value_of_type(int)))\ndiff --git a/config.h b/config.h\nindex ca994d77147..ef9eade6414 100644\n--- a/config.h\n+++ b/config.h\n@@ -206,6 +206,7 @@ int config_with_options(config_fn_t fn, void *,\n \n int git_parse_ssize_t(const char *, ssize_t *);\n int git_parse_ulong(const char *, unsigned long *);\n+int git_parse_int(const char *value, int *ret);\n \n /**\n * Same as `git_config_bool`, except that it returns -1 on error rather\n-- \ngitgitgadget\n\n"},{"id":"464699","messageId":"4d8cac67f66fc9c6477efabde795da95fafc51ec.1665579160.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v5.git.1665579160.gitgitgadget@gmail.com","subject":"[PATCH v5 05/12] bundle-uri: unit test \"key=value\" parsing","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T12:52:32Z","receivedAt":"2022-10-12T12:53:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nCreate a new 'test-tool bundle-uri' test helper. This helper will assist\nin testing logic deep in the bundle URI feature.\n\nThis change introduces the 'parse-key-values' subcommand, which parses\nan input file as a list of lines. These are fed into\nbundle_uri_parse_line() to test how we construct a 'struct bundle_list'\nfrom that data. The list is then output to stdout as if the key-value\npairs were a Git config file.\n\nWe use an input file instead of stdin because of a future change to\nparse in config-file format that works better as an input file.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n bundle-uri.c | 33 ++++++++++\n bundle-uri.h | 3 +\n t/helper/test-bundle-uri.c | 70 +++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5750-bundle-uri-parse.sh | 121 ++++++++++++++++++++++++++++++++++++\n t/test-lib-functions.sh | 11 ++++\n 8 files changed, 241 insertions(+)\n create mode 100644 t/helper/test-bundle-uri.c\n create mode 100755 t/t5750-bundle-uri-parse.sh\n\ndiff --git a/Makefile b/Makefile\nindex 7d5f48069ea..7dee0329c49 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -722,6 +722,7 @@ PROGRAMS += $(patsubst %.o,git-%$X,$(PROGRAM_OBJS))\n TEST_BUILTINS_OBJS += test-advise.o\n TEST_BUILTINS_OBJS += test-bitmap.o\n TEST_BUILTINS_OBJS += test-bloom.o\n+TEST_BUILTINS_OBJS += test-bundle-uri.o\n TEST_BUILTINS_OBJS += test-chmtime.o\n TEST_BUILTINS_OBJS += test-config.o\n TEST_BUILTINS_OBJS += test-crontab.o\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 372e6fac5cf..c02e7f62eb1 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -66,6 +66,39 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \treturn 0;\n }\n \n+static int summarize_bundle(struct remote_bundle_info *info, void *data)\n+{\n+\tFILE *fp = data;\n+\tfprintf(fp, \"[bundle \\\"%s\\\"]\\n\", info->id);\n+\tfprintf(fp, \"\\turi = %s\\n\", info->uri);\n+\treturn 0;\n+}\n+\n+void print_bundle_list(FILE *fp, struct bundle_list *list)\n+{\n+\tconst char *mode;\n+\n+\tswitch (list->mode) {\n+\tcase BUNDLE_MODE_ALL:\n+\t\tmode = \"all\";\n+\t\tbreak;\n+\n+\tcase BUNDLE_MODE_ANY:\n+\t\tmode = \"any\";\n+\t\tbreak;\n+\n+\tcase BUNDLE_MODE_NONE:\n+\tdefault:\n+\t\tmode = \"\";\n+\t}\n+\n+\tfprintf(fp, \"[bundle]\\n\");\n+\tfprintf(fp, \"\\tversion = %d\\n\", list->version);\n+\tfprintf(fp, \"\\tmode = %s\\n\", mode);\n+\n+\tfor_all_bundles_in_list(list, summarize_bundle, fp);\n+}\n+\n /**\n * Given a key-value pair, update the state of the given bundle list.\n * Returns 0 if the key-value pair is understood. Returns -1 if the key\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 90583461929..0e56ab2ae5a 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -59,6 +59,9 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \t\t\t bundle_iterator iter,\n \t\t\t void *data);\n \n+struct FILE;\n+void print_bundle_list(FILE *fp, struct bundle_list *list);\n+\n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n * based on that information.\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nnew file mode 100644\nindex 00000000000..0329c56544f\n--- /dev/null\n+++ b/t/helper/test-bundle-uri.c\n@@ -0,0 +1,70 @@\n+#include \"test-tool.h\"\n+#include \"parse-options.h\"\n+#include \"bundle-uri.h\"\n+#include \"strbuf.h\"\n+#include \"string-list.h\"\n+\n+static int cmd__bundle_uri_parse(int argc, const char **argv)\n+{\n+\tconst char *key_value_usage[] = {\n+\t\t\"test-tool bundle-uri parse-key-values \",\n+\t\tNULL\n+\t};\n+\tconst char **usage = key_value_usage;\n+\tstruct option options[] = {\n+\t\tOPT_END(),\n+\t};\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct bundle_list list;\n+\tint err = 0;\n+\tFILE *fp;\n+\n+\targc = parse_options(argc, argv, NULL, options, usage, 0);\n+\tif (argc != 1)\n+\t\tgoto usage;\n+\n+\tinit_bundle_list(&list);\n+\tfp = fopen(argv[0], \"r\");\n+\tif (!fp)\n+\t\tdie(\"failed to open '%s'\", argv[0]);\n+\n+\twhile (strbuf_getline(&sb, fp) != EOF) {\n+\t\tif (bundle_uri_parse_line(&list, sb.buf))\n+\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\t}\n+\tstrbuf_release(&sb);\n+\tfclose(fp);\n+\n+\tprint_bundle_list(stdout, &list);\n+\n+\tclear_bundle_list(&list);\n+\n+\treturn !!err;\n+\n+usage:\n+\tusage_with_options(usage, options);\n+}\n+\n+int cmd__bundle_uri(int argc, const char **argv)\n+{\n+\tconst char *usage[] = {\n+\t\t\"test-tool bundle-uri []\",\n+\t\tNULL\n+\t};\n+\tstruct option options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL, options, usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION |\n+\t\t\t PARSE_OPT_KEEP_ARGV0);\n+\tif (argc == 1)\n+\t\tgoto usage;\n+\n+\tif (!strcmp(argv[1], \"parse-key-values\"))\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1);\n+\terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n+\n+usage:\n+\tusage_with_options(usage, options);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 318fdbab0c3..fbe2d9d8108 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -17,6 +17,7 @@ static struct test_cmd cmds[] = {\n \t{ \"advise\", cmd__advise_if_enabled },\n \t{ \"bitmap\", cmd__bitmap },\n \t{ \"bloom\", cmd__bloom },\n+\t{ \"bundle-uri\", cmd__bundle_uri },\n \t{ \"chmtime\", cmd__chmtime },\n \t{ \"config\", cmd__config },\n \t{ \"crontab\", cmd__crontab },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex bb799271631..b2aa1f39a8f 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -7,6 +7,7 @@\n int cmd__advise_if_enabled(int argc, const char **argv);\n int cmd__bitmap(int argc, const char **argv);\n int cmd__bloom(int argc, const char **argv);\n+int cmd__bundle_uri(int argc, const char **argv);\n int cmd__chmtime(int argc, const char **argv);\n int cmd__config(int argc, const char **argv);\n int cmd__crontab(int argc, const char **argv);\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nnew file mode 100755\nindex 00000000000..fd142a66ad5\n--- /dev/null\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -0,0 +1,121 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri bundle_uri_parse_line()\"\n+\n+TEST_NO_CREATE_REPO=1\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success 'bundle_uri_parse_line() just URIs' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' '\n+\tcat >in <<-\\EOF &&\n+\t=bogus-value\n+\tbogus-key=\n+\tEOF\n+\n+\tcat >err.expect <<-EOF &&\n+\terror: bundle-uri: line has empty key or value\n+\terror: bad line: '\\''=bogus-value'\\''\n+\terror: bundle-uri: line has empty key or value\n+\terror: bad line: '\\''bogus-key='\\''\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty lines' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\terror: bundle-uri: got an empty line\n+\terror: bad line: '\\'''\\''\n+\terror: bundle-uri: got an empty line\n+\terror: bad line: '\\'''\\''\n+\tEOF\n+\n+\t# We fail, but try to continue parsing regardless\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: duplicate lines' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\tbundle.one.uri=https://example.com/bundle-2.bdl\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\terror: bad line: '\\''bundle.one.uri=https://example.com/bundle-2.bdl'\\''\n+\tEOF\n+\n+\t# We fail, but try to continue parsing regardless\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_done\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 6da7273f1d5..3175d665add 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1956,3 +1956,14 @@ test_is_magic_mtime () {\n \trm -f .git/test-mtime-actual\n \treturn $ret\n }\n+\n+# Given two filenames, parse both using 'git config --list --file'\n+# and compare the sorted output of those commands. Useful when\n+# wanting to ignore whitespace differences and sorting concerns.\n+test_cmp_config_output () {\n+\tgit config --list --file=\"$1\" >config-expect &&\n+\tgit config --list --file=\"$2\" >config-actual &&\n+\tsort config-expect >sorted-expect &&\n+\tsort config-actual >sorted-actual &&\n+\ttest_cmp sorted-expect sorted-actual\n+}\n-- \ngitgitgadget\n\n"},{"id":"464700","messageId":"cd915d57f3bced015dc154572612d4dd221d1618.1665579160.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v5.git.1665579160.gitgitgadget@gmail.com","subject":"[PATCH v5 04/12] bundle-uri: create \"key=value\" line parsing","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T12:52:31Z","receivedAt":"2022-10-12T12:53:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nWhen advertising a bundle list over Git's protocol v2, we will use\npacket lines. Each line will be of the form \"key=value\" representing a\nbundle list. Connect the API necessary for Git's transport to the\nkey-value pair parsing created in the previous change.\n\nWe are not currently implementing this protocol v2 functionality, but\ninstead preparing to expose this parsing to be unit-testable.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 27 ++++++++++++++++++++++++++-\n bundle-uri.h | 12 ++++++++++++\n 2 files changed, 38 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 0bc59dd9c34..372e6fac5cf 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -71,7 +71,6 @@ int for_all_bundles_in_list(struct bundle_list *list,\n * Returns 0 if the key-value pair is understood. Returns -1 if the key\n * is not understood or the value is malformed.\n */\n-MAYBE_UNUSED\n static int bundle_list_update(const char *key, const char *value,\n \t\t\t struct bundle_list *list)\n {\n@@ -306,3 +305,29 @@ cleanup:\n \tfree(filename);\n \treturn result;\n }\n+\n+/**\n+ * General API for {transport,connect}.c etc.\n+ */\n+int bundle_uri_parse_line(struct bundle_list *list, const char *line)\n+{\n+\tint result;\n+\tconst char *equals;\n+\tstruct strbuf key = STRBUF_INIT;\n+\n+\tif (!strlen(line))\n+\t\treturn error(_(\"bundle-uri: got an empty line\"));\n+\n+\tequals = strchr(line, '=');\n+\n+\tif (!equals)\n+\t\treturn error(_(\"bundle-uri: line is not of the form 'key=value'\"));\n+\tif (line == equals || !*(equals + 1))\n+\t\treturn error(_(\"bundle-uri: line has empty key or value\"));\n+\n+\tstrbuf_add(&key, line, equals - line);\n+\tresult = bundle_list_update(key.buf, equals + 1, list);\n+\tstrbuf_release(&key);\n+\n+\treturn result;\n+}\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex ff7e3fd3fb2..90583461929 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -67,4 +67,16 @@ int for_all_bundles_in_list(struct bundle_list *list,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n+/**\n+ * General API for {transport,connect}.c etc.\n+ */\n+\n+/**\n+ * Parse a \"key=value\" packet line from the bundle-uri verb.\n+ *\n+ * Returns 0 on success and non-zero on error.\n+ */\n+int bundle_uri_parse_line(struct bundle_list *list,\n+\t\t\t const char *line);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"464701","messageId":"0ecae3a44b360fbb03a52b73536f83c457a6668d.1665579160.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v5.git.1665579160.gitgitgadget@gmail.com","subject":"[PATCH v5 06/12] bundle-uri: parse bundle list in config format","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T12:52:33Z","receivedAt":"2022-10-12T12:53:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a bundle provider wants to operate independently from a Git remote,\nthey want to provide a single, consistent URI that users can use in\ntheir 'git clone --bundle-uri' commands. At this point, the Git client\nexpects that URI to be a single bundle that can be unbundled and used to\nbootstrap the rest of the clone from the Git server. This single bundle\ncannot be re-used to assist with future incremental fetches.\n\nTo allow for the incremental fetch case, teach Git to understand a\nbundle list that could be advertised at an independent bundle URI. Such\na bundle list is likely to be inspected by human readers, even if only\nby the bundle provider creating the list. For this reason, we can take\nour expected \"key=value\" pairs and instead format them using Git config\nformat.\n\nCreate bundle_uri_parse_config_format() to parse a file in config format\nand convert that into a 'struct bundle_list' filled with its\nunderstanding of the contents.\n\nBe careful to use error_action CONFIG_ERROR_ERROR when calling\ngit_config_from_file_with_options() because the default action for\ngit_config_from_file() is to die() on a parsing error. The current\nwarning isn't particularly helpful if it arises to a user, but it will\nbe made more verbose at a higher layer later.\n\nUpdate 'test-tool bundle-uri' to take this config file format as input.\nIt uses a filename instead of stdin because there is no existing way to\nparse a FILE pointer in the config machinery. Using\ngit_config_from_mem() is overly complicated and more likely to introduce\nbugs than this simpler version.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 27 ++++++++++++++++++++\n bundle-uri.h | 9 +++++++\n t/helper/test-bundle-uri.c | 49 +++++++++++++++++++++++++++---------\n t/t5750-bundle-uri-parse.sh | 50 +++++++++++++++++++++++++++++++++++++\n 4 files changed, 123 insertions(+), 12 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex c02e7f62eb1..3d44ec2b1e6 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -173,6 +173,33 @@ static int bundle_list_update(const char *key, const char *value,\n \treturn 0;\n }\n \n+static int config_to_bundle_list(const char *key, const char *value, void *data)\n+{\n+\tstruct bundle_list *list = data;\n+\treturn bundle_list_update(key, value, list);\n+}\n+\n+int bundle_uri_parse_config_format(const char *uri,\n+\t\t\t\t const char *filename,\n+\t\t\t\t struct bundle_list *list)\n+{\n+\tint result;\n+\tstruct config_options opts = {\n+\t\t.error_action = CONFIG_ERROR_ERROR,\n+\t};\n+\n+\tresult = git_config_from_file_with_options(config_to_bundle_list,\n+\t\t\t\t\t\t filename, list,\n+\t\t\t\t\t\t &opts);\n+\n+\tif (!result && list->mode == BUNDLE_MODE_NONE) {\n+\t\twarning(_(\"bundle list at '%s' has no mode\"), uri);\n+\t\tresult = 1;\n+\t}\n+\n+\treturn result;\n+}\n+\n static char *find_temp_filename(void)\n {\n \tint fd;\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 0e56ab2ae5a..bc13d4c9929 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -62,6 +62,15 @@ int for_all_bundles_in_list(struct bundle_list *list,\n struct FILE;\n void print_bundle_list(FILE *fp, struct bundle_list *list);\n \n+/**\n+ * A bundle URI may point to a bundle list where the key=value\n+ * pairs are provided in config file format. This method is\n+ * exposed publicly for testing purposes.\n+ */\n+int bundle_uri_parse_config_format(const char *uri,\n+\t\t\t\t const char *filename,\n+\t\t\t\t struct bundle_list *list);\n+\n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n * based on that information.\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex 0329c56544f..25afd393428 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -4,12 +4,21 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n \n-static int cmd__bundle_uri_parse(int argc, const char **argv)\n+enum input_mode {\n+\tKEY_VALUE_PAIRS,\n+\tCONFIG_FILE,\n+};\n+\n+static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mode)\n {\n \tconst char *key_value_usage[] = {\n \t\t\"test-tool bundle-uri parse-key-values \",\n \t\tNULL\n \t};\n+\tconst char *config_usage[] = {\n+\t\t\"test-tool bundle-uri parse-config \",\n+\t\tNULL\n+\t};\n \tconst char **usage = key_value_usage;\n \tstruct option options[] = {\n \t\tOPT_END(),\n@@ -19,21 +28,35 @@ static int cmd__bundle_uri_parse(int argc, const char **argv)\n \tint err = 0;\n \tFILE *fp;\n \n-\targc = parse_options(argc, argv, NULL, options, usage, 0);\n-\tif (argc != 1)\n-\t\tgoto usage;\n+\tif (mode == CONFIG_FILE)\n+\t\tusage = config_usage;\n+\n+\targc = parse_options(argc, argv, NULL, options, usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tinit_bundle_list(&list);\n-\tfp = fopen(argv[0], \"r\");\n-\tif (!fp)\n-\t\tdie(\"failed to open '%s'\", argv[0]);\n \n-\twhile (strbuf_getline(&sb, fp) != EOF) {\n-\t\tif (bundle_uri_parse_line(&list, sb.buf))\n-\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\tswitch (mode) {\n+\tcase KEY_VALUE_PAIRS:\n+\t\tif (argc != 1)\n+\t\t\tgoto usage;\n+\t\tfp = fopen(argv[0], \"r\");\n+\t\tif (!fp)\n+\t\t\tdie(\"failed to open '%s'\", argv[0]);\n+\t\twhile (strbuf_getline(&sb, fp) != EOF) {\n+\t\t\tif (bundle_uri_parse_line(&list, sb.buf))\n+\t\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\t\t}\n+\t\tfclose(fp);\n+\t\tbreak;\n+\n+\tcase CONFIG_FILE:\n+\t\tif (argc != 1)\n+\t\t\tgoto usage;\n+\t\terr = bundle_uri_parse_config_format(\"\", argv[0], &list);\n+\t\tbreak;\n \t}\n \tstrbuf_release(&sb);\n-\tfclose(fp);\n \n \tprint_bundle_list(stdout, &list);\n \n@@ -62,7 +85,9 @@ int cmd__bundle_uri(int argc, const char **argv)\n \t\tgoto usage;\n \n \tif (!strcmp(argv[1], \"parse-key-values\"))\n-\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1);\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS);\n+\tif (!strcmp(argv[1], \"parse-config\"))\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE);\n \terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n \n usage:\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex fd142a66ad5..c2fe3f9c5a5 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -118,4 +118,54 @@ test_expect_success 'bundle_uri_parse_line() parsing edge cases: duplicate lines\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format: just URIs' '\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config expect >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'parse config format edge cases: empty key or value' '\n+\tcat >in1 <<-\\EOF &&\n+\t= bogus-value\n+\tEOF\n+\n+\tcat >err1 <<-EOF &&\n+\terror: bad config line 1 in file in1\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-config in1 >actual 2>err &&\n+\ttest_cmp err1 err &&\n+\ttest_cmp_config_output expect actual &&\n+\n+\tcat >in2 <<-\\EOF &&\n+\tbogus-key =\n+\tEOF\n+\n+\tcat >err2 <<-EOF &&\n+\terror: bad config line 1 in file in2\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-config in2 >actual 2>err &&\n+\ttest_cmp err2 err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"464702","messageId":"7e6b32313b0f1922c0d0bfd104a288b8606306ec.1665579160.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v5.git.1665579160.gitgitgadget@gmail.com","subject":"[PATCH v5 07/12] bundle-uri: limit recursion depth for bundle lists","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T12:52:34Z","receivedAt":"2022-10-12T12:53:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe next change will start allowing us to parse bundle lists that are\ndownloaded from a provided bundle URI. Those lists might point to other\nlists, which could proceed to an arbitrary depth (and even create\ncycles). Restructure fetch_bundle_uri() to have an internal version that\nhas a recursion depth. Compare that to a new max_bundle_uri_depth\nconstant that is twice as high as we expect this depth to be for any\nlegitimate use of bundle list linking.\n\nWe can consider making max_bundle_uri_depth a configurable value if\nthere is demonstrated value in the future.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 21 ++++++++++++++++++++-\n 1 file changed, 20 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 3d44ec2b1e6..8a7c11c6393 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -334,11 +334,25 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \treturn result;\n }\n \n-int fetch_bundle_uri(struct repository *r, const char *uri)\n+/**\n+ * This limits the recursion on fetch_bundle_uri_internal() when following\n+ * bundle lists.\n+ */\n+static int max_bundle_uri_depth = 4;\n+\n+static int fetch_bundle_uri_internal(struct repository *r,\n+\t\t\t\t const char *uri,\n+\t\t\t\t int depth)\n {\n \tint result = 0;\n \tchar *filename;\n \n+\tif (depth >= max_bundle_uri_depth) {\n+\t\twarning(_(\"exceeded bundle URI recursion limit (%d)\"),\n+\t\t\tmax_bundle_uri_depth);\n+\t\treturn -1;\n+\t}\n+\n \tif (!(filename = find_temp_filename())) {\n \t\tresult = -1;\n \t\tgoto cleanup;\n@@ -366,6 +380,11 @@ cleanup:\n \treturn result;\n }\n \n+int fetch_bundle_uri(struct repository *r, const char *uri)\n+{\n+\treturn fetch_bundle_uri_internal(r, uri, 0);\n+}\n+\n /**\n * General API for {transport,connect}.c etc.\n */\n-- \ngitgitgadget\n\n"},{"id":"464703","messageId":"fba3a4a117e5996ba4a7e14d7391954b249b2b68.1665579160.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v5.git.1665579160.gitgitgadget@gmail.com","subject":"[PATCH v5 10/12] bundle: add flags to verify_bundle()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T12:52:37Z","receivedAt":"2022-10-12T12:53:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe verify_bundle() method has a 'verbose' option, but we will want to\nextend this method to have more granular control over its output. First,\nreplace this 'verbose' option with a new 'flags' option with a single\npossible value: VERIFY_BUNDLE_VERBOSE.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/bundle.c | 5 +++--\n bundle-uri.c | 7 ++++++-\n bundle.c | 9 +++++----\n bundle.h | 14 ++++++++++++--\n transport.c | 2 +-\n 5 files changed, 27 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex 2adad545a2e..7d983a238f0 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -119,7 +119,8 @@ static int cmd_bundle_verify(int argc, const char **argv, const char *prefix) {\n \t\tgoto cleanup;\n \t}\n \tclose(bundle_fd);\n-\tif (verify_bundle(the_repository, &header, !quiet)) {\n+\tif (verify_bundle(the_repository, &header,\n+\t\t\t quiet ? 0 : VERIFY_BUNDLE_VERBOSE)) {\n \t\tret = 1;\n \t\tgoto cleanup;\n \t}\n@@ -185,7 +186,7 @@ static int cmd_bundle_unbundle(int argc, const char **argv, const char *prefix)\n \t\tstrvec_pushl(&extra_index_pack_args, \"-v\", \"--progress-title\",\n \t\t\t _(\"Unbundling objects\"), NULL);\n \tret = !!unbundle(the_repository, &header, bundle_fd,\n-\t\t\t &extra_index_pack_args) ||\n+\t\t\t &extra_index_pack_args, 0) ||\n \t\tlist_bundle_refs(&header, argc, argv);\n \tbundle_header_release(&header);\n cleanup:\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 70bfd2defee..d9060be707e 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -303,7 +303,12 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \tif ((bundle_fd = read_bundle_header(file, &header)) < 0)\n \t\treturn 1;\n \n-\tif ((result = unbundle(r, &header, bundle_fd, NULL)))\n+\t/*\n+\t * Skip the reachability walk here, since we will be adding\n+\t * a reachable ref pointing to the new tips, which will reach\n+\t * the prerequisite commits.\n+\t */\n+\tif ((result = unbundle(r, &header, bundle_fd, NULL, 0)))\n \t\treturn 1;\n \n \t/*\ndiff --git a/bundle.c b/bundle.c\nindex c277f3b9360..1f6a7f782e1 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -189,7 +189,7 @@ static int list_refs(struct string_list *r, int argc, const char **argv)\n \n int verify_bundle(struct repository *r,\n \t\t struct bundle_header *header,\n-\t\t int verbose)\n+\t\t enum verify_bundle_flags flags)\n {\n \t/*\n \t * Do fast check, then if any prereqs are missing then go line by line\n@@ -248,7 +248,7 @@ int verify_bundle(struct repository *r,\n \t\terror(\"%s %s\", oid_to_hex(oid), name);\n \t}\n \n-\tif (verbose) {\n+\tif (flags & VERIFY_BUNDLE_VERBOSE) {\n \t\tstruct string_list *r;\n \n \t\tr = &header->references;\n@@ -617,7 +617,8 @@ err:\n }\n \n int unbundle(struct repository *r, struct bundle_header *header,\n-\t int bundle_fd, struct strvec *extra_index_pack_args)\n+\t int bundle_fd, struct strvec *extra_index_pack_args,\n+\t enum verify_bundle_flags flags)\n {\n \tstruct child_process ip = CHILD_PROCESS_INIT;\n \tstrvec_pushl(&ip.args, \"index-pack\", \"--fix-thin\", \"--stdin\", NULL);\n@@ -631,7 +632,7 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \t\tstrvec_clear(extra_index_pack_args);\n \t}\n \n-\tif (verify_bundle(r, header, 0))\n+\tif (verify_bundle(r, header, flags))\n \t\treturn -1;\n \tip.in = bundle_fd;\n \tip.no_stdout = 1;\ndiff --git a/bundle.h b/bundle.h\nindex 0c052f54964..6652e819981 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -29,7 +29,13 @@ int read_bundle_header_fd(int fd, struct bundle_header *header,\n int create_bundle(struct repository *r, const char *path,\n \t\t int argc, const char **argv, struct strvec *pack_options,\n \t\t int version);\n-int verify_bundle(struct repository *r, struct bundle_header *header, int verbose);\n+\n+enum verify_bundle_flags {\n+\tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n+};\n+\n+int verify_bundle(struct repository *r, struct bundle_header *header,\n+\t\t enum verify_bundle_flags flags);\n \n /**\n * Unbundle after reading the header with read_bundle_header().\n@@ -40,9 +46,13 @@ int verify_bundle(struct repository *r, struct bundle_header *header, int verbos\n * Provide \"extra_index_pack_args\" to pass any extra arguments\n * (e.g. \"-v\" for verbose/progress), NULL otherwise. The provided\n * \"extra_index_pack_args\" (if any) will be strvec_clear()'d for you.\n+ *\n+ * Before unbundling, this method will call verify_bundle() with the\n+ * given 'flags'.\n */\n int unbundle(struct repository *r, struct bundle_header *header,\n-\t int bundle_fd, struct strvec *extra_index_pack_args);\n+\t int bundle_fd, struct strvec *extra_index_pack_args,\n+\t enum verify_bundle_flags flags);\n int list_bundle_refs(struct bundle_header *header,\n \t\tint argc, const char **argv);\n \ndiff --git a/transport.c b/transport.c\nindex 52db7a3cb09..c5d3042731a 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -178,7 +178,7 @@ static int fetch_refs_from_bundle(struct transport *transport,\n \tif (!data->get_refs_from_bundle_called)\n \t\tget_refs_from_bundle_inner(transport);\n \tret = unbundle(the_repository, &data->header, data->fd,\n-\t\t &extra_index_pack_args);\n+\t\t &extra_index_pack_args, 0);\n \ttransport->hash_algo = data->header.hash_algo;\n \treturn ret;\n }\n-- \ngitgitgadget\n\n"},{"id":"464704","messageId":"8dc5a8e4e63dc98642176e5b78be739ef721d2d8.1665579160.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v5.git.1665579160.gitgitgadget@gmail.com","subject":"[PATCH v5 08/12] bundle: properly clear all revision flags","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T12:52:35Z","receivedAt":"2022-10-12T12:53:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe verify_bundle() method checks two things for a bundle's\nprerequisites:\n\n 1. Are these objects in the object store?\n 2. Are these objects reachable from our references?\n\nIn this second question, multiple uses of verify_bundle() in the same\nprocess can report an invalid bundle even though it is correct. The\nreason is due to not clearing all of the commit marks on the commits\npreviously walked.\n\nThe revision walk machinery was first introduced in-process by\nfb9a54150d3 (git-bundle: avoid fork() in verify_bundle(), 2007-02-22).\nThis implementation used \"-1\" as the set of flags to clear. The next\nmeaningful change came in 2b064697a5b (revision traversal: retire\nBOUNDARY_SHOW, 2007-03-05), which introduced the PREREQ_MARK flag\ninstead of a flag normally controlled by the revision-walk machinery.\n\nIn 86a0a408b90 (commit: factor out\nclear_commit_marks_for_object_array, 2011-10-01), the loop over the\narray of commits was replaced with a new\nclear_commit_marks_for_object_array(), but simultaneously the \"-1\" value\nwas replaced with \"ALL_REV_FLAGS\", which stopped un-setting the\nPREREQ_MARK flag. This means that if multiple commits were marked by the\nPREREQ_MARK in a previous run of verify_bundle(), then this loop could\nterminate early due to 'i' going to zero:\n\n\twhile (i && (commit = get_revision(&revs)))\n\t\tif (commit->object.flags & PREREQ_MARK)\n\t\t\ti--;\n\nThe flag clearing work was changed again in 63647391e6c (bundle: avoid\nusing the rev_info flag leak_pending, 2017-12-25), but that was only\ncosmetic and did not change the behavior.\n\nIt may seem that it would be sufficient to add the PREREQ_MARK flag to\nthe clear_commit_marks() call in its current location. However, we\nactually need to do it in the \"cleanup:\" step, since the first loop\nchecking \"Are these objects in the object store?\" might add the\nPREREQ_MARK flag to some objects and then terminate without performing a\nwalk due to one missing object. By clearing the flags in all cases, we\navoid this issue when running verify_bundle() multiple times in the same\nprocess.\n\nMoving this loop to the cleanup step alone would cause a segfault when\nrunning 'git bundle verify' outside of a repository, but this is because\nof that error condition using \"goto cleanup\" when returning is perfectly\nsafe. Nothing has been initialized at that point, so we can return\nimmediately without causing any leaks.\n\nThis behavior is verified carefully by a test that will be added soon\nwhen Git learns to download bundle lists in a 'git clone --bundle-uri'\ncommand.\n\nSigned-off-by: Derrick Stolee \n---\n bundle.c | 23 ++++++++++-------------\n 1 file changed, 10 insertions(+), 13 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex 0208e6d90d3..c277f3b9360 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -202,10 +202,8 @@ int verify_bundle(struct repository *r,\n \tint i, ret = 0, req_nr;\n \tconst char *message = _(\"Repository lacks these prerequisite commits:\");\n \n-\tif (!r || !r->objects || !r->objects->odb) {\n-\t\tret = error(_(\"need a repository to verify a bundle\"));\n-\t\tgoto cleanup;\n-\t}\n+\tif (!r || !r->objects || !r->objects->odb)\n+\t\treturn error(_(\"need a repository to verify a bundle\"));\n \n \trepo_init_revisions(r, &revs, NULL);\n \tfor (i = 0; i < p->nr; i++) {\n@@ -250,15 +248,6 @@ int verify_bundle(struct repository *r,\n \t\terror(\"%s %s\", oid_to_hex(oid), name);\n \t}\n \n-\t/* Clean up objects used, as they will be reused. */\n-\tfor (i = 0; i < p->nr; i++) {\n-\t\tstruct string_list_item *e = p->items + i;\n-\t\tstruct object_id *oid = e->util;\n-\t\tcommit = lookup_commit_reference_gently(r, oid, 1);\n-\t\tif (commit)\n-\t\t\tclear_commit_marks(commit, ALL_REV_FLAGS);\n-\t}\n-\n \tif (verbose) {\n \t\tstruct string_list *r;\n \n@@ -287,6 +276,14 @@ int verify_bundle(struct repository *r,\n \t\t\t\t list_objects_filter_spec(&header->filter));\n \t}\n cleanup:\n+\t/* Clean up objects used, as they will be reused. */\n+\tfor (i = 0; i < p->nr; i++) {\n+\t\tstruct string_list_item *e = p->items + i;\n+\t\tstruct object_id *oid = e->util;\n+\t\tcommit = lookup_commit_reference_gently(r, oid, 1);\n+\t\tif (commit)\n+\t\t\tclear_commit_marks(commit, ALL_REV_FLAGS | PREREQ_MARK);\n+\t}\n \trelease_revisions(&revs);\n \treturn ret;\n }\n-- \ngitgitgadget\n\n"},{"id":"464705","messageId":"51e9b8474fb040c4277b96d4b13a8da58bdb3682.1665579160.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v5.git.1665579160.gitgitgadget@gmail.com","subject":"[PATCH v5 09/12] bundle-uri: fetch a list of bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T12:52:36Z","receivedAt":"2022-10-12T12:53:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen the content at a given bundle URI is not understood as a bundle\n(based on inspecting the initial content), then Git currently gives up\nand ignores that content. Independent bundle providers may want to split\nup the bundle content into multiple bundles, but still make them\navailable from a single URI.\n\nTeach Git to attempt parsing the bundle URI content as a Git config file\nproviding the key=value pairs for a bundle list. Git then looks at the\nmode of the list to see if ANY single bundle is sufficient or if ALL\nbundles are required. The content at the selected URIs are downloaded\nand the content is inspected again, creating a recursive process.\n\nTo guard the recursion against malformed or malicious content, limit the\nrecursion depth to a reasonable four for now. This can be converted to a\nconfigured value in the future if necessary. The value of four is twice\nas high as expected to be useful (a bundle list is unlikely to point to\nmore bundle lists).\n\nTo test this scenario, create an interesting bundle topology where three\nincremental bundles are built on top of a single full bundle. By using a\nmerge commit, the two middle bundles are \"independent\" in that they do\nnot require each other in order to unbundle themselves. They each only\nneed the base bundle. The bundle containing the merge commit requires\nboth of the middle bundles, though. This leads to some interesting\ndecisions when unbundling, especially when we later implement heuristics\nthat promote downloading bundles until the prerequisite commits are\nsatisfied.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 203 ++++++++++++++++++++++++++---\n bundle-uri.h | 13 ++\n t/t5558-clone-bundle-uri.sh | 248 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 448 insertions(+), 16 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 8a7c11c6393..70bfd2defee 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -37,6 +37,8 @@ static int clear_remote_bundle_info(struct remote_bundle_info *bundle,\n {\n \tFREE_AND_NULL(bundle->id);\n \tFREE_AND_NULL(bundle->uri);\n+\tFREE_AND_NULL(bundle->file);\n+\tbundle->unbundled = 0;\n \treturn 0;\n }\n \n@@ -334,18 +336,117 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \treturn result;\n }\n \n+struct bundle_list_context {\n+\tstruct repository *r;\n+\tstruct bundle_list *list;\n+\tenum bundle_list_mode mode;\n+\tint count;\n+\tint depth;\n+};\n+\n+/*\n+ * This early definition is necessary because we use indirect recursion:\n+ *\n+ * While iterating through a bundle list that was downloaded as part\n+ * of fetch_bundle_uri_internal(), iterator methods eventually call it\n+ * again, but with depth + 1.\n+ */\n+static int fetch_bundle_uri_internal(struct repository *r,\n+\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t int depth,\n+\t\t\t\t struct bundle_list *list);\n+\n+static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data)\n+{\n+\tint res;\n+\tstruct bundle_list_context *ctx = data;\n+\n+\tif (ctx->mode == BUNDLE_MODE_ANY && ctx->count)\n+\t\treturn 0;\n+\n+\tres = fetch_bundle_uri_internal(ctx->r, bundle, ctx->depth + 1, ctx->list);\n+\n+\t/*\n+\t * Only increment count if the download succeeded. If our mode is\n+\t * BUNDLE_MODE_ANY, then we will want to try other URIs in the\n+\t * list in case they work instead.\n+\t */\n+\tif (!res)\n+\t\tctx->count++;\n+\n+\t/*\n+\t * To be opportunistic as possible, we continue iterating and\n+\t * download as many bundles as we can, so we can apply the ones\n+\t * that work, even in BUNDLE_MODE_ALL mode.\n+\t */\n+\treturn 0;\n+}\n+\n+static int download_bundle_list(struct repository *r,\n+\t\t\t\tstruct bundle_list *local_list,\n+\t\t\t\tstruct bundle_list *global_list,\n+\t\t\t\tint depth)\n+{\n+\tstruct bundle_list_context ctx = {\n+\t\t.r = r,\n+\t\t.list = global_list,\n+\t\t.depth = depth + 1,\n+\t\t.mode = local_list->mode,\n+\t};\n+\n+\treturn for_all_bundles_in_list(local_list, download_bundle_to_file, &ctx);\n+}\n+\n+static int fetch_bundle_list_in_config_format(struct repository *r,\n+\t\t\t\t\t struct bundle_list *global_list,\n+\t\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t\t int depth)\n+{\n+\tint result;\n+\tstruct bundle_list list_from_bundle;\n+\n+\tinit_bundle_list(&list_from_bundle);\n+\n+\tif ((result = bundle_uri_parse_config_format(bundle->uri,\n+\t\t\t\t\t\t bundle->file,\n+\t\t\t\t\t\t &list_from_bundle)))\n+\t\tgoto cleanup;\n+\n+\tif (list_from_bundle.mode == BUNDLE_MODE_NONE) {\n+\t\twarning(_(\"unrecognized bundle mode from URI '%s'\"),\n+\t\t\tbundle->uri);\n+\t\tresult = -1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif ((result = download_bundle_list(r, &list_from_bundle,\n+\t\t\t\t\t global_list, depth)))\n+\t\tgoto cleanup;\n+\n+cleanup:\n+\tclear_bundle_list(&list_from_bundle);\n+\treturn result;\n+}\n+\n /**\n * This limits the recursion on fetch_bundle_uri_internal() when following\n * bundle lists.\n */\n static int max_bundle_uri_depth = 4;\n \n+/**\n+ * Recursively download all bundles advertised at the given URI\n+ * to files. If the file is a bundle, then add it to the given\n+ * 'list'. Otherwise, expect a bundle list and recurse on the\n+ * URIs in that list according to the list mode (ANY or ALL).\n+ */\n static int fetch_bundle_uri_internal(struct repository *r,\n-\t\t\t\t const char *uri,\n-\t\t\t\t int depth)\n+\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t int depth,\n+\t\t\t\t struct bundle_list *list)\n {\n \tint result = 0;\n-\tchar *filename;\n+\tstruct remote_bundle_info *bcopy;\n \n \tif (depth >= max_bundle_uri_depth) {\n \t\twarning(_(\"exceeded bundle URI recursion limit (%d)\"),\n@@ -353,36 +454,106 @@ static int fetch_bundle_uri_internal(struct repository *r,\n \t\treturn -1;\n \t}\n \n-\tif (!(filename = find_temp_filename())) {\n+\tif (!bundle->file &&\n+\t !(bundle->file = find_temp_filename())) {\n \t\tresult = -1;\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = copy_uri_to_file(filename, uri))) {\n-\t\twarning(_(\"failed to download bundle from URI '%s'\"), uri);\n+\tif ((result = copy_uri_to_file(bundle->file, bundle->uri))) {\n+\t\twarning(_(\"failed to download bundle from URI '%s'\"), bundle->uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = !is_bundle(filename, 0))) {\n-\t\twarning(_(\"file at URI '%s' is not a bundle\"), uri);\n+\tif ((result = !is_bundle(bundle->file, 1))) {\n+\t\tresult = fetch_bundle_list_in_config_format(\n+\t\t\t\tr, list, bundle, depth);\n+\t\tif (result)\n+\t\t\twarning(_(\"file at URI '%s' is not a bundle or bundle list\"),\n+\t\t\t\tbundle->uri);\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = unbundle_from_file(r, filename))) {\n-\t\twarning(_(\"failed to unbundle bundle from URI '%s'\"), uri);\n-\t\tgoto cleanup;\n-\t}\n+\t/* Copy the bundle and insert it into the global list. */\n+\tCALLOC_ARRAY(bcopy, 1);\n+\tbcopy->id = xstrdup(bundle->id);\n+\tbcopy->file = xstrdup(bundle->file);\n+\thashmap_entry_init(&bcopy->ent, strhash(bcopy->id));\n+\thashmap_add(&list->bundles, &bcopy->ent);\n \n cleanup:\n-\tif (filename)\n-\t\tunlink(filename);\n-\tfree(filename);\n+\tif (result && bundle->file)\n+\t\tunlink(bundle->file);\n \treturn result;\n }\n \n+/**\n+ * This loop iterator breaks the loop with nonzero return code on the\n+ * first successful unbundling of a bundle.\n+ */\n+static int attempt_unbundle(struct remote_bundle_info *info, void *data)\n+{\n+\tstruct repository *r = data;\n+\n+\tif (!info->file || info->unbundled)\n+\t\treturn 0;\n+\n+\tif (!unbundle_from_file(r, info->file)) {\n+\t\tinfo->unbundled = 1;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int unbundle_all_bundles(struct repository *r,\n+\t\t\t\tstruct bundle_list *list)\n+{\n+\t/*\n+\t * Iterate through all bundles looking for ones that can\n+\t * successfully unbundle. If any succeed, then perhaps another\n+\t * will succeed in the next attempt.\n+\t *\n+\t * Keep in mind that a non-zero result for the loop here means\n+\t * the loop terminated early on a successful unbundling, which\n+\t * signals that we can try again.\n+\t */\n+\twhile (for_all_bundles_in_list(list, attempt_unbundle, r)) ;\n+\n+\treturn 0;\n+}\n+\n+static int unlink_bundle(struct remote_bundle_info *info, void *data)\n+{\n+\tif (info->file)\n+\t\tunlink_or_warn(info->file);\n+\treturn 0;\n+}\n+\n int fetch_bundle_uri(struct repository *r, const char *uri)\n {\n-\treturn fetch_bundle_uri_internal(r, uri, 0);\n+\tint result;\n+\tstruct bundle_list list;\n+\tstruct remote_bundle_info bundle = {\n+\t\t.uri = xstrdup(uri),\n+\t\t.id = xstrdup(\"\"),\n+\t};\n+\n+\tinit_bundle_list(&list);\n+\n+\t/* If a bundle is added to this global list, then it is required. */\n+\tlist.mode = BUNDLE_MODE_ALL;\n+\n+\tif ((result = fetch_bundle_uri_internal(r, &bundle, 0, &list)))\n+\t\tgoto cleanup;\n+\n+\tresult = unbundle_all_bundles(r, &list);\n+\n+cleanup:\n+\tfor_all_bundles_in_list(&list, unlink_bundle, NULL);\n+\tclear_bundle_list(&list);\n+\tclear_remote_bundle_info(&bundle, NULL);\n+\treturn result;\n }\n \n /**\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex bc13d4c9929..4dbc269823c 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -28,6 +28,19 @@ struct remote_bundle_info {\n \t * if there was no table of contents.\n \t */\n \tchar *uri;\n+\n+\t/**\n+\t * If the bundle has been downloaded, then 'file' is a\n+\t * filename storing its contents. Otherwise, 'file' is\n+\t * NULL.\n+\t */\n+\tchar *file;\n+\n+\t/**\n+\t * If the bundle has been unbundled successfully, then\n+\t * this boolean is true.\n+\t */\n+\tunsigned unbundled:1;\n };\n \n #define REMOTE_BUNDLE_INFO_INIT { 0 }\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex ad666a2d28a..a86dc04f528 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -41,6 +41,195 @@ test_expect_success 'clone with file:// bundle' '\n \ttest_cmp expect actual\n '\n \n+# To get interesting tests for bundle lists, we need to construct a\n+# somewhat-interesting commit history.\n+#\n+# ---------------- bundle-4\n+#\n+# 4\n+# / \\\n+# ----|---|------- bundle-3\n+# | |\n+# | 3\n+# | |\n+# ----|---|------- bundle-2\n+# | |\n+# 2 |\n+# | |\n+# ----|---|------- bundle-1\n+# \\ /\n+# 1\n+# |\n+# (previous commits)\n+test_expect_success 'construct incremental bundle list' '\n+\t(\n+\t\tcd clone-from &&\n+\t\tgit checkout -b base &&\n+\t\ttest_commit 1 &&\n+\t\tgit checkout -b left &&\n+\t\ttest_commit 2 &&\n+\t\tgit checkout -b right base &&\n+\t\ttest_commit 3 &&\n+\t\tgit checkout -b merge left &&\n+\t\tgit merge right -m \"4\" &&\n+\n+\t\tgit bundle create bundle-1.bundle base &&\n+\t\tgit bundle create bundle-2.bundle base..left &&\n+\t\tgit bundle create bundle-3.bundle base..right &&\n+\t\tgit bundle create bundle-4.bundle merge --not left right\n+\t)\n+'\n+\n+test_expect_success 'clone bundle list (file, no heuristic)' '\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-3.bundle\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-4.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-list-file &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-list-file cat-file --batch-check refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/merge\n+\trefs/bundles/right\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone bundle list (file, all mode, some failures)' '\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-0\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-0.bundle\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\n+\t# No bundle-3 means bundle-4 will not apply.\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-4.bundle\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-5\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-5.bundle\n+\tEOF\n+\n+\tGIT_TRACE2_PERF=1 \\\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-all-some &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-all-some cat-file --batch-check refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone bundle list (file, all mode, all failures)' '\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-0\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-0.bundle\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-5\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-5.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-all-fail &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-all-fail cat-file --batch-check refs &&\n+\t! grep \"refs/bundles/\" refs\n+'\n+\n+test_expect_success 'clone bundle list (file, any mode)' '\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = any\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-0\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-0.bundle\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-5\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-5.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-any-file &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-any-file cat-file --batch-check refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone bundle list (file, any mode, all failures)' '\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = any\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-0\"]\n+\t\turi = $HTTPD_URL/bundle-0.bundle\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-5\"]\n+\t\turi = $HTTPD_URL/bundle-5.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-any-fail &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-any-fail cat-file --batch-check refs &&\n+\t! grep \"refs/bundles/\" refs\n+'\n+\n #########################################################################\n # HTTP tests begin here\n \n@@ -75,6 +264,65 @@ test_expect_success 'clone HTTP bundle' '\n \ttest_config -C clone-http log.excludedecoration refs/bundle/\n '\n \n+test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n+\tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = $HTTPD_URL/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = $HTTPD_URL/bundle-2.bundle\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = $HTTPD_URL/bundle-3.bundle\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = $HTTPD_URL/bundle-4.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" clone-from clone-list-http &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-list-http cat-file --batch-check \"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = any\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-0\"]\n+\t\turi = $HTTPD_URL/bundle-0.bundle\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = $HTTPD_URL/bundle-1.bundle\n+\n+\t# Does not exist. Should be skipped.\n+\t[bundle \"bundle-5\"]\n+\t\turi = $HTTPD_URL/bundle-5.bundle\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" clone-from clone-any-http &&\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-any-http cat-file --batch-check refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/merge\n+\trefs/bundles/right\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n # Do not add tests here unless they use the HTTP server, as they will\n # not run unless the HTTP dependencies exist.\n \n-- \ngitgitgadget\n\n"},{"id":"464706","messageId":"5729ff2af4bb56a68624b7942b8afa67601adb43.1665579160.git.gitgitgadget@gmail.com","threadId":"58342","inReplyTo":"pull.1333.v5.git.1665579160.gitgitgadget@gmail.com","subject":"[PATCH v5 12/12] bundle-uri: suppress stderr from remote-https","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T12:52:39Z","receivedAt":"2022-10-12T12:53:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen downloading bundles from a git-remote-https subprocess, the bundle\nURI logic wants to be opportunistic and download as much as possible and\nwork with what did succeed. This is particularly important in the \"any\"\nmode, where any single bundle success will work.\n\nIf the URI is not available, the git-remote-https process will die()\nwith a \"fatal:\" error message, even though that error is not actually\nfatal to the super process. Since stderr is passed through, it looks\nlike a fatal error to the user.\n\nSuppress stderr to avoid these errors from bubbling to the surface. The\nbundle URI API adds its own warning() messages on these failures.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 1 +\n t/t5558-clone-bundle-uri.sh | 16 ++++++++++++++--\n 2 files changed, 15 insertions(+), 2 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex d872acf5ab0..79a914f961b 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -230,6 +230,7 @@ static int download_https_uri_to_file(const char *file, const char *uri)\n \tint found_get = 0;\n \n \tstrvec_pushl(&cp.args, \"git-remote-https\", uri, NULL);\n+\tcp.err = -1;\n \tcp.in = -1;\n \tcp.out = -1;\n \ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 9b159078386..9155f31fa2c 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -147,6 +147,8 @@ test_expect_success 'clone bundle list (file, all mode, some failures)' '\n \tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n \t\tclone-from clone-all-some 2>err &&\n \t! grep \"Repository lacks these prerequisite commits\" err &&\n+\t! grep \"fatal\" err &&\n+\tgrep \"warning: failed to download bundle from URI\" err &&\n \n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-all-some cat-file --batch-check err &&\n \t! grep \"Repository lacks these prerequisite commits\" err &&\n+\t! grep \"fatal\" err &&\n+\tgrep \"warning: failed to download bundle from URI\" err &&\n \n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-all-fail cat-file --batch-check err &&\n+\t! grep \"fatal\" err &&\n+\tgrep \"warning: failed to download bundle from URI\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-any-fail cat-file --batch-check err &&\n+\t! grep \"fatal\" err &&\n+\tgrep \"warning: failed to download bundle from URI\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-any-http cat-file --batch-check \n\nWhen downloading a list of bundles in \"all\" mode, Git has no\nunderstanding of the dependencies between the bundles. Git attempts to\nunbundle the bundles in some order, but some may not pass the\nverify_bundle() step because of missing prerequisites. This is passed as\nerror messages to the user, even when they eventually succeed in later\nattempts after their dependent bundles are unbundled.\n\nAdd a new VERIFY_BUNDLE_QUIET flag to verify_bundle() that avoids the\nerror messages from the missing prerequisite commits. The method still\nreturns the number of missing prerequisit commits, allowing callers to\nunbundle() to notice that the bundle failed to apply.\n\nUse this flag in bundle-uri.c and test that the messages go away for\n'git clone --bundle-uri' commands.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/bundle.c | 2 +-\n bundle-uri.c | 3 ++-\n bundle.c | 10 ++++++++--\n bundle.h | 1 +\n t/t5558-clone-bundle-uri.sh | 25 ++++++++++++++++++++-----\n 5 files changed, 32 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex 7d983a238f0..fd4586b09e0 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -120,7 +120,7 @@ static int cmd_bundle_verify(int argc, const char **argv, const char *prefix) {\n \t}\n \tclose(bundle_fd);\n \tif (verify_bundle(the_repository, &header,\n-\t\t\t quiet ? 0 : VERIFY_BUNDLE_VERBOSE)) {\n+\t\t\t quiet ? VERIFY_BUNDLE_QUIET : VERIFY_BUNDLE_VERBOSE)) {\n \t\tret = 1;\n \t\tgoto cleanup;\n \t}\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex d9060be707e..d872acf5ab0 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -308,7 +308,8 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t * a reachable ref pointing to the new tips, which will reach\n \t * the prerequisite commits.\n \t */\n-\tif ((result = unbundle(r, &header, bundle_fd, NULL, 0)))\n+\tif ((result = unbundle(r, &header, bundle_fd, NULL,\n+\t\t\t VERIFY_BUNDLE_QUIET)))\n \t\treturn 1;\n \n \t/*\ndiff --git a/bundle.c b/bundle.c\nindex 1f6a7f782e1..4ef7256aa11 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -216,7 +216,10 @@ int verify_bundle(struct repository *r,\n \t\t\tadd_pending_object(&revs, o, name);\n \t\t\tcontinue;\n \t\t}\n-\t\tif (++ret == 1)\n+\t\tret++;\n+\t\tif (flags & VERIFY_BUNDLE_QUIET)\n+\t\t\tcontinue;\n+\t\tif (ret == 1)\n \t\t\terror(\"%s\", message);\n \t\terror(\"%s %s\", oid_to_hex(oid), name);\n \t}\n@@ -243,7 +246,10 @@ int verify_bundle(struct repository *r,\n \t\tassert(o); /* otherwise we'd have returned early */\n \t\tif (o->flags & SHOWN)\n \t\t\tcontinue;\n-\t\tif (++ret == 1)\n+\t\tret++;\n+\t\tif (flags & VERIFY_BUNDLE_QUIET)\n+\t\t\tcontinue;\n+\t\tif (ret == 1)\n \t\t\terror(\"%s\", message);\n \t\terror(\"%s %s\", oid_to_hex(oid), name);\n \t}\ndiff --git a/bundle.h b/bundle.h\nindex 6652e819981..575c34245d1 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -32,6 +32,7 @@ int create_bundle(struct repository *r, const char *path,\n \n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n+\tVERIFY_BUNDLE_QUIET = (1 << 1),\n };\n \n int verify_bundle(struct repository *r, struct bundle_header *header,\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex a86dc04f528..9b159078386 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -99,7 +99,10 @@ test_expect_success 'clone bundle list (file, no heuristic)' '\n \t\turi = file://$(pwd)/clone-from/bundle-4.bundle\n \tEOF\n \n-\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" clone-from clone-list-file &&\n+\tgit clone --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from clone-list-file 2>err &&\n+\t! grep \"Repository lacks these prerequisite commits\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-list-file cat-file --batch-check err &&\n+\t! grep \"Repository lacks these prerequisite commits\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-all-some cat-file --batch-check err &&\n+\t! grep \"Repository lacks these prerequisite commits\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-all-fail cat-file --batch-check err &&\n+\t! grep \"Repository lacks these prerequisite commits\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-any-file cat-file --batch-check err &&\n+\t! grep \"Repository lacks these prerequisite commits\" err &&\n+\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-list-http cat-file --batch-check writes:\n\n> From: Derrick Stolee \n>\n> The verify_bundle() method checks two things for a bundle's\n> prerequisites:\n>\n> 1. Are these objects in the object store?\n> 2. Are these objects reachable from our references?\n>\n> In this second question, multiple uses of verify_bundle() in the same\n> process can report an invalid bundle even though it is correct. The\n> reason is due to not clearing all of the commit marks on the commits\n> previously walked.\n> ...\n> Moving this loop to the cleanup step alone would cause a segfault when\n> running 'git bundle verify' outside of a repository, but this is because\n> of that error condition using \"goto cleanup\" when returning is perfectly\n> safe. Nothing has been initialized at that point, so we can return\n> immediately without causing any leaks.\n\nNicely analyzed. The implementation clearly follows the design\ndescribed above. Much better than the previous iteration.\n\nThanks.\n"},{"id":"464727","messageId":"xmqqtu49knnu.fsf@gitster.g","threadId":"58342","inReplyTo":"2e0bfa834f14e1f4f66adf36d4326e5f1c2c9c20.1665579160.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 11/12] bundle-uri: quiet failed unbundlings","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-12T16:32:53Z","receivedAt":"2022-10-12T16:33:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> When downloading a list of bundles in \"all\" mode, Git has no\n> understanding of the dependencies between the bundles. Git attempts to\n> unbundle the bundles in some order, but some may not pass the\n> verify_bundle() step because of missing prerequisites. This is passed as\n> error messages to the user, even when they eventually succeed in later\n> attempts after their dependent bundles are unbundled.\n>\n> Add a new VERIFY_BUNDLE_QUIET flag to verify_bundle() that avoids the\n> error messages from the missing prerequisite commits. The method still\n> returns the number of missing prerequisit commits, allowing callers to\n> unbundle() to notice that the bundle failed to apply.\n>\n> Use this flag in bundle-uri.c and test that the messages go away for\n> 'git clone --bundle-uri' commands.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n\nInteresting that we ended up with verbosity\nlevels, but \"bundle verify --verbose\" does not (have to) exist, as\nthat is the default, and 0 (aka \"normal\") is no longer used to call\nverify_bundle() by anybody.\n\nI actually was wondering that with SKIP_REACHABLE gone, we would\nlose the \"enum verify_bundle_flags\" altogether, without the need for\na new \"quiet\" option. But that would not work as unbundle() calls\nverify_bundle() and callers of unbundle() do not necessarily want\nthe verification step to squelch errors.\n\nSo looks good overall.\n\nThanks.\n"},{"id":"465778","messageId":"0267162a-098b-2280-46b6-51932a8df6c3@github.com","threadId":"58342","inReplyTo":"pull.1333.v5.git.1665579160.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 00/12] Bundle URIs III: Parse and download from bundle lists","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-10-26T14:34:41Z","receivedAt":"2022-10-26T14:34:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/12/2022 8:52 AM, Derrick Stolee via GitGitGadget wrote:\n> This is the third series building the bundle URI feature. It is built on top\n> of ds/bundle-uri-clone, which introduced 'git clone --bundle-uri=' where is\n> a URI to a bundle file. This series adds the capability of downloading and\n> parsing a bundle list and then downloading the URIs in that list.\n> \n> The core functionality of bundle lists is implemented by creating data\n> structures from a list of key-value pairs. These pairs can come from a\n> plain-text file in Git config format, but in the future, we will support the\n> list being supplied by packet lines over Git's protocol v2 in the\n> 'bundle-uri' command (reserved for the next series).\n\nThis version has been available for a while now without comment. Could\nwe consider it for merging to 'next' soon?\n\nI want to wait for this series to merge into 'master' before sending\npart IV on top, which advertises bundle URIs over protocol v2.\n\nThanks,\n-Stolee\n"},{"id":"465806","messageId":"xmqq4jvqlgc6.fsf@gitster.g","threadId":"58342","inReplyTo":"0267162a-098b-2280-46b6-51932a8df6c3@github.com","subject":"Re: [PATCH v5 00/12] Bundle URIs III: Parse and download from bundle lists","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-26T16:06:33Z","receivedAt":"2022-10-26T16:06:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> This version has been available for a while now without comment. Could\n> we consider it for merging to 'next' soon?\n\nCould somebody who has reviewed it fully give an Ack (or two)? I\nknow an earlier rounds had some comments, but after v3 things have\nquieted down.\n\nI know the change from v4 to v5 has good improvements, but do not\nclaim to have read the other parts in detail.\n\nThanks.\n"},{"id":"465815","messageId":"xmqqmt9ih0um.fsf@gitster.g","threadId":"58342","inReplyTo":"5729ff2af4bb56a68624b7942b8afa67601adb43.1665579160.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 12/12] bundle-uri: suppress stderr from remote-https","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-26T18:54:41Z","receivedAt":"2022-10-26T18:54:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> When downloading bundles from a git-remote-https subprocess, the bundle\n> URI logic wants to be opportunistic and download as much as possible and\n> work with what did succeed. This is particularly important in the \"any\"\n> mode, where any single bundle success will work.\n>\n> If the URI is not available, the git-remote-https process will die()\n> with a \"fatal:\" error message, even though that error is not actually\n> fatal to the super process. Since stderr is passed through, it looks\n> like a fatal error to the user.\n>\n> Suppress stderr to avoid these errors from bubbling to the surface. The\n> bundle URI API adds its own warning() messages on these failures.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> bundle-uri.c | 1 +\n> t/t5558-clone-bundle-uri.sh | 16 ++++++++++++++--\n> 2 files changed, 15 insertions(+), 2 deletions(-)\n\nSo this is the same in spirit as [11/12] to squelch errors from an\naction that we are prepared to fail. If we had an easy way to\nsquelch only one class of errors (e.g. the resource no longer exists\nat the URI) while allowing others to pass (e.g. we downloaded but it\nwas corrupt), that might be even better when somebody is debugging\nthe thing, but to an end user, it is a hopefully ignorable failure\neither way, as long as there are other alternatives in the set of\nbundles in the \"any\" mode.\n"},{"id":"465816","messageId":"xmqqilk6h0be.fsf@gitster.g","threadId":"58342","inReplyTo":"51e9b8474fb040c4277b96d4b13a8da58bdb3682.1665579160.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 09/12] bundle-uri: fetch a list of bundles","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-26T19:06:13Z","receivedAt":"2022-10-26T19:09:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +static int fetch_bundle_list_in_config_format(struct repository *r,\n> +\t\t\t\t\t struct bundle_list *global_list,\n> +\t\t\t\t\t struct remote_bundle_info *bundle,\n> +\t\t\t\t\t int depth)\n> +{\n> +\tint result;\n> +\tstruct bundle_list list_from_bundle;\n> +\n> +\tinit_bundle_list(&list_from_bundle);\n> +\n> +\tif ((result = bundle_uri_parse_config_format(bundle->uri,\n> +\t\t\t\t\t\t bundle->file,\n> +\t\t\t\t\t\t &list_from_bundle)))\n> +\t\tgoto cleanup;\n\nIt makes us a bit nervous to apply the config parser directly on\ndata controlled by a third-party. bundle_uri_parse_config_format()\nhopefully is careful enough to avoid including other local files and\ncall generic callbacks to affect the actual configuration used by\nthe process.\n\nIt seems bundle_list_update() discards everything it does not (care\nto) understand, and safe to call from config_to_bundle_list(), which\nin turn is called from here.\n\nOK.\n"}]}