{"thread":{"id":"59040","subject":"[PATCH 0/8] Bundle URIs V: creationToken heuristic for incremental fetches","startedAt":"2023-01-06T20:36:58Z","lastAt":"2023-01-31T22:01:13Z","messageCount":79,"participants":["Derrick Stolee via GitGitGadget","Junio C Hamano","Derrick Stolee","Victoria Dye"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"469844","messageId":"pull.1454.git.1673037405.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":null,"subject":"[PATCH 0/8] Bundle URIs V: creationToken heuristic for incremental fetches","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-06T20:36:37Z","receivedAt":"2023-01-06T20:36:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This fifth part to the bundle URIs feature follows part IV (advertising via\nprotocol v2) which recently merged to 'master', so this series is based on\n'master'.\n\nThis part introduces the concept of a heuristic that a bundle list can\nadvertise. The purpose of the heuristic is to hint to the Git client that\nthe bundles can be downloaded and unbundled in a certain order. In\nparticular, that order can assist with using the same bundle URI to download\nnew bundles from an updated bundle list. This allows bundle URIs to assist\nwith incremental fetches, not just initial clones.\n\nThe only planned heuristic is the \"creationToken\" heuristic where the bundle\nlist adds a 64-bit unsigned integer \"creationToken\" value to each bundle in\nthe list. Those values provide an ordering on the bundles implying that the\nbundles can be unbundled in increasing creationToken order and at each point\nthe required commits for the ith bundle were provided by bundles with lower\ncreationTokens.\n\nAt clone time, the only difference implied by the creationToken order is\nthat the Git client does not need to guess at the order to apply the\nbundles, but instead can use the creationToken order to apply them without\nfailure and retry. However, this presents an interesting benefit during\nfetches: the Git client can check the bundle list and download bundles in\ndecreasing creationToken order until the required commits for these bundles\nare present within the repository's object store. This prevents downloading\nmore bundle information than required.\n\nThe creationToken value is also a promise that the Git client will not need\nto download a bundle if its creationToken is less than or equal to the\ncreationToken of a previously-downloaded bundle. This further improves the\nperformance during a fetch in that the client does not need to download any\nbundles at all if it recognizes that the maximum creationToken is the same\n(or smaller than) a previously-downloaded creationToken.\n\nThe creationToken concept is documented in the existing design document at\nDocumentation/technical/bundle-uri.txt, including suggested ways for bundle\nproviders to organize their bundle lists to take advantage of the heuristic.\n\nThis series formalizes the creationToken heuristic and the Git client logic\nfor understanding it. Further, for bundle lists provided by the git clone\n--bundle-uri option, the Git client will recognize the heuristic as being\nhelpful for incremental fetches and store config values so that future git\nfetch commands check the bundle list before communicating with any Git\nremotes.\n\nNote that this option does not integrate fetches with bundle lists\nadvertised via protocol v2. I spent some time working on this, but found the\nimplementation to be distinct enough that it merited its own attention in a\nseparate series. In particular, the configuration for indicating that a\nfetch should check the bundle-uri protocol v2 command seemed best to be\nlocated within a Git remote instead of a repository-global key such as is\nbeing used for a static URI. Further, the timing of querying the bundle-uri\ncommand during a git fetch command is significantly different and more\ncomplicated than how it is used in git clone.\n\n\nWhat Remains?\n=============\n\nOriginally, I had planned on making this bundle URI work a 5-part series,\nand this is part 5. Shouldn't we be done now?\n\nThere are two main things that should be done after this series, in any\norder:\n\n * Teach git fetch to check a bundle list advertised by a remote over the\n bundle-uri protocol v2 command.\n * Add the bundle..filter option to allow advertising bundles and\n partial bundles side-by-side.\n\nThere is also room for expanding tests for more error conditions, or for\nother tweaks that are not currently part of the design document. I do think\nthat after this series, the feature will be easier to work on different\nparts in parallel.\n\n\nPatch Outline\n=============\n\n * Patch 1 creates a test setup demonstrating a creationToken heuristic. At\n this point, the Git client ignores the heuristic and uses its ad-hoc\n strategy for ordering the bundles.\n * Patches 2 and 3 teach Git to parse the bundle.heuristic and\n bundle..creationToken keys in a bundle list.\n * Patch 4 teaches Git to download bundles using the creationToken order.\n This order uses a stack approach to start from the maximum creationToken\n and continue downloading the next bundle in the list until all bundles\n can successfully be unbundled. This is the algorithm required for\n incremental fetches, while initial clones could download in the opposite\n order. Since clones will download all bundles anyway, having a second\n code path just for clones seemed unnecessary.\n * Patch 5 teaches git clone --bundle-uri to set fetch.bundleURI when the\n advertised bundle list includs a heuristic that Git understands.\n * Patch 6 updates the design document to remove reference to a bundle.flag\n option that was previously going to indicate the list was designed for\n fetches, but the bundle.heuristic option already does that.\n * Patch 7 teaches git fetch to check fetch.bundleURI and download bundles\n from that static URI before connecting to remotes via the Git protocol.\n * Patch 8 introduces a new fetch.bundleCreationToken config value to store\n the maximum creationToken of downloaded bundles. This prevents\n downloading the latest bundle on every git fetch command, reducing waste.\n\nThanks,\n\n * Stolee\n\nDerrick Stolee (8):\n t5558: add tests for creationToken heuristic\n bundle-uri: parse bundle.heuristic=creationToken\n bundle-uri: parse bundle..creationToken values\n bundle-uri: download in creationToken order\n clone: set fetch.bundleURI if appropriate\n bundle-uri: drop bundle.flag from design doc\n fetch: fetch from an external bundle URI\n bundle-uri: store fetch.bundleCreationToken\n\n Documentation/config/bundle.txt | 7 +\n Documentation/config/fetch.txt | 16 ++\n Documentation/technical/bundle-uri.txt | 8 +-\n builtin/clone.c | 6 +-\n builtin/fetch.c | 8 +\n bundle-uri.c | 208 ++++++++++++++++++++++++-\n bundle-uri.h | 28 +++-\n t/t5558-clone-bundle-uri.sh | 204 +++++++++++++++++++++++-\n t/t5601-clone.sh | 50 ++++++\n t/t5750-bundle-uri-parse.sh | 37 +++++\n 10 files changed, 561 insertions(+), 11 deletions(-)\n\n\nbase-commit: 4dbebc36b0893f5094668ddea077d0e235560b16\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1454%2Fderrickstolee%2Fbundle-redo%2FcreationToken-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1454/derrickstolee/bundle-redo/creationToken-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1454\n-- \ngitgitgadget\n"},{"id":"469845","messageId":"9007249b9488c23f00c2d498ffd520e4af8b37a4.1673037405.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.git.1673037405.gitgitgadget@gmail.com","subject":"[PATCH 2/8] bundle-uri: parse bundle.heuristic=creationToken","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-06T20:36:39Z","receivedAt":"2023-01-06T20:37:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe bundle.heuristic value communicates that the bundle list is\norganized to make use of the bundle..creationToken values that may\nbe provided in the bundle list. Those values will create a total order\non the bundles, allowing the Git client to download them in a specific\norder and even remember previously-downloaded bundles by storing the\nmaximum creation token value.\n\nBefore implementing any logic that parses or uses the\nbundle..creationToken values, teach Git to parse the\nbundle.heuristic value from a bundle list. We can use 'test-tool\nbundle-uri' to print the heuristic value and verify that the parsing\nworks correctly.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/bundle.txt | 7 +++++++\n bundle-uri.c | 21 +++++++++++++++++++++\n bundle-uri.h | 14 ++++++++++++++\n t/t5750-bundle-uri-parse.sh | 19 +++++++++++++++++++\n 4 files changed, 61 insertions(+)\n\ndiff --git a/Documentation/config/bundle.txt b/Documentation/config/bundle.txt\nindex daa21eb674a..3faae386853 100644\n--- a/Documentation/config/bundle.txt\n+++ b/Documentation/config/bundle.txt\n@@ -15,6 +15,13 @@ bundle.mode::\n \tcomplete understanding of the bundled information (`all`) or if any one\n \tof the listed bundle URIs is sufficient (`any`).\n \n+bundle.heuristic::\n+\tIf this string-valued key exists, then the bundle list is designed to\n+\twork well with incremental `git fetch` commands. The heuristic signals\n+\tthat there are additional keys available for each bundle that help\n+\tdetermine which subset of bundles the client should download. The\n+\tonly value currently understood is `creationToken`.\n+\n bundle..*::\n \tThe `bundle..*` keys are used to describe a single item in the\n \tbundle list, grouped under `` for identification purposes.\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 36268dda172..56c94595c2a 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -9,6 +9,11 @@\n #include \"config.h\"\n #include \"remote.h\"\n \n+static const char *heuristics[] = {\n+\t[BUNDLE_HEURISTIC_NONE] = \"\",\n+\t[BUNDLE_HEURISTIC_CREATIONTOKEN] = \"creationToken\",\n+};\n+\n static int compare_bundles(const void *hashmap_cmp_fn_data,\n \t\t\t const struct hashmap_entry *he1,\n \t\t\t const struct hashmap_entry *he2,\n@@ -100,6 +105,9 @@ void print_bundle_list(FILE *fp, struct bundle_list *list)\n \tfprintf(fp, \"\\tversion = %d\\n\", list->version);\n \tfprintf(fp, \"\\tmode = %s\\n\", mode);\n \n+\tif (list->heuristic)\n+\t\tprintf(\"\\theuristic = %s\\n\", heuristics[list->heuristic]);\n+\n \tfor_all_bundles_in_list(list, summarize_bundle, fp);\n }\n \n@@ -142,6 +150,19 @@ static int bundle_list_update(const char *key, const char *value,\n \t\t\treturn 0;\n \t\t}\n \n+\t\tif (!strcmp(subkey, \"heuristic\")) {\n+\t\t\tint i;\n+\t\t\tfor (i = 0; i < BUNDLE_HEURISTIC__COUNT; i++) {\n+\t\t\t\tif (!strcmp(value, heuristics[i])) {\n+\t\t\t\t\tlist->heuristic = i;\n+\t\t\t\t\treturn 0;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\t/* Ignore unknown heuristics. */\n+\t\t\treturn 0;\n+\t\t}\n+\n \t\t/* Ignore other unknown global keys. */\n \t\treturn 0;\n \t}\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex d5e89f1671c..ad82174112d 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -52,6 +52,14 @@ enum bundle_list_mode {\n \tBUNDLE_MODE_ANY\n };\n \n+enum bundle_list_heuristic {\n+\tBUNDLE_HEURISTIC_NONE = 0,\n+\tBUNDLE_HEURISTIC_CREATIONTOKEN,\n+\n+\t/* Must be last. */\n+\tBUNDLE_HEURISTIC__COUNT,\n+};\n+\n /**\n * A bundle_list contains an unordered set of remote_bundle_info structs,\n * as well as information about the bundle listing, such as version and\n@@ -75,6 +83,12 @@ struct bundle_list {\n \t * advertised by the bundle list at that location.\n \t */\n \tchar *baseURI;\n+\n+\t/**\n+\t * A list can have a heuristic, which helps reduce the number of\n+\t * downloaded bundles.\n+\t */\n+\tenum bundle_list_heuristic heuristic;\n };\n \n void init_bundle_list(struct bundle_list *list);\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex 7b4f930e532..6fc92a9c0d4 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -250,4 +250,23 @@ test_expect_success 'parse config format edge cases: empty key or value' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format: creationToken heuristic' '\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config expect >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"469846","messageId":"39eed9148782c37f5184c5fff7d0e4d1a7a2a1fe.1673037405.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.git.1673037405.gitgitgadget@gmail.com","subject":"[PATCH 1/8] t5558: add tests for creationToken heuristic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-06T20:36:38Z","receivedAt":"2023-01-06T20:37:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs documented in the bundle URI design doc in 2da14fad8fe (docs:\ndocument bundle URI standard, 2022-08-09), the 'creationToken' member of\na bundle URI allows a bundle provider to specify a total order on the\nbundles.\n\nFuture changes will allow the Git client to understand these members and\nmodify its behavior around downloading the bundles in that order. In the\nmeantime, create tests that add creation tokens to the bundle list. For\nnow, the Git client correctly ignores these unknown keys.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5558-clone-bundle-uri.sh | 52 +++++++++++++++++++++++++++++++++++--\n 1 file changed, 50 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 9155f31fa2c..328caeeae9a 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -284,7 +284,17 @@ test_expect_success 'clone HTTP bundle' '\n \ttest_config -C clone-http log.excludedecoration refs/bundle/\n '\n \n+# usage: test_bundle_downloaded \n+test_bundle_downloaded () {\n+\tcat >pattern <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/$1\"\\]\n+\tEOF\n+\tgrep -f pattern \"$2\"\n+}\n+\n test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\n \tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n \tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n \t[bundle]\n@@ -304,12 +314,19 @@ test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n \t\turi = $HTTPD_URL/bundle-4.bundle\n \tEOF\n \n-\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" \\\n+\t\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n \t\tclone-from clone-list-http 2>err &&\n \t! grep \"Repository lacks these prerequisite commits\" err &&\n \n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n-\tgit -C clone-list-http cat-file --batch-check \"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\tEOF\n+\n+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" . clone-list-http-2 &&\n+\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-list-http-2 cat-file --batch-check \n\nThe Implementation Plan section lists a 'bundle.flag' option that is not\ndocumented anywhere else. What is documented elsewhere in the document\nand implemented by previous changes is the 'bundle.heuristic' config\nkey. For now, a heuristic is required to indicate that a bundle list is\norganized for use during 'git fetch', and it is also sufficient for all\nexisting designs.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/bundle-uri.txt | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/technical/bundle-uri.txt b/Documentation/technical/bundle-uri.txt\nindex b78d01d9adf..91d3a13e327 100644\n--- a/Documentation/technical/bundle-uri.txt\n+++ b/Documentation/technical/bundle-uri.txt\n@@ -479,14 +479,14 @@ outline for submitting these features:\n (This choice is an opt-in via a config option and a command-line\n option.)\n \n-4. Allow the client to understand the `bundle.flag=forFetch` configuration\n+4. Allow the client to understand the `bundle.heuristic` configuration key\n and the `bundle..creationToken` heuristic. When `git clone`\n- discovers a bundle URI with `bundle.flag=forFetch`, it configures the\n- client repository to check that bundle URI during later `git fetch `\n+ discovers a bundle URI with `bundle.heuristic`, it configures the client\n+ repository to check that bundle URI during later `git fetch `\n commands.\n \n 5. Allow clients to discover bundle URIs during `git fetch` and configure\n- a bundle URI for later fetches if `bundle.flag=forFetch`.\n+ a bundle URI for later fetches if `bundle.heuristic` is set.\n \n 6. Implement the \"inspect headers\" heuristic to reduce data downloads when\n the `bundle..creationToken` heuristic is not available.\n-- \ngitgitgadget\n\n"},{"id":"469848","messageId":"d9c6f50e4f218267c1e8da060ce5b190dc8a709c.1673037405.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.git.1673037405.gitgitgadget@gmail.com","subject":"[PATCH 5/8] clone: set fetch.bundleURI if appropriate","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-06T20:36:42Z","receivedAt":"2023-01-06T20:37:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBundle providers may organize their bundle lists in a way that is\nintended to improve incremental fetches, not just initial clones.\nHowever, they do need to state that they have organized with that in\nmind, or else the client will not expect to save time by downloading\nbundles after the initial clone. This is done by specifying a\nbundle.heuristic value.\n\nThere are two types of bundle lists: those at a static URI and those\nthat are advertised from a Git remote over protocol v2.\n\nThe new fetch.bundleURI config value applies for static bundle URIs that\nare not advertised over protocol v2. If the user specifies a static URI\nvia 'git clone --bundle-uri', then Git can set this config as a reminder\nfor future 'git fetch' operations to check the bundle list before\nconnecting to the remote(s).\n\nFor lists provided over protocol v2, we will want to take a different\napproach and create a property of the remote itself by creating a\nremote..* type config key. That is not implemented in this change.\n\nLater changes will update 'git fetch' to consume this option.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/fetch.txt | 8 ++++++++\n builtin/clone.c | 6 +++++-\n bundle-uri.c | 10 +++++++---\n bundle-uri.h | 8 +++++++-\n t/t5558-clone-bundle-uri.sh | 33 +++++++++++++++++++++++++++++++++\n 5 files changed, 60 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt\nindex cd65d236b43..4f796218aab 100644\n--- a/Documentation/config/fetch.txt\n+++ b/Documentation/config/fetch.txt\n@@ -96,3 +96,11 @@ fetch.writeCommitGraph::\n \tmerge and the write may take longer. Having an updated commit-graph\n \tfile helps performance of many Git commands, including `git merge-base`,\n \t`git push -f`, and `git log --graph`. Defaults to false.\n+\n+fetch.bundleURI::\n+\tThis value stores a URI for fetching Git object data from a bundle URI\n+\tbefore performing an incremental fetch from the origin Git server. If\n+\tthe value is `` then running `git fetch ` is equivalent to\n+\tfirst running `git fetch --bundle-uri=` immediately before\n+\t`git fetch `. See details of the `--bundle-uri` option in\n+\tlinkgit:git-fetch[1].\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 5453ba5277f..5370617664d 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1248,12 +1248,16 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t * data from the --bundle-uri option.\n \t */\n \tif (bundle_uri) {\n+\t\tint has_heuristic = 0;\n+\n \t\t/* At this point, we need the_repository to match the cloned repo. */\n \t\tif (repo_init(the_repository, git_dir, work_tree))\n \t\t\twarning(_(\"failed to initialize the repo, skipping bundle URI\"));\n-\t\telse if (fetch_bundle_uri(the_repository, bundle_uri))\n+\t\telse if (fetch_bundle_uri(the_repository, bundle_uri, &has_heuristic))\n \t\t\twarning(_(\"failed to fetch objects from bundle URI '%s'\"),\n \t\t\t\tbundle_uri);\n+\t\telse if (has_heuristic)\n+\t\t\tgit_config_set_gently(\"fetch.bundleuri\", bundle_uri);\n \t}\n \n \tstrvec_push(&transport_ls_refs_options.ref_prefixes, \"HEAD\");\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex b30c85ba6f2..1dbbbb980eb 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -594,9 +594,10 @@ static int fetch_bundle_list_in_config_format(struct repository *r,\n \t * it advertises are expected to be bundles, not nested lists.\n \t * We can drop 'global_list' and 'depth'.\n \t */\n-\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN)\n+\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN) {\n \t\tresult = fetch_bundles_by_token(r, &list_from_bundle);\n-\telse if ((result = download_bundle_list(r, &list_from_bundle,\n+\t\tglobal_list->heuristic = BUNDLE_HEURISTIC_CREATIONTOKEN;\n+\t} else if ((result = download_bundle_list(r, &list_from_bundle,\n \t\t\t\t\t global_list, depth)))\n \t\tgoto cleanup;\n \n@@ -707,7 +708,8 @@ static int unlink_bundle(struct remote_bundle_info *info, void *data)\n \treturn 0;\n }\n \n-int fetch_bundle_uri(struct repository *r, const char *uri)\n+int fetch_bundle_uri(struct repository *r, const char *uri,\n+\t\t int *has_heuristic)\n {\n \tint result;\n \tstruct bundle_list list;\n@@ -727,6 +729,8 @@ int fetch_bundle_uri(struct repository *r, const char *uri)\n \tresult = unbundle_all_bundles(r, &list);\n \n cleanup:\n+\tif (has_heuristic)\n+\t\t*has_heuristic = (list.heuristic != BUNDLE_HEURISTIC_NONE);\n \tfor_all_bundles_in_list(&list, unlink_bundle, NULL);\n \tclear_bundle_list(&list);\n \tclear_remote_bundle_info(&bundle, NULL);\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 1cae418211b..52b27cd10e3 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -124,8 +124,14 @@ int bundle_uri_parse_config_format(const char *uri,\n * based on that information.\n *\n * Returns non-zero if no bundle information is found at the given 'uri'.\n+ *\n+ * If the pointer 'has_heuristic' is non-NULL, then the value it points to\n+ * will be set to be non-zero if and only if the fetched list has a\n+ * heuristic value. Such a value indicates that the list was designed for\n+ * incremental fetches.\n */\n-int fetch_bundle_uri(struct repository *r, const char *uri);\n+int fetch_bundle_uri(struct repository *r, const char *uri,\n+\t\t int *has_heuristic);\n \n /**\n * Given a bundle list that was already advertised (likely by the\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex d7461ec907e..8ff560425ee 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -435,6 +435,39 @@ test_expect_success 'clone bundle list (http, creationToken)' '\n \ttest_bundle_downloaded bundle-2.bundle trace-clone.txt\n '\n \n+test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n+\ttest_when_finished rm -rf fetch-http-4 trace*.txt &&\n+\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" \\\n+\tgit clone --single-branch --branch=base \\\n+\t\t--bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\t\"$HTTPD_URL/smart/fetch.git\" fetch-http-4 &&\n+\n+\ttest_cmp_config -C fetch-http-4 \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n+\n+\t# The clone should copy two files: the list and bundle-1.\n+\ttest_bundle_downloaded bundle-list trace-clone.txt &&\n+\ttest_bundle_downloaded bundle-1.bundle trace-clone.txt &&\n+\n+\t# only received base ref from bundle-1\n+\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\tEOF\n+\ttest_cmp expect refs\n+'\n+\n # Do not add tests here unless they use the HTTP server, as they will\n # not run unless the HTTP dependencies exist.\n \n-- \ngitgitgadget\n\n"},{"id":"469849","messageId":"1627fc158b1e301a1663e24f9f21268b4f1caa55.1673037405.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.git.1673037405.gitgitgadget@gmail.com","subject":"[PATCH 7/8] fetch: fetch from an external bundle URI","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-06T20:36:44Z","receivedAt":"2023-01-06T20:37:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a user specifies a URI via 'git clone --bundle-uri', that URI may\nbe a bundle list that advertises a 'bundle.heuristic' value. In that\ncase, the Git client stores a 'fetch.bundleURI' config value storing\nthat URI.\n\nTeach 'git fetch' to check for this config value and download bundles\nfrom that URI before fetching from the Git remote(s). Likely, the bundle\nprovider has configured a heuristic (such as \"creationToken\") that will\nallow the Git client to download only a portion of the bundles before\ncontinuing the fetch.\n\nSince this URI is completely independent of the remote server, we want\nto be sure that we connect to the bundle URI before creating a\nconnection to the Git remote. We do not want to hold a stateful\nconnection for too long if we can avoid it.\n\nTo test that this works correctly, extend the previous tests that set\n'fetch.bundleURI' to do follow-up fetches. The bundle list is updated\nincrementally at each phase to demonstrate that the heuristic avoids\ndownloading older bundles. This includes the middle fetch downloading\nthe objects in bundle-3.bundle from the Git remote, and therefore not\nneeding that bundle in the third fetch.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/fetch.c | 8 +++++\n t/t5558-clone-bundle-uri.sh | 59 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 67 insertions(+)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 7378cafeec9..fbb1d470c38 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -29,6 +29,7 @@\n #include \"commit-graph.h\"\n #include \"shallow.h\"\n #include \"worktree.h\"\n+#include \"bundle-uri.h\"\n \n #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n \n@@ -2109,6 +2110,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,\n int cmd_fetch(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n+\tconst char *bundle_uri;\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n \tstruct remote *remote = NULL;\n \tint result = 0;\n@@ -2194,6 +2196,12 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (dry_run)\n \t\twrite_fetch_head = 0;\n \n+\tif (!git_config_get_string_tmp(\"fetch.bundleuri\", &bundle_uri) &&\n+\t !starts_with(bundle_uri, \"remote:\")) {\n+\t\tif (fetch_bundle_uri(the_repository, bundle_uri, NULL))\n+\t\t\twarning(_(\"failed to fetch bundles from '%s'\"), bundle_uri);\n+\t}\n+\n \tif (all) {\n \t\tif (argc == 1)\n \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 8ff560425ee..3f4d61a915c 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -465,6 +465,65 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \tcat >expect <<-\\EOF &&\n \trefs/bundles/base\n \tEOF\n+\ttest_cmp expect refs &&\n+\n+\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\t# Fetch the objects for bundle-2 _and_ bundle-3.\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace1.txt\" \\\n+\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n+\t\trefs/heads/left:refs/heads/left \\\n+\t\trefs/heads/right:refs/heads/right &&\n+\n+\t# This fetch should copy two files: the list and bundle-2.\n+\ttest_bundle_downloaded bundle-list trace1.txt &&\n+\ttest_bundle_downloaded bundle-2.bundle trace1.txt &&\n+\t! test_bundle_downloaded bundle-1.bundle trace1.txt &&\n+\n+\t# received left from bundle-2\n+\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect refs &&\n+\n+\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\tEOF\n+\n+\t# This fetch should skip bundle-3.bundle, since its objets are\n+\t# already local (we have the requisite commits for bundle-4.bundle).\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n+\t\trefs/heads/merge:refs/heads/merge &&\n+\n+\t# This fetch should copy three files: the list, bundle-3, and bundle-4.\n+\ttest_bundle_downloaded bundle-list trace2.txt &&\n+\ttest_bundle_downloaded bundle-4.bundle trace2.txt &&\n+\t! test_bundle_downloaded bundle-1.bundle trace2.txt &&\n+\t! test_bundle_downloaded bundle-2.bundle trace2.txt &&\n+\t! test_bundle_downloaded bundle-3.bundle trace2.txt &&\n+\n+\t# received merge ref from bundle-4, but right is missing\n+\t# because we did not download bundle-3.\n+\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/merge\n+\tEOF\n \ttest_cmp expect refs\n '\n \n-- \ngitgitgadget\n\n"},{"id":"469850","messageId":"57c0174d3752fb61a05e0653de9d3057616ed16a.1673037405.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.git.1673037405.gitgitgadget@gmail.com","subject":"[PATCH 4/8] bundle-uri: download in creationToken order","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-06T20:36:41Z","receivedAt":"2023-01-06T20:37:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe creationToken heuristic provides an ordering on the bundles\nadvertised by a bundle list. Teach the Git client to download bundles\ndifferently when this heuristic is advertised.\n\nThe bundles in the list are sorted by their advertised creationToken\nvalues, then downloaded in decreasing order. This avoids the previous\nstrategy of downloading bundles in an arbitrary order and attempting\nto apply them (likely failing in the case of required commits) until\ndiscovering the order through attempted unbundling.\n\nDuring a fresh 'git clone', it may make sense to download the bundles in\nincreasing order, since that would prevent the need to attempt\nunbundling a bundle with required commits that do not exist in our empty\nobject store. The cost of testing an unbundle is quite low, and instead\nthe chosen order is optimizing for a future bundle download during a\n'git fetch' operation with a non-empty object store.\n\nSince the Git client continues fetching from the Git remote after\ndownloading and unbundling bundles, the client's object store can be\nahead of the bundle provider's object store. The next time it attempts\nto download from the bundle list, it makes most sense to download only\nthe most-recent bundles until all tips successfully unbundle. The\nstrategy implemented here provides that short-circuit where the client\ndownloads a minimal set of bundles.\n\nA later implementation detail will store the maximum creationToken seen\nduring such a bundle download, and the client will avoid downloading a\nbundle unless its creationToken is strictly greater than that stored\nvalue. For now, if the client seeks to download from an identical\nbundle list since its previous download, it will download the\nmost-recent bundle then stop since its required commits are already in\nthe object store.\n\nAdd tests that exercise this behavior, but we will expand upon these\ntests when incremental downloads during 'git fetch' make use of\ncreationToken values.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 140 +++++++++++++++++++++++++++++++++++-\n t/t5558-clone-bundle-uri.sh | 41 ++++++++++-\n t/t5601-clone.sh | 50 +++++++++++++\n 3 files changed, 227 insertions(+), 4 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 63e2cc21057..b30c85ba6f2 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -434,6 +434,124 @@ static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data\n \treturn 0;\n }\n \n+struct sorted_bundle_list {\n+\tstruct remote_bundle_info **items;\n+\tsize_t alloc;\n+\tsize_t nr;\n+};\n+\n+static int insert_bundle(struct remote_bundle_info *bundle, void *data)\n+{\n+\tstruct sorted_bundle_list *list = data;\n+\tlist->items[list->nr++] = bundle;\n+\treturn 0;\n+}\n+\n+static int compare_creation_token(const void *va, const void *vb)\n+{\n+\tconst struct remote_bundle_info * const *a = va;\n+\tconst struct remote_bundle_info * const *b = vb;\n+\n+\tif ((*a)->creationToken > (*b)->creationToken)\n+\t\treturn -1;\n+\tif ((*a)->creationToken < (*b)->creationToken)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int fetch_bundles_by_token(struct repository *r,\n+\t\t\t\t struct bundle_list *list)\n+{\n+\tint cur;\n+\tint pop_or_push = 0;\n+\tstruct bundle_list_context ctx = {\n+\t\t.r = r,\n+\t\t.list = list,\n+\t\t.mode = list->mode,\n+\t};\n+\tstruct sorted_bundle_list sorted = {\n+\t\t.alloc = hashmap_get_size(&list->bundles),\n+\t};\n+\n+\tALLOC_ARRAY(sorted.items, sorted.alloc);\n+\n+\tfor_all_bundles_in_list(list, insert_bundle, &sorted);\n+\n+\tQSORT(sorted.items, sorted.nr, compare_creation_token);\n+\n+\t/*\n+\t * Use a stack-based approach to download the bundles and attempt\n+\t * to unbundle them in decreasing order by creation token. If we\n+\t * fail to unbundle (after a successful download) then move to the\n+\t * next non-downloaded bundle (push to the stack) and attempt\n+\t * downloading. Once we succeed in applying a bundle, move to the\n+\t * previous unapplied bundle (pop the stack) and attempt to unbundle\n+\t * it again.\n+\t *\n+\t * In the case of a fresh clone, we will likely download all of the\n+\t * bundles before successfully unbundling the oldest one, then the\n+\t * rest of the bundles unbundle successfully in increasing order\n+\t * of creationToken.\n+\t *\n+\t * If there are existing objects, then this process may terminate\n+\t * early when all required commits from \"new\" bundles exist in the\n+\t * repo's object store.\n+\t */\n+\tcur = 0;\n+\twhile (cur >= 0 && cur < sorted.nr) {\n+\t\tstruct remote_bundle_info *bundle = sorted.items[cur];\n+\t\tif (!bundle->file) {\n+\t\t\t/* Not downloaded yet. Try downloading. */\n+\t\t\tif (download_bundle_to_file(bundle, &ctx)) {\n+\t\t\t\t/* Failure. Push to the stack. */\n+\t\t\t\tpop_or_push = 1;\n+\t\t\t\tgoto stack_operation;\n+\t\t\t}\n+\n+\t\t\t/* We expect bundles when using creationTokens. */\n+\t\t\tif (!is_bundle(bundle->file, 1)) {\n+\t\t\t\twarning(_(\"file downloaded from '%s' is not a bundle\"),\n+\t\t\t\t\tbundle->uri);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (bundle->file && !bundle->unbundled) {\n+\t\t\t/*\n+\t\t\t * This was downloaded, but not successfully\n+\t\t\t * unbundled. Try unbundling again.\n+\t\t\t */\n+\t\t\tif (unbundle_from_file(ctx.r, bundle->file)) {\n+\t\t\t\t/* Failed to unbundle. Push to stack. */\n+\t\t\t\tpop_or_push = 1;\n+\t\t\t} else {\n+\t\t\t\t/* Succeeded in unbundle. Pop stack. */\n+\t\t\t\tpop_or_push = -1;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/*\n+\t\t * Else case: downloaded and unbundled successfully.\n+\t\t * Skip this by moving in the same direction as the\n+\t\t * previous step.\n+\t\t */\n+\n+stack_operation:\n+\t\t/* Move in the specified direction and repeat. */\n+\t\tcur += pop_or_push;\n+\t}\n+\n+\tfree(sorted.items);\n+\n+\t/*\n+\t * We succeed if the loop terminates because 'cur' drops below\n+\t * zero. The other case is that we terminate because 'cur'\n+\t * reaches the end of the list, so we have a failure no matter\n+\t * which bundles we apply from the list.\n+\t */\n+\treturn cur >= 0;\n+}\n+\n static int download_bundle_list(struct repository *r,\n \t\t\t\tstruct bundle_list *local_list,\n \t\t\t\tstruct bundle_list *global_list,\n@@ -471,7 +589,14 @@ static int fetch_bundle_list_in_config_format(struct repository *r,\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = download_bundle_list(r, &list_from_bundle,\n+\t/*\n+\t * If this list uses the creationToken heuristic, then the URIs\n+\t * it advertises are expected to be bundles, not nested lists.\n+\t * We can drop 'global_list' and 'depth'.\n+\t */\n+\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN)\n+\t\tresult = fetch_bundles_by_token(r, &list_from_bundle);\n+\telse if ((result = download_bundle_list(r, &list_from_bundle,\n \t\t\t\t\t global_list, depth)))\n \t\tgoto cleanup;\n \n@@ -613,6 +738,14 @@ int fetch_bundle_list(struct repository *r, struct bundle_list *list)\n \tint result;\n \tstruct bundle_list global_list;\n \n+\t/*\n+\t * If the creationToken heuristic is used, then the URIs\n+\t * advertised by 'list' are not nested lists and instead\n+\t * direct bundles. We do not need to use global_list.\n+\t */\n+\tif (list->heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN)\n+\t\treturn fetch_bundles_by_token(r, list);\n+\n \tinit_bundle_list(&global_list);\n \n \t/* If a bundle is added to this global list, then it is required. */\n@@ -621,7 +754,10 @@ int fetch_bundle_list(struct repository *r, struct bundle_list *list)\n \tif ((result = download_bundle_list(r, list, &global_list, 0)))\n \t\tgoto cleanup;\n \n-\tresult = unbundle_all_bundles(r, &global_list);\n+\tif (list->heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN)\n+\t\tresult = fetch_bundles_by_token(r, list);\n+\telse\n+\t\tresult = unbundle_all_bundles(r, &global_list);\n \n cleanup:\n \tfor_all_bundles_in_list(&global_list, unlink_bundle, NULL);\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 328caeeae9a..d7461ec907e 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -368,6 +368,8 @@ test_expect_success 'clone bundle list (HTTP, any mode)' '\n '\n \n test_expect_success 'clone bundle list (http, creationToken)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\n \tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n \tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n \t[bundle]\n@@ -392,10 +394,45 @@ test_expect_success 'clone bundle list (http, creationToken)' '\n \t\tcreationToken = 4\n \tEOF\n \n-\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" . clone-list-http-2 &&\n+\tGIT_TRACE2_EVENT=$(pwd)/trace-clone.txt \\\n+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\tclone-from clone-list-http-2 &&\n \n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n-\tgit -C clone-list-http-2 cat-file --batch-check \"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=$(pwd)/trace-clone.txt \\\n+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\tclone-from clone-token-http &&\n+\n+\ttest_bundle_downloaded bundle-1.bundle trace-clone.txt &&\n+\ttest_bundle_downloaded bundle-2.bundle trace-clone.txt\n '\n \n # Do not add tests here unless they use the HTTP server, as they will\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 1928ea1dd7c..57476b6e6d7 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -831,6 +831,56 @@ test_expect_success 'auto-discover multiple bundles from HTTP clone' '\n \tgrep -f pattern trace.txt\n '\n \n+# Usage: test_bundle_downloaded \n+test_bundle_downloaded () {\n+\tcat >pattern <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/$1.bundle\"\\]\n+\tEOF\n+\tgrep -f pattern \"$2\"\n+}\n+\n+test_expect_success 'auto-discover multiple bundles from HTTP clone: creationToken heuristic' '\n+\ttest_when_finished rm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/repo4.git\" &&\n+\ttest_when_finished rm -rf clone-heuristic trace*.txt &&\n+\n+\ttest_commit -C src newest &&\n+\tgit -C src bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/newest.bundle\" HEAD~1..HEAD &&\n+\tgit clone --bare --no-local src \"$HTTPD_DOCUMENT_ROOT_PATH/repo4.git\" &&\n+\n+\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/repo4.git/config\" <<-EOF &&\n+\t[uploadPack]\n+\t\tadvertiseBundleURIs = true\n+\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"everything\"]\n+\t\turi = $HTTPD_URL/everything.bundle\n+\t\tcreationtoken = 1\n+\n+\t[bundle \"new\"]\n+\t\turi = $HTTPD_URL/new.bundle\n+\t\tcreationtoken = 2\n+\n+\t[bundle \"newest\"]\n+\t\turi = $HTTPD_URL/newest.bundle\n+\t\tcreationtoken = 3\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" \\\n+\t\tgit -c protocol.version=2 \\\n+\t\t -c transfer.bundleURI=true clone \\\n+\t\t\"$HTTPD_URL/smart/repo4.git\" clone-heuristic &&\n+\n+\t# We should fetch all bundles\n+\tfor b in everything new newest\n+\tdo\n+\t\ttest_bundle_downloaded $b trace-clone.txt || return 1\n+\tdone\n+'\n+\n # DO NOT add non-httpd-specific tests here, because the last part of this\n # test script is only executed when httpd is available and enabled.\n \n-- \ngitgitgadget\n\n"},{"id":"469851","messageId":"51f210ddeb46fb06e885dc384a486c4bb16ad8cd.1673037405.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.git.1673037405.gitgitgadget@gmail.com","subject":"[PATCH 8/8] bundle-uri: store fetch.bundleCreationToken","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-06T20:36:45Z","receivedAt":"2023-01-06T20:37:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a bundle list specifies the \"creationToken\" heuristic, the Git\nclient downloads the list and then starts downloading bundles in\ndescending creationToken order. This process stops as soon as all\ndownloaded bundles can be applied to the repository (because all\nrequired commits are present in the repository or in the downloaded\nbundles).\n\nWhen checking the same bundle list twice, this strategy requires\ndownloading the bundle with the maximum creationToken again, which is\nwasteful. The creationToken heuristic promises that the client will not\nhave a use for that bundle if its creationToken value is the at most the\nprevious creationToken value.\n\nTo prevent these wasteful downloads, create a fetch.bundleCreationToken\nconfig setting that the Git client sets after downloading bundles. This\nvalue allows skipping that maximum bundle download when this config\nvalue is the same value (or larger).\n\nTo test that this works correctly, we can insert some \"duplicate\"\nfetches into existing tests and demonstrate that only the bundle list is\ndownloaded.\n\nThe previous logic for downloading bundles by creationToken worked even\nif the bundle list was empty, but now we have logic that depends on the\nfirst entry of the list. Terminate early in the (non-sensical) case of\nan empty bundle list.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/fetch.txt | 8 ++++++++\n bundle-uri.c | 35 ++++++++++++++++++++++++++++++++--\n t/t5558-clone-bundle-uri.sh | 25 +++++++++++++++++++++++-\n 3 files changed, 65 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt\nindex 4f796218aab..96755ba148b 100644\n--- a/Documentation/config/fetch.txt\n+++ b/Documentation/config/fetch.txt\n@@ -104,3 +104,11 @@ fetch.bundleURI::\n \tfirst running `git fetch --bundle-uri=` immediately before\n \t`git fetch `. See details of the `--bundle-uri` option in\n \tlinkgit:git-fetch[1].\n+\n+fetch.bundleCreationToken::\n+\tWhen using `fetch.bundleURI` to fetch incrementally from a bundle\n+\tlist that uses the \"creationToken\" heuristic, this config value\n+\tstores the maximum `creationToken` value of the downloaded bundles.\n+\tThis value is used to prevent downloading bundles in the future\n+\tif the advertised `creationToken` is not strictly larger than this\n+\tvalue.\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 1dbbbb980eb..98655bd6721 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -464,6 +464,8 @@ static int fetch_bundles_by_token(struct repository *r,\n {\n \tint cur;\n \tint pop_or_push = 0;\n+\tconst char *creationTokenStr;\n+\tuint64_t maxCreationToken;\n \tstruct bundle_list_context ctx = {\n \t\t.r = r,\n \t\t.list = list,\n@@ -477,8 +479,27 @@ static int fetch_bundles_by_token(struct repository *r,\n \n \tfor_all_bundles_in_list(list, insert_bundle, &sorted);\n \n+\tif (!sorted.nr) {\n+\t\tfree(sorted.items);\n+\t\treturn 0;\n+\t}\n+\n \tQSORT(sorted.items, sorted.nr, compare_creation_token);\n \n+\t/*\n+\t * If fetch.bundleCreationToken exists, parses to a uint64t, and\n+\t * is not strictly smaller than the maximum creation token in the\n+\t * bundle list, then do not download any bundles.\n+\t */\n+\tif (!repo_config_get_value(r,\n+\t\t\t\t \"fetch.bundlecreationtoken\",\n+\t\t\t\t &creationTokenStr) &&\n+\t sscanf(creationTokenStr, \"%\"PRIu64, &maxCreationToken) == 1 &&\n+\t sorted.items[0]->creationToken <= maxCreationToken) {\n+\t\tfree(sorted.items);\n+\t\treturn 0;\n+\t}\n+\n \t/*\n \t * Use a stack-based approach to download the bundles and attempt\n \t * to unbundle them in decreasing order by creation token. If we\n@@ -541,14 +562,24 @@ stack_operation:\n \t\tcur += pop_or_push;\n \t}\n \n-\tfree(sorted.items);\n-\n \t/*\n \t * We succeed if the loop terminates because 'cur' drops below\n \t * zero. The other case is that we terminate because 'cur'\n \t * reaches the end of the list, so we have a failure no matter\n \t * which bundles we apply from the list.\n \t */\n+\tif (cur < 0) {\n+\t\tstruct strbuf value = STRBUF_INIT;\n+\t\tstrbuf_addf(&value, \"%\"PRIu64\"\", sorted.items[0]->creationToken);\n+\t\tif (repo_config_set_multivar_gently(ctx.r,\n+\t\t\t\t\t\t \"fetch.bundleCreationToken\",\n+\t\t\t\t\t\t value.buf, NULL, 0))\n+\t\t\twarning(_(\"failed to store maximum creation token\"));\n+\n+\t\tstrbuf_release(&value);\n+\t}\n+\n+\tfree(sorted.items);\n \treturn cur >= 0;\n }\n \ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 3f4d61a915c..0604d721f1b 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -455,6 +455,7 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \t\t\"$HTTPD_URL/smart/fetch.git\" fetch-http-4 &&\n \n \ttest_cmp_config -C fetch-http-4 \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n+\ttest_cmp_config -C fetch-http-4 1 fetch.bundlecreationtoken &&\n \n \t# The clone should copy two files: the list and bundle-1.\n \ttest_bundle_downloaded bundle-list trace-clone.txt &&\n@@ -479,6 +480,8 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \t\trefs/heads/left:refs/heads/left \\\n \t\trefs/heads/right:refs/heads/right &&\n \n+\ttest_cmp_config -C fetch-http-4 2 fetch.bundlecreationtoken &&\n+\n \t# This fetch should copy two files: the list and bundle-2.\n \ttest_bundle_downloaded bundle-list trace1.txt &&\n \ttest_bundle_downloaded bundle-2.bundle trace1.txt &&\n@@ -492,6 +495,15 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \tEOF\n \ttest_cmp expect refs &&\n \n+\t# No-op fetch\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace1b.txt\" \\\n+\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n+\t\trefs/heads/left:refs/heads/left \\\n+\t\trefs/heads/right:refs/heads/right &&\n+\ttest_bundle_downloaded bundle-list trace1b.txt &&\n+\t! test_bundle_downloaded bundle-1.bundle trace1b.txt &&\n+\t! test_bundle_downloaded bundle-2.bundle trace1b.txt &&\n+\n \tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n \t[bundle \"bundle-3\"]\n \t\turi = bundle-3.bundle\n@@ -508,6 +520,8 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \t\tgit -C fetch-http-4 fetch origin --no-tags \\\n \t\trefs/heads/merge:refs/heads/merge &&\n \n+\ttest_cmp_config -C fetch-http-4 4 fetch.bundlecreationtoken &&\n+\n \t# This fetch should copy three files: the list, bundle-3, and bundle-4.\n \ttest_bundle_downloaded bundle-list trace2.txt &&\n \ttest_bundle_downloaded bundle-4.bundle trace2.txt &&\n@@ -524,7 +538,16 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \trefs/bundles/left\n \trefs/bundles/merge\n \tEOF\n-\ttest_cmp expect refs\n+\ttest_cmp expect refs &&\n+\n+\t# No-op fetch\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2b.txt\" \\\n+\t\tgit -C fetch-http-4 fetch origin &&\n+\ttest_bundle_downloaded bundle-list trace2b.txt &&\n+\t! test_bundle_downloaded bundle-1.bundle trace2b.txt &&\n+\t! test_bundle_downloaded bundle-2.bundle trace2b.txt &&\n+\t! test_bundle_downloaded bundle-3.bundle trace2b.txt &&\n+\t! test_bundle_downloaded bundle-4.bundle trace2b.txt\n '\n \n # Do not add tests here unless they use the HTTP server, as they will\n-- \ngitgitgadget\n"},{"id":"469852","messageId":"a1808f0b10cfb519613bc292e30b884962a83275.1673037405.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.git.1673037405.gitgitgadget@gmail.com","subject":"[PATCH 3/8] bundle-uri: parse bundle..creationToken values","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-06T20:36:40Z","receivedAt":"2023-01-06T20:37:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change taught Git to parse the bundle.heuristic value,\nespecially when its value is \"creationToken\". Now, teach Git to parse\nthe bundle..creationToken values on each bundle in a bundle list.\n\nBefore implementing any logic based on creationToken values for the\ncreationToken heuristic, parse and print these values for testing\npurposes.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 10 ++++++++++\n bundle-uri.h | 6 ++++++\n t/t5750-bundle-uri-parse.sh | 18 ++++++++++++++++++\n 3 files changed, 34 insertions(+)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 56c94595c2a..63e2cc21057 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -80,6 +80,9 @@ static int summarize_bundle(struct remote_bundle_info *info, void *data)\n \tFILE *fp = data;\n \tfprintf(fp, \"[bundle \\\"%s\\\"]\\n\", info->id);\n \tfprintf(fp, \"\\turi = %s\\n\", info->uri);\n+\n+\tif (info->creationToken)\n+\t\tfprintf(fp, \"\\tcreationToken = %\"PRIu64\"\\n\", info->creationToken);\n \treturn 0;\n }\n \n@@ -190,6 +193,13 @@ static int bundle_list_update(const char *key, const char *value,\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(subkey, \"creationtoken\")) {\n+\t\tif (sscanf(value, \"%\"PRIu64, &bundle->creationToken) != 1)\n+\t\t\twarning(_(\"could not parse bundle list key %s with value '%s'\"),\n+\t\t\t\t\"creationToken\", value);\n+\t\treturn 0;\n+\t}\n+\n \t/*\n \t * At this point, we ignore any information that we don't\n \t * understand, assuming it to be hints for a heuristic the client\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex ad82174112d..1cae418211b 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -42,6 +42,12 @@ struct remote_bundle_info {\n \t * this boolean is true.\n \t */\n \tunsigned unbundled:1;\n+\n+\t/**\n+\t * If the bundle is part of a list with the creationToken\n+\t * heuristic, then we use this member for sorting the bundles.\n+\t */\n+\tuint64_t creationToken;\n };\n \n #define REMOTE_BUNDLE_INFO_INIT { 0 }\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex 6fc92a9c0d4..81bdf58b944 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -258,10 +258,13 @@ test_expect_success 'parse config format: creationToken heuristic' '\n \t\theuristic = creationToken\n \t[bundle \"one\"]\n \t\turi = http://example.com/bundle.bdl\n+\t\tcreationToken = 123456\n \t[bundle \"two\"]\n \t\turi = https://example.com/bundle.bdl\n+\t\tcreationToken = 12345678901234567890\n \t[bundle \"three\"]\n \t\turi = file:///usr/share/git/bundle.bdl\n+\t\tcreationToken = 1\n \tEOF\n \n \ttest-tool bundle-uri parse-config expect >actual 2>err &&\n@@ -269,4 +272,19 @@ test_expect_success 'parse config format: creationToken heuristic' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format edge cases: creationToken heuristic' '\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t\tcreationToken = bogus\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config expect >actual 2>err &&\n+\tgrep \"could not parse bundle list key creationToken with value '\\''bogus'\\''\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"469930","messageId":"xmqqilhgxwj4.fsf@gitster.g","threadId":"59040","inReplyTo":"9007249b9488c23f00c2d498ffd520e4af8b37a4.1673037405.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/8] bundle-uri: parse bundle.heuristic=creationToken","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-09T02:38:07Z","receivedAt":"2023-01-09T02:38:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +static const char *heuristics[] = {\n> +\t[BUNDLE_HEURISTIC_NONE] = \"\",\n> +\t[BUNDLE_HEURISTIC_CREATIONTOKEN] = \"creationToken\",\n> +};\n\nIdeally it would require the least amount of maintenance if we could\ndefine BUNDLE_HEURISTIC__COUNT as ARRAY_SIZE() of this thing, but it\nbeing a file scope static, it might not be easy to arrange that. As\na lessor altenative, would it make it safer to size this array more\nexplicitly using BUNDLE_HEURISTIC__COUNT macro?\n\n\tstatic const char *heuristics[BUNDLE_HEURISTIC__COUNT] = {\n\t\t...\n\t};\n\nor is it more-or-less moot point to aim for safety because nobody\nenforces that these [indices] used to define the contents of this\narray are dense?\n\nThat is ...\n\n> @@ -142,6 +150,19 @@ static int bundle_list_update(const char *key, const char *value,\n> \t\t\treturn 0;\n> \t\t}\n> \n> +\t\tif (!strcmp(subkey, \"heuristic\")) {\n> +\t\t\tint i;\n> +\t\t\tfor (i = 0; i < BUNDLE_HEURISTIC__COUNT; i++) {\n> +\t\t\t\tif (!strcmp(value, heuristics[i])) {\n> +\t\t\t\t\tlist->heuristic = i;\n> +\t\t\t\t\treturn 0;\n> +\t\t\t\t}\n> +\t\t\t}\n\n... this strcmp() will segfault if heuristics[] array is sparse, or\nBUNDLE_HEURISTIC__COUNT is larger than the array (i.e. you add a new\nheuristic in \"enum bundle_heuristic\" before the __COUNT sentinel,\nbut forget to add it to the heuristics[] array).\n\n\"You are worrying too much. Our developers would notice a segfault\nand the current code, which may look risky to you, is something they\ncan live with\", is a perfectly acceptable response, but somehow I\nhave this nagging feeling that we should be able to make it easier\nto maintain without incurring extra runtime cost.\n\n> diff --git a/bundle-uri.h b/bundle-uri.h\n> index d5e89f1671c..ad82174112d 100644\n> --- a/bundle-uri.h\n> +++ b/bundle-uri.h\n> @@ -52,6 +52,14 @@ enum bundle_list_mode {\n> \tBUNDLE_MODE_ANY\n> };\n> \n> +enum bundle_list_heuristic {\n> +\tBUNDLE_HEURISTIC_NONE = 0,\n> +\tBUNDLE_HEURISTIC_CREATIONTOKEN,\n> +\n> +\t/* Must be last. */\n> +\tBUNDLE_HEURISTIC__COUNT,\n> +};\n\nThe only reason to leave a trailing comma is to make it easy to\nappend new values at the end. By omitting the trailing comma, you\ncan doubly signal \"Must be last\" here (not suggesting to remove the\ncomment; suggesting to remove the trailing comma).\n\nThanks.\n\n"},{"id":"469932","messageId":"xmqq8ricxv52.fsf@gitster.g","threadId":"59040","inReplyTo":"a1808f0b10cfb519613bc292e30b884962a83275.1673037405.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/8] bundle-uri: parse bundle..creationToken values","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-09T03:08:09Z","receivedAt":"2023-01-09T03:08:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +\tif (!strcmp(subkey, \"creationtoken\")) {\n> +\t\tif (sscanf(value, \"%\"PRIu64, &bundle->creationToken) != 1)\n> +\t\t\twarning(_(\"could not parse bundle list key %s with value '%s'\"),\n> +\t\t\t\t\"creationToken\", value);\n> +\t\treturn 0;\n> +\t}\n\nWe tend to avoid sscanf() to parse out integral values, as it is a\nbit too permissive to our liking (especially while parsing the\nobject header), but here it probably is OK, I guess.\n\n> +\t/**\n> +\t * If the bundle is part of a list with the creationToken\n> +\t * heuristic, then we use this member for sorting the bundles.\n> +\t */\n> +\tuint64_t creationToken;\n> };\n\nIs the idea behind the type is that creationTokens, while we leave\nup to the bundle providers what the actual values (other than zero)\nmean, must be comparable to give them a total order, and uint64\nwould be a usable type for bundle providers to come up with such\nvalues easily?\n\nThanks.\n"},{"id":"469933","messageId":"xmqqtu10wfwt.fsf@gitster.g","threadId":"59040","inReplyTo":"57c0174d3752fb61a05e0653de9d3057616ed16a.1673037405.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/8] bundle-uri: download in creationToken order","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-09T03:22:26Z","receivedAt":"2023-01-09T03:22:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +struct sorted_bundle_list {\n> +\tstruct remote_bundle_info **items;\n> +\tsize_t alloc;\n> +\tsize_t nr;\n> +};\n> +\n> +static int insert_bundle(struct remote_bundle_info *bundle, void *data)\n> +{\n> +\tstruct sorted_bundle_list *list = data;\n> +\tlist->items[list->nr++] = bundle;\n> +\treturn 0;\n> +}\n\nEspecially given that the type of the list claims it to be \"sorted\",\ninsert_bundle() is a misleading name for a helper that merely\nappends to it to make the list (tentatively) unsorted.\n\nI am not opposed to \"append all to make an unsorted list, then sort\nthe list at the end\" strategy.\n\n> +static int compare_creation_token(const void *va, const void *vb)\n> +{\n> +\tconst struct remote_bundle_info * const *a = va;\n> +\tconst struct remote_bundle_info * const *b = vb;\n> +\n> +\tif ((*a)->creationToken > (*b)->creationToken)\n> +\t\treturn -1;\n> +\tif ((*a)->creationToken < (*b)->creationToken)\n> +\t\treturn 1;\n> +\treturn 0;\n> +}\n\nUsually compare(a,b) returns the sign of (a-b), but the returned\nvalue from the above is the opposite. This is because we want the\nlist sorted from newer to older? It may help developers to name\nsuch a (reverse) \"compare\" function differently.\n\n> +static int fetch_bundles_by_token(struct repository *r,\n> +\t\t\t\t struct bundle_list *list)\n> +{\n> +\tint cur;\n> +\tint pop_or_push = 0;\n> +\tstruct bundle_list_context ctx = {\n> +\t\t.r = r,\n> +\t\t.list = list,\n> +\t\t.mode = list->mode,\n> +\t};\n> +\tstruct sorted_bundle_list sorted = {\n> +\t\t.alloc = hashmap_get_size(&list->bundles),\n> +\t};\n> +\n> +\tALLOC_ARRAY(sorted.items, sorted.alloc);\n> +\n> +\tfor_all_bundles_in_list(list, insert_bundle, &sorted);\n> +\n> +\tQSORT(sorted.items, sorted.nr, compare_creation_token);\n\nIf I were doing this patch, I would call the type of the list of\nbundles \"struct bundle_list\" (without \"sorted_\" in its name), and\nname the variable of that type used here \"sorted\". That would make\nit more clear that this particular bundle list starts its life as\nunsorted (with \"append_bundle\" function adding more elements) and\nthen gets sorted in the end, from the above several lines.\n\n"},{"id":"469976","messageId":"35bcfd1f-ff54-fc2a-0477-6454bca03a87@github.com","threadId":"59040","inReplyTo":"xmqqilhgxwj4.fsf@gitster.g","subject":"Re: [PATCH 2/8] bundle-uri: parse bundle.heuristic=creationToken","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-09T14:20:09Z","receivedAt":"2023-01-09T14:20:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/8/2023 9:38 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> +static const char *heuristics[] = {\n>> +\t[BUNDLE_HEURISTIC_NONE] = \"\",\n>> +\t[BUNDLE_HEURISTIC_CREATIONTOKEN] = \"creationToken\",\n>> +};\n> \n> Ideally it would require the least amount of maintenance if we could\n> define BUNDLE_HEURISTIC__COUNT as ARRAY_SIZE() of this thing, but it\n> being a file scope static, it might not be easy to arrange that. As\n> a lessor altenative, would it make it safer to size this array more\n> explicitly using BUNDLE_HEURISTIC__COUNT macro?\n> \n> \tstatic const char *heuristics[BUNDLE_HEURISTIC__COUNT] = {\n> \t\t...\n> \t};\n\nYes, I should have used this size indicator.\n \n> or is it more-or-less moot point to aim for safety because nobody\n> enforces that these [indices] used to define the contents of this\n> array are dense?\n> \n> That is ...\n> \n>> @@ -142,6 +150,19 @@ static int bundle_list_update(const char *key, const char *value,\n>> \t\t\treturn 0;\n>> \t\t}\n>> \n>> +\t\tif (!strcmp(subkey, \"heuristic\")) {\n>> +\t\t\tint i;\n>> +\t\t\tfor (i = 0; i < BUNDLE_HEURISTIC__COUNT; i++) {\n>> +\t\t\t\tif (!strcmp(value, heuristics[i])) {\n>> +\t\t\t\t\tlist->heuristic = i;\n>> +\t\t\t\t\treturn 0;\n>> +\t\t\t\t}\n>> +\t\t\t}\n> \n> ... this strcmp() will segfault if heuristics[] array is sparse, or\n> BUNDLE_HEURISTIC__COUNT is larger than the array (i.e. you add a new\n> heuristic in \"enum bundle_heuristic\" before the __COUNT sentinel,\n> but forget to add it to the heuristics[] array).\n> \n> \"You are worrying too much. Our developers would notice a segfault\n> and the current code, which may look risky to you, is something they\n> can live with\", is a perfectly acceptable response, but somehow I\n> have this nagging feeling that we should be able to make it easier\n> to maintain without incurring extra runtime cost.\n\nYou're right. I was following an established pattern of linking\nenums to values, but I'm not sure that those other examples will\nloop over the array like this looking for a value.\n\nA safer approach would be to have an array of (enum, string) pairs\nthat could either be iterated in a loop (fast enough for a small\nnumber of enum values, such as this case) or used to populate a\nhashmap at runtime if needed for a large number of queries.\n\n>> diff --git a/bundle-uri.h b/bundle-uri.h\n>> index d5e89f1671c..ad82174112d 100644\n>> --- a/bundle-uri.h\n>> +++ b/bundle-uri.h\n>> @@ -52,6 +52,14 @@ enum bundle_list_mode {\n>> \tBUNDLE_MODE_ANY\n>> };\n>> \n>> +enum bundle_list_heuristic {\n>> +\tBUNDLE_HEURISTIC_NONE = 0,\n>> +\tBUNDLE_HEURISTIC_CREATIONTOKEN,\n>> +\n>> +\t/* Must be last. */\n>> +\tBUNDLE_HEURISTIC__COUNT,\n>> +};\n> \n> The only reason to leave a trailing comma is to make it easy to\n> append new values at the end. By omitting the trailing comma, you\n> can doubly signal \"Must be last\" here (not suggesting to remove the\n> comment; suggesting to remove the trailing comma).\n\nThis is a great example of \"doing the typically right thing\" but\nwithout thinking of _why_ we do that thing. Thanks for pointing this\nout.\n\nThanks,\n-Stolee\n"},{"id":"469977","messageId":"3c0db749-5040-24eb-1b26-969e7c768906@github.com","threadId":"59040","inReplyTo":"xmqq8ricxv52.fsf@gitster.g","subject":"Re: [PATCH 3/8] bundle-uri: parse bundle..creationToken values","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-09T14:41:59Z","receivedAt":"2023-01-09T14:42:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/8/2023 10:08 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> +\tif (!strcmp(subkey, \"creationtoken\")) {\n>> +\t\tif (sscanf(value, \"%\"PRIu64, &bundle->creationToken) != 1)\n>> +\t\t\twarning(_(\"could not parse bundle list key %s with value '%s'\"),\n>> +\t\t\t\t\"creationToken\", value);\n>> +\t\treturn 0;\n>> +\t}\n> \n> We tend to avoid sscanf() to parse out integral values, as it is a\n> bit too permissive to our liking (especially while parsing the\n> object header), but here it probably is OK, I guess.\n\nI tried to find another way to parse uint64ts, but could not find\nanother example in the codebase. I'd be happy to change it if we\nhave a preferred method.\n \n>> +\t/**\n>> +\t * If the bundle is part of a list with the creationToken\n>> +\t * heuristic, then we use this member for sorting the bundles.\n>> +\t */\n>> +\tuint64_t creationToken;\n>> };\n> \n> Is the idea behind the type is that creationTokens, while we leave\n> up to the bundle providers what the actual values (other than zero)\n> mean, must be comparable to give them a total order, and uint64\n> would be a usable type for bundle providers to come up with such\n> values easily?\n\nOne easy way to create the total order is to use Unix epoch\ntimestamps (on the same machine, to avoid clock skew). We cannot\nuse the machine-local width of the timestamp types, though. And\nthere is no need to use timestamp-like values. The bundle provider\ncould keep an integer count.\n\nI did add a test that ensures we test a creationToken that would\nnot fit within 32 bits.\n\nThanks,\n-Stolee\n"},{"id":"469979","messageId":"9ac62cc6-db13-9a39-b3fd-a76311bdf92d@github.com","threadId":"59040","inReplyTo":"xmqqtu10wfwt.fsf@gitster.g","subject":"Re: [PATCH 4/8] bundle-uri: download in creationToken order","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-09T14:58:54Z","receivedAt":"2023-01-09T14:59:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/8/2023 10:22 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> +struct sorted_bundle_list {\n>> +\tstruct remote_bundle_info **items;\n>> +\tsize_t alloc;\n>> +\tsize_t nr;\n>> +};\n>> +\n>> +static int insert_bundle(struct remote_bundle_info *bundle, void *data)\n>> +{\n>> +\tstruct sorted_bundle_list *list = data;\n>> +\tlist->items[list->nr++] = bundle;\n>> +\treturn 0;\n>> +}\n> \n> Especially given that the type of the list claims it to be \"sorted\",\n> insert_bundle() is a misleading name for a helper that merely\n> appends to it to make the list (tentatively) unsorted.\n> \n> I am not opposed to \"append all to make an unsorted list, then sort\n> the list at the end\" strategy.\n\n...\n\n> If I were doing this patch, I would call the type of the list of\n> bundles \"struct bundle_list\" (without \"sorted_\" in its name), and\n> name the variable of that type used here \"sorted\". That would make\n> it more clear that this particular bundle list starts its life as\n> unsorted (with \"append_bundle\" function adding more elements) and\n> then gets sorted in the end, from the above several lines.\n\nSince \"struct bundle_list\" is taken, how about \"bundles_for_sorting\"\nsince that's the purpose of this struct (to be passed as data to\nthe for_all_bundles_in_list() and then to QSORT()).\n\nRenaming insert_bundle() to append_bundle() is clearly better.\n\n>> +static int compare_creation_token(const void *va, const void *vb)\n>> +{\n>> +\tconst struct remote_bundle_info * const *a = va;\n>> +\tconst struct remote_bundle_info * const *b = vb;\n>> +\n>> +\tif ((*a)->creationToken > (*b)->creationToken)\n>> +\t\treturn -1;\n>> +\tif ((*a)->creationToken < (*b)->creationToken)\n>> +\t\treturn 1;\n>> +\treturn 0;\n>> +}\n> \n> Usually compare(a,b) returns the sign of (a-b), but the returned\n> value from the above is the opposite. This is because we want the\n> list sorted from newer to older? It may help developers to name\n> such a (reverse) \"compare\" function differently.\n\nWould renaming this to \"compare_creation_token_decreasing\" be clear\nenough? (Plus a doc comment.)\n\nThanks,\n-Stolee\n"},{"id":"470525","messageId":"4d13cea1-e9f6-381f-14ef-8a5a645e2a8f@github.com","threadId":"59040","inReplyTo":"39eed9148782c37f5184c5fff7d0e4d1a7a2a1fe.1673037405.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/8] t5558: add tests for creationToken heuristic","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-17T18:17:42Z","receivedAt":"2023-01-17T19:03:27Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> As documented in the bundle URI design doc in 2da14fad8fe (docs:\n> document bundle URI standard, 2022-08-09), the 'creationToken' member of\n> a bundle URI allows a bundle provider to specify a total order on the\n> bundles.\n> \n> Future changes will allow the Git client to understand these members and\n> modify its behavior around downloading the bundles in that order. In the\n> meantime, create tests that add creation tokens to the bundle list. For\n> now, the Git client correctly ignores these unknown keys.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> t/t5558-clone-bundle-uri.sh | 52 +++++++++++++++++++++++++++++++++++--\n> 1 file changed, 50 insertions(+), 2 deletions(-)\n> \n> diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\n> index 9155f31fa2c..328caeeae9a 100755\n> --- a/t/t5558-clone-bundle-uri.sh\n> +++ b/t/t5558-clone-bundle-uri.sh\n> @@ -284,7 +284,17 @@ test_expect_success 'clone HTTP bundle' '\n> \ttest_config -C clone-http log.excludedecoration refs/bundle/\n> '\n> \n> +# usage: test_bundle_downloaded \n> +test_bundle_downloaded () {\n> +\tcat >pattern <<-EOF &&\n> +\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/$1\"\\]\n> +\tEOF\n> +\tgrep -f pattern \"$2\"\n> +}\n> +\n> test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n> +\ttest_when_finished rm -f trace*.txt &&\n> +\n> \tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n> \tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n> \t[bundle]\n> @@ -304,12 +314,19 @@ test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n> \t\turi = $HTTPD_URL/bundle-4.bundle\n> \tEOF\n> \n> -\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" \\\n> +\t\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n> \t\tclone-from clone-list-http 2>err &&\n> \t! grep \"Repository lacks these prerequisite commits\" err &&\n> \n> \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n> -\tgit -C clone-list-http cat-file --batch-check +\tgit -C clone-list-http cat-file --batch-check +\n> +\tfor b in 1 2 3 4\n> +\tdo\n> +\t\ttest_bundle_downloaded bundle-$b.bundle trace-clone.txt ||\n> +\t\t\treturn 1\n> +\tdone\n\nBecause the current state of bundle list handling is equivalent to \"no\nheuristic\", this pre-existing test is just updated to verify all bundles are\ndownloaded. This isn't new behavior, but it'll be relevant to compare with\nthe behavior of the 'creationToken' heuristic. \n\nI was going to ask how the tests verify that *only* the expected bundles are\ndownloaded, and it looks like later patches [1] handle that with\n'! test_bundle_downloaded' checks. That approach seems a bit fragile (if a\nbundle's name doesn't match the '! test_bundle_downloaded' check for some\nreason, the bundle can be either downloaded or not with no effect on the\ntest result). Would something like a 'test_downloaded_bundle_count' work\ninstead?\n\n-------- 8< --------\n\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 0604d721f1..b2f55dd983 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -292,6 +292,16 @@ test_bundle_downloaded () {\n \tgrep -f pattern \"$2\"\n }\n \n+test_download_bundle_count () {\n+\tcat >exclude <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/bundle-list\"\\]\n+\tEOF\n+\tcat >pattern <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/.*\"\\]\n+\tEOF\n+\ttest $(grep -f pattern \"$2\" | grep -v -f exclude | wc -l) -eq \"$1\"\n+}\n+\n test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n \ttest_when_finished rm -f trace*.txt &&\n \n@@ -322,6 +332,7 @@ test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n \tgit -C clone-list-http cat-file --batch-check 8 --------\n\n[1] https://lore.kernel.org/git/51f210ddeb46fb06e885dc384a486c4bb16ad8cd.1673037405.git.gitgitgadget@gmail.com/\n\n> '\n> \n> test_expect_success 'clone bundle list (HTTP, any mode)' '\n> @@ -350,6 +367,37 @@ test_expect_success 'clone bundle list (HTTP, any mode)' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'clone bundle list (http, creationToken)' '\n> +\tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n> +\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n> +\t[bundle]\n> +\t\tversion = 1\n> +\t\tmode = all\n> +\t\theuristic = creationToken\n> +\n> +\t[bundle \"bundle-1\"]\n> +\t\turi = bundle-1.bundle\n> +\t\tcreationToken = 1\n> +\n> +\t[bundle \"bundle-2\"]\n> +\t\turi = bundle-2.bundle\n> +\t\tcreationToken = 2\n> +\n> +\t[bundle \"bundle-3\"]\n> +\t\turi = bundle-3.bundle\n> +\t\tcreationToken = 3\n> +\n> +\t[bundle \"bundle-4\"]\n> +\t\turi = bundle-4.bundle\n> +\t\tcreationToken = 4\n> +\tEOF\n> +\n> +\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" . clone-list-http-2 &&\n> +\n> +\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n> +\tgit -C clone-list-http-2 cat-file --batch-check +'\n> +\n> # Do not add tests here unless they use the HTTP server, as they will\n> # not run unless the HTTP dependencies exist.\n> \n\n"},{"id":"470527","messageId":"5fb1385a-3821-8e1f-b31c-2c38f87dc965@github.com","threadId":"59040","inReplyTo":"9007249b9488c23f00c2d498ffd520e4af8b37a4.1673037405.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/8] bundle-uri: parse bundle.heuristic=creationToken","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-17T19:13:17Z","receivedAt":"2023-01-17T20:24:55Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The bundle.heuristic value communicates that the bundle list is\n> organized to make use of the bundle..creationToken values that may\n> be provided in the bundle list. Those values will create a total order\n> on the bundles, allowing the Git client to download them in a specific\n> order and even remember previously-downloaded bundles by storing the\n> maximum creation token value.\n> \n> Before implementing any logic that parses or uses the\n> bundle..creationToken values, teach Git to parse the\n> bundle.heuristic value from a bundle list. We can use 'test-tool\n> bundle-uri' to print the heuristic value and verify that the parsing\n> works correctly.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/config/bundle.txt | 7 +++++++\n> bundle-uri.c | 21 +++++++++++++++++++++\n> bundle-uri.h | 14 ++++++++++++++\n> t/t5750-bundle-uri-parse.sh | 19 +++++++++++++++++++\n> 4 files changed, 61 insertions(+)\n> \n> diff --git a/Documentation/config/bundle.txt b/Documentation/config/bundle.txt\n> index daa21eb674a..3faae386853 100644\n> --- a/Documentation/config/bundle.txt\n> +++ b/Documentation/config/bundle.txt\n> @@ -15,6 +15,13 @@ bundle.mode::\n> \tcomplete understanding of the bundled information (`all`) or if any one\n> \tof the listed bundle URIs is sufficient (`any`).\n> \n> +bundle.heuristic::\n> +\tIf this string-valued key exists, then the bundle list is designed to\n> +\twork well with incremental `git fetch` commands. The heuristic signals\n> +\tthat there are additional keys available for each bundle that help\n> +\tdetermine which subset of bundles the client should download. The\n> +\tonly value currently understood is `creationToken`.\n\nThis description clearly describes the 'heuristic' key and what it does.\n\n> +\n> bundle..*::\n> \tThe `bundle..*` keys are used to describe a single item in the\n> \tbundle list, grouped under `` for identification purposes.\n> diff --git a/bundle-uri.c b/bundle-uri.c\n> index 36268dda172..56c94595c2a 100644\n> --- a/bundle-uri.c\n> +++ b/bundle-uri.c\n> @@ -9,6 +9,11 @@\n> #include \"config.h\"\n> #include \"remote.h\"\n> \n> +static const char *heuristics[] = {\n> +\t[BUNDLE_HEURISTIC_NONE] = \"\",\n> +\t[BUNDLE_HEURISTIC_CREATIONTOKEN] = \"creationToken\",\n> +};\n> +\n> static int compare_bundles(const void *hashmap_cmp_fn_data,\n> \t\t\t const struct hashmap_entry *he1,\n> \t\t\t const struct hashmap_entry *he2,\n> @@ -100,6 +105,9 @@ void print_bundle_list(FILE *fp, struct bundle_list *list)\n> \tfprintf(fp, \"\\tversion = %d\\n\", list->version);\n> \tfprintf(fp, \"\\tmode = %s\\n\", mode);\n> \n> +\tif (list->heuristic)\n> +\t\tprintf(\"\\theuristic = %s\\n\", heuristics[list->heuristic]);\n\nGiven this condition, the 'heuristic' key should not be sent if it's\n'BUNDLE_HEURISTIC_NONE'. But, as a fallback...\n\n> +\n> \tfor_all_bundles_in_list(list, summarize_bundle, fp);\n> }\n> \n> @@ -142,6 +150,19 @@ static int bundle_list_update(const char *key, const char *value,\n> \t\t\treturn 0;\n> \t\t}\n> \n> +\t\tif (!strcmp(subkey, \"heuristic\")) {\n> +\t\t\tint i;\n> +\t\t\tfor (i = 0; i < BUNDLE_HEURISTIC__COUNT; i++) {\n> +\t\t\t\tif (!strcmp(value, heuristics[i])) {\n> +\t\t\t\t\tlist->heuristic = i;\n> +\t\t\t\t\treturn 0;\n> +\t\t\t\t}\n> +\t\t\t}\n\n...this condition seems to handle 'BUNDLE_HEURISTIC_NONE' anyway. There's no\nharm in this, since 'BUNDLE_HEURISTIC_NONE' is the default value of\n'list->heuristic' anyway.\n\n> void init_bundle_list(struct bundle_list *list);\n> diff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\n> index 7b4f930e532..6fc92a9c0d4 100755\n> --- a/t/t5750-bundle-uri-parse.sh\n> +++ b/t/t5750-bundle-uri-parse.sh\n> @@ -250,4 +250,23 @@ test_expect_success 'parse config format edge cases: empty key or value' '\n> \ttest_cmp_config_output expect actual\n> '\n> \n> +test_expect_success 'parse config format: creationToken heuristic' '\n> +\tcat >expect <<-\\EOF &&\n> +\t[bundle]\n> +\t\tversion = 1\n> +\t\tmode = all\n> +\t\theuristic = creationToken\n> +\t[bundle \"one\"]\n> +\t\turi = http://example.com/bundle.bdl\n> +\t[bundle \"two\"]\n> +\t\turi = https://example.com/bundle.bdl\n> +\t[bundle \"three\"]\n> +\t\turi = file:///usr/share/git/bundle.bdl\n> +\tEOF\n> +\n> +\ttest-tool bundle-uri parse-config expect >actual 2>err &&\n> +\ttest_must_be_empty err &&\n> +\ttest_cmp_config_output expect actual\n> +'\n\nAnd this test verifies that 'heuristic' is no longer being ignored. Looks\ngood!\n\n> +\n> test_done\n\n"},{"id":"470528","messageId":"d9ef2037-0365-dbe3-c5d6-04d7e96fc7c9@github.com","threadId":"59040","inReplyTo":"a1808f0b10cfb519613bc292e30b884962a83275.1673037405.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/8] bundle-uri: parse bundle..creationToken values","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-17T19:24:32Z","receivedAt":"2023-01-17T20:57:14Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The previous change taught Git to parse the bundle.heuristic value,\n> especially when its value is \"creationToken\". Now, teach Git to parse\n> the bundle..creationToken values on each bundle in a bundle list.\n> \n> Before implementing any logic based on creationToken values for the\n> creationToken heuristic, parse and print these values for testing\n> purposes.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> bundle-uri.c | 10 ++++++++++\n> bundle-uri.h | 6 ++++++\n> t/t5750-bundle-uri-parse.sh | 18 ++++++++++++++++++\n> 3 files changed, 34 insertions(+)\n> \n> diff --git a/bundle-uri.c b/bundle-uri.c\n> index 56c94595c2a..63e2cc21057 100644\n> --- a/bundle-uri.c\n> +++ b/bundle-uri.c\n> @@ -80,6 +80,9 @@ static int summarize_bundle(struct remote_bundle_info *info, void *data)\n> \tFILE *fp = data;\n> \tfprintf(fp, \"[bundle \\\"%s\\\"]\\n\", info->id);\n> \tfprintf(fp, \"\\turi = %s\\n\", info->uri);\n> +\n> +\tif (info->creationToken)\n> +\t\tfprintf(fp, \"\\tcreationToken = %\"PRIu64\"\\n\", info->creationToken);\n> \treturn 0;\n> }\n> \n> @@ -190,6 +193,13 @@ static int bundle_list_update(const char *key, const char *value,\n> \t\treturn 0;\n> \t}\n> \n> +\tif (!strcmp(subkey, \"creationtoken\")) {\n> +\t\tif (sscanf(value, \"%\"PRIu64, &bundle->creationToken) != 1)\n> +\t\t\twarning(_(\"could not parse bundle list key %s with value '%s'\"),\n> +\t\t\t\t\"creationToken\", value);\n> +\t\treturn 0;\n> +\t}\n\nLike the 'heuristic' key in the last patch, the parsing of 'creationToken'\nis pretty straightforward.\n\n> diff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\n> index 6fc92a9c0d4..81bdf58b944 100755\n> --- a/t/t5750-bundle-uri-parse.sh\n> +++ b/t/t5750-bundle-uri-parse.sh\n> @@ -258,10 +258,13 @@ test_expect_success 'parse config format: creationToken heuristic' '\n> \t\theuristic = creationToken\n> \t[bundle \"one\"]\n> \t\turi = http://example.com/bundle.bdl\n> +\t\tcreationToken = 123456\n> \t[bundle \"two\"]\n> \t\turi = https://example.com/bundle.bdl\n> +\t\tcreationToken = 12345678901234567890\n> \t[bundle \"three\"]\n> \t\turi = file:///usr/share/git/bundle.bdl\n> +\t\tcreationToken = 1\n> \tEOF\n> \n> \ttest-tool bundle-uri parse-config expect >actual 2>err &&\n> @@ -269,4 +272,19 @@ test_expect_success 'parse config format: creationToken heuristic' '\n> \ttest_cmp_config_output expect actual\n> '\n> \n> +test_expect_success 'parse config format edge cases: creationToken heuristic' '\n> +\tcat >expect <<-\\EOF &&\n> +\t[bundle]\n> +\t\tversion = 1\n> +\t\tmode = all\n> +\t\theuristic = creationToken\n> +\t[bundle \"one\"]\n> +\t\turi = http://example.com/bundle.bdl\n> +\t\tcreationToken = bogus\n> +\tEOF\n> +\n> +\ttest-tool bundle-uri parse-config expect >actual 2>err &&\n> +\tgrep \"could not parse bundle list key creationToken with value '\\''bogus'\\''\" err\n> +'\n\nAnd the tests cover both valid and invalid cases nicely.\n\n> +\n> test_done\n\n"},{"id":"470557","messageId":"be447e59-26be-75b1-65b4-4e976a2e28d4@github.com","threadId":"59040","inReplyTo":"4d13cea1-e9f6-381f-14ef-8a5a645e2a8f@github.com","subject":"Re: [PATCH 1/8] t5558: add tests for creationToken heuristic","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-17T21:00:27Z","receivedAt":"2023-01-17T23:15:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/17/2023 1:17 PM, Victoria Dye wrote:\n> Derrick Stolee via GitGitGadget wrote:\n\n>> +\tfor b in 1 2 3 4\n>> +\tdo\n>> +\t\ttest_bundle_downloaded bundle-$b.bundle trace-clone.txt ||\n>> +\t\t\treturn 1\n>> +\tdone\n> \n> Because the current state of bundle list handling is equivalent to \"no\n> heuristic\", this pre-existing test is just updated to verify all bundles are\n> downloaded. This isn't new behavior, but it'll be relevant to compare with\n> the behavior of the 'creationToken' heuristic. \n> \n> I was going to ask how the tests verify that *only* the expected bundles are\n> downloaded, and it looks like later patches [1] handle that with\n> '! test_bundle_downloaded' checks. That approach seems a bit fragile (if a\n> bundle's name doesn't match the '! test_bundle_downloaded' check for some\n> reason, the bundle can be either downloaded or not with no effect on the\n> test result). Would something like a 'test_downloaded_bundle_count' work\n> instead?\n\nOr, perhaps we could check the exact list _and order_ using this slightly\nmore generic helper?\n\n# Given a GIT_TRACE2_EVENT log over stdin, writes to stdout a list of URLs\n# sent to git-remote-https child processes.\ntest_remote_https_urls() {\n\tgrep -e '\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\".*\"\\]' |\n\t\tsed -e 's/{\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"//g' \\\n\t\t -e 's/\"\\]}//g'\n}\n\nWith a test example looking a lot like this:\n\n\tcat >expect <<-EOF &&\n\t$HTTPD_URL/newest.bundle\n\t$HTTPD_URL/new.bundle\n\t$HTTPD_URL/everything.bundle\n\tEOF\n\n\ttest_remote_https_urls actual &&\n\ttest_cmp expect actual\n\nThanks for the inspiration.\n\n>> +test_expect_success 'clone bundle list (http, creationToken)' '\n>> +\tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n>> +\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n>> +\t[bundle]\n>> +\t\tversion = 1\n>> +\t\tmode = all\n>> +\t\theuristic = creationToken\n>> +\n>> +\t[bundle \"bundle-1\"]\n>> +\t\turi = bundle-1.bundle\n>> +\t\tcreationToken = 1\n>> +\n>> +\t[bundle \"bundle-2\"]\n>> +\t\turi = bundle-2.bundle\n>> +\t\tcreationToken = 2\n>> +\n>> +\t[bundle \"bundle-3\"]\n>> +\t\turi = bundle-3.bundle\n>> +\t\tcreationToken = 3\n>> +\n>> +\t[bundle \"bundle-4\"]\n>> +\t\turi = bundle-4.bundle\n>> +\t\tcreationToken = 4\n>> +\tEOF\n>> +\n>> +\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" . clone-list-http-2 &&\n>> +\n>> +\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n>> +\tgit -C clone-list-http-2 cat-file --batch-check \n> This test looks like the one that was updated above, but adds the\n> 'creationToken' heuristic key. However, the 'test_bundle_downloaded' check\n> isn't included - if it were, it would need to verify that all bundles were\n> downloaded, with the heuristic being ignored, all bundles will be downloaded\n> (which isn't consistent with what the 'creationToken' heuristic will\n> *eventually* do).\n> \n> As a matter of personal preference (so no pressure to change if you\n> disagree), I find this test in its current state a bit misleading; because\n> it's a 'test_expect_success' and there's no \"NEEDSWORK\" or \"TODO\", I could\n> easily assume that cloning from a bundle list with the 'creationToken'\n> heuristic is working as-intended at this point (that is, there's no\n> indication that it's not implemented). \n\nIt's true that it's not implemented right now, and it is a bit misleading\nbecause of that. At clone time, the only thing that will change with the\nimplementation is possibly the order of the files being downloaded (and\nthe order is not predictable before that implementation).\n\nThe restriction of _not_ downloading some files comes only for the 'git\nfetch' implementation, so these test changes are only foundations for those\nfuture tests.\n\nThe only benefit of having these tests right now is that we get some\ndemonstration that the existing implementation ignores unknown properties\nin the bundle list.\n\n> If you did want to change it, adding a 'NEEDSWORK' comment, changing to\n> 'test_expect_failure' & including the appropriate 'test_bundle_downloaded'\n> check, or moving this test to the patch where the heuristic is implemented\n> would mitigate any confusion. That said, this \"issue\" is resolved by the\n> end of the series anyway, so it's really a low priority fix.\nI think if we wanted to go this route, we could do it with the \"download\nthe bundles in this order\" check, or possibly by adding the 'git fetch'\nbehavior into the test at this point. I'll consider these options for v2.\n\nThanks,\n-Stolee\n"},{"id":"470728","messageId":"ede340d1-bce4-0c1d-7afb-4874a67d1803@github.com","threadId":"59040","inReplyTo":"57c0174d3752fb61a05e0653de9d3057616ed16a.1673037405.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/8] bundle-uri: download in creationToken order","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-19T18:32:13Z","receivedAt":"2023-01-19T18:32:33Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> +static int fetch_bundles_by_token(struct repository *r,\n> +\t\t\t\t struct bundle_list *list)\n> +{\n> +\tint cur;\n> +\tint pop_or_push = 0;\n> +\tstruct bundle_list_context ctx = {\n> +\t\t.r = r,\n> +\t\t.list = list,\n> +\t\t.mode = list->mode,\n> +\t};\n> +\tstruct sorted_bundle_list sorted = {\n> +\t\t.alloc = hashmap_get_size(&list->bundles),\n> +\t};\n> +\n> +\tALLOC_ARRAY(sorted.items, sorted.alloc);\n> +\n> +\tfor_all_bundles_in_list(list, insert_bundle, &sorted);\n> +\n> +\tQSORT(sorted.items, sorted.nr, compare_creation_token);\n\nSo, at this point, 'sorted' is ordered by *decreasing* creation token? With\nthe loop below being somewhat complex, it would be nice to have a comment\nmention that explicitly so readers have a clear understanding of the\n\"initial state\" before entering the loop.\n\n> +\n> +\t/*\n> +\t * Use a stack-based approach to download the bundles and attempt\n> +\t * to unbundle them in decreasing order by creation token. If we\n> +\t * fail to unbundle (after a successful download) then move to the\n> +\t * next non-downloaded bundle (push to the stack) and attempt\n> +\t * downloading. Once we succeed in applying a bundle, move to the\n> +\t * previous unapplied bundle (pop the stack) and attempt to unbundle\n> +\t * it again.\n> +\t *\n> +\t * In the case of a fresh clone, we will likely download all of the\n> +\t * bundles before successfully unbundling the oldest one, then the\n> +\t * rest of the bundles unbundle successfully in increasing order\n> +\t * of creationToken.\n> +\t *\n> +\t * If there are existing objects, then this process may terminate\n> +\t * early when all required commits from \"new\" bundles exist in the\n> +\t * repo's object store.\n> +\t */\n> +\tcur = 0;\n> +\twhile (cur >= 0 && cur < sorted.nr) {\n> +\t\tstruct remote_bundle_info *bundle = sorted.items[cur];\n> +\t\tif (!bundle->file) {\n> +\t\t\t/* Not downloaded yet. Try downloading. */\n> +\t\t\tif (download_bundle_to_file(bundle, &ctx)) {\n> +\t\t\t\t/* Failure. Push to the stack. */\n> +\t\t\t\tpop_or_push = 1;\n> +\t\t\t\tgoto stack_operation;\n\nPersonally, I find the use of \"stack\" terminology more confusing than not.\n'sorted' isn't really a stack, it's a list with fixed contents being\ntraversed stepwise with 'cur'. For example, 'pop_or_push' being renamed to\n'move_direction' or 'step' something along those lines might more clearly\nindicate what's actually happening with 'cur' & 'sorted'.\n\n> +\t\t\t}\n> +\n> +\t\t\t/* We expect bundles when using creationTokens. */\n> +\t\t\tif (!is_bundle(bundle->file, 1)) {\n> +\t\t\t\twarning(_(\"file downloaded from '%s' is not a bundle\"),\n> +\t\t\t\t\tbundle->uri);\n> +\t\t\t\tbreak;\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\tif (bundle->file && !bundle->unbundled) {\n> +\t\t\t/*\n> +\t\t\t * This was downloaded, but not successfully\n> +\t\t\t * unbundled. Try unbundling again.\n> +\t\t\t */\n> +\t\t\tif (unbundle_from_file(ctx.r, bundle->file)) {\n> +\t\t\t\t/* Failed to unbundle. Push to stack. */\n> +\t\t\t\tpop_or_push = 1;\n> +\t\t\t} else {\n> +\t\t\t\t/* Succeeded in unbundle. Pop stack. */\n> +\t\t\t\tpop_or_push = -1;\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\t/*\n> +\t\t * Else case: downloaded and unbundled successfully.\n> +\t\t * Skip this by moving in the same direction as the\n> +\t\t * previous step.\n> +\t\t */\n> +\n> +stack_operation:\n> +\t\t/* Move in the specified direction and repeat. */\n> +\t\tcur += pop_or_push;\n> +\t}\n\nAfter reading through this loop, I generally understood *what* its doing,\nbut didn't really follow *why* the download & unbundling is done like this.\nI needed to refer back to the design doc\n('Documentation/technical/bundle-uri.txt') to understand some basic\nassumptions about bundles:\n\n- A new bundle's creation token should always be strictly greater than the\n previous newest bundle's creation token. I don't see any special handling\n for equal creation tokens, so my assumption is that the sorting of the\n list arbitrarily assigns one to be \"greater\" and it's dealt with that way.\n- The bundle with the lowest creation token should always be unbundleable,\n since it contains all objects in an initial clone.\n\nI do still have some questions, though:\n\n- Why would 'unbundle_from_file()' fail? From context clues, I'm guessing it\n fails if it has some unreachable objects (as in an incremental bundle), or\n if it's corrupted somehow.\n- Why would 'download_bundle_to_file()' to fail? Unlike\n 'unbundle_from_file()', it looks like that represents an unexpected error.\n\nAlso - it seems like one of the assumptions here is that, if a bundle can't\nbe downloaded & unbundled, no bundle with a higher creation token can be\nsuccessfully unbundled ('download_bundle_to_file()' sets 'pop_or_push' to\n'1', which will cause the loop to ignore all higher-token bundles and return\na nonzero value from the function). \n\nI don't think that assumption is necessarily true, though. Suppose you have\na \"base\" bundle 100 and incremental bundles 101 and 102. 101 has all objects\nfrom a new branch A, and 102 has all objects from a newer branch B (not\nbased on any objects in A). In this case, 102 could be unbundled even if 101\nis corrupted/can't be downloaded, but we'd run into issues if we store 102\nas the \"latest unbundled creation token\" (because it implies that 101 was\nunbundled).\n\nIs there any benefit to trying to unbundle those higher bundles *without*\nadvancing the \"latest creation token\"? E.g. in my example, unbundle 102 but\nstore '100' as the latest creation token?\n\n> diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\n> index 328caeeae9a..d7461ec907e 100755\n> --- a/t/t5558-clone-bundle-uri.sh\n> +++ b/t/t5558-clone-bundle-uri.sh\n> @@ -368,6 +368,8 @@ test_expect_success 'clone bundle list (HTTP, any mode)' '\n> '\n> \n> test_expect_success 'clone bundle list (http, creationToken)' '\n> +\ttest_when_finished rm -f trace*.txt &&\n> +\n> \tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n> \tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n> \t[bundle]\n> @@ -392,10 +394,45 @@ test_expect_success 'clone bundle list (http, creationToken)' '\n> \t\tcreationToken = 4\n> \tEOF\n> \n> -\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" . clone-list-http-2 &&\n> +\tGIT_TRACE2_EVENT=$(pwd)/trace-clone.txt \\\n> +\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n> +\t\tclone-from clone-list-http-2 &&\n> \n> \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n> -\tgit -C clone-list-http-2 cat-file --batch-check +\tgit -C clone-list-http-2 cat-file --batch-check +\n> +\tfor b in 1 2 3 4\n> +\tdo\n> +\t\ttest_bundle_downloaded bundle-$b.bundle trace-clone.txt ||\n> +\t\t\treturn 1\n> +\tdone\n\nIf I understand correctly, these added conditions would have passed even if\nthey were added when the test was initially created in patch 1, but they're\nadded here to tie them to the implementation of the creationToken heuristic?\nSeems reasonable.\n\n> +'\n> +\n> +test_expect_success 'clone bundle list (http, creationToken)' '\n\nThis new test has the same name as the one above it - how does it differ\nfrom that one? Whatever the difference is, can that be noted somehow in the\ntitle or a comment?\n\n> +\ttest_when_finished rm -f trace*.txt &&\n> +\n> +\tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n> +\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n> +\t[bundle]\n> +\t\tversion = 1\n> +\t\tmode = all\n> +\t\theuristic = creationToken\n> +\n> +\t[bundle \"bundle-1\"]\n> +\t\turi = bundle-1.bundle\n> +\t\tcreationToken = 1\n> +\n> +\t[bundle \"bundle-2\"]\n> +\t\turi = bundle-2.bundle\n> +\t\tcreationToken = 2\n> +\tEOF\n> +\n> +\tGIT_TRACE2_EVENT=$(pwd)/trace-clone.txt \\\n> +\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n> +\t\tclone-from clone-token-http &&\n> +\n> +\ttest_bundle_downloaded bundle-1.bundle trace-clone.txt &&\n> +\ttest_bundle_downloaded bundle-2.bundle trace-clone.txt\n> '\n> \n> # Do not add tests here unless they use the HTTP server, as they will\n> diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\n> index 1928ea1dd7c..57476b6e6d7 100755\n> --- a/t/t5601-clone.sh\n> +++ b/t/t5601-clone.sh\n> @@ -831,6 +831,56 @@ test_expect_success 'auto-discover multiple bundles from HTTP clone' '\n> \tgrep -f pattern trace.txt\n> '\n> \n> +# Usage: test_bundle_downloaded \n> +test_bundle_downloaded () {\n> +\tcat >pattern <<-EOF &&\n> +\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/$1.bundle\"\\]\n> +\tEOF\n> +\tgrep -f pattern \"$2\"\n> +}\n\nThis function is the same as the one created in 't5558'. Should it be moved\nto 'lib-bundle.sh' or 'test-lib.sh' to avoid duplicate code?\n\n> +\n> +test_expect_success 'auto-discover multiple bundles from HTTP clone: creationToken heuristic' '\n> +\ttest_when_finished rm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/repo4.git\" &&\n> +\ttest_when_finished rm -rf clone-heuristic trace*.txt &&\n> +\n> +\ttest_commit -C src newest &&\n> +\tgit -C src bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/newest.bundle\" HEAD~1..HEAD &&\n> +\tgit clone --bare --no-local src \"$HTTPD_DOCUMENT_ROOT_PATH/repo4.git\" &&\n> +\n> +\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/repo4.git/config\" <<-EOF &&\n> +\t[uploadPack]\n> +\t\tadvertiseBundleURIs = true\n> +\n> +\t[bundle]\n> +\t\tversion = 1\n> +\t\tmode = all\n> +\t\theuristic = creationToken\n> +\n> +\t[bundle \"everything\"]\n> +\t\turi = $HTTPD_URL/everything.bundle\n> +\t\tcreationtoken = 1\n> +\n> +\t[bundle \"new\"]\n> +\t\turi = $HTTPD_URL/new.bundle\n> +\t\tcreationtoken = 2\n> +\n> +\t[bundle \"newest\"]\n> +\t\turi = $HTTPD_URL/newest.bundle\n> +\t\tcreationtoken = 3\n> +\tEOF\n> +\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" \\\n> +\t\tgit -c protocol.version=2 \\\n> +\t\t -c transfer.bundleURI=true clone \\\n> +\t\t\"$HTTPD_URL/smart/repo4.git\" clone-heuristic &&\n> +\n> +\t# We should fetch all bundles\n> +\tfor b in everything new newest\n> +\tdo\n> +\t\ttest_bundle_downloaded $b trace-clone.txt || return 1\n> +\tdone\n> +'\n> +\n> # DO NOT add non-httpd-specific tests here, because the last part of this\n> # test script is only executed when httpd is available and enabled.\n> \n\n"},{"id":"470730","messageId":"61a830f0-a8d8-5a8a-e952-db213d571352@github.com","threadId":"59040","inReplyTo":"d9c6f50e4f218267c1e8da060ce5b190dc8a709c.1673037405.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/8] clone: set fetch.bundleURI if appropriate","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-19T19:42:53Z","receivedAt":"2023-01-19T19:43:01Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> diff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt\n> index cd65d236b43..4f796218aab 100644\n> --- a/Documentation/config/fetch.txt\n> +++ b/Documentation/config/fetch.txt\n> @@ -96,3 +96,11 @@ fetch.writeCommitGraph::\n> \tmerge and the write may take longer. Having an updated commit-graph\n> \tfile helps performance of many Git commands, including `git merge-base`,\n> \t`git push -f`, and `git log --graph`. Defaults to false.\n> +\n> +fetch.bundleURI::\n> +\tThis value stores a URI for fetching Git object data from a bundle URI\n> +\tbefore performing an incremental fetch from the origin Git server. If\n> +\tthe value is `` then running `git fetch ` is equivalent to\n> +\tfirst running `git fetch --bundle-uri=` immediately before\n> +\t`git fetch `. See details of the `--bundle-uri` option in\n> +\tlinkgit:git-fetch[1].\n\nSince it's not mentioned from this or any other user-facing documentation\n(AFAICT), could you note that this value is set automatically by 'git clone'\niff '--bundle-uri' is specified *and* 'bundle.heuristic' is set for the\ninitially downloaded bundle list?\n\nIt would also be nice to make note of that behavior in the documentation of\nthe '--bundle-uri' option in 'Documentation/git-clone.txt', since command\ndocumentation in general seems to be more popular/visible to users than\nconfig docs.\n\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index 5453ba5277f..5370617664d 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -1248,12 +1248,16 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> \t * data from the --bundle-uri option.\n> \t */\n> \tif (bundle_uri) {\n> +\t\tint has_heuristic = 0;\n> +\n> \t\t/* At this point, we need the_repository to match the cloned repo. */\n> \t\tif (repo_init(the_repository, git_dir, work_tree))\n> \t\t\twarning(_(\"failed to initialize the repo, skipping bundle URI\"));\n> -\t\telse if (fetch_bundle_uri(the_repository, bundle_uri))\n> +\t\telse if (fetch_bundle_uri(the_repository, bundle_uri, &has_heuristic))\n> \t\t\twarning(_(\"failed to fetch objects from bundle URI '%s'\"),\n> \t\t\t\tbundle_uri);\n> +\t\telse if (has_heuristic)\n> +\t\t\tgit_config_set_gently(\"fetch.bundleuri\", bundle_uri);\n\nIf the heuristic is anything other than \"none\", this config value is set in\nthe repository-scoped config file. Makes sense!\n\n> \t}\n> \n> \tstrvec_push(&transport_ls_refs_options.ref_prefixes, \"HEAD\");\n> diff --git a/bundle-uri.c b/bundle-uri.c\n> index b30c85ba6f2..1dbbbb980eb 100644\n> --- a/bundle-uri.c\n> +++ b/bundle-uri.c\n> @@ -594,9 +594,10 @@ static int fetch_bundle_list_in_config_format(struct repository *r,\n> \t * it advertises are expected to be bundles, not nested lists.\n> \t * We can drop 'global_list' and 'depth'.\n> \t */\n> -\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN)\n> +\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN) {\n> \t\tresult = fetch_bundles_by_token(r, &list_from_bundle);\n> -\telse if ((result = download_bundle_list(r, &list_from_bundle,\n> +\t\tglobal_list->heuristic = BUNDLE_HEURISTIC_CREATIONTOKEN;\n\nIf the 'heuristic' field already existed and was being used to apply\nbundles, why wasn't 'global_list->heuristic' already being set? Before this\npatch, was the 'global_list->heuristic' field not accurately reflecting the\nheuristic type of a given bundle list? \n\nIf so, I think it'd make sense to move this section to patch 4 [1], since\nthat's when the heuristic is first applied to the bundle list.\n\n[1] https://lore.kernel.org/git/57c0174d3752fb61a05e0653de9d3057616ed16a.1673037405.git.gitgitgadget@gmail.com/\n\n> diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\n> index d7461ec907e..8ff560425ee 100755\n> --- a/t/t5558-clone-bundle-uri.sh\n> +++ b/t/t5558-clone-bundle-uri.sh\n> @@ -435,6 +435,39 @@ test_expect_success 'clone bundle list (http, creationToken)' '\n> \ttest_bundle_downloaded bundle-2.bundle trace-clone.txt\n> '\n> \n> +test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n> +\ttest_when_finished rm -rf fetch-http-4 trace*.txt &&\n> +\n> +\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n> +\t[bundle]\n> +\t\tversion = 1\n> +\t\tmode = all\n> +\t\theuristic = creationToken\n> +\n> +\t[bundle \"bundle-1\"]\n> +\t\turi = bundle-1.bundle\n> +\t\tcreationToken = 1\n> +\tEOF\n> +\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" \\\n> +\tgit clone --single-branch --branch=base \\\n> +\t\t--bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n> +\t\t\"$HTTPD_URL/smart/fetch.git\" fetch-http-4 &&\n> +\n> +\ttest_cmp_config -C fetch-http-4 \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n> +\n> +\t# The clone should copy two files: the list and bundle-1.\n> +\ttest_bundle_downloaded bundle-list trace-clone.txt &&\n> +\ttest_bundle_downloaded bundle-1.bundle trace-clone.txt &&\n> +\n> +\t# only received base ref from bundle-1\n> +\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n> +\tcat >expect <<-\\EOF &&\n> +\trefs/bundles/base\n> +\tEOF\n> +\ttest_cmp expect refs\n> +'\n\nThis test looks good - it verifies the config update, bundle download, and\nunbundle all work as intended.\n\n> +\n> # Do not add tests here unless they use the HTTP server, as they will\n> # not run unless the HTTP dependencies exist.\n> \n\n"},{"id":"470731","messageId":"699c4d66-f159-c745-4287-a7815aa17189@github.com","threadId":"59040","inReplyTo":"afcfd27a883d16009a2c55c3dcfb5ade07132b50.1673037405.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/8] bundle-uri: drop bundle.flag from design doc","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-19T19:44:17Z","receivedAt":"2023-01-19T19:44:26Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The Implementation Plan section lists a 'bundle.flag' option that is not\n> documented anywhere else. What is documented elsewhere in the document\n> and implemented by previous changes is the 'bundle.heuristic' config\n> key. For now, a heuristic is required to indicate that a bundle list is\n> organized for use during 'git fetch', and it is also sufficient for all\n> existing designs.\n\nGood catch, thanks for keeping the documentation consistent!\n\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/technical/bundle-uri.txt | 8 ++++----\n> 1 file changed, 4 insertions(+), 4 deletions(-)\n> \n> diff --git a/Documentation/technical/bundle-uri.txt b/Documentation/technical/bundle-uri.txt\n> index b78d01d9adf..91d3a13e327 100644\n> --- a/Documentation/technical/bundle-uri.txt\n> +++ b/Documentation/technical/bundle-uri.txt\n> @@ -479,14 +479,14 @@ outline for submitting these features:\n> (This choice is an opt-in via a config option and a command-line\n> option.)\n> \n> -4. Allow the client to understand the `bundle.flag=forFetch` configuration\n> +4. Allow the client to understand the `bundle.heuristic` configuration key\n> and the `bundle..creationToken` heuristic. When `git clone`\n> - discovers a bundle URI with `bundle.flag=forFetch`, it configures the\n> - client repository to check that bundle URI during later `git fetch `\n> + discovers a bundle URI with `bundle.heuristic`, it configures the client\n> + repository to check that bundle URI during later `git fetch `\n> commands.\n> \n> 5. Allow clients to discover bundle URIs during `git fetch` and configure\n> - a bundle URI for later fetches if `bundle.flag=forFetch`.\n> + a bundle URI for later fetches if `bundle.heuristic` is set.\n> \n> 6. Implement the \"inspect headers\" heuristic to reduce data downloads when\n> the `bundle..creationToken` heuristic is not available.\n\n"},{"id":"470733","messageId":"58269fe6-ebe1-7b12-4cd9-2110a94543e5@github.com","threadId":"59040","inReplyTo":"1627fc158b1e301a1663e24f9f21268b4f1caa55.1673037405.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 7/8] fetch: fetch from an external bundle URI","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-19T20:34:13Z","receivedAt":"2023-01-19T20:34:28Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> When a user specifies a URI via 'git clone --bundle-uri', that URI may\n> be a bundle list that advertises a 'bundle.heuristic' value. In that\n> case, the Git client stores a 'fetch.bundleURI' config value storing\n> that URI.\n> \n> Teach 'git fetch' to check for this config value and download bundles\n> from that URI before fetching from the Git remote(s). Likely, the bundle\n> provider has configured a heuristic (such as \"creationToken\") that will\n> allow the Git client to download only a portion of the bundles before\n> continuing the fetch.\n> \n> Since this URI is completely independent of the remote server, we want\n> to be sure that we connect to the bundle URI before creating a\n> connection to the Git remote. We do not want to hold a stateful\n> connection for too long if we can avoid it.\n> \n> To test that this works correctly, extend the previous tests that set\n> 'fetch.bundleURI' to do follow-up fetches. The bundle list is updated\n> incrementally at each phase to demonstrate that the heuristic avoids\n> downloading older bundles. This includes the middle fetch downloading\n> the objects in bundle-3.bundle from the Git remote, and therefore not\n> needing that bundle in the third fetch.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/fetch.c | 8 +++++\n> t/t5558-clone-bundle-uri.sh | 59 +++++++++++++++++++++++++++++++++++++\n> 2 files changed, 67 insertions(+)\n> \n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 7378cafeec9..fbb1d470c38 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -29,6 +29,7 @@\n> #include \"commit-graph.h\"\n> #include \"shallow.h\"\n> #include \"worktree.h\"\n> +#include \"bundle-uri.h\"\n> \n> #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n> \n> @@ -2109,6 +2110,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,\n> int cmd_fetch(int argc, const char **argv, const char *prefix)\n> {\n> \tint i;\n> +\tconst char *bundle_uri;\n> \tstruct string_list list = STRING_LIST_INIT_DUP;\n> \tstruct remote *remote = NULL;\n> \tint result = 0;\n> @@ -2194,6 +2196,12 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tif (dry_run)\n> \t\twrite_fetch_head = 0;\n> \n> +\tif (!git_config_get_string_tmp(\"fetch.bundleuri\", &bundle_uri) &&\n> +\t !starts_with(bundle_uri, \"remote:\")) {\n\nMaybe a silly question, by why would the bundle URI start with 'remote:'\n(and why do we silently skip fetching from the URI in that case)?\n\n> +\t\tif (fetch_bundle_uri(the_repository, bundle_uri, NULL))\n> +\t\t\twarning(_(\"failed to fetch bundles from '%s'\"), bundle_uri);\n> +\t}\n> +\n> \tif (all) {\n> \t\tif (argc == 1)\n> \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\n> diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\n> index 8ff560425ee..3f4d61a915c 100755\n> --- a/t/t5558-clone-bundle-uri.sh\n> +++ b/t/t5558-clone-bundle-uri.sh\n> @@ -465,6 +465,65 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n> \tcat >expect <<-\\EOF &&\n> \trefs/bundles/base\n> \tEOF\n> +\ttest_cmp expect refs &&\n\nAt this point in the test, only 'base' is in the cloned repo (equivalent to\nthe contents of 'bundle-1').\n\n> +\n> +\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n> +\t[bundle \"bundle-2\"]\n> +\t\turi = bundle-2.bundle\n> +\t\tcreationToken = 2\n> +\tEOF\n> +\n> +\t# Fetch the objects for bundle-2 _and_ bundle-3.\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace1.txt\" \\\n> +\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n> +\t\trefs/heads/left:refs/heads/left \\\n> +\t\trefs/heads/right:refs/heads/right &&\n> +\n> +\t# This fetch should copy two files: the list and bundle-2.\n> +\ttest_bundle_downloaded bundle-list trace1.txt &&\n> +\ttest_bundle_downloaded bundle-2.bundle trace1.txt &&\n> +\t! test_bundle_downloaded bundle-1.bundle trace1.txt &&\n\nNow, with 'bundle-2' in the list, fetch 'left' and 'right'. 'bundle-1' is\nnot fetched because we already have its contents (in 'base'), 'bundle-2' is\ndownloaded to get 'left', and 'right' is fetched directly from the repo.\n\n> +\n> +\t# received left from bundle-2\n> +\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n> +\tcat >expect <<-\\EOF &&\n> +\trefs/bundles/base\n> +\trefs/bundles/left\n> +\tEOF\n> +\ttest_cmp expect refs &&\n> +\n> +\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n> +\t[bundle \"bundle-3\"]\n> +\t\turi = bundle-3.bundle\n> +\t\tcreationToken = 3\n> +\n> +\t[bundle \"bundle-4\"]\n> +\t\turi = bundle-4.bundle\n> +\t\tcreationToken = 4\n> +\tEOF\n> +\n> +\t# This fetch should skip bundle-3.bundle, since its objets are\n\ns/objets/objects\n\n> +\t# already local (we have the requisite commits for bundle-4.bundle).\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n> +\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n> +\t\trefs/heads/merge:refs/heads/merge &&\n> +\n> +\t# This fetch should copy three files: the list, bundle-3, and bundle-4.\n> +\ttest_bundle_downloaded bundle-list trace2.txt &&\n> +\ttest_bundle_downloaded bundle-4.bundle trace2.txt &&\n> +\t! test_bundle_downloaded bundle-1.bundle trace2.txt &&\n> +\t! test_bundle_downloaded bundle-2.bundle trace2.txt &&\n> +\t! test_bundle_downloaded bundle-3.bundle trace2.txt &&\n\n'bundle-3' and 'bundle-4' are then added to the list and we fetch 'merge'.\nThe repository already has 'base', 'left', and 'right' so we don't need to\ndownload 'bundle-1', 'bundle-2', and 'bundle-3' respectively; 'bundle-4' is\ndownloaded to get 'merge'.\n\n> +\n> +\t# received merge ref from bundle-4, but right is missing\n> +\t# because we did not download bundle-3.\n> +\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n> +\n> +\tcat >expect <<-\\EOF &&\n> +\trefs/bundles/base\n> +\trefs/bundles/left\n> +\trefs/bundles/merge\n\nAnd this confirms that 'base', 'left', and 'merge' all came from bundles\n(and 'right' did not), and everything is working as expected. This is test\nis both easy to understand (the comments clearly explain each step without\nbeing overly verbose) and thorough. Looks good!\n\n> +\tEOF\n> \ttest_cmp expect refs\n> '\n> \n\n"},{"id":"470741","messageId":"f7a777f0-3f37-86a1-f520-1ab6853b9ed1@github.com","threadId":"59040","inReplyTo":"51f210ddeb46fb06e885dc384a486c4bb16ad8cd.1673037405.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 8/8] bundle-uri: store fetch.bundleCreationToken","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-19T22:24:03Z","receivedAt":"2023-01-19T22:43:02Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> When a bundle list specifies the \"creationToken\" heuristic, the Git\n> client downloads the list and then starts downloading bundles in\n> descending creationToken order. This process stops as soon as all\n> downloaded bundles can be applied to the repository (because all\n> required commits are present in the repository or in the downloaded\n> bundles).\n> \n> When checking the same bundle list twice, this strategy requires\n> downloading the bundle with the maximum creationToken again, which is\n> wasteful. The creationToken heuristic promises that the client will not\n> have a use for that bundle if its creationToken value is the at most the\n\ns/is the at most/is at most(?)\n\n> previous creationToken value.\n> \n> To prevent these wasteful downloads, create a fetch.bundleCreationToken\n> config setting that the Git client sets after downloading bundles. This\n> value allows skipping that maximum bundle download when this config\n> value is the same value (or larger).\n> \n> To test that this works correctly, we can insert some \"duplicate\"\n> fetches into existing tests and demonstrate that only the bundle list is\n> downloaded.\n> \n> The previous logic for downloading bundles by creationToken worked even\n> if the bundle list was empty, but now we have logic that depends on the\n> first entry of the list. Terminate early in the (non-sensical) case of\n> an empty bundle list.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/config/fetch.txt | 8 ++++++++\n> bundle-uri.c | 35 ++++++++++++++++++++++++++++++++--\n> t/t5558-clone-bundle-uri.sh | 25 +++++++++++++++++++++++-\n> 3 files changed, 65 insertions(+), 3 deletions(-)\n> \n> diff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt\n> index 4f796218aab..96755ba148b 100644\n> --- a/Documentation/config/fetch.txt\n> +++ b/Documentation/config/fetch.txt\n> @@ -104,3 +104,11 @@ fetch.bundleURI::\n> \tfirst running `git fetch --bundle-uri=` immediately before\n> \t`git fetch `. See details of the `--bundle-uri` option in\n> \tlinkgit:git-fetch[1].\n> +\n> +fetch.bundleCreationToken::\n> +\tWhen using `fetch.bundleURI` to fetch incrementally from a bundle\n> +\tlist that uses the \"creationToken\" heuristic, this config value\n> +\tstores the maximum `creationToken` value of the downloaded bundles.\n> +\tThis value is used to prevent downloading bundles in the future\n> +\tif the advertised `creationToken` is not strictly larger than this\n> +\tvalue.\n> diff --git a/bundle-uri.c b/bundle-uri.c\n> index 1dbbbb980eb..98655bd6721 100644\n> --- a/bundle-uri.c\n> +++ b/bundle-uri.c\n> @@ -464,6 +464,8 @@ static int fetch_bundles_by_token(struct repository *r,\n> {\n> \tint cur;\n> \tint pop_or_push = 0;\n> +\tconst char *creationTokenStr;\n> +\tuint64_t maxCreationToken;\n> \tstruct bundle_list_context ctx = {\n> \t\t.r = r,\n> \t\t.list = list,\n> @@ -477,8 +479,27 @@ static int fetch_bundles_by_token(struct repository *r,\n> \n> \tfor_all_bundles_in_list(list, insert_bundle, &sorted);\n> \n> +\tif (!sorted.nr) {\n> +\t\tfree(sorted.items);\n> +\t\treturn 0;\n> +\t}\n\nThis check is added here because we're only now at risk for an invalid\naccess to 'sorted' (checking 'sorted.items[0]' below).\n\n> +\n> \tQSORT(sorted.items, sorted.nr, compare_creation_token);\n> \n> +\t/*\n> +\t * If fetch.bundleCreationToken exists, parses to a uint64t, and\n> +\t * is not strictly smaller than the maximum creation token in the\n> +\t * bundle list, then do not download any bundles.\n> +\t */\n> +\tif (!repo_config_get_value(r,\n> +\t\t\t\t \"fetch.bundlecreationtoken\",\n> +\t\t\t\t &creationTokenStr) &&\n> +\t sscanf(creationTokenStr, \"%\"PRIu64, &maxCreationToken) == 1 &&\n> +\t sorted.items[0]->creationToken <= maxCreationToken) {\n> +\t\tfree(sorted.items);\n> +\t\treturn 0;\n> +\t}\n\nAnd here, we exit if the cached creation token is greater than or equal to\nthe highest advertised token. Overall, this seems pretty safe:\n\n- If the value is (somehow) manually updated to something larger than it\n should be, objects will be fetched from the server that could have\n otherwise come from a bundle. Not ideal, but no worse than a regular\n fetch.\n- If the value is too small or invalid (i.e., not an unsigned integer),\n we'll download the first bundle, unbundle it, then overwrite the invalid\n 'fetch.bundlecreationtoken' with a new valid one.\n\nThe latter is self-correcting, but should the former be documented\nsomewhere? For example, if someone changes which bundle server they're using\nwith a repo, the creation token numbering scheme might be completely\ndifferent. In that case, a user would probably want to \"reset\" the\n'fetch.bundlecreationtoken' and start fresh with a new bundle list (even if\nthe recommended method is \"manually delete the config key from your repo\").\n\n> +\n> \t/*\n> \t * Use a stack-based approach to download the bundles and attempt\n> \t * to unbundle them in decreasing order by creation token. If we\n> @@ -541,14 +562,24 @@ stack_operation:\n> \t\tcur += pop_or_push;\n> \t}\n> \n> -\tfree(sorted.items);\n> -\n> \t/*\n> \t * We succeed if the loop terminates because 'cur' drops below\n> \t * zero. The other case is that we terminate because 'cur'\n> \t * reaches the end of the list, so we have a failure no matter\n> \t * which bundles we apply from the list.\n> \t */\n> +\tif (cur < 0) {\n> +\t\tstruct strbuf value = STRBUF_INIT;\n> +\t\tstrbuf_addf(&value, \"%\"PRIu64\"\", sorted.items[0]->creationToken);\n> +\t\tif (repo_config_set_multivar_gently(ctx.r,\n> +\t\t\t\t\t\t \"fetch.bundleCreationToken\",\n> +\t\t\t\t\t\t value.buf, NULL, 0))\n\nSet the new max bundle creation token if the value has been updated (if the\n'fetch.bundleCreationToken' was >= the first bundle's token, 'cur' is 0) in\nthe repository scope (like the cached bundle URI in patch 5 [1]). Looks\ngood.\n\n[1] https://lore.kernel.org/git/d9c6f50e4f218267c1e8da060ce5b190dc8a709c.1673037405.git.gitgitgadget@gmail.com/\n\n> +\t\t\twarning(_(\"failed to store maximum creation token\"));\n> +\n> +\t\tstrbuf_release(&value);\n> +\t}\n> +\n> +\tfree(sorted.items);\n> \treturn cur >= 0;\n> }\n> \n> diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\n> index 3f4d61a915c..0604d721f1b 100755\n> --- a/t/t5558-clone-bundle-uri.sh\n> +++ b/t/t5558-clone-bundle-uri.sh\n\nIt isn't exactly related to this patch, but it looks like the second \"clone\nbundle list (http, creationToken)\" never received any updates to\ndifferentiate it from the first test with that name (I noted the duplicate\nname in patch 4 [2]). Is it just a leftover duplicate?\n\n[1] https://lore.kernel.org/git/ede340d1-bce4-0c1d-7afb-4874a67d1803@github.com/\n\n> @@ -455,6 +455,7 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n> \t\t\"$HTTPD_URL/smart/fetch.git\" fetch-http-4 &&\n> \n> \ttest_cmp_config -C fetch-http-4 \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n> +\ttest_cmp_config -C fetch-http-4 1 fetch.bundlecreationtoken &&\n> \n> \t# The clone should copy two files: the list and bundle-1.\n> \ttest_bundle_downloaded bundle-list trace-clone.txt &&\n> @@ -479,6 +480,8 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n> \t\trefs/heads/left:refs/heads/left \\\n> \t\trefs/heads/right:refs/heads/right &&\n> \n> +\ttest_cmp_config -C fetch-http-4 2 fetch.bundlecreationtoken &&\n> +\n> \t# This fetch should copy two files: the list and bundle-2.\n> \ttest_bundle_downloaded bundle-list trace1.txt &&\n> \ttest_bundle_downloaded bundle-2.bundle trace1.txt &&\n\nVerifying the max creation token that's saved - nice!\n\n> @@ -492,6 +495,15 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n> \tEOF\n> \ttest_cmp expect refs &&\n> \n> +\t# No-op fetch\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace1b.txt\" \\\n> +\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n> +\t\trefs/heads/left:refs/heads/left \\\n> +\t\trefs/heads/right:refs/heads/right &&\n> +\ttest_bundle_downloaded bundle-list trace1b.txt &&\n> +\t! test_bundle_downloaded bundle-1.bundle trace1b.txt &&\n> +\t! test_bundle_downloaded bundle-2.bundle trace1b.txt &&\n\nNow we make sure we're not downloading that first bundle if it's already\nbeen unbundled.\n\n> +\n> \tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n> \t[bundle \"bundle-3\"]\n> \t\turi = bundle-3.bundle\n> @@ -508,6 +520,8 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n> \t\tgit -C fetch-http-4 fetch origin --no-tags \\\n> \t\trefs/heads/merge:refs/heads/merge &&\n> \n> +\ttest_cmp_config -C fetch-http-4 4 fetch.bundlecreationtoken &&\n> +\n> \t# This fetch should copy three files: the list, bundle-3, and bundle-4.\n> \ttest_bundle_downloaded bundle-list trace2.txt &&\n> \ttest_bundle_downloaded bundle-4.bundle trace2.txt &&\n> @@ -524,7 +538,16 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n> \trefs/bundles/left\n> \trefs/bundles/merge\n> \tEOF\n> -\ttest_cmp expect refs\n> +\ttest_cmp expect refs &&\n> +\n> +\t# No-op fetch\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace2b.txt\" \\\n> +\t\tgit -C fetch-http-4 fetch origin &&\n> +\ttest_bundle_downloaded bundle-list trace2b.txt &&\n> +\t! test_bundle_downloaded bundle-1.bundle trace2b.txt &&\n> +\t! test_bundle_downloaded bundle-2.bundle trace2b.txt &&\n> +\t! test_bundle_downloaded bundle-3.bundle trace2b.txt &&\n> +\t! test_bundle_downloaded bundle-4.bundle trace2b.txt\n\nAnd add another no-op fetch - this time, specifically making sure the\ndownloaded bundles have covered all objects fetched from 'origin'. As with\nprevious patches, these test updates are quite thorough; nice work!\n\n> '\n> \n> # Do not add tests here unless they use the HTTP server, as they will\n\n"},{"id":"470782","messageId":"1e787073-a1f6-58cd-ec5a-f99f2879624f@github.com","threadId":"59040","inReplyTo":"ede340d1-bce4-0c1d-7afb-4874a67d1803@github.com","subject":"Re: [PATCH 4/8] bundle-uri: download in creationToken order","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-20T14:56:50Z","receivedAt":"2023-01-20T14:56:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/19/2023 1:32 PM, Victoria Dye wrote:> Derrick Stolee via GitGitGadget wrote:\n>> +static int fetch_bundles_by_token(struct repository *r,\n>> +\t\t\t\t struct bundle_list *list)\n>> +{\n>> +\tint cur;\n>> +\tint pop_or_push = 0;\n>> +\tstruct bundle_list_context ctx = {\n>> +\t\t.r = r,\n>> +\t\t.list = list,\n>> +\t\t.mode = list->mode,\n>> +\t};\n>> +\tstruct sorted_bundle_list sorted = {\n>> +\t\t.alloc = hashmap_get_size(&list->bundles),\n>> +\t};\n>> +\n>> +\tALLOC_ARRAY(sorted.items, sorted.alloc);\n>> +\n>> +\tfor_all_bundles_in_list(list, insert_bundle, &sorted);\n>> +\n>> +\tQSORT(sorted.items, sorted.nr, compare_creation_token);\n>\n> So, at this point, 'sorted' is ordered by *decreasing* creation token? With\n> the loop below being somewhat complex, it would be nice to have a comment\n> mention that explicitly so readers have a clear understanding of the\n> \"initial state\" before entering the loop.\n\nThat's a good point, but also in my local version I have the following line:\n\n\tQSORT(bundles.items, bundles.nr, compare_creation_token_decreasing);\n\nThe comparison function was renamed based on Junio's feedback. After making\nthat change, this line is more self-documenting. Do you still think that it\nneeds a clarification comment if this rename occurs?\n\n>> +\t/*\n>> +\t * Use a stack-based approach to download the bundles and attempt\n>> +\t * to unbundle them in decreasing order by creation token. If we\n>> +\t * fail to unbundle (after a successful download) then move to the\n>> +\t * next non-downloaded bundle (push to the stack) and attempt\n>> +\t * downloading. Once we succeed in applying a bundle, move to the\n>> +\t * previous unapplied bundle (pop the stack) and attempt to unbundle\n>> +\t * it again.\n>> +\t *\n>> +\t * In the case of a fresh clone, we will likely download all of the\n>> +\t * bundles before successfully unbundling the oldest one, then the\n>> +\t * rest of the bundles unbundle successfully in increasing order\n>> +\t * of creationToken.\n>> +\t *\n>> +\t * If there are existing objects, then this process may terminate\n>> +\t * early when all required commits from \"new\" bundles exist in the\n>> +\t * repo's object store.\n>> +\t */\n>> +\tcur = 0;\n>> +\twhile (cur >= 0 && cur < sorted.nr) {\n>> +\t\tstruct remote_bundle_info *bundle = sorted.items[cur];\n>> +\t\tif (!bundle->file) {\n>> +\t\t\t/* Not downloaded yet. Try downloading. */\n>> +\t\t\tif (download_bundle_to_file(bundle, &ctx)) {\n>> +\t\t\t\t/* Failure. Push to the stack. */\n>> +\t\t\t\tpop_or_push = 1;\n>> +\t\t\t\tgoto stack_operation;\n>\n> Personally, I find the use of \"stack\" terminology more confusing than not.\n> 'sorted' isn't really a stack, it's a list with fixed contents being\n> traversed stepwise with 'cur'. For example, 'pop_or_push' being renamed to\n> 'move_direction' or 'step' something along those lines might more clearly\n> indicate what's actually happening with 'cur' & 'sorted'.\n\ns/pop_or_push/move_direction/ makes a lot of sense.\n\nI'll think about describing the strategy differently to avoid the \"stack\"\nlanguage. Mentally, I'm constructing a stack of \"downloaded but unable to\nunbundle bundles\", but they aren't actually arranged that way in any\nexplicit structure. Instead, they are just the bundles in the list that\nhave a file but haven't been unbundled.\n\n>> +\t\t\t}\n>> +\n>> +\t\t\t/* We expect bundles when using creationTokens. */\n>> +\t\t\tif (!is_bundle(bundle->file, 1)) {\n>> +\t\t\t\twarning(_(\"file downloaded from '%s' is not a bundle\"),\n>> +\t\t\t\t\tbundle->uri);\n>> +\t\t\t\tbreak;\n>> +\t\t\t}\n>> +\t\t}\n>> +\n>> +\t\tif (bundle->file && !bundle->unbundled) {\n>> +\t\t\t/*\n>> +\t\t\t * This was downloaded, but not successfully\n>> +\t\t\t * unbundled. Try unbundling again.\n>> +\t\t\t */\n>> +\t\t\tif (unbundle_from_file(ctx.r, bundle->file)) {\n>> +\t\t\t\t/* Failed to unbundle. Push to stack. */\n>> +\t\t\t\tpop_or_push = 1;\n>> +\t\t\t} else {\n>> +\t\t\t\t/* Succeeded in unbundle. Pop stack. */\n>> +\t\t\t\tpop_or_push = -1;\n>> +\t\t\t}\n>> +\t\t}\n>> +\n>> +\t\t/*\n>> +\t\t * Else case: downloaded and unbundled successfully.\n>> +\t\t * Skip this by moving in the same direction as the\n>> +\t\t * previous step.\n>> +\t\t */\n>> +\n>> +stack_operation:\n>> +\t\t/* Move in the specified direction and repeat. */\n>> +\t\tcur += pop_or_push;\n>> +\t}\n>\n> After reading through this loop, I generally understood *what* its doing,\n> but didn't really follow *why* the download & unbundling is done like this.\n\nThe commit message should be updated to point to refer to the previously-\nadded test setup in t5558:\n\n# To get interesting tests for bundle lists, we need to construct a\n# somewhat-interesting commit history.\n#\n# ---------------- bundle-4\n#\n# 4\n# / \\\n# ----|---|------- bundle-3\n# | |\n# | 3\n# | |\n# ----|---|------- bundle-2\n# | |\n# 2 |\n# | |\n# ----|---|------- bundle-1\n# \\ /\n# 1\n# |\n# (previous commits)\n\nAnd then this can be used to motivate the algorithm. Suppose we have\nalready downloaded commit 1 through a previous fetch. We try to download\nbundle-4 first, but it can't apply because it requires commits that are\nin bundle-3 _and_ bundle-2, but the client doesn't know which bundles\ncontain those commits. Downloading bundle-3 successfully unbundles, so a\nnaive algorithm would think we are \"done\" and expect to unbundle bundle-4.\nHowever, that unbundling fails, so we go deeper into the list to download\nbundle-2. That succeeds, and then retrying bundle-4 succeeds.\n\n> I needed to refer back to the design doc\n> ('Documentation/technical/bundle-uri.txt') to understand some basic\n> assumptions about bundles:\n>\n> - A new bundle's creation token should always be strictly greater than the\n> previous newest bundle's creation token. I don't see any special handling\n> for equal creation tokens, so my assumption is that the sorting of the\n> list arbitrarily assigns one to be \"greater\" and it's dealt with that way.\n\nYes, the bundle provider should not have equal values unless the bundles are\ntruly independent. That could be clarified in that doc.\n\n> - The bundle with the lowest creation token should always be unbundleable,\n> since it contains all objects in an initial clone.\n\nYes, at least it should not have any required commits.\n\n> I do still have some questions, though:\n>\n> - Why would 'unbundle_from_file()' fail? From context clues, I'm guessing it\n> fails if it has some unreachable objects (as in an incremental bundle), or\n> if it's corrupted somehow.\n\nYou are correct. We assume that the data is well-formed and so the problem\nmust be due to required commits not already present in the local object store.\n\n> - Why would 'download_bundle_to_file()' to fail? Unlike\n> 'unbundle_from_file()', it looks like that represents an unexpected error.\n\nYes, that could fail for network issues such as a server error or other\nnetwork failure. In such cases, the client should expect that we will not\nbe able to download that bundle for the process's lifetime. We may be able\nto opportunistically download other bundles, but we will rely on the Git\nprotocol to get the objects if the bundles fail.\n\nThese failure conditions are not tested deeply (there are some tests from\nearlier series that test the behavior, but there is room for improvement).\n\n> Also - it seems like one of the assumptions here is that, if a bundle can't\n> be downloaded & unbundled, no bundle with a higher creation token can be\n> successfully unbundled ('download_bundle_to_file()' sets 'pop_or_push' to\n> '1', which will cause the loop to ignore all higher-token bundles and return\n> a nonzero value from the function).\n>\n> I don't think that assumption is necessarily true, though. Suppose you have\n> a \"base\" bundle 100 and incremental bundles 101 and 102. 101 has all objects\n> from a new branch A, and 102 has all objects from a newer branch B (not\n> based on any objects in A). In this case, 102 could be unbundled even if 101\n> is corrupted/can't be downloaded, but we'd run into issues if we store 102\n> as the \"latest unbundled creation token\" (because it implies that 101 was\n> unbundled).\n\nYou are correct. bundle-3 can be unbundled even if bundle-2 fails in the\ntest example above.\n\n> Is there any benefit to trying to unbundle those higher bundles *without*\n> advancing the \"latest creation token\"? E.g. in my example, unbundle 102 but\n> store '100' as the latest creation token?\n\nI will need to think more about this.\n\nGenerally, most repositories that care about this will not have independent\nbundles because between every bundle creation step the default branch will\nadvance. (Of course, exceptions can still occur, such as over weekends.)\nThus, the latest bundle will have a required commit that only exists in the\nprevious bundle. This algorithm and its error conditions are then looking\nfor ways to recover when that is not the case.\n\nWhen a bundle fails to download, my gut feeling is that it is unlikely that\nit was completely independent of a bundle with higher creationToken. However,\nwe have already downloaded that bundle and it is a very low cost to attempt\nan unbundling of it.\n\nThe tricky part is that we want to avoid downloading _all_ the bundles just\nbecause one is failing to unbundle. If a failed download would cause the top\nbundle from unbundling, we don't want to go through the whole list of bundles\neven though they unbundle without issue. I'm thinking specifically about the\nincremental fetch case, where we don't want to blow up to a full clone worth\nof downloads.\n\nThis deserves a little more attention, so I'll think more on it and get\nback to you.\n\n>> \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n>> -\tgit -C clone-list-http-2 cat-file --batch-check > +\tgit -C clone-list-http-2 cat-file --batch-check > +\n>> +\tfor b in 1 2 3 4\n>> +\tdo\n>> +\t\ttest_bundle_downloaded bundle-$b.bundle trace-clone.txt ||\n>> +\t\t\treturn 1\n>> +\tdone\n>\n> If I understand correctly, these added conditions would have passed even if\n> they were added when the test was initially created in patch 1, but they're\n> added here to tie them to the implementation of the creationToken heuristic?\n> Seems reasonable.\n\nThey probably should have been added in patch 1 to be clear that behavior\nis not changing here.\n\n>> +'\n>> +\n>> +test_expect_success 'clone bundle list (http, creationToken)' '\n>\n> This new test has the same name as the one above it - how does it differ\n> from that one? Whatever the difference is, can that be noted somehow in the\n> title or a comment?\n\nThe title should change, pointing out that the bundle list is truncated\nand the rest of the clone is being fetched over the Git protocol. It will\nbe expanded with fetches later, I think, but it should be better motivated\nin this patch, even if that is so.\n\n>> +# Usage: test_bundle_downloaded \n>> +test_bundle_downloaded () {\n>> +\tcat >pattern <<-EOF &&\n>> +\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/$1.bundle\"\\]\n>> +\tEOF\n>> +\tgrep -f pattern \"$2\"\n>> +}\n>\n> This function is the same as the one created in 't5558'. Should it be moved\n> to 'lib-bundle.sh' or 'test-lib.sh' to avoid duplicate code?\n\nIt's slightly different, but that is just because we are using the advertisement\nand thus we never download a bundle-list and always download .bundle files. That\nis not an important distinction and I expect to replace it with the\ntest_remote_https_urls() helper discussed in an earlier response.\n\nThanks,\n-Stolee\n"},{"id":"470789","messageId":"75d25afd-f941-3090-5cac-f1a4ca7d9827@github.com","threadId":"59040","inReplyTo":"61a830f0-a8d8-5a8a-e952-db213d571352@github.com","subject":"Re: [PATCH 5/8] clone: set fetch.bundleURI if appropriate","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-20T15:42:30Z","receivedAt":"2023-01-20T15:42:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/19/2023 2:42 PM, Victoria Dye wrote:\n> Derrick Stolee via GitGitGadget wrote:\n>> diff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt\n>> index cd65d236b43..4f796218aab 100644\n>> --- a/Documentation/config/fetch.txt\n>> +++ b/Documentation/config/fetch.txt\n>> @@ -96,3 +96,11 @@ fetch.writeCommitGraph::\n>> \tmerge and the write may take longer. Having an updated commit-graph\n>> \tfile helps performance of many Git commands, including `git merge-base`,\n>> \t`git push -f`, and `git log --graph`. Defaults to false.\n>> +\n>> +fetch.bundleURI::\n>> +\tThis value stores a URI for fetching Git object data from a bundle URI\n>> +\tbefore performing an incremental fetch from the origin Git server. If\n>> +\tthe value is `` then running `git fetch ` is equivalent to\n>> +\tfirst running `git fetch --bundle-uri=` immediately before\n>> +\t`git fetch `. See details of the `--bundle-uri` option in\n>> +\tlinkgit:git-fetch[1].\n>\n> Since it's not mentioned from this or any other user-facing documentation\n> (AFAICT), could you note that this value is set automatically by 'git clone'\n> iff '--bundle-uri' is specified *and* 'bundle.heuristic' is set for the\n> initially downloaded bundle list?\n\nCan do.\n\n> It would also be nice to make note of that behavior in the documentation of\n> the '--bundle-uri' option in 'Documentation/git-clone.txt', since command\n> documentation in general seems to be more popular/visible to users than\n> config docs.\n\nYes. I also thought that I had updated this documentation to not refer\nto 'git fetch --bundle-uri', which doesn't exist anymore since an earlier\nRFC version. I'll be sure to update that, too.\n\n>> \tstrvec_push(&transport_ls_refs_options.ref_prefixes, \"HEAD\");\n>> diff --git a/bundle-uri.c b/bundle-uri.c\n>> index b30c85ba6f2..1dbbbb980eb 100644\n>> --- a/bundle-uri.c\n>> +++ b/bundle-uri.c\n>> @@ -594,9 +594,10 @@ static int fetch_bundle_list_in_config_format(struct repository *r,\n>> \t * it advertises are expected to be bundles, not nested lists.\n>> \t * We can drop 'global_list' and 'depth'.\n>> \t */\n>> -\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN)\n>> +\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN) {\n>> \t\tresult = fetch_bundles_by_token(r, &list_from_bundle);\n>> -\telse if ((result = download_bundle_list(r, &list_from_bundle,\n>> +\t\tglobal_list->heuristic = BUNDLE_HEURISTIC_CREATIONTOKEN;\n>\n> If the 'heuristic' field already existed and was being used to apply\n> bundles, why wasn't 'global_list->heuristic' already being set? Before this\n> patch, was the 'global_list->heuristic' field not accurately reflecting the\n> heuristic type of a given bundle list?\n>\n> If so, I think it'd make sense to move this section to patch 4 [1], since\n> that's when the heuristic is first applied to the bundle list.\n>\n> [1] https://lore.kernel.org/git/57c0174d3752fb61a05e0653de9d3057616ed16a.1673037405.git.gitgitgadget@gmail.com/\n\nCan do.\n\nThanks,\n-Stolee\n"},{"id":"470790","messageId":"4e2ac966-0f45-8018-ff8f-3831ea0c3c2e@github.com","threadId":"59040","inReplyTo":"58269fe6-ebe1-7b12-4cd9-2110a94543e5@github.com","subject":"Re: [PATCH 7/8] fetch: fetch from an external bundle URI","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-20T15:47:02Z","receivedAt":"2023-01-20T15:47:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/19/2023 3:34 PM, Victoria Dye wrote:\n> Derrick Stolee via GitGitGadget wrote:\n\n>> +\tif (!git_config_get_string_tmp(\"fetch.bundleuri\", &bundle_uri) &&\n>> +\t !starts_with(bundle_uri, \"remote:\")) {\n> \n> Maybe a silly question, by why would the bundle URI start with 'remote:'\n> (and why do we silently skip fetching from the URI in that case)?\n\nThanks for catching this. I originally was going to include fetching from\nlists advertised by a Git remote, and use the same `fetch.bundleURI` config.\nHowever, it makes more sense to make a `remote..bundles` config\ninstead, so I dropped that functionality from this series. I forgot to remove\nthis `remote:` case, but will do so in v2.\n\nI've locally fixed the \"objects\" typo you pointed out, too.\n\nThanks,\n-Stolee\n"},{"id":"470792","messageId":"7853f7f3-d999-5b39-ab6f-c43055c65085@github.com","threadId":"59040","inReplyTo":"f7a777f0-3f37-86a1-f520-1ab6853b9ed1@github.com","subject":"Re: [PATCH 8/8] bundle-uri: store fetch.bundleCreationToken","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-20T15:53:12Z","receivedAt":"2023-01-20T15:53:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/19/2023 5:24 PM, Victoria Dye wrote:\n> Derrick Stolee via GitGitGadget wrote:\n\n>> +\t/*\n>> +\t * If fetch.bundleCreationToken exists, parses to a uint64t, and\n>> +\t * is not strictly smaller than the maximum creation token in the\n>> +\t * bundle list, then do not download any bundles.\n>> +\t */\n>> +\tif (!repo_config_get_value(r,\n>> +\t\t\t\t \"fetch.bundlecreationtoken\",\n>> +\t\t\t\t &creationTokenStr) &&\n>> +\t sscanf(creationTokenStr, \"%\"PRIu64, &maxCreationToken) == 1 &&\n>> +\t sorted.items[0]->creationToken <= maxCreationToken) {\n>> +\t\tfree(sorted.items);\n>> +\t\treturn 0;\n>> +\t}\n>\n> And here, we exit if the cached creation token is greater than or equal to\n> the highest advertised token. Overall, this seems pretty safe:\n>\n> - If the value is (somehow) manually updated to something larger than it\n> should be, objects will be fetched from the server that could have\n> otherwise come from a bundle. Not ideal, but no worse than a regular\n> fetch.\n> - If the value is too small or invalid (i.e., not an unsigned integer),\n> we'll download the first bundle, unbundle it, then overwrite the invalid\n> 'fetch.bundlecreationtoken' with a new valid one.\n>\n> The latter is self-correcting, but should the former be documented\n> somewhere? For example, if someone changes which bundle server they're using\n> with a repo, the creation token numbering scheme might be completely\n> different. In that case, a user would probably want to \"reset\" the\n> 'fetch.bundlecreationtoken' and start fresh with a new bundle list (even if\n> the recommended method is \"manually delete the config key from your repo\").\n\nI can update the config documentation to include this.\n\n\n>> diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\n>> index 3f4d61a915c..0604d721f1b 100755\n>> --- a/t/t5558-clone-bundle-uri.sh\n>> +++ b/t/t5558-clone-bundle-uri.sh\n>\n> It isn't exactly related to this patch, but it looks like the second \"clone\n> bundle list (http, creationToken)\" never received any updates to\n> differentiate it from the first test with that name (I noted the duplicate\n> name in patch 4 [2]). Is it just a leftover duplicate?\n>\n> [1] https://lore.kernel.org/git/ede340d1-bce4-0c1d-7afb-4874a67d1803@github.com/\n\nI'll be sure to follow up and make these test changes be of higher value,\navoiding these confusions.\n\nThanks,\n-Stolee\n"},{"id":"470913","messageId":"b3828725bc8f8887b9b4777a0e3d84224a427f31.1674487310.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v2.git.1674487310.gitgitgadget@gmail.com","subject":"[PATCH v2 01/10] bundle: optionally skip reachability walk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-23T15:21:41Z","receivedAt":"2023-01-23T15:22:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen unbundling a bundle, the verify_bundle() method checks two things\nwith regards to the prerequisite commits:\n\n 1. Those commits are in the object store, and\n 2. Those commits are reachable from refs.\n\nDuring testing of the bundle URI feature, where multiple bundles are\nunbundled in the same process, the ref store did not appear to be\nrefreshing with the new refs/bundles/* references added within that\nprocess. This caused the second half -- the reachability walk -- report\nthat some commits were not present, despite actually being present.\n\nOne way to attempt to fix this would be to create a way to force-refresh\nthe ref state. That would correct this for these cases where the\nrefs/bundles/* references have been updated. However, this still is an\nexpensive operation in a repository with many references.\n\nInstead, optionally allow callers to skip this portion by instead just\nchecking for presence within the object store. Use this when unbundling\nin bundle-uri.c.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 8 +++++++-\n bundle.c | 3 ++-\n bundle.h | 1 +\n 3 files changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 36268dda172..2f079f713cf 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -322,9 +322,15 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t * Skip the reachability walk here, since we will be adding\n \t * a reachable ref pointing to the new tips, which will reach\n \t * the prerequisite commits.\n+\t *\n+\t * Since multiple iterations of unbundle_from_file() can create\n+\t * new commits in the object store that are not reachable from\n+\t * the current cached state of the ref store, skip the reachability\n+\t * walk and move forward as long as the objects are present in the\n+\t * object store.\n \t */\n \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n-\t\t\t VERIFY_BUNDLE_QUIET)))\n+\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_SKIP_REACHABLE)))\n \t\treturn 1;\n \n \t/*\ndiff --git a/bundle.c b/bundle.c\nindex 4ef7256aa11..b51974f0806 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -223,7 +223,8 @@ int verify_bundle(struct repository *r,\n \t\t\terror(\"%s\", message);\n \t\terror(\"%s %s\", oid_to_hex(oid), name);\n \t}\n-\tif (revs.pending.nr != p->nr)\n+\tif (revs.pending.nr != p->nr ||\n+\t (flags & VERIFY_BUNDLE_SKIP_REACHABLE))\n \t\tgoto cleanup;\n \treq_nr = revs.pending.nr;\n \tsetup_revisions(2, argv, &revs, NULL);\ndiff --git a/bundle.h b/bundle.h\nindex 9f2bd733a6a..24c30e5f74a 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -34,6 +34,7 @@ int create_bundle(struct repository *r, const char *path,\n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n \tVERIFY_BUNDLE_QUIET = (1 << 1),\n+\tVERIFY_BUNDLE_SKIP_REACHABLE = (1 << 2),\n };\n \n int verify_bundle(struct repository *r, struct bundle_header *header,\n-- \ngitgitgadget\n\n"},{"id":"470914","messageId":"427aff4d5e5c85b601f43af8b664515380e11453.1674487310.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v2.git.1674487310.gitgitgadget@gmail.com","subject":"[PATCH v2 02/10] t5558: add tests for creationToken heuristic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-23T15:21:42Z","receivedAt":"2023-01-23T15:22:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs documented in the bundle URI design doc in 2da14fad8fe (docs:\ndocument bundle URI standard, 2022-08-09), the 'creationToken' member of\na bundle URI allows a bundle provider to specify a total order on the\nbundles.\n\nFuture changes will allow the Git client to understand these members and\nmodify its behavior around downloading the bundles in that order. In the\nmeantime, create tests that add creation tokens to the bundle list. For\nnow, the Git client correctly ignores these unknown keys.\n\nCreate a new test helper function, test_remote_https_urls, which filters\nGIT_TRACE2_EVENT output to extract a list of URLs passed to\ngit-remote-https child processes. This can be used to verify the order\nof these requests as we implement the creationToken heuristic. For now,\nwe need to sort the actual output since the current client does not have\na well-defined order that it applies to the bundles.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5558-clone-bundle-uri.sh | 69 +++++++++++++++++++++++++++++++++++--\n t/test-lib-functions.sh | 8 +++++\n 2 files changed, 75 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 9155f31fa2c..474432c8ace 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -285,6 +285,8 @@ test_expect_success 'clone HTTP bundle' '\n '\n \n test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\n \tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n \tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n \t[bundle]\n@@ -304,12 +306,26 @@ test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n \t\turi = $HTTPD_URL/bundle-4.bundle\n \tEOF\n \n-\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" \\\n+\t\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n \t\tclone-from clone-list-http 2>err &&\n \t! grep \"Repository lacks these prerequisite commits\" err &&\n \n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n-\tgit -C clone-list-http cat-file --batch-check expect <<-EOF &&\n+\t$HTTPD_URL/bundle-1.bundle\n+\t$HTTPD_URL/bundle-2.bundle\n+\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/bundle-4.bundle\n+\t$HTTPD_URL/bundle-list\n+\tEOF\n+\n+\t# Sort the list, since the order is not well-defined\n+\t# without a heuristic.\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual\n '\n \n test_expect_success 'clone bundle list (HTTP, any mode)' '\n@@ -350,6 +366,55 @@ test_expect_success 'clone bundle list (HTTP, any mode)' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'clone bundle list (http, creationToken)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\n+\tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" git \\\n+\t\tclone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\t\"$HTTPD_URL/smart/fetch.git\" clone-list-http-2 &&\n+\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-list-http-2 cat-file --batch-check expect <<-EOF &&\n+\t$HTTPD_URL/bundle-1.bundle\n+\t$HTTPD_URL/bundle-2.bundle\n+\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/bundle-4.bundle\n+\t$HTTPD_URL/bundle-list\n+\tEOF\n+\n+\t# Since the creationToken heuristic is not yet understood by the\n+\t# client, the order cannot be verified at this moment. Sort the\n+\t# list for consistent results.\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual\n+'\n+\n # Do not add tests here unless they use the HTTP server, as they will\n # not run unless the HTTP dependencies exist.\n \ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex f036c4d3003..ace542f4226 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1833,6 +1833,14 @@ test_region () {\n \treturn 0\n }\n \n+# Given a GIT_TRACE2_EVENT log over stdin, writes to stdout a list of URLs\n+# sent to git-remote-https child processes.\n+test_remote_https_urls() {\n+\tgrep -e '\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\".*\"\\]' |\n+\t\tsed -e 's/{\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"//g' \\\n+\t\t -e 's/\"\\]}//g'\n+}\n+\n # Print the destination of symlink(s) provided as arguments. Basically\n # the same as the readlink command, but it's not available everywhere.\n test_readlink () {\n-- \ngitgitgadget\n\n"},{"id":"470915","messageId":"12efa228d04c9379ec0598c974045ed851b068d9.1674487310.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v2.git.1674487310.gitgitgadget@gmail.com","subject":"[PATCH v2 04/10] bundle-uri: parse bundle..creationToken values","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-23T15:21:44Z","receivedAt":"2023-01-23T15:22:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change taught Git to parse the bundle.heuristic value,\nespecially when its value is \"creationToken\". Now, teach Git to parse\nthe bundle..creationToken values on each bundle in a bundle list.\n\nBefore implementing any logic based on creationToken values for the\ncreationToken heuristic, parse and print these values for testing\npurposes.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 10 ++++++++++\n bundle-uri.h | 6 ++++++\n t/t5750-bundle-uri-parse.sh | 18 ++++++++++++++++++\n 3 files changed, 34 insertions(+)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 0d64b1d84ba..f46ab5c1743 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -83,6 +83,9 @@ static int summarize_bundle(struct remote_bundle_info *info, void *data)\n \tFILE *fp = data;\n \tfprintf(fp, \"[bundle \\\"%s\\\"]\\n\", info->id);\n \tfprintf(fp, \"\\turi = %s\\n\", info->uri);\n+\n+\tif (info->creationToken)\n+\t\tfprintf(fp, \"\\tcreationToken = %\"PRIu64\"\\n\", info->creationToken);\n \treturn 0;\n }\n \n@@ -203,6 +206,13 @@ static int bundle_list_update(const char *key, const char *value,\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(subkey, \"creationtoken\")) {\n+\t\tif (sscanf(value, \"%\"PRIu64, &bundle->creationToken) != 1)\n+\t\t\twarning(_(\"could not parse bundle list key %s with value '%s'\"),\n+\t\t\t\t\"creationToken\", value);\n+\t\treturn 0;\n+\t}\n+\n \t/*\n \t * At this point, we ignore any information that we don't\n \t * understand, assuming it to be hints for a heuristic the client\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 2e44a50a90b..ef32840bfa6 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -42,6 +42,12 @@ struct remote_bundle_info {\n \t * this boolean is true.\n \t */\n \tunsigned unbundled:1;\n+\n+\t/**\n+\t * If the bundle is part of a list with the creationToken\n+\t * heuristic, then we use this member for sorting the bundles.\n+\t */\n+\tuint64_t creationToken;\n };\n \n #define REMOTE_BUNDLE_INFO_INIT { 0 }\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex 6fc92a9c0d4..81bdf58b944 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -258,10 +258,13 @@ test_expect_success 'parse config format: creationToken heuristic' '\n \t\theuristic = creationToken\n \t[bundle \"one\"]\n \t\turi = http://example.com/bundle.bdl\n+\t\tcreationToken = 123456\n \t[bundle \"two\"]\n \t\turi = https://example.com/bundle.bdl\n+\t\tcreationToken = 12345678901234567890\n \t[bundle \"three\"]\n \t\turi = file:///usr/share/git/bundle.bdl\n+\t\tcreationToken = 1\n \tEOF\n \n \ttest-tool bundle-uri parse-config expect >actual 2>err &&\n@@ -269,4 +272,19 @@ test_expect_success 'parse config format: creationToken heuristic' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format edge cases: creationToken heuristic' '\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t\tcreationToken = bogus\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config expect >actual 2>err &&\n+\tgrep \"could not parse bundle list key creationToken with value '\\''bogus'\\''\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"470916","messageId":"f6f8197c9cc86a7b3b986953e4722b7c05241eba.1674487310.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v2.git.1674487310.gitgitgadget@gmail.com","subject":"[PATCH v2 03/10] bundle-uri: parse bundle.heuristic=creationToken","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-23T15:21:43Z","receivedAt":"2023-01-23T15:22:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe bundle.heuristic value communicates that the bundle list is\norganized to make use of the bundle..creationToken values that may\nbe provided in the bundle list. Those values will create a total order\non the bundles, allowing the Git client to download them in a specific\norder and even remember previously-downloaded bundles by storing the\nmaximum creation token value.\n\nBefore implementing any logic that parses or uses the\nbundle..creationToken values, teach Git to parse the\nbundle.heuristic value from a bundle list. We can use 'test-tool\nbundle-uri' to print the heuristic value and verify that the parsing\nworks correctly.\n\nAs an extra precaution, create the internal 'heuristics' array to be a\nlist of (enum, string) pairs so we can iterate through the array entries\ncarefully, regardless of the enum values.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/bundle.txt | 7 +++++++\n bundle-uri.c | 34 +++++++++++++++++++++++++++++++++\n bundle-uri.h | 14 ++++++++++++++\n t/t5750-bundle-uri-parse.sh | 19 ++++++++++++++++++\n 4 files changed, 74 insertions(+)\n\ndiff --git a/Documentation/config/bundle.txt b/Documentation/config/bundle.txt\nindex daa21eb674a..3faae386853 100644\n--- a/Documentation/config/bundle.txt\n+++ b/Documentation/config/bundle.txt\n@@ -15,6 +15,13 @@ bundle.mode::\n \tcomplete understanding of the bundled information (`all`) or if any one\n \tof the listed bundle URIs is sufficient (`any`).\n \n+bundle.heuristic::\n+\tIf this string-valued key exists, then the bundle list is designed to\n+\twork well with incremental `git fetch` commands. The heuristic signals\n+\tthat there are additional keys available for each bundle that help\n+\tdetermine which subset of bundles the client should download. The\n+\tonly value currently understood is `creationToken`.\n+\n bundle..*::\n \tThe `bundle..*` keys are used to describe a single item in the\n \tbundle list, grouped under `` for identification purposes.\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 2f079f713cf..0d64b1d84ba 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -9,6 +9,14 @@\n #include \"config.h\"\n #include \"remote.h\"\n \n+static struct {\n+\tenum bundle_list_heuristic heuristic;\n+\tconst char *name;\n+} heuristics[BUNDLE_HEURISTIC__COUNT] = {\n+\t{ BUNDLE_HEURISTIC_NONE, \"\"},\n+\t{ BUNDLE_HEURISTIC_CREATIONTOKEN, \"creationToken\" },\n+};\n+\n static int compare_bundles(const void *hashmap_cmp_fn_data,\n \t\t\t const struct hashmap_entry *he1,\n \t\t\t const struct hashmap_entry *he2,\n@@ -100,6 +108,17 @@ void print_bundle_list(FILE *fp, struct bundle_list *list)\n \tfprintf(fp, \"\\tversion = %d\\n\", list->version);\n \tfprintf(fp, \"\\tmode = %s\\n\", mode);\n \n+\tif (list->heuristic) {\n+\t\tint i;\n+\t\tfor (i = 0; i < BUNDLE_HEURISTIC__COUNT; i++) {\n+\t\t\tif (heuristics[i].heuristic == list->heuristic) {\n+\t\t\t\tprintf(\"\\theuristic = %s\\n\",\n+\t\t\t\t heuristics[list->heuristic].name);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n \tfor_all_bundles_in_list(list, summarize_bundle, fp);\n }\n \n@@ -142,6 +161,21 @@ static int bundle_list_update(const char *key, const char *value,\n \t\t\treturn 0;\n \t\t}\n \n+\t\tif (!strcmp(subkey, \"heuristic\")) {\n+\t\t\tint i;\n+\t\t\tfor (i = 0; i < BUNDLE_HEURISTIC__COUNT; i++) {\n+\t\t\t\tif (heuristics[i].heuristic &&\n+\t\t\t\t heuristics[i].name &&\n+\t\t\t\t !strcmp(value, heuristics[i].name)) {\n+\t\t\t\t\tlist->heuristic = heuristics[i].heuristic;\n+\t\t\t\t\treturn 0;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\t/* Ignore unknown heuristics. */\n+\t\t\treturn 0;\n+\t\t}\n+\n \t\t/* Ignore other unknown global keys. */\n \t\treturn 0;\n \t}\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex d5e89f1671c..2e44a50a90b 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -52,6 +52,14 @@ enum bundle_list_mode {\n \tBUNDLE_MODE_ANY\n };\n \n+enum bundle_list_heuristic {\n+\tBUNDLE_HEURISTIC_NONE = 0,\n+\tBUNDLE_HEURISTIC_CREATIONTOKEN,\n+\n+\t/* Must be last. */\n+\tBUNDLE_HEURISTIC__COUNT\n+};\n+\n /**\n * A bundle_list contains an unordered set of remote_bundle_info structs,\n * as well as information about the bundle listing, such as version and\n@@ -75,6 +83,12 @@ struct bundle_list {\n \t * advertised by the bundle list at that location.\n \t */\n \tchar *baseURI;\n+\n+\t/**\n+\t * A list can have a heuristic, which helps reduce the number of\n+\t * downloaded bundles.\n+\t */\n+\tenum bundle_list_heuristic heuristic;\n };\n \n void init_bundle_list(struct bundle_list *list);\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex 7b4f930e532..6fc92a9c0d4 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -250,4 +250,23 @@ test_expect_success 'parse config format edge cases: empty key or value' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format: creationToken heuristic' '\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config expect >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"470917","messageId":"7cfaa3c518cbedb65c585cc02015bb21ae24e9fa.1674487310.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v2.git.1674487310.gitgitgadget@gmail.com","subject":"[PATCH v2 05/10] bundle-uri: download in creationToken order","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-23T15:21:45Z","receivedAt":"2023-01-23T15:22:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe creationToken heuristic provides an ordering on the bundles\nadvertised by a bundle list. Teach the Git client to download bundles\ndifferently when this heuristic is advertised.\n\nThe bundles in the list are sorted by their advertised creationToken\nvalues, then downloaded in decreasing order. This avoids the previous\nstrategy of downloading bundles in an arbitrary order and attempting\nto apply them (likely failing in the case of required commits) until\ndiscovering the order through attempted unbundling.\n\nDuring a fresh 'git clone', it may make sense to download the bundles in\nincreasing order, since that would prevent the need to attempt\nunbundling a bundle with required commits that do not exist in our empty\nobject store. The cost of testing an unbundle is quite low, and instead\nthe chosen order is optimizing for a future bundle download during a\n'git fetch' operation with a non-empty object store.\n\nSince the Git client continues fetching from the Git remote after\ndownloading and unbundling bundles, the client's object store can be\nahead of the bundle provider's object store. The next time it attempts\nto download from the bundle list, it makes most sense to download only\nthe most-recent bundles until all tips successfully unbundle. The\nstrategy implemented here provides that short-circuit where the client\ndownloads a minimal set of bundles.\n\nHowever, we are not satisfied by the naive approach of downloading\nbundles until one successfully unbundles, expecting the earlier bundles\nto successfully unbundle now. The example repository in t5558\ndemonstrates this well:\n\n ---------------- bundle-4\n\n 4\n / \\\n ----|---|------- bundle-3\n | |\n | 3\n | |\n ----|---|------- bundle-2\n | |\n 2 |\n | |\n ----|---|------- bundle-1\n \\ /\n 1\n |\n (previous commits)\n\nIn this repository, if we already have the objects for bundle-1 and then\ntry to fetch from this list, the naive approach will fail. bundle-4\nrequires both bundle-3 and bundle-2, though bundle-3 will successfully\nunbundle without bundle-2. Thus, the algorithm needs to keep this in\nmind.\n\nA later implementation detail will store the maximum creationToken seen\nduring such a bundle download, and the client will avoid downloading a\nbundle unless its creationToken is strictly greater than that stored\nvalue. For now, if the client seeks to download from an identical\nbundle list since its previous download, it will download the\nmost-recent bundle then stop since its required commits are already in\nthe object store.\n\nAdd tests that exercise this behavior, but we will expand upon these\ntests when incremental downloads during 'git fetch' make use of\ncreationToken values.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 156 +++++++++++++++++++++++++++++++++++-\n t/t5558-clone-bundle-uri.sh | 40 +++++++--\n t/t5601-clone.sh | 46 +++++++++++\n 3 files changed, 233 insertions(+), 9 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex f46ab5c1743..39acd856fb9 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -453,6 +453,139 @@ static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data\n \treturn 0;\n }\n \n+struct bundles_for_sorting {\n+\tstruct remote_bundle_info **items;\n+\tsize_t alloc;\n+\tsize_t nr;\n+};\n+\n+static int append_bundle(struct remote_bundle_info *bundle, void *data)\n+{\n+\tstruct bundles_for_sorting *list = data;\n+\tlist->items[list->nr++] = bundle;\n+\treturn 0;\n+}\n+\n+/**\n+ * For use in QSORT() to get a list sorted by creationToken\n+ * in decreasing order.\n+ */\n+static int compare_creation_token_decreasing(const void *va, const void *vb)\n+{\n+\tconst struct remote_bundle_info * const *a = va;\n+\tconst struct remote_bundle_info * const *b = vb;\n+\n+\tif ((*a)->creationToken > (*b)->creationToken)\n+\t\treturn -1;\n+\tif ((*a)->creationToken < (*b)->creationToken)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int fetch_bundles_by_token(struct repository *r,\n+\t\t\t\t struct bundle_list *list)\n+{\n+\tint cur;\n+\tint move_direction = 0;\n+\tstruct bundle_list_context ctx = {\n+\t\t.r = r,\n+\t\t.list = list,\n+\t\t.mode = list->mode,\n+\t};\n+\tstruct bundles_for_sorting bundles = {\n+\t\t.alloc = hashmap_get_size(&list->bundles),\n+\t};\n+\n+\tALLOC_ARRAY(bundles.items, bundles.alloc);\n+\n+\tfor_all_bundles_in_list(list, append_bundle, &bundles);\n+\n+\tQSORT(bundles.items, bundles.nr, compare_creation_token_decreasing);\n+\n+\t/*\n+\t * Attempt to download and unbundle the minimum number of bundles by\n+\t * creationToken in decreasing order. If we fail to unbundle (after\n+\t * a successful download) then move to the next non-downloaded bundle\n+\t * and attempt downloading. Once we succeed in applying a bundle,\n+\t * move to the previous unapplied bundle and attempt to unbundle it\n+\t * again.\n+\t *\n+\t * In the case of a fresh clone, we will likely download all of the\n+\t * bundles before successfully unbundling the oldest one, then the\n+\t * rest of the bundles unbundle successfully in increasing order\n+\t * of creationToken.\n+\t *\n+\t * If there are existing objects, then this process may terminate\n+\t * early when all required commits from \"new\" bundles exist in the\n+\t * repo's object store.\n+\t */\n+\tcur = 0;\n+\twhile (cur >= 0 && cur < bundles.nr) {\n+\t\tstruct remote_bundle_info *bundle = bundles.items[cur];\n+\t\tif (!bundle->file) {\n+\t\t\t/*\n+\t\t\t * Not downloaded yet. Try downloading.\n+\t\t\t *\n+\t\t\t * Note that bundle->file is non-NULL if a download\n+\t\t\t * was attempted, even if it failed to download.\n+\t\t\t */\n+\t\t\tif (fetch_bundle_uri_internal(ctx.r, bundle, ctx.depth + 1, ctx.list)) {\n+\t\t\t\t/* Mark as unbundled so we do not retry. */\n+\t\t\t\tbundle->unbundled = 1;\n+\n+\t\t\t\t/* Try looking deeper in the list. */\n+\t\t\t\tmove_direction = 1;\n+\t\t\t\tgoto stack_operation;\n+\t\t\t}\n+\n+\t\t\t/* We expect bundles when using creationTokens. */\n+\t\t\tif (!is_bundle(bundle->file, 1)) {\n+\t\t\t\twarning(_(\"file downloaded from '%s' is not a bundle\"),\n+\t\t\t\t\tbundle->uri);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (bundle->file && !bundle->unbundled) {\n+\t\t\t/*\n+\t\t\t * This was downloaded, but not successfully\n+\t\t\t * unbundled. Try unbundling again.\n+\t\t\t */\n+\t\t\tif (unbundle_from_file(ctx.r, bundle->file)) {\n+\t\t\t\t/* Try looking deeper in the list. */\n+\t\t\t\tmove_direction = 1;\n+\t\t\t} else {\n+\t\t\t\t/*\n+\t\t\t\t * Succeeded in unbundle. Retry bundles\n+\t\t\t\t * that previously failed to unbundle.\n+\t\t\t\t */\n+\t\t\t\tmove_direction = -1;\n+\t\t\t\tbundle->unbundled = 1;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/*\n+\t\t * Else case: downloaded and unbundled successfully.\n+\t\t * Skip this by moving in the same direction as the\n+\t\t * previous step.\n+\t\t */\n+\n+stack_operation:\n+\t\t/* Move in the specified direction and repeat. */\n+\t\tcur += move_direction;\n+\t}\n+\n+\tfree(bundles.items);\n+\n+\t/*\n+\t * We succeed if the loop terminates because 'cur' drops below\n+\t * zero. The other case is that we terminate because 'cur'\n+\t * reaches the end of the list, so we have a failure no matter\n+\t * which bundles we apply from the list.\n+\t */\n+\treturn cur >= 0;\n+}\n+\n static int download_bundle_list(struct repository *r,\n \t\t\t\tstruct bundle_list *local_list,\n \t\t\t\tstruct bundle_list *global_list,\n@@ -490,7 +623,15 @@ static int fetch_bundle_list_in_config_format(struct repository *r,\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = download_bundle_list(r, &list_from_bundle,\n+\t/*\n+\t * If this list uses the creationToken heuristic, then the URIs\n+\t * it advertises are expected to be bundles, not nested lists.\n+\t * We can drop 'global_list' and 'depth'.\n+\t */\n+\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN) {\n+\t\tresult = fetch_bundles_by_token(r, &list_from_bundle);\n+\t\tglobal_list->heuristic = BUNDLE_HEURISTIC_CREATIONTOKEN;\n+\t} else if ((result = download_bundle_list(r, &list_from_bundle,\n \t\t\t\t\t global_list, depth)))\n \t\tgoto cleanup;\n \n@@ -632,6 +773,14 @@ int fetch_bundle_list(struct repository *r, struct bundle_list *list)\n \tint result;\n \tstruct bundle_list global_list;\n \n+\t/*\n+\t * If the creationToken heuristic is used, then the URIs\n+\t * advertised by 'list' are not nested lists and instead\n+\t * direct bundles. We do not need to use global_list.\n+\t */\n+\tif (list->heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN)\n+\t\treturn fetch_bundles_by_token(r, list);\n+\n \tinit_bundle_list(&global_list);\n \n \t/* If a bundle is added to this global list, then it is required. */\n@@ -640,7 +789,10 @@ int fetch_bundle_list(struct repository *r, struct bundle_list *list)\n \tif ((result = download_bundle_list(r, list, &global_list, 0)))\n \t\tgoto cleanup;\n \n-\tresult = unbundle_all_bundles(r, &global_list);\n+\tif (list->heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN)\n+\t\tresult = fetch_bundles_by_token(r, list);\n+\telse\n+\t\tresult = unbundle_all_bundles(r, &global_list);\n \n cleanup:\n \tfor_all_bundles_in_list(&global_list, unlink_bundle, NULL);\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 474432c8ace..6f9417a0afb 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -401,17 +401,43 @@ test_expect_success 'clone bundle list (http, creationToken)' '\n \tgit -C clone-list-http-2 cat-file --batch-check expect <<-EOF &&\n-\t$HTTPD_URL/bundle-1.bundle\n-\t$HTTPD_URL/bundle-2.bundle\n-\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/bundle-list\n \t$HTTPD_URL/bundle-4.bundle\n+\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/bundle-2.bundle\n+\t$HTTPD_URL/bundle-1.bundle\n+\tEOF\n+\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\n+\tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=$(pwd)/trace-clone.txt \\\n+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\t--single-branch --branch=base --no-tags \\\n+\t\t\"$HTTPD_URL/smart/fetch.git\" clone-token-http &&\n+\n+\tcat >expect <<-EOF &&\n \t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-1.bundle\n \tEOF\n \n-\t# Since the creationToken heuristic is not yet understood by the\n-\t# client, the order cannot be verified at this moment. Sort the\n-\t# list for consistent results.\n-\ttest_remote_https_urls actual &&\n+\ttest_remote_https_urls actual &&\n \ttest_cmp expect actual\n '\n \ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 1928ea1dd7c..b7d5551262c 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -831,6 +831,52 @@ test_expect_success 'auto-discover multiple bundles from HTTP clone' '\n \tgrep -f pattern trace.txt\n '\n \n+test_expect_success 'auto-discover multiple bundles from HTTP clone: creationToken heuristic' '\n+\ttest_when_finished rm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/repo4.git\" &&\n+\ttest_when_finished rm -rf clone-heuristic trace*.txt &&\n+\n+\ttest_commit -C src newest &&\n+\tgit -C src bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/newest.bundle\" HEAD~1..HEAD &&\n+\tgit clone --bare --no-local src \"$HTTPD_DOCUMENT_ROOT_PATH/repo4.git\" &&\n+\n+\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/repo4.git/config\" <<-EOF &&\n+\t[uploadPack]\n+\t\tadvertiseBundleURIs = true\n+\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"everything\"]\n+\t\turi = $HTTPD_URL/everything.bundle\n+\t\tcreationtoken = 1\n+\n+\t[bundle \"new\"]\n+\t\turi = $HTTPD_URL/new.bundle\n+\t\tcreationtoken = 2\n+\n+\t[bundle \"newest\"]\n+\t\turi = $HTTPD_URL/newest.bundle\n+\t\tcreationtoken = 3\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" \\\n+\t\tgit -c protocol.version=2 \\\n+\t\t -c transfer.bundleURI=true clone \\\n+\t\t\"$HTTPD_URL/smart/repo4.git\" clone-heuristic &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/newest.bundle\n+\t$HTTPD_URL/new.bundle\n+\t$HTTPD_URL/everything.bundle\n+\tEOF\n+\n+\t# We should fetch all bundles in the expected order.\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual\n+'\n+\n # DO NOT add non-httpd-specific tests here, because the last part of this\n # test script is only executed when httpd is available and enabled.\n \n-- \ngitgitgadget\n\n"},{"id":"470918","messageId":"17c404c1b836d7c160defe97e1667c631d292fb0.1674487310.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v2.git.1674487310.gitgitgadget@gmail.com","subject":"[PATCH v2 06/10] clone: set fetch.bundleURI if appropriate","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-23T15:21:46Z","receivedAt":"2023-01-23T15:22:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBundle providers may organize their bundle lists in a way that is\nintended to improve incremental fetches, not just initial clones.\nHowever, they do need to state that they have organized with that in\nmind, or else the client will not expect to save time by downloading\nbundles after the initial clone. This is done by specifying a\nbundle.heuristic value.\n\nThere are two types of bundle lists: those at a static URI and those\nthat are advertised from a Git remote over protocol v2.\n\nThe new fetch.bundleURI config value applies for static bundle URIs that\nare not advertised over protocol v2. If the user specifies a static URI\nvia 'git clone --bundle-uri', then Git can set this config as a reminder\nfor future 'git fetch' operations to check the bundle list before\nconnecting to the remote(s).\n\nFor lists provided over protocol v2, we will want to take a different\napproach and create a property of the remote itself by creating a\nremote..* type config key. That is not implemented in this change.\n\nLater changes will update 'git fetch' to consume this option.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/fetch.txt | 8 +++++++\n builtin/clone.c | 6 +++++-\n bundle-uri.c | 5 ++++-\n bundle-uri.h | 8 ++++++-\n t/t5558-clone-bundle-uri.sh | 39 ++++++++++++++++++++++++++++++++++\n 5 files changed, 63 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt\nindex cd65d236b43..244f44d460f 100644\n--- a/Documentation/config/fetch.txt\n+++ b/Documentation/config/fetch.txt\n@@ -96,3 +96,11 @@ fetch.writeCommitGraph::\n \tmerge and the write may take longer. Having an updated commit-graph\n \tfile helps performance of many Git commands, including `git merge-base`,\n \t`git push -f`, and `git log --graph`. Defaults to false.\n+\n+fetch.bundleURI::\n+\tThis value stores a URI for downloading Git object data from a bundle\n+\tURI before performing an incremental fetch from the origin Git server.\n+\tThis is similar to how the `--bundle-uri` option behaves in\n+\tlinkgit:git-clone[1]. `git clone --bundle-uri` will set the\n+\t`fetch.bundleURI` value if the supplied bundle URI contains a bundle\n+\tlist that is organized for incremental fetches.\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 5453ba5277f..5370617664d 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1248,12 +1248,16 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t * data from the --bundle-uri option.\n \t */\n \tif (bundle_uri) {\n+\t\tint has_heuristic = 0;\n+\n \t\t/* At this point, we need the_repository to match the cloned repo. */\n \t\tif (repo_init(the_repository, git_dir, work_tree))\n \t\t\twarning(_(\"failed to initialize the repo, skipping bundle URI\"));\n-\t\telse if (fetch_bundle_uri(the_repository, bundle_uri))\n+\t\telse if (fetch_bundle_uri(the_repository, bundle_uri, &has_heuristic))\n \t\t\twarning(_(\"failed to fetch objects from bundle URI '%s'\"),\n \t\t\t\tbundle_uri);\n+\t\telse if (has_heuristic)\n+\t\t\tgit_config_set_gently(\"fetch.bundleuri\", bundle_uri);\n \t}\n \n \tstrvec_push(&transport_ls_refs_options.ref_prefixes, \"HEAD\");\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 39acd856fb9..162a9276f31 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -742,7 +742,8 @@ static int unlink_bundle(struct remote_bundle_info *info, void *data)\n \treturn 0;\n }\n \n-int fetch_bundle_uri(struct repository *r, const char *uri)\n+int fetch_bundle_uri(struct repository *r, const char *uri,\n+\t\t int *has_heuristic)\n {\n \tint result;\n \tstruct bundle_list list;\n@@ -762,6 +763,8 @@ int fetch_bundle_uri(struct repository *r, const char *uri)\n \tresult = unbundle_all_bundles(r, &list);\n \n cleanup:\n+\tif (has_heuristic)\n+\t\t*has_heuristic = (list.heuristic != BUNDLE_HEURISTIC_NONE);\n \tfor_all_bundles_in_list(&list, unlink_bundle, NULL);\n \tclear_bundle_list(&list);\n \tclear_remote_bundle_info(&bundle, NULL);\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex ef32840bfa6..6dbc780f661 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -124,8 +124,14 @@ int bundle_uri_parse_config_format(const char *uri,\n * based on that information.\n *\n * Returns non-zero if no bundle information is found at the given 'uri'.\n+ *\n+ * If the pointer 'has_heuristic' is non-NULL, then the value it points to\n+ * will be set to be non-zero if and only if the fetched list has a\n+ * heuristic value. Such a value indicates that the list was designed for\n+ * incremental fetches.\n */\n-int fetch_bundle_uri(struct repository *r, const char *uri);\n+int fetch_bundle_uri(struct repository *r, const char *uri,\n+\t\t int *has_heuristic);\n \n /**\n * Given a bundle list that was already advertised (likely by the\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 6f9417a0afb..b2d15e141ca 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -432,6 +432,8 @@ test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n \t\t--single-branch --branch=base --no-tags \\\n \t\t\"$HTTPD_URL/smart/fetch.git\" clone-token-http &&\n \n+\ttest_cmp_config -C clone-token-http \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n+\n \tcat >expect <<-EOF &&\n \t$HTTPD_URL/bundle-list\n \t$HTTPD_URL/bundle-1.bundle\n@@ -441,6 +443,43 @@ test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n+\ttest_when_finished rm -rf fetch-http-4 trace*.txt &&\n+\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" \\\n+\tgit clone --single-branch --branch=base \\\n+\t\t--bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\t\"$HTTPD_URL/smart/fetch.git\" fetch-http-4 &&\n+\n+\ttest_cmp_config -C fetch-http-4 \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-1.bundle\n+\tEOF\n+\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# only received base ref from bundle-1\n+\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\tEOF\n+\ttest_cmp expect refs\n+'\n+\n # Do not add tests here unless they use the HTTP server, as they will\n # not run unless the HTTP dependencies exist.\n \n-- \ngitgitgadget\n\n"},{"id":"470919","messageId":"d491070efedffa95e9bf7d8f18d7707d43776488.1674487310.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v2.git.1674487310.gitgitgadget@gmail.com","subject":"[PATCH v2 07/10] bundle-uri: drop bundle.flag from design doc","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-23T15:21:47Z","receivedAt":"2023-01-23T15:22:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe Implementation Plan section lists a 'bundle.flag' option that is not\ndocumented anywhere else. What is documented elsewhere in the document\nand implemented by previous changes is the 'bundle.heuristic' config\nkey. For now, a heuristic is required to indicate that a bundle list is\norganized for use during 'git fetch', and it is also sufficient for all\nexisting designs.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/bundle-uri.txt | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/technical/bundle-uri.txt b/Documentation/technical/bundle-uri.txt\nindex b78d01d9adf..91d3a13e327 100644\n--- a/Documentation/technical/bundle-uri.txt\n+++ b/Documentation/technical/bundle-uri.txt\n@@ -479,14 +479,14 @@ outline for submitting these features:\n (This choice is an opt-in via a config option and a command-line\n option.)\n \n-4. Allow the client to understand the `bundle.flag=forFetch` configuration\n+4. Allow the client to understand the `bundle.heuristic` configuration key\n and the `bundle..creationToken` heuristic. When `git clone`\n- discovers a bundle URI with `bundle.flag=forFetch`, it configures the\n- client repository to check that bundle URI during later `git fetch `\n+ discovers a bundle URI with `bundle.heuristic`, it configures the client\n+ repository to check that bundle URI during later `git fetch `\n commands.\n \n 5. Allow clients to discover bundle URIs during `git fetch` and configure\n- a bundle URI for later fetches if `bundle.flag=forFetch`.\n+ a bundle URI for later fetches if `bundle.heuristic` is set.\n \n 6. Implement the \"inspect headers\" heuristic to reduce data downloads when\n the `bundle..creationToken` heuristic is not available.\n-- \ngitgitgadget\n\n"},{"id":"470920","messageId":"pull.1454.v2.git.1674487310.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.git.1673037405.gitgitgadget@gmail.com","subject":"[PATCH v2 00/10] Bundle URIs V: creationToken heuristic for incremental fetches","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-23T15:21:40Z","receivedAt":"2023-01-23T15:22:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This fifth part to the bundle URIs feature follows part IV (advertising via\nprotocol v2) which recently merged to 'master', so this series is based on\n'master'.\n\nThis part introduces the concept of a heuristic that a bundle list can\nadvertise. The purpose of the heuristic is to hint to the Git client that\nthe bundles can be downloaded and unbundled in a certain order. In\nparticular, that order can assist with using the same bundle URI to download\nnew bundles from an updated bundle list. This allows bundle URIs to assist\nwith incremental fetches, not just initial clones.\n\nThe only planned heuristic is the \"creationToken\" heuristic where the bundle\nlist adds a 64-bit unsigned integer \"creationToken\" value to each bundle in\nthe list. Those values provide an ordering on the bundles implying that the\nbundles can be unbundled in increasing creationToken order and at each point\nthe required commits for the ith bundle were provided by bundles with lower\ncreationTokens.\n\nAt clone time, the only difference implied by the creationToken order is\nthat the Git client does not need to guess at the order to apply the\nbundles, but instead can use the creationToken order to apply them without\nfailure and retry. However, this presents an interesting benefit during\nfetches: the Git client can check the bundle list and download bundles in\ndecreasing creationToken order until the required commits for these bundles\nare present within the repository's object store. This prevents downloading\nmore bundle information than required.\n\nThe creationToken value is also a promise that the Git client will not need\nto download a bundle if its creationToken is less than or equal to the\ncreationToken of a previously-downloaded bundle. This further improves the\nperformance during a fetch in that the client does not need to download any\nbundles at all if it recognizes that the maximum creationToken is the same\n(or smaller than) a previously-downloaded creationToken.\n\nThe creationToken concept is documented in the existing design document at\nDocumentation/technical/bundle-uri.txt, including suggested ways for bundle\nproviders to organize their bundle lists to take advantage of the heuristic.\n\nThis series formalizes the creationToken heuristic and the Git client logic\nfor understanding it. Further, for bundle lists provided by the git clone\n--bundle-uri option, the Git client will recognize the heuristic as being\nhelpful for incremental fetches and store config values so that future git\nfetch commands check the bundle list before communicating with any Git\nremotes.\n\nNote that this option does not integrate fetches with bundle lists\nadvertised via protocol v2. I spent some time working on this, but found the\nimplementation to be distinct enough that it merited its own attention in a\nseparate series. In particular, the configuration for indicating that a\nfetch should check the bundle-uri protocol v2 command seemed best to be\nlocated within a Git remote instead of a repository-global key such as is\nbeing used for a static URI. Further, the timing of querying the bundle-uri\ncommand during a git fetch command is significantly different and more\ncomplicated than how it is used in git clone.\n\n\nWhat Remains?\n=============\n\nOriginally, I had planned on making this bundle URI work a 5-part series,\nand this is part 5. Shouldn't we be done now?\n\nThere are two main things that should be done after this series, in any\norder:\n\n * Teach git fetch to check a bundle list advertised by a remote over the\n bundle-uri protocol v2 command.\n * Add the bundle..filter option to allow advertising bundles and\n partial bundles side-by-side.\n\nThere is also room for expanding tests for more error conditions, or for\nother tweaks that are not currently part of the design document. I do think\nthat after this series, the feature will be easier to work on different\nparts in parallel.\n\n\nPatch Outline\n=============\n\n * (New in v2) Patch 1 adds a new VERIFY_BUNDLE_SKIP_REACHABLE flag for\n verify_bundle() which is called by unbundle(); this fixes a probable\n exposed by patch 10 where a bundle would fail to unbundle due to the \"are\n the required commits reachable from refs?\" check.\n * Patch 2 creates a test setup demonstrating a creationToken heuristic. At\n this point, the Git client ignores the heuristic and uses its ad-hoc\n strategy for ordering the bundles.\n * Patches 3 and 4 teach Git to parse the bundle.heuristic and\n bundle..creationToken keys in a bundle list.\n * Patch 5 teaches Git to download bundles using the creationToken order.\n This order uses a stack approach to start from the maximum creationToken\n and continue downloading the next bundle in the list until all bundles\n can successfully be unbundled. This is the algorithm required for\n incremental fetches, while initial clones could download in the opposite\n order. Since clones will download all bundles anyway, having a second\n code path just for clones seemed unnecessary.\n * Patch 6 teaches git clone --bundle-uri to set fetch.bundleURI when the\n advertised bundle list includs a heuristic that Git understands.\n * Patch 7 updates the design document to remove reference to a bundle.flag\n option that was previously going to indicate the list was designed for\n fetches, but the bundle.heuristic option already does that.\n * Patch 8 teaches git fetch to check fetch.bundleURI and download bundles\n from that static URI before connecting to remotes via the Git protocol.\n * Patch 9 introduces a new fetch.bundleCreationToken config value to store\n the maximum creationToken of downloaded bundles. This prevents\n downloading the latest bundle on every git fetch command, reducing waste.\n * (New in v2) Patch 10 adds new tests for interesting incremental fetch\n shapes. Along with other test edits in other patches, these revealed\n several issues that required improvement within this series. These tests\n also check extra cases around failed bundle downloads.\n\n\nUpdates in v2\n=============\n\n * Patches 1 and 10 are new.\n * I started making the extra tests in patch 10 due to Victoria's concern\n around failed downloads. I extended the bundle list in a way that exposed\n other issues that are fixed in this version. Unfortunately, the test\n requires the full functionality of the entire series, so the tests are\n not isolated to where the code fixes are made. One thing that I noticed\n in the process is that some of the tests were using the local-clone trick\n to copy full object directories instead of copying only the requested\n object set. This was causing confusion in how the bundles were applying\n or failing to apply, so the tests are updated to use http whenever\n possible.\n * In Patch 2, I created a new test_remote_https_urls helper to get the full\n download list (in order). In this patch, the bundle download order is not\n well-defined, but is modified in later tests when it becomes\n well-defined.\n * In Patch 3, I updated the connection between config value and enum value\n to be an array of pairs instead of faking a hashmap-like interface that\n could be dangerous if the enum values were assigned incorrectly.\n * In Patch 5, the 'sorted' list and its type was renamed to be more\n descriptive. This also included updates to \"append_bundle()\" and\n \"compare_creation_token_decreasing()\" to be more descriptive. This had\n some side effects in Patch 8 due to the renames.\n * In Patch 5, I added the interesting bundle shape to the commit message to\n remind us of why the creationToken algorithm needs to be the way it is. I\n also removed the \"stack\" language in favor of discussing ranges of the\n sorted list. Some renames, such as \"pop_or_push\" is changed to\n \"move_direction\", resulted from this change of language.\n * The assignment of heuristic from the local list to global_list was moved\n into Patch 5.\n * In Patch 5, one of the tests removed bundle-2 because it allows a later\n test for git fetch to demonstrate the interesting behavior where bundle-4\n requires both bundle-2 and bundle-3.\n * In Patch 6, the fetch.bundleURI config is described differently,\n including dropping the defunct git fetch --bundle-uri reference and\n discussing that git clone --bundle-uri will set it automatically.\n * Patch 8 no longer refers to a config value starting with \"remote:\". It\n also expands a test that was previously not expanded in v1.\n * Patch 9 updates the documentation for fetch.bundleURI and\n fetch.bundleCreationToken to describe how the user should unset the\n latter if they edit the former.\n * Much of Patch 9's changes are due to context changes from the renames in\n Patch 5. However, it also adds the restriction that it will not attempt\n to download bundles unless their creationToken is strictly greater than\n the stored token. This ends up being critical to the failed download\n case, preventing an incremental fetch from downloading all bundles just\n because one bundle failed to download (and that case is tested in patch\n 10).\n * Patch 10 adds significant testing, including several tests of failed\n bundle downloads in various cases.\n\nThanks,\n\n * Stolee\n\nDerrick Stolee (10):\n bundle: optionally skip reachability walk\n t5558: add tests for creationToken heuristic\n bundle-uri: parse bundle.heuristic=creationToken\n bundle-uri: parse bundle..creationToken values\n bundle-uri: download in creationToken order\n clone: set fetch.bundleURI if appropriate\n bundle-uri: drop bundle.flag from design doc\n fetch: fetch from an external bundle URI\n bundle-uri: store fetch.bundleCreationToken\n bundle-uri: test missing bundles with heuristic\n\n Documentation/config/bundle.txt | 7 +\n Documentation/config/fetch.txt | 24 +\n Documentation/technical/bundle-uri.txt | 8 +-\n builtin/clone.c | 6 +-\n builtin/fetch.c | 7 +\n bundle-uri.c | 257 +++++++++-\n bundle-uri.h | 28 +-\n bundle.c | 3 +-\n bundle.h | 1 +\n t/t5558-clone-bundle-uri.sh | 672 ++++++++++++++++++++++++-\n t/t5601-clone.sh | 46 ++\n t/t5750-bundle-uri-parse.sh | 37 ++\n t/test-lib-functions.sh | 8 +\n 13 files changed, 1091 insertions(+), 13 deletions(-)\n\n\nbase-commit: 4dbebc36b0893f5094668ddea077d0e235560b16\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1454%2Fderrickstolee%2Fbundle-redo%2FcreationToken-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1454/derrickstolee/bundle-redo/creationToken-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1454\n\nRange-diff vs v1:\n\n -: ----------- > 1: b3828725bc8 bundle: optionally skip reachability walk\n 1: 39eed914878 ! 2: 427aff4d5e5 t5558: add tests for creationToken heuristic\n @@ Commit message\n meantime, create tests that add creation tokens to the bundle list. For\n now, the Git client correctly ignores these unknown keys.\n \n + Create a new test helper function, test_remote_https_urls, which filters\n + GIT_TRACE2_EVENT output to extract a list of URLs passed to\n + git-remote-https child processes. This can be used to verify the order\n + of these requests as we implement the creationToken heuristic. For now,\n + we need to sort the actual output since the current client does not have\n + a well-defined order that it applies to the bundles.\n +\n Signed-off-by: Derrick Stolee \n \n ## t/t5558-clone-bundle-uri.sh ##\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone HTTP bundle' '\n - \ttest_config -C clone-http log.excludedecoration refs/bundle/\n '\n \n -+# usage: test_bundle_downloaded \n -+test_bundle_downloaded () {\n -+\tcat >pattern <<-EOF &&\n -+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/$1\"\\]\n -+\tEOF\n -+\tgrep -f pattern \"$2\"\n -+}\n -+\n test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n +\ttest_when_finished rm -f trace*.txt &&\n +\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (HTTP, no he\n -\tgit -C clone-list-http cat-file --batch-check expect <<-EOF &&\n ++\t$HTTPD_URL/bundle-1.bundle\n ++\t$HTTPD_URL/bundle-2.bundle\n ++\t$HTTPD_URL/bundle-3.bundle\n ++\t$HTTPD_URL/bundle-4.bundle\n ++\t$HTTPD_URL/bundle-list\n ++\tEOF\n ++\n ++\t# Sort the list, since the order is not well-defined\n ++\t# without a heuristic.\n ++\ttest_remote_https_urls actual &&\n ++\ttest_cmp expect actual\n '\n \n test_expect_success 'clone bundle list (HTTP, any mode)' '\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (HTTP, any m\n '\n \n +test_expect_success 'clone bundle list (http, creationToken)' '\n ++\ttest_when_finished rm -f trace*.txt &&\n ++\n +\tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n +\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n +\t[bundle]\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (HTTP, any m\n +\t\tcreationToken = 4\n +\tEOF\n +\n -+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" . clone-list-http-2 &&\n ++\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" git \\\n ++\t\tclone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n ++\t\t\"$HTTPD_URL/smart/fetch.git\" clone-list-http-2 &&\n +\n +\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n -+\tgit -C clone-list-http-2 cat-file --batch-check expect <<-EOF &&\n ++\t$HTTPD_URL/bundle-1.bundle\n ++\t$HTTPD_URL/bundle-2.bundle\n ++\t$HTTPD_URL/bundle-3.bundle\n ++\t$HTTPD_URL/bundle-4.bundle\n ++\t$HTTPD_URL/bundle-list\n ++\tEOF\n ++\n ++\t# Since the creationToken heuristic is not yet understood by the\n ++\t# client, the order cannot be verified at this moment. Sort the\n ++\t# list for consistent results.\n ++\ttest_remote_https_urls actual &&\n ++\ttest_cmp expect actual\n +'\n +\n # Do not add tests here unless they use the HTTP server, as they will\n # not run unless the HTTP dependencies exist.\n \n +\n + ## t/test-lib-functions.sh ##\n +@@ t/test-lib-functions.sh: test_region () {\n + \treturn 0\n + }\n + \n ++# Given a GIT_TRACE2_EVENT log over stdin, writes to stdout a list of URLs\n ++# sent to git-remote-https child processes.\n ++test_remote_https_urls() {\n ++\tgrep -e '\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\".*\"\\]' |\n ++\t\tsed -e 's/{\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"//g' \\\n ++\t\t -e 's/\"\\]}//g'\n ++}\n ++\n + # Print the destination of symlink(s) provided as arguments. Basically\n + # the same as the readlink command, but it's not available everywhere.\n + test_readlink () {\n 2: 9007249b948 ! 3: f6f8197c9cc bundle-uri: parse bundle.heuristic=creationToken\n @@ Commit message\n bundle-uri' to print the heuristic value and verify that the parsing\n works correctly.\n \n + As an extra precaution, create the internal 'heuristics' array to be a\n + list of (enum, string) pairs so we can iterate through the array entries\n + carefully, regardless of the enum values.\n +\n Signed-off-by: Derrick Stolee \n \n ## Documentation/config/bundle.txt ##\n @@ bundle-uri.c\n #include \"config.h\"\n #include \"remote.h\"\n \n -+static const char *heuristics[] = {\n -+\t[BUNDLE_HEURISTIC_NONE] = \"\",\n -+\t[BUNDLE_HEURISTIC_CREATIONTOKEN] = \"creationToken\",\n ++static struct {\n ++\tenum bundle_list_heuristic heuristic;\n ++\tconst char *name;\n ++} heuristics[BUNDLE_HEURISTIC__COUNT] = {\n ++\t{ BUNDLE_HEURISTIC_NONE, \"\"},\n ++\t{ BUNDLE_HEURISTIC_CREATIONTOKEN, \"creationToken\" },\n +};\n +\n static int compare_bundles(const void *hashmap_cmp_fn_data,\n @@ bundle-uri.c: void print_bundle_list(FILE *fp, struct bundle_list *list)\n \tfprintf(fp, \"\\tversion = %d\\n\", list->version);\n \tfprintf(fp, \"\\tmode = %s\\n\", mode);\n \n -+\tif (list->heuristic)\n -+\t\tprintf(\"\\theuristic = %s\\n\", heuristics[list->heuristic]);\n ++\tif (list->heuristic) {\n ++\t\tint i;\n ++\t\tfor (i = 0; i < BUNDLE_HEURISTIC__COUNT; i++) {\n ++\t\t\tif (heuristics[i].heuristic == list->heuristic) {\n ++\t\t\t\tprintf(\"\\theuristic = %s\\n\",\n ++\t\t\t\t heuristics[list->heuristic].name);\n ++\t\t\t\tbreak;\n ++\t\t\t}\n ++\t\t}\n ++\t}\n +\n \tfor_all_bundles_in_list(list, summarize_bundle, fp);\n }\n @@ bundle-uri.c: static int bundle_list_update(const char *key, const char *value,\n +\t\tif (!strcmp(subkey, \"heuristic\")) {\n +\t\t\tint i;\n +\t\t\tfor (i = 0; i < BUNDLE_HEURISTIC__COUNT; i++) {\n -+\t\t\t\tif (!strcmp(value, heuristics[i])) {\n -+\t\t\t\t\tlist->heuristic = i;\n ++\t\t\t\tif (heuristics[i].heuristic &&\n ++\t\t\t\t heuristics[i].name &&\n ++\t\t\t\t !strcmp(value, heuristics[i].name)) {\n ++\t\t\t\t\tlist->heuristic = heuristics[i].heuristic;\n +\t\t\t\t\treturn 0;\n +\t\t\t\t}\n +\t\t\t}\n @@ bundle-uri.h: enum bundle_list_mode {\n +\tBUNDLE_HEURISTIC_CREATIONTOKEN,\n +\n +\t/* Must be last. */\n -+\tBUNDLE_HEURISTIC__COUNT,\n ++\tBUNDLE_HEURISTIC__COUNT\n +};\n +\n /**\n 3: a1808f0b10c = 4: 12efa228d04 bundle-uri: parse bundle..creationToken values\n 4: 57c0174d375 ! 5: 7cfaa3c518c bundle-uri: download in creationToken order\n @@ Commit message\n strategy implemented here provides that short-circuit where the client\n downloads a minimal set of bundles.\n \n + However, we are not satisfied by the naive approach of downloading\n + bundles until one successfully unbundles, expecting the earlier bundles\n + to successfully unbundle now. The example repository in t5558\n + demonstrates this well:\n +\n + ---------------- bundle-4\n +\n + 4\n + / \\\n + ----|---|------- bundle-3\n + | |\n + | 3\n + | |\n + ----|---|------- bundle-2\n + | |\n + 2 |\n + | |\n + ----|---|------- bundle-1\n + \\ /\n + 1\n + |\n + (previous commits)\n +\n + In this repository, if we already have the objects for bundle-1 and then\n + try to fetch from this list, the naive approach will fail. bundle-4\n + requires both bundle-3 and bundle-2, though bundle-3 will successfully\n + unbundle without bundle-2. Thus, the algorithm needs to keep this in\n + mind.\n +\n A later implementation detail will store the maximum creationToken seen\n during such a bundle download, and the client will avoid downloading a\n bundle unless its creationToken is strictly greater than that stored\n @@ bundle-uri.c: static int download_bundle_to_file(struct remote_bundle_info *bund\n \treturn 0;\n }\n \n -+struct sorted_bundle_list {\n ++struct bundles_for_sorting {\n +\tstruct remote_bundle_info **items;\n +\tsize_t alloc;\n +\tsize_t nr;\n +};\n +\n -+static int insert_bundle(struct remote_bundle_info *bundle, void *data)\n ++static int append_bundle(struct remote_bundle_info *bundle, void *data)\n +{\n -+\tstruct sorted_bundle_list *list = data;\n ++\tstruct bundles_for_sorting *list = data;\n +\tlist->items[list->nr++] = bundle;\n +\treturn 0;\n +}\n +\n -+static int compare_creation_token(const void *va, const void *vb)\n ++/**\n ++ * For use in QSORT() to get a list sorted by creationToken\n ++ * in decreasing order.\n ++ */\n ++static int compare_creation_token_decreasing(const void *va, const void *vb)\n +{\n +\tconst struct remote_bundle_info * const *a = va;\n +\tconst struct remote_bundle_info * const *b = vb;\n @@ bundle-uri.c: static int download_bundle_to_file(struct remote_bundle_info *bund\n +\t\t\t\t struct bundle_list *list)\n +{\n +\tint cur;\n -+\tint pop_or_push = 0;\n ++\tint move_direction = 0;\n +\tstruct bundle_list_context ctx = {\n +\t\t.r = r,\n +\t\t.list = list,\n +\t\t.mode = list->mode,\n +\t};\n -+\tstruct sorted_bundle_list sorted = {\n ++\tstruct bundles_for_sorting bundles = {\n +\t\t.alloc = hashmap_get_size(&list->bundles),\n +\t};\n +\n -+\tALLOC_ARRAY(sorted.items, sorted.alloc);\n ++\tALLOC_ARRAY(bundles.items, bundles.alloc);\n +\n -+\tfor_all_bundles_in_list(list, insert_bundle, &sorted);\n ++\tfor_all_bundles_in_list(list, append_bundle, &bundles);\n +\n -+\tQSORT(sorted.items, sorted.nr, compare_creation_token);\n ++\tQSORT(bundles.items, bundles.nr, compare_creation_token_decreasing);\n +\n +\t/*\n -+\t * Use a stack-based approach to download the bundles and attempt\n -+\t * to unbundle them in decreasing order by creation token. If we\n -+\t * fail to unbundle (after a successful download) then move to the\n -+\t * next non-downloaded bundle (push to the stack) and attempt\n -+\t * downloading. Once we succeed in applying a bundle, move to the\n -+\t * previous unapplied bundle (pop the stack) and attempt to unbundle\n -+\t * it again.\n ++\t * Attempt to download and unbundle the minimum number of bundles by\n ++\t * creationToken in decreasing order. If we fail to unbundle (after\n ++\t * a successful download) then move to the next non-downloaded bundle\n ++\t * and attempt downloading. Once we succeed in applying a bundle,\n ++\t * move to the previous unapplied bundle and attempt to unbundle it\n ++\t * again.\n +\t *\n +\t * In the case of a fresh clone, we will likely download all of the\n +\t * bundles before successfully unbundling the oldest one, then the\n @@ bundle-uri.c: static int download_bundle_to_file(struct remote_bundle_info *bund\n +\t * repo's object store.\n +\t */\n +\tcur = 0;\n -+\twhile (cur >= 0 && cur < sorted.nr) {\n -+\t\tstruct remote_bundle_info *bundle = sorted.items[cur];\n ++\twhile (cur >= 0 && cur < bundles.nr) {\n ++\t\tstruct remote_bundle_info *bundle = bundles.items[cur];\n +\t\tif (!bundle->file) {\n -+\t\t\t/* Not downloaded yet. Try downloading. */\n -+\t\t\tif (download_bundle_to_file(bundle, &ctx)) {\n -+\t\t\t\t/* Failure. Push to the stack. */\n -+\t\t\t\tpop_or_push = 1;\n ++\t\t\t/*\n ++\t\t\t * Not downloaded yet. Try downloading.\n ++\t\t\t *\n ++\t\t\t * Note that bundle->file is non-NULL if a download\n ++\t\t\t * was attempted, even if it failed to download.\n ++\t\t\t */\n ++\t\t\tif (fetch_bundle_uri_internal(ctx.r, bundle, ctx.depth + 1, ctx.list)) {\n ++\t\t\t\t/* Mark as unbundled so we do not retry. */\n ++\t\t\t\tbundle->unbundled = 1;\n ++\n ++\t\t\t\t/* Try looking deeper in the list. */\n ++\t\t\t\tmove_direction = 1;\n +\t\t\t\tgoto stack_operation;\n +\t\t\t}\n +\n @@ bundle-uri.c: static int download_bundle_to_file(struct remote_bundle_info *bund\n +\t\t\t * unbundled. Try unbundling again.\n +\t\t\t */\n +\t\t\tif (unbundle_from_file(ctx.r, bundle->file)) {\n -+\t\t\t\t/* Failed to unbundle. Push to stack. */\n -+\t\t\t\tpop_or_push = 1;\n ++\t\t\t\t/* Try looking deeper in the list. */\n ++\t\t\t\tmove_direction = 1;\n +\t\t\t} else {\n -+\t\t\t\t/* Succeeded in unbundle. Pop stack. */\n -+\t\t\t\tpop_or_push = -1;\n ++\t\t\t\t/*\n ++\t\t\t\t * Succeeded in unbundle. Retry bundles\n ++\t\t\t\t * that previously failed to unbundle.\n ++\t\t\t\t */\n ++\t\t\t\tmove_direction = -1;\n ++\t\t\t\tbundle->unbundled = 1;\n +\t\t\t}\n +\t\t}\n +\n @@ bundle-uri.c: static int download_bundle_to_file(struct remote_bundle_info *bund\n +\n +stack_operation:\n +\t\t/* Move in the specified direction and repeat. */\n -+\t\tcur += pop_or_push;\n ++\t\tcur += move_direction;\n +\t}\n +\n -+\tfree(sorted.items);\n ++\tfree(bundles.items);\n +\n +\t/*\n +\t * We succeed if the loop terminates because 'cur' drops below\n @@ bundle-uri.c: static int fetch_bundle_list_in_config_format(struct repository *r\n +\t * it advertises are expected to be bundles, not nested lists.\n +\t * We can drop 'global_list' and 'depth'.\n +\t */\n -+\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN)\n ++\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN) {\n +\t\tresult = fetch_bundles_by_token(r, &list_from_bundle);\n -+\telse if ((result = download_bundle_list(r, &list_from_bundle,\n ++\t\tglobal_list->heuristic = BUNDLE_HEURISTIC_CREATIONTOKEN;\n ++\t} else if ((result = download_bundle_list(r, &list_from_bundle,\n \t\t\t\t\t global_list, depth)))\n \t\tgoto cleanup;\n \n @@ bundle-uri.c: int fetch_bundle_list(struct repository *r, struct bundle_list *li\n \tfor_all_bundles_in_list(&global_list, unlink_bundle, NULL);\n \n ## t/t5558-clone-bundle-uri.sh ##\n -@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (HTTP, any mode)' '\n - '\n - \n - test_expect_success 'clone bundle list (http, creationToken)' '\n -+\ttest_when_finished rm -f trace*.txt &&\n -+\n - \tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n - \tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n - \t[bundle]\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (http, creationToken)' '\n - \t\tcreationToken = 4\n - \tEOF\n + \tgit -C clone-list-http-2 cat-file --batch-check oids &&\n --\tgit -C clone-list-http-2 cat-file --batch-check expect <<-EOF &&\n +-\t$HTTPD_URL/bundle-1.bundle\n +-\t$HTTPD_URL/bundle-2.bundle\n +-\t$HTTPD_URL/bundle-3.bundle\n ++\t$HTTPD_URL/bundle-list\n + \t$HTTPD_URL/bundle-4.bundle\n ++\t$HTTPD_URL/bundle-3.bundle\n ++\t$HTTPD_URL/bundle-2.bundle\n ++\t$HTTPD_URL/bundle-1.bundle\n ++\tEOF\n ++\n ++\ttest_remote_https_urls actual &&\n ++\ttest_cmp expect actual\n +'\n +\n -+test_expect_success 'clone bundle list (http, creationToken)' '\n ++test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n +\ttest_when_finished rm -f trace*.txt &&\n +\n +\tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (http, creat\n +\t[bundle \"bundle-1\"]\n +\t\turi = bundle-1.bundle\n +\t\tcreationToken = 1\n -+\n -+\t[bundle \"bundle-2\"]\n -+\t\turi = bundle-2.bundle\n -+\t\tcreationToken = 2\n +\tEOF\n +\n +\tGIT_TRACE2_EVENT=$(pwd)/trace-clone.txt \\\n +\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n -+\t\tclone-from clone-token-http &&\n ++\t\t--single-branch --branch=base --no-tags \\\n ++\t\t\"$HTTPD_URL/smart/fetch.git\" clone-token-http &&\n +\n -+\ttest_bundle_downloaded bundle-1.bundle trace-clone.txt &&\n -+\ttest_bundle_downloaded bundle-2.bundle trace-clone.txt\n ++\tcat >expect <<-EOF &&\n + \t$HTTPD_URL/bundle-list\n ++\t$HTTPD_URL/bundle-1.bundle\n + \tEOF\n + \n +-\t# Since the creationToken heuristic is not yet understood by the\n +-\t# client, the order cannot be verified at this moment. Sort the\n +-\t# list for consistent results.\n +-\ttest_remote_https_urls actual &&\n ++\ttest_remote_https_urls actual &&\n + \ttest_cmp expect actual\n '\n \n - # Do not add tests here unless they use the HTTP server, as they will\n \n ## t/t5601-clone.sh ##\n @@ t/t5601-clone.sh: test_expect_success 'auto-discover multiple bundles from HTTP clone' '\n \tgrep -f pattern trace.txt\n '\n \n -+# Usage: test_bundle_downloaded \n -+test_bundle_downloaded () {\n -+\tcat >pattern <<-EOF &&\n -+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/$1.bundle\"\\]\n -+\tEOF\n -+\tgrep -f pattern \"$2\"\n -+}\n -+\n +test_expect_success 'auto-discover multiple bundles from HTTP clone: creationToken heuristic' '\n +\ttest_when_finished rm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/repo4.git\" &&\n +\ttest_when_finished rm -rf clone-heuristic trace*.txt &&\n @@ t/t5601-clone.sh: test_expect_success 'auto-discover multiple bundles from HTTP\n +\t\t -c transfer.bundleURI=true clone \\\n +\t\t\"$HTTPD_URL/smart/repo4.git\" clone-heuristic &&\n +\n -+\t# We should fetch all bundles\n -+\tfor b in everything new newest\n -+\tdo\n -+\t\ttest_bundle_downloaded $b trace-clone.txt || return 1\n -+\tdone\n ++\tcat >expect <<-EOF &&\n ++\t$HTTPD_URL/newest.bundle\n ++\t$HTTPD_URL/new.bundle\n ++\t$HTTPD_URL/everything.bundle\n ++\tEOF\n ++\n ++\t# We should fetch all bundles in the expected order.\n ++\ttest_remote_https_urls actual &&\n ++\ttest_cmp expect actual\n +'\n +\n # DO NOT add non-httpd-specific tests here, because the last part of this\n 5: d9c6f50e4f2 ! 6: 17c404c1b83 clone: set fetch.bundleURI if appropriate\n @@ Documentation/config/fetch.txt: fetch.writeCommitGraph::\n \t`git push -f`, and `git log --graph`. Defaults to false.\n +\n +fetch.bundleURI::\n -+\tThis value stores a URI for fetching Git object data from a bundle URI\n -+\tbefore performing an incremental fetch from the origin Git server. If\n -+\tthe value is `` then running `git fetch ` is equivalent to\n -+\tfirst running `git fetch --bundle-uri=` immediately before\n -+\t`git fetch `. See details of the `--bundle-uri` option in\n -+\tlinkgit:git-fetch[1].\n ++\tThis value stores a URI for downloading Git object data from a bundle\n ++\tURI before performing an incremental fetch from the origin Git server.\n ++\tThis is similar to how the `--bundle-uri` option behaves in\n ++\tlinkgit:git-clone[1]. `git clone --bundle-uri` will set the\n ++\t`fetch.bundleURI` value if the supplied bundle URI contains a bundle\n ++\tlist that is organized for incremental fetches.\n \n ## builtin/clone.c ##\n @@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n @@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n \tstrvec_push(&transport_ls_refs_options.ref_prefixes, \"HEAD\");\n \n ## bundle-uri.c ##\n -@@ bundle-uri.c: static int fetch_bundle_list_in_config_format(struct repository *r,\n - \t * it advertises are expected to be bundles, not nested lists.\n - \t * We can drop 'global_list' and 'depth'.\n - \t */\n --\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN)\n -+\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN) {\n - \t\tresult = fetch_bundles_by_token(r, &list_from_bundle);\n --\telse if ((result = download_bundle_list(r, &list_from_bundle,\n -+\t\tglobal_list->heuristic = BUNDLE_HEURISTIC_CREATIONTOKEN;\n -+\t} else if ((result = download_bundle_list(r, &list_from_bundle,\n - \t\t\t\t\t global_list, depth)))\n - \t\tgoto cleanup;\n - \n @@ bundle-uri.c: static int unlink_bundle(struct remote_bundle_info *info, void *data)\n \treturn 0;\n }\n @@ bundle-uri.h: int bundle_uri_parse_config_format(const char *uri,\n * Given a bundle list that was already advertised (likely by the\n \n ## t/t5558-clone-bundle-uri.sh ##\n -@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (http, creationToken)' '\n - \ttest_bundle_downloaded bundle-2.bundle trace-clone.txt\n +@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n + \t\t--single-branch --branch=base --no-tags \\\n + \t\t\"$HTTPD_URL/smart/fetch.git\" clone-token-http &&\n + \n ++\ttest_cmp_config -C clone-token-http \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n ++\n + \tcat >expect <<-EOF &&\n + \t$HTTPD_URL/bundle-list\n + \t$HTTPD_URL/bundle-1.bundle\n +@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n + \ttest_cmp expect actual\n '\n \n +test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (http, creat\n +\n +\ttest_cmp_config -C fetch-http-4 \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n +\n -+\t# The clone should copy two files: the list and bundle-1.\n -+\ttest_bundle_downloaded bundle-list trace-clone.txt &&\n -+\ttest_bundle_downloaded bundle-1.bundle trace-clone.txt &&\n ++\tcat >expect <<-EOF &&\n ++\t$HTTPD_URL/bundle-list\n ++\t$HTTPD_URL/bundle-1.bundle\n ++\tEOF\n ++\n ++\ttest_remote_https_urls actual &&\n ++\ttest_cmp expect actual &&\n +\n +\t# only received base ref from bundle-1\n +\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n 6: afcfd27a883 = 7: d491070efed bundle-uri: drop bundle.flag from design doc\n 7: 1627fc158b1 ! 8: 59e57e04968 fetch: fetch from an external bundle URI\n @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (dry_run)\n \t\twrite_fetch_head = 0;\n \n -+\tif (!git_config_get_string_tmp(\"fetch.bundleuri\", &bundle_uri) &&\n -+\t !starts_with(bundle_uri, \"remote:\")) {\n ++\tif (!git_config_get_string_tmp(\"fetch.bundleuri\", &bundle_uri)) {\n +\t\tif (fetch_bundle_uri(the_repository, bundle_uri, NULL))\n +\t\t\twarning(_(\"failed to fetch bundles from '%s'\"), bundle_uri);\n +\t}\n @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\n \n ## t/t5558-clone-bundle-uri.sh ##\n +@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n + \tEOF\n + \n + \ttest_remote_https_urls actual &&\n +-\ttest_cmp expect actual\n ++\ttest_cmp expect actual &&\n ++\n ++\t# We now have only one bundle ref.\n ++\tgit -C clone-token-http for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n ++\tcat >expect <<-\\EOF &&\n ++\trefs/bundles/base\n ++\tEOF\n ++\ttest_cmp expect refs &&\n ++\n ++\t# Add remaining bundles, exercising the \"deepening\" strategy\n ++\t# for downloading via the creationToken heurisitc.\n ++\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n ++\t[bundle \"bundle-2\"]\n ++\t\turi = bundle-2.bundle\n ++\t\tcreationToken = 2\n ++\n ++\t[bundle \"bundle-3\"]\n ++\t\turi = bundle-3.bundle\n ++\t\tcreationToken = 3\n ++\n ++\t[bundle \"bundle-4\"]\n ++\t\turi = bundle-4.bundle\n ++\t\tcreationToken = 4\n ++\tEOF\n ++\n ++\tGIT_TRACE2_EVENT=\"$(pwd)/trace1.txt\" \\\n ++\t\tgit -C clone-token-http fetch origin --no-tags \\\n ++\t\trefs/heads/merge:refs/heads/merge &&\n ++\n ++\tcat >expect <<-EOF &&\n ++\t$HTTPD_URL/bundle-list\n ++\t$HTTPD_URL/bundle-4.bundle\n ++\t$HTTPD_URL/bundle-3.bundle\n ++\t$HTTPD_URL/bundle-2.bundle\n ++\tEOF\n ++\n ++\ttest_remote_https_urls actual &&\n ++\ttest_cmp expect actual &&\n ++\n ++\t# We now have all bundle refs.\n ++\tgit -C clone-token-http for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n ++\n ++\tcat >expect <<-\\EOF &&\n ++\trefs/bundles/base\n ++\trefs/bundles/left\n ++\trefs/bundles/merge\n ++\trefs/bundles/right\n ++\tEOF\n ++\ttest_cmp expect refs\n + '\n + \n + test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \tcat >expect <<-\\EOF &&\n \trefs/bundles/base\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heurist\n +\t\trefs/heads/left:refs/heads/left \\\n +\t\trefs/heads/right:refs/heads/right &&\n +\n -+\t# This fetch should copy two files: the list and bundle-2.\n -+\ttest_bundle_downloaded bundle-list trace1.txt &&\n -+\ttest_bundle_downloaded bundle-2.bundle trace1.txt &&\n -+\t! test_bundle_downloaded bundle-1.bundle trace1.txt &&\n ++\tcat >expect <<-EOF &&\n ++\t$HTTPD_URL/bundle-list\n ++\t$HTTPD_URL/bundle-2.bundle\n ++\tEOF\n ++\n ++\ttest_remote_https_urls actual &&\n ++\ttest_cmp expect actual &&\n +\n +\t# received left from bundle-2\n +\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heurist\n +\t\tcreationToken = 4\n +\tEOF\n +\n -+\t# This fetch should skip bundle-3.bundle, since its objets are\n ++\t# This fetch should skip bundle-3.bundle, since its objects are\n +\t# already local (we have the requisite commits for bundle-4.bundle).\n +\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n +\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n +\t\trefs/heads/merge:refs/heads/merge &&\n +\n -+\t# This fetch should copy three files: the list, bundle-3, and bundle-4.\n -+\ttest_bundle_downloaded bundle-list trace2.txt &&\n -+\ttest_bundle_downloaded bundle-4.bundle trace2.txt &&\n -+\t! test_bundle_downloaded bundle-1.bundle trace2.txt &&\n -+\t! test_bundle_downloaded bundle-2.bundle trace2.txt &&\n -+\t! test_bundle_downloaded bundle-3.bundle trace2.txt &&\n ++\tcat >expect <<-EOF &&\n ++\t$HTTPD_URL/bundle-list\n ++\t$HTTPD_URL/bundle-4.bundle\n ++\tEOF\n ++\n ++\ttest_remote_https_urls actual &&\n ++\ttest_cmp expect actual &&\n +\n +\t# received merge ref from bundle-4, but right is missing\n +\t# because we did not download bundle-3.\n 8: 51f210ddeb4 ! 9: 6a1504b1c3a bundle-uri: store fetch.bundleCreationToken\n @@ Commit message\n When checking the same bundle list twice, this strategy requires\n downloading the bundle with the maximum creationToken again, which is\n wasteful. The creationToken heuristic promises that the client will not\n - have a use for that bundle if its creationToken value is the at most the\n + have a use for that bundle if its creationToken value is at most the\n previous creationToken value.\n \n To prevent these wasteful downloads, create a fetch.bundleCreationToken\n @@ Commit message\n \n ## Documentation/config/fetch.txt ##\n @@ Documentation/config/fetch.txt: fetch.bundleURI::\n - \tfirst running `git fetch --bundle-uri=` immediately before\n - \t`git fetch `. See details of the `--bundle-uri` option in\n - \tlinkgit:git-fetch[1].\n + \tlinkgit:git-clone[1]. `git clone --bundle-uri` will set the\n + \t`fetch.bundleURI` value if the supplied bundle URI contains a bundle\n + \tlist that is organized for incremental fetches.\n +++\n ++If you modify this value and your repository has a `fetch.bundleCreationToken`\n ++value, then remove that `fetch.bundleCreationToken` value before fetching from\n ++the new bundle URI.\n +\n +fetch.bundleCreationToken::\n +\tWhen using `fetch.bundleURI` to fetch incrementally from a bundle\n @@ Documentation/config/fetch.txt: fetch.bundleURI::\n +\tThis value is used to prevent downloading bundles in the future\n +\tif the advertised `creationToken` is not strictly larger than this\n +\tvalue.\n +++\n ++The creation token values are chosen by the provider serving the specific\n ++bundle URI. If you modify the URI at `fetch.bundleURI`, then be sure to\n ++remove the value for the `fetch.bundleCreationToken` value before fetching.\n \n ## bundle-uri.c ##\n @@ bundle-uri.c: static int fetch_bundles_by_token(struct repository *r,\n {\n \tint cur;\n - \tint pop_or_push = 0;\n + \tint move_direction = 0;\n +\tconst char *creationTokenStr;\n -+\tuint64_t maxCreationToken;\n ++\tuint64_t maxCreationToken = 0, newMaxCreationToken = 0;\n \tstruct bundle_list_context ctx = {\n \t\t.r = r,\n \t\t.list = list,\n @@ bundle-uri.c: static int fetch_bundles_by_token(struct repository *r,\n \n - \tfor_all_bundles_in_list(list, insert_bundle, &sorted);\n + \tfor_all_bundles_in_list(list, append_bundle, &bundles);\n \n -+\tif (!sorted.nr) {\n -+\t\tfree(sorted.items);\n ++\tif (!bundles.nr) {\n ++\t\tfree(bundles.items);\n +\t\treturn 0;\n +\t}\n +\n - \tQSORT(sorted.items, sorted.nr, compare_creation_token);\n + \tQSORT(bundles.items, bundles.nr, compare_creation_token_decreasing);\n \n +\t/*\n +\t * If fetch.bundleCreationToken exists, parses to a uint64t, and\n @@ bundle-uri.c: static int fetch_bundles_by_token(struct repository *r,\n +\t\t\t\t \"fetch.bundlecreationtoken\",\n +\t\t\t\t &creationTokenStr) &&\n +\t sscanf(creationTokenStr, \"%\"PRIu64, &maxCreationToken) == 1 &&\n -+\t sorted.items[0]->creationToken <= maxCreationToken) {\n -+\t\tfree(sorted.items);\n ++\t bundles.items[0]->creationToken <= maxCreationToken) {\n ++\t\tfree(bundles.items);\n +\t\treturn 0;\n +\t}\n +\n \t/*\n - \t * Use a stack-based approach to download the bundles and attempt\n - \t * to unbundle them in decreasing order by creation token. If we\n + \t * Attempt to download and unbundle the minimum number of bundles by\n + \t * creationToken in decreasing order. If we fail to unbundle (after\n +@@ bundle-uri.c: static int fetch_bundles_by_token(struct repository *r,\n + \tcur = 0;\n + \twhile (cur >= 0 && cur < bundles.nr) {\n + \t\tstruct remote_bundle_info *bundle = bundles.items[cur];\n ++\n ++\t\t/*\n ++\t\t * If we need to dig into bundles below the previous\n ++\t\t * creation token value, then likely we are in an erroneous\n ++\t\t * state due to missing or invalid bundles. Halt the process\n ++\t\t * instead of continuing to download extra data.\n ++\t\t */\n ++\t\tif (bundle->creationToken <= maxCreationToken)\n ++\t\t\tbreak;\n ++\n + \t\tif (!bundle->file) {\n + \t\t\t/*\n + \t\t\t * Not downloaded yet. Try downloading.\n +@@ bundle-uri.c: static int fetch_bundles_by_token(struct repository *r,\n + \t\t\t\t */\n + \t\t\t\tmove_direction = -1;\n + \t\t\t\tbundle->unbundled = 1;\n ++\n ++\t\t\t\tif (bundle->creationToken > newMaxCreationToken)\n ++\t\t\t\t\tnewMaxCreationToken = bundle->creationToken;\n + \t\t\t}\n + \t\t}\n + \n @@ bundle-uri.c: stack_operation:\n - \t\tcur += pop_or_push;\n + \t\tcur += move_direction;\n \t}\n \n --\tfree(sorted.items);\n +-\tfree(bundles.items);\n -\n \t/*\n \t * We succeed if the loop terminates because 'cur' drops below\n @@ bundle-uri.c: stack_operation:\n \t */\n +\tif (cur < 0) {\n +\t\tstruct strbuf value = STRBUF_INIT;\n -+\t\tstrbuf_addf(&value, \"%\"PRIu64\"\", sorted.items[0]->creationToken);\n ++\t\tstrbuf_addf(&value, \"%\"PRIu64\"\", newMaxCreationToken);\n +\t\tif (repo_config_set_multivar_gently(ctx.r,\n +\t\t\t\t\t\t \"fetch.bundleCreationToken\",\n +\t\t\t\t\t\t value.buf, NULL, 0))\n @@ bundle-uri.c: stack_operation:\n +\t\tstrbuf_release(&value);\n +\t}\n +\n -+\tfree(sorted.items);\n ++\tfree(bundles.items);\n \treturn cur >= 0;\n }\n \n \n ## t/t5558-clone-bundle-uri.sh ##\n +@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n + \t\t\"$HTTPD_URL/smart/fetch.git\" clone-token-http &&\n + \n + \ttest_cmp_config -C clone-token-http \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n ++\ttest_cmp_config -C clone-token-http 1 fetch.bundlecreationtoken &&\n + \n + \tcat >expect <<-EOF &&\n + \t$HTTPD_URL/bundle-list\n +@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n + \tGIT_TRACE2_EVENT=\"$(pwd)/trace1.txt\" \\\n + \t\tgit -C clone-token-http fetch origin --no-tags \\\n + \t\trefs/heads/merge:refs/heads/merge &&\n ++\ttest_cmp_config -C clone-token-http 4 fetch.bundlecreationtoken &&\n + \n + \tcat >expect <<-EOF &&\n + \t$HTTPD_URL/bundle-list\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \t\t\"$HTTPD_URL/smart/fetch.git\" fetch-http-4 &&\n \n \ttest_cmp_config -C fetch-http-4 \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n +\ttest_cmp_config -C fetch-http-4 1 fetch.bundlecreationtoken &&\n \n - \t# The clone should copy two files: the list and bundle-1.\n - \ttest_bundle_downloaded bundle-list trace-clone.txt &&\n + \tcat >expect <<-EOF &&\n + \t$HTTPD_URL/bundle-list\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n + \t\tgit -C fetch-http-4 fetch origin --no-tags \\\n \t\trefs/heads/left:refs/heads/left \\\n \t\trefs/heads/right:refs/heads/right &&\n - \n +\ttest_cmp_config -C fetch-http-4 2 fetch.bundlecreationtoken &&\n -+\n - \t# This fetch should copy two files: the list and bundle-2.\n - \ttest_bundle_downloaded bundle-list trace1.txt &&\n - \ttest_bundle_downloaded bundle-2.bundle trace1.txt &&\n + \n + \tcat >expect <<-EOF &&\n + \t$HTTPD_URL/bundle-list\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \tEOF\n \ttest_cmp expect refs &&\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heurist\n +\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n +\t\trefs/heads/left:refs/heads/left \\\n +\t\trefs/heads/right:refs/heads/right &&\n -+\ttest_bundle_downloaded bundle-list trace1b.txt &&\n -+\t! test_bundle_downloaded bundle-1.bundle trace1b.txt &&\n -+\t! test_bundle_downloaded bundle-2.bundle trace1b.txt &&\n ++\n ++\tcat >expect <<-EOF &&\n ++\t$HTTPD_URL/bundle-list\n ++\tEOF\n ++\ttest_remote_https_urls actual &&\n ++\ttest_cmp expect actual &&\n +\n \tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n \t[bundle \"bundle-3\"]\n \t\turi = bundle-3.bundle\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n + \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n \t\tgit -C fetch-http-4 fetch origin --no-tags \\\n \t\trefs/heads/merge:refs/heads/merge &&\n - \n +\ttest_cmp_config -C fetch-http-4 4 fetch.bundlecreationtoken &&\n -+\n - \t# This fetch should copy three files: the list, bundle-3, and bundle-4.\n - \ttest_bundle_downloaded bundle-list trace2.txt &&\n - \ttest_bundle_downloaded bundle-4.bundle trace2.txt &&\n + \n + \tcat >expect <<-EOF &&\n + \t$HTTPD_URL/bundle-list\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \trefs/bundles/left\n \trefs/bundles/merge\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heurist\n +\t# No-op fetch\n +\tGIT_TRACE2_EVENT=\"$(pwd)/trace2b.txt\" \\\n +\t\tgit -C fetch-http-4 fetch origin &&\n -+\ttest_bundle_downloaded bundle-list trace2b.txt &&\n -+\t! test_bundle_downloaded bundle-1.bundle trace2b.txt &&\n -+\t! test_bundle_downloaded bundle-2.bundle trace2b.txt &&\n -+\t! test_bundle_downloaded bundle-3.bundle trace2b.txt &&\n -+\t! test_bundle_downloaded bundle-4.bundle trace2b.txt\n ++\n ++\tcat >expect <<-EOF &&\n ++\t$HTTPD_URL/bundle-list\n ++\tEOF\n ++\ttest_remote_https_urls actual &&\n ++\ttest_cmp expect actual\n '\n \n # Do not add tests here unless they use the HTTP server, as they will\n -: ----------- > 10: 676522615ad bundle-uri: test missing bundles with heuristic\n\n-- \ngitgitgadget\n"},{"id":"470921","messageId":"6a1504b1c3a24b45d48c093285dfcc9a3d6afd68.1674487310.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v2.git.1674487310.gitgitgadget@gmail.com","subject":"[PATCH v2 09/10] bundle-uri: store fetch.bundleCreationToken","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-23T15:21:49Z","receivedAt":"2023-01-23T15:23:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a bundle list specifies the \"creationToken\" heuristic, the Git\nclient downloads the list and then starts downloading bundles in\ndescending creationToken order. This process stops as soon as all\ndownloaded bundles can be applied to the repository (because all\nrequired commits are present in the repository or in the downloaded\nbundles).\n\nWhen checking the same bundle list twice, this strategy requires\ndownloading the bundle with the maximum creationToken again, which is\nwasteful. The creationToken heuristic promises that the client will not\nhave a use for that bundle if its creationToken value is at most the\nprevious creationToken value.\n\nTo prevent these wasteful downloads, create a fetch.bundleCreationToken\nconfig setting that the Git client sets after downloading bundles. This\nvalue allows skipping that maximum bundle download when this config\nvalue is the same value (or larger).\n\nTo test that this works correctly, we can insert some \"duplicate\"\nfetches into existing tests and demonstrate that only the bundle list is\ndownloaded.\n\nThe previous logic for downloading bundles by creationToken worked even\nif the bundle list was empty, but now we have logic that depends on the\nfirst entry of the list. Terminate early in the (non-sensical) case of\nan empty bundle list.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/fetch.txt | 16 ++++++++++++\n bundle-uri.c | 48 ++++++++++++++++++++++++++++++++--\n t/t5558-clone-bundle-uri.sh | 29 +++++++++++++++++++-\n 3 files changed, 90 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt\nindex 244f44d460f..568f0f75b30 100644\n--- a/Documentation/config/fetch.txt\n+++ b/Documentation/config/fetch.txt\n@@ -104,3 +104,19 @@ fetch.bundleURI::\n \tlinkgit:git-clone[1]. `git clone --bundle-uri` will set the\n \t`fetch.bundleURI` value if the supplied bundle URI contains a bundle\n \tlist that is organized for incremental fetches.\n++\n+If you modify this value and your repository has a `fetch.bundleCreationToken`\n+value, then remove that `fetch.bundleCreationToken` value before fetching from\n+the new bundle URI.\n+\n+fetch.bundleCreationToken::\n+\tWhen using `fetch.bundleURI` to fetch incrementally from a bundle\n+\tlist that uses the \"creationToken\" heuristic, this config value\n+\tstores the maximum `creationToken` value of the downloaded bundles.\n+\tThis value is used to prevent downloading bundles in the future\n+\tif the advertised `creationToken` is not strictly larger than this\n+\tvalue.\n++\n+The creation token values are chosen by the provider serving the specific\n+bundle URI. If you modify the URI at `fetch.bundleURI`, then be sure to\n+remove the value for the `fetch.bundleCreationToken` value before fetching.\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 162a9276f31..691853b2c56 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -487,6 +487,8 @@ static int fetch_bundles_by_token(struct repository *r,\n {\n \tint cur;\n \tint move_direction = 0;\n+\tconst char *creationTokenStr;\n+\tuint64_t maxCreationToken = 0, newMaxCreationToken = 0;\n \tstruct bundle_list_context ctx = {\n \t\t.r = r,\n \t\t.list = list,\n@@ -500,8 +502,27 @@ static int fetch_bundles_by_token(struct repository *r,\n \n \tfor_all_bundles_in_list(list, append_bundle, &bundles);\n \n+\tif (!bundles.nr) {\n+\t\tfree(bundles.items);\n+\t\treturn 0;\n+\t}\n+\n \tQSORT(bundles.items, bundles.nr, compare_creation_token_decreasing);\n \n+\t/*\n+\t * If fetch.bundleCreationToken exists, parses to a uint64t, and\n+\t * is not strictly smaller than the maximum creation token in the\n+\t * bundle list, then do not download any bundles.\n+\t */\n+\tif (!repo_config_get_value(r,\n+\t\t\t\t \"fetch.bundlecreationtoken\",\n+\t\t\t\t &creationTokenStr) &&\n+\t sscanf(creationTokenStr, \"%\"PRIu64, &maxCreationToken) == 1 &&\n+\t bundles.items[0]->creationToken <= maxCreationToken) {\n+\t\tfree(bundles.items);\n+\t\treturn 0;\n+\t}\n+\n \t/*\n \t * Attempt to download and unbundle the minimum number of bundles by\n \t * creationToken in decreasing order. If we fail to unbundle (after\n@@ -522,6 +543,16 @@ static int fetch_bundles_by_token(struct repository *r,\n \tcur = 0;\n \twhile (cur >= 0 && cur < bundles.nr) {\n \t\tstruct remote_bundle_info *bundle = bundles.items[cur];\n+\n+\t\t/*\n+\t\t * If we need to dig into bundles below the previous\n+\t\t * creation token value, then likely we are in an erroneous\n+\t\t * state due to missing or invalid bundles. Halt the process\n+\t\t * instead of continuing to download extra data.\n+\t\t */\n+\t\tif (bundle->creationToken <= maxCreationToken)\n+\t\t\tbreak;\n+\n \t\tif (!bundle->file) {\n \t\t\t/*\n \t\t\t * Not downloaded yet. Try downloading.\n@@ -561,6 +592,9 @@ static int fetch_bundles_by_token(struct repository *r,\n \t\t\t\t */\n \t\t\t\tmove_direction = -1;\n \t\t\t\tbundle->unbundled = 1;\n+\n+\t\t\t\tif (bundle->creationToken > newMaxCreationToken)\n+\t\t\t\t\tnewMaxCreationToken = bundle->creationToken;\n \t\t\t}\n \t\t}\n \n@@ -575,14 +609,24 @@ stack_operation:\n \t\tcur += move_direction;\n \t}\n \n-\tfree(bundles.items);\n-\n \t/*\n \t * We succeed if the loop terminates because 'cur' drops below\n \t * zero. The other case is that we terminate because 'cur'\n \t * reaches the end of the list, so we have a failure no matter\n \t * which bundles we apply from the list.\n \t */\n+\tif (cur < 0) {\n+\t\tstruct strbuf value = STRBUF_INIT;\n+\t\tstrbuf_addf(&value, \"%\"PRIu64\"\", newMaxCreationToken);\n+\t\tif (repo_config_set_multivar_gently(ctx.r,\n+\t\t\t\t\t\t \"fetch.bundleCreationToken\",\n+\t\t\t\t\t\t value.buf, NULL, 0))\n+\t\t\twarning(_(\"failed to store maximum creation token\"));\n+\n+\t\tstrbuf_release(&value);\n+\t}\n+\n+\tfree(bundles.items);\n \treturn cur >= 0;\n }\n \ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 7deeb4b8ad1..9c2b7934b9b 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -433,6 +433,7 @@ test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n \t\t\"$HTTPD_URL/smart/fetch.git\" clone-token-http &&\n \n \ttest_cmp_config -C clone-token-http \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n+\ttest_cmp_config -C clone-token-http 1 fetch.bundlecreationtoken &&\n \n \tcat >expect <<-EOF &&\n \t$HTTPD_URL/bundle-list\n@@ -468,6 +469,7 @@ test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace1.txt\" \\\n \t\tgit -C clone-token-http fetch origin --no-tags \\\n \t\trefs/heads/merge:refs/heads/merge &&\n+\ttest_cmp_config -C clone-token-http 4 fetch.bundlecreationtoken &&\n \n \tcat >expect <<-EOF &&\n \t$HTTPD_URL/bundle-list\n@@ -511,6 +513,7 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \t\t\"$HTTPD_URL/smart/fetch.git\" fetch-http-4 &&\n \n \ttest_cmp_config -C fetch-http-4 \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n+\ttest_cmp_config -C fetch-http-4 1 fetch.bundlecreationtoken &&\n \n \tcat >expect <<-EOF &&\n \t$HTTPD_URL/bundle-list\n@@ -538,6 +541,7 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \t\tgit -C fetch-http-4 fetch origin --no-tags \\\n \t\trefs/heads/left:refs/heads/left \\\n \t\trefs/heads/right:refs/heads/right &&\n+\ttest_cmp_config -C fetch-http-4 2 fetch.bundlecreationtoken &&\n \n \tcat >expect <<-EOF &&\n \t$HTTPD_URL/bundle-list\n@@ -555,6 +559,18 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \tEOF\n \ttest_cmp expect refs &&\n \n+\t# No-op fetch\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace1b.txt\" \\\n+\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n+\t\trefs/heads/left:refs/heads/left \\\n+\t\trefs/heads/right:refs/heads/right &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n \tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n \t[bundle \"bundle-3\"]\n \t\turi = bundle-3.bundle\n@@ -570,6 +586,7 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n \t\tgit -C fetch-http-4 fetch origin --no-tags \\\n \t\trefs/heads/merge:refs/heads/merge &&\n+\ttest_cmp_config -C fetch-http-4 4 fetch.bundlecreationtoken &&\n \n \tcat >expect <<-EOF &&\n \t$HTTPD_URL/bundle-list\n@@ -588,7 +605,17 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \trefs/bundles/left\n \trefs/bundles/merge\n \tEOF\n-\ttest_cmp expect refs\n+\ttest_cmp expect refs &&\n+\n+\t# No-op fetch\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2b.txt\" \\\n+\t\tgit -C fetch-http-4 fetch origin &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual\n '\n \n # Do not add tests here unless they use the HTTP server, as they will\n-- \ngitgitgadget\n\n"},{"id":"470922","messageId":"676522615ad0e8f24099ef35a0f39367e5f688ae.1674487310.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v2.git.1674487310.gitgitgadget@gmail.com","subject":"[PATCH v2 10/10] bundle-uri: test missing bundles with heuristic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-23T15:21:50Z","receivedAt":"2023-01-23T15:23:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe creationToken heuristic uses a different mechanism for downloading\nbundles from the \"standard\" approach. Specifically: it uses a concrete\norder based on the creationToken values and attempts to download as few\nbundles as possible. It also modifies local config to store a value for\nfuture fetches to avoid downloading bundles, if possible.\n\nHowever, if any of the individual bundles has a failed download, then\nthe logic for the ordering comes into question. It is important to avoid\ninfinite loops, assigning invalid creation token values in config, but\nalso to be opportunistic as possible when downloading as many bundles as\nseem appropriate.\n\nThese tests were used to inform the implementation of\nfetch_bundles_by_token() in bundle-uri.c, but are being added\nindependently here to allow focusing on faulty downloads. There may be\nmore cases that could be added that result in modifications to\nfetch_bundles_by_token() as interesting data shapes reveal themselves in\nreal scenarios.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5558-clone-bundle-uri.sh | 400 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 400 insertions(+)\n\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 9c2b7934b9b..e3ccfe872c4 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -618,6 +618,406 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'creationToken heuristic with failed downloads (clone)' '\n+\ttest_when_finished rm -rf download-* trace*.txt &&\n+\n+\t# Case 1: base bundle does not exist, nothing can unbundle\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = fake.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone-1.txt\" \\\n+\tgit clone --single-branch --branch=base \\\n+\t\t--bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\t\"$HTTPD_URL/smart/fetch.git\" download-1 &&\n+\n+\t# Bundle failure does not set these configs.\n+\ttest_must_fail git -C download-1 config fetch.bundleuri &&\n+\ttest_must_fail git -C download-1 config fetch.bundlecreationtoken &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-4.bundle\n+\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/bundle-2.bundle\n+\t$HTTPD_URL/fake.bundle\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# All bundles failed to unbundle\n+\tgit -C download-1 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\ttest_must_be_empty refs &&\n+\n+\t# Case 2: middle bundle does not exist, only two bundles can unbundle\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = fake.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone-2.txt\" \\\n+\tgit clone --single-branch --branch=base \\\n+\t\t--bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\t\"$HTTPD_URL/smart/fetch.git\" download-2 &&\n+\n+\t# Bundle failure does not set these configs.\n+\ttest_must_fail git -C download-2 config fetch.bundleuri &&\n+\ttest_must_fail git -C download-2 config fetch.bundlecreationtoken &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-4.bundle\n+\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/fake.bundle\n+\t$HTTPD_URL/bundle-1.bundle\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Only base bundle unbundled.\n+\tgit -C download-2 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/right\n+\tEOF\n+\ttest_cmp expect refs &&\n+\n+\t# Case 3: top bundle does not exist, rest unbundle fine.\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = fake.bundle\n+\t\tcreationToken = 4\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone-3.txt\" \\\n+\tgit clone --single-branch --branch=base \\\n+\t\t--bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\t\"$HTTPD_URL/smart/fetch.git\" download-3 &&\n+\n+\t# As long as we have continguous successful downloads,\n+\t# we _do_ set these configs.\n+\ttest_cmp_config -C download-3 \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n+\ttest_cmp_config -C download-3 3 fetch.bundlecreationtoken &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/fake.bundle\n+\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/bundle-2.bundle\n+\t$HTTPD_URL/bundle-1.bundle\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# All bundles failed to unbundle\n+\tgit -C download-3 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/right\n+\tEOF\n+\ttest_cmp expect refs\n+'\n+\n+# Expand the bundle list to include other interesting shapes, specifically\n+# interesting for use when fetching from a previous state.\n+#\n+# ---------------- bundle-7\n+# 7\n+# _/|\\_\n+# ---/--|--\\------ bundle-6\n+# 5 | 6\n+# --|---|---|----- bundle-4\n+# | 4 |\n+# | / \\ /\n+# --|-|---|/------ bundle-3 (the client will be caught up to this point.)\n+# \\ | 3\n+# ---\\|---|------- bundle-2\n+# 2 |\n+# ----|---|------- bundle-1\n+# \\ /\n+# 1\n+# |\n+# (previous commits)\n+test_expect_success 'expand incremental bundle list' '\n+\t(\n+\t\tcd clone-from &&\n+\t\tgit checkout -b lefter left &&\n+\t\ttest_commit 5 &&\n+\t\tgit checkout -b righter right &&\n+\t\ttest_commit 6 &&\n+\t\tgit checkout -b top lefter &&\n+\t\tgit merge -m \"7\" merge righter &&\n+\n+\t\tgit bundle create bundle-6.bundle lefter righter --not left right &&\n+\t\tgit bundle create bundle-7.bundle top --not lefter merge righter &&\n+\n+\t\tcp bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\"\n+\t) &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/fetch.git\" fetch origin +refs/heads/*:refs/heads/*\n+'\n+\n+test_expect_success 'creationToken heuristic with failed downloads (fetch)' '\n+\ttest_when_finished rm -rf download-* trace*.txt &&\n+\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\tEOF\n+\n+\tgit clone --single-branch --branch=left \\\n+\t\t--bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\t\"$HTTPD_URL/smart/fetch.git\" fetch-base &&\n+\ttest_cmp_config -C fetch-base \"$HTTPD_URL/bundle-list\" fetch.bundleURI &&\n+\ttest_cmp_config -C fetch-base 3 fetch.bundleCreationToken &&\n+\n+\t# Case 1: all bundles exist: successful unbundling of all bundles\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\n+\t[bundle \"bundle-6\"]\n+\t\turi = bundle-6.bundle\n+\t\tcreationToken = 6\n+\n+\t[bundle \"bundle-7\"]\n+\t\turi = bundle-7.bundle\n+\t\tcreationToken = 7\n+\tEOF\n+\n+\tcp -r fetch-base fetch-1 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-fetch-1.txt\" \\\n+\t\tgit -C fetch-1 fetch origin &&\n+\ttest_cmp_config -C fetch-1 7 fetch.bundlecreationtoken &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-7.bundle\n+\t$HTTPD_URL/bundle-6.bundle\n+\t$HTTPD_URL/bundle-4.bundle\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Check which bundles have unbundled by refs\n+\tgit -C fetch-1 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/lefter\n+\trefs/bundles/merge\n+\trefs/bundles/right\n+\trefs/bundles/righter\n+\trefs/bundles/top\n+\tEOF\n+\ttest_cmp expect refs &&\n+\n+\t# Case 2: middle bundle does not exist, only bundle-4 can unbundle\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\n+\t[bundle \"bundle-6\"]\n+\t\turi = fake.bundle\n+\t\tcreationToken = 6\n+\n+\t[bundle \"bundle-7\"]\n+\t\turi = bundle-7.bundle\n+\t\tcreationToken = 7\n+\tEOF\n+\n+\tcp -r fetch-base fetch-2 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-fetch-2.txt\" \\\n+\t\tgit -C fetch-2 fetch origin &&\n+\n+\t# Since bundle-7 fails to unbundle, do not update creation token.\n+\ttest_cmp_config -C fetch-2 3 fetch.bundlecreationtoken &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-7.bundle\n+\t$HTTPD_URL/fake.bundle\n+\t$HTTPD_URL/bundle-4.bundle\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Check which bundles have unbundled by refs\n+\tgit -C fetch-2 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/merge\n+\trefs/bundles/right\n+\tEOF\n+\ttest_cmp expect refs &&\n+\n+\t# Case 3: top bundle does not exist, rest unbundle fine.\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\n+\t[bundle \"bundle-6\"]\n+\t\turi = bundle-6.bundle\n+\t\tcreationToken = 6\n+\n+\t[bundle \"bundle-7\"]\n+\t\turi = fake.bundle\n+\t\tcreationToken = 7\n+\tEOF\n+\n+\tcp -r fetch-base fetch-3 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-fetch-3.txt\" \\\n+\t\tgit -C fetch-3 fetch origin &&\n+\n+\t# As long as we have continguous successful downloads,\n+\t# we _do_ set the maximum creation token.\n+\ttest_cmp_config -C fetch-3 6 fetch.bundlecreationtoken &&\n+\n+\t# NOTE: the fetch skips bundle-4 since bundle-6 successfully\n+\t# unbundles itself and bundle-7 failed to download.\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/fake.bundle\n+\t$HTTPD_URL/bundle-6.bundle\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Check which bundles have unbundled by refs\n+\tgit -C fetch-3 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/lefter\n+\trefs/bundles/right\n+\trefs/bundles/righter\n+\tEOF\n+\ttest_cmp expect refs\n+'\n+\n # Do not add tests here unless they use the HTTP server, as they will\n # not run unless the HTTP dependencies exist.\n \n-- \ngitgitgadget\n"},{"id":"470923","messageId":"59e57e049683e42248c270b3bfcad2d72769219d.1674487310.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v2.git.1674487310.gitgitgadget@gmail.com","subject":"[PATCH v2 08/10] fetch: fetch from an external bundle URI","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-23T15:21:48Z","receivedAt":"2023-01-23T15:23:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a user specifies a URI via 'git clone --bundle-uri', that URI may\nbe a bundle list that advertises a 'bundle.heuristic' value. In that\ncase, the Git client stores a 'fetch.bundleURI' config value storing\nthat URI.\n\nTeach 'git fetch' to check for this config value and download bundles\nfrom that URI before fetching from the Git remote(s). Likely, the bundle\nprovider has configured a heuristic (such as \"creationToken\") that will\nallow the Git client to download only a portion of the bundles before\ncontinuing the fetch.\n\nSince this URI is completely independent of the remote server, we want\nto be sure that we connect to the bundle URI before creating a\nconnection to the Git remote. We do not want to hold a stateful\nconnection for too long if we can avoid it.\n\nTo test that this works correctly, extend the previous tests that set\n'fetch.bundleURI' to do follow-up fetches. The bundle list is updated\nincrementally at each phase to demonstrate that the heuristic avoids\ndownloading older bundles. This includes the middle fetch downloading\nthe objects in bundle-3.bundle from the Git remote, and therefore not\nneeding that bundle in the third fetch.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/fetch.c | 7 +++\n t/t5558-clone-bundle-uri.sh | 113 +++++++++++++++++++++++++++++++++++-\n 2 files changed, 119 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 7378cafeec9..f101e454dc9 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -29,6 +29,7 @@\n #include \"commit-graph.h\"\n #include \"shallow.h\"\n #include \"worktree.h\"\n+#include \"bundle-uri.h\"\n \n #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n \n@@ -2109,6 +2110,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,\n int cmd_fetch(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n+\tconst char *bundle_uri;\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n \tstruct remote *remote = NULL;\n \tint result = 0;\n@@ -2194,6 +2196,11 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (dry_run)\n \t\twrite_fetch_head = 0;\n \n+\tif (!git_config_get_string_tmp(\"fetch.bundleuri\", &bundle_uri)) {\n+\t\tif (fetch_bundle_uri(the_repository, bundle_uri, NULL))\n+\t\t\twarning(_(\"failed to fetch bundles from '%s'\"), bundle_uri);\n+\t}\n+\n \tif (all) {\n \t\tif (argc == 1)\n \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex b2d15e141ca..7deeb4b8ad1 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -440,7 +440,55 @@ test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n \tEOF\n \n \ttest_remote_https_urls actual &&\n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\n+\t# We now have only one bundle ref.\n+\tgit -C clone-token-http for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\tEOF\n+\ttest_cmp expect refs &&\n+\n+\t# Add remaining bundles, exercising the \"deepening\" strategy\n+\t# for downloading via the creationToken heurisitc.\n+\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace1.txt\" \\\n+\t\tgit -C clone-token-http fetch origin --no-tags \\\n+\t\trefs/heads/merge:refs/heads/merge &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-4.bundle\n+\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/bundle-2.bundle\n+\tEOF\n+\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# We now have all bundle refs.\n+\tgit -C clone-token-http for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/merge\n+\trefs/bundles/right\n+\tEOF\n+\ttest_cmp expect refs\n '\n \n test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n@@ -477,6 +525,69 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \tcat >expect <<-\\EOF &&\n \trefs/bundles/base\n \tEOF\n+\ttest_cmp expect refs &&\n+\n+\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\t# Fetch the objects for bundle-2 _and_ bundle-3.\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace1.txt\" \\\n+\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n+\t\trefs/heads/left:refs/heads/left \\\n+\t\trefs/heads/right:refs/heads/right &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-2.bundle\n+\tEOF\n+\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# received left from bundle-2\n+\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect refs &&\n+\n+\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\tEOF\n+\n+\t# This fetch should skip bundle-3.bundle, since its objects are\n+\t# already local (we have the requisite commits for bundle-4.bundle).\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n+\t\trefs/heads/merge:refs/heads/merge &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-4.bundle\n+\tEOF\n+\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# received merge ref from bundle-4, but right is missing\n+\t# because we did not download bundle-3.\n+\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/merge\n+\tEOF\n \ttest_cmp expect refs\n '\n \n-- \ngitgitgadget\n\n"},{"id":"470943","messageId":"xmqqsfg1m8l6.fsf@gitster.g","threadId":"59040","inReplyTo":"b3828725bc8f8887b9b4777a0e3d84224a427f31.1674487310.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 01/10] bundle: optionally skip reachability walk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-23T18:03:49Z","receivedAt":"2023-01-23T18:03:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> When unbundling a bundle, the verify_bundle() method checks two things\n> with regards to the prerequisite commits:\n>\n> 1. Those commits are in the object store, and\n> 2. Those commits are reachable from refs.\n>\n> During testing of the bundle URI feature, where multiple bundles are\n> unbundled in the same process, the ref store did not appear to be\n> refreshing with the new refs/bundles/* references added within that\n> process. This caused the second half -- the reachability walk -- report\n> that some commits were not present, despite actually being present.\n>\n> One way to attempt to fix this would be to create a way to force-refresh\n> the ref state. That would correct this for these cases where the\n> refs/bundles/* references have been updated. However, this still is an\n> expensive operation in a repository with many references.\n>\n> Instead, optionally allow callers to skip this portion by instead just\n> checking for presence within the object store. Use this when unbundling\n> in bundle-uri.c.\n\nThis step is new in this round.\n\nI am assuming that this approach is to avoid repeated \"now we\nunbundled one, let's spend enormous cycles to update the in-core\nview of the ref store before processing the next bundle\"---instead\nwe unbundle all, assuming the prerequisites for each and every\nbundle are satisfied.\n\nI am OK as long as we check the assumption holds true at the end;\nthis looks like a good optimization.\n\n"},{"id":"470945","messageId":"eae85534-89c9-6eff-69d5-7d4b2be85fb6@github.com","threadId":"59040","inReplyTo":"xmqqsfg1m8l6.fsf@gitster.g","subject":"Re: [PATCH v2 01/10] bundle: optionally skip reachability walk","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-23T18:24:26Z","receivedAt":"2023-01-23T18:25:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/23/2023 1:03 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> When unbundling a bundle, the verify_bundle() method checks two things\n>> with regards to the prerequisite commits:\n>>\n>> 1. Those commits are in the object store, and\n>> 2. Those commits are reachable from refs.\n>>\n>> During testing of the bundle URI feature, where multiple bundles are\n>> unbundled in the same process, the ref store did not appear to be\n>> refreshing with the new refs/bundles/* references added within that\n>> process. This caused the second half -- the reachability walk -- report\n>> that some commits were not present, despite actually being present.\n>>\n>> One way to attempt to fix this would be to create a way to force-refresh\n>> the ref state. That would correct this for these cases where the\n>> refs/bundles/* references have been updated. However, this still is an\n>> expensive operation in a repository with many references.\n>>\n>> Instead, optionally allow callers to skip this portion by instead just\n>> checking for presence within the object store. Use this when unbundling\n>> in bundle-uri.c.\n> \n> This step is new in this round.\n> \n> I am assuming that this approach is to avoid repeated \"now we\n> unbundled one, let's spend enormous cycles to update the in-core\n> view of the ref store before processing the next bundle\"---instead\n> we unbundle all, assuming the prerequisites for each and every\n> bundle are satisfied.\n\nWe are specifically removing the requirement that the objects are\nreachable from refs, we still check that the objects are in the\nobject store. Thus, we can only be in a bad state afterwards if\nthe required objects for a bundle were in the object store,\npreviously unreachable, and one of these two things happened:\n\n1. Some objects reachable from those required commits were already\n missing in the repository (so the repo's object store was broken\n but only for some unreachable objects).\n\n2. A GC pruned those objects between verifying the bundle and\n writing the refs/bundles/* refs after unbundling.\n\nI think we should trust the repository to not be in the first state,\nbut the race condition in the second option will create a state\nwhere we have missing objects that are now reachable from refs.\n \n> I am OK as long as we check the assumption holds true at the end;\n> this looks like a good optimization.\n \nSo are you recommending that we verify all objects reachable from\nthe new refs/bundles/* are present after unbundling? That would\nprevent the possibility of a GC race, but at some significant run-\ntime performance costs. Do we do the same as we unpack from a\nfetch? Do we apply the same scrutiny to the objects during\nunbundling as we do from a fetched pack? They both use 'git\nindex-pack --stdin --fix-thin', so my guess is that we do the same\namount of checks for both cases.\n\nSince this is only one of multiple ways to add objects that depend\non possibly-unreachable objects, my preferred way to avoid the\nGC race is by using care in the GC process itself (such as the new\n--expire-to option to recover from these races).\n\nThanks,\n-Stolee\n"},{"id":"470949","messageId":"xmqqilgxm2ky.fsf@gitster.g","threadId":"59040","inReplyTo":"eae85534-89c9-6eff-69d5-7d4b2be85fb6@github.com","subject":"Re: [PATCH v2 01/10] bundle: optionally skip reachability walk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-23T20:13:33Z","receivedAt":"2023-01-23T20:13:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> We are specifically removing the requirement that the objects are\n> reachable from refs, we still check that the objects are in the\n> object store. Thus, we can only be in a bad state afterwards if\n> the required objects for a bundle were in the object store,\n> previously unreachable, and one of these two things happened:\n>\n> 1. Some objects reachable from those required commits were already\n> missing in the repository (so the repo's object store was broken\n> but only for some unreachable objects).\n\nA repository having some unreachable objects floating in the object\nstore is not corrupt. As long as all the objects reachable from refs\nare connected, that is a perfectly sane state.\n\nBut allowing unbundling with the sanity check loosened WILL corrupt\nit, at the moment you point some objects from the bundle with refs.\n\n> I think we should trust the repository to not be in the first state,\n\nSo, I think this line of thought is simply mistaken.\n\n>> I am OK as long as we check the assumption holds true at the end;\n>> this looks like a good optimization.\n> \n> So are you recommending that we verify all objects reachable from\n> the new refs/bundles/* are present after unbundling?\n\nMaking sure that prerequisites are connected will reduce the span of\nthe DAG we would need to verify. After unbundling all bundles, but\nbefore updating the refs to point at the tips in the bundles, if we\ncan make sure that these prerequisite objects named in the bundles\nare reachable from the tips recorded in the bundles, while stopping\nthe traversal at the tips of original refs (remember: we have only\nupdated objects in the object store, but haven't updated the refs from\nthe bundles), that would allow us to make sure that the updates to\nrefs proposed by the bundles will not corrupt the repository.\n\n"},{"id":"470950","messageId":"xmqqedrlm018.fsf@gitster.g","threadId":"59040","inReplyTo":"eae85534-89c9-6eff-69d5-7d4b2be85fb6@github.com","subject":"Re: [PATCH v2 01/10] bundle: optionally skip reachability walk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-23T21:08:35Z","receivedAt":"2023-01-23T21:08:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Do we do the same as we unpack from a fetch?\n\nWe should.\n\nWe only consider tips of refs and objects that are reachable from\nthem to be \"present\", and there may be random objects that float in\nthe object store without any guarantee that no the objects that\nought to be reachable from them are missing from the object store,\nbut they do not play part in the common ancestor discovery.\n\nAnd then after we unpack, we ensure that the proposed updates to\nrefs made by the fetch operation will not corrupt the repository.\nThis can be guaranteed by making sure that objects to be placed at\nthe updated tip can all reach some existing tips of refs. We trust\nrefs before the operation (in this case, 'git fetch') begins. We\nensure that refs after the operation can be trusted before updating\nthem. Where \"trust\" here means \"objects pointed at by them are\nconnected.\n\n"},{"id":"470952","messageId":"xmqqtu0glw81.fsf@gitster.g","threadId":"59040","inReplyTo":"xmqqilgxm2ky.fsf@gitster.g","subject":"Re: [PATCH v2 01/10] bundle: optionally skip reachability walk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-23T22:30:54Z","receivedAt":"2023-01-23T22:30:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> A repository having some unreachable objects floating in the object\n> store is not corrupt. As long as all the objects reachable from refs\n> are connected, that is a perfectly sane state.\n>\n> But allowing unbundling with the sanity check loosened WILL corrupt\n> it, at the moment you point some objects from the bundle with refs.\n\nWhile all of the above is true, I think existing check done by\nbundle.c::verify_bundle() is stricter than necessary. We make sure\nthat the prerequiste objects exist and are reachable from the refs.\nBut for the purpose of ensuring the health of the repo after the\noperation, it is also OK if the prerequisite objects exist and they\npass connected.c::check_connected() test to reach existing refs.\nverify_bundle() that is used in unbundle() does not allow it.\n\nIn a simplest case, with a single ref and a single strand of pearls\nhistory, with a few cruft history that are connected to the main\nhistory but are not anchored by any ref (e.g. the tip of 'main' was\nrewound recently and we haven't done any GC).\n\n 7---8 (cruft)\n /\n 0---1---2---3---4---5---6 refs/heads/main\n\nAnd they have another repository created back when '5' was the\nlatest and greatest, which built three commits on top of it.\n\n 0---1---2---3---4---5---A---B---C\n\nThey create a bundle of 5..C to update us to C. In the meantime, we\nhave created three commits ourselves (6, 7, and 8) but threw two\naway.\n\nIf a bundle requires commit '5', because it is reachable from an\nexisting ref (which points at the 'main' branch), we can unbundle it\nand point a ref at the tip of the history contained within the\nbundle safely. Commit '5' being pointed by a ref means the commit,\nits ancestors, and all trees and blobs referenced are available to\nthe repository (some may be fetched lazily from promisor), and\nunless the producer lied and placed unconnected data in the bundle,\nunpacking a bundle on top of '5' should give us all the objects that\nare needed to complete the new tip proposed by the bundle (e.g. 'C').\n\n 7---8 (cruft)\n /\n 0---1---2---3---4---5---6 refs/heads/main\n \\\n A---B---C refs/bundle-1/main\n\nAnd existing check that I called \"sticter than necessary\" easily\nmakes sure that it is safe to point 'C' with our ref.\n\nImagine another party cloned us back when 'main' was pointing at '8'\n(which we since then have rewound), and built a few commits on it.\n\n X---Y refs/bundle-2/main\n /\n 7---8 (cruft)\n /\n 0---1---2---3---4---5---6 refs/heads/main\n\nAs they did not know we'd rewind and discard 7 and 8, they would\nhave created their bundle to cover 8..Y. The current test will fail\nbecause '8' that is prerequisite is not reachable from any ref on\nour side. But that is too pessimistic.\n\nAs long as we haven't garbage collected so that all objects\nreachable from '7' and '8' are available to this repository,\nhowever, we should be able to unbundle the bundle that has '8' as\nits prerequisite. For that, we only need that '8' passes the\ncheck_connected() check, which essentially means \"we shouldn't find\nany missing link while traversing history from '8' that stops at any\nexisting refs\".\n\nAgain this relies on the fact that unbundling code makes sure that\nincoming data is fully connected (i.e. bundle producer did not lie\nabout the prerequisite).\n\n"},{"id":"470969","messageId":"771a2993-85bd-0831-0977-24204f84e206@github.com","threadId":"59040","inReplyTo":"xmqqtu0glw81.fsf@gitster.g","subject":"Re: [PATCH v2 01/10] bundle: optionally skip reachability walk","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-24T12:27:04Z","receivedAt":"2023-01-24T12:27:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/23/2023 5:30 PM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n>> A repository having some unreachable objects floating in the object\n>> store is not corrupt. As long as all the objects reachable from refs\n>> are connected, that is a perfectly sane state.\n>>\n>> But allowing unbundling with the sanity check loosened WILL corrupt\n>> it, at the moment you point some objects from the bundle with refs.\n> \n> While all of the above is true, I think existing check done by\n> bundle.c::verify_bundle() is stricter than necessary. We make sure\n> that the prerequiste objects exist and are reachable from the refs.\n> But for the purpose of ensuring the health of the repo after the\n> operation, it is also OK if the prerequisite objects exist and they\n> pass connected.c::check_connected() test to reach existing refs.\n> verify_bundle() that is used in unbundle() does not allow it.\n\nThank you for all of the detailed explanation, here and in other\nmessages.\n\nI'll focus on this area today and see what I can learn and how I\ncan approach this problem in a different way. The current options\nthat I see are:\n\n 1. Leave verify_bundle() as-is and figure out how to refresh the\n refs. (This would remain a stricter check than necessary.)\n\n 2. Find out how to modify verify_bundle() so it can do the more\n relaxed connectivity check.\n\n 3. Take the connectivity check that fetch uses before updating\n refs and add that check before updating refs in the bundle URI\n code.\n\nThere could also be a combination of (2) and (3), or others I have\nnot considered until I go poking around in the code.\n\nI'll let you know what I find.\n\nThanks,\n-Stolee\n"},{"id":"470971","messageId":"01f97aff-58a1-ef2c-e668-d37ea513c64e@github.com","threadId":"59040","inReplyTo":"771a2993-85bd-0831-0977-24204f84e206@github.com","subject":"[PATCH v2.5 01/11] bundle: test unbundling with incomplete history","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-24T14:14:36Z","receivedAt":"2023-01-24T14:14:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/24/2023 7:27 AM, Derrick Stolee wrote:\n\n> I'll focus on this area today and see what I can learn and how I\n> can approach this problem in a different way.\n\nThe first thing I did was try to figure out how things work today,\nso I created this test case. It appears we were not testing this\nat all previously.\n\nThis is just a candidate replacement for v3, so don't worry about\napplying it until I re-roll.\n\nThanks,\n-Stolee\n\n--- >8 ---\n\nFrom f9b0cc872ac44892fe6b1c973f16b35edfdc5b20 Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Tue, 24 Jan 2023 08:47:19 -0500\nSubject: [PATCH v2.5 01/11] bundle: test unbundling with incomplete history\n\nWhen verifying a bundle, Git checks first that all prerequisite commits\nexist in the object store, then adds an additional check: those\nprerequisite commits must be reachable from references in the\nrepository.\n\nThis check is stronger than what is checked for refs being added during\n'git fetch', which simply guarantees that the new refs have a complete\nhistory up to the point where it intersects with the current reachable\nhistory.\n\nHowever, we also do not have any tests that check the behavior under\nthis condition. Create a test that demonstrates its behavior.\n\nIn order to construct a broken history, perform a shallow clone of a\nrepository with a linear history, but whose default branch ('base') has\na single commit, so dropping the shallow markers leaves a complete\nhistory from that reference. However, the 'tip' reference adds a\nshallow commit whose parent is missing in the cloned repository. Trying\nto unbundle a bundle with the 'tip' as a prerequisite will succeed past\nthe object store check and move into the reachability check.\n\nThe two errors that are reported are of this form:\n\n error: Could not read \n fatal: Failed to traverse parents of commit \n\nThese messages are not particularly helpful for the person running the\nunbundle command, but they do prevent the command from succeeding.\n\nSigned-off-by: Derrick Stolee \n---\n t/t6020-bundle-misc.sh | 40 ++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 40 insertions(+)\n\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 3a1cf30b1d7..38dbbf89155 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -566,4 +566,44 @@ test_expect_success 'cloning from filtered bundle has useful error' '\n \tgrep \"cannot clone from filtered bundle\" err\n '\n\n+test_expect_success 'verify catches unreachable, broken prerequisites' '\n+\ttest_when_finished rm -rf clone-from clone-to &&\n+\tgit init clone-from &&\n+\t(\n+\t\tcd clone-from &&\n+\t\tgit checkout -b base &&\n+\t\ttest_commit A &&\n+\t\tgit checkout -b tip &&\n+\t\tgit commit --allow-empty -m \"will drop by shallow\" &&\n+\t\tgit commit --allow-empty -m \"will keep by shallow\" &&\n+\t\tgit commit --allow-empty -m \"for bundle, not clone\" &&\n+\t\tgit bundle create tip.bundle tip~1..tip &&\n+\t\tgit reset --hard HEAD~1 &&\n+\t\tgit checkout base\n+\t) &&\n+\tBAD_OID=$(git -C clone-from rev-parse tip~1) &&\n+\tTIP_OID=$(git -C clone-from rev-parse tip) &&\n+\tgit clone --depth=1 --no-single-branch \\\n+\t\t\"file://$(pwd)/clone-from\" clone-to &&\n+\t(\n+\t\tcd clone-to &&\n+\n+\t\t# Set up broken history by removing shallow markers\n+\t\tgit update-ref -d refs/remotes/origin/tip &&\n+\t\trm .git/shallow &&\n+\n+\t\t# Verify should fail\n+\t\ttest_must_fail git bundle verify \\\n+\t\t\t../clone-from/tip.bundle 2>err &&\n+\t\tgrep \"Could not read $BAD_OID\" err &&\n+\t\tgrep \"Failed to traverse parents of commit $TIP_OID\" err &&\n+\n+\t\t# Unbundling should fail\n+\t\ttest_must_fail git bundle unbundle \\\n+\t\t\t../clone-from/tip.bundle 2>err &&\n+\t\tgrep \"Could not read $BAD_OID\" err &&\n+\t\tgrep \"Failed to traverse parents of commit $TIP_OID\" err\n+\t)\n+'\n+\n test_done\n--\n2.39.1.vfs.0.0\n\n\n"},{"id":"470972","messageId":"ecc6b167-f5c4-48ce-3973-461d1659ed40@github.com","threadId":"59040","inReplyTo":"771a2993-85bd-0831-0977-24204f84e206@github.com","subject":"[PATCH v2.5 02/11] bundle: verify using connected()","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-24T14:16:30Z","receivedAt":"2023-01-24T14:16:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/24/2023 7:27 AM, Derrick Stolee wrote:\n\n> I'll focus on this area today and see what I can learn and how I\n> can approach this problem in a different way.\n\n> 2. Find out how to modify verify_bundle() so it can do the more\n> relaxed connectivity check.\n\nAnd here is the modification to verify_bundle() to depend on\ncheck_connected() instead. This also improves (in my opinion) the\nerror reporting from this situation, as seen in the test edits.\n\nAgain, this is a placeholder before I re-roll this series into\nan inevitable v3, so don't bother applying this patch until then.\n\nThanks,\n-Stolee\n\n--- >8 ---\n\nFrom 6a3d64761e9691994f9310b9ce2338f49aa72d48 Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Tue, 24 Jan 2023 08:47:00 -0500\nSubject: [PATCH v2.5 02/11] bundle: verify using connected()\n\nWhen Git verifies a bundle to see if it is safe for unbundling, it first\nlooks to see if the prerequisite commits are in the object store. This\nis usually a sufficient filter, and those missing commits are indicated\nclearly in the error messages. However, if the commits are present in\nthe object store, then there could still be issues if those commits are\nnot reachable from the repository's references. The repository only has\nguarantees that its object store is closed under reachability for the\nobjects that are reachable from references.\n\nThus, the code in verify_bundle() has previously had the additional\ncheck that all prerequisite commits are reachable from repository\nreferences. This is done via a revision walk from all references,\nstopping only if all prerequisite commits are discovered or all commits\nare walked. This uses a custom walk to verify_bundle().\n\nThis check is more strict than what Git applies even to fetched\npack-files. In the fetch case, Git guarantees that the new references\nare closed under reachability by walking from the new references until\nwalking commits that are reachable from repository refs. This is done\nthrough the well-used check_connected() method.\n\nTo better align with the restrictions required by 'git fetch',\nreimplement this check in verify_bundle() to use check_connected(). This\nalso simplifies the code significantly.\n\nThe previous change added a test that verified the behavior of 'git\nbundle verify' and 'git bundle unbundle' in this case, and the error\nmessages looked like this:\n\n error: Could not read \n fatal: Failed to traverse parents of commit \n\nHowever, by changing the revision walk slightly within check_connected()\nand using its quiet mode, we can omit those messages. Instead, we get\nonly this message, tailored to describing the current state of the\nrepository:\n\n error: some prerequisite commits exist in the object store,\n but are not connected to the repository's history\n\n(Line break added here for the commit message formatting, only.)\n\nWhile this message does not include any object IDs, there is no\nguarantee that those object IDs would help the user diagnose what is\ngoing on, as they could be separated from the prerequisite commits by\nsome distance. At minimum, this situation describes the situation in a\nmore informative way than the previous error messages.\n\nSigned-off-by: Derrick Stolee \n---\n bundle.c | 75 ++++++++++++++++--------------------------\n t/t6020-bundle-misc.sh | 8 ++---\n 2 files changed, 33 insertions(+), 50 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex 4ef7256aa11..76c3a904898 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -12,6 +12,7 @@\n #include \"refs.h\"\n #include \"strvec.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"connected.h\"\n\n static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n@@ -187,6 +188,21 @@ static int list_refs(struct string_list *r, int argc, const char **argv)\n /* Remember to update object flag allocation in object.h */\n #define PREREQ_MARK (1u<<16)\n\n+struct string_list_iterator {\n+\tstruct string_list *list;\n+\tsize_t cur;\n+};\n+\n+static const struct object_id *iterate_ref_map(void *cb_data)\n+{\n+\tstruct string_list_iterator *iter = cb_data;\n+\n+\tif (iter->cur >= iter->list->nr)\n+\t\treturn NULL;\n+\n+\treturn iter->list->items[iter->cur++].util;\n+}\n+\n int verify_bundle(struct repository *r,\n \t\t struct bundle_header *header,\n \t\t enum verify_bundle_flags flags)\n@@ -196,26 +212,25 @@ int verify_bundle(struct repository *r,\n \t * to be verbose about the errors\n \t */\n \tstruct string_list *p = &header->prerequisites;\n-\tstruct rev_info revs = REV_INFO_INIT;\n-\tconst char *argv[] = {NULL, \"--all\", NULL};\n-\tstruct commit *commit;\n-\tint i, ret = 0, req_nr;\n+\tint i, ret = 0;\n \tconst char *message = _(\"Repository lacks these prerequisite commits:\");\n+\tstruct string_list_iterator iter = {\n+\t\t.list = p,\n+\t};\n+\tstruct check_connected_options opts = {\n+\t\t.quiet = 1,\n+\t};\n\n \tif (!r || !r->objects || !r->objects->odb)\n \t\treturn error(_(\"need a repository to verify a bundle\"));\n\n-\trepo_init_revisions(r, &revs, NULL);\n \tfor (i = 0; i < p->nr; i++) {\n \t\tstruct string_list_item *e = p->items + i;\n \t\tconst char *name = e->string;\n \t\tstruct object_id *oid = e->util;\n \t\tstruct object *o = parse_object(r, oid);\n-\t\tif (o) {\n-\t\t\to->flags |= PREREQ_MARK;\n-\t\t\tadd_pending_object(&revs, o, name);\n+\t\tif (o)\n \t\t\tcontinue;\n-\t\t}\n \t\tret++;\n \t\tif (flags & VERIFY_BUNDLE_QUIET)\n \t\t\tcontinue;\n@@ -223,37 +238,14 @@ int verify_bundle(struct repository *r,\n \t\t\terror(\"%s\", message);\n \t\terror(\"%s %s\", oid_to_hex(oid), name);\n \t}\n-\tif (revs.pending.nr != p->nr)\n+\tif (ret)\n \t\tgoto cleanup;\n-\treq_nr = revs.pending.nr;\n-\tsetup_revisions(2, argv, &revs, NULL);\n-\n-\tlist_objects_filter_copy(&revs.filter, &header->filter);\n-\n-\tif (prepare_revision_walk(&revs))\n-\t\tdie(_(\"revision walk setup failed\"));\n\n-\ti = req_nr;\n-\twhile (i && (commit = get_revision(&revs)))\n-\t\tif (commit->object.flags & PREREQ_MARK)\n-\t\t\ti--;\n-\n-\tfor (i = 0; i < p->nr; i++) {\n-\t\tstruct string_list_item *e = p->items + i;\n-\t\tconst char *name = e->string;\n-\t\tconst struct object_id *oid = e->util;\n-\t\tstruct object *o = parse_object(r, oid);\n-\t\tassert(o); /* otherwise we'd have returned early */\n-\t\tif (o->flags & SHOWN)\n-\t\t\tcontinue;\n-\t\tret++;\n-\t\tif (flags & VERIFY_BUNDLE_QUIET)\n-\t\t\tcontinue;\n-\t\tif (ret == 1)\n-\t\t\terror(\"%s\", message);\n-\t\terror(\"%s %s\", oid_to_hex(oid), name);\n-\t}\n+\tif ((ret = check_connected(iterate_ref_map, &iter, &opts)))\n+\t\terror(_(\"some prerequisite commits exist in the object store, \"\n+\t\t\t\"but are not connected to the repository's history\"));\n\n+\t/* TODO: preserve this verbose language. */\n \tif (flags & VERIFY_BUNDLE_VERBOSE) {\n \t\tstruct string_list *r;\n\n@@ -282,15 +274,6 @@ int verify_bundle(struct repository *r,\n \t\t\t\t list_objects_filter_spec(&header->filter));\n \t}\n cleanup:\n-\t/* Clean up objects used, as they will be reused. */\n-\tfor (i = 0; i < p->nr; i++) {\n-\t\tstruct string_list_item *e = p->items + i;\n-\t\tstruct object_id *oid = e->util;\n-\t\tcommit = lookup_commit_reference_gently(r, oid, 1);\n-\t\tif (commit)\n-\t\t\tclear_commit_marks(commit, ALL_REV_FLAGS | PREREQ_MARK);\n-\t}\n-\trelease_revisions(&revs);\n \treturn ret;\n }\n\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 38dbbf89155..7d40994991e 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -595,14 +595,14 @@ test_expect_success 'verify catches unreachable, broken prerequisites' '\n \t\t# Verify should fail\n \t\ttest_must_fail git bundle verify \\\n \t\t\t../clone-from/tip.bundle 2>err &&\n-\t\tgrep \"Could not read $BAD_OID\" err &&\n-\t\tgrep \"Failed to traverse parents of commit $TIP_OID\" err &&\n+\t\tgrep \"some prerequisite commits .* are not connected\" err &&\n+\t\ttest_line_count = 1 err &&\n\n \t\t# Unbundling should fail\n \t\ttest_must_fail git bundle unbundle \\\n \t\t\t../clone-from/tip.bundle 2>err &&\n-\t\tgrep \"Could not read $BAD_OID\" err &&\n-\t\tgrep \"Failed to traverse parents of commit $TIP_OID\" err\n+\t\tgrep \"some prerequisite commits .* are not connected\" err &&\n+\t\ttest_line_count = 1 err\n \t)\n '\n\n--\n2.39.1.vfs.0.0\n\n\n"},{"id":"470980","messageId":"xmqq5ycwj6t9.fsf@gitster.g","threadId":"59040","inReplyTo":"771a2993-85bd-0831-0977-24204f84e206@github.com","subject":"Re: [PATCH v2 01/10] bundle: optionally skip reachability walk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-24T15:22:42Z","receivedAt":"2023-01-24T15:22:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> I'll focus on this area today and see what I can learn and how I\n> can approach this problem in a different way. The current options\n> that I see are:\n>\n> 1. Leave verify_bundle() as-is and figure out how to refresh the\n> refs. (This would remain a stricter check than necessary.)\n\nEven if we switch to \"assume everything is OK, remember a few key\nfacts (like prerequisites and tips) about each bundle as we unpack\nthem, and sanity check the results at the end\" approach, doesn't\nthat last step require us to be able to see the final state of the\nrefs? If so, wouldn't we need to figure out how to refresh the refs\nno matter what?\n\n> 2. Find out how to modify verify_bundle() so it can do the more\n> relaxed connectivity check.\n\nI am not sure what kind of relaxing you have in mind, but as long as\nwe can guarantee the connectedness of the end result\n\n> 3. Take the connectivity check that fetch uses before updating\n> refs and add that check before updating refs in the bundle URI\n> code.\n\nThis is optional at much lower priority, isn't it? In the second\nexample in the message you are responding to, I do not think it is\ntoo bad to reject the bundle based on '8' that has been rewound away\n(in other words, a bundle publisher ought to be basing their bundles\non well publicized and commonly available commits). Only when we\ntry to be overly helpful to such a use case, it becomes necessary to\nloosen the rule from \"all prerequisites must be reachable from\nexisting refs\" to \"or prerequisites that are not reachable from any\nrefs are also OK if they pass check_connected()\".\n\nThe current check to require that prerequisites are reachable from\nrefs does not have to check trees and blobs, because any commit that\nis reachable from an existing ref is complete[*] by definition.\n\n Let's define a term: a commit is \"complete\" iff it is not\n missing any objects that it (recursively) references to.\n\nThe check done by check_connected() is more expensive because it has\nto prove that a commit, which is found in the object store and may\nor may not be reachable from any refs, is complete. The tranversal\nstill can take advantage of the fact that commits _reachable_ from\nrefs are guaranteed to be complete, but until the traversal reaches\na commit that is reachable from refs (e.g. when inspecting commits\n'8' and then '7' until it reaches '6', in the second example in the\nmessage you are responding to) we need to look at trees and blobs.\n\n> There could also be a combination of (2) and (3), or others I have\n> not considered until I go poking around in the code.\n>\n> I'll let you know what I find.\n\nThanks. Unlike areas that allow glitches as long as workarounds are\navailable (e.g. UI), the object store + refs layer is where it is\nabsolutely required to be correct. I am happy to see capable minds\nare on it.\n"},{"id":"470990","messageId":"xmqqv8kvhmzz.fsf@gitster.g","threadId":"59040","inReplyTo":"01f97aff-58a1-ef2c-e668-d37ea513c64e@github.com","subject":"Re: [PATCH v2.5 01/11] bundle: test unbundling with incomplete history","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-24T17:16:00Z","receivedAt":"2023-01-24T17:16:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> In order to construct a broken history, perform a shallow clone of a\n> repository with a linear history, but whose default branch ('base') has\n> a single commit, so dropping the shallow markers leaves a complete\n> history from that reference. However, the 'tip' reference adds a\n> shallow commit whose parent is missing in the cloned repository. Trying\n> to unbundle a bundle with the 'tip' as a prerequisite will succeed past\n> the object store check and move into the reachability check.\n\nIt makes it sound convoluted set-up for tests, but I guess it is the\nmost direct way to get to the state you want to test, which is good.\n\nIn practice, the problem would appear when you create a multi-commit\nbranch, which then is discarded. GC then decides to expire the\nolder part of the commit chain while leaving the commits near the\ntip still in the object store. So the problem can happen without\nusers doing anything esoteric, and is very much worth testing.\n\n> +test_expect_success 'verify catches unreachable, broken prerequisites' '\n> +\ttest_when_finished rm -rf clone-from clone-to &&\n\nOK, so my understanding of what happens is ...\n\n> +\tgit init clone-from &&\n> +\t(\n> +\t\tcd clone-from &&\n> +\t\tgit checkout -b base &&\n> +\t\ttest_commit A &&\n> +\t\tgit checkout -b tip &&\n> +\t\tgit commit --allow-empty -m \"will drop by shallow\" &&\n> +\t\tgit commit --allow-empty -m \"will keep by shallow\" &&\n> +\t\tgit commit --allow-empty -m \"for bundle, not clone\" &&\n> +\t\tgit bundle create tip.bundle tip~1..tip &&\n\n... there is a single strand of pearls\n\n\tA---D---K---B tip\n\nwhere D is with \"will drop by shallow\" message. The bundle\nis prepared to give a history leading to B while requiring K.\n\n> +\t\tgit reset --hard HEAD~1 &&\n> +\t\tgit checkout base\n\nThen B is thrown away before the history is cloned.\n\n> +\t) &&\n> +\tBAD_OID=$(git -C clone-from rev-parse tip~1) &&\n> +\tTIP_OID=$(git -C clone-from rev-parse tip) &&\n> +\tgit clone --depth=1 --no-single-branch \\\n> +\t\t\"file://$(pwd)/clone-from\" clone-to &&\n> +\t(\n> +\t\tcd clone-to &&\n\nThe cloned repository should have\n\n\tA---d---K\n\nwhere D is missing behind the shallow boundary, origin/tip pointing\nat K.\n\n> +\t\t# Set up broken history by removing shallow markers\n> +\t\tgit update-ref -d refs/remotes/origin/tip &&\n\nBut we remove origin/tip, so K (and its trees and blobs) is totally\ndisconnected.\n\n> +\t\trm .git/shallow &&\n\nAnd then this removes the shallow info that makes us to pretend that\nK does not have D (missing) as its parent. Now we lack the required\nparent D if we start traversing from K.\n\n> +\t\t# Verify should fail\n> +\t\ttest_must_fail git bundle verify \\\n> +\t\t\t../clone-from/tip.bundle 2>err &&\n\nverify_bundle() wants to see traversal from \"--all\" to hit the\nprerequisite objects and K certainly cannot be reached by any ref.\n\nOK. So we ended up with a repository where we are on 'base' branch,\nand origin/HEAD and origin/base remote-tracking refs exist, all of\nthese refs pointing at A. Plus K exists but not D, but it is fine\nbecause K is not referenced by any ref.\n\nThis is perfectly constructed test case that checks a very\ninteresting scenario. It is as if the commit chain D---K was\ndiscarded (via \"git branch -D\") and then D got expired for being too\nold but K is not old enough.\n\nWe want to ensure \"git bundle verify\" and \"git fetch ./bundle.file\"\nin this healthy repository, where its refs do honor the promise, but\nits object store has unconnected commits (like \"K\") that are not\ncomplete, behaves sensibly. If we loosen \"prerequisites must be\nreachable from refs\" to \"prerequisites must exist\", it will lead to\nrepository corruption if we allow the bundle to be unbundled and its\ntips made into our refs, because these new refs point at incomplete\nobjects.\n\nExcellent.\n\n> +\t\t# Unbundling should fail\n> +\t\ttest_must_fail git bundle unbundle \\\n> +\t\t\t../clone-from/tip.bundle 2>err &&\n> +\t\tgrep \"Could not read $BAD_OID\" err &&\n> +\t\tgrep \"Failed to traverse parents of commit $TIP_OID\" err\n> +\t)\n> +'\n"},{"id":"470992","messageId":"xmqqfsbzhm7h.fsf@gitster.g","threadId":"59040","inReplyTo":"ecc6b167-f5c4-48ce-3973-461d1659ed40@github.com","subject":"Re: [PATCH v2.5 02/11] bundle: verify using connected()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-24T17:33:06Z","receivedAt":"2023-01-24T17:33:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> When Git verifies a bundle to see if it is safe for unbundling, it first\n> looks to see if the prerequisite commits are in the object store. This\n> is usually a sufficient filter, and those missing commits are indicated\n> clearly in the error messages.\n\nI am not sure if our early check is because \"does the prerequisite\neven exist?\" is sufficient. It is a short-cut that is cheap and can\nbe done without preparing the commit traversal.\n\n> However, if the commits are present in\n> the object store, then there could still be issues if those commits are\n> not reachable from the repository's references. The repository only has\n> guarantees that its object store is closed under reachability for the\n> objects that are reachable from references.\n\nCorrect.\n\n> Thus, the code in verify_bundle() has previously had the additional\n> check that all prerequisite commits are reachable from repository\n> references. This is done via a revision walk from all references,\n> stopping only if all prerequisite commits are discovered or all commits\n> are walked. This uses a custom walk to verify_bundle().\n\nCorrect.\n\n> This check is more strict than what Git applies even to fetched\n> pack-files.\n\nI do not see the need to say \"even\" here. In what other situation\ndo we make connectivity checks, and is there a need to be more\nstrict than others when checking fetched packfiles?\n\n> In the fetch case, Git guarantees that the new references\n> are closed under reachability by walking from the new references until\n> walking commits that are reachable from repository refs. This is done\n> through the well-used check_connected() method.\n\nCorrect and is a good point to make.\n\n> To better align with the restrictions required by 'git fetch',\n> reimplement this check in verify_bundle() to use check_connected(). This\n> also simplifies the code significantly.\n\nWonderful. I never liked the custom check done in unbundle code,\nwhich I am reasonably sure came from scripted hack to unbundle I\nwrote eons ago.\n\n"},{"id":"470995","messageId":"bad5d998-b34a-15dd-4f9d-9bf1cb74dda0@github.com","threadId":"59040","inReplyTo":"xmqqfsbzhm7h.fsf@gitster.g","subject":"Re: [PATCH v2.5 02/11] bundle: verify using connected()","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-24T18:46:02Z","receivedAt":"2023-01-24T18:46:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/24/2023 12:33 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> When Git verifies a bundle to see if it is safe for unbundling, it first\n>> looks to see if the prerequisite commits are in the object store. This\n>> is usually a sufficient filter, and those missing commits are indicated\n>> clearly in the error messages.\n> \n> I am not sure if our early check is because \"does the prerequisite\n> even exist?\" is sufficient. It is a short-cut that is cheap and can\n> be done without preparing the commit traversal.\n\nI suppose I should say \"Usually, existence in the object store is\ncorrelated with having all reachable objects, but this is not\nguaranteed.\" I'll also mention that it is a short-cut that can fail\nfaster than the reachability check.\n\n>> This check is more strict than what Git applies even to fetched\n>> pack-files.\n> \n> I do not see the need to say \"even\" here. In what other situation\n> do we make connectivity checks, and is there a need to be more\n> strict than others when checking fetched packfiles?\n\nI suppose that I was implying that fetches are the more common\noperation, and the scrutiny applied to an arbitrary pack-file from\na remote is probably higher there. However, who knows where a\nbundle came from, so the scrutiny should be the same.\n\n>> To better align with the restrictions required by 'git fetch',\n>> reimplement this check in verify_bundle() to use check_connected(). This\n>> also simplifies the code significantly.\n> \n> Wonderful. I never liked the custom check done in unbundle code,\n> which I am reasonably sure came from scripted hack to unbundle I\n> wrote eons ago.\n \nExcellent. Thanks for your feedback.\n\n-Stolee\n"},{"id":"471000","messageId":"xmqqr0vjfyxf.fsf@gitster.g","threadId":"59040","inReplyTo":"bad5d998-b34a-15dd-4f9d-9bf1cb74dda0@github.com","subject":"Re: [PATCH v2.5 02/11] bundle: verify using connected()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-24T20:41:16Z","receivedAt":"2023-01-24T20:41:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>> I do not see the need to say \"even\" here. In what other situation\n>> do we make connectivity checks, and is there a need to be more\n>> strict than others when checking fetched packfiles?\n>\n> I suppose that I was implying that fetches are the more common\n> operation, and the scrutiny applied to an arbitrary pack-file from\n> a remote is probably higher there. However, who knows where a\n> bundle came from, so the scrutiny should be the same.\n\nAh, I see that is where that \"even\" came from. And yes, I agree\nthat unbundling and fetch should be suspicious of their input to the\nsame degree.\n"},{"id":"471096","messageId":"39e4a13a-2230-919f-4cfc-95d70fb33e3a@github.com","threadId":"59040","inReplyTo":"427aff4d5e5c85b601f43af8b664515380e11453.1674487310.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 02/10] t5558: add tests for creationToken heuristic","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-27T19:15:09Z","receivedAt":"2023-01-27T19:15:18Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> As documented in the bundle URI design doc in 2da14fad8fe (docs:\n> document bundle URI standard, 2022-08-09), the 'creationToken' member of\n> a bundle URI allows a bundle provider to specify a total order on the\n> bundles.\n> \n> Future changes will allow the Git client to understand these members and\n> modify its behavior around downloading the bundles in that order. In the\n> meantime, create tests that add creation tokens to the bundle list. For\n> now, the Git client correctly ignores these unknown keys.\n> \n> Create a new test helper function, test_remote_https_urls, which filters\n> GIT_TRACE2_EVENT output to extract a list of URLs passed to\n> git-remote-https child processes. This can be used to verify the order\n> of these requests as we implement the creationToken heuristic. For now,\n> we need to sort the actual output since the current client does not have\n> a well-defined order that it applies to the bundles.\n\n...\n\n> +# Given a GIT_TRACE2_EVENT log over stdin, writes to stdout a list of URLs\n> +# sent to git-remote-https child processes.\n> +test_remote_https_urls() {\n> +\tgrep -e '\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\".*\"\\]' |\n> +\t\tsed -e 's/{\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"//g' \\\n> +\t\t -e 's/\"\\]}//g'\n> +}\n> +\n\n...\n\n> +\tcat >expect <<-EOF &&\n> +\t$HTTPD_URL/bundle-1.bundle\n> +\t$HTTPD_URL/bundle-2.bundle\n> +\t$HTTPD_URL/bundle-3.bundle\n> +\t$HTTPD_URL/bundle-4.bundle\n> +\t$HTTPD_URL/bundle-list\n> +\tEOF\n> +\n> +\t# Sort the list, since the order is not well-defined\n> +\t# without a heuristic.\n> +\ttest_remote_https_urls actual &&\n> +\ttest_cmp expect actual\n\n...\n\n> +\tcat >expect <<-EOF &&\n> +\t$HTTPD_URL/bundle-1.bundle\n> +\t$HTTPD_URL/bundle-2.bundle\n> +\t$HTTPD_URL/bundle-3.bundle\n> +\t$HTTPD_URL/bundle-4.bundle\n> +\t$HTTPD_URL/bundle-list\n> +\tEOF\n> +\n> +\t# Since the creationToken heuristic is not yet understood by the\n> +\t# client, the order cannot be verified at this moment. Sort the\n> +\t# list for consistent results.\n> +\ttest_remote_https_urls actual &&\n> +\ttest_cmp expect actual\n\nThese updates make the tests stronger (that is, less likely to let a\nregression slip through), and the additional comments are helpful for\nexplaining what is and is not implemented at this point in the series. \n\n"},{"id":"471097","messageId":"a2896d5b-f395-68df-1f23-356d0128cb9c@github.com","threadId":"59040","inReplyTo":"7cfaa3c518cbedb65c585cc02015bb21ae24e9fa.1674487310.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 05/10] bundle-uri: download in creationToken order","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-27T19:17:00Z","receivedAt":"2023-01-27T19:17:06Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> +struct bundles_for_sorting {\n\n...\n\n> +static int append_bundle(struct remote_bundle_info *bundle, void *data)\n\n...\n\n> +/**\n> + * For use in QSORT() to get a list sorted by creationToken\n> + * in decreasing order.\n> + */\n> +static int compare_creation_token_decreasing(const void *va, const void *vb)\n\nThese new function/struct names are all unambiguous. Looks good!\n\n> +\tcur = 0;\n> +\twhile (cur >= 0 && cur < bundles.nr) {\n> +\t\tstruct remote_bundle_info *bundle = bundles.items[cur];\n> +\t\tif (!bundle->file) {\n> +\t\t\t/*\n> +\t\t\t * Not downloaded yet. Try downloading.\n> +\t\t\t *\n> +\t\t\t * Note that bundle->file is non-NULL if a download\n> +\t\t\t * was attempted, even if it failed to download.\n> +\t\t\t */\n> +\t\t\tif (fetch_bundle_uri_internal(ctx.r, bundle, ctx.depth + 1, ctx.list)) {\n> +\t\t\t\t/* Mark as unbundled so we do not retry. */\n> +\t\t\t\tbundle->unbundled = 1;\n\nThis implicitly shows that, unlike a failed unbundling, a failed download is\nalways erroneous behavior, with the added benefit of avoiding (potentially\nexpensive) download re-attempts.\n\n> +\n> +\t\t\t\t/* Try looking deeper in the list. */\n> +\t\t\t\tmove_direction = 1;\n> +\t\t\t\tgoto stack_operation;\n> +\t\t\t}\n> +\n> +\t\t\t/* We expect bundles when using creationTokens. */\n> +\t\t\tif (!is_bundle(bundle->file, 1)) {\n> +\t\t\t\twarning(_(\"file downloaded from '%s' is not a bundle\"),\n> +\t\t\t\t\tbundle->uri);\n> +\t\t\t\tbreak;\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\tif (bundle->file && !bundle->unbundled) {\n> +\t\t\t/*\n> +\t\t\t * This was downloaded, but not successfully\n> +\t\t\t * unbundled. Try unbundling again.\n> +\t\t\t */\n> +\t\t\tif (unbundle_from_file(ctx.r, bundle->file)) {\n> +\t\t\t\t/* Try looking deeper in the list. */\n> +\t\t\t\tmove_direction = 1;\n> +\t\t\t} else {\n> +\t\t\t\t/*\n> +\t\t\t\t * Succeeded in unbundle. Retry bundles\n> +\t\t\t\t * that previously failed to unbundle.\n> +\t\t\t\t */\n> +\t\t\t\tmove_direction = -1;\n> +\t\t\t\tbundle->unbundled = 1;\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\t/*\n> +\t\t * Else case: downloaded and unbundled successfully.\n> +\t\t * Skip this by moving in the same direction as the\n> +\t\t * previous step.\n> +\t\t */\n> +\n> +stack_operation:\n\nOther than this label, it looks like you've replaced all of the\n\"stack-based\" language. Should this be replaced as well? No problem if not,\nI just wasn't sure whether it was left that way intentionally.\n\n> diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\n> index 474432c8ace..6f9417a0afb 100755\n> --- a/t/t5558-clone-bundle-uri.sh\n> +++ b/t/t5558-clone-bundle-uri.sh\n> @@ -401,17 +401,43 @@ test_expect_success 'clone bundle list (http, creationToken)' '\n> \tgit -C clone-list-http-2 cat-file --batch-check \n> \tcat >expect <<-EOF &&\n> -\t$HTTPD_URL/bundle-1.bundle\n> -\t$HTTPD_URL/bundle-2.bundle\n> -\t$HTTPD_URL/bundle-3.bundle\n> +\t$HTTPD_URL/bundle-list\n> \t$HTTPD_URL/bundle-4.bundle\n> +\t$HTTPD_URL/bundle-3.bundle\n> +\t$HTTPD_URL/bundle-2.bundle\n> +\t$HTTPD_URL/bundle-1.bundle\n> +\tEOF\n\nOoh, interesting - using the new \"test_remote_https_urls\", these tests now\nalso verify that the bundles were downloaded in decreasing order when using\nthe 'creationToken' heuristic. That's a nice extra confirmation that the\nheuristic is working as intended.\n\n> +test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n\n...\n\n> +test_expect_success 'auto-discover multiple bundles from HTTP clone: creationToken heuristic' '\n\nThese tests look good as well, especially 'clone incomplete bundle list's\nnow-more descriptive name.\n\n"},{"id":"471098","messageId":"f219288e-93c8-7791-0b1c-b36f658f8697@github.com","threadId":"59040","inReplyTo":"59e57e049683e42248c270b3bfcad2d72769219d.1674487310.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 08/10] fetch: fetch from an external bundle URI","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-27T19:18:31Z","receivedAt":"2023-01-27T19:18:36Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> @@ -2109,6 +2110,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,\n> int cmd_fetch(int argc, const char **argv, const char *prefix)\n> {\n> \tint i;\n> +\tconst char *bundle_uri;\n> \tstruct string_list list = STRING_LIST_INIT_DUP;\n> \tstruct remote *remote = NULL;\n> \tint result = 0;\n> @@ -2194,6 +2196,11 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tif (dry_run)\n> \t\twrite_fetch_head = 0;\n> \n> +\tif (!git_config_get_string_tmp(\"fetch.bundleuri\", &bundle_uri)) {\n> +\t\tif (fetch_bundle_uri(the_repository, bundle_uri, NULL))\n> +\t\t\twarning(_(\"failed to fetch bundles from '%s'\"), bundle_uri);\n\nnit: these conditions don't need to be nested and could instead be joined\nwith '&&' in the outer 'if ()' (unless you plan to add more to this block in\na future series - I didn't see anything in later patches here).\n\nEverything else (tests, doc updates since the last version) looks good.\n\n"},{"id":"471099","messageId":"b3f2992d-f614-5cc7-f606-d3607a154685@github.com","threadId":"59040","inReplyTo":"676522615ad0e8f24099ef35a0f39367e5f688ae.1674487310.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 10/10] bundle-uri: test missing bundles with heuristic","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-27T19:21:44Z","receivedAt":"2023-01-27T19:21:51Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The creationToken heuristic uses a different mechanism for downloading\n> bundles from the \"standard\" approach. Specifically: it uses a concrete\n> order based on the creationToken values and attempts to download as few\n> bundles as possible. It also modifies local config to store a value for\n> future fetches to avoid downloading bundles, if possible.\n> \n> However, if any of the individual bundles has a failed download, then\n> the logic for the ordering comes into question. It is important to avoid\n> infinite loops, assigning invalid creation token values in config, but\n> also to be opportunistic as possible when downloading as many bundles as\n> seem appropriate.\n> \n> These tests were used to inform the implementation of\n> fetch_bundles_by_token() in bundle-uri.c, but are being added\n> independently here to allow focusing on faulty downloads. There may be\n> more cases that could be added that result in modifications to\n> fetch_bundles_by_token() as interesting data shapes reveal themselves in\n> real scenarios.\n> \n\nThe expanded testing is great, thanks for adding it!\n\n> +\t# Case 2: middle bundle does not exist, only two bundles can unbundle\n> +\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n> +\t[bundle]\n> +\t\tversion = 1\n> +\t\tmode = all\n> +\t\theuristic = creationToken\n> +\n> +\t[bundle \"bundle-1\"]\n> +\t\turi = bundle-1.bundle\n> +\t\tcreationToken = 1\n> +\n> +\t[bundle \"bundle-2\"]\n> +\t\turi = fake.bundle\n> +\t\tcreationToken = 2\n> +\n> +\t[bundle \"bundle-3\"]\n> +\t\turi = bundle-3.bundle\n> +\t\tcreationToken = 3\n> +\n> +\t[bundle \"bundle-4\"]\n> +\t\turi = bundle-4.bundle\n> +\t\tcreationToken = 4\n> +\tEOF\n> +\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone-2.txt\" \\\n> +\tgit clone --single-branch --branch=base \\\n> +\t\t--bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n> +\t\t\"$HTTPD_URL/smart/fetch.git\" download-2 &&\n> +\n> +\t# Bundle failure does not set these configs.\n> +\ttest_must_fail git -C download-2 config fetch.bundleuri &&\n> +\ttest_must_fail git -C download-2 config fetch.bundlecreationtoken &&\n> +\n> +\tcat >expect <<-EOF &&\n> +\t$HTTPD_URL/bundle-list\n> +\t$HTTPD_URL/bundle-4.bundle\n> +\t$HTTPD_URL/bundle-3.bundle\n> +\t$HTTPD_URL/fake.bundle\n> +\t$HTTPD_URL/bundle-1.bundle\n> +\tEOF\n> +\ttest_remote_https_urls actual &&\n> +\ttest_cmp expect actual &&\n> +\n> +\t# Only base bundle unbundled.\n> +\tgit -C download-2 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n> +\tcat >expect <<-EOF &&\n> +\trefs/bundles/base\n> +\trefs/bundles/right\n> +\tEOF\n> +\ttest_cmp expect refs &&\n\nMaybe I'm misreading, but I don't think the comment (\"Only base bundle\nunbundled\") lines up with the expected bundle refs (both bundle-1\n('refs/bundles/base') and bundle-3 ('refs/bundles/right') seem to be\nunbundled). \n\n> +\n> +\t# Case 3: top bundle does not exist, rest unbundle fine.\n> +\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n> +\t[bundle]\n> +\t\tversion = 1\n> +\t\tmode = all\n> +\t\theuristic = creationToken\n> +\n> +\t[bundle \"bundle-1\"]\n> +\t\turi = bundle-1.bundle\n> +\t\tcreationToken = 1\n> +\n> +\t[bundle \"bundle-2\"]\n> +\t\turi = bundle-2.bundle\n> +\t\tcreationToken = 2\n> +\n> +\t[bundle \"bundle-3\"]\n> +\t\turi = bundle-3.bundle\n> +\t\tcreationToken = 3\n> +\n> +\t[bundle \"bundle-4\"]\n> +\t\turi = fake.bundle\n> +\t\tcreationToken = 4\n> +\tEOF\n> +\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone-3.txt\" \\\n> +\tgit clone --single-branch --branch=base \\\n> +\t\t--bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n> +\t\t\"$HTTPD_URL/smart/fetch.git\" download-3 &&\n> +\n> +\t# As long as we have continguous successful downloads,\n> +\t# we _do_ set these configs.\n> +\ttest_cmp_config -C download-3 \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n> +\ttest_cmp_config -C download-3 3 fetch.bundlecreationtoken &&\n> +\n> +\tcat >expect <<-EOF &&\n> +\t$HTTPD_URL/bundle-list\n> +\t$HTTPD_URL/fake.bundle\n> +\t$HTTPD_URL/bundle-3.bundle\n> +\t$HTTPD_URL/bundle-2.bundle\n> +\t$HTTPD_URL/bundle-1.bundle\n> +\tEOF\n> +\ttest_remote_https_urls actual &&\n> +\ttest_cmp expect actual &&\n> +\n> +\t# All bundles failed to unbundle\n> +\tgit -C download-3 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n> +\tcat >expect <<-EOF &&\n> +\trefs/bundles/base\n> +\trefs/bundles/left\n> +\trefs/bundles/right\n> +\tEOF\n> +\ttest_cmp expect refs\n\nSimilar issue with the comment here - it says that all bundles *failed* to\nunbundle, but the test case description (\"Case 3: top bundle does not exist,\nrest unbundle fine.\") and the result show bundle-1, bundle-2, and bundle-3\nall unbundling successfully.\n\n> +'\n> +\n> +# Expand the bundle list to include other interesting shapes, specifically\n> +# interesting for use when fetching from a previous state.\n> +#\n> +# ---------------- bundle-7\n> +# 7\n> +# _/|\\_\n> +# ---/--|--\\------ bundle-6\n> +# 5 | 6\n> +# --|---|---|----- bundle-4\n> +# | 4 |\n> +# | / \\ /\n> +# --|-|---|/------ bundle-3 (the client will be caught up to this point.)\n> +# \\ | 3\n> +# ---\\|---|------- bundle-2\n> +# 2 |\n> +# ----|---|------- bundle-1\n> +# \\ /\n> +# 1\n> +# |\n> +# (previous commits)\n\n...\n\n> +\t# Case 1: all bundles exist: successful unbundling of all bundles\n\n...\n\n> +\t# Case 2: middle bundle does not exist, only bundle-4 can unbundle\n\n...\n\n> +\t# Case 3: top bundle does not exist, rest unbundle fine.\n\nThe rest of these cases look okay and, at a high-level, it's helpful to have\nthese additional tests covering a different topology.\n\n"},{"id":"471100","messageId":"d8624caa-3c16-6b4d-03be-27d5531c6582@github.com","threadId":"59040","inReplyTo":"pull.1454.v2.git.1674487310.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/10] Bundle URIs V: creationToken heuristic for incremental fetches","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-27T19:28:27Z","receivedAt":"2023-01-27T19:28:36Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> Updates in v2\n> =============\n\nIn patches 2-9, I only had a few minor nits I commented on in the respective\npatches; everything else either addressed something directly noted in the\nlast set of reviews or otherwise was sufficiently explained by\ncomments/commit messages. \n\nAs for patch 1, I think the approach to relaxing the unbundling checks you\nsettled on in [1] after discussing with Junio makes sense.\n\n[1] https://lore.kernel.org/git/ecc6b167-f5c4-48ce-3973-461d1659ed40@github.com/\n\n> \n> Thanks,\n> \n> * Stolee\n> \n> Derrick Stolee (10):\n> bundle: optionally skip reachability walk\n> t5558: add tests for creationToken heuristic\n> bundle-uri: parse bundle.heuristic=creationToken\n> bundle-uri: parse bundle..creationToken values\n> bundle-uri: download in creationToken order\n> clone: set fetch.bundleURI if appropriate\n> bundle-uri: drop bundle.flag from design doc\n> fetch: fetch from an external bundle URI\n> bundle-uri: store fetch.bundleCreationToken\n> bundle-uri: test missing bundles with heuristic\n> \n> Documentation/config/bundle.txt | 7 +\n> Documentation/config/fetch.txt | 24 +\n> Documentation/technical/bundle-uri.txt | 8 +-\n> builtin/clone.c | 6 +-\n> builtin/fetch.c | 7 +\n> bundle-uri.c | 257 +++++++++-\n> bundle-uri.h | 28 +-\n> bundle.c | 3 +-\n> bundle.h | 1 +\n> t/t5558-clone-bundle-uri.sh | 672 ++++++++++++++++++++++++-\n> t/t5601-clone.sh | 46 ++\n> t/t5750-bundle-uri-parse.sh | 37 ++\n> t/test-lib-functions.sh | 8 +\n> 13 files changed, 1091 insertions(+), 13 deletions(-)\n> \n> \n> base-commit: 4dbebc36b0893f5094668ddea077d0e235560b16\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1454%2Fderrickstolee%2Fbundle-redo%2FcreationToken-v2\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1454/derrickstolee/bundle-redo/creationToken-v2\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1454\n> \n> Range-diff vs v1:\n> \n> -: ----------- > 1: b3828725bc8 bundle: optionally skip reachability walk\n> 1: 39eed914878 ! 2: 427aff4d5e5 t5558: add tests for creationToken heuristic\n> @@ Commit message\n> meantime, create tests that add creation tokens to the bundle list. For\n> now, the Git client correctly ignores these unknown keys.\n> \n> + Create a new test helper function, test_remote_https_urls, which filters\n> + GIT_TRACE2_EVENT output to extract a list of URLs passed to\n> + git-remote-https child processes. This can be used to verify the order\n> + of these requests as we implement the creationToken heuristic. For now,\n> + we need to sort the actual output since the current client does not have\n> + a well-defined order that it applies to the bundles.\n> +\n> Signed-off-by: Derrick Stolee \n> \n> ## t/t5558-clone-bundle-uri.sh ##\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone HTTP bundle' '\n> - \ttest_config -C clone-http log.excludedecoration refs/bundle/\n> '\n> \n> -+# usage: test_bundle_downloaded \n> -+test_bundle_downloaded () {\n> -+\tcat >pattern <<-EOF &&\n> -+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/$1\"\\]\n> -+\tEOF\n> -+\tgrep -f pattern \"$2\"\n> -+}\n> -+\n> test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n> +\ttest_when_finished rm -f trace*.txt &&\n> +\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (HTTP, no he\n> -\tgit -C clone-list-http cat-file --batch-check +\tgit -C clone-list-http cat-file --batch-check +\n> -+\tfor b in 1 2 3 4\n> -+\tdo\n> -+\t\ttest_bundle_downloaded bundle-$b.bundle trace-clone.txt ||\n> -+\t\t\treturn 1\n> -+\tdone\n> ++\tcat >expect <<-EOF &&\n> ++\t$HTTPD_URL/bundle-1.bundle\n> ++\t$HTTPD_URL/bundle-2.bundle\n> ++\t$HTTPD_URL/bundle-3.bundle\n> ++\t$HTTPD_URL/bundle-4.bundle\n> ++\t$HTTPD_URL/bundle-list\n> ++\tEOF\n> ++\n> ++\t# Sort the list, since the order is not well-defined\n> ++\t# without a heuristic.\n> ++\ttest_remote_https_urls actual &&\n> ++\ttest_cmp expect actual\n> '\n> \n> test_expect_success 'clone bundle list (HTTP, any mode)' '\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (HTTP, any m\n> '\n> \n> +test_expect_success 'clone bundle list (http, creationToken)' '\n> ++\ttest_when_finished rm -f trace*.txt &&\n> ++\n> +\tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n> +\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n> +\t[bundle]\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (HTTP, any m\n> +\t\tcreationToken = 4\n> +\tEOF\n> +\n> -+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" . clone-list-http-2 &&\n> ++\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" git \\\n> ++\t\tclone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n> ++\t\t\"$HTTPD_URL/smart/fetch.git\" clone-list-http-2 &&\n> +\n> +\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n> -+\tgit -C clone-list-http-2 cat-file --batch-check ++\tgit -C clone-list-http-2 cat-file --batch-check ++\n> ++\tcat >expect <<-EOF &&\n> ++\t$HTTPD_URL/bundle-1.bundle\n> ++\t$HTTPD_URL/bundle-2.bundle\n> ++\t$HTTPD_URL/bundle-3.bundle\n> ++\t$HTTPD_URL/bundle-4.bundle\n> ++\t$HTTPD_URL/bundle-list\n> ++\tEOF\n> ++\n> ++\t# Since the creationToken heuristic is not yet understood by the\n> ++\t# client, the order cannot be verified at this moment. Sort the\n> ++\t# list for consistent results.\n> ++\ttest_remote_https_urls actual &&\n> ++\ttest_cmp expect actual\n> +'\n> +\n> # Do not add tests here unless they use the HTTP server, as they will\n> # not run unless the HTTP dependencies exist.\n> \n> +\n> + ## t/test-lib-functions.sh ##\n> +@@ t/test-lib-functions.sh: test_region () {\n> + \treturn 0\n> + }\n> + \n> ++# Given a GIT_TRACE2_EVENT log over stdin, writes to stdout a list of URLs\n> ++# sent to git-remote-https child processes.\n> ++test_remote_https_urls() {\n> ++\tgrep -e '\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\".*\"\\]' |\n> ++\t\tsed -e 's/{\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"//g' \\\n> ++\t\t -e 's/\"\\]}//g'\n> ++}\n> ++\n> + # Print the destination of symlink(s) provided as arguments. Basically\n> + # the same as the readlink command, but it's not available everywhere.\n> + test_readlink () {\n> 2: 9007249b948 ! 3: f6f8197c9cc bundle-uri: parse bundle.heuristic=creationToken\n> @@ Commit message\n> bundle-uri' to print the heuristic value and verify that the parsing\n> works correctly.\n> \n> + As an extra precaution, create the internal 'heuristics' array to be a\n> + list of (enum, string) pairs so we can iterate through the array entries\n> + carefully, regardless of the enum values.\n> +\n> Signed-off-by: Derrick Stolee \n> \n> ## Documentation/config/bundle.txt ##\n> @@ bundle-uri.c\n> #include \"config.h\"\n> #include \"remote.h\"\n> \n> -+static const char *heuristics[] = {\n> -+\t[BUNDLE_HEURISTIC_NONE] = \"\",\n> -+\t[BUNDLE_HEURISTIC_CREATIONTOKEN] = \"creationToken\",\n> ++static struct {\n> ++\tenum bundle_list_heuristic heuristic;\n> ++\tconst char *name;\n> ++} heuristics[BUNDLE_HEURISTIC__COUNT] = {\n> ++\t{ BUNDLE_HEURISTIC_NONE, \"\"},\n> ++\t{ BUNDLE_HEURISTIC_CREATIONTOKEN, \"creationToken\" },\n> +};\n> +\n> static int compare_bundles(const void *hashmap_cmp_fn_data,\n> @@ bundle-uri.c: void print_bundle_list(FILE *fp, struct bundle_list *list)\n> \tfprintf(fp, \"\\tversion = %d\\n\", list->version);\n> \tfprintf(fp, \"\\tmode = %s\\n\", mode);\n> \n> -+\tif (list->heuristic)\n> -+\t\tprintf(\"\\theuristic = %s\\n\", heuristics[list->heuristic]);\n> ++\tif (list->heuristic) {\n> ++\t\tint i;\n> ++\t\tfor (i = 0; i < BUNDLE_HEURISTIC__COUNT; i++) {\n> ++\t\t\tif (heuristics[i].heuristic == list->heuristic) {\n> ++\t\t\t\tprintf(\"\\theuristic = %s\\n\",\n> ++\t\t\t\t heuristics[list->heuristic].name);\n> ++\t\t\t\tbreak;\n> ++\t\t\t}\n> ++\t\t}\n> ++\t}\n> +\n> \tfor_all_bundles_in_list(list, summarize_bundle, fp);\n> }\n> @@ bundle-uri.c: static int bundle_list_update(const char *key, const char *value,\n> +\t\tif (!strcmp(subkey, \"heuristic\")) {\n> +\t\t\tint i;\n> +\t\t\tfor (i = 0; i < BUNDLE_HEURISTIC__COUNT; i++) {\n> -+\t\t\t\tif (!strcmp(value, heuristics[i])) {\n> -+\t\t\t\t\tlist->heuristic = i;\n> ++\t\t\t\tif (heuristics[i].heuristic &&\n> ++\t\t\t\t heuristics[i].name &&\n> ++\t\t\t\t !strcmp(value, heuristics[i].name)) {\n> ++\t\t\t\t\tlist->heuristic = heuristics[i].heuristic;\n> +\t\t\t\t\treturn 0;\n> +\t\t\t\t}\n> +\t\t\t}\n> @@ bundle-uri.h: enum bundle_list_mode {\n> +\tBUNDLE_HEURISTIC_CREATIONTOKEN,\n> +\n> +\t/* Must be last. */\n> -+\tBUNDLE_HEURISTIC__COUNT,\n> ++\tBUNDLE_HEURISTIC__COUNT\n> +};\n> +\n> /**\n> 3: a1808f0b10c = 4: 12efa228d04 bundle-uri: parse bundle..creationToken values\n> 4: 57c0174d375 ! 5: 7cfaa3c518c bundle-uri: download in creationToken order\n> @@ Commit message\n> strategy implemented here provides that short-circuit where the client\n> downloads a minimal set of bundles.\n> \n> + However, we are not satisfied by the naive approach of downloading\n> + bundles until one successfully unbundles, expecting the earlier bundles\n> + to successfully unbundle now. The example repository in t5558\n> + demonstrates this well:\n> +\n> + ---------------- bundle-4\n> +\n> + 4\n> + / \\\n> + ----|---|------- bundle-3\n> + | |\n> + | 3\n> + | |\n> + ----|---|------- bundle-2\n> + | |\n> + 2 |\n> + | |\n> + ----|---|------- bundle-1\n> + \\ /\n> + 1\n> + |\n> + (previous commits)\n> +\n> + In this repository, if we already have the objects for bundle-1 and then\n> + try to fetch from this list, the naive approach will fail. bundle-4\n> + requires both bundle-3 and bundle-2, though bundle-3 will successfully\n> + unbundle without bundle-2. Thus, the algorithm needs to keep this in\n> + mind.\n> +\n> A later implementation detail will store the maximum creationToken seen\n> during such a bundle download, and the client will avoid downloading a\n> bundle unless its creationToken is strictly greater than that stored\n> @@ bundle-uri.c: static int download_bundle_to_file(struct remote_bundle_info *bund\n> \treturn 0;\n> }\n> \n> -+struct sorted_bundle_list {\n> ++struct bundles_for_sorting {\n> +\tstruct remote_bundle_info **items;\n> +\tsize_t alloc;\n> +\tsize_t nr;\n> +};\n> +\n> -+static int insert_bundle(struct remote_bundle_info *bundle, void *data)\n> ++static int append_bundle(struct remote_bundle_info *bundle, void *data)\n> +{\n> -+\tstruct sorted_bundle_list *list = data;\n> ++\tstruct bundles_for_sorting *list = data;\n> +\tlist->items[list->nr++] = bundle;\n> +\treturn 0;\n> +}\n> +\n> -+static int compare_creation_token(const void *va, const void *vb)\n> ++/**\n> ++ * For use in QSORT() to get a list sorted by creationToken\n> ++ * in decreasing order.\n> ++ */\n> ++static int compare_creation_token_decreasing(const void *va, const void *vb)\n> +{\n> +\tconst struct remote_bundle_info * const *a = va;\n> +\tconst struct remote_bundle_info * const *b = vb;\n> @@ bundle-uri.c: static int download_bundle_to_file(struct remote_bundle_info *bund\n> +\t\t\t\t struct bundle_list *list)\n> +{\n> +\tint cur;\n> -+\tint pop_or_push = 0;\n> ++\tint move_direction = 0;\n> +\tstruct bundle_list_context ctx = {\n> +\t\t.r = r,\n> +\t\t.list = list,\n> +\t\t.mode = list->mode,\n> +\t};\n> -+\tstruct sorted_bundle_list sorted = {\n> ++\tstruct bundles_for_sorting bundles = {\n> +\t\t.alloc = hashmap_get_size(&list->bundles),\n> +\t};\n> +\n> -+\tALLOC_ARRAY(sorted.items, sorted.alloc);\n> ++\tALLOC_ARRAY(bundles.items, bundles.alloc);\n> +\n> -+\tfor_all_bundles_in_list(list, insert_bundle, &sorted);\n> ++\tfor_all_bundles_in_list(list, append_bundle, &bundles);\n> +\n> -+\tQSORT(sorted.items, sorted.nr, compare_creation_token);\n> ++\tQSORT(bundles.items, bundles.nr, compare_creation_token_decreasing);\n> +\n> +\t/*\n> -+\t * Use a stack-based approach to download the bundles and attempt\n> -+\t * to unbundle them in decreasing order by creation token. If we\n> -+\t * fail to unbundle (after a successful download) then move to the\n> -+\t * next non-downloaded bundle (push to the stack) and attempt\n> -+\t * downloading. Once we succeed in applying a bundle, move to the\n> -+\t * previous unapplied bundle (pop the stack) and attempt to unbundle\n> -+\t * it again.\n> ++\t * Attempt to download and unbundle the minimum number of bundles by\n> ++\t * creationToken in decreasing order. If we fail to unbundle (after\n> ++\t * a successful download) then move to the next non-downloaded bundle\n> ++\t * and attempt downloading. Once we succeed in applying a bundle,\n> ++\t * move to the previous unapplied bundle and attempt to unbundle it\n> ++\t * again.\n> +\t *\n> +\t * In the case of a fresh clone, we will likely download all of the\n> +\t * bundles before successfully unbundling the oldest one, then the\n> @@ bundle-uri.c: static int download_bundle_to_file(struct remote_bundle_info *bund\n> +\t * repo's object store.\n> +\t */\n> +\tcur = 0;\n> -+\twhile (cur >= 0 && cur < sorted.nr) {\n> -+\t\tstruct remote_bundle_info *bundle = sorted.items[cur];\n> ++\twhile (cur >= 0 && cur < bundles.nr) {\n> ++\t\tstruct remote_bundle_info *bundle = bundles.items[cur];\n> +\t\tif (!bundle->file) {\n> -+\t\t\t/* Not downloaded yet. Try downloading. */\n> -+\t\t\tif (download_bundle_to_file(bundle, &ctx)) {\n> -+\t\t\t\t/* Failure. Push to the stack. */\n> -+\t\t\t\tpop_or_push = 1;\n> ++\t\t\t/*\n> ++\t\t\t * Not downloaded yet. Try downloading.\n> ++\t\t\t *\n> ++\t\t\t * Note that bundle->file is non-NULL if a download\n> ++\t\t\t * was attempted, even if it failed to download.\n> ++\t\t\t */\n> ++\t\t\tif (fetch_bundle_uri_internal(ctx.r, bundle, ctx.depth + 1, ctx.list)) {\n> ++\t\t\t\t/* Mark as unbundled so we do not retry. */\n> ++\t\t\t\tbundle->unbundled = 1;\n> ++\n> ++\t\t\t\t/* Try looking deeper in the list. */\n> ++\t\t\t\tmove_direction = 1;\n> +\t\t\t\tgoto stack_operation;\n> +\t\t\t}\n> +\n> @@ bundle-uri.c: static int download_bundle_to_file(struct remote_bundle_info *bund\n> +\t\t\t * unbundled. Try unbundling again.\n> +\t\t\t */\n> +\t\t\tif (unbundle_from_file(ctx.r, bundle->file)) {\n> -+\t\t\t\t/* Failed to unbundle. Push to stack. */\n> -+\t\t\t\tpop_or_push = 1;\n> ++\t\t\t\t/* Try looking deeper in the list. */\n> ++\t\t\t\tmove_direction = 1;\n> +\t\t\t} else {\n> -+\t\t\t\t/* Succeeded in unbundle. Pop stack. */\n> -+\t\t\t\tpop_or_push = -1;\n> ++\t\t\t\t/*\n> ++\t\t\t\t * Succeeded in unbundle. Retry bundles\n> ++\t\t\t\t * that previously failed to unbundle.\n> ++\t\t\t\t */\n> ++\t\t\t\tmove_direction = -1;\n> ++\t\t\t\tbundle->unbundled = 1;\n> +\t\t\t}\n> +\t\t}\n> +\n> @@ bundle-uri.c: static int download_bundle_to_file(struct remote_bundle_info *bund\n> +\n> +stack_operation:\n> +\t\t/* Move in the specified direction and repeat. */\n> -+\t\tcur += pop_or_push;\n> ++\t\tcur += move_direction;\n> +\t}\n> +\n> -+\tfree(sorted.items);\n> ++\tfree(bundles.items);\n> +\n> +\t/*\n> +\t * We succeed if the loop terminates because 'cur' drops below\n> @@ bundle-uri.c: static int fetch_bundle_list_in_config_format(struct repository *r\n> +\t * it advertises are expected to be bundles, not nested lists.\n> +\t * We can drop 'global_list' and 'depth'.\n> +\t */\n> -+\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN)\n> ++\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN) {\n> +\t\tresult = fetch_bundles_by_token(r, &list_from_bundle);\n> -+\telse if ((result = download_bundle_list(r, &list_from_bundle,\n> ++\t\tglobal_list->heuristic = BUNDLE_HEURISTIC_CREATIONTOKEN;\n> ++\t} else if ((result = download_bundle_list(r, &list_from_bundle,\n> \t\t\t\t\t global_list, depth)))\n> \t\tgoto cleanup;\n> \n> @@ bundle-uri.c: int fetch_bundle_list(struct repository *r, struct bundle_list *li\n> \tfor_all_bundles_in_list(&global_list, unlink_bundle, NULL);\n> \n> ## t/t5558-clone-bundle-uri.sh ##\n> -@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (HTTP, any mode)' '\n> - '\n> - \n> - test_expect_success 'clone bundle list (http, creationToken)' '\n> -+\ttest_when_finished rm -f trace*.txt &&\n> -+\n> - \tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n> - \tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n> - \t[bundle]\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (http, creationToken)' '\n> - \t\tcreationToken = 4\n> - \tEOF\n> + \tgit -C clone-list-http-2 cat-file --batch-check \n> --\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" . clone-list-http-2 &&\n> -+\tGIT_TRACE2_EVENT=$(pwd)/trace-clone.txt \\\n> -+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n> -+\t\tclone-from clone-list-http-2 &&\n> - \n> - \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n> --\tgit -C clone-list-http-2 cat-file --batch-check -+\tgit -C clone-list-http-2 cat-file --batch-check -+\n> -+\tfor b in 1 2 3 4\n> -+\tdo\n> -+\t\ttest_bundle_downloaded bundle-$b.bundle trace-clone.txt ||\n> -+\t\t\treturn 1\n> -+\tdone\n> + \tcat >expect <<-EOF &&\n> +-\t$HTTPD_URL/bundle-1.bundle\n> +-\t$HTTPD_URL/bundle-2.bundle\n> +-\t$HTTPD_URL/bundle-3.bundle\n> ++\t$HTTPD_URL/bundle-list\n> + \t$HTTPD_URL/bundle-4.bundle\n> ++\t$HTTPD_URL/bundle-3.bundle\n> ++\t$HTTPD_URL/bundle-2.bundle\n> ++\t$HTTPD_URL/bundle-1.bundle\n> ++\tEOF\n> ++\n> ++\ttest_remote_https_urls actual &&\n> ++\ttest_cmp expect actual\n> +'\n> +\n> -+test_expect_success 'clone bundle list (http, creationToken)' '\n> ++test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n> +\ttest_when_finished rm -f trace*.txt &&\n> +\n> +\tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (http, creat\n> +\t[bundle \"bundle-1\"]\n> +\t\turi = bundle-1.bundle\n> +\t\tcreationToken = 1\n> -+\n> -+\t[bundle \"bundle-2\"]\n> -+\t\turi = bundle-2.bundle\n> -+\t\tcreationToken = 2\n> +\tEOF\n> +\n> +\tGIT_TRACE2_EVENT=$(pwd)/trace-clone.txt \\\n> +\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n> -+\t\tclone-from clone-token-http &&\n> ++\t\t--single-branch --branch=base --no-tags \\\n> ++\t\t\"$HTTPD_URL/smart/fetch.git\" clone-token-http &&\n> +\n> -+\ttest_bundle_downloaded bundle-1.bundle trace-clone.txt &&\n> -+\ttest_bundle_downloaded bundle-2.bundle trace-clone.txt\n> ++\tcat >expect <<-EOF &&\n> + \t$HTTPD_URL/bundle-list\n> ++\t$HTTPD_URL/bundle-1.bundle\n> + \tEOF\n> + \n> +-\t# Since the creationToken heuristic is not yet understood by the\n> +-\t# client, the order cannot be verified at this moment. Sort the\n> +-\t# list for consistent results.\n> +-\ttest_remote_https_urls actual &&\n> ++\ttest_remote_https_urls actual &&\n> + \ttest_cmp expect actual\n> '\n> \n> - # Do not add tests here unless they use the HTTP server, as they will\n> \n> ## t/t5601-clone.sh ##\n> @@ t/t5601-clone.sh: test_expect_success 'auto-discover multiple bundles from HTTP clone' '\n> \tgrep -f pattern trace.txt\n> '\n> \n> -+# Usage: test_bundle_downloaded \n> -+test_bundle_downloaded () {\n> -+\tcat >pattern <<-EOF &&\n> -+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/$1.bundle\"\\]\n> -+\tEOF\n> -+\tgrep -f pattern \"$2\"\n> -+}\n> -+\n> +test_expect_success 'auto-discover multiple bundles from HTTP clone: creationToken heuristic' '\n> +\ttest_when_finished rm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/repo4.git\" &&\n> +\ttest_when_finished rm -rf clone-heuristic trace*.txt &&\n> @@ t/t5601-clone.sh: test_expect_success 'auto-discover multiple bundles from HTTP\n> +\t\t -c transfer.bundleURI=true clone \\\n> +\t\t\"$HTTPD_URL/smart/repo4.git\" clone-heuristic &&\n> +\n> -+\t# We should fetch all bundles\n> -+\tfor b in everything new newest\n> -+\tdo\n> -+\t\ttest_bundle_downloaded $b trace-clone.txt || return 1\n> -+\tdone\n> ++\tcat >expect <<-EOF &&\n> ++\t$HTTPD_URL/newest.bundle\n> ++\t$HTTPD_URL/new.bundle\n> ++\t$HTTPD_URL/everything.bundle\n> ++\tEOF\n> ++\n> ++\t# We should fetch all bundles in the expected order.\n> ++\ttest_remote_https_urls actual &&\n> ++\ttest_cmp expect actual\n> +'\n> +\n> # DO NOT add non-httpd-specific tests here, because the last part of this\n> 5: d9c6f50e4f2 ! 6: 17c404c1b83 clone: set fetch.bundleURI if appropriate\n> @@ Documentation/config/fetch.txt: fetch.writeCommitGraph::\n> \t`git push -f`, and `git log --graph`. Defaults to false.\n> +\n> +fetch.bundleURI::\n> -+\tThis value stores a URI for fetching Git object data from a bundle URI\n> -+\tbefore performing an incremental fetch from the origin Git server. If\n> -+\tthe value is `` then running `git fetch ` is equivalent to\n> -+\tfirst running `git fetch --bundle-uri=` immediately before\n> -+\t`git fetch `. See details of the `--bundle-uri` option in\n> -+\tlinkgit:git-fetch[1].\n> ++\tThis value stores a URI for downloading Git object data from a bundle\n> ++\tURI before performing an incremental fetch from the origin Git server.\n> ++\tThis is similar to how the `--bundle-uri` option behaves in\n> ++\tlinkgit:git-clone[1]. `git clone --bundle-uri` will set the\n> ++\t`fetch.bundleURI` value if the supplied bundle URI contains a bundle\n> ++\tlist that is organized for incremental fetches.\n> \n> ## builtin/clone.c ##\n> @@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n> @@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n> \tstrvec_push(&transport_ls_refs_options.ref_prefixes, \"HEAD\");\n> \n> ## bundle-uri.c ##\n> -@@ bundle-uri.c: static int fetch_bundle_list_in_config_format(struct repository *r,\n> - \t * it advertises are expected to be bundles, not nested lists.\n> - \t * We can drop 'global_list' and 'depth'.\n> - \t */\n> --\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN)\n> -+\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN) {\n> - \t\tresult = fetch_bundles_by_token(r, &list_from_bundle);\n> --\telse if ((result = download_bundle_list(r, &list_from_bundle,\n> -+\t\tglobal_list->heuristic = BUNDLE_HEURISTIC_CREATIONTOKEN;\n> -+\t} else if ((result = download_bundle_list(r, &list_from_bundle,\n> - \t\t\t\t\t global_list, depth)))\n> - \t\tgoto cleanup;\n> - \n> @@ bundle-uri.c: static int unlink_bundle(struct remote_bundle_info *info, void *data)\n> \treturn 0;\n> }\n> @@ bundle-uri.h: int bundle_uri_parse_config_format(const char *uri,\n> * Given a bundle list that was already advertised (likely by the\n> \n> ## t/t5558-clone-bundle-uri.sh ##\n> -@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (http, creationToken)' '\n> - \ttest_bundle_downloaded bundle-2.bundle trace-clone.txt\n> +@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n> + \t\t--single-branch --branch=base --no-tags \\\n> + \t\t\"$HTTPD_URL/smart/fetch.git\" clone-token-http &&\n> + \n> ++\ttest_cmp_config -C clone-token-http \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n> ++\n> + \tcat >expect <<-EOF &&\n> + \t$HTTPD_URL/bundle-list\n> + \t$HTTPD_URL/bundle-1.bundle\n> +@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n> + \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (http, creat\n> +\n> +\ttest_cmp_config -C fetch-http-4 \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n> +\n> -+\t# The clone should copy two files: the list and bundle-1.\n> -+\ttest_bundle_downloaded bundle-list trace-clone.txt &&\n> -+\ttest_bundle_downloaded bundle-1.bundle trace-clone.txt &&\n> ++\tcat >expect <<-EOF &&\n> ++\t$HTTPD_URL/bundle-list\n> ++\t$HTTPD_URL/bundle-1.bundle\n> ++\tEOF\n> ++\n> ++\ttest_remote_https_urls actual &&\n> ++\ttest_cmp expect actual &&\n> +\n> +\t# only received base ref from bundle-1\n> +\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n> 6: afcfd27a883 = 7: d491070efed bundle-uri: drop bundle.flag from design doc\n> 7: 1627fc158b1 ! 8: 59e57e04968 fetch: fetch from an external bundle URI\n> @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tif (dry_run)\n> \t\twrite_fetch_head = 0;\n> \n> -+\tif (!git_config_get_string_tmp(\"fetch.bundleuri\", &bundle_uri) &&\n> -+\t !starts_with(bundle_uri, \"remote:\")) {\n> ++\tif (!git_config_get_string_tmp(\"fetch.bundleuri\", &bundle_uri)) {\n> +\t\tif (fetch_bundle_uri(the_repository, bundle_uri, NULL))\n> +\t\t\twarning(_(\"failed to fetch bundles from '%s'\"), bundle_uri);\n> +\t}\n> @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\n> \n> ## t/t5558-clone-bundle-uri.sh ##\n> +@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n> + \tEOF\n> + \n> + \ttest_remote_https_urls actual &&\n> +-\ttest_cmp expect actual\n> ++\ttest_cmp expect actual &&\n> ++\n> ++\t# We now have only one bundle ref.\n> ++\tgit -C clone-token-http for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n> ++\tcat >expect <<-\\EOF &&\n> ++\trefs/bundles/base\n> ++\tEOF\n> ++\ttest_cmp expect refs &&\n> ++\n> ++\t# Add remaining bundles, exercising the \"deepening\" strategy\n> ++\t# for downloading via the creationToken heurisitc.\n> ++\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n> ++\t[bundle \"bundle-2\"]\n> ++\t\turi = bundle-2.bundle\n> ++\t\tcreationToken = 2\n> ++\n> ++\t[bundle \"bundle-3\"]\n> ++\t\turi = bundle-3.bundle\n> ++\t\tcreationToken = 3\n> ++\n> ++\t[bundle \"bundle-4\"]\n> ++\t\turi = bundle-4.bundle\n> ++\t\tcreationToken = 4\n> ++\tEOF\n> ++\n> ++\tGIT_TRACE2_EVENT=\"$(pwd)/trace1.txt\" \\\n> ++\t\tgit -C clone-token-http fetch origin --no-tags \\\n> ++\t\trefs/heads/merge:refs/heads/merge &&\n> ++\n> ++\tcat >expect <<-EOF &&\n> ++\t$HTTPD_URL/bundle-list\n> ++\t$HTTPD_URL/bundle-4.bundle\n> ++\t$HTTPD_URL/bundle-3.bundle\n> ++\t$HTTPD_URL/bundle-2.bundle\n> ++\tEOF\n> ++\n> ++\ttest_remote_https_urls actual &&\n> ++\ttest_cmp expect actual &&\n> ++\n> ++\t# We now have all bundle refs.\n> ++\tgit -C clone-token-http for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n> ++\n> ++\tcat >expect <<-\\EOF &&\n> ++\trefs/bundles/base\n> ++\trefs/bundles/left\n> ++\trefs/bundles/merge\n> ++\trefs/bundles/right\n> ++\tEOF\n> ++\ttest_cmp expect refs\n> + '\n> + \n> + test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n> \tcat >expect <<-\\EOF &&\n> \trefs/bundles/base\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heurist\n> +\t\trefs/heads/left:refs/heads/left \\\n> +\t\trefs/heads/right:refs/heads/right &&\n> +\n> -+\t# This fetch should copy two files: the list and bundle-2.\n> -+\ttest_bundle_downloaded bundle-list trace1.txt &&\n> -+\ttest_bundle_downloaded bundle-2.bundle trace1.txt &&\n> -+\t! test_bundle_downloaded bundle-1.bundle trace1.txt &&\n> ++\tcat >expect <<-EOF &&\n> ++\t$HTTPD_URL/bundle-list\n> ++\t$HTTPD_URL/bundle-2.bundle\n> ++\tEOF\n> ++\n> ++\ttest_remote_https_urls actual &&\n> ++\ttest_cmp expect actual &&\n> +\n> +\t# received left from bundle-2\n> +\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heurist\n> +\t\tcreationToken = 4\n> +\tEOF\n> +\n> -+\t# This fetch should skip bundle-3.bundle, since its objets are\n> ++\t# This fetch should skip bundle-3.bundle, since its objects are\n> +\t# already local (we have the requisite commits for bundle-4.bundle).\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n> +\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n> +\t\trefs/heads/merge:refs/heads/merge &&\n> +\n> -+\t# This fetch should copy three files: the list, bundle-3, and bundle-4.\n> -+\ttest_bundle_downloaded bundle-list trace2.txt &&\n> -+\ttest_bundle_downloaded bundle-4.bundle trace2.txt &&\n> -+\t! test_bundle_downloaded bundle-1.bundle trace2.txt &&\n> -+\t! test_bundle_downloaded bundle-2.bundle trace2.txt &&\n> -+\t! test_bundle_downloaded bundle-3.bundle trace2.txt &&\n> ++\tcat >expect <<-EOF &&\n> ++\t$HTTPD_URL/bundle-list\n> ++\t$HTTPD_URL/bundle-4.bundle\n> ++\tEOF\n> ++\n> ++\ttest_remote_https_urls actual &&\n> ++\ttest_cmp expect actual &&\n> +\n> +\t# received merge ref from bundle-4, but right is missing\n> +\t# because we did not download bundle-3.\n> 8: 51f210ddeb4 ! 9: 6a1504b1c3a bundle-uri: store fetch.bundleCreationToken\n> @@ Commit message\n> When checking the same bundle list twice, this strategy requires\n> downloading the bundle with the maximum creationToken again, which is\n> wasteful. The creationToken heuristic promises that the client will not\n> - have a use for that bundle if its creationToken value is the at most the\n> + have a use for that bundle if its creationToken value is at most the\n> previous creationToken value.\n> \n> To prevent these wasteful downloads, create a fetch.bundleCreationToken\n> @@ Commit message\n> \n> ## Documentation/config/fetch.txt ##\n> @@ Documentation/config/fetch.txt: fetch.bundleURI::\n> - \tfirst running `git fetch --bundle-uri=` immediately before\n> - \t`git fetch `. See details of the `--bundle-uri` option in\n> - \tlinkgit:git-fetch[1].\n> + \tlinkgit:git-clone[1]. `git clone --bundle-uri` will set the\n> + \t`fetch.bundleURI` value if the supplied bundle URI contains a bundle\n> + \tlist that is organized for incremental fetches.\n> +++\n> ++If you modify this value and your repository has a `fetch.bundleCreationToken`\n> ++value, then remove that `fetch.bundleCreationToken` value before fetching from\n> ++the new bundle URI.\n> +\n> +fetch.bundleCreationToken::\n> +\tWhen using `fetch.bundleURI` to fetch incrementally from a bundle\n> @@ Documentation/config/fetch.txt: fetch.bundleURI::\n> +\tThis value is used to prevent downloading bundles in the future\n> +\tif the advertised `creationToken` is not strictly larger than this\n> +\tvalue.\n> +++\n> ++The creation token values are chosen by the provider serving the specific\n> ++bundle URI. If you modify the URI at `fetch.bundleURI`, then be sure to\n> ++remove the value for the `fetch.bundleCreationToken` value before fetching.\n> \n> ## bundle-uri.c ##\n> @@ bundle-uri.c: static int fetch_bundles_by_token(struct repository *r,\n> {\n> \tint cur;\n> - \tint pop_or_push = 0;\n> + \tint move_direction = 0;\n> +\tconst char *creationTokenStr;\n> -+\tuint64_t maxCreationToken;\n> ++\tuint64_t maxCreationToken = 0, newMaxCreationToken = 0;\n> \tstruct bundle_list_context ctx = {\n> \t\t.r = r,\n> \t\t.list = list,\n> @@ bundle-uri.c: static int fetch_bundles_by_token(struct repository *r,\n> \n> - \tfor_all_bundles_in_list(list, insert_bundle, &sorted);\n> + \tfor_all_bundles_in_list(list, append_bundle, &bundles);\n> \n> -+\tif (!sorted.nr) {\n> -+\t\tfree(sorted.items);\n> ++\tif (!bundles.nr) {\n> ++\t\tfree(bundles.items);\n> +\t\treturn 0;\n> +\t}\n> +\n> - \tQSORT(sorted.items, sorted.nr, compare_creation_token);\n> + \tQSORT(bundles.items, bundles.nr, compare_creation_token_decreasing);\n> \n> +\t/*\n> +\t * If fetch.bundleCreationToken exists, parses to a uint64t, and\n> @@ bundle-uri.c: static int fetch_bundles_by_token(struct repository *r,\n> +\t\t\t\t \"fetch.bundlecreationtoken\",\n> +\t\t\t\t &creationTokenStr) &&\n> +\t sscanf(creationTokenStr, \"%\"PRIu64, &maxCreationToken) == 1 &&\n> -+\t sorted.items[0]->creationToken <= maxCreationToken) {\n> -+\t\tfree(sorted.items);\n> ++\t bundles.items[0]->creationToken <= maxCreationToken) {\n> ++\t\tfree(bundles.items);\n> +\t\treturn 0;\n> +\t}\n> +\n> \t/*\n> - \t * Use a stack-based approach to download the bundles and attempt\n> - \t * to unbundle them in decreasing order by creation token. If we\n> + \t * Attempt to download and unbundle the minimum number of bundles by\n> + \t * creationToken in decreasing order. If we fail to unbundle (after\n> +@@ bundle-uri.c: static int fetch_bundles_by_token(struct repository *r,\n> + \tcur = 0;\n> + \twhile (cur >= 0 && cur < bundles.nr) {\n> + \t\tstruct remote_bundle_info *bundle = bundles.items[cur];\n> ++\n> ++\t\t/*\n> ++\t\t * If we need to dig into bundles below the previous\n> ++\t\t * creation token value, then likely we are in an erroneous\n> ++\t\t * state due to missing or invalid bundles. Halt the process\n> ++\t\t * instead of continuing to download extra data.\n> ++\t\t */\n> ++\t\tif (bundle->creationToken <= maxCreationToken)\n> ++\t\t\tbreak;\n> ++\n> + \t\tif (!bundle->file) {\n> + \t\t\t/*\n> + \t\t\t * Not downloaded yet. Try downloading.\n> +@@ bundle-uri.c: static int fetch_bundles_by_token(struct repository *r,\n> + \t\t\t\t */\n> + \t\t\t\tmove_direction = -1;\n> + \t\t\t\tbundle->unbundled = 1;\n> ++\n> ++\t\t\t\tif (bundle->creationToken > newMaxCreationToken)\n> ++\t\t\t\t\tnewMaxCreationToken = bundle->creationToken;\n> + \t\t\t}\n> + \t\t}\n> + \n> @@ bundle-uri.c: stack_operation:\n> - \t\tcur += pop_or_push;\n> + \t\tcur += move_direction;\n> \t}\n> \n> --\tfree(sorted.items);\n> +-\tfree(bundles.items);\n> -\n> \t/*\n> \t * We succeed if the loop terminates because 'cur' drops below\n> @@ bundle-uri.c: stack_operation:\n> \t */\n> +\tif (cur < 0) {\n> +\t\tstruct strbuf value = STRBUF_INIT;\n> -+\t\tstrbuf_addf(&value, \"%\"PRIu64\"\", sorted.items[0]->creationToken);\n> ++\t\tstrbuf_addf(&value, \"%\"PRIu64\"\", newMaxCreationToken);\n> +\t\tif (repo_config_set_multivar_gently(ctx.r,\n> +\t\t\t\t\t\t \"fetch.bundleCreationToken\",\n> +\t\t\t\t\t\t value.buf, NULL, 0))\n> @@ bundle-uri.c: stack_operation:\n> +\t\tstrbuf_release(&value);\n> +\t}\n> +\n> -+\tfree(sorted.items);\n> ++\tfree(bundles.items);\n> \treturn cur >= 0;\n> }\n> \n> \n> ## t/t5558-clone-bundle-uri.sh ##\n> +@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n> + \t\t\"$HTTPD_URL/smart/fetch.git\" clone-token-http &&\n> + \n> + \ttest_cmp_config -C clone-token-http \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n> ++\ttest_cmp_config -C clone-token-http 1 fetch.bundlecreationtoken &&\n> + \n> + \tcat >expect <<-EOF &&\n> + \t$HTTPD_URL/bundle-list\n> +@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n> + \tGIT_TRACE2_EVENT=\"$(pwd)/trace1.txt\" \\\n> + \t\tgit -C clone-token-http fetch origin --no-tags \\\n> + \t\trefs/heads/merge:refs/heads/merge &&\n> ++\ttest_cmp_config -C clone-token-http 4 fetch.bundlecreationtoken &&\n> + \n> + \tcat >expect <<-EOF &&\n> + \t$HTTPD_URL/bundle-list\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n> \t\t\"$HTTPD_URL/smart/fetch.git\" fetch-http-4 &&\n> \n> \ttest_cmp_config -C fetch-http-4 \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n> +\ttest_cmp_config -C fetch-http-4 1 fetch.bundlecreationtoken &&\n> \n> - \t# The clone should copy two files: the list and bundle-1.\n> - \ttest_bundle_downloaded bundle-list trace-clone.txt &&\n> + \tcat >expect <<-EOF &&\n> + \t$HTTPD_URL/bundle-list\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n> + \t\tgit -C fetch-http-4 fetch origin --no-tags \\\n> \t\trefs/heads/left:refs/heads/left \\\n> \t\trefs/heads/right:refs/heads/right &&\n> - \n> +\ttest_cmp_config -C fetch-http-4 2 fetch.bundlecreationtoken &&\n> -+\n> - \t# This fetch should copy two files: the list and bundle-2.\n> - \ttest_bundle_downloaded bundle-list trace1.txt &&\n> - \ttest_bundle_downloaded bundle-2.bundle trace1.txt &&\n> + \n> + \tcat >expect <<-EOF &&\n> + \t$HTTPD_URL/bundle-list\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n> \tEOF\n> \ttest_cmp expect refs &&\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heurist\n> +\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n> +\t\trefs/heads/left:refs/heads/left \\\n> +\t\trefs/heads/right:refs/heads/right &&\n> -+\ttest_bundle_downloaded bundle-list trace1b.txt &&\n> -+\t! test_bundle_downloaded bundle-1.bundle trace1b.txt &&\n> -+\t! test_bundle_downloaded bundle-2.bundle trace1b.txt &&\n> ++\n> ++\tcat >expect <<-EOF &&\n> ++\t$HTTPD_URL/bundle-list\n> ++\tEOF\n> ++\ttest_remote_https_urls actual &&\n> ++\ttest_cmp expect actual &&\n> +\n> \tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n> \t[bundle \"bundle-3\"]\n> \t\turi = bundle-3.bundle\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n> + \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n> \t\tgit -C fetch-http-4 fetch origin --no-tags \\\n> \t\trefs/heads/merge:refs/heads/merge &&\n> - \n> +\ttest_cmp_config -C fetch-http-4 4 fetch.bundlecreationtoken &&\n> -+\n> - \t# This fetch should copy three files: the list, bundle-3, and bundle-4.\n> - \ttest_bundle_downloaded bundle-list trace2.txt &&\n> - \ttest_bundle_downloaded bundle-4.bundle trace2.txt &&\n> + \n> + \tcat >expect <<-EOF &&\n> + \t$HTTPD_URL/bundle-list\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n> \trefs/bundles/left\n> \trefs/bundles/merge\n> @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heurist\n> +\t# No-op fetch\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace2b.txt\" \\\n> +\t\tgit -C fetch-http-4 fetch origin &&\n> -+\ttest_bundle_downloaded bundle-list trace2b.txt &&\n> -+\t! test_bundle_downloaded bundle-1.bundle trace2b.txt &&\n> -+\t! test_bundle_downloaded bundle-2.bundle trace2b.txt &&\n> -+\t! test_bundle_downloaded bundle-3.bundle trace2b.txt &&\n> -+\t! test_bundle_downloaded bundle-4.bundle trace2b.txt\n> ++\n> ++\tcat >expect <<-EOF &&\n> ++\t$HTTPD_URL/bundle-list\n> ++\tEOF\n> ++\ttest_remote_https_urls actual &&\n> ++\ttest_cmp expect actual\n> '\n> \n> # Do not add tests here unless they use the HTTP server, as they will\n> -: ----------- > 10: 676522615ad bundle-uri: test missing bundles with heuristic\n> \n\n"},{"id":"471101","messageId":"xmqqpmaz93k3.fsf@gitster.g","threadId":"59040","inReplyTo":"a2896d5b-f395-68df-1f23-356d0128cb9c@github.com","subject":"Re: [PATCH v2 05/10] bundle-uri: download in creationToken order","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-27T19:32:12Z","receivedAt":"2023-01-27T19:32:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n>> +\t\t\t/*\n>> +\t\t\t * Not downloaded yet. Try downloading.\n>> +\t\t\t *\n>> +\t\t\t * Note that bundle->file is non-NULL if a download\n>> +\t\t\t * was attempted, even if it failed to download.\n>> +\t\t\t */\n>> +\t\t\tif (fetch_bundle_uri_internal(ctx.r, bundle, ctx.depth + 1, ctx.list)) {\n>> +\t\t\t\t/* Mark as unbundled so we do not retry. */\n>> +\t\t\t\tbundle->unbundled = 1;\n>\n> This implicitly shows that, unlike a failed unbundling, a failed download is\n> always erroneous behavior, with the added benefit of avoiding (potentially\n> expensive) download re-attempts.\n\nHmph, I somehow was hoping that we'd allow an option to use range\nrequests to resume an interrupted download in the future, so\noutright \"always avoid attempts to download again\" may not be what\nwe want in the longer run. But being able to tell if download\nfailed (and there will probably be more than \"success/failure\" bit,\nbut something like \"we got an explicit 401 not found\" vs \"we were\ndisconnected after downloading a few megabytes\"), and unbundling\nfailed (where there is no point attempting) is a good idea.\n\n>> \tcat >expect <<-EOF &&\n>> -\t$HTTPD_URL/bundle-1.bundle\n>> -\t$HTTPD_URL/bundle-2.bundle\n>> -\t$HTTPD_URL/bundle-3.bundle\n>> +\t$HTTPD_URL/bundle-list\n>> \t$HTTPD_URL/bundle-4.bundle\n>> +\t$HTTPD_URL/bundle-3.bundle\n>> +\t$HTTPD_URL/bundle-2.bundle\n>> +\t$HTTPD_URL/bundle-1.bundle\n>> +\tEOF\n>\n> Ooh, interesting - using the new \"test_remote_https_urls\", these tests now\n> also verify that the bundles were downloaded in decreasing order when using\n> the 'creationToken' heuristic. That's a nice extra confirmation that the\n> heuristic is working as intended.\n\nYes, that indeed is very nice.\n"},{"id":"471143","messageId":"07c4658e-89dd-0f82-77e9-e7c443f747cd@github.com","threadId":"59040","inReplyTo":"xmqqpmaz93k3.fsf@gitster.g","subject":"Re: [PATCH v2 05/10] bundle-uri: download in creationToken order","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-30T18:43:07Z","receivedAt":"2023-01-30T18:43:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/27/2023 2:32 PM, Junio C Hamano wrote:\n> Victoria Dye writes:\n> \n>>> +\t\t\t/*\n>>> +\t\t\t * Not downloaded yet. Try downloading.\n>>> +\t\t\t *\n>>> +\t\t\t * Note that bundle->file is non-NULL if a download\n>>> +\t\t\t * was attempted, even if it failed to download.\n>>> +\t\t\t */\n>>> +\t\t\tif (fetch_bundle_uri_internal(ctx.r, bundle, ctx.depth + 1, ctx.list)) {\n>>> +\t\t\t\t/* Mark as unbundled so we do not retry. */\n>>> +\t\t\t\tbundle->unbundled = 1;\n>>\n>> This implicitly shows that, unlike a failed unbundling, a failed download is\n>> always erroneous behavior, with the added benefit of avoiding (potentially\n>> expensive) download re-attempts.\n> \n> Hmph, I somehow was hoping that we'd allow an option to use range\n> requests to resume an interrupted download in the future, so\n> outright \"always avoid attempts to download again\" may not be what\n> we want in the longer run. But being able to tell if download\n> failed (and there will probably be more than \"success/failure\" bit,\n> but something like \"we got an explicit 401 not found\" vs \"we were\n> disconnected after downloading a few megabytes\"), and unbundling\n> failed (where there is no point attempting) is a good idea.\n\nI think there are two possible directions we can have when talking\nabout interrupted downloads:\n\n1. The network connection was disconnected, and the client may want\n to respond to that with a retry and a ranged request.\n\n2. The client process itself terminates for some reason, and a\n second process recognizes that some of the data already exists\n and could be used for a range request of the remainder.\n\nI think both of these would not be handled at this layer, but\ninstead further down, inside fetch_bundle_uri_internal()\n(specifically further down in download_https_uri_to_file()).\n\nAny retry logic should happen there, closer to the connection,\nand at the layer of the current patch, we should assume that any\nretry logic that was attempted ended up failing in the end.\n\nDoes that satisfy your concerns here?\n\nThanks,\n-Stolee\n"},{"id":"471144","messageId":"d93aff7f-2fcb-1d4d-4537-c23ea7171215@github.com","threadId":"59040","inReplyTo":"b3f2992d-f614-5cc7-f606-d3607a154685@github.com","subject":"Re: [PATCH v2 10/10] bundle-uri: test missing bundles with heuristic","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-30T18:47:29Z","receivedAt":"2023-01-30T18:47:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/27/2023 2:21 PM, Victoria Dye wrote:\n> Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n\n>> +\t# Only base bundle unbundled.\n>> +\tgit -C download-2 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n>> +\tcat >expect <<-EOF &&\n>> +\trefs/bundles/base\n>> +\trefs/bundles/right\n>> +\tEOF\n>> +\ttest_cmp expect refs &&\n> \n> Maybe I'm misreading, but I don't think the comment (\"Only base bundle\n> unbundled\") lines up with the expected bundle refs (both bundle-1\n> ('refs/bundles/base') and bundle-3 ('refs/bundles/right') seem to be\n> unbundled). \n\n>> +\t# All bundles failed to unbundle\n>> +\tgit -C download-3 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n>> +\tcat >expect <<-EOF &&\n>> +\trefs/bundles/base\n>> +\trefs/bundles/left\n>> +\trefs/bundles/right\n>> +\tEOF\n>> +\ttest_cmp expect refs\n> \n> Similar issue with the comment here - it says that all bundles *failed* to\n> unbundle, but the test case description (\"Case 3: top bundle does not exist,\n> rest unbundle fine.\") and the result show bundle-1, bundle-2, and bundle-3\n> all unbundling successfully.\n\nThank you for reading carefully. I'm sorry about not updating\nthe comments after copying these checks around the test file.\n\nThanks,\n-Stolee\n"},{"id":"471148","messageId":"xmqqtu073kx1.fsf@gitster.g","threadId":"59040","inReplyTo":"07c4658e-89dd-0f82-77e9-e7c443f747cd@github.com","subject":"Re: [PATCH v2 05/10] bundle-uri: download in creationToken order","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-30T19:02:50Z","receivedAt":"2023-01-30T19:02:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> I think there are two possible directions we can have when talking\n> about interrupted downloads:\n>\n> 1. The network connection was disconnected, and the client may want\n> to respond to that with a retry and a ranged request.\n>\n> 2. The client process itself terminates for some reason, and a\n> second process recognizes that some of the data already exists\n> and could be used for a range request of the remainder.\n>\n> I think both of these would not be handled at this layer, but\n> instead further down, inside fetch_bundle_uri_internal()\n> (specifically further down in download_https_uri_to_file()).\n>\n> Any retry logic should happen there, closer to the connection,\n> and at the layer of the current patch, we should assume that any\n> retry logic that was attempted ended up failing in the end.\n>\n> Does that satisfy your concerns here?\n\nMostly. We probably do not want / need to cater to \"I killed it\nwith ^C and would want to continue\".\n\nThanks.\n"},{"id":"471150","messageId":"b6b76b2c-3a81-a612-d2ca-93b2b660feeb@github.com","threadId":"59040","inReplyTo":"xmqqtu073kx1.fsf@gitster.g","subject":"Re: [PATCH v2 05/10] bundle-uri: download in creationToken order","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-30T19:12:41Z","receivedAt":"2023-01-30T19:13:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/30/2023 2:02 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> I think there are two possible directions we can have when talking\n>> about interrupted downloads:\n>>\n>> 1. The network connection was disconnected, and the client may want\n>> to respond to that with a retry and a ranged request.\n>>\n>> 2. The client process itself terminates for some reason, and a\n>> second process recognizes that some of the data already exists\n>> and could be used for a range request of the remainder.\n... \n> Mostly. We probably do not want / need to cater to \"I killed it\n> with ^C and would want to continue\".\n\nI mention it because it has been mentioned to me as an example use\ncase. This includes other failures, such as power outages.\n\nI don't think we have range requests enabled for the longer-lived\npackfile-uri, but we shall see if adoption of bundle URIs presents\nmore motivation to build that feature. Case (1) will be the easiest\nto consider, and I agree that it would be more generally useful.\n\nThanks,\n-Stolee\n"},{"id":"471172","messageId":"pull.1454.v3.git.1675171759.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v2.git.1674487310.gitgitgadget@gmail.com","subject":"[PATCH v3 00/11] Bundle URIs V: creationToken heuristic for incremental fetches","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-31T13:29:08Z","receivedAt":"2023-01-31T13:29:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This fifth part to the bundle URIs feature follows part IV (advertising via\nprotocol v2) which recently merged to 'master', so this series is based on\n'master'.\n\nThis part introduces the concept of a heuristic that a bundle list can\nadvertise. The purpose of the heuristic is to hint to the Git client that\nthe bundles can be downloaded and unbundled in a certain order. In\nparticular, that order can assist with using the same bundle URI to download\nnew bundles from an updated bundle list. This allows bundle URIs to assist\nwith incremental fetches, not just initial clones.\n\nThe only planned heuristic is the \"creationToken\" heuristic where the bundle\nlist adds a 64-bit unsigned integer \"creationToken\" value to each bundle in\nthe list. Those values provide an ordering on the bundles implying that the\nbundles can be unbundled in increasing creationToken order and at each point\nthe required commits for the ith bundle were provided by bundles with lower\ncreationTokens.\n\nAt clone time, the only difference implied by the creationToken order is\nthat the Git client does not need to guess at the order to apply the\nbundles, but instead can use the creationToken order to apply them without\nfailure and retry. However, this presents an interesting benefit during\nfetches: the Git client can check the bundle list and download bundles in\ndecreasing creationToken order until the required commits for these bundles\nare present within the repository's object store. This prevents downloading\nmore bundle information than required.\n\nThe creationToken value is also a promise that the Git client will not need\nto download a bundle if its creationToken is less than or equal to the\ncreationToken of a previously-downloaded bundle. This further improves the\nperformance during a fetch in that the client does not need to download any\nbundles at all if it recognizes that the maximum creationToken is the same\n(or smaller than) a previously-downloaded creationToken.\n\nThe creationToken concept is documented in the existing design document at\nDocumentation/technical/bundle-uri.txt, including suggested ways for bundle\nproviders to organize their bundle lists to take advantage of the heuristic.\n\nThis series formalizes the creationToken heuristic and the Git client logic\nfor understanding it. Further, for bundle lists provided by the git clone\n--bundle-uri option, the Git client will recognize the heuristic as being\nhelpful for incremental fetches and store config values so that future git\nfetch commands check the bundle list before communicating with any Git\nremotes.\n\nNote that this option does not integrate fetches with bundle lists\nadvertised via protocol v2. I spent some time working on this, but found the\nimplementation to be distinct enough that it merited its own attention in a\nseparate series. In particular, the configuration for indicating that a\nfetch should check the bundle-uri protocol v2 command seemed best to be\nlocated within a Git remote instead of a repository-global key such as is\nbeing used for a static URI. Further, the timing of querying the bundle-uri\ncommand during a git fetch command is significantly different and more\ncomplicated than how it is used in git clone.\n\n\nWhat Remains?\n=============\n\nOriginally, I had planned on making this bundle URI work a 5-part series,\nand this is part 5. Shouldn't we be done now?\n\nThere are two main things that should be done after this series, in any\norder:\n\n * Teach git fetch to check a bundle list advertised by a remote over the\n bundle-uri protocol v2 command.\n * Add the bundle..filter option to allow advertising bundles and\n partial bundles side-by-side.\n\nThere is also room for expanding tests for more error conditions, or for\nother tweaks that are not currently part of the design document. I do think\nthat after this series, the feature will be easier to work on different\nparts in parallel.\n\n\nPatch Outline\n=============\n\n * (New in v3) Patch 1 tests the behavior of 'git bundle verify' and 'git\n bundle unbundle' when in the strange situation where a prerequisite\n commit exists in the object store but is not closed under reachability\n (necessarily not reachable from refs, too). This helps motivate the new\n Patch 2.\n * (New in v3) Patch 2 updates the behavior in verify_bundle() to use the\n check_connected()\n * Patch 3 creates a test setup demonstrating a creationToken heuristic. At\n this point, the Git client ignores the heuristic and uses its ad-hoc\n strategy for ordering the bundles.\n * Patches 4 and 5 teach Git to parse the bundle.heuristic and\n bundle..creationToken keys in a bundle list.\n * Patch 6 teaches Git to download bundles using the creationToken order.\n This order uses a stack approach to start from the maximum creationToken\n and continue downloading the next bundle in the list until all bundles\n can successfully be unbundled. This is the algorithm required for\n incremental fetches, while initial clones could download in the opposite\n order. Since clones will download all bundles anyway, having a second\n code path just for clones seemed unnecessary.\n * Patch 7 teaches git clone --bundle-uri to set fetch.bundleURI when the\n advertised bundle list includs a heuristic that Git understands.\n * Patch 8 updates the design document to remove reference to a bundle.flag\n option that was previously going to indicate the list was designed for\n fetches, but the bundle.heuristic option already does that.\n * Patch 9 teaches git fetch to check fetch.bundleURI and download bundles\n from that static URI before connecting to remotes via the Git protocol.\n * Patch 10 introduces a new fetch.bundleCreationToken config value to store\n the maximum creationToken of downloaded bundles. This prevents\n downloading the latest bundle on every git fetch command, reducing waste.\n * Patch 11 adds new tests for interesting incremental fetch shapes. Along\n with other test edits in other patches, these revealed several issues\n that required improvement within this series. These tests also check\n extra cases around failed bundle downloads.\n\n\nUpdates in v3\n=============\n\n * Patches 1 and 2 are replacements for v3's patch 1. Instead of skipping\n the reachability walk, make it slightly more flexible by using\n check_connected(). The first patch adds tests that cover this behavior,\n which was previously untested.\n * Patch 6 replaces the \"stack_operation\" label with a \"move\" label.\n * Patch 9 simplifies nested ifs to use &&.\n * Patch 11 updates some incorrect test comments.\n\n\nUpdates in v2\n=============\n\n * Patches 1 and 10 are new.\n * I started making the extra tests in patch 10 due to Victoria's concern\n around failed downloads. I extended the bundle list in a way that exposed\n other issues that are fixed in this version. Unfortunately, the test\n requires the full functionality of the entire series, so the tests are\n not isolated to where the code fixes are made. One thing that I noticed\n in the process is that some of the tests were using the local-clone trick\n to copy full object directories instead of copying only the requested\n object set. This was causing confusion in how the bundles were applying\n or failing to apply, so the tests are updated to use http whenever\n possible.\n * In Patch 2, I created a new test_remote_https_urls helper to get the full\n download list (in order). In this patch, the bundle download order is not\n well-defined, but is modified in later tests when it becomes\n well-defined.\n * In Patch 3, I updated the connection between config value and enum value\n to be an array of pairs instead of faking a hashmap-like interface that\n could be dangerous if the enum values were assigned incorrectly.\n * In Patch 5, the 'sorted' list and its type was renamed to be more\n descriptive. This also included updates to \"append_bundle()\" and\n \"compare_creation_token_decreasing()\" to be more descriptive. This had\n some side effects in Patch 8 due to the renames.\n * In Patch 5, I added the interesting bundle shape to the commit message to\n remind us of why the creationToken algorithm needs to be the way it is. I\n also removed the \"stack\" language in favor of discussing ranges of the\n sorted list. Some renames, such as \"pop_or_push\" is changed to\n \"move_direction\", resulted from this change of language.\n * The assignment of heuristic from the local list to global_list was moved\n into Patch 5.\n * In Patch 5, one of the tests removed bundle-2 because it allows a later\n test for git fetch to demonstrate the interesting behavior where bundle-4\n requires both bundle-2 and bundle-3.\n * In Patch 6, the fetch.bundleURI config is described differently,\n including dropping the defunct git fetch --bundle-uri reference and\n discussing that git clone --bundle-uri will set it automatically.\n * Patch 8 no longer refers to a config value starting with \"remote:\". It\n also expands a test that was previously not expanded in v1.\n * Patch 9 updates the documentation for fetch.bundleURI and\n fetch.bundleCreationToken to describe how the user should unset the\n latter if they edit the former.\n * Much of Patch 9's changes are due to context changes from the renames in\n Patch 5. However, it also adds the restriction that it will not attempt\n to download bundles unless their creationToken is strictly greater than\n the stored token. This ends up being critical to the failed download\n case, preventing an incremental fetch from downloading all bundles just\n because one bundle failed to download (and that case is tested in patch\n 10).\n * Patch 10 adds significant testing, including several tests of failed\n bundle downloads in various cases.\n\nThanks,\n\n * Stolee\n\nDerrick Stolee (11):\n bundle: test unbundling with incomplete history\n bundle: verify using check_connected()\n t5558: add tests for creationToken heuristic\n bundle-uri: parse bundle.heuristic=creationToken\n bundle-uri: parse bundle..creationToken values\n bundle-uri: download in creationToken order\n clone: set fetch.bundleURI if appropriate\n bundle-uri: drop bundle.flag from design doc\n fetch: fetch from an external bundle URI\n bundle-uri: store fetch.bundleCreationToken\n bundle-uri: test missing bundles with heuristic\n\n Documentation/config/bundle.txt | 7 +\n Documentation/config/fetch.txt | 24 +\n Documentation/technical/bundle-uri.txt | 8 +-\n builtin/clone.c | 6 +-\n builtin/fetch.c | 6 +\n bundle-uri.c | 249 ++++++++-\n bundle-uri.h | 28 +-\n bundle.c | 75 ++-\n t/t5558-clone-bundle-uri.sh | 672 ++++++++++++++++++++++++-\n t/t5601-clone.sh | 46 ++\n t/t5750-bundle-uri-parse.sh | 37 ++\n t/t6020-bundle-misc.sh | 40 ++\n t/test-lib-functions.sh | 8 +\n 13 files changed, 1149 insertions(+), 57 deletions(-)\n\n\nbase-commit: 4dbebc36b0893f5094668ddea077d0e235560b16\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1454%2Fderrickstolee%2Fbundle-redo%2FcreationToken-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1454/derrickstolee/bundle-redo/creationToken-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1454\n\nRange-diff vs v2:\n\n 1: b3828725bc8 < -: ----------- bundle: optionally skip reachability walk\n -: ----------- > 1: f9b0cc872ac bundle: test unbundling with incomplete history\n -: ----------- > 2: 20c29d37f9c bundle: verify using check_connected()\n 2: 427aff4d5e5 = 3: 45cdf9d13a7 t5558: add tests for creationToken heuristic\n 3: f6f8197c9cc = 4: 49bf10e0fd4 bundle-uri: parse bundle.heuristic=creationToken\n 4: 12efa228d04 = 5: ff629bc119b bundle-uri: parse bundle..creationToken values\n 5: 7cfaa3c518c ! 6: 366db5f6931 bundle-uri: download in creationToken order\n @@ bundle-uri.c: static int download_bundle_to_file(struct remote_bundle_info *bund\n +\n +\t\t\t\t/* Try looking deeper in the list. */\n +\t\t\t\tmove_direction = 1;\n -+\t\t\t\tgoto stack_operation;\n ++\t\t\t\tgoto move;\n +\t\t\t}\n +\n +\t\t\t/* We expect bundles when using creationTokens. */\n @@ bundle-uri.c: static int download_bundle_to_file(struct remote_bundle_info *bund\n +\t\t * previous step.\n +\t\t */\n +\n -+stack_operation:\n ++move:\n +\t\t/* Move in the specified direction and repeat. */\n +\t\tcur += move_direction;\n +\t}\n 6: 17c404c1b83 = 7: b59c4e2d390 clone: set fetch.bundleURI if appropriate\n 7: d491070efed = 8: 83f49b37c69 bundle-uri: drop bundle.flag from design doc\n 8: 59e57e04968 ! 9: 314c60f2ae4 fetch: fetch from an external bundle URI\n @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (dry_run)\n \t\twrite_fetch_head = 0;\n \n -+\tif (!git_config_get_string_tmp(\"fetch.bundleuri\", &bundle_uri)) {\n -+\t\tif (fetch_bundle_uri(the_repository, bundle_uri, NULL))\n -+\t\t\twarning(_(\"failed to fetch bundles from '%s'\"), bundle_uri);\n -+\t}\n ++\tif (!git_config_get_string_tmp(\"fetch.bundleuri\", &bundle_uri) &&\n ++\t fetch_bundle_uri(the_repository, bundle_uri, NULL))\n ++\t\twarning(_(\"failed to fetch bundles from '%s'\"), bundle_uri);\n +\n \tif (all) {\n \t\tif (argc == 1)\n 9: 6a1504b1c3a ! 10: 4e0465efd19 bundle-uri: store fetch.bundleCreationToken\n @@ bundle-uri.c: static int fetch_bundles_by_token(struct repository *r,\n \t\t\t}\n \t\t}\n \n -@@ bundle-uri.c: stack_operation:\n +@@ bundle-uri.c: move:\n \t\tcur += move_direction;\n \t}\n \n 10: 676522615ad ! 11: c968b63feba bundle-uri: test missing bundles with heuristic\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heurist\n +\ttest_remote_https_urls actual &&\n +\ttest_cmp expect actual &&\n +\n -+\t# Only base bundle unbundled.\n ++\t# bundle-1 and bundle-3 could unbundle, but bundle-4 could not\n +\tgit -C download-2 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n +\tcat >expect <<-EOF &&\n +\trefs/bundles/base\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'http clone with bundle.heurist\n +\ttest_remote_https_urls actual &&\n +\ttest_cmp expect actual &&\n +\n -+\t# All bundles failed to unbundle\n ++\t# fake.bundle did not unbundle, but the others did.\n +\tgit -C download-3 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n +\tcat >expect <<-EOF &&\n +\trefs/bundles/base\n\n-- \ngitgitgadget\n"},{"id":"471171","messageId":"f9b0cc872ac44892fe6b1c973f16b35edfdc5b20.1675171759.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v3.git.1675171759.gitgitgadget@gmail.com","subject":"[PATCH v3 01/11] bundle: test unbundling with incomplete history","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-31T13:29:09Z","receivedAt":"2023-01-31T13:29:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen verifying a bundle, Git checks first that all prerequisite commits\nexist in the object store, then adds an additional check: those\nprerequisite commits must be reachable from references in the\nrepository.\n\nThis check is stronger than what is checked for refs being added during\n'git fetch', which simply guarantees that the new refs have a complete\nhistory up to the point where it intersects with the current reachable\nhistory.\n\nHowever, we also do not have any tests that check the behavior under\nthis condition. Create a test that demonstrates its behavior.\n\nIn order to construct a broken history, perform a shallow clone of a\nrepository with a linear history, but whose default branch ('base') has\na single commit, so dropping the shallow markers leaves a complete\nhistory from that reference. However, the 'tip' reference adds a\nshallow commit whose parent is missing in the cloned repository. Trying\nto unbundle a bundle with the 'tip' as a prerequisite will succeed past\nthe object store check and move into the reachability check.\n\nThe two errors that are reported are of this form:\n\n error: Could not read \n fatal: Failed to traverse parents of commit \n\nThese messages are not particularly helpful for the person running the\nunbundle command, but they do prevent the command from succeeding.\n\nSigned-off-by: Derrick Stolee \n---\n t/t6020-bundle-misc.sh | 40 ++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 40 insertions(+)\n\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 3a1cf30b1d7..38dbbf89155 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -566,4 +566,44 @@ test_expect_success 'cloning from filtered bundle has useful error' '\n \tgrep \"cannot clone from filtered bundle\" err\n '\n \n+test_expect_success 'verify catches unreachable, broken prerequisites' '\n+\ttest_when_finished rm -rf clone-from clone-to &&\n+\tgit init clone-from &&\n+\t(\n+\t\tcd clone-from &&\n+\t\tgit checkout -b base &&\n+\t\ttest_commit A &&\n+\t\tgit checkout -b tip &&\n+\t\tgit commit --allow-empty -m \"will drop by shallow\" &&\n+\t\tgit commit --allow-empty -m \"will keep by shallow\" &&\n+\t\tgit commit --allow-empty -m \"for bundle, not clone\" &&\n+\t\tgit bundle create tip.bundle tip~1..tip &&\n+\t\tgit reset --hard HEAD~1 &&\n+\t\tgit checkout base\n+\t) &&\n+\tBAD_OID=$(git -C clone-from rev-parse tip~1) &&\n+\tTIP_OID=$(git -C clone-from rev-parse tip) &&\n+\tgit clone --depth=1 --no-single-branch \\\n+\t\t\"file://$(pwd)/clone-from\" clone-to &&\n+\t(\n+\t\tcd clone-to &&\n+\n+\t\t# Set up broken history by removing shallow markers\n+\t\tgit update-ref -d refs/remotes/origin/tip &&\n+\t\trm .git/shallow &&\n+\n+\t\t# Verify should fail\n+\t\ttest_must_fail git bundle verify \\\n+\t\t\t../clone-from/tip.bundle 2>err &&\n+\t\tgrep \"Could not read $BAD_OID\" err &&\n+\t\tgrep \"Failed to traverse parents of commit $TIP_OID\" err &&\n+\n+\t\t# Unbundling should fail\n+\t\ttest_must_fail git bundle unbundle \\\n+\t\t\t../clone-from/tip.bundle 2>err &&\n+\t\tgrep \"Could not read $BAD_OID\" err &&\n+\t\tgrep \"Failed to traverse parents of commit $TIP_OID\" err\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"471173","messageId":"20c29d37f9c1ba1367145331d25dd27f966312cd.1675171759.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v3.git.1675171759.gitgitgadget@gmail.com","subject":"[PATCH v3 02/11] bundle: verify using check_connected()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-31T13:29:10Z","receivedAt":"2023-01-31T13:29:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen Git verifies a bundle to see if it is safe for unbundling, it first\nlooks to see if the prerequisite commits are in the object store. This\nis an easy way to \"fail fast\" but it is not a sufficient check for\nupdating refs that guarantee closure under reachability. There could\nstill be issues if those commits are not reachable from the repository's\nreferences. The repository only has guarantees that its object store is\nclosed under reachability for the objects that are reachable from\nreferences.\n\nThus, the code in verify_bundle() has previously had the additional\ncheck that all prerequisite commits are reachable from repository\nreferences. This is done via a revision walk from all references,\nstopping only if all prerequisite commits are discovered or all commits\nare walked. This uses a custom walk to verify_bundle().\n\nThis check is more strict than what Git applies to fetched pack-files.\nIn the fetch case, Git guarantees that the new references are closed\nunder reachability by walking from the new references until walking\ncommits that are reachable from repository refs. This is done through\nthe well-used check_connected() method.\n\nTo better align with the restrictions required by 'git fetch',\nreimplement this check in verify_bundle() to use check_connected(). This\nalso simplifies the code significantly.\n\nThe previous change added a test that verified the behavior of 'git\nbundle verify' and 'git bundle unbundle' in this case, and the error\nmessages looked like this:\n\n error: Could not read \n fatal: Failed to traverse parents of commit \n\nHowever, by changing the revision walk slightly within check_connected()\nand using its quiet mode, we can omit those messages. Instead, we get\nonly this message, tailored to describing the current state of the\nrepository:\n\n error: some prerequisite commits exist in the object store,\n but are not connected to the repository's history\n\n(Line break added here for the commit message formatting, only.)\n\nWhile this message does not include any object IDs, there is no\nguarantee that those object IDs would help the user diagnose what is\ngoing on, as they could be separated from the prerequisite commits by\nsome distance. At minimum, this situation describes the situation in a\nmore informative way than the previous error messages.\n\nSigned-off-by: Derrick Stolee \n---\n bundle.c | 75 ++++++++++++++++--------------------------\n t/t6020-bundle-misc.sh | 8 ++---\n 2 files changed, 33 insertions(+), 50 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex 4ef7256aa11..76c3a904898 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -12,6 +12,7 @@\n #include \"refs.h\"\n #include \"strvec.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"connected.h\"\n \n static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n@@ -187,6 +188,21 @@ static int list_refs(struct string_list *r, int argc, const char **argv)\n /* Remember to update object flag allocation in object.h */\n #define PREREQ_MARK (1u<<16)\n \n+struct string_list_iterator {\n+\tstruct string_list *list;\n+\tsize_t cur;\n+};\n+\n+static const struct object_id *iterate_ref_map(void *cb_data)\n+{\n+\tstruct string_list_iterator *iter = cb_data;\n+\n+\tif (iter->cur >= iter->list->nr)\n+\t\treturn NULL;\n+\n+\treturn iter->list->items[iter->cur++].util;\n+}\n+\n int verify_bundle(struct repository *r,\n \t\t struct bundle_header *header,\n \t\t enum verify_bundle_flags flags)\n@@ -196,26 +212,25 @@ int verify_bundle(struct repository *r,\n \t * to be verbose about the errors\n \t */\n \tstruct string_list *p = &header->prerequisites;\n-\tstruct rev_info revs = REV_INFO_INIT;\n-\tconst char *argv[] = {NULL, \"--all\", NULL};\n-\tstruct commit *commit;\n-\tint i, ret = 0, req_nr;\n+\tint i, ret = 0;\n \tconst char *message = _(\"Repository lacks these prerequisite commits:\");\n+\tstruct string_list_iterator iter = {\n+\t\t.list = p,\n+\t};\n+\tstruct check_connected_options opts = {\n+\t\t.quiet = 1,\n+\t};\n \n \tif (!r || !r->objects || !r->objects->odb)\n \t\treturn error(_(\"need a repository to verify a bundle\"));\n \n-\trepo_init_revisions(r, &revs, NULL);\n \tfor (i = 0; i < p->nr; i++) {\n \t\tstruct string_list_item *e = p->items + i;\n \t\tconst char *name = e->string;\n \t\tstruct object_id *oid = e->util;\n \t\tstruct object *o = parse_object(r, oid);\n-\t\tif (o) {\n-\t\t\to->flags |= PREREQ_MARK;\n-\t\t\tadd_pending_object(&revs, o, name);\n+\t\tif (o)\n \t\t\tcontinue;\n-\t\t}\n \t\tret++;\n \t\tif (flags & VERIFY_BUNDLE_QUIET)\n \t\t\tcontinue;\n@@ -223,37 +238,14 @@ int verify_bundle(struct repository *r,\n \t\t\terror(\"%s\", message);\n \t\terror(\"%s %s\", oid_to_hex(oid), name);\n \t}\n-\tif (revs.pending.nr != p->nr)\n+\tif (ret)\n \t\tgoto cleanup;\n-\treq_nr = revs.pending.nr;\n-\tsetup_revisions(2, argv, &revs, NULL);\n-\n-\tlist_objects_filter_copy(&revs.filter, &header->filter);\n-\n-\tif (prepare_revision_walk(&revs))\n-\t\tdie(_(\"revision walk setup failed\"));\n \n-\ti = req_nr;\n-\twhile (i && (commit = get_revision(&revs)))\n-\t\tif (commit->object.flags & PREREQ_MARK)\n-\t\t\ti--;\n-\n-\tfor (i = 0; i < p->nr; i++) {\n-\t\tstruct string_list_item *e = p->items + i;\n-\t\tconst char *name = e->string;\n-\t\tconst struct object_id *oid = e->util;\n-\t\tstruct object *o = parse_object(r, oid);\n-\t\tassert(o); /* otherwise we'd have returned early */\n-\t\tif (o->flags & SHOWN)\n-\t\t\tcontinue;\n-\t\tret++;\n-\t\tif (flags & VERIFY_BUNDLE_QUIET)\n-\t\t\tcontinue;\n-\t\tif (ret == 1)\n-\t\t\terror(\"%s\", message);\n-\t\terror(\"%s %s\", oid_to_hex(oid), name);\n-\t}\n+\tif ((ret = check_connected(iterate_ref_map, &iter, &opts)))\n+\t\terror(_(\"some prerequisite commits exist in the object store, \"\n+\t\t\t\"but are not connected to the repository's history\"));\n \n+\t/* TODO: preserve this verbose language. */\n \tif (flags & VERIFY_BUNDLE_VERBOSE) {\n \t\tstruct string_list *r;\n \n@@ -282,15 +274,6 @@ int verify_bundle(struct repository *r,\n \t\t\t\t list_objects_filter_spec(&header->filter));\n \t}\n cleanup:\n-\t/* Clean up objects used, as they will be reused. */\n-\tfor (i = 0; i < p->nr; i++) {\n-\t\tstruct string_list_item *e = p->items + i;\n-\t\tstruct object_id *oid = e->util;\n-\t\tcommit = lookup_commit_reference_gently(r, oid, 1);\n-\t\tif (commit)\n-\t\t\tclear_commit_marks(commit, ALL_REV_FLAGS | PREREQ_MARK);\n-\t}\n-\trelease_revisions(&revs);\n \treturn ret;\n }\n \ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 38dbbf89155..7d40994991e 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -595,14 +595,14 @@ test_expect_success 'verify catches unreachable, broken prerequisites' '\n \t\t# Verify should fail\n \t\ttest_must_fail git bundle verify \\\n \t\t\t../clone-from/tip.bundle 2>err &&\n-\t\tgrep \"Could not read $BAD_OID\" err &&\n-\t\tgrep \"Failed to traverse parents of commit $TIP_OID\" err &&\n+\t\tgrep \"some prerequisite commits .* are not connected\" err &&\n+\t\ttest_line_count = 1 err &&\n \n \t\t# Unbundling should fail\n \t\ttest_must_fail git bundle unbundle \\\n \t\t\t../clone-from/tip.bundle 2>err &&\n-\t\tgrep \"Could not read $BAD_OID\" err &&\n-\t\tgrep \"Failed to traverse parents of commit $TIP_OID\" err\n+\t\tgrep \"some prerequisite commits .* are not connected\" err &&\n+\t\ttest_line_count = 1 err\n \t)\n '\n \n-- \ngitgitgadget\n\n"},{"id":"471174","messageId":"45cdf9d13a7e0b1e68932defe46a790f6813ed96.1675171759.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v3.git.1675171759.gitgitgadget@gmail.com","subject":"[PATCH v3 03/11] t5558: add tests for creationToken heuristic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-31T13:29:11Z","receivedAt":"2023-01-31T13:29:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs documented in the bundle URI design doc in 2da14fad8fe (docs:\ndocument bundle URI standard, 2022-08-09), the 'creationToken' member of\na bundle URI allows a bundle provider to specify a total order on the\nbundles.\n\nFuture changes will allow the Git client to understand these members and\nmodify its behavior around downloading the bundles in that order. In the\nmeantime, create tests that add creation tokens to the bundle list. For\nnow, the Git client correctly ignores these unknown keys.\n\nCreate a new test helper function, test_remote_https_urls, which filters\nGIT_TRACE2_EVENT output to extract a list of URLs passed to\ngit-remote-https child processes. This can be used to verify the order\nof these requests as we implement the creationToken heuristic. For now,\nwe need to sort the actual output since the current client does not have\na well-defined order that it applies to the bundles.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5558-clone-bundle-uri.sh | 69 +++++++++++++++++++++++++++++++++++--\n t/test-lib-functions.sh | 8 +++++\n 2 files changed, 75 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 9155f31fa2c..474432c8ace 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -285,6 +285,8 @@ test_expect_success 'clone HTTP bundle' '\n '\n \n test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\n \tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n \tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n \t[bundle]\n@@ -304,12 +306,26 @@ test_expect_success 'clone bundle list (HTTP, no heuristic)' '\n \t\turi = $HTTPD_URL/bundle-4.bundle\n \tEOF\n \n-\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" \\\n+\t\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n \t\tclone-from clone-list-http 2>err &&\n \t! grep \"Repository lacks these prerequisite commits\" err &&\n \n \tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n-\tgit -C clone-list-http cat-file --batch-check expect <<-EOF &&\n+\t$HTTPD_URL/bundle-1.bundle\n+\t$HTTPD_URL/bundle-2.bundle\n+\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/bundle-4.bundle\n+\t$HTTPD_URL/bundle-list\n+\tEOF\n+\n+\t# Sort the list, since the order is not well-defined\n+\t# without a heuristic.\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual\n '\n \n test_expect_success 'clone bundle list (HTTP, any mode)' '\n@@ -350,6 +366,55 @@ test_expect_success 'clone bundle list (HTTP, any mode)' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'clone bundle list (http, creationToken)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\n+\tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" git \\\n+\t\tclone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\t\"$HTTPD_URL/smart/fetch.git\" clone-list-http-2 &&\n+\n+\tgit -C clone-from for-each-ref --format=\"%(objectname)\" >oids &&\n+\tgit -C clone-list-http-2 cat-file --batch-check expect <<-EOF &&\n+\t$HTTPD_URL/bundle-1.bundle\n+\t$HTTPD_URL/bundle-2.bundle\n+\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/bundle-4.bundle\n+\t$HTTPD_URL/bundle-list\n+\tEOF\n+\n+\t# Since the creationToken heuristic is not yet understood by the\n+\t# client, the order cannot be verified at this moment. Sort the\n+\t# list for consistent results.\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual\n+'\n+\n # Do not add tests here unless they use the HTTP server, as they will\n # not run unless the HTTP dependencies exist.\n \ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex f036c4d3003..ace542f4226 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1833,6 +1833,14 @@ test_region () {\n \treturn 0\n }\n \n+# Given a GIT_TRACE2_EVENT log over stdin, writes to stdout a list of URLs\n+# sent to git-remote-https child processes.\n+test_remote_https_urls() {\n+\tgrep -e '\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\".*\"\\]' |\n+\t\tsed -e 's/{\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"//g' \\\n+\t\t -e 's/\"\\]}//g'\n+}\n+\n # Print the destination of symlink(s) provided as arguments. Basically\n # the same as the readlink command, but it's not available everywhere.\n test_readlink () {\n-- \ngitgitgadget\n\n"},{"id":"471175","messageId":"49bf10e0fd4468720cd86cc47e425bf446a1a4f7.1675171759.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v3.git.1675171759.gitgitgadget@gmail.com","subject":"[PATCH v3 04/11] bundle-uri: parse bundle.heuristic=creationToken","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-31T13:29:12Z","receivedAt":"2023-01-31T13:29:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe bundle.heuristic value communicates that the bundle list is\norganized to make use of the bundle..creationToken values that may\nbe provided in the bundle list. Those values will create a total order\non the bundles, allowing the Git client to download them in a specific\norder and even remember previously-downloaded bundles by storing the\nmaximum creation token value.\n\nBefore implementing any logic that parses or uses the\nbundle..creationToken values, teach Git to parse the\nbundle.heuristic value from a bundle list. We can use 'test-tool\nbundle-uri' to print the heuristic value and verify that the parsing\nworks correctly.\n\nAs an extra precaution, create the internal 'heuristics' array to be a\nlist of (enum, string) pairs so we can iterate through the array entries\ncarefully, regardless of the enum values.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/bundle.txt | 7 +++++++\n bundle-uri.c | 34 +++++++++++++++++++++++++++++++++\n bundle-uri.h | 14 ++++++++++++++\n t/t5750-bundle-uri-parse.sh | 19 ++++++++++++++++++\n 4 files changed, 74 insertions(+)\n\ndiff --git a/Documentation/config/bundle.txt b/Documentation/config/bundle.txt\nindex daa21eb674a..3faae386853 100644\n--- a/Documentation/config/bundle.txt\n+++ b/Documentation/config/bundle.txt\n@@ -15,6 +15,13 @@ bundle.mode::\n \tcomplete understanding of the bundled information (`all`) or if any one\n \tof the listed bundle URIs is sufficient (`any`).\n \n+bundle.heuristic::\n+\tIf this string-valued key exists, then the bundle list is designed to\n+\twork well with incremental `git fetch` commands. The heuristic signals\n+\tthat there are additional keys available for each bundle that help\n+\tdetermine which subset of bundles the client should download. The\n+\tonly value currently understood is `creationToken`.\n+\n bundle..*::\n \tThe `bundle..*` keys are used to describe a single item in the\n \tbundle list, grouped under `` for identification purposes.\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 36268dda172..36ec542718d 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -9,6 +9,14 @@\n #include \"config.h\"\n #include \"remote.h\"\n \n+static struct {\n+\tenum bundle_list_heuristic heuristic;\n+\tconst char *name;\n+} heuristics[BUNDLE_HEURISTIC__COUNT] = {\n+\t{ BUNDLE_HEURISTIC_NONE, \"\"},\n+\t{ BUNDLE_HEURISTIC_CREATIONTOKEN, \"creationToken\" },\n+};\n+\n static int compare_bundles(const void *hashmap_cmp_fn_data,\n \t\t\t const struct hashmap_entry *he1,\n \t\t\t const struct hashmap_entry *he2,\n@@ -100,6 +108,17 @@ void print_bundle_list(FILE *fp, struct bundle_list *list)\n \tfprintf(fp, \"\\tversion = %d\\n\", list->version);\n \tfprintf(fp, \"\\tmode = %s\\n\", mode);\n \n+\tif (list->heuristic) {\n+\t\tint i;\n+\t\tfor (i = 0; i < BUNDLE_HEURISTIC__COUNT; i++) {\n+\t\t\tif (heuristics[i].heuristic == list->heuristic) {\n+\t\t\t\tprintf(\"\\theuristic = %s\\n\",\n+\t\t\t\t heuristics[list->heuristic].name);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n \tfor_all_bundles_in_list(list, summarize_bundle, fp);\n }\n \n@@ -142,6 +161,21 @@ static int bundle_list_update(const char *key, const char *value,\n \t\t\treturn 0;\n \t\t}\n \n+\t\tif (!strcmp(subkey, \"heuristic\")) {\n+\t\t\tint i;\n+\t\t\tfor (i = 0; i < BUNDLE_HEURISTIC__COUNT; i++) {\n+\t\t\t\tif (heuristics[i].heuristic &&\n+\t\t\t\t heuristics[i].name &&\n+\t\t\t\t !strcmp(value, heuristics[i].name)) {\n+\t\t\t\t\tlist->heuristic = heuristics[i].heuristic;\n+\t\t\t\t\treturn 0;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\t/* Ignore unknown heuristics. */\n+\t\t\treturn 0;\n+\t\t}\n+\n \t\t/* Ignore other unknown global keys. */\n \t\treturn 0;\n \t}\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex d5e89f1671c..2e44a50a90b 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -52,6 +52,14 @@ enum bundle_list_mode {\n \tBUNDLE_MODE_ANY\n };\n \n+enum bundle_list_heuristic {\n+\tBUNDLE_HEURISTIC_NONE = 0,\n+\tBUNDLE_HEURISTIC_CREATIONTOKEN,\n+\n+\t/* Must be last. */\n+\tBUNDLE_HEURISTIC__COUNT\n+};\n+\n /**\n * A bundle_list contains an unordered set of remote_bundle_info structs,\n * as well as information about the bundle listing, such as version and\n@@ -75,6 +83,12 @@ struct bundle_list {\n \t * advertised by the bundle list at that location.\n \t */\n \tchar *baseURI;\n+\n+\t/**\n+\t * A list can have a heuristic, which helps reduce the number of\n+\t * downloaded bundles.\n+\t */\n+\tenum bundle_list_heuristic heuristic;\n };\n \n void init_bundle_list(struct bundle_list *list);\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex 7b4f930e532..6fc92a9c0d4 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -250,4 +250,23 @@ test_expect_success 'parse config format edge cases: empty key or value' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format: creationToken heuristic' '\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config expect >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"471176","messageId":"ff629bc119b466dcd827f758b3d892fefd6a9703.1675171760.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v3.git.1675171759.gitgitgadget@gmail.com","subject":"[PATCH v3 05/11] bundle-uri: parse bundle..creationToken values","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-31T13:29:13Z","receivedAt":"2023-01-31T13:29:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change taught Git to parse the bundle.heuristic value,\nespecially when its value is \"creationToken\". Now, teach Git to parse\nthe bundle..creationToken values on each bundle in a bundle list.\n\nBefore implementing any logic based on creationToken values for the\ncreationToken heuristic, parse and print these values for testing\npurposes.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 10 ++++++++++\n bundle-uri.h | 6 ++++++\n t/t5750-bundle-uri-parse.sh | 18 ++++++++++++++++++\n 3 files changed, 34 insertions(+)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 36ec542718d..d4277b2e3a7 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -83,6 +83,9 @@ static int summarize_bundle(struct remote_bundle_info *info, void *data)\n \tFILE *fp = data;\n \tfprintf(fp, \"[bundle \\\"%s\\\"]\\n\", info->id);\n \tfprintf(fp, \"\\turi = %s\\n\", info->uri);\n+\n+\tif (info->creationToken)\n+\t\tfprintf(fp, \"\\tcreationToken = %\"PRIu64\"\\n\", info->creationToken);\n \treturn 0;\n }\n \n@@ -203,6 +206,13 @@ static int bundle_list_update(const char *key, const char *value,\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(subkey, \"creationtoken\")) {\n+\t\tif (sscanf(value, \"%\"PRIu64, &bundle->creationToken) != 1)\n+\t\t\twarning(_(\"could not parse bundle list key %s with value '%s'\"),\n+\t\t\t\t\"creationToken\", value);\n+\t\treturn 0;\n+\t}\n+\n \t/*\n \t * At this point, we ignore any information that we don't\n \t * understand, assuming it to be hints for a heuristic the client\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 2e44a50a90b..ef32840bfa6 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -42,6 +42,12 @@ struct remote_bundle_info {\n \t * this boolean is true.\n \t */\n \tunsigned unbundled:1;\n+\n+\t/**\n+\t * If the bundle is part of a list with the creationToken\n+\t * heuristic, then we use this member for sorting the bundles.\n+\t */\n+\tuint64_t creationToken;\n };\n \n #define REMOTE_BUNDLE_INFO_INIT { 0 }\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex 6fc92a9c0d4..81bdf58b944 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -258,10 +258,13 @@ test_expect_success 'parse config format: creationToken heuristic' '\n \t\theuristic = creationToken\n \t[bundle \"one\"]\n \t\turi = http://example.com/bundle.bdl\n+\t\tcreationToken = 123456\n \t[bundle \"two\"]\n \t\turi = https://example.com/bundle.bdl\n+\t\tcreationToken = 12345678901234567890\n \t[bundle \"three\"]\n \t\turi = file:///usr/share/git/bundle.bdl\n+\t\tcreationToken = 1\n \tEOF\n \n \ttest-tool bundle-uri parse-config expect >actual 2>err &&\n@@ -269,4 +272,19 @@ test_expect_success 'parse config format: creationToken heuristic' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format edge cases: creationToken heuristic' '\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t\tcreationToken = bogus\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config expect >actual 2>err &&\n+\tgrep \"could not parse bundle list key creationToken with value '\\''bogus'\\''\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"471177","messageId":"b59c4e2d39032628e029eb830f82963787eca176.1675171760.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v3.git.1675171759.gitgitgadget@gmail.com","subject":"[PATCH v3 07/11] clone: set fetch.bundleURI if appropriate","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-31T13:29:15Z","receivedAt":"2023-01-31T13:29:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBundle providers may organize their bundle lists in a way that is\nintended to improve incremental fetches, not just initial clones.\nHowever, they do need to state that they have organized with that in\nmind, or else the client will not expect to save time by downloading\nbundles after the initial clone. This is done by specifying a\nbundle.heuristic value.\n\nThere are two types of bundle lists: those at a static URI and those\nthat are advertised from a Git remote over protocol v2.\n\nThe new fetch.bundleURI config value applies for static bundle URIs that\nare not advertised over protocol v2. If the user specifies a static URI\nvia 'git clone --bundle-uri', then Git can set this config as a reminder\nfor future 'git fetch' operations to check the bundle list before\nconnecting to the remote(s).\n\nFor lists provided over protocol v2, we will want to take a different\napproach and create a property of the remote itself by creating a\nremote..* type config key. That is not implemented in this change.\n\nLater changes will update 'git fetch' to consume this option.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/fetch.txt | 8 +++++++\n builtin/clone.c | 6 +++++-\n bundle-uri.c | 5 ++++-\n bundle-uri.h | 8 ++++++-\n t/t5558-clone-bundle-uri.sh | 39 ++++++++++++++++++++++++++++++++++\n 5 files changed, 63 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt\nindex cd65d236b43..244f44d460f 100644\n--- a/Documentation/config/fetch.txt\n+++ b/Documentation/config/fetch.txt\n@@ -96,3 +96,11 @@ fetch.writeCommitGraph::\n \tmerge and the write may take longer. Having an updated commit-graph\n \tfile helps performance of many Git commands, including `git merge-base`,\n \t`git push -f`, and `git log --graph`. Defaults to false.\n+\n+fetch.bundleURI::\n+\tThis value stores a URI for downloading Git object data from a bundle\n+\tURI before performing an incremental fetch from the origin Git server.\n+\tThis is similar to how the `--bundle-uri` option behaves in\n+\tlinkgit:git-clone[1]. `git clone --bundle-uri` will set the\n+\t`fetch.bundleURI` value if the supplied bundle URI contains a bundle\n+\tlist that is organized for incremental fetches.\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 5453ba5277f..5370617664d 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1248,12 +1248,16 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t * data from the --bundle-uri option.\n \t */\n \tif (bundle_uri) {\n+\t\tint has_heuristic = 0;\n+\n \t\t/* At this point, we need the_repository to match the cloned repo. */\n \t\tif (repo_init(the_repository, git_dir, work_tree))\n \t\t\twarning(_(\"failed to initialize the repo, skipping bundle URI\"));\n-\t\telse if (fetch_bundle_uri(the_repository, bundle_uri))\n+\t\telse if (fetch_bundle_uri(the_repository, bundle_uri, &has_heuristic))\n \t\t\twarning(_(\"failed to fetch objects from bundle URI '%s'\"),\n \t\t\t\tbundle_uri);\n+\t\telse if (has_heuristic)\n+\t\t\tgit_config_set_gently(\"fetch.bundleuri\", bundle_uri);\n \t}\n \n \tstrvec_push(&transport_ls_refs_options.ref_prefixes, \"HEAD\");\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex af48938d243..7a1b6d94bf5 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -736,7 +736,8 @@ static int unlink_bundle(struct remote_bundle_info *info, void *data)\n \treturn 0;\n }\n \n-int fetch_bundle_uri(struct repository *r, const char *uri)\n+int fetch_bundle_uri(struct repository *r, const char *uri,\n+\t\t int *has_heuristic)\n {\n \tint result;\n \tstruct bundle_list list;\n@@ -756,6 +757,8 @@ int fetch_bundle_uri(struct repository *r, const char *uri)\n \tresult = unbundle_all_bundles(r, &list);\n \n cleanup:\n+\tif (has_heuristic)\n+\t\t*has_heuristic = (list.heuristic != BUNDLE_HEURISTIC_NONE);\n \tfor_all_bundles_in_list(&list, unlink_bundle, NULL);\n \tclear_bundle_list(&list);\n \tclear_remote_bundle_info(&bundle, NULL);\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex ef32840bfa6..6dbc780f661 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -124,8 +124,14 @@ int bundle_uri_parse_config_format(const char *uri,\n * based on that information.\n *\n * Returns non-zero if no bundle information is found at the given 'uri'.\n+ *\n+ * If the pointer 'has_heuristic' is non-NULL, then the value it points to\n+ * will be set to be non-zero if and only if the fetched list has a\n+ * heuristic value. Such a value indicates that the list was designed for\n+ * incremental fetches.\n */\n-int fetch_bundle_uri(struct repository *r, const char *uri);\n+int fetch_bundle_uri(struct repository *r, const char *uri,\n+\t\t int *has_heuristic);\n \n /**\n * Given a bundle list that was already advertised (likely by the\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 6f9417a0afb..b2d15e141ca 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -432,6 +432,8 @@ test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n \t\t--single-branch --branch=base --no-tags \\\n \t\t\"$HTTPD_URL/smart/fetch.git\" clone-token-http &&\n \n+\ttest_cmp_config -C clone-token-http \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n+\n \tcat >expect <<-EOF &&\n \t$HTTPD_URL/bundle-list\n \t$HTTPD_URL/bundle-1.bundle\n@@ -441,6 +443,43 @@ test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n+\ttest_when_finished rm -rf fetch-http-4 trace*.txt &&\n+\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" \\\n+\tgit clone --single-branch --branch=base \\\n+\t\t--bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\t\"$HTTPD_URL/smart/fetch.git\" fetch-http-4 &&\n+\n+\ttest_cmp_config -C fetch-http-4 \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-1.bundle\n+\tEOF\n+\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# only received base ref from bundle-1\n+\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\tEOF\n+\ttest_cmp expect refs\n+'\n+\n # Do not add tests here unless they use the HTTP server, as they will\n # not run unless the HTTP dependencies exist.\n \n-- \ngitgitgadget\n\n"},{"id":"471179","messageId":"366db5f6931df1a2fc832d998a69ac010e59293c.1675171760.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v3.git.1675171759.gitgitgadget@gmail.com","subject":"[PATCH v3 06/11] bundle-uri: download in creationToken order","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-31T13:29:14Z","receivedAt":"2023-01-31T13:29:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe creationToken heuristic provides an ordering on the bundles\nadvertised by a bundle list. Teach the Git client to download bundles\ndifferently when this heuristic is advertised.\n\nThe bundles in the list are sorted by their advertised creationToken\nvalues, then downloaded in decreasing order. This avoids the previous\nstrategy of downloading bundles in an arbitrary order and attempting\nto apply them (likely failing in the case of required commits) until\ndiscovering the order through attempted unbundling.\n\nDuring a fresh 'git clone', it may make sense to download the bundles in\nincreasing order, since that would prevent the need to attempt\nunbundling a bundle with required commits that do not exist in our empty\nobject store. The cost of testing an unbundle is quite low, and instead\nthe chosen order is optimizing for a future bundle download during a\n'git fetch' operation with a non-empty object store.\n\nSince the Git client continues fetching from the Git remote after\ndownloading and unbundling bundles, the client's object store can be\nahead of the bundle provider's object store. The next time it attempts\nto download from the bundle list, it makes most sense to download only\nthe most-recent bundles until all tips successfully unbundle. The\nstrategy implemented here provides that short-circuit where the client\ndownloads a minimal set of bundles.\n\nHowever, we are not satisfied by the naive approach of downloading\nbundles until one successfully unbundles, expecting the earlier bundles\nto successfully unbundle now. The example repository in t5558\ndemonstrates this well:\n\n ---------------- bundle-4\n\n 4\n / \\\n ----|---|------- bundle-3\n | |\n | 3\n | |\n ----|---|------- bundle-2\n | |\n 2 |\n | |\n ----|---|------- bundle-1\n \\ /\n 1\n |\n (previous commits)\n\nIn this repository, if we already have the objects for bundle-1 and then\ntry to fetch from this list, the naive approach will fail. bundle-4\nrequires both bundle-3 and bundle-2, though bundle-3 will successfully\nunbundle without bundle-2. Thus, the algorithm needs to keep this in\nmind.\n\nA later implementation detail will store the maximum creationToken seen\nduring such a bundle download, and the client will avoid downloading a\nbundle unless its creationToken is strictly greater than that stored\nvalue. For now, if the client seeks to download from an identical\nbundle list since its previous download, it will download the\nmost-recent bundle then stop since its required commits are already in\nthe object store.\n\nAdd tests that exercise this behavior, but we will expand upon these\ntests when incremental downloads during 'git fetch' make use of\ncreationToken values.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 156 +++++++++++++++++++++++++++++++++++-\n t/t5558-clone-bundle-uri.sh | 40 +++++++--\n t/t5601-clone.sh | 46 +++++++++++\n 3 files changed, 233 insertions(+), 9 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex d4277b2e3a7..af48938d243 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -447,6 +447,139 @@ static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data\n \treturn 0;\n }\n \n+struct bundles_for_sorting {\n+\tstruct remote_bundle_info **items;\n+\tsize_t alloc;\n+\tsize_t nr;\n+};\n+\n+static int append_bundle(struct remote_bundle_info *bundle, void *data)\n+{\n+\tstruct bundles_for_sorting *list = data;\n+\tlist->items[list->nr++] = bundle;\n+\treturn 0;\n+}\n+\n+/**\n+ * For use in QSORT() to get a list sorted by creationToken\n+ * in decreasing order.\n+ */\n+static int compare_creation_token_decreasing(const void *va, const void *vb)\n+{\n+\tconst struct remote_bundle_info * const *a = va;\n+\tconst struct remote_bundle_info * const *b = vb;\n+\n+\tif ((*a)->creationToken > (*b)->creationToken)\n+\t\treturn -1;\n+\tif ((*a)->creationToken < (*b)->creationToken)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int fetch_bundles_by_token(struct repository *r,\n+\t\t\t\t struct bundle_list *list)\n+{\n+\tint cur;\n+\tint move_direction = 0;\n+\tstruct bundle_list_context ctx = {\n+\t\t.r = r,\n+\t\t.list = list,\n+\t\t.mode = list->mode,\n+\t};\n+\tstruct bundles_for_sorting bundles = {\n+\t\t.alloc = hashmap_get_size(&list->bundles),\n+\t};\n+\n+\tALLOC_ARRAY(bundles.items, bundles.alloc);\n+\n+\tfor_all_bundles_in_list(list, append_bundle, &bundles);\n+\n+\tQSORT(bundles.items, bundles.nr, compare_creation_token_decreasing);\n+\n+\t/*\n+\t * Attempt to download and unbundle the minimum number of bundles by\n+\t * creationToken in decreasing order. If we fail to unbundle (after\n+\t * a successful download) then move to the next non-downloaded bundle\n+\t * and attempt downloading. Once we succeed in applying a bundle,\n+\t * move to the previous unapplied bundle and attempt to unbundle it\n+\t * again.\n+\t *\n+\t * In the case of a fresh clone, we will likely download all of the\n+\t * bundles before successfully unbundling the oldest one, then the\n+\t * rest of the bundles unbundle successfully in increasing order\n+\t * of creationToken.\n+\t *\n+\t * If there are existing objects, then this process may terminate\n+\t * early when all required commits from \"new\" bundles exist in the\n+\t * repo's object store.\n+\t */\n+\tcur = 0;\n+\twhile (cur >= 0 && cur < bundles.nr) {\n+\t\tstruct remote_bundle_info *bundle = bundles.items[cur];\n+\t\tif (!bundle->file) {\n+\t\t\t/*\n+\t\t\t * Not downloaded yet. Try downloading.\n+\t\t\t *\n+\t\t\t * Note that bundle->file is non-NULL if a download\n+\t\t\t * was attempted, even if it failed to download.\n+\t\t\t */\n+\t\t\tif (fetch_bundle_uri_internal(ctx.r, bundle, ctx.depth + 1, ctx.list)) {\n+\t\t\t\t/* Mark as unbundled so we do not retry. */\n+\t\t\t\tbundle->unbundled = 1;\n+\n+\t\t\t\t/* Try looking deeper in the list. */\n+\t\t\t\tmove_direction = 1;\n+\t\t\t\tgoto move;\n+\t\t\t}\n+\n+\t\t\t/* We expect bundles when using creationTokens. */\n+\t\t\tif (!is_bundle(bundle->file, 1)) {\n+\t\t\t\twarning(_(\"file downloaded from '%s' is not a bundle\"),\n+\t\t\t\t\tbundle->uri);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (bundle->file && !bundle->unbundled) {\n+\t\t\t/*\n+\t\t\t * This was downloaded, but not successfully\n+\t\t\t * unbundled. Try unbundling again.\n+\t\t\t */\n+\t\t\tif (unbundle_from_file(ctx.r, bundle->file)) {\n+\t\t\t\t/* Try looking deeper in the list. */\n+\t\t\t\tmove_direction = 1;\n+\t\t\t} else {\n+\t\t\t\t/*\n+\t\t\t\t * Succeeded in unbundle. Retry bundles\n+\t\t\t\t * that previously failed to unbundle.\n+\t\t\t\t */\n+\t\t\t\tmove_direction = -1;\n+\t\t\t\tbundle->unbundled = 1;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/*\n+\t\t * Else case: downloaded and unbundled successfully.\n+\t\t * Skip this by moving in the same direction as the\n+\t\t * previous step.\n+\t\t */\n+\n+move:\n+\t\t/* Move in the specified direction and repeat. */\n+\t\tcur += move_direction;\n+\t}\n+\n+\tfree(bundles.items);\n+\n+\t/*\n+\t * We succeed if the loop terminates because 'cur' drops below\n+\t * zero. The other case is that we terminate because 'cur'\n+\t * reaches the end of the list, so we have a failure no matter\n+\t * which bundles we apply from the list.\n+\t */\n+\treturn cur >= 0;\n+}\n+\n static int download_bundle_list(struct repository *r,\n \t\t\t\tstruct bundle_list *local_list,\n \t\t\t\tstruct bundle_list *global_list,\n@@ -484,7 +617,15 @@ static int fetch_bundle_list_in_config_format(struct repository *r,\n \t\tgoto cleanup;\n \t}\n \n-\tif ((result = download_bundle_list(r, &list_from_bundle,\n+\t/*\n+\t * If this list uses the creationToken heuristic, then the URIs\n+\t * it advertises are expected to be bundles, not nested lists.\n+\t * We can drop 'global_list' and 'depth'.\n+\t */\n+\tif (list_from_bundle.heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN) {\n+\t\tresult = fetch_bundles_by_token(r, &list_from_bundle);\n+\t\tglobal_list->heuristic = BUNDLE_HEURISTIC_CREATIONTOKEN;\n+\t} else if ((result = download_bundle_list(r, &list_from_bundle,\n \t\t\t\t\t global_list, depth)))\n \t\tgoto cleanup;\n \n@@ -626,6 +767,14 @@ int fetch_bundle_list(struct repository *r, struct bundle_list *list)\n \tint result;\n \tstruct bundle_list global_list;\n \n+\t/*\n+\t * If the creationToken heuristic is used, then the URIs\n+\t * advertised by 'list' are not nested lists and instead\n+\t * direct bundles. We do not need to use global_list.\n+\t */\n+\tif (list->heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN)\n+\t\treturn fetch_bundles_by_token(r, list);\n+\n \tinit_bundle_list(&global_list);\n \n \t/* If a bundle is added to this global list, then it is required. */\n@@ -634,7 +783,10 @@ int fetch_bundle_list(struct repository *r, struct bundle_list *list)\n \tif ((result = download_bundle_list(r, list, &global_list, 0)))\n \t\tgoto cleanup;\n \n-\tresult = unbundle_all_bundles(r, &global_list);\n+\tif (list->heuristic == BUNDLE_HEURISTIC_CREATIONTOKEN)\n+\t\tresult = fetch_bundles_by_token(r, list);\n+\telse\n+\t\tresult = unbundle_all_bundles(r, &global_list);\n \n cleanup:\n \tfor_all_bundles_in_list(&global_list, unlink_bundle, NULL);\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 474432c8ace..6f9417a0afb 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -401,17 +401,43 @@ test_expect_success 'clone bundle list (http, creationToken)' '\n \tgit -C clone-list-http-2 cat-file --batch-check expect <<-EOF &&\n-\t$HTTPD_URL/bundle-1.bundle\n-\t$HTTPD_URL/bundle-2.bundle\n-\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/bundle-list\n \t$HTTPD_URL/bundle-4.bundle\n+\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/bundle-2.bundle\n+\t$HTTPD_URL/bundle-1.bundle\n+\tEOF\n+\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\n+\tcp clone-from/bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\" &&\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=$(pwd)/trace-clone.txt \\\n+\tgit clone --bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\t--single-branch --branch=base --no-tags \\\n+\t\t\"$HTTPD_URL/smart/fetch.git\" clone-token-http &&\n+\n+\tcat >expect <<-EOF &&\n \t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-1.bundle\n \tEOF\n \n-\t# Since the creationToken heuristic is not yet understood by the\n-\t# client, the order cannot be verified at this moment. Sort the\n-\t# list for consistent results.\n-\ttest_remote_https_urls actual &&\n+\ttest_remote_https_urls actual &&\n \ttest_cmp expect actual\n '\n \ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 1928ea1dd7c..b7d5551262c 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -831,6 +831,52 @@ test_expect_success 'auto-discover multiple bundles from HTTP clone' '\n \tgrep -f pattern trace.txt\n '\n \n+test_expect_success 'auto-discover multiple bundles from HTTP clone: creationToken heuristic' '\n+\ttest_when_finished rm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/repo4.git\" &&\n+\ttest_when_finished rm -rf clone-heuristic trace*.txt &&\n+\n+\ttest_commit -C src newest &&\n+\tgit -C src bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/newest.bundle\" HEAD~1..HEAD &&\n+\tgit clone --bare --no-local src \"$HTTPD_DOCUMENT_ROOT_PATH/repo4.git\" &&\n+\n+\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/repo4.git/config\" <<-EOF &&\n+\t[uploadPack]\n+\t\tadvertiseBundleURIs = true\n+\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"everything\"]\n+\t\turi = $HTTPD_URL/everything.bundle\n+\t\tcreationtoken = 1\n+\n+\t[bundle \"new\"]\n+\t\turi = $HTTPD_URL/new.bundle\n+\t\tcreationtoken = 2\n+\n+\t[bundle \"newest\"]\n+\t\turi = $HTTPD_URL/newest.bundle\n+\t\tcreationtoken = 3\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone.txt\" \\\n+\t\tgit -c protocol.version=2 \\\n+\t\t -c transfer.bundleURI=true clone \\\n+\t\t\"$HTTPD_URL/smart/repo4.git\" clone-heuristic &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/newest.bundle\n+\t$HTTPD_URL/new.bundle\n+\t$HTTPD_URL/everything.bundle\n+\tEOF\n+\n+\t# We should fetch all bundles in the expected order.\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual\n+'\n+\n # DO NOT add non-httpd-specific tests here, because the last part of this\n # test script is only executed when httpd is available and enabled.\n \n-- \ngitgitgadget\n\n"},{"id":"471178","messageId":"83f49b37c6975a5898e66a420f13a70ef25afc59.1675171760.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v3.git.1675171759.gitgitgadget@gmail.com","subject":"[PATCH v3 08/11] bundle-uri: drop bundle.flag from design doc","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-31T13:29:16Z","receivedAt":"2023-01-31T13:29:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe Implementation Plan section lists a 'bundle.flag' option that is not\ndocumented anywhere else. What is documented elsewhere in the document\nand implemented by previous changes is the 'bundle.heuristic' config\nkey. For now, a heuristic is required to indicate that a bundle list is\norganized for use during 'git fetch', and it is also sufficient for all\nexisting designs.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/bundle-uri.txt | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/technical/bundle-uri.txt b/Documentation/technical/bundle-uri.txt\nindex b78d01d9adf..91d3a13e327 100644\n--- a/Documentation/technical/bundle-uri.txt\n+++ b/Documentation/technical/bundle-uri.txt\n@@ -479,14 +479,14 @@ outline for submitting these features:\n (This choice is an opt-in via a config option and a command-line\n option.)\n \n-4. Allow the client to understand the `bundle.flag=forFetch` configuration\n+4. Allow the client to understand the `bundle.heuristic` configuration key\n and the `bundle..creationToken` heuristic. When `git clone`\n- discovers a bundle URI with `bundle.flag=forFetch`, it configures the\n- client repository to check that bundle URI during later `git fetch `\n+ discovers a bundle URI with `bundle.heuristic`, it configures the client\n+ repository to check that bundle URI during later `git fetch `\n commands.\n \n 5. Allow clients to discover bundle URIs during `git fetch` and configure\n- a bundle URI for later fetches if `bundle.flag=forFetch`.\n+ a bundle URI for later fetches if `bundle.heuristic` is set.\n \n 6. Implement the \"inspect headers\" heuristic to reduce data downloads when\n the `bundle..creationToken` heuristic is not available.\n-- \ngitgitgadget\n\n"},{"id":"471180","messageId":"314c60f2ae4cf54a0d99cc4e6d8c4edd77fd6b22.1675171760.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v3.git.1675171759.gitgitgadget@gmail.com","subject":"[PATCH v3 09/11] fetch: fetch from an external bundle URI","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-31T13:29:17Z","receivedAt":"2023-01-31T13:30:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a user specifies a URI via 'git clone --bundle-uri', that URI may\nbe a bundle list that advertises a 'bundle.heuristic' value. In that\ncase, the Git client stores a 'fetch.bundleURI' config value storing\nthat URI.\n\nTeach 'git fetch' to check for this config value and download bundles\nfrom that URI before fetching from the Git remote(s). Likely, the bundle\nprovider has configured a heuristic (such as \"creationToken\") that will\nallow the Git client to download only a portion of the bundles before\ncontinuing the fetch.\n\nSince this URI is completely independent of the remote server, we want\nto be sure that we connect to the bundle URI before creating a\nconnection to the Git remote. We do not want to hold a stateful\nconnection for too long if we can avoid it.\n\nTo test that this works correctly, extend the previous tests that set\n'fetch.bundleURI' to do follow-up fetches. The bundle list is updated\nincrementally at each phase to demonstrate that the heuristic avoids\ndownloading older bundles. This includes the middle fetch downloading\nthe objects in bundle-3.bundle from the Git remote, and therefore not\nneeding that bundle in the third fetch.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/fetch.c | 6 ++\n t/t5558-clone-bundle-uri.sh | 113 +++++++++++++++++++++++++++++++++++-\n 2 files changed, 118 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 7378cafeec9..0477c379369 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -29,6 +29,7 @@\n #include \"commit-graph.h\"\n #include \"shallow.h\"\n #include \"worktree.h\"\n+#include \"bundle-uri.h\"\n \n #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n \n@@ -2109,6 +2110,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,\n int cmd_fetch(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n+\tconst char *bundle_uri;\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n \tstruct remote *remote = NULL;\n \tint result = 0;\n@@ -2194,6 +2196,10 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (dry_run)\n \t\twrite_fetch_head = 0;\n \n+\tif (!git_config_get_string_tmp(\"fetch.bundleuri\", &bundle_uri) &&\n+\t fetch_bundle_uri(the_repository, bundle_uri, NULL))\n+\t\twarning(_(\"failed to fetch bundles from '%s'\"), bundle_uri);\n+\n \tif (all) {\n \t\tif (argc == 1)\n \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex b2d15e141ca..7deeb4b8ad1 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -440,7 +440,55 @@ test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n \tEOF\n \n \ttest_remote_https_urls actual &&\n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\n+\t# We now have only one bundle ref.\n+\tgit -C clone-token-http for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\tEOF\n+\ttest_cmp expect refs &&\n+\n+\t# Add remaining bundles, exercising the \"deepening\" strategy\n+\t# for downloading via the creationToken heurisitc.\n+\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace1.txt\" \\\n+\t\tgit -C clone-token-http fetch origin --no-tags \\\n+\t\trefs/heads/merge:refs/heads/merge &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-4.bundle\n+\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/bundle-2.bundle\n+\tEOF\n+\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# We now have all bundle refs.\n+\tgit -C clone-token-http for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/merge\n+\trefs/bundles/right\n+\tEOF\n+\ttest_cmp expect refs\n '\n \n test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n@@ -477,6 +525,69 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \tcat >expect <<-\\EOF &&\n \trefs/bundles/base\n \tEOF\n+\ttest_cmp expect refs &&\n+\n+\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\t# Fetch the objects for bundle-2 _and_ bundle-3.\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace1.txt\" \\\n+\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n+\t\trefs/heads/left:refs/heads/left \\\n+\t\trefs/heads/right:refs/heads/right &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-2.bundle\n+\tEOF\n+\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# received left from bundle-2\n+\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect refs &&\n+\n+\tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\tEOF\n+\n+\t# This fetch should skip bundle-3.bundle, since its objects are\n+\t# already local (we have the requisite commits for bundle-4.bundle).\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n+\t\trefs/heads/merge:refs/heads/merge &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-4.bundle\n+\tEOF\n+\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# received merge ref from bundle-4, but right is missing\n+\t# because we did not download bundle-3.\n+\tgit -C fetch-http-4 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/merge\n+\tEOF\n \ttest_cmp expect refs\n '\n \n-- \ngitgitgadget\n\n"},{"id":"471181","messageId":"c968b63febab41f1f665910a103fcec56fac0a59.1675171760.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v3.git.1675171759.gitgitgadget@gmail.com","subject":"[PATCH v3 11/11] bundle-uri: test missing bundles with heuristic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-31T13:29:19Z","receivedAt":"2023-01-31T13:30:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe creationToken heuristic uses a different mechanism for downloading\nbundles from the \"standard\" approach. Specifically: it uses a concrete\norder based on the creationToken values and attempts to download as few\nbundles as possible. It also modifies local config to store a value for\nfuture fetches to avoid downloading bundles, if possible.\n\nHowever, if any of the individual bundles has a failed download, then\nthe logic for the ordering comes into question. It is important to avoid\ninfinite loops, assigning invalid creation token values in config, but\nalso to be opportunistic as possible when downloading as many bundles as\nseem appropriate.\n\nThese tests were used to inform the implementation of\nfetch_bundles_by_token() in bundle-uri.c, but are being added\nindependently here to allow focusing on faulty downloads. There may be\nmore cases that could be added that result in modifications to\nfetch_bundles_by_token() as interesting data shapes reveal themselves in\nreal scenarios.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5558-clone-bundle-uri.sh | 400 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 400 insertions(+)\n\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 9c2b7934b9b..afd56926c53 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -618,6 +618,406 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'creationToken heuristic with failed downloads (clone)' '\n+\ttest_when_finished rm -rf download-* trace*.txt &&\n+\n+\t# Case 1: base bundle does not exist, nothing can unbundle\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = fake.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone-1.txt\" \\\n+\tgit clone --single-branch --branch=base \\\n+\t\t--bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\t\"$HTTPD_URL/smart/fetch.git\" download-1 &&\n+\n+\t# Bundle failure does not set these configs.\n+\ttest_must_fail git -C download-1 config fetch.bundleuri &&\n+\ttest_must_fail git -C download-1 config fetch.bundlecreationtoken &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-4.bundle\n+\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/bundle-2.bundle\n+\t$HTTPD_URL/fake.bundle\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# All bundles failed to unbundle\n+\tgit -C download-1 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\ttest_must_be_empty refs &&\n+\n+\t# Case 2: middle bundle does not exist, only two bundles can unbundle\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = fake.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone-2.txt\" \\\n+\tgit clone --single-branch --branch=base \\\n+\t\t--bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\t\"$HTTPD_URL/smart/fetch.git\" download-2 &&\n+\n+\t# Bundle failure does not set these configs.\n+\ttest_must_fail git -C download-2 config fetch.bundleuri &&\n+\ttest_must_fail git -C download-2 config fetch.bundlecreationtoken &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-4.bundle\n+\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/fake.bundle\n+\t$HTTPD_URL/bundle-1.bundle\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# bundle-1 and bundle-3 could unbundle, but bundle-4 could not\n+\tgit -C download-2 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/right\n+\tEOF\n+\ttest_cmp expect refs &&\n+\n+\t# Case 3: top bundle does not exist, rest unbundle fine.\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = fake.bundle\n+\t\tcreationToken = 4\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-clone-3.txt\" \\\n+\tgit clone --single-branch --branch=base \\\n+\t\t--bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\t\"$HTTPD_URL/smart/fetch.git\" download-3 &&\n+\n+\t# As long as we have continguous successful downloads,\n+\t# we _do_ set these configs.\n+\ttest_cmp_config -C download-3 \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n+\ttest_cmp_config -C download-3 3 fetch.bundlecreationtoken &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/fake.bundle\n+\t$HTTPD_URL/bundle-3.bundle\n+\t$HTTPD_URL/bundle-2.bundle\n+\t$HTTPD_URL/bundle-1.bundle\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# fake.bundle did not unbundle, but the others did.\n+\tgit -C download-3 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/right\n+\tEOF\n+\ttest_cmp expect refs\n+'\n+\n+# Expand the bundle list to include other interesting shapes, specifically\n+# interesting for use when fetching from a previous state.\n+#\n+# ---------------- bundle-7\n+# 7\n+# _/|\\_\n+# ---/--|--\\------ bundle-6\n+# 5 | 6\n+# --|---|---|----- bundle-4\n+# | 4 |\n+# | / \\ /\n+# --|-|---|/------ bundle-3 (the client will be caught up to this point.)\n+# \\ | 3\n+# ---\\|---|------- bundle-2\n+# 2 |\n+# ----|---|------- bundle-1\n+# \\ /\n+# 1\n+# |\n+# (previous commits)\n+test_expect_success 'expand incremental bundle list' '\n+\t(\n+\t\tcd clone-from &&\n+\t\tgit checkout -b lefter left &&\n+\t\ttest_commit 5 &&\n+\t\tgit checkout -b righter right &&\n+\t\ttest_commit 6 &&\n+\t\tgit checkout -b top lefter &&\n+\t\tgit merge -m \"7\" merge righter &&\n+\n+\t\tgit bundle create bundle-6.bundle lefter righter --not left right &&\n+\t\tgit bundle create bundle-7.bundle top --not lefter merge righter &&\n+\n+\t\tcp bundle-*.bundle \"$HTTPD_DOCUMENT_ROOT_PATH/\"\n+\t) &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/fetch.git\" fetch origin +refs/heads/*:refs/heads/*\n+'\n+\n+test_expect_success 'creationToken heuristic with failed downloads (fetch)' '\n+\ttest_when_finished rm -rf download-* trace*.txt &&\n+\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\tEOF\n+\n+\tgit clone --single-branch --branch=left \\\n+\t\t--bundle-uri=\"$HTTPD_URL/bundle-list\" \\\n+\t\t\"$HTTPD_URL/smart/fetch.git\" fetch-base &&\n+\ttest_cmp_config -C fetch-base \"$HTTPD_URL/bundle-list\" fetch.bundleURI &&\n+\ttest_cmp_config -C fetch-base 3 fetch.bundleCreationToken &&\n+\n+\t# Case 1: all bundles exist: successful unbundling of all bundles\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\n+\t[bundle \"bundle-6\"]\n+\t\turi = bundle-6.bundle\n+\t\tcreationToken = 6\n+\n+\t[bundle \"bundle-7\"]\n+\t\turi = bundle-7.bundle\n+\t\tcreationToken = 7\n+\tEOF\n+\n+\tcp -r fetch-base fetch-1 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-fetch-1.txt\" \\\n+\t\tgit -C fetch-1 fetch origin &&\n+\ttest_cmp_config -C fetch-1 7 fetch.bundlecreationtoken &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-7.bundle\n+\t$HTTPD_URL/bundle-6.bundle\n+\t$HTTPD_URL/bundle-4.bundle\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Check which bundles have unbundled by refs\n+\tgit -C fetch-1 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/lefter\n+\trefs/bundles/merge\n+\trefs/bundles/right\n+\trefs/bundles/righter\n+\trefs/bundles/top\n+\tEOF\n+\ttest_cmp expect refs &&\n+\n+\t# Case 2: middle bundle does not exist, only bundle-4 can unbundle\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\n+\t[bundle \"bundle-6\"]\n+\t\turi = fake.bundle\n+\t\tcreationToken = 6\n+\n+\t[bundle \"bundle-7\"]\n+\t\turi = bundle-7.bundle\n+\t\tcreationToken = 7\n+\tEOF\n+\n+\tcp -r fetch-base fetch-2 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-fetch-2.txt\" \\\n+\t\tgit -C fetch-2 fetch origin &&\n+\n+\t# Since bundle-7 fails to unbundle, do not update creation token.\n+\ttest_cmp_config -C fetch-2 3 fetch.bundlecreationtoken &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/bundle-7.bundle\n+\t$HTTPD_URL/fake.bundle\n+\t$HTTPD_URL/bundle-4.bundle\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Check which bundles have unbundled by refs\n+\tgit -C fetch-2 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/merge\n+\trefs/bundles/right\n+\tEOF\n+\ttest_cmp expect refs &&\n+\n+\t# Case 3: top bundle does not exist, rest unbundle fine.\n+\tcat >\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = bundle-2.bundle\n+\t\tcreationToken = 2\n+\n+\t[bundle \"bundle-3\"]\n+\t\turi = bundle-3.bundle\n+\t\tcreationToken = 3\n+\n+\t[bundle \"bundle-4\"]\n+\t\turi = bundle-4.bundle\n+\t\tcreationToken = 4\n+\n+\t[bundle \"bundle-6\"]\n+\t\turi = bundle-6.bundle\n+\t\tcreationToken = 6\n+\n+\t[bundle \"bundle-7\"]\n+\t\turi = fake.bundle\n+\t\tcreationToken = 7\n+\tEOF\n+\n+\tcp -r fetch-base fetch-3 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-fetch-3.txt\" \\\n+\t\tgit -C fetch-3 fetch origin &&\n+\n+\t# As long as we have continguous successful downloads,\n+\t# we _do_ set the maximum creation token.\n+\ttest_cmp_config -C fetch-3 6 fetch.bundlecreationtoken &&\n+\n+\t# NOTE: the fetch skips bundle-4 since bundle-6 successfully\n+\t# unbundles itself and bundle-7 failed to download.\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\t$HTTPD_URL/fake.bundle\n+\t$HTTPD_URL/bundle-6.bundle\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Check which bundles have unbundled by refs\n+\tgit -C fetch-3 for-each-ref --format=\"%(refname)\" \"refs/bundles/*\" >refs &&\n+\tcat >expect <<-EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\trefs/bundles/lefter\n+\trefs/bundles/right\n+\trefs/bundles/righter\n+\tEOF\n+\ttest_cmp expect refs\n+'\n+\n # Do not add tests here unless they use the HTTP server, as they will\n # not run unless the HTTP dependencies exist.\n \n-- \ngitgitgadget\n"},{"id":"471182","messageId":"4e0465efd19798e7a14d653440233b9b3319b6de.1675171760.git.gitgitgadget@gmail.com","threadId":"59040","inReplyTo":"pull.1454.v3.git.1675171759.gitgitgadget@gmail.com","subject":"[PATCH v3 10/11] bundle-uri: store fetch.bundleCreationToken","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-31T13:29:18Z","receivedAt":"2023-01-31T13:30:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a bundle list specifies the \"creationToken\" heuristic, the Git\nclient downloads the list and then starts downloading bundles in\ndescending creationToken order. This process stops as soon as all\ndownloaded bundles can be applied to the repository (because all\nrequired commits are present in the repository or in the downloaded\nbundles).\n\nWhen checking the same bundle list twice, this strategy requires\ndownloading the bundle with the maximum creationToken again, which is\nwasteful. The creationToken heuristic promises that the client will not\nhave a use for that bundle if its creationToken value is at most the\nprevious creationToken value.\n\nTo prevent these wasteful downloads, create a fetch.bundleCreationToken\nconfig setting that the Git client sets after downloading bundles. This\nvalue allows skipping that maximum bundle download when this config\nvalue is the same value (or larger).\n\nTo test that this works correctly, we can insert some \"duplicate\"\nfetches into existing tests and demonstrate that only the bundle list is\ndownloaded.\n\nThe previous logic for downloading bundles by creationToken worked even\nif the bundle list was empty, but now we have logic that depends on the\nfirst entry of the list. Terminate early in the (non-sensical) case of\nan empty bundle list.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/fetch.txt | 16 ++++++++++++\n bundle-uri.c | 48 ++++++++++++++++++++++++++++++++--\n t/t5558-clone-bundle-uri.sh | 29 +++++++++++++++++++-\n 3 files changed, 90 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt\nindex 244f44d460f..568f0f75b30 100644\n--- a/Documentation/config/fetch.txt\n+++ b/Documentation/config/fetch.txt\n@@ -104,3 +104,19 @@ fetch.bundleURI::\n \tlinkgit:git-clone[1]. `git clone --bundle-uri` will set the\n \t`fetch.bundleURI` value if the supplied bundle URI contains a bundle\n \tlist that is organized for incremental fetches.\n++\n+If you modify this value and your repository has a `fetch.bundleCreationToken`\n+value, then remove that `fetch.bundleCreationToken` value before fetching from\n+the new bundle URI.\n+\n+fetch.bundleCreationToken::\n+\tWhen using `fetch.bundleURI` to fetch incrementally from a bundle\n+\tlist that uses the \"creationToken\" heuristic, this config value\n+\tstores the maximum `creationToken` value of the downloaded bundles.\n+\tThis value is used to prevent downloading bundles in the future\n+\tif the advertised `creationToken` is not strictly larger than this\n+\tvalue.\n++\n+The creation token values are chosen by the provider serving the specific\n+bundle URI. If you modify the URI at `fetch.bundleURI`, then be sure to\n+remove the value for the `fetch.bundleCreationToken` value before fetching.\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 7a1b6d94bf5..d6f7df7350f 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -481,6 +481,8 @@ static int fetch_bundles_by_token(struct repository *r,\n {\n \tint cur;\n \tint move_direction = 0;\n+\tconst char *creationTokenStr;\n+\tuint64_t maxCreationToken = 0, newMaxCreationToken = 0;\n \tstruct bundle_list_context ctx = {\n \t\t.r = r,\n \t\t.list = list,\n@@ -494,8 +496,27 @@ static int fetch_bundles_by_token(struct repository *r,\n \n \tfor_all_bundles_in_list(list, append_bundle, &bundles);\n \n+\tif (!bundles.nr) {\n+\t\tfree(bundles.items);\n+\t\treturn 0;\n+\t}\n+\n \tQSORT(bundles.items, bundles.nr, compare_creation_token_decreasing);\n \n+\t/*\n+\t * If fetch.bundleCreationToken exists, parses to a uint64t, and\n+\t * is not strictly smaller than the maximum creation token in the\n+\t * bundle list, then do not download any bundles.\n+\t */\n+\tif (!repo_config_get_value(r,\n+\t\t\t\t \"fetch.bundlecreationtoken\",\n+\t\t\t\t &creationTokenStr) &&\n+\t sscanf(creationTokenStr, \"%\"PRIu64, &maxCreationToken) == 1 &&\n+\t bundles.items[0]->creationToken <= maxCreationToken) {\n+\t\tfree(bundles.items);\n+\t\treturn 0;\n+\t}\n+\n \t/*\n \t * Attempt to download and unbundle the minimum number of bundles by\n \t * creationToken in decreasing order. If we fail to unbundle (after\n@@ -516,6 +537,16 @@ static int fetch_bundles_by_token(struct repository *r,\n \tcur = 0;\n \twhile (cur >= 0 && cur < bundles.nr) {\n \t\tstruct remote_bundle_info *bundle = bundles.items[cur];\n+\n+\t\t/*\n+\t\t * If we need to dig into bundles below the previous\n+\t\t * creation token value, then likely we are in an erroneous\n+\t\t * state due to missing or invalid bundles. Halt the process\n+\t\t * instead of continuing to download extra data.\n+\t\t */\n+\t\tif (bundle->creationToken <= maxCreationToken)\n+\t\t\tbreak;\n+\n \t\tif (!bundle->file) {\n \t\t\t/*\n \t\t\t * Not downloaded yet. Try downloading.\n@@ -555,6 +586,9 @@ static int fetch_bundles_by_token(struct repository *r,\n \t\t\t\t */\n \t\t\t\tmove_direction = -1;\n \t\t\t\tbundle->unbundled = 1;\n+\n+\t\t\t\tif (bundle->creationToken > newMaxCreationToken)\n+\t\t\t\t\tnewMaxCreationToken = bundle->creationToken;\n \t\t\t}\n \t\t}\n \n@@ -569,14 +603,24 @@ move:\n \t\tcur += move_direction;\n \t}\n \n-\tfree(bundles.items);\n-\n \t/*\n \t * We succeed if the loop terminates because 'cur' drops below\n \t * zero. The other case is that we terminate because 'cur'\n \t * reaches the end of the list, so we have a failure no matter\n \t * which bundles we apply from the list.\n \t */\n+\tif (cur < 0) {\n+\t\tstruct strbuf value = STRBUF_INIT;\n+\t\tstrbuf_addf(&value, \"%\"PRIu64\"\", newMaxCreationToken);\n+\t\tif (repo_config_set_multivar_gently(ctx.r,\n+\t\t\t\t\t\t \"fetch.bundleCreationToken\",\n+\t\t\t\t\t\t value.buf, NULL, 0))\n+\t\t\twarning(_(\"failed to store maximum creation token\"));\n+\n+\t\tstrbuf_release(&value);\n+\t}\n+\n+\tfree(bundles.items);\n \treturn cur >= 0;\n }\n \ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 7deeb4b8ad1..9c2b7934b9b 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -433,6 +433,7 @@ test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n \t\t\"$HTTPD_URL/smart/fetch.git\" clone-token-http &&\n \n \ttest_cmp_config -C clone-token-http \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n+\ttest_cmp_config -C clone-token-http 1 fetch.bundlecreationtoken &&\n \n \tcat >expect <<-EOF &&\n \t$HTTPD_URL/bundle-list\n@@ -468,6 +469,7 @@ test_expect_success 'clone incomplete bundle list (http, creationToken)' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace1.txt\" \\\n \t\tgit -C clone-token-http fetch origin --no-tags \\\n \t\trefs/heads/merge:refs/heads/merge &&\n+\ttest_cmp_config -C clone-token-http 4 fetch.bundlecreationtoken &&\n \n \tcat >expect <<-EOF &&\n \t$HTTPD_URL/bundle-list\n@@ -511,6 +513,7 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \t\t\"$HTTPD_URL/smart/fetch.git\" fetch-http-4 &&\n \n \ttest_cmp_config -C fetch-http-4 \"$HTTPD_URL/bundle-list\" fetch.bundleuri &&\n+\ttest_cmp_config -C fetch-http-4 1 fetch.bundlecreationtoken &&\n \n \tcat >expect <<-EOF &&\n \t$HTTPD_URL/bundle-list\n@@ -538,6 +541,7 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \t\tgit -C fetch-http-4 fetch origin --no-tags \\\n \t\trefs/heads/left:refs/heads/left \\\n \t\trefs/heads/right:refs/heads/right &&\n+\ttest_cmp_config -C fetch-http-4 2 fetch.bundlecreationtoken &&\n \n \tcat >expect <<-EOF &&\n \t$HTTPD_URL/bundle-list\n@@ -555,6 +559,18 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \tEOF\n \ttest_cmp expect refs &&\n \n+\t# No-op fetch\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace1b.txt\" \\\n+\t\tgit -C fetch-http-4 fetch origin --no-tags \\\n+\t\trefs/heads/left:refs/heads/left \\\n+\t\trefs/heads/right:refs/heads/right &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual &&\n+\n \tcat >>\"$HTTPD_DOCUMENT_ROOT_PATH/bundle-list\" <<-EOF &&\n \t[bundle \"bundle-3\"]\n \t\turi = bundle-3.bundle\n@@ -570,6 +586,7 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n \t\tgit -C fetch-http-4 fetch origin --no-tags \\\n \t\trefs/heads/merge:refs/heads/merge &&\n+\ttest_cmp_config -C fetch-http-4 4 fetch.bundlecreationtoken &&\n \n \tcat >expect <<-EOF &&\n \t$HTTPD_URL/bundle-list\n@@ -588,7 +605,17 @@ test_expect_success 'http clone with bundle.heuristic creates fetch.bundleURI' '\n \trefs/bundles/left\n \trefs/bundles/merge\n \tEOF\n-\ttest_cmp expect refs\n+\ttest_cmp expect refs &&\n+\n+\t# No-op fetch\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2b.txt\" \\\n+\t\tgit -C fetch-http-4 fetch origin &&\n+\n+\tcat >expect <<-EOF &&\n+\t$HTTPD_URL/bundle-list\n+\tEOF\n+\ttest_remote_https_urls actual &&\n+\ttest_cmp expect actual\n '\n \n # Do not add tests here unless they use the HTTP server, as they will\n-- \ngitgitgadget\n\n"},{"id":"471193","messageId":"xmqqr0vay5cz.fsf@gitster.g","threadId":"59040","inReplyTo":"20c29d37f9c1ba1367145331d25dd27f966312cd.1675171759.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 02/11] bundle: verify using check_connected()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-31T17:35:24Z","receivedAt":"2023-01-31T17:36:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> Thus, the code in verify_bundle() has previously had the additional\n> check that all prerequisite commits are reachable from repository\n> references. This is done via a revision walk from all references,\n> stopping only if all prerequisite commits are discovered or all commits\n> are walked. This uses a custom walk to verify_bundle().\n>\n> This check is more strict than what Git applies to fetched pack-files.\n> In the fetch case, Git guarantees that the new references are closed\n> under reachability by walking from the new references until walking\n> commits that are reachable from repository refs. This is done through\n> the well-used check_connected() method.\n>\n> To better align with the restrictions required by 'git fetch',\n> reimplement this check in verify_bundle() to use check_connected(). This\n> also simplifies the code significantly.\n\nAs I often say, breaking repository faster is not the kind of\nperformance gain we want to have in Git, and I am in favor of this\niteration compared to the earlier version that mostly punted on\nensuring the correctness (rather, it relied on assumption that \"most\nof the time this should be OK\").\n\n> bundle.c | 75 ++++++++++++++++--------------------------\n> t/t6020-bundle-misc.sh | 8 ++---\n> 2 files changed, 33 insertions(+), 50 deletions(-)\n\nThe diffstat is very pleasing to see.\n\nLet me read the postimage along aloud (preimage omitted).\n\n> diff --git a/bundle.c b/bundle.c\n> index 4ef7256aa11..76c3a904898 100644\n> --- a/bundle.c\n> +++ b/bundle.c\n> @@ -187,6 +188,21 @@ static int list_refs(struct string_list *r, int argc, const char **argv)\n> /* Remember to update object flag allocation in object.h */\n> #define PREREQ_MARK (1u<<16)\n> \n> +struct string_list_iterator {\n> +\tstruct string_list *list;\n> +\tsize_t cur;\n> +};\n> +\n> +static const struct object_id *iterate_ref_map(void *cb_data)\n> +{\n> +\tstruct string_list_iterator *iter = cb_data;\n> +\n> +\tif (iter->cur >= iter->list->nr)\n> +\t\treturn NULL;\n> +\n> +\treturn iter->list->items[iter->cur++].util;\n> +}\n\nThis is to let check_connected() collect all the prerequisite object\nnames. OK.\n\n> int verify_bundle(struct repository *r,\n> \t\t struct bundle_header *header,\n> \t\t enum verify_bundle_flags flags)\n> {\n> \t/*\n> \t * Do fast check, then if any prereqs are missing then go line by line\n> \t * to be verbose about the errors\n> \t */\n> \tstruct string_list *p = &header->prerequisites;\n> +\tint i, ret = 0;\n> \tconst char *message = _(\"Repository lacks these prerequisite commits:\");\n> +\tstruct string_list_iterator iter = {\n> +\t\t.list = p,\n> +\t};\n> +\tstruct check_connected_options opts = {\n> +\t\t.quiet = 1,\n> +\t};\n> \n> \tif (!r || !r->objects || !r->objects->odb)\n> \t\treturn error(_(\"need a repository to verify a bundle\"));\n> \n> \tfor (i = 0; i < p->nr; i++) {\n> \t\tstruct string_list_item *e = p->items + i;\n> \t\tconst char *name = e->string;\n> \t\tstruct object_id *oid = e->util;\n> \t\tstruct object *o = parse_object(r, oid);\n> +\t\tif (o)\n> \t\t\tcontinue;\n> \t\tret++;\n> \t\tif (flags & VERIFY_BUNDLE_QUIET)\n> \t\t\tcontinue;\n> \t\tif (ret == 1)\n> \t\t\terror(\"%s\", message);\n> \t\terror(\"%s %s\", oid_to_hex(oid), name);\n> \t}\n> +\tif (ret)\n> \t\tgoto cleanup;\n\nThe \"quick fail\" logic as before. Looking sensible.\n\n> \n> +\tif ((ret = check_connected(iterate_ref_map, &iter, &opts)))\n> +\t\terror(_(\"some prerequisite commits exist in the object store, \"\n> +\t\t\t\"but are not connected to the repository's history\"));\n\nAnd then we let check_connected() to ensure that traversing from\nthese prerequisite objects down to the DAG formed by existing refs\nwill not die from missing objects. Makes sense.\n\n> +\t/* TODO: preserve this verbose language. */\n\nI am lost -- aren't we preserving the BUNDLE_VERBOSE code below?\n\n> \tif (flags & VERIFY_BUNDLE_VERBOSE) {\n\n\n\n> diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\n> index 38dbbf89155..7d40994991e 100755\n> --- a/t/t6020-bundle-misc.sh\n> +++ b/t/t6020-bundle-misc.sh\n> @@ -595,14 +595,14 @@ test_expect_success 'verify catches unreachable, broken prerequisites' '\n> \t\t# Verify should fail\n> \t\ttest_must_fail git bundle verify \\\n> \t\t\t../clone-from/tip.bundle 2>err &&\n> -\t\tgrep \"Could not read $BAD_OID\" err &&\n> -\t\tgrep \"Failed to traverse parents of commit $TIP_OID\" err &&\n> +\t\tgrep \"some prerequisite commits .* are not connected\" err &&\n> +\t\ttest_line_count = 1 err &&\n> \n> \t\t# Unbundling should fail\n> \t\ttest_must_fail git bundle unbundle \\\n> \t\t\t../clone-from/tip.bundle 2>err &&\n> -\t\tgrep \"Could not read $BAD_OID\" err &&\n> -\t\tgrep \"Failed to traverse parents of commit $TIP_OID\" err\n> +\t\tgrep \"some prerequisite commits .* are not connected\" err &&\n> +\t\ttest_line_count = 1 err\n> \t)\n> '\n\nEspecially with the new test added in the previous step, we know we\nare not trading correctness off. Excellent.\n\nI wonder how much the performance hit does this version incur over\nthe \"not safe at all\" version and over the \"use custom and\nstricter-than-needed\" version, by the way?\n\nThanks.\n"},{"id":"471197","messageId":"73c1d863-036b-81bb-50d3-2a084c2c2fb5@github.com","threadId":"59040","inReplyTo":"xmqqr0vay5cz.fsf@gitster.g","subject":"Re: [PATCH v3 02/11] bundle: verify using check_connected()","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-31T19:31:08Z","receivedAt":"2023-01-31T19:31:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/31/2023 12:35 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n\n>> +\t/* TODO: preserve this verbose language. */\n> \n> I am lost -- aren't we preserving the BUNDLE_VERBOSE code below?\n\nSorry, I put this in so I wouldn't forget to test that the\nverbose message sticks, but I did forget to remove it.\n\n>> diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\n>> index 38dbbf89155..7d40994991e 100755\n>> --- a/t/t6020-bundle-misc.sh\n>> +++ b/t/t6020-bundle-misc.sh\n>> @@ -595,14 +595,14 @@ test_expect_success 'verify catches unreachable, broken prerequisites' '\n>> \t\t# Verify should fail\n>> \t\ttest_must_fail git bundle verify \\\n>> \t\t\t../clone-from/tip.bundle 2>err &&\n>> -\t\tgrep \"Could not read $BAD_OID\" err &&\n>> -\t\tgrep \"Failed to traverse parents of commit $TIP_OID\" err &&\n>> +\t\tgrep \"some prerequisite commits .* are not connected\" err &&\n>> +\t\ttest_line_count = 1 err &&\n>> \n>> \t\t# Unbundling should fail\n>> \t\ttest_must_fail git bundle unbundle \\\n>> \t\t\t../clone-from/tip.bundle 2>err &&\n>> -\t\tgrep \"Could not read $BAD_OID\" err &&\n>> -\t\tgrep \"Failed to traverse parents of commit $TIP_OID\" err\n>> +\t\tgrep \"some prerequisite commits .* are not connected\" err &&\n>> +\t\ttest_line_count = 1 err\n>> \t)\n>> '\n> \n> Especially with the new test added in the previous step, we know we\n> are not trading correctness off. Excellent.\n> \n> I wonder how much the performance hit does this version incur over\n> the \"not safe at all\" version and over the \"use custom and\n> stricter-than-needed\" version, by the way?\n\nI was able to simulate this in an extreme situation by taking a clone\nof the normal Git fork, creating a ref at every first parent, then\ncreating a bundle of the difference between git/git and gitster/git.\nFinally, I compared the performance of 'git bundle verify' on Git\ncompiled before this change and after this change:\n\nBenchmark 1: old\n Time (mean ± σ): 324.7 ms ± 7.5 ms [User: 228.0 ms, System: 95.7 ms]\n Range (min … max): 315.3 ms … 338.0 ms 10 runs\n \nBenchmark 2: new\n Time (mean ± σ): 331.2 ms ± 20.2 ms [User: 230.6 ms, System: 99.7 ms]\n Range (min … max): 302.8 ms … 360.2 ms 10 runs\n \nSummary\n 'old' ran\n 1.02 ± 0.07 times faster than 'new'\n\nSo, there is a performance difference in the two situations, but it\nis very slight, in what I can detect.\n\nOf course, there is the case where the behavior differs because of\nthe more permissible behavior, but I expect the old algorithm to be\nslower than the new case, because check_connected() can terminate\nwith success (due to seeing all the prerequisite commits) faster\nthan the old walk can terminate with failure (due to walking all\nreachable commits).\n\nThanks,\n-Stolee\n"},{"id":"471198","messageId":"xmqqilgmxzrf.fsf@gitster.g","threadId":"59040","inReplyTo":"73c1d863-036b-81bb-50d3-2a084c2c2fb5@github.com","subject":"Re: [PATCH v3 02/11] bundle: verify using check_connected()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-31T19:36:20Z","receivedAt":"2023-01-31T19:36:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> ... but I expect the old algorithm to be\n> slower than the new case, because check_connected() can terminate\n> with success (due to seeing all the prerequisite commits) faster\n> than the old walk can terminate with failure (due to walking all\n> reachable commits).\n\nThat is good to hear. As long as we are not making it worse than\nbefore, I am happy as a clam ;-)\n\nThanks.\n"},{"id":"471206","messageId":"xmqq4js6xuu2.fsf@gitster.g","threadId":"59040","inReplyTo":"ff629bc119b466dcd827f758b3d892fefd6a9703.1675171760.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 05/11] bundle-uri: parse bundle..creationToken values","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-31T21:22:45Z","receivedAt":"2023-01-31T21:22:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +\tif (info->creationToken)\n> +\t\tfprintf(fp, \"\\tcreationToken = %\"PRIu64\"\\n\", info->creationToken);\n\nSo, a bundle info with .creationToken set to 0 signals that the\ntoken is missing for that bundle. REMOTE_BUNDLE_INFO_INIT clears\nall members, so unless we find a concrete value and assign to the\nmember, we'll have 0 in the member (and never a random value). \n\nVery good.\n\nWe used to avoid camelCased variable names and member names and\ninstead prefer snake_case; I wonder if it is still the case.\nLooking at bundle.h it still seems to be the case, and we may want\nto match that.\n\n> @@ -203,6 +206,13 @@ static int bundle_list_update(const char *key, const char *value,\n> \t\treturn 0;\n> \t}\n> \n> +\tif (!strcmp(subkey, \"creationtoken\")) {\n> +\t\tif (sscanf(value, \"%\"PRIu64, &bundle->creationToken) != 1)\n> +\t\t\twarning(_(\"could not parse bundle list key %s with value '%s'\"),\n> +\t\t\t\t\"creationToken\", value);\n> +\t\treturn 0;\n> +\t}\n\nShould this be a warning, or a hard error? I _think_ it depends on\nhow much we trust creationToken supplied by the bundle providers for\ncorrectness. If we only use the values as hint to gain performance,\nand downloading the bundles in a wrong order (due to bogus\ncreationToken values assigned to them) does not lead to correctness\nerror, then warning is fine.\n\nLooking good.\n"},{"id":"471207","messageId":"xmqqzg9ywehp.fsf@gitster.g","threadId":"59040","inReplyTo":"pull.1454.v3.git.1675171759.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/11] Bundle URIs V: creationToken heuristic for incremental fetches","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-31T22:01:06Z","receivedAt":"2023-01-31T22:01:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> This fifth part to the bundle URIs feature follows part IV (advertising via\n> protocol v2) which recently merged to 'master', so this series is based on\n> 'master'.\n\nThanks for a pleasant read. Will queue.\n"}]}