{"thread":{"id":"58731","subject":"[PATCH 0/9] Bundle URIs IV: advertise over protocol v2","startedAt":"2022-11-01T01:07:41Z","lastAt":"2023-01-06T14:13:22Z","messageCount":87,"participants":["Derrick Stolee via GitGitGadget","Ævar Arnfjörð Bjarmason via GitGitGadget","Phillip Wood","Ævar Arnfjörð Bjarmason","SZEDER Gábor","Victoria Dye","Derrick Stolee","Jeff King","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"466147","messageId":"pull.1400.git.1667264854.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":null,"subject":"[PATCH 0/9] Bundle URIs IV: advertise over protocol v2","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-01T01:07:25Z","receivedAt":"2022-11-01T01:07:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is based on the recent master batch that included ds/bundle-uri-....\n\nNow that git clone --bundle-uri can download a bundle list from a plaintex\nfile in config format, we can use the same set of key-value pairs to\nadvertise a bundle list over protocol v2. At the end of this series:\n\n 1. A server can advertise bundles when uploadPack.advertiseBundleURIs is\n enabled. The bundle list comes from the server's local config,\n specifically the bundle.* namespace.\n 2. A client can notice a server's bundle-uri advertisement and request the\n bundle list if transfer.bundleURI is enabled. The bundles are downloaded\n as if the list was advertised from the --bundle-uri option.\n\nMany patches in this series were adapted from Ævar's v2 RFC [1]. He is\nretained as author and I added myself as co-author only if the modifications\nwere significant.\n\n[1]\nhttps://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/\n\n * Patches 1-5 are mostly taken from [1], again with mostly minor updates.\n The one major difference is the packet line format being a single\n key=value format instead of a sequence of pairs. This also means that\n Patch 4 is entirely new since it feeds these pairs directly from the\n server's config.\n\n * Patches 6-9 finish off the ability for the client to notice the\n capability, request the values, and download bundles before continuing\n with the rest of the download.\n\nOne thing that is not handled here but could be handled in a future change\nis to disconnect from the origin Git server while downloading the bundle\nURIs, then reconnecting afterwards. This does not make any difference for\nHTTPS, but SSH may benefit from the reduced connection time. The git clone\n--bundle-uri option did not suffer from this because the bundles are\ndownloaded before the server connection begins.\n\nAfter this series, there is one more before the original scope of the plan\nis complete: using creation tokens as a heuristic. See [2] for the RFC\nversion of those patches.\n\n[2] https://github.com/derrickstolee/git/pull/22\n\nThanks,\n\n * Stolee\n\nDerrick Stolee (5):\n bundle-uri: serve bundle.* keys from config\n strbuf: reintroduce strbuf_parent_directory()\n bundle-uri: allow relative URLs in bundle lists\n bundle-uri: download bundles from an advertised list\n clone: unbundle the advertised bundles\n\nÆvar Arnfjörð Bjarmason (4):\n protocol v2: add server-side \"bundle-uri\" skeleton\n bundle-uri client: add minimal NOOP client\n bundle-uri client: add helper for testing server\n bundle-uri client: add boolean transfer.bundleURI setting\n\n Documentation/config/transfer.txt | 6 +\n Documentation/gitprotocol-v2.txt | 193 +++++++++++++++++++++\n builtin/clone.c | 23 +++\n bundle-uri.c | 91 +++++++++-\n bundle-uri.h | 27 +++\n connect.c | 47 +++++\n remote.h | 5 +\n serve.c | 6 +\n strbuf.c | 9 +\n strbuf.h | 7 +\n t/helper/test-bundle-uri.c | 48 ++++++\n t/lib-t5730-protocol-v2-bundle-uri.sh | 229 +++++++++++++++++++++++++\n t/t5601-clone.sh | 59 +++++++\n t/t5701-git-serve.sh | 40 ++++-\n t/t5730-protocol-v2-bundle-uri-file.sh | 36 ++++\n t/t5731-protocol-v2-bundle-uri-git.sh | 17 ++\n t/t5732-protocol-v2-bundle-uri-http.sh | 17 ++\n t/t5750-bundle-uri-parse.sh | 54 ++++++\n transport-helper.c | 13 ++\n transport-internal.h | 7 +\n transport.c | 87 ++++++++++\n transport.h | 23 +++\n 22 files changed, 1042 insertions(+), 2 deletions(-)\n create mode 100644 t/lib-t5730-protocol-v2-bundle-uri.sh\n create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh\n create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh\n create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh\n\n\nbase-commit: c03801e19cb8ab36e9c0d17ff3d5e0c3b0f24193\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1400%2Fderrickstolee%2Fbundle-redo%2Fadvertise-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1400/derrickstolee/bundle-redo/advertise-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1400\n-- \ngitgitgadget\n"},{"id":"466148","messageId":"a02eee983185815d94ba1124b43eae43280aa963.1667264854.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.git.1667264854.gitgitgadget@gmail.com","subject":"[PATCH 1/9] protocol v2: add server-side \"bundle-uri\" skeleton","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-01T01:07:26Z","receivedAt":"2022-11-01T01:07:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nAdd a skeleton server-side implementation of a new \"bundle-uri\" command\nto protocol v2. This will allow conforming clients to optionally seed\ntheir initial clones or incremental fetches from URLs containing\n\"*.bundle\" files created with \"git bundle create\".\n\nThis change only performs the basic boilerplate of advertising a new\nprotocol v2 capability. The new 'bundle-uri' capability allows a client\nto request a list of bundles. Right now, the server only returns a flush\npacket, which corresponds to an empty advertisement. The bundle.* config\nnamespace describes which key-value pairs will be communicated across\nthis interface in future updates.\n\nThe critical bit right now is that the new boolean\nuploadPack.adverstiseBundleURIs config value signals whether or not this\ncapability should be advertised at all.\n\nAn earlier version of this patch [1] used a different transfer format\nthan the \"key=value\" pairs in the current implementation. The change was\nmade to unify the protocol v2 verb with the bundle lists provided by\nindependent bundle servers. Further, the standard allows for the server\nto advertise a URI that contains a bundle list. This allows users\nautomatically discovering bundle providers that are loosely associated\nwith the origin server, but without the origin server knowing exactly\nwhich bundles are currently available.\n\n[1] https://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/\n\nThe very-deep headings needed to be modified to stop at level 4 due to\ndocumentation build issues. These were not recognized in earlier builds\nsince the file was previously in the Documentation/technical/ directory\nand was built in a different way. With its current location, the\nheavily-nested details were causing build issues and they are now\nreplaced with a bulletted list of details.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Documentation/gitprotocol-v2.txt | 193 +++++++++++++++++++++++++++++++\n bundle-uri.c | 36 ++++++\n bundle-uri.h | 7 ++\n serve.c | 6 +\n t/t5701-git-serve.sh | 40 ++++++-\n 5 files changed, 281 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\nindex 59bf41cefb9..57642b4a415 100644\n--- a/Documentation/gitprotocol-v2.txt\n+++ b/Documentation/gitprotocol-v2.txt\n@@ -578,6 +578,199 @@ and associated requested information, each separated by a single space.\n \n \tobj-info = obj-id SP obj-size\n \n+bundle-uri\n+~~~~~~~~~~\n+\n+If the 'bundle-uri' capability is advertised, the server supports the\n+`bundle-uri' command.\n+\n+The capability is currently advertised with no value (i.e. not\n+\"bundle-uri=somevalue\"), a value may be added in the future for\n+supporting command-wide extensions. Clients MUST ignore any unknown\n+capability values and proceed with the 'bundle-uri` dialog they\n+support.\n+\n+The 'bundle-uri' command is intended to be issued before `fetch` to\n+get URIs to bundle files (see linkgit:git-bundle[1]) to \"seed\" and\n+inform the subsequent `fetch` command.\n+\n+The client CAN issue `bundle-uri` before or after any other valid\n+command. To be useful to clients it's expected that it'll be issued\n+after an `ls-refs` and before `fetch`, but CAN be issued at any time\n+in the dialog.\n+\n+DISCUSSION of bundle-uri\n+^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+The intent of the feature is optimize for server resource consumption\n+in the common case by changing the common case of fetching a very\n+large PACK during linkgit:git-clone[1] into a smaller incremental\n+fetch.\n+\n+It also allows servers to achieve better caching in combination with\n+an `uploadpack.packObjectsHook` (see linkgit:git-config[1]).\n+\n+By having new clones or fetches be a more predictable and common\n+negotiation against the tips of recently produces *.bundle file(s).\n+Servers might even pre-generate the results of such negotiations for\n+the `uploadpack.packObjectsHook` as new pushes come in.\n+\n+One way that servers could take advantage of these bundles is that the\n+server would anticipate that fresh clones will download a known bundle,\n+followed by catching up to the current state of the repository using ref\n+tips found in that bundle (or bundles).\n+\n+PROTOCOL for bundle-uri\n+^^^^^^^^^^^^^^^^^^^^^^^\n+\n+A `bundle-uri` request takes no arguments, and as noted above does not\n+currently advertise a capability value. Both may be added in the\n+future.\n+\n+When the client issues a `command=bundle-uri` the response is a list of\n+key-value pairs provided as packet lines with value `=`. The\n+meaning of these key-value pairs are provided by the config keys in the\n+`bundle.*` namespace (see linkgit:git-config[1]).\n+\n+Clients are still expected to fully parse the line according to the\n+above format, lines that do not conform to the format SHOULD be\n+discarded. The user MAY be warned in such a case.\n+\n+bundle-uri CLIENT AND SERVER EXPECTATIONS\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+URI CONTENTS::\n+The advertised URIs MUST be in one of two possible formats.\n++\n+The first possible format is a bundle file that `git bundle verify`\n+would accept. I.e. they MUST contain one or more reference tips for\n+use by the client, MUST indicate prerequisites (in any) with standard\n+\"-\" prefixes, and MUST indicate their \"object-format\", if\n+applicable. Create \"*.bundle\" files with `git bundle create`.\n++\n+The second possible format is a plaintext file that `git config --list`\n+would accept (with the `--file` option). The key-value pairs in this list\n+are in the `bundle.*` namespace (see linkgit:git-config[1]).\n+\n+bundle-uri CLIENT ERROR RECOVERY::\n+A client MUST above all gracefully degrade on errors, whether that\n+error is because of bad missing/data in the bundle URI(s), because\n+that client is too dumb to e.g. understand and fully parse out bundle\n+headers and their prerequisite relationships, or something else.\n++\n+Server operators should feel confident in turning on \"bundle-uri\" and\n+not worry if e.g. their CDN goes down that clones or fetches will run\n+into hard failures. Even if the server bundle bundle(s) are\n+incomplete, or bad in some way the client should still end up with a\n+functioning repository, just as if it had chosen not to use this\n+protocol extension.\n++\n+All subsequent discussion on client and server interaction MUST keep\n+this in mind.\n+\n+bundle-uri SERVER TO CLIENT::\n+The ordering of the returned bundle uris is not significant. Clients\n+MUST parse their headers to discover their contained OIDS and\n+prerequisites. A client MUST consider the content of the bundle(s)\n+themselves and their header as the ultimate source of truth.\n++\n+A server MAY even return bundle(s) that don't have any direct\n+relationship to the repository being cloned (either through accident,\n+or intentional \"clever\" configuration), and expect a client to sort\n+out what data they'd like from the bundle(s), if any.\n+\n+bundle-uri CLIENT TO SERVER::\n+The client SHOULD provide reference tips found in the bundle header(s)\n+as 'have' lines in any subsequent `fetch` request. A client MAY also\n+ignore the bundle(s) entirely if doing so is deemed worse for some\n+reason, e.g. if the bundles can't be downloaded, it doesn't like the\n+tips it finds etc.\n+\n+WHEN ADVERTISED BUNDLE(S) REQUIRE NO FURTHER NEGOTIATION::\n+If after issuing `bundle-uri` and `ls-refs`, and getting the header(s)\n+of the bundle(s) the client finds that the ref tips it wants can be\n+retrieved entirety from advertised bundle(s), it MAY disconnect. The\n+results of such a 'clone' or 'fetch' should be indistinguishable from\n+the state attained without using bundle-uri.\n+\n+EARLY CLIENT DISCONNECTIONS AND ERROR RECOVERY::\n+A client MAY perform an early disconnect while still downloading the\n+bundle(s) (having streamed and parsed their headers). In such a case\n+the client MUST gracefully recover from any errors related to\n+finishing the download and validation of the bundle(s).\n++\n+I.e. a client might need to re-connect and issue a 'fetch' command,\n+and possibly fall back to not making use of 'bundle-uri' at all.\n++\n+This \"MAY\" behavior is specified as such (and not a \"SHOULD\") on the\n+assumption that a server advertising bundle uris is more likely than\n+not to be serving up a relatively large repository, and to be pointing\n+to URIs that have a good chance of being in working order. A client\n+MAY e.g. look at the payload size of the bundles as a heuristic to see\n+if an early disconnect is worth it, should falling back on a full\n+\"fetch\" dialog be necessary.\n+\n+WHEN ADVERTISED BUNDLE(S) REQUIRE FURTHER NEGOTIATION::\n+A client SHOULD commence a negotiation of a PACK from the server via\n+the \"fetch\" command using the OID tips found in advertised bundles,\n+even if's still in the process of downloading those bundle(s).\n++\n+This allows for aggressive early disconnects from any interactive\n+server dialog. The client blindly trusts that the advertised OID tips\n+are relevant, and issues them as 'have' lines, it then requests any\n+tips it would like (usually from the \"ls-refs\" advertisement) via\n+'want' lines. The server will then compute a (hopefully small) PACK\n+with the expected difference between the tips from the bundle(s) and\n+the data requested.\n++\n+The only connection the client then needs to keep active is to the\n+concurrently downloading static bundle(s), when those and the\n+incremental PACK are retrieved they should be inflated and\n+validated. Any errors at this point should be gracefully recovered\n+from, see above.\n+\n+bundle-uri PROTOCOL FEATURES\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+As noted above the `=` definitions are documented by the\n+`bundle.*` config namespace.\n+\n+In particular, the `bundle.version` key specifies an integer value. The\n+only accepted value at the moment is `1`, but if the client sees an\n+unexpected value here then the client MUST ignore the bundle list.\n+\n+As long as `bundle.version` is understood, all other unknown keys MAY be\n+ignored by the client. The server will guarantee compatibility with older\n+clients, though newer clients may be better able to use the extra keys to\n+minimize downloads.\n+\n+Any backwards-incompatible addition of pre-URI key-value will be\n+guarded by a new `bundle.version` value or values in 'bundle-uri'\n+capability advertisement itself, and/or by new future `bundle-uri`\n+request arguments.\n+\n+Some example key-value pairs that are not currently implemented but could\n+be implemented in the future include:\n+\n+ * Add a \"hash=\" or \"size=\" advertise the expected hash or\n+ size of the bundle file.\n+\n+ * Advertise that one or more bundle files are the same (to e.g. have\n+ clients round-robin or otherwise choose one of N possible files).\n+\n+ * A \"oid=\" shortcut and \"prerequisite=\" shortcut. For\n+ expressing the common case of a bundle with one tip and no\n+ prerequisites, or one tip and one prerequisite.\n++\n+This would allow for optimizing the common case of servers who'd like\n+to provide one \"big bundle\" containing only their \"main\" branch,\n+and/or incremental updates thereof.\n++\n+A client receiving such a a response MAY assume that they can skip\n+retrieving the header from a bundle at the indicated URI, and thus\n+save themselves and the server(s) the request(s) needed to inspect the\n+headers of that bundle or bundles.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 79a914f961b..32022595964 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -563,6 +563,42 @@ cleanup:\n \treturn result;\n }\n \n+/**\n+ * API for serve.c.\n+ */\n+\n+int bundle_uri_advertise(struct repository *r, struct strbuf *value)\n+{\n+\tstatic int advertise_bundle_uri = -1;\n+\n+\tif (advertise_bundle_uri != -1)\n+\t\tgoto cached;\n+\n+\tadvertise_bundle_uri = 0;\n+\tgit_config_get_maybe_bool(\"uploadpack.advertisebundleuris\", &advertise_bundle_uri);\n+\n+cached:\n+\treturn advertise_bundle_uri;\n+}\n+\n+int bundle_uri_command(struct repository *r,\n+\t\t struct packet_reader *request)\n+{\n+\tstruct packet_writer writer;\n+\tpacket_writer_init(&writer, 1);\n+\n+\twhile (packet_reader_read(request) == PACKET_READ_NORMAL)\n+\t\tdie(_(\"bundle-uri: unexpected argument: '%s'\"), request->line);\n+\tif (request->status != PACKET_READ_FLUSH)\n+\t\tdie(_(\"bundle-uri: expected flush after arguments\"));\n+\n+\t/* TODO: Implement the communication */\n+\n+\tpacket_writer_flush(&writer);\n+\n+\treturn 0;\n+}\n+\n /**\n * General API for {transport,connect}.c etc.\n */\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 4dbc269823c..357111ecce8 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -4,6 +4,7 @@\n #include \"hashmap.h\"\n #include \"strbuf.h\"\n \n+struct packet_reader;\n struct repository;\n struct string_list;\n \n@@ -92,6 +93,12 @@ int bundle_uri_parse_config_format(const char *uri,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n+/**\n+ * API for serve.c.\n+ */\n+int bundle_uri_advertise(struct repository *r, struct strbuf *value);\n+int bundle_uri_command(struct repository *r, struct packet_reader *request);\n+\n /**\n * General API for {transport,connect}.c etc.\n */\ndiff --git a/serve.c b/serve.c\nindex 733347f602a..cbf4a143cfe 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -7,6 +7,7 @@\n #include \"protocol-caps.h\"\n #include \"serve.h\"\n #include \"upload-pack.h\"\n+#include \"bundle-uri.h\"\n \n static int advertise_sid = -1;\n static int client_hash_algo = GIT_HASH_SHA1;\n@@ -135,6 +136,11 @@ static struct protocol_capability capabilities[] = {\n \t\t.advertise = always_advertise,\n \t\t.command = cap_object_info,\n \t},\n+\t{\n+\t\t.name = \"bundle-uri\",\n+\t\t.advertise = bundle_uri_advertise,\n+\t\t.command = bundle_uri_command,\n+\t},\n };\n \n void protocol_v2_advertise_capabilities(void)\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex 1896f671cb3..f21e5e9d33d 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -13,7 +13,7 @@ test_expect_success 'test capability advertisement' '\n \twrong_algo sha1:sha256\n \twrong_algo sha256:sha1\n \tEOF\n-\tcat >expect <<-EOF &&\n+\tcat >expect.base <<-EOF &&\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n \tls-refs=unborn\n@@ -21,8 +21,11 @@ test_expect_success 'test capability advertisement' '\n \tserver-option\n \tobject-format=$(test_oid algo)\n \tobject-info\n+\tEOF\n+\tcat >expect.trailer <<-EOF &&\n \t0000\n \tEOF\n+\tcat expect.base expect.trailer >expect &&\n \n \tGIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \\\n \t\t--advertise-capabilities >out &&\n@@ -342,4 +345,39 @@ test_expect_success 'basics of object-info' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'test capability advertisement with uploadpack.advertiseBundleURIs' '\n+\ttest_config uploadpack.advertiseBundleURIs true &&\n+\n+\tcat >expect.extra <<-EOF &&\n+\tbundle-uri\n+\tEOF\n+\tcat expect.base \\\n+\t expect.extra \\\n+\t expect.trailer >expect &&\n+\n+\tGIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \\\n+\t\t--advertise-capabilities >out &&\n+\ttest-tool pkt-line unpack actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'basics of bundle-uri: dies if not enabled' '\n+\ttest-tool pkt-line pack >in <<-EOF &&\n+\tcommand=bundle-uri\n+\t0000\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid command '\"'\"'bundle-uri'\"'\"'\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\tERR serve: invalid command '\"'\"'bundle-uri'\"'\"'\n+\tEOF\n+\n+\ttest_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"466149","messageId":"64dd9bf41de0310bbed5e3623870819f1c876f83.1667264854.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.git.1667264854.gitgitgadget@gmail.com","subject":"[PATCH 2/9] bundle-uri client: add minimal NOOP client","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-01T01:07:27Z","receivedAt":"2022-11-01T01:07:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nSet up all the needed client parts of the \"bundle-uri\" protocol\nextension, without actually doing anything with the bundle URIs.\n\nI.e. if the server says it supports \"bundle-uri\" we'll issue a\ncommand=bundle-uri after command=ls-refs when we're cloning. We'll\nparse the returned output using the code already tested for in\nt5750-bundle-uri-parse.sh.\n\nWhat we aren't doing is actually acting on that data, i.e. downloading\nthe bundle(s) before we get to doing the command=fetch, and adjusting\nour negotiation dialog appropriately. I'll do that in subsequent\ncommits.\n\nThere's a question of what level of encapsulation we should use here,\nI've opted to use connect.h in clone.c, but we could also e.g. make\ntransport_get_remote_refs() invoke this, i.e. make it implicitly get\nthe bundle-uri list for later steps.\n\nThis approach means that we don't \"support\" this in \"git fetch\" for\nnow. I'm starting with the case of initial clones, although as noted\nin preceding commits to the protocol documentation nothing about this\napproach precludes getting bundles on incremental fetches.\n\nFor the t5732-protocol-v2-bundle-uri-http.sh it's not easy to set\nenvironment variables for git-upload-pack (it's started by Apache), so\nlet's skip the test under T5730_HTTP, and add unused T5730_{FILE,GIT}\nprerequisites for consistency and future use.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 7 ++\n bundle-uri.c | 4 +\n connect.c | 47 ++++++++\n remote.h | 5 +\n t/lib-t5730-protocol-v2-bundle-uri.sh | 148 +++++++++++++++++++++++++\n t/t5730-protocol-v2-bundle-uri-file.sh | 36 ++++++\n t/t5731-protocol-v2-bundle-uri-git.sh | 17 +++\n t/t5732-protocol-v2-bundle-uri-http.sh | 17 +++\n transport-helper.c | 13 +++\n transport-internal.h | 7 ++\n transport.c | 55 +++++++++\n transport.h | 19 ++++\n 12 files changed, 375 insertions(+)\n create mode 100644 t/lib-t5730-protocol-v2-bundle-uri.sh\n create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh\n create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh\n create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 547d6464b3c..edf98295af2 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -27,6 +27,7 @@\n #include \"iterator.h\"\n #include \"sigchain.h\"\n #include \"branch.h\"\n+#include \"connect.h\"\n #include \"remote.h\"\n #include \"run-command.h\"\n #include \"connected.h\"\n@@ -1266,6 +1267,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (refs)\n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n \n+\t/*\n+\t * Populate transport->got_remote_bundle_uri and\n+\t * transport->bundle_uri. We might get nothing.\n+\t */\n+\ttransport_get_remote_bundle_uri(transport);\n+\n \tif (mapped_refs) {\n \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n \ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 32022595964..2201b604b11 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -571,6 +571,10 @@ int bundle_uri_advertise(struct repository *r, struct strbuf *value)\n {\n \tstatic int advertise_bundle_uri = -1;\n \n+\tif (value &&\n+\t git_env_bool(\"GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE\", 0))\n+\t\tstrbuf_addstr(value, \"test-unknown-capability-value\");\n+\n \tif (advertise_bundle_uri != -1)\n \t\tgoto cached;\n \ndiff --git a/connect.c b/connect.c\nindex 5ea53deda23..d39effb7492 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -15,6 +15,7 @@\n #include \"version.h\"\n #include \"protocol.h\"\n #include \"alias.h\"\n+#include \"bundle-uri.h\"\n \n static char *server_capabilities_v1;\n static struct strvec server_capabilities_v2 = STRVEC_INIT;\n@@ -491,6 +492,52 @@ static void send_capabilities(int fd_out, struct packet_reader *reader)\n \t}\n }\n \n+int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\n+\t\t\t struct bundle_list *bundles, int stateless_rpc)\n+{\n+\tint line_nr = 1;\n+\n+\t/* Assert bundle-uri support */\n+\tserver_supports_v2(\"bundle-uri\", 1);\n+\n+\t/* (Re-)send capabilities */\n+\tsend_capabilities(fd_out, reader);\n+\n+\t/* Send command */\n+\tpacket_write_fmt(fd_out, \"command=bundle-uri\\n\");\n+\tpacket_delim(fd_out);\n+\n+\t/* Send options */\n+\tif (git_env_bool(\"GIT_TEST_PROTOCOL_BAD_BUNDLE_URI\", 0))\n+\t\tpacket_write_fmt(fd_out, \"test-bad-client\\n\");\n+\tpacket_flush(fd_out);\n+\n+\t/* Process response from server */\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\tconst char *line = reader->line;\n+\t\tline_nr++;\n+\n+\t\tif (!bundle_uri_parse_line(bundles, line))\n+\t\t\tcontinue;\n+\n+\t\treturn error(_(\"error on bundle-uri response line %d: %s\"),\n+\t\t\t line_nr, line);\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH)\n+\t\treturn error(_(\"expected flush after bundle-uri listing\"));\n+\n+\t/*\n+\t * Might die(), but obscure enough that that's OK, e.g. in\n+\t * serve.c we'll call BUG() on its equivalent (the\n+\t * PACKET_READ_RESPONSE_END check).\n+\t */\n+\tcheck_stateless_delimiter(stateless_rpc, reader,\n+\t\t\t\t _(\"expected response end packet after ref listing\"));\n+\n+\treturn 0;\n+}\n+\n struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\t struct ref **list, int for_push,\n \t\t\t struct transport_ls_refs_options *transport_options,\ndiff --git a/remote.h b/remote.h\nindex 1c4621b414b..1ebbe42792e 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -234,6 +234,11 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\t const struct string_list *server_options,\n \t\t\t int stateless_rpc);\n \n+/* Used for protocol v2 in order to retrieve refs from a remote */\n+struct bundle_list;\n+int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\n+\t\t\t struct bundle_list *bundles, int stateless_rpc);\n+\n int resolve_remote_symref(struct ref *ref, struct ref *list);\n \n /*\ndiff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh\nnew file mode 100644\nindex 00000000000..27294e9c976\n--- /dev/null\n+++ b/t/lib-t5730-protocol-v2-bundle-uri.sh\n@@ -0,0 +1,148 @@\n+# Included from t573*-protocol-v2-bundle-uri-*.sh\n+\n+T5730_PARENT=\n+T5730_URI=\n+T5730_BUNDLE_URI=\n+case \"$T5730_PROTOCOL\" in\n+file)\n+\tT5730_PARENT=file_parent\n+\tT5730_URI=\"file://$PWD/file_parent\"\n+\tT5730_BUNDLE_URI=\"$T5730_URI/fake.bdl\"\n+\ttest_set_prereq T5730_FILE\n+\t;;\n+git)\n+\t. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+\tstart_git_daemon --export-all --enable=receive-pack\n+\tT5730_PARENT=\"$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\"\n+\tT5730_URI=\"$GIT_DAEMON_URL/parent\"\n+\tT5730_BUNDLE_URI=\"https://example.com/fake.bdl\"\n+\ttest_set_prereq T5730_GIT\n+\t;;\n+http)\n+\t. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\tstart_httpd\n+\tT5730_PARENT=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\"\n+\tT5730_URI=\"$HTTPD_URL/smart/http_parent\"\n+\tT5730_BUNDLE_URI=\"https://example.com/fake.bdl\"\n+\ttest_set_prereq T5730_HTTP\n+\t;;\n+*)\n+\tBUG \"Need to pass valid T5730_PROTOCOL (was $T5730_PROTOCOL)\"\n+\t;;\n+esac\n+\n+test_expect_success \"setup protocol v2 $T5730_PROTOCOL:// tests\" '\n+\tgit init \"$T5730_PARENT\" &&\n+\ttest_commit -C \"$T5730_PARENT\" one &&\n+\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs true\n+'\n+\n+# Poor man's URI escaping. Good enough for the test suite whose trash\n+# directory has a space in it. See 93c3fcbe4d4 (git-svn: attempt to\n+# mimic SVN 1.7 URL canonicalization, 2012-07-28) for prior art.\n+test_uri_escape() {\n+\tsed 's/ /%20/g'\n+}\n+\n+case \"$T5730_PROTOCOL\" in\n+http)\n+\ttest_expect_success \"setup config for $T5730_PROTOCOL:// tests\" '\n+\t\tgit -C \"$T5730_PARENT\" config http.receivepack true\n+\t'\n+\t;;\n+*)\n+\t;;\n+esac\n+T5730_BUNDLE_URI_ESCAPED=$(echo \"$T5730_BUNDLE_URI\" | test_uri_escape)\n+\n+test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: no bundle-uri\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\ttest_when_finished \"git -C \\\"$T5730_PARENT\\\" config uploadpack.advertiseBundleURIs true\" &&\n+\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs false &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tgit \\\n+\t\t-c protocol.version=2 \\\n+\t\tls-remote --symref \"$T5730_URI\" \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\t! grep bundle-uri log\n+'\n+\n+test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: have bundle-uri\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tuploadpack.bundleURI \"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tgit \\\n+\t\t-c protocol.version=2 \\\n+\t\tls-remote --symref \"$T5730_URI\" \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\t# Server advertised bundle-uri capability\n+\tgrep bundle-uri log\n+'\n+\n+test_expect_success !T5730_HTTP \"bad client with $T5730_PROTOCOL:// using protocol v2\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_config -C \"$T5730_PARENT\" uploadpack.bundleURI \\\n+\t\t\"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tCloning into '\"'\"'child'\"'\"'...\n+\tEOF\n+\tcase \"$T5730_PROTOCOL\" in\n+\tfile)\n+\t\tcat >fatal-bundle-uri.expect <<-\\EOF\n+\t\tfatal: bundle-uri: unexpected argument: '\"'\"'test-bad-client'\"'\"'\n+\t\tEOF\n+\t\t;;\n+\t*)\n+\t\tcat >fatal.expect <<-\\EOF\n+\t\tfatal: read error: Connection reset by peer\n+\t\tEOF\n+\t\t;;\n+\tesac &&\n+\n+\ttest_when_finished \"rm -rf child\" &&\n+\ttest_must_fail ok=sigpipe env \\\n+\t\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\t\tGIT_TEST_PROTOCOL_BAD_BUNDLE_URI=true \\\n+\t\tgit -c protocol.version=2 \\\n+\t\tclone \"$T5730_URI\" child \\\n+\t\t>out 2>err &&\n+\ttest_must_be_empty out &&\n+\n+\tgrep -v -e ^fatal: -e ^error: err >err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\n+\tcase \"$T5730_PROTOCOL\" in\n+\tfile)\n+\t\t# Due to general race conditions with client/server replies we\n+\t\t# may or may not get \"fatal: the remote end hung up\n+\t\t# expectedly\" here\n+\t\tgrep \"^fatal: bundle-uri:\" err >fatal-bundle-uri.actual &&\n+\t\ttest_cmp fatal-bundle-uri.expect fatal-bundle-uri.actual\n+\t\t;;\n+\t*)\n+\t\tgrep \"^fatal:\" err >fatal.actual &&\n+\t\t# Due to the same race conditions this might be\n+\t\t# \"fatal: read error: Connection reset by peer\", \"fatal: the remote end\n+\t\t# hung up unexpectedly\" etc.\n+\t\tcat fatal.actual &&\n+\t\ttest_file_not_empty fatal.actual\n+\t\t;;\n+\tesac &&\n+\n+\tgrep \"clone> test-bad-client$\" log >sent-bad-request &&\n+\ttest_file_not_empty sent-bad-request\n+'\ndiff --git a/t/t5730-protocol-v2-bundle-uri-file.sh b/t/t5730-protocol-v2-bundle-uri-file.sh\nnew file mode 100755\nindex 00000000000..89203d3a23c\n--- /dev/null\n+++ b/t/t5730-protocol-v2-bundle-uri-file.sh\n@@ -0,0 +1,36 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri with protocol v2 and 'file://' transport\"\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'file://' transport\n+#\n+T5730_PROTOCOL=file\n+. \"$TEST_DIRECTORY\"/lib-t5730-protocol-v2-bundle-uri.sh\n+\n+test_expect_success \"unknown capability value with $T5730_PROTOCOL:// using protocol v2\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tuploadpack.bundleURI \"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tGIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE=true \\\n+\tgit \\\n+\t\t-c protocol.version=2 \\\n+\t\tls-remote --symref \"$T5730_URI\" \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\tgrep \"> bundle-uri=test-unknown-capability-value\" log\n+'\n+\n+test_done\ndiff --git a/t/t5731-protocol-v2-bundle-uri-git.sh b/t/t5731-protocol-v2-bundle-uri-git.sh\nnew file mode 100755\nindex 00000000000..282847b311f\n--- /dev/null\n+++ b/t/t5731-protocol-v2-bundle-uri-git.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri with protocol v2 and 'git://' transport\"\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'git://' transport\n+#\n+T5730_PROTOCOL=git\n+. \"$TEST_DIRECTORY\"/lib-t5730-protocol-v2-bundle-uri.sh\n+\n+test_done\ndiff --git a/t/t5732-protocol-v2-bundle-uri-http.sh b/t/t5732-protocol-v2-bundle-uri-http.sh\nnew file mode 100755\nindex 00000000000..fcc1cf3faef\n--- /dev/null\n+++ b/t/t5732-protocol-v2-bundle-uri-http.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri with protocol v2 and 'git://' transport\"\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'git://' transport\n+#\n+T5730_PROTOCOL=http\n+. \"$TEST_DIRECTORY\"/lib-t5730-protocol-v2-bundle-uri.sh\n+\n+test_done\ndiff --git a/transport-helper.c b/transport-helper.c\nindex e95267a4ab5..3ea7c2bb5ad 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -1267,9 +1267,22 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n \treturn ret;\n }\n \n+static int get_bundle_uri(struct transport *transport)\n+{\n+\tget_helper(transport);\n+\n+\tif (process_connect(transport, 0)) {\n+\t\tdo_take_over(transport);\n+\t\treturn transport->vtable->get_bundle_uri(transport);\n+\t}\n+\n+\treturn -1;\n+}\n+\n static struct transport_vtable vtable = {\n \t.set_option\t= set_helper_option,\n \t.get_refs_list\t= get_refs_list,\n+\t.get_bundle_uri = get_bundle_uri,\n \t.fetch_refs\t= fetch_refs,\n \t.push_refs\t= push_refs,\n \t.connect\t= connect_helper,\ndiff --git a/transport-internal.h b/transport-internal.h\nindex c4ca0b733ac..90ea749e5cf 100644\n--- a/transport-internal.h\n+++ b/transport-internal.h\n@@ -26,6 +26,13 @@ struct transport_vtable {\n \tstruct ref *(*get_refs_list)(struct transport *transport, int for_push,\n \t\t\t\t struct transport_ls_refs_options *transport_options);\n \n+\t/**\n+\t * Populates the remote side's bundle-uri under protocol v2,\n+\t * if the \"bundle-uri\" capability was advertised. Returns 0 if\n+\t * OK, negative values on error.\n+\t */\n+\tint (*get_bundle_uri)(struct transport *transport);\n+\n \t/**\n \t * Fetch the objects for the given refs. Note that this gets\n \t * an array, and should ignore the list structure.\ndiff --git a/transport.c b/transport.c\nindex e7b97194c10..a020adc1f56 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -22,6 +22,7 @@\n #include \"protocol.h\"\n #include \"object-store.h\"\n #include \"color.h\"\n+#include \"bundle-uri.h\"\n \n static int transport_use_color = -1;\n static char transport_colors[][COLOR_MAXLEN] = {\n@@ -359,6 +360,25 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \treturn handshake(transport, for_push, options, 1);\n }\n \n+static int get_bundle_uri(struct transport *transport)\n+{\n+\tstruct git_transport_data *data = transport->data;\n+\tstruct packet_reader reader;\n+\tint stateless_rpc = transport->stateless_rpc;\n+\n+\tif (!transport->bundles) {\n+\t\tCALLOC_ARRAY(transport->bundles, 1);\n+\t\tinit_bundle_list(transport->bundles);\n+\t}\n+\n+\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\treturn get_remote_bundle_uri(data->fd[1], &reader,\n+\t\t\t\t transport->bundles, stateless_rpc);\n+}\n+\n static int fetch_refs_via_pack(struct transport *transport,\n \t\t\t int nr_heads, struct ref **to_fetch)\n {\n@@ -902,6 +922,7 @@ static int disconnect_git(struct transport *transport)\n \n static struct transport_vtable taken_over_vtable = {\n \t.get_refs_list\t= get_refs_via_connect,\n+\t.get_bundle_uri = get_bundle_uri,\n \t.fetch_refs\t= fetch_refs_via_pack,\n \t.push_refs\t= git_transport_push,\n \t.disconnect\t= disconnect_git\n@@ -1054,6 +1075,7 @@ static struct transport_vtable bundle_vtable = {\n \n static struct transport_vtable builtin_smart_vtable = {\n \t.get_refs_list\t= get_refs_via_connect,\n+\t.get_bundle_uri = get_bundle_uri,\n \t.fetch_refs\t= fetch_refs_via_pack,\n \t.push_refs\t= git_transport_push,\n \t.connect\t= connect_git,\n@@ -1068,6 +1090,9 @@ struct transport *transport_get(struct remote *remote, const char *url)\n \tret->progress = isatty(2);\n \tstring_list_init_dup(&ret->pack_lockfiles);\n \n+\tCALLOC_ARRAY(ret->bundles, 1);\n+\tinit_bundle_list(ret->bundles);\n+\n \tif (!remote)\n \t\tBUG(\"No remote provided to transport_get()\");\n \n@@ -1482,6 +1507,34 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)\n \treturn rc;\n }\n \n+int transport_get_remote_bundle_uri(struct transport *transport)\n+{\n+\tconst struct transport_vtable *vtable = transport->vtable;\n+\n+\t/* Check config only once. */\n+\tif (transport->got_remote_bundle_uri++)\n+\t\treturn 0;\n+\n+\t/*\n+\t * \"Support\" protocol v0 and v2 without bundle-uri support by\n+\t * silently degrading to a NOOP.\n+\t */\n+\tif (!server_supports_v2(\"bundle-uri\", 0))\n+\t\treturn 0;\n+\n+\t/*\n+\t * This is intentionally below the transport.injectBundleURI,\n+\t * we want to be able to inject into protocol v0, or into the\n+\t * dialog of a server who doesn't support this.\n+\t */\n+\tif (!vtable->get_bundle_uri)\n+\t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n+\n+\tif (vtable->get_bundle_uri(transport) < 0)\n+\t\treturn error(_(\"could not retrieve server-advertised bundle-uri list\"));\n+\treturn 0;\n+}\n+\n void transport_unlock_pack(struct transport *transport, unsigned int flags)\n {\n \tint in_signal_handler = !!(flags & TRANSPORT_UNLOCK_PACK_IN_SIGNAL_HANDLER);\n@@ -1512,6 +1565,8 @@ int transport_disconnect(struct transport *transport)\n \t\tret = transport->vtable->disconnect(transport);\n \tif (transport->got_remote_refs)\n \t\tfree_refs((void *)transport->remote_refs);\n+\tclear_bundle_list(transport->bundles);\n+\tfree(transport->bundles);\n \tfree(transport);\n \treturn ret;\n }\ndiff --git a/transport.h b/transport.h\nindex b5bf7b3e704..85150f504fb 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -62,6 +62,7 @@ enum transport_family {\n \tTRANSPORT_FAMILY_IPV6\n };\n \n+struct bundle_list;\n struct transport {\n \tconst struct transport_vtable *vtable;\n \n@@ -76,6 +77,18 @@ struct transport {\n \t */\n \tunsigned got_remote_refs : 1;\n \n+\t/**\n+\t * Indicates whether we already called get_bundle_uri_list(); set by\n+\t * transport.c::transport_get_remote_bundle_uri().\n+\t */\n+\tunsigned got_remote_bundle_uri : 1;\n+\n+\t/*\n+\t * The results of \"command=bundle-uri\", if both sides support\n+\t * the \"bundle-uri\" capability.\n+\t */\n+\tstruct bundle_list *bundles;\n+\n \t/*\n \t * Transports that call take-over destroys the data specific to\n \t * the transport type while doing so, and cannot be reused.\n@@ -281,6 +294,12 @@ void transport_ls_refs_options_release(struct transport_ls_refs_options *opts);\n const struct ref *transport_get_remote_refs(struct transport *transport,\n \t\t\t\t\t struct transport_ls_refs_options *transport_options);\n \n+/**\n+ * Retrieve bundle URI(s) from a remote. Populates \"struct\n+ * transport\"'s \"bundle_uri\" and \"got_remote_bundle_uri\".\n+ */\n+int transport_get_remote_bundle_uri(struct transport *transport);\n+\n /*\n * Fetch the hash algorithm used by a remote.\n *\n-- \ngitgitgadget\n\n"},{"id":"466150","messageId":"ae0003bb39b486732928bde16a01d925afc12f96.1667264854.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.git.1667264854.gitgitgadget@gmail.com","subject":"[PATCH 3/9] bundle-uri client: add helper for testing server","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-01T01:07:28Z","receivedAt":"2022-11-01T01:07:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nAdd a 'test-tool bundle-uri ls-remote' command. This is a thin wrapper\nfor issuing protocol v2 \"bundle-uri\" commands to a server, and to the\nparsing routines in bundle-uri.c.\n\nSince in the \"git clone\" case we'll have already done the handshake(),\nbut not here, introduce a \"got_advertisement\" state along with\n\"got_remote_heads\". It seems to me that the \"got_remote_heads\" is\nbadly named in the first place, and the whole logic of eagerly getting\nls-refs on handshake() or not could be refactored somewhat, but let's\nnot do that now, and instead just add another self-documenting state\nvariable.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 2 +-\n t/helper/test-bundle-uri.c | 46 +++++++++++++++++++\n t/lib-t5730-protocol-v2-bundle-uri.sh | 63 ++++++++++++++++++++++-----\n transport.c | 43 ++++++++++++++----\n transport.h | 6 ++-\n 5 files changed, 139 insertions(+), 21 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex edf98295af2..22b1e506452 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1271,7 +1271,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t * Populate transport->got_remote_bundle_uri and\n \t * transport->bundle_uri. We might get nothing.\n \t */\n-\ttransport_get_remote_bundle_uri(transport);\n+\ttransport_get_remote_bundle_uri(transport, 1);\n \n \tif (mapped_refs) {\n \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex 25afd393428..ffb975b7b4f 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -3,6 +3,10 @@\n #include \"bundle-uri.h\"\n #include \"strbuf.h\"\n #include \"string-list.h\"\n+#include \"transport.h\"\n+#include \"ref-filter.h\"\n+#include \"remote.h\"\n+#include \"refs.h\"\n \n enum input_mode {\n \tKEY_VALUE_PAIRS,\n@@ -68,6 +72,46 @@ usage:\n \tusage_with_options(usage, options);\n }\n \n+static int cmd_ls_remote(int argc, const char **argv)\n+{\n+\tconst char *uploadpack = NULL;\n+\tstruct string_list server_options = STRING_LIST_INIT_DUP;\n+\tconst char *dest;\n+\tstruct remote *remote;\n+\tstruct transport *transport;\n+\tint status = 0;\n+\n+\tdest = argc > 1 ? argv[1] : NULL;\n+\n+\tremote = remote_get(dest);\n+\tif (!remote) {\n+\t\tif (dest)\n+\t\t\tdie(_(\"bad repository '%s'\"), dest);\n+\t\tdie(_(\"no remote configured to get bundle URIs from\"));\n+\t}\n+\tif (!remote->url_nr)\n+\t\tdie(_(\"remote '%s' has no configured URL\"), dest);\n+\n+\ttransport = transport_get(remote, NULL);\n+\tif (uploadpack)\n+\t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n+\tif (server_options.nr)\n+\t\ttransport->server_options = &server_options;\n+\n+\tif (transport_get_remote_bundle_uri(transport, 0) < 0) {\n+\t\terror(_(\"could not get the bundle-uri list\"));\n+\t\tstatus = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tprint_bundle_list(stdout, transport->bundles);\n+\n+cleanup:\n+\tif (transport_disconnect(transport))\n+\t\treturn 1;\n+\treturn status;\n+}\n+\n int cmd__bundle_uri(int argc, const char **argv)\n {\n \tconst char *usage[] = {\n@@ -88,6 +132,8 @@ int cmd__bundle_uri(int argc, const char **argv)\n \t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS);\n \tif (!strcmp(argv[1], \"parse-config\"))\n \t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE);\n+\tif (!strcmp(argv[1], \"ls-remote\"))\n+\t\treturn cmd_ls_remote(argc - 1, argv + 1);\n \terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n \n usage:\ndiff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh\nindex 27294e9c976..c327544641b 100644\n--- a/t/lib-t5730-protocol-v2-bundle-uri.sh\n+++ b/t/lib-t5730-protocol-v2-bundle-uri.sh\n@@ -34,7 +34,9 @@ esac\n test_expect_success \"setup protocol v2 $T5730_PROTOCOL:// tests\" '\n \tgit init \"$T5730_PARENT\" &&\n \ttest_commit -C \"$T5730_PARENT\" one &&\n-\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs true\n+\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs true &&\n+\tgit -C \"$T5730_PARENT\" config bundle.version 1 &&\n+\tgit -C \"$T5730_PARENT\" config bundle.mode all\n '\n \n # Poor man's URI escaping. Good enough for the test suite whose trash\n@@ -61,9 +63,8 @@ test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: no bundl\n \tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs false &&\n \n \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n-\tgit \\\n-\t\t-c protocol.version=2 \\\n-\t\tls-remote --symref \"$T5730_URI\" \\\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \"$T5730_URI\" \\\n \t\t>actual 2>err &&\n \n \t# Server responded using protocol v2\n@@ -76,12 +77,11 @@ test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: have bun\n \ttest_when_finished \"rm -f log\" &&\n \n \ttest_config -C \"$T5730_PARENT\" \\\n-\t\tuploadpack.bundleURI \"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\t\tbundle.only.uri \"$T5730_BUNDLE_URI_ESCAPED\" &&\n \n \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n-\tgit \\\n-\t\t-c protocol.version=2 \\\n-\t\tls-remote --symref \"$T5730_URI\" \\\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \"$T5730_URI\" \\\n \t\t>actual 2>err &&\n \n \t# Server responded using protocol v2\n@@ -94,8 +94,8 @@ test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: have bun\n test_expect_success !T5730_HTTP \"bad client with $T5730_PROTOCOL:// using protocol v2\" '\n \ttest_when_finished \"rm -f log\" &&\n \n-\ttest_config -C \"$T5730_PARENT\" uploadpack.bundleURI \\\n-\t\t\"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tbundle.only.uri \"$T5730_BUNDLE_URI_ESCAPED\" &&\n \n \tcat >err.expect <<-\\EOF &&\n \tCloning into '\"'\"'child'\"'\"'...\n@@ -146,3 +146,46 @@ test_expect_success !T5730_HTTP \"bad client with $T5730_PROTOCOL:// using protoc\n \tgrep \"clone> test-bad-client$\" log >sent-bad-request &&\n \ttest_file_not_empty sent-bad-request\n '\n+\n+test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tbundle.only.uri \"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\n+\t# All data about bundle URIs\n+\tcat >expect <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \\\n+\t\t\"$T5730_URI\" \\\n+\t\t>actual &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2 and extra data\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tbundle.only.uri \"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\n+\t# Extra data should be ignored\n+\ttest_config -C \"$T5730_PARENT\" bundle.only.extra bogus &&\n+\n+\t# All data about bundle URIs\n+\tcat >expect <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \\\n+\t\t\"$T5730_URI\" \\\n+\t\t>actual &&\n+\ttest_cmp_config_output expect actual\n+'\ndiff --git a/transport.c b/transport.c\nindex a020adc1f56..86460f5be28 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -198,6 +198,7 @@ struct git_transport_data {\n \tstruct git_transport_options options;\n \tstruct child_process *conn;\n \tint fd[2];\n+\tunsigned got_advertisement : 1;\n \tunsigned got_remote_heads : 1;\n \tenum protocol_version version;\n \tstruct oid_array extra_have;\n@@ -346,6 +347,7 @@ static struct ref *handshake(struct transport *transport, int for_push,\n \t\tBUG(\"unknown protocol version\");\n \t}\n \tdata->got_remote_heads = 1;\n+\tdata->got_advertisement = 1;\n \ttransport->hash_algo = reader.hash_algo;\n \n \tif (reader.line_peeked)\n@@ -371,6 +373,33 @@ static int get_bundle_uri(struct transport *transport)\n \t\tinit_bundle_list(transport->bundles);\n \t}\n \n+\tif (!data->got_advertisement) {\n+\t\tstruct ref *refs;\n+\t\tstruct git_transport_data *data = transport->data;\n+\t\tenum protocol_version version;\n+\n+\t\trefs = handshake(transport, 0, NULL, 0);\n+\t\tversion = data->version;\n+\n+\t\tswitch (version) {\n+\t\tcase protocol_v2:\n+\t\t\tassert(!refs);\n+\t\t\tbreak;\n+\t\tcase protocol_v0:\n+\t\tcase protocol_v1:\n+\t\tcase protocol_unknown_version:\n+\t\t\tassert(refs);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * \"Support\" protocol v0 and v2 without bundle-uri support by\n+\t * silently degrading to a NOOP.\n+\t */\n+\tif (!server_supports_v2(\"bundle-uri\", 0))\n+\t\treturn 0;\n+\n \tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n \t\t\t PACKET_READ_CHOMP_NEWLINE |\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n@@ -1507,7 +1536,7 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)\n \treturn rc;\n }\n \n-int transport_get_remote_bundle_uri(struct transport *transport)\n+int transport_get_remote_bundle_uri(struct transport *transport, int quiet)\n {\n \tconst struct transport_vtable *vtable = transport->vtable;\n \n@@ -1515,20 +1544,16 @@ int transport_get_remote_bundle_uri(struct transport *transport)\n \tif (transport->got_remote_bundle_uri++)\n \t\treturn 0;\n \n-\t/*\n-\t * \"Support\" protocol v0 and v2 without bundle-uri support by\n-\t * silently degrading to a NOOP.\n-\t */\n-\tif (!server_supports_v2(\"bundle-uri\", 0))\n-\t\treturn 0;\n-\n \t/*\n \t * This is intentionally below the transport.injectBundleURI,\n \t * we want to be able to inject into protocol v0, or into the\n \t * dialog of a server who doesn't support this.\n \t */\n-\tif (!vtable->get_bundle_uri)\n+\tif (!vtable->get_bundle_uri) {\n+\t\tif (quiet)\n+\t\t\treturn -1;\n \t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n+\t}\n \n \tif (vtable->get_bundle_uri(transport) < 0)\n \t\treturn error(_(\"could not retrieve server-advertised bundle-uri list\"));\ndiff --git a/transport.h b/transport.h\nindex 85150f504fb..dd0115b83bf 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -297,8 +297,12 @@ const struct ref *transport_get_remote_refs(struct transport *transport,\n /**\n * Retrieve bundle URI(s) from a remote. Populates \"struct\n * transport\"'s \"bundle_uri\" and \"got_remote_bundle_uri\".\n+ *\n+ * With `quiet=1` it will not complain if the serve doesn't support\n+ * the protocol, but only if we discover the server uses it, and\n+ * encounter issues then.\n */\n-int transport_get_remote_bundle_uri(struct transport *transport);\n+int transport_get_remote_bundle_uri(struct transport *transport, int quiet);\n \n /*\n * Fetch the hash algorithm used by a remote.\n-- \ngitgitgadget\n\n"},{"id":"466151","messageId":"431cd585184b29f981247a6fa298c5a555937728.1667264854.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.git.1667264854.gitgitgadget@gmail.com","subject":"[PATCH 4/9] bundle-uri: serve bundle.* keys from config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-01T01:07:29Z","receivedAt":"2022-11-01T01:07:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nImplement the \"bundle-uri\" protocol v2 capability by populating the\nkey=value packet lines from the local Git config. The list of bundles is\nprovided from the keys beginning with \"bundle.\".\n\nIn the future, we may want to filter this list to be more specific to\nthe exact known keys that the server intends to share, but for\nflexibility at the moment we will assume that the config values are\nwell-formed.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 16 +++++++++++-\n t/lib-t5730-protocol-v2-bundle-uri.sh | 35 +++++++++++++++++++++++++++\n 2 files changed, 50 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 2201b604b11..3469f1aaa98 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -585,6 +585,16 @@ cached:\n \treturn advertise_bundle_uri;\n }\n \n+static int config_to_packet_line(const char *key, const char *value, void *data)\n+{\n+\tstruct packet_reader *writer = data;\n+\n+\tif (!strncmp(key, \"bundle.\", 7))\n+\t\tpacket_write_fmt(writer->fd, \"%s=%s\", key, value);\n+\n+\treturn 0;\n+}\n+\n int bundle_uri_command(struct repository *r,\n \t\t struct packet_reader *request)\n {\n@@ -596,7 +606,11 @@ int bundle_uri_command(struct repository *r,\n \tif (request->status != PACKET_READ_FLUSH)\n \t\tdie(_(\"bundle-uri: expected flush after arguments\"));\n \n-\t/* TODO: Implement the communication */\n+\t/*\n+\t * Read all \"bundle.*\" config lines to the client as key=value\n+\t * packet lines.\n+\t */\n+\tgit_config(config_to_packet_line, &writer);\n \n \tpacket_writer_flush(&writer);\n \ndiff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh\nindex c327544641b..000fcc5e20b 100644\n--- a/t/lib-t5730-protocol-v2-bundle-uri.sh\n+++ b/t/lib-t5730-protocol-v2-bundle-uri.sh\n@@ -158,6 +158,8 @@ test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2\" '\n \t[bundle]\n \t\tversion = 1\n \t\tmode = all\n+\t[bundle \"only\"]\n+\t\turi = $T5730_BUNDLE_URI_ESCAPED\n \tEOF\n \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n \ttest-tool bundle-uri \\\n@@ -181,6 +183,39 @@ test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2 and ext\n \t[bundle]\n \t\tversion = 1\n \t\tmode = all\n+\t[bundle \"only\"]\n+\t\turi = $T5730_BUNDLE_URI_ESCAPED\n+\tEOF\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \\\n+\t\t\"$T5730_URI\" \\\n+\t\t>actual &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+\n+test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2 with list\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tbundle.bundle1.uri \"$T5730_BUNDLE_URI_ESCAPED-1.bdl\" &&\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tbundle.bundle2.uri \"$T5730_BUNDLE_URI_ESCAPED-2.bdl\" &&\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tbundle.bundle3.uri \"$T5730_BUNDLE_URI_ESCAPED-3.bdl\" &&\n+\n+\t# All data about bundle URIs\n+\tcat >expect <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"bundle1\"]\n+\t\turi = $T5730_BUNDLE_URI_ESCAPED-1.bdl\n+\t[bundle \"bundle2\"]\n+\t\turi = $T5730_BUNDLE_URI_ESCAPED-2.bdl\n+\t[bundle \"bundle3\"]\n+\t\turi = $T5730_BUNDLE_URI_ESCAPED-3.bdl\n \tEOF\n \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n \ttest-tool bundle-uri \\\n-- \ngitgitgadget\n\n"},{"id":"466152","messageId":"c877f7c033decc857bb58aff411faf6a27aa96d7.1667264854.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.git.1667264854.gitgitgadget@gmail.com","subject":"[PATCH 5/9] bundle-uri client: add boolean transfer.bundleURI setting","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-01T01:07:30Z","receivedAt":"2022-11-01T01:07:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nThe yet-to-be introduced client support for bundle-uri will always\nfall back on a full clone, but we'd still like to be able to ignore a\nserver's bundle-uri advertisement entirely.\n\nThe new transfer.bundleURI config option defaults to 'false', but a user\ncan set it to 'true' to enable checking for bundle URIs from the origin\nGit server using protocol v2.\n\nTo enable this setting by default in the correct tests, add a\nGIT_TEST_BUNDLE_URI environment variable.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Documentation/config/transfer.txt | 6 ++++++\n t/lib-t5730-protocol-v2-bundle-uri.sh | 3 +++\n transport.c | 10 +++++++---\n 3 files changed, 16 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/transfer.txt b/Documentation/config/transfer.txt\nindex 264812cca4d..c3ac767d1e4 100644\n--- a/Documentation/config/transfer.txt\n+++ b/Documentation/config/transfer.txt\n@@ -115,3 +115,9 @@ transfer.unpackLimit::\n transfer.advertiseSID::\n \tBoolean. When true, client and server processes will advertise their\n \tunique session IDs to their remote counterpart. Defaults to false.\n+\n+transfer.bundleURI::\n+\tWhen `true`, local `git clone` commands will request bundle\n+\tinformation from the remote server (if advertised) and download\n+\tbundles before continuing the clone through the Git protocol.\n+\tDefaults to `false`.\ndiff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh\nindex 000fcc5e20b..872bc39ad1b 100644\n--- a/t/lib-t5730-protocol-v2-bundle-uri.sh\n+++ b/t/lib-t5730-protocol-v2-bundle-uri.sh\n@@ -1,5 +1,8 @@\n # Included from t573*-protocol-v2-bundle-uri-*.sh\n \n+GIT_TEST_BUNDLE_URI=1\n+export GIT_TEST_BUNDLE_URI\n+\n T5730_PARENT=\n T5730_URI=\n T5730_BUNDLE_URI=\ndiff --git a/transport.c b/transport.c\nindex 86460f5be28..b33180226ae 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1538,6 +1538,7 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)\n \n int transport_get_remote_bundle_uri(struct transport *transport, int quiet)\n {\n+\tint value = 0;\n \tconst struct transport_vtable *vtable = transport->vtable;\n \n \t/* Check config only once. */\n@@ -1545,10 +1546,13 @@ int transport_get_remote_bundle_uri(struct transport *transport, int quiet)\n \t\treturn 0;\n \n \t/*\n-\t * This is intentionally below the transport.injectBundleURI,\n-\t * we want to be able to inject into protocol v0, or into the\n-\t * dialog of a server who doesn't support this.\n+\t * Don't use bundle-uri at all, if configured not to. Only proceed\n+\t * if GIT_TEST_BUNDLE_URI=1 or transfer.bundleURI=true.\n \t */\n+\tif (!git_env_bool(\"GIT_TEST_BUNDLE_URI\", 0) &&\n+\t (git_config_get_bool(\"transfer.bundleuri\", &value) || !value))\n+\t\treturn 0;\n+\n \tif (!vtable->get_bundle_uri) {\n \t\tif (quiet)\n \t\t\treturn -1;\n-- \ngitgitgadget\n\n"},{"id":"466153","messageId":"2200a70d279adcc6460dcb7bef7b0e681d19a414.1667264854.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.git.1667264854.gitgitgadget@gmail.com","subject":"[PATCH 6/9] strbuf: reintroduce strbuf_parent_directory()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-01T01:07:31Z","receivedAt":"2022-11-01T01:08:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe strbuf_parent_directory() method was added as a static method in\ncontrib/scalar by d0feac4e8c0 (scalar: 'register' sets recommended\nconfig and starts maintenance, 2021-12-03) and then removed in\n65f6a9eb0b9 (scalar: constrain enlistment search, 2022-08-18), but now\nthere is a need for a similar method in the bundle URI feature.\n\nRe-add the method, this time in strbuf.c. The method requirements are\nslightly modified to allow a trailing slash, in which case nothing is\ndone. The return value is the number of byte removed.\n\nSigned-off-by: Derrick Stolee \n---\n strbuf.c | 9 +++++++++\n strbuf.h | 7 +++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/strbuf.c b/strbuf.c\nindex 0890b1405c5..b5cb324c431 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -1200,3 +1200,12 @@ int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n \tfree(path2);\n \treturn res;\n }\n+\n+size_t strbuf_parent_directory(struct strbuf *buf)\n+{\n+\tsize_t len = buf->len;\n+\tsize_t offset = offset_1st_component(buf->buf);\n+\tchar *path_sep = find_last_dir_sep(buf->buf + offset);\n+\tstrbuf_setlen(buf, path_sep ? path_sep - buf->buf : offset);\n+\treturn len - buf->len;\n+}\ndiff --git a/strbuf.h b/strbuf.h\nindex 76965a17d44..8a964a08c31 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -664,6 +664,13 @@ int launch_sequence_editor(const char *path, struct strbuf *buffer,\n int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n \t\t\t const char *const *env);\n \n+/*\n+ * Remove the deepest subdirectory in the provided path string. If path\n+ * contains a trailing separator, then the path is considered a directory\n+ * and nothing is modified.\n+ */\n+size_t strbuf_parent_directory(struct strbuf *buf);\n+\n void strbuf_add_lines(struct strbuf *sb,\n \t\t const char *prefix,\n \t\t const char *buf,\n-- \ngitgitgadget\n\n"},{"id":"466154","messageId":"e002affe4bf1367b3397ca452900d6609280f248.1667264854.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.git.1667264854.gitgitgadget@gmail.com","subject":"[PATCH 8/9] bundle-uri: download bundles from an advertised list","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-01T01:07:33Z","receivedAt":"2022-11-01T01:08:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe logic in fetch_bundle_uri() is useful for the --bundle-uri option of\n'git clone', but is not helpful when the clone operation discovers a\nlist of URIs from the bundle-uri protocol v2 verb. To actually download\nand unbundle the advertised bundles, we need a different mechanism.\n\nCreate the new fetch_bundle_list() method which is very similar to\nfetch_bundle_uri() except that it relies on download_bundle_list()\ninstead of fetch_bundle_uri_internal(). The download_bundle_list()\nmethod will recursively call fetch_bundle_uri_internal() if any of the\nadvertised URIs serve a bundle list instead of a bundle. This will also\nfollow the bundle.list.mode setting from the input list: \"any\" will\ndownload only one such URI while \"all\" will download data from all of\nthe URIs.\n\nIn an identical way to fetch_bundle_uri(), the bundles are unbundled\nafter all of the bundle lists have been expanded and all necessary URIs.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 21 +++++++++++++++++++++\n bundle-uri.h | 11 +++++++++++\n 2 files changed, 32 insertions(+)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 0f3902bbd2b..0b0081ed269 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -577,6 +577,27 @@ cleanup:\n \treturn result;\n }\n \n+int fetch_bundle_list(struct repository *r, const char *uri, struct bundle_list *list)\n+{\n+\tint result;\n+\tstruct bundle_list global_list;\n+\n+\tinit_bundle_list(&global_list);\n+\n+\t/* If a bundle is added to this global list, then it is required. */\n+\tglobal_list.mode = BUNDLE_MODE_ALL;\n+\n+\tif ((result = download_bundle_list(r, list, &global_list, 0)))\n+\t\tgoto cleanup;\n+\n+\tresult = unbundle_all_bundles(r, &global_list);\n+\n+cleanup:\n+\tfor_all_bundles_in_list(&global_list, unlink_bundle, NULL);\n+\tclear_bundle_list(&global_list);\n+\treturn result;\n+}\n+\n /**\n * API for serve.c.\n */\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 7905e56732c..a75b68d2f5a 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -102,6 +102,17 @@ int bundle_uri_parse_config_format(const char *uri,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n+/**\n+ * Given a bundle list that was already advertised (likely by the\n+ * bundle-uri protocol v2 verb) at the given uri, fetch and unbundle the\n+ * bundles according to the bundle strategy of that list.\n+ *\n+ * Returns non-zero if no bundle information is found at the given 'uri'.\n+ */\n+int fetch_bundle_list(struct repository *r,\n+\t\t const char *uri,\n+\t\t struct bundle_list *list);\n+\n /**\n * API for serve.c.\n */\n-- \ngitgitgadget\n\n"},{"id":"466155","messageId":"3550f6fb91b0214c459cbc0dd9bb8c6d7c677f85.1667264854.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.git.1667264854.gitgitgadget@gmail.com","subject":"[PATCH 7/9] bundle-uri: allow relative URLs in bundle lists","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-01T01:07:32Z","receivedAt":"2022-11-01T01:08:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBundle providers may want to distribute that data across multiple CDNs.\nThis might require a change in the base URI, all the way to the domain\nname. If all bundles require an absolute URI in their 'uri' value, then\nevery push to a CDN would require altering the table of contents to\nmatch the expected domain and exact location within it.\n\nAllow a bundle list to specify a relative URI for the bundles.\nThis allows easier distribution of bundle data.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 16 ++++++++++-\n bundle-uri.h | 9 +++++++\n t/helper/test-bundle-uri.c | 2 ++\n t/t5750-bundle-uri-parse.sh | 54 +++++++++++++++++++++++++++++++++++++\n transport.c | 3 +++\n 5 files changed, 83 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 3469f1aaa98..0f3902bbd2b 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -7,6 +7,7 @@\n #include \"hashmap.h\"\n #include \"pkt-line.h\"\n #include \"config.h\"\n+#include \"remote.h\"\n \n static int compare_bundles(const void *hashmap_cmp_fn_data,\n \t\t\t const struct hashmap_entry *he1,\n@@ -49,6 +50,7 @@ void clear_bundle_list(struct bundle_list *list)\n \n \tfor_all_bundles_in_list(list, clear_remote_bundle_info, NULL);\n \thashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent);\n+\tfree(list->baseURI);\n }\n \n int for_all_bundles_in_list(struct bundle_list *list,\n@@ -163,7 +165,7 @@ static int bundle_list_update(const char *key, const char *value,\n \tif (!strcmp(subkey, \"uri\")) {\n \t\tif (bundle->uri)\n \t\t\treturn -1;\n-\t\tbundle->uri = xstrdup(value);\n+\t\tbundle->uri = relative_url(list->baseURI, value, NULL);\n \t\treturn 0;\n \t}\n \n@@ -190,6 +192,18 @@ int bundle_uri_parse_config_format(const char *uri,\n \t\t.error_action = CONFIG_ERROR_ERROR,\n \t};\n \n+\tif (!list->baseURI) {\n+\t\tstruct strbuf baseURI = STRBUF_INIT;\n+\t\tstrbuf_addstr(&baseURI, uri);\n+\n+\t\t/*\n+\t\t * If the URI does not end with a trailing slash, then\n+\t\t * remove the filename portion of the path. This is\n+\t\t * important for relative URIs.\n+\t\t */\n+\t\tstrbuf_parent_directory(&baseURI);\n+\t\tlist->baseURI = strbuf_detach(&baseURI, NULL);\n+\t}\n \tresult = git_config_from_file_with_options(config_to_bundle_list,\n \t\t\t\t\t\t filename, list,\n \t\t\t\t\t\t &opts);\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 357111ecce8..7905e56732c 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -61,6 +61,15 @@ struct bundle_list {\n \tint version;\n \tenum bundle_list_mode mode;\n \tstruct hashmap bundles;\n+\n+\t/**\n+\t * The baseURI of a bundle_list is used as the base for any\n+\t * relative URIs advertised by the bundle list at that location.\n+\t *\n+\t * When the list is generated from a Git server, then use that\n+\t * server's location.\n+\t */\n+\tchar *baseURI;\n };\n \n void init_bundle_list(struct bundle_list *list);\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex ffb975b7b4f..5aa0b494ce3 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -40,6 +40,8 @@ static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mo\n \n \tinit_bundle_list(&list);\n \n+\tlist.baseURI = xstrdup(\"\");\n+\n \tswitch (mode) {\n \tcase KEY_VALUE_PAIRS:\n \t\tif (argc != 1)\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex c2fe3f9c5a5..ed5262a8d2b 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -30,6 +30,30 @@ test_expect_success 'bundle_uri_parse_line() just URIs' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'bundle_uri_parse_line(): relative URIs' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=bundle.bdl\n+\tbundle.two.uri=../bundle.bdl\n+\tbundle.three.uri=sub/dir/bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = /bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = /sub/dir/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' '\n \tcat >in <<-\\EOF &&\n \t=bogus-value\n@@ -136,6 +160,36 @@ test_expect_success 'parse config format: just URIs' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format: relative URIs' '\n+\tcat >in <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = ../bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = sub/dir/bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = /bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = /sub/dir/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config in >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n test_expect_success 'parse config format edge cases: empty key or value' '\n \tcat >in1 <<-\\EOF &&\n \t= bogus-value\ndiff --git a/transport.c b/transport.c\nindex b33180226ae..2c4ff0c2023 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1553,6 +1553,9 @@ int transport_get_remote_bundle_uri(struct transport *transport, int quiet)\n \t (git_config_get_bool(\"transfer.bundleuri\", &value) || !value))\n \t\treturn 0;\n \n+\tif (!transport->bundles->baseURI)\n+\t\ttransport->bundles->baseURI = xstrdup(transport->url);\n+\n \tif (!vtable->get_bundle_uri) {\n \t\tif (quiet)\n \t\t\treturn -1;\n-- \ngitgitgadget\n\n"},{"id":"466156","messageId":"1c034bba744716b03c6ce4422a378001c9c7e39f.1667264854.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.git.1667264854.gitgitgadget@gmail.com","subject":"[PATCH 9/9] clone: unbundle the advertised bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-01T01:07:34Z","receivedAt":"2022-11-01T01:08:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA previous change introduced the transport methods to acquire a bundle\nlist from the 'bundle-uri' protocol v2 verb, when advertised _and_ when\nthe client has chosen to enable the feature.\n\nTeach Git to download and unbundle the data advertised by those bundles\nduring 'git clone'.\n\nAlso, since the --bundle-uri option exists, we do not want to mix the\nadvertised bundles with the user-specified bundles.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 26 +++++++++++++++++----\n t/t5601-clone.sh | 59 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 80 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 22b1e506452..09f10477ed6 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1267,11 +1267,27 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (refs)\n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n \n-\t/*\n-\t * Populate transport->got_remote_bundle_uri and\n-\t * transport->bundle_uri. We might get nothing.\n-\t */\n-\ttransport_get_remote_bundle_uri(transport, 1);\n+\tif (!bundle_uri) {\n+\t\t/*\n+\t\t* Populate transport->got_remote_bundle_uri and\n+\t\t* transport->bundle_uri. We might get nothing.\n+\t\t*/\n+\t\ttransport_get_remote_bundle_uri(transport, 1);\n+\n+\t\tif (transport->bundles &&\n+\t\t hashmap_get_size(&transport->bundles->bundles)) {\n+\t\t\t/* At this point, we need the_repository to match the cloned repo. */\n+\t\t\tif (repo_init(the_repository, git_dir, work_tree))\n+\t\t\t\twarning(_(\"failed to initialize the repo, skipping bundle URI\"));\n+\t\t\tif (fetch_bundle_list(the_repository,\n+\t\t\t\t\t remote->url[0],\n+\t\t\t\t\t transport->bundles))\n+\t\t\t\twarning(_(\"failed to fetch advertised bundles\"));\n+\t\t} else {\n+\t\t\tclear_bundle_list(transport->bundles);\n+\t\t\tFREE_AND_NULL(transport->bundles);\n+\t\t}\n+\t}\n \n \tif (mapped_refs) {\n \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 45f0803ed4d..d1d8139751e 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -795,6 +795,65 @@ test_expect_success 'reject cloning shallow repository using HTTP' '\n \tgit clone --no-reject-shallow $HTTPD_URL/smart/repo.git repo\n '\n \n+test_expect_success 'auto-discover bundle URI from HTTP clone' '\n+\ttest_when_finished rm -rf trace.txt repo2 \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" &&\n+\tgit -C src bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/everything.bundle\" --all &&\n+\tgit clone --bare --no-local src \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" &&\n+\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tuploadpack.advertiseBundleURIs true &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tbundle.version 1 &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tbundle.mode all &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tbundle.everything.uri \"$HTTPD_URL/everything.bundle\" &&\n+\n+\tGIT_TEST_BUNDLE_URI=1 \\\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n+\t\tgit -c protocol.version=2 clone \\\n+\t\t$HTTPD_URL/smart/repo2.git repo2 &&\n+\tcat >pattern <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/everything.bundle\"\\]\n+\tEOF\n+\tgrep -f pattern trace.txt\n+'\n+\n+test_expect_success 'auto-discover multiple bundles from HTTP clone' '\n+\ttest_when_finished rm -rf trace.txt repo3 \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" &&\n+\n+\ttest_commit -C src new &&\n+\tgit -C src bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/new.bundle\" HEAD~1..HEAD &&\n+\tgit clone --bare --no-local src \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" &&\n+\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tuploadpack.advertiseBundleURIs true &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.version 1 &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.mode all &&\n+\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.everything.uri \"$HTTPD_URL/everything.bundle\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.new.uri \"$HTTPD_URL/new.bundle\" &&\n+\n+\tGIT_TEST_BUNDLE_URI=1 \\\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n+\t\tgit -c protocol.version=2 clone \\\n+\t\t$HTTPD_URL/smart/repo3.git repo3 &&\n+\n+\t# We should fetch _both_ bundles\n+\tcat >pattern <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/everything.bundle\"\\]\n+\tEOF\n+\tgrep -f pattern trace.txt &&\n+\tcat >pattern <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/new.bundle\"\\]\n+\tEOF\n+\tgrep -f pattern trace.txt\n+'\n+\n # DO NOT add non-httpd-specific tests here, because the last part of this\n # test script is only executed when httpd is available and enabled.\n \n-- \ngitgitgadget\n"},{"id":"466385","messageId":"6efed789-7ea8-5135-aafb-59da1312b80d@dunelm.org.uk","threadId":"58731","inReplyTo":"2200a70d279adcc6460dcb7bef7b0e681d19a414.1667264854.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/9] strbuf: reintroduce strbuf_parent_directory()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-11-03T09:28:06Z","receivedAt":"2022-11-03T09:29:30Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Stolee\n\nOn 01/11/2022 01:07, Derrick Stolee via GitGitGadget wrote:\n> +size_t strbuf_parent_directory(struct strbuf *buf)\n> +{\n> +\tsize_t len = buf->len;\n> +\tsize_t offset = offset_1st_component(buf->buf);\n> +\tchar *path_sep = find_last_dir_sep(buf->buf + offset);\n> +\tstrbuf_setlen(buf, path_sep ? path_sep - buf->buf : offset);\n> +\treturn len - buf->len;\n> +}\n> diff --git a/strbuf.h b/strbuf.h\n> index 76965a17d44..8a964a08c31 100644\n> --- a/strbuf.h\n> +++ b/strbuf.h\n> @@ -664,6 +664,13 @@ int launch_sequence_editor(const char *path, struct strbuf *buffer,\n> int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n> \t\t\t const char *const *env);\n> \n> +/*\n> + * Remove the deepest subdirectory in the provided path string. If path\n> + * contains a trailing separator, then the path is considered a directory\n> + * and nothing is modified.\n> + */\n\nI found the name and description a bit confusing, if I've understood \ncorrectly it isn't really removing the deepest subdirectory but removes \nthe file name if there is one from the path. As such perhaps \nstrbuf_strip_filename() might make it clearer that it is a no-op if the \npath ends with a slash. It returns the number of characters removed but \nthat is undocumented.\n\nBest Wishes\n\nPhillip\n\n> +size_t strbuf_parent_directory(struct strbuf *buf);\n> +\n> void strbuf_add_lines(struct strbuf *sb,\n> \t\t const char *prefix,\n> \t\t const char *buf,\n"},{"id":"466393","messageId":"221103.86iljw1i5d.gmgdl@evledraar.gmail.com","threadId":"58731","inReplyTo":"2200a70d279adcc6460dcb7bef7b0e681d19a414.1667264854.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/9] strbuf: reintroduce strbuf_parent_directory()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-03T09:49:02Z","receivedAt":"2022-11-03T09:50:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Nov 01 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n\n> +/*\n\nPlease use a \"/**\" opening comment in strbuf.h, i.e. an API-doc comment.\n\nI see just above what you're adding we got it wrong :(, but most docs\nfor functions in that file correctly use that.\n\nSome recent background at:\nhttps://lore.kernel.org/git/220928.86pmffwmft.gmgdl@evledraar.gmail.com/\n"},{"id":"466850","messageId":"20221108170839.GA1731@szeder.dev","threadId":"58731","inReplyTo":"a02eee983185815d94ba1124b43eae43280aa963.1667264854.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/9] protocol v2: add server-side \"bundle-uri\" skeleton","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2022-11-08T17:08:39Z","receivedAt":"2022-11-08T17:08:47Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Nov 01, 2022 at 01:07:26AM +0000, Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n> An earlier version of this patch [1] used a different transfer format\n> than the \"key=value\" pairs in the current implementation. The change was\n> made to unify the protocol v2 verb with the bundle lists provided by\n\ns/verb/command/ here, and in a couple of later commit messages and an\nin-code comment as well.\n\nAs mentioned before on a previous bundle-uri patch series, protocol v2\nhas commands, not verbs.\n\n"},{"id":"467105","messageId":"508ba524-09d2-81f1-c5f8-815ab766791a@github.com","threadId":"58731","inReplyTo":"a02eee983185815d94ba1124b43eae43280aa963.1667264854.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/9] protocol v2: add server-side \"bundle-uri\" skeleton","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-11-11T01:59:37Z","receivedAt":"2022-11-11T01:59:46Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n> +\n> +PROTOCOL for bundle-uri\n> +^^^^^^^^^^^^^^^^^^^^^^^\n> +\n> +A `bundle-uri` request takes no arguments, and as noted above does not\n> +currently advertise a capability value. Both may be added in the\n> +future.\n> +\n> +When the client issues a `command=bundle-uri` the response is a list of\n\nnit: comma after `command=bundle-uri`\n\nI misread this a couple times dropping the \"the\", so it read like the\n`command=bundle-uri` was the *response*, not the request. I think the comma\nwould help avoid that?\n\n> +key-value pairs provided as packet lines with value `=`. The\n> +meaning of these key-value pairs are provided by the config keys in the\n> +`bundle.*` namespace (see linkgit:git-config[1]).\n\nWhat does this (\"the meaning of these key-value pairs are provided by the\nconfig keys...\") mean? Are the response keys in the `bundle.*` namespace? Or\ndo the client-side `bundle.*` keys provide some kind of translation of what\nthe keys mean?\n\n> +\n> +Clients are still expected to fully parse the line according to the\n> +above format, lines that do not conform to the format SHOULD be\n> +discarded. The user MAY be warned in such a case.\n\nWhy \"still\" - is there some reason they *wouldn't* parse the response line? \n\nIs \"the above format\" referring to `=` in general, or restricted\nto/guaranteed that the ``'s defined by the `bundle.*` namespace? I'm\nguessing \"still expected to fully parse\" == \"MUST parse\" (using\nMUST/SHOULD/MAY nomeclature), it would help to call that out explicitly to\nbe consistent with the rest of this doc.\n\n> +\n> +bundle-uri CLIENT AND SERVER EXPECTATIONS\n> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> +\n> +URI CONTENTS::\n> +The advertised URIs MUST be in one of two possible formats.\n> ++\n> +The first possible format is a bundle file that `git bundle verify`\n\nI don't think \"format\" is the right word to describe this (I'd reserve\n\"format\" for the literal format of the URI string). Maybe something like\nthis?\n\n\tThe advertised URIs MUST contain one of two types of content.\n\n\tThe advertised URI may contain a bundle file that `git bundle\n\tverify` would accept...\n\n\t...\n\t\n\tAlternatively, the advertised URI may provide a plaintext file...\n\n> +would accept. I.e. they MUST contain one or more reference tips for\n> +use by the client, MUST indicate prerequisites (in any) with standard\n> +\"-\" prefixes, and MUST indicate their \"object-format\", if\n> +applicable. Create \"*.bundle\" files with `git bundle create`.\n\nThe last sentence doesn't add anything that you don't know from the `git\nbundle verify` note in the first doesn't already tell you, and feels like a\nbit of a non-sequitur as a result. Although, it tangentially raises a\nquestion: do bundle files *have to* have the '.bundle' suffix to pass `git\nbundle verify`? If not, are they expected to when coming from these URIs?\n\n> ++\n> +The second possible format is a plaintext file that `git config --list`\n> +would accept (with the `--file` option). The key-value pairs in this list\n> +are in the `bundle.*` namespace (see linkgit:git-config[1]).\n> +\n> +bundle-uri CLIENT ERROR RECOVERY::\n> +A client MUST above all gracefully degrade on errors, whether that\n> +error is because of bad missing/data in the bundle URI(s), because\n> +that client is too dumb to e.g. understand and fully parse out bundle\n> +headers and their prerequisite relationships, or something else.\n> ++\n> +Server operators should feel confident in turning on \"bundle-uri\" and\n> +not worry if e.g. their CDN goes down that clones or fetches will run\n> +into hard failures. Even if the server bundle bundle(s) are\n> +incomplete, or bad in some way the client should still end up with a\n> +functioning repository, just as if it had chosen not to use this\n> +protocol extension.\n> ++\n> +All subsequent discussion on client and server interaction MUST keep\n> +this in mind.\n> +\n> +bundle-uri SERVER TO CLIENT::\n> +The ordering of the returned bundle uris is not significant. Clients\n> +MUST parse their headers to discover their contained OIDS and\n> +prerequisites. A client MUST consider the content of the bundle(s)\n> +themselves and their header as the ultimate source of truth.\n> ++\n> +A server MAY even return bundle(s) that don't have any direct\n> +relationship to the repository being cloned (either through accident,\n> +or intentional \"clever\" configuration), and expect a client to sort\n> +out what data they'd like from the bundle(s), if any.\n> +\n> +bundle-uri CLIENT TO SERVER::\n> +The client SHOULD provide reference tips found in the bundle header(s)\n> +as 'have' lines in any subsequent `fetch` request. A client MAY also\n> +ignore the bundle(s) entirely if doing so is deemed worse for some\n> +reason, e.g. if the bundles can't be downloaded, it doesn't like the\n> +tips it finds etc.\n> +\n> +WHEN ADVERTISED BUNDLE(S) REQUIRE NO FURTHER NEGOTIATION::\n> +If after issuing `bundle-uri` and `ls-refs`, and getting the header(s)\n> +of the bundle(s) the client finds that the ref tips it wants can be\n> +retrieved entirety from advertised bundle(s), it MAY disconnect. The\n\ns/entirety/entirely\n\nAnd to clarify, by \"it MAY disconnect\", you mean it may disconnect from the\nmain repository server? Or the bundle server? \n\n> +results of such a 'clone' or 'fetch' should be indistinguishable from\n> +the state attained without using bundle-uri.\n> +\n> +EARLY CLIENT DISCONNECTIONS AND ERROR RECOVERY::\n> +A client MAY perform an early disconnect while still downloading the\n> +bundle(s) (having streamed and parsed their headers). In such a case\n> +the client MUST gracefully recover from any errors related to\n> +finishing the download and validation of the bundle(s).\n> ++\n> +I.e. a client might need to re-connect and issue a 'fetch' command,\n> +and possibly fall back to not making use of 'bundle-uri' at all.\n> ++\n> +This \"MAY\" behavior is specified as such (and not a \"SHOULD\") on the\n> +assumption that a server advertising bundle uris is more likely than\n> +not to be serving up a relatively large repository, and to be pointing\n> +to URIs that have a good chance of being in working order. A client\n> +MAY e.g. look at the payload size of the bundles as a heuristic to see\n> +if an early disconnect is worth it, should falling back on a full\n> +\"fetch\" dialog be necessary.\n> +\n> +WHEN ADVERTISED BUNDLE(S) REQUIRE FURTHER NEGOTIATION::\n> +A client SHOULD commence a negotiation of a PACK from the server via\n> +the \"fetch\" command using the OID tips found in advertised bundles,\n> +even if's still in the process of downloading those bundle(s).\n> ++\n> +This allows for aggressive early disconnects from any interactive\n> +server dialog. The client blindly trusts that the advertised OID tips\n> +are relevant, and issues them as 'have' lines, it then requests any\n> +tips it would like (usually from the \"ls-refs\" advertisement) via\n> +'want' lines. The server will then compute a (hopefully small) PACK\n> +with the expected difference between the tips from the bundle(s) and\n> +the data requested.\n> ++\n> +The only connection the client then needs to keep active is to the\n> +concurrently downloading static bundle(s), when those and the\n> +incremental PACK are retrieved they should be inflated and\n> +validated. Any errors at this point should be gracefully recovered\n> +from, see above.\n> +\n> +bundle-uri PROTOCOL FEATURES\n> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> +\n> +As noted above the `=` definitions are documented by the\n> +`bundle.*` config namespace.\n\nSame comment as earlier - this is a confusing way to phrase this. If you\nmean \"the keys are part of the `bundle.*` namespace documented in\nlinkgit:git-config[1]\", I think you can just say that directly. If not, it\nwould help to clarify the relationship between the `bundle.*` namespace and\nthese keys.\n\n> +\n> +In particular, the `bundle.version` key specifies an integer value. The\n> +only accepted value at the moment is `1`, but if the client sees an\n> +unexpected value here then the client MUST ignore the bundle list.\n> +\n> +As long as `bundle.version` is understood, all other unknown keys MAY be\n> +ignored by the client. The server will guarantee compatibility with older\n> +clients, though newer clients may be better able to use the extra keys to\n> +minimize downloads.\n> +\n> +Any backwards-incompatible addition of pre-URI key-value will be\n> +guarded by a new `bundle.version` value or values in 'bundle-uri'\n> +capability advertisement itself, and/or by new future `bundle-uri`\n> +request arguments.\n> +\n> +Some example key-value pairs that are not currently implemented but could\n> +be implemented in the future include:\n> +\n> + * Add a \"hash=\" or \"size=\" advertise the expected hash or\n> + size of the bundle file.\n> +\n> + * Advertise that one or more bundle files are the same (to e.g. have\n> + clients round-robin or otherwise choose one of N possible files).\n> +\n> + * A \"oid=\" shortcut and \"prerequisite=\" shortcut. For\n> + expressing the common case of a bundle with one tip and no\n> + prerequisites, or one tip and one prerequisite.\n> ++\n> +This would allow for optimizing the common case of servers who'd like\n> +to provide one \"big bundle\" containing only their \"main\" branch,\n> +and/or incremental updates thereof.\n> ++\n> +A client receiving such a a response MAY assume that they can skip\n> +retrieving the header from a bundle at the indicated URI, and thus\n> +save themselves and the server(s) the request(s) needed to inspect the\n> +headers of that bundle or bundles.\n\nOverall, this document is quite thorough, especially with respect to edge\ncases/error handling. I found it a bit confusing at times (at least\npartially due to my unfamiliarity with protocol v2), including some\npotentially ambiguous phrasing or scenarios (especially those in the\ndisconnection & error recovery sections) that are difficult to clearly\ndescribe in generic terms.\n\nI think some sections (especially \"PROTOCOL for bundle-uri\" and \"bundle-uri\nCLIENT AND SERVER EXPECTATIONS\") would benefit from examples of what \"good\"\nand \"bad\" request/response values & behaviors look like; they would help\nillustrate some of those more complex situations. The rest of the patch (the\nimplementation & tests) looked good to me. \n\nThanks for your continued work on this, I'm really excited to see the next\nsteps of bundle servers in this series!\n\n"},{"id":"467365","messageId":"ca11478b-7b44-3018-04d8-0b84c4f43b56@github.com","threadId":"58731","inReplyTo":"508ba524-09d2-81f1-c5f8-815ab766791a@github.com","subject":"Re: [PATCH 1/9] protocol v2: add server-side \"bundle-uri\" skeleton","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-11-16T14:08:02Z","receivedAt":"2022-11-16T14:09:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/10/22 8:59 PM, Victoria Dye wrote:\n> Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n>> +\n>> +PROTOCOL for bundle-uri\n>> +^^^^^^^^^^^^^^^^^^^^^^^\n>> +\n>> +A `bundle-uri` request takes no arguments, and as noted above does not\n>> +currently advertise a capability value. Both may be added in the\n>> +future.\n>> +\n>> +When the client issues a `command=bundle-uri` the response is a list of\n> \n> nit: comma after `command=bundle-uri`\n> \n> I misread this a couple times dropping the \"the\", so it read like the\n> `command=bundle-uri` was the *response*, not the request. I think the comma\n> would help avoid that?\n\nI think this should be\n\n When the client issues a `command=bundle-uri` request, the response is...\n\n>> +key-value pairs provided as packet lines with value `=`. The\n>> +meaning of these key-value pairs are provided by the config keys in the\n>> +`bundle.*` namespace (see linkgit:git-config[1]).\n> \n> What does this (\"the meaning of these key-value pairs are provided by the\n> config keys...\") mean? Are the response keys in the `bundle.*` namespace? Or\n> do the client-side `bundle.*` keys provide some kind of translation of what\n> the keys mean?\n\nI can elaborate more here, but the intention is that the protocol defines\nonly how these key-value pairs are delivered, and how the client assigns\nmeaning to those values and acts upon them is defined elsewhere.\n\n>> +\n>> +Clients are still expected to fully parse the line according to the\n>> +above format, lines that do not conform to the format SHOULD be\n>> +discarded. The user MAY be warned in such a case.\n> \n> Why \"still\" - is there some reason they *wouldn't* parse the response line? \n\n\"still\" is not needed here.\n\n> Is \"the above format\" referring to `=` in general, or restricted\n> to/guaranteed that the ``'s defined by the `bundle.*` namespace? I'm\n> guessing \"still expected to fully parse\" == \"MUST parse\" (using\n> MUST/SHOULD/MAY nomeclature), it would help to call that out explicitly to\n> be consistent with the rest of this doc.\n\nUsing MUST simplifies things a lot.\n\n>> +\n>> +bundle-uri CLIENT AND SERVER EXPECTATIONS\n>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n>> +\n>> +URI CONTENTS::\n>> +The advertised URIs MUST be in one of two possible formats.\n>> ++\n>> +The first possible format is a bundle file that `git bundle verify`\n> \n> I don't think \"format\" is the right word to describe this (I'd reserve\n> \"format\" for the literal format of the URI string). Maybe something like\n> this?\n> \n> \tThe advertised URIs MUST contain one of two types of content.\n\nHow about...\n\n\"The content at the advertised URIs MUST be one of two types\" ?\n\n> \tThe advertised URI may contain a bundle file that `git bundle\n> \tverify` would accept...\n> \n> \t...\n> \t\n> \tAlternatively, the advertised URI may provide a plaintext file...\n> \n>> +would accept. I.e. they MUST contain one or more reference tips for\n>> +use by the client, MUST indicate prerequisites (in any) with standard\n>> +\"-\" prefixes, and MUST indicate their \"object-format\", if\n>> +applicable. Create \"*.bundle\" files with `git bundle create`.\n> \n> The last sentence doesn't add anything that you don't know from the `git\n> bundle verify` note in the first doesn't already tell you, and feels like a\n> bit of a non-sequitur as a result. Although, it tangentially raises a\n> question: do bundle files *have to* have the '.bundle' suffix to pass `git\n> bundle verify`? If not, are they expected to when coming from these URIs?\n \nThe files do not need that extension. This sentence can be removed.\n\n>> +WHEN ADVERTISED BUNDLE(S) REQUIRE NO FURTHER NEGOTIATION::\n>> +If after issuing `bundle-uri` and `ls-refs`, and getting the header(s)\n>> +of the bundle(s) the client finds that the ref tips it wants can be\n>> +retrieved entirety from advertised bundle(s), it MAY disconnect. The\n> \n> s/entirety/entirely\n\nThanks.\n\n> And to clarify, by \"it MAY disconnect\", you mean it may disconnect from the\n> main repository server? Or the bundle server? \n\nThe main repository server, since the bundle server is not speaking\nthe Git protocol, but there is definitely room to clarify here.\n\n>> +bundle-uri PROTOCOL FEATURES\n>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n>> +\n>> +As noted above the `=` definitions are documented by the\n>> +`bundle.*` config namespace.\n> \n> Same comment as earlier - this is a confusing way to phrase this. If you\n> mean \"the keys are part of the `bundle.*` namespace documented in\n> linkgit:git-config[1]\", I think you can just say that directly. If not, it\n> would help to clarify the relationship between the `bundle.*` namespace and\n> these keys.\n\nWill do.\n\n> Overall, this document is quite thorough, especially with respect to edge\n> cases/error handling. I found it a bit confusing at times (at least\n> partially due to my unfamiliarity with protocol v2), including some\n> potentially ambiguous phrasing or scenarios (especially those in the\n> disconnection & error recovery sections) that are difficult to clearly\n> describe in generic terms.\n> \n> I think some sections (especially \"PROTOCOL for bundle-uri\" and \"bundle-uri\n> CLIENT AND SERVER EXPECTATIONS\") would benefit from examples of what \"good\"\n> and \"bad\" request/response values & behaviors look like; they would help\n> illustrate some of those more complex situations. The rest of the patch (the\n> implementation & tests) looked good to me. \n\nThis is an interesting idea, although the examples of \"good\" and \"bad\" are\nprobably best left as the test cases. Looking through the rest of this\ndocument, this section is already much more verbose than the others, so I\nhesitate to add these examples at this point. Perhaps there is room to\nimprove the whole document with such examples as a follow-up.\n\nThanks,\n-Stolee\n"},{"id":"467380","messageId":"pull.1400.v2.git.1668628302.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.git.1667264854.gitgitgadget@gmail.com","subject":"[PATCH v2 0/9] Bundle URIs IV: advertise over protocol v2","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-16T19:51:33Z","receivedAt":"2022-11-16T19:51:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is based on the recent master batch that included ds/bundle-uri-....\n\nNow that git clone --bundle-uri can download a bundle list from a plaintex\nfile in config format, we can use the same set of key-value pairs to\nadvertise a bundle list over protocol v2. At the end of this series:\n\n 1. A server can advertise bundles when uploadPack.advertiseBundleURIs is\n enabled. The bundle list comes from the server's local config,\n specifically the bundle.* namespace.\n 2. A client can notice a server's bundle-uri advertisement and request the\n bundle list if transfer.bundleURI is enabled. The bundles are downloaded\n as if the list was advertised from the --bundle-uri option.\n\nMany patches in this series were adapted from Ævar's v2 RFC [1]. He is\nretained as author and I added myself as co-author only if the modifications\nwere significant.\n\n[1]\nhttps://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/\n\n * Patches 1-5 are mostly taken from [1], again with mostly minor updates.\n The one major difference is the packet line format being a single\n key=value format instead of a sequence of pairs. This also means that\n Patch 4 is entirely new since it feeds these pairs directly from the\n server's config.\n\n * Patches 6-9 finish off the ability for the client to notice the\n capability, request the values, and download bundles before continuing\n with the rest of the download.\n\nOne thing that is not handled here but could be handled in a future change\nis to disconnect from the origin Git server while downloading the bundle\nURIs, then reconnecting afterwards. This does not make any difference for\nHTTPS, but SSH may benefit from the reduced connection time. The git clone\n--bundle-uri option did not suffer from this because the bundles are\ndownloaded before the server connection begins.\n\nAfter this series, there is one more before the original scope of the plan\nis complete: using creation tokens as a heuristic. See [2] for the RFC\nversion of those patches.\n\n[2] https://github.com/derrickstolee/git/pull/22\n\n\nUpdates in v2\n=============\n\n * Commit messages now refer to protocol v2 \"commands\" not \"verbs\".\n * Several edits were made to gitprotocol-v2.txt thanks to Victoria's\n thorough review.\n * strbuf_parent_directory() is renamed strbuf_strip_file_from_path() to\n make it more clear how it behaves when ending with a slash.\n\nThanks,\n\n * Stolee\n\nDerrick Stolee (5):\n bundle-uri: serve bundle.* keys from config\n strbuf: introduce strbuf_strip_file_from_path()\n bundle-uri: allow relative URLs in bundle lists\n bundle-uri: download bundles from an advertised list\n clone: unbundle the advertised bundles\n\nÆvar Arnfjörð Bjarmason (4):\n protocol v2: add server-side \"bundle-uri\" skeleton\n bundle-uri client: add minimal NOOP client\n bundle-uri client: add helper for testing server\n bundle-uri client: add boolean transfer.bundleURI setting\n\n Documentation/config/transfer.txt | 6 +\n Documentation/gitprotocol-v2.txt | 201 ++++++++++++++++++++++\n builtin/clone.c | 23 +++\n bundle-uri.c | 91 +++++++++-\n bundle-uri.h | 27 +++\n connect.c | 47 +++++\n remote.h | 5 +\n serve.c | 6 +\n strbuf.c | 9 +\n strbuf.h | 12 ++\n t/helper/test-bundle-uri.c | 48 ++++++\n t/lib-t5730-protocol-v2-bundle-uri.sh | 229 +++++++++++++++++++++++++\n t/t5601-clone.sh | 59 +++++++\n t/t5701-git-serve.sh | 40 ++++-\n t/t5730-protocol-v2-bundle-uri-file.sh | 36 ++++\n t/t5731-protocol-v2-bundle-uri-git.sh | 17 ++\n t/t5732-protocol-v2-bundle-uri-http.sh | 17 ++\n t/t5750-bundle-uri-parse.sh | 54 ++++++\n transport-helper.c | 13 ++\n transport-internal.h | 7 +\n transport.c | 87 ++++++++++\n transport.h | 23 +++\n 22 files changed, 1055 insertions(+), 2 deletions(-)\n create mode 100644 t/lib-t5730-protocol-v2-bundle-uri.sh\n create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh\n create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh\n create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh\n\n\nbase-commit: c03801e19cb8ab36e9c0d17ff3d5e0c3b0f24193\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1400%2Fderrickstolee%2Fbundle-redo%2Fadvertise-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1400/derrickstolee/bundle-redo/advertise-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1400\n\nRange-diff vs v1:\n\n 1: a02eee98318 ! 1: beae335b855 protocol v2: add server-side \"bundle-uri\" skeleton\n @@ Commit message\n \n An earlier version of this patch [1] used a different transfer format\n than the \"key=value\" pairs in the current implementation. The change was\n - made to unify the protocol v2 verb with the bundle lists provided by\n + made to unify the protocol v2 command with the bundle lists provided by\n independent bundle servers. Further, the standard allows for the server\n to advertise a URI that contains a bundle list. This allows users\n automatically discovering bundle providers that are loosely associated\n @@ Documentation/gitprotocol-v2.txt: and associated requested information, each sep\n +currently advertise a capability value. Both may be added in the\n +future.\n +\n -+When the client issues a `command=bundle-uri` the response is a list of\n -+key-value pairs provided as packet lines with value `=`. The\n -+meaning of these key-value pairs are provided by the config keys in the\n -+`bundle.*` namespace (see linkgit:git-config[1]).\n ++When the client issues a `command=bundle-uri` request, the response is a\n ++list of key-value pairs provided as packet lines with value\n ++`=`. Each `` should be interpreted as a config key from\n ++the `bundle.*` namespace to construct a list of bundles. These keys are\n ++grouped by a `bundle..` subsection, where each key corresponding to a\n ++given `` contributes attributes to the bundle defined by that ``.\n ++See linkgit:git-config[1] for the specific details of these keys and how\n ++the Git client will interpret their values.\n +\n -+Clients are still expected to fully parse the line according to the\n -+above format, lines that do not conform to the format SHOULD be\n -+discarded. The user MAY be warned in such a case.\n ++Clients MUST parse the line according to the above format, lines that do\n ++not conform to the format SHOULD be discarded. The user MAY be warned in\n ++such a case.\n +\n +bundle-uri CLIENT AND SERVER EXPECTATIONS\n +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n +\n +URI CONTENTS::\n -+The advertised URIs MUST be in one of two possible formats.\n ++The content at the advertised URIs MUST be one of two types.\n ++\n -+The first possible format is a bundle file that `git bundle verify`\n ++The advertised URI may contain a bundle file that `git bundle verify`\n +would accept. I.e. they MUST contain one or more reference tips for\n +use by the client, MUST indicate prerequisites (in any) with standard\n +\"-\" prefixes, and MUST indicate their \"object-format\", if\n -+applicable. Create \"*.bundle\" files with `git bundle create`.\n ++applicable.\n ++\n -+The second possible format is a plaintext file that `git config --list`\n -+would accept (with the `--file` option). The key-value pairs in this list\n -+are in the `bundle.*` namespace (see linkgit:git-config[1]).\n ++The advertised URI may alternatively contain a plaintext file that `git\n ++config --list` would accept (with the `--file` option). The key-value\n ++pairs in this list are in the `bundle.*` namespace (see\n ++linkgit:git-config[1]).\n +\n +bundle-uri CLIENT ERROR RECOVERY::\n +A client MUST above all gracefully degrade on errors, whether that\n @@ Documentation/gitprotocol-v2.txt: and associated requested information, each sep\n +WHEN ADVERTISED BUNDLE(S) REQUIRE NO FURTHER NEGOTIATION::\n +If after issuing `bundle-uri` and `ls-refs`, and getting the header(s)\n +of the bundle(s) the client finds that the ref tips it wants can be\n -+retrieved entirety from advertised bundle(s), it MAY disconnect. The\n -+results of such a 'clone' or 'fetch' should be indistinguishable from\n -+the state attained without using bundle-uri.\n ++retrieved entirely from advertised bundle(s), the client MAY disconnect\n ++from the Git server. The results of such a 'clone' or 'fetch' should be\n ++indistinguishable from the state attained without using bundle-uri.\n +\n +EARLY CLIENT DISCONNECTIONS AND ERROR RECOVERY::\n +A client MAY perform an early disconnect while still downloading the\n @@ Documentation/gitprotocol-v2.txt: and associated requested information, each sep\n +bundle-uri PROTOCOL FEATURES\n +^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n +\n -+As noted above the `=` definitions are documented by the\n -+`bundle.*` config namespace.\n ++The client constructs a bundle list from the `=` pairs\n ++provided by the server. These pairs are part of the `bundle.*` namespace\n ++as documented in linkgit:git-config[1]. In this section, we discuss some\n ++of these keys and describe the actions the client will do in response to\n ++this information.\n +\n +In particular, the `bundle.version` key specifies an integer value. The\n +only accepted value at the moment is `1`, but if the client sees an\n 2: 64dd9bf41de = 2: 0d85aef965d bundle-uri client: add minimal NOOP client\n 3: ae0003bb39b = 3: c3269a24b57 bundle-uri client: add helper for testing server\n 4: 431cd585184 = 4: cd906f6d981 bundle-uri: serve bundle.* keys from config\n 5: c877f7c033d = 5: 93397468931 bundle-uri client: add boolean transfer.bundleURI setting\n 6: 2200a70d279 ! 6: 7d86852c015 strbuf: reintroduce strbuf_parent_directory()\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - strbuf: reintroduce strbuf_parent_directory()\n + strbuf: introduce strbuf_strip_file_from_path()\n \n The strbuf_parent_directory() method was added as a static method in\n contrib/scalar by d0feac4e8c0 (scalar: 'register' sets recommended\n @@ Commit message\n 65f6a9eb0b9 (scalar: constrain enlistment search, 2022-08-18), but now\n there is a need for a similar method in the bundle URI feature.\n \n - Re-add the method, this time in strbuf.c. The method requirements are\n - slightly modified to allow a trailing slash, in which case nothing is\n - done. The return value is the number of byte removed.\n + Re-add the method, this time in strbuf.c, but with a new name:\n + strbuf_strip_file_from_path(). The method requirements are slightly\n + modified to allow a trailing slash, in which case nothing is done, which\n + makes the name change valuable. The return value is the number of bytes\n + removed.\n \n Signed-off-by: Derrick Stolee \n \n @@ strbuf.c: int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n \treturn res;\n }\n +\n -+size_t strbuf_parent_directory(struct strbuf *buf)\n ++size_t strbuf_strip_file_from_path(struct strbuf *buf)\n +{\n +\tsize_t len = buf->len;\n +\tsize_t offset = offset_1st_component(buf->buf);\n +\tchar *path_sep = find_last_dir_sep(buf->buf + offset);\n -+\tstrbuf_setlen(buf, path_sep ? path_sep - buf->buf : offset);\n ++\tstrbuf_setlen(buf, path_sep ? path_sep - buf->buf + 1 : offset);\n +\treturn len - buf->len;\n +}\n \n @@ strbuf.h: int launch_sequence_editor(const char *path, struct strbuf *buffer,\n \t\t\t const char *const *env);\n \n +/*\n -+ * Remove the deepest subdirectory in the provided path string. If path\n ++ * Remove the filename from the provided path string. If the path\n + * contains a trailing separator, then the path is considered a directory\n -+ * and nothing is modified.\n ++ * and nothing is modified. Returns the number of characters removed from\n ++ * the path.\n ++ *\n ++ * Examples:\n ++ * - \"/path/to/file\" -> \"/path/to/\" (returns: 4)\n ++ * - \"/path/to/dir/\" -> \"/path/to/dir/\" (returns: 0)\n + */\n -+size_t strbuf_parent_directory(struct strbuf *buf);\n ++size_t strbuf_strip_file_from_path(struct strbuf *buf);\n +\n void strbuf_add_lines(struct strbuf *sb,\n \t\t const char *prefix,\n 7: 3550f6fb91b ! 7: 186e112d821 bundle-uri: allow relative URLs in bundle lists\n @@ bundle-uri.c: int bundle_uri_parse_config_format(const char *uri,\n +\t\t * remove the filename portion of the path. This is\n +\t\t * important for relative URIs.\n +\t\t */\n -+\t\tstrbuf_parent_directory(&baseURI);\n ++\t\tstrbuf_strip_file_from_path(&baseURI);\n +\t\tlist->baseURI = strbuf_detach(&baseURI, NULL);\n +\t}\n \tresult = git_config_from_file_with_options(config_to_bundle_list,\n 8: e002affe4bf ! 8: f254da46a2c bundle-uri: download bundles from an advertised list\n @@ Commit message\n \n The logic in fetch_bundle_uri() is useful for the --bundle-uri option of\n 'git clone', but is not helpful when the clone operation discovers a\n - list of URIs from the bundle-uri protocol v2 verb. To actually download\n - and unbundle the advertised bundles, we need a different mechanism.\n + list of URIs from the bundle-uri protocol v2 command. To actually\n + download and unbundle the advertised bundles, we need a different\n + mechanism.\n \n Create the new fetch_bundle_list() method which is very similar to\n fetch_bundle_uri() except that it relies on download_bundle_list()\n 9: 1c034bba744 ! 9: b62b4b17481 clone: unbundle the advertised bundles\n @@ Commit message\n clone: unbundle the advertised bundles\n \n A previous change introduced the transport methods to acquire a bundle\n - list from the 'bundle-uri' protocol v2 verb, when advertised _and_ when\n - the client has chosen to enable the feature.\n + list from the 'bundle-uri' protocol v2 command, when advertised _and_\n + when the client has chosen to enable the feature.\n \n Teach Git to download and unbundle the data advertised by those bundles\n during 'git clone'.\n\n-- \ngitgitgadget\n"},{"id":"467381","messageId":"beae335b85586ff88f89c551af62f91a04086781.1668628303.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v2.git.1668628302.gitgitgadget@gmail.com","subject":"[PATCH v2 1/9] protocol v2: add server-side \"bundle-uri\" skeleton","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-16T19:51:34Z","receivedAt":"2022-11-16T19:51:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nAdd a skeleton server-side implementation of a new \"bundle-uri\" command\nto protocol v2. This will allow conforming clients to optionally seed\ntheir initial clones or incremental fetches from URLs containing\n\"*.bundle\" files created with \"git bundle create\".\n\nThis change only performs the basic boilerplate of advertising a new\nprotocol v2 capability. The new 'bundle-uri' capability allows a client\nto request a list of bundles. Right now, the server only returns a flush\npacket, which corresponds to an empty advertisement. The bundle.* config\nnamespace describes which key-value pairs will be communicated across\nthis interface in future updates.\n\nThe critical bit right now is that the new boolean\nuploadPack.adverstiseBundleURIs config value signals whether or not this\ncapability should be advertised at all.\n\nAn earlier version of this patch [1] used a different transfer format\nthan the \"key=value\" pairs in the current implementation. The change was\nmade to unify the protocol v2 command with the bundle lists provided by\nindependent bundle servers. Further, the standard allows for the server\nto advertise a URI that contains a bundle list. This allows users\nautomatically discovering bundle providers that are loosely associated\nwith the origin server, but without the origin server knowing exactly\nwhich bundles are currently available.\n\n[1] https://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/\n\nThe very-deep headings needed to be modified to stop at level 4 due to\ndocumentation build issues. These were not recognized in earlier builds\nsince the file was previously in the Documentation/technical/ directory\nand was built in a different way. With its current location, the\nheavily-nested details were causing build issues and they are now\nreplaced with a bulletted list of details.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Documentation/gitprotocol-v2.txt | 201 +++++++++++++++++++++++++++++++\n bundle-uri.c | 36 ++++++\n bundle-uri.h | 7 ++\n serve.c | 6 +\n t/t5701-git-serve.sh | 40 +++++-\n 5 files changed, 289 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\nindex 59bf41cefb9..10bd2d40cec 100644\n--- a/Documentation/gitprotocol-v2.txt\n+++ b/Documentation/gitprotocol-v2.txt\n@@ -578,6 +578,207 @@ and associated requested information, each separated by a single space.\n \n \tobj-info = obj-id SP obj-size\n \n+bundle-uri\n+~~~~~~~~~~\n+\n+If the 'bundle-uri' capability is advertised, the server supports the\n+`bundle-uri' command.\n+\n+The capability is currently advertised with no value (i.e. not\n+\"bundle-uri=somevalue\"), a value may be added in the future for\n+supporting command-wide extensions. Clients MUST ignore any unknown\n+capability values and proceed with the 'bundle-uri` dialog they\n+support.\n+\n+The 'bundle-uri' command is intended to be issued before `fetch` to\n+get URIs to bundle files (see linkgit:git-bundle[1]) to \"seed\" and\n+inform the subsequent `fetch` command.\n+\n+The client CAN issue `bundle-uri` before or after any other valid\n+command. To be useful to clients it's expected that it'll be issued\n+after an `ls-refs` and before `fetch`, but CAN be issued at any time\n+in the dialog.\n+\n+DISCUSSION of bundle-uri\n+^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+The intent of the feature is optimize for server resource consumption\n+in the common case by changing the common case of fetching a very\n+large PACK during linkgit:git-clone[1] into a smaller incremental\n+fetch.\n+\n+It also allows servers to achieve better caching in combination with\n+an `uploadpack.packObjectsHook` (see linkgit:git-config[1]).\n+\n+By having new clones or fetches be a more predictable and common\n+negotiation against the tips of recently produces *.bundle file(s).\n+Servers might even pre-generate the results of such negotiations for\n+the `uploadpack.packObjectsHook` as new pushes come in.\n+\n+One way that servers could take advantage of these bundles is that the\n+server would anticipate that fresh clones will download a known bundle,\n+followed by catching up to the current state of the repository using ref\n+tips found in that bundle (or bundles).\n+\n+PROTOCOL for bundle-uri\n+^^^^^^^^^^^^^^^^^^^^^^^\n+\n+A `bundle-uri` request takes no arguments, and as noted above does not\n+currently advertise a capability value. Both may be added in the\n+future.\n+\n+When the client issues a `command=bundle-uri` request, the response is a\n+list of key-value pairs provided as packet lines with value\n+`=`. Each `` should be interpreted as a config key from\n+the `bundle.*` namespace to construct a list of bundles. These keys are\n+grouped by a `bundle..` subsection, where each key corresponding to a\n+given `` contributes attributes to the bundle defined by that ``.\n+See linkgit:git-config[1] for the specific details of these keys and how\n+the Git client will interpret their values.\n+\n+Clients MUST parse the line according to the above format, lines that do\n+not conform to the format SHOULD be discarded. The user MAY be warned in\n+such a case.\n+\n+bundle-uri CLIENT AND SERVER EXPECTATIONS\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+URI CONTENTS::\n+The content at the advertised URIs MUST be one of two types.\n++\n+The advertised URI may contain a bundle file that `git bundle verify`\n+would accept. I.e. they MUST contain one or more reference tips for\n+use by the client, MUST indicate prerequisites (in any) with standard\n+\"-\" prefixes, and MUST indicate their \"object-format\", if\n+applicable.\n++\n+The advertised URI may alternatively contain a plaintext file that `git\n+config --list` would accept (with the `--file` option). The key-value\n+pairs in this list are in the `bundle.*` namespace (see\n+linkgit:git-config[1]).\n+\n+bundle-uri CLIENT ERROR RECOVERY::\n+A client MUST above all gracefully degrade on errors, whether that\n+error is because of bad missing/data in the bundle URI(s), because\n+that client is too dumb to e.g. understand and fully parse out bundle\n+headers and their prerequisite relationships, or something else.\n++\n+Server operators should feel confident in turning on \"bundle-uri\" and\n+not worry if e.g. their CDN goes down that clones or fetches will run\n+into hard failures. Even if the server bundle bundle(s) are\n+incomplete, or bad in some way the client should still end up with a\n+functioning repository, just as if it had chosen not to use this\n+protocol extension.\n++\n+All subsequent discussion on client and server interaction MUST keep\n+this in mind.\n+\n+bundle-uri SERVER TO CLIENT::\n+The ordering of the returned bundle uris is not significant. Clients\n+MUST parse their headers to discover their contained OIDS and\n+prerequisites. A client MUST consider the content of the bundle(s)\n+themselves and their header as the ultimate source of truth.\n++\n+A server MAY even return bundle(s) that don't have any direct\n+relationship to the repository being cloned (either through accident,\n+or intentional \"clever\" configuration), and expect a client to sort\n+out what data they'd like from the bundle(s), if any.\n+\n+bundle-uri CLIENT TO SERVER::\n+The client SHOULD provide reference tips found in the bundle header(s)\n+as 'have' lines in any subsequent `fetch` request. A client MAY also\n+ignore the bundle(s) entirely if doing so is deemed worse for some\n+reason, e.g. if the bundles can't be downloaded, it doesn't like the\n+tips it finds etc.\n+\n+WHEN ADVERTISED BUNDLE(S) REQUIRE NO FURTHER NEGOTIATION::\n+If after issuing `bundle-uri` and `ls-refs`, and getting the header(s)\n+of the bundle(s) the client finds that the ref tips it wants can be\n+retrieved entirely from advertised bundle(s), the client MAY disconnect\n+from the Git server. The results of such a 'clone' or 'fetch' should be\n+indistinguishable from the state attained without using bundle-uri.\n+\n+EARLY CLIENT DISCONNECTIONS AND ERROR RECOVERY::\n+A client MAY perform an early disconnect while still downloading the\n+bundle(s) (having streamed and parsed their headers). In such a case\n+the client MUST gracefully recover from any errors related to\n+finishing the download and validation of the bundle(s).\n++\n+I.e. a client might need to re-connect and issue a 'fetch' command,\n+and possibly fall back to not making use of 'bundle-uri' at all.\n++\n+This \"MAY\" behavior is specified as such (and not a \"SHOULD\") on the\n+assumption that a server advertising bundle uris is more likely than\n+not to be serving up a relatively large repository, and to be pointing\n+to URIs that have a good chance of being in working order. A client\n+MAY e.g. look at the payload size of the bundles as a heuristic to see\n+if an early disconnect is worth it, should falling back on a full\n+\"fetch\" dialog be necessary.\n+\n+WHEN ADVERTISED BUNDLE(S) REQUIRE FURTHER NEGOTIATION::\n+A client SHOULD commence a negotiation of a PACK from the server via\n+the \"fetch\" command using the OID tips found in advertised bundles,\n+even if's still in the process of downloading those bundle(s).\n++\n+This allows for aggressive early disconnects from any interactive\n+server dialog. The client blindly trusts that the advertised OID tips\n+are relevant, and issues them as 'have' lines, it then requests any\n+tips it would like (usually from the \"ls-refs\" advertisement) via\n+'want' lines. The server will then compute a (hopefully small) PACK\n+with the expected difference between the tips from the bundle(s) and\n+the data requested.\n++\n+The only connection the client then needs to keep active is to the\n+concurrently downloading static bundle(s), when those and the\n+incremental PACK are retrieved they should be inflated and\n+validated. Any errors at this point should be gracefully recovered\n+from, see above.\n+\n+bundle-uri PROTOCOL FEATURES\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+The client constructs a bundle list from the `=` pairs\n+provided by the server. These pairs are part of the `bundle.*` namespace\n+as documented in linkgit:git-config[1]. In this section, we discuss some\n+of these keys and describe the actions the client will do in response to\n+this information.\n+\n+In particular, the `bundle.version` key specifies an integer value. The\n+only accepted value at the moment is `1`, but if the client sees an\n+unexpected value here then the client MUST ignore the bundle list.\n+\n+As long as `bundle.version` is understood, all other unknown keys MAY be\n+ignored by the client. The server will guarantee compatibility with older\n+clients, though newer clients may be better able to use the extra keys to\n+minimize downloads.\n+\n+Any backwards-incompatible addition of pre-URI key-value will be\n+guarded by a new `bundle.version` value or values in 'bundle-uri'\n+capability advertisement itself, and/or by new future `bundle-uri`\n+request arguments.\n+\n+Some example key-value pairs that are not currently implemented but could\n+be implemented in the future include:\n+\n+ * Add a \"hash=\" or \"size=\" advertise the expected hash or\n+ size of the bundle file.\n+\n+ * Advertise that one or more bundle files are the same (to e.g. have\n+ clients round-robin or otherwise choose one of N possible files).\n+\n+ * A \"oid=\" shortcut and \"prerequisite=\" shortcut. For\n+ expressing the common case of a bundle with one tip and no\n+ prerequisites, or one tip and one prerequisite.\n++\n+This would allow for optimizing the common case of servers who'd like\n+to provide one \"big bundle\" containing only their \"main\" branch,\n+and/or incremental updates thereof.\n++\n+A client receiving such a a response MAY assume that they can skip\n+retrieving the header from a bundle at the indicated URI, and thus\n+save themselves and the server(s) the request(s) needed to inspect the\n+headers of that bundle or bundles.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 79a914f961b..32022595964 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -563,6 +563,42 @@ cleanup:\n \treturn result;\n }\n \n+/**\n+ * API for serve.c.\n+ */\n+\n+int bundle_uri_advertise(struct repository *r, struct strbuf *value)\n+{\n+\tstatic int advertise_bundle_uri = -1;\n+\n+\tif (advertise_bundle_uri != -1)\n+\t\tgoto cached;\n+\n+\tadvertise_bundle_uri = 0;\n+\tgit_config_get_maybe_bool(\"uploadpack.advertisebundleuris\", &advertise_bundle_uri);\n+\n+cached:\n+\treturn advertise_bundle_uri;\n+}\n+\n+int bundle_uri_command(struct repository *r,\n+\t\t struct packet_reader *request)\n+{\n+\tstruct packet_writer writer;\n+\tpacket_writer_init(&writer, 1);\n+\n+\twhile (packet_reader_read(request) == PACKET_READ_NORMAL)\n+\t\tdie(_(\"bundle-uri: unexpected argument: '%s'\"), request->line);\n+\tif (request->status != PACKET_READ_FLUSH)\n+\t\tdie(_(\"bundle-uri: expected flush after arguments\"));\n+\n+\t/* TODO: Implement the communication */\n+\n+\tpacket_writer_flush(&writer);\n+\n+\treturn 0;\n+}\n+\n /**\n * General API for {transport,connect}.c etc.\n */\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 4dbc269823c..357111ecce8 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -4,6 +4,7 @@\n #include \"hashmap.h\"\n #include \"strbuf.h\"\n \n+struct packet_reader;\n struct repository;\n struct string_list;\n \n@@ -92,6 +93,12 @@ int bundle_uri_parse_config_format(const char *uri,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n+/**\n+ * API for serve.c.\n+ */\n+int bundle_uri_advertise(struct repository *r, struct strbuf *value);\n+int bundle_uri_command(struct repository *r, struct packet_reader *request);\n+\n /**\n * General API for {transport,connect}.c etc.\n */\ndiff --git a/serve.c b/serve.c\nindex 733347f602a..cbf4a143cfe 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -7,6 +7,7 @@\n #include \"protocol-caps.h\"\n #include \"serve.h\"\n #include \"upload-pack.h\"\n+#include \"bundle-uri.h\"\n \n static int advertise_sid = -1;\n static int client_hash_algo = GIT_HASH_SHA1;\n@@ -135,6 +136,11 @@ static struct protocol_capability capabilities[] = {\n \t\t.advertise = always_advertise,\n \t\t.command = cap_object_info,\n \t},\n+\t{\n+\t\t.name = \"bundle-uri\",\n+\t\t.advertise = bundle_uri_advertise,\n+\t\t.command = bundle_uri_command,\n+\t},\n };\n \n void protocol_v2_advertise_capabilities(void)\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex 1896f671cb3..f21e5e9d33d 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -13,7 +13,7 @@ test_expect_success 'test capability advertisement' '\n \twrong_algo sha1:sha256\n \twrong_algo sha256:sha1\n \tEOF\n-\tcat >expect <<-EOF &&\n+\tcat >expect.base <<-EOF &&\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n \tls-refs=unborn\n@@ -21,8 +21,11 @@ test_expect_success 'test capability advertisement' '\n \tserver-option\n \tobject-format=$(test_oid algo)\n \tobject-info\n+\tEOF\n+\tcat >expect.trailer <<-EOF &&\n \t0000\n \tEOF\n+\tcat expect.base expect.trailer >expect &&\n \n \tGIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \\\n \t\t--advertise-capabilities >out &&\n@@ -342,4 +345,39 @@ test_expect_success 'basics of object-info' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'test capability advertisement with uploadpack.advertiseBundleURIs' '\n+\ttest_config uploadpack.advertiseBundleURIs true &&\n+\n+\tcat >expect.extra <<-EOF &&\n+\tbundle-uri\n+\tEOF\n+\tcat expect.base \\\n+\t expect.extra \\\n+\t expect.trailer >expect &&\n+\n+\tGIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \\\n+\t\t--advertise-capabilities >out &&\n+\ttest-tool pkt-line unpack actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'basics of bundle-uri: dies if not enabled' '\n+\ttest-tool pkt-line pack >in <<-EOF &&\n+\tcommand=bundle-uri\n+\t0000\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid command '\"'\"'bundle-uri'\"'\"'\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\tERR serve: invalid command '\"'\"'bundle-uri'\"'\"'\n+\tEOF\n+\n+\ttest_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"467382","messageId":"c3269a24b5780023cbb4d173cb9cfb10c5a4b0d8.1668628303.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v2.git.1668628302.gitgitgadget@gmail.com","subject":"[PATCH v2 3/9] bundle-uri client: add helper for testing server","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-16T19:51:36Z","receivedAt":"2022-11-16T19:52:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nAdd a 'test-tool bundle-uri ls-remote' command. This is a thin wrapper\nfor issuing protocol v2 \"bundle-uri\" commands to a server, and to the\nparsing routines in bundle-uri.c.\n\nSince in the \"git clone\" case we'll have already done the handshake(),\nbut not here, introduce a \"got_advertisement\" state along with\n\"got_remote_heads\". It seems to me that the \"got_remote_heads\" is\nbadly named in the first place, and the whole logic of eagerly getting\nls-refs on handshake() or not could be refactored somewhat, but let's\nnot do that now, and instead just add another self-documenting state\nvariable.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 2 +-\n t/helper/test-bundle-uri.c | 46 +++++++++++++++++++\n t/lib-t5730-protocol-v2-bundle-uri.sh | 63 ++++++++++++++++++++++-----\n transport.c | 43 ++++++++++++++----\n transport.h | 6 ++-\n 5 files changed, 139 insertions(+), 21 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex edf98295af2..22b1e506452 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1271,7 +1271,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t * Populate transport->got_remote_bundle_uri and\n \t * transport->bundle_uri. We might get nothing.\n \t */\n-\ttransport_get_remote_bundle_uri(transport);\n+\ttransport_get_remote_bundle_uri(transport, 1);\n \n \tif (mapped_refs) {\n \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex 25afd393428..ffb975b7b4f 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -3,6 +3,10 @@\n #include \"bundle-uri.h\"\n #include \"strbuf.h\"\n #include \"string-list.h\"\n+#include \"transport.h\"\n+#include \"ref-filter.h\"\n+#include \"remote.h\"\n+#include \"refs.h\"\n \n enum input_mode {\n \tKEY_VALUE_PAIRS,\n@@ -68,6 +72,46 @@ usage:\n \tusage_with_options(usage, options);\n }\n \n+static int cmd_ls_remote(int argc, const char **argv)\n+{\n+\tconst char *uploadpack = NULL;\n+\tstruct string_list server_options = STRING_LIST_INIT_DUP;\n+\tconst char *dest;\n+\tstruct remote *remote;\n+\tstruct transport *transport;\n+\tint status = 0;\n+\n+\tdest = argc > 1 ? argv[1] : NULL;\n+\n+\tremote = remote_get(dest);\n+\tif (!remote) {\n+\t\tif (dest)\n+\t\t\tdie(_(\"bad repository '%s'\"), dest);\n+\t\tdie(_(\"no remote configured to get bundle URIs from\"));\n+\t}\n+\tif (!remote->url_nr)\n+\t\tdie(_(\"remote '%s' has no configured URL\"), dest);\n+\n+\ttransport = transport_get(remote, NULL);\n+\tif (uploadpack)\n+\t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n+\tif (server_options.nr)\n+\t\ttransport->server_options = &server_options;\n+\n+\tif (transport_get_remote_bundle_uri(transport, 0) < 0) {\n+\t\terror(_(\"could not get the bundle-uri list\"));\n+\t\tstatus = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tprint_bundle_list(stdout, transport->bundles);\n+\n+cleanup:\n+\tif (transport_disconnect(transport))\n+\t\treturn 1;\n+\treturn status;\n+}\n+\n int cmd__bundle_uri(int argc, const char **argv)\n {\n \tconst char *usage[] = {\n@@ -88,6 +132,8 @@ int cmd__bundle_uri(int argc, const char **argv)\n \t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS);\n \tif (!strcmp(argv[1], \"parse-config\"))\n \t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE);\n+\tif (!strcmp(argv[1], \"ls-remote\"))\n+\t\treturn cmd_ls_remote(argc - 1, argv + 1);\n \terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n \n usage:\ndiff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh\nindex 27294e9c976..c327544641b 100644\n--- a/t/lib-t5730-protocol-v2-bundle-uri.sh\n+++ b/t/lib-t5730-protocol-v2-bundle-uri.sh\n@@ -34,7 +34,9 @@ esac\n test_expect_success \"setup protocol v2 $T5730_PROTOCOL:// tests\" '\n \tgit init \"$T5730_PARENT\" &&\n \ttest_commit -C \"$T5730_PARENT\" one &&\n-\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs true\n+\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs true &&\n+\tgit -C \"$T5730_PARENT\" config bundle.version 1 &&\n+\tgit -C \"$T5730_PARENT\" config bundle.mode all\n '\n \n # Poor man's URI escaping. Good enough for the test suite whose trash\n@@ -61,9 +63,8 @@ test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: no bundl\n \tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs false &&\n \n \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n-\tgit \\\n-\t\t-c protocol.version=2 \\\n-\t\tls-remote --symref \"$T5730_URI\" \\\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \"$T5730_URI\" \\\n \t\t>actual 2>err &&\n \n \t# Server responded using protocol v2\n@@ -76,12 +77,11 @@ test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: have bun\n \ttest_when_finished \"rm -f log\" &&\n \n \ttest_config -C \"$T5730_PARENT\" \\\n-\t\tuploadpack.bundleURI \"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\t\tbundle.only.uri \"$T5730_BUNDLE_URI_ESCAPED\" &&\n \n \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n-\tgit \\\n-\t\t-c protocol.version=2 \\\n-\t\tls-remote --symref \"$T5730_URI\" \\\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \"$T5730_URI\" \\\n \t\t>actual 2>err &&\n \n \t# Server responded using protocol v2\n@@ -94,8 +94,8 @@ test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: have bun\n test_expect_success !T5730_HTTP \"bad client with $T5730_PROTOCOL:// using protocol v2\" '\n \ttest_when_finished \"rm -f log\" &&\n \n-\ttest_config -C \"$T5730_PARENT\" uploadpack.bundleURI \\\n-\t\t\"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tbundle.only.uri \"$T5730_BUNDLE_URI_ESCAPED\" &&\n \n \tcat >err.expect <<-\\EOF &&\n \tCloning into '\"'\"'child'\"'\"'...\n@@ -146,3 +146,46 @@ test_expect_success !T5730_HTTP \"bad client with $T5730_PROTOCOL:// using protoc\n \tgrep \"clone> test-bad-client$\" log >sent-bad-request &&\n \ttest_file_not_empty sent-bad-request\n '\n+\n+test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tbundle.only.uri \"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\n+\t# All data about bundle URIs\n+\tcat >expect <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \\\n+\t\t\"$T5730_URI\" \\\n+\t\t>actual &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2 and extra data\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tbundle.only.uri \"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\n+\t# Extra data should be ignored\n+\ttest_config -C \"$T5730_PARENT\" bundle.only.extra bogus &&\n+\n+\t# All data about bundle URIs\n+\tcat >expect <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \\\n+\t\t\"$T5730_URI\" \\\n+\t\t>actual &&\n+\ttest_cmp_config_output expect actual\n+'\ndiff --git a/transport.c b/transport.c\nindex a020adc1f56..86460f5be28 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -198,6 +198,7 @@ struct git_transport_data {\n \tstruct git_transport_options options;\n \tstruct child_process *conn;\n \tint fd[2];\n+\tunsigned got_advertisement : 1;\n \tunsigned got_remote_heads : 1;\n \tenum protocol_version version;\n \tstruct oid_array extra_have;\n@@ -346,6 +347,7 @@ static struct ref *handshake(struct transport *transport, int for_push,\n \t\tBUG(\"unknown protocol version\");\n \t}\n \tdata->got_remote_heads = 1;\n+\tdata->got_advertisement = 1;\n \ttransport->hash_algo = reader.hash_algo;\n \n \tif (reader.line_peeked)\n@@ -371,6 +373,33 @@ static int get_bundle_uri(struct transport *transport)\n \t\tinit_bundle_list(transport->bundles);\n \t}\n \n+\tif (!data->got_advertisement) {\n+\t\tstruct ref *refs;\n+\t\tstruct git_transport_data *data = transport->data;\n+\t\tenum protocol_version version;\n+\n+\t\trefs = handshake(transport, 0, NULL, 0);\n+\t\tversion = data->version;\n+\n+\t\tswitch (version) {\n+\t\tcase protocol_v2:\n+\t\t\tassert(!refs);\n+\t\t\tbreak;\n+\t\tcase protocol_v0:\n+\t\tcase protocol_v1:\n+\t\tcase protocol_unknown_version:\n+\t\t\tassert(refs);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * \"Support\" protocol v0 and v2 without bundle-uri support by\n+\t * silently degrading to a NOOP.\n+\t */\n+\tif (!server_supports_v2(\"bundle-uri\", 0))\n+\t\treturn 0;\n+\n \tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n \t\t\t PACKET_READ_CHOMP_NEWLINE |\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n@@ -1507,7 +1536,7 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)\n \treturn rc;\n }\n \n-int transport_get_remote_bundle_uri(struct transport *transport)\n+int transport_get_remote_bundle_uri(struct transport *transport, int quiet)\n {\n \tconst struct transport_vtable *vtable = transport->vtable;\n \n@@ -1515,20 +1544,16 @@ int transport_get_remote_bundle_uri(struct transport *transport)\n \tif (transport->got_remote_bundle_uri++)\n \t\treturn 0;\n \n-\t/*\n-\t * \"Support\" protocol v0 and v2 without bundle-uri support by\n-\t * silently degrading to a NOOP.\n-\t */\n-\tif (!server_supports_v2(\"bundle-uri\", 0))\n-\t\treturn 0;\n-\n \t/*\n \t * This is intentionally below the transport.injectBundleURI,\n \t * we want to be able to inject into protocol v0, or into the\n \t * dialog of a server who doesn't support this.\n \t */\n-\tif (!vtable->get_bundle_uri)\n+\tif (!vtable->get_bundle_uri) {\n+\t\tif (quiet)\n+\t\t\treturn -1;\n \t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n+\t}\n \n \tif (vtable->get_bundle_uri(transport) < 0)\n \t\treturn error(_(\"could not retrieve server-advertised bundle-uri list\"));\ndiff --git a/transport.h b/transport.h\nindex 85150f504fb..dd0115b83bf 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -297,8 +297,12 @@ const struct ref *transport_get_remote_refs(struct transport *transport,\n /**\n * Retrieve bundle URI(s) from a remote. Populates \"struct\n * transport\"'s \"bundle_uri\" and \"got_remote_bundle_uri\".\n+ *\n+ * With `quiet=1` it will not complain if the serve doesn't support\n+ * the protocol, but only if we discover the server uses it, and\n+ * encounter issues then.\n */\n-int transport_get_remote_bundle_uri(struct transport *transport);\n+int transport_get_remote_bundle_uri(struct transport *transport, int quiet);\n \n /*\n * Fetch the hash algorithm used by a remote.\n-- \ngitgitgadget\n\n"},{"id":"467383","messageId":"0d85aef965d62a684b25f388a946abd3c17809fc.1668628303.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v2.git.1668628302.gitgitgadget@gmail.com","subject":"[PATCH v2 2/9] bundle-uri client: add minimal NOOP client","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-16T19:51:35Z","receivedAt":"2022-11-16T19:52:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nSet up all the needed client parts of the \"bundle-uri\" protocol\nextension, without actually doing anything with the bundle URIs.\n\nI.e. if the server says it supports \"bundle-uri\" we'll issue a\ncommand=bundle-uri after command=ls-refs when we're cloning. We'll\nparse the returned output using the code already tested for in\nt5750-bundle-uri-parse.sh.\n\nWhat we aren't doing is actually acting on that data, i.e. downloading\nthe bundle(s) before we get to doing the command=fetch, and adjusting\nour negotiation dialog appropriately. I'll do that in subsequent\ncommits.\n\nThere's a question of what level of encapsulation we should use here,\nI've opted to use connect.h in clone.c, but we could also e.g. make\ntransport_get_remote_refs() invoke this, i.e. make it implicitly get\nthe bundle-uri list for later steps.\n\nThis approach means that we don't \"support\" this in \"git fetch\" for\nnow. I'm starting with the case of initial clones, although as noted\nin preceding commits to the protocol documentation nothing about this\napproach precludes getting bundles on incremental fetches.\n\nFor the t5732-protocol-v2-bundle-uri-http.sh it's not easy to set\nenvironment variables for git-upload-pack (it's started by Apache), so\nlet's skip the test under T5730_HTTP, and add unused T5730_{FILE,GIT}\nprerequisites for consistency and future use.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 7 ++\n bundle-uri.c | 4 +\n connect.c | 47 ++++++++\n remote.h | 5 +\n t/lib-t5730-protocol-v2-bundle-uri.sh | 148 +++++++++++++++++++++++++\n t/t5730-protocol-v2-bundle-uri-file.sh | 36 ++++++\n t/t5731-protocol-v2-bundle-uri-git.sh | 17 +++\n t/t5732-protocol-v2-bundle-uri-http.sh | 17 +++\n transport-helper.c | 13 +++\n transport-internal.h | 7 ++\n transport.c | 55 +++++++++\n transport.h | 19 ++++\n 12 files changed, 375 insertions(+)\n create mode 100644 t/lib-t5730-protocol-v2-bundle-uri.sh\n create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh\n create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh\n create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 547d6464b3c..edf98295af2 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -27,6 +27,7 @@\n #include \"iterator.h\"\n #include \"sigchain.h\"\n #include \"branch.h\"\n+#include \"connect.h\"\n #include \"remote.h\"\n #include \"run-command.h\"\n #include \"connected.h\"\n@@ -1266,6 +1267,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (refs)\n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n \n+\t/*\n+\t * Populate transport->got_remote_bundle_uri and\n+\t * transport->bundle_uri. We might get nothing.\n+\t */\n+\ttransport_get_remote_bundle_uri(transport);\n+\n \tif (mapped_refs) {\n \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n \ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 32022595964..2201b604b11 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -571,6 +571,10 @@ int bundle_uri_advertise(struct repository *r, struct strbuf *value)\n {\n \tstatic int advertise_bundle_uri = -1;\n \n+\tif (value &&\n+\t git_env_bool(\"GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE\", 0))\n+\t\tstrbuf_addstr(value, \"test-unknown-capability-value\");\n+\n \tif (advertise_bundle_uri != -1)\n \t\tgoto cached;\n \ndiff --git a/connect.c b/connect.c\nindex 5ea53deda23..d39effb7492 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -15,6 +15,7 @@\n #include \"version.h\"\n #include \"protocol.h\"\n #include \"alias.h\"\n+#include \"bundle-uri.h\"\n \n static char *server_capabilities_v1;\n static struct strvec server_capabilities_v2 = STRVEC_INIT;\n@@ -491,6 +492,52 @@ static void send_capabilities(int fd_out, struct packet_reader *reader)\n \t}\n }\n \n+int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\n+\t\t\t struct bundle_list *bundles, int stateless_rpc)\n+{\n+\tint line_nr = 1;\n+\n+\t/* Assert bundle-uri support */\n+\tserver_supports_v2(\"bundle-uri\", 1);\n+\n+\t/* (Re-)send capabilities */\n+\tsend_capabilities(fd_out, reader);\n+\n+\t/* Send command */\n+\tpacket_write_fmt(fd_out, \"command=bundle-uri\\n\");\n+\tpacket_delim(fd_out);\n+\n+\t/* Send options */\n+\tif (git_env_bool(\"GIT_TEST_PROTOCOL_BAD_BUNDLE_URI\", 0))\n+\t\tpacket_write_fmt(fd_out, \"test-bad-client\\n\");\n+\tpacket_flush(fd_out);\n+\n+\t/* Process response from server */\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\tconst char *line = reader->line;\n+\t\tline_nr++;\n+\n+\t\tif (!bundle_uri_parse_line(bundles, line))\n+\t\t\tcontinue;\n+\n+\t\treturn error(_(\"error on bundle-uri response line %d: %s\"),\n+\t\t\t line_nr, line);\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH)\n+\t\treturn error(_(\"expected flush after bundle-uri listing\"));\n+\n+\t/*\n+\t * Might die(), but obscure enough that that's OK, e.g. in\n+\t * serve.c we'll call BUG() on its equivalent (the\n+\t * PACKET_READ_RESPONSE_END check).\n+\t */\n+\tcheck_stateless_delimiter(stateless_rpc, reader,\n+\t\t\t\t _(\"expected response end packet after ref listing\"));\n+\n+\treturn 0;\n+}\n+\n struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\t struct ref **list, int for_push,\n \t\t\t struct transport_ls_refs_options *transport_options,\ndiff --git a/remote.h b/remote.h\nindex 1c4621b414b..1ebbe42792e 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -234,6 +234,11 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\t const struct string_list *server_options,\n \t\t\t int stateless_rpc);\n \n+/* Used for protocol v2 in order to retrieve refs from a remote */\n+struct bundle_list;\n+int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\n+\t\t\t struct bundle_list *bundles, int stateless_rpc);\n+\n int resolve_remote_symref(struct ref *ref, struct ref *list);\n \n /*\ndiff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh\nnew file mode 100644\nindex 00000000000..27294e9c976\n--- /dev/null\n+++ b/t/lib-t5730-protocol-v2-bundle-uri.sh\n@@ -0,0 +1,148 @@\n+# Included from t573*-protocol-v2-bundle-uri-*.sh\n+\n+T5730_PARENT=\n+T5730_URI=\n+T5730_BUNDLE_URI=\n+case \"$T5730_PROTOCOL\" in\n+file)\n+\tT5730_PARENT=file_parent\n+\tT5730_URI=\"file://$PWD/file_parent\"\n+\tT5730_BUNDLE_URI=\"$T5730_URI/fake.bdl\"\n+\ttest_set_prereq T5730_FILE\n+\t;;\n+git)\n+\t. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+\tstart_git_daemon --export-all --enable=receive-pack\n+\tT5730_PARENT=\"$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\"\n+\tT5730_URI=\"$GIT_DAEMON_URL/parent\"\n+\tT5730_BUNDLE_URI=\"https://example.com/fake.bdl\"\n+\ttest_set_prereq T5730_GIT\n+\t;;\n+http)\n+\t. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\tstart_httpd\n+\tT5730_PARENT=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\"\n+\tT5730_URI=\"$HTTPD_URL/smart/http_parent\"\n+\tT5730_BUNDLE_URI=\"https://example.com/fake.bdl\"\n+\ttest_set_prereq T5730_HTTP\n+\t;;\n+*)\n+\tBUG \"Need to pass valid T5730_PROTOCOL (was $T5730_PROTOCOL)\"\n+\t;;\n+esac\n+\n+test_expect_success \"setup protocol v2 $T5730_PROTOCOL:// tests\" '\n+\tgit init \"$T5730_PARENT\" &&\n+\ttest_commit -C \"$T5730_PARENT\" one &&\n+\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs true\n+'\n+\n+# Poor man's URI escaping. Good enough for the test suite whose trash\n+# directory has a space in it. See 93c3fcbe4d4 (git-svn: attempt to\n+# mimic SVN 1.7 URL canonicalization, 2012-07-28) for prior art.\n+test_uri_escape() {\n+\tsed 's/ /%20/g'\n+}\n+\n+case \"$T5730_PROTOCOL\" in\n+http)\n+\ttest_expect_success \"setup config for $T5730_PROTOCOL:// tests\" '\n+\t\tgit -C \"$T5730_PARENT\" config http.receivepack true\n+\t'\n+\t;;\n+*)\n+\t;;\n+esac\n+T5730_BUNDLE_URI_ESCAPED=$(echo \"$T5730_BUNDLE_URI\" | test_uri_escape)\n+\n+test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: no bundle-uri\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\ttest_when_finished \"git -C \\\"$T5730_PARENT\\\" config uploadpack.advertiseBundleURIs true\" &&\n+\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs false &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tgit \\\n+\t\t-c protocol.version=2 \\\n+\t\tls-remote --symref \"$T5730_URI\" \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\t! grep bundle-uri log\n+'\n+\n+test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: have bundle-uri\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tuploadpack.bundleURI \"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tgit \\\n+\t\t-c protocol.version=2 \\\n+\t\tls-remote --symref \"$T5730_URI\" \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\t# Server advertised bundle-uri capability\n+\tgrep bundle-uri log\n+'\n+\n+test_expect_success !T5730_HTTP \"bad client with $T5730_PROTOCOL:// using protocol v2\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_config -C \"$T5730_PARENT\" uploadpack.bundleURI \\\n+\t\t\"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tCloning into '\"'\"'child'\"'\"'...\n+\tEOF\n+\tcase \"$T5730_PROTOCOL\" in\n+\tfile)\n+\t\tcat >fatal-bundle-uri.expect <<-\\EOF\n+\t\tfatal: bundle-uri: unexpected argument: '\"'\"'test-bad-client'\"'\"'\n+\t\tEOF\n+\t\t;;\n+\t*)\n+\t\tcat >fatal.expect <<-\\EOF\n+\t\tfatal: read error: Connection reset by peer\n+\t\tEOF\n+\t\t;;\n+\tesac &&\n+\n+\ttest_when_finished \"rm -rf child\" &&\n+\ttest_must_fail ok=sigpipe env \\\n+\t\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\t\tGIT_TEST_PROTOCOL_BAD_BUNDLE_URI=true \\\n+\t\tgit -c protocol.version=2 \\\n+\t\tclone \"$T5730_URI\" child \\\n+\t\t>out 2>err &&\n+\ttest_must_be_empty out &&\n+\n+\tgrep -v -e ^fatal: -e ^error: err >err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\n+\tcase \"$T5730_PROTOCOL\" in\n+\tfile)\n+\t\t# Due to general race conditions with client/server replies we\n+\t\t# may or may not get \"fatal: the remote end hung up\n+\t\t# expectedly\" here\n+\t\tgrep \"^fatal: bundle-uri:\" err >fatal-bundle-uri.actual &&\n+\t\ttest_cmp fatal-bundle-uri.expect fatal-bundle-uri.actual\n+\t\t;;\n+\t*)\n+\t\tgrep \"^fatal:\" err >fatal.actual &&\n+\t\t# Due to the same race conditions this might be\n+\t\t# \"fatal: read error: Connection reset by peer\", \"fatal: the remote end\n+\t\t# hung up unexpectedly\" etc.\n+\t\tcat fatal.actual &&\n+\t\ttest_file_not_empty fatal.actual\n+\t\t;;\n+\tesac &&\n+\n+\tgrep \"clone> test-bad-client$\" log >sent-bad-request &&\n+\ttest_file_not_empty sent-bad-request\n+'\ndiff --git a/t/t5730-protocol-v2-bundle-uri-file.sh b/t/t5730-protocol-v2-bundle-uri-file.sh\nnew file mode 100755\nindex 00000000000..89203d3a23c\n--- /dev/null\n+++ b/t/t5730-protocol-v2-bundle-uri-file.sh\n@@ -0,0 +1,36 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri with protocol v2 and 'file://' transport\"\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'file://' transport\n+#\n+T5730_PROTOCOL=file\n+. \"$TEST_DIRECTORY\"/lib-t5730-protocol-v2-bundle-uri.sh\n+\n+test_expect_success \"unknown capability value with $T5730_PROTOCOL:// using protocol v2\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tuploadpack.bundleURI \"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tGIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE=true \\\n+\tgit \\\n+\t\t-c protocol.version=2 \\\n+\t\tls-remote --symref \"$T5730_URI\" \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\tgrep \"> bundle-uri=test-unknown-capability-value\" log\n+'\n+\n+test_done\ndiff --git a/t/t5731-protocol-v2-bundle-uri-git.sh b/t/t5731-protocol-v2-bundle-uri-git.sh\nnew file mode 100755\nindex 00000000000..282847b311f\n--- /dev/null\n+++ b/t/t5731-protocol-v2-bundle-uri-git.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri with protocol v2 and 'git://' transport\"\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'git://' transport\n+#\n+T5730_PROTOCOL=git\n+. \"$TEST_DIRECTORY\"/lib-t5730-protocol-v2-bundle-uri.sh\n+\n+test_done\ndiff --git a/t/t5732-protocol-v2-bundle-uri-http.sh b/t/t5732-protocol-v2-bundle-uri-http.sh\nnew file mode 100755\nindex 00000000000..fcc1cf3faef\n--- /dev/null\n+++ b/t/t5732-protocol-v2-bundle-uri-http.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri with protocol v2 and 'git://' transport\"\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'git://' transport\n+#\n+T5730_PROTOCOL=http\n+. \"$TEST_DIRECTORY\"/lib-t5730-protocol-v2-bundle-uri.sh\n+\n+test_done\ndiff --git a/transport-helper.c b/transport-helper.c\nindex e95267a4ab5..3ea7c2bb5ad 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -1267,9 +1267,22 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n \treturn ret;\n }\n \n+static int get_bundle_uri(struct transport *transport)\n+{\n+\tget_helper(transport);\n+\n+\tif (process_connect(transport, 0)) {\n+\t\tdo_take_over(transport);\n+\t\treturn transport->vtable->get_bundle_uri(transport);\n+\t}\n+\n+\treturn -1;\n+}\n+\n static struct transport_vtable vtable = {\n \t.set_option\t= set_helper_option,\n \t.get_refs_list\t= get_refs_list,\n+\t.get_bundle_uri = get_bundle_uri,\n \t.fetch_refs\t= fetch_refs,\n \t.push_refs\t= push_refs,\n \t.connect\t= connect_helper,\ndiff --git a/transport-internal.h b/transport-internal.h\nindex c4ca0b733ac..90ea749e5cf 100644\n--- a/transport-internal.h\n+++ b/transport-internal.h\n@@ -26,6 +26,13 @@ struct transport_vtable {\n \tstruct ref *(*get_refs_list)(struct transport *transport, int for_push,\n \t\t\t\t struct transport_ls_refs_options *transport_options);\n \n+\t/**\n+\t * Populates the remote side's bundle-uri under protocol v2,\n+\t * if the \"bundle-uri\" capability was advertised. Returns 0 if\n+\t * OK, negative values on error.\n+\t */\n+\tint (*get_bundle_uri)(struct transport *transport);\n+\n \t/**\n \t * Fetch the objects for the given refs. Note that this gets\n \t * an array, and should ignore the list structure.\ndiff --git a/transport.c b/transport.c\nindex e7b97194c10..a020adc1f56 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -22,6 +22,7 @@\n #include \"protocol.h\"\n #include \"object-store.h\"\n #include \"color.h\"\n+#include \"bundle-uri.h\"\n \n static int transport_use_color = -1;\n static char transport_colors[][COLOR_MAXLEN] = {\n@@ -359,6 +360,25 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \treturn handshake(transport, for_push, options, 1);\n }\n \n+static int get_bundle_uri(struct transport *transport)\n+{\n+\tstruct git_transport_data *data = transport->data;\n+\tstruct packet_reader reader;\n+\tint stateless_rpc = transport->stateless_rpc;\n+\n+\tif (!transport->bundles) {\n+\t\tCALLOC_ARRAY(transport->bundles, 1);\n+\t\tinit_bundle_list(transport->bundles);\n+\t}\n+\n+\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\treturn get_remote_bundle_uri(data->fd[1], &reader,\n+\t\t\t\t transport->bundles, stateless_rpc);\n+}\n+\n static int fetch_refs_via_pack(struct transport *transport,\n \t\t\t int nr_heads, struct ref **to_fetch)\n {\n@@ -902,6 +922,7 @@ static int disconnect_git(struct transport *transport)\n \n static struct transport_vtable taken_over_vtable = {\n \t.get_refs_list\t= get_refs_via_connect,\n+\t.get_bundle_uri = get_bundle_uri,\n \t.fetch_refs\t= fetch_refs_via_pack,\n \t.push_refs\t= git_transport_push,\n \t.disconnect\t= disconnect_git\n@@ -1054,6 +1075,7 @@ static struct transport_vtable bundle_vtable = {\n \n static struct transport_vtable builtin_smart_vtable = {\n \t.get_refs_list\t= get_refs_via_connect,\n+\t.get_bundle_uri = get_bundle_uri,\n \t.fetch_refs\t= fetch_refs_via_pack,\n \t.push_refs\t= git_transport_push,\n \t.connect\t= connect_git,\n@@ -1068,6 +1090,9 @@ struct transport *transport_get(struct remote *remote, const char *url)\n \tret->progress = isatty(2);\n \tstring_list_init_dup(&ret->pack_lockfiles);\n \n+\tCALLOC_ARRAY(ret->bundles, 1);\n+\tinit_bundle_list(ret->bundles);\n+\n \tif (!remote)\n \t\tBUG(\"No remote provided to transport_get()\");\n \n@@ -1482,6 +1507,34 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)\n \treturn rc;\n }\n \n+int transport_get_remote_bundle_uri(struct transport *transport)\n+{\n+\tconst struct transport_vtable *vtable = transport->vtable;\n+\n+\t/* Check config only once. */\n+\tif (transport->got_remote_bundle_uri++)\n+\t\treturn 0;\n+\n+\t/*\n+\t * \"Support\" protocol v0 and v2 without bundle-uri support by\n+\t * silently degrading to a NOOP.\n+\t */\n+\tif (!server_supports_v2(\"bundle-uri\", 0))\n+\t\treturn 0;\n+\n+\t/*\n+\t * This is intentionally below the transport.injectBundleURI,\n+\t * we want to be able to inject into protocol v0, or into the\n+\t * dialog of a server who doesn't support this.\n+\t */\n+\tif (!vtable->get_bundle_uri)\n+\t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n+\n+\tif (vtable->get_bundle_uri(transport) < 0)\n+\t\treturn error(_(\"could not retrieve server-advertised bundle-uri list\"));\n+\treturn 0;\n+}\n+\n void transport_unlock_pack(struct transport *transport, unsigned int flags)\n {\n \tint in_signal_handler = !!(flags & TRANSPORT_UNLOCK_PACK_IN_SIGNAL_HANDLER);\n@@ -1512,6 +1565,8 @@ int transport_disconnect(struct transport *transport)\n \t\tret = transport->vtable->disconnect(transport);\n \tif (transport->got_remote_refs)\n \t\tfree_refs((void *)transport->remote_refs);\n+\tclear_bundle_list(transport->bundles);\n+\tfree(transport->bundles);\n \tfree(transport);\n \treturn ret;\n }\ndiff --git a/transport.h b/transport.h\nindex b5bf7b3e704..85150f504fb 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -62,6 +62,7 @@ enum transport_family {\n \tTRANSPORT_FAMILY_IPV6\n };\n \n+struct bundle_list;\n struct transport {\n \tconst struct transport_vtable *vtable;\n \n@@ -76,6 +77,18 @@ struct transport {\n \t */\n \tunsigned got_remote_refs : 1;\n \n+\t/**\n+\t * Indicates whether we already called get_bundle_uri_list(); set by\n+\t * transport.c::transport_get_remote_bundle_uri().\n+\t */\n+\tunsigned got_remote_bundle_uri : 1;\n+\n+\t/*\n+\t * The results of \"command=bundle-uri\", if both sides support\n+\t * the \"bundle-uri\" capability.\n+\t */\n+\tstruct bundle_list *bundles;\n+\n \t/*\n \t * Transports that call take-over destroys the data specific to\n \t * the transport type while doing so, and cannot be reused.\n@@ -281,6 +294,12 @@ void transport_ls_refs_options_release(struct transport_ls_refs_options *opts);\n const struct ref *transport_get_remote_refs(struct transport *transport,\n \t\t\t\t\t struct transport_ls_refs_options *transport_options);\n \n+/**\n+ * Retrieve bundle URI(s) from a remote. Populates \"struct\n+ * transport\"'s \"bundle_uri\" and \"got_remote_bundle_uri\".\n+ */\n+int transport_get_remote_bundle_uri(struct transport *transport);\n+\n /*\n * Fetch the hash algorithm used by a remote.\n *\n-- \ngitgitgadget\n\n"},{"id":"467384","messageId":"7d86852c01513ee131bf993302416f4c881a0bc6.1668628303.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v2.git.1668628302.gitgitgadget@gmail.com","subject":"[PATCH v2 6/9] strbuf: introduce strbuf_strip_file_from_path()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-16T19:51:39Z","receivedAt":"2022-11-16T19:52:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe strbuf_parent_directory() method was added as a static method in\ncontrib/scalar by d0feac4e8c0 (scalar: 'register' sets recommended\nconfig and starts maintenance, 2021-12-03) and then removed in\n65f6a9eb0b9 (scalar: constrain enlistment search, 2022-08-18), but now\nthere is a need for a similar method in the bundle URI feature.\n\nRe-add the method, this time in strbuf.c, but with a new name:\nstrbuf_strip_file_from_path(). The method requirements are slightly\nmodified to allow a trailing slash, in which case nothing is done, which\nmakes the name change valuable. The return value is the number of bytes\nremoved.\n\nSigned-off-by: Derrick Stolee \n---\n strbuf.c | 9 +++++++++\n strbuf.h | 12 ++++++++++++\n 2 files changed, 21 insertions(+)\n\ndiff --git a/strbuf.c b/strbuf.c\nindex 0890b1405c5..8d1e2e8bb61 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -1200,3 +1200,12 @@ int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n \tfree(path2);\n \treturn res;\n }\n+\n+size_t strbuf_strip_file_from_path(struct strbuf *buf)\n+{\n+\tsize_t len = buf->len;\n+\tsize_t offset = offset_1st_component(buf->buf);\n+\tchar *path_sep = find_last_dir_sep(buf->buf + offset);\n+\tstrbuf_setlen(buf, path_sep ? path_sep - buf->buf + 1 : offset);\n+\treturn len - buf->len;\n+}\ndiff --git a/strbuf.h b/strbuf.h\nindex 76965a17d44..4822b713786 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -664,6 +664,18 @@ int launch_sequence_editor(const char *path, struct strbuf *buffer,\n int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n \t\t\t const char *const *env);\n \n+/*\n+ * Remove the filename from the provided path string. If the path\n+ * contains a trailing separator, then the path is considered a directory\n+ * and nothing is modified. Returns the number of characters removed from\n+ * the path.\n+ *\n+ * Examples:\n+ * - \"/path/to/file\" -> \"/path/to/\" (returns: 4)\n+ * - \"/path/to/dir/\" -> \"/path/to/dir/\" (returns: 0)\n+ */\n+size_t strbuf_strip_file_from_path(struct strbuf *buf);\n+\n void strbuf_add_lines(struct strbuf *sb,\n \t\t const char *prefix,\n \t\t const char *buf,\n-- \ngitgitgadget\n\n"},{"id":"467385","messageId":"933974689312bbb130236c81550ee3467f295a43.1668628303.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v2.git.1668628302.gitgitgadget@gmail.com","subject":"[PATCH v2 5/9] bundle-uri client: add boolean transfer.bundleURI setting","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-16T19:51:38Z","receivedAt":"2022-11-16T19:52:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nThe yet-to-be introduced client support for bundle-uri will always\nfall back on a full clone, but we'd still like to be able to ignore a\nserver's bundle-uri advertisement entirely.\n\nThe new transfer.bundleURI config option defaults to 'false', but a user\ncan set it to 'true' to enable checking for bundle URIs from the origin\nGit server using protocol v2.\n\nTo enable this setting by default in the correct tests, add a\nGIT_TEST_BUNDLE_URI environment variable.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Documentation/config/transfer.txt | 6 ++++++\n t/lib-t5730-protocol-v2-bundle-uri.sh | 3 +++\n transport.c | 10 +++++++---\n 3 files changed, 16 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/transfer.txt b/Documentation/config/transfer.txt\nindex 264812cca4d..c3ac767d1e4 100644\n--- a/Documentation/config/transfer.txt\n+++ b/Documentation/config/transfer.txt\n@@ -115,3 +115,9 @@ transfer.unpackLimit::\n transfer.advertiseSID::\n \tBoolean. When true, client and server processes will advertise their\n \tunique session IDs to their remote counterpart. Defaults to false.\n+\n+transfer.bundleURI::\n+\tWhen `true`, local `git clone` commands will request bundle\n+\tinformation from the remote server (if advertised) and download\n+\tbundles before continuing the clone through the Git protocol.\n+\tDefaults to `false`.\ndiff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh\nindex 000fcc5e20b..872bc39ad1b 100644\n--- a/t/lib-t5730-protocol-v2-bundle-uri.sh\n+++ b/t/lib-t5730-protocol-v2-bundle-uri.sh\n@@ -1,5 +1,8 @@\n # Included from t573*-protocol-v2-bundle-uri-*.sh\n \n+GIT_TEST_BUNDLE_URI=1\n+export GIT_TEST_BUNDLE_URI\n+\n T5730_PARENT=\n T5730_URI=\n T5730_BUNDLE_URI=\ndiff --git a/transport.c b/transport.c\nindex 86460f5be28..b33180226ae 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1538,6 +1538,7 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)\n \n int transport_get_remote_bundle_uri(struct transport *transport, int quiet)\n {\n+\tint value = 0;\n \tconst struct transport_vtable *vtable = transport->vtable;\n \n \t/* Check config only once. */\n@@ -1545,10 +1546,13 @@ int transport_get_remote_bundle_uri(struct transport *transport, int quiet)\n \t\treturn 0;\n \n \t/*\n-\t * This is intentionally below the transport.injectBundleURI,\n-\t * we want to be able to inject into protocol v0, or into the\n-\t * dialog of a server who doesn't support this.\n+\t * Don't use bundle-uri at all, if configured not to. Only proceed\n+\t * if GIT_TEST_BUNDLE_URI=1 or transfer.bundleURI=true.\n \t */\n+\tif (!git_env_bool(\"GIT_TEST_BUNDLE_URI\", 0) &&\n+\t (git_config_get_bool(\"transfer.bundleuri\", &value) || !value))\n+\t\treturn 0;\n+\n \tif (!vtable->get_bundle_uri) {\n \t\tif (quiet)\n \t\t\treturn -1;\n-- \ngitgitgadget\n\n"},{"id":"467386","messageId":"f254da46a2c2afa74c6fdfd8cc1e2110c70455a5.1668628303.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v2.git.1668628302.gitgitgadget@gmail.com","subject":"[PATCH v2 8/9] bundle-uri: download bundles from an advertised list","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-16T19:51:41Z","receivedAt":"2022-11-16T19:52:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe logic in fetch_bundle_uri() is useful for the --bundle-uri option of\n'git clone', but is not helpful when the clone operation discovers a\nlist of URIs from the bundle-uri protocol v2 command. To actually\ndownload and unbundle the advertised bundles, we need a different\nmechanism.\n\nCreate the new fetch_bundle_list() method which is very similar to\nfetch_bundle_uri() except that it relies on download_bundle_list()\ninstead of fetch_bundle_uri_internal(). The download_bundle_list()\nmethod will recursively call fetch_bundle_uri_internal() if any of the\nadvertised URIs serve a bundle list instead of a bundle. This will also\nfollow the bundle.list.mode setting from the input list: \"any\" will\ndownload only one such URI while \"all\" will download data from all of\nthe URIs.\n\nIn an identical way to fetch_bundle_uri(), the bundles are unbundled\nafter all of the bundle lists have been expanded and all necessary URIs.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 21 +++++++++++++++++++++\n bundle-uri.h | 11 +++++++++++\n 2 files changed, 32 insertions(+)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex ab91bb32e9b..5914d220c43 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -577,6 +577,27 @@ cleanup:\n \treturn result;\n }\n \n+int fetch_bundle_list(struct repository *r, const char *uri, struct bundle_list *list)\n+{\n+\tint result;\n+\tstruct bundle_list global_list;\n+\n+\tinit_bundle_list(&global_list);\n+\n+\t/* If a bundle is added to this global list, then it is required. */\n+\tglobal_list.mode = BUNDLE_MODE_ALL;\n+\n+\tif ((result = download_bundle_list(r, list, &global_list, 0)))\n+\t\tgoto cleanup;\n+\n+\tresult = unbundle_all_bundles(r, &global_list);\n+\n+cleanup:\n+\tfor_all_bundles_in_list(&global_list, unlink_bundle, NULL);\n+\tclear_bundle_list(&global_list);\n+\treturn result;\n+}\n+\n /**\n * API for serve.c.\n */\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 7905e56732c..a75b68d2f5a 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -102,6 +102,17 @@ int bundle_uri_parse_config_format(const char *uri,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n+/**\n+ * Given a bundle list that was already advertised (likely by the\n+ * bundle-uri protocol v2 verb) at the given uri, fetch and unbundle the\n+ * bundles according to the bundle strategy of that list.\n+ *\n+ * Returns non-zero if no bundle information is found at the given 'uri'.\n+ */\n+int fetch_bundle_list(struct repository *r,\n+\t\t const char *uri,\n+\t\t struct bundle_list *list);\n+\n /**\n * API for serve.c.\n */\n-- \ngitgitgadget\n\n"},{"id":"467387","messageId":"186e112d821d9a42ffbc3c8b46e24b2b4bb3dfe8.1668628303.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v2.git.1668628302.gitgitgadget@gmail.com","subject":"[PATCH v2 7/9] bundle-uri: allow relative URLs in bundle lists","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-16T19:51:40Z","receivedAt":"2022-11-16T19:52:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBundle providers may want to distribute that data across multiple CDNs.\nThis might require a change in the base URI, all the way to the domain\nname. If all bundles require an absolute URI in their 'uri' value, then\nevery push to a CDN would require altering the table of contents to\nmatch the expected domain and exact location within it.\n\nAllow a bundle list to specify a relative URI for the bundles.\nThis allows easier distribution of bundle data.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 16 ++++++++++-\n bundle-uri.h | 9 +++++++\n t/helper/test-bundle-uri.c | 2 ++\n t/t5750-bundle-uri-parse.sh | 54 +++++++++++++++++++++++++++++++++++++\n transport.c | 3 +++\n 5 files changed, 83 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 3469f1aaa98..ab91bb32e9b 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -7,6 +7,7 @@\n #include \"hashmap.h\"\n #include \"pkt-line.h\"\n #include \"config.h\"\n+#include \"remote.h\"\n \n static int compare_bundles(const void *hashmap_cmp_fn_data,\n \t\t\t const struct hashmap_entry *he1,\n@@ -49,6 +50,7 @@ void clear_bundle_list(struct bundle_list *list)\n \n \tfor_all_bundles_in_list(list, clear_remote_bundle_info, NULL);\n \thashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent);\n+\tfree(list->baseURI);\n }\n \n int for_all_bundles_in_list(struct bundle_list *list,\n@@ -163,7 +165,7 @@ static int bundle_list_update(const char *key, const char *value,\n \tif (!strcmp(subkey, \"uri\")) {\n \t\tif (bundle->uri)\n \t\t\treturn -1;\n-\t\tbundle->uri = xstrdup(value);\n+\t\tbundle->uri = relative_url(list->baseURI, value, NULL);\n \t\treturn 0;\n \t}\n \n@@ -190,6 +192,18 @@ int bundle_uri_parse_config_format(const char *uri,\n \t\t.error_action = CONFIG_ERROR_ERROR,\n \t};\n \n+\tif (!list->baseURI) {\n+\t\tstruct strbuf baseURI = STRBUF_INIT;\n+\t\tstrbuf_addstr(&baseURI, uri);\n+\n+\t\t/*\n+\t\t * If the URI does not end with a trailing slash, then\n+\t\t * remove the filename portion of the path. This is\n+\t\t * important for relative URIs.\n+\t\t */\n+\t\tstrbuf_strip_file_from_path(&baseURI);\n+\t\tlist->baseURI = strbuf_detach(&baseURI, NULL);\n+\t}\n \tresult = git_config_from_file_with_options(config_to_bundle_list,\n \t\t\t\t\t\t filename, list,\n \t\t\t\t\t\t &opts);\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 357111ecce8..7905e56732c 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -61,6 +61,15 @@ struct bundle_list {\n \tint version;\n \tenum bundle_list_mode mode;\n \tstruct hashmap bundles;\n+\n+\t/**\n+\t * The baseURI of a bundle_list is used as the base for any\n+\t * relative URIs advertised by the bundle list at that location.\n+\t *\n+\t * When the list is generated from a Git server, then use that\n+\t * server's location.\n+\t */\n+\tchar *baseURI;\n };\n \n void init_bundle_list(struct bundle_list *list);\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex ffb975b7b4f..5aa0b494ce3 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -40,6 +40,8 @@ static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mo\n \n \tinit_bundle_list(&list);\n \n+\tlist.baseURI = xstrdup(\"\");\n+\n \tswitch (mode) {\n \tcase KEY_VALUE_PAIRS:\n \t\tif (argc != 1)\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex c2fe3f9c5a5..ed5262a8d2b 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -30,6 +30,30 @@ test_expect_success 'bundle_uri_parse_line() just URIs' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'bundle_uri_parse_line(): relative URIs' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=bundle.bdl\n+\tbundle.two.uri=../bundle.bdl\n+\tbundle.three.uri=sub/dir/bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = /bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = /sub/dir/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' '\n \tcat >in <<-\\EOF &&\n \t=bogus-value\n@@ -136,6 +160,36 @@ test_expect_success 'parse config format: just URIs' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format: relative URIs' '\n+\tcat >in <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = ../bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = sub/dir/bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = /bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = /sub/dir/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config in >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n test_expect_success 'parse config format edge cases: empty key or value' '\n \tcat >in1 <<-\\EOF &&\n \t= bogus-value\ndiff --git a/transport.c b/transport.c\nindex b33180226ae..2c4ff0c2023 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1553,6 +1553,9 @@ int transport_get_remote_bundle_uri(struct transport *transport, int quiet)\n \t (git_config_get_bool(\"transfer.bundleuri\", &value) || !value))\n \t\treturn 0;\n \n+\tif (!transport->bundles->baseURI)\n+\t\ttransport->bundles->baseURI = xstrdup(transport->url);\n+\n \tif (!vtable->get_bundle_uri) {\n \t\tif (quiet)\n \t\t\treturn -1;\n-- \ngitgitgadget\n\n"},{"id":"467388","messageId":"cd906f6d9818a36cbdee896c1ede44d6de3f8990.1668628303.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v2.git.1668628302.gitgitgadget@gmail.com","subject":"[PATCH v2 4/9] bundle-uri: serve bundle.* keys from config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-16T19:51:37Z","receivedAt":"2022-11-16T19:52:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nImplement the \"bundle-uri\" protocol v2 capability by populating the\nkey=value packet lines from the local Git config. The list of bundles is\nprovided from the keys beginning with \"bundle.\".\n\nIn the future, we may want to filter this list to be more specific to\nthe exact known keys that the server intends to share, but for\nflexibility at the moment we will assume that the config values are\nwell-formed.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 16 +++++++++++-\n t/lib-t5730-protocol-v2-bundle-uri.sh | 35 +++++++++++++++++++++++++++\n 2 files changed, 50 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 2201b604b11..3469f1aaa98 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -585,6 +585,16 @@ cached:\n \treturn advertise_bundle_uri;\n }\n \n+static int config_to_packet_line(const char *key, const char *value, void *data)\n+{\n+\tstruct packet_reader *writer = data;\n+\n+\tif (!strncmp(key, \"bundle.\", 7))\n+\t\tpacket_write_fmt(writer->fd, \"%s=%s\", key, value);\n+\n+\treturn 0;\n+}\n+\n int bundle_uri_command(struct repository *r,\n \t\t struct packet_reader *request)\n {\n@@ -596,7 +606,11 @@ int bundle_uri_command(struct repository *r,\n \tif (request->status != PACKET_READ_FLUSH)\n \t\tdie(_(\"bundle-uri: expected flush after arguments\"));\n \n-\t/* TODO: Implement the communication */\n+\t/*\n+\t * Read all \"bundle.*\" config lines to the client as key=value\n+\t * packet lines.\n+\t */\n+\tgit_config(config_to_packet_line, &writer);\n \n \tpacket_writer_flush(&writer);\n \ndiff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh\nindex c327544641b..000fcc5e20b 100644\n--- a/t/lib-t5730-protocol-v2-bundle-uri.sh\n+++ b/t/lib-t5730-protocol-v2-bundle-uri.sh\n@@ -158,6 +158,8 @@ test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2\" '\n \t[bundle]\n \t\tversion = 1\n \t\tmode = all\n+\t[bundle \"only\"]\n+\t\turi = $T5730_BUNDLE_URI_ESCAPED\n \tEOF\n \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n \ttest-tool bundle-uri \\\n@@ -181,6 +183,39 @@ test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2 and ext\n \t[bundle]\n \t\tversion = 1\n \t\tmode = all\n+\t[bundle \"only\"]\n+\t\turi = $T5730_BUNDLE_URI_ESCAPED\n+\tEOF\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \\\n+\t\t\"$T5730_URI\" \\\n+\t\t>actual &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+\n+test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2 with list\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tbundle.bundle1.uri \"$T5730_BUNDLE_URI_ESCAPED-1.bdl\" &&\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tbundle.bundle2.uri \"$T5730_BUNDLE_URI_ESCAPED-2.bdl\" &&\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tbundle.bundle3.uri \"$T5730_BUNDLE_URI_ESCAPED-3.bdl\" &&\n+\n+\t# All data about bundle URIs\n+\tcat >expect <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"bundle1\"]\n+\t\turi = $T5730_BUNDLE_URI_ESCAPED-1.bdl\n+\t[bundle \"bundle2\"]\n+\t\turi = $T5730_BUNDLE_URI_ESCAPED-2.bdl\n+\t[bundle \"bundle3\"]\n+\t\turi = $T5730_BUNDLE_URI_ESCAPED-3.bdl\n \tEOF\n \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n \ttest-tool bundle-uri \\\n-- \ngitgitgadget\n\n"},{"id":"467389","messageId":"b62b4b1748194f0c7b81536701f15aa0df8e1d9b.1668628303.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v2.git.1668628302.gitgitgadget@gmail.com","subject":"[PATCH v2 9/9] clone: unbundle the advertised bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-16T19:51:42Z","receivedAt":"2022-11-16T19:52:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA previous change introduced the transport methods to acquire a bundle\nlist from the 'bundle-uri' protocol v2 command, when advertised _and_\nwhen the client has chosen to enable the feature.\n\nTeach Git to download and unbundle the data advertised by those bundles\nduring 'git clone'.\n\nAlso, since the --bundle-uri option exists, we do not want to mix the\nadvertised bundles with the user-specified bundles.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 26 +++++++++++++++++----\n t/t5601-clone.sh | 59 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 80 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 22b1e506452..09f10477ed6 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1267,11 +1267,27 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (refs)\n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n \n-\t/*\n-\t * Populate transport->got_remote_bundle_uri and\n-\t * transport->bundle_uri. We might get nothing.\n-\t */\n-\ttransport_get_remote_bundle_uri(transport, 1);\n+\tif (!bundle_uri) {\n+\t\t/*\n+\t\t* Populate transport->got_remote_bundle_uri and\n+\t\t* transport->bundle_uri. We might get nothing.\n+\t\t*/\n+\t\ttransport_get_remote_bundle_uri(transport, 1);\n+\n+\t\tif (transport->bundles &&\n+\t\t hashmap_get_size(&transport->bundles->bundles)) {\n+\t\t\t/* At this point, we need the_repository to match the cloned repo. */\n+\t\t\tif (repo_init(the_repository, git_dir, work_tree))\n+\t\t\t\twarning(_(\"failed to initialize the repo, skipping bundle URI\"));\n+\t\t\tif (fetch_bundle_list(the_repository,\n+\t\t\t\t\t remote->url[0],\n+\t\t\t\t\t transport->bundles))\n+\t\t\t\twarning(_(\"failed to fetch advertised bundles\"));\n+\t\t} else {\n+\t\t\tclear_bundle_list(transport->bundles);\n+\t\t\tFREE_AND_NULL(transport->bundles);\n+\t\t}\n+\t}\n \n \tif (mapped_refs) {\n \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 45f0803ed4d..d1d8139751e 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -795,6 +795,65 @@ test_expect_success 'reject cloning shallow repository using HTTP' '\n \tgit clone --no-reject-shallow $HTTPD_URL/smart/repo.git repo\n '\n \n+test_expect_success 'auto-discover bundle URI from HTTP clone' '\n+\ttest_when_finished rm -rf trace.txt repo2 \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" &&\n+\tgit -C src bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/everything.bundle\" --all &&\n+\tgit clone --bare --no-local src \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" &&\n+\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tuploadpack.advertiseBundleURIs true &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tbundle.version 1 &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tbundle.mode all &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tbundle.everything.uri \"$HTTPD_URL/everything.bundle\" &&\n+\n+\tGIT_TEST_BUNDLE_URI=1 \\\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n+\t\tgit -c protocol.version=2 clone \\\n+\t\t$HTTPD_URL/smart/repo2.git repo2 &&\n+\tcat >pattern <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/everything.bundle\"\\]\n+\tEOF\n+\tgrep -f pattern trace.txt\n+'\n+\n+test_expect_success 'auto-discover multiple bundles from HTTP clone' '\n+\ttest_when_finished rm -rf trace.txt repo3 \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" &&\n+\n+\ttest_commit -C src new &&\n+\tgit -C src bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/new.bundle\" HEAD~1..HEAD &&\n+\tgit clone --bare --no-local src \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" &&\n+\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tuploadpack.advertiseBundleURIs true &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.version 1 &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.mode all &&\n+\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.everything.uri \"$HTTPD_URL/everything.bundle\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.new.uri \"$HTTPD_URL/new.bundle\" &&\n+\n+\tGIT_TEST_BUNDLE_URI=1 \\\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n+\t\tgit -c protocol.version=2 clone \\\n+\t\t$HTTPD_URL/smart/repo3.git repo3 &&\n+\n+\t# We should fetch _both_ bundles\n+\tcat >pattern <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/everything.bundle\"\\]\n+\tEOF\n+\tgrep -f pattern trace.txt &&\n+\tcat >pattern <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/new.bundle\"\\]\n+\tEOF\n+\tgrep -f pattern trace.txt\n+'\n+\n # DO NOT add non-httpd-specific tests here, because the last part of this\n # test script is only executed when httpd is available and enabled.\n \n-- \ngitgitgadget\n"},{"id":"468135","messageId":"ca410bed-e8d1-415f-5235-b64fe18bed27@github.com","threadId":"58731","inReplyTo":"0d85aef965d62a684b25f388a946abd3c17809fc.1668628303.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/9] bundle-uri client: add minimal NOOP client","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-11-29T00:57:10Z","receivedAt":"2022-11-29T00:57:16Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n> From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n> \n> \n> Set up all the needed client parts of the \"bundle-uri\" protocol\n> extension, without actually doing anything with the bundle URIs.\n> \n> I.e. if the server says it supports \"bundle-uri\" we'll issue a\n> command=bundle-uri after command=ls-refs when we're cloning. We'll\n> parse the returned output using the code already tested for in\n> t5750-bundle-uri-parse.sh.\n> \n> What we aren't doing is actually acting on that data, i.e. downloading\n> the bundle(s) before we get to doing the command=fetch, and adjusting\n> our negotiation dialog appropriately. I'll do that in subsequent\n> commits.\n\nMakes sense.\n\n> \n> There's a question of what level of encapsulation we should use here,\n> I've opted to use connect.h in clone.c, but we could also e.g. make\n> transport_get_remote_refs() invoke this, i.e. make it implicitly get\n> the bundle-uri list for later steps.\n\nI'm not sure I follow what this sentence is saying. Looking at the\nimplementation below [1], you've added a call to\n'transport_get_remote_bundle_uri()' in 'clone.c', but that's defined in\n'transport.h' (which is already included in 'clone.c'). Why is 'connect.h'\nneeded at all?\n\n> \n> This approach means that we don't \"support\" this in \"git fetch\" for\n> now. I'm starting with the case of initial clones, although as noted\n> in preceding commits to the protocol documentation nothing about this\n> approach precludes getting bundles on incremental fetches.\n\nThis explanation seems more complicated than necessary. I think it's\nsufficient to say \"The no-op client is initially used only in 'clone' to\ntest the basic functionality. The bundle URI client will be integrated into\nfetch, pull, etc. in later patches\".\n\n> \n> For the t5732-protocol-v2-bundle-uri-http.sh it's not easy to set\n> environment variables for git-upload-pack (it's started by Apache), so\n> let's skip the test under T5730_HTTP, and add unused T5730_{FILE,GIT}\n> prerequisites for consistency and future use.\n\n\"skip the test\" doesn't explain *which* test is skipped (and it doesn't look\nlike you skip all of them). I think you're referring to \"bad client with\n$T5730_PROTOCOL:// using protocol v2\" specifically?\n\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/clone.c | 7 ++\n> bundle-uri.c | 4 +\n> connect.c | 47 ++++++++\n> remote.h | 5 +\n> t/lib-t5730-protocol-v2-bundle-uri.sh | 148 +++++++++++++++++++++++++\n> t/t5730-protocol-v2-bundle-uri-file.sh | 36 ++++++\n> t/t5731-protocol-v2-bundle-uri-git.sh | 17 +++\n> t/t5732-protocol-v2-bundle-uri-http.sh | 17 +++\n> transport-helper.c | 13 +++\n> transport-internal.h | 7 ++\n> transport.c | 55 +++++++++\n> transport.h | 19 ++++\n> 12 files changed, 375 insertions(+)\n> create mode 100644 t/lib-t5730-protocol-v2-bundle-uri.sh\n> create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh\n> create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh\n> create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh\n> \n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index 547d6464b3c..edf98295af2 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -27,6 +27,7 @@\n> #include \"iterator.h\"\n> #include \"sigchain.h\"\n> #include \"branch.h\"\n> +#include \"connect.h\"\n> #include \"remote.h\"\n> #include \"run-command.h\"\n> #include \"connected.h\"\n> @@ -1266,6 +1267,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> \tif (refs)\n> \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n> \n> +\t/*\n> +\t * Populate transport->got_remote_bundle_uri and\n> +\t * transport->bundle_uri. We might get nothing.\n> +\t */\n> +\ttransport_get_remote_bundle_uri(transport);\n\n[1] \n\n> +\n> \tif (mapped_refs) {\n> \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n> \n> diff --git a/bundle-uri.c b/bundle-uri.c\n> index 32022595964..2201b604b11 100644\n> --- a/bundle-uri.c\n> +++ b/bundle-uri.c\n> @@ -571,6 +571,10 @@ int bundle_uri_advertise(struct repository *r, struct strbuf *value)\n> {\n> \tstatic int advertise_bundle_uri = -1;\n> \n> +\tif (value &&\n> +\t git_env_bool(\"GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE\", 0))\n> +\t\tstrbuf_addstr(value, \"test-unknown-capability-value\");\n\nIt looks like 'GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE' is being used\nto \"mock\" server responses to test certain behavior on the client side. I'm\nsomewhat uncomfortable with how this mixes test-specific code paths with\napplication code, and AFAICT nothing similar is done for other\nadvertise/command functions in protocol v2. Is there a way to set up tests\nto intercept the client requests and customize the response? \n\n> +\n> \tif (advertise_bundle_uri != -1)\n> \t\tgoto cached;\n> \n> diff --git a/connect.c b/connect.c\n> index 5ea53deda23..d39effb7492 100644\n> --- a/connect.c\n> +++ b/connect.c\n> @@ -15,6 +15,7 @@\n> #include \"version.h\"\n> #include \"protocol.h\"\n> #include \"alias.h\"\n> +#include \"bundle-uri.h\"\n> \n> static char *server_capabilities_v1;\n> static struct strvec server_capabilities_v2 = STRVEC_INIT;\n> @@ -491,6 +492,52 @@ static void send_capabilities(int fd_out, struct packet_reader *reader)\n> \t}\n> }\n> \n> +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\n> +\t\t\t struct bundle_list *bundles, int stateless_rpc)\n> +{\n> +\tint line_nr = 1;\n> +\n> +\t/* Assert bundle-uri support */\n> +\tserver_supports_v2(\"bundle-uri\", 1);\n> +\n> +\t/* (Re-)send capabilities */\n> +\tsend_capabilities(fd_out, reader);\n> +\n> +\t/* Send command */\n> +\tpacket_write_fmt(fd_out, \"command=bundle-uri\\n\");\n> +\tpacket_delim(fd_out);\n> +\n> +\t/* Send options */\n> +\tif (git_env_bool(\"GIT_TEST_PROTOCOL_BAD_BUNDLE_URI\", 0))\n> +\t\tpacket_write_fmt(fd_out, \"test-bad-client\\n\");\n\nSame comment as on 'GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE'. There's\nno precedent that I can find for a test variable like this in 'connect.c', and\n\"in the middle of client code\" doesn't seem like an ideal place for it. \n\nIf there really isn't another way of doing this, could the addition of these\n'GIT_TEST' variables and their associated tests be split out into a\ndedicated commit? That would at least separate the test code paths from the\napplication code in the commit history.\n\n> +\tpacket_flush(fd_out);\n> +\n> +\t/* Process response from server */\n> +\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n> +\t\tconst char *line = reader->line;\n> +\t\tline_nr++;\n> +\n> +\t\tif (!bundle_uri_parse_line(bundles, line))\n> +\t\t\tcontinue;\n> +\n> +\t\treturn error(_(\"error on bundle-uri response line %d: %s\"),\n> +\t\t\t line_nr, line);\n> +\t}\n> +\n> +\tif (reader->status != PACKET_READ_FLUSH)\n> +\t\treturn error(_(\"expected flush after bundle-uri listing\"));\n> +\n> +\t/*\n> +\t * Might die(), but obscure enough that that's OK, e.g. in\n> +\t * serve.c we'll call BUG() on its equivalent (the\n> +\t * PACKET_READ_RESPONSE_END check).\n> +\t */\n> +\tcheck_stateless_delimiter(stateless_rpc, reader,\n> +\t\t\t\t _(\"expected response end packet after ref listing\"));\n> +\n> +\treturn 0;\n\nThe rest of this looks fine to me.\n\n> +}\n> +\n> struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n> \t\t\t struct ref **list, int for_push,\n> \t\t\t struct transport_ls_refs_options *transport_options,\n> diff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh\n> new file mode 100644\n> index 00000000000..27294e9c976\n> --- /dev/null\n> +++ b/t/lib-t5730-protocol-v2-bundle-uri.sh\n\nnit: this set of tests is used in more than just 't5730', so the name is\nsomewhat misleading. It also seems a bit overly-specific to include\n\"protocol-v2\" in the filename (the tests themselves mention \"protocol v2\"\nwhen it's relevant). What about something like 'lib-proto-bundle-uri.sh'\n(using \"proto\" to mimic 'lib-proto-disable.sh')?\n\n> @@ -0,0 +1,148 @@\n> +# Included from t573*-protocol-v2-bundle-uri-*.sh\n> +\n> +T5730_PARENT=\n> +T5730_URI=\n> +T5730_BUNDLE_URI=\n\nSimilar to the filename nit - these variables are used in tests outside of\n't5730', so their names are not quite accurate to their usage. \n\n> +# Poor man's URI escaping. Good enough for the test suite whose trash\n> +# directory has a space in it. See 93c3fcbe4d4 (git-svn: attempt to\n> +# mimic SVN 1.7 URL canonicalization, 2012-07-28) for prior art.\n> +test_uri_escape() {\n> +\tsed 's/ /%20/g'\n> +}\n\nThis is a good opportunity to unify on a single implementation rather than\nto have multiple similar ones floating around. Can this be moved into\n'test-lib.sh' (or 'test-lib-functions.sh'?), with 't9119' and 't9120'\nupdated to use the new 'test_uri_escape()'?\n\n> diff --git a/t/t5730-protocol-v2-bundle-uri-file.sh b/t/t5730-protocol-v2-bundle-uri-file.sh\n> new file mode 100755\n> index 00000000000..89203d3a23c\n> --- /dev/null\n> +++ b/t/t5730-protocol-v2-bundle-uri-file.sh\n> @@ -0,0 +1,36 @@\n> +#!/bin/sh\n> +\n> +test_description=\"Test bundle-uri with protocol v2 and 'file://' transport\"\n> +\n> +TEST_NO_CREATE_REPO=1\n> +\n> +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n> +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n> +\n> +. ./test-lib.sh\n> +\n> +# Test protocol v2 with 'file://' transport\n> +#\n> +T5730_PROTOCOL=file\n> +. \"$TEST_DIRECTORY\"/lib-t5730-protocol-v2-bundle-uri.sh\n> +\n> +test_expect_success \"unknown capability value with $T5730_PROTOCOL:// using protocol v2\" '\n\nWhy is this test only run for the 'file://' transport protocol? And, why\nisn't it in 'lib-t5730-protocol-v2-bundle-uri.sh'? If nothing else, moving\nthis test to that file (even if it needs to be conditional on a specific\nprotocol) puts the '$T5730_PARENT', '$T5730_BUNDLE_URI_ESCAPED' and\n'$T5730_URI' variables in scope for better readability.\n\n> +\ttest_when_finished \"rm -f log\" &&\n> +\n> +\ttest_config -C \"$T5730_PARENT\" \\\n> +\t\tuploadpack.bundleURI \"$T5730_BUNDLE_URI_ESCAPED\" &&\n> +\n> +\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n> +\tGIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE=true \\\n> +\tgit \\\n> +\t\t-c protocol.version=2 \\\n> +\t\tls-remote --symref \"$T5730_URI\" \\\n> +\t\t>actual 2>err &&\n\nThis test never does anything with the content of 'actual' or 'err'. Should\nit? If not, they probably shouldn't be redirected from stdout/stderr, since\nthe messages might be valuable when debugging.\n\n> +\n> +\t# Server responded using protocol v2\n> +\tgrep \"< version 2\" log &&\n> +\n> +\tgrep \"> bundle-uri=test-unknown-capability-value\" log\n> +'\n> +\n> +test_done\n> diff --git a/transport-helper.c b/transport-helper.c\n> index e95267a4ab5..3ea7c2bb5ad 100644\n> --- a/transport-helper.c\n> +++ b/transport-helper.c\n> @@ -1267,9 +1267,22 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n> \treturn ret;\n> }\n> \n> +static int get_bundle_uri(struct transport *transport)\n> +{\n> +\tget_helper(transport);\n> +\n> +\tif (process_connect(transport, 0)) {\n> +\t\tdo_take_over(transport);\n> +\t\treturn transport->vtable->get_bundle_uri(transport);\n> +\t}\n> +\n> +\treturn -1;\n> +}\n> +\n> static struct transport_vtable vtable = {\n> \t.set_option\t= set_helper_option,\n> \t.get_refs_list\t= get_refs_list,\n> +\t.get_bundle_uri = get_bundle_uri,\n> \t.fetch_refs\t= fetch_refs,\n> \t.push_refs\t= push_refs,\n> \t.connect\t= connect_helper,\n> diff --git a/transport-internal.h b/transport-internal.h\n> index c4ca0b733ac..90ea749e5cf 100644\n> --- a/transport-internal.h\n> +++ b/transport-internal.h\n> @@ -26,6 +26,13 @@ struct transport_vtable {\n> \tstruct ref *(*get_refs_list)(struct transport *transport, int for_push,\n> \t\t\t\t struct transport_ls_refs_options *transport_options);\n> \n> +\t/**\n> +\t * Populates the remote side's bundle-uri under protocol v2,\n> +\t * if the \"bundle-uri\" capability was advertised. Returns 0 if\n> +\t * OK, negative values on error.\n\nDouble-checked the call stack to make sure this is true, and it is (the\nreturn value is always either a hardcoded '0' or 'error()'.\n\n> +\t */\n> +\tint (*get_bundle_uri)(struct transport *transport);\n> +\n> \t/**\n> \t * Fetch the objects for the given refs. Note that this gets\n> \t * an array, and should ignore the list structure.\n> diff --git a/transport.c b/transport.c\n> index e7b97194c10..a020adc1f56 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -902,6 +922,7 @@ static int disconnect_git(struct transport *transport)\n> \n> static struct transport_vtable taken_over_vtable = {\n> \t.get_refs_list\t= get_refs_via_connect,\n> +\t.get_bundle_uri = get_bundle_uri,\n> \t.fetch_refs\t= fetch_refs_via_pack,\n> \t.push_refs\t= git_transport_push,\n> \t.disconnect\t= disconnect_git\n> @@ -1054,6 +1075,7 @@ static struct transport_vtable bundle_vtable = {\n> \n> static struct transport_vtable builtin_smart_vtable = {\n> \t.get_refs_list\t= get_refs_via_connect,\n> +\t.get_bundle_uri = get_bundle_uri,\n> \t.fetch_refs\t= fetch_refs_via_pack,\n> \t.push_refs\t= git_transport_push,\n> \t.connect\t= connect_git,\n\nI think I follow what this is doing (if I'm reading correctly) - add\n'get_bundle_uri' to all of the vtables (including in 'transport-helper.c')\n*except* 'bundle_vtable', since we aren't requesting bundle URIs from an\nalready-identified bundle URI.\n\n> @@ -1482,6 +1507,34 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)\n> \treturn rc;\n> }\n> \n> +int transport_get_remote_bundle_uri(struct transport *transport)\n> +{\n> +\tconst struct transport_vtable *vtable = transport->vtable;\n> +\n> +\t/* Check config only once. */\n> +\tif (transport->got_remote_bundle_uri++)\n> +\t\treturn 0;\n\nWe're only ever going to read the bundle list once per command (or, at least\nonce per 'transport' instance), so if 'transport_get_remote_bundle_uri()'\nhas already been called, we can safely assume the correct results (if any)\nare in the 'transport' structure.\n\n> +\n> +\t/*\n> +\t * \"Support\" protocol v0 and v2 without bundle-uri support by\n> +\t * silently degrading to a NOOP.\n> +\t */\n> +\tif (!server_supports_v2(\"bundle-uri\", 0))\n> +\t\treturn 0;\n> +\n> +\t/*\n> +\t * This is intentionally below the transport.injectBundleURI,\n> +\t * we want to be able to inject into protocol v0, or into the\n> +\t * dialog of a server who doesn't support this.\n> +\t */\n> +\tif (!vtable->get_bundle_uri)\n> +\t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n> +\n> +\tif (vtable->get_bundle_uri(transport) < 0)\n\nAs you noted earlier, 'get_bundle_uri()' always returns a value <= 0, so\nthis check works.\n\n> +\t\treturn error(_(\"could not retrieve server-advertised bundle-uri list\"));\n> +\treturn 0;\n> +}\n> +\n> void transport_unlock_pack(struct transport *transport, unsigned int flags)\n> {\n> \tint in_signal_handler = !!(flags & TRANSPORT_UNLOCK_PACK_IN_SIGNAL_HANDLER)\n"},{"id":"468136","messageId":"84c07e84-0805-6163-d77b-cb9f42db402e@github.com","threadId":"58731","inReplyTo":"c3269a24b5780023cbb4d173cb9cfb10c5a4b0d8.1668628303.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/9] bundle-uri client: add helper for testing server","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-11-29T00:59:52Z","receivedAt":"2022-11-29T00:59:59Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n> From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n> \n> \n> Add a 'test-tool bundle-uri ls-remote' command. This is a thin wrapper\n> for issuing protocol v2 \"bundle-uri\" commands to a server, and to the\n> parsing routines in bundle-uri.c.\n> \n> Since in the \"git clone\" case we'll have already done the handshake(),\n> but not here, introduce a \"got_advertisement\" state along with\n> \"got_remote_heads\". It seems to me that the \"got_remote_heads\" is\n> badly named in the first place, and the whole logic of eagerly getting\n> ls-refs on handshake() or not could be refactored somewhat, but let's\n> not do that now, and instead just add another self-documenting state\n> variable.\n\nMaybe I'm missing something, but why not just rename 'got_remote_heads' to\nsomething like 'finished_handshake' rather than adding 'got_advertisement'\n(since, AFAICT, it's always identical in value to 'got_remote_heads').\n\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Derrick Stolee \n\nThis commit also introduces the 'quiet' flag to\n'transport_get_remote_bundle_uri()', but there's no mention in the commit\nmessage. The message also doesn't explain the changes to existing tests\n(adding 'bundle.*' settings, swapping out 'git ls-remote' for the new\n'test-tool bundle-uri ls-remote' in existing tests, etc.). I think these are\nall relevant to fully understanding the patch, so could you mention them in\nyour next reroll?\n\n> ---\n> builtin/clone.c | 2 +-\n> t/helper/test-bundle-uri.c | 46 +++++++++++++++++++\n> t/lib-t5730-protocol-v2-bundle-uri.sh | 63 ++++++++++++++++++++++-----\n> transport.c | 43 ++++++++++++++----\n> transport.h | 6 ++-\n> 5 files changed, 139 insertions(+), 21 deletions(-)\n> \n> diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\n> index 25afd393428..ffb975b7b4f 100644\n> --- a/t/helper/test-bundle-uri.c\n> +++ b/t/helper/test-bundle-uri.c\n> @@ -88,6 +132,8 @@ int cmd__bundle_uri(int argc, const char **argv)\n> \t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS);\n> \tif (!strcmp(argv[1], \"parse-config\"))\n> \t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE);\n> +\tif (!strcmp(argv[1], \"ls-remote\"))\n> +\t\treturn cmd_ls_remote(argc - 1, argv + 1);\n\nWith this helper being added, I'm not sure if/why 'clone' was needed to test\nthe bundle URIs in patch 2 (I assumed integrating with a command was the\nonly way to test it, which is why I didn't mention this in my review [1]).\nIn the spirit of having commits avoid \"doing more than one thing\" could\nthese patches be reorganized into something like:\n\n1. Add the no-op client & some basic tests around fetching the bundle URI\n list using this test helper.\n2. Add the 'transport_get_remote_bundle_uri()' call to 'clone()' with\n clone-specific tests.\n\nIt probably wouldn't make the patches much shorter, but it would help avoid\nthe churn of test changes & changing assumptions around 'quiet' &\n'got_advertisement' in this patch.\n\n[1] https://lore.kernel.org/git/ca410bed-e8d1-415f-5235-b64fe18bed27@github.com/\n\n> \terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n> \n> usage:\n> diff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh\n> index 27294e9c976..c327544641b 100644\n> --- a/t/lib-t5730-protocol-v2-bundle-uri.sh\n> +++ b/t/lib-t5730-protocol-v2-bundle-uri.sh\n> @@ -34,7 +34,9 @@ esac\n> test_expect_success \"setup protocol v2 $T5730_PROTOCOL:// tests\" '\n> \tgit init \"$T5730_PARENT\" &&\n> \ttest_commit -C \"$T5730_PARENT\" one &&\n> -\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs true\n> +\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs true &&\n> +\tgit -C \"$T5730_PARENT\" config bundle.version 1 &&\n> +\tgit -C \"$T5730_PARENT\" config bundle.mode all\n\nWhy are these config settings added here? I don't see them used anywhere?\n\n> diff --git a/transport.c b/transport.c\n> index a020adc1f56..86460f5be28 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -371,6 +373,33 @@ static int get_bundle_uri(struct transport *transport)\n> \t\tinit_bundle_list(transport->bundles);\n> \t}\n> \n> +\tif (!data->got_advertisement) {\n> +\t\tstruct ref *refs;\n> +\t\tstruct git_transport_data *data = transport->data;\n> +\t\tenum protocol_version version;\n> +\n> +\t\trefs = handshake(transport, 0, NULL, 0);\n> +\t\tversion = data->version;\n> +\n> +\t\tswitch (version) {\n> +\t\tcase protocol_v2:\n> +\t\t\tassert(!refs);\n> +\t\t\tbreak;\n> +\t\tcase protocol_v0:\n> +\t\tcase protocol_v1:\n> +\t\tcase protocol_unknown_version:\n> +\t\t\tassert(refs);\n> +\t\t\tbreak;\n\nWhy were these 'refs' assertions added? What are they intended to validate?\n\n> +\t\t}\n> +\t}\n> +\n> +\t/*\n> +\t * \"Support\" protocol v0 and v2 without bundle-uri support by\n> +\t * silently degrading to a NOOP.\n> +\t */\n> +\tif (!server_supports_v2(\"bundle-uri\", 0))\n> +\t\treturn 0;\n\nI was originally confused as to why this was moved out of\n'transport_get_remote_bundle_uri()', but it looks like the answer is \"we\nwere previously relying on the handshake being done by the time we called\n'transport_get_remote_bundle_uri()', but we can't anymore.\"\n\n> +\n> \tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n> \t\t\t PACKET_READ_CHOMP_NEWLINE |\n> \t\t\t PACKET_READ_GENTLE_ON_EOF);\n"},{"id":"468137","messageId":"de6dc8f5-4750-db83-e58d-507aaa868ca4@github.com","threadId":"58731","inReplyTo":"cd906f6d9818a36cbdee896c1ede44d6de3f8990.1668628303.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/9] bundle-uri: serve bundle.* keys from config","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-11-29T01:00:56Z","receivedAt":"2022-11-29T01:01:02Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> Implement the \"bundle-uri\" protocol v2 capability by populating the\n> key=value packet lines from the local Git config. The list of bundles is\n> provided from the keys beginning with \"bundle.\".\n> \n> In the future, we may want to filter this list to be more specific to\n> the exact known keys that the server intends to share, but for\n> flexibility at the moment we will assume that the config values are\n> well-formed.\n\nThis patch looks good - the implementation is pretty straightforward (\"send\na config value if its key matches 'bundle.*'), and the tests cover both\nsingle and multiple bundle URIs returned by the server.\n\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> bundle-uri.c | 16 +++++++++++-\n> t/lib-t5730-protocol-v2-bundle-uri.sh | 35 +++++++++++++++++++++++++++\n> 2 files changed, 50 insertions(+), 1 deletion(-)\n> \n> diff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh\n> index c327544641b..000fcc5e20b 100644\n> --- a/t/lib-t5730-protocol-v2-bundle-uri.sh\n> +++ b/t/lib-t5730-protocol-v2-bundle-uri.sh\n> @@ -158,6 +158,8 @@ test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2\" '\n> \t[bundle]\n> \t\tversion = 1\n> \t\tmode = all\n> +\t[bundle \"only\"]\n> +\t\turi = $T5730_BUNDLE_URI_ESCAPED\n\n\nAh, okay, this explains why the 'bundle.only.uri' config was added to the\ntest in the last patch [1]. But, if the config is only being served in this\npatch, shouldn't that test change be moved to this patch? \n\n[1] https://lore.kernel.org/git/c3269a24b5780023cbb4d173cb9cfb10c5a4b0d8.1668628303.git.gitgitgadget@gmail.com/\n\n> \tEOF\n> \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n> \ttest-tool bundle-uri \\\n"},{"id":"468138","messageId":"4ec06afa-d804-16b1-f140-dfd9ebd912e5@github.com","threadId":"58731","inReplyTo":"933974689312bbb130236c81550ee3467f295a43.1668628303.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 5/9] bundle-uri client: add boolean transfer.bundleURI setting","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-11-29T01:03:13Z","receivedAt":"2022-11-29T01:03:26Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n> From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n> \n> \n> The yet-to-be introduced client support for bundle-uri will always\n> fall back on a full clone, but we'd still like to be able to ignore a\n> server's bundle-uri advertisement entirely.\n> \n> The new transfer.bundleURI config option defaults to 'false', but a user\n> can set it to 'true' to enable checking for bundle URIs from the origin\n> Git server using protocol v2.\n\nThanks for adding this, an \"opt-in\" approach seems reasonable for\nintroducing this feature.\n\n> \n> To enable this setting by default in the correct tests, add a\n> GIT_TEST_BUNDLE_URI environment variable.\n\nThis makes sense. I'm less concerned with this environment variable than\nthose in patch 2 [1], since it's in line with the test variables that\nenable/disable whole features ('GIT_TEST_SPLIT_INDEX',\n'GIT_TEST_COMMIT_GRAPH', etc.). \n\nThe only thing feedback I can think of would be that this patch could be\nmoved to earlier in the series (that is, immediately after creating\n'transport_get_remote_bundle_uri()'). That said, I don't feel strongly\neither way.\n\n[1] https://lore.kernel.org/git/ca410bed-e8d1-415f-5235-b64fe18bed27@github.com/\n\n> \n> Co-authored-by: Derrick Stolee \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Derrick Stolee \n\nThe implementation and documentation below align with the commit message.\nLooks good!\n\n> ---\n> Documentation/config/transfer.txt | 6 ++++++\n> t/lib-t5730-protocol-v2-bundle-uri.sh | 3 +++\n> transport.c | 10 +++++++---\n> 3 files changed, 16 insertions(+), 3 deletions(-)\n> \n> diff --git a/Documentation/config/transfer.txt b/Documentation/config/transfer.txt\n> index 264812cca4d..c3ac767d1e4 100644\n> --- a/Documentation/config/transfer.txt\n> +++ b/Documentation/config/transfer.txt\n> @@ -115,3 +115,9 @@ transfer.unpackLimit::\n> transfer.advertiseSID::\n> \tBoolean. When true, client and server processes will advertise their\n> \tunique session IDs to their remote counterpart. Defaults to false.\n> +\n> +transfer.bundleURI::\n> +\tWhen `true`, local `git clone` commands will request bundle\n> +\tinformation from the remote server (if advertised) and download\n> +\tbundles before continuing the clone through the Git protocol.\n> +\tDefaults to `false`.\n> diff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh\n> index 000fcc5e20b..872bc39ad1b 100644\n> --- a/t/lib-t5730-protocol-v2-bundle-uri.sh\n> +++ b/t/lib-t5730-protocol-v2-bundle-uri.sh\n> @@ -1,5 +1,8 @@\n> # Included from t573*-protocol-v2-bundle-uri-*.sh\n> \n> +GIT_TEST_BUNDLE_URI=1\n> +export GIT_TEST_BUNDLE_URI\n> +\n> T5730_PARENT=\n> T5730_URI=\n> T5730_BUNDLE_URI=\n> diff --git a/transport.c b/transport.c\n> index 86460f5be28..b33180226ae 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -1538,6 +1538,7 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)\n> \n> int transport_get_remote_bundle_uri(struct transport *transport, int quiet)\n> {\n> +\tint value = 0;\n> \tconst struct transport_vtable *vtable = transport->vtable;\n> \n> \t/* Check config only once. */\n> @@ -1545,10 +1546,13 @@ int transport_get_remote_bundle_uri(struct transport *transport, int quiet)\n> \t\treturn 0;\n> \n> \t/*\n> -\t * This is intentionally below the transport.injectBundleURI,\n> -\t * we want to be able to inject into protocol v0, or into the\n> -\t * dialog of a server who doesn't support this.\n> +\t * Don't use bundle-uri at all, if configured not to. Only proceed\n> +\t * if GIT_TEST_BUNDLE_URI=1 or transfer.bundleURI=true.\n> \t */\n> +\tif (!git_env_bool(\"GIT_TEST_BUNDLE_URI\", 0) &&> +\t (git_config_get_bool(\"transfer.bundleuri\", &value) || !value))\n> +\t\treturn 0;\n> +\n> \tif (!vtable->get_bundle_uri) {\n> \t\tif (quiet)\n> \t\t\treturn -1;\n\n"},{"id":"468139","messageId":"be89bbec-1a8b-1337-a5f7-f43420da64de@github.com","threadId":"58731","inReplyTo":"7d86852c01513ee131bf993302416f4c881a0bc6.1668628303.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 6/9] strbuf: introduce strbuf_strip_file_from_path()","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-11-29T01:03:32Z","receivedAt":"2022-11-29T01:03:43Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The strbuf_parent_directory() method was added as a static method in\n> contrib/scalar by d0feac4e8c0 (scalar: 'register' sets recommended\n> config and starts maintenance, 2021-12-03) and then removed in\n> 65f6a9eb0b9 (scalar: constrain enlistment search, 2022-08-18), but now\n> there is a need for a similar method in the bundle URI feature.\n> \n> Re-add the method, this time in strbuf.c, but with a new name:\n> strbuf_strip_file_from_path(). The method requirements are slightly\n> modified to allow a trailing slash, in which case nothing is done, which\n> makes the name change valuable. The return value is the number of bytes\n> removed.\n\n*Extremely* minor point, but why return anything at all? The call in the\nnext patch doesn't use the return value, and some similar-in-spirit 'strbuf'\nfunctions (like 'strbuf_trim()') return nothing. \n\nI don't think this is worth changing if you can imagine using that return\nvalue for something eventually; just wanted to point it out as something to\n(optionally) consider if you re-roll for something else anyway.\n\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> strbuf.c | 9 +++++++++\n> strbuf.h | 12 ++++++++++++\n> 2 files changed, 21 insertions(+)\n> \n> diff --git a/strbuf.c b/strbuf.c\n> index 0890b1405c5..8d1e2e8bb61 100644\n> --- a/strbuf.c\n> +++ b/strbuf.c\n> @@ -1200,3 +1200,12 @@ int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n> \tfree(path2);\n> \treturn res;\n> }\n> +\n> +size_t strbuf_strip_file_from_path(struct strbuf *buf)\n> +{\n> +\tsize_t len = buf->len;\n> +\tsize_t offset = offset_1st_component(buf->buf);\n> +\tchar *path_sep = find_last_dir_sep(buf->buf + offset);\n> +\tstrbuf_setlen(buf, path_sep ? path_sep - buf->buf + 1 : offset);\n> +\treturn len - buf->len;\n> +}\n> diff --git a/strbuf.h b/strbuf.h\n> index 76965a17d44..4822b713786 100644\n> --- a/strbuf.h\n> +++ b/strbuf.h\n> @@ -664,6 +664,18 @@ int launch_sequence_editor(const char *path, struct strbuf *buffer,\n> int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n> \t\t\t const char *const *env);\n> \n> +/*\n> + * Remove the filename from the provided path string. If the path\n> + * contains a trailing separator, then the path is considered a directory\n> + * and nothing is modified. Returns the number of characters removed from\n> + * the path.\n> + *\n> + * Examples:\n> + * - \"/path/to/file\" -> \"/path/to/\" (returns: 4)\n> + * - \"/path/to/dir/\" -> \"/path/to/dir/\" (returns: 0)\n> + */\n> +size_t strbuf_strip_file_from_path(struct strbuf *buf);\n> +\n> void strbuf_add_lines(struct strbuf *sb,\n> \t\t const char *prefix,\n> \t\t const char *buf,\n\n"},{"id":"468142","messageId":"06e45662-699b-52f9-2fdf-6e621444f446@github.com","threadId":"58731","inReplyTo":"186e112d821d9a42ffbc3c8b46e24b2b4bb3dfe8.1668628303.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 7/9] bundle-uri: allow relative URLs in bundle lists","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-11-29T01:25:38Z","receivedAt":"2022-11-29T01:26:37Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> Bundle providers may want to distribute that data across multiple CDNs.\n> This might require a change in the base URI, all the way to the domain\n> name. If all bundles require an absolute URI in their 'uri' value, then\n> every push to a CDN would require altering the table of contents to\n> match the expected domain and exact location within it.\n> \n> Allow a bundle list to specify a relative URI for the bundles.\n> This allows easier distribution of bundle data.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> bundle-uri.c | 16 ++++++++++-\n> bundle-uri.h | 9 +++++++\n> t/helper/test-bundle-uri.c | 2 ++\n> t/t5750-bundle-uri-parse.sh | 54 +++++++++++++++++++++++++++++++++++++\n> transport.c | 3 +++\n> 5 files changed, 83 insertions(+), 1 deletion(-)\n> \n> diff --git a/bundle-uri.c b/bundle-uri.c\n> @@ -190,6 +192,18 @@ int bundle_uri_parse_config_format(const char *uri,\n> \t\t.error_action = CONFIG_ERROR_ERROR,\n> \t};\n> \n> +\tif (!list->baseURI) {\n> +\t\tstruct strbuf baseURI = STRBUF_INIT;\n> +\t\tstrbuf_addstr(&baseURI, uri);\n> +\n> +\t\t/*\n> +\t\t * If the URI does not end with a trailing slash, then\n> +\t\t * remove the filename portion of the path. This is\n> +\t\t * important for relative URIs.\n> +\t\t */\n> +\t\tstrbuf_strip_file_from_path(&baseURI);\n> +\t\tlist->baseURI = strbuf_detach(&baseURI, NULL);\n\nIs the 'baseURI' is set to the URI of the first bundle (ordered by hash)? If\ndata is distributed across multiple CDNs, couldn't this be a suboptimal\nchoice? For example, if the first bundle is on 'A.com', but every other\nbundle is on 'B.org'?\n\n> +\t}\n> \tresult = git_config_from_file_with_options(config_to_bundle_list,\n> \t\t\t\t\t\t filename, list,\n> \t\t\t\t\t\t &opts);\n> diff --git a/bundle-uri.h b/bundle-uri.h\n> index 357111ecce8..7905e56732c 100644\n> --- a/bundle-uri.h\n> +++ b/bundle-uri.h\n> @@ -61,6 +61,15 @@ struct bundle_list {\n> \tint version;\n> \tenum bundle_list_mode mode;\n> \tstruct hashmap bundles;\n> +\n> +\t/**\n> +\t * The baseURI of a bundle_list is used as the base for any\n> +\t * relative URIs advertised by the bundle list at that location.\n> +\t *\n> +\t * When the list is generated from a Git server, then use that\n> +\t * server's location.\n\nHmmm, I think I'm missing something with my earlier comment. I thought the\n'uri' argument to 'bundle_uri_parse_config_format()' was an individual\nbundle's URI? What's the \"server's location\" in this context?\n\n> +\t */\n> +\tchar *baseURI;\n> };\n> \n> void init_bundle_list(struct bundle_list *list);\n> diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\n> index ffb975b7b4f..5aa0b494ce3 100644\n> --- a/t/helper/test-bundle-uri.c\n> +++ b/t/helper/test-bundle-uri.c\n> @@ -40,6 +40,8 @@ static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mo\n> \n> \tinit_bundle_list(&list);\n> \n> +\tlist.baseURI = xstrdup(\"\");\n\nUsing a hardcoded value here leads to pretty different behavior in\n'test-bundle-uri.c' vs. starting with an unset 'list.baseURI' in something\nlike 'clone'. Why does this need to be set to '' for the tests?\n\n> +\n> \tswitch (mode) {\n> \tcase KEY_VALUE_PAIRS:\n> \t\tif (argc != 1)\n> diff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\n> index c2fe3f9c5a5..ed5262a8d2b 100755\n> --- a/t/t5750-bundle-uri-parse.sh\n> +++ b/t/t5750-bundle-uri-parse.sh\n> @@ -30,6 +30,30 @@ test_expect_success 'bundle_uri_parse_line() just URIs' '\n> \ttest_cmp_config_output expect actual\n> '\n> \n> +test_expect_success 'bundle_uri_parse_line(): relative URIs' '\n> +\tcat >in <<-\\EOF &&\n> +\tbundle.one.uri=bundle.bdl\n> +\tbundle.two.uri=../bundle.bdl\n> +\tbundle.three.uri=sub/dir/bundle.bdl\n> +\tEOF\n> +\n> +\tcat >expect <<-\\EOF &&\n> +\t[bundle]\n> +\t\tversion = 1\n> +\t\tmode = all\n> +\t[bundle \"one\"]\n> +\t\turi = /bundle.bdl\n> +\t[bundle \"two\"]\n> +\t\turi = bundle.bdl\n\nThis seems a little strange, but it looks like '/../bundle.bdl'\nnormalizes to 'bundle.bdl' because '' is treated like a regular path\nelement (like a directory). \n\nOut of curiosity, what would happen if 'bundle.two.uri' was\n'../../bundle.bdl'?\n\n> +\t[bundle \"three\"]\n> +\t\turi = /sub/dir/bundle.bdl\n> +\tEOF\n> +\n> +\ttest-tool bundle-uri parse-key-values in >actual 2>err &&\n> +\ttest_must_be_empty err &&\n> +\ttest_cmp_config_output expect actual\n> +'\n> +\n> test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' '\n> \tcat >in <<-\\EOF &&\n> \t=bogus-value\n"},{"id":"468148","messageId":"c916a819-565e-e58e-3565-dbe2bc1cd9a5@github.com","threadId":"58731","inReplyTo":"f254da46a2c2afa74c6fdfd8cc1e2110c70455a5.1668628303.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 8/9] bundle-uri: download bundles from an advertised list","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-11-29T01:51:04Z","receivedAt":"2022-11-29T01:51:10Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The logic in fetch_bundle_uri() is useful for the --bundle-uri option of\n> 'git clone', but is not helpful when the clone operation discovers a\n> list of URIs from the bundle-uri protocol v2 command. To actually\n> download and unbundle the advertised bundles, we need a different\n> mechanism.\n> \n> Create the new fetch_bundle_list() method which is very similar to\n> fetch_bundle_uri() except that it relies on download_bundle_list()\n> instead of fetch_bundle_uri_internal(). The download_bundle_list()\n> method will recursively call fetch_bundle_uri_internal() if any of the\n> advertised URIs serve a bundle list instead of a bundle. This will also\n> follow the bundle.list.mode setting from the input list: \"any\" will\n> download only one such URI while \"all\" will download data from all of\n> the URIs.\n> \n> In an identical way to fetch_bundle_uri(), the bundles are unbundled\n> after all of the bundle lists have been expanded and all necessary URIs.\n\nThis explanation is clear and matches the implementation below. I'll admit\nit's a bit difficult to wrap my head around what's going on but, from what I\nunderstand, it does what it needs to do to set up for the next patch.\n\nThere's no way to test this change in this patch (since\n'fetch_bundle_list()' isn't called anywhere yet), but I think that's fine;\nmaking it testable would probably make the patch too long/complicated to\nfollow.\n\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> bundle-uri.c | 21 +++++++++++++++++++++\n> bundle-uri.h | 11 +++++++++++\n> 2 files changed, 32 insertions(+)\n> \n> diff --git a/bundle-uri.c b/bundle-uri.c\n> index ab91bb32e9b..5914d220c43 100644\n> --- a/bundle-uri.c\n> +++ b/bundle-uri.c\n> @@ -577,6 +577,27 @@ cleanup:\n> \treturn result;\n> }\n> \n> +int fetch_bundle_list(struct repository *r, const char *uri, struct bundle_list *list)\n> +{\n> +\tint result;\n> +\tstruct bundle_list global_list;\n> +\n> +\tinit_bundle_list(&global_list);\n> +\n> +\t/* If a bundle is added to this global list, then it is required. */\n> +\tglobal_list.mode = BUNDLE_MODE_ALL;\n> +\n> +\tif ((result = download_bundle_list(r, list, &global_list, 0)))\n> +\t\tgoto cleanup;\n> +\n> +\tresult = unbundle_all_bundles(r, &global_list);\n> +\n> +cleanup:\n> +\tfor_all_bundles_in_list(&global_list, unlink_bundle, NULL);\n> +\tclear_bundle_list(&global_list);\n> +\treturn result;\n> +}\n> +\n> /**\n> * API for serve.c.\n> */\n> diff --git a/bundle-uri.h b/bundle-uri.h\n> index 7905e56732c..a75b68d2f5a 100644\n> --- a/bundle-uri.h\n> +++ b/bundle-uri.h\n> @@ -102,6 +102,17 @@ int bundle_uri_parse_config_format(const char *uri,\n> */\n> int fetch_bundle_uri(struct repository *r, const char *uri);\n> \n> +/**\n> + * Given a bundle list that was already advertised (likely by the\n> + * bundle-uri protocol v2 verb) at the given uri, fetch and unbundle the\n> + * bundles according to the bundle strategy of that list.\n> + *\n> + * Returns non-zero if no bundle information is found at the given 'uri'.\n> + */\n> +int fetch_bundle_list(struct repository *r,\n> +\t\t const char *uri,\n> +\t\t struct bundle_list *list);\n> +\n> /**\n> * API for serve.c.\n> */\n\n"},{"id":"468149","messageId":"9522d667-da4d-6eb0-8f1c-1396ebc0351c@github.com","threadId":"58731","inReplyTo":"b62b4b1748194f0c7b81536701f15aa0df8e1d9b.1668628303.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 9/9] clone: unbundle the advertised bundles","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-11-29T01:59:13Z","receivedAt":"2022-11-29T01:59:26Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> A previous change introduced the transport methods to acquire a bundle\n> list from the 'bundle-uri' protocol v2 command, when advertised _and_\n> when the client has chosen to enable the feature.\n> \n> Teach Git to download and unbundle the data advertised by those bundles\n> during 'git clone'.\n> \n> Also, since the --bundle-uri option exists, we do not want to mix the\n> advertised bundles with the user-specified bundles.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/clone.c | 26 +++++++++++++++++----\n> t/t5601-clone.sh | 59 ++++++++++++++++++++++++++++++++++++++++++++++++\n> 2 files changed, 80 insertions(+), 5 deletions(-)\n> \n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index 22b1e506452..09f10477ed6 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -1267,11 +1267,27 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> \tif (refs)\n> \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n> \n> -\t/*\n> -\t * Populate transport->got_remote_bundle_uri and\n> -\t * transport->bundle_uri. We might get nothing.\n> -\t */\n> -\ttransport_get_remote_bundle_uri(transport, 1);\n> +\tif (!bundle_uri) {\n> +\t\t/*\n> +\t\t* Populate transport->got_remote_bundle_uri and\n> +\t\t* transport->bundle_uri. We might get nothing.\n> +\t\t*/\n> +\t\ttransport_get_remote_bundle_uri(transport, 1);\n> +\n> +\t\tif (transport->bundles &&\n> +\t\t hashmap_get_size(&transport->bundles->bundles)) {\n> +\t\t\t/* At this point, we need the_repository to match the cloned repo. */\n> +\t\t\tif (repo_init(the_repository, git_dir, work_tree))\n> +\t\t\t\twarning(_(\"failed to initialize the repo, skipping bundle URI\"));\n> +\t\t\tif (fetch_bundle_list(the_repository,\n> +\t\t\t\t\t remote->url[0],\n> +\t\t\t\t\t transport->bundles))\n\nIf the repo initialization fails, this line is still executed. Should the\ncondition be 'else if' to avoid that?\n\nOtherwise, all of the added logic looks good to me.\n\n> +\t\t\t\twarning(_(\"failed to fetch advertised bundles\"));\n> +\t\t} else {\n> +\t\t\tclear_bundle_list(transport->bundles);\n> +\t\t\tFREE_AND_NULL(transport->bundles);\n> +\t\t}\n> +\t}\n> \n> \tif (mapped_refs) {\n> \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n> diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\n> index 45f0803ed4d..d1d8139751e 100755\n> --- a/t/t5601-clone.sh\n> +++ b/t/t5601-clone.sh\n\nPer the commit message:\n\n> Also, since the --bundle-uri option exists, we do not want to mix the\n> advertised bundles with the user-specified bundles.\n\nCould you add a test verifying that '--bundle-uri' causes 'clone' to skip\nbundle URI auto-discovery? It's clear from the implementation above that\n'clone' is currently doing that as-expected, but it might be nice to have\nthe test for regression testing purposes.\n\n> @@ -795,6 +795,65 @@ test_expect_success 'reject cloning shallow repository using HTTP' '\n> \tgit clone --no-reject-shallow $HTTPD_URL/smart/repo.git repo\n> '\n> \n> +test_expect_success 'auto-discover bundle URI from HTTP clone' '\n> +\ttest_when_finished rm -rf trace.txt repo2 \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" &&\n> +\tgit -C src bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/everything.bundle\" --all &&\n> +\tgit clone --bare --no-local src \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" &&\n> +\n> +\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n> +\t\tuploadpack.advertiseBundleURIs true &&\n> +\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n> +\t\tbundle.version 1 &&\n> +\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n> +\t\tbundle.mode all &&\n> +\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n> +\t\tbundle.everything.uri \"$HTTPD_URL/everything.bundle\" &&\n> +\n> +\tGIT_TEST_BUNDLE_URI=1 \\\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n> +\t\tgit -c protocol.version=2 clone \\\n> +\t\t$HTTPD_URL/smart/repo2.git repo2 &&\n> +\tcat >pattern <<-EOF &&\n> +\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/everything.bundle\"\\]\n> +\tEOF\n> +\tgrep -f pattern trace.txt\n> +'\n> +\n> +test_expect_success 'auto-discover multiple bundles from HTTP clone' '\n> +\ttest_when_finished rm -rf trace.txt repo3 \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" &&\n> +\n> +\ttest_commit -C src new &&\n> +\tgit -C src bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/new.bundle\" HEAD~1..HEAD &&\n> +\tgit clone --bare --no-local src \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" &&\n> +\n> +\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n> +\t\tuploadpack.advertiseBundleURIs true &&\n> +\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n> +\t\tbundle.version 1 &&\n> +\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n> +\t\tbundle.mode all &&\n> +\n> +\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n> +\t\tbundle.everything.uri \"$HTTPD_URL/everything.bundle\" &&\n> +\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n> +\t\tbundle.new.uri \"$HTTPD_URL/new.bundle\" &&\n> +\n> +\tGIT_TEST_BUNDLE_URI=1 \\\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n> +\t\tgit -c protocol.version=2 clone \\\n> +\t\t$HTTPD_URL/smart/repo3.git repo3 &&\n> +\n> +\t# We should fetch _both_ bundles\n> +\tcat >pattern <<-EOF &&\n> +\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/everything.bundle\"\\]\n> +\tEOF\n> +\tgrep -f pattern trace.txt &&\n> +\tcat >pattern <<-EOF &&\n> +\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/new.bundle\"\\]\n> +\tEOF\n> +\tgrep -f pattern trace.txt\n> +'\n> +\n> # DO NOT add non-httpd-specific tests here, because the last part of this\n> # test script is only executed when httpd is available and enabled.\n> \n\n"},{"id":"468435","messageId":"4ec12008-620d-c186-38ef-08ea87fec856@github.com","threadId":"58731","inReplyTo":"ca410bed-e8d1-415f-5235-b64fe18bed27@github.com","subject":"Re: [PATCH v2 2/9] bundle-uri client: add minimal NOOP client","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-12-02T15:00:35Z","receivedAt":"2022-12-02T15:00:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/28/2022 7:57 PM, Victoria Dye wrote:\n> Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n>> From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n>> \n\n>> There's a question of what level of encapsulation we should use here,\n>> I've opted to use connect.h in clone.c, but we could also e.g. make\n>> transport_get_remote_refs() invoke this, i.e. make it implicitly get\n>> the bundle-uri list for later steps.\n>\n> I'm not sure I follow what this sentence is saying. Looking at the\n> implementation below [1], you've added a call to\n> 'transport_get_remote_bundle_uri()' in 'clone.c', but that's defined in\n> 'transport.h' (which is already included in 'clone.c'). Why is 'connect.h'\n> needed at all?\n\nIt's not. Good catch!\n\n>> This approach means that we don't \"support\" this in \"git fetch\" for\n>> now. I'm starting with the case of initial clones, although as noted\n>> in preceding commits to the protocol documentation nothing about this\n>> approach precludes getting bundles on incremental fetches.\n>\n> This explanation seems more complicated than necessary. I think it's\n> sufficient to say \"The no-op client is initially used only in 'clone' to\n> test the basic functionality. The bundle URI client will be integrated into\n> fetch, pull, etc. in later patches\".\n\nDefinitely can be reworded. Specifically, fetches need more functionality\n(coming in part V) before there is value in that integration.\n\n>> For the t5732-protocol-v2-bundle-uri-http.sh it's not easy to set\n>> environment variables for git-upload-pack (it's started by Apache), so\n>> let's skip the test under T5730_HTTP, and add unused T5730_{FILE,GIT}\n>> prerequisites for consistency and future use.\n>\n> \"skip the test\" doesn't explain *which* test is skipped (and it doesn't look\n> like you skip all of them). I think you're referring to \"bad client with\n> $T5730_PROTOCOL:// using protocol v2\" specifically?\n\nWill make the specific test to skip more clear.\n\n>> diff --git a/bundle-uri.c b/bundle-uri.c\n>> index 32022595964..2201b604b11 100644\n>> --- a/bundle-uri.c\n>> +++ b/bundle-uri.c\n>> @@ -571,6 +571,10 @@ int bundle_uri_advertise(struct repository *r, struct strbuf *value)\n>> {\n>> \tstatic int advertise_bundle_uri = -1;\n>>\n>> +\tif (value &&\n>> +\t git_env_bool(\"GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE\", 0))\n>> +\t\tstrbuf_addstr(value, \"test-unknown-capability-value\");\n>\n> It looks like 'GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE' is being used\n> to \"mock\" server responses to test certain behavior on the client side. I'm\n> somewhat uncomfortable with how this mixes test-specific code paths with\n> application code, and AFAICT nothing similar is done for other\n> advertise/command functions in protocol v2. Is there a way to set up tests\n> to intercept the client requests and customize the response?\n\nI'm going to investigate how we can test similar functionality within the\ntest-tool code instead, if possible.\n\n>> +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\n>> +\t\t\t struct bundle_list *bundles, int stateless_rpc)\n>> +{\n>> +\tint line_nr = 1;\n>> +\n>> +\t/* Assert bundle-uri support */\n>> +\tserver_supports_v2(\"bundle-uri\", 1);\n>> +\n>> +\t/* (Re-)send capabilities */\n>> +\tsend_capabilities(fd_out, reader);\n>> +\n>> +\t/* Send command */\n>> +\tpacket_write_fmt(fd_out, \"command=bundle-uri\\n\");\n>> +\tpacket_delim(fd_out);\n>> +\n>> +\t/* Send options */\n>> +\tif (git_env_bool(\"GIT_TEST_PROTOCOL_BAD_BUNDLE_URI\", 0))\n>> +\t\tpacket_write_fmt(fd_out, \"test-bad-client\\n\");\n>\n> Same comment as on 'GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE'. There's\n> no precedent that I can find for a test variable like this in 'connect.c', and\n> \"in the middle of client code\" doesn't seem like an ideal place for it.\n>\n> If there really isn't another way of doing this, could the addition of these\n> 'GIT_TEST' variables and their associated tests be split out into a\n> dedicated commit? That would at least separate the test code paths from the\n> application code in the commit history.\n\nI'll definitely split them out, making this change much more about the\ntest boilerplate. In addition, most of the test boilerplate actually works\nwithout the 'git clone' update, so this can be split into three commits:\n\n1. Create the test infrastructure to check that the server advertises\n the 'bundle-uri' command appropriately.\n\n2. Implement the basic client that issues and parses the 'bundle-uri'\n command. Add the request to 'git clone' and add a test that verifies\n that the client makes the request.\n\n3. Add the extra error condition tests.\n\n>> +++ b/t/lib-t5730-protocol-v2-bundle-uri.sh\n>\n> nit: this set of tests is used in more than just 't5730', so the name is\n> somewhat misleading. It also seems a bit overly-specific to include\n> \"protocol-v2\" in the filename (the tests themselves mention \"protocol v2\"\n> when it's relevant). What about something like 'lib-proto-bundle-uri.sh'\n> (using \"proto\" to mimic 'lib-proto-disable.sh')?\n\nI agree. I think 'lib-bundle-uri-protocol.sh' is a clearer name.\n\n>> +# Poor man's URI escaping. Good enough for the test suite whose trash\n>> +# directory has a space in it. See 93c3fcbe4d4 (git-svn: attempt to\n>> +# mimic SVN 1.7 URL canonicalization, 2012-07-28) for prior art.\n>> +test_uri_escape() {\n>> +\tsed 's/ /%20/g'\n>> +}\n>\n> This is a good opportunity to unify on a single implementation rather than\n> to have multiple similar ones floating around. Can this be moved into\n> 'test-lib.sh' (or 'test-lib-functions.sh'?), with 't9119' and 't9120'\n> updated to use the new 'test_uri_escape()'?\n\nWill move, although I was not able to find the use in t9120.\n\n>> +test_expect_success \"unknown capability value with $T5730_PROTOCOL:// using protocol v2\" '\n>\n> Why is this test only run for the 'file://' transport protocol? And, why\n> isn't it in 'lib-t5730-protocol-v2-bundle-uri.sh'? If nothing else, moving\n> this test to that file (even if it needs to be conditional on a specific\n> protocol) puts the '$T5730_PARENT', '$T5730_BUNDLE_URI_ESCAPED' and\n> '$T5730_URI' variables in scope for better readability.\n\nI think this is one of the tests that doesn't work in HTTP, but could be\nskipped using a prereq if it is placed in the common test script.\n\nI will rethink this test coverage to see if there is a different way to\ncheck similar behavior without as much insertion into the client/server\ncode.\n\n>> +int transport_get_remote_bundle_uri(struct transport *transport)\n>> +{\n>> +\tconst struct transport_vtable *vtable = transport->vtable;\n>> +\n>> +\t/* Check config only once. */\n>> +\tif (transport->got_remote_bundle_uri++)\n>> +\t\treturn 0;\n>\n> We're only ever going to read the bundle list once per command (or, at least\n> once per 'transport' instance), so if 'transport_get_remote_bundle_uri()'\n> has already been called, we can safely assume the correct results (if any)\n> are in the 'transport' structure.\n\nYes, although it suffers from a mistake of this form I've seen before:\ngot_remote_bundle_uri is a single bit, so this only works every other time.\nI will fix this.\n\nThanks for the detailed review!\n-Stolee\n"},{"id":"468437","messageId":"c3e1aaea-f509-cc00-b093-f114a15692a3@github.com","threadId":"58731","inReplyTo":"84c07e84-0805-6163-d77b-cb9f42db402e@github.com","subject":"Re: [PATCH v2 3/9] bundle-uri client: add helper for testing server","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-12-02T15:28:28Z","receivedAt":"2022-12-02T15:28:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/28/2022 7:59 PM, Victoria Dye wrote:\n> Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n>> From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n>> \n>>\n>> Add a 'test-tool bundle-uri ls-remote' command. This is a thin wrapper\n>> for issuing protocol v2 \"bundle-uri\" commands to a server, and to the\n>> parsing routines in bundle-uri.c.\n>>\n>> Since in the \"git clone\" case we'll have already done the handshake(),\n>> but not here, introduce a \"got_advertisement\" state along with\n>> \"got_remote_heads\". It seems to me that the \"got_remote_heads\" is\n>> badly named in the first place, and the whole logic of eagerly getting\n>> ls-refs on handshake() or not could be refactored somewhat, but let's\n>> not do that now, and instead just add another self-documenting state\n>> variable.\n>\n> Maybe I'm missing something, but why not just rename 'got_remote_heads' to\n> something like 'finished_handshake' rather than adding 'got_advertisement'\n> (since, AFAICT, it's always identical in value to 'got_remote_heads').\n\nI think that is a reasonable recommendation.\n\n>> --- a/t/helper/test-bundle-uri.c\n>> +++ b/t/helper/test-bundle-uri.c\n>> @@ -88,6 +132,8 @@ int cmd__bundle_uri(int argc, const char **argv)\n>> \t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS);\n>> \tif (!strcmp(argv[1], \"parse-config\"))\n>> \t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE);\n>> +\tif (!strcmp(argv[1], \"ls-remote\"))\n>> +\t\treturn cmd_ls_remote(argc - 1, argv + 1);\n>\n> With this helper being added, I'm not sure if/why 'clone' was needed to test\n> the bundle URIs in patch 2 (I assumed integrating with a command was the\n> only way to test it, which is why I didn't mention this in my review [1]).\n> In the spirit of having commits avoid \"doing more than one thing\" could\n> these patches be reorganized into something like:\n>\n> 1. Add the no-op client & some basic tests around fetching the bundle URI\n> list using this test helper.\n> 2. Add the 'transport_get_remote_bundle_uri()' call to 'clone()' with\n> clone-specific tests.\n>\n> It probably wouldn't make the patches much shorter, but it would help avoid\n> the churn of test changes & changing assumptions around 'quiet' &\n> 'got_advertisement' in this patch.\n\nI will think more on this as I get further into your review and figure out\na way to do the error case tests. At minimum, I've split out some things\nso they might be easier to rearrange, but the 'git clone' integration is\n(currently) still paired with the implementation in transport.c.\n\n>> test_expect_success \"setup protocol v2 $T5730_PROTOCOL:// tests\" '\n>> \tgit init \"$T5730_PARENT\" &&\n>> \ttest_commit -C \"$T5730_PARENT\" one &&\n>> -\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs true\n>> +\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs true &&\n>> +\tgit -C \"$T5730_PARENT\" config bundle.version 1 &&\n>> +\tgit -C \"$T5730_PARENT\" config bundle.mode all\n>\n> Why are these config settings added here? I don't see them used anywhere?\n\nThis can be delayed until the next change that actually reads that config.\n\n>> diff --git a/transport.c b/transport.c\n>> index a020adc1f56..86460f5be28 100644\n>> --- a/transport.c\n>> +++ b/transport.c\n>> @@ -371,6 +373,33 @@ static int get_bundle_uri(struct transport *transport)\n>> \t\tinit_bundle_list(transport->bundles);\n>> \t}\n>>\n>> +\tif (!data->got_advertisement) {\n>> +\t\tstruct ref *refs;\n>> +\t\tstruct git_transport_data *data = transport->data;\n>> +\t\tenum protocol_version version;\n>> +\n>> +\t\trefs = handshake(transport, 0, NULL, 0);\n>> +\t\tversion = data->version;\n>> +\n>> +\t\tswitch (version) {\n>> +\t\tcase protocol_v2:\n>> +\t\t\tassert(!refs);\n>> +\t\t\tbreak;\n>> +\t\tcase protocol_v0:\n>> +\t\tcase protocol_v1:\n>> +\t\tcase protocol_unknown_version:\n>> +\t\t\tassert(refs);\n>> +\t\t\tbreak;\n>\n> Why were these 'refs' assertions added? What are they intended to validate?\n\nYou're right. This is essentially inserting test code into the product\n(although the assert()s would be compiled out, I assume). The only differnce\nhere is that after the handshake, protocol v2 has not executed the 'ls-refs'\ncommand, while the other protocol versions start with a ref advertisement\nin the initial response.\n\nThanks,\n-Stolee\n"},{"id":"468438","messageId":"7f94bda0-9f08-f179-3e7e-a791a607657e@github.com","threadId":"58731","inReplyTo":"4ec06afa-d804-16b1-f140-dfd9ebd912e5@github.com","subject":"Re: [PATCH v2 5/9] bundle-uri client: add boolean transfer.bundleURI setting","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-12-02T15:38:19Z","receivedAt":"2022-12-02T15:38:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/28/2022 8:03 PM, Victoria Dye wrote:\n> Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n>> From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n>> \n>>\n>> The yet-to-be introduced client support for bundle-uri will always\n>> fall back on a full clone, but we'd still like to be able to ignore a\n>> server's bundle-uri advertisement entirely.\n>>\n>> The new transfer.bundleURI config option defaults to 'false', but a user\n>> can set it to 'true' to enable checking for bundle URIs from the origin\n>> Git server using protocol v2.\n> \n> Thanks for adding this, an \"opt-in\" approach seems reasonable for\n> introducing this feature.\n> \n>>\n>> To enable this setting by default in the correct tests, add a\n>> GIT_TEST_BUNDLE_URI environment variable.\n> \n> This makes sense. I'm less concerned with this environment variable than\n> those in patch 2 [1], since it's in line with the test variables that\n> enable/disable whole features ('GIT_TEST_SPLIT_INDEX',\n> 'GIT_TEST_COMMIT_GRAPH', etc.). \n> \n> The only thing feedback I can think of would be that this patch could be\n> moved to earlier in the series (that is, immediately after creating\n> 'transport_get_remote_bundle_uri()'). That said, I don't feel strongly\n> either way.\n\nIt was simple enough to reorder them, so I've done that.\n\nThanks,\n-Stolee\n"},{"id":"468439","messageId":"9ac3d892-0f67-3d4d-060e-a599551c0fd3@github.com","threadId":"58731","inReplyTo":"be89bbec-1a8b-1337-a5f7-f43420da64de@github.com","subject":"Re: [PATCH v2 6/9] strbuf: introduce strbuf_strip_file_from_path()","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-12-02T15:40:56Z","receivedAt":"2022-12-02T15:41:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/28/2022 8:03 PM, Victoria Dye wrote:\n> Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>>\n>> The strbuf_parent_directory() method was added as a static method in\n>> contrib/scalar by d0feac4e8c0 (scalar: 'register' sets recommended\n>> config and starts maintenance, 2021-12-03) and then removed in\n>> 65f6a9eb0b9 (scalar: constrain enlistment search, 2022-08-18), but now\n>> there is a need for a similar method in the bundle URI feature.\n>>\n>> Re-add the method, this time in strbuf.c, but with a new name:\n>> strbuf_strip_file_from_path(). The method requirements are slightly\n>> modified to allow a trailing slash, in which case nothing is done, which\n>> makes the name change valuable. The return value is the number of bytes\n>> removed.\n> \n> *Extremely* minor point, but why return anything at all? The call in the\n> next patch doesn't use the return value, and some similar-in-spirit 'strbuf'\n> functions (like 'strbuf_trim()') return nothing. \n> \n> I don't think this is worth changing if you can imagine using that return\n> value for something eventually; just wanted to point it out as something to\n> (optionally) consider if you re-roll for something else anyway.\n\nWhile I'm here, it's not too hard to remove that and save some lines.\nWe can always bring that back if someone needs it in the future.\n\nThanks,\n-Stolee\n"},{"id":"468442","messageId":"7000a8f3-9b80-75e9-3d36-36982ef5dbb3@github.com","threadId":"58731","inReplyTo":"06e45662-699b-52f9-2fdf-6e621444f446@github.com","subject":"Re: [PATCH v2 7/9] bundle-uri: allow relative URLs in bundle lists","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-12-02T16:03:40Z","receivedAt":"2022-12-02T16:04:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/28/2022 8:25 PM, Victoria Dye wrote:\n> Derrick Stolee via GitGitGadget wrote:\n\n>> +\tif (!list->baseURI) {\n>> +\t\tstruct strbuf baseURI = STRBUF_INIT;\n>> +\t\tstrbuf_addstr(&baseURI, uri);\n>> +\n>> +\t\t/*\n>> +\t\t * If the URI does not end with a trailing slash, then\n>> +\t\t * remove the filename portion of the path. This is\n>> +\t\t * important for relative URIs.\n>> +\t\t */\n>> +\t\tstrbuf_strip_file_from_path(&baseURI);\n>> +\t\tlist->baseURI = strbuf_detach(&baseURI, NULL);\n>\n> Is the 'baseURI' is set to the URI of the first bundle (ordered by hash)? If\n> data is distributed across multiple CDNs, couldn't this be a suboptimal\n> choice? For example, if the first bundle is on 'A.com', but every other\n> bundle is on 'B.org'?\n\nThe baseURI is set to one of two things:\n\n1. The URI used for the clone, specifying the way the client connected to\n the Git server, or\n\n2. The URI used to download the bundle list itself.\n\nThis allows the same bundle list file to be distributed to multiple CDNs,\nassuming that the bundles themselves will have the same relative position\nto the list.\n\n>> +\t/**\n>> +\t * The baseURI of a bundle_list is used as the base for any\n>> +\t * relative URIs advertised by the bundle list at that location.\n>> +\t *\n>> +\t * When the list is generated from a Git server, then use that\n>> +\t * server's location.\n>\n> Hmmm, I think I'm missing something with my earlier comment. I thought the\n> 'uri' argument to 'bundle_uri_parse_config_format()' was an individual\n> bundle's URI? What's the \"server's location\" in this context?\n\nI can work to make this concept clearer by rewording this comment.\n\n>> @@ -40,6 +40,8 @@ static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mo\n>>\n>> \tinit_bundle_list(&list);\n>>\n>> +\tlist.baseURI = xstrdup(\"\");\n>\n> Using a hardcoded value here leads to pretty different behavior in\n> 'test-bundle-uri.c' vs. starting with an unset 'list.baseURI' in something\n> like 'clone'. Why does this need to be set to '' for the tests?\n\nIn this part of the test helper, we are not making a connection to a server\nand instead parsing a bundle list file directly. To demonstrate how the\nrelative paths work during this parsing, we add a bogus baseURI here so\nwe can clearly see where the relative paths were parsed versus using the\nURI as an absolute URI.\n\n\n>> +test_expect_success 'bundle_uri_parse_line(): relative URIs' '\n>> +\tcat >in <<-\\EOF &&\n>> +\tbundle.one.uri=bundle.bdl\n>> +\tbundle.two.uri=../bundle.bdl\n>> +\tbundle.three.uri=sub/dir/bundle.bdl\n>> +\tEOF\n>> +\n>> +\tcat >expect <<-\\EOF &&\n>> +\t[bundle]\n>> +\t\tversion = 1\n>> +\t\tmode = all\n>> +\t[bundle \"one\"]\n>> +\t\turi = /bundle.bdl\n>> +\t[bundle \"two\"]\n>> +\t\turi = bundle.bdl\n>\n> This seems a little strange, but it looks like '/../bundle.bdl'\n> normalizes to 'bundle.bdl' because '' is treated like a regular path\n> element (like a directory).\n>\n> Out of curiosity, what would happen if 'bundle.two.uri' was\n> '../../bundle.bdl'?\n\nIt will fail! The error message is\n\n\t\"fatal: cannot strip one component off url '.'\"\n\nThis is disappointing that an erroneous bundle list could cause a 'git\nclone' command to die(), when we want the bundle URI feature to allow the\nclone to continue normally even if the bundle downloads fail. I will mark\nthis for #leftoverbits, since it would involve changing the interface for\nchop_last_dir() and relative_url() in remote.c.\n\nAt minimum, I will document this with a test case.\n\nThanks,\n-Stolee\n"},{"id":"468443","messageId":"e4b6363a-58a4-a11e-d33e-c0bc9c755d72@github.com","threadId":"58731","inReplyTo":"9522d667-da4d-6eb0-8f1c-1396ebc0351c@github.com","subject":"Re: [PATCH v2 9/9] clone: unbundle the advertised bundles","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-12-02T16:16:41Z","receivedAt":"2022-12-02T16:18:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/28/2022 8:59 PM, Victoria Dye wrote:\n> Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>>\n>> A previous change introduced the transport methods to acquire a bundle\n>> list from the 'bundle-uri' protocol v2 command, when advertised _and_\n>> when the client has chosen to enable the feature.\n>>\n>> Teach Git to download and unbundle the data advertised by those bundles\n>> during 'git clone'.\n>>\n>> Also, since the --bundle-uri option exists, we do not want to mix the\n>> advertised bundles with the user-specified bundles.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> builtin/clone.c | 26 +++++++++++++++++----\n>> t/t5601-clone.sh | 59 ++++++++++++++++++++++++++++++++++++++++++++++++\n>> 2 files changed, 80 insertions(+), 5 deletions(-)\n>>\n>> diff --git a/builtin/clone.c b/builtin/clone.c\n>> index 22b1e506452..09f10477ed6 100644\n>> --- a/builtin/clone.c\n>> +++ b/builtin/clone.c\n>> @@ -1267,11 +1267,27 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n>> \tif (refs)\n>> \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n>>\n>> -\t/*\n>> -\t * Populate transport->got_remote_bundle_uri and\n>> -\t * transport->bundle_uri. We might get nothing.\n>> -\t */\n>> -\ttransport_get_remote_bundle_uri(transport, 1);\n>> +\tif (!bundle_uri) {\n>> +\t\t/*\n>> +\t\t* Populate transport->got_remote_bundle_uri and\n>> +\t\t* transport->bundle_uri. We might get nothing.\n>> +\t\t*/\n>> +\t\ttransport_get_remote_bundle_uri(transport, 1);\n>> +\n>> +\t\tif (transport->bundles &&\n>> +\t\t hashmap_get_size(&transport->bundles->bundles)) {\n>> +\t\t\t/* At this point, we need the_repository to match the cloned repo. */\n>> +\t\t\tif (repo_init(the_repository, git_dir, work_tree))\n>> +\t\t\t\twarning(_(\"failed to initialize the repo, skipping bundle URI\"));\n>> +\t\t\tif (fetch_bundle_list(the_repository,\n>> +\t\t\t\t\t remote->url[0],\n>> +\t\t\t\t\t transport->bundles))\n>\n> If the repo initialization fails, this line is still executed. Should the\n> condition be 'else if' to avoid that?\n>\n> Otherwise, all of the added logic looks good to me.\n\nYes, it should. An earlier version of this follows the correct if/else if\npattern.\n\n>> diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\n>> index 45f0803ed4d..d1d8139751e 100755\n>> --- a/t/t5601-clone.sh\n>> +++ b/t/t5601-clone.sh\n>\n> Per the commit message:\n>\n>> Also, since the --bundle-uri option exists, we do not want to mix the\n>> advertised bundles with the user-specified bundles.\n>\n> Could you add a test verifying that '--bundle-uri' causes 'clone' to skip\n> bundle URI auto-discovery? It's clear from the implementation above that\n> 'clone' is currently doing that as-expected, but it might be nice to have\n> the test for regression testing purposes.\n\nI can add that to the lib-bundle-uri-protocol.sh tests pretty easily.\n\nThanks,\n-Stolee\n"},{"id":"468460","messageId":"221202.86bkolfwzv.gmgdl@evledraar.gmail.com","threadId":"58731","inReplyTo":"7d86852c01513ee131bf993302416f4c881a0bc6.1668628303.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 6/9] strbuf: introduce strbuf_strip_file_from_path()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-12-02T18:32:24Z","receivedAt":"2022-12-02T19:04:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Nov 16 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> The strbuf_parent_directory() method was added as a static method in\n> contrib/scalar by d0feac4e8c0 (scalar: 'register' sets recommended\n> config and starts maintenance, 2021-12-03) and then removed in\n> 65f6a9eb0b9 (scalar: constrain enlistment search, 2022-08-18), but now\n> there is a need for a similar method in the bundle URI feature.\n>\n> Re-add the method, this time in strbuf.c, but with a new name:\n> strbuf_strip_file_from_path(). The method requirements are slightly\n> modified to allow a trailing slash, in which case nothing is done, which\n> makes the name change valuable. The return value is the number of bytes\n> removed.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> strbuf.c | 9 +++++++++\n> strbuf.h | 12 ++++++++++++\n> 2 files changed, 21 insertions(+)\n>\n> diff --git a/strbuf.c b/strbuf.c\n> index 0890b1405c5..8d1e2e8bb61 100644\n> --- a/strbuf.c\n> +++ b/strbuf.c\n> @@ -1200,3 +1200,12 @@ int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n> \tfree(path2);\n> \treturn res;\n> }\n> +\n> +size_t strbuf_strip_file_from_path(struct strbuf *buf)\n\nNit: Almost every function in this API calls its argument \"sb\", let's do\nthat for new functions.\n\n> +{\n> +\tsize_t len = buf->len;\n> +\tsize_t offset = offset_1st_component(buf->buf);\n\nMm, isn't the return value of offset_1st_component() a boolean? it's\njust an \"is_dir_sep(buf->buf[0])\".\n\nSo this works to....\n\n> +\tchar *path_sep = find_last_dir_sep(buf->buf + offset);\n\n...find the last dir separator starting at either 0 or 1.\n\nBut anyway, what sort of string is this expecting to handle where the\nlast dir separator isn't >=1 offset into the string anyway? Shouldn't we\njust exclude the string \"/\" here? Maybe I'm missing something....\n\n\n> +\tstrbuf_setlen(buf, path_sep ? path_sep - buf->buf + 1 : offset);\n> +\treturn len - buf->len;\n> +}\n\nUrm, so isn't this literally one-byte away from being equivalent to a\nfunction that's already in the API?:\nstrbuf_trim_trailing_dir_sep. I.e. this seems to me to do the same as\nthis new function.\n\nContext manually adjusted so we can see the only difference is the\n\"is_dir_sep\" v.s. \"!is_dir_sep\".\n\nThere's a few strbuf functions like that, and we should probably\ngeneralize the ctype-like test they share into some callback mechanism,\nbut in the meantime keeping with the pattern & naming of existing\nfunctions seems better.\n\nBut again, I may be missing something.\n\nI removed the comment because if it's the same then the new function is\nself-documenting. It doesn't matter if the URI ends in a \"/\" or not, all\nwe need to get across is that we're stripping non-dirsep characters from\nthe URL, whether it ends in one or not.\n\nIn terms of correctness: The use of is_dir_sep() seems incorrect to me\nhere. On Windows won't that end up using is_xplatform_dir_sep(), so\nbundle-uri's behavior will differ there, and we'll support \\\\-paths as\nwell as /-paths, but elsewhere only /-paths.\n\nShouldn't this just test \"/\", not \"is_dir_sep()\"?\n\nAt which point (if the above is correct) we could also call this\nstrbuf_rtrim_notchr(), and just call strbuf_rtrim_notchr(sb, '/') (but\neven better would be a ctype-like callback).\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 5914d220c43..c3ed04eae0f 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -192,20 +192,15 @@ int bundle_uri_parse_config_format(const char *uri,\n \t\t.error_action = CONFIG_ERROR_ERROR,\n \t};\n \n \tif (!list->baseURI) {\n \t\tstruct strbuf baseURI = STRBUF_INIT;\n \t\tstrbuf_addstr(&baseURI, uri);\n \n-\t\t/*\n-\t\t * If the URI does not end with a trailing slash, then\n-\t\t * remove the filename portion of the path. This is\n-\t\t * important for relative URIs.\n-\t\t */\n-\t\tstrbuf_strip_file_from_path(&baseURI);\n+\t\tstrbuf_trim_trailing_not_dir_sep(&baseURI);\n \t\tlist->baseURI = strbuf_detach(&baseURI, NULL);\n \t}\n \tresult = git_config_from_file_with_options(config_to_bundle_list,\n \t\t\t\t\t\t filename, list,\n \t\t\t\t\t\t &opts);\n \n \tif (!result && list->mode == BUNDLE_MODE_NONE) {\ndiff --git a/strbuf.c b/strbuf.c\nindex 8d1e2e8bb61..3466552b854 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -117,14 +117,21 @@ void strbuf_rtrim(struct strbuf *sb)\n void strbuf_trim_trailing_dir_sep(struct strbuf *sb)\n {\n \twhile (sb->len > 0 && is_dir_sep((unsigned char)sb->buf[sb->len - 1]))\n \t\tsb->len--;\n \tsb->buf[sb->len] = '\\0';\n }\n \n+void strbuf_trim_trailing_not_dir_sep(struct strbuf *sb)\n+{\n+\twhile (sb->len > 0 && !is_dir_sep((unsigned char)sb->buf[sb->len - 1]))\n+\t\tsb->len--;\n+\tsb->buf[sb->len] = '\\0';\n+}\n+\n void strbuf_trim_trailing_newline(struct strbuf *sb)\n {\n \tif (sb->len > 0 && sb->buf[sb->len - 1] == '\\n') {\n \t\tif (--sb->len > 0 && sb->buf[sb->len - 1] == '\\r')\n \t\t\t--sb->len;\n \t\tsb->buf[sb->len] = '\\0';\n \t}\n@@ -1196,16 +1203,7 @@ int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n \t\t\tres = error_errno(_(\"could not edit '%s'\"), path);\n \t\tunlink(path);\n \t}\n \n \tfree(path2);\n \treturn res;\n }\n-\n-size_t strbuf_strip_file_from_path(struct strbuf *buf)\n-{\n-\tsize_t len = buf->len;\n-\tsize_t offset = offset_1st_component(buf->buf);\n-\tchar *path_sep = find_last_dir_sep(buf->buf + offset);\n-\tstrbuf_setlen(buf, path_sep ? path_sep - buf->buf + 1 : offset);\n-\treturn len - buf->len;\n-}\ndiff --git a/strbuf.h b/strbuf.h\nindex 4822b713786..b5929ecc8dd 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -185,14 +185,16 @@ static inline void strbuf_setlen(struct strbuf *sb, size_t len)\n */\n void strbuf_trim(struct strbuf *sb);\n void strbuf_rtrim(struct strbuf *sb);\n void strbuf_ltrim(struct strbuf *sb);\n \n /* Strip trailing directory separators */\n void strbuf_trim_trailing_dir_sep(struct strbuf *sb);\n+/* Strip trailing non-directory separators */\n+void strbuf_trim_trailing_not_dir_sep(struct strbuf *sb);\n \n /* Strip trailing LF or CR/LF */\n void strbuf_trim_trailing_newline(struct strbuf *sb);\n \n /**\n * Replace the contents of the strbuf with a reencoded form. Returns -1\n * on error, 0 on success.\n"},{"id":"468521","messageId":"0980dcd4-30eb-4ef4-9369-279abe5ca5a2@github.com","threadId":"58731","inReplyTo":"221202.86bkolfwzv.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 6/9] strbuf: introduce strbuf_strip_file_from_path()","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-12-05T15:11:59Z","receivedAt":"2022-12-05T15:12:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/2/2022 1:32 PM, Ævar Arnfjörð Bjarmason wrote:\n\n>> +size_t strbuf_strip_file_from_path(struct strbuf *buf)\n> \n> Nit: Almost every function in this API calls its argument \"sb\", let's do\n> that for new functions.\n\nSure.\n\n>> +{\n>> +\tsize_t len = buf->len;\n>> +\tsize_t offset = offset_1st_component(buf->buf);\n> \n> Mm, isn't the return value of offset_1st_component() a boolean? it's\n> just an \"is_dir_sep(buf->buf[0])\".\n> \n> So this works to....\n> \n>> +\tchar *path_sep = find_last_dir_sep(buf->buf + offset);\n> \n> ...find the last dir separator starting at either 0 or 1.\n> \n> But anyway, what sort of string is this expecting to handle where the\n> last dir separator isn't >=1 offset into the string anyway? Shouldn't we\n> just exclude the string \"/\" here? Maybe I'm missing something....\n\nThe difference is all about whether or not we start with a slash _and_ no\nother slash appears in the path.\n\n 1. \"/root-file\": offset becomes 1 and path_sep becomes NULL, so the\n length is reduced to offset (1), resulting in \"/\"\n 2. \"local-file\": offset becomes 0 and path_sep becomes NULL, so the\n length is reduced to offset (0), resulting in \"\"\n\nOf course, the first case would be the same if we did not set offset to 1\nand then path_sep points to that base slash, so the length calculation\nresults in \"/\" anyway. That will simplify things.\n\nvoid strbuf_strip_file_from_path(struct strbuf *sb)\n{\n\tchar *path_sep = find_last_dir_sep(sb->buf);\n\tstrbuf_setlen(sb, path_sep ? path_sep - sb->buf + 1 : 0);\n}\n\n>> +\tstrbuf_setlen(buf, path_sep ? path_sep - buf->buf + 1 : offset);\n>> +\treturn len - buf->len;\n>> +}\n> \n> Urm, so isn't this literally one-byte away from being equivalent to a\n> function that's already in the API?:\n> strbuf_trim_trailing_dir_sep. I.e. this seems to me to do the same as\n> this new function.\n> \n> Context manually adjusted so we can see the only difference is the\n> \"is_dir_sep\" v.s. \"!is_dir_sep\".\n> \n> There's a few strbuf functions like that, and we should probably\n> generalize the ctype-like test they share into some callback mechanism,\n> but in the meantime keeping with the pattern & naming of existing\n> functions seems better.\n\nUsing callbacks or extra options seems like overkill here.\n\n> I removed the comment because if it's the same then the new function is\n> self-documenting. It doesn't matter if the URI ends in a \"/\" or not, all\n> we need to get across is that we're stripping non-dirsep characters from\n> the URL, whether it ends in one or not.\n> \n> In terms of correctness: The use of is_dir_sep() seems incorrect to me\n> here. On Windows won't that end up using is_xplatform_dir_sep(), so\n> bundle-uri's behavior will differ there, and we'll support \\\\-paths as\n> well as /-paths, but elsewhere only /-paths.\n> \n> Shouldn't this just test \"/\", not \"is_dir_sep()\"?\n\nThis method is used for file paths, too, so is_dir_sep() is important to\nwork properly on Windows platforms.\n\nThanks,\n-Stolee\n\n"},{"id":"468527","messageId":"pull.1400.v3.git.1670262639.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v2.git.1668628302.gitgitgadget@gmail.com","subject":"[PATCH v3 00/11] Bundle URIs IV: advertise over protocol v2","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-05T17:50:28Z","receivedAt":"2022-12-05T17:52:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is based on the recent master batch that included ds/bundle-uri-....\n\nNow that git clone --bundle-uri can download a bundle list from a plaintex\nfile in config format, we can use the same set of key-value pairs to\nadvertise a bundle list over protocol v2. At the end of this series:\n\n 1. A server can advertise bundles when uploadPack.advertiseBundleURIs is\n enabled. The bundle list comes from the server's local config,\n specifically the bundle.* namespace.\n 2. A client can notice a server's bundle-uri advertisement and request the\n bundle list if transfer.bundleURI is enabled. The bundles are downloaded\n as if the list was advertised from the --bundle-uri option.\n\nMany patches in this series were adapted from Ævar's v2 RFC [1]. He is\nretained as author and I added myself as co-author only if the modifications\nwere significant.\n\n[1]\nhttps://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/\n\n * Patches 1-7 are mostly taken from [1], again with mostly minor updates.\n The one major difference is the packet line format being a single\n key=value format instead of a sequence of pairs. (In v3, these commits\n are significantly reorganized from [1].)\n\n * Patches 8-11 finish off the ability for the client to notice the\n capability, request the values, and download bundles before continuing\n with the rest of the download.\n\nOne thing that is not handled here but could be handled in a future change\nis to disconnect from the origin Git server while downloading the bundle\nURIs, then reconnecting afterwards. This does not make any difference for\nHTTPS, but SSH may benefit from the reduced connection time. The git clone\n--bundle-uri option did not suffer from this because the bundles are\ndownloaded before the server connection begins.\n\nAfter this series, there is one more before the original scope of the plan\nis complete: using creation tokens as a heuristic. See [2] for the RFC\nversion of those patches.\n\n[2] https://github.com/derrickstolee/git/pull/22\n\n\nUpdates in v3\n=============\n\nMost of these updates are due to Victoria's very thorough review. Thanks!\n\n * What was patch 2 was split to be better understood.\n * The new patch 2 is only the new test script infrastructure for testing\n whether or not the server provides the bundle-uri capability. It is\n extended with other more complicated examples in later patches. The name\n was rewritten from lib-t5730-*.sh to lib-bundle-uri-protocol.sh and the\n variable names are renamed with the BUNDLE_URI_ prefix.\n * The new patch 3 performs the basic client interaction with the\n 'bundle-uri' command, while still not being fully wired up on the server\n side. The tests do check that the client requests the bundle-uri command\n after seeing it in the server's capabilities. One important difference\n from earlier is that the check for server_supports_v2() was moved into\n the get_bundle_uri() method (underneath the vtable) because we need to\n check the handshake before calling that method. It makes most sense to\n put the handshake call there, so do it from the start.\n * Patch 4 carefully tests how the transfer.bundleURI config blocks the\n client-side request of the bundle-uri command. Later tests will use the\n GIT_TEST_BUNDLE_URI environment variable instead.\n * The new Patch 5 renames got_remote_heads to finished_handshake in 'struct\n git_transport_data' and that's it. That new value is then used in patch 6\n to indicate if we need to request the handshake in the bundle URI logic.\n * Patch 6 creates the ls-remote helper in 'test-tool bundle-uri' as before,\n but now only makes use of the finished_handshake member instead of\n creating a new one. The test helper represents an example consumer of\n transport_get_remote_bundle_uri() without first doing the server-side\n handshake, which motivates several of the placements of code within that\n method and get_bundle_uri() earlier in the series. The \"quiet\" option is\n also removed to simplify the test helper and to always communicate the\n inner errors to the user.\n * Patch 7 adds the server-side listing of bundle.* config values. The test\n scripts around these config values have been cleaned up since the\n previous version.\n * Patch 8 has another iteration of strbuf_strip_file_from_path() taking the\n feedback from Victoria and Ævar.\n * Patch 9 adds the relative path logic. The definition of the base path is\n clarified in the commit message and comments. An additional test shows\n what happens if the server advertises too many parent paths\n (unfortunately, a die(), and this is marked for cleanup later).\n * Patch 10 is identical to the old patch 8.\n * Patch 11 completes the work by having the client download the bundles\n provided by the server list. It fixes an if/else that should have been an\n if/else-if. A new test checks that the --bundle-uri=X option overrides\n the server advertisement.\n\n\nUpdates in v2\n=============\n\n * Commit messages now refer to protocol v2 \"commands\" not \"verbs\".\n * Several edits were made to gitprotocol-v2.txt thanks to Victoria's\n thorough review.\n * strbuf_parent_directory() is renamed strbuf_strip_file_from_path() to\n make it more clear how it behaves when ending with a slash.\n\nThanks,\n\n * Stolee\n\nDerrick Stolee (6):\n transport: rename got_remote_heads\n bundle-uri: serve bundle.* keys from config\n strbuf: introduce strbuf_strip_file_from_path()\n bundle-uri: allow relative URLs in bundle lists\n bundle-uri: download bundles from an advertised list\n clone: unbundle the advertised bundles\n\nÆvar Arnfjörð Bjarmason (5):\n protocol v2: add server-side \"bundle-uri\" skeleton\n t: create test harness for 'bundle-uri' command\n clone: request the 'bundle-uri' command when available\n bundle-uri client: add boolean transfer.bundleURI setting\n bundle-uri client: add helper for testing server\n\n Documentation/config/transfer.txt | 6 +\n Documentation/gitprotocol-v2.txt | 201 +++++++++++++++++++++++\n builtin/clone.c | 22 +++\n bundle-uri.c | 87 +++++++++-\n bundle-uri.h | 32 ++++\n connect.c | 44 +++++\n remote.h | 5 +\n serve.c | 6 +\n strbuf.c | 6 +\n strbuf.h | 11 ++\n t/helper/test-bundle-uri.c | 48 ++++++\n t/lib-bundle-uri-protocol.sh | 212 +++++++++++++++++++++++++\n t/t5601-clone.sh | 59 +++++++\n t/t5701-git-serve.sh | 40 ++++-\n t/t5730-protocol-v2-bundle-uri-file.sh | 17 ++\n t/t5731-protocol-v2-bundle-uri-git.sh | 17 ++\n t/t5732-protocol-v2-bundle-uri-http.sh | 17 ++\n t/t5750-bundle-uri-parse.sh | 82 ++++++++++\n t/t9119-git-svn-info.sh | 2 +-\n t/test-lib-functions.sh | 7 +\n transport-helper.c | 13 ++\n transport-internal.h | 7 +\n transport.c | 88 ++++++++--\n transport.h | 19 +++\n 24 files changed, 1036 insertions(+), 12 deletions(-)\n create mode 100644 t/lib-bundle-uri-protocol.sh\n create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh\n create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh\n create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh\n\n\nbase-commit: c03801e19cb8ab36e9c0d17ff3d5e0c3b0f24193\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1400%2Fderrickstolee%2Fbundle-redo%2Fadvertise-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1400/derrickstolee/bundle-redo/advertise-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1400\n\nRange-diff vs v2:\n\n 1: beae335b855 = 1: beae335b855 protocol v2: add server-side \"bundle-uri\" skeleton\n -: ----------- > 2: fcdfef2012a t: create test harness for 'bundle-uri' command\n 2: 0d85aef965d ! 3: a0188ae39c6 bundle-uri client: add minimal NOOP client\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - bundle-uri client: add minimal NOOP client\n + clone: request the 'bundle-uri' command when available\n \n - Set up all the needed client parts of the \"bundle-uri\" protocol\n - extension, without actually doing anything with the bundle URIs.\n + Set up all the needed client parts of the 'bundle-uri' protocol v2\n + command, without actually doing anything with the bundle URIs.\n \n - I.e. if the server says it supports \"bundle-uri\" we'll issue a\n - command=bundle-uri after command=ls-refs when we're cloning. We'll\n - parse the returned output using the code already tested for in\n - t5750-bundle-uri-parse.sh.\n + If the server says it supports 'bundle-uri' teach Git to issue the\n + 'bundle-uri' command after the 'ls-refs' during 'git clone'. The\n + returned key=value pairs are passed to the bundle list code which is\n + tested using a different ingest mechanism in t5750-bundle-uri-parse.sh.\n \n - What we aren't doing is actually acting on that data, i.e. downloading\n - the bundle(s) before we get to doing the command=fetch, and adjusting\n - our negotiation dialog appropriately. I'll do that in subsequent\n - commits.\n + At this point, Git does nothing with that bundle list. It will not\n + download any of the bundles. That will come in a later change after\n + these protocol bits are finalized.\n \n - There's a question of what level of encapsulation we should use here,\n - I've opted to use connect.h in clone.c, but we could also e.g. make\n - transport_get_remote_refs() invoke this, i.e. make it implicitly get\n - the bundle-uri list for later steps.\n -\n - This approach means that we don't \"support\" this in \"git fetch\" for\n - now. I'm starting with the case of initial clones, although as noted\n - in preceding commits to the protocol documentation nothing about this\n - approach precludes getting bundles on incremental fetches.\n -\n - For the t5732-protocol-v2-bundle-uri-http.sh it's not easy to set\n - environment variables for git-upload-pack (it's started by Apache), so\n - let's skip the test under T5730_HTTP, and add unused T5730_{FILE,GIT}\n - prerequisites for consistency and future use.\n + The no-op client is initially used only by 'git clone' to test the basic\n + functionality, and eventually will bootstrap the initial download of Git\n + objects during a fresh clone. The bundle URI client will not be\n + integrated into other fetches until a mechanism is created to select a\n + subset of bundles for download.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: Derrick Stolee \n \n ## builtin/clone.c ##\n -@@\n - #include \"iterator.h\"\n - #include \"sigchain.h\"\n - #include \"branch.h\"\n -+#include \"connect.h\"\n - #include \"remote.h\"\n - #include \"run-command.h\"\n - #include \"connected.h\"\n @@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (refs)\n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n @@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n \n \n - ## bundle-uri.c ##\n -@@ bundle-uri.c: int bundle_uri_advertise(struct repository *r, struct strbuf *value)\n - {\n - \tstatic int advertise_bundle_uri = -1;\n - \n -+\tif (value &&\n -+\t git_env_bool(\"GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE\", 0))\n -+\t\tstrbuf_addstr(value, \"test-unknown-capability-value\");\n -+\n - \tif (advertise_bundle_uri != -1)\n - \t\tgoto cached;\n - \n -\n ## connect.c ##\n @@\n #include \"version.h\"\n @@ connect.c: static void send_capabilities(int fd_out, struct packet_reader *reade\n +\tpacket_write_fmt(fd_out, \"command=bundle-uri\\n\");\n +\tpacket_delim(fd_out);\n +\n -+\t/* Send options */\n -+\tif (git_env_bool(\"GIT_TEST_PROTOCOL_BAD_BUNDLE_URI\", 0))\n -+\t\tpacket_write_fmt(fd_out, \"test-bad-client\\n\");\n +\tpacket_flush(fd_out);\n +\n +\t/* Process response from server */\n @@ remote.h: struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \n /*\n \n - ## t/lib-t5730-protocol-v2-bundle-uri.sh (new) ##\n -@@\n -+# Included from t573*-protocol-v2-bundle-uri-*.sh\n -+\n -+T5730_PARENT=\n -+T5730_URI=\n -+T5730_BUNDLE_URI=\n -+case \"$T5730_PROTOCOL\" in\n -+file)\n -+\tT5730_PARENT=file_parent\n -+\tT5730_URI=\"file://$PWD/file_parent\"\n -+\tT5730_BUNDLE_URI=\"$T5730_URI/fake.bdl\"\n -+\ttest_set_prereq T5730_FILE\n -+\t;;\n -+git)\n -+\t. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n -+\tstart_git_daemon --export-all --enable=receive-pack\n -+\tT5730_PARENT=\"$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\"\n -+\tT5730_URI=\"$GIT_DAEMON_URL/parent\"\n -+\tT5730_BUNDLE_URI=\"https://example.com/fake.bdl\"\n -+\ttest_set_prereq T5730_GIT\n -+\t;;\n -+http)\n -+\t. \"$TEST_DIRECTORY\"/lib-httpd.sh\n -+\tstart_httpd\n -+\tT5730_PARENT=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\"\n -+\tT5730_URI=\"$HTTPD_URL/smart/http_parent\"\n -+\tT5730_BUNDLE_URI=\"https://example.com/fake.bdl\"\n -+\ttest_set_prereq T5730_HTTP\n -+\t;;\n -+*)\n -+\tBUG \"Need to pass valid T5730_PROTOCOL (was $T5730_PROTOCOL)\"\n -+\t;;\n -+esac\n -+\n -+test_expect_success \"setup protocol v2 $T5730_PROTOCOL:// tests\" '\n -+\tgit init \"$T5730_PARENT\" &&\n -+\ttest_commit -C \"$T5730_PARENT\" one &&\n -+\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs true\n -+'\n + ## t/lib-bundle-uri-protocol.sh ##\n +@@ t/lib-bundle-uri-protocol.sh: test_expect_success \"connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav\n + \t# Server advertised bundle-uri capability\n + \tgrep \"< bundle-uri\" log\n + '\n +\n -+# Poor man's URI escaping. Good enough for the test suite whose trash\n -+# directory has a space in it. See 93c3fcbe4d4 (git-svn: attempt to\n -+# mimic SVN 1.7 URL canonicalization, 2012-07-28) for prior art.\n -+test_uri_escape() {\n -+\tsed 's/ /%20/g'\n -+}\n -+\n -+case \"$T5730_PROTOCOL\" in\n -+http)\n -+\ttest_expect_success \"setup config for $T5730_PROTOCOL:// tests\" '\n -+\t\tgit -C \"$T5730_PARENT\" config http.receivepack true\n -+\t'\n -+\t;;\n -+*)\n -+\t;;\n -+esac\n -+T5730_BUNDLE_URI_ESCAPED=$(echo \"$T5730_BUNDLE_URI\" | test_uri_escape)\n -+\n -+test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: no bundle-uri\" '\n -+\ttest_when_finished \"rm -f log\" &&\n -+\ttest_when_finished \"git -C \\\"$T5730_PARENT\\\" config uploadpack.advertiseBundleURIs true\" &&\n -+\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs false &&\n ++test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris\" '\n ++\ttest_when_finished \"rm -rf log cloned\" &&\n +\n +\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n +\tgit \\\n +\t\t-c protocol.version=2 \\\n -+\t\tls-remote --symref \"$T5730_URI\" \\\n -+\t\t>actual 2>err &&\n -+\n -+\t# Server responded using protocol v2\n -+\tgrep \"< version 2\" log &&\n -+\n -+\t! grep bundle-uri log\n -+'\n -+\n -+test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: have bundle-uri\" '\n -+\ttest_when_finished \"rm -f log\" &&\n -+\n -+\ttest_config -C \"$T5730_PARENT\" \\\n -+\t\tuploadpack.bundleURI \"$T5730_BUNDLE_URI_ESCAPED\" &&\n -+\n -+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n -+\tgit \\\n -+\t\t-c protocol.version=2 \\\n -+\t\tls-remote --symref \"$T5730_URI\" \\\n ++\t\tclone \"$BUNDLE_URI_REPO_URI\" cloned \\\n +\t\t>actual 2>err &&\n +\n +\t# Server responded using protocol v2\n +\tgrep \"< version 2\" log &&\n +\n +\t# Server advertised bundle-uri capability\n -+\tgrep bundle-uri log\n -+'\n ++\tgrep \"< bundle-uri\" log &&\n +\n -+test_expect_success !T5730_HTTP \"bad client with $T5730_PROTOCOL:// using protocol v2\" '\n -+\ttest_when_finished \"rm -f log\" &&\n -+\n -+\ttest_config -C \"$T5730_PARENT\" uploadpack.bundleURI \\\n -+\t\t\"$T5730_BUNDLE_URI_ESCAPED\" &&\n -+\n -+\tcat >err.expect <<-\\EOF &&\n -+\tCloning into '\"'\"'child'\"'\"'...\n -+\tEOF\n -+\tcase \"$T5730_PROTOCOL\" in\n -+\tfile)\n -+\t\tcat >fatal-bundle-uri.expect <<-\\EOF\n -+\t\tfatal: bundle-uri: unexpected argument: '\"'\"'test-bad-client'\"'\"'\n -+\t\tEOF\n -+\t\t;;\n -+\t*)\n -+\t\tcat >fatal.expect <<-\\EOF\n -+\t\tfatal: read error: Connection reset by peer\n -+\t\tEOF\n -+\t\t;;\n -+\tesac &&\n -+\n -+\ttest_when_finished \"rm -rf child\" &&\n -+\ttest_must_fail ok=sigpipe env \\\n -+\t\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n -+\t\tGIT_TEST_PROTOCOL_BAD_BUNDLE_URI=true \\\n -+\t\tgit -c protocol.version=2 \\\n -+\t\tclone \"$T5730_URI\" child \\\n -+\t\t>out 2>err &&\n -+\ttest_must_be_empty out &&\n -+\n -+\tgrep -v -e ^fatal: -e ^error: err >err.actual &&\n -+\ttest_cmp err.expect err.actual &&\n -+\n -+\tcase \"$T5730_PROTOCOL\" in\n -+\tfile)\n -+\t\t# Due to general race conditions with client/server replies we\n -+\t\t# may or may not get \"fatal: the remote end hung up\n -+\t\t# expectedly\" here\n -+\t\tgrep \"^fatal: bundle-uri:\" err >fatal-bundle-uri.actual &&\n -+\t\ttest_cmp fatal-bundle-uri.expect fatal-bundle-uri.actual\n -+\t\t;;\n -+\t*)\n -+\t\tgrep \"^fatal:\" err >fatal.actual &&\n -+\t\t# Due to the same race conditions this might be\n -+\t\t# \"fatal: read error: Connection reset by peer\", \"fatal: the remote end\n -+\t\t# hung up unexpectedly\" etc.\n -+\t\tcat fatal.actual &&\n -+\t\ttest_file_not_empty fatal.actual\n -+\t\t;;\n -+\tesac &&\n -+\n -+\tgrep \"clone> test-bad-client$\" log >sent-bad-request &&\n -+\ttest_file_not_empty sent-bad-request\n ++\t# Client issued bundle-uri command\n ++\tgrep \"> command=bundle-uri\" log\n +'\n \n - ## t/t5730-protocol-v2-bundle-uri-file.sh (new) ##\n -@@\n -+#!/bin/sh\n -+\n -+test_description=\"Test bundle-uri with protocol v2 and 'file://' transport\"\n -+\n -+TEST_NO_CREATE_REPO=1\n -+\n -+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n -+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n -+\n -+. ./test-lib.sh\n -+\n -+# Test protocol v2 with 'file://' transport\n -+#\n -+T5730_PROTOCOL=file\n -+. \"$TEST_DIRECTORY\"/lib-t5730-protocol-v2-bundle-uri.sh\n -+\n -+test_expect_success \"unknown capability value with $T5730_PROTOCOL:// using protocol v2\" '\n -+\ttest_when_finished \"rm -f log\" &&\n -+\n -+\ttest_config -C \"$T5730_PARENT\" \\\n -+\t\tuploadpack.bundleURI \"$T5730_BUNDLE_URI_ESCAPED\" &&\n -+\n -+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n -+\tGIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE=true \\\n -+\tgit \\\n -+\t\t-c protocol.version=2 \\\n -+\t\tls-remote --symref \"$T5730_URI\" \\\n -+\t\t>actual 2>err &&\n -+\n -+\t# Server responded using protocol v2\n -+\tgrep \"< version 2\" log &&\n -+\n -+\tgrep \"> bundle-uri=test-unknown-capability-value\" log\n -+'\n -+\n -+test_done\n -\n - ## t/t5731-protocol-v2-bundle-uri-git.sh (new) ##\n -@@\n -+#!/bin/sh\n -+\n -+test_description=\"Test bundle-uri with protocol v2 and 'git://' transport\"\n -+\n -+TEST_NO_CREATE_REPO=1\n -+\n -+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n -+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n -+\n -+. ./test-lib.sh\n -+\n -+# Test protocol v2 with 'git://' transport\n -+#\n -+T5730_PROTOCOL=git\n -+. \"$TEST_DIRECTORY\"/lib-t5730-protocol-v2-bundle-uri.sh\n -+\n -+test_done\n -\n - ## t/t5732-protocol-v2-bundle-uri-http.sh (new) ##\n -@@\n -+#!/bin/sh\n -+\n -+test_description=\"Test bundle-uri with protocol v2 and 'git://' transport\"\n -+\n -+TEST_NO_CREATE_REPO=1\n -+\n -+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n -+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n -+\n -+. ./test-lib.sh\n -+\n -+# Test protocol v2 with 'git://' transport\n -+#\n -+T5730_PROTOCOL=http\n -+. \"$TEST_DIRECTORY\"/lib-t5730-protocol-v2-bundle-uri.sh\n -+\n -+test_done\n -\n ## transport-helper.c ##\n @@ transport-helper.c: static struct ref *get_refs_list_using_list(struct transport *transport,\n \treturn ret;\n @@ transport.c: static struct ref *get_refs_via_connect(struct transport *transport\n +\t\tinit_bundle_list(transport->bundles);\n +\t}\n +\n ++\t/*\n ++\t * \"Support\" protocol v0 and v2 without bundle-uri support by\n ++\t * silently degrading to a NOOP.\n ++\t */\n ++\tif (!server_supports_v2(\"bundle-uri\", 0))\n ++\t\treturn 0;\n ++\n +\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n +\t\t\t PACKET_READ_CHOMP_NEWLINE |\n +\t\t\t PACKET_READ_GENTLE_ON_EOF);\n @@ transport.c: int transport_fetch_refs(struct transport *transport, struct ref *r\n +\tconst struct transport_vtable *vtable = transport->vtable;\n +\n +\t/* Check config only once. */\n -+\tif (transport->got_remote_bundle_uri++)\n ++\tif (transport->got_remote_bundle_uri)\n +\t\treturn 0;\n ++\ttransport->got_remote_bundle_uri = 1;\n +\n -+\t/*\n -+\t * \"Support\" protocol v0 and v2 without bundle-uri support by\n -+\t * silently degrading to a NOOP.\n -+\t */\n -+\tif (!server_supports_v2(\"bundle-uri\", 0))\n -+\t\treturn 0;\n -+\n -+\t/*\n -+\t * This is intentionally below the transport.injectBundleURI,\n -+\t * we want to be able to inject into protocol v0, or into the\n -+\t * dialog of a server who doesn't support this.\n -+\t */\n +\tif (!vtable->get_bundle_uri)\n +\t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n +\n 5: 93397468931 ! 4: e46118e60f7 bundle-uri client: add boolean transfer.bundleURI setting\n @@ Documentation/config/transfer.txt: transfer.unpackLimit::\n +\tbundles before continuing the clone through the Git protocol.\n +\tDefaults to `false`.\n \n - ## t/lib-t5730-protocol-v2-bundle-uri.sh ##\n -@@\n - # Included from t573*-protocol-v2-bundle-uri-*.sh\n + ## t/lib-bundle-uri-protocol.sh ##\n +@@ t/lib-bundle-uri-protocol.sh: test_expect_success \"connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav\n + '\n \n -+GIT_TEST_BUNDLE_URI=1\n -+export GIT_TEST_BUNDLE_URI\n + test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris\" '\n +-\ttest_when_finished \"rm -rf log cloned\" &&\n ++\ttest_when_finished \"rm -rf log cloned cloned2\" &&\n + \n + \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n ++\tGIT_TEST_BUNDLE_URI=0 \\\n + \tgit \\\n + \t\t-c protocol.version=2 \\\n + \t\tclone \"$BUNDLE_URI_REPO_URI\" cloned \\\n +@@ t/lib-bundle-uri-protocol.sh: test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque\n + \t# Server advertised bundle-uri capability\n + \tgrep \"< bundle-uri\" log &&\n + \n ++\t# Client did not issue bundle-uri command\n ++\t! grep \"> command=bundle-uri\" log &&\n ++\n ++\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n ++\tgit \\\n ++\t\t-c transfer.bundleURI=true \\\n ++\t\t-c protocol.version=2 \\\n ++\t\tclone \"$BUNDLE_URI_REPO_URI\" cloned2 \\\n ++\t\t>actual 2>err &&\n +\n - T5730_PARENT=\n - T5730_URI=\n - T5730_BUNDLE_URI=\n ++\t# Server responded using protocol v2\n ++\tgrep \"< version 2\" log &&\n ++\n ++\t# Server advertised bundle-uri capability\n ++\tgrep \"< bundle-uri\" log &&\n ++\n + \t# Client issued bundle-uri command\n + \tgrep \"> command=bundle-uri\" log\n + '\n \n ## transport.c ##\n @@ transport.c: int transport_fetch_refs(struct transport *transport, struct ref *refs)\n \n - int transport_get_remote_bundle_uri(struct transport *transport, int quiet)\n + int transport_get_remote_bundle_uri(struct transport *transport)\n {\n +\tint value = 0;\n \tconst struct transport_vtable *vtable = transport->vtable;\n \n \t/* Check config only once. */\n -@@ transport.c: int transport_get_remote_bundle_uri(struct transport *transport, int quiet)\n +@@ transport.c: int transport_get_remote_bundle_uri(struct transport *transport)\n \t\treturn 0;\n + \ttransport->got_remote_bundle_uri = 1;\n \n - \t/*\n --\t * This is intentionally below the transport.injectBundleURI,\n --\t * we want to be able to inject into protocol v0, or into the\n --\t * dialog of a server who doesn't support this.\n -+\t * Don't use bundle-uri at all, if configured not to. Only proceed\n -+\t * if GIT_TEST_BUNDLE_URI=1 or transfer.bundleURI=true.\n - \t */\n ++\t/*\n ++\t * Don't request bundle-uri from the server unless configured to\n ++\t * do so by GIT_TEST_BUNDLE_URI=1 or transfer.bundleURI=true.\n ++\t */\n +\tif (!git_env_bool(\"GIT_TEST_BUNDLE_URI\", 0) &&\n +\t (git_config_get_bool(\"transfer.bundleuri\", &value) || !value))\n +\t\treturn 0;\n +\n - \tif (!vtable->get_bundle_uri) {\n - \t\tif (quiet)\n - \t\t\treturn -1;\n + \tif (!vtable->get_bundle_uri)\n + \t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n + \n -: ----------- > 5: b009b4f58ea transport: rename got_remote_heads\n 3: c3269a24b57 ! 6: 46a58e83caf bundle-uri client: add helper for testing server\n @@ Commit message\n for issuing protocol v2 \"bundle-uri\" commands to a server, and to the\n parsing routines in bundle-uri.c.\n \n - Since in the \"git clone\" case we'll have already done the handshake(),\n - but not here, introduce a \"got_advertisement\" state along with\n - \"got_remote_heads\". It seems to me that the \"got_remote_heads\" is\n - badly named in the first place, and the whole logic of eagerly getting\n - ls-refs on handshake() or not could be refactored somewhat, but let's\n - not do that now, and instead just add another self-documenting state\n - variable.\n + In the \"git clone\" case we'll have already done the handshake(),\n + but not here. Add an extra case to check for this handshake in\n + get_bundle_uri() for ease of use for future callers. Rename the existing\n + 'got_remote_heads' to 'finished_handshake' to make it more clear what\n + that bit represents.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: Derrick Stolee \n \n - ## builtin/clone.c ##\n -@@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n - \t * Populate transport->got_remote_bundle_uri and\n - \t * transport->bundle_uri. We might get nothing.\n - \t */\n --\ttransport_get_remote_bundle_uri(transport);\n -+\ttransport_get_remote_bundle_uri(transport, 1);\n - \n - \tif (mapped_refs) {\n - \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n -\n ## t/helper/test-bundle-uri.c ##\n @@\n #include \"bundle-uri.h\"\n @@ t/helper/test-bundle-uri.c: usage:\n +\tif (server_options.nr)\n +\t\ttransport->server_options = &server_options;\n +\n -+\tif (transport_get_remote_bundle_uri(transport, 0) < 0) {\n ++\tif (transport_get_remote_bundle_uri(transport) < 0) {\n +\t\terror(_(\"could not get the bundle-uri list\"));\n +\t\tstatus = 1;\n +\t\tgoto cleanup;\n @@ t/helper/test-bundle-uri.c: int cmd__bundle_uri(int argc, const char **argv)\n \n usage:\n \n - ## t/lib-t5730-protocol-v2-bundle-uri.sh ##\n -@@ t/lib-t5730-protocol-v2-bundle-uri.sh: esac\n - test_expect_success \"setup protocol v2 $T5730_PROTOCOL:// tests\" '\n - \tgit init \"$T5730_PARENT\" &&\n - \ttest_commit -C \"$T5730_PARENT\" one &&\n --\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs true\n -+\tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs true &&\n -+\tgit -C \"$T5730_PARENT\" config bundle.version 1 &&\n -+\tgit -C \"$T5730_PARENT\" config bundle.mode all\n + ## t/lib-bundle-uri-protocol.sh ##\n +@@ t/lib-bundle-uri-protocol.sh: test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque\n + \t# Client issued bundle-uri command\n + \tgrep \"> command=bundle-uri\" log\n '\n - \n - # Poor man's URI escaping. Good enough for the test suite whose trash\n -@@ t/lib-t5730-protocol-v2-bundle-uri.sh: test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: no bundl\n - \tgit -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs false &&\n - \n - \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n --\tgit \\\n --\t\t-c protocol.version=2 \\\n --\t\tls-remote --symref \"$T5730_URI\" \\\n -+\ttest-tool bundle-uri \\\n -+\t\tls-remote \"$T5730_URI\" \\\n - \t\t>actual 2>err &&\n - \n - \t# Server responded using protocol v2\n -@@ t/lib-t5730-protocol-v2-bundle-uri.sh: test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: have bun\n - \ttest_when_finished \"rm -f log\" &&\n - \n - \ttest_config -C \"$T5730_PARENT\" \\\n --\t\tuploadpack.bundleURI \"$T5730_BUNDLE_URI_ESCAPED\" &&\n -+\t\tbundle.only.uri \"$T5730_BUNDLE_URI_ESCAPED\" &&\n - \n - \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n --\tgit \\\n --\t\t-c protocol.version=2 \\\n --\t\tls-remote --symref \"$T5730_URI\" \\\n -+\ttest-tool bundle-uri \\\n -+\t\tls-remote \"$T5730_URI\" \\\n - \t\t>actual 2>err &&\n - \n - \t# Server responded using protocol v2\n -@@ t/lib-t5730-protocol-v2-bundle-uri.sh: test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: have bun\n - test_expect_success !T5730_HTTP \"bad client with $T5730_PROTOCOL:// using protocol v2\" '\n - \ttest_when_finished \"rm -f log\" &&\n - \n --\ttest_config -C \"$T5730_PARENT\" uploadpack.bundleURI \\\n --\t\t\"$T5730_BUNDLE_URI_ESCAPED\" &&\n -+\ttest_config -C \"$T5730_PARENT\" \\\n -+\t\tbundle.only.uri \"$T5730_BUNDLE_URI_ESCAPED\" &&\n - \n - \tcat >err.expect <<-\\EOF &&\n - \tCloning into '\"'\"'child'\"'\"'...\n -@@ t/lib-t5730-protocol-v2-bundle-uri.sh: test_expect_success !T5730_HTTP \"bad client with $T5730_PROTOCOL:// using protoc\n - \tgrep \"clone> test-bad-client$\" log >sent-bad-request &&\n - \ttest_file_not_empty sent-bad-request\n - '\n -+\n -+test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2\" '\n -+\ttest_when_finished \"rm -f log\" &&\n +\n -+\ttest_config -C \"$T5730_PARENT\" \\\n -+\t\tbundle.only.uri \"$T5730_BUNDLE_URI_ESCAPED\" &&\n ++test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2\" '\n ++\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n ++\t\tbundle.only.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED\" &&\n +\n +\t# All data about bundle URIs\n +\tcat >expect <<-EOF &&\n @@ t/lib-t5730-protocol-v2-bundle-uri.sh: test_expect_success !T5730_HTTP \"bad clie\n +\t\tversion = 1\n +\t\tmode = all\n +\tEOF\n -+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n ++\n +\ttest-tool bundle-uri \\\n +\t\tls-remote \\\n -+\t\t\"$T5730_URI\" \\\n ++\t\t\"$BUNDLE_URI_REPO_URI\" \\\n +\t\t>actual &&\n +\ttest_cmp_config_output expect actual\n +'\n +\n -+test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2 and extra data\" '\n -+\ttest_when_finished \"rm -f log\" &&\n -+\n -+\ttest_config -C \"$T5730_PARENT\" \\\n -+\t\tbundle.only.uri \"$T5730_BUNDLE_URI_ESCAPED\" &&\n ++test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2 and extra data\" '\n ++\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n ++\t\tbundle.only.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED\" &&\n +\n +\t# Extra data should be ignored\n -+\ttest_config -C \"$T5730_PARENT\" bundle.only.extra bogus &&\n ++\ttest_config -C \"$BUNDLE_URI_PARENT\" bundle.only.extra bogus &&\n +\n +\t# All data about bundle URIs\n +\tcat >expect <<-EOF &&\n @@ t/lib-t5730-protocol-v2-bundle-uri.sh: test_expect_success !T5730_HTTP \"bad clie\n +\t\tversion = 1\n +\t\tmode = all\n +\tEOF\n -+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n ++\n +\ttest-tool bundle-uri \\\n +\t\tls-remote \\\n -+\t\t\"$T5730_URI\" \\\n ++\t\t\"$BUNDLE_URI_REPO_URI\" \\\n +\t\t>actual &&\n +\ttest_cmp_config_output expect actual\n +'\n \n ## transport.c ##\n -@@ transport.c: struct git_transport_data {\n - \tstruct git_transport_options options;\n - \tstruct child_process *conn;\n - \tint fd[2];\n -+\tunsigned got_advertisement : 1;\n - \tunsigned got_remote_heads : 1;\n - \tenum protocol_version version;\n - \tstruct oid_array extra_have;\n -@@ transport.c: static struct ref *handshake(struct transport *transport, int for_push,\n - \t\tBUG(\"unknown protocol version\");\n - \t}\n - \tdata->got_remote_heads = 1;\n -+\tdata->got_advertisement = 1;\n - \ttransport->hash_algo = reader.hash_algo;\n - \n - \tif (reader.line_peeked)\n @@ transport.c: static int get_bundle_uri(struct transport *transport)\n \t\tinit_bundle_list(transport->bundles);\n \t}\n \n -+\tif (!data->got_advertisement) {\n -+\t\tstruct ref *refs;\n -+\t\tstruct git_transport_data *data = transport->data;\n -+\t\tenum protocol_version version;\n ++\tif (!data->finished_handshake) {\n ++\t\tstruct ref *refs = handshake(transport, 0, NULL, 0);\n +\n -+\t\trefs = handshake(transport, 0, NULL, 0);\n -+\t\tversion = data->version;\n -+\n -+\t\tswitch (version) {\n -+\t\tcase protocol_v2:\n -+\t\t\tassert(!refs);\n -+\t\t\tbreak;\n -+\t\tcase protocol_v0:\n -+\t\tcase protocol_v1:\n -+\t\tcase protocol_unknown_version:\n -+\t\t\tassert(refs);\n -+\t\t\tbreak;\n -+\t\t}\n ++\t\tif (refs)\n ++\t\t\tfree_refs(refs);\n +\t}\n +\n -+\t/*\n -+\t * \"Support\" protocol v0 and v2 without bundle-uri support by\n -+\t * silently degrading to a NOOP.\n -+\t */\n -+\tif (!server_supports_v2(\"bundle-uri\", 0))\n -+\t\treturn 0;\n -+\n - \tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n - \t\t\t PACKET_READ_CHOMP_NEWLINE |\n - \t\t\t PACKET_READ_GENTLE_ON_EOF);\n -@@ transport.c: int transport_fetch_refs(struct transport *transport, struct ref *refs)\n - \treturn rc;\n - }\n - \n --int transport_get_remote_bundle_uri(struct transport *transport)\n -+int transport_get_remote_bundle_uri(struct transport *transport, int quiet)\n - {\n - \tconst struct transport_vtable *vtable = transport->vtable;\n - \n -@@ transport.c: int transport_get_remote_bundle_uri(struct transport *transport)\n - \tif (transport->got_remote_bundle_uri++)\n - \t\treturn 0;\n - \n --\t/*\n --\t * \"Support\" protocol v0 and v2 without bundle-uri support by\n --\t * silently degrading to a NOOP.\n --\t */\n --\tif (!server_supports_v2(\"bundle-uri\", 0))\n --\t\treturn 0;\n --\n \t/*\n - \t * This is intentionally below the transport.injectBundleURI,\n - \t * we want to be able to inject into protocol v0, or into the\n - \t * dialog of a server who doesn't support this.\n - \t */\n --\tif (!vtable->get_bundle_uri)\n -+\tif (!vtable->get_bundle_uri) {\n -+\t\tif (quiet)\n -+\t\t\treturn -1;\n - \t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n -+\t}\n - \n - \tif (vtable->get_bundle_uri(transport) < 0)\n - \t\treturn error(_(\"could not retrieve server-advertised bundle-uri list\"));\n -\n - ## transport.h ##\n -@@ transport.h: const struct ref *transport_get_remote_refs(struct transport *transport,\n - /**\n - * Retrieve bundle URI(s) from a remote. Populates \"struct\n - * transport\"'s \"bundle_uri\" and \"got_remote_bundle_uri\".\n -+ *\n -+ * With `quiet=1` it will not complain if the serve doesn't support\n -+ * the protocol, but only if we discover the server uses it, and\n -+ * encounter issues then.\n - */\n --int transport_get_remote_bundle_uri(struct transport *transport);\n -+int transport_get_remote_bundle_uri(struct transport *transport, int quiet);\n - \n - /*\n - * Fetch the hash algorithm used by a remote.\n + \t * \"Support\" protocol v0 and v2 without bundle-uri support by\n + \t * silently degrading to a NOOP.\n 4: cd906f6d981 ! 7: acc5a8f57f9 bundle-uri: serve bundle.* keys from config\n @@ bundle-uri.c: int bundle_uri_command(struct repository *r,\n \tpacket_writer_flush(&writer);\n \n \n - ## t/lib-t5730-protocol-v2-bundle-uri.sh ##\n -@@ t/lib-t5730-protocol-v2-bundle-uri.sh: test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2\" '\n + ## t/lib-bundle-uri-protocol.sh ##\n +@@ t/lib-bundle-uri-protocol.sh: test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol\n \t[bundle]\n \t\tversion = 1\n \t\tmode = all\n +\t[bundle \"only\"]\n -+\t\turi = $T5730_BUNDLE_URI_ESCAPED\n ++\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED\n \tEOF\n - \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n + \n ++\tGIT_TEST_BUNDLE_URI=1 \\\n \ttest-tool bundle-uri \\\n -@@ t/lib-t5730-protocol-v2-bundle-uri.sh: test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2 and ext\n + \t\tls-remote \\\n + \t\t\"$BUNDLE_URI_REPO_URI\" \\\n +@@ t/lib-bundle-uri-protocol.sh: test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol\n \t[bundle]\n \t\tversion = 1\n \t\tmode = all\n +\t[bundle \"only\"]\n -+\t\turi = $T5730_BUNDLE_URI_ESCAPED\n -+\tEOF\n -+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n ++\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED\n + \tEOF\n + \n ++\tGIT_TEST_BUNDLE_URI=1 \\\n +\ttest-tool bundle-uri \\\n +\t\tls-remote \\\n -+\t\t\"$T5730_URI\" \\\n ++\t\t\"$BUNDLE_URI_REPO_URI\" \\\n +\t\t>actual &&\n +\ttest_cmp_config_output expect actual\n +'\n +\n -+\n -+test_expect_success \"ls-remote with $T5730_PROTOCOL:// using protocol v2 with list\" '\n -+\ttest_when_finished \"rm -f log\" &&\n -+\n -+\ttest_config -C \"$T5730_PARENT\" \\\n -+\t\tbundle.bundle1.uri \"$T5730_BUNDLE_URI_ESCAPED-1.bdl\" &&\n -+\ttest_config -C \"$T5730_PARENT\" \\\n -+\t\tbundle.bundle2.uri \"$T5730_BUNDLE_URI_ESCAPED-2.bdl\" &&\n -+\ttest_config -C \"$T5730_PARENT\" \\\n -+\t\tbundle.bundle3.uri \"$T5730_BUNDLE_URI_ESCAPED-3.bdl\" &&\n ++test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2 with list\" '\n ++\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n ++\t\tbundle.bundle1.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED-1.bdl\" &&\n ++\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n ++\t\tbundle.bundle2.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED-2.bdl\" &&\n ++\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n ++\t\tbundle.bundle3.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED-3.bdl\" &&\n +\n +\t# All data about bundle URIs\n +\tcat >expect <<-EOF &&\n @@ t/lib-t5730-protocol-v2-bundle-uri.sh: test_expect_success \"ls-remote with $T573\n +\t\tversion = 1\n +\t\tmode = all\n +\t[bundle \"bundle1\"]\n -+\t\turi = $T5730_BUNDLE_URI_ESCAPED-1.bdl\n ++\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED-1.bdl\n +\t[bundle \"bundle2\"]\n -+\t\turi = $T5730_BUNDLE_URI_ESCAPED-2.bdl\n ++\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED-2.bdl\n +\t[bundle \"bundle3\"]\n -+\t\turi = $T5730_BUNDLE_URI_ESCAPED-3.bdl\n - \tEOF\n - \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n ++\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED-3.bdl\n ++\tEOF\n ++\n ++\tGIT_TEST_BUNDLE_URI=1 \\\n \ttest-tool bundle-uri \\\n + \t\tls-remote \\\n + \t\t\"$BUNDLE_URI_REPO_URI\" \\\n 6: 7d86852c015 ! 8: 1eec3426aee strbuf: introduce strbuf_strip_file_from_path()\n @@ Commit message\n Re-add the method, this time in strbuf.c, but with a new name:\n strbuf_strip_file_from_path(). The method requirements are slightly\n modified to allow a trailing slash, in which case nothing is done, which\n - makes the name change valuable. The return value is the number of bytes\n - removed.\n + makes the name change valuable.\n \n Signed-off-by: Derrick Stolee \n \n @@ strbuf.c: int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n \treturn res;\n }\n +\n -+size_t strbuf_strip_file_from_path(struct strbuf *buf)\n ++void strbuf_strip_file_from_path(struct strbuf *sb)\n +{\n -+\tsize_t len = buf->len;\n -+\tsize_t offset = offset_1st_component(buf->buf);\n -+\tchar *path_sep = find_last_dir_sep(buf->buf + offset);\n -+\tstrbuf_setlen(buf, path_sep ? path_sep - buf->buf + 1 : offset);\n -+\treturn len - buf->len;\n ++\tchar *path_sep = find_last_dir_sep(sb->buf);\n ++\tstrbuf_setlen(sb, path_sep ? path_sep - sb->buf + 1 : 0);\n +}\n \n ## strbuf.h ##\n @@ strbuf.h: int launch_sequence_editor(const char *path, struct strbuf *buffer,\n +/*\n + * Remove the filename from the provided path string. If the path\n + * contains a trailing separator, then the path is considered a directory\n -+ * and nothing is modified. Returns the number of characters removed from\n -+ * the path.\n ++ * and nothing is modified.\n + *\n + * Examples:\n -+ * - \"/path/to/file\" -> \"/path/to/\" (returns: 4)\n -+ * - \"/path/to/dir/\" -> \"/path/to/dir/\" (returns: 0)\n ++ * - \"/path/to/file\" -> \"/path/to/\"\n ++ * - \"/path/to/dir/\" -> \"/path/to/dir/\"\n + */\n -+size_t strbuf_strip_file_from_path(struct strbuf *buf);\n ++void strbuf_strip_file_from_path(struct strbuf *sb);\n +\n void strbuf_add_lines(struct strbuf *sb,\n \t\t const char *prefix,\n 7: 186e112d821 ! 9: 48731438d6a bundle-uri: allow relative URLs in bundle lists\n @@ Commit message\n every push to a CDN would require altering the table of contents to\n match the expected domain and exact location within it.\n \n - Allow a bundle list to specify a relative URI for the bundles.\n - This allows easier distribution of bundle data.\n + Allow a bundle list to specify a relative URI for the bundles. This URI\n + is based on where the client received the bundle list. For a list\n + provided in the 'bundle-uri' protocol v2 command, the Git remote URI is\n + the base URI. Otherwise, the bundle list was provided from an HTTP URI\n + not using the Git protocol, and that URI is the base URI. This allows\n + easier distribution of bundle data.\n \n Signed-off-by: Derrick Stolee \n \n @@ bundle-uri.h: struct bundle_list {\n \tstruct hashmap bundles;\n +\n +\t/**\n -+\t * The baseURI of a bundle_list is used as the base for any\n -+\t * relative URIs advertised by the bundle list at that location.\n ++\t * The baseURI of a bundle_list is the URI that provided the list.\n +\t *\n -+\t * When the list is generated from a Git server, then use that\n -+\t * server's location.\n ++\t * In the case of the 'bundle-uri' protocol v2 command, the base\n ++\t * URI is the URI of the Git remote.\n ++\t *\n ++\t * Otherewise, the bundle list was downloaded over HTTP from some\n ++\t * known URI.\n ++\t *\n ++\t * The baseURI is used as the base for any relative URIs\n ++\t * advertised by the bundle list at that location.\n +\t */\n +\tchar *baseURI;\n };\n @@ t/t5750-bundle-uri-parse.sh: test_expect_success 'bundle_uri_parse_line() just U\n +\ttest_must_be_empty err &&\n +\ttest_cmp_config_output expect actual\n +'\n ++\n ++test_expect_success 'bundle_uri_parse_line(): relative URIs and parent paths' '\n ++\tcat >in <<-\\EOF &&\n ++\tbundle.one.uri=bundle.bdl\n ++\tbundle.two.uri=../bundle.bdl\n ++\tbundle.three.uri=../../bundle.bdl\n ++\tEOF\n ++\n ++\tcat >expect <<-\\EOF &&\n ++\t[bundle]\n ++\t\tversion = 1\n ++\t\tmode = all\n ++\t[bundle \"one\"]\n ++\t\turi = /bundle.bdl\n ++\t[bundle \"two\"]\n ++\t\turi = bundle.bdl\n ++\t[bundle \"three\"]\n ++\t\turi = /../bundle.bdl\n ++\tEOF\n ++\n ++\t# TODO: We would prefer if parsing a bundle list would not cause\n ++\t# a die() and instead would give a warning and allow the rest of\n ++\t# a Git command to continue. This test_must_fail is necessary for\n ++\t# now until the interface for relative_url() allows for reporting\n ++\t# an error instead of die()ing.\n ++\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n ++\tgrep \"fatal: cannot strip one component off url\" err\n ++'\n +\n test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' '\n \tcat >in <<-\\EOF &&\n @@ t/t5750-bundle-uri-parse.sh: test_expect_success 'parse config format: just URIs\n \t= bogus-value\n \n ## transport.c ##\n -@@ transport.c: int transport_get_remote_bundle_uri(struct transport *transport, int quiet)\n +@@ transport.c: int transport_get_remote_bundle_uri(struct transport *transport)\n \t (git_config_get_bool(\"transfer.bundleuri\", &value) || !value))\n \t\treturn 0;\n \n +\tif (!transport->bundles->baseURI)\n +\t\ttransport->bundles->baseURI = xstrdup(transport->url);\n +\n - \tif (!vtable->get_bundle_uri) {\n - \t\tif (quiet)\n - \t\t\treturn -1;\n + \tif (!vtable->get_bundle_uri)\n + \t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n + \n 8: f254da46a2c = 10: 69bf154bec6 bundle-uri: download bundles from an advertised list\n 9: b62b4b17481 ! 11: 7e1819162b6 clone: unbundle the advertised bundles\n @@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n -\t * Populate transport->got_remote_bundle_uri and\n -\t * transport->bundle_uri. We might get nothing.\n -\t */\n --\ttransport_get_remote_bundle_uri(transport, 1);\n +-\ttransport_get_remote_bundle_uri(transport);\n +\tif (!bundle_uri) {\n +\t\t/*\n +\t\t* Populate transport->got_remote_bundle_uri and\n +\t\t* transport->bundle_uri. We might get nothing.\n +\t\t*/\n -+\t\ttransport_get_remote_bundle_uri(transport, 1);\n ++\t\ttransport_get_remote_bundle_uri(transport);\n +\n +\t\tif (transport->bundles &&\n +\t\t hashmap_get_size(&transport->bundles->bundles)) {\n +\t\t\t/* At this point, we need the_repository to match the cloned repo. */\n +\t\t\tif (repo_init(the_repository, git_dir, work_tree))\n +\t\t\t\twarning(_(\"failed to initialize the repo, skipping bundle URI\"));\n -+\t\t\tif (fetch_bundle_list(the_repository,\n -+\t\t\t\t\t remote->url[0],\n -+\t\t\t\t\t transport->bundles))\n ++\t\t\telse if (fetch_bundle_list(the_repository,\n ++\t\t\t\t\t\t remote->url[0],\n ++\t\t\t\t\t\t transport->bundles))\n +\t\t\t\twarning(_(\"failed to fetch advertised bundles\"));\n +\t\t} else {\n +\t\t\tclear_bundle_list(transport->bundles);\n @@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (mapped_refs) {\n \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n \n + ## t/lib-bundle-uri-protocol.sh ##\n +@@ t/lib-bundle-uri-protocol.sh: test_expect_success \"connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav\n + '\n + \n + test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris\" '\n +-\ttest_when_finished \"rm -rf log cloned cloned2\" &&\n ++\ttest_when_finished \"rm -rf log* cloned*\" &&\n + \n + \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n + \tGIT_TEST_BUNDLE_URI=0 \\\n +@@ t/lib-bundle-uri-protocol.sh: test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque\n + \tgrep \"< bundle-uri\" log &&\n + \n + \t# Client issued bundle-uri command\n +-\tgrep \"> command=bundle-uri\" log\n ++\tgrep \"> command=bundle-uri\" log &&\n ++\n ++\tGIT_TRACE_PACKET=\"$PWD/log3\" \\\n ++\tgit \\\n ++\t\t-c transfer.bundleURI=true \\\n ++\t\t-c protocol.version=2 \\\n ++\t\tclone --bundle-uri=\"$BUNDLE_URI_BUNDLE_URI\" \\\n ++\t\t\"$BUNDLE_URI_REPO_URI\" cloned3 \\\n ++\t\t>actual 2>err &&\n ++\n ++\t# Server responded using protocol v2\n ++\tgrep \"< version 2\" log3 &&\n ++\n ++\t# Server advertised bundle-uri capability\n ++\tgrep \"< bundle-uri\" log3 &&\n ++\n ++\t# Client did not issue bundle-uri command (--bundle-uri override)\n ++\t! grep \"> command=bundle-uri\" log3\n + '\n + \n + test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2\" '\n +\n ## t/t5601-clone.sh ##\n @@ t/t5601-clone.sh: test_expect_success 'reject cloning shallow repository using HTTP' '\n \tgit clone --no-reject-shallow $HTTPD_URL/smart/repo.git repo\n\n-- \ngitgitgadget\n"},{"id":"468528","messageId":"69bf154bec63a22df8e5eac89f975625ce73c8ac.1670262639.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v3.git.1670262639.gitgitgadget@gmail.com","subject":"[PATCH v3 10/11] bundle-uri: download bundles from an advertised list","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-05T17:50:38Z","receivedAt":"2022-12-05T17:52:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe logic in fetch_bundle_uri() is useful for the --bundle-uri option of\n'git clone', but is not helpful when the clone operation discovers a\nlist of URIs from the bundle-uri protocol v2 command. To actually\ndownload and unbundle the advertised bundles, we need a different\nmechanism.\n\nCreate the new fetch_bundle_list() method which is very similar to\nfetch_bundle_uri() except that it relies on download_bundle_list()\ninstead of fetch_bundle_uri_internal(). The download_bundle_list()\nmethod will recursively call fetch_bundle_uri_internal() if any of the\nadvertised URIs serve a bundle list instead of a bundle. This will also\nfollow the bundle.list.mode setting from the input list: \"any\" will\ndownload only one such URI while \"all\" will download data from all of\nthe URIs.\n\nIn an identical way to fetch_bundle_uri(), the bundles are unbundled\nafter all of the bundle lists have been expanded and all necessary URIs.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 21 +++++++++++++++++++++\n bundle-uri.h | 11 +++++++++++\n 2 files changed, 32 insertions(+)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 80370992773..c411b871bdd 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -577,6 +577,27 @@ cleanup:\n \treturn result;\n }\n \n+int fetch_bundle_list(struct repository *r, const char *uri, struct bundle_list *list)\n+{\n+\tint result;\n+\tstruct bundle_list global_list;\n+\n+\tinit_bundle_list(&global_list);\n+\n+\t/* If a bundle is added to this global list, then it is required. */\n+\tglobal_list.mode = BUNDLE_MODE_ALL;\n+\n+\tif ((result = download_bundle_list(r, list, &global_list, 0)))\n+\t\tgoto cleanup;\n+\n+\tresult = unbundle_all_bundles(r, &global_list);\n+\n+cleanup:\n+\tfor_all_bundles_in_list(&global_list, unlink_bundle, NULL);\n+\tclear_bundle_list(&global_list);\n+\treturn result;\n+}\n+\n /**\n * API for serve.c.\n */\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex e7e90a5f088..b2c9c160a52 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -107,6 +107,17 @@ int bundle_uri_parse_config_format(const char *uri,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n+/**\n+ * Given a bundle list that was already advertised (likely by the\n+ * bundle-uri protocol v2 verb) at the given uri, fetch and unbundle the\n+ * bundles according to the bundle strategy of that list.\n+ *\n+ * Returns non-zero if no bundle information is found at the given 'uri'.\n+ */\n+int fetch_bundle_list(struct repository *r,\n+\t\t const char *uri,\n+\t\t struct bundle_list *list);\n+\n /**\n * API for serve.c.\n */\n-- \ngitgitgadget\n\n"},{"id":"468529","messageId":"48731438d6a289129a5768b80af856fb49692426.1670262639.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v3.git.1670262639.gitgitgadget@gmail.com","subject":"[PATCH v3 09/11] bundle-uri: allow relative URLs in bundle lists","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-05T17:50:37Z","receivedAt":"2022-12-05T17:52:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBundle providers may want to distribute that data across multiple CDNs.\nThis might require a change in the base URI, all the way to the domain\nname. If all bundles require an absolute URI in their 'uri' value, then\nevery push to a CDN would require altering the table of contents to\nmatch the expected domain and exact location within it.\n\nAllow a bundle list to specify a relative URI for the bundles. This URI\nis based on where the client received the bundle list. For a list\nprovided in the 'bundle-uri' protocol v2 command, the Git remote URI is\nthe base URI. Otherwise, the bundle list was provided from an HTTP URI\nnot using the Git protocol, and that URI is the base URI. This allows\neasier distribution of bundle data.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 16 +++++++-\n bundle-uri.h | 14 +++++++\n t/helper/test-bundle-uri.c | 2 +\n t/t5750-bundle-uri-parse.sh | 82 +++++++++++++++++++++++++++++++++++++\n transport.c | 3 ++\n 5 files changed, 116 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 6919f541085..80370992773 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -7,6 +7,7 @@\n #include \"hashmap.h\"\n #include \"pkt-line.h\"\n #include \"config.h\"\n+#include \"remote.h\"\n \n static int compare_bundles(const void *hashmap_cmp_fn_data,\n \t\t\t const struct hashmap_entry *he1,\n@@ -49,6 +50,7 @@ void clear_bundle_list(struct bundle_list *list)\n \n \tfor_all_bundles_in_list(list, clear_remote_bundle_info, NULL);\n \thashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent);\n+\tfree(list->baseURI);\n }\n \n int for_all_bundles_in_list(struct bundle_list *list,\n@@ -163,7 +165,7 @@ static int bundle_list_update(const char *key, const char *value,\n \tif (!strcmp(subkey, \"uri\")) {\n \t\tif (bundle->uri)\n \t\t\treturn -1;\n-\t\tbundle->uri = xstrdup(value);\n+\t\tbundle->uri = relative_url(list->baseURI, value, NULL);\n \t\treturn 0;\n \t}\n \n@@ -190,6 +192,18 @@ int bundle_uri_parse_config_format(const char *uri,\n \t\t.error_action = CONFIG_ERROR_ERROR,\n \t};\n \n+\tif (!list->baseURI) {\n+\t\tstruct strbuf baseURI = STRBUF_INIT;\n+\t\tstrbuf_addstr(&baseURI, uri);\n+\n+\t\t/*\n+\t\t * If the URI does not end with a trailing slash, then\n+\t\t * remove the filename portion of the path. This is\n+\t\t * important for relative URIs.\n+\t\t */\n+\t\tstrbuf_strip_file_from_path(&baseURI);\n+\t\tlist->baseURI = strbuf_detach(&baseURI, NULL);\n+\t}\n \tresult = git_config_from_file_with_options(config_to_bundle_list,\n \t\t\t\t\t\t filename, list,\n \t\t\t\t\t\t &opts);\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 357111ecce8..e7e90a5f088 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -61,6 +61,20 @@ struct bundle_list {\n \tint version;\n \tenum bundle_list_mode mode;\n \tstruct hashmap bundles;\n+\n+\t/**\n+\t * The baseURI of a bundle_list is the URI that provided the list.\n+\t *\n+\t * In the case of the 'bundle-uri' protocol v2 command, the base\n+\t * URI is the URI of the Git remote.\n+\t *\n+\t * Otherewise, the bundle list was downloaded over HTTP from some\n+\t * known URI.\n+\t *\n+\t * The baseURI is used as the base for any relative URIs\n+\t * advertised by the bundle list at that location.\n+\t */\n+\tchar *baseURI;\n };\n \n void init_bundle_list(struct bundle_list *list);\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex f8159187014..5df5bc3b89e 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -40,6 +40,8 @@ static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mo\n \n \tinit_bundle_list(&list);\n \n+\tlist.baseURI = xstrdup(\"\");\n+\n \tswitch (mode) {\n \tcase KEY_VALUE_PAIRS:\n \t\tif (argc != 1)\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex c2fe3f9c5a5..7b4f930e532 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -30,6 +30,58 @@ test_expect_success 'bundle_uri_parse_line() just URIs' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'bundle_uri_parse_line(): relative URIs' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=bundle.bdl\n+\tbundle.two.uri=../bundle.bdl\n+\tbundle.three.uri=sub/dir/bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = /bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = /sub/dir/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line(): relative URIs and parent paths' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=bundle.bdl\n+\tbundle.two.uri=../bundle.bdl\n+\tbundle.three.uri=../../bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = /bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = /../bundle.bdl\n+\tEOF\n+\n+\t# TODO: We would prefer if parsing a bundle list would not cause\n+\t# a die() and instead would give a warning and allow the rest of\n+\t# a Git command to continue. This test_must_fail is necessary for\n+\t# now until the interface for relative_url() allows for reporting\n+\t# an error instead of die()ing.\n+\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n+\tgrep \"fatal: cannot strip one component off url\" err\n+'\n+\n test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' '\n \tcat >in <<-\\EOF &&\n \t=bogus-value\n@@ -136,6 +188,36 @@ test_expect_success 'parse config format: just URIs' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format: relative URIs' '\n+\tcat >in <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = ../bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = sub/dir/bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = /bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = /sub/dir/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config in >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n test_expect_success 'parse config format edge cases: empty key or value' '\n \tcat >in1 <<-\\EOF &&\n \t= bogus-value\ndiff --git a/transport.c b/transport.c\nindex 97d395e10a3..957dca4923c 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1539,6 +1539,9 @@ int transport_get_remote_bundle_uri(struct transport *transport)\n \t (git_config_get_bool(\"transfer.bundleuri\", &value) || !value))\n \t\treturn 0;\n \n+\tif (!transport->bundles->baseURI)\n+\t\ttransport->bundles->baseURI = xstrdup(transport->url);\n+\n \tif (!vtable->get_bundle_uri)\n \t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n \n-- \ngitgitgadget\n\n"},{"id":"468530","messageId":"beae335b85586ff88f89c551af62f91a04086781.1670262639.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v3.git.1670262639.gitgitgadget@gmail.com","subject":"[PATCH v3 01/11] protocol v2: add server-side \"bundle-uri\" skeleton","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-05T17:50:29Z","receivedAt":"2022-12-05T17:52:33Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nAdd a skeleton server-side implementation of a new \"bundle-uri\" command\nto protocol v2. This will allow conforming clients to optionally seed\ntheir initial clones or incremental fetches from URLs containing\n\"*.bundle\" files created with \"git bundle create\".\n\nThis change only performs the basic boilerplate of advertising a new\nprotocol v2 capability. The new 'bundle-uri' capability allows a client\nto request a list of bundles. Right now, the server only returns a flush\npacket, which corresponds to an empty advertisement. The bundle.* config\nnamespace describes which key-value pairs will be communicated across\nthis interface in future updates.\n\nThe critical bit right now is that the new boolean\nuploadPack.adverstiseBundleURIs config value signals whether or not this\ncapability should be advertised at all.\n\nAn earlier version of this patch [1] used a different transfer format\nthan the \"key=value\" pairs in the current implementation. The change was\nmade to unify the protocol v2 command with the bundle lists provided by\nindependent bundle servers. Further, the standard allows for the server\nto advertise a URI that contains a bundle list. This allows users\nautomatically discovering bundle providers that are loosely associated\nwith the origin server, but without the origin server knowing exactly\nwhich bundles are currently available.\n\n[1] https://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/\n\nThe very-deep headings needed to be modified to stop at level 4 due to\ndocumentation build issues. These were not recognized in earlier builds\nsince the file was previously in the Documentation/technical/ directory\nand was built in a different way. With its current location, the\nheavily-nested details were causing build issues and they are now\nreplaced with a bulletted list of details.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Documentation/gitprotocol-v2.txt | 201 +++++++++++++++++++++++++++++++\n bundle-uri.c | 36 ++++++\n bundle-uri.h | 7 ++\n serve.c | 6 +\n t/t5701-git-serve.sh | 40 +++++-\n 5 files changed, 289 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\nindex 59bf41cefb9..10bd2d40cec 100644\n--- a/Documentation/gitprotocol-v2.txt\n+++ b/Documentation/gitprotocol-v2.txt\n@@ -578,6 +578,207 @@ and associated requested information, each separated by a single space.\n \n \tobj-info = obj-id SP obj-size\n \n+bundle-uri\n+~~~~~~~~~~\n+\n+If the 'bundle-uri' capability is advertised, the server supports the\n+`bundle-uri' command.\n+\n+The capability is currently advertised with no value (i.e. not\n+\"bundle-uri=somevalue\"), a value may be added in the future for\n+supporting command-wide extensions. Clients MUST ignore any unknown\n+capability values and proceed with the 'bundle-uri` dialog they\n+support.\n+\n+The 'bundle-uri' command is intended to be issued before `fetch` to\n+get URIs to bundle files (see linkgit:git-bundle[1]) to \"seed\" and\n+inform the subsequent `fetch` command.\n+\n+The client CAN issue `bundle-uri` before or after any other valid\n+command. To be useful to clients it's expected that it'll be issued\n+after an `ls-refs` and before `fetch`, but CAN be issued at any time\n+in the dialog.\n+\n+DISCUSSION of bundle-uri\n+^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+The intent of the feature is optimize for server resource consumption\n+in the common case by changing the common case of fetching a very\n+large PACK during linkgit:git-clone[1] into a smaller incremental\n+fetch.\n+\n+It also allows servers to achieve better caching in combination with\n+an `uploadpack.packObjectsHook` (see linkgit:git-config[1]).\n+\n+By having new clones or fetches be a more predictable and common\n+negotiation against the tips of recently produces *.bundle file(s).\n+Servers might even pre-generate the results of such negotiations for\n+the `uploadpack.packObjectsHook` as new pushes come in.\n+\n+One way that servers could take advantage of these bundles is that the\n+server would anticipate that fresh clones will download a known bundle,\n+followed by catching up to the current state of the repository using ref\n+tips found in that bundle (or bundles).\n+\n+PROTOCOL for bundle-uri\n+^^^^^^^^^^^^^^^^^^^^^^^\n+\n+A `bundle-uri` request takes no arguments, and as noted above does not\n+currently advertise a capability value. Both may be added in the\n+future.\n+\n+When the client issues a `command=bundle-uri` request, the response is a\n+list of key-value pairs provided as packet lines with value\n+`=`. Each `` should be interpreted as a config key from\n+the `bundle.*` namespace to construct a list of bundles. These keys are\n+grouped by a `bundle..` subsection, where each key corresponding to a\n+given `` contributes attributes to the bundle defined by that ``.\n+See linkgit:git-config[1] for the specific details of these keys and how\n+the Git client will interpret their values.\n+\n+Clients MUST parse the line according to the above format, lines that do\n+not conform to the format SHOULD be discarded. The user MAY be warned in\n+such a case.\n+\n+bundle-uri CLIENT AND SERVER EXPECTATIONS\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+URI CONTENTS::\n+The content at the advertised URIs MUST be one of two types.\n++\n+The advertised URI may contain a bundle file that `git bundle verify`\n+would accept. I.e. they MUST contain one or more reference tips for\n+use by the client, MUST indicate prerequisites (in any) with standard\n+\"-\" prefixes, and MUST indicate their \"object-format\", if\n+applicable.\n++\n+The advertised URI may alternatively contain a plaintext file that `git\n+config --list` would accept (with the `--file` option). The key-value\n+pairs in this list are in the `bundle.*` namespace (see\n+linkgit:git-config[1]).\n+\n+bundle-uri CLIENT ERROR RECOVERY::\n+A client MUST above all gracefully degrade on errors, whether that\n+error is because of bad missing/data in the bundle URI(s), because\n+that client is too dumb to e.g. understand and fully parse out bundle\n+headers and their prerequisite relationships, or something else.\n++\n+Server operators should feel confident in turning on \"bundle-uri\" and\n+not worry if e.g. their CDN goes down that clones or fetches will run\n+into hard failures. Even if the server bundle bundle(s) are\n+incomplete, or bad in some way the client should still end up with a\n+functioning repository, just as if it had chosen not to use this\n+protocol extension.\n++\n+All subsequent discussion on client and server interaction MUST keep\n+this in mind.\n+\n+bundle-uri SERVER TO CLIENT::\n+The ordering of the returned bundle uris is not significant. Clients\n+MUST parse their headers to discover their contained OIDS and\n+prerequisites. A client MUST consider the content of the bundle(s)\n+themselves and their header as the ultimate source of truth.\n++\n+A server MAY even return bundle(s) that don't have any direct\n+relationship to the repository being cloned (either through accident,\n+or intentional \"clever\" configuration), and expect a client to sort\n+out what data they'd like from the bundle(s), if any.\n+\n+bundle-uri CLIENT TO SERVER::\n+The client SHOULD provide reference tips found in the bundle header(s)\n+as 'have' lines in any subsequent `fetch` request. A client MAY also\n+ignore the bundle(s) entirely if doing so is deemed worse for some\n+reason, e.g. if the bundles can't be downloaded, it doesn't like the\n+tips it finds etc.\n+\n+WHEN ADVERTISED BUNDLE(S) REQUIRE NO FURTHER NEGOTIATION::\n+If after issuing `bundle-uri` and `ls-refs`, and getting the header(s)\n+of the bundle(s) the client finds that the ref tips it wants can be\n+retrieved entirely from advertised bundle(s), the client MAY disconnect\n+from the Git server. The results of such a 'clone' or 'fetch' should be\n+indistinguishable from the state attained without using bundle-uri.\n+\n+EARLY CLIENT DISCONNECTIONS AND ERROR RECOVERY::\n+A client MAY perform an early disconnect while still downloading the\n+bundle(s) (having streamed and parsed their headers). In such a case\n+the client MUST gracefully recover from any errors related to\n+finishing the download and validation of the bundle(s).\n++\n+I.e. a client might need to re-connect and issue a 'fetch' command,\n+and possibly fall back to not making use of 'bundle-uri' at all.\n++\n+This \"MAY\" behavior is specified as such (and not a \"SHOULD\") on the\n+assumption that a server advertising bundle uris is more likely than\n+not to be serving up a relatively large repository, and to be pointing\n+to URIs that have a good chance of being in working order. A client\n+MAY e.g. look at the payload size of the bundles as a heuristic to see\n+if an early disconnect is worth it, should falling back on a full\n+\"fetch\" dialog be necessary.\n+\n+WHEN ADVERTISED BUNDLE(S) REQUIRE FURTHER NEGOTIATION::\n+A client SHOULD commence a negotiation of a PACK from the server via\n+the \"fetch\" command using the OID tips found in advertised bundles,\n+even if's still in the process of downloading those bundle(s).\n++\n+This allows for aggressive early disconnects from any interactive\n+server dialog. The client blindly trusts that the advertised OID tips\n+are relevant, and issues them as 'have' lines, it then requests any\n+tips it would like (usually from the \"ls-refs\" advertisement) via\n+'want' lines. The server will then compute a (hopefully small) PACK\n+with the expected difference between the tips from the bundle(s) and\n+the data requested.\n++\n+The only connection the client then needs to keep active is to the\n+concurrently downloading static bundle(s), when those and the\n+incremental PACK are retrieved they should be inflated and\n+validated. Any errors at this point should be gracefully recovered\n+from, see above.\n+\n+bundle-uri PROTOCOL FEATURES\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+The client constructs a bundle list from the `=` pairs\n+provided by the server. These pairs are part of the `bundle.*` namespace\n+as documented in linkgit:git-config[1]. In this section, we discuss some\n+of these keys and describe the actions the client will do in response to\n+this information.\n+\n+In particular, the `bundle.version` key specifies an integer value. The\n+only accepted value at the moment is `1`, but if the client sees an\n+unexpected value here then the client MUST ignore the bundle list.\n+\n+As long as `bundle.version` is understood, all other unknown keys MAY be\n+ignored by the client. The server will guarantee compatibility with older\n+clients, though newer clients may be better able to use the extra keys to\n+minimize downloads.\n+\n+Any backwards-incompatible addition of pre-URI key-value will be\n+guarded by a new `bundle.version` value or values in 'bundle-uri'\n+capability advertisement itself, and/or by new future `bundle-uri`\n+request arguments.\n+\n+Some example key-value pairs that are not currently implemented but could\n+be implemented in the future include:\n+\n+ * Add a \"hash=\" or \"size=\" advertise the expected hash or\n+ size of the bundle file.\n+\n+ * Advertise that one or more bundle files are the same (to e.g. have\n+ clients round-robin or otherwise choose one of N possible files).\n+\n+ * A \"oid=\" shortcut and \"prerequisite=\" shortcut. For\n+ expressing the common case of a bundle with one tip and no\n+ prerequisites, or one tip and one prerequisite.\n++\n+This would allow for optimizing the common case of servers who'd like\n+to provide one \"big bundle\" containing only their \"main\" branch,\n+and/or incremental updates thereof.\n++\n+A client receiving such a a response MAY assume that they can skip\n+retrieving the header from a bundle at the indicated URI, and thus\n+save themselves and the server(s) the request(s) needed to inspect the\n+headers of that bundle or bundles.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 79a914f961b..32022595964 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -563,6 +563,42 @@ cleanup:\n \treturn result;\n }\n \n+/**\n+ * API for serve.c.\n+ */\n+\n+int bundle_uri_advertise(struct repository *r, struct strbuf *value)\n+{\n+\tstatic int advertise_bundle_uri = -1;\n+\n+\tif (advertise_bundle_uri != -1)\n+\t\tgoto cached;\n+\n+\tadvertise_bundle_uri = 0;\n+\tgit_config_get_maybe_bool(\"uploadpack.advertisebundleuris\", &advertise_bundle_uri);\n+\n+cached:\n+\treturn advertise_bundle_uri;\n+}\n+\n+int bundle_uri_command(struct repository *r,\n+\t\t struct packet_reader *request)\n+{\n+\tstruct packet_writer writer;\n+\tpacket_writer_init(&writer, 1);\n+\n+\twhile (packet_reader_read(request) == PACKET_READ_NORMAL)\n+\t\tdie(_(\"bundle-uri: unexpected argument: '%s'\"), request->line);\n+\tif (request->status != PACKET_READ_FLUSH)\n+\t\tdie(_(\"bundle-uri: expected flush after arguments\"));\n+\n+\t/* TODO: Implement the communication */\n+\n+\tpacket_writer_flush(&writer);\n+\n+\treturn 0;\n+}\n+\n /**\n * General API for {transport,connect}.c etc.\n */\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 4dbc269823c..357111ecce8 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -4,6 +4,7 @@\n #include \"hashmap.h\"\n #include \"strbuf.h\"\n \n+struct packet_reader;\n struct repository;\n struct string_list;\n \n@@ -92,6 +93,12 @@ int bundle_uri_parse_config_format(const char *uri,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n+/**\n+ * API for serve.c.\n+ */\n+int bundle_uri_advertise(struct repository *r, struct strbuf *value);\n+int bundle_uri_command(struct repository *r, struct packet_reader *request);\n+\n /**\n * General API for {transport,connect}.c etc.\n */\ndiff --git a/serve.c b/serve.c\nindex 733347f602a..cbf4a143cfe 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -7,6 +7,7 @@\n #include \"protocol-caps.h\"\n #include \"serve.h\"\n #include \"upload-pack.h\"\n+#include \"bundle-uri.h\"\n \n static int advertise_sid = -1;\n static int client_hash_algo = GIT_HASH_SHA1;\n@@ -135,6 +136,11 @@ static struct protocol_capability capabilities[] = {\n \t\t.advertise = always_advertise,\n \t\t.command = cap_object_info,\n \t},\n+\t{\n+\t\t.name = \"bundle-uri\",\n+\t\t.advertise = bundle_uri_advertise,\n+\t\t.command = bundle_uri_command,\n+\t},\n };\n \n void protocol_v2_advertise_capabilities(void)\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex 1896f671cb3..f21e5e9d33d 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -13,7 +13,7 @@ test_expect_success 'test capability advertisement' '\n \twrong_algo sha1:sha256\n \twrong_algo sha256:sha1\n \tEOF\n-\tcat >expect <<-EOF &&\n+\tcat >expect.base <<-EOF &&\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n \tls-refs=unborn\n@@ -21,8 +21,11 @@ test_expect_success 'test capability advertisement' '\n \tserver-option\n \tobject-format=$(test_oid algo)\n \tobject-info\n+\tEOF\n+\tcat >expect.trailer <<-EOF &&\n \t0000\n \tEOF\n+\tcat expect.base expect.trailer >expect &&\n \n \tGIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \\\n \t\t--advertise-capabilities >out &&\n@@ -342,4 +345,39 @@ test_expect_success 'basics of object-info' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'test capability advertisement with uploadpack.advertiseBundleURIs' '\n+\ttest_config uploadpack.advertiseBundleURIs true &&\n+\n+\tcat >expect.extra <<-EOF &&\n+\tbundle-uri\n+\tEOF\n+\tcat expect.base \\\n+\t expect.extra \\\n+\t expect.trailer >expect &&\n+\n+\tGIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \\\n+\t\t--advertise-capabilities >out &&\n+\ttest-tool pkt-line unpack actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'basics of bundle-uri: dies if not enabled' '\n+\ttest-tool pkt-line pack >in <<-EOF &&\n+\tcommand=bundle-uri\n+\t0000\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid command '\"'\"'bundle-uri'\"'\"'\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\tERR serve: invalid command '\"'\"'bundle-uri'\"'\"'\n+\tEOF\n+\n+\ttest_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"468531","messageId":"acc5a8f57f903342c47802115f8e3de9e9d588dc.1670262639.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v3.git.1670262639.gitgitgadget@gmail.com","subject":"[PATCH v3 07/11] bundle-uri: serve bundle.* keys from config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-05T17:50:35Z","receivedAt":"2022-12-05T17:52:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nImplement the \"bundle-uri\" protocol v2 capability by populating the\nkey=value packet lines from the local Git config. The list of bundles is\nprovided from the keys beginning with \"bundle.\".\n\nIn the future, we may want to filter this list to be more specific to\nthe exact known keys that the server intends to share, but for\nflexibility at the moment we will assume that the config values are\nwell-formed.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 16 +++++++++++++++-\n t/lib-bundle-uri-protocol.sh | 35 +++++++++++++++++++++++++++++++++++\n 2 files changed, 50 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 32022595964..6919f541085 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -581,6 +581,16 @@ cached:\n \treturn advertise_bundle_uri;\n }\n \n+static int config_to_packet_line(const char *key, const char *value, void *data)\n+{\n+\tstruct packet_reader *writer = data;\n+\n+\tif (!strncmp(key, \"bundle.\", 7))\n+\t\tpacket_write_fmt(writer->fd, \"%s=%s\", key, value);\n+\n+\treturn 0;\n+}\n+\n int bundle_uri_command(struct repository *r,\n \t\t struct packet_reader *request)\n {\n@@ -592,7 +602,11 @@ int bundle_uri_command(struct repository *r,\n \tif (request->status != PACKET_READ_FLUSH)\n \t\tdie(_(\"bundle-uri: expected flush after arguments\"));\n \n-\t/* TODO: Implement the communication */\n+\t/*\n+\t * Read all \"bundle.*\" config lines to the client as key=value\n+\t * packet lines.\n+\t */\n+\tgit_config(config_to_packet_line, &writer);\n \n \tpacket_writer_flush(&writer);\n \ndiff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh\nindex 88e339ae9ad..6d3f871fa0f 100644\n--- a/t/lib-bundle-uri-protocol.sh\n+++ b/t/lib-bundle-uri-protocol.sh\n@@ -129,8 +129,11 @@ test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol\n \t[bundle]\n \t\tversion = 1\n \t\tmode = all\n+\t[bundle \"only\"]\n+\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED\n \tEOF\n \n+\tGIT_TEST_BUNDLE_URI=1 \\\n \ttest-tool bundle-uri \\\n \t\tls-remote \\\n \t\t\"$BUNDLE_URI_REPO_URI\" \\\n@@ -150,8 +153,40 @@ test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol\n \t[bundle]\n \t\tversion = 1\n \t\tmode = all\n+\t[bundle \"only\"]\n+\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED\n \tEOF\n \n+\tGIT_TEST_BUNDLE_URI=1 \\\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \\\n+\t\t\"$BUNDLE_URI_REPO_URI\" \\\n+\t\t>actual &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2 with list\" '\n+\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n+\t\tbundle.bundle1.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED-1.bdl\" &&\n+\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n+\t\tbundle.bundle2.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED-2.bdl\" &&\n+\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n+\t\tbundle.bundle3.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED-3.bdl\" &&\n+\n+\t# All data about bundle URIs\n+\tcat >expect <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"bundle1\"]\n+\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED-1.bdl\n+\t[bundle \"bundle2\"]\n+\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED-2.bdl\n+\t[bundle \"bundle3\"]\n+\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED-3.bdl\n+\tEOF\n+\n+\tGIT_TEST_BUNDLE_URI=1 \\\n \ttest-tool bundle-uri \\\n \t\tls-remote \\\n \t\t\"$BUNDLE_URI_REPO_URI\" \\\n-- \ngitgitgadget\n\n"},{"id":"468532","messageId":"e46118e60f7a59ef25edf5f1378b4ef0c007cce8.1670262639.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v3.git.1670262639.gitgitgadget@gmail.com","subject":"[PATCH v3 04/11] bundle-uri client: add boolean transfer.bundleURI setting","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-05T17:50:32Z","receivedAt":"2022-12-05T17:52:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nThe yet-to-be introduced client support for bundle-uri will always\nfall back on a full clone, but we'd still like to be able to ignore a\nserver's bundle-uri advertisement entirely.\n\nThe new transfer.bundleURI config option defaults to 'false', but a user\ncan set it to 'true' to enable checking for bundle URIs from the origin\nGit server using protocol v2.\n\nTo enable this setting by default in the correct tests, add a\nGIT_TEST_BUNDLE_URI environment variable.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Documentation/config/transfer.txt | 6 ++++++\n t/lib-bundle-uri-protocol.sh | 19 ++++++++++++++++++-\n transport.c | 9 +++++++++\n 3 files changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/transfer.txt b/Documentation/config/transfer.txt\nindex 264812cca4d..c3ac767d1e4 100644\n--- a/Documentation/config/transfer.txt\n+++ b/Documentation/config/transfer.txt\n@@ -115,3 +115,9 @@ transfer.unpackLimit::\n transfer.advertiseSID::\n \tBoolean. When true, client and server processes will advertise their\n \tunique session IDs to their remote counterpart. Defaults to false.\n+\n+transfer.bundleURI::\n+\tWhen `true`, local `git clone` commands will request bundle\n+\tinformation from the remote server (if advertised) and download\n+\tbundles before continuing the clone through the Git protocol.\n+\tDefaults to `false`.\ndiff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh\nindex d44c6e10f9e..77bfd4f0119 100644\n--- a/t/lib-bundle-uri-protocol.sh\n+++ b/t/lib-bundle-uri-protocol.sh\n@@ -85,9 +85,10 @@ test_expect_success \"connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav\n '\n \n test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris\" '\n-\ttest_when_finished \"rm -rf log cloned\" &&\n+\ttest_when_finished \"rm -rf log cloned cloned2\" &&\n \n \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tGIT_TEST_BUNDLE_URI=0 \\\n \tgit \\\n \t\t-c protocol.version=2 \\\n \t\tclone \"$BUNDLE_URI_REPO_URI\" cloned \\\n@@ -99,6 +100,22 @@ test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque\n \t# Server advertised bundle-uri capability\n \tgrep \"< bundle-uri\" log &&\n \n+\t# Client did not issue bundle-uri command\n+\t! grep \"> command=bundle-uri\" log &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tgit \\\n+\t\t-c transfer.bundleURI=true \\\n+\t\t-c protocol.version=2 \\\n+\t\tclone \"$BUNDLE_URI_REPO_URI\" cloned2 \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\t# Server advertised bundle-uri capability\n+\tgrep \"< bundle-uri\" log &&\n+\n \t# Client issued bundle-uri command\n \tgrep \"> command=bundle-uri\" log\n '\ndiff --git a/transport.c b/transport.c\nindex b6f279e92cb..9f9e38d66dd 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1516,6 +1516,7 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)\n \n int transport_get_remote_bundle_uri(struct transport *transport)\n {\n+\tint value = 0;\n \tconst struct transport_vtable *vtable = transport->vtable;\n \n \t/* Check config only once. */\n@@ -1523,6 +1524,14 @@ int transport_get_remote_bundle_uri(struct transport *transport)\n \t\treturn 0;\n \ttransport->got_remote_bundle_uri = 1;\n \n+\t/*\n+\t * Don't request bundle-uri from the server unless configured to\n+\t * do so by GIT_TEST_BUNDLE_URI=1 or transfer.bundleURI=true.\n+\t */\n+\tif (!git_env_bool(\"GIT_TEST_BUNDLE_URI\", 0) &&\n+\t (git_config_get_bool(\"transfer.bundleuri\", &value) || !value))\n+\t\treturn 0;\n+\n \tif (!vtable->get_bundle_uri)\n \t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n \n-- \ngitgitgadget\n\n"},{"id":"468533","messageId":"fcdfef2012a74a4e08df53834d3c32bfd2bd10a1.1670262639.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v3.git.1670262639.gitgitgadget@gmail.com","subject":"[PATCH v3 02/11] t: create test harness for 'bundle-uri' command","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-05T17:50:30Z","receivedAt":"2022-12-05T17:52:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nThe previous change allowed for a Git server to advertise the\n'bundle-uri' command as a capability based on the\nuploadPack.advertiseBundleURIs config option. Create a set of tests that\ncheck that this capability is advertised using 'git ls-remote'.\n\nIn order to test this functionality across three protocols (file, git,\nand http), create lib-bundle-uri-protocol.sh to generalize the tests,\nallowing the other test scripts to set an environment variable and\notherwise inherit the setup and tests from this script.\n\nThe tests currently only test that the 'bundle-uri' command is\nadvertised or not. Other actions will be tested as the Git client learns\nto request the 'bundle-uri' command and parse its response.\n\nTo help with URI escaping, specifically for file paths with a space in\nthem, extract a 'sed' invocation from t9199-git-svn-info.sh into a\nhelper function for use here, too.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n t/lib-bundle-uri-protocol.sh | 85 ++++++++++++++++++++++++++\n t/t5730-protocol-v2-bundle-uri-file.sh | 17 ++++++\n t/t5731-protocol-v2-bundle-uri-git.sh | 17 ++++++\n t/t5732-protocol-v2-bundle-uri-http.sh | 17 ++++++\n t/t9119-git-svn-info.sh | 2 +-\n t/test-lib-functions.sh | 7 +++\n 6 files changed, 144 insertions(+), 1 deletion(-)\n create mode 100644 t/lib-bundle-uri-protocol.sh\n create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh\n create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh\n create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh\n\ndiff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh\nnew file mode 100644\nindex 00000000000..2da22a39cb8\n--- /dev/null\n+++ b/t/lib-bundle-uri-protocol.sh\n@@ -0,0 +1,85 @@\n+# Set up and run tests of the 'bundle-uri' command in protocol v2\n+#\n+# The test that includes this script should set BUNDLE_URI_PROTOCOL\n+# to one of \"file\", \"git\", or \"http\".\n+\n+BUNDLE_URI_TEST_PARENT=\n+BUNDLE_URI_TEST_URI=\n+BUNDLE_URI_TEST_BUNDLE_URI=\n+case \"$BUNDLE_URI_PROTOCOL\" in\n+file)\n+\tBUNDLE_URI_PARENT=file_parent\n+\tBUNDLE_URI_REPO_URI=\"file://$PWD/file_parent\"\n+\tBUNDLE_URI_BUNDLE_URI=\"$BUNDLE_URI_REPO_URI/fake.bdl\"\n+\ttest_set_prereq BUNDLE_URI_FILE\n+\t;;\n+git)\n+\t. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+\tstart_git_daemon --export-all --enable=receive-pack\n+\tBUNDLE_URI_PARENT=\"$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\"\n+\tBUNDLE_URI_REPO_URI=\"$GIT_DAEMON_URL/parent\"\n+\tBUNDLE_URI_BUNDLE_URI=\"https://example.com/fake.bdl\"\n+\ttest_set_prereq BUNDLE_URI_GIT\n+\t;;\n+http)\n+\t. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\tstart_httpd\n+\tBUNDLE_URI_PARENT=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\"\n+\tBUNDLE_URI_REPO_URI=\"$HTTPD_URL/smart/http_parent\"\n+\tBUNDLE_URI_BUNDLE_URI=\"https://example.com/fake.bdl\"\n+\ttest_set_prereq BUNDLE_URI_HTTP\n+\t;;\n+*)\n+\tBUG \"Need to pass valid BUNDLE_URI_PROTOCOL (was \\\"$BUNDLE_URI_PROTOCOL\\\")\"\n+\t;;\n+esac\n+\n+test_expect_success \"setup protocol v2 $BUNDLE_URI_PROTOCOL:// tests\" '\n+\tgit init \"$BUNDLE_URI_PARENT\" &&\n+\ttest_commit -C \"$BUNDLE_URI_PARENT\" one &&\n+\tgit -C \"$BUNDLE_URI_PARENT\" config uploadpack.advertiseBundleURIs true\n+'\n+\n+case \"$BUNDLE_URI_PROTOCOL\" in\n+http)\n+\ttest_expect_success \"setup config for $BUNDLE_URI_PROTOCOL:// tests\" '\n+\t\tgit -C \"$BUNDLE_URI_PARENT\" config http.receivepack true\n+\t'\n+\t;;\n+*)\n+\t;;\n+esac\n+BUNDLE_URI_BUNDLE_URI_ESCAPED=$(echo \"$BUNDLE_URI_BUNDLE_URI\" | test_uri_escape)\n+\n+test_expect_success \"connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: no bundle-uri\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\ttest_when_finished \"git -C \\\"$BUNDLE_URI_PARENT\\\" config uploadpack.advertiseBundleURIs true\" &&\n+\tgit -C \"$BUNDLE_URI_PARENT\" config uploadpack.advertiseBundleURIs false &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tgit \\\n+\t\t-c protocol.version=2 \\\n+\t\tls-remote --symref \"$BUNDLE_URI_REPO_URI\" \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\t! grep bundle-uri log\n+'\n+\n+test_expect_success \"connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: have bundle-uri\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tgit \\\n+\t\t-c protocol.version=2 \\\n+\t\tls-remote --symref \"$BUNDLE_URI_REPO_URI\" \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\t# Server advertised bundle-uri capability\n+\tgrep \"< bundle-uri\" log\n+'\ndiff --git a/t/t5730-protocol-v2-bundle-uri-file.sh b/t/t5730-protocol-v2-bundle-uri-file.sh\nnew file mode 100755\nindex 00000000000..37bdb725bca\n--- /dev/null\n+++ b/t/t5730-protocol-v2-bundle-uri-file.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri with protocol v2 and 'file://' transport\"\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'file://' transport\n+#\n+BUNDLE_URI_PROTOCOL=file\n+. \"$TEST_DIRECTORY\"/lib-bundle-uri-protocol.sh\n+\n+test_done\ndiff --git a/t/t5731-protocol-v2-bundle-uri-git.sh b/t/t5731-protocol-v2-bundle-uri-git.sh\nnew file mode 100755\nindex 00000000000..8add1b37abc\n--- /dev/null\n+++ b/t/t5731-protocol-v2-bundle-uri-git.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri with protocol v2 and 'git://' transport\"\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'git://' transport\n+#\n+BUNDLE_URI_PROTOCOL=git\n+. \"$TEST_DIRECTORY\"/lib-bundle-uri-protocol.sh\n+\n+test_done\ndiff --git a/t/t5732-protocol-v2-bundle-uri-http.sh b/t/t5732-protocol-v2-bundle-uri-http.sh\nnew file mode 100755\nindex 00000000000..129daa02269\n--- /dev/null\n+++ b/t/t5732-protocol-v2-bundle-uri-http.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri with protocol v2 and 'http://' transport\"\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'http://' transport\n+#\n+BUNDLE_URI_PROTOCOL=http\n+. \"$TEST_DIRECTORY\"/lib-bundle-uri-protocol.sh\n+\n+test_done\ndiff --git a/t/t9119-git-svn-info.sh b/t/t9119-git-svn-info.sh\nindex 8201c3e808a..088d1c57a88 100755\n--- a/t/t9119-git-svn-info.sh\n+++ b/t/t9119-git-svn-info.sh\n@@ -28,7 +28,7 @@ test_cmp_info () {\n \trm -f tmp.expect tmp.actual\n }\n \n-quoted_svnrepo=\"$(echo $svnrepo | sed 's/ /%20/')\"\n+quoted_svnrepo=\"$(echo $svnrepo | test_uri_escape)\"\n \n test_expect_success 'setup repository and import' '\n \tmkdir info &&\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 29d914a12ba..5f6966a404b 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1755,6 +1755,13 @@ test_path_is_hidden () {\n \treturn 1\n }\n \n+# Poor man's URI escaping. Good enough for the test suite whose trash\n+# directory has a space in it. See 93c3fcbe4d4 (git-svn: attempt to\n+# mimic SVN 1.7 URL canonicalization, 2012-07-28) for prior art.\n+test_uri_escape() {\n+\tsed 's/ /%20/g'\n+}\n+\n # Check that the given command was invoked as part of the\n # trace2-format trace on stdin.\n #\n-- \ngitgitgadget\n\n"},{"id":"468534","messageId":"7e1819162b6129371168d89dd60a1bd3cb7de299.1670262639.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v3.git.1670262639.gitgitgadget@gmail.com","subject":"[PATCH v3 11/11] clone: unbundle the advertised bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-05T17:50:39Z","receivedAt":"2022-12-05T17:52:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA previous change introduced the transport methods to acquire a bundle\nlist from the 'bundle-uri' protocol v2 command, when advertised _and_\nwhen the client has chosen to enable the feature.\n\nTeach Git to download and unbundle the data advertised by those bundles\nduring 'git clone'.\n\nAlso, since the --bundle-uri option exists, we do not want to mix the\nadvertised bundles with the user-specified bundles.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 26 +++++++++++++---\n t/lib-bundle-uri-protocol.sh | 21 +++++++++++--\n t/t5601-clone.sh | 59 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 99 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 39364c25b15..af8b2a4df66 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1266,11 +1266,27 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (refs)\n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n \n-\t/*\n-\t * Populate transport->got_remote_bundle_uri and\n-\t * transport->bundle_uri. We might get nothing.\n-\t */\n-\ttransport_get_remote_bundle_uri(transport);\n+\tif (!bundle_uri) {\n+\t\t/*\n+\t\t* Populate transport->got_remote_bundle_uri and\n+\t\t* transport->bundle_uri. We might get nothing.\n+\t\t*/\n+\t\ttransport_get_remote_bundle_uri(transport);\n+\n+\t\tif (transport->bundles &&\n+\t\t hashmap_get_size(&transport->bundles->bundles)) {\n+\t\t\t/* At this point, we need the_repository to match the cloned repo. */\n+\t\t\tif (repo_init(the_repository, git_dir, work_tree))\n+\t\t\t\twarning(_(\"failed to initialize the repo, skipping bundle URI\"));\n+\t\t\telse if (fetch_bundle_list(the_repository,\n+\t\t\t\t\t\t remote->url[0],\n+\t\t\t\t\t\t transport->bundles))\n+\t\t\t\twarning(_(\"failed to fetch advertised bundles\"));\n+\t\t} else {\n+\t\t\tclear_bundle_list(transport->bundles);\n+\t\t\tFREE_AND_NULL(transport->bundles);\n+\t\t}\n+\t}\n \n \tif (mapped_refs) {\n \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\ndiff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh\nindex 6d3f871fa0f..73e2d45bc8b 100644\n--- a/t/lib-bundle-uri-protocol.sh\n+++ b/t/lib-bundle-uri-protocol.sh\n@@ -85,7 +85,7 @@ test_expect_success \"connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav\n '\n \n test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris\" '\n-\ttest_when_finished \"rm -rf log cloned cloned2\" &&\n+\ttest_when_finished \"rm -rf log* cloned*\" &&\n \n \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n \tGIT_TEST_BUNDLE_URI=0 \\\n@@ -117,7 +117,24 @@ test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque\n \tgrep \"< bundle-uri\" log &&\n \n \t# Client issued bundle-uri command\n-\tgrep \"> command=bundle-uri\" log\n+\tgrep \"> command=bundle-uri\" log &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log3\" \\\n+\tgit \\\n+\t\t-c transfer.bundleURI=true \\\n+\t\t-c protocol.version=2 \\\n+\t\tclone --bundle-uri=\"$BUNDLE_URI_BUNDLE_URI\" \\\n+\t\t\"$BUNDLE_URI_REPO_URI\" cloned3 \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log3 &&\n+\n+\t# Server advertised bundle-uri capability\n+\tgrep \"< bundle-uri\" log3 &&\n+\n+\t# Client did not issue bundle-uri command (--bundle-uri override)\n+\t! grep \"> command=bundle-uri\" log3\n '\n \n test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2\" '\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 45f0803ed4d..d1d8139751e 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -795,6 +795,65 @@ test_expect_success 'reject cloning shallow repository using HTTP' '\n \tgit clone --no-reject-shallow $HTTPD_URL/smart/repo.git repo\n '\n \n+test_expect_success 'auto-discover bundle URI from HTTP clone' '\n+\ttest_when_finished rm -rf trace.txt repo2 \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" &&\n+\tgit -C src bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/everything.bundle\" --all &&\n+\tgit clone --bare --no-local src \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" &&\n+\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tuploadpack.advertiseBundleURIs true &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tbundle.version 1 &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tbundle.mode all &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tbundle.everything.uri \"$HTTPD_URL/everything.bundle\" &&\n+\n+\tGIT_TEST_BUNDLE_URI=1 \\\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n+\t\tgit -c protocol.version=2 clone \\\n+\t\t$HTTPD_URL/smart/repo2.git repo2 &&\n+\tcat >pattern <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/everything.bundle\"\\]\n+\tEOF\n+\tgrep -f pattern trace.txt\n+'\n+\n+test_expect_success 'auto-discover multiple bundles from HTTP clone' '\n+\ttest_when_finished rm -rf trace.txt repo3 \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" &&\n+\n+\ttest_commit -C src new &&\n+\tgit -C src bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/new.bundle\" HEAD~1..HEAD &&\n+\tgit clone --bare --no-local src \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" &&\n+\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tuploadpack.advertiseBundleURIs true &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.version 1 &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.mode all &&\n+\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.everything.uri \"$HTTPD_URL/everything.bundle\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.new.uri \"$HTTPD_URL/new.bundle\" &&\n+\n+\tGIT_TEST_BUNDLE_URI=1 \\\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n+\t\tgit -c protocol.version=2 clone \\\n+\t\t$HTTPD_URL/smart/repo3.git repo3 &&\n+\n+\t# We should fetch _both_ bundles\n+\tcat >pattern <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/everything.bundle\"\\]\n+\tEOF\n+\tgrep -f pattern trace.txt &&\n+\tcat >pattern <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/new.bundle\"\\]\n+\tEOF\n+\tgrep -f pattern trace.txt\n+'\n+\n # DO NOT add non-httpd-specific tests here, because the last part of this\n # test script is only executed when httpd is available and enabled.\n \n-- \ngitgitgadget\n"},{"id":"468535","messageId":"46a58e83caf6d37e5a62b39d3c2e35e278211297.1670262639.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v3.git.1670262639.gitgitgadget@gmail.com","subject":"[PATCH v3 06/11] bundle-uri client: add helper for testing server","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-05T17:50:34Z","receivedAt":"2022-12-05T17:52:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nAdd a 'test-tool bundle-uri ls-remote' command. This is a thin wrapper\nfor issuing protocol v2 \"bundle-uri\" commands to a server, and to the\nparsing routines in bundle-uri.c.\n\nIn the \"git clone\" case we'll have already done the handshake(),\nbut not here. Add an extra case to check for this handshake in\nget_bundle_uri() for ease of use for future callers. Rename the existing\n'got_remote_heads' to 'finished_handshake' to make it more clear what\nthat bit represents.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n t/helper/test-bundle-uri.c | 46 ++++++++++++++++++++++++++++++++++++\n t/lib-bundle-uri-protocol.sh | 39 ++++++++++++++++++++++++++++++\n transport.c | 7 ++++++\n 3 files changed, 92 insertions(+)\n\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex 25afd393428..f8159187014 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -3,6 +3,10 @@\n #include \"bundle-uri.h\"\n #include \"strbuf.h\"\n #include \"string-list.h\"\n+#include \"transport.h\"\n+#include \"ref-filter.h\"\n+#include \"remote.h\"\n+#include \"refs.h\"\n \n enum input_mode {\n \tKEY_VALUE_PAIRS,\n@@ -68,6 +72,46 @@ usage:\n \tusage_with_options(usage, options);\n }\n \n+static int cmd_ls_remote(int argc, const char **argv)\n+{\n+\tconst char *uploadpack = NULL;\n+\tstruct string_list server_options = STRING_LIST_INIT_DUP;\n+\tconst char *dest;\n+\tstruct remote *remote;\n+\tstruct transport *transport;\n+\tint status = 0;\n+\n+\tdest = argc > 1 ? argv[1] : NULL;\n+\n+\tremote = remote_get(dest);\n+\tif (!remote) {\n+\t\tif (dest)\n+\t\t\tdie(_(\"bad repository '%s'\"), dest);\n+\t\tdie(_(\"no remote configured to get bundle URIs from\"));\n+\t}\n+\tif (!remote->url_nr)\n+\t\tdie(_(\"remote '%s' has no configured URL\"), dest);\n+\n+\ttransport = transport_get(remote, NULL);\n+\tif (uploadpack)\n+\t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n+\tif (server_options.nr)\n+\t\ttransport->server_options = &server_options;\n+\n+\tif (transport_get_remote_bundle_uri(transport) < 0) {\n+\t\terror(_(\"could not get the bundle-uri list\"));\n+\t\tstatus = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tprint_bundle_list(stdout, transport->bundles);\n+\n+cleanup:\n+\tif (transport_disconnect(transport))\n+\t\treturn 1;\n+\treturn status;\n+}\n+\n int cmd__bundle_uri(int argc, const char **argv)\n {\n \tconst char *usage[] = {\n@@ -88,6 +132,8 @@ int cmd__bundle_uri(int argc, const char **argv)\n \t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS);\n \tif (!strcmp(argv[1], \"parse-config\"))\n \t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE);\n+\tif (!strcmp(argv[1], \"ls-remote\"))\n+\t\treturn cmd_ls_remote(argc - 1, argv + 1);\n \terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n \n usage:\ndiff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh\nindex 77bfd4f0119..88e339ae9ad 100644\n--- a/t/lib-bundle-uri-protocol.sh\n+++ b/t/lib-bundle-uri-protocol.sh\n@@ -119,3 +119,42 @@ test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque\n \t# Client issued bundle-uri command\n \tgrep \"> command=bundle-uri\" log\n '\n+\n+test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2\" '\n+\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n+\t\tbundle.only.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED\" &&\n+\n+\t# All data about bundle URIs\n+\tcat >expect <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \\\n+\t\t\"$BUNDLE_URI_REPO_URI\" \\\n+\t\t>actual &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2 and extra data\" '\n+\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n+\t\tbundle.only.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED\" &&\n+\n+\t# Extra data should be ignored\n+\ttest_config -C \"$BUNDLE_URI_PARENT\" bundle.only.extra bogus &&\n+\n+\t# All data about bundle URIs\n+\tcat >expect <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \\\n+\t\t\"$BUNDLE_URI_REPO_URI\" \\\n+\t\t>actual &&\n+\ttest_cmp_config_output expect actual\n+'\ndiff --git a/transport.c b/transport.c\nindex a2281d95262..97d395e10a3 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -371,6 +371,13 @@ static int get_bundle_uri(struct transport *transport)\n \t\tinit_bundle_list(transport->bundles);\n \t}\n \n+\tif (!data->finished_handshake) {\n+\t\tstruct ref *refs = handshake(transport, 0, NULL, 0);\n+\n+\t\tif (refs)\n+\t\t\tfree_refs(refs);\n+\t}\n+\n \t/*\n \t * \"Support\" protocol v0 and v2 without bundle-uri support by\n \t * silently degrading to a NOOP.\n-- \ngitgitgadget\n\n"},{"id":"468536","messageId":"a0188ae39c6aba81f551e1f43d30072261ebdfb4.1670262639.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v3.git.1670262639.gitgitgadget@gmail.com","subject":"[PATCH v3 03/11] clone: request the 'bundle-uri' command when available","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-05T17:50:31Z","receivedAt":"2022-12-05T17:52:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nSet up all the needed client parts of the 'bundle-uri' protocol v2\ncommand, without actually doing anything with the bundle URIs.\n\nIf the server says it supports 'bundle-uri' teach Git to issue the\n'bundle-uri' command after the 'ls-refs' during 'git clone'. The\nreturned key=value pairs are passed to the bundle list code which is\ntested using a different ingest mechanism in t5750-bundle-uri-parse.sh.\n\nAt this point, Git does nothing with that bundle list. It will not\ndownload any of the bundles. That will come in a later change after\nthese protocol bits are finalized.\n\nThe no-op client is initially used only by 'git clone' to test the basic\nfunctionality, and eventually will bootstrap the initial download of Git\nobjects during a fresh clone. The bundle URI client will not be\nintegrated into other fetches until a mechanism is created to select a\nsubset of bundles for download.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 6 +++++\n connect.c | 44 +++++++++++++++++++++++++++++++\n remote.h | 5 ++++\n t/lib-bundle-uri-protocol.sh | 19 ++++++++++++++\n transport-helper.c | 13 +++++++++\n transport-internal.h | 7 +++++\n transport.c | 51 ++++++++++++++++++++++++++++++++++++\n transport.h | 19 ++++++++++++++\n 8 files changed, 164 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 547d6464b3c..39364c25b15 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1266,6 +1266,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (refs)\n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n \n+\t/*\n+\t * Populate transport->got_remote_bundle_uri and\n+\t * transport->bundle_uri. We might get nothing.\n+\t */\n+\ttransport_get_remote_bundle_uri(transport);\n+\n \tif (mapped_refs) {\n \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n \ndiff --git a/connect.c b/connect.c\nindex 5ea53deda23..624a10f18ee 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -15,6 +15,7 @@\n #include \"version.h\"\n #include \"protocol.h\"\n #include \"alias.h\"\n+#include \"bundle-uri.h\"\n \n static char *server_capabilities_v1;\n static struct strvec server_capabilities_v2 = STRVEC_INIT;\n@@ -491,6 +492,49 @@ static void send_capabilities(int fd_out, struct packet_reader *reader)\n \t}\n }\n \n+int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\n+\t\t\t struct bundle_list *bundles, int stateless_rpc)\n+{\n+\tint line_nr = 1;\n+\n+\t/* Assert bundle-uri support */\n+\tserver_supports_v2(\"bundle-uri\", 1);\n+\n+\t/* (Re-)send capabilities */\n+\tsend_capabilities(fd_out, reader);\n+\n+\t/* Send command */\n+\tpacket_write_fmt(fd_out, \"command=bundle-uri\\n\");\n+\tpacket_delim(fd_out);\n+\n+\tpacket_flush(fd_out);\n+\n+\t/* Process response from server */\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\tconst char *line = reader->line;\n+\t\tline_nr++;\n+\n+\t\tif (!bundle_uri_parse_line(bundles, line))\n+\t\t\tcontinue;\n+\n+\t\treturn error(_(\"error on bundle-uri response line %d: %s\"),\n+\t\t\t line_nr, line);\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH)\n+\t\treturn error(_(\"expected flush after bundle-uri listing\"));\n+\n+\t/*\n+\t * Might die(), but obscure enough that that's OK, e.g. in\n+\t * serve.c we'll call BUG() on its equivalent (the\n+\t * PACKET_READ_RESPONSE_END check).\n+\t */\n+\tcheck_stateless_delimiter(stateless_rpc, reader,\n+\t\t\t\t _(\"expected response end packet after ref listing\"));\n+\n+\treturn 0;\n+}\n+\n struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\t struct ref **list, int for_push,\n \t\t\t struct transport_ls_refs_options *transport_options,\ndiff --git a/remote.h b/remote.h\nindex 1c4621b414b..1ebbe42792e 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -234,6 +234,11 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\t const struct string_list *server_options,\n \t\t\t int stateless_rpc);\n \n+/* Used for protocol v2 in order to retrieve refs from a remote */\n+struct bundle_list;\n+int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\n+\t\t\t struct bundle_list *bundles, int stateless_rpc);\n+\n int resolve_remote_symref(struct ref *ref, struct ref *list);\n \n /*\ndiff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh\nindex 2da22a39cb8..d44c6e10f9e 100644\n--- a/t/lib-bundle-uri-protocol.sh\n+++ b/t/lib-bundle-uri-protocol.sh\n@@ -83,3 +83,22 @@ test_expect_success \"connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav\n \t# Server advertised bundle-uri capability\n \tgrep \"< bundle-uri\" log\n '\n+\n+test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris\" '\n+\ttest_when_finished \"rm -rf log cloned\" &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tgit \\\n+\t\t-c protocol.version=2 \\\n+\t\tclone \"$BUNDLE_URI_REPO_URI\" cloned \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\t# Server advertised bundle-uri capability\n+\tgrep \"< bundle-uri\" log &&\n+\n+\t# Client issued bundle-uri command\n+\tgrep \"> command=bundle-uri\" log\n+'\ndiff --git a/transport-helper.c b/transport-helper.c\nindex e95267a4ab5..3ea7c2bb5ad 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -1267,9 +1267,22 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n \treturn ret;\n }\n \n+static int get_bundle_uri(struct transport *transport)\n+{\n+\tget_helper(transport);\n+\n+\tif (process_connect(transport, 0)) {\n+\t\tdo_take_over(transport);\n+\t\treturn transport->vtable->get_bundle_uri(transport);\n+\t}\n+\n+\treturn -1;\n+}\n+\n static struct transport_vtable vtable = {\n \t.set_option\t= set_helper_option,\n \t.get_refs_list\t= get_refs_list,\n+\t.get_bundle_uri = get_bundle_uri,\n \t.fetch_refs\t= fetch_refs,\n \t.push_refs\t= push_refs,\n \t.connect\t= connect_helper,\ndiff --git a/transport-internal.h b/transport-internal.h\nindex c4ca0b733ac..90ea749e5cf 100644\n--- a/transport-internal.h\n+++ b/transport-internal.h\n@@ -26,6 +26,13 @@ struct transport_vtable {\n \tstruct ref *(*get_refs_list)(struct transport *transport, int for_push,\n \t\t\t\t struct transport_ls_refs_options *transport_options);\n \n+\t/**\n+\t * Populates the remote side's bundle-uri under protocol v2,\n+\t * if the \"bundle-uri\" capability was advertised. Returns 0 if\n+\t * OK, negative values on error.\n+\t */\n+\tint (*get_bundle_uri)(struct transport *transport);\n+\n \t/**\n \t * Fetch the objects for the given refs. Note that this gets\n \t * an array, and should ignore the list structure.\ndiff --git a/transport.c b/transport.c\nindex e7b97194c10..b6f279e92cb 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -22,6 +22,7 @@\n #include \"protocol.h\"\n #include \"object-store.h\"\n #include \"color.h\"\n+#include \"bundle-uri.h\"\n \n static int transport_use_color = -1;\n static char transport_colors[][COLOR_MAXLEN] = {\n@@ -359,6 +360,32 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \treturn handshake(transport, for_push, options, 1);\n }\n \n+static int get_bundle_uri(struct transport *transport)\n+{\n+\tstruct git_transport_data *data = transport->data;\n+\tstruct packet_reader reader;\n+\tint stateless_rpc = transport->stateless_rpc;\n+\n+\tif (!transport->bundles) {\n+\t\tCALLOC_ARRAY(transport->bundles, 1);\n+\t\tinit_bundle_list(transport->bundles);\n+\t}\n+\n+\t/*\n+\t * \"Support\" protocol v0 and v2 without bundle-uri support by\n+\t * silently degrading to a NOOP.\n+\t */\n+\tif (!server_supports_v2(\"bundle-uri\", 0))\n+\t\treturn 0;\n+\n+\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\treturn get_remote_bundle_uri(data->fd[1], &reader,\n+\t\t\t\t transport->bundles, stateless_rpc);\n+}\n+\n static int fetch_refs_via_pack(struct transport *transport,\n \t\t\t int nr_heads, struct ref **to_fetch)\n {\n@@ -902,6 +929,7 @@ static int disconnect_git(struct transport *transport)\n \n static struct transport_vtable taken_over_vtable = {\n \t.get_refs_list\t= get_refs_via_connect,\n+\t.get_bundle_uri = get_bundle_uri,\n \t.fetch_refs\t= fetch_refs_via_pack,\n \t.push_refs\t= git_transport_push,\n \t.disconnect\t= disconnect_git\n@@ -1054,6 +1082,7 @@ static struct transport_vtable bundle_vtable = {\n \n static struct transport_vtable builtin_smart_vtable = {\n \t.get_refs_list\t= get_refs_via_connect,\n+\t.get_bundle_uri = get_bundle_uri,\n \t.fetch_refs\t= fetch_refs_via_pack,\n \t.push_refs\t= git_transport_push,\n \t.connect\t= connect_git,\n@@ -1068,6 +1097,9 @@ struct transport *transport_get(struct remote *remote, const char *url)\n \tret->progress = isatty(2);\n \tstring_list_init_dup(&ret->pack_lockfiles);\n \n+\tCALLOC_ARRAY(ret->bundles, 1);\n+\tinit_bundle_list(ret->bundles);\n+\n \tif (!remote)\n \t\tBUG(\"No remote provided to transport_get()\");\n \n@@ -1482,6 +1514,23 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)\n \treturn rc;\n }\n \n+int transport_get_remote_bundle_uri(struct transport *transport)\n+{\n+\tconst struct transport_vtable *vtable = transport->vtable;\n+\n+\t/* Check config only once. */\n+\tif (transport->got_remote_bundle_uri)\n+\t\treturn 0;\n+\ttransport->got_remote_bundle_uri = 1;\n+\n+\tif (!vtable->get_bundle_uri)\n+\t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n+\n+\tif (vtable->get_bundle_uri(transport) < 0)\n+\t\treturn error(_(\"could not retrieve server-advertised bundle-uri list\"));\n+\treturn 0;\n+}\n+\n void transport_unlock_pack(struct transport *transport, unsigned int flags)\n {\n \tint in_signal_handler = !!(flags & TRANSPORT_UNLOCK_PACK_IN_SIGNAL_HANDLER);\n@@ -1512,6 +1561,8 @@ int transport_disconnect(struct transport *transport)\n \t\tret = transport->vtable->disconnect(transport);\n \tif (transport->got_remote_refs)\n \t\tfree_refs((void *)transport->remote_refs);\n+\tclear_bundle_list(transport->bundles);\n+\tfree(transport->bundles);\n \tfree(transport);\n \treturn ret;\n }\ndiff --git a/transport.h b/transport.h\nindex b5bf7b3e704..85150f504fb 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -62,6 +62,7 @@ enum transport_family {\n \tTRANSPORT_FAMILY_IPV6\n };\n \n+struct bundle_list;\n struct transport {\n \tconst struct transport_vtable *vtable;\n \n@@ -76,6 +77,18 @@ struct transport {\n \t */\n \tunsigned got_remote_refs : 1;\n \n+\t/**\n+\t * Indicates whether we already called get_bundle_uri_list(); set by\n+\t * transport.c::transport_get_remote_bundle_uri().\n+\t */\n+\tunsigned got_remote_bundle_uri : 1;\n+\n+\t/*\n+\t * The results of \"command=bundle-uri\", if both sides support\n+\t * the \"bundle-uri\" capability.\n+\t */\n+\tstruct bundle_list *bundles;\n+\n \t/*\n \t * Transports that call take-over destroys the data specific to\n \t * the transport type while doing so, and cannot be reused.\n@@ -281,6 +294,12 @@ void transport_ls_refs_options_release(struct transport_ls_refs_options *opts);\n const struct ref *transport_get_remote_refs(struct transport *transport,\n \t\t\t\t\t struct transport_ls_refs_options *transport_options);\n \n+/**\n+ * Retrieve bundle URI(s) from a remote. Populates \"struct\n+ * transport\"'s \"bundle_uri\" and \"got_remote_bundle_uri\".\n+ */\n+int transport_get_remote_bundle_uri(struct transport *transport);\n+\n /*\n * Fetch the hash algorithm used by a remote.\n *\n-- \ngitgitgadget\n\n"},{"id":"468537","messageId":"1eec3426aee12bbd674ebd646075f0d4c0b1f5af.1670262639.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v3.git.1670262639.gitgitgadget@gmail.com","subject":"[PATCH v3 08/11] strbuf: introduce strbuf_strip_file_from_path()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-05T17:50:36Z","receivedAt":"2022-12-05T17:52:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe strbuf_parent_directory() method was added as a static method in\ncontrib/scalar by d0feac4e8c0 (scalar: 'register' sets recommended\nconfig and starts maintenance, 2021-12-03) and then removed in\n65f6a9eb0b9 (scalar: constrain enlistment search, 2022-08-18), but now\nthere is a need for a similar method in the bundle URI feature.\n\nRe-add the method, this time in strbuf.c, but with a new name:\nstrbuf_strip_file_from_path(). The method requirements are slightly\nmodified to allow a trailing slash, in which case nothing is done, which\nmakes the name change valuable.\n\nSigned-off-by: Derrick Stolee \n---\n strbuf.c | 6 ++++++\n strbuf.h | 11 +++++++++++\n 2 files changed, 17 insertions(+)\n\ndiff --git a/strbuf.c b/strbuf.c\nindex 0890b1405c5..c383f41a3c5 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -1200,3 +1200,9 @@ int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n \tfree(path2);\n \treturn res;\n }\n+\n+void strbuf_strip_file_from_path(struct strbuf *sb)\n+{\n+\tchar *path_sep = find_last_dir_sep(sb->buf);\n+\tstrbuf_setlen(sb, path_sep ? path_sep - sb->buf + 1 : 0);\n+}\ndiff --git a/strbuf.h b/strbuf.h\nindex 76965a17d44..f6dbb9681ee 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -664,6 +664,17 @@ int launch_sequence_editor(const char *path, struct strbuf *buffer,\n int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n \t\t\t const char *const *env);\n \n+/*\n+ * Remove the filename from the provided path string. If the path\n+ * contains a trailing separator, then the path is considered a directory\n+ * and nothing is modified.\n+ *\n+ * Examples:\n+ * - \"/path/to/file\" -> \"/path/to/\"\n+ * - \"/path/to/dir/\" -> \"/path/to/dir/\"\n+ */\n+void strbuf_strip_file_from_path(struct strbuf *sb);\n+\n void strbuf_add_lines(struct strbuf *sb,\n \t\t const char *prefix,\n \t\t const char *buf,\n-- \ngitgitgadget\n\n"},{"id":"468538","messageId":"b009b4f58ea312e40af385ea5ca7ede5ea07391a.1670262639.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v3.git.1670262639.gitgitgadget@gmail.com","subject":"[PATCH v3 05/11] transport: rename got_remote_heads","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-05T17:50:33Z","receivedAt":"2022-12-05T17:52:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'got_remote_heads' member of 'struct git_transport_data' was used\nhistorically to indicate that the initial server connection was made and\nthe ref advertisement was returned. With protocol v2, that initial\nhandshake does not necessarily include the ref advertisement, so this\nmember is not an accurate name. Thankfully, all uses of the member are\nonly checking to see if the handshake should take place, not whether or\nnot some local data has the ref advertisement.\n\nRename the member to 'finished_handshake' to represent the proper state.\nNote that the variable is only set to 1 during the handshake() method.\n\nSigned-off-by: Derrick Stolee \n---\n transport.c | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex 9f9e38d66dd..a2281d95262 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -198,7 +198,7 @@ struct git_transport_data {\n \tstruct git_transport_options options;\n \tstruct child_process *conn;\n \tint fd[2];\n-\tunsigned got_remote_heads : 1;\n+\tunsigned finished_handshake : 1;\n \tenum protocol_version version;\n \tstruct oid_array extra_have;\n \tstruct oid_array shallow;\n@@ -345,7 +345,7 @@ static struct ref *handshake(struct transport *transport, int for_push,\n \tcase protocol_unknown_version:\n \t\tBUG(\"unknown protocol version\");\n \t}\n-\tdata->got_remote_heads = 1;\n+\tdata->finished_handshake = 1;\n \ttransport->hash_algo = reader.hash_algo;\n \n \tif (reader.line_peeked)\n@@ -421,7 +421,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.negotiation_tips = data->options.negotiation_tips;\n \targs.reject_shallow_remote = transport->smart_options->reject_shallow;\n \n-\tif (!data->got_remote_heads) {\n+\tif (!data->finished_handshake) {\n \t\tint i;\n \t\tint must_list_refs = 0;\n \t\tfor (i = 0; i < nr_heads; i++) {\n@@ -461,7 +461,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\t\t to_fetch, nr_heads, &data->shallow,\n \t\t\t &transport->pack_lockfiles, data->version);\n \n-\tdata->got_remote_heads = 0;\n+\tdata->finished_handshake = 0;\n \tdata->options.self_contained_and_connected =\n \t\targs.self_contained_and_connected;\n \tdata->options.connectivity_checked = args.connectivity_checked;\n@@ -846,7 +846,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \tif (transport_color_config() < 0)\n \t\treturn -1;\n \n-\tif (!data->got_remote_heads)\n+\tif (!data->finished_handshake)\n \t\tget_refs_via_connect(transport, 1, NULL);\n \n \tmemset(&args, 0, sizeof(args));\n@@ -894,7 +894,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \telse\n \t\tret = finish_connect(data->conn);\n \tdata->conn = NULL;\n-\tdata->got_remote_heads = 0;\n+\tdata->finished_handshake = 0;\n \n \treturn ret;\n }\n@@ -914,7 +914,7 @@ static int disconnect_git(struct transport *transport)\n {\n \tstruct git_transport_data *data = transport->data;\n \tif (data->conn) {\n-\t\tif (data->got_remote_heads && !transport->stateless_rpc)\n+\t\tif (data->finished_handshake && !transport->stateless_rpc)\n \t\t\tpacket_flush(data->fd[1]);\n \t\tclose(data->fd[0]);\n \t\tif (data->fd[1] >= 0)\n@@ -949,7 +949,7 @@ void transport_take_over(struct transport *transport,\n \tdata->conn = child;\n \tdata->fd[0] = data->conn->out;\n \tdata->fd[1] = data->conn->in;\n-\tdata->got_remote_heads = 0;\n+\tdata->finished_handshake = 0;\n \ttransport->data = data;\n \n \ttransport->vtable = &taken_over_vtable;\n@@ -1150,7 +1150,7 @@ struct transport *transport_get(struct remote *remote, const char *url)\n \t\tret->smart_options = &(data->options);\n \n \t\tdata->conn = NULL;\n-\t\tdata->got_remote_heads = 0;\n+\t\tdata->finished_handshake = 0;\n \t} else {\n \t\t/* Unknown protocol in URL. Pass to external handler. */\n \t\tint len = external_specification_len(url);\n-- \ngitgitgadget\n\n"},{"id":"468554","messageId":"e68b4b9e-5925-bf7b-4ffb-3f72adc970dc@github.com","threadId":"58731","inReplyTo":"beae335b85586ff88f89c551af62f91a04086781.1670262639.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 01/11] protocol v2: add server-side \"bundle-uri\" skeleton","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-12-05T23:31:33Z","receivedAt":"2022-12-05T23:31:39Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n> diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\n> index 59bf41cefb9..10bd2d40cec 100644\n> --- a/Documentation/gitprotocol-v2.txt\n> +++ b/Documentation/gitprotocol-v2.txt\n> @@ -578,6 +578,207 @@ and associated requested information, each separated by a single space.\n> \n> \tobj-info = obj-id SP obj-size\n> \n> +bundle-uri\n> +~~~~~~~~~~\n\nApologies for not following up on this patch when you updated it for v2.\nThis version is much clearer in describing the bundle URI command protocol,\nespecially how the 'bundle.*' config is used:\n\n> +When the client issues a `command=bundle-uri` request, the response is a\n> +list of key-value pairs provided as packet lines with value\n> +`=`. Each `` should be interpreted as a config key from\n> +the `bundle.*` namespace to construct a list of bundles. These keys are\n> +grouped by a `bundle..` subsection, where each key corresponding to a\n> +given `` contributes attributes to the bundle defined by that ``.\n> +See linkgit:git-config[1] for the specific details of these keys and how\n> +the Git client will interpret their values.\n> +\n> +Clients MUST parse the line according to the above format, lines that do\n> +not conform to the format SHOULD be discarded. The user MAY be warned in\n> +such a case.\n> +\n\nand the response types/formats:\n\n> +URI CONTENTS::\n> +The content at the advertised URIs MUST be one of two types.\n> ++\n> +The advertised URI may contain a bundle file that `git bundle verify`\n> +would accept. I.e. they MUST contain one or more reference tips for\n> +use by the client, MUST indicate prerequisites (in any) with standard\n> +\"-\" prefixes, and MUST indicate their \"object-format\", if\n> +applicable.\n> ++\n> +The advertised URI may alternatively contain a plaintext file that `git\n> +config --list` would accept (with the `--file` option). The key-value\n> +pairs in this list are in the `bundle.*` namespace (see\n> +linkgit:git-config[1]).\n> +\n\nRegarding your point about examples in [1]: after reading the remainder of\nthe series, I agree that the test cases in the later patches do a good job\nof documenting the behavior.\n\nThanks!\n\n[1] https://lore.kernel.org/git/ca11478b-7b44-3018-04d8-0b84c4f43b56@github.com/\n\n"},{"id":"468555","messageId":"b7b98123-0e78-757e-404d-024e167bfe60@github.com","threadId":"58731","inReplyTo":"e46118e60f7a59ef25edf5f1378b4ef0c007cce8.1670262639.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 04/11] bundle-uri client: add boolean transfer.bundleURI setting","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-12-05T23:32:11Z","receivedAt":"2022-12-05T23:32:16Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n> From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n> \n> \n> The yet-to-be introduced client support for bundle-uri will always\n> fall back on a full clone, but we'd still like to be able to ignore a\n> server's bundle-uri advertisement entirely.\n> \n> The new transfer.bundleURI config option defaults to 'false', but a user\n> can set it to 'true' to enable checking for bundle URIs from the origin\n> Git server using protocol v2.\n> \n> To enable this setting by default in the correct tests, add a\n> GIT_TEST_BUNDLE_URI environment variable.\n\nIt wasn't immediately clear to me from reading this patch, but it looks like\n'GIT_TEST_BUNDLE_URI' is mainly used to allow 'test-tool bundle-uri\nls-remote' to issue the bundle URI command (since it can't use a '-c\ntransfer.bundleURI=true' command line option) in patch 7 [1].\n\nIf that's the only use for 'GIT_TEST_BUNDLE_URI', could you avoid the\nenvironment variable altogether by setting 'transfer.bundleURI=true' with\n'test_config' before the 'test-tool' call (and 'test_unconfig' after, if\nneeded)? Alternatively, if you do want to be able to test the bundle URI\nprotocol wholesale across all tests (e.g., in the 'linux-TEST-vars' CI job),\nthen I think the environment variable makes sense.\n\n[1] https://lore.kernel.org/git/acc5a8f57f903342c47802115f8e3de9e9d588dc.1670262639.git.gitgitgadget@gmail.com/\n\n> diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh\n> index d44c6e10f9e..77bfd4f0119 100644\n> --- a/t/lib-bundle-uri-protocol.sh\n> +++ b/t/lib-bundle-uri-protocol.sh\n> @@ -85,9 +85,10 @@ test_expect_success \"connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav\n> '\n> \n> test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris\" '\n> -\ttest_when_finished \"rm -rf log cloned\" &&\n> +\ttest_when_finished \"rm -rf log cloned cloned2\" &&\n> \n> \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n> +\tGIT_TEST_BUNDLE_URI=0 \\\n> \tgit \\\n> \t\t-c protocol.version=2 \\\n> \t\tclone \"$BUNDLE_URI_REPO_URI\" cloned \\\n> @@ -99,6 +100,22 @@ test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque\n> \t# Server advertised bundle-uri capability\n> \tgrep \"< bundle-uri\" log &&\n> \n> +\t# Client did not issue bundle-uri command\n> +\t! grep \"> command=bundle-uri\" log &&\n> +\n> +\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n> +\tgit \\\n> +\t\t-c transfer.bundleURI=true \\\n> +\t\t-c protocol.version=2 \\\n> +\t\tclone \"$BUNDLE_URI_REPO_URI\" cloned2 \\\n> +\t\t>actual 2>err &&\n\nIf 'GIT_TEST_BUNDLE_URI' is set to '1' in a more global scope (by a CI job\nor user running the tests), then the '-c transfer.bundleURI' config isn't\nactually what's enabling the behavior. To make this more directly comparable\nto the case earlier in this test, could you add 'GIT_TEST_BUNDLE_URI=0' here\nas well?\n\n"},{"id":"468556","messageId":"c0a0dd7b-3ffa-eff7-8ddd-19e8efd24575@github.com","threadId":"58731","inReplyTo":"46a58e83caf6d37e5a62b39d3c2e35e278211297.1670262639.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 06/11] bundle-uri client: add helper for testing server","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-12-05T23:32:35Z","receivedAt":"2022-12-05T23:32:42Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n> From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n> \n> \n> Add a 'test-tool bundle-uri ls-remote' command. This is a thin wrapper\n> for issuing protocol v2 \"bundle-uri\" commands to a server, and to the\n> parsing routines in bundle-uri.c.\n> \n> In the \"git clone\" case we'll have already done the handshake(),\n> but not here. Add an extra case to check for this handshake in\n> get_bundle_uri() for ease of use for future callers. Rename the existing\n> 'got_remote_heads' to 'finished_handshake' to make it more clear what\n> that bit represents.\n\nnit: I think this last sentence (\"Rename...\") is out-of-place, since you\nmade that change in patch 5 [1].\n\n[1] https://lore.kernel.org/git/b009b4f58ea312e40af385ea5ca7ede5ea07391a.1670262639.git.gitgitgadget@gmail.com/\n\n> diff --git a/transport.c b/transport.c\n> index a2281d95262..97d395e10a3 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -371,6 +371,13 @@ static int get_bundle_uri(struct transport *transport)\n> \t\tinit_bundle_list(transport->bundles);\n> \t}\n> \n> +\tif (!data->finished_handshake) {\n> +\t\tstruct ref *refs = handshake(transport, 0, NULL, 0);\n> +\n> +\t\tif (refs)\n> +\t\t\tfree_refs(refs);\n> +\t}\n\nThis makes more sense without the extra assertions. Thanks!\n\n"},{"id":"468557","messageId":"2b21f22b-b707-7705-a41c-9e4acbdb0405@github.com","threadId":"58731","inReplyTo":"48731438d6a289129a5768b80af856fb49692426.1670262639.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 09/11] bundle-uri: allow relative URLs in bundle lists","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-12-05T23:33:03Z","receivedAt":"2022-12-05T23:34:05Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> Allow a bundle list to specify a relative URI for the bundles. This URI\n> is based on where the client received the bundle list. For a list\n> provided in the 'bundle-uri' protocol v2 command, the Git remote URI is\n> the base URI. Otherwise, the bundle list was provided from an HTTP URI\n> not using the Git protocol, and that URI is the base URI. This allows\n> easier distribution of bundle data.\n\nThanks, this clears up my confusion about the source of 'baseURI'.\n\n> +\t/**\n> +\t * The baseURI of a bundle_list is the URI that provided the list.\n> +\t *\n> +\t * In the case of the 'bundle-uri' protocol v2 command, the base\n> +\t * URI is the URI of the Git remote.\n> +\t *\n> +\t * Otherewise, the bundle list was downloaded over HTTP from some\n> +\t * known URI.\n\ns/Otherewise/Otherwise\n\nAlso, this sentence is a bit more vague than what was noted in the commit\nmessage; it doesn't actually say what the base URI is set to in this\nscenario. Feel free to ignore if you think it's overkill, but that could\nprobably be cleared up by adding another sentence after like \"The base URI\nis set to that known URI.\"\n\n> +\t *\n> +\t * The baseURI is used as the base for any relative URIs\n> +\t * advertised by the bundle list at that location.\n> +\t */\n> +\tchar *baseURI;\n\n...\n\n> +\t# TODO: We would prefer if parsing a bundle list would not cause\n> +\t# a die() and instead would give a warning and allow the rest of\n> +\t# a Git command to continue. This test_must_fail is necessary for\n> +\t# now until the interface for relative_url() allows for reporting\n> +\t# an error instead of die()ing.\n> +\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n> +\tgrep \"fatal: cannot strip one component off url\" err\n\nThanks for adding this, I'm content to leave this as a TODO for now.\n\n"},{"id":"468561","messageId":"bafe5d54-1322-d9b8-3721-af34067b5e87@github.com","threadId":"58731","inReplyTo":"pull.1400.v3.git.1670262639.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/11] Bundle URIs IV: advertise over protocol v2","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-12-05T23:42:40Z","receivedAt":"2022-12-05T23:42:48Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee via GitGitGadget wrote:\n> Updates in v3\n> =============\n> \n\nThanks for addressing all of my questions/comments from the past version(s)!\nOverall, the series is easy to follow and cleanly integrates the new\nfunctionality. In this version, I just had a few minor phrasing/typo nits &\nsome questions about 'GIT_TEST_BUNDLE_URI', but otherwise I'm happy with it. \n\n"},{"id":"468615","messageId":"221206.86a640dda3.gmgdl@evledraar.gmail.com","threadId":"58731","inReplyTo":"1eec3426aee12bbd674ebd646075f0d4c0b1f5af.1670262639.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 08/11] strbuf: introduce strbuf_strip_file_from_path()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-12-06T10:06:53Z","receivedAt":"2022-12-06T10:43:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Dec 05 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> The strbuf_parent_directory() method was added as a static method in\n> contrib/scalar by d0feac4e8c0 (scalar: 'register' sets recommended\n> config and starts maintenance, 2021-12-03) and then removed in\n> 65f6a9eb0b9 (scalar: constrain enlistment search, 2022-08-18), but now\n> there is a need for a similar method in the bundle URI feature.\n>\n> Re-add the method, this time in strbuf.c, but with a new name:\n> strbuf_strip_file_from_path(). The method requirements are slightly\n> modified to allow a trailing slash, in which case nothing is done, which\n> makes the name change valuable.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> strbuf.c | 6 ++++++\n> strbuf.h | 11 +++++++++++\n> 2 files changed, 17 insertions(+)\n>\n> diff --git a/strbuf.c b/strbuf.c\n> index 0890b1405c5..c383f41a3c5 100644\n> --- a/strbuf.c\n> +++ b/strbuf.c\n> @@ -1200,3 +1200,9 @@ int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n> \tfree(path2);\n> \treturn res;\n> }\n> +\n> +void strbuf_strip_file_from_path(struct strbuf *sb)\n> +{\n> +\tchar *path_sep = find_last_dir_sep(sb->buf);\n> +\tstrbuf_setlen(sb, path_sep ? path_sep - sb->buf + 1 : 0);\n> +}\n> diff --git a/strbuf.h b/strbuf.h\n> index 76965a17d44..f6dbb9681ee 100644\n> --- a/strbuf.h\n> +++ b/strbuf.h\n> @@ -664,6 +664,17 @@ int launch_sequence_editor(const char *path, struct strbuf *buffer,\n> int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n> \t\t\t const char *const *env);\n> \n> +/*\n> + * Remove the filename from the provided path string. If the path\n> + * contains a trailing separator, then the path is considered a directory\n> + * and nothing is modified.\n> + *\n> + * Examples:\n> + * - \"/path/to/file\" -> \"/path/to/\"\n> + * - \"/path/to/dir/\" -> \"/path/to/dir/\"\n> + */\n> +void strbuf_strip_file_from_path(struct strbuf *sb);\n> +\n> void strbuf_add_lines(struct strbuf *sb,\n> \t\t const char *prefix,\n> \t\t const char *buf,\n\nRe your reply in\nhttps://lore.kernel.org/git/0980dcd4-30eb-4ef4-9369-279abe5ca5a2@github.com/\nI still don't get how this is different from a 1-byte change to\nstrbuf_trim_trailing_dir_sep(), and if it isn't I think it's confusing\nAPI design to have two very different ways to return the same data.\n\nThere you said \"The difference is all about whether or not we start with\na slash _and_ no other slash appears in the path.\".\n\nBut I can't find a case where there's any difference between the two. I\ntried this ad-hoc test on top:\n\t\n\tdiff --git a/help.c b/help.c\n\tindex f1e090a4428..b0866b01439 100644\n\t--- a/help.c\n\t+++ b/help.c\n\t@@ -765,6 +765,16 @@ int cmd_version(int argc, const char **argv, const char *prefix)\n\t \t\t\t \"also print build options\"),\n\t \t\tOPT_END()\n\t \t};\n\t+\tstruct strbuf sb1 = STRBUF_INIT;\n\t+\tstruct strbuf sb2 = STRBUF_INIT;\n\t+\n\t+\tif (getenv(\"STR\")) {\n\t+\t\tstrbuf_addstr(&sb1, getenv(\"STR\"));\n\t+\t\tstrbuf_addstr(&sb2, getenv(\"STR\"));\n\t+\t\tstrbuf_strip_file_from_path(&sb1);\n\t+\t\tstrbuf_trim_trailing_not_dir_sep(&sb2);\n\t+\t\tfprintf(stderr, \"%s: %s | %s\\n\", strcmp(sb1.buf, sb2.buf) ? \"NEQ\" : \"EQ\", sb1.buf, sb2.buf);\n\t+\t}\n\t \n\t \targc = parse_options(argc, argv, prefix, options, usage, 0);\n\t \n\tdiff --git a/strbuf.c b/strbuf.c\n\tindex c383f41a3c5..f75d94556fc 100644\n\t--- a/strbuf.c\n\t+++ b/strbuf.c\n\t@@ -114,13 +114,23 @@ void strbuf_rtrim(struct strbuf *sb)\n\t \tsb->buf[sb->len] = '\\0';\n\t }\n\t \n\t-void strbuf_trim_trailing_dir_sep(struct strbuf *sb)\n\t+static void strbuf_trim_trailing_dir_sep_1(struct strbuf *sb, int flip)\n\t {\n\t-\twhile (sb->len > 0 && is_dir_sep((unsigned char)sb->buf[sb->len - 1]))\n\t+\twhile (sb->len > 0 && is_dir_sep((unsigned char)sb->buf[sb->len - 1]) - flip)\n\t \t\tsb->len--;\n\t \tsb->buf[sb->len] = '\\0';\n\t }\n\t \n\t+void strbuf_trim_trailing_dir_sep(struct strbuf *sb)\n\t+{\n\t+\tstrbuf_trim_trailing_dir_sep_1(sb, 1);\n\t+}\n\t+\n\t+void strbuf_trim_trailing_not_dir_sep(struct strbuf *sb)\n\t+{\n\t+\tstrbuf_trim_trailing_dir_sep_1(sb, 1);\n\t+}\n\t+\n\t void strbuf_trim_trailing_newline(struct strbuf *sb)\n\t {\n\t \tif (sb->len > 0 && sb->buf[sb->len - 1] == '\\n') {\n\tdiff --git a/strbuf.h b/strbuf.h\n\tindex f6dbb9681ee..b936f45ffad 100644\n\t--- a/strbuf.h\n\t+++ b/strbuf.h\n\t@@ -189,6 +189,8 @@ void strbuf_ltrim(struct strbuf *sb);\n\t \n\t /* Strip trailing directory separators */\n\t void strbuf_trim_trailing_dir_sep(struct strbuf *sb);\n\t+/* Strip trailing not-directory separators */\n\t+void strbuf_trim_trailing_not_dir_sep(struct strbuf *sb);\n\t \n\t /* Strip trailing LF or CR/LF */\n\t void strbuf_trim_trailing_newline(struct strbuf *sb);\n\nThen:\n\t\n\t$ for str in a / b/ /c /d/ /e/ /f/g /h/i/ j/k l//m n/o/p //q/r/s/t; do STR=$str ./git version; done 2>&1 | grep :\n\tEQ: | \n\tEQ: / | /\n\tEQ: b/ | b/\n\tEQ: / | /\n\tEQ: /d/ | /d/\n\tEQ: /e/ | /e/\n\tEQ: /f/ | /f/\n\tEQ: /h/i/ | /h/i/\n\tEQ: j/ | j/\n\tEQ: l// | l//\n\tEQ: n/o/ | n/o/\n\tEQ: //q/r/s/ | //q/r/s/\n\nI.e. for those inputs it's the same as the existing\nstrbuf_trim_trailing_dir_sep() with an inverted test. Is there some edge\ncase that I'm missing?\n"},{"id":"468620","messageId":"221206.86wn74bw35.gmgdl@evledraar.gmail.com","threadId":"58731","inReplyTo":"221206.86a640dda3.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 08/11] strbuf: introduce strbuf_strip_file_from_path()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-12-06T11:37:15Z","receivedAt":"2022-12-06T11:39:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Dec 06 2022, Ævar Arnfjörð Bjarmason wrote:\n\n> On Mon, Dec 05 2022, Derrick Stolee via GitGitGadget wrote:\n>\n>> From: Derrick Stolee \n>>\n>> The strbuf_parent_directory() method was added as a static method in\n>> contrib/scalar by d0feac4e8c0 (scalar: 'register' sets recommended\n>> config and starts maintenance, 2021-12-03) and then removed in\n>> 65f6a9eb0b9 (scalar: constrain enlistment search, 2022-08-18), but now\n>> there is a need for a similar method in the bundle URI feature.\n>>\n>> Re-add the method, this time in strbuf.c, but with a new name:\n>> strbuf_strip_file_from_path(). The method requirements are slightly\n>> modified to allow a trailing slash, in which case nothing is done, which\n>> makes the name change valuable.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> strbuf.c | 6 ++++++\n>> strbuf.h | 11 +++++++++++\n>> 2 files changed, 17 insertions(+)\n>>\n>> diff --git a/strbuf.c b/strbuf.c\n>> index 0890b1405c5..c383f41a3c5 100644\n>> --- a/strbuf.c\n>> +++ b/strbuf.c\n>> @@ -1200,3 +1200,9 @@ int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n>> \tfree(path2);\n>> \treturn res;\n>> }\n>> +\n>> +void strbuf_strip_file_from_path(struct strbuf *sb)\n>> +{\n>> +\tchar *path_sep = find_last_dir_sep(sb->buf);\n>> +\tstrbuf_setlen(sb, path_sep ? path_sep - sb->buf + 1 : 0);\n>> +}\n>> diff --git a/strbuf.h b/strbuf.h\n>> index 76965a17d44..f6dbb9681ee 100644\n>> --- a/strbuf.h\n>> +++ b/strbuf.h\n>> @@ -664,6 +664,17 @@ int launch_sequence_editor(const char *path, struct strbuf *buffer,\n>> int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n>> \t\t\t const char *const *env);\n>> \n>> +/*\n>> + * Remove the filename from the provided path string. If the path\n>> + * contains a trailing separator, then the path is considered a directory\n>> + * and nothing is modified.\n>> + *\n>> + * Examples:\n>> + * - \"/path/to/file\" -> \"/path/to/\"\n>> + * - \"/path/to/dir/\" -> \"/path/to/dir/\"\n>> + */\n>> +void strbuf_strip_file_from_path(struct strbuf *sb);\n>> +\n>> void strbuf_add_lines(struct strbuf *sb,\n>> \t\t const char *prefix,\n>> \t\t const char *buf,\n>\n> Re your reply in\n> https://lore.kernel.org/git/0980dcd4-30eb-4ef4-9369-279abe5ca5a2@github.com/\n> I still don't get how this is different from a 1-byte change to\n> strbuf_trim_trailing_dir_sep(), and if it isn't I think it's confusing\n> API design to have two very different ways to return the same data.\n>\n> There you said \"The difference is all about whether or not we start with\n> a slash _and_ no other slash appears in the path.\".\n>\n> But I can't find a case where there's any difference between the two. I\n> tried this ad-hoc test on top:\n> \t\n> \tdiff --git a/help.c b/help.c\n> \tindex f1e090a4428..b0866b01439 100644\n> \t--- a/help.c\n> \t+++ b/help.c\n> \t@@ -765,6 +765,16 @@ int cmd_version(int argc, const char **argv, const char *prefix)\n> \t \t\t\t \"also print build options\"),\n> \t \t\tOPT_END()\n> \t \t};\n> \t+\tstruct strbuf sb1 = STRBUF_INIT;\n> \t+\tstruct strbuf sb2 = STRBUF_INIT;\n> \t+\n> \t+\tif (getenv(\"STR\")) {\n> \t+\t\tstrbuf_addstr(&sb1, getenv(\"STR\"));\n> \t+\t\tstrbuf_addstr(&sb2, getenv(\"STR\"));\n> \t+\t\tstrbuf_strip_file_from_path(&sb1);\n> \t+\t\tstrbuf_trim_trailing_not_dir_sep(&sb2);\n> \t+\t\tfprintf(stderr, \"%s: %s | %s\\n\", strcmp(sb1.buf, sb2.buf) ? \"NEQ\" : \"EQ\", sb1.buf, sb2.buf);\n> \t+\t}\n> \t \n> \t \targc = parse_options(argc, argv, prefix, options, usage, 0);\n> \t \n> \tdiff --git a/strbuf.c b/strbuf.c\n> \tindex c383f41a3c5..f75d94556fc 100644\n> \t--- a/strbuf.c\n> \t+++ b/strbuf.c\n> \t@@ -114,13 +114,23 @@ void strbuf_rtrim(struct strbuf *sb)\n> \t \tsb->buf[sb->len] = '\\0';\n> \t }\n> \t \n> \t-void strbuf_trim_trailing_dir_sep(struct strbuf *sb)\n> \t+static void strbuf_trim_trailing_dir_sep_1(struct strbuf *sb, int flip)\n> \t {\n> \t-\twhile (sb->len > 0 && is_dir_sep((unsigned char)sb->buf[sb->len - 1]))\n> \t+\twhile (sb->len > 0 && is_dir_sep((unsigned char)sb->buf[sb->len - 1]) - flip)\n> \t \t\tsb->len--;\n> \t \tsb->buf[sb->len] = '\\0';\n> \t }\n> \t \n> \t+void strbuf_trim_trailing_dir_sep(struct strbuf *sb)\n> \t+{\n> \t+\tstrbuf_trim_trailing_dir_sep_1(sb, 1);\n> \t+}\n> \t+\n> \t+void strbuf_trim_trailing_not_dir_sep(struct strbuf *sb)\n> \t+{\n> \t+\tstrbuf_trim_trailing_dir_sep_1(sb, 1);\n> \t+}\n> \t+\n> \t void strbuf_trim_trailing_newline(struct strbuf *sb)\n> \t {\n> \t \tif (sb->len > 0 && sb->buf[sb->len - 1] == '\\n') {\n> \tdiff --git a/strbuf.h b/strbuf.h\n> \tindex f6dbb9681ee..b936f45ffad 100644\n> \t--- a/strbuf.h\n> \t+++ b/strbuf.h\n> \t@@ -189,6 +189,8 @@ void strbuf_ltrim(struct strbuf *sb);\n> \t \n> \t /* Strip trailing directory separators */\n> \t void strbuf_trim_trailing_dir_sep(struct strbuf *sb);\n> \t+/* Strip trailing not-directory separators */\n> \t+void strbuf_trim_trailing_not_dir_sep(struct strbuf *sb);\n> \t \n> \t /* Strip trailing LF or CR/LF */\n> \t void strbuf_trim_trailing_newline(struct strbuf *sb);\n>\n> Then:\n> \t\n> \t$ for str in a / b/ /c /d/ /e/ /f/g /h/i/ j/k l//m n/o/p //q/r/s/t; do STR=$str ./git version; done 2>&1 | grep :\n> \tEQ: | \n> \tEQ: / | /\n> \tEQ: b/ | b/\n> \tEQ: / | /\n> \tEQ: /d/ | /d/\n> \tEQ: /e/ | /e/\n> \tEQ: /f/ | /f/\n> \tEQ: /h/i/ | /h/i/\n> \tEQ: j/ | j/\n> \tEQ: l// | l//\n> \tEQ: n/o/ | n/o/\n> \tEQ: //q/r/s/ | //q/r/s/\n>\n> I.e. for those inputs it's the same as the existing\n> strbuf_trim_trailing_dir_sep() with an inverted test. Is there some edge\n> case that I'm missing?\n\nFWIW the \"overkill\" change on top to do this via callbacks is the\nbelow. Which I tested just to see how easy it was, and whether it would\nfail your tests (it doesn't).\n\n-- >8 --\nSubject: [PATCH] strbuf: generalize \"{,r,l}trim\" to a callback interface\n\nWe've had all three variants of \"trim\" for isspace(), then since\nc64a8d200f4 (worktree move: accept destination as directory,\n2018-02-12) we've had a \"is_dir_sep\" variant.\n\nA preceding change then added a \"!is_dir_sep\" variant. Let's\ngeneralize this, and have all these functions that want to trim\ncharacters matching some criteria be driven by the same logic.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n bundle-uri.c | 7 +------\n git-compat-util.h | 5 +++++\n strbuf.c | 44 ++++++++++++++++++++++++++++----------------\n strbuf.h | 41 +++++++++++++++++++++++++++--------------\n 4 files changed, 61 insertions(+), 36 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex c411b871bdd..7240dedcaee 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -195,13 +195,8 @@ int bundle_uri_parse_config_format(const char *uri,\n \tif (!list->baseURI) {\n \t\tstruct strbuf baseURI = STRBUF_INIT;\n \t\tstrbuf_addstr(&baseURI, uri);\n+\t\tstrbuf_trim_trailing_not_dir_sep(&baseURI);\n \n-\t\t/*\n-\t\t * If the URI does not end with a trailing slash, then\n-\t\t * remove the filename portion of the path. This is\n-\t\t * important for relative URIs.\n-\t\t */\n-\t\tstrbuf_strip_file_from_path(&baseURI);\n \t\tlist->baseURI = strbuf_detach(&baseURI, NULL);\n \t}\n \tresult = git_config_from_file_with_options(config_to_bundle_list,\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex a76d0526f79..5bce9fa768c 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -451,6 +451,11 @@ static inline int git_is_dir_sep(int c)\n #define is_dir_sep git_is_dir_sep\n #endif\n \n+static inline int is_not_dir_sep(int c)\n+{\n+\treturn !is_dir_sep(c);\n+}\n+\n #ifndef offset_1st_component\n static inline int git_offset_1st_component(const char *path)\n {\ndiff --git a/strbuf.c b/strbuf.c\nindex c383f41a3c5..a5a1c01d539 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -101,24 +101,37 @@ void strbuf_grow(struct strbuf *sb, size_t extra)\n \t\tsb->buf[0] = '\\0';\n }\n \n-void strbuf_trim(struct strbuf *sb)\n+void strbuf_trim_fn(struct strbuf *sb, strbuf_ctype_fn_t fn)\n {\n-\tstrbuf_rtrim(sb);\n-\tstrbuf_ltrim(sb);\n+\tstrbuf_rtrim_fn(sb, fn);\n+\tstrbuf_ltrim_fn(sb, fn);\n }\n \n-void strbuf_rtrim(struct strbuf *sb)\n+void strbuf_rtrim_fn(struct strbuf *sb, strbuf_ctype_fn_t fn)\n {\n-\twhile (sb->len > 0 && isspace((unsigned char)sb->buf[sb->len - 1]))\n+\twhile (sb->len > 0 && fn((unsigned char)sb->buf[sb->len - 1]))\n \t\tsb->len--;\n \tsb->buf[sb->len] = '\\0';\n }\n \n+void strbuf_trim(struct strbuf *sb)\n+{\n+\tstrbuf_trim_fn(sb, strbuf_ctype_isspace);\n+}\n+\n+void strbuf_rtrim(struct strbuf *sb)\n+{\n+\tstrbuf_rtrim_fn(sb, strbuf_ctype_isspace);\n+}\n+\n void strbuf_trim_trailing_dir_sep(struct strbuf *sb)\n {\n-\twhile (sb->len > 0 && is_dir_sep((unsigned char)sb->buf[sb->len - 1]))\n-\t\tsb->len--;\n-\tsb->buf[sb->len] = '\\0';\n+\tstrbuf_rtrim_fn(sb, is_dir_sep);\n+}\n+\n+void strbuf_trim_trailing_not_dir_sep(struct strbuf *sb)\n+{\n+\tstrbuf_rtrim_fn(sb, is_not_dir_sep);\n }\n \n void strbuf_trim_trailing_newline(struct strbuf *sb)\n@@ -130,10 +143,10 @@ void strbuf_trim_trailing_newline(struct strbuf *sb)\n \t}\n }\n \n-void strbuf_ltrim(struct strbuf *sb)\n+void strbuf_ltrim_fn(struct strbuf *sb, strbuf_ctype_fn_t fn)\n {\n \tchar *b = sb->buf;\n-\twhile (sb->len > 0 && isspace(*b)) {\n+\twhile (sb->len > 0 && fn(*b)) {\n \t\tb++;\n \t\tsb->len--;\n \t}\n@@ -141,6 +154,11 @@ void strbuf_ltrim(struct strbuf *sb)\n \tsb->buf[sb->len] = '\\0';\n }\n \n+void strbuf_ltrim(struct strbuf *sb)\n+{\n+\tstrbuf_ltrim_fn(sb, strbuf_ctype_isspace);\n+}\n+\n int strbuf_reencode(struct strbuf *sb, const char *from, const char *to)\n {\n \tchar *out;\n@@ -1200,9 +1218,3 @@ int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n \tfree(path2);\n \treturn res;\n }\n-\n-void strbuf_strip_file_from_path(struct strbuf *sb)\n-{\n-\tchar *path_sep = find_last_dir_sep(sb->buf);\n-\tstrbuf_setlen(sb, path_sep ? path_sep - sb->buf + 1 : 0);\n-}\ndiff --git a/strbuf.h b/strbuf.h\nindex f6dbb9681ee..bb7aa38816f 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -180,15 +180,39 @@ static inline void strbuf_setlen(struct strbuf *sb, size_t len)\n */\n \n /**\n- * Strip whitespace from the beginning (`ltrim`), end (`rtrim`), or both side\n- * (`trim`) of a string.\n+ * A callback function that acts like the macros defined in\n+ * . To be given to strbuf_{,r,l}trim() below.\n+ */\n+typedef int (*strbuf_ctype_fn_t)(int c);\n+static inline int strbuf_ctype_isspace(int c) { return isspace(c); }\n+\n+/**\n+ * Strip characters matching the 'strbuf_ctype_fn_t' from the\n+ * beginning (`ltrim`), end (`rtrim`) or both sides (`trim`) of a\n+ * string.\n+ */\n+void strbuf_trim_fn(struct strbuf *sb, strbuf_ctype_fn_t fn);\n+void strbuf_rtrim_fn(struct strbuf *sb, strbuf_ctype_fn_t fn);\n+void strbuf_ltrim_fn(struct strbuf *sb, strbuf_ctype_fn_t fn);\n+\n+/**\n+ * The common-case wrapper for strbuf_{,r,l}trim_fn() uses the\n+ * strbuf_ctype_isspace() callback function.\n */\n void strbuf_trim(struct strbuf *sb);\n void strbuf_rtrim(struct strbuf *sb);\n void strbuf_ltrim(struct strbuf *sb);\n \n-/* Strip trailing directory separators */\n+/**\n+ * Strip trailing directory separators. This is strbuf_rtrim_fn() with\n+ * is_dir_sep() as the callback..\n+ */\n void strbuf_trim_trailing_dir_sep(struct strbuf *sb);\n+/**\n+ * Strip trailing not-directory separators. This is strbuf_rtrim_fn()\n+ * with is_not_dir_sep() as the callback.\n+ */\n+void strbuf_trim_trailing_not_dir_sep(struct strbuf *sb);\n \n /* Strip trailing LF or CR/LF */\n void strbuf_trim_trailing_newline(struct strbuf *sb);\n@@ -664,17 +688,6 @@ int launch_sequence_editor(const char *path, struct strbuf *buffer,\n int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n \t\t\t const char *const *env);\n \n-/*\n- * Remove the filename from the provided path string. If the path\n- * contains a trailing separator, then the path is considered a directory\n- * and nothing is modified.\n- *\n- * Examples:\n- * - \"/path/to/file\" -> \"/path/to/\"\n- * - \"/path/to/dir/\" -> \"/path/to/dir/\"\n- */\n-void strbuf_strip_file_from_path(struct strbuf *sb);\n-\n void strbuf_add_lines(struct strbuf *sb,\n \t\t const char *prefix,\n \t\t const char *buf,\n-- \n2.39.0.rc1.1014.gc37e9814e18\n\n"},{"id":"468712","messageId":"Y5CNo5kyByIHDVYh@coredump.intra.peff.net","threadId":"58731","inReplyTo":"69bf154bec63a22df8e5eac89f975625ce73c8ac.1670262639.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 10/11] bundle-uri: download bundles from an advertised list","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-12-07T12:57:07Z","receivedAt":"2022-12-07T12:57:13Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Dec 05, 2022 at 05:50:38PM +0000, Derrick Stolee via GitGitGadget wrote:\n\n> +int fetch_bundle_list(struct repository *r, const char *uri, struct bundle_list *list)\n> +{\n> +\tint result;\n> +\tstruct bundle_list global_list;\n> +\n> +\tinit_bundle_list(&global_list);\n> +\n> +\t/* If a bundle is added to this global list, then it is required. */\n> +\tglobal_list.mode = BUNDLE_MODE_ALL;\n> +\n> +\tif ((result = download_bundle_list(r, list, &global_list, 0)))\n> +\t\tgoto cleanup;\n> +\n> +\tresult = unbundle_all_bundles(r, &global_list);\n> +\n> +cleanup:\n> +\tfor_all_bundles_in_list(&global_list, unlink_bundle, NULL);\n> +\tclear_bundle_list(&global_list);\n> +\treturn result;\n> +}\n\nThe \"uri\" parameter in this function is unused. I'm not sure if that's\nindicative of a bug or missing feature (e.g., could it be the base for a\nrelative url?), or if it's just a leftover from development.\n\nIf the latter, I'm happy to add it to my list of cleanups.\n\nThere are a couple other unused parameters in this series, too, but they\nare all in virtual functions and must be kept. I'll add them to my list\nof annotations.\n\n-Peff\n"},{"id":"468720","messageId":"98ee4b5b-9d72-31fc-d5b1-1a978450973f@github.com","threadId":"58731","inReplyTo":"221206.86wn74bw35.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 08/11] strbuf: introduce strbuf_strip_file_from_path()","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-12-07T14:44:35Z","receivedAt":"2022-12-07T14:44:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/6/22 6:37 AM, Ævar Arnfjörð Bjarmason wrote:\n\n> FWIW the \"overkill\" change on top to do this via callbacks is the\n> below. Which I tested just to see how easy it was, and whether it would\n> fail your tests (it doesn't).\n> \n> -- >8 --\n> Subject: [PATCH] strbuf: generalize \"{,r,l}trim\" to a callback interface\n\nI don't like this approach and think it distracts from the goal\nof the series. If you want to update it afterwards, then by all\nmeans go for it.\n\nThanks,\n-Stolee\n"},{"id":"468723","messageId":"8d9d01eb-4c2a-cd25-05f7-e1452c253626@github.com","threadId":"58731","inReplyTo":"b7b98123-0e78-757e-404d-024e167bfe60@github.com","subject":"Re: [PATCH v3 04/11] bundle-uri client: add boolean transfer.bundleURI setting","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-12-07T15:20:11Z","receivedAt":"2022-12-07T15:20:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/5/2022 6:32 PM, Victoria Dye wrote:\n> Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n>> +\t# Client did not issue bundle-uri command\n>> +\t! grep \"> command=bundle-uri\" log &&\n>> +\n>> +\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n>> +\tgit \\\n>> +\t\t-c transfer.bundleURI=true \\\n>> +\t\t-c protocol.version=2 \\\n>> +\t\tclone \"$BUNDLE_URI_REPO_URI\" cloned2 \\\n>> +\t\t>actual 2>err &&\n>\n> If 'GIT_TEST_BUNDLE_URI' is set to '1' in a more global scope (by a CI job\n> or user running the tests), then the '-c transfer.bundleURI' config isn't\n> actually what's enabling the behavior. To make this more directly comparable\n> to the case earlier in this test, could you add 'GIT_TEST_BUNDLE_URI=0' here\n> as well?\n\nYou're right that GIT_TEST_BUNDLE_URI is not needed if we can set\ntransfer.bundleURI globally earlier in the test. It doesn't make much sense\nto run the entire test suite with it on, since the server side does not\nadvertise bundles unless explicitly configured to do so.\n\nThanks,\n-Stolee\n"},{"id":"468724","messageId":"76f9a25b-721b-29c9-9ad1-6174387aeeb8@github.com","threadId":"58731","inReplyTo":"2b21f22b-b707-7705-a41c-9e4acbdb0405@github.com","subject":"Re: [PATCH v3 09/11] bundle-uri: allow relative URLs in bundle lists","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-12-07T15:22:20Z","receivedAt":"2022-12-07T15:22:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/5/2022 6:33 PM, Victoria Dye wrote:\n> Derrick Stolee via GitGitGadget wrote:\n>> Allow a bundle list to specify a relative URI for the bundles. This URI\n>> is based on where the client received the bundle list. For a list\n>> provided in the 'bundle-uri' protocol v2 command, the Git remote URI is\n>> the base URI. Otherwise, the bundle list was provided from an HTTP URI\n>> not using the Git protocol, and that URI is the base URI. This allows\n>> easier distribution of bundle data.\n> \n> Thanks, this clears up my confusion about the source of 'baseURI'.\n> \n>> +\t/**\n>> +\t * The baseURI of a bundle_list is the URI that provided the list.\n>> +\t *\n>> +\t * In the case of the 'bundle-uri' protocol v2 command, the base\n>> +\t * URI is the URI of the Git remote.\n>> +\t *\n>> +\t * Otherewise, the bundle list was downloaded over HTTP from some\n>> +\t * known URI.\n> \n> s/Otherewise/Otherwise\n> \n> Also, this sentence is a bit more vague than what was noted in the commit\n> message; it doesn't actually say what the base URI is set to in this\n> scenario. Feel free to ignore if you think it's overkill, but that could\n> probably be cleared up by adding another sentence after like \"The base URI\n> is set to that known URI.\"\n\nThanks for both of these suggestions.\n\n-Stolee\n"},{"id":"468725","messageId":"affbc458-d4f5-525f-d431-5ec1d489afc8@github.com","threadId":"58731","inReplyTo":"Y5CNo5kyByIHDVYh@coredump.intra.peff.net","subject":"Re: [PATCH v3 10/11] bundle-uri: download bundles from an advertised list","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-12-07T15:27:06Z","receivedAt":"2022-12-07T15:27:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/7/2022 7:57 AM, Jeff King wrote:\n> On Mon, Dec 05, 2022 at 05:50:38PM +0000, Derrick Stolee via GitGitGadget wrote:\n> \n>> +int fetch_bundle_list(struct repository *r, const char *uri, struct bundle_list *list)\n>> +{\n>> +\tint result;\n>> +\tstruct bundle_list global_list;\n>> +\n>> +\tinit_bundle_list(&global_list);\n>> +\n>> +\t/* If a bundle is added to this global list, then it is required. */\n>> +\tglobal_list.mode = BUNDLE_MODE_ALL;\n>> +\n>> +\tif ((result = download_bundle_list(r, list, &global_list, 0)))\n>> +\t\tgoto cleanup;\n>> +\n>> +\tresult = unbundle_all_bundles(r, &global_list);\n>> +\n>> +cleanup:\n>> +\tfor_all_bundles_in_list(&global_list, unlink_bundle, NULL);\n>> +\tclear_bundle_list(&global_list);\n>> +\treturn result;\n>> +}\n> \n> The \"uri\" parameter in this function is unused. I'm not sure if that's\n> indicative of a bug or missing feature (e.g., could it be the base for a\n> relative url?), or if it's just a leftover from development.\n\nThanks for your careful eye. This 'uri' is indeed not needed. I think it\nwas initially there for relative URIs, but the given 'list' is expected\nto have that value initialized. I'll make it clear in the doc comment.\n \n> If the latter, I'm happy to add it to my list of cleanups.\n> \n> There are a couple other unused parameters in this series, too, but they\n> are all in virtual functions and must be kept. I'll add them to my list\n> of annotations.\n\nYour UNUSED annotations exist in my tree, so I'll try my best to update\nthem in the next version.\n\nThanks,\n-Stolee\n"},{"id":"468726","messageId":"4d4e02c3-89dc-8372-7e8a-7ec76fdd6f4e@github.com","threadId":"58731","inReplyTo":"affbc458-d4f5-525f-d431-5ec1d489afc8@github.com","subject":"Re: [PATCH v3 10/11] bundle-uri: download bundles from an advertised list","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-12-07T15:54:31Z","receivedAt":"2022-12-07T15:54:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/7/2022 10:27 AM, Derrick Stolee wrote:\n> On 12/7/2022 7:57 AM, Jeff King wrote:\n\n>> There are a couple other unused parameters in this series, too, but they\n>> are all in virtual functions and must be kept. I'll add them to my list\n>> of annotations.\n> \n> Your UNUSED annotations exist in my tree, so I'll try my best to update\n> them in the next version.\n\nOne of these showed an example where I should have been using\nrepo_config_...() instead of git_config_...(). Thanks!\n\n-Stolee\n"},{"id":"468767","messageId":"Y5GF8ams0k03kiVU@coredump.intra.peff.net","threadId":"58731","inReplyTo":"affbc458-d4f5-525f-d431-5ec1d489afc8@github.com","subject":"Re: [PATCH v3 10/11] bundle-uri: download bundles from an advertised list","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-12-08T06:36:33Z","receivedAt":"2022-12-08T06:36:37Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 07, 2022 at 10:27:06AM -0500, Derrick Stolee wrote:\n\n> > The \"uri\" parameter in this function is unused. I'm not sure if that's\n> > indicative of a bug or missing feature (e.g., could it be the base for a\n> > relative url?), or if it's just a leftover from development.\n> \n> Thanks for your careful eye. This 'uri' is indeed not needed. I think it\n> was initially there for relative URIs, but the given 'list' is expected\n> to have that value initialized. I'll make it clear in the doc comment.\n\nThat makes sense. I've queued a patch locally to remove it (since\nlocally I build with -Wunused-parameters), which will eventually make\nits way to the list.\n\n> > If the latter, I'm happy to add it to my list of cleanups.\n> > \n> > There are a couple other unused parameters in this series, too, but they\n> > are all in virtual functions and must be kept. I'll add them to my list\n> > of annotations.\n> \n> Your UNUSED annotations exist in my tree, so I'll try my best to update\n> them in the next version.\n\nSounds good (and again, I've queued something locally, but if you beat\nme to it, it's easy to drop mine).\n\nNote that your series hit 'next' (which is how I noticed it), so there\nusually would not be a \"next version\". Though we will rewind\npost-release, so there may still be an opportunity (I didn't follow the\ntopic closely enough to know if you might want to re-roll for other\nreasons).\n\n-Peff\n"},{"id":"468768","messageId":"Y5GG0HT7V36vinb8@coredump.intra.peff.net","threadId":"58731","inReplyTo":"4d4e02c3-89dc-8372-7e8a-7ec76fdd6f4e@github.com","subject":"Re: [PATCH v3 10/11] bundle-uri: download bundles from an advertised list","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-12-08T06:40:16Z","receivedAt":"2022-12-08T06:40:22Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 07, 2022 at 10:54:31AM -0500, Derrick Stolee wrote:\n\n> On 12/7/2022 10:27 AM, Derrick Stolee wrote:\n> > On 12/7/2022 7:57 AM, Jeff King wrote:\n> \n> >> There are a couple other unused parameters in this series, too, but they\n> >> are all in virtual functions and must be kept. I'll add them to my list\n> >> of annotations.\n> > \n> > Your UNUSED annotations exist in my tree, so I'll try my best to update\n> > them in the next version.\n> \n> One of these showed an example where I should have been using\n> repo_config_...() instead of git_config_...(). Thanks!\n\nOh good. It is always satisfying when the warning finds a potential bug,\nas it sometimes feels a bit like annotation make-work. ;)\n\nIn this instance we're in server-side v2 protocol code, which is already\nvery global-heavy in its world-view. So I don't think it's a real bug\nhere, but just a nice-to-have.\n\nI have seen this \"oops, we don't really use our repository parameter\"\nissue in a few places. And while I do think it's best to use it if you\nhave it, I suspect it's the tip of the iceberg in terms of functions\nusing the_repository. In the long run, I think we'll really smoke those\nout from the bottom up, as more functions insist on taking a repository\nparameter (and then their callers will have to switch or face the\nembarrassment of passing the_repository themselves, and so on).\n\nAll of which is to say that yes, that is a fine change to make. But I\ndon't consider at all urgent in this instance.\n\n-Peff\n"},{"id":"468783","messageId":"221208.86k0326nse.gmgdl@evledraar.gmail.com","threadId":"58731","inReplyTo":"98ee4b5b-9d72-31fc-d5b1-1a978450973f@github.com","subject":"Re: [PATCH v3 08/11] strbuf: introduce strbuf_strip_file_from_path()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-12-08T12:52:44Z","receivedAt":"2022-12-08T13:14:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Dec 07 2022, Derrick Stolee wrote:\n\n> On 12/6/22 6:37 AM, Ævar Arnfjörð Bjarmason wrote:\n>\n>> FWIW the \"overkill\" change on top to do this via callbacks is the\n>> below. Which I tested just to see how easy it was, and whether it would\n>> fail your tests (it doesn't).\n>> \n>> -- >8 --\n>> Subject: [PATCH] strbuf: generalize \"{,r,l}trim\" to a callback interface\n>\n> I don't like this approach and think it distracts from the goal\n> of the series. If you want to update it afterwards, then by all\n> means go for it.\n\nYes, I think it shouldn't be part of this series at all.\n\nI.e. the part in [1] that you're replying to is just something I poked\nat because you nerd-sniped me into poking at it.\n\nI think it's probably not worth it, and I don't think I like the\nAPI[2]. What I do think is worth including in this series one way or\nanother is [3].\n\nYou're proposing a new addition to the strbuf API. I think it's relevant\nfeedback that you seem to be re-inventing close relative (literally a\n1-byte difference!) of a function that's there already.\n\nOr I'm just wrong, but then what input does your\nstrbuf_strip_file_from_path() handle differently than the\nstrbuf_trim_trailing_not_dir_sep() in [3]?\n\nMaking them sibling functions would make the API more discoverable. The\ncomment you're adding would also improve the existing code. I.e. we\ncould have this end-state in strbuf.h:\n\n\t/**\n\t * Strip trailing directory separators, or not-directory separators.\n\t *\n\t * The \"dir_sep\" variant portably trims redundant slash(es) from the\n\t * end, while the \"not_dir_sep\" gets you to the base directory, should\n\t * the path refer to a file:\n\t *\n\t * |---------------+---------------+-------------------|\n\t * | In | out (dir_sep) | out (not_dir_sep) |\n\t * |---------------+---------------+-------------------|\n\t * | /path/to/file | /path/to/file | /path/to/ |\n\t * | /path/to/dir/ | /path/to/dir | /path/to/dir/ |\n\t * |---------------+---------------+-------------------|\n\t */\n\tvoid strbuf_trim_trailing_dir_sep(struct strbuf *sb);\n\tvoid strbuf_trim_trailing_not_dir_sep(struct strbuf *sb);\n\nOr maybe you still think it's not worth it, I also think that's\nfine. I'd really appreciate knowing if it's a \"yeah maybe they're the\nsame, but I haven't checked\", or if it's \"I think you missed a case, but\nI haven't explained it to you\".\n\nOtherwise if I do follow-up I'd probably have to start by brute-force\ntesting the two to satisfy my own paranoia :)\n\nThanks.\n\n1. https://lore.kernel.org/git/221206.86wn74bw35.gmgdl@evledraar.gmail.com/\n2. Although once I started poking at it I found a lot of cases in-tree\n where we hardcoded that exact loop (or the equivalent strbuf_setlen()\n variant), which could be replaced by \"trim all cases of this\n character from the end of a strbuf\", or \"trim all cases of this\n character match ...\" function.\n3. https://lore.kernel.org/git/221206.86a640dda3.gmgdl@evledraar.gmail.com/\n"},{"id":"468789","messageId":"5ba28171-ede1-8f47-8813-239f591c165c@github.com","threadId":"58731","inReplyTo":"Y5GF8ams0k03kiVU@coredump.intra.peff.net","subject":"Re: [PATCH v3 10/11] bundle-uri: download bundles from an advertised list","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-12-08T14:58:30Z","receivedAt":"2022-12-08T14:58:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/8/2022 1:36 AM, Jeff King wrote:\n> On Wed, Dec 07, 2022 at 10:27:06AM -0500, Derrick Stolee wrote:\n> \n>>> The \"uri\" parameter in this function is unused. I'm not sure if that's\n>>> indicative of a bug or missing feature (e.g., could it be the base for a\n>>> relative url?), or if it's just a leftover from development.\n>>\n>> Thanks for your careful eye. This 'uri' is indeed not needed. I think it\n>> was initially there for relative URIs, but the given 'list' is expected\n>> to have that value initialized. I'll make it clear in the doc comment.\n> \n> That makes sense. I've queued a patch locally to remove it (since\n> locally I build with -Wunused-parameters), which will eventually make\n> its way to the list.\n> \n>>> If the latter, I'm happy to add it to my list of cleanups.\n>>>\n>>> There are a couple other unused parameters in this series, too, but they\n>>> are all in virtual functions and must be kept. I'll add them to my list\n>>> of annotations.\n>>\n>> Your UNUSED annotations exist in my tree, so I'll try my best to update\n>> them in the next version.\n> \n> Sounds good (and again, I've queued something locally, but if you beat\n> me to it, it's easy to drop mine).\n> \n> Note that your series hit 'next' (which is how I noticed it), so there\n> usually would not be a \"next version\". Though we will rewind\n> post-release, so there may still be an opportunity (I didn't follow the\n> topic closely enough to know if you might want to re-roll for other\n> reasons).\n\nI noticed that after reading this round of review, so I'll be preparing\nsome fixes on top. I noticed some UNUSED that would be necessary from\nearlier parts of the bundle URI work, and you've probably already queued\nthose changes.\n\nSince I no longer plan to re-roll this series, I'd be happy to review\nyour queued annotations, and I'll focus on the other fixups.\n\nThanks,\n-Stolee\n"},{"id":"469457","messageId":"pull.1400.v4.git.1671722058.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v3.git.1670262639.gitgitgadget@gmail.com","subject":"[PATCH v4 00/11] Bundle URIs IV: advertise over protocol v2","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-22T15:14:06Z","receivedAt":"2022-12-22T15:14:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is based on the recent master batch that included ds/bundle-uri-....\n\nNow that git clone --bundle-uri can download a bundle list from a plaintex\nfile in config format, we can use the same set of key-value pairs to\nadvertise a bundle list over protocol v2. At the end of this series:\n\n 1. A server can advertise bundles when uploadPack.advertiseBundleURIs is\n enabled. The bundle list comes from the server's local config,\n specifically the bundle.* namespace.\n 2. A client can notice a server's bundle-uri advertisement and request the\n bundle list if transfer.bundleURI is enabled. The bundles are downloaded\n as if the list was advertised from the --bundle-uri option.\n\nMany patches in this series were adapted from Ævar's v2 RFC [1]. He is\nretained as author and I added myself as co-author only if the modifications\nwere significant.\n\n[1]\nhttps://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/\n\n * Patches 1-7 are mostly taken from [1], again with mostly minor updates.\n The one major difference is the packet line format being a single\n key=value format instead of a sequence of pairs. (In v3, these commits\n are significantly reorganized from [1].)\n\n * Patches 8-11 finish off the ability for the client to notice the\n capability, request the values, and download bundles before continuing\n with the rest of the download.\n\nOne thing that is not handled here but could be handled in a future change\nis to disconnect from the origin Git server while downloading the bundle\nURIs, then reconnecting afterwards. This does not make any difference for\nHTTPS, but SSH may benefit from the reduced connection time. The git clone\n--bundle-uri option did not suffer from this because the bundles are\ndownloaded before the server connection begins.\n\nAfter this series, there is one more before the original scope of the plan\nis complete: using creation tokens as a heuristic. See [2] for the RFC\nversion of those patches.\n\n[2] https://github.com/derrickstolee/git/pull/22\n\n\nUpdates in v4\n=============\n\nThis version includes squashed-in versions of the fixups that were\npreviously known as ds/bundle-uri-4-fixup.\n\n * Some unused parameters are now marked with UNUSED, since we are\n introducing those parameters for the first time. In one case an unused\n parameter should have been used in repo_config_...() instead of\n git_config_...().\n * The GIT_TEST_BUNDLE_URI environment variable is removed in favor of the\n transfer.bundleURI config option in all cases.\n * A stale commit message is fixed to no longer refer to a rename that was\n split into a different commit as part of v3.\n * The documentation comment for fetch_bundle_list() explicitly defines a\n non-zero return value as an error.\n\n\nUpdates in v3\n=============\n\nMost of these updates are due to Victoria's very thorough review. Thanks!\n\n * What was patch 2 was split to be better understood.\n * The new patch 2 is only the new test script infrastructure for testing\n whether or not the server provides the bundle-uri capability. It is\n extended with other more complicated examples in later patches. The name\n was rewritten from lib-t5730-*.sh to lib-bundle-uri-protocol.sh and the\n variable names are renamed with the BUNDLE_URI_ prefix.\n * The new patch 3 performs the basic client interaction with the\n 'bundle-uri' command, while still not being fully wired up on the server\n side. The tests do check that the client requests the bundle-uri command\n after seeing it in the server's capabilities. One important difference\n from earlier is that the check for server_supports_v2() was moved into\n the get_bundle_uri() method (underneath the vtable) because we need to\n check the handshake before calling that method. It makes most sense to\n put the handshake call there, so do it from the start.\n * Patch 4 carefully tests how the transfer.bundleURI config blocks the\n client-side request of the bundle-uri command. Later tests will use the\n GIT_TEST_BUNDLE_URI environment variable instead.\n * The new Patch 5 renames got_remote_heads to finished_handshake in 'struct\n git_transport_data' and that's it. That new value is then used in patch 6\n to indicate if we need to request the handshake in the bundle URI logic.\n * Patch 6 creates the ls-remote helper in 'test-tool bundle-uri' as before,\n but now only makes use of the finished_handshake member instead of\n creating a new one. The test helper represents an example consumer of\n transport_get_remote_bundle_uri() without first doing the server-side\n handshake, which motivates several of the placements of code within that\n method and get_bundle_uri() earlier in the series. The \"quiet\" option is\n also removed to simplify the test helper and to always communicate the\n inner errors to the user.\n * Patch 7 adds the server-side listing of bundle.* config values. The test\n scripts around these config values have been cleaned up since the\n previous version.\n * Patch 8 has another iteration of strbuf_strip_file_from_path() taking the\n feedback from Victoria and Ævar.\n * Patch 9 adds the relative path logic. The definition of the base path is\n clarified in the commit message and comments. An additional test shows\n what happens if the server advertises too many parent paths\n (unfortunately, a die(), and this is marked for cleanup later).\n * Patch 10 is identical to the old patch 8.\n * Patch 11 completes the work by having the client download the bundles\n provided by the server list. It fixes an if/else that should have been an\n if/else-if. A new test checks that the --bundle-uri=X option overrides\n the server advertisement.\n\n\nUpdates in v2\n=============\n\n * Commit messages now refer to protocol v2 \"commands\" not \"verbs\".\n * Several edits were made to gitprotocol-v2.txt thanks to Victoria's\n thorough review.\n * strbuf_parent_directory() is renamed strbuf_strip_file_from_path() to\n make it more clear how it behaves when ending with a slash.\n\nThanks,\n\n * Stolee\n\nDerrick Stolee (6):\n transport: rename got_remote_heads\n bundle-uri: serve bundle.* keys from config\n strbuf: introduce strbuf_strip_file_from_path()\n bundle-uri: allow relative URLs in bundle lists\n bundle-uri: download bundles from an advertised list\n clone: unbundle the advertised bundles\n\nÆvar Arnfjörð Bjarmason (5):\n protocol v2: add server-side \"bundle-uri\" skeleton\n t: create test harness for 'bundle-uri' command\n clone: request the 'bundle-uri' command when available\n bundle-uri client: add boolean transfer.bundleURI setting\n bundle-uri client: add helper for testing server\n\n Documentation/config/transfer.txt | 6 +\n Documentation/gitprotocol-v2.txt | 201 +++++++++++++++++++++++\n builtin/clone.c | 21 +++\n bundle-uri.c | 87 +++++++++-\n bundle-uri.h | 35 ++++\n connect.c | 44 +++++\n remote.h | 5 +\n serve.c | 6 +\n strbuf.c | 6 +\n strbuf.h | 11 ++\n t/helper/test-bundle-uri.c | 48 ++++++\n t/lib-bundle-uri-protocol.sh | 216 +++++++++++++++++++++++++\n t/t5601-clone.sh | 59 +++++++\n t/t5701-git-serve.sh | 40 ++++-\n t/t5730-protocol-v2-bundle-uri-file.sh | 17 ++\n t/t5731-protocol-v2-bundle-uri-git.sh | 17 ++\n t/t5732-protocol-v2-bundle-uri-http.sh | 17 ++\n t/t5750-bundle-uri-parse.sh | 82 ++++++++++\n t/t9119-git-svn-info.sh | 2 +-\n t/test-lib-functions.sh | 7 +\n transport-helper.c | 13 ++\n transport-internal.h | 7 +\n transport.c | 87 ++++++++--\n transport.h | 19 +++\n 24 files changed, 1041 insertions(+), 12 deletions(-)\n create mode 100644 t/lib-bundle-uri-protocol.sh\n create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh\n create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh\n create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh\n\n\nbase-commit: c03801e19cb8ab36e9c0d17ff3d5e0c3b0f24193\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1400%2Fderrickstolee%2Fbundle-redo%2Fadvertise-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1400/derrickstolee/bundle-redo/advertise-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1400\n\nRange-diff vs v3:\n\n 1: beae335b855 ! 1: 5ba91813de3 protocol v2: add server-side \"bundle-uri\" skeleton\n @@ bundle-uri.c: cleanup:\n + * API for serve.c.\n + */\n +\n -+int bundle_uri_advertise(struct repository *r, struct strbuf *value)\n ++int bundle_uri_advertise(struct repository *r, struct strbuf *value UNUSED)\n +{\n +\tstatic int advertise_bundle_uri = -1;\n +\n @@ bundle-uri.c: cleanup:\n +\t\tgoto cached;\n +\n +\tadvertise_bundle_uri = 0;\n -+\tgit_config_get_maybe_bool(\"uploadpack.advertisebundleuris\", &advertise_bundle_uri);\n ++\trepo_config_get_maybe_bool(r, \"uploadpack.advertisebundleuris\", &advertise_bundle_uri);\n +\n +cached:\n +\treturn advertise_bundle_uri;\n 2: fcdfef2012a = 2: 3267a5b3b37 t: create test harness for 'bundle-uri' command\n 3: a0188ae39c6 = 3: 5bf6fe55771 clone: request the 'bundle-uri' command when available\n 4: e46118e60f7 ! 4: 876dd3f221f bundle-uri client: add boolean transfer.bundleURI setting\n @@ Commit message\n can set it to 'true' to enable checking for bundle URIs from the origin\n Git server using protocol v2.\n \n - To enable this setting by default in the correct tests, add a\n - GIT_TEST_BUNDLE_URI environment variable.\n -\n Co-authored-by: Derrick Stolee \n Signed-off-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: Derrick Stolee \n @@ t/lib-bundle-uri-protocol.sh: test_expect_success \"connect with $BUNDLE_URI_PROT\n +\ttest_when_finished \"rm -rf log cloned cloned2\" &&\n \n \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n -+\tGIT_TEST_BUNDLE_URI=0 \\\n \tgit \\\n ++\t\t-c transfer.bundleURI=false \\\n \t\t-c protocol.version=2 \\\n \t\tclone \"$BUNDLE_URI_REPO_URI\" cloned \\\n + \t\t>actual 2>err &&\n @@ t/lib-bundle-uri-protocol.sh: test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque\n \t# Server advertised bundle-uri capability\n \tgrep \"< bundle-uri\" log &&\n @@ transport.c: int transport_get_remote_bundle_uri(struct transport *transport)\n \n +\t/*\n +\t * Don't request bundle-uri from the server unless configured to\n -+\t * do so by GIT_TEST_BUNDLE_URI=1 or transfer.bundleURI=true.\n ++\t * do so by the transfer.bundleURI=true config option.\n +\t */\n -+\tif (!git_env_bool(\"GIT_TEST_BUNDLE_URI\", 0) &&\n -+\t (git_config_get_bool(\"transfer.bundleuri\", &value) || !value))\n ++\tif (git_config_get_bool(\"transfer.bundleuri\", &value) || !value)\n +\t\treturn 0;\n +\n \tif (!vtable->get_bundle_uri)\n 5: b009b4f58ea = 5: 8f5a483c329 transport: rename got_remote_heads\n 6: 46a58e83caf ! 6: 13e4c82e338 bundle-uri client: add helper for testing server\n @@ Commit message\n \n In the \"git clone\" case we'll have already done the handshake(),\n but not here. Add an extra case to check for this handshake in\n - get_bundle_uri() for ease of use for future callers. Rename the existing\n - 'got_remote_heads' to 'finished_handshake' to make it more clear what\n - that bit represents.\n + get_bundle_uri() for ease of use for future callers.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: Derrick Stolee \n @@ t/lib-bundle-uri-protocol.sh: test_expect_success \"clone with $BUNDLE_URI_PROTOC\n \tgrep \"> command=bundle-uri\" log\n '\n +\n ++# The remaining tests will all assume transfer.bundleURI=true\n ++#\n ++# This test can be removed when transfer.bundleURI is enabled by default.\n ++test_expect_success 'enable transfer.bundleURI for remaining tests' '\n ++\tgit config --global transfer.bundleURI true\n ++'\n ++\n +test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2\" '\n +\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n +\t\tbundle.only.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED\" &&\n 7: acc5a8f57f9 ! 7: c9b7d8779e4 bundle-uri: serve bundle.* keys from config\n @@ t/lib-bundle-uri-protocol.sh: test_expect_success \"test bundle-uri with $BUNDLE_\n +\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED\n \tEOF\n \n -+\tGIT_TEST_BUNDLE_URI=1 \\\n \ttest-tool bundle-uri \\\n - \t\tls-remote \\\n - \t\t\"$BUNDLE_URI_REPO_URI\" \\\n @@ t/lib-bundle-uri-protocol.sh: test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol\n \t[bundle]\n \t\tversion = 1\n \t\tmode = all\n +\t[bundle \"only\"]\n +\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED\n - \tEOF\n - \n -+\tGIT_TEST_BUNDLE_URI=1 \\\n ++\tEOF\n ++\n +\ttest-tool bundle-uri \\\n +\t\tls-remote \\\n +\t\t\"$BUNDLE_URI_REPO_URI\" \\\n @@ t/lib-bundle-uri-protocol.sh: test_expect_success \"test bundle-uri with $BUNDLE_\n +\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED-2.bdl\n +\t[bundle \"bundle3\"]\n +\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED-3.bdl\n -+\tEOF\n -+\n -+\tGIT_TEST_BUNDLE_URI=1 \\\n + \tEOF\n + \n \ttest-tool bundle-uri \\\n - \t\tls-remote \\\n - \t\t\"$BUNDLE_URI_REPO_URI\" \\\n 8: 1eec3426aee = 8: d13bd6cd95d strbuf: introduce strbuf_strip_file_from_path()\n 9: 48731438d6a ! 9: a188b38399d bundle-uri: allow relative URLs in bundle lists\n @@ bundle-uri.h: struct bundle_list {\n +\t * In the case of the 'bundle-uri' protocol v2 command, the base\n +\t * URI is the URI of the Git remote.\n +\t *\n -+\t * Otherewise, the bundle list was downloaded over HTTP from some\n -+\t * known URI.\n ++\t * Otherwise, the bundle list was downloaded over HTTP from some\n ++\t * known URI. 'baseURI' is set to that value.\n +\t *\n +\t * The baseURI is used as the base for any relative URIs\n +\t * advertised by the bundle list at that location.\n @@ t/t5750-bundle-uri-parse.sh: test_expect_success 'parse config format: just URIs\n \n ## transport.c ##\n @@ transport.c: int transport_get_remote_bundle_uri(struct transport *transport)\n - \t (git_config_get_bool(\"transfer.bundleuri\", &value) || !value))\n + \tif (git_config_get_bool(\"transfer.bundleuri\", &value) || !value)\n \t\treturn 0;\n \n +\tif (!transport->bundles->baseURI)\n 10: 69bf154bec6 ! 10: 72ca6f4254f bundle-uri: download bundles from an advertised list\n @@ bundle-uri.c: cleanup:\n \treturn result;\n }\n \n -+int fetch_bundle_list(struct repository *r, const char *uri, struct bundle_list *list)\n ++int fetch_bundle_list(struct repository *r, struct bundle_list *list)\n +{\n +\tint result;\n +\tstruct bundle_list global_list;\n @@ bundle-uri.h: int bundle_uri_parse_config_format(const char *uri,\n + * bundle-uri protocol v2 verb) at the given uri, fetch and unbundle the\n + * bundles according to the bundle strategy of that list.\n + *\n -+ * Returns non-zero if no bundle information is found at the given 'uri'.\n ++ * It is expected that the given 'list' is initialized, including its\n ++ * 'baseURI' value.\n ++ *\n ++ * Returns non-zero if there was an error trying to download the list\n ++ * or any of its advertised bundles.\n + */\n +int fetch_bundle_list(struct repository *r,\n -+\t\t const char *uri,\n +\t\t struct bundle_list *list);\n +\n /**\n 11: 7e1819162b6 ! 11: 46ab2b05b15 clone: unbundle the advertised bundles\n @@ Commit message\n when the client has chosen to enable the feature.\n \n Teach Git to download and unbundle the data advertised by those bundles\n - during 'git clone'.\n + during 'git clone'. This takes place between the ref advertisement and\n + the object data download, and stateful connections will linger while\n + the client downloads bundles. In the future, we should consider closing\n + the remote connection during this process.\n \n Also, since the --bundle-uri option exists, we do not want to mix the\n advertised bundles with the user-specified bundles.\n @@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n +\t\t\tif (repo_init(the_repository, git_dir, work_tree))\n +\t\t\t\twarning(_(\"failed to initialize the repo, skipping bundle URI\"));\n +\t\t\telse if (fetch_bundle_list(the_repository,\n -+\t\t\t\t\t\t remote->url[0],\n +\t\t\t\t\t\t transport->bundles))\n +\t\t\t\twarning(_(\"failed to fetch advertised bundles\"));\n +\t\t} else {\n @@ t/lib-bundle-uri-protocol.sh: test_expect_success \"connect with $BUNDLE_URI_PROT\n +\ttest_when_finished \"rm -rf log* cloned*\" &&\n \n \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n - \tGIT_TEST_BUNDLE_URI=0 \\\n + \tgit \\\n @@ t/lib-bundle-uri-protocol.sh: test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque\n \tgrep \"< bundle-uri\" log &&\n \n @@ t/lib-bundle-uri-protocol.sh: test_expect_success \"clone with $BUNDLE_URI_PROTOC\n +\t! grep \"> command=bundle-uri\" log3\n '\n \n - test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2\" '\n + # The remaining tests will all assume transfer.bundleURI=true\n \n ## t/t5601-clone.sh ##\n @@ t/t5601-clone.sh: test_expect_success 'reject cloning shallow repository using HTTP' '\n @@ t/t5601-clone.sh: test_expect_success 'reject cloning shallow repository using H\n +\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n +\t\tbundle.everything.uri \"$HTTPD_URL/everything.bundle\" &&\n +\n -+\tGIT_TEST_BUNDLE_URI=1 \\\n +\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n -+\t\tgit -c protocol.version=2 clone \\\n ++\t\tgit -c protocol.version=2 \\\n ++\t\t -c transfer.bundleURI=true clone \\\n +\t\t$HTTPD_URL/smart/repo2.git repo2 &&\n +\tcat >pattern <<-EOF &&\n +\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/everything.bundle\"\\]\n @@ t/t5601-clone.sh: test_expect_success 'reject cloning shallow repository using H\n +\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n +\t\tbundle.new.uri \"$HTTPD_URL/new.bundle\" &&\n +\n -+\tGIT_TEST_BUNDLE_URI=1 \\\n +\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n -+\t\tgit -c protocol.version=2 clone \\\n ++\t\tgit -c protocol.version=2 \\\n ++\t\t -c transfer.bundleURI=true clone \\\n +\t\t$HTTPD_URL/smart/repo3.git repo3 &&\n +\n +\t# We should fetch _both_ bundles\n\n-- \ngitgitgadget\n"},{"id":"469458","messageId":"5ba91813de38d2b6cb2018517460623d94003926.1671722058.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v4.git.1671722058.gitgitgadget@gmail.com","subject":"[PATCH v4 01/11] protocol v2: add server-side \"bundle-uri\" skeleton","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-22T15:14:07Z","receivedAt":"2022-12-22T15:14:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nAdd a skeleton server-side implementation of a new \"bundle-uri\" command\nto protocol v2. This will allow conforming clients to optionally seed\ntheir initial clones or incremental fetches from URLs containing\n\"*.bundle\" files created with \"git bundle create\".\n\nThis change only performs the basic boilerplate of advertising a new\nprotocol v2 capability. The new 'bundle-uri' capability allows a client\nto request a list of bundles. Right now, the server only returns a flush\npacket, which corresponds to an empty advertisement. The bundle.* config\nnamespace describes which key-value pairs will be communicated across\nthis interface in future updates.\n\nThe critical bit right now is that the new boolean\nuploadPack.adverstiseBundleURIs config value signals whether or not this\ncapability should be advertised at all.\n\nAn earlier version of this patch [1] used a different transfer format\nthan the \"key=value\" pairs in the current implementation. The change was\nmade to unify the protocol v2 command with the bundle lists provided by\nindependent bundle servers. Further, the standard allows for the server\nto advertise a URI that contains a bundle list. This allows users\nautomatically discovering bundle providers that are loosely associated\nwith the origin server, but without the origin server knowing exactly\nwhich bundles are currently available.\n\n[1] https://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/\n\nThe very-deep headings needed to be modified to stop at level 4 due to\ndocumentation build issues. These were not recognized in earlier builds\nsince the file was previously in the Documentation/technical/ directory\nand was built in a different way. With its current location, the\nheavily-nested details were causing build issues and they are now\nreplaced with a bulletted list of details.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Documentation/gitprotocol-v2.txt | 201 +++++++++++++++++++++++++++++++\n bundle-uri.c | 36 ++++++\n bundle-uri.h | 7 ++\n serve.c | 6 +\n t/t5701-git-serve.sh | 40 +++++-\n 5 files changed, 289 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\nindex 59bf41cefb9..10bd2d40cec 100644\n--- a/Documentation/gitprotocol-v2.txt\n+++ b/Documentation/gitprotocol-v2.txt\n@@ -578,6 +578,207 @@ and associated requested information, each separated by a single space.\n \n \tobj-info = obj-id SP obj-size\n \n+bundle-uri\n+~~~~~~~~~~\n+\n+If the 'bundle-uri' capability is advertised, the server supports the\n+`bundle-uri' command.\n+\n+The capability is currently advertised with no value (i.e. not\n+\"bundle-uri=somevalue\"), a value may be added in the future for\n+supporting command-wide extensions. Clients MUST ignore any unknown\n+capability values and proceed with the 'bundle-uri` dialog they\n+support.\n+\n+The 'bundle-uri' command is intended to be issued before `fetch` to\n+get URIs to bundle files (see linkgit:git-bundle[1]) to \"seed\" and\n+inform the subsequent `fetch` command.\n+\n+The client CAN issue `bundle-uri` before or after any other valid\n+command. To be useful to clients it's expected that it'll be issued\n+after an `ls-refs` and before `fetch`, but CAN be issued at any time\n+in the dialog.\n+\n+DISCUSSION of bundle-uri\n+^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+The intent of the feature is optimize for server resource consumption\n+in the common case by changing the common case of fetching a very\n+large PACK during linkgit:git-clone[1] into a smaller incremental\n+fetch.\n+\n+It also allows servers to achieve better caching in combination with\n+an `uploadpack.packObjectsHook` (see linkgit:git-config[1]).\n+\n+By having new clones or fetches be a more predictable and common\n+negotiation against the tips of recently produces *.bundle file(s).\n+Servers might even pre-generate the results of such negotiations for\n+the `uploadpack.packObjectsHook` as new pushes come in.\n+\n+One way that servers could take advantage of these bundles is that the\n+server would anticipate that fresh clones will download a known bundle,\n+followed by catching up to the current state of the repository using ref\n+tips found in that bundle (or bundles).\n+\n+PROTOCOL for bundle-uri\n+^^^^^^^^^^^^^^^^^^^^^^^\n+\n+A `bundle-uri` request takes no arguments, and as noted above does not\n+currently advertise a capability value. Both may be added in the\n+future.\n+\n+When the client issues a `command=bundle-uri` request, the response is a\n+list of key-value pairs provided as packet lines with value\n+`=`. Each `` should be interpreted as a config key from\n+the `bundle.*` namespace to construct a list of bundles. These keys are\n+grouped by a `bundle..` subsection, where each key corresponding to a\n+given `` contributes attributes to the bundle defined by that ``.\n+See linkgit:git-config[1] for the specific details of these keys and how\n+the Git client will interpret their values.\n+\n+Clients MUST parse the line according to the above format, lines that do\n+not conform to the format SHOULD be discarded. The user MAY be warned in\n+such a case.\n+\n+bundle-uri CLIENT AND SERVER EXPECTATIONS\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+URI CONTENTS::\n+The content at the advertised URIs MUST be one of two types.\n++\n+The advertised URI may contain a bundle file that `git bundle verify`\n+would accept. I.e. they MUST contain one or more reference tips for\n+use by the client, MUST indicate prerequisites (in any) with standard\n+\"-\" prefixes, and MUST indicate their \"object-format\", if\n+applicable.\n++\n+The advertised URI may alternatively contain a plaintext file that `git\n+config --list` would accept (with the `--file` option). The key-value\n+pairs in this list are in the `bundle.*` namespace (see\n+linkgit:git-config[1]).\n+\n+bundle-uri CLIENT ERROR RECOVERY::\n+A client MUST above all gracefully degrade on errors, whether that\n+error is because of bad missing/data in the bundle URI(s), because\n+that client is too dumb to e.g. understand and fully parse out bundle\n+headers and their prerequisite relationships, or something else.\n++\n+Server operators should feel confident in turning on \"bundle-uri\" and\n+not worry if e.g. their CDN goes down that clones or fetches will run\n+into hard failures. Even if the server bundle bundle(s) are\n+incomplete, or bad in some way the client should still end up with a\n+functioning repository, just as if it had chosen not to use this\n+protocol extension.\n++\n+All subsequent discussion on client and server interaction MUST keep\n+this in mind.\n+\n+bundle-uri SERVER TO CLIENT::\n+The ordering of the returned bundle uris is not significant. Clients\n+MUST parse their headers to discover their contained OIDS and\n+prerequisites. A client MUST consider the content of the bundle(s)\n+themselves and their header as the ultimate source of truth.\n++\n+A server MAY even return bundle(s) that don't have any direct\n+relationship to the repository being cloned (either through accident,\n+or intentional \"clever\" configuration), and expect a client to sort\n+out what data they'd like from the bundle(s), if any.\n+\n+bundle-uri CLIENT TO SERVER::\n+The client SHOULD provide reference tips found in the bundle header(s)\n+as 'have' lines in any subsequent `fetch` request. A client MAY also\n+ignore the bundle(s) entirely if doing so is deemed worse for some\n+reason, e.g. if the bundles can't be downloaded, it doesn't like the\n+tips it finds etc.\n+\n+WHEN ADVERTISED BUNDLE(S) REQUIRE NO FURTHER NEGOTIATION::\n+If after issuing `bundle-uri` and `ls-refs`, and getting the header(s)\n+of the bundle(s) the client finds that the ref tips it wants can be\n+retrieved entirely from advertised bundle(s), the client MAY disconnect\n+from the Git server. The results of such a 'clone' or 'fetch' should be\n+indistinguishable from the state attained without using bundle-uri.\n+\n+EARLY CLIENT DISCONNECTIONS AND ERROR RECOVERY::\n+A client MAY perform an early disconnect while still downloading the\n+bundle(s) (having streamed and parsed their headers). In such a case\n+the client MUST gracefully recover from any errors related to\n+finishing the download and validation of the bundle(s).\n++\n+I.e. a client might need to re-connect and issue a 'fetch' command,\n+and possibly fall back to not making use of 'bundle-uri' at all.\n++\n+This \"MAY\" behavior is specified as such (and not a \"SHOULD\") on the\n+assumption that a server advertising bundle uris is more likely than\n+not to be serving up a relatively large repository, and to be pointing\n+to URIs that have a good chance of being in working order. A client\n+MAY e.g. look at the payload size of the bundles as a heuristic to see\n+if an early disconnect is worth it, should falling back on a full\n+\"fetch\" dialog be necessary.\n+\n+WHEN ADVERTISED BUNDLE(S) REQUIRE FURTHER NEGOTIATION::\n+A client SHOULD commence a negotiation of a PACK from the server via\n+the \"fetch\" command using the OID tips found in advertised bundles,\n+even if's still in the process of downloading those bundle(s).\n++\n+This allows for aggressive early disconnects from any interactive\n+server dialog. The client blindly trusts that the advertised OID tips\n+are relevant, and issues them as 'have' lines, it then requests any\n+tips it would like (usually from the \"ls-refs\" advertisement) via\n+'want' lines. The server will then compute a (hopefully small) PACK\n+with the expected difference between the tips from the bundle(s) and\n+the data requested.\n++\n+The only connection the client then needs to keep active is to the\n+concurrently downloading static bundle(s), when those and the\n+incremental PACK are retrieved they should be inflated and\n+validated. Any errors at this point should be gracefully recovered\n+from, see above.\n+\n+bundle-uri PROTOCOL FEATURES\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+The client constructs a bundle list from the `=` pairs\n+provided by the server. These pairs are part of the `bundle.*` namespace\n+as documented in linkgit:git-config[1]. In this section, we discuss some\n+of these keys and describe the actions the client will do in response to\n+this information.\n+\n+In particular, the `bundle.version` key specifies an integer value. The\n+only accepted value at the moment is `1`, but if the client sees an\n+unexpected value here then the client MUST ignore the bundle list.\n+\n+As long as `bundle.version` is understood, all other unknown keys MAY be\n+ignored by the client. The server will guarantee compatibility with older\n+clients, though newer clients may be better able to use the extra keys to\n+minimize downloads.\n+\n+Any backwards-incompatible addition of pre-URI key-value will be\n+guarded by a new `bundle.version` value or values in 'bundle-uri'\n+capability advertisement itself, and/or by new future `bundle-uri`\n+request arguments.\n+\n+Some example key-value pairs that are not currently implemented but could\n+be implemented in the future include:\n+\n+ * Add a \"hash=\" or \"size=\" advertise the expected hash or\n+ size of the bundle file.\n+\n+ * Advertise that one or more bundle files are the same (to e.g. have\n+ clients round-robin or otherwise choose one of N possible files).\n+\n+ * A \"oid=\" shortcut and \"prerequisite=\" shortcut. For\n+ expressing the common case of a bundle with one tip and no\n+ prerequisites, or one tip and one prerequisite.\n++\n+This would allow for optimizing the common case of servers who'd like\n+to provide one \"big bundle\" containing only their \"main\" branch,\n+and/or incremental updates thereof.\n++\n+A client receiving such a a response MAY assume that they can skip\n+retrieving the header from a bundle at the indicated URI, and thus\n+save themselves and the server(s) the request(s) needed to inspect the\n+headers of that bundle or bundles.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 79a914f961b..28d8966005e 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -563,6 +563,42 @@ cleanup:\n \treturn result;\n }\n \n+/**\n+ * API for serve.c.\n+ */\n+\n+int bundle_uri_advertise(struct repository *r, struct strbuf *value UNUSED)\n+{\n+\tstatic int advertise_bundle_uri = -1;\n+\n+\tif (advertise_bundle_uri != -1)\n+\t\tgoto cached;\n+\n+\tadvertise_bundle_uri = 0;\n+\trepo_config_get_maybe_bool(r, \"uploadpack.advertisebundleuris\", &advertise_bundle_uri);\n+\n+cached:\n+\treturn advertise_bundle_uri;\n+}\n+\n+int bundle_uri_command(struct repository *r,\n+\t\t struct packet_reader *request)\n+{\n+\tstruct packet_writer writer;\n+\tpacket_writer_init(&writer, 1);\n+\n+\twhile (packet_reader_read(request) == PACKET_READ_NORMAL)\n+\t\tdie(_(\"bundle-uri: unexpected argument: '%s'\"), request->line);\n+\tif (request->status != PACKET_READ_FLUSH)\n+\t\tdie(_(\"bundle-uri: expected flush after arguments\"));\n+\n+\t/* TODO: Implement the communication */\n+\n+\tpacket_writer_flush(&writer);\n+\n+\treturn 0;\n+}\n+\n /**\n * General API for {transport,connect}.c etc.\n */\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 4dbc269823c..357111ecce8 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -4,6 +4,7 @@\n #include \"hashmap.h\"\n #include \"strbuf.h\"\n \n+struct packet_reader;\n struct repository;\n struct string_list;\n \n@@ -92,6 +93,12 @@ int bundle_uri_parse_config_format(const char *uri,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n+/**\n+ * API for serve.c.\n+ */\n+int bundle_uri_advertise(struct repository *r, struct strbuf *value);\n+int bundle_uri_command(struct repository *r, struct packet_reader *request);\n+\n /**\n * General API for {transport,connect}.c etc.\n */\ndiff --git a/serve.c b/serve.c\nindex 733347f602a..cbf4a143cfe 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -7,6 +7,7 @@\n #include \"protocol-caps.h\"\n #include \"serve.h\"\n #include \"upload-pack.h\"\n+#include \"bundle-uri.h\"\n \n static int advertise_sid = -1;\n static int client_hash_algo = GIT_HASH_SHA1;\n@@ -135,6 +136,11 @@ static struct protocol_capability capabilities[] = {\n \t\t.advertise = always_advertise,\n \t\t.command = cap_object_info,\n \t},\n+\t{\n+\t\t.name = \"bundle-uri\",\n+\t\t.advertise = bundle_uri_advertise,\n+\t\t.command = bundle_uri_command,\n+\t},\n };\n \n void protocol_v2_advertise_capabilities(void)\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex 1896f671cb3..f21e5e9d33d 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -13,7 +13,7 @@ test_expect_success 'test capability advertisement' '\n \twrong_algo sha1:sha256\n \twrong_algo sha256:sha1\n \tEOF\n-\tcat >expect <<-EOF &&\n+\tcat >expect.base <<-EOF &&\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n \tls-refs=unborn\n@@ -21,8 +21,11 @@ test_expect_success 'test capability advertisement' '\n \tserver-option\n \tobject-format=$(test_oid algo)\n \tobject-info\n+\tEOF\n+\tcat >expect.trailer <<-EOF &&\n \t0000\n \tEOF\n+\tcat expect.base expect.trailer >expect &&\n \n \tGIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \\\n \t\t--advertise-capabilities >out &&\n@@ -342,4 +345,39 @@ test_expect_success 'basics of object-info' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'test capability advertisement with uploadpack.advertiseBundleURIs' '\n+\ttest_config uploadpack.advertiseBundleURIs true &&\n+\n+\tcat >expect.extra <<-EOF &&\n+\tbundle-uri\n+\tEOF\n+\tcat expect.base \\\n+\t expect.extra \\\n+\t expect.trailer >expect &&\n+\n+\tGIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \\\n+\t\t--advertise-capabilities >out &&\n+\ttest-tool pkt-line unpack actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'basics of bundle-uri: dies if not enabled' '\n+\ttest-tool pkt-line pack >in <<-EOF &&\n+\tcommand=bundle-uri\n+\t0000\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid command '\"'\"'bundle-uri'\"'\"'\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\tERR serve: invalid command '\"'\"'bundle-uri'\"'\"'\n+\tEOF\n+\n+\ttest_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"469459","messageId":"3267a5b3b372d9b33ad3afd22bde7ba5559331ae.1671722058.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v4.git.1671722058.gitgitgadget@gmail.com","subject":"[PATCH v4 02/11] t: create test harness for 'bundle-uri' command","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-22T15:14:08Z","receivedAt":"2022-12-22T15:14:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nThe previous change allowed for a Git server to advertise the\n'bundle-uri' command as a capability based on the\nuploadPack.advertiseBundleURIs config option. Create a set of tests that\ncheck that this capability is advertised using 'git ls-remote'.\n\nIn order to test this functionality across three protocols (file, git,\nand http), create lib-bundle-uri-protocol.sh to generalize the tests,\nallowing the other test scripts to set an environment variable and\notherwise inherit the setup and tests from this script.\n\nThe tests currently only test that the 'bundle-uri' command is\nadvertised or not. Other actions will be tested as the Git client learns\nto request the 'bundle-uri' command and parse its response.\n\nTo help with URI escaping, specifically for file paths with a space in\nthem, extract a 'sed' invocation from t9199-git-svn-info.sh into a\nhelper function for use here, too.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n t/lib-bundle-uri-protocol.sh | 85 ++++++++++++++++++++++++++\n t/t5730-protocol-v2-bundle-uri-file.sh | 17 ++++++\n t/t5731-protocol-v2-bundle-uri-git.sh | 17 ++++++\n t/t5732-protocol-v2-bundle-uri-http.sh | 17 ++++++\n t/t9119-git-svn-info.sh | 2 +-\n t/test-lib-functions.sh | 7 +++\n 6 files changed, 144 insertions(+), 1 deletion(-)\n create mode 100644 t/lib-bundle-uri-protocol.sh\n create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh\n create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh\n create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh\n\ndiff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh\nnew file mode 100644\nindex 00000000000..2da22a39cb8\n--- /dev/null\n+++ b/t/lib-bundle-uri-protocol.sh\n@@ -0,0 +1,85 @@\n+# Set up and run tests of the 'bundle-uri' command in protocol v2\n+#\n+# The test that includes this script should set BUNDLE_URI_PROTOCOL\n+# to one of \"file\", \"git\", or \"http\".\n+\n+BUNDLE_URI_TEST_PARENT=\n+BUNDLE_URI_TEST_URI=\n+BUNDLE_URI_TEST_BUNDLE_URI=\n+case \"$BUNDLE_URI_PROTOCOL\" in\n+file)\n+\tBUNDLE_URI_PARENT=file_parent\n+\tBUNDLE_URI_REPO_URI=\"file://$PWD/file_parent\"\n+\tBUNDLE_URI_BUNDLE_URI=\"$BUNDLE_URI_REPO_URI/fake.bdl\"\n+\ttest_set_prereq BUNDLE_URI_FILE\n+\t;;\n+git)\n+\t. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+\tstart_git_daemon --export-all --enable=receive-pack\n+\tBUNDLE_URI_PARENT=\"$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\"\n+\tBUNDLE_URI_REPO_URI=\"$GIT_DAEMON_URL/parent\"\n+\tBUNDLE_URI_BUNDLE_URI=\"https://example.com/fake.bdl\"\n+\ttest_set_prereq BUNDLE_URI_GIT\n+\t;;\n+http)\n+\t. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\tstart_httpd\n+\tBUNDLE_URI_PARENT=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\"\n+\tBUNDLE_URI_REPO_URI=\"$HTTPD_URL/smart/http_parent\"\n+\tBUNDLE_URI_BUNDLE_URI=\"https://example.com/fake.bdl\"\n+\ttest_set_prereq BUNDLE_URI_HTTP\n+\t;;\n+*)\n+\tBUG \"Need to pass valid BUNDLE_URI_PROTOCOL (was \\\"$BUNDLE_URI_PROTOCOL\\\")\"\n+\t;;\n+esac\n+\n+test_expect_success \"setup protocol v2 $BUNDLE_URI_PROTOCOL:// tests\" '\n+\tgit init \"$BUNDLE_URI_PARENT\" &&\n+\ttest_commit -C \"$BUNDLE_URI_PARENT\" one &&\n+\tgit -C \"$BUNDLE_URI_PARENT\" config uploadpack.advertiseBundleURIs true\n+'\n+\n+case \"$BUNDLE_URI_PROTOCOL\" in\n+http)\n+\ttest_expect_success \"setup config for $BUNDLE_URI_PROTOCOL:// tests\" '\n+\t\tgit -C \"$BUNDLE_URI_PARENT\" config http.receivepack true\n+\t'\n+\t;;\n+*)\n+\t;;\n+esac\n+BUNDLE_URI_BUNDLE_URI_ESCAPED=$(echo \"$BUNDLE_URI_BUNDLE_URI\" | test_uri_escape)\n+\n+test_expect_success \"connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: no bundle-uri\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\ttest_when_finished \"git -C \\\"$BUNDLE_URI_PARENT\\\" config uploadpack.advertiseBundleURIs true\" &&\n+\tgit -C \"$BUNDLE_URI_PARENT\" config uploadpack.advertiseBundleURIs false &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tgit \\\n+\t\t-c protocol.version=2 \\\n+\t\tls-remote --symref \"$BUNDLE_URI_REPO_URI\" \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\t! grep bundle-uri log\n+'\n+\n+test_expect_success \"connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: have bundle-uri\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tgit \\\n+\t\t-c protocol.version=2 \\\n+\t\tls-remote --symref \"$BUNDLE_URI_REPO_URI\" \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\t# Server advertised bundle-uri capability\n+\tgrep \"< bundle-uri\" log\n+'\ndiff --git a/t/t5730-protocol-v2-bundle-uri-file.sh b/t/t5730-protocol-v2-bundle-uri-file.sh\nnew file mode 100755\nindex 00000000000..37bdb725bca\n--- /dev/null\n+++ b/t/t5730-protocol-v2-bundle-uri-file.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri with protocol v2 and 'file://' transport\"\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'file://' transport\n+#\n+BUNDLE_URI_PROTOCOL=file\n+. \"$TEST_DIRECTORY\"/lib-bundle-uri-protocol.sh\n+\n+test_done\ndiff --git a/t/t5731-protocol-v2-bundle-uri-git.sh b/t/t5731-protocol-v2-bundle-uri-git.sh\nnew file mode 100755\nindex 00000000000..8add1b37abc\n--- /dev/null\n+++ b/t/t5731-protocol-v2-bundle-uri-git.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri with protocol v2 and 'git://' transport\"\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'git://' transport\n+#\n+BUNDLE_URI_PROTOCOL=git\n+. \"$TEST_DIRECTORY\"/lib-bundle-uri-protocol.sh\n+\n+test_done\ndiff --git a/t/t5732-protocol-v2-bundle-uri-http.sh b/t/t5732-protocol-v2-bundle-uri-http.sh\nnew file mode 100755\nindex 00000000000..129daa02269\n--- /dev/null\n+++ b/t/t5732-protocol-v2-bundle-uri-http.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri with protocol v2 and 'http://' transport\"\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'http://' transport\n+#\n+BUNDLE_URI_PROTOCOL=http\n+. \"$TEST_DIRECTORY\"/lib-bundle-uri-protocol.sh\n+\n+test_done\ndiff --git a/t/t9119-git-svn-info.sh b/t/t9119-git-svn-info.sh\nindex 8201c3e808a..088d1c57a88 100755\n--- a/t/t9119-git-svn-info.sh\n+++ b/t/t9119-git-svn-info.sh\n@@ -28,7 +28,7 @@ test_cmp_info () {\n \trm -f tmp.expect tmp.actual\n }\n \n-quoted_svnrepo=\"$(echo $svnrepo | sed 's/ /%20/')\"\n+quoted_svnrepo=\"$(echo $svnrepo | test_uri_escape)\"\n \n test_expect_success 'setup repository and import' '\n \tmkdir info &&\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 29d914a12ba..5f6966a404b 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1755,6 +1755,13 @@ test_path_is_hidden () {\n \treturn 1\n }\n \n+# Poor man's URI escaping. Good enough for the test suite whose trash\n+# directory has a space in it. See 93c3fcbe4d4 (git-svn: attempt to\n+# mimic SVN 1.7 URL canonicalization, 2012-07-28) for prior art.\n+test_uri_escape() {\n+\tsed 's/ /%20/g'\n+}\n+\n # Check that the given command was invoked as part of the\n # trace2-format trace on stdin.\n #\n-- \ngitgitgadget\n\n"},{"id":"469460","messageId":"5bf6fe55771a5687d711c59cf0460baa087603a3.1671722058.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v4.git.1671722058.gitgitgadget@gmail.com","subject":"[PATCH v4 03/11] clone: request the 'bundle-uri' command when available","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-22T15:14:09Z","receivedAt":"2022-12-22T15:14:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nSet up all the needed client parts of the 'bundle-uri' protocol v2\ncommand, without actually doing anything with the bundle URIs.\n\nIf the server says it supports 'bundle-uri' teach Git to issue the\n'bundle-uri' command after the 'ls-refs' during 'git clone'. The\nreturned key=value pairs are passed to the bundle list code which is\ntested using a different ingest mechanism in t5750-bundle-uri-parse.sh.\n\nAt this point, Git does nothing with that bundle list. It will not\ndownload any of the bundles. That will come in a later change after\nthese protocol bits are finalized.\n\nThe no-op client is initially used only by 'git clone' to test the basic\nfunctionality, and eventually will bootstrap the initial download of Git\nobjects during a fresh clone. The bundle URI client will not be\nintegrated into other fetches until a mechanism is created to select a\nsubset of bundles for download.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 6 +++++\n connect.c | 44 +++++++++++++++++++++++++++++++\n remote.h | 5 ++++\n t/lib-bundle-uri-protocol.sh | 19 ++++++++++++++\n transport-helper.c | 13 +++++++++\n transport-internal.h | 7 +++++\n transport.c | 51 ++++++++++++++++++++++++++++++++++++\n transport.h | 19 ++++++++++++++\n 8 files changed, 164 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 547d6464b3c..39364c25b15 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1266,6 +1266,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (refs)\n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n \n+\t/*\n+\t * Populate transport->got_remote_bundle_uri and\n+\t * transport->bundle_uri. We might get nothing.\n+\t */\n+\ttransport_get_remote_bundle_uri(transport);\n+\n \tif (mapped_refs) {\n \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n \ndiff --git a/connect.c b/connect.c\nindex 5ea53deda23..624a10f18ee 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -15,6 +15,7 @@\n #include \"version.h\"\n #include \"protocol.h\"\n #include \"alias.h\"\n+#include \"bundle-uri.h\"\n \n static char *server_capabilities_v1;\n static struct strvec server_capabilities_v2 = STRVEC_INIT;\n@@ -491,6 +492,49 @@ static void send_capabilities(int fd_out, struct packet_reader *reader)\n \t}\n }\n \n+int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\n+\t\t\t struct bundle_list *bundles, int stateless_rpc)\n+{\n+\tint line_nr = 1;\n+\n+\t/* Assert bundle-uri support */\n+\tserver_supports_v2(\"bundle-uri\", 1);\n+\n+\t/* (Re-)send capabilities */\n+\tsend_capabilities(fd_out, reader);\n+\n+\t/* Send command */\n+\tpacket_write_fmt(fd_out, \"command=bundle-uri\\n\");\n+\tpacket_delim(fd_out);\n+\n+\tpacket_flush(fd_out);\n+\n+\t/* Process response from server */\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\tconst char *line = reader->line;\n+\t\tline_nr++;\n+\n+\t\tif (!bundle_uri_parse_line(bundles, line))\n+\t\t\tcontinue;\n+\n+\t\treturn error(_(\"error on bundle-uri response line %d: %s\"),\n+\t\t\t line_nr, line);\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH)\n+\t\treturn error(_(\"expected flush after bundle-uri listing\"));\n+\n+\t/*\n+\t * Might die(), but obscure enough that that's OK, e.g. in\n+\t * serve.c we'll call BUG() on its equivalent (the\n+\t * PACKET_READ_RESPONSE_END check).\n+\t */\n+\tcheck_stateless_delimiter(stateless_rpc, reader,\n+\t\t\t\t _(\"expected response end packet after ref listing\"));\n+\n+\treturn 0;\n+}\n+\n struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\t struct ref **list, int for_push,\n \t\t\t struct transport_ls_refs_options *transport_options,\ndiff --git a/remote.h b/remote.h\nindex 1c4621b414b..1ebbe42792e 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -234,6 +234,11 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\t const struct string_list *server_options,\n \t\t\t int stateless_rpc);\n \n+/* Used for protocol v2 in order to retrieve refs from a remote */\n+struct bundle_list;\n+int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\n+\t\t\t struct bundle_list *bundles, int stateless_rpc);\n+\n int resolve_remote_symref(struct ref *ref, struct ref *list);\n \n /*\ndiff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh\nindex 2da22a39cb8..d44c6e10f9e 100644\n--- a/t/lib-bundle-uri-protocol.sh\n+++ b/t/lib-bundle-uri-protocol.sh\n@@ -83,3 +83,22 @@ test_expect_success \"connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav\n \t# Server advertised bundle-uri capability\n \tgrep \"< bundle-uri\" log\n '\n+\n+test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris\" '\n+\ttest_when_finished \"rm -rf log cloned\" &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tgit \\\n+\t\t-c protocol.version=2 \\\n+\t\tclone \"$BUNDLE_URI_REPO_URI\" cloned \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\t# Server advertised bundle-uri capability\n+\tgrep \"< bundle-uri\" log &&\n+\n+\t# Client issued bundle-uri command\n+\tgrep \"> command=bundle-uri\" log\n+'\ndiff --git a/transport-helper.c b/transport-helper.c\nindex e95267a4ab5..3ea7c2bb5ad 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -1267,9 +1267,22 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n \treturn ret;\n }\n \n+static int get_bundle_uri(struct transport *transport)\n+{\n+\tget_helper(transport);\n+\n+\tif (process_connect(transport, 0)) {\n+\t\tdo_take_over(transport);\n+\t\treturn transport->vtable->get_bundle_uri(transport);\n+\t}\n+\n+\treturn -1;\n+}\n+\n static struct transport_vtable vtable = {\n \t.set_option\t= set_helper_option,\n \t.get_refs_list\t= get_refs_list,\n+\t.get_bundle_uri = get_bundle_uri,\n \t.fetch_refs\t= fetch_refs,\n \t.push_refs\t= push_refs,\n \t.connect\t= connect_helper,\ndiff --git a/transport-internal.h b/transport-internal.h\nindex c4ca0b733ac..90ea749e5cf 100644\n--- a/transport-internal.h\n+++ b/transport-internal.h\n@@ -26,6 +26,13 @@ struct transport_vtable {\n \tstruct ref *(*get_refs_list)(struct transport *transport, int for_push,\n \t\t\t\t struct transport_ls_refs_options *transport_options);\n \n+\t/**\n+\t * Populates the remote side's bundle-uri under protocol v2,\n+\t * if the \"bundle-uri\" capability was advertised. Returns 0 if\n+\t * OK, negative values on error.\n+\t */\n+\tint (*get_bundle_uri)(struct transport *transport);\n+\n \t/**\n \t * Fetch the objects for the given refs. Note that this gets\n \t * an array, and should ignore the list structure.\ndiff --git a/transport.c b/transport.c\nindex e7b97194c10..b6f279e92cb 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -22,6 +22,7 @@\n #include \"protocol.h\"\n #include \"object-store.h\"\n #include \"color.h\"\n+#include \"bundle-uri.h\"\n \n static int transport_use_color = -1;\n static char transport_colors[][COLOR_MAXLEN] = {\n@@ -359,6 +360,32 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \treturn handshake(transport, for_push, options, 1);\n }\n \n+static int get_bundle_uri(struct transport *transport)\n+{\n+\tstruct git_transport_data *data = transport->data;\n+\tstruct packet_reader reader;\n+\tint stateless_rpc = transport->stateless_rpc;\n+\n+\tif (!transport->bundles) {\n+\t\tCALLOC_ARRAY(transport->bundles, 1);\n+\t\tinit_bundle_list(transport->bundles);\n+\t}\n+\n+\t/*\n+\t * \"Support\" protocol v0 and v2 without bundle-uri support by\n+\t * silently degrading to a NOOP.\n+\t */\n+\tif (!server_supports_v2(\"bundle-uri\", 0))\n+\t\treturn 0;\n+\n+\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\treturn get_remote_bundle_uri(data->fd[1], &reader,\n+\t\t\t\t transport->bundles, stateless_rpc);\n+}\n+\n static int fetch_refs_via_pack(struct transport *transport,\n \t\t\t int nr_heads, struct ref **to_fetch)\n {\n@@ -902,6 +929,7 @@ static int disconnect_git(struct transport *transport)\n \n static struct transport_vtable taken_over_vtable = {\n \t.get_refs_list\t= get_refs_via_connect,\n+\t.get_bundle_uri = get_bundle_uri,\n \t.fetch_refs\t= fetch_refs_via_pack,\n \t.push_refs\t= git_transport_push,\n \t.disconnect\t= disconnect_git\n@@ -1054,6 +1082,7 @@ static struct transport_vtable bundle_vtable = {\n \n static struct transport_vtable builtin_smart_vtable = {\n \t.get_refs_list\t= get_refs_via_connect,\n+\t.get_bundle_uri = get_bundle_uri,\n \t.fetch_refs\t= fetch_refs_via_pack,\n \t.push_refs\t= git_transport_push,\n \t.connect\t= connect_git,\n@@ -1068,6 +1097,9 @@ struct transport *transport_get(struct remote *remote, const char *url)\n \tret->progress = isatty(2);\n \tstring_list_init_dup(&ret->pack_lockfiles);\n \n+\tCALLOC_ARRAY(ret->bundles, 1);\n+\tinit_bundle_list(ret->bundles);\n+\n \tif (!remote)\n \t\tBUG(\"No remote provided to transport_get()\");\n \n@@ -1482,6 +1514,23 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)\n \treturn rc;\n }\n \n+int transport_get_remote_bundle_uri(struct transport *transport)\n+{\n+\tconst struct transport_vtable *vtable = transport->vtable;\n+\n+\t/* Check config only once. */\n+\tif (transport->got_remote_bundle_uri)\n+\t\treturn 0;\n+\ttransport->got_remote_bundle_uri = 1;\n+\n+\tif (!vtable->get_bundle_uri)\n+\t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n+\n+\tif (vtable->get_bundle_uri(transport) < 0)\n+\t\treturn error(_(\"could not retrieve server-advertised bundle-uri list\"));\n+\treturn 0;\n+}\n+\n void transport_unlock_pack(struct transport *transport, unsigned int flags)\n {\n \tint in_signal_handler = !!(flags & TRANSPORT_UNLOCK_PACK_IN_SIGNAL_HANDLER);\n@@ -1512,6 +1561,8 @@ int transport_disconnect(struct transport *transport)\n \t\tret = transport->vtable->disconnect(transport);\n \tif (transport->got_remote_refs)\n \t\tfree_refs((void *)transport->remote_refs);\n+\tclear_bundle_list(transport->bundles);\n+\tfree(transport->bundles);\n \tfree(transport);\n \treturn ret;\n }\ndiff --git a/transport.h b/transport.h\nindex b5bf7b3e704..85150f504fb 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -62,6 +62,7 @@ enum transport_family {\n \tTRANSPORT_FAMILY_IPV6\n };\n \n+struct bundle_list;\n struct transport {\n \tconst struct transport_vtable *vtable;\n \n@@ -76,6 +77,18 @@ struct transport {\n \t */\n \tunsigned got_remote_refs : 1;\n \n+\t/**\n+\t * Indicates whether we already called get_bundle_uri_list(); set by\n+\t * transport.c::transport_get_remote_bundle_uri().\n+\t */\n+\tunsigned got_remote_bundle_uri : 1;\n+\n+\t/*\n+\t * The results of \"command=bundle-uri\", if both sides support\n+\t * the \"bundle-uri\" capability.\n+\t */\n+\tstruct bundle_list *bundles;\n+\n \t/*\n \t * Transports that call take-over destroys the data specific to\n \t * the transport type while doing so, and cannot be reused.\n@@ -281,6 +294,12 @@ void transport_ls_refs_options_release(struct transport_ls_refs_options *opts);\n const struct ref *transport_get_remote_refs(struct transport *transport,\n \t\t\t\t\t struct transport_ls_refs_options *transport_options);\n \n+/**\n+ * Retrieve bundle URI(s) from a remote. Populates \"struct\n+ * transport\"'s \"bundle_uri\" and \"got_remote_bundle_uri\".\n+ */\n+int transport_get_remote_bundle_uri(struct transport *transport);\n+\n /*\n * Fetch the hash algorithm used by a remote.\n *\n-- \ngitgitgadget\n\n"},{"id":"469461","messageId":"8f5a483c32958b8ab2ac6deeef383165e7214260.1671722058.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v4.git.1671722058.gitgitgadget@gmail.com","subject":"[PATCH v4 05/11] transport: rename got_remote_heads","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-22T15:14:11Z","receivedAt":"2022-12-22T15:14:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'got_remote_heads' member of 'struct git_transport_data' was used\nhistorically to indicate that the initial server connection was made and\nthe ref advertisement was returned. With protocol v2, that initial\nhandshake does not necessarily include the ref advertisement, so this\nmember is not an accurate name. Thankfully, all uses of the member are\nonly checking to see if the handshake should take place, not whether or\nnot some local data has the ref advertisement.\n\nRename the member to 'finished_handshake' to represent the proper state.\nNote that the variable is only set to 1 during the handshake() method.\n\nSigned-off-by: Derrick Stolee \n---\n transport.c | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex b4cf2c0252e..757ad552bf3 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -198,7 +198,7 @@ struct git_transport_data {\n \tstruct git_transport_options options;\n \tstruct child_process *conn;\n \tint fd[2];\n-\tunsigned got_remote_heads : 1;\n+\tunsigned finished_handshake : 1;\n \tenum protocol_version version;\n \tstruct oid_array extra_have;\n \tstruct oid_array shallow;\n@@ -345,7 +345,7 @@ static struct ref *handshake(struct transport *transport, int for_push,\n \tcase protocol_unknown_version:\n \t\tBUG(\"unknown protocol version\");\n \t}\n-\tdata->got_remote_heads = 1;\n+\tdata->finished_handshake = 1;\n \ttransport->hash_algo = reader.hash_algo;\n \n \tif (reader.line_peeked)\n@@ -421,7 +421,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.negotiation_tips = data->options.negotiation_tips;\n \targs.reject_shallow_remote = transport->smart_options->reject_shallow;\n \n-\tif (!data->got_remote_heads) {\n+\tif (!data->finished_handshake) {\n \t\tint i;\n \t\tint must_list_refs = 0;\n \t\tfor (i = 0; i < nr_heads; i++) {\n@@ -461,7 +461,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\t\t to_fetch, nr_heads, &data->shallow,\n \t\t\t &transport->pack_lockfiles, data->version);\n \n-\tdata->got_remote_heads = 0;\n+\tdata->finished_handshake = 0;\n \tdata->options.self_contained_and_connected =\n \t\targs.self_contained_and_connected;\n \tdata->options.connectivity_checked = args.connectivity_checked;\n@@ -846,7 +846,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \tif (transport_color_config() < 0)\n \t\treturn -1;\n \n-\tif (!data->got_remote_heads)\n+\tif (!data->finished_handshake)\n \t\tget_refs_via_connect(transport, 1, NULL);\n \n \tmemset(&args, 0, sizeof(args));\n@@ -894,7 +894,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \telse\n \t\tret = finish_connect(data->conn);\n \tdata->conn = NULL;\n-\tdata->got_remote_heads = 0;\n+\tdata->finished_handshake = 0;\n \n \treturn ret;\n }\n@@ -914,7 +914,7 @@ static int disconnect_git(struct transport *transport)\n {\n \tstruct git_transport_data *data = transport->data;\n \tif (data->conn) {\n-\t\tif (data->got_remote_heads && !transport->stateless_rpc)\n+\t\tif (data->finished_handshake && !transport->stateless_rpc)\n \t\t\tpacket_flush(data->fd[1]);\n \t\tclose(data->fd[0]);\n \t\tif (data->fd[1] >= 0)\n@@ -949,7 +949,7 @@ void transport_take_over(struct transport *transport,\n \tdata->conn = child;\n \tdata->fd[0] = data->conn->out;\n \tdata->fd[1] = data->conn->in;\n-\tdata->got_remote_heads = 0;\n+\tdata->finished_handshake = 0;\n \ttransport->data = data;\n \n \ttransport->vtable = &taken_over_vtable;\n@@ -1150,7 +1150,7 @@ struct transport *transport_get(struct remote *remote, const char *url)\n \t\tret->smart_options = &(data->options);\n \n \t\tdata->conn = NULL;\n-\t\tdata->got_remote_heads = 0;\n+\t\tdata->finished_handshake = 0;\n \t} else {\n \t\t/* Unknown protocol in URL. Pass to external handler. */\n \t\tint len = external_specification_len(url);\n-- \ngitgitgadget\n\n"},{"id":"469462","messageId":"876dd3f221fe575dc0407983e15280de4e9e18e0.1671722058.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v4.git.1671722058.gitgitgadget@gmail.com","subject":"[PATCH v4 04/11] bundle-uri client: add boolean transfer.bundleURI setting","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-22T15:14:10Z","receivedAt":"2022-12-22T15:14:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nThe yet-to-be introduced client support for bundle-uri will always\nfall back on a full clone, but we'd still like to be able to ignore a\nserver's bundle-uri advertisement entirely.\n\nThe new transfer.bundleURI config option defaults to 'false', but a user\ncan set it to 'true' to enable checking for bundle URIs from the origin\nGit server using protocol v2.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Documentation/config/transfer.txt | 6 ++++++\n t/lib-bundle-uri-protocol.sh | 19 ++++++++++++++++++-\n transport.c | 8 ++++++++\n 3 files changed, 32 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/transfer.txt b/Documentation/config/transfer.txt\nindex 264812cca4d..c3ac767d1e4 100644\n--- a/Documentation/config/transfer.txt\n+++ b/Documentation/config/transfer.txt\n@@ -115,3 +115,9 @@ transfer.unpackLimit::\n transfer.advertiseSID::\n \tBoolean. When true, client and server processes will advertise their\n \tunique session IDs to their remote counterpart. Defaults to false.\n+\n+transfer.bundleURI::\n+\tWhen `true`, local `git clone` commands will request bundle\n+\tinformation from the remote server (if advertised) and download\n+\tbundles before continuing the clone through the Git protocol.\n+\tDefaults to `false`.\ndiff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh\nindex d44c6e10f9e..75ea8c4418f 100644\n--- a/t/lib-bundle-uri-protocol.sh\n+++ b/t/lib-bundle-uri-protocol.sh\n@@ -85,10 +85,11 @@ test_expect_success \"connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav\n '\n \n test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris\" '\n-\ttest_when_finished \"rm -rf log cloned\" &&\n+\ttest_when_finished \"rm -rf log cloned cloned2\" &&\n \n \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n \tgit \\\n+\t\t-c transfer.bundleURI=false \\\n \t\t-c protocol.version=2 \\\n \t\tclone \"$BUNDLE_URI_REPO_URI\" cloned \\\n \t\t>actual 2>err &&\n@@ -99,6 +100,22 @@ test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque\n \t# Server advertised bundle-uri capability\n \tgrep \"< bundle-uri\" log &&\n \n+\t# Client did not issue bundle-uri command\n+\t! grep \"> command=bundle-uri\" log &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tgit \\\n+\t\t-c transfer.bundleURI=true \\\n+\t\t-c protocol.version=2 \\\n+\t\tclone \"$BUNDLE_URI_REPO_URI\" cloned2 \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\t# Server advertised bundle-uri capability\n+\tgrep \"< bundle-uri\" log &&\n+\n \t# Client issued bundle-uri command\n \tgrep \"> command=bundle-uri\" log\n '\ndiff --git a/transport.c b/transport.c\nindex b6f279e92cb..b4cf2c0252e 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1516,6 +1516,7 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)\n \n int transport_get_remote_bundle_uri(struct transport *transport)\n {\n+\tint value = 0;\n \tconst struct transport_vtable *vtable = transport->vtable;\n \n \t/* Check config only once. */\n@@ -1523,6 +1524,13 @@ int transport_get_remote_bundle_uri(struct transport *transport)\n \t\treturn 0;\n \ttransport->got_remote_bundle_uri = 1;\n \n+\t/*\n+\t * Don't request bundle-uri from the server unless configured to\n+\t * do so by the transfer.bundleURI=true config option.\n+\t */\n+\tif (git_config_get_bool(\"transfer.bundleuri\", &value) || !value)\n+\t\treturn 0;\n+\n \tif (!vtable->get_bundle_uri)\n \t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n \n-- \ngitgitgadget\n\n"},{"id":"469463","messageId":"c9b7d8779e44f9633c4abfc5588d61c0b87d9b78.1671722058.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v4.git.1671722058.gitgitgadget@gmail.com","subject":"[PATCH v4 07/11] bundle-uri: serve bundle.* keys from config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-22T15:14:13Z","receivedAt":"2022-12-22T15:14:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nImplement the \"bundle-uri\" protocol v2 capability by populating the\nkey=value packet lines from the local Git config. The list of bundles is\nprovided from the keys beginning with \"bundle.\".\n\nIn the future, we may want to filter this list to be more specific to\nthe exact known keys that the server intends to share, but for\nflexibility at the moment we will assume that the config values are\nwell-formed.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 16 +++++++++++++++-\n t/lib-bundle-uri-protocol.sh | 32 ++++++++++++++++++++++++++++++++\n 2 files changed, 47 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 28d8966005e..26ff4b062d7 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -581,6 +581,16 @@ cached:\n \treturn advertise_bundle_uri;\n }\n \n+static int config_to_packet_line(const char *key, const char *value, void *data)\n+{\n+\tstruct packet_reader *writer = data;\n+\n+\tif (!strncmp(key, \"bundle.\", 7))\n+\t\tpacket_write_fmt(writer->fd, \"%s=%s\", key, value);\n+\n+\treturn 0;\n+}\n+\n int bundle_uri_command(struct repository *r,\n \t\t struct packet_reader *request)\n {\n@@ -592,7 +602,11 @@ int bundle_uri_command(struct repository *r,\n \tif (request->status != PACKET_READ_FLUSH)\n \t\tdie(_(\"bundle-uri: expected flush after arguments\"));\n \n-\t/* TODO: Implement the communication */\n+\t/*\n+\t * Read all \"bundle.*\" config lines to the client as key=value\n+\t * packet lines.\n+\t */\n+\tgit_config(config_to_packet_line, &writer);\n \n \tpacket_writer_flush(&writer);\n \ndiff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh\nindex 5620e230387..3022ea4a95b 100644\n--- a/t/lib-bundle-uri-protocol.sh\n+++ b/t/lib-bundle-uri-protocol.sh\n@@ -136,6 +136,8 @@ test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol\n \t[bundle]\n \t\tversion = 1\n \t\tmode = all\n+\t[bundle \"only\"]\n+\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED\n \tEOF\n \n \ttest-tool bundle-uri \\\n@@ -157,6 +159,36 @@ test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol\n \t[bundle]\n \t\tversion = 1\n \t\tmode = all\n+\t[bundle \"only\"]\n+\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED\n+\tEOF\n+\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \\\n+\t\t\"$BUNDLE_URI_REPO_URI\" \\\n+\t\t>actual &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2 with list\" '\n+\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n+\t\tbundle.bundle1.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED-1.bdl\" &&\n+\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n+\t\tbundle.bundle2.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED-2.bdl\" &&\n+\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n+\t\tbundle.bundle3.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED-3.bdl\" &&\n+\n+\t# All data about bundle URIs\n+\tcat >expect <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"bundle1\"]\n+\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED-1.bdl\n+\t[bundle \"bundle2\"]\n+\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED-2.bdl\n+\t[bundle \"bundle3\"]\n+\t\turi = $BUNDLE_URI_BUNDLE_URI_ESCAPED-3.bdl\n \tEOF\n \n \ttest-tool bundle-uri \\\n-- \ngitgitgadget\n\n"},{"id":"469464","messageId":"13e4c82e3380d8b91583550e61671bd8eac69ab1.1671722058.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v4.git.1671722058.gitgitgadget@gmail.com","subject":"[PATCH v4 06/11] bundle-uri client: add helper for testing server","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-22T15:14:12Z","receivedAt":"2022-12-22T15:15:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nAdd a 'test-tool bundle-uri ls-remote' command. This is a thin wrapper\nfor issuing protocol v2 \"bundle-uri\" commands to a server, and to the\nparsing routines in bundle-uri.c.\n\nIn the \"git clone\" case we'll have already done the handshake(),\nbut not here. Add an extra case to check for this handshake in\nget_bundle_uri() for ease of use for future callers.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n t/helper/test-bundle-uri.c | 46 ++++++++++++++++++++++++++++++++++++\n t/lib-bundle-uri-protocol.sh | 46 ++++++++++++++++++++++++++++++++++++\n transport.c | 7 ++++++\n 3 files changed, 99 insertions(+)\n\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex 25afd393428..f8159187014 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -3,6 +3,10 @@\n #include \"bundle-uri.h\"\n #include \"strbuf.h\"\n #include \"string-list.h\"\n+#include \"transport.h\"\n+#include \"ref-filter.h\"\n+#include \"remote.h\"\n+#include \"refs.h\"\n \n enum input_mode {\n \tKEY_VALUE_PAIRS,\n@@ -68,6 +72,46 @@ usage:\n \tusage_with_options(usage, options);\n }\n \n+static int cmd_ls_remote(int argc, const char **argv)\n+{\n+\tconst char *uploadpack = NULL;\n+\tstruct string_list server_options = STRING_LIST_INIT_DUP;\n+\tconst char *dest;\n+\tstruct remote *remote;\n+\tstruct transport *transport;\n+\tint status = 0;\n+\n+\tdest = argc > 1 ? argv[1] : NULL;\n+\n+\tremote = remote_get(dest);\n+\tif (!remote) {\n+\t\tif (dest)\n+\t\t\tdie(_(\"bad repository '%s'\"), dest);\n+\t\tdie(_(\"no remote configured to get bundle URIs from\"));\n+\t}\n+\tif (!remote->url_nr)\n+\t\tdie(_(\"remote '%s' has no configured URL\"), dest);\n+\n+\ttransport = transport_get(remote, NULL);\n+\tif (uploadpack)\n+\t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n+\tif (server_options.nr)\n+\t\ttransport->server_options = &server_options;\n+\n+\tif (transport_get_remote_bundle_uri(transport) < 0) {\n+\t\terror(_(\"could not get the bundle-uri list\"));\n+\t\tstatus = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tprint_bundle_list(stdout, transport->bundles);\n+\n+cleanup:\n+\tif (transport_disconnect(transport))\n+\t\treturn 1;\n+\treturn status;\n+}\n+\n int cmd__bundle_uri(int argc, const char **argv)\n {\n \tconst char *usage[] = {\n@@ -88,6 +132,8 @@ int cmd__bundle_uri(int argc, const char **argv)\n \t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS);\n \tif (!strcmp(argv[1], \"parse-config\"))\n \t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE);\n+\tif (!strcmp(argv[1], \"ls-remote\"))\n+\t\treturn cmd_ls_remote(argc - 1, argv + 1);\n \terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n \n usage:\ndiff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh\nindex 75ea8c4418f..5620e230387 100644\n--- a/t/lib-bundle-uri-protocol.sh\n+++ b/t/lib-bundle-uri-protocol.sh\n@@ -119,3 +119,49 @@ test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque\n \t# Client issued bundle-uri command\n \tgrep \"> command=bundle-uri\" log\n '\n+\n+# The remaining tests will all assume transfer.bundleURI=true\n+#\n+# This test can be removed when transfer.bundleURI is enabled by default.\n+test_expect_success 'enable transfer.bundleURI for remaining tests' '\n+\tgit config --global transfer.bundleURI true\n+'\n+\n+test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2\" '\n+\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n+\t\tbundle.only.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED\" &&\n+\n+\t# All data about bundle URIs\n+\tcat >expect <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \\\n+\t\t\"$BUNDLE_URI_REPO_URI\" \\\n+\t\t>actual &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success \"test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2 and extra data\" '\n+\ttest_config -C \"$BUNDLE_URI_PARENT\" \\\n+\t\tbundle.only.uri \"$BUNDLE_URI_BUNDLE_URI_ESCAPED\" &&\n+\n+\t# Extra data should be ignored\n+\ttest_config -C \"$BUNDLE_URI_PARENT\" bundle.only.extra bogus &&\n+\n+\t# All data about bundle URIs\n+\tcat >expect <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\n+\ttest-tool bundle-uri \\\n+\t\tls-remote \\\n+\t\t\"$BUNDLE_URI_REPO_URI\" \\\n+\t\t>actual &&\n+\ttest_cmp_config_output expect actual\n+'\ndiff --git a/transport.c b/transport.c\nindex 757ad552bf3..0f35114a13e 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -371,6 +371,13 @@ static int get_bundle_uri(struct transport *transport)\n \t\tinit_bundle_list(transport->bundles);\n \t}\n \n+\tif (!data->finished_handshake) {\n+\t\tstruct ref *refs = handshake(transport, 0, NULL, 0);\n+\n+\t\tif (refs)\n+\t\t\tfree_refs(refs);\n+\t}\n+\n \t/*\n \t * \"Support\" protocol v0 and v2 without bundle-uri support by\n \t * silently degrading to a NOOP.\n-- \ngitgitgadget\n\n"},{"id":"469465","messageId":"d13bd6cd95dad95be1f6fd63e48f4923350d75cd.1671722058.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v4.git.1671722058.gitgitgadget@gmail.com","subject":"[PATCH v4 08/11] strbuf: introduce strbuf_strip_file_from_path()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-22T15:14:14Z","receivedAt":"2022-12-22T15:15:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe strbuf_parent_directory() method was added as a static method in\ncontrib/scalar by d0feac4e8c0 (scalar: 'register' sets recommended\nconfig and starts maintenance, 2021-12-03) and then removed in\n65f6a9eb0b9 (scalar: constrain enlistment search, 2022-08-18), but now\nthere is a need for a similar method in the bundle URI feature.\n\nRe-add the method, this time in strbuf.c, but with a new name:\nstrbuf_strip_file_from_path(). The method requirements are slightly\nmodified to allow a trailing slash, in which case nothing is done, which\nmakes the name change valuable.\n\nSigned-off-by: Derrick Stolee \n---\n strbuf.c | 6 ++++++\n strbuf.h | 11 +++++++++++\n 2 files changed, 17 insertions(+)\n\ndiff --git a/strbuf.c b/strbuf.c\nindex 0890b1405c5..c383f41a3c5 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -1200,3 +1200,9 @@ int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n \tfree(path2);\n \treturn res;\n }\n+\n+void strbuf_strip_file_from_path(struct strbuf *sb)\n+{\n+\tchar *path_sep = find_last_dir_sep(sb->buf);\n+\tstrbuf_setlen(sb, path_sep ? path_sep - sb->buf + 1 : 0);\n+}\ndiff --git a/strbuf.h b/strbuf.h\nindex 76965a17d44..f6dbb9681ee 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -664,6 +664,17 @@ int launch_sequence_editor(const char *path, struct strbuf *buffer,\n int strbuf_edit_interactively(struct strbuf *buffer, const char *path,\n \t\t\t const char *const *env);\n \n+/*\n+ * Remove the filename from the provided path string. If the path\n+ * contains a trailing separator, then the path is considered a directory\n+ * and nothing is modified.\n+ *\n+ * Examples:\n+ * - \"/path/to/file\" -> \"/path/to/\"\n+ * - \"/path/to/dir/\" -> \"/path/to/dir/\"\n+ */\n+void strbuf_strip_file_from_path(struct strbuf *sb);\n+\n void strbuf_add_lines(struct strbuf *sb,\n \t\t const char *prefix,\n \t\t const char *buf,\n-- \ngitgitgadget\n\n"},{"id":"469466","messageId":"a188b38399d767f4fc55ff8955c24c249f694e94.1671722058.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v4.git.1671722058.gitgitgadget@gmail.com","subject":"[PATCH v4 09/11] bundle-uri: allow relative URLs in bundle lists","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-22T15:14:15Z","receivedAt":"2022-12-22T15:15:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBundle providers may want to distribute that data across multiple CDNs.\nThis might require a change in the base URI, all the way to the domain\nname. If all bundles require an absolute URI in their 'uri' value, then\nevery push to a CDN would require altering the table of contents to\nmatch the expected domain and exact location within it.\n\nAllow a bundle list to specify a relative URI for the bundles. This URI\nis based on where the client received the bundle list. For a list\nprovided in the 'bundle-uri' protocol v2 command, the Git remote URI is\nthe base URI. Otherwise, the bundle list was provided from an HTTP URI\nnot using the Git protocol, and that URI is the base URI. This allows\neasier distribution of bundle data.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 16 +++++++-\n bundle-uri.h | 14 +++++++\n t/helper/test-bundle-uri.c | 2 +\n t/t5750-bundle-uri-parse.sh | 82 +++++++++++++++++++++++++++++++++++++\n transport.c | 3 ++\n 5 files changed, 116 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 26ff4b062d7..69929d363cc 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -7,6 +7,7 @@\n #include \"hashmap.h\"\n #include \"pkt-line.h\"\n #include \"config.h\"\n+#include \"remote.h\"\n \n static int compare_bundles(const void *hashmap_cmp_fn_data,\n \t\t\t const struct hashmap_entry *he1,\n@@ -49,6 +50,7 @@ void clear_bundle_list(struct bundle_list *list)\n \n \tfor_all_bundles_in_list(list, clear_remote_bundle_info, NULL);\n \thashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent);\n+\tfree(list->baseURI);\n }\n \n int for_all_bundles_in_list(struct bundle_list *list,\n@@ -163,7 +165,7 @@ static int bundle_list_update(const char *key, const char *value,\n \tif (!strcmp(subkey, \"uri\")) {\n \t\tif (bundle->uri)\n \t\t\treturn -1;\n-\t\tbundle->uri = xstrdup(value);\n+\t\tbundle->uri = relative_url(list->baseURI, value, NULL);\n \t\treturn 0;\n \t}\n \n@@ -190,6 +192,18 @@ int bundle_uri_parse_config_format(const char *uri,\n \t\t.error_action = CONFIG_ERROR_ERROR,\n \t};\n \n+\tif (!list->baseURI) {\n+\t\tstruct strbuf baseURI = STRBUF_INIT;\n+\t\tstrbuf_addstr(&baseURI, uri);\n+\n+\t\t/*\n+\t\t * If the URI does not end with a trailing slash, then\n+\t\t * remove the filename portion of the path. This is\n+\t\t * important for relative URIs.\n+\t\t */\n+\t\tstrbuf_strip_file_from_path(&baseURI);\n+\t\tlist->baseURI = strbuf_detach(&baseURI, NULL);\n+\t}\n \tresult = git_config_from_file_with_options(config_to_bundle_list,\n \t\t\t\t\t\t filename, list,\n \t\t\t\t\t\t &opts);\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 357111ecce8..c505444bc75 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -61,6 +61,20 @@ struct bundle_list {\n \tint version;\n \tenum bundle_list_mode mode;\n \tstruct hashmap bundles;\n+\n+\t/**\n+\t * The baseURI of a bundle_list is the URI that provided the list.\n+\t *\n+\t * In the case of the 'bundle-uri' protocol v2 command, the base\n+\t * URI is the URI of the Git remote.\n+\t *\n+\t * Otherwise, the bundle list was downloaded over HTTP from some\n+\t * known URI. 'baseURI' is set to that value.\n+\t *\n+\t * The baseURI is used as the base for any relative URIs\n+\t * advertised by the bundle list at that location.\n+\t */\n+\tchar *baseURI;\n };\n \n void init_bundle_list(struct bundle_list *list);\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex f8159187014..5df5bc3b89e 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -40,6 +40,8 @@ static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mo\n \n \tinit_bundle_list(&list);\n \n+\tlist.baseURI = xstrdup(\"\");\n+\n \tswitch (mode) {\n \tcase KEY_VALUE_PAIRS:\n \t\tif (argc != 1)\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex c2fe3f9c5a5..7b4f930e532 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -30,6 +30,58 @@ test_expect_success 'bundle_uri_parse_line() just URIs' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'bundle_uri_parse_line(): relative URIs' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=bundle.bdl\n+\tbundle.two.uri=../bundle.bdl\n+\tbundle.three.uri=sub/dir/bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = /bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = /sub/dir/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-key-values in >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line(): relative URIs and parent paths' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=bundle.bdl\n+\tbundle.two.uri=../bundle.bdl\n+\tbundle.three.uri=../../bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = /bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = /../bundle.bdl\n+\tEOF\n+\n+\t# TODO: We would prefer if parsing a bundle list would not cause\n+\t# a die() and instead would give a warning and allow the rest of\n+\t# a Git command to continue. This test_must_fail is necessary for\n+\t# now until the interface for relative_url() allows for reporting\n+\t# an error instead of die()ing.\n+\ttest_must_fail test-tool bundle-uri parse-key-values in >actual 2>err &&\n+\tgrep \"fatal: cannot strip one component off url\" err\n+'\n+\n test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' '\n \tcat >in <<-\\EOF &&\n \t=bogus-value\n@@ -136,6 +188,36 @@ test_expect_success 'parse config format: just URIs' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format: relative URIs' '\n+\tcat >in <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = ../bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = sub/dir/bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = /bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = /sub/dir/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config in >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n test_expect_success 'parse config format edge cases: empty key or value' '\n \tcat >in1 <<-\\EOF &&\n \t= bogus-value\ndiff --git a/transport.c b/transport.c\nindex 0f35114a13e..241f8a6ba2d 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1538,6 +1538,9 @@ int transport_get_remote_bundle_uri(struct transport *transport)\n \tif (git_config_get_bool(\"transfer.bundleuri\", &value) || !value)\n \t\treturn 0;\n \n+\tif (!transport->bundles->baseURI)\n+\t\ttransport->bundles->baseURI = xstrdup(transport->url);\n+\n \tif (!vtable->get_bundle_uri)\n \t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n \n-- \ngitgitgadget\n\n"},{"id":"469467","messageId":"72ca6f4254fd29edaba9f201324e22892b4dfb2d.1671722058.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v4.git.1671722058.gitgitgadget@gmail.com","subject":"[PATCH v4 10/11] bundle-uri: download bundles from an advertised list","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-22T15:14:16Z","receivedAt":"2022-12-22T15:15:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe logic in fetch_bundle_uri() is useful for the --bundle-uri option of\n'git clone', but is not helpful when the clone operation discovers a\nlist of URIs from the bundle-uri protocol v2 command. To actually\ndownload and unbundle the advertised bundles, we need a different\nmechanism.\n\nCreate the new fetch_bundle_list() method which is very similar to\nfetch_bundle_uri() except that it relies on download_bundle_list()\ninstead of fetch_bundle_uri_internal(). The download_bundle_list()\nmethod will recursively call fetch_bundle_uri_internal() if any of the\nadvertised URIs serve a bundle list instead of a bundle. This will also\nfollow the bundle.list.mode setting from the input list: \"any\" will\ndownload only one such URI while \"all\" will download data from all of\nthe URIs.\n\nIn an identical way to fetch_bundle_uri(), the bundles are unbundled\nafter all of the bundle lists have been expanded and all necessary URIs.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 21 +++++++++++++++++++++\n bundle-uri.h | 14 ++++++++++++++\n 2 files changed, 35 insertions(+)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 69929d363cc..36268dda172 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -577,6 +577,27 @@ cleanup:\n \treturn result;\n }\n \n+int fetch_bundle_list(struct repository *r, struct bundle_list *list)\n+{\n+\tint result;\n+\tstruct bundle_list global_list;\n+\n+\tinit_bundle_list(&global_list);\n+\n+\t/* If a bundle is added to this global list, then it is required. */\n+\tglobal_list.mode = BUNDLE_MODE_ALL;\n+\n+\tif ((result = download_bundle_list(r, list, &global_list, 0)))\n+\t\tgoto cleanup;\n+\n+\tresult = unbundle_all_bundles(r, &global_list);\n+\n+cleanup:\n+\tfor_all_bundles_in_list(&global_list, unlink_bundle, NULL);\n+\tclear_bundle_list(&global_list);\n+\treturn result;\n+}\n+\n /**\n * API for serve.c.\n */\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex c505444bc75..d5e89f1671c 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -107,6 +107,20 @@ int bundle_uri_parse_config_format(const char *uri,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n+/**\n+ * Given a bundle list that was already advertised (likely by the\n+ * bundle-uri protocol v2 verb) at the given uri, fetch and unbundle the\n+ * bundles according to the bundle strategy of that list.\n+ *\n+ * It is expected that the given 'list' is initialized, including its\n+ * 'baseURI' value.\n+ *\n+ * Returns non-zero if there was an error trying to download the list\n+ * or any of its advertised bundles.\n+ */\n+int fetch_bundle_list(struct repository *r,\n+\t\t struct bundle_list *list);\n+\n /**\n * API for serve.c.\n */\n-- \ngitgitgadget\n\n"},{"id":"469468","messageId":"46ab2b05b1559bf289d28ab3353ea9cecf84875d.1671722058.git.gitgitgadget@gmail.com","threadId":"58731","inReplyTo":"pull.1400.v4.git.1671722058.gitgitgadget@gmail.com","subject":"[PATCH v4 11/11] clone: unbundle the advertised bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-22T15:14:17Z","receivedAt":"2022-12-22T15:15:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA previous change introduced the transport methods to acquire a bundle\nlist from the 'bundle-uri' protocol v2 command, when advertised _and_\nwhen the client has chosen to enable the feature.\n\nTeach Git to download and unbundle the data advertised by those bundles\nduring 'git clone'. This takes place between the ref advertisement and\nthe object data download, and stateful connections will linger while\nthe client downloads bundles. In the future, we should consider closing\nthe remote connection during this process.\n\nAlso, since the --bundle-uri option exists, we do not want to mix the\nadvertised bundles with the user-specified bundles.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 25 ++++++++++++---\n t/lib-bundle-uri-protocol.sh | 21 +++++++++++--\n t/t5601-clone.sh | 59 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 98 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 39364c25b15..527839662b0 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1266,11 +1266,26 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (refs)\n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n \n-\t/*\n-\t * Populate transport->got_remote_bundle_uri and\n-\t * transport->bundle_uri. We might get nothing.\n-\t */\n-\ttransport_get_remote_bundle_uri(transport);\n+\tif (!bundle_uri) {\n+\t\t/*\n+\t\t* Populate transport->got_remote_bundle_uri and\n+\t\t* transport->bundle_uri. We might get nothing.\n+\t\t*/\n+\t\ttransport_get_remote_bundle_uri(transport);\n+\n+\t\tif (transport->bundles &&\n+\t\t hashmap_get_size(&transport->bundles->bundles)) {\n+\t\t\t/* At this point, we need the_repository to match the cloned repo. */\n+\t\t\tif (repo_init(the_repository, git_dir, work_tree))\n+\t\t\t\twarning(_(\"failed to initialize the repo, skipping bundle URI\"));\n+\t\t\telse if (fetch_bundle_list(the_repository,\n+\t\t\t\t\t\t transport->bundles))\n+\t\t\t\twarning(_(\"failed to fetch advertised bundles\"));\n+\t\t} else {\n+\t\t\tclear_bundle_list(transport->bundles);\n+\t\t\tFREE_AND_NULL(transport->bundles);\n+\t\t}\n+\t}\n \n \tif (mapped_refs) {\n \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\ndiff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh\nindex 3022ea4a95b..a4a1af8d029 100644\n--- a/t/lib-bundle-uri-protocol.sh\n+++ b/t/lib-bundle-uri-protocol.sh\n@@ -85,7 +85,7 @@ test_expect_success \"connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav\n '\n \n test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris\" '\n-\ttest_when_finished \"rm -rf log cloned cloned2\" &&\n+\ttest_when_finished \"rm -rf log* cloned*\" &&\n \n \tGIT_TRACE_PACKET=\"$PWD/log\" \\\n \tgit \\\n@@ -117,7 +117,24 @@ test_expect_success \"clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque\n \tgrep \"< bundle-uri\" log &&\n \n \t# Client issued bundle-uri command\n-\tgrep \"> command=bundle-uri\" log\n+\tgrep \"> command=bundle-uri\" log &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log3\" \\\n+\tgit \\\n+\t\t-c transfer.bundleURI=true \\\n+\t\t-c protocol.version=2 \\\n+\t\tclone --bundle-uri=\"$BUNDLE_URI_BUNDLE_URI\" \\\n+\t\t\"$BUNDLE_URI_REPO_URI\" cloned3 \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log3 &&\n+\n+\t# Server advertised bundle-uri capability\n+\tgrep \"< bundle-uri\" log3 &&\n+\n+\t# Client did not issue bundle-uri command (--bundle-uri override)\n+\t! grep \"> command=bundle-uri\" log3\n '\n \n # The remaining tests will all assume transfer.bundleURI=true\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 45f0803ed4d..00d4fae5136 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -795,6 +795,65 @@ test_expect_success 'reject cloning shallow repository using HTTP' '\n \tgit clone --no-reject-shallow $HTTPD_URL/smart/repo.git repo\n '\n \n+test_expect_success 'auto-discover bundle URI from HTTP clone' '\n+\ttest_when_finished rm -rf trace.txt repo2 \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" &&\n+\tgit -C src bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/everything.bundle\" --all &&\n+\tgit clone --bare --no-local src \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" &&\n+\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tuploadpack.advertiseBundleURIs true &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tbundle.version 1 &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tbundle.mode all &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo2.git\" config \\\n+\t\tbundle.everything.uri \"$HTTPD_URL/everything.bundle\" &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n+\t\tgit -c protocol.version=2 \\\n+\t\t -c transfer.bundleURI=true clone \\\n+\t\t$HTTPD_URL/smart/repo2.git repo2 &&\n+\tcat >pattern <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/everything.bundle\"\\]\n+\tEOF\n+\tgrep -f pattern trace.txt\n+'\n+\n+test_expect_success 'auto-discover multiple bundles from HTTP clone' '\n+\ttest_when_finished rm -rf trace.txt repo3 \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" &&\n+\n+\ttest_commit -C src new &&\n+\tgit -C src bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/new.bundle\" HEAD~1..HEAD &&\n+\tgit clone --bare --no-local src \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" &&\n+\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tuploadpack.advertiseBundleURIs true &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.version 1 &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.mode all &&\n+\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.everything.uri \"$HTTPD_URL/everything.bundle\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/repo3.git\" config \\\n+\t\tbundle.new.uri \"$HTTPD_URL/new.bundle\" &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n+\t\tgit -c protocol.version=2 \\\n+\t\t -c transfer.bundleURI=true clone \\\n+\t\t$HTTPD_URL/smart/repo3.git repo3 &&\n+\n+\t# We should fetch _both_ bundles\n+\tcat >pattern <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/everything.bundle\"\\]\n+\tEOF\n+\tgrep -f pattern trace.txt &&\n+\tcat >pattern <<-EOF &&\n+\t\"event\":\"child_start\".*\"argv\":\\[\"git-remote-https\",\"$HTTPD_URL/new.bundle\"\\]\n+\tEOF\n+\tgrep -f pattern trace.txt\n+'\n+\n # DO NOT add non-httpd-specific tests here, because the last part of this\n # test script is only executed when httpd is available and enabled.\n \n-- \ngitgitgadget\n"},{"id":"469522","messageId":"xmqqilhzogv4.fsf@gitster.g","threadId":"58731","inReplyTo":"pull.1400.v4.git.1671722058.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 00/11] Bundle URIs IV: advertise over protocol v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-12-25T11:35:59Z","receivedAt":"2022-12-25T11:38:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> This version includes squashed-in versions of the fixups that were\n> previously known as ds/bundle-uri-4-fixup.\n>\n> * Some unused parameters are now marked with UNUSED, since we are\n> introducing those parameters for the first time. In one case an unused\n> parameter should have been used in repo_config_...() instead of\n> git_config_...().\n> * The GIT_TEST_BUNDLE_URI environment variable is removed in favor of the\n> transfer.bundleURI config option in all cases.\n> * A stale commit message is fixed to no longer refer to a rename that was\n> split into a different commit as part of v3.\n> * The documentation comment for fetch_bundle_list() explicitly defines a\n> non-zero return value as an error.\n\nI guess that sprinkling the fixes from \"4-fixup\" series into the\noriginal commits will resolve many comments by Ævar about the\nlogical ordering of changes in the \"fixup\" series, mostly by making\nthem all moot points? Thanks for working on this, and let's start\nmerging it down to 'next'.\n\n"},{"id":"469682","messageId":"Y68Sc3V5x6xSucZW@coredump.intra.peff.net","threadId":"58731","inReplyTo":"13e4c82e3380d8b91583550e61671bd8eac69ab1.1671722058.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 06/11] bundle-uri client: add helper for testing server","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-12-30T16:31:47Z","receivedAt":"2022-12-30T16:31:57Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 22, 2022 at 03:14:12PM +0000, Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n\n> +static int cmd_ls_remote(int argc, const char **argv)\n> +{\n> +\tconst char *uploadpack = NULL;\n> +\tstruct string_list server_options = STRING_LIST_INIT_DUP;\n\nThese two variables are initialized to NULL and empty respectively, and\nthen not accessed until here:\n\n> +\ttransport = transport_get(remote, NULL);\n> +\tif (uploadpack)\n> +\t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n> +\tif (server_options.nr)\n> +\t\ttransport->server_options = &server_options;\n\nwhere neither conditional will trigger, since they will still be NULL\nand empty.\n\nIs this function missing some argv parsing that would affect these? Or\nif it's not needed, would we want to remove them, like:\n\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex 5df5bc3b89..b18e760310 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -76,8 +76,6 @@ static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mo\n \n static int cmd_ls_remote(int argc, const char **argv)\n {\n-\tconst char *uploadpack = NULL;\n-\tstruct string_list server_options = STRING_LIST_INIT_DUP;\n \tconst char *dest;\n \tstruct remote *remote;\n \tstruct transport *transport;\n@@ -95,11 +93,6 @@ static int cmd_ls_remote(int argc, const char **argv)\n \t\tdie(_(\"remote '%s' has no configured URL\"), dest);\n \n \ttransport = transport_get(remote, NULL);\n-\tif (uploadpack)\n-\t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n-\tif (server_options.nr)\n-\t\ttransport->server_options = &server_options;\n-\n \tif (transport_get_remote_bundle_uri(transport) < 0) {\n \t\terror(_(\"could not get the bundle-uri list\"));\n \t\tstatus = 1;\n\nNot a huge deal, but I noticed that Coverity complained about the\nuploadpack one because this hit 'next' (the server_options one I found\nmanually, but it was kind of obvious when looking at the other).\n\n-Peff\n"},{"id":"469813","messageId":"9bc96719-d6c7-a127-d80e-a42ecaa0803c@github.com","threadId":"58731","inReplyTo":"Y68Sc3V5x6xSucZW@coredump.intra.peff.net","subject":"Re: [PATCH v4 06/11] bundle-uri client: add helper for testing server","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-05T19:09:34Z","receivedAt":"2023-01-05T19:09:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/30/22 11:31 AM, Jeff King wrote:\n> On Thu, Dec 22, 2022 at 03:14:12PM +0000, Ævar Arnfjörð Bjarmason via GitGitGadget wrote:\n> \n>> +static int cmd_ls_remote(int argc, const char **argv)\n>> +{\n>> +\tconst char *uploadpack = NULL;\n>> +\tstruct string_list server_options = STRING_LIST_INIT_DUP;\n> \n> These two variables are initialized to NULL and empty respectively, and\n> then not accessed until here:\n> \n>> +\ttransport = transport_get(remote, NULL);\n>> +\tif (uploadpack)\n>> +\t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n>> +\tif (server_options.nr)\n>> +\t\ttransport->server_options = &server_options;\n> \n> where neither conditional will trigger, since they will still be NULL\n> and empty.\n> \n> Is this function missing some argv parsing that would affect these? Or\n> if it's not needed, would we want to remove them, like:\n\n...\n\n> Not a huge deal, but I noticed that Coverity complained about the\n> uploadpack one because this hit 'next' (the server_options one I found\n> manually, but it was kind of obvious when looking at the other).\n\nYes, removing these lines would be fine. Perhaps there were\nuses for these in an earlier version that were dropped. But\nwe can remove them now and then add them back when they\nactually connect to functionality.\n\nThanks,\n-Stolee\n"},{"id":"469817","messageId":"Y7fgV5eFx78BHdQ4@coredump.intra.peff.net","threadId":"58731","inReplyTo":"9bc96719-d6c7-a127-d80e-a42ecaa0803c@github.com","subject":"[PATCH] test-bundle-uri: drop unused variables","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-06T08:48:23Z","receivedAt":"2023-01-06T08:48:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 05, 2023 at 02:09:34PM -0500, Derrick Stolee wrote:\n\n> > Not a huge deal, but I noticed that Coverity complained about the\n> > uploadpack one because this hit 'next' (the server_options one I found\n> > manually, but it was kind of obvious when looking at the other).\n> \n> Yes, removing these lines would be fine. Perhaps there were\n> uses for these in an earlier version that were dropped. But\n> we can remove them now and then add them back when they\n> actually connect to functionality.\n\nThanks for confirming. Here's a patch that can go on top of\nds/bundle-uri-4 (or just on master, as that topic has now graduated).\n\n-- >8 --\nSubject: [PATCH] test-bundle-uri: drop unused variables\n\nCommit 70b9c10373 (bundle-uri client: add helper for testing server,\n2022-12-22) added a cmd_ls_remote() function which contains \"uploadpack\"\nand \"server_options\" variables. Neither of these variables is ever\nmodified after being initialized, so the code to handle non-NULL and\nnon-empty values is impossible to reach.\n\nWhile in theory we might add command-line parsing to set these, let's\ndrop the dead code for now in the name of cleanliness. It's easy enough\nto add it back later if need be.\n\nNoticed by Coverity.\n\nSigned-off-by: Jeff King \n---\n t/helper/test-bundle-uri.c | 7 -------\n 1 file changed, 7 deletions(-)\n\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex 5df5bc3b89..b18e760310 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -76,8 +76,6 @@ static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mo\n \n static int cmd_ls_remote(int argc, const char **argv)\n {\n-\tconst char *uploadpack = NULL;\n-\tstruct string_list server_options = STRING_LIST_INIT_DUP;\n \tconst char *dest;\n \tstruct remote *remote;\n \tstruct transport *transport;\n@@ -95,11 +93,6 @@ static int cmd_ls_remote(int argc, const char **argv)\n \t\tdie(_(\"remote '%s' has no configured URL\"), dest);\n \n \ttransport = transport_get(remote, NULL);\n-\tif (uploadpack)\n-\t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n-\tif (server_options.nr)\n-\t\ttransport->server_options = &server_options;\n-\n \tif (transport_get_remote_bundle_uri(transport) < 0) {\n \t\terror(_(\"could not get the bundle-uri list\"));\n \t\tstatus = 1;\n-- \n2.39.0.463.g3774f23bc9\n\n"},{"id":"469833","messageId":"12bfe644-1781-fb68-6b1d-edbe60ba9527@github.com","threadId":"58731","inReplyTo":"Y7fgV5eFx78BHdQ4@coredump.intra.peff.net","subject":"Re: [PATCH] test-bundle-uri: drop unused variables","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-06T14:13:16Z","receivedAt":"2023-01-06T14:13:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/6/2023 3:48 AM, Jeff King wrote:\n> On Thu, Jan 05, 2023 at 02:09:34PM -0500, Derrick Stolee wrote:\n> \n>>> Not a huge deal, but I noticed that Coverity complained about the\n>>> uploadpack one because this hit 'next' (the server_options one I found\n>>> manually, but it was kind of obvious when looking at the other).\n>>\n>> Yes, removing these lines would be fine. Perhaps there were\n>> uses for these in an earlier version that were dropped. But\n>> we can remove them now and then add them back when they\n>> actually connect to functionality.\n> \n> Thanks for confirming. Here's a patch that can go on top of\n> ds/bundle-uri-4 (or just on master, as that topic has now graduated).\n> \n> -- >8 --\n> Subject: [PATCH] test-bundle-uri: drop unused variables\n\nThanks for putting together a formal patch. LGTM.\n\n-Stolee\n"}]}