{"thread":{"id":"56220","subject":"[RFC PATCH 00/13] Add bundle-uri: resumably clones, static \"dumb\" CDN etc.","startedAt":"2021-08-05T15:07:39Z","lastAt":"2021-08-24T22:39:42Z","messageCount":26,"participants":["Ævar Arnfjörð Bjarmason","Jonathan Nieder","Derrick Stolee","brian m. carlson"],"isPatch":true,"patchVersion":1,"patchTotal":13},"messages":[{"id":"432104","messageId":"RFC-cover-00.13-0000000000-20210805T150534Z-avarab@gmail.com","threadId":"56220","inReplyTo":null,"subject":"[RFC PATCH 00/13] Add bundle-uri: resumably clones, static \"dumb\" CDN etc.","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-05T15:07:16Z","receivedAt":"2021-08-05T15:07:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"We're in the 2.33.0 rc cycle, and I'd hoped to have some more prep\nwork for this integrated already, but for now here's something\ninteresting I've been working on for early commentary/feedback.\n\nThis adds the the ability to protocol v2 for servers to optimistically\npre-seed supporting clients with one or more bundles via a new\n\"bundle-uri\" protocol extension.\n\nRight now only \"clone\" supports this, but it's a rather easy change on\ntop to add incremental \"fetch\" support as well.\n\nThe elevator pitch for this feature is (I guess it's a long elevator\nride..);\n\n * Allows for offloading most/all of a PACK \"fetch\" to \"dumb\" CDN's\n that *don't* have very close coordination with the server running\n \"git-upload-pack\" (unlike packfile-uri, more on that below).\n\n I.e. distributing an up-to-date-enough bundle via something like\n Debian's FTP mirror system, or a best-effort occasionally updated\n CDN.\n\n Should the bundle(s) be outdated, corrupt or whatever the client\n gracefully recovers by either ignoring the bad data, or catching up\n via negotiation with whatever data it did get.\n\n Server operators should be confident in using bundle URIs, even if\n the CDN they're pointing to is flaky, out of date, or even\n sometimes outright broken or unreachable. The client will recover\n in all those cases.\n\n * Makes performant git infrastructure more accessible, i.e. this\n feature helps the last with an up-to-date repack with up-to-date\n bitmaps when talking to a network-local git server, but a lot of\n users have more option for scaling or distributing things via dumb\n CDNs than a server that can run \"git-upload-pack\".\n\n * You can even bootstrap a clone of a remote server that doesn't\n support bundle-uri with a local or remote bundle with the\n \"transfer.injectBundleURI\" config.\n\n * Trivial path to resumable clones. Note that that's \"resumable\" in\n the sense that curl(1) will resume a partially downloaded bundle,\n we don't resume whatever state index-pack was in when the\n connection was broken.\n\n I have a POC of this working locally, it's just a matter of\n invoking curl(1) with \"--continue-at -\".\n\n The hindrance for resumably clones is now just the UI for git-clone\n (i.e. stashing the partial data somewhere), not protocol\n limitations.\n\nThis goes on top of the outstanding series I have for serve.c API\ncleanup & fixes at\nhttps://lore.kernel.org/git/cover-v4-00.10-00000000000-20210805T011823Z-avarab@gmail.com\n\nI also needed to grab one patch from my \"bundle unbundle progress\"\nseries:\nhttps://lore.kernel.org/git/cover-0.4-0000000000-20210727T004015Z-avarab@gmail.com/\n\nSomething like this approach had been suggested before in late 2011 by\nJeff King, see:\nhttps://lore.kernel.org/git/20111110074330.GA27925@sigill.intra.peff.net/;\nThere's significant differences in the approach, mainly due to\nprotocol v2 not existing at the time. I wrote most of this before\nfinding/seeing Jeff's earlier patches.\n\nFor a demo of how this works head over to 12/13:\nhttps://lore.kernel.org/git/RFC-patch-12.13-8dc5613e87-20210805T150534Z-avarab@gmail.com\n\nIn 13/13 there's a design doc discussing the approach, and major\ndifferences with the existing packfile-uri mechanism:\nhttps://lore.kernel.org/git/RFC-patch-13.13-1e657ed27a-20210805T150534Z-avarab@gmail.com\n\nThis can also be grabbed from the \"avar/bundle-uri-client-clone\"\nbranch of https://github.com/avar/git/\n\nÆvar Arnfjörð Bjarmason (13):\n serve: add command to advertise bundle URIs\n bundle-uri client: add \"bundle-uri\" parsing + tests\n connect.c: refactor sending of agent & object-format\n bundle-uri client: add minimal NOOP client\n bundle-uri client: add \"git ls-remote-bundle-uri\"\n bundle-uri client: add transfer.injectBundleURI support\n bundle-uri client: add boolean transfer.bundleURI setting\n bundle.h: make \"fd\" version of read_bundle_header() public\n fetch-pack: add a deref_without_lazy_fetch_extended()\n fetch-pack: move --keep=* option filling to a function\n index-pack: add --progress-title option\n bundle-uri client: support for bundle-uri with \"clone\"\n bundle-uri docs: add design notes\n\n Documentation/config/transfer.txt | 26 ++\n Documentation/git-index-pack.txt | 6 +\n Documentation/git-ls-remote-bundle-uri.txt | 63 +++\n Documentation/git-ls-remote.txt | 1 +\n Documentation/technical/bundle-uri.txt | 119 ++++++\n Documentation/technical/protocol-v2.txt | 145 +++++++\n Makefile | 3 +\n builtin.h | 1 +\n builtin/clone.c | 7 +\n builtin/index-pack.c | 6 +\n builtin/ls-remote-bundle-uri.c | 90 +++++\n bundle-uri.c | 151 ++++++++\n bundle-uri.h | 30 ++\n bundle.c | 8 +-\n bundle.h | 2 +\n command-list.txt | 1 +\n connect.c | 80 +++-\n fetch-pack.c | 304 ++++++++++++++-\n fetch-pack.h | 6 +\n git.c | 1 +\n remote.h | 4 +\n serve.c | 6 +\n t/helper/test-bundle-uri.c | 80 ++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/lib-t5730-protocol-v2-bundle-uri.sh | 425 +++++++++++++++++++++\n t/t5701-git-serve.sh | 124 +++++-\n t/t5730-protocol-v2-bundle-uri-file.sh | 36 ++\n t/t5731-protocol-v2-bundle-uri-git.sh | 17 +\n t/t5732-protocol-v2-bundle-uri-http.sh | 17 +\n t/t5750-bundle-uri-parse.sh | 98 +++++\n transport-helper.c | 13 +\n transport-internal.h | 7 +\n transport.c | 120 ++++++\n transport.h | 22 ++\n 35 files changed, 1988 insertions(+), 33 deletions(-)\n create mode 100644 Documentation/git-ls-remote-bundle-uri.txt\n create mode 100644 Documentation/technical/bundle-uri.txt\n create mode 100644 builtin/ls-remote-bundle-uri.c\n create mode 100644 bundle-uri.c\n create mode 100644 bundle-uri.h\n create mode 100644 t/helper/test-bundle-uri.c\n create mode 100644 t/lib-t5730-protocol-v2-bundle-uri.sh\n create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh\n create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh\n create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh\n create mode 100755 t/t5750-bundle-uri-parse.sh\n\n-- \n2.33.0.rc0.646.g585563e77f\n\n"},{"id":"432105","messageId":"RFC-patch-01.13-4e1a0dbef5-20210805T150534Z-avarab@gmail.com","threadId":"56220","inReplyTo":"RFC-cover-00.13-0000000000-20210805T150534Z-avarab@gmail.com","subject":"[RFC PATCH 01/13] serve: add command to advertise bundle URIs","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-05T15:07:17Z","receivedAt":"2021-08-05T15:07:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When the uploadpack.bundleURI config is set to a URI (or URIs, if set\n>1 times), advertise a \"bundle-uri\" command, then when the client\nrequests \"bundle-uri\" emit those URIs back at them.\n\nThe client CAN then request those URIs out of bounds, and after\nthey've done (after either disconnecting & coming back, or leaving us\nhanging), proceed with the rest of request flow. I.e. issuing a\n\"ls-refs\" followed by a \"fetch\". The client MAY then send us \"have\"\nlines with the tips they've unpacked from their newly acquired\nbundle(s).\n\nThis commit doesn't implement any of that required client behavior,\nonly the trivial server behavior of spewing a list of URLs at the\nclient on request.\n\nThere is already a uploadpack.blobPackfileUri setting for the server,\nso why is this needed? The Documentation/technical/bundle-uri.txt\nadded in a preceding commit discusses that in more detail, but in\nsummary:\n\n 1. There is no \"real\" support for in git.git. The\n uploadpack.blobPackfileUri setting allows carving out a list of\n blobs (actually any OIDs), but as alluded to in bfc2a36ff2a (Doc:\n clarify contents of packfile sent as URI, 2021-01-20) the only\n \"real\" implementation is JGit based.\n\n 2. The uploadpack.blobPackfileUri is a MUST where this is a\n \"CAN\". I.e. once a client says they support packfile-uri of given\n list of protocols the server will send them a PACK response\n assuming they've downloaded the URI they client was sent, if the\n client doesn't do that they don't have a valid repository.\n\n Pointing at a bundle and having the client send us \"have\"\n lines (or not, maybe they couldn't fetch it, or decided they\n didn't want to) is more flexible, and can gracefully recover\n e.g. if the CDN isn't reachable (maybe you do support \"https\", but\n the CDN provider is down, or blocked your whole country).\n\n 3. Because of the disconnect in #2 \"dumb\" servers can seed\n pre-clients, e.g. we might point to a repo.bundle whose exact\n state we aren't sure of (a cronjob updates it, sometimes). The\n client will discover its contents, and give us the \"have\" lines,\n the \"packfile-uri\" method effectively requires the server to have\n those exact \"have\" lines (or rather, it will produce a similar\n PACK using give-or-take the same exclusions).\n\n 4. This provides an easy way to the long sought after \"resumable\n clones\". I.e. since we can assume that it's in the server's\n interest to keep their bundle(s) as up-to-date as possible, most\n or all of the history we need to fetch will be in the bundle. If\n we fail midway through the \"clone\" we can offload the problem of\n resuming to wget/curl/rsync/whatever, instead of (as has been\n suggested, but not implemented for the \"normal\" dialog)\n \"repairing\" a partial PACK response or something.\n\nThere was a suggestion of implementing a similar feature long ago[1]\nby Jeff King. The main difference between it and this approach is that\nwe've since gained protocol v2, so we can add this as an optional path\nin the dialog between client and server. The 2011 implementation\nhooked into the transport mechanism to try to clone from a bundle\ndirectly. See also [2] and [3] for some later mentions of that\napproach.\n\nSee also [4] for the series that implemented\nuploadpack.blobPackfileUri, and [5] for a series on top that did the\n.gitmodules check in that context. See [6] for the \"ls-refs unborn\"\nfeature which modified code in similar areas of the request flow.\n\n1. https://lore.kernel.org/git/20111110074330.GA27925@sigill.intra.peff.net/\n2. https://lore.kernel.org/git/20190514092900.GA11679@sigill.intra.peff.net/\n3. https://lore.kernel.org/git/YFJWz5yIGng+a16k@coredump.intra.peff.net/\n4. https://lore.kernel.org/git/cover.1591821067.git.jonathantanmy@google.com/\n Merged as 34e849b05a4 (Merge branch 'jt/cdn-offload', 2020-06-25)\n5. https://lore.kernel.org/git/cover.1614021092.git.jonathantanmy@google.com/\n Merged as 6ee353d42f3 (Merge branch 'jt/transfer-fsck-across-packs',\n 2021-03-01)\n6. 69571dfe219 (Merge branch 'jt/clone-unborn-head', 2021-02-17)\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Documentation/technical/protocol-v2.txt | 140 ++++++++++++++++++++++++\n Makefile | 1 +\n bundle-uri.c | 65 +++++++++++\n bundle-uri.h | 14 +++\n serve.c | 6 +\n t/t5701-git-serve.sh | 124 ++++++++++++++++++++-\n 6 files changed, 349 insertions(+), 1 deletion(-)\n create mode 100644 bundle-uri.c\n create mode 100644 bundle-uri.h\n\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nindex 213538f1d0..d10d5e9ef6 100644\n--- a/Documentation/technical/protocol-v2.txt\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -556,3 +556,143 @@ and associated requested information, each separated by a single space.\n \tattr = \"size\"\n \n \tobj-info = obj-id SP obj-size\n+\n+bundle-uri\n+~~~~~~~~~~\n+\n+If the 'bundle-uri' capability is advertised, the server supports the\n+`bundle-uri' command.\n+\n+The capability is currently advertised with no value (i.e. not\n+\"bundle-uri=somevalue\"), a value may be added in the future for\n+supporting command-wide extensions. Clients MUST ignore any unknown\n+capability values and proceed with the 'bundle-uri` dialog they\n+support.\n+\n+The 'bundle-uri' command is intended to be issued before `fetch` to\n+get URIs to bundle files (see linkgit:git-bundle[1]) to \"seed\" and\n+inform the subsequent `fetch` command.\n+\n+The client CAN issue `bundle-uri` before or after any other valid\n+command. It's expected that it'll be issued after an `ls-refs` and\n+before `fetch`.\n+\n+DISCUSSION of bundle-uri\n+^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+The intent of the feature is optimize for server resource consumption\n+in the common case by changing the common case of fetching a very\n+large PACK during linkgit:git-clone[1] into a smaller incremental\n+fetch.\n+\n+It also allows servers to achieve better caching in combination with\n+an `uploadpack.packObjectsHook` (see linkgit:git-config[1]).\n+\n+By having new clones or fetches be a more predictable and common\n+negotiation against the tips of recently produces *.bundle file(s).\n+Servers might even pre-generate the results of such negotiations for\n+the `uploadpack.packObjectsHook` as new pushes come in.\n+\n+I.e. the server would anticipate that fresh clones will download a\n+known bundle, followed by catching up to the current state of the\n+repository using ref tips found in that bundle (or bundles).\n+\n+PROTOCOL for bundle-uri\n+^^^^^^^^^^^^^^^^^^^^^^^\n+\n+A `bundle-uri` request takes no arguments, and as noted above does not\n+currently advertise a capability value. Both may be added in the\n+future.\n+\n+When the client issues a `command=bundle-uri` the response is a list\n+of URIs the server would like the client to fetch out-of-bounds before\n+proceeding with the `fetch` request in this format:\n+\n+\toutput = bundle-uri-line\n+\t\t bundle-uri-line* flush-pkt\n+\n+\tbundle-uri-line = PKT-LINE(bundle-uri)\n+\t\t\t *(SP bundle-feature-key *(=bundle-feature-val))\n+\t\t\t LF\n+\n+\tbundle-uri = A URI such as a https://, ssh:// etc. URI\n+\n+\tbundle-feature-key = Any printable ASCII characters except SP or \"=\"\n+\tbundle-feature-val = Any printable ASCII characters except SP or \"=\"\n+\n+No `bundle-feature-key`=`bundle-feature-value` fields are currently\n+defined. See the discussion of features below.\n+\n+bundle-uri CLIENT AND SERVER EXPECTATIONS\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+The advertised bundles MUST contain one or more reference tips for use\n+by the client. Bundles that are not self-contained MUST use the\n+standard \"-\" prefixes in the bundle format to indicate their\n+prerequisites. I.e. they must be in the standard format \"git bundle\n+create\" would create.\n+\n+If after an `ls-refs` the client finds that the ref tips it wants can\n+be retrieved entirety from advertised bundle(s), it MAY\n+disconnect. The results of such a \"clone\" or \"fetch\" should be\n+indistinguishable from the state attained without using bundle-uri.\n+\n+The client MAY also keep the connection open pending download of the\n+bundle-uris, e.g. should on or more downloads (or their validation)\n+fail.\n+\n+The client MAY provide reference tips found in the bundle(s) to be\n+downloaded out-of-bounds as `have` lines in the `fetch` request. They\n+MAY also ignore the bundle(s) entirely (e.g. if they can't be\n+downloaded) or some combination of the two.\n+\n+For the convenience of clients bundles SHOULD be provided in the order\n+that they must be unpacked in if processed one-at-a-time by a dumber client.\n+\n+That usually means a \"big bundle\" first with most of the history\n+that's self-contained, optionally followed by incremental updates on\n+that \"big bundle\".\n+\n+This ordering is a mere convention and not a MUST, e.g. a repository\n+with N branches with disconnected histories might have N \"big\n+bundles\", each with their own self-contained history. A server might\n+also only provide \"incremental updates\".\n+\n+A client MUST consider the content of the bundles themselves and their\n+header as the ultimate source of truth. Servers MAY be tolerant of\n+simpler clients by using the convention outlined above.\n+\n+As noted before a client MUST gracefully degrade on errors, whether\n+that error is because of bad missing/data in the bundle URI(s), or\n+because that client is too dumb to e.g. understand and fully parse out\n+bundle headers and their prerequisite relationships.\n+\n+bundle-uri PROTOCOL FEATURES\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+As noted above no `bundle-feature-key`=`bundle-feature-value` fields\n+are currently defined.\n+\n+They are intended for future per-URI metadata which older clients MUST\n+ignore and gracefully degrade on. Any fields they do recognize they\n+CAN also ignore.\n+\n+Any backwards-incompatible addition of pre-URI key-value will be\n+guarded by a new value or values in 'bundle-uri' capability\n+advertisement itself, and/or by new future `bundle-uri` request\n+arguments.\n+\n+While no per-URI key-value are currently supported currently they're\n+intended to support future features such as:\n+\n+ * Add a \"hash=\" or \"size=\" advertise the expected hash or\n+ size of the bundle file.\n+\n+ * Advertise that one or more bundle files are the same (to e.g. have\n+ clients round-robin or otherwise choose one of N possible files).\n+\n+ * A \"tip=\" shortcut. A client who'd have the advertised \n+ would know there was no need to download the relevant bundle(s),\n+ they've got that OID already (for multi-tips the client would need\n+ to fetch the bundle, or do e.g. HTTP range requests to get its\n+ header).\ndiff --git a/Makefile b/Makefile\nindex 9573190f1d..877c6c47b6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -850,6 +850,7 @@ LIB_OBJS += blob.o\n LIB_OBJS += bloom.o\n LIB_OBJS += branch.o\n LIB_OBJS += bulk-checkin.o\n+LIB_OBJS += bundle-uri.o\n LIB_OBJS += bundle.o\n LIB_OBJS += cache-tree.o\n LIB_OBJS += cbtree.o\ndiff --git a/bundle-uri.c b/bundle-uri.c\nnew file mode 100644\nindex 0000000000..2d93e8b003\n--- /dev/null\n+++ b/bundle-uri.c\n@@ -0,0 +1,65 @@\n+#include \"cache.h\"\n+#include \"bundle-uri.h\"\n+#include \"pkt-line.h\"\n+#include \"config.h\"\n+\n+/**\n+ * serve.[ch] API.\n+ */\n+\n+/*\n+ * \"uploadpack.bundleURI\" is advertised only if there's URIs to serve\n+ * up per the config.\n+ */\n+static int advertise_bundle_uri = -1;\n+\n+static void send_bundle_uris(struct packet_writer *writer,\n+\t\t\t struct string_list *uris)\n+{\n+\tstruct string_list_item *item;\n+\tfor_each_string_list_item(item, uris) {\n+\t\tconst char *uri = item->string;\n+\n+\t\tpacket_writer_write(writer, \"%s\", uri);\n+\t}\n+}\n+\n+static struct string_list bundle_uris = STRING_LIST_INIT_DUP;\n+\n+static int bundle_uri_startup_config(const char *var, const char *value,\n+\t\t\t\t void *data)\n+{\n+\tif (!strcmp(var, \"uploadpack.bundleuri\")) {\n+\t\tadvertise_bundle_uri = 1;\n+\t\tstring_list_append(&bundle_uris, value);\n+\t}\n+\treturn 0;\n+}\n+\n+int bundle_uri_advertise(struct repository *r, struct strbuf *value)\n+{\n+\tif (advertise_bundle_uri == -1) {\n+\t\tgit_config(bundle_uri_startup_config, NULL);\n+\t\tif (advertise_bundle_uri == -1)\n+\t\t\tadvertise_bundle_uri = 0;\n+\t}\n+\treturn advertise_bundle_uri;\n+}\n+\n+int bundle_uri_command(struct repository *r,\n+\t\t struct packet_reader *request)\n+{\n+\tstruct packet_writer writer;\n+\tpacket_writer_init(&writer, 1);\n+\n+\twhile (packet_reader_read(request) == PACKET_READ_NORMAL)\n+\t\tdie(\"bundle-uri: unexpected argument: '%s'\", request->line);\n+\tif (request->status != PACKET_READ_FLUSH)\n+\t\tdie(\"bundle-uri: expected flush after arguments\");\n+\n+\tsend_bundle_uris(&writer, &bundle_uris);\n+\n+\tpacket_writer_flush(&writer);\n+\n+\treturn 0;\n+}\ndiff --git a/bundle-uri.h b/bundle-uri.h\nnew file mode 100644\nindex 0000000000..6a40efeb39\n--- /dev/null\n+++ b/bundle-uri.h\n@@ -0,0 +1,14 @@\n+#ifndef BUNDLE_URI_H\n+#define BUNDLE_URI_H\n+\n+struct repository;\n+struct packet_reader;\n+struct packet_writer;\n+\n+/**\n+ * serve.[ch] API.\n+ */\n+int bundle_uri_advertise(struct repository *r, struct strbuf *value);\n+int bundle_uri_command(struct repository *r, struct packet_reader *request);\n+\n+#endif /* BUNDLE_URI_H */\ndiff --git a/serve.c b/serve.c\nindex 1817edc7f5..789bf5fc38 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -8,6 +8,7 @@\n #include \"protocol-caps.h\"\n #include \"serve.h\"\n #include \"upload-pack.h\"\n+#include \"bundle-uri.h\"\n \n static int advertise_sid = -1;\n \n@@ -104,6 +105,11 @@ static struct protocol_capability capabilities[] = {\n \t\t.advertise = always_advertise,\n \t\t.command = cap_object_info,\n \t},\n+\t{\n+\t\t.name = \"bundle-uri\",\n+\t\t.advertise = bundle_uri_advertise,\n+\t\t.command = bundle_uri_command,\n+\t},\n };\n \n void protocol_v2_advertise_capabilities(void)\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex 930721f053..21d5314d83 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -12,7 +12,7 @@ test_expect_success 'test capability advertisement' '\n \twrong_algo sha1:sha256\n \twrong_algo sha256:sha1\n \tEOF\n-\tcat >expect <<-EOF &&\n+\tcat >expect.base <<-EOF &&\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n \tls-refs=unborn\n@@ -20,8 +20,11 @@ test_expect_success 'test capability advertisement' '\n \tserver-option\n \tobject-format=$(test_oid algo)\n \tobject-info\n+\tEOF\n+\tcat >expect.trailer <<-EOF &&\n \t0000\n \tEOF\n+\tcat expect.base expect.trailer >expect &&\n \n \tGIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \\\n \t\t--advertise-capabilities >out &&\n@@ -266,4 +269,123 @@ test_expect_success 'basics of object-info' '\n \ttest_cmp expect actual\n '\n \n+# Test the basics of bundle-uri\n+#\n+test_expect_success 'test capability advertisement with uploadpack.bundleURI' '\n+\ttest_config uploadpack.bundleURI FAKE &&\n+\n+\tcat >expect.extra <<-EOF &&\n+\tbundle-uri\n+\tEOF\n+\tcat expect.base \\\n+\t expect.extra \\\n+\t expect.trailer >expect &&\n+\n+\tGIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \\\n+\t\t--advertise-capabilities >out &&\n+\ttest-tool pkt-line unpack actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'basics of bundle-uri: dies if not enabled' '\n+\ttest-tool pkt-line pack >in <<-EOF &&\n+\tcommand=bundle-uri\n+\t0000\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid command '\"'\"'bundle-uri'\"'\"'\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\tERR serve: invalid command '\"'\"'bundle-uri'\"'\"'\n+\tEOF\n+\n+\ttest_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out\n+'\n+\n+\n+test_expect_success 'basics of bundle-uri: enabled with single URI' '\n+\ttest_config uploadpack.bundleURI https://cdn.example.com/repo.bdl &&\n+\n+\ttest-tool pkt-line pack >in <<-EOF &&\n+\tcommand=bundle-uri\n+\tobject-format=$(test_oid algo)\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\thttps://cdn.example.com/repo.bdl\n+\t0000\n+\tEOF\n+\n+\ttest-tool serve-v2 --stateless-rpc out &&\n+\ttest-tool pkt-line unpack actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'basics of bundle-uri: enabled with single URI' '\n+\ttest_config uploadpack.bundleURI https://cdn.example.com/repo.bdl &&\n+\n+\ttest-tool pkt-line pack >in <<-EOF &&\n+\tcommand=bundle-uri\n+\tobject-format=$(test_oid algo)\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\thttps://cdn.example.com/repo.bdl\n+\t0000\n+\tEOF\n+\n+\ttest-tool serve-v2 --stateless-rpc out &&\n+\ttest-tool pkt-line unpack actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'basics of bundle-uri: enabled with two URIs' '\n+\ttest_config uploadpack.bundleURI https://cdn.example.com/repo.bdl &&\n+\ttest_config uploadpack.bundleURI https://cdn.example.com/recent.bdl --add &&\n+\n+\ttest-tool pkt-line pack >in <<-EOF &&\n+\tcommand=bundle-uri\n+\tobject-format=$(test_oid algo)\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\thttps://cdn.example.com/repo.bdl\n+\thttps://cdn.example.com/recent.bdl\n+\t0000\n+\tEOF\n+\n+\ttest-tool serve-v2 --stateless-rpc out &&\n+\ttest-tool pkt-line unpack actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'basics of bundle-uri: unknown future feature(s)' '\n+\ttest_config uploadpack.bundleURI https://cdn.example.com/fake.bdl &&\n+\n+\ttest-tool pkt-line pack >in <<-EOF &&\n+\tcommand=bundle-uri\n+\tobject-format=$(test_oid algo)\n+\t0001\n+\tsome-feature\n+\twe-do-not\n+\tknow=about\n+\t0000\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: bundle-uri: unexpected argument: '\"'\"'some-feature'\"'\"'\n+\tEOF\n+\n+\ttest_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out\n+'\n+\n test_done\n-- \n2.33.0.rc0.646.g585563e77f\n\n"},{"id":"432106","messageId":"RFC-patch-02.13-70c2209c10-20210805T150534Z-avarab@gmail.com","threadId":"56220","inReplyTo":"RFC-cover-00.13-0000000000-20210805T150534Z-avarab@gmail.com","subject":"[RFC PATCH 02/13] bundle-uri client: add \"bundle-uri\" parsing + tests","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-05T15:07:18Z","receivedAt":"2021-08-05T15:07:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a \"test-tool bundle-uri parse\" which parses the format defined in\nthe newly specified \"bundle-uri\" command.\n\nAs note in the \"bundle-uri\" section in protocol-v2.txt we haven't\nspecified any key-values yet, just URI lines, but we need to make sure\nour client doesn't die if this optional data is provided by the\nserver. Let's add a bundle_uri_parse_line() to do that, in subsequent\ncommits the actual client code in {connect,transport}.c will make use\nof it.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 1 +\n bundle-uri.c | 80 ++++++++++++++++++++++++++++++\n bundle-uri.h | 16 ++++++\n t/helper/test-bundle-uri.c | 80 ++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5750-bundle-uri-parse.sh | 98 +++++++++++++++++++++++++++++++++++++\n 7 files changed, 277 insertions(+)\n create mode 100644 t/helper/test-bundle-uri.c\n create mode 100755 t/t5750-bundle-uri-parse.sh\n\ndiff --git a/Makefile b/Makefile\nindex 877c6c47b6..6d3612b962 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -699,6 +699,7 @@ PROGRAMS += $(patsubst %.o,git-%$X,$(PROGRAM_OBJS))\n TEST_BUILTINS_OBJS += test-advise.o\n TEST_BUILTINS_OBJS += test-bitmap.o\n TEST_BUILTINS_OBJS += test-bloom.o\n+TEST_BUILTINS_OBJS += test-bundle-uri.o\n TEST_BUILTINS_OBJS += test-chmtime.o\n TEST_BUILTINS_OBJS += test-config.o\n TEST_BUILTINS_OBJS += test-crontab.o\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 2d93e8b003..d48bb78012 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -63,3 +63,83 @@ int bundle_uri_command(struct repository *r,\n \n \treturn 0;\n }\n+\n+/**\n+ * General API for {transport,connect}.c etc.\n+ */\n+int bundle_uri_parse_line(struct string_list *bundle_uri, const char *line)\n+{\n+\tint i;\n+\tstruct string_list uri = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *item = NULL;\n+\tint err = 0;\n+\n+\t/*\n+\t * Right now we don't understand anything beyond the first SP,\n+\t * but let's be tolerant and ignore any future unknown\n+\t * fields. See the \"MUST\" note about \"bundle-feature-key\" in\n+\t * technical/protocol-v2.txt\n+\t */\n+\tif (string_list_split(&uri, line, ' ', -1) < 1)\n+\t\treturn error(_(\"bundle-uri line not in SP-delimited format: %s\"), line);\n+\n+\tfor (i = 0; i < uri.nr; i++) {\n+\t\tstruct string_list kv = STRING_LIST_INIT_DUP;\n+\t\tstruct string_list_item *kv_item = NULL;\n+\t\tconst char *arg = uri.items[i].string;\n+\t\tint fields;\n+\n+\t\t/*\n+\t\t * The \"string\" for each list item is the parsed URI\n+\t\t * at the start of the line\n+\t\t */\n+\t\tif (i == 0) {\n+\t\t\titem = string_list_append(bundle_uri, arg);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/*\n+\t\t * Anything else on the line is keys or key-value\n+\t\t * pairs separated by \"=\".\n+\t\t *\n+\t\t * Let's parse the format, even if we don't understand\n+\t\t * any of the keys or values yet.\n+\t\t */\n+\t\tassert(item);\n+\t\targ = uri.items[i].string;\n+\t\tif (i == 1) {\n+\t\t\titem->util = xcalloc(1, sizeof(struct string_list));\n+\t\t\tstring_list_init(item->util, 1);\n+\t\t}\n+\n+\t\tfields = string_list_split(&kv, arg, '=', 2);\n+\t\tif (fields < 1 || fields > 2) {\n+\t\t\terr = error(\"expected `k` or `k=v` in column %d of bundle-uri line '%s', got '%s'\",\n+\t\t\t\t i, line, arg);\n+\t\t\tstring_list_clear(&kv, 0);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tkv_item = string_list_append(item->util, kv.items[0].string);\n+\t\tif (kv.nr == 2)\n+\t\t\tkv_item->util = xstrdup(kv.items[1].string);\n+\n+\t\tstring_list_clear(&kv, 0);\n+\t}\n+\tstring_list_clear(&uri, 0);\n+\treturn err;\n+}\n+\n+static void bundle_uri_string_list_clear_cb(void *util, const char *string)\n+{\n+\tstruct string_list *fields = util;\n+\tif (!fields)\n+\t\treturn;\n+\tstring_list_clear(fields, 1);\n+\tfree(fields);\n+}\n+\n+void bundle_uri_string_list_clear(struct string_list *bundle_uri)\n+{\n+\tstring_list_clear_func(bundle_uri, bundle_uri_string_list_clear_cb);\n+}\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 6a40efeb39..2d271801b8 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -4,6 +4,7 @@\n struct repository;\n struct packet_reader;\n struct packet_writer;\n+struct string_list;\n \n /**\n * serve.[ch] API.\n@@ -11,4 +12,19 @@ struct packet_writer;\n int bundle_uri_advertise(struct repository *r, struct strbuf *value);\n int bundle_uri_command(struct repository *r, struct packet_reader *request);\n \n+/**\n+ * General API for {transport,connect}.c etc.\n+ */\n+\n+/**\n+ * bundle_uri_parse_line() returns 0 when a valid bundle-uri has been\n+ * added to `bundle_uri`, <0 on error.\n+ */\n+int bundle_uri_parse_line(struct string_list *bundle_uri, const char *line);\n+\n+/**\n+ * Clear the `bundle_uri` list. Just a very thin wrapper on\n+ * string_list_clear().\n+ */\n+void bundle_uri_string_list_clear(struct string_list *bundle_uri);\n #endif /* BUNDLE_URI_H */\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nnew file mode 100644\nindex 0000000000..014da1b6ab\n--- /dev/null\n+++ b/t/helper/test-bundle-uri.c\n@@ -0,0 +1,80 @@\n+#include \"test-tool.h\"\n+#include \"parse-options.h\"\n+#include \"bundle-uri.h\"\n+#include \"strbuf.h\"\n+#include \"string-list.h\"\n+\n+static int cmd__bundle_uri_parse(int argc, const char **argv)\n+{\n+\tconst char *usage[] = {\n+\t\t\"test-tool bundle-uri parse util;\n+\n+\t\tfprintf(stdout, \"%s\", item->string);\n+\t\tif (!kv) {\n+\t\t\tfprintf(stdout, \"\\n\");\n+\t\t\tcontinue;\n+\t\t}\n+\t\tfor_each_string_list_item(kv_item, kv) {\n+\t\t\tconst char *k = kv_item->string;\n+\t\t\tconst char *v = kv_item->util;\n+\n+\t\t\tif (v)\n+\t\t\t\tfprintf(stdout, \" [kv: %s => %s]\", k, v);\n+\t\t\telse\n+\t\t\t\tfprintf(stdout, \" [attr: %s]\", k);\n+\t\t}\n+\t\tfprintf(stdout, \"\\n\");\n+\t}\n+\tstrbuf_release(&sb);\n+\n+\tbundle_uri_string_list_clear(&list);\n+\n+\treturn err < 0 ? 1 : 0;\n+usage:\n+\tusage_with_options(usage, options);\n+}\n+\n+int cmd__bundle_uri(int argc, const char **argv)\n+{\n+\tconst char *usage[] = {\n+\t\t\"test-tool bundle-uri []\",\n+\t\tNULL\n+\t};\n+\tstruct option options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL, options, usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION |\n+\t\t\t PARSE_OPT_KEEP_ARGV0);\n+\tif (argc == 1)\n+\t\tgoto usage;\n+\n+\tif (!strcmp(argv[1], \"parse\"))\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1);\n+\terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n+\n+usage:\n+\tusage_with_options(usage, options);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 3ce5585e53..b6e1ee7b25 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -17,6 +17,7 @@ static struct test_cmd cmds[] = {\n \t{ \"advise\", cmd__advise_if_enabled },\n \t{ \"bitmap\", cmd__bitmap },\n \t{ \"bloom\", cmd__bloom },\n+\t{ \"bundle-uri\", cmd__bundle_uri },\n \t{ \"chmtime\", cmd__chmtime },\n \t{ \"config\", cmd__config },\n \t{ \"crontab\", cmd__crontab },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 9f0f522850..ef839ac726 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -7,6 +7,7 @@\n int cmd__advise_if_enabled(int argc, const char **argv);\n int cmd__bitmap(int argc, const char **argv);\n int cmd__bloom(int argc, const char **argv);\n+int cmd__bundle_uri(int argc, const char **argv);\n int cmd__chmtime(int argc, const char **argv);\n int cmd__config(int argc, const char **argv);\n int cmd__crontab(int argc, const char **argv);\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nnew file mode 100755\nindex 0000000000..ef1b9fedea\n--- /dev/null\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -0,0 +1,98 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri bundle_uri_parse_line()\"\n+\n+TEST_NO_CREATE_REPO=1\n+. ./test-lib.sh\n+\n+test_expect_success 'bundle_uri_parse_line() just URIs' '\n+\tcat >in <<-\\EOF &&\n+\thttp://example.com/bundle.bdl\n+\thttps://example.com/bundle.bdl\n+\tfile:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\t# For the simple case\n+\tcp in expect &&\n+\n+\ttest-tool bundle-uri parse actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() with attributes' '\n+\tcat >in <<-\\EOF &&\n+\thttp://example.com/bundle1.bdl attr\n+\thttp://example.com/bundle2.bdl ibute\n+\tEOF\n+\n+\n+\tcat >expect <<-\\EOF &&\n+\thttp://example.com/bundle1.bdl [attr: attr]\n+\thttp://example.com/bundle2.bdl [attr: ibute]\n+\tEOF\n+\n+\ttest-tool bundle-uri parse actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() with attributes and key-value attributes' '\n+\tcat >in <<-\\EOF &&\n+\thttp://example.com/bundle1.bdl x a=b y c=d z e=f a=b\n+\tEOF\n+\n+\n+\tcat >expect <<-\\EOF &&\n+\thttp://example.com/bundle1.bdl [attr: x] [kv: a => b] [attr: y] [kv: c => d] [attr: z] [kv: e => f] [kv: a => b]\n+\tEOF\n+\n+\ttest-tool bundle-uri parse actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: extra SP' '\n+\tcat >in <<-\\EOF &&\n+\thttp://example.com/bundle1.bdl one-space\n+\thttp://example.com/bundle1.bdl two-space\n+\thttp://example.com/bundle1.bdl three-space\n+\tEOF\n+\n+\t# We are anal just the one SP\n+\tcat >expect <<-\\EOF &&\n+\thttp://example.com/bundle1.bdl [attr: one-space]\n+\thttp://example.com/bundle1.bdl [attr: ] [attr: two-space]\n+\thttp://example.com/bundle1.bdl [attr: ] [attr: ] [attr: three-space]\n+\tEOF\n+\n+\ttest-tool bundle-uri parse actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: multiple = in key-values' '\n+\tcat >in <<-\\EOF &&\n+\thttp://example.com/bundle1.bdl k=v=extra\n+\thttp://example.com/bundle2.bdl a=b k=v=extra c=d\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\terror: expected `k` or `k=v` in column 1 of bundle-uri line '\"'\"'http://example.com/bundle1.bdl k=v=extra'\"'\"', got '\"'\"'k=v=extra'\"'\"'\n+\terror: bad line: http://example.com/bundle1.bdl k=v=extra\n+\terror: expected `k` or `k=v` in column 2 of bundle-uri line '\"'\"'http://example.com/bundle2.bdl a=b k=v=extra c=d'\"'\"', got '\"'\"'k=v=extra'\"'\"'\n+\terror: bad line: http://example.com/bundle2.bdl a=b k=v=extra c=d\n+\tEOF\n+\n+\t# We fail, but try to continue parsing regardless\n+\tcat >expect <<-\\EOF &&\n+\thttp://example.com/bundle1.bdl\n+\thttp://example.com/bundle2.bdl [kv: a => b] [kv: c => d]\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.33.0.rc0.646.g585563e77f\n\n"},{"id":"432107","messageId":"RFC-patch-04.13-21caf01775-20210805T150534Z-avarab@gmail.com","threadId":"56220","inReplyTo":"RFC-cover-00.13-0000000000-20210805T150534Z-avarab@gmail.com","subject":"[RFC PATCH 04/13] bundle-uri client: add minimal NOOP client","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-05T15:07:20Z","receivedAt":"2021-08-05T15:07:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Set up all the needed client parts of the \"bundle-uri\" protocol\nextension, without actually doing anything with the bundle URIs.\n\nI.e. if the server says it supports \"bundle-uri\" we'll issue a\ncommand=bundle-uri after command=ls-refs when we're cloning. We'll\nparse the returned output using the code already tested for in\nt5750-bundle-uri-parse.sh.\n\nWhat we aren't doing is actually acting on that data, i.e. downloading\nthe bundle(s) before we get to doing the command=fetch, and adjusting\nour negotiation dialog appropriately. I'll do that in subsequent\ncommits.\n\nThere's a question of what level of encapsulation we should use here,\nI've opted to use connect.h in clone.c, but we could also e.g. make\ntransport_get_remote_refs() invoke this, i.e. make it implicitly get\nthe bundle-uri list for later steps.\n\nThis approach means that we don't \"support\" this in \"git fetch\" for\nnow. I'm starting with the case of initial clones, although as noted\nin preceding commits to the protocol documentation nothing about this\napproach precludes getting bundles on incremental fetches.\n\nFor the t5732-protocol-v2-bundle-uri-http.sh it's not easy to set\nenvironment variables for git-upload-pack (it's started by Apache), so\nlet's skip the test under T5730_HTTP, and add unused T5730_{FILE,GIT}\nprerequisites for consistency and future use.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/clone.c | 7 ++\n bundle-uri.c | 10 +-\n connect.c | 47 +++++++++\n remote.h | 4 +\n t/lib-t5730-protocol-v2-bundle-uri.sh | 140 +++++++++++++++++++++++++\n t/t5730-protocol-v2-bundle-uri-file.sh | 36 +++++++\n t/t5731-protocol-v2-bundle-uri-git.sh | 17 +++\n t/t5732-protocol-v2-bundle-uri-http.sh | 17 +++\n transport-helper.c | 13 +++\n transport-internal.h | 7 ++\n transport.c | 48 +++++++++\n transport.h | 18 ++++\n 12 files changed, 362 insertions(+), 2 deletions(-)\n create mode 100644 t/lib-t5730-protocol-v2-bundle-uri.sh\n create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh\n create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh\n create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 66fe66679c..1e4e6be57d 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -27,6 +27,7 @@\n #include \"iterator.h\"\n #include \"sigchain.h\"\n #include \"branch.h\"\n+#include \"connect.h\"\n #include \"remote.h\"\n #include \"run-command.h\"\n #include \"connected.h\"\n@@ -1292,6 +1293,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \n \trefs = transport_get_remote_refs(transport, &transport_ls_refs_options);\n \n+\t/*\n+\t * Populate transport->got_remote_bundle_uri and\n+\t * transport->bundle_uri. We might get nothing.\n+\t */\n+\ttransport_get_remote_bundle_uri(transport);\n+\n \tif (refs) {\n \t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n \ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex d48bb78012..56619bde22 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -38,11 +38,16 @@ static int bundle_uri_startup_config(const char *var, const char *value,\n \n int bundle_uri_advertise(struct repository *r, struct strbuf *value)\n {\n+\tif (value &&\n+\t git_env_bool(\"GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE\", 0))\n+\t\tstrbuf_addstr(value, \"test-unknown-capability-value\");\n+\n \tif (advertise_bundle_uri == -1) {\n \t\tgit_config(bundle_uri_startup_config, NULL);\n \t\tif (advertise_bundle_uri == -1)\n \t\t\tadvertise_bundle_uri = 0;\n \t}\n+\n \treturn advertise_bundle_uri;\n }\n \n@@ -53,9 +58,10 @@ int bundle_uri_command(struct repository *r,\n \tpacket_writer_init(&writer, 1);\n \n \twhile (packet_reader_read(request) == PACKET_READ_NORMAL)\n-\t\tdie(\"bundle-uri: unexpected argument: '%s'\", request->line);\n+\t\tdie(_(\"bundle-uri: unexpected argument: '%s'\"), request->line);\n+\n \tif (request->status != PACKET_READ_FLUSH)\n-\t\tdie(\"bundle-uri: expected flush after arguments\");\n+\t\tdie(_(\"bundle-uri: expected flush after arguments\"));\n \n \tsend_bundle_uris(&writer, &bundle_uris);\n \ndiff --git a/connect.c b/connect.c\nindex 18dfeae2b8..c7601ffd83 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -15,6 +15,7 @@\n #include \"version.h\"\n #include \"protocol.h\"\n #include \"alias.h\"\n+#include \"bundle-uri.h\"\n \n static char *server_capabilities_v1;\n static struct strvec server_capabilities_v2 = STRVEC_INIT;\n@@ -489,6 +490,52 @@ static void send_capabilities(int fd_out, struct packet_reader *reader)\n \t}\n }\n \n+int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\n+\t\t\t struct string_list *bundle_uri, int stateless_rpc)\n+{\n+\tint line_nr = 1;\n+\n+\t/* Assert bundle-uri support */\n+\tserver_supports_v2(\"bundle-uri\", 1);\n+\n+\t/* (Re-)send capabilities */\n+\tsend_capabilities(fd_out, reader);\n+\n+\t/* Send command */\n+\tpacket_write_fmt(fd_out, \"command=bundle-uri\\n\");\n+\tpacket_delim(fd_out);\n+\n+\t/* Send options */\n+\tif (git_env_bool(\"GIT_TEST_PROTOCOL_BAD_BUNDLE_URI\", 0))\n+\t\tpacket_write_fmt(fd_out, \"test-bad-client\\n\");\n+\tpacket_flush(fd_out);\n+\n+\t/* Process response from server */\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\tconst char *line = reader->line;\n+\t\tline_nr++;\n+\n+\t\tif (!bundle_uri_parse_line(bundle_uri, line))\n+\t\t\tcontinue;\n+\n+\t\treturn error(_(\"error on bundle-uri response line %d: %s\"),\n+\t\t\t line_nr, line);\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH)\n+\t\treturn error(_(\"expected flush after bundle-uri listing\"));\n+\n+\t/*\n+\t * Might die(), but obscure enough that that's OK, e.g. in\n+\t * serve.c we'll call BUG() on its equivalent (the\n+\t * PACKET_READ_RESPONSE_END check).\n+\t */\n+\tcheck_stateless_delimiter(stateless_rpc, reader,\n+\t\t\t\t _(\"expected response end packet after ref listing\"));\n+\n+\treturn 0;\n+}\n+\n struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\t struct ref **list, int for_push,\n \t\t\t struct transport_ls_refs_options *transport_options,\ndiff --git a/remote.h b/remote.h\nindex 5a59198252..a0823fe4e9 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -202,6 +202,10 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\t const struct string_list *server_options,\n \t\t\t int stateless_rpc);\n \n+/* Used for protocol v2 in order to retrieve refs from a remote */\n+int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\n+\t\t\t struct string_list *bundle_uri, int stateless_rpc);\n+\n int resolve_remote_symref(struct ref *ref, struct ref *list);\n \n /*\ndiff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh\nnew file mode 100644\nindex 0000000000..6bd9d2dcfb\n--- /dev/null\n+++ b/t/lib-t5730-protocol-v2-bundle-uri.sh\n@@ -0,0 +1,140 @@\n+# Included from t573*-protocol-v2-bundle-uri-*.sh\n+\n+T5730_PARENT=\n+T5730_URI=\n+T5730_BUNDLE_URI=\n+case \"$T5730_PROTOCOL\" in\n+file)\n+\tT5730_PARENT=file_parent\n+\tT5730_URI=\"file://$PWD/file_parent\"\n+\tT5730_BUNDLE_URI=\"$T5730_URI/fake.bdl\"\n+\ttest_set_prereq T5730_FILE\n+\t;;\n+git)\n+\t. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+\tstart_git_daemon --export-all --enable=receive-pack\n+\tT5730_PARENT=\"$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\"\n+\tT5730_URI=\"$GIT_DAEMON_URL/parent\"\n+\tT5730_BUNDLE_URI=\"https://example.com/fake.bdl\"\n+\ttest_set_prereq T5730_GIT\n+\t;;\n+http)\n+\t. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\tstart_httpd\n+\tT5730_PARENT=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\"\n+\tT5730_URI=\"$HTTPD_URL/smart/http_parent\"\n+\ttest_set_prereq T5730_HTTP\n+\t;;\n+*)\n+\tBUG \"Need to pass valid T5730_PROTOCOL (was $T5730_PROTOCOL)\"\n+\t;;\n+esac\n+\n+test_expect_success \"setup protocol v2 $T5730_PROTOCOL:// tests\" '\n+\tgit init \"$T5730_PARENT\" &&\n+\ttest_commit -C \"$T5730_PARENT\" one\n+'\n+\n+# Poor man's URI escaping. Good enough for the test suite whose trash\n+# directory has a space in it. See 93c3fcbe4d4 (git-svn: attempt to\n+# mimic SVN 1.7 URL canonicalization, 2012-07-28) for prior art.\n+test_uri_escape() {\n+\tsed 's/ /%20/g'\n+}\n+\n+case \"$T5730_PROTOCOL\" in\n+http)\n+\ttest_expect_success \"setup config for $T5730_PROTOCOL:// tests\" '\n+\t\tgit -C \"$T5730_PARENT\" config http.receivepack true\n+\t'\n+\t;;\n+*)\n+\t;;\n+esac\n+T5730_BUNDLE_URI_ESCAPED=$(echo \"$T5730_BUNDLE_URI\" | test_uri_escape)\n+\n+test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: no bundle-uri\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tgit \\\n+\t\t-c protocol.version=2 \\\n+\t\tls-remote --symref \"$T5730_URI\" \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\t! grep bundle-uri log\n+'\n+\n+test_expect_success \"connect with $T5730_PROTOCOL:// using protocol v2: have bundle-uri\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tuploadpack.bundleURI \"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tgit \\\n+\t\t-c protocol.version=2 \\\n+\t\tls-remote --symref \"$T5730_URI\" \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\t# Server advertised bundle-uri capability\n+\tgrep bundle-uri log\n+'\n+\n+test_expect_success !T5730_HTTP \"bad client with $T5730_PROTOCOL:// using protocol v2\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_config -C \"$T5730_PARENT\" uploadpack.bundleURI \\\n+\t\t\"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tCloning into '\"'\"'child'\"'\"'...\n+\tEOF\n+\tcase \"$T5730_PROTOCOL\" in\n+\tfile)\n+\t\tcat >fatal-bundle-uri.expect <<-\\EOF\n+\t\tfatal: bundle-uri: unexpected argument: '\"'\"'test-bad-client'\"'\"'\n+\t\tEOF\n+\t\t;;\n+\t*)\n+\t\tcat >fatal.expect <<-\\EOF\n+\t\tfatal: read error: Connection reset by peer\n+\t\tEOF\n+\t\t;;\n+\tesac &&\n+\n+\ttest_when_finished \"rm -rf child\" &&\n+\ttest_must_fail ok=sigpipe env \\\n+\t\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\t\tGIT_TEST_PROTOCOL_BAD_BUNDLE_URI=true \\\n+\t\tgit -c protocol.version=2 \\\n+\t\tclone \"$T5730_URI\" child \\\n+\t\t>out 2>err &&\n+\ttest_must_be_empty out &&\n+\n+\tgrep -v -e ^fatal: -e ^error: err >err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\n+\tcase \"$T5730_PROTOCOL\" in\n+\tfile)\n+\t\t# Due to general race conditions with client/server replies we\n+\t\t# may or may not get \"fatal: the remote end hung up\n+\t\t# expectedly\" here\n+\t\tgrep \"^fatal: bundle-uri:\" err >fatal-bundle-uri.actual &&\n+\t\ttest_cmp fatal-bundle-uri.expect fatal-bundle-uri.actual\n+\t\t;;\n+\t*)\n+\t\tgrep \"^fatal:\" err >fatal.actual &&\n+\t\ttest_cmp fatal.expect fatal.actual\n+\t\t;;\n+\tesac &&\n+\n+\tgrep \"clone> test-bad-client$\" log >sent-bad-request &&\n+\ttest_file_not_empty sent-bad-request\n+'\ndiff --git a/t/t5730-protocol-v2-bundle-uri-file.sh b/t/t5730-protocol-v2-bundle-uri-file.sh\nnew file mode 100755\nindex 0000000000..89203d3a23\n--- /dev/null\n+++ b/t/t5730-protocol-v2-bundle-uri-file.sh\n@@ -0,0 +1,36 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri with protocol v2 and 'file://' transport\"\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'file://' transport\n+#\n+T5730_PROTOCOL=file\n+. \"$TEST_DIRECTORY\"/lib-t5730-protocol-v2-bundle-uri.sh\n+\n+test_expect_success \"unknown capability value with $T5730_PROTOCOL:// using protocol v2\" '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_config -C \"$T5730_PARENT\" \\\n+\t\tuploadpack.bundleURI \"$T5730_BUNDLE_URI_ESCAPED\" &&\n+\n+\tGIT_TRACE_PACKET=\"$PWD/log\" \\\n+\tGIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE=true \\\n+\tgit \\\n+\t\t-c protocol.version=2 \\\n+\t\tls-remote --symref \"$T5730_URI\" \\\n+\t\t>actual 2>err &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"< version 2\" log &&\n+\n+\tgrep \"> bundle-uri=test-unknown-capability-value\" log\n+'\n+\n+test_done\ndiff --git a/t/t5731-protocol-v2-bundle-uri-git.sh b/t/t5731-protocol-v2-bundle-uri-git.sh\nnew file mode 100755\nindex 0000000000..282847b311\n--- /dev/null\n+++ b/t/t5731-protocol-v2-bundle-uri-git.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri with protocol v2 and 'git://' transport\"\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'git://' transport\n+#\n+T5730_PROTOCOL=git\n+. \"$TEST_DIRECTORY\"/lib-t5730-protocol-v2-bundle-uri.sh\n+\n+test_done\ndiff --git a/t/t5732-protocol-v2-bundle-uri-http.sh b/t/t5732-protocol-v2-bundle-uri-http.sh\nnew file mode 100755\nindex 0000000000..fcc1cf3fae\n--- /dev/null\n+++ b/t/t5732-protocol-v2-bundle-uri-http.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri with protocol v2 and 'git://' transport\"\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'git://' transport\n+#\n+T5730_PROTOCOL=http\n+. \"$TEST_DIRECTORY\"/lib-t5730-protocol-v2-bundle-uri.sh\n+\n+test_done\ndiff --git a/transport-helper.c b/transport-helper.c\nindex e8dbdd1153..aff27bcf78 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -1260,9 +1260,22 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n \treturn ret;\n }\n \n+static int get_bundle_uri(struct transport *transport)\n+{\n+\tget_helper(transport);\n+\n+\tif (process_connect(transport, 0)) {\n+\t\tdo_take_over(transport);\n+\t\treturn transport->vtable->get_bundle_uri(transport);\n+\t}\n+\n+\treturn -1;\n+}\n+\n static struct transport_vtable vtable = {\n \t.set_option\t= set_helper_option,\n \t.get_refs_list\t= get_refs_list,\n+\t.get_bundle_uri = get_bundle_uri,\n \t.fetch_refs\t= fetch_refs,\n \t.push_refs\t= push_refs,\n \t.connect\t= connect_helper,\ndiff --git a/transport-internal.h b/transport-internal.h\nindex c4ca0b733a..90ea749e5c 100644\n--- a/transport-internal.h\n+++ b/transport-internal.h\n@@ -26,6 +26,13 @@ struct transport_vtable {\n \tstruct ref *(*get_refs_list)(struct transport *transport, int for_push,\n \t\t\t\t struct transport_ls_refs_options *transport_options);\n \n+\t/**\n+\t * Populates the remote side's bundle-uri under protocol v2,\n+\t * if the \"bundle-uri\" capability was advertised. Returns 0 if\n+\t * OK, negative values on error.\n+\t */\n+\tint (*get_bundle_uri)(struct transport *transport);\n+\n \t/**\n \t * Fetch the objects for the given refs. Note that this gets\n \t * an array, and should ignore the list structure.\ndiff --git a/transport.c b/transport.c\nindex f9400b9b0b..444cf74756 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -22,6 +22,7 @@\n #include \"protocol.h\"\n #include \"object-store.h\"\n #include \"color.h\"\n+#include \"bundle-uri.h\"\n \n static int transport_use_color = -1;\n static char transport_colors[][COLOR_MAXLEN] = {\n@@ -345,6 +346,21 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \treturn handshake(transport, for_push, options, 1);\n }\n \n+static int get_bundle_uri(struct transport *transport)\n+{\n+\tstruct git_transport_data *data = transport->data;\n+\tstruct packet_reader reader;\n+\tint stateless_rpc = transport->stateless_rpc;\n+\tstring_list_init(&transport->bundle_uri, 1);\n+\n+\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\treturn get_remote_bundle_uri(data->fd[1], &reader,\n+\t\t\t\t &transport->bundle_uri, stateless_rpc);\n+}\n+\n static int fetch_refs_via_pack(struct transport *transport,\n \t\t\t int nr_heads, struct ref **to_fetch)\n {\n@@ -884,6 +900,7 @@ static int disconnect_git(struct transport *transport)\n \n static struct transport_vtable taken_over_vtable = {\n \t.get_refs_list\t= get_refs_via_connect,\n+\t.get_bundle_uri = get_bundle_uri,\n \t.fetch_refs\t= fetch_refs_via_pack,\n \t.push_refs\t= git_transport_push,\n \t.disconnect\t= disconnect_git\n@@ -1037,6 +1054,7 @@ static struct transport_vtable bundle_vtable = {\n \n static struct transport_vtable builtin_smart_vtable = {\n \t.get_refs_list\t= get_refs_via_connect,\n+\t.get_bundle_uri = get_bundle_uri,\n \t.fetch_refs\t= fetch_refs_via_pack,\n \t.push_refs\t= git_transport_push,\n \t.connect\t= connect_git,\n@@ -1050,6 +1068,7 @@ struct transport *transport_get(struct remote *remote, const char *url)\n \n \tret->progress = isatty(2);\n \tstring_list_init_dup(&ret->pack_lockfiles);\n+\tstring_list_init_dup(&ret->bundle_uri);\n \n \tif (!remote)\n \t\tBUG(\"No remote provided to transport_get()\");\n@@ -1453,6 +1472,34 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)\n \treturn rc;\n }\n \n+int transport_get_remote_bundle_uri(struct transport *transport)\n+{\n+\tconst struct transport_vtable *vtable = transport->vtable;\n+\n+\t/* Lazily configured */\n+\tif (transport->got_remote_bundle_uri++)\n+\t\treturn 0;\n+\n+\t/*\n+\t * \"Support\" protocol v0 and v2 without bundle-uri support by\n+\t * silently degrading to a NOOP.\n+\t */\n+\tif (!server_supports_v2(\"bundle-uri\", 0))\n+\t\treturn 0;\n+\n+\t/*\n+\t * This is intentionally below the transport.injectBundleURI,\n+\t * we want to be able to inject into protocol v0, or into the\n+\t * dialog of a server who doesn't support this.\n+\t */\n+\tif (!vtable->get_bundle_uri)\n+\t\treturn error(_(\"bundle-uri operation not supported by protocol\"));\n+\n+\tif (vtable->get_bundle_uri(transport) < 0)\n+\t\treturn error(_(\"could not retrieve server-advertised bundle-uri list\"));\n+\treturn 0;\n+}\n+\n void transport_unlock_pack(struct transport *transport)\n {\n \tint i;\n@@ -1478,6 +1525,7 @@ int transport_disconnect(struct transport *transport)\n \t\tret = transport->vtable->disconnect(transport);\n \tif (transport->got_remote_refs)\n \t\tfree_refs((void *)transport->remote_refs);\n+\tbundle_uri_string_list_clear(&transport->bundle_uri);\n \tfree(transport);\n \treturn ret;\n }\ndiff --git a/transport.h b/transport.h\nindex 1cbab11373..fb9f89f956 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -75,6 +75,18 @@ struct transport {\n \t */\n \tunsigned got_remote_refs : 1;\n \n+\t/**\n+\t * Indicates whether we already called get_bundle_uri_list(); set by\n+\t * transport.c::transport_get_remote_bundle_uri().\n+\t */\n+\tunsigned got_remote_bundle_uri : 1;\n+\n+\t/*\n+\t * The results of \"command=bundle-uri\", if both sides support\n+\t * the \"bundle-uri\" capability.\n+\t */\n+\tstruct string_list bundle_uri;\n+\n \t/*\n \t * Transports that call take-over destroys the data specific to\n \t * the transport type while doing so, and cannot be reused.\n@@ -270,6 +282,12 @@ struct transport_ls_refs_options {\n const struct ref *transport_get_remote_refs(struct transport *transport,\n \t\t\t\t\t struct transport_ls_refs_options *transport_options);\n \n+/**\n+ * Retrieve bundle URI(s) from a remote. Populates \"struct\n+ * transport\"'s \"bundle_uri\" and \"got_remote_bundle_uri\".\n+ */\n+int transport_get_remote_bundle_uri(struct transport *transport);\n+\n /*\n * Fetch the hash algorithm used by a remote.\n *\n-- \n2.33.0.rc0.646.g585563e77f\n\n"},{"id":"432108","messageId":"RFC-patch-03.13-fc5c725307-20210805T150534Z-avarab@gmail.com","threadId":"56220","inReplyTo":"RFC-cover-00.13-0000000000-20210805T150534Z-avarab@gmail.com","subject":"[RFC PATCH 03/13] connect.c: refactor sending of agent & object-format","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-05T15:07:19Z","receivedAt":"2021-08-05T15:07:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor the sending of the \"agent\" and \"object-format\" capabilities\ninto a function.\n\nThis was added in its current form in ab67235bc4 (connect: parse v2\nrefs with correct hash algorithm, 2020-05-25). When we connect to a v2\nserver we need to know about its object-format, and it needs to know\nabout ours. Since most things in connect.c and transport.c piggy-back\non the eager getting of remote refs via the handshake() those commands\ncan make use of the just-sent-over object-format by ls-refs.\n\nBut I'm about to add a command that may come after ls-refs, and may\nnot, but we need the server to know about our user-agent and\nobject-format. So let's split this into a function.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n connect.c | 33 ++++++++++++++++++++-------------\n 1 file changed, 20 insertions(+), 13 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 70b13389ba..18dfeae2b8 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -471,6 +471,24 @@ void check_stateless_delimiter(int stateless_rpc,\n \t\tdie(\"%s\", error);\n }\n \n+static void send_capabilities(int fd_out, struct packet_reader *reader)\n+{\n+\tconst char *hash_name;\n+\n+\tif (server_supports_v2(\"agent\", 0))\n+\t\tpacket_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n+\n+\tif (server_feature_v2(\"object-format\", &hash_name)) {\n+\t\tint hash_algo = hash_algo_by_name(hash_name);\n+\t\tif (hash_algo == GIT_HASH_UNKNOWN)\n+\t\t\tdie(_(\"unknown object format '%s' specified by server\"), hash_name);\n+\t\treader->hash_algo = &hash_algos[hash_algo];\n+\t\tpacket_write_fmt(fd_out, \"object-format=%s\", reader->hash_algo->name);\n+\t} else {\n+\t\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n+\t}\n+}\n+\n struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\t struct ref **list, int for_push,\n \t\t\t struct transport_ls_refs_options *transport_options,\n@@ -478,7 +496,6 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\t int stateless_rpc)\n {\n \tint i;\n-\tconst char *hash_name;\n \tstruct strvec *ref_prefixes = transport_options ?\n \t\t&transport_options->ref_prefixes : NULL;\n \tchar **unborn_head_target = transport_options ?\n@@ -488,18 +505,8 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \tif (server_supports_v2(\"ls-refs\", 1))\n \t\tpacket_write_fmt(fd_out, \"command=ls-refs\\n\");\n \n-\tif (server_supports_v2(\"agent\", 0))\n-\t\tpacket_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n-\n-\tif (server_feature_v2(\"object-format\", &hash_name)) {\n-\t\tint hash_algo = hash_algo_by_name(hash_name);\n-\t\tif (hash_algo == GIT_HASH_UNKNOWN)\n-\t\t\tdie(_(\"unknown object format '%s' specified by server\"), hash_name);\n-\t\treader->hash_algo = &hash_algos[hash_algo];\n-\t\tpacket_write_fmt(fd_out, \"object-format=%s\", reader->hash_algo->name);\n-\t} else {\n-\t\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n-\t}\n+\t/* Send capabilities */\n+\tsend_capabilities(fd_out, reader);\n \n \tif (server_options && server_options->nr &&\n \t server_supports_v2(\"server-option\", 1))\n-- \n2.33.0.rc0.646.g585563e77f\n\n"},{"id":"432109","messageId":"RFC-patch-05.13-c4da15d9c0-20210805T150534Z-avarab@gmail.com","threadId":"56220","inReplyTo":"RFC-cover-00.13-0000000000-20210805T150534Z-avarab@gmail.com","subject":"[RFC PATCH 05/13] bundle-uri client: add \"git ls-remote-bundle-uri\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-05T15:07:21Z","receivedAt":"2021-08-05T15:07:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a git-ls-remote-bundle-uri command, this is a thin wrapper for\nissuing protocol v2 \"bundle-uri\" commands to a server, and to the\nparsing routines in bundle-uri.c.\n\nSince in the \"git clone\" case we'll have already done the handshake(),\nbut not here, introduce a \"got_advertisement\" state along with\n\"got_remote_heads\". It seems to me that the \"got_remote_heads\" is\nbadly named in the first place, and the whole logic of eagerly getting\nls-refs on handshake() or not could be refactored somewhat, but let's\nnot do that now, and instead just add another self-documenting state\nvariable.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Documentation/git-ls-remote-bundle-uri.txt | 63 ++++++++++\n Documentation/git-ls-remote.txt | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/clone.c | 2 +-\n builtin/ls-remote-bundle-uri.c | 90 ++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n t/lib-t5730-protocol-v2-bundle-uri.sh | 132 +++++++++++++++++++++\n transport.c | 43 +++++--\n transport.h | 6 +-\n 11 files changed, 330 insertions(+), 11 deletions(-)\n create mode 100644 Documentation/git-ls-remote-bundle-uri.txt\n create mode 100644 builtin/ls-remote-bundle-uri.c\n\ndiff --git a/Documentation/git-ls-remote-bundle-uri.txt b/Documentation/git-ls-remote-bundle-uri.txt\nnew file mode 100644\nindex 0000000000..7d3d16a36f\n--- /dev/null\n+++ b/Documentation/git-ls-remote-bundle-uri.txt\n@@ -0,0 +1,63 @@\n+git-ls-remote-bundle-uri(1)\n+===========================\n+\n+NAME\n+----\n+git-ls-remote-bundle-uri - List 'bundle-uri' in a remote repository\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git ls-remote-bundle-uri [--quiet] [--uri] [--upload-pack=]\n+\t[[-o | --server-option=]