{"thread":{"id":"57905","subject":"[PATCH 00/24] [RFC] Bundle URIs Combined RFC","startedAt":"2022-05-20T18:40:52Z","lastAt":"2022-05-20T18:42:38Z","messageCount":25,"participants":["Derrick Stolee via GitGitGadget","Ævar Arnfjörð Bjarmason via GitGitGadget"],"isPatch":true,"patchVersion":1,"patchTotal":24},"messages":[{"id":"455590","messageId":"pull.1234.git.1653072042.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":null,"subject":"[PATCH 00/24] [RFC] Bundle URIs Combined RFC","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:18Z","receivedAt":"2022-05-20T18:40:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Here is the latest RFC round of Bundle URIs. It combines most of the ideas\nfrom my previous RFC [5] and Ævar's latest RFC [7]. I'll start by talking\nabout the differences from those two approaches before going into the full\ndetail of the RFC (assuming no context of the previous discussion).\n\nThis series is based on ds/bundle-uri [8].\n\n\nComparison to Previous RFC Rounds\n=================================\n\nThe two design documents from Ævar's latest RFC are combined into one and\nare moved up to the first patch.\n\nThe \"table of contents\" concept is renamed to a \"bundle list\". When serving\nfrom an HTTPS GET request, that still takes the form of a config file\nformat, but the exact same key=value pairs are also used in the bundle-uri\nprotocol v2 verb. Both use a common underlying logic for downloading from\nlists that are found in either way. Since I modified Ævar's bundle-uri\nprotocol v2 logic to match this common model, I was able to preserve his\nauthorship on several patches, but some were different enough that I added\nmy co-authorship.\n\nThe bundle list has a new bundle.list.mode setting that can be all (get\neverything) or any (get one). This allows the bundle-uri verb to advertise\nmultiple bundle servers where any one of them would be sufficient to get a\nrecent view of the repository.\n\nThe timestamp heuristic is removed from this series for the sake of brevity.\nThe logic for downloading the bundles can be modified in the future to take\nadvantage of such a heuristic if it is added in the future.\n\nÆvar's use of shelling out to curl is completely removed. This was causing\ntest failures on Windows but also has no way of integrating with the\ncredential manager at the moment, which is critical to serving private\nrepository data via bundles. Similar to the timestamp heuristic, this \"fetch\nthe headers first\" heuristic can be added later.\n\nBecause these heuristics are removed, the incremental bundle fetches during\ngit fetch are also removed. This needs at least one heuristic, so is an\nintended target after the git clone integration is more stable.\n\nInstead of creating git bundle fetch, I implemented the same functionality\ninto git fetch --bundle-uri=X, which is perhaps more understandable to\nusers. This command will allow users to fetch from bundles on their own\ntimeline if they don't want to rely on Git auto-discovering or auto-fetching\nfrom bundles.\n\nBoth the git clone --bundle-uri and the discovery from the bundle-uri\nprotocol v2 system are provided.\n\nThe transport.injectBundleURI config option is removed. Instead, the\n--bundle-uri command-line option is provided.\n\nÆvar's series had some use of logic to feed the bundle ref advertisement\ndirectly into the fetch negotiation. This is removed in favor of storing the\nbundled refs in refs/bundles/ (along with\nlog.excludeDecoration=refs/bundles/) to automatically gain that benefit,\nincluding when using --bundle-uri.\n\nI kept and extended the tests that Ævar provided. There is a new test in the\nend-to-end test that includes the server advertising multiple bundles. The\nseries as a whole is still light on tests, but I wanted to continue to focus\non the viability of the design before committing too much on tests. I've\nmarked commit messages with RFC-TODO comments which will be resolved before\nsending any patches for full review.\n\n\nContext-Free Description\n========================\n\nThere have been several suggestions to improve Git clone speeds and\nreliability by supplementing the Git protocol with static content. The\nPackfile URI [0] feature lets the Git response include URIs that point to\npackfiles that the client must download to complete the request.\n\nLast year, Ævar suggested using bundles instead of packfiles [1] [2]. This\ndesign has the same benefits to the packfile URI feature because it offloads\nmost object downloads to static content fetches. The main advantage over\npackfile URIs is that the remote Git server does not need to know what is in\nthose bundles. The Git client tells the server what it downloaded during the\nfetch negotiation afterwards. This includes any chance that the client did\nnot have access to those bundles or otherwise failed to access them. I\nagreed that this was a much more desirable way to serve static content but\nhad concerns about the flexibility of that design [3]. I started\ninvestigating this idea myself in December, resulting in an RFC in February\n[5]. After some discussion, we agreed [6] that Ævar would combine my ideas\nwith his, including the patches that had not been shared publicly. Ævar's\nlatest RFC [7] includes the two implementations mostly side-by-side, with\nsome early refactoring patches that help both implementations. I submitted\nthose early patches for full review [8], now known as ds/bundle-uri.\n\nNotable to this topic, although not used directly in this RFC, is the notion\nof partial bundles [4] which landed as a feature in Git 2.36.0. This allows\nus to create bundles that respect a partial clone filter. While this RFC\ndoes not implement the awareness of filters when downloading bundles, it\nwould be relatively simple to modify the implementation to include those\nchecks. My previous RFC [5] had that integration, for reference.\n\nThe first patch of this series is a design document detailing how the bundle\nURI feature works in its ideal, feature-complete state (along with ways it\ncan be extended with more features in the future). I focused on maximizing\nflexibility for the service that organizes and serves bundles. This\nincludes:\n\n * Git servers can advertise bundle URIs using the Git protocol.\n\n * Users can set up bundle servers independent of the remote Git server if\n they specify the bundle URI via a --bundle-uri argument.\n\n * Bundle URIs work for full and partial clones.\n\n * Bundle URIs can assist with git fetch in addition to git clone.\n\nThe general breakdown is as follows:\n\n * Patch 1 adds documentation for the feature in its entirety.\n\n * Patches 2-8 add the git fetch --bundle-uri= and git clone\n --bundle-uri= functionality for a single bundle (no bunde lists).\n\n * Patches 9-15 add the bundle list data structures and basic parsing so\n that git clone --bundle-uri= can download bundles based on a bundle\n list.\n\n * Patches 16-24 add the bundle-uri protocol v2 verb and allows git clone to\n discover and download a bundle list advertised by the server.\n\nI consider the patches in their current form to be “RFC quality”. There are\nmultiple places where tests are missing or special cases are not checked.\nThe goal for this RFC is to seek feedback on the high-level ideas before\ncommitting to the deep work of creating mergeable patches.\n\nMy previous RFC [5] included a running server prototype using static HTTPS\ncontent that is generated by a daily job. I updated that prototype to use\nthe new bundle list format, but since this RFC does not include\nunderstanding the timestamp heuristic or the partial clone filters, the RFC\nisn't appropriate for performance testing with that data. The clones will\nstill work, but they will download extra data (both the full and partial\nbundles).\n\n\nIntended Focus of this RFC\n==========================\n\nThis RFC is very large, and that's even with many patches not including full\ndocumentation or tests. These commits are not intended to be reviewed as if\nI intended to merge this as-is.\n\nOne thing this feature establishes is a new standard by which the Git client\nwill communicate with external servers. The goal of this RFC is to determine\nif this standard is well designed, and whether we need to make it more\nrobust. Alternatively, the design might need to be changed for reasons I\ncannot predict.\n\n[0]\nhttps://github.com/git/git/blob/master/Documentation/technical/packfile-uri.txt\nThe packfile URI feature in Git (Created June 2020)\n\n[1]\nhttps://lore.kernel.org/git/RFC-cover-00.13-0000000000-20210805T150534Z-avarab@gmail.com/\nAn earlier RFC for a bundle URI feature. (August 2021)\n\n[2]\nhttps://lore.kernel.org/git/cover-0.3-00000000000-20211025T211159Z-avarab@gmail.com/\nAn earlier patch series creating a 'bundle-uri' protocol v2 capability.\n(October 2021)\n\n[3]\nhttps://lore.kernel.org/git/e7fe220b-2877-107e-8f7e-ea507a65feff@gmail.com/\nMy earlier thoughts on the previous RFCs, many of which are integrated into\nthis RFC (August 2021)\n\n[4]\nhttps://lore.kernel.org/git/pull.1159.git.1645638911.gitgitgadget@gmail.com/\nAdd object filters to bundles (February 2022)\n\n[5]\nhttps://lore.kernel.org/git/pull.1160.git.1645641063.gitgitgadget@gmail.com\n[RFC] Bundle URIs (February 2022)\n\n[6]\nhttps://lore.kernel.org/git/ddebc223-1e13-e758-f9b1-d3f23961e459@github.com/\nSummary of my conversation with Ævar (March 2022)\n\n[7]\nhttps://lore.kernel.org/git/RFC-cover-v2-00.36-00000000000-20220418T165545Z-avarab@gmail.com/\n[RFC PATCH v2 00/36] bundle-uri: a \"dumb CDN\" for git + TOC format (April\n2022)\n\n[8]\nhttps://lore.kernel.org/git/pull.1233.git.1652731865.gitgitgadget@gmail.com/\n[PATCH 0/8] Bundle URIs: Prepatory patches (May 2022)\n\nThanks, -Stolee\n\nDerrick Stolee (18):\n docs: document bundle URI standard\n remote-curl: add 'get' capability\n bundle-uri: create basic file-copy logic\n bundle-uri: add support for http(s):// and file://\n fetch: add --bundle-uri option\n fetch: add 'refs/bundle/' to log.excludeDecoration\n clone: add --bundle-uri option\n clone: --bundle-uri cannot be combined with --depth\n bundle-uri: create bundle_list struct and helpers\n bundle-uri: create base key-value pair parsing\n bundle-uri: limit recursion depth for bundle lists\n bundle-uri: parse bundle list in config format\n bundle-uri: fetch a list of bundles\n bundle-uri: serve URI advertisement from bundle.* config\n bundle-uri: allow relative URLs in bundle lists\n bundle-uri: download bundles from an advertised list\n clone: unbundle the advertised bundles\n t5601: basic bundle URI tests\n\nÆvar Arnfjörð Bjarmason (6):\n bundle-uri: create \"key=value\" line parsing\n bundle-uri: unit test \"key=value\" parsing\n protocol v2: add server-side \"bundle-uri\" skeleton\n bundle-uri client: add minimal NOOP client\n bundle-uri client: add \"git ls-remote-bundle-uri\"\n bundle-uri client: add boolean transfer.bundleURI setting\n\n .gitignore | 1 +\n Documentation/config/transfer.txt | 6 +\n Documentation/fetch-options.txt | 6 +\n Documentation/git-fetch.txt | 1 +\n Documentation/git-ls-remote-bundle-uri.txt | 62 ++\n Documentation/git-ls-remote.txt | 1 +\n Documentation/gitremote-helpers.txt | 6 +\n Documentation/technical/bundle-uri.txt | 477 +++++++++++++++\n Makefile | 3 +\n builtin.h | 1 +\n builtin/clone.c | 45 ++\n builtin/fetch.c | 10 +\n builtin/ls-remote-bundle-uri.c | 71 +++\n bundle-uri.c | 663 +++++++++++++++++++++\n bundle-uri.h | 139 +++++\n command-list.txt | 1 +\n connect.c | 47 ++\n git.c | 1 +\n remote-curl.c | 32 +\n remote.h | 5 +\n serve.c | 6 +\n t/helper/test-bundle-uri.c | 92 +++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/lib-t5730-protocol-v2-bundle-uri.sh | 294 +++++++++\n t/t5601-clone.sh | 59 ++\n t/t5701-git-serve.sh | 40 +-\n t/t5730-protocol-v2-bundle-uri-file.sh | 36 ++\n t/t5731-protocol-v2-bundle-uri-git.sh | 17 +\n t/t5732-protocol-v2-bundle-uri-http.sh | 17 +\n t/t5750-bundle-uri-parse.sh | 195 ++++++\n t/t6021-fetch-bundle.sh | 23 +\n t/test-lib-functions.sh | 11 +\n transport-helper.c | 18 +-\n transport-internal.h | 7 +\n transport.c | 87 +++\n transport.h | 23 +\n 37 files changed, 2503 insertions(+), 2 deletions(-)\n create mode 100644 Documentation/git-ls-remote-bundle-uri.txt\n create mode 100644 Documentation/technical/bundle-uri.txt\n create mode 100644 builtin/ls-remote-bundle-uri.c\n create mode 100644 bundle-uri.c\n create mode 100644 bundle-uri.h\n create mode 100644 t/helper/test-bundle-uri.c\n create mode 100644 t/lib-t5730-protocol-v2-bundle-uri.sh\n create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh\n create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh\n create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh\n create mode 100755 t/t5750-bundle-uri-parse.sh\n create mode 100755 t/t6021-fetch-bundle.sh\n\n\nbase-commit: 89c6e450fe4a919ecb6fa698005a935531c732cf\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1234%2Fderrickstolee%2Fbundle-redo%2Frfc-3-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1234/derrickstolee/bundle-redo/rfc-3-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1234\n-- \ngitgitgadget\n"},{"id":"455591","messageId":"897b50483c6b1db0994df9c5f01ff86f734e7f55.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 01/24] docs: document bundle URI standard","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:19Z","receivedAt":"2022-05-20T18:41:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIntroduce the idea of bundle URIs to the Git codebase through an\naspirational design document. This document includes the full design\nintended to include the feature in its fully-implemented form. This will\ntake several steps as detailed in the Implementation Plan section.\n\nBy committing this document now, it can be used to motivate changes\nnecessary to reach these final goals. The design can still be altered as\nnew information is discovered.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/bundle-uri.txt | 477 +++++++++++++++++++++++++\n 1 file changed, 477 insertions(+)\n create mode 100644 Documentation/technical/bundle-uri.txt\n\ndiff --git a/Documentation/technical/bundle-uri.txt b/Documentation/technical/bundle-uri.txt\nnew file mode 100644\nindex 00000000000..0862336c9eb\n--- /dev/null\n+++ b/Documentation/technical/bundle-uri.txt\n@@ -0,0 +1,477 @@\n+Bundle URIs\n+===========\n+\n+Bundle URIs are locations where Git can download one or more bundles in\n+order to bootstrap the object database in advance of fetching the remaining\n+objects from a remote.\n+\n+One goal is to speed up clones and fetches for users with poor network\n+connectivity to the origin server. Another benefit is to allow heavy users,\n+such as CI build farms, to use local resources for the majority of Git data\n+and thereby reducing the load on the origin server.\n+\n+To enable the bundle URI feature, users can specify a bundle URI using\n+command-line options or the origin server can advertise one or more URIs\n+via a protocol v2 capability.\n+\n+Design Goals\n+------------\n+\n+The bundle URI standard aims to be flexible enough to satisfy multiple\n+workloads. The bundle provider and the Git client have several choices in\n+how they create and consume bundle URIs.\n+\n+* Bundles can have whatever name the server desires. This name could refer\n+ to immutable data by using a hash of the bundle contents. However, this\n+ means that a new URI will be needed after every update of the content.\n+ This might be acceptable if the server is advertising the URI (and the\n+ server is aware of new bundles being generated) but would not be\n+ ergonomic for users using the command line option.\n+\n+* The bundles could be organized specifically for bootstrapping full\n+ clones, but could also be organized with the intention of bootstrapping\n+ incremental fetches. The bundle provider must decide on one of several\n+ organization schemes to minimize client downloads during incremental\n+ fetches, but the Git client can also choose whether to use bundles for\n+ either of these operations.\n+\n+* The bundle provider can choose to support full clones, partial clones,\n+ or both. The client can detect which bundles are appropriate for the\n+ repository's partial clone filter, if any.\n+\n+* The bundle provider can use a single bundle (for clones only), or a\n+ list of bundles. When using a list of bundles, the provider can specify\n+ whether or not the client needs _all_ of the bundle URIs for a full\n+ clone, or if _any_ one of the bundle URIs is sufficient. This allows the\n+ bundle provider to use different URIs for different geographies.\n+\n+* The bundle provider can organize the bundles using heuristics, such as\n+ timestamps or tokens, to help the client prevent downloading bundles it\n+ does not need. When the bundle provider does not provide these\n+ heuristics, the client can use optimizations to minimize how much of the\n+ data is downloaded.\n+\n+* The bundle provider does not need to be associated with the Git server.\n+ The client can choose to use the bundle provider without it being\n+ advertised by the Git server.\n+\n+* The client can choose to discover bundle providers that are advertised\n+ by the Git server. This could happen during `git clone`, during\n+ `git fetch`, both, or neither. The user can choose which combination\n+ works best for them.\n+\n+* The client can choose to configure a bundle provider manually at any\n+ time. The client can also choose to specify a bundle provider manually\n+ as a command-line option to `git clone`.\n+\n+Each repository is different and every Git server has different needs.\n+Hopefully the bundle URI feature is flexible enough to satisfy all needs.\n+If not, then the feature can be extended through its versioning mechanism.\n+\n+Server requirements\n+-------------------\n+\n+To provide a server-side implementation of bundle servers, no other parts\n+of the Git protocol are required. This allows server maintainers to use\n+static content solutions such as CDNs in order to serve the bundle files.\n+\n+At the current scope of the bundle URI feature, all URIs are expected to\n+be HTTP(S) URLs where content is downloaded to a local file using a `GET`\n+request to that URL. The server could include authentication requirements\n+to those requests with the aim of triggering the configured credential\n+helper for secure access. (Future extensions could use \"file://\" URIs or\n+SSH URIs.)\n+\n+Assuming a `200 OK` response from the server, the content at the URL is\n+expected to be of one of two forms:\n+\n+1. Bundle: A Git bundle file of version 2 or higher.\n+\n+2. Table of Contents: A plain-text file that is parsable using Git's\n+ config file parser. This file describes one or more bundles that are\n+ accessible from other URIs.\n+\n+Any other data provided by the server is considered erroneous.\n+\n+Bundle Lists\n+------------\n+\n+The Git server can advertise bundle URIs using a set of `key=value` pairs.\n+A bundle URI can also serve a plain-text file in the Git config format\n+containing these same `key=value` pairs. In both cases, we consider this\n+to be a _bundle list_. The pairs specify information about the bundles\n+that the client can use to make decisions for which bundles to download\n+and which to ignore.\n+\n+A few keys focus on properties of the list itself.\n+\n+bundle.list.version::\n+\t(Required) This value provides a version number for the table of\n+\tcontents. If a future Git change enables a feature that needs the\n+\tGit client to react to a new key in the table of contents file,\n+\tthen this version will increment. The only current version number\n+\tis 1, and if any other value is specified then Git will fail to\n+\tuse this file.\n+\n+bundle.list.mode::\n+\t(Required) This value has one of two values: `all` and `any`. When\n+\t`all` is specified, then the client should expect to need all of\n+\tthe listed bundle URIs that match their repository's requirements.\n+\tWhen `any` is specified, then the client should expect that any\n+\tone of the bundle URIs that match their repository's requirements\n+\twill suffice. Typically, the `any` option is used to list a number\n+\tof different bundle servers located in different geographies.\n+\n+bundle.list.forFetch::\n+\tThis boolean value is a signal to the Git client that\n+\tthe bundle server has designed its bundle organization to assist `git fetch`\n+\tcommands in addition to `git clone` commands. If this is missing,\n+\tGit should not use this table of contents for `git fetch` as it\n+\tmay lead to excess data downloads.\n+\n+The remaining keys include an `` segment which is a server-designated\n+name for each available bundle.\n+\n+bundle..uri::\n+\t(Required) This string value is the URI for downloading bundle\n+\t``. If the URI begins with a protocol (`http://` or\n+\t`https://`) then the URI is absolute. Otherwise, the URI is\n+\tinterpreted as relative to the URI used for the table of contents.\n+\tIf the URI begins with `/`, then that relative path is relative to\n+\tthe domain name used for the table of contents. (This use of\n+\trelative paths is intended to make it easier to distribute a set\n+\tof bundles across a large number of servers or CDNs with different\n+\tdomain names.)\n+\n+bundle..timestamp::\n+\tThis value is the number of seconds since Unix epoch (UTC) that\n+\tthis bundle was created. This is used as an approximation of a\n+\tpoint in time that the bundle matches the data available at the\n+\torigin server.\n+\n+bundle..requires::\n+\tThis string value represents the ID of another bundle. When\n+\tpresent, the server is indicating that this bundle contains a thin\n+\tpackfile. If the client does not have all necessary objects to\n+\tunbundle this packfile, then the client can download the bundle\n+\twith the `requires` ID and try again. (Note: it may be beneficial\n+\tto allow the server to specify multiple `requires` bundles.)\n+\n+bundle..filter::\n+\tThis string value represents an object filter that should also\n+\tappear in the header of this bundle. The server uses this value to\n+\tdifferentiate different kinds of bundles from which the client can\n+\tchoose those that match their object filters.\n+\n+bundle..list::\n+\tThis boolean value indicates whether the client should expect the\n+\tcontent from this URI to be a list (if `true`) or a bundle (if\n+\t`false`). This is typically used when `bundle.list.mode` is `any`.\n+\n+bundle..location::\n+\tThis string value advertises a real-world location from where the\n+\tbundle URI is served. This can be used to present the user with an\n+\toption for which bundle URI to use. This is only valuable when\n+\t`bundle.list.mode` is `any`.\n+\n+Here is an example bundle list using the Git config format:\n+\n+```\n+[bundle \"list\"]\n+\tversion = 1\n+\tmode = all\n+\tforFetch = true\n+\n+[bundle \"2022-02-09-1644442601-daily\"]\n+\turi = https://bundles.fake.com/git/git/2022-02-09-1644442601-daily.bundle\n+\ttimestamp = 1644442601\n+\trequires = 2022-02-02-1643842562\n+\n+[bundle \"2022-02-02-1643842562\"]\n+\turi = https://bundles.fake.com/git/git/2022-02-02-1643842562.bundle\n+\ttimestamp = 1643842562\n+\n+[bundle \"2022-02-09-1644442631-daily-blobless\"]\n+\turi = 2022-02-09-1644442631-daily-blobless.bundle\n+\ttimestamp = 1644442631\n+\trequires = 2022-02-02-1643842568-blobless\n+\tfilter = blob:none\n+\n+[bundle \"2022-02-02-1643842568-blobless\"]\n+\turi = /git/git/2022-02-02-1643842568-blobless.bundle\n+\ttimestamp = 1643842568\n+\tfilter = blob:none\n+```\n+\n+This example uses `bundle.list.mode=all` as well as the\n+`bundle..timestamp` heuristic. It also uses the `bundle..filter`\n+options to present two parallel sets of bundles: one for full clones and\n+another for blobless partial clones.\n+\n+Suppose that this bundle list was found at the URI\n+`https://bundles.fake.com/git/git/` and so the two blobless bundles have\n+the following fully-expanded URIs:\n+\n+* `https://bundles.fake.com/git/git/2022-02-09-1644442631-daily-blobless.bundle`\n+* `https://bundles.fake.com/git/git/2022-02-02-1643842568-blobless.bundle`\n+\n+Advertising Bundle URIs\n+-----------------------\n+\n+If a user knows a bundle URI for the repository they are cloning, then they\n+can specify that URI manually through a command-line option. However, a\n+Git host may want to advertise bundle URIs during the clone operation,\n+helping users unaware of the feature.\n+\n+The only thing required for this feature is that the server can advertise\n+one or more bundle URIs. This advertisement takes the form of a new\n+protocol v2 capability specifically for discovering bundle URIs.\n+\n+The client could choose an arbitrary bundle URI as an option _or_ select\n+the URI with lowest latency by some exploratory checks. It is up to the\n+bundle provider to decide if having multiple URIs is preferable to a\n+single URI that is geodistributed through server-side infrastructure.\n+\n+Cloning with Bundle URIs\n+------------------------\n+\n+The primary need for bundle URIs is to speed up clones. The Git client\n+will interact with bundle URIs according to the following flow:\n+\n+1. The user specifies a bundle URI with the `--bundle-uri` command-line\n+ option _or_ the client discovers a bundle list advertised by the\n+ Git server.\n+\n+2. If the downloaded data from a bundle URI is a bundle, then the client\n+ inspects the bundle headers to check that the negative commit OIDs are\n+ present in the client repository. If some are missing, then the client\n+ delays unbundling until other bundles have been unbundled, making those\n+ OIDs present. When all required OIDs are present, the client unbundles\n+ that data using a refspec. The default refspec is\n+ `+refs/heads/*:refs/bundles/*`, but this can be configured.\n+\n+3. If the file is instead a bundle list, then the client inspects the\n+ `bundle.list.mode` to see if the list is of the `all` or `any` form.\n+\n+ a. If `bundle.list.mode=all`, then the client considers all bundle\n+ URIs. The list is reduced based on the `bundle..filter` options\n+ matching the client repository's partial clone filter. Then, all\n+ bundle URIs are requested. If the `bundle..timestamp` heuristic\n+ is provided, then the bundles are downloaded in reverse-\n+ chronological order, stopping when a bundle has all required OIDs.\n+ The bundles can then be unbundled in chronological order. The client\n+ stores the latest timestamp as a heuristic for avoiding future\n+ downloads if the bundle list does not advertise newer bundles.\n+\n+ b. If `bundle.list.mode=any`, then the client can choose any one of the\n+ bundle URIs to inspect. The client can use a variety of ways to\n+ choose among these URIs. The client can also fallback to another URI\n+ if the initial choice fails to return a result.\n+\n+Note that during a clone we expect that all bundles will be required, and\n+heuristics such as `bundle..timestamp` can be used to download bundles\n+in chronological order or in parallel.\n+\n+If a given bundle URI is a bundle list with `bundle.list.forFetch=true`,\n+then the client can choose to store that URI as its chosen bundle URI. The\n+client can then navigate directly to that URI during later `git fetch`\n+calls.\n+\n+When downloading bundle URIs, the client can choose to inspect the initial\n+content before committing to downloading the entire content. This may\n+provide enough information to determine if the URI is a bundle list or\n+a bundle. In the case of a bundle, the client may inspect the bundle\n+header to determine that all advertised tips are already in the client\n+repository and cancel the remaining download.\n+\n+Fetching with Bundle URIs\n+-------------------------\n+\n+When the client fetches new data, it can decide to fetch from bundle\n+servers before fetching from the origin remote. This could be done via\n+a command-line option, but it is more likely useful to use a config value\n+such as the one specified during the clone.\n+\n+The fetch operation follows the same procedure to download bundles from a\n+bundle list (although we do _not_ want to use parallel downloads here). We\n+expect that the process will end when all negative commit OIDs in a thin\n+bundle are already in the object database.\n+\n+A further optimization is that the client can avoid downloading any\n+bundles if their timestamps are not larger than the stored timestamp.\n+After fetching new bundles, this local timestamp value is updated.\n+\n+If the bundle provider does not provide the `bundle..timestamp`\n+heuristic, then the client should attempt to inspect the bundle headers\n+before downloading the full bundle data in case the bundle tips already\n+exist in the client repository.\n+\n+Error Conditions\n+----------------\n+\n+If the Git client discovers something unexpected while downloading\n+information according to a bundle URI or the table of contents found at\n+that location, then Git can ignore that data and continue as if it was not\n+given a bundle URI. The remote Git server is the ultimate source of truth,\n+not the bundle URI.\n+\n+Here are a few example error conditions:\n+\n+* The client fails to connect with a server at the given URI or a connection\n+ is lost without any chance to recover.\n+\n+* The client receives a response other than `200 OK` (such as `404 Not Found`,\n+ `401 Not Authorized`, or `500 Internal Server Error`). The client should\n+ use the `credential.helper` to attempt authentication after the first\n+ `401 Not Authorized` response, but a second such response is a failure.\n+\n+* The client receives data that is not parsable as a bundle or table of\n+ contents.\n+\n+* The bundle list describes a directed cycle in the\n+ `bundle..requires` links.\n+\n+* A bundle includes a filter that does not match expectations.\n+\n+* The client cannot unbundle the bundles because the negative commit OIDs\n+ are not in the object database and there are no more\n+ `bundle..requires` links to follow.\n+\n+There are also situations that could be seen as wasteful, but are not\n+error conditions:\n+\n+* The downloaded bundles contain more information than is requested by\n+ the clone or fetch request. A primary example is if the user requests\n+ a clone with `--single-branch` but downloads bundles that store every\n+ reachable commit from all `refs/heads/*` references. This might be\n+ initially wasteful, but perhaps these objects will become reachable by\n+ a later ref update that the client cares about.\n+\n+* A bundle download during a `git fetch` contains objects already in the\n+ object database. This is probably unavoidable if we are using bundles\n+ for fetches, since the client will almost always be slightly ahead of\n+ the bundle servers after performing its \"catch-up\" fetch to the remote\n+ server. This extra work is most wasteful when the client is fetching\n+ much more frequently than the server is computing bundles, such as if\n+ the client is using hourly prefetches with background maintenance, but\n+ the server is computing bundles weekly. For this reason, the client\n+ should not use bundle URIs for fetch unless the server has explicitly\n+ recommended it through the `bundle.list.forFetch = true` value.\n+\n+Implementation Plan\n+-------------------\n+\n+This design document is being submitted on its own as an aspirational\n+document, with the goal of implementing all of the mentioned client\n+features over the course of several patch series. Here is a potential\n+outline for submitting these features:\n+\n+1. Integrate bundle URIs into `git clone` with a `--bundle-uri` option.\n+ This will include a new `git fetch --bundle-uri` mode for use as the\n+ implementation underneath `git clone`. The initial version here will\n+ expect a single bundle at the given URI.\n+\n+2. Implement the ability to parse a bundle list from a bundle URI and\n+ update the `git fetch --bundle-uri` logic to properly distinguish\n+ between `bundle.list.mode` options. Specifically design the feature so\n+ that the config format parsing feeds a list of key-value pairs into the\n+ bundle list logic.\n+\n+3. Create the `bundle-uri` protocol v2 verb so Git servers can advertise\n+ bundle URIs using the key-value pairs. Plug into the existing key-value\n+ input to the bundle list logic. Allow `git clone` to discover these\n+ bundle URIs and bootstrap the client repository from the bundle data.\n+ (This choice is an opt-in via a config option and a command-line\n+ option.)\n+\n+4. Allow the client to understand the `bundle.list.forFetch` configuration\n+ and the `bundle..timestamp` heuristic. When `git clone` discovers a\n+ bundle URI with `bundle.list.forFetch=true`, it configures the client\n+ repository to check that bundle URI during later `git fetch `\n+ commands.\n+\n+5. Allow clients to discover bundle URIs during `git fetch` and configure\n+ a bundle URI for later fetches if `bundle.list.forFetch=true`.\n+\n+6. Implement the \"inspect headers\" heuristic to reduce data downloads when\n+ the `bundle..timestamp` heuristic is not available.\n+\n+As these features are reviewed, this plan might be updated. We also expect\n+that new designs will be discovered and implemented as this feature\n+matures and becomes used in real-world scenarios.\n+\n+Related Work: Packfile URIs\n+---------------------------\n+\n+The Git protocol already has a capability where the Git server can list\n+a set of URLs along with the packfile response when serving a client\n+request. The client is then expected to download the packfiles at those\n+locations in order to have a complete understanding of the response.\n+\n+This mechanism is used by the Gerrit server (implemented with JGit) and\n+has been effective at reducing CPU load and improving user performance for\n+clones.\n+\n+A major downside to this mechanism is that the origin server needs to know\n+_exactly_ what is in those packfiles, and the packfiles need to be available\n+to the user for some time after the server has responded. This coupling\n+between the origin and the packfile data is difficult to manage.\n+\n+Further, this implementation is extremely hard to make work with fetches.\n+\n+Related Work: GVFS Cache Servers\n+--------------------------------\n+\n+The GVFS Protocol [2] is a set of HTTP endpoints designed independently of\n+the Git project before Git's partial clone was created. One feature of this\n+protocol is the idea of a \"cache server\" which can be colocated with build\n+machines or developer offices to transfer Git data without overloading the\n+central server.\n+\n+The endpoint that VFS for Git is famous for is the `GET /gvfs/objects/{oid}`\n+endpoint, which allows downloading an object on-demand. This is a critical\n+piece of the filesystem virtualization of that product.\n+\n+However, a more subtle need is the `GET /gvfs/prefetch?lastPackTimestamp=`\n+endpoint. Given an optional timestamp, the cache server responds with a list\n+of precomputed packfiles containing the commits and trees that were introduced\n+in those time intervals.\n+\n+The cache server computes these \"prefetch\" packfiles using the following\n+strategy:\n+\n+1. Every hour, an \"hourly\" pack is generated with a given timestamp.\n+2. Nightly, the previous 24 hourly packs are rolled up into a \"daily\" pack.\n+3. Nightly, all prefetch packs more than 30 days old are rolled up into\n+ one pack.\n+\n+When a user runs `gvfs clone` or `scalar clone` against a repo with cache\n+servers, the client requests all prefetch packfiles, which is at most\n+`24 + 30 + 1` packfiles downloading only commits and trees. The client\n+then follows with a request to the origin server for the references, and\n+attempts to checkout that tip reference. (There is an extra endpoint that\n+helps get all reachable trees from a given commit, in case that commit\n+was not already in a prefetch packfile.)\n+\n+During a `git fetch`, a hook requests the prefetch endpoint using the\n+most-recent timestamp from a previously-downloaded prefetch packfile.\n+Only the list of packfiles with later timestamps are downloaded. Most\n+users fetch hourly, so they get at most one hourly prefetch pack. Users\n+whose machines have been off or otherwise have not fetched in over 30 days\n+might redownload all prefetch packfiles. This is rare.\n+\n+It is important to note that the clients always contact the origin server\n+for the refs advertisement, so the refs are frequently \"ahead\" of the\n+prefetched pack data. The missing objects are downloaded on-demand using\n+the `GET gvfs/objects/{oid}` requests, when needed by a command such as\n+`git checkout` or `git log`. Some Git optimizations disable checks that\n+would cause these on-demand downloads to be too aggressive.\n+\n+See Also\n+--------\n+\n+[1] https://lore.kernel.org/git/RFC-cover-00.13-0000000000-20210805T150534Z-avarab@gmail.com/\n+ An earlier RFC for a bundle URI feature.\n+\n+[2] https://github.com/microsoft/VFSForGit/blob/master/Protocol.md\n+ The GVFS Protocol\n-- \ngitgitgadget\n\n"},{"id":"455592","messageId":"682326226783d25744ca72fefb61ec0362f7f141.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 02/24] remote-curl: add 'get' capability","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:20Z","receivedAt":"2022-05-20T18:41:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA future change will want a way to download a file over HTTP(S) using\nthe simplest of download mechanisms. We do not want to assume that the\nserver on the other side understands anything about the Git protocol but\ncould be a simple static web server.\n\nCreate the new 'get' capability for the remote helpers which advertises\nthat the 'get' command is avalable. A caller can send a line containing\n'get ' to download the file at into the file at\n.\n\nRFC-TODO: This change requires tests directly on the remote helper.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/gitremote-helpers.txt | 6 ++++++\n remote-curl.c | 32 +++++++++++++++++++++++++++++\n t/t6021-fetch-bundle.sh | 23 +++++++++++++++++++++\n transport-helper.c | 5 ++++-\n 4 files changed, 65 insertions(+), 1 deletion(-)\n create mode 100755 t/t6021-fetch-bundle.sh\n\ndiff --git a/Documentation/gitremote-helpers.txt b/Documentation/gitremote-helpers.txt\nindex 6f1e269ae43..f82588601a9 100644\n--- a/Documentation/gitremote-helpers.txt\n+++ b/Documentation/gitremote-helpers.txt\n@@ -168,6 +168,9 @@ Supported commands: 'list', 'import'.\n \tCan guarantee that when a clone is requested, the received\n \tpack is self contained and is connected.\n \n+'get'::\n+\tCan use the 'get' command to download a file from a given URI.\n+\n If a helper advertises 'connect', Git will use it if possible and\n fall back to another capability if the helper requests so when\n connecting (see the 'connect' command under COMMANDS).\n@@ -418,6 +421,9 @@ Supported if the helper has the \"connect\" capability.\n +\n Supported if the helper has the \"stateless-connect\" capability.\n \n+'get' ::\n+\tDownloads the file from the given `` to the given ``.\n+\n If a fatal error occurs, the program writes the error message to\n stderr and exits. The caller should expect that a suitable error\n message has been printed if the child closes the connection without\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 67f178b1120..53750d88e76 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -1276,6 +1276,33 @@ static void parse_fetch(struct strbuf *buf)\n \tstrbuf_reset(buf);\n }\n \n+static void parse_get(struct strbuf *buf)\n+{\n+\tstruct http_get_options opts = { 0 };\n+\tstruct strbuf url = STRBUF_INIT;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tconst char *p, *space;\n+\n+\tif (!skip_prefix(buf->buf, \"get \", &p))\n+\t\tdie(_(\"http transport does not support %s\"), buf->buf);\n+\n+\tspace = strchr(p, ' ');\n+\n+\tif (!space)\n+\t\tdie(_(\"protocol error: expected ' ', missing space\"));\n+\n+\tstrbuf_add(&url, p, space - p);\n+\tstrbuf_addstr(&path, space + 1);\n+\n+\thttp_get_file(url.buf, path.buf, &opts);\n+\n+\tstrbuf_release(&url);\n+\tstrbuf_release(&path);\n+\tprintf(\"\\n\");\n+\tfflush(stdout);\n+\tstrbuf_reset(buf);\n+}\n+\n static int push_dav(int nr_spec, const char **specs)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n@@ -1549,9 +1576,14 @@ int cmd_main(int argc, const char **argv)\n \t\t\t\tprintf(\"unsupported\\n\");\n \t\t\tfflush(stdout);\n \n+\t\t} else if (skip_prefix(buf.buf, \"get \", &arg)) {\n+\t\t\tparse_get(&buf);\n+\t\t\tfflush(stdout);\n+\n \t\t} else if (!strcmp(buf.buf, \"capabilities\")) {\n \t\t\tprintf(\"stateless-connect\\n\");\n \t\t\tprintf(\"fetch\\n\");\n+\t\t\tprintf(\"get\\n\");\n \t\t\tprintf(\"option\\n\");\n \t\t\tprintf(\"push\\n\");\n \t\t\tprintf(\"check-connectivity\\n\");\ndiff --git a/t/t6021-fetch-bundle.sh b/t/t6021-fetch-bundle.sh\nnew file mode 100755\nindex 00000000000..8ae16009ba0\n--- /dev/null\n+++ b/t/t6021-fetch-bundle.sh\n@@ -0,0 +1,23 @@\n+#!/bin/sh\n+\n+test_description='test fetching files and bundles over HTTP'\n+\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'set up source repo' '\n+\tgit init src &&\n+\ttest_commit -C src one\n+'\n+\n+test_expect_success 'fail to get a file over HTTP' '\n+\tcat >input <<-EOF &&\n+\tget $HTTPD_URL/does-not-exist.txt download-failed.txt\n+\n+\tEOF\n+\tgit remote-https \"$HTTPD_URL\" err &&\n+\ttest_path_is_missing download-failed.txt\n+'\n+\n+test_done\ndiff --git a/transport-helper.c b/transport-helper.c\nindex b4dbbabb0c2..dfbeaebe40c 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -33,7 +33,8 @@ struct helper_data {\n \t\tcheck_connectivity : 1,\n \t\tno_disconnect_req : 1,\n \t\tno_private_update : 1,\n-\t\tobject_format : 1;\n+\t\tobject_format : 1,\n+\t\tget : 1;\n \n \t/*\n \t * As an optimization, the transport code may invoke fetch before\n@@ -210,6 +211,8 @@ static struct child_process *get_helper(struct transport *transport)\n \t\t\tdata->no_private_update = 1;\n \t\t} else if (starts_with(capname, \"object-format\")) {\n \t\t\tdata->object_format = 1;\n+\t\t} else if (!strcmp(capname, \"get\")) {\n+\t\t\tdata->get = 1;\n \t\t} else if (mandatory) {\n \t\t\tdie(_(\"unknown mandatory capability %s; this remote \"\n \t\t\t \"helper probably needs newer version of Git\"),\n-- \ngitgitgadget\n\n"},{"id":"455593","messageId":"701a983eed2e695e91b7a9abc50c64b25415d8a6.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 06/24] fetch: add 'refs/bundle/' to log.excludeDecoration","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:24Z","receivedAt":"2022-05-20T18:41:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen fetching from a bundle URI, the branches of that bundle are stored\nin a different ref namespace: refs/bundles/. This namespace is intended\nto assist with later 'git fetch' negotiations with a Git server,\nallowing the client to advertise which data it already has from a bundle\nURI.\n\nThese references can be confusing for a user when they appear as a\ndecoration in 'git log' output. Add \"refs/bundles/\" to the multi-valued\nlog.excludeDecoration config value. This is similar to the way\n\"refs/prefetch/\" is hidden by background prefetch operations in 'git\nmaintenance' as added by 96eaffebb (maintenance: set\nlog.excludeDecoration durin prefetch, 2021-01-19).\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/fetch-options.txt | 3 ++-\n bundle-uri.c | 7 +++++++\n 2 files changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 09bd1feeed8..8b801bcc2f3 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -321,4 +321,5 @@ endif::git-pull[]\n --bundle-uri=::\n \tInstead of fetching from a remote, fetch a bundle from the given\n \t`` and unbundle the data into the local repository. The refs\n-\tin the bundle will be stored under the `refs/bundle/*` namespace.\n+\tin the bundle will be stored under the hidden `refs/bundle/*`\n+\tnamespace.\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 08de7257c74..0692a62071a 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"bundle-uri.h\"\n #include \"bundle.h\"\n+#include \"config.h\"\n #include \"object-store.h\"\n #include \"refs.h\"\n #include \"run-command.h\"\n@@ -148,6 +149,12 @@ int fetch_bundle_uri(struct repository *r, const char *uri)\n \tif ((result = unbundle_from_file(r, filename.buf)))\n \t\tgoto cleanup;\n \n+\tgit_config_set_multivar_gently(\"log.excludedecoration\",\n+\t\t\t\t\t\"refs/bundle/\",\n+\t\t\t\t\t\"refs/bundle/\",\n+\t\t\t\t\tCONFIG_FLAGS_FIXED_VALUE |\n+\t\t\t\t\tCONFIG_FLAGS_MULTI_REPLACE);\n+\n cleanup:\n \tunlink(filename.buf);\n \tstrbuf_release(&filename);\n-- \ngitgitgadget\n\n"},{"id":"455594","messageId":"c7ec97011e0b2b6a347a144549be93bd78526476.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 05/24] fetch: add --bundle-uri option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:23Z","receivedAt":"2022-05-20T18:41:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTeach 'git fetch' a new --bundle-uri= option which changes the mode\nfrom fetching from a remote using the Git protocol to fetching a bundle\nfrom the given . See Documentation/technical/bundle-uri.txt for\nmore information on the design of this feature.\n\nThis implementation is limited to the most basic version of the feature.\nWe expect the content at that URI to be a bundle file, not a bundle\nlist. Bundle lists will be implemented later.\n\nThis implementation is sufficient for a bundle provider to create a\nsingle bootstrap bundle for a large repository. The user would bootstrap\na repository using a sequence of Git commands, such as:\n\n 1. git init && cd \n 2. git fetch --bundle-uri=\n 3. git remote add origin \n 4. git fetch origin\n 5. git checkout FETCH_HEAD\n\nLater changes will make this seamless within a 'git clone' command, but\nthis implementation is large enough to delay that integration.\n\nCurrently, this option supports URIs using \"http(s)://\", \"file://\" or\nsimply specifying a filename. Other protocols could be added in the\nfuture.\n\nRFC TODO: add end-to-end tests of this workflow.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/fetch-options.txt | 5 +++++\n Documentation/git-fetch.txt | 1 +\n builtin/fetch.c | 10 ++++++++++\n 3 files changed, 16 insertions(+)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 622bd84768b..09bd1feeed8 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -317,3 +317,8 @@ endif::git-pull[]\n -6::\n --ipv6::\n \tUse IPv6 addresses only, ignoring IPv4 addresses.\n+\n+--bundle-uri=::\n+\tInstead of fetching from a remote, fetch a bundle from the given\n+\t`` and unbundle the data into the local repository. The refs\n+\tin the bundle will be stored under the `refs/bundle/*` namespace.\ndiff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt\nindex e9d364669af..4fd8911b336 100644\n--- a/Documentation/git-fetch.txt\n+++ b/Documentation/git-fetch.txt\n@@ -13,6 +13,7 @@ SYNOPSIS\n 'git fetch' [] \n 'git fetch' --multiple [] [( | )...]\n 'git fetch' --all []\n+'git fetch' --bundle-uri= []\n \n \n DESCRIPTION\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex e3791f09ed5..cb0d2fbe82c 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -29,6 +29,7 @@\n #include \"commit-graph.h\"\n #include \"shallow.h\"\n #include \"worktree.h\"\n+#include \"bundle-uri.h\"\n \n #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n \n@@ -37,6 +38,7 @@ static const char * const builtin_fetch_usage[] = {\n \tN_(\"git fetch [] \"),\n \tN_(\"git fetch --multiple [] [( | )...]\"),\n \tN_(\"git fetch --all []\"),\n+\tN_(\"git fetch --bundle-uri= []\"),\n \tNULL\n };\n \n@@ -86,6 +88,7 @@ static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;\n static int fetch_write_commit_graph = -1;\n static int stdin_refspecs = 0;\n static int negotiate_only;\n+static const char *bundle_uri;\n \n static int git_fetch_config(const char *k, const char *v, void *cb)\n {\n@@ -224,6 +227,8 @@ static struct option builtin_fetch_options[] = {\n \t\t N_(\"write the commit-graph after fetching\")),\n \tOPT_BOOL(0, \"stdin\", &stdin_refspecs,\n \t\t N_(\"accept refspecs from stdin\")),\n+\tOPT_STRING(0, \"bundle-uri\", &bundle_uri, N_(\"uri\"),\n+\t\t N_(\"download bundle data from the given URI instead of from a remote\")),\n \tOPT_END()\n };\n \n@@ -2181,6 +2186,11 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (dry_run)\n \t\twrite_fetch_head = 0;\n \n+\tif (bundle_uri) {\n+\t\tresult = fetch_bundle_uri(the_repository, bundle_uri);\n+\t\tgoto cleanup;\n+\t}\n+\n \tif (all) {\n \t\tif (argc == 1)\n \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\n-- \ngitgitgadget\n\n"},{"id":"455595","messageId":"7bd508d5ebb5433615bee4a6d7777486a38213c4.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 04/24] bundle-uri: add support for http(s):// and file://","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:22Z","receivedAt":"2022-05-20T18:41:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change created the logic for copying a file by name. Now,\nfirst inspect the URI for an HTTP(S) prefix and use git-remote-https as\nthe way to download the data at that URI. Otherwise, check to see if\nfile:// is present and modify the prefix accordingly.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 64 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 91aba061833..08de7257c74 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -19,10 +19,73 @@ static void find_temp_filename(struct strbuf *name)\n \tunlink(name->buf);\n }\n \n+static int download_https_uri_to_file(const char *uri, const char *file)\n+{\n+\tint result = 0;\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tFILE *child_in = NULL, *child_out = NULL;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tint found_get = 0;\n+\n+\tstrvec_pushl(&cp.args, \"git-remote-https\", \"origin\", uri, NULL);\n+\tcp.in = -1;\n+\tcp.out = -1;\n+\n+\tif (start_command(&cp))\n+\t\treturn 1;\n+\n+\tchild_in = fdopen(cp.in, \"w\");\n+\tif (!child_in) {\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tchild_out = fdopen(cp.out, \"r\");\n+\tif (!child_out) {\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tfprintf(child_in, \"capabilities\\n\");\n+\tfflush(child_in);\n+\n+\twhile (!strbuf_getline(&line, child_out)) {\n+\t\tif (!line.len)\n+\t\t\tbreak;\n+\t\tif (!strcmp(line.buf, \"get\"))\n+\t\t\tfound_get = 1;\n+\t}\n+\tstrbuf_release(&line);\n+\n+\tif (!found_get) {\n+\t\tresult = error(_(\"insufficient capabilities\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\tfprintf(child_in, \"get %s %s\\n\\n\", uri, file);\n+\n+cleanup:\n+\tif (child_in)\n+\t\tfclose(child_in);\n+\tif (finish_command(&cp))\n+\t\treturn 1;\n+\tif (child_out)\n+\t\tfclose(child_out);\n+\treturn result;\n+}\n+\n static int copy_uri_to_file(const char *uri, const char *file)\n {\n+\tconst char *out;\n+\tif (skip_prefix(uri, \"https:\", &out) ||\n+\t skip_prefix(uri, \"http:\", &out))\n+\t\treturn download_https_uri_to_file(uri, file);\n+\n+\tif (!skip_prefix(uri, \"file://\", &out))\n+\t\tout = uri;\n+\n \t/* Copy as a file */\n-\treturn copy_file(uri, file, 0444);\n+\treturn !!copy_file(out, file, 0);\n }\n \n static int unbundle_from_file(struct repository *r, const char *file)\n-- \ngitgitgadget\n\n"},{"id":"455596","messageId":"cb270ece66d2ef7435a8f0987008935d51fb9b3a.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 03/24] bundle-uri: create basic file-copy logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:21Z","receivedAt":"2022-05-20T18:41:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore implementing a way to fetch bundles into a repository, create the\nbasic logic. Assume that the URI is actually a file path. Future logic\nwill make this more careful to other protocols.\n\nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n bundle-uri.c | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n bundle-uri.h | 14 ++++++++\n 3 files changed, 107 insertions(+)\n create mode 100644 bundle-uri.c\n create mode 100644 bundle-uri.h\n\ndiff --git a/Makefile b/Makefile\nindex f8bccfab5e9..8f27310836d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -887,6 +887,7 @@ LIB_OBJS += blob.o\n LIB_OBJS += bloom.o\n LIB_OBJS += branch.o\n LIB_OBJS += bulk-checkin.o\n+LIB_OBJS += bundle-uri.o\n LIB_OBJS += bundle.o\n LIB_OBJS += cache-tree.o\n LIB_OBJS += cbtree.o\ndiff --git a/bundle-uri.c b/bundle-uri.c\nnew file mode 100644\nindex 00000000000..91aba061833\n--- /dev/null\n+++ b/bundle-uri.c\n@@ -0,0 +1,92 @@\n+#include \"cache.h\"\n+#include \"bundle-uri.h\"\n+#include \"bundle.h\"\n+#include \"object-store.h\"\n+#include \"refs.h\"\n+#include \"run-command.h\"\n+\n+static void find_temp_filename(struct strbuf *name)\n+{\n+\tint fd;\n+\t/*\n+\t * Find a temporray filename that is available. This is briefly\n+\t * racy, but unlikely to collide.\n+\t */\n+\tfd = odb_mkstemp(name, \"bundles/tmp_uri_XXXXXX\");\n+\tif (fd < 0)\n+\t\tdie(_(\"failed to create temporary file\"));\n+\tclose(fd);\n+\tunlink(name->buf);\n+}\n+\n+static int copy_uri_to_file(const char *uri, const char *file)\n+{\n+\t/* Copy as a file */\n+\treturn copy_file(uri, file, 0444);\n+}\n+\n+static int unbundle_from_file(struct repository *r, const char *file)\n+{\n+\tint result = 0;\n+\tint bundle_fd;\n+\tstruct bundle_header header = BUNDLE_HEADER_INIT;\n+\tstruct strvec extra_index_pack_args = STRVEC_INIT;\n+\tstruct string_list_item *refname;\n+\tstruct strbuf bundle_ref = STRBUF_INIT;\n+\tsize_t bundle_prefix_len;\n+\n+\tif ((bundle_fd = read_bundle_header(file, &header)) < 0)\n+\t\treturn 1;\n+\n+\tresult = unbundle(r, &header, bundle_fd, &extra_index_pack_args);\n+\n+\t/*\n+\t * Convert all refs/heads/ from the bundle into refs/bundles/\n+\t * in the local repository.\n+\t */\n+\tstrbuf_addstr(&bundle_ref, \"refs/bundles/\");\n+\tbundle_prefix_len = bundle_ref.len;\n+\n+\tfor_each_string_list_item(refname, &header.references) {\n+\t\tstruct object_id *oid = refname->util;\n+\t\tstruct object_id old_oid;\n+\t\tconst char *branch_name;\n+\t\tint has_old;\n+\n+\t\tif (!skip_prefix(refname->string, \"refs/heads/\", &branch_name))\n+\t\t\tcontinue;\n+\n+\t\tstrbuf_setlen(&bundle_ref, bundle_prefix_len);\n+\t\tstrbuf_addstr(&bundle_ref, branch_name);\n+\n+\t\thas_old = !read_ref(bundle_ref.buf, &old_oid);\n+\t\tupdate_ref(\"fetched bundle\", bundle_ref.buf, oid,\n+\t\t\t has_old ? &old_oid : NULL,\n+\t\t\t REF_SKIP_OID_VERIFICATION,\n+\t\t\t UPDATE_REFS_MSG_ON_ERR);\n+\t}\n+\n+\tbundle_header_release(&header);\n+\treturn result;\n+}\n+\n+int fetch_bundle_uri(struct repository *r, const char *uri)\n+{\n+\tint result = 0;\n+\tstruct strbuf filename = STRBUF_INIT;\n+\n+\tfind_temp_filename(&filename);\n+\tif ((result = copy_uri_to_file(uri, filename.buf)))\n+\t\tgoto cleanup;\n+\n+\tif ((result = !is_bundle(filename.buf, 0)))\n+\t\tgoto cleanup;\n+\n+\tif ((result = unbundle_from_file(r, filename.buf)))\n+\t\tgoto cleanup;\n+\n+cleanup:\n+\tunlink(filename.buf);\n+\tstrbuf_release(&filename);\n+\treturn result;\n+}\n\\ No newline at end of file\ndiff --git a/bundle-uri.h b/bundle-uri.h\nnew file mode 100644\nindex 00000000000..8a152f1ef14\n--- /dev/null\n+++ b/bundle-uri.h\n@@ -0,0 +1,14 @@\n+#ifndef BUNDLE_URI_H\n+#define BUNDLE_URI_H\n+\n+struct repository;\n+\n+/**\n+ * Fetch data from the given 'uri' and unbundle the bundle data found\n+ * based on that information.\n+ *\n+ * Returns non-zero if no bundle information is found at the given 'uri'.\n+ */\n+int fetch_bundle_uri(struct repository *r, const char *uri);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"455597","messageId":"8653864ec468a354f7d78f5902e47e787a10208e.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 12/24] bundle-uri: unit test \"key=value\" parsing","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:30Z","receivedAt":"2022-05-20T18:41:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nCreate a new 'test-tool bundle-uri' test helper. This helper will assist\nin testing logic deep in the bundle URI feature.\n\nThis change introduces the 'parse-key-values' subcommand, which parses\nstdin as a list of lines. These are fed into bundle_uri_parse_line() to\ntest how we construct a 'struct bundle_list' from that data. The list is\nthen output to stdout as if the key-value pairs were a Git config file.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n bundle-uri.c | 34 ++++++++++++++\n bundle-uri.h | 3 ++\n t/helper/test-bundle-uri.c | 63 +++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5750-bundle-uri-parse.sh | 91 +++++++++++++++++++++++++++++++++++++\n t/test-lib-functions.sh | 11 +++++\n 8 files changed, 205 insertions(+)\n create mode 100644 t/helper/test-bundle-uri.c\n create mode 100755 t/t5750-bundle-uri-parse.sh\n\ndiff --git a/Makefile b/Makefile\nindex 8f27310836d..c8a14793005 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -706,6 +706,7 @@ PROGRAMS += $(patsubst %.o,git-%$X,$(PROGRAM_OBJS))\n TEST_BUILTINS_OBJS += test-advise.o\n TEST_BUILTINS_OBJS += test-bitmap.o\n TEST_BUILTINS_OBJS += test-bloom.o\n+TEST_BUILTINS_OBJS += test-bundle-uri.o\n TEST_BUILTINS_OBJS += test-chmtime.o\n TEST_BUILTINS_OBJS += test-config.o\n TEST_BUILTINS_OBJS += test-crontab.o\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 948f974c478..ff215422888 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -67,6 +67,40 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \treturn 0;\n }\n \n+static int summarize_bundle(struct remote_bundle_info *info, void *data)\n+{\n+\tFILE *fp = data;\n+\tfprintf(fp, \"[bundle \\\"%s\\\"]\\n\", info->id);\n+\tfprintf(fp, \"\\turi = %s\\n\", info->uri);\n+\treturn 0;\n+}\n+\n+void print_bundle_list(FILE *fp, struct bundle_list *list)\n+{\n+\tconst char *mode;\n+\n+\tswitch (list->mode) {\n+\tcase BUNDLE_MODE_ALL:\n+\t\tmode = \"all\";\n+\t\tbreak;\n+\n+\tcase BUNDLE_MODE_ANY:\n+\t\tmode = \"any\";\n+\t\tbreak;\n+\n+\tcase BUNDLE_MODE_NONE:\n+\tdefault:\n+\t\tmode = \"\";\n+\t}\n+\n+\tprintf(\"[bundle \\\"list\\\"]\\n\");\n+\tprintf(\"\\tversion = %d\\n\", list->version);\n+\tprintf(\"\\tmode = %s\\n\", mode);\n+\n+\tfor_all_bundles_in_list(list, summarize_bundle, fp);\n+}\n+\n+\n /**\n * Given a key-value pair, update the state of the given bundle list.\n * Returns 0 if the key-value pair is understood. Returns 1 if the key\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 344d5943922..4a12a90bf42 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -70,6 +70,9 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \t\t\t bundle_iterator iter,\n \t\t\t void *data);\n \n+struct FILE;\n+void print_bundle_list(FILE *fp, struct bundle_list *list);\n+\n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n * based on that information.\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nnew file mode 100644\nindex 00000000000..5cb0c9196fa\n--- /dev/null\n+++ b/t/helper/test-bundle-uri.c\n@@ -0,0 +1,63 @@\n+#include \"test-tool.h\"\n+#include \"parse-options.h\"\n+#include \"bundle-uri.h\"\n+#include \"strbuf.h\"\n+#include \"string-list.h\"\n+\n+static int cmd__bundle_uri_parse_key_values(int argc, const char **argv)\n+{\n+\tconst char *usage[] = {\n+\t\t\"test-tool bundle-uri parse-key-values []\",\n+\t\tNULL\n+\t};\n+\tstruct option options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL, options, usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION |\n+\t\t\t PARSE_OPT_KEEP_ARGV0);\n+\tif (argc == 1)\n+\t\tgoto usage;\n+\n+\tif (!strcmp(argv[1], \"parse-key-values\"))\n+\t\treturn cmd__bundle_uri_parse_key_values(argc - 1, argv + 1);\n+\terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n+\n+usage:\n+\tusage_with_options(usage, options);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 0424f7adf5d..bff823fbd3e 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -17,6 +17,7 @@ static struct test_cmd cmds[] = {\n \t{ \"advise\", cmd__advise_if_enabled },\n \t{ \"bitmap\", cmd__bitmap },\n \t{ \"bloom\", cmd__bloom },\n+\t{ \"bundle-uri\", cmd__bundle_uri },\n \t{ \"chmtime\", cmd__chmtime },\n \t{ \"config\", cmd__config },\n \t{ \"crontab\", cmd__crontab },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex c876e8246fb..eb747e07dd1 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -7,6 +7,7 @@\n int cmd__advise_if_enabled(int argc, const char **argv);\n int cmd__bitmap(int argc, const char **argv);\n int cmd__bloom(int argc, const char **argv);\n+int cmd__bundle_uri(int argc, const char **argv);\n int cmd__chmtime(int argc, const char **argv);\n int cmd__config(int argc, const char **argv);\n int cmd__crontab(int argc, const char **argv);\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nnew file mode 100755\nindex 00000000000..ebb80596125\n--- /dev/null\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -0,0 +1,91 @@\n+#!/bin/sh\n+\n+test_description=\"Test bundle-uri bundle_uri_parse_line()\"\n+\n+TEST_NO_CREATE_REPO=1\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success 'bundle_uri_parse_line() just URIs' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle \"list\"]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-key-values actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' '\n+\tcat >in <<-\\EOF &&\n+\t=bogus-value\n+\tbogus-key=\n+\tEOF\n+\n+\tcat >err.expect <<-EOF &&\n+\terror: bundle-uri: line has empty key or value\n+\terror: bad line: '\\''=bogus-value'\\''\n+\terror: bundle-uri: line has empty key or value\n+\terror: bad line: '\\''bogus-key='\\''\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle \"list\"]\n+\t\tversion = 1\n+\t\tmode = all\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty lines' '\n+\tcat >in <<-\\EOF &&\n+\tbundle.one.uri=http://example.com/bundle.bdl\n+\n+\tbundle.two.uri=https://example.com/bundle.bdl\n+\n+\tbundle.three.uri=file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\terror: bundle-uri: got an empty line\n+\terror: bad line: '\\'''\\''\n+\terror: bundle-uri: got an empty line\n+\terror: bad line: '\\'''\\''\n+\tEOF\n+\n+\t# We fail, but try to continue parsing regardless\n+\tcat >expect <<-\\EOF &&\n+\t[bundle \"list\"]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-key-values actual 2>err &&\n+\ttest_cmp err.expect err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_done\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 93c03380d44..747739f81b7 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1944,3 +1944,14 @@ test_is_magic_mtime () {\n \trm -f .git/test-mtime-actual\n \treturn $ret\n }\n+\n+# Given two filenames, parse both using 'git config --list --file'\n+# and compare the sorted output of those commands. Useful when\n+# wanting to ignore whitespace differences and sorting concerns.\n+test_cmp_config_output () {\n+\tgit config --list --file=\"$1\" >config-expect &&\n+\tgit config --list --file=\"$2\" >config-actual &&\n+\tsort config-expect >sorted-expect &&\n+\tsort config-actual >sorted-actual &&\n+\ttest_cmp sorted-expect sorted-actual\n+}\n-- \ngitgitgadget\n\n"},{"id":"455598","messageId":"2c94a55553bffdf90df7f80c8e971bfe2887fa1d.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 10/24] bundle-uri: create base key-value pair parsing","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:28Z","receivedAt":"2022-05-20T18:41:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThere will be two primary ways to advertise a bundle list: as a list of\npacket lines in Git's protocol v2 and as a config file served from a\nbundle URI. Both of these fundamentally use a list of key-value pairs.\nWe will use the same set of key-value pairs across these formats.\n\nCreate a new bundle_list_update() method that is currently unusued, but\nwill be used in the next change. It inspects each key to see if it is\nunderstood and then applies it to the given bundle_list. Here are the\nkeys that we teach Git to understand:\n\n* bundle.list.version: This value should be an integer. Git currently\n understands only version 1 and will ignore the list if the version is\n any other value. This version can be increased in the future if we\n need to add new keys that Git should not ignore. We can add new\n \"heuristic\" keys without incrementing the version.\n\n* bundle.list.mode: This value should be one of \"all\" or \"any\". If this\n mode is not understood, then Git will ignore the list. This mode\n indicates whether Git needs all of the bundle list items to make a\n complete view of the content or if any single item is sufficient.\n\nThe rest of the keys use a bundle identifier \"\" as part of the key\nname, and this cannot equal \"list\". Keys using the same \"\" describe\na common bundle list item.\n\n* bundle..uri: This stores the URI of the bundle item. This\n currently is expected to be an absolute URI, but will be relaxed to be\n a relative URI in the future.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 80 insertions(+)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex b9a219d3202..f18bba7071d 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -67,6 +67,86 @@ int for_all_bundles_in_list(struct bundle_list *list,\n \treturn 0;\n }\n \n+/**\n+ * Given a key-value pair, update the state of the given bundle list.\n+ * Returns 0 if the key-value pair is understood. Returns 1 if the key\n+ * is not understood or the value is malformed.\n+ */\n+MAYBE_UNUSED\n+static int bundle_list_update(const char *key, const char *value,\n+\t\t\t struct bundle_list *list)\n+{\n+\tconst char *pkey, *dot;\n+\tstruct strbuf id = STRBUF_INIT;\n+\tstruct remote_bundle_info lookup = REMOTE_BUNDLE_INFO_INIT;\n+\tstruct remote_bundle_info *bundle;\n+\n+\tif (!skip_prefix(key, \"bundle.\", &pkey))\n+\t\treturn 1;\n+\n+\tif (!strcmp(pkey, \"list.version\")) {\n+\t\tint version = atoi(value);\n+\t\tif (version != 1)\n+\t\t\treturn 1;\n+\n+\t\tlist->version = version;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!strcmp(pkey, \"list.mode\")) {\n+\t\tif (!strcmp(value, \"all\"))\n+\t\t\tlist->mode = BUNDLE_MODE_ALL;\n+\t\telse if (!strcmp(value, \"any\"))\n+\t\t\tlist->mode = BUNDLE_MODE_ANY;\n+\t\telse\n+\t\t\treturn 1;\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * All remaining keys must be of the form \"bundle..*\" where\n+\t * != \"list\"\n+\t */\n+\n+\tdot = strchr(pkey, '.');\n+\tif (!dot)\n+\t\treturn 1;\n+\tif (dot - pkey == 4 &&\n+\t !strncmp(pkey, \"list\", 4))\n+\t\treturn 1;\n+\n+\tstrbuf_add(&id, pkey, dot - pkey);\n+\tdot++;\n+\n+\t/*\n+\t * Check for an existing bundle with this , or create one\n+\t * if necessary.\n+\t */\n+\tlookup.id = id.buf;\n+\thashmap_entry_init(&lookup.ent, strhash(lookup.id));\n+\tif (!(bundle = hashmap_get_entry(&list->bundles, &lookup, ent, NULL))) {\n+\t\tCALLOC_ARRAY(bundle, 1);\n+\t\tbundle->id = strbuf_detach(&id, NULL);\n+\t\tstrbuf_init(&bundle->file, 0);\n+\t\thashmap_entry_init(&bundle->ent, strhash(bundle->id));\n+\t\thashmap_add(&list->bundles, &bundle->ent);\n+\t}\n+\tstrbuf_release(&id);\n+\n+\tif (!strcmp(dot, \"uri\")) {\n+\t\tfree(bundle->uri);\n+\t\tbundle->uri = xstrdup(value);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * At this point, we ignore any information that we don't\n+\t * understand, assuming it to be hints for a heuristic the client\n+\t * does not currently understand.\n+\t */\n+\treturn 0;\n+}\n+\n static void find_temp_filename(struct strbuf *name)\n {\n \tint fd;\n-- \ngitgitgadget\n\n"},{"id":"455599","messageId":"7cb3bd1ddae2687724f2f514408d3c013d11bb78.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 07/24] clone: add --bundle-uri option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:25Z","receivedAt":"2022-05-20T18:41:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nCloning a remote repository is one of the most expensive operations in\nGit. The server can spend a lot of CPU time generating a pack-file for\nthe client's request. The amount of data can clog the network for a long\ntime, and the Git protocol is not resumable. For users with poor network\nconnections or are located far away from the origin server, this can be\nespecially painful.\n\nThe 'git bundle fetch' command allows users to bootstrap a repository\nusing a set of bundles. However, this would require them to use 'git\ninit' first, followed by the 'git bundle fetch', and finally add a\nremote, fetch, and checkout the branch they want.\n\nInstead, integrate this workflow directly into 'git clone' with the\n--bundle-uri' option. If the user is aware of a bundle server, then they\ncan tell Git to bootstrap the new repository with these bundles before\nfetching the remaining objects from the origin server.\n\nRFC-TODO: Document this option in git-clone.txt.\n\nRFC-TODO: I added a comment about the location of this code being\nnecessary for the later step of auto-discovering the bundle URI from the\norigin server. This is probably not actually a requirement, but rather a\npain point around how I implemented the feature. If a --bundle-uri\noption is specified, but SSH is used for the clone, then the SSH\nconnection is left open while Git downloads bundles from another server.\nThis is sub-optimal and should be reconsidered when fully reviewed.\n\nRFC-TODO: create tests for this option with a variety of URI types.\n\nRFC-TODO: a simple end-to-end test is available at the end of the\nseries.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 22 ++++++++++++++++++++++\n 1 file changed, 22 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 52316563795..fd1ae82e57b 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -34,6 +34,7 @@\n #include \"list-objects-filter-options.h\"\n #include \"hook.h\"\n #include \"bundle.h\"\n+#include \"bundle-uri.h\"\n \n /*\n * Overall FIXMEs:\n@@ -77,6 +78,7 @@ static int option_filter_submodules = -1; /* unspecified */\n static int config_filter_submodules = -1; /* unspecified */\n static struct string_list server_options = STRING_LIST_INIT_NODUP;\n static int option_remote_submodules;\n+static const char *bundle_uri;\n \n static int recurse_submodules_cb(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -160,6 +162,8 @@ static struct option builtin_clone_options[] = {\n \t\t N_(\"any cloned submodules will use their remote-tracking branch\")),\n \tOPT_BOOL(0, \"sparse\", &option_sparse_checkout,\n \t\t N_(\"initialize sparse-checkout file to include only files at root\")),\n+\tOPT_STRING(0, \"bundle-uri\", &bundle_uri,\n+\t\t N_(\"uri\"), N_(\"a URI for downloading bundles before fetching from origin remote\")),\n \tOPT_END()\n };\n \n@@ -1232,6 +1236,24 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \n \trefs = transport_get_remote_refs(transport, &transport_ls_refs_options);\n \n+\t/*\n+\t * NOTE: The bundle URI download takes place after transport_get_remote_refs()\n+\t * because a later change will introduce a check for recommended features,\n+\t * which might include a recommended bundle URI.\n+\t */\n+\n+\t/*\n+\t * Before fetching from the remote, download and install bundle\n+\t * data from the --bundle-uri option.\n+\t */\n+\tif (bundle_uri) {\n+\t\t/* At this point, we need the_repository to match the cloned repo. */\n+\t\trepo_init(the_repository, git_dir, work_tree);\n+\t\tif (fetch_bundle_uri(the_repository, bundle_uri))\n+\t\t\twarning(_(\"failed to fetch objects from bundle URI '%s'\"),\n+\t\t\t\tbundle_uri);\n+\t}\n+\n \tif (refs)\n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n \n-- \ngitgitgadget\n\n"},{"id":"455600","messageId":"4e0286f8e20da83c3d111e99eb9f6d9d764a84e6.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 11/24] bundle-uri: create \"key=value\" line parsing","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:29Z","receivedAt":"2022-05-20T18:41:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nWhen advertising a bundle list over Git's protocol v2, we will use\npacket lines. Each line will be of the form \"key=value\" representing a\nbundle list. Connect the API necessary for Git's transport to the\nkey-value pair parsing created in the previous change.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 29 +++++++++++++++++++++++++++--\n bundle-uri.h | 14 +++++++++++++-\n 2 files changed, 40 insertions(+), 3 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex f18bba7071d..948f974c478 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -72,7 +72,6 @@ int for_all_bundles_in_list(struct bundle_list *list,\n * Returns 0 if the key-value pair is understood. Returns 1 if the key\n * is not understood or the value is malformed.\n */\n-MAYBE_UNUSED\n static int bundle_list_update(const char *key, const char *value,\n \t\t\t struct bundle_list *list)\n {\n@@ -300,4 +299,30 @@ cleanup:\n \tunlink(filename.buf);\n \tstrbuf_release(&filename);\n \treturn result;\n-}\n\\ No newline at end of file\n+}\n+\n+/**\n+ * General API for {transport,connect}.c etc.\n+ */\n+int bundle_uri_parse_line(struct bundle_list *list, const char *line)\n+{\n+\tint result;\n+\tconst char *equals;\n+\tstruct strbuf key = STRBUF_INIT;\n+\n+\tif (!strlen(line))\n+\t\treturn error(_(\"bundle-uri: got an empty line\"));\n+\n+\tequals = strchr(line, '=');\n+\n+\tif (!equals)\n+\t\treturn error(_(\"bundle-uri: line is not of the form 'key=value'\"));\n+\tif (line == equals || !*(equals + 1))\n+\t\treturn error(_(\"bundle-uri: line has empty key or value\"));\n+\n+\tstrbuf_add(&key, line, equals - line);\n+\tresult = bundle_list_update(key.buf, equals + 1, list);\n+\tstrbuf_release(&key);\n+\n+\treturn result;\n+}\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 0f95197744c..344d5943922 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -78,4 +78,16 @@ int for_all_bundles_in_list(struct bundle_list *list,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n-#endif\n+/**\n+ * General API for {transport,connect}.c etc.\n+ */\n+\n+/**\n+ * Parse a \"key=value\" packet line from the bundle-uri verb.\n+ *\n+ * Returns 0 on success and non-zero on error.\n+ */\n+int bundle_uri_parse_line(struct bundle_list *list,\n+\t\t\t const char *line);\n+\n+#endif /* BUNDLE_URI_H */\n-- \ngitgitgadget\n\n"},{"id":"455601","messageId":"1fd302d0e56e877c13c5d4a5a5dc128104cc5fd3.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 14/24] bundle-uri: parse bundle list in config format","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:32Z","receivedAt":"2022-05-20T18:41:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a bundle provider wants to operate independently from a Git remote,\nthey want to provide a single, consistent URI that users can use in\ntheir 'git clone --bundle-uri' commands. At this point, the Git client\nexpects that URI to be a single bundle that can be unbundled and used to\nbootstrap the rest of the clone from the Git server. This single bundle\ncannot be re-used to assist with future incremental fetches.\n\nTo allow for the incremental fetch case, teach Git to understand a\nbundle list that could be advertised at an independent bundle URI. Such\na bundle list is likely to be inspected by human readers, even if only\nby the bundle provider creating the list. For this reason, we can take\nour expected \"key=value\" pairs and instead format them using Git config\nformat.\n\nCreate parse_bundle_list_in_config_format() to parse a file in config\nformat and convert that into a 'struct bundle_list' filled with its\nunderstanding of the contents.\n\nBe careful to call git_config_from_file_with_options() because the\ndefault action for git_config_from_file() is to die() on a parsing\nerror. The current warning isn't particularly helpful if it arises to a\nuser, but it will be made more verbose at a higher layer later.\n\nUpdate 'test-tool bundle-uri' to take this config file format as input.\nIt uses a filename instead of stdin because there is no existing way to\nparse a FILE pointer in the config machinery. Using\ngit_config_from_mem() is overly complicated and more likely to introduce\nbugs than this simpler version. I would rather have a slightly confusing\ntest helper than complicated product code.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 28 +++++++++++++++++++++\n bundle-uri.h | 10 ++++++++\n t/helper/test-bundle-uri.c | 45 ++++++++++++++++++++++++++-------\n t/t5750-bundle-uri-parse.sh | 50 +++++++++++++++++++++++++++++++++++++\n 4 files changed, 124 insertions(+), 9 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 74a26ce6c00..ab4b6385fc0 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -180,6 +180,34 @@ static int bundle_list_update(const char *key, const char *value,\n \treturn 0;\n }\n \n+static int config_to_bundle_list(const char *key, const char *value, void *data)\n+{\n+\tstruct bundle_list *list = data;\n+\treturn bundle_list_update(key, value, list);\n+}\n+\n+int parse_bundle_list_in_config_format(const char *uri,\n+\t\t\t\t const char *filename,\n+\t\t\t\t struct bundle_list *list)\n+{\n+\tint result;\n+\tstruct config_options opts = {\n+\t\t.error_action = CONFIG_ERROR_ERROR,\n+\t};\n+\n+\tlist->mode = BUNDLE_MODE_NONE;\n+\tresult = git_config_from_file_with_options(config_to_bundle_list,\n+\t\t\t\t\t\t filename, list,\n+\t\t\t\t\t\t &opts);\n+\n+\tif (!result && list->mode == BUNDLE_MODE_NONE) {\n+\t\twarning(_(\"bundle list at '%s' has no mode\"), uri);\n+\t\tresult = 1;\n+\t}\n+\n+\treturn result;\n+}\n+\n static void find_temp_filename(struct strbuf *name)\n {\n \tint fd;\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 4a12a90bf42..b0183870336 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -73,6 +73,16 @@ int for_all_bundles_in_list(struct bundle_list *list,\n struct FILE;\n void print_bundle_list(FILE *fp, struct bundle_list *list);\n \n+/**\n+ * A bundle URI may point to a bundle list where the key=value\n+ * pairs are provided in config file format. This method is\n+ * exposed publicly for testing purposes.\n+ */\n+\n+int parse_bundle_list_in_config_format(const char *uri,\n+\t\t\t\t const char *filename,\n+\t\t\t\t struct bundle_list *list);\n+\n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n * based on that information.\ndiff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c\nindex 5cb0c9196fa..23ce0eebca3 100644\n--- a/t/helper/test-bundle-uri.c\n+++ b/t/helper/test-bundle-uri.c\n@@ -4,27 +4,52 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n \n-static int cmd__bundle_uri_parse_key_values(int argc, const char **argv)\n+enum input_mode {\n+\tKEY_VALUE_PAIRS,\n+\tCONFIG_FILE,\n+};\n+\n+static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mode)\n {\n-\tconst char *usage[] = {\n+\tconst char *key_value_usage[] = {\n \t\t\"test-tool bundle-uri parse-key-values \",\n+\t\tNULL\n+\t};\n \tstruct option options[] = {\n \t\tOPT_END(),\n \t};\n+\tconst char **usage = key_value_usage;\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct bundle_list list;\n \tint err = 0;\n \n-\targc = parse_options(argc, argv, NULL, options, usage, 0);\n-\tif (argc)\n-\t\tgoto usage;\n+\tif (mode == CONFIG_FILE)\n+\t\tusage = config_usage;\n+\n+\targc = parse_options(argc, argv, NULL, options, usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tinit_bundle_list(&list);\n-\twhile (strbuf_getline(&sb, stdin) != EOF) {\n-\t\tif (bundle_uri_parse_line(&list, sb.buf) < 0)\n-\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\n+\tswitch (mode) {\n+\tcase KEY_VALUE_PAIRS:\n+\t\tif (argc)\n+\t\t\tgoto usage;\n+\t\twhile (strbuf_getline(&sb, stdin) != EOF) {\n+\t\t\tif (bundle_uri_parse_line(&list, sb.buf) < 0)\n+\t\t\t\terr = error(\"bad line: '%s'\", sb.buf);\n+\t\t}\n+\t\tbreak;\n+\n+\tcase CONFIG_FILE:\n+\t\tif (argc != 1)\n+\t\t\tgoto usage;\n+\t\terr = parse_bundle_list_in_config_format(\"\", argv[0], &list);\n+\t\tbreak;\n \t}\n \tstrbuf_release(&sb);\n \n@@ -55,7 +80,9 @@ int cmd__bundle_uri(int argc, const char **argv)\n \t\tgoto usage;\n \n \tif (!strcmp(argv[1], \"parse-key-values\"))\n-\t\treturn cmd__bundle_uri_parse_key_values(argc - 1, argv + 1);\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS);\n+\tif (!strcmp(argv[1], \"parse-config\"))\n+\t\treturn cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE);\n \terror(\"there is no test-tool bundle-uri tool '%s'\", argv[1]);\n \n usage:\ndiff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh\nindex ebb80596125..c2b7a8ce968 100755\n--- a/t/t5750-bundle-uri-parse.sh\n+++ b/t/t5750-bundle-uri-parse.sh\n@@ -88,4 +88,54 @@ test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty lines' '\n \ttest_cmp_config_output expect actual\n '\n \n+test_expect_success 'parse config format: just URIs' '\n+\tcat >expect <<-\\EOF &&\n+\t[bundle \"list\"]\n+\t\tversion = 1\n+\t\tmode = all\n+\t[bundle \"one\"]\n+\t\turi = http://example.com/bundle.bdl\n+\t[bundle \"two\"]\n+\t\turi = https://example.com/bundle.bdl\n+\t[bundle \"three\"]\n+\t\turi = file:///usr/share/git/bundle.bdl\n+\tEOF\n+\n+\ttest-tool bundle-uri parse-config expect >actual 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n+test_expect_success 'parse config format edge cases: empty key or value' '\n+\tcat >in1 <<-\\EOF &&\n+\t= bogus-value\n+\tEOF\n+\n+\tcat >err1 <<-EOF &&\n+\terror: bad config line 1 in file in1\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\t[bundle \"list\"]\n+\t\tversion = 1\n+\t\tmode = \n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-config in1 >actual 2>err &&\n+\ttest_cmp err1 err &&\n+\ttest_cmp_config_output expect actual &&\n+\n+\tcat >in2 <<-\\EOF &&\n+\tbogus-key =\n+\tEOF\n+\n+\tcat >err2 <<-EOF &&\n+\twarning: bundle list at '\\'''\\'' has no mode\n+\tEOF\n+\n+\ttest_must_fail test-tool bundle-uri parse-config in2 >actual 2>err &&\n+\ttest_cmp err2 err &&\n+\ttest_cmp_config_output expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"455602","messageId":"f87cb15a46d2da367e5abbd5da8bcdcc9e481514.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 09/24] bundle-uri: create bundle_list struct and helpers","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:27Z","receivedAt":"2022-05-20T18:42:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt will likely be rare where a user uses a single bundle URI and expects\nthat URI to point to a bundle. Instead, that URI will likely be a list\nof bundles provided in some format. Alternatively, the Git server could\nadvertise a list of bundles.\n\nIn anticipation of these two ways of advertising multiple bundles,\ncreate a data structure that represents such a list. This will be\npopulated using a common API, but for now focus on what data can be\nrepresented.\n\nEach list contains a number of remote_bundle_info structs. These contain\nan 'id' that is used to uniquely identify them in the list, and also a\n'uri' that contains the location of its data. Finally, there is a strbuf\ncontaining the filename used when Git downloads the contents to disk.\n\nThe list itself stores these remote_bundle_info structs in a hashtable\nusing 'id' as the key. The order of the structs in the input is\nconsidered unimportant, but future modifications to the format and these\ndata structures will place ordering possibilities on the set. The list\nalso has a few \"global\" properties, including the version (used when\nparsing the list) and the mode. The mode is one of these two options:\n\n1. BUNDLE_MODE_ALL: all listed URIs are intended to be combined\n together. The client should download all of the advertised data to\n have a complete copy of the data.\n\n2. BUNDLE_MODE_ANY: any one listed item is sufficient to have a complete\n copy of the data. The client can choose arbitrarily from these\n options. In the future, the client may use pings to find the closest\n URI among geodistributed replicas, or use some other heuristic\n information added to the format.\n\nThis API is currently unused, but will soon be expanded with parsing\nlogic and then be consumed by the bundle URI download logic.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++\n bundle-uri.h | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 128 insertions(+)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 0692a62071a..b9a219d3202 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -5,6 +5,67 @@\n #include \"object-store.h\"\n #include \"refs.h\"\n #include \"run-command.h\"\n+#include \"hashmap.h\"\n+#include \"pkt-line.h\"\n+\n+static int compare_bundles(const void *hashmap_cmp_fn_data,\n+\t\t\t const struct hashmap_entry *he1,\n+\t\t\t const struct hashmap_entry *he2,\n+\t\t\t const void *id)\n+{\n+\tconst struct remote_bundle_info *e1 =\n+\t\tcontainer_of(he1, const struct remote_bundle_info, ent);\n+\tconst struct remote_bundle_info *e2 =\n+\t\tcontainer_of(he2, const struct remote_bundle_info, ent);\n+\n+\treturn strcmp(e1->id, id ? (const char *)id : e2->id);\n+}\n+\n+void init_bundle_list(struct bundle_list *list)\n+{\n+\tmemset(list, 0, sizeof(*list));\n+\n+\t/* Implied defaults. */\n+\tlist->mode = BUNDLE_MODE_ALL;\n+\tlist->version = 1;\n+\n+\thashmap_init(&list->bundles, compare_bundles, NULL, 0);\n+}\n+\n+static int clear_remote_bundle_info(struct remote_bundle_info *bundle,\n+\t\t\t\t void *data)\n+{\n+\tfree(bundle->id);\n+\tfree(bundle->uri);\n+\tstrbuf_release(&bundle->file);\n+\treturn 0;\n+}\n+\n+void clear_bundle_list(struct bundle_list *list)\n+{\n+\tif (!list)\n+\t\treturn;\n+\n+\tfor_all_bundles_in_list(list, clear_remote_bundle_info, NULL);\n+\thashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent);\n+}\n+\n+int for_all_bundles_in_list(struct bundle_list *list,\n+\t\t\t bundle_iterator iter,\n+\t\t\t void *data)\n+{\n+\tstruct remote_bundle_info *info;\n+\tstruct hashmap_iter i;\n+\n+\thashmap_for_each_entry(&list->bundles, &i, info, ent) {\n+\t\tint result = iter(info, data);\n+\n+\t\tif (result)\n+\t\t\treturn result;\n+\t}\n+\n+\treturn 0;\n+}\n \n static void find_temp_filename(struct strbuf *name)\n {\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex 8a152f1ef14..0f95197744c 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -1,7 +1,74 @@\n #ifndef BUNDLE_URI_H\n #define BUNDLE_URI_H\n \n+#include \"hashmap.h\"\n+#include \"strbuf.h\"\n+\n struct repository;\n+struct strbuf;\n+struct string_list;\n+\n+/**\n+ * The remote_bundle_info struct contains information for a single bundle\n+ * URI. This may be initialized simply by a given URI or might have\n+ * additional metadata associated with it if the bundle was advertised by\n+ * a bundle list.\n+ */\n+struct remote_bundle_info {\n+\tstruct hashmap_entry ent;\n+\n+\t/**\n+\t * The 'id' is a name given to the bundle for reference\n+\t * by other bundle infos.\n+\t */\n+\tchar *id;\n+\n+\t/**\n+\t * The 'uri' is the location of the remote bundle so\n+\t * it can be downloaded on-demand. This will be NULL\n+\t * if there was no table of contents.\n+\t */\n+\tchar *uri;\n+\n+\t/**\n+\t * If the bundle has been downloaded, then 'file' is a\n+\t * filename storing its contents. Otherwise, 'file' is\n+\t * an empty string.\n+\t */\n+\tstruct strbuf file;\n+};\n+\n+#define REMOTE_BUNDLE_INFO_INIT { \\\n+\t.file = STRBUF_INIT, \\\n+}\n+\n+enum bundle_list_mode {\n+\tBUNDLE_MODE_NONE = 0,\n+\tBUNDLE_MODE_ALL,\n+\tBUNDLE_MODE_ANY\n+};\n+\n+/**\n+ * A bundle_list contains an unordered set of remote_bundle_info structs,\n+ * as well as information about the bundle listing, such as version and\n+ * mode.\n+ */\n+struct bundle_list {\n+\tint version;\n+\tenum bundle_list_mode mode;\n+\tunsigned forFetch : 1;\n+\tstruct hashmap bundles;\n+};\n+\n+void init_bundle_list(struct bundle_list *list);\n+void clear_bundle_list(struct bundle_list *list);\n+\n+typedef int (*bundle_iterator)(struct remote_bundle_info *bundle,\n+\t\t\t void *data);\n+\n+int for_all_bundles_in_list(struct bundle_list *list,\n+\t\t\t bundle_iterator iter,\n+\t\t\t void *data);\n \n /**\n * Fetch data from the given 'uri' and unbundle the bundle data found\n-- \ngitgitgadget\n\n"},{"id":"455603","messageId":"ceb253ec445b7275943e1c04ac51f12869271e91.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 15/24] bundle-uri: fetch a list of bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:33Z","receivedAt":"2022-05-20T18:42:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen the content at a given bundle URI is not understood as a bundle\n(based on inspecting the initial content), then Git currently gives up\nand ignores that content. Independent bundle providers may want to split\nup the bundle content into multiple bundles, but still make them\navailable from a single URI.\n\nTeach Git to attempt parsing the bundle URI content as a Git config file\nproviding the key=value pairs for a bundle list. Git then looks at the\nmode of the list to see if ANY single bundle is sufficient or if ALL\nbundles are required. The content at the selected URIs are downloaded\nand the content is inspected again, creating a recursive process.\n\nTo guard the recursion against malformed or malicious content, limit the\nrecursion depth to a reasonable four for now. This can be converted to a\nconfigured value in the future if necessary. The value of four is twice\nas high as expected to be useful (a bundle list is unlikely to point to\nmore bundle lists).\n\nRFC TODO: add tests for this feature right now.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 213 ++++++++++++++++++++++++++++++++++++++++++++++-----\n bundle-uri.h | 6 ++\n 2 files changed, 200 insertions(+), 19 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex ab4b6385fc0..601e6e87822 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -35,9 +35,10 @@ void init_bundle_list(struct bundle_list *list)\n static int clear_remote_bundle_info(struct remote_bundle_info *bundle,\n \t\t\t\t void *data)\n {\n-\tfree(bundle->id);\n-\tfree(bundle->uri);\n+\tFREE_AND_NULL(bundle->id);\n+\tFREE_AND_NULL(bundle->uri);\n \tstrbuf_release(&bundle->file);\n+\tbundle->unbundled = 0;\n \treturn 0;\n }\n \n@@ -336,18 +337,102 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \treturn result;\n }\n \n+struct bundle_list_context {\n+\tstruct repository *r;\n+\tstruct bundle_list *list;\n+\tenum bundle_list_mode mode;\n+\tint count;\n+\tint depth;\n+};\n+\n+/*\n+ * This early definition is necessary because we use indirect recursion:\n+ *\n+ * While iterating through a bundle list that was downloaded as part\n+ * of fetch_bundle_uri_internal(), iterator methods eventually call it\n+ * again, but with depth + 1.\n+ */\n+static int fetch_bundle_uri_internal(struct repository *r,\n+\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t int depth,\n+\t\t\t\t struct bundle_list *list);\n+\n+static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data)\n+{\n+\tstruct bundle_list_context *ctx = data;\n+\n+\tif (ctx->mode == BUNDLE_MODE_ANY && ctx->count)\n+\t\treturn 0;\n+\n+\tctx->count++;\n+\treturn fetch_bundle_uri_internal(ctx->r, bundle, ctx->depth + 1, ctx->list);\n+}\n+\n+static int download_bundle_list(struct repository *r,\n+\t\t\t\tstruct bundle_list *local_list,\n+\t\t\t\tstruct bundle_list *global_list,\n+\t\t\t\tint depth)\n+{\n+\tstruct bundle_list_context ctx = {\n+\t\t.r = r,\n+\t\t.list = global_list,\n+\t\t.depth = depth + 1,\n+\t\t.mode = local_list->mode,\n+\t};\n+\n+\treturn for_all_bundles_in_list(local_list, download_bundle_to_file, &ctx);\n+}\n+\n+static int fetch_bundle_list_in_config_format(struct repository *r,\n+\t\t\t\t\t struct bundle_list *global_list,\n+\t\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t\t int depth)\n+{\n+\tint result;\n+\tstruct bundle_list list_from_bundle;\n+\n+\tinit_bundle_list(&list_from_bundle);\n+\n+\tif ((result = parse_bundle_list_in_config_format(bundle->uri,\n+\t\t\t\t\t\t\t bundle->file.buf,\n+\t\t\t\t\t\t\t &list_from_bundle)))\n+\t\tgoto cleanup;\n+\n+\tif (list_from_bundle.mode == BUNDLE_MODE_NONE) {\n+\t\twarning(_(\"unrecognized bundle mode from URI '%s'\"),\n+\t\t\tbundle->uri);\n+\t\tresult = -1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif ((result = download_bundle_list(r, &list_from_bundle,\n+\t\t\t\t\t global_list, depth)))\n+\t\tgoto cleanup;\n+\n+cleanup:\n+\tclear_bundle_list(&list_from_bundle);\n+\treturn result;\n+}\n+\n /**\n * This limits the recursion on fetch_bundle_uri_internal() when following\n * bundle lists.\n */\n static int max_bundle_uri_depth = 4;\n \n+/**\n+ * Recursively download all bundles advertised at the given URI\n+ * to files. If the file is a bundle, then add it to the given\n+ * 'list'. Otherwise, expect a bundle list and recurse on the\n+ * URIs in that list according to the list mode (ANY or ALL).\n+ */\n static int fetch_bundle_uri_internal(struct repository *r,\n-\t\t\t\t const char *uri,\n-\t\t\t\t int depth)\n+\t\t\t\t struct remote_bundle_info *bundle,\n+\t\t\t\t int depth,\n+\t\t\t\t struct bundle_list *list)\n {\n \tint result = 0;\n-\tstruct strbuf filename = STRBUF_INIT;\n+\tstruct remote_bundle_info *bcopy;\n \n \tif (depth >= max_bundle_uri_depth) {\n \t\twarning(_(\"exceeded bundle URI recursion limit (%d)\"),\n@@ -355,31 +440,121 @@ static int fetch_bundle_uri_internal(struct repository *r,\n \t\treturn -1;\n \t}\n \n-\tfind_temp_filename(&filename);\n-\tif ((result = copy_uri_to_file(uri, filename.buf)))\n+\tif (!bundle->file.len)\n+\t\tfind_temp_filename(&bundle->file);\n+\tif ((result = copy_uri_to_file(bundle->uri, bundle->file.buf)))\n \t\tgoto cleanup;\n \n-\tif ((result = !is_bundle(filename.buf, 0)))\n-\t\tgoto cleanup;\n-\n-\tif ((result = unbundle_from_file(r, filename.buf)))\n+\tif ((result = !is_bundle(bundle->file.buf, 1))) {\n+\t\tresult = fetch_bundle_list_in_config_format(\n+\t\t\t\tr, list, bundle, depth);\n \t\tgoto cleanup;\n+\t}\n \n-\tgit_config_set_multivar_gently(\"log.excludedecoration\",\n-\t\t\t\t\t\"refs/bundle/\",\n-\t\t\t\t\t\"refs/bundle/\",\n-\t\t\t\t\tCONFIG_FLAGS_FIXED_VALUE |\n-\t\t\t\t\tCONFIG_FLAGS_MULTI_REPLACE);\n+\t/* Copy the bundle and insert it into the global list. */\n+\tCALLOC_ARRAY(bcopy, 1);\n+\tbcopy->id = xstrdup(bundle->id);\n+\tstrbuf_init(&bcopy->file, 0);\n+\tstrbuf_add(&bcopy->file, bundle->file.buf, bundle->file.len);\n+\tstrbuf_detach(&bundle->file, NULL);\n+\thashmap_entry_init(&bcopy->ent, strhash(bcopy->id));\n+\thashmap_add(&list->bundles, &bcopy->ent);\n \n cleanup:\n-\tunlink(filename.buf);\n-\tstrbuf_release(&filename);\n+\tif (bundle->file.len)\n+\t\tunlink(bundle->file.buf);\n \treturn result;\n }\n \n+struct attempt_unbundle_context {\n+\tstruct repository *r;\n+\tint success_count;\n+\tint failure_count;\n+};\n+\n+static int attempt_unbundle(struct remote_bundle_info *info, void *data)\n+{\n+\tstruct attempt_unbundle_context *ctx = data;\n+\n+\tif (info->unbundled || !unbundle_from_file(ctx->r, info->file.buf)) {\n+\t\tctx->success_count++;\n+\t\tinfo->unbundled = 1;\n+\t} else {\n+\t\tctx->failure_count++;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int unbundle_all_bundles(struct repository *r,\n+\t\t\t\tstruct bundle_list *list)\n+{\n+\tint last_success_count = -1;\n+\tstruct attempt_unbundle_context ctx = {\n+\t\t.r = r,\n+\t};\n+\n+\t/*\n+\t * Iterate through all bundles looking for ones that can\n+\t * successfully unbundle. If any succeed, then perhaps another\n+\t * will succeed in the next attempt.\n+\t */\n+\twhile (last_success_count < ctx.success_count) {\n+\t\tlast_success_count = ctx.success_count;\n+\n+\t\tctx.success_count = 0;\n+\t\tctx.failure_count = 0;\n+\t\tfor_all_bundles_in_list(list, attempt_unbundle, &ctx);\n+\t}\n+\n+\tif (ctx.success_count)\n+\t\tgit_config_set_multivar_gently(\"log.excludedecoration\",\n+\t\t\t\t\t\t\"refs/bundle/\",\n+\t\t\t\t\t\t\"refs/bundle/\",\n+\t\t\t\t\t\tCONFIG_FLAGS_FIXED_VALUE |\n+\t\t\t\t\t\tCONFIG_FLAGS_MULTI_REPLACE);\n+\n+\tif (ctx.failure_count) {\n+\t\twarning(_(\"failed to unbundle %d bundles\"),\n+\t\t\tctx.failure_count);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int unlink_bundle(struct remote_bundle_info *info, void *data)\n+{\n+\tif (info->file.buf)\n+\t\tunlink_or_warn(info->file.buf);\n+\treturn 0;\n+}\n+\n int fetch_bundle_uri(struct repository *r, const char *uri)\n {\n-\treturn fetch_bundle_uri_internal(r, uri, 0);\n+\tint result;\n+\tstruct bundle_list list;\n+\tstruct remote_bundle_info bundle = {\n+\t\t.uri = xstrdup(uri),\n+\t\t.id = xstrdup(\"\"),\n+\t\t.file = STRBUF_INIT,\n+\t};\n+\n+\tinit_bundle_list(&list);\n+\n+\t/* If a bundle is added to this global list, then it is required. */\n+\tlist.mode = BUNDLE_MODE_ALL;\n+\n+\tif ((result = fetch_bundle_uri_internal(r, &bundle, 0, &list)))\n+\t\tgoto cleanup;\n+\n+\tresult = unbundle_all_bundles(r, &list);\n+\n+cleanup:\n+\tfor_all_bundles_in_list(&list, unlink_bundle, NULL);\n+\tclear_bundle_list(&list);\n+\tclear_remote_bundle_info(&bundle, NULL);\n+\treturn result;\n }\n \n /**\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex b0183870336..adce5dc07e2 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -36,6 +36,12 @@ struct remote_bundle_info {\n \t * an empty string.\n \t */\n \tstruct strbuf file;\n+\n+\t/**\n+\t * If the bundle has been unbundled successfully, then\n+\t * this boolean is true.\n+\t */\n+\tunsigned unbundled:1;\n };\n \n #define REMOTE_BUNDLE_INFO_INIT { \\\n-- \ngitgitgadget\n\n"},{"id":"455604","messageId":"94f552e771644aef60c23e5d9379723951542031.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 16/24] protocol v2: add server-side \"bundle-uri\" skeleton","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:34Z","receivedAt":"2022-05-20T18:42:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nAdd a skeleton server-side implementation of a new \"bundle-uri\" command\nto protocol v2. This will allow conforming clients to optionally seed\ntheir initial clones or incremental fetches from URLs containing\n\"*.bundle\" files created with \"git bundle create\".\n\nThis change only performs the basic boilerplate of advertising a new\nprotocol v2 capability. The new 'bundle-uri' capability allows a client\nto request a list of bundles. Right now, the server only returns a flush\npacket, which corresponds to an empty advertisement.\n\nThe critical bit right now is that the new boolean\nuploadPack.adverstiseBundleURIs config value signals whether or not this\ncapability should be advertised at all.\n\nRFC TODO: Write documentation.\n\nRFC TODO: Compare to original implementation.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 36 ++++++++++++++++++++++++++++++++++++\n bundle-uri.h | 7 +++++++\n serve.c | 6 ++++++\n t/t5701-git-serve.sh | 40 +++++++++++++++++++++++++++++++++++++++-\n 4 files changed, 88 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 601e6e87822..79a6b7ed8b8 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -557,6 +557,42 @@ cleanup:\n \treturn result;\n }\n \n+/**\n+ * API for serve.c.\n+ */\n+\n+static int advertise_bundle_uri = -1;\n+\n+int bundle_uri_advertise(struct repository *r, struct strbuf *value)\n+{\n+\tif (advertise_bundle_uri != -1)\n+\t\tgoto cached;\n+\n+\tadvertise_bundle_uri = 0;\n+\tgit_config_get_maybe_bool(\"uploadpack.advertisebundleuris\", &advertise_bundle_uri);\n+\n+cached:\n+\treturn advertise_bundle_uri;\n+}\n+\n+int bundle_uri_command(struct repository *r,\n+\t\t struct packet_reader *request)\n+{\n+\tstruct packet_writer writer;\n+\tpacket_writer_init(&writer, 1);\n+\n+\twhile (packet_reader_read(request) == PACKET_READ_NORMAL)\n+\t\tdie(_(\"bundle-uri: unexpected argument: '%s'\"), request->line);\n+\tif (request->status != PACKET_READ_FLUSH)\n+\t\tdie(_(\"bundle-uri: expected flush after arguments\"));\n+\n+\t/* TODO: Implement the communication */\n+\n+\tpacket_writer_flush(&writer);\n+\n+\treturn 0;\n+}\n+\n /**\n * General API for {transport,connect}.c etc.\n */\ndiff --git a/bundle-uri.h b/bundle-uri.h\nindex adce5dc07e2..9f6fc2d75f9 100644\n--- a/bundle-uri.h\n+++ b/bundle-uri.h\n@@ -4,6 +4,7 @@\n #include \"hashmap.h\"\n #include \"strbuf.h\"\n \n+struct packet_reader;\n struct repository;\n struct strbuf;\n struct string_list;\n@@ -97,6 +98,12 @@ int parse_bundle_list_in_config_format(const char *uri,\n */\n int fetch_bundle_uri(struct repository *r, const char *uri);\n \n+/**\n+ * API for serve.c.\n+ */\n+int bundle_uri_advertise(struct repository *r, struct strbuf *value);\n+int bundle_uri_command(struct repository *r, struct packet_reader *request);\n+\n /**\n * General API for {transport,connect}.c etc.\n */\ndiff --git a/serve.c b/serve.c\nindex b3fe9b5126a..f3e0203d2c6 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -8,6 +8,7 @@\n #include \"protocol-caps.h\"\n #include \"serve.h\"\n #include \"upload-pack.h\"\n+#include \"bundle-uri.h\"\n \n static int advertise_sid = -1;\n static int client_hash_algo = GIT_HASH_SHA1;\n@@ -136,6 +137,11 @@ static struct protocol_capability capabilities[] = {\n \t\t.advertise = always_advertise,\n \t\t.command = cap_object_info,\n \t},\n+\t{\n+\t\t.name = \"bundle-uri\",\n+\t\t.advertise = bundle_uri_advertise,\n+\t\t.command = bundle_uri_command,\n+\t},\n };\n \n void protocol_v2_advertise_capabilities(void)\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex 1896f671cb3..f21e5e9d33d 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -13,7 +13,7 @@ test_expect_success 'test capability advertisement' '\n \twrong_algo sha1:sha256\n \twrong_algo sha256:sha1\n \tEOF\n-\tcat >expect <<-EOF &&\n+\tcat >expect.base <<-EOF &&\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n \tls-refs=unborn\n@@ -21,8 +21,11 @@ test_expect_success 'test capability advertisement' '\n \tserver-option\n \tobject-format=$(test_oid algo)\n \tobject-info\n+\tEOF\n+\tcat >expect.trailer <<-EOF &&\n \t0000\n \tEOF\n+\tcat expect.base expect.trailer >expect &&\n \n \tGIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \\\n \t\t--advertise-capabilities >out &&\n@@ -342,4 +345,39 @@ test_expect_success 'basics of object-info' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'test capability advertisement with uploadpack.advertiseBundleURIs' '\n+\ttest_config uploadpack.advertiseBundleURIs true &&\n+\n+\tcat >expect.extra <<-EOF &&\n+\tbundle-uri\n+\tEOF\n+\tcat expect.base \\\n+\t expect.extra \\\n+\t expect.trailer >expect &&\n+\n+\tGIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \\\n+\t\t--advertise-capabilities >out &&\n+\ttest-tool pkt-line unpack actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'basics of bundle-uri: dies if not enabled' '\n+\ttest-tool pkt-line pack >in <<-EOF &&\n+\tcommand=bundle-uri\n+\t0000\n+\tEOF\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid command '\"'\"'bundle-uri'\"'\"'\n+\tEOF\n+\n+\tcat >expect <<-\\EOF &&\n+\tERR serve: invalid command '\"'\"'bundle-uri'\"'\"'\n+\tEOF\n+\n+\ttest_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"455605","messageId":"333180f80904a2a0675bfa65fab62107fa13b2a9.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 08/24] clone: --bundle-uri cannot be combined with --depth","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:26Z","receivedAt":"2022-05-20T18:42:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change added the '--bundle-uri' option, but did not check\nif the --depth parameter was included. Since bundles are not compatible\nwith shallow clones, provide an error message to the user who is\nattempting this combination.\n\nI am leaving this as its own change, separate from the one that\nimplements '--bundle-uri', because this is more of an advisory for the\nuser. There is nothing wrong with bootstrapping with bundles and then\nfetching a shallow clone. However, that is likely going to involve too\nmuch work for the client _and_ the server. The client will download all\nof this bundle information containing the full history of the\nrepository only to ignore most of it. The server will get a shallow\nfetch request, but with a list of haves that might cause a more painful\ncomputation of that shallow pack-file.\n\nRFC-TODO: add a test case for this error message.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex fd1ae82e57b..a9caa5dfed6 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -926,6 +926,11 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\toption_no_checkout = 1;\n \t}\n \n+\tif (bundle_uri) {\n+\t\tif (deepen)\n+\t\t\tdie(_(\"--bundle-uri is incompatible with --depth, --shallow-since, and --shallow-exclude\"));\n+\t}\n+\n \trepo_name = argv[0];\n \n \tpath = get_repo_path(repo_name, &is_bundle);\n-- \ngitgitgadget\n\n"},{"id":"455606","messageId":"00dc67f5188b107d5ddaf8b32f03fa61f515a5ec.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 13/24] bundle-uri: limit recursion depth for bundle lists","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:31Z","receivedAt":"2022-05-20T18:42:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe next change will start allowing us to parse bundle lists that are\ndownloaded from a provided bundle URI. Those lists might point to other\nlists, which could proceed to an arbitrary depth (and even create\ncycles). Restructure fetch_bundle_uri() to have an internal version that\nhas a recursion depth. Compare that to a new max_bundle_uri_depth\nconstant that is twice as high as we expect this depth to be for any\nlegitimate use of bundle list linking.\n\nWe can consider making max_bundle_uri_depth a configurable value if\nthere is demonstrated value in the future.\n\nSigned-off-by: Derrick Stolee \n---\n bundle-uri.c | 21 ++++++++++++++++++++-\n 1 file changed, 20 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex ff215422888..74a26ce6c00 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -308,11 +308,25 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \treturn result;\n }\n \n-int fetch_bundle_uri(struct repository *r, const char *uri)\n+/**\n+ * This limits the recursion on fetch_bundle_uri_internal() when following\n+ * bundle lists.\n+ */\n+static int max_bundle_uri_depth = 4;\n+\n+static int fetch_bundle_uri_internal(struct repository *r,\n+\t\t\t\t const char *uri,\n+\t\t\t\t int depth)\n {\n \tint result = 0;\n \tstruct strbuf filename = STRBUF_INIT;\n \n+\tif (depth >= max_bundle_uri_depth) {\n+\t\twarning(_(\"exceeded bundle URI recursion limit (%d)\"),\n+\t\t\tmax_bundle_uri_depth);\n+\t\treturn -1;\n+\t}\n+\n \tfind_temp_filename(&filename);\n \tif ((result = copy_uri_to_file(uri, filename.buf)))\n \t\tgoto cleanup;\n@@ -335,6 +349,11 @@ cleanup:\n \treturn result;\n }\n \n+int fetch_bundle_uri(struct repository *r, const char *uri)\n+{\n+\treturn fetch_bundle_uri_internal(r, uri, 0);\n+}\n+\n /**\n * General API for {transport,connect}.c etc.\n */\n-- \ngitgitgadget\n\n"},{"id":"455607","messageId":"f14144ffada7bc426ca33d07b810b3ebfbde9fb2.1653072042.git.gitgitgadget@gmail.com","threadId":"57905","inReplyTo":"pull.1234.git.1653072042.gitgitgadget@gmail.com","subject":"[PATCH 18/24] bundle-uri client: add \"git ls-remote-bundle-uri\"","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-20T18:40:36Z","receivedAt":"2022-05-20T18:42:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nTODO: make this a test-helper instead of its own builtin?\n\nAdd a git-ls-remote-bundle-uri command, this is a thin wrapper for\nissuing protocol v2 \"bundle-uri\" commands to a server, and to the\nparsing routines in bundle-uri.c.\n\nSince in the \"git clone\" case we'll have already done the handshake(),\nbut not here, introduce a \"got_advertisement\" state along with\n\"got_remote_heads\". It seems to me that the \"got_remote_heads\" is\nbadly named in the first place, and the whole logic of eagerly getting\nls-refs on handshake() or not could be refactored somewhat, but let's\nnot do that now, and instead just add another self-documenting state\nvariable.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-ls-remote-bundle-uri.txt | 62 +++++++++\n Documentation/git-ls-remote.txt | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/clone.c | 2 +-\n builtin/ls-remote-bundle-uri.c | 71 ++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n t/lib-t5730-protocol-v2-bundle-uri.sh | 147 ++++++++++++++++++++-\n transport.c | 43 ++++--\n transport.h | 6 +-\n 12 files changed, 324 insertions(+), 13 deletions(-)\n create mode 100644 Documentation/git-ls-remote-bundle-uri.txt\n create mode 100644 builtin/ls-remote-bundle-uri.c\n\ndiff --git a/.gitignore b/.gitignore\nindex e81de1063a4..e1312b2795c 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -91,6 +91,7 @@\n /git-log\n /git-ls-files\n /git-ls-remote\n+/git-ls-remote-bundle-uri\n /git-ls-tree\n /git-mailinfo\n /git-mailsplit\ndiff --git a/Documentation/git-ls-remote-bundle-uri.txt b/Documentation/git-ls-remote-bundle-uri.txt\nnew file mode 100644\nindex 00000000000..793d7677f2f\n--- /dev/null\n+++ b/Documentation/git-ls-remote-bundle-uri.txt\n@@ -0,0 +1,62 @@\n+git-ls-remote-bundle-uri(1)\n+===========================\n+\n+NAME\n+----\n+git-ls-remote-bundle-uri - List 'bundle-uri' in a remote repository\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git ls-remote-bundle-uri' [-q |--quiet] [--uri] [--upload-pack=]\n+\t\t\t [[-o | --server-option=]