git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 1/6] docs: document bundle URI standard

From
Matthew John Cheetham <mjcheetham@outlook.com>
Date
Jul 21, 2022, 12:09 UTC
Message-ID
<AS8PR03MB8689A38CDA60565FED96EC99C0919@AS8PR03MB8689.eurprd03.prod.outlook.com>
In-Reply-To
<d444042dc4dcc1f9b218ca851fcf603a3afce92f.1656535245.git.gitgitgadget@gmail.com>
I had a few questions and suggestions; below.

On 2022-06-29 21:40, Derrick Stolee via GitGitGadget wrote:> +Assuming a `200 OK` response from the server, the content at the URL is

Show 8 quoted lines
> +inspected. First, Git attempts to parse the file as a bundle file of
> +version 2 or higher. If the file is not a bundle, then the file is parsed
> +as a plain-text file using Git's config parser. The key-value pairs in
> +that config file are expected to describe a list of bundle URIs. If
> +neither of these parse attempts succeed, then Git will report an error to
> +the user that the bundle URI provided erroneous data.
> +
> +Any other data provided by the server is considered erroneous.

I wonder if it may be worth considering adding an optional server requirement ("MAY" not "MUST") to provide a `Content-Type` header indicating if the response is a bundle list (or bundle) to skip straight to parsing the correct type of file? Eg "application/x-git-bundle-list"?

Even the simplest of content servers should be able to set Content-Type. If not falling back to 'try parse bundle else try parse bundle list' is still OK.

Show 8 quoted lines
> +bundle.mode::
> +	(Required) This value has one of two values: `all` and `any`. When `all`
> +	is specified, then the client should expect to need all of the listed
> +	bundle URIs that match their repository's requirements. When `any` is
> +	specified, then the client should expect that any one of the bundle URIs
> +	that match their repository's requirements will suffice. Typically, the
> +	`any` option is used to list a number of different bundle servers
> +	located in different geographies.

Do you forsee any future where we'd want or need to specify 'sets' of bundles where "all" _of_ "any" particular set would be required? Eg. there are 3 sets of bundles (A, B, C), and the client would need to download all bundles belonging to any of A, B, or C? Where ABC would be different geo-distributed sets?

I guess what I'm getting at here is with this design (which I appreciate is intentionally flexible), there are several different ways a server could direct a client to bundles stored in a nearby geography:

1. Serve an "all" bundle list with geo-located bundle URIs?
2. Serve a global "any" bundle list with each single bundle in a 
different geography (specified by `location`)
3. Serve a single bundle (not a list) with a different

Are any of these going to be preferred over another for potential client optimisations?

Show 9 quoted lines
> +
> +bundle.heuristic::
> +	If this string-valued key exists, then the bundle list is designed to
> +	work well with incremental `git fetch` commands. The heuristic signals
> +	that there are additional keys available for each bundle that help
> +	determine which subset of bundles the client should download.
> +
> +The remaining keys include an `<id>` segment which is a server-designated
> +name for each available bundle.
Case-sensitive ID? A-Za-z0-9 only? "Same as Git config rules"?
Show 6 quoted lines
> +bundle.<id>.location::
> +	This string value advertises a real-world location from where the bundle
> +	URI is served. This can be used to present the user with an option for
> +	which bundle URI to use or simply as an informative indicator of which
> +	bundle URI was selected by Git. This is only valuable when
> +	`bundle.mode` is `any`.

I assume `location` is just an opaque string that is just used for info or display purposes? Does it make sense for other 'display' type strings like 'name' or 'message'?

Show 26 quoted lines
> +Here is an example bundle list using the Git config format:
> +
> +```
> +[bundle]
> +	version = 1
> +	mode = all
> +	heuristic = creationToken
> +
> +[bundle "2022-02-09-1644442601-daily"]
> +	uri = https://bundles.example.com/git/git/2022-02-09-1644442601-daily.bundle
> +	timestamp = 1644442601
> +
> +[bundle "2022-02-02-1643842562"]
> +	uri = https://bundles.example.com/git/git/2022-02-02-1643842562.bundle
> +	timestamp = 1643842562
> +
> +[bundle "2022-02-09-1644442631-daily-blobless"]
> +	uri = 2022-02-09-1644442631-daily-blobless.bundle
> +	timestamp = 1644442631
> +	filter = blob:none
> +
> +[bundle "2022-02-02-1643842568-blobless"]
> +	uri = /git/git/2022-02-02-1643842568-blobless.bundle
> +	timestamp = 1643842568
> +	filter = blob:none
> +```

Do you mean to use `creationToken` in these examples rather than `timestamp`?

> +Advertising Bundle URIs
> +-----------------------
> +
...
> +The client could choose an arbitrary bundle URI as an option _or_ select
> +the URI with best performance by some exploratory checks. It is up to the
> +bundle provider to decide if having multiple URIs is preferable to a
> +single URI that is geodistributed through server-side infrastructure.

Would it make sense for the client to pick the first bundle URI rather than an arbitrary one? The server could use information about the request (origin IP/geography) to provide a sorted list of URIs by physical distance to the client.

I guess if arbitrary is 'random' then this provides some client-side load balancing over multiple potential servers too. Interested in your thoughts behind what would be 'best practice' for a bundle server here.

> +If the bundle provider does not provide a heuristic, then the client
> +should attempt to inspect the bundle headers before downloading the full
> +bundle data in case the bundle tips already exist in the client
> +repository.

Would this default behaviour also be considered another explicit heurisitic option? For example: `bundle.heuristic=default` or `inspect`. Is it ever likely that the default behaviour would change?

> +* The client receives a response other than `200 OK` (such as `404 Not Found`,
> +  `401 Not Authorized`, or `500 Internal Server Error`). The client should
> +  use the `credential.helper` to attempt authentication after the first
> +  `401 Not Authorized` response, but a second such response is a failure.

I'd probably say a 500 is not solvable with a different set of credentials, but potentially a retry or just `die`. Do we attempt to `credential_fill()` with anything other than a 401 (or maybe 403 or 404) elsewhere in Git?

Show 10 quoted lines
> +* A bundle download during a `git fetch` contains objects already in the
> +  object database. This is probably unavoidable if we are using bundles
> +  for fetches, since the client will almost always be slightly ahead of
> +  the bundle servers after performing its "catch-up" fetch to the remote
> +  server. This extra work is most wasteful when the client is fetching
> +  much more frequently than the server is computing bundles, such as if
> +  the client is using hourly prefetches with background maintenance, but
> +  the server is computing bundles weekly. For this reason, the client
> +  should not use bundle URIs for fetch unless the server has explicitly
> +  recommended it through the `bundle.flags = forFetch` value.

`bundle.flags` is not mentioned elsewhere in this document. Might be worth including this, and possible values, with the other key definitions above.

Show 41 quoted lines
> +Implementation Plan
> +-------------------
> +
> +This design document is being submitted on its own as an aspirational
> +document, with the goal of implementing all of the mentioned client
> +features over the course of several patch series. Here is a potential
> +outline for submitting these features:
> +
> +1. Integrate bundle URIs into `git clone` with a `--bundle-uri` option.
> +   This will include a new `git fetch --bundle-uri` mode for use as the
> +   implementation underneath `git clone`. The initial version here will
> +   expect a single bundle at the given URI.
> +
> +2. Implement the ability to parse a bundle list from a bundle URI and
> +   update the `git fetch --bundle-uri` logic to properly distinguish
> +   between `bundle.mode` options. Specifically design the feature so
> +   that the config format parsing feeds a list of key-value pairs into the
> +   bundle list logic.
> +
> +3. Create the `bundle-uri` protocol v2 verb so Git servers can advertise
> +   bundle URIs using the key-value pairs. Plug into the existing key-value
> +   input to the bundle list logic. Allow `git clone` to discover these
> +   bundle URIs and bootstrap the client repository from the bundle data.
> +   (This choice is an opt-in via a config option and a command-line
> +   option.)
> +
> +4. Allow the client to understand the `bundle.flag=forFetch` configuration
> +   and the `bundle.<id>.creationToken` heuristic. When `git clone`
> +   discovers a bundle URI with `bundle.flag=forFetch`, it configures the
> +   client repository to check that bundle URI during later `git fetch <remote>`
> +   commands.
> +
> +5. Allow clients to discover bundle URIs during `git fetch` and configure
> +   a bundle URI for later fetches if `bundle.flag=forFetch`.
> +
> +6. Implement the "inspect headers" heuristic to reduce data downloads when
> +   the `bundle.<id>.creationToken` heuristic is not available.
> +
> +As these features are reviewed, this plan might be updated. We also expect
> +that new designs will be discovered and implemented as this feature
> +matures and becomes used in real-world scenarios.
This plan seems logical to me at least! :-)

-- Thanks, Matthew

Previous: SZEDER GáborNext: Derrick Stolee
Message 35 of 64 in “bundle URIs: design doc and initial git fetch --bundle-uri implementation”
  1. 0/6 bundle URIs: design doc and initial git fetch --bundle-uri implementationDerrick Stolee via GitGitGadget, Jun 6, 2022
  2. 6/6 fetch: add 'refs/bundle/' to log.excludeDecorationDerrick Stolee via GitGitGadget, Jun 6, 2022
  3. 4/6 fetch: add --bundle-uri optionDerrick Stolee via GitGitGadget, Jun 6, 2022
  4. 5/6 bundle-uri: add support for http(s):// and file://Derrick Stolee via GitGitGadget, Jun 6, 2022
  5. 3/6 bundle-uri: create basic file-copy logicDerrick Stolee via GitGitGadget, Jun 6, 2022
  6. 1/6 docs: document bundle URI standardDerrick Stolee via GitGitGadget, Jun 6, 2022
  7. Junio C HamanoJun 6, 2022
  8. Derrick StoleeJun 8, 2022
  9. Junio C HamanoJun 8, 2022
  10. Junio C HamanoJun 8, 2022
  11. Junio C HamanoJun 8, 2022
  12. Derrick StoleeJun 8, 2022
  13. Junio C HamanoJun 7, 2022
  14. Derrick StoleeJun 8, 2022
  15. Junio C HamanoJun 8, 2022
  16. Derrick StoleeJun 9, 2022
  17. Junio C HamanoJun 9, 2022
  18. Ævar Arnfjörð BjarmasonJun 9, 2022
  19. Derrick StoleeJun 9, 2022
  20. Junio C HamanoJun 9, 2022
  21. Derrick StoleeJun 21, 2022
  22. Junio C HamanoJun 21, 2022
  23. Derrick StoleeJun 21, 2022
  24. Junio C HamanoJun 21, 2022
  25. 2/6 remote-curl: add 'get' capabilityDerrick Stolee via GitGitGadget, Jun 6, 2022
  26. Junio C HamanoJul 21, 2022
  27. 0/6 bundle URIs: design doc and initial git fetch --bundle-uri implementationDerrick Stolee via GitGitGadget, Jun 29, 2022
  28. 2/6 remote-curl: add 'get' capabilityDerrick Stolee via GitGitGadget, Jun 29, 2022
  29. Josh SteadmonJul 21, 2022
  30. 3/6 bundle-uri: create basic file-copy logicDerrick Stolee via GitGitGadget, Jun 29, 2022
  31. Josh SteadmonJul 21, 2022
  32. Derrick StoleeJul 22, 2022
  33. 1/6 docs: document bundle URI standardDerrick Stolee via GitGitGadget, Jun 29, 2022
  34. SZEDER GáborJul 18, 2022
  35. Matthew John CheethamJul 21, 2022
  36. Derrick StoleeJul 22, 2022
  37. Derrick StoleeJul 22, 2022
  38. Josh SteadmonJul 21, 2022
  39. Derrick StoleeJul 22, 2022
  40. Derrick StoleeJul 22, 2022
  41. 4/6 fetch: add --bundle-uri optionDerrick Stolee via GitGitGadget, Jun 29, 2022
  42. 5/6 bundle-uri: add support for http(s):// and file://Derrick Stolee via GitGitGadget, Jun 29, 2022
  43. 6/6 fetch: add 'refs/bundle/' to log.excludeDecorationDerrick Stolee via GitGitGadget, Jun 29, 2022
  44. Josh SteadmonJul 21, 2022
  45. Derrick StoleeJul 22, 2022
  46. Josh SteadmonJul 21, 2022
  47. Junio C HamanoJul 21, 2022
  48. 0/2 bundle URIs: design doc and initial git fetch --bundle-uri implementationDerrick Stolee via GitGitGadget, Jul 25, 2022
  49. 2/2 bundle-uri: add example bundle organizationDerrick Stolee via GitGitGadget, Jul 25, 2022
  50. Matthew John CheethamAug 4, 2022
  51. Derrick StoleeAug 4, 2022
  52. Ævar Arnfjörð BjarmasonAug 4, 2022
  53. Derrick StoleeAug 5, 2022
  54. 1/2 docs: document bundle URI standardDerrick Stolee via GitGitGadget, Jul 25, 2022
  55. 1/2 docs: document bundle URI standardtenglong.tl, Jul 28, 2022
  56. Derrick StoleeAug 1, 2022
  57. Josh SteadmonJul 25, 2022
  58. 0/2 bundle URIs: design docDerrick Stolee via GitGitGadget, Aug 9, 2022
  59. 1/2 docs: document bundle URI standardDerrick Stolee via GitGitGadget, Aug 9, 2022
  60. Philip OakleyOct 4, 2022
  61. 2/2 bundle-uri: add example bundle organizationDerrick Stolee via GitGitGadget, Aug 9, 2022
  62. Phillip WoodAug 9, 2022
  63. Derrick StoleeAug 9, 2022
  64. Phillip WoodAug 11, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.