git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 1/6] docs: document bundle URI standard

From
Derrick Stolee <derrickstolee@github.com>
Date
Jul 22, 2022, 13:15 UTC
Message-ID
<5a6a687d-4543-f375-13dd-b60b37ddce18@github.com>
In-Reply-To
<YtnHeUUnaLn6mSYK@google.com>
On 7/21/2022 5:39 PM, Josh Steadmon wrote:
> On 2022.06.29 20:40, Derrick Stolee via GitGitGadget wrote:
>> From: Derrick Stolee <derrickstolee@github.com>
Show 8 quoted lines
>> +bundle.heuristic::
>> +	If this string-valued key exists, then the bundle list is designed to
>> +	work well with incremental `git fetch` commands. The heuristic signals
>> +	that there are additional keys available for each bundle that help
>> +	determine which subset of bundles the client should download.
> 
> To be clear, the values of `bundle.heuristic` are keys underneath the
> following `bundle.<id>` segments?

At the moment, the only planned heuristic is associated with a single key (bundle.<id>.creationToken) but future heuristics could be more complicated. The general idea is to say "these bundles were organized with this heuristic in mind, so either take all of them (ignoring the heuristic) or apply a specific algorithm to download a subset at a time." The creationToken heuristic uses the total order on the bundles to do a greedy algorithm for downloading the most-recent bundles until the required commits are all satisfied.

Show 45 quoted lines
>> +Related Work: Packfile URIs
>> +---------------------------
> 
> Thanks for including this section; my very first question before I even
> started reading the doc was "How does this compare to Packfile URIs?".
> 
> 
>> +
>> +The Git protocol already has a capability where the Git server can list
>> +a set of URLs along with the packfile response when serving a client
>> +request. The client is then expected to download the packfiles at those
>> +locations in order to have a complete understanding of the response.
>> +
>> +This mechanism is used by the Gerrit server (implemented with JGit) and
>> +has been effective at reducing CPU load and improving user performance for
>> +clones.
>> +
>> +A major downside to this mechanism is that the origin server needs to know
>> +_exactly_ what is in those packfiles, and the packfiles need to be available
>> +to the user for some time after the server has responded. This coupling
>> +between the origin and the packfile data is difficult to manage.
> 
> It was not immediately clear to me why bundle URIs would avoid these two
> downsides, but after thinking about it and discussing it, I believe this
> is the reasoning (please correct me if I'm wrong):
> 
> Bundle URIs are intended to be fully processed before negotiation
> happens, so the server can rely solely on the client's reported Haves /
> Wants to determine its response, as usual, and therefore doesn't need to
> know the bundle contents. Packfile URIs are not processed by the client
> before negotiation, so the server needs to be aware of the packfile
> contents in order to determine which additional objects to send, and
> can't rely solely on the Haves/Wants from the client.
> 
> If that's accurate, then it seems fine that when the bundle URI is
> provided by the user on the command-line, Git can download and process
> the bundle before even attempting to contact the server. But in future
> series, when we add the ability for the server to provide a URI via a
> capability, the client will have to pause after seeing the server's
> advertised URI, fetch and process the bundle, and then proceed with a
> fetch command. This seems fine assuming the bundles can be handled in a
> reasonable amount of time. And even if the connection breaks before the
> fetch command can be issued, presumably the client would not need to
> attempt to re-download the bundle a second time when the client makes a
> second connection to re-attempt the fetch.

After the server has advertised the bundle URIs, the client should consider disconnecting while downloading bundles, then reconnect to the server after that is complete.

For stateless connections (https) we need to reconnect every time, so this isn't a problem.

I anticipate that in the long-term view of this feature, the server advertised bundle URIs will usually point to an independent bundle list at a static URI. In that situation, we can store the chosen bundle list URI in Git config and check that bundle list for new data before even connecting to the origin Git server. We would only "rediscover" the URIs when either the bundle URI returns 404 and/or the origin server lists new URIs.

Of course, there is the option that the origin Git server wants to track and advertise the individual bundles directly, in which case the client can't do this opportunistic bundle pre-check.

Show 8 quoted lines
> For the point about the packfiles needing to be available for some time,
> this makes sense because the server's response is processed by the
> client before the referenced packfile is downloaded, but the response is
> incomplete without the packfile. But with bundle URIs, the server's
> response doesn't depend on whether or not the client actually processed
> the referenced bundle, only on what was negotiated. So the server's
> response is still valid and useful even if the client is unable to get
> the bundle.

You're pointing out the critical feature: the bundles are completely optional. If the client fails to download the bundles due to something like this race, then it will just negotiate for the origin Git server to provide the data.

What's perhaps not entirely fair about this point is that the bundle provider should keep this race in mind between serving a bundle list and having the client request the associated bundles. Something like a 24-hour window between updating the bundle list and deleting an unreferenced bundle file should satisfy the vast majority of clients.

Thanks for the close look. While I removed them from the context, I've made note of your smaller comments on things like typos and will fix them in v3.

Thanks, -Stolee

Previous: Josh SteadmonNext: Derrick Stolee
Message 39 of 64 in “bundle URIs: design doc and initial git fetch --bundle-uri implementation”
  1. 0/6 bundle URIs: design doc and initial git fetch --bundle-uri implementationDerrick Stolee via GitGitGadget, Jun 6, 2022
  2. 6/6 fetch: add 'refs/bundle/' to log.excludeDecorationDerrick Stolee via GitGitGadget, Jun 6, 2022
  3. 4/6 fetch: add --bundle-uri optionDerrick Stolee via GitGitGadget, Jun 6, 2022
  4. 5/6 bundle-uri: add support for http(s):// and file://Derrick Stolee via GitGitGadget, Jun 6, 2022
  5. 3/6 bundle-uri: create basic file-copy logicDerrick Stolee via GitGitGadget, Jun 6, 2022
  6. 1/6 docs: document bundle URI standardDerrick Stolee via GitGitGadget, Jun 6, 2022
  7. Junio C HamanoJun 6, 2022
  8. Derrick StoleeJun 8, 2022
  9. Junio C HamanoJun 8, 2022
  10. Junio C HamanoJun 8, 2022
  11. Junio C HamanoJun 8, 2022
  12. Derrick StoleeJun 8, 2022
  13. Junio C HamanoJun 7, 2022
  14. Derrick StoleeJun 8, 2022
  15. Junio C HamanoJun 8, 2022
  16. Derrick StoleeJun 9, 2022
  17. Junio C HamanoJun 9, 2022
  18. Ævar Arnfjörð BjarmasonJun 9, 2022
  19. Derrick StoleeJun 9, 2022
  20. Junio C HamanoJun 9, 2022
  21. Derrick StoleeJun 21, 2022
  22. Junio C HamanoJun 21, 2022
  23. Derrick StoleeJun 21, 2022
  24. Junio C HamanoJun 21, 2022
  25. 2/6 remote-curl: add 'get' capabilityDerrick Stolee via GitGitGadget, Jun 6, 2022
  26. Junio C HamanoJul 21, 2022
  27. 0/6 bundle URIs: design doc and initial git fetch --bundle-uri implementationDerrick Stolee via GitGitGadget, Jun 29, 2022
  28. 2/6 remote-curl: add 'get' capabilityDerrick Stolee via GitGitGadget, Jun 29, 2022
  29. Josh SteadmonJul 21, 2022
  30. 3/6 bundle-uri: create basic file-copy logicDerrick Stolee via GitGitGadget, Jun 29, 2022
  31. Josh SteadmonJul 21, 2022
  32. Derrick StoleeJul 22, 2022
  33. 1/6 docs: document bundle URI standardDerrick Stolee via GitGitGadget, Jun 29, 2022
  34. SZEDER GáborJul 18, 2022
  35. Matthew John CheethamJul 21, 2022
  36. Derrick StoleeJul 22, 2022
  37. Derrick StoleeJul 22, 2022
  38. Josh SteadmonJul 21, 2022
  39. Derrick StoleeJul 22, 2022
  40. Derrick StoleeJul 22, 2022
  41. 4/6 fetch: add --bundle-uri optionDerrick Stolee via GitGitGadget, Jun 29, 2022
  42. 5/6 bundle-uri: add support for http(s):// and file://Derrick Stolee via GitGitGadget, Jun 29, 2022
  43. 6/6 fetch: add 'refs/bundle/' to log.excludeDecorationDerrick Stolee via GitGitGadget, Jun 29, 2022
  44. Josh SteadmonJul 21, 2022
  45. Derrick StoleeJul 22, 2022
  46. Josh SteadmonJul 21, 2022
  47. Junio C HamanoJul 21, 2022
  48. 0/2 bundle URIs: design doc and initial git fetch --bundle-uri implementationDerrick Stolee via GitGitGadget, Jul 25, 2022
  49. 2/2 bundle-uri: add example bundle organizationDerrick Stolee via GitGitGadget, Jul 25, 2022
  50. Matthew John CheethamAug 4, 2022
  51. Derrick StoleeAug 4, 2022
  52. Ævar Arnfjörð BjarmasonAug 4, 2022
  53. Derrick StoleeAug 5, 2022
  54. 1/2 docs: document bundle URI standardDerrick Stolee via GitGitGadget, Jul 25, 2022
  55. 1/2 docs: document bundle URI standardtenglong.tl, Jul 28, 2022
  56. Derrick StoleeAug 1, 2022
  57. Josh SteadmonJul 25, 2022
  58. 0/2 bundle URIs: design docDerrick Stolee via GitGitGadget, Aug 9, 2022
  59. 1/2 docs: document bundle URI standardDerrick Stolee via GitGitGadget, Aug 9, 2022
  60. Philip OakleyOct 4, 2022
  61. 2/2 bundle-uri: add example bundle organizationDerrick Stolee via GitGitGadget, Aug 9, 2022
  62. Phillip WoodAug 9, 2022
  63. Derrick StoleeAug 9, 2022
  64. Phillip WoodAug 11, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.