git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v4 0/2] bundle URIs: design doc

From
PWPhillip Wood <phillip.wood123@gmail.com>
Date
Aug 11, 2022, 15:42 UTC
Message-ID
<9524d32c-6726-d556-730d-2324b300dc72@gmail.com>
In-Reply-To
<bce18d3e-ac37-7c7a-d411-d0aad87a8f68@github.com>
Hi Stolee
On 09/08/2022 16:50, Derrick Stolee wrote:
Show 42 quoted lines
>> One small query - the document mentions CI farms as benefiting from this work
>> but my impression is that those commonly use shallow clones which are (quite
>> reasonably) not supported in this proposal.
> 
> There are two different kinds of CI farms.
> 
> The most common one is a SaaS CI system that provides machines on-demand,
> but each run starts from some "clean" state. For example, GitHub Actions
> runs CI builds of the Git project across a number of platforms. These
> machines need the source at HEAD, but do not need the full history. Further,
> they will erase the repository entirely at the end of the build, never
> fetching from those repositories. Thus, a shallow clone makes sense to
> minimize the data transfer. Bundles don't make sense here for multiple
> reasons, including that bundles must be closed under reachability and do
> not work for representing a shallow clone (see [1]). The other reason is
> that CI builds typically are triggered immediately after the commit appears
> on the origin Git server, so there is no time for a bundle provider to
> create a bundle representing that shallow clone.
> 
> [1] https://github.com/git/git/blob/c50926e1f48891e2671e1830dbcd2912a4563450/Documentation/technical/bundle-format.txt#L65-L69
> 
> The less common one is a private build farm. These machines are long-lived
> and controlled by the repository owner. They come pre-loaded with all of
> the software needed to build the repository. The best practice in this
> type of build farm is to keep a full clone of the repository in a well-
> known location and use incremental fetches to update the client repositories
> to download the commit necessary for the build. This type of build farm is
> typically self-hosted, but could also be hosted by a cloud provider. The
> bundle URI design allows ways to quickly bootstrap new build machines using
> a bundle provider (probably co-located with the build machines) as well as
> improving fetch times by creating frequent incremental bundles. The new
> commit being built is unlikely to exist immediately in the bundles, but it
> is unlikely to be too far ahead of any of the bundles.
> 
> While private build farms are less common, they do become necessary for
> large projects. Engineering teams that have the resources to self-host a
> build farm are likely to also have the resources to self-host a bundle
> server. They may not have the connections or desire to advertise those
> bundle server URIs from the origin Git server.
> 
> I hope this helps clarify my perspective as to why build farms using long-
> lived copies of the repository could take advantage of bundle URIs.
Yes it does, thanks for taking the time to explain that.
Phillip
> Thanks,
> -Stolee
Previous: Derrick Stolee
Message 64 of 64 in “bundle URIs: design doc and initial git fetch --bundle-uri implementation”
  1. 0/6 bundle URIs: design doc and initial git fetch --bundle-uri implementationDerrick Stolee via GitGitGadget, Jun 6, 2022
  2. 6/6 fetch: add 'refs/bundle/' to log.excludeDecorationDerrick Stolee via GitGitGadget, Jun 6, 2022
  3. 4/6 fetch: add --bundle-uri optionDerrick Stolee via GitGitGadget, Jun 6, 2022
  4. 5/6 bundle-uri: add support for http(s):// and file://Derrick Stolee via GitGitGadget, Jun 6, 2022
  5. 3/6 bundle-uri: create basic file-copy logicDerrick Stolee via GitGitGadget, Jun 6, 2022
  6. 1/6 docs: document bundle URI standardDerrick Stolee via GitGitGadget, Jun 6, 2022
  7. Junio C HamanoJun 6, 2022
  8. Derrick StoleeJun 8, 2022
  9. Junio C HamanoJun 8, 2022
  10. Junio C HamanoJun 8, 2022
  11. Junio C HamanoJun 8, 2022
  12. Derrick StoleeJun 8, 2022
  13. Junio C HamanoJun 7, 2022
  14. Derrick StoleeJun 8, 2022
  15. Junio C HamanoJun 8, 2022
  16. Derrick StoleeJun 9, 2022
  17. Junio C HamanoJun 9, 2022
  18. Ævar Arnfjörð BjarmasonJun 9, 2022
  19. Derrick StoleeJun 9, 2022
  20. Junio C HamanoJun 9, 2022
  21. Derrick StoleeJun 21, 2022
  22. Junio C HamanoJun 21, 2022
  23. Derrick StoleeJun 21, 2022
  24. Junio C HamanoJun 21, 2022
  25. 2/6 remote-curl: add 'get' capabilityDerrick Stolee via GitGitGadget, Jun 6, 2022
  26. Junio C HamanoJul 21, 2022
  27. 0/6 bundle URIs: design doc and initial git fetch --bundle-uri implementationDerrick Stolee via GitGitGadget, Jun 29, 2022
  28. 2/6 remote-curl: add 'get' capabilityDerrick Stolee via GitGitGadget, Jun 29, 2022
  29. Josh SteadmonJul 21, 2022
  30. 3/6 bundle-uri: create basic file-copy logicDerrick Stolee via GitGitGadget, Jun 29, 2022
  31. Josh SteadmonJul 21, 2022
  32. Derrick StoleeJul 22, 2022
  33. 1/6 docs: document bundle URI standardDerrick Stolee via GitGitGadget, Jun 29, 2022
  34. SZEDER GáborJul 18, 2022
  35. Matthew John CheethamJul 21, 2022
  36. Derrick StoleeJul 22, 2022
  37. Derrick StoleeJul 22, 2022
  38. Josh SteadmonJul 21, 2022
  39. Derrick StoleeJul 22, 2022
  40. Derrick StoleeJul 22, 2022
  41. 4/6 fetch: add --bundle-uri optionDerrick Stolee via GitGitGadget, Jun 29, 2022
  42. 5/6 bundle-uri: add support for http(s):// and file://Derrick Stolee via GitGitGadget, Jun 29, 2022
  43. 6/6 fetch: add 'refs/bundle/' to log.excludeDecorationDerrick Stolee via GitGitGadget, Jun 29, 2022
  44. Josh SteadmonJul 21, 2022
  45. Derrick StoleeJul 22, 2022
  46. Josh SteadmonJul 21, 2022
  47. Junio C HamanoJul 21, 2022
  48. 0/2 bundle URIs: design doc and initial git fetch --bundle-uri implementationDerrick Stolee via GitGitGadget, Jul 25, 2022
  49. 2/2 bundle-uri: add example bundle organizationDerrick Stolee via GitGitGadget, Jul 25, 2022
  50. Matthew John CheethamAug 4, 2022
  51. Derrick StoleeAug 4, 2022
  52. Ævar Arnfjörð BjarmasonAug 4, 2022
  53. Derrick StoleeAug 5, 2022
  54. 1/2 docs: document bundle URI standardDerrick Stolee via GitGitGadget, Jul 25, 2022
  55. 1/2 docs: document bundle URI standardtenglong.tl, Jul 28, 2022
  56. Derrick StoleeAug 1, 2022
  57. Josh SteadmonJul 25, 2022
  58. 0/2 bundle URIs: design docDerrick Stolee via GitGitGadget, Aug 9, 2022
  59. 1/2 docs: document bundle URI standardDerrick Stolee via GitGitGadget, Aug 9, 2022
  60. Philip OakleyOct 4, 2022
  61. 2/2 bundle-uri: add example bundle organizationDerrick Stolee via GitGitGadget, Aug 9, 2022
  62. Phillip WoodAug 9, 2022
  63. Derrick StoleeAug 9, 2022
  64. Phillip WoodAug 11, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.