git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[TOPIC 09/11] Bundle-URI on fetch / resume-able clone

From
Taylor Blau <me@ttaylorr.com>
Date
Sep 20, 2024, 14:22 UTC
Message-ID
<Zu2FCF+BOrshAzxN@nand.local>
In-Reply-To
<Zu2DmS30E0kKug2a@nand.local>

Bundle-URI on fetch / resume-able clone =======================================

(moderator: Toon; notetaker: Taylor)
* Toon: we have bundle-uris, which allow us to download bundles before
	cloning. What would it take to do them on fetch, too?
* Toon: Complex; you might end up downloading a lot of bundles that you
	don't need.
* Toon: there was a bundle token that could be used by the client to
	determine whether or not it needs the bundle. Unfortunately, since you
	don't know what amount of history you're missing, the bundle in and of
	itself may not be sufficient.
* Toon: How do we make bundles more efficient on fetch and aware of what
	the clients do and don't have.
* Johannes: In VFS for Git, the set of bundles is "opinionated" and
	regenerated often (e.g., weekly, daily, hourly, etc.). We don't really
	care about oversharing, because it's fast enough to download those
	static files.
* Toon: That's true for the server, not for a singular client.
* Johannes: right, but it's to protect the server. If you don't care
	about having too-big of a bundle, it's OK because you can just fetch
	the last hour and then see what you still need on top of that.
* Johannes: there is logic to determine what you need to download based
	on the time that you last fetched, etc.
* Elijah: are they thin packs? Yes.
* Patrick: the heuristics that we have to advertise what kind of bundles
	are on the server is not sufficient enough to determine which bundles
	the client may or may not want to download. The client must guess.
* Jonathan: ISTM that the property list for the bundles as they are
	today should probably be considered a starting point (as in "part of a
	feature"). The heuristics need to be extended.
* Jonathan: at Google we use packfile-URIs, and one of the advantages of
	how that works is that since the packfiles are advertised after the
	normal fetch negotiation, you get a curated list. For bundle-uris,
	need something analogous to the fetch negotiation for the client to be
	able to make a similar decision of which bundle to download.
* brian: extending the feature to store timestamps, then you could store
	the last fetch on the system would inform some better selection of
	which bundles to clone down during a fetch.
* brian: Would make a big difference for folks in environments which do
	not benefit from reliable Internet connections.
* Toon: Sure... but you have to put a lot of pressure to keep those
	bundles up-to-date on the server side.
* Jonathan: one of the advantages of bundles over packs is that they
	have information about the references. One potential property could be
	"here's the length of this header" and then having the client download
	it to examine whether or not it wants such a bundle.
* Patrick: probably would want to cache those on the client so that
	we're able to avoid re-downloading these every single time.
* Toon: resumability
* Jonathan: protocol supports it, just hasn't been implemented. Someone
	needs to just get it done. :)
* Beyond that, the main complication is how you store the state and what
	the UX is for resuming. But a person implementing it can figure those
	things out.
* Brian: broken-proxy situations require some configurability of where
	your bundles can be downloaded from.
Previous: Patrick SteinhardtNext: Taylor Blau
Message 33 of 38 in “Notes from the Git Contributor's Summit, 2024”
  1. Taylor BlauSep 20, 2024
  2. 01/11 RustTaylor Blau, Sep 20, 2024
  3. rsbecker@nexbridge.comSep 20, 2024
  4. Sean AllredSep 23, 2024
  5. rsbecker@nexbridge.comSep 23, 2024
  6. Phillip WoodSep 24, 2024
  7. rsbecker@nexbridge.comSep 24, 2024
  8. Sean AllredSep 27, 2024
  9. rsbecker@nexbridge.comSep 27, 2024
  10. rsbecker@nexbridge.comSep 27, 2024
  11. 02/11 Top-level lib/ directoryTaylor Blau, Sep 20, 2024
  12. 03/11 Structured Error HandlingTaylor Blau, Sep 20, 2024
  13. 04/11 Platform Support PolicyTaylor Blau, Sep 20, 2024
  14. 05/11 : SHA 256 / Git 3.0Taylor Blau, Sep 20, 2024
  15. Junio C HamanoSep 20, 2024
  16. 06/11 Git and Software Freedom ConservancyTaylor Blau, Sep 20, 2024
  17. 07/11 New Contributors and DiscordTaylor Blau, Sep 20, 2024
  18. Junio C HamanoSep 20, 2024
  19. Kousik SanagavarapuSep 21, 2024
  20. Junio C HamanoSep 22, 2024
  21. Junio C HamanoSep 22, 2024
  22. Konstantin RyabitsevSep 23, 2024
  23. Junio C HamanoSep 23, 2024
  24. Konstantin RyabitsevSep 24, 2024
  25. Junio C HamanoSep 24, 2024
  26. Konstantin RyabitsevSep 24, 2024
  27. Phillip WoodSep 27, 2024
  28. Junio C HamanoSep 27, 2024
  29. Phillip WoodOct 1, 2024
  30. 08/11 Modern Build SystemsTaylor Blau, Sep 20, 2024
  31. Eli SchwartzSep 23, 2024
  32. Patrick SteinhardtSep 24, 2024
  33. 09/11 Bundle-URI on fetch / resume-able cloneTaylor Blau, Sep 20, 2024
  34. 10/11 Project TrackingTaylor Blau, Sep 20, 2024
  35. Junio C HamanoSep 20, 2024
  36. Junio C HamanoSep 20, 2024
  37. Phillip WoodSep 23, 2024
  38. 11/11 git-scm.com state of the siteTaylor Blau, Sep 20, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.