git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 4/4] bundle v3: the beginning

From
Christian Couder <christian.couder@gmail.com>
Date
May 20, 2016, 12:39 UTC
Message-ID
<CAP8UFD1xqRMFE2Wzntu=XevCyj+acGLEO-cTq1fqn+NMe3x0vg@mail.gmail.com>
In-Reply-To
<1456950761-19759-5-git-send-email-gitster@pobox.com>

I am responding to this 2+ month old email because I am investigating adding an alternate object store at the same level as loose and packed objects. This alternate object store could be used for large files. I am working on this for GitLab. (Yeah, I am working, as a freelance, for both Booking.com and GitLab these days.)

On Wed, Mar 2, 2016 at 9:32 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 40 quoted lines
> The bundle v3 format introduces an ability to have the bundle header
> (which describes what references in the bundled history can be
> fetched, and what objects the receiving repository must have in
> order to unbundle it successfully) in one file, and the bundled pack
> stream data in a separate file.
>
> A v3 bundle file begins with a line with "# v3 git bundle", followed
> by zero or more "extended header" lines, and an empty line, finally
> followed by the list of prerequisites and references in the same
> format as v2 bundle.  If it uses the "split bundle" feature, there
> is a "data: $URL" extended header line, and nothing follows the list
> of prerequisites and references.  Also, "sha1: " extended header
> line may exist to help validating that the pack stream data matches
> the bundle header.
>
> A typical expected use of a split bundle is to help initial clone
> that involves a huge data transfer, and would go like this:
>
>  - Any repository people would clone and fetch from would regularly
>    be repacked, and it is expected that there would be a packfile
>    without prerequisites that holds all (or at least most) of the
>    history of it (call it pack-$name.pack).
>
>  - After arranging that packfile to be downloadable over popular
>    transfer methods used for serving static files (such as HTTP or
>    HTTPS) that are easily resumable as $URL/pack-$name.pack, a v3
>    bundle file (call it $name.bndl) can be prepared with an extended
>    header "data: $URL/pack-$name.pack" to point at the download
>    location for the packfile, and be served at "$URL/$name.bndl".
>
>  - An updated Git client, when trying to "git clone" from such a
>    repository, may be redirected to $URL/$name.bndl", which would be
>    a tiny text file (when split bundle feature is used).
>
>  - The client would then inspect the downloaded $name.bndl, learn
>    that the corresponding packfile exists at $URL/pack-$name.pack,
>    and downloads it as pack-$name.pack, until the download succeeds.
>    This can easily be done with "wget --continue" equivalent over an
>    unreliable link.  The checksum recorded on the "sha1: " header
>    line is expected to be used by this downloader (not written yet).

I wonder if this mechanism could also be used or extended to clone and fetch an alternate object database.

In [1], [2] and [3], and this was also discussed during the Contributor Summit last month, Peff says that he started working on alternate object database support a long time ago, and that the hard part is a protocol extension to tell remotes that you can access some objects in a different way.

If a Git client would download a "$name.bndl" v3 bundle file that would have a "data: $URL/alt-odb-$name.odb" extended header, the Git client would just need to download "$URL/alt-odb-$name.odb" and use the alternate object database support on this file.

This way it would know all it has to know to access the objects in the alternate database. The alternate object database may not contain the real objects, if they are too big for example, but just files that describe how to get the real objects.

Show 8 quoted lines
>  - After fully downloading $name.bndl and pack-$name.pack and
>    storing them next to each other, the client would clone from the
>    $name.bndl; this would populate the newly created repository with
>    reasonably recent history.
>
>  - Then the client can issue "git fetch" against the original
>    repository to obtain the most recent part of the history created
>    since the bundle was made.

[1] http://thread.gmane.org/gmane.comp.version-control.git/206886/focus=207040 [2] http://thread.gmane.org/gmane.comp.version-control.git/247171 [3] http://thread.gmane.org/gmane.comp.version-control.git/202902/focus=203020

Previous: Duy NguyenNext: Duy Nguyen
Message 14 of 37 in “bundle: plug resource leak”
  1. 1/2 bundle: plug resource leakJunio C Hamano, Mar 1, 2016
  2. 2/2 bundle: keep a copy of bundle file name in the in-core bundle headerJunio C Hamano, Mar 1, 2016
  3. Jeff KingMar 2, 2016
  4. Junio C HamanoMar 2, 2016
  5. 0/4 "split bundle" previewJunio C Hamano, Mar 2, 2016
  6. 1/4 bundle doc: 'verify' is not about verifying the bundleJunio C Hamano, Mar 2, 2016
  7. 2/4 bundle: plug resource leakJunio C Hamano, Mar 2, 2016
  8. 3/4 bundle: keep a copy of bundle file name in the in-core bundle headerJunio C Hamano, Mar 2, 2016
  9. Jeff KingMar 2, 2016
  10. 4/4 bundle v3: the beginningJunio C Hamano, Mar 2, 2016
  11. Duy NguyenMar 3, 2016
  12. Junio C HamanoMar 3, 2016
  13. Duy NguyenMar 3, 2016
  14. Christian CouderMay 20, 2016
  15. Duy NguyenMay 31, 2016
  16. Christian CouderMay 31, 2016
  17. Duy NguyenJun 1, 2016
  18. Christian CouderJun 7, 2016
  19. Duy NguyenJun 1, 2016
  20. Christian CouderJun 7, 2016
  21. Mike HommeyJun 7, 2016
  22. Duy NguyenJun 7, 2016
  23. Junio C HamanoJun 7, 2016
  24. Jeff KingJun 7, 2016
  25. Duy NguyenJun 8, 2016
  26. Jeff KingJun 8, 2016
  27. Duy NguyenJun 9, 2016
  28. Jeff KingJun 9, 2016
  29. Junio C HamanoJun 8, 2016
  30. Jeff KingJun 8, 2016
  31. Jeff KingMay 31, 2016
  32. Jeff KingMay 31, 2016
  33. Christian CouderJun 7, 2016
  34. Jeff KingJun 7, 2016
  35. Jeff KingMar 2, 2016
  36. Junio C HamanoMar 2, 2016
  37. Jeff KingMar 2, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.