git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v4 9/9] Documentation/config: add odb.<name>.promisorRemote

From
Taylor Blau <me@ttaylorr.com>
Date
Sep 26, 2018, 13:44 UTC
Message-ID
<20180926134424.GB25697@syl>
In-Reply-To
<20180926041222.GA743@sigill.intra.peff.net>
On Wed, Sep 26, 2018 at 12:12:22AM -0400, Jeff King wrote:
Show 39 quoted lines
> On Tue, Sep 25, 2018 at 03:31:36PM -0700, Junio C Hamano wrote:
>
> > Christian Couder <christian.couder@gmail.com> writes:
> >
> > > The main issue that this patch series tries to solve is that
> > > extensions.partialclone config option limits the partial clone and
> > > promisor features to only one remote. One related issue is that it
> > > also prevents to have other kind of promisor/partial clone/odb
> > > remotes. By other kind I mean remotes that would not necessarily be
> > > git repos, but that could store objects (that's where ODB, for Object
> > > DataBase, comes from) and could provide those objects to Git through a
> > > helper (or driver) script or program.
> >
> > I do not think "sources that are not git repositories" is all that
> > interesting, unless they can also serve as the source for ext::
> > remote helper.  And if they can serve "git fetch ext::...", I think
> > they can be treated just like a normal Git repository by the
> > backfill code that needs to lazily populate the partial clone.
>
> I don't know about that. Imagine I had a regular Git repo with a bunch
> of large blobs, and then I also stored those large blobs in something
> like S3 that provides caching, geographic locality, and resumable
> transfers.
>
> [ ... ]
>
> Now if you are arguing that the interface to the external-odb helper
> script should be that it _looks_ like upload-pack, but simply advertises
> no refs and will let you fetch any object, that makes more sense to me.
> It's not something you could "git clone", but you can "git fetch" from
> it.
>
> However, that may be an overly constricting interface for the helper.
> E.g., we might want to be able to issue several requests and have them
> transfer in parallel. But I suppose we could teach that trick to
> upload-pack in the long run, as it may be applicable even to fetching
> from "real" git repos.
>
> Hmm. Actually, I kind of like that direction the more I think about it.

Yes, this is an important design decision for Git LFS, which I believe is important to this series. Git LFS allows the caller to issue `n` parallel object transfers (uploads or downloads) at a time, which is useful when, say, checking out a repository that has many large objects.

We do this trick with 'filter.lfs.process', where we accumulate many Git LFS objects that we wish to tell Git about so that it can check them out into the working copy, and then promise that we will provide the contents later (e.g., by sending status=delayed).

We then "batch" up all of those requests, issue them all at once (after which the LFS API will tell us the URLs of where to upload/download each item), and then we open "N" threads to do that work.

After all of that, we respond back with all of the objects that we had to download, and close the process filter.

Thanks, Taylor

Previous: Jeff KingNext: Junio C Hamano
Message 15 of 23 in “Introducing remote ODBs”
  1. 0/9 Introducing remote ODBsChristian Couder, Aug 2, 2018
  2. 1/9 fetch-object: make functions return an error codeChristian Couder, Aug 2, 2018
  3. 2/9 Add initial remote odb supportChristian Couder, Aug 2, 2018
  4. 3/9 remote-odb: implement remote_odb_get_direct()Christian Couder, Aug 2, 2018
  5. 4/9 remote-odb: implement remote_odb_get_many_direct()Christian Couder, Aug 2, 2018
  6. 5/9 remote-odb: add remote_odb_reinit()Christian Couder, Aug 2, 2018
  7. 6/9 Use remote_odb_get_direct() and has_remote_odb()Christian Couder, Aug 2, 2018
  8. 7/9 Use odb.origin.partialclonefilter instead of core.partialclonefilterChristian Couder, Aug 2, 2018
  9. 8/9 t0410: test fetching from many promisor remotesChristian Couder, Aug 2, 2018
  10. 9/9 Documentation/config: add odb.<name>.promisorRemoteChristian Couder, Aug 2, 2018
  11. Stefan BellerAug 2, 2018
  12. Christian CouderSep 25, 2018
  13. Junio C HamanoSep 25, 2018
  14. Jeff KingSep 26, 2018
  15. Taylor BlauSep 26, 2018
  16. Junio C HamanoSep 26, 2018
  17. Jonathan NiederOct 16, 2018
  18. Jonathan TanOct 16, 2018
  19. Junio C HamanoOct 19, 2018
  20. Jonathan NiederOct 19, 2018
  21. Junio C HamanoOct 19, 2018
  22. Christian CouderOct 31, 2018
  23. Jeff KingNov 1, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.