git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Partial clone demo for large files (Re: Why Git LFS is not a built-in feature)

From
Christian Couder <christian.couder@gmail.com>
Date
Nov 18, 2020, 10:20 UTC
Message-ID
<CAP8UFD35kk10FpUnPpiAUzTHJbm=SJ-76OTmkTwBstGFe3Zgdw@mail.gmail.com>
In-Reply-To
<87blfzg5qa.fsf@evledraar.gmail.com>

On Sat, Nov 14, 2020 at 7:25 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

Show 24 quoted lines
>
>
> On Sat, Nov 14 2020, Konstantin Ryabitsev wrote:
>
> > On Sat, Nov 14, 2020 at 12:29:02AM +0000, brian m. carlson wrote:
> >> Additionally, in many cases, projects can avoid the need for storing
> >> large files at all by using repository best practices, like not storing
> >> build products or binary dependencies in the repository and instead
> >> using an artifact server or a standard packaging system.  If possible,
> >> that will almost always provide a better experience than any solution
> >> for storing large files in the repository.
> >
> > Well, I would argue that if the goal is ongoing archival and easy
> > replication, then storing objects in a repository like git makes a lot
> > more sense than keeping them on a central server that may or may not be
> > there a few years down the line. Having large file support native in git
> > is a laudable goal and I quite often wish that it existed.
>
> That native support does exist right now in the form of partial clones,
> the packfile-uris support, core.bigFileThreshold etc.
>
> It's got a lot of rough edges currently, but if it's something you're
> interested in you should try it out and see if the subset of features
> that works well now is something that would work for you.

I have been working on a partial clone demo that stores large files on an HTTP server:

https://gitlab.com/chriscool/partial-clone-demo/-/blob/master/http-promisor/demo.txt

It has a lot of rough edges indeed. Fetching from the HTTP server promisor remote is very slow. For the last part you need a hacked Git. The scripts have a lot of bugs and limitations and are not finished (to say the least).

The goal for now is just to give people (especially product managers, developers and managers inside GitLab) an outlook about how it could work.

Previous: Ævar Arnfjörð BjarmasonNext: brian m. carlson
Message 5 of 6 in “Why Git LFS is not a built-in feature”
  1. AlirezaNov 13, 2020
  2. brian m. carlsonNov 14, 2020
  3. Konstantin RyabitsevNov 14, 2020
  4. Ævar Arnfjörð BjarmasonNov 14, 2020
  5. Partial clone demo for large files (Re: Why Git LFS is not a built-in feature)Christian Couder, Nov 18, 2020
  6. brian m. carlsonNov 14, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.