Why Git LFS is not a built-in feature

Currently, having to set up git-lfs in each client and checking server compatibility is a huge barrier for using it in the first place, whilst it is generally a good practice to store large files in lfs.

As a consequence a lot of repos are not using it when they should.

Is there any reason that we don't have built-in support for such an important feature?

On 2020-11-13 at 09:45:52, Alireza wrote:

There are a couple reasons that it's not a built-in feature:

* First, there are several options in this space.  Git LFS is one,
  git-annex is another, and some people prefer to store large objects in
  the repository and use partial clone.  Git, as a project, tries to be
  flexible and meet the needs of various kinds of users without
  privileging one or another external tool.
* Git LFS is a complicated piece of software and it's currently written
  in Go, which is different from most of Git.  Re-implementing it in C
  would be burdensome, and there's little interest in maintaining Go
  software in the Git project.
* Git LFS uses a different protocol from Git, requiring additional
  configuration and a separate server-side component.
* The smudge and clean filter approach has some limitations, among them
  that users who don't have the external filter installed can commit
  uncleaned objects that then result in the working tree consistently
  being modified, even after git reset --hard.

It's my hope that the built-in support for partial clone will mature enough to the point where that's a clear win and the need for external tools isn't as great, since I think that will ultimately provide a better experience for users. Some people are already using it. So in some sense, we do have this as a built-in feature, maybe just not the one you were expecting.

Additionally, in many cases, projects can avoid the need for storing large files at all by using repository best practices, like not storing build products or binary dependencies in the repository and instead using an artifact server or a standard packaging system. If possible, that will almost always provide a better experience than any solution for storing large files in the repository.

Finally, if you do want to use an external tool like Git LFS, it's reasonably straightforward to specify a script to install and configure the required dependencies for your project on each system so that everything just works. One popular location for this kind of path is script/bootstrap.

On Sat, Nov 14, 2020 at 12:29:02AM +0000, brian m. carlson wrote:

Well, I would argue that if the goal is ongoing archival and easy replication, then storing objects in a repository like git makes a lot more sense than keeping them on a central server that may or may not be there a few years down the line. Having large file support native in git is a laudable goal and I quite often wish that it existed.

-K

On Sat, Nov 14 2020, Konstantin Ryabitsev wrote:

That native support does exist right now in the form of partial clones, the packfile-uris support, core.bigFileThreshold etc.

It's got a lot of rough edges currently, but if it's something you're interested in you should try it out and see if the subset of features that works well now is something that would work for you.

On Sat, Nov 14, 2020 at 7:25 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

I have been working on a partial clone demo that stores large files on an HTTP server:

https://gitlab.com/chriscool/partial-clone-demo/-/blob/master/http-promisor/demo.txt

It has a lot of rough edges indeed. Fetching from the HTTP server promisor remote is very slow. For the last part you need a hacked Git. The scripts have a lot of bugs and limitations and are not finished (to say the least).

The goal for now is just to give people (especially product managers, developers and managers inside GitLab) an outlook about how it could work.

On 2020-11-14 at 16:27:00, Konstantin Ryabitsev wrote:

Sure, and I think that's a different goal than the typical source code or writing project (documentation, book, etc.) repository. For example, one can use Git repositories to do backups using the tool bup, which actually works quite well but isn't a traditional use of Git.

The typical use case is that the user wants to store some reasonably sized project on their local system and possibly also collaborate with others for that, and with that goal, it makes sense to make the repository not be absurdly large, since most developer systems don't have tons of storage. As Ævar and I mentioned, there are built-in options for large files that make this use case more palatable with native Git tooling. But for this particular use case, it doesn't logically make sense to store build assets, whether they're yours or others', in this project repository.

If your goal is archival and replication, then a tool like bup might meet your needs, or simply a large repository with many objects. But in that context, you'll likely have more storage, CPU, and memory available to you and the need for large file support will look different (e.g., core.bigFileThreshold) or not be present at all.