{"thread":{"id":"54637","subject":"Why Git LFS is not a built-in feature","startedAt":"2020-11-13T09:46:25Z","lastAt":"2020-11-18T10:20:34Z","messageCount":6,"participants":["Alireza","brian m. carlson","Konstantin Ryabitsev","Ævar Arnfjörð Bjarmason","Christian Couder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"409862","messageId":"CAD9n_qjKyxNjtd1YrcHzshLg0-vbwXkHRwMveXHAWSOXMWLKAg@mail.gmail.com","threadId":"54637","inReplyTo":null,"subject":"Why Git LFS is not a built-in feature","fromName":"Alireza","fromEmail":"rezaxm@gmail.com","sentAt":"2020-11-13T09:45:52Z","receivedAt":"2020-11-13T09:46:25Z","isPatch":false,"sender":{"key":"rezaxm@gmail.com","avatar":null},"body":"Currently, having to set up git-lfs in each client and checking server\ncompatibility is a huge barrier for using it in the first place,\nwhilst it is generally a good practice to store large files in lfs.\n\nAs a consequence a lot of repos are not using it when they should.\n\nIs there any reason that we don't have built-in support for such an\nimportant feature?\n"},{"id":"409938","messageId":"20201114002902.GN6252@camp.crustytoothpaste.net","threadId":"54637","inReplyTo":"CAD9n_qjKyxNjtd1YrcHzshLg0-vbwXkHRwMveXHAWSOXMWLKAg@mail.gmail.com","subject":"Re: Why Git LFS is not a built-in feature","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-11-14T00:29:02Z","receivedAt":"2020-11-14T00:29:51Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-11-13 at 09:45:52, Alireza wrote:\n> Currently, having to set up git-lfs in each client and checking server\n> compatibility is a huge barrier for using it in the first place,\n> whilst it is generally a good practice to store large files in lfs.\n> \n> As a consequence a lot of repos are not using it when they should.\n> \n> Is there any reason that we don't have built-in support for such an\n> important feature?\n\nThere are a couple reasons that it's not a built-in feature:\n\n* First, there are several options in this space.  Git LFS is one,\n  git-annex is another, and some people prefer to store large objects in\n  the repository and use partial clone.  Git, as a project, tries to be\n  flexible and meet the needs of various kinds of users without\n  privileging one or another external tool.\n* Git LFS is a complicated piece of software and it's currently written\n  in Go, which is different from most of Git.  Re-implementing it in C\n  would be burdensome, and there's little interest in maintaining Go\n  software in the Git project.\n* Git LFS uses a different protocol from Git, requiring additional\n  configuration and a separate server-side component.\n* The smudge and clean filter approach has some limitations, among them\n  that users who don't have the external filter installed can commit\n  uncleaned objects that then result in the working tree consistently\n  being modified, even after git reset --hard.\n\nIt's my hope that the built-in support for partial clone will mature\nenough to the point where that's a clear win and the need for external\ntools isn't as great, since I think that will ultimately provide a\nbetter experience for users.  Some people are already using it.  So in\nsome sense, we do have this as a built-in feature, maybe just not the\none you were expecting.\n\nAdditionally, in many cases, projects can avoid the need for storing\nlarge files at all by using repository best practices, like not storing\nbuild products or binary dependencies in the repository and instead\nusing an artifact server or a standard packaging system.  If possible,\nthat will almost always provide a better experience than any solution\nfor storing large files in the repository.\n\nFinally, if you do want to use an external tool like Git LFS, it's\nreasonably straightforward to specify a script to install and configure\nthe required dependencies for your project on each system so that\neverything just works.  One popular location for this kind of path is\nscript/bootstrap.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"409957","messageId":"20201114162700.cvmxzcs4sdhsxpak@chatter.i7.local","threadId":"54637","inReplyTo":"20201114002902.GN6252@camp.crustytoothpaste.net","subject":"Re: Why Git LFS is not a built-in feature","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2020-11-14T16:27:00Z","receivedAt":"2020-11-14T16:27:05Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Sat, Nov 14, 2020 at 12:29:02AM +0000, brian m. carlson wrote:\n> Additionally, in many cases, projects can avoid the need for storing\n> large files at all by using repository best practices, like not storing\n> build products or binary dependencies in the repository and instead\n> using an artifact server or a standard packaging system.  If possible,\n> that will almost always provide a better experience than any solution\n> for storing large files in the repository.\n\nWell, I would argue that if the goal is ongoing archival and easy \nreplication, then storing objects in a repository like git makes a lot \nmore sense than keeping them on a central server that may or may not be \nthere a few years down the line. Having large file support native in git \nis a laudable goal and I quite often wish that it existed.\n\n-K\n"},{"id":"409959","messageId":"87blfzg5qa.fsf@evledraar.gmail.com","threadId":"54637","inReplyTo":"20201114162700.cvmxzcs4sdhsxpak@chatter.i7.local","subject":"Re: Why Git LFS is not a built-in feature","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2020-11-14T18:20:45Z","receivedAt":"2020-11-14T18:21:06Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Nov 14 2020, Konstantin Ryabitsev wrote:\n\n> On Sat, Nov 14, 2020 at 12:29:02AM +0000, brian m. carlson wrote:\n>> Additionally, in many cases, projects can avoid the need for storing\n>> large files at all by using repository best practices, like not storing\n>> build products or binary dependencies in the repository and instead\n>> using an artifact server or a standard packaging system.  If possible,\n>> that will almost always provide a better experience than any solution\n>> for storing large files in the repository.\n>\n> Well, I would argue that if the goal is ongoing archival and easy \n> replication, then storing objects in a repository like git makes a lot \n> more sense than keeping them on a central server that may or may not be \n> there a few years down the line. Having large file support native in git \n> is a laudable goal and I quite often wish that it existed.\n\nThat native support does exist right now in the form of partial clones,\nthe packfile-uris support, core.bigFileThreshold etc.\n\nIt's got a lot of rough edges currently, but if it's something you're\ninterested in you should try it out and see if the subset of features\nthat works well now is something that would work for you.\n"},{"id":"409962","messageId":"20201114191531.GO6252@camp.crustytoothpaste.net","threadId":"54637","inReplyTo":"20201114162700.cvmxzcs4sdhsxpak@chatter.i7.local","subject":"Re: Why Git LFS is not a built-in feature","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-11-14T19:15:31Z","receivedAt":"2020-11-14T19:15:40Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-11-14 at 16:27:00, Konstantin Ryabitsev wrote:\n> On Sat, Nov 14, 2020 at 12:29:02AM +0000, brian m. carlson wrote:\n> > Additionally, in many cases, projects can avoid the need for storing\n> > large files at all by using repository best practices, like not storing\n> > build products or binary dependencies in the repository and instead\n> > using an artifact server or a standard packaging system.  If possible,\n> > that will almost always provide a better experience than any solution\n> > for storing large files in the repository.\n> \n> Well, I would argue that if the goal is ongoing archival and easy\n> replication, then storing objects in a repository like git makes a lot\n> more sense than keeping them on a central server that may or may not be\n> there a few years down the line. Having large file support native in git\n> is a laudable goal and I quite often wish that it existed.\n\nSure, and I think that's a different goal than the typical source code\nor writing project (documentation, book, etc.) repository.  For example,\none can use Git repositories to do backups using the tool bup, which\nactually works quite well but isn't a traditional use of Git.\n\nThe typical use case is that the user wants to store some reasonably\nsized project on their local system and possibly also collaborate with\nothers for that, and with that goal, it makes sense to make the\nrepository not be absurdly large, since most developer systems don't\nhave tons of storage.  As Ævar and I mentioned, there are built-in\noptions for large files that make this use case more palatable with\nnative Git tooling.  But for this particular use case, it doesn't\nlogically make sense to store build assets, whether they're yours or\nothers', in this project repository.\n\nIf your goal is archival and replication, then a tool like bup might\nmeet your needs, or simply a large repository with many objects.  But in\nthat context, you'll likely have more storage, CPU, and memory available\nto you and the need for large file support will look different (e.g.,\ncore.bigFileThreshold) or not be present at all.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"410217","messageId":"CAP8UFD35kk10FpUnPpiAUzTHJbm=SJ-76OTmkTwBstGFe3Zgdw@mail.gmail.com","threadId":"54637","inReplyTo":"87blfzg5qa.fsf@evledraar.gmail.com","subject":"Partial clone demo for large files (Re: Why Git LFS is not a built-in feature)","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2020-11-18T10:20:20Z","receivedAt":"2020-11-18T10:20:34Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sat, Nov 14, 2020 at 7:25 PM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n>\n>\n> On Sat, Nov 14 2020, Konstantin Ryabitsev wrote:\n>\n> > On Sat, Nov 14, 2020 at 12:29:02AM +0000, brian m. carlson wrote:\n> >> Additionally, in many cases, projects can avoid the need for storing\n> >> large files at all by using repository best practices, like not storing\n> >> build products or binary dependencies in the repository and instead\n> >> using an artifact server or a standard packaging system.  If possible,\n> >> that will almost always provide a better experience than any solution\n> >> for storing large files in the repository.\n> >\n> > Well, I would argue that if the goal is ongoing archival and easy\n> > replication, then storing objects in a repository like git makes a lot\n> > more sense than keeping them on a central server that may or may not be\n> > there a few years down the line. Having large file support native in git\n> > is a laudable goal and I quite often wish that it existed.\n>\n> That native support does exist right now in the form of partial clones,\n> the packfile-uris support, core.bigFileThreshold etc.\n>\n> It's got a lot of rough edges currently, but if it's something you're\n> interested in you should try it out and see if the subset of features\n> that works well now is something that would work for you.\n\nI have been working on a partial clone demo that stores large files on\nan HTTP server:\n\nhttps://gitlab.com/chriscool/partial-clone-demo/-/blob/master/http-promisor/demo.txt\n\nIt has a lot of rough edges indeed. Fetching from the HTTP server\npromisor remote is very slow. For the last part you need a hacked Git.\nThe scripts have a lot of bugs and limitations and are not finished\n(to say the least).\n\nThe goal for now is just to give people (especially product managers,\ndevelopers and managers inside GitLab) an outlook about how it could\nwork.\n"}]}