{"thread":{"id":"50181","subject":"git-lfs integration?","startedAt":"2019-01-08T15:23:25Z","lastAt":"2019-01-10T03:10:09Z","messageCount":4,"participants":["Harald Dunkel","Ævar Arnfjörð Bjarmason","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"366332","messageId":"79fd2b4e-243c-a9f5-3485-2954fb0f50ef@aixigo.de","threadId":"50181","inReplyTo":null,"subject":"git-lfs integration?","fromName":"Harald Dunkel","fromEmail":"harald.dunkel@aixigo.de","sentAt":"2019-01-08T15:16:41Z","receivedAt":"2019-01-08T15:23:25Z","isPatch":false,"sender":{"key":"harald.dunkel@aixigo.de","avatar":null},"body":"Hi folks,\n\nI wonder why git-lfs is needed to efficiently handle large files\nin git. Would it be reasonable to integrate this functionality\ninto the native git?\n\nPlease excuse me asking. I read some pretty scary articles about\nrewriting history, asking everybody to clone existing repositories\nagain, and strange errors if git-lfs is *not* installed. Apparently\nthis is a one-way street, so I didn't dare to install git-lfs yet.\n\n\nRegards\nHarri\n"},{"id":"366333","messageId":"87k1jf6tls.fsf@evledraar.gmail.com","threadId":"50181","inReplyTo":"79fd2b4e-243c-a9f5-3485-2954fb0f50ef@aixigo.de","subject":"Re: git-lfs integration?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-08T15:45:35Z","receivedAt":"2019-01-08T15:45:39Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jan 08 2019, Harald Dunkel wrote:\n\n> I wonder why git-lfs is needed to efficiently handle large files\n> in git. Would it be reasonable to integrate this functionality\n> into the native git?\n>\n> Please excuse me asking. I read some pretty scary articles about\n> rewriting history, asking everybody to clone existing repositories\n> again, and strange errors if git-lfs is *not* installed. Apparently\n> this is a one-way street, so I didn't dare to install git-lfs yet.\n\nIt depends on what \"integrate this\" means.\n\nI think it's unlikely that git-lfs would be integrated as-is. There's\nvarious clean/smudge filters like it that do remote downloads (also\ngit-annex). Everyone's probably better off if git itself maintains the\ninfra needed for that ecosystem, and users can vote by their usage which\none(s) they like.\n\nBut in more general terms the problem of making git natively friendlier\nto \"large files\" is being worked on on multiple fronts.\n\nFor one, Microsoft has been upstreaming parts of their GVFS fork, if you\nsearch for \"partial clone\" in release notes since 2.16.0 (including in\n2.20.0) you'll see some of that work relevant to that. I.e. one part of\nthis is the general ability to have partially available local history,\nwhether it's skipping (big) blobs, some trees etc.\n\nAnother effort has been the introduction of the v2 protocol to Git,\nwhich has happened recently, and is only now starting to get rolled out\nat various hosting providers. That in and of itself hasn't helped with\nthis, but allows for future extensions to the protocol, such as \"this is\nnot the full data, you can find the rest at xyz\".\n\nThen there's the \"odb\" effort, see e.g. here:\nhttps://public-inbox.org/git/20180802061505.2983-1-chriscool@tuxfamily.org/\n\nI think that long-term (5-20yrs) those effors will probably completely\nsupplant the approach git-lfs is taking. It's a very useful tool, but\nultimately a bit of a hacky workaround in lieu of addressing fixable\nissues in git itself, i.e. native support for partially downloaded\nhistory. But getting to that point will take time & effort.\n"},{"id":"366397","messageId":"6b501c00-3d7e-5c58-b6bb-fd1405dafde8@aixigo.de","threadId":"50181","inReplyTo":"87k1jf6tls.fsf@evledraar.gmail.com","subject":"Re: git-lfs integration?","fromName":"Harald Dunkel","fromEmail":"harald.dunkel@aixigo.de","sentAt":"2019-01-09T06:10:02Z","receivedAt":"2019-01-09T06:10:06Z","isPatch":false,"sender":{"key":"harald.dunkel@aixigo.de","avatar":null},"body":"Hi Ævar,\n\nthanx very much for your detailed response. Exactly what I was\nlooking for.\n\n\nRegards\nHarri\n"},{"id":"366468","messageId":"20190110030959.GL423984@genre.crustytoothpaste.net","threadId":"50181","inReplyTo":"79fd2b4e-243c-a9f5-3485-2954fb0f50ef@aixigo.de","subject":"Re: git-lfs integration?","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2019-01-10T03:09:59Z","receivedAt":"2019-01-10T03:10:09Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Tue, Jan 08, 2019 at 04:16:41PM +0100, Harald Dunkel wrote:\n> Hi folks,\n> \n> I wonder why git-lfs is needed to efficiently handle large files\n> in git. Would it be reasonable to integrate this functionality\n> into the native git?\n\nMost of the problems Git has with handling large files aren't really\nproblems if you have unlimited resources, but they are practical\nproblems in some situations.\n\nGit doesn't really handle files that exceed memory capacity very well.\nIn order to deltify files, we need to have them in memory, so the option\nis either to not deltify large files and have a lot of storage used, or\ndeltify and spend a lot of CPU and memory compressing, decompressing,\nand deltifying them.\n\nThis means that Git can require a lot of resources to store and repack\nlarge files. This is not only a problem on your system, but on whatever\nremote system you host your repos on (your own server, GitHub, GitLab,\netc.). Your host, while probably having more resources than your local\nmachine, also probably has more repos as well.\n\nGit LFS makes the trade-off to store files uncompressed and only copy\nthe needed files from the server to your system. That means that you\ndon't need to bloat your local clone with files you may never check out,\nbut you have the downside that your clone isn't necessarily complete.\n\nI'm a maintainer of Git LFS, and I'm perfectly happy with solutions that\nhelp Git handle large files better. Ævar gave a great explanation of\nsome of the work that's going on in this regard, and I'm also happy to\nhear about other improvements that may come up as well.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"}]}