{"thread":{"id":"53747","subject":"Re: your mail","startedAt":"2020-06-24T00:38:55Z","lastAt":"2020-06-24T01:31:09Z","messageCount":2,"participants":["brian m. carlson","shejan shuza"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"401047","messageId":"CABO4vy0WTYbeAiyWf7hE_4QurwH_c1wJUdU=A4FSC0uSxiEO7g@mail.gmail.com","threadId":"53747","inReplyTo":null,"subject":"","fromName":"shejan shuza","fromEmail":"shejan0@gmail.com","sentAt":"2020-06-24T00:38:39Z","receivedAt":"2020-06-24T00:38:55Z","isPatch":false,"sender":{"key":"shejan0@gmail.com","avatar":null},"body":"Hi, I have a repository with about 55GB of contents, with binary files\nthat are less than 100MB each (so no LFS mode) from a project which\nhas almost filled up an entire hard drive. I am trying to add all of\nthe contents to a git repo and push it to GitHub but every time I do\n\ngit add .\n\nin the folder with my contents after initializing and setting my\nremote, git starts caching all the files to .git/objects, making the\n.git folder grow in size rapidly. All the files are binaries, so git\ncannot stage changes between versions anyway, so there is no reason to\ncache versions.\n\nIs there any way, such as editing the git attributes or changing\nsomething about how files are staged in the git repository, to only\njust add indexes or references to files in the repository rather than\ncache them into the .git folder, while also being able to push all the\ndata to GitHub?\n"},{"id":"400485","messageId":"20200624013101.GA6451@camp.crustytoothpaste.net","threadId":"53747","inReplyTo":"CABO4vy0WTYbeAiyWf7hE_4QurwH_c1wJUdU=A4FSC0uSxiEO7g@mail.gmail.com","subject":"Re: your mail","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-24T01:31:01Z","receivedAt":"2020-06-24T01:31:09Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-06-24 at 00:38:39, shejan shuza wrote:\n> Hi, I have a repository with about 55GB of contents, with binary files\n> that are less than 100MB each (so no LFS mode) from a project which\n> has almost filled up an entire hard drive. I am trying to add all of\n> the contents to a git repo and push it to GitHub but every time I do\n> \n> git add .\n> \n> in the folder with my contents after initializing and setting my\n> remote, git starts caching all the files to .git/objects, making the\n> .git folder grow in size rapidly. All the files are binaries, so git\n> cannot stage changes between versions anyway, so there is no reason to\n> cache versions.\n\nWhat you're experiencing is normal; storing files in the .git directory\nis how Git keeps track of them.  It can't rely on the copies in your\nworking tree because you can modify those files at any time, and if you\ndid so, relying on them would corrupt the repository.\n\nAlso, note that Git can and does deltify changes between revisions once\nthe data is packed, regardless of whether the file is binary, but how\nwell it does so depends on your data.  For example, if it's compressed,\nit likely doesn't deltify very well, so storing things like compressed\nimages or zip files using deflate is generally going to result in a\nbloated repository.\n\nHowever, if you don't need versioning for these files, then you don't\nneed a Git repository.  Git is a tool for versioning files, not a\ngeneral storage mechanism.  You may find a cloud storage bucket or some\nother artifact storage service may meet your needs better.\n\nI will also tell you from a practical point of view that almost nobody\n(including you) will want to host a 55 GB repository filled with binary\nblobs.  Usually repacking these repositories is very expensive,\nrequiring extensive CPU and memory usage for a prolonged time for little\nuseful benefit.\n\n> Is there any way, such as editing the git attributes or changing\n> something about how files are staged in the git repository, to only\n> just add indexes or references to files in the repository rather than\n> cache them into the .git folder, while also being able to push all the\n> data to GitHub?\n\nThis is how Git LFS and similar tools, like git-annex, work.  Git LFS\nwill create copies of the objects in your .git directory though, at\nleast until they're pushed to the server, at which point they can be\npruned.  Git LFS has the same limitation as Git here.  I'm less familiar\nwith git-annex, but it is also a popular choice.\n\nHowever, as mentioned, it sounds like you don't need versioning at all,\nso unless you do, Git with Git LFS will be no more suitable for this\nthan plain Git.  If that's the case, I encourage you to explore\nalternate solutions.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"}]}