# Roadmap to better handle big files? 5 messages from 2010-02-24 to 2010-02-25. Participants: Nick Triantos, Nicolas Pitre, Jakub Narebski, Joshua Jensen. Thread: https://gitlist.dev/t/22809 ## Nick Triantos, 2010-02-24 23:00 Subject: Roadmap to better handle big files? Message-ID: URL: https://gitlist.dev/e/B85968F5-E7C2-499D-A8BE-0160BA575F10%40perceptivepixel.com ``` Hi, Is there any planned functionality to better support large files in git? (> 100MB / file) We've been happily using git but we now have some files which we'd very much like to have under the same version control as our source code, and some of those files have been as large as 450MB/file. We are looking at chunking the file up before commiting it to git, but is there any plan to better support chunking of these files during repacks or other operations? Right now, it appears either the whole file, or the whole collection of files in a commit (not sure which) can need to be resident in memory up to twice, from reading various places on the web. Our poor 32-bit server is barfing on this. We are going to put more RAM and a 64bit OS on the machine, but this still seems like an unnecessary design decision. thanks very much, -Nick ``` ## Nicolas Pitre, 2010-02-24 23:39 Subject: Re: Roadmap to better handle big files? Message-ID: URL: https://gitlist.dev/e/alpine.LFD.2.00.1002241837510.1946%40xanadu.home In-Reply-To: ``` On Wed, 24 Feb 2010, Nick Triantos wrote: > Hi, > > Is there any planned functionality to better support large files in git? (> 100MB / file) Yes. It's just a matter of available time to implement it. Nicolas ``` ## Jakub Narebski, 2010-02-24 23:51 Subject: Re: Roadmap to better handle big files? Message-ID: URL: https://gitlist.dev/e/m3fx4qmbwr.fsf%40localhost.localdomain In-Reply-To: ``` Nick Triantos writes: > Is there any planned functionality to better support large files in > git? (> 100MB / file) > > We've been happily using git but we now have some files which we'd > very much like to have under the same version control as our source > code, and some of those files have been as large as 450MB/file. We > are looking at chunking the file up before commiting it to git, but > is there any plan to better support chunking of these files during > repacks or other operations? Right now, it appears either the whole > file, or the whole collection of files in a commit (not sure which) > can need to be resident in memory up to twice, from reading various > places on the web. Our poor 32-bit server is barfing on this. We > are going to put more RAM and a 64bit OS on the machine, but this > still seems like an unnecessary design decision. Git has a roadmap??? More seriously, take a look at git-bigfiles project (fork): http://caca.zoy.org/wiki/git-bigfiles HTH -- Jakub Narebski Poland ShadeHawk on #git ``` ## Nick Triantos, 2010-02-25 00:02 Subject: Re: Roadmap to better handle big files? Message-ID: <2009C5FE-F0B7-4D64-BF5C-04087E17EDF1@perceptivepixel.com> URL: https://gitlist.dev/e/2009C5FE-F0B7-4D64-BF5C-04087E17EDF1%40perceptivepixel.com In-Reply-To: ``` Thanks. I had looked at that project, but the logo being a piece of poop sort of scared me away from it (and it looked to be very early on in their design work so far)... thanks! -Nick On Feb 24, 2010, at 3:51 PM, Jakub Narebski wrote: > Nick Triantos writes: > >> Is there any planned functionality to better support large files in >> git? (> 100MB / file) >> >> We've been happily using git but we now have some files which we'd >> very much like to have under the same version control as our source >> code, and some of those files have been as large as 450MB/file. We >> are looking at chunking the file up before commiting it to git, but >> is there any plan to better support chunking of these files during >> repacks or other operations? Right now, it appears either the whole >> file, or the whole collection of files in a commit (not sure which) >> can need to be resident in memory up to twice, from reading various >> places on the web. Our poor 32-bit server is barfing on this. We >> are going to put more RAM and a 64bit OS on the machine, but this >> still seems like an unnecessary design decision. > > Git has a roadmap??? > > More seriously, take a look at git-bigfiles project (fork): > http://caca.zoy.org/wiki/git-bigfiles > > HTH > -- > Jakub Narebski > Poland > ShadeHawk on #git ``` ## Joshua Jensen, 2010-02-25 18:06 Subject: Re: Roadmap to better handle big files? Message-ID: <4B86BC12.9080201@workspacewhiz.com> URL: https://gitlist.dev/e/4B86BC12.9080201%40workspacewhiz.com In-Reply-To: ``` ----- Original Message ----- From: Nick Triantos Date: 2/24/2010 4:00 PM > Is there any planned functionality to better support large files in git? (> 100MB / file) > I once used Git alternates to point to a network share filled with the really large files hashed into a .git/objects directory. It worked, although it was slower than having the entire repository locally. Josh ```