threads / discuss / 22809

Roadmap to better handle big files?

Subject: Roadmap to better handle big files?

## tl;dr

5 messages between Feb 24, 2010 and Feb 25, 2010.

replies: 4people: 4as markdown or json

Nick Triantos· Feb 24, 2010, 23:00 UTC · lore

Hi,

Is there any planned functionality to better support large files in git?  (> 100MB / file)

We've been happily using git but we now have some files which we'd very much like to have under the same version control as our source code, and some of those files have been as large as 450MB/file.  We are looking at chunking the file up before commiting it to git, but is there any plan to better support chunking of these files during repacks or other operations?  Right now, it appears either the whole file, or the whole collection of files in a commit (not sure which) can need to be resident in memory up to twice, from reading various places on the web.  Our poor 32-bit server is barfing on this.  We are going to put more RAM and a 64bit OS on the machine, but this still seems like an unnecessary design decision.

thanks very much, -Nick

Nicolas Pitre· Feb 24, 2010, 23:39 UTC · re: Nick Triantos · lore

Re: Roadmap to better handle big files?

On Wed, 24 Feb 2010, Nick Triantos wrote:

> Hi,
> 
> Is there any planned functionality to better support large files in git?  (> 100MB / file)

Yes.  It's just a matter of available time to implement it.

Nicolas

Jakub Narebski· Feb 24, 2010, 23:51 UTC · re: Nick Triantos · lore

Re: Roadmap to better handle big files?

Nick Triantos <nick@perceptivepixel.com> writes:

Show 14 quoted lines

> Is there any planned functionality to better support large files in
> git?  (> 100MB / file)
> 
> We've been happily using git but we now have some files which we'd
> very much like to have under the same version control as our source
> code, and some of those files have been as large as 450MB/file.  We
> are looking at chunking the file up before commiting it to git, but
> is there any plan to better support chunking of these files during
> repacks or other operations?  Right now, it appears either the whole
> file, or the whole collection of files in a commit (not sure which)
> can need to be resident in memory up to twice, from reading various
> places on the web.  Our poor 32-bit server is barfing on this.  We
> are going to put more RAM and a 64bit OS on the machine, but this
> still seems like an unnecessary design decision.

Git has a roadmap???

More seriously, take a look at git-bigfiles project (fork): http://caca.zoy.org/wiki/git-bigfiles

HTH

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Nick Triantos· Feb 25, 2010, 00:02 UTC · re: Jakub Narebski · lore

Re: Roadmap to better handle big files?

Thanks.  I had looked at that project, but the logo being a piece of poop sort of scared me away from it (and it looked to be very early on in their design work so far)...

thanks! -Nick

On Feb 24, 2010, at 3:51 PM, Jakub Narebski wrote:

Show 27 quoted lines

> Nick Triantos <nick@perceptivepixel.com> writes:
> 
>> Is there any planned functionality to better support large files in
>> git?  (> 100MB / file)
>> 
>> We've been happily using git but we now have some files which we'd
>> very much like to have under the same version control as our source
>> code, and some of those files have been as large as 450MB/file.  We
>> are looking at chunking the file up before commiting it to git, but
>> is there any plan to better support chunking of these files during
>> repacks or other operations?  Right now, it appears either the whole
>> file, or the whole collection of files in a commit (not sure which)
>> can need to be resident in memory up to twice, from reading various
>> places on the web.  Our poor 32-bit server is barfing on this.  We
>> are going to put more RAM and a 64bit OS on the machine, but this
>> still seems like an unnecessary design decision.
> 
> Git has a roadmap???
> 
> More seriously, take a look at git-bigfiles project (fork):
> http://caca.zoy.org/wiki/git-bigfiles
> 
> HTH
> --
> Jakub Narebski
> Poland
> ShadeHawk on #git

Joshua Jensen· Feb 25, 2010, 18:06 UTC · re: Nick Triantos · lore

Re: Roadmap to better handle big files?

----- Original Message -----
From: Nick Triantos
Date: 2/24/2010 4:00 PM

> Is there any planned functionality to better support large files in git?  (>  100MB / file)
>

I once used Git alternates to point to a network share filled with the really large files hashed into a .git/objects directory. It worked, although it was slower than having the entire repository locally.

Josh