Re: [PATCH 2/2] index-pack: reduce memory usage when the pack has large blobs

From: Ian Kumlien <pomac@vapor.com>
Date: Feb 25, 2012, 22:45 UTC
Message-ID: <20120225224533.GJ9526@pomac.netswarm.net>
In-Reply-To: <CACsJy8C-8dvXpNTU=JpdupSpS8OuqqTpGvDs6s1ASeKdk9d5Dg@mail.gmail.com>

On Sat, Feb 25, 2012 at 08:49:55AM +0700, Nguyen Thai Ngoc Duy wrote:

Show 17 quoted lines

> 2012/2/24 Ian Kumlien <pomac@vapor.com>:
> > Writing objects: 100% (1425/1425), 56.06 MiB | 4.62 MiB/s, done.
> > Total 1425 (delta 790), reused 1425 (delta 790)
> > fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
> > fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
> > fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
> > fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
> > To ../test_data/
> >  ! [remote rejected] master -> master (missing necessary objects)
> >  ! [remote rejected] origin/HEAD -> origin/HEAD (missing necessary objects)
> >  ! [remote rejected] origin/master -> origin/master (missing necessary objects)
> > error: failed to push some refs to '../test_data/'
> >
> > So there are additional code paths to look at... =(
> 
> I can't say where that came from. Does this help? (Space damaged, may
> need manual application)

> If not, you might need to apply this to generate coredump, then look
> and see where that failed malloc comes from

Actually, i added a backtrace and used addr2line to confirm my suspicion... which is: builtin/index-pack.c:414

ie get_data_from_pack...

It looks to me like, if we are to support this kind of things, we need a slightly different approach - instead of passing the data around, it feels like passing a function pointer around would be beneficial.

Looking at the code i see alot of places where this would be a issue, just the fact that get_data_from_pack is used in several functions that might do some small operation and then just free it.

I understand and recognize that my "problem" is not what git was designed for; it was designed for small files, which is very evident in how it approaches the data... And I'd most definetly have to look alot closer to this code... =)

> -- 
> Duy