Re: [FYI] very large text files and their problems.
I'm uncertain if you got my reply since i did it out of bounds - so i'll repeat myself - sorry... =)
On Wed, Feb 22, 2012 at 11:18:19PM +0700, Nguyen Thai Ngoc Duy wrote:
Show 6 quoted lines
> On Wed, Feb 22, 2012 at 10:49 PM, Ian Kumlien <pomac@vapor.com> wrote:
> > Hi,
> >
> > We just saw a interesting issue, git compressed a ~3.4 gb project to ~57 mb.
>
> How big are those files? How many of them? How often do they change?
This was the first check in, there is no deltas yet.
The file in question is ~3.3 gb in size - ie exactly: 3310214313 bytes (as seen below in the malloc failure)
git show <blob sha1 id> |wc -c gives the same exact result.
Show 8 quoted lines
> > But when we tried to clone it on a big machine we got:
> >
> > fatal: Out of memory, malloc failed (tried to allocate
> > 18446744072724798634 bytes)
> >
> > This is already fixed in the 1.7.10 mainline - but it also seems like
>
> Does 1.7.9 have this problem?
Only tested 1.7.8 and 1.7.9.1 - works in mainline git (pre-1.7.10)
Show 14 quoted lines
> > git needs to have atleast the same ammount of memory as the largest
> > file free... Couldn't this be worked around?
> >
> > On a (32 bit) machine with 4GB memory - results in:
> > fatal: Out of memory, malloc failed (tried to allocate 3310214313 bytes)
> >
> > (and i see how this could be a problem, but couldn't it be mitigated? or
> > is it bydesign and intended behaviour?)
>
> I think that it's delta resolving that hogs all your memory. If your
> files are smaller than 512M, try lower core.bigFileThreshold. The
> topic jc/split-blob, which stores a big file are several smaller
> pieces, might solve your problem. Unfortunately the topic is not
> complete yet.
Well, in this case it's just stream unpacking gzip data to disk, i understand if delta would be a problem... But wouldn't delta be a problem in the sence of <size_of_change>+<size_of_subdata>+<result> ?
Ie, if the file is mmapped - it shouldn't have to be allocated, right?
> --
> Duy
