{"thread":{"id":"29695","subject":"Re: [FYI] very large text files and their problems.","startedAt":"2012-02-22T18:39:40Z","lastAt":"2012-02-22T18:39:40Z","messageCount":1,"participants":["Ian Kumlien"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"185185","messageId":"1329935980.23912.5.camel@pi","threadId":"29695","inReplyTo":null,"subject":"Re: [FYI] very large text files and their problems.","fromName":"Ian Kumlien","fromEmail":"pomac@vapor.com","sentAt":"2012-02-22T18:39:40Z","receivedAt":"2012-02-22T18:39:40Z","isPatch":false,"sender":{"key":"pomac@vapor.com","avatar":null},"body":"Seems like i ruined my dovecot config in a recent upgrade - which also\naffected my mail... =/\n\nAnyway, it's all fixed now.\n\nfrom: Nguyen Thai Ngoc Duy <pclouds () gmail ! com>\n> On Wed, Feb 22, 2012 at 10:49 PM, Ian Kumlien <pomac@vapor.com> wrote:\n> > Hi,\n> >\n> > We just saw a interesting issue, git compressed a ~3.4 gb project to\n> ~57 mb.\n> \n> How big are those files? How many of them? How often do they change?\n\nThis is the initial check in, one of the files is a 3.3 gb text file.\n\n> > But when we tried to clone it on a big machine we got:\n> >\n> > fatal: Out of memory, malloc failed (tried to allocate\n> > 18446744072724798634 bytes)\n> >\n> > This is already fixed in the 1.7.10 mainline - but it also seems\n> like\n> \n> Does 1.7.9 have this problem?\n\nI've tested with 1.7.9.1, haven't downgraded to test with 1.7.9...\n\n> > git needs to have atleast the same ammount of memory as the largest\n> > file free... Couldn't this be worked around?\n> >\n> > On a (32 bit) machine with 4GB memory - results in:\n> > fatal: Out of memory, malloc failed (tried to allocate 3310214313\n> bytes)\n> >\n> > (and i see how this could be a problem, but couldn't it be\n> mitigated? or\n> > is it bydesign and intended behaviour?)\n> \n> I think that it's delta resolving that hogs all your memory. If your\n> files are smaller than 512M, try lower core.bigFileThreshold. The\n> topic jc/split-blob, which stores a big file are several smaller\n> pieces, might solve your problem. Unfortunately the topic is not\n> complete yet.\n\nthe problem here is that there is one file that is exactly: 3310214313\nbytes, so it should all be one \"blob\".\n\nsplit-blob would be really interesting for several reasons though =)\n\n> -- \n> Duy\n> --\n-- \nIan Kumlien  -- http://demius.net || http://pomac.netswarm.net\n"}]}