{"thread":{"id":"3507","subject":"git doesn't like big files when pushing","startedAt":"2006-03-01T22:08:02Z","lastAt":"2006-03-02T12:43:06Z","messageCount":10,"participants":["Greg KH","Linus Torvalds","Junio C Hamano","Martin Langhoff","Simon Richter"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"16983","messageId":"20060301220802.GA18250@kroah.com","threadId":"3507","inReplyTo":null,"subject":"git doesn't like big files when pushing","fromName":"Greg KH","fromEmail":"greg@kroah.com","sentAt":"2006-03-01T22:08:02Z","receivedAt":"2006-03-01T22:08:02Z","isPatch":false,"sender":{"key":"greg@kroah.com","avatar":"https://gravatar.com/avatar/5bb5aa0cc2e01c00ec899d11130c07796bc186e465bae57bc34873b13b72c7c8?d=mp&s=160"},"body":"I have a mail archive stored with git, in mbox form, and I made some\nchanges to a few of the files and checked them back in.\n\nThat worked fine, but when I went to push the stuff to my server, I got\nthe following errors:\n\n$ git push origin\nupdating 'refs/heads/master'\n from 490badd9bec9ada3a21be275c97fb2a3a390f49e\n to 16be8985abc8a9c89ad2cc8f46a0d8e9786e832f\nGenerating pack...\nDone counting 8 objects.\nDeltifying 8 objects.\nfatal: Out of memory, malloc failed\nfatal: early EOF\nunpack unpacker exited with error code\nng refs/heads/master n/a (unpacker error)\n\n\nAnd here are the file sizes of the files that were changed:\n$ ls -lh linux-usb-devel.save.200*\n-rw-r--r-- 1 greg users 41M Jan 6 14:30 linux-usb-devel.save.2001\n-rw-r--r-- 1 greg users 80M Jan 6 14:30 linux-usb-devel.save.2002\n-rw-r--r-- 1 greg users 74M Jan 6 14:30 linux-usb-devel.save.2003\n-rw-r--r-- 1 greg users 99M Jan 6 14:30 linux-usb-devel.save.2004\n-rw-r--r-- 1 greg users 89M Jan 6 14:30 linux-usb-devel.save.2005\n\nSo, am I just foolish for trying to use git for this? Should I just go\nback to using rsync to back stuff like this up with?\n\nthanks,\n\ngreg k-h\n"},{"id":"16984","messageId":"20060301220840.GB18250@kroah.com","threadId":"3507","inReplyTo":"20060301220802.GA18250@kroah.com","subject":"Re: git doesn't like big files when pushing","fromName":"Greg KH","fromEmail":"greg@kroah.com","sentAt":"2006-03-01T22:08:40Z","receivedAt":"2006-03-01T22:08:40Z","isPatch":false,"sender":{"key":"greg@kroah.com","avatar":"https://gravatar.com/avatar/5bb5aa0cc2e01c00ec899d11130c07796bc186e465bae57bc34873b13b72c7c8?d=mp&s=160"},"body":"On Wed, Mar 01, 2006 at 02:08:02PM -0800, Greg KH wrote:\n> I have a mail archive stored with git, in mbox form, and I made some\n> changes to a few of the files and checked them back in.\n> \n> That worked fine, but when I went to push the stuff to my server, I got\n> the following errors:\n> \n> $ git push origin\n> updating 'refs/heads/master'\n> from 490badd9bec9ada3a21be275c97fb2a3a390f49e\n> to 16be8985abc8a9c89ad2cc8f46a0d8e9786e832f\n> Generating pack...\n> Done counting 8 objects.\n> Deltifying 8 objects.\n> fatal: Out of memory, malloc failed\n> fatal: early EOF\n> unpack unpacker exited with error code\n> ng refs/heads/master n/a (unpacker error)\n\nOh, and I'm using:\n\t$ git --version\n\tgit version 1.2.3.g8c2f\n\nif that helps or not.\n\nthanks,\n\ngreg k-h\n"},{"id":"16986","messageId":"20060301222018.GA9965@kroah.com","threadId":"3507","inReplyTo":"20060301220840.GB18250@kroah.com","subject":"Re: git doesn't like big files when pushing","fromName":"Greg KH","fromEmail":"greg@kroah.com","sentAt":"2006-03-01T22:20:18Z","receivedAt":"2006-03-01T22:20:18Z","isPatch":false,"sender":{"key":"greg@kroah.com","avatar":"https://gravatar.com/avatar/5bb5aa0cc2e01c00ec899d11130c07796bc186e465bae57bc34873b13b72c7c8?d=mp&s=160"},"body":"On Wed, Mar 01, 2006 at 02:08:40PM -0800, Greg KH wrote:\n> On Wed, Mar 01, 2006 at 02:08:02PM -0800, Greg KH wrote:\n> > I have a mail archive stored with git, in mbox form, and I made some\n> > changes to a few of the files and checked them back in.\n> > \n> > That worked fine, but when I went to push the stuff to my server, I got\n> > the following errors:\n> > \n> > $ git push origin\n> > updating 'refs/heads/master'\n> > from 490badd9bec9ada3a21be275c97fb2a3a390f49e\n> > to 16be8985abc8a9c89ad2cc8f46a0d8e9786e832f\n> > Generating pack...\n> > Done counting 8 objects.\n> > Deltifying 8 objects.\n> > fatal: Out of memory, malloc failed\n> > fatal: early EOF\n> > unpack unpacker exited with error code\n> > ng refs/heads/master n/a (unpacker error)\n> \n> Oh, and I'm using:\n> \t$ git --version\n> \tgit version 1.2.3.g8c2f\n\nHm, 1.2.4.g6177 seems better, it's still trying to pack things, after\nabout 10 minutes, but at least it isn't dying yet. I'll let you know if\nit finishes properly or not...\n\nthanks,\n\ngreg k-h\n"},{"id":"16990","messageId":"Pine.LNX.4.64.0603011448230.22647@g5.osdl.org","threadId":"3507","inReplyTo":"20060301220802.GA18250@kroah.com","subject":"Re: git doesn't like big files when pushing","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-03-01T23:00:49Z","receivedAt":"2006-03-01T23:00:49Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 1 Mar 2006, Greg KH wrote:\n> \n> That worked fine, but when I went to push the stuff to my server, I got\n> the following errors:\n> \n> $ git push origin\n> updating 'refs/heads/master'\n> from 490badd9bec9ada3a21be275c97fb2a3a390f49e\n> to 16be8985abc8a9c89ad2cc8f46a0d8e9786e832f\n> Generating pack...\n> Done counting 8 objects.\n> Deltifying 8 objects.\n> fatal: Out of memory, malloc failed\n> fatal: early EOF\n\nGaah. We probably have a memory leak somewhere, and it just normally \ndoesn't much matter.\n\nGit does want to keep the \"window\" of the objects it packs in memory while \npacking (it would be really costly to read them in one at a time, over and \nover again), but it should hopefully not really not need tons more memory \nthan that. Since the window is normally 10, and you only have 8 objects, \nit really wants to have all eight in memory, but it shouldn't need a whole \nlot more.\n\nBut maybe it's really the case that you can't fit those 8 objects in \nmemory. One option (which might also solve some of the performance issues) \nis to make the window be based on object _size_ rather than just be a \nfixed number (ie with an 80MB object, you'd only try a couple of objects \naround it, not the full window).\n\n\t\tLinus\n"},{"id":"16992","messageId":"7v8xrtepje.fsf@assigned-by-dhcp.cox.net","threadId":"3507","inReplyTo":"20060301220840.GB18250@kroah.com","subject":"Re: git doesn't like big files when pushing","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-03-01T23:03:17Z","receivedAt":"2006-03-01T23:03:17Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Greg KH writes:\n\n> On Wed, Mar 01, 2006 at 02:08:02PM -0800, Greg KH wrote:\n>> I have a mail archive stored with git, in mbox form, and I made some\n>> changes to a few of the files and checked them back in.\n>...\n\nOuch. Running out of memory while deltifying sounds really bad.\n\n> Oh, and I'm using:\n> \t$ git --version\n> \tgit version 1.2.3.g8c2f\n>\n> if that helps or not.\n\nIt doen't, since I do not have 8c2fXXXX commit ;-).\n\nI suspect \"git push --thin origin\" might help, if you are on my\n\"master\" branch:\n\n diff-tree a79a276... (from 2245be3...)\n Author: Junio C Hamano \n Date: Mon Feb 20 00:09:41 2006 -0800\n\n Add git-push --thin.\n\n Maybe we would want to make this default before it graduates to\n the master branch, but in the meantime to help testing things,\n this allows you to say \"git push --thin destination\".\n\n Signed-off-by: Junio C Hamano \n\n :100755 100755 706db99... 73dcf06... M\tgit-push.sh\n"},{"id":"16994","messageId":"20060301232719.GA22068@kroah.com","threadId":"3507","inReplyTo":"7v8xrtepje.fsf@assigned-by-dhcp.cox.net","subject":"Re: git doesn't like big files when pushing","fromName":"Greg KH","fromEmail":"greg@kroah.com","sentAt":"2006-03-01T23:27:19Z","receivedAt":"2006-03-01T23:27:19Z","isPatch":false,"sender":{"key":"greg@kroah.com","avatar":"https://gravatar.com/avatar/5bb5aa0cc2e01c00ec899d11130c07796bc186e465bae57bc34873b13b72c7c8?d=mp&s=160"},"body":"On Wed, Mar 01, 2006 at 03:03:17PM -0800, Junio C Hamano wrote:\n> Greg KH writes:\n> \n> > On Wed, Mar 01, 2006 at 02:08:02PM -0800, Greg KH wrote:\n> >> I have a mail archive stored with git, in mbox form, and I made some\n> >> changes to a few of the files and checked them back in.\n> >...\n> \n> Ouch. Running out of memory while deltifying sounds really bad.\n> \n> > Oh, and I'm using:\n> > \t$ git --version\n> > \tgit version 1.2.3.g8c2f\n> >\n> > if that helps or not.\n> \n> It doen't, since I do not have 8c2fXXXX commit ;-).\n\nHeh, that's a merge on my tree, sorry. My tree only has one change to\nthe git-format-patch.sh script.\n\n858cbfbabe4ede5f5eba32041eb7448319e53e2a is the most recent commit from\nyour tree.\n\n> I suspect \"git push --thin origin\" might help, if you are on my\n> \"master\" branch:\n> \n> diff-tree a79a276... (from 2245be3...)\n> Author: Junio C Hamano \n> Date: Mon Feb 20 00:09:41 2006 -0800\n> \n> Add git-push --thin.\n> \n> Maybe we would want to make this default before it graduates to\n> the master branch, but in the meantime to help testing things,\n> this allows you to say \"git push --thin destination\".\n> \n> Signed-off-by: Junio C Hamano \n\nWill try that. I eventually gave up on the last push when it ran for 45\nminutes at full cpu usage, and X got killed by the OOM killer in the\nkernel for some reason...\n\nthanks,\n\ngreg k-h\n"},{"id":"16995","messageId":"20060301233506.GA25209@kroah.com","threadId":"3507","inReplyTo":"20060301232719.GA22068@kroah.com","subject":"Re: git doesn't like big files when pushing","fromName":"Greg KH","fromEmail":"greg@kroah.com","sentAt":"2006-03-01T23:35:06Z","receivedAt":"2006-03-01T23:35:06Z","isPatch":false,"sender":{"key":"greg@kroah.com","avatar":"https://gravatar.com/avatar/5bb5aa0cc2e01c00ec899d11130c07796bc186e465bae57bc34873b13b72c7c8?d=mp&s=160"},"body":"On Wed, Mar 01, 2006 at 03:27:19PM -0800, Greg KH wrote:\n> On Wed, Mar 01, 2006 at 03:03:17PM -0800, Junio C Hamano wrote:\n> > I suspect \"git push --thin origin\" might help, if you are on my\n> > \"master\" branch:\n> > \n> > diff-tree a79a276... (from 2245be3...)\n> > Author: Junio C Hamano \n> > Date: Mon Feb 20 00:09:41 2006 -0800\n> > \n> > Add git-push --thin.\n> > \n> > Maybe we would want to make this default before it graduates to\n> > the master branch, but in the meantime to help testing things,\n> > this allows you to say \"git push --thin destination\".\n> > \n> > Signed-off-by: Junio C Hamano \n> \n> Will try that. I eventually gave up on the last push when it ran for 45\n> minutes at full cpu usage, and X got killed by the OOM killer in the\n> kernel for some reason...\n\nNice, this worked!\n\nNow what's the odds that when I pull from the server to another box\nthese same objects, the server will have the same problem as git-push\ndid?\n\nwill go try that now...\n\nthanks,\n\ngreg k-h\n"},{"id":"16998","messageId":"20060302003418.GA11119@kroah.com","threadId":"3507","inReplyTo":"20060301233506.GA25209@kroah.com","subject":"Re: git doesn't like big files when pushing","fromName":"Greg KH","fromEmail":"greg@kroah.com","sentAt":"2006-03-02T00:34:18Z","receivedAt":"2006-03-02T00:34:18Z","isPatch":false,"sender":{"key":"greg@kroah.com","avatar":"https://gravatar.com/avatar/5bb5aa0cc2e01c00ec899d11130c07796bc186e465bae57bc34873b13b72c7c8?d=mp&s=160"},"body":"On Wed, Mar 01, 2006 at 03:35:06PM -0800, Greg KH wrote:\n> On Wed, Mar 01, 2006 at 03:27:19PM -0800, Greg KH wrote:\n> > On Wed, Mar 01, 2006 at 03:03:17PM -0800, Junio C Hamano wrote:\n> > > I suspect \"git push --thin origin\" might help, if you are on my\n> > > \"master\" branch:\n> > > \n> > > diff-tree a79a276... (from 2245be3...)\n> > > Author: Junio C Hamano \n> > > Date: Mon Feb 20 00:09:41 2006 -0800\n> > > \n> > > Add git-push --thin.\n> > > \n> > > Maybe we would want to make this default before it graduates to\n> > > the master branch, but in the meantime to help testing things,\n> > > this allows you to say \"git push --thin destination\".\n> > > \n> > > Signed-off-by: Junio C Hamano \n> > \n> > Will try that. I eventually gave up on the last push when it ran for 45\n> > minutes at full cpu usage, and X got killed by the OOM killer in the\n> > kernel for some reason...\n> \n> Nice, this worked!\n> \n> Now what's the odds that when I pull from the server to another box\n> these same objects, the server will have the same problem as git-push\n> did?\n\nOk, no problem there either.\n\nthanks for the pointer to that option.\n\ngreg k-h\n"},{"id":"17000","messageId":"46a038f90603011645v7d039b1bif3da9ebb3fa3e1c1@mail.gmail.com","threadId":"3507","inReplyTo":"20060302003418.GA11119@kroah.com","subject":"Re: git doesn't like big files when pushing","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-03-02T00:45:34Z","receivedAt":"2006-03-02T00:45:34Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 3/2/06, Greg KH wrote:\n> Ok, no problem there either.\n>\n> thanks for the pointer to that option.\n\nNow that it has been tested twice, it's about time it becomes the default ;-)\n\ncheers,\n\n\nmartin\n"},{"id":"17029","messageId":"4406E85A.9030908@hogyros.de","threadId":"3507","inReplyTo":"Pine.LNX.4.64.0603011448230.22647@g5.osdl.org","subject":"Re: git doesn't like big files when pushing","fromName":"Simon Richter","fromEmail":"simon.richter@hogyros.de","sentAt":"2006-03-02T12:43:06Z","receivedAt":"2006-03-02T12:43:06Z","isPatch":false,"sender":{"key":"simon.richter@hogyros.de","avatar":"https://gravatar.com/avatar/1192aa9fa5dd19ce258b12b044cc27111dd7cdb58d920dc2123a02a24f55b5c5?d=mp&s=160"},"body":"Hi,\n\nLinus Torvalds wrote:\n\n> But maybe it's really the case that you can't fit those 8 objects in \n> memory. One option (which might also solve some of the performance issues) \n> is to make the window be based on object _size_ rather than just be a \n> fixed number (ie with an 80MB object, you'd only try a couple of objects \n> around it, not the full window).\n\nWell, doesn't the pack process involve keeping objects of similar size \nin memory next to each other? My suspicion is that the system is running \nout of virtual address space because almost every malloc() will attempt \nto get an entirely new block because the one that was just freed is a \nfew bytes too small.\n\n Simon\n"}]}