{"thread":{"id":"12726","subject":"Excessive memory consumption.","startedAt":"2008-03-17T15:43:53Z","lastAt":"2008-03-17T17:04:50Z","messageCount":2,"participants":["David Brown","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"72297","messageId":"20080317154353.GA9062@old.davidb.org","threadId":"12726","inReplyTo":null,"subject":"Excessive memory consumption.","fromName":"David Brown","fromEmail":"git@davidb.org","sentAt":"2008-03-17T15:43:53Z","receivedAt":"2008-03-17T15:43:53Z","isPatch":false,"sender":{"key":"git@davidb.org","avatar":"https://gravatar.com/avatar/94c86a2938470a74c2eac5e2b69afc0871f79a660295c02219597aba8cb101c1?d=mp&s=160"},"body":"I'm working with a repository containing a large number of revisions (it\nhas the full 2.6 kernel history) along with a bunch of large tarfiles of\nprebuilt images (basically firmware images).  These tarfiles are about 50MB\nand are updated frequently.\n\nI'm trying to figure out how to limit memory consumption when fetching from\nthis repo.  The repo lives on a network drive, and during the\n\n   \"remote: Compressing objects: ...\"\n\nphase, git-pack-objects quickly grows to 3 or more GB in size.  I've tried\nsetting  pack.windowMemory = 512m, and pack.deltaCacheLimit = 512m.  Any\nother suggested things to try?\n\nThis is running 1.5.4.3.\n\nThanks,\nDavid\n"},{"id":"72302","messageId":"alpine.LFD.1.00.0803170956430.3020@woody.linux-foundation.org","threadId":"12726","inReplyTo":"20080317154353.GA9062@old.davidb.org","subject":"Re: Excessive memory consumption.","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-03-17T17:04:50Z","receivedAt":"2008-03-17T17:04:50Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 17 Mar 2008, David Brown wrote:\n> \n> phase, git-pack-objects quickly grows to 3 or more GB in size.  I've tried\n> setting  pack.windowMemory = 512m, and pack.deltaCacheLimit = 512m.  Any\n> other suggested things to try?\n\nThis *might* be the malloc() fragmentation issue. If you have lots of \nlarge files with deltas in between them, you might well end up with a lot \nof the allocations having been 'free()'d, but even if you have \"only\" one \ngig in active use, there might be lots of holes in the allocation patterns \nwith \"free\" memory.\n\nDoes lowering the limits more aggressively make any difference (ie try \nwith those limits set to just a few tens of megs?) It will cause much \nworse deltas (since now the delta machinery won't work well on those big \ntar-files), so you probably don't want to do this for real, but as a test \nof whether it's those allocations or soemthing else it would still be \ninteresting.\n\nBut using valgrind (--tool=massif) to show the real allocations might also \ngive an even better view. Maybe valgrind tells you only 1 gig is used, and \nthen yes, it would be about some kind of fragmentation issue. Or maybe \nvalgrind points out some other memory user entirely..\n\n\t\tLinus\n"}]}