{"thread":{"id":"11180","subject":"RAM consumption when working with the gcc repo","startedAt":"2007-12-07T20:07:05Z","lastAt":"2007-12-08T19:12:07Z","messageCount":15,"participants":["Jon Smirl","david@lang.hm","Marco Costalba","Nicolas Pitre","Jeff King","Johannes Schindelin","Martin Koegler"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"62311","messageId":"9e4733910712071207p750c14f4h7abc5d637da3a478@mail.gmail.com","threadId":"11180","inReplyTo":null,"subject":"RAM consumption when working with the gcc repo","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-12-07T20:07:05Z","receivedAt":"2007-12-07T20:07:05Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"I noticed two things when doing a repack of the gcc repo. First is\nthat the git process is getting to be way too big. Turning off the\ndelta caches had minimal impact. Why does the process still grow to\n4.8GB?\n\nPutting this in perspective, this is a 4.8GB process constructing a\n330MB file. Something isn't right. Memory leak or inefficient data\nstructure?\n\nThe second issue is that the repack process slows way down on the last\n10% of the packing process. I don't believe this was caused by\nswapping since my disk light wasn't on. It takes a long to do the last\n10% as it did for the first 70%. This seems to be correlated with the\nsize of the process getting so large.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"62315","messageId":"e5bfff550712071236t26fe77cchf75ca64adf6a33@mail.gmail.com","threadId":"11180","inReplyTo":"Pine.LNX.4.64.0712071323260.12607@asgard.lang.hm","subject":"Re: RAM consumption when working with the gcc repo","fromName":"Marco Costalba","fromEmail":"mcostalba@gmail.com","sentAt":"2007-12-07T20:36:38Z","receivedAt":"2007-12-07T20:36:38Z","isPatch":false,"sender":{"key":"mcostalba@gmail.com","avatar":null},"body":"On Dec 7, 2007 10:24 PM, wrote:\n> On Fri, 7 Dec 2007, Jon Smirl wrote:\n>\n> keep in mind that that 330MB file is _very_ heavily compressed. the simple\n> zlib compression is probably getting you 10:1 or 20:1 compression and the\n> delta compression is a significant multiplier on top of that.\n>\n\nIf the delta is good the zlib is poor and the contrary stands too.\n\nIt is very difficult to _guess_ in this cases.\n\nMarco\n"},{"id":"62316","messageId":"alpine.LFD.0.99999.0712071529580.555@xanadu.home","threadId":"11180","inReplyTo":"Pine.LNX.4.64.0712071323260.12607@asgard.lang.hm","subject":"Re: RAM consumption when working with the gcc repo","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-12-07T20:46:30Z","receivedAt":"2007-12-07T20:46:30Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 7 Dec 2007, david@lang.hm wrote:\n\n> On Fri, 7 Dec 2007, Jon Smirl wrote:\n> \n> > I noticed two things when doing a repack of the gcc repo. First is\n> > that the git process is getting to be way too big. Turning off the\n> > delta caches had minimal impact. Why does the process still grow to\n> > 4.8GB?\n> > \n> > Putting this in perspective, this is a 4.8GB process constructing a\n> > 330MB file. Something isn't right. Memory leak or inefficient data\n> > structure?\n> \n> keep in mind that that 330MB file is _very_ heavily compressed. the simple\n> zlib compression is probably getting you 10:1 or 20:1 compression and the\n> delta compression is a significant multiplier on top of that.\n\nDoesn't matter. Something is indeed fishy.\n\nThe bulk of pack-objects memory consumption can be estimated as follows:\n\n1M objects * sizeof(struct object_entry) ~= 100MB\n256 window entries with data (assuming a big 1MB per entry) = 256MB\nDelta result caching was disabled therefore 0MB\nread-side delta cache limited to 16MB\n\nSo the purely ram allocation might get to roughly 400MB.\n\nThen add the pack and index map, which, depending on the original pack \nsize,\nmight be 2GB.\n\nSo we're pessimistically talking of about 2.5GB of virtual space.\n\nThe other 2.3GB is hard to explain.\n\n\nNicolas\n"},{"id":"62319","messageId":"9e4733910712071323w5edccdb1pb5d6b05eb00ec97c@mail.gmail.com","threadId":"11180","inReplyTo":"alpine.LFD.0.99999.0712071529580.555@xanadu.home","subject":"Re: RAM consumption when working with the gcc repo","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-12-07T21:23:35Z","receivedAt":"2007-12-07T21:23:35Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 12/7/07, Nicolas Pitre wrote:\n> On Fri, 7 Dec 2007, david@lang.hm wrote:\n>\n> > On Fri, 7 Dec 2007, Jon Smirl wrote:\n> >\n> > > I noticed two things when doing a repack of the gcc repo. First is\n> > > that the git process is getting to be way too big. Turning off the\n> > > delta caches had minimal impact. Why does the process still grow to\n> > > 4.8GB?\n> > >\n> > > Putting this in perspective, this is a 4.8GB process constructing a\n> > > 330MB file. Something isn't right. Memory leak or inefficient data\n> > > structure?\n> >\n> > keep in mind that that 330MB file is _very_ heavily compressed. the simple\n> > zlib compression is probably getting you 10:1 or 20:1 compression and the\n> > delta compression is a significant multiplier on top of that.\n>\n> Doesn't matter. Something is indeed fishy.\n>\n> The bulk of pack-objects memory consumption can be estimated as follows:\n>\n> 1M objects * sizeof(struct object_entry) ~= 100MB\n> 256 window entries with data (assuming a big 1MB per entry) = 256MB\n> Delta result caching was disabled therefore 0MB\n> read-side delta cache limited to 16MB\n>\n> So the purely ram allocation might get to roughly 400MB.\n>\n> Then add the pack and index map, which, depending on the original pack\n> size,\n> might be 2GB.\n\nI'm repacking the heavily compress pack, so input pack and index are\nabout 360MB, not 2GB.\n\n>\n> So we're pessimistically talking of about 2.5GB of virtual space.\n>\n> The other 2.3GB is hard to explain.\n\nMore like 3.5MB that is hard to explain.\n\nIs there a simple way to tell what percent is mmap vs anon allocation?\n\n\n>\n>\n> Nicolas\n>\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"62313","messageId":"Pine.LNX.4.64.0712071323260.12607@asgard.lang.hm","threadId":"11180","inReplyTo":"9e4733910712071207p750c14f4h7abc5d637da3a478@mail.gmail.com","subject":"Re: RAM consumption when working with the gcc repo","fromName":"","fromEmail":"david@lang.hm","sentAt":"2007-12-07T21:24:41Z","receivedAt":"2007-12-07T21:24:41Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Fri, 7 Dec 2007, Jon Smirl wrote:\n\n> I noticed two things when doing a repack of the gcc repo. First is\n> that the git process is getting to be way too big. Turning off the\n> delta caches had minimal impact. Why does the process still grow to\n> 4.8GB?\n>\n> Putting this in perspective, this is a 4.8GB process constructing a\n> 330MB file. Something isn't right. Memory leak or inefficient data\n> structure?\n\nkeep in mind that that 330MB file is _very_ heavily compressed. the simple \nzlib compression is probably getting you 10:1 or 20:1 compression and the \ndelta compression is a significant multiplier on top of that.\n\nDavid Lang\n\n> The second issue is that the repack process slows way down on the last\n> 10% of the packing process. I don't believe this was caused by\n> swapping since my disk light wasn't on. It takes a long to do the last\n> 10% as it did for the first 70%. This seems to be correlated with the\n> size of the process getting so large.\n>\n>\n"},{"id":"62322","messageId":"e5bfff550712071325k3d4dadf2gf34c069022cc005@mail.gmail.com","threadId":"11180","inReplyTo":"alpine.LFD.0.99999.0712071529580.555@xanadu.home","subject":"Re: RAM consumption when working with the gcc repo","fromName":"Marco Costalba","fromEmail":"mcostalba@gmail.com","sentAt":"2007-12-07T21:25:50Z","receivedAt":"2007-12-07T21:25:50Z","isPatch":false,"sender":{"key":"mcostalba@gmail.com","avatar":null},"body":"On Dec 7, 2007 9:46 PM, Nicolas Pitre wrote:\n>\n> The other 2.3GB is hard to explain.\n>\n\nBTW does exist a tool to profile memory consumption by each source\nlevel struct / vector/ or any other data container?\n\nValgrind checks mainly memory leaks, callgrind gives profiling\ninformation in terms of call graphs and times/cycles consumed by each\nfunction.\n\nWhat I _really_ would like it's a tool that allows me to *easily*\ncheck how much memory is used in a given point in time by each data\ncontainer at source level.\n\nSomething like this:\n\n\nAt checkpoint \"trigger_now\":\n\nstruct my_data is instantiated 120234 times\nstruct super_delta is instantiated 100000 times\n\n\nAt checkpoint \"trigger_also_now\":\n\nstruct my_data is instantiated 12 times\nstruct super_delta is instantiated 70 times\n\n.....\n\n\nThat would be AWSOME!!! a super debugging killing tool!\n\nThanks\nMarco\n"},{"id":"62324","messageId":"9e4733910712071327u4b74e8a0idcd601dd2dcc13f8@mail.gmail.com","threadId":"11180","inReplyTo":"alpine.LFD.0.99999.0712071529580.555@xanadu.home","subject":"Re: RAM consumption when working with the gcc repo","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-12-07T21:27:13Z","receivedAt":"2007-12-07T21:27:13Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 12/7/07, Nicolas Pitre wrote:\n> On Fri, 7 Dec 2007, david@lang.hm wrote:\n>\n> > On Fri, 7 Dec 2007, Jon Smirl wrote:\n> >\n> > > I noticed two things when doing a repack of the gcc repo. First is\n> > > that the git process is getting to be way too big. Turning off the\n> > > delta caches had minimal impact. Why does the process still grow to\n> > > 4.8GB?\n> > >\n> > > Putting this in perspective, this is a 4.8GB process constructing a\n> > > 330MB file. Something isn't right. Memory leak or inefficient data\n> > > structure?\n> >\n> > keep in mind that that 330MB file is _very_ heavily compressed. the simple\n> > zlib compression is probably getting you 10:1 or 20:1 compression and the\n> > delta compression is a significant multiplier on top of that.\n>\n> Doesn't matter. Something is indeed fishy.\n\nI didn't have any problem repacking Mozilla and it ends up as a 450MB\npack file with 1.5M entries. So something has changed. With Mozilla I\nhad a 3GB machine, and now I can't finish a 330MB pack on a 4GB\nmachine. I don't recall the Mozilla process ever exceeding 2GB.\n\n\n>\n> The bulk of pack-objects memory consumption can be estimated as follows:\n>\n> 1M objects * sizeof(struct object_entry) ~= 100MB\n> 256 window entries with data (assuming a big 1MB per entry) = 256MB\n> Delta result caching was disabled therefore 0MB\n> read-side delta cache limited to 16MB\n>\n> So the purely ram allocation might get to roughly 400MB.\n>\n> Then add the pack and index map, which, depending on the original pack\n> size,\n> might be 2GB.\n>\n> So we're pessimistically talking of about 2.5GB of virtual space.\n>\n> The other 2.3GB is hard to explain.\n>\n>\n> Nicolas\n>\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"62330","messageId":"20071207213928.GA11613@coredump.intra.peff.net","threadId":"11180","inReplyTo":"9e4733910712071207p750c14f4h7abc5d637da3a478@mail.gmail.com","subject":"Re: RAM consumption when working with the gcc repo","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2007-12-07T21:39:28Z","receivedAt":"2007-12-07T21:39:28Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 07, 2007 at 03:07:05PM -0500, Jon Smirl wrote:\n\n> I noticed two things when doing a repack of the gcc repo. First is\n> that the git process is getting to be way too big. Turning off the\n> delta caches had minimal impact. Why does the process still grow to\n> 4.8GB?\n> \n> Putting this in perspective, this is a 4.8GB process constructing a\n> 330MB file. Something isn't right. Memory leak or inefficient data\n> structure?\n\nKeep in mind that you are trying many different deltas, which are being\nheld in memory, to find the right one and generate the 330MB file. And\nwhen you multiply that times N threads going at once, _each one_ is\nusing a bunch of memory.\n\nAs Nico suggested, you could probably drop the memory usage by reducing\nthe size of the delta cache.\n\n-Peff\n"},{"id":"62331","messageId":"9e4733910712071339o5b1efd37g8ff471e767c0ae76@mail.gmail.com","threadId":"11180","inReplyTo":"alpine.LFD.0.99999.0712071529580.555@xanadu.home","subject":"Re: RAM consumption when working with the gcc repo","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-12-07T21:39:44Z","receivedAt":"2007-12-07T21:39:44Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"Here's a big clue.\n\nWhen I repack the 300MB file the process grows to 4.8GB\nWhen I repack the 2,000MB file the process grows to 3.3GB\n\nIn both cases the last 10% of the repack is taking as much time as the\nfirst 90%.\n\nAt the end I am packing 60 objects/sec. In the beginning i was packing\n1,000s of objects per second.\n\nI'm not swapping\n\njonsmirl@terra:/video/gcc/.git/objects/pack/foo$ vmstat 1\nprocs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----\n r b swpd free buff cache si so bi bo in cs us sy id wa\n 2 0 1416 25668 3904 2756404 0 0 62 45 115 398 6 0 93 1\n 3 0 1416 26880 3900 2754852 0 0 0 0 414 2453 26 1 73 0\n 2 0 1416 26880 3900 2754852 0 0 0 0 472 3518 26 1 73 0\n 4 0 1416 26912 3900 2754768 0 0 0 0 394 1642 26 1 74 0\n 2 0 1416 26912 3900 2754768 0 0 0 0 401 1364 25 0 75 0\n 2 0 1416 26896 3900 2754768 0 0 0 0 456 1922 25 1 75 0\n\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"62332","messageId":"20071207214049.GB11613@coredump.intra.peff.net","threadId":"11180","inReplyTo":"20071207213928.GA11613@coredump.intra.peff.net","subject":"Re: RAM consumption when working with the gcc repo","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2007-12-07T21:40:49Z","receivedAt":"2007-12-07T21:40:49Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 07, 2007 at 04:39:28PM -0500, Jeff King wrote:\n\n> Keep in mind that you are trying many different deltas, which are being\n> held in memory, to find the right one and generate the 330MB file. And\n> when you multiply that times N threads going at once, _each one_ is\n> using a bunch of memory.\n> \n> As Nico suggested, you could probably drop the memory usage by reducing\n> the size of the delta cache.\n\nSorry, I clearly need to start reading to the ends of threads before\ngetting involved; I think Nico has already explained this with actual\nnumbers later on.\n\n-Peff\n"},{"id":"62333","messageId":"9e4733910712071343k2d88d2fck1fa3fa333258e8e3@mail.gmail.com","threadId":"11180","inReplyTo":"20071207213928.GA11613@coredump.intra.peff.net","subject":"Re: RAM consumption when working with the gcc repo","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-12-07T21:43:58Z","receivedAt":"2007-12-07T21:43:58Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 12/7/07, Jeff King wrote:\n> On Fri, Dec 07, 2007 at 03:07:05PM -0500, Jon Smirl wrote:\n>\n> > I noticed two things when doing a repack of the gcc repo. First is\n> > that the git process is getting to be way too big. Turning off the\n> > delta caches had minimal impact. Why does the process still grow to\n> > 4.8GB?\n> >\n> > Putting this in perspective, this is a 4.8GB process constructing a\n> > 330MB file. Something isn't right. Memory leak or inefficient data\n> > structure?\n>\n> Keep in mind that you are trying many different deltas, which are being\n> held in memory, to find the right one and generate the 330MB file. And\n> when you multiply that times N threads going at once, _each one_ is\n> using a bunch of memory.\n>\n> As Nico suggested, you could probably drop the memory usage by reducing\n> the size of the delta cache.\n\nDelta cache is disabled --\n\npack.deltacachelimit = 0\n\nUnless this option is broken?\n\n>\n> -Peff\n>\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"62336","messageId":"9e4733910712071350w3c0e12dfkf366d33d61b55266@mail.gmail.com","threadId":"11180","inReplyTo":"alpine.LFD.0.99999.0712071529580.555@xanadu.home","subject":"Re: RAM consumption when working with the gcc repo","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-12-07T21:50:31Z","receivedAt":"2007-12-07T21:50:31Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"This is for a 3.3GB process with the 2GB pack as input\nLooking at my process map, why is the pack file in the map four times?\n\n2ba1f703b000-2ba23703b000 r--p 00000000 09:01 33079321\n /video/gcc/.git/objects/pack/pack-bd163555ea9240a7fdd07d2708a293872665f48b.pack\n2ba23703b000-2ba23703c000 rw-p 2ba23703b000 00:00 0\n2ba237c86000-2ba239352000 r--p 80000000 09:01 33079321\n /video/gcc/.git/objects/pack/pack-bd163555ea9240a7fdd07d2708a293872665f48b.pack\n2ba2394b1000-2ba2794b1000 r--p 40000000 09:01 33079321\n /video/gcc/.git/objects/pack/pack-bd163555ea9240a7fdd07d2708a293872665f48b.pack\n2ba2794b1000-2ba27a4b2000 rw-p 2ba2794b1000 00:00 0\n2ba27bcb2000-2ba281c29000 rw-p 2ba23703c000 00:00 0\n2ba281c29000-2ba2a32f5000 r--p 60000000 09:01 33079321\n /video/gcc/.git/objects/pack/pack-bd163555ea9240a7fdd07d2708a293872665f48b.pack\n7fffb75e2000-7fffb75f7000 rw-p 7ffffffea000 00:00 0 [stack]\n7fffb75fe000-7fffb7600000 r-xp 7fffb75fe000 00:00 0 [vdso]\nffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0\n [vsyscall]\n\nHere's the heap:\n\n00400000-004b9000 r-xp 00000000 08:16 296588\n /usr/local/bin/git\n006b9000-006bd000 rw-p 000b9000 08:16 296588\n /usr/local/bin/git\n006bd000-0c17f000 rw-p 006bd000 00:00 0 [heap]\n40000000-40001000 ---p 40000000 00:00 0\n40001000-40801000 rw-p 40001000 00:00 0\n40801000-40802000 ---p 40801000 00:00 0\n40802000-41002000 rw-p 40802000 00:00 0\n41002000-41003000 ---p 41002000 00:00 0\n41003000-41803000 rw-p 41003000 00:00 0\n41803000-41804000 ---p 41803000 00:00 0\n41804000-42004000 rw-p 41804000 00:00 0\n\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"62407","messageId":"Pine.LNX.4.64.0712081153500.27959@racer.site","threadId":"11180","inReplyTo":"e5bfff550712071325k3d4dadf2gf34c069022cc005@mail.gmail.com","subject":"Re: RAM consumption when working with the gcc repo","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-12-08T11:54:17Z","receivedAt":"2007-12-08T11:54:17Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 7 Dec 2007, Marco Costalba wrote:\n\n> On Dec 7, 2007 9:46 PM, Nicolas Pitre wrote:\n> >\n> > The other 2.3GB is hard to explain.\n> \n> BTW does exist a tool to profile memory consumption by each source level \n> struct / vector/ or any other data container?\n> \n> Valgrind checks mainly memory leaks, callgrind gives profiling \n> information in terms of call graphs and times/cycles consumed by each \n> function.\n\nHave you looked at Massif (also part of Valgrind)?\n\nCiao,\nDscho\n"},{"id":"62423","messageId":"20071208172437.GA24442@auto.tuwien.ac.at","threadId":"11180","inReplyTo":"alpine.LFD.0.99999.0712071529580.555@xanadu.home","subject":"Re: RAM consumption when working with the gcc repo","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2007-12-08T17:24:37Z","receivedAt":"2007-12-08T17:24:37Z","isPatch":false,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"On Fri, Dec 07, 2007 at 03:46:30PM -0500, Nicolas Pitre wrote:\n> On Fri, 7 Dec 2007, david@lang.hm wrote:\n> > On Fri, 7 Dec 2007, Jon Smirl wrote:\n> > \n> > > I noticed two things when doing a repack of the gcc repo. First is\n> > > that the git process is getting to be way too big. Turning off the\n> > > delta caches had minimal impact. Why does the process still grow to\n> > > 4.8GB?\n> > > \n> > > Putting this in perspective, this is a 4.8GB process constructing a\n> > > 330MB file. Something isn't right. Memory leak or inefficient data\n> > > structure?\n> > \n> > keep in mind that that 330MB file is _very_ heavily compressed. the simple\n> > zlib compression is probably getting you 10:1 or 20:1 compression and the\n> > delta compression is a significant multiplier on top of that.\n> \n> Doesn't matter. Something is indeed fishy.\n> \n> The bulk of pack-objects memory consumption can be estimated as follows:\n> \n> 1M objects * sizeof(struct object_entry) ~= 100MB\n> 256 window entries with data (assuming a big 1MB per entry) = 256MB\n\nFor each (uncompressed) object in the delta window, a delta index is\ncreated. It can have the same size as the uncompressed object.\n\nEach thread has its own window, so using 4 threads means having 1024 objects \nin memory => 1 GB\n\n> Delta result caching was disabled therefore 0MB\n> read-side delta cache limited to 16MB\n> \n> So the purely ram allocation might get to roughly 400MB.\n> \n> Then add the pack and index map, which, depending on the original pack \n> size,\n> might be 2GB.\n> \n> So we're pessimistically talking of about 2.5GB of virtual space.\n> \n> The other 2.3GB is hard to explain.\n\nmfg Martin Kögler\n"},{"id":"62425","messageId":"e5bfff550712081112xf34eb1cua95da3f9397030f0@mail.gmail.com","threadId":"11180","inReplyTo":"Pine.LNX.4.64.0712081153500.27959@racer.site","subject":"Re: RAM consumption when working with the gcc repo","fromName":"Marco Costalba","fromEmail":"mcostalba@gmail.com","sentAt":"2007-12-08T19:12:07Z","receivedAt":"2007-12-08T19:12:07Z","isPatch":false,"sender":{"key":"mcostalba@gmail.com","avatar":null},"body":"On Dec 8, 2007 12:54 PM, Johannes Schindelin wrote:\n> >\n> > BTW does exist a tool to profile memory consumption by each source level\n> > struct / vector/ or any other data container?\n> >\n> > Valgrind checks mainly memory leaks, callgrind gives profiling\n> > information in terms of call graphs and times/cycles consumed by each\n> > function.\n>\n> Have you looked at Massif (also part of Valgrind)?\n>\n\nOnly very quickly, so probably I've missed something, but anyway\nthat's my comment on Massiv:\n\n- Interesting output is in html format, graph is nice but gives very\ngeneral info.\n\n- The tool mainly tracks who called malloc and friends also going back\nin the stack frame (in my box if I try to set stack deep at 4 instead\nof default 3 program crashes)\n\n- Does not seem to give information regarding the structures where\nmemory is allocated, only the function names that allocate directly or\nindirectly the memory. Nothing like\n\nstruct my_data has 34.256 instantiations\nstruct stuff has 2.456 instantiations\n\n\n- Relation between memory and time of allocation is IMHO a little bit\nconfusing, it's like a parameter obtained from multipling allocated\nbytes x time, not very useful.\n\n- Regarding the previous point, it seems missing a way to\ntrigger/snapshotting the memory map in terms of source level\nstructures used by the application in a given time, triggable through\ncode.\n\nJust to be clear, the much better callgrind tool allows to insert in\nthe code the macros\n\nCALLGRIND_START_INSTRUMENTATION and\nCALLGRIND_STOP_INSTRUMENTATION\n\nthat start/stop recording of events in the code ranges specified by\nthe developer. What it seems to be missing in massif it's a macro like\n\nMASSIV_SNAPSHOT_MEM_MAP\n\nto be used _where_ developer needs and that gives information on heap\nallocation in terms of source level entities that *use* that memory,\nnot low level addresses or allocator functions.\n\n\nThanks\nMarco\n"}]}