{"thread":{"id":"11200","subject":"dmalloc and leaks in git","startedAt":"2007-12-08T20:53:44Z","lastAt":"2007-12-10T16:54:17Z","messageCount":7,"participants":["Jon Smirl","Johannes Schindelin","Linus Torvalds","Nicolas Pitre"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"62429","messageId":"9e4733910712081253t7cd43f87o6001f32fddc01565@mail.gmail.com","threadId":"11200","inReplyTo":null,"subject":"dmalloc and leaks in git","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-12-08T20:53:44Z","receivedAt":"2007-12-08T20:53:44Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"It is very easy to use dmalloc with git. Follow the instructions here,\nhttp://dmalloc.com/docs/latest/online/dmalloc_4.html\n\nBut using dmalloc shows a pervasive problem, none of the git commands\nare cleaning up after themselves. For example I ran a simple command,\ngit-status, and thousands of objects were not freed.\n\nNormally this doesn't hurt since exiting the process obviously frees\nall of the memory. But when programming this way it becomes impossible\nto tell which leaks were on purpose and which were accidental.\n\nTo sort this out an #ifdef DMALLOC needs to be created and then code\nfor freeing all of the 'on purpose' leaks needs to be added in an\nIFDEF right before the process exits. The test scripts can then be\nmodified to ensure that everything is freed when the command exits.\n\nI've used this process on several projects I've managed and it is a\nvery good thing to do. Once the new infrastructure is in place leaks\ncan be detected automatically and nipped in the bud before they get\nout of control. The key to making this work is getting code in place\nin the #ifdef to free those \"on-purpose\" leaks.\n\nI tried a couple of year ago to add leak detection to Mozilla but\nMozilla is way too far gone. There are 10,000 places where things are\nallocated and not being freed. It is a huge manually intensive task\nsorting out which of these were on-purpose vs accidental. If Mozilla\nhad followed a discipline of ensuring that nothing was every leaked\n(by using the scheme above) a lot of recent leak clean up work could\nhave been avoided.\n\nI don't know enough about the structure of git to add the cleanups in\n#ifdefs before exit. People who wrote the commands are going to have\nto help out with this.\n\ndiff --git a/Makefile b/Makefile\nindex 0a5df7a..426830c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -752,7 +752,7 @@ SHELL_PATH_SQ = $(subst ','\\'',$(SHELL_PATH))\n PERL_PATH_SQ = $(subst ','\\'',$(PERL_PATH))\n TCLTK_PATH_SQ = $(subst ','\\'',$(TCLTK_PATH))\n\n-LIBS = $(GITLIBS) $(EXTLIBS)\n+LIBS = $(GITLIBS) $(EXTLIBS) -ldmalloc\n\n BASIC_CFLAGS += -DSHA1_HEADER='$(SHA1_HEADER_SQ)' \\\n \t$(COMPAT_CFLAGS)\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 79eb10e..8894c30 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -428,3 +428,5 @@ static inline int strtol_i(char const *s, int\nbase, int *result)\n }\n\n #endif\n+\n+#include \"dmalloc.h\"\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"62430","messageId":"Pine.LNX.4.64.0712082058280.27959@racer.site","threadId":"11200","inReplyTo":"9e4733910712081253t7cd43f87o6001f32fddc01565@mail.gmail.com","subject":"Re: dmalloc and leaks in git","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-12-08T20:58:50Z","receivedAt":"2007-12-08T20:58:50Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 8 Dec 2007, Jon Smirl wrote:\n\n> It is very easy to use dmalloc with git. Follow the instructions here,\n> http://dmalloc.com/docs/latest/online/dmalloc_4.html\n> \n> But using dmalloc shows a pervasive problem, none of the git commands\n> are cleaning up after themselves. For example I ran a simple command,\n> git-status, and thousands of objects were not freed.\n\nKnown problem.  Goes by the name of \"libification\" on this list.\n\nCiao,\nDscho\n"},{"id":"62431","messageId":"9e4733910712081302h64d9b062kad02c4f13818d59f@mail.gmail.com","threadId":"11200","inReplyTo":"Pine.LNX.4.64.0712082058280.27959@racer.site","subject":"Re: dmalloc and leaks in git","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-12-08T21:02:37Z","receivedAt":"2007-12-08T21:02:37Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 12/8/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n> Hi,\n>\n> On Sat, 8 Dec 2007, Jon Smirl wrote:\n>\n> > It is very easy to use dmalloc with git. Follow the instructions here,\n> > http://dmalloc.com/docs/latest/online/dmalloc_4.html\n> >\n> > But using dmalloc shows a pervasive problem, none of the git commands\n> > are cleaning up after themselves. For example I ran a simple command,\n> > git-status, and thousands of objects were not freed.\n>\n> Known problem.  Goes by the name of \"libification\" on this list.\n\nI tried using dmalloc to find the leak in repack but it is impossible\nto sort out the accidental leaks from the on-purpose ones. On exit\nthere were millions of unfreed objects coming from all over the place.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"62432","messageId":"Pine.LNX.4.64.0712082118210.27959@racer.site","threadId":"11200","inReplyTo":"9e4733910712081302h64d9b062kad02c4f13818d59f@mail.gmail.com","subject":"Re: dmalloc and leaks in git","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-12-08T21:19:29Z","receivedAt":"2007-12-08T21:19:29Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 8 Dec 2007, Jon Smirl wrote:\n\n> On 12/8/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n> > Hi,\n> >\n> > On Sat, 8 Dec 2007, Jon Smirl wrote:\n> >\n> > > It is very easy to use dmalloc with git. Follow the instructions here,\n> > > http://dmalloc.com/docs/latest/online/dmalloc_4.html\n> > >\n> > > But using dmalloc shows a pervasive problem, none of the git commands\n> > > are cleaning up after themselves. For example I ran a simple command,\n> > > git-status, and thousands of objects were not freed.\n> >\n> > Known problem.  Goes by the name of \"libification\" on this list.\n> \n> I tried using dmalloc to find the leak in repack but it is impossible\n> to sort out the accidental leaks from the on-purpose ones. On exit\n> there were millions of unfreed objects coming from all over the place.\n\nThis might be a starting point:\n\nhttp://repo.or.cz/w/git/dscho.git?a=commitdiff;h=2083418c5010f04fbcd6e1f67de522ad6acd863d\n\nHth,\nDscho\n"},{"id":"62522","messageId":"alpine.LFD.0.9999.0712091251330.12046@woody.linux-foundation.org","threadId":"11200","inReplyTo":"9e4733910712081253t7cd43f87o6001f32fddc01565@mail.gmail.com","subject":"Re: dmalloc and leaks in git","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-12-09T20:57:02Z","receivedAt":"2007-12-09T20:57:02Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 8 Dec 2007, Jon Smirl wrote:\n> \n> But using dmalloc shows a pervasive problem, none of the git commands\n> are cleaning up after themselves. For example I ran a simple command,\n> git-status, and thousands of objects were not freed.\n\nOne thing to do is to use a better reporting tool.\n\nFor example, if you use\n\n\tvalgrind --tool=massif --heap=yes ...\n\nit will generate a postscript file with the allocation history as a graph \nof the different allocators in different colors etc. That would likely \nshow where the big users come from..\n\n\t\tLinus\n"},{"id":"62593","messageId":"alpine.LFD.0.9999.0712100828590.12046@woody.linux-foundation.org","threadId":"11200","inReplyTo":"alpine.LFD.0.9999.0712091251330.12046@woody.linux-foundation.org","subject":"Re: dmalloc and leaks in git","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-12-10T16:34:37Z","receivedAt":"2007-12-10T16:34:37Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sun, 9 Dec 2007, Linus Torvalds wrote:\n> \n> For example, if you use\n> \n> \tvalgrind --tool=massif --heap=yes ...\n\nI tried this on my copy of the gcc thing, but I didn't do the extreme \npacking thing, so I never saw the 3.4GB usage. Massif just reported a 200M \nheap, and about half of that was \"add_object_entry\".\n\nOf course, that doesn't report any mmap usage at all, so it totally \nignores the mapping of the original pack-file itself (which will obviously \nbe totally dense by the end, since we look at all objects).\n\nIt also doesn't take into account various secondary effects. For example, \nI don't think it looks at heap fragmentation issues etc, which normally \naren't a noticeable thing, but maybe some particular allocation pattern \ncan make the glibc allocator waste horrid amounts of memory or something \nlike that.\n\n\t\t\tLinus\n"},{"id":"62596","messageId":"alpine.LFD.0.99999.0712101148250.555@xanadu.home","threadId":"11200","inReplyTo":"alpine.LFD.0.9999.0712100828590.12046@woody.linux-foundation.org","subject":"Re: dmalloc and leaks in git","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-12-10T16:54:17Z","receivedAt":"2007-12-10T16:54:17Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 10 Dec 2007, Linus Torvalds wrote:\n\n> \n> \n> On Sun, 9 Dec 2007, Linus Torvalds wrote:\n> > \n> > For example, if you use\n> > \n> > \tvalgrind --tool=massif --heap=yes ...\n> \n> I tried this on my copy of the gcc thing, but I didn't do the extreme \n> packing thing, so I never saw the 3.4GB usage. Massif just reported a 200M \n> heap, and about half of that was \"add_object_entry\".\n\nSo far, it seems that the problem occurs much more severely when you run \n'git repack -a -f' while using the already highly packed gcc repo as a \nstarting point.\n\nRemains to determine if it occurs only when the repack is threaded, or \nif that has no significance.\n\n\nNicolas\n"}]}