{"thread":{"id":"15619","subject":"out of memory problem","startedAt":"2008-09-21T22:59:30Z","lastAt":"2008-09-24T16:45:44Z","messageCount":4,"participants":["Guo Tang","Andreas Ericsson","guo tang"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"91275","messageId":"alpine.LFD.2.00.0809211556030.19373@homelinux","threadId":"15619","inReplyTo":null,"subject":"out of memory problem","fromName":"Guo Tang","fromEmail":"tangguo77@gmail.com","sentAt":"2008-09-21T22:59:30Z","receivedAt":"2008-09-21T22:59:30Z","isPatch":false,"sender":{"key":"tangguo77@gmail.com","avatar":null},"body":"Gentlemen,\n\nI try to run \"git gc\" on linux kernel tree. The virtual memory keeps \ngoing up until over 3GB, then crash. \nTried twice with the v1.6.0.2, same result.\nThen I used the git coming with FC9 (v1.5.5.1), the peak virutal memory \nusage is about 1.5GB. \"git gc\" finished without any trouble. \n\nCould there be a memory leak in v1.6.0.2?\n\nThanks,\nGuo\n"},{"id":"91283","messageId":"48D749F7.3030008@op5.se","threadId":"15619","inReplyTo":"alpine.LFD.2.00.0809211556030.19373@homelinux","subject":"Re: out of memory problem","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2008-09-22T07:32:07Z","receivedAt":"2008-09-22T07:32:07Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Guo Tang wrote:\n> Gentlemen,\n> \n> I try to run \"git gc\" on linux kernel tree. The virtual memory keeps \n> going up until over 3GB, then crash. \n> Tried twice with the v1.6.0.2, same result.\n> Then I used the git coming with FC9 (v1.5.5.1), the peak virutal memory \n> usage is about 1.5GB. \"git gc\" finished without any trouble. \n> \n> Could there be a memory leak in v1.6.0.2?\n> \n\nThere could be, but most likely it's commit\n38bd64979a2a3ffa178af801c6a62e6fcd658274 (Enable threaded delta\nsearch on BSD and Linux). Do you have multiple cpu's in the\ncomputer where 'git gc' was running? If so, and if you've set\npack.threads = 0, or --threads=0 it will autodetect the number\nof CPU's you have and then saturate all of them with work. Each\nthread will however consume memory close to that of a single\nprocess running the repack, so for large repositories you might\nwant to set pack.threads = 1 in such large repositories.\n\nIt's a shame you didn't save the unpacked repository, or this could\nhave been properly debugged. While it's possible there is a memory\nleak, it's a dismal project trying to locate it by staring at the\ncode, and the time it takes to repack huge repositories with memory\nintensive parameters is sort of prohibitive for finding the possible\nleak by bisection.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"},{"id":"91362","messageId":"48D89FB7.3000509@op5.se","threadId":"15619","inReplyTo":"4120f6ec0809220828i3b38eda3tfc4974df8a2568cb@mail.gmail.com","subject":"Re: out of memory problem","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2008-09-23T07:50:15Z","receivedAt":"2008-09-23T07:50:15Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"guo tang wrote:\n> On Mon, Sep 22, 2008 at 12:32 AM, Andreas Ericsson wrote:\n> \n>> Guo Tang wrote:\n>>\n>>> Gentlemen,\n>>>\n>>> I try to run \"git gc\" on linux kernel tree. The virtual memory keeps going\n>>> up until over 3GB, then crash. Tried twice with the v1.6.0.2, same result.\n>>> Then I used the git coming with FC9 (v1.5.5.1), the peak virutal memory\n>>> usage is about 1.5GB. \"git gc\" finished without any trouble.\n>>> Could there be a memory leak in v1.6.0.2?\n>>>\n>>>\n>> There could be, but most likely it's commit\n>> 38bd64979a2a3ffa178af801c6a62e6fcd658274 (Enable threaded delta\n>> search on BSD and Linux). Do you have multiple cpu's in the\n>> computer where 'git gc' was running? If so, and if you've set\n>> pack.threads = 0, or --threads=0 it will autodetect the number\n>> of CPU's you have and then saturate all of them with work. Each\n>> thread will however consume memory close to that of a single\n>> process running the repack, so for large repositories you might\n>> want to set pack.threads = 1 in such large repositories.\n> \n> \n> It is a Pentium M single core machine. But I am not sure whether it is using\n> just a single thread or\n> multiple threads. I will try setting pack.threads parameter next I run into\n> trouble.\n> \n\nUnless you explicitly told it to run multiple threads (which\nwould be a bit silly on a single-core machine), it just ran\none thread.\n\n>> It's a shame you didn't save the unpacked repository, or this could\n>> have been properly debugged. While it's possible there is a memory\n>> leak, it's a dismal project trying to locate it by staring at the\n>> code, and the time it takes to repack huge repositories with memory\n>> intensive parameters is sort of prohibitive for finding the possible\n>> leak by bisection.\n> \n> \n> Yes, the repository is already packed now. One question, beside the\n> bisecting method, do we have\n> this ability built into kernel:\n> 1. Turn a flag on for a process.\n> 2. OS will keep track off process malloc(), free() calls and the call stack.\n> \n> 3. For the malloc() calls without the the free() call (a memory leak), OS\n> will keep it count based on malloc() call stack.\n> 4. After some time, be able to dump this information out based on biggest\n> leak spot.\n> \n\nNo, there's not. The kernel isn't the one handing out the memory when you\ncall malloc(). That's handled by the C library, which can (and usually does)\nallocate a larger area of memory than the application needs, so that it\ndoesn't have to run a system call for every malloc() call you do.\n\nYou can pre-load a different memory allocator though, which can do whatever\nit wants with calls to malloc(), including ofcourse logging which function\ncalled them and how much memory was requested.\n\nGoogle for \"memory leak check linux\" and you'll get something like 750000\nresults.\n\n\n> The complain when I ran out of memory if from mmap failure. Is it the same\n> as malloc() failure?\n> \n\nSort of. Read 'man 2 mmap' for a more exhaustive description.\n\n> This kind of tool is available in Windows with its umdh (user mode heap\n> dump) tool.\n> \n\nThere are a number of tools to detect leaks under Linux/Unix as well.\nvalgrind is probably the most frequently used of all such leak checkers.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"},{"id":"91492","messageId":"4120f6ec0809240945t7b9313eagb1aa53f67ba9a76c@mail.gmail.com","threadId":"15619","inReplyTo":"48D89FB7.3000509@op5.se","subject":"Re: out of memory problem","fromName":"guo tang","fromEmail":"tangguo77@gmail.com","sentAt":"2008-09-24T16:45:44Z","receivedAt":"2008-09-24T16:45:44Z","isPatch":false,"sender":{"key":"tangguo77@gmail.com","avatar":null},"body":"It happened again. I cannot make a copy of repository and try bisect.\nThe repository is 2.5GB, too time consuming to make a copy.\n\nWhen I tried bisect without start from fresh. The bisect failed to\nfind the bad commit since the bad commit it found out can work now\nonce the repository is packed.\n\nI did notice 3 things.\n\n1. When I have memory trouble, \"git gc\" cannot finish the count\nobjects stage. It appears to be in an infinite loop.\n2. When I ran \"git count-objects -v\", it reported there is an error: 1\ngarbage object found. The garbage object is named something like\n.git/objects/35/tmp_xxx. Even if \"git gc\" finished without error, it\nwon't delete that garbage object. I guess there might be some\ndifference on how garbage object is handled between v1.5.5.1 and\nv1.6.2\n3. If I manually remove that garbage object using \"rm\", everything\nseems still fine.\n\nI will give valgrid a shoot next time I am in trouble.\n\nThanks,\nGuo\n\nOn Tue, Sep 23, 2008 at 12:50 AM, Andreas Ericsson wrote:\n> guo tang wrote:\n>>\n>> On Mon, Sep 22, 2008 at 12:32 AM, Andreas Ericsson wrote:\n>>\n>>> Guo Tang wrote:\n>>>\n>>>> Gentlemen,\n>>>>\n>>>> I try to run \"git gc\" on linux kernel tree. The virtual memory keeps\n>>>> going\n>>>> up until over 3GB, then crash. Tried twice with the v1.6.0.2, same\n>>>> result.\n>>>> Then I used the git coming with FC9 (v1.5.5.1), the peak virutal memory\n>>>> usage is about 1.5GB. \"git gc\" finished without any trouble.\n>>>> Could there be a memory leak in v1.6.0.2?\n>>>>\n>>>>\n>>> There could be, but most likely it's commit\n>>> 38bd64979a2a3ffa178af801c6a62e6fcd658274 (Enable threaded delta\n>>> search on BSD and Linux). Do you have multiple cpu's in the\n>>> computer where 'git gc' was running? If so, and if you've set\n>>> pack.threads = 0, or --threads=0 it will autodetect the number\n>>> of CPU's you have and then saturate all of them with work. Each\n>>> thread will however consume memory close to that of a single\n>>> process running the repack, so for large repositories you might\n>>> want to set pack.threads = 1 in such large repositories.\n>>\n>>\n>> It is a Pentium M single core machine. But I am not sure whether it is\n>> using\n>> just a single thread or\n>> multiple threads. I will try setting pack.threads parameter next I run\n>> into\n>> trouble.\n>>\n>\n> Unless you explicitly told it to run multiple threads (which\n> would be a bit silly on a single-core machine), it just ran\n> one thread.\n>\n>>> It's a shame you didn't save the unpacked repository, or this could\n>>> have been properly debugged. While it's possible there is a memory\n>>> leak, it's a dismal project trying to locate it by staring at the\n>>> code, and the time it takes to repack huge repositories with memory\n>>> intensive parameters is sort of prohibitive for finding the possible\n>>> leak by bisection.\n>>\n>>\n>> Yes, the repository is already packed now. One question, beside the\n>> bisecting method, do we have\n>> this ability built into kernel:\n>> 1. Turn a flag on for a process.\n>> 2. OS will keep track off process malloc(), free() calls and the call\n>> stack.\n>>\n>> 3. For the malloc() calls without the the free() call (a memory leak), OS\n>> will keep it count based on malloc() call stack.\n>> 4. After some time, be able to dump this information out based on biggest\n>> leak spot.\n>>\n>\n> No, there's not. The kernel isn't the one handing out the memory when you\n> call malloc(). That's handled by the C library, which can (and usually does)\n> allocate a larger area of memory than the application needs, so that it\n> doesn't have to run a system call for every malloc() call you do.\n>\n> You can pre-load a different memory allocator though, which can do whatever\n> it wants with calls to malloc(), including ofcourse logging which function\n> called them and how much memory was requested.\n>\n> Google for \"memory leak check linux\" and you'll get something like 750000\n> results.\n>\n>\n>> The complain when I ran out of memory if from mmap failure. Is it the same\n>> as malloc() failure?\n>>\n>\n> Sort of. Read 'man 2 mmap' for a more exhaustive description.\n>\n>> This kind of tool is available in Windows with its umdh (user mode heap\n>> dump) tool.\n>>\n>\n> There are a number of tools to detect leaks under Linux/Unix as well.\n> valgrind is probably the most frequently used of all such leak checkers.\n>\n> --\n> Andreas Ericsson andreas.ericsson@op5.se\n> OP5 AB www.op5.se\n> Tel: +46 8-230225 Fax: +46 8-230231\n>\n"}]}