{"thread":{"id":"31825","subject":"error: git-fast-import died of signal 11","startedAt":"2012-10-15T15:53:57Z","lastAt":"2012-10-17T20:06:10Z","messageCount":8,"participants":["Uri Moszkowicz","Andrew Wong","Michael Haggerty"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"201267","messageId":"CAMJd5AQ_vsQBGnMRrZUUqztjYjaHkU0_FOteNpEvE8NTrPPvQQ@mail.gmail.com","threadId":"31825","inReplyTo":"CAMJd5ATv5XfTK++4=Rs+RUkgb7F-ssrz2Lrch_WxvxZt+yF33A@mail.gmail.com","subject":"error: git-fast-import died of signal 11","fromName":"Uri Moszkowicz","fromEmail":"uri@4refs.com","sentAt":"2012-10-15T15:53:57Z","receivedAt":"2012-10-15T15:53:57Z","isPatch":false,"sender":{"key":"uri@4refs.com","avatar":null},"body":"Hi,\nI'm trying to convert a CVS repository to Git using cvs2git. I was able to\ngenerate the dump file without problem but am unable to get Git to\nfast-import it. The dump file is 328GB and I ran git fast-import on a\nmachine with 512GB of RAM.\n\nfatal: Out of memory? mmap failed: Cannot allocate memory\nfast-import: dumping crash report to fast_import_crash_18192\nerror: git-fast-import died of signal 11\n\nHow can I import the repository?\n\nThanks,\nUri\n"},{"id":"201289","messageId":"507C7C35.7080906@gmail.com","threadId":"31825","inReplyTo":"CAMJd5AQ_vsQBGnMRrZUUqztjYjaHkU0_FOteNpEvE8NTrPPvQQ@mail.gmail.com","subject":"Re: error: git-fast-import died of signal 11","fromName":"Andrew Wong","fromEmail":"andrew.kw.w@gmail.com","sentAt":"2012-10-15T21:12:21Z","receivedAt":"2012-10-15T21:12:21Z","isPatch":false,"sender":{"key":"andrew.kw.w@gmail.com","avatar":"https://avatars.githubusercontent.com/u/489311?v=4"},"body":"On 10/15/2012 11:53 AM, Uri Moszkowicz wrote:\n> I'm trying to convert a CVS repository to Git using cvs2git. I was able to\n> generate the dump file without problem but am unable to get Git to\n> fast-import it. The dump file is 328GB and I ran git fast-import on a\n> machine with 512GB of RAM.\nJust taking a wild guess here. Are you using 64bit version of git? If \nnot, maybe it'd help to try 64bit?\n\n> fatal: Out of memory? mmap failed: Cannot allocate memory\n> fast-import: dumping crash report to fast_import_crash_18192\n> error: git-fast-import died of signal 11\n\"fast-import\" also produced a crash report. It might help to diagnose \nthe issue if you can post that report?\nThe report shouldn't be too big. And you might want to strip any \nsensitive information before posting.\n"},{"id":"201290","messageId":"CAMJd5ARTmud_nrUKF8USXJAKVwvLH1gSmhys2o_mjimTW5B+Vg@mail.gmail.com","threadId":"31825","inReplyTo":"507C7C35.7080906@gmail.com","subject":"Re: error: git-fast-import died of signal 11","fromName":"Uri Moszkowicz","fromEmail":"uri@4refs.com","sentAt":"2012-10-15T21:28:22Z","receivedAt":"2012-10-15T21:28:22Z","isPatch":false,"sender":{"key":"uri@4refs.com","avatar":null},"body":"Thanks for the reply. Yes I am using a 64-bit build of Git. The report\nis too large to attach to email so I've uploaded it here (~6MB tar.xz\nfile):\n\nhttp://www.tempfiles.net/download/201210/267447/fast_import_crash18192.html\n\nOn Mon, Oct 15, 2012 at 4:12 PM, Andrew Wong wrote:\n> On 10/15/2012 11:53 AM, Uri Moszkowicz wrote:\n>>\n>> I'm trying to convert a CVS repository to Git using cvs2git. I was able to\n>> generate the dump file without problem but am unable to get Git to\n>> fast-import it. The dump file is 328GB and I ran git fast-import on a\n>> machine with 512GB of RAM.\n>\n> Just taking a wild guess here. Are you using 64bit version of git? If not,\n> maybe it'd help to try 64bit?\n>\n>\n>> fatal: Out of memory? mmap failed: Cannot allocate memory\n>> fast-import: dumping crash report to fast_import_crash_18192\n>> error: git-fast-import died of signal 11\n>\n> \"fast-import\" also produced a crash report. It might help to diagnose the\n> issue if you can post that report?\n> The report shouldn't be too big. And you might want to strip any sensitive\n> information before posting.\n"},{"id":"201294","messageId":"507C95A3.8010704@gmail.com","threadId":"31825","inReplyTo":"CAMJd5ARTmud_nrUKF8USXJAKVwvLH1gSmhys2o_mjimTW5B+Vg@mail.gmail.com","subject":"Re: error: git-fast-import died of signal 11","fromName":"Andrew Wong","fromEmail":"andrew.kw.w@gmail.com","sentAt":"2012-10-15T23:00:51Z","receivedAt":"2012-10-15T23:00:51Z","isPatch":false,"sender":{"key":"andrew.kw.w@gmail.com","avatar":"https://avatars.githubusercontent.com/u/489311?v=4"},"body":"On 10/15/2012 05:28 PM, Uri Moszkowicz wrote:\n> Thanks for the reply. Yes I am using a 64-bit build of Git. The report\n> is too large to attach to email so I've uploaded it here (~6MB tar.xz\n> file):\n>\n> http://www.tempfiles.net/download/201210/267447/fast_import_crash18192.html\nHm, there are some blanks in the \"recent commands\" section:\n\n D someFile\n D someFile\n D someFile\n D someFile\n (blank here)\n reset refs/tags/someTag\n from :1000045763\n reset refs/heads/TAG.FIXUP\n * (blank here)\n\nThere should have been some commands there. Maybe that has something to \ndo with the crash? Would you be able to locate where this is in the cvs \ndump? You might be able to use those tag names around those lines to \nhelp locate them.\nWhat are those blanks supposed to be? Probably commits?\n"},{"id":"201316","messageId":"507D0A53.1030707@alum.mit.edu","threadId":"31825","inReplyTo":"CAMJd5AQ_vsQBGnMRrZUUqztjYjaHkU0_FOteNpEvE8NTrPPvQQ@mail.gmail.com","subject":"Re: error: git-fast-import died of signal 11","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2012-10-16T07:18:43Z","receivedAt":"2012-10-16T07:18:43Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 10/15/2012 05:53 PM, Uri Moszkowicz wrote:\n> I'm trying to convert a CVS repository to Git using cvs2git. I was able to\n> generate the dump file without problem but am unable to get Git to\n> fast-import it. The dump file is 328GB and I ran git fast-import on a\n> machine with 512GB of RAM.\n> \n> fatal: Out of memory? mmap failed: Cannot allocate memory\n> fast-import: dumping crash report to fast_import_crash_18192\n> error: git-fast-import died of signal 11\n> \n> How can I import the repository?\n\nWhat versions of git and of cvs2git are you using? If not the current\nversions, please try with the current versions.\n\nWhat is the nature of your repository (i.e., why is it so big)? Does it\nconsist of extremely large files? A very deep history? Extremely many\nbranches/tags? Extremely many files?\n\nDid you check whether the RAM usage of git-fast-import process was\ngrowing gradually to fill RAM while it was running vs. whether the usage\nseemed reasonable until it suddenly crashed?\n\nThere are a few obvious possibilities:\n\n0. There is some reason that too little of your computer's RAM is\navailable to git-fast-import (e.g., ulimit, other processes running at\nthe same time, much RAM being used as a ramdisk, etc).\n\n1. Your import is simply too big for git-fast-import to hold in memory\nthe accumulated things that it has to remember. I'm not familiar with\nthe internals of git-fast-import, but I believe that the main thing that\nit has to keep in RAM is the list of \"marks\" (references to git objects\nthat can be referred to later in the import). From your crash file, it\nlooks like there were about 350k marks loaded at the time of the crash.\n Supposing each mark is about 100 bytes, this would only amount to 35\nMb, which should not be a problem (*if* my assumptions are correct).\n\n2. Your import contains a gigantic object which individually is so big\nthat it overflows some component of the import. (I don't know whether\nlarge objects are handled streamily; they might be read into memory at\nsome point.) But since your computer had so much RAM this is hardly\nimaginable.\n\n3. git-fast-import has a memory leak and the accumulated memory leakage\nis exhausting your RAM.\n\n4. git-fast-import has some other kind of a bug.\n\n5. The contents of the dumpfile are corrupt in a way that is triggering\nthe problem. This could either be invalid input (e.g., an object that\nis reported to be quaggabytes large), or some invalid input that\ntriggers a bug in git-fast-import.\n\nIf (1), then you either need a bigger machine or git-fast-import needs\narchitectural changes.\n\nIf (2), then you either need a bigger machine or git-fast-import and/or\ngit needs architectural changes.\n\nIf (3), then it would be good to get more information about the problem\nso that the leak can be fixed. If this is the case, it might be\npossible to work around the problem by splitting the dumpfile into\nseveral parts and loading them one after the other (outputting the marks\nfrom one run and loading them into the next).\n\nIf (4) or (5), then it would be helpful to narrow down the problem. It\nmight be possible to do so by following the instructions in the cvs2svn\nFAQ [1] for systematically shrinking a test case to smaller size using\ndestroy_repository.py and shrink_test_case.py. If you can create a\nsmall repository that triggers the same problem, then there is a good\nchance that it is easy to fix.\n\nMichael\n(the cvs2git maintainer)\n\n[1] http://cvs2svn.tigris.org/faq.html#testcase\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\nhttp://softwareswirl.blogspot.com/\n"},{"id":"201383","messageId":"CAMJd5ATOZGrRUWQCey8KHY=JfOtZCmhzpvCBou2EpY1tfLLFZA@mail.gmail.com","threadId":"31825","inReplyTo":"507D0A53.1030707@alum.mit.edu","subject":"Re: error: git-fast-import died of signal 11","fromName":"Uri Moszkowicz","fromEmail":"uri@4refs.com","sentAt":"2012-10-16T19:27:50Z","receivedAt":"2012-10-16T19:27:50Z","isPatch":false,"sender":{"key":"uri@4refs.com","avatar":null},"body":"I'm using Git 1.8.0-rc2 and cvs2git version 2.5.0-dev (trunk). The\nrepository is almost 20 years old and should consist of mostly\nsmallish plain text files. We've been tagging every commit, in\naddition to for releases and development branches, so there's a lot of\ntags and branches. I didn't see the memory usage of the process before\nexiting but after ~3.5 hours in a subsequent run it seems to be using\nabout 8.5GB of virtual memory with a resident size of only .5GB, which\nshould have easily fit on the 512GB machine that I was using. I'm\ntrying on a 1TB machine now but it doesn't look like it'll make a\ndifference. There is no ram disk and I have exclusive access to the\nmachine so only from the OS, which is trivial. The only significant\nlimit from my environment would be on the stack:\n\n[umoszkow@mawhp5 ~] limit\ncputime unlimited\nfilesize unlimited\ndatasize unlimited\nstacksize 8000 kbytes\ncoredumpsize 0 kbytes\nmemoryuse unlimited\nvmemoryuse unlimited\ndescriptors 1024\nmemorylocked 32 kbytes\nmaxproc 8388608\n\nWould that result in an mmap error though? I'll try with unlimited\nstacksize and descriptors anyway.\n\nI don't think modifying the original repository or a clone of it is\npossible at this point but breaking up the import into a few steps may\nbe - will try that next if this fails.\n\nOn Tue, Oct 16, 2012 at 2:18 AM, Michael Haggerty wrote:\n> On 10/15/2012 05:53 PM, Uri Moszkowicz wrote:\n>> I'm trying to convert a CVS repository to Git using cvs2git. I was able to\n>> generate the dump file without problem but am unable to get Git to\n>> fast-import it. The dump file is 328GB and I ran git fast-import on a\n>> machine with 512GB of RAM.\n>>\n>> fatal: Out of memory? mmap failed: Cannot allocate memory\n>> fast-import: dumping crash report to fast_import_crash_18192\n>> error: git-fast-import died of signal 11\n>>\n>> How can I import the repository?\n>\n> What versions of git and of cvs2git are you using? If not the current\n> versions, please try with the current versions.\n>\n> What is the nature of your repository (i.e., why is it so big)? Does it\n> consist of extremely large files? A very deep history? Extremely many\n> branches/tags? Extremely many files?\n>\n> Did you check whether the RAM usage of git-fast-import process was\n> growing gradually to fill RAM while it was running vs. whether the usage\n> seemed reasonable until it suddenly crashed?\n>\n> There are a few obvious possibilities:\n>\n> 0. There is some reason that too little of your computer's RAM is\n> available to git-fast-import (e.g., ulimit, other processes running at\n> the same time, much RAM being used as a ramdisk, etc).\n>\n> 1. Your import is simply too big for git-fast-import to hold in memory\n> the accumulated things that it has to remember. I'm not familiar with\n> the internals of git-fast-import, but I believe that the main thing that\n> it has to keep in RAM is the list of \"marks\" (references to git objects\n> that can be referred to later in the import). From your crash file, it\n> looks like there were about 350k marks loaded at the time of the crash.\n> Supposing each mark is about 100 bytes, this would only amount to 35\n> Mb, which should not be a problem (*if* my assumptions are correct).\n>\n> 2. Your import contains a gigantic object which individually is so big\n> that it overflows some component of the import. (I don't know whether\n> large objects are handled streamily; they might be read into memory at\n> some point.) But since your computer had so much RAM this is hardly\n> imaginable.\n>\n> 3. git-fast-import has a memory leak and the accumulated memory leakage\n> is exhausting your RAM.\n>\n> 4. git-fast-import has some other kind of a bug.\n>\n> 5. The contents of the dumpfile are corrupt in a way that is triggering\n> the problem. This could either be invalid input (e.g., an object that\n> is reported to be quaggabytes large), or some invalid input that\n> triggers a bug in git-fast-import.\n>\n> If (1), then you either need a bigger machine or git-fast-import needs\n> architectural changes.\n>\n> If (2), then you either need a bigger machine or git-fast-import and/or\n> git needs architectural changes.\n>\n> If (3), then it would be good to get more information about the problem\n> so that the leak can be fixed. If this is the case, it might be\n> possible to work around the problem by splitting the dumpfile into\n> several parts and loading them one after the other (outputting the marks\n> from one run and loading them into the next).\n>\n> If (4) or (5), then it would be helpful to narrow down the problem. It\n> might be possible to do so by following the instructions in the cvs2svn\n> FAQ [1] for systematically shrinking a test case to smaller size using\n> destroy_repository.py and shrink_test_case.py. If you can create a\n> small repository that triggers the same problem, then there is a good\n> chance that it is easy to fix.\n>\n> Michael\n> (the cvs2git maintainer)\n>\n> [1] http://cvs2svn.tigris.org/faq.html#testcase\n>\n> --\n> Michael Haggerty\n> mhagger@alum.mit.edu\n> http://softwareswirl.blogspot.com/\n"},{"id":"201388","messageId":"CADgNjanD=gYF_J7onuY6gK2mXuj1XGjWWX28nb2aAYPfmWH3PQ@mail.gmail.com","threadId":"31825","inReplyTo":"CAMJd5AR2gsyymKhT_hK9=4bHbcVnn+qEaDSxrZeJL1dfbmDxTw@mail.gmail.com","subject":"Re: error: git-fast-import died of signal 11","fromName":"Andrew Wong","fromEmail":"andrew.kw.w@gmail.com","sentAt":"2012-10-16T20:12:12Z","receivedAt":"2012-10-16T20:12:12Z","isPatch":false,"sender":{"key":"andrew.kw.w@gmail.com","avatar":"https://avatars.githubusercontent.com/u/489311?v=4"},"body":"On Tue, Oct 16, 2012 at 3:41 PM, Uri Moszkowicz wrote:\n> I can do that if it still fails tomorrow. How do I build a debug version of git?\n>\n> On Tue, Oct 16, 2012 at 2:35 PM, Andrew Wong wrote:\n>> Yea, it's a difficult problem to diagnose. It'd be really helpful if\n>> you can run a debug version of git and run the import process under a\n>> debugger.\n\nAfter getting git's source, you can simply run \"make\", and it'll\ncompile with debug info by default. When compiling is done, you will\nsee all the binaries in the source's root folder. Then, from the\nsource folder, you can start gdb by the command:\n gdb ./git-fast-import\n\nWhen you're inside gdb, put a breakpoint on \"die_nicely\" by entering:\n b die_nicely\n\nThen, you can finally run your import process by entering:\n r < your_cvs_dump\n\nWhen fast-import crashes/dies, you can find the stacktrace by entering:\n bt\n\nAnd that should tell us where it crashed, and, hopefully, where the\nmemory error happened.\n"},{"id":"201446","messageId":"CAMJd5AS92QtPyJ0sFWSQYtd-SSoZLHtR_NidRTBuNzaxv9t94A@mail.gmail.com","threadId":"31825","inReplyTo":"507D0A53.1030707@alum.mit.edu","subject":"Re: error: git-fast-import died of signal 11","fromName":"Uri Moszkowicz","fromEmail":"uri@4refs.com","sentAt":"2012-10-17T20:06:10Z","receivedAt":"2012-10-17T20:06:10Z","isPatch":false,"sender":{"key":"uri@4refs.com","avatar":null},"body":"Hi Michael,\nLooks like the changes to limit solved the problem. I didn't verify if\nit was the stacksize or descriptors but one of those. Final repository\nsize was 14GB from a 328GB dump file.\n\nThanks,\nUri\n\nOn Tue, Oct 16, 2012 at 2:18 AM, Michael Haggerty wrote:\n> On 10/15/2012 05:53 PM, Uri Moszkowicz wrote:\n>> I'm trying to convert a CVS repository to Git using cvs2git. I was able to\n>> generate the dump file without problem but am unable to get Git to\n>> fast-import it. The dump file is 328GB and I ran git fast-import on a\n>> machine with 512GB of RAM.\n>>\n>> fatal: Out of memory? mmap failed: Cannot allocate memory\n>> fast-import: dumping crash report to fast_import_crash_18192\n>> error: git-fast-import died of signal 11\n>>\n>> How can I import the repository?\n>\n> What versions of git and of cvs2git are you using? If not the current\n> versions, please try with the current versions.\n>\n> What is the nature of your repository (i.e., why is it so big)? Does it\n> consist of extremely large files? A very deep history? Extremely many\n> branches/tags? Extremely many files?\n>\n> Did you check whether the RAM usage of git-fast-import process was\n> growing gradually to fill RAM while it was running vs. whether the usage\n> seemed reasonable until it suddenly crashed?\n>\n> There are a few obvious possibilities:\n>\n> 0. There is some reason that too little of your computer's RAM is\n> available to git-fast-import (e.g., ulimit, other processes running at\n> the same time, much RAM being used as a ramdisk, etc).\n>\n> 1. Your import is simply too big for git-fast-import to hold in memory\n> the accumulated things that it has to remember. I'm not familiar with\n> the internals of git-fast-import, but I believe that the main thing that\n> it has to keep in RAM is the list of \"marks\" (references to git objects\n> that can be referred to later in the import). From your crash file, it\n> looks like there were about 350k marks loaded at the time of the crash.\n> Supposing each mark is about 100 bytes, this would only amount to 35\n> Mb, which should not be a problem (*if* my assumptions are correct).\n>\n> 2. Your import contains a gigantic object which individually is so big\n> that it overflows some component of the import. (I don't know whether\n> large objects are handled streamily; they might be read into memory at\n> some point.) But since your computer had so much RAM this is hardly\n> imaginable.\n>\n> 3. git-fast-import has a memory leak and the accumulated memory leakage\n> is exhausting your RAM.\n>\n> 4. git-fast-import has some other kind of a bug.\n>\n> 5. The contents of the dumpfile are corrupt in a way that is triggering\n> the problem. This could either be invalid input (e.g., an object that\n> is reported to be quaggabytes large), or some invalid input that\n> triggers a bug in git-fast-import.\n>\n> If (1), then you either need a bigger machine or git-fast-import needs\n> architectural changes.\n>\n> If (2), then you either need a bigger machine or git-fast-import and/or\n> git needs architectural changes.\n>\n> If (3), then it would be good to get more information about the problem\n> so that the leak can be fixed. If this is the case, it might be\n> possible to work around the problem by splitting the dumpfile into\n> several parts and loading them one after the other (outputting the marks\n> from one run and loading them into the next).\n>\n> If (4) or (5), then it would be helpful to narrow down the problem. It\n> might be possible to do so by following the instructions in the cvs2svn\n> FAQ [1] for systematically shrinking a test case to smaller size using\n> destroy_repository.py and shrink_test_case.py. If you can create a\n> small repository that triggers the same problem, then there is a good\n> chance that it is easy to fix.\n>\n> Michael\n> (the cvs2git maintainer)\n>\n> [1] http://cvs2svn.tigris.org/faq.html#testcase\n>\n> --\n> Michael Haggerty\n> mhagger@alum.mit.edu\n> http://softwareswirl.blogspot.com/\n"}]}