{"thread":{"id":"34746","subject":"git-p4 out of memory for very large repository","startedAt":"2013-08-23T01:12:45Z","lastAt":"2013-09-07T08:19:05Z","messageCount":13,"participants":["Corey Thompson","Luke Diamand","Pete Wyckoff"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"225706","messageId":"20130823011245.GA7693@jerec","threadId":"34746","inReplyTo":null,"subject":"git-p4 out of memory for very large repository","fromName":"Corey Thompson","fromEmail":"cmtptr@gmail.com","sentAt":"2013-08-23T01:12:45Z","receivedAt":"2013-08-23T01:12:45Z","isPatch":false,"sender":{"key":"cmtptr@gmail.com","avatar":null},"body":"Hello,\n\nHas anyone actually gotten git-p4 to clone a large Perforce repository?\nI have one codebase in particular that gets to about 67%, then\nconsistently gets get-fast-import (and often times a few other\nprocesses) killed by the OOM killer.\n\nI've found some patches out there that claim to resolve this, but\nthey're all for versions of git-p4.py from several years ago.  Not only\nwill they not apply cleanly, but as far as I can tell the issues that\nthese patches are meant to address aren't in the current version,\nanyway.\n\nAny suggestions would be greatly appreciated.\n\nThanks,\nCorey\n"},{"id":"225712","messageId":"52170C6A.4080708@diamand.org","threadId":"34746","inReplyTo":"20130823011245.GA7693@jerec","subject":"Re: git-p4 out of memory for very large repository","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2013-08-23T07:16:58Z","receivedAt":"2013-08-23T07:16:58Z","isPatch":false,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"On 23/08/13 02:12, Corey Thompson wrote:\n> Hello,\n>\n> Has anyone actually gotten git-p4 to clone a large Perforce repository?\n\nYes. I've cloned repos with a couple of Gig of files.\n\n> I have one codebase in particular that gets to about 67%, then\n> consistently gets get-fast-import (and often times a few other\n> processes) killed by the OOM killer.\n\nWhat size is this codebase? Which version and platform of git are you using?\n\nMaybe it's a regression, or perhaps you've hit some new, previously \nunknown size limit?\n\nThanks\nLuke\n\n\n>\n> I've found some patches out there that claim to resolve this, but\n> they're all for versions of git-p4.py from several years ago.  Not only\n> will they not apply cleanly, but as far as I can tell the issues that\n> these patches are meant to address aren't in the current version,\n> anyway.\n>\n> Any suggestions would be greatly appreciated.\n>\n> Thanks,\n> Corey\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"225716","messageId":"20130823114856.GA8182@jerec","threadId":"34746","inReplyTo":"52170C6A.4080708@diamand.org","subject":"Re: git-p4 out of memory for very large repository","fromName":"Corey Thompson","fromEmail":"cmtptr@gmail.com","sentAt":"2013-08-23T11:48:56Z","receivedAt":"2013-08-23T11:48:56Z","isPatch":false,"sender":{"key":"cmtptr@gmail.com","avatar":null},"body":"On Fri, Aug 23, 2013 at 08:16:58AM +0100, Luke Diamand wrote:\n> On 23/08/13 02:12, Corey Thompson wrote:\n> >Hello,\n> >\n> >Has anyone actually gotten git-p4 to clone a large Perforce repository?\n> \n> Yes. I've cloned repos with a couple of Gig of files.\n> \n> >I have one codebase in particular that gets to about 67%, then\n> >consistently gets get-fast-import (and often times a few other\n> >processes) killed by the OOM killer.\n> \n> What size is this codebase? Which version and platform of git are you using?\n> \n> Maybe it's a regression, or perhaps you've hit some new, previously\n> unknown size limit?\n> \n> Thanks\n> Luke\n> \n> \n> >\n> >I've found some patches out there that claim to resolve this, but\n> >they're all for versions of git-p4.py from several years ago.  Not only\n> >will they not apply cleanly, but as far as I can tell the issues that\n> >these patches are meant to address aren't in the current version,\n> >anyway.\n> >\n> >Any suggestions would be greatly appreciated.\n> >\n> >Thanks,\n> >Corey\n> >--\n> >To unsubscribe from this list: send the line \"unsubscribe git\" in\n> >the body of a message to majordomo@vger.kernel.org\n> >More majordomo info at  http://vger.kernel.org/majordomo-info.html\n> \n\nSorry, I guess I could have included more details in my original post.\nSince then, I have also made an attempt to clone another (slightly more\nrecent) branch, and at last had success.  So I see this does indeed\nwork, it just seems to be very unhappy with one particular branch.\n\nSo, here are a few statistics I collected on the two branches.\n\nbranch-that-fails:\ntotal workspace disk usage (current head): 12GB\n68 files over 20MB\nlargest three being about 118MB\n\nbranch-that-clones:\ntotal workspace disk usage (current head): 11GB\n22 files over 20MB\nlargest three being about 80MB\n\nI suspect that part of the problem here might be that my company likes\nto submit very large binaries into our repo (.tar.gzs, pre-compiled\nthird party binaries, etc.).\n\nIs there any way I can clone this in pieces?  The best I've come up with\nis to clone only up to a change number just before it tends to fail, and\nthen rebase to the latest.  My clone succeeded, but the rebase still\nruns out of memory.  It would be great if I could specify a change\nnumber to rebase up to, so that I can just take this thing a few hundred\nchanges at a time.\n\nThanks,\nCorey\n"},{"id":"225717","messageId":"20130823115920.GB8182@jerec","threadId":"34746","inReplyTo":"20130823114856.GA8182@jerec","subject":"Re: git-p4 out of memory for very large repository","fromName":"Corey Thompson","fromEmail":"cmtptr@gmail.com","sentAt":"2013-08-23T11:59:20Z","receivedAt":"2013-08-23T11:59:20Z","isPatch":false,"sender":{"key":"cmtptr@gmail.com","avatar":null},"body":"On Fri, Aug 23, 2013 at 07:48:56AM -0400, Corey Thompson wrote:\n> Sorry, I guess I could have included more details in my original post.\n> Since then, I have also made an attempt to clone another (slightly more\n> recent) branch, and at last had success.  So I see this does indeed\n> work, it just seems to be very unhappy with one particular branch.\n> \n> So, here are a few statistics I collected on the two branches.\n> \n> branch-that-fails:\n> total workspace disk usage (current head): 12GB\n> 68 files over 20MB\n> largest three being about 118MB\n> \n> branch-that-clones:\n> total workspace disk usage (current head): 11GB\n> 22 files over 20MB\n> largest three being about 80MB\n> \n> I suspect that part of the problem here might be that my company likes\n> to submit very large binaries into our repo (.tar.gzs, pre-compiled\n> third party binaries, etc.).\n> \n> Is there any way I can clone this in pieces?  The best I've come up with\n> is to clone only up to a change number just before it tends to fail, and\n> then rebase to the latest.  My clone succeeded, but the rebase still\n> runs out of memory.  It would be great if I could specify a change\n> number to rebase up to, so that I can just take this thing a few hundred\n> changes at a time.\n> \n> Thanks,\n> Corey\n\nAnd I still haven't told you anything about my platform or git\nversion...\n\nThis is on Fedora Core 11, with git 1.8.3.4 built from the github repo\n(117eea7e).\n"},{"id":"225739","messageId":"5217BB34.9080502@diamand.org","threadId":"34746","inReplyTo":"20130823115920.GB8182@jerec","subject":"Re: git-p4 out of memory for very large repository","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2013-08-23T19:42:44Z","receivedAt":"2013-08-23T19:42:44Z","isPatch":false,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"\nI think I've cloned files as large as that or larger. If you just want to\nclone this and move on, perhaps you just need a bit more memory? What's the\nsize of your physical memory and swap partition? Per process memory limit?\n\n\nOn 23 Aug 2013 12:59, \"Corey Thompson\" <cmtptr@gmail.com> wrote:\nOn 23/08/13 12:59, Corey Thompson wrote:\n> On Fri, Aug 23, 2013 at 07:48:56AM -0400, Corey Thompson wrote:\n>> Sorry, I guess I could have included more details in my original post.\n>> Since then, I have also made an attempt to clone another (slightly more\n>> recent) branch, and at last had success.  So I see this does indeed\n>> work, it just seems to be very unhappy with one particular branch.\n>>\n>> So, here are a few statistics I collected on the two branches.\n>>\n>> branch-that-fails:\n>> total workspace disk usage (current head): 12GB\n>> 68 files over 20MB\n>> largest three being about 118MB\n>>\n>> branch-that-clones:\n>> total workspace disk usage (current head): 11GB\n>> 22 files over 20MB\n>> largest three being about 80MB\n>>\n>> I suspect that part of the problem here might be that my company likes\n>> to submit very large binaries into our repo (.tar.gzs, pre-compiled\n>> third party binaries, etc.).\n>>\n>> Is there any way I can clone this in pieces?  The best I've come up with\n>> is to clone only up to a change number just before it tends to fail, and\n>> then rebase to the latest.  My clone succeeded, but the rebase still\n>> runs out of memory.  It would be great if I could specify a change\n>> number to rebase up to, so that I can just take this thing a few hundred\n>> changes at a time.\n>>\n>> Thanks,\n>> Corey\n> \n> And I still haven't told you anything about my platform or git\n> version...\n> \n> This is on Fedora Core 11, with git 1.8.3.4 built from the github repo\n> (117eea7e).\n"},{"id":"225764","messageId":"20130824005614.GC8182@jerec","threadId":"34746","inReplyTo":"5217BB34.9080502@diamand.org","subject":"Re: git-p4 out of memory for very large repository","fromName":"Corey Thompson","fromEmail":"cmtptr@gmail.com","sentAt":"2013-08-24T00:56:14Z","receivedAt":"2013-08-24T00:56:14Z","isPatch":false,"sender":{"key":"cmtptr@gmail.com","avatar":null},"body":"On Fri, Aug 23, 2013 at 08:42:44PM +0100, Luke Diamand wrote:\n> \n> I think I've cloned files as large as that or larger. If you just want to\n> clone this and move on, perhaps you just need a bit more memory? What's the\n> size of your physical memory and swap partition? Per process memory limit?\n> \n\nThe machine has 32GB of memory, so I hope that should be more than\nsufficient!\n\n$ ulimit -a\ncore file size          (blocks, -c) 0\ndata seg size           (kbytes, -d) unlimited\nscheduling priority             (-e) 0\nfile size               (blocks, -f) unlimited\npending signals                 (-i) 268288\nmax locked memory       (kbytes, -l) 64\nmax memory size         (kbytes, -m) unlimited\nopen files                      (-n) 1024\npipe size            (512 bytes, -p) 8\nPOSIX message queues     (bytes, -q) 819200\nreal-time priority              (-r) 0\nstack size              (kbytes, -s) 10240\ncpu time               (seconds, -t) unlimited\nmax user processes              (-u) 1024\nvirtual memory          (kbytes, -v) unlimited\nfile locks                      (-x) unlimited\n\nAdmittedly I don't typically look at ulimit, so please excuse me if I\ninterpret this wrong, but I feel like this is indicating that the only\nartificial limit in place is a maximum of 64kB mlock()'d memory.\n\nThanks,\nCorey\n"},{"id":"225867","messageId":"20130825155001.GA875@padd.com","threadId":"34746","inReplyTo":"20130823114856.GA8182@jerec","subject":"Re: git-p4 out of memory for very large repository","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2013-08-25T15:50:01Z","receivedAt":"2013-08-25T15:50:01Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"cmtptr@gmail.com wrote on Fri, 23 Aug 2013 07:48 -0400:\n> On Fri, Aug 23, 2013 at 08:16:58AM +0100, Luke Diamand wrote:\n> > On 23/08/13 02:12, Corey Thompson wrote:\n> > >Hello,\n> > >\n> > >Has anyone actually gotten git-p4 to clone a large Perforce repository?\n> > \n> > Yes. I've cloned repos with a couple of Gig of files.\n> > \n> > >I have one codebase in particular that gets to about 67%, then\n> > >consistently gets get-fast-import (and often times a few other\n> > >processes) killed by the OOM killer.\n[..]\n> Sorry, I guess I could have included more details in my original post.\n> Since then, I have also made an attempt to clone another (slightly more\n> recent) branch, and at last had success.  So I see this does indeed\n> work, it just seems to be very unhappy with one particular branch.\n> \n> So, here are a few statistics I collected on the two branches.\n> \n> branch-that-fails:\n> total workspace disk usage (current head): 12GB\n> 68 files over 20MB\n> largest three being about 118MB\n> \n> branch-that-clones:\n> total workspace disk usage (current head): 11GB\n> 22 files over 20MB\n> largest three being about 80MB\n> \n> I suspect that part of the problem here might be that my company likes\n> to submit very large binaries into our repo (.tar.gzs, pre-compiled\n> third party binaries, etc.).\n> \n> Is there any way I can clone this in pieces?  The best I've come up with\n> is to clone only up to a change number just before it tends to fail, and\n> then rebase to the latest.  My clone succeeded, but the rebase still\n> runs out of memory.  It would be great if I could specify a change\n> number to rebase up to, so that I can just take this thing a few hundred\n> changes at a time.\n\nModern git, including your version, do \"streaming\" reads from p4,\nso the git-p4 python process never even holds a whole file's\nworth of data.  You're seeing git-fast-import die, it seems.  It\nwill hold onto the entire file contents.  But just one, not the\nentire repo.  How big is the single largest file?\n\nYou can import in pieces.  See the change numbers like this:\n\n    p4 changes -m 1000 //depot/big/...\n    p4 changes -m 1000 //depot/big/...@<some-old-change>\n\nImport something far enough back in history so that it seems\nto work:\n\n    git p4 clone --destination=big //depot/big@60602\n    cd big\n\nSync up a bit at a time:\n\n    git p4 sync @60700\n    git p4 sync @60800\n    ...\n\nI don't expect this to get around the problem you describe,\nhowever.  Sounds like there is one gigantic file that is causing\ngit-fast-import to fill all of memory.  You will at least isolate\nthe change.\n\nThere are options to git-fast-import to limit max pack size\nand to cause it to skip importing files that are too big, if\nthat would help.\n\nYou can also use a client spec to hide the offending files\nfrom git.\n\nCan you watch with \"top\"?  Hit \"M\" to sort by memory usage, and\nsee how big the processes get before falling over.\n\n\t\t-- Pete\n"},{"id":"225899","messageId":"20130826134756.GA1335@jerec","threadId":"34746","inReplyTo":"20130825155001.GA875@padd.com","subject":"Re: git-p4 out of memory for very large repository","fromName":"Corey Thompson","fromEmail":"cmtptr@gmail.com","sentAt":"2013-08-26T13:47:56Z","receivedAt":"2013-08-26T13:47:56Z","isPatch":false,"sender":{"key":"cmtptr@gmail.com","avatar":null},"body":"On Sun, Aug 25, 2013 at 11:50:01AM -0400, Pete Wyckoff wrote:\n> Modern git, including your version, do \"streaming\" reads from p4,\n> so the git-p4 python process never even holds a whole file's\n> worth of data.  You're seeing git-fast-import die, it seems.  It\n> will hold onto the entire file contents.  But just one, not the\n> entire repo.  How big is the single largest file?\n> \n> You can import in pieces.  See the change numbers like this:\n> \n>     p4 changes -m 1000 //depot/big/...\n>     p4 changes -m 1000 //depot/big/...@<some-old-change>\n> \n> Import something far enough back in history so that it seems\n> to work:\n> \n>     git p4 clone --destination=big //depot/big@60602\n>     cd big\n> \n> Sync up a bit at a time:\n> \n>     git p4 sync @60700\n>     git p4 sync @60800\n>     ...\n> \n> I don't expect this to get around the problem you describe,\n> however.  Sounds like there is one gigantic file that is causing\n> git-fast-import to fill all of memory.  You will at least isolate\n> the change.\n> \n> There are options to git-fast-import to limit max pack size\n> and to cause it to skip importing files that are too big, if\n> that would help.\n> \n> You can also use a client spec to hide the offending files\n> from git.\n> \n> Can you watch with \"top\"?  Hit \"M\" to sort by memory usage, and\n> see how big the processes get before falling over.\n> \n> \t\t-- Pete\n\nYou are correct that git-fast-import is killed by the OOM killer, but I\nwas unclear about which process was malloc()ing so much memory that the\nOOM killer got invoked (as other completely unrelated processes usually\nalso get killed when this happens).\n\nUnless there's one gigantic file in one change that gets removed by\nanother change, I don't think that's the problem; as I mentioned in\nanother email, the machine has 32GB physical memory and the largest\nsingle file in the current head is only 118MB.  Even if there is a very\nlarge transient file somewhere in the history, I seriously doubt it's\ntens of gigabytes in size.\n\nI have tried watching it with top before, but it takes several hours\nbefore it dies.  I haven't been able to see any explosion of memory\nusage, even within the final hour, but I've never caught it just before\nit dies, either.  I suspect that whatever the issue is here, it happens\nvery quickly.\n\nIf I'm unable to get through this today using the incremental p4 sync\nmethod you described, I'll try running a full-blown clone overnight with\ntop in batch mode writing to a log file to see whether it catches\nanything.\n\nThanks again,\nCorey\n"},{"id":"226091","messageId":"20130828154135.GA16921@jerec","threadId":"34746","inReplyTo":"20130826134756.GA1335@jerec","subject":"Re: git-p4 out of memory for very large repository","fromName":"Corey Thompson","fromEmail":"cmtptr@gmail.com","sentAt":"2013-08-28T15:41:35Z","receivedAt":"2013-08-28T15:41:35Z","isPatch":false,"sender":{"key":"cmtptr@gmail.com","avatar":null},"body":"On Mon, Aug 26, 2013 at 09:47:56AM -0400, Corey Thompson wrote:\n> You are correct that git-fast-import is killed by the OOM killer, but I\n> was unclear about which process was malloc()ing so much memory that the\n> OOM killer got invoked (as other completely unrelated processes usually\n> also get killed when this happens).\n> \n> Unless there's one gigantic file in one change that gets removed by\n> another change, I don't think that's the problem; as I mentioned in\n> another email, the machine has 32GB physical memory and the largest\n> single file in the current head is only 118MB.  Even if there is a very\n> large transient file somewhere in the history, I seriously doubt it's\n> tens of gigabytes in size.\n> \n> I have tried watching it with top before, but it takes several hours\n> before it dies.  I haven't been able to see any explosion of memory\n> usage, even within the final hour, but I've never caught it just before\n> it dies, either.  I suspect that whatever the issue is here, it happens\n> very quickly.\n> \n> If I'm unable to get through this today using the incremental p4 sync\n> method you described, I'll try running a full-blown clone overnight with\n> top in batch mode writing to a log file to see whether it catches\n> anything.\n> \n> Thanks again,\n> Corey\n\nUnforunately I have not made much progress.  The incremental sync method\nfails with the output pasted below.  The change I specified is only one\nchange number above where that repo was cloned...\n\nSo I tried a 'git p4 rebase' overnight with top running, and as I feared\nI did not see anything out of the ordinary.  git, git-fast-import, and\ngit-p4 all hovered under 1.5% MEM the entire time, right up until\ndeath.  The last entry in my log shows git-fast-import at 0.8%, with git\nand git-p4 at 0.0% and 0.1%, respectively.  I could try again with a\nmore granular period, but I feel like this method is ultimately a goose\nchase.\n\nCorey\n\n\n$ git p4 sync //path/to/some/branch@505859\nDoing initial import of //path/to/some/branch/ from revision @505859 into refs/remotes/p4/master\nfast-import failed: warning: Not updating refs/remotes/p4/master (new tip 29ef6ff25f1448fa2f907d22fd704594dc8769bd does not contain d477672be5ac6a00cc9175ba2713d5395660e840)\ngit-fast-import statistics:\n---------------------------------------------------------------------\nAlloc'd objects:     165000\nTotal objects:           69 (    232434 duplicates                  )\n      blobs  :           45 (    209904 duplicates         40 deltas of         42 attempts) \n      trees  :           23 (     22530 duplicates          0 deltas of         23 attempts) \n      commits:            1 (         0 duplicates          0 deltas of          0 attempts) \n      tags   :            0 (         0 duplicates          0 deltas of          0 attempts)\nTotal branches:           1 (         1 loads     )\n      marks:           1024 (         0 unique    )\n      atoms:         105170\nMemory total:         24421 KiB\n       pools:         17976 KiB\n     objects:          6445 KiB\n---------------------------------------------------------------------\npack_report: getpagesize()            =       4096\npack_report: core.packedGitWindowSize =   33554432\npack_report: core.packedGitLimit      =  268435456\npack_report: pack_used_ctr            =       4371\npack_report: pack_mmap_calls          =        124\npack_report: pack_open_windows        =          8 /          9\npack_report: pack_mapped              =  268435456 /  268435456\n---------------------------------------------------------------------\n"},{"id":"226280","messageId":"20130829224609.GB25879@padd.com","threadId":"34746","inReplyTo":"20130828154135.GA16921@jerec","subject":"Re: git-p4 out of memory for very large repository","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2013-08-29T22:46:09Z","receivedAt":"2013-08-29T22:46:09Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"cmtptr@gmail.com wrote on Wed, 28 Aug 2013 11:41 -0400:\n> On Mon, Aug 26, 2013 at 09:47:56AM -0400, Corey Thompson wrote:\n> > You are correct that git-fast-import is killed by the OOM killer, but I\n> > was unclear about which process was malloc()ing so much memory that the\n> > OOM killer got invoked (as other completely unrelated processes usually\n> > also get killed when this happens).\n> > \n> > Unless there's one gigantic file in one change that gets removed by\n> > another change, I don't think that's the problem; as I mentioned in\n> > another email, the machine has 32GB physical memory and the largest\n> > single file in the current head is only 118MB.  Even if there is a very\n> > large transient file somewhere in the history, I seriously doubt it's\n> > tens of gigabytes in size.\n> > \n> > I have tried watching it with top before, but it takes several hours\n> > before it dies.  I haven't been able to see any explosion of memory\n> > usage, even within the final hour, but I've never caught it just before\n> > it dies, either.  I suspect that whatever the issue is here, it happens\n> > very quickly.\n> > \n> > If I'm unable to get through this today using the incremental p4 sync\n> > method you described, I'll try running a full-blown clone overnight with\n> > top in batch mode writing to a log file to see whether it catches\n> > anything.\n> > \n> > Thanks again,\n> > Corey\n> \n> Unforunately I have not made much progress.  The incremental sync method\n> fails with the output pasted below.  The change I specified is only one\n> change number above where that repo was cloned...\n\nI usually just do \"git p4 sync @505859\".  The error message below\ncrops up when things get confused.  Usually after a previous\nerror.  I tend to destroy the repo and try again.  Sorry I don't\ncan't explain better what's happening here.  It's not a memory\nissue; it reports only 24 MB used.\n\n> So I tried a 'git p4 rebase' overnight with top running, and as I feared\n> I did not see anything out of the ordinary.  git, git-fast-import, and\n> git-p4 all hovered under 1.5% MEM the entire time, right up until\n> death.  The last entry in my log shows git-fast-import at 0.8%, with git\n> and git-p4 at 0.0% and 0.1%, respectively.  I could try again with a\n> more granular period, but I feel like this method is ultimately a goose\n> chase.\n\nBizarre.  There is no good explanation why memory usage would go\nup to 32 GB (?) within one top interval (3 sec ?).  My theory\nabout one gigantic object is debunked:  you have only the 118 MB\none.  Perhaps there's some container or process memory limit, as\nLuke guessed, but it's not obvious here.\n\nThe other big hammer is \"strace\".  If you're still interested in\nplaying with this, you could do:\n\n    strace -vf -tt -s 200 -o /tmp/strace.out git p4 clone ....\n\nand hours later, see if something suggests itself toward the\nend of that output file.\n\n\t\t-- Pete\n"},{"id":"226551","messageId":"5224EA2C.7090001@diamand.org","threadId":"34746","inReplyTo":"20130829224609.GB25879@padd.com","subject":"Re: git-p4 out of memory for very large repository","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2013-09-02T19:42:36Z","receivedAt":"2013-09-02T19:42:36Z","isPatch":false,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"I guess you could try changing the OOM score for git-fast-import.\n\nchange /proc/<pid>/oomadj.\n\nI think a value of -31 would make it very unlikely to be killed.\n\nOn 29/08/13 23:46, Pete Wyckoff wrote:\n> cmtptr@gmail.com wrote on Wed, 28 Aug 2013 11:41 -0400:\n>> On Mon, Aug 26, 2013 at 09:47:56AM -0400, Corey Thompson wrote:\n>>> You are correct that git-fast-import is killed by the OOM killer, but I\n>>> was unclear about which process was malloc()ing so much memory that the\n>>> OOM killer got invoked (as other completely unrelated processes usually\n>>> also get killed when this happens).\n>>>\n>>> Unless there's one gigantic file in one change that gets removed by\n>>> another change, I don't think that's the problem; as I mentioned in\n>>> another email, the machine has 32GB physical memory and the largest\n>>> single file in the current head is only 118MB.  Even if there is a very\n>>> large transient file somewhere in the history, I seriously doubt it's\n>>> tens of gigabytes in size.\n>>>\n>>> I have tried watching it with top before, but it takes several hours\n>>> before it dies.  I haven't been able to see any explosion of memory\n>>> usage, even within the final hour, but I've never caught it just before\n>>> it dies, either.  I suspect that whatever the issue is here, it happens\n>>> very quickly.\n>>>\n>>> If I'm unable to get through this today using the incremental p4 sync\n>>> method you described, I'll try running a full-blown clone overnight with\n>>> top in batch mode writing to a log file to see whether it catches\n>>> anything.\n>>>\n>>> Thanks again,\n>>> Corey\n>>\n>> Unforunately I have not made much progress.  The incremental sync method\n>> fails with the output pasted below.  The change I specified is only one\n>> change number above where that repo was cloned...\n>\n> I usually just do \"git p4 sync @505859\".  The error message below\n> crops up when things get confused.  Usually after a previous\n> error.  I tend to destroy the repo and try again.  Sorry I don't\n> can't explain better what's happening here.  It's not a memory\n> issue; it reports only 24 MB used.\n>\n>> So I tried a 'git p4 rebase' overnight with top running, and as I feared\n>> I did not see anything out of the ordinary.  git, git-fast-import, and\n>> git-p4 all hovered under 1.5% MEM the entire time, right up until\n>> death.  The last entry in my log shows git-fast-import at 0.8%, with git\n>> and git-p4 at 0.0% and 0.1%, respectively.  I could try again with a\n>> more granular period, but I feel like this method is ultimately a goose\n>> chase.\n>\n> Bizarre.  There is no good explanation why memory usage would go\n> up to 32 GB (?) within one top interval (3 sec ?).  My theory\n> about one gigantic object is debunked:  you have only the 118 MB\n> one.  Perhaps there's some container or process memory limit, as\n> Luke guessed, but it's not obvious here.\n>\n> The other big hammer is \"strace\".  If you're still interested in\n> playing with this, you could do:\n>\n>      strace -vf -tt -s 200 -o /tmp/strace.out git p4 clone ....\n>\n> and hours later, see if something suggests itself toward the\n> end of that output file.\n>\n> \t\t-- Pete\n"},{"id":"226977","messageId":"20130906190345.GA3239@jerec","threadId":"34746","inReplyTo":"5224EA2C.7090001@diamand.org","subject":"Re: git-p4 out of memory for very large repository","fromName":"Corey Thompson","fromEmail":"cmtptr@gmail.com","sentAt":"2013-09-06T19:03:45Z","receivedAt":"2013-09-06T19:03:45Z","isPatch":false,"sender":{"key":"cmtptr@gmail.com","avatar":null},"body":"On Mon, Sep 02, 2013 at 08:42:36PM +0100, Luke Diamand wrote:\n> I guess you could try changing the OOM score for git-fast-import.\n> \n> change /proc/<pid>/oomadj.\n> \n> I think a value of -31 would make it very unlikely to be killed.\n> \n> On 29/08/13 23:46, Pete Wyckoff wrote:\n> >I usually just do \"git p4 sync @505859\".  The error message below\n> >crops up when things get confused.  Usually after a previous\n> >error.  I tend to destroy the repo and try again.  Sorry I don't\n> >can't explain better what's happening here.  It's not a memory\n> >issue; it reports only 24 MB used.\n> >\n> >Bizarre.  There is no good explanation why memory usage would go\n> >up to 32 GB (?) within one top interval (3 sec ?).  My theory\n> >about one gigantic object is debunked:  you have only the 118 MB\n> >one.  Perhaps there's some container or process memory limit, as\n> >Luke guessed, but it's not obvious here.\n> >\n> >The other big hammer is \"strace\".  If you're still interested in\n> >playing with this, you could do:\n> >\n> >     strace -vf -tt -s 200 -o /tmp/strace.out git p4 clone ....\n> >\n> >and hours later, see if something suggests itself toward the\n> >end of that output file.\n> >\n> >\t\t-- Pete\n> \n\nFinally, I claim success!  Unfortunately I did not try either of the OOM\nscore or strace suggestions - sorry!  After spending so much time on\nthis, I've gotten to the point that I'm more interested in getting it to\nwork than in figuring out why the direct approach isn't working; it\nsounds like you're both pretty confident that git is working as it\nshould, and I don't maintain the system I'm doing this on so I don't\ndoubt that there might be some artificial limit or other quirk here that\nwe just aren't seeing.\n\nAnyway, what I found is that Pete's incremental method does work, I just\nhave to know how to do it properly!  This is what I WAS doing to\ngenerate the error message I pasted several posts ago:\n\ngit clone //path/to/branch@<begin>,<stage1>\ncd branch\ngit sync //path/to/branch@<stage2>\n# ERROR!\n# (I also tried //path/to/branch@<stage1+1>,<stage2>, same error)\n\nEventually what happened is that I downloaded the free 20-user p4d, set\nup a very small repository with only 4 changes, and started some old\nfashioned trial-and-error.  Here's what I should have been doing all\nalong:\n\ngit clone //path/to/branch@<begin>,<stage1>\ncd branch\ngit sync //path/to/branch@<begin>,<stage2>\ngit sync //path/to/branch@<begin>,<stage3>\n# and so on...\n\nAnd syncing a few thousand changes every day over the course of the past\nweek, my git repo is finally up to the Perforce HEAD.  So I suppose\nultimately this was my own misunderstanding, partly because when you\nbegin your range at the original first change number the output looks\nsuspiciously like it's importing changes again that it's already\nimported.  Maybe this is all documented somewhere, and if it is I just\nfailed to find it.\n\nThanks to both of you for all your help!\nCorey\n"},{"id":"227005","messageId":"20130907081905.GA4377@padd.com","threadId":"34746","inReplyTo":"20130906190345.GA3239@jerec","subject":"Re: git-p4 out of memory for very large repository","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2013-09-07T08:19:05Z","receivedAt":"2013-09-07T08:19:05Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"cmtptr@gmail.com wrote on Fri, 06 Sep 2013 15:03 -0400:\n> Finally, I claim success!  Unfortunately I did not try either of the OOM\n> score or strace suggestions - sorry!  After spending so much time on\n> this, I've gotten to the point that I'm more interested in getting it to\n> work than in figuring out why the direct approach isn't working; it\n> sounds like you're both pretty confident that git is working as it\n> should, and I don't maintain the system I'm doing this on so I don't\n> doubt that there might be some artificial limit or other quirk here that\n> we just aren't seeing.\n> \n> Anyway, what I found is that Pete's incremental method does work, I just\n> have to know how to do it properly!  This is what I WAS doing to\n> generate the error message I pasted several posts ago:\n> \n> git clone //path/to/branch@<begin>,<stage1>\n> cd branch\n> git sync //path/to/branch@<stage2>\n> # ERROR!\n> # (I also tried //path/to/branch@<stage1+1>,<stage2>, same error)\n> \n> Eventually what happened is that I downloaded the free 20-user p4d, set\n> up a very small repository with only 4 changes, and started some old\n> fashioned trial-and-error.  Here's what I should have been doing all\n> along:\n> \n> git clone //path/to/branch@<begin>,<stage1>\n> cd branch\n> git sync //path/to/branch@<begin>,<stage2>\n> git sync //path/to/branch@<begin>,<stage3>\n> # and so on...\n> \n> And syncing a few thousand changes every day over the course of the past\n> week, my git repo is finally up to the Perforce HEAD.  So I suppose\n> ultimately this was my own misunderstanding, partly because when you\n> begin your range at the original first change number the output looks\n> suspiciously like it's importing changes again that it's already\n> imported.  Maybe this is all documented somewhere, and if it is I just\n> failed to find it.\n> \n> Thanks to both of you for all your help!\n\nThat you got it to work is the most important thing.  Amazing all\nthe effort you put into it; a lesser hacker would have walked\naway much earlier.\n\nThe changes don't overlap.  If you give it a range that includes\nchanges already synced, git-p4 makes sure to start only at the\nlowest change it has not yet seen.  I'll see if I can update the\ndocs somewhere.\n\n\t\t-- Pete\n"}]}