From: Luke Diamand <luke@diamand.org>
Date: Mon, 02 Sep 2013 19:42:36 GMT
Subject: Re: git-p4 out of memory for very large repository
Message-ID: <5224EA2C.7090001@diamand.org>
In-Reply-To: <20130829224609.GB25879@padd.com>

I guess you could try changing the OOM score for git-fast-import.

change /proc/<pid>/oomadj.

I think a value of -31 would make it very unlikely to be killed.

On 29/08/13 23:46, Pete Wyckoff wrote:
> cmtptr@gmail.com wrote on Wed, 28 Aug 2013 11:41 -0400:
>> On Mon, Aug 26, 2013 at 09:47:56AM -0400, Corey Thompson wrote:
>>> You are correct that git-fast-import is killed by the OOM killer, but I
>>> was unclear about which process was malloc()ing so much memory that the
>>> OOM killer got invoked (as other completely unrelated processes usually
>>> also get killed when this happens).
>>>
>>> Unless there's one gigantic file in one change that gets removed by
>>> another change, I don't think that's the problem; as I mentioned in
>>> another email, the machine has 32GB physical memory and the largest
>>> single file in the current head is only 118MB.  Even if there is a very
>>> large transient file somewhere in the history, I seriously doubt it's
>>> tens of gigabytes in size.
>>>
>>> I have tried watching it with top before, but it takes several hours
>>> before it dies.  I haven't been able to see any explosion of memory
>>> usage, even within the final hour, but I've never caught it just before
>>> it dies, either.  I suspect that whatever the issue is here, it happens
>>> very quickly.
>>>
>>> If I'm unable to get through this today using the incremental p4 sync
>>> method you described, I'll try running a full-blown clone overnight with
>>> top in batch mode writing to a log file to see whether it catches
>>> anything.
>>>
>>> Thanks again,
>>> Corey
>>
>> Unforunately I have not made much progress.  The incremental sync method
>> fails with the output pasted below.  The change I specified is only one
>> change number above where that repo was cloned...
>
> I usually just do "git p4 sync @505859".  The error message below
> crops up when things get confused.  Usually after a previous
> error.  I tend to destroy the repo and try again.  Sorry I don't
> can't explain better what's happening here.  It's not a memory
> issue; it reports only 24 MB used.
>
>> So I tried a 'git p4 rebase' overnight with top running, and as I feared
>> I did not see anything out of the ordinary.  git, git-fast-import, and
>> git-p4 all hovered under 1.5% MEM the entire time, right up until
>> death.  The last entry in my log shows git-fast-import at 0.8%, with git
>> and git-p4 at 0.0% and 0.1%, respectively.  I could try again with a
>> more granular period, but I feel like this method is ultimately a goose
>> chase.
>
> Bizarre.  There is no good explanation why memory usage would go
> up to 32 GB (?) within one top interval (3 sec ?).  My theory
> about one gigantic object is debunked:  you have only the 118 MB
> one.  Perhaps there's some container or process memory limit, as
> Luke guessed, but it's not obvious here.
>
> The other big hammer is "strace".  If you're still interested in
> playing with this, you could do:
>
>      strace -vf -tt -s 200 -o /tmp/strace.out git p4 clone ....
>
> and hours later, see if something suggests itself toward the
> end of that output file.
>
> 		-- Pete