Re: git-p4 out of memory for very large repository

From: Corey Thompson <cmtptr@gmail.com>
Date: Sep 6, 2013, 19:03 UTC
Message-ID: <20130906190345.GA3239@jerec>
In-Reply-To: <5224EA2C.7090001@diamand.org>

On Mon, Sep 02, 2013 at 08:42:36PM +0100, Luke Diamand wrote:

Show 29 quoted lines

> I guess you could try changing the OOM score for git-fast-import.
> 
> change /proc/<pid>/oomadj.
> 
> I think a value of -31 would make it very unlikely to be killed.
> 
> On 29/08/13 23:46, Pete Wyckoff wrote:
> >I usually just do "git p4 sync @505859".  The error message below
> >crops up when things get confused.  Usually after a previous
> >error.  I tend to destroy the repo and try again.  Sorry I don't
> >can't explain better what's happening here.  It's not a memory
> >issue; it reports only 24 MB used.
> >
> >Bizarre.  There is no good explanation why memory usage would go
> >up to 32 GB (?) within one top interval (3 sec ?).  My theory
> >about one gigantic object is debunked:  you have only the 118 MB
> >one.  Perhaps there's some container or process memory limit, as
> >Luke guessed, but it's not obvious here.
> >
> >The other big hammer is "strace".  If you're still interested in
> >playing with this, you could do:
> >
> >     strace -vf -tt -s 200 -o /tmp/strace.out git p4 clone ....
> >
> >and hours later, see if something suggests itself toward the
> >end of that output file.
> >
> >		-- Pete
>

Finally, I claim success! Unfortunately I did not try either of the OOM score or strace suggestions - sorry! After spending so much time on this, I've gotten to the point that I'm more interested in getting it to work than in figuring out why the direct approach isn't working; it sounds like you're both pretty confident that git is working as it should, and I don't maintain the system I'm doing this on so I don't doubt that there might be some artificial limit or other quirk here that we just aren't seeing.

Anyway, what I found is that Pete's incremental method does work, I just have to know how to do it properly! This is what I WAS doing to generate the error message I pasted several posts ago:

git clone //path/to/branch@<begin>,<stage1> cd branch git sync //path/to/branch@<stage2> # ERROR! # (I also tried //path/to/branch@<stage1+1>,<stage2>, same error)

Eventually what happened is that I downloaded the free 20-user p4d, set up a very small repository with only 4 changes, and started some old fashioned trial-and-error. Here's what I should have been doing all along:

git clone //path/to/branch@<begin>,<stage1> cd branch git sync //path/to/branch@<begin>,<stage2> git sync //path/to/branch@<begin>,<stage3> # and so on...

And syncing a few thousand changes every day over the course of the past week, my git repo is finally up to the Perforce HEAD. So I suppose ultimately this was my own misunderstanding, partly because when you begin your range at the original first change number the output looks suspiciously like it's importing changes again that it's already imported. Maybe this is all documented somewhere, and if it is I just failed to find it.

Thanks to both of you for all your help! Corey