From: Pete Wyckoff <pw@padd.com>
Date: Sat, 07 Sep 2013 08:19:05 GMT
Subject: Re: git-p4 out of memory for very large repository
Message-ID: <20130907081905.GA4377@padd.com>
In-Reply-To: <20130906190345.GA3239@jerec>

cmtptr@gmail.com wrote on Fri, 06 Sep 2013 15:03 -0400:
> Finally, I claim success!  Unfortunately I did not try either of the OOM
> score or strace suggestions - sorry!  After spending so much time on
> this, I've gotten to the point that I'm more interested in getting it to
> work than in figuring out why the direct approach isn't working; it
> sounds like you're both pretty confident that git is working as it
> should, and I don't maintain the system I'm doing this on so I don't
> doubt that there might be some artificial limit or other quirk here that
> we just aren't seeing.
> 
> Anyway, what I found is that Pete's incremental method does work, I just
> have to know how to do it properly!  This is what I WAS doing to
> generate the error message I pasted several posts ago:
> 
> git clone //path/to/branch@<begin>,<stage1>
> cd branch
> git sync //path/to/branch@<stage2>
> # ERROR!
> # (I also tried //path/to/branch@<stage1+1>,<stage2>, same error)
> 
> Eventually what happened is that I downloaded the free 20-user p4d, set
> up a very small repository with only 4 changes, and started some old
> fashioned trial-and-error.  Here's what I should have been doing all
> along:
> 
> git clone //path/to/branch@<begin>,<stage1>
> cd branch
> git sync //path/to/branch@<begin>,<stage2>
> git sync //path/to/branch@<begin>,<stage3>
> # and so on...
> 
> And syncing a few thousand changes every day over the course of the past
> week, my git repo is finally up to the Perforce HEAD.  So I suppose
> ultimately this was my own misunderstanding, partly because when you
> begin your range at the original first change number the output looks
> suspiciously like it's importing changes again that it's already
> imported.  Maybe this is all documented somewhere, and if it is I just
> failed to find it.
> 
> Thanks to both of you for all your help!

That you got it to work is the most important thing.  Amazing all
the effort you put into it; a lesser hacker would have walked
away much earlier.

The changes don't overlap.  If you give it a range that includes
changes already synced, git-p4 makes sure to start only at the
lowest change it has not yet seen.  I'll see if I can update the
docs somewhere.

		-- Pete