Re: git-p4 out of memory for very large repository
- From
- Pete Wyckoff <pw@padd.com>
- Date
- Sep 7, 2013, 08:19 UTC
- Message-ID
- <20130907081905.GA4377@padd.com>
- In-Reply-To
- <20130906190345.GA3239@jerec>
cmtptr@gmail.com wrote on Fri, 06 Sep 2013 15:03 -0400:
Show 39 quoted lines
> Finally, I claim success! Unfortunately I did not try either of the OOM > score or strace suggestions - sorry! After spending so much time on > this, I've gotten to the point that I'm more interested in getting it to > work than in figuring out why the direct approach isn't working; it > sounds like you're both pretty confident that git is working as it > should, and I don't maintain the system I'm doing this on so I don't > doubt that there might be some artificial limit or other quirk here that > we just aren't seeing. > > Anyway, what I found is that Pete's incremental method does work, I just > have to know how to do it properly! This is what I WAS doing to > generate the error message I pasted several posts ago: > > git clone //path/to/branch@<begin>,<stage1> > cd branch > git sync //path/to/branch@<stage2> > # ERROR! > # (I also tried //path/to/branch@<stage1+1>,<stage2>, same error) > > Eventually what happened is that I downloaded the free 20-user p4d, set > up a very small repository with only 4 changes, and started some old > fashioned trial-and-error. Here's what I should have been doing all > along: > > git clone //path/to/branch@<begin>,<stage1> > cd branch > git sync //path/to/branch@<begin>,<stage2> > git sync //path/to/branch@<begin>,<stage3> > # and so on... > > And syncing a few thousand changes every day over the course of the past > week, my git repo is finally up to the Perforce HEAD. So I suppose > ultimately this was my own misunderstanding, partly because when you > begin your range at the original first change number the output looks > suspiciously like it's importing changes again that it's already > imported. Maybe this is all documented somewhere, and if it is I just > failed to find it. > > Thanks to both of you for all your help!
That you got it to work is the most important thing. Amazing all the effort you put into it; a lesser hacker would have walked away much earlier.
The changes don't overlap. If you give it a range that includes changes already synced, git-p4 makes sure to start only at the lowest change it has not yet seen. I'll see if I can update the docs somewhere.
-- Pete