git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: git-p4 out of memory for very large repository

From
PWPete Wyckoff <pw@padd.com>
Date
Sep 7, 2013, 08:19 UTC
Message-ID
<20130907081905.GA4377@padd.com>
In-Reply-To
<20130906190345.GA3239@jerec>
cmtptr@gmail.com wrote on Fri, 06 Sep 2013 15:03 -0400:
Show 39 quoted lines
> Finally, I claim success!  Unfortunately I did not try either of the OOM
> score or strace suggestions - sorry!  After spending so much time on
> this, I've gotten to the point that I'm more interested in getting it to
> work than in figuring out why the direct approach isn't working; it
> sounds like you're both pretty confident that git is working as it
> should, and I don't maintain the system I'm doing this on so I don't
> doubt that there might be some artificial limit or other quirk here that
> we just aren't seeing.
> 
> Anyway, what I found is that Pete's incremental method does work, I just
> have to know how to do it properly!  This is what I WAS doing to
> generate the error message I pasted several posts ago:
> 
> git clone //path/to/branch@<begin>,<stage1>
> cd branch
> git sync //path/to/branch@<stage2>
> # ERROR!
> # (I also tried //path/to/branch@<stage1+1>,<stage2>, same error)
> 
> Eventually what happened is that I downloaded the free 20-user p4d, set
> up a very small repository with only 4 changes, and started some old
> fashioned trial-and-error.  Here's what I should have been doing all
> along:
> 
> git clone //path/to/branch@<begin>,<stage1>
> cd branch
> git sync //path/to/branch@<begin>,<stage2>
> git sync //path/to/branch@<begin>,<stage3>
> # and so on...
> 
> And syncing a few thousand changes every day over the course of the past
> week, my git repo is finally up to the Perforce HEAD.  So I suppose
> ultimately this was my own misunderstanding, partly because when you
> begin your range at the original first change number the output looks
> suspiciously like it's importing changes again that it's already
> imported.  Maybe this is all documented somewhere, and if it is I just
> failed to find it.
> 
> Thanks to both of you for all your help!

That you got it to work is the most important thing. Amazing all the effort you put into it; a lesser hacker would have walked away much earlier.

The changes don't overlap. If you give it a range that includes changes already synced, git-p4 makes sure to start only at the lowest change it has not yet seen. I'll see if I can update the docs somewhere.

		-- Pete
Previous: Corey Thompson
Message 13 of 13 in “git-p4 out of memory for very large repository”
  1. Corey ThompsonAug 23, 2013
  2. Luke DiamandAug 23, 2013
  3. Corey ThompsonAug 23, 2013
  4. Corey ThompsonAug 23, 2013
  5. Luke DiamandAug 23, 2013
  6. Corey ThompsonAug 24, 2013
  7. Pete WyckoffAug 25, 2013
  8. Corey ThompsonAug 26, 2013
  9. Corey ThompsonAug 28, 2013
  10. Pete WyckoffAug 29, 2013
  11. Luke DiamandSep 2, 2013
  12. Corey ThompsonSep 6, 2013
  13. Pete WyckoffSep 7, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.