Re: git-p4 out of memory for very large repository

From: Corey Thompson <cmtptr@gmail.com>
Date: Aug 28, 2013, 15:41 UTC
Message-ID: <20130828154135.GA16921@jerec>
In-Reply-To: <20130826134756.GA1335@jerec>

On Mon, Aug 26, 2013 at 09:47:56AM -0400, Corey Thompson wrote:

Show 25 quoted lines

> You are correct that git-fast-import is killed by the OOM killer, but I
> was unclear about which process was malloc()ing so much memory that the
> OOM killer got invoked (as other completely unrelated processes usually
> also get killed when this happens).
> 
> Unless there's one gigantic file in one change that gets removed by
> another change, I don't think that's the problem; as I mentioned in
> another email, the machine has 32GB physical memory and the largest
> single file in the current head is only 118MB.  Even if there is a very
> large transient file somewhere in the history, I seriously doubt it's
> tens of gigabytes in size.
> 
> I have tried watching it with top before, but it takes several hours
> before it dies.  I haven't been able to see any explosion of memory
> usage, even within the final hour, but I've never caught it just before
> it dies, either.  I suspect that whatever the issue is here, it happens
> very quickly.
> 
> If I'm unable to get through this today using the incremental p4 sync
> method you described, I'll try running a full-blown clone overnight with
> top in batch mode writing to a log file to see whether it catches
> anything.
> 
> Thanks again,
> Corey

Unforunately I have not made much progress. The incremental sync method fails with the output pasted below. The change I specified is only one change number above where that repo was cloned...

So I tried a 'git p4 rebase' overnight with top running, and as I feared I did not see anything out of the ordinary. git, git-fast-import, and git-p4 all hovered under 1.5% MEM the entire time, right up until death. The last entry in my log shows git-fast-import at 0.8%, with git and git-p4 at 0.0% and 0.1%, respectively. I could try again with a more granular period, but I feel like this method is ultimately a goose chase.

Corey

$ git p4 sync //path/to/some/branch@505859
Doing initial import of //path/to/some/branch/ from revision @505859 into refs/remotes/p4/master
fast-import failed: warning: Not updating refs/remotes/p4/master (new tip 29ef6ff25f1448fa2f907d22fd704594dc8769bd does not contain d477672be5ac6a00cc9175ba2713d5395660e840)
git-fast-import statistics:
---------------------------------------------------------------------
Alloc'd objects:     165000
Total objects:           69 (    232434 duplicates                  )
      blobs  :           45 (    209904 duplicates         40 deltas of         42 attempts) 
      trees  :           23 (     22530 duplicates          0 deltas of         23 attempts) 
      commits:            1 (         0 duplicates          0 deltas of          0 attempts) 
      tags   :            0 (         0 duplicates          0 deltas of          0 attempts)
Total branches:           1 (         1 loads     )
      marks:           1024 (         0 unique    )
      atoms:         105170
Memory total:         24421 KiB
       pools:         17976 KiB
     objects:          6445 KiB
---------------------------------------------------------------------
pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize =   33554432
pack_report: core.packedGitLimit      =  268435456
pack_report: pack_used_ctr            =       4371
pack_report: pack_mmap_calls          =        124
pack_report: pack_open_windows        =          8 /          9
pack_report: pack_mapped              =  268435456 /  268435456
---------------------------------------------------------------------