Re: [PATCH/RFC v2] git-p4: stream from perforce to speed up clones

From: Pete Wyckoff <pw@padd.com>
Date: Jul 25, 2009, 14:23 UTC
Message-ID: <20090725142329.GA3841@arf.padd.com>
In-Reply-To: <1247301071-9541-1-git-send-email-luke@diamand.org>

luke@diamand.org wrote on Sat, 11 Jul 2009 09:31 +0100:

Show 14 quoted lines

> Change commit() to stream data from Perforce and into fast-import
> rather than reading into memory first, and then writing out. This
> hugely reduces the memory requirements when cloning non-incrementally.
> 
> Signed-off-by: Luke Diamand <luke@diamand.org>
> ---
> I've modified git-p4 so that it streams/pipes data into fast-import rather
> that reading everything into memory first. The old scheme meant that
> for a large repository (mine is around 2G) my PC just grinds to a
> halt and never actually finishes. With this change it takes around ten
> minutes.
> 
> This is a resend of a patch I sent earlier, which my MUA helpfully
> managed to word-wrap.

This is definitely good stuff. While I'd like to get Daniel's native transport code working in my environment, I had a need to import some big trees quickly from p4 to git. Usually I resort to a rare big-memory machine to do the git-p4 import, but with your change, I can use my laptop without going into swap or out-of-memory.

For the content, I would recommend some changes. Here are some patches on top of yours:

    [PATCH 1/5] git-p4 stream: remove unused function
    [PATCH 2/5] git-p4 stream: do not pass branchPrefixes so much
    [PATCH 3/5] git-p4 stream: show relative path in debug messages
    [PATCH 4/5] git-p4 stream: check apple file type
    [PATCH 5/5] git-p4 stream: use existing p4CmdList with callback

They clean up the code and change your main iterator loop to reuse the existing p4CmdList.

I'd suggest you merge these into your patch and send it back out. It's a worthwhile change and we should get Simon to review it.

		-- Pete