Re: git-p4import.py robustness changes

From: Shawn O. Pearce <spearce@spearce.org>
Date: Jun 4, 2007, 05:54 UTC
Message-ID: <20070604055433.GD4507@spearce.org>
In-Reply-To: <839AEF71-ED29-4A79-BE97-C79EAFEDC466@slamb.org>

Scott Lamb <slamb@slamb.org> wrote:

Show 9 quoted lines

> On Jun 3, 2007, at 6:11 AM, Simon Hausmann wrote:
> >On the topic of git integration with perforce, what are the chances  
> >of getting
> >git-p4 ( http://repo.or.cz/w/fast-export.git ) into git's contrib/ 
> >fast-export
> >area? :)
> 
> I missed that one...I just saw Tailor and the Perl script someone  
> else had written.

Perhaps why it should be in contrib/fast-import?  ;-)

Show 13 quoted lines

> As for performance...hmm. Looks like git-p4import.py runs these  
> commands for each Perforce revision:
> 
>     realtime  operation
>         3.4%  p4 describe -s N
>        66.6%  p4 sync ...@N
>    [*] 10.2%  git ls-files -m -d -o -z | git update-index --add -- 
> remove -z --stdin
>         2.6%  git rev-parse --verify HEAD
>         4.2%  git write-tree
>         2.8%  git commit-tree xxxxxx
>         7.5%  git tag -f p4/N xxxxxx
>         2.7%  git update-ref HEAD xxxxxx

...

> git-p4 seems to use "git fast-import". I guess the big performance  
> improvement there is removing the ls-files operation? So we're  
> talking about a 0-10% speedup, right? Plus some fork()/exec() overhead.

fast-import folds all of the git commands you list above behind a single engine that is *fast*. So its actually a 0-30% gain that is available by using the fast-import backend, with a single fork()/exec() for the *entire import*. The local object IO performed by Git is also minimized, so large imports have much better IO behavior from the Git perspective. Its not something to sneeze at.

fast-import also can run in parallel with the frontend process, allowing you to use a dual-core system, to the extent that your disk(s) and network can keep up. Generally p4 is going to be the bottleneck.

I think writing data to fast-import is much easier than running the raw Git commands, especially when you are talking about an import engine where you need to set all of the special environment variables for git-commit-tree or git-tag to do its job properly. Its a good tool that simply doesn't get enough use, partly because nobody is using it...

-- 
Shawn.