Re: From P4 to Git

From: Sam Vilain <sam@vilain.net>
Date: Aug 3, 2009, 20:32 UTC
Message-ID: <1249331540.12801.10.camel@maia.lan>
In-Reply-To: <81b0412b0908030650oc39f4a3s7c059e300b65addb@mail.gmail.com>

On Mon, 2009-08-03 at 15:50 +0200, Alex Riesen wrote:

Show 13 quoted lines

> On Mon, Aug 3, 2009 at 13:30, Sam Vilain<sam@vilain.net> wrote:
> > On Mon, 2009-08-03 at 10:47 +0200, Alex Riesen wrote:
> >> Is it an import-once tool, or can the process be restarted? (because it looks
> >> like the script needs a complicated setup).
> >
> > It's fully restartable.  Not only that but it uses transaction
> > protection to make sure that its internal state doesn't get corrupted
> > when performing the various options.
> 
> "varios options"? Operations? As when working on a live server?
> Aren't P4 changelist numbers always increasing? Or you mean
> the protection against multiple running instances of p4raw,
> so it is also parallelizable?

The "live" parts are never touched - only the write-only files that perforce writes; and the rcs files are read using rcs.

What I was referring to by "various options" was the commands in the program which perform the migration. It's a multi-step process - load the metadata from perforce 'load', import the file images to git blobs 'export-blobs', trace through the path structure and decide which parts are the roots, and make parents 'find-branches', etc. All parts are fully transaction protected, restartable and rewindable. It also means that if a command crashes that you are left in a sane state. I had to build it like that to keep my sanity writing and using it :-). After all it was something of a reverse engineering of Perforce's internals.

All you need is the RCS backing files, a 'checkpoint' and (optionally) 'journal' files. No access to the live perforce DBs is required.

> > No, it's server only. ...
> Darn. I hoped it wasn't. Can't play with it, then.

Yeah, in theory you could derive the internal table information by issuing zillions of 'p4 integrated', 'p4 filelog' commands etc. I've written p4raw sub-commands which can produce the output of those commands but not tried to go the other way; I wasn't interested.

Show 6 quoted lines

> > ... I think I did get around to implementing not having
> > to go through all the stages for branches you didn't care to import.  It's
> > difficult though, the stage which correlates those thousands of 'integrate'
> > records is never going to be fast.
> Maybe if it is done locally, it can be improved? You seem to use the
> Postgre for lookups, right?

I'm sure that the queries could be improved, but Postgres is actually quite good at crunching the numbers. It's just a lot of data to crunch.

> That's were I hoped your project could help. I thought, if I pull in all the
> needed changelists (selected by path/CL), there may be a way to
> recreate a mergeable history out of this dump. At least, one involving less
> labor then I have to do now.

Yeah... well the design of my tool is that it needs to have the perforce internal information. But really you can probably consider the converter orphaned, I have no current need to work on it; it served a purpose, which was converting perl's perforce to perl.git, and that's history now.

Sam