From P4 to Git

Hello,
I am investigating the use of Git in a setting where we:
 1. branches a lot
 2. have a very large codebase

Given Git is developed to handle these 2 issues, I suspect it would be a very good choice, but I need to gather some experiments feedback and hard figures on how Git performs (storage use, necessary bandwidth/resources, maintainance of repositories, cleanup & gc...).

For the experiment part, I am started working on it but would be interested in other people's experiences.

For the figures part, I think I read somewhere there exists some statistics on Git usage for Linux kernel. Is this correct ? If true, where could I find them ?

Thanks in advance for answering my (maybe pointless) questions and for producing such a nice piece of software.

On Tue, 28 Jul 2009, Arnaud Bailly wrote:

the one thing that I have seen P4 reported as handling better than git is the case where the repository is very large. this is typically not caused by lots of lines of code, but by checking in lots of binary blobs to the repository.

David Lang

Arnaud Bailly <abailly@oqube.com> writes:

Git handles large number of branches very well.  

The workflow of topic branches, where each new feature is developed on separate branch, which can be individually picked to be merged (or not) is used by git itself, for example.

How large codebase? What it means "large codebase"? Large number of files, or files large in size (usually binary)?

Git can deal comfortably with codebase of the size of Linux kernel. Perl 5 core was converted from Perforce to Git.

But git is snapshot based, not changeset based, and treats project (repository) as whole, not as a combination of single file histories. This means that it would be unproductive to use 'everything in single repository' approach. If your codebase is of the size of whole KDE tree, or the whole GNOME tree, you would need to organize it and split it into smaller, reasonably sized repositories (you can urganize them back together in a superproject using submodules).

If ou can't do that, you would probably be better with other version control system, like Subversion (IIRC both KDE and OpenOffice.org chosen this free centralized version control system).

Because Git was created to version control source code, it might not work well with large binary files, meaning that performance would suffer.

Partial checkouts (where you check out only part of repository) were proposed, but are not implemented yet. Neither did lazy clone / remote alternates idea. You can do a bit with undocumented `delta` gitattribute, and with putting large binary blobs into separate packfile, which is 'kept' (using *.keep file) against repacking, and perhaps available on networked filesystem.

I think you can use refs/replace/ mechanizm (IIRC currently in 'pu', (proposed updates) branch) to have two versions of repository: one with binary blobs and one without.

Check out Sam Vilain (?) reports from converting Perl 5 repository from Perforce to Git.

There is GitStat project: http://mirror.celinuxforum.org/gitstat/

There was also Git-Statistics project at Google Summer of Code 2008 which repository can be found at http://repo.or.cz/w/git-stats.git See http://git.or.cz/gitwiki/SoC2008Projects

2009/7/29 Jakub Narebski <jnareb@gmail.com>:

Going a bit off topic - but to clarify, KDE hasn't really 'chosen' svn over git, it's that we haven't switched to git _yet_. It just takes time. We already have trial projects (Amarok, for example, has switched to git to lead the way) and I'm not sure if git submodule support is mature enough, but baring any major problems we will switch.

John

I have re-added git mailing list to Cc.

On Fri, 31 July 2009, Arnaud Bailly wrote:

Linux kernel, for development of which git was initially designed, is around 7.5M LoC of code (10M LoC with comments and blank lines)[1]. The performance for such large codebase has to be good (at least on Linux and other POSIX systems), as good performance was one of goal decisions of git.

Perl 5 core, which version control history was converted from Perforce to Git in December 2008[1][2] by Sam Vilain (you might want to take a look how it was done; unfortunately it seems that Sam Vilain blog vanished from Internet), is 2.3M LoC of mixed language code (mainly Perl and C, with twice as much Perl), so it is smaller than yours codebase.

[1] According to OSS software metric site Ohloh (http://www.ohloh.net) [2] http://news.perlfoundation.org/2008/12/perl_5_development_now_on_git.html [3] http://use.perl.org/article.pl?sid=08/12/22/0830205

So it looks like you wouldn't _need_ to split source tree into separate smaller repositories for performance reasons. Still it might be good idea to put separate (sub)projects into separate repositories. But I guess you can do that even at later time (although it would be best to do this upfront).

Branching and merging in Git is very easy (with Subversion 1.5 merging
is supposedly to get easier).  Git itself uses 'topic branches' workflow,
where each feature (each series of patches) gets its own branch, and
branches are cherry-picked to be merged (or to be dropped, or replaced
by newer version of series).
 

Which follows Linux kernel. There are also some GitStat deployments tracking other code.

And there are tools such as Ohloh, which do software metric, and some of code is available. GitHub hosting site also offers some software metric / statistic tools in its web interface; I don't know about other sites such as Gitorious or InDefero. Gitweb currently doesn't offer any statistics.

On Fri, Jul 31, 2009 at 11:22, Jakub Narebski<jnareb@gmail.com> wrote:

It looks like they use P4 branches. Which are NOT branches as Git understands it (a line of history). The P4 "branches" are just directories with some metadata regarding recording where the data in the directory were before an act of "branching" (just a server-side recursive copy).

In this case (and this must be, as there are no other branches in P4), they'll have to split their repository.

Alex Riesen wrote:

p4's branches are 'close enough'. My tool travels through the history of the repository forwards, detects paths where new content was placed and calls those roots. When branching is detected (recorded file-by-file in perforce), it puts the branch source as a new parent. When branches are 'integrated', it makes a fuzzy decision based on the number of outstanding merges it thinks are due, also based on a merge base calculation on the current view of the derived history. It allows manual correction of this fuzzy information, and arbitrary grafting along the way. Discrepancies between the fuzzy decision and the actual integration records are recorded in the commit log along with other metadata including the commit ID.

Sam

On Mon, Aug 3, 2009 at 09:49, Sam Vilain<sam@vilain.net> wrote:

Sounds dangerous. And interesting...

Is it an import-once tool, or can the process be restarted? (because it looks like the script needs a complicated setup). Can it be used from a client machine? And more importantly: can the branches be found from incomplete history, restricted by path and changelist range? (because, in a corporate setup, clients seldom have full access to all data).

On Mon, 2009-08-03 at 10:47 +0200, Alex Riesen wrote:

It's fully restartable. Not only that but it uses transaction protection to make sure that its internal state doesn't get corrupted when performing the various options.

No, it's server only. I think I did get around to implementing not having to go through all the stages for branches you didn't care to import. It's difficult though, the stage which correlates those thousands of 'integrate' records is never going to be fast. Be prepared to tune your postgres - add lots of shared_buffers and sort memory if your project is as large as Perl.

Sam.

On Mon, Aug 3, 2009 at 13:30, Sam Vilain<sam@vilain.net> wrote:

"varios options"? Operations? As when working on a live server? Aren't P4 changelist numbers always increasing? Or you mean the protection against multiple running instances of p4raw, so it is also parallelizable?

Darn. I hoped it wasn't. Can't play with it, then.

Maybe if it is done locally, it can be improved? You seem to use the Postgre for lookups, right?

Mine isn't, but it is thrown on a server with a lot of others. Some branched off mine, some integrate it, some just copied and re-introduced into repository (these will probably worked with manually forever). There is also a small problem of different P4 servers, hosting code from the same project (they pull different repos together on client, imagine that!)

And all of them go back for 5-6 years, so it is kind of largish (not to mention a mess of binaries).

Some branches I hope to merge back. I do that already, but it is a lot of manual work (I use that git-p4-import.bat I posted early, which only can import a sequence of changelists with files they touch).

That's were I hoped your project could help. I thought, if I pull in all the needed changelists (selected by path/CL), there may be a way to recreate a mergeable history out of this dump. At least, one involving less labor then I have to do now.

On Mon, 2009-08-03 at 15:50 +0200, Alex Riesen wrote:

The "live" parts are never touched - only the write-only files that perforce writes; and the rcs files are read using rcs.

What I was referring to by "various options" was the commands in the program which perform the migration. It's a multi-step process - load the metadata from perforce 'load', import the file images to git blobs 'export-blobs', trace through the path structure and decide which parts are the roots, and make parents 'find-branches', etc. All parts are fully transaction protected, restartable and rewindable. It also means that if a command crashes that you are left in a sane state. I had to build it like that to keep my sanity writing and using it :-). After all it was something of a reverse engineering of Perforce's internals.

All you need is the RCS backing files, a 'checkpoint' and (optionally) 'journal' files. No access to the live perforce DBs is required.

Yeah, in theory you could derive the internal table information by issuing zillions of 'p4 integrated', 'p4 filelog' commands etc. I've written p4raw sub-commands which can produce the output of those commands but not tried to go the other way; I wasn't interested.

I'm sure that the queries could be improved, but Postgres is actually quite good at crunching the numbers. It's just a lot of data to crunch.

Yeah... well the design of my tool is that it needs to have the perforce internal information. But really you can probably consider the converter orphaned, I have no current need to work on it; it served a purpose, which was converting perl's perforce to perl.git, and that's history now.

Sam

Sam Vilain, Mon, Aug 03, 2009 22:32:20 +0200:

I always felt Perforce had a touch of pure evil...

Which in my case is all the same. I don't dare to ask, frankly.

Thanks anyway for bringing it to public attention. At least, it is known to be done once. Could be an encouragement for next one to try. A guide, even.

Alex Riesen wrote:

Perforce is not really that evil imho. It uses rcs for what it's good at
- storing file revisions. All the metadata it kept in table stores which
could be checkpointed and journalled like a regular database. Simple,
elegant, fast, robust. Overall I'd say it was a well engineered piece of
software which delivered great benefits to its users, far less evil than
CVS or Subversion. Which is funny because Subversion rips off many
concepts from Perforce including the 'branching is just copying' idea.
But they left behind the integration system. Even now svn's merging
support can't even handle merging correctly in the face of branched
branches, or so I hear. That's evil - pretending to have a feature
solved when they haven't even met the standard set by bzr, hg, mtn, git,
etc.

Sam

On Fri, 2009-07-31 at 11:22 +0200, Jakub Narebski wrote:

You think that was unfortunate? Wow maybe I should pull finger and get something back up again :-). Didn't try since the drupal blog I had finally bitrot, silly php.

So if you're looking for the actual importer ... google git-p4raw, should be on http://utsl.gen.nz/gitweb/ or http://git.catalyst.net.nz/gw

Sam

Thanks all for your detailed answers.

Playing with git on various environments, I have a couple more questions
that I could not (quickly) find answers on the web.
 1. when checkout-ing on NTFS, I got all my files marked as changed
 because of file permissions set from 100644 to 100755. I see there is a
 config flag about file permissions (core.sharedRepository ?) but not
 sure how to use it
 2. I am working with NTFS on a truecrypt managed USB HD and I noticed
 an order of magnitude difference in speed (ie. about 10x for checkout
 of 50k files) between this and my ext3 FS  on laptop HD. Is this
 expected ? Some people here reported slowness on non-encrypted NTFS and
 I think I read a couple of web fragments about speed difference between
 the 2. WDYT ?

Regards,

On Tue, Aug 04, 2009 at 02:31:42PM +0200, Arnaud Bailly wrote:

No, that config flag is for sharing a repository with other people. You want something like core.filemode = false

-Peter