threads / discuss / 27411

[git-p4] Trouble importing all perforce branches into git repository

Subject: [git-p4] Trouble importing all perforce branches into git repository

## tl;dr

11 messages between May 19, 2011 and Jun 16, 2011.

replies: 10people: 5as markdown or json

Grant Limberg· May 19, 2011, 19:31 UTC · lore
I'm trying to import a project from my company's Perforce depot into a git repository. I've been successful getting a single branch into a single git repo, but what I'd like to do is to pull in the entire project plus all branches, but I can't get it working correctly.
I have my .p4settings file:
> P4PORT=perforce-server.local:1666
> P4USER=my.username
> P4CLIENT=my.clientspec.name
Clientspec is fairly simple:
> //project/... //my.clientspec.name/project/...
> -//project/External/... //my.clientspec.name/project/External/...
The second line is to exclude the external library binaries from the checkout.
I use the following command line to start git-p4 importing:
> git p4 clone --verbose --detect-branches --max-changes=100 //project/...@all
> 
This goes through and checks all the branch names, etc, and attempts to import starting at the first commit (in this case its changelists 771 for this project).
Show 18 quoted lines
> Importing from //project/...@all into project
> Initialized empty Git repository in /Users/grant/Development/git_test/project/.git/
> Reading pipe: git config git-p4.useclientspec
> Reading pipe: git config git-p4.user
> Reading pipe: git config git-p4.password
> Reading pipe: git config git-p4.port
> Reading pipe: git config git-p4.host
> Reading pipe: git config git-p4.client
> p4 -G branches
> Opening pipe: p4 -G branches
> p4 -G branch -o branch-1
> Opening pipe: p4 -G branch -o branch-1
> Reading pipe: git config --bool core.ignorecase
> p4 -G branch -o branch-2
> ...
> Opening pipe: p4 -G branch -o branch-n
> p4-git branches: []
> initial parents: {}
I'm not sure if p4-git branches, and initial parents are supposed to be empty here, but they are.
Finally we get to the point where it's about to start importing changelists into git and the following happens:
Show 21 quoted lines
> Getting p4 changes for //project/...
> p4 changes //project/...
> Reading pipe: p4 changes //project/...
> p4 -G describe 771
> Opening pipe: p4 -G describe 771
> Importing revision 771 (1%)branch is MAIN
> 
>     Importing new branch RCMerge/MAIN
> p4 changes //RCMerge/MAIN/...@1,770
> Reading pipe: p4 changes //RCMerge/MAIN/...@1,770
> 
>     Resuming with change 771
> parent determined through known branches: Branch-foo
> looking for initial parent for refs/remotes/p4/project/MAIN; current parent is refs/remotes/p4/project/Branch-foo
> commit into refs/remotes/p4/project/MAIN
> parent refs/remotes/p4/project/Branch-foo
> p4 -G -x - print
> Opening pipe: p4 -G -x - print
> Glue/source/.empty
> fatal: Invalid ref name or SHA1 expression: refs/remotes/p4/project/Branch-foo
> fast-import: dumping crash report to .git/fast_import_crash_26002
Here's the file referenced above:
Show 38 quoted lines
> fast-import crash report:
>     fast-import process: 26002
>     parent process     : 26000
>     at Thu May 19 11:51:54 2011
> 
> fatal: Invalid ref name or SHA1 expression: refs/remotes/p4/project/Branch-foo
> 
> Most Recent Commands Before Crash
> ---------------------------------
>   checkpoint
>   commit refs/remotes/p4/project/MAIN
>   committer Some User <some.user@domain.com> 1253574589 -0800
>   data <<EOT
> * from refs/remotes/p4/project/Branch-foo
> 
> Active Branch LRU
> -----------------
>     active_branches = 0 cur, 5 max
> 
>   pos  clock name
>   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> Inactive Branches
> -----------------
> refs/remotes/p4/project/MAIN:
>   status      : dirty
>   tip commit  : 0000000000000000000000000000000000000000
>   old tree    : 0000000000000000000000000000000000000000
>   cur tree    : 0000000000000000000000000000000000000000
>   commit clock: 0
>   last pack   : 
> 
> 
> Marks
> -----
> 
> -------------------
> END OF CRASH REPORT
Now, being a perforce novice, I have no idea what this means and have no idea how to resolve this issue, or if its even possible. Has anyone run into a similar issue before? If so, how did you resolve it?

Grant Limberg glimberg@gmail.com

Pete Wyckoff· May 22, 2011, 11:49 UTC · re: Grant Limberg · lore

Re: [git-p4] Trouble importing all perforce branches into git repository

glimberg@gmail.com wrote on Thu, 19 May 2011 19:32 +0000:
> [git-p4 automatic branch detection]
> > git p4 clone --verbose --detect-branches --max-changes=100 //project/...@all

I worry that --max-changes prevents it from pulling in the ref for Branch-foo, and this confuses the automatic branch detection that assumes it has all its parents.

It would certainly be more reliable if you did "git-p4 sync --branch foo //project/foo@all" for each branch you care about.

Vitor has some modifications to auto-branch detection that might be useful, or maybe some thoughts as to what's going on.

		-- Pete

(Vitor: full mail is here: http://article.gmane.org/gmane.comp.version-control.git/173996/ )

Grant Limberg· May 23, 2011, 03:27 UTC · re: Pete Wyckoff · lore

Re: [git-p4] Trouble importing all perforce branches into git repository

On May 22, 2011, at 4:49 AM, Pete Wyckoff wrote:
Show 19 quoted lines
> glimberg@gmail.com wrote on Thu, 19 May 2011 19:32 +0000:
>> [git-p4 automatic branch detection]
>>> git p4 clone --verbose --detect-branches --max-changes=100 //project/...@all
> 
> I worry that --max-changes prevents it from pulling in the
> ref for Branch-foo, and this confuses the automatic branch
> detection that assumes it has all its parents.
> 
> It would certainly be more reliable if you did "git-p4 sync
> --branch foo //project/foo@all" for each branch you care about.
> 
> Vitor has some modifications to auto-branch detection that might
> be useful, or maybe some thoughts as to what's going on.
> 
> 		-- Pete
> 
> (Vitor: full mail is here:
> http://article.gmane.org/gmane.comp.version-control.git/173996/ )
> 
Removing --max-changes=100 has no effect on the crash I'm experiencing.   Same crash still happens on the very first imported commit.
If i do "git-p4 sync --branch foo //project/foo@all" for each branch, does that convert 'p4 integrate' commands into git merges?  I"m trying to keep all history intact when importing from p4 into git.
Thanks,
-Grant
Vitor Antunes· May 23, 2011, 10:54 UTC · re: Grant Limberg · lore

Re: [git-p4] Trouble importing all perforce branches into git repository

Hi Grant,

I think I saw that happening when there is nothing to import or in situations where the parent of a certain branch does not exist.

Perforce does not strictly require branch definitions to allow a branch structure/flow. It is quite possible that you are integrating stuff around without using "branch specs". When git-p4 is importing your P4 database it will look at all branches available in the server, including ones from other depots. So it is possible that while you are seeing a big list of branches being processed, none apply to the branch structure you are trying to import.

Could you please confirm that you have Perforce branch specs for all branches that you are trying to import?

Thanks, Vitor

Grant Limberg· May 23, 2011, 16:18 UTC · re: Vitor Antunes · lore

Re: [git-p4] Trouble importing all perforce branches into git repository

On May 23, 2011, at 3:54 AM, Vitor Antunes wrote:
Show 18 quoted lines
> Hi Grant,
> 
> I think I saw that happening when there is nothing to import or in
> situations where the parent of a certain branch does not exist.
> 
> Perforce does not strictly require branch definitions to allow a branch
> structure/flow. It is quite possible that you are integrating stuff
> around without using "branch specs". When git-p4 is importing your P4
> database it will look at all branches available in the server, including
> ones from other depots. So it is possible that while you are seeing a
> big list of branches being processed, none apply to the branch structure
> you are trying to import.
> 
> Could you please confirm that you have Perforce branch specs for all
> branches that you are trying to import?
> 
> Thanks,
> Vitor
Vitor, 
The "Branch-foo" that git-p4 is crashing on does have a branchspec in Perforce.  I cannot, however guarantee that branchspecs were correctly created and used for all integrates in the repository.  It was created long before I started working here.  Is there a way to limit --detect-branches to only certain branches that I can ensure do have branchspecs?
- Grant
Vitor Antunes· May 23, 2011, 17:38 UTC · re: Grant Limberg · lore

Re: [git-p4] Trouble importing all perforce branches into git repository

Hi Grant,
> The "Branch-foo" that git-p4 is crashing on does have a branchspec in
> Perforce.  I cannot, however guarantee that branchspecs were
> correctly created and used for all integrates in the repository.  It
> was created long before I started working here.

You can create the branch specs now. There is no requirement for them to have been created at the time of the integration.

I've sent a patch to this mailing list [1] that adds the possibility of creating these branches in a gitconfig file. If you feel brave enough, please give it a try. The patch is missing a test case, which I still did not have time to review and send

> Is there a way to limit --detect-branches to only certain branches
> that I can ensure do have branchspecs?

If you have an incremental structure you could limit the cloning to a subset of the directories. I am not aware of any other way. Maybe someone else in the mailing list can shed some light...? :)

Vitor
[1] http://thread.gmane.org/gmane.comp.version-control.git/167998/focus=168000
Grant Limberg· May 23, 2011, 20:06 UTC · re: Vitor Antunes · lore

Re: [git-p4] Trouble importing all perforce branches into git repository

On May 23, 2011, at 10:38 AM, Vitor Antunes wrote:
Show 25 quoted lines
> Hi Grant,
> 
>> The "Branch-foo" that git-p4 is crashing on does have a branchspec in
>> Perforce.  I cannot, however guarantee that branchspecs were
>> correctly created and used for all integrates in the repository.  It
>> was created long before I started working here.
> 
> You can create the branch specs now. There is no requirement for them to
> have been created at the time of the integration.
> 
> I've sent a patch to this mailing list [1] that adds the possibility of
> creating these branches in a gitconfig file. If you feel brave enough,
> please give it a try. The patch is missing a test case, which I still
> did not have time to review and send
> 
>> Is there a way to limit --detect-branches to only certain branches
>> that I can ensure do have branchspecs?
> 
> If you have an incremental structure you could limit the cloning to a
> subset of the directories. I am not aware of any other way. Maybe
> someone else in the mailing list can shed some light...? :)
> 
> Vitor
> 
> [1] http://thread.gmane.org/gmane.comp.version-control.git/167998/focus=168000
I think I just figured out the problem.  The branch mapping between //project/MAIN/... and //project/Branch-foo/... was done backwards, thus git-p4 was expecting Branch-foo to be a parent of MAIN when the inverse was actually true.  MAIN's first changelist number was 771.  Branch-foo's first changelist number was 7652.  That's why git-p4 was getting confused.  I fixed the branch mapping in Perforce and the clone with --detect-branches seems to be working fine now. 

Grant Limberg glimberg@gmail.com

tumik· Jun 16, 2011, 08:28 UTC · re: Grant Limberg · lore

Re: [git-p4] Trouble importing all perforce branches into git repository

Grant Limberg wrote:
Show 6 quoted lines
> 
> I think I just figured out the problem.  The branch mapping between
> //project/MAIN/... and //project/Branch-foo/... was done backwards, thus
> git-p4 was expecting Branch-foo to be a parent of MAIN when the inverse
> was actually true.
> 

Did you get it working so that the branch integrates at Perforce show up as merges in git? I have been trying to do this for tens of hours now and just can't get it working.. I have tried making the branch mapping both ways, and even both at the same time but with no success.

I have succesfully imported the two testing branches to git, but if I look at it in gitk for example, the history is linear and there are no merges from the other branch (it shows up as a normal commit, without the parent from other branch, just adding all lines that the merging did). in p4v it shows up as it should!

-- View this message in context: http://git.661346.n2.nabble.com/git-p4-Trouble-importing-all-perforce-branches-into-git-repository-tp6383422p6482174.html Sent from the git mailing list archive at Nabble.com.

Thomas Berg· Jun 16, 2011, 08:51 UTC · re: tumik · lore

Re: [git-p4] Trouble importing all perforce branches into git repository

On Thu, Jun 16, 2011 at 10:28 AM, tumik <tumik@tuomisalo.com> wrote:
Show 10 quoted lines
> Did you get it working so that the branch integrates at Perforce show up as
> merges in git? I have been trying to do this for tens of hours now and just
> can't get it working.. I have tried making the branch mapping both ways, and
> even both at the same time but with no success.
>
> I have succesfully imported the two testing branches to git, but if I look
> at it in gitk for example, the history is linear and there are no merges
> from the other branch (it shows up as a normal commit, without the parent
> from other branch, just adding all lines that the merging did). in p4v it
> shows up as it should!

git-p4 only uses the branch mapping for the initial commit on a branch. It does not attempt to detect merges after that point. After importing to git, merges just appear as regular commits in the history.

Unfortunately, implementing a general solution for this is non-trivial. Perforce branches and merges per file, meaning that partial merges are possible in Perforce. Such states can not be correctly represented in Git, and depending on your needs, you may or may not want a particular partial merge to be considered as a git merge.

There was another script called git-p4raw for importing Perforce history to git, which does merge detection. It solved the problem by having configurable rules for what should be regarded as a merge, and it supported manually reviewing the decisions afterwards (iirc).

One possibly useful feature in git-p4 would be an option to regard every integration as a complete merge, which shouldn't be too hard to implement. But this would break every time someone did a single-file integration between two branches (which happens quite often in my experience, for cherry-picking changes etc.).

Thomas
tumik· Jun 16, 2011, 09:35 UTC · re: Thomas Berg · lore

Re: [git-p4] Trouble importing all perforce branches into git repository

Thomas Berg wrote:
Show 6 quoted lines
> 
> git-p4 only uses the branch mapping for the initial commit on a
> branch. It does not attempt to detect merges after that point. After
> importing to git, merges just appear as regular commits in the
> history.
> 

Thank you for the info, that's the way it seems to be working! It's a shame if the cloning from perforce -> git can't be made succesfully keeping all the history information :(

Thomas Berg wrote:
Show 6 quoted lines
> 
> There was another script called git-p4raw for importing Perforce
> history to git, which does merge detection. It solved the problem by
> having configurable rules for what should be regarded as a merge, and
> it supported manually reviewing the decisions afterwards (iirc).
> 

Yes, I tried the git-p4raw too. Actually even several forks of it! But I wasn't able to get any of those working, as none of them could load the perforce database correctly. Maybe the perforce database has changed lately or something, as git-p4raw gave errors that it's database tables don't have as much columns as perforce's does. Tried also to change the code to add those columns but there were so many of them and in many tables so I decided it's too hard and time-consuming to start reverse-engineering them..

- tumik

-- View this message in context: http://git.661346.n2.nabble.com/git-p4-Trouble-importing-all-perforce-branches-into-git-repository-tp6383422p6482333.html Sent from the git mailing list archive at Nabble.com.

Thomas Berg· Jun 16, 2011, 10:05 UTC · re: tumik · lore

Re: [git-p4] Trouble importing all perforce branches into git repository

On Thu, Jun 16, 2011 at 11:35 AM, tumik <tumik@tuomisalo.com> wrote:
Show 9 quoted lines
> Yes, I tried the git-p4raw too. Actually even several forks of it! But I
> wasn't able to get any of those working, as none of them could load the
> perforce database correctly. Maybe the perforce database has changed lately
> or something, as git-p4raw gave errors that it's database tables don't have
> as much columns as perforce's does. Tried also to change the code to add
> those columns but there were so many of them and in many tables so I decided
> it's too hard and time-consuming to start reverse-engineering them..
>
> - tumik

I could maybe mention another approach: if your Perforce history isn't too large so you can extract the info manually, or if you are able to create a script to figure out your merges (changelist number, revision of source and target branch in Perforce) it is easy to modify the git history with this info afterwards.

You can add merge arrows to your git-p4 imported repository, either by creating a .git/info/graft file and running git-filter-branch afterwards (to make it permanent and distributable), or by using git-replace. I may have a script somewhere that generates the graft file from p4 changelist numbers and branch names, if you're interested.

The graft + git-filter-branch is good for a one-time conversion, but since it rewrites history it may not work well for branches that still get updated in Perforce.

I have not tested git-replace myself, but it should be possible to use this to add merge arrows too, without modifying the hashes of commits later in the history. I'm not sure how well git-replace works with the rest of git though.

Thomas

← back to recent threads