{"thread":{"id":"27411","subject":"[git-p4] Trouble importing all perforce branches into git repository","startedAt":"2011-05-19T19:31:54Z","lastAt":"2011-06-16T10:05:38Z","messageCount":11,"participants":["Grant Limberg","Pete Wyckoff","Vitor Antunes","tumik","Thomas Berg"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"168252","messageId":"5A927B4F-7242-48AD-BC1C-BCA490A251C4@gmail.com","threadId":"27411","inReplyTo":null,"subject":"[git-p4] Trouble importing all perforce branches into git repository","fromName":"Grant Limberg","fromEmail":"glimberg@gmail.com","sentAt":"2011-05-19T19:31:54Z","receivedAt":"2011-05-19T19:31:54Z","isPatch":false,"sender":{"key":"glimberg@gmail.com","avatar":"https://gravatar.com/avatar/ab54b5130a6b126e24d0db59b75bd4518527df9f6b26ca5822b59c98add8f7cc?d=mp&s=160"},"body":"I'm trying to import a project from my company's Perforce depot into a git repository. I've been successful getting a single branch into a single git repo, but what I'd like to do is to pull in the entire project plus all branches, but I can't get it working correctly.\n\nI have my .p4settings file:\n\n> P4PORT=perforce-server.local:1666\n> P4USER=my.username\n> P4CLIENT=my.clientspec.name\n\nClientspec is fairly simple:\n\n> //project/... //my.clientspec.name/project/...\n> -//project/External/... //my.clientspec.name/project/External/...\n\nThe second line is to exclude the external library binaries from the checkout.\n\nI use the following command line to start git-p4 importing:\n\n> git p4 clone --verbose --detect-branches --max-changes=100 //project/...@all\n> \nThis goes through and checks all the branch names, etc, and attempts to import starting at the first commit (in this case its changelists 771 for this project).\n\n> Importing from //project/...@all into project\n> Initialized empty Git repository in /Users/grant/Development/git_test/project/.git/\n> Reading pipe: git config git-p4.useclientspec\n> Reading pipe: git config git-p4.user\n> Reading pipe: git config git-p4.password\n> Reading pipe: git config git-p4.port\n> Reading pipe: git config git-p4.host\n> Reading pipe: git config git-p4.client\n> p4 -G branches\n> Opening pipe: p4 -G branches\n> p4 -G branch -o branch-1\n> Opening pipe: p4 -G branch -o branch-1\n> Reading pipe: git config --bool core.ignorecase\n> p4 -G branch -o branch-2\n> ...\n> Opening pipe: p4 -G branch -o branch-n\n> p4-git branches: []\n> initial parents: {}\n\nI'm not sure if p4-git branches, and initial parents are supposed to be empty here, but they are.\n\nFinally we get to the point where it's about to start importing changelists into git and the following happens:\n\n> Getting p4 changes for //project/...\n> p4 changes //project/...\n> Reading pipe: p4 changes //project/...\n> p4 -G describe 771\n> Opening pipe: p4 -G describe 771\n> Importing revision 771 (1%)branch is MAIN\n> \n>     Importing new branch RCMerge/MAIN\n> p4 changes //RCMerge/MAIN/...@1,770\n> Reading pipe: p4 changes //RCMerge/MAIN/...@1,770\n> \n>     Resuming with change 771\n> parent determined through known branches: Branch-foo\n> looking for initial parent for refs/remotes/p4/project/MAIN; current parent is refs/remotes/p4/project/Branch-foo\n> commit into refs/remotes/p4/project/MAIN\n> parent refs/remotes/p4/project/Branch-foo\n> p4 -G -x - print\n> Opening pipe: p4 -G -x - print\n> Glue/source/.empty\n> fatal: Invalid ref name or SHA1 expression: refs/remotes/p4/project/Branch-foo\n> fast-import: dumping crash report to .git/fast_import_crash_26002\n\nHere's the file referenced above:\n\n> fast-import crash report:\n>     fast-import process: 26002\n>     parent process     : 26000\n>     at Thu May 19 11:51:54 2011\n> \n> fatal: Invalid ref name or SHA1 expression: refs/remotes/p4/project/Branch-foo\n> \n> Most Recent Commands Before Crash\n> ---------------------------------\n>   checkpoint\n>   commit refs/remotes/p4/project/MAIN\n>   committer Some User <some.user@domain.com> 1253574589 -0800\n>   data <<EOT\n> * from refs/remotes/p4/project/Branch-foo\n> \n> Active Branch LRU\n> -----------------\n>     active_branches = 0 cur, 5 max\n> \n>   pos  clock name\n>   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> \n> Inactive Branches\n> -----------------\n> refs/remotes/p4/project/MAIN:\n>   status      : dirty\n>   tip commit  : 0000000000000000000000000000000000000000\n>   old tree    : 0000000000000000000000000000000000000000\n>   cur tree    : 0000000000000000000000000000000000000000\n>   commit clock: 0\n>   last pack   : \n> \n> \n> Marks\n> -----\n> \n> -------------------\n> END OF CRASH REPORT\n\nNow, being a perforce novice, I have no idea what this means and have no idea how to resolve this issue, or if its even possible. Has anyone run into a similar issue before? If so, how did you resolve it?\n\n\nGrant Limberg\nglimberg@gmail.com\n\n\n\n"},{"id":"168446","messageId":"20110522114917.GA19927@arf.padd.com","threadId":"27411","inReplyTo":"5A927B4F-7242-48AD-BC1C-BCA490A251C4@gmail.com","subject":"Re: [git-p4] Trouble importing all perforce branches into git repository","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2011-05-22T11:49:17Z","receivedAt":"2011-05-22T11:49:17Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"glimberg@gmail.com wrote on Thu, 19 May 2011 19:32 +0000:\n> [git-p4 automatic branch detection]\n> > git p4 clone --verbose --detect-branches --max-changes=100 //project/...@all\n\nI worry that --max-changes prevents it from pulling in the\nref for Branch-foo, and this confuses the automatic branch\ndetection that assumes it has all its parents.\n\nIt would certainly be more reliable if you did \"git-p4 sync\n--branch foo //project/foo@all\" for each branch you care about.\n\nVitor has some modifications to auto-branch detection that might\nbe useful, or maybe some thoughts as to what's going on.\n\n\t\t-- Pete\n\n(Vitor: full mail is here:\nhttp://article.gmane.org/gmane.comp.version-control.git/173996/ )\n"},{"id":"168478","messageId":"398FA881-E4A1-49AC-80F2-2D46E9F2ABB9@gmail.com","threadId":"27411","inReplyTo":"20110522114917.GA19927@arf.padd.com","subject":"Re: [git-p4] Trouble importing all perforce branches into git repository","fromName":"Grant Limberg","fromEmail":"glimberg@gmail.com","sentAt":"2011-05-23T03:27:48Z","receivedAt":"2011-05-23T03:27:48Z","isPatch":false,"sender":{"key":"glimberg@gmail.com","avatar":"https://gravatar.com/avatar/ab54b5130a6b126e24d0db59b75bd4518527df9f6b26ca5822b59c98add8f7cc?d=mp&s=160"},"body":"On May 22, 2011, at 4:49 AM, Pete Wyckoff wrote:\n\n> glimberg@gmail.com wrote on Thu, 19 May 2011 19:32 +0000:\n>> [git-p4 automatic branch detection]\n>>> git p4 clone --verbose --detect-branches --max-changes=100 //project/...@all\n> \n> I worry that --max-changes prevents it from pulling in the\n> ref for Branch-foo, and this confuses the automatic branch\n> detection that assumes it has all its parents.\n> \n> It would certainly be more reliable if you did \"git-p4 sync\n> --branch foo //project/foo@all\" for each branch you care about.\n> \n> Vitor has some modifications to auto-branch detection that might\n> be useful, or maybe some thoughts as to what's going on.\n> \n> \t\t-- Pete\n> \n> (Vitor: full mail is here:\n> http://article.gmane.org/gmane.comp.version-control.git/173996/ )\n> \n\nRemoving --max-changes=100 has no effect on the crash I'm experiencing.   Same crash still happens on the very first imported commit.\n\nIf i do \"git-p4 sync --branch foo //project/foo@all\" for each branch, does that convert 'p4 integrate' commands into git merges?  I\"m trying to keep all history intact when importing from p4 into git.\n\nThanks,\n\n-Grant\n\n"},{"id":"168492","messageId":"BANLkTi=TCyyS7Q=3BnLcG=yhL_boH=w1XA@mail.gmail.com","threadId":"27411","inReplyTo":"398FA881-E4A1-49AC-80F2-2D46E9F2ABB9@gmail.com","subject":"Re: [git-p4] Trouble importing all perforce branches into git repository","fromName":"Vitor Antunes","fromEmail":"vitor.hda@gmail.com","sentAt":"2011-05-23T10:54:18Z","receivedAt":"2011-05-23T10:54:18Z","isPatch":false,"sender":{"key":"vitor.hda@gmail.com","avatar":"https://avatars.githubusercontent.com/u/606876?v=4"},"body":"Hi Grant,\n\nI think I saw that happening when there is nothing to import or in\nsituations where the parent of a certain branch does not exist.\n\nPerforce does not strictly require branch definitions to allow a branch\nstructure/flow. It is quite possible that you are integrating stuff\naround without using \"branch specs\". When git-p4 is importing your P4\ndatabase it will look at all branches available in the server, including\nones from other depots. So it is possible that while you are seeing a\nbig list of branches being processed, none apply to the branch structure\nyou are trying to import.\n\nCould you please confirm that you have Perforce branch specs for all\nbranches that you are trying to import?\n\nThanks,\nVitor\n"},{"id":"168506","messageId":"34E33A18-B9C4-4CA9-B96C-79B0E2BDCD44@gmail.com","threadId":"27411","inReplyTo":"BANLkTi=TCyyS7Q=3BnLcG=yhL_boH=w1XA@mail.gmail.com","subject":"Re: [git-p4] Trouble importing all perforce branches into git repository","fromName":"Grant Limberg","fromEmail":"glimberg@gmail.com","sentAt":"2011-05-23T16:18:20Z","receivedAt":"2011-05-23T16:18:20Z","isPatch":false,"sender":{"key":"glimberg@gmail.com","avatar":"https://gravatar.com/avatar/ab54b5130a6b126e24d0db59b75bd4518527df9f6b26ca5822b59c98add8f7cc?d=mp&s=160"},"body":"On May 23, 2011, at 3:54 AM, Vitor Antunes wrote:\n\n> Hi Grant,\n> \n> I think I saw that happening when there is nothing to import or in\n> situations where the parent of a certain branch does not exist.\n> \n> Perforce does not strictly require branch definitions to allow a branch\n> structure/flow. It is quite possible that you are integrating stuff\n> around without using \"branch specs\". When git-p4 is importing your P4\n> database it will look at all branches available in the server, including\n> ones from other depots. So it is possible that while you are seeing a\n> big list of branches being processed, none apply to the branch structure\n> you are trying to import.\n> \n> Could you please confirm that you have Perforce branch specs for all\n> branches that you are trying to import?\n> \n> Thanks,\n> Vitor\n\nVitor, \n\nThe \"Branch-foo\" that git-p4 is crashing on does have a branchspec in Perforce.  I cannot, however guarantee that branchspecs were correctly created and used for all integrates in the repository.  It was created long before I started working here.  Is there a way to limit --detect-branches to only certain branches that I can ensure do have branchspecs?\n\n- Grant"},{"id":"168512","messageId":"BANLkTik+Zp1Fvi_zABCtAZH0RKA68n5Svw@mail.gmail.com","threadId":"27411","inReplyTo":"34E33A18-B9C4-4CA9-B96C-79B0E2BDCD44@gmail.com","subject":"Re: [git-p4] Trouble importing all perforce branches into git repository","fromName":"Vitor Antunes","fromEmail":"vitor.hda@gmail.com","sentAt":"2011-05-23T17:38:03Z","receivedAt":"2011-05-23T17:38:03Z","isPatch":false,"sender":{"key":"vitor.hda@gmail.com","avatar":"https://avatars.githubusercontent.com/u/606876?v=4"},"body":"Hi Grant,\n\n> The \"Branch-foo\" that git-p4 is crashing on does have a branchspec in\n> Perforce.  I cannot, however guarantee that branchspecs were\n> correctly created and used for all integrates in the repository.  It\n> was created long before I started working here.\n\nYou can create the branch specs now. There is no requirement for them to\nhave been created at the time of the integration.\n\nI've sent a patch to this mailing list [1] that adds the possibility of\ncreating these branches in a gitconfig file. If you feel brave enough,\nplease give it a try. The patch is missing a test case, which I still\ndid not have time to review and send\n\n> Is there a way to limit --detect-branches to only certain branches\n> that I can ensure do have branchspecs?\n\nIf you have an incremental structure you could limit the cloning to a\nsubset of the directories. I am not aware of any other way. Maybe\nsomeone else in the mailing list can shed some light...? :)\n\nVitor\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/167998/focus=168000\n"},{"id":"168520","messageId":"0AC6DD14-3042-4A18-91AC-1CE77D8B4CD2@gmail.com","threadId":"27411","inReplyTo":"BANLkTik+Zp1Fvi_zABCtAZH0RKA68n5Svw@mail.gmail.com","subject":"Re: [git-p4] Trouble importing all perforce branches into git repository","fromName":"Grant Limberg","fromEmail":"glimberg@gmail.com","sentAt":"2011-05-23T20:06:15Z","receivedAt":"2011-05-23T20:06:15Z","isPatch":false,"sender":{"key":"glimberg@gmail.com","avatar":"https://gravatar.com/avatar/ab54b5130a6b126e24d0db59b75bd4518527df9f6b26ca5822b59c98add8f7cc?d=mp&s=160"},"body":"On May 23, 2011, at 10:38 AM, Vitor Antunes wrote:\n\n> Hi Grant,\n> \n>> The \"Branch-foo\" that git-p4 is crashing on does have a branchspec in\n>> Perforce.  I cannot, however guarantee that branchspecs were\n>> correctly created and used for all integrates in the repository.  It\n>> was created long before I started working here.\n> \n> You can create the branch specs now. There is no requirement for them to\n> have been created at the time of the integration.\n> \n> I've sent a patch to this mailing list [1] that adds the possibility of\n> creating these branches in a gitconfig file. If you feel brave enough,\n> please give it a try. The patch is missing a test case, which I still\n> did not have time to review and send\n> \n>> Is there a way to limit --detect-branches to only certain branches\n>> that I can ensure do have branchspecs?\n> \n> If you have an incremental structure you could limit the cloning to a\n> subset of the directories. I am not aware of any other way. Maybe\n> someone else in the mailing list can shed some light...? :)\n> \n> Vitor\n> \n> [1] http://thread.gmane.org/gmane.comp.version-control.git/167998/focus=168000\n\nI think I just figured out the problem.  The branch mapping between //project/MAIN/... and //project/Branch-foo/... was done backwards, thus git-p4 was expecting Branch-foo to be a parent of MAIN when the inverse was actually true.  MAIN's first changelist number was 771.  Branch-foo's first changelist number was 7652.  That's why git-p4 was getting confused.  I fixed the branch mapping in Perforce and the clone with --detect-branches seems to be working fine now. \n\nGrant Limberg\nglimberg@gmail.com\n\n\n\n\n"},{"id":"170118","messageId":"1308212906486-6482174.post@n2.nabble.com","threadId":"27411","inReplyTo":"0AC6DD14-3042-4A18-91AC-1CE77D8B4CD2@gmail.com","subject":"Re: [git-p4] Trouble importing all perforce branches into git repository","fromName":"tumik","fromEmail":"tumik@tuomisalo.com","sentAt":"2011-06-16T08:28:26Z","receivedAt":"2011-06-16T08:28:26Z","isPatch":false,"sender":{"key":"tumik@tuomisalo.com","avatar":null},"body":"\nGrant Limberg wrote:\n> \n> I think I just figured out the problem.  The branch mapping between\n> //project/MAIN/... and //project/Branch-foo/... was done backwards, thus\n> git-p4 was expecting Branch-foo to be a parent of MAIN when the inverse\n> was actually true.\n> \n\nDid you get it working so that the branch integrates at Perforce show up as\nmerges in git? I have been trying to do this for tens of hours now and just\ncan't get it working.. I have tried making the branch mapping both ways, and\neven both at the same time but with no success.\n\nI have succesfully imported the two testing branches to git, but if I look\nat it in gitk for example, the history is linear and there are no merges\nfrom the other branch (it shows up as a normal commit, without the parent\nfrom other branch, just adding all lines that the merging did). in p4v it\nshows up as it should!\n\n--\nView this message in context: http://git.661346.n2.nabble.com/git-p4-Trouble-importing-all-perforce-branches-into-git-repository-tp6383422p6482174.html\nSent from the git mailing list archive at Nabble.com.\n"},{"id":"170120","messageId":"BANLkTikTt4vMj+iZMhGEi6JefRUkZGZ8dg@mail.gmail.com","threadId":"27411","inReplyTo":"1308212906486-6482174.post@n2.nabble.com","subject":"Re: [git-p4] Trouble importing all perforce branches into git repository","fromName":"Thomas Berg","fromEmail":"merlin66b@gmail.com","sentAt":"2011-06-16T08:51:59Z","receivedAt":"2011-06-16T08:51:59Z","isPatch":false,"sender":{"key":"merlin66b@gmail.com","avatar":null},"body":"On Thu, Jun 16, 2011 at 10:28 AM, tumik <tumik@tuomisalo.com> wrote:\n> Did you get it working so that the branch integrates at Perforce show up as\n> merges in git? I have been trying to do this for tens of hours now and just\n> can't get it working.. I have tried making the branch mapping both ways, and\n> even both at the same time but with no success.\n>\n> I have succesfully imported the two testing branches to git, but if I look\n> at it in gitk for example, the history is linear and there are no merges\n> from the other branch (it shows up as a normal commit, without the parent\n> from other branch, just adding all lines that the merging did). in p4v it\n> shows up as it should!\n\ngit-p4 only uses the branch mapping for the initial commit on a\nbranch. It does not attempt to detect merges after that point. After\nimporting to git, merges just appear as regular commits in the\nhistory.\n\nUnfortunately, implementing a general solution for this is\nnon-trivial. Perforce branches and merges per file, meaning that\npartial merges are possible in Perforce. Such states can not be\ncorrectly represented in Git, and depending on your needs, you may or\nmay not want a particular partial merge to be considered as a git\nmerge.\n\nThere was another script called git-p4raw for importing Perforce\nhistory to git, which does merge detection. It solved the problem by\nhaving configurable rules for what should be regarded as a merge, and\nit supported manually reviewing the decisions afterwards (iirc).\n\nOne possibly useful feature in git-p4 would be an option to regard\nevery integration as a complete merge, which shouldn't be too hard to\nimplement. But this would break every time someone did a single-file\nintegration between two branches (which happens quite often in my\nexperience, for cherry-picking changes etc.).\n\nThomas\n"},{"id":"170122","messageId":"1308216928289-6482333.post@n2.nabble.com","threadId":"27411","inReplyTo":"BANLkTikTt4vMj+iZMhGEi6JefRUkZGZ8dg@mail.gmail.com","subject":"Re: [git-p4] Trouble importing all perforce branches into git repository","fromName":"tumik","fromEmail":"tumik@tuomisalo.com","sentAt":"2011-06-16T09:35:28Z","receivedAt":"2011-06-16T09:35:28Z","isPatch":false,"sender":{"key":"tumik@tuomisalo.com","avatar":null},"body":"\nThomas Berg wrote:\n> \n> git-p4 only uses the branch mapping for the initial commit on a\n> branch. It does not attempt to detect merges after that point. After\n> importing to git, merges just appear as regular commits in the\n> history.\n> \nThank you for the info, that's the way it seems to be working! It's a shame\nif the cloning from perforce -> git can't be made succesfully keeping all\nthe history information :(\n\n\nThomas Berg wrote:\n> \n> There was another script called git-p4raw for importing Perforce\n> history to git, which does merge detection. It solved the problem by\n> having configurable rules for what should be regarded as a merge, and\n> it supported manually reviewing the decisions afterwards (iirc).\n> \nYes, I tried the git-p4raw too. Actually even several forks of it! But I\nwasn't able to get any of those working, as none of them could load the\nperforce database correctly. Maybe the perforce database has changed lately\nor something, as git-p4raw gave errors that it's database tables don't have\nas much columns as perforce's does. Tried also to change the code to add\nthose columns but there were so many of them and in many tables so I decided\nit's too hard and time-consuming to start reverse-engineering them..\n\n- tumik\n\n--\nView this message in context: http://git.661346.n2.nabble.com/git-p4-Trouble-importing-all-perforce-branches-into-git-repository-tp6383422p6482333.html\nSent from the git mailing list archive at Nabble.com.\n"},{"id":"170123","messageId":"BANLkTikMzxGDqX7FnNFx+fmFuXVi6NncJA@mail.gmail.com","threadId":"27411","inReplyTo":"1308216928289-6482333.post@n2.nabble.com","subject":"Re: [git-p4] Trouble importing all perforce branches into git repository","fromName":"Thomas Berg","fromEmail":"merlin66b@gmail.com","sentAt":"2011-06-16T10:05:38Z","receivedAt":"2011-06-16T10:05:38Z","isPatch":false,"sender":{"key":"merlin66b@gmail.com","avatar":null},"body":"On Thu, Jun 16, 2011 at 11:35 AM, tumik <tumik@tuomisalo.com> wrote:\n> Yes, I tried the git-p4raw too. Actually even several forks of it! But I\n> wasn't able to get any of those working, as none of them could load the\n> perforce database correctly. Maybe the perforce database has changed lately\n> or something, as git-p4raw gave errors that it's database tables don't have\n> as much columns as perforce's does. Tried also to change the code to add\n> those columns but there were so many of them and in many tables so I decided\n> it's too hard and time-consuming to start reverse-engineering them..\n>\n> - tumik\n\nI could maybe mention another approach: if your Perforce history isn't\ntoo large so you can extract the info manually, or if you are able to\ncreate a script to figure out your merges (changelist number, revision\nof source and target branch in Perforce) it is easy to modify the git\nhistory with this info afterwards.\n\nYou can add merge arrows to your git-p4 imported repository, either by\ncreating a .git/info/graft file and running git-filter-branch\nafterwards (to make it permanent and distributable), or by using\ngit-replace. I may have a script somewhere that generates the graft\nfile from p4 changelist numbers and branch names, if you're\ninterested.\n\nThe graft + git-filter-branch is good for a one-time conversion, but\nsince it rewrites history it may not work well for branches that still\nget updated in Perforce.\n\nI have not tested git-replace myself, but it should be possible to use\nthis to add merge arrows too, without modifying the hashes of commits\nlater in the history. I'm not sure how well git-replace works with the\nrest of git though.\n\nThomas\n"}]}