git-p4: Clone p4 path with bidirectional integrations

Hi all,

Is it possible to `git p4 clone --detect-branches` from a Perforce path which contains bidirectional integrations?

I've tried a bunch of things to get this to work, but here's an example which hopefully illustrates what I'm trying to accomplish and the issue I'm having.

Perforce setup, assuming PWD is mapped to //depot/... in your client spec:

  1. mkdir -p testing/master
  2. touch testing/master/test1 && p4 add testing/master/test1 && p4 submit
  3. p4 integrate //depot/testing/master/...
//depot/testing/staging/... && p4 submit
  3. touch testing/staging/test2 && p4 add testing/staging/test2 && p4 submit
  4. p4 integrate //depot/testing/staging/...
//depot/testing/master/... && p4 submit

Now try to clone with git-p4:

  1. git init p4_git_test && cd p4_git_test
  2. git config git-p4.branchList master:staging
  3. git config --add git-p4.branchList staging:master
  4. git p4 clone //depot/testing/...@all --detect-branches .

You end up with a failure like:

  Importing from //depot/testing/...@all into .
  Reinitialized existing Git repository in /home/amiller/p4_git_test/.git/
  Importing revision 1205832 (25%)
      Importing new branch testing/master

      Resuming with change 1205832
  fatal: ambiguous argument 'refs/remotes/p4/testing/staging': unknown
revision or path not in the working tree.
  Use '--' to separate paths from revisions, like this:
  'git <command> [<revision>...] -- [<file>...]'
  Command failed: ['git', 'rev-list', '--reverse', '--no-merges',
'refs/remotes/p4/testing/staging']

I'm using Git 2.22.1.

Thanks, Aaron

On Mon, 19 Aug 2019 at 18:30, Aaron Miller <aaronkmiller@gmail.com> wrote:

I have to admit I don't use the detect-branches code myself.

It's possible that running with "-v" might give a bit more information.

Can you write a test case, or even just a shell script, that might help figure out what's going on.

Unfortunately Perforce doesn't really know about branches, so git-p4 has to make a guess!

Hi Luke,

Here's the output from that. I've set git-p4.branchUser in this test to avoid needlessly cluttering the output since I have a huge amount of branches in my Perforce repo, but otherwise I used the exact script which I've included later in this email:

Importing from //depot/testing/...@all into . Reinitialized existing Git repository in /home/amiller/Code/git-migration/repos/testing/.git/ Reading pipe: ['git', 'config', '--bool', 'git-p4.useclientspec'] Reading pipe: ['git', 'config', 'git-p4.branchUser'] Reading pipe: ['git', 'config', 'git-p4.user'] Reading pipe: ['git', 'config', 'git-p4.password'] Reading pipe: ['git', 'config', 'git-p4.port'] Reading pipe: ['git', 'config', 'git-p4.host'] Reading pipe: ['git', 'config', 'git-p4.client'] Reading pipe: ['git', 'config', '--int', 'git-p4.retries'] Reading pipe: ['git', 'config', '--int', 'git-p4.retries'] Opening pipe: ['p4', '-r', '3', '-G', 'login', '-s'] Opening pipe: p4 -r 3 -G branches -u amiller Reading pipe: ['git', 'config', '--get-all', 'git-p4.branchList'] p4-git branches: [] initial parents: {} Getting p4 changes for //depot/testing/... Opening pipe: ['p4', '-r', '3', '-G', 'changes', '-m', '1'] Opening pipe: ['p4', '-r', '3', '-G', 'changes', '//depot/testing/...@1,1048577'] Opening pipe: ['p4', '-r', '3', '-G', 'changes', '//depot/testing/...@1048578,1206544'] Opening pipe: ['p4', '-r', '3', '-G', 'describe', '-s', '1206099'] Importing revision 1206099 (25%)Reading pipe: ['git', 'config', '--bool', 'core.ignorecase'] branch is master

    Importing new branch testing/master
Opening pipe: ['p4', '-r', '3', '-G', 'changes',
'//depot/testing/master/...@1,1048577']
Opening pipe: ['p4', '-r', '3', '-G', 'changes',
'//depot/testing/master/...@1048578,1206098']

    Resuming with change 1206099
parent determined through known branches: staging
looking for initial parent for refs/remotes/p4/testing/master; current
parent is refs/remotes/p4/testing/staging
Creating temporary branch: refs/git-p4-tmp/1206099
commit into refs/git-p4-tmp/1206099
Reading pipe: ['git', 'config', '--bool', 'git-p4.keepEmptyCommits']
Opening pipe: ['p4', '-r', '3', '-G', '-x', '-', 'print']
//depot/testing/master/test1 --> test1 (0 MB)
checkpoint finished: progress checkpoint

Reading pipe: ['git', 'rev-list', '--reverse', '--no-merges',
'refs/remotes/p4/testing/staging']
fatal: ambiguous argument 'refs/remotes/p4/testing/staging': unknown
revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Traceback (most recent call last):
  File "/home/amiller/.bin/git-p4.py", line 4173, in <module>
    main()
  File "/home/amiller/.bin/git-p4.py", line 4167, in main
    if not cmd.run(args):
  File "/home/amiller/.bin/git-p4.py", line 3923, in run
    if not P4Sync.run(self, depotPaths):
  File "/home/amiller/.bin/git-p4.py", line 3790, in run
    self.importChanges(changes)
  File "/home/amiller/.bin/git-p4.py", line 3451, in importChanges
    blob = self.searchParent(parent, branch, tempBranch)
  File "/home/amiller/.bin/git-p4.py", line 3374, in searchParent
    "--no-merges", parent]):
  File "/home/amiller/.bin/git-p4.py", line 237, in read_pipe_lines
    die('Command failed: %s' % str(c))
  File "/home/amiller/.bin/git-p4.py", line 165, in die
    raise Exception(msg)
Exception: Command failed: ['git', 'rev-list', '--reverse',
'--no-merges', 'refs/remotes/p4/testing/staging']

No problem:

#!/bin/bash

# perforce setup - assumes PWD is mapped to //depot/... mkdir -p testing/master touch testing/master/test1 p4 add testing/master/test1 p4 submit -d 'test changelist 1'

p4 integrate //depot/testing/master/... //depot/testing/staging/... p4 submit -d 'test changelist 2'

touch testing/staging/test2 p4 add testing/staging/test2 p4 submit -d 'test changelist 3'

p4 integrate //depot/testing/staging/... //depot/testing/master/... p4 submit -d 'test changelist 4'

# clone with git-p4: git init p4_git_test cd p4_git_test git config git-p4.branchList master:staging git config --add git-p4.branchList staging:master git p4 clone //depot/testing/...@all --detect-branches --verbose .

Thanks, Aaron

19.08.2019, 13:30, "Aaron Miller" <aaronkmiller@gmail.com>:

Yes, but it would require some manual work most likely.

First of all, git-p4 should normally take only one direction from bidirectional integrations on its own. Do you see "p4 branch <branchABC> defines a mapping from <path1> to <path2>, but there exists another mapping from <path2> to <path1> already!"? If you do, it means that git-p4 will ignore <branchABC> mapping.

Also, just FYI, as far as I know, git-p4 doesn't create "merge" commits, so bidirectional integrations won't look different from ordinary commits in git commit graph.

Uh-oh, 5M commits! But given that it fails at 25% instead of 1%, you've got some luck. :)

This might not be just because of bidirectional integrations per se. This error may happen if, say, there's a P4 branch mapping from staging to master, but master was actually created before staging. git-p4 tries to find a "parent branch" for master, but it doesn't exist yet, so git-p4 fails in an ugly way.

One way to filter out troublesome P4 branch mappings is to set git-p4.branchUser to a particular user. But most likely, this won't help you because different people created different branch mappings over time.

Unfortunately, there's _no_ git-p4.branchRegexp config option, but it's fairly straightforward to implement -- patches welcome! ;) (getBranchMapping() needs to apply a regex to branch names before doing anything serious with them)

The other option is to manually set git-p4.branchList for all your branch pairs like git config --add git-p4.branchList staging:master git config --add git-p4.branchList master:branchA git config --add git-p4.branchList master:branchB ... (or by manually editing .git/config) Note, that you can't have master:staging together with staging:master, otherwise you'll likely run into the same problem as before.

This might be simple or quite tedious depending on the history and branching strategies of your repositories. It may be as easy as just dropping some p4 branch mappings. However, one of the repositories I had to deal with had almost random branching strategy (with most integrations done without predefined branch mappings), so I had to spend quite some time to trace the history and figure out which branches make the most sense in git. (Revision Graph in p4v was very helpful for figuring branching history out)

As I said, git-p4 doesn't create merge commits in git (or I can't see how to make them), so for repositories with simple/short history, I recreated those merge commits manually (well, in a bash script) using `git replace --graft <commit> <parent1> <parent2>` followed by `git filter-branch --tag-name-filter cat -- --all` to make grafts permanent. (`git filter-branch` is only needed once in the very end after all `git replace` manipulations are done) It's perhaps better to teach git-p4 to produce merge commits, but a bash script was a low-tech low-risk option for me.

Also, beware that git-p4 doesn't handle branch-into-non-empty directory properly. If I remember correctly, something like `p4 copy //depot/branchA/... //depot/branchB/... ; p4 submit; p4 copy //depot/branchC/... //depot/branchB/...; p4 submit` will result in branchB having _both_ branchA and branchC contents in git. `git filter-branch` or `git rebase` are your friends to workaround this. (or better fix git-p4, of course)

Hope this help, Andrey.

Hi Andrey,

Thanks so much for this detailed response, I really appreciate it.

I was afraid that was the case!

Ah, I didn't realize that, thank you. Perhaps I should just sync each branch separately then, ignoring branch mappings entirely and be done with it.

I was hoping to generate a commit graph that properly represents integrations as merge commits because our Perforce branches are quite large. Storing diffs of integrations rather than discrete commits would result in a much smaller Git repository for us. There are *quite* a lot of integration commits.

I wonder how hard it would be to modify git-p4 to use merge commits? I will take a look at the source. I'm somewhat surprised that's not the case to begin with though? Maybe someone else can chime in on why merge commits aren't used.

Hehe :) Fortunately I don't have to migrate *all* of those commits, but still a fair number.

I should have mentioned - there were no P4 branch mappings defined for the depot paths in the test case I shared (maybe that's obvious).

Yeah this is exactly correct, I've tested precisely this scenario. Actually that's what two of the real branches I am trying to migrate look like.

I was thinking I could set git-p4.branchUser to a user that hasn't created any branches at all and then define branch mappings I care about in git-p4.branchList.

But pretty much all of our branches either have bidirectional integrations or a situation like you described above where branchA was created before branchB, and branchB integrates into branchA.

That would be a neat option! It's not really filtering the branch mappings which is my issue though. The number of branches is small enough that I can manage the mappings manually.

Yeah this is really the crux of my issue.

I'm thinking if I'm not getting merge commits anyway I may as well just forget about the branch mappings entirely.

Fortunately our branching strategy was *mostly* well-defined and we do have Perforce mappings defined between our branches. However most integrations didn't actually use the branch mappings, just manually specified depot paths. But they did fall under the branch mapping scopes.

Unfortunately our repo history is anything but short or simple :(

These are great tips, thanks! Maybe what I will do is sync each branch separately with no branch mappings, then use this technique to create merge commits for the initial branch creation commits only and not worry about any other integrations.

-Aaron

Aaron,

20.08.2019, 19:46, "Aaron Miller" <aaronkmiller@gmail.com>:

You could do it this way too. But this way your branches will be completely independent. However, with --detect-branches, you'll preserve the information on which branch was branched off at which point. (it's useful for release branches or feature branches; but not really for main and testing branching model)

Not sure if there'll be a big difference in terms of Git repository size. Git is quite efficient with its delta compression and pack files.

It's somewhere around importChanges() and commit(), I suppose. It seems simple on git side (just specify additional parents when creating a commit using "merge" keyword for git fast-import), but may be more complicated on Perforce side (to find the parent(s)).

Yeah, it comes as a bad surprise. One reason I can think of is bi-directional interaction between Perforce and Git -- perhaps some information is lost during submit to P4 and then reimport back to Git. It might also get tricky during import of changelists that span several branches (shouldn't probably be considered a merge though).

Well, you had git-p4.branchList defined, which is enough for --detect-branches logic to kick in.

Yeah, it's a good idea. (I did the same) As far as I remember, you can even use a non-existing user for git-p4.branchUser.

Another suggestion: after import into git is done, it might be useful to cross-check tips of git branches against tips of corresponding Perforce branches. (some files may exist in git when they are deleted from Perforce due to the aforementioned branch-into-non-empty directory issue)

If commits for integrations have consistent commit messages, you might be able to get merge commits using `git replace` cheaper than by modifying git-p4 itself (which can't rely on commit messages).