{"thread":{"id":"40193","subject":"[PATCHv1 2/3] git-p4: do not terminate creating tag for unknown commit","startedAt":"2015-08-27T07:18:55Z","lastAt":"2015-09-14T06:45:31Z","messageCount":6,"participants":["Luke Diamand","Eric Sunshine","Holl, Marcus"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"268781","messageId":"1440659938-12952-1-git-send-email-luke@diamand.org","threadId":"40193","inReplyTo":null,"subject":"[PATCHv1 0/3] git-p4: fixing import with labels when commit is not transferred","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2015-08-27T07:18:55Z","receivedAt":"2015-08-27T07:18:55Z","isPatch":false,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"Marcus Holl discovered that git-p4 fails to import labels sometimes.\n\nhttp://permalink.gmane.org/gmane.comp.version-control.git/273034\n\nIt turns out there are at least two related problems.\n\nThe first is that if git-p4 tries to import a P4 tag which references\na commit that we don't know about at all (e.g. because the P4\nchangelist was created before the point at which the repo was\ncloned) then when it tries to find the commit, the git rev-list\ncommand fails, and git-p4 terminates.\n\nThis can be fixed by detecting the problem and ignoring that label.\n\nThe second problem is that git-p4 can end up trying to find out\nabout a commit which is still in the git fast-import stream and not\nyet processed. This is fixed by using fast-import \"marks\" to reference\nthese commits, which avoids having to query for commits which don't\nyet exist. The normal path is used for commits that have not been\nimported during this run of the program (and so won't have marks).\n\nA test case is added to demonstrate the problem.\n\nMarcus - if you're able to take a look at this and see if it\nfixes your problem that would be very useful.\n\nThanks,\nLuke\n\nLuke Diamand (3):\n git-p4: failing test for ignoring invalid p4 labels\n git-p4: do not terminate creating tag for unknown commit\n git-p4: fix P4 label import for unprocessed commits\n\n git-p4.py | 25 +++++++++++++++--------\n t/t9811-git-p4-label-import.sh | 45 ++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 62 insertions(+), 8 deletions(-)\n\n-- \n2.5.0.rc0\n"},{"id":"268780","messageId":"1440659938-12952-2-git-send-email-luke@diamand.org","threadId":"40193","inReplyTo":"1440659938-12952-1-git-send-email-luke@diamand.org","subject":"[PATCHv1 1/3] git-p4: failing test for ignoring invalid p4 labels","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2015-08-27T07:18:56Z","receivedAt":"2015-08-27T07:18:56Z","isPatch":false,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"When importing a label which references a commit that git-p4 does\nnot know about, git-p4 should skip it and go on to process other\nlabels that can be imported.\n\nInstead it crashes when attempting to find the missing commit in\nthe git history. This test demonstrates the problem.\n\nSigned-off-by: Luke Diamand \n---\n t/t9811-git-p4-label-import.sh | 45 ++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 45 insertions(+)\n\ndiff --git a/t/t9811-git-p4-label-import.sh b/t/t9811-git-p4-label-import.sh\nindex 095238f..f7d5048 100755\n--- a/t/t9811-git-p4-label-import.sh\n+++ b/t/t9811-git-p4-label-import.sh\n@@ -214,6 +214,51 @@ test_expect_success 'use git config to enable import/export of tags' '\n \t)\n '\n \n+p4_head_revision() {\n+\tp4 changes -m 1 \"$@\" | awk '{print $2}'\n+}\n+\n+# Importing a label that references a P4 commit that has\n+# has not been seen. The presence of a label on a commit\n+# we haven't seen should not cause git-p4 to fail. It should\n+# merely skip that label, and still import other labels.\n+test_expect_failure 'importing labels with missing revisions' '\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\trm -fr \"$cli\" \"$git\" &&\n+\t\tmkdir \"$cli\" &&\n+\t\tP4CLIENT=missing-revision &&\n+\t\tclient_view \"//depot/missing-revision/... //missing-revision/...\" &&\n+\t\tcd \"$cli\" &&\n+\t\t>f1 && p4 add f1 && p4 submit -d \"start\" &&\n+\n+\t\tp4 tag -l TAG_S0 ... &&\n+\n+\t\t>f2 && p4 add f2 && p4 submit -d \"second\" &&\n+\n+\t\tstartrev=$(p4_head_revision //depot/missing-revision/...) &&\n+\n+\t\t>f3 && p4 add f3 && p4 submit -d \"third\" &&\n+\n+\t\tp4 edit f2 && date >f2 && p4 submit -d \"change\" f2 &&\n+\n+\t\tendrev=$(p4_head_revision //depot/missing-revision/...) &&\n+\n+\t\tp4 tag -l TAG_S1 ... &&\n+\n+\t\t# we should skip TAG_S0 since it is before our startpoint,\n+\t\t# but pick up TAG_S1.\n+\n+\t\tgit p4 clone --dest=\"$git\" --import-labels -v \\\n+\t\t\t//depot/missing-revision/...@$startrev,$endrev &&\n+\t\t(\n+\t\t\tcd \"$git\" &&\n+\t\t\tgit rev-parse TAG_S1 &&\n+\t\t\t! git rev-parse TAG_S0\n+\t\t)\n+\t)\n+'\n+\n \n test_expect_success 'kill p4d' '\n \tkill_p4d\n-- \n2.5.0.rc0\n"},{"id":"268779","messageId":"1440659938-12952-3-git-send-email-luke@diamand.org","threadId":"40193","inReplyTo":"1440659938-12952-1-git-send-email-luke@diamand.org","subject":"[PATCHv1 2/3] git-p4: do not terminate creating tag for unknown commit","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2015-08-27T07:18:57Z","receivedAt":"2015-08-27T07:18:57Z","isPatch":false,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"If p4 reports a tag for a commit that git-p4 does not know\nabout (e.g. because it references a P4 changelist that was\nimported prior to the point at which the repo was cloned into\ngit), make sure that the error is correctly caught and handled.\nrather than just crashing.\n\nSigned-off-by: Luke Diamand \n---\n git-p4.py | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 073f87b..a62611a 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -2494,9 +2494,9 @@ class P4Sync(Command, P4UserMap):\n # find the corresponding git commit; take the oldest commit\n changelist = int(change['change'])\n gitCommit = read_pipe([\"git\", \"rev-list\", \"--max-count=1\",\n- \"--reverse\", \":/\\[git-p4:.*change = %d\\]\" % changelist])\n+ \"--reverse\", \":/\\[git-p4:.*change = %d\\]\" % changelist], ignore_error=True)\n if len(gitCommit) == 0:\n- print \"could not find git commit for changelist %d\" % changelist\n+ print \"importing label %s: could not find git commit for changelist %d\" % (name, changelist)\n else:\n gitCommit = gitCommit.strip()\n commitFound = True\n-- \n2.5.0.rc0\n"},{"id":"268782","messageId":"1440659938-12952-4-git-send-email-luke@diamand.org","threadId":"40193","inReplyTo":"1440659938-12952-1-git-send-email-luke@diamand.org","subject":"[PATCHv1 3/3] git-p4: fix P4 label import for unprocessed commits","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2015-08-27T07:18:58Z","receivedAt":"2015-08-27T07:18:58Z","isPatch":false,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"With --detect-labels enabled, git-p4 will try to create tags\nusing git fast-import by writing a \"tag\" clause to the\nfast-import stream.\n\nIf the commit that the tag references has not yet actually\nbeen processed by fast-import, then the tag can't be created\nand git-p4 fails to import the P4 label.\n\nTeach git-p4 to use fast-import \"marks\" when creating tags\nwhich reference commits created during the current run of the\nprogram.\n\nCommits created before the current run are still referenced\nin the old way using a normal git commit.\n\nSigned-off-by: Luke Diamand \n---\n git-p4.py | 25 +++++++++++++++++--------\n t/t9811-git-p4-label-import.sh | 2 +-\n 2 files changed, 18 insertions(+), 9 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex a62611a..2018011 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -2322,8 +2322,11 @@ class P4Sync(Command, P4UserMap):\n else:\n return \"%s \" % userid\n \n- # Stream a p4 tag\n def streamTag(self, gitStream, labelName, labelDetails, commit, epoch):\n+ \"\"\" Stream a p4 tag.\n+ commit is either a git commit, or a fast-import mark, \":\"\n+ \"\"\"\n+\n if verbose:\n print \"writing tag %s for commit %s\" % (labelName, commit)\n gitStream.write(\"tag %s\\n\" % labelName)\n@@ -2374,7 +2377,7 @@ class P4Sync(Command, P4UserMap):\n self.clientSpecDirs.update_client_spec_path_cache(files)\n \n self.gitStream.write(\"commit %s\\n\" % branch)\n-# gitStream.write(\"mark :%s\\n\" % details[\"change\"])\n+ self.gitStream.write(\"mark :%s\\n\" % details[\"change\"])\n self.committedChanges.add(int(details[\"change\"]))\n committer = \"\"\n if author not in self.users:\n@@ -2493,13 +2496,19 @@ class P4Sync(Command, P4UserMap):\n if change.has_key('change'):\n # find the corresponding git commit; take the oldest commit\n changelist = int(change['change'])\n- gitCommit = read_pipe([\"git\", \"rev-list\", \"--max-count=1\",\n- \"--reverse\", \":/\\[git-p4:.*change = %d\\]\" % changelist], ignore_error=True)\n- if len(gitCommit) == 0:\n- print \"importing label %s: could not find git commit for changelist %d\" % (name, changelist)\n- else:\n- gitCommit = gitCommit.strip()\n+ if changelist in self.committedChanges:\n+ gitCommit = \":%d\" % changelist # use a fast-import mark\n commitFound = True\n+ else:\n+ gitCommit = read_pipe([\"git\", \"rev-list\", \"--max-count=1\",\n+ \"--reverse\", \":/\\[git-p4:.*change = %d\\]\" % changelist], ignore_error=True)\n+ if len(gitCommit) == 0:\n+ print \"importing label %s: could not find git commit for changelist %d\" % (name, changelist)\n+ else:\n+ commitFound = True\n+ gitCommit = gitCommit.strip()\n+\n+ if commitFound:\n # Convert from p4 time format\n try:\n tmwhen = time.strptime(labelDetails['Update'], \"%Y/%m/%d %H:%M:%S\")\ndiff --git a/t/t9811-git-p4-label-import.sh b/t/t9811-git-p4-label-import.sh\nindex f7d5048..f994640 100755\n--- a/t/t9811-git-p4-label-import.sh\n+++ b/t/t9811-git-p4-label-import.sh\n@@ -222,7 +222,7 @@ p4_head_revision() {\n # has not been seen. The presence of a label on a commit\n # we haven't seen should not cause git-p4 to fail. It should\n # merely skip that label, and still import other labels.\n-test_expect_failure 'importing labels with missing revisions' '\n+test_expect_success 'importing labels with missing revisions' '\n \ttest_when_finished cleanup_git &&\n \t(\n \t\trm -fr \"$cli\" \"$git\" &&\n-- \n2.5.0.rc0\n"},{"id":"268790","messageId":"CAPig+cQUX5sD5Bh7bS6PaNeYiBw413vk2+TLTPzFi0AQwt+8nA@mail.gmail.com","threadId":"40193","inReplyTo":"1440659938-12952-2-git-send-email-luke@diamand.org","subject":"Re: [PATCHv1 1/3] git-p4: failing test for ignoring invalid p4 labels","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-08-27T12:53:53Z","receivedAt":"2015-08-27T12:53:53Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Aug 27, 2015 at 3:18 AM, Luke Diamand wrote:\n> When importing a label which references a commit that git-p4 does\n> not know about, git-p4 should skip it and go on to process other\n> labels that can be imported.\n>\n> Instead it crashes when attempting to find the missing commit in\n> the git history. This test demonstrates the problem.\n>\n> Signed-off-by: Luke Diamand \n> ---\n> diff --git a/t/t9811-git-p4-label-import.sh b/t/t9811-git-p4-label-import.sh\n> index 095238f..f7d5048 100755\n> --- a/t/t9811-git-p4-label-import.sh\n> +++ b/t/t9811-git-p4-label-import.sh\n> @@ -214,6 +214,51 @@ test_expect_success 'use git config to enable import/export of tags' '\n> )\n> '\n>\n> +p4_head_revision() {\n> + p4 changes -m 1 \"$@\" | awk '{print $2}'\n> +}\n> +\n> +# Importing a label that references a P4 commit that has\n> +# has not been seen. The presence of a label on a commit\n\ns/has has/has/\n\n> +# we haven't seen should not cause git-p4 to fail. It should\n> +# merely skip that label, and still import other labels.\n> +test_expect_failure 'importing labels with missing revisions' '\n"},{"id":"269907","messageId":"D21C3878.15603%marcus.holl@sap.com","threadId":"40193","inReplyTo":"1440659938-12952-1-git-send-email-luke@diamand.org","subject":"Re: [PATCHv1 0/3] git-p4: fixing import with labels when commit is not transferred","fromName":"Holl, Marcus","fromEmail":"marcus.holl@sap.com","sentAt":"2015-09-14T06:45:31Z","receivedAt":"2015-09-14T06:45:31Z","isPatch":false,"sender":{"key":"marcus.holl@sap.com","avatar":null},"body":"Hi Luke,\n\n>Marcus - if you're able to take a look at this and see if it\n>fixes your problem that would be very useful.\n\n\nFix works for me.\n\nThanks a lot for fixing this issue.\n\nBest regards,\nMarcus\n\nOn 27/08/15 09:18, \"Luke Diamand\" wrote:\n\n>Marcus Holl discovered that git-p4 fails to import labels sometimes.\n>\n>http://permalink.gmane.org/gmane.comp.version-control.git/273034\n>\n>It turns out there are at least two related problems.\n>\n>The first is that if git-p4 tries to import a P4 tag which references\n>a commit that we don't know about at all (e.g. because the P4\n>changelist was created before the point at which the repo was\n>cloned) then when it tries to find the commit, the git rev-list\n>command fails, and git-p4 terminates.\n>\n>This can be fixed by detecting the problem and ignoring that label.\n>\n>The second problem is that git-p4 can end up trying to find out\n>about a commit which is still in the git fast-import stream and not\n>yet processed. This is fixed by using fast-import \"marks\" to reference\n>these commits, which avoids having to query for commits which don't\n>yet exist. The normal path is used for commits that have not been\n>imported during this run of the program (and so won't have marks).\n>\n>A test case is added to demonstrate the problem.\n>\n>Marcus - if you're able to take a look at this and see if it\n>fixes your problem that would be very useful.\n>\n>Thanks,\n>Luke\n>\n>Luke Diamand (3):\n> git-p4: failing test for ignoring invalid p4 labels\n> git-p4: do not terminate creating tag for unknown commit\n> git-p4: fix P4 label import for unprocessed commits\n>\n> git-p4.py | 25 +++++++++++++++--------\n> t/t9811-git-p4-label-import.sh | 45\n>++++++++++++++++++++++++++++++++++++++++++\n> 2 files changed, 62 insertions(+), 8 deletions(-)\n>\n>-- \n>2.5.0.rc0\n"}]}