# [PATCHv1 2/3] git-p4: do not terminate creating tag for unknown commit 6 messages from 2015-08-27 to 2015-09-14. Participants: Luke Diamand, Eric Sunshine, Holl, Marcus. Thread: https://gitlist.dev/t/40193 ## Luke Diamand, 2015-08-27 07:18 Subject: [PATCHv1 0/3] git-p4: fixing import with labels when commit is not transferred Message-ID: <1440659938-12952-1-git-send-email-luke@diamand.org> URL: https://gitlist.dev/e/1440659938-12952-1-git-send-email-luke%40diamand.org ``` Marcus Holl discovered that git-p4 fails to import labels sometimes. http://permalink.gmane.org/gmane.comp.version-control.git/273034 It turns out there are at least two related problems. The first is that if git-p4 tries to import a P4 tag which references a commit that we don't know about at all (e.g. because the P4 changelist was created before the point at which the repo was cloned) then when it tries to find the commit, the git rev-list command fails, and git-p4 terminates. This can be fixed by detecting the problem and ignoring that label. The second problem is that git-p4 can end up trying to find out about a commit which is still in the git fast-import stream and not yet processed. This is fixed by using fast-import "marks" to reference these commits, which avoids having to query for commits which don't yet exist. The normal path is used for commits that have not been imported during this run of the program (and so won't have marks). A test case is added to demonstrate the problem. Marcus - if you're able to take a look at this and see if it fixes your problem that would be very useful. Thanks, Luke Luke Diamand (3): git-p4: failing test for ignoring invalid p4 labels git-p4: do not terminate creating tag for unknown commit git-p4: fix P4 label import for unprocessed commits git-p4.py | 25 +++++++++++++++-------- t/t9811-git-p4-label-import.sh | 45 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+), 8 deletions(-) -- 2.5.0.rc0 ``` ## Luke Diamand, 2015-08-27 07:18 Subject: [PATCHv1 1/3] git-p4: failing test for ignoring invalid p4 labels Message-ID: <1440659938-12952-2-git-send-email-luke@diamand.org> URL: https://gitlist.dev/e/1440659938-12952-2-git-send-email-luke%40diamand.org In-Reply-To: <1440659938-12952-1-git-send-email-luke@diamand.org> ``` When importing a label which references a commit that git-p4 does not know about, git-p4 should skip it and go on to process other labels that can be imported. Instead it crashes when attempting to find the missing commit in the git history. This test demonstrates the problem. Signed-off-by: Luke Diamand --- t/t9811-git-p4-label-import.sh | 45 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/t/t9811-git-p4-label-import.sh b/t/t9811-git-p4-label-import.sh index 095238f..f7d5048 100755 --- a/t/t9811-git-p4-label-import.sh +++ b/t/t9811-git-p4-label-import.sh @@ -214,6 +214,51 @@ test_expect_success 'use git config to enable import/export of tags' ' ) ' +p4_head_revision() { + p4 changes -m 1 "$@" | awk '{print $2}' +} + +# Importing a label that references a P4 commit that has +# has not been seen. The presence of a label on a commit +# we haven't seen should not cause git-p4 to fail. It should +# merely skip that label, and still import other labels. +test_expect_failure 'importing labels with missing revisions' ' + test_when_finished cleanup_git && + ( + rm -fr "$cli" "$git" && + mkdir "$cli" && + P4CLIENT=missing-revision && + client_view "//depot/missing-revision/... //missing-revision/..." && + cd "$cli" && + >f1 && p4 add f1 && p4 submit -d "start" && + + p4 tag -l TAG_S0 ... && + + >f2 && p4 add f2 && p4 submit -d "second" && + + startrev=$(p4_head_revision //depot/missing-revision/...) && + + >f3 && p4 add f3 && p4 submit -d "third" && + + p4 edit f2 && date >f2 && p4 submit -d "change" f2 && + + endrev=$(p4_head_revision //depot/missing-revision/...) && + + p4 tag -l TAG_S1 ... && + + # we should skip TAG_S0 since it is before our startpoint, + # but pick up TAG_S1. + + git p4 clone --dest="$git" --import-labels -v \ + //depot/missing-revision/...@$startrev,$endrev && + ( + cd "$git" && + git rev-parse TAG_S1 && + ! git rev-parse TAG_S0 + ) + ) +' + test_expect_success 'kill p4d' ' kill_p4d -- 2.5.0.rc0 ``` ## Luke Diamand, 2015-08-27 07:18 Subject: [PATCHv1 2/3] git-p4: do not terminate creating tag for unknown commit Message-ID: <1440659938-12952-3-git-send-email-luke@diamand.org> URL: https://gitlist.dev/e/1440659938-12952-3-git-send-email-luke%40diamand.org In-Reply-To: <1440659938-12952-1-git-send-email-luke@diamand.org> ``` If p4 reports a tag for a commit that git-p4 does not know about (e.g. because it references a P4 changelist that was imported prior to the point at which the repo was cloned into git), make sure that the error is correctly caught and handled. rather than just crashing. Signed-off-by: Luke Diamand --- git-p4.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/git-p4.py b/git-p4.py index 073f87b..a62611a 100755 --- a/git-p4.py +++ b/git-p4.py @@ -2494,9 +2494,9 @@ class P4Sync(Command, P4UserMap): # find the corresponding git commit; take the oldest commit changelist = int(change['change']) gitCommit = read_pipe(["git", "rev-list", "--max-count=1", - "--reverse", ":/\[git-p4:.*change = %d\]" % changelist]) + "--reverse", ":/\[git-p4:.*change = %d\]" % changelist], ignore_error=True) if len(gitCommit) == 0: - print "could not find git commit for changelist %d" % changelist + print "importing label %s: could not find git commit for changelist %d" % (name, changelist) else: gitCommit = gitCommit.strip() commitFound = True -- 2.5.0.rc0 ``` ## Luke Diamand, 2015-08-27 07:18 Subject: [PATCHv1 3/3] git-p4: fix P4 label import for unprocessed commits Message-ID: <1440659938-12952-4-git-send-email-luke@diamand.org> URL: https://gitlist.dev/e/1440659938-12952-4-git-send-email-luke%40diamand.org In-Reply-To: <1440659938-12952-1-git-send-email-luke@diamand.org> ``` With --detect-labels enabled, git-p4 will try to create tags using git fast-import by writing a "tag" clause to the fast-import stream. If the commit that the tag references has not yet actually been processed by fast-import, then the tag can't be created and git-p4 fails to import the P4 label. Teach git-p4 to use fast-import "marks" when creating tags which reference commits created during the current run of the program. Commits created before the current run are still referenced in the old way using a normal git commit. Signed-off-by: Luke Diamand --- git-p4.py | 25 +++++++++++++++++-------- t/t9811-git-p4-label-import.sh | 2 +- 2 files changed, 18 insertions(+), 9 deletions(-) diff --git a/git-p4.py b/git-p4.py index a62611a..2018011 100755 --- a/git-p4.py +++ b/git-p4.py @@ -2322,8 +2322,11 @@ class P4Sync(Command, P4UserMap): else: return "%s " % userid - # Stream a p4 tag def streamTag(self, gitStream, labelName, labelDetails, commit, epoch): + """ Stream a p4 tag. + commit is either a git commit, or a fast-import mark, ":" + """ + if verbose: print "writing tag %s for commit %s" % (labelName, commit) gitStream.write("tag %s\n" % labelName) @@ -2374,7 +2377,7 @@ class P4Sync(Command, P4UserMap): self.clientSpecDirs.update_client_spec_path_cache(files) self.gitStream.write("commit %s\n" % branch) -# gitStream.write("mark :%s\n" % details["change"]) + self.gitStream.write("mark :%s\n" % details["change"]) self.committedChanges.add(int(details["change"])) committer = "" if author not in self.users: @@ -2493,13 +2496,19 @@ class P4Sync(Command, P4UserMap): if change.has_key('change'): # find the corresponding git commit; take the oldest commit changelist = int(change['change']) - gitCommit = read_pipe(["git", "rev-list", "--max-count=1", - "--reverse", ":/\[git-p4:.*change = %d\]" % changelist], ignore_error=True) - if len(gitCommit) == 0: - print "importing label %s: could not find git commit for changelist %d" % (name, changelist) - else: - gitCommit = gitCommit.strip() + if changelist in self.committedChanges: + gitCommit = ":%d" % changelist # use a fast-import mark commitFound = True + else: + gitCommit = read_pipe(["git", "rev-list", "--max-count=1", + "--reverse", ":/\[git-p4:.*change = %d\]" % changelist], ignore_error=True) + if len(gitCommit) == 0: + print "importing label %s: could not find git commit for changelist %d" % (name, changelist) + else: + commitFound = True + gitCommit = gitCommit.strip() + + if commitFound: # Convert from p4 time format try: tmwhen = time.strptime(labelDetails['Update'], "%Y/%m/%d %H:%M:%S") diff --git a/t/t9811-git-p4-label-import.sh b/t/t9811-git-p4-label-import.sh index f7d5048..f994640 100755 --- a/t/t9811-git-p4-label-import.sh +++ b/t/t9811-git-p4-label-import.sh @@ -222,7 +222,7 @@ p4_head_revision() { # has not been seen. The presence of a label on a commit # we haven't seen should not cause git-p4 to fail. It should # merely skip that label, and still import other labels. -test_expect_failure 'importing labels with missing revisions' ' +test_expect_success 'importing labels with missing revisions' ' test_when_finished cleanup_git && ( rm -fr "$cli" "$git" && -- 2.5.0.rc0 ``` ## Eric Sunshine, 2015-08-27 12:53 Subject: Re: [PATCHv1 1/3] git-p4: failing test for ignoring invalid p4 labels Message-ID: URL: https://gitlist.dev/e/CAPig%2BcQUX5sD5Bh7bS6PaNeYiBw413vk2%2BTLTPzFi0AQwt%2B8nA%40mail.gmail.com In-Reply-To: <1440659938-12952-2-git-send-email-luke@diamand.org> ``` On Thu, Aug 27, 2015 at 3:18 AM, Luke Diamand wrote: > When importing a label which references a commit that git-p4 does > not know about, git-p4 should skip it and go on to process other > labels that can be imported. > > Instead it crashes when attempting to find the missing commit in > the git history. This test demonstrates the problem. > > Signed-off-by: Luke Diamand > --- > diff --git a/t/t9811-git-p4-label-import.sh b/t/t9811-git-p4-label-import.sh > index 095238f..f7d5048 100755 > --- a/t/t9811-git-p4-label-import.sh > +++ b/t/t9811-git-p4-label-import.sh > @@ -214,6 +214,51 @@ test_expect_success 'use git config to enable import/export of tags' ' > ) > ' > > +p4_head_revision() { > + p4 changes -m 1 "$@" | awk '{print $2}' > +} > + > +# Importing a label that references a P4 commit that has > +# has not been seen. The presence of a label on a commit s/has has/has/ > +# we haven't seen should not cause git-p4 to fail. It should > +# merely skip that label, and still import other labels. > +test_expect_failure 'importing labels with missing revisions' ' ``` ## Holl, Marcus, 2015-09-14 06:45 Subject: Re: [PATCHv1 0/3] git-p4: fixing import with labels when commit is not transferred Message-ID: URL: https://gitlist.dev/e/D21C3878.15603%25marcus.holl%40sap.com In-Reply-To: <1440659938-12952-1-git-send-email-luke@diamand.org> ``` Hi Luke, >Marcus - if you're able to take a look at this and see if it >fixes your problem that would be very useful. Fix works for me. Thanks a lot for fixing this issue. Best regards, Marcus On 27/08/15 09:18, "Luke Diamand" wrote: >Marcus Holl discovered that git-p4 fails to import labels sometimes. > >http://permalink.gmane.org/gmane.comp.version-control.git/273034 > >It turns out there are at least two related problems. > >The first is that if git-p4 tries to import a P4 tag which references >a commit that we don't know about at all (e.g. because the P4 >changelist was created before the point at which the repo was >cloned) then when it tries to find the commit, the git rev-list >command fails, and git-p4 terminates. > >This can be fixed by detecting the problem and ignoring that label. > >The second problem is that git-p4 can end up trying to find out >about a commit which is still in the git fast-import stream and not >yet processed. This is fixed by using fast-import "marks" to reference >these commits, which avoids having to query for commits which don't >yet exist. The normal path is used for commits that have not been >imported during this run of the program (and so won't have marks). > >A test case is added to demonstrate the problem. > >Marcus - if you're able to take a look at this and see if it >fixes your problem that would be very useful. > >Thanks, >Luke > >Luke Diamand (3): > git-p4: failing test for ignoring invalid p4 labels > git-p4: do not terminate creating tag for unknown commit > git-p4: fix P4 label import for unprocessed commits > > git-p4.py | 25 +++++++++++++++-------- > t/t9811-git-p4-label-import.sh | 45 >++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 62 insertions(+), 8 deletions(-) > >-- >2.5.0.rc0 ```