# [PATCH] cvsimport: strip question marks from tags 11 messages from 2012-09-05 to 2012-09-06. Participants: Ken Dreyer, Junio C Hamano, Alex Vandiver, Andreas Schwab. Thread: https://gitlist.dev/t/31440 ## Ken Dreyer, 2012-09-05 02:53 Subject: [PATCH] cvsimport: strip question marks from tags Message-ID: <1346813618-20279-1-git-send-email-ktdreyer@ktdreyer.com> URL: https://gitlist.dev/e/1346813618-20279-1-git-send-email-ktdreyer%40ktdreyer.com ``` The "?" character can be present in a CVS tag name, but git's bad_ref_char does not allow question marks in git tags. If git-cvsimport encounters a CVS tag with a question mark, it will error and refuse to continue the import beyond that point. When importing CVS tags, strip "?" characters from the tag names as we translate them to git tag names. Signed-off-by: Ken Dreyer --- git-cvsimport.perl | 1 + 1 file changed, 1 insertion(+) diff --git a/git-cvsimport.perl b/git-cvsimport.perl index 8d41610..36f59fe 100755 --- a/git-cvsimport.perl +++ b/git-cvsimport.perl @@ -890,6 +890,7 @@ sub commit { $xtag =~ tr/_/\./ if ( $opt_u ); $xtag =~ s/[\/]/$opt_s/g; $xtag =~ s/\[//g; + $xtag =~ s/\?//g; system('git' , 'tag', '-f', $xtag, $cid) == 0 or die "Cannot create tag $xtag: $!\n"; -- 1.7.11.4 ``` ## Junio C Hamano, 2012-09-05 03:19 Subject: Re: [PATCH] cvsimport: strip question marks from tags Message-ID: <7vsjax6trn.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vsjax6trn.fsf%40alter.siamese.dyndns.org In-Reply-To: <1346813618-20279-1-git-send-email-ktdreyer@ktdreyer.com> ``` Ken Dreyer writes: > The "?" character can be present in a CVS tag name, but git's > bad_ref_char does not allow question marks in git tags. If > git-cvsimport encounters a CVS tag with a question mark, it will error > and refuse to continue the import beyond that point. > > When importing CVS tags, strip "?" characters from the tag names as we > translate them to git tag names. > > Signed-off-by: Ken Dreyer > --- > git-cvsimport.perl | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/git-cvsimport.perl b/git-cvsimport.perl > index 8d41610..36f59fe 100755 > --- a/git-cvsimport.perl > +++ b/git-cvsimport.perl > @@ -890,6 +890,7 @@ sub commit { > $xtag =~ tr/_/\./ if ( $opt_u ); > $xtag =~ s/[\/]/$opt_s/g; > $xtag =~ s/\[//g; > + $xtag =~ s/\?//g; I do not think this is a right and sustainable approach. The next patch would probably be to strip "~" and then another patch that strips "^", and yet another that squashes ".." into one would surely follow. How about extending the s/\[//g we can see in the context to cover everything that are unacceptable (see refs.c:bad_ref_char()) once and for all? The result needs to be further massaged to avoid component that has two or more dots in a row, a dot at the beginning or at the end (see the comment at the beginning of refs.c, and also refs.c:check_refname_component()). > system('git' , 'tag', '-f', $xtag, $cid) == 0 > or die "Cannot create tag $xtag: $!\n"; ``` ## Ken Dreyer, 2012-09-05 04:26 Subject: [PATCH] cvsimport: strip all inappropriate tag strings Message-ID: <1346819164-8116-1-git-send-email-ktdreyer@ktdreyer.com> URL: https://gitlist.dev/e/1346819164-8116-1-git-send-email-ktdreyer%40ktdreyer.com In-Reply-To: <7vsjax6trn.fsf@alter.siamese.dyndns.org> ``` Certain characters such as "?" can be present in a CVS tag name, but git does not allow these characters in tags. If git-cvsimport encounters a CVS tag that git cannot handle, cvsimport will error and refuse to continue the import beyond that point. When importing CVS tags, strip all the inappropriate strings from the tag names as we translate them to git tag names. Signed-off-by: Ken Dreyer --- Thank you Junio for the review. I've taken your suggestion and amended my patch to eliminate all the bad strings in ref.c. git-cvsimport.perl | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/git-cvsimport.perl b/git-cvsimport.perl index 8d41610..0dc598d 100755 --- a/git-cvsimport.perl +++ b/git-cvsimport.perl @@ -889,7 +889,25 @@ sub commit { $xtag =~ s/\s+\*\*.*$//; # Remove stuff like ** INVALID ** and ** FUNKY ** $xtag =~ tr/_/\./ if ( $opt_u ); $xtag =~ s/[\/]/$opt_s/g; - $xtag =~ s/\[//g; + + # See ref.c for these rules. + # Tag cannot end with a '/' - this is already handled above. + # Tag cannot contain bad chars. See bad_ref_char in ref.c. + $xtag =~ s/[ ~\^:\\\*\?\[]//g; + # Tag cannot contain '..'. + $xtag =~ s/\.\.//g; + # Tag cannot contain '@{'. + $xtag =~ s/\@{//g; + # Tag cannot end with '.lock'. + $xtag =~ s/(?:\.lock)+$//; + # Tag cannot begin or end with '.'. + $xtag =~ s/^\.+//; + $xtag =~ s/\.+$//; + # Tag cannot consist of a single '.' - already handled above. + # Tag cannot be empty. + if ($xtag eq '') { + return; + } system('git' , 'tag', '-f', $xtag, $cid) == 0 or die "Cannot create tag $xtag: $!\n"; -- 1.7.11.4 ``` ## Junio C Hamano, 2012-09-05 05:52 Subject: Re: [PATCH] cvsimport: strip all inappropriate tag strings Message-ID: <7vobll6moz.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vobll6moz.fsf%40alter.siamese.dyndns.org In-Reply-To: <1346819164-8116-1-git-send-email-ktdreyer@ktdreyer.com> ``` Ken Dreyer writes: > Certain characters such as "?" can be present in a CVS tag name, but > git does not allow these characters in tags. If git-cvsimport > encounters a CVS tag that git cannot handle, cvsimport will error and > refuse to continue the import beyond that point. > > When importing CVS tags, strip all the inappropriate strings from the > tag names as we translate them to git tag names. > > Signed-off-by: Ken Dreyer > --- Thanks, will queue. I think we also forbid tagnames (or branchnames for that matter) that begin with a dash on the creation side, even though the reading side tries to be lenient (i.e. if for some bad tool already created a file .git/refs/tags/-foobar, we allow "git show tags/-foobar" to show it). The routines in refs.c enforces primarily on the reading codepath. So this part: > + # Tag cannot begin or end with '.'. > + $xtag =~ s/^\.+//; > + $xtag =~ s/\.+$//; may need to become # Tag cannot begin with '.' or '-', or end with '.'. $xtag =~ s/^[-.]+//; $xtag =~ s/\.+$//; or something. > git-cvsimport.perl | 20 +++++++++++++++++++- > 1 file changed, 19 insertions(+), 1 deletion(-) > > diff --git a/git-cvsimport.perl b/git-cvsimport.perl > index 8d41610..0dc598d 100755 > --- a/git-cvsimport.perl > +++ b/git-cvsimport.perl > @@ -889,7 +889,25 @@ sub commit { > $xtag =~ s/\s+\*\*.*$//; # Remove stuff like ** INVALID ** and ** FUNKY ** > $xtag =~ tr/_/\./ if ( $opt_u ); > $xtag =~ s/[\/]/$opt_s/g; > - $xtag =~ s/\[//g; > + > + # See ref.c for these rules. > + # Tag cannot end with a '/' - this is already handled above. > + # Tag cannot contain bad chars. See bad_ref_char in ref.c. > + $xtag =~ s/[ ~\^:\\\*\?\[]//g; > + # Tag cannot contain '..'. > + $xtag =~ s/\.\.//g; > + # Tag cannot contain '@{'. > + $xtag =~ s/\@{//g; > + # Tag cannot end with '.lock'. > + $xtag =~ s/(?:\.lock)+$//; > + # Tag cannot begin or end with '.'. > + $xtag =~ s/^\.+//; > + $xtag =~ s/\.+$//; > + # Tag cannot consist of a single '.' - already handled above. > + # Tag cannot be empty. > + if ($xtag eq '') { > + return; > + } > > system('git' , 'tag', '-f', $xtag, $cid) == 0 > or die "Cannot create tag $xtag: $!\n"; ``` ## Alex Vandiver, 2012-09-05 06:44 Subject: Re: [PATCH] cvsimport: strip all inappropriate tag strings Message-ID: <1346827469.1137.22.camel@umgah.localdomain> URL: https://gitlist.dev/e/1346827469.1137.22.camel%40umgah.localdomain In-Reply-To: <1346819164-8116-1-git-send-email-ktdreyer@ktdreyer.com> ``` On Tue, 2012-09-04 at 22:26 -0600, Ken Dreyer wrote: > When importing CVS tags, strip all the inappropriate strings from the > tag names as we translate them to git tag names. > > [snip] > diff --git a/git-cvsimport.perl b/git-cvsimport.perl > index 8d41610..0dc598d 100755 > --- a/git-cvsimport.perl > +++ b/git-cvsimport.perl > @@ -889,7 +889,25 @@ sub commit { > $xtag =~ s/\s+\*\*.*$//; # Remove stuff like ** INVALID ** and ** FUNKY ** > $xtag =~ tr/_/\./ if ( $opt_u ); > $xtag =~ s/[\/]/$opt_s/g; > - $xtag =~ s/\[//g; > + > + # See ref.c for these rules. > + # Tag cannot end with a '/' - this is already handled above. > + # Tag cannot contain bad chars. See bad_ref_char in ref.c. > + $xtag =~ s/[ ~\^:\\\*\?\[]//g; > + # Tag cannot contain '..'. > + $xtag =~ s/\.\.//g; > + # Tag cannot contain '@{'. > + $xtag =~ s/\@{//g; > + # Tag cannot end with '.lock'. > + $xtag =~ s/(?:\.lock)+$//; > + # Tag cannot begin or end with '.'. > + $xtag =~ s/^\.+//; > + $xtag =~ s/\.+$//; > + # Tag cannot consist of a single '.' - already handled above. > + # Tag cannot be empty. > + if ($xtag eq '') { > + return; > + } Unfortunately, this isn't quite sufficient. Consider the case of a tag named "foo.lock." The .lock rule doesn't match, because it's not at the end of the string -- but after s/\.+$// runs, it _is_ at the end, and hence invalid. A similar problem exists with a tag named "a.@{.b", given the ordering of @{ and .. removal. Something like the following would suffice: 1 while $xtag =~ s/ (?: \.\. # Tag cannot contain '..'. | \@{ # Tag cannot contain '@{'. | \.lock $ # Tag cannot end with '.lock'. | ^ \. # Tag cannot begin... | \. $ # ...or end with '.' )//xg; - Alex ``` ## Ken Dreyer, 2012-09-05 21:39 Subject: [PATCH] cvsimport: strip all inappropriate tag strings Message-ID: <1346881194-19076-1-git-send-email-ktdreyer@ktdreyer.com> URL: https://gitlist.dev/e/1346881194-19076-1-git-send-email-ktdreyer%40ktdreyer.com In-Reply-To: <1346827469.1137.22.camel@umgah.localdomain> ``` Certain characters such as "?" can be present in a CVS tag name, but git does not allow these characters in tags. If git-cvsimport encounters a CVS tag that git cannot handle, cvsimport will error and refuse to continue the import beyond that point. When importing CVS tags, strip all the inappropriate strings from the tag names as we translate them to git tag names. Signed-off-by: Ken Dreyer --- Thanks Junio and Alex for your review and comments. I've implemented both of your suggestions in this patch. git-cvsimport.perl | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/git-cvsimport.perl b/git-cvsimport.perl index 8d41610..dda8a6d 100755 --- a/git-cvsimport.perl +++ b/git-cvsimport.perl @@ -889,7 +889,23 @@ sub commit { $xtag =~ s/\s+\*\*.*$//; # Remove stuff like ** INVALID ** and ** FUNKY ** $xtag =~ tr/_/\./ if ( $opt_u ); $xtag =~ s/[\/]/$opt_s/g; - $xtag =~ s/\[//g; + + # See ref.c for these rules. + # Tag cannot contain bad chars. See bad_ref_char in ref.c. + $xtag =~ s/[ ~\^:\\\*\?\[]//g; + # Other bad strings for tags: + 1 while $xtag =~ s/ + (?: \.\. # Tag cannot contain '..'. + | \@{ # Tag cannot contain '@{'. + | ^ - # Tag cannot begin with '-'. + | \.lock $ # Tag cannot end with '.lock'. + | ^ \. # Tag cannot begin... + | \. $ # ...or end with '.' + )//xg; + # Tag cannot be empty. + if ($xtag eq '') { + return; + } system('git' , 'tag', '-f', $xtag, $cid) == 0 or die "Cannot create tag $xtag: $!\n"; -- 1.7.11.4 ``` ## Junio C Hamano, 2012-09-06 03:52 Subject: Re: [PATCH] cvsimport: strip all inappropriate tag strings Message-ID: <7vipbr6c4q.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vipbr6c4q.fsf%40alter.siamese.dyndns.org In-Reply-To: <1346881194-19076-1-git-send-email-ktdreyer@ktdreyer.com> ``` Ken Dreyer writes: > Certain characters such as "?" can be present in a CVS tag name, but > git does not allow these characters in tags. If git-cvsimport > encounters a CVS tag that git cannot handle, cvsimport will error and > refuse to continue the import beyond that point. > > When importing CVS tags, strip all the inappropriate strings from the > tag names as we translate them to git tag names. > > Signed-off-by: Ken Dreyer > --- > > Thanks Junio and Alex for your review and comments. I've implemented > both of your suggestions in this patch. Thanks. Do we want to give a warning instead of silently dropping a tag on the floor, or is the output verbose enough that such a warning will be drowned in the noise? > git-cvsimport.perl | 18 +++++++++++++++++- > 1 file changed, 17 insertions(+), 1 deletion(-) > > diff --git a/git-cvsimport.perl b/git-cvsimport.perl > index 8d41610..dda8a6d 100755 > --- a/git-cvsimport.perl > +++ b/git-cvsimport.perl > @@ -889,7 +889,23 @@ sub commit { > $xtag =~ s/\s+\*\*.*$//; # Remove stuff like ** INVALID ** and ** FUNKY ** > $xtag =~ tr/_/\./ if ( $opt_u ); > $xtag =~ s/[\/]/$opt_s/g; > - $xtag =~ s/\[//g; > + > + # See ref.c for these rules. > + # Tag cannot contain bad chars. See bad_ref_char in ref.c. > + $xtag =~ s/[ ~\^:\\\*\?\[]//g; > + # Other bad strings for tags: > + 1 while $xtag =~ s/ > + (?: \.\. # Tag cannot contain '..'. > + | \@{ # Tag cannot contain '@{'. > + | ^ - # Tag cannot begin with '-'. > + | \.lock $ # Tag cannot end with '.lock'. > + | ^ \. # Tag cannot begin... > + | \. $ # ...or end with '.' > + )//xg; > + # Tag cannot be empty. > + if ($xtag eq '') { That is, adding something like: print STDERR "warning: ignoring tag '$tag' with invalid tagname"; here. > + return; > + } > > system('git' , 'tag', '-f', $xtag, $cid) == 0 > or die "Cannot create tag $xtag: $!\n"; It also may be worthwhile to show the original tagname ($tag) somewhere in this message to help diagnosis. ``` ## Ken Dreyer, 2012-09-06 05:42 Subject: [PATCH] cvsimport: strip all inappropriate tag strings Message-ID: <1346910178-13902-1-git-send-email-ktdreyer@ktdreyer.com> URL: https://gitlist.dev/e/1346910178-13902-1-git-send-email-ktdreyer%40ktdreyer.com In-Reply-To: <7vipbr6c4q.fsf@alter.siamese.dyndns.org> ``` Certain characters such as "?" can be present in a CVS tag name, but git does not allow these characters in tags. If git-cvsimport encounters a CVS tag that git cannot handle, cvsimport will error and refuse to continue the import beyond that point. When importing CVS tags, strip all the inappropriate strings from the tag names as we translate them to git tag names. Provide more debugging information to the user if we've altered the tag and the "git tag" command still fails. Also, warn the user if we end up skipping an (unusable) tag altogether. Signed-off-by: Ken Dreyer --- Thanks Junio for your suggestion about diagnosis messages. I've implemented your suggestion by adding a warning statement if we skip a tag altogether, and I also added some output if we've translated a tag and the system() call still fails. git-cvsimport.perl | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/git-cvsimport.perl b/git-cvsimport.perl index 8d41610..3a30754 100755 --- a/git-cvsimport.perl +++ b/git-cvsimport.perl @@ -889,10 +889,36 @@ sub commit { $xtag =~ s/\s+\*\*.*$//; # Remove stuff like ** INVALID ** and ** FUNKY ** $xtag =~ tr/_/\./ if ( $opt_u ); $xtag =~ s/[\/]/$opt_s/g; - $xtag =~ s/\[//g; - system('git' , 'tag', '-f', $xtag, $cid) == 0 - or die "Cannot create tag $xtag: $!\n"; + # See ref.c for these rules. + # Tag cannot contain bad chars. See bad_ref_char in ref.c. + $xtag =~ s/[ ~\^:\\\*\?\[]//g; + # Other bad strings for tags: + 1 while $xtag =~ s/ + (?: \.\. # Tag cannot contain '..'. + | \@{ # Tag cannot contain '@{'. + | ^ - # Tag cannot begin with '-'. + | \.lock $ # Tag cannot end with '.lock'. + | ^ \. # Tag cannot begin... + | \. $ # ...or end with '.' + )//xg; + # Tag cannot be empty. + if ($xtag eq '') { + warn("warning: ignoring tag '$tag'", + " with invalid tagname\n"); + return; + } + + if (system('git' , 'tag', '-f', $xtag, $cid) != 0) { + # We did our best to sanitize the tag, but still failed + # for whatever reason. Bail out, and give the user + # enough information to understand if/how we should + # improve the translation in the future. + if ($tag ne $xtag) { + print "Translated '$tag' tag to '$xtag'\n"; + } + die "Cannot create tag $xtag: $!\n"; + } print "Created tag '$xtag' on '$branch'\n" if $opt_v; } -- 1.7.11.4 ``` ## Andreas Schwab, 2012-09-06 09:02 Subject: Re: [PATCH] cvsimport: strip all inappropriate tag strings Message-ID: URL: https://gitlist.dev/e/m2y5knjzh5.fsf%40igel.home In-Reply-To: <1346910178-13902-1-git-send-email-ktdreyer@ktdreyer.com> ``` Ken Dreyer writes: > + # See ref.c for these rules. > + # Tag cannot contain bad chars. See bad_ref_char in ref.c. s/ref.c/refs.c/ Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ``` ## Ken Dreyer, 2012-09-06 16:36 Subject: [PATCH] cvsimport: strip all inappropriate tag strings Message-ID: <1346949413-17287-1-git-send-email-ktdreyer@ktdreyer.com> URL: https://gitlist.dev/e/1346949413-17287-1-git-send-email-ktdreyer%40ktdreyer.com In-Reply-To: ``` Certain characters such as "?" can be present in a CVS tag name, but git does not allow these characters in tags. If git-cvsimport encounters a CVS tag that git cannot handle, cvsimport will error and refuse to continue the import beyond that point. When importing CVS tags, strip all the inappropriate strings from the tag names as we translate them to git tag names. Provide more debugging information to the user if we've altered the tag and the "git tag" command still fails. Also, warn the user if we end up skipping an (unusable) tag altogether. Signed-off-by: Ken Dreyer --- Thanks Andreas for catching that "ref.c" in the comments ought to be "refs.c". I've corrected that in this latest version of the patch. git-cvsimport.perl | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/git-cvsimport.perl b/git-cvsimport.perl index 8d41610..8032f23 100755 --- a/git-cvsimport.perl +++ b/git-cvsimport.perl @@ -889,10 +889,37 @@ sub commit { $xtag =~ s/\s+\*\*.*$//; # Remove stuff like ** INVALID ** and ** FUNKY ** $xtag =~ tr/_/\./ if ( $opt_u ); $xtag =~ s/[\/]/$opt_s/g; - $xtag =~ s/\[//g; - system('git' , 'tag', '-f', $xtag, $cid) == 0 - or die "Cannot create tag $xtag: $!\n"; + # See refs.c for these rules. + # Tag cannot contain bad chars. (See bad_ref_char in refs.c.) + $xtag =~ s/[ ~\^:\\\*\?\[]//g; + # Other bad strings for tags: + # (See check_refname_component in refs.c.) + 1 while $xtag =~ s/ + (?: \.\. # Tag cannot contain '..'. + | \@{ # Tag cannot contain '@{'. + | ^ - # Tag cannot begin with '-'. + | \.lock $ # Tag cannot end with '.lock'. + | ^ \. # Tag cannot begin... + | \. $ # ...or end with '.' + )//xg; + # Tag cannot be empty. + if ($xtag eq '') { + warn("warning: ignoring tag '$tag'", + " with invalid tagname\n"); + return; + } + + if (system('git' , 'tag', '-f', $xtag, $cid) != 0) { + # We did our best to sanitize the tag, but still failed + # for whatever reason. Bail out, and give the user + # enough information to understand if/how we should + # improve the translation in the future. + if ($tag ne $xtag) { + print "Translated '$tag' tag to '$xtag'\n"; + } + die "Cannot create tag $xtag: $!\n"; + } print "Created tag '$xtag' on '$branch'\n" if $opt_v; } -- 1.7.11.4 ``` ## Junio C Hamano, 2012-09-06 17:41 Subject: Re: [PATCH] cvsimport: strip all inappropriate tag strings Message-ID: <7v627r3v6l.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7v627r3v6l.fsf%40alter.siamese.dyndns.org In-Reply-To: <1346949413-17287-1-git-send-email-ktdreyer@ktdreyer.com> ``` Ken Dreyer writes: > Thanks Andreas for catching that "ref.c" in the comments ought to be > "refs.c". I've corrected that in this latest version of the patch. Yeah, thanks, all. Will queue. ```