threads / discuss / 40184

git clone svn: authors from authors file are ignored, authors-prog works, but crashes on branch points

Subject: git clone svn: authors from authors file are ignored, authors-prog works, but crashes on branch points

## tl;dr

7 messages between Aug 26, 2015 and Sep 11, 2015.

replies: 6people: 4as markdown or json

Till Schäfer· Aug 26, 2015, 19:57 UTC · lore

Hi, i am observing some weired "git svn clone" behavior during my try to migrate the Scaffold Hunter [1] SVN repository [2] to Git: 

if i just use the command 
$ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --no-metadata -s code
everything went smoothly. Now i wanted to replace the authors SVN-logins by useful names and created a mapping file with the following content:

anjenson = Andrew Zhilka <> bernhard.dick = Bernhard Dick <> dominic.sacre = Dominic Sacré <> doxmoxbox = doxmoxbox <> falkn = Falk Nette <> henning.garus = Henning Garus <> kakl = Karsten Klein <> klein = Karsten Klein <> lappie00 = Jeroen Lappenschaar <> michael.hesse = Michael Hesse <> nlskrg = Nils Kriege <nlskrg@gmail.com> philipp.kopp = Philipp Kopp <> philipp.lewe = Philipp Lewe <> schrins = Sven Schrinner <> shamshadnpti = Shamshad Alam <> srenner7 = Steffen Renner <> sturm89 = Werner Sturm <> thomas.schmitz = Thomas Schmitz <> thorsten.fluegel = Thorsten Flügel <> till.schaefer = Till Schäfer <till2.schaefer@tu-dortmund.de> tillschaefer = Till Schäfer <till2.schaefer@tu-dortmund.de>

I executed:
$ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --authors-file=/home/till/temp/code-authors-transform.txt --no-metadata -s code

and got the error message "Author: klein not defined in /home/till/temp/code-authors-transform.txt file". I tried a workaround by using a python script to return the entries i have already defined in the authors file: 

$ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --authors-file=/home/till/temp/code-authors-transform.txt --authors-prog=/home/till/temp/authors.py --no-metadata -s code
The import process now went over the "klein" commit. The weired thing is, that a few SVN usernames seem to be recognized in the authors file and a few are passed to my script (I logged the output). However, at the first revision, where a tag was added in the SVN repo the cloning process crashed with the error message:

Found possible branch point: svn://svn.code.sf.net/p/scaffoldhunter/code/trunk => svn://svn.code.sf.net/p/scaffoldhunter/code/branches/subsearch, 17 Use of uninitialized value $u in substitution (s///) at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101. Use of uninitialized value $u in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101. refs/remotes/origin/trunk: 'svn://svn.code.sf.net/p/scaffoldhunter/code' not found in ''

One problem I am seeing in the SVN repo is, that between the revisions 97 and 102 the trunk folder was absent. the old one was moved to some subfolder and a new one was created. Therefore, I started the cloning beginning with rev 102 using the command line option "-r102:HEAD". However, the same error occurred for some later tag (the first tag after rev 102).

Found possible branch point: svn://svn.code.sf.net/p/scaffoldhunter/code/trunk => svn://svn.code.sf.net/p/scaffoldhunter/code/tags/release-2.0, 1565 Use of uninitialized value $u in substitution (s///) at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101. Use of uninitialized value $u in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101. refs/remotes/origin/trunk: 'svn://svn.code.sf.net/p/scaffoldhunter/code' not found in ''

I would be very glad if someone has a hint about what is going wrong here. Is this a bug in git or is something wrong with the SVN repo?
used software versions: 
- git 2.5.0 
- subversion 1.8.14

Regards, Till

[1] http://scaffoldhunter.sourceforge.net/ [2] https://svn.code.sf.net/p/scaffoldhunter/code/trunk

-- 
Dipl.-Inf. Till Schäfer
TU Dortmund University
Chair 11 - Algorithm Engineering
Otto-Hahn-Str. 14 / Room 237
44227 Dortmund, Germany

e-mail: till.schaefer@cs.tu-dortmund.de
phone: +49(231)755-7706
fax: +49(231)755-7740
web: http://ls11-www.cs.uni-dortmund.de/staff/schaefer
pgp: https://keyserver2.pgp.com/vkd/SubmitSearch.event?&&SearchCriteria=0xD84DED79
Michael J Gruber· Sep 10, 2015, 12:32 UTC · re: Till Schäfer · lore

Re: git clone svn: authors from authors file are ignored, authors-prog works, but crashes on branch points

Till Schäfer venit, vidit, dixit 26.08.2015 21:57:
Show 6 quoted lines
> Hi,
> i am observing some weired "git svn clone" behavior during my try to migrate the Scaffold Hunter [1] SVN repository [2] to Git: 
> 
> if i just use the command 
> 
> $ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --no-metadata -s code

Is there a specific reason for "--no-metadata"? The man page advises against it. Basically, it keeps you from converting the history in chunks. (You can always filter-branch everything after a complete, "one shot" conversion which may be done in stages.)

Show 35 quoted lines
> everything went smoothly. Now i wanted to replace the authors SVN-logins by useful names and created a mapping file with the following content: 
> 
> anjenson = Andrew Zhilka <>
> bernhard.dick = Bernhard Dick <>
> dominic.sacre = Dominic Sacré <>
> doxmoxbox = doxmoxbox <>
> falkn = Falk Nette <>
> henning.garus = Henning Garus <>
> kakl = Karsten Klein <>
> klein = Karsten Klein <>
> lappie00 = Jeroen Lappenschaar <>
> michael.hesse = Michael Hesse <>
> nlskrg = Nils Kriege <nlskrg@gmail.com>
> philipp.kopp = Philipp Kopp <>
> philipp.lewe = Philipp Lewe <>
> schrins = Sven Schrinner <>
> shamshadnpti = Shamshad Alam <>
> srenner7 = Steffen Renner <>
> sturm89 = Werner Sturm <>
> thomas.schmitz = Thomas Schmitz <>
> thorsten.fluegel = Thorsten Flügel <>
> till.schaefer = Till Schäfer <till2.schaefer@tu-dortmund.de>
> tillschaefer = Till Schäfer <till2.schaefer@tu-dortmund.de>
> 
> 
> I executed:
> 
> $ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --authors-file=/home/till/temp/code-authors-transform.txt --no-metadata -s code
> 
> and got the error message "Author: klein not defined in /home/till/temp/code-authors-transform.txt file". 
> I tried a workaround by using a python script to return the entries i have already defined in the authors file: 
> 
> $ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --authors-file=/home/till/temp/code-authors-transform.txt --authors-prog=/home/till/temp/authors.py --no-metadata -s code
> 
> The import process now went over the "klein" commit. The weired thing is, that a few SVN usernames seem to be recognized in the authors file and a few are passed to my script (I logged the output). However, at the first revision, where a tag was added in the SVN repo the cloning process crashed with the error message:

That first part is weird already. The reason is that author-file processing and author-prog output processing use different perl regexps. I will follow up with a patch.

Show 13 quoted lines
> Found possible branch point: svn://svn.code.sf.net/p/scaffoldhunter/code/trunk => svn://svn.code.sf.net/p/scaffoldhunter/code/branches/subsearch, 17
> Use of uninitialized value $u in substitution (s///) at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> Use of uninitialized value $u in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> refs/remotes/origin/trunk: 'svn://svn.code.sf.net/p/scaffoldhunter/code' not found in ''
> 
> One problem I am seeing in the SVN repo is, that between the revisions 97 and 102 the trunk folder was absent. the old one was moved to some subfolder and a new one was created. Therefore, I started the cloning beginning with rev 102 using the command line option "-r102:HEAD". However, the same error occurred for some later tag (the first tag after rev 102). 
> 
> Found possible branch point: svn://svn.code.sf.net/p/scaffoldhunter/code/trunk => svn://svn.code.sf.net/p/scaffoldhunter/code/tags/release-2.0, 1565
> Use of uninitialized value $u in substitution (s///) at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> Use of uninitialized value $u in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> refs/remotes/origin/trunk: 'svn://svn.code.sf.net/p/scaffoldhunter/code' not found in ''
> 
> I would be very glad if someone has a hint about what is going wrong here. Is this a bug in git or is something wrong with the SVN repo?

The layout change (no root) certainly doesn't help. We shouldn't crash, though.

> used software versions: 
> - git 2.5.0 
> - subversion 1.8.14

The conversion goes through for me with (your other options and) your authors file and specifying a stupid authors prog:

--->8--- #!/bin/sh grep "$1" /tmp/t/authors | head -n 1 | cut -d'=' -f2 | cut -c'2-' --->8---

and:

git version 2.6.0.rc0.178.g4177fa5 svn, Version 1.8.13 (r1667537)

with our without --authors-file...
(It went also through with a patched git-svn and just the authors file.)

I don't think git-svn has changed since git 2.5.0. Maybe you can downgrade svn for the one-shot conversion? I don't see anything suspicious in their changelog but don't remember it as being overly complete.

git svn served us well in making itself mostly obsolete. The downside is that not many are using it any more. Keeping up with subversion changes (to keep git svn working) has become more and more difficult.

Michael
Till Schäfer· Sep 10, 2015, 17:29 UTC · re: Michael J Gruber · lore

Re: git clone svn: authors from authors file are ignored, authors-prog works, but crashes on branch points

Hi, I can confirm, that the patch (see mail: "[PATCH] git-svn: parse authors file more leniently") fixes the "Author: **** not defined in **** file" problem in my case. 

Downgrading to subversion 1.8.13 (from 1.8.14) also fixes the branching crash in revision 17. With subversion 1.8.14 i still get:

Found possible branch point: svn://svn.code.sf.net/p/scaffoldhunter/code/trunk => svn://svn.code.sf.net/p/scaffoldhunter/code/branches/subsearch, 17 Use of uninitialized value $u in substitution (s///) at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101. Use of uninitialized value $u in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101. refs/remotes/origin/trunk: 'svn://svn.code.sf.net/p/scaffoldhunter/code' not found in ''

Am Donnerstag, 10. September 2015, 14:32:07 schrieb Michael J Gruber:
Show 12 quoted lines
> Till Schäfer venit, vidit, dixit 26.08.2015 21:57:
> > Hi,
> > i am observing some weired "git svn clone" behavior during my try to migrate the Scaffold Hunter [1] SVN repository [2] to Git: 
> > 
> > if i just use the command 
> > 
> > $ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --no-metadata -s code
> 
> Is there a specific reason for "--no-metadata"? The man page advises
> against it. Basically, it keeps you from converting the history in
> chunks. (You can always filter-branch everything after a complete, "one
> shot" conversion which may be done in stages.)
i was basically following the tutorial at git-scm [1] (which is also somewhat outdated in the later svn branch conversion). Anyway, i did not read the manual :-/ (man-page)

Regards, Till

[1] https://git-scm.com/book/en/v2/Git-and-Other-Systems-Migrating-to-Git
Show 89 quoted lines
> 
> > everything went smoothly. Now i wanted to replace the authors SVN-logins by useful names and created a mapping file with the following content: 
> > 
> > anjenson = Andrew Zhilka <>
> > bernhard.dick = Bernhard Dick <>
> > dominic.sacre = Dominic Sacré <>
> > doxmoxbox = doxmoxbox <>
> > falkn = Falk Nette <>
> > henning.garus = Henning Garus <>
> > kakl = Karsten Klein <>
> > klein = Karsten Klein <>
> > lappie00 = Jeroen Lappenschaar <>
> > michael.hesse = Michael Hesse <>
> > nlskrg = Nils Kriege <nlskrg@gmail.com>
> > philipp.kopp = Philipp Kopp <>
> > philipp.lewe = Philipp Lewe <>
> > schrins = Sven Schrinner <>
> > shamshadnpti = Shamshad Alam <>
> > srenner7 = Steffen Renner <>
> > sturm89 = Werner Sturm <>
> > thomas.schmitz = Thomas Schmitz <>
> > thorsten.fluegel = Thorsten Flügel <>
> > till.schaefer = Till Schäfer <till2.schaefer@tu-dortmund.de>
> > tillschaefer = Till Schäfer <till2.schaefer@tu-dortmund.de>
> > 
> > 
> > I executed:
> > 
> > $ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --authors-file=/home/till/temp/code-authors-transform.txt --no-metadata -s code
> > 
> > and got the error message "Author: klein not defined in /home/till/temp/code-authors-transform.txt file". 
> > I tried a workaround by using a python script to return the entries i have already defined in the authors file: 
> > 
> > $ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --authors-file=/home/till/temp/code-authors-transform.txt --authors-prog=/home/till/temp/authors.py --no-metadata -s code
> > 
> > The import process now went over the "klein" commit. The weired thing is, that a few SVN usernames seem to be recognized in the authors file and a few are passed to my script (I logged the output). However, at the first revision, where a tag was added in the SVN repo the cloning process crashed with the error message: 
> 
> That first part is weird already. The reason is that author-file
> processing and author-prog output processing use different perl regexps.
> I will follow up with a patch.
> 
> > Found possible branch point: svn://svn.code.sf.net/p/scaffoldhunter/code/trunk => svn://svn.code.sf.net/p/scaffoldhunter/code/branches/subsearch, 17
> > Use of uninitialized value $u in substitution (s///) at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> > Use of uninitialized value $u in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> > refs/remotes/origin/trunk: 'svn://svn.code.sf.net/p/scaffoldhunter/code' not found in ''
> > 
> > One problem I am seeing in the SVN repo is, that between the revisions 97 and 102 the trunk folder was absent. the old one was moved to some subfolder and a new one was created. Therefore, I started the cloning beginning with rev 102 using the command line option "-r102:HEAD". However, the same error occurred for some later tag (the first tag after rev 102). 
> > 
> > Found possible branch point: svn://svn.code.sf.net/p/scaffoldhunter/code/trunk => svn://svn.code.sf.net/p/scaffoldhunter/code/tags/release-2.0, 1565
> > Use of uninitialized value $u in substitution (s///) at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> > Use of uninitialized value $u in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> > refs/remotes/origin/trunk: 'svn://svn.code.sf.net/p/scaffoldhunter/code' not found in ''
> > 
> > I would be very glad if someone has a hint about what is going wrong here. Is this a bug in git or is something wrong with the SVN repo?
> 
> The layout change (no root) certainly doesn't help. We shouldn't crash,
> though.
> 
> > used software versions: 
> > - git 2.5.0 
> > - subversion 1.8.14
> 
> The conversion goes through for me with (your other options and) your
> authors file and specifying a stupid authors prog:
> 
> --->8---
> #!/bin/sh
> grep "$1" /tmp/t/authors | head -n 1 | cut -d'=' -f2 | cut -c'2-'
> --->8---
> 
> and:
> 
> git version 2.6.0.rc0.178.g4177fa5
> svn, Version 1.8.13 (r1667537)
> 
> with our without --authors-file...
> 
> (It went also through with a patched git-svn and just the authors file.)
> 
> I don't think git-svn has changed since git 2.5.0. Maybe you can
> downgrade svn for the one-shot conversion? I don't see anything
> suspicious in their changelog but don't remember it as being overly
> complete.
> 
> git svn served us well in making itself mostly obsolete. The downside is
> that not many are using it any more. Keeping up with subversion changes
> (to keep git svn working) has become more and more difficult.
> 
> Michael
-- 
Dipl.-Inf. Till Schäfer
TU Dortmund University
Chair 11 - Algorithm Engineering
Otto-Hahn-Str. 14 / Room 237
44227 Dortmund, Germany

e-mail: till.schaefer@cs.tu-dortmund.de
phone: +49(231)755-7706
fax: +49(231)755-7740
web: http://ls11-www.cs.uni-dortmund.de/staff/schaefer
pgp: https://keyserver2.pgp.com/vkd/SubmitSearch.event?&&SearchCriteria=0xD84DED79
Michael J Gruber· Sep 10, 2015, 12:32 UTC · re: Till Schäfer · lore

[PATCH] git-svn: parse authors file more leniently

Currently, git-svn parses an authors file using the perl regex
/^(.+?|\(no author\))\s*=\s*(.+?)\s*<(.+)>\s*$/

in order to extract svn user name, real name and e-mail. This does not match an empty e-mail field like "<>". On the other hand, the output of an authors-prog is parsed with the perl regex

/^\s*(.+?)\s*<(.*)>\s*$/
in order to extract real name and e-mail.
So, specifying a trivial file grep such as
grep "$1" /tmp/authors | head -n 1 | cut -d'=' -f2 | cut -c'2-'

as the authors prog gives different results compared to specifying /tmp/authors as the authors file directly.

Instead, make git svn uses the perl regex
/^(.+?|\(no author\))\s*=\s*(.+?)\s*<(.*)>\s*$/

for parsing the authors file so that the same (slightly more lenient) regex is used in both cases.

Reported-by: Till Schäfer <till2.schaefer@tu-dortmund.de>
Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
---
 git-svn.perl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/git-svn.perl b/git-svn.perl
index 36f7240..fa5f253 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -1924,7 +1924,7 @@ sub load_authors {
 	my $log = $cmd eq 'log';
 	while (<$authors>) {
 		chomp;
-		next unless /^(.+?|\(no author\))\s*=\s*(.+?)\s*<(.+)>\s*$/;
+		next unless /^(.+?|\(no author\))\s*=\s*(.+?)\s*<(.*)>\s*$/;
 		my ($user, $name, $email) = ($1, $2, $3);
 		if ($log) {
 			$Git::SVN::Log::rusers{"$name <$email>"} = $user;

-- 
2.6.0.rc0.178.g4177fa5
Eric Wong· Sep 10, 2015, 18:08 UTC · re: Michael J Gruber · lore

Re: [PATCH] git-svn: parse authors file more leniently

Michael J Gruber <git@drmicha.warpmail.net> wrote:
Show 9 quoted lines
> Instead, make git svn uses the perl regex
> 
> /^(.+?|\(no author\))\s*=\s*(.+?)\s*<(.*)>\s*$/
> 
> for parsing the authors file so that the same (slightly more lenient)
> regex is used in both cases.
> 
> Reported-by: Till Schäfer <till2.schaefer@tu-dortmund.de>
> Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
Thanks.
Signed-off-by: Eric Wong <normalperson@yhbt.net>

And pushed to master of git://bogomips.org/git-svn (commit f7c6de0ea1bd5722a1181c6279676c6831b38a34)

By the way, I also had some other patches sitting around for you. Did you ever have time to revisit them? (I haven't)

      t/lib-httpd: load mod_unixd
      t/lib-git-svn: check same httpd module dirs as lib-httpd
Junio C Hamano· Sep 10, 2015, 21:08 UTC · re: Eric Wong · lore

Re: [PATCH] git-svn: parse authors file more leniently

Eric Wong <normalperson@yhbt.net> writes:
Show 16 quoted lines
> Michael J Gruber <git@drmicha.warpmail.net> wrote:
>> Instead, make git svn uses the perl regex
>> 
>> /^(.+?|\(no author\))\s*=\s*(.+?)\s*<(.*)>\s*$/
>> 
>> for parsing the authors file so that the same (slightly more lenient)
>> regex is used in both cases.
>> 
>> Reported-by: Till Schäfer <till2.schaefer@tu-dortmund.de>
>> Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
>
> Thanks.
> Signed-off-by: Eric Wong <normalperson@yhbt.net>
>
> And pushed to master of git://bogomips.org/git-svn
> (commit f7c6de0ea1bd5722a1181c6279676c6831b38a34)
Thanks, pulled.
Michael J Gruber· Sep 11, 2015, 08:36 UTC · re: Eric Wong · lore

Re: [PATCH] git-svn: parse authors file more leniently

Eric Wong venit, vidit, dixit 10.09.2015 20:08:
Show 23 quoted lines
> Michael J Gruber <git@drmicha.warpmail.net> wrote:
>> Instead, make git svn uses the perl regex
>>
>> /^(.+?|\(no author\))\s*=\s*(.+?)\s*<(.*)>\s*$/
>>
>> for parsing the authors file so that the same (slightly more lenient)
>> regex is used in both cases.
>>
>> Reported-by: Till Schäfer <till2.schaefer@tu-dortmund.de>
>> Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
> 
> Thanks.
> Signed-off-by: Eric Wong <normalperson@yhbt.net>
> 
> And pushed to master of git://bogomips.org/git-svn
> (commit f7c6de0ea1bd5722a1181c6279676c6831b38a34)
> 
> By the way, I also had some other patches sitting around for you.
> Did you ever have time to revisit them?  (I haven't)
> 
>       t/lib-httpd: load mod_unixd
>       t/lib-git-svn: check same httpd module dirs as lib-httpd
> 
Also "from me".
Short answer: No

If I remember correctly, they were correct bit not complete in the sense that on a standard Fedora install (with newer apache), svn tests still wouldn't run over http. But I/we learned that those tests were simply run over local file protocol instead when svn over http didn't work. On a standard debian install (which apparantly has non-standard, thus downwards compatible apache config) everything was fine with or without those patches.

I still plan to look at them when I find time. (I'll be retiring sometime between 20 and 30 years from now, so there's hope.)

Michael

← back to recent threads