Re: [bug] git svn fetch: defect history - missing merges and wrong tag ancestors - third-party tool found

> On 2020-12-25 23:24, M. Buecher wrote:
>> Dear all,
>> 
>> I finally had the time to start converting some older Subversion
>> repositories to git repositiers and run into issues with repos using
>> "parallel" branches, so-called vendor branches [1] in Subversion to
>> track upstream changes via snapshots in a separate branch and merge
>> them into the custom build on trunk.
>> 
>> Just wanting to convert the Subversion history as-is to git.
>> I studied the git-svn reference docs [2] plus the related chapter of
>> the ProGit book [3] and assume that I understood how git-svn works.
>> Still I'm not a git expert, just a sporadic user.
>> 
>> Somehow `git svn fetch` (2.29.2.windows.3) looses merge information
>> between vendor branch and trunk plus tags are referencing the
>> predecessors instead of the original ancestor.
>> Maybe there is a manual way to fix this for that small repository, but
>> it wouldn't be feasible for larger repositories, that's why I deceided
>> to write this bug report.
>> Is there anything I missed? (see my procedure below)
>> 
>> Fortunately I ran quite early into these issues and also with a very
>> small repository of just 11 commits, so I can provide a small
>> reproduction case (see below after the links).
>> Hoping you can enhance `git svn fetch`.
> 
> Tested further with re-created Subversion repositories, that either
> looked the same but made sure that svn:mergeinfo is present
> (Subversion >=1.5 via "cherry pick merge", no "2-URL merge"), or that
> has the vendor branch being copied from trunk (both attached).
> Only when the vendor branch was a copy from trunk, then svn-git got
> the merges correct. Otherwise - even with svn:mergeinfo - it does not
> get the merges.
> 
> Assumption:
> It seems that git svn handles trunk and branches in a special way, but
> Subversion actually does not have branches.
> In Subversion there are just directories and files, and a branch/tag
> is just a copy of another directory+revision and merges can happen
> between any directories independent if they have related ancestry or
> not.
> 
> Workaround:
> As git svn does not recognize all merges correctly (especially
> cross-branch copies) those lost links must be added manually.
> I wrote a small GNU awk script (attached) to determine all
> cross-branch copies from an `svnadmin dump`. This way the first
> revision when something got copied is known.
> Running git svn just to that revision, then fixing the parents and
> continuing with git svn up the next revision.
> This helps git svn to find the correct ancestors and correctly build
> the follow merges.
> The parents can be changed with `git replace -f --graft <commit>
> <existing correct parents> <additional correct parents>`.
> Additionally before continuing with git svn this `git replace` change
> can be made permanent with `git-filter-repo --force --replace-refs
> delete-no-add`.
> 
> 
>> Any help is appreciated, thanks in advance
>> Matthias Bücher

>> [Links]
>> [1]
>> http://svnbook.red-bean.com/en/1.8/svn.advanced.vendorbr.html#svn.advanced.vendorbr.mirrored-sources
>> [2] https://git-scm.com/docs/git-svn
>> [3] 
>> https://git-scm.com/book/en/v2/Git-and-Other-Systems-Migrating-to-Git
>> 
>> 
>> [System Info]
>> git version:
>> git version 2.29.2.windows.3
>> cpu: x86_64
>> built from commit: d054eb1fc46ff23e7c95756a7c747e2f2864b478
>> sizeof-long: 4
>> sizeof-size_t: 8
>> shell-path: /bin/sh
>> uname: Windows 10.0 19042
>> compiler info: gnuc: 10.2
>> libc info: no libc information available
>> $SHELL (typically, interactive shell): C:\Program 
>> Files\Git\usr\bin\bash.exe
>> 
>> [Enabled Hooks]
>> none
>> 
>> 
>> [Commits in Subversion]
>> A Subversion repository dump is attached, plus a test where I
>> recreated the same history directly in a git repository without an
>> issue.
>> 
>> * trunk:  a------------d--e--h--i--j--k
>> *                     /     /
>> * vendor: a (empty)--b-----f
>> *                    ^     ^
>> * tags:              c     g
>> 
>> * Vendor releases: b, f
>> * Custom modifications: e, i, j, k
>> * Tags: really just used as tags, although Subversion internally they
>> are branches. Therefore wanting to create lightweight git tags,
>> although annotated git tags would be fine too.
>> 
>> 
>> [Expected Subversion to git repo conversion]
>> * trunk => branch "main"
>> * branches/* => branch "*"
>> * tags/* => tag "*"
>> * vendor/current => branch "vendor/current" (can be renamed later to
>> just "vendor")
>> * vendor/* => tag "vendor/*" (except for vendor/current)
>> 
>> Expected "tags":
>> vendor/5.4 = rev b (maybe c when annotated)
>> vendor/5.8 = rev f (maybe g when annotated)
>> 
>> 
>> [Wrong Results]
>> Merges from "vendor/current" branch to trunk get lost.
>> Tags are referencing the predecessors of the expected commit.
>> 
>> 
>> [Procedure]
>> ```
>> ### a) preparation
>> cd /d/Coding
>> #
>> cd dd-formmailer
>> svn log --xml --quiet | grep author | sort -u | perl -pe
>> 's/.*>(.*?)<.*/$1 = /' > authors-transform.txt
>> ## edit authors-transform.txt accordingly
>> 
>> ### b) git-svn adapted from ProGit book, but with "svn/" prefix
>> cd /d/Coding
>> git svn init --stdlayout --no-metadata --prefix="svn/" --
>> 'svn+ssh://svn@vcs/dd-formmailer' dd-formmailer-git
>> cd dd-formmailer-git
>> ## edit .git/config for additional vendor branches and tags
>> << __EOF
>> ...
>> [svn-remote "svn"]
>> 	...
>> 	fetch = trunk:refs/remotes/svn/trunk
>> #	branches = vendor/current:refs/remotes/svn/vendor/current ##
>> non-glob definition not working for branches
>> 	branches = vendor/{current}:refs/remotes/svn/vendor/*
>> 	branches = branches/*:refs/remotes/svn/*
>> 	tags = vendor/*:refs/remotes/svn/tags/vendor/*
>> 	tags = tags/*:refs/remotes/svn/tags/*
>> __EOF
>> ##
>> cat .git/config
>> git svn fetch --authors-file ../dd-formmailer/authors-transform.txt
>> #
>> git branch -vv --list ; git for-each-ref
>> gitk --all &
>> ```
>> 
>> [Log]
>> $ cat .git/config
>> [core]
>>         repositoryformatversion = 0
>>         filemode = false
>>         bare = false
>>         logallrefupdates = true
>>         symlinks = false
>>         ignorecase = true
>> [svn-remote "svn"]
>>         noMetadata = 1
>>         url = svn+ssh://svn@vcs/dd-formmailer
>>         fetch = trunk:refs/remotes/svn/trunk
>>         branches = vendor/{current}:refs/remotes/svn/vendor/*
>>         branches = branches/*:refs/remotes/svn/*
>>         tags = vendor/*:refs/remotes/svn/tags/vendor/*
>>         tags = tags/*:refs/remotes/svn/tags/*
>> 
>> $ git svn fetch --authors-file ../dd-formmailer/authors-transform.txt
>> r1 = 158d53044a5628897379403647a19ea13594b532 
>> (refs/remotes/svn/vendor/current)
>>         A       _svn__guideline.txt
>>         A       _svn_client_config.txt
>>         A       _svn_dir_ignore_list.txt
>> r1 = a795b654edbc296e6f38da398f032a4851fd0a9e (refs/remotes/svn/trunk)
>>         A       dd-formmailer.css
>>         A       dd-formmailer.php
>>         A       lang/BrazilianPortuguese.php
>>         A       lang/Catalan.php
>>         A       lang/Danish.php
>>         A       lang/Deutsch.php
>>         A       lang/Dutch.php
>>         A       lang/English.php
>>         A       lang/Finnish.php
>>         A       lang/French.php
>>         A       lang/Greek.php
>>         A       lang/Italian.php
>>         A       lang/NorwegianBokmaal.php
>>         A       lang/Polish.php
>>         A       lang/Portuguese.php
>>         A       lang/Romanian.php
>>         A       lang/Russian.php
>>         A       lang/Slovak.php
>>         A       lang/Slovene.php
>>         A       lang/Spanish.php
>>         A       lang/Swedish.php
>>         A       lang/Turkish.php
>>         A       recaptchalib.php
>> r2 = 8fd8668dbed2dde6b55306c71b3b629f5ed794ec 
>> (refs/remotes/svn/vendor/current)
>> Found possible branch point:
>> svn+ssh://svn@vcs/dd-formmailer/vendor/current =>
>> svn+ssh://svn@vcs/dd-formmailer/vendor/5.4, 1
>> Found branch parent: (refs/remotes/svn/tags/vendor/5.4)
>> 158d53044a5628897379403647a19ea13594b532
>> Following parent with do_switch
>>         A       dd-formmailer.css
>>         A       dd-formmailer.php
>>         A       lang/BrazilianPortuguese.php
>>         A       lang/Catalan.php
>>         A       lang/Danish.php
>>         A       lang/Deutsch.php
>>         A       lang/Dutch.php
>>         A       lang/English.php
>>         A       lang/Finnish.php
>>         A       lang/French.php
>>         A       lang/Greek.php
>>         A       lang/Italian.php
>>         A       lang/NorwegianBokmaal.php
>>         A       lang/Polish.php
>>         A       lang/Portuguese.php
>>         A       lang/Romanian.php
>>         A       lang/Russian.php
>>         A       lang/Slovak.php
>>         A       lang/Slovene.php
>>         A       lang/Spanish.php
>>         A       lang/Swedish.php
>>         A       lang/Turkish.php
>>         A       recaptchalib.php
>> Successfully followed parent
>> r3 = 7ab4f436cafc8af3ed2e727a6d8cbef1a8f8b39f 
>> (refs/remotes/svn/tags/vendor/5.4)
>>         A       dd-formmailer.css
>>         A       dd-formmailer.php
>>         A       lang/BrazilianPortuguese.php
>>         A       lang/Catalan.php
>>         A       lang/Danish.php
>>         A       lang/Deutsch.php
>>         A       lang/Dutch.php
>>         A       lang/English.php
>>         A       lang/Finnish.php
>>         A       lang/French.php
>>         A       lang/Greek.php
>>         A       lang/Italian.php
>>         A       lang/NorwegianBokmaal.php
>>         A       lang/Polish.php
>>         A       lang/Portuguese.php
>>         A       lang/Romanian.php
>>         A       lang/Russian.php
>>         A       lang/Slovak.php
>>         A       lang/Slovene.php
>>         A       lang/Spanish.php
>>         A       lang/Swedish.php
>>         A       lang/Turkish.php
>>         A       recaptchalib.php
>> r4 = 7ec3663fd8e7c9fcfeb2742968a948d6978776d4 (refs/remotes/svn/trunk)
>>         M       dd-formmailer.css
>>         M       dd-formmailer.php
>>         A       dd-verify.php
>> r5 = e65ba2bdcd12a1935c1a327507dad6b7117f452b (refs/remotes/svn/trunk)
>>         A       calendar.gif
>>         A       date_chooser.js
>>         M       dd-formmailer.css
>>         M       dd-formmailer.php
>>         A       lang/Belarussian.php
>>         A       lang/Czech.php
>>         A       lang/Estonian.php
>>         A       lang/Japanese.php
>>         A       lang/Vietnamese.php
>>         M       recaptchalib.php
>> r6 = 9f95df7ce49c5d1cb4715017d223a8bd1c8dcffc 
>> (refs/remotes/svn/vendor/current)
>> Found possible branch point:
>> svn+ssh://svn@vcs/dd-formmailer/vendor/current =>
>> svn+ssh://svn@vcs/dd-formmailer/vendor/5.8, 3
>> Found branch parent: (refs/remotes/svn/tags/vendor/5.8)
>> 8fd8668dbed2dde6b55306c71b3b629f5ed794ec
>> Following parent with do_switch
>>         A       calendar.gif
>>         A       date_chooser.js
>>         M       dd-formmailer.css
>>         M       dd-formmailer.php
>>         A       lang/Belarussian.php
>>         A       lang/Czech.php
>>         A       lang/Estonian.php
>>         A       lang/Japanese.php
>>         A       lang/Vietnamese.php
>>         M       recaptchalib.php
>> Successfully followed parent
>> r7 = cc00cb187386298cf974dda69f151d2ad4795917 
>> (refs/remotes/svn/tags/vendor/5.8)
>>         A       calendar.gif
>>         A       date_chooser.js
>>         M       dd-formmailer.css
>>         M       dd-formmailer.php
>>         M       dd-verify.php
>>         A       lang/Belarussian.php
>>         A       lang/Czech.php
>>         A       lang/Estonian.php
>>         A       lang/Japanese.php
>>         A       lang/Vietnamese.php
>>         M       recaptchalib.php
>> Checking svn:mergeinfo changes since r5: 1 sources, 1 changed
>> W: Cannot find common ancestor between
>> e65ba2bdcd12a1935c1a327507dad6b7117f452b and
>> 9f95df7ce49c5d1cb4715017d223a8bd1c8dcffc. Ignoring merge info.
>> r8 = 9a5cd8f55f377f469280171cd87219b8f528c693 (refs/remotes/svn/trunk)
>>         M       dd-formmailer.php
>> r9 = 8143782376d9fbc91a9181b367b72701205b017f (refs/remotes/svn/trunk)
>>         M       dd-formmailer.php
>> r10 = 6f563fc4c511ddcc53c2ffc5d26c1e116725bc6b 
>> (refs/remotes/svn/trunk)
>>         M       _svn_client_config.txt
>> r11 = e9130854178d2c2743a981f303cdd7f34e54b052 
>> (refs/remotes/svn/trunk)
>> svnserve: E210002: Network connection closed unexpectedly
>> Checked out HEAD:
>>   svn+ssh://svn@vcs/dd-formmailer/trunk r11
>> 
>> $ git branch -vv --list ; git for-each-ref
>> * main e913085 Updated client config for Windows Scripts
>> e9130854178d2c2743a981f303cdd7f34e54b052 commit refs/heads/main
>> 7ab4f436cafc8af3ed2e727a6d8cbef1a8f8b39f commit 
>> refs/remotes/svn/tags/vendor/5.4
>> cc00cb187386298cf974dda69f151d2ad4795917 commit 
>> refs/remotes/svn/tags/vendor/5.8
>> e9130854178d2c2743a981f303cdd7f34e54b052 commit refs/remotes/svn/trunk
>> 9f95df7ce49c5d1cb4715017d223a8bd1c8dcffc commit 
>> refs/remotes/svn/vendor/current