{"thread":{"id":"7748","subject":"Using git to bridge two svn repositories: a success story","startedAt":"2007-04-20T04:12:52Z","lastAt":"2007-04-20T07:18:37Z","messageCount":2,"participants":["Steven Grimm","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"39937","messageId":"46283DC4.7070604@midwinter.com","threadId":"7748","inReplyTo":null,"subject":"Using git to bridge two svn repositories: a success story","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2007-04-20T04:12:52Z","receivedAt":"2007-04-20T04:12:52Z","isPatch":false,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"I complain too much of the time on this list, so here's a success story \nI can share for a change. I just used git to merge two separate svn \nrepositories: the official repo for an open-source program and an \ninternal repo with our locally-modified version of the same program. The \nlocal copy has been tracking the official one off and on over time; it \nhas a bunch of changes that were contributed back to the official code \nbase at various points, other changes that weren't, and some directory \nlayout changes to accommodate our internal build system.\n\nWe had fallen fairly far behind the official version, so yesterday I \ndecided to bring us up to date. Not a trivial merge; various of our \nchanges had been applied to different branches in the official svn \nrepository, which had gotten merged back into their trunk at various \npoints. In many cases local change A appeared before remote change B in \nour history but in the opposite order in the official repo since they \ncommitted our change after the other one.\n\nObviously svn is nowhere near adequate to the task of normalizing these \ntwo code bases. So I used git instead, and it worked out great. \nSpecifically, here's what I did, minus a few false starts:\n\n1. Made two git-svn repositories, one based on our local code base and \none based on the official svn repository.\n\n2. Created a git repository and pulled from both of the git-svn repos. \n(I know I could have done this with one repo instead of three, but I \nwanted to make sure I could easily blow away one of the parts of this \nand start over.)\n\n3. Added a couple of .git/info/grafts entries for places where I knew \nthe original project had merged branches back into trunk, but where \ngit-svn hadn't detected the merge. Probably not git-svn's fault, given \nhow brittle merging is in svn and the fact that a couple of the merges \nwere split across multiple svn revisions.\n\n4. Found an early point in our history when we had a fairly close to \nunmodified copy of the distribution at the time and created a branch \nfrom that revision.\n\n5. Renamed the files from our layout back to the distribution's. (I'll \ntalk more about this below.)\n\n6. Did a baseless merge with the corresponding revision of the \ndistribution's history. Resolved the conflicts, which weren't too severe \nthanks to step 4.\n\n7. Walked through the revision history on both sides merging into my \nintegration branch. I was more cautious about this than I probably \nneeded to be (though more on that below too); my approach was to merge \nup to a particular change on our side that I knew we'd contributed \nupstream, then merge up to the corresponding revision on the official \nside, repeat until done. In cases where our stuff had been integrated \ninto a branch in the official repo, I followed that branch rather than \ntrunk for the most part. I ended up walking three branches plus trunk.\n\n8. Once I had merged the last of our local changes, I merged the head of \nthe official trunk into my integration branch, picking up a bunch of \nofficial revs in one step.\n\n9. Renamed everything back to our naming conventions.\n\nThis was kind of an iterative process and the main reason I did it \nincrementally at first was mostly to limit the amount of conflict \nresolution at any one step, as well as to make sure that each of our \ncontributions had in fact been merged correctly. (I wrote most of the \ncode we contributed so I was able to quickly tell if it looked right.)\n\nThe gitk display for this repo looks like a ladder; nearly every \nrevision of my integration branch is a merge.\n\nNow, about those renames. The major change in structure was to rename \nthe source directory from \"server\" in the official repository to \"src\" \nwhich our build system expects. So before I did any merges, I committed \na revision where I did \"git mv src server\" (along with a couple other \nsimilar renames) so there'd be an explicit rename-only revision for \ngit's rename detection to use to apply changes to the right files.\n\nUnfortunately, that broke down as soon as I got to a contribution of \nours that added a new file. I merged the contribution on our side (where \neverything lives in src/), and it correctly applied the modifications to \nthe existing files in server/ thanks to the renames in the history. But \nthe new files were created in src/. I didn't notice the file missing \nfrom server/ at first, and merged the revision from the official repo \nthat created the same file there. The new file was identical on both \nsides, so I didn't think it was odd that there wasn't a conflict, and \nproceeded to the next rev. It was only after several more revisions \nmerged from both sides that I noticed the server/ copy of the file was \nmissing changes I'd sworn I'd just merged from our side. Naturally all \nour local changes were getting successfully applied to the copy in src/ \nand all the changes from the official repo were showing up in the \nserver/ copy.\n\nSo I ended up resetting back to the first revision that created a new \nfile in src/, and making sure I stopped at each revision that introduced \na new file there so I could commit an extra revision after the merge to \nmanually rename it into server/. Then the subsequent merge with the \nrevision that created the file in server/ would correctly flag any \ndifferences between the two versions as conflicts, and I could go \nthrough and do the right thing with them. There were only three or four \nsuch cases so it wasn't too much extra work.\n\nThe only other glitch I ran into was missing one merge from the official \nsvn repository when I created my grafts file. That caused me to get a \nbunch of repeat conflicts when I merged the subsequent svn trunk \nrevision. But I immediately realized what was happening there; I reset, \nadded the missing merge to my grafts file, and did the merge again, and \nthe repeat conflicts went away.\n\nAside from those two minor things, it was a painless exercise, and now I \nhave a reasonably coherent (if a bit convoluted) combined history of the \ntwo versions of the code base without the svn repositories on either \nside being aware of each other. I plan to keep all these git \nrepositories around so I can quickly integrate subsequent changes from \nboth sides.\n\nSo, kudos all around. Without git this would have been a much more \ntime-consuming and error-prone exercise!\n\n-Steve\n"},{"id":"39947","messageId":"7vk5w7344y.fsf@assigned-by-dhcp.cox.net","threadId":"7748","inReplyTo":"46283DC4.7070604@midwinter.com","subject":"Re: Using git to bridge two svn repositories: a success story","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-04-20T07:18:37Z","receivedAt":"2007-04-20T07:18:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Steven Grimm <koreth@midwinter.com> writes:\n\n> ..., so here's a success\n> story I can share for a change. I just used git to merge two separate\n> svn repositories: the official repo for an open-source program and an\n> internal repo with our locally-modified version of the same\n> program.\n\nNice story.  Thanks for sharing.\n\n> Now, about those renames. The major change in structure was to rename\n> the source directory from \"server\" in the official repository to \"src\"\n> which our build system expects. So before I did any merges, I\n> committed a revision where I did \"git mv src server\" (along with a\n> couple other similar renames) so there'd be an explicit rename-only\n> revision for git's rename detection to use to apply changes to the\n> right files.\n\nIn the work you did in the story, your rename from server to src\nwas indeed an atomic action YOU wanted to have, which was done\nto match the two tree structure.  It was your project, not git,\nthat did not want the name upstream uses.  So it makes perfect\nsense to have that rename-only commit.  But if you do it only\nbecause you think it would help later merges, don't.\n\nI do not know who started this myth, but \"rename only commit\"\ndoes not help rename detection in merges AT ALL, as rename\ndeteciton is not done step-by-step, but between ancestor and the\ntip of each branch.\n\nA \"rename-only commit\" does help if you are following history\nwith \"git log -p -M\", where rename detection logic compares\ntrees stepwise.  I think somebody confused this with the rename\ndetection done by the merge machinery to start the myth.  Please\ndo not spread it any further.\n"}]}