{"thread":{"id":"11452","subject":"Stitching together private svn repo and public git repo","startedAt":"2008-01-02T19:25:41Z","lastAt":"2008-01-03T17:24:31Z","messageCount":6,"participants":["Gregory Jefferis","Charles Bailey","Dmitry Potapov","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"64346","messageId":"C3A195B5.10839%jefferis@gmail.com","threadId":"11452","inReplyTo":null,"subject":"Stitching together private svn repo and public git repo","fromName":"Gregory Jefferis","fromEmail":"jefferis@gmail.com","sentAt":"2008-01-02T19:25:41Z","receivedAt":"2008-01-02T19:25:41Z","isPatch":false,"sender":{"key":"jefferis@gmail.com","avatar":"https://gravatar.com/avatar/8528f4e227ebc88380c203c816e263246064e31f6180681dc2c6dabfaff24d6b?d=mp&s=160"},"body":"Short Version: \n\nRepo B is a completely linear repo that has been tracking public repo A for\nsome time. A number of manual merges were done to bring new A releases\ninto B but there are no connections between the two repos. (How) can I\nstitch B to A to reflect this relationship? I want to retain B's history\nand leave myself with an arrangement in which I can pull from A into B\nperiodically? Many thanks for your suggestions, Greg.\n\n--\nLong Version:\n\nI have been tracking my modifications of a public project released as tar\nballs with my own local svn repository. My svn repo was completely linear\nand I always merged in changes manually each time a new tar ball was\nreleased. This started to get painful.\n\nNow I want to do better. I have set up a public git repository (\"A\") to\ntrack the tar ball releases from the public project. I have converted my\nsvn repo with git-svn to a git repo (\"B\").\n\nMy goal is to end up with a (new?) git repo tracking the public repo (A) so\nthat I can pull in any changes and merge easily. Fine, but I also want to\nkeep the history that I have imported from my svn repository (B). So my\nquestion is, can anyone suggest some pointers for how to stitch together A\nand B?\n\nRight now I have been trying to pull B into A to splice:\n\nA $ git checkout v1.91\nB $ git checkout v1.91-manualmerge\nB $ git pull --no-commit -s ours ../A\n\nThis looks right when I run gitk, but if I repeat this for a second merge\npoint then the previous merge seems to disappear from history when I bring\nup gitk.\n\nI haven't found any docs/wiki info that has enlightened me yet. Any\nsuggestions, wisdom, links etc very much appreciated - I do want to try to\nget this right now, so I don't have to rejig everything in the future. Best\nwishes and many thanks,\n\nGreg.\n\n-- \nGregory Jefferis, PhD and:\nResearch Fellow\nDepartment of Zoology St John's College\nDowning Street Cambridge\nCambridge, CB2 3EJ CB2 1TP\n"},{"id":"64353","messageId":"20080102214005.GA10924@hashpling.org","threadId":"11452","inReplyTo":"C3A195B5.10839%jefferis@gmail.com","subject":"Re: Stitching together private svn repo and public git repo","fromName":"Charles Bailey","fromEmail":"charles@hashpling.org","sentAt":"2008-01-02T21:40:05Z","receivedAt":"2008-01-02T21:40:05Z","isPatch":false,"sender":{"key":"charles@hashpling.org","avatar":"https://avatars.githubusercontent.com/u/1668475?v=4"},"body":"On Wed, Jan 02, 2008 at 07:25:41PM +0000, Gregory Jefferis wrote:\n> \n> Right now I have been trying to pull B into A to splice:\n> \n> A $ git checkout v1.91\n> B $ git checkout v1.91-manualmerge\n> B $ git pull --no-commit -s ours ../A\n\nSo you have something like this in B:\n\nAv1* -- Av1_b -- Av1_c -- Av1_d -- Av2_d* -- Av2_e -- Av3_e*\n\nWhere the * are manual merges of the offical Avn releases...\n\nAnd you want this:\n\nAv1 -- -- -- Av2 -- -- Av3\n \\ \\ \\\n Av1_b -- Av1_c -- Av1_d -- Av2_d -- Av2_e -- Av3_e ???\n\nThis is a \"rewrite history\" job as parent lists are part of each\ncommit. (See Linus' big bold-face warning about history re-writes.)\n\nFirst of all I would add a remote for the 'A' repository so that those\ncommits are available in the 'B' repository.\n\nSomething like:\ngit remote add repoA /path/to/A\ngit fetch repoA\n\nYou could then do this with a 'git filter-branch --parent-filter' to\nrewrite the parents of the merge commits. As far as I can see, you\nwould need to call filter-branch once per merge to rewrite everything\nfrom the merge commit forwards. At this point all later commits would\nhave different ids, so attempting to rewrite subsequent parent-ids in\nthe same filter-branch invocation is probably futile.\n\nIt might be possible to use an intelligent script in a single\n--commit-filter invocation of filter-branch, but the script of the\nactual filter would have to be a bit (a lot!) more sophisticated,\nremember the ids of the new commits as it created them with 'git\ncommit-tree' and merging in the repoA parents at the right points.\n\nLeaving that aside and concentrating on the multiple filter-branch\ninvocation option... for example, the first parent-filter script could\nbe:\nsed -e 's/^$/-p /'\n\n( This is if you want your exisiting tree to branch from Av1 in the\noriginal repo. If you wanted to replace your tree root with a root at\nAv1, because your current root is just a copy of Av1 then you want:\nsed -e 's///'\n)\n\nThen, for whatever the Av2_d commit's new id is, you could do a\nparent-filter of:\nsed -e 's/\\(\\)/\\1 -p /'\n\nThis causes the Av2_d commit to be rewritten, maintaining its original\nparent and adding another parent which comes from repoA. It then\nrewrites all the descendent commits of Av2_d to take account of this\nnew commit id and all the subsequent new commit ids.\n\nNow you can do similar for the next merge commit:\nsed -e 's/\\(\\)/\\1 -p /'\n\nAs noted, you end up with completely new commit objects with a\ncompletely new history, so you will screw everyone who is\npulling/cloning from you.\n\nCharles.\n"},{"id":"64357","messageId":"37fcd2780801021413g5f2a602dkb9fc289a13517375@mail.gmail.com","threadId":"11452","inReplyTo":"20080102214005.GA10924@hashpling.org","subject":"Re: Stitching together private svn repo and public git repo","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2008-01-02T22:13:54Z","receivedAt":"2008-01-02T22:13:54Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Jan 3, 2008 12:40 AM, Charles Bailey wrote:\n> You could then do this with a 'git filter-branch --parent-filter' to\n> rewrite the parents of the merge commits. As far as I can see, you\n> would need to call filter-branch once per merge to rewrite everything\n> from the merge commit forwards.\n\nI believe there is a much easier way to do that using .git/info/grafts\nThe first step is to create .git/info/grafts, which specifies commit-id\nand its parents for each commit that you want to change. Then you\ncan check the result using gitk, and if you are satisfied with what\nyou see then you run git filter-branch on it to convert 'fake' parents\ninto real ones.\n\nDmitry\n"},{"id":"64360","messageId":"20080102224611.GA14984@hashpling.org","threadId":"11452","inReplyTo":"37fcd2780801021413g5f2a602dkb9fc289a13517375@mail.gmail.com","subject":"Re: Stitching together private svn repo and public git repo","fromName":"Charles Bailey","fromEmail":"charles@hashpling.org","sentAt":"2008-01-02T22:46:11Z","receivedAt":"2008-01-02T22:46:11Z","isPatch":false,"sender":{"key":"charles@hashpling.org","avatar":"https://avatars.githubusercontent.com/u/1668475?v=4"},"body":"On Thu, Jan 03, 2008 at 01:13:54AM +0300, Dmitry Potapov wrote:\n> I believe there is a much easier way to do that using .git/info/grafts\n> The first step is to create .git/info/grafts, which specifies commit-id\n> and its parents for each commit that you want to change. Then you\n> can check the result using gitk, and if you are satisfied with what\n> you see then you run git filter-branch on it to convert 'fake' parents\n> into real ones.\n> \n> Dmitry\n> \n\nOh yes, this is much easier. Unless I'm missing something, the\ndocumentation on grafts is fairly sparse, though. They are mentioned\n(almost in passing) in git help filter-branch but the file format is\nonly documented in repository-layout.txt which seems more developer\nthan user oriented.\n\nCharles.\n"},{"id":"64387","messageId":"C3A2A4FD.1089D%jefferis@gmail.com","threadId":"11452","inReplyTo":"20080102224611.GA14984@hashpling.org","subject":"Re: Stitching together private svn repo and public git repo","fromName":"Gregory Jefferis","fromEmail":"jefferis@gmail.com","sentAt":"2008-01-03T14:43:09Z","receivedAt":"2008-01-03T14:43:09Z","isPatch":false,"sender":{"key":"jefferis@gmail.com","avatar":"https://gravatar.com/avatar/8528f4e227ebc88380c203c816e263246064e31f6180681dc2c6dabfaff24d6b?d=mp&s=160"},"body":"Many thanks to both for your replies. Starting from your comments and\nconsulting:\n\nhttp://www.kernel.org/pub/software/scm/git/docs/repository-layout.html\n\nfor (fairly scant) details of .git/info/grafts and\n \nhttp://www.kernel.org/pub/software/scm/git/docs/git-filter-branch.html\n\nI think I have now been able to do the necessary. Since I found this a\ntrivial operation (in the true mathematical sense of quasi-impossible until\nsomeone helps you to figure it out and simple once you have) and I tripped\nover a couple of times, I thought I would share my final shell script below.\n\nOne point that I missed initially. The graft file format is unforgiving.\nYou are allowed comments but each regular line is:\n\nCommit-SHA-1 Parent1-SHA-1 Parent2-SHA-1\\n\n \nIf you put e.g. more than a single space between your SHA-1s you will get\n\"bad graft data:\" - I missed one in the fake merge line and ran round in\ncircles for a bit thinking it was something to do with the state of my\ntrees.\n\nAnyway, boy did it feel great when I was able to git-pull the latest public\nrelease into my private repository:\n\n 34 files changed, 11563 insertions, 8399 deletions\n\nand only 2 simple conflicts to fix with git-mergetool. That was not a merge\nI wanted to do by hand!\n\nBest wishes and thanks again,\n\nGreg.\n-- \n\n#!/usr/bin/env bash\n# makemygitbygraft.sh\n# merge my private repo (converted from svn) with a public git repo\n# retains history of both and will allow 3 way merges etc in future\n\n# work on a copy of my svn imported repo just in case\ncd ~/dev/Physiology\nrm -rf ~/dev/Physiology/nm-graft\ncp -R nm-git-svn2 nm-graft\ncd nm-graft\n\n# Now get to work, bring in public repo as a remote\ngit remote add nm-ucl ../nm-ucl\ngit fetch nm-ucl\n\n# set up the grafts\ncat > .git/info/grafts < wrote:\n\n> On Thu, Jan 03, 2008 at 01:13:54AM +0300, Dmitry Potapov wrote:\n>> I believe there is a much easier way to do that using .git/info/grafts\n>> The first step is to create .git/info/grafts, which specifies commit-id\n>> and its parents for each commit that you want to change. Then you\n>> can check the result using gitk, and if you are satisfied with what\n>> you see then you run git filter-branch on it to convert 'fake' parents\n>> into real ones.\n>> \n>> Dmitry\n>> \n> \n> Oh yes, this is much easier. Unless I'm missing something, the\n> documentation on grafts is fairly sparse, though. They are mentioned\n> (almost in passing) in git help filter-branch but the file format is\n> only documented in repository-layout.txt which seems more developer\n> than user oriented.\n> \n> Charles.\n\n-- \nGregory Jefferis, PhD\nDivision of Neurobiology\nMRC Laboratory of Molecular Biology,\nHills Road,\nCambridge, CB2 0QH, UK.\n\nhttp://www2.mrc-lmb.cam.ac.uk/NB/jefferis_g\nhttp://www.neuroscience.cam.ac.uk/directory/profile.php?gsxej2\nhttp://flybrain.stanford.edu\n"},{"id":"64388","messageId":"7vd4sistyo.fsf@gitster.siamese.dyndns.org","threadId":"11452","inReplyTo":"C3A2A4FD.1089D%jefferis@gmail.com","subject":"Re: Stitching together private svn repo and public git repo","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-01-03T17:24:31Z","receivedAt":"2008-01-03T17:24:31Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"I suspect your message was linewrapped by your MUA and ...\n\nGregory Jefferis writes:\n\n> ...\n> # set up the grafts\n> cat > .git/info/grafts < # connect r2 of git-svn to v1.86 of nm-ucl\n> # r1 (which was an identical initial import) will now be detached (and\n> forgotten)\n> 52b1933421ef524811407fa4c240da58ceec5749\n> eca3db14fcf25744fdf585456f03599a9db2af96\n> #\n> # Make fake merge with Neuromatic v1.91b\n> # svn-manualmerge1 svn-manualmerge1~1 public-v1.91b\n> 120282ee5275027312dde386c8995218f361cf35\n> 0962cb27f7ad92f44def04630e8e6a22b86e0699\n> 6735057f53dd57248c7ec23f6ae9f22085d98fba\n> EOF\n>\n> # running git-status seems to stop git thinking that the working dir is\n> dirty\n> git-status\n> ...\n\n... your grafts file look as if you are marking five commits as\nroot commits, which is not what you wanted ;-).\n\nThe reason I mention this is because I think with a bit more\npolishing and some background description your message is a good\nmaterial to add to Documentation/howto/.\n"}]}