{"thread":{"id":"46198","subject":"Best practices for updating old repos","startedAt":"2017-06-16T01:00:43Z","lastAt":"2017-06-16T10:17:01Z","messageCount":5,"participants":["Michael Eager","Stefan Beller","Michael O'Cleirigh"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"322429","messageId":"59432BCC.2040901@eagerm.com","threadId":"46198","inReplyTo":null,"subject":"Best practices for updating old repos","fromName":"Michael Eager","fromEmail":"eager@eagerm.com","sentAt":"2017-06-16T00:52:28Z","receivedAt":"2017-06-16T01:00:43Z","isPatch":false,"sender":{"key":"eager@eagerm.com","avatar":null},"body":"Hi All --\n\nI'm working with code that is based on a five year old repository.\nThere are 130 local commits since the repo was forked. Naturally,\nthe upstream project has moved on significantly.\n\nI'm wondering about best approaches to updating the repo to the\ncurrent upstream version. Here are the approaches I've considered:\n\n- Rebase from upstream. Likely almost every patch will fail with\n multiple merge conflicts.\n\n- Merge local branch into upstream. Likely many merge failures, but\n fewer than with rebase.\n\n- Apply individual patches from the old repo to the upstream repo.\n Fix merge conflicts, rebuild, fix build failures. There may be\n some duplication and additional merge problems created, where a\n later patch from the old repo fixes the same conflict or build\n failure.\n\nI've tried each of these approaches on various projects. Each has\nproblems. After resolving merge issues there are build failures which\nneed to be resolved and additional patches created. The result is\nthat the patch history is a bit chaotic, where there are later patches\nwhich fix problems with early patches. I've tried to sort the fix\npatches to follow the patch they correct, so that the fixes were\ntogether and I could merge them, but that can be difficult.\n\nI've used Stacked Git a little, but don't know if it will make\nany of this easier.\n\nOn some projects, I've reimplemented changes in the upstream repo,\nabandoning the patch history from the old repo:\n\n- Create diff of old repo and upstream. Apply only the changes\n to add new functionality, which are in the patches to the\n old repo. Fix problems caused by API changes, renamed files, etc.\n\n- Re-implement the changes on the upstream repo. Some of the old\n code would be re-used, but modified to fit in the current upstream.\n Some new code would be written.\n\nOne other variant of the rebase approach I've thought of is to do\nthis incrementally, rebasing the old repo against an upstream commit\na short time after the old repo was forked, fixing any conflicts,\nrebuilding and fixing build failures. Then repeat, with a bit\nnewer commit. Then repeat, until I get to the top. This sounds\ntedious, but some of it can be automated. It also might result in\nmy making the changes compatible with upstream code which was later\nabandoned or significantly changed.\n\nAnyone have a different approach that I should consider? Or maybe\noffer advice on how to make one of these approaches work better?\nWhat is best practice to update an old repo?\n\n-- \nMichael Eager\t eager@eagercon.com\n1960 Park Blvd., Palo Alto, CA 94306 650-325-8077\n"},{"id":"322431","messageId":"CAGZ79kZMCz9g7PW6km3gQJU5qipw27jvkVDXD64MUNKJykYZOw@mail.gmail.com","threadId":"46198","inReplyTo":"59432BCC.2040901@eagerm.com","subject":"Re: Best practices for updating old repos","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-06-16T04:22:53Z","receivedAt":"2017-06-16T04:22:59Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Jun 15, 2017 at 5:52 PM, Michael Eager wrote:\n\n> One other variant of the rebase approach I've thought of is to do\n> this incrementally, rebasing the old repo against an upstream commit\n> a short time after the old repo was forked, fixing any conflicts,\n> rebuilding and fixing build failures. Then repeat, with a bit\n> newer commit. Then repeat, until I get to the top. This sounds\n> tedious, but some of it can be automated. It also might result in\n> my making the changes compatible with upstream code which was later\n> abandoned or significantly changed.\n\nThis sounds like\n\nhttps://github.com/mhagger/git-imerge\nhttps://www.youtube.com/watch?v=FMZ2_-Ny_zc\n"},{"id":"322442","messageId":"59437994.4010609@eagerm.com","threadId":"46198","inReplyTo":"CAGUfqurYdFJDT4+XzyPvo3sxeT=zjXqZGCPpDbUFwjqG1B3pBw@mail.gmail.com","subject":"Re: Best practices for updating old repos","fromName":"Michael Eager","fromEmail":"eager@eagerm.com","sentAt":"2017-06-16T06:24:20Z","receivedAt":"2017-06-16T06:24:27Z","isPatch":false,"sender":{"key":"eager@eagerm.com","avatar":null},"body":"Thanks for your comments.\n\nOn 06/15/2017 07:57 PM, Michael O'Cleirigh wrote:\n> Hi Michael,\n>\n> In git if you don't merge often then you get these merge conflict hell situations.\n>\n> In my experience the main conflicts come not from the unified diff of those 130 commits but from\n> differences in the surrounding code.\n>\n> Merging/rebase/cherrypicking directly to the latest upstream sounds impossible to me.\n>\n> These conflicts come from the distance between the local fork branch and the upstream branch.\n>\n> You need to merge through closer commits first to have a hope of getting something automatic to work.\n>\n> Something like getting the list of releases made in the upstream in the last 5 years and merging\n> them in order into the fork branch.\n>\n> i.e. merge v1, merge v2, ... merge v300\n>\n> I went through something similiar with a subversion repo we converted to git.\n>\n> In subversion they were cherry picking done work into a release branch.\n>\n> In git a feature branch mode was being used.\n>\n> It turned out some commits were never cherry picked and bringing them to the latest release was hard.\n>\n> We tried many of the approaches you outlined, took what git would give us automatically and in the\n> most hairy cases recreated the changes on the latest upstream by reading the diff of the original\n> commit and rewriting it on the latest code.\n>\n> In terms of how the history looks after the merge conflicts are resolved you could internalize the\n> fixups into a single commit applied onto the original fork branch. So that history would show the\n> 130 commit branch directly merged into the upstream.\n>\n> You would use the git-commit-tree command to reuse the merged tree id and then use it as a merge\n> commit between the 130th commit id and the upstream commit id.\n>\n> Regards,\n>\n> Michael\n>\n> On Thu, Jun 15, 2017 at 8:52 PM, Michael Eager > wrote:\n>\n> Hi All --\n>\n> I'm working with code that is based on a five year old repository.\n> There are 130 local commits since the repo was forked. Naturally,\n> the upstream project has moved on significantly.\n>\n> I'm wondering about best approaches to updating the repo to the\n> current upstream version. Here are the approaches I've considered:\n>\n> - Rebase from upstream. Likely almost every patch will fail with\n> multiple merge conflicts.\n>\n> - Merge local branch into upstream. Likely many merge failures, but\n> fewer than with rebase.\n>\n> - Apply individual patches from the old repo to the upstream repo.\n> Fix merge conflicts, rebuild, fix build failures. There may be\n> some duplication and additional merge problems created, where a\n> later patch from the old repo fixes the same conflict or build\n> failure.\n>\n> I've tried each of these approaches on various projects. Each has\n> problems. After resolving merge issues there are build failures which\n> need to be resolved and additional patches created. The result is\n> that the patch history is a bit chaotic, where there are later patches\n> which fix problems with early patches. I've tried to sort the fix\n> patches to follow the patch they correct, so that the fixes were\n> together and I could merge them, but that can be difficult.\n>\n> I've used Stacked Git a little, but don't know if it will make\n> any of this easier.\n>\n> On some projects, I've reimplemented changes in the upstream repo,\n> abandoning the patch history from the old repo:\n>\n> - Create diff of old repo and upstream. Apply only the changes\n> to add new functionality, which are in the patches to the\n> old repo. Fix problems caused by API changes, renamed files, etc.\n>\n> - Re-implement the changes on the upstream repo. Some of the old\n> code would be re-used, but modified to fit in the current upstream.\n> Some new code would be written.\n>\n> One other variant of the rebase approach I've thought of is to do\n> this incrementally, rebasing the old repo against an upstream commit\n> a short time after the old repo was forked, fixing any conflicts,\n> rebuilding and fixing build failures. Then repeat, with a bit\n> newer commit. Then repeat, until I get to the top. This sounds\n> tedious, but some of it can be automated. It also might result in\n> my making the changes compatible with upstream code which was later\n> abandoned or significantly changed.\n>\n> Anyone have a different approach that I should consider? Or maybe\n> offer advice on how to make one of these approaches work better?\n> What is best practice to update an old repo?\n>\n> --\n> Michael Eager eager@eagercon.com \n> 1960 Park Blvd., Palo Alto, CA 94306 650-325-8077 \n>\n>\n\n\n-- \nMichael Eager\t eager@eagercon.com\n1960 Park Blvd., Palo Alto, CA 94306 650-325-8077\n"},{"id":"322444","messageId":"59437B68.5090601@eagerm.com","threadId":"46198","inReplyTo":"CAGZ79kZMCz9g7PW6km3gQJU5qipw27jvkVDXD64MUNKJykYZOw@mail.gmail.com","subject":"Re: Best practices for updating old repos","fromName":"Michael Eager","fromEmail":"eager@eagerm.com","sentAt":"2017-06-16T06:32:08Z","receivedAt":"2017-06-16T06:32:13Z","isPatch":false,"sender":{"key":"eager@eagerm.com","avatar":null},"body":"On 06/15/2017 09:22 PM, Stefan Beller wrote:\n> On Thu, Jun 15, 2017 at 5:52 PM, Michael Eager wrote:\n>\n>> One other variant of the rebase approach I've thought of is to do\n>> this incrementally, rebasing the old repo against an upstream commit\n>> a short time after the old repo was forked, fixing any conflicts,\n>> rebuilding and fixing build failures. Then repeat, with a bit\n>> newer commit. Then repeat, until I get to the top. This sounds\n>> tedious, but some of it can be automated. It also might result in\n>> my making the changes compatible with upstream code which was later\n>> abandoned or significantly changed.\n>\n> This sounds like\n>\n> https://github.com/mhagger/git-imerge\n> https://www.youtube.com/watch?v=FMZ2_-Ny_zc\n\nThanks, Stefan. I'll look into this; it may be similar to what\nI was thinking of doing. I'll have to watch the video.\n\n-- \nMichael Eager\t eager@eagercon.com\n1960 Park Blvd., Palo Alto, CA 94306 650-325-8077\n"},{"id":"322447","messageId":"CAGUfqup7aVzoYq-1h=u8ONTT7VvL05x3-Svm1e4d0XfYjnv3sA@mail.gmail.com","threadId":"46198","inReplyTo":"CAGUfqurYdFJDT4+XzyPvo3sxeT=zjXqZGCPpDbUFwjqG1B3pBw@mail.gmail.com","subject":"Fwd: Best practices for updating old repos","fromName":"Michael O'Cleirigh","fromEmail":"michael.ocleirigh@gmail.com","sentAt":"2017-06-16T10:16:54Z","receivedAt":"2017-06-16T10:17:01Z","isPatch":false,"sender":{"key":"michael.ocleirigh@gmail.com","avatar":null},"body":"(Sorry I sent this originally last night in gmail but not in plain\ntext mode and it bounced)\n\n\nHi Michael,\n\nIn git if you don't merge often then you get these merge conflict hell\nsituations.\n\nIn my experience the main conflicts come not from the unified diff of\nthose 130 commits but from differences in the surrounding code.\n\nMerging/rebase/cherrypicking directly to the latest upstream sounds\nimpossible to me.\n\nThese conflicts come from the distance between the local fork branch\nand the upstream branch.\n\nYou need to merge through closer commits first to have a hope of\ngetting something automatic to work.\n\nSomething like getting the list of releases made in the upstream in\nthe last 5 years and merging them in order into the fork branch.\n\ni.e. merge v1, merge v2, ... merge v300\n\nI went through something similiar with a subversion repo we converted to git.\n\nIn subversion they were cherry picking done work into a release branch.\n\nIn git a feature branch mode was being used.\n\nIt turned out some commits were never cherry picked and bringing them\nto the latest release was hard.\n\nWe tried many of the approaches you outlined, took what git would give\nus automatically and in the most hairy cases recreated the changes on\nthe latest upstream by reading the diff of the original commit and\nrewriting it on the latest code.\n\nIn terms of how the history looks after the merge conflicts are\nresolved you could internalize the fixups into a single commit applied\nonto the original fork branch. So that history would show the 130\ncommit branch directly merged into the upstream.\n\nYou would use the git-commit-tree command to reuse the merged tree id\nand then use it as a merge commit between the 130th commit id and the\nupstream commit id.\n\nRegards,\n\nMichael\n\nOn Thu, Jun 15, 2017 at 8:52 PM, Michael Eager wrote:\n>\n> Hi All --\n>\n> I'm working with code that is based on a five year old repository.\n> There are 130 local commits since the repo was forked. Naturally,\n> the upstream project has moved on significantly.\n>\n> I'm wondering about best approaches to updating the repo to the\n> current upstream version. Here are the approaches I've considered:\n>\n> - Rebase from upstream. Likely almost every patch will fail with\n> multiple merge conflicts.\n>\n> - Merge local branch into upstream. Likely many merge failures, but\n> fewer than with rebase.\n>\n> - Apply individual patches from the old repo to the upstream repo.\n> Fix merge conflicts, rebuild, fix build failures. There may be\n> some duplication and additional merge problems created, where a\n> later patch from the old repo fixes the same conflict or build\n> failure.\n>\n> I've tried each of these approaches on various projects. Each has\n> problems. After resolving merge issues there are build failures which\n> need to be resolved and additional patches created. The result is\n> that the patch history is a bit chaotic, where there are later patches\n> which fix problems with early patches. I've tried to sort the fix\n> patches to follow the patch they correct, so that the fixes were\n> together and I could merge them, but that can be difficult.\n>\n> I've used Stacked Git a little, but don't know if it will make\n> any of this easier.\n>\n> On some projects, I've reimplemented changes in the upstream repo,\n> abandoning the patch history from the old repo:\n>\n> - Create diff of old repo and upstream. Apply only the changes\n> to add new functionality, which are in the patches to the\n> old repo. Fix problems caused by API changes, renamed files, etc.\n>\n> - Re-implement the changes on the upstream repo. Some of the old\n> code would be re-used, but modified to fit in the current upstream.\n> Some new code would be written.\n>\n> One other variant of the rebase approach I've thought of is to do\n> this incrementally, rebasing the old repo against an upstream commit\n> a short time after the old repo was forked, fixing any conflicts,\n> rebuilding and fixing build failures. Then repeat, with a bit\n> newer commit. Then repeat, until I get to the top. This sounds\n> tedious, but some of it can be automated. It also might result in\n> my making the changes compatible with upstream code which was later\n> abandoned or significantly changed.\n>\n> Anyone have a different approach that I should consider? Or maybe\n> offer advice on how to make one of these approaches work better?\n> What is best practice to update an old repo?\n>\n> --\n> Michael Eager eager@eagercon.com\n> 1960 Park Blvd., Palo Alto, CA 94306 650-325-8077\n"}]}