{"thread":{"id":"59970","subject":"Beyond Merge and Rebase: The Upstream Import Approach in Git","startedAt":"2023-07-11T08:25:03Z","lastAt":"2023-08-01T09:19:28Z","messageCount":9,"participants":["Aleksander Korzyński","Junio C Hamano","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"479372","messageId":"CADWu+UnThMq2M+kCMADP9rZ5c6nL+Hz+z0-OqRnuG2oYVzbvWw@mail.gmail.com","threadId":"59970","inReplyTo":null,"subject":"Beyond Merge and Rebase: The Upstream Import Approach in Git","fromName":"Aleksander Korzyński","fromEmail":"ak@akorzy.net","sentAt":"2023-07-11T08:24:42Z","receivedAt":"2023-07-11T08:25:03Z","isPatch":false,"sender":{"key":"ak@akorzy.net","avatar":"https://gravatar.com/avatar/e6e5228420ba48122e2fec385efde4bf553448300290cdfd01577c7a39c23a6f?d=mp&s=160"},"body":"Hello,\n\nGit users often have to make a choice: to merge or rebase. I'm going\nto describe a third way that has the characteristics of both and is\nvery well suited for tracking an open-source project or any other\nupstream branch. I'm looking for feedback on the approach.\n\nMERGE OR REBASE?\n\nLet's assume that you have forked an upstream open-source repository\nand keep the fork in your own repo. The default branch of the upstream\nrepository is called \"main\" and is called the same in your own fork.\nYou have made a few changes to the source code and committed them to\nthe \"main\" branch of your fork. In the meantime, new changes have been\ncommitted to the upstream \"main\" branch of the project. How do you\nimport the upstream changes to your fork?\n\nLet's assume that your local fork also contains a branch called\n\"upstream/main\", which reflects the state of the upstream's \"main\"\nbranch. So the \"main\" branch contains your own changes and the\n\"upstream/main\" branch contains the community's changes:\n\n  time -->\n\n  o---o---o---o---o  upstream/main\n       \\\n        o---o---o  main\n\nSo a different way to ask the question is: how do you bring\nupstream/main's changes into main?\n\nOne solution is to merge \"upstream/main\" into \"main\":\n\n  o---o---o---o---o  upstream/main\n       \\           \\\n        o---o---o---M  main\n\nThe merge above would certainly work, but it becomes problematic as\ntime passes and you get a lot of these merges in your \"main\" branch.\nYou then no longer have visibility into the differences between\n\"upstream/main\" and \"main\", because your commits get lost deep in the\nhistory of the branch, as illustrated below:\n\n  o---o---o---o---o---o---o---o---o---o---o  upstream/main\n       \\           \\       \\       \\       \\\n        o---o---o---M---o---M---o---M---o---M  main\n\nSo the alternative solution is to rebase your \"main\" branch on top of\n\"upstream/main\":\n\n  o---o---o---o---o  upstream/main\n                   \\\n                    o'---o'---o'  main\n\nYou now have the advantage of having greater visibility into the\ndifferences between \"upstream/main\" and \"main\". However, a rebase\ncomes with a different problem: if any user of your fork had the\n\"main\" branch checked out in their local repository and they run \"git\npull\", they are going to get an error stating that the local and\nupstream branches have diverged. They will have to take special steps\nto recover from the rebase of the \"main\" branch.\n\nSo how to solve that problem?\n\nTHE THIRD WAY - UPSTREAM IMPORT\n\nThe proposed third way is a special operation that (in the described\nuse case) has the advantages of both a merge and a rebase, without the\ndisadvantages. The approach is illustrated below:\n\n  o---o---o---o---o  upstream/main\n       \\           \\\n        \\           o'---o'---o'\n         \\                     \\\n          o---o---o-------------S  main\n\nFirst, the divergent commits from \"main\" are rebased on top of\n\"upstream/main\", but then they are combined back with \"main\" using a\nspecial merge commit, which has a custom strategy: it replaces the old\ncontent of \"main\" with the new rebased content. This last commit is\nthe secret sauce of this solution: the commit has two parents, like an\nordinary merge, but has the semantics of a rebase.\n\nThe structure above has the advantages of both a merge and a rebase.\nOn the one hand, just like with an ordinary merge, a user who runs\n\"git pull\" on their local copy of \"main\" is not going to see the error\nabout divergent branches. On the other hand, just like with an\nordinary rebase, there is visibility into the last imported commit\nfrom \"upstream/main\" and the differences between that commit and the\ntip of \"main\".\n\nDROPPING PATCHES\n\nWhat is supposed to happen if one of the commits from \"main\" is ported\nto \"upstream/main\", as illustrated below?\n\n  o---o---o---A'---o  upstream/main\n       \\\n        \\\n         \\\n          A---B---C  main\n\nIn that case, the upstream importing operation should drop that patch,\nas illustrated below:\n\n  o---o---o---A'---o  upstream/main\n       \\            \\\n        \\            B'---C'\n         \\                 \\\n          A---B---C---------S  main\n\n\nBut how would the upstream importing operation know which patches to\ndrop? There are one of two ways.\n\nFirstly, it can look at the git's patch-id, which is the SHA of the\nfile changes with line numbers ignored. This is the same strategy that\nrebase uses to drop duplicate commits.\n\nSecondly, it can use an arbitrary change-id associated with a commit\n(for example, for projects that use Gerrit, it can be the Gerrit's\nChange-Id, which is saved in the commit message). This is useful when\na given patch lands upstream in a slightly changed form, but is meant\nto replace the version in \"main\".\n\nIMPLEMENTATION\n\nThe solution above has already been implemented in an open-source\nPython script called git-upstream[1], published 10 years ago. It was\noriginally implemented for the OpenStack project, but the solution is\ngeneric and applicable to any open-source project. It is going to be\neasier for users to benefit from the ideas behind git-upstream if the\nfunctionality is integrated directly into git.\n\nWould you like to see the above functionality integrated directly into git?\n\nBest regards,\nAleksander Korzynski\n\nwww.linkedin.com/in/akorzy\nwww.devopsera.com/blog\n\nP.S.\n\nFor completeness, I'm providing links to alternative solutions for\ntracking patches:\n\n* git-upstream[1] uses the strategy described above\n* quilt[2] uses patch files saved in a source code repository\n* StGit[3] is inspired by quilt and uses git commits to store patches\n* MQ[4] is also inspired by quilt and implements a patch queue in Mercurial\n\n[1] https://opendev.org/x/git-upstream\n[2] https://savannah.nongnu.org/projects/quilt\n[3] https://stacked-git.github.io\n[4] https://wiki.mercurial-scm.org/MqExtension\n"},{"id":"479382","messageId":"xmqqwmz6z8j1.fsf@gitster.g","threadId":"59970","inReplyTo":"CADWu+UnThMq2M+kCMADP9rZ5c6nL+Hz+z0-OqRnuG2oYVzbvWw@mail.gmail.com","subject":"Re: Beyond Merge and Rebase: The Upstream Import Approach in Git","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-11T17:02:10Z","receivedAt":"2023-07-11T17:02:19Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Aleksander Korzyński <ak@akorzy.net> writes:\n\n> So the alternative solution is to rebase your \"main\" branch on top of\n> \"upstream/main\":\n>\n>   o---o---o---o---o  upstream/main\n>                    \\\n>                     o'---o'---o'  main\n>\n> You now have the advantage of having greater visibility into the\n> differences between \"upstream/main\" and \"main\". However, a rebase\n> comes with a different problem: if any user of your fork had the\n> \"main\" branch checked out in their local repository and they run \"git\n> pull\", they are going to get an error stating that the local and\n> upstream branches have diverged. They will have to take special steps\n> to recover from the rebase of the \"main\" branch.\n>\n> So how to solve that problem?\n\nIn short, what you wrote is a way to use rebase but help those who\nhave older versions of your work to bring themselves up to date.\nThat is a useful thing for downstream contributors to have, and it\nis a valuable goal to aim to help these downstream contributors to\ncoordinate sharing of their work.  Because in general, downstream\ncontributors tend to outnumber upstream maintainers.  It would help\nyou to hear perspective from upstream maintainers as well, and here\nare a few things that come to my mind.\n\n    o---o---o---o---o  upstream/main\n         \\           \\\n          \\           a'---b'---c'\n           \\                     \\\n            a---b---c-------------S  main\n\n * It certainly would help folks who received a copy of c from you\n   and then want to observe your progress after you rebased c to c',\n   but how does this help those who have older versions of your\n   work, *and* built their own changes on top?  They would not just\n   need to update their remote-tracking branch that has your older\n   version of the work to the latest, but also rebase their work on\n   top.\n\n    o---o---o---o---o  upstream/main\n         \\           \\\n          \\           a'---b'---c'---d'---e'\n           \\                     \\\n            a---b---c-------------S  main\n                     \\\n                      d---e  your coworker\n\n * It is a reasonable way to for keeping your work as a fork from\n   the upstream up-to-date, but it is unclear what the eventual\n   presentation to and adoption by the upstream would look like.  As\n   an upstream maintainer, for example, I do not want to merge S\n   above to the upstream tree.\n\n * There is no need to say that it is undesirable to merge from\n   upstream to your working topic branch like 'main' repeatedly, as\n   everybody knows it will clutter your history, but more\n   importantly, the resulting history becomes more useless from the\n   upstream's point of view as you have more such reverse merges.\n   The upstream wants to see your work and only your work delineated\n   on your repository.  If you repeat the \"rebase and merge\", then\n   the next round would create a new history a\", b\" and c\" forked on\n   top of an updated upstream/main, merged on top of S, perhaps\n   looking like this:\n\n    o---o---o---o---o---------------o  upstream/main\n         \\           \\               \\\n          \\           a'---b'---c'    a\"--b\"--c\"\n           \\                     \\             \\\n            a---b---c-------------S-------------T  main\n\n   However, once you keep going this way for several rounds, would\n   the result really be much better than bushy history with full of\n   reverse merges from upstream?  Would it help to add new history\n   simplification mechanisms and options to help visualize the\n   history, or do we already have necessary support (e.g. if the\n   convention for these \"merge to cauterize the older versions of\n   history with the newly rebased history\" S and T merges is to\n   record the rebased history as the first-parent, then \"git log\n   --first-parent upstream/main..main\" should be sufficient).  The\n   users would benefit to have an easy way, given only T or S, to\n   get range-diff among (a,b,c) and (a',b',c') and (a\",b\",c\").\n\nWhat is interesting is that, because S and T are essentially \"ours\"\nmerges of your local history into the history that would result if\nyou rebased on top of the upstream (i.e. merge S and T would have\nthe same tree as c' and c\"), what is tested and used by the holder\nof S and T are the changes represented by the latest rebased\nversions of the commits.  So from the upstream point of view,\nthrowing a pull request for c\" (not the original a, b and c) would\nbe a reasonable way to finalize your work.  That way, what you are\noffering to the upstream is not an ancient original commits (i.e. a,\nb, and c) that you haven't been using at all once you created S and\nT.\n\nThanks.\n"},{"id":"479405","messageId":"71f7d816-b139-03ab-88fc-7552d65e4498@gmx.de","threadId":"59970","inReplyTo":"CADWu+UnThMq2M+kCMADP9rZ5c6nL+Hz+z0-OqRnuG2oYVzbvWw@mail.gmail.com","subject":"Re: Beyond Merge and Rebase: The Upstream Import Approach in Git","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2023-07-12T11:34:05Z","receivedAt":"2023-07-12T11:34:13Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Aleksander,\n\nOn Tue, 11 Jul 2023, Aleksander Korzyński wrote:\n\n> THE THIRD WAY - UPSTREAM IMPORT\n>\n> The proposed third way is a special operation that (in the described\n> use case) has the advantages of both a merge and a rebase, without the\n> disadvantages. The approach is illustrated below:\n>\n>   o---o---o---o---o  upstream/main\n>        \\           \\\n>         \\           o'---o'---o'\n>          \\                     \\\n>           o---o---o-------------S  main\n>\n> First, the divergent commits from \"main\" are rebased on top of\n> \"upstream/main\", but then they are combined back with \"main\" using a\n> special merge commit, which has a custom strategy: it replaces the old\n> content of \"main\" with the new rebased content. This last commit is\n> the secret sauce of this solution: the commit has two parents, like an\n> ordinary merge, but has the semantics of a rebase.\n>\n> The structure above has the advantages of both a merge and a rebase.\n> On the one hand, just like with an ordinary merge, a user who runs\n> \"git pull\" on their local copy of \"main\" is not going to see the error\n> about divergent branches. On the other hand, just like with an\n> ordinary rebase, there is visibility into the last imported commit\n> from \"upstream/main\" and the differences between that commit and the\n> tip of \"main\".\n\nI know this strategy well, having used it initially to maintain Git for\nWindows' patches on top of Git releases. I refer to it as `rebasing merge`\nstrategy.\n\nThe main benefit for me was that the patches were always kept in an\n\"upstreamable state\", which incidentally also helped resolving the\nmerge conflicts that occurred by continually rebasing them onto upstream\nreleases.\n\nHowever, I soon realized that the delineation between upstream and\ndownstream patches was unsatisfactory, in particular when new downstream\npatches are added. In the context of the example above, try to find a `git\nrebase` invocation that rebases the current set of downstream patches:\n\n   o---o---o---o---o---o---o---o  upstream/main\n        \\           \\\n         \\           o'---o'---o'\n          \\                     \\\n           o---o---o-------------S---o---o---o  main\n\nA candidate to describe this in a commit range would be\n`upstream/main..main ^S^`, but you cannot pass that to `git rebase -i`,\nwhich expects a single upstream.\n\nSide note: You could _simulate_ this by calling `git replace --graft\nupstream/main upstream/main^ S^` before calling `git rebase -i\nupstream/main`, but I found it really easy to forget to remove the replace\nobject afterwards, and I managed to confuse myself many times before\ndeciding to use replace objects only very rarely.\n\nSo I switched to a different scheme instead that I dub \"merging rebase\".\nInstead of finishing the rebase with a merge, I start it with that merge.\nIn your example, it would look like this:\n\n   o---o---o---o---o  upstream/main\n        \\           \\\n         o---o---o---M---o'---o'---o' main\n\nNaturally, `M` needs to be a merge that _must_ be made with `-s ours` in\norder to be \"tree-same with upstream/main\".\n\nThis strategy was implemented initially in\nhttps://github.com/msysgit/msysgit/commit/95ae63b8c6c0b275f460897c15a44a7df5246dfb\nand is in use to this day:\nhttps://github.com/git-for-windows/build-extra/blob/main/shears.sh\n\nThis strategy is not without problems, though, which becomes quite clear\nwhen you accept PRs that are based on commits prior to the most recent\nmerging rebase (or rebasing merge, both strategies suffer from the same\nproblem): the _next_ merging rebase will not necessarily find the most\nappropriate base commit, in particular when rebasing with\n`--rebase-merges`, causing unnecessary merge conflicts.\n\nThe underlying problem is, of course, the lack of mapping between\npre-rebase and post-rebase versions of the commits: Git has no idea\nthat two commits should be considered identical for the purposes of the\nrebase, even if their SHA-1 differs. And in my hands, the patch ID has\nbeen a poor tool to address this lack of mapping, almost always failing\nfor me. Not even hacked-up `git range-diff` was able to reconstruct the\nmapping reliably enough.\n\nAnd that problem, as far as I can tell, is still unsolved.\n\nThere have been efforts to this end, including\nhttps://lore.kernel.org/git/pull.1356.v2.git.1664981957.gitgitgadget@gmail.com/,\nbut I do not think that any satisfying consensus was reached.\n\nCiao,\nJohannes\n"},{"id":"479409","messageId":"xmqq7cr5uonp.fsf@gitster.g","threadId":"59970","inReplyTo":"71f7d816-b139-03ab-88fc-7552d65e4498@gmx.de","subject":"Re: Beyond Merge and Rebase: The Upstream Import Approach in Git","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-12T15:37:14Z","receivedAt":"2023-07-12T15:37:34Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> I know this strategy well, having used it initially to maintain Git for\n> Windows' patches on top of Git releases. I refer to it as `rebasing merge`\n> strategy.\n\n;-) Thanks. This did look familiar.\n\n>    o---o---o---o---o---o---o---o  upstream/main\n>         \\           \\\n>          \\           o'---o'---o'\n>           \\                     \\\n>            o---o---o-------------S---o---o---o  main\n>\n> A candidate to describe this in a commit range would be\n> `upstream/main..main ^S^`, but you cannot pass that to `git rebase -i`,\n> which expects a single upstream.\n\nIf \"git rebase\" is taught the `--ancestry-path` option and made to\npass it down to the underlying \"which commits do I want to replay\nand in what order\" logic, it would be sufficient to help the above\ntopology, I would think.  But offhand I do not know what other\nrev-list options will become useful in different scenarios.\n\n> So I switched to a different scheme instead that I dub \"merging rebase\".\n> Instead of finishing the rebase with a merge, I start it with that merge.\n> In your example, it would look like this:\n>\n>    o---o---o---o---o  upstream/main\n>         \\           \\\n>          o---o---o---M---o'---o'---o' main\n>\n> Naturally, `M` needs to be a merge that _must_ be made with `-s ours` in\n> order to be \"tree-same with upstream/main\".\n\nAnd this will let you say \"rebase -i upstream/main\" to further\nrebase the most recent round of commits.  That does look quite\nnatural.\n\n> This strategy is not without problems, though, which becomes quite clear\n> when you accept PRs that are based on commits prior to the most recent\n> merging rebase (or rebasing merge, both strategies suffer from the same\n> problem): the _next_ merging rebase will not necessarily find the most\n> appropriate base commit, in particular when rebasing with\n> `--rebase-merges`, causing unnecessary merge conflicts.\n\nEven without rebasing merge or merging rebase, changes, which could\nbe useful if they were not based on a stale base, do happen, and it\nis more effective to have the original authors of these changes to\nupdate them to your most recent tree, than you dealing with them\nyourself, for two reasons.  There are more ICs than you alone, and\nthey are more familiar with their work.\n\nIn other words, isn't the real cause of the above that the workflow\nis not taking advantage of the distributed development?  \"This PR\nseems to solve the right problem, but it is based on an old version\nof the code, please update?\"\n"},{"id":"479420","messageId":"xmqqpm4wswny.fsf@gitster.g","threadId":"59970","inReplyTo":"xmqq7cr5uonp.fsf@gitster.g","subject":"Re: Beyond Merge and Rebase: The Upstream Import Approach in Git","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-12T20:27:13Z","receivedAt":"2023-07-12T20:27:19Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n>> So I switched to a different scheme instead that I dub \"merging rebase\".\n>> Instead of finishing the rebase with a merge, I start it with that merge.\n>> In your example, it would look like this:\n>>\n>>    o---o---o---o---o  upstream/main\n>>         \\           \\\n>>          o---o---o---M---o'---o'---o' main\n> ...\n>> This strategy is not without problems, though, which becomes quite clear\n>> when you accept PRs that are based on commits prior to the most recent\n>> merging rebase (or rebasing merge, both strategies suffer from the same\n>> problem): the _next_ merging rebase will not necessarily find the most\n>> appropriate base commit, in particular when rebasing with\n>> `--rebase-merges`, causing unnecessary merge conflicts.\n\nIn Git, any commit, be it a single parent commit or a merge, makes\nthis statement:\n\n    I considered all the parents of this commit, and it is my belief\n    that it suits the purpose of the branch I am growing better than\n    all of them.\n\nThis is the foundation of the correctness of three-way merges.\nComing from a common ancestor, because M suits the purpose of the\nbranch better than M^1 or M^2, when merging anything forked from M^1\n(or M^2) into a decendant of M (say, 'main'), as long as the\ndescendant of M still shares the same purpose of the branch, it does\nnot need to consider what the commits before M^1 (or M^2) did.\n\nM in the \"merging rebase\", however, claims that M, i.e. the recent\nupstream, fits the purpose of the branch better than the earlier\nthree commits did, which is not quite right.  In contrast, rebasing\nmerge does not have such a problem, i.e.\n\n    o---o---o---o---o  upstream/main\n         \\           \\\n          \\           a'---b'---c'\n           \\                     \\\n            a---b---c-------------M main\n\nThe commit c, a parent of M, implemented the features the topic\nwanted to, and the commit c', another parent of M, implements the\nsame on top of a newer upstream.  The tree of M is the same as c'\nand it matches the purpose, which presumably is to implement\nwhatever (a,b,c) or (a',b',c') wanted to on top of reasonably recent\nupstream, of the branch.\n\nAnyway, I do not think building on top of M would help from this\nstate, so let's stop seeing if there is a way to make rebasing merge\na bit more useful.\n\nThanks.\n\n\n"},{"id":"479457","messageId":"CADWu+Ukgnc241jsYwqorAJ20qeGvqYvyqRbDa4CFXXjiEeDOkw@mail.gmail.com","threadId":"59970","inReplyTo":"xmqqwmz6z8j1.fsf@gitster.g","subject":"Re: Beyond Merge and Rebase: The Upstream Import Approach in Git","fromName":"Aleksander Korzyński","fromEmail":"ak@akorzy.net","sentAt":"2023-07-13T10:55:07Z","receivedAt":"2023-07-13T10:55:27Z","isPatch":false,"sender":{"key":"ak@akorzy.net","avatar":"https://gravatar.com/avatar/e6e5228420ba48122e2fec385efde4bf553448300290cdfd01577c7a39c23a6f?d=mp&s=160"},"body":"Hello Junio,\n\nThanks for the insightful response.\n\nFirstly, let me add some context, which will be helpful for responding\nto your comments. This strategy was created at a large organization,\nwhere the \"main\" branch was used as the integration branch for changes\nmade within the organization and for the organization. So whenever an\nemployee needed to make a change, they would start a topic branch off\n\"main\" and the topic branch would be later merged or rebased onto\n\"main\". When the employee was ready to contribute the change upstream,\nthey would only present the topic branch upstream. The \"main\" branch\nwas never presented upstream. Release branches were created from the\n\"main\" branch to work on releases to production at the organization.\nTags were created on the release branches to mark stable and tested\nversions. Deployments to production were made from the tagged\nversions.\n\nThe rest of my response is inline:\n\n> So from the upstream point of view,\n> throwing a pull request for c\" (not the original a, b and c) would\n> be a reasonable way to finalize your work.  That way, what you are\n> offering to the upstream is not an ancient original commits (i.e. a,\n> b, and c) that you haven't been using at all once you created S and\n> T.\n\nYes, this is how it's intended to be used.\n\n> it is unclear what the eventual\n> presentation to and adoption by the upstream would look like.  As\n> an upstream maintainer, for example, I do not want to merge S\n> above to the upstream tree.\n\nPresentation to upstream would only involve submitting a pull request\nfor c\". Merge S would not be presented to upstream.\n\n> how does this help those who have older versions of your\n> work, *and* built their own changes on top?  They would not just\n> need to update their remote-tracking branch that has your older\n> version of the work to the latest, but also rebase their work on\n> top.\n>\n>     o---o---o---o---o  upstream/main\n>          \\           \\\n>           \\           a'---b'---c'---d'---e'\n>            \\                     \\\n>             a---b---c-------------S  main\n>                      \\\n>                       d---e  your coworker\n\nThe coworker would either merge their topic branch to \"main\" or rebase\nit on top of \"main\":\n\n    o---o---o---o---o  upstream/main\n         \\           \\\n          \\           a'---b'---c'\n           \\                     \\\n            a---b---c-------------S---M  main\n                     \\               /\n                      d-------------e\n\n\n    o---o---o---o---o  upstream/main\n         \\           \\\n          \\           a'---b'---c'\n           \\                     \\\n            a---b---c-------------S---d'---e'  main\n\n> However, once you keep going this way for several rounds, would\n> the result really be much better than bushy history with full of\n> reverse merges from upstream?   Would it help to add new history\n> simplification mechanisms and options to help visualize the\n> history, or do we already have necessary support (e.g. if the\n> convention for these \"merge to cauterize the older versions of\n> history with the newly rebased history\" S and T merges is to\n> record the rebased history as the first-parent, then \"git log\n> --first-parent upstream/main..main\" should be sufficient).\n\nGood point. I believe the proposed method has two advantages over\nusing \"git log --first-parent\":\n\n* consider the scenario where the cauterizing merge is not the only\nmerge in the \"main\" branch - a topic branch from a coworker has also\nbeen merged to \"main\". In that case, \"git log --first-parent\" would\nnot show the commits from the merged topic branch.\n\n* a user unfamiliar with the command-line interface can use a history\nvisualization tool (such as gitk or tig) to obtain a clear view of the\ndifferences between the last imported version of upstream and the tip\nof \"main\".\n\n> The users would benefit to have an easy way, given only T or S, to\n> get range-diff among (a,b,c) and (a',b',c') and (a\",b\",c\").\n\nI like that idea. It would be great to have a simple git command to\ndisplay such a range-diff. The command would have to correctly\nidentify the commits.\n\n--\nBest regards,\nAleksander Korzynski\n"},{"id":"479513","messageId":"CADWu+UmSotSA3Usn=z3C5wBpYp8DWnb0_s1ATv=RKZn+YrERvA@mail.gmail.com","threadId":"59970","inReplyTo":"71f7d816-b139-03ab-88fc-7552d65e4498@gmx.de","subject":"Re: Beyond Merge and Rebase: The Upstream Import Approach in Git","fromName":"Aleksander Korzyński","fromEmail":"ak@akorzy.net","sentAt":"2023-07-14T10:06:26Z","receivedAt":"2023-07-14T10:06:51Z","isPatch":false,"sender":{"key":"ak@akorzy.net","avatar":"https://gravatar.com/avatar/e6e5228420ba48122e2fec385efde4bf553448300290cdfd01577c7a39c23a6f?d=mp&s=160"},"body":"Hi Johannes,\n\nMy responses are inline:\n\n> I know this strategy well, having used it initially to maintain Git for\n> Windows' patches on top of Git releases.\n\nIt's good to know others are using similar ideas :-)\n\n> However, I soon realized that the delineation between upstream and\n> downstream patches was unsatisfactory, in particular when new downstream\n> patches are added. In the context of the example above, try to find a `git\n> rebase` invocation that rebases the current set of downstream patches:\n>\n>    o---o---o---o---o---o---o---o  upstream/main\n>         \\           \\\n>          \\           o'---o'---o'\n>           \\                     \\\n>            o---o---o-------------S---o---o---o  main\n\nWe have solved that problem with custom scripting. The git-upstream[1]\ntool properly rebases the commits in that case. This is one of the\nreasons why I would like to see the git-upstream functionality\nreimplemented in git itself. With git today, you can't achieve that\nwith a single `git rebase` command, but you can with a series of\ncommands. Introducing a new command or switch to git would allow us to\nperform that operation with a single command.\n\nLet's recap what the automation needs to do. Assume the following situation:\n\n  o---o---o---o---x---o---o---t  tag=v1.2.3  branch=upstream/main\n       \\           \\\n        \\           a'---b'---c'\n         \\                     \\\n          a---b---c-------------S---d---e  branch=main\n\nLet's assume the user has the \"main\" branch checked out and they want\nto import the latest tag from the \"upstream/main\" branch. The commands\nthey run are:\n\ngit checkout main\ngit upstream import v1.2.3\n\nThe automation should now perform the following:\n\n* create a new branch \"import/v1.2.3\" starting from tag \"v1.2.3\"\n* rebase a', b', c' onto \"import/v1.2.3\"\n* rebase d, e onto \"import/v1.2.3\"\n* perform the cauterizing merge of \"import/v1.2.3\" to \"main\"\n\nHere are important observations:\n\n* the first rebase operates on commits present on the main branch,\nstarting from the first commit after x, ending with the last commit\nbefore S\n* the second rebase operates on commits present on the main branch,\nstarting from the first commit after S, ending with the tip of main\n\nSo the problem boils down to identifying commits x and S. Once we\nidentify these commits, we can perform the rebases.\n\nTo identify x we need to find the most recent common ancestor of\n\"main\" and \"v1.2.3\". To identify S we need to iterate over branch\n\"main\" starting from x and forward in time until we find the first\nmerge. That's the logic that needs to be implemented. If that logic\nwas available under a single command or switch in git, we'd be able to\nperform the upstream import operation without a helper script such as\ngit-upstream.\n\n> This strategy is not without problems, though, which becomes quite clear\n> when you accept PRs that are based on commits prior to the most recent\n> merging rebase (or rebasing merge, both strategies suffer from the same\n> problem): the _next_ merging rebase will not necessarily find the most\n> appropriate base commit, in particular when rebasing with\n> `--rebase-merges`, causing unnecessary merge conflicts.\n\nThis can also be solved with custom logic. Let's consider the scenario\nin detail:\n\n  o---o---o---o---x---o---o---t  tag=v1.2.3  branch=upstream/main\n       \\           \\\n        \\           a'---b'\n         \\                \\\n          a---b------------S---c---M---f  branch=main\n               \\                  /\n                d----------------e  branch=topic\n\nAs before, the user runs the following commands:\n\ngit checkout main\ngit upstream import v1.2.3\n\nIn this case, the automation should rebase d and e between c and f:\n\n  o---o---o---o---x---o---o---t  tag=v1.2.3  branch=upstream/main\n       \\           \\           \\\n        \\           a'---b'     a\"--b\"--c\"--d\"--e\"--f\"\n         \\                \\                          \\\n          a---b------------S---c---M---f--------------S'  branch=main\n               \\                  /\n                d----------------e  branch=topic\n\nThis logic can be implemented as follows. When the automation reaches\nthe merge commit M, it finds the second parent e and then searches for\nthe most recent common ancestor of e and main, so that it finds b. The\nrebase then operates on commits starting from the first commit after b\nand ending with the second parent of M.\n\nThe logic above could also be incorporated into git.\n\n> The underlying problem is, of course, the lack of mapping between\n> pre-rebase and post-rebase versions of the commits: Git has no idea\n> that two commits should be considered identical for the purposes of the\n> rebase, even if their SHA-1 differs. And in my hands, the patch ID has\n> been a poor tool to address this lack of mapping, almost always failing\n> for me. Not even hacked-up `git range-diff` was able to reconstruct the\n> mapping reliably enough.\n>\n> And that problem, as far as I can tell, is still unsolved.\n\nAs shown above, we don't actually need to be able to map pre-rebase\nand post-rebase versions of the commits in order to correctly perform\nthe \"git upstream import\" operation. The \"git-upstream\" helper script\nis a working implementation of the strategy without doing the mapping.\n\nThat being said, being able to map pre-rebase and post-rebase versions\nof the commits is useful for something else: dropping patches that\nhave been incorporated upstream. The \"git-upstream\" script utilizes\ntwo strategies for that purpose. One of them is to use patch-id. The\nother one is to use an arbitrary identifier that you attach to the\ncommit both in the \"main\" and \"upstream/main\" branches. In our case,\nwe have used the Gerrit's Change-Id as the identifier, but it could be\nsomething else. The Gerrit's Change-Id is just a random string added\nto the bottom of a commit message by a git commit hook.\n\n> So I switched to a different scheme instead that I dub \"merging rebase\".\n> Instead of finishing the rebase with a merge, I start it with that merge.\n> In your example, it would look like this:\n>\n>    o---o---o---o---o  upstream/main\n>         \\           \\\n>          o---o---o---M---o'---o'---o' main\n\nI like Junio's word \"cauterize\" to describe the special merge :-) So\nI'm going to call this strategy \"cauterize & rebase\" and the strategy\nI described in the initial email \"rebase & cauterize\".\n\nWe have also considered \"cauterize & rebase\" instead of \"rebase &\ncauterize\" and the reason we opted for the latter was peer review in\nGerrit. When we rebase first, we can store the rebased commits on a\ntemporary import branch and push the import branch to a shared\nrepository. The import branch then contains everything except for the\nlast cauterizing merge. We then need to push only the cauterizing\nmerge into the Gerrit review system. The reviewer then only has to\napprove the cauterizing merge to approve the entire \"upstream import\"\nstructure. We didn't need to make any changes to the Gerrit review\nsystem to utilize it in that way. These considerations may not apply\nto other review systems.\n\n> This strategy was implemented initially in\n> https://github.com/msysgit/msysgit/commit/95ae63b8c6c0b275f460897c15a44a7df5246dfb\n> and is in use to this day:\n> https://github.com/git-for-windows/build-extra/blob/main/shears.sh\n> (...)\n> https://lore.kernel.org/git/pull.1356.v2.git.1664981957.gitgitgadget@gmail.com/\n\nThanks for the links, they are useful :-)\n\nWith the content of this email in mind, what are your thoughts? Would\nyou like to see the strategy becoming a first-class feature in git?\n\nBest regards,\nAleksander Korzynski\n\n[1] https://opendev.org/x/git-upstream\n"},{"id":"479515","messageId":"CADWu+UnvastFbWsjfHvJhvS1RBgD8M1LXuA2VMBMSkTqpiLS5w@mail.gmail.com","threadId":"59970","inReplyTo":"xmqqpm4wswny.fsf@gitster.g","subject":"Re: Beyond Merge and Rebase: The Upstream Import Approach in Git","fromName":"Aleksander Korzyński","fromEmail":"ak@akorzy.net","sentAt":"2023-07-14T10:56:51Z","receivedAt":"2023-07-14T10:58:52Z","isPatch":false,"sender":{"key":"ak@akorzy.net","avatar":"https://gravatar.com/avatar/e6e5228420ba48122e2fec385efde4bf553448300290cdfd01577c7a39c23a6f?d=mp&s=160"},"body":"> it is more effective to have the original authors of these changes to\n> update them to your most recent tree, than you dealing with them\n> yourself, for two reasons.  There are more ICs than you alone, and\n> they are more familiar with their work.\n>\n> In other words, isn't the real cause of the above that the workflow\n> is not taking advantage of the distributed development?  \"This PR\n> seems to solve the right problem, but it is based on an old version\n> of the code, please update?\"\n\nThat's a valid point. Let me describe how it used to work.\n\nWe tracked a busy project (OpenStack Nova), which used to have as many\nas 50 commits per day. At night we used to run an automated job that\nwould attempt to import the latest upstream (rebase and cauterize),\ndeploy to a test environment and test it with our configuration. It\nwould have been impractically slow to require the developer of every\ninternal patch to manually update to the latest version of the code,\nbefore deploying and testing. Also, it wouldn't be acceptable to an\nenterprise to always require the original author to rebase their\npatch, because the author may be on holiday or may have left the\ncompany, but the business has to move on.\n\nIn the morning, we used to check if the automated job returned green\nor red. In practice, however, most of the time our patches would\ncleanly rebase automatically without manual intervention. That was\nbecause of the way we used to write the patches - changing as few\nlines as possible. Also, patches were typically only temporary in\nnature, as they were eventually contributed to the open-source\nupstream project.\n\nIf the automated job returned red, there was a designated engineer who\nwould investigate the issue on a given day. They would try to rebase\nthe patches themselves and fix any issues. If they had any questions\nor concerns they would contact the original author, as long as the\noriginal author was at work. Most of the time contacting the original\nauthor wasn't necessary.\n\n> In Git, any commit, be it a single parent commit or a merge, makes\n> this statement:\n>\n>     I considered all the parents of this commit, and it is my belief\n>     that it suits the purpose of the branch I am growing better than\n>     all of them.\n> (...)\n> M in the \"merging rebase\", however, claims that M, i.e. the recent\n> upstream, fits the purpose of the branch better than the earlier\n> three commits did, which is not quite right.  In contrast, rebasing\n> merge does not have such a problem, i.e.\n>\n>     o---o---o---o---o  upstream/main\n>          \\           \\\n>           \\           a'---b'---c'\n>            \\                     \\\n>             a---b---c-------------M main\n\nI second that observation.\n\nAny other comments? :-)\n\n--\nBest regards,\nAleksander Korzynski\n"},{"id":"480037","messageId":"CADWu+UmjFiKa2h4qqumA-jOfEGvVnsV85bOTKgYhpQwYCf=VFw@mail.gmail.com","threadId":"59970","inReplyTo":"CADWu+UnvastFbWsjfHvJhvS1RBgD8M1LXuA2VMBMSkTqpiLS5w@mail.gmail.com","subject":"Re: Beyond Merge and Rebase: The Upstream Import Approach in Git","fromName":"Aleksander Korzyński","fromEmail":"ak@akorzy.net","sentAt":"2023-08-01T09:17:06Z","receivedAt":"2023-08-01T09:19:28Z","isPatch":false,"sender":{"key":"ak@akorzy.net","avatar":"https://gravatar.com/avatar/e6e5228420ba48122e2fec385efde4bf553448300290cdfd01577c7a39c23a6f?d=mp&s=160"},"body":"Hi,\n\nThanks again for your responses, Junio and Johannes. I'm looking to\nimplement the discussed structure in git. As the first step, I'd like\nto implement:\n\ngit merge -s theirs\n\nThe name of the `theirs` strategy above is inspired by the existing\n`ours` strategy. The command above is going to be the equivalent of\nthe following three commands:\n\ngit merge --no-commit -s ours <commit>\ngit read-tree --reset -u <commit>\ngit commit --no-edit\n\nThe new command is going to be used to create the last \"welding merge\"\nat the end of the structure below:\n\n     o---o---o---o---o  upstream/main\n          \\           \\\n           \\           a'---b'---c'\n            \\                     \\\n             a---b---c-------------M main\n\nThe strategy above could be called \"rebase & weld\". The new command\ncan also be used with the \"weld & rebase\" strategy described by\nJohannes:\n\n   o---o---o---o---o  upstream/main\n        \\           \\\n         a---b---c---M---a'---b'---c' main\n\nIn addition, `git merge -s theirs` could be called `git weld`, which\nwould make it shorter to type. What do you think?\n\nAlso, I'm thinking about the eventual interface for creating the\nentire structure. Perhaps \"rebase & weld\" could be created with the\nfollowing command:\n\ngit rebase --weld\n\nand \"weld & rebase\" with the following:\n\ngit rebase --pre-weld\n\nWhat are your thoughts?\n\n--\nBest regards,\nAleksander Korzynski\n"}]}