{"thread":{"id":"42467","subject":"RFC: new git-splice subcommand for non-interactive branch splicing","startedAt":"2016-05-27T14:08:11Z","lastAt":"2016-05-30T00:34:48Z","messageCount":6,"participants":["Adam Spiers","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"287676","messageId":"20160527140811.GB11256@pacific.linksys.moosehall","threadId":"42467","inReplyTo":null,"subject":"RFC: new git-splice subcommand for non-interactive branch splicing","fromName":"Adam Spiers","fromEmail":"git@adamspiers.org","sentAt":"2016-05-27T14:08:11Z","receivedAt":"2016-05-27T14:08:11Z","isPatch":false,"sender":{"key":"git@adamspiers.org","avatar":"https://avatars.githubusercontent.com/u/100738?v=4"},"body":"Hi all,\n\nI finally got around to implementing a new git subcommand which I've\nwanted for quite a while.  I've called it git-splice.\n\nDescription\n-----------\n\ngit-splice(1) non-interactively splices the current branch by removing\na range of commits from within it and/or cherry-picking a range of\ncommits into it.  It's essentially just a glorified wrapper around\ncherry-pick and rebase -i.\n\nUsage\n-----\n\nExamples:\n\n    # Remove commit A from the current branch\n    git splice A^!\n\n    # Remove commits A..B from the current branch\n    git splice A..B\n\n    # Remove commits A..B from the current branch, and cherry-pick\n    # commits C..D at the same point\n    git splice A..B C..D\n\n    # Cherry-pick commits C..D, splicing them in just after commit A\n    git splice A C..D\n\n    # Remove first commit mentioning 'foo', and insert all commits\n    # in the 'elsewhere' branch which mention 'bar'\n    git splice --grep=foo -n1 HEAD -- --grep=bar HEAD..elsewhere\n\n    # Abort a splice which failed during cherry-pick or rebase\n    git splice --abort\n\n    # Resume a splice after manually fixing conflicts caused by\n    # cherry-pick or rebase\n    git splice --continue\n\nN.B. Obviously this command rewrites history!  As with git rebase,\nyou should be aware of all the implications of history rewriting\nbefore using it.\n\nCode\n----\n\nCurrently this is in alpha state:\n\n  https://github.com/git/git/compare/master...aspiers:splice\n\nand I reserve the right to rewrite the history of that branch in the\nnear future ;-)\n\nI realise that the code does not yet conform to the coding standards\nof the git project.  For example, it relies on non-POSIX bash\nfeatures, like arrays.  I would be happy to fix this if there is a\nchance git-splice might be accepted for inclusion within the git\ndistribution.  (Presumably contrib/ is another possibility.)\nAlso, I haven't yet written a proper man page for it.\n\nMotivation\n----------\n\nI wrote git-splice as the next step in the journey towards being able\nto implement a tool which automatically (or at least\nsemi-automatically) splits a linear sequence of commits into a commit\ngraph where ancestry exactly mirrors commit \"dependency\".[0]  In other\nwords, in this commit graph, a commit B would have commit A as an\nancestor if and *only* if commit B cannot cleanly apply without A\nalready being present in the branch.  As a corollary, if commit F\ndepends on D and E, but D and E are mutually independent, F would\nneed to depend on a merge commit which contains D and E.\n\nSuch a tool could be useful for a few reasons.  Firstly, large patch\nseries are much harder to review than single commits or small patch\nseries, but typical development workflows often lead to large patch\nseries.\n\nFor example, if I work privately on a new feature for some hours /\ndays / weeks, I will typically amass a bunch of commits which are not\nall directly related to the new feature: there are often refactorings,\nfixes for bugs discovered during development of the new feature, etc.\n\nI doubt I'm the only git user not disciplined enough to maintain neat\nbranch organization for the whole of a long period of hacking!\ni.e. religiously maintaining one branch per bugfix, one branch per\nrefactoring, and one branch for the new feature.[1]  Typically, tidying\nup the branches comes a bit later, when I want to start feeding stuff\nupstream for review.\n\nTherefore being able to reduce the effort involved with breaking a\nlarge patch series into smaller related chunks seems potentially very\nuseful.\n\nAs well as making reviews smaller easier, this allows both the reviews\nand any corresponding CI to proceed in a more parallelized fashion.\n\nSome review systems can implicitly discourage reviews of large patch\nseries, by treating each commit as a review in its own right and/or not\nproviding sophisticated support for patch series.  Gerrit is one\nexample; gitlab and GitHub are counter-examples.\n\nI'm sure there are other use cases which I didn't think of yet.\n\nNext steps, and the future\n--------------------------\n\nObviously, I'd welcome thoughts on whether it would make sense to\ninclude this in the git distribution.\n\nIn the longer term however, I'd like to write two more subcommands:\n\n  - git-transplant(1) which wraps around git-splice(1) and enables\n    easy non-interactive transplanting of a range of commits from\n    one branch to another.  This should be pretty straightforward\n    to implement.\n\n  - git-explode(1) which wraps around git-transplant(1) and\n    git-deps(1), and automatically breaks a linear sequence of commits\n    into multiple smaller sequences, forming a commit graph where\n    ancestry mirrors commit dependency, as mentioned above.  I expect\n    this to be more difficult, and would probably write it in Python.\n\n    Ideally, this tool would also be able to integrate with other\n    workflow management tools[1] in order to effectively create /\n    manage topic branches and track dependencies between them.\n\nEventually, the utopia I'm dreaming about would become a reality and\nlook something like this:\n\n    git checkout -b new-feature\n\n    while in_long_frenzied_period_of_hacking; do\n        # don't worry too much about branch maintenance here, just hack\n        git add ...\n        git commit ...\n    done\n\n    # Break lots of commits from new-feature into new topic branches:\n    git explode\n\n    # List topic branches\n    git work list\n\n    # Manually complete tidy-up of those branches\n    git push ...\n    git send-email ...\n\n\nFeedback on any of this is very welcome!\n\nThanks,\nAdam\n\n\n[0] https://github.com/aspiers/git-deps/#user-content-use-case-2-splitting-a-patch-series\n\n    This type of dependency could be described as textual or syntactic or\n    lexical, and is automatically detected by git-deps:\n\n        https://github.com/aspiers/git-deps/\n\n    which I wrote a couple of years ago and previously announced on this list:\n\n        http://thread.gmane.org/gmane.comp.version-control.git/262000/focus=262606\n\n    Of course, this is a somewhat naive approach in that it has no\n    awareness of semantic dependencies, e.g. commit A changing file X\n    in a way which only makes sense if commit B changing file Y is\n    already present.  However in my experience it's still a useful\n    start in the right direction, saving a lot of time by detecting\n    the \"obvious\" dependencies, and often revealing dependencies which\n    I would have otherwise missed.\n\n[1] There are tools which can help with this, e.g. topgit, git-flow,\n    and gitwork, which IMHO is particularly interesting.\n"},{"id":"287682","messageId":"alpine.DEB.2.20.1605271701500.4449@virtualbox","threadId":"42467","inReplyTo":"20160527140811.GB11256@pacific.linksys.moosehall","subject":"Re: RFC: new git-splice subcommand for non-interactive branch splicing","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-05-27T15:27:14Z","receivedAt":"2016-05-27T15:27:14Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Adam,\n\nOn Fri, 27 May 2016, Adam Spiers wrote:\n\n> Description\n> -----------\n> \n> git-splice(1) non-interactively splices the current branch by removing\n> a range of commits from within it and/or cherry-picking a range of\n> commits into it.  It's essentially just a glorified wrapper around\n> cherry-pick and rebase -i.\n\nIt sounds as if you could accomplish the same with\n\n\tgit checkout -b former-commits <split>\n\tgit checkout -b latter-commits <base>\n\tgit cherry-pick <split>..HEAD@{2}\n\n> Next steps, and the future\n> --------------------------\n> \n> Obviously, I'd welcome thoughts on whether it would make sense to\n> include this in the git distribution.\n\nFar be I from discouraging you to work on these scripts, but I think that\na really good place for such subcommands is a separate repository, as you\nhave it already. There are already some rarely used subcommands in\nlibexec/git-core/ cluttering up the space and I would be reluctant to add\neven more subcommands to the default Git installation delivered to every\nuser.\n\nYou can *always* just extend the PATH so that git-splice can be found;\nThen `git splice ...` will do exactly what you want. That is e.g. how\ngit-flow works. (Of course I hope that you will maintain your scripts\nmuch, much better than git-flow, i.e. not abandon all users).\n\n> In the longer term however, I'd like to write two more subcommands:\n> \n>   - git-transplant(1) which wraps around git-splice(1) and enables\n>     easy non-interactive transplanting of a range of commits from\n>     one branch to another.  This should be pretty straightforward\n>     to implement.\n\nThis is just cherry-pick with a range...\n\n>   - git-explode(1) which wraps around git-transplant(1) and\n>     git-deps(1), and automatically breaks a linear sequence of commits\n>     into multiple smaller sequences, forming a commit graph where\n>     ancestry mirrors commit dependency, as mentioned above.  I expect\n>     this to be more difficult, and would probably write it in Python.\n\nYou mean something like Darcs on top of Git. Essentially, you want to end\nup with an octopus merge of branches whose commits would conflict if\nexchanged.\n\nI implemented the logic for this in a shell script somewhere, so it is not\n*all* that hard (Python not required). But I ended up never quite using it\nbecause it turns out that in practice, the commit \"dependency\" (as defined\nby the commit diffs) does not really reflect the true dependency.\n\nFor example, in my work to move large parts of rebase -i into a builtin, I\nhave an entire series of commits that do nothing else but prepare the\nsequencer for rebase -i's functionality. Most of these commits touch\ncompletely separate parts of the code, so they would make independent\nbranches in your git-explode command. Yet, that would destroy the story\nthat the patch series tells, as the natural flow would get lost.\n\nAnother major complication is that sometimes the \"dependency as per the\ndiff\" is totally bogus. Take for example Git for Windows' patches on top\nof Git: there are a couple \"sub branches\" that add global variables to\nenvironment.c. By the logic of the overlapping (or touching) hunks, these\nsub branches should build on top of each other, right? But they are\nlogically completely independent.\n\nSo I think that this is a nice exercise, but in practice it will require a\nhuman to determine which commits really depend on each other.\n\n> Eventually, the utopia I'm dreaming about would become a reality and\n> look something like this:\n> \n>     git checkout -b new-feature\n> \n>     while in_long_frenzied_period_of_hacking; do\n>         # don't worry too much about branch maintenance here, just hack\n>         git add ...\n>         git commit ...\n>     done\n> \n>     # Break lots of commits from new-feature into new topic branches:\n>     git explode\n> \n>     # List topic branches\n>     git work list\n\nYou would render me *really* impressed if you could come up with an\nautomated way to determine logical dependencies between patches.\n\nCiao,\nJohannes\n"},{"id":"287688","messageId":"20160527163652.GC11256@pacific.linksys.moosehall","threadId":"42467","inReplyTo":"alpine.DEB.2.20.1605271701500.4449@virtualbox","subject":"Re: RFC: new git-splice subcommand for non-interactive branch splicing","fromName":"Adam Spiers","fromEmail":"git@adamspiers.org","sentAt":"2016-05-27T16:36:52Z","receivedAt":"2016-05-27T16:36:52Z","isPatch":false,"sender":{"key":"git@adamspiers.org","avatar":"https://avatars.githubusercontent.com/u/100738?v=4"},"body":"Hi Johannes,\n\nThanks for the quick reply!  Responses inline below:\n\nOn Fri, May 27, 2016 at 05:27:14PM +0200, Johannes Schindelin wrote:\n> On Fri, 27 May 2016, Adam Spiers wrote:\n> \n> > Description\n> > -----------\n> > \n> > git-splice(1) non-interactively splices the current branch by removing\n> > a range of commits from within it and/or cherry-picking a range of\n> > commits into it.  It's essentially just a glorified wrapper around\n> > cherry-pick and rebase -i.\n> \n> It sounds as if you could accomplish the same with\n> \n>       git checkout -b former-commits <split>\n>       git checkout -b latter-commits <base>\n>       git cherry-pick <split>..HEAD@{2}\n\nNot really - that is missing several features which git-splice\nprovides, e.g.\n\n  - The ability to remove a non-consecutive list of commits\n    from the branch.\n\n  - The ability to insert commits at the same time as removing\n    (granted, that's just in extra cherry-pick your method, but again\n    that's another thing to orchestrate).\n\n  - The ability to specify commits to remove / insert using\n    arguments understood by git-rev-list.\n\n  - The patch-id magic which is built into git-rebase.  This\n    would kick in if any of the commits to insert are already\n    in <split>..HEAD@{2} (using your reference terminology).\n\n  - A single command to orchestrate the whole workflow, including\n    cleanup, and --abort and --continue when manual conflict\n    resolution is required.  This modularity should help a lot when\n    building further tools which wrap around it in order to perform\n    more complex tasks.\n\nThis last point is perhaps the most important.  Of course it's\npossible to do this manually already.  But the whole point of\ngit-splice is to automate it in a convenient and reliable manner.\n\n> > Next steps, and the future\n> > --------------------------\n> > \n> > Obviously, I'd welcome thoughts on whether it would make sense to\n> > include this in the git distribution.\n> \n> Far be I from discouraging you to work on these scripts, but I think that\n> a really good place for such subcommands is a separate repository, as you\n> have it already. There are already some rarely used subcommands in\n> libexec/git-core/ cluttering up the space and I would be reluctant to add\n> even more subcommands to the default Git installation delivered to every\n> user.\n\nSure, I appreciate the difficulty in deciding where to draw the line.\nMy feeling is that rebase -i provides something tremendously\nimportant, which the vast majority of users use on a regular basis,\nbut that git is currently missing a convenient way to\n*non-interactively* perform the same magic which rebase -i\nfacilitates.  And removing / reordering commits is surely one of the\nmost common use cases of rebase -i, so I think a lot of people could\nbenefit from some porcelain to automate that and allow building\nhigher-level tools on top of it.\n\nI suspect the most popular use-case in the short term would be the\ninfamous \"oops, I only just noticed that I put that commit on the\nwrong branch, and now there's already a whole bunch of other commits\non top of it\".  I would expect that reducing this solution to a single\ngit-transplant(1) command would be pretty attractive for a lot of\npeople.  And of course GUIs / IDEs could incorporate it into their\nmore beautiful front-ends.  However, if it's not in git core, that's\nunlikely to happen.\n\n> You can *always* just extend the PATH so that git-splice can be found;\n> Then `git splice ...` will do exactly what you want. That is e.g. how\n> git-flow works.\n\nSure, I've been using that trick since at least 2009 ;-) [0]\n\n> (Of course I hope that you will maintain your scripts\n> much, much better than git-flow, i.e. not abandon all users).\n\nI hope so too ;-)\n\n> > In the longer term however, I'd like to write two more subcommands:\n> > \n> >   - git-transplant(1) which wraps around git-splice(1) and enables\n> >     easy non-interactive transplanting of a range of commits from\n> >     one branch to another.  This should be pretty straightforward\n> >     to implement.\n> \n> This is just cherry-pick with a range...\n\nNo it's not:\n\n  - git-transplant would be able to splice commits from one branch\n    *into* (i.e. inside, *not* onto) another branch.\n\n  - git-transplant would also take care of removing the commits from\n    the source branch, but not before they were safely inside the\n    destination branch.\n\n  - git-transplant would orchestrate the whole workflow with a single\n    command, complete with --abort and --continue.\n\n> >   - git-explode(1) which wraps around git-transplant(1) and\n> >     git-deps(1), and automatically breaks a linear sequence of commits\n> >     into multiple smaller sequences, forming a commit graph where\n> >     ancestry mirrors commit dependency, as mentioned above.  I expect\n> >     this to be more difficult, and would probably write it in Python.\n> \n> You mean something like Darcs on top of Git. Essentially, you want to end\n> up with an octopus merge of branches whose commits would conflict if\n> exchanged.\n\nSomething like that, yes, but it's not as simple as a single octopus\nmerge.  It would support arbitrarily deep DAGs of topic branches.\n\n> I implemented the logic for this in a shell script somewhere, so it is not\n> *all* that hard (Python not required). But I ended up never quite using it\n> because it turns out that in practice, the commit \"dependency\" (as defined\n> by the commit diffs) does not really reflect the true dependency.\n>\n> For example,\n\n[snipped examples]\n\nSure - I already covered this concern in footnote [0] of my previous\nmail; maybe you missed that?  As I said there, in my experience, I\nhave found it very useful to be able to automatically detect textual\ndependencies via git-deps, even though they do not represent the\nentire set of dependencies.  I've even recorded a YouTube screencast\ndemonstrating one such use case[1].  So please don't let the question\nfor perfection become the enemy of the good ;-)\n\n> So I think that this is a nice exercise, but in practice it will require a\n> human to determine which commits really depend on each other.\n\nOf course - this is exactly why I wrote \"or at least\nsemi-automatically\" in the first mail of this thread.  But even though\ngit-deps / git-explode can never automatically handle *all*\ndependencies, they can handle enough dependencies to be significantly\nuseful.  I have concrete real-world experience of that.  In fact there\nis one scenario I am working on right now which is current proof of\nthis (no coincidence, since that's what motivated me to take this next\nstep on this journey and write git-splice):\n\nI have made a large bunch of small commits to a single text file\n(design document).  Some are possibly contentious; some aren't.  So I\nneed to split them out into a series of smaller independent patch\nseries which I can submit to gerrit for review, thereby making life\neasier for the reviewers and minimizing any bottlenecks where reviews\nfor one change are blocked because another change hasn't been reviewed\nyet.  And in this case, because the changes are all applying to a\nsingle file containing only natural language, git-deps correctly\ndetermines *all* dependencies, not just textual ones.\n\n> You would render me *really* impressed if you could come up with an\n> automated way to determine logical dependencies between patches.\n\nHey, I would *really* impress myself if I could do that, too; after\nall, that would be a pretty sophisticated form of artificial\nintelligence :-)\n\nThanks again for the feedback!\n\nAdam\n\n\n[0] https://github.com/aspiers/git-config/commit/287685408326\n\n    BTW there are currently 36 git-* scripts in that repo; you may or\n    may not find it interesting to browse through them.\n\n[1] https://github.com/aspiers/git-deps#use-case-1-porting-between-branches\n\n    In this case, git-deps can serve as an early warning system to\n    flag when a backporting task's cost:reward ratio would be too\n    high to justify starting work on it.\n"},{"id":"287725","messageId":"alpine.DEB.2.20.1605280841000.4449@virtualbox","threadId":"42467","inReplyTo":"20160527163652.GC11256@pacific.linksys.moosehall","subject":"Re: RFC: new git-splice subcommand for non-interactive branch splicing","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-05-28T07:06:59Z","receivedAt":"2016-05-28T07:06:59Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Adam,\n\nplease reply-to-all on this list.\n\nOn Fri, 27 May 2016, Adam Spiers wrote:\n\n> My feeling is that rebase -i provides something tremendously\n> important, which the vast majority of users use on a regular basis,\n> but that git is currently missing a convenient way to\n> *non-interactively* perform the same magic which rebase -i\n> facilitates.\n\nWould it not make sense to enhance `rebase -i`, then?\n\nI, for one, plan to port my Git garden shears (at least partially) once I\nmanaged to get my rebase--helper work in. The shears script is kind of a\n\"--preseve-merges done right\":\n\n\thttps://github.com/git-for-windows/build-extra/blob/master/shears.sh\n\nIt allows you to (re-)create a branch structure like this:\n\n\tbud\n\tpick cafebabe XYZ\n\tpick 01234567 Another patch\n\tmark the-first-branch\n\n\tbud\n\tpick 98765432 This patch was unrelated\n\tmark the-second-branch\n\n\tbud\n\tmerge the-first-branch\n\tmerge the-second-branch\n\nOf course, this is interactive. But quite frankly, you want to be able to\nperform quite complicated stuff, and I think the command-line offers only\nan inadequate interface for this.\n\n> I suspect the most popular use-case in the short term would be the\n> infamous \"oops, I only just noticed that I put that commit on the\n> wrong branch, and now there's already a whole bunch of other commits\n> on top of it\".\n\nI have two workflows for that. The simpler one:\n\n\tgit checkout other-branch\n\tgit commit\n\tgit checkout @{-1}\n\nSometimes I need to call `git stash -p` before, and `git stash apply`\nafter those calls.\n\nThe more complicated one comes in handy when a complete rebuild takes a\nlong time, and branch switching would trigger a rebuild:\n\n\t# Here, I stash what I *want* on the other branch\n\tgit stash -p\n\tgit worktree add throwaway other-branch\n\tcd throwaway\n\tgit stash apply\n\tgit commit\n\nI did use the approach you proposed a couple of times: just commit in the\nmiddle, and sort things out later. The problem: I frequently forgot, and\nif I did not, reordering the commits resulted in stupid and avoidable\nmerge conflicts.\n\n> > > In the longer term however, I'd like to write two more subcommands:\n> > > \n> > >   - git-transplant(1) which wraps around git-splice(1) and enables\n> > >     easy non-interactive transplanting of a range of commits from\n> > >     one branch to another.  This should be pretty straightforward\n> > >     to implement.\n> > \n> > This is just cherry-pick with a range...\n> \n> No it's not:\n> \n>   - git-transplant would be able to splice commits from one branch\n>     *into* (i.e. inside, *not* onto) another branch.\n\nOkay, but in case of merge conflicts, you still have to switch to the\nother branch, right?\n\n>   - git-transplant would also take care of removing the commits from\n>     the source branch, but not before they were safely inside the\n>     destination branch.\n\nThat assumes a workflow where you develop on one big messy branch and\nlater sort it out into the appropriate, separate branches, right? I admit\nthat I used to do that, too, but ever since worktrees arrived, I do not do\nthat anymore: it resulted in too much clean-up work. Better to put the\ncommits into the correct branch right away. Of course, that is just *my*\npreference.\n\n>   - git-transplant would orchestrate the whole workflow with a single\n>     command, complete with --abort and --continue.\n\ncherry-pick also sports --abort and --continue.\n\n> > >   - git-explode(1) which wraps around git-transplant(1) and\n> > >     git-deps(1), and automatically breaks a linear sequence of commits\n> > >     into multiple smaller sequences, forming a commit graph where\n> > >     ancestry mirrors commit dependency, as mentioned above.  I expect\n> > >     this to be more difficult, and would probably write it in Python.\n> > \n> > You mean something like Darcs on top of Git. Essentially, you want to end\n> > up with an octopus merge of branches whose commits would conflict if\n> > exchanged.\n> \n> Something like that, yes, but it's not as simple as a single octopus\n> merge.  It would support arbitrarily deep DAGs of topic branches.\n\nYes, of course. Because\n\nA - B - C - D\n\nmight need to resolve into\n\nA - M1 - C - M3\n  X        /\nB - M2 - D\n\n> > I implemented the logic for this in a shell script somewhere, so it is not\n> > *all* that hard (Python not required). But I ended up never quite using it\n> > because it turns out that in practice, the commit \"dependency\" (as defined\n> > by the commit diffs) does not really reflect the true dependency.\n> >\n> > For example,\n> \n> [snipped examples]\n> \n> Sure - I already covered this concern in footnote [0] of my previous\n> mail; maybe you missed that?\n\nI think it deserves more prominent a place than a footnote.\n\n> > So I think that this is a nice exercise, but in practice it will\n> > require a human to determine which commits really depend on each\n> > other.\n> \n> Of course - this is exactly why I wrote \"or at least\n> semi-automatically\" in the first mail of this thread.  But even though\n> git-deps / git-explode can never automatically handle *all*\n> dependencies, they can handle enough dependencies to be significantly\n> useful.  I have concrete real-world experience of that.\n\nI'd love to see those examples where it worked, because it sure did not\nwork for me (I wasted two weeks to implement that script that I never used\nsuccessfully).\n\n> I have made a large bunch of small commits to a single text file\n> (design document).  Some are possibly contentious; some aren't.\n\nAh. Well, as I said, I changed my workflow to use multiple worktrees with\nmultiple branches. The contentious changes would go into at least one\nbranch, more likely multiple. The uncontentious changes would go into\nanother.\n\nAnd most likely at least some of those branches would cause merge\nconflicts. However, they would do so only once, not multiple times during\nthe cleaning-up phase.\n\nI did actually track the time at some stage to determine what is faster.\nSorting things into multiple branches won hands down (in my hands). And\nno: I did not believe it would.\n\n> So I need to split them out into a series of smaller independent patch\n> series which I can submit to gerrit for review, thereby making life\n> easier for the reviewers and minimizing any bottlenecks where reviews\n> for one change are blocked because another change hasn't been reviewed\n> yet.  And in this case, because the changes are all applying to a single\n> file containing only natural language, git-deps correctly determines\n> *all* dependencies, not just textual ones.\n\nI do not buy that. When you introduce a section way down in the document\nfor which you have to introduce a new definition in one of the first\nsections, logically those two changes belong to the same topic branch. Yet\ngit-deps would have no chance to determine that.\n\n> > You would render me *really* impressed if you could come up with an\n> > automated way to determine logical dependencies between patches.\n> \n> Hey, I would *really* impress myself if I could do that, too; after\n> all, that would be a pretty sophisticated form of artificial\n> intelligence :-)\n\nYep, I will definitely follow your progress!\n\nCiao,\nJohannes\n"},{"id":"287731","messageId":"20160528112417.GD11256@pacific.linksys.moosehall","threadId":"42467","inReplyTo":"alpine.DEB.2.20.1605280841000.4449@virtualbox","subject":"Re: RFC: new git-splice subcommand for non-interactive branch splicing","fromName":"Adam Spiers","fromEmail":"git@adamspiers.org","sentAt":"2016-05-28T11:24:18Z","receivedAt":"2016-05-28T11:24:18Z","isPatch":false,"sender":{"key":"git@adamspiers.org","avatar":"https://avatars.githubusercontent.com/u/100738?v=4"},"body":"On Sat, May 28, 2016 at 09:06:59AM +0200, Johannes Schindelin wrote:\n> Hi Adam,\n> \n> please reply-to-all on this list.\n\nSorry, I forgot that was the policy here.  Every list and individual\nhas different preferences on whether to Cc: on list mail, so I find it\nalmost impossible to keep track of who prefers what :-/\n\n> On Fri, 27 May 2016, Adam Spiers wrote:\n> > My feeling is that rebase -i provides something tremendously\n> > important, which the vast majority of users use on a regular basis,\n> > but that git is currently missing a convenient way to\n> > *non-interactively* perform the same magic which rebase -i\n> > facilitates.\n> \n> Would it not make sense to enhance `rebase -i`, then?\n\nYou mean enhance it to support non-interactive usage?  That wouldn't\nmake much sense to me, given that -i is short for --interactive.  Even\nif we added a new non-interactive rebase mode which let you edit the\ncommits prior to rebasing them, I can't imagine how it would need to\nbe any different to how non-interactive rebase -i currently works,\ni.e. setting GIT_SEQUENCE_EDITOR to a non-interactive command which\nmodifies the rebase todo file passed via $1.\n\nOr maybe you are suggesting to enhance it to perform operations on the\nrebase todo list, such as removing a commit from the todo list, or\nmoving a commit to a different position?  But that sounds like scope\ncreep to me; IMHO it would be cleaner for rebase -i to remain an\nunopinionated platform for history editing, rather than to make\nassumptions about common history editing workflows.  I think those\nassumptions belong in higher-level porcelain tools.\n\nOr if you have some other enhancement in mind, please share details!\n\n> I, for one, plan to port my Git garden shears (at least partially) once I\n> managed to get my rebase--helper work in. The shears script is kind of a\n> \"--preseve-merges done right\":\n> \n>     https://github.com/git-for-windows/build-extra/blob/master/shears.sh\n> \n> It allows you to (re-)create a branch structure like this:\n> \n>     bud\n>     pick cafebabe XYZ\n>     pick 01234567 Another patch\n>     mark the-first-branch\n> \n>     bud\n>     pick 98765432 This patch was unrelated\n>     mark the-second-branch\n> \n>     bud\n>     merge the-first-branch\n>     merge the-second-branch\n> \n> Of course, this is interactive.\n\nInteresting approach; thanks for sharing.  At a first glance, this\ndoes sound similar to what topgit and gitwork are trying to achieve.\nI don't entirely understand it yet, however; it's hard to without\nknowing more about the structure of Git for Windows' integration\nbranch and seeing a concrete example and/or comprehensive\ndocumentation.  For example, it's not clear to me how\ngenerate_script() works, or how it lets you modify just one of the\ntopic \"sub-branches\" and then automatically update all other\nsub-branches which depend on it?\n\n> But quite frankly, you want to be able to\n> perform quite complicated stuff\n\nWhy do you say that?  Splice and transplant operations are\nconceptually very straightforward, and not even particularly hard to\nimplement.  git-splice only took me a day or so, and I expect\ngit-transplant to be quicker.\n\nThe only complicated thing I want to implement is git-explode, but if\nI have splice and transplant \"primitives\" available, then it should be\nquite easy to implement git-explode as a series of splice/transplant\noperations.\n\n> and I think the command-line offers only an inadequate interface for\n> this.\n\nPlease can you give an example where it would be inadequate?\n\n> > I suspect the most popular use-case in the short term would be the\n> > infamous \"oops, I only just noticed that I put that commit on the\n> > wrong branch, and now there's already a whole bunch of other commits\n> > on top of it\".\n> \n> I have two workflows for that. The simpler one:\n> \n>     git checkout other-branch\n>     git commit\n>     git checkout @{-1}\n>\n> after those calls.\n> \n> The more complicated one comes in handy when a complete rebuild takes a\n> long time, and branch switching would trigger a rebuild:\n> \n>     # Here, I stash what I *want* on the other branch\n>     git stash -p\n>     git worktree add throwaway other-branch\n>     cd throwaway\n>     git stash apply\n>     git commit\n\nNeither of these workflows work with the scenario I described.  In my\nscenario, the commit is already buried beneath other commits.\n\n> I did use the approach you proposed a couple of times: just commit in the\n> middle, and sort things out later. The problem: I frequently forgot, and\n> if I did not, reordering the commits resulted in stupid and avoidable\n> merge conflicts.\n\nI think you are misunderstanding me - I am not proposing this\nworkflow; in fact that would be stupid because it's already in common\nusage.  But I'm not even advocating it.  I'm saying:\n\n  - I do it regularly by accident (since when I am in the hacking\n    zone, I am usually focused on the code, not on branch maintenance).\n\n  - When I eventually realise I've done it, I need to go back afterwards\n    and clean up the mess by decomposing the linear series of commits\n    into separate topic branches.\n\n  - No really good higher-level topic branch maintenance tools exist yet.\n\n  - This is a common problem which many git users suffer from, based\n    on a) my experience collaborating with others, b) personal\n    feedback I've received on this topic, and c) googling for things\n    like \"git move commit to another branch\".\n\n  - There needs to be an easier way to clean up the mess, which minimises\n    the pain resulting from merge conflicts.\n\n> > > > In the longer term however, I'd like to write two more subcommands:\n> > > > \n> > > >   - git-transplant(1) which wraps around git-splice(1) and enables\n> > > >     easy non-interactive transplanting of a range of commits from\n> > > >     one branch to another.  This should be pretty straightforward\n> > > >     to implement.\n> > > \n> > > This is just cherry-pick with a range...\n> > \n> > No it's not:\n> > \n> >   - git-transplant would be able to splice commits from one branch\n> >     *into* (i.e. inside, *not* onto) another branch.\n> \n> Okay, but in case of merge conflicts, you still have to switch to the\n> other branch, right?\n\nNope, if there are conflicts, you would either do\n\n    git transplant --abort\n\nleaving you where you started, or fix the conflicts and then do\n\n    git transplant --continue\n\nleaving you on the original branch, but with the transplanted commits\nno longer in that branch and instead in the target branch.\n\n> >   - git-transplant would also take care of removing the commits from\n> >     the source branch, but not before they were safely inside the\n> >     destination branch.\n> \n> That assumes a workflow where you develop on one big messy branch and\n> later sort it out into the appropriate, separate branches, right? I admit\n> that I used to do that, too, but ever since worktrees arrived, I do not do\n> that anymore: it resulted in too much clean-up work. Better to put the\n> commits into the correct branch right away. Of course, that is just *my*\n> preference.\n\nIt's *my* preference too, as I already stated above.  That does not\nchange the fact that even with the best intentions to avoid this\nworkflow, it will still happen, *especially* when there are still no\ngood higher-level tools which help avoid it.  BTW gitwork is the\nclosest I've seen to a tool which correctly addresses this:\n\n    https://jonseymour.s3.amazonaws.com/git-work.html#_discussion\n\nbut AFAIK it hasn't been updated in 3 years.  Maybe Jon (cc'd) can\ncomment on whether that means it already worked perfectly 3 years ago ;-)\n\n> >   - git-transplant would orchestrate the whole workflow with a single\n> >     command, complete with --abort and --continue.\n> \n> cherry-pick also sports --abort and --continue.\n\nIf you look at my implementation of git-splice you'll see that it uses\ncherry-pick --abort and --continue.  So thanks but I'm already fully\naware of that :-)  However that misses the point.\n\nTransplant needs to be a composite of multiple operations (splice into\nthe target branch followed by splice to remove from the source\nbranch).  Therefore given that either of these splice operations can\nfail due to conflicts, the overall transplant needs to be orchestrated\nas a single workflow which supports --abort and --continue.  For\nexample, if the second splice fails with conflicts, the user needs to\nbe able _with_a_single_command_ to roll back to the state before the\ntransplant started.  That means aborting the second splice, and\nreverting the first.\n\nIf rollback is not as easy as a single command, the UX will create\nfear of failure, which will discourage users from using the tool.\n\n> > > >   - git-explode(1) which wraps around git-transplant(1) and\n> > > >     git-deps(1), and automatically breaks a linear sequence of commits\n> > > >     into multiple smaller sequences, forming a commit graph where\n> > > >     ancestry mirrors commit dependency, as mentioned above.  I expect\n> > > >     this to be more difficult, and would probably write it in Python.\n> > > \n> > > You mean something like Darcs on top of Git. Essentially, you want to end\n> > > up with an octopus merge of branches whose commits would conflict if\n> > > exchanged.\n> > \n> > Something like that, yes, but it's not as simple as a single octopus\n> > merge.  It would support arbitrarily deep DAGs of topic branches.\n> \n> Yes, of course. Because\n> \n> A - B - C - D\n> \n> might need to resolve into\n> \n> A - M1 - C - M3\n>   X        /\n> B - M2 - D\n\nWhy are both M1 and M2 needed here?  Can't D be a child of M1?\n\n    A---M1---C---M3\n       /  \\     /\n    B-'    `-D-'\n\n> > > I implemented the logic for this in a shell script somewhere, so it is not\n> > > *all* that hard (Python not required). But I ended up never quite using it\n> > > because it turns out that in practice, the commit \"dependency\" (as defined\n> > > by the commit diffs) does not really reflect the true dependency.\n> > >\n> > > For example,\n> > \n> > [snipped examples]\n> > \n> > Sure - I already covered this concern in footnote [0] of my previous\n> > mail; maybe you missed that?\n> \n> I think it deserves more prominent a place than a footnote.\n\nI originally had it inline ;-/ but then moved it to a footnote since\nit was somewhat orthogonal to the main subject (git-splice) and I\nthought it was at risk of diluting focus.  Sorry if that was the\nwrong decision.\n\n> > > So I think that this is a nice exercise, but in practice it will\n> > > require a human to determine which commits really depend on each\n> > > other.\n> > \n> > Of course - this is exactly why I wrote \"or at least\n> > semi-automatically\" in the first mail of this thread.  But even though\n> > git-deps / git-explode can never automatically handle *all*\n> > dependencies, they can handle enough dependencies to be significantly\n> > useful.  I have concrete real-world experience of that.\n> \n> I'd love to see those examples where it worked, because it sure did not\n> work for me (I wasted two weeks to implement that script that I never used\n> successfully).\n\nYeah - I was hoping to make another YouTube video demonstrating one of\nthem, but I've gone over my time budget simply by implementing\ngit-splice (and unit tests) and then discussing it here :-/\n\n> > I have made a large bunch of small commits to a single text file\n> > (design document).  Some are possibly contentious; some aren't.\n> \n> Ah. Well, as I said, I changed my workflow to use multiple worktrees with\n> multiple branches. The contentious changes would go into at least one\n> branch, more likely multiple. The uncontentious changes would go into\n> another.\n\nExactly, that's the ideal situation we're aiming for.  The question is\nhow to get there.  Doing this branch maintenance manually is currently\ntoo inconvenient for most users.  My belief is that building primitive\noperations such as splice and transplant will encourage the\ndevelopment of higher-level tools which can further automate the\nmaintenance tasks.\n\n> And most likely at least some of those branches would cause merge\n> conflicts.\n\nSmall nitpick: technically, branches can't cause merge conflicts; only\noperations such as cherry-pick / merge / rebase can cause them.\n\n> However, they would do so only once, not multiple times during\n> the cleaning-up phase.\n\nSorry, I'm not sure I understand this, because I'm not sure exactly\nwhich operations you are referring to.\n\n> I did actually track the time at some stage to determine what is faster.\n> Sorting things into multiple branches won hands down (in my hands). And\n> no: I did not believe it would.\n\nIt can certainly win, yes, and as I already said, this is my favoured\napproach too, when I remember to do it.  But the win gets less likely\nfor each dependency which exists between branches, in the absence of\ngood branch management tools.\n\n> > So I need to split them out into a series of smaller independent patch\n> > series which I can submit to gerrit for review, thereby making life\n> > easier for the reviewers and minimizing any bottlenecks where reviews\n> > for one change are blocked because another change hasn't been reviewed\n> > yet.  And in this case, because the changes are all applying to a single\n> > file containing only natural language, git-deps correctly determines\n> > *all* dependencies, not just textual ones.\n> \n> I do not buy that. When you introduce a section way down in the document\n> for which you have to introduce a new definition in one of the first\n> sections, logically those two changes belong to the same topic branch. Yet\n> git-deps would have no chance to determine that.\n\nSure, but none of the 24 commits I am referring to do that - a fact of\nwhich I was entirely aware when I performed the analysis with\ngit-deps.  Sorry for omitting this from the above \"because ...\" clause.\n\nBut I would like to reiterate: just because it is known not to work in\nall situations, it does *not* mean it can't be of use in any\nsituation.  Again, don't let the perfect become the enemy of the good.\n\nCheers,\nAdam\n"},{"id":"287770","messageId":"20160530003448.GF11256@pacific.linksys.moosehall","threadId":"42467","inReplyTo":"20160527140811.GB11256@pacific.linksys.moosehall","subject":"RFC: new git-transplant subcommand for non-interactively moving commits between branches","fromName":"Adam Spiers","fromEmail":"git@adamspiers.org","sentAt":"2016-05-30T00:34:48Z","receivedAt":"2016-05-30T00:34:48Z","isPatch":false,"sender":{"key":"git@adamspiers.org","avatar":"https://avatars.githubusercontent.com/u/100738?v=4"},"body":"On Fri, May 27, 2016 at 03:08:11PM +0100, Adam Spiers wrote:\n> Hi all,\n>\n> I finally got around to implementing a new git subcommand which I've\n> wanted for quite a while.  I've called it git-splice.\n\n[snipped]\n\n> Next steps, and the future\n> --------------------------\n>\n> Obviously, I'd welcome thoughts on whether it would make sense to\n> include this in the git distribution.\n>\n> In the longer term however, I'd like to write two more subcommands:\n>\n>   - git-transplant(1) which wraps around git-splice(1) and enables\n>     easy non-interactive transplanting of a range of commits from\n>     one branch to another.  This should be pretty straightforward\n>     to implement.\n\nI've now written git-transplant too (and in the process of doing so,\nmade many more enhancements to git-splice).\n\n\nUsage\n-----\n\nExamples:\n\n    # Move commits A..B from the current branch onto branch X\n    git transplant A..B X\n\n    # Move commits A..B from the current branch into branch X after commit C\n    git transplant --after=C A..B X\n\n    # Create a new branch X starting at ref Y, then\n    # move commits A..B from the current branch onto X\n    git transplant --new-from=Y C A..B X\n\n    # Abort a transplant which failed during cherry-pick or rebase\n    git transplant --abort\n\n    # Resume a transplant after manually fixing conflicts caused by\n    # cherry-pick or rebase\n    git transplant --continue\n\nN.B. this command rewrites history, since after splicing the commits\ninto the target branch, it removes them from the current branch.  As\nwith git rebase, you should be aware of all the implications of\nhistory rewriting before using it.\n\n\nMotivation\n----------\n\nSee the rest of this mail thread.\n\n\nCode\n----\n\n> Currently this is in alpha state:\n>\n>   https://github.com/git/git/compare/master...aspiers:splice\n>\n> and I reserve the right to rewrite the history of that branch in the\n> near future ;-)\n\nThis still holds for git-splice, but also now for git-transplant too:\nthe same branch now contains git-transplant and its test suite.\n\n\nNext steps\n----------\n\n- Wait for feedback.  (Ideally I hope people will actually try both tools.)\n\n- If there is any appetite for eventually moving this into the\n  distribution, it will still need quite a few things cleaning up first,\n  e.g. making the coding style consistent with git's, getting rid of\n  bashisms ...\n\n- (Eventually) write git-explode, as explained elsewhere in this thread.\n\n\nCheers!\nAdam\n"}]}