Merging commits together into a super-commit

Hi,

I really love the fact I can micro-commit changes when I'm developing. However at some point the combination of changes I have made can be considered a single body of work. This is especially true when you start doing things like re-basing on code that has moved around a lot. You don't want to be correcting a whole bunch of merge failures for every commit in your current tree.

So far the only was I can see to do this is a:

git-diff master..HEAD > my.patch

And then re-applying your patch in stages, manually doing the commits.

Am I missing something?

I'm thinking something like git-cherrypick taking multiple commits and create a new super commit on a new tree. i.e.:

git-cherrypick -m "Valgrind fixes" 12345.. 12678.. 565757..

Merging the existing commit comments would be nice too.

git merge --squash ?

On Thu, 2007-05-10 at 13:19 +0200, Raimund Bauer wrote:

Hmm, that would do although it only works on whole trees, you can't specify a range of commits. So you would have to cherrypick groups of changes onto other branches and then merge them together with a couple of --squashes

However thanks for pointing that one out :-)

Hi,

On Thu, 10 May 2007, Alex Bennee wrote:

Since specifying several commit ranges can always lead to merge conflicts, it makes no sense to _not_ accumulate the patches in a separate branch.

Ciao, Dscho

Alex Bennee wrote:

Here we go:

- cherry-pick them before commit

  $ git cherry-pick -n x
  $ git cherry-pick -n y
  $ git cherry-pick -n z
  $ git commit -m "$(for c in x y z; do git show --stat $c; done)" -e

- merge in a single commit

  $ git merge --squash foo

You didn't really think that git couldn't do that, did you? ;)

-- Hannes

On Thu, 10 May 2007, Johannes Sixt wrote:

I've done this (actually, mostly with "revert", but cherry-pick and revert are literally the same things).

However:

I'm too lazy to do this part, so I always do it by hand.

Clearly git can, but equally clearly it really *would* be pretty nice if you could just do

	git cherry-pick x y z

and create one commit and have the message already somewhat done for you (and "git revert" doing the same).

So if somebody does that, I'll certainly applaud..

		Linus

On Thu, 10 May 2007 09:01:15 -0700 (PDT), Linus Torvalds wrote:

Perhaps with a --squash option. I've already been wanting a cherry-pick that accepts a range, but that makes separate commits.

Yes, I know there's some git-rebase thing that will do it, but I can never remember the right syntax for that without studying the documentation[*]. What I find myself wanting to type is just:

	git cherry-pick A..B

But there is the whole problem of how to deal with any conflict that appears during the process.

-Carl

[*] I do use one form of rebase without ever needing to consult the documentation, but it's in the opposite "direction", if you will, from the cherry-pick-a-range operation. I use it when I want to rebase a set of commits from my current branch to some other branch, such as:

	git rebase origin

Not surprisingly, what's common about both of the operations above is that they are really only accepting a single argument, (a range in one case, and a branch-name in the other). That's what makes for an easy-to-use command. Things that require multiple branch names in a particular order, or extra options, (see "git rebase --onto newbase upstream branch"), need someone smarter than me to drive them.

Another problem is that using the optional final [<branch>] argument to git-rebase makes it change the current branch, (which no other git-rebase operation does), and that's another thing I find very confusing.

I'm sure the most complex form of git-rebase solves some precise problem that someone has, (and maybe even gets used regularly). But it's got enough complications that I just ignore it, (and would instead really prefer being able to just cherry-pick a whole range).

On Thu, May 10, 2007 at 09:57:10AM -0700, Carl Worth wrote:

I use it all the time to fix up old commits with

	git checkout sha1-of-bad-commit
	...edit, test,...
	git commit -a ---amend
	git rebase --onto HEAD sha1-of-bad-commit original-branch

But though it usually does what I want, I'm in total agreement about the confusing syntax and branch-switching behavior....

--b.

On Thu, 10 May 2007 13:14:57 -0400, "J. Bruce Fields" wrote:

Excellent. Thanks for the description.

Fixing up some broken (unpublished) commit in the past is a useful thing to support[*]. Of course, "commit --amend" takes care of this for the case where it's the most recent commit. For an older commit, here is exactly what I'd like to do, (showing that my suggestion of cherry-pick with a range would cover exactly this scenario):

	git tag tainted		# Could be "git branch tainted" just as well

	git reset --hard bad-commit
	...edit, test...
	git commit --amend
	git cherry-pick bad-commit..tainted

	git tag -d tainted

So, compared to the rebase usage, this does add two commands for bookkeeping the original state and cleaning it up. But the syntax for the cherry-pick part is quite a bit simpler than the original rebase at least.

Also, "reset --hard" isn't actually what I want in this case. I'd like this recipe to use something that would move the current branch to some other point, but in a safe way, (that is, not destroy any uncommitted changes that might exist at the beginning). I don't have any proposal for what that would be.

-Carl

[*] I've also experimented with using other "non-core git" ways of addressing this same problem. Here are a couple that I haven't found satisfactory for this particular use case:

stg - This probably works great if you're using it as a primary
      interface. But trying to use it as a quick one-off when
      generally using core git does not work well at all. Instead of
      the two "git tag" commands in my recipe above, an stg recipe
      would involve a lot of additional bookkeeping with stg init, stg
      uncommit [N times for fixing a commit N steps back in the
      history], stg goto, stg push, etc.

cg-admin-rewritehist - This is a powerful tool, but its interface
      isn't geared toward interactively fixing things up, (instead
      requiring a filter to be executed at all steps). So, while it's
      great for changing history, (as its name suggests), such as
      performing a sed operation on the commit messages, or
      eliminating a file, it doesn't help much with the "fix up one
      broken commit in the past".

      It is worth noting that quite unlike rebase,
      cg-admin-rewritehist can deal quite nicely with history that
      involves branching and merging.

On Thu, May 10, 2007 at 08:30:37PM CEST, Carl Worth wrote:

I think you are underestimating stg here. You can stg init just once per branch (ever), I think. Then,

	stg uncommit -n N
	stg pop -n N-1
	..hack..
	stg refresh
	stg push -a

It seems to be a bit shorter than the sequence you've presented above, and overally working with volatile commits using StGIT feels much more natural to me - and I haven't even ever used quilt seriously! (I have special antipathy to the git reset UI, too.)

Few days ago Santi Bejar has sent me a bundle with some updates to the Git homepage (thanks again a lot!). Since I didn't want some of the patches and wanted to tweak others, what I eventually did was pretty much this: I fast-forwarded my master to his bundle's head, then uncommitted the patches, popped them all and repeated the sequence

	stg push
	..review..
		stg refresh
		stg commit
	..or..
		stg delete `stg top`

for each patch.

On Thu, 10 May 2007 21:21:06 +0200, Petr Baudis wrote:

Yes, maybe I didn't learn to use it well enough.

I don't have details now, but I know I ran into some difficulty when leaving the extra stg state around. It seems that it added stuff that resulted in some reference of mine becoming ambiguous, ("refspec <foo> matches more than one" perhaps?). What I do remember is that I couldn't get one of my standard git push commands to work until I deleted all of .git/refs/bases and .git/refs/patches and then things started to work again.

The -n option is something I hadn't noticed, and that helps, (except that what I've got to start with is a git revision name, not a number).

But there are still some places where an experienced git user runs into some awkward situations trying to use stg. For example, "stg refresh" is basically always doing the equivalent of "commit -a" so there's annoyingly no way to refresh only some of the modified state into the commit.

Also, if I want to edit a commit message while under the influence of stg, how do I do that? If I do "git commit --amend" will I seriously confuse stg, (I'm guessing I would, but I don't know).

It's that kind of uncertainty that makes me uncomfortable to mix git and stg. And personally, I couldn't get excited about using it alone, (for example, in addition to the commit message with headline, stg makes me invent yet _another_ name for every commit---yuck). Not to mention I'm already quite comfortable with git alone, and all the flexibility it provides.

Plus, all the stuff that stg provides to allow it to be used standalone ends up just being noise to the git user that just wants to do some stack-based manipulation of an unpublished branch, for example.

So, I'd really like to see something more integrated into git itself that provides some of the missing functionality.

-Carl

On Thu, May 10, 2007 at 09:48:05PM CEST, Carl Worth wrote:

Hmm, yes, I've been thinking myself that it would be quite nice if I could just tell uncommit git revname right away.

Yes, I fear that StGIT hides the index in a similar way that Cogito does. It seems like user index usage is undergoing kind of renaissance these days in Git community (at least it seems to me this way, maybe it's always been this way), it would probably make sense to allow making use of index in StGIT as well.

I have no idea, but there's stg refresh -e.

I wouldn't normally use it for projects I have commit access to myself, but for maintaining own patches for an "external" project, I just find it much more comfortable than using git. But then again, if this part of git UI improved as much as some of the other parts in the last half a year...

And yes, it would be cool if stg new could guess patch name from the subject line in a similar manner that stg uncommit does.

I'm sorry, I couldn't parse this. :-)

On Thu, 10 May 2007 22:02:53 +0200, Petr Baudis wrote:

I'll try again.

I like the git user interface. I like it a lot. (It's got a couple of tiny things that I would do differently if I could start over, but more importantly it has a lot of big things that I wouldn't have even thought of if I had started from scratch.)

But with respect to the current topic, there are a couple of features that the git interface is missing compared to something like stg:

1. Amend a commit that's somewhere besides the tip of a branch,
   (rebuilding every commit that follows)

2. Re-ordering commits that exist on a branch, (again, rebuilding
   every commit that follows).

And what I was trying to say in my confusing paragraph, is that if I look to stg to add one or both pieces of this functionality, then it comes with a lot of baggage. For example, "stg --help" lists about 38 sub-commands. And some of those are wholly unnecessary if already using git, (4 repository commands 6 working-copy commands, for example). While others exist only to allow a notion of "git commits" vs. "stg commits" and translating back and forth between them, (assimilate and uncommit for example).

Now, that's not a critique of stg itself. As you say, it can work really well if you use it in a standalone fashion to track some project.

I'd just love to see something more minimal, and incorporated into git itself, to address the missing functionality. Right now, "cherry-pick A..B" is all I have to suggest. But maybe later there could be some sort of push/pop addition as well, (except that obviously the name "push" isn't available as a sub-command).

-Carl

Hello Folks,

On Thu, May 10, 2007 at 14:16:43 -0700, Carl Worth wrote:

I would actually propose to (gradually) add stg functionality into git. If it was done in stgit-compatible fashion, it would allow using stgit for the bits still not ported to git and switching back and forth according to user's taste.

Many commands from stgit either already have git equivalent or do just a little work beyond what the git command already does, so they could be easily integrated.

I think that many of them would be actually pretty simple.

On 2007-05-10 22:02:53 +0200, Petr Baudis wrote:

It shouldn't be hard to do. Instead of uncommitting a fixed number of commits, loop until you reach a specified commit.

I agree. It's bad UI for StGIT to behave different from git, given that easy interoperation is a goal.

Yes, you would confuse it; the patch ref would still point to the old commit object, but that would no longer be an ancestor of HEAD. StGIT doesn't know how to recover from this, so you'd have to do it by hand, which is annoying.

I'm working on making StGIT more robust against this kind of damage, but I'm not quite done yet.

Good idea. This would be embarrassingly easy to do.

But you can kind of do it today. Just commit with git (my favorite here is the emacs modes) and "stg assimilate"!

On Fri, May 11, 2007 at 12:23:47AM +0200, Karl Hasselström wrote:

Well, that's an idea that already appeared in some discussions - I can't speak for Catalin, but I too think it could be a good thing.

Eg, if we're going to use the patchlogs a bit more (and I wish so), it will be much less cluttered by using the index to select what to commit with several git-add's, than when using several stg-refresh's.

As noted elsewhere, there are some commands that are a bit superfluous (add, rm, and the branch-switching feature directly come to mind). It is especially annoying, eg when "stg rm" behaves differently than "git rm", by not removing the real file. I have tried recently to avoid using "git add" and "stg rm", and I am quite pleased with that :)

Well, that's arguably a non-orthodox way of doing things, I like the idea your "stg new" patch much better :)

Best regards,

On 2007-05-11 22:40:17 +0200, Yann Dirson wrote:

It's only unothodox if you expect git and stgit to not always mix so well. But if we have the ambition that they should interoperate as near to seamlessly as we can make them, this kind of workflow becomes very natural.

It shouldn't be necessary with a manual "assimilate" step. If stgit finds that there are unadorned git commits on top of the patch stack, it should do the assimilation automatically. With that in place, "stg new" and "stg refresh" would be nearly superfluous, since git-commit with and without --amend does the same thing -- the only thing they won't do is give the user the option of manually choosing the patch name.

I believe this sort of integration is the way to go. It'll be beneficial for git users who want to occasionally use some stgit to rebase their patch series, since they'll not have to learn more than two or three new commands in addition to the git they already know. Heavy stgit users will benefit from having the much larger git community maintaining a large subset of the porcelain they use, instead of having to duplicate the effort and always lag behind.

This is no binary choice, of course. One could certainly imagine a compromise where stgit becomes much easier to mix with git than today, but still retains the current command set.

On Sat, May 12, 2007 at 12:43:25AM +0200, Karl Hasselström wrote:

It is great that this is possible, but I'm not sure I'll ever see it as "very natural" :)

Hm.  I'm not that convinced :)

Eg, imagine a merge commit somewhere in the stack. What would stgit do with that ?

I quite like the idea of makeing it easier to mix them, and removing the real duplicates from stgit, but I think that we should be careful not to remove power from stgit while doing this.

Best regards,

On 2007-05-12 09:10:23 +0200, Yann Dirson wrote:

There are two cases:

  1. The merge commit is below the bottommost patch. This is perfectly
     OK, and nothing special has to be done. The only restriction is
     that we can't uncommit past the merge.

  2. The merge commit is above the topmost patch. (There may or may
     not also be other not-yet-stgitified commits above the topmost
     patch, below or above the merge commit.) In this case, stgit
     should not auto-assimilate the commits on top of the stack (since
     it can't be done for the merge commit), and a number of stgit
     commands (push, pop, new, ...) should refuse to work until the
     user has either reset the branch so that the merge disappears, or
     done "stg commit" on all the patches below the merge.

Note that these are the only cases: stgit should (and does) enforce the invariant that the applied patches form a consecutive series of commits, without "holes". This is why "stg new" would be forbidden in case (2).

The point is not that you should commit merges on top of your patches, of course. The point is that if you do, stgit should handle it gracefully. Right now you can commit all your patches and do a merge, but if you try to do it the other way around, stgit will break down on you -- but there's no real reason why it should.

I agree.

torsdag 10 maj 2007 skrev Carl Worth:

List the files to refresh and you get what you want.

	stg refresh file1 file2...

	stg refresh -e

[...]

I agree mixing stgit and git is not really comfy until you learn it and still I mess things up somtimes. Also rebase seems quite a bit faster than stgit, but I have not intuitive understanding for rebase so I get scared everytime. Having a gui that lets me mark commits and "drag" them to the new location would be nice.

-- robin

On Thu, May 10, 2007 at 12:48:05PM -0700, Carl Worth wrote:

I really think we should have a "stg uninit" command. Note that currently "stg branch --delete" on master will just do that instead of really deleting the branch, but that is a known bug (#8732 on gna).

I remember quite some time ago that cg-push exhibited this behaviour. However, nowadays I frequently push stgit stacks with git-push without a problem.

While "uncommit to named commit" that Karl implemented helps here, and that "stg goto <patchname>" may be a viable alternative to "pop -n", you may also want to try:

	stg uncommit -t <commit>
	..hack..
	stg refresh -p <patchname>

There are still some rough edges with "refresh -p", though[*], but when it works I love this comfort :)

Best regards,

Yann Dirson wrote:

It would be also nice to have command to remove applied patches. Sometimes I'd muck up StGIT stack by rebasing in git. Applied patches are in repository, but I'm interested in preserving unapplied ones.

On 2007-05-12 13:34:30 +0200, Yann Dirson wrote:

What we should do is delete all stgit metadata when the last patch goes away.

And we shouldn't have "stg init", either. Initing should be done automatically when needed.

On Sat, May 12, 2007 at 04:02:28PM +0200, Karl Hasselström wrote:

This supposes there is no valuable branch-level metadata. Currently we have the description - something which could arguably be moved to the git level as well. Otherwise that sounds reasonable to me.

Good idea as well, that would make stg more accessible to the average plain-git user.

Best regards,

On 2007-05-12 16:41:45 +0200, Yann Dirson wrote:

I left the branch description out of the discussion on purpose, since it's not that interesting -- if there is a description, we can simply delete everything except that. And I wholeheartedly agree that the branch description doesn't belong in stgit; it's orthogonal to the business of managing a patch stack.

Yes, that's my secret evil master plan.

Yann Dirson <ydirson@altern.org> writes:

Will it be something like

	[branch "master"]
        	description = "My primary development line"

if so I think that is a reasonable thing to do, from git-core's point of view. Obviously, gitk, tig, gitweb and friends can use this, too.

Are there other per-branch information StGIT wants to keep on an active branch that might benefit the core as well?

Yes, I wished for this often myself.

On 2007-05-12 12:27:49 -0700, Junio C Hamano wrote:

Yes, exactly. It's just a simple per-branch description string, just like in your suggestion.

Absolutely.

No, I'm pretty sure there isn't.

On Sat, May 12, 2007 at 12:27:49PM -0700, Junio C Hamano wrote:

Maybe the "protected" attribute, to forbid commands to touch to the stack ? Not sure, however, since the semantics would probably be a bit different in git (eg. just forbid update-ref) and in StGIT (also protects unapplied patches).

Best regards,

Instead of storing the branch description in an StGIT-specific file, store it in the git config file, where tools other than StGIT can read and write it.

Signed-off-by: Karl Hasselström <kha@treskal.com>
---

On 2007-05-12 12:27:49 -0700, Junio C Hamano wrote:

This was easier to do than I'd thought. I don't get quotes around the description, though; do I have to insert them manually? And what purpose do they serve?

 stgit/stack.py |   19 ++++++++++++++-----
 1 files changed, 14 insertions(+), 5 deletions(-)

On Thu, May 10, 2007 at 11:30:37AM -0700, Carl Worth wrote:

Yeah, something like that would be great. I think I've seen others suggest similar syntax before, so it's probably just a question of one of us who want this finding time to write the patches.

The tag creation and cleanup could get to be annoying too. You could scrounge through the reflog instead of using a temporary tag, but depending on the amount of --amend'ing and cherry-picking you do the reflog entry may end up in a different place each time, so it's probably hard to make this automatic.

I also didn't like having to come up with another name for each patch--I'd rather just run git-log or gitk and cut-n-paste the sha1.

For kernel work I started out working with multiple (sym- or hard-linked) trees, then used akpm's patch scripts, then stgit. I think I'm happiest just using plain git.

The one thing I've never been good at is keeping the history of the patch series itself.

--b.

On Thu, May 10, 2007 at 09:22:12PM CEST, J. Bruce Fields wrote:

Actually, you don't have to - if you don't specify the patch names, stgit will make them up itself using the subject of the commit message as a base.

And by the way, I absolutely love that - when viewing the stack, it's very useful to see what commits you still have to go etc. - stg series is concise yet fully descriptive. I'm pondering about whether something like this couldn't be incorporated into other git UIs somehow as well.

On Thu, May 10, 2007 at 09:47:42PM +0200, Petr Baudis wrote:

You still have to on new patches, right?

Sure.

I mainly find myself using gitk origin.. for that now. My main problem there is just that it's slow. (And File->Update) seems possibly even slower than just killing and restarting it.--b.

On Thu, May 10, 2007 at 09:57:10AM -0700, Carl Worth wrote:

Indeed this is a problem we also have in StGIT, when pushing multiple patches after rebasing. Currently we have to deal with the conflict and forge a new command-line to finish the job, which is quite awkward.

In this respect, the --continue/--skip/--abort set of flags that git-rebase has are really useful.

In fact, I have plans to deal with such behaviours with stgit transactions: in this case, we have the need of user interaction in the middle of a transaction, and the rebase flags mentionned above are just a way for the user of continuing or aborting the transaction.

However, currently I'm not sure that git-rebase would be very robust if the user would mess with HEAD before issuing one of these commands. Maybe git would also benefit from a generic transaction mechanism of some sort, so "cherry-pick A..B" and possibly others can behave in a consistent way with rebase ?

It could even be more sensible to implement transactions at the git level rather than at the stgit one...

Best regards,

On 2007-05-12 11:53:12 +0200, Yann Dirson wrote:

Yes, please. (Unless a convincing technical argument pops up against it, of course.) Any stgit invariant that isn't based on a git invariant is one more thing that can break when git and stgit commands are mixed.

On Sat, May 12, 2007 at 12:49:19PM +0200, Karl Hasselström wrote:

For reference, I have written down some design ideas in january[1]. They were written with StGIT in mind, we'll have to see if it transposes easily to plain git.

I fear it will not be that easy, at least with this design :)

OTOH, implementing transactions in StGIT could provide a first experience on this particular field, that may later be transposed to git core - not unlike cogito did for other features. There is the risk, however, of seeing a different (hopefully better) design for the feature in git afterwards, and this in turn is likely to make life harder for StGIT...

[1] http://marc.info/?t=116803935800001&r=1&w=2

Best regards,