threads / discuss / 44119

Limitiations of git rebase --preserve-merges --interactive

Subject: Limitiations of git rebase --preserve-merges --interactive

## tl;dr

15 messages between Sep 20, 2016 and Sep 23, 2016.

replies: 14people: 7as markdown or json

Stefan Haller· Sep 20, 2016, 08:46 UTC · lore

The BUGS section of the git-rebase manpage says that editing or rewording commits "should work fine", but attempts to reorder commits usually don't do what you want.

I'd like to know more about what does or doesn't work. For example, will squashing commits work? (I.e. using the "fixup" or "squash" commands in the todo sheet.)

Will dropping commits work?

Does it make sense to insert "exec" commands, or will they run at arbitrary times?

-- 
Stefan Haller
Berlin, Germany
http://www.haller-berlin.de/
Anatoly Borodin· Sep 22, 2016, 19:33 UTC · re: Stefan Haller · lore

Re: Limitiations of git rebase --preserve-merges --interactive

Hi Stefan,

this section was added to the manual in the commit cddb42d2c58a9de9b2b5ef68817778e7afaace3e by "Jonathan Nieder" <jrnieder@gmail.com> 6 years ago. Maybe he remembers better?

-- 
Mit freundlichen Grüßen,
Anatoly Borodin
Kevin Daudt· Sep 22, 2016, 19:48 UTC · re: Anatoly Borodin · lore

Re: Limitiations of git rebase --preserve-merges --interactive

On Thu, Sep 22, 2016 at 07:33:11PM +0000, Anatoly Borodin wrote:
Show 6 quoted lines
> Hi Stefan,
> 
> this section was added to the manual in the commit
> cddb42d2c58a9de9b2b5ef68817778e7afaace3e by "Jonathan Nieder"
> <jrnieder@gmail.com> 6 years ago. Maybe he remembers better?
> 

Just to make it clear, this section explicitly talks about 'bugs' with preserve-merges and interactive rebase. Without the --preserve-merges option, those operations works as expected.

The reason, as that section explains, is that it's not possible to store the merge structure in the flat todo list. I assume this means git internally remembers where the merge commit was, and then restores it while rebasing.

Changing the order, or dropping commits might then give unexpected results.

Stefan Beller· Sep 22, 2016, 20:54 UTC · re: Kevin Daudt · lore

Re: Limitiations of git rebase --preserve-merges --interactive

On Thu, Sep 22, 2016 at 12:48 PM, Kevin Daudt <me@ikke.info> wrote:
Show 20 quoted lines
> On Thu, Sep 22, 2016 at 07:33:11PM +0000, Anatoly Borodin wrote:
>> Hi Stefan,
>>
>> this section was added to the manual in the commit
>> cddb42d2c58a9de9b2b5ef68817778e7afaace3e by "Jonathan Nieder"
>> <jrnieder@gmail.com> 6 years ago. Maybe he remembers better?
>>
>
> Just to make it clear, this section explicitly talks about 'bugs' with
> preserve-merges and interactive rebase.  Without the --preserve-merges
> option, those operations works as expected.
>
> The reason, as that section explains, is that it's not possible to store
> the merge structure in the flat todo list. I assume this means git
> internally remembers where the merge commit was, and then restores it
> while rebasing.
>
> Changing the order, or dropping commits might then give unexpected
> results.
>
The commit message may help as well:
    rebase -i -p: document shortcomings
    The rebase --preserve-merges facility presents a list of commits
    in its instruction sheet and uses a separate table to keep
    track of their parents.  Unfortunately, in practice this means
    that with -p after most attempts to rearrange patches, some
    commits have the "wrong" parent and the resulting history is
    rarely what the caller expected.
    Yes, it would be nice to fix that.  But first, add a warning to the
    manual to help the uninitiated understand what is going on.
Stefan Haller· Sep 23, 2016, 10:40 UTC · re: Stefan Beller · lore

Re: Limitiations of git rebase --preserve-merges --interactive

Stefan Beller <sbeller@google.com> wrote:
Show 35 quoted lines
> On Thu, Sep 22, 2016 at 12:48 PM, Kevin Daudt <me@ikke.info> wrote:
> > On Thu, Sep 22, 2016 at 07:33:11PM +0000, Anatoly Borodin wrote:
> >> Hi Stefan,
> >>
> >> this section was added to the manual in the commit
> >> cddb42d2c58a9de9b2b5ef68817778e7afaace3e by "Jonathan Nieder"
> >> <jrnieder@gmail.com> 6 years ago. Maybe he remembers better?
> >>
> >
> > Just to make it clear, this section explicitly talks about 'bugs' with
> > preserve-merges and interactive rebase.  Without the --preserve-merges
> > option, those operations works as expected.
> >
> > The reason, as that section explains, is that it's not possible to store
> > the merge structure in the flat todo list. I assume this means git
> > internally remembers where the merge commit was, and then restores it
> > while rebasing.
> >
> > Changing the order, or dropping commits might then give unexpected
> > results.
> >
> 
> The commit message may help as well:
> 
>     rebase -i -p: document shortcomings
> 
>     The rebase --preserve-merges facility presents a list of commits
>     in its instruction sheet and uses a separate table to keep
>     track of their parents.  Unfortunately, in practice this means
>     that with -p after most attempts to rearrange patches, some
>     commits have the "wrong" parent and the resulting history is
>     rarely what the caller expected.
> 
>     Yes, it would be nice to fix that.  But first, add a warning to the
>     manual to help the uninitiated understand what is going on.

Thanks, but all of this still talks about the issues in very generic terms ("most attempts to rearrange patches"). I'm interested in more details as to exactly what kind of attempts do or don't work. In particular, I'm interested in fixup/squash commands (without reordering anything else), or dropping (non-merge) commits.

I could of course experiment with these and try to find out myself, but I was hoping someone would just know the answer off the top of their head, saving me some time.

-- 
Stefan Haller
Berlin, Germany
http://www.haller-berlin.de/
Johannes Schindelin· Sep 23, 2016, 21:13 UTC · re: Stefan Haller · lore

Re: Limitiations of git rebase --preserve-merges --interactive

Hi Stefan,
On Fri, 23 Sep 2016, Stefan Haller wrote:
Show 47 quoted lines
> Stefan Beller <sbeller@google.com> wrote:
> 
> > On Thu, Sep 22, 2016 at 12:48 PM, Kevin Daudt <me@ikke.info> wrote:
> > > On Thu, Sep 22, 2016 at 07:33:11PM +0000, Anatoly Borodin wrote:
> > >> Hi Stefan,
> > >>
> > >> this section was added to the manual in the commit
> > >> cddb42d2c58a9de9b2b5ef68817778e7afaace3e by "Jonathan Nieder"
> > >> <jrnieder@gmail.com> 6 years ago. Maybe he remembers better?
> > >>
> > >
> > > Just to make it clear, this section explicitly talks about 'bugs' with
> > > preserve-merges and interactive rebase.  Without the --preserve-merges
> > > option, those operations works as expected.
> > >
> > > The reason, as that section explains, is that it's not possible to store
> > > the merge structure in the flat todo list. I assume this means git
> > > internally remembers where the merge commit was, and then restores it
> > > while rebasing.
> > >
> > > Changing the order, or dropping commits might then give unexpected
> > > results.
> > >
> > 
> > The commit message may help as well:
> > 
> >     rebase -i -p: document shortcomings
> > 
> >     The rebase --preserve-merges facility presents a list of commits
> >     in its instruction sheet and uses a separate table to keep
> >     track of their parents.  Unfortunately, in practice this means
> >     that with -p after most attempts to rearrange patches, some
> >     commits have the "wrong" parent and the resulting history is
> >     rarely what the caller expected.
> > 
> >     Yes, it would be nice to fix that.  But first, add a warning to the
> >     manual to help the uninitiated understand what is going on.
> 
> Thanks, but all of this still talks about the issues in very generic
> terms ("most attempts to rearrange patches"). I'm interested in more
> details as to exactly what kind of attempts do or don't work. In
> particular, I'm interested in fixup/squash commands (without reordering
> anything else), or dropping (non-merge) commits.
> 
> I could of course experiment with these and try to find out myself, but
> I was hoping someone would just know the answer off the top of their
> head, saving me some time.

The fundamental problem here is the underlying design of bolting on the "recreate a merge" functionality onto the "pick" command.

That is, if you try to rebase non-linear commit history, it will still generate a linear list of "pick <commit-name>" lines, as if it were linear, except that it will include the merge commits, too.

It then will try to guess what you want to do by recording which commit was rewritten as which commit. And when it encounters a "pick" with a merge commit, it will try to merge the *rewritten* commit.

In other words, the design does not allow for changing the tip of any merged branch. Not reordering, not dropping.

And I do not think that there is a way to fix that design. That is why I came up with the Git garden shears (see the link I sent elsewhere in this thread).

Ciao, Johannes

Stefan Beller· Sep 23, 2016, 22:26 UTC · re: Johannes Schindelin · lore

Re: Limitiations of git rebase --preserve-merges --interactive

On Fri, Sep 23, 2016 at 2:13 PM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 58 quoted lines
> Hi Stefan,
>
> On Fri, 23 Sep 2016, Stefan Haller wrote:
>
>> Stefan Beller <sbeller@google.com> wrote:
>>
>> > On Thu, Sep 22, 2016 at 12:48 PM, Kevin Daudt <me@ikke.info> wrote:
>> > > On Thu, Sep 22, 2016 at 07:33:11PM +0000, Anatoly Borodin wrote:
>> > >> Hi Stefan,
>> > >>
>> > >> this section was added to the manual in the commit
>> > >> cddb42d2c58a9de9b2b5ef68817778e7afaace3e by "Jonathan Nieder"
>> > >> <jrnieder@gmail.com> 6 years ago. Maybe he remembers better?
>> > >>
>> > >
>> > > Just to make it clear, this section explicitly talks about 'bugs' with
>> > > preserve-merges and interactive rebase.  Without the --preserve-merges
>> > > option, those operations works as expected.
>> > >
>> > > The reason, as that section explains, is that it's not possible to store
>> > > the merge structure in the flat todo list. I assume this means git
>> > > internally remembers where the merge commit was, and then restores it
>> > > while rebasing.
>> > >
>> > > Changing the order, or dropping commits might then give unexpected
>> > > results.
>> > >
>> >
>> > The commit message may help as well:
>> >
>> >     rebase -i -p: document shortcomings
>> >
>> >     The rebase --preserve-merges facility presents a list of commits
>> >     in its instruction sheet and uses a separate table to keep
>> >     track of their parents.  Unfortunately, in practice this means
>> >     that with -p after most attempts to rearrange patches, some
>> >     commits have the "wrong" parent and the resulting history is
>> >     rarely what the caller expected.
>> >
>> >     Yes, it would be nice to fix that.  But first, add a warning to the
>> >     manual to help the uninitiated understand what is going on.
>>
>> Thanks, but all of this still talks about the issues in very generic
>> terms ("most attempts to rearrange patches"). I'm interested in more
>> details as to exactly what kind of attempts do or don't work. In
>> particular, I'm interested in fixup/squash commands (without reordering
>> anything else), or dropping (non-merge) commits.
>>
>> I could of course experiment with these and try to find out myself, but
>> I was hoping someone would just know the answer off the top of their
>> head, saving me some time.
>
> The fundamental problem here is the underlying design of bolting on the
> "recreate a merge" functionality onto the "pick" command.
>
> That is, if you try to rebase non-linear commit history, it will still
> generate a linear list of "pick <commit-name>" lines, as if it were
> linear, except that it will include the merge commits, too.

Which on a more fundamental design level would be ok. (C.f. your shell history is a linear list of git commands, but it deals just fine with non linear DAGSs)

>
> It then will try to guess what you want to do by recording which commit
> was rewritten as which commit. And when it encounters a "pick" with a
> merge commit, it will try to merge the *rewritten* commit.

Instead of guessing we'd need to differentiate between "pick" and "pickmerge", whereas the later describes creating commits with more than one parent (i.e. the prior pick line).

I could imagine the "pickmerge" to list all additional parents (The first parent being the previously picked commit) via symbolic naming:

    pick 1234affe implement foo
    pickmerge 3456feed origin/js/new-feature-1 # Merge origin/js/new-feature-1
    pick 45678ead implement feature-2

The "pickmerge" would have first the merge tips, and then the old subject line after a # character.

>
> In other words, the design does not allow for changing the tip of any
> merged branch. Not reordering, not dropping.

I see how the current design is problematic as there is no argument possible that allows the user to correct the wrong guess.

>
> And I do not think that there is a way to fix that design. That is why I
> came up with the Git garden shears (see the link I sent elsewhere in this
> thread).
I'll look into that.

Thanks, Stefan

Anatoly Borodin· Sep 22, 2016, 21:04 UTC · re: Kevin Daudt · lore

Re: Limitiations of git rebase --preserve-merges --interactive

Hi Kevin,
Kevin Daudt <me@ikke.info> wrote:
> Changing the order, or dropping commits might then give unexpected
> results.

The question that Stefan has is rather "what is *supposed* to work / give *expected* results?". Some stuff can be found in the tests (t/t*rebase*preserve*), but maybe there is more?

-- 
Mit freundlichen Grüßen,
Anatoly Borodin
Anatoly Borodin· Sep 22, 2016, 21:01 UTC · re: Stefan Haller · lore

Re: Limitiations of git rebase --preserve-merges --interactive

Hi Stefan,

I've also done some archaeology and found that the original version of the merge preserving code was written by Johannes Schindelin <Johannes.Schindelin@gmx.de>, see e.g. f09c9b8c5ff9d8a15499b09ccd6c3e7b3c76af77

There were also some big discussion threads in 2007-2008 regarding a better mechanism to "mark" or "tag" the heads of rebased branches, but I haven't seen consensus and development in that direction after that.

PS There are also some pieces of "what should work" in these tests:

t/t3409-rebase-preserve-merges.sh* t/t3410-rebase-preserve-dropped-merges.sh* t/t3411-rebase-preserve-around-merges.sh* t/t3414-rebase-preserve-onto.sh*

-- 
Mit freundlichen Grüßen,
Anatoly Borodin
Stefan Beller· Sep 22, 2016, 21:08 UTC · re: Anatoly Borodin · lore

Re: Limitiations of git rebase --preserve-merges --interactive

On Thu, Sep 22, 2016 at 2:01 PM, Anatoly Borodin <anatoly.borodin@gmail.com> wrote:

Show 5 quoted lines
> Hi Stefan,
>
> I've also done some archaeology and found that the original version of
> the merge preserving code was written by Johannes Schindelin
> <Johannes.Schindelin@gmx.de>, see e.g.

I think it would be helpful if you'd cc those folks involved, not just the mailing list.

Johannes Schindelin· Sep 23, 2016, 21:04 UTC · re: Stefan Beller · lore

Re: Limitiations of git rebase --preserve-merges --interactive

Hi,
On Thu, 22 Sep 2016, Stefan Beller wrote:
Show 10 quoted lines
> On Thu, Sep 22, 2016 at 2:01 PM, Anatoly Borodin
> <anatoly.borodin@gmail.com> wrote:
> > Hi Stefan,
> >
> > I've also done some archaeology and found that the original version of
> > the merge preserving code was written by Johannes Schindelin
> > <Johannes.Schindelin@gmx.de>, see e.g.
> 
> I think it would be helpful if you'd cc those folks involved, not just
> the mailing list.

Indeed. It is quite tedious to re-Cc: people. Anatoly, please do not force me to put in that work in the future.

Also: I mentioned recently that I am not happy with the original design,
either, and that I came up with a new design that I intend to port to the
rebase--helper, once it is included in an official Git version:
https://public-inbox.org/git/alpine.DEB.2.20.1609111027330.129229@virtualbox/

Ciao, Johannes

Stefan Haller· Sep 23, 2016, 10:40 UTC · re: Anatoly Borodin · lore

Re: Limitiations of git rebase --preserve-merges --interactive

Anatoly Borodin <anatoly.borodin@gmail.com> wrote:
Show 6 quoted lines
> PS There are also some pieces of "what should work" in these tests:
> 
> t/t3409-rebase-preserve-merges.sh*
> t/t3410-rebase-preserve-dropped-merges.sh*
> t/t3411-rebase-preserve-around-merges.sh*
> t/t3414-rebase-preserve-onto.sh*

Thanks, this is interesting; I'm having trouble understanding the tests though. Some of them use rebase -p -i, but I don't understand why they use -i, or why that even works in a test (i.e. why it doesn't open an editor).

In one test I saw "GIT_EDITOR=: git rebase -i -p", which I guess means "use the initially given todo sheet unchanged". I don't see any tests that do an interactive rebase and actually change the todo list.

-- 
Stefan Haller
Berlin, Germany
http://www.haller-berlin.de/
Junio C Hamano· Sep 23, 2016, 15:37 UTC · re: Stefan Haller · lore

Re: Limitiations of git rebase --preserve-merges --interactive

lists@haller-berlin.de (Stefan Haller) writes:
> Thanks, this is interesting; I'm having trouble understanding the tests
> though. Some of them use rebase -p -i, but I don't understand why they
> use -i, or why that even works in a test (i.e. why it doesn't open an
> editor).

Upon starting up, tests dot-source t/test-lib.sh file and it unsets most of GIT_* environment variables to obtain a stable testing environment that is not affected by things that testers may have in their environment.

There is EDITOR=: in t/test-lib.sh, which was added in 2006 before GIT_EDITOR was invented. That is the one in effect for git subcommands that usually interacts with editors during the test, unless specific tests further override it with test_set_editor helper.

Stefan Haller· Sep 23, 2016, 15:50 UTC · re: Junio C Hamano · lore

Re: Limitiations of git rebase --preserve-merges --interactive

Junio C Hamano <gitster@pobox.com> wrote:
Show 17 quoted lines
> lists@haller-berlin.de (Stefan Haller) writes:
> 
> > Thanks, this is interesting; I'm having trouble understanding the tests
> > though. Some of them use rebase -p -i, but I don't understand why they
> > use -i, or why that even works in a test (i.e. why it doesn't open an
> > editor).
> 
> Upon starting up, tests dot-source t/test-lib.sh file and it
> unsets most of GIT_* environment variables to obtain a stable
> testing environment that is not affected by things that testers
> may have in their environment.
> 
> There is EDITOR=: in t/test-lib.sh, which was added in 2006 before
> GIT_EDITOR was invented.  That is the one in effect for git
> subcommands that usually interacts with editors during the test,
> unless specific tests further override it with test_set_editor
> helper.

Thanks for the explanation. So this explains why -i works at all here; it doesn't explain why -i is used in these tests. Unless I'm missing something, they should all work with just -p.

And I don't see any tests that do rebase -p -i and actually do something interesting with the -i part. So my original question still remains. :-)

-- 
Stefan Haller
Berlin, Germany
http://www.haller-berlin.de/
Johannes Sixt· Sep 23, 2016, 19:24 UTC · re: Stefan Haller · lore

Re: Limitiations of git rebase --preserve-merges --interactive

Am 23.09.2016 um 17:50 schrieb Stefan Haller:
> And I don't see any tests that do rebase -p -i and actually do something
> interesting with the -i part. So my original question still remains. :-)
-i -p came first. -p without -i was bolted on later.
-- Hannes

← back to recent threads