How to switch kernel customizations from 2.6.15.6 to 2.6.16?

Dear git people,

I made a customized Linux kernel based on 2.6.15.6 by cloning the stable 2.6.15 kernel repository (which was then at version 2.6.15.6) and making several commits. Now I would like a Linux kernel based on 2.6.16 with the same customizations. This seems to be a very simple task, but I have been trying various combinations of git commands for several days and have not figured out how to do it.

I believe that means I should pull the 2.6.16 kernel into the "origin" branch and then rebase the "master" branch, merging my customizations with 2.6.16. To this end, I switched my remote file to point to the 2.6.16 stable repository and tried to pull. The result was not what I wanted. The situation is complicated by the fact that 2.6.15.6 is not an ancestor of 2.6.16. The warning in the man page about branches that are modified nonlinearly seems to apply.

How do I make my customized 2.6.16 kernel?

On Tue, 28 Mar 2006, Matt McCutchen wrote:

Don't "Pull". "Fetch".

"Pull" implies a merge, which is not what you want.

Just realize that you can have any number of branches, and instead of forcing "origin" to be something that it simply is not, just create a new branch called "linus".

Make that point to my tree, and do

	git fetch linus

to update it. It really is that easy.

Exact commands something like this:

 (1) Edit your .git/remotes/linus file so that it has the following 
     contents:

	URL: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6
	Pull: refs/heads/master:refs/heads/linus

 (2) Use that to fetch my tree as the "linus" branch:

	git fetch linus

and now you have at least a nice new branch that contains the "standard" kernel. At which point you just need to do

 (3) rebase the work since "origin" onto "linus":

	git rebase --onto linus origin

I really really despise the "git rebase" command line syntax, and I find it very non-inuitive, but what this does is take your current branch, compare its contents to "origin" (ie where you started from), and then just rebase the commits onto the state that is "linus".

[ Junio: syntax comments:

  Personally, I think the rebase syntax sucks, because the _natural_ way 
  to do it is to just describe the set of commits to rebase the same way 
  we describe all _other_ commit sets: as a "begin..end" sequence.

  So I think rebase _should_ work something like this:

	git rebase origin.. [--onto] linus

  ie just giving an arbitrary range. This is even more noticeable for 
  "git-format-patch", where that insane "<his> [<mine>]" syntax is even 
  worse, for no good reason, when again it should really just work like 
  "git diff" where giving a single revision implies a single revision, and 
  giving a range implies a range, and no strange "mine" vs "his" rules ]

Oh well. Syntax rant over.

			Linus

Linus Torvalds <torvalds@osdl.org> writes:

Yeah, inertia, backward compatibility wart, craziness.

Linus Torvalds <torvalds@osdl.org> writes:

I'd agree in general, and I am not happy about them.

But I have an excuse.

rev-parse's A..B notation was invented on June 13th (178cb24). But format-patch was originally posted on May 30th:

	http://article.gmane.org/gmane.comp.version-control.git/4279

before the convenience of rev-parse was invented ;-).

In addition, both rebase and format-patch does a bit more than straight his..mine.

    *---x---x---o---o---o---o
     \                      ^mine
      .---.---.---.
                  ^his

We do _not_ want to process all six of his..mine commits when doing "format-patch his mine" in the above picture, because upstream might have accepted some of them already, and we filter them out with git-cherry.

Having said that, you have been able to say format-patch A..B C..D E..F for quite some time (since November 21, 2005).

Rebase is even more strange, especially with --onto.  When you do

    $ rebase --onto his origin mine

in this picture,

    *---x---x---o---o---o---o
     \      ^origin         ^mine
      .---.---.---.
                  ^his

you are discarding two 'x' commits, and lost-found is the only thing that would help you to recover them.

Unlike format-patch which takes ranges, rebase does not let you say "rebase --onto base A..B C..D E..F"; what happens might be too confusing, especially if B, D, F are not coming from the current branch. The current branch is rewound to base and then the chosen sets of patches are applied, which is kind-of scary. It would feel safer to do:

	$ git checkout -b newbranch base
        $ git format-patch --stdout A..B C..D E..F | git am -3

and after making sure the result is really what you want resetting the original branch to the current (newbranch) head.

On Tue, 28 Mar 2006, Junio C Hamano wrote:

Yes, and the documentation still talks only about the old insane format..

.. and this is largely because the whole interface is broken.

The breakage shows up as inflexibility and having a hard time explaining what the thing does. It shows up as confusion about the usage, and about the meaning of a simple command line.

For example, in the most trivial format, doing a

	git rebase <branchname>

the _logical_ thing (from just reading the command) to believe the above does is to think that it rebases the named branch. I pretty much guarantee that that is what any native English speaker would think it does, if they thought about it.

The fact that it actually does the _reverse_, and rebases not the named branch, but the branch you are on right now.

The fact that when things go south, the old branch has gotten lost (unless you remember about ORIG_HEAD) is all related to the same thing. It's just a very non-intuitive interface.

That was fine when people weren't supposed to use it, and it was doing something very very special, but that has clearly changed over time. Now people are encouraged to use it, it's pretty well-known, and even new git users seem to want to do it.

I think "git rebase" is a lost cause. It's just fundamentally a very very badly designed command, because it does everything the wrong way around.

I personally believe that the _sane_ way of doing rebasing is to 
 - get rid of "git rebase" entirely
 - teach "git cherry-pick" to take a _range_ of commits instead.

Lookie here, let's look at your example a bit more (which in turn comes from the original question that started this thread):

    *---x---x---o---o---o---o
     \      ^origin         ^mine
      .---.---.---.
                  ^his

We have three branch points, and we want to move the commits on "mine" from "origin" onto "his". How would you do this so that it's _not_ confusing, and so that you can explain to a newbie user what he is doing, _especially_ if things go wrong in the middle?

Right now, the sequence is:

	git checkout mine			# if required
	git rebase --onto his origin

and if things go south during the rebase, it's immediately total chaos, and you really _really_ need to understand what you are doing.

The above just doesn't make any sense. It's hard to explain why you would do something like that.

In contrast, here's an alternate workflow that is much easier to explain, and doesn't involve "rebase" at all:

	git checkout his
	git cherry-pick origin..mine

Notice what this does? Show these two sequences to anybody who has some basic familiarity with git terminology, but has perhaps never actually used it, and ask them what the two sequences do. I pretty much guarantee that the second sequence will make sense and get people to generally pick the right answer, while the first sequence will make people maybe _guess_ the right answer, but it's not intuitive.

In particular, what do you think happens when a patch in the series doesn't apply under the two circumstances? Which workflow has the "intuitive" way of recovering, and which does not?

Right. The second one has a very intuitive way to recover. In fact, it's so intuitive that the answer may be "ok, I'll skip that one commit entirely because I don't know how to resolve it, and instead cherry-pick the rest, and ask the original author to cherry-pick it for me later". And doing so is as easy as

	git reset --hard	# undo the mess from the failed one,
				# the same way we always do for all
				# other failed things

	git cherry-pick next..mine	# do the rest

See? That's a very logical thing to do. It's different from "git rebase", but it's different in a _good_ way.

In contrast, the "rebase" conflict resolution is pretty damn esoteric, and using a simple "gitk --all" when things go wrong won't help you pick things up at all.

Now, "git cherry-pick" doesn't actually support the above format, and I'm not saying that the "git rebase" name itself is evil. I think we could fix "git rebase" to work better, but the semantics - the way they are _designed_ right now - are just horrible.

			Linus

On Wed, 29 Mar 2006, Linus Torvalds wrote:

vs

Btw, I realize that the advantage of "git rebase" is that it doesn't change somebody elses branch. However, we'd still be a lot better off with us simply doing the equivalent of something like

    git checkout -b new-mine his && 
	git cherry-pick origin..mine &&
	git rename-branch -f new-mine mine

instead of what git-rebase does now.

			Linus

Linus Torvalds <torvalds@osdl.org> writes:

Yes, I think it would be good to deprecate/discard the current rebase and make cherry-pick the recommended workflow.

Optionally. I've been thinking about not using git-cherry while cherrypicking, because falling back to three-way merge seems to work equally well in dealing with "patch already applied" case.

Anyway, teaching the range notation to cherry-pick would be something like this, I suppose.

---

Linus Torvalds wrote:

[...] 

Perhaps if possible also have

git cherry-pick --whole-branch branchname

meaning

git cherry-pick branchname:begining..branchname:HEAD

Jakub Narebski <jnareb@gmail.com> writes:

There is no branchname:beginning in git.

On Wed, 29 Mar 2006, Junio C Hamano wrote:

Well, in this case you don't need it. You just do

	git cherry-pick HEAD..branch

and it just magically does the right thing.

For consistency reasons, we should probably allow that to be written as just "..branch", the same way we can write "branch.." to mean "everything in HEAD but not in "branch".

		Linus

Linus Torvalds <torvalds@osdl.org> writes:

Something like this, perhaps.

-- >8 -- revision arguments: ..B means HEAD..B, just like A.. means A..HEAD

For consistency reasons, we should probably allow that to be written as just "..branch", the same way we can write "branch.." to mean "everything in HEAD but not in "branch".

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

Linus Torvalds <torvalds@osdl.org> writes:

Not really. Most of the commands in git suite operate on the current branch.

After thinking about it a bit more, I still agree with you that things are not as easy to explain as they should be, but I do not think rebase is so broken anymore.

This is _easy_ to explain, yes. However, I do not necessarily agree with what you said here:

It is not so easy to figure out what the next should be. If we limit ourselves to the simplest case that origin is an ancestor of mine, yes, but in general, no. We are rebasing presumably because upstream made independent progress so "origin" would not be an ancestor of mine anymore; and you are talking about the generic rev-list syntax next..mine === ^next mine.

"reset --hard" to stop cherry picking is easy. I do not think continuing is as easy as you made it sound like.

There was a nontrivial amount of thought went into making the "rebase" restartable. Actually, that thought was really for making the "am" restartable, and the hope was once the user becomes familiar with how to restart "am", restarting "rebase" is just as easy because you restart them the same way. You have it fall back to 3-way merge (and in the case of rebase, it _can_ fall back to 3-way when the patch does not apply, because all the blob object names recorded in the intermediate patch format are from your local repository), you resolve and prepare the index to be committed, and say "git am --resolved". We _could_ make "git rebase --resolved" a synonym for "git am --resolved", because "rebase" being tied to "am" only because the former is implemented in terms of the latter behind the user _is_ unintiutive.

As I said, yes, the part to punt is easy. But that is different from being able to continue smoothly. And if you want to punt during "rebase", you could just as easily do "git reset --hard ORIG_HEAD", just like you would punt a failed merge with "git reset --hard ORIG_HEAD".

The non-English (and no natural language I presume) syntax rebase takes is a mistake from understandability point of view. I fully agree with that. Let me think aloud how we could rephrase them better.

(1) git rebase origin

         A---B---C master (HEAD)
        /
    ---o---o---o---o origin

    I started building on tip of his but while I was woking on
    it he made independent progress.  I want to rebuild my
    branch as if I started at the tip of his current branch.

                     A---B---C master (HEAD)
                    /
    ---o---o---o---o origin

(2) git rebase --onto origin A..C

         A---B---C master (HEAD)
        /
    ---o---o---o---o origin

    I started building on tip of his but while I was woking on
    it he made independent progress.  I want to rebuild my
    branch as if I started at the tip of his current branch, but
    come to think of it I do not need A anymore.

                     B---C master (HEAD)
                    /
    ---o---o---o---o origin

    I personally feel _this_ form is the most logical, and form
    (1) for the sake of consistency could be spelled as:

        $ git rebase --onto origin origin..master

    So you could think of (1) a convenient shorthand for this
    spelled-out form.

(3) git rebase --onto origin A..C topic

               B---C topic
              /
         .---A---. master (HEAD)
        /
    ---o---o---o---o origin

    I have a topic that interferes with what he did in his
    latest updates, and I'd like to resolve the conflicts
    early.  Currently I am not on that branch so first let me
    switch to it.

         .---A---. master
        /
    ---o---o---o---o origin
                    \
                     B---C topic (HEAD)

    This form was done only as a shorthand to save typing "git
    checkout topic" at the beginning, just like "git checkout -b
    newbranch" can be used to save typing "git branch newbranch"
    before the checkout, but I agree it may have made things
    more confusing.  We _could_ deprecate this form and require
    the user to always switch branches before starting the
    rebase.

Matt McCutchen <hashproduct@verizon.net> wrote:

I think you want to use `git-fetch --force` to download origin but not immediately merge it yet. This will bypass the not-an-ancestor check you are running into.

Then you can perform the rebase yourself with:

	# Export your local changes into a series of patches.
	#
	git-format-patch -k --stdout --full-index v2.6.16.6 >changes.mbox

	# Checkout the new origin (2.6.16) into master.
	#
	git-reset --hard origin

	# Now apply your patches.
	#
	git-am --binary -3 changes.mbox

If you get merge conflicts fix them up and restart with `git-am --resolved`.

Note this is the logic of `git-rebase` except it doesn't require you to actually have a common ancestor, while `git-rebase` does.

Matt McCutchen <hashproduct@verizon.net> writes:

Drawing ancestry graph would help visualizing what you want to achieve. You have:

       v2.6.15
    ---o
        \ 
         \
          \                 
           o---o---o v2.6.15.6
                    \
                     x---x---x v2.6.15.6-matt

where x---x---x are your own changes, and you want:

       v2.6.15           v2.6.16
    ---o---o---o---...---o---o
        \                     \
         \                     y---y---y 2.6.16-matt
          \                 
           o---o---o v2.6.15.6
                    \
                     x---x---x 2.6.15.6-matt

to happen, where y---y---y are analogous to x---x---x.

Assuming your branches are:

        origin - v2.6.15.6 (from stable team)
        master - your changes (2.6.15.6-matt)

you could:

        $ git fetch git://../torvalds/linux-2.6.git tag v2.6.16
        $ git checkout -b 2.6.16-matt v2.6.16
        $ git format-patch origin master | git am -3

Alternatively, you might want to do a real merge:

       v2.6.15           v2.6.16
    ---o---o---o---...---o---o
        \                     \
         \                     \ 
          \                     m 2.6.16-matt
           o---o---o v2.6.15.6 /
                    \         /
                     x---x---x 2.6.15.6-matt

Presumably the stable team backported safer changes from the history between v2.6.15-v2.6.16, and the way things are fixed are probably quite different from the equivalent fixes in the development track that led to v2.6.16 (because what's being patched has also changed), so it is very likely you would see serious conflicts during this merge. If you do not understand what the stable team did in order to reimplement certain fixes, you would have a very difficult time deciding on how to resolve conflicts with this merge.

At that point it would not be a git question but the kernel question I am not qualified to answer ;-), but it might be an interesting exercise.

On Tue, 2006-03-28 at 18:26 -0800, Junio C Hamano wrote:

Beautiful diagram.  This is exactly my situation.

This looks like what I want. When I run the third command, however, I get "no patch found". Four files corresponding to my four commits appear in my repository; I have attached them. What is wrong?

Yes, I think this is the problem what I ran into before when I was trying to pull.

Perhaps this is just politics, but which kernel repository is more
official, and why?  Linus's or the one I have been using,
	git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.16.y.git
?

Thanks for the help!

Matt McCutchen <hashproduct@verizon.net> writes:

Me ;-).

It should have been 

        $ git format-patch --stdout origin master | git am -3

Sorry.

Linus tree is the main development for the 2.6.(X+1), and 2.6.x.y are managed by the stable team. They serve different purposes, and both are "official".

On Wed, 2006-03-29 at 19:22 -0800, Junio C Hamano wrote:

Thanks! My kernel is finally merged and is building as I write! It turns out that parts of my commits #2-#4 had already been addressed between 2.6.15.6 and 2.6.16. I had to merge #2 with what had already been done, and #3 and #4 became unnecessary.

On Wed, 29 Mar 2006, Matt McCutchen wrote:

The beauty of git should be (and maybe that's not entirely true simply because of practical concerns) that there really need not be any notion of "more official".

You can fetch multiple different git trees from different sources into the same git tree, and then keep them _all_ around equally as different branches. You can move fixes between the different branches with "git cherry-pick", and you can merge different branches with "git merge"

Now, the reason I say "_should_ be" rather than "is" is two-fold:

 - right now, a lot of the infrastructure is simply set up more towards 
   the "one single source repository" model. When you do a "git clone", it 
   kind of makes the origin special. That's how all the documentation is 
   written, and that's also the only remote branch that git creates 
   _automatically_ for you.

   This really isn't a technical issue: the git code code doesn't care 
   about any special "original" repository. But the fact that you have to 
   create the ".git/remotes/linus" file by hand, and that all the examples 
   in the docs end up talking about a single "origin" branch means that 
   people _think_ of git as a "single origin" thing.

 - the more fundamental one is that when you start mixing branches, you 
   have to be very careful if you expect the upstream projects to pull the 
   changes _back_. In particular, that's where you have to think twice (or 
   three times) about doing a "git merge" (or a "git pull", which 
   implicitly merges for you if it's not a pure fast-forward).

   In particular, the fact that _you_ want to merge two trees that came 
   from different sources does _not_ imply that either of the two sources 
   might want to merge with each other. So if you merge the two together, 
   you may find it impossible to have either of them then pull from you: 
   they way want your changes, but they might not like the merge you did, 
   because they have different policies about that work than you did.

So while the first point is purely a "mental model" issue and about lack of helper scripts, the second point is fundamental.

For example, in your case it was almost certainly the right thing to do to cherry-pick your changes from the 2.6.15.6 branch onto the development branch, because I simply don't want to merge the 2.6.15.6 stuff into the standard tree: part of the _rules_ for the stable branch is that the things it fixes should have been fixed in the development tree already, so merging the stable tree should always be unnecessary (and often clash, although _hopefully_ in many cases the fixes in the stable tree are 1:1 identical and will merge beautifully).

Anyway: from a technical standpoint, no tree should be "special" or "more 
official" for git usage. But when merging data back to any of those trees 
that aren't special, the source/history of the data is important to keep 
in mind. Branch "a" may not be any more special than branch "b", but when 
you push changes back to the source of branch "a", the history of those 
changes (relative to what the source was) is meaningful.

			Linus

On Thu, 2006-03-30 at 09:32 -0800, Linus Torvalds wrote:

I understand this, and it is one of several reasons why I prefer git to other version control systems. However, I thought there would be a single official kernel repository even if git didn't require it. Junio explained to me that both yours and the stable one are official for different purposes. I think I will use the stable one because it is current enough for my needs.

True. It might help several branches coordinate development if a commit could be marked as "equivalent" to another commit so that, if both were involved in a merge, one could be thrown out.