Multi-head fetches, pulls, and a King Ghidorah

The current "pu" branch has most of the necessary plumbing for multi-head fetching, pulling and creating Octopus merges based on multiple heads. I have made "git pull" multi-head aware but did not make it multi-head capable.

I have been trying out Tony Luck's excellent topic-branch workflow (see Documentation/using-topic-branches.txt), and the head of the "pu" branch is an Octopus on top of the "master" and my two private topic branches [*1*] made with "git octopus".

I have been thinking about how the multi-head fetch and merge should interact with each other. It appears to me that coming up with a reasonably semantics for "git pull" that would fit everybody is very difficult, because there are at least two different valid workflows that involve multi-head pulling, and they need to do different things. I'll talk about "git pull" that can deal with more than one remote heads, which does not exist yet, in the next couple of paragraphs.

Suppose I were the Emperor Penguin [*2*], and Jeff asks "Please pull from my netdev-2.6.git repository, sis190 and e100 branches." It is very reasonable to expect the following to fetch heads of those two branches and attempt to make an Octopus [*3*] on top of a copy of the Penguin head:

    $ cat .git/branches/jgarzik
    kernel.org:/pub/linux/kernel/git/jgarzik/
    $ git checkout -b try-jeffs-update master
    $ git pull jgarzik/netdev-2.6.git/ sis190 e100
    $ git diff -p master..HEAD | git apply --stat --summary

The arguments to pull, especially the refspec parameters ("sis190" and "e100" in the above example) are given to the underlying "git fetch", and because neither of them have a colon, they are not stored in corresponding places under the local refs/heads/. Instead, fetched heads would be fed directly to "git octopus" inside "git pull" (just like the current "git pull" invokes "git resolve" internally after fetching just one head).

However, suppose then I were Joe Random, an individual netdev contributor who is interested in these two netdev branches. Upon seeing the pull request, I might decide it is a good time to get changes from there, my upstream.

    $ git pull jgarzik/netdev-2.6.git/ sis190:sis190 e100:e100

Because I am keeping track of copies of these two branches, I use "sis190:sis190 e100:e100" to update my local heads.

I might have some local changes in these two branches. The old "git fetch" unconditionally overwrote local heads when told to, but lately it acquired the "reverse push" semantics Johannes Schindelin proposed to make it safer. The fetch process only overwrites local heads when the upstream change results in a fast-forward merge; practically, that happens only when I have not worked on that branch since I pulled from the upstream the last time. So my refs/heads/sis190 and refs/heads/e100 may be copies of Jeff's heads, or they may be the same heads as I had before starting the fetch.

At this point, it might be reasonable to expect that the above "git pull" command would behave as if I pulled (i.e. fetched and merged) these heads separately, using traditional single-head pull:

    $ git checkout sis190
    $ git pull jgarzik/netdev-2.6.git/ sis190:sis190
    $ git checkout e100
    $ git pull jgarzik/netdev-2.6.git/ e100:e100

That is, fetch and resolve them independently and individually.

Back in the Emperor Penguin example, he _could_ also have been interested in keeping copies of Jeff's branch heads, so he could have written refspecs on the command line the same way as Joe Random did. I.e. instead of:

    $ git pull jgarzik/netdev-2.6.git/ sis190 e100

he could have said:

    $ git pull jgarzik/netdev-2.6.git/ sis190:sis190 e100:e100

Both "Octopus" and "Multiple independent pull" semantics are valid to support different workflows, and there is no way to differenciate the two from the command line, without giving an extra flag and making the implementation more complicated.

Currently, I am inclined to leave the current "not more than one remote head" implementation, and possibly extend it to support the "Octopus" semantics later, for three very simple reasons.

 (1) What the latter "git pull" is buying us compared to two
     traditional single-head pulls is very little; that the
     underlying "git fetch" _could_ obtain packs more
     efficiently than two independent fetches.

 (2) The netdev example happened to involve multiple heads from
     a single repository, but the pull request could as well
     have been "jgarzik/netdev-2.6.git#sis190 and
     jgarzik/libata-dev.git#sil24", in which case I wouldn't
     have to be worrying about multi-head pull at all; the user
     would just have used two independent traditional "git
     pull" --- there is no other option.

 (3) Or course, because I am lazy ;-).  Seriously, multiple
     independent merges is a nightmere when you start thinking
     about what to do when you get a conflict and need to have
     the user hand merge in the middle of the first one.

Since "git pull" tentatively would not do multi-head natively, the Emperor Penguin example needs to be done this way:

    $ git checkout -b try-jeffs-update master
    $ git fetch jgarzik/netdev-2.6.git/ sis190 e100
    $ git octopus
    $ git diff -p master..HEAD | git apply --stat --summary

I think this should already work with the current "pu" branch head.

[Footnotes]

*1* I should probably write a bit about how I do things in a separate message as a how-to.

*2* I am not a penguin. Figuring out what kind of animal I am is left as an easter-egg hunt. I have had the answer somewhere in the current git.git archive for some time ;-).

*3* This one has only three heads, so that would be a Tripus, but what X-pus counts is not heads but legs, so we should really be calling this a King Ghidorah who has three heads.

*4* What's currently in "pu" branch dies when a head cannot be fast forwarded, but I think that is simply a thinko. It should just refuse to fast-forward and warn, leave the obtained remote head somewhere other tools can find later, just as the original "reverse push" script by Johannes did.

Hi,

If I understand correctly, the multi-head fetch would not write any ref if used like this:

	git fetch remote:repository/ head tail

but it would try a fast-forward when used like this:

	git fetch remote:repository/ head:head tail:tail

Correct? If yes: This is fantastic! It obsoletes my dumb script.

On Fri, 19 Aug 2005, Junio C Hamano wrote:

I propose a "--separate" flag to git pull. This would do exactly the same as a plain git pull, but for each fetched branch which could not be fast-forwarded

	- try to switch to the branch (dying if it is not the current,
	  and the working tree is dirty)

	- try a merge

	- if the merge fails, reset the branch to original state,
	  write out a temporary head and output a warning

After that, it would switch back to the original branch and check that out.

For all failed merges, the user needs to "git resolve" (the exact command line could be output by "git pull --separate").

Ciao, Dscho

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

That is the intent, but what is currently in "pu" is somewhat broken.

Junio C Hamano <junkio@cox.net> writes:

So here it is.

Note that the version of git on my $PATH is usually the one from the proposed updates branch, so some of the commands I use in the following text may not work for you unless you also have built "pu" one yourself.

I am planning to finish updating, testing and documenting what's in the current proposed updates branch, and have most of them graduate to the master branch over the weekend. I am aiming for doing the 0.99.5 on Wednesday next week.

------------

I have the following heads all the time in my private repository:

    master	- the one to be pushed to public "master" branch
    pu		- master plus various proposed changes
    rc		- master plus minimum release engineering

    ko-master	- a copy of public "master" branch head
    ko-rc	- a copy of public "rc" branch head

My GIT day always starts with this command:

    $ git fetch ko

I have this in .git/remotes/ko:

    $ cat .git/remotes/ko
    URL: master.kernel.org:/pub/scm/git/git.git/
    Pull: master:ko-master rc:ko-rc
    Push: master pu rc

The Pull: line gives me the default set of <refspec>s to give to the "git fetch" command. I slurp "master" and "rc" heads from the public repository at master.kernel.org, and fast-forward my ko-master and ko-rc branches with them. I do not touch these two branches in any way other than this "git fetch" fast forwarding.

I have a few "topic branches" in addition to the above; they change from time to time. Recently I've been looking at multi-head fetches, and that work is done in "mhf" branch. There also is a catch-all topic called "misc". I started them like this:

    $ git branch mhf master
    $ git branch misc master

The first thing I do during my GIT day is to process the patches I received via e-mail. I store them one topic per file in my working tree, like this:

    $ ls +*.txt
    +js-glossary.txt  +mc-mailinfo.txt

Depending on the quality of the patch, seriousness of the bug they fix, and the area of the code they touch, they either go directly to "master", "misc", or sent back to the sender, but the last one, luckily for me, rarely happens:

    $ git checkout master
    $ git applymbox -q ./+js-glossary.txt .git/info/signoff

    $ git checkout misc
    $ git applymbox -q ./+mc-mailinfo.txt please

The last parameter to the applymbox command is the name of a file that has my signoff message. The latest applymbox in the "pu" branch has a bit more useful extension to do the same thing as what "git commit" does.

At this point, I may push out the "master" branch (and nothing else), like this:

    $ git push ko master

This pushes only "master", ignoring the default <refspec>s defined in the .git/remotes/ko file you saw earlier.

Once I am done with the outside patches, I go back to where I left off the previous day:

    $ git checkout mhf

And I check where my head is relative to the master:

    $ git show-branches master mhf
    ! [master] Yet another tweak
     * [mhf] Make git-fetch-script a bit more chatty.
    +  Yet another tweak
    +  Another tweak in Makefile
     + Make git-fetch-script a bit more chatty.
     + Update git-ls-remote-script
     + ...
     + Start adding the $GIT_DIR/remotes/ support.
    ++ [PATCH] Allow file removal when "git commit --all" is used.

The output from show-branches is a poor-man's gitk. The named branches are shown, and '+' sign in each column shows whether the commit is contained in each branch, and the output stops where all branches converge, or you hit ^C ;-).

If the mhf branch is way behind, I may choose to first rebase it, to clean up my history:

    $ git rebase master
    $ git show-branches master mhf
    ! [master] Yet another tweak
     * [mhf] Make git-fetch-script a bit more chatty.
     + Make git-fetch-script a bit more chatty.
     + Update git-ls-remote-script
     + ...
     + Start adding the $GIT_DIR/remotes/ support.
    ++ Yet another tweak

I keep working in my topic branches. I may make some other changes in "misc" topic branch. It's a simple cycle of:

    $ edit-and-test
    $ git commit -s -a -v

Eventually I get to a good point where it makes sense to push things to the public repository.

    $ git show-branches master mhf misc
    ! [master] Yet another tweak
     * [mhf] Make git-fetch-script a bit more chatty.
      ! [misc] Add hooks to tools/git-applypatch.
     +  Make git-fetch-script a bit more chatty.
     +  Update git-ls-remote-script
     +  ...
     +  Start adding the $GIT_DIR/remotes/ support.
    ++  Yet another tweak
    ++  Another tweak in Makefile
      + Add hooks to tools/git-applypatch.
      + Add commit hook and make the verification customizable.
    +++ [PATCH] Allow file removal when "git commit --all" is used.

As you may have noticed, my topic branches are private and not pushed to the public repository. Instead, I make a grand total merge of them into "pu". The proposed update branch is always rewound and made from the head of the master:

    $ git checkout pu
    $ git reset master

This checks out the head of "pu" branch, and then resets the index file to match "master" and updates .git/refs/heads/pu.

What it does _not_ do is to update my working tree to match the index file. Linus recommends to do "git checkout -f" at this point, but I typically do this instead:

    $ git diff -R -p | git apply

Note. This is an embarrassingly expensive way; the only thing it buys me over "git checkout -f" is that it removes the files that were introduced in "pu" branch that did not exist in the "master" head. I have to come up with a not so expensive way to do this.

Then before doing the grand total merge, check again where those heads are:

    $ git show-branches master mhf misc pu 
    ! [master] Yet another tweak
     ! [mhf] Make git-fetch-script a bit more chatty.
      ! [misc] Add hooks to tools/git-applypatch.
       * [pu] Yet another tweak
     +   Make git-fetch-script a bit more chatty.
     +   Update git-ls-remote-script
     +   ...
     +   Start adding the $GIT_DIR/remotes/ support.
    ++ + Yet another tweak
    ++ + Another tweak in Makefile
      +  Add hooks to tools/git-applypatch.
      +  Add commit hook and make the verification customizable.
    ++++ [PATCH] Allow file removal when "git commit --all" is used.

Notice that I did not rebase "misc" above, but that is OK. What I want to do here is to make new "pu" an Octopus over "master", merging in all my topic branches (currently "mhf" and "misc").

    $ git fetch . mhf misc
    Packing 0 objects
    Unpacking 0 objects
    * committish: a101f32...e5580	refs/heads/mhf from .
    * committish: 4426ac7...0c5bc	refs/heads/misc from .

This fetches two heads from the current repository (!). The only reason I do it is that the tentative implementation of "git octopus" always reads from $GIT_DIR/FETCH_HEAD, and "git fetch" is the way to populate that file.

    $ git octopus
    Removing git-parse-remote
    Removing git-parse-remote
    Committed octopus merge fe1899156bffa4be6722b2ca0b74ff17b76523da
     Makefile                        |    3 -
     git-commit-script               |   75 +++++-------------
     ...
     tools/git-applypatch            |   87 ++++++++++++++++----
     15 files changed, 551 insertions(+), 224 deletions(-)

This makes an Octopus out of the master and other two topic branches. I can make sure that resulting "pu" contains all the necessary commits from the branches involved:

    $ git show-branches master mhf misc pu
    ! [master] Yet another tweak
     ! [mhf] Make git-fetch-script a bit more chatty.
      ! [misc] Add hooks to tools/git-applypatch.
       * [pu] Octopus merge of the following: 
       + Octopus merge of the following:
     + + Make git-fetch-script a bit more chatty.
     + + Update git-ls-remote-script
     + + ...
     + + Start adding the $GIT_DIR/remotes/ support.
    ++ + Yet another tweak
    ++ + Another tweak in Makefile
      ++ Add hooks to tools/git-applypatch.
      ++ Add commit hook and make the verification customizable.
    ++++ [PATCH] Allow file removal when "git commit --all" is used.

And things are now ready to be pushed out. First I sanity check the differences between ko-master and master (the earlier "git fetch ko" was done only for this step):

    $ git show-branches master ko-master

... and then run "git push":

    $ git push ko

This will push "master" and "rc" but would fail to push "pu", because that is rebased and not based on what is on the public repository. So I push once more, this time with --force, like this:

    $ git push --force ko pu

This pushes only "pu", ignoring the default <refspec>s defined in the .git/remotes/ko file you saw earlier. After that, I go back to reading the mail and wait until the kernel.org mirror network catches up.

Hi,

On Fri, 19 Aug 2005, Junio C Hamano wrote:

How about this?

  $ git reset master
  $ rm -i $(git-diff-tree ORIG_HEAD HEAD | grep "^.\{97\}D" | cut -c 100-)
  $ git checkout -f

Ciao, Dscho