Trying to Update All Heads of a Repository

Guys and Gals,

*sigh*

This is the sort of thing that mystifies me still. I think there is (should be?) an easy command and I just missed that day in class. Feel free to smack me with the obvious as needed...

I re-cloned (rsync) by git.git tree yesteday. Today is a new day, and I want to start with a fresh update of the entire git.git tree. I have not changed anything in my copy yet. My goal is a simple update of the entire repository.

    % git --version
    git version 0.99.9.GIT

Uh, foo. That's a version from 02-Nov-2005. I guess I could dig out a SHA1 and we'd know exactly what version it is.

(Request:  Does it make sense to have a hard SHA1 as
           part of git's output version string?)

First, I have this, because it came with a git-clone:

    % cat .git/remotes/origin
    URL: rsync://rsync.kernel.org/pub/scm/git/git.git
    Pull: master:origin

I don't know how to say "Grab all the updates for all the heads for which I have heads in my .git/refs".

So I hope "git pull --help" will tell me:

    % git pull --help
    usage: git-fetch-pack [-q] [-v] [--exec=upload-pack] [host:]directory <refs>...
    Fetch failure: --help

That would be no. So I hit the man page, and I can't quite decipher it. I'm dense, so I have to re-read it a few more times until I fully grok it. Dinner interferes.

Like could I have just said?:

    % git fetch rsync://rsync.kernel.org/pub/scm/git/git.git
or
    % git fetch rsync://rsync.kernel.org/pub/scm/git/git.git master:origin
    % git fetch rsync://rsync.kernel.org/pub/scm/git/git.git pu:pu
    % git fetch rsync://rsync.kernel.org/pub/scm/git/git.git todo:todo

Dunno yet.

In the meantime, I figure I can effectively achieve my goal by:

    for each head $h
        git checkout -f $h
        git pull rsync://rsync.kernel.org/pub/scm/git/git.git $h:$h

So I start with master. I know I can cheat here and rely on the "Pull: master:origin":

    % git pull origin
    receiving file list ... done

    sent 102 bytes  received 12339 bytes  8294.00 bytes/sec
    total size is 4708148  speedup is 378.44
    * committish: a3114b3428595710d4719dd668531210af5993c9
      branch 'master' of rsync://rsync.kernel.org/pub/scm/git/git
    * refs/heads/origin: same as branch 'master' of rsync://rsync.kernel.org/pub/scm/git/git
    Already up-to-date.

That worked fine:

    % git show-branch | head -15
    ! [maint] GIT 0.99.9b
     * [master] Document --since and --until options to rev-parse.
      ! [origin] Document --since and --until options to rev-parse.
       ! [pu] Merge branch 'master'
	! [todo] GIT 0.99.9
    -----
	+ [todo] GIT 0.99.9
	+ [todo^] 2005-10-26 (prepare for 0.99.9)
	+ [todo~2] Update 2005-10-19 afternoon.
	+ [todo~3] 2005-10-18 midnight
	+ [todo~4] 2005-10-16 early afternoon.
	+ [todo~5] 2005-10-16 midnight.
	+ [todo~6] Updates 2005-10-14 night
	+ [todo~7] Updates 2005-10-12 night.

On to the pu branch:

    % git checkout -f pu
    % git show-branch | head -10
    ! [maint] GIT 0.99.9b
     ! [master] Document --since and --until options to rev-parse.
      ! [origin] Document --since and --until options to rev-parse.
       * [pu] Merge branch 'master'
	! [todo] GIT 0.99.9
    -----
	+ [todo] GIT 0.99.9
	+ [todo^] 2005-10-26 (prepare for 0.99.9)
	+ [todo~2] Update 2005-10-19 afternoon.
	+ [todo~3] 2005-10-18 midnight

And pull it now:

    % git pull rsync://rsync.kernel.org/pub/scm/git/git.git pu:pu

    receiving file list ... done

    sent 102 bytes  received 12325 bytes  8284.67 bytes/sec
    total size is 4708148  speedup is 378.86
    * committish: 6c86914344ea64e3a7b287721bdeaa7d41f8a1c3
      branch 'pu' of rsync://rsync.kernel.org/pub/scm/git/git
    * refs/heads/pu: does not fast forward to branch 'pu' of rsync://rsync.kernel.org/pub/scm/git/git;
      not updating.
    Trying really trivial in-index merge...
    fatal: Merge requires file-level merging
    Nope.
    Trying simple merge.
    Simple merge failed, trying Automatic merge.
    Auto-merging Makefile.
    ERROR: git-merge-ours.sh: Not handling case  -> a64704f3e52be43b427792ef9c1bbe42af057c87 -> 4f3d053889de4a5ba8e6e5d519c014a51220accd
    fatal: merge program failed
    Automatic merge failed; fix up by hand

Rats. But why did this fail? I've changed nothing in my tree. I'm pulling remote pu into local pu directly. I think this should Just Work...

Uh, now what?

    % git status

    # On branch refs/heads/pu
    #
    # Updated but not checked in:
    #   (will commit)
    #
    #       new file: Documentation/git-http-push.txt
    #       modified: Documentation/git-rev-parse.txt
    #       modified: Makefile
    #       modified: commit.c
    #       modified: config.c
    #       modified: fetch-pack.c
    #       modified: git-branch.sh
    #       modified: git-checkout.sh
    #       unmerged: git-merge-ours.sh
    #       modified: git-push.sh
    #       new file: git-shallow-pack.sh
    #       modified: git-svnimport.perl
    #       new file: http-push.c
    #       modified: ls-files.c
    #       modified: name-rev.c
    #       modified: send-pack.c
    #       modified: server-info.c
    #       modified: sha1_name.c
    #       modified: t/t3001-ls-files-others-exclude.sh
    #       modified: tag.c
    #       modified: tag.h
    #       modified: upload-pack.c
    #
    #
    # Changed but not updated:
    #   (use git-update-index to mark for commit)
    #
    #       unmerged: git-merge-ours.sh
    #
    #
    # Untracked files:
    #   (use "git add" to add to commit)
    #
    #       git-merge-ours.sh

    % git diff
    * Unmerged path git-merge-ours.sh

    % cat .git/HEAD
    0e6a92bd0b33b3461fb7b7f7eb34647ad93d451d

    % cat .git/FETCH_HEAD
    6c86914344ea64e3a7b287721bdeaa7d41f8a1c3                branch 'pu' of rsync://rsync.kernel.org/pub/scm/git/git

So. Yes, I know I can "git-add git-merge-ours.sh" and then do a "git-update-index git-merge-ours.sh" and get a spiffy new index. And then I can "git-commit" it all, and that should be everything.

But I really thought that this should Just Work? Shouldn't this be a fast forward, nice-n-clean-n-easy?

Do I need to add .git/remote entries for each head? Can I place multiple "Pull: pu:pu" type refspecs into the one .git/remote/origin? Is that what people do? Even if that I do that, I still have to manually iterate:

    for each head $h
        git checkout -f $h
        git pull origin $h

More tequilla.

Thanks, jdl

On Thu, 3 Nov 2005, Jon Loeliger wrote:

Well, you can't per se - there's no guarantee that the heads on the other side match, and in fact usually they won't (at the very minimum, the remote "master" tends to be the local "origin", although if you never do any development at _all_, you can make even that match).

I sent out a patch some time ago that added "--all" and "--tags", but only the "--tags" part was accepted. Arguably for good reasons.

But "git fetch" will take a list of heads. So you can do

	git fetch repo master:origin pu todo

and it will update the three local heads (origin, pu and todo) from the remote heads master, pu and todo respectively).

IF you have a 100% match between heads names locally and remotely, you could also do something like

	git fetch repo $(cd .git/refs ; echo heads/*)

which is pretty cheesy, but should work ;)

As to your experiences with "git pull" - it should work fine too, but you really do need to make sure that the branch you are in always matches the branch you're pulling. Otherwise you'll be doing a merge between two different branches - which is a valid operation, but not what you're trying to achieve here.

As we saw earlier, the way to undo an unsuccessful "git pull" is to do a "git reset --hard". Undoing a _successful_ git pull is actually slightly harder in the general case (it's really undoing a series of commit), but if you do it right after the pull, you can do

	git reset --hard ORIG_HEAD

because "git pull" will save the original head before pulling.

			Linus

Jon Loeliger <jdl@freescale.com> writes:

My "guinea pig" repository has this in $GIT_DIR/remotes/origin:

        URL: git://git.kernel.org/pub/scm/git/git.git
        Pull: master:origin
        Pull: +pu:pu
        Pull: maint:maint

This means that my "master" is copied to the "origin" branch of the guinea pig repository and "pu" and "maint" are copies of my "pu" and "maint" branches. You never do your own development on branches that appear on the right hand side of colon on "Pull" lines (i.e. origin, pu and maint) in this repository. They are to be updated by git-fetch.

And the regular workflow over there is:

	$ cd over-there
        $ git checkout master
        $ git pull
        ... build, test, install

You're right.  We need to add --help to that command.

You could. Alternatively, with your original remotes/origin file, you should be able to do:

	$ git checkout master
	$ git fetch origin master:origin +pu:pu maint:maint
        $ git pull . origin

My "pu" is somewhat special; it is rewound and rebased all the time, so merging with its older self would conflict with it. That's why my example remotes/origin file has '+' in front of it. It tells git-fetch that the other side _might_ rebase and fetch would not result in a fast-forward merge when that happens.

I should advertise that fact more often, for people who came here after I did the last such advertisement.

Sorry, my "pu" does not fast forward. The branch is to showcase what I've received or picked up from the list, and what changes are under consideration for inclusion.

The "pu" head is built by pulling a handful topic branches on top of the "master" branch. My private repository right now looks like this:

    ! [hold/draw] Illustration: "Commit DAG Revision Naming"
     ! [hold/http] [PATCH 2/2] Add support for git-http-push to git-pu...
      ! [hold/svn] Several fixes to import mono's svn tree
       ! [jcmu] git-tag: Do not assume the working tree root is writab...
        ! [maint] GIT 0.99.9c
         * [master] Illustration: "Commit DAG Revision Naming"
          ! [pu] Merge branch 'toys'
           ! [toys] git-shallow-pack
    --------
          +  [pu] Merge branch 'toys'
          ++ [toys] git-shallow-pack
          +  [pu^] Merge branches 'hold/svn' and 'hold/http'
     +    +  [hold/http] [PATCH 2/2] Add support for git-http-push to ...
     +    +  [hold/http^] [PATCH 1/2] Add support for pushing to a rem...
      +   +  [hold/svn] Several fixes to import mono's svn tree
       +  +  [jcmu] git-tag: Do not assume the working tree root is wr...
        +    [maint] GIT 0.99.9c
    +  ++++  [hold/draw] Illustration: "Commit DAG Revision Naming"
    +  ++++  [hold/draw^] Illustration: "Git Diff Types"
    +  ++++  [hold/draw~2] Illustration: "Fundamental Git Index Operat...
    +  ++++  [hold/draw~3] git-merge-ours: make sure our index matches...
    +  ++++  [hold/draw~4] Add 'ours' merge strategy.
    +  ++++  [hold/draw~5] Add --no-commit to git-merge/git-pull.
    +  ++++  [hold/draw~6] Document --since and --until options to rev...
    ++++++++ [hold/draw~7] Be careful when dereferencing tags.

The "hold/draw" topic branch (thanks for your ASCII art) is fully merged into my "master"; in fact it's head is the master branch head. Nick's http-push and Yaacov's svnimport fixes are kept in separate topic branches, which originated at the previous "master" (now master~7 == hold/draw~7), and they have not been rebased, partly I am lazy and partly there are no changes that would cause merge conflicts with other branches.

On the other hand, the commit structure of my public repository right now looks like this:

    ! [maint] GIT 0.99.9c
     * [master] Illustration: "Commit DAG Revision Naming"
      ! [pu] Merge branch 'toys'
    ---
      + [pu] Merge branch 'toys'
      + [pu^2] git-shallow-pack
      + [pu^] Merge branches 'hold/svn' and 'hold/http'
      + [pu^^3] [PATCH 2/2] Add support for git-http-push to git-push script
      + [pu^^3^] [PATCH 1/2] Add support for pushing to a remote reposito...
      + [pu^^2] Several fixes to import mono's svn tree
      + [pu~2] git-tag: Do not assume the working tree root is writable.
    +   [maint] GIT 0.99.9c
    +++ [master] Illustration: "Commit DAG Revision Naming"

That is, I do not publish the heads of my topic branches, but you can get them yourselves by looking at the merge logs in the "pu" branch. "toys" head is pu^2, "hold/svn" head is pu^^2, etc.

I think this "bundling together into one pu" head policy somewhat discourages contributor developments on top of "pu" material, but on the other hand the topic branches in my repository tend to have relatively short lifespan, and I'd rather not clutter people's cloned repositories with these transient branches.

If you wanted to try out Nick's http-push from "hold/http" branch, for example, but prefer not to worry about bugs in other things in "pu", you need to do this:

1. Find out if I keep the change as a separate topic branch.
   Either run "gitk --all" and inspect the commit structure
   between "master" and "pu", or look at the "pu" merge log from
   "git-show-branch" output above.

2. Isolate the topic branch from "pu", discarding other stuff
   you are not interested in.  After step 1, you figured out
   that pu^^3 is the tip of hold/http branch, so make a local
   branch for that, and merge the tip of "origin" on top of it,
   because hold/http branch may not have been rebased to the
   current master:

    $ git checkout -b http-push pu^^3
    $ git pull -n . origin
    $ git show-branch origin http-push http-push^
    ! [origin] Illustration: "Commit DAG Revision Naming"
     * [http-push] Merge branch 'master'
      ! [http-push^] [PATCH 2/2] Add support for git-http-push to ...
    ---
     +  [http-push] Merge branch 'master'
    ++  [origin] Illustration: "Commit DAG Revision Naming"
    ++  [origin^] Illustration: "Git Diff Types"
    ++  [origin~2] Illustration: "Fundamental Git Index Operations"
    ++  [origin~3] git-merge-ours: make sure our index matches HEAD
    ++  [origin~4] Add 'ours' merge strategy.
    ++  [origin~5] Add --no-commit to git-merge/git-pull.
    ++  [origin~6] Document --since and --until options to rev-parse.
     ++ [http-push^] [PATCH 2/2] Add support for git-http-push to ...
     ++ [http-push^^] [PATCH 1/2] Add support for pushing to a rem...
    +++ [http-push^~2] Be careful when dereferencing tags.

3. Build, test, install and have fun.

4. Suppose you came up with a neat enhancement idea or two to
   Nick's http-push.  Do your development and make commits on
   top of the current branch.  Your commit structure would look
   like this after a couple of commits:

    $ git show-branch origin http-push
    ! [origin] Illustration: "Commit DAG Revision Naming"
     * [http-push] Add feature nitfol.
    --
     + [http-push] Add feature nitfol.
     + [http-push^] Add feature frotz.
     + [http-push~2] Merge branch 'master'
    ++ [origin] Illustration: "Commit DAG Revision Naming"

5. Then feed me your changes, perhaps CC:ing the original author:

    $ git format-patch -k -m http-push~2
    * 0001-Add-feature-frotz.txt
    * 0002-Add-feature-nitfol.txt

You could publish the head of this branch and tell me to pull from there, but I have a slight preference to receiving e-mail patches if the changes are about the material still in "pu".

On Thu, 3 Nov 2005, Junio C Hamano wrote:

Shouldn't approximately this be what git-clone sets up for you? If it's getting those heads initially, it should keep the information for future use. The only thing it wouldn't know would be the "+" on pu.

On the "pu" issue, maybe "pu" shouldn't be a branch, but should be something different, like a tag. I think, actually, that a tag would describe "pu" pretty well: no development happens on its line; rather, development happens elsewhere (topic branches), and a merge of some of them is then released as being interesting. What is interesting as "pu" today isn't necessarily related to yesterday or tomorrow. Then you wouldn't exactly be rebasing things all the time, but you'd instead be marking different and unrelated commits in the same way at different times. (Probably need to do more to handle the case where a tag ref changes and define the semantics properly)

	-Daniel
*This .sig left intentionally blank*

Daniel Barkalow <barkalow@iabervon.org> writes:

Yes, and I am actually thinking along the lines of:

         URL: git://git.kernel.org/pub/scm/git/git.git
         Pull: master:origin/master
         Pull: +pu:origin/pu
         Pull: maint:origin/maint

to further prevent the puller's namespace being cluttered. This has an added benefit that it gives a convention that "remotes/origin" file describes the "refs/heads/origin" hierarchy. In the past, somebody complained that branches/* files are configuration per local branch, but remotes/* files are not.

Hmph.  That is a thought.

Another thing that bothers me somewhat about git.git/ repository is the todo branch. The "todo" branch does not have any common commit from the mainline and it is deliberately so --- I'd want to maintain a unified TODO list even after we go 1.0, at that time I plan to have master that continues post 1.0 development, maint which corresponds the current maint (i.e. maintenance for post 1.0 releases) and maint1.0 which is for backporting important fixes to frozen 1.0. I could maintain separate TODO-for-post-1.0 and TODO-for-1.0-maint, but that is rather a waste --- items on the two list both end up requirign attention and time from the same brain. Also merging "todo" into master and maint1.0 the way "gitk" project was merged into git.git would not make much sense, because that merge is pretty much one-way street.

The first thing I clone from git.git/ repository is almost always "rm -f .git/refs/heads/todo", because it clutters git-show-branch output otherwise ;-). In fact, it comes from totally separate repository on my private end, and just happens to be pushed into the same git.git/ repository (partly because I do not have any other repository there at kernel.org; i.e. /pub/scm/git/ is not writable by me).

Daniel Barkalow <barkalow@iabervon.org> writes:

I think this makes sense, and especially so, once we later change the default behaviour of 'git clone' to clone but rename the remote refs to somewhere else.

-- >8 -- cut here -- >8 --

This implements the idea Daniel Barkalow came up with, to match the remotes/origin created by clone by default to the workflow I use myself in my guinea pig repository, to have me eat my own dog food.

We probably would want to use either .git/refs/local/heads/* (idea by Linus) or .git/refs/heads/origin/* instead to reduce the local ref namespace pollution.

Signed-off-by: Junio C Hamano <junkio@cox.net>

---

 Documentation/git-clone.txt |    3 ++-
 git-clone.sh                |   12 ++++++++++--
 2 files changed, 12 insertions(+), 3 deletions(-)

applies-to: 9cc3928769f8f4e3597fa446321112c0568514df a0b45ee2b1362d664cca2d81f2915ed3b903b252