"True" git merge in git-pasky

  Hello,

  so I've implemented "true" git merge in git-pasky, using core git's
merging capabilities. It seems to even work. :-)

  I tested it briefly, and even did one non-conflicting and one
conflicting merge with Linus with this, but I'd like to hear your
comments and possibly more testing before releasing it.

  To get the lastest git-pasky, get the tarball at

	http://pasky.or.cz/~pasky/dev/git

unpack, build, install, do

	git pull

rebuild and reinstall.

  The semantics is trivial (and it might get changed so that you would
do git update instead of git pull at most of places). If you don't have
a given GIT repository ready yet, do

	git init rsync://example.com/repo

in a new directory. It is by default tracking, therefore if you do

	git pull

anytime later, git merge will be automatically invoked. If you want to prevent this, do

	git track

which will untrack your tree; the remote branch you were tracking is called "origin", shall you want to pull/merge it later. You might want to also merge with someone else. Do

	git addremote elsewhere rsync://example.org/another
	git pull elsewhere
	git merge elsewhere

(Note that merge won't pull automatically; you must do that on your own if you want to pull.)

If the merge didn't succeed and you have conflicts, don't panic. The merge told you about the conflicts, you can also do

	git diff

to see the changes, you'll probably spot the conflict markers. Resolve the conflicts and then simply do

	git commit

to commit the pending merge.

Now you decided to do a little bit of parallel development and stick your patches not ready for 2.6.12 to a separate tree. That's fine, do

	git fork experimental ~/linux-2.6.experimental

and get some coffee. (It takes about 8 minutes here, but I think git isn't at fault - it is probably all spent in

	read-tree $(tree-id)
	checkout-cache -a
	update-cache --refresh

and you pretty much need to call that.)

Then, do some work there, syncing with your main tree periodically:

	git merge master

(that's how your first init'd branch is called). You decide to make it more fun for Linus and push your experimental stuff into your master tree. Fine, cd there and do

	git merge experimental

and there you go!

Have fun,

On Tue, Apr 19, 2005 at 05:51:07AM +0200, Petr Baudis wrote:

I pulled the tar.bz2 and did make: gcc -g -O3 -Wall -o merge-cache merge-cache.o libgit.a libgit.a -lssl -lz gcc -g -O3 -Wall -c -o unpack-file.o unpack-file.c gcc -g -O3 -Wall -o unpack-file unpack-file.o libgit.a libgit.a -lssl -lz make: commit-id: Command not found Generating gitversion.sh...

Is this bad?

Dear diary, on Tue, Apr 19, 2005 at 07:43:07AM CEST, I got a letter where bert hubert <ahu@ds9a.nl> told me that...

It will cause a 40-digit hexadecimal number missing in your git help and git version output.

Petr Baudis <pasky@ucw.cz> : [...]

Tip of the day: cat the whole tree to /dev/null before the fork

-- Ueimor

I was reading this comment in gitcommit.sh and started thinking...

    # We bother with added/removed files here instead of updating
    # the cache at the time of git(add|rm).sh, since we want to
    # have the cache in a consistent state representing the tree
    # as it was the last time we committed. Otherwise, e.g. partial
    # conflicts would be a PITA since added/removed files would
    # be committed along automagically as well.

Let's for a moment forget what git-pasky currently does, which is not to touch .git/index until the user says "Ok, let's commit". I am wondering if that is the root cause of all the trouble git-pasky needs to go through. Specifically I think having to deal with add/remove queue seems to affect not just commit you have that comment above but also with diffs.

I'd like to start from a different premise and see what happens:

 - What .git/index records is *not* the state as the last
   commit.  It is just an cache Cogito uses to speed up access
   to the user's working tree.  From the user's point of view,
   it does not even exist.

 - The way this hypothetical Cogito uses .git/index is to always
   reflect add and remove but modification may be out of sync.
   It is updated lazily when .git/index must match the working
   tree.  Again, this is invisible to the user.  From the user's
   point of view, there are only two things: the last commit
   represented as .git/HEAD and his own working tree.

I call this hypothetical implementation of Cogito "jit-*" in the following description. Also this is just to convey the idea, so all the error checking (e.g. "what the user gave jit-merge is not a valid commit id") and sugarcoating (e.g. tags, symbolic foreign repository names instead of rsync URL etc) are omitted.

* jit-checkout $commit_id

  This is like "cvs co".  Same as what you are doing I suppose.

    committed_tree=$(cat-file commit $commit_id | sed -e 's/^tree //;q')
    read-tree $committed_tree
    checkout-cache -f -a
    echo $commit_id >.git/HEAD

* jit-add files... | jit-remove files...

  Like "cvs add".  Here, .git/index is treated as just a cache
  of the working tree, not the mirror of previous commit.  So
  unlike git-pasky, jit-* touches .git/index here.

    update-cache --add "$@"

    ---

    rm -f "$@" ;# this is debatable...
    update-cache --remove "$@"

* jit-diff [files...]

  Like "cvs diff".  The user wants to see what's different
  between his working tree and the last commit.

    case "$#" in 0) set x $(show-files --cached); shift ;; esac
    update-cache --add --remove "$@" --refresh
    current_tree=$(write-tree)

    committed_tree=$(cat-file commit $commit_id | sed -e 's/^tree //;q')
    diff-tree -r -z $committed_tree $current_tree |
      filter-output-to-limit-to-given-filelist "$@" |
      parse-diff-tree-output-and-show-real-file-diffs

  Unlike git-pasky, jit-* does not keep the state from the last
  commit in .git/index.  Instead, .git/index is meant to cache
  the state of the working tree.  So the first three lines in
  the above updates .git/index lazily from what is in the
  working tree for the part that needs to be diffed.  Then it
  uses helper scripts to filter and parse diff-tree output and
  generates per-file diffs.  Since add and remove are already
  recorded in .git/index, it does not have to special case
  "uncommitted add" and such.

* jit-commit

  Like "cvs commit".

    set x $(show-files --cached); shift
    update-cache --add --remove "$@"

    current_tree=$(write-tree)
    next_commit=$(commmit-tree $current_tree -p $(cat .git/HEAD))
    echo $next_commit >.git/HEAD

  Unlike git-pasky, .git/index already has adds and removes but
  it does not know about local modifications.  So it runs
  update-cache to make it match the working tree first, and then
  does the usual commit thing.  

  The above only allows the whole tree commit.  But allowing
  single file commit is not that hard:

    (
        set x $(show-files --cached); shift
        update-cache --add --remove "$@"
    ) ;# we use subshell to preserve "$@" here...
    current_tree=$(write-tree)

    committed_tree=$(cat-file commit $commit_id | sed -e 's/^tree //;q')
    read-tree $(committed_tree)
    update-cache --add --remove "$@"
    next_commit=$(commmit-tree $current_tree -p $(cat .git/HEAD))
    echo $next_commit >.git/HEAD

    read-tree $current_tree

  The first four lines are to preserve the current tree state.
  Then we rewind the dircache to the last committed state,
  update only the named files to bring it to the state the user
  wanted to commit, and commit.  Once done, we re-read the state
  to match the user's original intention (e.g. adds recorded in
  .git/index previously but not committed in this run is
  preserved).

* jit-merge $commit_id

  LIke "cvs up -j".  I have working tree which is based on some
  commit, and I want to merge somebody else's head $commit_id.
  Stated more exactly: I want to have the result of my changes
  in my working tree, if I started out from the merge between
  the commit I am actually based on and $commit_id.

    # First get my changes and stash away in a safe place.
    jit-diff >,,working-tree-changes-as-patch

    # After the above, we know .git/index matches the working tree, so...
    current_tree=$(write-tree)

    # Usual 3-way Linus merge.
    merge_base=$(merge-base $(cat .git/HEAD) $commit_id)

    base_tree=$(cat-file commit $merge_base | sed -e 's/^tree //;q')
    committed_tree=$(cat-file commit $(cat .git/HEAD) | sed -e 's/^tree //;q')
    his_tree=$(cat-file commit $commit_id | sed -e 's/^tree //;q')

    read-tree -m $base_tree $committed_tree $his_tree
    merge-cache three-way-merge-script -a

    # Now our .git/index has the merge result.  Match working
    # tree to it.
    checkout-cache -f -a

    # Apply our precious changes.
    patch <,,working-tree-changes-as-patch

    # Here we need to detect adds and removes and issue
    # appropriate update-cache --add --remove.

* jit-pull $foreign_repository

  I do not think we need this.  Just rsync but not merge.

It looks quite simple. I am asking your opinion because I am sure you have thought about issues involved through, and the above outline looks simple only because it is missing something important that you already had to deal with and solved---and the solution looks convoluted to me only because I am not aware of the problem you had to solve.

On Tue, 19 Apr 2005, Junio C Hamano wrote:

I think git-pasky is wrong.

It's true that we want to often (almost always) diff against the last "released" thing, and I actually think git-pasky does what it does because I never wrote a tool to diff the current working directory against a "tree".

At the same time, I very much worked with a model where you do _not_ have a traditional "work file", but the index really _is_ the "work file".

Yes. Yes. YES.

That is indeed the whole point of the index file. In my world-view, the index file does _everything_. It's the staging area ("work file"), it's the merging area ("merge directory") and it's the cache file ("stat cache").

I'll immediately write a tool to diff the current working directory against a tree object, and hopefully that will just make pasky happy with this model too.

Is there any other reason why git-pasky wants to have a work file?

		Linus

LT> Is there any other reason why git-pasky wants to have a work file?

Do you mean "why does a user wants to check things out in the working directory and make changes, possibly run compile tests before pushing the result to Linus?" ;-) I'm confused what you mean by "a work file", I guess...

On Tue, 19 Apr 2005, Linus Torvalds wrote:

Ok, "immediately" took a bit longer than I wanted to, and quite frankly, the end result is not very well tested. It was a bit more complex than I was hoping for to match up the index file against a tree object, since unlike the tree<->tree comparison in diff-tree, you have to compare two cases where the layout isn't the same.

No matter. It seems to work to a first approximation, and the result is such a cool tool that it's worth committing and pushing out immediately.

The code ain't exactly pretty, but hey, maybe that's just me having higher standards of beauty than most. Or maybe you just shudder at what I consider pretty in the first place, in which case you probably shouldn't look too closely at this one.

What the new "diff-cache" does is basically emulate "diff-tree", except one of the trees is always the index file.

You can also choose whether you want to trust the index file entirely (using the "--cached" flag) or ask the diff logic to show any files that don't match the stat state as being "tentatively changed". Both of these operations are very useful indeed.

For example, let's say that you have worked on your index file, and are ready to commit. You want to see eactly _what_ you are going to commit is without having to write a new tree object and compare it that way, and to do that, you just do

	diff-cache --cached $(cat .git/HEAD)

(another difference between diff-tree and diff-cache is that the new diff-cache can take a "commit" object, and it automatically just extracts the tree information from there).

Example: let's say I had renamed "commit.c" to "git-commit.c", and I had 
done an "upate-cache" to make that effective in the index file. 
"show-diff" wouldn't show anything at all, since the index file matches 
my working directory. But doing a diff-cache does:

	torvalds@ppc970:~/git> diff-cache --cached $(cat .git/HEAD)
	-100644 blob    4161aecc6700a2eb579e842af0b7f22b98443f74        commit.c
	+100644 blob    4161aecc6700a2eb579e842af0b7f22b98443f74        git-commit.c

So what the above "diff-cache" command line does is to say

   "show me the differences between HEAD and the current index contents 
    (the ones I'd write with a "write-tree")"

And as you can see, the output matches "diff-tree -r" output (we always do "-r", since the index is always fully populated). All the same rules: "+" means added file, "-" means removed file, and "*" means changed file. You can trivially see that the above is a rename.

In fact, "diff-tree --cached" _should_ always be entirely equivalent to actually doing a "write-tree" and comparing that. Except this one is much nicer for the case where you just want to check. Maybe you don't want to do the tree.

So doing a "diff-cache --cached" is basically very useful when you are asking yourself "what have I already marked for being committed, and what's the difference to a previous tree".

However, the "non-cached" version takes a different approach, and is potentially the even more useful of the two in that what it does can't be emulated with a "write-tree + diff-tree". Thus that's the default mode. The non-cached version asks the question

   "show me the differences between HEAD and the currently checked out 
    tree - index contents _and_ files that aren't up-to-date"

which is obviously a very useful question too, since that tells you what you _could_ commit. Again, the output matches the "diff-tree -r" output to a tee, but with a twist.

The twist is that if some file doesn't match the cache, we don't have a backing store thing for it, and we use the magic "all-zero" sha1 to show that. So let's say that you have edited "kernel/sched.c", but have not actually done an update-cache on it yet - there is no "object" associated with the new state, and you get:

	torvalds@ppc970:~/v2.6/linux> diff-cache $(cat .git/HEAD )
	*100644->100664 blob    7476bbcfe5ef5a1dd87d745f298b831143e4d77e->0000000000000000000000000000000000000000      kernel/sched.c

ie it shows that the tree has changed, and that "kernel/sched.c" has is not up-to-date and may contain new stuff. The all-zero sha1 means that to get the real diff, you need to look at the object in the working directory directly rather than do an object-to-object diff.

NOTE! As with other commands of this type, "diff-cache" does not actually look at the contents of the file at all. So maybe "kernel/sched.c" hasn't actually changed, and it's just that you touched it. In either case, it's a note that you need to upate-cache it to make the cache be in sync.

NOTE 2! You can have a mixture of files show up as "has been updated" and "is still dirty in the working directory" together. You can always tell which file is in which state, since the "has been updated" ones show a valid sha1, and the "not in sync with the index" ones will always have the special all-zero sha1.

I think this should obviate the need for Pasky keeping a separate work file. You can always tell what the difference to the last commit is with this, and you don't need to have a separate file to tell you about what you're supposed to do.

			Linus

The model you have always had is that there are three things the user needs to be aware of:

 * files in working tree -- this is what you touch with your
   editor and feed compilers with.

 * files in dircache -- update-cache copies from working
   tree to here, checkout-cache copies from here to working
   tree.

 * committed tree state -- write-tree + commit-tree copies from
   dircache to this state, read-tree copies from here to
   dircache.

The original message I started this thread with suggested that I wish if Cogito sugarcoating layer treated the dircache invisible to the user by keeping it virtually and lazily in sync with the working tree, as opposed to the way the current git-pasky does, which is to keep it in sync with the committed state.

But after thinking about it more, I changed my mind. With something like diff-cache available to the user, making aware of the three hierarchy to the user might be cleaner.

The workflow becomes:
 
 * Initial read-tree + checkout-cache -f -a; makes the three in
   sync.

 * Hack away.  Makes the working tree drift from dircache.

 * show-diff to see what's changed since your last "checkpoint".
   update-cache when happy.  Working tree is in sync with
   dircache which is the "staging area" for my half-baked but
   still good stuff.  Makes the dircache different from the
   committed.

 * Hack away more.  show-diff does not show your earlier changes
   anymore.  This is sometimes inconvenient when you want to see
   what you earlier changed but not committed.  Here comes the
   new shiny diff-cache to rescue.

 * When satisfied with all the changes diff-cache --cached
   shows, finally, say write-tree + commit-tree.  This makes all
   three in sync again.

I vaguely recall having heard about some SCM that distinguishes check-in and commit. Maybe this two-staged update-cache and write-tree + commit-tree workflow is similar to it?

LT> And as you can see, the output matches "diff-tree -r" output (we always do LT> "-r", since the index is always fully populated). All the same rules: "+" LT> means added file, "-" means removed file, and "*" means changed file. You LT> can trivially see that the above is a rename.

I do not know if Pasky tools already have something like this already, or not; but just FIY, here is what I use to extract a "patch" out of a working tree.

Usage:

    $ diff-tree -z [-r] ... | jit-diff-tree-helper [ | less ]
    $ diff-cache -z ...     | jit-diff-tree-helper [ | less ]

This would be useful for the merge I described in my initial message in this thread to take a snapshot of what the user has done since the last commit, to be applied on the result of the merge.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

--- jit-diff-tree-helper 2005-03-19 15:28:25.000000000 -0800 +++ jit-diff-tree-helper 2005-04-20 19:15:32.000000000 -0700