git index: how does it work?

Hello,

I am wondering if someone could explain and/or point me to an explanation of how the git index works.

For instance, suppose I have a tracked file: "foo.c"

1) [I modify "foo.c"]
2) git add foo.c
3) [modify again]
4) git commit -m "blah blah"

Since I don't include the "-a" switch, the version I added on step 2 is committed. But how does the index keep track of these changes? Does the index file actually contain the hunks of "foo.c" that have been modified? Or is there a "temporary" blob created, which the index points to?

In either case, is there some interface to access these hunks and/or get a reference to the blob?

Thanks,

-- Shaun

PS I'm considering writing an extension to git where the "diff" understands the semantics of certain types of files: hunks wouldn't just be textual blobs but would try to represent a minimal change from one version to the next based on an edit distance, so that, e.g. changing the location of a function would be represented by a "move" edit, rather than two text changes.

I have been building a prototype as a wrapper around git, intervening to store extra information, etc before passing commands on to git. Blobs, commits, etc are nice abstractions I can leave as is, but the index seems sort of foggy to me. Any advice appreciated!

Shaun Cutts <shaun@cuttshome.net> writes:

Step 2 hashes foo.c and creates a blob object and registers it to the index. Step 4 writes out the index as a tree and makes a commit out of it.

Running this sequence might be instructive.

	1$ edit foo.c
        2$ git add foo.c
        2a$ git ls-files -s foo.c
	2b$ git diff foo.c
        2c$ git diff --cached foo.c
        3$ edit foo.c
        3a$ git ls-files -s foo.c
	3b$ git diff foo.c
        3c$ git diff --cached foo.c
        4$ git commit -m 'half-edit of foo.c'
        4a$ git ls-files -s foo.c
	4b$ git ls-tree HEAD foo.c
        4c$ git diff foo.c
        4d$ git diff --cached foo.c

 - 2a shows the actual blob object that was created out of foo.c in step 2.

 - 2b shows the difference between that blob (now in the index) and foo.c,
   which should be empty.

 - 2c shows the difference between the HEAD commit and the index, which
   should show your edit in step 1.

 - 3a shows the blob in the index; you haven't added, so it should show
   the same as 2a.

 - 3b shows the difference between the index and foo.c, which should show
   the edit in step 3.

 - 3c shows the difference between the HEAD commit and the index, which
   should show your edit in step 1.

 - 4a shows the blob in the index; you haven't added, so it should show
   the same as 2a.

 - 4b shows the blob in the committed tree and the blob object should be
   identical to 2a.

 - 4c shows the difference between the index and foo.c, which should show
   the edit in step 3.

 - 4d shows the difference between the HEAD commit and the index, which
   should now be empty.

Junio,

Your advice was very helpful.

Digging in, however, I find I still am in the dark on one point: how does the index track renamed files, and how to query it for information about them?

For instance, if I add a 5th step to the sequence:

5) git mv foo.c bar.c

Then I am told by "git status" that the file is renamed, but I can't seem to elict this info using "git ls-files". Under some circumstances even "git status" lists a new and deleted file after a rename.

Are renames being tracked by the index, and is there a more basic interface than "status" to query about them?

Thanks for any help, --- Shaun

On Aug 5, 2009, at 8:00 PM, Junio C Hamano wrote:

Heya,

On Wed, Aug 12, 2009 at 04:52, Shaun Cutts<shaun@cuttshome.net> wrote:

Nope, git never explicitly tracks renames. Try this: $ mv foo bar $ git rm --cached foo $ git add bar $ git status

It'll tell you that you renamed foo to bar, even if you never executed 'git mv'.

This is because git does rename _detection_, that is, it'll notice that you have another file with (almost) the same contents, so it assumes you did a rename.

Aha ---

that explains it, then.

Is there a lower-level interface to rename detection than via "status"? And... um... hmmm.... how does it work? The hash codes don't help for "almost" the same. Is there an approximate string matching algorithm built in somewhere?

Thanks,

-- Shaun

On Aug 12, 2009, at 7:47 PM, Sverre Rabbelier wrote:

[Please don't top-post, fixed that for you...]

On 2009.08.12 20:45:48 +0200, Shaun Cutts wrote:

Roughly, it works like this: The files are split into small chunks, those are hashed, and if there are chunks with the same hash in both files, those chunks are treated as being common to both files. The more the files have in common, the higher the similarity score. See estimate_similarity() for details.

As the splitting also happens at newlines this has some interesting effects, for example, you can completely reorder the lines in a file after renaming it, and git will still detect it as a rename.

Björn

Shaun Cutts <shaun@cuttshome.net> writes:

No. Index, nor git in general, never records renames. git records contents, not content changes. The index records a state, so does the tree object pointed at by the HEAD commit.

When you ask for "status", git will notice that you have lost a file, and added a new one, between these two states, by comparing them. The contents of these lost files and added files are then compared, and ones with similar contents are paired up.

That way, you do not have to use "git mv A B" to "rename" A to B. You can just as well "mv A B; git rm A; git add B", and get the same outcome, exactly because git does not record renames.

Instead, we track them by deducing that you renamed from the result.

The tree-vs-index comparison "git status" does to figure all this out is "git diff-index -M --cached HEAD".

As it should be obvious from the above description,

	git diff-index -M --cached HEAD -- A

is *NOT* the way for you to ask about "possible renames of A". You need to run the diff for the whole tree without path limitation so that you can pair deletions and creations up in order to deduce renames.

Heya,

On Wed, Aug 5, 2009 at 09:21, Shaun Cutts<shaun@cuttshome.net> wrote:

This sounds very similar to what Daniel was discussing in "[PATCH 2/3 v3] Use an external program to implement fetching with curl git" [0], if you're truly interested in doing this, please do keep me posted (and I suspect Dscho might also be interested in being cc-ed) :).

[0] http://thread.gmane.org/gmane.comp.version-control.git/124503

I'll be happy to keep you posted....

... I'll put up a description once I get things worked out a bit more. It will take me a month or two, though, probably.

... but as a quickie... :)

The general idea is to use actual syntax parsing to understand what happens in particular files, but be able to fall back on text if necessary. (Maybe "smarter text" as described by Daniel would be an intermediate fallback step.)

No matter what the target language, files have a hierarchical organization (at least as far as I am going to care about :)). My idea is to write a "delta" in yaml with the tree-edit operations, as a universal representation of changes. This could be edited by the user if necessary -- for example: a move with edits in it might not be detected, but the user could explicitly replace the delete/add pair with the move/edit. Tools would be provided to verify that the edited deltas actually produce the changes stated (& update them to capture the next set of deltas, etc.)

Suggestions from you guys as to the best way to tie this in would be greatly appreciated. I think the analysis of particular file types should only be loosely coupled with the rest of the system, though, as otherwise it will create a rats' nest.

Ideally, there would be a mechanism for an outside diff tool to specify "these are the hunks", and to register a utility to apply them... the smart diff tool would use the yaml tree-operation format and have its own registry (or config section) for how to analyze particular file types.

The diff tool would also be coupled with a merge tool... in general, it would be nice if there were more hooks for providing specialized diff & merge.

-- Shaun

On Aug 5, 2009, at 8:21 PM, Sverre Rabbelier wrote: