{"thread":{"id":"20411","subject":"git index: how does it work?","startedAt":"2009-08-05T16:21:38Z","lastAt":"2009-08-12T20:31:24Z","messageCount":9,"participants":["Shaun Cutts","Junio C Hamano","Sverre Rabbelier","Björn Steinbrink"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"119633","messageId":"loom.20090805T160528-69@post.gmane.org","threadId":"20411","inReplyTo":null,"subject":"git index: how does it work?","fromName":"Shaun Cutts","fromEmail":"shaun@cuttshome.net","sentAt":"2009-08-05T16:21:38Z","receivedAt":"2009-08-05T16:21:38Z","isPatch":false,"sender":{"key":"shaun@cuttshome.net","avatar":"https://gravatar.com/avatar/6238ceb14c3a11096c8574e76b7e991a156522e521f0a9d099abed81c20c4451?d=mp&s=160"},"body":"Hello,\n\nI am wondering if someone could explain and/or point me to an explanation of how\nthe git index works.\n\nFor instance, suppose I have a tracked file: \"foo.c\"\n\n1) [I modify \"foo.c\"]\n2) git add foo.c\n3) [modify again]\n4) git commit -m \"blah blah\"\n\nSince I don't include the \"-a\" switch, the version I added on step 2 is\ncommitted. But how does the index keep track of these changes? Does the index\nfile actually contain the hunks of \"foo.c\" that have been modified? Or is there\na \"temporary\" blob created, which the index points to? \n\nIn either case, is there some interface to access these hunks and/or get a\nreference to the blob?\n\nThanks,\n\n-- Shaun\n\nPS I'm considering writing an extension to git where the \"diff\" understands the\nsemantics of certain types of files: hunks wouldn't just be textual blobs but\nwould try to represent a minimal change from one version to the next based on an\nedit distance, so that, e.g. changing the location of a function would be\nrepresented by a \"move\" edit, rather than two text changes.\n\nI have been building a prototype as a wrapper around git, intervening to store\nextra information, etc before passing commands on to git. Blobs, commits, etc\nare nice abstractions I can leave as is, but the index seems sort of foggy to\nme. Any advice appreciated!\n"},{"id":"119643","messageId":"7v7hxi9m4k.fsf@alter.siamese.dyndns.org","threadId":"20411","inReplyTo":"loom.20090805T160528-69@post.gmane.org","subject":"Re: git index: how does it work?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-08-05T18:00:27Z","receivedAt":"2009-08-05T18:00:27Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shaun Cutts <shaun@cuttshome.net> writes:\n\n> I am wondering if someone could explain and/or point me to an explanation of how\n> the git index works.\n>\n> For instance, suppose I have a tracked file: \"foo.c\"\n>\n> 1) [I modify \"foo.c\"]\n> 2) git add foo.c\n> 3) [modify again]\n> 4) git commit -m \"blah blah\"\n>\n> Since I don't include the \"-a\" switch, the version I added on step 2 is\n> committed. But how does the index keep track of these changes? Does the index\n> file actually contain the hunks of \"foo.c\" that have been modified? Or is there\n> a \"temporary\" blob created, which the index points to? \n\nStep 2 hashes foo.c and creates a blob object and registers it to the\nindex.  Step 4 writes out the index as a tree and makes a commit out of\nit.\n\nRunning this sequence might be instructive.\n\n\t1$ edit foo.c\n        2$ git add foo.c\n        2a$ git ls-files -s foo.c\n\t2b$ git diff foo.c\n        2c$ git diff --cached foo.c\n        3$ edit foo.c\n        3a$ git ls-files -s foo.c\n\t3b$ git diff foo.c\n        3c$ git diff --cached foo.c\n        4$ git commit -m 'half-edit of foo.c'\n        4a$ git ls-files -s foo.c\n\t4b$ git ls-tree HEAD foo.c\n        4c$ git diff foo.c\n        4d$ git diff --cached foo.c\n\n - 2a shows the actual blob object that was created out of foo.c in step 2.\n\n - 2b shows the difference between that blob (now in the index) and foo.c,\n   which should be empty.\n\n - 2c shows the difference between the HEAD commit and the index, which\n   should show your edit in step 1.\n\n - 3a shows the blob in the index; you haven't added, so it should show\n   the same as 2a.\n\n - 3b shows the difference between the index and foo.c, which should show\n   the edit in step 3.\n\n - 3c shows the difference between the HEAD commit and the index, which\n   should show your edit in step 1.\n\n - 4a shows the blob in the index; you haven't added, so it should show\n   the same as 2a.\n\n - 4b shows the blob in the committed tree and the blob object should be\n   identical to 2a.\n\n - 4c shows the difference between the index and foo.c, which should show\n   the edit in step 3.\n\n - 4d shows the difference between the HEAD commit and the index, which\n   should now be empty.\n"},{"id":"119650","messageId":"fabb9a1e0908051121m6efafc02gd468a92784a73ecd@mail.gmail.com","threadId":"20411","inReplyTo":"loom.20090805T160528-69@post.gmane.org","subject":"Re: git index: how does it work?","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2009-08-05T18:21:00Z","receivedAt":"2009-08-05T18:21:00Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Wed, Aug 5, 2009 at 09:21, Shaun Cutts<shaun@cuttshome.net> wrote:\n> PS I'm considering writing an extension to git where the \"diff\" understands the\n> semantics of certain types of files: hunks wouldn't just be textual blobs but\n> would try to represent a minimal change from one version to the next based on an\n> edit distance, so that, e.g. changing the location of a function would be\n> represented by a \"move\" edit, rather than two text changes.\n\nThis sounds very similar to what Daniel was discussing in \"[PATCH 2/3\nv3] Use an external program to implement fetching with curl git\" [0],\nif you're truly interested in doing this, please do keep me posted\n(and I suspect Dscho might also be interested in being cc-ed) :).\n\n[0] http://thread.gmane.org/gmane.comp.version-control.git/124503\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"119665","messageId":"C2C8DCE2-388B-49F4-A231-CD52779CDF48@cuttshome.net","threadId":"20411","inReplyTo":"fabb9a1e0908051121m6efafc02gd468a92784a73ecd@mail.gmail.com","subject":"Re: git index: how does it work?","fromName":"Shaun Cutts","fromEmail":"shaun@cuttshome.net","sentAt":"2009-08-05T19:31:50Z","receivedAt":"2009-08-05T19:31:50Z","isPatch":false,"sender":{"key":"shaun@cuttshome.net","avatar":"https://gravatar.com/avatar/6238ceb14c3a11096c8574e76b7e991a156522e521f0a9d099abed81c20c4451?d=mp&s=160"},"body":"I'll be happy to keep you posted....\n\n... I'll put up a description once I get things worked out a bit more.  \nIt will take me a month or two, though, probably.\n\n... but as a quickie... :)\n\nThe general idea is to use actual syntax parsing to understand what  \nhappens in particular files, but be able to fall back on text if  \nnecessary. (Maybe \"smarter text\" as described by Daniel would be an  \nintermediate fallback step.)\n\nNo matter what the target language, files have a hierarchical  \norganization (at least as far as I am going to care about :)). My idea  \nis to write a \"delta\" in yaml with the tree-edit operations, as a  \nuniversal representation of changes. This could be edited by the user  \nif necessary -- for example: a move with edits in it might not be  \ndetected, but the user could explicitly replace the delete/add pair  \nwith the move/edit. Tools would be provided to verify that the edited  \ndeltas actually produce the changes stated (& update them to capture  \nthe next set of deltas, etc.)\n\nSuggestions from you guys as to the best way to tie this in would be  \ngreatly appreciated. I think the analysis of particular file types  \nshould only be loosely coupled with the rest of the system, though, as  \notherwise it will create a rats' nest.\n\nIdeally, there would be a mechanism for an outside diff tool to  \nspecify \"these are the hunks\", and to register a utility to apply  \nthem... the smart diff tool would use the yaml tree-operation format  \nand have its own registry (or config section) for how to analyze  \nparticular file types.\n\nThe diff tool would also be coupled with a merge tool... in general,  \nit would be nice if there were more hooks for providing specialized  \ndiff & merge.\n\n-- Shaun\n\nOn Aug 5, 2009, at 8:21 PM, Sverre Rabbelier wrote:\n\n> Heya,\n>\n> On Wed, Aug 5, 2009 at 09:21, Shaun Cutts<shaun@cuttshome.net> wrote:\n>> PS I'm considering writing an extension to git where the \"diff\"  \n>> understands the\n>> semantics of certain types of files: hunks wouldn't just be textual  \n>> blobs but\n>> would try to represent a minimal change from one version to the  \n>> next based on an\n>> edit distance, so that, e.g. changing the location of a function  \n>> would be\n>> represented by a \"move\" edit, rather than two text changes.\n>\n> This sounds very similar to what Daniel was discussing in \"[PATCH 2/3\n> v3] Use an external program to implement fetching with curl git\" [0],\n> if you're truly interested in doing this, please do keep me posted\n> (and I suspect Dscho might also be interested in being cc-ed) :).\n>\n> [0] http://thread.gmane.org/gmane.comp.version-control.git/124503\n>\n> -- \n> Cheers,\n>\n> Sverre Rabbelier\n>\n"},{"id":"120401","messageId":"436D5ED1-2F0E-4227-AC4A-3A5FD16B2DCF@cuttshome.net","threadId":"20411","inReplyTo":"7v7hxi9m4k.fsf@alter.siamese.dyndns.org","subject":"Re: git index: how does it work?","fromName":"Shaun Cutts","fromEmail":"shaun@cuttshome.net","sentAt":"2009-08-12T11:52:31Z","receivedAt":"2009-08-12T11:52:31Z","isPatch":false,"sender":{"key":"shaun@cuttshome.net","avatar":"https://gravatar.com/avatar/6238ceb14c3a11096c8574e76b7e991a156522e521f0a9d099abed81c20c4451?d=mp&s=160"},"body":"Junio,\n\nYour advice was very helpful.\n\nDigging in, however, I find I still am in the dark on one point: how  \ndoes the index track renamed files, and how to query it for  \ninformation about them?\n\nFor instance, if I add a 5th step to the sequence:\n\n5) git mv foo.c bar.c\n\nThen I am told by \"git status\" that the file is renamed, but I can't  \nseem to elict this info using \"git ls-files\". Under some circumstances  \neven \"git status\" lists a new and deleted file after a rename.\n\nAre renames being tracked by the index, and is there a more basic  \ninterface than \"status\" to query about them?\n\nThanks for any help,\n--- Shaun\n\nOn Aug 5, 2009, at 8:00 PM, Junio C Hamano wrote:\n\n> Shaun Cutts <shaun@cuttshome.net> writes:\n>\n>> I am wondering if someone could explain and/or point me to an  \n>> explanation of how\n>> the git index works.\n>>\n>> For instance, suppose I have a tracked file: \"foo.c\"\n>>\n>> 1) [I modify \"foo.c\"]\n>> 2) git add foo.c\n>> 3) [modify again]\n>> 4) git commit -m \"blah blah\"\n>>\n>> Since I don't include the \"-a\" switch, the version I added on step  \n>> 2 is\n>> committed. But how does the index keep track of these changes? Does  \n>> the index\n>> file actually contain the hunks of \"foo.c\" that have been modified?  \n>> Or is there\n>> a \"temporary\" blob created, which the index points to?\n>\n> Step 2 hashes foo.c and creates a blob object and registers it to the\n> index.  Step 4 writes out the index as a tree and makes a commit out  \n> of\n> it.\n>\n> Running this sequence might be instructive.\n>\n> \t1$ edit foo.c\n>        2$ git add foo.c\n>        2a$ git ls-files -s foo.c\n> \t2b$ git diff foo.c\n>        2c$ git diff --cached foo.c\n>        3$ edit foo.c\n>        3a$ git ls-files -s foo.c\n> \t3b$ git diff foo.c\n>        3c$ git diff --cached foo.c\n>        4$ git commit -m 'half-edit of foo.c'\n>        4a$ git ls-files -s foo.c\n> \t4b$ git ls-tree HEAD foo.c\n>        4c$ git diff foo.c\n>        4d$ git diff --cached foo.c\n>\n> - 2a shows the actual blob object that was created out of foo.c in  \n> step 2.\n>\n> - 2b shows the difference between that blob (now in the index) and  \n> foo.c,\n>   which should be empty.\n>\n> - 2c shows the difference between the HEAD commit and the index, which\n>   should show your edit in step 1.\n>\n> - 3a shows the blob in the index; you haven't added, so it should show\n>   the same as 2a.\n>\n> - 3b shows the difference between the index and foo.c, which should  \n> show\n>   the edit in step 3.\n>\n> - 3c shows the difference between the HEAD commit and the index, which\n>   should show your edit in step 1.\n>\n> - 4a shows the blob in the index; you haven't added, so it should show\n>   the same as 2a.\n>\n> - 4b shows the blob in the committed tree and the blob object should  \n> be\n>   identical to 2a.\n>\n> - 4c shows the difference between the index and foo.c, which should  \n> show\n>   the edit in step 3.\n>\n> - 4d shows the difference between the HEAD commit and the index, which\n>   should now be empty.\n>\n"},{"id":"120427","messageId":"fabb9a1e0908121047jf7e6cf1n757cde626c79830c@mail.gmail.com","threadId":"20411","inReplyTo":"436D5ED1-2F0E-4227-AC4A-3A5FD16B2DCF@cuttshome.net","subject":"Re: git index: how does it work?","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2009-08-12T17:47:47Z","receivedAt":"2009-08-12T17:47:47Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Wed, Aug 12, 2009 at 04:52, Shaun Cutts<shaun@cuttshome.net> wrote:\n> Are renames being tracked by the index, and is there a more basic interface\n> than \"status\" to query about them?\n\nNope, git never explicitly tracks renames. Try this:\n$ mv foo bar\n$ git rm --cached foo\n$ git add bar\n$ git status\n\nIt'll tell you that you renamed foo to bar, even if you never executed 'git mv'.\n\nThis is because git does rename _detection_, that is, it'll notice\nthat you have another file with (almost) the same contents, so it\nassumes you did a rename.\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"120432","messageId":"410DBF0B-80A2-4235-B566-B622CA976DA0@cuttshome.net","threadId":"20411","inReplyTo":"fabb9a1e0908121047jf7e6cf1n757cde626c79830c@mail.gmail.com","subject":"Re: git index: how does it work?","fromName":"Shaun Cutts","fromEmail":"shaun@cuttshome.net","sentAt":"2009-08-12T18:45:48Z","receivedAt":"2009-08-12T18:45:48Z","isPatch":false,"sender":{"key":"shaun@cuttshome.net","avatar":"https://gravatar.com/avatar/6238ceb14c3a11096c8574e76b7e991a156522e521f0a9d099abed81c20c4451?d=mp&s=160"},"body":"Aha ---\n\nthat explains it, then.\n\nIs there a lower-level interface to rename detection than via  \n\"status\"? And... um... hmmm.... how does it work? The hash codes don't  \nhelp for \"almost\" the same. Is there an approximate string matching  \nalgorithm built in somewhere?\n\nThanks,\n\n-- Shaun\n\nOn Aug 12, 2009, at 7:47 PM, Sverre Rabbelier wrote:\n\n> Heya,\n>\n> On Wed, Aug 12, 2009 at 04:52, Shaun Cutts<shaun@cuttshome.net> wrote:\n>> Are renames being tracked by the index, and is there a more basic  \n>> interface\n>> than \"status\" to query about them?\n>\n> Nope, git never explicitly tracks renames. Try this:\n> $ mv foo bar\n> $ git rm --cached foo\n> $ git add bar\n> $ git status\n>\n> It'll tell you that you renamed foo to bar, even if you never  \n> executed 'git mv'.\n>\n> This is because git does rename _detection_, that is, it'll notice\n> that you have another file with (almost) the same contents, so it\n> assumes you did a rename.\n>\n> -- \n> Cheers,\n>\n> Sverre Rabbelier\n>\n"},{"id":"120435","messageId":"20090812193943.GA3718@atjola.homenet","threadId":"20411","inReplyTo":"410DBF0B-80A2-4235-B566-B622CA976DA0@cuttshome.net","subject":"Re: git index: how does it work?","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2009-08-12T19:39:43Z","receivedAt":"2009-08-12T19:39:43Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"[Please don't top-post, fixed that for you...]\n\nOn 2009.08.12 20:45:48 +0200, Shaun Cutts wrote:\n> On Aug 12, 2009, at 7:47 PM, Sverre Rabbelier wrote:\n> >Heya,\n> >\n> >On Wed, Aug 12, 2009 at 04:52, Shaun Cutts<shaun@cuttshome.net> wrote:\n> >>Are renames being tracked by the index, and is there a more\n> >>basic interface\n> >>than \"status\" to query about them?\n> >\n> >Nope, git never explicitly tracks renames. Try this:\n> >$ mv foo bar\n> >$ git rm --cached foo\n> >$ git add bar\n> >$ git status\n> >\n> >It'll tell you that you renamed foo to bar, even if you never\n> >executed 'git mv'.\n> >\n> >This is because git does rename _detection_, that is, it'll notice\n> >that you have another file with (almost) the same contents, so it\n> >assumes you did a rename.\n>\n> Aha ---\n> \n> that explains it, then.\n> \n> Is there a lower-level interface to rename detection than via\n> \"status\"? And... um... hmmm.... how does it work? The hash codes\n> don't help for \"almost\" the same. Is there an approximate string\n> matching algorithm built in somewhere?\n\nRoughly, it works like this:\nThe files are split into small chunks, those are hashed, and if there\nare chunks with the same hash in both files, those chunks are treated as\nbeing common to both files. The more the files have in common, the\nhigher the similarity score. See estimate_similarity() for details.\n\nAs the splitting also happens at newlines this has some interesting\neffects, for example, you can completely reorder the lines in a file\nafter renaming it, and git will still detect it as a rename.\n\nBjörn\n"},{"id":"120449","messageId":"7vab24ydsz.fsf@alter.siamese.dyndns.org","threadId":"20411","inReplyTo":"436D5ED1-2F0E-4227-AC4A-3A5FD16B2DCF@cuttshome.net","subject":"Re: git index: how does it work?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-08-12T20:31:24Z","receivedAt":"2009-08-12T20:31:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shaun Cutts <shaun@cuttshome.net> writes:\n\n> Are renames being tracked by the index, and is there a more basic\n> interface than \"status\" to query about them?\n\nNo.  Index, nor git in general, never records renames.  git records\ncontents, not content changes.  The index records a state, so does the\ntree object pointed at by the HEAD commit.\n\nWhen you ask for \"status\", git will notice that you have lost a file, and\nadded a new one, between these two states, by comparing them.  The\ncontents of these lost files and added files are then compared, and ones\nwith similar contents are paired up.\n\nThat way, you do not have to use \"git mv A B\" to \"rename\" A to B.  You can\njust as well \"mv A B; git rm A; git add B\", and get the same outcome,\nexactly because git does not record renames.\n\nInstead, we track them by deducing that you renamed from the result.\n\nThe tree-vs-index comparison \"git status\" does to figure all this out is\n\"git diff-index -M --cached HEAD\".\n\nAs it should be obvious from the above description,\n\n\tgit diff-index -M --cached HEAD -- A\n\nis *NOT* the way for you to ask about \"possible renames of A\".  You need\nto run the diff for the whole tree without path limitation so that you can\npair deletions and creations up in order to deduce renames.\n"}]}