{"thread":{"id":"38587","subject":"Get a git diff without taking index into account","startedAt":"2015-02-18T14:57:33Z","lastAt":"2015-02-18T22:30:13Z","messageCount":11,"participants":["Eric Frederich","Junio C Hamano","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"256271","messageId":"CAAoZyYN-ohiq-Od=u-cd5FRH8=NpJNGS+zEo+NYgwAK7Kjaz_w@mail.gmail.com","threadId":"38587","inReplyTo":null,"subject":"Get a git diff without taking index into account","fromName":"Eric Frederich","fromEmail":"eric.frederich@gmail.com","sentAt":"2015-02-18T14:57:33Z","receivedAt":"2015-02-18T14:57:33Z","isPatch":false,"sender":{"key":"eric.frederich@gmail.com","avatar":"https://gravatar.com/avatar/a047bc6d3404a87c923025574cdee9abf57ae9efc1d96c7bbedf30faf47ee12c?d=mp&s=160"},"body":"Some background.\nI'm trying to use Git as an object store for trees.\nI put trees into the repo and can retrieve them.\nI'm having issues with diffing these trees that I retrieve from the repo.\nIf I use a \"git checkout\" the diffs seem to work but if I create the\ntree myself user lower level ls-tree and cat-file commands then the\ndiff doesn't work.\nIt seems to take the index into account.\n\nBelow is a complete working example.\nShould be able to copy / paste line by line.\nI am trying to run the diff of form:\n git diff [--options] [--] [...]\nwhere it says it does a diff from working tree to a commit\n\nMaybe git is interpreting my command as one of the other forms?\nCan someone help me understand what is going on?\n\n#\n# EXAMPLE\n#\n\n# cleanup and create dummy data\nrm -rf /tmp/mydatastore && mkdir -p /tmp/mydatastore\nrm -rf /tmp/test /tmp/test2 && mkdir -p /tmp/test/d1 /tmp/test2\necho \"this is f1\" > /tmp/test/f1\necho \"this is f2\" > /tmp/test/d1/f2\n\n# create a new branch called test with data from /tmp/test\ngit --git-dir=/tmp/mydatastore/.db init --bare\ngit --git-dir=/tmp/mydatastore/.db hash-object -w /tmp/test/d1/f2 /tmp/test/f1\necho -e \"100644 blob c837441e09d13d3a0a2d906d7c3813adda504833\\tf2\" |\ngit --git-dir=/tmp/mydatastore/.db mktree --batch\necho -e \"100644 blob\n11ac5613caf504eec18b2e60f1e6b3c598b085eb\\tf1\\n40755 tree\n055f1133fbc9872f3093cca5f182b16611e6789a\\td1\" | git\n--git-dir=/tmp/mydatastore/.db mktree\ncommit_sha=`git --git-dir=/tmp/mydatastore/.db commit-tree -m \"initial\ncommit\" c427094b22e74d1eaeebdc9e49e6790b5b6a706a`\ngit --git-dir=/tmp/mydatastore/.db update-ref refs/heads/test $commit_sha\n\n# why does this show diffs?\ngit --git-dir=/tmp/mydatastore/.db --work-tree=/tmp/test diff $commit_sha\n\n# after doing a checkout somewhere else it doesn't show any diffs\ngit --git-dir=/tmp/mydatastore/.db --work-tree=/tmp/test2 checkout test .\ngit --git-dir=/tmp/mydatastore/.db --work-tree=/tmp/test diff $commit_sha\n\n# remove the index and it shows diffs again\nrm /tmp/mydatastore/.db/index\ngit --git-dir=/tmp/mydatastore/.db --work-tree=/tmp/test diff $commit_sha\n\n# it was my understanding from \"git help diff\" that this form of diff\nis supposed to\n# compare a work tree against a commit or branch and not take into\naccount the index.\n# Clearly it takes into account the index because we get different\nresults with and without it\n"},{"id":"256272","messageId":"CAAoZyYPhiKX1F5ymdSijR7=e8CT1sqaomehBjt-NVDz_A4V4UA@mail.gmail.com","threadId":"38587","inReplyTo":"CAAoZyYN-ohiq-Od=u-cd5FRH8=NpJNGS+zEo+NYgwAK7Kjaz_w@mail.gmail.com","subject":"Re: Get a git diff without taking index into account","fromName":"Eric Frederich","fromEmail":"eric.frederich@gmail.com","sentAt":"2015-02-18T15:06:30Z","receivedAt":"2015-02-18T15:06:30Z","isPatch":false,"sender":{"key":"eric.frederich@gmail.com","avatar":"https://gravatar.com/avatar/a047bc6d3404a87c923025574cdee9abf57ae9efc1d96c7bbedf30faf47ee12c?d=mp&s=160"},"body":"More concise example:\n\ncd /tmp\ngit clone --bare https://github.com/defnull/bottle.git\nmkdir /tmp/bottlecopy\ngit --git-dir=/tmp/bottle.git --work-tree=/tmp/bottlecopy checkout master .\n\n# this shows no diffs\ngit --git-dir=/tmp/bottle.git --work-tree=/tmp/bottlecopy diff master\nrm /tmp/bottle.git/index\n\n# why does this one now show a diff?\n# how can I compare a working directory to a commit without taking the\nindex into account?\ngit --git-dir=/tmp/bottle.git --work-tree=/tmp/bottlecopy diff master\n\nOn Wed, Feb 18, 2015 at 9:57 AM, Eric Frederich\n wrote:\n> Some background.\n> I'm trying to use Git as an object store for trees.\n> I put trees into the repo and can retrieve them.\n> I'm having issues with diffing these trees that I retrieve from the repo.\n> If I use a \"git checkout\" the diffs seem to work but if I create the\n> tree myself user lower level ls-tree and cat-file commands then the\n> diff doesn't work.\n> It seems to take the index into account.\n>\n> Below is a complete working example.\n> Should be able to copy / paste line by line.\n> I am trying to run the diff of form:\n> git diff [--options] [--] [...]\n> where it says it does a diff from working tree to a commit\n>\n> Maybe git is interpreting my command as one of the other forms?\n> Can someone help me understand what is going on?\n>\n> #\n> # EXAMPLE\n> #\n>\n> # cleanup and create dummy data\n> rm -rf /tmp/mydatastore && mkdir -p /tmp/mydatastore\n> rm -rf /tmp/test /tmp/test2 && mkdir -p /tmp/test/d1 /tmp/test2\n> echo \"this is f1\" > /tmp/test/f1\n> echo \"this is f2\" > /tmp/test/d1/f2\n>\n> # create a new branch called test with data from /tmp/test\n> git --git-dir=/tmp/mydatastore/.db init --bare\n> git --git-dir=/tmp/mydatastore/.db hash-object -w /tmp/test/d1/f2 /tmp/test/f1\n> echo -e \"100644 blob c837441e09d13d3a0a2d906d7c3813adda504833\\tf2\" |\n> git --git-dir=/tmp/mydatastore/.db mktree --batch\n> echo -e \"100644 blob\n> 11ac5613caf504eec18b2e60f1e6b3c598b085eb\\tf1\\n40755 tree\n> 055f1133fbc9872f3093cca5f182b16611e6789a\\td1\" | git\n> --git-dir=/tmp/mydatastore/.db mktree\n> commit_sha=`git --git-dir=/tmp/mydatastore/.db commit-tree -m \"initial\n> commit\" c427094b22e74d1eaeebdc9e49e6790b5b6a706a`\n> git --git-dir=/tmp/mydatastore/.db update-ref refs/heads/test $commit_sha\n>\n> # why does this show diffs?\n> git --git-dir=/tmp/mydatastore/.db --work-tree=/tmp/test diff $commit_sha\n>\n> # after doing a checkout somewhere else it doesn't show any diffs\n> git --git-dir=/tmp/mydatastore/.db --work-tree=/tmp/test2 checkout test .\n> git --git-dir=/tmp/mydatastore/.db --work-tree=/tmp/test diff $commit_sha\n>\n> # remove the index and it shows diffs again\n> rm /tmp/mydatastore/.db/index\n> git --git-dir=/tmp/mydatastore/.db --work-tree=/tmp/test diff $commit_sha\n>\n> # it was my understanding from \"git help diff\" that this form of diff\n> is supposed to\n> # compare a work tree against a commit or branch and not take into\n> account the index.\n> # Clearly it takes into account the index because we get different\n> results with and without it\n"},{"id":"256273","messageId":"CAPc5daU9km+gr-DHJzJF59mugwGeNX69H27E_DaoyBZnuzoiFw@mail.gmail.com","threadId":"38587","inReplyTo":"CAAoZyYPhiKX1F5ymdSijR7=e8CT1sqaomehBjt-NVDz_A4V4UA@mail.gmail.com","subject":"Re: Get a git diff without taking index into account","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-02-18T15:37:05Z","receivedAt":"2015-02-18T15:37:05Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"On Wed, Feb 18, 2015 at 7:06 AM, Eric Frederich\n wrote:\n> # how can I compare a working directory to a commit without taking the\n> index into account?\n\nYou don't.\n"},{"id":"256274","messageId":"CAAoZyYPVopmP_bv7EZS912R4bxpzNm49_q0XXZXqa52dTDDM2Q@mail.gmail.com","threadId":"38587","inReplyTo":"CAPc5daU9km+gr-DHJzJF59mugwGeNX69H27E_DaoyBZnuzoiFw@mail.gmail.com","subject":"Re: Get a git diff without taking index into account","fromName":"Eric Frederich","fromEmail":"eric.frederich@gmail.com","sentAt":"2015-02-18T15:42:20Z","receivedAt":"2015-02-18T15:42:20Z","isPatch":false,"sender":{"key":"eric.frederich@gmail.com","avatar":"https://gravatar.com/avatar/a047bc6d3404a87c923025574cdee9abf57ae9efc1d96c7bbedf30faf47ee12c?d=mp&s=160"},"body":"This is from \"git help diff\". It seems to imply that I should be able to do it.\nIt mentions nothing of the index.\n\n git diff [--options] [--] [...]\n This form is to view the changes you have in your working\ntree relative to the named . You can use HEAD to compare it\nwith the latest commit, or a branch name\n to compare with the tip of a different branch.\n\nOn Wed, Feb 18, 2015 at 10:37 AM, Junio C Hamano wrote:\n> On Wed, Feb 18, 2015 at 7:06 AM, Eric Frederich\n> wrote:\n>> # how can I compare a working directory to a commit without taking the\n>> index into account?\n>\n> You don't.\n"},{"id":"256279","messageId":"xmqqfva341sf.fsf@gitster.dls.corp.google.com","threadId":"38587","inReplyTo":"CAAoZyYPVopmP_bv7EZS912R4bxpzNm49_q0XXZXqa52dTDDM2Q@mail.gmail.com","subject":"Re: Get a git diff without taking index into account","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-02-18T16:33:52Z","receivedAt":"2015-02-18T16:33:52Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Frederich writes:\n\n> This is from \"git help diff\". It seems to imply that I should be able to do it.\n> It mentions nothing of the index.\n\nMost of the documentation on early subcommands (and \"git diff\"\ncertainly is one of the early subcommands) were written back when\neverybody knew that Git almost always talks about _tracked_ files\nthat are known to the index, and the only time it even cares about\nuntracked ones that are not in the index was when it tries to help\nusers by reminding what the user may have forgot to \"git add\".\n\nDocumentation pages do not bother repeating \"this only looks at\ntracked paths\" for this reason; Git is about tracked files by\ndefault.\n\nPerhaps you can suggest how to improve the description of commands\nwithout being too repetitive?\n\nThanks.\n"},{"id":"256288","messageId":"CAAoZyYOst-5cD7qtV=T3Oahja1JN1ZmeyAcELrn7xD0bMc7Mrg@mail.gmail.com","threadId":"38587","inReplyTo":"xmqqfva341sf.fsf@gitster.dls.corp.google.com","subject":"Re: Get a git diff without taking index into account","fromName":"Eric Frederich","fromEmail":"eric.frederich@gmail.com","sentAt":"2015-02-18T18:27:50Z","receivedAt":"2015-02-18T18:27:50Z","isPatch":false,"sender":{"key":"eric.frederich@gmail.com","avatar":"https://gravatar.com/avatar/a047bc6d3404a87c923025574cdee9abf57ae9efc1d96c7bbedf30faf47ee12c?d=mp&s=160"},"body":"Thanks for the reply.\nMy immediate concern is not to fix the documentation but to get some\nsort of status or diff.\nI want to avoid using an index because I want to allow multiple\nprocesses to do different diffs at the same time.\n\nRight now I can put trees into the repo and get trees out without\nusing the index but I had to write routines that use the lower level\ncommands.\nIf I could use the index it would be a couple of out of the box\ncommands and I wouldn't have to write my own routine.\nWhile I've already written ones for \"git add\" and \"git checkout\" I'm\ntrying to avoid writing one for \"git diff\"\n\nI looks like this is unavoidable.\nOh well... thanks for the help.\n\n\n\nOn Wed, Feb 18, 2015 at 11:33 AM, Junio C Hamano wrote:\n> Eric Frederich writes:\n>\n>> This is from \"git help diff\". It seems to imply that I should be able to do it.\n>> It mentions nothing of the index.\n>\n> Most of the documentation on early subcommands (and \"git diff\"\n> certainly is one of the early subcommands) were written back when\n> everybody knew that Git almost always talks about _tracked_ files\n> that are known to the index, and the only time it even cares about\n> untracked ones that are not in the index was when it tries to help\n> users by reminding what the user may have forgot to \"git add\".\n>\n> Documentation pages do not bother repeating \"this only looks at\n> tracked paths\" for this reason; Git is about tracked files by\n> default.\n>\n> Perhaps you can suggest how to improve the description of commands\n> without being too repetitive?\n>\n> Thanks.\n"},{"id":"256289","messageId":"20150218183251.GB6346@peff.net","threadId":"38587","inReplyTo":"CAAoZyYOst-5cD7qtV=T3Oahja1JN1ZmeyAcELrn7xD0bMc7Mrg@mail.gmail.com","subject":"Re: Get a git diff without taking index into account","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-02-18T18:32:51Z","receivedAt":"2015-02-18T18:32:51Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 18, 2015 at 01:27:50PM -0500, Eric Frederich wrote:\n\n> My immediate concern is not to fix the documentation but to get some\n> sort of status or diff.\n> I want to avoid using an index because I want to allow multiple\n> processes to do different diffs at the same time.\n\nIf you only have one working tree, can't all of the processes use the\nsame index (that matches the working tree) and do different diffs\nagainst it?\n\nIf you have multiple working trees, can you use one index per working\ntree, and specify it using GIT_INDEX_FILE?\n\nIf you can persist the index file for each working tree, this will be\nmuch faster in the long run, too (you can just refresh the index before\neach diff, which means that git does not have to actually open the files\nin most cases; we can compare their stat information to what is in the\nindex, and then the index sha1 with what is in the tree).\n\n-Peff\n"},{"id":"256308","messageId":"CAAoZyYNY5V=WNf9dSG1p2iQqk11CjDB9Qm=t9mv97vgjY_ZWHA@mail.gmail.com","threadId":"38587","inReplyTo":"20150218183251.GB6346@peff.net","subject":"Re: Get a git diff without taking index into account","fromName":"Eric Frederich","fromEmail":"eric.frederich@gmail.com","sentAt":"2015-02-18T19:36:39Z","receivedAt":"2015-02-18T19:36:39Z","isPatch":false,"sender":{"key":"eric.frederich@gmail.com","avatar":"https://gravatar.com/avatar/a047bc6d3404a87c923025574cdee9abf57ae9efc1d96c7bbedf30faf47ee12c?d=mp&s=160"},"body":"Thanks Jeff.\n\nI recognize your picture from here...\n http://git.661346.n2.nabble.com/push-race-td7569254.html\n... which helped me figure out how two processes trying to update a\nref at the same time works out.\n\nI will try using a separate GIT_INDEX_FILE for each working tree.\n\nI'm not certain that what I'm trying to do is even a good idea.\n\nWe have a system which is the official storage of things we'll just\ncall \"items\".\nThese items have one or more \"revisions\". These revisions can have\nall sorts of relationships to other revisions.\nOnce a revision is released it is locked down and cannot be changed\nincluding the revisions.\n\nRather than working directly with this official storage system, we\nwant our application to work against a concept of a \"local workspace\".\nThis is where I want to use Git.\nWe can map a released revision to a tree structure by getting all the\nfiles, serializing all the attributes, relation data, etc.\nThat tree structure is what I would store in Git.\n\nMy options seem to be\n 1) use a single Git repo to store all items in a disconnected manner\n(each item has a branch disconnected (orphaned?) from the other\nbranches)\n 2) each item gets its own Git repo\n 3) use a single Git repo to store all items but have them all\ntogether in a workspace\n\nI'm pursuing option (1) right now and trying to see how much work it would take.\nWith option (2) I think that would limit my ability to send a bunch of\nitems from one repo to another.\nOption (3) doesn't really map to the system we're trying to mimic\nbecause releases are done at the \"item revision\" level, not at a\nhigher workspace level.\n\nOn Wed, Feb 18, 2015 at 1:32 PM, Jeff King wrote:\n> On Wed, Feb 18, 2015 at 01:27:50PM -0500, Eric Frederich wrote:\n>\n>> My immediate concern is not to fix the documentation but to get some\n>> sort of status or diff.\n>> I want to avoid using an index because I want to allow multiple\n>> processes to do different diffs at the same time.\n>\n> If you only have one working tree, can't all of the processes use the\n> same index (that matches the working tree) and do different diffs\n> against it?\n>\n> If you have multiple working trees, can you use one index per working\n> tree, and specify it using GIT_INDEX_FILE?\n>\n> If you can persist the index file for each working tree, this will be\n> much faster in the long run, too (you can just refresh the index before\n> each diff, which means that git does not have to actually open the files\n> in most cases; we can compare their stat information to what is in the\n> index, and then the index sha1 with what is in the tree).\n>\n> -Peff\n"},{"id":"256315","messageId":"CAAoZyYOT_OQZ+rrwFvnsVBV_sYRA4Oti3vRNnE6_RaV9w8qxdg@mail.gmail.com","threadId":"38587","inReplyTo":"20150218183251.GB6346@peff.net","subject":"Re: Get a git diff without taking index into account","fromName":"Eric Frederich","fromEmail":"eric.frederich@gmail.com","sentAt":"2015-02-18T21:38:55Z","receivedAt":"2015-02-18T21:38:55Z","isPatch":false,"sender":{"key":"eric.frederich@gmail.com","avatar":"https://gravatar.com/avatar/a047bc6d3404a87c923025574cdee9abf57ae9efc1d96c7bbedf30faf47ee12c?d=mp&s=160"},"body":"On Wed, Feb 18, 2015 at 1:32 PM, Jeff King wrote:\n> On Wed, Feb 18, 2015 at 01:27:50PM -0500, Eric Frederich wrote:\n>\n> If you can persist the index file for each working tree, this will be\n> much faster in the long run, too (you can just refresh the index before\n> each diff, which means that git does not have to actually open the files\n> in most cases; we can compare their stat information to what is in the\n> index, and then the index sha1 with what is in the tree).\n\nCould you elaborate on \"you can just refresh the index before each diff\"\nWhat command would I use to do this?\nI don't want to store some object just to get a diff of it.\n\nAlso, how would I go about detecting untracked files the way status does?\nThere is no way to specify a HEAD per git command using switches or\nenvironment variables.\nI can't change the HEAD of the Git repo because other processes may be\nusing it at the same time.\n"},{"id":"256318","messageId":"CAPc5daUw8F6fFZk0+-_9Z5nxPG4D56P88yFz=BSfK+xYxLhBxg@mail.gmail.com","threadId":"38587","inReplyTo":"CAAoZyYOT_OQZ+rrwFvnsVBV_sYRA4Oti3vRNnE6_RaV9w8qxdg@mail.gmail.com","subject":"Re: Get a git diff without taking index into account","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-02-18T22:16:52Z","receivedAt":"2015-02-18T22:16:52Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"On Wed, Feb 18, 2015 at 1:38 PM, Eric Frederich\n wrote:\n> Could you elaborate on \"you can just refresh the index before each diff\"\n> What command would I use to do this?\n\n\"update-index --refresh\", perhaps?\n\n> Also, how would I go about detecting untracked files the way status does?\n\n\"ls-files\"?\n"},{"id":"256319","messageId":"20150218223013.GA16287@peff.net","threadId":"38587","inReplyTo":"CAAoZyYOT_OQZ+rrwFvnsVBV_sYRA4Oti3vRNnE6_RaV9w8qxdg@mail.gmail.com","subject":"Re: Get a git diff without taking index into account","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-02-18T22:30:13Z","receivedAt":"2015-02-18T22:30:13Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 18, 2015 at 04:38:55PM -0500, Eric Frederich wrote:\n\n> Also, how would I go about detecting untracked files the way status does?\n> There is no way to specify a HEAD per git command using switches or\n> environment variables.\n> I can't change the HEAD of the Git repo because other processes may be\n> using it at the same time.\n\nUntracked files are a function of the index, not of the HEAD. So you\nwould load whatever tree you like into your index (either \"the\" index,\nor a temporary one you specify with GIT_INDEX_FILE), and then \"git\nls-files -o\".\n\n-Peff\n"}]}