{"thread":{"id":"31242","subject":"git diff vs git diff-files","startedAt":"2012-08-13T08:55:11Z","lastAt":"2012-08-14T13:11:46Z","messageCount":9,"participants":["Bernd Jendrissek","Thomas Rast","Matthieu Moy","Andreas Schwab"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"196906","messageId":"CAF7PVPrex6sJMU+uvxQOSpB12qSTp2barTycow_-LZ8Ngbv7uQ@mail.gmail.com","threadId":"31242","inReplyTo":null,"subject":"git diff vs git diff-files","fromName":"Bernd Jendrissek","fromEmail":"bernd.jendrissek@gmail.com","sentAt":"2012-08-13T08:55:11Z","receivedAt":"2012-08-13T08:55:11Z","isPatch":false,"sender":{"key":"bernd.jendrissek@gmail.com","avatar":null},"body":"I have a billion testsuite golden output files that have changed due\nto an explicit ordering of objects I've imposed on output. A helper\nscript I wrote to help parse the diffs (to ignore order-only\ndifferences) noticed that one hunk had a different number of additions\nand deletions. I'm manually copying and pasting the hunks from git add\n-i into the script. However, when I used git diff to get at the\nchanges, to discover which line was the offending one, all additions\nand deletions were exactly matched!\n\nWith strace I noticed that git add -i calls git diff-files, and that's\nabout as far as I can conveniently trace where things are going weird.\nWeird:\n\n$ /usr/local/git/bin/git diff-files -p --color -- TwoStageAmp-output.net\ndiff --git a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\nb/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\nindex a5ee1e7..a9f3620 100644\n--- a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n+++ b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n@@ -1,47 +1,47 @@\n-Part unknown { Name Cout }\n-Part unknown { Name R5 }\n-Part unknown { Name R4 }\n-Part unknown { Name RE2 }\n-Part unknown { Name Q2 }\n+Part unknown { Name R8 }\n Part unknown { Name A3 }\n-Part unknown { Name R3 }\n Part unknown { Name A2 }\n-Part unknown { Name RE1 }\n Part unknown { Name A1 }\n[snip]\n\n$ /usr/local/git/bin/git diff TwoStageAmp-output.netdiff --git\na/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\nb/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.n\nindex a5ee1e7..a9f3620 100644\n--- a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n+++ b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n@@ -1,47 +1,47 @@\n-Part unknown { Name Cout }\n-Part unknown { Name R5 }\n-Part unknown { Name R4 }\n-Part unknown { Name RE2 }\n-Part unknown { Name Q2 }\n+Part unknown { Name R8 }\n Part unknown { Name A3 }\n-Part unknown { Name R3 }\n Part unknown { Name A2 }\n-Part unknown { Name RE1 }\n-Part unknown { Name Q1 }  <--- Not present with diff-files output\n Part unknown { Name A1 }\n[snip]\n\nThe index is clean on TwoStageAmp-output.net (but by now a million\nother files are dirty in the index). While I discovered this issue\nwith Ubuntu's 1.7.1, I got the same behaviour with git\n1.7.12.rc2.18.g61b472e\n\nWhat's going on here? I'll leave my tree in whatever weird state it\nmight be, for a while, in case anyone wants me to try various things.\nAt this point I want to understand what's going on, in case there's\neither a learning moment for me or a bug, rather than just finding a\nworkaround.\n"},{"id":"196911","messageId":"87fw7qhm31.fsf@thomas.inf.ethz.ch","threadId":"31242","inReplyTo":"CAF7PVPrex6sJMU+uvxQOSpB12qSTp2barTycow_-LZ8Ngbv7uQ@mail.gmail.com","subject":"Re: git diff vs git diff-files","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-08-13T15:02:10Z","receivedAt":"2012-08-13T15:02:10Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Bernd Jendrissek <bernd.jendrissek@gmail.com> writes:\n\n> $ /usr/local/git/bin/git diff-files -p --color -- TwoStageAmp-output.net\n> diff --git a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n> b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n> index a5ee1e7..a9f3620 100644\n> --- a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n> +++ b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n> @@ -1,47 +1,47 @@\n> -Part unknown { Name Cout }\n> -Part unknown { Name R5 }\n> -Part unknown { Name R4 }\n> -Part unknown { Name RE2 }\n> -Part unknown { Name Q2 }\n> +Part unknown { Name R8 }\n>  Part unknown { Name A3 }\n> -Part unknown { Name R3 }\n>  Part unknown { Name A2 }\n> -Part unknown { Name RE1 }\n>  Part unknown { Name A1 }\n> [snip]\n>\n> $ /usr/local/git/bin/git diff TwoStageAmp-output.netdiff --git\n> a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n> b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.n\n> index a5ee1e7..a9f3620 100644\n> --- a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n> +++ b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n> @@ -1,47 +1,47 @@\n> -Part unknown { Name Cout }\n> -Part unknown { Name R5 }\n> -Part unknown { Name R4 }\n> -Part unknown { Name RE2 }\n> -Part unknown { Name Q2 }\n> +Part unknown { Name R8 }\n>  Part unknown { Name A3 }\n> -Part unknown { Name R3 }\n>  Part unknown { Name A2 }\n> -Part unknown { Name RE1 }\n> -Part unknown { Name Q1 }  <--- Not present with diff-files output\n>  Part unknown { Name A1 }\n> [snip]\n\nCan you share this repository?  Or at least the pre- and post-change\nfiles, transferred in such a way that there won't be any whitespace\ndamage (your snippets above show obvious damage).  You can use\n\n  git cat-file blob :gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n\nto get at the contents of the version in the index.\n\nDo you have any diff config that could be of interest?  A textconv\nfilter would be an obvious example that could produce the above, but\nperhaps you could just look at\n\n  git config --get-regexp '.*diff.*'\n  grep diff .gitattributes .git/info/attributes\n\netc.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"196912","messageId":"vpq1uja25b1.fsf@bauges.imag.fr","threadId":"31242","inReplyTo":"CAF7PVPrex6sJMU+uvxQOSpB12qSTp2barTycow_-LZ8Ngbv7uQ@mail.gmail.com","subject":"Re: git diff vs git diff-files","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2012-08-13T15:13:38Z","receivedAt":"2012-08-13T15:13:38Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Bernd Jendrissek <bernd.jendrissek@gmail.com> writes:\n\n> $ /usr/local/git/bin/git diff-files -p --color -- TwoStageAmp-output.net\n> diff --git a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n> b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n> index a5ee1e7..a9f3620 100644\n\n> $ /usr/local/git/bin/git diff TwoStageAmp-output.netdiff --git\n> a/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n> b/gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.n\n> index a5ee1e7..a9f3620 100644\n\nWhat's surprising is that both diff show the same \"index\" line, so both\ncommands actually diff the same content, and then show a different\noutput.\n\nYou can try something like\n\ngit cat-file blob a5ee1e7 > /tmp/staged.txt\ndiff -u /tmp/staged.txt TwoStageAmp-output.netdiff\n\nto recover the content from the index (Thomas' version should also work\nand give the same result), and use another diff tool. This \"diff -u\"\nshould give you an output similar to one of \"git diff\" and \"git\ndiff-files\" (my understanding is that it should match diff-files, and\n\"git diff\" is the one being wrong here).\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"196913","messageId":"CAF7PVPqyaZQtca0KfWHirBY2Dvdtn2RT-_mxR8x5uUNsfbHmdQ@mail.gmail.com","threadId":"31242","inReplyTo":"87fw7qhm31.fsf@thomas.inf.ethz.ch","subject":"Re: git diff vs git diff-files","fromName":"Bernd Jendrissek","fromEmail":"bernd.jendrissek@gmail.com","sentAt":"2012-08-13T16:06:04Z","receivedAt":"2012-08-13T16:06:04Z","isPatch":false,"sender":{"key":"bernd.jendrissek@gmail.com","avatar":null},"body":"On Mon, Aug 13, 2012 at 5:02 PM, Thomas Rast <trast@student.ethz.ch> wrote:\n> Can you share this repository?\n\nThis weird behaviour doesn't even survive making a copy (cp -a) of the\nwhole repository, so I very much doubt making it available would be\nilluminative. My disk's SMART data seems okay. The weird-quotient just\nrose a bit.\n\nBesides, .git/ is 60MB and my upload speed is 128kbps. A bit inconvenient.\n\n> Or at least the pre- and post-change\n> files, transferred in such a way that there won't be any whitespace\n> damage (your snippets above show obvious damage).  You can use\n\nhttp://www.bpj-code.co.za/downloads.php/bugs/TwoStageAmp-output.net?text\ncontains the output from git show a5ee1e7. Leave off the ?text for an\napplication/octet-stream download.\n\nThat file (with the Q1 line present) is consistent with the earlier\ncommit that added the file. It's diff-files that's lying.\n\n> Do you have any diff config that could be of interest?  A textconv\n> filter would be an obvious example that could produce the above, but\n> perhaps you could just look at\n\nNothing that fancy. I have just diff.color = auto and user.* = blah in\nglobal config, and similarly benign config in the repository.\n"},{"id":"196961","messageId":"87vcgl999v.fsf@thomas.inf.ethz.ch","threadId":"31242","inReplyTo":"CAF7PVPqyaZQtca0KfWHirBY2Dvdtn2RT-_mxR8x5uUNsfbHmdQ@mail.gmail.com","subject":"Re: git diff vs git diff-files","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-08-14T08:18:20Z","receivedAt":"2012-08-14T08:18:20Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Bernd Jendrissek <bernd.jendrissek@gmail.com> writes:\n\n> On Mon, Aug 13, 2012 at 5:02 PM, Thomas Rast <trast@student.ethz.ch> wrote:\n>> Can you share this repository?\n>\n> This weird behaviour doesn't even survive making a copy (cp -a) of the\n> whole repository, so I very much doubt making it available would be\n> illuminative. My disk's SMART data seems okay. The weird-quotient just\n> rose a bit.\n\nThat sounds like an issue with stat() data, and then it doesn't.  cp -a\nwould change all the inode numbers, triggering a full refresh of the\nindex in 'git diff'.  But I'm not sure exactly how this can lead to\ndisagreement, since both commands read *both* index and worktree version\nof the file.  Very confusing.\n\nCan you try the following:\n\n  git ls-files --debug gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n  cp .git/index .git/index.orig\n  touch gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n  # note, it is important that you run diff first\n  git diff gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n  git diff-files -p gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n  git ls-files --debug gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n\n>> Or at least the pre- and post-change\n>> files, transferred in such a way that there won't be any whitespace\n>> damage (your snippets above show obvious damage).  You can use\n>\n> http://www.bpj-code.co.za/downloads.php/bugs/TwoStageAmp-output.net?text\n> contains the output from git show a5ee1e7. Leave off the ?text for an\n> application/octet-stream download.\n\nUmm, that's only one side of the diff, isn't it?  The hunk header claims\nthat the hunk goes up to line 47, so there must be more changes beyond\nwhat you showed.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"196966","messageId":"CAF7PVPq0Wjck+53G4=Ofz0drMdx9NtpfgzMFR4j=32N-ez2xBg@mail.gmail.com","threadId":"31242","inReplyTo":"87vcgl999v.fsf@thomas.inf.ethz.ch","subject":"Re: git diff vs git diff-files","fromName":"Bernd Jendrissek","fromEmail":"bernd.jendrissek@gmail.com","sentAt":"2012-08-14T09:15:26Z","receivedAt":"2012-08-14T09:15:26Z","isPatch":false,"sender":{"key":"bernd.jendrissek@gmail.com","avatar":null},"body":"On Tue, Aug 14, 2012 at 10:18 AM, Thomas Rast <trast@student.ethz.ch> wrote:\n> That sounds like an issue with stat() data, and then it doesn't.  cp -a\n> would change all the inode numbers, triggering a full refresh of the\n> index in 'git diff'.  But I'm not sure exactly how this can lead to\n> disagreement, since both commands read *both* index and worktree version\n> of the file.  Very confusing.\n>\n> Can you try the following:\n>\n>   git ls-files --debug gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n>   cp .git/index .git/index.orig\n>   touch gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n>   # note, it is important that you run diff first\n>   git diff gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n>   git diff-files -p gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n>   git ls-files --debug gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n\nI'm can't tell what this was supposed to prove, but I think it's\nproven that I'm in the wrong mailing list, and that I should be taking\nthis up with gnome-terminal. The Q1 part deletion line seems to be\nappearing (in the terminal) only after I've copy&pasted it into\nanother terminal.\n\nRunning the same commands in xterm (correctly) shows the Q1 part\ndeletion line without having to copy & paste.\n\nAt this point I doubt there's much value in publishing the output, but\njust in case (and it may be relevant when i take it to\ngnome-terminal), it's here:\n\nhttp://www.bpj-code.co.za/downloads.php/bugs/git-output-87vcgl999v.fsf?text\n\nNote that line 104 of that output appears in the same terminal from\nwhich I copy&paste it only after I switch back to it, after having\npasted the output. Oi vey.\n\n>> http://www.bpj-code.co.za/downloads.php/bugs/TwoStageAmp-output.net?text\n>> contains the output from git show a5ee1e7. Leave off the ?text for an\n>> application/octet-stream download.\n>\n> Umm, that's only one side of the diff, isn't it?  The hunk header claims\n> that the hunk goes up to line 47, so there must be more changes beyond\n> what you showed.\n\nYou're right - the hunk isn't even self-consistent!\nhttp://www.bpj-code.co.za/downloads.php/bugs/git-add--p-TwoStageAmp-output.net?text\nis the \"bad\" diff, and I count only 46 lines of (context + deletions),\nwhile the hunk header claims the (correct) 47 lines corresponding to\nthe whole file covered in the hunk.\n\nThanks for your time; I'm fairly convinced now that git is just an\ninnocent accessory to another program's bug.\n"},{"id":"196967","messageId":"87393pztk4.fsf@thomas.inf.ethz.ch","threadId":"31242","inReplyTo":"CAF7PVPq0Wjck+53G4=Ofz0drMdx9NtpfgzMFR4j=32N-ez2xBg@mail.gmail.com","subject":"Re: git diff vs git diff-files","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-08-14T09:55:39Z","receivedAt":"2012-08-14T09:55:39Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Bernd Jendrissek <bernd.jendrissek@gmail.com> writes:\n\n> On Tue, Aug 14, 2012 at 10:18 AM, Thomas Rast <trast@student.ethz.ch> wrote:\n>> Can you try the following:\n>>\n>>   git ls-files --debug gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n>>   cp .git/index .git/index.orig\n>>   touch gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n>>   # note, it is important that you run diff first\n>>   git diff gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n>>   git diff-files -p gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n>>   git ls-files --debug gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net\n>\n> I'm can't tell what this was supposed to prove, but I think it's\n> proven that I'm in the wrong mailing list, and that I should be taking\n> this up with gnome-terminal. The Q1 part deletion line seems to be\n> appearing (in the terminal) only after I've copy&pasted it into\n> another terminal.\n>\n> Running the same commands in xterm (correctly) shows the Q1 part\n> deletion line without having to copy & paste.\n\nFun stuff.  This will be very interesting to debug.  It's still\npossible, though perhaps not extremely likely, that git is writing\ngarbage to the terminal and it just happens to work for xterm.\n\nWhatever you do next, *please* post the *exact* file contents *and*\noutput in a format that does not suffer any transport damage, neither to\nwhitespace nor to binary data (such as terminal escapes).  Piping\nthrough xxd comes to mind, perhaps by using\n\n  GIT_PAGER=xxd git -p diff ....  >diff-hexdump\n\netc. to ensure that git takes the same code paths as when writing to\nless.\n\nCut&paste from a terminal will at best give you exactly what the\nterminal figures it is *currently displaying*, which is completely\ndifferent from what programs write to it.  At worst, it will also suffer\nwhitespace damage.  So pretty much everything you showed so far, except\nfor the raw data of one (why only one?!) side of the diff, is useless.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"196969","messageId":"m2mx1xd929.fsf@igel.home","threadId":"31242","inReplyTo":"87393pztk4.fsf@thomas.inf.ethz.ch","subject":"Re: git diff vs git diff-files","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2012-08-14T11:09:18Z","receivedAt":"2012-08-14T11:09:18Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"Thomas Rast <trast@student.ethz.ch> writes:\n\n> Whatever you do next, *please* post the *exact* file contents *and*\n> output in a format that does not suffer any transport damage, neither to\n> whitespace nor to binary data (such as terminal escapes).  Piping\n> through xxd comes to mind, perhaps by using\n>\n>   GIT_PAGER=xxd git -p diff ....  >diff-hexdump\n>\n> etc. to ensure that git takes the same code paths as when writing to\n> less.\n\nOr use script(1).\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"196970","messageId":"CAF7PVPqZRjWv4Z=a21yf=_NXrWd4b5M3ROx=dLQ6MbNRFKt1dg@mail.gmail.com","threadId":"31242","inReplyTo":"87393pztk4.fsf@thomas.inf.ethz.ch","subject":"Re: git diff vs git diff-files","fromName":"Bernd Jendrissek","fromEmail":"bernd.jendrissek@gmail.com","sentAt":"2012-08-14T13:11:46Z","receivedAt":"2012-08-14T13:11:46Z","isPatch":false,"sender":{"key":"bernd.jendrissek@gmail.com","avatar":null},"body":"On Tue, Aug 14, 2012 at 11:55 AM, Thomas Rast <trast@student.ethz.ch> wrote:\n> Fun stuff.  This will be very interesting to debug.  It's still\n> possible, though perhaps not extremely likely, that git is writing\n> garbage to the terminal and it just happens to work for xterm.\n>\n> Whatever you do next, *please* post the *exact* file contents *and*\n\nHmm, this really isn't about git anymore, as I've reduced this problem\nto the following file:\n\n0000000: 200a 200a 200a 200a 200a 200a 200a 200a   . . . . . . . .\n0000010: 200a 200a 200a 200a 200a 200a 200a 200a   . . . . . . . .\n0000020: 200a 200a 200a 200a 200a 200a 200a 200a   . . . . . . . .\n0000030: 200a 200a 200a 200a 200a 200a 200a 200a   . . . . . . . .\n0000040: 200a 200a 200a 200a 200a 200a 200a 200a   . . . . . . . .\n0000050: 200a 200a 200a 200a 200a 200a 200a 200a   . . . . . . . .\n0000060: 200a 200a 200a 200a 200a 200a 200a 200a   . . . . . . . .\n0000070: 200a 200a 200a 200a 200a 200a             . . . . . .\n\nberndj@capybara:~/gaf.git$ nl /tmp/g\n     2\t\n     3\t\n     4\t\n     5\t\n     6\t\n     7\n[snip]\n    60\t\n    61\t\n    62\t\nberndj@capybara:~/gaf.git$\n\nAnything shorter than that causes first the command line (what you\nmight call line 0, not the one you might call line 63, and yes, I do\nnot that this is close to a binary round number), and after that,\nnothing to disappear.\n\nstrace confirms that gnome-terminal reads, but neglects to write, line 1.\n\nI notice now that a second gnome-terminal tab in the same directory\nrunning the same commands reliably displays the correct output. I must\nhave dumped an ELF file to the terminal at some point, or something\nsimilarly daft like that, and forgotten to reset the terminal. Not\nthat I recall having done such a thing recently.\n\nIn any case, it's too late now; I ran reset(1) and the terminal is\nback behaving itself.\n\nThanks for helping me get (close to) the bottom of this. If git were\nto be guilty (which I very much doubt), the evidence is long gone now.\nI'll chalk this up to cosmic rays or something.\n"}]}