git diff vs git diff-files

I have a billion testsuite golden output files that have changed due to an explicit ordering of objects I've imposed on output. A helper script I wrote to help parse the diffs (to ignore order-only differences) noticed that one hunk had a different number of additions and deletions. I'm manually copying and pasting the hunks from git add -i into the script. However, when I used git diff to get at the changes, to discover which line was the offending one, all additions and deletions were exactly matched!

With strace I noticed that git add -i calls git diff-files, and that's about as far as I can conveniently trace where things are going weird. Weird:

$ /usr/local/git/bin/git diff-files -p --color -- TwoStageAmp-output.net

Bernd Jendrissek <bernd.jendrissek@gmail.com> writes:

Can you share this repository? Or at least the pre- and post-change files, transferred in such a way that there won't be any whitespace damage (your snippets above show obvious damage). You can use

  git cat-file blob :gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net

to get at the contents of the version in the index.

Do you have any diff config that could be of interest? A textconv filter would be an obvious example that could produce the above, but perhaps you could just look at

  git config --get-regexp '.*diff.*'
  grep diff .gitattributes .git/info/attributes

etc.

On Mon, Aug 13, 2012 at 5:02 PM, Thomas Rast <trast@student.ethz.ch> wrote:

This weird behaviour doesn't even survive making a copy (cp -a) of the whole repository, so I very much doubt making it available would be illuminative. My disk's SMART data seems okay. The weird-quotient just rose a bit.

Besides, .git/ is 60MB and my upload speed is 128kbps. A bit inconvenient.

http://www.bpj-code.co.za/downloads.php/bugs/TwoStageAmp-output.net?text contains the output from git show a5ee1e7. Leave off the ?text for an application/octet-stream download.

That file (with the Q1 line present) is consistent with the earlier commit that added the file. It's diff-files that's lying.

Nothing that fancy. I have just diff.color = auto and user.* = blah in global config, and similarly benign config in the repository.

Bernd Jendrissek <bernd.jendrissek@gmail.com> writes:

That sounds like an issue with stat() data, and then it doesn't. cp -a would change all the inode numbers, triggering a full refresh of the index in 'git diff'. But I'm not sure exactly how this can lead to disagreement, since both commands read *both* index and worktree version of the file. Very confusing.

Can you try the following:

  git ls-files --debug gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
  cp .git/index .git/index.orig
  touch gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
  # note, it is important that you run diff first
  git diff gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
  git diff-files -p gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net
  git ls-files --debug gnetlist/tests/common/outputs/osmond/TwoStageAmp-output.net

Umm, that's only one side of the diff, isn't it? The hunk header claims that the hunk goes up to line 47, so there must be more changes beyond what you showed.

On Tue, Aug 14, 2012 at 10:18 AM, Thomas Rast <trast@student.ethz.ch> wrote:

I'm can't tell what this was supposed to prove, but I think it's proven that I'm in the wrong mailing list, and that I should be taking this up with gnome-terminal. The Q1 part deletion line seems to be appearing (in the terminal) only after I've copy&pasted it into another terminal.

Running the same commands in xterm (correctly) shows the Q1 part deletion line without having to copy & paste.

At this point I doubt there's much value in publishing the output, but just in case (and it may be relevant when i take it to gnome-terminal), it's here:

http://www.bpj-code.co.za/downloads.php/bugs/git-output-87vcgl999v.fsf?text

Note that line 104 of that output appears in the same terminal from which I copy&paste it only after I switch back to it, after having pasted the output. Oi vey.

You're right - the hunk isn't even self-consistent! http://www.bpj-code.co.za/downloads.php/bugs/git-add--p-TwoStageAmp-output.net?text is the "bad" diff, and I count only 46 lines of (context + deletions), while the hunk header claims the (correct) 47 lines corresponding to the whole file covered in the hunk.

Thanks for your time; I'm fairly convinced now that git is just an innocent accessory to another program's bug.

Bernd Jendrissek <bernd.jendrissek@gmail.com> writes:

Fun stuff. This will be very interesting to debug. It's still possible, though perhaps not extremely likely, that git is writing garbage to the terminal and it just happens to work for xterm.

Whatever you do next, *please* post the *exact* file contents *and* output in a format that does not suffer any transport damage, neither to whitespace nor to binary data (such as terminal escapes). Piping through xxd comes to mind, perhaps by using

  GIT_PAGER=xxd git -p diff ....  >diff-hexdump

etc. to ensure that git takes the same code paths as when writing to less.

Cut&paste from a terminal will at best give you exactly what the terminal figures it is *currently displaying*, which is completely different from what programs write to it. At worst, it will also suffer whitespace damage. So pretty much everything you showed so far, except for the raw data of one (why only one?!) side of the diff, is useless.

Thomas Rast <trast@student.ethz.ch> writes:

Or use script(1).

Andreas.

On Tue, Aug 14, 2012 at 11:55 AM, Thomas Rast <trast@student.ethz.ch> wrote:

Hmm, this really isn't about git anymore, as I've reduced this problem to the following file:

0000000: 200a 200a 200a 200a 200a 200a 200a 200a . . . . . . . . 0000010: 200a 200a 200a 200a 200a 200a 200a 200a . . . . . . . . 0000020: 200a 200a 200a 200a 200a 200a 200a 200a . . . . . . . . 0000030: 200a 200a 200a 200a 200a 200a 200a 200a . . . . . . . . 0000040: 200a 200a 200a 200a 200a 200a 200a 200a . . . . . . . . 0000050: 200a 200a 200a 200a 200a 200a 200a 200a . . . . . . . . 0000060: 200a 200a 200a 200a 200a 200a 200a 200a . . . . . . . . 0000070: 200a 200a 200a 200a 200a 200a . . . . . .

berndj@capybara:~/gaf.git$ nl /tmp/g
     2	
     3	
     4	
     5	
     6	
     7
[snip]
    60	
    61	
    62	
berndj@capybara:~/gaf.git$

Anything shorter than that causes first the command line (what you might call line 0, not the one you might call line 63, and yes, I do not that this is close to a binary round number), and after that, nothing to disappear.

strace confirms that gnome-terminal reads, but neglects to write, line 1.

I notice now that a second gnome-terminal tab in the same directory running the same commands reliably displays the correct output. I must have dumped an ELF file to the terminal at some point, or something similarly daft like that, and forgotten to reset the terminal. Not that I recall having done such a thing recently.

In any case, it's too late now; I ran reset(1) and the terminal is back behaving itself.

Thanks for helping me get (close to) the bottom of this. If git were to be guilty (which I very much doubt), the evidence is long gone now. I'll chalk this up to cosmic rays or something.

Bernd Jendrissek <bernd.jendrissek@gmail.com> writes:

What's surprising is that both diff show the same "index" line, so both commands actually diff the same content, and then show a different output.

You can try something like

git cat-file blob a5ee1e7 > /tmp/staged.txt diff -u /tmp/staged.txt TwoStageAmp-output.netdiff

to recover the content from the index (Thomas' version should also work and give the same result), and use another diff tool. This "diff -u" should give you an output similar to one of "git diff" and "git diff-files" (my understanding is that it should match diff-files, and "git diff" is the one being wrong here).