Inconsistent results of git blame --porcelain when detecting copies from other files

Hi Folks!

The issue is best explained on an example. You can reproduce it using the Lucene repo https://github.com/apache/lucene-solr.git. Tested with the following versions:  1.8.1.6 (Ubuntu), 2.11.0.windows.1, 2.11.1.windows.1.

First, let's produce the correct results without using --procelain:

The following excerpt shows lines 501-505 from the output. In particular we can see that lines 502-503 originate from IndexReader.java.

10ba9abeb208d37df985e95a742f756de067353f lucene/src/java/org/apache/lucene/index/DirectoryReader.java 501 501) * <p>This method ^d1165b19726fa0cd13a539827a7cd43237a4fee lucene/src/java/org/apache/lucene/index/IndexReader.java 496 502) * returns the version recorded in the commit that the ^d1165b19726fa0cd13a539827a7cd43237a4fee lucene/src/java/org/apache/lucene/index/IndexReader.java 497 503) * reader opened. This version is advanced every time ^d1165b19726fa0cd13a539827a7cd43237a4fee lucene/src/java/org/apache/lucene/index/IndexReader.java 498 504) * a change is made with {@link IndexWriter}.</p> 10ba9abeb208d37df985e95a742f756de067353f lucene/src/java/org/apache/lucene/index/DirectoryReader.java 505 505) */

The same information can be obtained as well by using --line-porcelain:

Here is the output for line 502:

d1165b19726fa0cd13a539827a7cd43237a4feef 496 502 3
author Michael McCandless
author-mail <mikemccand@apache.org>
author-time 1327877325
author-tz +0000
committer Michael McCandless
committer-mail <mikemccand@apache.org>
committer-time 1327877325
committer-tz +0000
summary LUCENE-3725: add optional packing to FSTs
boundary
filename lucene/src/java/org/apache/lucene/index/IndexReader.java
        * returns the version recorded in the commit that the

However, when using --porcelain DirectoryReader.java is reported as the origin of lines 502-504:

10ba9abeb208d37df985e95a742f756de067353f 501 501 1
author Uwe Schindler
author-mail <uschindler@apache.org>
author-time 1327879145
author-tz +0000
committer Uwe Schindler
committer-mail <uschindler@apache.org>
committer-time 1327879145
committer-tz +0000
summary Reverse merged revision(s) from lucene/dev/trunk up to 1237502
previous f5dba8b76709ff0ef8715b8b288a4b64d4993fa3 lucene/src/java/org/apache/lucene/index/DirectoryReader.java
filename lucene/src/java/org/apache/lucene/index/DirectoryReader.java
        * <p>This method
d1165b19726fa0cd13a539827a7cd43237a4feef 496 502 3
        * returns the version recorded in the commit that the
d1165b19726fa0cd13a539827a7cd43237a4feef 497 503
        * reader opened.  This version is advanced every time
d1165b19726fa0cd13a539827a7cd43237a4feef 498 504

This is not only inconsistent with the other outputs but the output is also inconsistent in itself because lines 496 -498 do not even exist in a previous version of DirectoryReader.java.

Thanks for any feedback.

Kind Regards Konstantin Sokolov

"Sokolov, Konstantin" <konstantin.sokolov.ext@siemens.com> writes:

Hmph, this sounds vaguely familiar with

    http://public-inbox.org/git/20170106042051.nwjiuyyp7ljhs4sr@sigill.intra.peff.net

which is part of Git 2.12-rc0

On Mon, Feb 20, 2017 at 01:30:29PM -0800, Junio C Hamano wrote:

Yeah, I had the same thought while reading Konstantin's report.

I'm not sure Git is wrong, though. I think it's just the way the porcelain output works.

Here's a minimal reproduction; the interesting thing is when a commit is mentioned twice (as happens on lines 1 and 5 here):

  git init repo
  cd repo
  
  # use long lines to make sure we trigger content-movement detection
  for i in $(seq 1 5); do
  	echo this is really long line number $i
  done >file
  git add file
  git commit -m initial
  
  sed 's/1/one/; s/5/five/' <file >renamed
  git rm file
  git add renamed
  git commit -m 'rename and use english'
  
  git blame renamed
  git blame --line-porcelain renamed
  git blame --porcelain renamed

The first blame output looks something like this:

  bab03701 renamed ... line number 1
  ^dda1349 file    ... line number 2
  ^dda1349 file    ... line number 3
  ^dda1349 file    ... line number 4
  bab03701 renamed ... line number 5

so we can see it's the same case. The --line-porcelain similarly matches the commits and filenames.

But the --porcelain output is:

  bab037010dcabaf0509db27bf232d25659b180fa 1 1 1
  ...
  filename renamed
          this is really long line number one
  dda1349d41da859f4c37e018dbed714ba6c1aa18 2 2 3
  ...
  filename file
          this is really long line number 2
  dda1349d41da859f4c37e018dbed714ba6c1aa18 3 3
          this is really long line number 3
  dda1349d41da859f4c37e018dbed714ba6c1aa18 4 4
          this is really long line number 4
  bab037010dcabaf0509db27bf232d25659b180fa 5 5 1
          this is really long line number five

You might be tempted to say that the fifth line comes from "filename file", because that was the last "filename" entry we saw. But that's _not_ how the porcelain output works. That "filename" entry was associated with dda1349, but the line comes from bab0370 here.

The simplest way (IMHO) to parse --porcelain output is:

  - maintain a mapping of commit sha1s to the commit's details

  - whenever you see a "<sha1> <line_nr> <orig_nr> [<size-of-hunk>]"
    line, any key-value fields which follow impact _only_ that sha1, and
    you should update the details for that map entry

  - when you see the actual tab-indented line content, you have gotten
    all of the key-value updates for that sha1. You can now safely do
    what you like with the line entry.

Another way, if you don't want to update your mapping, is to actually pay attention to the size-of-hunk headers. In this case the middle three lines come in their own hunk (which you can see from the "2 2 3" header on the second line). The "filename" field we get applies to that hunk, but once we switch to a different one, the filename field needs to be looked up in the commit mapping.

But it's definitely not correct to blindly apply one "filename" field to subsequent lines in other hunks.

And yes, I do think this is probably more complex than it needs to be. I didn't write it. And I don't think it is worth the backwards compatibility headache of trying to change it now. It's possible this could be better documented (I didn't look at the documentation to write that explanation; I happened to puzzle it out for somebody else recently who had a similar case. That's what led to the bug-fix in the message you linked).

-Peff

Jeff King <peff@peff.net> writes:

Yup, that was how the output was meant to be read. At least in the mind of the person who designed the output format ;-)

Thanks for going into the issue. As far as I understand 2.12 won't change the discussed behavior of --procelain. We will switch to --line-procelain. After the current discussion it seems to be less error prone, more future-proof and our current parser can handle it without any changes.

Regards Konstantin

-----Ursprüngliche Nachricht-----
Von: Jeff King [mailto:peff@peff.net]
Gesendet: Montag, 20. Februar 2017 23:16
An: Junio C Hamano
Cc: Sokolov, Konstantin (ext) (CT RDA SSI ADM-DE); git@vger.kernel.org
Betreff: Re: Inconsistent results of git blame --porcelain when detecting copies from other files

On Mon, Feb 20, 2017 at 01:30:29PM -0800, Junio C Hamano wrote:

Yeah, I had the same thought while reading Konstantin's report.

I'm not sure Git is wrong, though. I think it's just the way the porcelain output works.

Here's a minimal reproduction; the interesting thing is when a commit is mentioned twice (as happens on lines 1 and 5 here):

  git init repo
  cd repo

  # use long lines to make sure we trigger content-movement detection
  for i in $(seq 1 5); do
        echo this is really long line number $i
  done >file
  git add file
  git commit -m initial

  sed 's/1/one/; s/5/five/' <file >renamed
  git rm file
  git add renamed
  git commit -m 'rename and use english'

  git blame renamed
  git blame --line-porcelain renamed
  git blame --porcelain renamed

The first blame output looks something like this:

  bab03701 renamed ... line number 1
  ^dda1349 file    ... line number 2
  ^dda1349 file    ... line number 3
  ^dda1349 file    ... line number 4
  bab03701 renamed ... line number 5

so we can see it's the same case. The --line-porcelain similarly matches the commits and filenames.

But the --porcelain output is:

  bab037010dcabaf0509db27bf232d25659b180fa 1 1 1
  ...
  filename renamed
          this is really long line number one
  dda1349d41da859f4c37e018dbed714ba6c1aa18 2 2 3
  ...
  filename file
          this is really long line number 2
  dda1349d41da859f4c37e018dbed714ba6c1aa18 3 3
          this is really long line number 3
  dda1349d41da859f4c37e018dbed714ba6c1aa18 4 4
          this is really long line number 4
  bab037010dcabaf0509db27bf232d25659b180fa 5 5 1
          this is really long line number five

You might be tempted to say that the fifth line comes from "filename file", because that was the last "filename" entry we saw. But that's _not_ how the porcelain output works. That "filename" entry was associated with dda1349, but the line comes from bab0370 here.

The simplest way (IMHO) to parse --porcelain output is:

  - maintain a mapping of commit sha1s to the commit's details

  - whenever you see a "<sha1> <line_nr> <orig_nr> [<size-of-hunk>]"
    line, any key-value fields which follow impact _only_ that sha1, and
    you should update the details for that map entry

  - when you see the actual tab-indented line content, you have gotten
    all of the key-value updates for that sha1. You can now safely do
    what you like with the line entry.

Another way, if you don't want to update your mapping, is to actually pay attention to the size-of-hunk headers. In this case the middle three lines come in their own hunk (which you can see from the "2 2 3" header on the second line). The "filename" field we get applies to that hunk, but once we switch to a different one, the filename field needs to be looked up in the commit mapping.

But it's definitely not correct to blindly apply one "filename" field to subsequent lines in other hunks.

And yes, I do think this is probably more complex than it needs to be. I didn't write it. And I don't think it is worth the backwards compatibility headache of trying to change it now. It's possible this could be better documented (I didn't look at the documentation to write that explanation; I happened to puzzle it out for somebody else recently who had a similar case. That's what led to the bug-fix in the message you linked).

-Peff

On Tue, Feb 21, 2017 at 12:25:37PM +0000, Sokolov, Konstantin wrote:

Right, the 2.12 change is only fixing a case where the "previous" key/value line was not repeated at the correct spots. The same parsing rules still hold.

-Peff