threads / discuss / 16973

how to track the history of a line in a file

Subject: how to track the history of a line in a file

## tl;dr

17 messages between Jan 2, 2009 and Jan 6, 2009.

replies: 16people: 7as markdown or json

david@lang.hm· Jan 2, 2009, 22:13 UTC · lore

I have a need to setup a repository where I'm storing config files, and I need to be able to search the history of a particular line, not just when the last edit of the line was (which is what I see from git blame)

I'm not seeing a obvious way to do this, am I missing something or does it need a non-obvious approach?

for example, if I do
git blame -L /SUBLEVEL/,+1 -M Makefile
on the linux kernel it currently shows
57f8f7b6 (Linus Torvalds 2008-10-23 20:06:52 -0700 3) SUBLEVEL = 28

what I would want it to show would be a list of the commits that have changed this line.

It looks like I can write a script to do this

git blame -L /SUBLEVEL/,+1 -M Makefile 57f8f7b6^ 6e86841d (Linus Torvalds 2008-07-28 19:40:31 -0700 3) SUBLEVEL = 27 git blame -L /SUBLEVEL/,+1 -M Makefile 6e86841d^ 2ddcca36 (Linus Torvalds 2008-05-03 11:59:44 -0700 3) SUBLEVEL = 26

etc.
is there a better way to do this?
David Lang
Jeff King· Jan 2, 2009, 21:26 UTC · re: david@lang.hm · lore

Re: how to track the history of a line in a file

On Fri, Jan 02, 2009 at 02:13:32PM -0800, david@lang.hm wrote:
> I have a need to setup a repository where I'm storing config files, and I  
> need to be able to search the history of a particular line, not just when  
> the last edit of the line was (which is what I see from git blame)

As you figured out, the "manual" way is to just keep reblaming from the parent of each blame. Recent versions of "git gui blame" have a "reblame from parent" option in the context menu which makes this a lot less painful.

> 57f8f7b6 (Linus Torvalds 2008-10-23 20:06:52 -0700 3) SUBLEVEL = 28
>
> what I would want it to show would be a list of the commits that have  
> changed this line.

The tricky thing here is what is "this line"? Using the line number isn't right, since it will change based on other content coming in and out of the file. You can keep drilling down by reblaming parent commits, but remember that each time you do that you are manually looking at the content and saying "Oh, this is the line I am still interested in." So I a script would have to correlate the old version and new version of the line and realize how to follow the "interesting" thing.

In your case, I think you want to see any commit in Makefile which changed a line with SUBLEVEL in it. Which is maybe easiest done as:

  git log -z -p Makefile |
    perl -0ne 'print if /\n[+-]SUBLEVEL/' |
    tr '\0' '\n'

and is pretty fast. But obviously we're leveraging some content-specific knowledge about what's in the Makefile.

-Peff
david@lang.hm· Jan 2, 2009, 22:43 UTC · re: Jeff King · lore

Re: how to track the history of a line in a file

On Fri, 2 Jan 2009, Jeff King wrote:
Show 10 quoted lines
> On Fri, Jan 02, 2009 at 02:13:32PM -0800, david@lang.hm wrote:
>
>> I have a need to setup a repository where I'm storing config files, and I
>> need to be able to search the history of a particular line, not just when
>> the last edit of the line was (which is what I see from git blame)
>
> As you figured out, the "manual" way is to just keep reblaming from the
> parent of each blame. Recent versions of "git gui blame" have a "reblame
> from parent" option in the context menu which makes this a lot less
> painful.
unfortunantly I am needing to do this from the command line.
Show 22 quoted lines
>> 57f8f7b6 (Linus Torvalds 2008-10-23 20:06:52 -0700 3) SUBLEVEL = 28
>>
>> what I would want it to show would be a list of the commits that have
>> changed this line.
>
> The tricky thing here is what is "this line"? Using the line number
> isn't right, since it will change based on other content coming in and
> out of the file. You can keep drilling down by reblaming parent commits,
> but remember that each time you do that you are manually looking at the
> content and saying "Oh, this is the line I am still interested in." So I
> a script would have to correlate the old version and new version of the
> line and realize how to follow the "interesting" thing.
>
> In your case, I think you want to see any commit in Makefile which
> changed a line with SUBLEVEL in it. Which is maybe easiest done as:
>
>  git log -z -p Makefile |
>    perl -0ne 'print if /\n[+-]SUBLEVEL/' |
>    tr '\0' '\n'
>
> and is pretty fast. But obviously we're leveraging some content-specific
> knowledge about what's in the Makefile.

using the line number shouldn't be _that_ hard becouse git knows what lines came and went from the file, so it can calculate the new line number (and does with the -M option)

In my case I would consider 'the same line' to be any lines in the diff that were taken out when this line was put in

so in the usual case (for me) of

-oldline +newline

it's a 1-1 correspondence

if it's instead -oldline1 -oldline2 +newline1 +newline2

I can't know for sure which oldline corresponds to the newline, but the odds are very good that they are related, so if I widen the search to cover each of the lines I am probably good.

David Lang
Jeff Whiteside· Jan 2, 2009, 21:49 UTC · re: david@lang.hm · lore

Re: how to track the history of a line in a file

Show 10 quoted lines
> -oldline
> +newline
>
> it's a 1-1 correspondence
>
> if it's instead
> -oldline1
> -oldline2
> +newline1
> +newline2

what a neat idea. i'm going to start malloc'ing 10,000 lines x 120 chars for each file i add, and edit them so that no new lines replace removed lines unless it's intended that they were the same line.

david@lang.hm· Jan 2, 2009, 22:58 UTC · re: Jeff Whiteside · lore

Re: how to track the history of a line in a file

On Fri, 2 Jan 2009, Jeff Whiteside wrote:
Show 14 quoted lines
>> -oldline
>> +newline
>>
>> it's a 1-1 correspondence
>>
>> if it's instead
>> -oldline1
>> -oldline2
>> +newline1
>> +newline2
>
> what a neat idea.  i'm going to start malloc'ing 10,000 lines x 120
> chars for each file i add, and edit them so that no new lines replace
> removed lines unless it's intended that they were the same line.

I don't understand your comment. that isn't nessasary to do the tracking that I'm needing (you don't have to look at every line, only some of the lines that are in the same hunk of the patch as the line(s) you are interested in)

in my situation the use case is config files. in them each line is usually edited independantly of other lines (with stuff being added, usually, but not always on the end)

David Lang
david@lang.hm· Jan 2, 2009, 23:01 UTC · re: Jeff King · lore

Re: how to track the history of a line in a file

On Fri, 2 Jan 2009, Jeff King wrote:
Show 17 quoted lines
> The tricky thing here is what is "this line"? Using the line number
> isn't right, since it will change based on other content coming in and
> out of the file. You can keep drilling down by reblaming parent commits,
> but remember that each time you do that you are manually looking at the
> content and saying "Oh, this is the line I am still interested in." So I
> a script would have to correlate the old version and new version of the
> line and realize how to follow the "interesting" thing.
>
> In your case, I think you want to see any commit in Makefile which
> changed a line with SUBLEVEL in it. Which is maybe easiest done as:
>
>  git log -z -p Makefile |
>    perl -0ne 'print if /\n[+-]SUBLEVEL/' |
>    tr '\0' '\n'
>
> and is pretty fast. But obviously we're leveraging some content-specific
> knowledge about what's in the Makefile.
Ok, I hacked togeather a quick bash script to try this

#!/bin/bash line=`git blame -L /$1/,+1 -M $2` COMMIT=`echo $line |cut -f 1 -d " "` foundline=`echo $line |cut -f 6 -d " "|sed s/")"//` echo "$foundline $COMMIT" echo "$line" while [ "$COMMIT" != "" ] ;do echo "git blame -L $foundline,+1 -M $2 $COMMIT^" line=`git blame -L $foundline,+1 -M $2 $COMMIT^` COMMIT=`echo $line |cut -f 1 -d " "` foundline=`echo $line |cut -f 6 -d " "|sed s/")"//` echo "$line" done

the problem that this has is that line 3 of $COMMIT may not be line 3 of $COMMIT^, and if they aren't it ends up hunting down the wrong data

either that or I am not understanding the output of git blame properly (also very possible)

David Lang
david@lang.hm· Jan 2, 2009, 23:48 UTC · re: david@lang.hm · lore

Re: how to track the history of a line in a file

On Fri, 2 Jan 2009, david@lang.hm wrote:
Show 22 quoted lines
> On Fri, 2 Jan 2009, Jeff King wrote:
>
>> The tricky thing here is what is "this line"? Using the line number
>> isn't right, since it will change based on other content coming in and
>> out of the file. You can keep drilling down by reblaming parent commits,
>> but remember that each time you do that you are manually looking at the
>> content and saying "Oh, this is the line I am still interested in." So I
>> a script would have to correlate the old version and new version of the
>> line and realize how to follow the "interesting" thing.
>> 
>> In your case, I think you want to see any commit in Makefile which
>> changed a line with SUBLEVEL in it. Which is maybe easiest done as:
>>
>>  git log -z -p Makefile |
>>    perl -0ne 'print if /\n[+-]SUBLEVEL/' |
>>    tr '\0' '\n'
>> 
>> and is pretty fast. But obviously we're leveraging some content-specific
>> knowledge about what's in the Makefile.
>
> Ok, I hacked togeather a quick bash script to try this
>
<SNIP>
Show 5 quoted lines
> the problem that this has is that line 3 of $COMMIT may not be line 3 of 
> $COMMIT^, and if they aren't it ends up hunting down the wrong data
>
> either that or I am not understanding the output of git blame properly (also 
> very possible)
I was misunderstanding git blame
new script is

#!/bin/bash line=`git blame -n -b -l -L /$1/,+1 -M $2` echo "-$line" foundCOMMIT=`echo "$line" |cut -c -40` foundline=`echo "$line" |cut -c 42- |cut -f 1 -d " "` while [ "$foundCOMMIT" != " " ] ;do #git diff -U0 $foundCOMMIT..$foundCOMMIT^ $2 line=`git blame -n -b -l -L $foundline,+1 -M $2 $foundCOMMIT^` echo "-$line" foundCOMMIT=`echo "$line" |cut -c -40` foundline=`echo "$line" |cut -c 42- |cut -f 1 -d " "` done

this seems to be working for me now.
David Lang
Junio C Hamano· Jan 2, 2009, 22:54 UTC · re: david@lang.hm · lore

Re: how to track the history of a line in a file

david@lang.hm writes:
> I was misunderstanding git blame

You may find the --porcelain output format of git-blame useful to make your script safer and more robust.

david@lang.hm· Jan 3, 2009, 00:59 UTC · re: Junio C Hamano · lore

Re: how to track the history of a line in a file

On Fri, 2 Jan 2009, Junio C Hamano wrote:
Show 6 quoted lines
> david@lang.hm writes:
>
>> I was misunderstanding git blame
>
> You may find the --porcelain output format of git-blame useful to make
> your script safer and more robust.

where is this output defined? I can look at it and understand it, but I don't know what is guarenteeded to exist, what will be on a specific line, and what may vanish on me.

David Lang
Thomas Rast· Jan 3, 2009, 00:07 UTC · re: david@lang.hm · lore

Re: how to track the history of a line in a file

david@lang.hm wrote:
Show 7 quoted lines
> On Fri, 2 Jan 2009, Junio C Hamano wrote:
> > You may find the --porcelain output format of git-blame useful to make
> > your script safer and more robust.
> 
> where is this output defined? I can look at it and understand it, but I 
> don't know what is guarenteeded to exist, what will be on a specific line, 
> and what may vanish on me.
man git-blame | less -p PORCELAIN\ FORMAT
-- 
Thomas Rast
trast@{inf,student}.ethz.ch
david@lang.hm· Jan 3, 2009, 02:05 UTC · re: Thomas Rast · lore

Re: how to track the history of a line in a file

On Sat, 3 Jan 2009, Thomas Rast wrote:
Show 10 quoted lines
> david@lang.hm wrote:
>> On Fri, 2 Jan 2009, Junio C Hamano wrote:
>>> You may find the --porcelain output format of git-blame useful to make
>>> your script safer and more robust.
>>
>> where is this output defined? I can look at it and understand it, but I
>> don't know what is guarenteeded to exist, what will be on a specific line,
>> and what may vanish on me.
>
> man git-blame | less -p PORCELAIN\ FORMAT
thanks,

am I missing something else or will git blame with a regex only return info on the first line it matches?

David Lang
david@lang.hm· Jan 2, 2009, 23:56 UTC · re: david@lang.hm · lore

Re: how to track the history of a line in a file

On Fri, 2 Jan 2009, david@lang.hm wrote:
Show 49 quoted lines
> On Fri, 2 Jan 2009, david@lang.hm wrote:
>
>> On Fri, 2 Jan 2009, Jeff King wrote:
>> 
>>> The tricky thing here is what is "this line"? Using the line number
>>> isn't right, since it will change based on other content coming in and
>>> out of the file. You can keep drilling down by reblaming parent commits,
>>> but remember that each time you do that you are manually looking at the
>>> content and saying "Oh, this is the line I am still interested in." So I
>>> a script would have to correlate the old version and new version of the
>>> line and realize how to follow the "interesting" thing.
>>> 
>>> In your case, I think you want to see any commit in Makefile which
>>> changed a line with SUBLEVEL in it. Which is maybe easiest done as:
>>>
>>>  git log -z -p Makefile |
>>>    perl -0ne 'print if /\n[+-]SUBLEVEL/' |
>>>    tr '\0' '\n'
>>> 
>>> and is pretty fast. But obviously we're leveraging some content-specific
>>> knowledge about what's in the Makefile.
>> 
>> Ok, I hacked togeather a quick bash script to try this
>> 
> <SNIP>
>> the problem that this has is that line 3 of $COMMIT may not be line 3 of 
>> $COMMIT^, and if they aren't it ends up hunting down the wrong data
>> 
>> either that or I am not understanding the output of git blame properly 
>> (also very possible)
>
> I was misunderstanding git blame
>
> new script is
>
> #!/bin/bash
> line=`git blame -n -b -l -L /$1/,+1 -M $2`
> echo "-$line"
> foundCOMMIT=`echo "$line" |cut -c -40`
> foundline=`echo "$line" |cut -c 42- |cut -f 1 -d " "`
> while [ "$foundCOMMIT" != "                                        " ] ;do
> #git diff -U0 $foundCOMMIT..$foundCOMMIT^ $2
> line=`git blame -n -b -l -L $foundline,+1 -M $2 $foundCOMMIT^`
> echo "-$line"
> foundCOMMIT=`echo "$line" |cut -c -40`
> foundline=`echo "$line" |cut -c 42- |cut -f 1 -d " "`
> done
>
> this seems to be working for me now.

not quite, it works as long as the line doesn't move in the commit where it changes.

David Lang
david@lang.hm· Jan 3, 2009, 04:25 UTC · re: david@lang.hm · lore

Re: how to track the history of a line in a file

On Fri, 2 Jan 2009, david@lang.hm wrote:
Show 29 quoted lines
> On Fri, 2 Jan 2009, david@lang.hm wrote:
>
>> On Fri, 2 Jan 2009, david@lang.hm wrote:
>> 
>>> On Fri, 2 Jan 2009, Jeff King wrote:
>>> 
>>>> The tricky thing here is what is "this line"? Using the line number
>>>> isn't right, since it will change based on other content coming in and
>>>> out of the file. You can keep drilling down by reblaming parent commits,
>>>> but remember that each time you do that you are manually looking at the
>>>> content and saying "Oh, this is the line I am still interested in." So I
>>>> a script would have to correlate the old version and new version of the
>>>> line and realize how to follow the "interesting" thing.
>>>> 
>>>> In your case, I think you want to see any commit in Makefile which
>>>> changed a line with SUBLEVEL in it. Which is maybe easiest done as:
>>>>
>>>>  git log -z -p Makefile |
>>>>    perl -0ne 'print if /\n[+-]SUBLEVEL/' |
>>>>    tr '\0' '\n'
>>>> 
>>>> and is pretty fast. But obviously we're leveraging some content-specific
>>>> knowledge about what's in the Makefile.
>>> 
>>> Ok, I hacked togeather a quick bash script to try this
>>> 
>> <SNIP>
>>> the problem that this has is that line 3 of $COMMIT may not be line 3 of 
>>> $COMMIT^, and if they aren't it ends up hunting down the wrong data

solving this problem was a bit more work than expected. I ended up invoking git diff to do the step back and find what lines were replaced by the one I care about.

this still can run into problems if the lines that are examined are not unique (git blame only finds the first match)

#!/usr/bin/perl
# program to trace the history of a line through a git repository
#
# useage findlines pattern filespec
#
# finds all occurances of the pattern in the file and then uses git blame
#  to find where those lines were introduced. It then takes a step back 
and
#  looks at what the line replaced (if anything) and does a git blame on 
those
#  lines as well
#
# written by David Lang and released under GPLv2
#
$target = shift @ARGV;
$file = shift @ARGV;
open($infile,"<$file");
$linecount=0;
# find all lines in the file that match the target pattern
# we have been provided. add them to a stack of items to research
while(<$infile>){
   $linecount++;
   if ($_ =~ /$target/) {
     $items[++$#items]{line} = $linecount;
     $items[$#items]{tree} = "HEAD";
     chomp;
     $items[$#items]{target} = $_;
   }
}
# go through all the items and research them in turn
#  (note that more items may be added to the list while inside this loop)
for ($i=0;$i<=$#items;$i++) {
# do a git blame to find where the lines came from
   $result = `git blame -b -l -L "$items[$i]{line}",+1 -M  $file 
$items[$i]{tree}` ;
   print "in ".$items[$i]{tree}." found:\n$result";
   $commit=substr($result,0,40);
   # if this is not the root commit,
   #  do a diff to find what lines may have been replaced by the line we 
are looking for
   if($commit ne "                                        "){
     $diff=`git diff -U0 $commit^..$commit $file`;
     @lines=split("\n",$diff);
     #strip the first four lines from the diff (as they just talk about the 
file)
     shift(@lines);
     shift(@lines);
     shift(@lines);
     shift(@lines);
     $match=0;
     $hunk='';
     # find what hunk of the diff introduced the lines we are looking for,
     #  add the lines that they replaced to the list of items to examine
     foreach(@lines){
       if ($_ =~ /^\@\@/ ) {
         if ($match) {
           process_hunk();
         } else {
           $hunk = '';
         };
       } else {
         $hunk .= $_."\n";
         if ($_ =~ /$items[$i]{target}/) {
           $match = 1;
         }
       }
     }
     if ($match) {
       process_hunk();
     }
   }
}
sub process_hunk(){
   # find lines in the hunk of diff that are being replaced
   foreach (split("\n",$hunk)){
     if ($_ =~ /^-/ ){
       # for now queue the line contents for further processing.
       #   this really should be the line number becouse otherwise
       #   git blame will use the first match it finds. since we are
       # matching the entire line this is less of a problem than it could 
be.
       $items[++$#items]{line} = "/".substr($_,1)."/";
       $items[$#items]{target} = substr($_,1);
       $items[$#items]{tree} = "$commit^";
     }
   }
}
Bernt Hansen· Jan 6, 2009, 15:48 UTC · re: david@lang.hm · lore

Re: how to track the history of a line in a file

david@lang.hm writes:
Show 31 quoted lines
> I have a need to setup a repository where I'm storing config files,
> and I need to be able to search the history of a particular line, not
> just when the last edit of the line was (which is what I see from git
> blame)
>
> I'm not seeing a obvious way to do this, am I missing something or
> does it need a non-obvious approach?
>
> for example, if I do
>
> git blame -L /SUBLEVEL/,+1 -M Makefile
>
> on the linux kernel it currently shows
>
> 57f8f7b6 (Linus Torvalds 2008-10-23 20:06:52 -0700 3) SUBLEVEL = 28
>
> what I would want it to show would be a list of the commits that have
> changed this line.
>
> It looks like I can write a script to do this
>
> git blame -L /SUBLEVEL/,+1 -M Makefile 57f8f7b6^
> 6e86841d (Linus Torvalds 2008-07-28 19:40:31 -0700 3) SUBLEVEL = 27
> git blame -L /SUBLEVEL/,+1 -M Makefile 6e86841d^
> 2ddcca36 (Linus Torvalds 2008-05-03 11:59:44 -0700 3) SUBLEVEL = 26
>
> etc.
>
> is there a better way to do this?
>
> David Lang
I think you need a script to do what you want.  I think this works...

Save the following script in ~/bin/git-rblame.sh, make it executable, and then create a global git alias for it as follows:

$ git config --global alias.rblame '!~/bin/git-rblame.sh $*'
Then you can just use

$ git rblame -L /SUBLEVEL/,+1 -M Makefile 6e86841d (Linus Torvalds 2008-07-28 19:40:31 -0700 3) SUBLEVEL = 27 2ddcca36 (Linus Torvalds 2008-05-03 11:59:44 -0700 3) SUBLEVEL = 26 ... 4c91aedb (Linus Torvalds 2005-06-28 22:57:29 -0700 3) SUBLEVEL = 13 ^1da177e (Linus Torvalds 2005-04-16 15:20:36 -0700 3) SUBLEVEL = 12

------cut here ------ !/bin/git-rblame.sh ---
#!/bin/sh
PARAMS="$*"
LINE=$(git blame $PARAMS)
while test $? == 0
do
    echo $LINE
    COMMIT="${LINE:0:8}^"
    LINE=$(git blame $PARAMS $COMMIT 2>/dev/null)
done
------cut here ------------------------------

Cheers, Bernt

Miklos Vajna· Jan 6, 2009, 16:08 UTC · re: Bernt Hansen · lore

Re: how to track the history of a line in a file

On Tue, Jan 06, 2009 at 10:48:10AM -0500, Bernt Hansen <bernt@norang.ca> wrote:
> Save the following script in ~/bin/git-rblame.sh, make it executable,
> and then create a global git alias for it as follows:
> 
> $ git config --global alias.rblame '!~/bin/git-rblame.sh $*'

Given that you have ~/bin in PATH, just name the script ~/bin/git-rblame and you won't even have to define an alias on each machine. ;-)

Bernt Hansen· Jan 6, 2009, 16:21 UTC · re: Miklos Vajna · lore

Re: how to track the history of a line in a file

Miklos Vajna <vmiklos@frugalware.org> writes:
Show 8 quoted lines
> On Tue, Jan 06, 2009 at 10:48:10AM -0500, Bernt Hansen <bernt@norang.ca> wrote:
>> Save the following script in ~/bin/git-rblame.sh, make it executable,
>> and then create a global git alias for it as follows:
>> 
>> $ git config --global alias.rblame '!~/bin/git-rblame.sh $*'
>
> Given that you have ~/bin in PATH, just name the script ~/bin/git-rblame
> and you won't even have to define an alias on each machine. ;-)

Yes but I don't want to use 'git-rblame' as the command since I've broken my habit of using the dashed versions of commands. 'git rblame' just feels better to me.

Now I probably should have named the script something that won't ever clash with possible future git commands (like my-git-rblame.sh or something) but since it's in my ~/bin I'll just deal with that if it ever happens in the future :)

-Bernt
Bernt Hansen· Jan 6, 2009, 16:28 UTC · re: Bernt Hansen · lore

Re: how to track the history of a line in a file

Bernt Hansen <bernt@norang.ca> writes:
Show 19 quoted lines
> Miklos Vajna <vmiklos@frugalware.org> writes:
>
>> On Tue, Jan 06, 2009 at 10:48:10AM -0500, Bernt Hansen <bernt@norang.ca> wrote:
>>> Save the following script in ~/bin/git-rblame.sh, make it executable,
>>> and then create a global git alias for it as follows:
>>> 
>>> $ git config --global alias.rblame '!~/bin/git-rblame.sh $*'
>>
>> Given that you have ~/bin in PATH, just name the script ~/bin/git-rblame
>> and you won't even have to define an alias on each machine. ;-)
>
> Yes but I don't want to use 'git-rblame' as the command since I've
> broken my habit of using the dashed versions of commands. 'git rblame'
> just feels better to me.
>
> Now I probably should have named the script something that won't ever
> clash with possible future git commands (like my-git-rblame.sh or
> something) but since it's in my ~/bin I'll just deal with that if it
> ever happens in the future :)

Oops. That actually does do what I want. Thanks for pointing that out Miklos!

(sorry for replying to my own post - next time I'll try it first before
 I post :)
-Bernt

← back to recent threads