threads / rfc / 2859

[RFC] faking cvs annotate

Subject: [RFC] faking cvs annotate

## tl;dr

10 messages between Dec 16, 2005 and Dec 16, 2005.

replies: 9people: 7as markdown or json

Martin Langhoff· Dec 16, 2005, 01:13 UTC · lore
I know, I know, use pickaxe :-)

Any suggestions as to how to implement a not-too-slow annotate that looks reasonable enough so as to fool real cvs clients? Let's assume I'll have the "per-file-version numbers" to translate commit SHA1s into cvs-ish version numbers in a magic hat, right next to the fluffy bunnies.

Suggestions of GIT machinery that would shortcut the trip from
     git-rev-list HEAD $path

to a annotate-ish output. Did I dream it or is qgit showing something annotate-ish in its screenshots?

Note! Just told a white lie: I won't be actually using git-rev-list. Instead, I'll have a rev list that will gloss over branched development, picking only one "path" of the history every time (initially using the algorithm formerly known as rand() ) , and showing merge commits as one.

cheers,
martin
Johannes Schindelin· Dec 16, 2005, 01:31 UTC · re: Martin Langhoff · lore

Re: [RFC] faking cvs annotate

Hi,
On Fri, 16 Dec 2005, Martin Langhoff wrote:
> I know, I know, use pickaxe :-)
> 
> Any suggestions as to how to implement a not-too-slow annotate that
> looks reasonable enough so as to fool real cvs clients?
For starters, you could use my attempt:
http://www.gelato.unsw.edu.au/archives/git/0508/7171.html

It does not fool any CVS user, because it does not fake version numbers. Instead, you can display SHA1s, short SHA1s, author names, or mixtures thereof.

However, I learnt to use git-whatchanged in the meantime, and I'll probably never go back.

Hth, Dscho

Martin Langhoff· Dec 16, 2005, 04:37 UTC · re: Johannes Schindelin · lore

Re: [RFC] faking cvs annotate

On 12/16/05, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> For starters, you could use my attempt:
> http://www.gelato.unsw.edu.au/archives/git/0508/7171.html

Not bad at all! I might simplify it a bit (won't need the commitmsg parsing), and see if how fast I can make it.

> It does not fool any CVS user,
Ah, no. Those have been fooled already ;-)
> However, I learnt to use git-whatchanged in the meantime, and I'll
> probably never go back.

Of course. I'm trying to meet a few the expectations of popular cvs clients (cli, TortoiseCVS, Eclipse, etc). If a workable git-annotate falls out of this, oh well.

OTOH, yours didn't get included. Any particular reasons? (other than stressing the point that you should use whatchanged/pickaxe?)

Also! On 12/16/05, Junio C Hamano <junkio@cox.net> wrote:
Show 5 quoted lines
> I haven't actively done anything but one of the good things that
> could happen is to split out the access routines for annotate
> database qgit build when run the first time in the repository,
> and make them available to other Porcelains.  There is no need
> to reinvent the wheel.

That would be cool! Though my particular implementation may be a tad unconventional in its view of the "file history" as a strictly linear thing. Normal git porcelains don't even blink when dealing with branching/merging development histories.

cheers,
martin
Linus Torvalds· Dec 16, 2005, 17:07 UTC · re: Martin Langhoff · lore

Re: [RFC] faking cvs annotate

On Fri, 16 Dec 2005, Martin Langhoff wrote:
Show 7 quoted lines
>
> On 12/16/05, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > For starters, you could use my attempt:
> > http://www.gelato.unsw.edu.au/archives/git/0508/7171.html
> 
> Not bad at all! I might simplify it a bit (won't need the commitmsg
> parsing), and see if how fast I can make it.

NOTE! You can make it a whole lot faster these days by passing the path down to git-rev-list. But as usual, I'm perl-incompatible, so I can't help you with that modification.

> OTOH, yours didn't get included. Any particular reasons? (other than
> stressing the point that you should use whatchanged/pickaxe?)

I'd love to see a git-annotate, but I want it to be efficient enough that it's not embarrassing.

For example, "git-whatchanged" is certainly efficient enough, and you could just parse that output (possible without using -p and comparing file contents on your own against the current HEAD as you progress backwards)

Or, you could now use "git-rev-list <rev> <filename>", which is even more efficient, but the downside of that is that it computes the _full_ graph before it starts outputting the early ones.

But, for example:
	[torvalds@g5 linux-history]$ time git-whatchanged drivers/char/Makefile > /dev/null 
	real    0m37.993s
	user    0m41.912s
	sys     0m0.400s
	[torvalds@g5 linux-history]$ time git-rev-list HEAD drivers/char/Makefile > /dev/null 
	real    0m6.713s
	user    0m6.656s
	sys     0m0.056s

that's the old Linux history thing, with 60,000+ commit. The path-based git-rev-list thing cuts it down to just 103 commits you need to look at (so the annotate itself should be really fast, and the cost really is all in that "git-rev-list" thing).

Of course, with the current kernel tree, the git-rev-list takes a lot less time, and only results in six commits, but that's a tree that only has a few months of history, so when looking at performance, you really should look at some bigger tree.

NOTE! You can also decide to cut down on time by saying "annotate back only to a specific date", ie something like

	git-rev-list $(git-rev-parse --since=2.years.ago HEAD) drivers/char/Makefile

which limits the thing by filename _and_ date (a good way to handle really really big projects) where the rev-parse might otherwise be very expensive.

And, of course, a quality implementation will notice when a file disappears, and use "git-diff-tree -M" on that commit to see if it was renamed, and then continuing with the old name...

			Linus
Nicolas Pitre· Dec 16, 2005, 21:40 UTC · re: Linus Torvalds · lore

Re: [RFC] faking cvs annotate

On Fri, 16 Dec 2005, Linus Torvalds wrote:
Show 6 quoted lines
> But, for example:
> 
> 	[torvalds@g5 linux-history]$ time git-whatchanged drivers/char/Makefile > /dev/null 
> 	real    0m37.993s
> 	user    0m41.912s
> 	sys     0m0.400s

Hmmm... I'd like to have a system like that too, where reality is smaller than the sum of system and user time. ;-)

Nicolas
Linus Torvalds· Dec 16, 2005, 22:12 UTC · re: Nicolas Pitre · lore

Re: [RFC] faking cvs annotate

On Fri, 16 Dec 2005, Nicolas Pitre wrote:
Show 11 quoted lines
> On Fri, 16 Dec 2005, Linus Torvalds wrote:
> 
> > But, for example:
> > 
> > 	[torvalds@g5 linux-history]$ time git-whatchanged drivers/char/Makefile > /dev/null 
> > 	real    0m37.993s
> > 	user    0m41.912s
> > 	sys     0m0.400s
> 
> Hmmm... I'd like to have a system like that too, where reality is 
> smaller than the sum of system and user time.  ;-)
It's called "smp".
You can buy then anywhere.
		Linus
Nicolas Pitre· Dec 16, 2005, 22:36 UTC · re: Linus Torvalds · lore

Re: [RFC] faking cvs annotate

On Fri, 16 Dec 2005, Linus Torvalds wrote:
Show 17 quoted lines
> 
> 
> On Fri, 16 Dec 2005, Nicolas Pitre wrote:
> 
> > On Fri, 16 Dec 2005, Linus Torvalds wrote:
> > 
> > > But, for example:
> > > 
> > > 	[torvalds@g5 linux-history]$ time git-whatchanged drivers/char/Makefile > /dev/null 
> > > 	real    0m37.993s
> > > 	user    0m41.912s
> > > 	sys     0m0.400s
> > 
> > Hmmm... I'd like to have a system like that too, where reality is 
> > smaller than the sum of system and user time.  ;-)
> 
> It's called "smp".

Gah! Maybe I should have tried the same before posting, since I get this on my P4 with HT:

	real    0m36.930s
	user    0m41.971s
	sys     0m0.656s
/me hides shamefully
Nicolas
Junio C Hamano· Dec 16, 2005, 01:42 UTC · re: Martin Langhoff · lore

Re: [RFC] faking cvs annotate

Martin Langhoff <martin.langhoff@gmail.com> writes:
Show 6 quoted lines
> Suggestions of GIT machinery that would shortcut the trip from
>
>      git-rev-list HEAD $path
>
> to a annotate-ish output. Did I dream it or is qgit showing something
> annotate-ish in its screenshots?

I haven't actively done anything but one of the good things that could happen is to split out the access routines for annotate database qgit build when run the first time in the repository, and make them available to other Porcelains. There is no need to reinvent the wheel.

Eric Wong· Dec 16, 2005, 06:09 UTC · re: Junio C Hamano · lore

Re: [RFC] faking cvs annotate

Junio C Hamano <junkio@cox.net> wrote:
Show 14 quoted lines
> Martin Langhoff <martin.langhoff@gmail.com> writes:
> 
> > Suggestions of GIT machinery that would shortcut the trip from
> >
> >      git-rev-list HEAD $path
> >
> > to a annotate-ish output. Did I dream it or is qgit showing something
> > annotate-ish in its screenshots?
> 
> I haven't actively done anything but one of the good things that
> could happen is to split out the access routines for annotate
> database qgit build when run the first time in the repository,
> and make them available to other Porcelains.  There is no need
> to reinvent the wheel.

I started working on something using Algorithm::Annotate on CPAN during an ADD excursion a few weeks ago, but mostly forgot about it (thanks again to ADD :). A few versions of git later and it still seems to work as well as I remembered it.

It doesn't handle renames/copies yet, but it does let you customize the left-hand side output unlike most/(all?) annotate implementations.

I guess it's also quite useful when the output is piped to vim with a repository browser like the one I brought up in <20051124093322.GA3899@mail.yhbt.net>

---------- 8< ---------

#!/usr/bin/env perl # Copyright (c) 2005, Eric Wong <normalperson@yhbt.net> # # This file is licensed under the GPL v2, or a later version # at the discretion of Linus. # # Read git-whatchanged output and let all Algorithm::Annotate do all # the hard work for us. Thanks to Chia-liang Kao for the awesome # Algorithm::Annotate module. # # Formatting options of annotation output is available. # Dates can be formatted through strftime strings using the -D switch, # and annotation expansion can be done using the -F switch # on following variables:

=cut
	%commit% %tree% %author_name% %author_email% %author_date%
	%committer_name% %committer_email% %commit_date% %message%
=cut
use warnings;
use strict;
sub usage ($) {
	print '* git-annotate ',
		'[-w <width>] [-D <date-format>] [-F <annotate-format>] ',
		'[--localtime] [--line-prefix <str>] [--line-postfix <str>] ',
		'<file>', "\n";
	exit $_[0];
}

# everybody with Perl 5 should have these, right? use POSIX qw(strftime); use Getopt::Long qw(:config gnu_getopt no_ignore_case no_auto_abbrev);

my %_o; # options
usage(1) unless (GetOptions(\%_o, qw(help|H|h width|w=i
		line_prefix|line-prefix:s line_postfix|line-postfix:s
		format|F=s date_format|date-format|D=s localtime|l)));
usage(0) if $_o{help};
# ok, see if we can do more than show a help message:
require Algorithm::Annotate or die
		"Can't find Algorithm::Annotate, get it from CPAN ($!)\n";
my $file = shift or usage(1);
unless (-f $file) {
	print STDERR "File does not exist or is not a file: $file\n";
	exit 1;
}
my $date_format = $_o{date_format} || '%Y-%m-%dT%H-%M-%SZ'; # ISO 8601

my $re_sha1 = qr/[a-f0-9]{40}/; # match 40 char hex strings my @blobs; # we read from newest to oldest, but $ann needs it reversed

sub t ($) {
	return localtime($_[0]) if $_o{'localtime'};
	return gmtime($_[0]);
}
if (my $pid = open my $child, '-|') {
	my $meta;
	while (<$child>) {
		chomp;
		if (/^diff-tree ($re_sha1) /o) {
			$meta = { commit => $1, message => [] };
		} elsif (/^tree ($re_sha1)$/) {
			$meta->{tree} = $1;
		} elsif (/^author (.*) <(\S+)> (\d+) [\+\-]?\d+$/) {
			$meta->{author_name} = $1;
			$meta->{author_email} = $2;
			$meta->{author} = "$1 <$2>";
			$meta->{author_date} = strftime($date_format, t($3));
		} elsif (/^committer (.*) <(\S+)> (\d+) [\+\-]?\d+$/) {
			$meta->{committer_name} = $1;
			$meta->{committer_email} = $2;
			$meta->{committer} = "$1 <$2>";
			$meta->{commit_date} = strftime($date_format, t($3));
		} elsif (/^\s{4}(.*)$/) {
			push @{$meta->{message}}, $1;
		} elsif (/^:\d{6} \d{6} $re_sha1 ($re_sha1)/o) {
			my $blob = $1;
			if ($blob =~ /^0{40}$/) {
				# nonexistent (hopefully!) blob
				$meta = undef;
				next;
			}
			
			$meta->{message} = join("\n",@{$meta->{message}});
			push @blobs, { meta => $meta, blob => $blob };
			$meta = undef;
		}
	}
} else {
	exec('git-whatchanged','-m','--pretty=raw','--',$file) or die
			"Unable to execute git-whatchanged $file: $? $!";
}

my $msg_format = $_o{format} || '%commit% %commit_date%'; my $max_len = 0; my $ann = Algorithm::Annotate->new;

foreach my $x (reverse @blobs) {
	my @contents;
	
	# no need to do safe pipe opens since we know $x->{blob} is a hex sum
	@contents = `git-cat-file blob $x->{blob}`;
	# $msg is the left hand side of our final annotation output,
	# format it according to $msg_format
	my $msg = $msg_format;
	my $meta = $x->{meta};
	# our annotation message templating engine:
	$msg =~ s/\%$_\%/$meta->{$_}/g foreach keys %$meta;
	$msg =~ s/\\t/\t/g;
	$msg =~ s/\\n/\n/g;
	$ann->add($msg, \@contents);
	if ($msg !~ /\n/) {
		$max_len = length $msg if length $msg > $max_len;
	} else {
		foreach (split /\n/, $msg) {
			$max_len = length $_ if length $_ > $max_len;
		}
	}
}
# see if we can annotate local changes, too
my $local_change;
my @diff_opts = qw(-z -C --find-copies-harder --name-only);
{ # see if we've changed $file in the working tree
	if (my $pid = open my $child, '-|') {
		$local_change = (<$child>);
	} else {
		exec('git-diff-files',@diff_opts,'--',$file);
	}
}
unless ($local_change) {
	# see if we've changed $file in the working tree
	if (my $pid = open my $child, '-|') {
		$local_change = (<$child>);
	} else {
		exec('git-diff-index',@diff_opts,'--cached','HEAD','--',$file);
	}
}

open my $fd,'<',$file or die "Error opening file: $file: $!\n"; my @contents = <$fd>;

if ($local_change) {
	my $msg = '(uncommitted): ';
	$ann->add($msg, \@contents);
	$max_len = length $msg if length $msg > $max_len;
}
# $result here is an array ref, each element corresponding to a line of
# the annotated file contents
my $result = $ann->result or do {
	print STDERR "Unable to annotate $file.  Is it tracked by git?\n";
	exit 1;
};

my $pre = defined $_o{line_prefix} ? $_o{line_prefix} : ''; my $post = defined $_o{line_postfix} ? $_o{line_postfix} : ': '; my $width = $_o{width} || $max_len;

foreach (@contents) {
	my $msg = shift @$result;
	if ($msg !~ /\n/) {
		print ($pre, pack("A$width", $msg), $post, $_);
	} else {
		my @msgs = split /\n/, $msg;
		print ($pre, pack("A$width", shift @msgs), $post, $_);
		foreach my $m (@msgs) {
			print ($pre, pack("A$width", $m), $post,"\n");
		}
	}
}
close $fd;
exit 0;
-- 
Eric Wong
Ryan Anderson· Dec 16, 2005, 15:05 UTC · re: Eric Wong · lore

Re: [RFC] faking cvs annotate

Eric Wong wrote:
Show 5 quoted lines
> #!/usr/bin/env perl
> # Copyright (c) 2005, Eric Wong <normalperson@yhbt.net>
> #
> # This file is licensed under the GPL v2, or a later version
> # at the discretion of Linus.
Please use full names for this kind of authority delegation.
> # Read git-whatchanged output and let all Algorithm::Annotate do all
> # the hard work for us.  Thanks to Chia-liang Kao for the awesome 
> # Algorithm::Annotate module.
> #
You'll want to add libalgorithm-annotate-perl to the Debian/control
Depends: line when you submit this as a patch.  (I dont' have an
RPM-based machine handy, so I can't doublecheck that.)

I was going to comment on the rest, but I don't really have time to play with it at the moment.

-- 
Ryan Anderson
  sometimes Pug Majere

← back to recent threads