patchcvsimport move over to using git for each ref to read refs

cvsimport: move over to using git-for-each-ref to read refs

cvsimport opens all of the files in $GIT_DIR/refs/heads and reads out the sha1's in order to work out what time the last commit on that branch was made (in CVS) thus allowing incremental updates. However, this takes no account of hierachical refs naming producing the following error for each directory in $GIT_DIR/refs:

  Use of uninitialized value in chomp at /usr/bin/git-cvsimport line 503.
  Use of uninitialized value in concatenation (.) or string at
					/usr/bin/git-cvsimport line 505.
  usage: git-cat-file [-t|-s|-e|-p|<type>] <sha1>

Take advantage of the new packed refs work to use the new for-each-ref iterator to get this information. Use the format specifier to ensure we are neutral to changes in default.

[Junio: although although for-each-ref offers a --perl quoting mode this patch does not use it as it seems only to make parsing the output harder in perl. If there is a neat trick for handling this 'perl' form please educate me. Here, rely on sha1's to contain no spaces.]

Signed-off-by: Andy Whitcroft <apw@shadowen.org>
---

Andy Whitcroft wrote:

By the way, this is equivalent to using "git-show-ref" introduced by Linus. But if you want commit timestamp

you can use it in --format as well.

Jakub Narebski wrote:

Unfortuantly, for-each-ref only offers us the textual version of this information not the numeric offset from the epoch which is what cvsimport is after.

I guess we could teach for-each-ref to output this as well? Perhaps something like authorstamp?

-apw

for-each-ref: add a raw timestamp field type

cvsimport is interested in the raw time stamps (in seconds since the epoch) to do its time comparisons. Export the raw timestamp under {author,committer,tagger}stamp.

Signed-off-by: Andy Whitcroft <apw@shadowen.org>
---

cvsimport: move over to using git-for-each-ref to read refs V2

cvsimport opens all of the files in $GIT_DIR/refs/heads and reads out the sha1's in order to work out what time the last commit on that branch was made (in CVS) thus allowing incremental updates. However, this takes no account of hierachical refs naming producing the following error for each directory in $GIT_DIR/refs:

  Use of uninitialized value in chomp at /usr/bin/git-cvsimport line 503.
  Use of uninitialized value in concatenation (.) or string at
					/usr/bin/git-cvsimport line 505.
  usage: git-cat-file [-t|-s|-e|-p|<type>] <sha1>

Take advantage of the new packed refs work to use the new for-each-ref iterator to get this information. Use the format specifier to ensure we are neutral to changes in default.

[Junio: although although for-each-ref offers a --perl quoting mode this patch does not use it as it seems only to make parsing the output harder in perl. If there is a neat trick for handling this 'perl' form please educate me. Here, rely on sha1's and refnames to contain no spaces.]

Signed-off-by: Andy Whitcroft <apw@shadowen.org>
---

Andy Whitcroft wrote:

Or authorepoch. 

I thought about following rpm idea of format modifiers, like authorepoch:date for strftime(3) "%c" format or commiterepoch:day for strftime(3) "%a %b %d %Y" format.

Feel free to provide patches ;-)

Andy Whitcroft <apw@shadowen.org> writes:

I think you can work with "author" or "committer" to grab the whole raw line.

About the quoting and parsing, language specific quoting mode is meant for git-for-each-ref to produce a string that can be eval'ed in the host language. Think of the command as a tool to write a short program for you.

The original is like this:

	# Get the last import timestamps
	opendir(D,"$git_dir/refs/heads");
	while(defined(my $head = readdir(D))) {
		next if $head =~ /^\./;
		open(F,"$git_dir/refs/heads/$head")
			or die "Bad head branch: $head: $!\n";
		chomp(my $ftag = <F>);
		close(F);
		open(F,"git-cat-file commit $ftag |");
		while(<F>) {
			next unless /^author\s.*\s(\d+)\s[-+]\d{4}$/;
			$branch_date{$head} = $1;
			last;
		}
		close(F);
	}
	closedir(D);

The purpose of the loop is to grab all branch heads, and grab author timestamp for all of them, _and_ stash that in %branch_date hash indexed by head name. You would want to have something like this executed for each branch:

	$refname = %(refname);
	$authorline = %(author);
	$authorline =~ /^author\s.*\s(\d+)\s[-+]\d{4}$/;
        $branch_date{$refname} = $1;

So, you have the tool to write such a program for you, by doing something like this:

	my $template = '
                $refname = %(refname);
                $authorline = %(author);
                $authorline =~ /^author\s.*\s(\d+)\s[-+]\d{4}$/;
                $branch_date{$refname} = $1;
	';
        my @cmd = ('git-for-each-ref', '--perl', "--format=$template");

	open I, '-|', @cmd, 'refs/heads';
        my $script = join('',<I>);
        close I;

	my ($refname, $authorline);
        eval "$script";

The language specific quoting flag --perl affects how %() are interpolated into the generated program text as literals. That's why there is no quote around %(refname) or %(author) in the example above when defining the $template.

Junio C Hamano wrote:

Thanks for the education. Very simple, and very powerful. I knew there was a reason for it out there. Will respin a V3 patch in a bit.

-apw

cvsimport: move over to using git-for-each-ref to read refs V3

cvsimport opens all of the files in $GIT_DIR/refs/heads and reads out the sha1's in order to work out what time the last commit on that branch was made (in CVS) thus allowing incremental updates. However, this takes no account of hierachical refs naming producing the following error for each directory in $GIT_DIR/refs:

  Use of uninitialized value in chomp at /usr/bin/git-cvsimport line 503.
  Use of uninitialized value in concatenation (.) or string at
					/usr/bin/git-cvsimport line 505.
  usage: git-cat-file [-t|-s|-e|-p|<type>] <sha1>

Take advantage of the new packed refs work to use the new for-each-ref iterator to get this information.

Signed-off-by: Andy Whitcroft <apw@shadowen.org>
---

Andy Whitcroft <apw@shadowen.org> writes:

for-each-ref let's you limit the refs by leading path, so I do not think "next if" inside the loop is needed if you say

	for-each-ref --perl --format=$fmt refs/heads

Any reason you did not like my version that allows you to get rid of the while() loop altogether? (hint: replace eval in my example with "print" and see what you are evaling)

Junio C Hamano wrote:

Stupid boy :)

To my mind we avoid the 'mozilla repo' issue of 1000's of heads busting some string length limit, or requiring some _vast_ string to hold it (as perl is likely to cope). Given we are sifting a small percentage of the data out of it.

Let me know which way you want it and we'll go that way :).

-apw

Andy Whitcroft <apw@shadowen.org> writes:

Quite valid reasoning.

Will apply V3 with refs/heads filtering added; no need to resend.

Andy Whitcroft <apw@shadowen.org> writes:

This probably showed that my initial description and example for the feature found in Documentation/git-for-each-ref was lacking.

I would appreciate a separate patch to enhance it if you are so inclined.

Junio C Hamano wrote:

Actually the description in that manual page is pretty good. But I thought it was undocumented as the standard install does not install the manual pages and I was getting the ones from my stable install to fool me into thinking they were installed. New developer error, doh.

That said I've taken the liberty of updating and clarifying the language to make it very obvious from the outset that the language specific output formats are for generating eval'able snippets and added a simple example of it to complement your complex one.

Patch to follow.

-apw

Andy Whitcroft <apw@shadowen.org> writes:

Thanks.