threads / discuss / 4639

CVS import broken?

Subject: CVS import broken?

## tl;dr

9 messages between Jun 23, 2006 and Jun 24, 2006.

replies: 8people: 4as markdown or json

Johannes Schindelin· Jun 23, 2006, 16:14 UTC · lore
Hi,

I track quite a few CVS repos with 'git-cvsimport -k -i', but recently it stopped working (of course, I was not reimporting, but incrementally tracking them). I think it is the introduction of one-index-per-branch policy:

> fatal: index file mmap failed (Invalid argument)
> unable to write to git-update-index:  at /.../git-cvsimport line 606, <CVS> line 277425.

It seems that git-cvsimport makes a temporary file of size 0, which cannot get mmap()ed, because it has size 0.

Any help appreciated, Dscho

Martin Langhoff· Jun 24, 2006, 05:09 UTC · re: Johannes Schindelin · lore

[PATCH] cvsimport - streamline temp index file creation and avoid creating empty tmpfiles

On 6/24/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> It seems that git-cvsimport makes a temporary file of size 0, which cannot
> get mmap()ed, because it has size 0.

This switch to tmpnam() avoids creating the tmpfile in the first place and streamlines the code. This handling of tmpfiles is slightly safer, but there is an inherent race condition.

---
NOTE: (a) I cannot reproduce the problem and (b) this is only lightly tested,
if trivial.

However, this switch to tmpnam() avoids creating the tmpfile in the first place and streamlines the code. This usage of tempfiles is open to a race condition if someone could guess the name returned by tmpnam, but even this is safer than what we did before, which was creating a file, closing the fh and then clobbering it from git-read-tree.

And if someone can guess the name that tmpnam() returns their magic is strong
enough that they'll go for more interesting targets.
Signed-off-by: Martin Langhoff <martin@catalyst.net.nz>
---
 git-cvsimport.perl |   20 +++++---------------
 1 files changed, 5 insertions(+), 15 deletions(-)
diff --git a/git-cvsimport.perl b/git-cvsimport.perl
index f3daa6c..d961b7b 100644
--- a/git-cvsimport.perl
+++ b/git-cvsimport.perl
@@ -17,7 +17,7 @@ use strict;
 use warnings;
 use Getopt::Std;
 use File::Spec;
-use File::Temp qw(tempfile);
+use File::Temp qw(tempfile tmpnam);
 use File::Path qw(mkpath);
 use File::Basename qw(basename dirname);
 use Time::Local;
@@ -467,12 +467,8 @@ my $orig_git_index;
 $orig_git_index = $ENV{GIT_INDEX_FILE} if exists $ENV{GIT_INDEX_FILE};
 
 my %index; # holds filenames of one index per branch
-{   # init with an index for origin
-    my ($fh, $fn) = tempfile('gitXXXXXX', SUFFIX => '.idx',
-			     DIR => File::Spec->tmpdir());
-    close ($fh);
-    $index{$opt_o} = $fn;
-}
+$index{$opt_o} = tmpnam();
+
 $ENV{GIT_INDEX_FILE} = $index{$opt_o};
 unless(-d $git_dir) {
 	system("git-init-db");
@@ -502,10 +498,7 @@ unless(-d $git_dir) {
 
 	# populate index
 	unless ($index{$last_branch}) {
-	    my ($fh, $fn) = tempfile('gitXXXXXX', SUFFIX => '.idx',
-				     DIR => File::Spec->tmpdir());
-	    close ($fh);
-	    $index{$last_branch} = $fn;
+	    $index{$last_branch} = tmpnam();
 	}
 	$ENV{GIT_INDEX_FILE} = $index{$last_branch};
 	system('git-read-tree', $last_branch);
@@ -818,10 +811,7 @@ while(<CVS>) {
 		if(($ancestor || $branch) ne $last_branch) {
 			print "Switching from $last_branch to $branch\n" if $opt_v;
 			unless ($index{$branch}) {
-			    my ($fh, $fn) = tempfile('gitXXXXXX', SUFFIX => '.idx',
-						     DIR => File::Spec->tmpdir());
-			    close ($fh);
-			    $index{$branch} = $fn;
+			    $index{$branch} = tmpnam();
 			    $ENV{GIT_INDEX_FILE} = $index{$branch};
 			    system("git-read-tree", $branch);
 			    die "read-tree failed: $?\n" if $?;

-- 
1.4.0.gcda2
Junio C Hamano· Jun 24, 2006, 06:59 UTC · re: Martin Langhoff · lore

Re: [PATCH] cvsimport - streamline temp index file creation and avoid creating empty tmpfiles

Martin Langhoff <martin@catalyst.net.nz> writes:
Show 11 quoted lines
> On 6/24/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>> It seems that git-cvsimport makes a temporary file of size 0, which cannot
>> get mmap()ed, because it has size 0.
>
> This switch to tmpnam() avoids creating the tmpfile in the first place and
> streamlines the code. This handling of tmpfiles is slightly safer, but there
> is an inherent race condition.
>
> ---
> NOTE: (a) I cannot reproduce the problem and (b) this is only lightly tested,
> if trivial.
Thanks both.

I'd take this to "next" after I hear from somebody, most likely Johannes, who had trouble with the code earlier that the problem is fixed.

Johannes Schindelin· Jun 24, 2006, 09:50 UTC · re: Martin Langhoff · lore

Re: [PATCH] cvsimport - streamline temp index file creation and avoid creating empty tmpfiles

Hi,
On Sat, 24 Jun 2006, Martin Langhoff wrote:
Show 7 quoted lines
> On 6/24/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > It seems that git-cvsimport makes a temporary file of size 0, which cannot
> > get mmap()ed, because it has size 0.
> 
> This switch to tmpnam() avoids creating the tmpfile in the first place and
> streamlines the code. This handling of tmpfiles is slightly safer, but there
> is an inherent race condition.
Thank you. This fixes the error.

HOWEVER, it does not fix the main problem: when I try to git-cvsimport, there is no index for that branch yet, since I used to git-cvsimport with the old cvsimport.

Now, when cvsimport sees there is no index, it evidently assumes that the current state is an empty tree, which is *not* true.

The effect is: the first commit removes all files from the tree which were not touched by the cvs commit. Bad.

> This usage of tempfiles is open to a race condition

I would not care too strongly about that. Eventually, I really would like this file to reside in $GIT_DIR, not /tmp, but whatever. That is not my biggest concern right now. That I cannot update since June 18th, however, is.

Ciao, Dscho

Junio C Hamano· Jun 24, 2006, 10:05 UTC · re: Johannes Schindelin · lore

Re: [PATCH] cvsimport - streamline temp index file creation and avoid creating empty tmpfiles

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> I would not care too strongly about that. Eventually, I really would like 
> this file to reside in $GIT_DIR, not /tmp, but whatever. That is not my 
> biggest concern right now. That I cannot update since June 18th, however, 
> is.
Would reverting 8f732649 in the meantime be an option for you?
Martin Langhoff· Jun 24, 2006, 11:16 UTC · re: Junio C Hamano · lore

Re: [PATCH] cvsimport - streamline temp index file creation and avoid creating empty tmpfiles

Johannes, Junio,

I've managed to repro the problem -- which was totally reproduceable, I was just testing the wrong version of the script. The problem was quite obvious: when running an incremental, the first head would not get the index created properly. Even worse, when forking a new branch, the index would be empty too.

Fixed both cases and posted separately. Thanks for the sharp eyes, and sorry about the bug!

martin
Junio C Hamano· Jun 24, 2006, 11:28 UTC · re: Martin Langhoff · lore

Re: [PATCH] cvsimport - streamline temp index file creation and avoid creating empty tmpfiles

"Martin Langhoff" <martin.langhoff@gmail.com> writes:
Show 9 quoted lines
> Johannes, Junio,
>
> I've managed to repro the problem -- which was totally reproduceable,
> I was just testing the wrong version of the script. The problem was
> quite obvious: when running an incremental, the first head would not
> get the index created properly. Even worse, when forking a new branch,
> the index would be empty too.
>
> Fixed both cases and posted separately.
Thanks.  Will be in "next".
Johannes Schindelin· Jun 24, 2006, 19:05 UTC · re: Junio C Hamano · lore

Re: [PATCH] cvsimport - streamline temp index file creation and avoid creating empty tmpfiles

Hi,
On Sat, 24 Jun 2006, Junio C Hamano wrote:
Show 8 quoted lines
> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > I would not care too strongly about that. Eventually, I really would like 
> > this file to reside in $GIT_DIR, not /tmp, but whatever. That is not my 
> > biggest concern right now. That I cannot update since June 18th, however, 
> > is.
> 
> Would reverting 8f732649 in the meantime be an option for you?
I think it is worth fixing instead of working around.

Ciao, Dscho

Martin Langhoff· Jun 24, 2006, 10:08 UTC · re: Johannes Schindelin · lore

Re: [PATCH] cvsimport - streamline temp index file creation and avoid creating empty tmpfiles

On 6/24/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Thank you. This fixes the error.
Your welcome!
Show 9 quoted lines
> HOWEVER, it does not fix the main problem: when I try to git-cvsimport,
> there is no index for that branch yet, since I used to git-cvsimport with
> the old cvsimport.
>
> Now, when cvsimport sees there is no index, it evidently assumes that the
> current state is an empty tree, which is *not* true.
>
> The effect is: the first commit removes all files from the tree which were
> not touched by the cvs commit. Bad.

I don't quite understand. No it shouldn't be the case -- it should create the index using git-read-tree based on the tip of the branch. Right after the call to tmpnam() the code looks like

 $index{$branch} = tmpnam();
 $ENV{GIT_INDEX_FILE} = $index{$branch};
 system("git-read-tree", $branch);
 die "read-tree failed: $?\n" if $?;
Show 6 quoted lines
> > This usage of tempfiles is open to a race condition
>
> I would not care too strongly about that. Eventually, I really would like
> this file to reside in $GIT_DIR, not /tmp, but whatever. That is not my
> biggest concern right now. That I cannot update since June 18th, however,
> is.
It's worrying me too. Running some tests now...
martin

← back to recent threads