threads / discuss / 4273

Incremental cvsimports

Subject: Incremental cvsimports

## tl;dr

10 messages between May 24, 2006 and May 24, 2006.

replies: 9people: 4as markdown or json

Geoff Russell· May 24, 2006, 07:46 UTC · lore
Dear Git,
I have a 3Gb repository (10 years of history) which I want to import into git.

I have tested little pieces of that and a 23Mb directory expands to 120Mb prior to packing/pruning. Therefore I don't have a big enough disk to import the whole repository at once. So I figure I could do it in pieces with a pack after each piece.

So I tried something like:
 1.      $ git cvsimport  -o austrics -v -d /cvsroot -C /GitRepo /cvsroot/A
followed by
 2.     $ git cvsimport  -o austrics -v -d /cvsroot -C /GitRepo /cvsroot/B

After 1, /GitRepo/.git is set up but /GitRepo has the members of A, not A itself. After 2, /GitRepo also has members of B. This isn't quite what I wanted!

Is there a way to do this?

Cheers, Geoff Russell

Martin Langhoff· May 24, 2006, 08:21 UTC · re: Geoff Russell · lore

Re: Incremental cvsimports

On 5/24/06, Geoff Russell <geoffrey.russell@gmail.com> wrote:
> Dear Git,
Dear Geoff,

if you look at the list archive for the last couple of days, you'll see there's been quite a bit of activity in tuning cvsimport so that it scales better with large imports like yours. We have been playing with a gentoo cvs repo with 300K commits / 1.6GB uncompressed.

Don't split up the tree... that'll lead to something rather ackward. Instead, fetch and build git from Junio's 'master' branch which seems to have collected most (all?) of the patches posted, including one from Linus that will repack the repo every 1K commits -- keeping the import size down.

You _will_ need a lot of memory though, as cvsps grows large (working on a workaround now) and cvsimport grows a bit over time (where is that last leak?!). And a fast machine -- specially fast IO. I've just switched from an old test machine to an AMD64 with fast disks, and it's importing around 10K commits per hour.

You will probably want to run cvsps by hand, and later use the -P flag.
cheers,
martin
Martin Langhoff· May 24, 2006, 08:25 UTC · re: Martin Langhoff · lore

Re: Incremental cvsimports

On 5/24/06, Martin Langhoff <martin.langhoff@gmail.com> wrote:
> You _will_ need a lot of memory though, as cvsps grows large (working
> on a workaround now)

While the workaround is worked around, you can create the cvsps file using something like

CVSROOT=~/tmp/gentoorepo/ cvsps --norc -x -u -A gentoo-x86 > gentoo.cvsps 2> gentoo.err

and then pass a -P gentoo.cvsps to cvsimport. s/gentoo/myproject/ ;-)
martin
Geoff Russell· May 24, 2006, 11:19 UTC · re: Martin Langhoff · lore

Re: Incremental cvsimports

Dear Martin,
On 5/24/06, Martin Langhoff <martin.langhoff@gmail.com> wrote:
Show 15 quoted lines
> On 5/24/06, Geoff Russell <geoffrey.russell@gmail.com> wrote:
> > Dear Git,
>
> Dear Geoff,
>
> if you look at the list archive for the last couple of days, you'll
> see there's been quite a bit of activity in tuning cvsimport so that
> it scales better with large imports like yours. We have been playing
> with a gentoo cvs repo with 300K commits / 1.6GB uncompressed.
>
> Don't split up the tree... that'll lead to something rather ackward.
> Instead, fetch and build git from Junio's 'master' branch which seems
> to have collected most (all?) of the patches posted, including one
> from Linus that will repack the repo every 1K commits -- keeping the
> import size down.

I got the latest git and yes, the size is kept down. I've only tried with a smaller repository but it looks promising. When I ran git-cvsimport without a CVS-module name (wanting the entire repository), it gave me a Usage message indicating that the CVS-module name was optional - but it isn't :)

I did have to change 2 lines in git-cvsimport to get it to run with my 5.8.0 perl (problems with POSIX errno). I've attached a patch but my work around isn't as quick as what it replaced.

Many thanks, I'll have a go with the big repository at work tomorrow!

Cheers, Geoff Russell

P.S. I've just started to look with git. We have wanted a cvs replacement for a while but have been too scared to change (until now).

Show 6 quoted lines
>
> You _will_ need a lot of memory though, as cvsps grows large (working
> on a workaround now) and cvsimport grows a bit over time (where is
> that last leak?!). And a fast machine -- specially fast IO. I've just
> switched from an old test machine to an AMD64 with fast disks, and
> it's importing around 10K commits per hour.
I
Show 9 quoted lines
>
> You will probably want to run cvsps by hand, and later use the -P flag.
>
> cheers,
>
>
> martin
>
>
Jeff King· May 24, 2006, 12:22 UTC · re: Geoff Russell · lore

Re: Incremental cvsimports

On Wed, May 24, 2006 at 08:49:03PM +0930, Geoff Russell wrote:
> I did have to change 2 lines in git-cvsimport to get it to run with my
> 5.8.0 perl (problems with POSIX errno). I've attached a patch but my
> work around isn't as quick as what it replaced.

Can you describe your problem in more detail? The POSIX errno constants have been available since long before 5.8.0, so we should be able to use them.

(btw, the change was introduced in my commit() cleanups:
  e73aefe4fdba0d161d9878642c69b40d83a0204c).
-Peff
Geoff Russell· May 24, 2006, 12:33 UTC · re: Jeff King · lore

Re: Incremental cvsimports

Dear Jeff,
See below.
On 5/24/06, Jeff King <peff@peff.net> wrote:
Show 9 quoted lines
> On Wed, May 24, 2006 at 08:49:03PM +0930, Geoff Russell wrote:
>
> > I did have to change 2 lines in git-cvsimport to get it to run with my
> > 5.8.0 perl (problems with POSIX errno). I've attached a patch but my
> > work around isn't as quick as what it replaced.
>
> Can you describe your problem in more detail? The POSIX errno constants
> have been available since long before 5.8.0, so we should be able to use
> them.
   $ ./git-cvsimport
   ":errno_h" is not exported by the POSIX module
   Can't continue after import errors at
/usr/lib/perl5/5.8.0/i386-linux-thread-multi/POSIX.pm line 19
    BEGIN failed--compilation aborted at ./git-cvsimport line 26.

When I deleted ":errno_h" I needed to patch the place it was used (as per patch I attached in original post).

Cheers, Geoff Russell

Show 11 quoted lines
>
> (btw, the change was introduced in my commit() cleanups:
>   e73aefe4fdba0d161d9878642c69b40d83a0204c).
>
> -Peff
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
Jeff King· May 24, 2006, 13:23 UTC · re: Geoff Russell · lore

Re: Incremental cvsimports

On Wed, May 24, 2006 at 10:03:44PM +0930, Geoff Russell wrote:
>   ":errno_h" is not exported by the POSIX module
>   Can't continue after import errors at
> /usr/lib/perl5/5.8.0/i386-linux-thread-multi/POSIX.pm line 19
>    BEGIN failed--compilation aborted at ./git-cvsimport line 26.
Hmm. It looks like something is nonstandard in your setup. I just compiled
5.8.0 from source and the :errno_h tag works fine. What is your
platform?  Can you try the following and let me know which work:
  $ perl -e 'use POSIX qw(:errno_h)'
  $ perl -e 'use POSIX qw(errno_h)'
  $ perl -e 'use Errno'
-Peff
Geoff Russell· May 24, 2006, 13:47 UTC · re: Jeff King · lore

Re: Incremental cvsimports

Hi Jeff,
On 5/24/06, Jeff King <peff@peff.net> wrote:
Show 10 quoted lines
> On Wed, May 24, 2006 at 10:03:44PM +0930, Geoff Russell wrote:
>
> >   ":errno_h" is not exported by the POSIX module
> >   Can't continue after import errors at
> > /usr/lib/perl5/5.8.0/i386-linux-thread-multi/POSIX.pm line 19
> >    BEGIN failed--compilation aborted at ./git-cvsimport line 26.
>
> Hmm. It looks like something is nonstandard in your setup. I just compiled
> 5.8.0 from source and the :errno_h tag works fine. What is your
> platform?  Can you try the following and let me know which work:
I compiled perl from source on Mandrake 9.1.
>   $ perl -e 'use POSIX qw(:errno_h)'
>   $ perl -e 'use POSIX qw(errno_h)'
>   $ perl -e 'use Errno'

All 3 work. But if I add a second tag before the ':errno_h", then I get an error.

The "use" line that makes git-cvsimport compile for me is:
        use POSIX qw(strftime dup2 ENOENT);
Which just imports the required symbol and not the full tag list.

Cheers, Geoff.

Show 8 quoted lines
>
> -Peff
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
Jeff King· May 24, 2006, 13:58 UTC · re: Geoff Russell · lore

Re: Incremental cvsimports

On Wed, May 24, 2006 at 11:17:32PM +0930, Geoff Russell wrote:
Show 6 quoted lines
> All 3 work.  But if I add a second tag before the ':errno_h", then I
> get an error.
> 
> The "use" line that makes git-cvsimport compile for me is:
> 
>        use POSIX qw(strftime dup2 ENOENT);

Odd. It's either a bug with importing tags in older versions, or there's some deep perl voodoo that I don't understand (either way, it is "fixed" in more recent versions). Importing ENOENT directly is reasonable.

Junio, can you apply the following fix?
diff --git a/git-cvsimport.perl b/git-cvsimport.perl
index af331d9..76f6246 100755
--- a/git-cvsimport.perl
+++ b/git-cvsimport.perl
@@ -23,7 +23,7 @@ use File::Basename qw(basename dirname);
 use Time::Local;
 use IO::Socket;
 use IO::Pipe;
-use POSIX qw(strftime dup2 :errno_h);
+use POSIX qw(strftime dup2 ENOENT);
 use IPC::Open2;
 
 $SIG{'PIPE'}="IGNORE";
Junio C Hamano· May 24, 2006, 17:05 UTC · re: Jeff King · lore

Re: Incremental cvsimports

Jeff King <peff@peff.net> writes:
> Odd. It's either a bug with importing tags in older versions, or there's
> some deep perl voodoo that I don't understand (either way, it is "fixed"
> in more recent versions).  Importing ENOENT directly is reasonable.

Sounds good. Thanks for the back-and-forth helping others in the community. I appreciate it.

> Junio, can you apply the following fix?

Will do, but I would have preferred if you did the commit log message and the stuff properly. Less work for me ;-).

Show 14 quoted lines
>
> diff --git a/git-cvsimport.perl b/git-cvsimport.perl
> index af331d9..76f6246 100755
> --- a/git-cvsimport.perl
> +++ b/git-cvsimport.perl
> @@ -23,7 +23,7 @@ use File::Basename qw(basename dirname);
>  use Time::Local;
>  use IO::Socket;
>  use IO::Pipe;
> -use POSIX qw(strftime dup2 :errno_h);
> +use POSIX qw(strftime dup2 ENOENT);
>  use IPC::Open2;
>  
>  $SIG{'PIPE'}="IGNORE";

← back to recent threads