{"thread":{"id":"4273","subject":"Incremental cvsimports","startedAt":"2006-05-24T07:46:57Z","lastAt":"2006-05-24T17:05:23Z","messageCount":10,"participants":["Geoff Russell","Martin Langhoff","Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"20612","messageId":"93c3eada0605240046t10e00119n4cfc39ec33fe1d92@mail.gmail.com","threadId":"4273","inReplyTo":null,"subject":"Incremental cvsimports","fromName":"Geoff Russell","fromEmail":"geoffrey.russell@gmail.com","sentAt":"2006-05-24T07:46:57Z","receivedAt":"2006-05-24T07:46:57Z","isPatch":false,"sender":{"key":"geoffrey.russell@gmail.com","avatar":"https://gravatar.com/avatar/c30f497ccfa6bf06d86f30bd2ba092a2dd124c61c6bc902f7cb5c3f6486947de?d=mp&s=160"},"body":"Dear Git,\n\nI have a 3Gb repository (10 years of history) which I want to import into git.\n\nI have tested little pieces of that and a 23Mb directory expands to 120Mb prior\nto packing/pruning.  Therefore I don't have a big enough disk to import the\nwhole repository at once. So I figure I could do it in pieces with a pack after\neach piece.\n\nSo I tried something like:\n\n 1.      $ git cvsimport  -o austrics -v -d /cvsroot -C /GitRepo /cvsroot/A\n\nfollowed by\n\n 2.     $ git cvsimport  -o austrics -v -d /cvsroot -C /GitRepo /cvsroot/B\n\nAfter 1, /GitRepo/.git is set up but /GitRepo has the members of A,\nnot A itself.\nAfter 2, /GitRepo also has members of B. This isn't quite what I wanted!\n\nIs there a way to do this?\n\nCheers,\nGeoff Russell\n"},{"id":"20615","messageId":"46a038f90605240121o117fadb6vf3ce910a3ad3e90@mail.gmail.com","threadId":"4273","inReplyTo":"93c3eada0605240046t10e00119n4cfc39ec33fe1d92@mail.gmail.com","subject":"Re: Incremental cvsimports","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-05-24T08:21:19Z","receivedAt":"2006-05-24T08:21:19Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 5/24/06, Geoff Russell <geoffrey.russell@gmail.com> wrote:\n> Dear Git,\n\nDear Geoff,\n\nif you look at the list archive for the last couple of days, you'll\nsee there's been quite a bit of activity in tuning cvsimport so that\nit scales better with large imports like yours. We have been playing\nwith a gentoo cvs repo with 300K commits / 1.6GB uncompressed.\n\nDon't split up the tree... that'll lead to something rather ackward.\nInstead, fetch and build git from Junio's 'master' branch which seems\nto have collected most (all?) of the patches posted, including one\nfrom Linus that will repack the repo every 1K commits -- keeping the\nimport size down.\n\nYou _will_ need a lot of memory though, as cvsps grows large (working\non a workaround now) and cvsimport grows a bit over time (where is\nthat last leak?!). And a fast machine -- specially fast IO. I've just\nswitched from an old test machine to an AMD64 with fast disks, and\nit's importing around 10K commits per hour.\n\nYou will probably want to run cvsps by hand, and later use the -P flag.\n\ncheers,\n\n\nmartin\n"},{"id":"20616","messageId":"46a038f90605240125q3fb58466w55745348846c48fc@mail.gmail.com","threadId":"4273","inReplyTo":"46a038f90605240121o117fadb6vf3ce910a3ad3e90@mail.gmail.com","subject":"Re: Incremental cvsimports","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-05-24T08:25:39Z","receivedAt":"2006-05-24T08:25:39Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 5/24/06, Martin Langhoff <martin.langhoff@gmail.com> wrote:\n> You _will_ need a lot of memory though, as cvsps grows large (working\n> on a workaround now)\n\nWhile the workaround is worked around, you can create the cvsps file\nusing something like\n\nCVSROOT=~/tmp/gentoorepo/ cvsps --norc -x -u -A gentoo-x86  >\ngentoo.cvsps 2> gentoo.err\n\nand then pass a -P gentoo.cvsps to cvsimport. s/gentoo/myproject/ ;-)\n\n\nmartin\n"},{"id":"20622","messageId":"93c3eada0605240419o48891cdle6c100fc0ac870ff@mail.gmail.com","threadId":"4273","inReplyTo":"46a038f90605240121o117fadb6vf3ce910a3ad3e90@mail.gmail.com","subject":"Re: Incremental cvsimports","fromName":"Geoff Russell","fromEmail":"geoffrey.russell@gmail.com","sentAt":"2006-05-24T11:19:03Z","receivedAt":"2006-05-24T11:19:03Z","isPatch":false,"sender":{"key":"geoffrey.russell@gmail.com","avatar":"https://gravatar.com/avatar/c30f497ccfa6bf06d86f30bd2ba092a2dd124c61c6bc902f7cb5c3f6486947de?d=mp&s=160"},"body":"Dear Martin,\n\n\nOn 5/24/06, Martin Langhoff <martin.langhoff@gmail.com> wrote:\n> On 5/24/06, Geoff Russell <geoffrey.russell@gmail.com> wrote:\n> > Dear Git,\n>\n> Dear Geoff,\n>\n> if you look at the list archive for the last couple of days, you'll\n> see there's been quite a bit of activity in tuning cvsimport so that\n> it scales better with large imports like yours. We have been playing\n> with a gentoo cvs repo with 300K commits / 1.6GB uncompressed.\n>\n> Don't split up the tree... that'll lead to something rather ackward.\n> Instead, fetch and build git from Junio's 'master' branch which seems\n> to have collected most (all?) of the patches posted, including one\n> from Linus that will repack the repo every 1K commits -- keeping the\n> import size down.\n\nI got the latest git and yes, the size is kept down. I've only tried with\na smaller repository but it looks promising. When I ran git-cvsimport without a\nCVS-module name (wanting the entire repository), it gave me a Usage message\nindicating that the CVS-module name was optional - but it isn't :)\n\nI did have to change\n2 lines in git-cvsimport to get it to run with my 5.8.0 perl (problems with\nPOSIX errno). I've attached a patch but my work around isn't as quick as\nwhat it replaced.\n\nMany thanks, I'll have a go with the big repository at work tomorrow!\n\nCheers,\nGeoff Russell\n\nP.S. I've just started to look with git. We have wanted a cvs replacement for\na while but have been too scared to change (until now).\n\n\n\n>\n> You _will_ need a lot of memory though, as cvsps grows large (working\n> on a workaround now) and cvsimport grows a bit over time (where is\n> that last leak?!). And a fast machine -- specially fast IO. I've just\n> switched from an old test machine to an AMD64 with fast disks, and\n> it's importing around 10K commits per hour.\n\nI\n\n>\n> You will probably want to run cvsps by hand, and later use the -P flag.\n>\n> cheers,\n>\n>\n> martin\n>\n>\n"},{"id":"20623","messageId":"20060524122246.GA3997@coredump.intra.peff.net","threadId":"4273","inReplyTo":"93c3eada0605240419o48891cdle6c100fc0ac870ff@mail.gmail.com","subject":"Re: Incremental cvsimports","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2006-05-24T12:22:46Z","receivedAt":"2006-05-24T12:22:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, May 24, 2006 at 08:49:03PM +0930, Geoff Russell wrote:\n\n> I did have to change 2 lines in git-cvsimport to get it to run with my\n> 5.8.0 perl (problems with POSIX errno). I've attached a patch but my\n> work around isn't as quick as what it replaced.\n\nCan you describe your problem in more detail? The POSIX errno constants\nhave been available since long before 5.8.0, so we should be able to use\nthem.\n\n(btw, the change was introduced in my commit() cleanups:\n  e73aefe4fdba0d161d9878642c69b40d83a0204c).\n\n-Peff\n"},{"id":"20624","messageId":"93c3eada0605240533q4d1b5b81p128dc2b905aa9976@mail.gmail.com","threadId":"4273","inReplyTo":"20060524122246.GA3997@coredump.intra.peff.net","subject":"Re: Incremental cvsimports","fromName":"Geoff Russell","fromEmail":"geoffrey.russell@gmail.com","sentAt":"2006-05-24T12:33:44Z","receivedAt":"2006-05-24T12:33:44Z","isPatch":false,"sender":{"key":"geoffrey.russell@gmail.com","avatar":"https://gravatar.com/avatar/c30f497ccfa6bf06d86f30bd2ba092a2dd124c61c6bc902f7cb5c3f6486947de?d=mp&s=160"},"body":"Dear Jeff,\n\nSee below.\n\nOn 5/24/06, Jeff King <peff@peff.net> wrote:\n> On Wed, May 24, 2006 at 08:49:03PM +0930, Geoff Russell wrote:\n>\n> > I did have to change 2 lines in git-cvsimport to get it to run with my\n> > 5.8.0 perl (problems with POSIX errno). I've attached a patch but my\n> > work around isn't as quick as what it replaced.\n>\n> Can you describe your problem in more detail? The POSIX errno constants\n> have been available since long before 5.8.0, so we should be able to use\n> them.\n\n\n   $ ./git-cvsimport\n\n   \":errno_h\" is not exported by the POSIX module\n   Can't continue after import errors at\n/usr/lib/perl5/5.8.0/i386-linux-thread-multi/POSIX.pm line 19\n    BEGIN failed--compilation aborted at ./git-cvsimport line 26.\n\nWhen I deleted \":errno_h\" I needed to patch the place it was used (as per patch\nI attached in original post).\n\nCheers,\nGeoff Russell\n\n\n\n>\n> (btw, the change was introduced in my commit() cleanups:\n>   e73aefe4fdba0d161d9878642c69b40d83a0204c).\n>\n> -Peff\n> -\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n>\n"},{"id":"20626","messageId":"20060524132317.GA4594@coredump.intra.peff.net","threadId":"4273","inReplyTo":"93c3eada0605240533q4d1b5b81p128dc2b905aa9976@mail.gmail.com","subject":"Re: Incremental cvsimports","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2006-05-24T13:23:17Z","receivedAt":"2006-05-24T13:23:17Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, May 24, 2006 at 10:03:44PM +0930, Geoff Russell wrote:\n\n>   \":errno_h\" is not exported by the POSIX module\n>   Can't continue after import errors at\n> /usr/lib/perl5/5.8.0/i386-linux-thread-multi/POSIX.pm line 19\n>    BEGIN failed--compilation aborted at ./git-cvsimport line 26.\n\nHmm. It looks like something is nonstandard in your setup. I just compiled\n5.8.0 from source and the :errno_h tag works fine. What is your\nplatform?  Can you try the following and let me know which work:\n  $ perl -e 'use POSIX qw(:errno_h)'\n  $ perl -e 'use POSIX qw(errno_h)'\n  $ perl -e 'use Errno'\n\n-Peff\n"},{"id":"20628","messageId":"93c3eada0605240647i48db0588ja343e348f5feb08e@mail.gmail.com","threadId":"4273","inReplyTo":"20060524132317.GA4594@coredump.intra.peff.net","subject":"Re: Incremental cvsimports","fromName":"Geoff Russell","fromEmail":"geoffrey.russell@gmail.com","sentAt":"2006-05-24T13:47:32Z","receivedAt":"2006-05-24T13:47:32Z","isPatch":false,"sender":{"key":"geoffrey.russell@gmail.com","avatar":"https://gravatar.com/avatar/c30f497ccfa6bf06d86f30bd2ba092a2dd124c61c6bc902f7cb5c3f6486947de?d=mp&s=160"},"body":"Hi Jeff,\n\nOn 5/24/06, Jeff King <peff@peff.net> wrote:\n> On Wed, May 24, 2006 at 10:03:44PM +0930, Geoff Russell wrote:\n>\n> >   \":errno_h\" is not exported by the POSIX module\n> >   Can't continue after import errors at\n> > /usr/lib/perl5/5.8.0/i386-linux-thread-multi/POSIX.pm line 19\n> >    BEGIN failed--compilation aborted at ./git-cvsimport line 26.\n>\n> Hmm. It looks like something is nonstandard in your setup. I just compiled\n> 5.8.0 from source and the :errno_h tag works fine. What is your\n> platform?  Can you try the following and let me know which work:\n\nI compiled perl from source on Mandrake 9.1.\n\n>   $ perl -e 'use POSIX qw(:errno_h)'\n>   $ perl -e 'use POSIX qw(errno_h)'\n>   $ perl -e 'use Errno'\n\nAll 3 work.  But if I add a second tag before the ':errno_h\", then I\nget an error.\n\nThe \"use\" line that makes git-cvsimport compile for me is:\n\n        use POSIX qw(strftime dup2 ENOENT);\n\nWhich just imports the required symbol and not the full tag list.\n\nCheers,\nGeoff.\n\n>\n> -Peff\n> -\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n>\n"},{"id":"20629","messageId":"20060524135828.GA23934@coredump.intra.peff.net","threadId":"4273","inReplyTo":"93c3eada0605240647i48db0588ja343e348f5feb08e@mail.gmail.com","subject":"Re: Incremental cvsimports","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2006-05-24T13:58:28Z","receivedAt":"2006-05-24T13:58:28Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, May 24, 2006 at 11:17:32PM +0930, Geoff Russell wrote:\n\n> All 3 work.  But if I add a second tag before the ':errno_h\", then I\n> get an error.\n> \n> The \"use\" line that makes git-cvsimport compile for me is:\n> \n>        use POSIX qw(strftime dup2 ENOENT);\n\nOdd. It's either a bug with importing tags in older versions, or there's\nsome deep perl voodoo that I don't understand (either way, it is \"fixed\"\nin more recent versions).  Importing ENOENT directly is reasonable.\n\nJunio, can you apply the following fix?\n\ndiff --git a/git-cvsimport.perl b/git-cvsimport.perl\nindex af331d9..76f6246 100755\n--- a/git-cvsimport.perl\n+++ b/git-cvsimport.perl\n@@ -23,7 +23,7 @@ use File::Basename qw(basename dirname);\n use Time::Local;\n use IO::Socket;\n use IO::Pipe;\n-use POSIX qw(strftime dup2 :errno_h);\n+use POSIX qw(strftime dup2 ENOENT);\n use IPC::Open2;\n \n $SIG{'PIPE'}=\"IGNORE\";\n"},{"id":"20636","messageId":"7virnv8ib0.fsf@assigned-by-dhcp.cox.net","threadId":"4273","inReplyTo":"20060524135828.GA23934@coredump.intra.peff.net","subject":"Re: Incremental cvsimports","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-05-24T17:05:23Z","receivedAt":"2006-05-24T17:05:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> Odd. It's either a bug with importing tags in older versions, or there's\n> some deep perl voodoo that I don't understand (either way, it is \"fixed\"\n> in more recent versions).  Importing ENOENT directly is reasonable.\n\nSounds good.  Thanks for the back-and-forth helping others in\nthe community.  I appreciate it.\n\n> Junio, can you apply the following fix?\n\nWill do, but I would have preferred if you did the commit log\nmessage and the stuff properly.  Less work for me ;-).\n\n>\n> diff --git a/git-cvsimport.perl b/git-cvsimport.perl\n> index af331d9..76f6246 100755\n> --- a/git-cvsimport.perl\n> +++ b/git-cvsimport.perl\n> @@ -23,7 +23,7 @@ use File::Basename qw(basename dirname);\n>  use Time::Local;\n>  use IO::Socket;\n>  use IO::Pipe;\n> -use POSIX qw(strftime dup2 :errno_h);\n> +use POSIX qw(strftime dup2 ENOENT);\n>  use IPC::Open2;\n>  \n>  $SIG{'PIPE'}=\"IGNORE\";\n"}]}