Re: Importing Mozilla CVS into git

From: Keith Packard <keithp@keithp.com>
Date: Jun 2, 2006, 02:07 UTC
Message-ID: <1149214075.5521.31.camel@neko.keithp.com>
In-Reply-To: <9e4733910606011755n29a149f2m1409c5a23888f1c5@mail.gmail.com>

On Thu, 2006-06-01 at 20:55 -0400, Jon Smirl wrote:

> With the attached patch you can parse the entire Mozilla tree. The
> tree has over 100,000 files in it and about 300 branches.

that's good news.

> I ran this overnight and it failed with out of memory on a 1GB machine
> with 1GB swap. If failed in the branch processing, the parse phase
> succeeded.

yeah, parsecvs has some internal storage inefficiencies which need to be addressed. In particular, every commit has a pointer to the related revision of every file in the commit. Much like git used to store every filename in the commit object and was changed to share common directory contents, parsecvs should be fixed to do the same.

> How much memory does something like this need?

It's basically 4 * nrev * nfile bytes on a 32-bit machine, multiply by 2 for a 64-bit machine.

Show 19 quoted lines

> If you want to quickly check out the branch processing problems use
> rsync to pull down just a copy of repository files.
> 
> I am getting 1000s of warnings like these and I haven't figured out why yet.
> 
> Warning: ../mozilla/mozilla/build/mac/build_scripts/Attic/MozillaCheckoutList.txt,v:
> unnamed branch from master
> Warning: ../mozilla/mozilla/build/unix/run-mozilla.sh,v: unnamed
> branch from master
> Warning: ../mozilla/mozilla/Makefile.in,v: unnamed branch from master of 99855
> Warning: ../mozilla/mozilla/Makefile.in,v: unnamed branch from master
> Warning: ../mozilla/mozilla/allmakefiles.sh,v: unnamed branch from master99855
> Warning: ../mozilla/mozilla/allmakefiles.sh,v: unnamed branch from master
> Warning: ../mozilla/mozilla/cmd/macfe/MailNews/AddressBook/Attic/UAddressBookUtilities.cp,v:
> unnamed branch from master
> Warning: ../mozilla/mozilla/cmd/macfe/MailNews/AddressBook/Attic/UAddressBookUtilities.h,v:
> unnamed branch from master
> Warning: ../mozilla/mozilla/cmd/macfe/central/Attic/msv2dsk.cp,v:
> unnamed branch from master

yeah, these happen when vendor branches go awry. I'll pull the repository and take a look. X.org had similar 'issues' as the current CVS repo was built by merging mesa, XFree86 and X.org together in a rather haphazard fashion.

-- 
keith.packard@intel.com