Re: Incremental CVS update
- From
Jon Smirl <jonsmirl@gmail.com>
- Date
- Jun 22, 2006, 20:08 UTC
- Message-ID
- <9e4733910606221308v2f995adev8c5b721be0a009e2@mail.gmail.com>
- In-Reply-To
- <46a038f90606221236j2c5c9692yecef924aa769c1c9@mail.gmail.com>
On 6/22/06, Martin Langhoff <martin.langhoff@gmail.com> wrote:
Show 13 quoted lines
> On 6/23/06, Jon Smirl <jonsmirl@gmail.com> wrote: > > cvsps keeps it's incremental status in ~/.cvps/*. parsecvs might want > > to keep it's status in the .git repository and use tags to locate it. > > You could even have a utility to show when and what was imported. By > > keeping everything in git it doesn't matter who runs the incremental > > update commands. > > Jon, > > what cvsps keeps is a cache of what it knows about the repo history, > to ask only for new commits. Now, cvsps will always write to STDOUT > the full history, and git-cvsimport discards the commits it has > already seen, based on reading the state of each git head.
The cache is 723MB for the Mozilla repo. Since the info gets cached in my home directory anyone else who needs to sync the repo doesn't get to use the cache.
[jonsmirl@jonsmirl .cvsps]$ pwd /home/jonsmirl/.cvsps [jonsmirl@jonsmirl .cvsps]$ ls -l total 707492 -rw-rw-r-- 1 jonsmirl jonsmirl 723758657 Jun 15 16:10 #home#mozcvs##mozilla [jonsmirl@jonsmirl .cvsps]$
Keith is rewriting parsecvs. If you analyze all of the data structures, the info needed for the conversion should be able to fit into well under 100MB instead of the ~2GB the current programs are using.
There are lots of ways to reduce memory consumption. You can turm CVS revisions into git IDs as soon as the revision is seen. That lets you get away from tracking file names and long CVS revision numbers. It also works to turn the author/log fields immediately into a hash. When possible switching to arrays instead of linked list is smaller too.
Some stats: 1M revisions 200K unique changesets (author/log combos) 200KB symbols 1,800 branches
cvsps has the lowest memory consumption, it uses 1200 bytes per revision. It looks like it is possible to lower this to less than 100 bytes per rev.
Show 9 quoted lines
> > So cvsps + git-cvsimport don't keep any extra data around, and I am > 100% certain that parsecvs don't need that either. > > cheers, > > > martin >
-- Jon Smirl jonsmirl@gmail.com