Re: packs and trees

From: Keith Packard <keithp@keithp.com>
Date: Jun 20, 2006, 15:18 UTC
Message-ID: <1150816728.5382.27.camel@neko.keithp.com>
In-Reply-To: <9e4733910606200735u5741a9adr83264ae7d51dd37@mail.gmail.com>

On Tue, 2006-06-20 at 10:35 -0400, Jon Smirl wrote:

> Keith's parsecvs run ended up in a loop and mine hit a parsecvs error
> and then had memory corruption after about eight hours. That was last
> week,  I just checked the logs and I don't see any comments about
> fixing it.

Yeah, I'm rewriting the tool; the current codebase isn't supportable.

> Even after spending eight hours building the changeset info iit is
> still going to take it a couple of days to retrieve the versions one
> at a time and write them to git. Reparsing 50MB delta files n^2/2
> times is a major bottleneck for all three programs.

The eight hours in question *were* writing out the deltas and packing the resulting trees. All that remained was to construct actual commit objects and write them out.

The problem was that parsecvs's internals are structured so that this processes would take a large amount of memory, so I'm reworking the code to free stuff as it goes along.

With a rewritten parsecvs, I'm hoping to be able to steal the algorithms from cvs2svn and stick those in place. Then work on truncating the history so it can deal with incremental updates to the repository, which I think will be straightforward if we stick a few breadcrumbs in the git repository to recover state from.

-- 
keith.packard@intel.com