From: Jon Smirl <jonsmirl@gmail.com>
Date: Sun, 06 Aug 2006 15:53:43 GMT
Subject: Re: fast-import and unique objects.
Message-ID: <9e4733910608060853ua0eabc1w9b35b8414d3c9bae@mail.gmail.com>
In-Reply-To: <9e4733910608060532w51fca2c0r8038828df0d41eeb@mail.gmail.com>

On 8/6/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> This model has a lot of object duplication. I generated 949,305
> revisions, but only 754,165 are unique. I'll modify my code to build a
> hash of the objects it has seen and then not send the duplicates to
> fast-import. Those 195,140 duplicated objects may be what is tripping
> index-pack up.

New run is finished with duplicate removal.

Time to run is unchanged, still 2hrs. Run time is IO bound not CPU.
Pack file is 845MB instead of 934MB.
git-index-pack works now, it takes 4 CPU minutes to run.
Index file is 18MB.

So it looks like the first stage code is working. Next I need to
modify cvs2svn to keep track of the sha-1 through it's sorting process
instead of file:revision.

-- 
Jon Smirl
jonsmirl@gmail.com