Re: fast-import and unique objects.

From: Shawn Pearce <spearce@spearce.org>
Date: Aug 7, 2006, 23:02 UTC
Message-ID: <20060807230258.GA19067@spearce.org>
In-Reply-To: <20060807075744.GB15477@h4x0r5.com>

Ryan Anderson <ryan@michonline.com> wrote:

Show 15 quoted lines

> On Sun, Aug 06, 2006 at 02:03:24PM -0400, Shawn Pearce wrote:
> > 
> >   - It expects an estimated object count as its second parameter.
> >     In your case this would be something around 760000.  This tells
> >     it how large of an object table to allocate, with each entry
> >     being 24 bytes + 1 pointer (28 or 32 bytes).  Overshooting
> > 	this number will cause it to degrade by allocating one
> > 	overflow entry at a time from malloc.
> 
> Hrm, you're allocating a big table and then assigning consecutive
> entries out of it, as pointers.
> 
> Why not just malloc a big block, and assign offsets into it, as if it
> were a really big array.  Every time it runs out, realloc it to double
> the current size, and update the base pointer.

Because I didn't want to move a 24 MB block of memory.  :-)

I'm probably going to clean that section of code up tonight and allocate a large block at the beginning then allocate overflow blocks at about 5000 entries at a time. There's no need for the blocks to be contiguous in memory, I just didn't want to have a high overhead from malloc when there would be a large number of them...

-- 
Shawn.