Re: GPF in index-pack

From: Shawn Pearce <spearce@spearce.org>
Date: Aug 6, 2006, 04:08 UTC
Message-ID: <20060806040848.GF20565@spearce.org>
In-Reply-To: <9e4733910608052100t463e6fb8gba54e4afde1ab19a@mail.gmail.com>

Jon Smirl <jonsmirl@gmail.com> wrote:

Show 36 quoted lines

> On 8/5/06, Shawn Pearce <spearce@spearce.org> wrote:
> >Jon Smirl <jonsmirl@gmail.com> wrote:
> >> On 8/5/06, Shawn Pearce <spearce@spearce.org> wrote:
> >> >Jon Smirl <jonsmirl@gmail.com> wrote:
> >> >> Process size is 2.6GB when the seg fault happen. That's a lot of
> >> >> memory to build a pack index over 1M objects.
> >> >>
> >> >> I'm running a 3:1 process address space split. I wonder why it didn't
> >> >> grow all the way to 3GB. I still have RAM and swap available.
> >> >
> >> >Was the pack you are trying to index built with that fast-import.c
> >> >I sent last night?  Its possible its doing something weird that
> >> >pack-index can't handle, such as insert a duplicate object into
> >> >the same pack...
> >>
> >> built with fast-import.
> >>
> >> >How big is the pack file?  I'd expect pack-index to be using
> >> >something around 24 MB of memory (24 bytes/entry) but maybe its
> >> >hanging onto a lot of data (memory leak?) as it decompresses the
> >> >entries to compute the checksums.
> >>
> >> It is 934MB in size with 985,000 entries.
> >>
> >> Why does resolve_delta in index-pack.c need to be recursive? Is there
> >> a better way to code that routine? If it mmaps the file that uses 1GB
> >> address space, why does it need another 1.5GB to build an index?
> >
> >Probably the easiest way to code the routine.  Delta depth is
> >bounded; in the fast-import.c that I sent out last night I hardcoded
> >it to 10, which is (I believe) the default for GIT.  So long as that
> >routine is recursive only along a single delta chain the recursion
> >depth won't be very high and shouldn't be the problem.
> 
> When I put index-pack in gdb at the seg fault, resolve_delta had
> recursed more than 20,000 times. I stopped looking after that.

Ouch.  I'm not familiar with this code, but looking at right now its
also not entirely obviously what its recursing for.  Plus dinner
is trying to be burned on the grill, so my attention is on that
more than on GIT.  :-)

Show 7 quoted lines

> >> I had a prior 400MB pack file built with fast-import that I was able
> >> to index ok.
> >
> >Dumb luck?  Maybe that had no duplicates while this one does?
> 
> Is there a git command to list the sha1's in a pack that doesn't have
> an index? I could sort it, sort it unqiue, and then diff the outputs.

Not that I know of. Packs themselves don't have the SHA1 values and getting them from a pack without an index is a painful exercise as you don't know where the base of an object resides within the pack when you need it to generate the object's raw content to determine its ID.

Yea, I'm sure the pack format has some great properties, but storing the base as a 20 byte object ID rather the offset of the base within the pack has its limitations...

-- 
Shawn.