Re: fact-import: failed to apply delta

From: Shawn O. Pearce <spearce@spearce.org>
Date: Feb 10, 2009, 15:56 UTC
Message-ID: <20090210155626.GM30949@spearce.org>
In-Reply-To: <alpine.DEB.1.00.0902101126040.10279@pacific.mpi-cbg.de>

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 16 quoted lines

> On Mon, 9 Feb 2009, Daniel Barkalow wrote:
> 
> > I'm getting a "fatal: failed to apply delta" from fast-import. I'm using 
> > a lot of checkpoints, and I haven't had it happen without making 
> > progress, so I was eventually able to import what I was importing (bunch 
> > of stuff I can't distribute, imported from perforce with the latest 
> > version of my p4 importer that I'm still working on). Also, everything 
> > that was saved by the checkpoints was valid and correct (at least after 
> > the fact).
> > 
> > I'm going to see if it's reproducable, and, if so, if I can get a test 
> > case that I can distribute, but I wanted to post to see if anyone had 
> > any special debugging advice for this error message and program 
> > combination.
> 
> I see three likely candidates: two in index-pack.c and one in sha1_file.c.

It has to be the one in sha1_file.c. fast-import never calls into the code in index-pack.c.

> My advice: instrument the code (IOW litter the code with debug output that 
> tells you where it did what), and then run it on the same test case you 
> had the problems.

My initial guess is, we're probably having trouble reading from the pack we are writing. This shouldn't be possible; fast-import uses an index of all objects in memory to locate the byte offset and then reads from that block.

I wonder if this is a write(2) vs. mmap(2) inconsistency in the VM system of your OS. fast-import plays some games here where it is actively writing into areas that might have been mmap'd. We should be closing the existing mmap's and opening them again when we switch from writing to reading (see gfi_unpack_entry), but if that's failing for some reason then we might be trying to read a VM page which isn't correctly initialized, and the delta isn't valid, and we can't inflate it.

-- 
Shawn.