Re: [PATCH/RFC] Allow writing loose objects that are corrupted in a pack file

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Jan 8, 2009, 00:37 UTC
Message-ID: <alpine.LFD.2.00.0901071621340.3283@localhost.localdomain>
In-Reply-To: <alpine.LFD.2.00.0901071520330.3057@localhost.localdomain>

On Wed, 7 Jan 2009, Linus Torvalds wrote:

Show 17 quoted lines

>
> >         > limit ~1.5GB -> corrupt file
> >         > limit ~3GB -> magically no longer corrupt.
> 
> That is interesting, although I also worry that there might be other 
> issues going on (ie since you've reported thigns magically fixing 
> themselves, maybe the ulimit tests just _happened_ to show that, even if 
> it wasn't the core reason).
> 
> BUT! This is definitely worth looking at.
> 
> For example, we do have some cases where we try to do "mmap()", and if it 
> fails, we try to free some memory and try again. In particular, in 
> xmmap(), if an mmap() fails - which may be due to running out of virtual 
> address space - we'll actually try to release some pack-file memory and 
> try again. Maybe there's a bug there - and it would be one that seldom 
> triggers for others.

Ho humm. We really do have some interesting things there.

Is this a 64-bit machine? I didn't think OS X did that, but if there is some limited 64-bit support there, maybe "sizeof(void *)" is 8, then we default the default git pack-window to a pretty healthy 1GB.

I could easily see that if you have a virtual memory size limit of 1.5GB, and the pack window size is 1GB, we might have trouble. Because we could only keep one such pack window in memory at a time.

I have _not_ looked at the code, though. I'd have expected a SIGSEGV if we really had issues with the window handling.

Anyway, _if_ your system has 64-bit pointers, then _maybe_ something the default 1GB pack window causes problem.

If so, then adding a

	[core]
		packedgitwindowsize = 64M

might make a difference. It would certainly be very interesting to hear if there's any impact.

		Linus