Re: Problem with large files on different OSes

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: May 27, 2009, 16:30 UTC
Message-ID: <alpine.LFD.2.01.0905270922250.3435@localhost.localdomain>
In-Reply-To: <m3y6siboij.fsf@localhost.localdomain>

On Wed, 27 May 2009, Jakub Narebski wrote:

> 
> Do seting `pack.packSizeLimit`, or adjusting values of
> `core.packedGitWindowSize` and/or `core.packedGitLimit`
> (see git-config(1)) help in your situation?

No, that will help just the packfile mmap (and even there, it won't help with things like index file size - we'll always mmap the whole index file). It's definitely worth doing, though - but I think we already default to 32MB pack-file windows on 32-bit architectures.

Individual files we always handle in one go. It's what git was designed for, after all - fairly small files. And so git is limited to files smaller than the virtual address space.

On a 32-bit setup, that often limits you to roughly a gigabyte. You have 4GB of virtual address space, of which one or two is used for the OS kernel. So say you have 2GB for user mode - you then have the executable mapping and libraries and stack, all spread out in that 2GB virtual address space.

In fact, even if it's 3GB for user (I don't know what OS X does), getting one contiguous area may well be limited to ~1GB depending on layout of shared library mappings etc VM fragmentation. Older Linux systems tended to map things in ways that made it hard to get more than 1GB of contiguous data mapping if you compiled with dynamic libraries.

64-bit mostly makes this a non-issue.

In fact, if you do "diff", you're going to be even _more_ limited, since for simplicity, our version of xdiff really wants both sources in memory at a time. So you can't really diff >500MB files. Not that you generally want to, of course.

I'll see if I can make us handle the "big file without diff" case better by chunking.

		Linus