Re: git status OOM on mmap of large file

From: Jeff King <peff@peff.net>
Date: Jan 24, 2019, 12:14 UTC
Message-ID: <20190124121427.GB4949@sigill.intra.peff.net>
In-Reply-To: <20190124003948.GS423984@genre.crustytoothpaste.net>

On Thu, Jan 24, 2019 at 12:39:49AM +0000, brian m. carlson wrote:

Show 9 quoted lines

> > [pid  6573] mmap(NULL, 11811164160, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
> > 
> > Why status needs to mmap a large file that is not modified
> > and that is configured to pass through smudge/clean, I don't know.
> 
> I believe that currently, Git stores the smudge/clean output in memory
> until it writes it out. When using the persistent filter process, it's
> possible for the process to choose to abort the operation, so we store
> the data in memory until we get the status.

For the clean step, that should be OK, since that filter output is tiny (but see my other message for a silly heuristic and an easy fix). And that should be all that "git status" needs. But...

> Theoretically, it should be possible for us to write this to a temporary
> file, and if necessary, rename into place, although I'm not sure how
> well that will work on Windows. File modes may also be tricky here.
> Patches are of course welcome.

I didn't experiment with the smudge side, but I think it uses the same apply_filter() code. Which means that yes, it would try to store the 11GB in memory before writing it out. And I agree writing it out to a file and moving it directly into place is the sanest option there. If that doesn't work, spooling to a tempfile and then streaming it into place would also work.

-Peff