Re: git status OOM on mmap of large file

From: Jeff King <peff@peff.net>
Date: Jan 24, 2019, 19:28 UTC
Message-ID: <20190124192831.GA14201@sigill.intra.peff.net>
In-Reply-To: <20190124191836.GA31073@sigill.intra.peff.net>

On Thu, Jan 24, 2019 at 02:18:36PM -0500, Jeff King wrote:

Show 10 quoted lines

> > I did some benchmarking, using cat as the clean filter:
> > [...]
> > From this, it looks like the file has to be quite large before the
> > preallocation makes a sizable improvement to runtime, and the
> > smudge/clean filters have to be used for actual content filtering
> > (not for hash generation purposes as git-annex and git-lfs use it).
> > An unusual edge case I think. So hint == 0 seems fine.
> 
> Thanks for these timings! I agree that "hint == 0" is probably
> reasonable, then.

One other minor point to consider: on some systems over-allocating actually isn't that expensive, because pages are actually allocated until we write to them, and malloc() is perfectly happy to overcommit memory. Your case would only run into problems on Linux when malloc() actually refuses the allocation (so limiting ourselves to "too large but still reasonable" is a valid strategy there).

But I doubt that's something we should be relying on in general. There are many systems that don't overcommit.

-Peff