Re: git status OOM on mmap of large file

From: Jeff King <peff@peff.net>
Date: Jan 24, 2019, 19:18 UTC
Message-ID: <20190124191836.GA31073@sigill.intra.peff.net>
In-Reply-To: <20190124183810.GC29200@kitenet.net>

On Thu, Jan 24, 2019 at 02:38:10PM -0400, Joey Hess wrote:

Show 22 quoted lines

> > Just off the top of my head, something like:
> > 
> >   /* guess that the filtered output will be the same size as the original */
> >   hint = len;
> > 
> >   /* allocate 10% extra in case the clean size is slightly larger */
> >   hint *= 1.1;
> > 
> >   /*
> >    * in any case, never go higher than half of core.bigfileThreshold.
> >    * We'd like to avoid allocating more bytes than that, and that still
> >    * gives us room for our strbuf to preemptively double if our guess is
> >    * just a little on the low side.
> >    */
> >   if (hint > big_file_threshold / 2)
> > 	hint = big_file_threshold / 2;
> > 
> > But to be honest, I have no idea if that would even produce measurable
> > benefits over simply growing the strbuf from scratch (i.e., hint==0).
> 
> Half of 512 MB is still quite a lot of memory to default to using in
> this situation. Eg smaller VPS's still often only have a GB or two of ram.

I think you'd want to drop core.bigFileThreshold on such a server, just because Git will happily keep 2*(bigFileThreshold-1) in memory to do a diff. But that nit aside...

Show 7 quoted lines

> I did some benchmarking, using cat as the clean filter:
> [...]
> From this, it looks like the file has to be quite large before the
> preallocation makes a sizable improvement to runtime, and the
> smudge/clean filters have to be used for actual content filtering
> (not for hash generation purposes as git-annex and git-lfs use it).
> An unusual edge case I think. So hint == 0 seems fine.

Thanks for these timings! I agree that "hint == 0" is probably reasonable, then.

I suppose there's no reason not to proceed with a patch around this. For most cases it's really only half the solution (since smudging is going to run into the same problem). But fixing that is quite a bit more involved, and the change itself will be largely orthogonal.

-Peff