Re: [RFC PATCH] http: use xmalloc with cURL

From: Jeff King <peff@peff.net>
Date: Aug 12, 2019, 19:55 UTC
Message-ID: <20190812195537.GA14223@sigill.intra.peff.net>
In-Reply-To: <CAPUEspg62pRNH6=_VvWDxQ4YujHUJAoTTampc0L4t68QMj30xg@mail.gmail.com>

On Sun, Aug 11, 2019 at 12:08:53PM -0700, Carlo Arenas wrote:

Show 6 quoted lines

> there is also the risk that xmalloc might not be sufficiently thread
> safe (ex: when it triggers unuse_one_window() through the use of a
> try_to_free_routine in packfile.c but we could mitigate the risk for
> now by doing it only first #ifdef NED (I promise to take it out before
> it even gets to next, assuming Junio would trust me enough after the
> last blow up to even get it into pu)

Yes, our xmalloc() is very much not thread-safe, and I suspect multithreaded bits that don't override try_to_free_routine, like git-grep, are probably buggy right now (assuming they xmalloc() outside of the big global lock). This has been the case for ages (see [1] from 2011, for example) and I don't think anybody has ever complained. Which leads me to believe that try_to_free_routine() rarely actually kicks in.

I've long suspected that the whole try_to_free_pack_memory thing is more trouble than it's worth these days. We're dealing with mmap'd files, so the OS is going to do a much better job at responding to memory pressure by dropping individual pages.

So the only thing we're really freeing is address space. On 64-bit systems, we have plenty of that to go around. On 32-bit systems, it might. But we introduced it[2] at the same time as sliding windows[3], so it's not clear how much it helps (keep in mind that even without this we'd still free old pack windows to make room for more mmap'd packs).

And as a cost, xmalloc() isn't thread safe, and is a lot harder to reason about (e.g., something simple like strbuf_grow() might trigger quite a bit more work than you'd expect).

I think it might be worth just eliminating the whole idea.

[1] https://public-inbox.org/git/20110831015936.GB2519@sigill.intra.peff.net/ [2] 97bfeb34df (Release pack windows before reporting out of memory., 2006-12-24) [3] 60bb8b1453 (Fully activate the sliding window pack access., 2006-12-23)

-Peff