Re: [PATCH V2 3/3] strbuf: allow to use preallocated memory

From: Jeff King <peff@peff.net>
Date: Jun 8, 2016, 19:52 UTC
Message-ID: <20160608195248.GA4264@sigill.intra.peff.net>
In-Reply-To: <xmqq37onlawb.fsf@gitster.mtv.corp.google.com>

On Wed, Jun 08, 2016 at 12:48:36PM -0700, Junio C Hamano wrote:

Show 17 quoted lines

> Jeff King <peff@peff.net> writes:
> 
> > That made me wonder if we could repeatedly reuse a buffer attached to
> > the file descriptor. And indeed, isn't that what stdio is? The whole
> > reason this buffer exists is because we are using a direct descriptor
> > write. If we switched this function to use fprintf(), we'd avoid the
> > whole buffer question, have a fixed cap on our memory use (since we just
> > flush anytime the buffer is full) _and_ we'd reduce the number of
> > write syscalls we're making by almost a factor of 100.
> 
> The primary reason why we avoid stdio in the lower level part of I/O
> is that the error reporting and handling is horrible.
> 
> e.g. c.f. http://article.gmane.org/gmane.comp.version-control.git/27019
> 
> Otherwise, I'd agree with your "Why aren't we using stdio if
> counting and avoiding overflow is so hard?".

I agree it can be confusing (especially on the output side your errors are likely deferred until the next flush). But in this particular case, I think it's an improvement (see the patch I just sent and its discussion of error handling).

I also think we could smooth over the rough edges by wrapping the complexity (we already have fprintf_or_die, which arguably could be used in this case, but I went with the solution that stayed closer to what the original code was doing). And if stdio is truly too horrible, I'd suggest we implement our own buffered I/O. Because that's effectively what such call-sites are doing, but in a really ad-hoc and non-performant way.

-Peff