Re: [PATCH 2/2] archive: avoid spawning `gzip`

From: Jeff King <peff@peff.net>
Date: May 1, 2019, 18:18 UTC
Message-ID: <20190501181807.GC4109@sigill.intra.peff.net>
In-Reply-To: <05e97774-5dd1-7224-940f-e50558118d93@web.de>

On Wed, May 01, 2019 at 07:45:05PM +0200, René Scharfe wrote:

Show 25 quoted lines

> > But since the performance is still not quite on par with `gzip`, I would
> > actually rather not, and really, just punt on that one, stating that
> > people interested in higher performance should use `pigz`.
> 
> Here are my performance numbers for generating .tar.gz files again:
> 
> master, using gzip(1):
>   Time (mean ± σ):     16.683 s ±  0.451 s    [User: 20.230 s, System: 0.375 s]
>   Range (min … max):   16.308 s … 17.852 s    10 runs
> 
> using zlib sequentially:
>   Time (mean ± σ):     19.898 s ±  0.228 s    [User: 19.825 s, System: 0.073 s]
>   Range (min … max):   19.627 s … 20.355 s    10 runs
> 
> using zlib asynchronously:
>   Time (mean ± σ):     17.300 s ±  0.198 s    [User: 20.825 s, System: 0.356 s]
>   Range (min … max):   17.042 s … 17.638 s    10 runs
> 
> using a gzip-lookalike:
>   Time (mean ± σ):     17.256 s ±  0.299 s    [User: 20.380 s, System: 0.294 s]
>   Range (min … max):   16.940 s … 17.804 s    10 runs
> 
> The last two have comparable system time, ca. 1% more user time and
> ca. 5% longer duration.  The second one has much better system time
> and 2% less user time and 19% longer duration.  Hmm.

I think the start_async() one seems like a good option. It reclaims most of the (wall-clock) performance, isn't very much code, and doesn't leave any ugly user-visible traces.

I'd be fine to see it come later, though, on top of the patches Dscho is sending. Even though changing to sequential zlib is technically a change in behavior, the existing behavior wasn't really planned. And given the wall-clock versus CPU time tradeoff, it's not entirely clear that one solution is better than the other.

> > And who knows, maybe nobody will complain at all about the performance?
> 
> Probably.  And popular tarballs would be cached anyway, I guess.

At GitHub we certainly do cache the git-archive output. We'd also be just fine with the sequential solution. We generally turn down pack.threads to 1, and keep our CPUs busy by serving multiple users anyway.

So whatever has the lowest overall CPU time is generally preferable, but the times are close enough that I don't think we'd care much either way (and it's probably not worth having a config option or similar).

-Peff