Re: [PATCH 06/18] chainlint.pl: validate test scripts in parallel

> I did some timings the other night, and I found something quite curious
> with the thread stuff.
>
> I was quite surprised that it made things slower! It's nice that we're
> only calling it once per script instead of once per test, but it seems
> the startup overhead of the script is really high.
>
> And since in this mode we're only feeding it one script at a time, I
> tried reverting the "chainlint.pl: validate test scripts in parallel"
> commit. And indeed, now things are much faster:
>
>   Benchmark 1: make
>     Time (mean ± σ):     61.544 s ±  3.364 s    [User: 556.486 s, System: 384.001 s]
>     Range (min … max):   57.660 s … 63.490 s    3 runs
>
> And you can see the same thing just running chainlint by itself:
>
>   $ time perl chainlint.pl /dev/null
>   real  0m0.069s
>   user  0m0.042s
>   sys   0m0.020s
>
>   $ git revert HEAD^{/validate.test.scripts.in.parallel}
>   $ time perl chainlint.pl /dev/null
>   real  0m0.014s
>   user  0m0.010s
>   sys   0m0.004s
>
> I didn't track down the source of the slowness. Maybe it's loading extra
> modules, or maybe it's opening /proc/cpuinfo, or maybe it's the thread
> setup. But it's a surprising slowdown.

> Now of course your intent is to do a single repo-wide invocation. And
> that is indeed a bit faster. Here it is without the parallel code:
>
>   Benchmark 1: make
>     Time (mean ± σ):     61.727 s ±  2.140 s    [User: 507.712 s, System: 377.753 s]
>     Range (min … max):   59.259 s … 63.074 s    3 runs
>
> The wall-clock time didn't improve much, but the CPU time did. Restoring
> the parallel code does improve the wall-clock time a bit, but at the
> cost of some extra CPU:
>
>   Benchmark 1: make
>     Time (mean ± σ):     59.029 s ±  2.851 s    [User: 515.690 s, System: 380.369 s]
>     Range (min … max):   55.736 s … 60.693 s    3 runs
>
> which makes sense. If I do a with/without of just "make test-chainlint",
> the parallelism is buying a few seconds of wall-clock:
>
>   Benchmark 1: make test-chainlint
>     Time (mean ± σ):     900.1 ms ± 102.9 ms    [User: 12049.8 ms, System: 79.7 ms]
>     Range (min … max):   704.2 ms … 994.4 ms    10 runs
>
>   Benchmark 1: make test-chainlint
>     Time (mean ± σ):      3.778 s ±  0.042 s    [User: 3.756 s, System: 0.023 s]
>     Range (min … max):    3.706 s …  3.833 s    10 runs
>
> I'm not sure what it all means. For Linux, I think I'd be just as happy
> with a single non-parallelized test-chainlint run for each file. But
> maybe on Windows the startup overhead is worse? OTOH, the whole test run
> is so much worse there. One process per script is not going to be that
> much in relative terms either way.

> And if we did cache the results and avoid extra invocations via "make",
> then we'd want all the parallelism to move to there anyway.
>
> Maybe that gives you more food for thought about whether perl's "use
> threads" is worth having.