Re: [PATCH] coccicheck: process every source file at once

From: Jeff King <peff@peff.net>
Date: Oct 2, 2018, 19:55 UTC
Message-ID: <20181002195519.GB2014@sigill.intra.peff.net>
In-Reply-To: <20181002191642.21504-1-jacob.e.keller@intel.com>

On Tue, Oct 02, 2018 at 12:16:42PM -0700, Jacob Keller wrote:

Show 19 quoted lines

> make coccicheck is used in order to apply coccinelle semantic patches,
> and see if any of the transformations found within contrib/coccinelle/
> can be applied to the current code base.
> 
> Pass every file to a single invocation of spatch, instead of running
> spatch once per source file.
> 
> This reduces the time required to run make coccicheck by a significant
> amount of time:
> 
> Prior timing of make coccicheck
>   real    6m14.090s
>   user    25m2.606s
>   sys     1m22.919s
> 
> New timing of make coccicheck
>   real    1m36.580s
>   user    7m55.933s
>   sys     0m18.219s

Yay! This is a nice result.

It's also one of the things that Julia suggested in an earlier thread. One thing I wasn't quite sure about after digging into the various versions (1.0.4 on Debian stable/unstable, 1.0.6 in experimental, and pre-1.0.7 at the bleeding edge) was whether the old versions would be similarly helped (or work at all).

I just replicated your results with 1.0.4.deb-3+b2 from Debian stable. It's possible there are older versions floating around, but for something developer-only like this, I think "in Debian stable" is a reasonable enough cutoff.

> This is nearly a 4x decrease in the time required to run make
> coccicheck. This is due to the overhead of restarting spatch for every
> file. By processing all files at once, we can amortize this startup cost
> across the total number of files, rather than paying it once per file.

One minor side effect is that we lose the opportunity to parallelize quite as much. However, I think the reduction in total CPU makes it well worth that (and we still have 8 cocci files and growing, which should keep at least 8 cores busy).

I think recent versions of Coccinelle will actually parallelize internally, too, but my 1.0.4 doesn't seem to. That's probably what I was thinking of earlier (but this is still a win even without that).

It looks like this also fixes a problem I ran into when doing the oideq work, which is that the patch for a header file would get shown multiple times (once for each file that includes it). So this is faster _and_ more correct. Double-yay.

Show 14 quoted lines

> diff --git a/Makefile b/Makefile
> index df1df9db78da..b9947f3f51ec 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2715,10 +2715,8 @@ endif
>  %.cocci.patch: %.cocci $(COCCI_SOURCES)
>  	@echo '    ' SPATCH $<; \
>  	ret=0; \
> -	for f in $(COCCI_SOURCES); do \
> -		$(SPATCH) --sp-file $< $$f $(SPATCH_FLAGS) || \
> -			{ ret=$$?; break; }; \
> -	done >$@+ 2>$@.log; \
> +	( $(SPATCH) --sp-file $< $(COCCI_SOURCES) $(SPATCH_FLAGS) || \
> +		{ ret=$$?; }; ) >$@+ 2>$@.log; \

This looks pretty straight-forward. I wondered if we could get rid of the "ret" handling, since we don't need to pass the error back out of the loop. But it's also used for the "show the log only on error" logic below:

>  	if test $$ret != 0; \
>  	then \
>  		cat $@.log; \

The subshell could be a {}, though, I think, but it's not that big a deal either way.

-Peff