Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Aug 3, 2018, 06:44 UTC
Message-ID: <20180803064419.GC237521@aiede.svl.corp.google.com>
In-Reply-To: <alpine.DEB.2.20.1808030755350.2446@hadrien>

Hi,

Julia Lawall wrote:

Show 5 quoted lines

> This was already possible.  Make coccicheck is not supposed to be used
> with -j, but rather with J=n.  That tells Coccinelle to parallelize the
> treatment of the files internally.  In this case, the semantic patch is
> only parsed once, and then n worker processes are forked to treat the
> different files.

Thanks for this hint.

I wonder if we can make this happen automatically under suitable conditions. We can parse -j<num> out of MAKEFLAGS and convert it into a J=<num> argument, but that only solves half the problem: the "make" parent process would think that the coccinelle run only deserves to occupy one job slot.

Technically we could do all this using a wrapper that pretends to be a submake <https://www.gnu.org/software/make/manual/html_node/Job-Slots.html> (prefixing the command with '+', parsing jobserver options from the MAKEFLAGS envvar) but it gets ugly.

It's likely that the best we can do is just to advertise J more prominently.

[...]

>> On Thu, Aug 02 2018, Jeff King wrote:

Show 9 quoted lines

>>>   cat contrib/coccinelle/*.cocci >mega.cocci
>>>   make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci
>
> There was already a COCCI=foo.cocci argument to focus on a single semantic
> patch.
>
> I'm surprised that the above cat command would work.  Semantic patch rules
> have names, and Coccinelle will not be happy isf two rules have the same
> name.

Yes, Git's semantic patches (in contrib/coccinelle) tend to be relatively undemanding.

>       Some may also have variables declared in initializers, although
> perhaps the ones in the kernel don't do this.  Causing these variables to
> be shared would not have a good effect.

... oh!  You're thinking of the Linux kernel.

It looks like Linux's scripts/coccicheck has a lot we can crib from. That's where the J envvar and automatic parallelism you mentioned are implemented, too.

So it sounds to me like at a minimum we should use all of that. ;-)

Thanks again for the pointers.

Sincerely, Jonathan