So, I have this PoC patch below on top of this patch series for some months now; actually this series was all fallout from working on this patch. It makes 'make -j<N> coccicheck' much faster (speedup of over 99% :) in some scenarios that, in my experience, occur rather frequently when working on Coccinelle semantic patches (as opposed to occasionally running 'make coccicheck' to see whether there are any regressions).
However, it's not ready for inclusion as it is, because, unfortunately, the improvements are not without some disadvantages, as explained in the second half of the commit message below.
Anyway, I'm sending this out, because I don't see how I could make it any better, but other contributors working on semantic patches might still find it useful to keep this in their own fork, even in its current form. And perhaps someone will come up with a brilliant idea to address those disadvantages...
Also available at:
https://github.com/szeder/git make-coccicheck-finegrained
-- >8 --
Subject: [PATCH] [PoC] coccinelle: make Coccinelle-related make targets more fine-grained
When running 'make coccicheck', each semantic patch is applied to all source files in a separate shell for loop. This approach has the following disadvantages:
1. Even if you only modified a single C source file, that shell loop
will still iterate over all source files and apply the semantic
patches to all of them, wasting a lot of time. 2. If you apply only a single semantic patch (either implicitly,
after you modified only one of them (and the results of the
previous 'make coccicheck' are still there), or explicitly, by
running e.g. 'make contrib/coccinelle/array.cocci.patch'), then
you won't be able to exploit multiple CPU cores to speed up the
operation, because that shell loop will iterate over all source
files sequentially. 3. 'make coccicheck' can only use as many parallel jobs as the
number of semantic patches, so even if you have more available
CPU cores than that, you can't exploit them all to speed up this
operation. 4. During 'make coccicheck' there is only a single
SPATCH contrib/coccinelle/<whatever>.cocci
line (or as many lines as the number of parallel jobs) of output
when starting to apply each semantic patch, and then comes a long
silence without any indication of progress, because applying some
of our semantic patches to all source files can take over a
minute. This can trick developers new to semantic patches into
thinking that something went wrong, hung, ended up in an endless
loop, etc. It certainly confused my the first time.Let's add a bit of Makefile metaprogramming to generate finer-grained make targets applying one semantic patch to only a single source file, and specify these as dependencies of the targets applying one semantic patch to all source files. This way that shell loop can be avoided, semantic patches will only be applied to changed source files, and the same semantic patch can be applied in parallel to multiple source files. The only remaining sequential part is aggregating the suggested transformations from the individual targets into a single patch file, which is comparatively cheap (especially since ideally there aren't any suggestions).
This change brings spectacular speedup in the scenario described in point (1) above. When the results of a previous 'make coccicheck' are there, the time needed to run
touch apply.c ; time make -j4 coccicheck
went from 3m42s to 1.848s, which is just over 99% speedup, yay!, and 'apply.c' is the second longest source file in our codebase. In the scenario in point (2), running
touch contrib/coccinelle/array.cocci ; time make -j4 coccicheck
went from 56.364s to 35.883s, which is ~36% speedup.
All the above timings are best-of-five on a machine with 2 physical and 2 logical cores. I don't have the hardware to bring any numbers for point (3). The time needed to run 'make -j4 coccicheck' in a clean state didn't change, it's ~3m42s both with and without this patch.
Unfortunately, this new approach has some disadvantages compared to the current situation:
- [RFC]
With this patch 'make coccicheck's output will look like this
('make' apparently doesn't iterate over semantic patches in
lexicographical order): SPATCH commit.cocci abspath.c
SPATCH commit.cocci advice.c
<... lots of lines ...>
SPATCH array.cocci http-walker.c
SPATCH array.cocci remote-curl.c which means that the number of lines in the output grows from
"Nr-of-semantic-patches" to "Nr-of-semantic-patches *
Nr-of-source-files". Now, while this certainly addresses point (4) above, it can be
considered too much, and I'm not sure that (4) is that big an
issue to justify this much output. OTOH, I couldn't yet figure out a way to print a single 'SPATCH
contrib/coccinelle/<whatever>.cocci' line at the beginning of
applying that semantic patch without triggering unnecessary work,
effectively killing most of the runtime benefits. Or to somehow
iterate over source files in the outer loop and over semantic
patches in the inner loop, so we could have one output line per
file. - [RFC]
The overhead of applying a semantic patch to all source files
became larger. 'make coccicheck' currently runs only one shell
process and creates two output files for each semantic patch.
With this patch it will run approx. "Nr-of-semantic-patches *
Nr-of-source-files" shell processes and create twice as many
output files. This overhead can be quantified by measuring the runtime of:
make -j4 SPATCH=true coccicheck
and this patch increases it from 0.142s to 1.479s. While this is
indeed an order of magnitude increase, it's still negligible
compared to the "real" runtime of 3m42s. So the increased overhead doesn't seem to matter on Linux, but I
would expect that it's worse on OSX and Windows; though I'm not
sure whether Coccinelle (with all its OCaml dependencies) works on
those platforms in the first place. - [RFC]
This approach uses $(eval), which we haven't used in any of our
Makefiles yet. I wonder whether it's portable enough. And it's
ugly and complicated.Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
Makefile | 52 ++++++++++++++++++++++++-----------
contrib/coccinelle/.gitignore | 3 +-
2 files changed, 38 insertions(+), 17 deletions(-)
SZEDER Gábor
