# [PATCH 0/5] Misc Coccinelle-related improvements 31 messages from 2018-07-23 to 2018-08-05. Participants: SZEDER Gábor, Derrick Stolee, Duy Nguyen, René Scharfe, Eric Sunshine, Junio C Hamano, Jeff King, Ævar Arnfjörð Bjarmason, Jonathan Nieder, Julia Lawall. Thread: https://gitlist.dev/t/48941 ## SZEDER Gábor, 2018-07-23 13:50 Subject: [PATCH 0/5] Misc Coccinelle-related improvements Message-ID: <20180723135100.24288-1-szeder.dev@gmail.com> URL: https://gitlist.dev/e/20180723135100.24288-1-szeder.dev%40gmail.com ``` Just a couple of minor Coccinelle-related improvements: - The first two patches are small cleanups. - The last three could make life perhaps just a tad bit easier for devs running 'make coccicheck'. SZEDER Gábor (5): coccinelle: mark the 'coccicheck' make target as .PHONY coccinelle: use $(addsuffix) in 'coccicheck' make target coccinelle: exclude sha1dc source files from static analysis coccinelle: put sane filenames into output patches coccinelle: extract dedicated make target to clean Coccinelle's results Makefile | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) -- 2.18.0.408.g42635c01bc ``` ## SZEDER Gábor, 2018-07-23 13:50 Subject: [PATCH 1/5] coccinelle: mark the 'coccicheck' make target as .PHONY Message-ID: <20180723135100.24288-2-szeder.dev@gmail.com> URL: https://gitlist.dev/e/20180723135100.24288-2-szeder.dev%40gmail.com In-Reply-To: <20180723135100.24288-1-szeder.dev@gmail.com> ``` The 'coccicheck' target doesn't create a file with the same name, so mark it as .PHONY. Signed-off-by: SZEDER Gábor --- Makefile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Makefile b/Makefile index e4b503d259..27bfc196dd 100644 --- a/Makefile +++ b/Makefile @@ -2685,6 +2685,8 @@ C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ)) fi coccicheck: $(patsubst %.cocci,%.cocci.patch,$(wildcard contrib/coccinelle/*.cocci)) +.PHONY: coccicheck + ### Installation rules ifneq ($(filter /%,$(firstword $(template_dir))),) -- 2.18.0.408.g42635c01bc ``` ## SZEDER Gábor, 2018-07-23 13:50 Subject: [PATCH 2/5] coccinelle: use $(addsuffix) in 'coccicheck' make target Message-ID: <20180723135100.24288-3-szeder.dev@gmail.com> URL: https://gitlist.dev/e/20180723135100.24288-3-szeder.dev%40gmail.com In-Reply-To: <20180723135100.24288-1-szeder.dev@gmail.com> ``` The dependencies of the 'coccicheck' make target are listed with the help of the $(patsubst) make function, which in this case doesn't do any pattern substitution, but only adds the '.patch' suffix. Use the shorter and more idiomatic $(addsuffix) make function instead. Signed-off-by: SZEDER Gábor --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 27bfc196dd..8f509576e9 100644 --- a/Makefile +++ b/Makefile @@ -2683,7 +2683,7 @@ C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ)) then \ echo ' ' SPATCH result: $@; \ fi -coccicheck: $(patsubst %.cocci,%.cocci.patch,$(wildcard contrib/coccinelle/*.cocci)) +coccicheck: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.cocci)) .PHONY: coccicheck -- 2.18.0.408.g42635c01bc ``` ## SZEDER Gábor, 2018-07-23 13:50 Subject: [PATCH 3/5] coccinelle: exclude sha1dc source files from static analysis Message-ID: <20180723135100.24288-4-szeder.dev@gmail.com> URL: https://gitlist.dev/e/20180723135100.24288-4-szeder.dev%40gmail.com In-Reply-To: <20180723135100.24288-1-szeder.dev@gmail.com> ``` sha1dc is an external library, that we carry in-tree for convenience or grab as a submodule, so there is no use in applying our semantic patches to its source files. Therefore, exclude sha1dc's source files from Coccinelle's static analysis. This change also makes the static analysis somewhat faster: presumably because of the heavy use of repetitive macro declarations, applying the semantic patches 'array.cocci' and 'swap.cocci' to 'sha1dc/sha1.c' takes over half a minute each on my machine, which amounts to about a third of the runtime of applying these two semantic patches to the whole git source tree. Signed-off-by: SZEDER Gábor --- Makefile | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 8f509576e9..73e2d16926 100644 --- a/Makefile +++ b/Makefile @@ -2666,10 +2666,16 @@ check: command-list.h fi C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ)) -%.cocci.patch: %.cocci $(C_SOURCES) +ifdef DC_SHA1_SUBMODULE +COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) +else +COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) +endif + +%.cocci.patch: %.cocci $(COCCI_SOURCES) @echo ' ' SPATCH $<; \ ret=0; \ - for f in $(C_SOURCES); do \ + for f in $(COCCI_SOURCES); do \ $(SPATCH) --sp-file $< $$f $(SPATCH_FLAGS) || \ { ret=$$?; break; }; \ done >$@+ 2>$@.log; \ -- 2.18.0.408.g42635c01bc ``` ## SZEDER Gábor, 2018-07-23 13:50 Subject: [PATCH 4/5] coccinelle: put sane filenames into output patches Message-ID: <20180723135100.24288-5-szeder.dev@gmail.com> URL: https://gitlist.dev/e/20180723135100.24288-5-szeder.dev%40gmail.com In-Reply-To: <20180723135100.24288-1-szeder.dev@gmail.com> ``` Coccinelle outputs its suggested transformations as patches, whose header looks something like this: --- commit.c +++ /tmp/cocci-output-19250-7ae78a-commit.c Note the lack of 'diff --opts ' line, the differing number of path components on the --- and +++ lines, and the nonsensical filename on the +++ line. 'patch -p0' can still apply these patches, as it takes the filename to be modified from the --- line. Alas, 'git apply' can't, because it takes the filename from the +++ line, and then complains about the nonexisting file. Pass the '--patch .' options to Coccinelle via the SPATCH_FLAGS 'make' variable, as it seems to make it generate proper context diff patches, with the header starting with a 'diff ...' line and containing sane filenames. The resulting 'contrib/coccinelle/*.cocci.patch' files then can be applied both with 'git apply' and 'patch' (even without '-p0'). Signed-off-by: SZEDER Gábor --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 73e2d16926..72ea29df4e 100644 --- a/Makefile +++ b/Makefile @@ -564,7 +564,7 @@ SPATCH = spatch export TCL_PATH TCLTK_PATH SPARSE_FLAGS = -SPATCH_FLAGS = --all-includes +SPATCH_FLAGS = --all-includes --patch . -- 2.18.0.408.g42635c01bc ``` ## SZEDER Gábor, 2018-07-23 13:51 Subject: [PATCH 5/5] coccinelle: extract dedicated make target to clean Coccinelle's results Message-ID: <20180723135100.24288-6-szeder.dev@gmail.com> URL: https://gitlist.dev/e/20180723135100.24288-6-szeder.dev%40gmail.com In-Reply-To: <20180723135100.24288-1-szeder.dev@gmail.com> ``` Sometimes I want to remove only Coccinelle's results, but keep all other build artifacts left after my usual 'make all man' build. This new 'cocciclean' make target will allow just that. Signed-off-by: SZEDER Gábor --- Makefile | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index 72ea29df4e..aee629cbaf 100644 --- a/Makefile +++ b/Makefile @@ -2903,7 +2903,10 @@ profile-clean: $(RM) $(addsuffix *.gcda,$(addprefix $(PROFILE_DIR)/, $(object_dirs))) $(RM) $(addsuffix *.gcno,$(addprefix $(PROFILE_DIR)/, $(object_dirs))) -clean: profile-clean coverage-clean +cocciclean: + $(RM) contrib/coccinelle/*.cocci.patch* + +clean: profile-clean coverage-clean cocciclean $(RM) *.res $(RM) $(OBJECTS) $(RM) $(LIB_FILE) $(XDIFF_LIB) $(VCSSVN_LIB) @@ -2915,7 +2918,6 @@ clean: profile-clean coverage-clean $(RM) -r $(GIT_TARNAME) .doc-tmp-dir $(RM) $(GIT_TARNAME).tar.gz git-core_$(GIT_VERSION)-*.tar.gz $(RM) $(htmldocs).tar.gz $(manpages).tar.gz - $(RM) contrib/coccinelle/*.cocci.patch* $(MAKE) -C Documentation/ clean ifndef NO_PERL $(MAKE) -C gitweb clean @@ -2931,7 +2933,7 @@ endif $(RM) GIT-USER-AGENT GIT-PREFIX $(RM) GIT-SCRIPT-DEFINES GIT-PERL-DEFINES GIT-PERL-HEADER GIT-PYTHON-VARS -.PHONY: all install profile-clean clean strip +.PHONY: all install profile-clean cocciclean clean strip .PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell .PHONY: FORCE cscope -- 2.18.0.408.g42635c01bc ``` ## Derrick Stolee, 2018-07-23 14:34 Subject: Re: [PATCH 4/5] coccinelle: put sane filenames into output patches Message-ID: <9f560db6-b50a-e2f3-7215-d046c60e255a@gmail.com> URL: https://gitlist.dev/e/9f560db6-b50a-e2f3-7215-d046c60e255a%40gmail.com In-Reply-To: <20180723135100.24288-5-szeder.dev@gmail.com> ``` On 7/23/2018 9:50 AM, SZEDER Gábor wrote: > Coccinelle outputs its suggested transformations as patches, whose > header looks something like this: > > --- commit.c > +++ /tmp/cocci-output-19250-7ae78a-commit.c > > Note the lack of 'diff --opts ' line, the differing number > of path components on the --- and +++ lines, and the nonsensical > filename on the +++ line. 'patch -p0' can still apply these patches, > as it takes the filename to be modified from the --- line. Alas, 'git > apply' can't, because it takes the filename from the +++ line, and > then complains about the nonexisting file. > > Pass the '--patch .' options to Coccinelle via the SPATCH_FLAGS 'make' > variable, as it seems to make it generate proper context diff patches, > with the header starting with a 'diff ...' line and containing sane > filenames. The resulting 'contrib/coccinelle/*.cocci.patch' files > then can be applied both with 'git apply' and 'patch' (even without > '-p0'). > > Signed-off-by: SZEDER Gábor > --- > Makefile | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/Makefile b/Makefile > index 73e2d16926..72ea29df4e 100644 > --- a/Makefile > +++ b/Makefile > @@ -564,7 +564,7 @@ SPATCH = spatch > export TCL_PATH TCLTK_PATH > > SPARSE_FLAGS = > -SPATCH_FLAGS = --all-includes > +SPATCH_FLAGS = --all-includes --patch . Thank you for this! These garbage filenames cause significant pain when trying to apply the suggested patch. ``` ## Derrick Stolee, 2018-07-23 14:36 Subject: Re: [PATCH 1/5] coccinelle: mark the 'coccicheck' make target as .PHONY Message-ID: <0023a8c0-9239-d8d7-4187-922f9305449b@gmail.com> URL: https://gitlist.dev/e/0023a8c0-9239-d8d7-4187-922f9305449b%40gmail.com In-Reply-To: <20180723135100.24288-2-szeder.dev@gmail.com> ``` On 7/23/2018 9:50 AM, SZEDER Gábor wrote: > The 'coccicheck' target doesn't create a file with the same name, so > mark it as .PHONY. > > Signed-off-by: SZEDER Gábor > --- > Makefile | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/Makefile b/Makefile > index e4b503d259..27bfc196dd 100644 > --- a/Makefile > +++ b/Makefile > @@ -2685,6 +2685,8 @@ C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ)) > fi > coccicheck: $(patsubst %.cocci,%.cocci.patch,$(wildcard contrib/coccinelle/*.cocci)) > > +.PHONY: coccicheck > + > ### Installation rules > > ifneq ($(filter /%,$(firstword $(template_dir))),) I did not know about phony targets [1] so thanks for teaching me something today. [1] https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html ``` ## Derrick Stolee, 2018-07-23 14:38 Subject: Re: [PATCH 0/5] Misc Coccinelle-related improvements Message-ID: URL: https://gitlist.dev/e/fbf95215-a174-6917-256c-3ed71f960a6a%40gmail.com In-Reply-To: <20180723135100.24288-1-szeder.dev@gmail.com> ``` On 7/23/2018 9:50 AM, SZEDER Gábor wrote: > Just a couple of minor Coccinelle-related improvements: > > - The first two patches are small cleanups. > > - The last three could make life perhaps just a tad bit easier for > devs running 'make coccicheck'. I appreciate your focus on making 'make coccicheck' easier to use! Reviewed-by: Derrick Stolee ``` ## Duy Nguyen, 2018-07-23 15:29 Subject: Re: [PATCH 0/5] Misc Coccinelle-related improvements Message-ID: URL: https://gitlist.dev/e/CACsJy8Av5vBY_fGGyHXdqezZJpdE-_-roro96%2BWAc2xiSbOYPQ%40mail.gmail.com In-Reply-To: <20180723135100.24288-1-szeder.dev@gmail.com> ``` On Mon, Jul 23, 2018 at 3:51 PM SZEDER Gábor wrote: > > Just a couple of minor Coccinelle-related improvements: > > - The first two patches are small cleanups. > > - The last three could make life perhaps just a tad bit easier for > devs running 'make coccicheck'. I'm not a heavy cocci user and probably won't spot any corner case problems. With that in mind, I've read this though and the series looks good to me. -- Duy ``` ## René Scharfe, 2018-07-23 16:30 Subject: Re: [PATCH 0/5] Misc Coccinelle-related improvements Message-ID: URL: https://gitlist.dev/e/b405bbd5-9af5-6884-6582-9aa7f7db0878%40web.de In-Reply-To: <20180723135100.24288-1-szeder.dev@gmail.com> ``` Am 23.07.2018 um 15:50 schrieb SZEDER Gábor: > Just a couple of minor Coccinelle-related improvements: > > - The first two patches are small cleanups. > > - The last three could make life perhaps just a tad bit easier for > devs running 'make coccicheck'. > > > SZEDER Gábor (5): > coccinelle: mark the 'coccicheck' make target as .PHONY > coccinelle: use $(addsuffix) in 'coccicheck' make target > coccinelle: exclude sha1dc source files from static analysis > coccinelle: put sane filenames into output patches > coccinelle: extract dedicated make target to clean Coccinelle's > results These patches look good to me. Thanks a lot! René ``` ## Eric Sunshine, 2018-07-23 18:27 Subject: Re: [PATCH 3/5] coccinelle: exclude sha1dc source files from static analysis Message-ID: URL: https://gitlist.dev/e/CAPig%2BcSOZd%2Bt17j7FSCYAydS34ZtfcRFZyE6E9fz%3Du7xB-01Mg%40mail.gmail.com In-Reply-To: <20180723135100.24288-4-szeder.dev@gmail.com> ``` On Mon, Jul 23, 2018 at 9:51 AM SZEDER Gábor wrote: > sha1dc is an external library, that we carry in-tree for convenience > or grab as a submodule, so there is no use in applying our semantic > patches to its source files. > > Therefore, exclude sha1dc's source files from Coccinelle's static > analysis. > > Signed-off-by: SZEDER Gábor > --- > diff --git a/Makefile b/Makefile > @@ -2666,10 +2666,16 @@ check: command-list.h > +ifdef DC_SHA1_SUBMODULE > +COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) > +else > +COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) > +endif Can't you just filter out both of these unconditionally without worrying about DC_SHA1_SUBMODULE? ``` ## SZEDER Gábor, 2018-07-23 18:43 Subject: Re: [PATCH 3/5] coccinelle: exclude sha1dc source files from static analysis Message-ID: URL: https://gitlist.dev/e/CAM0VKjkJoqRFmXdnuujSaiZ%3Dhvz6MeAmgoUQNAkZ%2B82ZrKtotw%40mail.gmail.com In-Reply-To: ``` On Mon, Jul 23, 2018 at 8:28 PM Eric Sunshine wrote: > > On Mon, Jul 23, 2018 at 9:51 AM SZEDER Gábor wrote: > > sha1dc is an external library, that we carry in-tree for convenience > > or grab as a submodule, so there is no use in applying our semantic > > patches to its source files. > > > > Therefore, exclude sha1dc's source files from Coccinelle's static > > analysis. > > > > Signed-off-by: SZEDER Gábor > > --- > > diff --git a/Makefile b/Makefile > > @@ -2666,10 +2666,16 @@ check: command-list.h > > +ifdef DC_SHA1_SUBMODULE > > +COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) > > +else > > +COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) > > +endif > > Can't you just filter out both of these unconditionally without > worrying about DC_SHA1_SUBMODULE? I'm not sure what you mean by that. Like this perhaps? COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(filter-out sha1dc/%,$(C_SOURCES))) While it's only a single line, I don't think it's any easier on the eyes. ``` ## Eric Sunshine, 2018-07-23 18:57 Subject: Re: [PATCH 3/5] coccinelle: exclude sha1dc source files from static analysis Message-ID: URL: https://gitlist.dev/e/CAPig%2BcQM-hdXXe6uZ768HbJ5T5QqJNKo9kuz6F%3Di7D%2B9Q-M90Q%40mail.gmail.com In-Reply-To: ``` On Mon, Jul 23, 2018 at 2:44 PM SZEDER Gábor wrote: > On Mon, Jul 23, 2018 at 8:28 PM Eric Sunshine wrote: > > On Mon, Jul 23, 2018 at 9:51 AM SZEDER Gábor wrote: > > > +ifdef DC_SHA1_SUBMODULE > > > +COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) > > > +else > > > +COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) > > > +endif > > > > Can't you just filter out both of these unconditionally without > > worrying about DC_SHA1_SUBMODULE? > > I'm not sure what you mean by that. Like this perhaps? > > COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(filter-out > sha1dc/%,$(C_SOURCES))) > > While it's only a single line, I don't think it's any easier on the > eyes. I wasn't worried about readability or one or two lines (indeed, you could still do the filtering over two statements). What I meant was that sha1dc/ contains files whether DC_SHA1_SUBMODULE is defined or not. If the idea of this change is that there's no point in having Coccinelle check those foreign, imported files (and waste time in the process), then I was thinking that you'd want to omit sha1dc/* regardless of whether DC_SHA1_SUBMODULE is defined. Looking more closely at the Makefile, however, I see that C_SOURCES holds only one or the other of sha1dc/* or sha1collisiondetection/lib/*, so my concern is unfounded, which explains why my question confused you. ``` ## Junio C Hamano, 2018-07-23 19:37 Subject: Re: [PATCH 1/5] coccinelle: mark the 'coccicheck' make target as .PHONY Message-ID: URL: https://gitlist.dev/e/xmqqefft68wv.fsf%40gitster-ct.c.googlers.com In-Reply-To: <20180723135100.24288-2-szeder.dev@gmail.com> ``` SZEDER Gábor writes: > The 'coccicheck' target doesn't create a file with the same name, so > mark it as .PHONY. > > Signed-off-by: SZEDER Gábor > --- Good. It is customary to do so immediately before the target, not after a blank line, though. > Makefile | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/Makefile b/Makefile > index e4b503d259..27bfc196dd 100644 > --- a/Makefile > +++ b/Makefile > @@ -2685,6 +2685,8 @@ C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ)) > fi > coccicheck: $(patsubst %.cocci,%.cocci.patch,$(wildcard contrib/coccinelle/*.cocci)) > > +.PHONY: coccicheck > + > ### Installation rules > > ifneq ($(filter /%,$(firstword $(template_dir))),) ``` ## Junio C Hamano, 2018-07-23 19:37 Subject: Re: [PATCH 2/5] coccinelle: use $(addsuffix) in 'coccicheck' make target Message-ID: URL: https://gitlist.dev/e/xmqqa7qh68vr.fsf%40gitster-ct.c.googlers.com In-Reply-To: <20180723135100.24288-3-szeder.dev@gmail.com> ``` SZEDER Gábor writes: > The dependencies of the 'coccicheck' make target are listed with the > help of the $(patsubst) make function, which in this case doesn't do > any pattern substitution, but only adds the '.patch' suffix. > > Use the shorter and more idiomatic $(addsuffix) make function instead. Makes sense. > > Signed-off-by: SZEDER Gábor > --- > Makefile | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/Makefile b/Makefile > index 27bfc196dd..8f509576e9 100644 > --- a/Makefile > +++ b/Makefile > @@ -2683,7 +2683,7 @@ C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ)) > then \ > echo ' ' SPATCH result: $@; \ > fi > -coccicheck: $(patsubst %.cocci,%.cocci.patch,$(wildcard contrib/coccinelle/*.cocci)) > +coccicheck: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.cocci)) > > .PHONY: coccicheck ``` ## Junio C Hamano, 2018-07-23 19:40 Subject: Re: [PATCH 0/5] Misc Coccinelle-related improvements Message-ID: URL: https://gitlist.dev/e/xmqq601568re.fsf%40gitster-ct.c.googlers.com In-Reply-To: <20180723135100.24288-1-szeder.dev@gmail.com> ``` SZEDER Gábor writes: > Just a couple of minor Coccinelle-related improvements: > > - The first two patches are small cleanups. > > - The last three could make life perhaps just a tad bit easier for > devs running 'make coccicheck'. Thanks. All 5 patches make sense. Queued. ``` ## SZEDER Gábor, 2018-08-02 11:55 Subject: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained Message-ID: <20180802115522.16107-1-szeder.dev@gmail.com> URL: https://gitlist.dev/e/20180802115522.16107-1-szeder.dev%40gmail.com In-Reply-To: <20180723135100.24288-1-szeder.dev@gmail.com> ``` So, I have this PoC patch below on top of this patch series for some months now; actually this series was all fallout from working on this patch. It makes 'make -j coccicheck' much faster (speedup of over 99% :) in some scenarios that, in my experience, occur rather frequently when working on Coccinelle semantic patches (as opposed to occasionally running 'make coccicheck' to see whether there are any regressions). However, it's not ready for inclusion as it is, because, unfortunately, the improvements are not without some disadvantages, as explained in the second half of the commit message below. Anyway, I'm sending this out, because I don't see how I could make it any better, but other contributors working on semantic patches might still find it useful to keep this in their own fork, even in its current form. And perhaps someone will come up with a brilliant idea to address those disadvantages... Also available at: https://github.com/szeder/git make-coccicheck-finegrained -- >8 -- Subject: [PATCH] [PoC] coccinelle: make Coccinelle-related make targets more fine-grained When running 'make coccicheck', each semantic patch is applied to all source files in a separate shell for loop. This approach has the following disadvantages: 1. Even if you only modified a single C source file, that shell loop will still iterate over all source files and apply the semantic patches to all of them, wasting a lot of time. 2. If you apply only a single semantic patch (either implicitly, after you modified only one of them (and the results of the previous 'make coccicheck' are still there), or explicitly, by running e.g. 'make contrib/coccinelle/array.cocci.patch'), then you won't be able to exploit multiple CPU cores to speed up the operation, because that shell loop will iterate over all source files sequentially. 3. 'make coccicheck' can only use as many parallel jobs as the number of semantic patches, so even if you have more available CPU cores than that, you can't exploit them all to speed up this operation. 4. During 'make coccicheck' there is only a single SPATCH contrib/coccinelle/.cocci line (or as many lines as the number of parallel jobs) of output when starting to apply each semantic patch, and then comes a long silence without any indication of progress, because applying some of our semantic patches to all source files can take over a minute. This can trick developers new to semantic patches into thinking that something went wrong, hung, ended up in an endless loop, etc. It certainly confused my the first time. Let's add a bit of Makefile metaprogramming to generate finer-grained make targets applying one semantic patch to only a single source file, and specify these as dependencies of the targets applying one semantic patch to all source files. This way that shell loop can be avoided, semantic patches will only be applied to changed source files, and the same semantic patch can be applied in parallel to multiple source files. The only remaining sequential part is aggregating the suggested transformations from the individual targets into a single patch file, which is comparatively cheap (especially since ideally there aren't any suggestions). This change brings spectacular speedup in the scenario described in point (1) above. When the results of a previous 'make coccicheck' are there, the time needed to run touch apply.c ; time make -j4 coccicheck went from 3m42s to 1.848s, which is just over 99% speedup, yay!, and 'apply.c' is the second longest source file in our codebase. In the scenario in point (2), running touch contrib/coccinelle/array.cocci ; time make -j4 coccicheck went from 56.364s to 35.883s, which is ~36% speedup. All the above timings are best-of-five on a machine with 2 physical and 2 logical cores. I don't have the hardware to bring any numbers for point (3). The time needed to run 'make -j4 coccicheck' in a clean state didn't change, it's ~3m42s both with and without this patch. Unfortunately, this new approach has some disadvantages compared to the current situation: - [RFC] With this patch 'make coccicheck's output will look like this ('make' apparently doesn't iterate over semantic patches in lexicographical order): SPATCH commit.cocci abspath.c SPATCH commit.cocci advice.c <... lots of lines ...> SPATCH array.cocci http-walker.c SPATCH array.cocci remote-curl.c which means that the number of lines in the output grows from "Nr-of-semantic-patches" to "Nr-of-semantic-patches * Nr-of-source-files". Now, while this certainly addresses point (4) above, it can be considered too much, and I'm not sure that (4) is that big an issue to justify this much output. OTOH, I couldn't yet figure out a way to print a single 'SPATCH contrib/coccinelle/.cocci' line at the beginning of applying that semantic patch without triggering unnecessary work, effectively killing most of the runtime benefits. Or to somehow iterate over source files in the outer loop and over semantic patches in the inner loop, so we could have one output line per file. - [RFC] The overhead of applying a semantic patch to all source files became larger. 'make coccicheck' currently runs only one shell process and creates two output files for each semantic patch. With this patch it will run approx. "Nr-of-semantic-patches * Nr-of-source-files" shell processes and create twice as many output files. This overhead can be quantified by measuring the runtime of: make -j4 SPATCH=true coccicheck and this patch increases it from 0.142s to 1.479s. While this is indeed an order of magnitude increase, it's still negligible compared to the "real" runtime of 3m42s. So the increased overhead doesn't seem to matter on Linux, but I would expect that it's worse on OSX and Windows; though I'm not sure whether Coccinelle (with all its OCaml dependencies) works on those platforms in the first place. - [RFC] This approach uses $(eval), which we haven't used in any of our Makefiles yet. I wonder whether it's portable enough. And it's ugly and complicated. Signed-off-by: SZEDER Gábor --- Makefile | 52 ++++++++++++++++++++++++----------- contrib/coccinelle/.gitignore | 3 +- 2 files changed, 38 insertions(+), 17 deletions(-) diff --git a/Makefile b/Makefile index d616c04125..f516dd93d1 100644 --- a/Makefile +++ b/Makefile @@ -2674,25 +2674,44 @@ COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) else COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) endif +COCCI_SEM_PATCHES = $(wildcard contrib/coccinelle/*.cocci) -%.cocci.patch: %.cocci $(COCCI_SOURCES) - @echo ' ' SPATCH $<; \ - ret=0; \ - for f in $(COCCI_SOURCES); do \ - $(SPATCH) --sp-file $< $$f $(SPATCH_FLAGS) || \ - { ret=$$?; break; }; \ - done >$@+ 2>$@.log; \ - if test $$ret != 0; \ +define cocci_template +$(cocci_sem_patch)_dirs := $$(addprefix $(cocci_sem_patch).patches/,$$(sort $$(dir $$(COCCI_SOURCES)))) + +$$($(cocci_sem_patch)_dirs): + @mkdir -p $$@ + +# e.g. 'contrib/coccinelle/strbuf.cocci.patches/builtin/commit.c.patch' +# Applies one semantic patch to _one_ source file. +$(cocci_sem_patch).patches/%.patch: % $(cocci_sem_patch) + @printf ' SPATCH %-25s %s\n' $$(notdir $(cocci_sem_patch)) $$<; \ + if ! $$(SPATCH) --sp-file $(cocci_sem_patch) $$< $$(SPATCH_FLAGS) >$$@ 2>$$@.log; \ then \ - cat $@.log; \ + rm -f $$@; \ + cat $$@.log; \ exit 1; \ - fi; \ - mv $@+ $@; \ - if test -s $@; \ - then \ - echo ' ' SPATCH result: $@; \ fi -coccicheck: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.cocci)) + +# e.g. 'contrib/coccinelle/strbuf.cocci.patch' +# Applies one semantic patch to _all_ source files. +$(cocci_sem_patch).patch: $(cocci_sem_patch) $$($(cocci_sem_patch)_dirs) $$(patsubst %,$(cocci_sem_patch).patches/%.patch,$(COCCI_SOURCES)) + @>$$@+; \ + for f in $$(filter %.patch,$$^); do \ + if test -s $$$$f; \ + then \ + cat $$$$f >>$$@+; \ + fi \ + done; \ + mv $$@+ $$@; \ + if test -s $$@; then \ + echo ' ' SPATCH result: $$@; \ + fi +endef + +$(foreach cocci_sem_patch,$(COCCI_SEM_PATCHES),$(eval $(cocci_template))) + +coccicheck: $(addsuffix .patch,$(COCCI_SEM_PATCHES)) .PHONY: coccicheck @@ -2907,7 +2926,8 @@ profile-clean: $(RM) $(addsuffix *.gcno,$(addprefix $(PROFILE_DIR)/, $(object_dirs))) cocciclean: - $(RM) contrib/coccinelle/*.cocci.patch* + $(RM) contrib/coccinelle/*.cocci.patch + $(RM) -r contrib/coccinelle/*.cocci.patches/ clean: profile-clean coverage-clean cocciclean $(RM) *.res diff --git a/contrib/coccinelle/.gitignore b/contrib/coccinelle/.gitignore index d3f29646dc..7ae6ffa983 100644 --- a/contrib/coccinelle/.gitignore +++ b/contrib/coccinelle/.gitignore @@ -1 +1,2 @@ -*.patch* +*.cocci.patch +*.cocci.patches/ -- 2.18.0.408.g42635c01bc ``` ## René Scharfe, 2018-08-02 13:24 Subject: Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained Message-ID: URL: https://gitlist.dev/e/edd896c5-9d73-7771-984e-f7443b74d01f%40web.de In-Reply-To: <20180802115522.16107-1-szeder.dev@gmail.com> ``` Am 02.08.2018 um 13:55 schrieb SZEDER Gábor: > Let's add a bit of Makefile metaprogramming to generate finer-grained > make targets applying one semantic patch to only a single source file, > and specify these as dependencies of the targets applying one semantic > patch to all source files. This way that shell loop can be avoided, > semantic patches will only be applied to changed source files, and the > same semantic patch can be applied in parallel to multiple source > files. The only remaining sequential part is aggregating the > suggested transformations from the individual targets into a single > patch file, which is comparatively cheap (especially since ideally > there aren't any suggestions). > > This change brings spectacular speedup in the scenario described in > point (1) above. When the results of a previous 'make coccicheck' are > there, the time needed to run > > touch apply.c ; time make -j4 coccicheck > > went from 3m42s to 1.848s, which is just over 99% speedup, yay!, and > 'apply.c' is the second longest source file in our codebase. In the > scenario in point (2), running > > touch contrib/coccinelle/array.cocci ; time make -j4 coccicheck > > went from 56.364s to 35.883s, which is ~36% speedup. Awesome! > Unfortunately, this new approach has some disadvantages compared to > the current situation: > > - [RFC] > With this patch 'make coccicheck's output will look like this > ('make' apparently doesn't iterate over semantic patches in > lexicographical order): > > SPATCH commit.cocci abspath.c > SPATCH commit.cocci advice.c > <... lots of lines ...> > SPATCH array.cocci http-walker.c > SPATCH array.cocci remote-curl.c > > which means that the number of lines in the output grows from > "Nr-of-semantic-patches" to "Nr-of-semantic-patches * > Nr-of-source-files". The output is mostly interesting to see if something is happening, to see if some semantic patch generated a non-empty diff and to get error details. We have ca. 400 .c files and 8 .cocci files -- that means the lines would fill up my scroll-back buffer. Hmm. Would it be possible to have a phony target (or directory) per source file that just prints a line and depends on the individual file+cocci targets? (I don't know much make, you probably thought about it already.) Or to select one random .cocci file (let's say the first one) and only print SPATCH lines for this one (without mentioning its name)? (A dirty hack..) > --- > Makefile | 52 ++++++++++++++++++++++++----------- > contrib/coccinelle/.gitignore | 3 +- > 2 files changed, 38 insertions(+), 17 deletions(-) > > diff --git a/Makefile b/Makefile > index d616c04125..f516dd93d1 100644 > --- a/Makefile > +++ b/Makefile > @@ -2674,25 +2674,44 @@ COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) > else > COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) > endif > +COCCI_SEM_PATCHES = $(wildcard contrib/coccinelle/*.cocci) > > -%.cocci.patch: %.cocci $(COCCI_SOURCES) > - @echo ' ' SPATCH $<; \ > - ret=0; \ > - for f in $(COCCI_SOURCES); do \ > - $(SPATCH) --sp-file $< $$f $(SPATCH_FLAGS) || \ > - { ret=$$?; break; }; \ > - done >$@+ 2>$@.log; \ > - if test $$ret != 0; \ > +define cocci_template > +$(cocci_sem_patch)_dirs := $$(addprefix $(cocci_sem_patch).patches/,$$(sort $$(dir $$(COCCI_SOURCES)))) > + > +$$($(cocci_sem_patch)_dirs): > + @mkdir -p $$@ > + > +# e.g. 'contrib/coccinelle/strbuf.cocci.patches/builtin/commit.c.patch' > +# Applies one semantic patch to _one_ source file. > +$(cocci_sem_patch).patches/%.patch: % $(cocci_sem_patch) > + @printf ' SPATCH %-25s %s\n' $$(notdir $(cocci_sem_patch)) $$<; \ > + if ! $$(SPATCH) --sp-file $(cocci_sem_patch) $$< $$(SPATCH_FLAGS) >$$@ 2>$$@.log; \ > then \ > - cat $@.log; \ > + rm -f $$@; \ > + cat $$@.log; \ > exit 1; \ > - fi; \ > - mv $@+ $@; \ > - if test -s $@; \ > - then \ > - echo ' ' SPATCH result: $@; \ > fi > -coccicheck: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.cocci)) > + > +# e.g. 'contrib/coccinelle/strbuf.cocci.patch' > +# Applies one semantic patch to _all_ source files. > +$(cocci_sem_patch).patch: $(cocci_sem_patch) $$($(cocci_sem_patch)_dirs) $$(patsubst %,$(cocci_sem_patch).patches/%.patch,$(COCCI_SOURCES)) Are the dependencies correct (before and after the patch)? E.g. are all semantic patches reapplied after cache.h was changed? I think we ignore header files currently, and that becomes more of a problem if the check becomes more fine-grained. René ``` ## Jeff King, 2018-08-02 18:01 Subject: Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained Message-ID: <20180802180155.GD15984@sigill.intra.peff.net> URL: https://gitlist.dev/e/20180802180155.GD15984%40sigill.intra.peff.net In-Reply-To: <20180802115522.16107-1-szeder.dev@gmail.com> ``` On Thu, Aug 02, 2018 at 01:55:22PM +0200, SZEDER Gábor wrote: > Let's add a bit of Makefile metaprogramming to generate finer-grained > make targets applying one semantic patch to only a single source file, > and specify these as dependencies of the targets applying one semantic > patch to all source files. This way that shell loop can be avoided, > semantic patches will only be applied to changed source files, and the > same semantic patch can be applied in parallel to multiple source > files. The only remaining sequential part is aggregating the > suggested transformations from the individual targets into a single > patch file, which is comparatively cheap (especially since ideally > there aren't any suggestions). > > This change brings spectacular speedup in the scenario described in > point (1) above. When the results of a previous 'make coccicheck' are > there, the time needed to run > > touch apply.c ; time make -j4 coccicheck > > went from 3m42s to 1.848s, which is just over 99% speedup, yay!, and > 'apply.c' is the second longest source file in our codebase. In the > scenario in point (2), running > > touch contrib/coccinelle/array.cocci ; time make -j4 coccicheck > > went from 56.364s to 35.883s, which is ~36% speedup. I really like this direction. The slowness of coccicheck is one of the things that has prevented me from running it more frequently. And I'm a big fan of breaking steps down as much as possible into make targets. It lets make do its job (avoiding repeated work and parallelizing). > All the above timings are best-of-five on a machine with 2 physical > and 2 logical cores. I don't have the hardware to bring any numbers > for point (3). The time needed to run 'make -j4 coccicheck' in a > clean state didn't change, it's ~3m42s both with and without this > patch. On a 40-core (20+20) machine, doing "make -j40 coccicheck" from scratch went from: real 1m25.520s user 5m41.492s sys 0m26.776s to: real 0m24.300s user 14m35.084s sys 0m50.108s I was surprised by the jump in CPU times. Doing "make -j1 coccicheck" with your patch results in: real 5m34.887s user 5m5.620s sys 0m19.908s so it's really the parallelizing that seems to be to blame (possibly because this CPU boosts from 2.3Ghz to 3.0Ghz, and we're only using 8 threads in the first example). > - [RFC] > With this patch 'make coccicheck's output will look like this > ('make' apparently doesn't iterate over semantic patches in > lexicographical order): > > SPATCH commit.cocci abspath.c > SPATCH commit.cocci advice.c > <... lots of lines ...> > SPATCH array.cocci http-walker.c > SPATCH array.cocci remote-curl.c > > which means that the number of lines in the output grows from > "Nr-of-semantic-patches" to "Nr-of-semantic-patches * > Nr-of-source-files". IMHO this is not really that big a deal. We are doing useful work for each line, so to me it's just more eye candy (and it's really satisfying to watch it zip by on the 40-core machine ;) ). What if we inverted the current loop? That is, right now we iterate over the cocci patches at the Makefile level, and then for each target we iterate over the giant list of source files. Instead, we could teach the Makefile to iterate over the source files, with a target for each, and then hit each cocci patch inside there. That would give roughly the same output as a normal build. But moreover, I wonder if we could make things faster by actually combining the cocci files into a single set of rules to be applied to each source file. That would mean invoking spatch 1/8 as much. It does give fewer opportunities for "make" to reuse work, but that only matters when the cocci files change (which is much less frequent than source files changing). Doing: cat contrib/coccinelle/*.cocci >mega.cocci make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci gives me: real 0m17.573s user 10m56.724s sys 0m11.548s And that should show an improvement on more normal-sized systems, too, because we really are eliminating some of the startup overhead. The other obvious downside is that you don't get individual patches for each class of transformation. Do we care? Certainly for a routine "make coccicheck" I primarily want to know: - is there something that needs fixing? - give me the patch for all fixes So I wonder if we'd want to have that be the default, and then perhaps optionally give some targets to let people run single scripts (or not; they could probably just run spatch themselves). > - [RFC] > The overhead of applying a semantic patch to all source files > became larger. 'make coccicheck' currently runs only one shell > process and creates two output files for each semantic patch. > With this patch it will run approx. "Nr-of-semantic-patches * > Nr-of-source-files" shell processes and create twice as many > output files. Doing the "one big .cocci" would help with this, too (and again puts us in the same ballpark as a compile). > - [RFC] > This approach uses $(eval), which we haven't used in any of our > Makefiles yet. I wonder whether it's portable enough. And it's > ugly and complicated. I looked into this a long time ago for another set of Makefile patches I was considering. $(eval) was added to GNU make in 3.80, released in 2002. Which is probably fine by now. If it isn't, one more exotic option would be to push the coccicheck stuff into its own Makefile, and just run it via recursive make. Then anybody doing a vanilla build can do so even with an antique make, but you could only "make coccicheck" with something from the last 16 years (but good luck getting ocaml running there). I suspect if we go with the one-spatch-per-source route, though, that we could do this just with regular make rules. -Peff ``` ## Jeff King, 2018-08-02 18:31 Subject: Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained Message-ID: <20180802183145.GA23690@sigill.intra.peff.net> URL: https://gitlist.dev/e/20180802183145.GA23690%40sigill.intra.peff.net In-Reply-To: <20180802180155.GD15984@sigill.intra.peff.net> ``` On Thu, Aug 02, 2018 at 02:01:55PM -0400, Jeff King wrote: > I suspect if we go with the one-spatch-per-source route, though, that we > could do this just with regular make rules. Yeah, it's pretty straightforward: diff --git a/Makefile b/Makefile index d616c0412..86fdcf567 100644 --- a/Makefile +++ b/Makefile @@ -2674,15 +2674,17 @@ COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) else COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) endif +COCCI_COMBINED = contrib/coccinelle/combined.cocci +COCCI_SEM_PATCHES = $(filter-out $(COCCI_COMBINED), $(wildcard contrib/coccinelle/*.cocci)) -%.cocci.patch: %.cocci $(COCCI_SOURCES) +$(COCCI_COMBINED): $(COCCI_SEM_PATCHES) + cat $^ >$@+ + mv $@+ $@ + +$(COCCI_COMBINED).patches/%.patch: % $(COCCI_COMBINED) @echo ' ' SPATCH $<; \ - ret=0; \ - for f in $(COCCI_SOURCES); do \ - $(SPATCH) --sp-file $< $$f $(SPATCH_FLAGS) || \ - { ret=$$?; break; }; \ - done >$@+ 2>$@.log; \ - if test $$ret != 0; \ + mkdir -p $(dir $@) || exit 1; \ + if ! $(SPATCH) --sp-file $(COCCI_COMBINED) $< $(SPATCH_FLAGS) >$@+ 2>$@.log; \ then \ cat $@.log; \ exit 1; \ @@ -2692,7 +2694,8 @@ endif then \ echo ' ' SPATCH result: $@; \ fi -coccicheck: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.cocci)) + +coccicheck: $(patsubst %, $(COCCI_COMBINED).patches/%.patch, $(COCCI_SOURCES)) .PHONY: coccicheck @@ -2907,7 +2910,7 @@ profile-clean: $(RM) $(addsuffix *.gcno,$(addprefix $(PROFILE_DIR)/, $(object_dirs))) cocciclean: - $(RM) contrib/coccinelle/*.cocci.patch* + $(RM) -r contrib/coccinelle/*.cocci.patches clean: profile-clean coverage-clean cocciclean $(RM) *.res I guess you could even replace "COCCI_COMBINED" with "COCCI_PATCH" in most of the targets, and that would let people do individual: make COCCI_PATCH=contrib/coccinelle/foo.cocci coccicheck The default would just be the concatenated version. -Peff ``` ## Eric Sunshine, 2018-08-02 19:46 Subject: Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained Message-ID: URL: https://gitlist.dev/e/CAPig%2BcQSg%3DMBB7zp5bx5Z6tep89FaqNfTXV0tATRRG3WAPn2MQ%40mail.gmail.com In-Reply-To: <20180802180155.GD15984@sigill.intra.peff.net> ``` On Thu, Aug 2, 2018 at 2:02 PM Jeff King wrote: > On Thu, Aug 02, 2018 at 01:55:22PM +0200, SZEDER Gábor wrote: > > This approach uses $(eval), which we haven't used in any of our > > Makefiles yet. I wonder whether it's portable enough. And it's > > ugly and complicated. > > I looked into this a long time ago for another set of Makefile patches I > was considering. $(eval) was added to GNU make in 3.80, released in > 2002. Which is probably fine by now. For the record, MacOS developer tools are stuck at GNU make 3.81 (from 2006). ``` ## Jeff King, 2018-08-02 21:29 Subject: Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained Message-ID: <20180802212930.GB32538@sigill.intra.peff.net> URL: https://gitlist.dev/e/20180802212930.GB32538%40sigill.intra.peff.net In-Reply-To: ``` On Thu, Aug 02, 2018 at 03:46:08PM -0400, Eric Sunshine wrote: > On Thu, Aug 2, 2018 at 2:02 PM Jeff King wrote: > > On Thu, Aug 02, 2018 at 01:55:22PM +0200, SZEDER Gábor wrote: > > > This approach uses $(eval), which we haven't used in any of our > > > Makefiles yet. I wonder whether it's portable enough. And it's > > > ugly and complicated. > > > > I looked into this a long time ago for another set of Makefile patches I > > was considering. $(eval) was added to GNU make in 3.80, released in > > 2002. Which is probably fine by now. > > For the record, MacOS developer tools are stuck at GNU make 3.81 (from 2006). Thanks, that's interesting to know. Fortunately from my past research that means that it should have both $(eval) and $(call). If we follow my suggestions here, we don't need to care for now, but I won't be surprised if it comes up again at some point. -Peff ``` ## Ævar Arnfjörð Bjarmason, 2018-08-02 21:45 Subject: Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained Message-ID: <87bmaktpbt.fsf@evledraar.gmail.com> URL: https://gitlist.dev/e/87bmaktpbt.fsf%40evledraar.gmail.com In-Reply-To: <20180802180155.GD15984@sigill.intra.peff.net> ``` On Thu, Aug 02 2018, Jeff King wrote: > On Thu, Aug 02, 2018 at 01:55:22PM +0200, SZEDER Gábor wrote: > >> Let's add a bit of Makefile metaprogramming to generate finer-grained >> make targets applying one semantic patch to only a single source file, >> and specify these as dependencies of the targets applying one semantic >> patch to all source files. This way that shell loop can be avoided, >> semantic patches will only be applied to changed source files, and the >> same semantic patch can be applied in parallel to multiple source >> files. The only remaining sequential part is aggregating the >> suggested transformations from the individual targets into a single >> patch file, which is comparatively cheap (especially since ideally >> there aren't any suggestions). >> >> This change brings spectacular speedup in the scenario described in >> point (1) above. When the results of a previous 'make coccicheck' are >> there, the time needed to run >> >> touch apply.c ; time make -j4 coccicheck >> >> went from 3m42s to 1.848s, which is just over 99% speedup, yay!, and >> 'apply.c' is the second longest source file in our codebase. In the >> scenario in point (2), running >> >> touch contrib/coccinelle/array.cocci ; time make -j4 coccicheck >> >> went from 56.364s to 35.883s, which is ~36% speedup. > > I really like this direction. The slowness of coccicheck is one of the > things that has prevented me from running it more frequently. And I'm a > big fan of breaking steps down as much as possible into make targets. It > lets make do its job (avoiding repeated work and parallelizing). Yeah, this is great. Also, CC-ing some of the recent contributors to linux.git's coccinelle, perhaps they're interested / have comments. >> All the above timings are best-of-five on a machine with 2 physical >> and 2 logical cores. I don't have the hardware to bring any numbers >> for point (3). The time needed to run 'make -j4 coccicheck' in a >> clean state didn't change, it's ~3m42s both with and without this >> patch. > > On a 40-core (20+20) machine, doing "make -j40 coccicheck" from scratch > went from: > > real 1m25.520s > user 5m41.492s > sys 0m26.776s > > to: > > real 0m24.300s > user 14m35.084s > sys 0m50.108s > > I was surprised by the jump in CPU times. Doing "make -j1 coccicheck" > with your patch results in: > > real 5m34.887s > user 5m5.620s > sys 0m19.908s > > so it's really the parallelizing that seems to be to blame (possibly > because this CPU boosts from 2.3Ghz to 3.0Ghz, and we're only using 8 > threads in the first example). > >> - [RFC] >> With this patch 'make coccicheck's output will look like this >> ('make' apparently doesn't iterate over semantic patches in >> lexicographical order): >> >> SPATCH commit.cocci abspath.c >> SPATCH commit.cocci advice.c >> <... lots of lines ...> >> SPATCH array.cocci http-walker.c >> SPATCH array.cocci remote-curl.c >> >> which means that the number of lines in the output grows from >> "Nr-of-semantic-patches" to "Nr-of-semantic-patches * >> Nr-of-source-files". > > IMHO this is not really that big a deal. We are doing useful work for > each line, so to me it's just more eye candy (and it's really satisfying > to watch it zip by on the 40-core machine ;) ). FWIW on the 8-cpu box I usually test on this went from 2m30s to 1m50s, so not a huge improvement in time, but nice to have the per-file progress. > What if we inverted the current loop? That is, right now we iterate over > the cocci patches at the Makefile level, and then for each target we > iterate over the giant list of source files. Instead, we could teach the > Makefile to iterate over the source files, with a target for each, and > then hit each cocci patch inside there. > > That would give roughly the same output as a normal build. But moreover, > I wonder if we could make things faster by actually combining the cocci > files into a single set of rules to be applied to each source file. That > would mean invoking spatch 1/8 as much. It does give fewer opportunities > for "make" to reuse work, but that only matters when the cocci files > change (which is much less frequent than source files changing). > > Doing: > > cat contrib/coccinelle/*.cocci >mega.cocci > make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci > > gives me: > > real 0m17.573s > user 10m56.724s > sys 0m11.548s > > And that should show an improvement on more normal-sized systems, too, > because we really are eliminating some of the startup overhead. > > The other obvious downside is that you don't get individual patches for > each class of transformation. Do we care? Certainly for a routine "make > coccicheck" I primarily want to know: > > - is there something that needs fixing? > > - give me the patch for all fixes > > So I wonder if we'd want to have that be the default, and then perhaps > optionally give some targets to let people run single scripts (or not; > they could probably just run spatch themselves). > >> - [RFC] >> The overhead of applying a semantic patch to all source files >> became larger. 'make coccicheck' currently runs only one shell >> process and creates two output files for each semantic patch. >> With this patch it will run approx. "Nr-of-semantic-patches * >> Nr-of-source-files" shell processes and create twice as many >> output files. > > Doing the "one big .cocci" would help with this, too (and again puts us > in the same ballpark as a compile). > >> - [RFC] >> This approach uses $(eval), which we haven't used in any of our >> Makefiles yet. I wonder whether it's portable enough. And it's >> ugly and complicated. > > I looked into this a long time ago for another set of Makefile patches I > was considering. $(eval) was added to GNU make in 3.80, released in > 2002. Which is probably fine by now. > > If it isn't, one more exotic option would be to push the coccicheck > stuff into its own Makefile, and just run it via recursive make. Then > anybody doing a vanilla build can do so even with an antique make, but > you could only "make coccicheck" with something from the last 16 years > (but good luck getting ocaml running there). > > I suspect if we go with the one-spatch-per-source route, though, that we > could do this just with regular make rules. > > -Peff ``` ## Jonathan Nieder, 2018-08-03 06:21 Subject: Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained Message-ID: <20180803062144.GB237521@aiede.svl.corp.google.com> URL: https://gitlist.dev/e/20180803062144.GB237521%40aiede.svl.corp.google.com In-Reply-To: <20180802183145.GA23690@sigill.intra.peff.net> ``` Hi, Jeff King wrote: > On Thu, Aug 02, 2018 at 02:01:55PM -0400, Jeff King wrote: >> I suspect if we go with the one-spatch-per-source route, though, that we >> could do this just with regular make rules. > > Yeah, it's pretty straightforward: > > diff --git a/Makefile b/Makefile > index d616c0412..86fdcf567 100644 > --- a/Makefile > +++ b/Makefile > @@ -2674,15 +2674,17 @@ COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) > else > COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) > endif > +COCCI_COMBINED = contrib/coccinelle/combined.cocci I like this approach. [...] > I guess you could even replace "COCCI_COMBINED" with "COCCI_PATCH" in > most of the targets, and that would let people do individual: > > make COCCI_PATCH=contrib/coccinelle/foo.cocci coccicheck The issue here is that the dependencies for foo.cocci become unreliable, so I'd rather have a separate target for that (e.g. depending on FORCE) if we go that way. Thanks, Jonathan ``` ## Julia Lawall, 2018-08-03 06:22 Subject: Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained Message-ID: URL: https://gitlist.dev/e/alpine.DEB.2.20.1808030755350.2446%40hadrien In-Reply-To: <87bmaktpbt.fsf@evledraar.gmail.com> ``` On Thu, 2 Aug 2018, Ævar Arnfjörð Bjarmason wrote: > > On Thu, Aug 02 2018, Jeff King wrote: > > > On Thu, Aug 02, 2018 at 01:55:22PM +0200, SZEDER Gábor wrote: > > > >> Let's add a bit of Makefile metaprogramming to generate finer-grained > >> make targets applying one semantic patch to only a single source file, > >> and specify these as dependencies of the targets applying one semantic > >> patch to all source files. This way that shell loop can be avoided, > >> semantic patches will only be applied to changed source files, and the > >> same semantic patch can be applied in parallel to multiple source > >> files. This was already possible. Make coccicheck is not supposed to be used with -j, but rather with J=n. That tells Coccinelle to parallelize the treatment of the files internally. In this case, the semantic patch is only parsed once, and then n worker processes are forked to treat the different files. > The only remaining sequential part is aggregating the > >> suggested transformations from the individual targets into a single > >> patch file, which is comparatively cheap (especially since ideally > >> there aren't any suggestions). > >> > >> This change brings spectacular speedup in the scenario described in > >> point (1) above. When the results of a previous 'make coccicheck' are > >> there, the time needed to run > >> > >> touch apply.c ; time make -j4 coccicheck > >> > >> went from 3m42s to 1.848s, which is just over 99% speedup, yay!, and > >> 'apply.c' is the second longest source file in our codebase. In the > >> scenario in point (2), running > >> > >> touch contrib/coccinelle/array.cocci ; time make -j4 coccicheck > >> > >> went from 56.364s to 35.883s, which is ~36% speedup. > > > > I really like this direction. The slowness of coccicheck is one of the > > things that has prevented me from running it more frequently. And I'm a > > big fan of breaking steps down as much as possible into make targets. It > > lets make do its job (avoiding repeated work and parallelizing). > > Yeah, this is great. Also, CC-ing some of the recent contributors to > linux.git's coccinelle, perhaps they're interested / have comments. I have extended the list of recipients with Nicolas Palix and the Coccinelle mailing list. In particular, Nicolas should comment on any changes. > > >> All the above timings are best-of-five on a machine with 2 physical > >> and 2 logical cores. I don't have the hardware to bring any numbers > >> for point (3). The time needed to run 'make -j4 coccicheck' in a > >> clean state didn't change, it's ~3m42s both with and without this > >> patch. > > > > On a 40-core (20+20) machine, doing "make -j40 coccicheck" from scratch > > went from: > > > > real 1m25.520s > > user 5m41.492s > > sys 0m26.776s > > > > to: > > > > real 0m24.300s > > user 14m35.084s > > sys 0m50.108s By default, when the -j option is not used, make coccicheck causes Coccinelle to run on all of the cores of the machine. In my experience, on a laptop (2 physical cores with hyperthreading), this is basically OK. And on a server, it is a disaster. On a machine with 80 physical cores and hyperthreading, make coccicheck will run each instance of Coccinelle such that it parallelizes on 160 cores. But in reality, there is not much difference between 20 and 40 cores, and after 40 cores the performance starts to degrade. So basically, using more than half of the physical cores on each socket is a loss. > > > > I was surprised by the jump in CPU times. Doing "make -j1 coccicheck" > > with your patch results in: > > > > real 5m34.887s > > user 5m5.620s > > sys 0m19.908s > > > > so it's really the parallelizing that seems to be to blame (possibly > > because this CPU boosts from 2.3Ghz to 3.0Ghz, and we're only using 8 > > threads in the first example). > > > >> - [RFC] > >> With this patch 'make coccicheck's output will look like this > >> ('make' apparently doesn't iterate over semantic patches in > >> lexicographical order): > >> > >> SPATCH commit.cocci abspath.c > >> SPATCH commit.cocci advice.c > >> <... lots of lines ...> > >> SPATCH array.cocci http-walker.c > >> SPATCH array.cocci remote-curl.c > >> > >> which means that the number of lines in the output grows from > >> "Nr-of-semantic-patches" to "Nr-of-semantic-patches * > >> Nr-of-source-files". > > > > IMHO this is not really that big a deal. We are doing useful work for > > each line, so to me it's just more eye candy (and it's really satisfying > > to watch it zip by on the 40-core machine ;) ). > > FWIW on the 8-cpu box I usually test on this went from 2m30s to 1m50s, > so not a huge improvement in time, but nice to have the per-file > progress. Coccinelle already does this when make coccicheck is used with J=n. It makes a subdirectory for the semantic patch that it is currently working on, and this subdirectory contains stderr.n files that contain HANDLING and the name of the file being treated. It also prints information about performance bottlenecks. I think that make coccicheck turns off all this reporting by default, but you can get it back with SPFLAGS="--quiet" > > What if we inverted the current loop? That is, right now we iterate over > > the cocci patches at the Makefile level, and then for each target we > > iterate over the giant list of source files. Instead, we could teach the > > Makefile to iterate over the source files, with a target for each, and > > then hit each cocci patch inside there. > > > > That would give roughly the same output as a normal build. But moreover, > > I wonder if we could make things faster by actually combining the cocci > > files into a single set of rules to be applied to each source file. That > > would mean invoking spatch 1/8 as much. It does give fewer opportunities > > for "make" to reuse work, but that only matters when the cocci files > > change (which is much less frequent than source files changing). > > > > Doing: > > > > cat contrib/coccinelle/*.cocci >mega.cocci > > make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci There was already a COCCI=foo.cocci argument to focus on a single semantic patch. I'm surprised that the above cat command would work. Semantic patch rules have names, and Coccinelle will not be happy isf two rules have the same name. Some may also have variables declared in initializers, although perhaps the ones in the kernel don't do this. Causing these variables to be shared would not have a good effect. Putting everything together can also go counter to the optimizations that Coccinelle provides. You can speed up spatch a lot on rules that mention specific functions by running id-utils on your code base in advance (sh coccinelle/scripts/idutils_index.sh). The if you have one semantic patch that only applies to files that contain foo and another that only applies to files that contain bar, then each will only be applied to its respective files. If you don't have an id-utils index this optimization will be done by Coccinelle by first scanning files for foo and bar, but the index is obviously much faster. If your semantic patch can be relavant to files that contain foo or bar, then the rules from the foo semantic patch (which could be very slow) will also be uselessly applied to the bar files. Whether this is relevant in practice depends on the specific semantic patches of course. > > > > gives me: > > > > real 0m17.573s > > user 10m56.724s > > sys 0m11.548s > > > > And that should show an improvement on more normal-sized systems, too, > > because we really are eliminating some of the startup overhead. > > > > The other obvious downside is that you don't get individual patches for > > each class of transformation. Do we care? Certainly for a routine "make > > coccicheck" I primarily want to know: > > > > - is there something that needs fixing? > > > > - give me the patch for all fixes > > > > So I wonder if we'd want to have that be the default, and then perhaps > > optionally give some targets to let people run single scripts (or not; > > they could probably just run spatch themselves). > > > >> - [RFC] > >> The overhead of applying a semantic patch to all source files > >> became larger. 'make coccicheck' currently runs only one shell > >> process Not sure to understand this. To my understanding it runs as many shell processes as there are cores on the machine. julia > and creates two output files for each semantic patch. > >> With this patch it will run approx. "Nr-of-semantic-patches * > >> Nr-of-source-files" shell processes and create twice as many > >> output files. > > > > Doing the "one big .cocci" would help with this, too (and again puts us > > in the same ballpark as a compile). > > > >> - [RFC] > >> This approach uses $(eval), which we haven't used in any of our > >> Makefiles yet. I wonder whether it's portable enough. And it's > >> ugly and complicated. > > > > I looked into this a long time ago for another set of Makefile patches I > > was considering. $(eval) was added to GNU make in 3.80, released in > > 2002. Which is probably fine by now. > > > > If it isn't, one more exotic option would be to push the coccicheck > > stuff into its own Makefile, and just run it via recursive make. Then > > anybody doing a vanilla build can do so even with an antique make, but > > you could only "make coccicheck" with something from the last 16 years > > (but good luck getting ocaml running there). > > > > I suspect if we go with the one-spatch-per-source route, though, that we > > could do this just with regular make rules. > > > > -Peff > ``` ## Julia Lawall, 2018-08-03 06:25 Subject: Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained Message-ID: URL: https://gitlist.dev/e/alpine.DEB.2.20.1808030824330.2446%40hadrien In-Reply-To: <87bmaktpbt.fsf@evledraar.gmail.com> ``` By the way, you can see my performance numbers here: https://www.usenix.org/system/files/conference/atc18/atc18-lawall.pdf Page 8, figures 4 and 5. julia ``` ## Jonathan Nieder, 2018-08-03 06:44 Subject: Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained Message-ID: <20180803064419.GC237521@aiede.svl.corp.google.com> URL: https://gitlist.dev/e/20180803064419.GC237521%40aiede.svl.corp.google.com In-Reply-To: ``` Hi, Julia Lawall wrote: > This was already possible. Make coccicheck is not supposed to be used > with -j, but rather with J=n. That tells Coccinelle to parallelize the > treatment of the files internally. In this case, the semantic patch is > only parsed once, and then n worker processes are forked to treat the > different files. Thanks for this hint. I wonder if we can make this happen automatically under suitable conditions. We can parse -j out of MAKEFLAGS and convert it into a J= argument, but that only solves half the problem: the "make" parent process would think that the coccinelle run only deserves to occupy one job slot. Technically we could do all this using a wrapper that pretends to be a submake (prefixing the command with '+', parsing jobserver options from the MAKEFLAGS envvar) but it gets ugly. It's likely that the best we can do is just to advertise J more prominently. [...] >> On Thu, Aug 02 2018, Jeff King wrote: >>> cat contrib/coccinelle/*.cocci >mega.cocci >>> make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci > > There was already a COCCI=foo.cocci argument to focus on a single semantic > patch. > > I'm surprised that the above cat command would work. Semantic patch rules > have names, and Coccinelle will not be happy isf two rules have the same > name. Yes, Git's semantic patches (in contrib/coccinelle) tend to be relatively undemanding. > Some may also have variables declared in initializers, although > perhaps the ones in the kernel don't do this. Causing these variables to > be shared would not have a good effect. ... oh! You're thinking of the Linux kernel. It looks like Linux's scripts/coccicheck has a lot we can crib from. That's where the J envvar and automatic parallelism you mentioned are implemented, too. So it sounds to me like at a minimum we should use all of that. ;-) Thanks again for the pointers. Sincerely, Jonathan ``` ## Julia Lawall, 2018-08-03 06:52 Subject: Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained Message-ID: URL: https://gitlist.dev/e/alpine.DEB.2.20.1808030848160.2446%40hadrien In-Reply-To: <20180803064419.GC237521@aiede.svl.corp.google.com> ``` On Thu, 2 Aug 2018, Jonathan Nieder wrote: > Hi, > > Julia Lawall wrote: > > > This was already possible. Make coccicheck is not supposed to be used > > with -j, but rather with J=n. That tells Coccinelle to parallelize the > > treatment of the files internally. In this case, the semantic patch is > > only parsed once, and then n worker processes are forked to treat the > > different files. > > Thanks for this hint. > > I wonder if we can make this happen automatically under suitable > conditions. We can parse -j out of MAKEFLAGS and convert it into a > J= argument, It does happen automatically, in the sense that the default with no -j or J=option is to use all the cores on the machine. With -j, it seems to have to default to one core internally: d7059ca0147adcd495f3c5b41f260e1ac55bb679 Also, if it didn't do that, it would end up with the product of the -j option and the number of cores on the machine, which would give very bad performance. > but that only solves half the problem: the "make" > parent process would think that the coccinelle run only deserves to > occupy one job slot. Yes, it seems to be intended to be run by itself. > Technically we could do all this using a wrapper that pretends to be a > submake > (prefixing the command with '+', parsing jobserver options from the > MAKEFLAGS envvar) but it gets ugly. > > It's likely that the best we can do is just to advertise J more > prominently. > > [...] > >> On Thu, Aug 02 2018, Jeff King wrote: > > >>> cat contrib/coccinelle/*.cocci >mega.cocci > >>> make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci > > > > There was already a COCCI=foo.cocci argument to focus on a single semantic > > patch. > > > > I'm surprised that the above cat command would work. Semantic patch rules > > have names, and Coccinelle will not be happy isf two rules have the same > > name. > > Yes, Git's semantic patches (in contrib/coccinelle) tend to be > relatively undemanding. > > > Some may also have variables declared in initializers, although > > perhaps the ones in the kernel don't do this. Causing these variables to > > be shared would not have a good effect. > > ... oh! You're thinking of the Linux kernel. > > It looks like Linux's scripts/coccicheck has a lot we can crib from. > That's where the J envvar and automatic parallelism you mentioned are > implemented, too. > > So it sounds to me like at a minimum we should use all of that. ;-) Yes. julia ``` ## Jeff King, 2018-08-03 13:08 Subject: Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained Message-ID: <20180803130824.GC4671@sigill.intra.peff.net> URL: https://gitlist.dev/e/20180803130824.GC4671%40sigill.intra.peff.net In-Reply-To: <20180803062144.GB237521@aiede.svl.corp.google.com> ``` On Thu, Aug 02, 2018 at 11:21:44PM -0700, Jonathan Nieder wrote: > > diff --git a/Makefile b/Makefile > > index d616c0412..86fdcf567 100644 > > --- a/Makefile > > +++ b/Makefile > > @@ -2674,15 +2674,17 @@ COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) > > else > > COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) > > endif > > +COCCI_COMBINED = contrib/coccinelle/combined.cocci > > I like this approach. I was pretty pleased with myself, too, but I had a lingering doubt about whether just cat-ing the files was legitimate. It sounds from the response elsewhere that it's not (but just happens to work now for out limited case). But it also sounds like there may be even better options. > > I guess you could even replace "COCCI_COMBINED" with "COCCI_PATCH" in > > most of the targets, and that would let people do individual: > > > > make COCCI_PATCH=contrib/coccinelle/foo.cocci coccicheck > > The issue here is that the dependencies for foo.cocci become > unreliable, so I'd rather have a separate target for that (e.g. > depending on FORCE) if we go that way. Can you be more specific? I don't see how it's unreliable, unless you mean that anything relying on "coccicheck" would depend on the exact value of COCCI_PATCH. But it may all be moot anyway, based no the responses elsewhere in the thread. -Peff ``` ## Jonathan Nieder, 2018-08-05 23:02 Subject: Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained Message-ID: <20180805230245.GB208811@aiede.svl.corp.google.com> URL: https://gitlist.dev/e/20180805230245.GB208811%40aiede.svl.corp.google.com In-Reply-To: <20180803130824.GC4671@sigill.intra.peff.net> ``` Hi, Jeff King wrote: > On Thu, Aug 02, 2018 at 11:21:44PM -0700, Jonathan Nieder wrote: >> Jeff King wrote: >>> I guess you could even replace "COCCI_COMBINED" with "COCCI_PATCH" in >>> most of the targets, and that would let people do individual: >>> >>> make COCCI_PATCH=contrib/coccinelle/foo.cocci coccicheck >> >> The issue here is that the dependencies for foo.cocci become >> unreliable, so I'd rather have a separate target for that (e.g. >> depending on FORCE) if we go that way. > > Can you be more specific? I don't see how it's unreliable, unless you > mean that anything relying on "coccicheck" would depend on the exact > value of COCCI_PATCH. Yes, that's what I mean: changing which COCCI_PATCH to try would need to trigger re-runs. > But it may all be moot anyway, based no the responses elsewhere in the > thread. Sure. Thanks, Jonathan ```