{"thread":{"id":"58360","subject":"[PATCH 0/5] cocci: make \"incremental\" possible + a ccache-like tool","startedAt":"2022-08-25T14:36:35Z","lastAt":"2022-11-11T22:23:22Z","messageCount":72,"participants":["Ævar Arnfjörð Bjarmason","SZEDER Gábor","Taylor Blau","Jeff King","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"461950","messageId":"cover-0.5-00000000000-20220825T141212Z-avarab@gmail.com","threadId":"58360","inReplyTo":null,"subject":"[PATCH 0/5] cocci: make \"incremental\" possible + a ccache-like tool","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-25T14:36:12Z","receivedAt":"2022-08-25T14:36:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This series makes the \"coccicheck\" rule incremental by only re-running\n\"spatch\" for those file/rule combinations that changed.\n\nThe result is that e.g. running \"make coccicheck\" followed by a:\n\n\tmake -W grep.c coccicheck\n\nWill take around 2s instead of a *lot* longer. The 3/5 change has all\nthe details about this, and why it's safe.\n\nFurthermore we'll properly handle re-building if any of the required\nheaders change (e.g. column.h).\n\nI had a previous stab at trying to make \"coccicheck\" faster a while\nago in [1], but that was fatally flawed because it would break in\ncases where grep.c e.g. dependen on a header that changed.\n\nThis series doesn't suffer from that issue, because we piggy-back on\nthe corresponding .depend/* files to extract dependency info. This\ndoes mean that the incremental speed-ups depend on\nCOMPUTE_HEADER_DEPENDENCIES=yes (which is the case with \"gcc\" and\n\"clang\" by default).\n\nI've been running with this series locally for a while. I think it\nshould fully solve the lamentations about the coccicheck being too\nslow (e.g. in [2]), particularly when paired with the new\n\"spatchcache\" helper added in 5/5.\n\nEven without that helper running \"make coccicheck\" e.g. after applying\na patch series now becomes viable, provided the patches don't modify\none of the \"rebuild everything\" files we have (such as cache.h and\ngit-compat-util.h).\n\n1. https://lore.kernel.org/git/20210302205103.12230-1-avarab@gmail.com/\n2. https://lore.kernel.org/git/xmqqh7249b8d.fsf@gitster.g/\n\nÆvar Arnfjörð Bjarmason (5):\n Makefile: add ability to TAB-complete cocci *.patch rules\n Makefile: have \"coccicheck\" re-run if flags change\n cocci: make \"coccicheck\" rule incremental\n cocci: make incremental compilation even faster\n spatchcache: add a ccache-alike for \"spatch\"\n\n .gitignore | 1 +\n Makefile | 114 +++++++--\n contrib/coccinelle/spatchcache | 217 ++++++++++++++++++\n .../coccinelle/the_repository.pending.cocci | 1 -\n shared.mak | 4 +-\n 5 files changed, 310 insertions(+), 27 deletions(-)\n create mode 100755 contrib/coccinelle/spatchcache\n\n-- \n2.37.2.1279.g64dec4e13cf\n\n"},{"id":"461951","messageId":"patch-1.5-8604ac9bfbc-20220825T141212Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-0.5-00000000000-20220825T141212Z-avarab@gmail.com","subject":"[PATCH 1/5] Makefile: add ability to TAB-complete cocci *.patch rules","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-25T14:36:13Z","receivedAt":"2022-08-25T14:36:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Declare the contrib/coccinelle/.cocci.patch rules in such a way\nas to allow TAB-completion, and slightly optimize the Makefile by\ncutting down on the number of $(wildcard) in favor of defining\n\"coccicheck\" and \"coccicheck-pending\" in terms of the same\nincrementally filtered list.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 14 ++++++++++----\n 1 file changed, 10 insertions(+), 4 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex e8adeb09f1c..bd8faac61ee 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3135,9 +3135,15 @@ check: $(GENERATED_H)\n \t\texit 1; \\\n \tfi\n \n+COCCI = $(wildcard contrib/coccinelle/*.cocci)\n+COCCI_PATCHES = $(addsuffix .patch,$(COCCI))\n+COCCICHECK_PENDING = $(filter %.pending.cocci.patch,$(COCCI_PATCHES))\n+COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES))\n+\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n-%.cocci.patch: %.cocci $(COCCI_SOURCES)\n+$(COCCI_PATCHES): $(COCCI_SOURCES)\n+$(COCCI_PATCHES): %.patch: %\n \t$(QUIET_SPATCH) \\\n \tif test $(SPATCH_BATCH_SIZE) = 0; then \\\n \t\tlimit=; \\\n@@ -3174,11 +3180,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell\n coccicheck-test: $(COCCI_TEST_RES_GEN)\n \n coccicheck: coccicheck-test\n-coccicheck: $(addsuffix .patch,$(filter-out %.pending.cocci,$(wildcard contrib/coccinelle/*.cocci)))\n+coccicheck: $(COCCICHECK)\n \n # See contrib/coccinelle/README\n coccicheck-pending: coccicheck-test\n-coccicheck-pending: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.pending.cocci))\n+coccicheck-pending: $(COCCICHECK_PENDING)\n \n .PHONY: coccicheck coccicheck-pending\n \n@@ -3446,7 +3452,7 @@ profile-clean:\n \n cocciclean:\n \t$(RM) -r .build/contrib/coccinelle\n-\t$(RM) contrib/coccinelle/*.cocci.patch*\n+\t$(RM) $(COCCI_PATCHES)*\n \n clean: profile-clean coverage-clean cocciclean\n \t$(RM) -r .build\n-- \n2.37.2.1279.g64dec4e13cf\n\n"},{"id":"461952","messageId":"patch-2.5-db3b3427668-20220825T141212Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-0.5-00000000000-20220825T141212Z-avarab@gmail.com","subject":"[PATCH 2/5] Makefile: have \"coccicheck\" re-run if flags change","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-25T14:36:14Z","receivedAt":"2022-08-25T14:36:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix an issue with the \"coccicheck\" family of rules that's been here\nsince 63f0a758a06 (add coccicheck make target, 2016-09-15), unlike\ne.g. \"make grep.o\" we wouldn't re-run it when $(SPATCH) or\n$(SPATCH_FLAGS) changed. To test new flags we needed to first do a\n\"make clean\".\n\nThis now uses the same (copy/pasted) pattern as other \"DEFINES\"\nrules. As a result we'll re-run properly. This can be demonstrated\ne.g. on the issue noted in [1]:\n\n\t$ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c V=1\n\t[...]\n\t SPATCH contrib/coccinelle/xcalloc.cocci\n\t$ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c SPATCH_FLAGS=\"--all-includes --recursive-includes\"\n\t * new spatch flags\n\t SPATCH contrib/coccinelle/xcalloc.cocci\n\t SPATCH result: contrib/coccinelle/xcalloc.cocci.patch\n\t$\n\n1. https://lore.kernel.org/git/20220823095602.GC1735@szeder.dev/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .gitignore | 1 +\n Makefile | 14 ++++++++++++++\n 2 files changed, 15 insertions(+)\n\ndiff --git a/.gitignore b/.gitignore\nindex 42fd7253b44..a32ad64d4c9 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -10,6 +10,7 @@\n /GIT-PERL-HEADER\n /GIT-PYTHON-VARS\n /GIT-SCRIPT-DEFINES\n+/GIT-SPATCH-DEFINES\n /GIT-USER-AGENT\n /GIT-VERSION-FILE\n /bin-wrappers/\ndiff --git a/Makefile b/Makefile\nindex bd8faac61ee..749eb9b25e3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1296,6 +1296,18 @@ SANITIZE_ADDRESS =\n SPATCH_FLAGS = --all-includes\n SPATCH_BATCH_SIZE = 1\n \n+# Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n+TRACK_SPATCH_DEFINES =\n+TRACK_SPATCH_DEFINES += $(SPATCH)\n+TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n+TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n+GIT-SPATCH-DEFINES: FORCE\n+\t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n+\t if test x\"$$FLAGS\" != x\"`cat GIT-SPATCH-DEFINES 2>/dev/null`\" ; then \\\n+\t\techo >&2 \" * new spatch flags\"; \\\n+\t\techo \"$$FLAGS\" >GIT-SPATCH-DEFINES; \\\n+ fi\n+\n include config.mak.uname\n -include config.mak.autogen\n -include config.mak\n@@ -3142,6 +3154,7 @@ COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES))\n \n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n+$(COCCI_PATCHES): GIT-SPATCH-DEFINES\n $(COCCI_PATCHES): $(COCCI_SOURCES)\n $(COCCI_PATCHES): %.patch: %\n \t$(QUIET_SPATCH) \\\n@@ -3165,6 +3178,7 @@ $(COCCI_PATCHES): %.patch: %\n \tfi\n \n COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n+$(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n-- \n2.37.2.1279.g64dec4e13cf\n\n"},{"id":"461953","messageId":"patch-4.5-cda20790146-20220825T141212Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-0.5-00000000000-20220825T141212Z-avarab@gmail.com","subject":"[PATCH 4/5] cocci: make incremental compilation even faster","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-25T14:36:16Z","receivedAt":"2022-08-25T14:36:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the incremental compilation of \"make coccicheck\" even faster, at\nthe cost of losing dependency discovery for a small whitelist of *.c\nfiles which we can't create *.o files for.\n\nThis will break obscure edge cases since e.g.:\n\n time make -W block-sha1/sha1.h coccicheck\n\nWould correctly re-build before, as block-sha1/sha1.c uses\nblock-sha1/sha1.h, but as it and these other\n$(USE_TINY_FOUND_H_SOURCES) files depended on the union of all such\nheaders (and more, all of $(FOUND_H_SOURCES)) the above command would\ntake ~10 seconds before.\n\nAs noted in the preceding commit these obscure cases were broken\nbefore the preceding commit fixed them. I think it makes sense to make\na correctness v.s. performance trade-off here. By doing this it\nbecomes viable to e.g. run \"make coccicheck\" in the same loop that\nmerges various topic branches down (along with \"ccache\" sped-up\ncompilation).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 12 +++++++++++-\n 1 file changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex a374204aaca..9410a587fc0 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3160,6 +3160,16 @@ COCCICHECK_OK = $(COCCICHECK:%=.build/%.ok)\n COCCICHECK_PENDING_OK = $(COCCICHECK_PENDING:%=.build/%.ok)\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n+USE_TINY_FOUND_H_SOURCES =\n+USE_TINY_FOUND_H_SOURCES += compat/%\n+USE_TINY_FOUND_H_SOURCES += block-sha1/%\n+USE_TINY_FOUND_H_SOURCES += ppc/%\n+\n+TINY_FOUND_H_SOURCES =\n+TINY_FOUND_H_SOURCES += cache.h\n+TINY_FOUND_H_SOURCES += git-compat-util.h\n+TINY_FOUND_H_SOURCES += strbuf.h\n+\n .build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES)\n \t$(call mkdir_p_parent_template)\n \t$(QUIET_GEN) >$@\n@@ -3171,7 +3181,7 @@ define cocci-rule\n # 2 = $(2)\n COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n .build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n-.build/$(1).patch/$(2): $(if $(wildcard $(3)),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES)\n+.build/$(1).patch/$(2): $(if $(wildcard $(3)),$(3),$(if $(filter $(USE_TINY_FOUND_H_SOURCES),$(3)),$(TINY_FOUND_H_SOURCES),.build/contrib/coccinelle/FOUND_H_SOURCES))\n .build/$(1).patch/$(2): $(1)\n .build/$(1).patch/$(2): .build/$(1).patch/% : %\n \t$$(call mkdir_p_parent_template)\n-- \n2.37.2.1279.g64dec4e13cf\n\n"},{"id":"461954","messageId":"patch-3.5-6fa83695f1f-20220825T141212Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-0.5-00000000000-20220825T141212Z-avarab@gmail.com","subject":"[PATCH 3/5] cocci: make \"coccicheck\" rule incremental","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-25T14:36:15Z","receivedAt":"2022-08-25T14:36:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Optimize the very slow \"coccicheck\" target to take advantage of\nincremental rebuilding, and fix outstanding bugs and dependency\nproblems with the existing rule.\n\nThe rule is now faster both on the initial run as we can make better\nuse of GNU make's parallelism than the old ad-hoc combination of\nmake's parallelism combined with $(SPATCH_BATCH_SIZE) and \"xargs -n\".\n\nIt also makes us *much* faster when incrementally building, it's now\nviable to \"make coccicheck\" as topic branches are merged down.\n\nThe rule didn't use FORCE (or its equivalents) before, so a:\n\n\tmake coccicheck\n\tmake coccicheck\n\nWould report nothing to do on the second iteration. But all of our\npatch output depended on all $(COCCI_SOURCES) files, therefore e.g.:\n\n make -W grep.c coccicheck\n\nWould be the same as doing a full re-run. A change in a single file\nwould force us to do a full re-run.\n\nThe reason for this (not the initial rationale, but my analysis) is:\n\n* Since we create a single \"*.cocci.patch+\" we don't know where to\n pick up where we left off. Note that (per [1]) we've had a race\n condition since 960154b9c17 (coccicheck: optionally batch spatch\n invocations, 2019-05-06) which might result in us producing corrupt\n patches to to the concurrent appending to \"$@+\" in the pre-image.\n\n That bug is now fixed.\n\n* We've been carrying forward the dependency on the *.c files since\n 63f0a758a06 (add coccicheck make target, 2016-09-15) the rule was\n initially added as a sort of poor man's dependency discovery.\n\n As we don't include other *.c files depending on other *.c files\n has always been broken, as could be trivially demonstrated\n e.g. with:\n\n make coccicheck\n make -W strbuf.h coccicheck\n\n However, depending on the corresponding *.c files has been doing\n something, namely that *if* an API change modified both *.c and *.h\n files we'd catch the change to the *.h we care about via the *.c\n being changed.\n\n For API changes that happened only via *.h files we'd do the wrong\n thing before this change, but e.g. for function additions (not\n \"static inline\" ones) catch the *.h change by proxy.\n\nNow we'll instead:\n\n * Create a / pair in the .build directory, E.g. for\n swap.cocci and grep.c we'll create\n .build/contrib/coccinelle/swap.cocci.patch/grep.c.\n\n That file is the diff we'll apply for that -\n combination, if there's no changes to me made (the common case)\n it'll be an empty file.\n\n * Our generated *.patch\n file (e.g. contrib/coccinelle/swap.cocci.patch) is now a simple \"cat\n $^\" of all of all of the / files for a given .\n\n See 1cc0425a27c (Makefile: have \"make pot\" not \"reset --hard\",\n 2022-05-26) for another recent rule that used that technique.\n\n * We depend on a .build/.ok file, which in turn depends on our\n . This emulates the previous \"test -s\" test, but in a more\n Makefile-native way (e.g. -k works as expected).\n\nAs noted above this change fixes existing dependency issues,\ne.g. changing strbuf.h will now trigger a re-build. Our default is to\ndepend on $(FOUND_H_SOURCES). This makes incremental re-building\npossible, but it's still fairly slow, as modifying one header will\ncause a full re-build of everything.\n\nWhich is why we'll not depend on $(FOUND_H_SOURCES) but the *.o file\ncorresponding to the *.c file, if it exists already. This means that\nwe can do:\n\n make all\n make coccicheck\n make -W column.h coccicheck\n\nBy depending on the *.o we piggy-back on\nCOMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the\n\"sparse\" target non-.PHONY, 2021-09-23) for prior art of doing that\nfor the *.sp files. E.g.:\n\n make contrib/coccinelle/free.cocci.patch\n make -W column.h contrib/coccinelle/free.cocci.patch\n\nWill take around 15 seconds for the second command on my 8 core box if\nI didn't run \"make\" beforehand to create the *.o files. But around 2\nseconds if I did and we have those \"*.o\" files.\n\nNotes about the approach of piggy-backing on *.o for dependencies:\n\n* It *is* a trade-off since we'll pay the extra cost of running the C\n compiler, but we're probably doing that anyway. The compiler is much\n faster than \"spatch\", so even though we need to re-compile the *.o to\n create the dependency info for the *.c for \"spatch\" it's\n faster (especially if using \"ccache\").\n\n * If that *.o file doesn't exist we'll depend on an intermediate file\n of ours which in turn depends on $(FOUND_H_SOURCES).\n\n This covers both an initial build, or where \"coccicheck\" is run\n without running \"all\" beforehand, and because we run \"coccicheck\"\n on e.g. files in compat/* that we don't know how to build unless\n the requisite flag was provided to the Makefile.\n\n Most of the runtime of \"incremental\" runs is now spent on various\n compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and\n therefore conflate whether we *can* compile an object and generate\n dependency information for it with whether we'd like to link it\n into our binary.\n\n Before this change the distinction didn't matter, but now one way\n to make this even faster on incremental builds would be to peel\n those concerns apart so that we can see that e.g. compat/mmap.c\n doesn't depend on column.h.\n\nFurther implementation details & notes:\n\n * We can take better advantage of parallelism, while making sure that\n we don't racily append to the contrib/coccinelle/swap.cocci.patch\n file from multiple workers.\n\n Before this change running \"make coccicheck\" would by default end\n up pegging just one CPU at the very end for a while, usually as\n we'd finish whichever *.cocci rule was the most expensive.\n\n This could be mitigated by combining \"make -jN\" with\n SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch\n spatch invocations, 2019-05-06). But doing so required careful\n juggling, as e.g. setting both to 4 would yield 16 workers.\n\n There will be cases where getting rid of \"SPATCH_BATCH_SIZE\" makes\n things worse, but a from-scratch \"make coccicheck\" with the default\n of SPATCH_BATCH_SIZE=1 (and tweaking it doesn't make a difference)\n is faster (~3m36s v.s. ~3m56s) with this approach, as we can feed\n the CPU more work in a less staggered way.\n\n * Getting rid of \"SPATCH_BATCH_SIZE\" particularly helps in cases\n where the default of 1 yields parallelism under \"make coccicheck\",\n but then running e.g.:\n\n make -W contrib/coccinelle/swap.cocci coccicheck\n\n I.e. before that would use only one CPU core, until the user\n remembered to adjust \"SPATCH_BATCH_SIZE\" differently than the\n setting that makes sense when doing a non-incremental run of \"make\n coccicheck\".\n\n * The change here to\n \"contrib/coccinelle/the_repository.pending.cocci\" is because we've\n been spewing out this warning into our $@.log since that rule was\n added:\n\n warning: rule starting on line 21: metavariable F not used in the - or context code\n\n It's being fixed here because we'll now notice on \"make\n coccicheck-pending\", since we're not redirecting all of STDERR to a\n log file, instead we use the \"--very-quiet\" option.\n\nOutstanding issues & future work:\n\n * We could get rid of \"--all-includes\" in favor of manually\n specifying a list of includes to give to \"spatch(1)\".\n\n As noted upthread of [2] a naïve removal of \"--all-includes\" will\n result in broken *.cocci patches, but if we know the exhaustive\n list of includes via COMPUTE_HEADER_DEPENDENCIES we don't need to\n re-scan for them, we could grab the headers to include from the\n .depend.d/.o.d and supply them with the \"--include\" option to\n spatch(1).q\n\n * We can generalize COMPUTE_HEADER_DEPENDENCIES to spew out the list\n of *.h dependencies without making a *.o object, which is slightly\n faster, and would also benefit the \"make sparse\" target. We'd still\n need to run the compiler, it would just have less work to do.\n\n1. https://lore.kernel.org/git/877dlwotjc.fsf@evledraar.gmail.com/\n2. https://lore.kernel.org/git/87ft18tcog.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 92 +++++++++++++------\n .../coccinelle/the_repository.pending.cocci | 1 -\n shared.mak | 4 +-\n 3 files changed, 66 insertions(+), 31 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 749eb9b25e3..a374204aaca 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1290,17 +1290,12 @@ SP_EXTRA_FLAGS = -Wno-universal-initializer\n SANITIZE_LEAK =\n SANITIZE_ADDRESS =\n \n-# For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n-# usually result in less CPU usage at the cost of higher peak memory.\n-# Setting it to 0 will feed all files in a single spatch invocation.\n SPATCH_FLAGS = --all-includes\n-SPATCH_BATCH_SIZE = 1\n \n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n TRACK_SPATCH_DEFINES += $(SPATCH)\n TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n-TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n GIT-SPATCH-DEFINES: FORCE\n \t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n \t if test x\"$$FLAGS\" != x\"`cat GIT-SPATCH-DEFINES 2>/dev/null`\" ; then \\\n@@ -3147,35 +3142,74 @@ check: $(GENERATED_H)\n \t\texit 1; \\\n \tfi\n \n+# It's expensive to compute the many=many rules below, only eval them\n+# on $(MAKECMDGOALS) that match these $(COCCI_RULES)\n+COCCI_RULES =\n+COCCI_RULES += cocci%\n+COCCI_RULES += contrib/coccinelle/%\n+COCCI_RULES += .build/contrib/coccinelle/%\n+\n COCCI = $(wildcard contrib/coccinelle/*.cocci)\n+COCCI_NAMES = $(COCCI:contrib/coccinelle/%.cocci=%)\n COCCI_PATCHES = $(addsuffix .patch,$(COCCI))\n+COCCI_PATCHES_OK = $(COCCI_PATCHES:%=.build/%.ok)\n COCCICHECK_PENDING = $(filter %.pending.cocci.patch,$(COCCI_PATCHES))\n-COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES))\n \n+COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES))\n+COCCICHECK_OK = $(COCCICHECK:%=.build/%.ok)\n+COCCICHECK_PENDING_OK = $(COCCICHECK_PENDING:%=.build/%.ok)\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n-$(COCCI_PATCHES): GIT-SPATCH-DEFINES\n-$(COCCI_PATCHES): $(COCCI_SOURCES)\n-$(COCCI_PATCHES): %.patch: %\n-\t$(QUIET_SPATCH) \\\n-\tif test $(SPATCH_BATCH_SIZE) = 0; then \\\n-\t\tlimit=; \\\n-\telse \\\n-\t\tlimit='-n $(SPATCH_BATCH_SIZE)'; \\\n-\tfi; \\\n-\tif ! echo $(COCCI_SOURCES) | xargs $$limit \\\n-\t\t$(SPATCH) $(SPATCH_FLAGS) \\\n-\t\t--sp-file $< --patch . \\\n-\t\t>$@+ 2>$@.log; \\\n+.build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES)\n+\t$(call mkdir_p_parent_template)\n+\t$(QUIET_GEN) >$@\n+\n+define cocci-rule\n+\n+## Rule for .build/$(1).patch/$(2); Params:\n+# 1 = $(1)\n+# 2 = $(2)\n+COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n+.build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n+.build/$(1).patch/$(2): $(if $(wildcard $(3)),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES)\n+.build/$(1).patch/$(2): $(1)\n+.build/$(1).patch/$(2): .build/$(1).patch/% : %\n+\t$$(call mkdir_p_parent_template)\n+\t$$(QUIET_SPATCH)$$(SPATCH) $$(SPATCH_FLAGS) \\\n+\t\t--very-quiet \\\n+\t\t--sp-file $(1) --patch . \\\n+\t\t$$(SPATCH_PATCH_FLAGS) $$< >$$@\n+endef\n+\n+define cocci-matrix-2\n+\n+$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o)))\n+endef\n+define cocci-matrix-1\n+$(foreach c,$(COCCI),$(call cocci-matrix-2,$(c)))\n+endef\n+\n+ifneq ($(filter $(COCCI_RULES),$(MAKECMDGOALS)),)\n+$(eval $(call cocci-matrix-1))\n+endif\n+\n+define spatch-rule\n+\n+contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1))\n+\t$(QUIET_SPATCH_M)cat $$^ >$$@\n+endef\n+\n+$(COCCI_PATCHES_OK): .build/%.ok : %\n+\t$(QUIET_SPATCH_R)if test -s $<; \\\n \tthen \\\n-\t\tcat $@.log; \\\n+\t\tcat $< | sed -e 's/^/\t/'; \\\n \t\texit 1; \\\n-\tfi; \\\n-\tmv $@+ $@; \\\n-\tif test -s $@; \\\n-\tthen \\\n-\t\techo ' ' SPATCH result: $@; \\\n-\tfi\n+\tfi && \\\n+\t>$@\n+\n+ifneq ($(filter $(COCCI_RULES),$(MAKECMDGOALS)),)\n+$(eval $(foreach n,$(COCCI_NAMES),$(call spatch-rule,$(n))))\n+endif\n \n COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n@@ -3194,11 +3228,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell\n coccicheck-test: $(COCCI_TEST_RES_GEN)\n \n coccicheck: coccicheck-test\n-coccicheck: $(COCCICHECK)\n+coccicheck: $(COCCICHECK_OK)\n \n # See contrib/coccinelle/README\n coccicheck-pending: coccicheck-test\n-coccicheck-pending: $(COCCICHECK_PENDING)\n+coccicheck-pending: $(COCCICHECK_PENDING_OK)\n \n .PHONY: coccicheck coccicheck-pending\n \n@@ -3466,7 +3500,7 @@ profile-clean:\n \n cocciclean:\n \t$(RM) -r .build/contrib/coccinelle\n-\t$(RM) $(COCCI_PATCHES)*\n+\t$(RM) $(COCCI_PATCHES)\n \n clean: profile-clean coverage-clean cocciclean\n \t$(RM) -r .build\ndiff --git a/contrib/coccinelle/the_repository.pending.cocci b/contrib/coccinelle/the_repository.pending.cocci\nindex 072ea0d9228..747d382ff5f 100644\n--- a/contrib/coccinelle/the_repository.pending.cocci\n+++ b/contrib/coccinelle/the_repository.pending.cocci\n@@ -20,7 +20,6 @@ expression E;\n \n @@\n expression E;\n-expression F;\n @@\n - has_object_file_with_flags(\n + repo_has_object_file_with_flags(the_repository,\ndiff --git a/shared.mak b/shared.mak\nindex 33f43edbf9a..c0e88308069 100644\n--- a/shared.mak\n+++ b/shared.mak\n@@ -69,8 +69,10 @@ ifndef V\n \tQUIET_SP = @echo ' ' SP $<;\n \tQUIET_HDR = @echo ' ' HDR $(<:hcc=h);\n \tQUIET_RC = @echo ' ' RC $@;\n-\tQUIET_SPATCH = @echo ' ' SPATCH $<;\n+\tQUIET_SPATCH = @echo ' ' SPATCH $@;\n \tQUIET_SPATCH_T = @echo ' ' SPATCH TEST $(@:.build/%=%);\n+\tQUIET_SPATCH_M = @echo ' ' SPATCH MERGE $$@;\n+\tQUIET_SPATCH_R = @echo ' ' SPATCH RESULT $<;\n \n ## Used in \"Documentation/Makefile\"\n \tQUIET_ASCIIDOC\t= @echo ' ' ASCIIDOC $@;\n-- \n2.37.2.1279.g64dec4e13cf\n\n"},{"id":"461955","messageId":"patch-5.5-ce4734e5d79-20220825T141212Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-0.5-00000000000-20220825T141212Z-avarab@gmail.com","subject":"[PATCH 5/5] spatchcache: add a ccache-alike for \"spatch\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-25T14:36:17Z","receivedAt":"2022-08-25T14:36:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a rather trivial \"spatchcache\", with this running e.g.:\n\n\tmake clean\n\tmake contrib/coccinelle/free.cocci.patch\n\nIs cut down from ~20s to ~3s on my system (and much of it is fixable\nshell overhead). This uses \"redis\" as a cache by default, but it's\nconfigurable. See the embedded documentation.\n\nThis is *not* like ccache in that we won't cache failed spatch\ninvocations, or those where spatch suggests changes for us. Those\ncases are so rare that I didn't think it was worth the bother, by far\nthe most common case is that it has no suggested changes.\n\nBecause we narrow the cache to that we don't need to save away stdout,\nstderr & the exit code. We simply cache the cases where we had no\nsuggested changes.\n\nAnother benchmark is to compare this with the previous\nSPATCH_BATCH_SIZE=N, as noted in [1]. Before this (on my 8 core system) running:\n\n\tmake clean; time make contrib/coccinelle/array.cocci.patch SPATCH_BATCH_SIZE=0\n\nWould take 33s, but with the preceding changes running without this\n\"spatchcache\" is slightly slower, or around 35s:\n\n\tmake clean; time make contrib/coccinelle/array.cocci.patch\n\nNow doing the same with SPATCH=contrib/coccinelle/spatchcache will\ntake around 6s, but we'll need to compile the *.o files first to take\nfull advantage of it (which can be fast with \"ccache\"):\n\n\tmake clean; make; time make contrib/coccinelle/array.cocci.patch SPATCH=contrib/coccinelle/spatchcache\n\n1. https://lore.kernel.org/git/YwdRqP1CyUAzCEn2@coredump.intra.peff.net/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n contrib/coccinelle/spatchcache | 217 +++++++++++++++++++++++++++++++++\n 1 file changed, 217 insertions(+)\n create mode 100755 contrib/coccinelle/spatchcache\n\ndiff --git a/contrib/coccinelle/spatchcache b/contrib/coccinelle/spatchcache\nnew file mode 100755\nindex 00000000000..f4f44700cfc\n--- /dev/null\n+++ b/contrib/coccinelle/spatchcache\n@@ -0,0 +1,217 @@\n+#!/bin/sh\n+#\n+# spatchcache: a poor-man's \"ccache\"-alike for \"spatch\" in git.git\n+#\n+# This caching command relies on the peculiarities of the Makefile\n+# driving \"spatch\" in git.git, in particular if we invoke:\n+#\n+#\tmake\n+#\tmake coccicheck\n+#\n+# We can with COMPUTE_HEADER_DEPENDENCIES (auto-detected as true with\n+# \"gcc\" and \"clang\") write e.g. a .depend/grep.o.d for grep.c, when we\n+# compile grep.o.\n+#\n+# The .depend/grep.o.d will have the full header dependency tree of\n+# grep.c, and we can thus cache the output of \"spatch\" by:\n+#\n+#\t1. Hashing all of those files\n+#\t2. Hashing our source file, and the *.cocci rule we're\n+#\t applying\n+#\t3. Running spatch, if suggests no changes (by far the common\n+#\t case) we invoke \"spatchCache.getCmd\" and\n+#\t \"spatchCache.setCmd\" with a hash SHA-256 to ask \"does this\n+#\t ID have no changes\" or \"say that ID had no changes>\n+#\t4. If no \"spatchCache.{set,get}Cmd\" is specified we'll use\n+#\t \"redis-cli\" and maintain a SET called \"spatch-cache\". Set\n+#\t appropriate redis memory policies to keep it from growing\n+#\t out of control.\n+#\n+# This along with the general incremental \"make\" support for\n+# \"contrib/coccinelle\" makes it viable to (re-)run coccicheck\n+# e.g. when merging integration branches.\n+#\n+# To use this, simply set SPATCH to\n+# contrib/coccinelle/spatchcache. Then optionally set:\n+#\n+#\t[spatchCache]\n+#\t\t# path to a custom spatch\n+#\t\tspatch = ~/g/coccicheck/spatch.opt\n+#\n+# As well as this trace config (debug implies trace):\n+#\n+#\t\ttrace = false\n+#\t\tdebug = false\n+#\n+# Setting \"trace\" to \"true\" allows for seeing when we have a cache HIT\n+# or MISS.\n+\n+set -e\n+\n+## Our own configuration & options\n+debug=$(git config --bool \"spatchCache.debug\")\n+if test \"$debug\" != \"true\"\n+then\n+\tdebug=\n+fi\n+if test -n \"$debug\"\n+then\n+\tset -x\n+fi\n+\n+trace=$(git config --bool \"spatchCache.trace\")\n+if test \"$trace\" != \"true\"\n+then\n+\ttrace=\n+fi\n+if test -n \"$debug\"\n+then\n+\t# debug implies trace\n+\ttrace=true\n+fi\n+\n+trace_it () {\n+\tif test -z \"$trace\"\n+\tthen\n+\t\treturn\n+\tfi\n+\techo \"$@\" >&2\n+}\n+\n+spatch=$(git config --path \"spatchCache.spatch\" || :)\n+if test -n \"$spatch\"\n+then\n+\tif test -n \"$debug\"\n+\tthen\n+\t\ttrace_it \"custom spatchCache.spatch='$spatch'\"\n+\tfi\n+else\n+\tspatch=spatch\n+fi\n+\n+set=$(git config spatchCache.setCmd || :)\n+get=$(git config spatchCache.getCmd || :)\n+\n+## Parse spatch()-like command-line for caching info\n+arg_sp=\n+arg_file=\n+args=\"$@\"\n+spatch_opts() {\n+\twhile test $# != 0\n+\tdo\n+\t\targ_file=\"$1\"\n+\t\tcase \"$1\" in\n+\t\t--sp-file)\n+\t\t\targ_sp=\"$2\"\n+\t\t\t;;\n+\t\tesac\n+\t\tshift\n+\tdone\n+}\n+spatch_opts \"$@\"\n+if ! test -f \"$arg_file\"\n+then\n+\targ_file=\n+fi\n+\n+hash_for_cache() {\n+\t# Parameters that should affect the cache\n+\techo \"spatch=$spatch\"\n+\techo \"args=$args\"\n+\techo\n+\n+\t# Our target file and its dependencies\n+\tgit hash-object \"$1\" \"$2\" $(grep -E -o '^[^:]+:$' \"$3\" | tr -d ':')\n+}\n+\n+# Sanity checks\n+if ! test -f \"$arg_sp\" && ! test -f \"$arg_file\"\n+then\n+\techo $0: no idea how to cache \"$@\" >&2\n+\texit 128\n+fi\n+\n+# Main logic\n+d=$(dirname \"$arg_file\")\n+b=$(basename \"$arg_file\")\n+bwoext=\"${b%.c}\"\n+dep=\"$d/.depend/$bwoext.o.d\"\n+\n+if ! test -f \"$dep\"\n+then\n+\ttrace_it \"$0: have no '$dep' for '$arg_file', can't cache!\"\n+\texec \"$spatch\" \"$@\"\n+fi\n+\n+if test -n \"$debug\"\n+then\n+\ttrace_it \"$0: The full cache input for '$arg_sp' '$arg_file' '$dep'\"\n+\thash_for_cache \"$arg_sp\" \"$arg_file\" \"$dep\" >&2\n+fi\n+sum=$(hash_for_cache \"$arg_sp\" \"$arg_file\" \"$dep\" | git hash-object --stdin)\n+\n+trace_it \"$0: processing '$arg_file' with '$arg_sp' rule, and got hash '$sum' for it + '$dep'\"\n+\n+getret=\n+if test -z \"$get\"\n+then\n+\tif test $(redis-cli SISMEMBER spatch-cache \"$sum\") = 1\n+\tthen\n+\t\tgetret=0\n+\telse\n+\t\tgetret=1\n+\tfi\n+else\n+\t$set \"$sum\"\n+\tgetret=$?\n+fi\n+\n+if test \"$getret\" = 0\n+then\n+\ttrace_it \"$0: HIT for '$arg_file' with '$arg_sp'\"\n+\texit 0\n+else\n+\ttrace_it \"$0: MISS: for '$arg_file' with '$arg_sp'\"\n+fi\n+\n+out=\"$(mktemp)\"\n+err=\"$(mktemp)\"\n+\n+set +e\n+\"$spatch\" \"$@\" >\"$out\" 2>\"$err\"\n+ret=$?\n+set -e\n+\n+if test $ret = 0 && ! test -s \"$out\" && ! test -s \"$err\"\n+then\n+\trm -f \"$out\" \"$err\"\n+\n+\ttrace_it \"$0: SET: for '$arg_file' with '$arg_sp'\"\n+\n+\tsetret=\n+\tif test -z \"$set\"\n+\tthen\n+\t\tif test $(redis-cli SADD spatch-cache \"$sum\") = 1\n+\t\tthen\n+\t\t\tsetret=0\n+\t\telse\n+\t\t\tsetret=1\n+\t\tfi\n+\telse\n+\t\t\"$set\" \"$sum\"\n+\t\tsetret=$?\n+\tfi\n+\n+\tif test \"$setret\" != 0\n+\tthen\n+\t\techo \"FAILED to set '$sum' in cache!\" >&2\n+\t\texit 128\n+\tfi\n+else\n+\ttrace_it \"$0: NO CACHE (have changes): for '$arg_file' with '$arg_sp'\"\n+\tcat \"$out\"\n+\tcat \"$err\" >&2\n+\trm -f \"$out\" \"$err\"\n+\texit \"$ret\"\n+fi\n+rm -f \"$out\" \"$err\"\n-- \n2.37.2.1279.g64dec4e13cf\n\n"},{"id":"461957","messageId":"20220825152916.GH1735@szeder.dev","threadId":"58360","inReplyTo":"patch-2.5-db3b3427668-20220825T141212Z-avarab@gmail.com","subject":"Re: [PATCH 2/5] Makefile: have \"coccicheck\" re-run if flags change","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2022-08-25T15:29:16Z","receivedAt":"2022-08-25T15:30:22Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Aug 25, 2022 at 04:36:14PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Fix an issue with the \"coccicheck\" family of rules that's been here\n> since 63f0a758a06 (add coccicheck make target, 2016-09-15), unlike\n> e.g. \"make grep.o\" we wouldn't re-run it when $(SPATCH) or\n> $(SPATCH_FLAGS) changed. To test new flags we needed to first do a\n> \"make clean\".\n\nNit: make cocciclean\n\n"},{"id":"461967","messageId":"20220825194418.GI1735@szeder.dev","threadId":"58360","inReplyTo":"patch-3.5-6fa83695f1f-20220825T141212Z-avarab@gmail.com","subject":"Re: [PATCH 3/5] cocci: make \"coccicheck\" rule incremental","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2022-08-25T19:44:18Z","receivedAt":"2022-08-25T19:44:25Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Aug 25, 2022 at 04:36:15PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> * Since we create a single \"*.cocci.patch+\" we don't know where to\n> pick up where we left off. Note that (per [1]) we've had a race\n> condition since 960154b9c17 (coccicheck: optionally batch spatch\n> invocations, 2019-05-06) which might result in us producing corrupt\n> patches to to the concurrent appending to \"$@+\" in the pre-image.\n> \n> That bug is now fixed.\n\nThere is no bug, because there is no concurrent appending to \"$@+\".\nThe message you mention seems to be irrelevant, as it talks about\n'xargs -P', but the invocation in '%.cocci.patch' targets never used\n'-P'.\n\n\n> Which is why we'll not depend on $(FOUND_H_SOURCES) but the *.o file\n> corresponding to the *.c file, if it exists already. This means that\n> we can do:\n> \n> make all\n> make coccicheck\n> make -W column.h coccicheck\n> \n> By depending on the *.o we piggy-back on\n> COMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the\n> \"sparse\" target non-.PHONY, 2021-09-23) for prior art of doing that\n> for the *.sp files. E.g.:\n> \n> make contrib/coccinelle/free.cocci.patch\n> make -W column.h contrib/coccinelle/free.cocci.patch\n> \n> Will take around 15 seconds for the second command on my 8 core box if\n> I didn't run \"make\" beforehand to create the *.o files. But around 2\n> seconds if I did and we have those \"*.o\" files.\n> \n> Notes about the approach of piggy-backing on *.o for dependencies:\n> \n> * It *is* a trade-off since we'll pay the extra cost of running the C\n> compiler, but we're probably doing that anyway.\n\nThis assumption doesn't hold, and I very much dislike the idea of\ndepending on *.o files:\n\n - Our static-analysis CI job doesn't build Git, now it will have to.\n\n - I don't have Coccinelle installed, because my distro doesn't ship\n it, and though the previous release did ship it, it was outdated.\n Instead I use Coccinelle's most recent version from a container\n which doesn't contain any build tools apart from 'make' for 'make\n coccicheck'.\n\n With this patch series I can't use this containerized Coccinelle\n at all, because even though I've already built git on the host,\n the dependency on *.o files triggers a BUILD-OPTIONS check during\n 'make coccicheck', and due to the missing 'curl-config' the build\n options do differ, triggering a rebuild, which in the absence of a\n compiler fails.\n\n And then the next 'make' on the host will have to rebuild\n everything again...\n\n\n> * We can take better advantage of parallelism, while making sure that\n> we don't racily append to the contrib/coccinelle/swap.cocci.patch\n> file from multiple workers.\n> \n> Before this change running \"make coccicheck\" would by default end\n> up pegging just one CPU at the very end for a while, usually as\n> we'd finish whichever *.cocci rule was the most expensive.\n> \n> This could be mitigated by combining \"make -jN\" with\n> SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch\n> spatch invocations, 2019-05-06). But doing so required careful\n> juggling, as e.g. setting both to 4 would yield 16 workers.\n\nNo, setting both to 4 does yield 4 workers.\n\nSPATCH_BATCH_SIZE has nothing to do with parallelism; it is merely the\nnumber of C source files that we pass to a single 'spatch' invocation,\nbut for any given semantic patch it's still a sequential loop.\n\n"},{"id":"461971","messageId":"220826.861qt43pcc.gmgdl@evledraar.gmail.com","threadId":"58360","inReplyTo":"20220825194418.GI1735@szeder.dev","subject":"Re: [PATCH 3/5] cocci: make \"coccicheck\" rule incremental","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-25T22:18:28Z","receivedAt":"2022-08-25T22:55:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Aug 25 2022, SZEDER Gábor wrote:\n\nThanks for taking a look!\n\n> On Thu, Aug 25, 2022 at 04:36:15PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> * Since we create a single \"*.cocci.patch+\" we don't know where to\n>> pick up where we left off. Note that (per [1]) we've had a race\n>> condition since 960154b9c17 (coccicheck: optionally batch spatch\n>> invocations, 2019-05-06) which might result in us producing corrupt\n>> patches to to the concurrent appending to \"$@+\" in the pre-image.\n>> \n>> That bug is now fixed.\n>\n> There is no bug, because there is no concurrent appending to \"$@+\".\n> The message you mention seems to be irrelevant, as it talks about\n> 'xargs -P', but the invocation in '%.cocci.patch' targets never used\n> '-P'.\n\nI think this is just confusing, I'll amend/rephrase.\n\nAnd at this point I honestly can't remember if I'm conflating this with\nan issue with my earlier proposed series here (I drafted this a while\nago), or a rather obscure \"hidden\" feature that I did use to speed up\ncoccicheck for myself for a while. Which is that you can do e.g.:\n\n\tmake coccicheck SPATCH_BATCH_SIZE=\"8 -P 8\"\n\nWhich will invoke xargs with the \"-P 8\" option in batches of 8 files.\n\nThat's never a thing that 960154b9c17 expected, and it only worked as an\naccident, but it *would* work, and unless you got unlucky with the races\ninvolved would generally speed up your coccicheck.\n\n>> Which is why we'll not depend on $(FOUND_H_SOURCES) but the *.o file\n>> corresponding to the *.c file, if it exists already. This means that\n>> we can do:\n>> \n>> make all\n>> make coccicheck\n>> make -W column.h coccicheck\n>> \n>> By depending on the *.o we piggy-back on\n>> COMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the\n>> \"sparse\" target non-.PHONY, 2021-09-23) for prior art of doing that\n>> for the *.sp files. E.g.:\n>> \n>> make contrib/coccinelle/free.cocci.patch\n>> make -W column.h contrib/coccinelle/free.cocci.patch\n>> \n>> Will take around 15 seconds for the second command on my 8 core box if\n>> I didn't run \"make\" beforehand to create the *.o files. But around 2\n>> seconds if I did and we have those \"*.o\" files.\n>> \n>> Notes about the approach of piggy-backing on *.o for dependencies:\n>> \n>> * It *is* a trade-off since we'll pay the extra cost of running the C\n>> compiler, but we're probably doing that anyway.\n>\n> This assumption doesn't hold, and I very much dislike the idea of\n> depending on *.o files:\n\nIt's my fault for not calling this out more explicitly, but it\n*optionally* depends on the *.o files, but if you don't have them\ncompiled already a \"make coccicheck\" will just use \"spatch\", and nothing\nelse.\n\nSee the CI run/output for this series:\nhttps://github.com/avar/git/runs/8017916844?check_suite_focus=true\n\n> - Our static-analysis CI job doesn't build Git, now it will have to.\n\nAside from this series which doesn't change how it works, maintaining\nthis doesn't seem important to me. E.g. as I noted in [1] coccinelle\nwill happily run on code that doesn't even compile.\n\nSo running the compiler during \"static-analysis\" (or another step that\nit would depend on, maybe \"pedantic\" or \"sparse\") seems like a good (but\nseparate change).\n\nYou don't want to wonder about odd coccinelle output, only to see it's\ntrying to make sense of C source that doesn't even compile.\n\n> - I don't have Coccinelle installed, because my distro doesn't ship\n> it, and though the previous release did ship it, it was outdated.\n> Instead I use Coccinelle's most recent version from a container\n> which doesn't contain any build tools apart from 'make' for 'make\n> coccicheck'.\n>\n> With this patch series I can't use this containerized Coccinelle\n> at all, because even though I've already built git on the host,\n> the dependency on *.o files triggers a BUILD-OPTIONS check during\n> 'make coccicheck', and due to the missing 'curl-config' the build\n> options do differ, triggering a rebuild, which in the absence of a\n> compiler fails.\n>\n> And then the next 'make' on the host will have to rebuild\n> everything again...\n\nThere may be some odd interaction here, but it's unclear if you've\nactually tried to do this with this series, because unless I've missed\nsome edge case this should all still work, per the above.\n\nWhat *won't* work is avoiding potential re-compilation of *.o files if\nyou *have them already* when you run \"make coccicheck\". I'm not familiar\nwith this type of setup, are you saying you're running \"make coccicheck\"\non a working directory that already has *.o files, but you want it to\nignore the *.o?\n\nThat could easily be made optional, but I just assumed that nobody would\ncare. If you want that can you try this on top and see if it works for\nyou?:\n\t\n\tdiff --git a/Makefile b/Makefile\n\tindex 9410a587fc0..11d83c490b4 100644\n\t--- a/Makefile\n\t+++ b/Makefile\n\t@@ -3174,6 +3174,11 @@ TINY_FOUND_H_SOURCES += strbuf.h\n\t \t$(call mkdir_p_parent_template)\n\t \t$(QUIET_GEN) >$@\n\t \n\t+SPATCH_USE_O_DEPENDENCIES = yes\n\t+ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no)\n\t+SPATCH_USE_O_DEPENDENCIES =\n\t+endif\n\t+\n\t define cocci-rule\n\t \n\t ## Rule for .build/$(1).patch/$(2); Params:\n\t@@ -3181,7 +3186,7 @@ define cocci-rule\n\t # 2 = $(2)\n\t COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n\t .build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n\t-.build/$(1).patch/$(2): $(if $(wildcard $(3)),$(3),$(if $(filter $(USE_TINY_FOUND_H_SOURCES),$(3)),$(TINY_FOUND_H_SOURCES),.build/contrib/coccinelle/FOUND_H_SOURCES))\n\t+.build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),$(if $(filter $(USE_TINY_FOUND_H_SOURCES),$(3)),$(TINY_FOUND_H_SOURCES),.build/contrib/coccinelle/FOUND_H_SOURCES))\n\t .build/$(1).patch/$(2): $(1)\n\t .build/$(1).patch/$(2): .build/$(1).patch/% : %\n\t \t$$(call mkdir_p_parent_template)\n\nI'd need to split up the already long line, but with *.o files compiled\nand:\n\t\n\t$ time make -W column.h contrib/coccinelle/free.cocci.patch COMPUTE_HEADER_DEPENDENCIES=yes\n\t CC wt-status.o\n\t CC builtin/branch.o\n\t CC builtin/clean.o\n\t CC builtin/column.o\n\t CC builtin/commit.o\n\t CC builtin/tag.o\n\t CC column.o\n\t CC help.o\n\t SPATCH .build/contrib/coccinelle/free.cocci.patch/builtin/commit.c\n\t SPATCH .build/contrib/coccinelle/free.cocci.patch/builtin/tag.c\n\t SPATCH .build/contrib/coccinelle/free.cocci.patch/builtin/column.c\n\t SPATCH .build/contrib/coccinelle/free.cocci.patch/column.c\n\t SPATCH .build/contrib/coccinelle/free.cocci.patch/wt-status.c\n\t SPATCH .build/contrib/coccinelle/free.cocci.patch/builtin/branch.c\n\t SPATCH .build/contrib/coccinelle/free.cocci.patch/builtin/clean.c\n\t SPATCH .build/contrib/coccinelle/free.cocci.patch/help.c\n\t SPATCH MERGE contrib/coccinelle/free.cocci.patch\n\t\n\treal 0m0.550s\n\tuser 0m0.505s\n\tsys 0m0.096s\n\nBut if you set COMPUTE_HEADER_DEPENDENCIES=yes it'll take ~4s (on that\nbox), and well re-apply the free.cocci rule to all the *.c files (and\nthis is all with the caching mechanism in 5/5, real \"spatch\" will be\nmuch slower).\n\nIf you already have git compiled (or partially compiled) the \"happy\npath\" to avoiding work is almost definitely to re-compile that *.o\nbecause the *.c changed, at which point we'll be able to see if we even\nneed to re-run any of the coccinelle rules. Usually we'll need to re-run\na far smaller set than the full set we operate on now.\n\n>> * We can take better advantage of parallelism, while making sure that\n>> we don't racily append to the contrib/coccinelle/swap.cocci.patch\n>> file from multiple workers.\n>> \n>> Before this change running \"make coccicheck\" would by default end\n>> up pegging just one CPU at the very end for a while, usually as\n>> we'd finish whichever *.cocci rule was the most expensive.\n>> \n>> This could be mitigated by combining \"make -jN\" with\n>> SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch\n>> spatch invocations, 2019-05-06). But doing so required careful\n>> juggling, as e.g. setting both to 4 would yield 16 workers.\n>\n> No, setting both to 4 does yield 4 workers.\n>\n> SPATCH_BATCH_SIZE has nothing to do with parallelism; it is merely the\n> number of C source files that we pass to a single 'spatch' invocation,\n> but for any given semantic patch it's still a sequential loop.\n\nThanks, will fix. I see I conflated SPATCH_BATCH_SIZE with spatch's\n--jobs there (although from experimentation that seems to have pretty\nlimited parallelism).\n\n1. https://lore.kernel.org/git/220825.86ilmg4mil.gmgdl@evledraar.gmail.com/\n"},{"id":"461991","messageId":"20220826104312.GJ1735@szeder.dev","threadId":"58360","inReplyTo":"220826.861qt43pcc.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 3/5] cocci: make \"coccicheck\" rule incremental","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2022-08-26T10:43:12Z","receivedAt":"2022-08-26T10:43:23Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Aug 26, 2022 at 12:18:28AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Aug 25 2022, SZEDER Gábor wrote:\n> \n> Thanks for taking a look!\n> \n> > On Thu, Aug 25, 2022 at 04:36:15PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> >> * Since we create a single \"*.cocci.patch+\" we don't know where to\n> >> pick up where we left off. Note that (per [1]) we've had a race\n> >> condition since 960154b9c17 (coccicheck: optionally batch spatch\n> >> invocations, 2019-05-06) which might result in us producing corrupt\n> >> patches to to the concurrent appending to \"$@+\" in the pre-image.\n> >> \n> >> That bug is now fixed.\n> >\n> > There is no bug, because there is no concurrent appending to \"$@+\".\n> > The message you mention seems to be irrelevant, as it talks about\n> > 'xargs -P', but the invocation in '%.cocci.patch' targets never used\n> > '-P'.\n> \n> I think this is just confusing, I'll amend/rephrase.\n> \n> And at this point I honestly can't remember if I'm conflating this with\n> an issue with my earlier proposed series here (I drafted this a while\n> ago), or a rather obscure \"hidden\" feature that I did use to speed up\n> coccicheck for myself for a while. Which is that you can do e.g.:\n> \n> \tmake coccicheck SPATCH_BATCH_SIZE=\"8 -P 8\"\n> \n> Which will invoke xargs with the \"-P 8\" option in batches of 8 files.\n\nAbusing SPATCH_BATCH_SIZE to try to shoot yourself in the foot\ndefinitely doesn't qualify as \"bug\" in my book :)\n\n> That's never a thing that 960154b9c17 expected, and it only worked as an\n> accident, but it *would* work, and unless you got unlucky with the races\n> involved would generally speed up your coccicheck.\n> \n> >> Which is why we'll not depend on $(FOUND_H_SOURCES) but the *.o file\n> >> corresponding to the *.c file, if it exists already. This means that\n> >> we can do:\n> >> \n> >> make all\n> >> make coccicheck\n> >> make -W column.h coccicheck\n> >> \n> >> By depending on the *.o we piggy-back on\n> >> COMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the\n> >> \"sparse\" target non-.PHONY, 2021-09-23) for prior art of doing that\n> >> for the *.sp files. E.g.:\n> >> \n> >> make contrib/coccinelle/free.cocci.patch\n> >> make -W column.h contrib/coccinelle/free.cocci.patch\n> >> \n> >> Will take around 15 seconds for the second command on my 8 core box if\n> >> I didn't run \"make\" beforehand to create the *.o files. But around 2\n> >> seconds if I did and we have those \"*.o\" files.\n> >> \n> >> Notes about the approach of piggy-backing on *.o for dependencies:\n> >> \n> >> * It *is* a trade-off since we'll pay the extra cost of running the C\n> >> compiler, but we're probably doing that anyway.\n> >\n> > This assumption doesn't hold, and I very much dislike the idea of\n> > depending on *.o files:\n> \n> It's my fault for not calling this out more explicitly, but it\n> *optionally* depends on the *.o files, but if you don't have them\n> compiled already a \"make coccicheck\" will just use \"spatch\", and nothing\n> else.\n> \n> See the CI run/output for this series:\n> https://github.com/avar/git/runs/8017916844?check_suite_focus=true\n> \n> > - Our static-analysis CI job doesn't build Git, now it will have to.\n> \n> Aside from this series which doesn't change how it works, maintaining\n> this doesn't seem important to me. E.g. as I noted in [1] coccinelle\n> will happily run on code that doesn't even compile.\n> \n> So running the compiler during \"static-analysis\" (or another step that\n> it would depend on, maybe \"pedantic\" or \"sparse\") seems like a good (but\n> separate change).\n\nIt seems like a bad and undesired change to me. We have plenty of\nother jobs that do build Git. It's just a waste of time to do it in a\njob that doesn't actually need it.\n\n> You don't want to wonder about odd coccinelle output, only to see it's\n> trying to make sense of C source that doesn't even compile.\n\nYou don't even want to look at Coccinelle output at all, if any of the\nother jobs were to fail for whatever reason.\n\n> > - I don't have Coccinelle installed, because my distro doesn't ship\n> > it, and though the previous release did ship it, it was outdated.\n> > Instead I use Coccinelle's most recent version from a container\n> > which doesn't contain any build tools apart from 'make' for 'make\n> > coccicheck'.\n> >\n> > With this patch series I can't use this containerized Coccinelle\n> > at all, because even though I've already built git on the host,\n> > the dependency on *.o files triggers a BUILD-OPTIONS check during\n> > 'make coccicheck', and due to the missing 'curl-config' the build\n> > options do differ, triggering a rebuild, which in the absence of a\n> > compiler fails.\n> >\n> > And then the next 'make' on the host will have to rebuild\n> > everything again...\n> \n> There may be some odd interaction here, but it's unclear if you've\n> actually tried to do this with this series, because unless I've missed\n> some edge case this should all still work, per the above.\n\nIt did not work:\n\n # Host:\n $ make\n SUBDIR git-gui\n SUBDIR gitk-git\n SUBDIR templates\n\n # Container:\n $ time make coccicheck -j4 SPATCH_FLAGS=--all-includes\n GIT_VERSION = 2.37.GIT\n make: curl-config: Command not found\n * new build flags\n CC chdir-notify.o\n CC imap-send.o\n /bin/sh: 1: gcc-10: not found\n /bin/sh: 1: gcc-10: not found\n make: *** [Makefile:2605: chdir-notify.o] Error 127\n make: *** Waiting for unfinished jobs....\n make: *** [Makefile:2605: imap-send.o] Error 127\n CC usage.o\n /bin/sh: 1: gcc-10: not found\n make: *** [Makefile:2605: usage.o] Error 127\n\n # Host again:\n $ make\n GIT_VERSION = 2.37.2.387.g804ba1c054\n * new build flags\n CC fuzz-commit-graph.o\n [ ... rebuild ... ]\n\n> What *won't* work is avoiding potential re-compilation of *.o files if\n> you *have them already* when you run \"make coccicheck\". I'm not familiar\n> with this type of setup, are you saying you're running \"make coccicheck\"\n> on a working directory that already has *.o files, but you want it to\n> ignore the *.o?\n> \n> That could easily be made optional, but I just assumed that nobody would\n> care. If you want that can you try this on top and see if it works for\n> you?:\n> \t\n> \tdiff --git a/Makefile b/Makefile\n> \tindex 9410a587fc0..11d83c490b4 100644\n> \t--- a/Makefile\n> \t+++ b/Makefile\n> \t@@ -3174,6 +3174,11 @@ TINY_FOUND_H_SOURCES += strbuf.h\n> \t \t$(call mkdir_p_parent_template)\n> \t \t$(QUIET_GEN) >$@\n> \t \n> \t+SPATCH_USE_O_DEPENDENCIES = yes\n> \t+ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no)\n> \t+SPATCH_USE_O_DEPENDENCIES =\n> \t+endif\n> \t+\n> \t define cocci-rule\n> \t \n> \t ## Rule for .build/$(1).patch/$(2); Params:\n> \t@@ -3181,7 +3186,7 @@ define cocci-rule\n> \t # 2 = $(2)\n> \t COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n> \t .build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n> \t-.build/$(1).patch/$(2): $(if $(wildcard $(3)),$(3),$(if $(filter $(USE_TINY_FOUND_H_SOURCES),$(3)),$(TINY_FOUND_H_SOURCES),.build/contrib/coccinelle/FOUND_H_SOURCES))\n> \t+.build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),$(if $(filter $(USE_TINY_FOUND_H_SOURCES),$(3)),$(TINY_FOUND_H_SOURCES),.build/contrib/coccinelle/FOUND_H_SOURCES))\n> \t .build/$(1).patch/$(2): $(1)\n> \t .build/$(1).patch/$(2): .build/$(1).patch/% : %\n> \t \t$$(call mkdir_p_parent_template)\n\nOK, with these changes on top I can run 'make coccicheck' in the\ncontainer after Git was built on the host.\n\nAnd I noticed a few behavior changes:\n\n - 'make coccicheck' currently prints ~\"Nr-of-semantic-patches\"\n lines, but with this change it grows to \"Nr-of-semantic-patches *\n Nr-of-source-files\" lines.\n\n I didn't like this when I did something similar a few years ago in\n\n https://public-inbox.org/git/20180802115522.16107-1-szeder.dev@gmail.com/\n\n and I don't like it now, either, but oh well, there is always\n 'make -s' (though Peff thought it's \"eye candy\" back then :).\n\n - Any transformation found during 'make coccicheck' is now\n considered an error.\n\n With 'SPATCH_FLAGS=--recursive-includes' I got this:\n\n [... gazillions of SPATCH .build/... lines ... ]\n SPATCH .build/contrib/coccinelle/hashmap.cocci.patch/config.c\n SPATCH MERGE contrib/coccinelle/array.cocci.patch\n SPATCH MERGE contrib/coccinelle/commit.cocci.patch\n SPATCH MERGE contrib/coccinelle/xcalloc.cocci.patch\n SPATCH MERGE contrib/coccinelle/preincr.cocci.patch\n SPATCH MERGE contrib/coccinelle/free.cocci.patch\n SPATCH MERGE contrib/coccinelle/equals-null.cocci.patch\n SPATCH MERGE contrib/coccinelle/xopen.cocci.patch\n SPATCH MERGE contrib/coccinelle/qsort.cocci.patch\n SPATCH MERGE contrib/coccinelle/xstrdup_or_null.cocci.patch\n SPATCH MERGE contrib/coccinelle/swap.cocci.patch\n SPATCH MERGE contrib/coccinelle/strbuf.cocci.patch\n SPATCH MERGE contrib/coccinelle/object_id.cocci.patch\n SPATCH MERGE contrib/coccinelle/flex_alloc.cocci.patch\n SPATCH RESULT contrib/coccinelle/array.cocci.patch\n SPATCH RESULT contrib/coccinelle/commit.cocci.patch\n SPATCH RESULT contrib/coccinelle/xcalloc.cocci.patch\n diff -u -p a/promisor-remote.c b/promisor-remote.c\n --- a/promisor-remote.c\n +++ b/promisor-remote.c\n @@ -146,7 +146,7 @@ static void promisor_remote_init(struct\n \tif (r->promisor_remote_config)\n \t\treturn;\n \tconfig = r->promisor_remote_config =\n -\t\txcalloc(sizeof(*r->promisor_remote_config), 1);\n +\t\txcalloc(1, sizeof(*r->promisor_remote_config));\n \tconfig->promisors_tail = &config->promisors;\n \n \trepo_config(r, promisor_remote_config, config);\n make: *** [Makefile:3218: .build/contrib/coccinelle/xcalloc.cocci.patch.ok] Error 1\n make: *** Waiting for unfinished jobs....\n\n So it seems that if more than one semantic patches find something\n to transform, then only the transformations from the first will\n be generated (and printed).\n\n Note, however, that if transformations are found while \"manually\"\n applying semantic patches, i.e. running 'make foo.cocci.patch',\n then it's not an error.\n\n - The resulting transformations from (only one of) the semantic\n patches are now printed at the end of 'make coccicheck'. I doubt\n it's desirable, but even if it is, it's an independent change, and\n the static analysis CI job needs a corresponding update, because\n it already 'cat's all non-empty '.cocci.patch' files.\n\n Note, however, that if transformations are found while \"manually\"\n applying semantic patches, then neither the transformations are\n printed, nor anything that would indicate that there were any\n transformations.\n\n - As I remember before this series the SPATCH TESTs were always run\n first during 'make coccicheck'. I don't know whether that was\n intentional and there were dependencies to enforce that order, or\n 'make' just happened to schedule them in that order on my system.\n Or maybe I'm just misremembering, but it does make sense to me to\n run those tests first (or at least before applying the\n corresponding semantic patches to the whole tree).\n\n With this series the tests were run near the end:\n\n [...]\n SPATCH .build/contrib/coccinelle/unused.cocci.patch/git.c\n SPATCH .build/contrib/coccinelle/unused.cocci.patch/color.c\n MKDIR -p .build/contrib/coccinelle/tests\n SPATCH TEST contrib/coccinelle/tests/unused.res\n SPATCH TEST contrib/coccinelle/tests/free.res\n SPATCH .build/contrib/coccinelle/hashmap.cocci.patch/upload-pack.c\n SPATCH .build/contrib/coccinelle/hashmap.cocci.patch/merge-ort-wrappers.c\n [...]\n\n 'hashmap.cocci' happened to be applied last.\n\n> I'd need to split up the already long line, but with *.o files compiled\n> and:\n> \t\n> \t$ time make -W column.h contrib/coccinelle/free.cocci.patch COMPUTE_HEADER_DEPENDENCIES=yes\n> \t CC wt-status.o\n> \t CC builtin/branch.o\n> \t CC builtin/clean.o\n> \t CC builtin/column.o\n> \t CC builtin/commit.o\n> \t CC builtin/tag.o\n> \t CC column.o\n> \t CC help.o\n> \t SPATCH .build/contrib/coccinelle/free.cocci.patch/builtin/commit.c\n> \t SPATCH .build/contrib/coccinelle/free.cocci.patch/builtin/tag.c\n> \t SPATCH .build/contrib/coccinelle/free.cocci.patch/builtin/column.c\n> \t SPATCH .build/contrib/coccinelle/free.cocci.patch/column.c\n> \t SPATCH .build/contrib/coccinelle/free.cocci.patch/wt-status.c\n> \t SPATCH .build/contrib/coccinelle/free.cocci.patch/builtin/branch.c\n> \t SPATCH .build/contrib/coccinelle/free.cocci.patch/builtin/clean.c\n> \t SPATCH .build/contrib/coccinelle/free.cocci.patch/help.c\n> \t SPATCH MERGE contrib/coccinelle/free.cocci.patch\n> \t\n> \treal 0m0.550s\n> \tuser 0m0.505s\n> \tsys 0m0.096s\n> \n> But if you set COMPUTE_HEADER_DEPENDENCIES=yes it'll take ~4s (on that\n> box), and well re-apply the free.cocci rule to all the *.c files (and\n> this is all with the caching mechanism in 5/5, real \"spatch\" will be\n> much slower).\n\nUnfortunately:\n\n # After a successful 'make coccicheck' in the container:\n $ time make -W column.h contrib/coccinelle/free.cocci.patch COMPUTE_HEADER_DEPENDENCIES=yes\n make: curl-config: Command not found\n * new spatch flags\n * new build flags\n CC chdir-notify.o\n /bin/sh: 1: gcc-10: not found\n CC imap-send.o\n /bin/sh: 1: gcc-10: not found\n make: *** [Makefile:2605: chdir-notify.o] Error 127\n make: *** Waiting for unfinished jobs....\n make: *** [Makefile:2605: imap-send.o] Error 127\n CC usage.o\n /bin/sh: 1: gcc-10: not found\n make: *** [Makefile:2605: usage.o] Error 127\n\n> If you already have git compiled (or partially compiled) the \"happy\n> path\" to avoiding work is almost definitely to re-compile that *.o\n> because the *.c changed, at which point we'll be able to see if we even\n> need to re-run any of the coccinelle rules. Usually we'll need to re-run\n> a far smaller set than the full set we operate on now.\n> \n> >> * We can take better advantage of parallelism, while making sure that\n> >> we don't racily append to the contrib/coccinelle/swap.cocci.patch\n> >> file from multiple workers.\n> >> \n> >> Before this change running \"make coccicheck\" would by default end\n> >> up pegging just one CPU at the very end for a while, usually as\n> >> we'd finish whichever *.cocci rule was the most expensive.\n> >> \n> >> This could be mitigated by combining \"make -jN\" with\n> >> SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch\n> >> spatch invocations, 2019-05-06). But doing so required careful\n> >> juggling, as e.g. setting both to 4 would yield 16 workers.\n> >\n> > No, setting both to 4 does yield 4 workers.\n> >\n> > SPATCH_BATCH_SIZE has nothing to do with parallelism; it is merely the\n> > number of C source files that we pass to a single 'spatch' invocation,\n> > but for any given semantic patch it's still a sequential loop.\n> \n> Thanks, will fix. I see I conflated SPATCH_BATCH_SIZE with spatch's\n> --jobs there (although from experimentation that seems to have pretty\n> limited parallelism).\n\n'spatch --jobs' works just fine; the problem is that we use 'spatch'\nwrong.\n\n> \n> 1. https://lore.kernel.org/git/220825.86ilmg4mil.gmgdl@evledraar.gmail.com/\n"},{"id":"462320","messageId":"patch-v2-2.9-0998948b881-20220831T205130Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v2-0.9-00000000000-20220831T205130Z-avarab@gmail.com","subject":"[PATCH v2 2/9] Makefile: add ability to TAB-complete cocci *.patch rules","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-31T20:57:49Z","receivedAt":"2022-08-31T20:58:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Declare the contrib/coccinelle/.cocci.patch rules in such a way\nas to allow TAB-completion, and slightly optimize the Makefile by\ncutting down on the number of $(wildcard) in favor of defining\n\"coccicheck\" and \"coccicheck-pending\" in terms of the same\nincrementally filtered list.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 14 ++++++++++----\n 1 file changed, 10 insertions(+), 4 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex eac30126e29..42a70eeaef4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3139,9 +3139,15 @@ check: $(GENERATED_H)\n \t\texit 1; \\\n \tfi\n \n+COCCI = $(wildcard contrib/coccinelle/*.cocci)\n+COCCI_PATCHES = $(addsuffix .patch,$(COCCI))\n+COCCICHECK_PENDING = $(filter %.pending.cocci.patch,$(COCCI_PATCHES))\n+COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES))\n+\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n-%.cocci.patch: %.cocci $(COCCI_SOURCES)\n+$(COCCI_PATCHES): $(COCCI_SOURCES)\n+$(COCCI_PATCHES): %.patch: %\n \t$(QUIET_SPATCH) \\\n \tif test $(SPATCH_BATCH_SIZE) = 0; then \\\n \t\tlimit=; \\\n@@ -3178,11 +3184,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell\n coccicheck-test: $(COCCI_TEST_RES_GEN)\n \n coccicheck: coccicheck-test\n-coccicheck: $(addsuffix .patch,$(filter-out %.pending.cocci,$(wildcard contrib/coccinelle/*.cocci)))\n+coccicheck: $(COCCICHECK)\n \n # See contrib/coccinelle/README\n coccicheck-pending: coccicheck-test\n-coccicheck-pending: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.pending.cocci))\n+coccicheck-pending: $(COCCICHECK_PENDING)\n \n .PHONY: coccicheck coccicheck-pending\n \n@@ -3450,7 +3456,7 @@ profile-clean:\n \n cocciclean:\n \t$(RM) -r .build/contrib/coccinelle\n-\t$(RM) contrib/coccinelle/*.cocci.patch*\n+\t$(RM) $(COCCI_PATCHES)*\n \n clean: profile-clean coverage-clean cocciclean\n \t$(RM) -r .build\n-- \n2.37.3.1420.g76f8a3d556c\n\n"},{"id":"462321","messageId":"cover-v2-0.9-00000000000-20220831T205130Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-0.5-00000000000-20220825T141212Z-avarab@gmail.com","subject":"[PATCH v2 0/9] cocci: make \"incremental\" possible + a ccache-like tool","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-31T20:57:47Z","receivedAt":"2022-08-31T20:58:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A re-roll of the series to have \"make coccicheck\" run\nincrementally. For the v1 see:\nhttps://lore.kernel.org/git/cover-0.5-00000000000-20220825T141212Z-avarab@gmail.com/\n\nChanges since v1 (based on SZEDER's feedback, thanks!):\n\n * Corrected small bits of commit message, removed mention of a \"bug\"\n (which was really something in an earlier series of mine...)\n\n * Split up the previous \"incremental\" commit. Now 1/9 is an unrelated\n fix that was in it (but which was spotted by it).\n\n * Added some preceding patches to make tweaking spatch flags easier.\n\n * The part where we optionally piggy-back on the dependencies of *.o\n files is now in 8/9, the \"pure\" incremental coccicheck in 7/9 will\n never need a compiler.\n\n * The \"make coccicheck\" now acts as before, i.e. no \"make -k\" needed,\n it runs to completion. I think having it \"make -k\" friendly would\n optionally make sense, but that can be left for some other time.\n\n * The previous change dropped the cocci *.log, we now log the stderr\n we had as before.\n\n * The spatchcache contrib command in 9/9 has updated docs & behavior.\n\n * The spatchcache now requires SPATCH_FLAGS=--very-quiet, which was\n the new default in this series before. I.e. we never cache things\n that have any stderr output.\n\nÆvar Arnfjörð Bjarmason (9):\n cocci rules: remove unused \"F\" metavariable from pending rule\n Makefile: add ability to TAB-complete cocci *.patch rules\n Makefile: have \"coccicheck\" re-run if flags change\n Makefile: split off SPATCH_BATCH_SIZE comment from \"cocci\" heading\n cocci: split off include-less \"tests\" from SPATCH_FLAGS\n cocci: split off \"--all-includes\" from SPATCH_FLAGS\n cocci: make \"coccicheck\" rule incremental\n cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES\n spatchcache: add a ccache-alike for \"spatch\"\n\n .gitignore | 1 +\n Makefile | 113 +++++++--\n contrib/coccinelle/spatchcache | 234 ++++++++++++++++++\n .../coccinelle/the_repository.pending.cocci | 1 -\n shared.mak | 3 +-\n 5 files changed, 325 insertions(+), 27 deletions(-)\n create mode 100755 contrib/coccinelle/spatchcache\n\nRange-diff against v1:\n -: ----------- > 1: 72b6a8e4e0b cocci rules: remove unused \"F\" metavariable from pending rule\n 1: 8604ac9bfbc = 2: 0998948b881 Makefile: add ability to TAB-complete cocci *.patch rules\n 2: db3b3427668 ! 3: 63cf9f58d99 Makefile: have \"coccicheck\" re-run if flags change\n @@ Commit message\n since 63f0a758a06 (add coccicheck make target, 2016-09-15), unlike\n e.g. \"make grep.o\" we wouldn't re-run it when $(SPATCH) or\n $(SPATCH_FLAGS) changed. To test new flags we needed to first do a\n - \"make clean\".\n + \"make cocciclean\".\n \n This now uses the same (copy/pasted) pattern as other \"DEFINES\"\n rules. As a result we'll re-run properly. This can be demonstrated\n -: ----------- > 4: 54d6bae3984 Makefile: split off SPATCH_BATCH_SIZE comment from \"cocci\" heading\n -: ----------- > 5: 4f165bf6128 cocci: split off include-less \"tests\" from SPATCH_FLAGS\n -: ----------- > 6: c74d09f4825 cocci: split off \"--all-includes\" from SPATCH_FLAGS\n 3: 6fa83695f1f ! 7: 120607b5da6 cocci: make \"coccicheck\" rule incremental\n @@ Commit message\n cocci: make \"coccicheck\" rule incremental\n \n Optimize the very slow \"coccicheck\" target to take advantage of\n - incremental rebuilding, and fix outstanding bugs and dependency\n - problems with the existing rule.\n + incremental rebuilding, and fix outstanding dependency problems with\n + the existing rule.\n \n The rule is now faster both on the initial run as we can make better\n use of GNU make's parallelism than the old ad-hoc combination of\n - make's parallelism combined with $(SPATCH_BATCH_SIZE) and \"xargs -n\".\n + make's parallelism combined with $(SPATCH_BATCH_SIZE) and/or the\n + \"--jobs\" argument to \"spatch(1)\".\n \n It also makes us *much* faster when incrementally building, it's now\n viable to \"make coccicheck\" as topic branches are merged down.\n @@ Commit message\n \n make -W grep.c coccicheck\n \n - Would be the same as doing a full re-run. A change in a single file\n - would force us to do a full re-run.\n + Would do a full re-run, i.e. a a change in a single file would force\n + us to do a full re-run.\n \n The reason for this (not the initial rationale, but my analysis) is:\n \n * Since we create a single \"*.cocci.patch+\" we don't know where to\n - pick up where we left off. Note that (per [1]) we've had a race\n - condition since 960154b9c17 (coccicheck: optionally batch spatch\n - invocations, 2019-05-06) which might result in us producing corrupt\n - patches to to the concurrent appending to \"$@+\" in the pre-image.\n -\n - That bug is now fixed.\n + pick up where we left off, or how to incrementally merge e.g. a\n + \"grep.c\" change with an existing *.cocci.patch.\n \n * We've been carrying forward the dependency on the *.c files since\n 63f0a758a06 (add coccicheck make target, 2016-09-15) the rule was\n @@ Commit message\n file (e.g. contrib/coccinelle/swap.cocci.patch) is now a simple \"cat\n $^\" of all of all of the / files for a given .\n \n + In the case discussed above of \"grep.c\" being changed we'll do the\n + full \"cat\" every time, so they resulting *.cocci.patch will always\n + be correct and up-to-date, even if it's \"incrementally updated\".\n +\n See 1cc0425a27c (Makefile: have \"make pot\" not \"reset --hard\",\n 2022-05-26) for another recent rule that used that technique.\n \n - * We depend on a .build/.ok file, which in turn depends on our\n - . This emulates the previous \"test -s\" test, but in a more\n - Makefile-native way (e.g. -k works as expected).\n -\n - As noted above this change fixes existing dependency issues,\n - e.g. changing strbuf.h will now trigger a re-build. Our default is to\n - depend on $(FOUND_H_SOURCES). This makes incremental re-building\n - possible, but it's still fairly slow, as modifying one header will\n - cause a full re-build of everything.\n -\n - Which is why we'll not depend on $(FOUND_H_SOURCES) but the *.o file\n - corresponding to the *.c file, if it exists already. This means that\n - we can do:\n -\n - make all\n - make coccicheck\n - make -W column.h coccicheck\n -\n - By depending on the *.o we piggy-back on\n - COMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the\n - \"sparse\" target non-.PHONY, 2021-09-23) for prior art of doing that\n - for the *.sp files. E.g.:\n -\n - make contrib/coccinelle/free.cocci.patch\n - make -W column.h contrib/coccinelle/free.cocci.patch\n -\n - Will take around 15 seconds for the second command on my 8 core box if\n - I didn't run \"make\" beforehand to create the *.o files. But around 2\n - seconds if I did and we have those \"*.o\" files.\n -\n - Notes about the approach of piggy-backing on *.o for dependencies:\n -\n - * It *is* a trade-off since we'll pay the extra cost of running the C\n - compiler, but we're probably doing that anyway. The compiler is much\n - faster than \"spatch\", so even though we need to re-compile the *.o to\n - create the dependency info for the *.c for \"spatch\" it's\n - faster (especially if using \"ccache\").\n + As before we'll:\n \n - * If that *.o file doesn't exist we'll depend on an intermediate file\n - of ours which in turn depends on $(FOUND_H_SOURCES).\n + * End up generating a contrib/coccinelle/swap.cocci.patch, if we\n + \"fail\" by creating a non-empty patch we'll still exit with a zero\n + exit code.\n \n - This covers both an initial build, or where \"coccicheck\" is run\n - without running \"all\" beforehand, and because we run \"coccicheck\"\n - on e.g. files in compat/* that we don't know how to build unless\n - the requisite flag was provided to the Makefile.\n -\n - Most of the runtime of \"incremental\" runs is now spent on various\n - compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and\n - therefore conflate whether we *can* compile an object and generate\n - dependency information for it with whether we'd like to link it\n - into our binary.\n -\n - Before this change the distinction didn't matter, but now one way\n - to make this even faster on incremental builds would be to peel\n - those concerns apart so that we can see that e.g. compat/mmap.c\n - doesn't depend on column.h.\n + Arguably we should move to a more Makefile-native way of doing\n + this, i.e. fail early, and if we want all of the \"failed\" changes\n + we can use \"make -k\", but as the current\n + \"ci/run-static-analysis.sh\" expects us to behave this way let's\n + keep the existing behavior of exhaustively discovering all cocci\n + changes, and only failing if spatch itself errors out.\n \n Further implementation details & notes:\n \n - * We can take better advantage of parallelism, while making sure that\n - we don't racily append to the contrib/coccinelle/swap.cocci.patch\n - file from multiple workers.\n -\n - Before this change running \"make coccicheck\" would by default end\n + * Before this change running \"make coccicheck\" would by default end\n up pegging just one CPU at the very end for a while, usually as\n we'd finish whichever *.cocci rule was the most expensive.\n \n @@ Commit message\n setting that makes sense when doing a non-incremental run of \"make\n coccicheck\".\n \n - * The change here to\n - \"contrib/coccinelle/the_repository.pending.cocci\" is because we've\n - been spewing out this warning into our $@.log since that rule was\n - added:\n -\n - warning: rule starting on line 21: metavariable F not used in the - or context code\n -\n - It's being fixed here because we'll now notice on \"make\n - coccicheck-pending\", since we're not redirecting all of STDERR to a\n - log file, instead we use the \"--very-quiet\" option.\n -\n Outstanding issues & future work:\n \n * We could get rid of \"--all-includes\" in favor of manually\n specifying a list of includes to give to \"spatch(1)\".\n \n - As noted upthread of [2] a naïve removal of \"--all-includes\" will\n + As noted upthread of [1] a naïve removal of \"--all-includes\" will\n result in broken *.cocci patches, but if we know the exhaustive\n list of includes via COMPUTE_HEADER_DEPENDENCIES we don't need to\n re-scan for them, we could grab the headers to include from the\n .depend.d/.o.d and supply them with the \"--include\" option to\n spatch(1).q\n \n - * We can generalize COMPUTE_HEADER_DEPENDENCIES to spew out the list\n - of *.h dependencies without making a *.o object, which is slightly\n - faster, and would also benefit the \"make sparse\" target. We'd still\n - need to run the compiler, it would just have less work to do.\n -\n - 1. https://lore.kernel.org/git/877dlwotjc.fsf@evledraar.gmail.com/\n - 2. https://lore.kernel.org/git/87ft18tcog.fsf@evledraar.gmail.com/\n + 1. https://lore.kernel.org/git/87ft18tcog.fsf@evledraar.gmail.com/\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## Makefile ##\n -@@ Makefile: SP_EXTRA_FLAGS = -Wno-universal-initializer\n - SANITIZE_LEAK =\n - SANITIZE_ADDRESS =\n - \n --# For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n +@@ Makefile: SANITIZE_ADDRESS =\n + SPATCH_INCLUDE_FLAGS = --all-includes\n + SPATCH_FLAGS =\n + SPATCH_TEST_FLAGS =\n +-# Setting SPATCH_BATCH_SIZE higher will\n -# usually result in less CPU usage at the cost of higher peak memory.\n -# Setting it to 0 will feed all files in a single spatch invocation.\n - SPATCH_FLAGS = --all-includes\n -SPATCH_BATCH_SIZE = 1\n \n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n - TRACK_SPATCH_DEFINES += $(SPATCH)\n +@@ Makefile: TRACK_SPATCH_DEFINES += $(SPATCH)\n + TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n + TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS)\n -TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n GIT-SPATCH-DEFINES: FORCE\n \t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n @@ Makefile: check: $(GENERATED_H)\n COCCI = $(wildcard contrib/coccinelle/*.cocci)\n +COCCI_NAMES = $(COCCI:contrib/coccinelle/%.cocci=%)\n COCCI_PATCHES = $(addsuffix .patch,$(COCCI))\n -+COCCI_PATCHES_OK = $(COCCI_PATCHES:%=.build/%.ok)\n COCCICHECK_PENDING = $(filter %.pending.cocci.patch,$(COCCI_PATCHES))\n -COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES))\n \n +COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES))\n -+COCCICHECK_OK = $(COCCICHECK:%=.build/%.ok)\n -+COCCICHECK_PENDING_OK = $(COCCICHECK_PENDING:%=.build/%.ok)\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n -$(COCCI_PATCHES): GIT-SPATCH-DEFINES\n @@ Makefile: check: $(GENERATED_H)\n -\tfi; \\\n -\tif ! echo $(COCCI_SOURCES) | xargs $$limit \\\n -\t\t$(SPATCH) $(SPATCH_FLAGS) \\\n +-\t\t$(SPATCH_INCLUDE_FLAGS) \\\n -\t\t--sp-file $< --patch . \\\n -\t\t>$@+ 2>$@.log; \\\n +.build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES)\n @@ Makefile: check: $(GENERATED_H)\n +define cocci-rule\n +\n +## Rule for .build/$(1).patch/$(2); Params:\n -+# 1 = $(1)\n -+# 2 = $(2)\n ++# $(1) = e.g. \"free.cocci\"\n ++# $(2) = e.g. \"grep.c\"\n +COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n +.build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n -+.build/$(1).patch/$(2): $(if $(wildcard $(3)),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES)\n ++.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES\n +.build/$(1).patch/$(2): $(1)\n +.build/$(1).patch/$(2): .build/$(1).patch/% : %\n +\t$$(call mkdir_p_parent_template)\n -+\t$$(QUIET_SPATCH)$$(SPATCH) $$(SPATCH_FLAGS) \\\n -+\t\t--very-quiet \\\n -+\t\t--sp-file $(1) --patch . \\\n -+\t\t$$(SPATCH_PATCH_FLAGS) $$< >$$@\n ++\t$$(QUIET_SPATCH)if ! $$(SPATCH) $$(SPATCH_FLAGS) \\\n ++\t\t$$(SPATCH_INCLUDE_FLAGS) \\\n ++\t\t--sp-file $(1) --patch . $$< \\\n ++\t\t>$$@ 2>$$@.log; \\\n + \tthen \\\n +-\t\tcat $@.log; \\\n ++\t\techo \"ERROR when applying '$(1)' to '$$<'; '$$@.log' follows:\"; \\\n ++\t\tcat $$@.log; \\\n + \t\texit 1; \\\n +-\tfi; \\\n +-\tmv $@+ $@; \\\n +-\tif test -s $@; \\\n ++\tfi\n +endef\n +\n +define cocci-matrix-2\n +\n -+$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o)))\n ++$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s)))\n +endef\n +define cocci-matrix-1\n +$(foreach c,$(COCCI),$(call cocci-matrix-2,$(c)))\n @@ Makefile: check: $(GENERATED_H)\n +define spatch-rule\n +\n +contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1))\n -+\t$(QUIET_SPATCH_M)cat $$^ >$$@\n -+endef\n -+\n -+$(COCCI_PATCHES_OK): .build/%.ok : %\n -+\t$(QUIET_SPATCH_R)if test -s $<; \\\n ++\t$(QUIET_SPATCH_M)cat $$^ >$$@ && \\\n ++\tif test -s $$@; \\\n \tthen \\\n --\t\tcat $@.log; \\\n -+\t\tcat $< | sed -e 's/^/\t/'; \\\n - \t\texit 1; \\\n --\tfi; \\\n --\tmv $@+ $@; \\\n --\tif test -s $@; \\\n --\tthen \\\n -\t\techo ' ' SPATCH result: $@; \\\n --\tfi\n -+\tfi && \\\n -+\t>$@\n ++\t\techo ' ' SPATCH result: $$@; \\\n + \tfi\n ++endef\n +\n +ifneq ($(filter $(COCCI_RULES),$(MAKECMDGOALS)),)\n +$(eval $(foreach n,$(COCCI_NAMES),$(call spatch-rule,$(n))))\n @@ Makefile: check: $(GENERATED_H)\n \n COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n -@@ Makefile: $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell\n - coccicheck-test: $(COCCI_TEST_RES_GEN)\n - \n - coccicheck: coccicheck-test\n --coccicheck: $(COCCICHECK)\n -+coccicheck: $(COCCICHECK_OK)\n - \n - # See contrib/coccinelle/README\n - coccicheck-pending: coccicheck-test\n --coccicheck-pending: $(COCCICHECK_PENDING)\n -+coccicheck-pending: $(COCCICHECK_PENDING_OK)\n - \n - .PHONY: coccicheck coccicheck-pending\n - \n @@ Makefile: profile-clean:\n \n cocciclean:\n @@ Makefile: profile-clean:\n clean: profile-clean coverage-clean cocciclean\n \t$(RM) -r .build\n \n - ## contrib/coccinelle/the_repository.pending.cocci ##\n -@@ contrib/coccinelle/the_repository.pending.cocci: expression E;\n - \n - @@\n - expression E;\n --expression F;\n - @@\n - - has_object_file_with_flags(\n - + repo_has_object_file_with_flags(the_repository,\n -\n ## shared.mak ##\n @@ shared.mak: ifndef V\n \tQUIET_SP = @echo ' ' SP $<;\n @@ shared.mak: ifndef V\n +\tQUIET_SPATCH = @echo ' ' SPATCH $@;\n \tQUIET_SPATCH_T = @echo ' ' SPATCH TEST $(@:.build/%=%);\n +\tQUIET_SPATCH_M = @echo ' ' SPATCH MERGE $$@;\n -+\tQUIET_SPATCH_R = @echo ' ' SPATCH RESULT $<;\n \n ## Used in \"Documentation/Makefile\"\n \tQUIET_ASCIIDOC\t= @echo ' ' ASCIIDOC $@;\n 4: cda20790146 ! 8: 536dce45eef cocci: make incremental compilation even faster\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - cocci: make incremental compilation even faster\n + cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES\n \n - Make the incremental compilation of \"make coccicheck\" even faster, at\n - the cost of losing dependency discovery for a small whitelist of *.c\n - files which we can't create *.o files for.\n + Improve the incremental rebuilding support of \"coccicheck\" by\n + piggy-backing on the computed dependency information of the\n + corresponding *.o file, rather than rebuilding all / pairs\n + if either their corresponding file changes, or if any header changes.\n \n - This will break obscure edge cases since e.g.:\n + This in effect uses the same method that the \"sparse\" target was made\n + to use in c234e8a0ecf (Makefile: make the \"sparse\" target non-.PHONY,\n + 2021-09-23), except that the dependency on the *.o file isn't a hard\n + one, we check with $(wildcard) if the *.o file exists, and if so we'll\n + depend on it.\n \n - time make -W block-sha1/sha1.h coccicheck\n + This means that the common case of:\n \n - Would correctly re-build before, as block-sha1/sha1.c uses\n - block-sha1/sha1.h, but as it and these other\n - $(USE_TINY_FOUND_H_SOURCES) files depended on the union of all such\n - headers (and more, all of $(FOUND_H_SOURCES)) the above command would\n - take ~10 seconds before.\n + make\n + make coccicheck\n \n - As noted in the preceding commit these obscure cases were broken\n - before the preceding commit fixed them. I think it makes sense to make\n - a correctness v.s. performance trade-off here. By doing this it\n - becomes viable to e.g. run \"make coccicheck\" in the same loop that\n - merges various topic branches down (along with \"ccache\" sped-up\n - compilation).\n + Will benefit from incremental rebuilding, now changing e.g. a header\n + will only re-run \"spatch\" on those those *.c files that make use of\n + it:\n +\n + By depending on the *.o we piggy-back on\n + COMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the\n + \"sparse\" target non-.PHONY, 2021-09-23) for prior art of doing that\n + for the *.sp files. E.g.:\n +\n + make contrib/coccinelle/free.cocci.patch\n + make -W column.h contrib/coccinelle/free.cocci.patch\n +\n + Will take around 15 seconds for the second command on my 8 core box if\n + I didn't run \"make\" beforehand to create the *.o files. But around 2\n + seconds if I did and we have those \"*.o\" files.\n +\n + Notes about the approach of piggy-backing on *.o for dependencies:\n +\n + * It *is* a trade-off since we'll pay the extra cost of running the C\n + compiler, but we're probably doing that anyway. The compiler is much\n + faster than \"spatch\", so even though we need to re-compile the *.o to\n + create the dependency info for the *.c for \"spatch\" it's\n + faster (especially if using \"ccache\").\n +\n + * There *are* use-cases where some would like to have *.o files\n + around, but to have the \"make coccicheck\" ignore them. See:\n + https://lore.kernel.org/git/20220826104312.GJ1735@szeder.dev/\n +\n + For those users a:\n +\n + make\n + make coccicheck SPATCH_USE_O_DEPENDENCIES=\n +\n + Will avoid considering the *.o files.\n +\n + * If that *.o file doesn't exist we'll depend on an intermediate file\n + of ours which in turn depends on $(FOUND_H_SOURCES).\n +\n + This covers both an initial build, or where \"coccicheck\" is run\n + without running \"all\" beforehand, and because we run \"coccicheck\"\n + on e.g. files in compat/* that we don't know how to build unless\n + the requisite flag was provided to the Makefile.\n +\n + Most of the runtime of \"incremental\" runs is now spent on various\n + compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and\n + therefore conflate whether we *can* compile an object and generate\n + dependency information for it with whether we'd like to link it\n + into our binary.\n +\n + Before this change the distinction didn't matter, but now one way\n + to make this even faster on incremental builds would be to peel\n + those concerns apart so that we can see that e.g. compat/mmap.c\n + doesn't depend on column.h.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## Makefile ##\n -@@ Makefile: COCCICHECK_OK = $(COCCICHECK:%=.build/%.ok)\n - COCCICHECK_PENDING_OK = $(COCCICHECK_PENDING:%=.build/%.ok)\n - COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n - \n -+USE_TINY_FOUND_H_SOURCES =\n -+USE_TINY_FOUND_H_SOURCES += compat/%\n -+USE_TINY_FOUND_H_SOURCES += block-sha1/%\n -+USE_TINY_FOUND_H_SOURCES += ppc/%\n -+\n -+TINY_FOUND_H_SOURCES =\n -+TINY_FOUND_H_SOURCES += cache.h\n -+TINY_FOUND_H_SOURCES += git-compat-util.h\n -+TINY_FOUND_H_SOURCES += strbuf.h\n -+\n - .build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES)\n +@@ Makefile: COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \t$(call mkdir_p_parent_template)\n \t$(QUIET_GEN) >$@\n -@@ Makefile: define cocci-rule\n - # 2 = $(2)\n + \n ++SPATCH_USE_O_DEPENDENCIES = yes\n ++ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no)\n ++SPATCH_USE_O_DEPENDENCIES =\n ++endif\n ++\n + define cocci-rule\n + \n + ## Rule for .build/$(1).patch/$(2); Params:\n + # $(1) = e.g. \"free.cocci\"\n + # $(2) = e.g. \"grep.c\"\n ++# $(3) = e.g. \"grep.o\"\n COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n .build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n --.build/$(1).patch/$(2): $(if $(wildcard $(3)),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES)\n -+.build/$(1).patch/$(2): $(if $(wildcard $(3)),$(3),$(if $(filter $(USE_TINY_FOUND_H_SOURCES),$(3)),$(TINY_FOUND_H_SOURCES),.build/contrib/coccinelle/FOUND_H_SOURCES))\n +-.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES\n ++.build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES)\n .build/$(1).patch/$(2): $(1)\n .build/$(1).patch/$(2): .build/$(1).patch/% : %\n \t$$(call mkdir_p_parent_template)\n +@@ Makefile: endef\n + \n + define cocci-matrix-2\n + \n +-$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s)))\n ++$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o)))\n + endef\n + define cocci-matrix-1\n + $(foreach c,$(COCCI),$(call cocci-matrix-2,$(c)))\n 5: ce4734e5d79 ! 9: 2b978676a56 spatchcache: add a ccache-alike for \"spatch\"\n @@ Commit message\n \n Add a rather trivial \"spatchcache\", with this running e.g.:\n \n - make clean\n - make contrib/coccinelle/free.cocci.patch\n + make cocciclean\n + make contrib/coccinelle/free.cocci.patch \\\n + SPATCH=contrib/coccicheck/spatchcache \\\n + SPATCH_FLAGS=--very-quiet\n \n - Is cut down from ~20s to ~3s on my system (and much of it is fixable\n - shell overhead). This uses \"redis\" as a cache by default, but it's\n - configurable. See the embedded documentation.\n + Is cut down from ~20s to ~5s on my system. Much of that is either\n + fixable shell overhead, or the around 40 files we \"CANTCACHE\" (see the\n + implementation).\n +\n + This uses \"redis\" as a cache by default, but it's configurable. See\n + the embedded documentation.\n \n This is *not* like ccache in that we won't cache failed spatch\n invocations, or those where spatch suggests changes for us. Those\n cases are so rare that I didn't think it was worth the bother, by far\n - the most common case is that it has no suggested changes.\n + the most common case is that it has no suggested changes. We'll also\n + refuse to cache any \"spatch\" invocation that has output on stderr,\n + which means that \"--very-quiet\" must be added to \"SPATCH_FLAGS\".\n \n Because we narrow the cache to that we don't need to save away stdout,\n stderr & the exit code. We simply cache the cases where we had no\n @@ contrib/coccinelle/spatchcache (new)\n +# driving \"spatch\" in git.git, in particular if we invoke:\n +#\n +#\tmake\n -+#\tmake coccicheck\n ++#\tmake coccicheck SPATCH_FLAGS=--very-quiet\n +#\n +# We can with COMPUTE_HEADER_DEPENDENCIES (auto-detected as true with\n +# \"gcc\" and \"clang\") write e.g. a .depend/grep.o.d for grep.c, when we\n @@ contrib/coccinelle/spatchcache (new)\n +# \"contrib/coccinelle\" makes it viable to (re-)run coccicheck\n +# e.g. when merging integration branches.\n +#\n ++# Note that the \"--very-quiet\" flag is currently critical. The cache\n ++# will refuse to cache anything that has output on STDERR (which might\n ++# be errors from spatch). The STDERR (and exit code) could in\n ++# principle be cached (as with ccache), but then the simple structure\n ++# in the Redis cache would need to change, so just supply\n ++# \"--very-quiet\" for now.\n ++#\n +# To use this, simply set SPATCH to\n +# contrib/coccinelle/spatchcache. Then optionally set:\n +#\n +#\t[spatchCache]\n -+#\t\t# path to a custom spatch\n ++#\t\t# Optional: path to a custom spatch\n +#\t\tspatch = ~/g/coccicheck/spatch.opt\n +#\n +# As well as this trace config (debug implies trace):\n @@ contrib/coccinelle/spatchcache (new)\n +#\t\tdebug = false\n +#\n +# Setting \"trace\" to \"true\" allows for seeing when we have a cache HIT\n -+# or MISS.\n ++# or MISS. To debug whether the cache is working do that, and run e.g.:\n ++#\n ++#\tredis-cli FLUSHALL\n ++#\t\n ++#\tgrep -hore HIT -e MISS -e SET -e NOCACHE -e CANTCACHE .build/contrib/coccinelle | sort | uniq -c\n ++#\t 600 CANTCACHE\n ++#\t 7365 MISS\n ++#\t 7365 SET\n ++#\n ++# A subsequent \"make cocciclean && make coccicheck\" should then have\n ++# all \"HIT\"'s and \"CANTCACHE\"'s.\n +\n +set -e\n +\n @@ contrib/coccinelle/spatchcache (new)\n +\n +if ! test -f \"$dep\"\n +then\n -+\ttrace_it \"$0: have no '$dep' for '$arg_file', can't cache!\"\n ++\ttrace_it \"$0: CANTCACHE have no '$dep' for '$arg_file'!\"\n +\texec \"$spatch\" \"$@\"\n +fi\n +\n @@ contrib/coccinelle/spatchcache (new)\n +\t\texit 128\n +\tfi\n +else\n -+\ttrace_it \"$0: NO CACHE (have changes): for '$arg_file' with '$arg_sp'\"\n ++\ttrace_it \"$0: NOCACHE (have changes): for '$arg_file' with '$arg_sp'\"\n +\tcat \"$out\"\n +\tcat \"$err\" >&2\n +\trm -f \"$out\" \"$err\"\n-- \n2.37.3.1420.g76f8a3d556c\n\n"},{"id":"462322","messageId":"patch-v2-3.9-63cf9f58d99-20220831T205130Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v2-0.9-00000000000-20220831T205130Z-avarab@gmail.com","subject":"[PATCH v2 3/9] Makefile: have \"coccicheck\" re-run if flags change","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-31T20:57:50Z","receivedAt":"2022-08-31T20:58:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix an issue with the \"coccicheck\" family of rules that's been here\nsince 63f0a758a06 (add coccicheck make target, 2016-09-15), unlike\ne.g. \"make grep.o\" we wouldn't re-run it when $(SPATCH) or\n$(SPATCH_FLAGS) changed. To test new flags we needed to first do a\n\"make cocciclean\".\n\nThis now uses the same (copy/pasted) pattern as other \"DEFINES\"\nrules. As a result we'll re-run properly. This can be demonstrated\ne.g. on the issue noted in [1]:\n\n\t$ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c V=1\n\t[...]\n\t SPATCH contrib/coccinelle/xcalloc.cocci\n\t$ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c SPATCH_FLAGS=\"--all-includes --recursive-includes\"\n\t * new spatch flags\n\t SPATCH contrib/coccinelle/xcalloc.cocci\n\t SPATCH result: contrib/coccinelle/xcalloc.cocci.patch\n\t$\n\n1. https://lore.kernel.org/git/20220823095602.GC1735@szeder.dev/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .gitignore | 1 +\n Makefile | 14 ++++++++++++++\n 2 files changed, 15 insertions(+)\n\ndiff --git a/.gitignore b/.gitignore\nindex 80b530bbed2..e67085f6918 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -10,6 +10,7 @@\n /GIT-PERL-HEADER\n /GIT-PYTHON-VARS\n /GIT-SCRIPT-DEFINES\n+/GIT-SPATCH-DEFINES\n /GIT-USER-AGENT\n /GIT-VERSION-FILE\n /bin-wrappers/\ndiff --git a/Makefile b/Makefile\nindex 42a70eeaef4..55ae3a2feda 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1300,6 +1300,18 @@ SANITIZE_ADDRESS =\n SPATCH_FLAGS = --all-includes\n SPATCH_BATCH_SIZE = 1\n \n+# Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n+TRACK_SPATCH_DEFINES =\n+TRACK_SPATCH_DEFINES += $(SPATCH)\n+TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n+TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n+GIT-SPATCH-DEFINES: FORCE\n+\t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n+\t if test x\"$$FLAGS\" != x\"`cat GIT-SPATCH-DEFINES 2>/dev/null`\" ; then \\\n+\t\techo >&2 \" * new spatch flags\"; \\\n+\t\techo \"$$FLAGS\" >GIT-SPATCH-DEFINES; \\\n+ fi\n+\n include config.mak.uname\n -include config.mak.autogen\n -include config.mak\n@@ -3146,6 +3158,7 @@ COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES))\n \n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n+$(COCCI_PATCHES): GIT-SPATCH-DEFINES\n $(COCCI_PATCHES): $(COCCI_SOURCES)\n $(COCCI_PATCHES): %.patch: %\n \t$(QUIET_SPATCH) \\\n@@ -3169,6 +3182,7 @@ $(COCCI_PATCHES): %.patch: %\n \tfi\n \n COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n+$(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n-- \n2.37.3.1420.g76f8a3d556c\n\n"},{"id":"462323","messageId":"patch-v2-1.9-72b6a8e4e0b-20220831T205130Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v2-0.9-00000000000-20220831T205130Z-avarab@gmail.com","subject":"[PATCH v2 1/9] cocci rules: remove unused \"F\" metavariable from pending rule","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-31T20:57:48Z","receivedAt":"2022-08-31T20:58:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix an issue with a rule added in 9b45f499818 (object-store: prepare\nhas_{sha1, object}_file to handle any repo, 2018-11-13). We've been\nspewing out this warning into our $@.log since that rule was added:\n\n\twarning: rule starting on line 21: metavariable F not used in the - or context code\n\nWe should do a better job of scouring our coccinelle log files for\nsuch issues, but for now let's fix this as a one-off.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n contrib/coccinelle/the_repository.pending.cocci | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/contrib/coccinelle/the_repository.pending.cocci b/contrib/coccinelle/the_repository.pending.cocci\nindex 072ea0d9228..747d382ff5f 100644\n--- a/contrib/coccinelle/the_repository.pending.cocci\n+++ b/contrib/coccinelle/the_repository.pending.cocci\n@@ -20,7 +20,6 @@ expression E;\n \n @@\n expression E;\n-expression F;\n @@\n - has_object_file_with_flags(\n + repo_has_object_file_with_flags(the_repository,\n-- \n2.37.3.1420.g76f8a3d556c\n\n"},{"id":"462324","messageId":"patch-v2-4.9-54d6bae3984-20220831T205130Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v2-0.9-00000000000-20220831T205130Z-avarab@gmail.com","subject":"[PATCH v2 4/9] Makefile: split off SPATCH_BATCH_SIZE comment from \"cocci\" heading","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-31T20:57:51Z","receivedAt":"2022-08-31T20:58:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Split off the \"; setting[...]\" part of the comment added in In\n960154b9c17 (coccicheck: optionally batch spatch invocations,\n2019-05-06), and restore what we had before that, which was a comment\nindicating that variables for the \"coccicheck\" target were being set\nhere.\n\nWhen 960154b9c17 amended the heading to discuss SPATCH_BATCH_SIZE it\nleft no natural place to add a new comment about other flags that\npreceded it. As subsequent commits will add such comments we need to\nsplit the existing comment up.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 55ae3a2feda..fa2e2193e99 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1294,10 +1294,11 @@ SP_EXTRA_FLAGS = -Wno-universal-initializer\n SANITIZE_LEAK =\n SANITIZE_ADDRESS =\n \n-# For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n+# For the 'coccicheck' target\n+SPATCH_FLAGS = --all-includes\n+# Setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n-SPATCH_FLAGS = --all-includes\n SPATCH_BATCH_SIZE = 1\n \n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n-- \n2.37.3.1420.g76f8a3d556c\n\n"},{"id":"462325","messageId":"patch-v2-5.9-4f165bf6128-20220831T205130Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v2-0.9-00000000000-20220831T205130Z-avarab@gmail.com","subject":"[PATCH v2 5/9] cocci: split off include-less \"tests\" from SPATCH_FLAGS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-31T20:57:52Z","receivedAt":"2022-08-31T20:58:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Amend the \"coccicheck-test\" rule added in f7ff6597a75 (cocci: add a\n\"coccicheck-test\" target and test *.cocci rules, 2022-07-05) to stop\nusing \"--all-includes\". The flags we'll need for the tests are\ndifferent than the ones we'll need for our main source code.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex fa2e2193e99..143c012b3be 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1296,6 +1296,7 @@ SANITIZE_ADDRESS =\n \n # For the 'coccicheck' target\n SPATCH_FLAGS = --all-includes\n+SPATCH_TEST_FLAGS =\n # Setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n@@ -1305,6 +1306,7 @@ SPATCH_BATCH_SIZE = 1\n TRACK_SPATCH_DEFINES =\n TRACK_SPATCH_DEFINES += $(SPATCH)\n TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n+TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n GIT-SPATCH-DEFINES: FORCE\n \t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n@@ -3188,7 +3190,7 @@ $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n \t$(call mkdir_p_parent_template)\n-\t$(QUIET_SPATCH_T)$(SPATCH) $(SPATCH_FLAGS) \\\n+\t$(QUIET_SPATCH_T)$(SPATCH) $(SPATCH_TEST_FLAGS) \\\n \t\t--very-quiet --no-show-diff \\\n \t\t--sp-file $< -o $@ \\\n \t\t$(@:.build/%.res=%.c) && \\\n-- \n2.37.3.1420.g76f8a3d556c\n\n"},{"id":"462326","messageId":"patch-v2-6.9-c74d09f4825-20220831T205130Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v2-0.9-00000000000-20220831T205130Z-avarab@gmail.com","subject":"[PATCH v2 6/9] cocci: split off \"--all-includes\" from SPATCH_FLAGS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-31T20:57:53Z","receivedAt":"2022-08-31T20:58:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Per the rationale in 7b63ea57500 (Makefile: remove mandatory \"spatch\"\narguments from SPATCH_FLAGS, 2022-07-05) we have certain flags that\nare truly mandatory, such as \"--sp-file\" and \"--patch .\". The\n\"--all-includes\" flag is also critical, but per [1] we might want to\nad-hoc tweak it occasionally for testing or one-offs.\n\nBut being unable to set e.g. SPATCH_FLAGS=\"--verbose-parsing\" without\nbreaking how our \"spatch\" works isn't ideal, i.e. before this we'd\nneed to know about the default include flags, and specify:\nSPATCH_FLAGS=\"--all-includes --verbose-parsing\".\n\nIf we were then to change the default include flag (e.g. to\n\"--recursive-includes\") in the future any such one-off commands would\nneed to be correspondingly updated.\n\nLet's instead leave the SPATCH_FLAGS for the user, while creating a\nnew SPATCH_INCLUDE_FLAGS to allow for ad-hoc testing of the include\nstrategy itself.\n\n1. https://lore.kernel.org/git/20220823095733.58685-1-szeder.dev@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex 143c012b3be..662f950e075 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1295,7 +1295,8 @@ SANITIZE_LEAK =\n SANITIZE_ADDRESS =\n \n # For the 'coccicheck' target\n-SPATCH_FLAGS = --all-includes\n+SPATCH_INCLUDE_FLAGS = --all-includes\n+SPATCH_FLAGS =\n SPATCH_TEST_FLAGS =\n # Setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n@@ -1305,6 +1306,7 @@ SPATCH_BATCH_SIZE = 1\n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n TRACK_SPATCH_DEFINES += $(SPATCH)\n+TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n@@ -3172,6 +3174,7 @@ $(COCCI_PATCHES): %.patch: %\n \tfi; \\\n \tif ! echo $(COCCI_SOURCES) | xargs $$limit \\\n \t\t$(SPATCH) $(SPATCH_FLAGS) \\\n+\t\t$(SPATCH_INCLUDE_FLAGS) \\\n \t\t--sp-file $< --patch . \\\n \t\t>$@+ 2>$@.log; \\\n \tthen \\\n-- \n2.37.3.1420.g76f8a3d556c\n\n"},{"id":"462327","messageId":"patch-v2-8.9-536dce45eef-20220831T205130Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v2-0.9-00000000000-20220831T205130Z-avarab@gmail.com","subject":"[PATCH v2 8/9] cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-31T20:57:55Z","receivedAt":"2022-08-31T20:58:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improve the incremental rebuilding support of \"coccicheck\" by\npiggy-backing on the computed dependency information of the\ncorresponding *.o file, rather than rebuilding all / pairs\nif either their corresponding file changes, or if any header changes.\n\nThis in effect uses the same method that the \"sparse\" target was made\nto use in c234e8a0ecf (Makefile: make the \"sparse\" target non-.PHONY,\n2021-09-23), except that the dependency on the *.o file isn't a hard\none, we check with $(wildcard) if the *.o file exists, and if so we'll\ndepend on it.\n\nThis means that the common case of:\n\n\tmake\n\tmake coccicheck\n\nWill benefit from incremental rebuilding, now changing e.g. a header\nwill only re-run \"spatch\" on those those *.c files that make use of\nit:\n\nBy depending on the *.o we piggy-back on\nCOMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the\n\"sparse\" target non-.PHONY, 2021-09-23) for prior art of doing that\nfor the *.sp files. E.g.:\n\n make contrib/coccinelle/free.cocci.patch\n make -W column.h contrib/coccinelle/free.cocci.patch\n\nWill take around 15 seconds for the second command on my 8 core box if\nI didn't run \"make\" beforehand to create the *.o files. But around 2\nseconds if I did and we have those \"*.o\" files.\n\nNotes about the approach of piggy-backing on *.o for dependencies:\n\n * It *is* a trade-off since we'll pay the extra cost of running the C\n compiler, but we're probably doing that anyway. The compiler is much\n faster than \"spatch\", so even though we need to re-compile the *.o to\n create the dependency info for the *.c for \"spatch\" it's\n faster (especially if using \"ccache\").\n\n * There *are* use-cases where some would like to have *.o files\n around, but to have the \"make coccicheck\" ignore them. See:\n https://lore.kernel.org/git/20220826104312.GJ1735@szeder.dev/\n\n For those users a:\n\n\tmake\n\tmake coccicheck SPATCH_USE_O_DEPENDENCIES=\n\n Will avoid considering the *.o files.\n\n * If that *.o file doesn't exist we'll depend on an intermediate file\n of ours which in turn depends on $(FOUND_H_SOURCES).\n\n This covers both an initial build, or where \"coccicheck\" is run\n without running \"all\" beforehand, and because we run \"coccicheck\"\n on e.g. files in compat/* that we don't know how to build unless\n the requisite flag was provided to the Makefile.\n\n Most of the runtime of \"incremental\" runs is now spent on various\n compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and\n therefore conflate whether we *can* compile an object and generate\n dependency information for it with whether we'd like to link it\n into our binary.\n\n Before this change the distinction didn't matter, but now one way\n to make this even faster on incremental builds would be to peel\n those concerns apart so that we can see that e.g. compat/mmap.c\n doesn't depend on column.h.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 562820301c4..9e2ef8f0dc4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3170,14 +3170,20 @@ COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \t$(call mkdir_p_parent_template)\n \t$(QUIET_GEN) >$@\n \n+SPATCH_USE_O_DEPENDENCIES = yes\n+ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no)\n+SPATCH_USE_O_DEPENDENCIES =\n+endif\n+\n define cocci-rule\n \n ## Rule for .build/$(1).patch/$(2); Params:\n # $(1) = e.g. \"free.cocci\"\n # $(2) = e.g. \"grep.c\"\n+# $(3) = e.g. \"grep.o\"\n COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n .build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n-.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES\n+.build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES)\n .build/$(1).patch/$(2): $(1)\n .build/$(1).patch/$(2): .build/$(1).patch/% : %\n \t$$(call mkdir_p_parent_template)\n@@ -3194,7 +3200,7 @@ endef\n \n define cocci-matrix-2\n \n-$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s)))\n+$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o)))\n endef\n define cocci-matrix-1\n $(foreach c,$(COCCI),$(call cocci-matrix-2,$(c)))\n-- \n2.37.3.1420.g76f8a3d556c\n\n"},{"id":"462328","messageId":"patch-v2-7.9-120607b5da6-20220831T205130Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v2-0.9-00000000000-20220831T205130Z-avarab@gmail.com","subject":"[PATCH v2 7/9] cocci: make \"coccicheck\" rule incremental","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-31T20:57:54Z","receivedAt":"2022-08-31T20:58:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Optimize the very slow \"coccicheck\" target to take advantage of\nincremental rebuilding, and fix outstanding dependency problems with\nthe existing rule.\n\nThe rule is now faster both on the initial run as we can make better\nuse of GNU make's parallelism than the old ad-hoc combination of\nmake's parallelism combined with $(SPATCH_BATCH_SIZE) and/or the\n\"--jobs\" argument to \"spatch(1)\".\n\nIt also makes us *much* faster when incrementally building, it's now\nviable to \"make coccicheck\" as topic branches are merged down.\n\nThe rule didn't use FORCE (or its equivalents) before, so a:\n\n\tmake coccicheck\n\tmake coccicheck\n\nWould report nothing to do on the second iteration. But all of our\npatch output depended on all $(COCCI_SOURCES) files, therefore e.g.:\n\n make -W grep.c coccicheck\n\nWould do a full re-run, i.e. a a change in a single file would force\nus to do a full re-run.\n\nThe reason for this (not the initial rationale, but my analysis) is:\n\n* Since we create a single \"*.cocci.patch+\" we don't know where to\n pick up where we left off, or how to incrementally merge e.g. a\n \"grep.c\" change with an existing *.cocci.patch.\n\n* We've been carrying forward the dependency on the *.c files since\n 63f0a758a06 (add coccicheck make target, 2016-09-15) the rule was\n initially added as a sort of poor man's dependency discovery.\n\n As we don't include other *.c files depending on other *.c files\n has always been broken, as could be trivially demonstrated\n e.g. with:\n\n make coccicheck\n make -W strbuf.h coccicheck\n\n However, depending on the corresponding *.c files has been doing\n something, namely that *if* an API change modified both *.c and *.h\n files we'd catch the change to the *.h we care about via the *.c\n being changed.\n\n For API changes that happened only via *.h files we'd do the wrong\n thing before this change, but e.g. for function additions (not\n \"static inline\" ones) catch the *.h change by proxy.\n\nNow we'll instead:\n\n * Create a / pair in the .build directory, E.g. for\n swap.cocci and grep.c we'll create\n .build/contrib/coccinelle/swap.cocci.patch/grep.c.\n\n That file is the diff we'll apply for that -\n combination, if there's no changes to me made (the common case)\n it'll be an empty file.\n\n * Our generated *.patch\n file (e.g. contrib/coccinelle/swap.cocci.patch) is now a simple \"cat\n $^\" of all of all of the / files for a given .\n\n In the case discussed above of \"grep.c\" being changed we'll do the\n full \"cat\" every time, so they resulting *.cocci.patch will always\n be correct and up-to-date, even if it's \"incrementally updated\".\n\n See 1cc0425a27c (Makefile: have \"make pot\" not \"reset --hard\",\n 2022-05-26) for another recent rule that used that technique.\n\nAs before we'll:\n\n * End up generating a contrib/coccinelle/swap.cocci.patch, if we\n \"fail\" by creating a non-empty patch we'll still exit with a zero\n exit code.\n\n Arguably we should move to a more Makefile-native way of doing\n this, i.e. fail early, and if we want all of the \"failed\" changes\n we can use \"make -k\", but as the current\n \"ci/run-static-analysis.sh\" expects us to behave this way let's\n keep the existing behavior of exhaustively discovering all cocci\n changes, and only failing if spatch itself errors out.\n\nFurther implementation details & notes:\n\n * Before this change running \"make coccicheck\" would by default end\n up pegging just one CPU at the very end for a while, usually as\n we'd finish whichever *.cocci rule was the most expensive.\n\n This could be mitigated by combining \"make -jN\" with\n SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch\n spatch invocations, 2019-05-06). But doing so required careful\n juggling, as e.g. setting both to 4 would yield 16 workers.\n\n There will be cases where getting rid of \"SPATCH_BATCH_SIZE\" makes\n things worse, but a from-scratch \"make coccicheck\" with the default\n of SPATCH_BATCH_SIZE=1 (and tweaking it doesn't make a difference)\n is faster (~3m36s v.s. ~3m56s) with this approach, as we can feed\n the CPU more work in a less staggered way.\n\n * Getting rid of \"SPATCH_BATCH_SIZE\" particularly helps in cases\n where the default of 1 yields parallelism under \"make coccicheck\",\n but then running e.g.:\n\n make -W contrib/coccinelle/swap.cocci coccicheck\n\n I.e. before that would use only one CPU core, until the user\n remembered to adjust \"SPATCH_BATCH_SIZE\" differently than the\n setting that makes sense when doing a non-incremental run of \"make\n coccicheck\".\n\nOutstanding issues & future work:\n\n * We could get rid of \"--all-includes\" in favor of manually\n specifying a list of includes to give to \"spatch(1)\".\n\n As noted upthread of [1] a naïve removal of \"--all-includes\" will\n result in broken *.cocci patches, but if we know the exhaustive\n list of includes via COMPUTE_HEADER_DEPENDENCIES we don't need to\n re-scan for them, we could grab the headers to include from the\n .depend.d/.o.d and supply them with the \"--include\" option to\n spatch(1).q\n\n1. https://lore.kernel.org/git/87ft18tcog.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 83 +++++++++++++++++++++++++++++++++++++-----------------\n shared.mak | 3 +-\n 2 files changed, 59 insertions(+), 27 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 662f950e075..562820301c4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1298,10 +1298,6 @@ SANITIZE_ADDRESS =\n SPATCH_INCLUDE_FLAGS = --all-includes\n SPATCH_FLAGS =\n SPATCH_TEST_FLAGS =\n-# Setting SPATCH_BATCH_SIZE higher will\n-# usually result in less CPU usage at the cost of higher peak memory.\n-# Setting it to 0 will feed all files in a single spatch invocation.\n-SPATCH_BATCH_SIZE = 1\n \n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n@@ -1309,7 +1305,6 @@ TRACK_SPATCH_DEFINES += $(SPATCH)\n TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS)\n-TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n GIT-SPATCH-DEFINES: FORCE\n \t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n \t if test x\"$$FLAGS\" != x\"`cat GIT-SPATCH-DEFINES 2>/dev/null`\" ; then \\\n@@ -3156,36 +3151,72 @@ check: $(GENERATED_H)\n \t\texit 1; \\\n \tfi\n \n+# It's expensive to compute the many=many rules below, only eval them\n+# on $(MAKECMDGOALS) that match these $(COCCI_RULES)\n+COCCI_RULES =\n+COCCI_RULES += cocci%\n+COCCI_RULES += contrib/coccinelle/%\n+COCCI_RULES += .build/contrib/coccinelle/%\n+\n COCCI = $(wildcard contrib/coccinelle/*.cocci)\n+COCCI_NAMES = $(COCCI:contrib/coccinelle/%.cocci=%)\n COCCI_PATCHES = $(addsuffix .patch,$(COCCI))\n COCCICHECK_PENDING = $(filter %.pending.cocci.patch,$(COCCI_PATCHES))\n-COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES))\n \n+COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES))\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n-$(COCCI_PATCHES): GIT-SPATCH-DEFINES\n-$(COCCI_PATCHES): $(COCCI_SOURCES)\n-$(COCCI_PATCHES): %.patch: %\n-\t$(QUIET_SPATCH) \\\n-\tif test $(SPATCH_BATCH_SIZE) = 0; then \\\n-\t\tlimit=; \\\n-\telse \\\n-\t\tlimit='-n $(SPATCH_BATCH_SIZE)'; \\\n-\tfi; \\\n-\tif ! echo $(COCCI_SOURCES) | xargs $$limit \\\n-\t\t$(SPATCH) $(SPATCH_FLAGS) \\\n-\t\t$(SPATCH_INCLUDE_FLAGS) \\\n-\t\t--sp-file $< --patch . \\\n-\t\t>$@+ 2>$@.log; \\\n+.build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES)\n+\t$(call mkdir_p_parent_template)\n+\t$(QUIET_GEN) >$@\n+\n+define cocci-rule\n+\n+## Rule for .build/$(1).patch/$(2); Params:\n+# $(1) = e.g. \"free.cocci\"\n+# $(2) = e.g. \"grep.c\"\n+COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n+.build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n+.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES\n+.build/$(1).patch/$(2): $(1)\n+.build/$(1).patch/$(2): .build/$(1).patch/% : %\n+\t$$(call mkdir_p_parent_template)\n+\t$$(QUIET_SPATCH)if ! $$(SPATCH) $$(SPATCH_FLAGS) \\\n+\t\t$$(SPATCH_INCLUDE_FLAGS) \\\n+\t\t--sp-file $(1) --patch . $$< \\\n+\t\t>$$@ 2>$$@.log; \\\n \tthen \\\n-\t\tcat $@.log; \\\n+\t\techo \"ERROR when applying '$(1)' to '$$<'; '$$@.log' follows:\"; \\\n+\t\tcat $$@.log; \\\n \t\texit 1; \\\n-\tfi; \\\n-\tmv $@+ $@; \\\n-\tif test -s $@; \\\n+\tfi\n+endef\n+\n+define cocci-matrix-2\n+\n+$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s)))\n+endef\n+define cocci-matrix-1\n+$(foreach c,$(COCCI),$(call cocci-matrix-2,$(c)))\n+endef\n+\n+ifneq ($(filter $(COCCI_RULES),$(MAKECMDGOALS)),)\n+$(eval $(call cocci-matrix-1))\n+endif\n+\n+define spatch-rule\n+\n+contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1))\n+\t$(QUIET_SPATCH_M)cat $$^ >$$@ && \\\n+\tif test -s $$@; \\\n \tthen \\\n-\t\techo ' ' SPATCH result: $@; \\\n+\t\techo ' ' SPATCH result: $$@; \\\n \tfi\n+endef\n+\n+ifneq ($(filter $(COCCI_RULES),$(MAKECMDGOALS)),)\n+$(eval $(foreach n,$(COCCI_NAMES),$(call spatch-rule,$(n))))\n+endif\n \n COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n@@ -3476,7 +3507,7 @@ profile-clean:\n \n cocciclean:\n \t$(RM) -r .build/contrib/coccinelle\n-\t$(RM) $(COCCI_PATCHES)*\n+\t$(RM) $(COCCI_PATCHES)\n \n clean: profile-clean coverage-clean cocciclean\n \t$(RM) -r .build\ndiff --git a/shared.mak b/shared.mak\nindex 33f43edbf9a..d9fdd6b4484 100644\n--- a/shared.mak\n+++ b/shared.mak\n@@ -69,8 +69,9 @@ ifndef V\n \tQUIET_SP = @echo ' ' SP $<;\n \tQUIET_HDR = @echo ' ' HDR $(<:hcc=h);\n \tQUIET_RC = @echo ' ' RC $@;\n-\tQUIET_SPATCH = @echo ' ' SPATCH $<;\n+\tQUIET_SPATCH = @echo ' ' SPATCH $@;\n \tQUIET_SPATCH_T = @echo ' ' SPATCH TEST $(@:.build/%=%);\n+\tQUIET_SPATCH_M = @echo ' ' SPATCH MERGE $$@;\n \n ## Used in \"Documentation/Makefile\"\n \tQUIET_ASCIIDOC\t= @echo ' ' ASCIIDOC $@;\n-- \n2.37.3.1420.g76f8a3d556c\n\n"},{"id":"462329","messageId":"patch-v2-9.9-2b978676a56-20220831T205130Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v2-0.9-00000000000-20220831T205130Z-avarab@gmail.com","subject":"[PATCH v2 9/9] spatchcache: add a ccache-alike for \"spatch\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-31T20:57:56Z","receivedAt":"2022-08-31T20:58:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a rather trivial \"spatchcache\", with this running e.g.:\n\n\tmake cocciclean\n\tmake contrib/coccinelle/free.cocci.patch \\\n\t\tSPATCH=contrib/coccicheck/spatchcache \\\n\t\tSPATCH_FLAGS=--very-quiet\n\nIs cut down from ~20s to ~5s on my system. Much of that is either\nfixable shell overhead, or the around 40 files we \"CANTCACHE\" (see the\nimplementation).\n\nThis uses \"redis\" as a cache by default, but it's configurable. See\nthe embedded documentation.\n\nThis is *not* like ccache in that we won't cache failed spatch\ninvocations, or those where spatch suggests changes for us. Those\ncases are so rare that I didn't think it was worth the bother, by far\nthe most common case is that it has no suggested changes. We'll also\nrefuse to cache any \"spatch\" invocation that has output on stderr,\nwhich means that \"--very-quiet\" must be added to \"SPATCH_FLAGS\".\n\nBecause we narrow the cache to that we don't need to save away stdout,\nstderr & the exit code. We simply cache the cases where we had no\nsuggested changes.\n\nAnother benchmark is to compare this with the previous\nSPATCH_BATCH_SIZE=N, as noted in [1]. Before this (on my 8 core system) running:\n\n\tmake clean; time make contrib/coccinelle/array.cocci.patch SPATCH_BATCH_SIZE=0\n\nWould take 33s, but with the preceding changes running without this\n\"spatchcache\" is slightly slower, or around 35s:\n\n\tmake clean; time make contrib/coccinelle/array.cocci.patch\n\nNow doing the same with SPATCH=contrib/coccinelle/spatchcache will\ntake around 6s, but we'll need to compile the *.o files first to take\nfull advantage of it (which can be fast with \"ccache\"):\n\n\tmake clean; make; time make contrib/coccinelle/array.cocci.patch SPATCH=contrib/coccinelle/spatchcache\n\n1. https://lore.kernel.org/git/YwdRqP1CyUAzCEn2@coredump.intra.peff.net/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n contrib/coccinelle/spatchcache | 234 +++++++++++++++++++++++++++++++++\n 1 file changed, 234 insertions(+)\n create mode 100755 contrib/coccinelle/spatchcache\n\ndiff --git a/contrib/coccinelle/spatchcache b/contrib/coccinelle/spatchcache\nnew file mode 100755\nindex 00000000000..f0b7b24e0ae\n--- /dev/null\n+++ b/contrib/coccinelle/spatchcache\n@@ -0,0 +1,234 @@\n+#!/bin/sh\n+#\n+# spatchcache: a poor-man's \"ccache\"-alike for \"spatch\" in git.git\n+#\n+# This caching command relies on the peculiarities of the Makefile\n+# driving \"spatch\" in git.git, in particular if we invoke:\n+#\n+#\tmake\n+#\tmake coccicheck SPATCH_FLAGS=--very-quiet\n+#\n+# We can with COMPUTE_HEADER_DEPENDENCIES (auto-detected as true with\n+# \"gcc\" and \"clang\") write e.g. a .depend/grep.o.d for grep.c, when we\n+# compile grep.o.\n+#\n+# The .depend/grep.o.d will have the full header dependency tree of\n+# grep.c, and we can thus cache the output of \"spatch\" by:\n+#\n+#\t1. Hashing all of those files\n+#\t2. Hashing our source file, and the *.cocci rule we're\n+#\t applying\n+#\t3. Running spatch, if suggests no changes (by far the common\n+#\t case) we invoke \"spatchCache.getCmd\" and\n+#\t \"spatchCache.setCmd\" with a hash SHA-256 to ask \"does this\n+#\t ID have no changes\" or \"say that ID had no changes>\n+#\t4. If no \"spatchCache.{set,get}Cmd\" is specified we'll use\n+#\t \"redis-cli\" and maintain a SET called \"spatch-cache\". Set\n+#\t appropriate redis memory policies to keep it from growing\n+#\t out of control.\n+#\n+# This along with the general incremental \"make\" support for\n+# \"contrib/coccinelle\" makes it viable to (re-)run coccicheck\n+# e.g. when merging integration branches.\n+#\n+# Note that the \"--very-quiet\" flag is currently critical. The cache\n+# will refuse to cache anything that has output on STDERR (which might\n+# be errors from spatch). The STDERR (and exit code) could in\n+# principle be cached (as with ccache), but then the simple structure\n+# in the Redis cache would need to change, so just supply\n+# \"--very-quiet\" for now.\n+#\n+# To use this, simply set SPATCH to\n+# contrib/coccinelle/spatchcache. Then optionally set:\n+#\n+#\t[spatchCache]\n+#\t\t# Optional: path to a custom spatch\n+#\t\tspatch = ~/g/coccicheck/spatch.opt\n+#\n+# As well as this trace config (debug implies trace):\n+#\n+#\t\ttrace = false\n+#\t\tdebug = false\n+#\n+# Setting \"trace\" to \"true\" allows for seeing when we have a cache HIT\n+# or MISS. To debug whether the cache is working do that, and run e.g.:\n+#\n+#\tredis-cli FLUSHALL\n+#\t\n+#\tgrep -hore HIT -e MISS -e SET -e NOCACHE -e CANTCACHE .build/contrib/coccinelle | sort | uniq -c\n+#\t 600 CANTCACHE\n+#\t 7365 MISS\n+#\t 7365 SET\n+#\n+# A subsequent \"make cocciclean && make coccicheck\" should then have\n+# all \"HIT\"'s and \"CANTCACHE\"'s.\n+\n+set -e\n+\n+## Our own configuration & options\n+debug=$(git config --bool \"spatchCache.debug\")\n+if test \"$debug\" != \"true\"\n+then\n+\tdebug=\n+fi\n+if test -n \"$debug\"\n+then\n+\tset -x\n+fi\n+\n+trace=$(git config --bool \"spatchCache.trace\")\n+if test \"$trace\" != \"true\"\n+then\n+\ttrace=\n+fi\n+if test -n \"$debug\"\n+then\n+\t# debug implies trace\n+\ttrace=true\n+fi\n+\n+trace_it () {\n+\tif test -z \"$trace\"\n+\tthen\n+\t\treturn\n+\tfi\n+\techo \"$@\" >&2\n+}\n+\n+spatch=$(git config --path \"spatchCache.spatch\" || :)\n+if test -n \"$spatch\"\n+then\n+\tif test -n \"$debug\"\n+\tthen\n+\t\ttrace_it \"custom spatchCache.spatch='$spatch'\"\n+\tfi\n+else\n+\tspatch=spatch\n+fi\n+\n+set=$(git config spatchCache.setCmd || :)\n+get=$(git config spatchCache.getCmd || :)\n+\n+## Parse spatch()-like command-line for caching info\n+arg_sp=\n+arg_file=\n+args=\"$@\"\n+spatch_opts() {\n+\twhile test $# != 0\n+\tdo\n+\t\targ_file=\"$1\"\n+\t\tcase \"$1\" in\n+\t\t--sp-file)\n+\t\t\targ_sp=\"$2\"\n+\t\t\t;;\n+\t\tesac\n+\t\tshift\n+\tdone\n+}\n+spatch_opts \"$@\"\n+if ! test -f \"$arg_file\"\n+then\n+\targ_file=\n+fi\n+\n+hash_for_cache() {\n+\t# Parameters that should affect the cache\n+\techo \"spatch=$spatch\"\n+\techo \"args=$args\"\n+\techo\n+\n+\t# Our target file and its dependencies\n+\tgit hash-object \"$1\" \"$2\" $(grep -E -o '^[^:]+:$' \"$3\" | tr -d ':')\n+}\n+\n+# Sanity checks\n+if ! test -f \"$arg_sp\" && ! test -f \"$arg_file\"\n+then\n+\techo $0: no idea how to cache \"$@\" >&2\n+\texit 128\n+fi\n+\n+# Main logic\n+d=$(dirname \"$arg_file\")\n+b=$(basename \"$arg_file\")\n+bwoext=\"${b%.c}\"\n+dep=\"$d/.depend/$bwoext.o.d\"\n+\n+if ! test -f \"$dep\"\n+then\n+\ttrace_it \"$0: CANTCACHE have no '$dep' for '$arg_file'!\"\n+\texec \"$spatch\" \"$@\"\n+fi\n+\n+if test -n \"$debug\"\n+then\n+\ttrace_it \"$0: The full cache input for '$arg_sp' '$arg_file' '$dep'\"\n+\thash_for_cache \"$arg_sp\" \"$arg_file\" \"$dep\" >&2\n+fi\n+sum=$(hash_for_cache \"$arg_sp\" \"$arg_file\" \"$dep\" | git hash-object --stdin)\n+\n+trace_it \"$0: processing '$arg_file' with '$arg_sp' rule, and got hash '$sum' for it + '$dep'\"\n+\n+getret=\n+if test -z \"$get\"\n+then\n+\tif test $(redis-cli SISMEMBER spatch-cache \"$sum\") = 1\n+\tthen\n+\t\tgetret=0\n+\telse\n+\t\tgetret=1\n+\tfi\n+else\n+\t$set \"$sum\"\n+\tgetret=$?\n+fi\n+\n+if test \"$getret\" = 0\n+then\n+\ttrace_it \"$0: HIT for '$arg_file' with '$arg_sp'\"\n+\texit 0\n+else\n+\ttrace_it \"$0: MISS: for '$arg_file' with '$arg_sp'\"\n+fi\n+\n+out=\"$(mktemp)\"\n+err=\"$(mktemp)\"\n+\n+set +e\n+\"$spatch\" \"$@\" >\"$out\" 2>\"$err\"\n+ret=$?\n+set -e\n+\n+if test $ret = 0 && ! test -s \"$out\" && ! test -s \"$err\"\n+then\n+\trm -f \"$out\" \"$err\"\n+\n+\ttrace_it \"$0: SET: for '$arg_file' with '$arg_sp'\"\n+\n+\tsetret=\n+\tif test -z \"$set\"\n+\tthen\n+\t\tif test $(redis-cli SADD spatch-cache \"$sum\") = 1\n+\t\tthen\n+\t\t\tsetret=0\n+\t\telse\n+\t\t\tsetret=1\n+\t\tfi\n+\telse\n+\t\t\"$set\" \"$sum\"\n+\t\tsetret=$?\n+\tfi\n+\n+\tif test \"$setret\" != 0\n+\tthen\n+\t\techo \"FAILED to set '$sum' in cache!\" >&2\n+\t\texit 128\n+\tfi\n+else\n+\ttrace_it \"$0: NOCACHE (have changes): for '$arg_file' with '$arg_sp'\"\n+\tcat \"$out\"\n+\tcat \"$err\" >&2\n+\trm -f \"$out\" \"$err\"\n+\texit \"$ret\"\n+fi\n+rm -f \"$out\" \"$err\"\n-- \n2.37.3.1420.g76f8a3d556c\n\n"},{"id":"462469","messageId":"20220901163859.GA122568@szeder.dev","threadId":"58360","inReplyTo":"patch-v2-7.9-120607b5da6-20220831T205130Z-avarab@gmail.com","subject":"Re: [PATCH v2 7/9] cocci: make \"coccicheck\" rule incremental","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2022-09-01T16:38:59Z","receivedAt":"2022-09-01T16:39:21Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Aug 31, 2022 at 10:57:54PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Optimize the very slow \"coccicheck\" target to take advantage of\n> incremental rebuilding, and fix outstanding dependency problems with\n> the existing rule.\n> \n> The rule is now faster both on the initial run as we can make better\n> use of GNU make's parallelism than the old ad-hoc combination of\n> make's parallelism combined with $(SPATCH_BATCH_SIZE) and/or the\n> \"--jobs\" argument to \"spatch(1)\".\n\nOn my machine a \"from scratch\" 'make -j4 coccicheck' without the\ninacceptably slow 'unused.cocci' takes 9m28s, with\nSPATCH_BATCH_SIZE=32 it takes 6m39s. If we invoke 'spatch' like in\nthe patch below and let one 'spatch' invocation process all source\nfiles one by one (i.e. unlike the batched invocations) and using its\n'--jobs' option then it takes 4m56s.\n\nThis patch series is slower than any of the above, as it takes 10m3s.\n\n> * Before this change running \"make coccicheck\" would by default end\n> up pegging just one CPU at the very end for a while, usually as\n> we'd finish whichever *.cocci rule was the most expensive.\n> \n> This could be mitigated by combining \"make -jN\" with\n> SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch\n> spatch invocations, 2019-05-06). But doing so required careful\n> juggling, as e.g. setting both to 4 would yield 16 workers.\n\nAs pointed out previously, this is not the case.\n\n\n --- >8 ---\n\ndiff --git a/Makefile b/Makefile\nindex e8adeb09f1..b93b1b62e1 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1290,11 +1290,9 @@ SP_EXTRA_FLAGS = -Wno-universal-initializer\n SANITIZE_LEAK =\n SANITIZE_ADDRESS =\n \n-# For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n-# usually result in less CPU usage at the cost of higher peak memory.\n-# Setting it to 0 will feed all files in a single spatch invocation.\n+# For the 'coccicheck' target\n SPATCH_FLAGS = --all-includes\n-SPATCH_BATCH_SIZE = 1\n+SPATCH_JOBS = 1\n \n include config.mak.uname\n -include config.mak.autogen\n@@ -3139,19 +3137,17 @@ COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n %.cocci.patch: %.cocci $(COCCI_SOURCES)\n \t$(QUIET_SPATCH) \\\n-\tif test $(SPATCH_BATCH_SIZE) = 0; then \\\n-\t\tlimit=; \\\n-\telse \\\n-\t\tlimit='-n $(SPATCH_BATCH_SIZE)'; \\\n-\tfi; \\\n-\tif ! echo $(COCCI_SOURCES) | xargs $$limit \\\n-\t\t$(SPATCH) $(SPATCH_FLAGS) \\\n-\t\t--sp-file $< --patch . \\\n+\tfilelist=$@.filelist; \\\n+\t: Blank lines between filenames are necessary to put each file in separate group ; \\\n+\tprintf \"%s\\\\n\\\\n\" $(COCCI_SOURCES) >$$filelist && \\\n+\tif ! $(SPATCH) $(SPATCH_FLAGS) --jobs $(SPATCH_JOBS) \\\n+\t\t--sp-file $< --patch . --file-groups $$filelist \\\n \t\t>$@+ 2>$@.log; \\\n \tthen \\\n \t\tcat $@.log; \\\n \t\texit 1; \\\n \tfi; \\\n+\trm -f $$filelist; \\\n \tmv $@+ $@; \\\n \tif test -s $@; \\\n \tthen \\\n"},{"id":"462481","messageId":"220901.86y1v3hrii.gmgdl@evledraar.gmail.com","threadId":"58360","inReplyTo":"20220901163859.GA122568@szeder.dev","subject":"Re: [PATCH v2 7/9] cocci: make \"coccicheck\" rule incremental","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-01T18:04:45Z","receivedAt":"2022-09-01T18:36:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Sep 01 2022, SZEDER Gábor wrote:\n\n> On Wed, Aug 31, 2022 at 10:57:54PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> Optimize the very slow \"coccicheck\" target to take advantage of\n>> incremental rebuilding, and fix outstanding dependency problems with\n>> the existing rule.\n>> \n>> The rule is now faster both on the initial run as we can make better\n>> use of GNU make's parallelism than the old ad-hoc combination of\n>> make's parallelism combined with $(SPATCH_BATCH_SIZE) and/or the\n>> \"--jobs\" argument to \"spatch(1)\".\n>\n> On my machine a \"from scratch\" 'make -j4 coccicheck' without the\n> inacceptably slow 'unused.cocci' takes 9m28s, with\n> SPATCH_BATCH_SIZE=32 it takes 6m39s. If we invoke 'spatch' like in\n> the patch below and let one 'spatch' invocation process all source\n> files one by one (i.e. unlike the batched invocations) and using its\n> '--jobs' option then it takes 4m56s.\n>\n> This patch series is slower than any of the above, as it takes 10m3s.\n\nBut is such from-scratch building your primary use-case?\n\nWhen I e.g. do a \"make coccicheck\" on master, then:\n\n\tgit merge gitster/ac/bitmap-lookup-table\n\nOr whatever, it takes much less time than before, ditto things like:\n\n\ttime make -W grep.h contrib/coccinelle/free.cocci.patch\n\nI.e. changing a file and wanting to re-run it for all of its reverse\ndependencies (this is with the *.o files).\n\n>> * Before this change running \"make coccicheck\" would by default end\n>> up pegging just one CPU at the very end for a while, usually as\n>> we'd finish whichever *.cocci rule was the most expensive.\n>> \n>> This could be mitigated by combining \"make -jN\" with\n>> SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch\n>> spatch invocations, 2019-05-06). But doing so required careful\n>> juggling, as e.g. setting both to 4 would yield 16 workers.\n>\n> As pointed out previously, this is not the case.\n\nYes we discussed this in v1, I just missed this in the re-roll. Sorry!\nWill fix it in the next iteration.\n\n> include config.mak.uname\n> -include config.mak.autogen\n> @@ -3139,19 +3137,17 @@ COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n> \n> %.cocci.patch: %.cocci $(COCCI_SOURCES)\n> \t$(QUIET_SPATCH) \\\n> -\tif test $(SPATCH_BATCH_SIZE) = 0; then \\\n> -\t\tlimit=; \\\n> -\telse \\\n> -\t\tlimit='-n $(SPATCH_BATCH_SIZE)'; \\\n> -\tfi; \\\n> -\tif ! echo $(COCCI_SOURCES) | xargs $$limit \\\n> -\t\t$(SPATCH) $(SPATCH_FLAGS) \\\n> -\t\t--sp-file $< --patch . \\\n> +\tfilelist=$@.filelist; \\\n> +\t: Blank lines between filenames are necessary to put each file in separate group ; \\\n> +\tprintf \"%s\\\\n\\\\n\" $(COCCI_SOURCES) >$$filelist && \\\n> +\tif ! $(SPATCH) $(SPATCH_FLAGS) --jobs $(SPATCH_JOBS) \\\n> +\t\t--sp-file $< --patch . --file-groups $$filelist \\\n\nYes, I think it's unfortunate that there *are* faster ways to run it,\nbut they rely on doing things in batches of N, which doesn't really jive\nwell with make's incremental model of the world.\n\nBut I think we can either have that, or the ability to incrementally\n(re)build.\n\nDo you see some easy way to have our cake & eat it too here? If\nyou/others really feel strongly about this use case I could also keep\nthe old method, and just hide this all behind a SPATCH_INCREMENTAL=Y\nflag or something. But I'm hoping we can turn this (mostly) into some\nwin-win.\n"},{"id":"464937","messageId":"cover-v3-00.11-00000000000-20221014T152552Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v2-0.9-00000000000-20220831T205130Z-avarab@gmail.com","subject":"[PATCH v3 00/11] cocci: make \"incremental\" possible + a ccache-like tool","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-14T15:31:16Z","receivedAt":"2022-10-14T15:32:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A re-roll of the series to have \"make coccicheck\" run\nincrementally. For the v1 see:\nhttps://lore.kernel.org/git/cover-0.5-00000000000-20220825T141212Z-avarab@gmail.com/\n\nThe big change in this belated v3 is that we now run against a\nconcatenated version of all our *.cocci files. This is something I\ndiscussed with Jeff King at Git Merge, after having confirmed the\nviability of that approach on the cocci mailing list.\n\nThe result is that even with a from-scratch run of \"make coccicheck\"\nthis is faster than on \"master\", while of course adding the ability to\nrun incrementally, so a subsequent \"coccicheck\" is only run against\nwhat we'd recompile.\n\nThe churn here is because I had to move/change some variables around\nearly in the series to make the eventual 08/11 smaller.\n\nThere's also improvements here to the bundled \"spatchcache\", it can\nnow optionally interoperate with non--very-quiet output (but won't\ncache stderr), which is handy for debugging it. It can also be\nconfigured to tell it where the .depend/* files are, which is a small\nchange to make it posible to use it outside of git.git..\n\nÆvar Arnfjörð Bjarmason (11):\n Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)\n cocci rules: remove unused \"F\" metavariable from pending rule\n Makefile: add ability to TAB-complete cocci *.patch rules\n Makefile: have \"coccicheck\" re-run if flags change\n Makefile: split off SPATCH_BATCH_SIZE comment from \"cocci\" heading\n cocci: split off include-less \"tests\" from SPATCH_FLAGS\n cocci: split off \"--all-includes\" from SPATCH_FLAGS\n cocci: make \"coccicheck\" rule incremental\n cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES\n cocci: run against a generated ALL.cocci\n spatchcache: add a ccache-alike for \"spatch\"\n\n .gitignore | 1 +\n Makefile | 159 ++++++++--\n contrib/coccinelle/.gitignore | 1 +\n contrib/coccinelle/spatchcache | 272 ++++++++++++++++++\n .../coccinelle/the_repository.pending.cocci | 1 -\n shared.mak | 10 +-\n 6 files changed, 416 insertions(+), 28 deletions(-)\n create mode 100755 contrib/coccinelle/spatchcache\n\nRange-diff against v2:\n -: ----------- > 1: 4494c91df9a Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)\n 1: 72b6a8e4e0b = 2: 8219b1b12f2 cocci rules: remove unused \"F\" metavariable from pending rule\n 2: 0998948b881 ! 3: 6dbfafa08fd Makefile: add ability to TAB-complete cocci *.patch rules\n @@ Makefile: check: $(GENERATED_H)\n \t\texit 1; \\\n \tfi\n \n -+COCCI = $(wildcard contrib/coccinelle/*.cocci)\n -+COCCI_PATCHES = $(addsuffix .patch,$(COCCI))\n -+COCCICHECK_PENDING = $(filter %.pending.cocci.patch,$(COCCI_PATCHES))\n -+COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES))\n ++COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci)\n ++COCCI_RULES = $(COCCI_GLOB)\n ++\n ++COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES))\n ++COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n ++\n ++COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n ++COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n +\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n -%.cocci.patch: %.cocci $(COCCI_SOURCES)\n ++COCCI_PATCHES = $(COCCI_RULES:%=%.patch)\n +$(COCCI_PATCHES): $(COCCI_SOURCES)\n +$(COCCI_PATCHES): %.patch: %\n \t$(QUIET_SPATCH) \\\n @@ Makefile: $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib\n \n coccicheck: coccicheck-test\n -coccicheck: $(addsuffix .patch,$(filter-out %.pending.cocci,$(wildcard contrib/coccinelle/*.cocci)))\n -+coccicheck: $(COCCICHECK)\n ++coccicheck: $(COCCICHECK_PATCHES)\n \n # See contrib/coccinelle/README\n coccicheck-pending: coccicheck-test\n -coccicheck-pending: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.pending.cocci))\n -+coccicheck-pending: $(COCCICHECK_PENDING)\n ++coccicheck-pending: $(COCCICHECK_PATCHES_PENDING)\n \n .PHONY: coccicheck coccicheck-pending\n \n -@@ Makefile: profile-clean:\n - \n - cocciclean:\n - \t$(RM) -r .build/contrib/coccinelle\n --\t$(RM) contrib/coccinelle/*.cocci.patch*\n -+\t$(RM) $(COCCI_PATCHES)*\n - \n - clean: profile-clean coverage-clean cocciclean\n - \t$(RM) -r .build\n 3: 63cf9f58d99 ! 4: f779a2d22aa Makefile: have \"coccicheck\" re-run if flags change\n @@ Makefile: SANITIZE_ADDRESS =\n include config.mak.uname\n -include config.mak.autogen\n -include config.mak\n -@@ Makefile: COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES))\n - \n +@@ Makefile: COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n + COCCI_PATCHES = $(COCCI_RULES:%=%.patch)\n +$(COCCI_PATCHES): GIT-SPATCH-DEFINES\n $(COCCI_PATCHES): $(COCCI_SOURCES)\n $(COCCI_PATCHES): %.patch: %\n @@ Makefile: $(COCCI_PATCHES): %.patch: %\n $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n +@@ Makefile: profile-clean:\n + \t$(RM) $(addsuffix *.gcno,$(addprefix $(PROFILE_DIR)/, $(object_dirs)))\n + \n + cocciclean:\n ++\t$(RM) GIT-SPATCH-DEFINES\n + \t$(RM) -r .build/contrib/coccinelle\n + \t$(RM) contrib/coccinelle/*.cocci.patch*\n + \n 4: 54d6bae3984 = 5: ab25b586f38 Makefile: split off SPATCH_BATCH_SIZE comment from \"cocci\" heading\n 5: 4f165bf6128 ! 6: 691be73b6fb cocci: split off include-less \"tests\" from SPATCH_FLAGS\n @@ Makefile: $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n \t$(call mkdir_p_parent_template)\n --\t$(QUIET_SPATCH_T)$(SPATCH) $(SPATCH_FLAGS) \\\n -+\t$(QUIET_SPATCH_T)$(SPATCH) $(SPATCH_TEST_FLAGS) \\\n +-\t$(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_FLAGS) \\\n ++\t$(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_TEST_FLAGS) \\\n \t\t--very-quiet --no-show-diff \\\n \t\t--sp-file $< -o $@ \\\n \t\t$(@:.build/%.res=%.c) && \\\n 6: c74d09f4825 = 7: 2ca5ea5beca cocci: split off \"--all-includes\" from SPATCH_FLAGS\n 7: 120607b5da6 ! 8: 2072a508064 cocci: make \"coccicheck\" rule incremental\n @@ Commit message\n \n This could be mitigated by combining \"make -jN\" with\n SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch\n - spatch invocations, 2019-05-06). But doing so required careful\n - juggling, as e.g. setting both to 4 would yield 16 workers.\n + spatch invocations, 2019-05-06).\n \n There will be cases where getting rid of \"SPATCH_BATCH_SIZE\" makes\n things worse, but a from-scratch \"make coccicheck\" with the default\n @@ Commit message\n setting that makes sense when doing a non-incremental run of \"make\n coccicheck\".\n \n + * Before the \"make coccicheck\" rule would have to clean\n + \"contrib/coccinelle/*.cocci.patch*\", since we'd create \"*+\" and\n + \"*.log\" files there. Now those are created in\n + .build/contrib/coccinelle/, which is covered by the \"cocciclean\" rule\n + already.\n +\n Outstanding issues & future work:\n \n * We could get rid of \"--all-includes\" in favor of manually\n @@ Makefile: TRACK_SPATCH_DEFINES += $(SPATCH)\n \t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n \t if test x\"$$FLAGS\" != x\"`cat GIT-SPATCH-DEFINES 2>/dev/null`\" ; then \\\n @@ Makefile: check: $(GENERATED_H)\n - \t\texit 1; \\\n - \tfi\n \n + COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci)\n + COCCI_RULES = $(COCCI_GLOB)\n ++COCCI_NAMES = $(COCCI_RULES:contrib/coccinelle/%.cocci=%)\n + \n + COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES))\n ++COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n + COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n ++COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n ++\n +# It's expensive to compute the many=many rules below, only eval them\n +# on $(MAKECMDGOALS) that match these $(COCCI_RULES)\n -+COCCI_RULES =\n -+COCCI_RULES += cocci%\n -+COCCI_RULES += contrib/coccinelle/%\n -+COCCI_RULES += .build/contrib/coccinelle/%\n -+\n - COCCI = $(wildcard contrib/coccinelle/*.cocci)\n -+COCCI_NAMES = $(COCCI:contrib/coccinelle/%.cocci=%)\n - COCCI_PATCHES = $(addsuffix .patch,$(COCCI))\n - COCCICHECK_PENDING = $(filter %.pending.cocci.patch,$(COCCI_PATCHES))\n --COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES))\n ++COCCI_RULES_GLOB =\n ++COCCI_RULES_GLOB += cocci%\n ++COCCI_RULES_GLOB += .build/contrib/coccinelle/%\n ++COCCI_RULES_GLOB += $(COCCICHECK_PATCHES)\n ++COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING)\n ++COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS))\n + \n + COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n + COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n \n -+COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES))\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n +-COCCI_PATCHES = $(COCCI_RULES:%=%.patch)\n -$(COCCI_PATCHES): GIT-SPATCH-DEFINES\n -$(COCCI_PATCHES): $(COCCI_SOURCES)\n -$(COCCI_PATCHES): %.patch: %\n @@ Makefile: check: $(GENERATED_H)\n +\tfi\n +endef\n +\n -+define cocci-matrix-2\n ++define cocci-matrix\n +\n -+$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s)))\n -+endef\n -+define cocci-matrix-1\n -+$(foreach c,$(COCCI),$(call cocci-matrix-2,$(c)))\n ++$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(1),$(s)))\n +endef\n +\n -+ifneq ($(filter $(COCCI_RULES),$(MAKECMDGOALS)),)\n -+$(eval $(call cocci-matrix-1))\n ++ifdef COCCI_GOALS\n ++$(eval $(foreach c,$(COCCI_RULES),$(call cocci-matrix,$(c))))\n +endif\n +\n +define spatch-rule\n +\n +contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1))\n -+\t$(QUIET_SPATCH_M)cat $$^ >$$@ && \\\n ++\t$(QUIET_SPATCH_CAT_TMPL)cat $$^ >$$@ && \\\n +\tif test -s $$@; \\\n \tthen \\\n -\t\techo ' ' SPATCH result: $@; \\\n @@ Makefile: check: $(GENERATED_H)\n \tfi\n +endef\n +\n -+ifneq ($(filter $(COCCI_RULES),$(MAKECMDGOALS)),)\n ++ifdef COCCI_GOALS\n +$(eval $(foreach n,$(COCCI_NAMES),$(call spatch-rule,$(n))))\n +endif\n \n COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n @@ Makefile: profile-clean:\n - \n cocciclean:\n + \t$(RM) GIT-SPATCH-DEFINES\n \t$(RM) -r .build/contrib/coccinelle\n --\t$(RM) $(COCCI_PATCHES)*\n -+\t$(RM) $(COCCI_PATCHES)\n +-\t$(RM) contrib/coccinelle/*.cocci.patch*\n ++\t$(RM) $(COCCICHECK_PATCHES)\n ++\t$(RM) $(COCCICHECK_PATCHES_PENDING)\n \n clean: profile-clean coverage-clean cocciclean\n \t$(RM) -r .build\n \n ## shared.mak ##\n @@ shared.mak: ifndef V\n - \tQUIET_SP = @echo ' ' SP $<;\n - \tQUIET_HDR = @echo ' ' HDR $(<:hcc=h);\n \tQUIET_RC = @echo ' ' RC $@;\n --\tQUIET_SPATCH = @echo ' ' SPATCH $<;\n -+\tQUIET_SPATCH = @echo ' ' SPATCH $@;\n - \tQUIET_SPATCH_T = @echo ' ' SPATCH TEST $(@:.build/%=%);\n -+\tQUIET_SPATCH_M = @echo ' ' SPATCH MERGE $$@;\n \n + ## Used in \"Makefile\": SPATCH\n +-\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $<;\n ++\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $@;\n + \tQUIET_SPATCH_TEST\t\t= @echo ' ' SPATCH TEST $(@:.build/%=%);\n + \n ++## Used in \"Makefile\": SPATCH_*TMPL (quoted for use in \"define\"'s)\n ++\tQUIET_SPATCH_CAT_TMPL\t\t= @echo ' ' SPATCH CAT $$$$^ \\>$$@;\n ++\n ## Used in \"Documentation/Makefile\"\n \tQUIET_ASCIIDOC\t= @echo ' ' ASCIIDOC $@;\n + \tQUIET_XMLTO\t= @echo ' ' XMLTO $@;\n 8: 536dce45eef ! 9: 739652eada9 cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES\n @@ Commit message\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## Makefile ##\n +@@ Makefile: SPATCH_INCLUDE_FLAGS = --all-includes\n + SPATCH_FLAGS =\n + SPATCH_TEST_FLAGS =\n + \n ++# If *.o files are present, have \"coccicheck\" depend on them, with\n ++# COMPUTE_HEADER_DEPENDENCIES this will speed up the common-case of\n ++# only needing to re-generate coccicheck results for the users of a\n ++# given API if it's changed, and not all files in the project. If\n ++# COMPUTE_HEADER_DEPENDENCIES=no this will be unset too.\n ++SPATCH_USE_O_DEPENDENCIES = YesPlease\n ++\n + # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n + TRACK_SPATCH_DEFINES =\n + TRACK_SPATCH_DEFINES += $(SPATCH)\n @@ Makefile: COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \t$(call mkdir_p_parent_template)\n \t$(QUIET_GEN) >$@\n \n -+SPATCH_USE_O_DEPENDENCIES = yes\n +ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no)\n +SPATCH_USE_O_DEPENDENCIES =\n +endif\n -+\n define cocci-rule\n \n ## Rule for .build/$(1).patch/$(2); Params:\n @@ Makefile: COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \t$$(call mkdir_p_parent_template)\n @@ Makefile: endef\n \n - define cocci-matrix-2\n + define cocci-matrix\n \n --$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s)))\n +-$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(1),$(s)))\n +$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o)))\n endef\n - define cocci-matrix-1\n - $(foreach c,$(COCCI),$(call cocci-matrix-2,$(c)))\n + \n + ifdef COCCI_GOALS\n -: ----------- > 10: 52177ea2a68 cocci: run against a generated ALL.cocci\n 9: 2b978676a56 ! 11: f7ca3f9c9af spatchcache: add a ccache-alike for \"spatch\"\n @@ contrib/coccinelle/spatchcache (new)\n +#\n +# Note that the \"--very-quiet\" flag is currently critical. The cache\n +# will refuse to cache anything that has output on STDERR (which might\n -+# be errors from spatch). The STDERR (and exit code) could in\n -+# principle be cached (as with ccache), but then the simple structure\n -+# in the Redis cache would need to change, so just supply\n -+# \"--very-quiet\" for now.\n ++# be errors from spatch), but see spatchCache.cacheWhenStderr below.\n ++#\n ++# The STDERR (and exit code) could in principle be cached (as with\n ++# ccache), but then the simple structure in the Redis cache would need\n ++# to change, so just supply \"--very-quiet\" for now.\n +#\n +# To use this, simply set SPATCH to\n +# contrib/coccinelle/spatchcache. Then optionally set:\n @@ contrib/coccinelle/spatchcache (new)\n +#\n +# As well as this trace config (debug implies trace):\n +#\n ++#\t\tcacheWhenStderr = true\n +#\t\ttrace = false\n +#\t\tdebug = false\n +#\n ++# The \".depend/grep.o.d\" can also be customized, as a string that will\n ++# be eval'd, it has access to a \"$dirname\" and \"$basename\":\n ++#\n ++#\t[spatchCache]\n ++#\t\tdependFormat = \"$dirname/.depend/${basename%.c}.o.d\"\n ++#\n +# Setting \"trace\" to \"true\" allows for seeing when we have a cache HIT\n +# or MISS. To debug whether the cache is working do that, and run e.g.:\n +#\n @@ contrib/coccinelle/spatchcache (new)\n +#\n +# A subsequent \"make cocciclean && make coccicheck\" should then have\n +# all \"HIT\"'s and \"CANTCACHE\"'s.\n ++#\n ++# The \"spatchCache.cacheWhenStderr\" option is critical when using\n ++# spatchCache.{trace,debug} to debug whether something is set in the\n ++# cache, as we'll write to the spatch logs in .build/* we'd otherwise\n ++# always emit a NOCACHE.\n +\n +set -e\n +\n @@ contrib/coccinelle/spatchcache (new)\n +\ttrace=true\n +fi\n +\n ++cacheWhenStderr=$(git config --bool \"spatchCache.cacheWhenStderr\")\n ++if test \"$cacheWhenStderr\" != \"true\"\n ++then\n ++\tcacheWhenStderr=\n ++fi\n ++\n +trace_it () {\n +\tif test -z \"$trace\"\n +\tthen\n @@ contrib/coccinelle/spatchcache (new)\n +\tspatch=spatch\n +fi\n +\n ++dependFormat='$dirname/.depend/${basename%.c}.o.d'\n ++dependFormatCfg=$(git config \"spatchCache.dependFormat\" || :)\n ++if test -n \"$dependFormatCfg\"\n ++then\n ++\tdependFormat=\"$dependFormatCfg\"\n ++fi\n ++\n +set=$(git config spatchCache.setCmd || :)\n +get=$(git config spatchCache.getCmd || :)\n +\n @@ contrib/coccinelle/spatchcache (new)\n +\n +hash_for_cache() {\n +\t# Parameters that should affect the cache\n -+\techo \"spatch=$spatch\"\n +\techo \"args=$args\"\n ++\techo \"config spatchCache.spatch=$spatch\"\n ++\techo \"config spatchCache.debug=$debug\"\n ++\techo \"config spatchCache.trace=$trace\"\n ++\techo \"config spatchCache.cacheWhenStderr=$cacheWhenStderr\"\n +\techo\n +\n +\t# Our target file and its dependencies\n @@ contrib/coccinelle/spatchcache (new)\n +fi\n +\n +# Main logic\n -+d=$(dirname \"$arg_file\")\n -+b=$(basename \"$arg_file\")\n -+bwoext=\"${b%.c}\"\n -+dep=\"$d/.depend/$bwoext.o.d\"\n ++dirname=$(dirname \"$arg_file\")\n ++basename=$(basename \"$arg_file\")\n ++eval \"dep=$dependFormat\"\n +\n +if ! test -f \"$dep\"\n +then\n @@ contrib/coccinelle/spatchcache (new)\n +err=\"$(mktemp)\"\n +\n +set +e\n -+\"$spatch\" \"$@\" >\"$out\" 2>\"$err\"\n ++\"$spatch\" \"$@\" >\"$out\" 2>>\"$err\"\n +ret=$?\n ++cat \"$out\"\n ++cat \"$err\" >&2\n +set -e\n +\n -+if test $ret = 0 && ! test -s \"$out\" && ! test -s \"$err\"\n ++nocache=\n ++if test $ret != 0\n ++then\n ++\tnocache=\"exited non-zero: $ret\"\n ++elif test -s \"$out\"\n ++then\n ++\tnocache=\"had patch output\"\n ++elif test -z \"$cacheWhenStderr\" && test -s \"$err\"\n ++then\n ++\tnocache=\"had stderr (use --very-quiet or spatchCache.cacheWhenStderr=true?)\"\n ++fi\n ++\n ++if test -n \"$nocache\"\n +then\n -+\trm -f \"$out\" \"$err\"\n ++\ttrace_it \"$0: NOCACHE ($nocache): for '$arg_file' with '$arg_sp'\"\n ++\texit \"$ret\"\n ++fi\n +\n -+\ttrace_it \"$0: SET: for '$arg_file' with '$arg_sp'\"\n ++trace_it \"$0: SET: for '$arg_file' with '$arg_sp'\"\n +\n -+\tsetret=\n -+\tif test -z \"$set\"\n ++setret=\n ++if test -z \"$set\"\n ++then\n ++\tif test $(redis-cli SADD spatch-cache \"$sum\") = 1\n +\tthen\n -+\t\tif test $(redis-cli SADD spatch-cache \"$sum\") = 1\n -+\t\tthen\n -+\t\t\tsetret=0\n -+\t\telse\n -+\t\t\tsetret=1\n -+\t\tfi\n ++\t\tsetret=0\n +\telse\n -+\t\t\"$set\" \"$sum\"\n -+\t\tsetret=$?\n -+\tfi\n -+\n -+\tif test \"$setret\" != 0\n -+\tthen\n -+\t\techo \"FAILED to set '$sum' in cache!\" >&2\n -+\t\texit 128\n ++\t\tsetret=1\n +\tfi\n +else\n -+\ttrace_it \"$0: NOCACHE (have changes): for '$arg_file' with '$arg_sp'\"\n -+\tcat \"$out\"\n -+\tcat \"$err\" >&2\n -+\trm -f \"$out\" \"$err\"\n -+\texit \"$ret\"\n ++\t\"$set\" \"$sum\"\n ++\tsetret=$?\n +fi\n -+rm -f \"$out\" \"$err\"\n ++\n ++if test \"$setret\" != 0\n ++then\n ++\techo \"FAILED to set '$sum' in cache!\" >&2\n ++\texit 128\n ++fi\n ++\n ++exit \"$ret\"\n-- \n2.38.0.1092.g8c0298861b0\n\n"},{"id":"464938","messageId":"patch-v3-01.11-4494c91df9a-20221014T152552Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v3-00.11-00000000000-20221014T152552Z-avarab@gmail.com","subject":"[PATCH v3 01/11] Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-14T15:31:17Z","receivedAt":"2022-10-14T15:32:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"In f7ff6597a75 (cocci: add a \"coccicheck-test\" target and test *.cocci\nrules, 2022-07-05) we abbreviated \"_TEST\" to \"_T\" to have it align\nwith the rest of the \"=\"'s above it.\n\nSubsequent commits will add more QUIET_SPATCH_* variables, so let's\nstop abbreviating this, and indent it in preparation for adding more\nof these variables.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 2 +-\n shared.mak | 6 ++++--\n 2 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 6bfb62cbe94..f641adab2fd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3164,7 +3164,7 @@ $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n \t$(call mkdir_p_parent_template)\n-\t$(QUIET_SPATCH_T)$(SPATCH) $(SPATCH_FLAGS) \\\n+\t$(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_FLAGS) \\\n \t\t--very-quiet --no-show-diff \\\n \t\t--sp-file $< -o $@ \\\n \t\t$(@:.build/%.res=%.c) && \\\ndiff --git a/shared.mak b/shared.mak\nindex 33f43edbf9a..96b06acc455 100644\n--- a/shared.mak\n+++ b/shared.mak\n@@ -69,8 +69,10 @@ ifndef V\n \tQUIET_SP = @echo ' ' SP $<;\n \tQUIET_HDR = @echo ' ' HDR $(<:hcc=h);\n \tQUIET_RC = @echo ' ' RC $@;\n-\tQUIET_SPATCH = @echo ' ' SPATCH $<;\n-\tQUIET_SPATCH_T = @echo ' ' SPATCH TEST $(@:.build/%=%);\n+\n+## Used in \"Makefile\": SPATCH\n+\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $<;\n+\tQUIET_SPATCH_TEST\t\t= @echo ' ' SPATCH TEST $(@:.build/%=%);\n \n ## Used in \"Documentation/Makefile\"\n \tQUIET_ASCIIDOC\t= @echo ' ' ASCIIDOC $@;\n-- \n2.38.0.1092.g8c0298861b0\n\n"},{"id":"464939","messageId":"patch-v3-02.11-8219b1b12f2-20221014T152552Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v3-00.11-00000000000-20221014T152552Z-avarab@gmail.com","subject":"[PATCH v3 02/11] cocci rules: remove unused \"F\" metavariable from pending rule","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-14T15:31:18Z","receivedAt":"2022-10-14T15:32:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix an issue with a rule added in 9b45f499818 (object-store: prepare\nhas_{sha1, object}_file to handle any repo, 2018-11-13). We've been\nspewing out this warning into our $@.log since that rule was added:\n\n\twarning: rule starting on line 21: metavariable F not used in the - or context code\n\nWe should do a better job of scouring our coccinelle log files for\nsuch issues, but for now let's fix this as a one-off.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n contrib/coccinelle/the_repository.pending.cocci | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/contrib/coccinelle/the_repository.pending.cocci b/contrib/coccinelle/the_repository.pending.cocci\nindex 072ea0d9228..747d382ff5f 100644\n--- a/contrib/coccinelle/the_repository.pending.cocci\n+++ b/contrib/coccinelle/the_repository.pending.cocci\n@@ -20,7 +20,6 @@ expression E;\n \n @@\n expression E;\n-expression F;\n @@\n - has_object_file_with_flags(\n + repo_has_object_file_with_flags(the_repository,\n-- \n2.38.0.1092.g8c0298861b0\n\n"},{"id":"464940","messageId":"patch-v3-05.11-ab25b586f38-20221014T152553Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v3-00.11-00000000000-20221014T152552Z-avarab@gmail.com","subject":"[PATCH v3 05/11] Makefile: split off SPATCH_BATCH_SIZE comment from \"cocci\" heading","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-14T15:31:21Z","receivedAt":"2022-10-14T15:32:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Split off the \"; setting[...]\" part of the comment added in In\n960154b9c17 (coccicheck: optionally batch spatch invocations,\n2019-05-06), and restore what we had before that, which was a comment\nindicating that variables for the \"coccicheck\" target were being set\nhere.\n\nWhen 960154b9c17 amended the heading to discuss SPATCH_BATCH_SIZE it\nleft no natural place to add a new comment about other flags that\npreceded it. As subsequent commits will add such comments we need to\nsplit the existing comment up.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex d62f2311107..2a106f633b9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1298,10 +1298,11 @@ SP_EXTRA_FLAGS = -Wno-universal-initializer\n SANITIZE_LEAK =\n SANITIZE_ADDRESS =\n \n-# For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n+# For the 'coccicheck' target\n+SPATCH_FLAGS = --all-includes\n+# Setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n-SPATCH_FLAGS = --all-includes\n SPATCH_BATCH_SIZE = 1\n \n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n-- \n2.38.0.1092.g8c0298861b0\n\n"},{"id":"464941","messageId":"patch-v3-03.11-6dbfafa08fd-20221014T152552Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v3-00.11-00000000000-20221014T152552Z-avarab@gmail.com","subject":"[PATCH v3 03/11] Makefile: add ability to TAB-complete cocci *.patch rules","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-14T15:31:19Z","receivedAt":"2022-10-14T15:32:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Declare the contrib/coccinelle/.cocci.patch rules in such a way\nas to allow TAB-completion, and slightly optimize the Makefile by\ncutting down on the number of $(wildcard) in favor of defining\n\"coccicheck\" and \"coccicheck-pending\" in terms of the same\nincrementally filtered list.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 17 ++++++++++++++---\n 1 file changed, 14 insertions(+), 3 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex f641adab2fd..4647f317ede 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3136,9 +3136,20 @@ check: $(GENERATED_H)\n \t\texit 1; \\\n \tfi\n \n+COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci)\n+COCCI_RULES = $(COCCI_GLOB)\n+\n+COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES))\n+COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n+\n+COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n+COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n+\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n-%.cocci.patch: %.cocci $(COCCI_SOURCES)\n+COCCI_PATCHES = $(COCCI_RULES:%=%.patch)\n+$(COCCI_PATCHES): $(COCCI_SOURCES)\n+$(COCCI_PATCHES): %.patch: %\n \t$(QUIET_SPATCH) \\\n \tif test $(SPATCH_BATCH_SIZE) = 0; then \\\n \t\tlimit=; \\\n@@ -3175,11 +3186,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell\n coccicheck-test: $(COCCI_TEST_RES_GEN)\n \n coccicheck: coccicheck-test\n-coccicheck: $(addsuffix .patch,$(filter-out %.pending.cocci,$(wildcard contrib/coccinelle/*.cocci)))\n+coccicheck: $(COCCICHECK_PATCHES)\n \n # See contrib/coccinelle/README\n coccicheck-pending: coccicheck-test\n-coccicheck-pending: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.pending.cocci))\n+coccicheck-pending: $(COCCICHECK_PATCHES_PENDING)\n \n .PHONY: coccicheck coccicheck-pending\n \n-- \n2.38.0.1092.g8c0298861b0\n\n"},{"id":"464942","messageId":"patch-v3-07.11-2ca5ea5beca-20221014T152553Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v3-00.11-00000000000-20221014T152552Z-avarab@gmail.com","subject":"[PATCH v3 07/11] cocci: split off \"--all-includes\" from SPATCH_FLAGS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-14T15:31:23Z","receivedAt":"2022-10-14T15:32:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Per the rationale in 7b63ea57500 (Makefile: remove mandatory \"spatch\"\narguments from SPATCH_FLAGS, 2022-07-05) we have certain flags that\nare truly mandatory, such as \"--sp-file\" and \"--patch .\". The\n\"--all-includes\" flag is also critical, but per [1] we might want to\nad-hoc tweak it occasionally for testing or one-offs.\n\nBut being unable to set e.g. SPATCH_FLAGS=\"--verbose-parsing\" without\nbreaking how our \"spatch\" works isn't ideal, i.e. before this we'd\nneed to know about the default include flags, and specify:\nSPATCH_FLAGS=\"--all-includes --verbose-parsing\".\n\nIf we were then to change the default include flag (e.g. to\n\"--recursive-includes\") in the future any such one-off commands would\nneed to be correspondingly updated.\n\nLet's instead leave the SPATCH_FLAGS for the user, while creating a\nnew SPATCH_INCLUDE_FLAGS to allow for ad-hoc testing of the include\nstrategy itself.\n\n1. https://lore.kernel.org/git/20220823095733.58685-1-szeder.dev@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex 535dece5d48..f79697053bc 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1299,7 +1299,8 @@ SANITIZE_LEAK =\n SANITIZE_ADDRESS =\n \n # For the 'coccicheck' target\n-SPATCH_FLAGS = --all-includes\n+SPATCH_INCLUDE_FLAGS = --all-includes\n+SPATCH_FLAGS =\n SPATCH_TEST_FLAGS =\n # Setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n@@ -1309,6 +1310,7 @@ SPATCH_BATCH_SIZE = 1\n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n TRACK_SPATCH_DEFINES += $(SPATCH)\n+TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n@@ -3174,6 +3176,7 @@ $(COCCI_PATCHES): %.patch: %\n \tfi; \\\n \tif ! echo $(COCCI_SOURCES) | xargs $$limit \\\n \t\t$(SPATCH) $(SPATCH_FLAGS) \\\n+\t\t$(SPATCH_INCLUDE_FLAGS) \\\n \t\t--sp-file $< --patch . \\\n \t\t>$@+ 2>$@.log; \\\n \tthen \\\n-- \n2.38.0.1092.g8c0298861b0\n\n"},{"id":"464943","messageId":"patch-v3-04.11-f779a2d22aa-20221014T152552Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v3-00.11-00000000000-20221014T152552Z-avarab@gmail.com","subject":"[PATCH v3 04/11] Makefile: have \"coccicheck\" re-run if flags change","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-14T15:31:20Z","receivedAt":"2022-10-14T15:32:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix an issue with the \"coccicheck\" family of rules that's been here\nsince 63f0a758a06 (add coccicheck make target, 2016-09-15), unlike\ne.g. \"make grep.o\" we wouldn't re-run it when $(SPATCH) or\n$(SPATCH_FLAGS) changed. To test new flags we needed to first do a\n\"make cocciclean\".\n\nThis now uses the same (copy/pasted) pattern as other \"DEFINES\"\nrules. As a result we'll re-run properly. This can be demonstrated\ne.g. on the issue noted in [1]:\n\n\t$ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c V=1\n\t[...]\n\t SPATCH contrib/coccinelle/xcalloc.cocci\n\t$ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c SPATCH_FLAGS=\"--all-includes --recursive-includes\"\n\t * new spatch flags\n\t SPATCH contrib/coccinelle/xcalloc.cocci\n\t SPATCH result: contrib/coccinelle/xcalloc.cocci.patch\n\t$\n\n1. https://lore.kernel.org/git/20220823095602.GC1735@szeder.dev/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .gitignore | 1 +\n Makefile | 15 +++++++++++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/.gitignore b/.gitignore\nindex 62720c6135d..7cce39270d4 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -7,6 +7,7 @@\n /GIT-PERL-HEADER\n /GIT-PYTHON-VARS\n /GIT-SCRIPT-DEFINES\n+/GIT-SPATCH-DEFINES\n /GIT-USER-AGENT\n /GIT-VERSION-FILE\n /bin-wrappers/\ndiff --git a/Makefile b/Makefile\nindex 4647f317ede..d62f2311107 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1304,6 +1304,18 @@ SANITIZE_ADDRESS =\n SPATCH_FLAGS = --all-includes\n SPATCH_BATCH_SIZE = 1\n \n+# Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n+TRACK_SPATCH_DEFINES =\n+TRACK_SPATCH_DEFINES += $(SPATCH)\n+TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n+TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n+GIT-SPATCH-DEFINES: FORCE\n+\t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n+\t if test x\"$$FLAGS\" != x\"`cat GIT-SPATCH-DEFINES 2>/dev/null`\" ; then \\\n+\t\techo >&2 \" * new spatch flags\"; \\\n+\t\techo \"$$FLAGS\" >GIT-SPATCH-DEFINES; \\\n+ fi\n+\n include config.mak.uname\n -include config.mak.autogen\n -include config.mak\n@@ -3148,6 +3160,7 @@ COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n COCCI_PATCHES = $(COCCI_RULES:%=%.patch)\n+$(COCCI_PATCHES): GIT-SPATCH-DEFINES\n $(COCCI_PATCHES): $(COCCI_SOURCES)\n $(COCCI_PATCHES): %.patch: %\n \t$(QUIET_SPATCH) \\\n@@ -3171,6 +3184,7 @@ $(COCCI_PATCHES): %.patch: %\n \tfi\n \n COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n+$(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n@@ -3457,6 +3471,7 @@ profile-clean:\n \t$(RM) $(addsuffix *.gcno,$(addprefix $(PROFILE_DIR)/, $(object_dirs)))\n \n cocciclean:\n+\t$(RM) GIT-SPATCH-DEFINES\n \t$(RM) -r .build/contrib/coccinelle\n \t$(RM) contrib/coccinelle/*.cocci.patch*\n \n-- \n2.38.0.1092.g8c0298861b0\n\n"},{"id":"464944","messageId":"patch-v3-06.11-691be73b6fb-20221014T152553Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v3-00.11-00000000000-20221014T152552Z-avarab@gmail.com","subject":"[PATCH v3 06/11] cocci: split off include-less \"tests\" from SPATCH_FLAGS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-14T15:31:22Z","receivedAt":"2022-10-14T15:32:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Amend the \"coccicheck-test\" rule added in f7ff6597a75 (cocci: add a\n\"coccicheck-test\" target and test *.cocci rules, 2022-07-05) to stop\nusing \"--all-includes\". The flags we'll need for the tests are\ndifferent than the ones we'll need for our main source code.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex 2a106f633b9..535dece5d48 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1300,6 +1300,7 @@ SANITIZE_ADDRESS =\n \n # For the 'coccicheck' target\n SPATCH_FLAGS = --all-includes\n+SPATCH_TEST_FLAGS =\n # Setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n@@ -1309,6 +1310,7 @@ SPATCH_BATCH_SIZE = 1\n TRACK_SPATCH_DEFINES =\n TRACK_SPATCH_DEFINES += $(SPATCH)\n TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n+TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n GIT-SPATCH-DEFINES: FORCE\n \t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n@@ -3190,7 +3192,7 @@ $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n \t$(call mkdir_p_parent_template)\n-\t$(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_FLAGS) \\\n+\t$(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_TEST_FLAGS) \\\n \t\t--very-quiet --no-show-diff \\\n \t\t--sp-file $< -o $@ \\\n \t\t$(@:.build/%.res=%.c) && \\\n-- \n2.38.0.1092.g8c0298861b0\n\n"},{"id":"464945","messageId":"patch-v3-08.11-2072a508064-20221014T152553Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v3-00.11-00000000000-20221014T152552Z-avarab@gmail.com","subject":"[PATCH v3 08/11] cocci: make \"coccicheck\" rule incremental","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-14T15:31:24Z","receivedAt":"2022-10-14T15:32:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Optimize the very slow \"coccicheck\" target to take advantage of\nincremental rebuilding, and fix outstanding dependency problems with\nthe existing rule.\n\nThe rule is now faster both on the initial run as we can make better\nuse of GNU make's parallelism than the old ad-hoc combination of\nmake's parallelism combined with $(SPATCH_BATCH_SIZE) and/or the\n\"--jobs\" argument to \"spatch(1)\".\n\nIt also makes us *much* faster when incrementally building, it's now\nviable to \"make coccicheck\" as topic branches are merged down.\n\nThe rule didn't use FORCE (or its equivalents) before, so a:\n\n\tmake coccicheck\n\tmake coccicheck\n\nWould report nothing to do on the second iteration. But all of our\npatch output depended on all $(COCCI_SOURCES) files, therefore e.g.:\n\n make -W grep.c coccicheck\n\nWould do a full re-run, i.e. a a change in a single file would force\nus to do a full re-run.\n\nThe reason for this (not the initial rationale, but my analysis) is:\n\n* Since we create a single \"*.cocci.patch+\" we don't know where to\n pick up where we left off, or how to incrementally merge e.g. a\n \"grep.c\" change with an existing *.cocci.patch.\n\n* We've been carrying forward the dependency on the *.c files since\n 63f0a758a06 (add coccicheck make target, 2016-09-15) the rule was\n initially added as a sort of poor man's dependency discovery.\n\n As we don't include other *.c files depending on other *.c files\n has always been broken, as could be trivially demonstrated\n e.g. with:\n\n make coccicheck\n make -W strbuf.h coccicheck\n\n However, depending on the corresponding *.c files has been doing\n something, namely that *if* an API change modified both *.c and *.h\n files we'd catch the change to the *.h we care about via the *.c\n being changed.\n\n For API changes that happened only via *.h files we'd do the wrong\n thing before this change, but e.g. for function additions (not\n \"static inline\" ones) catch the *.h change by proxy.\n\nNow we'll instead:\n\n * Create a / pair in the .build directory, E.g. for\n swap.cocci and grep.c we'll create\n .build/contrib/coccinelle/swap.cocci.patch/grep.c.\n\n That file is the diff we'll apply for that -\n combination, if there's no changes to me made (the common case)\n it'll be an empty file.\n\n * Our generated *.patch\n file (e.g. contrib/coccinelle/swap.cocci.patch) is now a simple \"cat\n $^\" of all of all of the / files for a given .\n\n In the case discussed above of \"grep.c\" being changed we'll do the\n full \"cat\" every time, so they resulting *.cocci.patch will always\n be correct and up-to-date, even if it's \"incrementally updated\".\n\n See 1cc0425a27c (Makefile: have \"make pot\" not \"reset --hard\",\n 2022-05-26) for another recent rule that used that technique.\n\nAs before we'll:\n\n * End up generating a contrib/coccinelle/swap.cocci.patch, if we\n \"fail\" by creating a non-empty patch we'll still exit with a zero\n exit code.\n\n Arguably we should move to a more Makefile-native way of doing\n this, i.e. fail early, and if we want all of the \"failed\" changes\n we can use \"make -k\", but as the current\n \"ci/run-static-analysis.sh\" expects us to behave this way let's\n keep the existing behavior of exhaustively discovering all cocci\n changes, and only failing if spatch itself errors out.\n\nFurther implementation details & notes:\n\n * Before this change running \"make coccicheck\" would by default end\n up pegging just one CPU at the very end for a while, usually as\n we'd finish whichever *.cocci rule was the most expensive.\n\n This could be mitigated by combining \"make -jN\" with\n SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch\n spatch invocations, 2019-05-06).\n\n There will be cases where getting rid of \"SPATCH_BATCH_SIZE\" makes\n things worse, but a from-scratch \"make coccicheck\" with the default\n of SPATCH_BATCH_SIZE=1 (and tweaking it doesn't make a difference)\n is faster (~3m36s v.s. ~3m56s) with this approach, as we can feed\n the CPU more work in a less staggered way.\n\n * Getting rid of \"SPATCH_BATCH_SIZE\" particularly helps in cases\n where the default of 1 yields parallelism under \"make coccicheck\",\n but then running e.g.:\n\n make -W contrib/coccinelle/swap.cocci coccicheck\n\n I.e. before that would use only one CPU core, until the user\n remembered to adjust \"SPATCH_BATCH_SIZE\" differently than the\n setting that makes sense when doing a non-incremental run of \"make\n coccicheck\".\n\n * Before the \"make coccicheck\" rule would have to clean\n \"contrib/coccinelle/*.cocci.patch*\", since we'd create \"*+\" and\n \"*.log\" files there. Now those are created in\n .build/contrib/coccinelle/, which is covered by the \"cocciclean\" rule\n already.\n\nOutstanding issues & future work:\n\n * We could get rid of \"--all-includes\" in favor of manually\n specifying a list of includes to give to \"spatch(1)\".\n\n As noted upthread of [1] a naïve removal of \"--all-includes\" will\n result in broken *.cocci patches, but if we know the exhaustive\n list of includes via COMPUTE_HEADER_DEPENDENCIES we don't need to\n re-scan for them, we could grab the headers to include from the\n .depend.d/.o.d and supply them with the \"--include\" option to\n spatch(1).q\n\n1. https://lore.kernel.org/git/87ft18tcog.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 84 +++++++++++++++++++++++++++++++++++++-----------------\n shared.mak | 5 +++-\n 2 files changed, 62 insertions(+), 27 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex f79697053bc..4e1da7206db 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1302,10 +1302,6 @@ SANITIZE_ADDRESS =\n SPATCH_INCLUDE_FLAGS = --all-includes\n SPATCH_FLAGS =\n SPATCH_TEST_FLAGS =\n-# Setting SPATCH_BATCH_SIZE higher will\n-# usually result in less CPU usage at the cost of higher peak memory.\n-# Setting it to 0 will feed all files in a single spatch invocation.\n-SPATCH_BATCH_SIZE = 1\n \n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n@@ -1313,7 +1309,6 @@ TRACK_SPATCH_DEFINES += $(SPATCH)\n TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS)\n-TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n GIT-SPATCH-DEFINES: FORCE\n \t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n \t if test x\"$$FLAGS\" != x\"`cat GIT-SPATCH-DEFINES 2>/dev/null`\" ; then \\\n@@ -3155,39 +3150,75 @@ check: $(GENERATED_H)\n \n COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci)\n COCCI_RULES = $(COCCI_GLOB)\n+COCCI_NAMES = $(COCCI_RULES:contrib/coccinelle/%.cocci=%)\n \n COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES))\n+COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n+COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n+\n+# It's expensive to compute the many=many rules below, only eval them\n+# on $(MAKECMDGOALS) that match these $(COCCI_RULES)\n+COCCI_RULES_GLOB =\n+COCCI_RULES_GLOB += cocci%\n+COCCI_RULES_GLOB += .build/contrib/coccinelle/%\n+COCCI_RULES_GLOB += $(COCCICHECK_PATCHES)\n+COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING)\n+COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS))\n \n COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n \n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n-COCCI_PATCHES = $(COCCI_RULES:%=%.patch)\n-$(COCCI_PATCHES): GIT-SPATCH-DEFINES\n-$(COCCI_PATCHES): $(COCCI_SOURCES)\n-$(COCCI_PATCHES): %.patch: %\n-\t$(QUIET_SPATCH) \\\n-\tif test $(SPATCH_BATCH_SIZE) = 0; then \\\n-\t\tlimit=; \\\n-\telse \\\n-\t\tlimit='-n $(SPATCH_BATCH_SIZE)'; \\\n-\tfi; \\\n-\tif ! echo $(COCCI_SOURCES) | xargs $$limit \\\n-\t\t$(SPATCH) $(SPATCH_FLAGS) \\\n-\t\t$(SPATCH_INCLUDE_FLAGS) \\\n-\t\t--sp-file $< --patch . \\\n-\t\t>$@+ 2>$@.log; \\\n+.build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES)\n+\t$(call mkdir_p_parent_template)\n+\t$(QUIET_GEN) >$@\n+\n+define cocci-rule\n+\n+## Rule for .build/$(1).patch/$(2); Params:\n+# $(1) = e.g. \"free.cocci\"\n+# $(2) = e.g. \"grep.c\"\n+COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n+.build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n+.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES\n+.build/$(1).patch/$(2): $(1)\n+.build/$(1).patch/$(2): .build/$(1).patch/% : %\n+\t$$(call mkdir_p_parent_template)\n+\t$$(QUIET_SPATCH)if ! $$(SPATCH) $$(SPATCH_FLAGS) \\\n+\t\t$$(SPATCH_INCLUDE_FLAGS) \\\n+\t\t--sp-file $(1) --patch . $$< \\\n+\t\t>$$@ 2>$$@.log; \\\n \tthen \\\n-\t\tcat $@.log; \\\n+\t\techo \"ERROR when applying '$(1)' to '$$<'; '$$@.log' follows:\"; \\\n+\t\tcat $$@.log; \\\n \t\texit 1; \\\n-\tfi; \\\n-\tmv $@+ $@; \\\n-\tif test -s $@; \\\n+\tfi\n+endef\n+\n+define cocci-matrix\n+\n+$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(1),$(s)))\n+endef\n+\n+ifdef COCCI_GOALS\n+$(eval $(foreach c,$(COCCI_RULES),$(call cocci-matrix,$(c))))\n+endif\n+\n+define spatch-rule\n+\n+contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1))\n+\t$(QUIET_SPATCH_CAT_TMPL)cat $$^ >$$@ && \\\n+\tif test -s $$@; \\\n \tthen \\\n-\t\techo ' ' SPATCH result: $@; \\\n+\t\techo ' ' SPATCH result: $$@; \\\n \tfi\n+endef\n+\n+ifdef COCCI_GOALS\n+$(eval $(foreach n,$(COCCI_NAMES),$(call spatch-rule,$(n))))\n+endif\n \n COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n@@ -3479,7 +3510,8 @@ profile-clean:\n cocciclean:\n \t$(RM) GIT-SPATCH-DEFINES\n \t$(RM) -r .build/contrib/coccinelle\n-\t$(RM) contrib/coccinelle/*.cocci.patch*\n+\t$(RM) $(COCCICHECK_PATCHES)\n+\t$(RM) $(COCCICHECK_PATCHES_PENDING)\n \n clean: profile-clean coverage-clean cocciclean\n \t$(RM) -r .build\ndiff --git a/shared.mak b/shared.mak\nindex 96b06acc455..5ccd6889fcb 100644\n--- a/shared.mak\n+++ b/shared.mak\n@@ -71,9 +71,12 @@ ifndef V\n \tQUIET_RC = @echo ' ' RC $@;\n \n ## Used in \"Makefile\": SPATCH\n-\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $<;\n+\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $@;\n \tQUIET_SPATCH_TEST\t\t= @echo ' ' SPATCH TEST $(@:.build/%=%);\n \n+## Used in \"Makefile\": SPATCH_*TMPL (quoted for use in \"define\"'s)\n+\tQUIET_SPATCH_CAT_TMPL\t\t= @echo ' ' SPATCH CAT $$$$^ \\>$$@;\n+\n ## Used in \"Documentation/Makefile\"\n \tQUIET_ASCIIDOC\t= @echo ' ' ASCIIDOC $@;\n \tQUIET_XMLTO\t= @echo ' ' XMLTO $@;\n-- \n2.38.0.1092.g8c0298861b0\n\n"},{"id":"464946","messageId":"patch-v3-09.11-739652eada9-20221014T152553Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v3-00.11-00000000000-20221014T152552Z-avarab@gmail.com","subject":"[PATCH v3 09/11] cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-14T15:31:25Z","receivedAt":"2022-10-14T15:32:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improve the incremental rebuilding support of \"coccicheck\" by\npiggy-backing on the computed dependency information of the\ncorresponding *.o file, rather than rebuilding all / pairs\nif either their corresponding file changes, or if any header changes.\n\nThis in effect uses the same method that the \"sparse\" target was made\nto use in c234e8a0ecf (Makefile: make the \"sparse\" target non-.PHONY,\n2021-09-23), except that the dependency on the *.o file isn't a hard\none, we check with $(wildcard) if the *.o file exists, and if so we'll\ndepend on it.\n\nThis means that the common case of:\n\n\tmake\n\tmake coccicheck\n\nWill benefit from incremental rebuilding, now changing e.g. a header\nwill only re-run \"spatch\" on those those *.c files that make use of\nit:\n\nBy depending on the *.o we piggy-back on\nCOMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the\n\"sparse\" target non-.PHONY, 2021-09-23) for prior art of doing that\nfor the *.sp files. E.g.:\n\n make contrib/coccinelle/free.cocci.patch\n make -W column.h contrib/coccinelle/free.cocci.patch\n\nWill take around 15 seconds for the second command on my 8 core box if\nI didn't run \"make\" beforehand to create the *.o files. But around 2\nseconds if I did and we have those \"*.o\" files.\n\nNotes about the approach of piggy-backing on *.o for dependencies:\n\n * It *is* a trade-off since we'll pay the extra cost of running the C\n compiler, but we're probably doing that anyway. The compiler is much\n faster than \"spatch\", so even though we need to re-compile the *.o to\n create the dependency info for the *.c for \"spatch\" it's\n faster (especially if using \"ccache\").\n\n * There *are* use-cases where some would like to have *.o files\n around, but to have the \"make coccicheck\" ignore them. See:\n https://lore.kernel.org/git/20220826104312.GJ1735@szeder.dev/\n\n For those users a:\n\n\tmake\n\tmake coccicheck SPATCH_USE_O_DEPENDENCIES=\n\n Will avoid considering the *.o files.\n\n * If that *.o file doesn't exist we'll depend on an intermediate file\n of ours which in turn depends on $(FOUND_H_SOURCES).\n\n This covers both an initial build, or where \"coccicheck\" is run\n without running \"all\" beforehand, and because we run \"coccicheck\"\n on e.g. files in compat/* that we don't know how to build unless\n the requisite flag was provided to the Makefile.\n\n Most of the runtime of \"incremental\" runs is now spent on various\n compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and\n therefore conflate whether we *can* compile an object and generate\n dependency information for it with whether we'd like to link it\n into our binary.\n\n Before this change the distinction didn't matter, but now one way\n to make this even faster on incremental builds would be to peel\n those concerns apart so that we can see that e.g. compat/mmap.c\n doesn't depend on column.h.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 15 +++++++++++++--\n 1 file changed, 13 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 4e1da7206db..bd3b5c21d6f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1303,6 +1303,13 @@ SPATCH_INCLUDE_FLAGS = --all-includes\n SPATCH_FLAGS =\n SPATCH_TEST_FLAGS =\n \n+# If *.o files are present, have \"coccicheck\" depend on them, with\n+# COMPUTE_HEADER_DEPENDENCIES this will speed up the common-case of\n+# only needing to re-generate coccicheck results for the users of a\n+# given API if it's changed, and not all files in the project. If\n+# COMPUTE_HEADER_DEPENDENCIES=no this will be unset too.\n+SPATCH_USE_O_DEPENDENCIES = YesPlease\n+\n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n TRACK_SPATCH_DEFINES += $(SPATCH)\n@@ -3175,14 +3182,18 @@ COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \t$(call mkdir_p_parent_template)\n \t$(QUIET_GEN) >$@\n \n+ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no)\n+SPATCH_USE_O_DEPENDENCIES =\n+endif\n define cocci-rule\n \n ## Rule for .build/$(1).patch/$(2); Params:\n # $(1) = e.g. \"free.cocci\"\n # $(2) = e.g. \"grep.c\"\n+# $(3) = e.g. \"grep.o\"\n COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n .build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n-.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES\n+.build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES)\n .build/$(1).patch/$(2): $(1)\n .build/$(1).patch/$(2): .build/$(1).patch/% : %\n \t$$(call mkdir_p_parent_template)\n@@ -3199,7 +3210,7 @@ endef\n \n define cocci-matrix\n \n-$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(1),$(s)))\n+$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o)))\n endef\n \n ifdef COCCI_GOALS\n-- \n2.38.0.1092.g8c0298861b0\n\n"},{"id":"464947","messageId":"patch-v3-11.11-f7ca3f9c9af-20221014T152553Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v3-00.11-00000000000-20221014T152552Z-avarab@gmail.com","subject":"[PATCH v3 11/11] spatchcache: add a ccache-alike for \"spatch\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-14T15:31:27Z","receivedAt":"2022-10-14T15:32:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a rather trivial \"spatchcache\", with this running e.g.:\n\n\tmake cocciclean\n\tmake contrib/coccinelle/free.cocci.patch \\\n\t\tSPATCH=contrib/coccicheck/spatchcache \\\n\t\tSPATCH_FLAGS=--very-quiet\n\nIs cut down from ~20s to ~5s on my system. Much of that is either\nfixable shell overhead, or the around 40 files we \"CANTCACHE\" (see the\nimplementation).\n\nThis uses \"redis\" as a cache by default, but it's configurable. See\nthe embedded documentation.\n\nThis is *not* like ccache in that we won't cache failed spatch\ninvocations, or those where spatch suggests changes for us. Those\ncases are so rare that I didn't think it was worth the bother, by far\nthe most common case is that it has no suggested changes. We'll also\nrefuse to cache any \"spatch\" invocation that has output on stderr,\nwhich means that \"--very-quiet\" must be added to \"SPATCH_FLAGS\".\n\nBecause we narrow the cache to that we don't need to save away stdout,\nstderr & the exit code. We simply cache the cases where we had no\nsuggested changes.\n\nAnother benchmark is to compare this with the previous\nSPATCH_BATCH_SIZE=N, as noted in [1]. Before this (on my 8 core system) running:\n\n\tmake clean; time make contrib/coccinelle/array.cocci.patch SPATCH_BATCH_SIZE=0\n\nWould take 33s, but with the preceding changes running without this\n\"spatchcache\" is slightly slower, or around 35s:\n\n\tmake clean; time make contrib/coccinelle/array.cocci.patch\n\nNow doing the same with SPATCH=contrib/coccinelle/spatchcache will\ntake around 6s, but we'll need to compile the *.o files first to take\nfull advantage of it (which can be fast with \"ccache\"):\n\n\tmake clean; make; time make contrib/coccinelle/array.cocci.patch SPATCH=contrib/coccinelle/spatchcache\n\n1. https://lore.kernel.org/git/YwdRqP1CyUAzCEn2@coredump.intra.peff.net/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n contrib/coccinelle/spatchcache | 272 +++++++++++++++++++++++++++++++++\n 1 file changed, 272 insertions(+)\n create mode 100755 contrib/coccinelle/spatchcache\n\ndiff --git a/contrib/coccinelle/spatchcache b/contrib/coccinelle/spatchcache\nnew file mode 100755\nindex 00000000000..7ec0dfcb1e4\n--- /dev/null\n+++ b/contrib/coccinelle/spatchcache\n@@ -0,0 +1,272 @@\n+#!/bin/sh\n+#\n+# spatchcache: a poor-man's \"ccache\"-alike for \"spatch\" in git.git\n+#\n+# This caching command relies on the peculiarities of the Makefile\n+# driving \"spatch\" in git.git, in particular if we invoke:\n+#\n+#\tmake\n+#\tmake coccicheck SPATCH_FLAGS=--very-quiet\n+#\n+# We can with COMPUTE_HEADER_DEPENDENCIES (auto-detected as true with\n+# \"gcc\" and \"clang\") write e.g. a .depend/grep.o.d for grep.c, when we\n+# compile grep.o.\n+#\n+# The .depend/grep.o.d will have the full header dependency tree of\n+# grep.c, and we can thus cache the output of \"spatch\" by:\n+#\n+#\t1. Hashing all of those files\n+#\t2. Hashing our source file, and the *.cocci rule we're\n+#\t applying\n+#\t3. Running spatch, if suggests no changes (by far the common\n+#\t case) we invoke \"spatchCache.getCmd\" and\n+#\t \"spatchCache.setCmd\" with a hash SHA-256 to ask \"does this\n+#\t ID have no changes\" or \"say that ID had no changes>\n+#\t4. If no \"spatchCache.{set,get}Cmd\" is specified we'll use\n+#\t \"redis-cli\" and maintain a SET called \"spatch-cache\". Set\n+#\t appropriate redis memory policies to keep it from growing\n+#\t out of control.\n+#\n+# This along with the general incremental \"make\" support for\n+# \"contrib/coccinelle\" makes it viable to (re-)run coccicheck\n+# e.g. when merging integration branches.\n+#\n+# Note that the \"--very-quiet\" flag is currently critical. The cache\n+# will refuse to cache anything that has output on STDERR (which might\n+# be errors from spatch), but see spatchCache.cacheWhenStderr below.\n+#\n+# The STDERR (and exit code) could in principle be cached (as with\n+# ccache), but then the simple structure in the Redis cache would need\n+# to change, so just supply \"--very-quiet\" for now.\n+#\n+# To use this, simply set SPATCH to\n+# contrib/coccinelle/spatchcache. Then optionally set:\n+#\n+#\t[spatchCache]\n+#\t\t# Optional: path to a custom spatch\n+#\t\tspatch = ~/g/coccicheck/spatch.opt\n+#\n+# As well as this trace config (debug implies trace):\n+#\n+#\t\tcacheWhenStderr = true\n+#\t\ttrace = false\n+#\t\tdebug = false\n+#\n+# The \".depend/grep.o.d\" can also be customized, as a string that will\n+# be eval'd, it has access to a \"$dirname\" and \"$basename\":\n+#\n+#\t[spatchCache]\n+#\t\tdependFormat = \"$dirname/.depend/${basename%.c}.o.d\"\n+#\n+# Setting \"trace\" to \"true\" allows for seeing when we have a cache HIT\n+# or MISS. To debug whether the cache is working do that, and run e.g.:\n+#\n+#\tredis-cli FLUSHALL\n+#\t\n+#\tgrep -hore HIT -e MISS -e SET -e NOCACHE -e CANTCACHE .build/contrib/coccinelle | sort | uniq -c\n+#\t 600 CANTCACHE\n+#\t 7365 MISS\n+#\t 7365 SET\n+#\n+# A subsequent \"make cocciclean && make coccicheck\" should then have\n+# all \"HIT\"'s and \"CANTCACHE\"'s.\n+#\n+# The \"spatchCache.cacheWhenStderr\" option is critical when using\n+# spatchCache.{trace,debug} to debug whether something is set in the\n+# cache, as we'll write to the spatch logs in .build/* we'd otherwise\n+# always emit a NOCACHE.\n+\n+set -e\n+\n+## Our own configuration & options\n+debug=$(git config --bool \"spatchCache.debug\")\n+if test \"$debug\" != \"true\"\n+then\n+\tdebug=\n+fi\n+if test -n \"$debug\"\n+then\n+\tset -x\n+fi\n+\n+trace=$(git config --bool \"spatchCache.trace\")\n+if test \"$trace\" != \"true\"\n+then\n+\ttrace=\n+fi\n+if test -n \"$debug\"\n+then\n+\t# debug implies trace\n+\ttrace=true\n+fi\n+\n+cacheWhenStderr=$(git config --bool \"spatchCache.cacheWhenStderr\")\n+if test \"$cacheWhenStderr\" != \"true\"\n+then\n+\tcacheWhenStderr=\n+fi\n+\n+trace_it () {\n+\tif test -z \"$trace\"\n+\tthen\n+\t\treturn\n+\tfi\n+\techo \"$@\" >&2\n+}\n+\n+spatch=$(git config --path \"spatchCache.spatch\" || :)\n+if test -n \"$spatch\"\n+then\n+\tif test -n \"$debug\"\n+\tthen\n+\t\ttrace_it \"custom spatchCache.spatch='$spatch'\"\n+\tfi\n+else\n+\tspatch=spatch\n+fi\n+\n+dependFormat='$dirname/.depend/${basename%.c}.o.d'\n+dependFormatCfg=$(git config \"spatchCache.dependFormat\" || :)\n+if test -n \"$dependFormatCfg\"\n+then\n+\tdependFormat=\"$dependFormatCfg\"\n+fi\n+\n+set=$(git config spatchCache.setCmd || :)\n+get=$(git config spatchCache.getCmd || :)\n+\n+## Parse spatch()-like command-line for caching info\n+arg_sp=\n+arg_file=\n+args=\"$@\"\n+spatch_opts() {\n+\twhile test $# != 0\n+\tdo\n+\t\targ_file=\"$1\"\n+\t\tcase \"$1\" in\n+\t\t--sp-file)\n+\t\t\targ_sp=\"$2\"\n+\t\t\t;;\n+\t\tesac\n+\t\tshift\n+\tdone\n+}\n+spatch_opts \"$@\"\n+if ! test -f \"$arg_file\"\n+then\n+\targ_file=\n+fi\n+\n+hash_for_cache() {\n+\t# Parameters that should affect the cache\n+\techo \"args=$args\"\n+\techo \"config spatchCache.spatch=$spatch\"\n+\techo \"config spatchCache.debug=$debug\"\n+\techo \"config spatchCache.trace=$trace\"\n+\techo \"config spatchCache.cacheWhenStderr=$cacheWhenStderr\"\n+\techo\n+\n+\t# Our target file and its dependencies\n+\tgit hash-object \"$1\" \"$2\" $(grep -E -o '^[^:]+:$' \"$3\" | tr -d ':')\n+}\n+\n+# Sanity checks\n+if ! test -f \"$arg_sp\" && ! test -f \"$arg_file\"\n+then\n+\techo $0: no idea how to cache \"$@\" >&2\n+\texit 128\n+fi\n+\n+# Main logic\n+dirname=$(dirname \"$arg_file\")\n+basename=$(basename \"$arg_file\")\n+eval \"dep=$dependFormat\"\n+\n+if ! test -f \"$dep\"\n+then\n+\ttrace_it \"$0: CANTCACHE have no '$dep' for '$arg_file'!\"\n+\texec \"$spatch\" \"$@\"\n+fi\n+\n+if test -n \"$debug\"\n+then\n+\ttrace_it \"$0: The full cache input for '$arg_sp' '$arg_file' '$dep'\"\n+\thash_for_cache \"$arg_sp\" \"$arg_file\" \"$dep\" >&2\n+fi\n+sum=$(hash_for_cache \"$arg_sp\" \"$arg_file\" \"$dep\" | git hash-object --stdin)\n+\n+trace_it \"$0: processing '$arg_file' with '$arg_sp' rule, and got hash '$sum' for it + '$dep'\"\n+\n+getret=\n+if test -z \"$get\"\n+then\n+\tif test $(redis-cli SISMEMBER spatch-cache \"$sum\") = 1\n+\tthen\n+\t\tgetret=0\n+\telse\n+\t\tgetret=1\n+\tfi\n+else\n+\t$set \"$sum\"\n+\tgetret=$?\n+fi\n+\n+if test \"$getret\" = 0\n+then\n+\ttrace_it \"$0: HIT for '$arg_file' with '$arg_sp'\"\n+\texit 0\n+else\n+\ttrace_it \"$0: MISS: for '$arg_file' with '$arg_sp'\"\n+fi\n+\n+out=\"$(mktemp)\"\n+err=\"$(mktemp)\"\n+\n+set +e\n+\"$spatch\" \"$@\" >\"$out\" 2>>\"$err\"\n+ret=$?\n+cat \"$out\"\n+cat \"$err\" >&2\n+set -e\n+\n+nocache=\n+if test $ret != 0\n+then\n+\tnocache=\"exited non-zero: $ret\"\n+elif test -s \"$out\"\n+then\n+\tnocache=\"had patch output\"\n+elif test -z \"$cacheWhenStderr\" && test -s \"$err\"\n+then\n+\tnocache=\"had stderr (use --very-quiet or spatchCache.cacheWhenStderr=true?)\"\n+fi\n+\n+if test -n \"$nocache\"\n+then\n+\ttrace_it \"$0: NOCACHE ($nocache): for '$arg_file' with '$arg_sp'\"\n+\texit \"$ret\"\n+fi\n+\n+trace_it \"$0: SET: for '$arg_file' with '$arg_sp'\"\n+\n+setret=\n+if test -z \"$set\"\n+then\n+\tif test $(redis-cli SADD spatch-cache \"$sum\") = 1\n+\tthen\n+\t\tsetret=0\n+\telse\n+\t\tsetret=1\n+\tfi\n+else\n+\t\"$set\" \"$sum\"\n+\tsetret=$?\n+fi\n+\n+if test \"$setret\" != 0\n+then\n+\techo \"FAILED to set '$sum' in cache!\" >&2\n+\texit 128\n+fi\n+\n+exit \"$ret\"\n-- \n2.38.0.1092.g8c0298861b0\n\n"},{"id":"464948","messageId":"patch-v3-10.11-52177ea2a68-20221014T152553Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v3-00.11-00000000000-20221014T152552Z-avarab@gmail.com","subject":"[PATCH v3 10/11] cocci: run against a generated ALL.cocci","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-14T15:31:26Z","receivedAt":"2022-10-14T15:33:07Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The preceding commits to make the \"coccicheck\" target incremental made\nit slower in some cases. As an optimization let's not have the\nmany=many mapping of <*.cocci>=<*.[ch]>, but instead concat the\n<*.cocci> into an ALL.cocci, and then run one-to-many\nALL.cocci=<*.[ch]>.\n\nA \"make coccicheck\" is now around 2x as fast as it was on \"master\",\nand around 1.5x as fast as the preceding change to make the run\nincremental:\n\n\t$ git hyperfine -L rev origin/master,HEAD~,HEAD -p 'make clean' 'make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' -r 3\n\tBenchmark 1: make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'origin/master\n\t Time (mean ± σ): 4.258 s ± 0.015 s [User: 27.432 s, System: 1.532 s]\n\t Range (min … max): 4.241 s … 4.268 s 3 runs\n\n\tBenchmark 2: make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'HEAD~\n\t Time (mean ± σ): 5.365 s ± 0.079 s [User: 36.899 s, System: 1.810 s]\n\t Range (min … max): 5.281 s … 5.436 s 3 runs\n\n\tBenchmark 3: make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'HEAD\n\t Time (mean ± σ): 2.725 s ± 0.063 s [User: 14.796 s, System: 0.233 s]\n\t Range (min … max): 2.667 s … 2.792 s 3 runs\n\n\tSummary\n\t 'make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'HEAD' ran\n\t 1.56 ± 0.04 times faster than 'make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'origin/master'\n\t 1.97 ± 0.05 times faster than 'make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'HEAD~'\n\nThis can be turned off with SPATCH_CONCAT_COCCI, but as the\nbeneficiaries of \"SPATCH_CONCAT_COCCI=\" would mainly be those\ndeveloping the *.cocci rules themselves, let's leave this optimization\non by default.\n\nFor more information see my \"Optimizing *.cocci rules by concat'ing\nthem\" (<220901.8635dbjfko.gmgdl@evledraar.gmail.com>) on the\ncocci@inria.fr mailing list.\n\nThis potentially changes the results of our *.cocci rules, but as\nnoted in that discussion it should be safe for our use. We don't name\nrules, or if we do their names don't conflict across our *.cocci\nfiles.\n\nTo the extent that we'd have any inter-dependencies between rules this\ndoesn't make that worse, as we'd have them now if we ran \"make\ncoccicheck\", applied the results, and would then have (due to\nhypothetical interdependencies) suggested changes on the subsequent\n\"make coccicheck\".\n\nOur \"coccicheck-test\" target makes use of the ALL.cocci when running\ntests, e.g. when testing unused.{c,out} we test it against ALL.cocci,\nnot unused.cocci. We thus assert (to the extent that we have test\ncoverage) that this concatenation doesn't change the expected results\nof running these rules.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 40 ++++++++++++++++++++++++++++++++---\n contrib/coccinelle/.gitignore | 1 +\n shared.mak | 1 +\n 3 files changed, 39 insertions(+), 3 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex bd3b5c21d6f..0dc25a2c7ad 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1310,6 +1310,25 @@ SPATCH_TEST_FLAGS =\n # COMPUTE_HEADER_DEPENDENCIES=no this will be unset too.\n SPATCH_USE_O_DEPENDENCIES = YesPlease\n \n+# Set SPATCH_CONCAT_COCCI to concatenate the contrib/cocci/*.cocci\n+# files into a single contrib/cocci/ALL.cocci before running\n+# \"coccicheck\".\n+#\n+# Pros:\n+#\n+# - Speeds up a one-shot run of \"make coccicheck\", as we won't have to\n+# parse *.[ch] files N times for the N *.cocci rules\n+#\n+# Cons:\n+#\n+# - Will make incremental development of *.cocci slower, as\n+# e.g. changing strbuf.cocci will re-run all *.cocci.\n+#\n+# - Makes error and performance analysis harder, as rules will be\n+# applied from a monolithic ALL.cocci, rather than\n+# e.g. strbuf.cocci.\n+SPATCH_CONCAT_COCCI = YesPlease\n+\n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n TRACK_SPATCH_DEFINES += $(SPATCH)\n@@ -3155,9 +3174,10 @@ check: $(GENERATED_H)\n \t\texit 1; \\\n \tfi\n \n+COCCI_GEN_ALL = contrib/coccinelle/ALL.cocci\n COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci)\n-COCCI_RULES = $(COCCI_GLOB)\n-COCCI_NAMES = $(COCCI_RULES:contrib/coccinelle/%.cocci=%)\n+COCCI_NAMES = $(sort ALL $(COCCI_GLOB:contrib/coccinelle/%.cocci=%))\n+COCCI_RULES = $(filter-out $(COCCI_GEN_ALL),$(COCCI_GLOB))\n \n COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES))\n COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n@@ -3171,6 +3191,7 @@ COCCI_RULES_GLOB += cocci%\n COCCI_RULES_GLOB += .build/contrib/coccinelle/%\n COCCI_RULES_GLOB += $(COCCICHECK_PATCHES)\n COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING)\n+COCCI_RULES_GLOB += $(COCCI_GEN_ALL)\n COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS))\n \n COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n@@ -3182,6 +3203,10 @@ COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \t$(call mkdir_p_parent_template)\n \t$(QUIET_GEN) >$@\n \n+$(COCCI_GEN_ALL): $(COCCICHECK)\n+\t$(call mkdir_p_parent_template)\n+\t$(QUIET_SPATCH_CAT)cat $^ >$@\n+\n ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no)\n SPATCH_USE_O_DEPENDENCIES =\n endif\n@@ -3214,7 +3239,7 @@ $(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o)))\n endef\n \n ifdef COCCI_GOALS\n-$(eval $(foreach c,$(COCCI_RULES),$(call cocci-matrix,$(c))))\n+$(eval $(foreach c,$(COCCI_RULES) $(COCCI_GEN_ALL),$(call cocci-matrix,$(c))))\n endif\n \n define spatch-rule\n@@ -3235,7 +3260,11 @@ COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n+ifdef SPATCH_CONCAT_COCCI\n+$(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : $(COCCI_GEN_ALL)\n+else\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n+endif\n \t$(call mkdir_p_parent_template)\n \t$(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_TEST_FLAGS) \\\n \t\t--very-quiet --no-show-diff \\\n@@ -3248,7 +3277,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell\n coccicheck-test: $(COCCI_TEST_RES_GEN)\n \n coccicheck: coccicheck-test\n+ifdef SPATCH_CONCAT_COCCI\n+coccicheck: contrib/coccinelle/ALL.cocci.patch\n+else\n coccicheck: $(COCCICHECK_PATCHES)\n+endif\n \n # See contrib/coccinelle/README\n coccicheck-pending: coccicheck-test\n@@ -3523,6 +3556,7 @@ cocciclean:\n \t$(RM) -r .build/contrib/coccinelle\n \t$(RM) $(COCCICHECK_PATCHES)\n \t$(RM) $(COCCICHECK_PATCHES_PENDING)\n+\t$(RM) $(COCCI_GEN_ALL)\n \n clean: profile-clean coverage-clean cocciclean\n \t$(RM) -r .build\ndiff --git a/contrib/coccinelle/.gitignore b/contrib/coccinelle/.gitignore\nindex d3f29646dc3..2709d98eb91 100644\n--- a/contrib/coccinelle/.gitignore\n+++ b/contrib/coccinelle/.gitignore\n@@ -1 +1,2 @@\n+/ALL.cocci\n *.patch*\ndiff --git a/shared.mak b/shared.mak\nindex 5ccd6889fcb..66dcf6768db 100644\n--- a/shared.mak\n+++ b/shared.mak\n@@ -73,6 +73,7 @@ ifndef V\n ## Used in \"Makefile\": SPATCH\n \tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $@;\n \tQUIET_SPATCH_TEST\t\t= @echo ' ' SPATCH TEST $(@:.build/%=%);\n+\tQUIET_SPATCH_CAT\t\t= @echo ' ' SPATCH CAT $$^ \\>$@;\n \n ## Used in \"Makefile\": SPATCH_*TMPL (quoted for use in \"define\"'s)\n \tQUIET_SPATCH_CAT_TMPL\t\t= @echo ' ' SPATCH CAT $$$$^ \\>$$@;\n-- \n2.38.0.1092.g8c0298861b0\n\n"},{"id":"464987","messageId":"Y0nJB6MEjTsGLWOJ@nand.local","threadId":"58360","inReplyTo":"patch-v3-05.11-ab25b586f38-20221014T152553Z-avarab@gmail.com","subject":"Re: [PATCH v3 05/11] Makefile: split off SPATCH_BATCH_SIZE comment from \"cocci\" heading","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-10-14T20:39:35Z","receivedAt":"2022-10-14T20:40:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Oct 14, 2022 at 05:31:21PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> @@ -1298,10 +1298,11 @@ SP_EXTRA_FLAGS = -Wno-universal-initializer\n> SANITIZE_LEAK =\n> SANITIZE_ADDRESS =\n>\n> -# For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n> +# For the 'coccicheck' target\n> +SPATCH_FLAGS = --all-includes\n> +# Setting SPATCH_BATCH_SIZE higher will\n\nI suppose it goes away once the coccicheck rule becomes incremental a\nfew patches later, but I did find this rewrapping odd.\n\nThanks,\nTaylor\n"},{"id":"465131","messageId":"Y02V0E0JRSNa4Sb9@coredump.intra.peff.net","threadId":"58360","inReplyTo":"cover-v3-00.11-00000000000-20221014T152552Z-avarab@gmail.com","subject":"Re: [PATCH v3 00/11] cocci: make \"incremental\" possible + a ccache-like tool","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-10-17T17:50:08Z","receivedAt":"2022-10-17T17:51:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 14, 2022 at 05:31:16PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> The big change in this belated v3 is that we now run against a\n> concatenated version of all our *.cocci files. This is something I\n> discussed with Jeff King at Git Merge, after having confirmed the\n> viability of that approach on the cocci mailing list.\n\nIs that safe? The last time it was brought up on this list (AFAIK) was\nin:\n\n https://lore.kernel.org/git/alpine.DEB.2.20.1808030755350.2446@hadrien/\n\nwhere Julia said:\n\n I'm surprised that the above cat command would work. Semantic patch rules\n have names, and Coccinelle will not be happy isf two rules have the same\n name. Some may also have variables declared in initializers, although\n perhaps the ones in the kernel don't do this. Causing these variables to\n be shared would not have a good effect.\n\n Putting everything together can also go counter to the optimizations that\n Coccinelle provides. [...]\n\nMaybe we don't do any of the things that could trigger problems in our\nspatch rules. But it's not clear to me what we're risking. Do you have a\nlink for further discussion?\n\n-Peff\n"},{"id":"465135","messageId":"221017.86czaqjnhy.gmgdl@evledraar.gmail.com","threadId":"58360","inReplyTo":"Y02V0E0JRSNa4Sb9@coredump.intra.peff.net","subject":"Re: [PATCH v3 00/11] cocci: make \"incremental\" possible + a ccache-like tool","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-17T18:36:46Z","receivedAt":"2022-10-17T18:47:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Oct 17 2022, Jeff King wrote:\n\n> On Fri, Oct 14, 2022 at 05:31:16PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> The big change in this belated v3 is that we now run against a\n>> concatenated version of all our *.cocci files. This is something I\n>> discussed with Jeff King at Git Merge, after having confirmed the\n>> viability of that approach on the cocci mailing list.\n>\n> Is that safe? The last time it was brought up on this list (AFAIK) was\n> in:\n>\n> https://lore.kernel.org/git/alpine.DEB.2.20.1808030755350.2446@hadrien/\n>\n> where Julia said:\n>\n> I'm surprised that the above cat command would work. Semantic patch rules\n> have names, and Coccinelle will not be happy isf two rules have the same\n> name. Some may also have variables declared in initializers, although\n> perhaps the ones in the kernel don't do this. Causing these variables to\n> be shared would not have a good effect.\n>\n> Putting everything together can also go counter to the optimizations that\n> Coccinelle provides. [...]\n>\n> Maybe we don't do any of the things that could trigger problems in our\n> spatch rules. But it's not clear to me what we're risking. Do you have a\n> link for further discussion?\n\nI think 10/11's commit message should answer your question:\nhttps://lore.kernel.org/git/patch-v3-10.11-52177ea2a68-20221014T152553Z-avarab@gmail.com/\n\nThe tl;dr is that it's not safe in the general case, as noted in the\npost you & the more recent one I linked to in 10/11.\n\nSo, with this series doing:\n\n\tperl -pi -e 's/swap/preincrement/g' contrib/coccinelle/swap.cocci\n\nWill error it if you run it with \"SPATCH_CONCAT_COCCI=Y\", but not with\n\"SPATCH_CONCAT_COCCI=\", as the rule names conflict in the ALL.cocci.\n\nBut as 10/11 notes we can just avoid this by not picking conflicting\nnames, which doesn't seem like an undue burden.\n\nAFAICT we have 5 named rules, and seemingly only 1/4 actually needs its\nname, the rest are apparently only using it for self-documentation, and\nwe could either remove the name, or accomplish the same with a comment:\n\t\n\tdiff --git a/contrib/coccinelle/hashmap.cocci b/contrib/coccinelle/hashmap.cocci\n\tindex d69e120ccff..c5dbb4557b5 100644\n\t--- a/contrib/coccinelle/hashmap.cocci\n\t+++ b/contrib/coccinelle/hashmap.cocci\n\t@@ -1,4 +1,4 @@\n\t-@ hashmap_entry_init_usage @\n\t+@@\n\t expression E;\n\t struct hashmap_entry HME;\n\t @@\n\tdiff --git a/contrib/coccinelle/preincr.cocci b/contrib/coccinelle/preincr.cocci\n\tindex 7fe1e8d2d9a..ae42cb07302 100644\n\t--- a/contrib/coccinelle/preincr.cocci\n\t+++ b/contrib/coccinelle/preincr.cocci\n\t@@ -1,4 +1,4 @@\n\t-@ preincrement @\n\t+@@\n\t identifier i;\n\t @@\n\t - ++i > 1\n\tdiff --git a/contrib/coccinelle/strbuf.cocci b/contrib/coccinelle/strbuf.cocci\n\tindex 0970d98ad72..5f06105df6d 100644\n\t--- a/contrib/coccinelle/strbuf.cocci\n\t+++ b/contrib/coccinelle/strbuf.cocci\n\t@@ -1,4 +1,4 @@\n\t-@ strbuf_addf_with_format_only @\n\t+@@\n\t expression E;\n\t constant fmt !~ \"%\";\n\t @@\n\tdiff --git a/contrib/coccinelle/swap.cocci b/contrib/coccinelle/swap.cocci\n\tindex a0934d1fdaf..522177afb66 100644\n\t--- a/contrib/coccinelle/swap.cocci\n\t+++ b/contrib/coccinelle/swap.cocci\n\t@@ -1,4 +1,4 @@\n\t-@ swap_with_declaration @\n\t+@@\n\t type T;\n\t identifier tmp;\n\t T a, b;\n\t\n"},{"id":"465138","messageId":"xmqqpmeqz2rn.fsf@gitster.g","threadId":"58360","inReplyTo":"221017.86czaqjnhy.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 00/11] cocci: make \"incremental\" possible + a ccache-like tool","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-17T19:08:44Z","receivedAt":"2022-10-17T19:08:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> But as 10/11 notes we can just avoid this by not picking conflicting\n> names, which doesn't seem like an undue burden.\n\nAs long as it is clearly documented to make it well known to\neverybody who would attempt to touch or add .cocci files, I am OK\nwith that direction.\n\n> AFAICT we have 5 named rules, and seemingly only 1/4 actually needs its\n> name, the rest are apparently only using it for self-documentation, and\n> we could either remove the name, or accomplish the same with a comment:\n\nWell, if they are for self-documentation, they'd better be unique,\nor they are not self-documenting enough, no ;-)?\n"},{"id":"465140","messageId":"Y02qgLP12p0c+XWB@coredump.intra.peff.net","threadId":"58360","inReplyTo":"221017.86czaqjnhy.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 00/11] cocci: make \"incremental\" possible + a ccache-like tool","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-10-17T19:18:24Z","receivedAt":"2022-10-17T19:18:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 17, 2022 at 08:36:46PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> > Maybe we don't do any of the things that could trigger problems in our\n> > spatch rules. But it's not clear to me what we're risking. Do you have a\n> > link for further discussion?\n> \n> I think 10/11's commit message should answer your question:\n> https://lore.kernel.org/git/patch-v3-10.11-52177ea2a68-20221014T152553Z-avarab@gmail.com/\n> \n> The tl;dr is that it's not safe in the general case, as noted in the\n> post you & the more recent one I linked to in 10/11.\n\nThanks. Holy cow, the coccinelle list is hard to use compared to\nlore/public-inbox. The direct link to the thread in question is:\n\n https://sympa.inria.fr/sympa/arc/cocci/2022-09/msg00001.html\n\n(you'll need to click a javascript button to see it, though).\n\n> So, with this series doing:\n> \n> \tperl -pi -e 's/swap/preincrement/g' contrib/coccinelle/swap.cocci\n> \n> Will error it if you run it with \"SPATCH_CONCAT_COCCI=Y\", but not with\n> \"SPATCH_CONCAT_COCCI=\", as the rule names conflict in the ALL.cocci.\n> \n> But as 10/11 notes we can just avoid this by not picking conflicting\n> names, which doesn't seem like an undue burden.\n\nYeah, that seems OK, then. In fact, I'd be fine with guidance in the\nREADME saying \"don't bother with a name if you don't need it\".\n\n-Peff\n"},{"id":"465765","messageId":"cover-v4-00.12-00000000000-20221026T141005Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v3-00.11-00000000000-20221014T152552Z-avarab@gmail.com","subject":"[PATCH v4 00/12] cocci: make \"incremental\" possible + a ccache-like tool","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-26T14:20:27Z","receivedAt":"2022-10-26T14:21:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A re-roll of the series to have \"make coccicheck\" run\nincrementally. For the v1 and general overview see:\nhttps://lore.kernel.org/git/cover-0.5-00000000000-20220825T141212Z-avarab@gmail.com/\n\nFor me a \"make coccicheck\" on master takes ~7m, with\n\"SPATCH_BATCH_SIZE=0\" (the usual way to speed it up on a beefy box)\nit's ~6m.\n\nWith this topic it takes 2m30s with no caching, and using the\n\"spatchcache\" tool in 12/12 10s. The ~6-7m to ~2m30s speed-up is\nmostly 11/12, i.e. we now concatenate the rules.\n\nBut being able to run this in ~10s once you set up \"spatchcache\" means\nthat incremental runs become viable (e.g. sanity checks when merging\ntopics in a loop).\n\nThese numbers are on a 8 core colo'd box, so YMMV. For GitHub CI the\n\"static-analysis\" (which has more than just \"make coccicheck\") now\ntakes ~8m instead of ~15m.\n\nChanges since v4:\n\n * update contrib/coccinelle/README as we go, see the range-diff\n below.\n\n * Added a 10/12 to remove named rules that didn't need their names,\n as we make an ALL.cocci these are more likely to collide.\n\n * I made the spatchcache tool faster, it spend ~40ms per run on a\n cache hit, mostly on running \"git config\", now it can be told via\n environment variables to use its default, and not ask the config,\n which makes it take ~10ms on a cache hit.\n\n * We now explain in 11/12's README addition that rule names are now\n global, as we concatenate all the *.cocci files.\n\nÆvar Arnfjörð Bjarmason (12):\n Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)\n cocci rules: remove unused \"F\" metavariable from pending rule\n Makefile: add ability to TAB-complete cocci *.patch rules\n Makefile: have \"coccicheck\" re-run if flags change\n Makefile: split off SPATCH_BATCH_SIZE comment from \"cocci\" heading\n cocci: split off include-less \"tests\" from SPATCH_FLAGS\n cocci: split off \"--all-includes\" from SPATCH_FLAGS\n cocci: make \"coccicheck\" rule incremental\n cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES\n cocci rules: remove 's from rules that don't need them\n cocci: run against a generated ALL.cocci\n spatchcache: add a ccache-alike for \"spatch\"\n\n .gitignore | 1 +\n Makefile | 159 +++++++--\n contrib/coccinelle/.gitignore | 1 +\n contrib/coccinelle/README | 49 +++\n contrib/coccinelle/hashmap.cocci | 2 +-\n contrib/coccinelle/preincr.cocci | 2 +-\n contrib/coccinelle/spatchcache | 304 ++++++++++++++++++\n contrib/coccinelle/strbuf.cocci | 2 +-\n contrib/coccinelle/swap.cocci | 2 +-\n .../coccinelle/the_repository.pending.cocci | 1 -\n shared.mak | 10 +-\n 11 files changed, 501 insertions(+), 32 deletions(-)\n create mode 100755 contrib/coccinelle/spatchcache\n\nRange-diff against v3:\n 1: 4494c91df9a = 1: c0306e3c397 Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)\n 2: 8219b1b12f2 = 2: 14e757cc020 cocci rules: remove unused \"F\" metavariable from pending rule\n 3: 6dbfafa08fd = 3: 210dbc6b101 Makefile: add ability to TAB-complete cocci *.patch rules\n 4: f779a2d22aa = 4: b859dae7509 Makefile: have \"coccicheck\" re-run if flags change\n 5: ab25b586f38 ! 5: 11b9227722b Makefile: split off SPATCH_BATCH_SIZE comment from \"cocci\" heading\n @@ Commit message\n preceded it. As subsequent commits will add such comments we need to\n split the existing comment up.\n \n + The wrapping for the \"SPATCH_BATCH_SIZE\" is now a bit odd, but\n + minimizes the diff size. As a subsequent commit will remove that\n + feature altogether this is worth it.\n +\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## Makefile ##\n 6: 691be73b6fb = 6: 1c30f4b543b cocci: split off include-less \"tests\" from SPATCH_FLAGS\n 7: 2ca5ea5beca = 7: 86827388627 cocci: split off \"--all-includes\" from SPATCH_FLAGS\n 8: 2072a508064 = 8: 340ce523436 cocci: make \"coccicheck\" rule incremental\n 9: 739652eada9 ! 9: 134b3743a60 cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES\n @@ Makefile: endef\n endef\n \n ifdef COCCI_GOALS\n +\n + ## contrib/coccinelle/README ##\n +@@ contrib/coccinelle/README: There are two types of semantic patches:\n + \n + This allows to expose plans of pending large scale refactorings without\n + impacting the bad pattern checks.\n ++\n ++Git-specific tips & things to know about how we run \"spatch\":\n ++\n ++ * The \"make coccicheck\" will piggy-back on\n ++ \"COMPUTE_HEADER_DEPENDENCIES\". If you've built a given object file\n ++ the \"coccicheck\" target will consider its depednency to decide if\n ++ it needs to re-run on the corresponding source file.\n ++\n ++ This means that a \"make coccicheck\" will re-compile object files\n ++ before running. This might be unexpected, but speeds up the run in\n ++ the common case, as e.g. a change to \"column.h\" won't require all\n ++ coccinelle rules to be re-run against \"grep.c\" (or another file\n ++ that happens not to use \"column.h\").\n ++\n ++ To disable this behavior use the \"SPATCH_USE_O_DEPENDENCIES=NoThanks\"\n ++ flag.\n -: ----------- > 10: 89f778ac055 cocci rules: remove 's from rules that don't need them\n10: 52177ea2a68 ! 11: a848d09527f cocci: run against a generated ALL.cocci\n @@ contrib/coccinelle/.gitignore\n +/ALL.cocci\n *.patch*\n \n + ## contrib/coccinelle/README ##\n +@@ contrib/coccinelle/README: Git-specific tips & things to know about how we run \"spatch\":\n + \n + To disable this behavior use the \"SPATCH_USE_O_DEPENDENCIES=NoThanks\"\n + flag.\n ++\n ++ * To speed up our rules the \"make coccicheck\" target will by default\n ++ concatenate all of the *.cocci files here into an \"ALL.cocci\", and\n ++ apply it to each source file.\n ++\n ++ This makes the run faster, as we don't need to run each rule\n ++ against each source file. See the Makefile for further discussion,\n ++ this behavior can be disabled with \"SPATCH_CONCAT_COCCI=\".\n ++\n ++ But since they're concatenated any in the (e.g. \"@\n ++ my_name\", v.s. anonymous \"@@\") needs to be unique across all our\n ++ *.cocci files. You should only need to name rules if other rules\n ++ depend on them (currently only one rule is named).\n +\n ## shared.mak ##\n @@ shared.mak: ifndef V\n ## Used in \"Makefile\": SPATCH\n11: f7ca3f9c9af ! 12: a728aa301a6 spatchcache: add a ccache-alike for \"spatch\"\n @@ Commit message\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n + ## contrib/coccinelle/README ##\n +@@ contrib/coccinelle/README: Git-specific tips & things to know about how we run \"spatch\":\n + my_name\", v.s. anonymous \"@@\") needs to be unique across all our\n + *.cocci files. You should only need to name rules if other rules\n + depend on them (currently only one rule is named).\n ++\n ++ * To speed up incremental runs even more use the \"spatchcache\" tool\n ++ in this directory as your \"SPATCH\". It aimns to be a \"ccache\" for\n ++ coccinelle, and piggy-backs on \"COMPUTE_HEADER_DEPENDENCIES\".\n ++\n ++ It caches in Redis by default, see it source for a how-to.\n ++\n ++ In one setup with a primed cache \"make coccicheck\" followed by a\n ++ \"make clean && make\" takes around 10s to run, but 2m30s with the\n ++ default of \"SPATCH_CONCAT_COCCI=Y\".\n ++\n ++ With \"SPATCH_CONCAT_COCCI=\" the total runtime is around ~6m, sped\n ++ up to ~1m with \"spatchcache\".\n ++\n ++ Most of the 10s (or ~1m) being spent on re-running \"spatch\" on\n ++ files we couldn't cache, as we didn't compile them (in contrib/*\n ++ and compat/* mostly).\n ++\n ++ The absolute times will differ for you, but the relative speedup\n ++ from caching should be on that order.\n +\n ## contrib/coccinelle/spatchcache (new) ##\n @@\n +#!/bin/sh\n @@ contrib/coccinelle/spatchcache (new)\n +# driving \"spatch\" in git.git, in particular if we invoke:\n +#\n +#\tmake\n ++#\t# See \"spatchCache.cacheWhenStderr\" for why \"--very-quiet\" is\n ++#\t# used\n +#\tmake coccicheck SPATCH_FLAGS=--very-quiet\n +#\n +# We can with COMPUTE_HEADER_DEPENDENCIES (auto-detected as true with\n @@ contrib/coccinelle/spatchcache (new)\n +# spatchCache.{trace,debug} to debug whether something is set in the\n +# cache, as we'll write to the spatch logs in .build/* we'd otherwise\n +# always emit a NOCACHE.\n ++#\n ++# Reading the config can make the command much slower, to work around\n ++# this the config can be set in the environment, with environment\n ++# variable name corresponding to the config key. \"default\" can be used\n ++# to use whatever's the script default, e.g. setting\n ++# spatchCache.cacheWhenStderr=true and deferring to the defaults for\n ++# the rest is:\n ++#\n ++#\texport GIT_CONTRIB_SPATCHCACHE_DEBUG=default\n ++#\texport GIT_CONTRIB_SPATCHCACHE_TRACE=default\n ++#\texport GIT_CONTRIB_SPATCHCACHE_CACHEWHENSTDERR=true\n ++#\texport GIT_CONTRIB_SPATCHCACHE_SPATCH=default\n ++#\texport GIT_CONTRIB_SPATCHCACHE_DEPENDFORMAT=default\n ++#\texport GIT_CONTRIB_SPATCHCACHE_SETCMD=default\n ++#\texport GIT_CONTRIB_SPATCHCACHE_GETCMD=default\n +\n +set -e\n +\n ++env_or_config () {\n ++\tenv=\"$1\"\n ++\tshift\n ++\tif test \"$env\" = \"default\"\n ++\tthen\n ++\t\t# Avoid expensive \"git config\" invocation\n ++\t\treturn\n ++\telif test -n \"$env\"\n ++\tthen\n ++\t\techo \"$env\"\n ++\telse\n ++\t\tgit config $@ || :\n ++\tfi\n ++}\n ++\n +## Our own configuration & options\n -+debug=$(git config --bool \"spatchCache.debug\")\n ++debug=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_DEBUG\" --bool \"spatchCache.debug\")\n +if test \"$debug\" != \"true\"\n +then\n +\tdebug=\n @@ contrib/coccinelle/spatchcache (new)\n +\tset -x\n +fi\n +\n -+trace=$(git config --bool \"spatchCache.trace\")\n ++trace=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_TRACE\" --bool \"spatchCache.trace\")\n +if test \"$trace\" != \"true\"\n +then\n +\ttrace=\n @@ contrib/coccinelle/spatchcache (new)\n +\ttrace=true\n +fi\n +\n -+cacheWhenStderr=$(git config --bool \"spatchCache.cacheWhenStderr\")\n ++cacheWhenStderr=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_CACHEWHENSTDERR\" --bool \"spatchCache.cacheWhenStderr\")\n +if test \"$cacheWhenStderr\" != \"true\"\n +then\n +\tcacheWhenStderr=\n @@ contrib/coccinelle/spatchcache (new)\n +\techo \"$@\" >&2\n +}\n +\n -+spatch=$(git config --path \"spatchCache.spatch\" || :)\n ++spatch=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_SPATCH\" --path \"spatchCache.spatch\")\n +if test -n \"$spatch\"\n +then\n +\tif test -n \"$debug\"\n @@ contrib/coccinelle/spatchcache (new)\n +fi\n +\n +dependFormat='$dirname/.depend/${basename%.c}.o.d'\n -+dependFormatCfg=$(git config \"spatchCache.dependFormat\" || :)\n ++dependFormatCfg=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_DEPENDFORMAT\" \"spatchCache.dependFormat\")\n +if test -n \"$dependFormatCfg\"\n +then\n +\tdependFormat=\"$dependFormatCfg\"\n +fi\n +\n -+set=$(git config spatchCache.setCmd || :)\n -+get=$(git config spatchCache.getCmd || :)\n ++set=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_SETCMD\" \"spatchCache.setCmd\")\n ++get=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_GETCMD\" \"spatchCache.getCmd\")\n +\n +## Parse spatch()-like command-line for caching info\n +arg_sp=\n-- \n2.38.0.1251.g3eefdfb5e7a\n\n"},{"id":"465766","messageId":"patch-v4-01.12-c0306e3c397-20221026T141005Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v4-00.12-00000000000-20221026T141005Z-avarab@gmail.com","subject":"[PATCH v4 01/12] Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-26T14:20:28Z","receivedAt":"2022-10-26T14:21:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"In f7ff6597a75 (cocci: add a \"coccicheck-test\" target and test *.cocci\nrules, 2022-07-05) we abbreviated \"_TEST\" to \"_T\" to have it align\nwith the rest of the \"=\"'s above it.\n\nSubsequent commits will add more QUIET_SPATCH_* variables, so let's\nstop abbreviating this, and indent it in preparation for adding more\nof these variables.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 2 +-\n shared.mak | 6 ++++--\n 2 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 85f03c6aed1..0ca1781510e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3168,7 +3168,7 @@ $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n \t$(call mkdir_p_parent_template)\n-\t$(QUIET_SPATCH_T)$(SPATCH) $(SPATCH_FLAGS) \\\n+\t$(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_FLAGS) \\\n \t\t--very-quiet --no-show-diff \\\n \t\t--sp-file $< -o $@ \\\n \t\t$(@:.build/%.res=%.c) && \\\ndiff --git a/shared.mak b/shared.mak\nindex 33f43edbf9a..96b06acc455 100644\n--- a/shared.mak\n+++ b/shared.mak\n@@ -69,8 +69,10 @@ ifndef V\n \tQUIET_SP = @echo ' ' SP $<;\n \tQUIET_HDR = @echo ' ' HDR $(<:hcc=h);\n \tQUIET_RC = @echo ' ' RC $@;\n-\tQUIET_SPATCH = @echo ' ' SPATCH $<;\n-\tQUIET_SPATCH_T = @echo ' ' SPATCH TEST $(@:.build/%=%);\n+\n+## Used in \"Makefile\": SPATCH\n+\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $<;\n+\tQUIET_SPATCH_TEST\t\t= @echo ' ' SPATCH TEST $(@:.build/%=%);\n \n ## Used in \"Documentation/Makefile\"\n \tQUIET_ASCIIDOC\t= @echo ' ' ASCIIDOC $@;\n-- \n2.38.0.1251.g3eefdfb5e7a\n\n"},{"id":"465767","messageId":"patch-v4-02.12-14e757cc020-20221026T141005Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v4-00.12-00000000000-20221026T141005Z-avarab@gmail.com","subject":"[PATCH v4 02/12] cocci rules: remove unused \"F\" metavariable from pending rule","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-26T14:20:29Z","receivedAt":"2022-10-26T14:21:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix an issue with a rule added in 9b45f499818 (object-store: prepare\nhas_{sha1, object}_file to handle any repo, 2018-11-13). We've been\nspewing out this warning into our $@.log since that rule was added:\n\n\twarning: rule starting on line 21: metavariable F not used in the - or context code\n\nWe should do a better job of scouring our coccinelle log files for\nsuch issues, but for now let's fix this as a one-off.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n contrib/coccinelle/the_repository.pending.cocci | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/contrib/coccinelle/the_repository.pending.cocci b/contrib/coccinelle/the_repository.pending.cocci\nindex 072ea0d9228..747d382ff5f 100644\n--- a/contrib/coccinelle/the_repository.pending.cocci\n+++ b/contrib/coccinelle/the_repository.pending.cocci\n@@ -20,7 +20,6 @@ expression E;\n \n @@\n expression E;\n-expression F;\n @@\n - has_object_file_with_flags(\n + repo_has_object_file_with_flags(the_repository,\n-- \n2.38.0.1251.g3eefdfb5e7a\n\n"},{"id":"465768","messageId":"patch-v4-03.12-210dbc6b101-20221026T141005Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v4-00.12-00000000000-20221026T141005Z-avarab@gmail.com","subject":"[PATCH v4 03/12] Makefile: add ability to TAB-complete cocci *.patch rules","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-26T14:20:30Z","receivedAt":"2022-10-26T14:21:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Declare the contrib/coccinelle/.cocci.patch rules in such a way\nas to allow TAB-completion, and slightly optimize the Makefile by\ncutting down on the number of $(wildcard) in favor of defining\n\"coccicheck\" and \"coccicheck-pending\" in terms of the same\nincrementally filtered list.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 17 ++++++++++++++---\n 1 file changed, 14 insertions(+), 3 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 0ca1781510e..b638a548182 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3140,9 +3140,20 @@ check: $(GENERATED_H)\n \t\texit 1; \\\n \tfi\n \n+COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci)\n+COCCI_RULES = $(COCCI_GLOB)\n+\n+COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES))\n+COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n+\n+COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n+COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n+\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n-%.cocci.patch: %.cocci $(COCCI_SOURCES)\n+COCCI_PATCHES = $(COCCI_RULES:%=%.patch)\n+$(COCCI_PATCHES): $(COCCI_SOURCES)\n+$(COCCI_PATCHES): %.patch: %\n \t$(QUIET_SPATCH) \\\n \tif test $(SPATCH_BATCH_SIZE) = 0; then \\\n \t\tlimit=; \\\n@@ -3179,11 +3190,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell\n coccicheck-test: $(COCCI_TEST_RES_GEN)\n \n coccicheck: coccicheck-test\n-coccicheck: $(addsuffix .patch,$(filter-out %.pending.cocci,$(wildcard contrib/coccinelle/*.cocci)))\n+coccicheck: $(COCCICHECK_PATCHES)\n \n # See contrib/coccinelle/README\n coccicheck-pending: coccicheck-test\n-coccicheck-pending: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.pending.cocci))\n+coccicheck-pending: $(COCCICHECK_PATCHES_PENDING)\n \n .PHONY: coccicheck coccicheck-pending\n \n-- \n2.38.0.1251.g3eefdfb5e7a\n\n"},{"id":"465769","messageId":"patch-v4-04.12-b859dae7509-20221026T141005Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v4-00.12-00000000000-20221026T141005Z-avarab@gmail.com","subject":"[PATCH v4 04/12] Makefile: have \"coccicheck\" re-run if flags change","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-26T14:20:31Z","receivedAt":"2022-10-26T14:21:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix an issue with the \"coccicheck\" family of rules that's been here\nsince 63f0a758a06 (add coccicheck make target, 2016-09-15), unlike\ne.g. \"make grep.o\" we wouldn't re-run it when $(SPATCH) or\n$(SPATCH_FLAGS) changed. To test new flags we needed to first do a\n\"make cocciclean\".\n\nThis now uses the same (copy/pasted) pattern as other \"DEFINES\"\nrules. As a result we'll re-run properly. This can be demonstrated\ne.g. on the issue noted in [1]:\n\n\t$ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c V=1\n\t[...]\n\t SPATCH contrib/coccinelle/xcalloc.cocci\n\t$ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c SPATCH_FLAGS=\"--all-includes --recursive-includes\"\n\t * new spatch flags\n\t SPATCH contrib/coccinelle/xcalloc.cocci\n\t SPATCH result: contrib/coccinelle/xcalloc.cocci.patch\n\t$\n\n1. https://lore.kernel.org/git/20220823095602.GC1735@szeder.dev/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .gitignore | 1 +\n Makefile | 15 +++++++++++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/.gitignore b/.gitignore\nindex 62720c6135d..7cce39270d4 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -7,6 +7,7 @@\n /GIT-PERL-HEADER\n /GIT-PYTHON-VARS\n /GIT-SCRIPT-DEFINES\n+/GIT-SPATCH-DEFINES\n /GIT-USER-AGENT\n /GIT-VERSION-FILE\n /bin-wrappers/\ndiff --git a/Makefile b/Makefile\nindex b638a548182..7d4518653ce 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1305,6 +1305,18 @@ SANITIZE_ADDRESS =\n SPATCH_FLAGS = --all-includes\n SPATCH_BATCH_SIZE = 1\n \n+# Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n+TRACK_SPATCH_DEFINES =\n+TRACK_SPATCH_DEFINES += $(SPATCH)\n+TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n+TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n+GIT-SPATCH-DEFINES: FORCE\n+\t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n+\t if test x\"$$FLAGS\" != x\"`cat GIT-SPATCH-DEFINES 2>/dev/null`\" ; then \\\n+\t\techo >&2 \" * new spatch flags\"; \\\n+\t\techo \"$$FLAGS\" >GIT-SPATCH-DEFINES; \\\n+ fi\n+\n include config.mak.uname\n -include config.mak.autogen\n -include config.mak\n@@ -3152,6 +3164,7 @@ COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n COCCI_PATCHES = $(COCCI_RULES:%=%.patch)\n+$(COCCI_PATCHES): GIT-SPATCH-DEFINES\n $(COCCI_PATCHES): $(COCCI_SOURCES)\n $(COCCI_PATCHES): %.patch: %\n \t$(QUIET_SPATCH) \\\n@@ -3175,6 +3188,7 @@ $(COCCI_PATCHES): %.patch: %\n \tfi\n \n COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n+$(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n@@ -3461,6 +3475,7 @@ profile-clean:\n \t$(RM) $(addsuffix *.gcno,$(addprefix $(PROFILE_DIR)/, $(object_dirs)))\n \n cocciclean:\n+\t$(RM) GIT-SPATCH-DEFINES\n \t$(RM) -r .build/contrib/coccinelle\n \t$(RM) contrib/coccinelle/*.cocci.patch*\n \n-- \n2.38.0.1251.g3eefdfb5e7a\n\n"},{"id":"465770","messageId":"patch-v4-05.12-11b9227722b-20221026T141005Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v4-00.12-00000000000-20221026T141005Z-avarab@gmail.com","subject":"[PATCH v4 05/12] Makefile: split off SPATCH_BATCH_SIZE comment from \"cocci\" heading","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-26T14:20:32Z","receivedAt":"2022-10-26T14:21:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Split off the \"; setting[...]\" part of the comment added in In\n960154b9c17 (coccicheck: optionally batch spatch invocations,\n2019-05-06), and restore what we had before that, which was a comment\nindicating that variables for the \"coccicheck\" target were being set\nhere.\n\nWhen 960154b9c17 amended the heading to discuss SPATCH_BATCH_SIZE it\nleft no natural place to add a new comment about other flags that\npreceded it. As subsequent commits will add such comments we need to\nsplit the existing comment up.\n\nThe wrapping for the \"SPATCH_BATCH_SIZE\" is now a bit odd, but\nminimizes the diff size. As a subsequent commit will remove that\nfeature altogether this is worth it.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 7d4518653ce..585d259c841 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1299,10 +1299,11 @@ SP_EXTRA_FLAGS = -Wno-universal-initializer\n SANITIZE_LEAK =\n SANITIZE_ADDRESS =\n \n-# For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n+# For the 'coccicheck' target\n+SPATCH_FLAGS = --all-includes\n+# Setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n-SPATCH_FLAGS = --all-includes\n SPATCH_BATCH_SIZE = 1\n \n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n-- \n2.38.0.1251.g3eefdfb5e7a\n\n"},{"id":"465771","messageId":"patch-v4-07.12-86827388627-20221026T141005Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v4-00.12-00000000000-20221026T141005Z-avarab@gmail.com","subject":"[PATCH v4 07/12] cocci: split off \"--all-includes\" from SPATCH_FLAGS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-26T14:20:34Z","receivedAt":"2022-10-26T14:21:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Per the rationale in 7b63ea57500 (Makefile: remove mandatory \"spatch\"\narguments from SPATCH_FLAGS, 2022-07-05) we have certain flags that\nare truly mandatory, such as \"--sp-file\" and \"--patch .\". The\n\"--all-includes\" flag is also critical, but per [1] we might want to\nad-hoc tweak it occasionally for testing or one-offs.\n\nBut being unable to set e.g. SPATCH_FLAGS=\"--verbose-parsing\" without\nbreaking how our \"spatch\" works isn't ideal, i.e. before this we'd\nneed to know about the default include flags, and specify:\nSPATCH_FLAGS=\"--all-includes --verbose-parsing\".\n\nIf we were then to change the default include flag (e.g. to\n\"--recursive-includes\") in the future any such one-off commands would\nneed to be correspondingly updated.\n\nLet's instead leave the SPATCH_FLAGS for the user, while creating a\nnew SPATCH_INCLUDE_FLAGS to allow for ad-hoc testing of the include\nstrategy itself.\n\n1. https://lore.kernel.org/git/20220823095733.58685-1-szeder.dev@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex 9d04b18a5b5..b153a9db0ac 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1300,7 +1300,8 @@ SANITIZE_LEAK =\n SANITIZE_ADDRESS =\n \n # For the 'coccicheck' target\n-SPATCH_FLAGS = --all-includes\n+SPATCH_INCLUDE_FLAGS = --all-includes\n+SPATCH_FLAGS =\n SPATCH_TEST_FLAGS =\n # Setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n@@ -1310,6 +1311,7 @@ SPATCH_BATCH_SIZE = 1\n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n TRACK_SPATCH_DEFINES += $(SPATCH)\n+TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n@@ -3178,6 +3180,7 @@ $(COCCI_PATCHES): %.patch: %\n \tfi; \\\n \tif ! echo $(COCCI_SOURCES) | xargs $$limit \\\n \t\t$(SPATCH) $(SPATCH_FLAGS) \\\n+\t\t$(SPATCH_INCLUDE_FLAGS) \\\n \t\t--sp-file $< --patch . \\\n \t\t>$@+ 2>$@.log; \\\n \tthen \\\n-- \n2.38.0.1251.g3eefdfb5e7a\n\n"},{"id":"465772","messageId":"patch-v4-06.12-1c30f4b543b-20221026T141005Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v4-00.12-00000000000-20221026T141005Z-avarab@gmail.com","subject":"[PATCH v4 06/12] cocci: split off include-less \"tests\" from SPATCH_FLAGS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-26T14:20:33Z","receivedAt":"2022-10-26T14:21:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Amend the \"coccicheck-test\" rule added in f7ff6597a75 (cocci: add a\n\"coccicheck-test\" target and test *.cocci rules, 2022-07-05) to stop\nusing \"--all-includes\". The flags we'll need for the tests are\ndifferent than the ones we'll need for our main source code.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex 585d259c841..9d04b18a5b5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1301,6 +1301,7 @@ SANITIZE_ADDRESS =\n \n # For the 'coccicheck' target\n SPATCH_FLAGS = --all-includes\n+SPATCH_TEST_FLAGS =\n # Setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n@@ -1310,6 +1311,7 @@ SPATCH_BATCH_SIZE = 1\n TRACK_SPATCH_DEFINES =\n TRACK_SPATCH_DEFINES += $(SPATCH)\n TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n+TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n GIT-SPATCH-DEFINES: FORCE\n \t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n@@ -3194,7 +3196,7 @@ $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n \t$(call mkdir_p_parent_template)\n-\t$(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_FLAGS) \\\n+\t$(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_TEST_FLAGS) \\\n \t\t--very-quiet --no-show-diff \\\n \t\t--sp-file $< -o $@ \\\n \t\t$(@:.build/%.res=%.c) && \\\n-- \n2.38.0.1251.g3eefdfb5e7a\n\n"},{"id":"465773","messageId":"patch-v4-08.12-340ce523436-20221026T141005Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v4-00.12-00000000000-20221026T141005Z-avarab@gmail.com","subject":"[PATCH v4 08/12] cocci: make \"coccicheck\" rule incremental","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-26T14:20:35Z","receivedAt":"2022-10-26T14:21:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Optimize the very slow \"coccicheck\" target to take advantage of\nincremental rebuilding, and fix outstanding dependency problems with\nthe existing rule.\n\nThe rule is now faster both on the initial run as we can make better\nuse of GNU make's parallelism than the old ad-hoc combination of\nmake's parallelism combined with $(SPATCH_BATCH_SIZE) and/or the\n\"--jobs\" argument to \"spatch(1)\".\n\nIt also makes us *much* faster when incrementally building, it's now\nviable to \"make coccicheck\" as topic branches are merged down.\n\nThe rule didn't use FORCE (or its equivalents) before, so a:\n\n\tmake coccicheck\n\tmake coccicheck\n\nWould report nothing to do on the second iteration. But all of our\npatch output depended on all $(COCCI_SOURCES) files, therefore e.g.:\n\n make -W grep.c coccicheck\n\nWould do a full re-run, i.e. a a change in a single file would force\nus to do a full re-run.\n\nThe reason for this (not the initial rationale, but my analysis) is:\n\n* Since we create a single \"*.cocci.patch+\" we don't know where to\n pick up where we left off, or how to incrementally merge e.g. a\n \"grep.c\" change with an existing *.cocci.patch.\n\n* We've been carrying forward the dependency on the *.c files since\n 63f0a758a06 (add coccicheck make target, 2016-09-15) the rule was\n initially added as a sort of poor man's dependency discovery.\n\n As we don't include other *.c files depending on other *.c files\n has always been broken, as could be trivially demonstrated\n e.g. with:\n\n make coccicheck\n make -W strbuf.h coccicheck\n\n However, depending on the corresponding *.c files has been doing\n something, namely that *if* an API change modified both *.c and *.h\n files we'd catch the change to the *.h we care about via the *.c\n being changed.\n\n For API changes that happened only via *.h files we'd do the wrong\n thing before this change, but e.g. for function additions (not\n \"static inline\" ones) catch the *.h change by proxy.\n\nNow we'll instead:\n\n * Create a / pair in the .build directory, E.g. for\n swap.cocci and grep.c we'll create\n .build/contrib/coccinelle/swap.cocci.patch/grep.c.\n\n That file is the diff we'll apply for that -\n combination, if there's no changes to me made (the common case)\n it'll be an empty file.\n\n * Our generated *.patch\n file (e.g. contrib/coccinelle/swap.cocci.patch) is now a simple \"cat\n $^\" of all of all of the / files for a given .\n\n In the case discussed above of \"grep.c\" being changed we'll do the\n full \"cat\" every time, so they resulting *.cocci.patch will always\n be correct and up-to-date, even if it's \"incrementally updated\".\n\n See 1cc0425a27c (Makefile: have \"make pot\" not \"reset --hard\",\n 2022-05-26) for another recent rule that used that technique.\n\nAs before we'll:\n\n * End up generating a contrib/coccinelle/swap.cocci.patch, if we\n \"fail\" by creating a non-empty patch we'll still exit with a zero\n exit code.\n\n Arguably we should move to a more Makefile-native way of doing\n this, i.e. fail early, and if we want all of the \"failed\" changes\n we can use \"make -k\", but as the current\n \"ci/run-static-analysis.sh\" expects us to behave this way let's\n keep the existing behavior of exhaustively discovering all cocci\n changes, and only failing if spatch itself errors out.\n\nFurther implementation details & notes:\n\n * Before this change running \"make coccicheck\" would by default end\n up pegging just one CPU at the very end for a while, usually as\n we'd finish whichever *.cocci rule was the most expensive.\n\n This could be mitigated by combining \"make -jN\" with\n SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch\n spatch invocations, 2019-05-06).\n\n There will be cases where getting rid of \"SPATCH_BATCH_SIZE\" makes\n things worse, but a from-scratch \"make coccicheck\" with the default\n of SPATCH_BATCH_SIZE=1 (and tweaking it doesn't make a difference)\n is faster (~3m36s v.s. ~3m56s) with this approach, as we can feed\n the CPU more work in a less staggered way.\n\n * Getting rid of \"SPATCH_BATCH_SIZE\" particularly helps in cases\n where the default of 1 yields parallelism under \"make coccicheck\",\n but then running e.g.:\n\n make -W contrib/coccinelle/swap.cocci coccicheck\n\n I.e. before that would use only one CPU core, until the user\n remembered to adjust \"SPATCH_BATCH_SIZE\" differently than the\n setting that makes sense when doing a non-incremental run of \"make\n coccicheck\".\n\n * Before the \"make coccicheck\" rule would have to clean\n \"contrib/coccinelle/*.cocci.patch*\", since we'd create \"*+\" and\n \"*.log\" files there. Now those are created in\n .build/contrib/coccinelle/, which is covered by the \"cocciclean\" rule\n already.\n\nOutstanding issues & future work:\n\n * We could get rid of \"--all-includes\" in favor of manually\n specifying a list of includes to give to \"spatch(1)\".\n\n As noted upthread of [1] a naïve removal of \"--all-includes\" will\n result in broken *.cocci patches, but if we know the exhaustive\n list of includes via COMPUTE_HEADER_DEPENDENCIES we don't need to\n re-scan for them, we could grab the headers to include from the\n .depend.d/.o.d and supply them with the \"--include\" option to\n spatch(1).q\n\n1. https://lore.kernel.org/git/87ft18tcog.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 84 +++++++++++++++++++++++++++++++++++++-----------------\n shared.mak | 5 +++-\n 2 files changed, 62 insertions(+), 27 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex b153a9db0ac..7cdbd050339 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1303,10 +1303,6 @@ SANITIZE_ADDRESS =\n SPATCH_INCLUDE_FLAGS = --all-includes\n SPATCH_FLAGS =\n SPATCH_TEST_FLAGS =\n-# Setting SPATCH_BATCH_SIZE higher will\n-# usually result in less CPU usage at the cost of higher peak memory.\n-# Setting it to 0 will feed all files in a single spatch invocation.\n-SPATCH_BATCH_SIZE = 1\n \n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n@@ -1314,7 +1310,6 @@ TRACK_SPATCH_DEFINES += $(SPATCH)\n TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS)\n-TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n GIT-SPATCH-DEFINES: FORCE\n \t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n \t if test x\"$$FLAGS\" != x\"`cat GIT-SPATCH-DEFINES 2>/dev/null`\" ; then \\\n@@ -3159,39 +3154,75 @@ check: $(GENERATED_H)\n \n COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci)\n COCCI_RULES = $(COCCI_GLOB)\n+COCCI_NAMES = $(COCCI_RULES:contrib/coccinelle/%.cocci=%)\n \n COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES))\n+COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n+COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n+\n+# It's expensive to compute the many=many rules below, only eval them\n+# on $(MAKECMDGOALS) that match these $(COCCI_RULES)\n+COCCI_RULES_GLOB =\n+COCCI_RULES_GLOB += cocci%\n+COCCI_RULES_GLOB += .build/contrib/coccinelle/%\n+COCCI_RULES_GLOB += $(COCCICHECK_PATCHES)\n+COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING)\n+COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS))\n \n COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n \n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n-COCCI_PATCHES = $(COCCI_RULES:%=%.patch)\n-$(COCCI_PATCHES): GIT-SPATCH-DEFINES\n-$(COCCI_PATCHES): $(COCCI_SOURCES)\n-$(COCCI_PATCHES): %.patch: %\n-\t$(QUIET_SPATCH) \\\n-\tif test $(SPATCH_BATCH_SIZE) = 0; then \\\n-\t\tlimit=; \\\n-\telse \\\n-\t\tlimit='-n $(SPATCH_BATCH_SIZE)'; \\\n-\tfi; \\\n-\tif ! echo $(COCCI_SOURCES) | xargs $$limit \\\n-\t\t$(SPATCH) $(SPATCH_FLAGS) \\\n-\t\t$(SPATCH_INCLUDE_FLAGS) \\\n-\t\t--sp-file $< --patch . \\\n-\t\t>$@+ 2>$@.log; \\\n+.build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES)\n+\t$(call mkdir_p_parent_template)\n+\t$(QUIET_GEN) >$@\n+\n+define cocci-rule\n+\n+## Rule for .build/$(1).patch/$(2); Params:\n+# $(1) = e.g. \"free.cocci\"\n+# $(2) = e.g. \"grep.c\"\n+COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n+.build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n+.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES\n+.build/$(1).patch/$(2): $(1)\n+.build/$(1).patch/$(2): .build/$(1).patch/% : %\n+\t$$(call mkdir_p_parent_template)\n+\t$$(QUIET_SPATCH)if ! $$(SPATCH) $$(SPATCH_FLAGS) \\\n+\t\t$$(SPATCH_INCLUDE_FLAGS) \\\n+\t\t--sp-file $(1) --patch . $$< \\\n+\t\t>$$@ 2>$$@.log; \\\n \tthen \\\n-\t\tcat $@.log; \\\n+\t\techo \"ERROR when applying '$(1)' to '$$<'; '$$@.log' follows:\"; \\\n+\t\tcat $$@.log; \\\n \t\texit 1; \\\n-\tfi; \\\n-\tmv $@+ $@; \\\n-\tif test -s $@; \\\n+\tfi\n+endef\n+\n+define cocci-matrix\n+\n+$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(1),$(s)))\n+endef\n+\n+ifdef COCCI_GOALS\n+$(eval $(foreach c,$(COCCI_RULES),$(call cocci-matrix,$(c))))\n+endif\n+\n+define spatch-rule\n+\n+contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1))\n+\t$(QUIET_SPATCH_CAT_TMPL)cat $$^ >$$@ && \\\n+\tif test -s $$@; \\\n \tthen \\\n-\t\techo ' ' SPATCH result: $@; \\\n+\t\techo ' ' SPATCH result: $$@; \\\n \tfi\n+endef\n+\n+ifdef COCCI_GOALS\n+$(eval $(foreach n,$(COCCI_NAMES),$(call spatch-rule,$(n))))\n+endif\n \n COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n@@ -3483,7 +3514,8 @@ profile-clean:\n cocciclean:\n \t$(RM) GIT-SPATCH-DEFINES\n \t$(RM) -r .build/contrib/coccinelle\n-\t$(RM) contrib/coccinelle/*.cocci.patch*\n+\t$(RM) $(COCCICHECK_PATCHES)\n+\t$(RM) $(COCCICHECK_PATCHES_PENDING)\n \n clean: profile-clean coverage-clean cocciclean\n \t$(RM) -r .build\ndiff --git a/shared.mak b/shared.mak\nindex 96b06acc455..5ccd6889fcb 100644\n--- a/shared.mak\n+++ b/shared.mak\n@@ -71,9 +71,12 @@ ifndef V\n \tQUIET_RC = @echo ' ' RC $@;\n \n ## Used in \"Makefile\": SPATCH\n-\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $<;\n+\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $@;\n \tQUIET_SPATCH_TEST\t\t= @echo ' ' SPATCH TEST $(@:.build/%=%);\n \n+## Used in \"Makefile\": SPATCH_*TMPL (quoted for use in \"define\"'s)\n+\tQUIET_SPATCH_CAT_TMPL\t\t= @echo ' ' SPATCH CAT $$$$^ \\>$$@;\n+\n ## Used in \"Documentation/Makefile\"\n \tQUIET_ASCIIDOC\t= @echo ' ' ASCIIDOC $@;\n \tQUIET_XMLTO\t= @echo ' ' XMLTO $@;\n-- \n2.38.0.1251.g3eefdfb5e7a\n\n"},{"id":"465774","messageId":"patch-v4-09.12-134b3743a60-20221026T141005Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v4-00.12-00000000000-20221026T141005Z-avarab@gmail.com","subject":"[PATCH v4 09/12] cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-26T14:20:36Z","receivedAt":"2022-10-26T14:21:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improve the incremental rebuilding support of \"coccicheck\" by\npiggy-backing on the computed dependency information of the\ncorresponding *.o file, rather than rebuilding all / pairs\nif either their corresponding file changes, or if any header changes.\n\nThis in effect uses the same method that the \"sparse\" target was made\nto use in c234e8a0ecf (Makefile: make the \"sparse\" target non-.PHONY,\n2021-09-23), except that the dependency on the *.o file isn't a hard\none, we check with $(wildcard) if the *.o file exists, and if so we'll\ndepend on it.\n\nThis means that the common case of:\n\n\tmake\n\tmake coccicheck\n\nWill benefit from incremental rebuilding, now changing e.g. a header\nwill only re-run \"spatch\" on those those *.c files that make use of\nit:\n\nBy depending on the *.o we piggy-back on\nCOMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the\n\"sparse\" target non-.PHONY, 2021-09-23) for prior art of doing that\nfor the *.sp files. E.g.:\n\n make contrib/coccinelle/free.cocci.patch\n make -W column.h contrib/coccinelle/free.cocci.patch\n\nWill take around 15 seconds for the second command on my 8 core box if\nI didn't run \"make\" beforehand to create the *.o files. But around 2\nseconds if I did and we have those \"*.o\" files.\n\nNotes about the approach of piggy-backing on *.o for dependencies:\n\n * It *is* a trade-off since we'll pay the extra cost of running the C\n compiler, but we're probably doing that anyway. The compiler is much\n faster than \"spatch\", so even though we need to re-compile the *.o to\n create the dependency info for the *.c for \"spatch\" it's\n faster (especially if using \"ccache\").\n\n * There *are* use-cases where some would like to have *.o files\n around, but to have the \"make coccicheck\" ignore them. See:\n https://lore.kernel.org/git/20220826104312.GJ1735@szeder.dev/\n\n For those users a:\n\n\tmake\n\tmake coccicheck SPATCH_USE_O_DEPENDENCIES=\n\n Will avoid considering the *.o files.\n\n * If that *.o file doesn't exist we'll depend on an intermediate file\n of ours which in turn depends on $(FOUND_H_SOURCES).\n\n This covers both an initial build, or where \"coccicheck\" is run\n without running \"all\" beforehand, and because we run \"coccicheck\"\n on e.g. files in compat/* that we don't know how to build unless\n the requisite flag was provided to the Makefile.\n\n Most of the runtime of \"incremental\" runs is now spent on various\n compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and\n therefore conflate whether we *can* compile an object and generate\n dependency information for it with whether we'd like to link it\n into our binary.\n\n Before this change the distinction didn't matter, but now one way\n to make this even faster on incremental builds would be to peel\n those concerns apart so that we can see that e.g. compat/mmap.c\n doesn't depend on column.h.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 15 +++++++++++++--\n contrib/coccinelle/README | 16 ++++++++++++++++\n 2 files changed, 29 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 7cdbd050339..531077031d8 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1304,6 +1304,13 @@ SPATCH_INCLUDE_FLAGS = --all-includes\n SPATCH_FLAGS =\n SPATCH_TEST_FLAGS =\n \n+# If *.o files are present, have \"coccicheck\" depend on them, with\n+# COMPUTE_HEADER_DEPENDENCIES this will speed up the common-case of\n+# only needing to re-generate coccicheck results for the users of a\n+# given API if it's changed, and not all files in the project. If\n+# COMPUTE_HEADER_DEPENDENCIES=no this will be unset too.\n+SPATCH_USE_O_DEPENDENCIES = YesPlease\n+\n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n TRACK_SPATCH_DEFINES += $(SPATCH)\n@@ -3179,14 +3186,18 @@ COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \t$(call mkdir_p_parent_template)\n \t$(QUIET_GEN) >$@\n \n+ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no)\n+SPATCH_USE_O_DEPENDENCIES =\n+endif\n define cocci-rule\n \n ## Rule for .build/$(1).patch/$(2); Params:\n # $(1) = e.g. \"free.cocci\"\n # $(2) = e.g. \"grep.c\"\n+# $(3) = e.g. \"grep.o\"\n COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n .build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n-.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES\n+.build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES)\n .build/$(1).patch/$(2): $(1)\n .build/$(1).patch/$(2): .build/$(1).patch/% : %\n \t$$(call mkdir_p_parent_template)\n@@ -3203,7 +3214,7 @@ endef\n \n define cocci-matrix\n \n-$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(1),$(s)))\n+$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o)))\n endef\n \n ifdef COCCI_GOALS\ndiff --git a/contrib/coccinelle/README b/contrib/coccinelle/README\nindex f0e80bd7f03..09ea8298e19 100644\n--- a/contrib/coccinelle/README\n+++ b/contrib/coccinelle/README\n@@ -41,3 +41,19 @@ There are two types of semantic patches:\n \n This allows to expose plans of pending large scale refactorings without\n impacting the bad pattern checks.\n+\n+Git-specific tips & things to know about how we run \"spatch\":\n+\n+ * The \"make coccicheck\" will piggy-back on\n+ \"COMPUTE_HEADER_DEPENDENCIES\". If you've built a given object file\n+ the \"coccicheck\" target will consider its depednency to decide if\n+ it needs to re-run on the corresponding source file.\n+\n+ This means that a \"make coccicheck\" will re-compile object files\n+ before running. This might be unexpected, but speeds up the run in\n+ the common case, as e.g. a change to \"column.h\" won't require all\n+ coccinelle rules to be re-run against \"grep.c\" (or another file\n+ that happens not to use \"column.h\").\n+\n+ To disable this behavior use the \"SPATCH_USE_O_DEPENDENCIES=NoThanks\"\n+ flag.\n-- \n2.38.0.1251.g3eefdfb5e7a\n\n"},{"id":"465775","messageId":"patch-v4-10.12-89f778ac055-20221026T141005Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v4-00.12-00000000000-20221026T141005Z-avarab@gmail.com","subject":"[PATCH v4 10/12] cocci rules: remove 's from rules that don't need them","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-26T14:20:37Z","receivedAt":"2022-10-26T14:21:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The in the part of the coccinelle syntax[1] is for our\npurposes there to declares if we have inter-dependencies between\ndifferent rules.\n\nBut such 's must be unique within a given semantic patch file. As\nwe'll be processing a concatenated version of our rules in the\nsubsequent commit let's remove these names. They weren't being used\nfor the semantic patches themselves, and equated to a short comment\nabout the rule.\n\nBoth the filename and context of the rules makes it clear what they're\ndoing, so we're not gaining anything from keeping these. Retaining\nthem goes against recommendations that \"contrib/coccinelle/README\"\nwill be making in the subsequent commit.\n\nThis leaves only one named rule in our sources, where it's needed for\na \" <-> \" relationship:\n\n\t$ git -P grep '^@ ' -- contrib/coccinelle/\n\tcontrib/coccinelle/swap.cocci:@ swap @\n\tcontrib/coccinelle/swap.cocci:@ extends swap @\n\n1. https://coccinelle.gitlabpages.inria.fr/website/docs/main_grammar.html\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n contrib/coccinelle/hashmap.cocci | 2 +-\n contrib/coccinelle/preincr.cocci | 2 +-\n contrib/coccinelle/strbuf.cocci | 2 +-\n contrib/coccinelle/swap.cocci | 2 +-\n 4 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/contrib/coccinelle/hashmap.cocci b/contrib/coccinelle/hashmap.cocci\nindex d69e120ccff..c5dbb4557b5 100644\n--- a/contrib/coccinelle/hashmap.cocci\n+++ b/contrib/coccinelle/hashmap.cocci\n@@ -1,4 +1,4 @@\n-@ hashmap_entry_init_usage @\n+@@\n expression E;\n struct hashmap_entry HME;\n @@\ndiff --git a/contrib/coccinelle/preincr.cocci b/contrib/coccinelle/preincr.cocci\nindex 7fe1e8d2d9a..ae42cb07302 100644\n--- a/contrib/coccinelle/preincr.cocci\n+++ b/contrib/coccinelle/preincr.cocci\n@@ -1,4 +1,4 @@\n-@ preincrement @\n+@@\n identifier i;\n @@\n - ++i > 1\ndiff --git a/contrib/coccinelle/strbuf.cocci b/contrib/coccinelle/strbuf.cocci\nindex 0970d98ad72..5f06105df6d 100644\n--- a/contrib/coccinelle/strbuf.cocci\n+++ b/contrib/coccinelle/strbuf.cocci\n@@ -1,4 +1,4 @@\n-@ strbuf_addf_with_format_only @\n+@@\n expression E;\n constant fmt !~ \"%\";\n @@\ndiff --git a/contrib/coccinelle/swap.cocci b/contrib/coccinelle/swap.cocci\nindex a0934d1fdaf..522177afb66 100644\n--- a/contrib/coccinelle/swap.cocci\n+++ b/contrib/coccinelle/swap.cocci\n@@ -1,4 +1,4 @@\n-@ swap_with_declaration @\n+@@\n type T;\n identifier tmp;\n T a, b;\n-- \n2.38.0.1251.g3eefdfb5e7a\n\n"},{"id":"465776","messageId":"patch-v4-11.12-a848d09527f-20221026T141005Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v4-00.12-00000000000-20221026T141005Z-avarab@gmail.com","subject":"[PATCH v4 11/12] cocci: run against a generated ALL.cocci","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-26T14:20:38Z","receivedAt":"2022-10-26T14:21:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The preceding commits to make the \"coccicheck\" target incremental made\nit slower in some cases. As an optimization let's not have the\nmany=many mapping of <*.cocci>=<*.[ch]>, but instead concat the\n<*.cocci> into an ALL.cocci, and then run one-to-many\nALL.cocci=<*.[ch]>.\n\nA \"make coccicheck\" is now around 2x as fast as it was on \"master\",\nand around 1.5x as fast as the preceding change to make the run\nincremental:\n\n\t$ git hyperfine -L rev origin/master,HEAD~,HEAD -p 'make clean' 'make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' -r 3\n\tBenchmark 1: make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'origin/master\n\t Time (mean ± σ): 4.258 s ± 0.015 s [User: 27.432 s, System: 1.532 s]\n\t Range (min … max): 4.241 s … 4.268 s 3 runs\n\n\tBenchmark 2: make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'HEAD~\n\t Time (mean ± σ): 5.365 s ± 0.079 s [User: 36.899 s, System: 1.810 s]\n\t Range (min … max): 5.281 s … 5.436 s 3 runs\n\n\tBenchmark 3: make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'HEAD\n\t Time (mean ± σ): 2.725 s ± 0.063 s [User: 14.796 s, System: 0.233 s]\n\t Range (min … max): 2.667 s … 2.792 s 3 runs\n\n\tSummary\n\t 'make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'HEAD' ran\n\t 1.56 ± 0.04 times faster than 'make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'origin/master'\n\t 1.97 ± 0.05 times faster than 'make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'HEAD~'\n\nThis can be turned off with SPATCH_CONCAT_COCCI, but as the\nbeneficiaries of \"SPATCH_CONCAT_COCCI=\" would mainly be those\ndeveloping the *.cocci rules themselves, let's leave this optimization\non by default.\n\nFor more information see my \"Optimizing *.cocci rules by concat'ing\nthem\" (<220901.8635dbjfko.gmgdl@evledraar.gmail.com>) on the\ncocci@inria.fr mailing list.\n\nThis potentially changes the results of our *.cocci rules, but as\nnoted in that discussion it should be safe for our use. We don't name\nrules, or if we do their names don't conflict across our *.cocci\nfiles.\n\nTo the extent that we'd have any inter-dependencies between rules this\ndoesn't make that worse, as we'd have them now if we ran \"make\ncoccicheck\", applied the results, and would then have (due to\nhypothetical interdependencies) suggested changes on the subsequent\n\"make coccicheck\".\n\nOur \"coccicheck-test\" target makes use of the ALL.cocci when running\ntests, e.g. when testing unused.{c,out} we test it against ALL.cocci,\nnot unused.cocci. We thus assert (to the extent that we have test\ncoverage) that this concatenation doesn't change the expected results\nof running these rules.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 40 ++++++++++++++++++++++++++++++++---\n contrib/coccinelle/.gitignore | 1 +\n contrib/coccinelle/README | 13 ++++++++++++\n shared.mak | 1 +\n 4 files changed, 52 insertions(+), 3 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 531077031d8..0d65c09ad5a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1311,6 +1311,25 @@ SPATCH_TEST_FLAGS =\n # COMPUTE_HEADER_DEPENDENCIES=no this will be unset too.\n SPATCH_USE_O_DEPENDENCIES = YesPlease\n \n+# Set SPATCH_CONCAT_COCCI to concatenate the contrib/cocci/*.cocci\n+# files into a single contrib/cocci/ALL.cocci before running\n+# \"coccicheck\".\n+#\n+# Pros:\n+#\n+# - Speeds up a one-shot run of \"make coccicheck\", as we won't have to\n+# parse *.[ch] files N times for the N *.cocci rules\n+#\n+# Cons:\n+#\n+# - Will make incremental development of *.cocci slower, as\n+# e.g. changing strbuf.cocci will re-run all *.cocci.\n+#\n+# - Makes error and performance analysis harder, as rules will be\n+# applied from a monolithic ALL.cocci, rather than\n+# e.g. strbuf.cocci.\n+SPATCH_CONCAT_COCCI = YesPlease\n+\n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n TRACK_SPATCH_DEFINES += $(SPATCH)\n@@ -3159,9 +3178,10 @@ check: $(GENERATED_H)\n \t\texit 1; \\\n \tfi\n \n+COCCI_GEN_ALL = contrib/coccinelle/ALL.cocci\n COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci)\n-COCCI_RULES = $(COCCI_GLOB)\n-COCCI_NAMES = $(COCCI_RULES:contrib/coccinelle/%.cocci=%)\n+COCCI_NAMES = $(sort ALL $(COCCI_GLOB:contrib/coccinelle/%.cocci=%))\n+COCCI_RULES = $(filter-out $(COCCI_GEN_ALL),$(COCCI_GLOB))\n \n COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES))\n COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n@@ -3175,6 +3195,7 @@ COCCI_RULES_GLOB += cocci%\n COCCI_RULES_GLOB += .build/contrib/coccinelle/%\n COCCI_RULES_GLOB += $(COCCICHECK_PATCHES)\n COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING)\n+COCCI_RULES_GLOB += $(COCCI_GEN_ALL)\n COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS))\n \n COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n@@ -3186,6 +3207,10 @@ COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \t$(call mkdir_p_parent_template)\n \t$(QUIET_GEN) >$@\n \n+$(COCCI_GEN_ALL): $(COCCICHECK)\n+\t$(call mkdir_p_parent_template)\n+\t$(QUIET_SPATCH_CAT)cat $^ >$@\n+\n ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no)\n SPATCH_USE_O_DEPENDENCIES =\n endif\n@@ -3218,7 +3243,7 @@ $(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o)))\n endef\n \n ifdef COCCI_GOALS\n-$(eval $(foreach c,$(COCCI_RULES),$(call cocci-matrix,$(c))))\n+$(eval $(foreach c,$(COCCI_RULES) $(COCCI_GEN_ALL),$(call cocci-matrix,$(c))))\n endif\n \n define spatch-rule\n@@ -3239,7 +3264,11 @@ COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n+ifdef SPATCH_CONCAT_COCCI\n+$(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : $(COCCI_GEN_ALL)\n+else\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n+endif\n \t$(call mkdir_p_parent_template)\n \t$(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_TEST_FLAGS) \\\n \t\t--very-quiet --no-show-diff \\\n@@ -3252,7 +3281,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell\n coccicheck-test: $(COCCI_TEST_RES_GEN)\n \n coccicheck: coccicheck-test\n+ifdef SPATCH_CONCAT_COCCI\n+coccicheck: contrib/coccinelle/ALL.cocci.patch\n+else\n coccicheck: $(COCCICHECK_PATCHES)\n+endif\n \n # See contrib/coccinelle/README\n coccicheck-pending: coccicheck-test\n@@ -3527,6 +3560,7 @@ cocciclean:\n \t$(RM) -r .build/contrib/coccinelle\n \t$(RM) $(COCCICHECK_PATCHES)\n \t$(RM) $(COCCICHECK_PATCHES_PENDING)\n+\t$(RM) $(COCCI_GEN_ALL)\n \n clean: profile-clean coverage-clean cocciclean\n \t$(RM) -r .build\ndiff --git a/contrib/coccinelle/.gitignore b/contrib/coccinelle/.gitignore\nindex d3f29646dc3..2709d98eb91 100644\n--- a/contrib/coccinelle/.gitignore\n+++ b/contrib/coccinelle/.gitignore\n@@ -1 +1,2 @@\n+/ALL.cocci\n *.patch*\ndiff --git a/contrib/coccinelle/README b/contrib/coccinelle/README\nindex 09ea8298e19..09b72bfd4e7 100644\n--- a/contrib/coccinelle/README\n+++ b/contrib/coccinelle/README\n@@ -57,3 +57,16 @@ Git-specific tips & things to know about how we run \"spatch\":\n \n To disable this behavior use the \"SPATCH_USE_O_DEPENDENCIES=NoThanks\"\n flag.\n+\n+ * To speed up our rules the \"make coccicheck\" target will by default\n+ concatenate all of the *.cocci files here into an \"ALL.cocci\", and\n+ apply it to each source file.\n+\n+ This makes the run faster, as we don't need to run each rule\n+ against each source file. See the Makefile for further discussion,\n+ this behavior can be disabled with \"SPATCH_CONCAT_COCCI=\".\n+\n+ But since they're concatenated any in the (e.g. \"@\n+ my_name\", v.s. anonymous \"@@\") needs to be unique across all our\n+ *.cocci files. You should only need to name rules if other rules\n+ depend on them (currently only one rule is named).\ndiff --git a/shared.mak b/shared.mak\nindex 5ccd6889fcb..66dcf6768db 100644\n--- a/shared.mak\n+++ b/shared.mak\n@@ -73,6 +73,7 @@ ifndef V\n ## Used in \"Makefile\": SPATCH\n \tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $@;\n \tQUIET_SPATCH_TEST\t\t= @echo ' ' SPATCH TEST $(@:.build/%=%);\n+\tQUIET_SPATCH_CAT\t\t= @echo ' ' SPATCH CAT $$^ \\>$@;\n \n ## Used in \"Makefile\": SPATCH_*TMPL (quoted for use in \"define\"'s)\n \tQUIET_SPATCH_CAT_TMPL\t\t= @echo ' ' SPATCH CAT $$$$^ \\>$$@;\n-- \n2.38.0.1251.g3eefdfb5e7a\n\n"},{"id":"465777","messageId":"patch-v4-12.12-a728aa301a6-20221026T141005Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v4-00.12-00000000000-20221026T141005Z-avarab@gmail.com","subject":"[PATCH v4 12/12] spatchcache: add a ccache-alike for \"spatch\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-26T14:20:39Z","receivedAt":"2022-10-26T14:22:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a rather trivial \"spatchcache\", with this running e.g.:\n\n\tmake cocciclean\n\tmake contrib/coccinelle/free.cocci.patch \\\n\t\tSPATCH=contrib/coccicheck/spatchcache \\\n\t\tSPATCH_FLAGS=--very-quiet\n\nIs cut down from ~20s to ~5s on my system. Much of that is either\nfixable shell overhead, or the around 40 files we \"CANTCACHE\" (see the\nimplementation).\n\nThis uses \"redis\" as a cache by default, but it's configurable. See\nthe embedded documentation.\n\nThis is *not* like ccache in that we won't cache failed spatch\ninvocations, or those where spatch suggests changes for us. Those\ncases are so rare that I didn't think it was worth the bother, by far\nthe most common case is that it has no suggested changes. We'll also\nrefuse to cache any \"spatch\" invocation that has output on stderr,\nwhich means that \"--very-quiet\" must be added to \"SPATCH_FLAGS\".\n\nBecause we narrow the cache to that we don't need to save away stdout,\nstderr & the exit code. We simply cache the cases where we had no\nsuggested changes.\n\nAnother benchmark is to compare this with the previous\nSPATCH_BATCH_SIZE=N, as noted in [1]. Before this (on my 8 core system) running:\n\n\tmake clean; time make contrib/coccinelle/array.cocci.patch SPATCH_BATCH_SIZE=0\n\nWould take 33s, but with the preceding changes running without this\n\"spatchcache\" is slightly slower, or around 35s:\n\n\tmake clean; time make contrib/coccinelle/array.cocci.patch\n\nNow doing the same with SPATCH=contrib/coccinelle/spatchcache will\ntake around 6s, but we'll need to compile the *.o files first to take\nfull advantage of it (which can be fast with \"ccache\"):\n\n\tmake clean; make; time make contrib/coccinelle/array.cocci.patch SPATCH=contrib/coccinelle/spatchcache\n\n1. https://lore.kernel.org/git/YwdRqP1CyUAzCEn2@coredump.intra.peff.net/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n contrib/coccinelle/README | 20 +++\n contrib/coccinelle/spatchcache | 304 +++++++++++++++++++++++++++++++++\n 2 files changed, 324 insertions(+)\n create mode 100755 contrib/coccinelle/spatchcache\n\ndiff --git a/contrib/coccinelle/README b/contrib/coccinelle/README\nindex 09b72bfd4e7..d1daa1f6263 100644\n--- a/contrib/coccinelle/README\n+++ b/contrib/coccinelle/README\n@@ -70,3 +70,23 @@ Git-specific tips & things to know about how we run \"spatch\":\n my_name\", v.s. anonymous \"@@\") needs to be unique across all our\n *.cocci files. You should only need to name rules if other rules\n depend on them (currently only one rule is named).\n+\n+ * To speed up incremental runs even more use the \"spatchcache\" tool\n+ in this directory as your \"SPATCH\". It aimns to be a \"ccache\" for\n+ coccinelle, and piggy-backs on \"COMPUTE_HEADER_DEPENDENCIES\".\n+\n+ It caches in Redis by default, see it source for a how-to.\n+\n+ In one setup with a primed cache \"make coccicheck\" followed by a\n+ \"make clean && make\" takes around 10s to run, but 2m30s with the\n+ default of \"SPATCH_CONCAT_COCCI=Y\".\n+\n+ With \"SPATCH_CONCAT_COCCI=\" the total runtime is around ~6m, sped\n+ up to ~1m with \"spatchcache\".\n+\n+ Most of the 10s (or ~1m) being spent on re-running \"spatch\" on\n+ files we couldn't cache, as we didn't compile them (in contrib/*\n+ and compat/* mostly).\n+\n+ The absolute times will differ for you, but the relative speedup\n+ from caching should be on that order.\ndiff --git a/contrib/coccinelle/spatchcache b/contrib/coccinelle/spatchcache\nnew file mode 100755\nindex 00000000000..29e9352d8a2\n--- /dev/null\n+++ b/contrib/coccinelle/spatchcache\n@@ -0,0 +1,304 @@\n+#!/bin/sh\n+#\n+# spatchcache: a poor-man's \"ccache\"-alike for \"spatch\" in git.git\n+#\n+# This caching command relies on the peculiarities of the Makefile\n+# driving \"spatch\" in git.git, in particular if we invoke:\n+#\n+#\tmake\n+#\t# See \"spatchCache.cacheWhenStderr\" for why \"--very-quiet\" is\n+#\t# used\n+#\tmake coccicheck SPATCH_FLAGS=--very-quiet\n+#\n+# We can with COMPUTE_HEADER_DEPENDENCIES (auto-detected as true with\n+# \"gcc\" and \"clang\") write e.g. a .depend/grep.o.d for grep.c, when we\n+# compile grep.o.\n+#\n+# The .depend/grep.o.d will have the full header dependency tree of\n+# grep.c, and we can thus cache the output of \"spatch\" by:\n+#\n+#\t1. Hashing all of those files\n+#\t2. Hashing our source file, and the *.cocci rule we're\n+#\t applying\n+#\t3. Running spatch, if suggests no changes (by far the common\n+#\t case) we invoke \"spatchCache.getCmd\" and\n+#\t \"spatchCache.setCmd\" with a hash SHA-256 to ask \"does this\n+#\t ID have no changes\" or \"say that ID had no changes>\n+#\t4. If no \"spatchCache.{set,get}Cmd\" is specified we'll use\n+#\t \"redis-cli\" and maintain a SET called \"spatch-cache\". Set\n+#\t appropriate redis memory policies to keep it from growing\n+#\t out of control.\n+#\n+# This along with the general incremental \"make\" support for\n+# \"contrib/coccinelle\" makes it viable to (re-)run coccicheck\n+# e.g. when merging integration branches.\n+#\n+# Note that the \"--very-quiet\" flag is currently critical. The cache\n+# will refuse to cache anything that has output on STDERR (which might\n+# be errors from spatch), but see spatchCache.cacheWhenStderr below.\n+#\n+# The STDERR (and exit code) could in principle be cached (as with\n+# ccache), but then the simple structure in the Redis cache would need\n+# to change, so just supply \"--very-quiet\" for now.\n+#\n+# To use this, simply set SPATCH to\n+# contrib/coccinelle/spatchcache. Then optionally set:\n+#\n+#\t[spatchCache]\n+#\t\t# Optional: path to a custom spatch\n+#\t\tspatch = ~/g/coccicheck/spatch.opt\n+#\n+# As well as this trace config (debug implies trace):\n+#\n+#\t\tcacheWhenStderr = true\n+#\t\ttrace = false\n+#\t\tdebug = false\n+#\n+# The \".depend/grep.o.d\" can also be customized, as a string that will\n+# be eval'd, it has access to a \"$dirname\" and \"$basename\":\n+#\n+#\t[spatchCache]\n+#\t\tdependFormat = \"$dirname/.depend/${basename%.c}.o.d\"\n+#\n+# Setting \"trace\" to \"true\" allows for seeing when we have a cache HIT\n+# or MISS. To debug whether the cache is working do that, and run e.g.:\n+#\n+#\tredis-cli FLUSHALL\n+#\t\n+#\tgrep -hore HIT -e MISS -e SET -e NOCACHE -e CANTCACHE .build/contrib/coccinelle | sort | uniq -c\n+#\t 600 CANTCACHE\n+#\t 7365 MISS\n+#\t 7365 SET\n+#\n+# A subsequent \"make cocciclean && make coccicheck\" should then have\n+# all \"HIT\"'s and \"CANTCACHE\"'s.\n+#\n+# The \"spatchCache.cacheWhenStderr\" option is critical when using\n+# spatchCache.{trace,debug} to debug whether something is set in the\n+# cache, as we'll write to the spatch logs in .build/* we'd otherwise\n+# always emit a NOCACHE.\n+#\n+# Reading the config can make the command much slower, to work around\n+# this the config can be set in the environment, with environment\n+# variable name corresponding to the config key. \"default\" can be used\n+# to use whatever's the script default, e.g. setting\n+# spatchCache.cacheWhenStderr=true and deferring to the defaults for\n+# the rest is:\n+#\n+#\texport GIT_CONTRIB_SPATCHCACHE_DEBUG=default\n+#\texport GIT_CONTRIB_SPATCHCACHE_TRACE=default\n+#\texport GIT_CONTRIB_SPATCHCACHE_CACHEWHENSTDERR=true\n+#\texport GIT_CONTRIB_SPATCHCACHE_SPATCH=default\n+#\texport GIT_CONTRIB_SPATCHCACHE_DEPENDFORMAT=default\n+#\texport GIT_CONTRIB_SPATCHCACHE_SETCMD=default\n+#\texport GIT_CONTRIB_SPATCHCACHE_GETCMD=default\n+\n+set -e\n+\n+env_or_config () {\n+\tenv=\"$1\"\n+\tshift\n+\tif test \"$env\" = \"default\"\n+\tthen\n+\t\t# Avoid expensive \"git config\" invocation\n+\t\treturn\n+\telif test -n \"$env\"\n+\tthen\n+\t\techo \"$env\"\n+\telse\n+\t\tgit config $@ || :\n+\tfi\n+}\n+\n+## Our own configuration & options\n+debug=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_DEBUG\" --bool \"spatchCache.debug\")\n+if test \"$debug\" != \"true\"\n+then\n+\tdebug=\n+fi\n+if test -n \"$debug\"\n+then\n+\tset -x\n+fi\n+\n+trace=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_TRACE\" --bool \"spatchCache.trace\")\n+if test \"$trace\" != \"true\"\n+then\n+\ttrace=\n+fi\n+if test -n \"$debug\"\n+then\n+\t# debug implies trace\n+\ttrace=true\n+fi\n+\n+cacheWhenStderr=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_CACHEWHENSTDERR\" --bool \"spatchCache.cacheWhenStderr\")\n+if test \"$cacheWhenStderr\" != \"true\"\n+then\n+\tcacheWhenStderr=\n+fi\n+\n+trace_it () {\n+\tif test -z \"$trace\"\n+\tthen\n+\t\treturn\n+\tfi\n+\techo \"$@\" >&2\n+}\n+\n+spatch=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_SPATCH\" --path \"spatchCache.spatch\")\n+if test -n \"$spatch\"\n+then\n+\tif test -n \"$debug\"\n+\tthen\n+\t\ttrace_it \"custom spatchCache.spatch='$spatch'\"\n+\tfi\n+else\n+\tspatch=spatch\n+fi\n+\n+dependFormat='$dirname/.depend/${basename%.c}.o.d'\n+dependFormatCfg=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_DEPENDFORMAT\" \"spatchCache.dependFormat\")\n+if test -n \"$dependFormatCfg\"\n+then\n+\tdependFormat=\"$dependFormatCfg\"\n+fi\n+\n+set=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_SETCMD\" \"spatchCache.setCmd\")\n+get=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_GETCMD\" \"spatchCache.getCmd\")\n+\n+## Parse spatch()-like command-line for caching info\n+arg_sp=\n+arg_file=\n+args=\"$@\"\n+spatch_opts() {\n+\twhile test $# != 0\n+\tdo\n+\t\targ_file=\"$1\"\n+\t\tcase \"$1\" in\n+\t\t--sp-file)\n+\t\t\targ_sp=\"$2\"\n+\t\t\t;;\n+\t\tesac\n+\t\tshift\n+\tdone\n+}\n+spatch_opts \"$@\"\n+if ! test -f \"$arg_file\"\n+then\n+\targ_file=\n+fi\n+\n+hash_for_cache() {\n+\t# Parameters that should affect the cache\n+\techo \"args=$args\"\n+\techo \"config spatchCache.spatch=$spatch\"\n+\techo \"config spatchCache.debug=$debug\"\n+\techo \"config spatchCache.trace=$trace\"\n+\techo \"config spatchCache.cacheWhenStderr=$cacheWhenStderr\"\n+\techo\n+\n+\t# Our target file and its dependencies\n+\tgit hash-object \"$1\" \"$2\" $(grep -E -o '^[^:]+:$' \"$3\" | tr -d ':')\n+}\n+\n+# Sanity checks\n+if ! test -f \"$arg_sp\" && ! test -f \"$arg_file\"\n+then\n+\techo $0: no idea how to cache \"$@\" >&2\n+\texit 128\n+fi\n+\n+# Main logic\n+dirname=$(dirname \"$arg_file\")\n+basename=$(basename \"$arg_file\")\n+eval \"dep=$dependFormat\"\n+\n+if ! test -f \"$dep\"\n+then\n+\ttrace_it \"$0: CANTCACHE have no '$dep' for '$arg_file'!\"\n+\texec \"$spatch\" \"$@\"\n+fi\n+\n+if test -n \"$debug\"\n+then\n+\ttrace_it \"$0: The full cache input for '$arg_sp' '$arg_file' '$dep'\"\n+\thash_for_cache \"$arg_sp\" \"$arg_file\" \"$dep\" >&2\n+fi\n+sum=$(hash_for_cache \"$arg_sp\" \"$arg_file\" \"$dep\" | git hash-object --stdin)\n+\n+trace_it \"$0: processing '$arg_file' with '$arg_sp' rule, and got hash '$sum' for it + '$dep'\"\n+\n+getret=\n+if test -z \"$get\"\n+then\n+\tif test $(redis-cli SISMEMBER spatch-cache \"$sum\") = 1\n+\tthen\n+\t\tgetret=0\n+\telse\n+\t\tgetret=1\n+\tfi\n+else\n+\t$set \"$sum\"\n+\tgetret=$?\n+fi\n+\n+if test \"$getret\" = 0\n+then\n+\ttrace_it \"$0: HIT for '$arg_file' with '$arg_sp'\"\n+\texit 0\n+else\n+\ttrace_it \"$0: MISS: for '$arg_file' with '$arg_sp'\"\n+fi\n+\n+out=\"$(mktemp)\"\n+err=\"$(mktemp)\"\n+\n+set +e\n+\"$spatch\" \"$@\" >\"$out\" 2>>\"$err\"\n+ret=$?\n+cat \"$out\"\n+cat \"$err\" >&2\n+set -e\n+\n+nocache=\n+if test $ret != 0\n+then\n+\tnocache=\"exited non-zero: $ret\"\n+elif test -s \"$out\"\n+then\n+\tnocache=\"had patch output\"\n+elif test -z \"$cacheWhenStderr\" && test -s \"$err\"\n+then\n+\tnocache=\"had stderr (use --very-quiet or spatchCache.cacheWhenStderr=true?)\"\n+fi\n+\n+if test -n \"$nocache\"\n+then\n+\ttrace_it \"$0: NOCACHE ($nocache): for '$arg_file' with '$arg_sp'\"\n+\texit \"$ret\"\n+fi\n+\n+trace_it \"$0: SET: for '$arg_file' with '$arg_sp'\"\n+\n+setret=\n+if test -z \"$set\"\n+then\n+\tif test $(redis-cli SADD spatch-cache \"$sum\") = 1\n+\tthen\n+\t\tsetret=0\n+\telse\n+\t\tsetret=1\n+\tfi\n+else\n+\t\"$set\" \"$sum\"\n+\tsetret=$?\n+fi\n+\n+if test \"$setret\" != 0\n+then\n+\techo \"FAILED to set '$sum' in cache!\" >&2\n+\texit 128\n+fi\n+\n+exit \"$ret\"\n-- \n2.38.0.1251.g3eefdfb5e7a\n\n"},{"id":"465940","messageId":"20221028125827.GA1682@szeder.dev","threadId":"58360","inReplyTo":"patch-v4-11.12-a848d09527f-20221026T141005Z-avarab@gmail.com","subject":"Re: [PATCH v4 11/12] cocci: run against a generated ALL.cocci","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2022-10-28T12:58:27Z","receivedAt":"2022-10-28T12:58:35Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Oct 26, 2022 at 04:20:38PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> The preceding commits to make the \"coccicheck\" target incremental made\n> it slower in some cases. As an optimization let's not have the\n> many=many mapping of <*.cocci>=<*.[ch]>, but instead concat the\n> <*.cocci> into an ALL.cocci, and then run one-to-many\n> ALL.cocci=<*.[ch]>.\n\n\n> @@ -3252,7 +3281,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell\n> coccicheck-test: $(COCCI_TEST_RES_GEN)\n> \n> coccicheck: coccicheck-test\n> +ifdef SPATCH_CONCAT_COCCI\n> +coccicheck: contrib/coccinelle/ALL.cocci.patch\n> +else\n> coccicheck: $(COCCICHECK_PATCHES)\n> +endif\n> \n> # See contrib/coccinelle/README\n> coccicheck-pending: coccicheck-test\n> @@ -3527,6 +3560,7 @@ cocciclean:\n> \t$(RM) -r .build/contrib/coccinelle\n> \t$(RM) $(COCCICHECK_PATCHES)\n> \t$(RM) $(COCCICHECK_PATCHES_PENDING)\n> +\t$(RM) $(COCCI_GEN_ALL)\n\nThis should also remove 'contrib/coccinelle/ALL.cocci.patch'.\n\n"},{"id":"466226","messageId":"cover-v5-00.13-00000000000-20221101T222616Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v4-00.12-00000000000-20221026T141005Z-avarab@gmail.com","subject":"[PATCH v5 00/13] cocci: make \"incremental\" possible + a ccache-like tool","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T22:35:42Z","receivedAt":"2022-11-01T22:36:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A re-roll of the series to have \"make coccicheck\" run\nincrementally. For the v1 and general overview see:\nhttps://lore.kernel.org/git/cover-0.5-00000000000-20220825T141212Z-avarab@gmail.com/\n\nFor a performance overview see the v4 CL:\nhttp://lore.kernel.org/git/cover-v4-00.12-00000000000-20221026T141005Z-avarab@gmail.com\n\nUpdates since v4:\n\n * SZEDER pointed out I needed to remove the ALL.cocci. I did that by\n having that ALL.cocci generated in .build/contrib/coccinelle,\n instead among the user-maintained *.cocci files in\n contrib/coccinelle.\n\n I think it leaves things in a beter place, but there's some churn\n from 8/13 onwards in service of that.\n * To make that easier the *.cocci files are copied to .build before\n being executed, which makes making ALL.cocci and the\n \"non-generated\" *.cocci easier.\n * Fixed typo/breakage in \"make coccicheck-pending\" Updated .gitignore\n * rule along with 8/13 (no more *.patch+ files.FixeFixeFixeFixe\n * A few other misc changes (e.g. commit mesages, docs), see\n range-diff below.\n\nPassing CI for this:\nhttps://github.com/avar/git/actions/runs/3363433825/jobs/5586038736;\nIt finished static-analysis in ~7m now, instead of ~17m (master result\nat:\nhttps://github.com/git/git/actions/runs/3358829617/jobs/5566255602)\n\nÆvar Arnfjörð Bjarmason (13):\n Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)\n cocci rules: remove unused \"F\" metavariable from pending rule\n Makefile: add ability to TAB-complete cocci *.patch rules\n Makefile: have \"coccicheck\" re-run if flags change\n Makefile: split off SPATCH_BATCH_SIZE comment from \"cocci\" heading\n cocci: split off include-less \"tests\" from SPATCH_FLAGS\n cocci: split off \"--all-includes\" from SPATCH_FLAGS\n cocci: make \"coccicheck\" rule incremental\n cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES\n Makefile: copy contrib/coccinelle/*.cocci to build/\n cocci rules: remove 's from rules that don't need them\n cocci: run against a generated ALL.cocci\n spatchcache: add a ccache-alike for \"spatch\"\n\n .gitignore | 1 +\n Makefile | 175 ++++++++--\n contrib/coccinelle/.gitignore | 2 +-\n contrib/coccinelle/README | 49 +++\n contrib/coccinelle/hashmap.cocci | 2 +-\n contrib/coccinelle/preincr.cocci | 2 +-\n contrib/coccinelle/spatchcache | 304 ++++++++++++++++++\n contrib/coccinelle/strbuf.cocci | 2 +-\n contrib/coccinelle/swap.cocci | 2 +-\n .../coccinelle/the_repository.pending.cocci | 1 -\n shared.mak | 8 +-\n 11 files changed, 515 insertions(+), 33 deletions(-)\n create mode 100755 contrib/coccinelle/spatchcache\n\nRange-diff against v4:\n 1: c0306e3c397 = 1: 1c77706db26 Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)\n 2: 14e757cc020 = 2: d8556f94cb4 cocci rules: remove unused \"F\" metavariable from pending rule\n 3: 210dbc6b101 = 3: 6bf908044eb Makefile: add ability to TAB-complete cocci *.patch rules\n 4: b859dae7509 = 4: 460025dbbf6 Makefile: have \"coccicheck\" re-run if flags change\n 5: 11b9227722b = 5: b00ceccf1d1 Makefile: split off SPATCH_BATCH_SIZE comment from \"cocci\" heading\n 6: 1c30f4b543b = 6: 04d268e246b cocci: split off include-less \"tests\" from SPATCH_FLAGS\n 7: 86827388627 = 7: cae46461ebf cocci: split off \"--all-includes\" from SPATCH_FLAGS\n 8: 340ce523436 ! 8: ee9ad433ab5 cocci: make \"coccicheck\" rule incremental\n @@ Makefile: check: $(GENERATED_H)\n +COCCI_NAMES = $(COCCI_RULES:contrib/coccinelle/%.cocci=%)\n \n COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES))\n -+COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n -+COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n -+\n +@@ Makefile: COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n + COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n + COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n + \n +# It's expensive to compute the many=many rules below, only eval them\n +# on $(MAKECMDGOALS) that match these $(COCCI_RULES)\n +COCCI_RULES_GLOB =\n @@ Makefile: check: $(GENERATED_H)\n +COCCI_RULES_GLOB += $(COCCICHECK_PATCHES)\n +COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING)\n +COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS))\n - \n - COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n - COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n - \n ++\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n -COCCI_PATCHES = $(COCCI_RULES:%=%.patch)\n @@ Makefile: check: $(GENERATED_H)\n +define spatch-rule\n +\n +contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1))\n -+\t$(QUIET_SPATCH_CAT_TMPL)cat $$^ >$$@ && \\\n ++\t$$(QUIET_SPATCH_CAT)cat $$^ >$$@ && \\\n +\tif test -s $$@; \\\n \tthen \\\n -\t\techo ' ' SPATCH result: $@; \\\n @@ Makefile: profile-clean:\n \t$(RM) GIT-SPATCH-DEFINES\n \t$(RM) -r .build/contrib/coccinelle\n -\t$(RM) contrib/coccinelle/*.cocci.patch*\n -+\t$(RM) $(COCCICHECK_PATCHES)\n -+\t$(RM) $(COCCICHECK_PATCHES_PENDING)\n ++\t$(RM) contrib/coccinelle/*.cocci.patch\n \n clean: profile-clean coverage-clean cocciclean\n \t$(RM) -r .build\n \n + ## contrib/coccinelle/.gitignore ##\n +@@\n +-*.patch*\n ++*.patch\n +\n ## shared.mak ##\n @@ shared.mak: ifndef V\n \tQUIET_RC = @echo ' ' RC $@;\n @@ shared.mak: ifndef V\n -\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $<;\n +\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $@;\n \tQUIET_SPATCH_TEST\t\t= @echo ' ' SPATCH TEST $(@:.build/%=%);\n ++\tQUIET_SPATCH_CAT\t\t= @echo ' ' SPATCH CAT $$^ \\>$@;\n \n -+## Used in \"Makefile\": SPATCH_*TMPL (quoted for use in \"define\"'s)\n -+\tQUIET_SPATCH_CAT_TMPL\t\t= @echo ' ' SPATCH CAT $$$$^ \\>$$@;\n -+\n ## Used in \"Documentation/Makefile\"\n \tQUIET_ASCIIDOC\t= @echo ' ' ASCIIDOC $@;\n - \tQUIET_XMLTO\t= @echo ' ' XMLTO $@;\n 9: 134b3743a60 = 9: 332fb6c8640 cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES\n -: ----------- > 10: 56ca8f5720a Makefile: copy contrib/coccinelle/*.cocci to build/\n10: 89f778ac055 = 11: 73bb897c965 cocci rules: remove 's from rules that don't need them\n11: a848d09527f ! 12: 55d2f638359 cocci: run against a generated ALL.cocci\n @@ Makefile: SPATCH_TEST_FLAGS =\n +#\n +# - Makes error and performance analysis harder, as rules will be\n +# applied from a monolithic ALL.cocci, rather than\n -+# e.g. strbuf.cocci.\n ++# e.g. strbuf.cocci. To work around this either undefine this, or\n ++# generate a specific patch, e.g. this will always use strbuf.cocci,\n ++# not ALL.cocci:\n ++#\n ++#\tmake contrib/coccinelle/strbuf.cocci.patch\n +SPATCH_CONCAT_COCCI = YesPlease\n +\n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n @@ Makefile: check: $(GENERATED_H)\n \t\texit 1; \\\n \tfi\n \n -+COCCI_GEN_ALL = contrib/coccinelle/ALL.cocci\n ++COCCI_GEN_ALL = .build/contrib/coccinelle/ALL.cocci\n COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci)\n --COCCI_RULES = $(COCCI_GLOB)\n --COCCI_NAMES = $(COCCI_RULES:contrib/coccinelle/%.cocci=%)\n -+COCCI_NAMES = $(sort ALL $(COCCI_GLOB:contrib/coccinelle/%.cocci=%))\n -+COCCI_RULES = $(filter-out $(COCCI_GEN_ALL),$(COCCI_GLOB))\n - \n - COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES))\n - COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n -@@ Makefile: COCCI_RULES_GLOB += cocci%\n - COCCI_RULES_GLOB += .build/contrib/coccinelle/%\n - COCCI_RULES_GLOB += $(COCCICHECK_PATCHES)\n - COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING)\n -+COCCI_RULES_GLOB += $(COCCI_GEN_ALL)\n - COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS))\n - \n - COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n -@@ Makefile: COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n + COCCI_RULES_TRACKED = $(COCCI_GLOB:%=.build/%)\n ++COCCI_RULES_TRACKED_NO_PENDING = $(filter-out %.pending.cocci,$(COCCI_RULES_TRACKED))\n + COCCI_RULES =\n ++COCCI_RULES += $(COCCI_GEN_ALL)\n + COCCI_RULES += $(COCCI_RULES_TRACKED)\n + COCCI_NAMES =\n + COCCI_NAMES += $(COCCI_RULES:.build/contrib/coccinelle/%.cocci=%)\n +@@ Makefile: $(COCCI_RULES_TRACKED): .build/% : %\n \t$(call mkdir_p_parent_template)\n \t$(QUIET_GEN) >$@\n \n -+$(COCCI_GEN_ALL): $(COCCICHECK)\n ++$(COCCI_GEN_ALL): $(COCCI_RULES_TRACKED_NO_PENDING)\n +\t$(call mkdir_p_parent_template)\n +\t$(QUIET_SPATCH_CAT)cat $^ >$@\n +\n ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no)\n SPATCH_USE_O_DEPENDENCIES =\n endif\n -@@ Makefile: $(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o)))\n - endef\n - \n - ifdef COCCI_GOALS\n --$(eval $(foreach c,$(COCCI_RULES),$(call cocci-matrix,$(c))))\n -+$(eval $(foreach c,$(COCCI_RULES) $(COCCI_GEN_ALL),$(call cocci-matrix,$(c))))\n - endif\n - \n - define spatch-rule\n @@ Makefile: COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n @@ Makefile: $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib\n +ifdef SPATCH_CONCAT_COCCI\n +coccicheck: contrib/coccinelle/ALL.cocci.patch\n +else\n - coccicheck: $(COCCICHECK_PATCHES)\n + coccicheck: $(COCCICHECK_PATCHES_INTREE)\n +endif\n \n # See contrib/coccinelle/README\n coccicheck-pending: coccicheck-test\n -@@ Makefile: cocciclean:\n - \t$(RM) -r .build/contrib/coccinelle\n - \t$(RM) $(COCCICHECK_PATCHES)\n - \t$(RM) $(COCCICHECK_PATCHES_PENDING)\n -+\t$(RM) $(COCCI_GEN_ALL)\n - \n - clean: profile-clean coverage-clean cocciclean\n - \t$(RM) -r .build\n -\n - ## contrib/coccinelle/.gitignore ##\n -@@\n -+/ALL.cocci\n - *.patch*\n \n ## contrib/coccinelle/README ##\n @@ contrib/coccinelle/README: Git-specific tips & things to know about how we run \"spatch\":\n @@ contrib/coccinelle/README: Git-specific tips & things to know about how we run \"\n + my_name\", v.s. anonymous \"@@\") needs to be unique across all our\n + *.cocci files. You should only need to name rules if other rules\n + depend on them (currently only one rule is named).\n -\n - ## shared.mak ##\n -@@ shared.mak: ifndef V\n - ## Used in \"Makefile\": SPATCH\n - \tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $@;\n - \tQUIET_SPATCH_TEST\t\t= @echo ' ' SPATCH TEST $(@:.build/%=%);\n -+\tQUIET_SPATCH_CAT\t\t= @echo ' ' SPATCH CAT $$^ \\>$@;\n - \n - ## Used in \"Makefile\": SPATCH_*TMPL (quoted for use in \"define\"'s)\n - \tQUIET_SPATCH_CAT_TMPL\t\t= @echo ' ' SPATCH CAT $$$$^ \\>$$@;\n12: a728aa301a6 = 13: ef578c230ea spatchcache: add a ccache-alike for \"spatch\"\n-- \n2.38.0.1280.g8136eb6fab2\n\n"},{"id":"466227","messageId":"patch-v5-01.13-1c77706db26-20221101T222616Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v5-00.13-00000000000-20221101T222616Z-avarab@gmail.com","subject":"[PATCH v5 01/13] Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T22:35:43Z","receivedAt":"2022-11-01T22:36:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"In f7ff6597a75 (cocci: add a \"coccicheck-test\" target and test *.cocci\nrules, 2022-07-05) we abbreviated \"_TEST\" to \"_T\" to have it align\nwith the rest of the \"=\"'s above it.\n\nSubsequent commits will add more QUIET_SPATCH_* variables, so let's\nstop abbreviating this, and indent it in preparation for adding more\nof these variables.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 2 +-\n shared.mak | 6 ++++--\n 2 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 4927379184c..b07ee35e17d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3172,7 +3172,7 @@ $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n \t$(call mkdir_p_parent_template)\n-\t$(QUIET_SPATCH_T)$(SPATCH) $(SPATCH_FLAGS) \\\n+\t$(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_FLAGS) \\\n \t\t--very-quiet --no-show-diff \\\n \t\t--sp-file $< -o $@ \\\n \t\t$(@:.build/%.res=%.c) && \\\ndiff --git a/shared.mak b/shared.mak\nindex 33f43edbf9a..96b06acc455 100644\n--- a/shared.mak\n+++ b/shared.mak\n@@ -69,8 +69,10 @@ ifndef V\n \tQUIET_SP = @echo ' ' SP $<;\n \tQUIET_HDR = @echo ' ' HDR $(<:hcc=h);\n \tQUIET_RC = @echo ' ' RC $@;\n-\tQUIET_SPATCH = @echo ' ' SPATCH $<;\n-\tQUIET_SPATCH_T = @echo ' ' SPATCH TEST $(@:.build/%=%);\n+\n+## Used in \"Makefile\": SPATCH\n+\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $<;\n+\tQUIET_SPATCH_TEST\t\t= @echo ' ' SPATCH TEST $(@:.build/%=%);\n \n ## Used in \"Documentation/Makefile\"\n \tQUIET_ASCIIDOC\t= @echo ' ' ASCIIDOC $@;\n-- \n2.38.0.1280.g8136eb6fab2\n\n"},{"id":"466228","messageId":"patch-v5-02.13-d8556f94cb4-20221101T222616Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v5-00.13-00000000000-20221101T222616Z-avarab@gmail.com","subject":"[PATCH v5 02/13] cocci rules: remove unused \"F\" metavariable from pending rule","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T22:35:44Z","receivedAt":"2022-11-01T22:36:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix an issue with a rule added in 9b45f499818 (object-store: prepare\nhas_{sha1, object}_file to handle any repo, 2018-11-13). We've been\nspewing out this warning into our $@.log since that rule was added:\n\n\twarning: rule starting on line 21: metavariable F not used in the - or context code\n\nWe should do a better job of scouring our coccinelle log files for\nsuch issues, but for now let's fix this as a one-off.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n contrib/coccinelle/the_repository.pending.cocci | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/contrib/coccinelle/the_repository.pending.cocci b/contrib/coccinelle/the_repository.pending.cocci\nindex 072ea0d9228..747d382ff5f 100644\n--- a/contrib/coccinelle/the_repository.pending.cocci\n+++ b/contrib/coccinelle/the_repository.pending.cocci\n@@ -20,7 +20,6 @@ expression E;\n \n @@\n expression E;\n-expression F;\n @@\n - has_object_file_with_flags(\n + repo_has_object_file_with_flags(the_repository,\n-- \n2.38.0.1280.g8136eb6fab2\n\n"},{"id":"466229","messageId":"patch-v5-03.13-6bf908044eb-20221101T222616Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v5-00.13-00000000000-20221101T222616Z-avarab@gmail.com","subject":"[PATCH v5 03/13] Makefile: add ability to TAB-complete cocci *.patch rules","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T22:35:45Z","receivedAt":"2022-11-01T22:37:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Declare the contrib/coccinelle/.cocci.patch rules in such a way\nas to allow TAB-completion, and slightly optimize the Makefile by\ncutting down on the number of $(wildcard) in favor of defining\n\"coccicheck\" and \"coccicheck-pending\" in terms of the same\nincrementally filtered list.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 17 ++++++++++++++---\n 1 file changed, 14 insertions(+), 3 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex b07ee35e17d..828d332096a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3144,9 +3144,20 @@ check: $(GENERATED_H)\n \t\texit 1; \\\n \tfi\n \n+COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci)\n+COCCI_RULES = $(COCCI_GLOB)\n+\n+COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES))\n+COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n+\n+COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n+COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n+\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n-%.cocci.patch: %.cocci $(COCCI_SOURCES)\n+COCCI_PATCHES = $(COCCI_RULES:%=%.patch)\n+$(COCCI_PATCHES): $(COCCI_SOURCES)\n+$(COCCI_PATCHES): %.patch: %\n \t$(QUIET_SPATCH) \\\n \tif test $(SPATCH_BATCH_SIZE) = 0; then \\\n \t\tlimit=; \\\n@@ -3183,11 +3194,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell\n coccicheck-test: $(COCCI_TEST_RES_GEN)\n \n coccicheck: coccicheck-test\n-coccicheck: $(addsuffix .patch,$(filter-out %.pending.cocci,$(wildcard contrib/coccinelle/*.cocci)))\n+coccicheck: $(COCCICHECK_PATCHES)\n \n # See contrib/coccinelle/README\n coccicheck-pending: coccicheck-test\n-coccicheck-pending: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.pending.cocci))\n+coccicheck-pending: $(COCCICHECK_PATCHES_PENDING)\n \n .PHONY: coccicheck coccicheck-pending\n \n-- \n2.38.0.1280.g8136eb6fab2\n\n"},{"id":"466230","messageId":"patch-v5-04.13-460025dbbf6-20221101T222616Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v5-00.13-00000000000-20221101T222616Z-avarab@gmail.com","subject":"[PATCH v5 04/13] Makefile: have \"coccicheck\" re-run if flags change","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T22:35:46Z","receivedAt":"2022-11-01T22:37:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix an issue with the \"coccicheck\" family of rules that's been here\nsince 63f0a758a06 (add coccicheck make target, 2016-09-15), unlike\ne.g. \"make grep.o\" we wouldn't re-run it when $(SPATCH) or\n$(SPATCH_FLAGS) changed. To test new flags we needed to first do a\n\"make cocciclean\".\n\nThis now uses the same (copy/pasted) pattern as other \"DEFINES\"\nrules. As a result we'll re-run properly. This can be demonstrated\ne.g. on the issue noted in [1]:\n\n\t$ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c V=1\n\t[...]\n\t SPATCH contrib/coccinelle/xcalloc.cocci\n\t$ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c SPATCH_FLAGS=\"--all-includes --recursive-includes\"\n\t * new spatch flags\n\t SPATCH contrib/coccinelle/xcalloc.cocci\n\t SPATCH result: contrib/coccinelle/xcalloc.cocci.patch\n\t$\n\n1. https://lore.kernel.org/git/20220823095602.GC1735@szeder.dev/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .gitignore | 1 +\n Makefile | 15 +++++++++++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/.gitignore b/.gitignore\nindex cb0231fb401..0832f1da77b 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -8,6 +8,7 @@\n /GIT-PERL-HEADER\n /GIT-PYTHON-VARS\n /GIT-SCRIPT-DEFINES\n+/GIT-SPATCH-DEFINES\n /GIT-USER-AGENT\n /GIT-VERSION-FILE\n /bin-wrappers/\ndiff --git a/Makefile b/Makefile\nindex 828d332096a..d81b0931ddd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1308,6 +1308,18 @@ SANITIZE_ADDRESS =\n SPATCH_FLAGS = --all-includes\n SPATCH_BATCH_SIZE = 1\n \n+# Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n+TRACK_SPATCH_DEFINES =\n+TRACK_SPATCH_DEFINES += $(SPATCH)\n+TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n+TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n+GIT-SPATCH-DEFINES: FORCE\n+\t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n+\t if test x\"$$FLAGS\" != x\"`cat GIT-SPATCH-DEFINES 2>/dev/null`\" ; then \\\n+\t\techo >&2 \" * new spatch flags\"; \\\n+\t\techo \"$$FLAGS\" >GIT-SPATCH-DEFINES; \\\n+ fi\n+\n include config.mak.uname\n -include config.mak.autogen\n -include config.mak\n@@ -3156,6 +3168,7 @@ COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n COCCI_PATCHES = $(COCCI_RULES:%=%.patch)\n+$(COCCI_PATCHES): GIT-SPATCH-DEFINES\n $(COCCI_PATCHES): $(COCCI_SOURCES)\n $(COCCI_PATCHES): %.patch: %\n \t$(QUIET_SPATCH) \\\n@@ -3179,6 +3192,7 @@ $(COCCI_PATCHES): %.patch: %\n \tfi\n \n COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n+$(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n@@ -3465,6 +3479,7 @@ profile-clean:\n \t$(RM) $(addsuffix *.gcno,$(addprefix $(PROFILE_DIR)/, $(object_dirs)))\n \n cocciclean:\n+\t$(RM) GIT-SPATCH-DEFINES\n \t$(RM) -r .build/contrib/coccinelle\n \t$(RM) contrib/coccinelle/*.cocci.patch*\n \n-- \n2.38.0.1280.g8136eb6fab2\n\n"},{"id":"466231","messageId":"patch-v5-06.13-04d268e246b-20221101T222616Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v5-00.13-00000000000-20221101T222616Z-avarab@gmail.com","subject":"[PATCH v5 06/13] cocci: split off include-less \"tests\" from SPATCH_FLAGS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T22:35:48Z","receivedAt":"2022-11-01T22:37:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Amend the \"coccicheck-test\" rule added in f7ff6597a75 (cocci: add a\n\"coccicheck-test\" target and test *.cocci rules, 2022-07-05) to stop\nusing \"--all-includes\". The flags we'll need for the tests are\ndifferent than the ones we'll need for our main source code.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex 8c4751e1bd6..6785e8175b5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1304,6 +1304,7 @@ SANITIZE_ADDRESS =\n \n # For the 'coccicheck' target\n SPATCH_FLAGS = --all-includes\n+SPATCH_TEST_FLAGS =\n # Setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n@@ -1313,6 +1314,7 @@ SPATCH_BATCH_SIZE = 1\n TRACK_SPATCH_DEFINES =\n TRACK_SPATCH_DEFINES += $(SPATCH)\n TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n+TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n GIT-SPATCH-DEFINES: FORCE\n \t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n@@ -3198,7 +3200,7 @@ $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n \t$(call mkdir_p_parent_template)\n-\t$(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_FLAGS) \\\n+\t$(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_TEST_FLAGS) \\\n \t\t--very-quiet --no-show-diff \\\n \t\t--sp-file $< -o $@ \\\n \t\t$(@:.build/%.res=%.c) && \\\n-- \n2.38.0.1280.g8136eb6fab2\n\n"},{"id":"466232","messageId":"patch-v5-05.13-b00ceccf1d1-20221101T222616Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v5-00.13-00000000000-20221101T222616Z-avarab@gmail.com","subject":"[PATCH v5 05/13] Makefile: split off SPATCH_BATCH_SIZE comment from \"cocci\" heading","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T22:35:47Z","receivedAt":"2022-11-01T22:37:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Split off the \"; setting[...]\" part of the comment added in In\n960154b9c17 (coccicheck: optionally batch spatch invocations,\n2019-05-06), and restore what we had before that, which was a comment\nindicating that variables for the \"coccicheck\" target were being set\nhere.\n\nWhen 960154b9c17 amended the heading to discuss SPATCH_BATCH_SIZE it\nleft no natural place to add a new comment about other flags that\npreceded it. As subsequent commits will add such comments we need to\nsplit the existing comment up.\n\nThe wrapping for the \"SPATCH_BATCH_SIZE\" is now a bit odd, but\nminimizes the diff size. As a subsequent commit will remove that\nfeature altogether this is worth it.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex d81b0931ddd..8c4751e1bd6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1302,10 +1302,11 @@ SP_EXTRA_FLAGS = -Wno-universal-initializer\n SANITIZE_LEAK =\n SANITIZE_ADDRESS =\n \n-# For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will\n+# For the 'coccicheck' target\n+SPATCH_FLAGS = --all-includes\n+# Setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n # Setting it to 0 will feed all files in a single spatch invocation.\n-SPATCH_FLAGS = --all-includes\n SPATCH_BATCH_SIZE = 1\n \n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n-- \n2.38.0.1280.g8136eb6fab2\n\n"},{"id":"466233","messageId":"patch-v5-07.13-cae46461ebf-20221101T222616Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v5-00.13-00000000000-20221101T222616Z-avarab@gmail.com","subject":"[PATCH v5 07/13] cocci: split off \"--all-includes\" from SPATCH_FLAGS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T22:35:49Z","receivedAt":"2022-11-01T22:37:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Per the rationale in 7b63ea57500 (Makefile: remove mandatory \"spatch\"\narguments from SPATCH_FLAGS, 2022-07-05) we have certain flags that\nare truly mandatory, such as \"--sp-file\" and \"--patch .\". The\n\"--all-includes\" flag is also critical, but per [1] we might want to\nad-hoc tweak it occasionally for testing or one-offs.\n\nBut being unable to set e.g. SPATCH_FLAGS=\"--verbose-parsing\" without\nbreaking how our \"spatch\" works isn't ideal, i.e. before this we'd\nneed to know about the default include flags, and specify:\nSPATCH_FLAGS=\"--all-includes --verbose-parsing\".\n\nIf we were then to change the default include flag (e.g. to\n\"--recursive-includes\") in the future any such one-off commands would\nneed to be correspondingly updated.\n\nLet's instead leave the SPATCH_FLAGS for the user, while creating a\nnew SPATCH_INCLUDE_FLAGS to allow for ad-hoc testing of the include\nstrategy itself.\n\n1. https://lore.kernel.org/git/20220823095733.58685-1-szeder.dev@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex 6785e8175b5..25f0c7e837d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1303,7 +1303,8 @@ SANITIZE_LEAK =\n SANITIZE_ADDRESS =\n \n # For the 'coccicheck' target\n-SPATCH_FLAGS = --all-includes\n+SPATCH_INCLUDE_FLAGS = --all-includes\n+SPATCH_FLAGS =\n SPATCH_TEST_FLAGS =\n # Setting SPATCH_BATCH_SIZE higher will\n # usually result in less CPU usage at the cost of higher peak memory.\n@@ -1313,6 +1314,7 @@ SPATCH_BATCH_SIZE = 1\n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n TRACK_SPATCH_DEFINES += $(SPATCH)\n+TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n@@ -3182,6 +3184,7 @@ $(COCCI_PATCHES): %.patch: %\n \tfi; \\\n \tif ! echo $(COCCI_SOURCES) | xargs $$limit \\\n \t\t$(SPATCH) $(SPATCH_FLAGS) \\\n+\t\t$(SPATCH_INCLUDE_FLAGS) \\\n \t\t--sp-file $< --patch . \\\n \t\t>$@+ 2>$@.log; \\\n \tthen \\\n-- \n2.38.0.1280.g8136eb6fab2\n\n"},{"id":"466234","messageId":"patch-v5-08.13-ee9ad433ab5-20221101T222616Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v5-00.13-00000000000-20221101T222616Z-avarab@gmail.com","subject":"[PATCH v5 08/13] cocci: make \"coccicheck\" rule incremental","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T22:35:50Z","receivedAt":"2022-11-01T22:37:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Optimize the very slow \"coccicheck\" target to take advantage of\nincremental rebuilding, and fix outstanding dependency problems with\nthe existing rule.\n\nThe rule is now faster both on the initial run as we can make better\nuse of GNU make's parallelism than the old ad-hoc combination of\nmake's parallelism combined with $(SPATCH_BATCH_SIZE) and/or the\n\"--jobs\" argument to \"spatch(1)\".\n\nIt also makes us *much* faster when incrementally building, it's now\nviable to \"make coccicheck\" as topic branches are merged down.\n\nThe rule didn't use FORCE (or its equivalents) before, so a:\n\n\tmake coccicheck\n\tmake coccicheck\n\nWould report nothing to do on the second iteration. But all of our\npatch output depended on all $(COCCI_SOURCES) files, therefore e.g.:\n\n make -W grep.c coccicheck\n\nWould do a full re-run, i.e. a a change in a single file would force\nus to do a full re-run.\n\nThe reason for this (not the initial rationale, but my analysis) is:\n\n* Since we create a single \"*.cocci.patch+\" we don't know where to\n pick up where we left off, or how to incrementally merge e.g. a\n \"grep.c\" change with an existing *.cocci.patch.\n\n* We've been carrying forward the dependency on the *.c files since\n 63f0a758a06 (add coccicheck make target, 2016-09-15) the rule was\n initially added as a sort of poor man's dependency discovery.\n\n As we don't include other *.c files depending on other *.c files\n has always been broken, as could be trivially demonstrated\n e.g. with:\n\n make coccicheck\n make -W strbuf.h coccicheck\n\n However, depending on the corresponding *.c files has been doing\n something, namely that *if* an API change modified both *.c and *.h\n files we'd catch the change to the *.h we care about via the *.c\n being changed.\n\n For API changes that happened only via *.h files we'd do the wrong\n thing before this change, but e.g. for function additions (not\n \"static inline\" ones) catch the *.h change by proxy.\n\nNow we'll instead:\n\n * Create a / pair in the .build directory, E.g. for\n swap.cocci and grep.c we'll create\n .build/contrib/coccinelle/swap.cocci.patch/grep.c.\n\n That file is the diff we'll apply for that -\n combination, if there's no changes to me made (the common case)\n it'll be an empty file.\n\n * Our generated *.patch\n file (e.g. contrib/coccinelle/swap.cocci.patch) is now a simple \"cat\n $^\" of all of all of the / files for a given .\n\n In the case discussed above of \"grep.c\" being changed we'll do the\n full \"cat\" every time, so they resulting *.cocci.patch will always\n be correct and up-to-date, even if it's \"incrementally updated\".\n\n See 1cc0425a27c (Makefile: have \"make pot\" not \"reset --hard\",\n 2022-05-26) for another recent rule that used that technique.\n\nAs before we'll:\n\n * End up generating a contrib/coccinelle/swap.cocci.patch, if we\n \"fail\" by creating a non-empty patch we'll still exit with a zero\n exit code.\n\n Arguably we should move to a more Makefile-native way of doing\n this, i.e. fail early, and if we want all of the \"failed\" changes\n we can use \"make -k\", but as the current\n \"ci/run-static-analysis.sh\" expects us to behave this way let's\n keep the existing behavior of exhaustively discovering all cocci\n changes, and only failing if spatch itself errors out.\n\nFurther implementation details & notes:\n\n * Before this change running \"make coccicheck\" would by default end\n up pegging just one CPU at the very end for a while, usually as\n we'd finish whichever *.cocci rule was the most expensive.\n\n This could be mitigated by combining \"make -jN\" with\n SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch\n spatch invocations, 2019-05-06).\n\n There will be cases where getting rid of \"SPATCH_BATCH_SIZE\" makes\n things worse, but a from-scratch \"make coccicheck\" with the default\n of SPATCH_BATCH_SIZE=1 (and tweaking it doesn't make a difference)\n is faster (~3m36s v.s. ~3m56s) with this approach, as we can feed\n the CPU more work in a less staggered way.\n\n * Getting rid of \"SPATCH_BATCH_SIZE\" particularly helps in cases\n where the default of 1 yields parallelism under \"make coccicheck\",\n but then running e.g.:\n\n make -W contrib/coccinelle/swap.cocci coccicheck\n\n I.e. before that would use only one CPU core, until the user\n remembered to adjust \"SPATCH_BATCH_SIZE\" differently than the\n setting that makes sense when doing a non-incremental run of \"make\n coccicheck\".\n\n * Before the \"make coccicheck\" rule would have to clean\n \"contrib/coccinelle/*.cocci.patch*\", since we'd create \"*+\" and\n \"*.log\" files there. Now those are created in\n .build/contrib/coccinelle/, which is covered by the \"cocciclean\" rule\n already.\n\nOutstanding issues & future work:\n\n * We could get rid of \"--all-includes\" in favor of manually\n specifying a list of includes to give to \"spatch(1)\".\n\n As noted upthread of [1] a naïve removal of \"--all-includes\" will\n result in broken *.cocci patches, but if we know the exhaustive\n list of includes via COMPUTE_HEADER_DEPENDENCIES we don't need to\n re-scan for them, we could grab the headers to include from the\n .depend.d/.o.d and supply them with the \"--include\" option to\n spatch(1).q\n\n1. https://lore.kernel.org/git/87ft18tcog.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 81 ++++++++++++++++++++++++-----------\n contrib/coccinelle/.gitignore | 2 +-\n shared.mak | 3 +-\n 3 files changed, 58 insertions(+), 28 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 25f0c7e837d..4bedff26285 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1306,10 +1306,6 @@ SANITIZE_ADDRESS =\n SPATCH_INCLUDE_FLAGS = --all-includes\n SPATCH_FLAGS =\n SPATCH_TEST_FLAGS =\n-# Setting SPATCH_BATCH_SIZE higher will\n-# usually result in less CPU usage at the cost of higher peak memory.\n-# Setting it to 0 will feed all files in a single spatch invocation.\n-SPATCH_BATCH_SIZE = 1\n \n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n@@ -1317,7 +1313,6 @@ TRACK_SPATCH_DEFINES += $(SPATCH)\n TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS)\n-TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n GIT-SPATCH-DEFINES: FORCE\n \t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n \t if test x\"$$FLAGS\" != x\"`cat GIT-SPATCH-DEFINES 2>/dev/null`\" ; then \\\n@@ -3163,6 +3158,7 @@ check: $(GENERATED_H)\n \n COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci)\n COCCI_RULES = $(COCCI_GLOB)\n+COCCI_NAMES = $(COCCI_RULES:contrib/coccinelle/%.cocci=%)\n \n COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES))\n COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n@@ -3170,32 +3166,65 @@ COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n \n+# It's expensive to compute the many=many rules below, only eval them\n+# on $(MAKECMDGOALS) that match these $(COCCI_RULES)\n+COCCI_RULES_GLOB =\n+COCCI_RULES_GLOB += cocci%\n+COCCI_RULES_GLOB += .build/contrib/coccinelle/%\n+COCCI_RULES_GLOB += $(COCCICHECK_PATCHES)\n+COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING)\n+COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS))\n+\n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n-COCCI_PATCHES = $(COCCI_RULES:%=%.patch)\n-$(COCCI_PATCHES): GIT-SPATCH-DEFINES\n-$(COCCI_PATCHES): $(COCCI_SOURCES)\n-$(COCCI_PATCHES): %.patch: %\n-\t$(QUIET_SPATCH) \\\n-\tif test $(SPATCH_BATCH_SIZE) = 0; then \\\n-\t\tlimit=; \\\n-\telse \\\n-\t\tlimit='-n $(SPATCH_BATCH_SIZE)'; \\\n-\tfi; \\\n-\tif ! echo $(COCCI_SOURCES) | xargs $$limit \\\n-\t\t$(SPATCH) $(SPATCH_FLAGS) \\\n-\t\t$(SPATCH_INCLUDE_FLAGS) \\\n-\t\t--sp-file $< --patch . \\\n-\t\t>$@+ 2>$@.log; \\\n+.build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES)\n+\t$(call mkdir_p_parent_template)\n+\t$(QUIET_GEN) >$@\n+\n+define cocci-rule\n+\n+## Rule for .build/$(1).patch/$(2); Params:\n+# $(1) = e.g. \"free.cocci\"\n+# $(2) = e.g. \"grep.c\"\n+COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n+.build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n+.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES\n+.build/$(1).patch/$(2): $(1)\n+.build/$(1).patch/$(2): .build/$(1).patch/% : %\n+\t$$(call mkdir_p_parent_template)\n+\t$$(QUIET_SPATCH)if ! $$(SPATCH) $$(SPATCH_FLAGS) \\\n+\t\t$$(SPATCH_INCLUDE_FLAGS) \\\n+\t\t--sp-file $(1) --patch . $$< \\\n+\t\t>$$@ 2>$$@.log; \\\n \tthen \\\n-\t\tcat $@.log; \\\n+\t\techo \"ERROR when applying '$(1)' to '$$<'; '$$@.log' follows:\"; \\\n+\t\tcat $$@.log; \\\n \t\texit 1; \\\n-\tfi; \\\n-\tmv $@+ $@; \\\n-\tif test -s $@; \\\n+\tfi\n+endef\n+\n+define cocci-matrix\n+\n+$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(1),$(s)))\n+endef\n+\n+ifdef COCCI_GOALS\n+$(eval $(foreach c,$(COCCI_RULES),$(call cocci-matrix,$(c))))\n+endif\n+\n+define spatch-rule\n+\n+contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1))\n+\t$$(QUIET_SPATCH_CAT)cat $$^ >$$@ && \\\n+\tif test -s $$@; \\\n \tthen \\\n-\t\techo ' ' SPATCH result: $@; \\\n+\t\techo ' ' SPATCH result: $$@; \\\n \tfi\n+endef\n+\n+ifdef COCCI_GOALS\n+$(eval $(foreach n,$(COCCI_NAMES),$(call spatch-rule,$(n))))\n+endif\n \n COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n@@ -3487,7 +3516,7 @@ profile-clean:\n cocciclean:\n \t$(RM) GIT-SPATCH-DEFINES\n \t$(RM) -r .build/contrib/coccinelle\n-\t$(RM) contrib/coccinelle/*.cocci.patch*\n+\t$(RM) contrib/coccinelle/*.cocci.patch\n \n clean: profile-clean coverage-clean cocciclean\n \t$(RM) -r .build\ndiff --git a/contrib/coccinelle/.gitignore b/contrib/coccinelle/.gitignore\nindex d3f29646dc3..1d45c0a40c8 100644\n--- a/contrib/coccinelle/.gitignore\n+++ b/contrib/coccinelle/.gitignore\n@@ -1 +1 @@\n-*.patch*\n+*.patch\ndiff --git a/shared.mak b/shared.mak\nindex 96b06acc455..f437073e48c 100644\n--- a/shared.mak\n+++ b/shared.mak\n@@ -71,8 +71,9 @@ ifndef V\n \tQUIET_RC = @echo ' ' RC $@;\n \n ## Used in \"Makefile\": SPATCH\n-\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $<;\n+\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $@;\n \tQUIET_SPATCH_TEST\t\t= @echo ' ' SPATCH TEST $(@:.build/%=%);\n+\tQUIET_SPATCH_CAT\t\t= @echo ' ' SPATCH CAT $$^ \\>$@;\n \n ## Used in \"Documentation/Makefile\"\n \tQUIET_ASCIIDOC\t= @echo ' ' ASCIIDOC $@;\n-- \n2.38.0.1280.g8136eb6fab2\n\n"},{"id":"466235","messageId":"patch-v5-10.13-56ca8f5720a-20221101T222616Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v5-00.13-00000000000-20221101T222616Z-avarab@gmail.com","subject":"[PATCH v5 10/13] Makefile: copy contrib/coccinelle/*.cocci to build/","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T22:35:52Z","receivedAt":"2022-11-01T22:37:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the \"coccinelle\" rule so that we first copy the *.cocci source\nin e.g. \"contrib/coccinelle/strbuf.cocci\" to\n\".build/contrib/coccinelle/strbuf.cocci\" before operating on it.\n\nFor now this serves as a rather pointless indirection, but prepares us\nfor the subsequent commit where we'll be able to inject generated\n*.cocci files. Having the entire dependency tree live inside .build/*\nsimplifies both the globbing we'd need to do, and any \"clean\" rules.\n\nIt will also help for future targets which will want to act on the\ngenerated patches or the logs, e.g. targets to alert if we can't parse\ncertain files (or, less so than usual) with \"spatch\", and e.g. a\nreplacement for \"ci/run-static-analysis.sh\". Such a replacement won't\ncare about placing the patches in the in-tree, only whether they're\n\"OK\" (and about the diff).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 27 +++++++++++++++++++++------\n shared.mak | 1 +\n 2 files changed, 22 insertions(+), 6 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex c7c96d284dd..44c906b65d5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3164,8 +3164,11 @@ check: $(GENERATED_H)\n \tfi\n \n COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci)\n-COCCI_RULES = $(COCCI_GLOB)\n-COCCI_NAMES = $(COCCI_RULES:contrib/coccinelle/%.cocci=%)\n+COCCI_RULES_TRACKED = $(COCCI_GLOB:%=.build/%)\n+COCCI_RULES =\n+COCCI_RULES += $(COCCI_RULES_TRACKED)\n+COCCI_NAMES =\n+COCCI_NAMES += $(COCCI_RULES:.build/contrib/coccinelle/%.cocci=%)\n \n COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES))\n COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n@@ -3173,6 +3176,9 @@ COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n \n+COCCICHECK_PATCHES_INTREE = $(COCCICHECK_PATCHES:.build/%=%)\n+COCCICHECK_PATCHES_PENDING_INTREE = $(COCCICHECK_PATCHES_PENDING:.build/%=%)\n+\n # It's expensive to compute the many=many rules below, only eval them\n # on $(MAKECMDGOALS) that match these $(COCCI_RULES)\n COCCI_RULES_GLOB =\n@@ -3180,10 +3186,16 @@ COCCI_RULES_GLOB += cocci%\n COCCI_RULES_GLOB += .build/contrib/coccinelle/%\n COCCI_RULES_GLOB += $(COCCICHECK_PATCHES)\n COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING)\n+COCCI_RULES_GLOB += $(COCCICHECK_PATCHES_INTREE)\n+COCCI_RULES_GLOB += $(COCCICHECK_PATCHES_PENDING_INTREE)\n COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS))\n \n COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \n+$(COCCI_RULES_TRACKED): .build/% : %\n+\t$(call mkdir_p_parent_template)\n+\t$(QUIET_CP)cp $< $@\n+\n .build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES)\n \t$(call mkdir_p_parent_template)\n \t$(QUIET_GEN) >$@\n@@ -3197,7 +3209,7 @@ define cocci-rule\n # $(1) = e.g. \"free.cocci\"\n # $(2) = e.g. \"grep.c\"\n # $(3) = e.g. \"grep.o\"\n-COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n+COCCI_$(1:.build/contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n .build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n .build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES)\n .build/$(1).patch/$(2): $(1)\n@@ -3225,12 +3237,15 @@ endif\n \n define spatch-rule\n \n-contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1))\n+.build/contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1))\n \t$$(QUIET_SPATCH_CAT)cat $$^ >$$@ && \\\n \tif test -s $$@; \\\n \tthen \\\n \t\techo ' ' SPATCH result: $$@; \\\n \tfi\n+contrib/coccinelle/$(1).cocci.patch: .build/contrib/coccinelle/$(1).cocci.patch\n+\t$$(QUIET_CP)cp $$< $$@\n+\n endef\n \n ifdef COCCI_GOALS\n@@ -3254,11 +3269,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell\n coccicheck-test: $(COCCI_TEST_RES_GEN)\n \n coccicheck: coccicheck-test\n-coccicheck: $(COCCICHECK_PATCHES)\n+coccicheck: $(COCCICHECK_PATCHES_INTREE)\n \n # See contrib/coccinelle/README\n coccicheck-pending: coccicheck-test\n-coccicheck-pending: $(COCCICHECK_PATCHES_PENDING)\n+coccicheck-pending: $(COCCICHECK_PATCHES_PENDING_INTREE)\n \n .PHONY: coccicheck coccicheck-pending\n \ndiff --git a/shared.mak b/shared.mak\nindex f437073e48c..a34b66c926d 100644\n--- a/shared.mak\n+++ b/shared.mak\n@@ -60,6 +60,7 @@ ifndef V\n \tQUIET_AR = @echo ' ' AR $@;\n \tQUIET_LINK = @echo ' ' LINK $@;\n \tQUIET_BUILT_IN = @echo ' ' BUILTIN $@;\n+\tQUIET_CP = @echo ' ' CP $< $@;\n \tQUIET_LNCP = @echo ' ' LN/CP $@;\n \tQUIET_XGETTEXT = @echo ' ' XGETTEXT $@;\n \tQUIET_MSGINIT = @echo ' ' MSGINIT $@;\n-- \n2.38.0.1280.g8136eb6fab2\n\n"},{"id":"466236","messageId":"patch-v5-09.13-332fb6c8640-20221101T222616Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v5-00.13-00000000000-20221101T222616Z-avarab@gmail.com","subject":"[PATCH v5 09/13] cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T22:35:51Z","receivedAt":"2022-11-01T22:37:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improve the incremental rebuilding support of \"coccicheck\" by\npiggy-backing on the computed dependency information of the\ncorresponding *.o file, rather than rebuilding all / pairs\nif either their corresponding file changes, or if any header changes.\n\nThis in effect uses the same method that the \"sparse\" target was made\nto use in c234e8a0ecf (Makefile: make the \"sparse\" target non-.PHONY,\n2021-09-23), except that the dependency on the *.o file isn't a hard\none, we check with $(wildcard) if the *.o file exists, and if so we'll\ndepend on it.\n\nThis means that the common case of:\n\n\tmake\n\tmake coccicheck\n\nWill benefit from incremental rebuilding, now changing e.g. a header\nwill only re-run \"spatch\" on those those *.c files that make use of\nit:\n\nBy depending on the *.o we piggy-back on\nCOMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the\n\"sparse\" target non-.PHONY, 2021-09-23) for prior art of doing that\nfor the *.sp files. E.g.:\n\n make contrib/coccinelle/free.cocci.patch\n make -W column.h contrib/coccinelle/free.cocci.patch\n\nWill take around 15 seconds for the second command on my 8 core box if\nI didn't run \"make\" beforehand to create the *.o files. But around 2\nseconds if I did and we have those \"*.o\" files.\n\nNotes about the approach of piggy-backing on *.o for dependencies:\n\n * It *is* a trade-off since we'll pay the extra cost of running the C\n compiler, but we're probably doing that anyway. The compiler is much\n faster than \"spatch\", so even though we need to re-compile the *.o to\n create the dependency info for the *.c for \"spatch\" it's\n faster (especially if using \"ccache\").\n\n * There *are* use-cases where some would like to have *.o files\n around, but to have the \"make coccicheck\" ignore them. See:\n https://lore.kernel.org/git/20220826104312.GJ1735@szeder.dev/\n\n For those users a:\n\n\tmake\n\tmake coccicheck SPATCH_USE_O_DEPENDENCIES=\n\n Will avoid considering the *.o files.\n\n * If that *.o file doesn't exist we'll depend on an intermediate file\n of ours which in turn depends on $(FOUND_H_SOURCES).\n\n This covers both an initial build, or where \"coccicheck\" is run\n without running \"all\" beforehand, and because we run \"coccicheck\"\n on e.g. files in compat/* that we don't know how to build unless\n the requisite flag was provided to the Makefile.\n\n Most of the runtime of \"incremental\" runs is now spent on various\n compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and\n therefore conflate whether we *can* compile an object and generate\n dependency information for it with whether we'd like to link it\n into our binary.\n\n Before this change the distinction didn't matter, but now one way\n to make this even faster on incremental builds would be to peel\n those concerns apart so that we can see that e.g. compat/mmap.c\n doesn't depend on column.h.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 15 +++++++++++++--\n contrib/coccinelle/README | 16 ++++++++++++++++\n 2 files changed, 29 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 4bedff26285..c7c96d284dd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1307,6 +1307,13 @@ SPATCH_INCLUDE_FLAGS = --all-includes\n SPATCH_FLAGS =\n SPATCH_TEST_FLAGS =\n \n+# If *.o files are present, have \"coccicheck\" depend on them, with\n+# COMPUTE_HEADER_DEPENDENCIES this will speed up the common-case of\n+# only needing to re-generate coccicheck results for the users of a\n+# given API if it's changed, and not all files in the project. If\n+# COMPUTE_HEADER_DEPENDENCIES=no this will be unset too.\n+SPATCH_USE_O_DEPENDENCIES = YesPlease\n+\n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n TRACK_SPATCH_DEFINES += $(SPATCH)\n@@ -3181,14 +3188,18 @@ COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n \t$(call mkdir_p_parent_template)\n \t$(QUIET_GEN) >$@\n \n+ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no)\n+SPATCH_USE_O_DEPENDENCIES =\n+endif\n define cocci-rule\n \n ## Rule for .build/$(1).patch/$(2); Params:\n # $(1) = e.g. \"free.cocci\"\n # $(2) = e.g. \"grep.c\"\n+# $(3) = e.g. \"grep.o\"\n COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n .build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n-.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES\n+.build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES)\n .build/$(1).patch/$(2): $(1)\n .build/$(1).patch/$(2): .build/$(1).patch/% : %\n \t$$(call mkdir_p_parent_template)\n@@ -3205,7 +3216,7 @@ endef\n \n define cocci-matrix\n \n-$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(1),$(s)))\n+$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o)))\n endef\n \n ifdef COCCI_GOALS\ndiff --git a/contrib/coccinelle/README b/contrib/coccinelle/README\nindex f0e80bd7f03..09ea8298e19 100644\n--- a/contrib/coccinelle/README\n+++ b/contrib/coccinelle/README\n@@ -41,3 +41,19 @@ There are two types of semantic patches:\n \n This allows to expose plans of pending large scale refactorings without\n impacting the bad pattern checks.\n+\n+Git-specific tips & things to know about how we run \"spatch\":\n+\n+ * The \"make coccicheck\" will piggy-back on\n+ \"COMPUTE_HEADER_DEPENDENCIES\". If you've built a given object file\n+ the \"coccicheck\" target will consider its depednency to decide if\n+ it needs to re-run on the corresponding source file.\n+\n+ This means that a \"make coccicheck\" will re-compile object files\n+ before running. This might be unexpected, but speeds up the run in\n+ the common case, as e.g. a change to \"column.h\" won't require all\n+ coccinelle rules to be re-run against \"grep.c\" (or another file\n+ that happens not to use \"column.h\").\n+\n+ To disable this behavior use the \"SPATCH_USE_O_DEPENDENCIES=NoThanks\"\n+ flag.\n-- \n2.38.0.1280.g8136eb6fab2\n\n"},{"id":"466237","messageId":"patch-v5-11.13-73bb897c965-20221101T222616Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v5-00.13-00000000000-20221101T222616Z-avarab@gmail.com","subject":"[PATCH v5 11/13] cocci rules: remove 's from rules that don't need them","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T22:35:53Z","receivedAt":"2022-11-01T22:37:31Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The in the part of the coccinelle syntax[1] is for our\npurposes there to declares if we have inter-dependencies between\ndifferent rules.\n\nBut such 's must be unique within a given semantic patch file. As\nwe'll be processing a concatenated version of our rules in the\nsubsequent commit let's remove these names. They weren't being used\nfor the semantic patches themselves, and equated to a short comment\nabout the rule.\n\nBoth the filename and context of the rules makes it clear what they're\ndoing, so we're not gaining anything from keeping these. Retaining\nthem goes against recommendations that \"contrib/coccinelle/README\"\nwill be making in the subsequent commit.\n\nThis leaves only one named rule in our sources, where it's needed for\na \" <-> \" relationship:\n\n\t$ git -P grep '^@ ' -- contrib/coccinelle/\n\tcontrib/coccinelle/swap.cocci:@ swap @\n\tcontrib/coccinelle/swap.cocci:@ extends swap @\n\n1. https://coccinelle.gitlabpages.inria.fr/website/docs/main_grammar.html\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n contrib/coccinelle/hashmap.cocci | 2 +-\n contrib/coccinelle/preincr.cocci | 2 +-\n contrib/coccinelle/strbuf.cocci | 2 +-\n contrib/coccinelle/swap.cocci | 2 +-\n 4 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/contrib/coccinelle/hashmap.cocci b/contrib/coccinelle/hashmap.cocci\nindex d69e120ccff..c5dbb4557b5 100644\n--- a/contrib/coccinelle/hashmap.cocci\n+++ b/contrib/coccinelle/hashmap.cocci\n@@ -1,4 +1,4 @@\n-@ hashmap_entry_init_usage @\n+@@\n expression E;\n struct hashmap_entry HME;\n @@\ndiff --git a/contrib/coccinelle/preincr.cocci b/contrib/coccinelle/preincr.cocci\nindex 7fe1e8d2d9a..ae42cb07302 100644\n--- a/contrib/coccinelle/preincr.cocci\n+++ b/contrib/coccinelle/preincr.cocci\n@@ -1,4 +1,4 @@\n-@ preincrement @\n+@@\n identifier i;\n @@\n - ++i > 1\ndiff --git a/contrib/coccinelle/strbuf.cocci b/contrib/coccinelle/strbuf.cocci\nindex 0970d98ad72..5f06105df6d 100644\n--- a/contrib/coccinelle/strbuf.cocci\n+++ b/contrib/coccinelle/strbuf.cocci\n@@ -1,4 +1,4 @@\n-@ strbuf_addf_with_format_only @\n+@@\n expression E;\n constant fmt !~ \"%\";\n @@\ndiff --git a/contrib/coccinelle/swap.cocci b/contrib/coccinelle/swap.cocci\nindex a0934d1fdaf..522177afb66 100644\n--- a/contrib/coccinelle/swap.cocci\n+++ b/contrib/coccinelle/swap.cocci\n@@ -1,4 +1,4 @@\n-@ swap_with_declaration @\n+@@\n type T;\n identifier tmp;\n T a, b;\n-- \n2.38.0.1280.g8136eb6fab2\n\n"},{"id":"466238","messageId":"patch-v5-12.13-55d2f638359-20221101T222616Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v5-00.13-00000000000-20221101T222616Z-avarab@gmail.com","subject":"[PATCH v5 12/13] cocci: run against a generated ALL.cocci","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T22:35:54Z","receivedAt":"2022-11-01T22:37:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The preceding commits to make the \"coccicheck\" target incremental made\nit slower in some cases. As an optimization let's not have the\nmany=many mapping of <*.cocci>=<*.[ch]>, but instead concat the\n<*.cocci> into an ALL.cocci, and then run one-to-many\nALL.cocci=<*.[ch]>.\n\nA \"make coccicheck\" is now around 2x as fast as it was on \"master\",\nand around 1.5x as fast as the preceding change to make the run\nincremental:\n\n\t$ git hyperfine -L rev origin/master,HEAD~,HEAD -p 'make clean' 'make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' -r 3\n\tBenchmark 1: make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'origin/master\n\t Time (mean ± σ): 4.258 s ± 0.015 s [User: 27.432 s, System: 1.532 s]\n\t Range (min … max): 4.241 s … 4.268 s 3 runs\n\n\tBenchmark 2: make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'HEAD~\n\t Time (mean ± σ): 5.365 s ± 0.079 s [User: 36.899 s, System: 1.810 s]\n\t Range (min … max): 5.281 s … 5.436 s 3 runs\n\n\tBenchmark 3: make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'HEAD\n\t Time (mean ± σ): 2.725 s ± 0.063 s [User: 14.796 s, System: 0.233 s]\n\t Range (min … max): 2.667 s … 2.792 s 3 runs\n\n\tSummary\n\t 'make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'HEAD' ran\n\t 1.56 ± 0.04 times faster than 'make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'origin/master'\n\t 1.97 ± 0.05 times faster than 'make coccicheck SPATCH=spatch COCCI_SOURCES=\"$(echo $(ls o*.c builtin/h*.c))\"' in 'HEAD~'\n\nThis can be turned off with SPATCH_CONCAT_COCCI, but as the\nbeneficiaries of \"SPATCH_CONCAT_COCCI=\" would mainly be those\ndeveloping the *.cocci rules themselves, let's leave this optimization\non by default.\n\nFor more information see my \"Optimizing *.cocci rules by concat'ing\nthem\" (<220901.8635dbjfko.gmgdl@evledraar.gmail.com>) on the\ncocci@inria.fr mailing list.\n\nThis potentially changes the results of our *.cocci rules, but as\nnoted in that discussion it should be safe for our use. We don't name\nrules, or if we do their names don't conflict across our *.cocci\nfiles.\n\nTo the extent that we'd have any inter-dependencies between rules this\ndoesn't make that worse, as we'd have them now if we ran \"make\ncoccicheck\", applied the results, and would then have (due to\nhypothetical interdependencies) suggested changes on the subsequent\n\"make coccicheck\".\n\nOur \"coccicheck-test\" target makes use of the ALL.cocci when running\ntests, e.g. when testing unused.{c,out} we test it against ALL.cocci,\nnot unused.cocci. We thus assert (to the extent that we have test\ncoverage) that this concatenation doesn't change the expected results\nof running these rules.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 38 ++++++++++++++++++++++++++++++++++++++\n contrib/coccinelle/README | 13 +++++++++++++\n 2 files changed, 51 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex 44c906b65d5..36cefe3b1d6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1314,6 +1314,29 @@ SPATCH_TEST_FLAGS =\n # COMPUTE_HEADER_DEPENDENCIES=no this will be unset too.\n SPATCH_USE_O_DEPENDENCIES = YesPlease\n \n+# Set SPATCH_CONCAT_COCCI to concatenate the contrib/cocci/*.cocci\n+# files into a single contrib/cocci/ALL.cocci before running\n+# \"coccicheck\".\n+#\n+# Pros:\n+#\n+# - Speeds up a one-shot run of \"make coccicheck\", as we won't have to\n+# parse *.[ch] files N times for the N *.cocci rules\n+#\n+# Cons:\n+#\n+# - Will make incremental development of *.cocci slower, as\n+# e.g. changing strbuf.cocci will re-run all *.cocci.\n+#\n+# - Makes error and performance analysis harder, as rules will be\n+# applied from a monolithic ALL.cocci, rather than\n+# e.g. strbuf.cocci. To work around this either undefine this, or\n+# generate a specific patch, e.g. this will always use strbuf.cocci,\n+# not ALL.cocci:\n+#\n+#\tmake contrib/coccinelle/strbuf.cocci.patch\n+SPATCH_CONCAT_COCCI = YesPlease\n+\n # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n TRACK_SPATCH_DEFINES =\n TRACK_SPATCH_DEFINES += $(SPATCH)\n@@ -3163,9 +3186,12 @@ check: $(GENERATED_H)\n \t\texit 1; \\\n \tfi\n \n+COCCI_GEN_ALL = .build/contrib/coccinelle/ALL.cocci\n COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci)\n COCCI_RULES_TRACKED = $(COCCI_GLOB:%=.build/%)\n+COCCI_RULES_TRACKED_NO_PENDING = $(filter-out %.pending.cocci,$(COCCI_RULES_TRACKED))\n COCCI_RULES =\n+COCCI_RULES += $(COCCI_GEN_ALL)\n COCCI_RULES += $(COCCI_RULES_TRACKED)\n COCCI_NAMES =\n COCCI_NAMES += $(COCCI_RULES:.build/contrib/coccinelle/%.cocci=%)\n@@ -3200,6 +3226,10 @@ $(COCCI_RULES_TRACKED): .build/% : %\n \t$(call mkdir_p_parent_template)\n \t$(QUIET_GEN) >$@\n \n+$(COCCI_GEN_ALL): $(COCCI_RULES_TRACKED_NO_PENDING)\n+\t$(call mkdir_p_parent_template)\n+\t$(QUIET_SPATCH_CAT)cat $^ >$@\n+\n ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no)\n SPATCH_USE_O_DEPENDENCIES =\n endif\n@@ -3256,7 +3286,11 @@ COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n $(COCCI_TEST_RES_GEN): .build/%.res : %.c\n $(COCCI_TEST_RES_GEN): .build/%.res : %.res\n+ifdef SPATCH_CONCAT_COCCI\n+$(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : $(COCCI_GEN_ALL)\n+else\n $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci\n+endif\n \t$(call mkdir_p_parent_template)\n \t$(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_TEST_FLAGS) \\\n \t\t--very-quiet --no-show-diff \\\n@@ -3269,7 +3303,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell\n coccicheck-test: $(COCCI_TEST_RES_GEN)\n \n coccicheck: coccicheck-test\n+ifdef SPATCH_CONCAT_COCCI\n+coccicheck: contrib/coccinelle/ALL.cocci.patch\n+else\n coccicheck: $(COCCICHECK_PATCHES_INTREE)\n+endif\n \n # See contrib/coccinelle/README\n coccicheck-pending: coccicheck-test\ndiff --git a/contrib/coccinelle/README b/contrib/coccinelle/README\nindex 09ea8298e19..09b72bfd4e7 100644\n--- a/contrib/coccinelle/README\n+++ b/contrib/coccinelle/README\n@@ -57,3 +57,16 @@ Git-specific tips & things to know about how we run \"spatch\":\n \n To disable this behavior use the \"SPATCH_USE_O_DEPENDENCIES=NoThanks\"\n flag.\n+\n+ * To speed up our rules the \"make coccicheck\" target will by default\n+ concatenate all of the *.cocci files here into an \"ALL.cocci\", and\n+ apply it to each source file.\n+\n+ This makes the run faster, as we don't need to run each rule\n+ against each source file. See the Makefile for further discussion,\n+ this behavior can be disabled with \"SPATCH_CONCAT_COCCI=\".\n+\n+ But since they're concatenated any in the (e.g. \"@\n+ my_name\", v.s. anonymous \"@@\") needs to be unique across all our\n+ *.cocci files. You should only need to name rules if other rules\n+ depend on them (currently only one rule is named).\n-- \n2.38.0.1280.g8136eb6fab2\n\n"},{"id":"466239","messageId":"patch-v5-13.13-ef578c230ea-20221101T222616Z-avarab@gmail.com","threadId":"58360","inReplyTo":"cover-v5-00.13-00000000000-20221101T222616Z-avarab@gmail.com","subject":"[PATCH v5 13/13] spatchcache: add a ccache-alike for \"spatch\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T22:35:55Z","receivedAt":"2022-11-01T22:37:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a rather trivial \"spatchcache\", with this running e.g.:\n\n\tmake cocciclean\n\tmake contrib/coccinelle/free.cocci.patch \\\n\t\tSPATCH=contrib/coccicheck/spatchcache \\\n\t\tSPATCH_FLAGS=--very-quiet\n\nIs cut down from ~20s to ~5s on my system. Much of that is either\nfixable shell overhead, or the around 40 files we \"CANTCACHE\" (see the\nimplementation).\n\nThis uses \"redis\" as a cache by default, but it's configurable. See\nthe embedded documentation.\n\nThis is *not* like ccache in that we won't cache failed spatch\ninvocations, or those where spatch suggests changes for us. Those\ncases are so rare that I didn't think it was worth the bother, by far\nthe most common case is that it has no suggested changes. We'll also\nrefuse to cache any \"spatch\" invocation that has output on stderr,\nwhich means that \"--very-quiet\" must be added to \"SPATCH_FLAGS\".\n\nBecause we narrow the cache to that we don't need to save away stdout,\nstderr & the exit code. We simply cache the cases where we had no\nsuggested changes.\n\nAnother benchmark is to compare this with the previous\nSPATCH_BATCH_SIZE=N, as noted in [1]. Before this (on my 8 core system) running:\n\n\tmake clean; time make contrib/coccinelle/array.cocci.patch SPATCH_BATCH_SIZE=0\n\nWould take 33s, but with the preceding changes running without this\n\"spatchcache\" is slightly slower, or around 35s:\n\n\tmake clean; time make contrib/coccinelle/array.cocci.patch\n\nNow doing the same with SPATCH=contrib/coccinelle/spatchcache will\ntake around 6s, but we'll need to compile the *.o files first to take\nfull advantage of it (which can be fast with \"ccache\"):\n\n\tmake clean; make; time make contrib/coccinelle/array.cocci.patch SPATCH=contrib/coccinelle/spatchcache\n\n1. https://lore.kernel.org/git/YwdRqP1CyUAzCEn2@coredump.intra.peff.net/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n contrib/coccinelle/README | 20 +++\n contrib/coccinelle/spatchcache | 304 +++++++++++++++++++++++++++++++++\n 2 files changed, 324 insertions(+)\n create mode 100755 contrib/coccinelle/spatchcache\n\ndiff --git a/contrib/coccinelle/README b/contrib/coccinelle/README\nindex 09b72bfd4e7..d1daa1f6263 100644\n--- a/contrib/coccinelle/README\n+++ b/contrib/coccinelle/README\n@@ -70,3 +70,23 @@ Git-specific tips & things to know about how we run \"spatch\":\n my_name\", v.s. anonymous \"@@\") needs to be unique across all our\n *.cocci files. You should only need to name rules if other rules\n depend on them (currently only one rule is named).\n+\n+ * To speed up incremental runs even more use the \"spatchcache\" tool\n+ in this directory as your \"SPATCH\". It aimns to be a \"ccache\" for\n+ coccinelle, and piggy-backs on \"COMPUTE_HEADER_DEPENDENCIES\".\n+\n+ It caches in Redis by default, see it source for a how-to.\n+\n+ In one setup with a primed cache \"make coccicheck\" followed by a\n+ \"make clean && make\" takes around 10s to run, but 2m30s with the\n+ default of \"SPATCH_CONCAT_COCCI=Y\".\n+\n+ With \"SPATCH_CONCAT_COCCI=\" the total runtime is around ~6m, sped\n+ up to ~1m with \"spatchcache\".\n+\n+ Most of the 10s (or ~1m) being spent on re-running \"spatch\" on\n+ files we couldn't cache, as we didn't compile them (in contrib/*\n+ and compat/* mostly).\n+\n+ The absolute times will differ for you, but the relative speedup\n+ from caching should be on that order.\ndiff --git a/contrib/coccinelle/spatchcache b/contrib/coccinelle/spatchcache\nnew file mode 100755\nindex 00000000000..29e9352d8a2\n--- /dev/null\n+++ b/contrib/coccinelle/spatchcache\n@@ -0,0 +1,304 @@\n+#!/bin/sh\n+#\n+# spatchcache: a poor-man's \"ccache\"-alike for \"spatch\" in git.git\n+#\n+# This caching command relies on the peculiarities of the Makefile\n+# driving \"spatch\" in git.git, in particular if we invoke:\n+#\n+#\tmake\n+#\t# See \"spatchCache.cacheWhenStderr\" for why \"--very-quiet\" is\n+#\t# used\n+#\tmake coccicheck SPATCH_FLAGS=--very-quiet\n+#\n+# We can with COMPUTE_HEADER_DEPENDENCIES (auto-detected as true with\n+# \"gcc\" and \"clang\") write e.g. a .depend/grep.o.d for grep.c, when we\n+# compile grep.o.\n+#\n+# The .depend/grep.o.d will have the full header dependency tree of\n+# grep.c, and we can thus cache the output of \"spatch\" by:\n+#\n+#\t1. Hashing all of those files\n+#\t2. Hashing our source file, and the *.cocci rule we're\n+#\t applying\n+#\t3. Running spatch, if suggests no changes (by far the common\n+#\t case) we invoke \"spatchCache.getCmd\" and\n+#\t \"spatchCache.setCmd\" with a hash SHA-256 to ask \"does this\n+#\t ID have no changes\" or \"say that ID had no changes>\n+#\t4. If no \"spatchCache.{set,get}Cmd\" is specified we'll use\n+#\t \"redis-cli\" and maintain a SET called \"spatch-cache\". Set\n+#\t appropriate redis memory policies to keep it from growing\n+#\t out of control.\n+#\n+# This along with the general incremental \"make\" support for\n+# \"contrib/coccinelle\" makes it viable to (re-)run coccicheck\n+# e.g. when merging integration branches.\n+#\n+# Note that the \"--very-quiet\" flag is currently critical. The cache\n+# will refuse to cache anything that has output on STDERR (which might\n+# be errors from spatch), but see spatchCache.cacheWhenStderr below.\n+#\n+# The STDERR (and exit code) could in principle be cached (as with\n+# ccache), but then the simple structure in the Redis cache would need\n+# to change, so just supply \"--very-quiet\" for now.\n+#\n+# To use this, simply set SPATCH to\n+# contrib/coccinelle/spatchcache. Then optionally set:\n+#\n+#\t[spatchCache]\n+#\t\t# Optional: path to a custom spatch\n+#\t\tspatch = ~/g/coccicheck/spatch.opt\n+#\n+# As well as this trace config (debug implies trace):\n+#\n+#\t\tcacheWhenStderr = true\n+#\t\ttrace = false\n+#\t\tdebug = false\n+#\n+# The \".depend/grep.o.d\" can also be customized, as a string that will\n+# be eval'd, it has access to a \"$dirname\" and \"$basename\":\n+#\n+#\t[spatchCache]\n+#\t\tdependFormat = \"$dirname/.depend/${basename%.c}.o.d\"\n+#\n+# Setting \"trace\" to \"true\" allows for seeing when we have a cache HIT\n+# or MISS. To debug whether the cache is working do that, and run e.g.:\n+#\n+#\tredis-cli FLUSHALL\n+#\t\n+#\tgrep -hore HIT -e MISS -e SET -e NOCACHE -e CANTCACHE .build/contrib/coccinelle | sort | uniq -c\n+#\t 600 CANTCACHE\n+#\t 7365 MISS\n+#\t 7365 SET\n+#\n+# A subsequent \"make cocciclean && make coccicheck\" should then have\n+# all \"HIT\"'s and \"CANTCACHE\"'s.\n+#\n+# The \"spatchCache.cacheWhenStderr\" option is critical when using\n+# spatchCache.{trace,debug} to debug whether something is set in the\n+# cache, as we'll write to the spatch logs in .build/* we'd otherwise\n+# always emit a NOCACHE.\n+#\n+# Reading the config can make the command much slower, to work around\n+# this the config can be set in the environment, with environment\n+# variable name corresponding to the config key. \"default\" can be used\n+# to use whatever's the script default, e.g. setting\n+# spatchCache.cacheWhenStderr=true and deferring to the defaults for\n+# the rest is:\n+#\n+#\texport GIT_CONTRIB_SPATCHCACHE_DEBUG=default\n+#\texport GIT_CONTRIB_SPATCHCACHE_TRACE=default\n+#\texport GIT_CONTRIB_SPATCHCACHE_CACHEWHENSTDERR=true\n+#\texport GIT_CONTRIB_SPATCHCACHE_SPATCH=default\n+#\texport GIT_CONTRIB_SPATCHCACHE_DEPENDFORMAT=default\n+#\texport GIT_CONTRIB_SPATCHCACHE_SETCMD=default\n+#\texport GIT_CONTRIB_SPATCHCACHE_GETCMD=default\n+\n+set -e\n+\n+env_or_config () {\n+\tenv=\"$1\"\n+\tshift\n+\tif test \"$env\" = \"default\"\n+\tthen\n+\t\t# Avoid expensive \"git config\" invocation\n+\t\treturn\n+\telif test -n \"$env\"\n+\tthen\n+\t\techo \"$env\"\n+\telse\n+\t\tgit config $@ || :\n+\tfi\n+}\n+\n+## Our own configuration & options\n+debug=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_DEBUG\" --bool \"spatchCache.debug\")\n+if test \"$debug\" != \"true\"\n+then\n+\tdebug=\n+fi\n+if test -n \"$debug\"\n+then\n+\tset -x\n+fi\n+\n+trace=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_TRACE\" --bool \"spatchCache.trace\")\n+if test \"$trace\" != \"true\"\n+then\n+\ttrace=\n+fi\n+if test -n \"$debug\"\n+then\n+\t# debug implies trace\n+\ttrace=true\n+fi\n+\n+cacheWhenStderr=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_CACHEWHENSTDERR\" --bool \"spatchCache.cacheWhenStderr\")\n+if test \"$cacheWhenStderr\" != \"true\"\n+then\n+\tcacheWhenStderr=\n+fi\n+\n+trace_it () {\n+\tif test -z \"$trace\"\n+\tthen\n+\t\treturn\n+\tfi\n+\techo \"$@\" >&2\n+}\n+\n+spatch=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_SPATCH\" --path \"spatchCache.spatch\")\n+if test -n \"$spatch\"\n+then\n+\tif test -n \"$debug\"\n+\tthen\n+\t\ttrace_it \"custom spatchCache.spatch='$spatch'\"\n+\tfi\n+else\n+\tspatch=spatch\n+fi\n+\n+dependFormat='$dirname/.depend/${basename%.c}.o.d'\n+dependFormatCfg=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_DEPENDFORMAT\" \"spatchCache.dependFormat\")\n+if test -n \"$dependFormatCfg\"\n+then\n+\tdependFormat=\"$dependFormatCfg\"\n+fi\n+\n+set=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_SETCMD\" \"spatchCache.setCmd\")\n+get=$(env_or_config \"$GIT_CONTRIB_SPATCHCACHE_GETCMD\" \"spatchCache.getCmd\")\n+\n+## Parse spatch()-like command-line for caching info\n+arg_sp=\n+arg_file=\n+args=\"$@\"\n+spatch_opts() {\n+\twhile test $# != 0\n+\tdo\n+\t\targ_file=\"$1\"\n+\t\tcase \"$1\" in\n+\t\t--sp-file)\n+\t\t\targ_sp=\"$2\"\n+\t\t\t;;\n+\t\tesac\n+\t\tshift\n+\tdone\n+}\n+spatch_opts \"$@\"\n+if ! test -f \"$arg_file\"\n+then\n+\targ_file=\n+fi\n+\n+hash_for_cache() {\n+\t# Parameters that should affect the cache\n+\techo \"args=$args\"\n+\techo \"config spatchCache.spatch=$spatch\"\n+\techo \"config spatchCache.debug=$debug\"\n+\techo \"config spatchCache.trace=$trace\"\n+\techo \"config spatchCache.cacheWhenStderr=$cacheWhenStderr\"\n+\techo\n+\n+\t# Our target file and its dependencies\n+\tgit hash-object \"$1\" \"$2\" $(grep -E -o '^[^:]+:$' \"$3\" | tr -d ':')\n+}\n+\n+# Sanity checks\n+if ! test -f \"$arg_sp\" && ! test -f \"$arg_file\"\n+then\n+\techo $0: no idea how to cache \"$@\" >&2\n+\texit 128\n+fi\n+\n+# Main logic\n+dirname=$(dirname \"$arg_file\")\n+basename=$(basename \"$arg_file\")\n+eval \"dep=$dependFormat\"\n+\n+if ! test -f \"$dep\"\n+then\n+\ttrace_it \"$0: CANTCACHE have no '$dep' for '$arg_file'!\"\n+\texec \"$spatch\" \"$@\"\n+fi\n+\n+if test -n \"$debug\"\n+then\n+\ttrace_it \"$0: The full cache input for '$arg_sp' '$arg_file' '$dep'\"\n+\thash_for_cache \"$arg_sp\" \"$arg_file\" \"$dep\" >&2\n+fi\n+sum=$(hash_for_cache \"$arg_sp\" \"$arg_file\" \"$dep\" | git hash-object --stdin)\n+\n+trace_it \"$0: processing '$arg_file' with '$arg_sp' rule, and got hash '$sum' for it + '$dep'\"\n+\n+getret=\n+if test -z \"$get\"\n+then\n+\tif test $(redis-cli SISMEMBER spatch-cache \"$sum\") = 1\n+\tthen\n+\t\tgetret=0\n+\telse\n+\t\tgetret=1\n+\tfi\n+else\n+\t$set \"$sum\"\n+\tgetret=$?\n+fi\n+\n+if test \"$getret\" = 0\n+then\n+\ttrace_it \"$0: HIT for '$arg_file' with '$arg_sp'\"\n+\texit 0\n+else\n+\ttrace_it \"$0: MISS: for '$arg_file' with '$arg_sp'\"\n+fi\n+\n+out=\"$(mktemp)\"\n+err=\"$(mktemp)\"\n+\n+set +e\n+\"$spatch\" \"$@\" >\"$out\" 2>>\"$err\"\n+ret=$?\n+cat \"$out\"\n+cat \"$err\" >&2\n+set -e\n+\n+nocache=\n+if test $ret != 0\n+then\n+\tnocache=\"exited non-zero: $ret\"\n+elif test -s \"$out\"\n+then\n+\tnocache=\"had patch output\"\n+elif test -z \"$cacheWhenStderr\" && test -s \"$err\"\n+then\n+\tnocache=\"had stderr (use --very-quiet or spatchCache.cacheWhenStderr=true?)\"\n+fi\n+\n+if test -n \"$nocache\"\n+then\n+\ttrace_it \"$0: NOCACHE ($nocache): for '$arg_file' with '$arg_sp'\"\n+\texit \"$ret\"\n+fi\n+\n+trace_it \"$0: SET: for '$arg_file' with '$arg_sp'\"\n+\n+setret=\n+if test -z \"$set\"\n+then\n+\tif test $(redis-cli SADD spatch-cache \"$sum\") = 1\n+\tthen\n+\t\tsetret=0\n+\telse\n+\t\tsetret=1\n+\tfi\n+else\n+\t\"$set\" \"$sum\"\n+\tsetret=$?\n+fi\n+\n+if test \"$setret\" != 0\n+then\n+\techo \"FAILED to set '$sum' in cache!\" >&2\n+\texit 128\n+fi\n+\n+exit \"$ret\"\n-- \n2.38.0.1280.g8136eb6fab2\n\n"},{"id":"466958","messageId":"20221109145701.GD1731@szeder.dev","threadId":"58360","inReplyTo":"patch-v5-08.13-ee9ad433ab5-20221101T222616Z-avarab@gmail.com","subject":"Re: [PATCH v5 08/13] cocci: make \"coccicheck\" rule incremental","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2022-11-09T14:57:01Z","receivedAt":"2022-11-09T14:58:00Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Nov 01, 2022 at 11:35:50PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> Optimize the very slow \"coccicheck\" target to take advantage of\n> incremental rebuilding, and fix outstanding dependency problems with\n> the existing rule.\n> \n> The rule is now faster both on the initial run as we can make better\n> use of GNU make's parallelism than the old ad-hoc combination of\n> make's parallelism combined with $(SPATCH_BATCH_SIZE) and/or the\n> \"--jobs\" argument to \"spatch(1)\".\n> \n> It also makes us *much* faster when incrementally building, it's now\n> viable to \"make coccicheck\" as topic branches are merged down.\n> \n> The rule didn't use FORCE (or its equivalents) before, so a:\n> \n> \tmake coccicheck\n> \tmake coccicheck\n> \n> Would report nothing to do on the second iteration. But all of our\n> patch output depended on all $(COCCI_SOURCES) files, therefore e.g.:\n> \n> make -W grep.c coccicheck\n> \n> Would do a full re-run, i.e. a a change in a single file would force\n> us to do a full re-run.\n> \n> The reason for this (not the initial rationale, but my analysis) is:\n> \n> * Since we create a single \"*.cocci.patch+\" we don't know where to\n> pick up where we left off, or how to incrementally merge e.g. a\n> \"grep.c\" change with an existing *.cocci.patch.\n> \n> * We've been carrying forward the dependency on the *.c files since\n> 63f0a758a06 (add coccicheck make target, 2016-09-15) the rule was\n> initially added as a sort of poor man's dependency discovery.\n> \n> As we don't include other *.c files depending on other *.c files\n> has always been broken, as could be trivially demonstrated\n> e.g. with:\n> \n> make coccicheck\n> make -W strbuf.h coccicheck\n> \n> However, depending on the corresponding *.c files has been doing\n> something, namely that *if* an API change modified both *.c and *.h\n> files we'd catch the change to the *.h we care about via the *.c\n> being changed.\n> \n> For API changes that happened only via *.h files we'd do the wrong\n> thing before this change, but e.g. for function additions (not\n> \"static inline\" ones) catch the *.h change by proxy.\n> \n> Now we'll instead:\n> \n> * Create a / pair in the .build directory, E.g. for\n> swap.cocci and grep.c we'll create\n> .build/contrib/coccinelle/swap.cocci.patch/grep.c.\n\nIndeed, and the output of 'make coccicheck' with this patch looks like\nthis:\n\n MKDIR -p .build/contrib/coccinelle/hashmap.cocci.patch\n SPATCH .build/contrib/coccinelle/hashmap.cocci.patch/upload-pack.c\n SPATCH .build/contrib/coccinelle/hashmap.cocci.patch/merge-ort-wrappers.c\n SPATCH .build/contrib/coccinelle/hashmap.cocci.patch/unpack-trees.c\n SPATCH .build/contrib/coccinelle/hashmap.cocci.patch/gpg-interface.c\n SPATCH .build/contrib/coccinelle/hashmap.cocci.patch/linear-assignment.c\n\nThis is misleading, because it appears as 'spatch' would generate C\nsource files, but that's not the case, of course.\n\n> That file is the diff we'll apply for that -\n> combination, if there's no changes to me made (the common case)\n> it'll be an empty file.\n> \n> * Our generated *.patch\n> file (e.g. contrib/coccinelle/swap.cocci.patch) is now a simple \"cat\n> $^\" of all of all of the / files for a given .\n> \n> In the case discussed above of \"grep.c\" being changed we'll do the\n> full \"cat\" every time, so they resulting *.cocci.patch will always\n> be correct and up-to-date, even if it's \"incrementally updated\".\n> \n> See 1cc0425a27c (Makefile: have \"make pot\" not \"reset --hard\",\n> 2022-05-26) for another recent rule that used that technique.\n> \n> As before we'll:\n> \n> * End up generating a contrib/coccinelle/swap.cocci.patch, if we\n> \"fail\" by creating a non-empty patch we'll still exit with a zero\n> exit code.\n> \n> Arguably we should move to a more Makefile-native way of doing\n> this, i.e. fail early, and if we want all of the \"failed\" changes\n> we can use \"make -k\", but as the current\n> \"ci/run-static-analysis.sh\" expects us to behave this way let's\n> keep the existing behavior of exhaustively discovering all cocci\n> changes, and only failing if spatch itself errors out.\n> \n> Further implementation details & notes:\n> \n> * Before this change running \"make coccicheck\" would by default end\n> up pegging just one CPU at the very end for a while, usually as\n> we'd finish whichever *.cocci rule was the most expensive.\n> \n> This could be mitigated by combining \"make -jN\" with\n> SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch\n> spatch invocations, 2019-05-06).\n> \n> There will be cases where getting rid of \"SPATCH_BATCH_SIZE\" makes\n> things worse, but a from-scratch \"make coccicheck\" with the default\n> of SPATCH_BATCH_SIZE=1 (and tweaking it doesn't make a difference)\n> is faster (~3m36s v.s. ~3m56s) with this approach, as we can feed\n> the CPU more work in a less staggered way.\n> \n> * Getting rid of \"SPATCH_BATCH_SIZE\" particularly helps in cases\n> where the default of 1 yields parallelism under \"make coccicheck\",\n> but then running e.g.:\n> \n> make -W contrib/coccinelle/swap.cocci coccicheck\n> \n> I.e. before that would use only one CPU core, until the user\n> remembered to adjust \"SPATCH_BATCH_SIZE\" differently than the\n> setting that makes sense when doing a non-incremental run of \"make\n> coccicheck\".\n> \n> * Before the \"make coccicheck\" rule would have to clean\n> \"contrib/coccinelle/*.cocci.patch*\", since we'd create \"*+\" and\n> \"*.log\" files there. Now those are created in\n> .build/contrib/coccinelle/, which is covered by the \"cocciclean\" rule\n> already.\n> \n> Outstanding issues & future work:\n> \n> * We could get rid of \"--all-includes\" in favor of manually\n> specifying a list of includes to give to \"spatch(1)\".\n> \n> As noted upthread of [1] a naïve removal of \"--all-includes\" will\n> result in broken *.cocci patches, but if we know the exhaustive\n> list of includes via COMPUTE_HEADER_DEPENDENCIES we don't need to\n> re-scan for them, we could grab the headers to include from the\n> .depend.d/.o.d and supply them with the \"--include\" option to\n> spatch(1).q\n> \n> 1. https://lore.kernel.org/git/87ft18tcog.fsf@evledraar.gmail.com/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> Makefile | 81 ++++++++++++++++++++++++-----------\n> contrib/coccinelle/.gitignore | 2 +-\n> shared.mak | 3 +-\n> 3 files changed, 58 insertions(+), 28 deletions(-)\n> \n> diff --git a/Makefile b/Makefile\n> index 25f0c7e837d..4bedff26285 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1306,10 +1306,6 @@ SANITIZE_ADDRESS =\n> SPATCH_INCLUDE_FLAGS = --all-includes\n> SPATCH_FLAGS =\n> SPATCH_TEST_FLAGS =\n> -# Setting SPATCH_BATCH_SIZE higher will\n> -# usually result in less CPU usage at the cost of higher peak memory.\n> -# Setting it to 0 will feed all files in a single spatch invocation.\n> -SPATCH_BATCH_SIZE = 1\n> \n> # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change\n> TRACK_SPATCH_DEFINES =\n> @@ -1317,7 +1313,6 @@ TRACK_SPATCH_DEFINES += $(SPATCH)\n> TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS)\n> TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS)\n> TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS)\n> -TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE)\n> GIT-SPATCH-DEFINES: FORCE\n> \t@FLAGS='$(TRACK_SPATCH_DEFINES)'; \\\n> \t if test x\"$$FLAGS\" != x\"`cat GIT-SPATCH-DEFINES 2>/dev/null`\" ; then \\\n> @@ -3163,6 +3158,7 @@ check: $(GENERATED_H)\n> \n> COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci)\n> COCCI_RULES = $(COCCI_GLOB)\n> +COCCI_NAMES = $(COCCI_RULES:contrib/coccinelle/%.cocci=%)\n> \n> COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES))\n> COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n> @@ -3170,32 +3166,65 @@ COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n> COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n> COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n> \n> +# It's expensive to compute the many=many rules below, only eval them\n> +# on $(MAKECMDGOALS) that match these $(COCCI_RULES)\n> +COCCI_RULES_GLOB =\n> +COCCI_RULES_GLOB += cocci%\n> +COCCI_RULES_GLOB += .build/contrib/coccinelle/%\n> +COCCI_RULES_GLOB += $(COCCICHECK_PATCHES)\n> +COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING)\n> +COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS))\n> +\n> COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n> \n> -COCCI_PATCHES = $(COCCI_RULES:%=%.patch)\n> -$(COCCI_PATCHES): GIT-SPATCH-DEFINES\n> -$(COCCI_PATCHES): $(COCCI_SOURCES)\n> -$(COCCI_PATCHES): %.patch: %\n> -\t$(QUIET_SPATCH) \\\n> -\tif test $(SPATCH_BATCH_SIZE) = 0; then \\\n> -\t\tlimit=; \\\n> -\telse \\\n> -\t\tlimit='-n $(SPATCH_BATCH_SIZE)'; \\\n> -\tfi; \\\n> -\tif ! echo $(COCCI_SOURCES) | xargs $$limit \\\n> -\t\t$(SPATCH) $(SPATCH_FLAGS) \\\n> -\t\t$(SPATCH_INCLUDE_FLAGS) \\\n> -\t\t--sp-file $< --patch . \\\n> -\t\t>$@+ 2>$@.log; \\\n> +.build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES)\n> +\t$(call mkdir_p_parent_template)\n> +\t$(QUIET_GEN) >$@\n> +\n> +define cocci-rule\n> +\n> +## Rule for .build/$(1).patch/$(2); Params:\n> +# $(1) = e.g. \"free.cocci\"\n> +# $(2) = e.g. \"grep.c\"\n> +COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n> +.build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n> +.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES\n> +.build/$(1).patch/$(2): $(1)\n> +.build/$(1).patch/$(2): .build/$(1).patch/% : %\n> +\t$$(call mkdir_p_parent_template)\n> +\t$$(QUIET_SPATCH)if ! $$(SPATCH) $$(SPATCH_FLAGS) \\\n> +\t\t$$(SPATCH_INCLUDE_FLAGS) \\\n> +\t\t--sp-file $(1) --patch . $$< \\\n> +\t\t>$$@ 2>$$@.log; \\\n> \tthen \\\n> -\t\tcat $@.log; \\\n> +\t\techo \"ERROR when applying '$(1)' to '$$<'; '$$@.log' follows:\"; \\\n> +\t\tcat $$@.log; \\\n> \t\texit 1; \\\n> -\tfi; \\\n> -\tmv $@+ $@; \\\n> -\tif test -s $@; \\\n> +\tfi\n> +endef\n> +\n> +define cocci-matrix\n> +\n> +$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(1),$(s)))\n> +endef\n> +\n> +ifdef COCCI_GOALS\n> +$(eval $(foreach c,$(COCCI_RULES),$(call cocci-matrix,$(c))))\n> +endif\n> +\n> +define spatch-rule\n> +\n> +contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1))\n> +\t$$(QUIET_SPATCH_CAT)cat $$^ >$$@ && \\\n> +\tif test -s $$@; \\\n> \tthen \\\n> -\t\techo ' ' SPATCH result: $@; \\\n> +\t\techo ' ' SPATCH result: $$@; \\\n> \tfi\n> +endef\n> +\n> +ifdef COCCI_GOALS\n> +$(eval $(foreach n,$(COCCI_NAMES),$(call spatch-rule,$(n))))\n> +endif\n> \n> COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES))\n> $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES\n> @@ -3487,7 +3516,7 @@ profile-clean:\n> cocciclean:\n> \t$(RM) GIT-SPATCH-DEFINES\n> \t$(RM) -r .build/contrib/coccinelle\n> -\t$(RM) contrib/coccinelle/*.cocci.patch*\n> +\t$(RM) contrib/coccinelle/*.cocci.patch\n> \n> clean: profile-clean coverage-clean cocciclean\n> \t$(RM) -r .build\n> diff --git a/contrib/coccinelle/.gitignore b/contrib/coccinelle/.gitignore\n> index d3f29646dc3..1d45c0a40c8 100644\n> --- a/contrib/coccinelle/.gitignore\n> +++ b/contrib/coccinelle/.gitignore\n> @@ -1 +1 @@\n> -*.patch*\n> +*.patch\n> diff --git a/shared.mak b/shared.mak\n> index 96b06acc455..f437073e48c 100644\n> --- a/shared.mak\n> +++ b/shared.mak\n> @@ -71,8 +71,9 @@ ifndef V\n> \tQUIET_RC = @echo ' ' RC $@;\n> \n> ## Used in \"Makefile\": SPATCH\n> -\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $<;\n> +\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $@;\n> \tQUIET_SPATCH_TEST\t\t= @echo ' ' SPATCH TEST $(@:.build/%=%);\n> +\tQUIET_SPATCH_CAT\t\t= @echo ' ' SPATCH CAT $$^ \\>$@;\n> \n> ## Used in \"Documentation/Makefile\"\n> \tQUIET_ASCIIDOC\t= @echo ' ' ASCIIDOC $@;\n> -- \n> 2.38.0.1280.g8136eb6fab2\n> \n"},{"id":"466960","messageId":"20221109150556.GE1731@szeder.dev","threadId":"58360","inReplyTo":"patch-v5-10.13-56ca8f5720a-20221101T222616Z-avarab@gmail.com","subject":"Re: [PATCH v5 10/13] Makefile: copy contrib/coccinelle/*.cocci to build/","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2022-11-09T15:05:56Z","receivedAt":"2022-11-09T15:06:05Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Nov 01, 2022 at 11:35:52PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> Change the \"coccinelle\" rule so that we first copy the *.cocci source\n> in e.g. \"contrib/coccinelle/strbuf.cocci\" to\n> \".build/contrib/coccinelle/strbuf.cocci\" before operating on it.\n\nAfter this patch the output of 'make coccicheck' looks like this:\n\n CP contrib/coccinelle/hashmap.cocci .build/contrib/coccinelle/hashmap.cocci\n MKDIR -p .build/.build/contrib/coccinelle/hashmap.cocci.patch\n SPATCH .build/.build/contrib/coccinelle/hashmap.cocci.patch/upload-pack.c\n SPATCH .build/.build/contrib/coccinelle/hashmap.cocci.patch/merge-ort-wrappers.c\n SPATCH .build/.build/contrib/coccinelle/hashmap.cocci.patch/unpack-trees.c\n SPATCH .build/.build/contrib/coccinelle/hashmap.cocci.patch/gpg-interface.c\n SPATCH .build/.build/contrib/coccinelle/hashmap.cocci.patch/linear-assignment.c\n\nNotice how there is not one but two leading '.build' path components.\nSurely one would be enough :)\n\nThis also breaks 'make cocciclean':\n\n $ make cocciclean\n rm -f GIT-SPATCH-DEFINES\n rm -f -r .build/contrib/coccinelle\n rm -f contrib/coccinelle/*.cocci.patch\n $ find .build/\n .build/\n .build/contrib\n .build/.build\n .build/.build/contrib\n .build/.build/contrib/coccinelle\n .build/.build/contrib/coccinelle/hashmap.cocci.patch\n .build/.build/contrib/coccinelle/hashmap.cocci.patch/upload-pack.c\n .build/.build/contrib/coccinelle/hashmap.cocci.patch/merge-ort-wrappers.c\n .build/.build/contrib/coccinelle/hashmap.cocci.patch/unpack-trees.c\n .build/.build/contrib/coccinelle/hashmap.cocci.patch/gpg-interface.c\n .build/.build/contrib/coccinelle/hashmap.cocci.patch/linear-assignment.c\n\n\n> For now this serves as a rather pointless indirection, but prepares us\n> for the subsequent commit where we'll be able to inject generated\n> *.cocci files. Having the entire dependency tree live inside .build/*\n> simplifies both the globbing we'd need to do, and any \"clean\" rules.\n> \n> It will also help for future targets which will want to act on the\n> generated patches or the logs, e.g. targets to alert if we can't parse\n> certain files (or, less so than usual) with \"spatch\", and e.g. a\n> replacement for \"ci/run-static-analysis.sh\". Such a replacement won't\n> care about placing the patches in the in-tree, only whether they're\n> \"OK\" (and about the diff).\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> Makefile | 27 +++++++++++++++++++++------\n> shared.mak | 1 +\n> 2 files changed, 22 insertions(+), 6 deletions(-)\n> \n> diff --git a/Makefile b/Makefile\n> index c7c96d284dd..44c906b65d5 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -3164,8 +3164,11 @@ check: $(GENERATED_H)\n> \tfi\n> \n> COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci)\n> -COCCI_RULES = $(COCCI_GLOB)\n> -COCCI_NAMES = $(COCCI_RULES:contrib/coccinelle/%.cocci=%)\n> +COCCI_RULES_TRACKED = $(COCCI_GLOB:%=.build/%)\n> +COCCI_RULES =\n> +COCCI_RULES += $(COCCI_RULES_TRACKED)\n> +COCCI_NAMES =\n> +COCCI_NAMES += $(COCCI_RULES:.build/contrib/coccinelle/%.cocci=%)\n> \n> COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES))\n> COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n> @@ -3173,6 +3176,9 @@ COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES))\n> COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch)\n> COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch)\n> \n> +COCCICHECK_PATCHES_INTREE = $(COCCICHECK_PATCHES:.build/%=%)\n> +COCCICHECK_PATCHES_PENDING_INTREE = $(COCCICHECK_PATCHES_PENDING:.build/%=%)\n> +\n> # It's expensive to compute the many=many rules below, only eval them\n> # on $(MAKECMDGOALS) that match these $(COCCI_RULES)\n> COCCI_RULES_GLOB =\n> @@ -3180,10 +3186,16 @@ COCCI_RULES_GLOB += cocci%\n> COCCI_RULES_GLOB += .build/contrib/coccinelle/%\n> COCCI_RULES_GLOB += $(COCCICHECK_PATCHES)\n> COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING)\n> +COCCI_RULES_GLOB += $(COCCICHECK_PATCHES_INTREE)\n> +COCCI_RULES_GLOB += $(COCCICHECK_PATCHES_PENDING_INTREE)\n> COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS))\n> \n> COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)\n> \n> +$(COCCI_RULES_TRACKED): .build/% : %\n> +\t$(call mkdir_p_parent_template)\n> +\t$(QUIET_CP)cp $< $@\n> +\n> .build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES)\n> \t$(call mkdir_p_parent_template)\n> \t$(QUIET_GEN) >$@\n> @@ -3197,7 +3209,7 @@ define cocci-rule\n> # $(1) = e.g. \"free.cocci\"\n> # $(2) = e.g. \"grep.c\"\n> # $(3) = e.g. \"grep.o\"\n> -COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n> +COCCI_$(1:.build/contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n> .build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n> .build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES)\n> .build/$(1).patch/$(2): $(1)\n> @@ -3225,12 +3237,15 @@ endif\n> \n> define spatch-rule\n> \n> -contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1))\n> +.build/contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1))\n> \t$$(QUIET_SPATCH_CAT)cat $$^ >$$@ && \\\n> \tif test -s $$@; \\\n> \tthen \\\n> \t\techo ' ' SPATCH result: $$@; \\\n> \tfi\n> +contrib/coccinelle/$(1).cocci.patch: .build/contrib/coccinelle/$(1).cocci.patch\n> +\t$$(QUIET_CP)cp $$< $$@\n> +\n> endef\n> \n> ifdef COCCI_GOALS\n> @@ -3254,11 +3269,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell\n> coccicheck-test: $(COCCI_TEST_RES_GEN)\n> \n> coccicheck: coccicheck-test\n> -coccicheck: $(COCCICHECK_PATCHES)\n> +coccicheck: $(COCCICHECK_PATCHES_INTREE)\n> \n> # See contrib/coccinelle/README\n> coccicheck-pending: coccicheck-test\n> -coccicheck-pending: $(COCCICHECK_PATCHES_PENDING)\n> +coccicheck-pending: $(COCCICHECK_PATCHES_PENDING_INTREE)\n> \n> .PHONY: coccicheck coccicheck-pending\n> \n> diff --git a/shared.mak b/shared.mak\n> index f437073e48c..a34b66c926d 100644\n> --- a/shared.mak\n> +++ b/shared.mak\n> @@ -60,6 +60,7 @@ ifndef V\n> \tQUIET_AR = @echo ' ' AR $@;\n> \tQUIET_LINK = @echo ' ' LINK $@;\n> \tQUIET_BUILT_IN = @echo ' ' BUILTIN $@;\n> +\tQUIET_CP = @echo ' ' CP $< $@;\n> \tQUIET_LNCP = @echo ' ' LN/CP $@;\n> \tQUIET_XGETTEXT = @echo ' ' XGETTEXT $@;\n> \tQUIET_MSGINIT = @echo ' ' MSGINIT $@;\n> -- \n> 2.38.0.1280.g8136eb6fab2\n> \n"},{"id":"466963","messageId":"221109.86tu38p1x8.gmgdl@evledraar.gmail.com","threadId":"58360","inReplyTo":"20221109150556.GE1731@szeder.dev","subject":"Re: [PATCH v5 10/13] Makefile: copy contrib/coccinelle/*.cocci to build/","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-09T15:42:33Z","receivedAt":"2022-11-09T15:45:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Nov 09 2022, SZEDER Gábor wrote:\n\n> On Tue, Nov 01, 2022 at 11:35:52PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> Change the \"coccinelle\" rule so that we first copy the *.cocci source\n>> in e.g. \"contrib/coccinelle/strbuf.cocci\" to\n>> \".build/contrib/coccinelle/strbuf.cocci\" before operating on it.\n>\n> After this patch the output of 'make coccicheck' looks like this:\n>\n> CP contrib/coccinelle/hashmap.cocci .build/contrib/coccinelle/hashmap.cocci\n> MKDIR -p .build/.build/contrib/coccinelle/hashmap.cocci.patch\n> SPATCH .build/.build/contrib/coccinelle/hashmap.cocci.patch/upload-pack.c\n> SPATCH .build/.build/contrib/coccinelle/hashmap.cocci.patch/merge-ort-wrappers.c\n> SPATCH .build/.build/contrib/coccinelle/hashmap.cocci.patch/unpack-trees.c\n> SPATCH .build/.build/contrib/coccinelle/hashmap.cocci.patch/gpg-interface.c\n> SPATCH .build/.build/contrib/coccinelle/hashmap.cocci.patch/linear-assignment.c\n>\n> Notice how there is not one but two leading '.build' path components.\n> Surely one would be enough :)\n\nOops, well spotted, I'll submit a patch on top soon to fix that...\n\n> This also breaks 'make cocciclean':\n>\n> $ make cocciclean\n> rm -f GIT-SPATCH-DEFINES\n> rm -f -r .build/contrib/coccinelle\n> rm -f contrib/coccinelle/*.cocci.patch\n> $ find .build/\n> .build/\n> .build/contrib\n> .build/.build\n> .build/.build/contrib\n> .build/.build/contrib/coccinelle\n> .build/.build/contrib/coccinelle/hashmap.cocci.patch\n> .build/.build/contrib/coccinelle/hashmap.cocci.patch/upload-pack.c\n> .build/.build/contrib/coccinelle/hashmap.cocci.patch/merge-ort-wrappers.c\n> .build/.build/contrib/coccinelle/hashmap.cocci.patch/unpack-trees.c\n> .build/.build/contrib/coccinelle/hashmap.cocci.patch/gpg-interface.c\n> .build/.build/contrib/coccinelle/hashmap.cocci.patch/linear-assignment.c\n\n...and that, i.e. the nested .build is clearly unintended...\n\nAside: Now that \"coccicheck\" is a well-behaved (well, mostly, sans the\nabove) target that knows its deps etc. I wonder if it makes sense to\nhave it clean this at all, and just leave it for \"make clean\". I.e. it\nshould clean the worktree litter, but we could just leave the\n\".build/contrib/coccinelle\".\n\nAnyway, that's just a thought, and something to leave for some later\n\"what should we clean in .build\" topic, if any. I'll make sure it rm\n-rf's the right .build/ dir, and that we put stuff in it...\n"},{"id":"467050","messageId":"patch-1.1-5bdb8deb0c7-20221110T161232Z-avarab@gmail.com","threadId":"58360","inReplyTo":"221109.86tu38p1x8.gmgdl@evledraar.gmail.com","subject":"[PATCH] Makefile: don't create a \".build/.build/\" for cocci, fix output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-10T16:14:18Z","receivedAt":"2022-11-10T16:14:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a couple of issues in the recently merged 0f3c55d4c2b (Merge\nbranch 'ab/coccicheck-incremental' into next, 2022-11-08):\n\nIn copying over the \"contrib/coccinelle/\" rules to\n\".build/contrib/coccinelle/\" we inadvertently ended up with a\n\".build/.build/contrib/coccinelle/\" as well. We'd generate the\nper-file patches in the former, and keep the rule and overall result\nin the latter. E.g. running:\n\n\tmake contrib/coccinelle/free.cocci.patch COCCI_SOURCES=\"attr.c grep.c\"\n\nWould, per \"tree -a .build\" yield the following result:\n\n\t.build\n\t├── .build\n\t│   └── contrib\n\t│   └── coccinelle\n\t│   └── free.cocci.patch\n\t│   ├── attr.c\n\t│   ├── attr.c.log\n\t│   ├── grep.c\n\t│   └── grep.c.log\n\t└── contrib\n\t └── coccinelle\n\t ├── FOUND_H_SOURCES\n\t ├── free.cocci\n\t └── free.cocci.patch\n\nNow we'll instead generate all of our files in\n\".build/contrib/coccinelle/\". Fixing this required renaming the\ndirectory where we keep our per-file patches, as we'd otherwise\nconflict with the result.\n\nNow the per-file patch directory is named e.g. \"free.cocci.d\". And the\nend result will now be:\n\n\t.build\n\t└── contrib\n\t └── coccinelle\n\t ├── FOUND_H_SOURCES\n\t ├── free.cocci\n\t ├── free.cocci.d\n\t │   ├── attr.c.patch\n\t │   ├── attr.c.patch.log\n\t │   ├── grep.c.patch\n\t │   └── grep.c.patch.log\n\t └── free.cocci.patch\n\nThe per-file patches now have a \".patch\" file suffix, which fixes\nanother issue reported against 0f3c55d4c2b: The summary output was\nconfusing. Before for the \"make\" command above we'd emit:\n\n\t[...]\n\tMKDIR -p .build/contrib/coccinelle\n\tCP contrib/coccinelle/free.cocci .build/contrib/coccinelle/free.cocci\n\tGEN .build/contrib/coccinelle/FOUND_H_SOURCES\n\tMKDIR -p .build/.build/contrib/coccinelle/free.cocci.patch\n\tSPATCH .build/.build/contrib/coccinelle/free.cocci.patch/grep.c\n\tSPATCH .build/.build/contrib/coccinelle/free.cocci.patch/attr.c\n\tSPATCH CAT $^ >.build/contrib/coccinelle/free.cocci.patch\n\tCP .build/contrib/coccinelle/free.cocci.patch contrib/coccinelle/free.cocci.patch\n\nBut now we'll instead emit (identical output at the start omitted):\n\n\t[...]\n\tMKDIR -p .build/contrib/coccinelle/free.cocci.d\n\tSPATCH grep.c >.build/contrib/coccinelle/free.cocci.d/grep.c.patch\n\tSPATCH attr.c >.build/contrib/coccinelle/free.cocci.d/attr.c.patch\n\tSPATCH CAT .build/contrib/coccinelle/free.cocci.d/**.patch >.build/contrib/coccinelle/free.cocci.patch\n\tCP .build/contrib/coccinelle/free.cocci.patch contrib/coccinelle/free.cocci.patch\n\nI.e. we have an \"SPATCH\" line that makes it clear that we're running\nagainst the \"{attr,grep}.c\" file. The \"SPATCH CAT\" is then altered to\ncorrespond to it, showing that we're concatenating the\n\"free.cocci.d/**.patch\" files into one generated \"free.cocci.patch\" at\nthe end.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n\nThis fixes the issues SZEDER reported against the\nab/coccicheck-incremental topic (now in \"next\") in\nhttps://lore.kernel.org/git/20221109150556.GE1731@szeder.dev/ and\nhttps://lore.kernel.org/git/20221109145701.GD1731@szeder.dev/\n\n Makefile | 12 ++++++------\n shared.mak | 4 ++--\n 2 files changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 36cefe3b1d6..15b29652221 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3236,14 +3236,14 @@ endif\n define cocci-rule\n \n ## Rule for .build/$(1).patch/$(2); Params:\n-# $(1) = e.g. \"free.cocci\"\n+# $(1) = e.g. \".build/contrib/coccinelle/free.cocci\"\n # $(2) = e.g. \"grep.c\"\n # $(3) = e.g. \"grep.o\"\n-COCCI_$(1:.build/contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)\n-.build/$(1).patch/$(2): GIT-SPATCH-DEFINES\n-.build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES)\n-.build/$(1).patch/$(2): $(1)\n-.build/$(1).patch/$(2): .build/$(1).patch/% : %\n+COCCI_$(1:.build/contrib/coccinelle/%.cocci=%) += $(1).d/$(2).patch\n+$(1).d/$(2).patch: GIT-SPATCH-DEFINES\n+$(1).d/$(2).patch: $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES)\n+$(1).d/$(2).patch: $(1)\n+$(1).d/$(2).patch: $(1).d/%.patch : %\n \t$$(call mkdir_p_parent_template)\n \t$$(QUIET_SPATCH)if ! $$(SPATCH) $$(SPATCH_FLAGS) \\\n \t\t$$(SPATCH_INCLUDE_FLAGS) \\\ndiff --git a/shared.mak b/shared.mak\nindex a34b66c926d..be1f30ff206 100644\n--- a/shared.mak\n+++ b/shared.mak\n@@ -72,9 +72,9 @@ ifndef V\n \tQUIET_RC = @echo ' ' RC $@;\n \n ## Used in \"Makefile\": SPATCH\n-\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $@;\n+\tQUIET_SPATCH\t\t\t= @echo ' ' SPATCH $< \\>$@;\n \tQUIET_SPATCH_TEST\t\t= @echo ' ' SPATCH TEST $(@:.build/%=%);\n-\tQUIET_SPATCH_CAT\t\t= @echo ' ' SPATCH CAT $$^ \\>$@;\n+\tQUIET_SPATCH_CAT\t\t= @echo ' ' SPATCH CAT $(@:%.patch=%.d/)\\*\\*.patch \\>$@;\n \n ## Used in \"Documentation/Makefile\"\n \tQUIET_ASCIIDOC\t= @echo ' ' ASCIIDOC $@;\n-- \n2.38.0.1465.g0d50a1dab40\n\n"},{"id":"467166","messageId":"Y27LMXTCF7o1Vv1i@nand.local","threadId":"58360","inReplyTo":"patch-1.1-5bdb8deb0c7-20221110T161232Z-avarab@gmail.com","subject":"Re: [PATCH] Makefile: don't create a \".build/.build/\" for cocci, fix output","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-11-11T22:22:41Z","receivedAt":"2022-11-11T22:23:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Nov 10, 2022 at 05:14:18PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> Fix a couple of issues in the recently merged 0f3c55d4c2b (Merge\n> branch 'ab/coccicheck-incremental' into next, 2022-11-08):\n\nThanks for coming back for these. Will queue this on top of\nab/coccicheck-incremental and re-merge back into 'next'.\n\n\nThanks,\nTaylor\n"}]}