git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 09/11] cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Oct 14, 2022, 15:31 UTC
Message-ID
<patch-v3-09.11-739652eada9-20221014T152553Z-avarab@gmail.com>
In-Reply-To
<cover-v3-00.11-00000000000-20221014T152552Z-avarab@gmail.com>

Improve the incremental rebuilding support of "coccicheck" by piggy-backing on the computed dependency information of the corresponding *.o file, rather than rebuilding all <RULE>/<FILE> pairs if either their corresponding file changes, or if any header changes.

This in effect uses the same method that the "sparse" target was made to use in c234e8a0ecf (Makefile: make the "sparse" target non-.PHONY, 2021-09-23), except that the dependency on the *.o file isn't a hard one, we check with $(wildcard) if the *.o file exists, and if so we'll depend on it.

This means that the common case of:
	make
	make coccicheck

Will benefit from incremental rebuilding, now changing e.g. a header will only re-run "spatch" on those those *.c files that make use of it:

By depending on the *.o we piggy-back on COMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the "sparse" target non-.PHONY, 2021-09-23) for prior art of doing that for the *.sp files. E.g.:

    make contrib/coccinelle/free.cocci.patch
    make -W column.h contrib/coccinelle/free.cocci.patch

Will take around 15 seconds for the second command on my 8 core box if I didn't run "make" beforehand to create the *.o files. But around 2 seconds if I did and we have those "*.o" files.

Notes about the approach of piggy-backing on *.o for dependencies:
 * It *is* a trade-off since we'll pay the extra cost of running the C
   compiler, but we're probably doing that anyway. The compiler is much
   faster than "spatch", so even though we need to re-compile the *.o to
   create the dependency info for the *.c for "spatch" it's
   faster (especially if using "ccache").
 * There *are* use-cases where some would like to have *.o files
   around, but to have the "make coccicheck" ignore them. See:
   https://lore.kernel.org/git/20220826104312.GJ1735@szeder.dev/
   For those users a:
	make
	make coccicheck SPATCH_USE_O_DEPENDENCIES=
   Will avoid considering the *.o files.
 * If that *.o file doesn't exist we'll depend on an intermediate file
   of ours which in turn depends on $(FOUND_H_SOURCES).
   This covers both an initial build, or where "coccicheck" is run
   without running "all" beforehand, and because we run "coccicheck"
   on e.g. files in compat/* that we don't know how to build unless
   the requisite flag was provided to the Makefile.
   Most of the runtime of "incremental" runs is now spent on various
   compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and
   therefore conflate whether we *can* compile an object and generate
   dependency information for it with whether we'd like to link it
   into our binary.
   Before this change the distinction didn't matter, but now one way
   to make this even faster on incremental builds would be to peel
   those concerns apart so that we can see that e.g. compat/mmap.c
   doesn't depend on column.h.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Makefile | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index 4e1da7206db..bd3b5c21d6f 100644
--- a/Makefile
+++ b/Makefile
@@ -1303,6 +1303,13 @@ SPATCH_INCLUDE_FLAGS = --all-includes
 SPATCH_FLAGS =
 SPATCH_TEST_FLAGS =
 
+# If *.o files are present, have "coccicheck" depend on them, with
+# COMPUTE_HEADER_DEPENDENCIES this will speed up the common-case of
+# only needing to re-generate coccicheck results for the users of a
+# given API if it's changed, and not all files in the project. If
+# COMPUTE_HEADER_DEPENDENCIES=no this will be unset too.
+SPATCH_USE_O_DEPENDENCIES = YesPlease
+
 # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change
 TRACK_SPATCH_DEFINES =
 TRACK_SPATCH_DEFINES += $(SPATCH)
@@ -3175,14 +3182,18 @@ COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res)
 	$(call mkdir_p_parent_template)
 	$(QUIET_GEN) >$@
 
+ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no)
+SPATCH_USE_O_DEPENDENCIES =
+endif
 define cocci-rule
 
 ## Rule for .build/$(1).patch/$(2); Params:
 # $(1) = e.g. "free.cocci"
 # $(2) = e.g. "grep.c"
+# $(3) = e.g. "grep.o"
 COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2)
 .build/$(1).patch/$(2): GIT-SPATCH-DEFINES
-.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES
+.build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES)
 .build/$(1).patch/$(2): $(1)
 .build/$(1).patch/$(2): .build/$(1).patch/% : %
 	$$(call mkdir_p_parent_template)
@@ -3199,7 +3210,7 @@ endef
 
 define cocci-matrix
 
-$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(1),$(s)))
+$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o)))
 endef
 
 ifdef COCCI_GOALS
-- 
2.38.0.1092.g8c0298861b0
Previous: Ævar Arnfjörð BjarmasonNext: Ævar Arnfjörð Bjarmason
Message 33 of 72 in “cocci: make "incremental" possible + a ccache-like tool”
  1. 0/5 cocci: make "incremental" possible + a ccache-like toolÆvar Arnfjörð Bjarmason, Aug 25, 2022
  2. 1/5 Makefile: add ability to TAB-complete cocci *.patch rulesÆvar Arnfjörð Bjarmason, Aug 25, 2022
  3. 2/5 Makefile: have "coccicheck" re-run if flags changeÆvar Arnfjörð Bjarmason, Aug 25, 2022
  4. SZEDER GáborAug 25, 2022
  5. 4/5 cocci: make incremental compilation even fasterÆvar Arnfjörð Bjarmason, Aug 25, 2022
  6. 3/5 cocci: make "coccicheck" rule incrementalÆvar Arnfjörð Bjarmason, Aug 25, 2022
  7. SZEDER GáborAug 25, 2022
  8. Ævar Arnfjörð BjarmasonAug 25, 2022
  9. SZEDER GáborAug 26, 2022
  10. 5/5 spatchcache: add a ccache-alike for "spatch"Ævar Arnfjörð Bjarmason, Aug 25, 2022
  11. 0/9 cocci: make "incremental" possible + a ccache-like toolÆvar Arnfjörð Bjarmason, Aug 31, 2022
  12. 2/9 Makefile: add ability to TAB-complete cocci *.patch rulesÆvar Arnfjörð Bjarmason, Aug 31, 2022
  13. 3/9 Makefile: have "coccicheck" re-run if flags changeÆvar Arnfjörð Bjarmason, Aug 31, 2022
  14. 1/9 cocci rules: remove unused "F" metavariable from pending ruleÆvar Arnfjörð Bjarmason, Aug 31, 2022
  15. 4/9 Makefile: split off SPATCH_BATCH_SIZE comment from "cocci" headingÆvar Arnfjörð Bjarmason, Aug 31, 2022
  16. 5/9 cocci: split off include-less "tests" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Aug 31, 2022
  17. 6/9 cocci: split off "--all-includes" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Aug 31, 2022
  18. 8/9 cocci: optimistically use COMPUTE_HEADER_DEPENDENCIESÆvar Arnfjörð Bjarmason, Aug 31, 2022
  19. 7/9 cocci: make "coccicheck" rule incrementalÆvar Arnfjörð Bjarmason, Aug 31, 2022
  20. SZEDER GáborSep 1, 2022
  21. Ævar Arnfjörð BjarmasonSep 1, 2022
  22. 9/9 spatchcache: add a ccache-alike for "spatch"Ævar Arnfjörð Bjarmason, Aug 31, 2022
  23. 00/11 cocci: make "incremental" possible + a ccache-like toolÆvar Arnfjörð Bjarmason, Oct 14, 2022
  24. 01/11 Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)Ævar Arnfjörð Bjarmason, Oct 14, 2022
  25. 02/11 cocci rules: remove unused "F" metavariable from pending ruleÆvar Arnfjörð Bjarmason, Oct 14, 2022
  26. 05/11 Makefile: split off SPATCH_BATCH_SIZE comment from "cocci" headingÆvar Arnfjörð Bjarmason, Oct 14, 2022
  27. Taylor BlauOct 14, 2022
  28. 03/11 Makefile: add ability to TAB-complete cocci *.patch rulesÆvar Arnfjörð Bjarmason, Oct 14, 2022
  29. 07/11 cocci: split off "--all-includes" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Oct 14, 2022
  30. 04/11 Makefile: have "coccicheck" re-run if flags changeÆvar Arnfjörð Bjarmason, Oct 14, 2022
  31. 06/11 cocci: split off include-less "tests" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Oct 14, 2022
  32. 08/11 cocci: make "coccicheck" rule incrementalÆvar Arnfjörð Bjarmason, Oct 14, 2022
  33. 09/11 cocci: optimistically use COMPUTE_HEADER_DEPENDENCIESÆvar Arnfjörð Bjarmason, Oct 14, 2022
  34. 11/11 spatchcache: add a ccache-alike for "spatch"Ævar Arnfjörð Bjarmason, Oct 14, 2022
  35. 10/11 cocci: run against a generated ALL.cocciÆvar Arnfjörð Bjarmason, Oct 14, 2022
  36. Jeff KingOct 17, 2022
  37. Ævar Arnfjörð BjarmasonOct 17, 2022
  38. Junio C HamanoOct 17, 2022
  39. Jeff KingOct 17, 2022
  40. 00/12 cocci: make "incremental" possible + a ccache-like toolÆvar Arnfjörð Bjarmason, Oct 26, 2022
  41. 01/12 Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)Ævar Arnfjörð Bjarmason, Oct 26, 2022
  42. 02/12 cocci rules: remove unused "F" metavariable from pending ruleÆvar Arnfjörð Bjarmason, Oct 26, 2022
  43. 03/12 Makefile: add ability to TAB-complete cocci *.patch rulesÆvar Arnfjörð Bjarmason, Oct 26, 2022
  44. 04/12 Makefile: have "coccicheck" re-run if flags changeÆvar Arnfjörð Bjarmason, Oct 26, 2022
  45. 05/12 Makefile: split off SPATCH_BATCH_SIZE comment from "cocci" headingÆvar Arnfjörð Bjarmason, Oct 26, 2022
  46. 07/12 cocci: split off "--all-includes" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Oct 26, 2022
  47. 06/12 cocci: split off include-less "tests" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Oct 26, 2022
  48. 08/12 cocci: make "coccicheck" rule incrementalÆvar Arnfjörð Bjarmason, Oct 26, 2022
  49. 09/12 cocci: optimistically use COMPUTE_HEADER_DEPENDENCIESÆvar Arnfjörð Bjarmason, Oct 26, 2022
  50. 10/12 cocci rules: remove <id>'s from rules that don't need themÆvar Arnfjörð Bjarmason, Oct 26, 2022
  51. 11/12 cocci: run against a generated ALL.cocciÆvar Arnfjörð Bjarmason, Oct 26, 2022
  52. SZEDER GáborOct 28, 2022
  53. 12/12 spatchcache: add a ccache-alike for "spatch"Ævar Arnfjörð Bjarmason, Oct 26, 2022
  54. 00/13 cocci: make "incremental" possible + a ccache-like toolÆvar Arnfjörð Bjarmason, Nov 1, 2022
  55. 01/13 Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)Ævar Arnfjörð Bjarmason, Nov 1, 2022
  56. 02/13 cocci rules: remove unused "F" metavariable from pending ruleÆvar Arnfjörð Bjarmason, Nov 1, 2022
  57. 03/13 Makefile: add ability to TAB-complete cocci *.patch rulesÆvar Arnfjörð Bjarmason, Nov 1, 2022
  58. 04/13 Makefile: have "coccicheck" re-run if flags changeÆvar Arnfjörð Bjarmason, Nov 1, 2022
  59. 06/13 cocci: split off include-less "tests" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Nov 1, 2022
  60. 05/13 Makefile: split off SPATCH_BATCH_SIZE comment from "cocci" headingÆvar Arnfjörð Bjarmason, Nov 1, 2022
  61. 07/13 cocci: split off "--all-includes" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Nov 1, 2022
  62. 08/13 cocci: make "coccicheck" rule incrementalÆvar Arnfjörð Bjarmason, Nov 1, 2022
  63. SZEDER GáborNov 9, 2022
  64. 10/13 Makefile: copy contrib/coccinelle/*.cocci to build/Ævar Arnfjörð Bjarmason, Nov 1, 2022
  65. SZEDER GáborNov 9, 2022
  66. Ævar Arnfjörð BjarmasonNov 9, 2022
  67. Makefile: don't create a ".build/.build/" for cocci, fix outputÆvar Arnfjörð Bjarmason, Nov 10, 2022
  68. Taylor BlauNov 11, 2022
  69. 09/13 cocci: optimistically use COMPUTE_HEADER_DEPENDENCIESÆvar Arnfjörð Bjarmason, Nov 1, 2022
  70. 11/13 cocci rules: remove <id>'s from rules that don't need themÆvar Arnfjörð Bjarmason, Nov 1, 2022
  71. 12/13 cocci: run against a generated ALL.cocciÆvar Arnfjörð Bjarmason, Nov 1, 2022
  72. 13/13 spatchcache: add a ccache-alike for "spatch"Ævar Arnfjörð Bjarmason, Nov 1, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.