git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v4 2/4] Makefile/coccicheck: speed up and fix bug with duplicate hunks

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Mar 22, 2021, 12:11 UTC
Message-ID
<3bca3239cb84488a1638f2bb269392f47f78c6da.1616414951.git.avarab@gmail.com>
In-Reply-To
<cover.1616414951.git.avarab@gmail.com>

Change the coccicheck target to run on all of our *.c and *.h files with --include-headers-for-types, instead of trusting it to find *.h files and other includes to modify from its recursive walking of includes as it has been doing with only --all-includes.

The --all-includes option introduced in a9a884aea57 (coccicheck: use --all-includes by default, 2016-09-30) is needed because we have rules that e.g. use the "type T" construct that wouldn't match unless we scoured our headers for the definition of the relevant type.

But combining --all-includes it with processing N files at a time with xargs as we've done since 960154b9c17 (coccicheck: optionally batch spatch invocations, 2019-05-06) introduced a subtle bug with duplicate hunks being written to the generated *.patch files. I did not dig down to the root cause of that, but I think it's due to spatch doing (and failing to do) some magical locking/racy mtime checking to decide if it emits a hunk. See [1] for a way to reproduce the issue, and a discussion of it.

This change speeds up the runtime of "make -j8 coccicheck" from ~1m50s to ~1m20s for me. See [2] for more timings.

We could also use --no-includes for a runtime of ~55s, but that would produce broken patches (we miss some hunks) in cases where we need the type or other definitions from included files.

If anyone cares there's probably an optimization opportunity here to e.g. detect that the whole file doesn't need to consider includes, since the rules would be unambiguous without considering them.

Note on portability: The --include-headers-for-types option is not in my "man spatch", but it's been part of spatch since 2014. See its fe3a327a (include headers for types option, 2014-07-27), it was first released with version 1.0.0 of spatch, released in April 2015. The spatch developers are just inconsistent about updating their TeX docs and manpages at the same time.

1. https://lore.kernel.org/git/20210305170724.23859-3-avarab@gmail.com/
2. https://lore.kernel.org/git/20210306192525.15197-1-avarab@gmail.com/
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Makefile | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index eef99b4705d..e43a9618df5 100644
--- a/Makefile
+++ b/Makefile
@@ -1199,7 +1199,8 @@ SPARSE_FLAGS ?=
 SP_EXTRA_FLAGS = -Wno-universal-initializer
 
 # For the 'coccicheck' target
-SPATCH_FLAGS = --all-includes --patch .
+SPATCH_FLAGS = --all-includes --include-headers-for-types --patch .
+
 # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will
 # usually result in less CPU usage at the cost of higher peak memory.
 # Setting it to 0 will feed all files in a single spatch invocation.
@@ -2860,7 +2861,7 @@ check: config-list.h command-list.h
 		exit 1; \
 	fi
 
-FOUND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
+FOUND_C_SOURCES = $(filter %.c %.h,$(shell $(FIND_SOURCE_FILES)))
 COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))
 
 %.cocci.patch: %.cocci $(COCCI_SOURCES)
-- 
2.31.0.285.gb40d23e604f
Previous: Jeff KingNext: René Scharfe.
Message 73 of 103 in “*.h: remove extern from function declarations”
  1. *.h: remove extern from function declarationsDenton Liu, Apr 12, 2019
  2. Jeff KingApr 13, 2019
  3. Junio C HamanoApr 13, 2019
  4. 0/3 remove extern from function declarationsDenton Liu, Apr 15, 2019
  5. 2/3 *.[ch]: remove extern from function declarations using sedDenton Liu, Apr 15, 2019
  6. 3/3 cocci: prevent extern function declarationsDenton Liu, Apr 15, 2019
  7. 1/3 *.[ch]: remove extern from function declarations using spatchDenton Liu, Apr 15, 2019
  8. Thomas GummererApr 15, 2019
  9. 0/4 remove extern from function declarationsDenton Liu, Apr 17, 2019
  10. 2/4 *.[ch]: remove extern from function declarations using sedDenton Liu, Apr 17, 2019
  11. 1/4 *.[ch]: remove extern from function declarations using spatchDenton Liu, Apr 17, 2019
  12. 4/4 cocci: prevent extern function declarationsDenton Liu, Apr 17, 2019
  13. 3/4 *.[ch]: manually align parameter listsDenton Liu, Apr 17, 2019
  14. cache.h: fix mismerge of 'dl/no-extern-in-func-decl'Denton Liu, Apr 22, 2019
  15. Junio C HamanoApr 22, 2019
  16. Junio C HamanoApr 22, 2019
  17. Jeff KingApr 22, 2019
  18. SZEDER GáborApr 25, 2019
  19. Denton LiuApr 25, 2019
  20. Johannes SchindelinApr 30, 2019
  21. Denton LiuMay 1, 2019
  22. Jeff KingMay 1, 2019
  23. SZEDER GáborMay 2, 2019
  24. Johannes SchindelinMay 3, 2019
  25. SZEDER GáborMay 3, 2019
  26. SZEDER GáborMay 3, 2019
  27. Jeff KingMay 3, 2019
  28. SZEDER GáborMay 3, 2019
  29. Junio C HamanoMay 5, 2019
  30. Jacob KellerMay 5, 2019
  31. Jacob KellerMay 5, 2019
  32. coccicheck: optionally process every source file at onceJeff King, May 6, 2019
  33. Duy NguyenMay 6, 2019
  34. coccicheck: optionally batch spatch invocationsJeff King, May 6, 2019
  35. Jacob KellerMay 7, 2019
  36. Jeff KingMay 7, 2019
  37. Junio C HamanoMay 7, 2019
  38. Jeff KingMay 7, 2019
  39. Jacob KellerMay 7, 2019
  40. Junio C HamanoMay 7, 2019
  41. Jeff KingMay 8, 2019
  42. Denton LiuMay 8, 2019
  43. Jeff KingMay 8, 2019
  44. Duy NguyenMay 7, 2019
  45. SZEDER GáborMay 7, 2019
  46. Makefile: fix bugs in coccicheck and speed it upÆvar Arnfjörð Bjarmason, Mar 2, 2021
  47. Ævar Arnfjörð BjarmasonMar 3, 2021
  48. Denton LiuMar 3, 2021
  49. Junio C HamanoMar 4, 2021
  50. Ævar Arnfjörð BjarmasonMar 5, 2021
  51. Jeff KingMar 5, 2021
  52. Ævar Arnfjörð BjarmasonMar 5, 2021
  53. Jeff KingMar 6, 2021
  54. 1/4 Makefile/coccicheck: add comment heading for all SPATCH flagsÆvar Arnfjörð Bjarmason, Mar 5, 2021
  55. 0/4 Makefile/coccicheck: fix bugs and speed it upÆvar Arnfjörð Bjarmason, Mar 5, 2021
  56. René Scharfe.Mar 5, 2021
  57. Ævar Arnfjörð BjarmasonMar 6, 2021
  58. René Scharfe.Mar 6, 2021
  59. Ævar Arnfjörð BjarmasonMar 6, 2021
  60. René Scharfe.Mar 6, 2021
  61. 3/4 Makefile/coccicheck: allow for setting xargs concurrencyÆvar Arnfjörð Bjarmason, Mar 5, 2021
  62. Jeff KingMar 6, 2021
  63. 4/4 Makefile/coccicheck: set SPATCH_BATCH_SIZE to 8Ævar Arnfjörð Bjarmason, Mar 5, 2021
  64. 5/4 Makefile/coccicheck: use --include-headers-for-typesÆvar Arnfjörð Bjarmason, Mar 6, 2021
  65. SZEDER GáborMar 18, 2021
  66. Ævar Arnfjörð BjarmasonMar 19, 2021
  67. 0/4 Makefile/coccicheck: fix bugs and speed it upÆvar Arnfjörð Bjarmason, Mar 22, 2021
  68. 1/4 Makefile/coccicheck: add comment heading for all SPATCH flagsÆvar Arnfjörð Bjarmason, Mar 22, 2021
  69. René Scharfe.Mar 22, 2021
  70. 4/4 Makefile/coccicheck: set SPATCH_BATCH_SIZE to 8Ævar Arnfjörð Bjarmason, Mar 22, 2021
  71. René Scharfe.Mar 22, 2021
  72. Jeff KingMar 24, 2021
  73. 2/4 Makefile/coccicheck: speed up and fix bug with duplicate hunksÆvar Arnfjörð Bjarmason, Mar 22, 2021
  74. René Scharfe.Mar 22, 2021
  75. Jeff KingMar 24, 2021
  76. Junio C HamanoMar 22, 2021
  77. 3/4 Makefile/coccicheck: allow for setting xargs concurrencyÆvar Arnfjörð Bjarmason, Mar 22, 2021
  78. Jeff KingMar 24, 2021
  79. Ævar Arnfjörð BjarmasonMar 25, 2021
  80. Jeff KingMar 26, 2021
  81. Junio C HamanoMar 27, 2021
  82. Ævar Arnfjörð BjarmasonMar 27, 2021
  83. 2/4 Makefile/coccicheck: speed up and fix bug with duplicate hunksÆvar Arnfjörð Bjarmason, Mar 5, 2021
  84. Jeff KingMar 6, 2021
  85. Ævar Arnfjörð BjarmasonMar 6, 2021
  86. Denton LiuMay 3, 2019
  87. 0/4 remove extern from function declarationsDenton Liu, Apr 23, 2019
  88. 2/4 *.[ch]: remove extern from function declarations using sedDenton Liu, Apr 23, 2019
  89. Junio C HamanoApr 24, 2019
  90. Denton LiuApr 25, 2019
  91. 3/4 *.[ch]: manually align parameter listsDenton Liu, Apr 23, 2019
  92. 4/4 cocci: prevent extern function declarationsDenton Liu, Apr 23, 2019
  93. 1/4 *.[ch]: remove extern from function declarations using spatchDenton Liu, Apr 23, 2019
  94. 0/3 *** SUBJECT HERE ***Denton Liu, Apr 29, 2019
  95. 2/3 *.[ch]: remove extern from function declarations using sedDenton Liu, Apr 29, 2019
  96. 1/3 *.[ch]: remove extern from function declarations using spatchDenton Liu, Apr 29, 2019
  97. 3/3 *.[ch]: manually align parameter listsDenton Liu, Apr 29, 2019
  98. Denton LiuApr 29, 2019
  99. Ævar Arnfjörð BjarmasonMay 6, 2019
  100. Denton LiuMay 6, 2019
  101. René Scharfe.Mar 6, 2021
  102. René Scharfe.Mar 6, 2021
  103. René Scharfe.Mar 13, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.