git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 0/5] cocci: make "incremental" possible + a ccache-like tool

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Aug 25, 2022, 14:36 UTC
Message-ID
<cover-0.5-00000000000-20220825T141212Z-avarab@gmail.com>

This series makes the "coccicheck" rule incremental by only re-running "spatch" for those file/rule combinations that changed.

The result is that e.g. running "make coccicheck" followed by a:
	make -W grep.c coccicheck

Will take around 2s instead of a *lot* longer. The 3/5 change has all the details about this, and why it's safe.

Furthermore we'll properly handle re-building if any of the required headers change (e.g. column.h).

I had a previous stab at trying to make "coccicheck" faster a while ago in [1], but that was fatally flawed because it would break in cases where grep.c e.g. dependen on a header that changed.

This series doesn't suffer from that issue, because we piggy-back on the corresponding .depend/* files to extract dependency info. This does mean that the incremental speed-ups depend on COMPUTE_HEADER_DEPENDENCIES=yes (which is the case with "gcc" and "clang" by default).

I've been running with this series locally for a while. I think it should fully solve the lamentations about the coccicheck being too slow (e.g. in [2]), particularly when paired with the new "spatchcache" helper added in 5/5.

Even without that helper running "make coccicheck" e.g. after applying a patch series now becomes viable, provided the patches don't modify one of the "rebuild everything" files we have (such as cache.h and git-compat-util.h).

1. https://lore.kernel.org/git/20210302205103.12230-1-avarab@gmail.com/
2. https://lore.kernel.org/git/xmqqh7249b8d.fsf@gitster.g/
Ævar Arnfjörð Bjarmason (5):
  Makefile: add ability to TAB-complete cocci *.patch rules
  Makefile: have "coccicheck" re-run if flags change
  cocci: make "coccicheck" rule incremental
  cocci: make incremental compilation even faster
  spatchcache: add a ccache-alike for "spatch"
 .gitignore                                    |   1 +
 Makefile                                      | 114 +++++++--
 contrib/coccinelle/spatchcache                | 217 ++++++++++++++++++
 .../coccinelle/the_repository.pending.cocci   |   1 -
 shared.mak                                    |   4 +-
 5 files changed, 310 insertions(+), 27 deletions(-)
 create mode 100755 contrib/coccinelle/spatchcache
-- 
2.37.2.1279.g64dec4e13cf
Next: Ævar Arnfjörð Bjarmason
Message 1 of 72 in “cocci: make "incremental" possible + a ccache-like tool”
  1. 0/5 cocci: make "incremental" possible + a ccache-like toolÆvar Arnfjörð Bjarmason, Aug 25, 2022
  2. 1/5 Makefile: add ability to TAB-complete cocci *.patch rulesÆvar Arnfjörð Bjarmason, Aug 25, 2022
  3. 2/5 Makefile: have "coccicheck" re-run if flags changeÆvar Arnfjörð Bjarmason, Aug 25, 2022
  4. SZEDER GáborAug 25, 2022
  5. 4/5 cocci: make incremental compilation even fasterÆvar Arnfjörð Bjarmason, Aug 25, 2022
  6. 3/5 cocci: make "coccicheck" rule incrementalÆvar Arnfjörð Bjarmason, Aug 25, 2022
  7. SZEDER GáborAug 25, 2022
  8. Ævar Arnfjörð BjarmasonAug 25, 2022
  9. SZEDER GáborAug 26, 2022
  10. 5/5 spatchcache: add a ccache-alike for "spatch"Ævar Arnfjörð Bjarmason, Aug 25, 2022
  11. 0/9 cocci: make "incremental" possible + a ccache-like toolÆvar Arnfjörð Bjarmason, Aug 31, 2022
  12. 2/9 Makefile: add ability to TAB-complete cocci *.patch rulesÆvar Arnfjörð Bjarmason, Aug 31, 2022
  13. 3/9 Makefile: have "coccicheck" re-run if flags changeÆvar Arnfjörð Bjarmason, Aug 31, 2022
  14. 1/9 cocci rules: remove unused "F" metavariable from pending ruleÆvar Arnfjörð Bjarmason, Aug 31, 2022
  15. 4/9 Makefile: split off SPATCH_BATCH_SIZE comment from "cocci" headingÆvar Arnfjörð Bjarmason, Aug 31, 2022
  16. 5/9 cocci: split off include-less "tests" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Aug 31, 2022
  17. 6/9 cocci: split off "--all-includes" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Aug 31, 2022
  18. 8/9 cocci: optimistically use COMPUTE_HEADER_DEPENDENCIESÆvar Arnfjörð Bjarmason, Aug 31, 2022
  19. 7/9 cocci: make "coccicheck" rule incrementalÆvar Arnfjörð Bjarmason, Aug 31, 2022
  20. SZEDER GáborSep 1, 2022
  21. Ævar Arnfjörð BjarmasonSep 1, 2022
  22. 9/9 spatchcache: add a ccache-alike for "spatch"Ævar Arnfjörð Bjarmason, Aug 31, 2022
  23. 00/11 cocci: make "incremental" possible + a ccache-like toolÆvar Arnfjörð Bjarmason, Oct 14, 2022
  24. 01/11 Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)Ævar Arnfjörð Bjarmason, Oct 14, 2022
  25. 02/11 cocci rules: remove unused "F" metavariable from pending ruleÆvar Arnfjörð Bjarmason, Oct 14, 2022
  26. 05/11 Makefile: split off SPATCH_BATCH_SIZE comment from "cocci" headingÆvar Arnfjörð Bjarmason, Oct 14, 2022
  27. Taylor BlauOct 14, 2022
  28. 03/11 Makefile: add ability to TAB-complete cocci *.patch rulesÆvar Arnfjörð Bjarmason, Oct 14, 2022
  29. 07/11 cocci: split off "--all-includes" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Oct 14, 2022
  30. 04/11 Makefile: have "coccicheck" re-run if flags changeÆvar Arnfjörð Bjarmason, Oct 14, 2022
  31. 06/11 cocci: split off include-less "tests" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Oct 14, 2022
  32. 08/11 cocci: make "coccicheck" rule incrementalÆvar Arnfjörð Bjarmason, Oct 14, 2022
  33. 09/11 cocci: optimistically use COMPUTE_HEADER_DEPENDENCIESÆvar Arnfjörð Bjarmason, Oct 14, 2022
  34. 11/11 spatchcache: add a ccache-alike for "spatch"Ævar Arnfjörð Bjarmason, Oct 14, 2022
  35. 10/11 cocci: run against a generated ALL.cocciÆvar Arnfjörð Bjarmason, Oct 14, 2022
  36. Jeff KingOct 17, 2022
  37. Ævar Arnfjörð BjarmasonOct 17, 2022
  38. Junio C HamanoOct 17, 2022
  39. Jeff KingOct 17, 2022
  40. 00/12 cocci: make "incremental" possible + a ccache-like toolÆvar Arnfjörð Bjarmason, Oct 26, 2022
  41. 01/12 Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)Ævar Arnfjörð Bjarmason, Oct 26, 2022
  42. 02/12 cocci rules: remove unused "F" metavariable from pending ruleÆvar Arnfjörð Bjarmason, Oct 26, 2022
  43. 03/12 Makefile: add ability to TAB-complete cocci *.patch rulesÆvar Arnfjörð Bjarmason, Oct 26, 2022
  44. 04/12 Makefile: have "coccicheck" re-run if flags changeÆvar Arnfjörð Bjarmason, Oct 26, 2022
  45. 05/12 Makefile: split off SPATCH_BATCH_SIZE comment from "cocci" headingÆvar Arnfjörð Bjarmason, Oct 26, 2022
  46. 07/12 cocci: split off "--all-includes" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Oct 26, 2022
  47. 06/12 cocci: split off include-less "tests" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Oct 26, 2022
  48. 08/12 cocci: make "coccicheck" rule incrementalÆvar Arnfjörð Bjarmason, Oct 26, 2022
  49. 09/12 cocci: optimistically use COMPUTE_HEADER_DEPENDENCIESÆvar Arnfjörð Bjarmason, Oct 26, 2022
  50. 10/12 cocci rules: remove <id>'s from rules that don't need themÆvar Arnfjörð Bjarmason, Oct 26, 2022
  51. 11/12 cocci: run against a generated ALL.cocciÆvar Arnfjörð Bjarmason, Oct 26, 2022
  52. SZEDER GáborOct 28, 2022
  53. 12/12 spatchcache: add a ccache-alike for "spatch"Ævar Arnfjörð Bjarmason, Oct 26, 2022
  54. 00/13 cocci: make "incremental" possible + a ccache-like toolÆvar Arnfjörð Bjarmason, Nov 1, 2022
  55. 01/13 Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)Ævar Arnfjörð Bjarmason, Nov 1, 2022
  56. 02/13 cocci rules: remove unused "F" metavariable from pending ruleÆvar Arnfjörð Bjarmason, Nov 1, 2022
  57. 03/13 Makefile: add ability to TAB-complete cocci *.patch rulesÆvar Arnfjörð Bjarmason, Nov 1, 2022
  58. 04/13 Makefile: have "coccicheck" re-run if flags changeÆvar Arnfjörð Bjarmason, Nov 1, 2022
  59. 06/13 cocci: split off include-less "tests" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Nov 1, 2022
  60. 05/13 Makefile: split off SPATCH_BATCH_SIZE comment from "cocci" headingÆvar Arnfjörð Bjarmason, Nov 1, 2022
  61. 07/13 cocci: split off "--all-includes" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason, Nov 1, 2022
  62. 08/13 cocci: make "coccicheck" rule incrementalÆvar Arnfjörð Bjarmason, Nov 1, 2022
  63. SZEDER GáborNov 9, 2022
  64. 10/13 Makefile: copy contrib/coccinelle/*.cocci to build/Ævar Arnfjörð Bjarmason, Nov 1, 2022
  65. SZEDER GáborNov 9, 2022
  66. Ævar Arnfjörð BjarmasonNov 9, 2022
  67. Makefile: don't create a ".build/.build/" for cocci, fix outputÆvar Arnfjörð Bjarmason, Nov 10, 2022
  68. Taylor BlauNov 11, 2022
  69. 09/13 cocci: optimistically use COMPUTE_HEADER_DEPENDENCIESÆvar Arnfjörð Bjarmason, Nov 1, 2022
  70. 11/13 cocci rules: remove <id>'s from rules that don't need themÆvar Arnfjörð Bjarmason, Nov 1, 2022
  71. 12/13 cocci: run against a generated ALL.cocciÆvar Arnfjörð Bjarmason, Nov 1, 2022
  72. 13/13 spatchcache: add a ccache-alike for "spatch"Ævar Arnfjörð Bjarmason, Nov 1, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.