git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Why the Makefile is so eager to re-build & re-link

From
Øystein Walle <oystwa@gmail.com>
Date
Jun 24, 2021, 23:35 UTC
Message-ID
<20210624233516.23564-1-oystwa@gmail.com>
In-Reply-To
<874kdn1j6i.fsf@evledraar.gmail.com>
Hi, Ævar
Show 13 quoted lines
> {command,config}-list.h (and in-flight, my hook-list.h): Every time
> you touch a Documentation/git-*.txt we need to re-generate these, and
> since their mtime changes we re-compile and re-link all the way up to
> libgit and our other tools.
> 
> I think the best solution here is to make the generate-*.sh
> shellscripts faster (just one takes ~300ms of nested shellscripting,
> just to grep out the first few lines of every git-*.txt, in e.g. Perl
> or a smarter awk script this would be <5ms).
> 
> Then we make those FORCE, but most of the time the config or command
> summary (or list of hooks) doesn't change, so we don't need to
> replace the file.

One possible technique to fix this is to generate to a temporary file, and copy the temporary file over the real file if it's actually different. I see the Makefile already does create a temporary file, so how about something like this:

diff --git a/Makefile b/Makefile
index 93664d6714..9b2f081a2a 100644
--- a/Makefile
+++ b/Makefile
@@ -2216,7 +2216,8 @@ command-list.h: generate-cmdlist.sh command-list.txt
 command-list.h: $(wildcard Documentation/git*.txt)
 	$(QUIET_GEN)$(SHELL_PATH) ./generate-cmdlist.sh \
 		$(patsubst %,--exclude-program %,$(EXCLUDED_PROGRAMS)) \
-		command-list.txt >$@+ && mv $@+ $@
+		command-list.txt >$@+ && \
+		if ! cmp -s $@ $@+; then mv $@+ $@; fi
 
 SCRIPT_DEFINES = $(SHELL_PATH_SQ):$(DIFF_SQ):$(GIT_VERSION):\
 	$(localedir_SQ):$(NO_CURL):$(USE_GETTEXT_SCHEME):$(SANE_TOOL_PATH_SQ):\


This seems to work fine from my basic testing. I think it can even be written
more terse as `&& ! cmp ... &&` but that looks a bit weird to me. In any case
it looks like it can easily be added the other relevant places in the Makefile
too.

Øsse
Previous: Eric SunshineNext: Øystein Walle
Message 76 of 87 in “Why the Makefile is so eager to re-build & re-link”
  1. Ævar Arnfjörð BjarmasonJun 24, 2021
  2. Jeff KingJun 24, 2021
  3. Ævar Arnfjörð BjarmasonJun 24, 2021
  4. Johannes SixtJun 24, 2021
  5. Ævar Arnfjörð BjarmasonJun 25, 2021
  6. Ævar Arnfjörð BjarmasonJun 25, 2021
  7. Jeff KingJun 29, 2021
  8. 0/8 Makefile: make command-list.h 2-5x as fast with -jNÆvar Arnfjörð Bjarmason, Oct 20, 2021
  9. 1/8 command-list.txt: sort with "LC_ALL=C sort"Ævar Arnfjörð Bjarmason, Oct 20, 2021
  10. 2/8 generate-cmdlist.sh: trivial whitespace changeÆvar Arnfjörð Bjarmason, Oct 20, 2021
  11. 3/8 generate-cmdlist.sh: spawn fewer processesÆvar Arnfjörð Bjarmason, Oct 20, 2021
  12. 4/8 generate-cmdlist.sh: don't call get_categories() from category_list()Ævar Arnfjörð Bjarmason, Oct 20, 2021
  13. 5/8 generate-cmdlist.sh: run "grep | sort", not "sort | grep"Ævar Arnfjörð Bjarmason, Oct 20, 2021
  14. 6/8 generate-cmdlist.sh: replace for loop by printf's auto-repeat featureÆvar Arnfjörð Bjarmason, Oct 20, 2021
  15. Jeff KingOct 21, 2021
  16. Jeff KingOct 21, 2021
  17. 7/8 Makefile: stop having command-list.h depend on a wildcardÆvar Arnfjörð Bjarmason, Oct 20, 2021
  18. Jeff KingOct 21, 2021
  19. Junio C HamanoOct 21, 2021
  20. Øystein WalleOct 21, 2021
  21. 8/8 Makefile: assert correct generate-cmdlist.sh outputÆvar Arnfjörð Bjarmason, Oct 20, 2021
  22. Jeff KingOct 20, 2021
  23. Taylor BlauOct 20, 2021
  24. Ævar Arnfjörð BjarmasonOct 20, 2021
  25. Jeff KingOct 20, 2021
  26. Ævar Arnfjörð BjarmasonOct 21, 2021
  27. Taylor BlauOct 21, 2021
  28. Ævar Arnfjörð BjarmasonOct 22, 2021
  29. Jeff KingOct 21, 2021
  30. Junio C HamanoOct 21, 2021
  31. Ævar Arnfjörð BjarmasonOct 22, 2021
  32. Jeff KingOct 22, 2021
  33. Ævar Arnfjörð BjarmasonOct 22, 2021
  34. Eric SunshineOct 21, 2021
  35. 00/10 Makefile: make generate-cmdlist.sh much fasterÆvar Arnfjörð Bjarmason, Oct 22, 2021
  36. 01/10 command-list.txt: sort with "LC_ALL=C sort"Ævar Arnfjörð Bjarmason, Oct 22, 2021
  37. Junio C HamanoOct 25, 2021
  38. Ævar Arnfjörð BjarmasonOct 25, 2021
  39. Junio C HamanoOct 25, 2021
  40. 02/10 generate-cmdlist.sh: trivial whitespace changeÆvar Arnfjörð Bjarmason, Oct 22, 2021
  41. 03/10 generate-cmdlist.sh: spawn fewer processesÆvar Arnfjörð Bjarmason, Oct 22, 2021
  42. 04/10 generate-cmdlist.sh: don't call get_categories() from category_list()Ævar Arnfjörð Bjarmason, Oct 22, 2021
  43. 05/10 generate-cmdlist.sh: run "grep | sort", not "sort | grep"Ævar Arnfjörð Bjarmason, Oct 22, 2021
  44. 06/10 generate-cmdlist.sh: replace for loop by printf's auto-repeat featureÆvar Arnfjörð Bjarmason, Oct 22, 2021
  45. 07/10 generate-cmdlist.sh: stop sorting category linesÆvar Arnfjörð Bjarmason, Oct 22, 2021
  46. Jeff KingOct 25, 2021
  47. 08/10 generate-cmdlist.sh: do not shell out to "sed"Ævar Arnfjörð Bjarmason, Oct 22, 2021
  48. Jeff KingOct 25, 2021
  49. Jeff KingOct 25, 2021
  50. 10/10 generate-cmdlist.sh: replace "cut", "tr" and "grep" with pure-shellÆvar Arnfjörð Bjarmason, Oct 22, 2021
  51. Junio C HamanoOct 23, 2021
  52. 09/10 generate-cmdlist.sh: replace "grep' invocation with a shell versionÆvar Arnfjörð Bjarmason, Oct 22, 2021
  53. Junio C HamanoOct 23, 2021
  54. Junio C HamanoOct 23, 2021
  55. Taylor BlauOct 22, 2021
  56. Junio C HamanoOct 23, 2021
  57. Jeff KingOct 25, 2021
  58. 00/10 generate-cmdlist.sh: make it (and "make") run fasterÆvar Arnfjörð Bjarmason, Nov 5, 2021
  59. 01/10 command-list.txt: sort with "LC_ALL=C sort"Ævar Arnfjörð Bjarmason, Nov 5, 2021
  60. Junio C HamanoNov 5, 2021
  61. Ævar Arnfjörð BjarmasonNov 6, 2021
  62. Junio C HamanoNov 8, 2021
  63. 02/10 generate-cmdlist.sh: trivial whitespace changeÆvar Arnfjörð Bjarmason, Nov 5, 2021
  64. 03/10 generate-cmdlist.sh: spawn fewer processesÆvar Arnfjörð Bjarmason, Nov 5, 2021
  65. Junio C HamanoNov 5, 2021
  66. Ævar Arnfjörð BjarmasonNov 6, 2021
  67. 04/10 generate-cmdlist.sh: don't call get_categories() from category_list()Ævar Arnfjörð Bjarmason, Nov 5, 2021
  68. 05/10 generate-cmdlist.sh: run "grep | sort", not "sort | grep"Ævar Arnfjörð Bjarmason, Nov 5, 2021
  69. 06/10 generate-cmdlist.sh: replace for loop by printf's auto-repeat featureÆvar Arnfjörð Bjarmason, Nov 5, 2021
  70. 07/10 generate-cmdlist.sh: stop sorting category linesÆvar Arnfjörð Bjarmason, Nov 5, 2021
  71. 08/10 generate-cmdlist.sh: do not shell out to "sed"Ævar Arnfjörð Bjarmason, Nov 5, 2021
  72. 09/10 generate-cmdlist.sh: replace "grep' invocation with a shell versionÆvar Arnfjörð Bjarmason, Nov 5, 2021
  73. 10/10 generate-cmdlist.sh: don't parse command-list.txt thriceÆvar Arnfjörð Bjarmason, Nov 5, 2021
  74. Felipe ContrerasJun 25, 2021
  75. Eric SunshineJun 29, 2021
  76. Øystein WalleJun 24, 2021
  77. Øystein WalleJun 24, 2021
  78. Ævar Arnfjörð BjarmasonJun 25, 2021
  79. Documentation/Makefile: don't re-build on 'git version' changesÆvar Arnfjörð Bjarmason, Jul 2, 2021
  80. Junio C HamanoJul 2, 2021
  81. Ævar Arnfjörð BjarmasonJul 3, 2021
  82. Junio C HamanoJul 5, 2021
  83. Felipe ContrerasJul 3, 2021
  84. Ævar Arnfjörð BjarmasonJul 3, 2021
  85. Felipe ContrerasJul 3, 2021
  86. Junio C HamanoJul 5, 2021
  87. Felipe ContrerasJul 6, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.