git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 8/8] Makefiles: add and use wildcard "mkdir -p" template

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Dec 24, 2021, 17:37 UTC
Message-ID
<patch-v2-8.8-59c1b7032db-20211224T173558Z-avarab@gmail.com>
In-Reply-To
<cover-v2-0.8-00000000000-20211224T173558Z-avarab@gmail.com>

Add a template to do the "mkdir -p" of $(@D) (the parent dir of $@) for us, and use it for the "make lint-docs" targets I added in 8650c6298c1 (doc lint: make "lint-docs" non-.PHONY, 2021-10-15).

As seen in 4c64fb5aad9 (Documentation/Makefile: fix lint-docs mkdir dependency, 2021-10-26) maintaining these manual lists of parent directory dependencies is fragile, in addition to being obviously verbose.

I used this pattern at the time because I couldn't find another method than "order-only" prerequisites to avoid doing a "mkdir -p $(@D)" for every file being created, which as noted in [1] would be significantly slower.

But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run of a performance test similar to that noted downthread of [1] in [2] shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]):

    $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation lint-docs' -p 'rm -rf Documentation/.build' 'make -C Documentation -j1 lint-docs'
    Benchmark 1: make -C Documentation -j1 lint-docs' in 'HEAD~1
      Time (mean ± σ):      2.914 s ±  0.062 s    [User: 2.449 s, System: 0.489 s]
      Range (min … max):    2.834 s …  3.020 s    10 runs
    Benchmark 2: make -C Documentation -j1 lint-docs' in 'HEAD~0
      Time (mean ± σ):      2.315 s ±  0.062 s    [User: 1.950 s, System: 0.386 s]
      Range (min … max):    2.229 s …  2.397 s    10 runs
    Summary
      'make -C Documentation -j1 lint-docs' in 'HEAD~0' ran
        1.26 ± 0.04 times faster than 'make -C Documentation -j1 lint-docs' in 'HEAD~1'

So let's use that pattern both for the "lint-docs" target, and a few miscellaneous other targets.

This method of creating parent directories is explicitly racy in that we don't know if we're going to say always create a "foo" followed by a "foo/bar" under parallelism, or skip the "foo" because we created "foo/bar" first. In this case it doesn't matter for anything except that we aren't guaranteed to get the same number of rules firing when running make in parallel.

1. https://lore.kernel.org/git/211028.861r45y3pt.gmgdl@evledraar.gmail.com/
2. https://lore.kernel.org/git/211028.86o879vvtp.gmgdl@evledraar.gmail.com/
3. https://gitlab.com/avar/git-hyperfine/
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/Makefile | 25 +++----------------------
 Makefile               | 12 +++++++-----
 shared.mak             | 17 +++++++++++++++++
 3 files changed, 27 insertions(+), 27 deletions(-)
diff --git a/Documentation/Makefile b/Documentation/Makefile
index 0f4ebdeda8a..1eb9192dae8 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -434,25 +434,11 @@ quick-install-html: require-htmlrepo
 print-man1:
 	@for i in $(MAN1_TXT); do echo $$i; done
 
-## Lint: Common
-.build:
-	$(QUIET)mkdir $@
-.build/lint-docs: | .build
-	$(QUIET)mkdir $@
-
 ## Lint: gitlink
-.build/lint-docs/gitlink: | .build/lint-docs
-	$(QUIET)mkdir $@
-.build/lint-docs/gitlink/howto: | .build/lint-docs/gitlink
-	$(QUIET)mkdir $@
-.build/lint-docs/gitlink/config: | .build/lint-docs/gitlink
-	$(QUIET)mkdir $@
 LINT_DOCS_GITLINK = $(patsubst %.txt,.build/lint-docs/gitlink/%.ok,$(HOWTO_TXT) $(DOC_DEP_TXT))
-$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink
-$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/howto
-$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/config
 $(LINT_DOCS_GITLINK): lint-gitlink.perl
 $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt
+	$(call mkdir_p_parent_template)
 	$(QUIET_LINT_GITLINK)$(PERL_PATH) lint-gitlink.perl \
 		$< \
 		$(HOWTO_TXT) $(DOC_DEP_TXT) \
@@ -463,23 +449,18 @@ $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt
 lint-docs-gitlink: $(LINT_DOCS_GITLINK)
 
 ## Lint: man-end-blurb
-.build/lint-docs/man-end-blurb: | .build/lint-docs
-	$(QUIET)mkdir $@
 LINT_DOCS_MAN_END_BLURB = $(patsubst %.txt,.build/lint-docs/man-end-blurb/%.ok,$(MAN_TXT))
-$(LINT_DOCS_MAN_END_BLURB): | .build/lint-docs/man-end-blurb
 $(LINT_DOCS_MAN_END_BLURB): lint-man-end-blurb.perl
 $(LINT_DOCS_MAN_END_BLURB): .build/lint-docs/man-end-blurb/%.ok: %.txt
+	$(call mkdir_p_parent_template)
 	$(QUIET_LINT_MANEND)$(PERL_PATH) lint-man-end-blurb.perl $< >$@
 .PHONY: lint-docs-man-end-blurb
-lint-docs-man-end-blurb: $(LINT_DOCS_MAN_END_BLURB)
 
 ## Lint: man-section-order
-.build/lint-docs/man-section-order: | .build/lint-docs
-	$(QUIET)mkdir $@
 LINT_DOCS_MAN_SECTION_ORDER = $(patsubst %.txt,.build/lint-docs/man-section-order/%.ok,$(MAN_TXT))
-$(LINT_DOCS_MAN_SECTION_ORDER): | .build/lint-docs/man-section-order
 $(LINT_DOCS_MAN_SECTION_ORDER): lint-man-section-order.perl
 $(LINT_DOCS_MAN_SECTION_ORDER): .build/lint-docs/man-section-order/%.ok: %.txt
+	$(call mkdir_p_parent_template)
 	$(QUIET_LINT_MANSEC)$(PERL_PATH) lint-man-section-order.perl $< >$@
 .PHONY: lint-docs-man-section-order
 lint-docs-man-section-order: $(LINT_DOCS_MAN_SECTION_ORDER)
diff --git a/Makefile b/Makefile
index d8051278519..5474c27f2b1 100644
--- a/Makefile
+++ b/Makefile
@@ -2714,7 +2714,8 @@ all:: $(MOFILES)
 endif
 
 po/build/locale/%/LC_MESSAGES/git.mo: po/%.po
-	$(QUIET_MSGFMT)mkdir -p $(dir $@) && $(MSGFMT) -o $@ $<
+	$(call mkdir_p_parent_template)
+	$(QUIET_MSGFMT)$(MSGFMT) -o $@ $<
 
 LIB_PERL := $(wildcard perl/Git.pm perl/Git/*.pm perl/Git/*/*.pm perl/Git/*/*/*.pm)
 LIB_PERL_GEN := $(patsubst perl/%.pm,perl/build/lib/%.pm,$(LIB_PERL))
@@ -2730,15 +2731,16 @@ NO_PERL_CPAN_FALLBACKS_SQ = $(subst ','\'',$(NO_PERL_CPAN_FALLBACKS))
 endif
 
 perl/build/lib/%.pm: perl/%.pm GIT-PERL-DEFINES
-	$(QUIET_GEN)mkdir -p $(dir $@) && \
+	$(call mkdir_p_parent_template)
+	$(QUIET_GEN) \
 	sed -e 's|@@LOCALEDIR@@|$(perl_localedir_SQ)|g' \
 	    -e 's|@@NO_GETTEXT@@|$(NO_GETTEXT_SQ)|g' \
 	    -e 's|@@NO_PERL_CPAN_FALLBACKS@@|$(NO_PERL_CPAN_FALLBACKS_SQ)|g' \
 	< $< > $@
 
 perl/build/man/man3/Git.3pm: perl/Git.pm
-	$(QUIET_GEN)mkdir -p $(dir $@) && \
-	pod2man $< $@
+	$(call mkdir_p_parent_template)
+	$(QUIET_GEN)pod2man $< $@
 
 $(ETAGS_TARGET): $(FOUND_SOURCE_FILES)
 	$(QUIET_GEN)$(RM) $@+ && \
@@ -2872,7 +2874,7 @@ test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(
 all:: $(TEST_PROGRAMS) $(test_bindir_programs)
 
 bin-wrappers/%: wrap-for-bin.sh
-	@mkdir -p bin-wrappers
+	$(call mkdir_p_parent_template)
 	$(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
 	     -e 's|@@BUILD_DIR@@|$(shell pwd)|' \
 	     -e 's|@@PROG@@|$(patsubst test-%,t/helper/test-%$(X),$(@F))$(patsubst git%,$(X),$(filter $(@F),$(BINDIR_PROGRAMS_NEED_X)))|' < $< > $@ && \
diff --git a/shared.mak b/shared.mak
index f8ebaf30569..7787f81856c 100644
--- a/shared.mak
+++ b/shared.mak
@@ -59,6 +59,8 @@ ifndef V
 	QUIET          = @
 	QUIET_GEN      = @echo $(wspfx_SQ) GEN $@;
 
+	QUIET_MKDIR_P_PARENT  = @echo $(wspfx_SQ) MKDIR -p $(@D);
+
 ## Used in "Makefile"
 	QUIET_CC       = @echo $(wspfx_SQ) CC $@;
 	QUIET_AR       = @echo $(wspfx_SQ) AR $@;
@@ -90,3 +92,18 @@ ifndef V
 	export V
 endif
 endif
+
+### Templates
+
+## mkdir_p_parent: lazily "mkdir -p" the path needed for a $@
+## file. Uses $(wildcard) to avoid the "mkdir -p" if it's not
+## needed.
+##
+## Is racy, but in a good way; we might redundantly (and safely)
+## "mkdir -p" when running in parallel, but won't need to exhaustively create
+## individual rules for "a" -> "prefix" -> "dir" -> "file" if given a
+## "a/prefix/dir/file". This can instead be inserted at the start of
+## the "a/prefix/dir/file" rule.
+define mkdir_p_parent_template
+$(if $(wildcard $(@D)),,$(QUIET_MKDIR_P_PARENT)$(shell mkdir -p $(@D)))
+endef

-- 
2.34.1.1215.g6e154b84c77
Previous: Ævar Arnfjörð BjarmasonNext: Ævar Arnfjörð Bjarmason
Message 11 of 67 in “Makefile: optimize noop runs, add shared.mak”
  1. 0/8 Makefile: optimize noop runs, add shared.makÆvar Arnfjörð Bjarmason, Dec 24, 2021
  2. 1/8 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to itÆvar Arnfjörð Bjarmason, Dec 24, 2021
  3. 2/8 Makefile: disable GNU make built-in wildcard rulesÆvar Arnfjörð Bjarmason, Dec 24, 2021
  4. 3/8 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)Ævar Arnfjörð Bjarmason, Dec 24, 2021
  5. 4/8 Makefile: move ".SUFFIXES" rule to shared.makÆvar Arnfjörð Bjarmason, Dec 24, 2021
  6. Taylor BlauFeb 22, 2022
  7. Ævar Arnfjörð BjarmasonFeb 22, 2022
  8. 6/8 Makefile: add "$(QUIET)" boilerplate to shared.makÆvar Arnfjörð Bjarmason, Dec 24, 2021
  9. 5/8 Makefile: move $(comma), $(empty) and $(space) to shared.makÆvar Arnfjörð Bjarmason, Dec 24, 2021
  10. 7/8 Makefile: use $(wspfx) for $(QUIET...) in shared.makÆvar Arnfjörð Bjarmason, Dec 24, 2021
  11. 8/8 Makefiles: add and use wildcard "mkdir -p" templateÆvar Arnfjörð Bjarmason, Dec 24, 2021
  12. Ævar Arnfjörð BjarmasonFeb 21, 2022
  13. 0/9 Makefile: optimize noop runs, add shared.makÆvar Arnfjörð Bjarmason, Feb 25, 2022
  14. 1/9 scalar Makefile: set the default target after the includesÆvar Arnfjörð Bjarmason, Feb 25, 2022
  15. Junio C HamanoFeb 25, 2022
  16. 2/9 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to itÆvar Arnfjörð Bjarmason, Feb 25, 2022
  17. Junio C HamanoFeb 25, 2022
  18. Ævar Arnfjörð BjarmasonFeb 25, 2022
  19. Junio C HamanoFeb 25, 2022
  20. Phillip WoodFeb 28, 2022
  21. Ævar Arnfjörð BjarmasonFeb 28, 2022
  22. Phillip WoodFeb 28, 2022
  23. Ævar Arnfjörð BjarmasonFeb 28, 2022
  24. 3/9 Makefile: disable GNU make built-in wildcard rulesÆvar Arnfjörð Bjarmason, Feb 25, 2022
  25. Junio C HamanoFeb 25, 2022
  26. 5/9 Makefile: move ".SUFFIXES" rule to shared.makÆvar Arnfjörð Bjarmason, Feb 25, 2022
  27. Junio C HamanoFeb 25, 2022
  28. 7/9 Makefile: add "$(QUIET)" boilerplate to shared.makÆvar Arnfjörð Bjarmason, Feb 25, 2022
  29. Junio C HamanoFeb 25, 2022
  30. 6/9 Makefile: move $(comma), $(empty) and $(space) to shared.makÆvar Arnfjörð Bjarmason, Feb 25, 2022
  31. Junio C HamanoFeb 25, 2022
  32. 4/9 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)Ævar Arnfjörð Bjarmason, Feb 25, 2022
  33. 8/9 Makefile: use $(wspfx) for $(QUIET...) in shared.makÆvar Arnfjörð Bjarmason, Feb 25, 2022
  34. Junio C HamanoFeb 25, 2022
  35. 9/9 Makefiles: add and use wildcard "mkdir -p" templateÆvar Arnfjörð Bjarmason, Feb 25, 2022
  36. 0/9 Makefile: optimize noop runs, add shared.makÆvar Arnfjörð Bjarmason, Mar 2, 2022
  37. 1/9 scalar Makefile: use "The default target of..." patternÆvar Arnfjörð Bjarmason, Mar 2, 2022
  38. Junio C HamanoMar 2, 2022
  39. 2/9 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to itÆvar Arnfjörð Bjarmason, Mar 2, 2022
  40. 3/9 Makefile: disable GNU make built-in wildcard rulesÆvar Arnfjörð Bjarmason, Mar 2, 2022
  41. 4/9 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)Ævar Arnfjörð Bjarmason, Mar 2, 2022
  42. 5/9 Makefile: move ".SUFFIXES" rule to shared.makÆvar Arnfjörð Bjarmason, Mar 2, 2022
  43. 7/9 Makefile: add "$(QUIET)" boilerplate to shared.makÆvar Arnfjörð Bjarmason, Mar 2, 2022
  44. 6/9 Makefile: move $(comma), $(empty) and $(space) to shared.makÆvar Arnfjörð Bjarmason, Mar 2, 2022
  45. 8/9 Makefile: use $(wspfx) for $(QUIET...) in shared.makÆvar Arnfjörð Bjarmason, Mar 2, 2022
  46. Junio C HamanoMar 2, 2022
  47. 9/9 Makefiles: add and use wildcard "mkdir -p" templateÆvar Arnfjörð Bjarmason, Mar 2, 2022
  48. Junio C HamanoMar 2, 2022
  49. Junio C HamanoMar 2, 2022
  50. Phillip WoodMar 3, 2022
  51. 0/8 Makefile: optimize noop runs, add shared.makÆvar Arnfjörð Bjarmason, Mar 3, 2022
  52. 1/8 scalar Makefile: use "The default target of..." patternÆvar Arnfjörð Bjarmason, Mar 3, 2022
  53. 2/8 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to itÆvar Arnfjörð Bjarmason, Mar 3, 2022
  54. 3/8 Makefile: disable GNU make built-in wildcard rulesÆvar Arnfjörð Bjarmason, Mar 3, 2022
  55. Rene KitaApr 11, 2022
  56. Ævar Arnfjörð BjarmasonApr 11, 2022
  57. 4/8 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES)Ævar Arnfjörð Bjarmason, Mar 3, 2022
  58. 5/8 Makefile: move ".SUFFIXES" rule to shared.makÆvar Arnfjörð Bjarmason, Mar 3, 2022
  59. Adam DinwoodieApr 5, 2022
  60. Ævar Arnfjörð BjarmasonApr 5, 2022
  61. Documentation/Makefile: fix "make info" regression in dad9cd7d518Ævar Arnfjörð Bjarmason, Apr 5, 2022
  62. Adam DinwoodieApr 6, 2022
  63. Junio C HamanoApr 6, 2022
  64. Taylor BlauApr 6, 2022
  65. 7/8 Makefile: add "$(QUIET)" boilerplate to shared.makÆvar Arnfjörð Bjarmason, Mar 3, 2022
  66. 6/8 Makefile: move $(comma), $(empty) and $(space) to shared.makÆvar Arnfjörð Bjarmason, Mar 3, 2022
  67. 8/8 Makefiles: add and use wildcard "mkdir -p" templateÆvar Arnfjörð Bjarmason, Mar 3, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.