[PATCH 7/8] Makefile: stop having command-list.h depend on a wildcard

Change the dependency graph for "command-list.h" so that it's not re-built in its entirety every time one file in Documentation/git*.txt changes. This makes the generation of the command-list.h use a pattern similar to that established in 8650c6298c1 (doc lint: make "lint-docs" non-.PHONY, 2021-10-15) for the "lint-docs" target.

This change replaces the monolithic "generate-cmdlist.sh" invocation with a dependency chain like:

    Documentation/git-add.txt ->
    .build/command-list.h.d/git-add.gen (and "git-add.txt" intermediary) ->
    command-list.h

I.e. we'll now generate intermediary files when extracting the "NAME" section the documentation. Here the 6th line of "Documentation/git.txt" will be extracted to ".build/command-list.h.d/git-add.gen.txt":

    git-add - Add file contents to the index

We'll then create an intermediate ".build/command-list.h.d/git-add.gen" (leading "\t" stripped):

    ./generate-cmdlist.sh --entry-only .build/command-list.h.d/git-add.gen.txt
    { "git-add", N_("Add file contents to the index"), 0 | CAT_mainporcelain | CAT_worktree },

The "command-list.h" itself can the be made by having "generate-cmdlist.sh" emit only the header section, followed by:

    LC_ALL=sort .build/command-list.h.d/*.gen

Unfortunately we can't drop the old code completely due to the CMake integration, see 061c2240b1b (Introduce CMake support for configuring Git, 2020-06-12). It will keep using the older and slower script.

With this the initial creation of the command-list.h is a bit slower with -j1, but around 2x as fast -j8[1]. The real benefit comes from the more common case of an incremental build, say when only "Documentation/git-add.txt" was updated after a "pull". There we're 4-5x as fast with this new method[2].

The benefit of optimizing this is because this file is very frequently re-generated, e.g. for "git rebase -i --exec 'make git'" with "git-add.txt" modified we're around 1.7 times as fast[3]. That target will need to re-make "git" (via "help.o") due to help.c's use of command-list.h.

In terms of implementation: I'm adding $(QUIET) while I'm at it, which is here so we don't quiet the equivalent of trace output under V=1, this could be used in other places that use "@cmd" to quiet "cmd" output.

Some of the dependencies between "command-list.h" and "$(COMMAND_LIST_GEN)" are redundant, considering that the former depends on the latter. I'm sticking to listing dependencies mentioned in the rule itself, e.g. "command-list.h" itself calls "generate-cmdlist.sh", so I list the dependency even though it would get it via a recursive dependency.

These rules can be lazy about leaving behind files on error thanks to the .DELETE_ON_ERROR flag, see 7b76d6bf221 (Makefile: add and use the ".DELETE_ON_ERROR" flag, 2021-06-29), or in the case of the "*.txt.gen" files because we'll unconditionally clobber them anyway if the relevant source file is touched, so we can leave them for "make clean".

1. Piped through: <cmd> 2>/dev/null | grep -P -v "^\s*$"; ditto below:
   hyperfine -s basic -L j ", -j8" -L s ,.old -p 'rm -rf .build command-list.h' 'make{j} -f Makefile{s} command-list.h'
   Benchmark #1: make -f Makefile command-list.h
     Time (mean ± σ):     769.3 ms ±  90.4 ms    [User: 892.4 ms, System: 98.8 ms]
     Range (min … max):   665.6 ms … 941.9 ms    10 runs
   Benchmark #2: make -j8 -f Makefile command-list.h
     Time (mean ± σ):     212.5 ms ±  45.0 ms    [User: 954.4 ms, System: 136.9 ms]
     Range (min … max):   187.6 ms … 326.2 ms    11 runs
   Benchmark #3: make -f Makefile.old command-list.h
     Time (mean ± σ):     515.0 ms ±  70.5 ms    [User: 613.8 ms, System: 110.9 ms]
     Range (min … max):   407.6 ms … 603.9 ms    10 runs
   Benchmark #4: make -j8 -f Makefile.old command-list.h
     Time (mean ± σ):     474.9 ms ±  62.7 ms    [User: 569.7 ms, System: 106.0 ms]
     Range (min … max):   407.0 ms … 556.8 ms    10 runs
   Summary
     'make -j8 -f Makefile command-list.h' ran
       2.24 ± 0.56 times faster than 'make -j8 -f Makefile.old command-list.h'
       2.42 ± 0.61 times faster than 'make -f Makefile.old command-list.h'
       3.62 ± 0.88 times faster than 'make -f Makefile command-list.h'
2. hyperfine -s basic -L j ", -j8" -L s ,.old -p 'touch Documentation/git-add.txt' 'make{j} -f Makefile{s} command-list.h'
   Benchmark #1: make -f Makefile command-list.h
     Time (mean ± σ):      94.6 ms ±  26.7 ms    [User: 83.0 ms, System: 14.1 ms]
     Range (min … max):    79.4 ms … 186.0 ms    15 runs
   Benchmark #2: make -j8 -f Makefile command-list.h
     Time (mean ± σ):      80.3 ms ±  10.5 ms    [User: 81.3 ms, System: 12.9 ms]
     Range (min … max):    76.9 ms … 127.6 ms    37 runs
   Benchmark #3: make -f Makefile.old command-list.h
     Time (mean ± σ):     348.9 ms ± 174.5 ms    [User: 415.3 ms, System: 70.6 ms]
     Range (min … max):    66.0 ms … 550.3 ms    43 runs
   Benchmark #4: make -j8 -f Makefile.old command-list.h
     Time (mean ± σ):     406.7 ms ± 157.1 ms    [User: 473.4 ms, System: 78.3 ms]
     Range (min … max):    67.4 ms … 560.4 ms    34 runs
   Summary
     'make -j8 -f Makefile command-list.h' ran
       1.18 ± 0.37 times faster than 'make -f Makefile command-list.h'
       4.34 ± 2.25 times faster than 'make -f Makefile.old command-list.h'
       5.07 ± 2.07 times faster than 'make -j8 -f Makefile.old command-list.h'
3. hyperfine -s basic -L j " -j8" -L s ,.old -p 'touch Documentation/git-add.txt' 'make{j} -f Makefile{s} git'
   Benchmark #1: make -j8 -f Makefile git
     Time (mean ± σ):     484.2 ms ±  53.5 ms    [User: 253.2 ms, System: 83.5 ms]
     Range (min … max):   400.7 ms … 580.5 ms    10 runs
   Benchmark #2: make -j8 -f Makefile.old git
     Time (mean ± σ):     836.7 ms ±  37.3 ms    [User: 729.2 ms, System: 157.1 ms]
     Range (min … max):   774.2 ms … 885.6 ms    10 runs
   Summary
     'make -j8 -f Makefile git' ran
       1.73 ± 0.21 times faster than 'make -j8 -f Makefile.old git'

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 .gitignore          |  1 +
 Makefile            | 52 +++++++++++++++++++++++++++++++++++++++------
 generate-cmdlist.sh | 36 +++++++++++++++++++++++++------
 3 files changed, 77 insertions(+), 12 deletions(-)