git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 00/10] Makefile: make generate-cmdlist.sh much faster

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Oct 22, 2021, 19:36 UTC
Message-ID
<cover-v2-00.10-00000000000-20211022T193027Z-avarab@gmail.com>
In-Reply-To
<cover-0.8-00000000000-20211020T183533Z-avarab@gmail.com>

This version of this series drops the Makefile-powered version of the cmdlist in favor of making the shellscript much faster, mostly with suggestions from Jeff King.

I still think that splitting out the generated data into files may be useful for unifying the Documentation/ and C code build processes, there's another custom parser for command-list.txt in Documentation/cmd-list.perl.

But if and when I've got something for that I can dig that out of the v1, in the meantime the v1 of this should be mostly uncontroversial.

The last tow patches make things a bit slower for me, but since they replace command invocations with pure-shell logic they presumably make things a bit less painful on e.g. Windows, and the 8th patch here already made things quite very fast already.

Jeff King (1):
  generate-cmdlist.sh: do not shell out to "sed"
Johannes Sixt (2):
  generate-cmdlist.sh: spawn fewer processes
  generate-cmdlist.sh: replace for loop by printf's auto-repeat feature
Ævar Arnfjörð Bjarmason (7):
  command-list.txt: sort with "LC_ALL=C sort"
  generate-cmdlist.sh: trivial whitespace change
  generate-cmdlist.sh: don't call get_categories() from category_list()
  generate-cmdlist.sh: run "grep | sort", not "sort | grep"
  generate-cmdlist.sh: stop sorting category lines
  generate-cmdlist.sh: replace "grep' invocation with a shell version
  generate-cmdlist.sh: replace "cut", "tr" and "grep" with pure-shell
 command-list.txt    | 20 +++++++-------
 generate-cmdlist.sh | 66 ++++++++++++++++++++++++++-------------------
 2 files changed, 48 insertions(+), 38 deletions(-)
Range-diff against v1:
 1:  96885282988 =  1:  96885282988 command-list.txt: sort with "LC_ALL=C sort"
 2:  5e8fef90e42 =  2:  5e8fef90e42 generate-cmdlist.sh: trivial whitespace change
 3:  6b4de6a6088 =  3:  6b4de6a6088 generate-cmdlist.sh: spawn fewer processes
 4:  074685cf714 =  4:  074685cf714 generate-cmdlist.sh: don't call get_categories() from category_list()
 5:  f01c1fd8088 =  5:  f01c1fd8088 generate-cmdlist.sh: run "grep | sort", not "sort | grep"
 6:  e0b11514b8d =  6:  e0b11514b8d generate-cmdlist.sh: replace for loop by printf's auto-repeat feature
 7:  0c6f9b80d3b <  -:  ----------- Makefile: stop having command-list.h depend on a wildcard
 8:  23d4cc77b6c <  -:  ----------- Makefile: assert correct generate-cmdlist.sh output
 -:  ----------- >  7:  f2f37c2963b generate-cmdlist.sh: stop sorting category lines
 -:  ----------- >  8:  83318d6c0da generate-cmdlist.sh: do not shell out to "sed"
 -:  ----------- >  9:  7903dd1f8c2 generate-cmdlist.sh: replace "grep' invocation with a shell version
 -:  ----------- > 10:  e10a43756d1 generate-cmdlist.sh: replace "cut", "tr" and "grep" with pure-shell
-- 
2.33.1.1505.g075a284c562
Previous: Eric SunshineNext: Ævar Arnfjörð Bjarmason
Message 35 of 87 in “Why the Makefile is so eager to re-build & re-link”
  1. Ævar Arnfjörð BjarmasonJun 24, 2021
  2. Jeff KingJun 24, 2021
  3. Ævar Arnfjörð BjarmasonJun 24, 2021
  4. Johannes SixtJun 24, 2021
  5. Ævar Arnfjörð BjarmasonJun 25, 2021
  6. Ævar Arnfjörð BjarmasonJun 25, 2021
  7. Jeff KingJun 29, 2021
  8. 0/8 Makefile: make command-list.h 2-5x as fast with -jNÆvar Arnfjörð Bjarmason, Oct 20, 2021
  9. 1/8 command-list.txt: sort with "LC_ALL=C sort"Ævar Arnfjörð Bjarmason, Oct 20, 2021
  10. 2/8 generate-cmdlist.sh: trivial whitespace changeÆvar Arnfjörð Bjarmason, Oct 20, 2021
  11. 3/8 generate-cmdlist.sh: spawn fewer processesÆvar Arnfjörð Bjarmason, Oct 20, 2021
  12. 4/8 generate-cmdlist.sh: don't call get_categories() from category_list()Ævar Arnfjörð Bjarmason, Oct 20, 2021
  13. 5/8 generate-cmdlist.sh: run "grep | sort", not "sort | grep"Ævar Arnfjörð Bjarmason, Oct 20, 2021
  14. 6/8 generate-cmdlist.sh: replace for loop by printf's auto-repeat featureÆvar Arnfjörð Bjarmason, Oct 20, 2021
  15. Jeff KingOct 21, 2021
  16. Jeff KingOct 21, 2021
  17. 7/8 Makefile: stop having command-list.h depend on a wildcardÆvar Arnfjörð Bjarmason, Oct 20, 2021
  18. Jeff KingOct 21, 2021
  19. Junio C HamanoOct 21, 2021
  20. Øystein WalleOct 21, 2021
  21. 8/8 Makefile: assert correct generate-cmdlist.sh outputÆvar Arnfjörð Bjarmason, Oct 20, 2021
  22. Jeff KingOct 20, 2021
  23. Taylor BlauOct 20, 2021
  24. Ævar Arnfjörð BjarmasonOct 20, 2021
  25. Jeff KingOct 20, 2021
  26. Ævar Arnfjörð BjarmasonOct 21, 2021
  27. Taylor BlauOct 21, 2021
  28. Ævar Arnfjörð BjarmasonOct 22, 2021
  29. Jeff KingOct 21, 2021
  30. Junio C HamanoOct 21, 2021
  31. Ævar Arnfjörð BjarmasonOct 22, 2021
  32. Jeff KingOct 22, 2021
  33. Ævar Arnfjörð BjarmasonOct 22, 2021
  34. Eric SunshineOct 21, 2021
  35. 00/10 Makefile: make generate-cmdlist.sh much fasterÆvar Arnfjörð Bjarmason, Oct 22, 2021
  36. 01/10 command-list.txt: sort with "LC_ALL=C sort"Ævar Arnfjörð Bjarmason, Oct 22, 2021
  37. Junio C HamanoOct 25, 2021
  38. Ævar Arnfjörð BjarmasonOct 25, 2021
  39. Junio C HamanoOct 25, 2021
  40. 02/10 generate-cmdlist.sh: trivial whitespace changeÆvar Arnfjörð Bjarmason, Oct 22, 2021
  41. 03/10 generate-cmdlist.sh: spawn fewer processesÆvar Arnfjörð Bjarmason, Oct 22, 2021
  42. 04/10 generate-cmdlist.sh: don't call get_categories() from category_list()Ævar Arnfjörð Bjarmason, Oct 22, 2021
  43. 05/10 generate-cmdlist.sh: run "grep | sort", not "sort | grep"Ævar Arnfjörð Bjarmason, Oct 22, 2021
  44. 06/10 generate-cmdlist.sh: replace for loop by printf's auto-repeat featureÆvar Arnfjörð Bjarmason, Oct 22, 2021
  45. 07/10 generate-cmdlist.sh: stop sorting category linesÆvar Arnfjörð Bjarmason, Oct 22, 2021
  46. Jeff KingOct 25, 2021
  47. 08/10 generate-cmdlist.sh: do not shell out to "sed"Ævar Arnfjörð Bjarmason, Oct 22, 2021
  48. Jeff KingOct 25, 2021
  49. Jeff KingOct 25, 2021
  50. 10/10 generate-cmdlist.sh: replace "cut", "tr" and "grep" with pure-shellÆvar Arnfjörð Bjarmason, Oct 22, 2021
  51. Junio C HamanoOct 23, 2021
  52. 09/10 generate-cmdlist.sh: replace "grep' invocation with a shell versionÆvar Arnfjörð Bjarmason, Oct 22, 2021
  53. Junio C HamanoOct 23, 2021
  54. Junio C HamanoOct 23, 2021
  55. Taylor BlauOct 22, 2021
  56. Junio C HamanoOct 23, 2021
  57. Jeff KingOct 25, 2021
  58. 00/10 generate-cmdlist.sh: make it (and "make") run fasterÆvar Arnfjörð Bjarmason, Nov 5, 2021
  59. 01/10 command-list.txt: sort with "LC_ALL=C sort"Ævar Arnfjörð Bjarmason, Nov 5, 2021
  60. Junio C HamanoNov 5, 2021
  61. Ævar Arnfjörð BjarmasonNov 6, 2021
  62. Junio C HamanoNov 8, 2021
  63. 02/10 generate-cmdlist.sh: trivial whitespace changeÆvar Arnfjörð Bjarmason, Nov 5, 2021
  64. 03/10 generate-cmdlist.sh: spawn fewer processesÆvar Arnfjörð Bjarmason, Nov 5, 2021
  65. Junio C HamanoNov 5, 2021
  66. Ævar Arnfjörð BjarmasonNov 6, 2021
  67. 04/10 generate-cmdlist.sh: don't call get_categories() from category_list()Ævar Arnfjörð Bjarmason, Nov 5, 2021
  68. 05/10 generate-cmdlist.sh: run "grep | sort", not "sort | grep"Ævar Arnfjörð Bjarmason, Nov 5, 2021
  69. 06/10 generate-cmdlist.sh: replace for loop by printf's auto-repeat featureÆvar Arnfjörð Bjarmason, Nov 5, 2021
  70. 07/10 generate-cmdlist.sh: stop sorting category linesÆvar Arnfjörð Bjarmason, Nov 5, 2021
  71. 08/10 generate-cmdlist.sh: do not shell out to "sed"Ævar Arnfjörð Bjarmason, Nov 5, 2021
  72. 09/10 generate-cmdlist.sh: replace "grep' invocation with a shell versionÆvar Arnfjörð Bjarmason, Nov 5, 2021
  73. 10/10 generate-cmdlist.sh: don't parse command-list.txt thriceÆvar Arnfjörð Bjarmason, Nov 5, 2021
  74. Felipe ContrerasJun 25, 2021
  75. Eric SunshineJun 29, 2021
  76. Øystein WalleJun 24, 2021
  77. Øystein WalleJun 24, 2021
  78. Ævar Arnfjörð BjarmasonJun 25, 2021
  79. Documentation/Makefile: don't re-build on 'git version' changesÆvar Arnfjörð Bjarmason, Jul 2, 2021
  80. Junio C HamanoJul 2, 2021
  81. Ævar Arnfjörð BjarmasonJul 3, 2021
  82. Junio C HamanoJul 5, 2021
  83. Felipe ContrerasJul 3, 2021
  84. Ævar Arnfjörð BjarmasonJul 3, 2021
  85. Felipe ContrerasJul 3, 2021
  86. Junio C HamanoJul 5, 2021
  87. Felipe ContrerasJul 6, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.