{"thread":{"id":"54769","subject":"[PATCH 01/17] doc: propose hooks managed by the config","startedAt":"2020-12-05T01:46:59Z","lastAt":"2021-03-10T19:31:12Z","messageCount":89,"participants":["Emily Shaffer","SZEDER Gábor","Phillip Wood","Josh Steadmon","Junio C Hamano","Ævar Arnfjörð Bjarmason","Jonathan Tan"],"isPatch":true,"patchVersion":1,"patchTotal":17},"messages":[{"id":"411451","messageId":"20201205014607.1464119-2-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 01/17] doc: propose hooks managed by the config","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:45:51Z","receivedAt":"2020-12-05T01:46:59Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Begin a design document for config-based hooks, managed via git-hook.\nFocus on an overview of the implementation and motivation for design\ndecisions. Briefly discuss the alternatives considered before this\npoint. Also, attempt to redefine terms to fit into a multihook world.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Since v4, addressed comments from Jonathan Tan about wording. However, I have\n not addressed AEvar's comments or done a full re-review of this document.\n I wanted to get the rest of the series out for initial review first.\n \n - Emily\n \n Since v4, addressed comments from Jonathan Tan about wording.\n\n Documentation/Makefile | 1 +\n .../technical/config-based-hooks.txt | 367 ++++++++++++++++++\n 2 files changed, 368 insertions(+)\n create mode 100644 Documentation/technical/config-based-hooks.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex 80d1908a44..58d6b3acbe 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -81,6 +81,7 @@ SP_ARTICLES += $(API_DOCS)\n TECH_DOCS += MyFirstContribution\n TECH_DOCS += MyFirstObjectWalk\n TECH_DOCS += SubmittingPatches\n+TECH_DOCS += technical/config-based-hooks\n TECH_DOCS += technical/hash-function-transition\n TECH_DOCS += technical/http-protocol\n TECH_DOCS += technical/index-format\ndiff --git a/Documentation/technical/config-based-hooks.txt b/Documentation/technical/config-based-hooks.txt\nnew file mode 100644\nindex 0000000000..dac391f505\n--- /dev/null\n+++ b/Documentation/technical/config-based-hooks.txt\n@@ -0,0 +1,367 @@\n+Configuration-based hook management\n+===================================\n+:sectanchors:\n+\n+[[motivation]]\n+== Motivation\n+\n+Replace the .git/hook/hookname path as the only source of hooks to execute;\n+allow users to define hooks using config files, in a way which is friendly to\n+users with multiple repos which have similar needs.\n+\n+Redefine \"hook\" as an event rather than a single script, allowing users to\n+perform unrelated actions on a single event.\n+\n+Take a step closer to safety when copying zipped Git repositories from untrusted\n+users by making it more apparent to users which scripts will be run during\n+normal Git operations.\n+\n+Make it easier for users to discover Git's hook feature and automate their\n+workflows.\n+\n+[[user-interfaces]]\n+== User interfaces\n+\n+[[config-schema]]\n+=== Config schema\n+\n+Hooks can be introduced by editing the configuration manually. There are two new\n+sections added, `hook` and `hookcmd`.\n+\n+[[config-schema-hook]]\n+==== `hook`\n+\n+Primarily contains subsections for each hook event. The order of variables in\n+these subsections defines the hook command execution order; hook commands can be\n+specified by setting the value directly to the command if no additional\n+configuration is needed, or by setting the value as the name of a `hookcmd`. If\n+Git does not find a `hookcmd` whose subsection matches the value of the given\n+command string, Git will try to execute the string directly. Hooks are executed\n+by passing the resolved command string to the shell. In the future, hook event\n+subsections could also contain per-hook-event settings; see\n+<> for more details.\n+\n+Also contains top-level hook execution settings, for example, `hook.runHookDir`\n+or `hook.disableAll`. (These settings are described more in\n+<>.)\n+\n+----\n+[hook \"pre-commit\"]\n+ command = perl-linter\n+ command = /usr/bin/git-secrets --pre-commit\n+\n+[hook \"pre-applypatch\"]\n+ command = perl-linter\n+ # for illustration purposes; error behavior isn't planned yet\n+ error = ignore\n+\n+[hook]\n+ runHookDir = interactive\n+----\n+\n+[[config-schema-hookcmd]]\n+==== `hookcmd`\n+\n+Defines a hook command and its attributes, which will be used when a hook event\n+occurs. Unqualified attributes are assumed to apply to this hook during all hook\n+events, but event-specific attributes can also be supplied. The example runs\n+`/usr/bin/lint-it --language=perl `, but for repos which\n+include this config, the hook command will be skipped for all events to which\n+it's normally subscribed _except_ `pre-commit`.\n+\n+----\n+[hookcmd \"perl-linter\"]\n+ command = /usr/bin/lint-it --language=perl\n+ skip = true\n+ # for illustration purposes; below hasn't been defined yet\n+ pre-commit-skip = false\n+----\n+\n+[[command-line-api]]\n+=== Command-line API\n+\n+Users should be able to view, reorder, and create hook commands via the command\n+line. External tools should be able to view a list of hooks in the correct order\n+to run.\n+\n+*`git hook list `*\n+\n+*`git hook list (--system|--global|--local|--worktree)`*\n+\n+*`git hook edit `*\n+\n+*`git hook add `*\n+\n+[[hook-editor]]\n+=== Hook editor\n+\n+The tool which is presented by `git hook edit `. Ideally, this\n+tool should be easier to use than manually editing the config, and then produce\n+a concise config afterwards. It may take a form similar to `git rebase\n+--interactive`.\n+\n+[[implementation]]\n+== Implementation\n+\n+[[library]]\n+=== Library\n+\n+`hook.c` and `hook.h` are responsible for interacting with the config files. In\n+the case when the code generating a hook event doesn't have special concerns\n+about how to run the hooks, the hook library will provide a basic API to call\n+all hooks in config order with an `strvec` provided by the code which\n+generates the hook event:\n+\n+*`int run_hooks(const char *hookname, struct strvec *args)`*\n+\n+This call includes the hook command provided by `run-command.h:find_hook()`;\n+eventually, this legacy hook will be gated by a config `hook.runHookDir`. The\n+config is checked against a number of cases:\n+\n+- \"no\": the legacy hook will not be run\n+- \"interactive\": Git will prompt the user before running the legacy hook\n+- \"warn\": Git will print a warning to stderr before running the legacy hook\n+- \"yes\" (default): Git will silently run the legacy hook\n+\n+In case this list is expanded in the future, if a value for `hook.runHookDir` is\n+given which Git does not recognize, Git should discard that config entry. For\n+example, if \"warn\" was specified at system level and \"junk\" was specified at\n+global level, Git would resolve the value to \"warn\"; if the only time the config\n+was set was to \"junk\", Git would use the default value of \"yes\".\n+\n+If the caller wants to do something more complicated, the hook library can also\n+provide a callback API:\n+\n+*`int for_each_hookcmd(const char *hookname, hookcmd_function *cb)`*\n+\n+Finally, to facilitate the builtin, the library will also provide the following\n+APIs to interact with the config:\n+\n+----\n+int set_hook_commands(const char *hookname, struct string_list *commands,\n+\tenum config_scope scope);\n+int set_hookcmd(const char *hookcmd, struct hookcmd options);\n+\n+int list_hook_commands(const char *hookname, struct string_list *commands);\n+int list_hooks_in_scope(enum config_scope scope, struct string_list *commands);\n+----\n+\n+`struct hookcmd` is expected to grow in size over time as more functionality is\n+added to hooks; so that other parts of the code don't need to understand the\n+config schema, `struct hookcmd` should contain logical values instead of string\n+pairs.\n+\n+----\n+struct hookcmd {\n+ const char *name;\n+ const char *command;\n+\n+ /* for illustration only; not planned at present */\n+ int parallelizable;\n+ const char *hookcmd_before;\n+ const char *hookcmd_after;\n+ enum recovery_action on_fail;\n+}\n+----\n+\n+[[builtin]]\n+=== Builtin\n+\n+`builtin/hook.c` is responsible for providing the frontend. It's responsible for\n+formatting user-provided data and then calling the library API to set the\n+configs as appropriate. The builtin frontend is not responsible for calling the\n+config directly, so that other areas of Git can rely on the hook library to\n+understand the most recent config schema for hooks.\n+\n+[[migration]]\n+=== Migration path\n+\n+[[stage-0]]\n+==== Stage 0\n+\n+Hooks are called by running `run-command.h:find_hook()` with the hookname and\n+executing the result. The hook library and builtin do not exist. Hooks only\n+exist as specially named scripts within `.git/hooks/`.\n+\n+[[stage-1]]\n+==== Stage 1\n+\n+`git hook list --porcelain ` is implemented. Users can replace their\n+`.git/hooks/` scripts with a trampoline based on `git hook list`'s\n+output. Modifier commands like `git hook add` and `git hook edit` can be\n+implemented around this time as well.\n+\n+[[stage-2]]\n+==== Stage 2\n+\n+`hook.h:run_hooks()` is taught to include `run-command.h:find_hook()` at the\n+end; calls to `find_hook()` are replaced with calls to `run_hooks()`. Users can\n+opt-in to config-based hooks simply by creating some in their config; otherwise\n+users should remain unaffected by the change.\n+\n+[[stage-3]]\n+==== Stage 3\n+\n+The call to `find_hook()` inside of `run_hooks()` learns to check for a config,\n+`hook.runHookDir`. Users can opt into managing their hooks completely via the\n+config this way.\n+\n+[[stage-4]]\n+==== Stage 4\n+\n+`.git/hooks` is removed from the template and the hook directory is considered\n+deprecated. To avoid breaking older repos, the default of `hook.runHookDir` is\n+not changed, and `find_hook()` is not removed.\n+\n+[[caveats]]\n+== Caveats\n+\n+[[security]]\n+=== Security and repo config\n+\n+Part of the motivation behind this refactor is to mitigate hooks as an attack\n+vector;footnote:[https://lore.kernel.org/git/20171002234517.GV19555@aiede.mtv.corp.google.com/]\n+however, as the design stands, users can still provide hooks in the repo-level\n+config, which is included when a repo is zipped and sent elsewhere. The\n+security of the repo-level config is still under discussion; this design\n+generally assumes the repo-level config is secure, which is not true yet. The\n+goal is to avoid an overcomplicated design to work around a problem which has\n+ceased to exist.\n+\n+[[ease-of-use]]\n+=== Ease of use\n+\n+The config schema is nontrivial; that's why it's important for the `git hook`\n+modifier commands to be usable. Contributors with UX expertise are encouraged to\n+share their suggestions.\n+\n+[[alternatives]]\n+== Alternative approaches\n+\n+A previous summary of alternatives exists in the\n+archives.footnote:[https://lore.kernel.org/git/20191116011125.GG22855@google.com]\n+\n+[[status-quo]]\n+=== Status quo\n+\n+Today users can implement multihooks themselves by using a \"trampoline script\"\n+as their hook, and pointing that script to a directory or list of other scripts\n+they wish to run.\n+\n+[[hook-directories]]\n+=== Hook directories\n+\n+Other contributors have suggested Git learn about the existence of a directory\n+such as `.git/hooks/.d` and execute those hooks in alphabetical order.\n+\n+[[comparison]]\n+=== Comparison table\n+\n+.Comparison of alternatives\n+|===\n+|Feature |Config-based hooks |Hook directories |Status quo\n+\n+|Supports multiple hooks\n+|Natively\n+|Natively\n+|With user effort\n+\n+|Safer for zipped repos\n+|A little\n+|No\n+|No\n+\n+|Previous hooks just work\n+|If configured\n+|Yes\n+|Yes\n+\n+|Can install one hook to many repos\n+|Yes\n+|No\n+|No\n+\n+|Discoverability\n+|Better (in `git help git`)\n+|Same as before\n+|Same as before\n+\n+|Hard to run unexpected hook\n+|If configured\n+|No\n+|No\n+|===\n+\n+[[future-work]]\n+== Future work\n+\n+[[execution-ordering]]\n+=== Execution ordering\n+\n+We may find that config order is insufficient for some users; for example,\n+config order makes it difficult to add a new hook to the system or global config\n+which runs at the end of the hook list. A new ordering schema should be:\n+\n+1) Specified by a `hook.order` config, so that users will not unexpectedly see\n+their order change;\n+\n+2) Either dependency or numerically based.\n+\n+Dependency-based ordering is prone to classic linked-list problems, like a\n+cycles and handling of missing dependencies. But, it paves the way for enabling\n+parallelization if some tasks truly depend on others.\n+\n+Numerical ordering makes it tricky for Git to generate suggested ordering\n+numbers for each command, but is easy to determine a definitive order.\n+\n+[[parallelization]]\n+=== Parallelization\n+\n+Users with many hooks might want to run them simultaneously, if the hooks don't\n+modify state; if one hook depends on another's output, then users will want to\n+specify those dependencies. If we decide to solve this problem, we may want to\n+look to modern build systems for inspiration on how to manage dependencies and\n+parallel tasks.\n+\n+[[securing-hookdir-hooks]]\n+=== Securing hookdir hooks\n+\n+With the design as written in this doc, it's still possible for a malicious user\n+to modify `.git/config` to include `hook.pre-receive.command = rm -rf /`, then\n+zip their repo and send it to another user. It may be necessary to teach Git to\n+only allow inlined hooks like this if they were configured outside of the local\n+scope (in other words, only run hookcmds, and only allow hookcmds to be\n+configured in global or system scope); or another approach, like a list of safe\n+projects, might be useful. It may also be sufficient (or at least useful) to\n+teach a `hook.disableAll` config or similar flag to the Git executable.\n+\n+[[submodule-inheritance]]\n+=== Submodule inheritance\n+\n+It's possible some submodules may want to run the identical set of hooks that\n+their superrepo runs. While a globally-configured hook set is helpful, it's not\n+a great solution for users who have multiple repos-with-submodules under the\n+same user. It would be useful for submodules to learn how to run hooks from\n+their superrepo's config, or inherit that hook setting.\n+\n+[[per-hook-event-settings]]\n+=== Per-hook-event settings\n+\n+It might be desirable to keep settings specifically for some hook events, but\n+not for others - for example, a user may wish to disable hookdir hooks for all\n+events but pre-commit, which they haven't had time to convert yet; or, a user\n+may wish for execution order settings to differ based on hook event. In that\n+case, it would be useful to set something like `hook.pre-commit.executionOrder`\n+which would not apply to the 'prepare-commit-msg' hook, for example.\n+\n+[[glossary]]\n+== Glossary\n+\n+*hook event*\n+\n+A point during Git's execution where user scripts may be run, for example,\n+_prepare-commit-msg_ or _pre-push_.\n+\n+*hook command*\n+\n+A user script or executable which will be run on one or more hook events.\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411452","messageId":"20201205014607.1464119-3-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 02/17] hook: scaffolding for git-hook subcommand","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:45:52Z","receivedAt":"2020-12-05T01:47:01Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Introduce infrastructure for a new subcommand, git-hook, which will be\nused to ease config-based hook management. This command will handle\nparsing configs to compose a list of hooks to run for a given event, as\nwell as adding or modifying hook configs in an interactive fashion.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Since v4, mainly changed to RUN_SETUP_GENTLY so that 'git hook list' can\n be executed outside of a repo.\n\n .gitignore | 1 +\n Documentation/git-hook.txt | 20 ++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/hook.c | 21 +++++++++++++++++++++\n git.c | 1 +\n t/t1360-config-based-hooks.sh | 11 +++++++++++\n 7 files changed, 56 insertions(+)\n create mode 100644 Documentation/git-hook.txt\n create mode 100644 builtin/hook.c\n create mode 100755 t/t1360-config-based-hooks.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex f22b7a4cf1..094f58a175 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -76,6 +76,7 @@\n /git-grep\n /git-hash-object\n /git-help\n+/git-hook\n /git-http-backend\n /git-http-fetch\n /git-http-push\ndiff --git a/Documentation/git-hook.txt b/Documentation/git-hook.txt\nnew file mode 100644\nindex 0000000000..9eeab0009d\n--- /dev/null\n+++ b/Documentation/git-hook.txt\n@@ -0,0 +1,20 @@\n+git-hook(1)\n+===========\n+\n+NAME\n+----\n+git-hook - Manage configured hooks\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git hook'\n+\n+DESCRIPTION\n+-----------\n+A placeholder command. Later, you will be able to list, add, and modify hooks\n+with this command.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex 45bce31016..6ef9c0ee4e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1100,6 +1100,7 @@ BUILTIN_OBJS += builtin/get-tar-commit-id.o\n BUILTIN_OBJS += builtin/grep.o\n BUILTIN_OBJS += builtin/hash-object.o\n BUILTIN_OBJS += builtin/help.o\n+BUILTIN_OBJS += builtin/hook.o\n BUILTIN_OBJS += builtin/index-pack.o\n BUILTIN_OBJS += builtin/init-db.o\n BUILTIN_OBJS += builtin/interpret-trailers.o\ndiff --git a/builtin.h b/builtin.h\nindex b6ce981b73..8df1d36a7a 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -163,6 +163,7 @@ int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);\n int cmd_grep(int argc, const char **argv, const char *prefix);\n int cmd_hash_object(int argc, const char **argv, const char *prefix);\n int cmd_help(int argc, const char **argv, const char *prefix);\n+int cmd_hook(int argc, const char **argv, const char *prefix);\n int cmd_index_pack(int argc, const char **argv, const char *prefix);\n int cmd_init_db(int argc, const char **argv, const char *prefix);\n int cmd_interpret_trailers(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/hook.c b/builtin/hook.c\nnew file mode 100644\nindex 0000000000..b2bbc84d4d\n--- /dev/null\n+++ b/builtin/hook.c\n@@ -0,0 +1,21 @@\n+#include \"cache.h\"\n+\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+\n+static const char * const builtin_hook_usage[] = {\n+\tN_(\"git hook\"),\n+\tNULL\n+};\n+\n+int cmd_hook(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct option builtin_hook_options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, builtin_hook_options,\n+\t\t\t builtin_hook_usage, 0);\n+\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex 4b7bd77b80..8e92b5d3f6 100644\n--- a/git.c\n+++ b/git.c\n@@ -525,6 +525,7 @@ static struct cmd_struct commands[] = {\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n \t{ \"hash-object\", cmd_hash_object },\n \t{ \"help\", cmd_help },\n+\t{ \"hook\", cmd_hook, RUN_SETUP_GENTLY },\n \t{ \"index-pack\", cmd_index_pack, RUN_SETUP_GENTLY | NO_PARSEOPT },\n \t{ \"init\", cmd_init_db },\n \t{ \"init-db\", cmd_init_db },\ndiff --git a/t/t1360-config-based-hooks.sh b/t/t1360-config-based-hooks.sh\nnew file mode 100755\nindex 0000000000..34b0df5216\n--- /dev/null\n+++ b/t/t1360-config-based-hooks.sh\n@@ -0,0 +1,11 @@\n+#!/bin/bash\n+\n+test_description='config-managed multihooks, including git-hook command'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'git hook command does not crash' '\n+\tgit hook\n+'\n+\n+test_done\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411453","messageId":"20201205014607.1464119-1-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201014232447.3050579-1-emilyshaffer@google.com","subject":"[PATCH v6 00/17] propose config-based hooks (part I)","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:45:50Z","receivedAt":"2020-12-05T01:47:03Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Hi folks, and thanks for the patience - I ran into many, many last-mile\nchallenges.\n\nI haven't addressed many comments on the design doc yet - I was keen to get the\n\"functionally complete\" implementation and conversion to the list.\n\nNext on my plate:\n - Update the design doc to make sense with what's in the implementation.\n - A blog post! How to set up new hooks, why they're neat, etc.\n - We seem to have some Googlers interested in trying it out internally, so\n I'm hoping we'll gather and collate feedback from that soon too.\n - And of course addressing comments on this series.\n\nThanks!\n - Emily\n\nEmily Shaffer (17):\n doc: propose hooks managed by the config\n hook: scaffolding for git-hook subcommand\n hook: add list command\n hook: include hookdir hook in list\n hook: respect hook.runHookDir\n hook: implement hookcmd..skip\n parse-options: parse into strvec\n hook: add 'run' subcommand\n hook: replace find_hook() with hook_exists()\n hook: support passing stdin to hooks\n run-command: allow stdin for run_processes_parallel\n hook: allow parallel hook execution\n hook: allow specifying working directory for hooks\n run-command: add stdin callback for parallelization\n hook: provide stdin by string_list or callback\n run-command: allow capturing of collated output\n hooks: allow callers to capture output\n\n .gitignore | 1 +\n Documentation/Makefile | 1 +\n Documentation/config/hook.txt | 19 +\n Documentation/git-hook.txt | 118 +++++\n Documentation/technical/api-parse-options.txt | 5 +\n .../technical/config-based-hooks.txt | 367 +++++++++++++++\n Makefile | 2 +\n builtin.h | 1 +\n builtin/bugreport.c | 4 +-\n builtin/fetch.c | 1 +\n builtin/hook.c | 174 ++++++++\n builtin/submodule--helper.c | 2 +-\n git.c | 1 +\n hook.c | 417 ++++++++++++++++++\n hook.h | 154 +++++++\n parse-options-cb.c | 16 +\n parse-options.h | 4 +\n run-command.c | 85 +++-\n run-command.h | 31 ++\n submodule.c | 1 +\n t/helper/test-run-command.c | 46 +-\n t/t0061-run-command.sh | 37 ++\n t/t1360-config-based-hooks.sh | 256 +++++++++++\n 23 files changed, 1728 insertions(+), 15 deletions(-)\n create mode 100644 Documentation/config/hook.txt\n create mode 100644 Documentation/git-hook.txt\n create mode 100644 Documentation/technical/config-based-hooks.txt\n create mode 100644 builtin/hook.c\n create mode 100644 hook.c\n create mode 100644 hook.h\n create mode 100755 t/t1360-config-based-hooks.sh\n\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411454","messageId":"20201205014607.1464119-4-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 03/17] hook: add list command","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:45:53Z","receivedAt":"2020-12-05T01:47:08Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Teach 'git hook list ', which checks the known configs in\norder to create an ordered list of hooks to run on a given hook event.\n\nMultiple commands can be specified for a given hook by providing\nmultiple \"hook..command = \" lines. Hooks will be\nrun in config order. If more properties need to be set on a given hook\nin the future, commands can also be specified by providing\n\"hook..command = \", as well as a \"[hookcmd\n]\" subsection; at minimum, this subsection must contain a\n\"hookcmd..command = \" line.\n\nFor example:\n\n $ git config --list | grep ^hook\n hook.pre-commit.command=baz\n hook.pre-commit.command=~/bar.sh\n hookcmd.baz.command=~/baz/from/hookcmd.sh\n\n $ git hook list pre-commit\n global: ~/baz/from/hookcmd.sh\n local: ~/bar.sh\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Since v4, updated the sample in the commit message to reflect reality better.\n \n Since v4, more work on the documentation. Also a slight change to the\n output format (space instead of tab).\n\n Documentation/config/hook.txt | 9 +++\n Documentation/git-hook.txt | 59 ++++++++++++++++-\n Makefile | 1 +\n builtin/hook.c | 56 +++++++++++++++--\n hook.c | 115 ++++++++++++++++++++++++++++++++++\n hook.h | 26 ++++++++\n t/t1360-config-based-hooks.sh | 81 +++++++++++++++++++++++-\n 7 files changed, 338 insertions(+), 9 deletions(-)\n create mode 100644 Documentation/config/hook.txt\n create mode 100644 hook.c\n create mode 100644 hook.h\n\ndiff --git a/Documentation/config/hook.txt b/Documentation/config/hook.txt\nnew file mode 100644\nindex 0000000000..71449ecbc7\n--- /dev/null\n+++ b/Documentation/config/hook.txt\n@@ -0,0 +1,9 @@\n+hook..command::\n+\tA command to execute during the hook event. This can be an\n+\texecutable on your device, a oneliner for your shell, or the name of a\n+\thookcmd. See linkgit:git-hook[1].\n+\n+hookcmd..command::\n+\tA command to execute during a hook for which has been specified\n+\tas a command. This can be an executable on your device or a oneliner for\n+\tyour shell. See linkgit:git-hook[1].\ndiff --git a/Documentation/git-hook.txt b/Documentation/git-hook.txt\nindex 9eeab0009d..f19875ed68 100644\n--- a/Documentation/git-hook.txt\n+++ b/Documentation/git-hook.txt\n@@ -8,12 +8,65 @@ git-hook - Manage configured hooks\n SYNOPSIS\n --------\n [verse]\n-'git hook'\n+'git hook' list \n \n DESCRIPTION\n -----------\n-A placeholder command. Later, you will be able to list, add, and modify hooks\n-with this command.\n+You can list configured hooks with this command. Later, you will be able to run,\n+add, and modify hooks with this command.\n+\n+This command parses the default configuration files for sections `hook` and\n+`hookcmd`. `hook` is used to describe the commands which will be run during a\n+particular hook event; commands are run in the order Git encounters them during\n+the configuration parse (see linkgit:git-config[1]). `hookcmd` is used to\n+describe attributes of a specific command. If additional attributes don't need\n+to be specified, a command to run can be specified directly in the `hook`\n+section; if a `hookcmd` by that name isn't found, Git will attempt to run the\n+provided value directly. For example:\n+\n+Global config\n+----\n+ [hook \"post-commit\"]\n+ command = \"linter\"\n+ command = \"~/typocheck.sh\"\n+\n+ [hookcmd \"linter\"]\n+ command = \"/bin/linter --c\"\n+----\n+\n+Local config\n+----\n+ [hook \"prepare-commit-msg\"]\n+ command = \"linter\"\n+ [hook \"post-commit\"]\n+ command = \"python ~/run-test-suite.py\"\n+----\n+\n+With these configs, you'd then see:\n+\n+----\n+$ git hook list \"post-commit\"\n+global: /bin/linter --c\n+global: ~/typocheck.sh\n+local: python ~/run-test-suite.py\n+\n+$ git hook list \"prepare-commit-msg\"\n+local: /bin/linter --c\n+----\n+\n+COMMANDS\n+--------\n+\n+list ``::\n+\n+List the hooks which have been configured for ``. Hooks appear\n+in the order they should be run, and print the config scope where the relevant\n+`hook..command` was specified, not the `hookcmd` (if applicable).\n+This output is human-readable and the format is subject to change over time.\n+\n+CONFIGURATION\n+-------------\n+include::config/hook.txt[]\n \n GIT\n ---\ndiff --git a/Makefile b/Makefile\nindex 6ef9c0ee4e..4bf158c4f8 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -903,6 +903,7 @@ LIB_OBJS += grep.o\n LIB_OBJS += hashmap.o\n LIB_OBJS += help.o\n LIB_OBJS += hex.o\n+LIB_OBJS += hook.o\n LIB_OBJS += ident.o\n LIB_OBJS += json-writer.o\n LIB_OBJS += kwset.o\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex b2bbc84d4d..4d36de52f8 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -1,21 +1,69 @@\n #include \"cache.h\"\n \n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"hook.h\"\n #include \"parse-options.h\"\n+#include \"strbuf.h\"\n \n static const char * const builtin_hook_usage[] = {\n-\tN_(\"git hook\"),\n+\tN_(\"git hook list \"),\n \tNULL\n };\n \n-int cmd_hook(int argc, const char **argv, const char *prefix)\n+static int list(int argc, const char **argv, const char *prefix)\n {\n-\tstruct option builtin_hook_options[] = {\n+\tstruct list_head *head, *pos;\n+\tstruct hook *item;\n+\tstruct strbuf hookname = STRBUF_INIT;\n+\n+\tstruct option list_options[] = {\n \t\tOPT_END(),\n \t};\n \n-\targc = parse_options(argc, argv, prefix, builtin_hook_options,\n+\targc = parse_options(argc, argv, prefix, list_options,\n \t\t\t builtin_hook_usage, 0);\n \n+\tif (argc < 1) {\n+\t\tusage_msg_opt(_(\"You must specify a hook event name to list.\"),\n+\t\t\t builtin_hook_usage, list_options);\n+\t}\n+\n+\tstrbuf_addstr(&hookname, argv[0]);\n+\n+\thead = hook_list(&hookname);\n+\n+\tif (list_empty(head)) {\n+\t\tprintf(_(\"no commands configured for hook '%s'\\n\"),\n+\t\t hookname.buf);\n+\t\tstrbuf_release(&hookname);\n+\t\treturn 0;\n+\t}\n+\n+\tlist_for_each(pos, head) {\n+\t\titem = list_entry(pos, struct hook, list);\n+\t\tif (item)\n+\t\t\tprintf(\"%s: %s\\n\",\n+\t\t\t config_scope_name(item->origin),\n+\t\t\t item->command.buf);\n+\t}\n+\n+\tclear_hook_list(head);\n+\tstrbuf_release(&hookname);\n+\n \treturn 0;\n }\n+\n+int cmd_hook(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct option builtin_hook_options[] = {\n+\t\tOPT_END(),\n+\t};\n+\tif (argc < 2)\n+\t\tusage_with_options(builtin_hook_usage, builtin_hook_options);\n+\n+\tif (!strcmp(argv[1], \"list\"))\n+\t\treturn list(argc - 1, argv + 1, prefix);\n+\n+\tusage_with_options(builtin_hook_usage, builtin_hook_options);\n+}\ndiff --git a/hook.c b/hook.c\nnew file mode 100644\nindex 0000000000..937dc768c8\n--- /dev/null\n+++ b/hook.c\n@@ -0,0 +1,115 @@\n+#include \"cache.h\"\n+\n+#include \"hook.h\"\n+#include \"config.h\"\n+\n+void free_hook(struct hook *ptr)\n+{\n+\tif (ptr) {\n+\t\tstrbuf_release(&ptr->command);\n+\t\tfree(ptr);\n+\t}\n+}\n+\n+static void append_or_move_hook(struct list_head *head, const char *command)\n+{\n+\tstruct list_head *pos = NULL, *tmp = NULL;\n+\tstruct hook *to_add = NULL;\n+\n+\t/*\n+\t * remove the prior entry with this command; we'll replace it at the\n+\t * end.\n+\t */\n+\tlist_for_each_safe(pos, tmp, head) {\n+\t\tstruct hook *it = list_entry(pos, struct hook, list);\n+\t\tif (!strcmp(it->command.buf, command)) {\n+\t\t list_del(pos);\n+\t\t /* we'll simply move the hook to the end */\n+\t\t to_add = it;\n+\t\t}\n+\t}\n+\n+\tif (!to_add) {\n+\t\t/* adding a new hook, not moving an old one */\n+\t\tto_add = xmalloc(sizeof(struct hook));\n+\t\tstrbuf_init(&to_add->command, 0);\n+\t\tstrbuf_addstr(&to_add->command, command);\n+\t}\n+\n+\t/* re-set the scope so we show where an override was specified */\n+\tto_add->origin = current_config_scope();\n+\n+\tlist_add_tail(&to_add->list, pos);\n+}\n+\n+static void remove_hook(struct list_head *to_remove)\n+{\n+\tstruct hook *hook_to_remove = list_entry(to_remove, struct hook, list);\n+\tlist_del(to_remove);\n+\tfree_hook(hook_to_remove);\n+}\n+\n+void clear_hook_list(struct list_head *head)\n+{\n+\tstruct list_head *pos, *tmp;\n+\tlist_for_each_safe(pos, tmp, head)\n+\t\tremove_hook(pos);\n+}\n+\n+struct hook_config_cb\n+{\n+\tstruct strbuf *hookname;\n+\tstruct list_head *list;\n+};\n+\n+static int hook_config_lookup(const char *key, const char *value, void *cb_data)\n+{\n+\tstruct hook_config_cb *data = cb_data;\n+\tconst char *hook_key = data->hookname->buf;\n+\tstruct list_head *head = data->list;\n+\n+\tif (!strcmp(key, hook_key)) {\n+\t\tconst char *command = value;\n+\t\tstruct strbuf hookcmd_name = STRBUF_INIT;\n+\n+\t\t/* Check if a hookcmd with that name exists. */\n+\t\tstrbuf_addf(&hookcmd_name, \"hookcmd.%s.command\", command);\n+\t\tgit_config_get_value(hookcmd_name.buf, &command);\n+\n+\t\tif (!command) {\n+\t\t\tstrbuf_release(&hookcmd_name);\n+\t\t\tBUG(\"git_config_get_value overwrote a string it shouldn't have\");\n+\t\t}\n+\n+\t\t/*\n+\t\t * TODO: implement an option-getting callback, e.g.\n+\t\t * get configs by pattern hookcmd.$value.*\n+\t\t * for each key+value, do_callback(key, value, cb_data)\n+\t\t */\n+\n+\t\tappend_or_move_hook(head, command);\n+\n+\t\tstrbuf_release(&hookcmd_name);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+struct list_head* hook_list(const struct strbuf* hookname)\n+{\n+\tstruct strbuf hook_key = STRBUF_INIT;\n+\tstruct list_head *hook_head = xmalloc(sizeof(struct list_head));\n+\tstruct hook_config_cb cb_data = { &hook_key, hook_head };\n+\n+\tINIT_LIST_HEAD(hook_head);\n+\n+\tif (!hookname)\n+\t\treturn NULL;\n+\n+\tstrbuf_addf(&hook_key, \"hook.%s.command\", hookname->buf);\n+\n+\tgit_config(hook_config_lookup, (void*)&cb_data);\n+\n+\tstrbuf_release(&hook_key);\n+\treturn hook_head;\n+}\ndiff --git a/hook.h b/hook.h\nnew file mode 100644\nindex 0000000000..8ffc4f14b6\n--- /dev/null\n+++ b/hook.h\n@@ -0,0 +1,26 @@\n+#include \"config.h\"\n+#include \"list.h\"\n+#include \"strbuf.h\"\n+\n+struct hook\n+{\n+\tstruct list_head list;\n+\t/*\n+\t * Config file which holds the hook.*.command definition.\n+\t * (This has nothing to do with the hookcmd..* configs.)\n+\t */\n+\tenum config_scope origin;\n+\t/* The literal command to run. */\n+\tstruct strbuf command;\n+};\n+\n+/*\n+ * Provides a linked list of 'struct hook' detailing commands which should run\n+ * in response to the 'hookname' event, in execution order.\n+ */\n+struct list_head* hook_list(const struct strbuf *hookname);\n+\n+/* Free memory associated with a 'struct hook' */\n+void free_hook(struct hook *ptr);\n+/* Empties the list at 'head', calling 'free_hook()' on each entry */\n+void clear_hook_list(struct list_head *head);\ndiff --git a/t/t1360-config-based-hooks.sh b/t/t1360-config-based-hooks.sh\nindex 34b0df5216..6e4a3e763f 100755\n--- a/t/t1360-config-based-hooks.sh\n+++ b/t/t1360-config-based-hooks.sh\n@@ -4,8 +4,85 @@ test_description='config-managed multihooks, including git-hook command'\n \n . ./test-lib.sh\n \n-test_expect_success 'git hook command does not crash' '\n-\tgit hook\n+ROOT=\n+if test_have_prereq MINGW\n+then\n+\t# In Git for Windows, Unix-like paths work only in shell scripts;\n+\t# `git.exe`, however, will prefix them with the pseudo root directory\n+\t# (of the Unix shell). Let's accommodate for that.\n+\tROOT=\"$(cd / && pwd)\"\n+fi\n+\n+setup_hooks () {\n+\ttest_config hook.pre-commit.command \"/path/ghi\" --add\n+\ttest_config_global hook.pre-commit.command \"/path/def\" --add\n+}\n+\n+setup_hookcmd () {\n+\ttest_config hook.pre-commit.command \"abc\" --add\n+\ttest_config_global hookcmd.abc.command \"/path/abc\" --add\n+}\n+\n+test_expect_success 'git hook rejects commands without a mode' '\n+\ttest_must_fail git hook pre-commit\n+'\n+\n+\n+test_expect_success 'git hook rejects commands without a hookname' '\n+\ttest_must_fail git hook list\n+'\n+\n+test_expect_success 'git hook runs outside of a repo' '\n+\tsetup_hooks &&\n+\n+\tcat >expected <<-EOF &&\n+\tglobal: $ROOT/path/def\n+\tEOF\n+\n+\tnongit git config --list --global &&\n+\n+\tnongit git hook list pre-commit >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'git hook list orders by config order' '\n+\tsetup_hooks &&\n+\n+\tcat >expected <<-EOF &&\n+\tglobal: $ROOT/path/def\n+\tlocal: $ROOT/path/ghi\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'git hook list dereferences a hookcmd' '\n+\tsetup_hooks &&\n+\tsetup_hookcmd &&\n+\n+\tcat >expected <<-EOF &&\n+\tglobal: $ROOT/path/def\n+\tlocal: $ROOT/path/ghi\n+\tlocal: $ROOT/path/abc\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'git hook list reorders on duplicate commands' '\n+\tsetup_hooks &&\n+\n+\ttest_config hook.pre-commit.command \"/path/def\" --add &&\n+\n+\tcat >expected <<-EOF &&\n+\tlocal: $ROOT/path/ghi\n+\tlocal: $ROOT/path/def\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\ttest_cmp expected actual\n '\n \n test_done\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411455","messageId":"20201205014607.1464119-5-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 04/17] hook: include hookdir hook in list","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:45:54Z","receivedAt":"2020-12-05T01:47:10Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Historically, hooks are declared by placing an executable into\n$GIT_DIR/hooks/$HOOKNAME (or $HOOKDIR/$HOOKNAME). Although hooks taken\nfrom the config are more featureful than hooks placed in the $HOOKDIR,\nthose hooks should not stop working for users who already have them.\n\nLegacy hooks should be run directly, not in shell. We know that they are\na path to an executable, not a oneliner script - and running them\ndirectly takes care of path quoting concerns for us for free.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Newly split into its own commit since v4, and taking place much sooner.\n \n An unfortunate side effect of adding this support *before* the\n hook.runHookDir support is that the labels on the list are not clear -\n because we aren't yet flagging which hooks are from the hookdir versus\n the config. I suppose we could move the addition of that field to the\n struct hook up to this patch, but it didn't make a lot of sense to me to\n do it just for cosmetic purposes.\n\n builtin/hook.c | 16 ++++++++++++----\n hook.c | 15 +++++++++++++++\n hook.h | 1 +\n t/t1360-config-based-hooks.sh | 19 +++++++++++++++++++\n 4 files changed, 47 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex 4d36de52f8..45bbc83b2b 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -16,6 +16,7 @@ static int list(int argc, const char **argv, const char *prefix)\n \tstruct list_head *head, *pos;\n \tstruct hook *item;\n \tstruct strbuf hookname = STRBUF_INIT;\n+\tstruct strbuf hookdir_annotation = STRBUF_INIT;\n \n \tstruct option list_options[] = {\n \t\tOPT_END(),\n@@ -42,10 +43,17 @@ static int list(int argc, const char **argv, const char *prefix)\n \n \tlist_for_each(pos, head) {\n \t\titem = list_entry(pos, struct hook, list);\n-\t\tif (item)\n-\t\t\tprintf(\"%s: %s\\n\",\n-\t\t\t config_scope_name(item->origin),\n-\t\t\t item->command.buf);\n+\t\tif (item) {\n+\t\t\t/* Don't translate 'hookdir' - it matches the config */\n+\t\t\tprintf(\"%s: %s%s\\n\",\n+\t\t\t (item->from_hookdir\n+\t\t\t\t? \"hookdir\"\n+\t\t\t\t: config_scope_name(item->origin)),\n+\t\t\t item->command.buf,\n+\t\t\t (item->from_hookdir\n+\t\t\t\t? hookdir_annotation.buf\n+\t\t\t\t: \"\"));\n+\t\t}\n \t}\n \n \tclear_hook_list(head);\ndiff --git a/hook.c b/hook.c\nindex 937dc768c8..ffbdcfd987 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -2,6 +2,7 @@\n \n #include \"hook.h\"\n #include \"config.h\"\n+#include \"run-command.h\"\n \n void free_hook(struct hook *ptr)\n {\n@@ -34,6 +35,7 @@ static void append_or_move_hook(struct list_head *head, const char *command)\n \t\tto_add = xmalloc(sizeof(struct hook));\n \t\tstrbuf_init(&to_add->command, 0);\n \t\tstrbuf_addstr(&to_add->command, command);\n+\t\tto_add->from_hookdir = 0;\n \t}\n \n \t/* re-set the scope so we show where an override was specified */\n@@ -100,6 +102,7 @@ struct list_head* hook_list(const struct strbuf* hookname)\n \tstruct strbuf hook_key = STRBUF_INIT;\n \tstruct list_head *hook_head = xmalloc(sizeof(struct list_head));\n \tstruct hook_config_cb cb_data = { &hook_key, hook_head };\n+\tconst char *legacy_hook_path = NULL;\n \n \tINIT_LIST_HEAD(hook_head);\n \n@@ -110,6 +113,18 @@ struct list_head* hook_list(const struct strbuf* hookname)\n \n \tgit_config(hook_config_lookup, (void*)&cb_data);\n \n+\tif (have_git_dir())\n+\t\tlegacy_hook_path = find_hook(hookname->buf);\n+\n+\t/* Unconditionally add legacy hook, but annotate it. */\n+\tif (legacy_hook_path) {\n+\t\tstruct hook *legacy_hook;\n+\n+\t\tappend_or_move_hook(hook_head, absolute_path(legacy_hook_path));\n+\t\tlegacy_hook = list_entry(hook_head->prev, struct hook, list);\n+\t\tlegacy_hook->from_hookdir = 1;\n+\t}\n+\n \tstrbuf_release(&hook_key);\n \treturn hook_head;\n }\ndiff --git a/hook.h b/hook.h\nindex 8ffc4f14b6..5750634c83 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -12,6 +12,7 @@ struct hook\n \tenum config_scope origin;\n \t/* The literal command to run. */\n \tstruct strbuf command;\n+\tint from_hookdir;\n };\n \n /*\ndiff --git a/t/t1360-config-based-hooks.sh b/t/t1360-config-based-hooks.sh\nindex 6e4a3e763f..0f12af4659 100755\n--- a/t/t1360-config-based-hooks.sh\n+++ b/t/t1360-config-based-hooks.sh\n@@ -23,6 +23,14 @@ setup_hookcmd () {\n \ttest_config_global hookcmd.abc.command \"/path/abc\" --add\n }\n \n+setup_hookdir () {\n+\tmkdir .git/hooks\n+\twrite_script .git/hooks/pre-commit <<-EOF\n+\techo \\\"Legacy Hook\\\"\n+\tEOF\n+\ttest_when_finished rm -rf .git/hooks\n+}\n+\n test_expect_success 'git hook rejects commands without a mode' '\n \ttest_must_fail git hook pre-commit\n '\n@@ -85,4 +93,15 @@ test_expect_success 'git hook list reorders on duplicate commands' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'git hook list shows hooks from the hookdir' '\n+\tsetup_hookdir &&\n+\n+\tcat >expected <<-EOF &&\n+\thookdir: $(pwd)/.git/hooks/pre-commit\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411457","messageId":"20201205014607.1464119-7-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 06/17] hook: implement hookcmd..skip","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:45:56Z","receivedAt":"2020-12-05T01:47:39Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"If a user wants a specific repo to skip execution of a hook which is set\nat a global or system level, they can now do so by specifying 'skip' in\ntheir repo config:\n\n~/.gitconfig\n [hook.pre-commit]\n command = skippable-oneliner\n command = skippable-hookcmd\n\n [hookcmd.skippable-hookcmd]\n command = foo.sh\n\n$GIT_DIR/.git/config\n [hookcmd.skippable-oneliner]\n skip = true\n [hookcmd.skippable-hookcmd]\n skip = true\n\nLater it may make sense to add an option like\n\"hookcmd..-skip\" - but for simplicity, let's start\nwith a universal skip setting like this.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n In addition to being handy for turning off global hooks one project doesn't\n care about, this setting will be necessary much later for the 'proc-receive'\n hook, which can only cope with up to one hook being specified.\n \n New since v4.\n \n During the Google team's review club I was reminded about this whole\n 'skip' option I never implemented. It's true that it's impossible to\n exclude a given hook without this; however, I think I have some more\n work to do on it, so consider it RFC for now and tell me what you think\n :)\n - Emily\n \n During the Google team's review club this week I was reminded about this whole\n 'skip' option I never implemented. It's true that it's impossible to exclude\n a given hook without this; however, I think we have some more work to do on it,\n so consider it RFC for now and tell me what you think :)\n \n - Emily\n\n hook.c | 37 +++++++++++++++++++++++++----------\n t/t1360-config-based-hooks.sh | 23 ++++++++++++++++++++++\n 2 files changed, 50 insertions(+), 10 deletions(-)\n\ndiff --git a/hook.c b/hook.c\nindex 340e5a35c8..f4084e33c8 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -12,23 +12,24 @@ void free_hook(struct hook *ptr)\n \t}\n }\n \n-static void append_or_move_hook(struct list_head *head, const char *command)\n+static struct hook* find_hook_by_command(struct list_head *head, const char *command)\n {\n \tstruct list_head *pos = NULL, *tmp = NULL;\n-\tstruct hook *to_add = NULL;\n+\tstruct hook *found = NULL;\n \n-\t/*\n-\t * remove the prior entry with this command; we'll replace it at the\n-\t * end.\n-\t */\n \tlist_for_each_safe(pos, tmp, head) {\n \t\tstruct hook *it = list_entry(pos, struct hook, list);\n \t\tif (!strcmp(it->command.buf, command)) {\n \t\t list_del(pos);\n-\t\t /* we'll simply move the hook to the end */\n-\t\t to_add = it;\n+\t\t found = it;\n \t\t}\n \t}\n+\treturn found;\n+}\n+\n+static void append_or_move_hook(struct list_head *head, const char *command)\n+{\n+\tstruct hook *to_add = find_hook_by_command(head, command);\n \n \tif (!to_add) {\n \t\t/* adding a new hook, not moving an old one */\n@@ -41,7 +42,7 @@ static void append_or_move_hook(struct list_head *head, const char *command)\n \t/* re-set the scope so we show where an override was specified */\n \tto_add->origin = current_config_scope();\n \n-\tlist_add_tail(&to_add->list, pos);\n+\tlist_add_tail(&to_add->list, head);\n }\n \n static void remove_hook(struct list_head *to_remove)\n@@ -73,8 +74,18 @@ static int hook_config_lookup(const char *key, const char *value, void *cb_data)\n \tif (!strcmp(key, hook_key)) {\n \t\tconst char *command = value;\n \t\tstruct strbuf hookcmd_name = STRBUF_INIT;\n+\t\tint skip = 0;\n+\n+\t\t/*\n+\t\t * Check if we're removing that hook instead. Hookcmds are\n+\t\t * removed by name, and inlined hooks are removed by command\n+\t\t * content.\n+\t\t */\n+\t\tstrbuf_addf(&hookcmd_name, \"hookcmd.%s.skip\", command);\n+\t\tgit_config_get_bool(hookcmd_name.buf, &skip);\n \n \t\t/* Check if a hookcmd with that name exists. */\n+\t\tstrbuf_reset(&hookcmd_name);\n \t\tstrbuf_addf(&hookcmd_name, \"hookcmd.%s.command\", command);\n \t\tgit_config_get_value(hookcmd_name.buf, &command);\n \n@@ -89,7 +100,13 @@ static int hook_config_lookup(const char *key, const char *value, void *cb_data)\n \t\t * for each key+value, do_callback(key, value, cb_data)\n \t\t */\n \n-\t\tappend_or_move_hook(head, command);\n+\t\tif (skip) {\n+\t\t\tstruct hook *to_remove = find_hook_by_command(head, command);\n+\t\t\tif (to_remove)\n+\t\t\t\tremove_hook(&(to_remove->list));\n+\t\t} else {\n+\t\t\tappend_or_move_hook(head, command);\n+\t\t}\n \n \t\tstrbuf_release(&hookcmd_name);\n \t}\ndiff --git a/t/t1360-config-based-hooks.sh b/t/t1360-config-based-hooks.sh\nindex 91127a50a4..ebd3bc623f 100755\n--- a/t/t1360-config-based-hooks.sh\n+++ b/t/t1360-config-based-hooks.sh\n@@ -132,6 +132,29 @@ test_expect_success 'hook.runHookDir = warn is respected by list' '\n \ttest_i18ncmp expected actual\n '\n \n+test_expect_success 'git hook list removes skipped hookcmd' '\n+\tsetup_hookcmd &&\n+\ttest_config hookcmd.abc.skip \"true\" --add &&\n+\n+\tcat >expected <<-EOF &&\n+\tno commands configured for hook '\\''pre-commit'\\''\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\ttest_i18ncmp expected actual\n+'\n+\n+test_expect_success 'git hook list removes skipped inlined hook' '\n+\tsetup_hooks &&\n+\ttest_config hookcmd.\"$ROOT/path/ghi\".skip \"true\" --add &&\n+\n+\tcat >expected <<-EOF &&\n+\tglobal: $ROOT/path/def\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\ttest_cmp expected actual\n+'\n \n test_expect_success 'hook.runHookDir = interactive is respected by list' '\n \tsetup_hookdir &&\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411456","messageId":"20201205014607.1464119-6-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 05/17] hook: respect hook.runHookDir","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:45:55Z","receivedAt":"2020-12-05T01:47:40Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Include hooks specified in the hook directory in the list of hooks to\nrun. These hooks do need to be treated differently from config-specified\nones - they do not need to run in a shell, and later on may be disabled\nor warned about based on a config setting.\n\nBecause they are at least as local as the local config, we'll run them\nlast - to keep the hook execution order from global to local.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Newly split into its own commit since v4, and taking place much sooner.\n \n An unfortunate side effect of adding this support *before* the\n hook.runHookDir support is that the labels on the list are not clear -\n because we aren't yet flagging which hooks are from the hookdir versus\n the config. I suppose we could move the addition of that field to the\n struct hook up to this patch, but it didn't make a lot of sense to me to\n do it just for cosmetic purposes.\n\n Documentation/config/hook.txt | 5 ++++\n builtin/hook.c | 54 +++++++++++++++++++++++++++++++++--\n hook.c | 21 ++++++++++++++\n hook.h | 15 ++++++++++\n t/t1360-config-based-hooks.sh | 43 ++++++++++++++++++++++++++++\n 5 files changed, 135 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/hook.txt b/Documentation/config/hook.txt\nindex 71449ecbc7..75312754ae 100644\n--- a/Documentation/config/hook.txt\n+++ b/Documentation/config/hook.txt\n@@ -7,3 +7,8 @@ hookcmd..command::\n \tA command to execute during a hook for which has been specified\n \tas a command. This can be an executable on your device or a oneliner for\n \tyour shell. See linkgit:git-hook[1].\n+\n+hook.runHookDir::\n+\tControls how hooks contained in your hookdir are executed. Can be any of\n+\t\"yes\", \"warn\", \"interactive\", or \"no\". Defaults to \"yes\". See\n+\tlinkgit:git-hook[1] and linkgit:git-config[1] \"core.hooksPath\").\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex 45bbc83b2b..16324d4195 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -11,6 +11,8 @@ static const char * const builtin_hook_usage[] = {\n \tNULL\n };\n \n+static enum hookdir_opt should_run_hookdir;\n+\n static int list(int argc, const char **argv, const char *prefix)\n {\n \tstruct list_head *head, *pos;\n@@ -41,6 +43,26 @@ static int list(int argc, const char **argv, const char *prefix)\n \t\treturn 0;\n \t}\n \n+\tswitch (should_run_hookdir) {\n+\t\tcase hookdir_no:\n+\t\t\tstrbuf_addstr(&hookdir_annotation, _(\" (will not run)\"));\n+\t\t\tbreak;\n+\t\tcase hookdir_interactive:\n+\t\t\tstrbuf_addstr(&hookdir_annotation, _(\" (will prompt)\"));\n+\t\t\tbreak;\n+\t\tcase hookdir_warn:\n+\t\tcase hookdir_unknown:\n+\t\t\tstrbuf_addstr(&hookdir_annotation, _(\" (will warn)\"));\n+\t\t\tbreak;\n+\t\tcase hookdir_yes:\n+\t\t/*\n+\t\t * The default behavior should agree with\n+\t\t * hook.c:configured_hookdir_opt().\n+\t\t */\n+\t\tdefault:\n+\t\t\tbreak;\n+\t}\n+\n \tlist_for_each(pos, head) {\n \t\titem = list_entry(pos, struct hook, list);\n \t\tif (item) {\n@@ -64,14 +86,40 @@ static int list(int argc, const char **argv, const char *prefix)\n \n int cmd_hook(int argc, const char **argv, const char *prefix)\n {\n+\tconst char *run_hookdir = NULL;\n+\n \tstruct option builtin_hook_options[] = {\n+\t\tOPT_STRING(0, \"run-hookdir\", &run_hookdir, N_(\"option\"),\n+\t\t\t N_(\"what to do with hooks found in the hookdir\")),\n \t\tOPT_END(),\n \t};\n-\tif (argc < 2)\n+\n+\targc = parse_options(argc, argv, prefix, builtin_hook_options,\n+\t\t\t builtin_hook_usage, 0);\n+\n+\t/* after the parse, we should have \" \" */\n+\tif (argc < 1)\n \t\tusage_with_options(builtin_hook_usage, builtin_hook_options);\n \n-\tif (!strcmp(argv[1], \"list\"))\n-\t\treturn list(argc - 1, argv + 1, prefix);\n+\n+\t/* argument > config */\n+\tif (run_hookdir)\n+\t\tif (!strcmp(run_hookdir, \"no\"))\n+\t\t\tshould_run_hookdir = hookdir_no;\n+\t\telse if (!strcmp(run_hookdir, \"yes\"))\n+\t\t\tshould_run_hookdir = hookdir_yes;\n+\t\telse if (!strcmp(run_hookdir, \"warn\"))\n+\t\t\tshould_run_hookdir = hookdir_warn;\n+\t\telse if (!strcmp(run_hookdir, \"interactive\"))\n+\t\t\tshould_run_hookdir = hookdir_interactive;\n+\t\telse\n+\t\t\tdie(_(\"'%s' is not a valid option for --run-hookdir \"\n+\t\t\t \"(yes, warn, interactive, no)\"), run_hookdir);\n+\telse\n+\t\tshould_run_hookdir = configured_hookdir_opt();\n+\n+\tif (!strcmp(argv[0], \"list\"))\n+\t\treturn list(argc, argv, prefix);\n \n \tusage_with_options(builtin_hook_usage, builtin_hook_options);\n }\ndiff --git a/hook.c b/hook.c\nindex ffbdcfd987..340e5a35c8 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -97,6 +97,27 @@ static int hook_config_lookup(const char *key, const char *value, void *cb_data)\n \treturn 0;\n }\n \n+enum hookdir_opt configured_hookdir_opt(void)\n+{\n+\tconst char *key;\n+\tif (git_config_get_value(\"hook.runhookdir\", &key))\n+\t\treturn hookdir_yes; /* by default, just run it. */\n+\n+\tif (!strcmp(key, \"no\"))\n+\t\treturn hookdir_no;\n+\n+\tif (!strcmp(key, \"yes\"))\n+\t\treturn hookdir_yes;\n+\n+\tif (!strcmp(key, \"warn\"))\n+\t\treturn hookdir_warn;\n+\n+\tif (!strcmp(key, \"interactive\"))\n+\t\treturn hookdir_interactive;\n+\n+\treturn hookdir_unknown;\n+}\n+\n struct list_head* hook_list(const struct strbuf* hookname)\n {\n \tstruct strbuf hook_key = STRBUF_INIT;\ndiff --git a/hook.h b/hook.h\nindex 5750634c83..ca45d388d3 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -21,6 +21,21 @@ struct hook\n */\n struct list_head* hook_list(const struct strbuf *hookname);\n \n+enum hookdir_opt\n+{\n+\thookdir_no,\n+\thookdir_warn,\n+\thookdir_interactive,\n+\thookdir_yes,\n+\thookdir_unknown,\n+};\n+\n+/*\n+ * Provides the hookdir_opt specified in the config without consulting any\n+ * command line arguments.\n+ */\n+enum hookdir_opt configured_hookdir_opt(void);\n+\n /* Free memory associated with a 'struct hook' */\n void free_hook(struct hook *ptr);\n /* Empties the list at 'head', calling 'free_hook()' on each entry */\ndiff --git a/t/t1360-config-based-hooks.sh b/t/t1360-config-based-hooks.sh\nindex 0f12af4659..91127a50a4 100755\n--- a/t/t1360-config-based-hooks.sh\n+++ b/t/t1360-config-based-hooks.sh\n@@ -104,4 +104,47 @@ test_expect_success 'git hook list shows hooks from the hookdir' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'hook.runHookDir = no is respected by list' '\n+\tsetup_hookdir &&\n+\n+\ttest_config hook.runHookDir \"no\" &&\n+\n+\tcat >expected <<-EOF &&\n+\thookdir: $(pwd)/.git/hooks/pre-commit (will not run)\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\t# the hookdir annotation is translated\n+\ttest_i18ncmp expected actual\n+'\n+\n+test_expect_success 'hook.runHookDir = warn is respected by list' '\n+\tsetup_hookdir &&\n+\n+\ttest_config hook.runHookDir \"warn\" &&\n+\n+\tcat >expected <<-EOF &&\n+\thookdir: $(pwd)/.git/hooks/pre-commit (will warn)\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\t# the hookdir annotation is translated\n+\ttest_i18ncmp expected actual\n+'\n+\n+\n+test_expect_success 'hook.runHookDir = interactive is respected by list' '\n+\tsetup_hookdir &&\n+\n+\ttest_config hook.runHookDir \"interactive\" &&\n+\n+\tcat >expected <<-EOF &&\n+\thookdir: $(pwd)/.git/hooks/pre-commit (will prompt)\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\t# the hookdir annotation is translated\n+\ttest_i18ncmp expected actual\n+'\n+\n test_done\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411458","messageId":"20201205014607.1464119-9-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 08/17] hook: add 'run' subcommand","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:45:58Z","receivedAt":"2020-12-05T01:47:41Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"In order to enable hooks to be run as an external process, by a\nstandalone Git command, or by tools which wrap Git, provide an external\nmeans to run all configured hook commands for a given hook event.\n\nFor now, the hook commands will run in config order, in series. As\nalternate ordering or parallelism is supported in the future, we should\nadd knobs to use those to the command line as well.\n\nAs with the legacy hook implementation, all stdout generated by hook\ncommands is redirected to stderr. Piping from stdin is not yet\nsupported.\n\nLegacy hooks (those present in $GITDIR/hooks) are run at the end of the\nexecution list. For now, there is no way to disable them.\n\nUsers may wish to provide hook commands like 'git config\nhook.pre-commit.command \"~/linter.sh --pre-commit\"'. To enable this, the\ncontents of the 'hook.*.command' and 'hookcmd.*.command' strings are\nfirst split by space or quotes into an argv_array, then expanded with\n'expand_user_path()'.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Since v4, updated the docs, and did less local application of single\n quotes. In order for hookdir hooks to run successfully with a space in\n the path, though, they must not be run with 'sh -c'. So we can treat the\n hookdir hooks specially, and warn users via doc about special\n considerations for configured hooks with spaces in their path.\n\n Documentation/git-hook.txt | 31 +++++++++-\n builtin/hook.c | 48 ++++++++++++++-\n hook.c | 112 ++++++++++++++++++++++++++++++++++\n hook.h | 32 ++++++++++\n t/t1360-config-based-hooks.sh | 65 +++++++++++++++++++-\n 5 files changed, 281 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-hook.txt b/Documentation/git-hook.txt\nindex f19875ed68..18a817d832 100644\n--- a/Documentation/git-hook.txt\n+++ b/Documentation/git-hook.txt\n@@ -9,11 +9,12 @@ SYNOPSIS\n --------\n [verse]\n 'git hook' list \n+'git hook' run [(-e|--env)=...] [(-a|--arg)=...] \n \n DESCRIPTION\n -----------\n-You can list configured hooks with this command. Later, you will be able to run,\n-add, and modify hooks with this command.\n+You can list and run configured hooks with this command. Later, you will be able\n+to add and modify hooks with this command.\n \n This command parses the default configuration files for sections `hook` and\n `hookcmd`. `hook` is used to describe the commands which will be run during a\n@@ -64,6 +65,32 @@ in the order they should be run, and print the config scope where the relevant\n `hook..command` was specified, not the `hookcmd` (if applicable).\n This output is human-readable and the format is subject to change over time.\n \n+run [(-e|--env)=...] [(-a|--arg)=...] ``::\n+\n+Runs hooks configured for ``, in the same order displayed by `git\n+hook list`. Hooks configured this way are run prepended with `sh -c`, so paths\n+containing special characters or spaces should be wrapped in single quotes:\n+`command = '/my/path with spaces/script.sh' some args`.\n+\n+OPTIONS\n+-------\n+--run-hookdir::\n+\tOverrides the hook.runHookDir config. Must be 'yes', 'warn',\n+\t'interactive', or 'no'. Specifies how to handle hooks located in the Git\n+\thook directory (core.hooksPath).\n+\n+-a::\n+--arg::\n+\tOnly valid for `run`.\n++\n+Specify arguments to pass to every hook that is run.\n+\n+-e::\n+--env::\n+\tOnly valid for `run`.\n++\n+Specify environment variables to set for every hook that is run.\n+\n CONFIGURATION\n -------------\n include::config/hook.txt[]\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex 16324d4195..26f7050387 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -5,9 +5,11 @@\n #include \"hook.h\"\n #include \"parse-options.h\"\n #include \"strbuf.h\"\n+#include \"strvec.h\"\n \n static const char * const builtin_hook_usage[] = {\n \tN_(\"git hook list \"),\n+\tN_(\"git hook run [(-e|--env)=...] [(-a|--arg)=...] \"),\n \tNULL\n };\n \n@@ -84,6 +86,46 @@ static int list(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n+static int run(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct strbuf hookname = STRBUF_INIT;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tint rc = 0;\n+\n+\tstruct option run_options[] = {\n+\t\tOPT_STRVEC('e', \"env\", &opt.env, N_(\"var\"),\n+\t\t\t N_(\"environment variables for hook to use\")),\n+\t\tOPT_STRVEC('a', \"arg\", &opt.args, N_(\"args\"),\n+\t\t\t N_(\"argument to pass to hook\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\t/*\n+\t * While it makes sense to list hooks out-of-repo, it doesn't make sense\n+\t * to execute them. Hooks usually want to look at repository artifacts.\n+\t */\n+\tif (!have_git_dir())\n+\t\tusage_msg_opt(_(\"You must be in a Git repo to execute hooks.\"),\n+\t\t\t builtin_hook_usage, run_options);\n+\n+\targc = parse_options(argc, argv, prefix, run_options,\n+\t\t\t builtin_hook_usage, 0);\n+\n+\tif (argc < 1)\n+\t\tusage_msg_opt(_(\"You must specify a hook event to run.\"),\n+\t\t\t builtin_hook_usage, run_options);\n+\n+\tstrbuf_addstr(&hookname, argv[0]);\n+\topt.run_hookdir = should_run_hookdir;\n+\n+\trc = run_hooks(hookname.buf, &opt);\n+\n+\tstrbuf_release(&hookname);\n+\trun_hooks_opt_clear(&opt);\n+\n+\treturn rc;\n+}\n+\n int cmd_hook(int argc, const char **argv, const char *prefix)\n {\n \tconst char *run_hookdir = NULL;\n@@ -95,10 +137,10 @@ int cmd_hook(int argc, const char **argv, const char *prefix)\n \t};\n \n \targc = parse_options(argc, argv, prefix, builtin_hook_options,\n-\t\t\t builtin_hook_usage, 0);\n+\t\t\t builtin_hook_usage, PARSE_OPT_KEEP_UNKNOWN);\n \n \t/* after the parse, we should have \" \" */\n-\tif (argc < 1)\n+\tif (argc < 2)\n \t\tusage_with_options(builtin_hook_usage, builtin_hook_options);\n \n \n@@ -120,6 +162,8 @@ int cmd_hook(int argc, const char **argv, const char *prefix)\n \n \tif (!strcmp(argv[0], \"list\"))\n \t\treturn list(argc, argv, prefix);\n+\tif (!strcmp(argv[0], \"run\"))\n+\t\treturn run(argc, argv, prefix);\n \n \tusage_with_options(builtin_hook_usage, builtin_hook_options);\n }\ndiff --git a/hook.c b/hook.c\nindex f4084e33c8..c4595a2324 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -3,6 +3,7 @@\n #include \"hook.h\"\n #include \"config.h\"\n #include \"run-command.h\"\n+#include \"prompt.h\"\n \n void free_hook(struct hook *ptr)\n {\n@@ -135,6 +136,56 @@ enum hookdir_opt configured_hookdir_opt(void)\n \treturn hookdir_unknown;\n }\n \n+static int should_include_hookdir(const char *path, enum hookdir_opt cfg)\n+{\n+\tstruct strbuf prompt = STRBUF_INIT;\n+\t/*\n+\t * If the path doesn't exist, don't bother adding the empty hook and\n+\t * don't bother checking the config or prompting the user.\n+\t */\n+\tif (!path)\n+\t\treturn 0;\n+\n+\tswitch (cfg)\n+\t{\n+\t\tcase hookdir_no:\n+\t\t\treturn 0;\n+\t\tcase hookdir_unknown:\n+\t\t\tfprintf(stderr,\n+\t\t\t\t_(\"Unrecognized value for 'hook.runHookDir'. \"\n+\t\t\t\t \"Is there a typo? \"));\n+\t\t\t/* FALLTHROUGH */\n+\t\tcase hookdir_warn:\n+\t\t\tfprintf(stderr, _(\"Running legacy hook at '%s'\\n\"),\n+\t\t\t\tpath);\n+\t\t\treturn 1;\n+\t\tcase hookdir_interactive:\n+\t\t\tdo {\n+\t\t\t\t/*\n+\t\t\t\t * TRANSLATORS: Make sure to include [Y] and [n]\n+\t\t\t\t * in your translation. Only English input is\n+\t\t\t\t * accepted. Default option is \"yes\".\n+\t\t\t\t */\n+\t\t\t\tfprintf(stderr, _(\"Run '%s'? [Yn] \"), path);\n+\t\t\t\tgit_read_line_interactively(&prompt);\n+\t\t\t\tstrbuf_tolower(&prompt);\n+\t\t\t\tif (starts_with(prompt.buf, \"n\")) {\n+\t\t\t\t\tstrbuf_release(&prompt);\n+\t\t\t\t\treturn 0;\n+\t\t\t\t} else if (starts_with(prompt.buf, \"y\")) {\n+\t\t\t\t\tstrbuf_release(&prompt);\n+\t\t\t\t\treturn 1;\n+\t\t\t\t}\n+\t\t\t\t/* otherwise, we didn't understand the input */\n+\t\t\t} while (prompt.len); /* an empty reply means \"Yes\" */\n+\t\t\tstrbuf_release(&prompt);\n+\t\t\treturn 1;\n+\t\tcase hookdir_yes:\n+\t\tdefault:\n+\t\t\treturn 1;\n+\t}\n+}\n+\n struct list_head* hook_list(const struct strbuf* hookname)\n {\n \tstruct strbuf hook_key = STRBUF_INIT;\n@@ -166,3 +217,64 @@ struct list_head* hook_list(const struct strbuf* hookname)\n \tstrbuf_release(&hook_key);\n \treturn hook_head;\n }\n+\n+void run_hooks_opt_init(struct run_hooks_opt *o)\n+{\n+\tstrvec_init(&o->env);\n+\tstrvec_init(&o->args);\n+\to->run_hookdir = configured_hookdir_opt();\n+}\n+\n+void run_hooks_opt_clear(struct run_hooks_opt *o)\n+{\n+\tstrvec_clear(&o->env);\n+\tstrvec_clear(&o->args);\n+}\n+\n+int run_hooks(const char *hookname, struct run_hooks_opt *options)\n+{\n+\tstruct strbuf hookname_str = STRBUF_INIT;\n+\tstruct list_head *to_run, *pos = NULL, *tmp = NULL;\n+\tint rc = 0;\n+\n+\tif (!options)\n+\t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n+\n+\tstrbuf_addstr(&hookname_str, hookname);\n+\n+\tto_run = hook_list(&hookname_str);\n+\n+\tlist_for_each_safe(pos, tmp, to_run) {\n+\t\tstruct child_process hook_proc = CHILD_PROCESS_INIT;\n+\t\tstruct hook *hook = list_entry(pos, struct hook, list);\n+\n+\t\thook_proc.env = options->env.v;\n+\t\thook_proc.no_stdin = 1;\n+\t\thook_proc.stdout_to_stderr = 1;\n+\t\thook_proc.trace2_hook_name = hook->command.buf;\n+\t\thook_proc.use_shell = 1;\n+\n+\t\tif (hook->from_hookdir) {\n+\t\t if (!should_include_hookdir(hook->command.buf, options->run_hookdir))\n+\t\t\tcontinue;\n+\t\t /*\n+\t\t * Commands from the config could be oneliners, but we know\n+\t\t * for certain that hookdir commands are not.\n+\t\t */\n+\t\t hook_proc.use_shell = 0;\n+\t\t}\n+\n+\t\t/* add command */\n+\t\tstrvec_push(&hook_proc.args, hook->command.buf);\n+\n+\t\t/*\n+\t\t * add passed-in argv, without expanding - let the user get back\n+\t\t * exactly what they put in\n+\t\t */\n+\t\tstrvec_pushv(&hook_proc.args, options->args.v);\n+\n+\t\trc |= run_command(&hook_proc);\n+\t}\n+\n+\treturn rc;\n+}\ndiff --git a/hook.h b/hook.h\nindex ca45d388d3..d1c3d71e82 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -1,6 +1,7 @@\n #include \"config.h\"\n #include \"list.h\"\n #include \"strbuf.h\"\n+#include \"strvec.h\"\n \n struct hook\n {\n@@ -36,6 +37,37 @@ enum hookdir_opt\n */\n enum hookdir_opt configured_hookdir_opt(void);\n \n+struct run_hooks_opt\n+{\n+\t/* Environment vars to be set for each hook */\n+\tstruct strvec env;\n+\n+\t/* Args to be passed to each hook */\n+\tstruct strvec args;\n+\n+\t/*\n+\t * How should the hookdir be handled?\n+\t * Leave the RUN_HOOKS_OPT_INIT default in most cases; this only needs\n+\t * to be overridden if the user can override it at the command line.\n+\t */\n+\tenum hookdir_opt run_hookdir;\n+};\n+\n+#define RUN_HOOKS_OPT_INIT { \t\t\\\n+\t.env = STRVEC_INIT, \t\t\t\t\\\n+\t.args = STRVEC_INIT, \t\t\t\\\n+\t.run_hookdir = configured_hookdir_opt()\t\\\n+}\n+\n+void run_hooks_opt_init(struct run_hooks_opt *o);\n+void run_hooks_opt_clear(struct run_hooks_opt *o);\n+\n+/*\n+ * Runs all hooks associated to the 'hookname' event in order. Each hook will be\n+ * passed 'env' and 'args'.\n+ */\n+int run_hooks(const char *hookname, struct run_hooks_opt *options);\n+\n /* Free memory associated with a 'struct hook' */\n void free_hook(struct hook *ptr);\n /* Empties the list at 'head', calling 'free_hook()' on each entry */\ndiff --git a/t/t1360-config-based-hooks.sh b/t/t1360-config-based-hooks.sh\nindex ebd3bc623f..5b3003d59b 100755\n--- a/t/t1360-config-based-hooks.sh\n+++ b/t/t1360-config-based-hooks.sh\n@@ -115,7 +115,10 @@ test_expect_success 'hook.runHookDir = no is respected by list' '\n \n \tgit hook list pre-commit >actual &&\n \t# the hookdir annotation is translated\n-\ttest_i18ncmp expected actual\n+\ttest_i18ncmp expected actual &&\n+\n+\tgit hook run pre-commit 2>actual &&\n+\ttest_must_be_empty actual\n '\n \n test_expect_success 'hook.runHookDir = warn is respected by list' '\n@@ -129,6 +132,14 @@ test_expect_success 'hook.runHookDir = warn is respected by list' '\n \n \tgit hook list pre-commit >actual &&\n \t# the hookdir annotation is translated\n+\ttest_i18ncmp expected actual &&\n+\n+\tcat >expected <<-EOF &&\n+\tRunning legacy hook at '\\''$(pwd)/.git/hooks/pre-commit'\\''\n+\t\"Legacy Hook\"\n+\tEOF\n+\n+\tgit hook run pre-commit 2>actual &&\n \ttest_i18ncmp expected actual\n '\n \n@@ -156,7 +167,7 @@ test_expect_success 'git hook list removes skipped inlined hook' '\n \ttest_cmp expected actual\n '\n \n-test_expect_success 'hook.runHookDir = interactive is respected by list' '\n+test_expect_success 'hook.runHookDir = interactive is respected by list and run' '\n \tsetup_hookdir &&\n \n \ttest_config hook.runHookDir \"interactive\" &&\n@@ -167,7 +178,55 @@ test_expect_success 'hook.runHookDir = interactive is respected by list' '\n \n \tgit hook list pre-commit >actual &&\n \t# the hookdir annotation is translated\n-\ttest_i18ncmp expected actual\n+\ttest_i18ncmp expected actual &&\n+\n+\ttest_write_lines n | git hook run pre-commit 2>actual &&\n+\t! grep \"Legacy Hook\" actual &&\n+\n+\ttest_write_lines y | git hook run pre-commit 2>actual &&\n+\tgrep \"Legacy Hook\" actual\n+'\n+\n+test_expect_success 'inline hook definitions execute oneliners' '\n+\ttest_config hook.pre-commit.command \"echo \\\"Hello World\\\"\" &&\n+\n+\techo \"Hello World\" >expected &&\n+\n+\t# hooks are run with stdout_to_stderr = 1\n+\tgit hook run pre-commit 2>actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'inline hook definitions resolve paths' '\n+\twrite_script sample-hook.sh <<-EOF &&\n+\techo \\\"Sample Hook\\\"\n+\tEOF\n+\n+\ttest_when_finished \"rm sample-hook.sh\" &&\n+\n+\ttest_config hook.pre-commit.command \"\\\"$(pwd)/sample-hook.sh\\\"\" &&\n+\n+\techo \\\"Sample Hook\\\" >expected &&\n+\n+\t# hooks are run with stdout_to_stderr = 1\n+\tgit hook run pre-commit 2>actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'hookdir hook included in git hook run' '\n+\tsetup_hookdir &&\n+\n+\techo \\\"Legacy Hook\\\" >expected &&\n+\n+\t# hooks are run with stdout_to_stderr = 1\n+\tgit hook run pre-commit 2>actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'out-of-repo runs excluded' '\n+\tsetup_hooks &&\n+\n+\tnongit test_must_fail git hook run pre-commit\n '\n \n test_done\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411459","messageId":"20201205014607.1464119-10-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 09/17] hook: replace find_hook() with hook_exists()","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:45:59Z","receivedAt":"2020-12-05T01:47:43Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Add a helper to easily determine whether any hooks exist for a given\nhook event.\n\nMany callers want to check whether some state could be modified by a\nhook; that check should include the config-based hooks as well. Optimize\nby checking the config directly. Since commands which execute hooks\nmight want to take args to replace 'hook.runHookDir', let\n'hook_exists()' mirror the behavior of 'hook.runHookDir'.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Since v4, updated this commit to include bugreport as a builtin instead of\n as a standalone.\n \n Since v4, a little more nuance when deciding whether a hookdir hook can happen.\n\n builtin/bugreport.c | 4 ++--\n hook.c | 15 +++++++++++++++\n hook.h | 9 +++++++++\n 3 files changed, 26 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/bugreport.c b/builtin/bugreport.c\nindex 3ad4b9b62e..11043f4a22 100644\n--- a/builtin/bugreport.c\n+++ b/builtin/bugreport.c\n@@ -3,7 +3,7 @@\n #include \"strbuf.h\"\n #include \"help.h\"\n #include \"compat/compiler.h\"\n-#include \"run-command.h\"\n+#include \"hook.h\"\n \n \n static void get_system_info(struct strbuf *sys_info)\n@@ -82,7 +82,7 @@ static void get_populated_hooks(struct strbuf *hook_info, int nongit)\n \t}\n \n \tfor (i = 0; i < ARRAY_SIZE(hook); i++)\n-\t\tif (find_hook(hook[i]))\n+\t\tif (hook_exists(hook[i], configured_hookdir_opt()))\n \t\t\tstrbuf_addf(hook_info, \"%s\\n\", hook[i]);\n }\n \ndiff --git a/hook.c b/hook.c\nindex c4595a2324..a7a4abdcac 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -225,6 +225,21 @@ void run_hooks_opt_init(struct run_hooks_opt *o)\n \to->run_hookdir = configured_hookdir_opt();\n }\n \n+int hook_exists(const char *hookname, enum hookdir_opt should_run_hookdir)\n+{\n+\tconst char *value = NULL; /* throwaway */\n+\tstruct strbuf hook_key = STRBUF_INIT;\n+\n+\tint could_run_hookdir = (should_run_hookdir == hookdir_interactive ||\n+\t\t\t\tshould_run_hookdir == hookdir_warn ||\n+\t\t\t\tshould_run_hookdir == hookdir_yes)\n+\t\t\t\t&& !!find_hook(hookname);\n+\n+\tstrbuf_addf(&hook_key, \"hook.%s.command\", hookname);\n+\n+\treturn (!git_config_get_value(hook_key.buf, &value)) || could_run_hookdir;\n+}\n+\n void run_hooks_opt_clear(struct run_hooks_opt *o)\n {\n \tstrvec_clear(&o->env);\ndiff --git a/hook.h b/hook.h\nindex d1c3d71e82..94a25c7cd0 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -62,6 +62,15 @@ struct run_hooks_opt\n void run_hooks_opt_init(struct run_hooks_opt *o);\n void run_hooks_opt_clear(struct run_hooks_opt *o);\n \n+/*\n+ * Returns 1 if any hooks are specified in the config or if a hook exists in the\n+ * hookdir. Typically, invoke hook_exsts() like:\n+ * hook_exists(hookname, configured_hookdir_opt());\n+ * Like with run_hooks, if you take a --run-hookdir flag, reflect that\n+ * user-specified behavior here instead.\n+ */\n+int hook_exists(const char *hookname, enum hookdir_opt should_run_hookdir);\n+\n /*\n * Runs all hooks associated to the 'hookname' event in order. Each hook will be\n * passed 'env' and 'args'.\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411460","messageId":"20201205014607.1464119-8-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 07/17] parse-options: parse into strvec","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:45:57Z","receivedAt":"2020-12-05T01:47:47Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"parse-options already knows how to read into a string_list, and it knows\nhow to read into an strvec as a passthrough (that is, including the\nargument as well as its value). string_list and strvec serve similar\npurposes but are somewhat painful to convert between; so, let's teach\nparse-options to read values of string arguments directly into an\nstrvec without preserving the argument name.\n\nThis is useful if collecting generic arguments to pass through to\nanother command, for example, 'git hook run --arg \"--quiet\" --arg\n\"--format=pretty\" some-hook'. The resulting strvec would contain\n{ \"--quiet\", \"--format=pretty\" }.\n\nThe implementation is based on that of OPT_STRING_LIST.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Since v4, fixed one or two more places where I missed the argv_array->strvec\n rename.\n\n Documentation/technical/api-parse-options.txt | 5 +++++\n parse-options-cb.c | 16 ++++++++++++++++\n parse-options.h | 4 ++++\n 3 files changed, 25 insertions(+)\n\ndiff --git a/Documentation/technical/api-parse-options.txt b/Documentation/technical/api-parse-options.txt\nindex 5a60bbfa7f..679bd98629 100644\n--- a/Documentation/technical/api-parse-options.txt\n+++ b/Documentation/technical/api-parse-options.txt\n@@ -173,6 +173,11 @@ There are some macros to easily define options:\n \tThe string argument is stored as an element in `string_list`.\n \tUse of `--no-option` will clear the list of preceding values.\n \n+`OPT_STRVEC(short, long, &struct strvec, arg_str, description)`::\n+\tIntroduce an option with a string argument.\n+\tThe string argument is stored as an element in `strvec`.\n+\tUse of `--no-option` will clear the list of preceding values.\n+\n `OPT_INTEGER(short, long, &int_var, description)`::\n \tIntroduce an option with integer argument.\n \tThe integer is put into `int_var`.\ndiff --git a/parse-options-cb.c b/parse-options-cb.c\nindex 4542d4d3f9..c2451dfb1b 100644\n--- a/parse-options-cb.c\n+++ b/parse-options-cb.c\n@@ -207,6 +207,22 @@ int parse_opt_string_list(const struct option *opt, const char *arg, int unset)\n \treturn 0;\n }\n \n+int parse_opt_strvec(const struct option *opt, const char *arg, int unset)\n+{\n+\tstruct strvec *v = opt->value;\n+\n+\tif (unset) {\n+\t\tstrvec_clear(v);\n+\t\treturn 0;\n+\t}\n+\n+\tif (!arg)\n+\t\treturn -1;\n+\n+\tstrvec_push(v, arg);\n+\treturn 0;\n+}\n+\n int parse_opt_noop_cb(const struct option *opt, const char *arg, int unset)\n {\n \treturn 0;\ndiff --git a/parse-options.h b/parse-options.h\nindex 7030d8f3da..75cc8c7c96 100644\n--- a/parse-options.h\n+++ b/parse-options.h\n@@ -177,6 +177,9 @@ struct option {\n #define OPT_STRING_LIST(s, l, v, a, h) \\\n \t\t\t\t { OPTION_CALLBACK, (s), (l), (v), (a), \\\n \t\t\t\t (h), 0, &parse_opt_string_list }\n+#define OPT_STRVEC(s, l, v, a, h) \\\n+\t\t\t\t { OPTION_CALLBACK, (s), (l), (v), (a), \\\n+\t\t\t\t (h), 0, &parse_opt_strvec }\n #define OPT_UYN(s, l, v, h) { OPTION_CALLBACK, (s), (l), (v), NULL, \\\n \t\t\t\t (h), PARSE_OPT_NOARG, &parse_opt_tertiary }\n #define OPT_EXPIRY_DATE(s, l, v, h) \\\n@@ -296,6 +299,7 @@ int parse_opt_commits(const struct option *, const char *, int);\n int parse_opt_commit(const struct option *, const char *, int);\n int parse_opt_tertiary(const struct option *, const char *, int);\n int parse_opt_string_list(const struct option *, const char *, int);\n+int parse_opt_strvec(const struct option *, const char *, int);\n int parse_opt_noop_cb(const struct option *, const char *, int);\n enum parse_opt_result parse_opt_unknown_cb(struct parse_opt_ctx_t *ctx,\n \t\t\t\t\t const struct option *,\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411461","messageId":"20201205014607.1464119-12-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 11/17] run-command: allow stdin for run_processes_parallel","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:46:01Z","receivedAt":"2020-12-05T01:47:48Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"While it makes sense not to inherit stdin from the parent process to\navoid deadlocking, it's not necessary to completely ban stdin to\nchildren. An informed user should be able to configure stdin safely. By\nsetting `some_child.process.no_stdin=1` before calling `get_next_task()`\nwe provide a reasonable default behavior but enable users to set up\nstdin streaming for themselves during the callback.\n\n`some_child.process.stdout_to_stderr`, however, remains unmodifiable by\n`get_next_task()` - the rest of the run_processes_parallel() API depends\non child output in stderr.\n\nSigned-off-by: Emily Shaffer \n---\n run-command.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex ea4d0fb4b1..80c8c97bc1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1683,6 +1683,9 @@ static int pp_start_one(struct parallel_processes *pp)\n \tif (i == pp->max_processes)\n \t\tBUG(\"bookkeeping is hard\");\n \n+\t/* disallow by default, but allow users to set up stdin if they wish */\n+\tpp->children[i].process.no_stdin = 1;\n+\n \tcode = pp->get_next_task(&pp->children[i].process,\n \t\t\t\t &pp->children[i].err,\n \t\t\t\t pp->data,\n@@ -1694,7 +1697,6 @@ static int pp_start_one(struct parallel_processes *pp)\n \t}\n \tpp->children[i].process.err = -1;\n \tpp->children[i].process.stdout_to_stderr = 1;\n-\tpp->children[i].process.no_stdin = 1;\n \n \tif (start_command(&pp->children[i].process)) {\n \t\tcode = pp->start_failure(&pp->children[i].err,\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411462","messageId":"20201205014607.1464119-11-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 10/17] hook: support passing stdin to hooks","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:46:00Z","receivedAt":"2020-12-05T01:47:50Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Some hooks (such as post-rewrite) need to take input via stdin.\nPreviously, callers provided stdin to hooks by setting\nrun-command.h:child_process.in, which takes a FD. Callers would open the\nfile in question themselves before calling run-command(). However, since\nwe will now need to seek to the front of the file and read it again for\nevery hook which runs, hook.h:run_command() takes a path and handles FD\nmanagement itself. Since this file is opened for read only, it should\nnot prevent later parallel execution support.\n\nOn the frontend, this is supported by asking for a file path, rather\nthan by reading stdin. Reading directly from stdin would involve caching\nthe entire stdin (to memory or to disk) and reading it back from the\nbeginning to each hook. We'd want to support cases like insufficient\nmemory or storage for the file. While this may prove useful later, for\nnow the path of least resistance is to just ask the user to make this\ninterim file themselves.\n\nSigned-off-by: Emily Shaffer \n---\n Documentation/git-hook.txt | 11 +++++++++--\n builtin/hook.c | 5 ++++-\n hook.c | 7 ++++++-\n hook.h | 9 +++++++--\n t/t1360-config-based-hooks.sh | 24 ++++++++++++++++++++++++\n 5 files changed, 50 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-hook.txt b/Documentation/git-hook.txt\nindex 18a817d832..cce30a80d0 100644\n--- a/Documentation/git-hook.txt\n+++ b/Documentation/git-hook.txt\n@@ -9,7 +9,8 @@ SYNOPSIS\n --------\n [verse]\n 'git hook' list \n-'git hook' run [(-e|--env)=...] [(-a|--arg)=...] \n+'git hook' run [(-e|--env)=...] [(-a|--arg)=...] [--to-stdin=]\n+\t\n \n DESCRIPTION\n -----------\n@@ -65,7 +66,7 @@ in the order they should be run, and print the config scope where the relevant\n `hook..command` was specified, not the `hookcmd` (if applicable).\n This output is human-readable and the format is subject to change over time.\n \n-run [(-e|--env)=...] [(-a|--arg)=...] ``::\n+run [(-e|--env)=...] [(-a|--arg)=...] [--to-stdin=] ``::\n \n Runs hooks configured for ``, in the same order displayed by `git\n hook list`. Hooks configured this way are run prepended with `sh -c`, so paths\n@@ -91,6 +92,12 @@ Specify arguments to pass to every hook that is run.\n +\n Specify environment variables to set for every hook that is run.\n \n+--to-stdin::\n+\tOnly valid for `run`.\n++\n+Specify a file which will be streamed into stdin for every hook that is run.\n+Each hook will receive the entire file from beginning to EOF.\n+\n CONFIGURATION\n -------------\n include::config/hook.txt[]\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex 26f7050387..e45831e01d 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -9,7 +9,8 @@\n \n static const char * const builtin_hook_usage[] = {\n \tN_(\"git hook list \"),\n-\tN_(\"git hook run [(-e|--env)=...] [(-a|--arg)=...] \"),\n+\tN_(\"git hook run [(-e|--env)=...] [(-a|--arg)=...]\"\n+\t \"[--to-stdin=] \"),\n \tNULL\n };\n \n@@ -97,6 +98,8 @@ static int run(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"environment variables for hook to use\")),\n \t\tOPT_STRVEC('a', \"arg\", &opt.args, N_(\"args\"),\n \t\t\t N_(\"argument to pass to hook\")),\n+\t\tOPT_STRING(0, \"to-stdin\", &opt.path_to_stdin, N_(\"path\"),\n+\t\t\t N_(\"file to read into hooks' stdin\")),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/hook.c b/hook.c\nindex a7a4abdcac..c7fdf556fe 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -263,8 +263,13 @@ int run_hooks(const char *hookname, struct run_hooks_opt *options)\n \t\tstruct child_process hook_proc = CHILD_PROCESS_INIT;\n \t\tstruct hook *hook = list_entry(pos, struct hook, list);\n \n+\t\t/* reopen the file for stdin; run_command closes it. */\n+\t\tif (options->path_to_stdin)\n+\t\t\thook_proc.in = xopen(options->path_to_stdin, O_RDONLY);\n+\t\telse\n+\t\t\thook_proc.no_stdin = 1;\n+\n \t\thook_proc.env = options->env.v;\n-\t\thook_proc.no_stdin = 1;\n \t\thook_proc.stdout_to_stderr = 1;\n \t\thook_proc.trace2_hook_name = hook->command.buf;\n \t\thook_proc.use_shell = 1;\ndiff --git a/hook.h b/hook.h\nindex 94a25c7cd0..5184dcaa5a 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -51,11 +51,15 @@ struct run_hooks_opt\n \t * to be overridden if the user can override it at the command line.\n \t */\n \tenum hookdir_opt run_hookdir;\n+\n+\t/* Path to file which should be piped to stdin for each hook */\n+\tconst char *path_to_stdin;\n };\n \n #define RUN_HOOKS_OPT_INIT { \t\t\\\n-\t.env = STRVEC_INIT, \t\t\t\t\\\n+\t.env = STRVEC_INIT, \t\t\t\\\n \t.args = STRVEC_INIT, \t\t\t\\\n+\t.path_to_stdin = NULL,\t\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n@@ -73,7 +77,8 @@ int hook_exists(const char *hookname, enum hookdir_opt should_run_hookdir);\n \n /*\n * Runs all hooks associated to the 'hookname' event in order. Each hook will be\n- * passed 'env' and 'args'.\n+ * passed 'env' and 'args'. The file at 'stdin_path' will be closed and reopened\n+ * for each hook that runs.\n */\n int run_hooks(const char *hookname, struct run_hooks_opt *options);\n \ndiff --git a/t/t1360-config-based-hooks.sh b/t/t1360-config-based-hooks.sh\nindex 5b3003d59b..c672269ee4 100755\n--- a/t/t1360-config-based-hooks.sh\n+++ b/t/t1360-config-based-hooks.sh\n@@ -229,4 +229,28 @@ test_expect_success 'out-of-repo runs excluded' '\n \tnongit test_must_fail git hook run pre-commit\n '\n \n+test_expect_success 'stdin to multiple hooks' '\n+\tgit config --add hook.test.command \"xargs -P1 -I% echo a%\" &&\n+\tgit config --add hook.test.command \"xargs -P1 -I% echo b%\" &&\n+\ttest_when_finished \"test_unconfig hook.test.command\" &&\n+\n+\tcat >input <<-EOF &&\n+\t1\n+\t2\n+\t3\n+\tEOF\n+\n+\tcat >expected <<-EOF &&\n+\ta1\n+\ta2\n+\ta3\n+\tb1\n+\tb2\n+\tb3\n+\tEOF\n+\n+\tgit hook run --to-stdin=input test 2>actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411463","messageId":"20201205014607.1464119-13-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 12/17] hook: allow parallel hook execution","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:46:02Z","receivedAt":"2020-12-05T01:47:50Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"In many cases, there's no reason not to allow hooks to execute in\nparallel. run_processes_parallel() is well-suited - it's a task queue\nthat runs its housekeeping in series, which means users don't\nneed to worry about thread safety on their callback data. True\nmultithreaded execution with the async_* functions isn't necessary here.\nSynchronous hook execution can be achieved by only allowing 1 job to run\nat a time.\n\nTeach run_hooks() to use that function for simple hooks which don't\nrequire stdin or capture of stderr.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Per AEvar's request - parallel hook execution on day zero.\n \n In most ways run_processes_parallel() worked great for me - but it didn't\n have great support for hooks where we pipe to and from. I had to add this\n support later in the series.\n \n Since I modified an existing and in-use library I'd appreciate a keen look on\n these patches.\n \n - Emily\n\n Documentation/config/hook.txt | 5 ++\n Documentation/git-hook.txt | 15 +++-\n builtin/hook.c | 6 +-\n hook.c | 142 ++++++++++++++++++++++++++--------\n hook.h | 28 ++++++-\n 5 files changed, 158 insertions(+), 38 deletions(-)\n\ndiff --git a/Documentation/config/hook.txt b/Documentation/config/hook.txt\nindex 75312754ae..a423d13781 100644\n--- a/Documentation/config/hook.txt\n+++ b/Documentation/config/hook.txt\n@@ -12,3 +12,8 @@ hook.runHookDir::\n \tControls how hooks contained in your hookdir are executed. Can be any of\n \t\"yes\", \"warn\", \"interactive\", or \"no\". Defaults to \"yes\". See\n \tlinkgit:git-hook[1] and linkgit:git-config[1] \"core.hooksPath\").\n+\n+hook.jobs::\n+\tSpecifies how many hooks can be run simultaneously during parallelized\n+\thook execution. If unspecified, defaults to the number of processors on\n+\tthe current system.\ndiff --git a/Documentation/git-hook.txt b/Documentation/git-hook.txt\nindex cce30a80d0..c2678c61b2 100644\n--- a/Documentation/git-hook.txt\n+++ b/Documentation/git-hook.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n [verse]\n 'git hook' list \n 'git hook' run [(-e|--env)=...] [(-a|--arg)=...] [--to-stdin=]\n-\t\n+\t[(-j|--jobs) ] \n \n DESCRIPTION\n -----------\n@@ -66,7 +66,8 @@ in the order they should be run, and print the config scope where the relevant\n `hook..command` was specified, not the `hookcmd` (if applicable).\n This output is human-readable and the format is subject to change over time.\n \n-run [(-e|--env)=...] [(-a|--arg)=...] [--to-stdin=] ``::\n+run [(-e|--env)=...] [(-a|--arg)=...] [--to-stdin=]\n+\t[(-j|--jobs)]``::\n \n Runs hooks configured for ``, in the same order displayed by `git\n hook list`. Hooks configured this way are run prepended with `sh -c`, so paths\n@@ -98,6 +99,16 @@ Specify environment variables to set for every hook that is run.\n Specify a file which will be streamed into stdin for every hook that is run.\n Each hook will receive the entire file from beginning to EOF.\n \n+-j::\n+--jobs::\n+\tOnly valid for `run`.\n++\n+Specify how many hooks to run simultaneously. If this flag is not specified, use\n+the value of the `hook.jobs` config. If the config is not specified, use the\n+number of CPUs on the current system. Some hooks may be ineligible for\n+parallelization: for example, 'commit-msg' intends hooks modify the commit\n+message body and cannot be parallelized.\n+\n CONFIGURATION\n -------------\n include::config/hook.txt[]\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex e45831e01d..064a0fea29 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -10,7 +10,7 @@\n static const char * const builtin_hook_usage[] = {\n \tN_(\"git hook list \"),\n \tN_(\"git hook run [(-e|--env)=...] [(-a|--arg)=...]\"\n-\t \"[--to-stdin=] \"),\n+\t \"[--to-stdin=] [(-j|--jobs) ] \"),\n \tNULL\n };\n \n@@ -90,7 +90,7 @@ static int list(int argc, const char **argv, const char *prefix)\n static int run(int argc, const char **argv, const char *prefix)\n {\n \tstruct strbuf hookname = STRBUF_INIT;\n-\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT_ASYNC;\n \tint rc = 0;\n \n \tstruct option run_options[] = {\n@@ -100,6 +100,8 @@ static int run(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"argument to pass to hook\")),\n \t\tOPT_STRING(0, \"to-stdin\", &opt.path_to_stdin, N_(\"path\"),\n \t\t\t N_(\"file to read into hooks' stdin\")),\n+\t\tOPT_INTEGER('j', \"jobs\", &opt.jobs,\n+\t\t\t N_(\"run up to hooks simultaneously\")),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/hook.c b/hook.c\nindex c7fdf556fe..edea54f95c 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -136,6 +136,14 @@ enum hookdir_opt configured_hookdir_opt(void)\n \treturn hookdir_unknown;\n }\n \n+int configured_hook_jobs(void)\n+{\n+\tint n = online_cpus();\n+\tgit_config_get_int(\"hook.jobs\", &n);\n+\n+\treturn n;\n+}\n+\n static int should_include_hookdir(const char *path, enum hookdir_opt cfg)\n {\n \tstruct strbuf prompt = STRBUF_INIT;\n@@ -223,6 +231,7 @@ void run_hooks_opt_init(struct run_hooks_opt *o)\n \tstrvec_init(&o->env);\n \tstrvec_init(&o->args);\n \to->run_hookdir = configured_hookdir_opt();\n+\to->jobs = configured_hook_jobs();\n }\n \n int hook_exists(const char *hookname, enum hookdir_opt should_run_hookdir)\n@@ -246,11 +255,96 @@ void run_hooks_opt_clear(struct run_hooks_opt *o)\n \tstrvec_clear(&o->args);\n }\n \n+\n+static int pick_next_hook(struct child_process *cp,\n+\t\t\t struct strbuf *out,\n+\t\t\t void *pp_cb,\n+\t\t\t void **pp_task_cb)\n+{\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\n+\tstruct hook *hook = list_entry(hook_cb->run_me, struct hook, list);\n+\n+\tif (hook_cb->head == hook_cb->run_me)\n+\t\treturn 0;\n+\n+\tcp->env = hook_cb->options->env.v;\n+\tcp->stdout_to_stderr = 1;\n+\tcp->trace2_hook_name = hook->command.buf;\n+\n+\t/* reopen the file for stdin; run_command closes it. */\n+\tif (hook_cb->options->path_to_stdin) {\n+\t\tcp->no_stdin = 0;\n+\t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n+\t} else {\n+\t\tcp->no_stdin = 1;\n+\t}\n+\n+\t/*\n+\t * Commands from the config could be oneliners, but we know\n+\t * for certain that hookdir commands are not.\n+\t */\n+\tif (hook->from_hookdir)\n+\t\tcp->use_shell = 0;\n+\telse\n+\t\tcp->use_shell = 1;\n+\n+\t/* add command */\n+\tstrvec_push(&cp->args, hook->command.buf);\n+\n+\t/*\n+\t * add passed-in argv, without expanding - let the user get back\n+\t * exactly what they put in\n+\t */\n+\tstrvec_pushv(&cp->args, hook_cb->options->args.v);\n+\n+\t/* Provide context for errors if necessary */\n+\t*pp_task_cb = hook;\n+\n+\t/* Get the next entry ready */\n+\thook_cb->run_me = hook_cb->run_me->next;\n+\n+\treturn 1;\n+}\n+\n+static int notify_start_failure(struct strbuf *out,\n+\t\t\t\tvoid *pp_cb,\n+\t\t\t\tvoid *pp_task_cp)\n+{\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct hook *attempted = pp_task_cp;\n+\n+\t/* |= rc in cb */\n+\thook_cb->rc |= 1;\n+\n+\tstrbuf_addf(out, _(\"Couldn't start '%s', configured in '%s'\\n\"),\n+\t\t attempted->command.buf,\n+\t\t attempted->from_hookdir ? \"hookdir\"\n+\t\t \t: config_scope_name(attempted->origin));\n+\n+\t/* NEEDSWORK: if halt_on_error is desired, do it here. */\n+\treturn 0;\n+}\n+\n+static int notify_hook_finished(int result,\n+\t\t\t\tstruct strbuf *out,\n+\t\t\t\tvoid *pp_cb,\n+\t\t\t\tvoid *pp_task_cb)\n+{\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\n+\t/* |= rc in cb */\n+\thook_cb->rc |= result;\n+\n+\t/* NEEDSWORK: if halt_on_error is desired, do it here. */\n+\treturn 0;\n+}\n+\n int run_hooks(const char *hookname, struct run_hooks_opt *options)\n {\n \tstruct strbuf hookname_str = STRBUF_INIT;\n \tstruct list_head *to_run, *pos = NULL, *tmp = NULL;\n-\tint rc = 0;\n+\tstruct hook_cb_data cb_data = { 0, NULL, NULL, options };\n \n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n@@ -260,41 +354,23 @@ int run_hooks(const char *hookname, struct run_hooks_opt *options)\n \tto_run = hook_list(&hookname_str);\n \n \tlist_for_each_safe(pos, tmp, to_run) {\n-\t\tstruct child_process hook_proc = CHILD_PROCESS_INIT;\n \t\tstruct hook *hook = list_entry(pos, struct hook, list);\n \n-\t\t/* reopen the file for stdin; run_command closes it. */\n-\t\tif (options->path_to_stdin)\n-\t\t\thook_proc.in = xopen(options->path_to_stdin, O_RDONLY);\n-\t\telse\n-\t\t\thook_proc.no_stdin = 1;\n-\n-\t\thook_proc.env = options->env.v;\n-\t\thook_proc.stdout_to_stderr = 1;\n-\t\thook_proc.trace2_hook_name = hook->command.buf;\n-\t\thook_proc.use_shell = 1;\n-\n-\t\tif (hook->from_hookdir) {\n-\t\t if (!should_include_hookdir(hook->command.buf, options->run_hookdir))\n-\t\t\tcontinue;\n-\t\t /*\n-\t\t * Commands from the config could be oneliners, but we know\n-\t\t * for certain that hookdir commands are not.\n-\t\t */\n-\t\t hook_proc.use_shell = 0;\n-\t\t}\n-\n-\t\t/* add command */\n-\t\tstrvec_push(&hook_proc.args, hook->command.buf);\n+\t\tif (hook->from_hookdir &&\n+\t\t !should_include_hookdir(hook->command.buf, options->run_hookdir))\n+\t\t\t list_del(pos);\n+\t}\n \n-\t\t/*\n-\t\t * add passed-in argv, without expanding - let the user get back\n-\t\t * exactly what they put in\n-\t\t */\n-\t\tstrvec_pushv(&hook_proc.args, options->args.v);\n+\tcb_data.head = to_run;\n+\tcb_data.run_me = to_run->next;\n \n-\t\trc |= run_command(&hook_proc);\n-\t}\n+\trun_processes_parallel_tr2(options->jobs,\n+\t\t\t\t pick_next_hook,\n+\t\t\t\t notify_start_failure,\n+\t\t\t\t notify_hook_finished,\n+\t\t\t\t &cb_data,\n+\t\t\t\t \"hook\",\n+\t\t\t\t hookname);\n \n-\treturn rc;\n+\treturn cb_data.rc;\n }\ndiff --git a/hook.h b/hook.h\nindex 5184dcaa5a..f54568afe3 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -37,6 +37,9 @@ enum hookdir_opt\n */\n enum hookdir_opt configured_hookdir_opt(void);\n \n+/* Provides the number of threads to use for parallel hook execution. */\n+int configured_hook_jobs(void);\n+\n struct run_hooks_opt\n {\n \t/* Environment vars to be set for each hook */\n@@ -54,15 +57,38 @@ struct run_hooks_opt\n \n \t/* Path to file which should be piped to stdin for each hook */\n \tconst char *path_to_stdin;\n+\n+\t/* Number of threads to parallelize across */\n+\tint jobs;\n };\n \n-#define RUN_HOOKS_OPT_INIT { \t\t\\\n+/*\n+ * Callback provided to feed_pipe_fn and consume_sideband_fn.\n+ */\n+struct hook_cb_data {\n+\tint rc;\n+\tstruct list_head *head;\n+\tstruct list_head *run_me;\n+\tstruct run_hooks_opt *options;\n+};\n+\n+#define RUN_HOOKS_OPT_INIT_SYNC { \t\t\\\n \t.env = STRVEC_INIT, \t\t\t\\\n \t.args = STRVEC_INIT, \t\t\t\\\n \t.path_to_stdin = NULL,\t\t\t\\\n+\t.jobs = 1,\t\t\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n+#define RUN_HOOKS_OPT_INIT_ASYNC {\t\t\\\n+\t.env = STRVEC_INIT, \t\t\t\\\n+\t.args = STRVEC_INIT, \t\t\t\\\n+\t.path_to_stdin = NULL,\t\t\t\\\n+\t.jobs = configured_hook_jobs(),\t\t\\\n+\t.run_hookdir = configured_hookdir_opt()\t\\\n+}\n+\n+\n void run_hooks_opt_init(struct run_hooks_opt *o);\n void run_hooks_opt_clear(struct run_hooks_opt *o);\n \n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411464","messageId":"20201205014607.1464119-15-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 14/17] run-command: add stdin callback for parallelization","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:46:04Z","receivedAt":"2020-12-05T01:47:51Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"If a user of the run_processes_parallel() API wants to pipe a large\namount of information to stdin of each parallel command, that\ninformation could exceed the buffer of the pipe allocated for that\nprocess's stdin. Generally this is solved by repeatedly writing to\nchild_process.in between calls to start_command() and finish_command();\nrun_processes_parallel() did not provide users an opportunity to access\nchild_process at that time.\n\nBecause the data might be extremely large (for example, a list of all\nrefs received during a push from a client) simply taking a string_list\nor strbuf is not as scalable as using a callback; the rest of the\nrun_processes_parallel() API also uses callbacks, so making this feature\nmatch the rest of the API reduces mental load on the user.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Since run_processes_parallel() is used elsewhere, I'd appreciate a close\n look on this patch which modifies it. Thanks :)\n\n builtin/fetch.c | 1 +\n builtin/submodule--helper.c | 2 +-\n run-command.c | 54 +++++++++++++++++++++++++++++++++++--\n run-command.h | 17 +++++++++++-\n submodule.c | 1 +\n t/helper/test-run-command.c | 31 ++++++++++++++++++---\n t/t0061-run-command.sh | 30 +++++++++++++++++++++\n 7 files changed, 128 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex ecf8537605..5e153b5193 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1647,6 +1647,7 @@ static int fetch_multiple(struct string_list *list, int max_children)\n \t\tresult = run_processes_parallel_tr2(max_children,\n \t\t\t\t\t\t &fetch_next_remote,\n \t\t\t\t\t\t &fetch_failed_to_start,\n+\t\t\t\t\t\t NULL,\n \t\t\t\t\t\t &fetch_finished,\n \t\t\t\t\t\t &state,\n \t\t\t\t\t\t \"fetch\", \"parallel/fetch\");\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex c30896c897..bb623c1852 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -2294,7 +2294,7 @@ static int update_submodules(struct submodule_update_clone *suc)\n \tint i;\n \n \trun_processes_parallel_tr2(suc->max_jobs, update_clone_get_next_task,\n-\t\t\t\t update_clone_start_failure,\n+\t\t\t\t update_clone_start_failure, NULL,\n \t\t\t\t update_clone_task_finished, suc, \"submodule\",\n \t\t\t\t \"parallel/update\");\n \ndiff --git a/run-command.c b/run-command.c\nindex 80c8c97bc1..7b65c087f8 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1548,6 +1548,7 @@ struct parallel_processes {\n \n \tget_next_task_fn get_next_task;\n \tstart_failure_fn start_failure;\n+\tfeed_pipe_fn feed_pipe;\n \ttask_finished_fn task_finished;\n \n \tstruct {\n@@ -1575,6 +1576,13 @@ static int default_start_failure(struct strbuf *out,\n \treturn 0;\n }\n \n+static int default_feed_pipe(struct strbuf *pipe,\n+\t\t\t void *pp_cb,\n+\t\t\t void *pp_task_cb)\n+{\n+\treturn 1;\n+}\n+\n static int default_task_finished(int result,\n \t\t\t\t struct strbuf *out,\n \t\t\t\t void *pp_cb,\n@@ -1605,6 +1613,7 @@ static void pp_init(struct parallel_processes *pp,\n \t\t int n,\n \t\t get_next_task_fn get_next_task,\n \t\t start_failure_fn start_failure,\n+\t\t feed_pipe_fn feed_pipe,\n \t\t task_finished_fn task_finished,\n \t\t void *data)\n {\n@@ -1623,6 +1632,7 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->get_next_task = get_next_task;\n \n \tpp->start_failure = start_failure ? start_failure : default_start_failure;\n+\tpp->feed_pipe = feed_pipe ? feed_pipe : default_feed_pipe;\n \tpp->task_finished = task_finished ? task_finished : default_task_finished;\n \n \tpp->nr_processes = 0;\n@@ -1715,6 +1725,37 @@ static int pp_start_one(struct parallel_processes *pp)\n \treturn 0;\n }\n \n+static void pp_buffer_stdin(struct parallel_processes *pp)\n+{\n+\tint i;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t/* Buffer stdin for each pipe. */\n+\tfor (i = 0; i < pp->max_processes; i++) {\n+\t\tif (pp->children[i].state == GIT_CP_WORKING &&\n+\t\t pp->children[i].process.in > 0) {\n+\t\t\tint done;\n+\t\t\tstrbuf_reset(&sb);\n+\t\t\tdone = pp->feed_pipe(&sb, pp->data,\n+\t\t\t\t\t pp->children[i].data);\n+\t\t\tif (sb.len) {\n+\t\t\t\tif (write_in_full(pp->children[i].process.in,\n+\t\t\t\t\t sb.buf, sb.len) < 0) {\n+\t\t\t\t\tif (errno != EPIPE)\n+\t\t\t\t\t\tdie_errno(\"write\");\n+\t\t\t\t\tdone = 1;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tif (done) {\n+\t\t\t\tclose(pp->children[i].process.in);\n+\t\t\t\tpp->children[i].process.in = 0;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&sb);\n+}\n+\n static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n {\n \tint i;\n@@ -1779,6 +1820,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \t\tpp->nr_processes--;\n \t\tpp->children[i].state = GIT_CP_FREE;\n \t\tpp->pfd[i].fd = -1;\n+\t\tpp->children[i].process.in = 0;\n \t\tchild_process_init(&pp->children[i].process);\n \n \t\tif (i != pp->output_owner) {\n@@ -1812,6 +1854,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n int run_processes_parallel(int n,\n \t\t\t get_next_task_fn get_next_task,\n \t\t\t start_failure_fn start_failure,\n+\t\t\t feed_pipe_fn feed_pipe,\n \t\t\t task_finished_fn task_finished,\n \t\t\t void *pp_cb)\n {\n@@ -1820,7 +1863,9 @@ int run_processes_parallel(int n,\n \tint spawn_cap = 4;\n \tstruct parallel_processes pp;\n \n-\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\tpp_init(&pp, n, get_next_task, start_failure, feed_pipe, task_finished, pp_cb);\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1837,6 +1882,7 @@ int run_processes_parallel(int n,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n+\t\tpp_buffer_stdin(&pp);\n \t\tpp_buffer_stderr(&pp, output_timeout);\n \t\tpp_output(&pp);\n \t\tcode = pp_collect_finished(&pp);\n@@ -1848,11 +1894,15 @@ int run_processes_parallel(int n,\n \t}\n \n \tpp_cleanup(&pp);\n+\n+\tsigchain_pop(SIGPIPE);\n+\n \treturn 0;\n }\n \n int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \t\t\t start_failure_fn start_failure,\n+\t\t\t feed_pipe_fn feed_pipe,\n \t\t\t task_finished_fn task_finished, void *pp_cb,\n \t\t\t const char *tr2_category, const char *tr2_label)\n {\n@@ -1862,7 +1912,7 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \t\t\t\t ((n < 1) ? online_cpus() : n));\n \n \tresult = run_processes_parallel(n, get_next_task, start_failure,\n-\t\t\t\t\ttask_finished, pp_cb);\n+\t\t\t\t\tfeed_pipe, task_finished, pp_cb);\n \n \ttrace2_region_leave(tr2_category, tr2_label, NULL);\n \ndiff --git a/run-command.h b/run-command.h\nindex 6472b38bde..e058c0e2c8 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -436,6 +436,20 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * This callback is called repeatedly on every child process who requests\n+ * start_command() to create a pipe by setting child_process.in < 0.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel, and\n+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n+ * The contents of 'send' will be read into the pipe and passed to the pipe.\n+ *\n+ * Return nonzero to close the pipe.\n+ */\n+typedef int (*feed_pipe_fn)(struct strbuf *pipe,\n+\t\t\t void *pp_cb,\n+\t\t\t void *pp_task_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -470,10 +484,11 @@ typedef int (*task_finished_fn)(int result,\n int run_processes_parallel(int n,\n \t\t\t get_next_task_fn,\n \t\t\t start_failure_fn,\n+\t\t\t feed_pipe_fn,\n \t\t\t task_finished_fn,\n \t\t\t void *pp_cb);\n int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n-\t\t\t task_finished_fn, void *pp_cb,\n+\t\t\t feed_pipe_fn, task_finished_fn, void *pp_cb,\n \t\t\t const char *tr2_category, const char *tr2_label);\n \n #endif\ndiff --git a/submodule.c b/submodule.c\nindex b3bb59f066..953f41818c 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1638,6 +1638,7 @@ int fetch_populated_submodules(struct repository *r,\n \trun_processes_parallel_tr2(max_parallel_jobs,\n \t\t\t\t get_next_submodule,\n \t\t\t\t fetch_start_failure,\n+\t\t\t\t NULL,\n \t\t\t\t fetch_finish,\n \t\t\t\t &spf,\n \t\t\t\t \"submodule\", \"parallel/fetch\");\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 7ae03dc712..9348184d30 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -32,8 +32,13 @@ static int parallel_next(struct child_process *cp,\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->argv);\n+\tcp->in = d->in;\n+\tcp->no_stdin = d->no_stdin;\n \tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n \tnumber_callbacks++;\n+\n+\t*task_cb = xmalloc(sizeof(int));\n+\t*(int*)(*task_cb) = 2;\n \treturn 1;\n }\n \n@@ -55,6 +60,17 @@ static int task_finished(int result,\n \treturn 1;\n }\n \n+static int test_stdin(struct strbuf *pipe, void *cb, void *task_cb)\n+{\n+\tint *lines_remaining = task_cb;\n+\n+\tif (*lines_remaining)\n+\t\tstrbuf_addf(pipe, \"sample stdin %d\\n\", --(*lines_remaining));\n+\n+\treturn !(*lines_remaining);\n+}\n+\n+\n struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n@@ -185,7 +201,7 @@ static int testsuite(int argc, const char **argv)\n \t\tsuite.tests.nr, max_jobs);\n \n \tret = run_processes_parallel(max_jobs, next_test, test_failed,\n-\t\t\t\t test_finished, &suite);\n+\t\t\t\t test_stdin, test_finished, &suite);\n \n \tif (suite.failed.nr > 0) {\n \t\tret = 1;\n@@ -413,15 +429,22 @@ int cmd__run_command(int argc, const char **argv)\n \n \tif (!strcmp(argv[1], \"run-command-parallel\"))\n \t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, NULL, &proc));\n+\t\t\t\t\t NULL, NULL, NULL, &proc));\n \n \tif (!strcmp(argv[1], \"run-command-abort\"))\n \t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, task_finished, &proc));\n+\t\t\t\t\t NULL, NULL, task_finished, &proc));\n \n \tif (!strcmp(argv[1], \"run-command-no-jobs\"))\n \t\texit(run_processes_parallel(jobs, no_job,\n-\t\t\t\t\t NULL, task_finished, &proc));\n+\t\t\t\t\t NULL, NULL, task_finished, &proc));\n+\n+\tif (!strcmp(argv[1], \"run-command-stdin\")) {\n+\t\tproc.in = -1;\n+\t\tproc.no_stdin = 0;\n+\t\texit (run_processes_parallel(jobs, parallel_next, NULL,\n+\t\t\t\t\t test_stdin, NULL, &proc));\n+\t}\n \n \tfprintf(stderr, \"check usage\\n\");\n \treturn 1;\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 7d599675e3..3eb572e6cd 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -143,6 +143,36 @@ test_expect_success 'run_command runs in parallel with more tasks than jobs avai\n \ttest_cmp expect actual\n '\n \n+cat >expect <<-EOF\n+preloaded output of a child\n+listening for stdin:\n+sample stdin 1\n+sample stdin 0\n+preloaded output of a child\n+listening for stdin:\n+sample stdin 1\n+sample stdin 0\n+preloaded output of a child\n+listening for stdin:\n+sample stdin 1\n+sample stdin 0\n+preloaded output of a child\n+listening for stdin:\n+sample stdin 1\n+sample stdin 0\n+EOF\n+\n+test_expect_success 'run_command listens to stdin' '\n+\twrite_script stdin-script <<-\\EOF &&\n+\techo \"listening for stdin:\"\n+\twhile read line; do\n+\t\techo \"$line\"\n+\tdone actual &&\n+\ttest_cmp expect actual\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411465","messageId":"20201205014607.1464119-14-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 13/17] hook: allow specifying working directory for hooks","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:46:03Z","receivedAt":"2020-12-05T01:47:51Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Hooks like \"post-checkout\" require that hooks have a different working\ndirectory than the initial process. Pipe that directly through to struct\nchild_process.\n\nBecause we can just run 'git -C hook run ...' it shouldn't be\nnecessary to pipe this option through the frontend. In fact, this\nreduces the possibility of users running hooks which affect some part of\nthe filesystem outside of the repo in question.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Needed later for \"post-checkout\" conversion.\n\n hook.c | 1 +\n hook.h | 5 +++++\n 2 files changed, 6 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex edea54f95c..f0c052d847 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -271,6 +271,7 @@ static int pick_next_hook(struct child_process *cp,\n \tcp->env = hook_cb->options->env.v;\n \tcp->stdout_to_stderr = 1;\n \tcp->trace2_hook_name = hook->command.buf;\n+\tcp->dir = hook_cb->options->dir;\n \n \t/* reopen the file for stdin; run_command closes it. */\n \tif (hook_cb->options->path_to_stdin) {\ndiff --git a/hook.h b/hook.h\nindex f54568afe3..4aae8e2dbb 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -60,6 +60,9 @@ struct run_hooks_opt\n \n \t/* Number of threads to parallelize across */\n \tint jobs;\n+\n+\t/* Path to initial working directory for subprocess */\n+\tconst char *dir;\n };\n \n /*\n@@ -77,6 +80,7 @@ struct hook_cb_data {\n \t.args = STRVEC_INIT, \t\t\t\\\n \t.path_to_stdin = NULL,\t\t\t\\\n \t.jobs = 1,\t\t\t\t\\\n+\t.dir = NULL,\t\t\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n@@ -85,6 +89,7 @@ struct hook_cb_data {\n \t.args = STRVEC_INIT, \t\t\t\\\n \t.path_to_stdin = NULL,\t\t\t\\\n \t.jobs = configured_hook_jobs(),\t\t\\\n+\t.dir = NULL,\t\t\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411466","messageId":"20201205014607.1464119-18-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 17/17] hooks: allow callers to capture output","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:46:07Z","receivedAt":"2020-12-05T01:47:54Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Some server-side hooks will require capturing output to send over\nsideband instead of printing directly to stderr. Expose that capability.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n You can see this in practice in the conversions for some of the push hooks,\n like 'receive-pack'.\n\n hook.c | 2 +-\n hook.h | 10 ++++++++++\n 2 files changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/hook.c b/hook.c\nindex 1186ee41b3..78d7721b74 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -407,7 +407,7 @@ int run_hooks(const char *hookname, struct run_hooks_opt *options)\n \t\t\t\t pick_next_hook,\n \t\t\t\t notify_start_failure,\n \t\t\t\t options->feed_pipe,\n-\t\t\t\t NULL,\n+\t\t\t\t options->consume_sideband,\n \t\t\t\t notify_hook_finished,\n \t\t\t\t &cb_data,\n \t\t\t\t \"hook\",\ndiff --git a/hook.h b/hook.h\nindex ace26c637e..7059e0db77 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -76,6 +76,14 @@ struct run_hooks_opt\n \tfeed_pipe_fn feed_pipe;\n \tvoid *feed_pipe_ctx;\n \n+\t/*\n+\t * Populate this to capture output and prevent it from being printed to\n+\t * stderr. This will be passed directly through to\n+\t * run_command:run_parallel_processes(). See t/helper/test-run-command.c\n+\t * for an example.\n+\t */\n+\tconsume_sideband_fn consume_sideband;\n+\n \t/* Number of threads to parallelize across */\n \tint jobs;\n \n@@ -103,6 +111,7 @@ struct hook_cb_data {\n \t.str_stdin = STRING_LIST_INIT_DUP,\t\\\n \t.feed_pipe = NULL,\t\t\t\\\n \t.feed_pipe_ctx = NULL,\t\t\t\\\n+\t.consume_sideband = NULL,\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n@@ -115,6 +124,7 @@ struct hook_cb_data {\n \t.str_stdin = STRING_LIST_INIT_DUP,\t\\\n \t.feed_pipe = NULL,\t\t\t\\\n \t.feed_pipe_ctx = NULL,\t\t\t\\\n+\t.consume_sideband = NULL,\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411467","messageId":"20201205014607.1464119-16-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 15/17] hook: provide stdin by string_list or callback","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:46:05Z","receivedAt":"2020-12-05T01:48:00Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"In cases where a hook requires only a small amount of information via\nstdin, it should be simple for users to provide a string_list alone. But\nin more complicated cases where the stdin is too large to hold in\nmemory, let's provide a callback the users can populate line after line\nwith instead.\n\nSigned-off-by: Emily Shaffer \n---\n hook.c | 39 +++++++++++++++++++++++++++++++++++++++\n hook.h | 25 +++++++++++++++++++++++++\n 2 files changed, 64 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex f0c052d847..fbb49f241d 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -9,6 +9,8 @@ void free_hook(struct hook *ptr)\n {\n \tif (ptr) {\n \t\tstrbuf_release(&ptr->command);\n+\t\tif (ptr->feed_pipe_cb_data)\n+\t\t\tfree(ptr->feed_pipe_cb_data);\n \t\tfree(ptr);\n \t}\n }\n@@ -38,6 +40,7 @@ static void append_or_move_hook(struct list_head *head, const char *command)\n \t\tstrbuf_init(&to_add->command, 0);\n \t\tstrbuf_addstr(&to_add->command, command);\n \t\tto_add->from_hookdir = 0;\n+\t\tto_add->feed_pipe_cb_data = NULL;\n \t}\n \n \t/* re-set the scope so we show where an override was specified */\n@@ -253,9 +256,32 @@ void run_hooks_opt_clear(struct run_hooks_opt *o)\n {\n \tstrvec_clear(&o->env);\n \tstrvec_clear(&o->args);\n+\tstring_list_clear(&o->str_stdin, 0);\n }\n \n \n+static int pipe_from_string_list(struct strbuf *pipe, void *pp_cb, void *pp_task_cb)\n+{\n+\tint *item_idx;\n+\tstruct hook *ctx = pp_task_cb;\n+\tstruct string_list *to_pipe = &((struct hook_cb_data*)pp_cb)->options->str_stdin;\n+\n+\t/* Bootstrap the state manager if necessary. */\n+\tif (!ctx->feed_pipe_cb_data) {\n+\t\tctx->feed_pipe_cb_data = xmalloc(sizeof(unsigned int));\n+\t\t*(int*)ctx->feed_pipe_cb_data = 0;\n+\t}\n+\n+\titem_idx = ctx->feed_pipe_cb_data;\n+\n+\tif (*item_idx < to_pipe->nr) {\n+\t\tstrbuf_addf(pipe, \"%s\\n\", to_pipe->items[*item_idx].string);\n+\t\t(*item_idx)++;\n+\t\treturn 0;\n+\t}\n+\treturn 1;\n+}\n+\n static int pick_next_hook(struct child_process *cp,\n \t\t\t struct strbuf *out,\n \t\t\t void *pp_cb,\n@@ -277,6 +303,10 @@ static int pick_next_hook(struct child_process *cp,\n \tif (hook_cb->options->path_to_stdin) {\n \t\tcp->no_stdin = 0;\n \t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n+\t} else if (hook_cb->options->feed_pipe) {\n+\t\t/* ask for start_command() to make a pipe for us */\n+\t\tcp->in = -1;\n+\t\tcp->no_stdin = 0;\n \t} else {\n \t\tcp->no_stdin = 1;\n \t}\n@@ -350,6 +380,14 @@ int run_hooks(const char *hookname, struct run_hooks_opt *options)\n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n \n+\tif ((options->path_to_stdin && options->str_stdin.nr) ||\n+\t (options->path_to_stdin && options->feed_pipe) ||\n+\t (options->str_stdin.nr && options->feed_pipe))\n+\t\tBUG(\"choose only one method to populate stdin\");\n+\n+\tif (options->str_stdin.nr)\n+\t\toptions->feed_pipe = &pipe_from_string_list;\n+\n \tstrbuf_addstr(&hookname_str, hookname);\n \n \tto_run = hook_list(&hookname_str);\n@@ -368,6 +406,7 @@ int run_hooks(const char *hookname, struct run_hooks_opt *options)\n \trun_processes_parallel_tr2(options->jobs,\n \t\t\t\t pick_next_hook,\n \t\t\t\t notify_start_failure,\n+\t\t\t\t options->feed_pipe,\n \t\t\t\t notify_hook_finished,\n \t\t\t\t &cb_data,\n \t\t\t\t \"hook\",\ndiff --git a/hook.h b/hook.h\nindex 4aae8e2dbb..ace26c637e 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -2,6 +2,7 @@\n #include \"list.h\"\n #include \"strbuf.h\"\n #include \"strvec.h\"\n+#include \"run-command.h\"\n \n struct hook\n {\n@@ -14,6 +15,12 @@ struct hook\n \t/* The literal command to run. */\n \tstruct strbuf command;\n \tint from_hookdir;\n+\n+\t/*\n+\t * Use this to keep state for your feed_pipe_fn if you are using\n+\t * run_hooks_opt.feed_pipe. Otherwise, do not touch it.\n+\t */\n+\tvoid *feed_pipe_cb_data;\n };\n \n /*\n@@ -57,12 +64,24 @@ struct run_hooks_opt\n \n \t/* Path to file which should be piped to stdin for each hook */\n \tconst char *path_to_stdin;\n+\t/* Pipe each string to stdin, separated by newlines */\n+\tstruct string_list str_stdin;\n+\t/*\n+\t * Callback and state pointer to ask for more content to pipe to stdin.\n+\t * Will be called repeatedly, for each hook. See\n+\t * hook.c:pipe_from_stdin() for an example. Keep per-hook state in\n+\t * hook.feed_pipe_cb_data (per process). Keep initialization context in\n+\t * feed_pipe_ctx (shared by all processes).\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\tvoid *feed_pipe_ctx;\n \n \t/* Number of threads to parallelize across */\n \tint jobs;\n \n \t/* Path to initial working directory for subprocess */\n \tconst char *dir;\n+\n };\n \n /*\n@@ -81,6 +100,9 @@ struct hook_cb_data {\n \t.path_to_stdin = NULL,\t\t\t\\\n \t.jobs = 1,\t\t\t\t\\\n \t.dir = NULL,\t\t\t\t\\\n+\t.str_stdin = STRING_LIST_INIT_DUP,\t\\\n+\t.feed_pipe = NULL,\t\t\t\\\n+\t.feed_pipe_ctx = NULL,\t\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n@@ -90,6 +112,9 @@ struct hook_cb_data {\n \t.path_to_stdin = NULL,\t\t\t\\\n \t.jobs = configured_hook_jobs(),\t\t\\\n \t.dir = NULL,\t\t\t\t\\\n+\t.str_stdin = STRING_LIST_INIT_DUP,\t\\\n+\t.feed_pipe = NULL,\t\t\t\\\n+\t.feed_pipe_ctx = NULL,\t\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411468","messageId":"20201205014607.1464119-17-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH 16/17] run-command: allow capturing of collated output","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-05T01:46:06Z","receivedAt":"2020-12-05T01:48:25Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Some callers, for example server-side hooks which wish to relay hook\noutput to clients across a transport, want to capture what would\nnormally print to stderr and do something else with it. Allow that via a\ncallback.\n\nBy calling the callback regardless of whether there's output available,\nwe allow clients to send e.g. a keepalive if necessary.\n\nBecause we expose a strbuf, not a fd or FILE*, there's no need to create\na temporary pipe or similar - we can just skip the print to stderr and\ninstead hand it to the caller.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Originally when writing this patch I attempted to use a pipe in memory -\n but managing its lifetime was actually pretty tricky, and I found I could\n achieve the same thing with less code by doing it this way. Critique welcome,\n including \"no, you really need to do it with a pipe\".\n\n builtin/fetch.c | 2 +-\n builtin/submodule--helper.c | 2 +-\n hook.c | 1 +\n run-command.c | 33 +++++++++++++++++++++++++--------\n run-command.h | 18 +++++++++++++++++-\n submodule.c | 2 +-\n t/helper/test-run-command.c | 25 ++++++++++++++++++++-----\n t/t0061-run-command.sh | 7 +++++++\n 8 files changed, 73 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 5e153b5193..6a634085d9 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1647,7 +1647,7 @@ static int fetch_multiple(struct string_list *list, int max_children)\n \t\tresult = run_processes_parallel_tr2(max_children,\n \t\t\t\t\t\t &fetch_next_remote,\n \t\t\t\t\t\t &fetch_failed_to_start,\n-\t\t\t\t\t\t NULL,\n+\t\t\t\t\t\t NULL, NULL,\n \t\t\t\t\t\t &fetch_finished,\n \t\t\t\t\t\t &state,\n \t\t\t\t\t\t \"fetch\", \"parallel/fetch\");\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex bb623c1852..8c543d33fd 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -2294,7 +2294,7 @@ static int update_submodules(struct submodule_update_clone *suc)\n \tint i;\n \n \trun_processes_parallel_tr2(suc->max_jobs, update_clone_get_next_task,\n-\t\t\t\t update_clone_start_failure, NULL,\n+\t\t\t\t update_clone_start_failure, NULL, NULL,\n \t\t\t\t update_clone_task_finished, suc, \"submodule\",\n \t\t\t\t \"parallel/update\");\n \ndiff --git a/hook.c b/hook.c\nindex fbb49f241d..1186ee41b3 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -407,6 +407,7 @@ int run_hooks(const char *hookname, struct run_hooks_opt *options)\n \t\t\t\t pick_next_hook,\n \t\t\t\t notify_start_failure,\n \t\t\t\t options->feed_pipe,\n+\t\t\t\t NULL,\n \t\t\t\t notify_hook_finished,\n \t\t\t\t &cb_data,\n \t\t\t\t \"hook\",\ndiff --git a/run-command.c b/run-command.c\nindex 7b65c087f8..0dce6bec83 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1549,6 +1549,7 @@ struct parallel_processes {\n \tget_next_task_fn get_next_task;\n \tstart_failure_fn start_failure;\n \tfeed_pipe_fn feed_pipe;\n+\tconsume_sideband_fn consume_sideband;\n \ttask_finished_fn task_finished;\n \n \tstruct {\n@@ -1614,6 +1615,7 @@ static void pp_init(struct parallel_processes *pp,\n \t\t get_next_task_fn get_next_task,\n \t\t start_failure_fn start_failure,\n \t\t feed_pipe_fn feed_pipe,\n+\t\t consume_sideband_fn consume_sideband,\n \t\t task_finished_fn task_finished,\n \t\t void *data)\n {\n@@ -1634,6 +1636,7 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->start_failure = start_failure ? start_failure : default_start_failure;\n \tpp->feed_pipe = feed_pipe ? feed_pipe : default_feed_pipe;\n \tpp->task_finished = task_finished ? task_finished : default_task_finished;\n+\tpp->consume_sideband = consume_sideband;\n \n \tpp->nr_processes = 0;\n \tpp->output_owner = 0;\n@@ -1670,7 +1673,10 @@ static void pp_cleanup(struct parallel_processes *pp)\n \t * When get_next_task added messages to the buffer in its last\n \t * iteration, the buffered output is non empty.\n \t */\n-\tstrbuf_write(&pp->buffered_output, stderr);\n+\tif (pp->consume_sideband)\n+\t\tpp->consume_sideband(&pp->buffered_output, pp->data);\n+\telse\n+\t\tstrbuf_write(&pp->buffered_output, stderr);\n \tstrbuf_release(&pp->buffered_output);\n \n \tsigchain_pop_common();\n@@ -1786,9 +1792,13 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n static void pp_output(struct parallel_processes *pp)\n {\n \tint i = pp->output_owner;\n+\n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n-\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\tif (pp->consume_sideband)\n+\t\t\tpp->consume_sideband(&pp->children[i].err, pp->data);\n+\t\telse\n+\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\tstrbuf_reset(&pp->children[i].err);\n \t}\n }\n@@ -1827,11 +1837,15 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t} else {\n-\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t/* Output errors, then all other finished child processes */\n+\t\t\tif (pp->consume_sideband) {\n+\t\t\t\tpp->consume_sideband(&pp->children[i].err, pp->data);\n+\t\t\t\tpp->consume_sideband(&pp->buffered_output, pp->data);\n+\t\t\t} else {\n+\t\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n+\t\t\t}\n \t\t\tstrbuf_reset(&pp->children[i].err);\n-\n-\t\t\t/* Output all other finished child processes */\n-\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n \t\t\tstrbuf_reset(&pp->buffered_output);\n \n \t\t\t/*\n@@ -1855,6 +1869,7 @@ int run_processes_parallel(int n,\n \t\t\t get_next_task_fn get_next_task,\n \t\t\t start_failure_fn start_failure,\n \t\t\t feed_pipe_fn feed_pipe,\n+\t\t\t consume_sideband_fn consume_sideband,\n \t\t\t task_finished_fn task_finished,\n \t\t\t void *pp_cb)\n {\n@@ -1865,7 +1880,7 @@ int run_processes_parallel(int n,\n \n \tsigchain_push(SIGPIPE, SIG_IGN);\n \n-\tpp_init(&pp, n, get_next_task, start_failure, feed_pipe, task_finished, pp_cb);\n+\tpp_init(&pp, n, get_next_task, start_failure, feed_pipe, consume_sideband, task_finished, pp_cb);\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1903,6 +1918,7 @@ int run_processes_parallel(int n,\n int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \t\t\t start_failure_fn start_failure,\n \t\t\t feed_pipe_fn feed_pipe,\n+\t\t\t consume_sideband_fn consume_sideband,\n \t\t\t task_finished_fn task_finished, void *pp_cb,\n \t\t\t const char *tr2_category, const char *tr2_label)\n {\n@@ -1912,7 +1928,8 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \t\t\t\t ((n < 1) ? online_cpus() : n));\n \n \tresult = run_processes_parallel(n, get_next_task, start_failure,\n-\t\t\t\t\tfeed_pipe, task_finished, pp_cb);\n+\t\t\t\t\tfeed_pipe, consume_sideband,\n+\t\t\t\t\ttask_finished, pp_cb);\n \n \ttrace2_region_leave(tr2_category, tr2_label, NULL);\n \ndiff --git a/run-command.h b/run-command.h\nindex e058c0e2c8..2ad8271f56 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -450,6 +450,20 @@ typedef int (*feed_pipe_fn)(struct strbuf *pipe,\n \t\t\t void *pp_cb,\n \t\t\t void *pp_task_cb);\n \n+/**\n+ * If this callback is provided, instead of collating process output to stderr,\n+ * they will be collated into a new pipe. consume_sideband_fn will be called\n+ * repeatedly. When output is available on that pipe, it will be contained in\n+ * 'output'. But it will be called with an empty 'output' too, to allow for\n+ * keepalives or similar operations if necessary.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel.\n+ *\n+ * Since this callback is provided with the collated output, no task cookie is\n+ * provided.\n+ */\n+typedef void (*consume_sideband_fn)(struct strbuf *output, void *pp_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -485,10 +499,12 @@ int run_processes_parallel(int n,\n \t\t\t get_next_task_fn,\n \t\t\t start_failure_fn,\n \t\t\t feed_pipe_fn,\n+\t\t\t consume_sideband_fn,\n \t\t\t task_finished_fn,\n \t\t\t void *pp_cb);\n int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n-\t\t\t feed_pipe_fn, task_finished_fn, void *pp_cb,\n+\t\t\t feed_pipe_fn, consume_sideband_fn,\n+\t\t\t task_finished_fn, void *pp_cb,\n \t\t\t const char *tr2_category, const char *tr2_label);\n \n #endif\ndiff --git a/submodule.c b/submodule.c\nindex 953f41818c..215bff22d9 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1638,7 +1638,7 @@ int fetch_populated_submodules(struct repository *r,\n \trun_processes_parallel_tr2(max_parallel_jobs,\n \t\t\t\t get_next_submodule,\n \t\t\t\t fetch_start_failure,\n-\t\t\t\t NULL,\n+\t\t\t\t NULL, NULL,\n \t\t\t\t fetch_finish,\n \t\t\t\t &spf,\n \t\t\t\t \"submodule\", \"parallel/fetch\");\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 9348184d30..d53db6d11c 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -51,6 +51,16 @@ static int no_job(struct child_process *cp,\n \treturn 0;\n }\n \n+static void test_consume_sideband(struct strbuf *output, void *cb)\n+{\n+\tFILE *sideband;\n+\n+\tsideband = fopen(\"./sideband\", \"a\");\n+\n+\tstrbuf_write(output, sideband);\n+\tfclose(sideband);\n+}\n+\n static int task_finished(int result,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb,\n@@ -201,7 +211,7 @@ static int testsuite(int argc, const char **argv)\n \t\tsuite.tests.nr, max_jobs);\n \n \tret = run_processes_parallel(max_jobs, next_test, test_failed,\n-\t\t\t\t test_stdin, test_finished, &suite);\n+\t\t\t\t test_stdin, NULL, test_finished, &suite);\n \n \tif (suite.failed.nr > 0) {\n \t\tret = 1;\n@@ -429,23 +439,28 @@ int cmd__run_command(int argc, const char **argv)\n \n \tif (!strcmp(argv[1], \"run-command-parallel\"))\n \t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, NULL, NULL, &proc));\n+\t\t\t\t\t NULL, NULL, NULL, NULL, &proc));\n \n \tif (!strcmp(argv[1], \"run-command-abort\"))\n \t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, NULL, task_finished, &proc));\n+\t\t\t\t\t NULL, NULL, NULL, task_finished, &proc));\n \n \tif (!strcmp(argv[1], \"run-command-no-jobs\"))\n \t\texit(run_processes_parallel(jobs, no_job,\n-\t\t\t\t\t NULL, NULL, task_finished, &proc));\n+\t\t\t\t\t NULL, NULL, NULL, task_finished, &proc));\n \n \tif (!strcmp(argv[1], \"run-command-stdin\")) {\n \t\tproc.in = -1;\n \t\tproc.no_stdin = 0;\n \t\texit (run_processes_parallel(jobs, parallel_next, NULL,\n-\t\t\t\t\t test_stdin, NULL, &proc));\n+\t\t\t\t\t test_stdin, NULL, NULL, &proc));\n \t}\n \n+\tif (!strcmp(argv[1], \"run-command-sideband\"))\n+\t\texit(run_processes_parallel(jobs, parallel_next, NULL, NULL,\n+\t\t\t\t\t test_consume_sideband, NULL,\n+\t\t\t\t\t &proc));\n+\n \tfprintf(stderr, \"check usage\\n\");\n \treturn 1;\n }\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 3eb572e6cd..c5a5b6df6c 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -143,6 +143,13 @@ test_expect_success 'run_command runs in parallel with more tasks than jobs avai\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command can divert output' '\n+\ttest_when_finished rm sideband &&\n+\ttest-tool run-command run-command-sideband 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n+\ttest_must_be_empty actual &&\n+\ttest_cmp expect sideband\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n listening for stdin:\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"411747","messageId":"20201208210925.GI8396@szeder.dev","threadId":"54769","inReplyTo":"20201205014607.1464119-16-emilyshaffer@google.com","subject":"Re: [PATCH 15/17] hook: provide stdin by string_list or callback","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-12-08T21:09:25Z","receivedAt":"2020-12-08T21:10:11Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Dec 04, 2020 at 05:46:05PM -0800, Emily Shaffer wrote:\n> diff --git a/hook.c b/hook.c\n> index f0c052d847..fbb49f241d 100644\n> --- a/hook.c\n> +++ b/hook.c\n> @@ -9,6 +9,8 @@ void free_hook(struct hook *ptr)\n> {\n> \tif (ptr) {\n> \t\tstrbuf_release(&ptr->command);\n> +\t\tif (ptr->feed_pipe_cb_data)\n\nCoccinelle suggests to drop this condition, because free() can handle\na NULL pointer just fine.\n\n> +\t\t\tfree(ptr->feed_pipe_cb_data);\n> \t\tfree(ptr);\n> \t}\n> }\n"},{"id":"411780","messageId":"20201208221155.GC3783238@google.com","threadId":"54769","inReplyTo":"20201208210925.GI8396@szeder.dev","subject":"Re: [PATCH 15/17] hook: provide stdin by string_list or callback","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-08T22:11:55Z","receivedAt":"2020-12-08T22:12:47Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Dec 08, 2020 at 10:09:25PM +0100, SZEDER Gábor wrote:\n> \n> On Fri, Dec 04, 2020 at 05:46:05PM -0800, Emily Shaffer wrote:\n> > diff --git a/hook.c b/hook.c\n> > index f0c052d847..fbb49f241d 100644\n> > --- a/hook.c\n> > +++ b/hook.c\n> > @@ -9,6 +9,8 @@ void free_hook(struct hook *ptr)\n> > {\n> > \tif (ptr) {\n> > \t\tstrbuf_release(&ptr->command);\n> > +\t\tif (ptr->feed_pipe_cb_data)\n> \n> Coccinelle suggests to drop this condition, because free() can handle\n> a NULL pointer just fine.\n\nDone (locally). Thanks (and thanks for checking the coccinelle output\ntoo).\n\n> \n> > +\t\t\tfree(ptr->feed_pipe_cb_data);\n> > \t\tfree(ptr);\n> > \t}\n> > }\n"},{"id":"412066","messageId":"23cb3575-7706-45a6-7a50-0fc9ef850b9f@gmail.com","threadId":"54769","inReplyTo":"20201205014607.1464119-9-emilyshaffer@google.com","subject":"Re: [PATCH 08/17] hook: add 'run' subcommand","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2020-12-11T10:15:26Z","receivedAt":"2020-12-11T10:17:19Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Emily\n\nOn 05/12/2020 01:45, Emily Shaffer wrote:\n> In order to enable hooks to be run as an external process, by a\n> standalone Git command, or by tools which wrap Git, provide an external\n> means to run all configured hook commands for a given hook event.\n> \n> For now, the hook commands will run in config order, in series. As\n> alternate ordering or parallelism is supported in the future, we should\n> add knobs to use those to the command line as well.\n> \n> As with the legacy hook implementation, all stdout generated by hook\n> commands is redirected to stderr. Piping from stdin is not yet\n> supported.\n> \n> Legacy hooks (those present in $GITDIR/hooks) are run at the end of the\n> execution list. For now, there is no way to disable them.\n> \n> Users may wish to provide hook commands like 'git config\n> hook.pre-commit.command \"~/linter.sh --pre-commit\"'. To enable this, the\n> contents of the 'hook.*.command' and 'hookcmd.*.command' strings are\n> first split by space or quotes into an argv_array, then expanded with\n> 'expand_user_path()'.\n\nI'm a bit confused by this last paragraph, the docs below say we pass \nthe string to the shell and that's what the implementation seems to do. \nIf we're running a lot of hooks then maybe it would be worth using \nsplit_cmdline() and expand_user_path() rather than invoking the shell \nfor each hook we run.\n\nI'm afraid I've only had time to skip the patch, there are a couple of \nminor comments below.\n\n> Signed-off-by: Emily Shaffer \n> ---\n> \n> Notes:\n> Since v4, updated the docs, and did less local application of single\n> quotes. In order for hookdir hooks to run successfully with a space in\n> the path, though, they must not be run with 'sh -c'. So we can treat the\n> hookdir hooks specially, and warn users via doc about special\n> considerations for configured hooks with spaces in their path.\n> \n> Documentation/git-hook.txt | 31 +++++++++-\n> builtin/hook.c | 48 ++++++++++++++-\n> hook.c | 112 ++++++++++++++++++++++++++++++++++\n> hook.h | 32 ++++++++++\n> t/t1360-config-based-hooks.sh | 65 +++++++++++++++++++-\n> 5 files changed, 281 insertions(+), 7 deletions(-)\n> \n> diff --git a/Documentation/git-hook.txt b/Documentation/git-hook.txt\n> index f19875ed68..18a817d832 100644\n> --- a/Documentation/git-hook.txt\n> +++ b/Documentation/git-hook.txt\n> @@ -9,11 +9,12 @@ SYNOPSIS\n> --------\n> [verse]\n> 'git hook' list \n> +'git hook' run [(-e|--env)=...] [(-a|--arg)=...] \n> \n> DESCRIPTION\n> -----------\n> -You can list configured hooks with this command. Later, you will be able to run,\n> -add, and modify hooks with this command.\n> +You can list and run configured hooks with this command. Later, you will be able\n> +to add and modify hooks with this command.\n> \n> This command parses the default configuration files for sections `hook` and\n> `hookcmd`. `hook` is used to describe the commands which will be run during a\n> @@ -64,6 +65,32 @@ in the order they should be run, and print the config scope where the relevant\n> `hook..command` was specified, not the `hookcmd` (if applicable).\n> This output is human-readable and the format is subject to change over time.\n> \n> +run [(-e|--env)=...] [(-a|--arg)=...] ``::\n> +\n> +Runs hooks configured for ``, in the same order displayed by `git\n> +hook list`. Hooks configured this way are run prepended with `sh -c`, so paths\n> +containing special characters or spaces should be wrapped in single quotes:\n> +`command = '/my/path with spaces/script.sh' some args`.\n> +\n> +OPTIONS\n> +-------\n> +--run-hookdir::\n> +\tOverrides the hook.runHookDir config. Must be 'yes', 'warn',\n> +\t'interactive', or 'no'. Specifies how to handle hooks located in the Git\n> +\thook directory (core.hooksPath).\n> +\n> +-a::\n> +--arg::\n> +\tOnly valid for `run`.\n> ++\n> +Specify arguments to pass to every hook that is run.\n> +\n> +-e::\n> +--env::\n> +\tOnly valid for `run`.\n> ++\n> +Specify environment variables to set for every hook that is run.\n> +\n> CONFIGURATION\n> -------------\n> include::config/hook.txt[]\n> diff --git a/builtin/hook.c b/builtin/hook.c\n> index 16324d4195..26f7050387 100644\n> --- a/builtin/hook.c\n> +++ b/builtin/hook.c\n> @@ -5,9 +5,11 @@\n> #include \"hook.h\"\n> #include \"parse-options.h\"\n> #include \"strbuf.h\"\n> +#include \"strvec.h\"\n> \n> static const char * const builtin_hook_usage[] = {\n> \tN_(\"git hook list \"),\n> +\tN_(\"git hook run [(-e|--env)=...] [(-a|--arg)=...] \"),\n> \tNULL\n> };\n> \n> @@ -84,6 +86,46 @@ static int list(int argc, const char **argv, const char *prefix)\n> \treturn 0;\n> }\n> \n> +static int run(int argc, const char **argv, const char *prefix)\n> +{\n> +\tstruct strbuf hookname = STRBUF_INIT;\n> +\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n> +\tint rc = 0;\n> +\n> +\tstruct option run_options[] = {\n> +\t\tOPT_STRVEC('e', \"env\", &opt.env, N_(\"var\"),\n> +\t\t\t N_(\"environment variables for hook to use\")),\n> +\t\tOPT_STRVEC('a', \"arg\", &opt.args, N_(\"args\"),\n> +\t\t\t N_(\"argument to pass to hook\")),\n> +\t\tOPT_END(),\n> +\t};\n> +\n> +\t/*\n> +\t * While it makes sense to list hooks out-of-repo, it doesn't make sense\n> +\t * to execute them. Hooks usually want to look at repository artifacts.\n> +\t */\n> +\tif (!have_git_dir())\n> +\t\tusage_msg_opt(_(\"You must be in a Git repo to execute hooks.\"),\n> +\t\t\t builtin_hook_usage, run_options);\n> +\n> +\targc = parse_options(argc, argv, prefix, run_options,\n> +\t\t\t builtin_hook_usage, 0);\n> +\n> +\tif (argc < 1)\n> +\t\tusage_msg_opt(_(\"You must specify a hook event to run.\"),\n> +\t\t\t builtin_hook_usage, run_options);\n> +\n> +\tstrbuf_addstr(&hookname, argv[0]);\n> +\topt.run_hookdir = should_run_hookdir;\n> +\n> +\trc = run_hooks(hookname.buf, &opt);\n> +\n> +\tstrbuf_release(&hookname);\n> +\trun_hooks_opt_clear(&opt);\n> +\n> +\treturn rc;\n> +}\n> +\n> int cmd_hook(int argc, const char **argv, const char *prefix)\n> {\n> \tconst char *run_hookdir = NULL;\n> @@ -95,10 +137,10 @@ int cmd_hook(int argc, const char **argv, const char *prefix)\n> \t};\n> \n> \targc = parse_options(argc, argv, prefix, builtin_hook_options,\n> -\t\t\t builtin_hook_usage, 0);\n> +\t\t\t builtin_hook_usage, PARSE_OPT_KEEP_UNKNOWN);\n> \n> \t/* after the parse, we should have \" \" */\n> -\tif (argc < 1)\n> +\tif (argc < 2)\n> \t\tusage_with_options(builtin_hook_usage, builtin_hook_options);\n> \n> \n> @@ -120,6 +162,8 @@ int cmd_hook(int argc, const char **argv, const char *prefix)\n> \n> \tif (!strcmp(argv[0], \"list\"))\n> \t\treturn list(argc, argv, prefix);\n> +\tif (!strcmp(argv[0], \"run\"))\n> +\t\treturn run(argc, argv, prefix);\n> \n> \tusage_with_options(builtin_hook_usage, builtin_hook_options);\n> }\n> diff --git a/hook.c b/hook.c\n> index f4084e33c8..c4595a2324 100644\n> --- a/hook.c\n> +++ b/hook.c\n> @@ -3,6 +3,7 @@\n> #include \"hook.h\"\n> #include \"config.h\"\n> #include \"run-command.h\"\n> +#include \"prompt.h\"\n> \n> void free_hook(struct hook *ptr)\n> {\n> @@ -135,6 +136,56 @@ enum hookdir_opt configured_hookdir_opt(void)\n> \treturn hookdir_unknown;\n> }\n> \n> +static int should_include_hookdir(const char *path, enum hookdir_opt cfg)\n> +{\n> +\tstruct strbuf prompt = STRBUF_INIT;\n> +\t/*\n> +\t * If the path doesn't exist, don't bother adding the empty hook and\n> +\t * don't bother checking the config or prompting the user.\n> +\t */\n> +\tif (!path)\n> +\t\treturn 0;\n> +\n> +\tswitch (cfg)\n> +\t{\n> +\t\tcase hookdir_no:\n\nStyle nit: we normally use uppercase for constants and enums.\n\n> +\t\t\treturn 0;\n> +\t\tcase hookdir_unknown:\n> +\t\t\tfprintf(stderr,\n> +\t\t\t\t_(\"Unrecognized value for 'hook.runHookDir'. \"\n> +\t\t\t\t \"Is there a typo? \"));\n\nWhat happens at the moment if core.hooksPath does not exist?\n\nBest Wishes\n\nPhillip\n\n> +\t\t\t/* FALLTHROUGH */\n> +\t\tcase hookdir_warn:\n> +\t\t\tfprintf(stderr, _(\"Running legacy hook at '%s'\\n\"),\n> +\t\t\t\tpath);\n> +\t\t\treturn 1;\n> +\t\tcase hookdir_interactive:\n> +\t\t\tdo {\n> +\t\t\t\t/*\n> +\t\t\t\t * TRANSLATORS: Make sure to include [Y] and [n]\n> +\t\t\t\t * in your translation. Only English input is\n> +\t\t\t\t * accepted. Default option is \"yes\".\n> +\t\t\t\t */\n> +\t\t\t\tfprintf(stderr, _(\"Run '%s'? [Yn] \"), path);\n> +\t\t\t\tgit_read_line_interactively(&prompt);\n> +\t\t\t\tstrbuf_tolower(&prompt);\n> +\t\t\t\tif (starts_with(prompt.buf, \"n\")) {\n> +\t\t\t\t\tstrbuf_release(&prompt);\n> +\t\t\t\t\treturn 0;\n> +\t\t\t\t} else if (starts_with(prompt.buf, \"y\")) {\n> +\t\t\t\t\tstrbuf_release(&prompt);\n> +\t\t\t\t\treturn 1;\n> +\t\t\t\t}\n> +\t\t\t\t/* otherwise, we didn't understand the input */\n> +\t\t\t} while (prompt.len); /* an empty reply means \"Yes\" */\n> +\t\t\tstrbuf_release(&prompt);\n> +\t\t\treturn 1;\n> +\t\tcase hookdir_yes:\n> +\t\tdefault:\n> +\t\t\treturn 1;\n> +\t}\n> +}\n> +\n> struct list_head* hook_list(const struct strbuf* hookname)\n> {\n> \tstruct strbuf hook_key = STRBUF_INIT;\n> @@ -166,3 +217,64 @@ struct list_head* hook_list(const struct strbuf* hookname)\n> \tstrbuf_release(&hook_key);\n> \treturn hook_head;\n> }\n> +\n> +void run_hooks_opt_init(struct run_hooks_opt *o)\n> +{\n> +\tstrvec_init(&o->env);\n> +\tstrvec_init(&o->args);\n> +\to->run_hookdir = configured_hookdir_opt();\n> +}\n> +\n> +void run_hooks_opt_clear(struct run_hooks_opt *o)\n> +{\n> +\tstrvec_clear(&o->env);\n> +\tstrvec_clear(&o->args);\n> +}\n> +\n> +int run_hooks(const char *hookname, struct run_hooks_opt *options)\n> +{\n> +\tstruct strbuf hookname_str = STRBUF_INIT;\n> +\tstruct list_head *to_run, *pos = NULL, *tmp = NULL;\n> +\tint rc = 0;\n> +\n> +\tif (!options)\n> +\t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n> +\n> +\tstrbuf_addstr(&hookname_str, hookname);\n> +\n> +\tto_run = hook_list(&hookname_str);\n> +\n> +\tlist_for_each_safe(pos, tmp, to_run) {\n> +\t\tstruct child_process hook_proc = CHILD_PROCESS_INIT;\n> +\t\tstruct hook *hook = list_entry(pos, struct hook, list);\n> +\n> +\t\thook_proc.env = options->env.v;\n> +\t\thook_proc.no_stdin = 1;\n> +\t\thook_proc.stdout_to_stderr = 1;\n> +\t\thook_proc.trace2_hook_name = hook->command.buf;\n> +\t\thook_proc.use_shell = 1;\n> +\n> +\t\tif (hook->from_hookdir) {\n> +\t\t if (!should_include_hookdir(hook->command.buf, options->run_hookdir))\n> +\t\t\tcontinue;\n> +\t\t /*\n> +\t\t * Commands from the config could be oneliners, but we know\n> +\t\t * for certain that hookdir commands are not.\n> +\t\t */\n> +\t\t hook_proc.use_shell = 0;\n> +\t\t}\n> +\n> +\t\t/* add command */\n> +\t\tstrvec_push(&hook_proc.args, hook->command.buf);\n> +\n> +\t\t/*\n> +\t\t * add passed-in argv, without expanding - let the user get back\n> +\t\t * exactly what they put in\n> +\t\t */\n> +\t\tstrvec_pushv(&hook_proc.args, options->args.v);\n> +\n> +\t\trc |= run_command(&hook_proc);\n> +\t}\n> +\n> +\treturn rc;\n> +}\n> diff --git a/hook.h b/hook.h\n> index ca45d388d3..d1c3d71e82 100644\n> --- a/hook.h\n> +++ b/hook.h\n> @@ -1,6 +1,7 @@\n> #include \"config.h\"\n> #include \"list.h\"\n> #include \"strbuf.h\"\n> +#include \"strvec.h\"\n> \n> struct hook\n> {\n> @@ -36,6 +37,37 @@ enum hookdir_opt\n> */\n> enum hookdir_opt configured_hookdir_opt(void);\n> \n> +struct run_hooks_opt\n> +{\n> +\t/* Environment vars to be set for each hook */\n> +\tstruct strvec env;\n> +\n> +\t/* Args to be passed to each hook */\n> +\tstruct strvec args;\n> +\n> +\t/*\n> +\t * How should the hookdir be handled?\n> +\t * Leave the RUN_HOOKS_OPT_INIT default in most cases; this only needs\n> +\t * to be overridden if the user can override it at the command line.\n> +\t */\n> +\tenum hookdir_opt run_hookdir;\n> +};\n> +\n> +#define RUN_HOOKS_OPT_INIT { \t\t\\\n> +\t.env = STRVEC_INIT, \t\t\t\t\\\n> +\t.args = STRVEC_INIT, \t\t\t\\\n> +\t.run_hookdir = configured_hookdir_opt()\t\\\n> +}\n> +\n> +void run_hooks_opt_init(struct run_hooks_opt *o);\n> +void run_hooks_opt_clear(struct run_hooks_opt *o);\n> +\n> +/*\n> + * Runs all hooks associated to the 'hookname' event in order. Each hook will be\n> + * passed 'env' and 'args'.\n> + */\n> +int run_hooks(const char *hookname, struct run_hooks_opt *options);\n> +\n> /* Free memory associated with a 'struct hook' */\n> void free_hook(struct hook *ptr);\n> /* Empties the list at 'head', calling 'free_hook()' on each entry */\n> diff --git a/t/t1360-config-based-hooks.sh b/t/t1360-config-based-hooks.sh\n> index ebd3bc623f..5b3003d59b 100755\n> --- a/t/t1360-config-based-hooks.sh\n> +++ b/t/t1360-config-based-hooks.sh\n> @@ -115,7 +115,10 @@ test_expect_success 'hook.runHookDir = no is respected by list' '\n> \n> \tgit hook list pre-commit >actual &&\n> \t# the hookdir annotation is translated\n> -\ttest_i18ncmp expected actual\n> +\ttest_i18ncmp expected actual &&\n> +\n> +\tgit hook run pre-commit 2>actual &&\n> +\ttest_must_be_empty actual\n> '\n> \n> test_expect_success 'hook.runHookDir = warn is respected by list' '\n> @@ -129,6 +132,14 @@ test_expect_success 'hook.runHookDir = warn is respected by list' '\n> \n> \tgit hook list pre-commit >actual &&\n> \t# the hookdir annotation is translated\n> +\ttest_i18ncmp expected actual &&\n> +\n> +\tcat >expected <<-EOF &&\n> +\tRunning legacy hook at '\\''$(pwd)/.git/hooks/pre-commit'\\''\n> +\t\"Legacy Hook\"\n> +\tEOF\n> +\n> +\tgit hook run pre-commit 2>actual &&\n> \ttest_i18ncmp expected actual\n> '\n> \n> @@ -156,7 +167,7 @@ test_expect_success 'git hook list removes skipped inlined hook' '\n> \ttest_cmp expected actual\n> '\n> \n> -test_expect_success 'hook.runHookDir = interactive is respected by list' '\n> +test_expect_success 'hook.runHookDir = interactive is respected by list and run' '\n> \tsetup_hookdir &&\n> \n> \ttest_config hook.runHookDir \"interactive\" &&\n> @@ -167,7 +178,55 @@ test_expect_success 'hook.runHookDir = interactive is respected by list' '\n> \n> \tgit hook list pre-commit >actual &&\n> \t# the hookdir annotation is translated\n> -\ttest_i18ncmp expected actual\n> +\ttest_i18ncmp expected actual &&\n> +\n> +\ttest_write_lines n | git hook run pre-commit 2>actual &&\n> +\t! grep \"Legacy Hook\" actual &&\n> +\n> +\ttest_write_lines y | git hook run pre-commit 2>actual &&\n> +\tgrep \"Legacy Hook\" actual\n> +'\n> +\n> +test_expect_success 'inline hook definitions execute oneliners' '\n> +\ttest_config hook.pre-commit.command \"echo \\\"Hello World\\\"\" &&\n> +\n> +\techo \"Hello World\" >expected &&\n> +\n> +\t# hooks are run with stdout_to_stderr = 1\n> +\tgit hook run pre-commit 2>actual &&\n> +\ttest_cmp expected actual\n> +'\n> +\n> +test_expect_success 'inline hook definitions resolve paths' '\n> +\twrite_script sample-hook.sh <<-EOF &&\n> +\techo \\\"Sample Hook\\\"\n> +\tEOF\n> +\n> +\ttest_when_finished \"rm sample-hook.sh\" &&\n> +\n> +\ttest_config hook.pre-commit.command \"\\\"$(pwd)/sample-hook.sh\\\"\" &&\n> +\n> +\techo \\\"Sample Hook\\\" >expected &&\n> +\n> +\t# hooks are run with stdout_to_stderr = 1\n> +\tgit hook run pre-commit 2>actual &&\n> +\ttest_cmp expected actual\n> +'\n> +\n> +test_expect_success 'hookdir hook included in git hook run' '\n> +\tsetup_hookdir &&\n> +\n> +\techo \\\"Legacy Hook\\\" >expected &&\n> +\n> +\t# hooks are run with stdout_to_stderr = 1\n> +\tgit hook run pre-commit 2>actual &&\n> +\ttest_cmp expected actual\n> +'\n> +\n> +test_expect_success 'out-of-repo runs excluded' '\n> +\tsetup_hooks &&\n> +\n> +\tnongit test_must_fail git hook run pre-commit\n> '\n> \n> test_done\n> \n"},{"id":"412302","messageId":"20201215214109.GD3783238@google.com","threadId":"54769","inReplyTo":"23cb3575-7706-45a6-7a50-0fc9ef850b9f@gmail.com","subject":"Re: [PATCH 08/17] hook: add 'run' subcommand","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-15T21:41:09Z","receivedAt":"2020-12-15T21:43:34Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Fri, Dec 11, 2020 at 10:15:26AM +0000, Phillip Wood wrote:\n> \n> Hi Emily\n> \n> On 05/12/2020 01:45, Emily Shaffer wrote:\n> > In order to enable hooks to be run as an external process, by a\n> > standalone Git command, or by tools which wrap Git, provide an external\n> > means to run all configured hook commands for a given hook event.\n> > \n> > For now, the hook commands will run in config order, in series. As\n> > alternate ordering or parallelism is supported in the future, we should\n> > add knobs to use those to the command line as well.\n> > \n> > As with the legacy hook implementation, all stdout generated by hook\n> > commands is redirected to stderr. Piping from stdin is not yet\n> > supported.\n> > \n> > Legacy hooks (those present in $GITDIR/hooks) are run at the end of the\n> > execution list. For now, there is no way to disable them.\n> > \n> > Users may wish to provide hook commands like 'git config\n> > hook.pre-commit.command \"~/linter.sh --pre-commit\"'. To enable this, the\n> > contents of the 'hook.*.command' and 'hookcmd.*.command' strings are\n> > first split by space or quotes into an argv_array, then expanded with\n> > 'expand_user_path()'.\n> \n> I'm a bit confused by this last paragraph, the docs below say we pass the\n> string to the shell and that's what the implementation seems to do. If we're\n> running a lot of hooks then maybe it would be worth using split_cmdline()\n> and expand_user_path() rather than invoking the shell for each hook we run.\n\nYeah, I think you are right that the commit message is stale. I had some\ntrouble getting things to work correctly with split_cmdline() and\nexpand_user_path(), so I'd prefer to run with shell.\n\n> \n> I'm afraid I've only had time to skip the patch, there are a couple of minor\n> comments below.\n\nNo problem. Thanks for having a look.\n\n> > +static int should_include_hookdir(const char *path, enum hookdir_opt cfg)\n> > +{\n> > +\tstruct strbuf prompt = STRBUF_INIT;\n> > +\t/*\n> > +\t * If the path doesn't exist, don't bother adding the empty hook and\n> > +\t * don't bother checking the config or prompting the user.\n> > +\t */\n> > +\tif (!path)\n> > +\t\treturn 0;\n> > +\n> > +\tswitch (cfg)\n> > +\t{\n> > +\t\tcase hookdir_no:\n> \n> Style nit: we normally use uppercase for constants and enums.\n\nOK. Thanks - will fix where it's introduced and update subsequent\npatches.\n\n> \n> > +\t\t\treturn 0;\n> > +\t\tcase hookdir_unknown:\n> > +\t\t\tfprintf(stderr,\n> > +\t\t\t\t_(\"Unrecognized value for 'hook.runHookDir'. \"\n> > +\t\t\t\t \"Is there a typo? \"));\n> \n> What happens at the moment if core.hooksPath does not exist?\n\nWhen core.hooksPath does not exist then $GIT_DIR/hooks/ is used instead.\nMy setup currently doesn't have $GIT_DIR/hooks/ and runs happily. That\nbit of logic (core.hooksPath or $GIT_DIR/hooks) is done in\nrun-command.h:find_hook() so I don't worry about it manually here.\n\nHowever, your comment caused me to investigate what happens when\ncore.hooksPath DOES exist - and I found a bug. Because the 'git hook'\nbuiltin doesn't call the default configuration callback, I miss\ncore.hooksPath hooks during 'git hook list' - but not during hooks\ninvoked during regular Git process runs. Very confusing :) So thanks for\nthe hint.\n\n - Emily\n"},{"id":"412318","messageId":"20201216003408.GR36751@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"Re: [PATCH v6 00/17] propose config-based hooks (part I)","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2020-12-16T00:34:08Z","receivedAt":"2020-12-16T00:35:24Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2020.12.04 17:45, Emily Shaffer wrote:\n> Hi folks, and thanks for the patience - I ran into many, many last-mile\n> challenges.\n> \n> I haven't addressed many comments on the design doc yet - I was keen to get the\n> \"functionally complete\" implementation and conversion to the list.\n> \n> Next on my plate:\n> - Update the design doc to make sense with what's in the implementation.\n> - A blog post! How to set up new hooks, why they're neat, etc.\n> - We seem to have some Googlers interested in trying it out internally, so\n> I'm hoping we'll gather and collate feedback from that soon too.\n> - And of course addressing comments on this series.\n> \n> Thanks!\n> - Emily\n\nThis approach looks good to me. I'll look forward to seeing the updated\ndesign and the feedback from the internal tests.\n"},{"id":"412321","messageId":"xmqqlfdy6025.fsf@gitster.c.googlers.com","threadId":"54769","inReplyTo":"20201216003408.GR36751@google.com","subject":"Re: [PATCH v6 00/17] propose config-based hooks (part I)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-12-16T00:56:18Z","receivedAt":"2020-12-16T00:57:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> On 2020.12.04 17:45, Emily Shaffer wrote:\n>> Hi folks, and thanks for the patience - I ran into many, many last-mile\n>> challenges.\n>> \n>> I haven't addressed many comments on the design doc yet - I was keen to get the\n>> \"functionally complete\" implementation and conversion to the list.\n>> \n>> Next on my plate:\n>> - Update the design doc to make sense with what's in the implementation.\n>> - A blog post! How to set up new hooks, why they're neat, etc.\n>> - We seem to have some Googlers interested in trying it out internally, so\n>> I'm hoping we'll gather and collate feedback from that soon too.\n>> - And of course addressing comments on this series.\n>> \n>> Thanks!\n>> - Emily\n>\n> This approach looks good to me. I'll look forward to seeing the updated\n> design and the feedback from the internal tests.\n\nThanks.\n\nBy the way, es/config-hooks does not seem to pass 5411 (at least)\neven as a standalone topic, and has been kicked out of 'seen' for\nsome time. Has anybody took a look into the issue?\n\n\n"},{"id":"412427","messageId":"20201216201625.GF3783238@google.com","threadId":"54769","inReplyTo":"xmqqlfdy6025.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v6 00/17] propose config-based hooks (part I)","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-16T20:16:25Z","receivedAt":"2020-12-16T20:17:12Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Dec 15, 2020 at 04:56:18PM -0800, Junio C Hamano wrote:\n> \n> Josh Steadmon writes:\n> \n> > On 2020.12.04 17:45, Emily Shaffer wrote:\n> >> Hi folks, and thanks for the patience - I ran into many, many last-mile\n> >> challenges.\n> >> \n> >> I haven't addressed many comments on the design doc yet - I was keen to get the\n> >> \"functionally complete\" implementation and conversion to the list.\n> >> \n> >> Next on my plate:\n> >> - Update the design doc to make sense with what's in the implementation.\n> >> - A blog post! How to set up new hooks, why they're neat, etc.\n> >> - We seem to have some Googlers interested in trying it out internally, so\n> >> I'm hoping we'll gather and collate feedback from that soon too.\n> >> - And of course addressing comments on this series.\n> >> \n> >> Thanks!\n> >> - Emily\n> >\n> > This approach looks good to me. I'll look forward to seeing the updated\n> > design and the feedback from the internal tests.\n> \n> Thanks.\n> \n> By the way, es/config-hooks does not seem to pass 5411 (at least)\n> even as a standalone topic, and has been kicked out of 'seen' for\n> some time. Has anybody took a look into the issue?\n\nYeah, I looked at it today. Looks like an issue with not paying\nattention to master->main default config, since I added a new test to\nthe 5411 suite (which means it wouldn't have made a conflict for someone\nto say \"ah yes, s/master/main/g\"). I am tracking down couple of other CI\nerrors today and will send a reroll today or tomorrow.\n"},{"id":"412452","messageId":"xmqqwnxh1g4h.fsf@gitster.c.googlers.com","threadId":"54769","inReplyTo":"20201216201625.GF3783238@google.com","subject":"Re: [PATCH v6 00/17] propose config-based hooks (part I)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-12-16T23:32:46Z","receivedAt":"2020-12-16T23:33:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Emily Shaffer writes:\n\n>> By the way, es/config-hooks does not seem to pass 5411 (at least)\n>> even as a standalone topic, and has been kicked out of 'seen' for\n>> some time. Has anybody took a look into the issue?\n>\n> Yeah, I looked at it today. Looks like an issue with not paying\n> attention to master->main default config, since I added a new test to\n> the 5411 suite (which means it wouldn't have made a conflict for someone\n> to say \"ah yes, s/master/main/g\"). I am tracking down couple of other CI\n> errors today and will send a reroll today or tomorrow.\n\nThanks.\n"},{"id":"412511","messageId":"20201218020748.GI3783238@google.com","threadId":"54769","inReplyTo":"xmqqwnxh1g4h.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v6 00/17] propose config-based hooks (part I)","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-18T02:07:48Z","receivedAt":"2020-12-18T02:08:35Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Wed, Dec 16, 2020 at 03:32:46PM -0800, Junio C Hamano wrote:\n> \n> Emily Shaffer writes:\n> \n> >> By the way, es/config-hooks does not seem to pass 5411 (at least)\n> >> even as a standalone topic, and has been kicked out of 'seen' for\n> >> some time. Has anybody took a look into the issue?\n> >\n> > Yeah, I looked at it today. Looks like an issue with not paying\n> > attention to master->main default config, since I added a new test to\n> > the 5411 suite (which means it wouldn't have made a conflict for someone\n> > to say \"ah yes, s/master/main/g\"). I am tracking down couple of other CI\n> > errors today and will send a reroll today or tomorrow.\n> \n> Thanks.\n\nI don't have a reroll today. I have been trying to get to the bottom of\na test which fails when built with clang but passes when built with gcc\n(t6030-bisect-porcelain.sh after patch 12 of the part II series) and\nhave not made progress on that, let alone on the other tasks I wanted to\ndo before sending the next version.\n\nNext week I will only work one day, so I'd anticipate a reroll sometime\nthe week following. Sorry for the wait - but I think even if I sent it\nwith the fix for this t5411 failure, it would still break 'seen' because\nof whatever this clang vs. gcc problem is.\n\nHope you enjoy your holidays.\n\n - Emily\n"},{"id":"412518","messageId":"xmqq7dpfy95q.fsf@gitster.c.googlers.com","threadId":"54769","inReplyTo":"20201218020748.GI3783238@google.com","subject":"Re: [PATCH v6 00/17] propose config-based hooks (part I)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-12-18T05:29:05Z","receivedAt":"2020-12-18T05:29:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Emily Shaffer writes:\n\n> I don't have a reroll today. I have been trying to get to the bottom of\n> a test which fails when built with clang but passes when built with gcc\n> (t6030-bisect-porcelain.sh after patch 12 of the part II series) and\n> have not made progress on that, let alone on the other tasks I wanted to\n> do before sending the next version.\n\nThanks for an interim report. No need to rush.\n\n> Hope you enjoy your holidays.\n\nYou too, and have fun.\n"},{"id":"412792","messageId":"20201222000220.1491091-1-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201205014607.1464119-1-emilyshaffer@google.com","subject":"[PATCH v7 00/17] propose config-based hooks (part I)","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:03Z","receivedAt":"2020-12-22T00:03:06Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Since v6:\n\n - Converted 'enum hookdir_opt' to UPPER_SNAKE\n - Coccinelle fix in the hook destructor\n - Fixed a bug where builtin/hook.c wasn't running the default git config setup\n and therefore missed hooks in core.hooksPath when it was set. (These hooks\n would still run except when invoked by 'git hook run' as the config was\n called by the processes which invoked the hook library.)\n\nCI run: https://github.com/nasamuffin/git/actions/runs/436864964\n\nThanks!\n - Emily\n\nEmily Shaffer (17):\n doc: propose hooks managed by the config\n hook: scaffolding for git-hook subcommand\n hook: add list command\n hook: include hookdir hook in list\n hook: respect hook.runHookDir\n hook: implement hookcmd..skip\n parse-options: parse into strvec\n hook: add 'run' subcommand\n hook: replace find_hook() with hook_exists()\n hook: support passing stdin to hooks\n run-command: allow stdin for run_processes_parallel\n hook: allow parallel hook execution\n hook: allow specifying working directory for hooks\n run-command: add stdin callback for parallelization\n hook: provide stdin by string_list or callback\n run-command: allow capturing of collated output\n hooks: allow callers to capture output\n\n .gitignore | 1 +\n Documentation/Makefile | 1 +\n Documentation/config/hook.txt | 19 +\n Documentation/git-hook.txt | 117 +++++\n Documentation/technical/api-parse-options.txt | 5 +\n .../technical/config-based-hooks.txt | 355 +++++++++++++++\n Makefile | 2 +\n builtin.h | 1 +\n builtin/bugreport.c | 4 +-\n builtin/fetch.c | 1 +\n builtin/hook.c | 176 ++++++++\n builtin/submodule--helper.c | 2 +-\n command-list.txt | 1 +\n git.c | 1 +\n hook.c | 416 ++++++++++++++++++\n hook.h | 154 +++++++\n parse-options-cb.c | 16 +\n parse-options.h | 4 +\n run-command.c | 85 +++-\n run-command.h | 31 ++\n submodule.c | 1 +\n t/helper/test-run-command.c | 46 +-\n t/t0061-run-command.sh | 37 ++\n t/t1360-config-based-hooks.sh | 256 +++++++++++\n 24 files changed, 1717 insertions(+), 15 deletions(-)\n create mode 100644 Documentation/config/hook.txt\n create mode 100644 Documentation/git-hook.txt\n create mode 100644 Documentation/technical/config-based-hooks.txt\n create mode 100644 builtin/hook.c\n create mode 100644 hook.c\n create mode 100644 hook.h\n create mode 100755 t/t1360-config-based-hooks.sh\n\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412793","messageId":"20201222000220.1491091-2-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 01/17] doc: propose hooks managed by the config","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:04Z","receivedAt":"2020-12-22T00:03:08Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Begin a design document for config-based hooks, managed via git-hook.\nFocus on an overview of the implementation and motivation for design\ndecisions. Briefly discuss the alternatives considered before this\npoint. Also, attempt to redefine terms to fit into a multihook world.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Since v6, checked for inconsistencies with implementation and added lots of\n caveats about whether 'git hook add' and 'git hook edit' will ever materialize.\n \n Hopefully this reflects reality now; please review accordingly.\n \n Since v6, checked for inconsistencies with implementation and added lots of\n caveats about whether 'git hook add' and 'git hook edit' will ever materialize.\n \n Hopefully this reflects reality now; please review accordingly.\n \n Since v4, addressed comments from Jonathan Tan about wording. However, I have\n not addressed AEvar's comments or done a full re-review of this document.\n I wanted to get the rest of the series out for initial review first.\n \n - Emily\n \n Since v4, addressed comments from Jonathan Tan about wording.\n\n Documentation/Makefile | 1 +\n .../technical/config-based-hooks.txt | 355 ++++++++++++++++++\n 2 files changed, 356 insertions(+)\n create mode 100644 Documentation/technical/config-based-hooks.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex 69dbe4bb0b..505d318da1 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -81,6 +81,7 @@ SP_ARTICLES += $(API_DOCS)\n TECH_DOCS += MyFirstContribution\n TECH_DOCS += MyFirstObjectWalk\n TECH_DOCS += SubmittingPatches\n+TECH_DOCS += technical/config-based-hooks\n TECH_DOCS += technical/hash-function-transition\n TECH_DOCS += technical/http-protocol\n TECH_DOCS += technical/index-format\ndiff --git a/Documentation/technical/config-based-hooks.txt b/Documentation/technical/config-based-hooks.txt\nnew file mode 100644\nindex 0000000000..3217faba47\n--- /dev/null\n+++ b/Documentation/technical/config-based-hooks.txt\n@@ -0,0 +1,355 @@\n+Configuration-based hook management\n+===================================\n+:sectanchors:\n+\n+[[motivation]]\n+== Motivation\n+\n+Replace the .git/hook/hookname path as the only source of hooks to execute;\n+allow users to define hooks using config files, in a way which is friendly to\n+users with multiple repos which have similar needs.\n+\n+Redefine \"hook\" as an event rather than a single script, allowing users to\n+perform multiple unrelated actions on a single event.\n+\n+Take a step closer to safety when copying zipped Git repositories from untrusted\n+users by making it more apparent to users which scripts will be run during\n+normal Git operations.\n+\n+Make it easier for users to discover Git's hook feature and automate their\n+workflows.\n+\n+[[user-interfaces]]\n+== User interfaces\n+\n+[[config-schema]]\n+=== Config schema\n+\n+Hooks can be introduced by editing the configuration manually. There are two new\n+sections added, `hook` and `hookcmd`.\n+\n+[[config-schema-hook]]\n+==== `hook`\n+\n+Primarily contains subsections for each hook event. The order of variables in\n+these subsections defines the hook command execution order; hook commands can be\n+specified by setting the value directly to the command if no additional\n+configuration is needed, or by setting the value as the name of a `hookcmd`. If\n+Git does not find a `hookcmd` whose subsection matches the value of the given\n+command string, Git will try to execute the string directly. Hooks are executed\n+by passing the resolved command string to the shell. In the future, hook event\n+subsections could also contain per-hook-event settings; see\n+<> for more details.\n+\n+Also contains top-level hook execution settings, for example, `hook.runHookDir`.\n+(These settings are described more in <>.)\n+\n+----\n+[hook \"pre-commit\"]\n+ command = perl-linter\n+ command = /usr/bin/git-secrets --pre-commit\n+\n+[hook \"pre-applypatch\"]\n+ command = perl-linter\n+ # for illustration purposes; error behavior isn't planned yet\n+ error = ignore\n+\n+[hook]\n+ runHookDir = interactive\n+----\n+\n+[[config-schema-hookcmd]]\n+==== `hookcmd`\n+\n+Defines a hook command and its attributes, which will be used when a hook event\n+occurs. Unqualified attributes are assumed to apply to this hook during all hook\n+events, but event-specific attributes can also be supplied. The example runs\n+`/usr/bin/lint-it --language=perl `, but for repos which\n+include this config, the hook command will be skipped for all events.\n+Theoretically, the last line could be used to \"un-skip\" the hook command for\n+`pre-commit` hooks, but this hasn't been scoped or implemented yet.\n+\n+----\n+[hookcmd \"perl-linter\"]\n+ command = /usr/bin/lint-it --language=perl\n+ skip = true\n+ # for illustration purposes; below hasn't been defined yet\n+ pre-commit-skip = false\n+----\n+\n+[[command-line-api]]\n+=== Command-line API\n+\n+Users should be able to view, run, reorder, and create hook commands via the\n+command line. External tools should be able to view a list of hooks in the\n+correct order to run. Modifier commands (`edit` and `add`) have not been\n+implemented yet and may not be if manually editing the config proves usable\n+enough.\n+\n+*`git hook list `*\n+\n+*`git hook run [-a ]... [-e ]...`*\n+\n+*`git hook edit `*\n+\n+*`git hook add `*\n+\n+[[hook-editor]]\n+=== Hook editor\n+\n+The tool which is presented by `git hook edit `. Ideally, this\n+tool should be easier to use than manually editing the config, and then produce\n+a concise config afterwards. It may take a form similar to `git rebase\n+--interactive`. This has not been designed or implemented yet and may not be if\n+the config proves usable enough.\n+\n+[[implementation]]\n+== Implementation\n+\n+[[library]]\n+=== Library\n+\n+`hook.c` and `hook.h` are responsible for interacting with the config files. The\n+hook library provides a basic API to call all hooks in config order with more\n+complex options passed via `struct run_hooks_opt`:\n+\n+*`int run_hooks(const char *hookname, struct run_hooks_opt *options)`*\n+\n+`struct run_hooks_opt` allows callers to set:\n+\n+- environment variables\n+- command-line arguments\n+- behavior for the hook command provided by `run-command.h:find_hook()` (see\n+ below)\n+- a method to provide stdin to each hook, either via a file containing stdin, a\n+ `struct string_list` containing a list of lines to print, or a callback\n+ function to allow the caller to populate stdin manually\n+- a method to process stdout from each hook, e.g. for printing to sideband\n+ during a network operation\n+- parallelism\n+- a custom working directory for hooks to execute in\n+\n+And this struct can be extended with more options as necessary in the future.\n+\n+The \"legacy\" hook provided by `run-command.h:find_hook()` - that is, the hook\n+present in `.git/hooks/` or\n+`$(git config --get core.hooksPath)/` - can be handled in a number of\n+ways, providing an avenue to deprecate these \"legacy\" hooks if desired. The\n+handling is based on a config `hook.runHookDir`, which is checked against a\n+number of cases:\n+\n+- \"no\": the legacy hook will not be run\n+- \"interactive\": Git will prompt the user before running the legacy hook\n+- \"warn\": Git will print a warning to stderr before running the legacy hook\n+- \"yes\" (default): Git will silently run the legacy hook\n+\n+In case this list is expanded in the future, if a value for `hook.runHookDir` is\n+given which Git does not recognize, Git should discard that config entry. For\n+example, if \"warn\" was specified at system level and \"junk\" was specified at\n+global level, Git would resolve the value to \"warn\"; if the only time the config\n+was set was to \"junk\", Git would use the default value of \"yes\".\n+\n+`struct hookcmd` is expected to grow in size over time as more functionality is\n+added to hooks; so that other parts of the code don't need to understand the\n+config schema, `struct hookcmd` should contain logical values instead of string\n+pairs.\n+\n+By default, hook parallelism is chosen based on the semantics of each hook;\n+callsites initialize their `struct run_hooks_opt` via one of two macros,\n+`RUN_HOOKS_OPT_INIT_SYNC` or `RUN_HOOKS_OPT_INIT_ASYNC`. The default number of\n+jobs can be configured in `hook.jobs`; this config applies across all hook\n+events. If unset, the value of `online_cpus()` (equivalent to `nproc`) is used.\n+\n+[[builtin]]\n+=== Builtin\n+\n+`builtin/hook.c` is responsible for providing the frontend. It's responsible for\n+formatting user-provided data and then calling the library API to set the\n+configs as appropriate. The builtin frontend is not responsible for calling the\n+config directly, so that other areas of Git can rely on the hook library to\n+understand the most recent config schema for hooks.\n+\n+[[migration]]\n+=== Migration path\n+\n+[[stage-0]]\n+==== Stage 0\n+\n+Hooks are called by running `run-command.h:find_hook()` with the hookname and\n+executing the result. The hook library and builtin do not exist. Hooks only\n+exist as specially named scripts within `.git/hooks/`.\n+\n+[[stage-1]]\n+==== Stage 1\n+\n+`git hook list --porcelain ` is implemented. `hook.h:run_hooks()` is\n+taught to include `run-command.h:find_hook()` at the end; calls to `find_hook()`\n+are replaced with calls to `run_hooks()`. Users can opt-in to config-based hooks\n+simply by creating some in their config; otherwise users should remain\n+unaffected by the change.\n+\n+[[stage-2]]\n+==== Stage 2\n+\n+The call to `find_hook()` inside of `run_hooks()` learns to check for a config,\n+`hook.runHookDir`. Users can opt into managing their hooks completely via the\n+config this way.\n+\n+[[stage-3]]\n+==== Stage 3\n+\n+`.git/hooks` is removed from the template and the hook directory is considered\n+deprecated. To avoid breaking older repos, the default of `hook.runHookDir` is\n+not changed, and `find_hook()` is not removed.\n+\n+[[caveats]]\n+== Caveats\n+\n+[[security]]\n+=== Security and repo config\n+\n+Part of the motivation behind this refactor is to mitigate hooks as an attack\n+vector;footnote:[https://lore.kernel.org/git/20171002234517.GV19555@aiede.mtv.corp.google.com/]\n+however, as the design stands, users can still provide hooks in the repo-level\n+config, which is included when a repo is zipped and sent elsewhere. The\n+security of the repo-level config is still under discussion; this design\n+generally assumes the repo-level config is secure, which is not true yet. The\n+goal is to avoid an overcomplicated design to work around a problem which has\n+ceased to exist.\n+\n+[[ease-of-use]]\n+=== Ease of use\n+\n+The config schema is nontrivial; that's why it's important for the `git hook`\n+modifier commands to be usable. Contributors with UX expertise are encouraged to\n+share their suggestions.\n+\n+[[alternatives]]\n+== Alternative approaches\n+\n+A previous summary of alternatives exists in the\n+archives.footnote:[https://lore.kernel.org/git/20191116011125.GG22855@google.com]\n+\n+[[status-quo]]\n+=== Status quo\n+\n+Today users can implement multihooks themselves by using a \"trampoline script\"\n+as their hook, and pointing that script to a directory or list of other scripts\n+they wish to run.\n+\n+[[hook-directories]]\n+=== Hook directories\n+\n+Other contributors have suggested Git learn about the existence of a directory\n+such as `.git/hooks/.d` and execute those hooks in alphabetical order.\n+\n+[[comparison]]\n+=== Comparison table\n+\n+.Comparison of alternatives\n+|===\n+|Feature |Config-based hooks |Hook directories |Status quo\n+\n+|Supports multiple hooks\n+|Natively\n+|Natively\n+|With user effort\n+\n+|Safer for zipped repos\n+|A little\n+|No\n+|No\n+\n+|Previous hooks just work\n+|If configured\n+|Yes\n+|Yes\n+\n+|Can install one hook to many repos\n+|Yes\n+|No\n+|No\n+\n+|Discoverability\n+|Better (in `git help git`)\n+|Same as before\n+|Same as before\n+\n+|Hard to run unexpected hook\n+|If configured\n+|No\n+|No\n+|===\n+\n+[[future-work]]\n+== Future work\n+\n+[[execution-ordering]]\n+=== Execution ordering\n+\n+We may find that config order is insufficient for some users; for example,\n+config order makes it difficult to add a new hook to the system or global config\n+which runs at the end of the hook list. A new ordering schema should be:\n+\n+1) Specified by a `hook.order` config, so that users will not unexpectedly see\n+their order change;\n+\n+2) Either dependency or numerically based.\n+\n+Dependency-based ordering is prone to classic linked-list problems, like a\n+cycles and handling of missing dependencies. But, it paves the way for enabling\n+parallelization if some tasks truly depend on others.\n+\n+Numerical ordering makes it tricky for Git to generate suggested ordering\n+numbers for each command, but is easy to determine a definitive order.\n+\n+[[parallelization]]\n+=== Parallelization with dependencies\n+\n+Currently hooks use a naive parallelization scheme or are run in series. But if\n+one hook depends on another's output, then users will want to specify those\n+dependencies. If we decide to solve this problem, we may want to look to modern\n+build systems for inspiration on how to manage dependencies and parallel tasks.\n+\n+[[securing-hookdir-hooks]]\n+=== Securing hookdir hooks\n+\n+With the design as written in this doc, it's still possible for a malicious user\n+to modify `.git/config` to include `hook.pre-receive.command = rm -rf /`, then\n+zip their repo and send it to another user. It may be necessary to teach Git to\n+only allow inlined hooks like this if they were configured outside of the local\n+scope (in other words, only run hookcmds, and only allow hookcmds to be\n+configured in global or system scope); or another approach, like a list of safe\n+projects, might be useful. It may also be sufficient (or at least useful) to\n+teach a `hook.disableAll` config or similar flag to the Git executable.\n+\n+[[submodule-inheritance]]\n+=== Submodule inheritance\n+\n+It's possible some submodules may want to run the identical set of hooks that\n+their superrepo runs. While a globally-configured hook set is helpful, it's not\n+a great solution for users who have multiple repos-with-submodules under the\n+same user. It would be useful for submodules to learn how to run hooks from\n+their superrepo's config, or inherit that hook setting.\n+\n+[[per-hook-event-settings]]\n+=== Per-hook-event settings\n+\n+It might be desirable to keep settings specifically for some hook events, but\n+not for others - for example, a user may wish to disable hookdir hooks for all\n+events but pre-commit, which they haven't had time to convert yet; or, a user\n+may wish for execution order settings to differ based on hook event. In that\n+case, it would be useful to set something like `hook.pre-commit.executionOrder`\n+which would not apply to the 'prepare-commit-msg' hook, for example.\n+\n+[[glossary]]\n+== Glossary\n+\n+*hook event*\n+\n+A point during Git's execution where user scripts may be run, for example,\n+_prepare-commit-msg_ or _pre-push_.\n+\n+*hook command*\n+\n+A user script or executable which will be run on one or more hook events.\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412794","messageId":"20201222000220.1491091-3-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 02/17] hook: scaffolding for git-hook subcommand","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:05Z","receivedAt":"2020-12-22T00:03:10Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Introduce infrastructure for a new subcommand, git-hook, which will be\nused to ease config-based hook management. This command will handle\nparsing configs to compose a list of hooks to run for a given event, as\nwell as adding or modifying hook configs in an interactive fashion.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Since v4, mainly changed to RUN_SETUP_GENTLY so that 'git hook list' can\n be executed outside of a repo.\n\n .gitignore | 1 +\n Documentation/git-hook.txt | 20 ++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/hook.c | 21 +++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n t/t1360-config-based-hooks.sh | 11 +++++++++++\n 8 files changed, 57 insertions(+)\n create mode 100644 Documentation/git-hook.txt\n create mode 100644 builtin/hook.c\n create mode 100755 t/t1360-config-based-hooks.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 3dcdb6bb5a..3608c35b73 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -76,6 +76,7 @@\n /git-grep\n /git-hash-object\n /git-help\n+/git-hook\n /git-http-backend\n /git-http-fetch\n /git-http-push\ndiff --git a/Documentation/git-hook.txt b/Documentation/git-hook.txt\nnew file mode 100644\nindex 0000000000..9eeab0009d\n--- /dev/null\n+++ b/Documentation/git-hook.txt\n@@ -0,0 +1,20 @@\n+git-hook(1)\n+===========\n+\n+NAME\n+----\n+git-hook - Manage configured hooks\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git hook'\n+\n+DESCRIPTION\n+-----------\n+A placeholder command. Later, you will be able to list, add, and modify hooks\n+with this command.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex 6fb86c5862..24cee44400 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1101,6 +1101,7 @@ BUILTIN_OBJS += builtin/get-tar-commit-id.o\n BUILTIN_OBJS += builtin/grep.o\n BUILTIN_OBJS += builtin/hash-object.o\n BUILTIN_OBJS += builtin/help.o\n+BUILTIN_OBJS += builtin/hook.o\n BUILTIN_OBJS += builtin/index-pack.o\n BUILTIN_OBJS += builtin/init-db.o\n BUILTIN_OBJS += builtin/interpret-trailers.o\ndiff --git a/builtin.h b/builtin.h\nindex b6ce981b73..8df1d36a7a 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -163,6 +163,7 @@ int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);\n int cmd_grep(int argc, const char **argv, const char *prefix);\n int cmd_hash_object(int argc, const char **argv, const char *prefix);\n int cmd_help(int argc, const char **argv, const char *prefix);\n+int cmd_hook(int argc, const char **argv, const char *prefix);\n int cmd_index_pack(int argc, const char **argv, const char *prefix);\n int cmd_init_db(int argc, const char **argv, const char *prefix);\n int cmd_interpret_trailers(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/hook.c b/builtin/hook.c\nnew file mode 100644\nindex 0000000000..b2bbc84d4d\n--- /dev/null\n+++ b/builtin/hook.c\n@@ -0,0 +1,21 @@\n+#include \"cache.h\"\n+\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+\n+static const char * const builtin_hook_usage[] = {\n+\tN_(\"git hook\"),\n+\tNULL\n+};\n+\n+int cmd_hook(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct option builtin_hook_options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, builtin_hook_options,\n+\t\t\t builtin_hook_usage, 0);\n+\n+\treturn 0;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex 9379b02e5e..75909bf602 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -103,6 +103,7 @@ git-grep mainporcelain info\n git-gui mainporcelain\n git-hash-object plumbingmanipulators\n git-help ancillaryinterrogators complete\n+git-hook mainporcelain\n git-http-backend synchingrepositories\n git-http-fetch synchelpers\n git-http-push synchelpers\ndiff --git a/git.c b/git.c\nindex a00a0a4d94..9d1768b8e8 100644\n--- a/git.c\n+++ b/git.c\n@@ -525,6 +525,7 @@ static struct cmd_struct commands[] = {\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n \t{ \"hash-object\", cmd_hash_object },\n \t{ \"help\", cmd_help },\n+\t{ \"hook\", cmd_hook, RUN_SETUP_GENTLY },\n \t{ \"index-pack\", cmd_index_pack, RUN_SETUP_GENTLY | NO_PARSEOPT },\n \t{ \"init\", cmd_init_db },\n \t{ \"init-db\", cmd_init_db },\ndiff --git a/t/t1360-config-based-hooks.sh b/t/t1360-config-based-hooks.sh\nnew file mode 100755\nindex 0000000000..34b0df5216\n--- /dev/null\n+++ b/t/t1360-config-based-hooks.sh\n@@ -0,0 +1,11 @@\n+#!/bin/bash\n+\n+test_description='config-managed multihooks, including git-hook command'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'git hook command does not crash' '\n+\tgit hook\n+'\n+\n+test_done\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412795","messageId":"20201222000220.1491091-4-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 03/17] hook: add list command","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:06Z","receivedAt":"2020-12-22T00:03:12Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Teach 'git hook list ', which checks the known configs in\norder to create an ordered list of hooks to run on a given hook event.\n\nMultiple commands can be specified for a given hook by providing\nmultiple \"hook..command = \" lines. Hooks will be\nrun in config order. If more properties need to be set on a given hook\nin the future, commands can also be specified by providing\n\"hook..command = \", as well as a \"[hookcmd\n]\" subsection; at minimum, this subsection must contain a\n\"hookcmd..command = \" line.\n\nFor example:\n\n $ git config --list | grep ^hook\n hook.pre-commit.command=baz\n hook.pre-commit.command=~/bar.sh\n hookcmd.baz.command=~/baz/from/hookcmd.sh\n\n $ git hook list pre-commit\n global: ~/baz/from/hookcmd.sh\n local: ~/bar.sh\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Since v4, updated the sample in the commit message to reflect reality better.\n \n Since v4, more work on the documentation. Also a slight change to the\n output format (space instead of tab).\n\n Documentation/config/hook.txt | 9 +++\n Documentation/git-hook.txt | 59 ++++++++++++++++-\n Makefile | 1 +\n builtin/hook.c | 56 +++++++++++++++--\n hook.c | 115 ++++++++++++++++++++++++++++++++++\n hook.h | 26 ++++++++\n t/t1360-config-based-hooks.sh | 81 +++++++++++++++++++++++-\n 7 files changed, 338 insertions(+), 9 deletions(-)\n create mode 100644 Documentation/config/hook.txt\n create mode 100644 hook.c\n create mode 100644 hook.h\n\ndiff --git a/Documentation/config/hook.txt b/Documentation/config/hook.txt\nnew file mode 100644\nindex 0000000000..71449ecbc7\n--- /dev/null\n+++ b/Documentation/config/hook.txt\n@@ -0,0 +1,9 @@\n+hook..command::\n+\tA command to execute during the hook event. This can be an\n+\texecutable on your device, a oneliner for your shell, or the name of a\n+\thookcmd. See linkgit:git-hook[1].\n+\n+hookcmd..command::\n+\tA command to execute during a hook for which has been specified\n+\tas a command. This can be an executable on your device or a oneliner for\n+\tyour shell. See linkgit:git-hook[1].\ndiff --git a/Documentation/git-hook.txt b/Documentation/git-hook.txt\nindex 9eeab0009d..f19875ed68 100644\n--- a/Documentation/git-hook.txt\n+++ b/Documentation/git-hook.txt\n@@ -8,12 +8,65 @@ git-hook - Manage configured hooks\n SYNOPSIS\n --------\n [verse]\n-'git hook'\n+'git hook' list \n \n DESCRIPTION\n -----------\n-A placeholder command. Later, you will be able to list, add, and modify hooks\n-with this command.\n+You can list configured hooks with this command. Later, you will be able to run,\n+add, and modify hooks with this command.\n+\n+This command parses the default configuration files for sections `hook` and\n+`hookcmd`. `hook` is used to describe the commands which will be run during a\n+particular hook event; commands are run in the order Git encounters them during\n+the configuration parse (see linkgit:git-config[1]). `hookcmd` is used to\n+describe attributes of a specific command. If additional attributes don't need\n+to be specified, a command to run can be specified directly in the `hook`\n+section; if a `hookcmd` by that name isn't found, Git will attempt to run the\n+provided value directly. For example:\n+\n+Global config\n+----\n+ [hook \"post-commit\"]\n+ command = \"linter\"\n+ command = \"~/typocheck.sh\"\n+\n+ [hookcmd \"linter\"]\n+ command = \"/bin/linter --c\"\n+----\n+\n+Local config\n+----\n+ [hook \"prepare-commit-msg\"]\n+ command = \"linter\"\n+ [hook \"post-commit\"]\n+ command = \"python ~/run-test-suite.py\"\n+----\n+\n+With these configs, you'd then see:\n+\n+----\n+$ git hook list \"post-commit\"\n+global: /bin/linter --c\n+global: ~/typocheck.sh\n+local: python ~/run-test-suite.py\n+\n+$ git hook list \"prepare-commit-msg\"\n+local: /bin/linter --c\n+----\n+\n+COMMANDS\n+--------\n+\n+list ``::\n+\n+List the hooks which have been configured for ``. Hooks appear\n+in the order they should be run, and print the config scope where the relevant\n+`hook..command` was specified, not the `hookcmd` (if applicable).\n+This output is human-readable and the format is subject to change over time.\n+\n+CONFIGURATION\n+-------------\n+include::config/hook.txt[]\n \n GIT\n ---\ndiff --git a/Makefile b/Makefile\nindex 24cee44400..d9f43dc8fe 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -904,6 +904,7 @@ LIB_OBJS += grep.o\n LIB_OBJS += hashmap.o\n LIB_OBJS += help.o\n LIB_OBJS += hex.o\n+LIB_OBJS += hook.o\n LIB_OBJS += ident.o\n LIB_OBJS += json-writer.o\n LIB_OBJS += kwset.o\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex b2bbc84d4d..4d36de52f8 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -1,21 +1,69 @@\n #include \"cache.h\"\n \n #include \"builtin.h\"\n+#include \"config.h\"\n+#include \"hook.h\"\n #include \"parse-options.h\"\n+#include \"strbuf.h\"\n \n static const char * const builtin_hook_usage[] = {\n-\tN_(\"git hook\"),\n+\tN_(\"git hook list \"),\n \tNULL\n };\n \n-int cmd_hook(int argc, const char **argv, const char *prefix)\n+static int list(int argc, const char **argv, const char *prefix)\n {\n-\tstruct option builtin_hook_options[] = {\n+\tstruct list_head *head, *pos;\n+\tstruct hook *item;\n+\tstruct strbuf hookname = STRBUF_INIT;\n+\n+\tstruct option list_options[] = {\n \t\tOPT_END(),\n \t};\n \n-\targc = parse_options(argc, argv, prefix, builtin_hook_options,\n+\targc = parse_options(argc, argv, prefix, list_options,\n \t\t\t builtin_hook_usage, 0);\n \n+\tif (argc < 1) {\n+\t\tusage_msg_opt(_(\"You must specify a hook event name to list.\"),\n+\t\t\t builtin_hook_usage, list_options);\n+\t}\n+\n+\tstrbuf_addstr(&hookname, argv[0]);\n+\n+\thead = hook_list(&hookname);\n+\n+\tif (list_empty(head)) {\n+\t\tprintf(_(\"no commands configured for hook '%s'\\n\"),\n+\t\t hookname.buf);\n+\t\tstrbuf_release(&hookname);\n+\t\treturn 0;\n+\t}\n+\n+\tlist_for_each(pos, head) {\n+\t\titem = list_entry(pos, struct hook, list);\n+\t\tif (item)\n+\t\t\tprintf(\"%s: %s\\n\",\n+\t\t\t config_scope_name(item->origin),\n+\t\t\t item->command.buf);\n+\t}\n+\n+\tclear_hook_list(head);\n+\tstrbuf_release(&hookname);\n+\n \treturn 0;\n }\n+\n+int cmd_hook(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct option builtin_hook_options[] = {\n+\t\tOPT_END(),\n+\t};\n+\tif (argc < 2)\n+\t\tusage_with_options(builtin_hook_usage, builtin_hook_options);\n+\n+\tif (!strcmp(argv[1], \"list\"))\n+\t\treturn list(argc - 1, argv + 1, prefix);\n+\n+\tusage_with_options(builtin_hook_usage, builtin_hook_options);\n+}\ndiff --git a/hook.c b/hook.c\nnew file mode 100644\nindex 0000000000..937dc768c8\n--- /dev/null\n+++ b/hook.c\n@@ -0,0 +1,115 @@\n+#include \"cache.h\"\n+\n+#include \"hook.h\"\n+#include \"config.h\"\n+\n+void free_hook(struct hook *ptr)\n+{\n+\tif (ptr) {\n+\t\tstrbuf_release(&ptr->command);\n+\t\tfree(ptr);\n+\t}\n+}\n+\n+static void append_or_move_hook(struct list_head *head, const char *command)\n+{\n+\tstruct list_head *pos = NULL, *tmp = NULL;\n+\tstruct hook *to_add = NULL;\n+\n+\t/*\n+\t * remove the prior entry with this command; we'll replace it at the\n+\t * end.\n+\t */\n+\tlist_for_each_safe(pos, tmp, head) {\n+\t\tstruct hook *it = list_entry(pos, struct hook, list);\n+\t\tif (!strcmp(it->command.buf, command)) {\n+\t\t list_del(pos);\n+\t\t /* we'll simply move the hook to the end */\n+\t\t to_add = it;\n+\t\t}\n+\t}\n+\n+\tif (!to_add) {\n+\t\t/* adding a new hook, not moving an old one */\n+\t\tto_add = xmalloc(sizeof(struct hook));\n+\t\tstrbuf_init(&to_add->command, 0);\n+\t\tstrbuf_addstr(&to_add->command, command);\n+\t}\n+\n+\t/* re-set the scope so we show where an override was specified */\n+\tto_add->origin = current_config_scope();\n+\n+\tlist_add_tail(&to_add->list, pos);\n+}\n+\n+static void remove_hook(struct list_head *to_remove)\n+{\n+\tstruct hook *hook_to_remove = list_entry(to_remove, struct hook, list);\n+\tlist_del(to_remove);\n+\tfree_hook(hook_to_remove);\n+}\n+\n+void clear_hook_list(struct list_head *head)\n+{\n+\tstruct list_head *pos, *tmp;\n+\tlist_for_each_safe(pos, tmp, head)\n+\t\tremove_hook(pos);\n+}\n+\n+struct hook_config_cb\n+{\n+\tstruct strbuf *hookname;\n+\tstruct list_head *list;\n+};\n+\n+static int hook_config_lookup(const char *key, const char *value, void *cb_data)\n+{\n+\tstruct hook_config_cb *data = cb_data;\n+\tconst char *hook_key = data->hookname->buf;\n+\tstruct list_head *head = data->list;\n+\n+\tif (!strcmp(key, hook_key)) {\n+\t\tconst char *command = value;\n+\t\tstruct strbuf hookcmd_name = STRBUF_INIT;\n+\n+\t\t/* Check if a hookcmd with that name exists. */\n+\t\tstrbuf_addf(&hookcmd_name, \"hookcmd.%s.command\", command);\n+\t\tgit_config_get_value(hookcmd_name.buf, &command);\n+\n+\t\tif (!command) {\n+\t\t\tstrbuf_release(&hookcmd_name);\n+\t\t\tBUG(\"git_config_get_value overwrote a string it shouldn't have\");\n+\t\t}\n+\n+\t\t/*\n+\t\t * TODO: implement an option-getting callback, e.g.\n+\t\t * get configs by pattern hookcmd.$value.*\n+\t\t * for each key+value, do_callback(key, value, cb_data)\n+\t\t */\n+\n+\t\tappend_or_move_hook(head, command);\n+\n+\t\tstrbuf_release(&hookcmd_name);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+struct list_head* hook_list(const struct strbuf* hookname)\n+{\n+\tstruct strbuf hook_key = STRBUF_INIT;\n+\tstruct list_head *hook_head = xmalloc(sizeof(struct list_head));\n+\tstruct hook_config_cb cb_data = { &hook_key, hook_head };\n+\n+\tINIT_LIST_HEAD(hook_head);\n+\n+\tif (!hookname)\n+\t\treturn NULL;\n+\n+\tstrbuf_addf(&hook_key, \"hook.%s.command\", hookname->buf);\n+\n+\tgit_config(hook_config_lookup, (void*)&cb_data);\n+\n+\tstrbuf_release(&hook_key);\n+\treturn hook_head;\n+}\ndiff --git a/hook.h b/hook.h\nnew file mode 100644\nindex 0000000000..8ffc4f14b6\n--- /dev/null\n+++ b/hook.h\n@@ -0,0 +1,26 @@\n+#include \"config.h\"\n+#include \"list.h\"\n+#include \"strbuf.h\"\n+\n+struct hook\n+{\n+\tstruct list_head list;\n+\t/*\n+\t * Config file which holds the hook.*.command definition.\n+\t * (This has nothing to do with the hookcmd..* configs.)\n+\t */\n+\tenum config_scope origin;\n+\t/* The literal command to run. */\n+\tstruct strbuf command;\n+};\n+\n+/*\n+ * Provides a linked list of 'struct hook' detailing commands which should run\n+ * in response to the 'hookname' event, in execution order.\n+ */\n+struct list_head* hook_list(const struct strbuf *hookname);\n+\n+/* Free memory associated with a 'struct hook' */\n+void free_hook(struct hook *ptr);\n+/* Empties the list at 'head', calling 'free_hook()' on each entry */\n+void clear_hook_list(struct list_head *head);\ndiff --git a/t/t1360-config-based-hooks.sh b/t/t1360-config-based-hooks.sh\nindex 34b0df5216..6e4a3e763f 100755\n--- a/t/t1360-config-based-hooks.sh\n+++ b/t/t1360-config-based-hooks.sh\n@@ -4,8 +4,85 @@ test_description='config-managed multihooks, including git-hook command'\n \n . ./test-lib.sh\n \n-test_expect_success 'git hook command does not crash' '\n-\tgit hook\n+ROOT=\n+if test_have_prereq MINGW\n+then\n+\t# In Git for Windows, Unix-like paths work only in shell scripts;\n+\t# `git.exe`, however, will prefix them with the pseudo root directory\n+\t# (of the Unix shell). Let's accommodate for that.\n+\tROOT=\"$(cd / && pwd)\"\n+fi\n+\n+setup_hooks () {\n+\ttest_config hook.pre-commit.command \"/path/ghi\" --add\n+\ttest_config_global hook.pre-commit.command \"/path/def\" --add\n+}\n+\n+setup_hookcmd () {\n+\ttest_config hook.pre-commit.command \"abc\" --add\n+\ttest_config_global hookcmd.abc.command \"/path/abc\" --add\n+}\n+\n+test_expect_success 'git hook rejects commands without a mode' '\n+\ttest_must_fail git hook pre-commit\n+'\n+\n+\n+test_expect_success 'git hook rejects commands without a hookname' '\n+\ttest_must_fail git hook list\n+'\n+\n+test_expect_success 'git hook runs outside of a repo' '\n+\tsetup_hooks &&\n+\n+\tcat >expected <<-EOF &&\n+\tglobal: $ROOT/path/def\n+\tEOF\n+\n+\tnongit git config --list --global &&\n+\n+\tnongit git hook list pre-commit >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'git hook list orders by config order' '\n+\tsetup_hooks &&\n+\n+\tcat >expected <<-EOF &&\n+\tglobal: $ROOT/path/def\n+\tlocal: $ROOT/path/ghi\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'git hook list dereferences a hookcmd' '\n+\tsetup_hooks &&\n+\tsetup_hookcmd &&\n+\n+\tcat >expected <<-EOF &&\n+\tglobal: $ROOT/path/def\n+\tlocal: $ROOT/path/ghi\n+\tlocal: $ROOT/path/abc\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'git hook list reorders on duplicate commands' '\n+\tsetup_hooks &&\n+\n+\ttest_config hook.pre-commit.command \"/path/def\" --add &&\n+\n+\tcat >expected <<-EOF &&\n+\tlocal: $ROOT/path/ghi\n+\tlocal: $ROOT/path/def\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\ttest_cmp expected actual\n '\n \n test_done\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412796","messageId":"20201222000220.1491091-5-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 04/17] hook: include hookdir hook in list","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:07Z","receivedAt":"2020-12-22T00:03:13Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Historically, hooks are declared by placing an executable into\n$GIT_DIR/hooks/$HOOKNAME (or $HOOKDIR/$HOOKNAME). Although hooks taken\nfrom the config are more featureful than hooks placed in the $HOOKDIR,\nthose hooks should not stop working for users who already have them.\n\nLegacy hooks should be run directly, not in shell. We know that they are\na path to an executable, not a oneliner script - and running them\ndirectly takes care of path quoting concerns for us for free.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Newly split into its own commit since v4, and taking place much sooner.\n \n An unfortunate side effect of adding this support *before* the\n hook.runHookDir support is that the labels on the list are not clear -\n because we aren't yet flagging which hooks are from the hookdir versus\n the config. I suppose we could move the addition of that field to the\n struct hook up to this patch, but it didn't make a lot of sense to me to\n do it just for cosmetic purposes.\n\n builtin/hook.c | 18 ++++++++++++++----\n hook.c | 15 +++++++++++++++\n hook.h | 1 +\n t/t1360-config-based-hooks.sh | 19 +++++++++++++++++++\n 4 files changed, 49 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex 4d36de52f8..a0013ae4d7 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -16,6 +16,7 @@ static int list(int argc, const char **argv, const char *prefix)\n \tstruct list_head *head, *pos;\n \tstruct hook *item;\n \tstruct strbuf hookname = STRBUF_INIT;\n+\tstruct strbuf hookdir_annotation = STRBUF_INIT;\n \n \tstruct option list_options[] = {\n \t\tOPT_END(),\n@@ -42,10 +43,17 @@ static int list(int argc, const char **argv, const char *prefix)\n \n \tlist_for_each(pos, head) {\n \t\titem = list_entry(pos, struct hook, list);\n-\t\tif (item)\n-\t\t\tprintf(\"%s: %s\\n\",\n-\t\t\t config_scope_name(item->origin),\n-\t\t\t item->command.buf);\n+\t\tif (item) {\n+\t\t\t/* Don't translate 'hookdir' - it matches the config */\n+\t\t\tprintf(\"%s: %s%s\\n\",\n+\t\t\t (item->from_hookdir\n+\t\t\t\t? \"hookdir\"\n+\t\t\t\t: config_scope_name(item->origin)),\n+\t\t\t item->command.buf,\n+\t\t\t (item->from_hookdir\n+\t\t\t\t? hookdir_annotation.buf\n+\t\t\t\t: \"\"));\n+\t\t}\n \t}\n \n \tclear_hook_list(head);\n@@ -62,6 +70,8 @@ int cmd_hook(int argc, const char **argv, const char *prefix)\n \tif (argc < 2)\n \t\tusage_with_options(builtin_hook_usage, builtin_hook_options);\n \n+\tgit_config(git_default_config, NULL);\n+\n \tif (!strcmp(argv[1], \"list\"))\n \t\treturn list(argc - 1, argv + 1, prefix);\n \ndiff --git a/hook.c b/hook.c\nindex 937dc768c8..ffbdcfd987 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -2,6 +2,7 @@\n \n #include \"hook.h\"\n #include \"config.h\"\n+#include \"run-command.h\"\n \n void free_hook(struct hook *ptr)\n {\n@@ -34,6 +35,7 @@ static void append_or_move_hook(struct list_head *head, const char *command)\n \t\tto_add = xmalloc(sizeof(struct hook));\n \t\tstrbuf_init(&to_add->command, 0);\n \t\tstrbuf_addstr(&to_add->command, command);\n+\t\tto_add->from_hookdir = 0;\n \t}\n \n \t/* re-set the scope so we show where an override was specified */\n@@ -100,6 +102,7 @@ struct list_head* hook_list(const struct strbuf* hookname)\n \tstruct strbuf hook_key = STRBUF_INIT;\n \tstruct list_head *hook_head = xmalloc(sizeof(struct list_head));\n \tstruct hook_config_cb cb_data = { &hook_key, hook_head };\n+\tconst char *legacy_hook_path = NULL;\n \n \tINIT_LIST_HEAD(hook_head);\n \n@@ -110,6 +113,18 @@ struct list_head* hook_list(const struct strbuf* hookname)\n \n \tgit_config(hook_config_lookup, (void*)&cb_data);\n \n+\tif (have_git_dir())\n+\t\tlegacy_hook_path = find_hook(hookname->buf);\n+\n+\t/* Unconditionally add legacy hook, but annotate it. */\n+\tif (legacy_hook_path) {\n+\t\tstruct hook *legacy_hook;\n+\n+\t\tappend_or_move_hook(hook_head, absolute_path(legacy_hook_path));\n+\t\tlegacy_hook = list_entry(hook_head->prev, struct hook, list);\n+\t\tlegacy_hook->from_hookdir = 1;\n+\t}\n+\n \tstrbuf_release(&hook_key);\n \treturn hook_head;\n }\ndiff --git a/hook.h b/hook.h\nindex 8ffc4f14b6..5750634c83 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -12,6 +12,7 @@ struct hook\n \tenum config_scope origin;\n \t/* The literal command to run. */\n \tstruct strbuf command;\n+\tint from_hookdir;\n };\n \n /*\ndiff --git a/t/t1360-config-based-hooks.sh b/t/t1360-config-based-hooks.sh\nindex 6e4a3e763f..0f12af4659 100755\n--- a/t/t1360-config-based-hooks.sh\n+++ b/t/t1360-config-based-hooks.sh\n@@ -23,6 +23,14 @@ setup_hookcmd () {\n \ttest_config_global hookcmd.abc.command \"/path/abc\" --add\n }\n \n+setup_hookdir () {\n+\tmkdir .git/hooks\n+\twrite_script .git/hooks/pre-commit <<-EOF\n+\techo \\\"Legacy Hook\\\"\n+\tEOF\n+\ttest_when_finished rm -rf .git/hooks\n+}\n+\n test_expect_success 'git hook rejects commands without a mode' '\n \ttest_must_fail git hook pre-commit\n '\n@@ -85,4 +93,15 @@ test_expect_success 'git hook list reorders on duplicate commands' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'git hook list shows hooks from the hookdir' '\n+\tsetup_hookdir &&\n+\n+\tcat >expected <<-EOF &&\n+\thookdir: $(pwd)/.git/hooks/pre-commit\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412797","messageId":"20201222000220.1491091-6-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 05/17] hook: respect hook.runHookDir","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:08Z","receivedAt":"2020-12-22T00:03:47Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Include hooks specified in the hook directory in the list of hooks to\nrun. These hooks do need to be treated differently from config-specified\nones - they do not need to run in a shell, and later on may be disabled\nor warned about based on a config setting.\n\nBecause they are at least as local as the local config, we'll run them\nlast - to keep the hook execution order from global to local.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Newly split into its own commit since v4, and taking place much sooner.\n \n An unfortunate side effect of adding this support *before* the\n hook.runHookDir support is that the labels on the list are not clear -\n because we aren't yet flagging which hooks are from the hookdir versus\n the config. I suppose we could move the addition of that field to the\n struct hook up to this patch, but it didn't make a lot of sense to me to\n do it just for cosmetic purposes.\n\n Documentation/config/hook.txt | 5 ++++\n builtin/hook.c | 54 +++++++++++++++++++++++++++++++++--\n hook.c | 21 ++++++++++++++\n hook.h | 15 ++++++++++\n t/t1360-config-based-hooks.sh | 43 ++++++++++++++++++++++++++++\n 5 files changed, 135 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/hook.txt b/Documentation/config/hook.txt\nindex 71449ecbc7..75312754ae 100644\n--- a/Documentation/config/hook.txt\n+++ b/Documentation/config/hook.txt\n@@ -7,3 +7,8 @@ hookcmd..command::\n \tA command to execute during a hook for which has been specified\n \tas a command. This can be an executable on your device or a oneliner for\n \tyour shell. See linkgit:git-hook[1].\n+\n+hook.runHookDir::\n+\tControls how hooks contained in your hookdir are executed. Can be any of\n+\t\"yes\", \"warn\", \"interactive\", or \"no\". Defaults to \"yes\". See\n+\tlinkgit:git-hook[1] and linkgit:git-config[1] \"core.hooksPath\").\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex a0013ae4d7..d087e6f5b0 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -11,6 +11,8 @@ static const char * const builtin_hook_usage[] = {\n \tNULL\n };\n \n+static enum hookdir_opt should_run_hookdir;\n+\n static int list(int argc, const char **argv, const char *prefix)\n {\n \tstruct list_head *head, *pos;\n@@ -41,6 +43,26 @@ static int list(int argc, const char **argv, const char *prefix)\n \t\treturn 0;\n \t}\n \n+\tswitch (should_run_hookdir) {\n+\t\tcase HOOKDIR_NO:\n+\t\t\tstrbuf_addstr(&hookdir_annotation, _(\" (will not run)\"));\n+\t\t\tbreak;\n+\t\tcase HOOKDIR_INTERACTIVE:\n+\t\t\tstrbuf_addstr(&hookdir_annotation, _(\" (will prompt)\"));\n+\t\t\tbreak;\n+\t\tcase HOOKDIR_WARN:\n+\t\tcase HOOKDIR_UNKNOWN:\n+\t\t\tstrbuf_addstr(&hookdir_annotation, _(\" (will warn)\"));\n+\t\t\tbreak;\n+\t\tcase HOOKDIR_YES:\n+\t\t/*\n+\t\t * The default behavior should agree with\n+\t\t * hook.c:configured_hookdir_opt().\n+\t\t */\n+\t\tdefault:\n+\t\t\tbreak;\n+\t}\n+\n \tlist_for_each(pos, head) {\n \t\titem = list_entry(pos, struct hook, list);\n \t\tif (item) {\n@@ -64,16 +86,42 @@ static int list(int argc, const char **argv, const char *prefix)\n \n int cmd_hook(int argc, const char **argv, const char *prefix)\n {\n+\tconst char *run_hookdir = NULL;\n+\n \tstruct option builtin_hook_options[] = {\n+\t\tOPT_STRING(0, \"run-hookdir\", &run_hookdir, N_(\"option\"),\n+\t\t\t N_(\"what to do with hooks found in the hookdir\")),\n \t\tOPT_END(),\n \t};\n-\tif (argc < 2)\n+\n+\targc = parse_options(argc, argv, prefix, builtin_hook_options,\n+\t\t\t builtin_hook_usage, 0);\n+\n+\t/* after the parse, we should have \" \" */\n+\tif (argc < 1)\n \t\tusage_with_options(builtin_hook_usage, builtin_hook_options);\n \n \tgit_config(git_default_config, NULL);\n \n-\tif (!strcmp(argv[1], \"list\"))\n-\t\treturn list(argc - 1, argv + 1, prefix);\n+\n+\t/* argument > config */\n+\tif (run_hookdir)\n+\t\tif (!strcmp(run_hookdir, \"no\"))\n+\t\t\tshould_run_hookdir = HOOKDIR_NO;\n+\t\telse if (!strcmp(run_hookdir, \"yes\"))\n+\t\t\tshould_run_hookdir = HOOKDIR_YES;\n+\t\telse if (!strcmp(run_hookdir, \"warn\"))\n+\t\t\tshould_run_hookdir = HOOKDIR_WARN;\n+\t\telse if (!strcmp(run_hookdir, \"interactive\"))\n+\t\t\tshould_run_hookdir = HOOKDIR_INTERACTIVE;\n+\t\telse\n+\t\t\tdie(_(\"'%s' is not a valid option for --run-hookdir \"\n+\t\t\t \"(yes, warn, interactive, no)\"), run_hookdir);\n+\telse\n+\t\tshould_run_hookdir = configured_hookdir_opt();\n+\n+\tif (!strcmp(argv[0], \"list\"))\n+\t\treturn list(argc, argv, prefix);\n \n \tusage_with_options(builtin_hook_usage, builtin_hook_options);\n }\ndiff --git a/hook.c b/hook.c\nindex ffbdcfd987..ed52e85159 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -97,6 +97,27 @@ static int hook_config_lookup(const char *key, const char *value, void *cb_data)\n \treturn 0;\n }\n \n+enum hookdir_opt configured_hookdir_opt(void)\n+{\n+\tconst char *key;\n+\tif (git_config_get_value(\"hook.runhookdir\", &key))\n+\t\treturn HOOKDIR_YES; /* by default, just run it. */\n+\n+\tif (!strcmp(key, \"no\"))\n+\t\treturn HOOKDIR_NO;\n+\n+\tif (!strcmp(key, \"yes\"))\n+\t\treturn HOOKDIR_YES;\n+\n+\tif (!strcmp(key, \"warn\"))\n+\t\treturn HOOKDIR_WARN;\n+\n+\tif (!strcmp(key, \"interactive\"))\n+\t\treturn HOOKDIR_INTERACTIVE;\n+\n+\treturn HOOKDIR_UNKNOWN;\n+}\n+\n struct list_head* hook_list(const struct strbuf* hookname)\n {\n \tstruct strbuf hook_key = STRBUF_INIT;\ndiff --git a/hook.h b/hook.h\nindex 5750634c83..ccdf6272f2 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -21,6 +21,21 @@ struct hook\n */\n struct list_head* hook_list(const struct strbuf *hookname);\n \n+enum hookdir_opt\n+{\n+\tHOOKDIR_NO,\n+\tHOOKDIR_WARN,\n+\tHOOKDIR_INTERACTIVE,\n+\tHOOKDIR_YES,\n+\tHOOKDIR_UNKNOWN,\n+};\n+\n+/*\n+ * Provides the hookdir_opt specified in the config without consulting any\n+ * command line arguments.\n+ */\n+enum hookdir_opt configured_hookdir_opt(void);\n+\n /* Free memory associated with a 'struct hook' */\n void free_hook(struct hook *ptr);\n /* Empties the list at 'head', calling 'free_hook()' on each entry */\ndiff --git a/t/t1360-config-based-hooks.sh b/t/t1360-config-based-hooks.sh\nindex 0f12af4659..91127a50a4 100755\n--- a/t/t1360-config-based-hooks.sh\n+++ b/t/t1360-config-based-hooks.sh\n@@ -104,4 +104,47 @@ test_expect_success 'git hook list shows hooks from the hookdir' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'hook.runHookDir = no is respected by list' '\n+\tsetup_hookdir &&\n+\n+\ttest_config hook.runHookDir \"no\" &&\n+\n+\tcat >expected <<-EOF &&\n+\thookdir: $(pwd)/.git/hooks/pre-commit (will not run)\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\t# the hookdir annotation is translated\n+\ttest_i18ncmp expected actual\n+'\n+\n+test_expect_success 'hook.runHookDir = warn is respected by list' '\n+\tsetup_hookdir &&\n+\n+\ttest_config hook.runHookDir \"warn\" &&\n+\n+\tcat >expected <<-EOF &&\n+\thookdir: $(pwd)/.git/hooks/pre-commit (will warn)\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\t# the hookdir annotation is translated\n+\ttest_i18ncmp expected actual\n+'\n+\n+\n+test_expect_success 'hook.runHookDir = interactive is respected by list' '\n+\tsetup_hookdir &&\n+\n+\ttest_config hook.runHookDir \"interactive\" &&\n+\n+\tcat >expected <<-EOF &&\n+\thookdir: $(pwd)/.git/hooks/pre-commit (will prompt)\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\t# the hookdir annotation is translated\n+\ttest_i18ncmp expected actual\n+'\n+\n test_done\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412798","messageId":"20201222000220.1491091-7-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 06/17] hook: implement hookcmd..skip","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:09Z","receivedAt":"2020-12-22T00:03:47Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"If a user wants a specific repo to skip execution of a hook which is set\nat a global or system level, they can now do so by specifying 'skip' in\ntheir repo config:\n\n~/.gitconfig\n [hook.pre-commit]\n command = skippable-oneliner\n command = skippable-hookcmd\n\n [hookcmd.skippable-hookcmd]\n command = foo.sh\n\n$GIT_DIR/.git/config\n [hookcmd.skippable-oneliner]\n skip = true\n [hookcmd.skippable-hookcmd]\n skip = true\n\nLater it may make sense to add an option like\n\"hookcmd..-skip\" - but for simplicity, let's start\nwith a universal skip setting like this.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n In addition to being handy for turning off global hooks one project doesn't\n care about, this setting will be necessary much later for the 'proc-receive'\n hook, which can only cope with up to one hook being specified.\n \n New since v4.\n \n During the Google team's review club I was reminded about this whole\n 'skip' option I never implemented. It's true that it's impossible to\n exclude a given hook without this; however, I think I have some more\n work to do on it, so consider it RFC for now and tell me what you think\n :)\n - Emily\n \n During the Google team's review club this week I was reminded about this whole\n 'skip' option I never implemented. It's true that it's impossible to exclude\n a given hook without this; however, I think we have some more work to do on it,\n so consider it RFC for now and tell me what you think :)\n \n - Emily\n\n hook.c | 37 +++++++++++++++++++++++++----------\n t/t1360-config-based-hooks.sh | 23 ++++++++++++++++++++++\n 2 files changed, 50 insertions(+), 10 deletions(-)\n\ndiff --git a/hook.c b/hook.c\nindex ed52e85159..d262503725 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -12,23 +12,24 @@ void free_hook(struct hook *ptr)\n \t}\n }\n \n-static void append_or_move_hook(struct list_head *head, const char *command)\n+static struct hook* find_hook_by_command(struct list_head *head, const char *command)\n {\n \tstruct list_head *pos = NULL, *tmp = NULL;\n-\tstruct hook *to_add = NULL;\n+\tstruct hook *found = NULL;\n \n-\t/*\n-\t * remove the prior entry with this command; we'll replace it at the\n-\t * end.\n-\t */\n \tlist_for_each_safe(pos, tmp, head) {\n \t\tstruct hook *it = list_entry(pos, struct hook, list);\n \t\tif (!strcmp(it->command.buf, command)) {\n \t\t list_del(pos);\n-\t\t /* we'll simply move the hook to the end */\n-\t\t to_add = it;\n+\t\t found = it;\n \t\t}\n \t}\n+\treturn found;\n+}\n+\n+static void append_or_move_hook(struct list_head *head, const char *command)\n+{\n+\tstruct hook *to_add = find_hook_by_command(head, command);\n \n \tif (!to_add) {\n \t\t/* adding a new hook, not moving an old one */\n@@ -41,7 +42,7 @@ static void append_or_move_hook(struct list_head *head, const char *command)\n \t/* re-set the scope so we show where an override was specified */\n \tto_add->origin = current_config_scope();\n \n-\tlist_add_tail(&to_add->list, pos);\n+\tlist_add_tail(&to_add->list, head);\n }\n \n static void remove_hook(struct list_head *to_remove)\n@@ -73,8 +74,18 @@ static int hook_config_lookup(const char *key, const char *value, void *cb_data)\n \tif (!strcmp(key, hook_key)) {\n \t\tconst char *command = value;\n \t\tstruct strbuf hookcmd_name = STRBUF_INIT;\n+\t\tint skip = 0;\n+\n+\t\t/*\n+\t\t * Check if we're removing that hook instead. Hookcmds are\n+\t\t * removed by name, and inlined hooks are removed by command\n+\t\t * content.\n+\t\t */\n+\t\tstrbuf_addf(&hookcmd_name, \"hookcmd.%s.skip\", command);\n+\t\tgit_config_get_bool(hookcmd_name.buf, &skip);\n \n \t\t/* Check if a hookcmd with that name exists. */\n+\t\tstrbuf_reset(&hookcmd_name);\n \t\tstrbuf_addf(&hookcmd_name, \"hookcmd.%s.command\", command);\n \t\tgit_config_get_value(hookcmd_name.buf, &command);\n \n@@ -89,7 +100,13 @@ static int hook_config_lookup(const char *key, const char *value, void *cb_data)\n \t\t * for each key+value, do_callback(key, value, cb_data)\n \t\t */\n \n-\t\tappend_or_move_hook(head, command);\n+\t\tif (skip) {\n+\t\t\tstruct hook *to_remove = find_hook_by_command(head, command);\n+\t\t\tif (to_remove)\n+\t\t\t\tremove_hook(&(to_remove->list));\n+\t\t} else {\n+\t\t\tappend_or_move_hook(head, command);\n+\t\t}\n \n \t\tstrbuf_release(&hookcmd_name);\n \t}\ndiff --git a/t/t1360-config-based-hooks.sh b/t/t1360-config-based-hooks.sh\nindex 91127a50a4..ebd3bc623f 100755\n--- a/t/t1360-config-based-hooks.sh\n+++ b/t/t1360-config-based-hooks.sh\n@@ -132,6 +132,29 @@ test_expect_success 'hook.runHookDir = warn is respected by list' '\n \ttest_i18ncmp expected actual\n '\n \n+test_expect_success 'git hook list removes skipped hookcmd' '\n+\tsetup_hookcmd &&\n+\ttest_config hookcmd.abc.skip \"true\" --add &&\n+\n+\tcat >expected <<-EOF &&\n+\tno commands configured for hook '\\''pre-commit'\\''\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\ttest_i18ncmp expected actual\n+'\n+\n+test_expect_success 'git hook list removes skipped inlined hook' '\n+\tsetup_hooks &&\n+\ttest_config hookcmd.\"$ROOT/path/ghi\".skip \"true\" --add &&\n+\n+\tcat >expected <<-EOF &&\n+\tglobal: $ROOT/path/def\n+\tEOF\n+\n+\tgit hook list pre-commit >actual &&\n+\ttest_cmp expected actual\n+'\n \n test_expect_success 'hook.runHookDir = interactive is respected by list' '\n \tsetup_hookdir &&\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412799","messageId":"20201222000220.1491091-9-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 08/17] hook: add 'run' subcommand","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:11Z","receivedAt":"2020-12-22T00:03:49Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"In order to enable hooks to be run as an external process, by a\nstandalone Git command, or by tools which wrap Git, provide an external\nmeans to run all configured hook commands for a given hook event.\n\nFor now, the hook commands will run in config order, in series. As\nalternate ordering or parallelism is supported in the future, we should\nadd knobs to use those to the command line as well.\n\nAs with the legacy hook implementation, all stdout generated by hook\ncommands is redirected to stderr. Piping from stdin is not yet\nsupported.\n\nLegacy hooks (those present in $GITDIR/hooks) are run at the end of the\nexecution list. For now, there is no way to disable them.\n\nUsers may wish to provide hook commands like 'git config\nhook.pre-commit.command \"~/linter.sh --pre-commit\"'. To enable this,\nconfig-defined hooks are run in a shell. (Since hooks in $GITDIR/hooks\ncan't be specified with included arguments or paths which need expansion\nlike this, they are run without a shell instead.)\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Since v4, updated the docs, and did less local application of single\n quotes. In order for hookdir hooks to run successfully with a space in\n the path, though, they must not be run with 'sh -c'. So we can treat the\n hookdir hooks specially, and warn users via doc about special\n considerations for configured hooks with spaces in their path.\n\n Documentation/git-hook.txt | 31 +++++++++-\n builtin/hook.c | 48 ++++++++++++++-\n hook.c | 112 ++++++++++++++++++++++++++++++++++\n hook.h | 32 ++++++++++\n t/t1360-config-based-hooks.sh | 65 +++++++++++++++++++-\n 5 files changed, 281 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-hook.txt b/Documentation/git-hook.txt\nindex f19875ed68..18a817d832 100644\n--- a/Documentation/git-hook.txt\n+++ b/Documentation/git-hook.txt\n@@ -9,11 +9,12 @@ SYNOPSIS\n --------\n [verse]\n 'git hook' list \n+'git hook' run [(-e|--env)=...] [(-a|--arg)=...] \n \n DESCRIPTION\n -----------\n-You can list configured hooks with this command. Later, you will be able to run,\n-add, and modify hooks with this command.\n+You can list and run configured hooks with this command. Later, you will be able\n+to add and modify hooks with this command.\n \n This command parses the default configuration files for sections `hook` and\n `hookcmd`. `hook` is used to describe the commands which will be run during a\n@@ -64,6 +65,32 @@ in the order they should be run, and print the config scope where the relevant\n `hook..command` was specified, not the `hookcmd` (if applicable).\n This output is human-readable and the format is subject to change over time.\n \n+run [(-e|--env)=...] [(-a|--arg)=...] ``::\n+\n+Runs hooks configured for ``, in the same order displayed by `git\n+hook list`. Hooks configured this way are run prepended with `sh -c`, so paths\n+containing special characters or spaces should be wrapped in single quotes:\n+`command = '/my/path with spaces/script.sh' some args`.\n+\n+OPTIONS\n+-------\n+--run-hookdir::\n+\tOverrides the hook.runHookDir config. Must be 'yes', 'warn',\n+\t'interactive', or 'no'. Specifies how to handle hooks located in the Git\n+\thook directory (core.hooksPath).\n+\n+-a::\n+--arg::\n+\tOnly valid for `run`.\n++\n+Specify arguments to pass to every hook that is run.\n+\n+-e::\n+--env::\n+\tOnly valid for `run`.\n++\n+Specify environment variables to set for every hook that is run.\n+\n CONFIGURATION\n -------------\n include::config/hook.txt[]\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex d087e6f5b0..07ba00e07a 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -5,9 +5,11 @@\n #include \"hook.h\"\n #include \"parse-options.h\"\n #include \"strbuf.h\"\n+#include \"strvec.h\"\n \n static const char * const builtin_hook_usage[] = {\n \tN_(\"git hook list \"),\n+\tN_(\"git hook run [(-e|--env)=...] [(-a|--arg)=...] \"),\n \tNULL\n };\n \n@@ -84,6 +86,46 @@ static int list(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n+static int run(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct strbuf hookname = STRBUF_INIT;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tint rc = 0;\n+\n+\tstruct option run_options[] = {\n+\t\tOPT_STRVEC('e', \"env\", &opt.env, N_(\"var\"),\n+\t\t\t N_(\"environment variables for hook to use\")),\n+\t\tOPT_STRVEC('a', \"arg\", &opt.args, N_(\"args\"),\n+\t\t\t N_(\"argument to pass to hook\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\t/*\n+\t * While it makes sense to list hooks out-of-repo, it doesn't make sense\n+\t * to execute them. Hooks usually want to look at repository artifacts.\n+\t */\n+\tif (!have_git_dir())\n+\t\tusage_msg_opt(_(\"You must be in a Git repo to execute hooks.\"),\n+\t\t\t builtin_hook_usage, run_options);\n+\n+\targc = parse_options(argc, argv, prefix, run_options,\n+\t\t\t builtin_hook_usage, 0);\n+\n+\tif (argc < 1)\n+\t\tusage_msg_opt(_(\"You must specify a hook event to run.\"),\n+\t\t\t builtin_hook_usage, run_options);\n+\n+\tstrbuf_addstr(&hookname, argv[0]);\n+\topt.run_hookdir = should_run_hookdir;\n+\n+\trc = run_hooks(hookname.buf, &opt);\n+\n+\tstrbuf_release(&hookname);\n+\trun_hooks_opt_clear(&opt);\n+\n+\treturn rc;\n+}\n+\n int cmd_hook(int argc, const char **argv, const char *prefix)\n {\n \tconst char *run_hookdir = NULL;\n@@ -95,10 +137,10 @@ int cmd_hook(int argc, const char **argv, const char *prefix)\n \t};\n \n \targc = parse_options(argc, argv, prefix, builtin_hook_options,\n-\t\t\t builtin_hook_usage, 0);\n+\t\t\t builtin_hook_usage, PARSE_OPT_KEEP_UNKNOWN);\n \n \t/* after the parse, we should have \" \" */\n-\tif (argc < 1)\n+\tif (argc < 2)\n \t\tusage_with_options(builtin_hook_usage, builtin_hook_options);\n \n \tgit_config(git_default_config, NULL);\n@@ -122,6 +164,8 @@ int cmd_hook(int argc, const char **argv, const char *prefix)\n \n \tif (!strcmp(argv[0], \"list\"))\n \t\treturn list(argc, argv, prefix);\n+\tif (!strcmp(argv[0], \"run\"))\n+\t\treturn run(argc, argv, prefix);\n \n \tusage_with_options(builtin_hook_usage, builtin_hook_options);\n }\ndiff --git a/hook.c b/hook.c\nindex d262503725..5836bbb739 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -3,6 +3,7 @@\n #include \"hook.h\"\n #include \"config.h\"\n #include \"run-command.h\"\n+#include \"prompt.h\"\n \n void free_hook(struct hook *ptr)\n {\n@@ -135,6 +136,56 @@ enum hookdir_opt configured_hookdir_opt(void)\n \treturn HOOKDIR_UNKNOWN;\n }\n \n+static int should_include_hookdir(const char *path, enum hookdir_opt cfg)\n+{\n+\tstruct strbuf prompt = STRBUF_INIT;\n+\t/*\n+\t * If the path doesn't exist, don't bother adding the empty hook and\n+\t * don't bother checking the config or prompting the user.\n+\t */\n+\tif (!path)\n+\t\treturn 0;\n+\n+\tswitch (cfg)\n+\t{\n+\t\tcase HOOKDIR_NO:\n+\t\t\treturn 0;\n+\t\tcase HOOKDIR_UNKNOWN:\n+\t\t\tfprintf(stderr,\n+\t\t\t\t_(\"Unrecognized value for 'hook.runHookDir'. \"\n+\t\t\t\t \"Is there a typo? \"));\n+\t\t\t/* FALLTHROUGH */\n+\t\tcase HOOKDIR_WARN:\n+\t\t\tfprintf(stderr, _(\"Running legacy hook at '%s'\\n\"),\n+\t\t\t\tpath);\n+\t\t\treturn 1;\n+\t\tcase HOOKDIR_INTERACTIVE:\n+\t\t\tdo {\n+\t\t\t\t/*\n+\t\t\t\t * TRANSLATORS: Make sure to include [Y] and [n]\n+\t\t\t\t * in your translation. Only English input is\n+\t\t\t\t * accepted. Default option is \"yes\".\n+\t\t\t\t */\n+\t\t\t\tfprintf(stderr, _(\"Run '%s'? [Yn] \"), path);\n+\t\t\t\tgit_read_line_interactively(&prompt);\n+\t\t\t\tstrbuf_tolower(&prompt);\n+\t\t\t\tif (starts_with(prompt.buf, \"n\")) {\n+\t\t\t\t\tstrbuf_release(&prompt);\n+\t\t\t\t\treturn 0;\n+\t\t\t\t} else if (starts_with(prompt.buf, \"y\")) {\n+\t\t\t\t\tstrbuf_release(&prompt);\n+\t\t\t\t\treturn 1;\n+\t\t\t\t}\n+\t\t\t\t/* otherwise, we didn't understand the input */\n+\t\t\t} while (prompt.len); /* an empty reply means \"Yes\" */\n+\t\t\tstrbuf_release(&prompt);\n+\t\t\treturn 1;\n+\t\tcase HOOKDIR_YES:\n+\t\tdefault:\n+\t\t\treturn 1;\n+\t}\n+}\n+\n struct list_head* hook_list(const struct strbuf* hookname)\n {\n \tstruct strbuf hook_key = STRBUF_INIT;\n@@ -166,3 +217,64 @@ struct list_head* hook_list(const struct strbuf* hookname)\n \tstrbuf_release(&hook_key);\n \treturn hook_head;\n }\n+\n+void run_hooks_opt_init(struct run_hooks_opt *o)\n+{\n+\tstrvec_init(&o->env);\n+\tstrvec_init(&o->args);\n+\to->run_hookdir = configured_hookdir_opt();\n+}\n+\n+void run_hooks_opt_clear(struct run_hooks_opt *o)\n+{\n+\tstrvec_clear(&o->env);\n+\tstrvec_clear(&o->args);\n+}\n+\n+int run_hooks(const char *hookname, struct run_hooks_opt *options)\n+{\n+\tstruct strbuf hookname_str = STRBUF_INIT;\n+\tstruct list_head *to_run, *pos = NULL, *tmp = NULL;\n+\tint rc = 0;\n+\n+\tif (!options)\n+\t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n+\n+\tstrbuf_addstr(&hookname_str, hookname);\n+\n+\tto_run = hook_list(&hookname_str);\n+\n+\tlist_for_each_safe(pos, tmp, to_run) {\n+\t\tstruct child_process hook_proc = CHILD_PROCESS_INIT;\n+\t\tstruct hook *hook = list_entry(pos, struct hook, list);\n+\n+\t\thook_proc.env = options->env.v;\n+\t\thook_proc.no_stdin = 1;\n+\t\thook_proc.stdout_to_stderr = 1;\n+\t\thook_proc.trace2_hook_name = hook->command.buf;\n+\t\thook_proc.use_shell = 1;\n+\n+\t\tif (hook->from_hookdir) {\n+\t\t if (!should_include_hookdir(hook->command.buf, options->run_hookdir))\n+\t\t\tcontinue;\n+\t\t /*\n+\t\t * Commands from the config could be oneliners, but we know\n+\t\t * for certain that hookdir commands are not.\n+\t\t */\n+\t\t hook_proc.use_shell = 0;\n+\t\t}\n+\n+\t\t/* add command */\n+\t\tstrvec_push(&hook_proc.args, hook->command.buf);\n+\n+\t\t/*\n+\t\t * add passed-in argv, without expanding - let the user get back\n+\t\t * exactly what they put in\n+\t\t */\n+\t\tstrvec_pushv(&hook_proc.args, options->args.v);\n+\n+\t\trc |= run_command(&hook_proc);\n+\t}\n+\n+\treturn rc;\n+}\ndiff --git a/hook.h b/hook.h\nindex ccdf6272f2..259662968f 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -1,6 +1,7 @@\n #include \"config.h\"\n #include \"list.h\"\n #include \"strbuf.h\"\n+#include \"strvec.h\"\n \n struct hook\n {\n@@ -36,6 +37,37 @@ enum hookdir_opt\n */\n enum hookdir_opt configured_hookdir_opt(void);\n \n+struct run_hooks_opt\n+{\n+\t/* Environment vars to be set for each hook */\n+\tstruct strvec env;\n+\n+\t/* Args to be passed to each hook */\n+\tstruct strvec args;\n+\n+\t/*\n+\t * How should the hookdir be handled?\n+\t * Leave the RUN_HOOKS_OPT_INIT default in most cases; this only needs\n+\t * to be overridden if the user can override it at the command line.\n+\t */\n+\tenum hookdir_opt run_hookdir;\n+};\n+\n+#define RUN_HOOKS_OPT_INIT { \t\t\\\n+\t.env = STRVEC_INIT, \t\t\t\t\\\n+\t.args = STRVEC_INIT, \t\t\t\\\n+\t.run_hookdir = configured_hookdir_opt()\t\\\n+}\n+\n+void run_hooks_opt_init(struct run_hooks_opt *o);\n+void run_hooks_opt_clear(struct run_hooks_opt *o);\n+\n+/*\n+ * Runs all hooks associated to the 'hookname' event in order. Each hook will be\n+ * passed 'env' and 'args'.\n+ */\n+int run_hooks(const char *hookname, struct run_hooks_opt *options);\n+\n /* Free memory associated with a 'struct hook' */\n void free_hook(struct hook *ptr);\n /* Empties the list at 'head', calling 'free_hook()' on each entry */\ndiff --git a/t/t1360-config-based-hooks.sh b/t/t1360-config-based-hooks.sh\nindex ebd3bc623f..5b3003d59b 100755\n--- a/t/t1360-config-based-hooks.sh\n+++ b/t/t1360-config-based-hooks.sh\n@@ -115,7 +115,10 @@ test_expect_success 'hook.runHookDir = no is respected by list' '\n \n \tgit hook list pre-commit >actual &&\n \t# the hookdir annotation is translated\n-\ttest_i18ncmp expected actual\n+\ttest_i18ncmp expected actual &&\n+\n+\tgit hook run pre-commit 2>actual &&\n+\ttest_must_be_empty actual\n '\n \n test_expect_success 'hook.runHookDir = warn is respected by list' '\n@@ -129,6 +132,14 @@ test_expect_success 'hook.runHookDir = warn is respected by list' '\n \n \tgit hook list pre-commit >actual &&\n \t# the hookdir annotation is translated\n+\ttest_i18ncmp expected actual &&\n+\n+\tcat >expected <<-EOF &&\n+\tRunning legacy hook at '\\''$(pwd)/.git/hooks/pre-commit'\\''\n+\t\"Legacy Hook\"\n+\tEOF\n+\n+\tgit hook run pre-commit 2>actual &&\n \ttest_i18ncmp expected actual\n '\n \n@@ -156,7 +167,7 @@ test_expect_success 'git hook list removes skipped inlined hook' '\n \ttest_cmp expected actual\n '\n \n-test_expect_success 'hook.runHookDir = interactive is respected by list' '\n+test_expect_success 'hook.runHookDir = interactive is respected by list and run' '\n \tsetup_hookdir &&\n \n \ttest_config hook.runHookDir \"interactive\" &&\n@@ -167,7 +178,55 @@ test_expect_success 'hook.runHookDir = interactive is respected by list' '\n \n \tgit hook list pre-commit >actual &&\n \t# the hookdir annotation is translated\n-\ttest_i18ncmp expected actual\n+\ttest_i18ncmp expected actual &&\n+\n+\ttest_write_lines n | git hook run pre-commit 2>actual &&\n+\t! grep \"Legacy Hook\" actual &&\n+\n+\ttest_write_lines y | git hook run pre-commit 2>actual &&\n+\tgrep \"Legacy Hook\" actual\n+'\n+\n+test_expect_success 'inline hook definitions execute oneliners' '\n+\ttest_config hook.pre-commit.command \"echo \\\"Hello World\\\"\" &&\n+\n+\techo \"Hello World\" >expected &&\n+\n+\t# hooks are run with stdout_to_stderr = 1\n+\tgit hook run pre-commit 2>actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'inline hook definitions resolve paths' '\n+\twrite_script sample-hook.sh <<-EOF &&\n+\techo \\\"Sample Hook\\\"\n+\tEOF\n+\n+\ttest_when_finished \"rm sample-hook.sh\" &&\n+\n+\ttest_config hook.pre-commit.command \"\\\"$(pwd)/sample-hook.sh\\\"\" &&\n+\n+\techo \\\"Sample Hook\\\" >expected &&\n+\n+\t# hooks are run with stdout_to_stderr = 1\n+\tgit hook run pre-commit 2>actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'hookdir hook included in git hook run' '\n+\tsetup_hookdir &&\n+\n+\techo \\\"Legacy Hook\\\" >expected &&\n+\n+\t# hooks are run with stdout_to_stderr = 1\n+\tgit hook run pre-commit 2>actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'out-of-repo runs excluded' '\n+\tsetup_hooks &&\n+\n+\tnongit test_must_fail git hook run pre-commit\n '\n \n test_done\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412800","messageId":"20201222000220.1491091-8-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 07/17] parse-options: parse into strvec","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:10Z","receivedAt":"2020-12-22T00:03:50Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"parse-options already knows how to read into a string_list, and it knows\nhow to read into an strvec as a passthrough (that is, including the\nargument as well as its value). string_list and strvec serve similar\npurposes but are somewhat painful to convert between; so, let's teach\nparse-options to read values of string arguments directly into an\nstrvec without preserving the argument name.\n\nThis is useful if collecting generic arguments to pass through to\nanother command, for example, 'git hook run --arg \"--quiet\" --arg\n\"--format=pretty\" some-hook'. The resulting strvec would contain\n{ \"--quiet\", \"--format=pretty\" }.\n\nThe implementation is based on that of OPT_STRING_LIST.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Since v4, fixed one or two more places where I missed the argv_array->strvec\n rename.\n\n Documentation/technical/api-parse-options.txt | 5 +++++\n parse-options-cb.c | 16 ++++++++++++++++\n parse-options.h | 4 ++++\n 3 files changed, 25 insertions(+)\n\ndiff --git a/Documentation/technical/api-parse-options.txt b/Documentation/technical/api-parse-options.txt\nindex 5a60bbfa7f..679bd98629 100644\n--- a/Documentation/technical/api-parse-options.txt\n+++ b/Documentation/technical/api-parse-options.txt\n@@ -173,6 +173,11 @@ There are some macros to easily define options:\n \tThe string argument is stored as an element in `string_list`.\n \tUse of `--no-option` will clear the list of preceding values.\n \n+`OPT_STRVEC(short, long, &struct strvec, arg_str, description)`::\n+\tIntroduce an option with a string argument.\n+\tThe string argument is stored as an element in `strvec`.\n+\tUse of `--no-option` will clear the list of preceding values.\n+\n `OPT_INTEGER(short, long, &int_var, description)`::\n \tIntroduce an option with integer argument.\n \tThe integer is put into `int_var`.\ndiff --git a/parse-options-cb.c b/parse-options-cb.c\nindex 4542d4d3f9..c2451dfb1b 100644\n--- a/parse-options-cb.c\n+++ b/parse-options-cb.c\n@@ -207,6 +207,22 @@ int parse_opt_string_list(const struct option *opt, const char *arg, int unset)\n \treturn 0;\n }\n \n+int parse_opt_strvec(const struct option *opt, const char *arg, int unset)\n+{\n+\tstruct strvec *v = opt->value;\n+\n+\tif (unset) {\n+\t\tstrvec_clear(v);\n+\t\treturn 0;\n+\t}\n+\n+\tif (!arg)\n+\t\treturn -1;\n+\n+\tstrvec_push(v, arg);\n+\treturn 0;\n+}\n+\n int parse_opt_noop_cb(const struct option *opt, const char *arg, int unset)\n {\n \treturn 0;\ndiff --git a/parse-options.h b/parse-options.h\nindex 7030d8f3da..75cc8c7c96 100644\n--- a/parse-options.h\n+++ b/parse-options.h\n@@ -177,6 +177,9 @@ struct option {\n #define OPT_STRING_LIST(s, l, v, a, h) \\\n \t\t\t\t { OPTION_CALLBACK, (s), (l), (v), (a), \\\n \t\t\t\t (h), 0, &parse_opt_string_list }\n+#define OPT_STRVEC(s, l, v, a, h) \\\n+\t\t\t\t { OPTION_CALLBACK, (s), (l), (v), (a), \\\n+\t\t\t\t (h), 0, &parse_opt_strvec }\n #define OPT_UYN(s, l, v, h) { OPTION_CALLBACK, (s), (l), (v), NULL, \\\n \t\t\t\t (h), PARSE_OPT_NOARG, &parse_opt_tertiary }\n #define OPT_EXPIRY_DATE(s, l, v, h) \\\n@@ -296,6 +299,7 @@ int parse_opt_commits(const struct option *, const char *, int);\n int parse_opt_commit(const struct option *, const char *, int);\n int parse_opt_tertiary(const struct option *, const char *, int);\n int parse_opt_string_list(const struct option *, const char *, int);\n+int parse_opt_strvec(const struct option *, const char *, int);\n int parse_opt_noop_cb(const struct option *, const char *, int);\n enum parse_opt_result parse_opt_unknown_cb(struct parse_opt_ctx_t *ctx,\n \t\t\t\t\t const struct option *,\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412801","messageId":"20201222000220.1491091-10-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 09/17] hook: replace find_hook() with hook_exists()","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:12Z","receivedAt":"2020-12-22T00:03:51Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Add a helper to easily determine whether any hooks exist for a given\nhook event.\n\nMany callers want to check whether some state could be modified by a\nhook; that check should include the config-based hooks as well. Optimize\nby checking the config directly. Since commands which execute hooks\nmight want to take args to replace 'hook.runHookDir', let\n'hook_exists()' mirror the behavior of 'hook.runHookDir'.\n\nSigned-off-by: Emily Shaffer \n---\n builtin/bugreport.c | 4 ++--\n hook.c | 15 +++++++++++++++\n hook.h | 9 +++++++++\n 3 files changed, 26 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/bugreport.c b/builtin/bugreport.c\nindex ad3cc9c02f..2fe65d8f1e 100644\n--- a/builtin/bugreport.c\n+++ b/builtin/bugreport.c\n@@ -3,7 +3,7 @@\n #include \"strbuf.h\"\n #include \"help.h\"\n #include \"compat/compiler.h\"\n-#include \"run-command.h\"\n+#include \"hook.h\"\n \n \n static void get_system_info(struct strbuf *sys_info)\n@@ -82,7 +82,7 @@ static void get_populated_hooks(struct strbuf *hook_info, int nongit)\n \t}\n \n \tfor (i = 0; i < ARRAY_SIZE(hook); i++)\n-\t\tif (find_hook(hook[i]))\n+\t\tif (hook_exists(hook[i], configured_hookdir_opt()))\n \t\t\tstrbuf_addf(hook_info, \"%s\\n\", hook[i]);\n }\n \ndiff --git a/hook.c b/hook.c\nindex 5836bbb739..fbb69706d8 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -225,6 +225,21 @@ void run_hooks_opt_init(struct run_hooks_opt *o)\n \to->run_hookdir = configured_hookdir_opt();\n }\n \n+int hook_exists(const char *hookname, enum hookdir_opt should_run_hookdir)\n+{\n+\tconst char *value = NULL; /* throwaway */\n+\tstruct strbuf hook_key = STRBUF_INIT;\n+\n+\tint could_run_hookdir = (should_run_hookdir == HOOKDIR_INTERACTIVE ||\n+\t\t\t\tshould_run_hookdir == HOOKDIR_WARN ||\n+\t\t\t\tshould_run_hookdir == HOOKDIR_YES)\n+\t\t\t\t&& !!find_hook(hookname);\n+\n+\tstrbuf_addf(&hook_key, \"hook.%s.command\", hookname);\n+\n+\treturn (!git_config_get_value(hook_key.buf, &value)) || could_run_hookdir;\n+}\n+\n void run_hooks_opt_clear(struct run_hooks_opt *o)\n {\n \tstrvec_clear(&o->env);\ndiff --git a/hook.h b/hook.h\nindex 259662968f..762b6fadad 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -62,6 +62,15 @@ struct run_hooks_opt\n void run_hooks_opt_init(struct run_hooks_opt *o);\n void run_hooks_opt_clear(struct run_hooks_opt *o);\n \n+/*\n+ * Returns 1 if any hooks are specified in the config or if a hook exists in the\n+ * hookdir. Typically, invoke hook_exsts() like:\n+ * hook_exists(hookname, configured_hookdir_opt());\n+ * Like with run_hooks, if you take a --run-hookdir flag, reflect that\n+ * user-specified behavior here instead.\n+ */\n+int hook_exists(const char *hookname, enum hookdir_opt should_run_hookdir);\n+\n /*\n * Runs all hooks associated to the 'hookname' event in order. Each hook will be\n * passed 'env' and 'args'.\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412802","messageId":"20201222000220.1491091-11-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 10/17] hook: support passing stdin to hooks","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:13Z","receivedAt":"2020-12-22T00:03:51Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Some hooks (such as post-rewrite) need to take input via stdin.\nPreviously, callers provided stdin to hooks by setting\nrun-command.h:child_process.in, which takes a FD. Callers would open the\nfile in question themselves before calling run-command(). However, since\nwe will now need to seek to the front of the file and read it again for\nevery hook which runs, hook.h:run_command() takes a path and handles FD\nmanagement itself. Since this file is opened for read only, it should\nnot prevent later parallel execution support.\n\nOn the frontend, this is supported by asking for a file path, rather\nthan by reading stdin. Reading directly from stdin would involve caching\nthe entire stdin (to memory or to disk) and reading it back from the\nbeginning to each hook. We'd want to support cases like insufficient\nmemory or storage for the file. While this may prove useful later, for\nnow the path of least resistance is to just ask the user to make this\ninterim file themselves.\n\nSigned-off-by: Emily Shaffer \n---\n Documentation/git-hook.txt | 11 +++++++++--\n builtin/hook.c | 5 ++++-\n hook.c | 7 ++++++-\n hook.h | 9 +++++++--\n t/t1360-config-based-hooks.sh | 24 ++++++++++++++++++++++++\n 5 files changed, 50 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-hook.txt b/Documentation/git-hook.txt\nindex 18a817d832..cce30a80d0 100644\n--- a/Documentation/git-hook.txt\n+++ b/Documentation/git-hook.txt\n@@ -9,7 +9,8 @@ SYNOPSIS\n --------\n [verse]\n 'git hook' list \n-'git hook' run [(-e|--env)=...] [(-a|--arg)=...] \n+'git hook' run [(-e|--env)=...] [(-a|--arg)=...] [--to-stdin=]\n+\t\n \n DESCRIPTION\n -----------\n@@ -65,7 +66,7 @@ in the order they should be run, and print the config scope where the relevant\n `hook..command` was specified, not the `hookcmd` (if applicable).\n This output is human-readable and the format is subject to change over time.\n \n-run [(-e|--env)=...] [(-a|--arg)=...] ``::\n+run [(-e|--env)=...] [(-a|--arg)=...] [--to-stdin=] ``::\n \n Runs hooks configured for ``, in the same order displayed by `git\n hook list`. Hooks configured this way are run prepended with `sh -c`, so paths\n@@ -91,6 +92,12 @@ Specify arguments to pass to every hook that is run.\n +\n Specify environment variables to set for every hook that is run.\n \n+--to-stdin::\n+\tOnly valid for `run`.\n++\n+Specify a file which will be streamed into stdin for every hook that is run.\n+Each hook will receive the entire file from beginning to EOF.\n+\n CONFIGURATION\n -------------\n include::config/hook.txt[]\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex 07ba00e07a..be104f2938 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -9,7 +9,8 @@\n \n static const char * const builtin_hook_usage[] = {\n \tN_(\"git hook list \"),\n-\tN_(\"git hook run [(-e|--env)=...] [(-a|--arg)=...] \"),\n+\tN_(\"git hook run [(-e|--env)=...] [(-a|--arg)=...]\"\n+\t \"[--to-stdin=] \"),\n \tNULL\n };\n \n@@ -97,6 +98,8 @@ static int run(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"environment variables for hook to use\")),\n \t\tOPT_STRVEC('a', \"arg\", &opt.args, N_(\"args\"),\n \t\t\t N_(\"argument to pass to hook\")),\n+\t\tOPT_STRING(0, \"to-stdin\", &opt.path_to_stdin, N_(\"path\"),\n+\t\t\t N_(\"file to read into hooks' stdin\")),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/hook.c b/hook.c\nindex fbb69706d8..ce5c443206 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -263,8 +263,13 @@ int run_hooks(const char *hookname, struct run_hooks_opt *options)\n \t\tstruct child_process hook_proc = CHILD_PROCESS_INIT;\n \t\tstruct hook *hook = list_entry(pos, struct hook, list);\n \n+\t\t/* reopen the file for stdin; run_command closes it. */\n+\t\tif (options->path_to_stdin)\n+\t\t\thook_proc.in = xopen(options->path_to_stdin, O_RDONLY);\n+\t\telse\n+\t\t\thook_proc.no_stdin = 1;\n+\n \t\thook_proc.env = options->env.v;\n-\t\thook_proc.no_stdin = 1;\n \t\thook_proc.stdout_to_stderr = 1;\n \t\thook_proc.trace2_hook_name = hook->command.buf;\n \t\thook_proc.use_shell = 1;\ndiff --git a/hook.h b/hook.h\nindex 762b6fadad..e22a6db832 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -51,11 +51,15 @@ struct run_hooks_opt\n \t * to be overridden if the user can override it at the command line.\n \t */\n \tenum hookdir_opt run_hookdir;\n+\n+\t/* Path to file which should be piped to stdin for each hook */\n+\tconst char *path_to_stdin;\n };\n \n #define RUN_HOOKS_OPT_INIT { \t\t\\\n-\t.env = STRVEC_INIT, \t\t\t\t\\\n+\t.env = STRVEC_INIT, \t\t\t\\\n \t.args = STRVEC_INIT, \t\t\t\\\n+\t.path_to_stdin = NULL,\t\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n@@ -73,7 +77,8 @@ int hook_exists(const char *hookname, enum hookdir_opt should_run_hookdir);\n \n /*\n * Runs all hooks associated to the 'hookname' event in order. Each hook will be\n- * passed 'env' and 'args'.\n+ * passed 'env' and 'args'. The file at 'stdin_path' will be closed and reopened\n+ * for each hook that runs.\n */\n int run_hooks(const char *hookname, struct run_hooks_opt *options);\n \ndiff --git a/t/t1360-config-based-hooks.sh b/t/t1360-config-based-hooks.sh\nindex 5b3003d59b..c672269ee4 100755\n--- a/t/t1360-config-based-hooks.sh\n+++ b/t/t1360-config-based-hooks.sh\n@@ -229,4 +229,28 @@ test_expect_success 'out-of-repo runs excluded' '\n \tnongit test_must_fail git hook run pre-commit\n '\n \n+test_expect_success 'stdin to multiple hooks' '\n+\tgit config --add hook.test.command \"xargs -P1 -I% echo a%\" &&\n+\tgit config --add hook.test.command \"xargs -P1 -I% echo b%\" &&\n+\ttest_when_finished \"test_unconfig hook.test.command\" &&\n+\n+\tcat >input <<-EOF &&\n+\t1\n+\t2\n+\t3\n+\tEOF\n+\n+\tcat >expected <<-EOF &&\n+\ta1\n+\ta2\n+\ta3\n+\tb1\n+\tb2\n+\tb3\n+\tEOF\n+\n+\tgit hook run --to-stdin=input test 2>actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412803","messageId":"20201222000220.1491091-13-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 12/17] hook: allow parallel hook execution","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:15Z","receivedAt":"2020-12-22T00:03:52Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"In many cases, there's no reason not to allow hooks to execute in\nparallel. run_processes_parallel() is well-suited - it's a task queue\nthat runs its housekeeping in series, which means users don't\nneed to worry about thread safety on their callback data. True\nmultithreaded execution with the async_* functions isn't necessary here.\nSynchronous hook execution can be achieved by only allowing 1 job to run\nat a time.\n\nTeach run_hooks() to use that function for simple hooks which don't\nrequire stdin or capture of stderr.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Per AEvar's request - parallel hook execution on day zero.\n \n In most ways run_processes_parallel() worked great for me - but it didn't\n have great support for hooks where we pipe to and from. I had to add this\n support later in the series.\n \n Since I modified an existing and in-use library I'd appreciate a keen look on\n these patches.\n \n - Emily\n\n Documentation/config/hook.txt | 5 ++\n Documentation/git-hook.txt | 14 +++-\n builtin/hook.c | 6 +-\n hook.c | 142 ++++++++++++++++++++++++++--------\n hook.h | 28 ++++++-\n 5 files changed, 157 insertions(+), 38 deletions(-)\n\ndiff --git a/Documentation/config/hook.txt b/Documentation/config/hook.txt\nindex 75312754ae..a423d13781 100644\n--- a/Documentation/config/hook.txt\n+++ b/Documentation/config/hook.txt\n@@ -12,3 +12,8 @@ hook.runHookDir::\n \tControls how hooks contained in your hookdir are executed. Can be any of\n \t\"yes\", \"warn\", \"interactive\", or \"no\". Defaults to \"yes\". See\n \tlinkgit:git-hook[1] and linkgit:git-config[1] \"core.hooksPath\").\n+\n+hook.jobs::\n+\tSpecifies how many hooks can be run simultaneously during parallelized\n+\thook execution. If unspecified, defaults to the number of processors on\n+\tthe current system.\ndiff --git a/Documentation/git-hook.txt b/Documentation/git-hook.txt\nindex cce30a80d0..01cee4ad81 100644\n--- a/Documentation/git-hook.txt\n+++ b/Documentation/git-hook.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n [verse]\n 'git hook' list \n 'git hook' run [(-e|--env)=...] [(-a|--arg)=...] [--to-stdin=]\n-\t\n+\t[(-j|--jobs) ] \n \n DESCRIPTION\n -----------\n@@ -66,7 +66,7 @@ in the order they should be run, and print the config scope where the relevant\n `hook..command` was specified, not the `hookcmd` (if applicable).\n This output is human-readable and the format is subject to change over time.\n \n-run [(-e|--env)=...] [(-a|--arg)=...] [--to-stdin=] ``::\n+run [(-e|--env)=...] [(-a|--arg)=...] [--to-stdin=] [(-j|--jobs)] ``::\n \n Runs hooks configured for ``, in the same order displayed by `git\n hook list`. Hooks configured this way are run prepended with `sh -c`, so paths\n@@ -98,6 +98,16 @@ Specify environment variables to set for every hook that is run.\n Specify a file which will be streamed into stdin for every hook that is run.\n Each hook will receive the entire file from beginning to EOF.\n \n+-j::\n+--jobs::\n+\tOnly valid for `run`.\n++\n+Specify how many hooks to run simultaneously. If this flag is not specified, use\n+the value of the `hook.jobs` config. If the config is not specified, use the\n+number of CPUs on the current system. Some hooks may be ineligible for\n+parallelization: for example, 'commit-msg' intends hooks modify the commit\n+message body and cannot be parallelized.\n+\n CONFIGURATION\n -------------\n include::config/hook.txt[]\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex be104f2938..7fbc84ab64 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -10,7 +10,7 @@\n static const char * const builtin_hook_usage[] = {\n \tN_(\"git hook list \"),\n \tN_(\"git hook run [(-e|--env)=...] [(-a|--arg)=...]\"\n-\t \"[--to-stdin=] \"),\n+\t \"[--to-stdin=] [(-j|--jobs) ] \"),\n \tNULL\n };\n \n@@ -90,7 +90,7 @@ static int list(int argc, const char **argv, const char *prefix)\n static int run(int argc, const char **argv, const char *prefix)\n {\n \tstruct strbuf hookname = STRBUF_INIT;\n-\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT_ASYNC;\n \tint rc = 0;\n \n \tstruct option run_options[] = {\n@@ -100,6 +100,8 @@ static int run(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"argument to pass to hook\")),\n \t\tOPT_STRING(0, \"to-stdin\", &opt.path_to_stdin, N_(\"path\"),\n \t\t\t N_(\"file to read into hooks' stdin\")),\n+\t\tOPT_INTEGER('j', \"jobs\", &opt.jobs,\n+\t\t\t N_(\"run up to hooks simultaneously\")),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/hook.c b/hook.c\nindex ce5c443206..b190afa33b 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -136,6 +136,14 @@ enum hookdir_opt configured_hookdir_opt(void)\n \treturn HOOKDIR_UNKNOWN;\n }\n \n+int configured_hook_jobs(void)\n+{\n+\tint n = online_cpus();\n+\tgit_config_get_int(\"hook.jobs\", &n);\n+\n+\treturn n;\n+}\n+\n static int should_include_hookdir(const char *path, enum hookdir_opt cfg)\n {\n \tstruct strbuf prompt = STRBUF_INIT;\n@@ -223,6 +231,7 @@ void run_hooks_opt_init(struct run_hooks_opt *o)\n \tstrvec_init(&o->env);\n \tstrvec_init(&o->args);\n \to->run_hookdir = configured_hookdir_opt();\n+\to->jobs = configured_hook_jobs();\n }\n \n int hook_exists(const char *hookname, enum hookdir_opt should_run_hookdir)\n@@ -246,11 +255,96 @@ void run_hooks_opt_clear(struct run_hooks_opt *o)\n \tstrvec_clear(&o->args);\n }\n \n+\n+static int pick_next_hook(struct child_process *cp,\n+\t\t\t struct strbuf *out,\n+\t\t\t void *pp_cb,\n+\t\t\t void **pp_task_cb)\n+{\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\n+\tstruct hook *hook = list_entry(hook_cb->run_me, struct hook, list);\n+\n+\tif (hook_cb->head == hook_cb->run_me)\n+\t\treturn 0;\n+\n+\tcp->env = hook_cb->options->env.v;\n+\tcp->stdout_to_stderr = 1;\n+\tcp->trace2_hook_name = hook->command.buf;\n+\n+\t/* reopen the file for stdin; run_command closes it. */\n+\tif (hook_cb->options->path_to_stdin) {\n+\t\tcp->no_stdin = 0;\n+\t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n+\t} else {\n+\t\tcp->no_stdin = 1;\n+\t}\n+\n+\t/*\n+\t * Commands from the config could be oneliners, but we know\n+\t * for certain that hookdir commands are not.\n+\t */\n+\tif (hook->from_hookdir)\n+\t\tcp->use_shell = 0;\n+\telse\n+\t\tcp->use_shell = 1;\n+\n+\t/* add command */\n+\tstrvec_push(&cp->args, hook->command.buf);\n+\n+\t/*\n+\t * add passed-in argv, without expanding - let the user get back\n+\t * exactly what they put in\n+\t */\n+\tstrvec_pushv(&cp->args, hook_cb->options->args.v);\n+\n+\t/* Provide context for errors if necessary */\n+\t*pp_task_cb = hook;\n+\n+\t/* Get the next entry ready */\n+\thook_cb->run_me = hook_cb->run_me->next;\n+\n+\treturn 1;\n+}\n+\n+static int notify_start_failure(struct strbuf *out,\n+\t\t\t\tvoid *pp_cb,\n+\t\t\t\tvoid *pp_task_cp)\n+{\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct hook *attempted = pp_task_cp;\n+\n+\t/* |= rc in cb */\n+\thook_cb->rc |= 1;\n+\n+\tstrbuf_addf(out, _(\"Couldn't start '%s', configured in '%s'\\n\"),\n+\t\t attempted->command.buf,\n+\t\t attempted->from_hookdir ? \"hookdir\"\n+\t\t \t: config_scope_name(attempted->origin));\n+\n+\t/* NEEDSWORK: if halt_on_error is desired, do it here. */\n+\treturn 0;\n+}\n+\n+static int notify_hook_finished(int result,\n+\t\t\t\tstruct strbuf *out,\n+\t\t\t\tvoid *pp_cb,\n+\t\t\t\tvoid *pp_task_cb)\n+{\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\n+\t/* |= rc in cb */\n+\thook_cb->rc |= result;\n+\n+\t/* NEEDSWORK: if halt_on_error is desired, do it here. */\n+\treturn 0;\n+}\n+\n int run_hooks(const char *hookname, struct run_hooks_opt *options)\n {\n \tstruct strbuf hookname_str = STRBUF_INIT;\n \tstruct list_head *to_run, *pos = NULL, *tmp = NULL;\n-\tint rc = 0;\n+\tstruct hook_cb_data cb_data = { 0, NULL, NULL, options };\n \n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n@@ -260,41 +354,23 @@ int run_hooks(const char *hookname, struct run_hooks_opt *options)\n \tto_run = hook_list(&hookname_str);\n \n \tlist_for_each_safe(pos, tmp, to_run) {\n-\t\tstruct child_process hook_proc = CHILD_PROCESS_INIT;\n \t\tstruct hook *hook = list_entry(pos, struct hook, list);\n \n-\t\t/* reopen the file for stdin; run_command closes it. */\n-\t\tif (options->path_to_stdin)\n-\t\t\thook_proc.in = xopen(options->path_to_stdin, O_RDONLY);\n-\t\telse\n-\t\t\thook_proc.no_stdin = 1;\n-\n-\t\thook_proc.env = options->env.v;\n-\t\thook_proc.stdout_to_stderr = 1;\n-\t\thook_proc.trace2_hook_name = hook->command.buf;\n-\t\thook_proc.use_shell = 1;\n-\n-\t\tif (hook->from_hookdir) {\n-\t\t if (!should_include_hookdir(hook->command.buf, options->run_hookdir))\n-\t\t\tcontinue;\n-\t\t /*\n-\t\t * Commands from the config could be oneliners, but we know\n-\t\t * for certain that hookdir commands are not.\n-\t\t */\n-\t\t hook_proc.use_shell = 0;\n-\t\t}\n-\n-\t\t/* add command */\n-\t\tstrvec_push(&hook_proc.args, hook->command.buf);\n+\t\tif (hook->from_hookdir &&\n+\t\t !should_include_hookdir(hook->command.buf, options->run_hookdir))\n+\t\t\t list_del(pos);\n+\t}\n \n-\t\t/*\n-\t\t * add passed-in argv, without expanding - let the user get back\n-\t\t * exactly what they put in\n-\t\t */\n-\t\tstrvec_pushv(&hook_proc.args, options->args.v);\n+\tcb_data.head = to_run;\n+\tcb_data.run_me = to_run->next;\n \n-\t\trc |= run_command(&hook_proc);\n-\t}\n+\trun_processes_parallel_tr2(options->jobs,\n+\t\t\t\t pick_next_hook,\n+\t\t\t\t notify_start_failure,\n+\t\t\t\t notify_hook_finished,\n+\t\t\t\t &cb_data,\n+\t\t\t\t \"hook\",\n+\t\t\t\t hookname);\n \n-\treturn rc;\n+\treturn cb_data.rc;\n }\ndiff --git a/hook.h b/hook.h\nindex e22a6db832..0d973d090f 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -37,6 +37,9 @@ enum hookdir_opt\n */\n enum hookdir_opt configured_hookdir_opt(void);\n \n+/* Provides the number of threads to use for parallel hook execution. */\n+int configured_hook_jobs(void);\n+\n struct run_hooks_opt\n {\n \t/* Environment vars to be set for each hook */\n@@ -54,15 +57,38 @@ struct run_hooks_opt\n \n \t/* Path to file which should be piped to stdin for each hook */\n \tconst char *path_to_stdin;\n+\n+\t/* Number of threads to parallelize across */\n+\tint jobs;\n };\n \n-#define RUN_HOOKS_OPT_INIT { \t\t\\\n+/*\n+ * Callback provided to feed_pipe_fn and consume_sideband_fn.\n+ */\n+struct hook_cb_data {\n+\tint rc;\n+\tstruct list_head *head;\n+\tstruct list_head *run_me;\n+\tstruct run_hooks_opt *options;\n+};\n+\n+#define RUN_HOOKS_OPT_INIT_SYNC { \t\t\\\n \t.env = STRVEC_INIT, \t\t\t\\\n \t.args = STRVEC_INIT, \t\t\t\\\n \t.path_to_stdin = NULL,\t\t\t\\\n+\t.jobs = 1,\t\t\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n+#define RUN_HOOKS_OPT_INIT_ASYNC {\t\t\\\n+\t.env = STRVEC_INIT, \t\t\t\\\n+\t.args = STRVEC_INIT, \t\t\t\\\n+\t.path_to_stdin = NULL,\t\t\t\\\n+\t.jobs = configured_hook_jobs(),\t\t\\\n+\t.run_hookdir = configured_hookdir_opt()\t\\\n+}\n+\n+\n void run_hooks_opt_init(struct run_hooks_opt *o);\n void run_hooks_opt_clear(struct run_hooks_opt *o);\n \n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412804","messageId":"20201222000220.1491091-12-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 11/17] run-command: allow stdin for run_processes_parallel","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:14Z","receivedAt":"2020-12-22T00:03:53Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"While it makes sense not to inherit stdin from the parent process to\navoid deadlocking, it's not necessary to completely ban stdin to\nchildren. An informed user should be able to configure stdin safely. By\nsetting `some_child.process.no_stdin=1` before calling `get_next_task()`\nwe provide a reasonable default behavior but enable users to set up\nstdin streaming for themselves during the callback.\n\n`some_child.process.stdout_to_stderr`, however, remains unmodifiable by\n`get_next_task()` - the rest of the run_processes_parallel() API depends\non child output in stderr.\n\nSigned-off-by: Emily Shaffer \n---\n run-command.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex ea4d0fb4b1..80c8c97bc1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1683,6 +1683,9 @@ static int pp_start_one(struct parallel_processes *pp)\n \tif (i == pp->max_processes)\n \t\tBUG(\"bookkeeping is hard\");\n \n+\t/* disallow by default, but allow users to set up stdin if they wish */\n+\tpp->children[i].process.no_stdin = 1;\n+\n \tcode = pp->get_next_task(&pp->children[i].process,\n \t\t\t\t &pp->children[i].err,\n \t\t\t\t pp->data,\n@@ -1694,7 +1697,6 @@ static int pp_start_one(struct parallel_processes *pp)\n \t}\n \tpp->children[i].process.err = -1;\n \tpp->children[i].process.stdout_to_stderr = 1;\n-\tpp->children[i].process.no_stdin = 1;\n \n \tif (start_command(&pp->children[i].process)) {\n \t\tcode = pp->start_failure(&pp->children[i].err,\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412805","messageId":"20201222000220.1491091-15-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 14/17] run-command: add stdin callback for parallelization","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:17Z","receivedAt":"2020-12-22T00:03:55Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"If a user of the run_processes_parallel() API wants to pipe a large\namount of information to stdin of each parallel command, that\ninformation could exceed the buffer of the pipe allocated for that\nprocess's stdin. Generally this is solved by repeatedly writing to\nchild_process.in between calls to start_command() and finish_command();\nrun_processes_parallel() did not provide users an opportunity to access\nchild_process at that time.\n\nBecause the data might be extremely large (for example, a list of all\nrefs received during a push from a client) simply taking a string_list\nor strbuf is not as scalable as using a callback; the rest of the\nrun_processes_parallel() API also uses callbacks, so making this feature\nmatch the rest of the API reduces mental load on the user.\n\nSigned-off-by: Emily Shaffer \n---\n builtin/fetch.c | 1 +\n builtin/submodule--helper.c | 2 +-\n hook.c | 1 +\n run-command.c | 54 +++++++++++++++++++++++++++++++++++--\n run-command.h | 17 +++++++++++-\n submodule.c | 1 +\n t/helper/test-run-command.c | 31 ++++++++++++++++++---\n t/t0061-run-command.sh | 30 +++++++++++++++++++++\n 8 files changed, 129 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex ecf8537605..5e153b5193 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1647,6 +1647,7 @@ static int fetch_multiple(struct string_list *list, int max_children)\n \t\tresult = run_processes_parallel_tr2(max_children,\n \t\t\t\t\t\t &fetch_next_remote,\n \t\t\t\t\t\t &fetch_failed_to_start,\n+\t\t\t\t\t\t NULL,\n \t\t\t\t\t\t &fetch_finished,\n \t\t\t\t\t\t &state,\n \t\t\t\t\t\t \"fetch\", \"parallel/fetch\");\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex c30896c897..bb623c1852 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -2294,7 +2294,7 @@ static int update_submodules(struct submodule_update_clone *suc)\n \tint i;\n \n \trun_processes_parallel_tr2(suc->max_jobs, update_clone_get_next_task,\n-\t\t\t\t update_clone_start_failure,\n+\t\t\t\t update_clone_start_failure, NULL,\n \t\t\t\t update_clone_task_finished, suc, \"submodule\",\n \t\t\t\t \"parallel/update\");\n \ndiff --git a/hook.c b/hook.c\nindex eea90ec1d0..312ede1251 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -368,6 +368,7 @@ int run_hooks(const char *hookname, struct run_hooks_opt *options)\n \trun_processes_parallel_tr2(options->jobs,\n \t\t\t\t pick_next_hook,\n \t\t\t\t notify_start_failure,\n+\t\t\t\t NULL,\n \t\t\t\t notify_hook_finished,\n \t\t\t\t &cb_data,\n \t\t\t\t \"hook\",\ndiff --git a/run-command.c b/run-command.c\nindex 80c8c97bc1..7b65c087f8 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1548,6 +1548,7 @@ struct parallel_processes {\n \n \tget_next_task_fn get_next_task;\n \tstart_failure_fn start_failure;\n+\tfeed_pipe_fn feed_pipe;\n \ttask_finished_fn task_finished;\n \n \tstruct {\n@@ -1575,6 +1576,13 @@ static int default_start_failure(struct strbuf *out,\n \treturn 0;\n }\n \n+static int default_feed_pipe(struct strbuf *pipe,\n+\t\t\t void *pp_cb,\n+\t\t\t void *pp_task_cb)\n+{\n+\treturn 1;\n+}\n+\n static int default_task_finished(int result,\n \t\t\t\t struct strbuf *out,\n \t\t\t\t void *pp_cb,\n@@ -1605,6 +1613,7 @@ static void pp_init(struct parallel_processes *pp,\n \t\t int n,\n \t\t get_next_task_fn get_next_task,\n \t\t start_failure_fn start_failure,\n+\t\t feed_pipe_fn feed_pipe,\n \t\t task_finished_fn task_finished,\n \t\t void *data)\n {\n@@ -1623,6 +1632,7 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->get_next_task = get_next_task;\n \n \tpp->start_failure = start_failure ? start_failure : default_start_failure;\n+\tpp->feed_pipe = feed_pipe ? feed_pipe : default_feed_pipe;\n \tpp->task_finished = task_finished ? task_finished : default_task_finished;\n \n \tpp->nr_processes = 0;\n@@ -1715,6 +1725,37 @@ static int pp_start_one(struct parallel_processes *pp)\n \treturn 0;\n }\n \n+static void pp_buffer_stdin(struct parallel_processes *pp)\n+{\n+\tint i;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t/* Buffer stdin for each pipe. */\n+\tfor (i = 0; i < pp->max_processes; i++) {\n+\t\tif (pp->children[i].state == GIT_CP_WORKING &&\n+\t\t pp->children[i].process.in > 0) {\n+\t\t\tint done;\n+\t\t\tstrbuf_reset(&sb);\n+\t\t\tdone = pp->feed_pipe(&sb, pp->data,\n+\t\t\t\t\t pp->children[i].data);\n+\t\t\tif (sb.len) {\n+\t\t\t\tif (write_in_full(pp->children[i].process.in,\n+\t\t\t\t\t sb.buf, sb.len) < 0) {\n+\t\t\t\t\tif (errno != EPIPE)\n+\t\t\t\t\t\tdie_errno(\"write\");\n+\t\t\t\t\tdone = 1;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tif (done) {\n+\t\t\t\tclose(pp->children[i].process.in);\n+\t\t\t\tpp->children[i].process.in = 0;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&sb);\n+}\n+\n static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n {\n \tint i;\n@@ -1779,6 +1820,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \t\tpp->nr_processes--;\n \t\tpp->children[i].state = GIT_CP_FREE;\n \t\tpp->pfd[i].fd = -1;\n+\t\tpp->children[i].process.in = 0;\n \t\tchild_process_init(&pp->children[i].process);\n \n \t\tif (i != pp->output_owner) {\n@@ -1812,6 +1854,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n int run_processes_parallel(int n,\n \t\t\t get_next_task_fn get_next_task,\n \t\t\t start_failure_fn start_failure,\n+\t\t\t feed_pipe_fn feed_pipe,\n \t\t\t task_finished_fn task_finished,\n \t\t\t void *pp_cb)\n {\n@@ -1820,7 +1863,9 @@ int run_processes_parallel(int n,\n \tint spawn_cap = 4;\n \tstruct parallel_processes pp;\n \n-\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\tpp_init(&pp, n, get_next_task, start_failure, feed_pipe, task_finished, pp_cb);\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1837,6 +1882,7 @@ int run_processes_parallel(int n,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n+\t\tpp_buffer_stdin(&pp);\n \t\tpp_buffer_stderr(&pp, output_timeout);\n \t\tpp_output(&pp);\n \t\tcode = pp_collect_finished(&pp);\n@@ -1848,11 +1894,15 @@ int run_processes_parallel(int n,\n \t}\n \n \tpp_cleanup(&pp);\n+\n+\tsigchain_pop(SIGPIPE);\n+\n \treturn 0;\n }\n \n int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \t\t\t start_failure_fn start_failure,\n+\t\t\t feed_pipe_fn feed_pipe,\n \t\t\t task_finished_fn task_finished, void *pp_cb,\n \t\t\t const char *tr2_category, const char *tr2_label)\n {\n@@ -1862,7 +1912,7 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \t\t\t\t ((n < 1) ? online_cpus() : n));\n \n \tresult = run_processes_parallel(n, get_next_task, start_failure,\n-\t\t\t\t\ttask_finished, pp_cb);\n+\t\t\t\t\tfeed_pipe, task_finished, pp_cb);\n \n \ttrace2_region_leave(tr2_category, tr2_label, NULL);\n \ndiff --git a/run-command.h b/run-command.h\nindex 6472b38bde..e058c0e2c8 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -436,6 +436,20 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * This callback is called repeatedly on every child process who requests\n+ * start_command() to create a pipe by setting child_process.in < 0.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel, and\n+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n+ * The contents of 'send' will be read into the pipe and passed to the pipe.\n+ *\n+ * Return nonzero to close the pipe.\n+ */\n+typedef int (*feed_pipe_fn)(struct strbuf *pipe,\n+\t\t\t void *pp_cb,\n+\t\t\t void *pp_task_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -470,10 +484,11 @@ typedef int (*task_finished_fn)(int result,\n int run_processes_parallel(int n,\n \t\t\t get_next_task_fn,\n \t\t\t start_failure_fn,\n+\t\t\t feed_pipe_fn,\n \t\t\t task_finished_fn,\n \t\t\t void *pp_cb);\n int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n-\t\t\t task_finished_fn, void *pp_cb,\n+\t\t\t feed_pipe_fn, task_finished_fn, void *pp_cb,\n \t\t\t const char *tr2_category, const char *tr2_label);\n \n #endif\ndiff --git a/submodule.c b/submodule.c\nindex b3bb59f066..953f41818c 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1638,6 +1638,7 @@ int fetch_populated_submodules(struct repository *r,\n \trun_processes_parallel_tr2(max_parallel_jobs,\n \t\t\t\t get_next_submodule,\n \t\t\t\t fetch_start_failure,\n+\t\t\t\t NULL,\n \t\t\t\t fetch_finish,\n \t\t\t\t &spf,\n \t\t\t\t \"submodule\", \"parallel/fetch\");\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 7ae03dc712..9348184d30 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -32,8 +32,13 @@ static int parallel_next(struct child_process *cp,\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->argv);\n+\tcp->in = d->in;\n+\tcp->no_stdin = d->no_stdin;\n \tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n \tnumber_callbacks++;\n+\n+\t*task_cb = xmalloc(sizeof(int));\n+\t*(int*)(*task_cb) = 2;\n \treturn 1;\n }\n \n@@ -55,6 +60,17 @@ static int task_finished(int result,\n \treturn 1;\n }\n \n+static int test_stdin(struct strbuf *pipe, void *cb, void *task_cb)\n+{\n+\tint *lines_remaining = task_cb;\n+\n+\tif (*lines_remaining)\n+\t\tstrbuf_addf(pipe, \"sample stdin %d\\n\", --(*lines_remaining));\n+\n+\treturn !(*lines_remaining);\n+}\n+\n+\n struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n@@ -185,7 +201,7 @@ static int testsuite(int argc, const char **argv)\n \t\tsuite.tests.nr, max_jobs);\n \n \tret = run_processes_parallel(max_jobs, next_test, test_failed,\n-\t\t\t\t test_finished, &suite);\n+\t\t\t\t test_stdin, test_finished, &suite);\n \n \tif (suite.failed.nr > 0) {\n \t\tret = 1;\n@@ -413,15 +429,22 @@ int cmd__run_command(int argc, const char **argv)\n \n \tif (!strcmp(argv[1], \"run-command-parallel\"))\n \t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, NULL, &proc));\n+\t\t\t\t\t NULL, NULL, NULL, &proc));\n \n \tif (!strcmp(argv[1], \"run-command-abort\"))\n \t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, task_finished, &proc));\n+\t\t\t\t\t NULL, NULL, task_finished, &proc));\n \n \tif (!strcmp(argv[1], \"run-command-no-jobs\"))\n \t\texit(run_processes_parallel(jobs, no_job,\n-\t\t\t\t\t NULL, task_finished, &proc));\n+\t\t\t\t\t NULL, NULL, task_finished, &proc));\n+\n+\tif (!strcmp(argv[1], \"run-command-stdin\")) {\n+\t\tproc.in = -1;\n+\t\tproc.no_stdin = 0;\n+\t\texit (run_processes_parallel(jobs, parallel_next, NULL,\n+\t\t\t\t\t test_stdin, NULL, &proc));\n+\t}\n \n \tfprintf(stderr, \"check usage\\n\");\n \treturn 1;\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 7d599675e3..87759482ad 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -143,6 +143,36 @@ test_expect_success 'run_command runs in parallel with more tasks than jobs avai\n \ttest_cmp expect actual\n '\n \n+cat >expect <<-EOF\n+preloaded output of a child\n+listening for stdin:\n+sample stdin 1\n+sample stdin 0\n+preloaded output of a child\n+listening for stdin:\n+sample stdin 1\n+sample stdin 0\n+preloaded output of a child\n+listening for stdin:\n+sample stdin 1\n+sample stdin 0\n+preloaded output of a child\n+listening for stdin:\n+sample stdin 1\n+sample stdin 0\n+EOF\n+\n+test_expect_success 'run_command listens to stdin' '\n+\twrite_script stdin-script <<-\\EOF &&\n+\techo \"listening for stdin:\"\n+\twhile read line; do\n+\t\techo \"$line\"\n+\tdone\n+\tEOF\n+\ttest-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412806","messageId":"20201222000220.1491091-14-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 13/17] hook: allow specifying working directory for hooks","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:16Z","receivedAt":"2020-12-22T00:03:56Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Hooks like \"post-checkout\" require that hooks have a different working\ndirectory than the initial process. Pipe that directly through to struct\nchild_process.\n\nBecause we can just run 'git -C hook run ...' it shouldn't be\nnecessary to pipe this option through the frontend. In fact, this\nreduces the possibility of users running hooks which affect some part of\nthe filesystem outside of the repo in question.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Needed later for \"post-checkout\" conversion.\n\n hook.c | 1 +\n hook.h | 5 +++++\n 2 files changed, 6 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex b190afa33b..eea90ec1d0 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -271,6 +271,7 @@ static int pick_next_hook(struct child_process *cp,\n \tcp->env = hook_cb->options->env.v;\n \tcp->stdout_to_stderr = 1;\n \tcp->trace2_hook_name = hook->command.buf;\n+\tcp->dir = hook_cb->options->dir;\n \n \t/* reopen the file for stdin; run_command closes it. */\n \tif (hook_cb->options->path_to_stdin) {\ndiff --git a/hook.h b/hook.h\nindex 0d973d090f..8a7542610c 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -60,6 +60,9 @@ struct run_hooks_opt\n \n \t/* Number of threads to parallelize across */\n \tint jobs;\n+\n+\t/* Path to initial working directory for subprocess */\n+\tconst char *dir;\n };\n \n /*\n@@ -77,6 +80,7 @@ struct hook_cb_data {\n \t.args = STRVEC_INIT, \t\t\t\\\n \t.path_to_stdin = NULL,\t\t\t\\\n \t.jobs = 1,\t\t\t\t\\\n+\t.dir = NULL,\t\t\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n@@ -85,6 +89,7 @@ struct hook_cb_data {\n \t.args = STRVEC_INIT, \t\t\t\\\n \t.path_to_stdin = NULL,\t\t\t\\\n \t.jobs = configured_hook_jobs(),\t\t\\\n+\t.dir = NULL,\t\t\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412807","messageId":"20201222000220.1491091-16-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 15/17] hook: provide stdin by string_list or callback","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:18Z","receivedAt":"2020-12-22T00:04:02Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"In cases where a hook requires only a small amount of information via\nstdin, it should be simple for users to provide a string_list alone. But\nin more complicated cases where the stdin is too large to hold in\nmemory, let's provide a callback the users can populate line after line\nwith instead.\n\nSigned-off-by: Emily Shaffer \n---\n hook.c | 39 ++++++++++++++++++++++++++++++++++++++-\n hook.h | 25 +++++++++++++++++++++++++\n 2 files changed, 63 insertions(+), 1 deletion(-)\n\ndiff --git a/hook.c b/hook.c\nindex 312ede1251..b63a34d0a6 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -9,6 +9,7 @@ void free_hook(struct hook *ptr)\n {\n \tif (ptr) {\n \t\tstrbuf_release(&ptr->command);\n+\t\tfree(ptr->feed_pipe_cb_data);\n \t\tfree(ptr);\n \t}\n }\n@@ -38,6 +39,7 @@ static void append_or_move_hook(struct list_head *head, const char *command)\n \t\tstrbuf_init(&to_add->command, 0);\n \t\tstrbuf_addstr(&to_add->command, command);\n \t\tto_add->from_hookdir = 0;\n+\t\tto_add->feed_pipe_cb_data = NULL;\n \t}\n \n \t/* re-set the scope so we show where an override was specified */\n@@ -253,9 +255,32 @@ void run_hooks_opt_clear(struct run_hooks_opt *o)\n {\n \tstrvec_clear(&o->env);\n \tstrvec_clear(&o->args);\n+\tstring_list_clear(&o->str_stdin, 0);\n }\n \n \n+static int pipe_from_string_list(struct strbuf *pipe, void *pp_cb, void *pp_task_cb)\n+{\n+\tint *item_idx;\n+\tstruct hook *ctx = pp_task_cb;\n+\tstruct string_list *to_pipe = &((struct hook_cb_data*)pp_cb)->options->str_stdin;\n+\n+\t/* Bootstrap the state manager if necessary. */\n+\tif (!ctx->feed_pipe_cb_data) {\n+\t\tctx->feed_pipe_cb_data = xmalloc(sizeof(unsigned int));\n+\t\t*(int*)ctx->feed_pipe_cb_data = 0;\n+\t}\n+\n+\titem_idx = ctx->feed_pipe_cb_data;\n+\n+\tif (*item_idx < to_pipe->nr) {\n+\t\tstrbuf_addf(pipe, \"%s\\n\", to_pipe->items[*item_idx].string);\n+\t\t(*item_idx)++;\n+\t\treturn 0;\n+\t}\n+\treturn 1;\n+}\n+\n static int pick_next_hook(struct child_process *cp,\n \t\t\t struct strbuf *out,\n \t\t\t void *pp_cb,\n@@ -277,6 +302,10 @@ static int pick_next_hook(struct child_process *cp,\n \tif (hook_cb->options->path_to_stdin) {\n \t\tcp->no_stdin = 0;\n \t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n+\t} else if (hook_cb->options->feed_pipe) {\n+\t\t/* ask for start_command() to make a pipe for us */\n+\t\tcp->in = -1;\n+\t\tcp->no_stdin = 0;\n \t} else {\n \t\tcp->no_stdin = 1;\n \t}\n@@ -350,6 +379,14 @@ int run_hooks(const char *hookname, struct run_hooks_opt *options)\n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n \n+\tif ((options->path_to_stdin && options->str_stdin.nr) ||\n+\t (options->path_to_stdin && options->feed_pipe) ||\n+\t (options->str_stdin.nr && options->feed_pipe))\n+\t\tBUG(\"choose only one method to populate stdin\");\n+\n+\tif (options->str_stdin.nr)\n+\t\toptions->feed_pipe = &pipe_from_string_list;\n+\n \tstrbuf_addstr(&hookname_str, hookname);\n \n \tto_run = hook_list(&hookname_str);\n@@ -368,7 +405,7 @@ int run_hooks(const char *hookname, struct run_hooks_opt *options)\n \trun_processes_parallel_tr2(options->jobs,\n \t\t\t\t pick_next_hook,\n \t\t\t\t notify_start_failure,\n-\t\t\t\t NULL,\n+\t\t\t\t options->feed_pipe,\n \t\t\t\t notify_hook_finished,\n \t\t\t\t &cb_data,\n \t\t\t\t \"hook\",\ndiff --git a/hook.h b/hook.h\nindex 8a7542610c..0ac83fa7ca 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -2,6 +2,7 @@\n #include \"list.h\"\n #include \"strbuf.h\"\n #include \"strvec.h\"\n+#include \"run-command.h\"\n \n struct hook\n {\n@@ -14,6 +15,12 @@ struct hook\n \t/* The literal command to run. */\n \tstruct strbuf command;\n \tint from_hookdir;\n+\n+\t/*\n+\t * Use this to keep state for your feed_pipe_fn if you are using\n+\t * run_hooks_opt.feed_pipe. Otherwise, do not touch it.\n+\t */\n+\tvoid *feed_pipe_cb_data;\n };\n \n /*\n@@ -57,12 +64,24 @@ struct run_hooks_opt\n \n \t/* Path to file which should be piped to stdin for each hook */\n \tconst char *path_to_stdin;\n+\t/* Pipe each string to stdin, separated by newlines */\n+\tstruct string_list str_stdin;\n+\t/*\n+\t * Callback and state pointer to ask for more content to pipe to stdin.\n+\t * Will be called repeatedly, for each hook. See\n+\t * hook.c:pipe_from_stdin() for an example. Keep per-hook state in\n+\t * hook.feed_pipe_cb_data (per process). Keep initialization context in\n+\t * feed_pipe_ctx (shared by all processes).\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\tvoid *feed_pipe_ctx;\n \n \t/* Number of threads to parallelize across */\n \tint jobs;\n \n \t/* Path to initial working directory for subprocess */\n \tconst char *dir;\n+\n };\n \n /*\n@@ -81,6 +100,9 @@ struct hook_cb_data {\n \t.path_to_stdin = NULL,\t\t\t\\\n \t.jobs = 1,\t\t\t\t\\\n \t.dir = NULL,\t\t\t\t\\\n+\t.str_stdin = STRING_LIST_INIT_DUP,\t\\\n+\t.feed_pipe = NULL,\t\t\t\\\n+\t.feed_pipe_ctx = NULL,\t\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n@@ -90,6 +112,9 @@ struct hook_cb_data {\n \t.path_to_stdin = NULL,\t\t\t\\\n \t.jobs = configured_hook_jobs(),\t\t\\\n \t.dir = NULL,\t\t\t\t\\\n+\t.str_stdin = STRING_LIST_INIT_DUP,\t\\\n+\t.feed_pipe = NULL,\t\t\t\\\n+\t.feed_pipe_ctx = NULL,\t\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412808","messageId":"20201222000220.1491091-17-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 16/17] run-command: allow capturing of collated output","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:19Z","receivedAt":"2020-12-22T00:04:02Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Some callers, for example server-side hooks which wish to relay hook\noutput to clients across a transport, want to capture what would\nnormally print to stderr and do something else with it. Allow that via a\ncallback.\n\nBy calling the callback regardless of whether there's output available,\nwe allow clients to send e.g. a keepalive if necessary.\n\nBecause we expose a strbuf, not a fd or FILE*, there's no need to create\na temporary pipe or similar - we can just skip the print to stderr and\ninstead hand it to the caller.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n Originally when writing this patch I attempted to use a pipe in memory -\n but managing its lifetime was actually pretty tricky, and I found I could\n achieve the same thing with less code by doing it this way. Critique welcome,\n including \"no, you really need to do it with a pipe\".\n\n builtin/fetch.c | 2 +-\n builtin/submodule--helper.c | 2 +-\n hook.c | 1 +\n run-command.c | 33 +++++++++++++++++++++++++--------\n run-command.h | 18 +++++++++++++++++-\n submodule.c | 2 +-\n t/helper/test-run-command.c | 25 ++++++++++++++++++++-----\n t/t0061-run-command.sh | 7 +++++++\n 8 files changed, 73 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 5e153b5193..6a634085d9 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1647,7 +1647,7 @@ static int fetch_multiple(struct string_list *list, int max_children)\n \t\tresult = run_processes_parallel_tr2(max_children,\n \t\t\t\t\t\t &fetch_next_remote,\n \t\t\t\t\t\t &fetch_failed_to_start,\n-\t\t\t\t\t\t NULL,\n+\t\t\t\t\t\t NULL, NULL,\n \t\t\t\t\t\t &fetch_finished,\n \t\t\t\t\t\t &state,\n \t\t\t\t\t\t \"fetch\", \"parallel/fetch\");\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex bb623c1852..8c543d33fd 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -2294,7 +2294,7 @@ static int update_submodules(struct submodule_update_clone *suc)\n \tint i;\n \n \trun_processes_parallel_tr2(suc->max_jobs, update_clone_get_next_task,\n-\t\t\t\t update_clone_start_failure, NULL,\n+\t\t\t\t update_clone_start_failure, NULL, NULL,\n \t\t\t\t update_clone_task_finished, suc, \"submodule\",\n \t\t\t\t \"parallel/update\");\n \ndiff --git a/hook.c b/hook.c\nindex b63a34d0a6..1439322a29 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -406,6 +406,7 @@ int run_hooks(const char *hookname, struct run_hooks_opt *options)\n \t\t\t\t pick_next_hook,\n \t\t\t\t notify_start_failure,\n \t\t\t\t options->feed_pipe,\n+\t\t\t\t NULL,\n \t\t\t\t notify_hook_finished,\n \t\t\t\t &cb_data,\n \t\t\t\t \"hook\",\ndiff --git a/run-command.c b/run-command.c\nindex 7b65c087f8..0dce6bec83 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1549,6 +1549,7 @@ struct parallel_processes {\n \tget_next_task_fn get_next_task;\n \tstart_failure_fn start_failure;\n \tfeed_pipe_fn feed_pipe;\n+\tconsume_sideband_fn consume_sideband;\n \ttask_finished_fn task_finished;\n \n \tstruct {\n@@ -1614,6 +1615,7 @@ static void pp_init(struct parallel_processes *pp,\n \t\t get_next_task_fn get_next_task,\n \t\t start_failure_fn start_failure,\n \t\t feed_pipe_fn feed_pipe,\n+\t\t consume_sideband_fn consume_sideband,\n \t\t task_finished_fn task_finished,\n \t\t void *data)\n {\n@@ -1634,6 +1636,7 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->start_failure = start_failure ? start_failure : default_start_failure;\n \tpp->feed_pipe = feed_pipe ? feed_pipe : default_feed_pipe;\n \tpp->task_finished = task_finished ? task_finished : default_task_finished;\n+\tpp->consume_sideband = consume_sideband;\n \n \tpp->nr_processes = 0;\n \tpp->output_owner = 0;\n@@ -1670,7 +1673,10 @@ static void pp_cleanup(struct parallel_processes *pp)\n \t * When get_next_task added messages to the buffer in its last\n \t * iteration, the buffered output is non empty.\n \t */\n-\tstrbuf_write(&pp->buffered_output, stderr);\n+\tif (pp->consume_sideband)\n+\t\tpp->consume_sideband(&pp->buffered_output, pp->data);\n+\telse\n+\t\tstrbuf_write(&pp->buffered_output, stderr);\n \tstrbuf_release(&pp->buffered_output);\n \n \tsigchain_pop_common();\n@@ -1786,9 +1792,13 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n static void pp_output(struct parallel_processes *pp)\n {\n \tint i = pp->output_owner;\n+\n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n-\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\tif (pp->consume_sideband)\n+\t\t\tpp->consume_sideband(&pp->children[i].err, pp->data);\n+\t\telse\n+\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\tstrbuf_reset(&pp->children[i].err);\n \t}\n }\n@@ -1827,11 +1837,15 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t} else {\n-\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t/* Output errors, then all other finished child processes */\n+\t\t\tif (pp->consume_sideband) {\n+\t\t\t\tpp->consume_sideband(&pp->children[i].err, pp->data);\n+\t\t\t\tpp->consume_sideband(&pp->buffered_output, pp->data);\n+\t\t\t} else {\n+\t\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n+\t\t\t}\n \t\t\tstrbuf_reset(&pp->children[i].err);\n-\n-\t\t\t/* Output all other finished child processes */\n-\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n \t\t\tstrbuf_reset(&pp->buffered_output);\n \n \t\t\t/*\n@@ -1855,6 +1869,7 @@ int run_processes_parallel(int n,\n \t\t\t get_next_task_fn get_next_task,\n \t\t\t start_failure_fn start_failure,\n \t\t\t feed_pipe_fn feed_pipe,\n+\t\t\t consume_sideband_fn consume_sideband,\n \t\t\t task_finished_fn task_finished,\n \t\t\t void *pp_cb)\n {\n@@ -1865,7 +1880,7 @@ int run_processes_parallel(int n,\n \n \tsigchain_push(SIGPIPE, SIG_IGN);\n \n-\tpp_init(&pp, n, get_next_task, start_failure, feed_pipe, task_finished, pp_cb);\n+\tpp_init(&pp, n, get_next_task, start_failure, feed_pipe, consume_sideband, task_finished, pp_cb);\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1903,6 +1918,7 @@ int run_processes_parallel(int n,\n int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \t\t\t start_failure_fn start_failure,\n \t\t\t feed_pipe_fn feed_pipe,\n+\t\t\t consume_sideband_fn consume_sideband,\n \t\t\t task_finished_fn task_finished, void *pp_cb,\n \t\t\t const char *tr2_category, const char *tr2_label)\n {\n@@ -1912,7 +1928,8 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \t\t\t\t ((n < 1) ? online_cpus() : n));\n \n \tresult = run_processes_parallel(n, get_next_task, start_failure,\n-\t\t\t\t\tfeed_pipe, task_finished, pp_cb);\n+\t\t\t\t\tfeed_pipe, consume_sideband,\n+\t\t\t\t\ttask_finished, pp_cb);\n \n \ttrace2_region_leave(tr2_category, tr2_label, NULL);\n \ndiff --git a/run-command.h b/run-command.h\nindex e058c0e2c8..2ad8271f56 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -450,6 +450,20 @@ typedef int (*feed_pipe_fn)(struct strbuf *pipe,\n \t\t\t void *pp_cb,\n \t\t\t void *pp_task_cb);\n \n+/**\n+ * If this callback is provided, instead of collating process output to stderr,\n+ * they will be collated into a new pipe. consume_sideband_fn will be called\n+ * repeatedly. When output is available on that pipe, it will be contained in\n+ * 'output'. But it will be called with an empty 'output' too, to allow for\n+ * keepalives or similar operations if necessary.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel.\n+ *\n+ * Since this callback is provided with the collated output, no task cookie is\n+ * provided.\n+ */\n+typedef void (*consume_sideband_fn)(struct strbuf *output, void *pp_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -485,10 +499,12 @@ int run_processes_parallel(int n,\n \t\t\t get_next_task_fn,\n \t\t\t start_failure_fn,\n \t\t\t feed_pipe_fn,\n+\t\t\t consume_sideband_fn,\n \t\t\t task_finished_fn,\n \t\t\t void *pp_cb);\n int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n-\t\t\t feed_pipe_fn, task_finished_fn, void *pp_cb,\n+\t\t\t feed_pipe_fn, consume_sideband_fn,\n+\t\t\t task_finished_fn, void *pp_cb,\n \t\t\t const char *tr2_category, const char *tr2_label);\n \n #endif\ndiff --git a/submodule.c b/submodule.c\nindex 953f41818c..215bff22d9 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1638,7 +1638,7 @@ int fetch_populated_submodules(struct repository *r,\n \trun_processes_parallel_tr2(max_parallel_jobs,\n \t\t\t\t get_next_submodule,\n \t\t\t\t fetch_start_failure,\n-\t\t\t\t NULL,\n+\t\t\t\t NULL, NULL,\n \t\t\t\t fetch_finish,\n \t\t\t\t &spf,\n \t\t\t\t \"submodule\", \"parallel/fetch\");\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 9348184d30..d53db6d11c 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -51,6 +51,16 @@ static int no_job(struct child_process *cp,\n \treturn 0;\n }\n \n+static void test_consume_sideband(struct strbuf *output, void *cb)\n+{\n+\tFILE *sideband;\n+\n+\tsideband = fopen(\"./sideband\", \"a\");\n+\n+\tstrbuf_write(output, sideband);\n+\tfclose(sideband);\n+}\n+\n static int task_finished(int result,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb,\n@@ -201,7 +211,7 @@ static int testsuite(int argc, const char **argv)\n \t\tsuite.tests.nr, max_jobs);\n \n \tret = run_processes_parallel(max_jobs, next_test, test_failed,\n-\t\t\t\t test_stdin, test_finished, &suite);\n+\t\t\t\t test_stdin, NULL, test_finished, &suite);\n \n \tif (suite.failed.nr > 0) {\n \t\tret = 1;\n@@ -429,23 +439,28 @@ int cmd__run_command(int argc, const char **argv)\n \n \tif (!strcmp(argv[1], \"run-command-parallel\"))\n \t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, NULL, NULL, &proc));\n+\t\t\t\t\t NULL, NULL, NULL, NULL, &proc));\n \n \tif (!strcmp(argv[1], \"run-command-abort\"))\n \t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, NULL, task_finished, &proc));\n+\t\t\t\t\t NULL, NULL, NULL, task_finished, &proc));\n \n \tif (!strcmp(argv[1], \"run-command-no-jobs\"))\n \t\texit(run_processes_parallel(jobs, no_job,\n-\t\t\t\t\t NULL, NULL, task_finished, &proc));\n+\t\t\t\t\t NULL, NULL, NULL, task_finished, &proc));\n \n \tif (!strcmp(argv[1], \"run-command-stdin\")) {\n \t\tproc.in = -1;\n \t\tproc.no_stdin = 0;\n \t\texit (run_processes_parallel(jobs, parallel_next, NULL,\n-\t\t\t\t\t test_stdin, NULL, &proc));\n+\t\t\t\t\t test_stdin, NULL, NULL, &proc));\n \t}\n \n+\tif (!strcmp(argv[1], \"run-command-sideband\"))\n+\t\texit(run_processes_parallel(jobs, parallel_next, NULL, NULL,\n+\t\t\t\t\t test_consume_sideband, NULL,\n+\t\t\t\t\t &proc));\n+\n \tfprintf(stderr, \"check usage\\n\");\n \treturn 1;\n }\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 87759482ad..e99f6c7f44 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -143,6 +143,13 @@ test_expect_success 'run_command runs in parallel with more tasks than jobs avai\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command can divert output' '\n+\ttest_when_finished rm sideband &&\n+\ttest-tool run-command run-command-sideband 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n+\ttest_must_be_empty actual &&\n+\ttest_cmp expect sideband\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n listening for stdin:\n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412809","messageId":"20201222000220.1491091-18-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v7 17/17] hooks: allow callers to capture output","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-22T00:02:20Z","receivedAt":"2020-12-22T00:04:27Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"Some server-side hooks will require capturing output to send over\nsideband instead of printing directly to stderr. Expose that capability.\n\nSigned-off-by: Emily Shaffer \n---\n\nNotes:\n You can see this in practice in the conversions for some of the push hooks,\n like 'receive-pack'.\n\n hook.c | 2 +-\n hook.h | 10 ++++++++++\n 2 files changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/hook.c b/hook.c\nindex 1439322a29..dc241f7ec5 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -406,7 +406,7 @@ int run_hooks(const char *hookname, struct run_hooks_opt *options)\n \t\t\t\t pick_next_hook,\n \t\t\t\t notify_start_failure,\n \t\t\t\t options->feed_pipe,\n-\t\t\t\t NULL,\n+\t\t\t\t options->consume_sideband,\n \t\t\t\t notify_hook_finished,\n \t\t\t\t &cb_data,\n \t\t\t\t \"hook\",\ndiff --git a/hook.h b/hook.h\nindex 0ac83fa7ca..1e92379bb8 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -76,6 +76,14 @@ struct run_hooks_opt\n \tfeed_pipe_fn feed_pipe;\n \tvoid *feed_pipe_ctx;\n \n+\t/*\n+\t * Populate this to capture output and prevent it from being printed to\n+\t * stderr. This will be passed directly through to\n+\t * run_command:run_parallel_processes(). See t/helper/test-run-command.c\n+\t * for an example.\n+\t */\n+\tconsume_sideband_fn consume_sideband;\n+\n \t/* Number of threads to parallelize across */\n \tint jobs;\n \n@@ -103,6 +111,7 @@ struct hook_cb_data {\n \t.str_stdin = STRING_LIST_INIT_DUP,\t\\\n \t.feed_pipe = NULL,\t\t\t\\\n \t.feed_pipe_ctx = NULL,\t\t\t\\\n+\t.consume_sideband = NULL,\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n@@ -115,6 +124,7 @@ struct hook_cb_data {\n \t.str_stdin = STRING_LIST_INIT_DUP,\t\\\n \t.feed_pipe = NULL,\t\t\t\\\n \t.feed_pipe_ctx = NULL,\t\t\t\\\n+\t.consume_sideband = NULL,\t\t\\\n \t.run_hookdir = configured_hookdir_opt()\t\\\n }\n \n-- \n2.28.0.rc0.142.g3c755180ce-goog\n\n"},{"id":"412835","messageId":"xmqqft3yk2ti.fsf@gitster.c.googlers.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"Re: [PATCH v7 00/17] propose config-based hooks (part I)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-12-22T02:11:05Z","receivedAt":"2020-12-22T02:11:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Emily Shaffer writes:\n\n> Since v6:\n>\n> - Converted 'enum hookdir_opt' to UPPER_SNAKE\n> - Coccinelle fix in the hook destructor\n> - Fixed a bug where builtin/hook.c wasn't running the default git config setup\n> and therefore missed hooks in core.hooksPath when it was set. (These hooks\n> would still run except when invoked by 'git hook run' as the config was\n> called by the processes which invoked the hook library.)\n\nThanks. Queued both series (it probably is easier to think of these\nas a single 34-patch series, as long as they both are in flight at\nthe same time).\n\n"},{"id":"413082","messageId":"20201228183445.GJ3783238@google.com","threadId":"54769","inReplyTo":"xmqqft3yk2ti.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v7 00/17] propose config-based hooks (part I)","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-28T18:34:45Z","receivedAt":"2020-12-28T18:35:55Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Mon, Dec 21, 2020 at 06:11:05PM -0800, Junio C Hamano wrote:\n> \n> Emily Shaffer writes:\n> \n> > Since v6:\n> >\n> > - Converted 'enum hookdir_opt' to UPPER_SNAKE\n> > - Coccinelle fix in the hook destructor\n> > - Fixed a bug where builtin/hook.c wasn't running the default git config setup\n> > and therefore missed hooks in core.hooksPath when it was set. (These hooks\n> > would still run except when invoked by 'git hook run' as the config was\n> > called by the processes which invoked the hook library.)\n> \n> Thanks. Queued both series (it probably is easier to think of these\n> as a single 34-patch series, as long as they both are in flight at\n> the same time).\n> \n\nDo you want me to send them as a single thread for next version?\n"},{"id":"413098","messageId":"X+pesdee13Uia9PR@google.com","threadId":"54769","inReplyTo":"20201228223716.275101-1-emilyshaffer@google.com","subject":"Re: [PATCH v3 18/17] doc: make git-hook.txt point of truth","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-28T22:39:45Z","receivedAt":"2020-12-28T23:23:00Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Mon, Dec 28, 2020 at 02:37:16PM -0800, Emily Shaffer wrote:\n\nArgh. I am having awful Monday brain and this should have been\nin-reply-to the other thread. I guess that's a point in opposition of\nsplitting big topics into multiple threads. :|\n\nI'll resend it on the other topic. I'm sorry.\n\n - Emily\n"},{"id":"413099","messageId":"20201228223716.275101-1-emilyshaffer@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"[PATCH v3 18/17] doc: make git-hook.txt point of truth","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-12-28T22:37:16Z","receivedAt":"2020-12-28T23:23:02Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"By showing the list of all hooks in 'git help hook' for users to refer\nto, 'git help hook' becomes a one-stop shop for hook authorship. Since\nsome may still have muscle memory for 'git help githooks', though,\nreference the 'git hook' commands and otherwise don't remove content.\n\nSigned-off-by: Emily Shaffer \n---\n\nSorry for the wonky subject. It seemed unnecessary to send the entirety\nof the topic again when there were no changes. (I was able to push this\npatch to my fork without -f, so indeed the rest is unchanged.)\n\nI'd really prefer if 'git help githooks' opens the 'git-hook.txt' manpage,\nbut I couldn't figure out how to do that. This seemed like the next best thing\nto me - users can still find information at the old manpage, but get a little\nnudge towards the new manpage in case they didn't notice it.\n\nExtremely open to other notes about the direction of these docs; I'm not\nconfident in anything except that it'd be annoying to have 'git help\ngithooks' remain unchanged.\n\n - Emily\n\n Documentation/git-hook.txt | 5 +\n Documentation/githooks.txt | 701 +------------------------------------\n 2 files changed, 13 insertions(+), 693 deletions(-)\n\ndiff --git a/Documentation/git-hook.txt b/Documentation/git-hook.txt\nindex 01cee4ad81..cb8e383ec9 100644\n--- a/Documentation/git-hook.txt\n+++ b/Documentation/git-hook.txt\n@@ -112,6 +112,11 @@ CONFIGURATION\n -------------\n include::config/hook.txt[]\n \n+HOOKS\n+-----\n+\n+include::native-hooks.txt[]\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex c450f7a27e..4d06369924 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -8,15 +8,20 @@ githooks - Hooks used by Git\n SYNOPSIS\n --------\n $GIT_DIR/hooks/* (or \\`git config core.hooksPath`/*)\n+[verse]\n+'git hook' list \n+'git hook' run \n \n \n DESCRIPTION\n -----------\n \n-Hooks are programs you can place in a hooks directory to trigger\n-actions at certain points in git's execution. Hooks that don't have\n+Hooks are programs you can place in a hooks directory or specify in the config\n+to trigger actions at certain points in Git's execution. Hooks that don't have\n the executable bit set are ignored.\n \n+For information about specifying hooks in the config, check linkgit:git-hook[1].\n+\n By default the hooks directory is `$GIT_DIR/hooks`, but that can be\n changed via the `core.hooksPath` configuration variable (see\n linkgit:git-config[1]).\n@@ -42,697 +47,7 @@ The currently supported hooks are described below.\n HOOKS\n -----\n \n-applypatch-msg\n-~~~~~~~~~~~~~~\n-\n-This hook is invoked by linkgit:git-am[1]. It takes a single\n-parameter, the name of the file that holds the proposed commit\n-log message. Exiting with a non-zero status causes `git am` to abort\n-before applying the patch.\n-\n-The hook is allowed to edit the message file in place, and can\n-be used to normalize the message into some project standard\n-format. It can also be used to refuse the commit after inspecting\n-the message file.\n-\n-The default 'applypatch-msg' hook, when enabled, runs the\n-'commit-msg' hook, if the latter is enabled.\n-\n-Hooks run during 'applypatch-msg' will not be parallelized.\n-\n-pre-applypatch\n-~~~~~~~~~~~~~~\n-\n-This hook is invoked by linkgit:git-am[1]. It takes no parameter, and is\n-invoked after the patch is applied, but before a commit is made.\n-\n-If it exits with non-zero status, then the working tree will not be\n-committed after applying the patch.\n-\n-It can be used to inspect the current working tree and refuse to\n-make a commit if it does not pass certain test.\n-\n-The default 'pre-applypatch' hook, when enabled, runs the\n-'pre-commit' hook, if the latter is enabled.\n-\n-Hooks run during 'pre-applypatch' will be run in parallel by default.\n-\n-post-applypatch\n-~~~~~~~~~~~~~~~\n-\n-This hook is invoked by linkgit:git-am[1]. It takes no parameter,\n-and is invoked after the patch is applied and a commit is made.\n-\n-This hook is meant primarily for notification, and cannot affect\n-the outcome of `git am`.\n-\n-Hooks run during 'post-applypatch' will be run in parallel by default.\n-\n-pre-commit\n-~~~~~~~~~~\n-\n-This hook is invoked by linkgit:git-commit[1], and can be bypassed\n-with the `--no-verify` option. It takes no parameters, and is\n-invoked before obtaining the proposed commit log message and\n-making a commit. Exiting with a non-zero status from this script\n-causes the `git commit` command to abort before creating a commit.\n-\n-The default 'pre-commit' hook, when enabled, catches introduction\n-of lines with trailing whitespaces and aborts the commit when\n-such a line is found.\n-\n-All the `git commit` hooks are invoked with the environment\n-variable `GIT_EDITOR=:` if the command will not bring up an editor\n-to modify the commit message.\n-\n-The default 'pre-commit' hook, when enabled--and with the\n-`hooks.allownonascii` config option unset or set to false--prevents\n-the use of non-ASCII filenames.\n-\n-Hooks executed during 'pre-commit' will not be parallelized.\n-\n-pre-merge-commit\n-~~~~~~~~~~~~~~~~\n-\n-This hook is invoked by linkgit:git-merge[1], and can be bypassed\n-with the `--no-verify` option. It takes no parameters, and is\n-invoked after the merge has been carried out successfully and before\n-obtaining the proposed commit log message to\n-make a commit. Exiting with a non-zero status from this script\n-causes the `git merge` command to abort before creating a commit.\n-\n-The default 'pre-merge-commit' hook, when enabled, runs the\n-'pre-commit' hook, if the latter is enabled.\n-\n-This hook is invoked with the environment variable\n-`GIT_EDITOR=:` if the command will not bring up an editor\n-to modify the commit message.\n-\n-If the merge cannot be carried out automatically, the conflicts\n-need to be resolved and the result committed separately (see\n-linkgit:git-merge[1]). At that point, this hook will not be executed,\n-but the 'pre-commit' hook will, if it is enabled.\n-\n-Hooks executed during 'pre-merge-commit' will not be parallelized.\n-\n-prepare-commit-msg\n-~~~~~~~~~~~~~~~~~~\n-\n-This hook is invoked by linkgit:git-commit[1] right after preparing the\n-default log message, and before the editor is started.\n-\n-It takes one to three parameters. The first is the name of the file\n-that contains the commit log message. The second is the source of the commit\n-message, and can be: `message` (if a `-m` or `-F` option was\n-given); `template` (if a `-t` option was given or the\n-configuration option `commit.template` is set); `merge` (if the\n-commit is a merge or a `.git/MERGE_MSG` file exists); `squash`\n-(if a `.git/SQUASH_MSG` file exists); or `commit`, followed by\n-a commit SHA-1 (if a `-c`, `-C` or `--amend` option was given).\n-\n-If the exit status is non-zero, `git commit` will abort.\n-\n-The purpose of the hook is to edit the message file in place, and\n-it is not suppressed by the `--no-verify` option. A non-zero exit\n-means a failure of the hook and aborts the commit. It should not\n-be used as replacement for pre-commit hook.\n-\n-The sample `prepare-commit-msg` hook that comes with Git removes the\n-help message found in the commented portion of the commit template.\n-\n-Hooks executed during 'prepare-commit-msg' will not be parallelized.\n-\n-commit-msg\n-~~~~~~~~~~\n-\n-This hook is invoked by linkgit:git-commit[1] and linkgit:git-merge[1], and can be\n-bypassed with the `--no-verify` option. It takes a single parameter,\n-the name of the file that holds the proposed commit log message.\n-Exiting with a non-zero status causes the command to abort.\n-\n-The hook is allowed to edit the message file in place, and can be used\n-to normalize the message into some project standard format. It\n-can also be used to refuse the commit after inspecting the message\n-file.\n-\n-The default 'commit-msg' hook, when enabled, detects duplicate\n-`Signed-off-by` trailers, and aborts the commit if one is found.\n-\n-Hooks executed during 'commit-msg' will not be parallelized.\n-\n-post-commit\n-~~~~~~~~~~~\n-\n-This hook is invoked by linkgit:git-commit[1]. It takes no parameters, and is\n-invoked after a commit is made.\n-\n-This hook is meant primarily for notification, and cannot affect\n-the outcome of `git commit`.\n-\n-Hooks executed during 'post-commit' will run in parallel by default.\n-\n-pre-rebase\n-~~~~~~~~~~\n-\n-This hook is called by linkgit:git-rebase[1] and can be used to prevent a\n-branch from getting rebased. The hook may be called with one or\n-two parameters. The first parameter is the upstream from which\n-the series was forked. The second parameter is the branch being\n-rebased, and is not set when rebasing the current branch.\n-\n-Hooks executed during 'pre-rebase' will run in parallel by default.\n-\n-post-checkout\n-~~~~~~~~~~~~~\n-\n-This hook is invoked when a linkgit:git-checkout[1] or\n-linkgit:git-switch[1] is run after having updated the\n-worktree. The hook is given three parameters: the ref of the previous HEAD,\n-the ref of the new HEAD (which may or may not have changed), and a flag\n-indicating whether the checkout was a branch checkout (changing branches,\n-flag=1) or a file checkout (retrieving a file from the index, flag=0).\n-This hook cannot affect the outcome of `git switch` or `git checkout`,\n-other than that the hook's exit status becomes the exit status of\n-these two commands.\n-\n-It is also run after linkgit:git-clone[1], unless the `--no-checkout` (`-n`) option is\n-used. The first parameter given to the hook is the null-ref, the second the\n-ref of the new HEAD and the flag is always 1. Likewise for `git worktree add`\n-unless `--no-checkout` is used.\n-\n-This hook can be used to perform repository validity checks, auto-display\n-differences from the previous HEAD if different, or set working dir metadata\n-properties.\n-\n-Hooks executed during 'post-checkout' will not be parallelized.\n-\n-post-merge\n-~~~~~~~~~~\n-\n-This hook is invoked by linkgit:git-merge[1], which happens when a `git pull`\n-is done on a local repository. The hook takes a single parameter, a status\n-flag specifying whether or not the merge being done was a squash merge.\n-This hook cannot affect the outcome of `git merge` and is not executed,\n-if the merge failed due to conflicts.\n-\n-This hook can be used in conjunction with a corresponding pre-commit hook to\n-save and restore any form of metadata associated with the working tree\n-(e.g.: permissions/ownership, ACLS, etc). See contrib/hooks/setgitperms.perl\n-for an example of how to do this.\n-\n-Hooks executed during 'post-merge' will run in parallel by default.\n-\n-pre-push\n-~~~~~~~~\n-\n-This hook is called by linkgit:git-push[1] and can be used to prevent\n-a push from taking place. The hook is called with two parameters\n-which provide the name and location of the destination remote, if a\n-named remote is not being used both values will be the same.\n-\n-Information about what is to be pushed is provided on the hook's standard\n-input with lines of the form:\n-\n- SP SP SP LF\n-\n-For instance, if the command +git push origin master:foreign+ were run the\n-hook would receive a line like the following:\n-\n- refs/heads/master 67890 refs/heads/foreign 12345\n-\n-although the full, 40-character SHA-1s would be supplied. If the foreign ref\n-does not yet exist the `` will be 40 `0`. If a ref is to be\n-deleted, the `` will be supplied as `(delete)` and the `` will be 40 `0`. If the local commit was specified by something other\n-than a name which could be expanded (such as `HEAD~`, or a SHA-1) it will be\n-supplied as it was originally given.\n-\n-If this hook exits with a non-zero status, `git push` will abort without\n-pushing anything. Information about why the push is rejected may be sent\n-to the user by writing to standard error.\n-\n-Hooks executed during 'pre-push' will run in parallel by default.\n-\n-[[pre-receive]]\n-pre-receive\n-~~~~~~~~~~~\n-\n-This hook is invoked by linkgit:git-receive-pack[1] when it reacts to\n-`git push` and updates reference(s) in its repository.\n-Just before starting to update refs on the remote repository, the\n-pre-receive hook is invoked. Its exit status determines the success\n-or failure of the update.\n-\n-This hook executes once for the receive operation. It takes no\n-arguments, but for each ref to be updated it receives on standard\n-input a line of the format:\n-\n- SP SP LF\n-\n-where `` is the old object name stored in the ref,\n-`` is the new object name to be stored in the ref and\n-`` is the full name of the ref.\n-When creating a new ref, `` is 40 `0`.\n-\n-If the hook exits with non-zero status, none of the refs will be\n-updated. If the hook exits with zero, updating of individual refs can\n-still be prevented by the <> hook.\n-\n-Both standard output and standard error output are forwarded to\n-`git send-pack` on the other end, so you can simply `echo` messages\n-for the user.\n-\n-The number of push options given on the command line of\n-`git push --push-option=...` can be read from the environment\n-variable `GIT_PUSH_OPTION_COUNT`, and the options themselves are\n-found in `GIT_PUSH_OPTION_0`, `GIT_PUSH_OPTION_1`,...\n-If it is negotiated to not use the push options phase, the\n-environment variables will not be set. If the client selects\n-to use push options, but doesn't transmit any, the count variable\n-will be set to zero, `GIT_PUSH_OPTION_COUNT=0`.\n-\n-See the section on \"Quarantine Environment\" in\n-linkgit:git-receive-pack[1] for some caveats.\n-\n-Hooks executed during 'pre-receive' will not be parallelized.\n-\n-[[update]]\n-update\n-~~~~~~\n-\n-This hook is invoked by linkgit:git-receive-pack[1] when it reacts to\n-`git push` and updates reference(s) in its repository.\n-Just before updating the ref on the remote repository, the update hook\n-is invoked. Its exit status determines the success or failure of\n-the ref update.\n-\n-The hook executes once for each ref to be updated, and takes\n-three parameters:\n-\n- - the name of the ref being updated,\n- - the old object name stored in the ref,\n- - and the new object name to be stored in the ref.\n-\n-A zero exit from the update hook allows the ref to be updated.\n-Exiting with a non-zero status prevents `git receive-pack`\n-from updating that ref.\n-\n-This hook can be used to prevent 'forced' update on certain refs by\n-making sure that the object name is a commit object that is a\n-descendant of the commit object named by the old object name.\n-That is, to enforce a \"fast-forward only\" policy.\n-\n-It could also be used to log the old..new status. However, it\n-does not know the entire set of branches, so it would end up\n-firing one e-mail per ref when used naively, though. The\n-<> hook is more suited to that.\n-\n-In an environment that restricts the users' access only to git\n-commands over the wire, this hook can be used to implement access\n-control without relying on filesystem ownership and group\n-membership. See linkgit:git-shell[1] for how you might use the login\n-shell to restrict the user's access to only git commands.\n-\n-Both standard output and standard error output are forwarded to\n-`git send-pack` on the other end, so you can simply `echo` messages\n-for the user.\n-\n-The default 'update' hook, when enabled--and with\n-`hooks.allowunannotated` config option unset or set to false--prevents\n-unannotated tags to be pushed.\n-\n-Hooks executed during 'update' are run in parallel by default.\n-\n-[[proc-receive]]\n-proc-receive\n-~~~~~~~~~~~~\n-\n-This hook is invoked by linkgit:git-receive-pack[1]. If the server has\n-set the multi-valued config variable `receive.procReceiveRefs`, and the\n-commands sent to 'receive-pack' have matching reference names, these\n-commands will be executed by this hook, instead of by the internal\n-`execute_commands()` function. This hook is responsible for updating\n-the relevant references and reporting the results back to 'receive-pack'.\n-\n-This hook executes once for the receive operation. It takes no\n-arguments, but uses a pkt-line format protocol to communicate with\n-'receive-pack' to read commands, push-options and send results. In the\n-following example for the protocol, the letter 'S' stands for\n-'receive-pack' and the letter 'H' stands for this hook.\n-\n- # Version and features negotiation.\n- S: PKT-LINE(version=1\\0push-options atomic...)\n- S: flush-pkt\n- H: PKT-LINE(version=1\\0push-options...)\n- H: flush-pkt\n-\n- # Send commands from server to the hook.\n- S: PKT-LINE( )\n- S: ... ...\n- S: flush-pkt\n- # Send push-options only if the 'push-options' feature is enabled.\n- S: PKT-LINE(push-option)\n- S: ... ...\n- S: flush-pkt\n-\n- # Receive result from the hook.\n- # OK, run this command successfully.\n- H: PKT-LINE(ok )\n- # NO, I reject it.\n- H: PKT-LINE(ng )\n- # Fall through, let 'receive-pack' to execute it.\n- H: PKT-LINE(ok )\n- H: PKT-LINE(option fall-through)\n- # OK, but has an alternate reference. The alternate reference name\n- # and other status can be given in option directives.\n- H: PKT-LINE(ok )\n- H: PKT-LINE(option refname )\n- H: PKT-LINE(option old-oid )\n- H: PKT-LINE(option new-oid )\n- H: PKT-LINE(option forced-update)\n- H: ... ...\n- H: flush-pkt\n-\n-Each command for the 'proc-receive' hook may point to a pseudo-reference\n-and always has a zero-old as its old-oid, while the 'proc-receive' hook\n-may update an alternate reference and the alternate reference may exist\n-already with a non-zero old-oid. For this case, this hook will use\n-\"option\" directives to report extended attributes for the reference given\n-by the leading \"ok\" directive.\n-\n-The report of the commands of this hook should have the same order as\n-the input. The exit status of the 'proc-receive' hook only determines\n-the success or failure of the group of commands sent to it, unless\n-atomic push is in use.\n-\n-It is forbidden to specify more than one hook for 'proc-receive'. If a\n-globally-configured 'proc-receive' must be overridden, use\n-'hookcmd..skip = true' to ignore it.\n-\n-[[post-receive]]\n-post-receive\n-~~~~~~~~~~~~\n-\n-This hook is invoked by linkgit:git-receive-pack[1] when it reacts to\n-`git push` and updates reference(s) in its repository.\n-It executes on the remote repository once after all the refs have\n-been updated.\n-\n-This hook executes once for the receive operation. It takes no\n-arguments, but gets the same information as the\n-<>\n-hook does on its standard input.\n-\n-This hook does not affect the outcome of `git receive-pack`, as it\n-is called after the real work is done.\n-\n-This supersedes the <> hook in that it gets\n-both old and new values of all the refs in addition to their\n-names.\n-\n-Both standard output and standard error output are forwarded to\n-`git send-pack` on the other end, so you can simply `echo` messages\n-for the user.\n-\n-The default 'post-receive' hook is empty, but there is\n-a sample script `post-receive-email` provided in the `contrib/hooks`\n-directory in Git distribution, which implements sending commit\n-emails.\n-\n-The number of push options given on the command line of\n-`git push --push-option=...` can be read from the environment\n-variable `GIT_PUSH_OPTION_COUNT`, and the options themselves are\n-found in `GIT_PUSH_OPTION_0`, `GIT_PUSH_OPTION_1`,...\n-If it is negotiated to not use the push options phase, the\n-environment variables will not be set. If the client selects\n-to use push options, but doesn't transmit any, the count variable\n-will be set to zero, `GIT_PUSH_OPTION_COUNT=0`.\n-\n-Hooks executed during 'post-receive' are run in parallel by default.\n-\n-[[post-update]]\n-post-update\n-~~~~~~~~~~~\n-\n-This hook is invoked by linkgit:git-receive-pack[1] when it reacts to\n-`git push` and updates reference(s) in its repository.\n-It executes on the remote repository once after all the refs have\n-been updated.\n-\n-It takes a variable number of parameters, each of which is the\n-name of ref that was actually updated.\n-\n-This hook is meant primarily for notification, and cannot affect\n-the outcome of `git receive-pack`.\n-\n-The 'post-update' hook can tell what are the heads that were pushed,\n-but it does not know what their original and updated values are,\n-so it is a poor place to do log old..new. The\n-<> hook does get both original and\n-updated values of the refs. You might consider it instead if you need\n-them.\n-\n-When enabled, the default 'post-update' hook runs\n-`git update-server-info` to keep the information used by dumb\n-transports (e.g., HTTP) up to date. If you are publishing\n-a Git repository that is accessible via HTTP, you should\n-probably enable this hook.\n-\n-Both standard output and standard error output are forwarded to\n-`git send-pack` on the other end, so you can simply `echo` messages\n-for the user.\n-\n-Hooks run during 'post-update' will be run in parallel by default.\n-\n-reference-transaction\n-~~~~~~~~~~~~~~~~~~~~~\n-\n-This hook is invoked by any Git command that performs reference\n-updates. It executes whenever a reference transaction is prepared,\n-committed or aborted and may thus get called multiple times.\n-\n-The hook takes exactly one argument, which is the current state the\n-given reference transaction is in:\n-\n- - \"prepared\": All reference updates have been queued to the\n- transaction and references were locked on disk.\n-\n- - \"committed\": The reference transaction was committed and all\n- references now have their respective new value.\n-\n- - \"aborted\": The reference transaction was aborted, no changes\n- were performed and the locks have been released.\n-\n-For each reference update that was added to the transaction, the hook\n-receives on standard input a line of the format:\n-\n- SP SP LF\n-\n-The exit status of the hook is ignored for any state except for the\n-\"prepared\" state. In the \"prepared\" state, a non-zero exit status will\n-cause the transaction to be aborted. The hook will not be called with\n-\"aborted\" state in that case.\n-\n-Hooks run during 'reference-transaction' will be run in parallel by default.\n-\n-push-to-checkout\n-~~~~~~~~~~~~~~~~\n-\n-This hook is invoked by linkgit:git-receive-pack[1] when it reacts to\n-`git push` and updates reference(s) in its repository, and when\n-the push tries to update the branch that is currently checked out\n-and the `receive.denyCurrentBranch` configuration variable is set to\n-`updateInstead`. Such a push by default is refused if the working\n-tree and the index of the remote repository has any difference from\n-the currently checked out commit; when both the working tree and the\n-index match the current commit, they are updated to match the newly\n-pushed tip of the branch. This hook is to be used to override the\n-default behaviour.\n-\n-The hook receives the commit with which the tip of the current\n-branch is going to be updated. It can exit with a non-zero status\n-to refuse the push (when it does so, it must not modify the index or\n-the working tree). Or it can make any necessary changes to the\n-working tree and to the index to bring them to the desired state\n-when the tip of the current branch is updated to the new commit, and\n-exit with a zero status.\n-\n-For example, the hook can simply run `git read-tree -u -m HEAD \"$1\"`\n-in order to emulate `git fetch` that is run in the reverse direction\n-with `git push`, as the two-tree form of `git read-tree -u -m` is\n-essentially the same as `git switch` or `git checkout`\n-that switches branches while\n-keeping the local changes in the working tree that do not interfere\n-with the difference between the branches.\n-\n-Hooks executed during 'push-to-checkout' will not be parallelized.\n-\n-pre-auto-gc\n-~~~~~~~~~~~\n-\n-This hook is invoked by `git gc --auto` (see linkgit:git-gc[1]). It\n-takes no parameter, and exiting with non-zero status from this script\n-causes the `git gc --auto` to abort.\n-\n-Hooks run during 'pre-auto-gc' will be run in parallel by default.\n-\n-post-rewrite\n-~~~~~~~~~~~~\n-\n-This hook is invoked by commands that rewrite commits\n-(linkgit:git-commit[1] when called with `--amend` and\n-linkgit:git-rebase[1]; however, full-history (re)writing tools like\n-linkgit:git-fast-import[1] or\n-https://github.com/newren/git-filter-repo[git-filter-repo] typically\n-do not call it!). Its first argument denotes the command it was\n-invoked by: currently one of `amend` or `rebase`. Further\n-command-dependent arguments may be passed in the future.\n-\n-The hook receives a list of the rewritten commits on stdin, in the\n-format\n-\n- SP [ SP ] LF\n-\n-The 'extra-info' is again command-dependent. If it is empty, the\n-preceding SP is also omitted. Currently, no commands pass any\n-'extra-info'.\n-\n-The hook always runs after the automatic note copying (see\n-\"notes.rewrite.\" in linkgit:git-config[1]) has happened, and\n-thus has access to these notes.\n-\n-Hooks run during 'post-rewrite' will be run in parallel by default.\n-\n-The following command-specific comments apply:\n-\n-rebase::\n-\tFor the 'squash' and 'fixup' operation, all commits that were\n-\tsquashed are listed as being rewritten to the squashed commit.\n-\tThis means that there will be several lines sharing the same\n-\t'new-sha1'.\n-+\n-The commits are guaranteed to be listed in the order that they were\n-processed by rebase.\n-\n-sendemail-validate\n-~~~~~~~~~~~~~~~~~~\n-\n-This hook is invoked by linkgit:git-send-email[1]. It takes a single parameter,\n-the name of the file that holds the e-mail to be sent. Exiting with a\n-non-zero status causes `git send-email` to abort before sending any\n-e-mails.\n-\n-fsmonitor-watchman\n-~~~~~~~~~~~~~~~~~~\n-\n-This hook is invoked when the configuration option `core.fsmonitor` is\n-set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2`\n-depending on the version of the hook to use.\n-\n-Version 1 takes two arguments, a version (1) and the time in elapsed\n-nanoseconds since midnight, January 1, 1970.\n-\n-Version 2 takes two arguments, a version (2) and a token that is used\n-for identifying changes since the token. For watchman this would be\n-a clock id. This version must output to stdout the new token followed\n-by a NUL before the list of files.\n-\n-The hook should output to stdout the list of all files in the working\n-directory that may have changed since the requested time. The logic\n-should be inclusive so that it does not miss any potential changes.\n-The paths should be relative to the root of the working directory\n-and be separated by a single NUL.\n-\n-It is OK to include files which have not actually changed. All changes\n-including newly-created and deleted files should be included. When\n-files are renamed, both the old and the new name should be included.\n-\n-Git will limit what files it checks for changes as well as which\n-directories are checked for untracked files based on the path names\n-given.\n-\n-An optimized way to tell git \"all files have changed\" is to return\n-the filename `/`.\n-\n-The exit status determines whether git will use the data from the\n-hook to limit its search. On error, it will fall back to verifying\n-all files and folders.\n-\n-p4-changelist\n-~~~~~~~~~~~~~\n-\n-This hook is invoked by `git-p4 submit`.\n-\n-The `p4-changelist` hook is executed after the changelist\n-message has been edited by the user. It can be bypassed with the\n-`--no-verify` option. It takes a single parameter, the name\n-of the file that holds the proposed changelist text. Exiting\n-with a non-zero status causes the command to abort.\n-\n-The hook is allowed to edit the changelist file and can be used\n-to normalize the text into some project standard format. It can\n-also be used to refuse the Submit after inspect the message file.\n-\n-Run `git-p4 submit --help` for details.\n-\n-p4-prepare-changelist\n-~~~~~~~~~~~~~~~~~~~~~\n-\n-This hook is invoked by `git-p4 submit`.\n-\n-The `p4-prepare-changelist` hook is executed right after preparing\n-the default changelist message and before the editor is started.\n-It takes one parameter, the name of the file that contains the\n-changelist text. Exiting with a non-zero status from the script\n-will abort the process.\n-\n-The purpose of the hook is to edit the message file in place,\n-and it is not supressed by the `--no-verify` option. This hook\n-is called even if `--prepare-p4-only` is set.\n-\n-Run `git-p4 submit --help` for details.\n-\n-p4-post-changelist\n-~~~~~~~~~~~~~~~~~~\n-\n-This hook is invoked by `git-p4 submit`.\n-\n-The `p4-post-changelist` hook is invoked after the submit has\n-successfully occurred in P4. It takes no parameters and is meant\n-primarily for notification and cannot affect the outcome of the\n-git p4 submit action.\n-\n-Run `git-p4 submit --help` for details.\n-\n-p4-pre-submit\n-~~~~~~~~~~~~~\n-\n-This hook is invoked by `git-p4 submit`. It takes no parameters and nothing\n-from standard input. Exiting with non-zero status from this script prevent\n-`git-p4 submit` from launching. It can be bypassed with the `--no-verify`\n-command line option. Run `git-p4 submit --help` for details.\n-\n-\n-\n-post-index-change\n-~~~~~~~~~~~~~~~~~\n-\n-This hook is invoked when the index is written in read-cache.c\n-do_write_locked_index.\n-\n-The first parameter passed to the hook is the indicator for the\n-working directory being updated. \"1\" meaning working directory\n-was updated or \"0\" when the working directory was not updated.\n-\n-The second parameter passed to the hook is the indicator for whether\n-or not the index was updated and the skip-worktree bit could have\n-changed. \"1\" meaning skip-worktree bits could have been updated\n-and \"0\" meaning they were not.\n-\n-Only one parameter should be set to \"1\" when the hook runs. The hook\n-running passing \"1\", \"1\" should not be possible.\n-\n-Hooks run during 'post-index-change' will be run in parallel by default.\n+include::native-hooks.txt[]\n \n GIT\n ---\n-- \n2.29.2.490.gc7ae633391\n\n"},{"id":"413108","messageId":"xmqqr1n9ee9p.fsf@gitster.c.googlers.com","threadId":"54769","inReplyTo":"20201228183445.GJ3783238@google.com","subject":"Re: [PATCH v7 00/17] propose config-based hooks (part I)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-12-28T22:50:42Z","receivedAt":"2020-12-28T23:23:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Emily Shaffer writes:\n\n> On Mon, Dec 21, 2020 at 06:11:05PM -0800, Junio C Hamano wrote:\n>> \n>> Emily Shaffer writes:\n>> \n>> > Since v6:\n>> >\n>> > - Converted 'enum hookdir_opt' to UPPER_SNAKE\n>> > - Coccinelle fix in the hook destructor\n>> > - Fixed a bug where builtin/hook.c wasn't running the default git config setup\n>> > and therefore missed hooks in core.hooksPath when it was set. (These hooks\n>> > would still run except when invoked by 'git hook run' as the config was\n>> > called by the processes which invoked the hook library.)\n>> \n>> Thanks. Queued both series (it probably is easier to think of these\n>> as a single 34-patch series, as long as they both are in flight at\n>> the same time).\n>> \n>\n> Do you want me to send them as a single thread for next version?\n\nUnless we deliberately focus on stabilizing the early 17 patches\ninto a shape that they won't need updating while working on the\nlater part of the series, I'd guess that your next resend would\ncontain updated versions of these 17 patches, so the only effect\nthat it has to pretend that the patches belong to two separate\nseries is to invite mistakes while queuing on my part. So either\n(1) a single thread of all patches, or (2) just the early part to\nreally make sure everybody is happy with them, so that we can\ngraduate it early even while the remainder may be going through\nrevisions, would be more preferrable than the way they have been\nstructured so far.\n\nThanks.\n"},{"id":"415060","messageId":"87mtwz7jew.fsf@evledraar.gmail.com","threadId":"54769","inReplyTo":"20201222000220.1491091-2-emilyshaffer@google.com","subject":"Re: [PATCH v7 01/17] doc: propose hooks managed by the config","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-01-23T15:38:31Z","receivedAt":"2021-01-23T15:39:31Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Dec 22 2020, Emily Shaffer wrote:\n\n> Since v4, addressed comments from Jonathan Tan about wording. However, I have\n> not addressed AEvar's comments or done a full re-review of this document.\n> I wanted to get the rest of the series out for initial review first.\n\nAs you note here and in a couple of other patch notes a lot of the\ncurrent state was based on my v4 commentary. I see we have parallel hook\nexecution by default now, woohoo!\n\nI've been keeping an eye on this series, but have been kicking the can\ndown the road on reviewing it.\n\nSkimming it now I think the state of it looks mostly good now when\nviewing the end result, I think it's mainly got one big problem, but the\ngood news is that it's relatively easy to solve :)\n\nWhich is that I think it's really hard to follow along with it because\n01/17 starts with a big design doc that's partially outdated, and\npartially saying things that aren't in or should be in either a\nuser-facing doc or commit message.\n\nAnd then individual patches (e.g. 12/17) either don't have tests\nassociated with them to test the feature they add, don't update/add\ndocs, or the docs are at the very beginning.\n\nI think we should aim to mostly or entirely get rid of\nDocumentation/technical/config-based-hooks.txt, it was more of a \"what\nabout this design?\" document in the beginning.\n\nIn a series we'd apply most or all of it should really be in end-user\ndoc (and stuff like \"Future work\" can just be noted in commit messages\nas we go along).\n\nSo long story short, I started trying to review this, but found myself\ntrying to reply to one patch and then grabbing docs from 01/17, or\n(e.g. for the parallel stuff) not having tests and starting to come up\nwith them myself.\n\nSo I thought I'd send this E-Mail instead as prodding to maybe convince\nyou to re-roll it again to make it easier to follow along in a piecemeal\nfashion.\n\nAs noted before I'm happy to help with this series if needed. I just\nthought I'd send this first given that it's been a month since the last\nsubmission, perhaps you've got some more local WIP changes by now...\n"},{"id":"415620","messageId":"YBSfrAfNCvk5LJb1@google.com","threadId":"54769","inReplyTo":"87mtwz7jew.fsf@evledraar.gmail.com","subject":"Re: [PATCH v7 01/17] doc: propose hooks managed by the config","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-01-29T23:52:12Z","receivedAt":"2021-01-29T23:53:06Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Sat, Jan 23, 2021 at 04:38:31PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> \n> \n> On Tue, Dec 22 2020, Emily Shaffer wrote:\n> \n> > Since v4, addressed comments from Jonathan Tan about wording. However, I have\n> > not addressed AEvar's comments or done a full re-review of this document.\n> > I wanted to get the rest of the series out for initial review first.\n> \n> As you note here and in a couple of other patch notes a lot of the\n> current state was based on my v4 commentary. I see we have parallel hook\n> execution by default now, woohoo!\n> \n> I've been keeping an eye on this series, but have been kicking the can\n> down the road on reviewing it.\n> \n> Skimming it now I think the state of it looks mostly good now when\n> viewing the end result, I think it's mainly got one big problem, but the\n> good news is that it's relatively easy to solve :)\n> \n> Which is that I think it's really hard to follow along with it because\n> 01/17 starts with a big design doc that's partially outdated, and\n> partially saying things that aren't in or should be in either a\n> user-facing doc or commit message.\n\nSure.\n\n> \n> And then individual patches (e.g. 12/17) either don't have tests\n> associated with them to test the feature they add, don't update/add\n> docs, or the docs are at the very beginning.\n\nThanks, sure.\n\n> \n> I think we should aim to mostly or entirely get rid of\n> Documentation/technical/config-based-hooks.txt, it was more of a \"what\n> about this design?\" document in the beginning.\n\nI'm not 100% sure that I agree - there are a couple other design docs in\nDocumentation/technical which I still refer to from time to time, e.g.\nfor sparse checkout. But I *do* agree that there's a lot of info there\nthat needs to be in end-user docs.\n\n> \n> In a series we'd apply most or all of it should really be in end-user\n> doc (and stuff like \"Future work\" can just be noted in commit messages\n> as we go along).\n> \n> So long story short, I started trying to review this, but found myself\n> trying to reply to one patch and then grabbing docs from 01/17, or\n> (e.g. for the parallel stuff) not having tests and starting to come up\n> with them myself.\n\nYeah. A related issue I could imagine, although not what you mentioned\nhere, is needing to do the same thing between part I and part II of the\nseries, as I often added some functionality late in part I and then used\nit first late in part II. I don't think this is worth reordering for,\nbut probably better notes would be handy.\n\n> \n> So I thought I'd send this E-Mail instead as prodding to maybe convince\n> you to re-roll it again to make it easier to follow along in a piecemeal\n> fashion.\n> \n> As noted before I'm happy to help with this series if needed. I just\n> thought I'd send this first given that it's been a month since the last\n> submission, perhaps you've got some more local WIP changes by now...\n\nThe biggest help for me would be review focused on nits, code style,\nmissed free(), etc. - I still haven't gotten that kind of review yet,\nand I feel the series as it is now is pretty much \"feature complete\".\n\nIn fact, and I'll mention this in reply to the cover letter in a moment,\nwe picked this series up as-is from Junio's branch in 'gitster/git' and\nhave been using it at Google for a couple of weeks now - primarily with\nusers running their legacy hooks living in hookdir, but also with a\nsubset of users encouraged to try out the config functionality.\n\nThanks for the bump.\n - Emily\n\n"},{"id":"415621","messageId":"YBShXpaDnxWKxYMf@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"Re: [PATCH v7 00/17] propose config-based hooks (part I)","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-01-29T23:59:26Z","receivedAt":"2021-01-30T00:00:59Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Mon, Dec 21, 2020 at 04:02:03PM -0800, Emily Shaffer wrote:\n> \n> Since v6:\n> \n> - Converted 'enum hookdir_opt' to UPPER_SNAKE\n> - Coccinelle fix in the hook destructor\n> - Fixed a bug where builtin/hook.c wasn't running the default git config setup\n> and therefore missed hooks in core.hooksPath when it was set. (These hooks\n> would still run except when invoked by 'git hook run' as the config was\n> called by the processes which invoked the hook library.)\n> \n> CI run: https://github.com/nasamuffin/git/actions/runs/436864964\n\nSome updates on this series...\n\nSince Jan 21 we've been running this series as picked from\ngitster/git:es/config-hooks on Googler machines, with a subset of users\nasked to try out putting their hooks into config instead of hookdir. So\nfar we haven't heard any crashes or bugs like that, although I did hear\na couple places where the user documentation is lacking. I feel\nencouraged by that, and I'm hoping to improve the documentation in the\nnext week or so, pending $DAYJOB concerns.\n\nWe also addressed some of this series in our every-other-week review\nclub (me, Jonathan Tan, Jonathan Nieder, and Josh Steadmon; although in\nthis case I tried to be quiet :) ) and so I hope there will be some\ncomments from my three teammates coming to list sometime next week.\n\nSince I feel pretty comfortable that it doesn't seem to explode\nanywhere, I'm really keen to hear nitpicky reviews and try to push to\nget this into 'next'; maybe I can barter my eyes on someone else's\nneglected review? That sounds pretty mercenary but I think Junio is the\none who suggested it a few weeks ago... ;)\n\n - Emily\n"},{"id":"415676","messageId":"20210131031011.1016240-1-jonathantanmy@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-4-emilyshaffer@google.com","subject":"Re: [PATCH v7 03/17] hook: add list command","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-01-31T03:10:11Z","receivedAt":"2021-01-31T03:17:15Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Teach 'git hook list ', which checks the known configs in\n> order to create an ordered list of hooks to run on a given hook event.\n> \n> Multiple commands can be specified for a given hook by providing\n> multiple \"hook..command = \" lines. Hooks will be\n> run in config order. If more properties need to be set on a given hook\n> in the future, commands can also be specified by providing\n> \"hook..command = \", as well as a \"[hookcmd\n> ]\" subsection; at minimum, this subsection must contain a\n> \"hookcmd..command = \" line.\n\nI learned later that this isn't true - in patch 6, the commit message\nand one of the tests therein describe being able to skip a previously\ninline command by making a hookcmd section of the same name and just\nspecifying \"skip = true\" there (without any command).\n\nMaybe just delete the \"at minimum\" part.\n\n> +static int list(int argc, const char **argv, const char *prefix)\n> {\n> -\tstruct option builtin_hook_options[] = {\n> +\tstruct list_head *head, *pos;\n> +\tstruct hook *item;\n\nYou asked for review on nits too so here's one: \"item\" should be\ndeclared in the list_for_each block. (That also makes it easier to see\nthat we don't need to free it.)\n\n> diff --git a/hook.c b/hook.c\n> new file mode 100644\n> index 0000000000..937dc768c8\n> --- /dev/null\n> +++ b/hook.c\n> @@ -0,0 +1,115 @@\n> +#include \"cache.h\"\n> +\n> +#include \"hook.h\"\n> +#include \"config.h\"\n\nUsually we put all the includes together without any intervening blank\nlines.\n\n> +static void append_or_move_hook(struct list_head *head, const char *command)\n> +{\n> +\tstruct list_head *pos = NULL, *tmp = NULL;\n> +\tstruct hook *to_add = NULL;\n> +\n> +\t/*\n> +\t * remove the prior entry with this command; we'll replace it at the\n> +\t * end.\n> +\t */\n> +\tlist_for_each_safe(pos, tmp, head) {\n> +\t\tstruct hook *it = list_entry(pos, struct hook, list);\n> +\t\tif (!strcmp(it->command.buf, command)) {\n> +\t\t list_del(pos);\n> +\t\t /* we'll simply move the hook to the end */\n> +\t\t to_add = it;\n\n\"break\" here?\n\n> +\t\t}\n> +\t}\n> +\n> +\tif (!to_add) {\n> +\t\t/* adding a new hook, not moving an old one */\n> +\t\tto_add = xmalloc(sizeof(struct hook));\n\nStyle is to write sizeof(*to_add), I think.\n\n[snip]\n\n> +struct hook_config_cb\n> +{\n> +\tstruct strbuf *hookname;\n\nstruct declarations have \"{\" not on a line on its own.\n\nAlso, \"hookname\" could just be a char *?\n\t\n> +\tstruct list_head *list;\n> +};\n> +\n> +static int hook_config_lookup(const char *key, const char *value, void *cb_data)\n> +{\n> +\tstruct hook_config_cb *data = cb_data;\n> +\tconst char *hook_key = data->hookname->buf;\n> +\tstruct list_head *head = data->list;\n> +\n> +\tif (!strcmp(key, hook_key)) {\n> +\t\tconst char *command = value;\n> +\t\tstruct strbuf hookcmd_name = STRBUF_INIT;\n> +\n> +\t\t/* Check if a hookcmd with that name exists. */\n> +\t\tstrbuf_addf(&hookcmd_name, \"hookcmd.%s.command\", command);\n> +\t\tgit_config_get_value(hookcmd_name.buf, &command);\n\nSo we don't really care whether git_config_get_value returns 0 or 1 as\nlong as it doesn't touch \"command\" if there is no such hookcmd. That\nfits with how git_config_get_value() is documented, so that's great.\nPerhaps better to document as:\n\n If a hookcmd.%s.command config exists, replace the command with the\n value of that config. (If not, do nothing - git_config_get_value() is\n documented to not overwrite the value argument in this case.)\n\n> +\n> +\t\tif (!command) {\n> +\t\t\tstrbuf_release(&hookcmd_name);\n> +\t\t\tBUG(\"git_config_get_value overwrote a string it shouldn't have\");\n> +\t\t}\n> +\n> +\t\t/*\n> +\t\t * TODO: implement an option-getting callback, e.g.\n> +\t\t * get configs by pattern hookcmd.$value.*\n> +\t\t * for each key+value, do_callback(key, value, cb_data)\n> +\t\t */\n> +\n> +\t\tappend_or_move_hook(head, command);\n> +\n> +\t\tstrbuf_release(&hookcmd_name);\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> +struct list_head* hook_list(const struct strbuf* hookname)\n\n\"const char *hookname\" should suffice?\n\nAlso, search for \"* \" and replace with \" *\" where applicable (also in\nthe .h file).\n\n> +{\n> +\tstruct strbuf hook_key = STRBUF_INIT;\n> +\tstruct list_head *hook_head = xmalloc(sizeof(struct list_head));\n> +\tstruct hook_config_cb cb_data = { &hook_key, hook_head };\n> +\n> +\tINIT_LIST_HEAD(hook_head);\n> +\n> +\tif (!hookname)\n> +\t\treturn NULL;\n> +\n> +\tstrbuf_addf(&hook_key, \"hook.%s.command\", hookname->buf);\n> +\n> +\tgit_config(hook_config_lookup, (void*)&cb_data);\n\nDo we need this void* cast?\n\n> +\n> +\tstrbuf_release(&hook_key);\n> +\treturn hook_head;\n> +}\n> diff --git a/hook.h b/hook.h\n> new file mode 100644\n> index 0000000000..8ffc4f14b6\n> --- /dev/null\n> +++ b/hook.h\n> @@ -0,0 +1,26 @@\n> +#include \"config.h\"\n> +#include \"list.h\"\n> +#include \"strbuf.h\"\n> +\n> +struct hook\n> +{\n> +\tstruct list_head list;\n> +\t/*\n> +\t * Config file which holds the hook.*.command definition.\n> +\t * (This has nothing to do with the hookcmd..* configs.)\n> +\t */\n> +\tenum config_scope origin;\n> +\t/* The literal command to run. */\n> +\tstruct strbuf command;\n\n\"char *\" would suffice?\n\nThe tests look fine.\n"},{"id":"415677","messageId":"20210131032022.1018820-1-jonathantanmy@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-5-emilyshaffer@google.com","subject":"Re: [PATCH v7 04/17] hook: include hookdir hook in list","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-01-31T03:20:22Z","receivedAt":"2021-01-31T03:28:14Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Historically, hooks are declared by placing an executable into\n> $GIT_DIR/hooks/$HOOKNAME (or $HOOKDIR/$HOOKNAME). Although hooks taken\n> from the config are more featureful than hooks placed in the $HOOKDIR,\n> those hooks should not stop working for users who already have them.\n\nMaybe explicitly add that we're listing them in the list with a \"hookdir:\"\nprefix.\n\n> Legacy hooks should be run directly, not in shell. We know that they are\n> a path to an executable, not a oneliner script - and running them\n> directly takes care of path quoting concerns for us for free.\n\nNot sure what this paragraph is doing here.\n\n> diff --git a/builtin/hook.c b/builtin/hook.c\n> index 4d36de52f8..a0013ae4d7 100644\n> --- a/builtin/hook.c\n> +++ b/builtin/hook.c\n> @@ -16,6 +16,7 @@ static int list(int argc, const char **argv, const char *prefix)\n> \tstruct list_head *head, *pos;\n> \tstruct hook *item;\n> \tstruct strbuf hookname = STRBUF_INIT;\n> +\tstruct strbuf hookdir_annotation = STRBUF_INIT;\n\nRight now this is never set? Maybe hold off on adding this until we set\nsomething.\n\n> @@ -110,6 +113,18 @@ struct list_head* hook_list(const struct strbuf* hookname)\n> \n> \tgit_config(hook_config_lookup, (void*)&cb_data);\n> \n> +\tif (have_git_dir())\n> +\t\tlegacy_hook_path = find_hook(hookname->buf);\n> +\n> +\t/* Unconditionally add legacy hook, but annotate it. */\n> +\tif (legacy_hook_path) {\n> +\t\tstruct hook *legacy_hook;\n> +\n> +\t\tappend_or_move_hook(hook_head, absolute_path(legacy_hook_path));\n\nBoth find_hook() and absolute_path() use static buffers to hold their\nreturn values, which makes me a bit nervous. Perhaps put them all under\nthe same \"if (have_git_dir())\" so that it's clearer that we're not\nsupposed to insert code arbitrarily between their invocation and their\nusage.\n\n> diff --git a/hook.h b/hook.h\n> index 8ffc4f14b6..5750634c83 100644\n> --- a/hook.h\n> +++ b/hook.h\n> @@ -12,6 +12,7 @@ struct hook\n> \tenum config_scope origin;\n> \t/* The literal command to run. */\n> \tstruct strbuf command;\n> +\tint from_hookdir;\n\nunsigned from_hookdir : 1?\n\nThe tests look good.\n"},{"id":"415678","messageId":"20210131033503.1022138-1-jonathantanmy@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-6-emilyshaffer@google.com","subject":"Re: [PATCH v7 05/17] hook: respect hook.runHookDir","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-01-31T03:35:03Z","receivedAt":"2021-01-31T03:36:03Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Include hooks specified in the hook directory in the list of hooks to\n> run. These hooks do need to be treated differently from config-specified\n> ones - they do not need to run in a shell, and later on may be disabled\n> or warned about based on a config setting.\n> \n> Because they are at least as local as the local config, we'll run them\n> last - to keep the hook execution order from global to local.\n\nThis commit message doesn't seem to match the code change. Firstly,\nwe're teaching hook.runHookDir, not respecting it (since it did not\nexist before this commit), and it's about showing it in \"list\" and not\nabout running it at all.\n\nPerhaps just \"hook: teach hook.runHookDir\" as the subject and as the\nbody:\n\n For now, this just affects the output of \"git hook list\". In the\n future, this will affect the behavior of \"git hook run\" and when Git\n runs hooks before or after its operations.\n\n> +\tswitch (should_run_hookdir) {\n> +\t\tcase HOOKDIR_NO:\n> +\t\t\tstrbuf_addstr(&hookdir_annotation, _(\" (will not run)\"));\n> +\t\t\tbreak;\n> +\t\tcase HOOKDIR_INTERACTIVE:\n> +\t\t\tstrbuf_addstr(&hookdir_annotation, _(\" (will prompt)\"));\n> +\t\t\tbreak;\n> +\t\tcase HOOKDIR_WARN:\n> +\t\tcase HOOKDIR_UNKNOWN:\n\nHmm...UNKNOWN is the same as WARN? This doesn't agree with what is said\nin patch 1:\n\n +In case this list is expanded in the future, if a value for `hook.runHookDir` is\n +given which Git does not recognize, Git should discard that config entry. For\n +example, if \"warn\" was specified at system level and \"junk\" was specified at\n +global level, Git would resolve the value to \"warn\"; if the only time the config\n +was set was to \"junk\", Git would use the default value of \"yes\".\n\nBut having said that, I would prefer if Git just errored out in this\ncase.\n\nThe rest looks good.\n"},{"id":"415680","messageId":"20210131034030.1025259-1-jonathantanmy@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-7-emilyshaffer@google.com","subject":"Re: [PATCH v7 06/17] hook: implement hookcmd..skip","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-01-31T03:40:30Z","receivedAt":"2021-01-31T03:41:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> If a user wants a specific repo to skip execution of a hook which is set\n> at a global or system level, they can now do so by specifying 'skip' in\n> their repo config:\n\nUsually the present tense describes the situation before the commit, so\nmaybe s/they can now do so/they will be able to do so/.\n\n> -static void append_or_move_hook(struct list_head *head, const char *command)\n> +static struct hook* find_hook_by_command(struct list_head *head, const char *command)\n\n\"* \" -> \" *\"\n\n[snip tests]\n\nFor the tests, I thought of the case in which we skip a hookcmd that was\nnever specified as a hook, but that's probably not very useful.\n"},{"id":"415684","messageId":"20210131042254.1032233-1-jonathantanmy@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-9-emilyshaffer@google.com","subject":"Re: [PATCH v7 08/17] hook: add 'run' subcommand","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-01-31T04:22:54Z","receivedAt":"2021-01-31T04:26:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> In order to enable hooks to be run as an external process, by a\n> standalone Git command, or by tools which wrap Git, provide an external\n> means to run all configured hook commands for a given hook event.\n> \n> For now, the hook commands will run in config order, in series. As\n> alternate ordering or parallelism is supported in the future, we should\n> add knobs to use those to the command line as well.\n> \n> As with the legacy hook implementation, all stdout generated by hook\n> commands is redirected to stderr. Piping from stdin is not yet\n> supported.\n> \n> Legacy hooks (those present in $GITDIR/hooks) are run at the end of the\n> execution list. For now, there is no way to disable them.\n\nNot true anymore now that we have hook.runHookDir :-)\n\n> @@ -64,6 +65,32 @@ in the order they should be run, and print the config scope where the relevant\n> `hook..command` was specified, not the `hookcmd` (if applicable).\n> This output is human-readable and the format is subject to change over time.\n> \n> +run [(-e|--env)=...] [(-a|--arg)=...] ``::\n> +\n> +Runs hooks configured for ``, in the same order displayed by `git\n> +hook list`. Hooks configured this way are run prepended with `sh -c`, so paths\n> +containing special characters or spaces should be wrapped in single quotes:\n> +`command = '/my/path with spaces/script.sh' some args`.\n\nI learned recently that this may not work the way I expect [1], so you\nmight want to specifically call this out for someone who knows how\nrun-command and running-with-shell works.\n\n[1] https://lore.kernel.org/git/YAs9pTBsdskC8CPN@coredump.intra.peff.net/\n\n> @@ -135,6 +136,56 @@ enum hookdir_opt configured_hookdir_opt(void)\n> \treturn HOOKDIR_UNKNOWN;\n> }\n> \n> +static int should_include_hookdir(const char *path, enum hookdir_opt cfg)\n> +{\n> +\tstruct strbuf prompt = STRBUF_INIT;\n> +\t/*\n> +\t * If the path doesn't exist, don't bother adding the empty hook and\n> +\t * don't bother checking the config or prompting the user.\n> +\t */\n> +\tif (!path)\n> +\t\treturn 0;\n> +\n> +\tswitch (cfg)\n> +\t{\n> +\t\tcase HOOKDIR_NO:\n> +\t\t\treturn 0;\n> +\t\tcase HOOKDIR_UNKNOWN:\n> +\t\t\tfprintf(stderr,\n> +\t\t\t\t_(\"Unrecognized value for 'hook.runHookDir'. \"\n> +\t\t\t\t \"Is there a typo? \"));\n> +\t\t\t/* FALLTHROUGH */\n\nSame comment (about UNKNOWN and defaulting to WARN instead of YES) as in\none of the previous patches.\n\n> +\t\tcase HOOKDIR_WARN:\n> +\t\t\tfprintf(stderr, _(\"Running legacy hook at '%s'\\n\"),\n> +\t\t\t\tpath);\n> +\t\t\treturn 1;\n> +\t\tcase HOOKDIR_INTERACTIVE:\n> +\t\t\tdo {\n> +\t\t\t\t/*\n> +\t\t\t\t * TRANSLATORS: Make sure to include [Y] and [n]\n> +\t\t\t\t * in your translation. Only English input is\n> +\t\t\t\t * accepted. Default option is \"yes\".\n> +\t\t\t\t */\n> +\t\t\t\tfprintf(stderr, _(\"Run '%s'? [Yn] \"), path);\n> +\t\t\t\tgit_read_line_interactively(&prompt);\n> +\t\t\t\tstrbuf_tolower(&prompt);\n> +\t\t\t\tif (starts_with(prompt.buf, \"n\")) {\n> +\t\t\t\t\tstrbuf_release(&prompt);\n> +\t\t\t\t\treturn 0;\n> +\t\t\t\t} else if (starts_with(prompt.buf, \"y\")) {\n> +\t\t\t\t\tstrbuf_release(&prompt);\n> +\t\t\t\t\treturn 1;\n> +\t\t\t\t}\n> +\t\t\t\t/* otherwise, we didn't understand the input */\n> +\t\t\t} while (prompt.len); /* an empty reply means \"Yes\" */\n> +\t\t\tstrbuf_release(&prompt);\n> +\t\t\treturn 1;\n> +\t\tcase HOOKDIR_YES:\n> +\t\tdefault:\n> +\t\t\treturn 1;\n> +\t}\n> +}\n\n[snip]\n\n> +int run_hooks(const char *hookname, struct run_hooks_opt *options)\n> +{\n> +\tstruct strbuf hookname_str = STRBUF_INIT;\n> +\tstruct list_head *to_run, *pos = NULL, *tmp = NULL;\n> +\tint rc = 0;\n> +\n> +\tif (!options)\n> +\t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n> +\n> +\tstrbuf_addstr(&hookname_str, hookname);\n> +\n> +\tto_run = hook_list(&hookname_str);\n> +\n> +\tlist_for_each_safe(pos, tmp, to_run) {\n> +\t\tstruct child_process hook_proc = CHILD_PROCESS_INIT;\n> +\t\tstruct hook *hook = list_entry(pos, struct hook, list);\n> +\n> +\t\thook_proc.env = options->env.v;\n> +\t\thook_proc.no_stdin = 1;\n> +\t\thook_proc.stdout_to_stderr = 1;\n> +\t\thook_proc.trace2_hook_name = hook->command.buf;\n> +\t\thook_proc.use_shell = 1;\n\nI think this is based on run_hook_ve() in run-command.c - could we\nrefactor that to avoid duplication of code?\n\n> +\n> +\t\tif (hook->from_hookdir) {\n> +\t\t if (!should_include_hookdir(hook->command.buf, options->run_hookdir))\n> +\t\t\tcontinue;\n> +\t\t /*\n> +\t\t * Commands from the config could be oneliners, but we know\n> +\t\t * for certain that hookdir commands are not.\n> +\t\t */\n> +\t\t hook_proc.use_shell = 0;\n> +\t\t}\n> +\n> +\t\t/* add command */\n> +\t\tstrvec_push(&hook_proc.args, hook->command.buf);\n> +\n> +\t\t/*\n> +\t\t * add passed-in argv, without expanding - let the user get back\n> +\t\t * exactly what they put in\n> +\t\t */\n> +\t\tstrvec_pushv(&hook_proc.args, options->args.v);\n> +\n> +\t\trc |= run_command(&hook_proc);\n> +\t}\n> +\n> +\treturn rc;\n> +}\n\n[snip]\n\n> +struct run_hooks_opt\n> +{\n> +\t/* Environment vars to be set for each hook */\n> +\tstruct strvec env;\n> +\n> +\t/* Args to be passed to each hook */\n> +\tstruct strvec args;\n> +\n> +\t/*\n> +\t * How should the hookdir be handled?\n> +\t * Leave the RUN_HOOKS_OPT_INIT default in most cases; this only needs\n> +\t * to be overridden if the user can override it at the command line.\n> +\t */\n> +\tenum hookdir_opt run_hookdir;\n> +};\n> +\n> +#define RUN_HOOKS_OPT_INIT { \t\t\\\n> +\t.env = STRVEC_INIT, \t\t\t\t\\\n> +\t.args = STRVEC_INIT, \t\t\t\\\n> +\t.run_hookdir = configured_hookdir_opt()\t\\\n> +}\n\nI don't think we have function invocations in our declarations like\nthis. Maybe stick to just using run_hooks_opt_init().\n\n[snip tests]\n\nThe tests look good.\n"},{"id":"415686","messageId":"20210131043928.1036246-1-jonathantanmy@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-10-emilyshaffer@google.com","subject":"Re: [PATCH v7 09/17] hook: replace find_hook() with hook_exists()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-01-31T04:39:28Z","receivedAt":"2021-01-31T04:41:25Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Add a helper to easily determine whether any hooks exist for a given\n> hook event.\n> \n> Many callers want to check whether some state could be modified by a\n> hook; that check should include the config-based hooks as well. Optimize\n> by checking the config directly. Since commands which execute hooks\n> might want to take args to replace 'hook.runHookDir', let\n> 'hook_exists()' mirror the behavior of 'hook.runHookDir'.\n\nThe text makes sense, but the title might better be \"introduce\nhook_exists()\" instead of \"replace\", since find_hook() is still around.\n\nAlso maybe briefly mention the future plans - e.g. in the future, no\ncode will use find_hook() except , because all of them will use hook_exists() and run_hook().\n\n> +/*\n> + * Returns 1 if any hooks are specified in the config or if a hook exists in the\n> + * hookdir. Typically, invoke hook_exsts() like:\n> + * hook_exists(hookname, configured_hookdir_opt());\n> + * Like with run_hooks, if you take a --run-hookdir flag, reflect that\n> + * user-specified behavior here instead.\n> + */\n> +int hook_exists(const char *hookname, enum hookdir_opt should_run_hookdir);\n\nI wonder if enum hookdir_opt should support a \"unspecified\" instead, in\nwhich case hook_exists() will automatically read the config (instead of\nrelying on the caller to call configured_hookdir_opt()), but I see that\nthis patch set is version 7 and perhaps this design point has already\nbeen discussed.\n"},{"id":"415707","messageId":"20210201053808.1309469-1-jonathantanmy@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-12-emilyshaffer@google.com","subject":"Re: [PATCH v7 11/17] run-command: allow stdin for run_processes_parallel","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-02-01T05:38:08Z","receivedAt":"2021-02-01T05:41:47Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> diff --git a/run-command.c b/run-command.c\n> index ea4d0fb4b1..80c8c97bc1 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1683,6 +1683,9 @@ static int pp_start_one(struct parallel_processes *pp)\n> \tif (i == pp->max_processes)\n> \t\tBUG(\"bookkeeping is hard\");\n> \n> +\t/* disallow by default, but allow users to set up stdin if they wish */\n> +\tpp->children[i].process.no_stdin = 1;\n> +\n\nThis makes sense. May be worth a more detailed comment, e.g.:\n\n By default, do not inherit stdin from the parent process. (If not, all\n children would share it!) Users may overwrite this by having the\n get_next_task function assign 0 to no_stdin and an appropriate integer\n to in.\n"},{"id":"415709","messageId":"20210201060422.1313603-1-jonathantanmy@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-13-emilyshaffer@google.com","subject":"Re: [PATCH v7 12/17] hook: allow parallel hook execution","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-02-01T06:04:22Z","receivedAt":"2021-02-01T06:27:35Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> In many cases, there's no reason not to allow hooks to execute in\n> parallel. run_processes_parallel() is well-suited - it's a task queue\n> that runs its housekeeping in series, which means users don't\n> need to worry about thread safety on their callback data. True\n> multithreaded execution with the async_* functions isn't necessary here.\n> Synchronous hook execution can be achieved by only allowing 1 job to run\n> at a time.\n> \n> Teach run_hooks() to use that function for simple hooks which don't\n> require stdin or capture of stderr.\n\nWhich hooks would be run in parallel, and which hooks in series? I don't\nsee code that distinguishes between them.\n\n> \n> Signed-off-by: Emily Shaffer \n> ---\n> \n> Notes:\n> Per AEvar's request - parallel hook execution on day zero.\n> \n> In most ways run_processes_parallel() worked great for me - but it didn't\n> have great support for hooks where we pipe to and from. I had to add this\n> support later in the series.\n> \n> Since I modified an existing and in-use library I'd appreciate a keen look on\n> these patches.\n\nWhat is the existing and in-use library that you're modifying?\n\n> @@ -246,11 +255,96 @@ void run_hooks_opt_clear(struct run_hooks_opt *o)\n> \tstrvec_clear(&o->args);\n> }\n> \n> +\n> +static int pick_next_hook(struct child_process *cp,\n> +\t\t\t struct strbuf *out,\n> +\t\t\t void *pp_cb,\n> +\t\t\t void **pp_task_cb)\n> +{\n> +\tstruct hook_cb_data *hook_cb = pp_cb;\n> +\n> +\tstruct hook *hook = list_entry(hook_cb->run_me, struct hook, list);\n> +\n> +\tif (hook_cb->head == hook_cb->run_me)\n> +\t\treturn 0;\n> +\n> +\tcp->env = hook_cb->options->env.v;\n> +\tcp->stdout_to_stderr = 1;\n> +\tcp->trace2_hook_name = hook->command.buf;\n> +\n> +\t/* reopen the file for stdin; run_command closes it. */\n> +\tif (hook_cb->options->path_to_stdin) {\n> +\t\tcp->no_stdin = 0;\n> +\t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n> +\t} else {\n> +\t\tcp->no_stdin = 1;\n> +\t}\n> +\n> +\t/*\n> +\t * Commands from the config could be oneliners, but we know\n> +\t * for certain that hookdir commands are not.\n> +\t */\n> +\tif (hook->from_hookdir)\n> +\t\tcp->use_shell = 0;\n> +\telse\n> +\t\tcp->use_shell = 1;\n> +\n> +\t/* add command */\n> +\tstrvec_push(&cp->args, hook->command.buf);\n> +\n> +\t/*\n> +\t * add passed-in argv, without expanding - let the user get back\n> +\t * exactly what they put in\n> +\t */\n> +\tstrvec_pushv(&cp->args, hook_cb->options->args.v);\n\nI just skimmed over this setup-process-for-hook part - it would have\nbeen much clearer if it was refactored into its own function before this\npatch (or better yet, written as its own function in the first place).\nAs it is, there are some unnecessary rewritings - e.g. setting stdin\nafter env, and the use_shell setup.\n\n> diff --git a/hook.h b/hook.h\n\n[snip]\n\n> +/*\n> + * Callback provided to feed_pipe_fn and consume_sideband_fn.\n> + */\n> +struct hook_cb_data {\n> +\tint rc;\n> +\tstruct list_head *head;\n> +\tstruct list_head *run_me;\n> +\tstruct run_hooks_opt *options;\n> +};\n\nCould this be in hook.c instead?\n\nAlso, I think it's clearer if run_me was a struct hook, and set to NULL\nwhen iteration reaches the end. If you disagree, I think it needs some\ndocumentation (e.g. \"the embedded linked list part of the hook that must\nbe run next; if equal to head, then iteration has ended\" or something\nlike that).\n\n> +#define RUN_HOOKS_OPT_INIT_SYNC { \t\t\\\n> \t.env = STRVEC_INIT, \t\t\t\\\n> \t.args = STRVEC_INIT, \t\t\t\\\n> \t.path_to_stdin = NULL,\t\t\t\\\n> +\t.jobs = 1,\t\t\t\t\\\n> \t.run_hookdir = configured_hookdir_opt()\t\\\n> }\n\nThis is not used anywhere.\n"},{"id":"415710","messageId":"20210201065153.1322296-1-jonathantanmy@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-15-emilyshaffer@google.com","subject":"Re: [PATCH v7 14/17] run-command: add stdin callback for parallelization","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-02-01T06:51:53Z","receivedAt":"2021-02-01T06:54:05Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> If a user of the run_processes_parallel() API wants to pipe a large\n> amount of information to stdin of each parallel command, that\n> information could exceed the buffer of the pipe allocated for that\n> process's stdin. Generally this is solved by repeatedly writing to\n> child_process.in between calls to start_command() and finish_command();\n> run_processes_parallel() did not provide users an opportunity to access\n> child_process at that time.\n\n[snip]\n\n> diff --git a/run-command.h b/run-command.h\n> index 6472b38bde..e058c0e2c8 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -436,6 +436,20 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n> \t\t\t\tvoid *pp_cb,\n> \t\t\t\tvoid *pp_task_cb);\n> \n> +/**\n> + * This callback is called repeatedly on every child process who requests\n> + * start_command() to create a pipe by setting child_process.in < 0.\n> + *\n> + * pp_cb is the callback cookie as passed into run_processes_parallel, and\n> + * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> + * The contents of 'send' will be read into the pipe and passed to the pipe.\n> + *\n> + * Return nonzero to close the pipe.\n> + */\n> +typedef int (*feed_pipe_fn)(struct strbuf *pipe,\n> +\t\t\t void *pp_cb,\n> +\t\t\t void *pp_task_cb);\n> +\n\nAs you mention above in the commit message, I think the clearest API to\nsupport what we need is to just have a callback (that has access to\nchild_process) that is executed between process start and finish.\n\nAs it is, I think this callback is too specific in that it takes a\nstruct strbuf. I think that this struct strbuf will just end up being\nunnecessary copying much of the time, when the user could have just\nwritten to the fd directly.\n"},{"id":"415725","messageId":"20210201070448.1325624-1-jonathantanmy@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-16-emilyshaffer@google.com","subject":"Re: [PATCH v7 15/17] hook: provide stdin by string_list or callback","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-02-01T07:04:48Z","receivedAt":"2021-02-01T07:08:31Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> In cases where a hook requires only a small amount of information via\n> stdin, it should be simple for users to provide a string_list alone. But\n> in more complicated cases where the stdin is too large to hold in\n> memory, let's provide a callback the users can populate line after line\n> with instead.\n\n[snip]\n\n> diff --git a/hook.h b/hook.h\n> index 8a7542610c..0ac83fa7ca 100644\n> --- a/hook.h\n> +++ b/hook.h\n> @@ -2,6 +2,7 @@\n> #include \"list.h\"\n> #include \"strbuf.h\"\n> #include \"strvec.h\"\n> +#include \"run-command.h\"\n> \n> struct hook\n> {\n> @@ -14,6 +15,12 @@ struct hook\n> \t/* The literal command to run. */\n> \tstruct strbuf command;\n> \tint from_hookdir;\n> +\n> +\t/*\n> +\t * Use this to keep state for your feed_pipe_fn if you are using\n> +\t * run_hooks_opt.feed_pipe. Otherwise, do not touch it.\n> +\t */\n> +\tvoid *feed_pipe_cb_data;\n\nWhen would we need per-hook state? I see in patch 14 that you give each\nrunning process little by little (in pp_buffer_stdin()), perhaps so that\neach hook can make progress at roughly the same pace, but I don't think\nwe can expect all hooks to work the same, so I don't think it's worth\ncomplicating the design for all that.\n\n> };\n> \n> /*\n> @@ -57,12 +64,24 @@ struct run_hooks_opt\n> \n> \t/* Path to file which should be piped to stdin for each hook */\n> \tconst char *path_to_stdin;\n> +\t/* Pipe each string to stdin, separated by newlines */\n> +\tstruct string_list str_stdin;\n> +\t/*\n> +\t * Callback and state pointer to ask for more content to pipe to stdin.\n> +\t * Will be called repeatedly, for each hook. See\n> +\t * hook.c:pipe_from_stdin() for an example. Keep per-hook state in\n> +\t * hook.feed_pipe_cb_data (per process). Keep initialization context in\n> +\t * feed_pipe_ctx (shared by all processes).\n> +\t */\n> +\tfeed_pipe_fn feed_pipe;\n> +\tvoid *feed_pipe_ctx;\n\nInstead of 3 fields, I think 2 suffice - the function and the data\n(called \"ctx\" here). We can supply a function that treats the data as a\nstring_list.\n"},{"id":"415834","messageId":"xmqqa6snjv4m.fsf@gitster.c.googlers.com","threadId":"54769","inReplyTo":"20201222000220.1491091-2-emilyshaffer@google.com","subject":"Re: [PATCH v7 01/17] doc: propose hooks managed by the config","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-01T22:11:53Z","receivedAt":"2021-02-01T22:12:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Emily Shaffer writes:\n\n> +ways, providing an avenue to deprecate these \"legacy\" hooks if desired. The\n> +handling is based on a config `hook.runHookDir`, which is checked against a\n> +number of cases:\n\nDon't we want to also warn when the setting \"no\" or something\nsimilar prevents the legacy hook from running, to help users\nwho wonder why their hook scripts are not running? I.e.\n\n> +- \"no\": the legacy hook will not be run\n\n+- \"warn-no\": Git will print a warning to stderr before ignoring the\n+ legacy hook\n\n> +- \"interactive\": Git will prompt the user before running the legacy hook\n> +- \"warn\": Git will print a warning to stderr before running the legacy hook\n> +- \"yes\" (default): Git will silently run the legacy hook\n\n> +In case this list is expanded in the future, if a value for `hook.runHookDir` is\n> +given which Git does not recognize, Git should discard that config entry. For\n> +example, if \"warn\" was specified at system level and \"junk\" was specified at\n> +global level, Git would resolve the value to \"warn\"; if the only time the config\n> +was set was to \"junk\", Git would use the default value of \"yes\".\n\nHmph, instead of complaining \"value 'junk' is not recognized\" and\nerroring out? Why?\n\n> +[[stage-3]]\n> +==== Stage 3\n> +\n> +`.git/hooks` is removed from the template and the hook directory is considered\n> +deprecated. To avoid breaking older repos, the default of `hook.runHookDir` is\n> +not changed, and `find_hook()` is not removed.\n\nPresumably, we'll have documentation somewhere that instructs users\n(who were taught by slashdot and other site to add certain scripts\nunder their .git/hooks/) how to do the equivalent without adding\nscripts in .git/hooks/ directory and instead using the config\nmechanism (e.g. \"when told to add script X in .git/hooks/, read such\nan instruction as if telling you to do Y instead\") by the time this\nhappens? It probably makes sense to do so as part of stage-2, at\nwhich point the users are _ready_ to migrate.\n\n> +[[security]]\n> +=== Security and repo config\n> +\n> +Part of the motivation behind this refactor is to mitigate hooks as an attack\n> +vector;footnote:[https://lore.kernel.org/git/20171002234517.GV19555@aiede.mtv.corp.google.com/]\n> +however, as the design stands, users can still provide hooks in the repo-level\n> +config, which is included when a repo is zipped and sent elsewhere. The\n> +security of the repo-level config is still under discussion; this design\n> +generally assumes the repo-level config is secure, which is not true yet. The\n> +goal is to avoid an overcomplicated design to work around a problem which has\n> +ceased to exist.\n\nI doubt we want to claim anything about security as part of this\nseries. As you say in the paragraph, .git/config and .git/hooks/\nare equally (un)protected and if we decide to punt .git/config\nsecurity, then not moving away from .git/hooks would not hurt\nsecurity-wise, either (in other words, security is not a viable\nmotivation behind this series).\n\nAnd if we stop advertising 'security merit' that does not exist,\nwhat remains? Isn't the biggest selling point that an identical set\nof hook configuration can be shared among multiple repositories, and\nit allows more than one hook scripts to be triggered by a single\n\"hook event\"? There may be other good things we should be able to\nsell the new mechanism to our users, and we do stress on them, which\nis done in the motivation section. So...\n\n> +.Comparison of alternatives\n> +|===\n> +|Feature |Config-based hooks |Hook directories |Status quo\n\nSorry, but I did not find this table particularly convincing.\n\nThe only thing I sense is a hand-wavy desire that \"we could make it\nbetter than everybody else if we work on it in this area\", which can\napply equally for other approaches---they could enhance what they\nalready have (e.g. \"discoverability & documentation\").\n\nAs a list of \"these are the points we aspire to do better than other\npeople\", I think it is an excellent idea to have a table like this\nhere in the documentation. But that is not a \"comparison\".\n\n> +[[execution-ordering]]\n> +=== Execution ordering\n> +\n> +We may find that config order is insufficient for some users; for example,\n> +config order makes it difficult to add a new hook to the system or global config\n> +which runs at the end of the hook list. A new ordering schema should be:\n> +\n> +1) Specified by a `hook.order` config, so that users will not unexpectedly see\n> +their order change;\n> +\n> +2) Either dependency or numerically based.\n> +\n> +Dependency-based ordering is prone to classic linked-list problems, like a\n> +cycles and handling of missing dependencies. But, it paves the way for enabling\n> +parallelization if some tasks truly depend on others.\n> +\n> +Numerical ordering makes it tricky for Git to generate suggested ordering\n> +numbers for each command, but is easy to determine a definitive order.\n\nOK.\n\nHave we decided what we do for hooks whose interface is to feed\ntheir input from their standard input? The current system, I think,\njust feeds the single hook by writing into a pipe to it, but if we\nwere to drive multiple hooks, we'd need to write the same thing to\neach of these hook programs? \n\nDo we have a plan to deal with hooks whose outcome is not just\n\"yes/no\", e.g. \"proc-receive\" hook that munges the list of refs to\nbe updated and the new values for them, or \"applypatch-msg\" that\nmunges the incoming proposed commit log message? Does the second\nhook work on the result of the first hook? Do the two hooks work on\nthe vanilla state and their output have to agree with each other?\n\n> +[[parallelization]]\n> +=== Parallelization with dependencies\n> +\n> +Currently hooks use a naive parallelization scheme or are run in series. But if\n> +one hook depends on another's output, then users will want to specify those\n> +dependencies.\n\nAn untold assumption here is that the questions I asked earlier on\nhaving more than one hooks that is not just yes/no is something\nreaders know the same answer, and that answer is \"the outcome of the\nfirst hook is passed along (as if it were the input given by Git\ndirectly, if the first hook did not exist), to the second hook. It\nshould be spelled out somewhere before the execution ordering\nsection, I think.\n"},{"id":"416562","messageId":"YCL5W31A0ClF16ug@google.com","threadId":"54769","inReplyTo":"20210131031011.1016240-1-jonathantanmy@google.com","subject":"Re: [PATCH v7 03/17] hook: add list command","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-02-09T21:06:35Z","receivedAt":"2021-02-09T21:44:56Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Sat, Jan 30, 2021 at 07:10:11PM -0800, Jonathan Tan wrote:\n> \n> > Teach 'git hook list ', which checks the known configs in\n> > order to create an ordered list of hooks to run on a given hook event.\n> > \n> > Multiple commands can be specified for a given hook by providing\n> > multiple \"hook..command = \" lines. Hooks will be\n> > run in config order. If more properties need to be set on a given hook\n> > in the future, commands can also be specified by providing\n> > \"hook..command = \", as well as a \"[hookcmd\n> > ]\" subsection; at minimum, this subsection must contain a\n> > \"hookcmd..command = \" line.\n> \n> I learned later that this isn't true - in patch 6, the commit message\n> and one of the tests therein describe being able to skip a previously\n> inline command by making a hookcmd section of the same name and just\n> specifying \"skip = true\" there (without any command).\n> \n> Maybe just delete the \"at minimum\" part.\n\nThanks, nice catch. I dropped \"at minimum\" and s/must/should/. Since I\nreceived feedback during the dogfood internally that there's no\ndocumentation on \"hookcmd..skip\" whatsoever, I'll try to make\na more detailed overview of hookcmd acting as an alias in the\nuser-facing docs in that commit.\n\n> \n> > +static int list(int argc, const char **argv, const char *prefix)\n> > {\n> > -\tstruct option builtin_hook_options[] = {\n> > +\tstruct list_head *head, *pos;\n> > +\tstruct hook *item;\n> \n> You asked for review on nits too so here's one: \"item\" should be\n> declared in the list_for_each block. (That also makes it easier to see\n> that we don't need to free it.)\n\nOk, done.\n\n> \n> > diff --git a/hook.c b/hook.c\n> > new file mode 100644\n> > index 0000000000..937dc768c8\n> > --- /dev/null\n> > +++ b/hook.c\n> > @@ -0,0 +1,115 @@\n> > +#include \"cache.h\"\n> > +\n> > +#include \"hook.h\"\n> > +#include \"config.h\"\n> \n> Usually we put all the includes together without any intervening blank\n> lines.\n\nDone.\n\n> \n> > +static void append_or_move_hook(struct list_head *head, const char *command)\n> > +{\n> > +\tstruct list_head *pos = NULL, *tmp = NULL;\n> > +\tstruct hook *to_add = NULL;\n> > +\n> > +\t/*\n> > +\t * remove the prior entry with this command; we'll replace it at the\n> > +\t * end.\n> > +\t */\n> > +\tlist_for_each_safe(pos, tmp, head) {\n> > +\t\tstruct hook *it = list_entry(pos, struct hook, list);\n> > +\t\tif (!strcmp(it->command.buf, command)) {\n> > +\t\t list_del(pos);\n> > +\t\t /* we'll simply move the hook to the end */\n> > +\t\t to_add = it;\n> \n> \"break\" here?\n\nI think it's safe to do so, but I think I left it out in case duplicates\ndo make it into the list somehow. But if they're always being inserted\nvia \"append_or_move_hook()\" that should not be an issue, so I'll add the\nbreak.\n\nIn fact, fixing this exposed a bug. Later, I add using 'pos' instead of\n'head':\n list_add_tail(&to_add->list, pos);\nWhen I'm guaranteed to iterate to the end of the list, that works fine,\nbecause in this implementation the last element of the list has \"->next\"\nset back to \"head\". But when 'pos' isn't sure to be at the end of the\nlist, it breaks the list. Whoops.. :)\n\n\n> \n> > +\t\t}\n> > +\t}\n> > +\n> > +\tif (!to_add) {\n> > +\t\t/* adding a new hook, not moving an old one */\n> > +\t\tto_add = xmalloc(sizeof(struct hook));\n> \n> Style is to write sizeof(*to_add), I think.\n\nSure.\n\n> \n> [snip]\n> \n> > +struct hook_config_cb\n> > +{\n> > +\tstruct strbuf *hookname;\n> \n> struct declarations have \"{\" not on a line on its own.\n\nSure, I can change it. Although, 'git grep -E \"struct \\w+$\"' tells me\nthere are a few other offenders :) But the count (60ish minus\ndoc/relnotes references) is much lower than 'git grep -E \"struct \\w+\n\\{$\"' (800+ matches) so I'm definitely doing it wrong :D\n\n> \n> Also, \"hookname\" could just be a char *?\n\nHrmph. I think my C++ background is showing - to me, calling \".buf\"\n(which is cheap) a few times is a small price to pay to get length\ninfo and so on for free. \"hookname\" can come from the user - \"git\nhook (list|run) repo-special-magic-hook\" - so I worry about leaving it as a raw\nchar* with no size info associated. Am I being too paranoid?\n\n> \t\n> > +\tstruct list_head *list;\n> > +};\n> > +\n> > +static int hook_config_lookup(const char *key, const char *value, void *cb_data)\n> > +{\n> > +\tstruct hook_config_cb *data = cb_data;\n> > +\tconst char *hook_key = data->hookname->buf;\n> > +\tstruct list_head *head = data->list;\n> > +\n> > +\tif (!strcmp(key, hook_key)) {\n> > +\t\tconst char *command = value;\n> > +\t\tstruct strbuf hookcmd_name = STRBUF_INIT;\n> > +\n> > +\t\t/* Check if a hookcmd with that name exists. */\n> > +\t\tstrbuf_addf(&hookcmd_name, \"hookcmd.%s.command\", command);\n> > +\t\tgit_config_get_value(hookcmd_name.buf, &command);\n> \n> So we don't really care whether git_config_get_value returns 0 or 1 as\n> long as it doesn't touch \"command\" if there is no such hookcmd. That\n> fits with how git_config_get_value() is documented, so that's great.\n> Perhaps better to document as:\n> \n> If a hookcmd.%s.command config exists, replace the command with the\n> value of that config. (If not, do nothing - git_config_get_value() is\n> documented to not overwrite the value argument in this case.)\n\nOK. Thanks, done.\n\n> \n> > +\n> > +\t\tif (!command) {\n> > +\t\t\tstrbuf_release(&hookcmd_name);\n> > +\t\t\tBUG(\"git_config_get_value overwrote a string it shouldn't have\");\n> > +\t\t}\n> > +\n> > +\t\t/*\n> > +\t\t * TODO: implement an option-getting callback, e.g.\n> > +\t\t * get configs by pattern hookcmd.$value.*\n> > +\t\t * for each key+value, do_callback(key, value, cb_data)\n> > +\t\t */\n> > +\n> > +\t\tappend_or_move_hook(head, command);\n> > +\n> > +\t\tstrbuf_release(&hookcmd_name);\n> > +\t}\n> > +\n> > +\treturn 0;\n> > +}\n> > +\n> > +struct list_head* hook_list(const struct strbuf* hookname)\n> \n> \"const char *hookname\" should suffice?\n> \n> Also, search for \"* \" and replace with \" *\" where applicable (also in\n> the .h file).\n\nSee above.\n\n> \n> > +{\n> > +\tstruct strbuf hook_key = STRBUF_INIT;\n> > +\tstruct list_head *hook_head = xmalloc(sizeof(struct list_head));\n> > +\tstruct hook_config_cb cb_data = { &hook_key, hook_head };\n> > +\n> > +\tINIT_LIST_HEAD(hook_head);\n> > +\n> > +\tif (!hookname)\n> > +\t\treturn NULL;\n> > +\n> > +\tstrbuf_addf(&hook_key, \"hook.%s.command\", hookname->buf);\n> > +\n> > +\tgit_config(hook_config_lookup, (void*)&cb_data);\n> \n> Do we need this void* cast?\n\nNope. Dropped, thanks.\n\n> \n> > +\n> > +\tstrbuf_release(&hook_key);\n> > +\treturn hook_head;\n> > +}\n> > diff --git a/hook.h b/hook.h\n> > new file mode 100644\n> > index 0000000000..8ffc4f14b6\n> > --- /dev/null\n> > +++ b/hook.h\n> > @@ -0,0 +1,26 @@\n> > +#include \"config.h\"\n> > +#include \"list.h\"\n> > +#include \"strbuf.h\"\n> > +\n> > +struct hook\n> > +{\n> > +\tstruct list_head list;\n> > +\t/*\n> > +\t * Config file which holds the hook.*.command definition.\n> > +\t * (This has nothing to do with the hookcmd..* configs.)\n> > +\t */\n> > +\tenum config_scope origin;\n> > +\t/* The literal command to run. */\n> > +\tstruct strbuf command;\n> \n> \"char *\" would suffice?\n\nSince 'command' comes directly from user input and is executed, I'm\nnervous about using a char* instead of a strbuf even more so than for\nthe hookname.\n\n> \n> The tests look fine.\n\nThanks for the nitpicky review, it was great! :)\n\n - Emily\n"},{"id":"416585","messageId":"YCMNQe/ibc/xOL97@google.com","threadId":"54769","inReplyTo":"20210131033503.1022138-1-jonathantanmy@google.com","subject":"Re: [PATCH v7 05/17] hook: respect hook.runHookDir","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-02-09T22:31:29Z","receivedAt":"2021-02-09T23:11:43Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Sat, Jan 30, 2021 at 07:35:03PM -0800, Jonathan Tan wrote:\n> \n> > Include hooks specified in the hook directory in the list of hooks to\n> > run. These hooks do need to be treated differently from config-specified\n> > ones - they do not need to run in a shell, and later on may be disabled\n> > or warned about based on a config setting.\n> > \n> > Because they are at least as local as the local config, we'll run them\n> > last - to keep the hook execution order from global to local.\n> \n> This commit message doesn't seem to match the code change. Firstly,\n> we're teaching hook.runHookDir, not respecting it (since it did not\n> exist before this commit), and it's about showing it in \"list\" and not\n> about running it at all.\n\nYeah, thanks for noticing. Now that I'm rereading it, it looks like this\nis the old commit message for \"include hookdir hook in list\". Yikes.\n> \n> Perhaps just \"hook: teach hook.runHookDir\" as the subject and as the\n> body:\n> \n> For now, this just affects the output of \"git hook list\". In the\n> future, this will affect the behavior of \"git hook run\" and when Git\n> runs hooks before or after its operations.\n> \n> > +\tswitch (should_run_hookdir) {\n> > +\t\tcase HOOKDIR_NO:\n> > +\t\t\tstrbuf_addstr(&hookdir_annotation, _(\" (will not run)\"));\n> > +\t\t\tbreak;\n> > +\t\tcase HOOKDIR_INTERACTIVE:\n> > +\t\t\tstrbuf_addstr(&hookdir_annotation, _(\" (will prompt)\"));\n> > +\t\t\tbreak;\n> > +\t\tcase HOOKDIR_WARN:\n> > +\t\tcase HOOKDIR_UNKNOWN:\n> \n> Hmm...UNKNOWN is the same as WARN? This doesn't agree with what is said\n> in patch 1:\n> \n> +In case this list is expanded in the future, if a value for `hook.runHookDir` is\n> +given which Git does not recognize, Git should discard that config entry. For\n> +example, if \"warn\" was specified at system level and \"junk\" was specified at\n> +global level, Git would resolve the value to \"warn\"; if the only time the config\n> +was set was to \"junk\", Git would use the default value of \"yes\".\n> \n> But having said that, I would prefer if Git just errored out in this\n> case.\n\nEh, I think I'd still like it to be tolerant. Thanks for keeping me\nhonest :) I added a test to enforce the tolerant behavior, too.\n\n> \n> The rest looks good.\n\nThanks.\n"},{"id":"416589","messageId":"YCMTTlYF2WR4RIOV@google.com","threadId":"54769","inReplyTo":"20210131034030.1025259-1-jonathantanmy@google.com","subject":"Re: [PATCH v7 06/17] hook: implement hookcmd..skip","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-02-09T22:57:18Z","receivedAt":"2021-02-09T23:15:29Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Sat, Jan 30, 2021 at 07:40:30PM -0800, Jonathan Tan wrote:\n> \n> > If a user wants a specific repo to skip execution of a hook which is set\n> > at a global or system level, they can now do so by specifying 'skip' in\n> > their repo config:\n> \n> Usually the present tense describes the situation before the commit, so\n> maybe s/they can now do so/they will be able to do so/.\n\nSure.\n\n> \n> > -static void append_or_move_hook(struct list_head *head, const char *command)\n> > +static struct hook* find_hook_by_command(struct list_head *head, const char *command)\n> \n> \"* \" -> \" *\"\n\nThanks.\n\n> \n> [snip tests]\n> \n> For the tests, I thought of the case in which we skip a hookcmd that was\n> never specified as a hook, but that's probably not very useful.\n\nAh, it might be useful to make sure we don't choke trying to remove\nsomething that isn't there - I'll add one.\n\n\nBy the way, I got feedback from Googlers using config hooks that \"skip\"\nisn't actually documented anywhere public-facing. For v8 I've added a\nsection on it to Documentation/git-hook.txt as well as to\nDocumentation/config/hook.txt.\n\n - Emily\n"},{"id":"416590","messageId":"YCMHOeLqESHokPZL@google.com","threadId":"54769","inReplyTo":"20210131032022.1018820-1-jonathantanmy@google.com","subject":"Re: [PATCH v7 04/17] hook: include hookdir hook in list","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-02-09T22:05:45Z","receivedAt":"2021-02-10T00:09:50Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Sat, Jan 30, 2021 at 07:20:22PM -0800, Jonathan Tan wrote:\n> \n> > Historically, hooks are declared by placing an executable into\n> > $GIT_DIR/hooks/$HOOKNAME (or $HOOKDIR/$HOOKNAME). Although hooks taken\n> > from the config are more featureful than hooks placed in the $HOOKDIR,\n> > those hooks should not stop working for users who already have them.\n> \n> Maybe explicitly add that we're listing them in the list with a \"hookdir:\"\n> prefix.\n\nSure.\n\n> \n> > Legacy hooks should be run directly, not in shell. We know that they are\n> > a path to an executable, not a oneliner script - and running them\n> > directly takes care of path quoting concerns for us for free.\n> \n> Not sure what this paragraph is doing here.\n\nYep, this is an artifact of the review process (explaining why I didn't\ndo something weird, which I did in an earlier version, but now it\ndoesn't make sense to mention it at all). Deleted.\n\n> \n> > diff --git a/builtin/hook.c b/builtin/hook.c\n> > index 4d36de52f8..a0013ae4d7 100644\n> > --- a/builtin/hook.c\n> > +++ b/builtin/hook.c\n> > @@ -16,6 +16,7 @@ static int list(int argc, const char **argv, const char *prefix)\n> > \tstruct list_head *head, *pos;\n> > \tstruct hook *item;\n> > \tstruct strbuf hookname = STRBUF_INIT;\n> > +\tstruct strbuf hookdir_annotation = STRBUF_INIT;\n> \n> Right now this is never set? Maybe hold off on adding this until we set\n> something.\n\nYeah, that makes sense. Will do.\n\n> \n> > @@ -110,6 +113,18 @@ struct list_head* hook_list(const struct strbuf* hookname)\n> > \n> > \tgit_config(hook_config_lookup, (void*)&cb_data);\n> > \n> > +\tif (have_git_dir())\n> > +\t\tlegacy_hook_path = find_hook(hookname->buf);\n> > +\n> > +\t/* Unconditionally add legacy hook, but annotate it. */\n> > +\tif (legacy_hook_path) {\n> > +\t\tstruct hook *legacy_hook;\n> > +\n> > +\t\tappend_or_move_hook(hook_head, absolute_path(legacy_hook_path));\n> \n> Both find_hook() and absolute_path() use static buffers to hold their\n> return values, which makes me a bit nervous. Perhaps put them all under\n> the same \"if (have_git_dir())\" so that it's clearer that we're not\n> supposed to insert code arbitrarily between their invocation and their\n> usage.\n\nOh, that's a cool way to indicate that. Thanks, I did that, and learned\nsomething new!\n\n> \n> > diff --git a/hook.h b/hook.h\n> > index 8ffc4f14b6..5750634c83 100644\n> > --- a/hook.h\n> > +++ b/hook.h\n> > @@ -12,6 +12,7 @@ struct hook\n> > \tenum config_scope origin;\n> > \t/* The literal command to run. */\n> > \tstruct strbuf command;\n> > +\tint from_hookdir;\n> \n> unsigned from_hookdir : 1?\n\nSure. It doesn't make a difference now but I see that would be nice for\nfutureproofing.\n\n> \n> The tests look good.\n\nThanks.\n"},{"id":"416778","messageId":"YCWzQi2vbiZ18tOF@google.com","threadId":"54769","inReplyTo":"20210131042254.1032233-1-jonathantanmy@google.com","subject":"Re: [PATCH v7 08/17] hook: add 'run' subcommand","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-02-11T22:44:18Z","receivedAt":"2021-02-11T22:45:24Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Sat, Jan 30, 2021 at 08:22:54PM -0800, Jonathan Tan wrote:\n> \n> > In order to enable hooks to be run as an external process, by a\n> > standalone Git command, or by tools which wrap Git, provide an external\n> > means to run all configured hook commands for a given hook event.\n> > \n> > For now, the hook commands will run in config order, in series. As\n> > alternate ordering or parallelism is supported in the future, we should\n> > add knobs to use those to the command line as well.\n> > \n> > As with the legacy hook implementation, all stdout generated by hook\n> > commands is redirected to stderr. Piping from stdin is not yet\n> > supported.\n> > \n> > Legacy hooks (those present in $GITDIR/hooks) are run at the end of the\n> > execution list. For now, there is no way to disable them.\n> \n> Not true anymore now that we have hook.runHookDir :-)\n\nUpdated.\n\n> \n> > @@ -64,6 +65,32 @@ in the order they should be run, and print the config scope where the relevant\n> > `hook..command` was specified, not the `hookcmd` (if applicable).\n> > This output is human-readable and the format is subject to change over time.\n> > \n> > +run [(-e|--env)=...] [(-a|--arg)=...] ``::\n> > +\n> > +Runs hooks configured for ``, in the same order displayed by `git\n> > +hook list`. Hooks configured this way are run prepended with `sh -c`, so paths\n> > +containing special characters or spaces should be wrapped in single quotes:\n> > +`command = '/my/path with spaces/script.sh' some args`.\n> \n> I learned recently that this may not work the way I expect [1], so you\n> might want to specifically call this out for someone who knows how\n> run-command and running-with-shell works.\n\nI think it might be good enough to say \"may be prepended\" instead - the\nquoting advice (wrap your paths) still holds.\n\n> \n> [1] https://lore.kernel.org/git/YAs9pTBsdskC8CPN@coredump.intra.peff.net/\n> \n> > @@ -135,6 +136,56 @@ enum hookdir_opt configured_hookdir_opt(void)\n> > \treturn HOOKDIR_UNKNOWN;\n> > }\n> > \n> > +static int should_include_hookdir(const char *path, enum hookdir_opt cfg)\n> > +{\n> > +\tstruct strbuf prompt = STRBUF_INIT;\n> > +\t/*\n> > +\t * If the path doesn't exist, don't bother adding the empty hook and\n> > +\t * don't bother checking the config or prompting the user.\n> > +\t */\n> > +\tif (!path)\n> > +\t\treturn 0;\n> > +\n> > +\tswitch (cfg)\n> > +\t{\n> > +\t\tcase HOOKDIR_NO:\n> > +\t\t\treturn 0;\n> > +\t\tcase HOOKDIR_UNKNOWN:\n> > +\t\t\tfprintf(stderr,\n> > +\t\t\t\t_(\"Unrecognized value for 'hook.runHookDir'. \"\n> > +\t\t\t\t \"Is there a typo? \"));\n> > +\t\t\t/* FALLTHROUGH */\n> \n> Same comment (about UNKNOWN and defaulting to WARN instead of YES) as in\n> one of the previous patches.\n\nLike in the previous patch, I opted to make it match the design doc\n(UNKNOWN matches default, aka YES). I left the typo warning, though, as\nit might be useful for someone trying to debug why \"itneractive\" isn't\nworking as they expect.\n\n> \n> > +\t\tcase HOOKDIR_WARN:\n> > +\t\t\tfprintf(stderr, _(\"Running legacy hook at '%s'\\n\"),\n> > +\t\t\t\tpath);\n> > +\t\t\treturn 1;\n> > +\t\tcase HOOKDIR_INTERACTIVE:\n> > +\t\t\tdo {\n> > +\t\t\t\t/*\n> > +\t\t\t\t * TRANSLATORS: Make sure to include [Y] and [n]\n> > +\t\t\t\t * in your translation. Only English input is\n> > +\t\t\t\t * accepted. Default option is \"yes\".\n> > +\t\t\t\t */\n> > +\t\t\t\tfprintf(stderr, _(\"Run '%s'? [Yn] \"), path);\n> > +\t\t\t\tgit_read_line_interactively(&prompt);\n> > +\t\t\t\tstrbuf_tolower(&prompt);\n> > +\t\t\t\tif (starts_with(prompt.buf, \"n\")) {\n> > +\t\t\t\t\tstrbuf_release(&prompt);\n> > +\t\t\t\t\treturn 0;\n> > +\t\t\t\t} else if (starts_with(prompt.buf, \"y\")) {\n> > +\t\t\t\t\tstrbuf_release(&prompt);\n> > +\t\t\t\t\treturn 1;\n> > +\t\t\t\t}\n> > +\t\t\t\t/* otherwise, we didn't understand the input */\n> > +\t\t\t} while (prompt.len); /* an empty reply means \"Yes\" */\n> > +\t\t\tstrbuf_release(&prompt);\n> > +\t\t\treturn 1;\n> > +\t\tcase HOOKDIR_YES:\n> > +\t\tdefault:\n> > +\t\t\treturn 1;\n> > +\t}\n> > +}\n> \n> [snip]\n> \n> > +int run_hooks(const char *hookname, struct run_hooks_opt *options)\n> > +{\n> > +\tstruct strbuf hookname_str = STRBUF_INIT;\n> > +\tstruct list_head *to_run, *pos = NULL, *tmp = NULL;\n> > +\tint rc = 0;\n> > +\n> > +\tif (!options)\n> > +\t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n> > +\n> > +\tstrbuf_addstr(&hookname_str, hookname);\n> > +\n> > +\tto_run = hook_list(&hookname_str);\n> > +\n> > +\tlist_for_each_safe(pos, tmp, to_run) {\n> > +\t\tstruct child_process hook_proc = CHILD_PROCESS_INIT;\n> > +\t\tstruct hook *hook = list_entry(pos, struct hook, list);\n> > +\n> > +\t\thook_proc.env = options->env.v;\n> > +\t\thook_proc.no_stdin = 1;\n> > +\t\thook_proc.stdout_to_stderr = 1;\n> > +\t\thook_proc.trace2_hook_name = hook->command.buf;\n> > +\t\thook_proc.use_shell = 1;\n> \n> I think this is based on run_hook_ve() in run-command.c - could we\n> refactor that to avoid duplication of code?\n\nHm. At the end of part II of this series \"run_hook_ve()\" is deleted\nentirely; the implementation of \"run_hooks\" diverges significantly as\nthe series progresses (supporting stdin/stdout, etc) so I'd rather not\ntry to keep one of them based on the other, as I think it'll be more\ncomplicated than it seems in this patch.\n\n> \n> > +\n> > +\t\tif (hook->from_hookdir) {\n> > +\t\t if (!should_include_hookdir(hook->command.buf, options->run_hookdir))\n> > +\t\t\tcontinue;\n> > +\t\t /*\n> > +\t\t * Commands from the config could be oneliners, but we know\n> > +\t\t * for certain that hookdir commands are not.\n> > +\t\t */\n> > +\t\t hook_proc.use_shell = 0;\n> > +\t\t}\n> > +\n> > +\t\t/* add command */\n> > +\t\tstrvec_push(&hook_proc.args, hook->command.buf);\n> > +\n> > +\t\t/*\n> > +\t\t * add passed-in argv, without expanding - let the user get back\n> > +\t\t * exactly what they put in\n> > +\t\t */\n> > +\t\tstrvec_pushv(&hook_proc.args, options->args.v);\n> > +\n> > +\t\trc |= run_command(&hook_proc);\n> > +\t}\n> > +\n> > +\treturn rc;\n> > +}\n> \n> [snip]\n> \n> > +struct run_hooks_opt\n> > +{\n> > +\t/* Environment vars to be set for each hook */\n> > +\tstruct strvec env;\n> > +\n> > +\t/* Args to be passed to each hook */\n> > +\tstruct strvec args;\n> > +\n> > +\t/*\n> > +\t * How should the hookdir be handled?\n> > +\t * Leave the RUN_HOOKS_OPT_INIT default in most cases; this only needs\n> > +\t * to be overridden if the user can override it at the command line.\n> > +\t */\n> > +\tenum hookdir_opt run_hookdir;\n> > +};\n> > +\n> > +#define RUN_HOOKS_OPT_INIT { \t\t\\\n> > +\t.env = STRVEC_INIT, \t\t\t\t\\\n> > +\t.args = STRVEC_INIT, \t\t\t\\\n> > +\t.run_hookdir = configured_hookdir_opt()\t\\\n> > +}\n> \n> I don't think we have function invocations in our declarations like\n> this. Maybe stick to just using run_hooks_opt_init().\n\nSure.\n\n> \n> [snip tests]\n> \n> The tests look good.\n\nThanks for the detailed review.\n\n - Emily\n"},{"id":"416804","messageId":"YCb99egCYoTJUU8B@google.com","threadId":"54769","inReplyTo":"20210131043928.1036246-1-jonathantanmy@google.com","subject":"Re: [PATCH v7 09/17] hook: replace find_hook() with hook_exists()","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-02-12T22:15:17Z","receivedAt":"2021-02-12T22:16:26Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Sat, Jan 30, 2021 at 08:39:28PM -0800, Jonathan Tan wrote:\n> \n> > Add a helper to easily determine whether any hooks exist for a given\n> > hook event.\n> > \n> > Many callers want to check whether some state could be modified by a\n> > hook; that check should include the config-based hooks as well. Optimize\n> > by checking the config directly. Since commands which execute hooks\n> > might want to take args to replace 'hook.runHookDir', let\n> > 'hook_exists()' mirror the behavior of 'hook.runHookDir'.\n> \n> The text makes sense, but the title might better be \"introduce\n> hook_exists()\" instead of \"replace\", since find_hook() is still around.\n> \n> Also maybe briefly mention the future plans - e.g. in the future, no\n> code will use find_hook() except are>, because all of them will use hook_exists() and run_hook().\n> \n> > +/*\n> > + * Returns 1 if any hooks are specified in the config or if a hook exists in the\n> > + * hookdir. Typically, invoke hook_exsts() like:\n> > + * hook_exists(hookname, configured_hookdir_opt());\n> > + * Like with run_hooks, if you take a --run-hookdir flag, reflect that\n> > + * user-specified behavior here instead.\n> > + */\n> > +int hook_exists(const char *hookname, enum hookdir_opt should_run_hookdir);\n> \n> I wonder if enum hookdir_opt should support a \"unspecified\" instead, in\n> which case hook_exists() will automatically read the config (instead of\n> relying on the caller to call configured_hookdir_opt()), but I see that\n> this patch set is version 7 and perhaps this design point has already\n> been discussed.\n\nNo, I don't think it has been. So you mean something like:\n\n enum hookdir_opt\n {\n \tHOOKDIR_NO,\n \tHOOKDIR_WARN,\n \tHOOKDIR_INTERACTIVE,\n \tHOOKDIR_YES,\n \tHOOKDIR_USE_CFG,\n \tHOOKDIR_UNKNOWN,\n };\n\n(name subject to quibbling) and then reimagining\nconfigured_hookdir_opt() to something like:\n\n enum hookdir_opt resolve_hookdir_opt(enum hookdir_opt o)\n {\n \tif (o != HOOKDIR_USE_CFG)\n\t\treturn o;\n\t/* former contents of configured_hookdir_opt here */\n }\n\nI like that, if nobody has complaints.\n\n - Emily\n"},{"id":"417099","messageId":"YCwhA6VIs16uMnJG@google.com","threadId":"54769","inReplyTo":"20201222000220.1491091-1-emilyshaffer@google.com","subject":"Re: [PATCH v7 00/17] propose config-based hooks (part I)","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2021-02-16T19:46:11Z","receivedAt":"2021-02-16T19:47:19Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2020.12.21 16:02, Emily Shaffer wrote:\n> Since v6:\n> \n> - Converted 'enum hookdir_opt' to UPPER_SNAKE\n> - Coccinelle fix in the hook destructor\n> - Fixed a bug where builtin/hook.c wasn't running the default git config setup\n> and therefore missed hooks in core.hooksPath when it was set. (These hooks\n> would still run except when invoked by 'git hook run' as the config was\n> called by the processes which invoked the hook library.)\n> \n> CI run: https://github.com/nasamuffin/git/actions/runs/436864964\n> \n> Thanks!\n> - Emily\n> \n> Emily Shaffer (17):\n> doc: propose hooks managed by the config\n> hook: scaffolding for git-hook subcommand\n> hook: add list command\n> hook: include hookdir hook in list\n> hook: respect hook.runHookDir\n> hook: implement hookcmd..skip\n> parse-options: parse into strvec\n> hook: add 'run' subcommand\n> hook: replace find_hook() with hook_exists()\n> hook: support passing stdin to hooks\n> run-command: allow stdin for run_processes_parallel\n> hook: allow parallel hook execution\n> hook: allow specifying working directory for hooks\n> run-command: add stdin callback for parallelization\n> hook: provide stdin by string_list or callback\n> run-command: allow capturing of collated output\n> hooks: allow callers to capture output\n> \n> .gitignore | 1 +\n> Documentation/Makefile | 1 +\n> Documentation/config/hook.txt | 19 +\n> Documentation/git-hook.txt | 117 +++++\n> Documentation/technical/api-parse-options.txt | 5 +\n> .../technical/config-based-hooks.txt | 355 +++++++++++++++\n> Makefile | 2 +\n> builtin.h | 1 +\n> builtin/bugreport.c | 4 +-\n> builtin/fetch.c | 1 +\n> builtin/hook.c | 176 ++++++++\n> builtin/submodule--helper.c | 2 +-\n> command-list.txt | 1 +\n> git.c | 1 +\n> hook.c | 416 ++++++++++++++++++\n> hook.h | 154 +++++++\n> parse-options-cb.c | 16 +\n> parse-options.h | 4 +\n> run-command.c | 85 +++-\n> run-command.h | 31 ++\n> submodule.c | 1 +\n> t/helper/test-run-command.c | 46 +-\n> t/t0061-run-command.sh | 37 ++\n> t/t1360-config-based-hooks.sh | 256 +++++++++++\n> 24 files changed, 1717 insertions(+), 15 deletions(-)\n> create mode 100644 Documentation/config/hook.txt\n> create mode 100644 Documentation/git-hook.txt\n> create mode 100644 Documentation/technical/config-based-hooks.txt\n> create mode 100644 builtin/hook.c\n> create mode 100644 hook.c\n> create mode 100644 hook.h\n> create mode 100755 t/t1360-config-based-hooks.sh\n> \n> -- \n> 2.28.0.rc0.142.g3c755180ce-goog\n> \n\nSorry for the delayed reply. I am happy with this series as-is. Thanks\nfor all your work on it!\n\nReviewed-by: Josh Steadmon \n"},{"id":"417116","messageId":"xmqqy2fnocjc.fsf@gitster.c.googlers.com","threadId":"54769","inReplyTo":"YCwhA6VIs16uMnJG@google.com","subject":"Re: [PATCH v7 00/17] propose config-based hooks (part I)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-16T22:47:35Z","receivedAt":"2021-02-16T22:48:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n>> Emily Shaffer (17):\n>> doc: propose hooks managed by the config\n>> hook: scaffolding for git-hook subcommand\n>> hook: add list command\n>> hook: include hookdir hook in list\n>> hook: respect hook.runHookDir\n>> hook: implement hookcmd..skip\n>> parse-options: parse into strvec\n>> hook: add 'run' subcommand\n>> hook: replace find_hook() with hook_exists()\n>> hook: support passing stdin to hooks\n>> run-command: allow stdin for run_processes_parallel\n>> hook: allow parallel hook execution\n>> hook: allow specifying working directory for hooks\n>> run-command: add stdin callback for parallelization\n>> hook: provide stdin by string_list or callback\n>> run-command: allow capturing of collated output\n>> hooks: allow callers to capture output\n>> \n> Sorry for the delayed reply. I am happy with this series as-is. Thanks\n> for all your work on it!\n>\n> Reviewed-by: Josh Steadmon \n\nThe topic branch has a lot more commits than these 17; I am\nwondering if the reviewed-by applies only to the bottom 17, or as\nthe whole? I recall that the upper half was expecting at least some\ndocumentation updates.\n\nThanks.\n"},{"id":"417222","messageId":"YC2IzVxLgvC7N8qd@google.com","threadId":"54769","inReplyTo":"xmqqy2fnocjc.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v7 00/17] propose config-based hooks (part I)","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2021-02-17T21:21:17Z","receivedAt":"2021-02-17T21:22:24Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2021.02.16 14:47, Junio C Hamano wrote:\n> Josh Steadmon writes:\n> \n> >> Emily Shaffer (17):\n> >> doc: propose hooks managed by the config\n> >> hook: scaffolding for git-hook subcommand\n> >> hook: add list command\n> >> hook: include hookdir hook in list\n> >> hook: respect hook.runHookDir\n> >> hook: implement hookcmd..skip\n> >> parse-options: parse into strvec\n> >> hook: add 'run' subcommand\n> >> hook: replace find_hook() with hook_exists()\n> >> hook: support passing stdin to hooks\n> >> run-command: allow stdin for run_processes_parallel\n> >> hook: allow parallel hook execution\n> >> hook: allow specifying working directory for hooks\n> >> run-command: add stdin callback for parallelization\n> >> hook: provide stdin by string_list or callback\n> >> run-command: allow capturing of collated output\n> >> hooks: allow callers to capture output\n> >> \n> > Sorry for the delayed reply. I am happy with this series as-is. Thanks\n> > for all your work on it!\n> >\n> > Reviewed-by: Josh Steadmon \n> \n> The topic branch has a lot more commits than these 17; I am\n> wondering if the reviewed-by applies only to the bottom 17, or as\n> the whole? I recall that the upper half was expecting at least some\n> documentation updates.\n> \n> Thanks.\n\nJust to these 17, sorry for being unclear.\n"},{"id":"417242","messageId":"xmqqlfbmi99f.fsf@gitster.g","threadId":"54769","inReplyTo":"YC2IzVxLgvC7N8qd@google.com","subject":"Re: [PATCH v7 00/17] propose config-based hooks (part I)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-17T23:07:08Z","receivedAt":"2021-02-17T23:08:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n>> The topic branch has a lot more commits than these 17; I am\n>> wondering if the reviewed-by applies only to the bottom 17, or as\n>> the whole? I recall that the upper half was expecting at least some\n>> documentation updates.\n>> \n>> Thanks.\n>\n> Just to these 17, sorry for being unclear.\n\nThanks for reading them through.\n\nI am tempted to say we should merge these \"mechanism\" part down to\n'next', hoping that the \"rewrite existing ones using the new\nmechansim\" part can follow soon.\n\n"},{"id":"417362","messageId":"YC7o2rUQOEdiMdqh@google.com","threadId":"54769","inReplyTo":"20210131043928.1036246-1-jonathantanmy@google.com","subject":"Re: [PATCH v7 09/17] hook: replace find_hook() with hook_exists()","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-02-18T22:23:22Z","receivedAt":"2021-02-18T22:24:38Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Sat, Jan 30, 2021 at 08:39:28PM -0800, Jonathan Tan wrote:\n> \n> > Add a helper to easily determine whether any hooks exist for a given\n> > hook event.\n> > \n> > Many callers want to check whether some state could be modified by a\n> > hook; that check should include the config-based hooks as well. Optimize\n> > by checking the config directly. Since commands which execute hooks\n> > might want to take args to replace 'hook.runHookDir', let\n> > 'hook_exists()' mirror the behavior of 'hook.runHookDir'.\n> \n> The text makes sense, but the title might better be \"introduce\n> hook_exists()\" instead of \"replace\", since find_hook() is still around.\n\nYep. Also, removed the one instance I did replace in this commit for\nsome reason (builtin/bugreport.c). Not sure why that's here and not\nlater on in part II.\n\n> \n> Also maybe briefly mention the future plans - e.g. in the future, no\n> code will use find_hook() except are>, because all of them will use hook_exists() and run_hook().\n\nNice, good point - I'll do that.\n\n\n> \n> > +/*\n> > + * Returns 1 if any hooks are specified in the config or if a hook exists in the\n> > + * hookdir. Typically, invoke hook_exsts() like:\n> > + * hook_exists(hookname, configured_hookdir_opt());\n> > + * Like with run_hooks, if you take a --run-hookdir flag, reflect that\n> > + * user-specified behavior here instead.\n> > + */\n> > +int hook_exists(const char *hookname, enum hookdir_opt should_run_hookdir);\n> \n> I wonder if enum hookdir_opt should support a \"unspecified\" instead, in\n> which case hook_exists() will automatically read the config (instead of\n> relying on the caller to call configured_hookdir_opt()), but I see that\n> this patch set is version 7 and perhaps this design point has already\n> been discussed.\n\nNope, and I think that's a pretty neat idea. I've done so.\n"},{"id":"417419","messageId":"YDAeSwOx4MkCda9H@google.com","threadId":"54769","inReplyTo":"20210201053808.1309469-1-jonathantanmy@google.com","subject":"Re: [PATCH v7 11/17] run-command: allow stdin for run_processes_parallel","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-02-19T20:23:39Z","receivedAt":"2021-02-19T20:24:28Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Sun, Jan 31, 2021 at 09:38:08PM -0800, Jonathan Tan wrote:\n> \n> > diff --git a/run-command.c b/run-command.c\n> > index ea4d0fb4b1..80c8c97bc1 100644\n> > --- a/run-command.c\n> > +++ b/run-command.c\n> > @@ -1683,6 +1683,9 @@ static int pp_start_one(struct parallel_processes *pp)\n> > \tif (i == pp->max_processes)\n> > \t\tBUG(\"bookkeeping is hard\");\n> > \n> > +\t/* disallow by default, but allow users to set up stdin if they wish */\n> > +\tpp->children[i].process.no_stdin = 1;\n> > +\n> \n> This makes sense. May be worth a more detailed comment, e.g.:\n> \n> By default, do not inherit stdin from the parent process. (If not, all\n> children would share it!) Users may overwrite this by having the\n> get_next_task function assign 0 to no_stdin and an appropriate integer\n> to in.\n\nThanks, took it slightly modified:\n\n /*\n * By default, do not inherit stdin from the parent process - otherwise,\n * all children would share stdin! Users may overwrite this to provide\n * something to the child's stdin by having their 'get_next_task'\n * callback assign 0 to .no_stdin and an appropriate integer to .in.\n */\n\n - Emily\n"},{"id":"417498","messageId":"YDQmS2qN/lrm547L@google.com","threadId":"54769","inReplyTo":"20210201060422.1313603-1-jonathantanmy@google.com","subject":"Re: [PATCH v7 12/17] hook: allow parallel hook execution","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-02-22T21:46:51Z","receivedAt":"2021-02-22T21:47:45Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Sun, Jan 31, 2021 at 10:04:22PM -0800, Jonathan Tan wrote:\n> \n> > In many cases, there's no reason not to allow hooks to execute in\n> > parallel. run_processes_parallel() is well-suited - it's a task queue\n> > that runs its housekeeping in series, which means users don't\n> > need to worry about thread safety on their callback data. True\n> > multithreaded execution with the async_* functions isn't necessary here.\n> > Synchronous hook execution can be achieved by only allowing 1 job to run\n> > at a time.\n> > \n> > Teach run_hooks() to use that function for simple hooks which don't\n> > require stdin or capture of stderr.\n> \n> Which hooks would be run in parallel, and which hooks in series? I don't\n> see code that distinguishes between them.\n\nIt's up to the caller, who can set run_hooks_opt.jobs. In part II of\nthis series I made a guess at which ones should run in parallel or in\nseries and specified it in Documentation/githooks.txt.\n\n> \n> > \n> > Signed-off-by: Emily Shaffer \n> > ---\n> > \n> > Notes:\n> > Per AEvar's request - parallel hook execution on day zero.\n> > \n> > In most ways run_processes_parallel() worked great for me - but it didn't\n> > have great support for hooks where we pipe to and from. I had to add this\n> > support later in the series.\n> > \n> > Since I modified an existing and in-use library I'd appreciate a keen look on\n> > these patches.\n> \n> What is the existing and in-use library that you're modifying?\n\nHm, this note wasn't super specific. From this point onwards in the\nseries I make changes to the run-command.h:run_processes_parallel()\nlibrary, although not in this commit itself. I think I meant \"from here\non out, help me look at run-command.h\".\n\nI'll try to make the note a little better next series, sorry for the\nconfusion :) :)\n\n> \n> > @@ -246,11 +255,96 @@ void run_hooks_opt_clear(struct run_hooks_opt *o)\n> > \tstrvec_clear(&o->args);\n> > }\n> > \n> > +\n> > +static int pick_next_hook(struct child_process *cp,\n> > +\t\t\t struct strbuf *out,\n> > +\t\t\t void *pp_cb,\n> > +\t\t\t void **pp_task_cb)\n> > +{\n> > +\tstruct hook_cb_data *hook_cb = pp_cb;\n> > +\n> > +\tstruct hook *hook = list_entry(hook_cb->run_me, struct hook, list);\n> > +\n> > +\tif (hook_cb->head == hook_cb->run_me)\n> > +\t\treturn 0;\n> > +\n> > +\tcp->env = hook_cb->options->env.v;\n> > +\tcp->stdout_to_stderr = 1;\n> > +\tcp->trace2_hook_name = hook->command.buf;\n> > +\n> > +\t/* reopen the file for stdin; run_command closes it. */\n> > +\tif (hook_cb->options->path_to_stdin) {\n> > +\t\tcp->no_stdin = 0;\n> > +\t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n> > +\t} else {\n> > +\t\tcp->no_stdin = 1;\n> > +\t}\n> > +\n> > +\t/*\n> > +\t * Commands from the config could be oneliners, but we know\n> > +\t * for certain that hookdir commands are not.\n> > +\t */\n> > +\tif (hook->from_hookdir)\n> > +\t\tcp->use_shell = 0;\n> > +\telse\n> > +\t\tcp->use_shell = 1;\n> > +\n> > +\t/* add command */\n> > +\tstrvec_push(&cp->args, hook->command.buf);\n> > +\n> > +\t/*\n> > +\t * add passed-in argv, without expanding - let the user get back\n> > +\t * exactly what they put in\n> > +\t */\n> > +\tstrvec_pushv(&cp->args, hook_cb->options->args.v);\n> \n> I just skimmed over this setup-process-for-hook part - it would have\n> been much clearer if it was refactored into its own function before this\n> patch (or better yet, written as its own function in the first place).\n> As it is, there are some unnecessary rewritings - e.g. setting stdin\n> after env, and the use_shell setup.\n\nYeah, that makes sense. Will see if I can change it for next round :)\n\n> \n> > diff --git a/hook.h b/hook.h\n> \n> [snip]\n> \n> > +/*\n> > + * Callback provided to feed_pipe_fn and consume_sideband_fn.\n> > + */\n> > +struct hook_cb_data {\n> > +\tint rc;\n> > +\tstruct list_head *head;\n> > +\tstruct list_head *run_me;\n> > +\tstruct run_hooks_opt *options;\n> > +};\n> \n> Could this be in hook.c instead?\n\nIt ends up being needed publicly by\nhttps://lore.kernel.org/git/20201222000435.1529768-17-emilyshaffer@google.com\n(receive-pack: convert receive hooks to hook.h), which writes its own\nstdin provider callback. (In a later commit, a \"void* options\" gets\nadded to this struct.)\n\nAt that point it's needed because the run-command callback structure can\nprovide one context pointer for the overall work queue, and one context\npointer for the individual task; this one is the \"overall work queue\"\npointer.\n\nFrom hook.h's perspective, the entire hook_cb_data is needed for\npick_next_hook; but run-command.h:run_processes_parallel() doesn't have\na way to tease out a smaller amount of the context pointer for various\ncallbacks. If we wanted to obfuscate \"hook_cb_data\" we'd need to add\nanother indirection and call back to hook.h first, who could then tease\nout the client-provided context and then call the client callback, but\nto me it sounds unnecessarily complex.\n\n> \n> Also, I think it's clearer if run_me was a struct hook, and set to NULL\n> when iteration reaches the end. If you disagree, I think it needs some\n> documentation (e.g. \"the embedded linked list part of the hook that must\n> be run next; if equal to head, then iteration has ended\" or something\n> like that).\n\nYeah, I don't see a huge reason not to do that, sure.\n\n> \n> > +#define RUN_HOOKS_OPT_INIT_SYNC { \t\t\\\n> > \t.env = STRVEC_INIT, \t\t\t\\\n> > \t.args = STRVEC_INIT, \t\t\t\\\n> > \t.path_to_stdin = NULL,\t\t\t\\\n> > +\t.jobs = 1,\t\t\t\t\\\n> > \t.run_hookdir = configured_hookdir_opt()\t\\\n> > }\n> \n> This is not used anywhere.\n\nIt is used in part II by hooks which are not able to be parallelized.\n\n - Emily\n"},{"id":"417504","messageId":"YDRAbcqtWRpVn703@google.com","threadId":"54769","inReplyTo":"20210201065153.1322296-1-jonathantanmy@google.com","subject":"Re: [PATCH v7 14/17] run-command: add stdin callback for parallelization","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-02-22T23:38:21Z","receivedAt":"2021-02-22T23:39:11Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Sun, Jan 31, 2021 at 10:51:53PM -0800, Jonathan Tan wrote:\n> \n> > If a user of the run_processes_parallel() API wants to pipe a large\n> > amount of information to stdin of each parallel command, that\n> > information could exceed the buffer of the pipe allocated for that\n> > process's stdin. Generally this is solved by repeatedly writing to\n> > child_process.in between calls to start_command() and finish_command();\n> > run_processes_parallel() did not provide users an opportunity to access\n> > child_process at that time.\n> \n> [snip]\n> \n> > diff --git a/run-command.h b/run-command.h\n> > index 6472b38bde..e058c0e2c8 100644\n> > --- a/run-command.h\n> > +++ b/run-command.h\n> > @@ -436,6 +436,20 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n> > \t\t\t\tvoid *pp_cb,\n> > \t\t\t\tvoid *pp_task_cb);\n> > \n> > +/**\n> > + * This callback is called repeatedly on every child process who requests\n> > + * start_command() to create a pipe by setting child_process.in < 0.\n> > + *\n> > + * pp_cb is the callback cookie as passed into run_processes_parallel, and\n> > + * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> > + * The contents of 'send' will be read into the pipe and passed to the pipe.\n> > + *\n> > + * Return nonzero to close the pipe.\n> > + */\n> > +typedef int (*feed_pipe_fn)(struct strbuf *pipe,\n> > +\t\t\t void *pp_cb,\n> > +\t\t\t void *pp_task_cb);\n> > +\n> \n> As you mention above in the commit message, I think the clearest API to\n> support what we need is to just have a callback (that has access to\n> child_process) that is executed between process start and finish.\n> \n> As it is, I think this callback is too specific in that it takes a\n> struct strbuf. I think that this struct strbuf will just end up being\n> unnecessary copying much of the time, when the user could have just\n> written to the fd directly.\n\nSince the rest of the run_processes_parallel() API passes strings around\nwith strbufs, I'd prefer to leave it as-is to match the general API\nexpectations and style.\n\n - Emily\n"},{"id":"417588","messageId":"20210223193324.1660502-1-jonathantanmy@google.com","threadId":"54769","inReplyTo":"YDRAbcqtWRpVn703@google.com","subject":"Re: [PATCH v7 14/17] run-command: add stdin callback for parallelization","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-02-23T19:33:24Z","receivedAt":"2021-02-23T19:34:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > > +/**\n> > > + * This callback is called repeatedly on every child process who requests\n> > > + * start_command() to create a pipe by setting child_process.in < 0.\n> > > + *\n> > > + * pp_cb is the callback cookie as passed into run_processes_parallel, and\n> > > + * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> > > + * The contents of 'send' will be read into the pipe and passed to the pipe.\n> > > + *\n> > > + * Return nonzero to close the pipe.\n> > > + */\n> > > +typedef int (*feed_pipe_fn)(struct strbuf *pipe,\n> > > +\t\t\t void *pp_cb,\n> > > +\t\t\t void *pp_task_cb);\n> > > +\n> > \n> > As you mention above in the commit message, I think the clearest API to\n> > support what we need is to just have a callback (that has access to\n> > child_process) that is executed between process start and finish.\n> > \n> > As it is, I think this callback is too specific in that it takes a\n> > struct strbuf. I think that this struct strbuf will just end up being\n> > unnecessary copying much of the time, when the user could have just\n> > written to the fd directly.\n> \n> Since the rest of the run_processes_parallel() API passes strings around\n> with strbufs, I'd prefer to leave it as-is to match the general API\n> expectations and style.\n> \n> - Emily\n\nBy the rest of the run_processes_parallel() API, do you mean\nget_next_task_fn, start_failure_fn, and task_finished_fn? If yes, I\nthink that it makes sense for them to be strbuf, because buffering is\nneeded to avoid outputs from individual child processes interleaving,\nbut that's not true here.\n\nHaving said that, this is an internal API so we could just leave it\nas-is and then refactor it if we ever need something more flexible.\n"},{"id":"417591","messageId":"YDVdGNefUehBknRB@google.com","threadId":"54769","inReplyTo":"20210201070448.1325624-1-jonathantanmy@google.com","subject":"Re: [PATCH v7 15/17] hook: provide stdin by string_list or callback","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-02-23T19:52:56Z","receivedAt":"2021-02-23T19:54:01Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Sun, Jan 31, 2021 at 11:04:48PM -0800, Jonathan Tan wrote:\n> \n> > In cases where a hook requires only a small amount of information via\n> > stdin, it should be simple for users to provide a string_list alone. But\n> > in more complicated cases where the stdin is too large to hold in\n> > memory, let's provide a callback the users can populate line after line\n> > with instead.\n> \n> [snip]\n> \n> > diff --git a/hook.h b/hook.h\n> > index 8a7542610c..0ac83fa7ca 100644\n> > --- a/hook.h\n> > +++ b/hook.h\n> > @@ -2,6 +2,7 @@\n> > #include \"list.h\"\n> > #include \"strbuf.h\"\n> > #include \"strvec.h\"\n> > +#include \"run-command.h\"\n> > \n> > struct hook\n> > {\n> > @@ -14,6 +15,12 @@ struct hook\n> > \t/* The literal command to run. */\n> > \tstruct strbuf command;\n> > \tint from_hookdir;\n> > +\n> > +\t/*\n> > +\t * Use this to keep state for your feed_pipe_fn if you are using\n> > +\t * run_hooks_opt.feed_pipe. Otherwise, do not touch it.\n> > +\t */\n> > +\tvoid *feed_pipe_cb_data;\n> \n> When would we need per-hook state? I see in patch 14 that you give each\n> running process little by little (in pp_buffer_stdin()), perhaps so that\n> each hook can make progress at roughly the same pace, but I don't think\n> we can expect all hooks to work the same, so I don't think it's worth\n> complicating the design for all that.\n\nI agree that this is a complicated way of doing it, and if you have a\nbetter design I'd be really excited to hear it.\n\nIt seemed like this was what was necessary for hooks like\nhttps://lore.kernel.org/git/20201222000435.1529768-15-emilyshaffer@google.com\nwhere the hook and the invoking process talk back and forth, or like\nhttps://lore.kernel.org/git/20201222000435.1529768-17-emilyshaffer@google.com\nwhich generates stdin on the fly for hooks which cannot be parallelized\n(and so won't run at the same pace).\n\nThe former example - proc-receive - does have a constraint that multiple\nhooks can't be specified, so we could theoretically keep the old\nimplementation and just pick up the single hook's location from the new\nhook library. But the latter example still makes me think this much\ncomplexity is needed.\n\n> \n> > };\n> > \n> > /*\n> > @@ -57,12 +64,24 @@ struct run_hooks_opt\n> > \n> > \t/* Path to file which should be piped to stdin for each hook */\n> > \tconst char *path_to_stdin;\n> > +\t/* Pipe each string to stdin, separated by newlines */\n> > +\tstruct string_list str_stdin;\n> > +\t/*\n> > +\t * Callback and state pointer to ask for more content to pipe to stdin.\n> > +\t * Will be called repeatedly, for each hook. See\n> > +\t * hook.c:pipe_from_stdin() for an example. Keep per-hook state in\n> > +\t * hook.feed_pipe_cb_data (per process). Keep initialization context in\n> > +\t * feed_pipe_ctx (shared by all processes).\n> > +\t */\n> > +\tfeed_pipe_fn feed_pipe;\n> > +\tvoid *feed_pipe_ctx;\n> \n> Instead of 3 fields, I think 2 suffice - the function and the data\n> (called \"ctx\" here). We can supply a function that treats the data as a\n> string_list.\n\nNice catch, sure.\n\n - Emily\n"},{"id":"417840","messageId":"xmqq4ki0udu4.fsf@gitster.g","threadId":"54769","inReplyTo":"xmqqlfbmi99f.fsf@gitster.g","subject":"Re: [PATCH v7 00/17] propose config-based hooks (part I)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-25T19:50:11Z","receivedAt":"2021-02-25T19:53:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Josh Steadmon writes:\n>\n>>> The topic branch has a lot more commits than these 17; I am\n>>> wondering if the reviewed-by applies only to the bottom 17, or as\n>>> the whole? I recall that the upper half was expecting at least some\n>>> documentation updates.\n>>> \n>>> Thanks.\n>>\n>> Just to these 17, sorry for being unclear.\n>\n> Thanks for reading them through.\n>\n> I am tempted to say we should merge these \"mechanism\" part down to\n> 'next', hoping that the \"rewrite existing ones using the new\n> mechansim\" part can follow soon.\n\nI said this on Feb 17th, but since then I think I saw you answer\n\"I'll do that\" in responses to JTan's reviews in the past few days\n(e.g. ). Would I regret if I merge the\ntopic down to 'next' today?\n\nThanks.\n"},{"id":"417849","messageId":"20210225205612.2061342-1-jonathantanmy@google.com","threadId":"54769","inReplyTo":"YDVdGNefUehBknRB@google.com","subject":"Re: [PATCH v7 15/17] hook: provide stdin by string_list or callback","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-02-25T20:56:11Z","receivedAt":"2021-02-25T20:59:53Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > When would we need per-hook state? I see in patch 14 that you give each\n> > running process little by little (in pp_buffer_stdin()), perhaps so that\n> > each hook can make progress at roughly the same pace, but I don't think\n> > we can expect all hooks to work the same, so I don't think it's worth\n> > complicating the design for all that.\n> \n> I agree that this is a complicated way of doing it, and if you have a\n> better design I'd be really excited to hear it.\n> \n> It seemed like this was what was necessary for hooks like\n> https://lore.kernel.org/git/20201222000435.1529768-15-emilyshaffer@google.com\n> where the hook and the invoking process talk back and forth, or like\n> https://lore.kernel.org/git/20201222000435.1529768-17-emilyshaffer@google.com\n> which generates stdin on the fly for hooks which cannot be parallelized\n> (and so won't run at the same pace).\n> \n> The former example - proc-receive - does have a constraint that multiple\n> hooks can't be specified, so we could theoretically keep the old\n> implementation and just pick up the single hook's location from the new\n> hook library. But the latter example still makes me think this much\n> complexity is needed.\n\nAh, I see. From your explanation, in these 2 cases, only one hook\nexecutes at a time (in the former case, because there is only one hook,\nand in the latter case, you said that the hooks cannot be parallelized).\nSo it seems to me that the global state (in struct run_hooks_opt) would\nbe sufficient to keep track of what's going on. (The feed_pipe_fn\nfunction can use pp_cb to keep track of the last executing pp_task_cb\nand then compare it against the new pp_task_cb, I think, to keep track\nof when a new hook has started.)\n\nEven in the case of multiple hooks run in series (as opposed to a single\nhook), I would think that the reason they can't be run in parallel is\nbecause the nature of execution of a hook depends on what happened\nduring the execution of the previous hook, which seems to me to be even\nmore reason to centralize the state in struct run_hooks_opt.\n\nHaving said that, if my suggestion of not having per-hook state makes\ncertain patches more complicated, then that might be reason enough to\nhave per-hook state. In that case, you should write \"per-hook state,\nthough strictly not necessary, makes simpler\" (or something like\nthat) in the commit message.\n"},{"id":"418076","messageId":"YD1hfZqfu9bcMkra@google.com","threadId":"54769","inReplyTo":"xmqq4ki0udu4.fsf@gitster.g","subject":"Re: [PATCH v7 00/17] propose config-based hooks (part I)","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-03-01T21:51:31Z","receivedAt":"2021-03-01T21:54:52Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Feb 25, 2021 at 11:50:11AM -0800, Junio C Hamano wrote:\n> \n> Junio C Hamano writes:\n> \n> > Josh Steadmon writes:\n> >\n> >>> The topic branch has a lot more commits than these 17; I am\n> >>> wondering if the reviewed-by applies only to the bottom 17, or as\n> >>> the whole? I recall that the upper half was expecting at least some\n> >>> documentation updates.\n> >>> \n> >>> Thanks.\n> >>\n> >> Just to these 17, sorry for being unclear.\n> >\n> > Thanks for reading them through.\n> >\n> > I am tempted to say we should merge these \"mechanism\" part down to\n> > 'next', hoping that the \"rewrite existing ones using the new\n> > mechansim\" part can follow soon.\n> \n> I said this on Feb 17th, but since then I think I saw you answer\n> \"I'll do that\" in responses to JTan's reviews in the past few days\n> (e.g. ). Would I regret if I merge the\n> topic down to 'next' today?\n\nBah, I'm sorry I missed this - I had a broken mutt config and wasn't\nseeing replies, my own fault. Argh.\n\nI have some pretty significant changes from JTan's reviews, so I'd\nprefer if you would wait since it would be tricky to turn them into a\npatch commit now. But if you'd rather merge it and see a patch instead,\nthat is fine with me.\n\n - Emily\n"},{"id":"418082","messageId":"xmqqtupums98.fsf@gitster.c.googlers.com","threadId":"54769","inReplyTo":"YD1hfZqfu9bcMkra@google.com","subject":"Re: [PATCH v7 00/17] propose config-based hooks (part I)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-01T22:19:31Z","receivedAt":"2021-03-01T22:24:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Emily Shaffer writes:\n\n>> I said this on Feb 17th, but since then I think I saw you answer\n>> \"I'll do that\" in responses to JTan's reviews in the past few days\n>> (e.g. ). Would I regret if I merge the\n>> topic down to 'next' today?\n>\n> Bah, I'm sorry I missed this - I had a broken mutt config and wasn't\n> seeing replies, my own fault. Argh.\n>\n> I have some pretty significant changes from JTan's reviews, so I'd\n> prefer if you would wait since it would be tricky to turn them into a\n> patch commit now. But if you'd rather merge it and see a patch instead,\n> that is fine with me.\n\nOK, I still have it outside 'next', I think.\n"},{"id":"418094","messageId":"YD2ZQIRfcwGzpFuQ@google.com","threadId":"54769","inReplyTo":"20210225205612.2061342-1-jonathantanmy@google.com","subject":"Re: [PATCH v7 15/17] hook: provide stdin by string_list or callback","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-03-02T01:47:44Z","receivedAt":"2021-03-02T15:28:22Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Feb 25, 2021 at 12:56:11PM -0800, Jonathan Tan wrote:\n> \n> > > When would we need per-hook state? I see in patch 14 that you give each\n> > > running process little by little (in pp_buffer_stdin()), perhaps so that\n> > > each hook can make progress at roughly the same pace, but I don't think\n> > > we can expect all hooks to work the same, so I don't think it's worth\n> > > complicating the design for all that.\n> > \n> > I agree that this is a complicated way of doing it, and if you have a\n> > better design I'd be really excited to hear it.\n> > \n> > It seemed like this was what was necessary for hooks like\n> > https://lore.kernel.org/git/20201222000435.1529768-15-emilyshaffer@google.com\n> > where the hook and the invoking process talk back and forth, or like\n> > https://lore.kernel.org/git/20201222000435.1529768-17-emilyshaffer@google.com\n> > which generates stdin on the fly for hooks which cannot be parallelized\n> > (and so won't run at the same pace).\n> > \n> > The former example - proc-receive - does have a constraint that multiple\n> > hooks can't be specified, so we could theoretically keep the old\n> > implementation and just pick up the single hook's location from the new\n> > hook library. But the latter example still makes me think this much\n> > complexity is needed.\n> \n> Ah, I see. From your explanation, in these 2 cases, only one hook\n> executes at a time (in the former case, because there is only one hook,\n> and in the latter case, you said that the hooks cannot be parallelized).\n> So it seems to me that the global state (in struct run_hooks_opt) would\n> be sufficient to keep track of what's going on. (The feed_pipe_fn\n> function can use pp_cb to keep track of the last executing pp_task_cb\n> and then compare it against the new pp_task_cb, I think, to keep track\n> of when a new hook has started.)\n> \n> Even in the case of multiple hooks run in series (as opposed to a single\n> hook), I would think that the reason they can't be run in parallel is\n> because the nature of execution of a hook depends on what happened\n> during the execution of the previous hook, which seems to me to be even\n> more reason to centralize the state in struct run_hooks_opt.\n> \n> Having said that, if my suggestion of not having per-hook state makes\n> certain patches more complicated, then that might be reason enough to\n> have per-hook state. In that case, you should write \"per-hook state,\n> though strictly not necessary, makes simpler\" (or something like\n> that) in the commit message.\n\nJonathan and I discussed this a little more offline and agreed to leave\nthe implementation as is.\n\nJonathan had suggested \"have one callback invocation apply to all hooks\nthat are running now\", either by having the callback iterate over the\ntask queue or by having the run-command lib take the result from the\ncallback and have *that* iterate over the task queue. The idea being,\none pointer to one copy of source material is easier to handle than\nmany.\n\nI suggested that the callback's implementation of the second version of\nthat, where the library takes care of the \"and do it for each task in\nprogress\" part, would be pretty much identical to the callback's\nimplementation as it is in this patch, except that as it is here the\ncontext pointer is per-task and as Jonathan suggests the context pointer\nis per-entire-hook-invocation - so there isn't much complexity\ndifference between the two, from the user's perspective.\n\nWe also talked about cases where N=# of hooks > M=# of jobs, that is,\nwhere some hooks must wait for other hooks to finish executing before\nthat could start. In this case, users' callback implementations would\nneed to be able to start over from the beginning of the source material,\nand a long-running hook would block other short-running hooks from\nbeginning (because the long-running hook would be confused by hearing\nthe source material to its stdin again).\n\nHopefully this diagram illustrates better based on my understanding of\nJonathan's suggestion:\n\n A B <- \"hey everyone, stdin 'foo'\"\n A B <- \"hey everyone, stdin 'bar'\"\n A B <- \"hey everyone, stdin 'baz'\"\n B\n B\n B\n B\n C <- \"hey everyone, stdin 'foo'\"\n C <- \"hey everyone, stdin 'bar'\"\n C <- \"hey everyone, stdin 'baz'\"\n C\n\n Anyway, since the complexity is probably about the same to the end user\n and using per-hook context means we don't have to wait like this, we\n agreed to stick with the implementation as is.\n\n - Emily\n"},{"id":"418120","messageId":"20210302233314.43539-1-jonathantanmy@google.com","threadId":"54769","inReplyTo":"YD2ZQIRfcwGzpFuQ@google.com","subject":"Re: [PATCH v7 15/17] hook: provide stdin by string_list or callback","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-03-02T23:33:14Z","receivedAt":"2021-03-03T06:42:13Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Jonathan and I discussed this a little more offline and agreed to leave\n> the implementation as is.\n> \n> Jonathan had suggested \"have one callback invocation apply to all hooks\n> that are running now\", either by having the callback iterate over the\n> task queue or by having the run-command lib take the result from the\n> callback and have *that* iterate over the task queue. The idea being,\n> one pointer to one copy of source material is easier to handle than\n> many.\n> \n> I suggested that the callback's implementation of the second version of\n> that, where the library takes care of the \"and do it for each task in\n> progress\" part, would be pretty much identical to the callback's\n> implementation as it is in this patch, except that as it is here the\n> context pointer is per-task and as Jonathan suggests the context pointer\n> is per-entire-hook-invocation - so there isn't much complexity\n> difference between the two, from the user's perspective.\n> \n> We also talked about cases where N=# of hooks > M=# of jobs, that is,\n> where some hooks must wait for other hooks to finish executing before\n> that could start. In this case, users' callback implementations would\n> need to be able to start over from the beginning of the source material,\n> and a long-running hook would block other short-running hooks from\n> beginning (because the long-running hook would be confused by hearing\n> the source material to its stdin again).\n\nYes - this (number of hooks greater than number of jobs allowed to run\nin parallel) was the case in which my suggestion of not having\nhook-specific state would not work. The case we were talking about is\nwhen there's a large amount of dynamically-generated data to be\ntransmitted to the hooks' stdins and I was thinking that it would be\nbest anyway if the callback looped over all hooks as data was generated,\nbut it would not be possible to only do a single pass if the number of\nhooks is greater than the number of jobs.\n"},{"id":"418698","messageId":"YEkO1CGw0nD7/69w@google.com","threadId":"54769","inReplyTo":"20210223193324.1660502-1-jonathantanmy@google.com","subject":"Re: [PATCH v7 14/17] run-command: add stdin callback for parallelization","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-03-10T18:24:20Z","receivedAt":"2021-03-10T18:25:03Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Feb 23, 2021 at 11:33:24AM -0800, Jonathan Tan wrote:\n> \n> > > > +/**\n> > > > + * This callback is called repeatedly on every child process who requests\n> > > > + * start_command() to create a pipe by setting child_process.in < 0.\n> > > > + *\n> > > > + * pp_cb is the callback cookie as passed into run_processes_parallel, and\n> > > > + * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> > > > + * The contents of 'send' will be read into the pipe and passed to the pipe.\n> > > > + *\n> > > > + * Return nonzero to close the pipe.\n> > > > + */\n> > > > +typedef int (*feed_pipe_fn)(struct strbuf *pipe,\n> > > > +\t\t\t void *pp_cb,\n> > > > +\t\t\t void *pp_task_cb);\n> > > > +\n> > > \n> > > As you mention above in the commit message, I think the clearest API to\n> > > support what we need is to just have a callback (that has access to\n> > > child_process) that is executed between process start and finish.\n> > > \n> > > As it is, I think this callback is too specific in that it takes a\n> > > struct strbuf. I think that this struct strbuf will just end up being\n> > > unnecessary copying much of the time, when the user could have just\n> > > written to the fd directly.\n> > \n> > Since the rest of the run_processes_parallel() API passes strings around\n> > with strbufs, I'd prefer to leave it as-is to match the general API\n> > expectations and style.\n> > \n> > - Emily\n> \n> By the rest of the run_processes_parallel() API, do you mean\n> get_next_task_fn, start_failure_fn, and task_finished_fn? If yes, I\n> think that it makes sense for them to be strbuf, because buffering is\n> needed to avoid outputs from individual child processes interleaving,\n> but that's not true here.\n> \n> Having said that, this is an internal API so we could just leave it\n> as-is and then refactor it if we ever need something more flexible.\n\nYeah, with that in mind I'll leave it as it is. I don't like the idea of\ndirectly exposing the child's pipe via callback; to me it feels like bad\nobject-oriented design, but that maybe doesn't apply here :)\n\nAnyway, like you say, we can change it later if someone doesn't like\nthis, no problem.\n"},{"id":"418703","messageId":"YEkeTB5MPjXJQRrO@google.com","threadId":"54769","inReplyTo":"xmqqa6snjv4m.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v7 01/17] doc: propose hooks managed by the config","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-03-10T19:30:20Z","receivedAt":"2021-03-10T19:31:12Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Mon, Feb 01, 2021 at 02:11:53PM -0800, Junio C Hamano wrote:\n> \n> Emily Shaffer writes:\n> \n> > +ways, providing an avenue to deprecate these \"legacy\" hooks if desired. The\n> > +handling is based on a config `hook.runHookDir`, which is checked against a\n> > +number of cases:\n> \n> Don't we want to also warn when the setting \"no\" or something\n> similar prevents the legacy hook from running, to help users\n> who wonder why their hook scripts are not running? I.e.\n> \n> > +- \"no\": the legacy hook will not be run\n> \n> +- \"warn-no\": Git will print a warning to stderr before ignoring the\n> + legacy hook\n\nYeah, I think you are right. Jonathan N suggested such an option as\n\"error\" (as opposed to warning); I'll add it here plus to the enum and\ntake your description verbatim. Thanks.\n\n> \n> > +- \"interactive\": Git will prompt the user before running the legacy hook\n> > +- \"warn\": Git will print a warning to stderr before running the legacy hook\n> > +- \"yes\" (default): Git will silently run the legacy hook\n> \n> > +In case this list is expanded in the future, if a value for `hook.runHookDir` is\n> > +given which Git does not recognize, Git should discard that config entry. For\n> > +example, if \"warn\" was specified at system level and \"junk\" was specified at\n> > +global level, Git would resolve the value to \"warn\"; if the only time the config\n> > +was set was to \"junk\", Git would use the default value of \"yes\".\n> \n> Hmph, instead of complaining \"value 'junk' is not recognized\" and\n> erroring out? Why?\n\nI think for the exact case you're describing above - where I forgot some\nuseful combination of \"run/don't run\" and \"warn/don't warn\" (or, one\ncould forsee, \"first\" e.g. to run the legacy hook early on instead of\nlate, or \"only\" e.g. config hooks don't exist, etc etc), we add this\nflag, and some enterprise (like Jonathan N's team) distributes the new\nconfig to folks' system configs before 100% of machines are using the\nnewer Git executable. It's a long shot :) but I'd rather be flexible\nthan inflexible.\n\nI will update the design doc anyway - I ended up implementing it as\n\"complain about 'junk' and then do default behavior\".\n\n> \n> > +[[stage-3]]\n> > +==== Stage 3\n> > +\n> > +`.git/hooks` is removed from the template and the hook directory is considered\n> > +deprecated. To avoid breaking older repos, the default of `hook.runHookDir` is\n> > +not changed, and `find_hook()` is not removed.\n> \n> Presumably, we'll have documentation somewhere that instructs users\n> (who were taught by slashdot and other site to add certain scripts\n> under their .git/hooks/) how to do the equivalent without adding\n> scripts in .git/hooks/ directory and instead using the config\n> mechanism (e.g. \"when told to add script X in .git/hooks/, read such\n> an instruction as if telling you to do Y instead\") by the time this\n> happens? It probably makes sense to do so as part of stage-2, at\n> which point the users are _ready_ to migrate.\n\nHm. Where should we distribute such a documentation? git-hook.txt?\ngithooks.txt? I think it's a good idea, sure.\n\n> \n> > +[[security]]\n> > +=== Security and repo config\n> > +\n> > +Part of the motivation behind this refactor is to mitigate hooks as an attack\n> > +vector;footnote:[https://lore.kernel.org/git/20171002234517.GV19555@aiede.mtv.corp.google.com/]\n> > +however, as the design stands, users can still provide hooks in the repo-level\n> > +config, which is included when a repo is zipped and sent elsewhere. The\n> > +security of the repo-level config is still under discussion; this design\n> > +generally assumes the repo-level config is secure, which is not true yet. The\n> > +goal is to avoid an overcomplicated design to work around a problem which has\n> > +ceased to exist.\n> \n> I doubt we want to claim anything about security as part of this\n> series. As you say in the paragraph, .git/config and .git/hooks/\n> are equally (un)protected and if we decide to punt .git/config\n> security, then not moving away from .git/hooks would not hurt\n> security-wise, either (in other words, security is not a viable\n> motivation behind this series).\n\nYeah, I think you are right that despite my best efforts to spin it that\nway, it just isn't a security win :) Ah well...\n\n> \n> And if we stop advertising 'security merit' that does not exist,\n> what remains? Isn't the biggest selling point that an identical set\n> of hook configuration can be shared among multiple repositories, and\n> it allows more than one hook scripts to be triggered by a single\n> \"hook event\"? There may be other good things we should be able to\n> sell the new mechanism to our users, and we do stress on them, which\n> is done in the motivation section. So...\n\nOk.\n\nI would like to keep this header and make it more clear that we didn't\nfully succeed in the wish to make zip attacks impossible, but I will\nremove it from the motivations section at the top. I also added a little\nmore clarity (I hope) on the other pieces in the motivation section.\n\n> \n> > +.Comparison of alternatives\n> > +|===\n> > +|Feature |Config-based hooks |Hook directories |Status quo\n> \n> Sorry, but I did not find this table particularly convincing.\n> \n> The only thing I sense is a hand-wavy desire that \"we could make it\n> better than everybody else if we work on it in this area\", which can\n> apply equally for other approaches---they could enhance what they\n> already have (e.g. \"discoverability & documentation\").\n> \n> As a list of \"these are the points we aspire to do better than other\n> people\", I think it is an excellent idea to have a table like this\n> here in the documentation. But that is not a \"comparison\".\n\nOk. I'll see what I can do to frame it better.\n\n> \n> > +[[execution-ordering]]\n> > +=== Execution ordering\n> > +\n> > +We may find that config order is insufficient for some users; for example,\n> > +config order makes it difficult to add a new hook to the system or global config\n> > +which runs at the end of the hook list. A new ordering schema should be:\n> > +\n> > +1) Specified by a `hook.order` config, so that users will not unexpectedly see\n> > +their order change;\n> > +\n> > +2) Either dependency or numerically based.\n> > +\n> > +Dependency-based ordering is prone to classic linked-list problems, like a\n> > +cycles and handling of missing dependencies. But, it paves the way for enabling\n> > +parallelization if some tasks truly depend on others.\n> > +\n> > +Numerical ordering makes it tricky for Git to generate suggested ordering\n> > +numbers for each command, but is easy to determine a definitive order.\n> \n> OK.\n> \n> Have we decided what we do for hooks whose interface is to feed\n> their input from their standard input? The current system, I think,\n> just feeds the single hook by writing into a pipe to it, but if we\n> were to drive multiple hooks, we'd need to write the same thing to\n> each of these hook programs? \n\nYep. There are a few ways to do it:\n1. Provide a file to be used as stdin. This way we just hook up that\nfile's fd to each hook process's stdin fd. (Example:\nhttps://lore.kernel.org/git/20201222000435.1529768-11-emilyshaffer@google.com\nin builtin/am.c:run_post_rewrite_hook())\n2. Provide a string_list in the run_hooks_opt struct; hook.c treats each\nentry in the string_list as a line to stdin, separated by a newline, and\n\"replays\" the list to each hook in order. (Example:\nhttps://lore.kernel.org/git/20201222000435.1529768-12-emilyshaffer@google.com)\n3. Set up your own more complicated version of (2) by writing your own\ncallback to provide \"next line of stdin\" based on a context pointer. The\ncontext pointer is per-hook-task. (Example:\nhttps://lore.kernel.org/git/20201222000435.1529768-17-emilyshaffer@google.com)\n\n> \n> Do we have a plan to deal with hooks whose outcome is not just\n> \"yes/no\", e.g. \"proc-receive\" hook that munges the list of refs to\n> be updated and the new values for them, or \"applypatch-msg\" that\n> munges the incoming proposed commit log message? Does the second\n> hook work on the result of the first hook? Do the two hooks work on\n> the vanilla state and their output have to agree with each other?\n\nThe only hook I found this way was 'proc-receive' itself - and in the\nend, because it's somewhat interactive with two-way communication with\nthe caller, it didn't seem like there was a good way to reason about\nmultiple hooks, like you say. So in the 'proc-receive' case, we disallow\nmultiple hooks. See\nhttps://lore.kernel.org/git/20201222000435.1529768-15-emilyshaffer@google.com.\n> \n> > +[[parallelization]]\n> > +=== Parallelization with dependencies\n> > +\n> > +Currently hooks use a naive parallelization scheme or are run in series. But if\n> > +one hook depends on another's output, then users will want to specify those\n> > +dependencies.\n> \n> An untold assumption here is that the questions I asked earlier on\n> having more than one hooks that is not just yes/no is something\n> readers know the same answer, and that answer is \"the outcome of the\n> first hook is passed along (as if it were the input given by Git\n> directly, if the first hook did not exist), to the second hook. It\n> should be spelled out somewhere before the execution ordering\n> section, I think.\n\nI'll make an explicit callout about that case, thanks.\n\n - Emily\n"}]}