{"thread":{"id":"57764","subject":"git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty","startedAt":"2022-04-19T19:00:17Z","lastAt":"2022-06-17T00:07:11Z","messageCount":85,"participants":["Anthony Sottile","Emily Shaffer","Junio C Hamano","Phillip Wood","Ævar Arnfjörð Bjarmason","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"453937","messageId":"CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com","threadId":"57764","inReplyTo":null,"subject":"git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty","fromName":"Anthony Sottile","fromEmail":"asottile@umich.edu","sentAt":"2022-04-19T18:59:36Z","receivedAt":"2022-04-19T19:00:17Z","isPatch":false,"sender":{"key":"asottile@umich.edu","avatar":"https://avatars.githubusercontent.com/u/1810591?v=4"},"body":"here's the shortest reproduction --\n\n```console\n$ cat ../testrepo/.git/hooks/pre-commit\n#!/usr/bin/env bash\nif [ -t 1 ]; then\n echo GOOD\nfi\n```\n\nin previous git versions:\n\n```\n$ git commit -q --allow-empty -m foo\nGOOD\n$\n```\n\nwith git 2.36.0:\n\n````\n$ git commit -q --allow-empty -m foo\n$\n```\n\nwhy I care: I maintain a git hooks framework which uses `isatty` to\ndetect whether it's appropriate to color the output. many tools\nutilize the same check. in 2.36.0+ isatty is false for stdout and\nstderr causing coloring to be turned off.\n\nI bisected this (it was a little complicated, needed to force a pty):\n\n`../testrepo`: a git repo set up with the hook above\n\n`../bisect.sh`:\n\n```bash\n#!/usr/bin/env bash\nset -eux\ngit clean -fxfd >& /dev/null\nmake -j6 prefix=\"$PWD/prefix\" NO_GETTEXT=1 NO_TCLTK=1 install >& /dev/null\nexport PATH=\"$PWD/prefix/bin:$PATH\"\ncd ../testrepo\n(../pty git commit -q --allow-empty -m foo || true) | grep GOOD\n```\n\n`../pty`:\n\n```python\n#!/usr/bin/env python3\nimport errno\nimport os\nimport subprocess\nimport sys\n\nx: int = 'nope'\n\n\nclass Pty(object):\n def __init__(self):\n self.r = self.w = None\n\n def __enter__(self):\n self.r, self.w = os.openpty()\n\n return self\n\n def close_w(self):\n if self.w is not None:\n os.close(self.w)\n self.w = None\n\n def close_r(self):\n assert self.r is not None\n os.close(self.r)\n self.r = None\n\n def __exit__(self, exc_type, exc_value, traceback):\n self.close_w()\n self.close_r()\n\n\ndef cmd_output_p(*cmd, **kwargs):\n with open(os.devnull) as devnull, Pty() as pty:\n kwargs = {'stdin': devnull, 'stdout': pty.w, 'stderr': pty.w}\n proc = subprocess.Popen(cmd, **kwargs)\n pty.close_w()\n\n buf = b''\n while True:\n try:\n bts = os.read(pty.r, 4096)\n except OSError as e:\n if e.errno == errno.EIO:\n bts = b''\n else:\n raise\n else:\n buf += bts\n if not bts:\n break\n\n return proc.wait(), buf, None\n\n\nif __name__ == '__main__':\n _, buf, _ = cmd_output_p(*sys.argv[1:])\n sys.stdout.buffer.write(buf)\n```\n\nthe first commit it points out:\n\n```\nf443246b9f29b815f0b98a07bb2d425628ae6522 is the first bad commit\ncommit f443246b9f29b815f0b98a07bb2d425628ae6522\nAuthor: Emily Shaffer \nDate: Wed Dec 22 04:59:40 2021 +0100\n\n commit: convert {pre-commit,prepare-commit-msg} hook to hook.h\n\n Move these hooks hook away from run-command.h to and over to the new\n hook.h library.\n\n Signed-off-by: Emily Shaffer \n Signed-off-by: Ævar Arnfjörð Bjarmason \n Acked-by: Emily Shaffer \n Signed-off-by: Junio C Hamano \n\n commit.c | 15 ++++++++-------\n 1 file changed, 8 insertions(+), 7 deletions(-)\nbisect run success\n```\n\n\nAnthony\n"},{"id":"453945","messageId":"Yl9Hn0C0TwalASC0@google.com","threadId":"57764","inReplyTo":"CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com","subject":"Re: git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2022-04-19T23:37:03Z","receivedAt":"2022-04-19T23:37:50Z","isPatch":false,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Apr 19, 2022 at 02:59:36PM -0400, Anthony Sottile wrote:\n> \n> here's the shortest reproduction --\n> \n> ```console\n> $ cat ../testrepo/.git/hooks/pre-commit\n> #!/usr/bin/env bash\n> if [ -t 1 ]; then\n> echo GOOD\n> fi\n> ```\n> \n> in previous git versions:\n> \n> ```\n> $ git commit -q --allow-empty -m foo\n> GOOD\n> $\n> ```\n> \n> with git 2.36.0:\n> \n> ````\n> $ git commit -q --allow-empty -m foo\n> $\n> ```\n> \n> why I care: I maintain a git hooks framework which uses `isatty` to\n> detect whether it's appropriate to color the output. many tools\n> utilize the same check. in 2.36.0+ isatty is false for stdout and\n> stderr causing coloring to be turned off.\n> \n> I bisected this (it was a little complicated, needed to force a pty):\n> \n> `../testrepo`: a git repo set up with the hook above\n> \n> `../bisect.sh`:\n> \n> ```bash\n> #!/usr/bin/env bash\n> set -eux\n> git clean -fxfd >& /dev/null\n> make -j6 prefix=\"$PWD/prefix\" NO_GETTEXT=1 NO_TCLTK=1 install >& /dev/null\n> export PATH=\"$PWD/prefix/bin:$PATH\"\n> cd ../testrepo\n> (../pty git commit -q --allow-empty -m foo || true) | grep GOOD\n> ```\n> \n> `../pty`:\n> \n> ```python\n> #!/usr/bin/env python3\n> import errno\n> import os\n> import subprocess\n> import sys\n> \n> x: int = 'nope'\n> \n> \n> class Pty(object):\n> def __init__(self):\n> self.r = self.w = None\n> \n> def __enter__(self):\n> self.r, self.w = os.openpty()\n> \n> return self\n> \n> def close_w(self):\n> if self.w is not None:\n> os.close(self.w)\n> self.w = None\n> \n> def close_r(self):\n> assert self.r is not None\n> os.close(self.r)\n> self.r = None\n> \n> def __exit__(self, exc_type, exc_value, traceback):\n> self.close_w()\n> self.close_r()\n> \n> \n> def cmd_output_p(*cmd, **kwargs):\n> with open(os.devnull) as devnull, Pty() as pty:\n> kwargs = {'stdin': devnull, 'stdout': pty.w, 'stderr': pty.w}\n> proc = subprocess.Popen(cmd, **kwargs)\n> pty.close_w()\n> \n> buf = b''\n> while True:\n> try:\n> bts = os.read(pty.r, 4096)\n> except OSError as e:\n> if e.errno == errno.EIO:\n> bts = b''\n> else:\n> raise\n> else:\n> buf += bts\n> if not bts:\n> break\n> \n> return proc.wait(), buf, None\n> \n> \n> if __name__ == '__main__':\n> _, buf, _ = cmd_output_p(*sys.argv[1:])\n> sys.stdout.buffer.write(buf)\n> ```\n> \n> the first commit it points out:\n> \n> ```\n> f443246b9f29b815f0b98a07bb2d425628ae6522 is the first bad commit\n> commit f443246b9f29b815f0b98a07bb2d425628ae6522\n> Author: Emily Shaffer \n> \n> commit: convert {pre-commit,prepare-commit-msg} hook to hook.h\n> \n> Move these hooks hook away from run-command.h to and over to the new\n> hook.h library.\n> \n> Signed-off-by: Emily Shaffer \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Acked-by: Emily Shaffer \n> Signed-off-by: Junio C Hamano \n> \n> commit.c | 15 ++++++++-------\n> 1 file changed, 8 insertions(+), 7 deletions(-)\n> bisect run success\n> ```\n\nInteresting. I'm surprised to see the tty-ness of hooks changing with\nthis patch, as the way the hook is called is pretty much the same:\n\nrun_hook_ve() (\"the old way\") sets no_stdin, stdout_to_stderr, args,\nenvvars, and some trace variables, and then runs 'run_command()';\nrun_command() invokes start_command().\n\nrun_hooks_opt (\"the new way\") ultimately kicks off the hook with a\ncallback that sets up a child_process with no_stdin, stdout_to_stderr,\nargs, envvars, and some trace variables (hook.c:pick_next_hook); the\ntask queue manager also sets .err to -1 on that child_process; then it\ncalls start_command() directly (run-command.c:pp_start_one()).\n\nI'm not sure I see why the tty-ness would change between the two. If I'm\nbeing honest, I'm actually slightly surprised that `isatty` returned\ntrue for your hook before - since the hook process is a child of Git and\nits output is, presumably, being consumed by Git first rather than by an\ninteractive user shell.\n\nI suppose that with stdout_to_stderr being set, the tty-ness of the main\nprocess's stderr would then apply to the child process's stdout (we do\nthat by calling `dup(2)`). But that's being set in both \"the old way\"\nand \"the new way\", so I'm pretty surprised to see a change here.\n\nIt *is* true that run-command.c:pp_start_one() sets child_process:err=-1\nfor the child and run-command.c:run_hook_ve() didn't do that; that -1\nmeans that start_command() will create a new fd for the child's stderr.\nSince run_hook_ve() didn't care about the child's stderr before, I\nwonder if that is why? Could it be that now that we're processing the\nchild's stderr, the child no longer thinks stderr is in tty, because the\nparent is consuming its output?\n\nI think if that's the case, a fix would involve\nrun-command.c:pp_start_one() not setting .err, .stdout_to_stderr, or\n.no_stdin at all on its own, and relying on the 'get_next_task' callback\nto set those things. It's a little more painful than I initially thought\nbecause the run_processes_parallel() library depends on that err capture\nto run pp_buffer_stderr() unconditionally; I guess it needs a tiny bit\nof shim logic to deal with callers who don't care to see their\nchildren's stderr.\n\nAll that said.... I'd expect that the dup() from the child's stdout to\nthe parent's stderr would still result in a happy isatty(1). So I'm not\nconvinced this is actually the right solution.... From your repro\nscript, I can't quite tell which fd the isatty call is against (to be\nhonest, I can't find the isatty call, either). So maybe I'm going the\nwrong direction :)\n\n - Emily\n"},{"id":"453946","messageId":"CA+dzEBntTx++n0QVcd3KHr_ri5Vmo4wEqY4_BBg8zuT7R4e7-Q@mail.gmail.com","threadId":"57764","inReplyTo":"Yl9Hn0C0TwalASC0@google.com","subject":"Re: git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty","fromName":"Anthony Sottile","fromEmail":"asottile@umich.edu","sentAt":"2022-04-19T23:52:15Z","receivedAt":"2022-04-19T23:52:30Z","isPatch":false,"sender":{"key":"asottile@umich.edu","avatar":"https://avatars.githubusercontent.com/u/1810591?v=4"},"body":"On Tue, Apr 19, 2022 at 7:37 PM Emily Shaffer wrote:\n>\n> On Tue, Apr 19, 2022 at 02:59:36PM -0400, Anthony Sottile wrote:\n> >\n> > here's the shortest reproduction --\n> >\n> > ```console\n> > $ cat ../testrepo/.git/hooks/pre-commit\n> > #!/usr/bin/env bash\n> > if [ -t 1 ]; then\n> > echo GOOD\n> > fi\n> > ```\n> >\n> > in previous git versions:\n> >\n> > ```\n> > $ git commit -q --allow-empty -m foo\n> > GOOD\n> > $\n> > ```\n> >\n> > with git 2.36.0:\n> >\n> > ````\n> > $ git commit -q --allow-empty -m foo\n> > $\n> > ```\n> >\n> > why I care: I maintain a git hooks framework which uses `isatty` to\n> > detect whether it's appropriate to color the output. many tools\n> > utilize the same check. in 2.36.0+ isatty is false for stdout and\n> > stderr causing coloring to be turned off.\n> >\n> > I bisected this (it was a little complicated, needed to force a pty):\n> >\n> > `../testrepo`: a git repo set up with the hook above\n> >\n> > `../bisect.sh`:\n> >\n> > ```bash\n> > #!/usr/bin/env bash\n> > set -eux\n> > git clean -fxfd >& /dev/null\n> > make -j6 prefix=\"$PWD/prefix\" NO_GETTEXT=1 NO_TCLTK=1 install >& /dev/null\n> > export PATH=\"$PWD/prefix/bin:$PATH\"\n> > cd ../testrepo\n> > (../pty git commit -q --allow-empty -m foo || true) | grep GOOD\n> > ```\n> >\n> > `../pty`:\n> >\n> > ```python\n> > #!/usr/bin/env python3\n> > import errno\n> > import os\n> > import subprocess\n> > import sys\n> >\n> > x: int = 'nope'\n> >\n> >\n> > class Pty(object):\n> > def __init__(self):\n> > self.r = self.w = None\n> >\n> > def __enter__(self):\n> > self.r, self.w = os.openpty()\n> >\n> > return self\n> >\n> > def close_w(self):\n> > if self.w is not None:\n> > os.close(self.w)\n> > self.w = None\n> >\n> > def close_r(self):\n> > assert self.r is not None\n> > os.close(self.r)\n> > self.r = None\n> >\n> > def __exit__(self, exc_type, exc_value, traceback):\n> > self.close_w()\n> > self.close_r()\n> >\n> >\n> > def cmd_output_p(*cmd, **kwargs):\n> > with open(os.devnull) as devnull, Pty() as pty:\n> > kwargs = {'stdin': devnull, 'stdout': pty.w, 'stderr': pty.w}\n> > proc = subprocess.Popen(cmd, **kwargs)\n> > pty.close_w()\n> >\n> > buf = b''\n> > while True:\n> > try:\n> > bts = os.read(pty.r, 4096)\n> > except OSError as e:\n> > if e.errno == errno.EIO:\n> > bts = b''\n> > else:\n> > raise\n> > else:\n> > buf += bts\n> > if not bts:\n> > break\n> >\n> > return proc.wait(), buf, None\n> >\n> >\n> > if __name__ == '__main__':\n> > _, buf, _ = cmd_output_p(*sys.argv[1:])\n> > sys.stdout.buffer.write(buf)\n> > ```\n> >\n> > the first commit it points out:\n> >\n> > ```\n> > f443246b9f29b815f0b98a07bb2d425628ae6522 is the first bad commit\n> > commit f443246b9f29b815f0b98a07bb2d425628ae6522\n> > Author: Emily Shaffer \n> >\n> > commit: convert {pre-commit,prepare-commit-msg} hook to hook.h\n> >\n> > Move these hooks hook away from run-command.h to and over to the new\n> > hook.h library.\n> >\n> > Signed-off-by: Emily Shaffer \n> > Signed-off-by: Ævar Arnfjörð Bjarmason \n> > Acked-by: Emily Shaffer \n> > Signed-off-by: Junio C Hamano \n> >\n> > commit.c | 15 ++++++++-------\n> > 1 file changed, 8 insertions(+), 7 deletions(-)\n> > bisect run success\n> > ```\n>\n> Interesting. I'm surprised to see the tty-ness of hooks changing with\n> this patch, as the way the hook is called is pretty much the same:\n>\n> run_hook_ve() (\"the old way\") sets no_stdin, stdout_to_stderr, args,\n> envvars, and some trace variables, and then runs 'run_command()';\n> run_command() invokes start_command().\n>\n> run_hooks_opt (\"the new way\") ultimately kicks off the hook with a\n> callback that sets up a child_process with no_stdin, stdout_to_stderr,\n> args, envvars, and some trace variables (hook.c:pick_next_hook); the\n> task queue manager also sets .err to -1 on that child_process; then it\n> calls start_command() directly (run-command.c:pp_start_one()).\n>\n> I'm not sure I see why the tty-ness would change between the two. If I'm\n> being honest, I'm actually slightly surprised that `isatty` returned\n> true for your hook before - since the hook process is a child of Git and\n> its output is, presumably, being consumed by Git first rather than by an\n> interactive user shell.\n>\n> I suppose that with stdout_to_stderr being set, the tty-ness of the main\n> process's stderr would then apply to the child process's stdout (we do\n> that by calling `dup(2)`). But that's being set in both \"the old way\"\n> and \"the new way\", so I'm pretty surprised to see a change here.\n>\n> It *is* true that run-command.c:pp_start_one() sets child_process:err=-1\n> for the child and run-command.c:run_hook_ve() didn't do that; that -1\n> means that start_command() will create a new fd for the child's stderr.\n> Since run_hook_ve() didn't care about the child's stderr before, I\n> wonder if that is why? Could it be that now that we're processing the\n> child's stderr, the child no longer thinks stderr is in tty, because the\n> parent is consuming its output?\n>\n> I think if that's the case, a fix would involve\n> run-command.c:pp_start_one() not setting .err, .stdout_to_stderr, or\n> .no_stdin at all on its own, and relying on the 'get_next_task' callback\n> to set those things. It's a little more painful than I initially thought\n> because the run_processes_parallel() library depends on that err capture\n> to run pp_buffer_stderr() unconditionally; I guess it needs a tiny bit\n> of shim logic to deal with callers who don't care to see their\n> children's stderr.\n>\n> All that said.... I'd expect that the dup() from the child's stdout to\n> the parent's stderr would still result in a happy isatty(1). So I'm not\n> convinced this is actually the right solution.... From your repro\n> script, I can't quite tell which fd the isatty call is against (to be\n> honest, I can't find the isatty call, either). So maybe I'm going the\n> wrong direction :)\n\nah, most of the repro script was just so I could bisect -- you can\nignore pretty much all of it except for the `pre-commit` file:\n\n#!/usr/bin/env bash\nif [ -t 1 ]; then\n echo GOOD\nfi\n\nthis is doing \"isatty\" against fd 1 which is stdout (it could also try\nthe same against fd 2 which was also a tty previously)\n\nAnthony\n\n>\n> - Emily\n"},{"id":"453948","messageId":"xmqqy200s7r7.fsf@gitster.g","threadId":"57764","inReplyTo":"CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com","subject":"Re: git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-20T04:23:24Z","receivedAt":"2022-04-20T04:23:32Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Anthony Sottile writes:\n\n> here's the shortest reproduction --\n>\n> ```console\n> $ cat ../testrepo/.git/hooks/pre-commit\n> #!/usr/bin/env bash\n> if [ -t 1 ]; then\n> echo GOOD\n> fi\n> ```\n\n> f443246b9f29b815f0b98a07bb2d425628ae6522 is the first bad commit\n> commit f443246b9f29b815f0b98a07bb2d425628ae6522\n> Author: Emily Shaffer \n> Date: Wed Dec 22 04:59:40 2021 +0100\n>\n> commit: convert {pre-commit,prepare-commit-msg} hook to hook.h\n>\n> Move these hooks hook away from run-command.h to and over to the new\n> hook.h library.\n>\n> Signed-off-by: Emily Shaffer \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Acked-by: Emily Shaffer \n> Signed-off-by: Junio C Hamano \n>\n> commit.c | 15 ++++++++-------\n> 1 file changed, 8 insertions(+), 7 deletions(-)\n> bisect run success\n\nNicely bisected. Thanks.\n\nI have a feeling that it may have been a deliberate design decision\nwhen Ævar revamped the code that drives the hook invocation based on\nEmily's code. Ævar, Emily, do any of you remember why we did this,\nor is this a mere regression?\n\nThanks.\n"},{"id":"453956","messageId":"6aabbcd6-f6c2-fe97-eb73-593bcf2e9e75@gmail.com","threadId":"57764","inReplyTo":"Yl9Hn0C0TwalASC0@google.com","subject":"Re: git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-04-20T09:00:45Z","receivedAt":"2022-04-20T09:01:20Z","isPatch":false,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Emily\n\nOn 20/04/2022 00:37, Emily Shaffer wrote:\n> On Tue, Apr 19, 2022 at 02:59:36PM -0400, Anthony Sottile wrote:\n>> [...]\n> Interesting. I'm surprised to see the tty-ness of hooks changing with\n> this patch, as the way the hook is called is pretty much the same:\n> \n> run_hook_ve() (\"the old way\") sets no_stdin, stdout_to_stderr, args,\n> envvars, and some trace variables, and then runs 'run_command()';\n> run_command() invokes start_command().\n> \n> run_hooks_opt (\"the new way\") ultimately kicks off the hook with a\n> callback that sets up a child_process with no_stdin, stdout_to_stderr,\n> args, envvars, and some trace variables (hook.c:pick_next_hook); the\n> task queue manager also sets .err to -1 on that child_process; then it\n> calls start_command() directly (run-command.c:pp_start_one()).\n> \n> I'm not sure I see why the tty-ness would change between the two. If I'm\n> being honest, I'm actually slightly surprised that `isatty` returned\n> true for your hook before - since the hook process is a child of Git and\n> its output is, presumably, being consumed by Git first rather than by an\n> interactive user shell.\n> \n> I suppose that with stdout_to_stderr being set, the tty-ness of the main\n> process's stderr would then apply to the child process's stdout (we do\n> that by calling `dup(2)`). But that's being set in both \"the old way\"\n> and \"the new way\", so I'm pretty surprised to see a change here.\n >\n> It *is* true that run-command.c:pp_start_one() sets child_process:err=-1\n> for the child and run-command.c:run_hook_ve() didn't do that; that -1\n> means that start_command() will create a new fd for the child's stderr.\n> Since run_hook_ve() didn't care about the child's stderr before, I\n> wonder if that is why? Could it be that now that we're processing the\n> child's stderr, the child no longer thinks stderr is in tty, because the\n> parent is consuming its output?\n\nExactly, stderr is redirected to a pipe so that we can buffer the output \nfrom each process and then write it to the real stdout when the process \nhas finished to avoid the output from different processes getting mixed \ntogether. Ideally in this case we'd see that stdout is a tty and create \na pty rather than a pipe when buffering the output from the process.\n\nBest Wishes\n\nPhillip\n"},{"id":"453976","messageId":"220420.86wnfk6isy.gmgdl@evledraar.gmail.com","threadId":"57764","inReplyTo":"6aabbcd6-f6c2-fe97-eb73-593bcf2e9e75@gmail.com","subject":"Re: git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-04-20T12:25:15Z","receivedAt":"2022-04-20T12:28:04Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Apr 20 2022, Phillip Wood wrote:\n\n> Hi Emily\n>\n> On 20/04/2022 00:37, Emily Shaffer wrote:\n>> On Tue, Apr 19, 2022 at 02:59:36PM -0400, Anthony Sottile wrote:\n>>> [...]\n>> Interesting. I'm surprised to see the tty-ness of hooks changing with\n>> this patch, as the way the hook is called is pretty much the same:\n>> run_hook_ve() (\"the old way\") sets no_stdin, stdout_to_stderr, args,\n>> envvars, and some trace variables, and then runs 'run_command()';\n>> run_command() invokes start_command().\n>> run_hooks_opt (\"the new way\") ultimately kicks off the hook with a\n>> callback that sets up a child_process with no_stdin, stdout_to_stderr,\n>> args, envvars, and some trace variables (hook.c:pick_next_hook); the\n>> task queue manager also sets .err to -1 on that child_process; then it\n>> calls start_command() directly (run-command.c:pp_start_one()).\n>> I'm not sure I see why the tty-ness would change between the two. If\n>> I'm\n>> being honest, I'm actually slightly surprised that `isatty` returned\n>> true for your hook before - since the hook process is a child of Git and\n>> its output is, presumably, being consumed by Git first rather than by an\n>> interactive user shell.\n>> I suppose that with stdout_to_stderr being set, the tty-ness of the\n>> main\n>> process's stderr would then apply to the child process's stdout (we do\n>> that by calling `dup(2)`). But that's being set in both \"the old way\"\n>> and \"the new way\", so I'm pretty surprised to see a change here.\n>>\n>> It *is* true that run-command.c:pp_start_one() sets child_process:err=-1\n>> for the child and run-command.c:run_hook_ve() didn't do that; that -1\n>> means that start_command() will create a new fd for the child's stderr.\n>> Since run_hook_ve() didn't care about the child's stderr before, I\n>> wonder if that is why? Could it be that now that we're processing the\n>> child's stderr, the child no longer thinks stderr is in tty, because the\n>> parent is consuming its output?\n>\n> Exactly, stderr is redirected to a pipe so that we can buffer the\n> output from each process and then write it to the real stdout when the\n> process has finished to avoid the output from different processes\n> getting mixed together. Ideally in this case we'd see that stdout is a\n> tty and create a pty rather than a pipe when buffering the output from\n> the process.\n\nAll: I have a fix for this, currently CI-ing, testing etc. Basically it\njust adds an option to run_process_parallel() to stop doing the\nstdout/stderr interception.\n\nIt means that for the current jobs=1 we'll behave as before.\n\nFor jobs >1 in the future we'll need to decide what we want to do,\ni.e. you can have TTY, or guaranteed non-interleaved output, but not\nboth.\n\nI'd think for hooks no interception makes sense, but in any case we can\ndefer that until sometime later...\n\nPreview of the fix below, this is on top of an earlier change to add the\n\"struct run_process_parallel_opts\" to pass such options along:\n\ndiff --git a/hook.c b/hook.c\nindex eadb2d58a7b..1f20e5db447 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -126,6 +126,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \tstruct run_process_parallel_opts run_opts = {\n \t\t.tr2_category = \"hook\",\n \t\t.tr2_label = hook_name,\n+\t\t.no_buffering = 1,\n \t};\n \n \tif (!options)\ndiff --git a/run-command.c b/run-command.c\nindex 2383375ee07..0f9d84433ad 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1604,7 +1604,7 @@ static void pp_cleanup(struct parallel_processes *pp)\n * <0 no new job was started, user wishes to shutdown early. Use negative code\n * to signal the children.\n */\n-static int pp_start_one(struct parallel_processes *pp)\n+static int pp_start_one(struct parallel_processes *pp, const int no_buffering)\n {\n \tint i, code;\n \n@@ -1623,9 +1623,12 @@ static int pp_start_one(struct parallel_processes *pp)\n \t\tstrbuf_reset(&pp->children[i].err);\n \t\treturn 1;\n \t}\n-\tpp->children[i].process.err = -1;\n-\tpp->children[i].process.stdout_to_stderr = 1;\n-\tpp->children[i].process.no_stdin = 1;\n+\n+\tif (!no_buffering) {\n+\t\tpp->children[i].process.err = -1;\n+\t\tpp->children[i].process.stdout_to_stderr = 1;\n+\t\tpp->children[i].process.no_stdin = 1;\n+\t}\n \n \tif (start_command(&pp->children[i].process)) {\n \t\tcode = pp->start_failure(&pp->children[i].err,\n@@ -1681,12 +1684,17 @@ static void pp_output(struct parallel_processes *pp)\n \t}\n }\n \n-static int pp_collect_finished(struct parallel_processes *pp)\n+static int pp_collect_finished(struct parallel_processes *pp,\n+\t\t\t const int no_buffering)\n {\n \tint i, code;\n \tint n = pp->max_processes;\n \tint result = 0;\n \n+\tif (no_buffering)\n+\t\tfor (i = 0; i < pp->max_processes; i++)\n+\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+\n \twhile (pp->nr_processes > 0) {\n \t\tfor (i = 0; i < pp->max_processes; i++)\n \t\t\tif (pp->children[i].state == GIT_CP_WAIT_CLEANUP)\n@@ -1741,7 +1749,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n static int run_processes_parallel_1(int n, get_next_task_fn get_next_task,\n \t\t\t\t start_failure_fn start_failure,\n \t\t\t\t task_finished_fn task_finished,\n-\t\t\t\t void *pp_cb)\n+\t\t\t\t void *pp_cb, const int no_buffering)\n {\n \tint i, code;\n \tint output_timeout = 100;\n@@ -1754,7 +1762,7 @@ static int run_processes_parallel_1(int n, get_next_task_fn get_next_task,\n \t\t i < spawn_cap && !pp.shutdown &&\n \t\t pp.nr_processes < pp.max_processes;\n \t\t i++) {\n-\t\t\tcode = pp_start_one(&pp);\n+\t\t\tcode = pp_start_one(&pp, no_buffering);\n \t\t\tif (!code)\n \t\t\t\tcontinue;\n \t\t\tif (code < 0) {\n@@ -1765,9 +1773,11 @@ static int run_processes_parallel_1(int n, get_next_task_fn get_next_task,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tpp_buffer_stderr(&pp, output_timeout);\n-\t\tpp_output(&pp);\n-\t\tcode = pp_collect_finished(&pp);\n+\t\tif (!no_buffering) {\n+\t\t\tpp_buffer_stderr(&pp, output_timeout);\n+\t\t\tpp_output(&pp);\n+\t\t}\n+\t\tcode = pp_collect_finished(&pp, no_buffering);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n \t\t\tif (code < 0)\n@@ -1783,7 +1793,8 @@ static int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \t\t\t\t start_failure_fn start_failure,\n \t\t\t\t task_finished_fn task_finished,\n \t\t\t\t void *pp_cb, const char *tr2_category,\n-\t\t\t\t const char *tr2_label)\n+\t\t\t\t const char *tr2_label,\n+\t\t\t\t const int no_buffering)\n {\n \tint result;\n \n@@ -1791,7 +1802,7 @@ static int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \t\t\t\t ((n < 1) ? online_cpus() : n));\n \n \tresult = run_processes_parallel_1(n, get_next_task, start_failure,\n-\t\t\t\t\t task_finished, pp_cb);\n+\t\t\t\t\t task_finished, pp_cb, no_buffering);\n \n \ttrace2_region_leave(tr2_category, tr2_label, NULL);\n \n@@ -1803,6 +1814,8 @@ int run_processes_parallel(int n, get_next_task_fn get_next_task,\n \t\t\t task_finished_fn task_finished, void *pp_cb,\n \t\t\t struct run_process_parallel_opts *opts)\n {\n+\tconst int no_buffering = opts && opts->no_buffering;\n+\n \tif (!opts)\n \t\tgoto no_opts;\n \n@@ -1811,12 +1824,13 @@ int run_processes_parallel(int n, get_next_task_fn get_next_task,\n \t\treturn run_processes_parallel_tr2(n, get_next_task,\n \t\t\t\t\t\t start_failure, task_finished,\n \t\t\t\t\t\t pp_cb, opts->tr2_category,\n-\t\t\t\t\t\t opts->tr2_label);\n+\t\t\t\t\t\t opts->tr2_label,\n+\t\t\t\t\t\t no_buffering);\n \t}\n \n no_opts:\n \treturn run_processes_parallel_1(n, get_next_task, start_failure,\n-\t\t\t\t\ttask_finished, pp_cb);\n+\t\t\t\t\ttask_finished, pp_cb, no_buffering);\n }\n \n \ndiff --git a/run-command.h b/run-command.h\nindex 9ec57a25de4..062eff81e17 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -463,11 +463,17 @@ typedef int (*task_finished_fn)(int result,\n *\n * tr2_category & tr2_label: sets the trace2 category and label for\n * logging. These must either be unset, or both of them must be set.\n+ *\n+ * no_buffering: Don't redirect stderr to stdout, and don't \"buffer\"\n+ * the output of the N children started. The output will not be\n+ * deterministic and may be interleaved, but we won't interfere with\n+ * the connection to the TTY.\n */\n struct run_process_parallel_opts\n {\n \tconst char *tr2_category;\n \tconst char *tr2_label;\n+\tunsigned int no_buffering:1;\n };\n \n /**\n@@ -477,7 +483,8 @@ struct run_process_parallel_opts\n *\n * The children started via this function run in parallel. Their output\n * (both stdout and stderr) is routed to stderr in a manner that output\n- * from different tasks does not interleave.\n+ * from different tasks does not interleave. This can be disabled by setting\n+ * \"no_buffering\" in \"struct run_process_parallel_opts\".\n *\n * start_failure_fn and task_finished_fn can be NULL to omit any\n * special handling.\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex ee281909bc3..fb6ad0bf4f7 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -130,7 +130,7 @@ World\n EOF\n \n test_expect_success 'run_command runs in parallel with more jobs available than tasks' '\n-\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n+\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >actual 2>&1 &&\n \ttest_cmp expect actual\n '\n \ndiff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\nindex 26ed5e11bc8..c0eda4e9237 100755\n--- a/t/t1800-hook.sh\n+++ b/t/t1800-hook.sh\n@@ -4,6 +4,7 @@ test_description='git-hook command'\n \n TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-terminal.sh\n \n test_expect_success 'git hook usage' '\n \ttest_expect_code 129 git hook &&\n@@ -120,4 +121,49 @@ test_expect_success 'git -c core.hooksPath= hook run' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDOUT redirect' '\n+\trm -rf .git &&\n+\ttest_when_finished \"rm -rf .git\" &&\n+\tgit init . &&\n+\n+\ttest_hook pre-commit <<-EOF &&\n+\t{\n+\t\ttest -t 1 && echo STDOUT TTY || echo STDOUT NO TTY &&\n+\t\ttest -t 2 && echo STDERR TTY || echo STDERR NO TTY\n+\t} >actual\n+\tEOF\n+\n+\ttest_commit A &&\n+\ttest_commit B &&\n+\tgit reset --soft HEAD^ &&\n+\tcat >expect <<-\\EOF &&\n+\tSTDOUT NO TTY\n+\tSTDERR TTY\n+\tEOF\n+\ttest_terminal git commit -m\"msg\" &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDERR redirect' '\n+\ttest_when_finished \"rm -rf .git\" &&\n+\tgit init . &&\n+\n+\ttest_hook pre-commit <<-EOF &&\n+\t{\n+\t\ttest -t 1 && echo >&2 STDOUT TTY || echo >&2 STDOUT NO TTY &&\n+\t\ttest -t 2 && echo >&2 STDERR TTY || echo >&2 STDERR NO TTY\n+\t} 2>actual\n+\tEOF\n+\n+\ttest_commit A &&\n+\ttest_commit B &&\n+\tgit reset --soft HEAD^ &&\n+\tcat >expect <<-\\EOF &&\n+\tSTDOUT TTY\n+\tSTDERR NO TTY\n+\tEOF\n+\ttest_terminal git commit -m\"msg\" &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n"},{"id":"453982","messageId":"CAJoAoZ=ysz6GDjUVCzUC-4OEwkyfrUzDyAws1xPbKXeLufUa0w@mail.gmail.com","threadId":"57764","inReplyTo":"220420.86wnfk6isy.gmgdl@evledraar.gmail.com","subject":"Re: git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2022-04-20T16:22:09Z","receivedAt":"2022-04-20T16:22:25Z","isPatch":false,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Wed, Apr 20, 2022 at 5:28 AM Ævar Arnfjörð Bjarmason\n wrote:\n> >> It *is* true that run-command.c:pp_start_one() sets child_process:err=-1\n> >> for the child and run-command.c:run_hook_ve() didn't do that; that -1\n> >> means that start_command() will create a new fd for the child's stderr.\n> >> Since run_hook_ve() didn't care about the child's stderr before, I\n> >> wonder if that is why? Could it be that now that we're processing the\n> >> child's stderr, the child no longer thinks stderr is in tty, because the\n> >> parent is consuming its output?\n> >\n> > Exactly, stderr is redirected to a pipe so that we can buffer the\n> > output from each process and then write it to the real stdout when the\n> > process has finished to avoid the output from different processes\n> > getting mixed together. Ideally in this case we'd see that stdout is a\n> > tty and create a pty rather than a pipe when buffering the output from\n> > the process.\n>\n> All: I have a fix for this, currently CI-ing, testing etc. Basically it\n> just adds an option to run_process_parallel() to stop doing the\n> stdout/stderr interception.\n>\n> It means that for the current jobs=1 we'll behave as before.\n>\n> For jobs >1 in the future we'll need to decide what we want to do,\n> i.e. you can have TTY, or guaranteed non-interleaved output, but not\n> both.\n>\n> I'd think for hooks no interception makes sense, but in any case we can\n> defer that until sometime later...\n\nI'm curious what your reasoning is there. I rely on hooks which give\nme user-readable output quite frequently, so the interleaving is\nimportant to keep them from being useless if I trigger more than one\nhook (e.g. I have separate hooks to check for secret keys and for\ndebug strings).\n\nWould it make sense to start by setting it based on the number of\nhooks available?\n\nLeft a quick thought below, but please don't consider it as a full\nreview - I haven't got time to look much more yet.\n\n>\n> Preview of the fix below, this is on top of an earlier change to add the\n> \"struct run_process_parallel_opts\" to pass such options along:\n>\n> diff --git a/hook.c b/hook.c\n> index eadb2d58a7b..1f20e5db447 100644\n> --- a/hook.c\n> +++ b/hook.c\n> @@ -126,6 +126,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n> struct run_process_parallel_opts run_opts = {\n> .tr2_category = \"hook\",\n> .tr2_label = hook_name,\n> + .no_buffering = 1,\n> };\n>\n> if (!options)\n> diff --git a/run-command.c b/run-command.c\n> index 2383375ee07..0f9d84433ad 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1604,7 +1604,7 @@ static void pp_cleanup(struct parallel_processes *pp)\n> * <0 no new job was started, user wishes to shutdown early. Use negative code\n> * to signal the children.\n> */\n> -static int pp_start_one(struct parallel_processes *pp)\n> +static int pp_start_one(struct parallel_processes *pp, const int no_buffering)\n> {\n> int i, code;\n>\n> @@ -1623,9 +1623,12 @@ static int pp_start_one(struct parallel_processes *pp)\n> strbuf_reset(&pp->children[i].err);\n> return 1;\n> }\n> - pp->children[i].process.err = -1;\n> - pp->children[i].process.stdout_to_stderr = 1;\n> - pp->children[i].process.no_stdin = 1;\n> +\n> + if (!no_buffering) {\n> + pp->children[i].process.err = -1;\n> + pp->children[i].process.stdout_to_stderr = 1;\n> + pp->children[i].process.no_stdin = 1;\n> + }\n\nIs it not possible to let run_processes_parallel() callers set these\nflags manually (as they are providing a child_process in the \"get next\ntask\" callback), and then to decide whether to buffer the output based\non the fd status instead? I'd prefer that rather than an all-or-none\noption that may not apply to every process, I think... But I could be\nwrong :)\n\n>\n> if (start_command(&pp->children[i].process)) {\n> code = pp->start_failure(&pp->children[i].err,\n> @@ -1681,12 +1684,17 @@ static void pp_output(struct parallel_processes *pp)\n> }\n> }\n>\n> -static int pp_collect_finished(struct parallel_processes *pp)\n> +static int pp_collect_finished(struct parallel_processes *pp,\n> + const int no_buffering)\n> {\n> int i, code;\n> int n = pp->max_processes;\n> int result = 0;\n>\n> + if (no_buffering)\n> + for (i = 0; i < pp->max_processes; i++)\n> + pp->children[i].state = GIT_CP_WAIT_CLEANUP;\n> +\n> while (pp->nr_processes > 0) {\n> for (i = 0; i < pp->max_processes; i++)\n> if (pp->children[i].state == GIT_CP_WAIT_CLEANUP)\n> @@ -1741,7 +1749,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> static int run_processes_parallel_1(int n, get_next_task_fn get_next_task,\n> start_failure_fn start_failure,\n> task_finished_fn task_finished,\n> - void *pp_cb)\n> + void *pp_cb, const int no_buffering)\n> {\n> int i, code;\n> int output_timeout = 100;\n> @@ -1754,7 +1762,7 @@ static int run_processes_parallel_1(int n, get_next_task_fn get_next_task,\n> i < spawn_cap && !pp.shutdown &&\n> pp.nr_processes < pp.max_processes;\n> i++) {\n> - code = pp_start_one(&pp);\n> + code = pp_start_one(&pp, no_buffering);\n> if (!code)\n> continue;\n> if (code < 0) {\n> @@ -1765,9 +1773,11 @@ static int run_processes_parallel_1(int n, get_next_task_fn get_next_task,\n> }\n> if (!pp.nr_processes)\n> break;\n> - pp_buffer_stderr(&pp, output_timeout);\n> - pp_output(&pp);\n> - code = pp_collect_finished(&pp);\n> + if (!no_buffering) {\n> + pp_buffer_stderr(&pp, output_timeout);\n> + pp_output(&pp);\n> + }\n> + code = pp_collect_finished(&pp, no_buffering);\n> if (code) {\n> pp.shutdown = 1;\n> if (code < 0)\n> @@ -1783,7 +1793,8 @@ static int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n> start_failure_fn start_failure,\n> task_finished_fn task_finished,\n> void *pp_cb, const char *tr2_category,\n> - const char *tr2_label)\n> + const char *tr2_label,\n> + const int no_buffering)\n> {\n> int result;\n>\n> @@ -1791,7 +1802,7 @@ static int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n> ((n < 1) ? online_cpus() : n));\n>\n> result = run_processes_parallel_1(n, get_next_task, start_failure,\n> - task_finished, pp_cb);\n> + task_finished, pp_cb, no_buffering);\n>\n> trace2_region_leave(tr2_category, tr2_label, NULL);\n>\n> @@ -1803,6 +1814,8 @@ int run_processes_parallel(int n, get_next_task_fn get_next_task,\n> task_finished_fn task_finished, void *pp_cb,\n> struct run_process_parallel_opts *opts)\n> {\n> + const int no_buffering = opts && opts->no_buffering;\n> +\n> if (!opts)\n> goto no_opts;\n>\n> @@ -1811,12 +1824,13 @@ int run_processes_parallel(int n, get_next_task_fn get_next_task,\n> return run_processes_parallel_tr2(n, get_next_task,\n> start_failure, task_finished,\n> pp_cb, opts->tr2_category,\n> - opts->tr2_label);\n> + opts->tr2_label,\n> + no_buffering);\n> }\n>\n> no_opts:\n> return run_processes_parallel_1(n, get_next_task, start_failure,\n> - task_finished, pp_cb);\n> + task_finished, pp_cb, no_buffering);\n> }\n>\n>\n> diff --git a/run-command.h b/run-command.h\n> index 9ec57a25de4..062eff81e17 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -463,11 +463,17 @@ typedef int (*task_finished_fn)(int result,\n> *\n> * tr2_category & tr2_label: sets the trace2 category and label for\n> * logging. These must either be unset, or both of them must be set.\n> + *\n> + * no_buffering: Don't redirect stderr to stdout, and don't \"buffer\"\n> + * the output of the N children started. The output will not be\n> + * deterministic and may be interleaved, but we won't interfere with\n> + * the connection to the TTY.\n> */\n> struct run_process_parallel_opts\n> {\n> const char *tr2_category;\n> const char *tr2_label;\n> + unsigned int no_buffering:1;\n> };\n>\n> /**\n> @@ -477,7 +483,8 @@ struct run_process_parallel_opts\n> *\n> * The children started via this function run in parallel. Their output\n> * (both stdout and stderr) is routed to stderr in a manner that output\n> - * from different tasks does not interleave.\n> + * from different tasks does not interleave. This can be disabled by setting\n> + * \"no_buffering\" in \"struct run_process_parallel_opts\".\n> *\n> * start_failure_fn and task_finished_fn can be NULL to omit any\n> * special handling.\n> diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\n> index ee281909bc3..fb6ad0bf4f7 100755\n> --- a/t/t0061-run-command.sh\n> +++ b/t/t0061-run-command.sh\n> @@ -130,7 +130,7 @@ World\n> EOF\n>\n> test_expect_success 'run_command runs in parallel with more jobs available than tasks' '\n> - test-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n> + test-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >actual 2>&1 &&\n> test_cmp expect actual\n> '\n>\n> diff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\n> index 26ed5e11bc8..c0eda4e9237 100755\n> --- a/t/t1800-hook.sh\n> +++ b/t/t1800-hook.sh\n> @@ -4,6 +4,7 @@ test_description='git-hook command'\n>\n> TEST_PASSES_SANITIZE_LEAK=true\n> . ./test-lib.sh\n> +. \"$TEST_DIRECTORY\"/lib-terminal.sh\n>\n> test_expect_success 'git hook usage' '\n> test_expect_code 129 git hook &&\n> @@ -120,4 +121,49 @@ test_expect_success 'git -c core.hooksPath= hook run' '\n> test_cmp expect actual\n> '\n>\n> +test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDOUT redirect' '\n> + rm -rf .git &&\n> + test_when_finished \"rm -rf .git\" &&\n> + git init . &&\n> +\n> + test_hook pre-commit <<-EOF &&\n> + {\n> + test -t 1 && echo STDOUT TTY || echo STDOUT NO TTY &&\n> + test -t 2 && echo STDERR TTY || echo STDERR NO TTY\n> + } >actual\n> + EOF\n> +\n> + test_commit A &&\n> + test_commit B &&\n> + git reset --soft HEAD^ &&\n> + cat >expect <<-\\EOF &&\n> + STDOUT NO TTY\n> + STDERR TTY\n> + EOF\n> + test_terminal git commit -m\"msg\" &&\n> + test_cmp expect actual\n> +'\n> +\n> +test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDERR redirect' '\n> + test_when_finished \"rm -rf .git\" &&\n> + git init . &&\n> +\n> + test_hook pre-commit <<-EOF &&\n> + {\n> + test -t 1 && echo >&2 STDOUT TTY || echo >&2 STDOUT NO TTY &&\n> + test -t 2 && echo >&2 STDERR TTY || echo >&2 STDERR NO TTY\n> + } 2>actual\n> + EOF\n> +\n> + test_commit A &&\n> + test_commit B &&\n> + git reset --soft HEAD^ &&\n> + cat >expect <<-\\EOF &&\n> + STDOUT TTY\n> + STDERR NO TTY\n> + EOF\n> + test_terminal git commit -m\"msg\" &&\n> + test_cmp expect actual\n> +'\n> +\n> test_done\n"},{"id":"453983","messageId":"xmqqr15rr9k6.fsf@gitster.g","threadId":"57764","inReplyTo":"6aabbcd6-f6c2-fe97-eb73-593bcf2e9e75@gmail.com","subject":"Re: git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-20T16:42:01Z","receivedAt":"2022-04-20T16:42:10Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n>> It *is* true that run-command.c:pp_start_one() sets child_process:err=-1\n>> for the child and run-command.c:run_hook_ve() didn't do that; that -1\n>> means that start_command() will create a new fd for the child's stderr.\n>> Since run_hook_ve() didn't care about the child's stderr before, I\n>> wonder if that is why? Could it be that now that we're processing the\n>> child's stderr, the child no longer thinks stderr is in tty, because the\n>> parent is consuming its output?\n>\n> Exactly, stderr is redirected to a pipe so that we can buffer the\n> output from each process and then write it to the real stdout when the\n> process has finished to avoid the output from different processes\n> getting mixed together. Ideally in this case we'd see that stdout is a\n> tty and create a pty rather than a pipe when buffering the output from\n> the process.\n\nAh, thanks, and sigh. That means this was an unintended regression\ncaused by use of parallel infrastructure, mixed with a bit of \"the\noriginal problem report wrote hook properly so that when it is not\nconnected to a terminal (such as in this new implementation) it\nrefrains to do terminal-y things like coloring, so everything is\nworking as intended\" ;-).\n\nIIRC, the parallel subprocess stuff was invented to spawn multiple\ntasks we internally need (like \"checkout these submodules\") that are\nnot interactive (hence does not need access to stdin) en masse, and\nthe output buffering is there to avoid interleaving the output that\nwould make it unreadable.\n\nUse of the parallel subprocess API means that we inherently cannot\ngive access to the standard input to the hooks. The users of the\noriginal run_hooks_ve() API would be OK with that, because it did\n.no_stdin=1 before the problematic hooks API rewrite, but I wonder\nwhat our plans should be for hooks that want to go interactive.\nThey could open /dev/tty themselves (and that would have been the\nonly way to go interactive even in the old world order, so it is\nperfectly acceptable to keep it that way with .no_stdin=1), but if\nthey run in parallel, the end-user would not know whom they are\ntyping to (and which output lines are the prompts they are expected\nto respond to).\n\nIn the longer term, there are multiple possible action items.\n\n * We probably would want to design a bit better anti-interleaving\n machinery than \"buffer everything and show only after the process\n exists\", if we want to keep using the parallel subprocess API.\n And that would help the original \"do this thing in multiple\n submodules at the same time\" use case, too. \n\n * We should teach hooks API to make it _optional_ to use the\n parallel subprocess API. If we are not spawning hooks in\n parallel today, there is no reason to incur this regression by\n using the parallel subprocess API---this was a needress bug, and\n I am angry.\n\n * the hooks API should learn a mechanism for multiple hooks to\n coordinate their executions. Perhaps they indicate their\n preference if they are OK to be run in parallel, and those that\n want isolation will be run one-at-a-time before or after others\n run in parallel, or something.\n\n * The hooks API should learn a mechanism for us to tell what\n execution environment they are in. Ideally, the hooks, if it is\n sane to run under the parallel subprocess API, shouldn't have\n been learning if they are talking to an interactive human user by\n looking at isatty(), but we should have been explicitly telling\n them that they are, perhaps by exporting an environment\n variable. There may probably be more clue hooks writers want\n other than \"am I talking to human user?\" that we would want to\n enumerate before going this route.\n\nThanks for analyzing.\n"},{"id":"453984","messageId":"CAJoAoZm7p32Hn=TLQeWUqp_nMjo_TQ2whR4F=cXk4c6PV1M5bA@mail.gmail.com","threadId":"57764","inReplyTo":"xmqqr15rr9k6.fsf@gitster.g","subject":"Re: git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2022-04-20T17:09:38Z","receivedAt":"2022-04-20T17:09:59Z","isPatch":false,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Wed, Apr 20, 2022 at 9:42 AM Junio C Hamano wrote:\n>\n> Phillip Wood writes:\n>\n> >> It *is* true that run-command.c:pp_start_one() sets child_process:err=-1\n> >> for the child and run-command.c:run_hook_ve() didn't do that; that -1\n> >> means that start_command() will create a new fd for the child's stderr.\n> >> Since run_hook_ve() didn't care about the child's stderr before, I\n> >> wonder if that is why? Could it be that now that we're processing the\n> >> child's stderr, the child no longer thinks stderr is in tty, because the\n> >> parent is consuming its output?\n> >\n> > Exactly, stderr is redirected to a pipe so that we can buffer the\n> > output from each process and then write it to the real stdout when the\n> > process has finished to avoid the output from different processes\n> > getting mixed together. Ideally in this case we'd see that stdout is a\n> > tty and create a pty rather than a pipe when buffering the output from\n> > the process.\n>\n> Ah, thanks, and sigh. That means this was an unintended regression\n> caused by use of parallel infrastructure, mixed with a bit of \"the\n> original problem report wrote hook properly so that when it is not\n> connected to a terminal (such as in this new implementation) it\n> refrains to do terminal-y things like coloring, so everything is\n> working as intended\" ;-).\n>\n> IIRC, the parallel subprocess stuff was invented to spawn multiple\n> tasks we internally need (like \"checkout these submodules\") that are\n> not interactive (hence does not need access to stdin) en masse, and\n> the output buffering is there to avoid interleaving the output that\n> would make it unreadable.\n>\n> Use of the parallel subprocess API means that we inherently cannot\n> give access to the standard input to the hooks. The users of the\n> original run_hooks_ve() API would be OK with that, because it did\n> .no_stdin=1 before the problematic hooks API rewrite, but I wonder\n> what our plans should be for hooks that want to go interactive.\n> They could open /dev/tty themselves (and that would have been the\n> only way to go interactive even in the old world order, so it is\n> perfectly acceptable to keep it that way with .no_stdin=1), but if\n> they run in parallel, the end-user would not know whom they are\n> typing to (and which output lines are the prompts they are expected\n> to respond to).\n>\n> In the longer term, there are multiple possible action items.\n>\n> * We probably would want to design a bit better anti-interleaving\n> machinery than \"buffer everything and show only after the process\n> exists\", if we want to keep using the parallel subprocess API.\n> And that would help the original \"do this thing in multiple\n> submodules at the same time\" use case, too.\n\nI've noticed this too, but for very noisy things which are\nparallelized, I'm not sure a better user experience is possible. I\nsuppose we could pick the \"first\" job in the task queue and print that\noutput as it comes in, so that users are aware that *something* is\nhappening?\n\n[job 0 starts]\n[job 1 starts]\njob 0 says 0-foo\n[job 1 says 1-foo, but it's buffered]\njob 0 says 0-bar\n[job 1 says 1-bar, but it's buffered]\n[job 0 finishes]\n[we replay the buffer from job 1 so far:]\njob 1 says 1-foo\njob 1 says 1-bar\njob 1 says 1-baz\n[job 1 finishes]\n\nI think it could be possible, but then job 1 still will never learn\nthat it's a tty, because it's being buffered to prevent interleaving,\neven if we have the illusion of non-buffering.\n\n>\n> * We should teach hooks API to make it _optional_ to use the\n> parallel subprocess API. If we are not spawning hooks in\n> parallel today, there is no reason to incur this regression by\n> using the parallel subprocess API---this was a needress bug, and\n> I am angry.\n\nTo counter, I think that having hooks invoked via two different\nmechanisms depending on how many are provided or whether they are\nparallelized is a mess to debug and maintain. I still stand by the\ndecision to use the parallel subprocess API, which I think was\nreasonable to expect to do the same thing when jobs=1, and I think we\nshould continue to do so. It simplifies the hook code significantly.\n\n>\n> * the hooks API should learn a mechanism for multiple hooks to\n> coordinate their executions. Perhaps they indicate their\n> preference if they are OK to be run in parallel, and those that\n> want isolation will be run one-at-a-time before or after others\n> run in parallel, or something.\n\nThere is such a mechanism for hooks overall, but not yet for\nindividual hooks. I know we discussed it at length[1] before, and\ndecided it would be okay to figure this out later on. I suppose \"later\non\" may have come :)\n\n>\n> * The hooks API should learn a mechanism for us to tell what\n> execution environment they are in. Ideally, the hooks, if it is\n> sane to run under the parallel subprocess API, shouldn't have\n> been learning if they are talking to an interactive human user by\n> looking at isatty(), but we should have been explicitly telling\n> them that they are, perhaps by exporting an environment\n> variable. There may probably be more clue hooks writers want\n> other than \"am I talking to human user?\" that we would want to\n> enumerate before going this route.\n\nHm. I was going to mention that Ævar and I discussed the possibility\nof setting an environment variable for hook child processes, telling\nthem which hook they are being run as - e.g.\n\"GIT_HOOK=prepare-commit-msg\" - but I suppose that relying on that\nalone doesn't tell us anything about whether the parent is being run\nin tty. I agree it could be very useful to simply pass\nGIT_PARENT_ISATTY to hooks (and I suppose other child processes).\nCould we simply do that from start_command() or something else deep in\nrun-command.h machinery? Then Anthony's use case becomes\n\nif [-t 1|| GIT_PARENT_ISATTY]\n ...\n\nand no need to examine Git version.\n\n - Emily\n\n1: https://lore.kernel.org/git/20210527000856.695702-2-emilyshaffer%40google.com\nunder \"Parallelization with dependencies\" (and preceding\nconversations)\n"},{"id":"453986","messageId":"xmqqilr3r7ki.fsf@gitster.g","threadId":"57764","inReplyTo":"CAJoAoZm7p32Hn=TLQeWUqp_nMjo_TQ2whR4F=cXk4c6PV1M5bA@mail.gmail.com","subject":"Re: git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-20T17:25:01Z","receivedAt":"2022-04-20T17:25:09Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Emily Shaffer writes:\n\n>> In the longer term, there are multiple possible action items.\n>> ...\n>>\n>> * We should teach hooks API to make it _optional_ to use the\n>> parallel subprocess API. If we are not spawning hooks in\n>> parallel today, there is no reason to incur this regression by\n>> using the parallel subprocess API---this was a needress bug, and\n>> I am angry.\n>\n> To counter, I think that having hooks invoked via two different\n> mechanisms depending on how many are provided or whether they are\n> parallelized is a mess to debug and maintain. I still stand by the\n> decision to use the parallel subprocess API, which I think was\n> reasonable to expect to do the same thing when jobs=1, and I think we\n> should continue to do so. It simplifies the hook code significantly.\n\nA simple code that does not behave as it should and causes end-user\nregression is not a code worth defending. Admitting it was a bad\nmove we made in the past is the first step to make it better.\n\nThe use of the parallel subprocess API in the hooks was prematurely\ndone, before we had clear use cases for running multiple hooks in\nparallel, and due to the lack of use cases, we didn't have chance to\nthink about the issues that need to be addressed before we can start\nusing the parallel subprocess API. The message you are responding to\nwas written with an explicit purpose of starting to list them.\n\n>> * the hooks API should learn a mechanism for multiple hooks to\n>> coordinate their executions. Perhaps they indicate their\n>> preference if they are OK to be run in parallel, and those that\n>> want isolation will be run one-at-a-time before or after others\n>> run in parallel, or something.\n>\n> There is such a mechanism for hooks overall, but not yet for\n> individual hooks. I know we discussed it at length[1] before, and\n\nThis...\n\n> decided it would be okay to figure this out later on. I suppose \"later\n> on\" may have come :)\n\nYes, besides patching up this regression for short term, I listed it\nas a possible ation item for the longer term.\n\n>> * The hooks API should learn a mechanism for us to tell what\n>> execution environment they are in. Ideally, the hooks, if it is\n>> sane to run under the parallel subprocess API, shouldn't have\n>> been learning if they are talking to an interactive human user by\n>> looking at isatty(), but we should have been explicitly telling\n>> them that they are, perhaps by exporting an environment\n>> variable. There may probably be more clue hooks writers want\n>> other than \"am I talking to human user?\" that we would want to\n>> enumerate before going this route.\n>\n> Hm. I was going to mention that Ævar and I discussed the possibility\n> of setting an environment variable for hook child processes, telling\n\nThat...\n\n> them which hook they are being run as - e.g.\n> \"GIT_HOOK=prepare-commit-msg\" - but I suppose that relying on that\n> alone doesn't tell us anything about whether the parent is being run\n> in tty. I agree it could be very useful to simply pass\n> GIT_PARENT_ISATTY to hooks (and I suppose other child processes).\n> Could we simply do that from start_command() or something else deep in\n> run-command.h machinery? Then Anthony's use case becomes\n>\n> if [-t 1|| GIT_PARENT_ISATTY]\n> ...\n>\n> and no need to examine Git version.\n\nBut DO NOT call it ISATTY. \"Are we showing the output to human\nend-users\" is the question it is answering to, and isatty() happens\nto be an implementation detail on POSIXy system.\n\n\"This\" and \"That\" above make it smell like discussion was done, but\neverybody got tired of discussing and the topic was shipped without\nnecessary polishment? That sounds like a process failure, which we\nmay want to address in the new development cycle, not limited to this\nparticular topic.\n\nThanks.\n"},{"id":"453989","messageId":"CAJoAoZnw6cNBwWpa5w-rhQ4p_zw6w6Q-NHzNeRKrrqPpDCjY2A@mail.gmail.com","threadId":"57764","inReplyTo":"xmqqilr3r7ki.fsf@gitster.g","subject":"Re: git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2022-04-20T17:41:49Z","receivedAt":"2022-04-20T17:42:06Z","isPatch":false,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Wed, Apr 20, 2022 at 10:25 AM Junio C Hamano wrote:\n>\n> Emily Shaffer writes:\n>\n> >> In the longer term, there are multiple possible action items.\n> >> ...\n> >>\n> >> * We should teach hooks API to make it _optional_ to use the\n> >> parallel subprocess API. If we are not spawning hooks in\n> >> parallel today, there is no reason to incur this regression by\n> >> using the parallel subprocess API---this was a needress bug, and\n> >> I am angry.\n> >\n> > To counter, I think that having hooks invoked via two different\n> > mechanisms depending on how many are provided or whether they are\n> > parallelized is a mess to debug and maintain. I still stand by the\n> > decision to use the parallel subprocess API, which I think was\n> > reasonable to expect to do the same thing when jobs=1, and I think we\n> > should continue to do so. It simplifies the hook code significantly.\n>\n> A simple code that does not behave as it should and causes end-user\n> regression is not a code worth defending. Admitting it was a bad\n> move we made in the past is the first step to make it better.\n\nI am also sorry that this use case was broken. However, I don't see\nthat it's documented in 'git help githooks' or elsewhere that we\nguarantee isatty() (or similar) of hooks matches that of the parent\nprocess. I think it is an accident that this worked before, and not\nsomething that was guaranteed by Git documentation - for example, we\nalso do not have regression tests ensuring that behavior for hooks\ntoday, either, or else we would not be having this conversation. (If I\nsimply missed the documentation promising that behavior, then I am\nsorry, and please point me to it.)\n\n>\n> The use of the parallel subprocess API in the hooks was prematurely\n> done, before we had clear use cases for running multiple hooks in\n> parallel, and due to the lack of use cases, we didn't have chance to\n> think about the issues that need to be addressed before we can start\n> using the parallel subprocess API. The message you are responding to\n> was written with an explicit purpose of starting to list them.\n>\n> >> * the hooks API should learn a mechanism for multiple hooks to\n> >> coordinate their executions. Perhaps they indicate their\n> >> preference if they are OK to be run in parallel, and those that\n> >> want isolation will be run one-at-a-time before or after others\n> >> run in parallel, or something.\n> >\n> > There is such a mechanism for hooks overall, but not yet for\n> > individual hooks. I know we discussed it at length[1] before, and\n>\n> This...\n>\n> > decided it would be okay to figure this out later on. I suppose \"later\n> > on\" may have come :)\n>\n> Yes, besides patching up this regression for short term, I listed it\n> as a possible ation item for the longer term.\n>\n> >> * The hooks API should learn a mechanism for us to tell what\n> >> execution environment they are in. Ideally, the hooks, if it is\n> >> sane to run under the parallel subprocess API, shouldn't have\n> >> been learning if they are talking to an interactive human user by\n> >> looking at isatty(), but we should have been explicitly telling\n> >> them that they are, perhaps by exporting an environment\n> >> variable. There may probably be more clue hooks writers want\n> >> other than \"am I talking to human user?\" that we would want to\n> >> enumerate before going this route.\n> >\n> > Hm. I was going to mention that Ævar and I discussed the possibility\n> > of setting an environment variable for hook child processes, telling\n>\n> That...\n>\n> > them which hook they are being run as - e.g.\n> > \"GIT_HOOK=prepare-commit-msg\" - but I suppose that relying on that\n> > alone doesn't tell us anything about whether the parent is being run\n> > in tty. I agree it could be very useful to simply pass\n> > GIT_PARENT_ISATTY to hooks (and I suppose other child processes).\n> > Could we simply do that from start_command() or something else deep in\n> > run-command.h machinery? Then Anthony's use case becomes\n> >\n> > if [-t 1|| GIT_PARENT_ISATTY]\n> > ...\n> >\n> > and no need to examine Git version.\n>\n> But DO NOT call it ISATTY. \"Are we showing the output to human\n> end-users\" is the question it is answering to, and isatty() happens\n> to be an implementation detail on POSIXy system.\n>\n> \"This\" and \"That\" above make it smell like discussion was done, but\n> everybody got tired of discussing and the topic was shipped without\n> necessary polishment? That sounds like a process failure, which we\n> may want to address in the new development cycle, not limited to this\n> particular topic.\n\nI think, rather, during discussion we said \"without knowing how real\nusers want to use hooks, it's not possible for us to make a good\ndesign for individual hooks to state whether they need to be\nparallelized or not.\" Perhaps that means this body of work should have\nstayed in 'next' longer, rather than making it to a release?\n\nFor what it's worth, Google internally has been using multiple hooks\nvia config for something like a year, with this design, from a\ncombination of 'next' and pending hooks patches. But we haven't\nimagined the need to color hook output for users and check isatty() or\nsimilar. I think there are not many other consumers of 'next' besides\nthe Google internal release. So I'm not sure that longer time in\n'next' would have allowed us to see this issue, either.\n\n - Emily\n"},{"id":"454057","messageId":"220421.86sfq67hlr.gmgdl@evledraar.gmail.com","threadId":"57764","inReplyTo":"CAJoAoZnw6cNBwWpa5w-rhQ4p_zw6w6Q-NHzNeRKrrqPpDCjY2A@mail.gmail.com","subject":"Re: git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-04-21T12:03:12Z","receivedAt":"2022-04-21T12:20:56Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Apr 20 2022, Emily Shaffer wrote:\n\n[I'll reply to most of this & other questions in the form of patches,\njust on some of this]\n\n> On Wed, Apr 20, 2022 at 10:25 AM Junio C Hamano wrote:\n>>\n>> Emily Shaffer writes:\n>>\n>> >> In the longer term, there are multiple possible action items.\n>> >> ...\n>> >>\n>> >> * We should teach hooks API to make it _optional_ to use the\n>> >> parallel subprocess API. If we are not spawning hooks in\n>> >> parallel today, there is no reason to incur this regression by\n>> >> using the parallel subprocess API---this was a needress bug, and\n>> >> I am angry.\n>> >\n>> > To counter, I think that having hooks invoked via two different\n>> > mechanisms depending on how many are provided or whether they are\n>> > parallelized is a mess to debug and maintain. I still stand by the\n>> > decision to use the parallel subprocess API, which I think was\n>> > reasonable to expect to do the same thing when jobs=1, and I think we\n>> > should continue to do so. It simplifies the hook code significantly.\n>>\n>> A simple code that does not behave as it should and causes end-user\n>> regression is not a code worth defending. Admitting it was a bad\n>> move we made in the past is the first step to make it better.\n>\n> I am also sorry that this use case was broken. However, I don't see\n> that it's documented in 'git help githooks' or elsewhere that we\n> guarantee isatty() (or similar) of hooks matches that of the parent\n> process. I think it is an accident that this worked before, and not\n> something that was guaranteed by Git documentation - for example, we\n> also do not have regression tests ensuring that behavior for hooks\n> today, either, or else we would not be having this conversation. (If I\n> simply missed the documentation promising that behavior, then I am\n> sorry, and please point me to it.)\n\nYou're correct that it wasn't documented, and as regressions go that\nmakes it *slightly* better. I.e. at least it's not a publicly documented\npromise.\n\nAnyone using this part of the interface would have discovered it by\nexperimentation, or (reasonably) assumed that git was invoking the hook\nwithout any special redirection or buffering.\n\nAnd you're also right that we didn't have any test coverage for this,\nactually before the t/t1800-hook.sh we didn't have any test coverage at\nall on stdout_to_stderr for hooks (at least those converted to the API\nso far), which is pretty fundimental.\n\nBut none of that (except perhaps the doc omission) makes this any less\nof a regression. We don't have 100% test coverage, and can't assume that\njust because something isn't documented or tested for that it's not\nbeing relied on in the wild. It is, as this upthread report indicates.\n\nIn this case \"100% test coverage\" in the \"make coverage\" sense wouldn't\nhelp, this is part of 200% test coverage. I.e. it's in how an external\nuser expects to use and interact with the command. So it can remain\nuncovered even if our own tests touch 100% of our own code.\n\n> [...]\n>> > Hm. I was going to mention that Ævar and I discussed the possibility\n>> > of setting an environment variable for hook child processes, telling\n>>\n>> That...\n>>\n>> > them which hook they are being run as - e.g.\n>> > \"GIT_HOOK=prepare-commit-msg\" - but I suppose that relying on that\n>> > alone doesn't tell us anything about whether the parent is being run\n>> > in tty. I agree it could be very useful to simply pass\n>> > GIT_PARENT_ISATTY to hooks (and I suppose other child processes).\n>> > Could we simply do that from start_command() or something else deep in\n>> > run-command.h machinery? Then Anthony's use case becomes\n>> >\n>> > if [-t 1|| GIT_PARENT_ISATTY]\n>> > ...\n>> >\n>> > and no need to examine Git version.\n\nJust to clarify this a bit, we discussed passing down GIT_HOOK so that\nyou could e.g. symlink all your hooks and dispatch to some \"hook\nrouter\".\n\nWhich right now you can do with the file-based hooks, because you'll\nneed to symlink them to such a router, but couldn't with future\nconfig-based hooks.\n\nIOW it's entirely separate conceptually from a \"how does this hook\nexpect to behave\" vis-a-vis calling isatty() or whatever. It would just\nbe working around or own implementation details, i.e. whether we invoke\na path or a configured command.\n\n>> But DO NOT call it ISATTY. \"Are we showing the output to human\n>> end-users\" is the question it is answering to, and isatty() happens\n>> to be an implementation detail on POSIXy system.\n>>\n>> \"This\" and \"That\" above make it smell like discussion was done, but\n>> everybody got tired of discussing and the topic was shipped without\n>> necessary polishment? That sounds like a process failure, which we\n>> may want to address in the new development cycle, not limited to this\n>> particular topic.\n>\n> I think, rather, during discussion we said \"without knowing how real\n> users want to use hooks, it's not possible for us to make a good\n> design for individual hooks to state whether they need to be\n> parallelized or not.\" Perhaps that means this body of work should have\n> stayed in 'next' longer, rather than making it to a release?\n>\n> For what it's worth, Google internally has been using multiple hooks\n> via config for something like a year, with this design, from a\n> combination of 'next' and pending hooks patches. But we haven't\n> imagined the need to color hook output for users and check isatty() or\n> similar. I think there are not many other consumers of 'next' besides\n> the Google internal release. So I'm not sure that longer time in\n> 'next' would have allowed us to see this issue, either.\n\nWe're both thoroughly \"on inside\" of this particular process failure, so\nwe're both bound to have biases here.\n\nBut having said that I agree with you here. I.e. as a mechanism for\nmitigating mistakes and catching obscure edge cases just being more\ncareful or having things sit in 'next' for longer has, I think, proved\nitself to not be an effective method (not just in this case, but a few\nsimilar cases).\n\nI'm not sure what the solution is exactly, but I'm pretty sure it\ninvolves more controlled exposure to the wild (e.g. shipping certain\nthings as feature flags first), not deferring that exposure for long\nperiods, which is what having things sit it \"next\" for longer amounts\nto.\n"},{"id":"454058","messageId":"cover-0.6-00000000000-20220421T122108Z-avarab@gmail.com","threadId":"57764","inReplyTo":"CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com","subject":"[PATCH 0/6] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-04-21T12:25:25Z","receivedAt":"2022-04-21T12:25:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This fixes the regression reported by Anthony Sottile[1] with hooks\nnot being connected to a TTY. See 6/6 for details.\n\nIt would also have been possible to rip the\nrun_processes_parallel_tr2() out of hook.c, as we currently only use\nit for nproc=1. However it's the plan to have it run multiple hooks,\nand as 3/6 argues it's a good idea in general for our parallel\nexecution API to learn a mode similar to GNU parallel's \"--ungroup\",\neven though at the conclusion of this series the hook API is its only\nuser.\n\n1. https://lore.kernel.org/git/CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com/\n\nÆvar Arnfjörð Bjarmason (6):\n run-command API: replace run_processes_parallel_tr2() with opts struct\n run-command tests: test stdout of run_command_parallel()\n run-command: add an \"ungroup\" option to run_process_parallel()\n hook tests: fix redirection logic error in 96e7225b310\n hook API: don't redundantly re-set \"no_stdin\" and \"stdout_to_stderr\"\n hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n\n builtin/fetch.c | 15 ++--\n builtin/submodule--helper.c | 12 ++--\n hook.c | 19 ++---\n run-command.c | 135 +++++++++++++++++++++++++++---------\n run-command.h | 56 +++++++++++----\n submodule.c | 13 ++--\n t/helper/test-run-command.c | 44 ++++++++----\n t/t0061-run-command.sh | 45 ++++++++++--\n t/t1800-hook.sh | 39 ++++++++++-\n 9 files changed, 287 insertions(+), 91 deletions(-)\n\n-- \n2.36.0.893.g80a51c675f6\n\n"},{"id":"454059","messageId":"patch-1.6-8bf71ce63dd-20220421T122108Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-0.6-00000000000-20220421T122108Z-avarab@gmail.com","subject":"[PATCH 1/6] run-command API: replace run_processes_parallel_tr2() with opts struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-04-21T12:25:26Z","receivedAt":"2022-04-21T12:25:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a new \"struct run_process_parallel_opts\" to cover the trace2\nuse-case added in ee4512ed481 (trace2: create new combined trace\nfacility, 2019-02-22). A subsequent commit will add more options, and\nhaving a proliferation of new functions or extra parameters would\nresult in needless churn.\n\nIt makes for a smaller change to make run_processes_parallel() and\nrun_processes_parallel_tr2() wrapper functions for the new \"static\"\nrun_processes_parallel_1(), which contains the main logic. We pass\ndown \"opts\" to the *_1() function even though it isn't used there\nyet (only in the *_tr2() function), a subsequent commit will make more\nuse of it.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fetch.c | 15 ++++++++------\n builtin/submodule--helper.c | 12 +++++++----\n hook.c | 13 ++++++------\n run-command.c | 40 +++++++++++++++++++++++++++----------\n run-command.h | 26 ++++++++++++++++--------\n submodule.c | 13 ++++++------\n t/helper/test-run-command.c | 13 ++++++------\n 7 files changed, 84 insertions(+), 48 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex e3791f09ed5..9bc99183191 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1948,14 +1948,17 @@ static int fetch_multiple(struct string_list *list, int max_children)\n \n \tif (max_children != 1 && list->nr != 1) {\n \t\tstruct parallel_fetch_state state = { argv.v, list, 0, 0 };\n+\t\tstruct run_process_parallel_opts run_opts = {\n+\t\t\t.tr2_category = \"fetch\",\n+\t\t\t.tr2_label = \"parallel/fetch\",\n+\t\t};\n \n \t\tstrvec_push(&argv, \"--end-of-options\");\n-\t\tresult = run_processes_parallel_tr2(max_children,\n-\t\t\t\t\t\t &fetch_next_remote,\n-\t\t\t\t\t\t &fetch_failed_to_start,\n-\t\t\t\t\t\t &fetch_finished,\n-\t\t\t\t\t\t &state,\n-\t\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n+\t\tresult = run_processes_parallel(max_children,\n+\t\t\t\t\t\t&fetch_next_remote,\n+\t\t\t\t\t\t&fetch_failed_to_start,\n+\t\t\t\t\t\t&fetch_finished, &state,\n+\t\t\t\t\t\t&run_opts);\n \n \t\tif (!result)\n \t\t\tresult = state.result;\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 2c87ef9364f..c3d1aace546 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -2652,12 +2652,16 @@ static int update_submodules(struct update_data *update_data)\n {\n \tint i, res = 0;\n \tstruct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;\n+\tstruct run_process_parallel_opts run_opts = {\n+\t\t.tr2_category = \"submodule\",\n+\t\t.tr2_label = \"parallel/update\",\n+\t};\n \n \tsuc.update_data = update_data;\n-\trun_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,\n-\t\t\t\t update_clone_start_failure,\n-\t\t\t\t update_clone_task_finished, &suc, \"submodule\",\n-\t\t\t\t \"parallel/update\");\n+\trun_processes_parallel(suc.update_data->max_jobs,\n+\t\t\t update_clone_get_next_task,\n+\t\t\t update_clone_start_failure,\n+\t\t\t update_clone_task_finished, &suc, &run_opts);\n \n \t/*\n \t * We saved the output and put it out all at once now.\ndiff --git a/hook.c b/hook.c\nindex 1d51be3b77a..eadb2d58a7b 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -123,6 +123,10 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \tconst char *const hook_path = find_hook(hook_name);\n \tint jobs = 1;\n \tint ret = 0;\n+\tstruct run_process_parallel_opts run_opts = {\n+\t\t.tr2_category = \"hook\",\n+\t\t.tr2_label = hook_name,\n+\t};\n \n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n@@ -144,13 +148,8 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \t\tcb_data.hook_path = abs_path.buf;\n \t}\n \n-\trun_processes_parallel_tr2(jobs,\n-\t\t\t\t pick_next_hook,\n-\t\t\t\t notify_start_failure,\n-\t\t\t\t notify_hook_finished,\n-\t\t\t\t &cb_data,\n-\t\t\t\t \"hook\",\n-\t\t\t\t hook_name);\n+\trun_processes_parallel(jobs, pick_next_hook, notify_start_failure,\n+\t\t\t notify_hook_finished, &cb_data, &run_opts);\n \tret = cb_data.rc;\n cleanup:\n \tstrbuf_release(&abs_path);\ndiff --git a/run-command.c b/run-command.c\nindex a8501e38ceb..7b8159aa235 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1738,11 +1738,11 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n-int run_processes_parallel(int n,\n-\t\t\t get_next_task_fn get_next_task,\n-\t\t\t start_failure_fn start_failure,\n-\t\t\t task_finished_fn task_finished,\n-\t\t\t void *pp_cb)\n+static int run_processes_parallel_1(int n, get_next_task_fn get_next_task,\n+\t\t\t\t start_failure_fn start_failure,\n+\t\t\t\t task_finished_fn task_finished,\n+\t\t\t\t void *pp_cb,\n+\t\t\t\t struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n \tint output_timeout = 100;\n@@ -1780,24 +1780,42 @@ int run_processes_parallel(int n,\n \treturn 0;\n }\n \n-int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n-\t\t\t start_failure_fn start_failure,\n-\t\t\t task_finished_fn task_finished, void *pp_cb,\n-\t\t\t const char *tr2_category, const char *tr2_label)\n+static int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n+\t\t\t\t start_failure_fn start_failure,\n+\t\t\t\t task_finished_fn task_finished,\n+\t\t\t\t void *pp_cb,\n+\t\t\t\t struct run_process_parallel_opts *opts)\n {\n+\tconst char *tr2_category = opts->tr2_category;\n+\tconst char *tr2_label = opts->tr2_label;\n \tint result;\n \n \ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n \t\t\t\t ((n < 1) ? online_cpus() : n));\n \n-\tresult = run_processes_parallel(n, get_next_task, start_failure,\n-\t\t\t\t\ttask_finished, pp_cb);\n+\tresult = run_processes_parallel_1(n, get_next_task, start_failure,\n+\t\t\t\t\t task_finished, pp_cb, opts);\n \n \ttrace2_region_leave(tr2_category, tr2_label, NULL);\n \n \treturn result;\n }\n \n+int run_processes_parallel(int n, get_next_task_fn get_next_task,\n+\t\t\t start_failure_fn start_failure,\n+\t\t\t task_finished_fn task_finished, void *pp_cb,\n+\t\t\t struct run_process_parallel_opts *opts)\n+{\n+\tif (opts->tr2_category && opts->tr2_label)\n+\t\treturn run_processes_parallel_tr2(n, get_next_task,\n+\t\t\t\t\t\t start_failure, task_finished,\n+\t\t\t\t\t\t pp_cb, opts);\n+\n+\treturn run_processes_parallel_1(n, get_next_task, start_failure,\n+\t\t\t\t\ttask_finished, pp_cb, opts);\n+}\n+\n+\n int run_auto_maintenance(int quiet)\n {\n \tint enabled;\ndiff --git a/run-command.h b/run-command.h\nindex 07bed6c31b4..66e7bebd88a 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -458,6 +458,19 @@ typedef int (*task_finished_fn)(int result,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * Options to pass to run_processes_parallel(), { 0 }-initialized\n+ * means no options. Fields:\n+ *\n+ * tr2_category & tr2_label: sets the trace2 category and label for\n+ * logging. These must either be unset, or both of them must be set.\n+ */\n+struct run_process_parallel_opts\n+{\n+\tconst char *tr2_category;\n+\tconst char *tr2_label;\n+};\n+\n /**\n * Runs up to n processes at the same time. Whenever a process can be\n * started, the callback get_next_task_fn is called to obtain the data\n@@ -469,15 +482,12 @@ typedef int (*task_finished_fn)(int result,\n *\n * start_failure_fn and task_finished_fn can be NULL to omit any\n * special handling.\n+ *\n+ * Options are passed via a \"struct run_process_parallel_opts\".\n */\n-int run_processes_parallel(int n,\n-\t\t\t get_next_task_fn,\n-\t\t\t start_failure_fn,\n-\t\t\t task_finished_fn,\n-\t\t\t void *pp_cb);\n-int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n-\t\t\t task_finished_fn, void *pp_cb,\n-\t\t\t const char *tr2_category, const char *tr2_label);\n+int run_processes_parallel(int n, get_next_task_fn, start_failure_fn,\n+\t\t\t task_finished_fn, void *pp_cb,\n+\t\t\t struct run_process_parallel_opts *opts);\n \n /**\n * Convenience function which prepares env_array for a command to be run in a\ndiff --git a/submodule.c b/submodule.c\nindex 86c8f0f89db..256c6bb4b8f 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1817,6 +1817,10 @@ int fetch_submodules(struct repository *r,\n {\n \tint i;\n \tstruct submodule_parallel_fetch spf = SPF_INIT;\n+\tstruct run_process_parallel_opts run_opts = {\n+\t\t.tr2_category = \"submodule\",\n+\t\t.tr2_label = \"parallel/fetch\",\n+\t};\n \n \tspf.r = r;\n \tspf.command_line_option = command_line_option;\n@@ -1838,12 +1842,9 @@ int fetch_submodules(struct repository *r,\n \n \tcalculate_changed_submodule_paths(r, &spf.changed_submodule_names);\n \tstring_list_sort(&spf.changed_submodule_names);\n-\trun_processes_parallel_tr2(max_parallel_jobs,\n-\t\t\t\t get_next_submodule,\n-\t\t\t\t fetch_start_failure,\n-\t\t\t\t fetch_finish,\n-\t\t\t\t &spf,\n-\t\t\t\t \"submodule\", \"parallel/fetch\");\n+\trun_processes_parallel(max_parallel_jobs, get_next_submodule,\n+\t\t\t fetch_start_failure, fetch_finish, &spf,\n+\t\t\t &run_opts);\n \n \tif (spf.submodules_with_errors.len > 0)\n \t\tfprintf(stderr, _(\"Errors during submodule fetch:\\n%s\"),\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex f3b90aa834a..9b21f2f9f83 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -183,7 +183,7 @@ static int testsuite(int argc, const char **argv)\n \t\t(uintmax_t)suite.tests.nr, max_jobs);\n \n \tret = run_processes_parallel(max_jobs, next_test, test_failed,\n-\t\t\t\t test_finished, &suite);\n+\t\t\t\t test_finished, &suite, NULL);\n \n \tif (suite.failed.nr > 0) {\n \t\tret = 1;\n@@ -371,6 +371,7 @@ int cmd__run_command(int argc, const char **argv)\n {\n \tstruct child_process proc = CHILD_PROCESS_INIT;\n \tint jobs;\n+\tstruct run_process_parallel_opts opts = { 0 };\n \n \tif (argc > 1 && !strcmp(argv[1], \"testsuite\"))\n \t\texit(testsuite(argc - 1, argv + 1));\n@@ -413,15 +414,15 @@ int cmd__run_command(int argc, const char **argv)\n \n \tif (!strcmp(argv[1], \"run-command-parallel\"))\n \t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, NULL, &proc));\n+\t\t\t\t\t NULL, NULL, &proc, &opts));\n \n \tif (!strcmp(argv[1], \"run-command-abort\"))\n-\t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, task_finished, &proc));\n+\t\texit(run_processes_parallel(jobs, parallel_next, NULL,\n+\t\t\t\t\t task_finished, &proc, &opts));\n \n \tif (!strcmp(argv[1], \"run-command-no-jobs\"))\n-\t\texit(run_processes_parallel(jobs, no_job,\n-\t\t\t\t\t NULL, task_finished, &proc));\n+\t\texit(run_processes_parallel(jobs, no_job, NULL, task_finished,\n+\t\t\t\t\t &proc, &opts));\n \n \tfprintf(stderr, \"check usage\\n\");\n \treturn 1;\n-- \n2.36.0.893.g80a51c675f6\n\n"},{"id":"454060","messageId":"patch-2.6-d9c9b158130-20220421T122108Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-0.6-00000000000-20220421T122108Z-avarab@gmail.com","subject":"[PATCH 2/6] run-command tests: test stdout of run_command_parallel()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-04-21T12:25:27Z","receivedAt":"2022-04-21T12:25:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Extend the tests added in c553c72eed6 (run-command: add an\nasynchronous parallel child processor, 2015-12-15) to test stdout in\naddition to stderr. A subsequent commit will add additional related\ntests for a new feature, making it obvious how the output of the two\ncompares on both stdout and stderr will make this easier to reason\nabout.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t0061-run-command.sh | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex ee281909bc3..131fcfda90f 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -130,17 +130,20 @@ World\n EOF\n \n test_expect_success 'run_command runs in parallel with more jobs available than tasks' '\n-\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n+\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n+\ttest_must_be_empty out &&\n \ttest_cmp expect actual\n '\n \n test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n-\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n+\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n+\ttest_must_be_empty out &&\n \ttest_cmp expect actual\n '\n \n test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n-\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n+\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n+\ttest_must_be_empty out &&\n \ttest_cmp expect actual\n '\n \n@@ -154,7 +157,8 @@ asking for a quick stop\n EOF\n \n test_expect_success 'run_command is asked to abort gracefully' '\n-\ttest-tool run-command run-command-abort 3 false 2>actual &&\n+\ttest-tool run-command run-command-abort 3 false >out 2>actual &&\n+\ttest_must_be_empty out &&\n \ttest_cmp expect actual\n '\n \n@@ -163,7 +167,8 @@ no further jobs available\n EOF\n \n test_expect_success 'run_command outputs ' '\n-\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n+\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n+\ttest_must_be_empty out &&\n \ttest_cmp expect actual\n '\n \n-- \n2.36.0.893.g80a51c675f6\n\n"},{"id":"454061","messageId":"patch-6.6-de3664f6d2b-20220421T122108Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-0.6-00000000000-20220421T122108Z-avarab@gmail.com","subject":"[PATCH 6/6] hook API: fix v2.36.0 regression: hooks should be connected to a TTY","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-04-21T12:25:31Z","receivedAt":"2022-04-21T12:25:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a regression reported[1] in f443246b9f2 (commit: convert\n{pre-commit,prepare-commit-msg} hook to hook.h, 2021-12-22): Due to\nusing the run_process_parallel() API in the earlier 96e7225b310 (hook:\nadd 'run' subcommand, 2021-12-22) we'd capture the hook's stderr and\nstdout, and thus lose the connection to the TTY in the case of\ne.g. the \"pre-commit\" hook.\n\nAs a preceding commit notes GNU parallel's similar --ungroup option\nalso has it emit output faster. While we're unlikely to have hooks\nthat emit truly massive amounts of output (or where the performance\nthereof matters) it's still informative to measure the overhead. In a\nsimilar \"seq\" test we're now ~30% faster:\n\n\t$ cat .git/hooks/seq-hook; git hyperfine -L rev origin/master,HEAD~0 -s 'make CFLAGS=-O3' './git hook run seq-hook'\n\t#!/bin/sh\n\n\tseq 100000000\n\tBenchmark 1: ./git hook run seq-hook' in 'origin/master\n\t Time (mean ± σ): 787.1 ms ± 13.6 ms [User: 701.6 ms, System: 534.4 ms]\n\t Range (min … max): 773.2 ms … 806.3 ms 10 runs\n\n\tBenchmark 2: ./git hook run seq-hook' in 'HEAD~0\n\t Time (mean ± σ): 603.4 ms ± 1.6 ms [User: 573.1 ms, System: 30.3 ms]\n\t Range (min … max): 601.0 ms … 606.2 ms 10 runs\n\n\tSummary\n\t './git hook run seq-hook' in 'HEAD~0' ran\n\t 1.30 ± 0.02 times faster than './git hook run seq-hook' in 'origin/master'\n\nIn the preceding commit we removed the \"no_stdin=1\" and\n\"stdout_to_stderr=1\" assignments. This change brings them back as with\n\".ungroup=1\" the run_process_parallel() function doesn't provide them\nfor us implicitly.\n\nAs an aside omitting the stdout_to_stderr=1 here would have all tests\npass, except those that test \"git hook run\" itself in\nt1800-hook.sh. But our tests passing is the result of another test\nblind spot, as was the case with the regression being fixed here. The\n\"stdout_to_stderr=1\" for hooks is long-standing behavior, see\ne.g. 1d9e8b56fe3 (Split back out update_hook handling in receive-pack,\n2007-03-10) and other follow-up commits (running \"git log\" with\n\"--reverse -p -Gstdout_to_stderr\" is a good start).\n\n1. https://lore.kernel.org/git/CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com/\n\nReported-by: Anthony Sottile \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 8 +++++++-\n t/t1800-hook.sh | 37 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 44 insertions(+), 1 deletion(-)\n\ndiff --git a/hook.c b/hook.c\nindex 68ee4030551..f5eef1d561b 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -53,7 +53,9 @@ static int pick_next_hook(struct child_process *cp,\n \tif (!hook_path)\n \t\treturn 0;\n \n+\tcp->no_stdin = 1;\n \tstrvec_pushv(&cp->env_array, hook_cb->options->env.v);\n+\tcp->stdout_to_stderr = 1;\n \tcp->trace2_hook_name = hook_cb->hook_name;\n \tcp->dir = hook_cb->options->dir;\n \n@@ -119,16 +121,20 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \t\t.options = options,\n \t};\n \tconst char *const hook_path = find_hook(hook_name);\n-\tint jobs = 1;\n+\tconst int jobs = 1;\n \tint ret = 0;\n \tstruct run_process_parallel_opts run_opts = {\n \t\t.tr2_category = \"hook\",\n \t\t.tr2_label = hook_name,\n+\t\t.ungroup = jobs == 1,\n \t};\n \n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n \n+\tif (jobs != 1 || !run_opts.ungroup)\n+\t\tBUG(\"TODO: think about & document order & interleaving of parallel hook output\");\n+\n \tif (options->invoked_hook)\n \t\t*options->invoked_hook = 0;\n \ndiff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\nindex 1e4adc3d53e..f22754deccc 100755\n--- a/t/t1800-hook.sh\n+++ b/t/t1800-hook.sh\n@@ -4,6 +4,7 @@ test_description='git-hook command'\n \n TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-terminal.sh\n \n test_expect_success 'git hook usage' '\n \ttest_expect_code 129 git hook &&\n@@ -120,4 +121,40 @@ test_expect_success 'git -c core.hooksPath= hook run' '\n \ttest_cmp expect actual\n '\n \n+test_hook_tty() {\n+\tlocal fd=\"$1\" &&\n+\n+\tcat >expect &&\n+\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\n+\ttest_hook -C repo pre-commit <<-EOF &&\n+\t{\n+\t\ttest -t 1 && echo >&$fd STDOUT TTY || echo >&$fd STDOUT NO TTY &&\n+\t\ttest -t 2 && echo >&$fd STDERR TTY || echo >&$fd STDERR NO TTY\n+\t} $fd>actual\n+\tEOF\n+\n+\ttest_commit -C repo A &&\n+\ttest_commit -C repo B &&\n+\tgit -C repo reset --soft HEAD^ &&\n+\ttest_terminal git -C repo commit -m\"B.new\" &&\n+\ttest_cmp expect repo/actual\n+}\n+\n+test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDOUT redirect' '\n+\ttest_hook_tty 1 <<-\\EOF\n+\tSTDOUT NO TTY\n+\tSTDERR TTY\n+\tEOF\n+'\n+\n+test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDERR redirect' '\n+\ttest_hook_tty 2 <<-\\EOF\n+\tSTDOUT TTY\n+\tSTDERR NO TTY\n+\tEOF\n+'\n+\n test_done\n-- \n2.36.0.893.g80a51c675f6\n\n"},{"id":"454062","messageId":"patch-3.6-d76f63c2948-20220421T122108Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-0.6-00000000000-20220421T122108Z-avarab@gmail.com","subject":"[PATCH 3/6] run-command: add an \"ungroup\" option to run_process_parallel()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-04-21T12:25:28Z","receivedAt":"2022-04-21T12:25:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Extend the parallel execution API added in c553c72eed6 (run-command:\nadd an asynchronous parallel child processor, 2015-12-15) to support a\nmode where the stdout and stderr of the processes isn't captured and\noutput in a deterministic order, instead we'll leave it to the kernel\nand stdio to sort it out.\n\nThis gives the API same functionality as GNU parallel's --ungroup\noption. As we'll see in a subsequent commit the main reason to want\nthis is to support stdout and stderr being connected to the TTY in the\ncase of jobs=1, demonstrated here with GNU parallel:\n\n\t$ parallel --ungroup 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n\tTTY\n\tTTY\n\t$ parallel 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n\tNTTY\n\tNTTY\n\nAnother is as GNU parallel's documentation notes a potential for\noptimization. Our results will be a bit different, but in cases where\nyou want to run processes in parallel where the exact order isn't\nimportant this can be a lot faster:\n\n\t$ hyperfine -r 3 -L o ,--ungroup 'parallel {o} seq ::: 10000000 >/dev/null '\n\tBenchmark 1: parallel seq ::: 10000000 >/dev/null\n\t Time (mean ± σ): 220.2 ms ± 9.3 ms [User: 124.9 ms, System: 96.1 ms]\n\t Range (min … max): 212.3 ms … 230.5 ms 3 runs\n\n\tBenchmark 2: parallel --ungroup seq ::: 10000000 >/dev/null\n\t Time (mean ± σ): 154.7 ms ± 0.9 ms [User: 136.2 ms, System: 25.1 ms]\n\t Range (min … max): 153.9 ms … 155.7 ms 3 runs\n\n\tSummary\n\t 'parallel --ungroup seq ::: 10000000 >/dev/null ' ran\n\t 1.42 ± 0.06 times faster than 'parallel seq ::: 10000000 >/dev/null '\n\nA large part of the juggling in the API is to make the API safer for\nits maintenance and consumers alike.\n\nFor the maintenance of the API we e.g. avoid malloc()-ing the\n\"pp->pfd\", ensuring that SANITIZE=address and other similar tools will\ncatch any unexpected misuse.\n\nFor API consumers we take pains to never pass the non-NULL \"out\"\nbuffer to an API user that provided the \"ungroup\" option. The\nresulting code in t/helper/test-run-command.c isn't typical of such a\nuser, i.e. they'd typically use one mode or the other, and would know\nwhether they'd provided \"ungroup\" or not.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 95 ++++++++++++++++++++++++++++---------\n run-command.h | 32 +++++++++----\n t/helper/test-run-command.c | 31 +++++++++---\n t/t0061-run-command.sh | 30 ++++++++++++\n 4 files changed, 151 insertions(+), 37 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 7b8159aa235..873de21ffaf 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1468,7 +1468,7 @@ int pipe_command(struct child_process *cmd,\n enum child_state {\n \tGIT_CP_FREE,\n \tGIT_CP_WORKING,\n-\tGIT_CP_WAIT_CLEANUP,\n+\tGIT_CP_WAIT_CLEANUP, /* only for !ungroup */\n };\n \n struct parallel_processes {\n@@ -1494,6 +1494,7 @@ struct parallel_processes {\n \tstruct pollfd *pfd;\n \n \tunsigned shutdown : 1;\n+\tunsigned ungroup:1;\n \n \tint output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n@@ -1537,8 +1538,9 @@ static void pp_init(struct parallel_processes *pp,\n \t\t get_next_task_fn get_next_task,\n \t\t start_failure_fn start_failure,\n \t\t task_finished_fn task_finished,\n-\t\t void *data)\n+\t\t void *data, struct run_process_parallel_opts *opts)\n {\n+\tconst int ungroup = opts->ungroup;\n \tint i;\n \n \tif (n < 1)\n@@ -1556,16 +1558,22 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->start_failure = start_failure ? start_failure : default_start_failure;\n \tpp->task_finished = task_finished ? task_finished : default_task_finished;\n \n+\tpp->ungroup = ungroup;\n+\n \tpp->nr_processes = 0;\n \tpp->output_owner = 0;\n \tpp->shutdown = 0;\n \tCALLOC_ARRAY(pp->children, n);\n-\tCALLOC_ARRAY(pp->pfd, n);\n+\tif (!ungroup)\n+\t\tCALLOC_ARRAY(pp->pfd, n);\n+\n \tstrbuf_init(&pp->buffered_output, 0);\n \n \tfor (i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n \t\tchild_process_init(&pp->children[i].process);\n+\t\tif (ungroup)\n+\t\t\tcontinue;\n \t\tpp->pfd[i].events = POLLIN | POLLHUP;\n \t\tpp->pfd[i].fd = -1;\n \t}\n@@ -1576,6 +1584,7 @@ static void pp_init(struct parallel_processes *pp,\n \n static void pp_cleanup(struct parallel_processes *pp)\n {\n+\tconst int ungroup = pp->ungroup;\n \tint i;\n \n \ttrace_printf(\"run_processes_parallel: done\");\n@@ -1585,14 +1594,17 @@ static void pp_cleanup(struct parallel_processes *pp)\n \t}\n \n \tfree(pp->children);\n-\tfree(pp->pfd);\n+\tif (!ungroup)\n+\t\tfree(pp->pfd);\n \n \t/*\n \t * When get_next_task added messages to the buffer in its last\n \t * iteration, the buffered output is non empty.\n \t */\n-\tstrbuf_write(&pp->buffered_output, stderr);\n-\tstrbuf_release(&pp->buffered_output);\n+\tif (!ungroup) {\n+\t\tstrbuf_write(&pp->buffered_output, stderr);\n+\t\tstrbuf_release(&pp->buffered_output);\n+\t}\n \n \tsigchain_pop_common();\n }\n@@ -1606,6 +1618,7 @@ static void pp_cleanup(struct parallel_processes *pp)\n */\n static int pp_start_one(struct parallel_processes *pp)\n {\n+\tconst int ungroup = pp->ungroup;\n \tint i, code;\n \n \tfor (i = 0; i < pp->max_processes; i++)\n@@ -1615,24 +1628,31 @@ static int pp_start_one(struct parallel_processes *pp)\n \t\tBUG(\"bookkeeping is hard\");\n \n \tcode = pp->get_next_task(&pp->children[i].process,\n-\t\t\t\t &pp->children[i].err,\n+\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n \t\t\t\t pp->data,\n \t\t\t\t &pp->children[i].data);\n \tif (!code) {\n-\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n-\t\tstrbuf_reset(&pp->children[i].err);\n+\t\tif (!ungroup) {\n+\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n+\t\t\tstrbuf_reset(&pp->children[i].err);\n+\t\t}\n \t\treturn 1;\n \t}\n-\tpp->children[i].process.err = -1;\n-\tpp->children[i].process.stdout_to_stderr = 1;\n-\tpp->children[i].process.no_stdin = 1;\n+\n+\tif (!ungroup) {\n+\t\tpp->children[i].process.err = -1;\n+\t\tpp->children[i].process.stdout_to_stderr = 1;\n+\t\tpp->children[i].process.no_stdin = 1;\n+\t}\n \n \tif (start_command(&pp->children[i].process)) {\n-\t\tcode = pp->start_failure(&pp->children[i].err,\n+\t\tcode = pp->start_failure(ungroup ? NULL : &pp->children[i].err,\n \t\t\t\t\t pp->data,\n \t\t\t\t\t pp->children[i].data);\n-\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n-\t\tstrbuf_reset(&pp->children[i].err);\n+\t\tif (!ungroup) {\n+\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n+\t\t\tstrbuf_reset(&pp->children[i].err);\n+\t\t}\n \t\tif (code)\n \t\t\tpp->shutdown = 1;\n \t\treturn code;\n@@ -1640,14 +1660,26 @@ static int pp_start_one(struct parallel_processes *pp)\n \n \tpp->nr_processes++;\n \tpp->children[i].state = GIT_CP_WORKING;\n-\tpp->pfd[i].fd = pp->children[i].process.err;\n+\tif (!ungroup)\n+\t\tpp->pfd[i].fd = pp->children[i].process.err;\n \treturn 0;\n }\n \n+static void pp_mark_working_for_cleanup(struct parallel_processes *pp)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < pp->max_processes; i++)\n+\t\tif (pp->children[i].state == GIT_CP_WORKING)\n+\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+}\n+\n static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n {\n \tint i;\n \n+\tassert(!pp->ungroup);\n+\n \twhile ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n \t\tif (errno == EINTR)\n \t\t\tcontinue;\n@@ -1674,6 +1706,9 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n static void pp_output(struct parallel_processes *pp)\n {\n \tint i = pp->output_owner;\n+\n+\tassert(!pp->ungroup);\n+\n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n \t\tstrbuf_write(&pp->children[i].err, stderr);\n@@ -1683,10 +1718,15 @@ static void pp_output(struct parallel_processes *pp)\n \n static int pp_collect_finished(struct parallel_processes *pp)\n {\n+\tconst int ungroup = pp->ungroup;\n \tint i, code;\n \tint n = pp->max_processes;\n \tint result = 0;\n \n+\tif (ungroup)\n+\t\tfor (i = 0; i < pp->max_processes; i++)\n+\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+\n \twhile (pp->nr_processes > 0) {\n \t\tfor (i = 0; i < pp->max_processes; i++)\n \t\t\tif (pp->children[i].state == GIT_CP_WAIT_CLEANUP)\n@@ -1697,8 +1737,8 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \t\tcode = finish_command(&pp->children[i].process);\n \n \t\tcode = pp->task_finished(code,\n-\t\t\t\t\t &pp->children[i].err, pp->data,\n-\t\t\t\t\t pp->children[i].data);\n+\t\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n+\t\t\t\t\t pp->data, pp->children[i].data);\n \n \t\tif (code)\n \t\t\tresult = code;\n@@ -1707,10 +1747,13 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \n \t\tpp->nr_processes--;\n \t\tpp->children[i].state = GIT_CP_FREE;\n-\t\tpp->pfd[i].fd = -1;\n+\t\tif (!ungroup)\n+\t\t\tpp->pfd[i].fd = -1;\n \t\tchild_process_init(&pp->children[i].process);\n \n-\t\tif (i != pp->output_owner) {\n+\t\tif (ungroup) {\n+\t\t\t/* no strbuf_*() work to do here */\n+\t\t} else if (i != pp->output_owner) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t} else {\n@@ -1744,12 +1787,14 @@ static int run_processes_parallel_1(int n, get_next_task_fn get_next_task,\n \t\t\t\t void *pp_cb,\n \t\t\t\t struct run_process_parallel_opts *opts)\n {\n+\tconst int ungroup = opts->ungroup;\n \tint i, code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n \tstruct parallel_processes pp;\n \n-\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n+\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n+\t\topts);\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1766,8 +1811,12 @@ static int run_processes_parallel_1(int n, get_next_task_fn get_next_task,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tpp_buffer_stderr(&pp, output_timeout);\n-\t\tpp_output(&pp);\n+\t\tif (ungroup) {\n+\t\t\tpp_mark_working_for_cleanup(&pp);\n+\t\t} else {\n+\t\t\tpp_buffer_stderr(&pp, output_timeout);\n+\t\t\tpp_output(&pp);\n+\t\t}\n \t\tcode = pp_collect_finished(&pp);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\ndiff --git a/run-command.h b/run-command.h\nindex 66e7bebd88a..936d334eee0 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -406,6 +406,10 @@ void check_pipe(int err);\n * pp_cb is the callback cookie as passed to run_processes_parallel.\n * You can store a child process specific callback cookie in pp_task_cb.\n *\n+ * The \"struct strbuf *err\" parameter is either a pointer to a string\n+ * to write errors to, or NULL if the \"ungroup\" option was\n+ * provided. See run_processes_parallel() below.\n+ *\n * Even after returning 0 to indicate that there are no more processes,\n * this function will be called again until there are no more running\n * child processes.\n@@ -424,9 +428,9 @@ typedef int (*get_next_task_fn)(struct child_process *cp,\n * This callback is called whenever there are problems starting\n * a new process.\n *\n- * You must not write to stdout or stderr in this function. Add your\n- * message to the strbuf out instead, which will be printed without\n- * messing up the output of the other parallel processes.\n+ * The \"struct strbuf *err\" parameter is either a pointer to a string\n+ * to write errors to, or NULL if the \"ungroup\" option was\n+ * provided. See run_processes_parallel() below.\n *\n * pp_cb is the callback cookie as passed into run_processes_parallel,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n@@ -442,9 +446,9 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n /**\n * This callback is called on every child process that finished processing.\n *\n- * You must not write to stdout or stderr in this function. Add your\n- * message to the strbuf out instead, which will be printed without\n- * messing up the output of the other parallel processes.\n+ * The \"struct strbuf *err\" parameter is either a pointer to a string\n+ * to write errors to, or NULL if the \"ungroup\" option was\n+ * provided. See run_processes_parallel() below.\n *\n * pp_cb is the callback cookie as passed into run_processes_parallel,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n@@ -464,11 +468,16 @@ typedef int (*task_finished_fn)(int result,\n *\n * tr2_category & tr2_label: sets the trace2 category and label for\n * logging. These must either be unset, or both of them must be set.\n+ *\n+ * ungroup: Ungroup output. Output is printed as soon as possible and\n+ * bypasses run-command's internal processing. This may cause output\n+ * from different commands to be mixed.\n */\n struct run_process_parallel_opts\n {\n \tconst char *tr2_category;\n \tconst char *tr2_label;\n+\tunsigned int ungroup:1;\n };\n \n /**\n@@ -478,12 +487,19 @@ struct run_process_parallel_opts\n *\n * The children started via this function run in parallel. Their output\n * (both stdout and stderr) is routed to stderr in a manner that output\n- * from different tasks does not interleave.\n+ * from different tasks does not interleave (but see \"ungroup\" above).\n *\n * start_failure_fn and task_finished_fn can be NULL to omit any\n * special handling.\n *\n- * Options are passed via a \"struct run_process_parallel_opts\".\n+ * Options are passed via a \"struct run_process_parallel_opts\". If the\n+ * \"ungroup\" option isn't specified the callbacks will get a pointer\n+ * to a \"struct strbuf *out\", and must not write to stdout or stderr\n+ * as such output will mess up the output of the other parallel\n+ * processes. If \"ungroup\" option is specified callbacks will get a\n+ * NULL \"struct strbuf *out\" parameter, and are responsible for\n+ * emitting their own output, including dealing with any race\n+ * conditions due to writing in parallel to stdout and stderr.\n */\n int run_processes_parallel(int n, get_next_task_fn, start_failure_fn,\n \t\t\t task_finished_fn, void *pp_cb,\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 9b21f2f9f83..747e57ef536 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -31,7 +31,11 @@ static int parallel_next(struct child_process *cp,\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->args.v);\n-\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n+\telse\n+\t\tfprintf(stderr, \"preloaded output of a child\\n\");\n+\n \tnumber_callbacks++;\n \treturn 1;\n }\n@@ -41,7 +45,10 @@ static int no_job(struct child_process *cp,\n \t\t void *cb,\n \t\t void **task_cb)\n {\n-\tstrbuf_addstr(err, \"no further jobs available\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"no further jobs available\\n\");\n+\telse\n+\t\tfprintf(stderr, \"no further jobs available\\n\");\n \treturn 0;\n }\n \n@@ -50,7 +57,10 @@ static int task_finished(int result,\n \t\t\t void *pp_cb,\n \t\t\t void *pp_task_cb)\n {\n-\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n+\telse\n+\t\tfprintf(stderr, \"asking for a quick stop\\n\");\n \treturn 1;\n }\n \n@@ -412,17 +422,26 @@ int cmd__run_command(int argc, const char **argv)\n \tstrvec_clear(&proc.args);\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n-\tif (!strcmp(argv[1], \"run-command-parallel\"))\n+\tif (!strcmp(argv[1], \"run-command-parallel\") ||\n+\t !strcmp(argv[1], \"run-command-parallel-ungroup\")) {\n+\t\topts.ungroup = !strcmp(argv[1], \"run-command-parallel-ungroup\");\n \t\texit(run_processes_parallel(jobs, parallel_next,\n \t\t\t\t\t NULL, NULL, &proc, &opts));\n+\t}\n \n-\tif (!strcmp(argv[1], \"run-command-abort\"))\n+\tif (!strcmp(argv[1], \"run-command-abort\") ||\n+\t !strcmp(argv[1], \"run-command-abort-ungroup\")) {\n+\t\topts.ungroup = !strcmp(argv[1], \"run-command-abort-ungroup\");\n \t\texit(run_processes_parallel(jobs, parallel_next, NULL,\n \t\t\t\t\t task_finished, &proc, &opts));\n+\t}\n \n-\tif (!strcmp(argv[1], \"run-command-no-jobs\"))\n+\tif (!strcmp(argv[1], \"run-command-no-jobs\") ||\n+\t !strcmp(argv[1], \"run-command-no-jobs-ungroup\")) {\n+\t\topts.ungroup = !strcmp(argv[1], \"run-command-no-jobs-ungroup\");\n \t\texit(run_processes_parallel(jobs, no_job, NULL, task_finished,\n \t\t\t\t\t &proc, &opts));\n+\t}\n \n \tfprintf(stderr, \"check usage\\n\");\n \treturn 1;\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 131fcfda90f..0a82db965e8 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -135,18 +135,36 @@ test_expect_success 'run_command runs in parallel with more jobs available than\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n+\ttest-tool run-command run-command-parallel-ungroup 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n \ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n \ttest_must_be_empty out &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n+\ttest-tool run-command run-command-parallel-ungroup 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n \ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n \ttest_must_be_empty out &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n+\ttest-tool run-command run-command-parallel-ungroup 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n@@ -162,6 +180,12 @@ test_expect_success 'run_command is asked to abort gracefully' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n+\ttest-tool run-command run-command-abort-ungroup 3 false >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_line_count = 6 err\n+'\n+\n cat >expect <<-EOF\n no further jobs available\n EOF\n@@ -172,6 +196,12 @@ test_expect_success 'run_command outputs ' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command outputs (ungroup) ' '\n+\ttest-tool run-command run-command-no-jobs-ungroup 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect actual\n+'\n+\n test_trace () {\n \texpect=\"$1\"\n \tshift\n-- \n2.36.0.893.g80a51c675f6\n\n"},{"id":"454063","messageId":"patch-4.6-cf62569b2e0-20220421T122108Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-0.6-00000000000-20220421T122108Z-avarab@gmail.com","subject":"[PATCH 4/6] hook tests: fix redirection logic error in 96e7225b310","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-04-21T12:25:29Z","receivedAt":"2022-04-21T12:26:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The tests added in 96e7225b310 (hook: add 'run' subcommand,\n2021-12-22) were redirecting to \"actual\" both in the body of the hook\nitself and in the testing code below.\n\nThe net result was that the \"2>>actual\" redirection later in the test\nwasn't doing anything. Let's have those redirection do what it looks\nlike they're doing.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1800-hook.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\nindex 26ed5e11bc8..1e4adc3d53e 100755\n--- a/t/t1800-hook.sh\n+++ b/t/t1800-hook.sh\n@@ -94,7 +94,7 @@ test_expect_success 'git hook run -- out-of-repo runs excluded' '\n test_expect_success 'git -c core.hooksPath= hook run' '\n \tmkdir my-hooks &&\n \twrite_script my-hooks/test-hook <<-\\EOF &&\n-\techo Hook ran $1 >>actual\n+\techo Hook ran $1\n \tEOF\n \n \tcat >expect <<-\\EOF &&\n-- \n2.36.0.893.g80a51c675f6\n\n"},{"id":"454064","messageId":"patch-5.6-98c26c9917b-20220421T122108Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-0.6-00000000000-20220421T122108Z-avarab@gmail.com","subject":"[PATCH 5/6] hook API: don't redundantly re-set \"no_stdin\" and \"stdout_to_stderr\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-04-21T12:25:30Z","receivedAt":"2022-04-21T12:26:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Amend code added in 96e7225b310 (hook: add 'run' subcommand,\n2021-12-22) top stop setting these two flags. We use the\nrun_process_parallel() API added in c553c72eed6 (run-command: add an\nasynchronous parallel child processor, 2015-12-15), which always sets\nthese in pp_start_one() (in addition to setting .err = -1).\n\nNote that an assert() to check that these values are already what\nwe're setting them to here would fail. That's because in\npp_start_one() we'll set these after calling this \"get_next_task\"\ncallback (which we call pick_next_hook()). But the only case where we\nweren't setting these just after returning from this function was if\nwe took the \"return 0\" path here, in which case we wouldn't have set\nthese.\n\nSo while this code wasn't wrong, it was entirely redundant. The\nrun_process_parallel() also can't work with a generic \"struct\nchild_process\", it needs one that's behaving in a way that it expects\nwhen it comes to stderr/stdout. So we shouldn't be changing these\nvalues, or in this case keeping around code that gives the impression\nthat doing in the general case is OK.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/hook.c b/hook.c\nindex eadb2d58a7b..68ee4030551 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -53,9 +53,7 @@ static int pick_next_hook(struct child_process *cp,\n \tif (!hook_path)\n \t\treturn 0;\n \n-\tcp->no_stdin = 1;\n \tstrvec_pushv(&cp->env_array, hook_cb->options->env.v);\n-\tcp->stdout_to_stderr = 1;\n \tcp->trace2_hook_name = hook_cb->hook_name;\n \tcp->dir = hook_cb->options->dir;\n \n-- \n2.36.0.893.g80a51c675f6\n\n"},{"id":"454090","messageId":"xmqqsfq6jqnt.fsf@gitster.g","threadId":"57764","inReplyTo":"220421.86sfq67hlr.gmgdl@evledraar.gmail.com","subject":"Re: git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-21T17:24:22Z","receivedAt":"2022-04-21T17:24:31Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> ... I.e. as a mechanism for\n> mitigating mistakes and catching obscure edge cases just being more\n> careful or having things sit in 'next' for longer has, I think, proved\n> itself to not be an effective method (not just in this case, but a few\n> similar cases).\n\nI tend to agree. Given that people do not discover possible\nregression that will affect even after a change hits 'master',\ncooking in 'next' alone would not be all that effective.\n\nBut that does not mean we shouldn't cook in 'next' at all.\n\nEspecially the previous cycle, I was experimenting with a tweak in\nmy workflow to have topics in 'next' to cook for one week and have\nthem graduate to 'master' unless we saw regression in a week.\nPreviously, I tended to keep topics on the larger side in 'next' and\nwe did see \"oops we found this after the topic hit 'next' and here\nis a fix-up\" to them, which I think helped to catch bugs before it\nbroke 'master'.\n\n> I'm not sure what the solution is exactly, but I'm pretty sure it\n> involves more controlled exposure to the wild (e.g. shipping certain\n> things as feature flags first), not deferring that exposure for long\n> periods, which is what having things sit it \"next\" for longer amounts\n> to.\n\nTo be fair, \"is the hook invoked with its standard output stream\nconnected to the original standard output of the main process?\" is\nnot something either of you cared while working on and reviewing the\nchanges, and releases based on 'next' $BIGCOMPANY have its users use\ninternally wouldn't have helped to catch this particular regression,\nas the primary reason we didn't think of it as a problem is because\ninternal users of $BIGCOMPANY tend to be more monoculture than folks\nin the wild.\n\nSo the fundamental solution would be to find a way to involve those\nwho found the regression after a release was done in the development\nprocess at a much earlier stage. I do not offhand know how to get\nthere.\n\nIt may be very hard for us to do with end-users, who typically have\nonly one instance of Git they use and a single valuable repository\nthey cannot subject to \"experiments\". They may depend on certain\naspects of the behaviour of the current version, but they lack an\nenvironment to try out our new version to see if we broke them.\nThey probably may not even be aware of what they are relying on,\njust as we are unaware of their dependence.\n\nBut for toolsmiths who integrate their gears with Git, there may be\nsomething we can do. Perhaps we can start giving \"works with Git\"\nbadge out (we need to control its use with some kind of trademark\nregistration), to those who \"maintain X that enhances Git\" when they\nregularly test their ware with 'next' or much earlier. And when\nthey stop us before unleashing a possible regression by reporting\nbugs early, give them a \"star\", so that their \"works with Git *****\"\nlogo can boast how much contribution in testing they are maing\nupstream.\n\nOr something like that, perhaps?\n"},{"id":"454094","messageId":"xmqqmtgejq4o.fsf@gitster.g","threadId":"57764","inReplyTo":"cover-0.6-00000000000-20220421T122108Z-avarab@gmail.com","subject":"Re: [PATCH 0/6] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-21T17:35:51Z","receivedAt":"2022-04-21T17:36:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> This fixes the regression reported by Anthony Sottile[1] with hooks\n> not being connected to a TTY. See 6/6 for details.\n\nIt is surprising that it takes 6 patches that rewrites ~100 lines\nand adds ~200 new lines, which would need to be treated as a new\ndevelopment with its own risk of regressions (hence going through\nthe normal review cycle, starting out of 'next', and gradually\ngetting merged down), instead of being able to be fast-tracked.\n\nLet's see who comments on the patches first and perhaps people may\nshoot to gain \"works with Git\" star by testing it ;-)\n\nThanks.\n"},{"id":"454144","messageId":"xmqq8rryi8l2.fsf@gitster.g","threadId":"57764","inReplyTo":"xmqqsfq6jqnt.fsf@gitster.g","subject":"Re: git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-21T18:40:09Z","receivedAt":"2022-04-21T18:40:19Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Especially the previous cycle, I was experimenting with a tweak in\n> my workflow to have topics in 'next' to cook for one week and have\n> them graduate to 'master' unless we saw regression in a week.\n\n\"and mechanically have them graduate\" is what I meant. Also\n\n> Previously, I tended to keep topics on the larger side in 'next' and\n\n\"in 'next' longer, and\" is what I meant.\n\n> we did see \"oops we found this after the topic hit 'next' and here\n> is a fix-up\" to them, which I think helped to catch bugs before it\n> broke 'master'.\n\n"},{"id":"454148","messageId":"220421.86bkwu6zd2.gmgdl@evledraar.gmail.com","threadId":"57764","inReplyTo":"xmqqmtgejq4o.fsf@gitster.g","subject":"Re: [PATCH 0/6] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-04-21T18:50:25Z","receivedAt":"2022-04-21T18:55:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Apr 21 2022, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> This fixes the regression reported by Anthony Sottile[1] with hooks\n>> not being connected to a TTY. See 6/6 for details.\n>\n> It is surprising that it takes 6 patches that rewrites ~100 lines\n> and adds ~200 new lines, which would need to be treated as a new\n> development with its own risk of regressions (hence going through\n> the normal review cycle, starting out of 'next', and gradually\n> getting merged down), instead of being able to be fast-tracked.\n\nYes, it's unfortunate. Perhaps there's some way I'm missing in coming up\nwith a smaller isolated fix, but I wasn't able to come up with one.\n\nIt's not just the pre-commit hook, that's just where the problem was\ndiscovered, but the dozen or so other run_hook*() users (not all of whom\nmay be practically impacted).\n\nRewriting the hook.c API to use another \"runner\" would probably be the\nsmallest change, but even if that's going to be smaller by line-count\nI'd think it's much bigger in terms of review time. Most of this series\nis boilerplate changes, or changes where it's easy to review that no\ncaller of run-command.[ch] that isn't hook.c will be impacted by them...\n"},{"id":"454319","messageId":"xmqqilr04fq1.fsf@gitster.g","threadId":"57764","inReplyTo":"patch-3.6-d76f63c2948-20220421T122108Z-avarab@gmail.com","subject":"Re: [PATCH 3/6] run-command: add an \"ungroup\" option to run_process_parallel()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-23T03:54:14Z","receivedAt":"2022-04-23T03:54:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> @@ -1494,6 +1494,7 @@ struct parallel_processes {\n> \tstruct pollfd *pfd;\n> \n> \tunsigned shutdown : 1;\n> +\tunsigned ungroup:1;\n\nMatch the style with the above (either with or without SP\nconsistently, I would choose to match existing one if I were doing\nthis myself).\n\n> @@ -1537,8 +1538,9 @@ static void pp_init(struct parallel_processes *pp,\n> \t\t get_next_task_fn get_next_task,\n> \t\t start_failure_fn start_failure,\n> \t\t task_finished_fn task_finished,\n> -\t\t void *data)\n> +\t\t void *data, struct run_process_parallel_opts *opts)\n> {\n> +\tconst int ungroup = opts->ungroup;\n> \tint i;\n> \n> \tif (n < 1)\n> @@ -1556,16 +1558,22 @@ static void pp_init(struct parallel_processes *pp,\n> \tpp->start_failure = start_failure ? start_failure : default_start_failure;\n> \tpp->task_finished = task_finished ? task_finished : default_task_finished;\n> \n> +\tpp->ungroup = ungroup;\n> +\n\nOK, now this makes it clear that the new structure introduced in the\nfirst step is about run_process_parallel() and not about trace2, so\nit would probably make sense to go back to that step and throw these\n*_fn callbacks and callback state to the structure, too.\n\n> \tpp->nr_processes = 0;\n> \tpp->output_owner = 0;\n> \tpp->shutdown = 0;\n> \tCALLOC_ARRAY(pp->children, n);\n> -\tCALLOC_ARRAY(pp->pfd, n);\n> +\tif (!ungroup)\n> +\t\tCALLOC_ARRAY(pp->pfd, n);\n\nOK, we will not poll under ungroup option, so we do not need pfd[]\nin that case. It would be cleaner to clear pp->pfd = NULL if not\ndone already when ungroup is in effect.\n\n> +\n> \tstrbuf_init(&pp->buffered_output, 0);\n> \n> \tfor (i = 0; i < n; i++) {\n> \t\tstrbuf_init(&pp->children[i].err, 0);\n> \t\tchild_process_init(&pp->children[i].process);\n> +\t\tif (ungroup)\n> +\t\t\tcontinue;\n> \t\tpp->pfd[i].events = POLLIN | POLLHUP;\n> \t\tpp->pfd[i].fd = -1;\n> \t}\n\nThis does not make practical difference _right_ _now_, but as a\ngeneral code hygiene discipline, it would be more future-proof not\nto rely on \"ungroup\" being the _only_ thing that allows us to omit\nallocating pfd. IOW, conditional allocation of pp->pfd based on\nungroup before the loop is perfectly fine, but inside the loop, it\nwould be better to say \"if pp->pfd is not there, no matter the\nreason why pp->pfd is missing, we refrain from filling the array\nbecause we are not polling\".\n\n\t\tif (!pp->pfd)\n\t\t\tcontinue;\n\nWe may know that the only reason we decided not to poll is with the\nungroup bit in the current code, but we do not have to depend on the\nknowledge. The only thing we need to know, in order to refrain from\nsetting POLLIN/POLLHUP bits, is that we decided that we will not poll,\nand the decision should be more directly found in pp->pfd than inferring\nwhat the ungroup says.\n\n> @@ -1576,6 +1584,7 @@ static void pp_init(struct parallel_processes *pp,\n> \n> static void pp_cleanup(struct parallel_processes *pp)\n> {\n> +\tconst int ungroup = pp->ungroup;\n> \tint i;\n> \n> \ttrace_printf(\"run_processes_parallel: done\");\n> @@ -1585,14 +1594,17 @@ static void pp_cleanup(struct parallel_processes *pp)\n> \t}\n> \n> \tfree(pp->children);\n> -\tfree(pp->pfd);\n> +\tif (!ungroup)\n> +\t\tfree(pp->pfd);\n\nLikewise, the NULLness of pp->pfd should be what matters.\n\n> \t/*\n> \t * When get_next_task added messages to the buffer in its last\n> \t * iteration, the buffered output is non empty.\n> \t */\n> -\tstrbuf_write(&pp->buffered_output, stderr);\n> -\tstrbuf_release(&pp->buffered_output);\n> +\tif (!ungroup) {\n> +\t\tstrbuf_write(&pp->buffered_output, stderr);\n> +\t\tstrbuf_release(&pp->buffered_output);\n> +\t}\n\nOK, this need to happen only when we are buffering.\n\n> \tsigchain_pop_common();\n> }\n> @@ -1606,6 +1618,7 @@ static void pp_cleanup(struct parallel_processes *pp)\n> */\n> static int pp_start_one(struct parallel_processes *pp)\n> {\n> +\tconst int ungroup = pp->ungroup;\n> \tint i, code;\n> \n> \tfor (i = 0; i < pp->max_processes; i++)\n> @@ -1615,24 +1628,31 @@ static int pp_start_one(struct parallel_processes *pp)\n> \t\tBUG(\"bookkeeping is hard\");\n> \n> \tcode = pp->get_next_task(&pp->children[i].process,\n> -\t\t\t\t &pp->children[i].err,\n> +\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n\nOK.\n\n> \t\t\t\t pp->data,\n> \t\t\t\t &pp->children[i].data);\n> \tif (!code) {\n> -\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> -\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\tif (!ungroup) {\n> +\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> +\t\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\t}\n> \t\treturn 1;\n> \t}\n> -\tpp->children[i].process.err = -1;\n> -\tpp->children[i].process.stdout_to_stderr = 1;\n> -\tpp->children[i].process.no_stdin = 1;\n> +\n> +\tif (!ungroup) {\n> +\t\tpp->children[i].process.err = -1;\n> +\t\tpp->children[i].process.stdout_to_stderr = 1;\n> +\t\tpp->children[i].process.no_stdin = 1;\n> +\t}\n\nOK, except for .no_stdin bit. Even before the \"we started using the\nparallel running API to drive hooks, losing the direct access to the\nreal standard output from the hooks\" regression, we didn't expose the\ninput side to the hooks. run_hook_ve() did set .no_stdin and I do\nnot think we want to change that with \"--ungroup\". If there is any\nreason why you needed not to keep .no_stdin bit set in the ungroup\nmode, deviating from what the code before the regression did, that\nneeds to be explained (but I suspect this was simply a bug in this\nround of the patch, not a deliberate behaviour change).\n\n> \tif (start_command(&pp->children[i].process)) {\n> -\t\tcode = pp->start_failure(&pp->children[i].err,\n> +\t\tcode = pp->start_failure(ungroup ? NULL : &pp->children[i].err,\n> \t\t\t\t\t pp->data,\n> \t\t\t\t\t pp->children[i].data);\n> -\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> -\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\tif (!ungroup) {\n> +\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> +\t\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\t}\n> \t\tif (code)\n> \t\t\tpp->shutdown = 1;\n> \t\treturn code;\n> @@ -1640,14 +1660,26 @@ static int pp_start_one(struct parallel_processes *pp)\n> \n> \tpp->nr_processes++;\n> \tpp->children[i].state = GIT_CP_WORKING;\n> -\tpp->pfd[i].fd = pp->children[i].process.err;\n> +\tif (!ungroup)\n> +\t\tpp->pfd[i].fd = pp->children[i].process.err;\n> \treturn 0;\n> }\n> \n> +static void pp_mark_working_for_cleanup(struct parallel_processes *pp)\n> +{\n> +\tint i;\n> +\n> +\tfor (i = 0; i < pp->max_processes; i++)\n> +\t\tif (pp->children[i].state == GIT_CP_WORKING)\n> +\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n> +}\n> +\n> static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> {\n> \tint i;\n> \n> +\tassert(!pp->ungroup);\n> +\n> \twhile ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n> \t\tif (errno == EINTR)\n> \t\t\tcontinue;\n> @@ -1674,6 +1706,9 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> static void pp_output(struct parallel_processes *pp)\n> {\n> \tint i = pp->output_owner;\n> +\n> +\tassert(!pp->ungroup);\n> +\n> \tif (pp->children[i].state == GIT_CP_WORKING &&\n> \t pp->children[i].err.len) {\n> \t\tstrbuf_write(&pp->children[i].err, stderr);\n> @@ -1683,10 +1718,15 @@ static void pp_output(struct parallel_processes *pp)\n> \n> static int pp_collect_finished(struct parallel_processes *pp)\n> {\n> +\tconst int ungroup = pp->ungroup;\n> \tint i, code;\n> \tint n = pp->max_processes;\n> \tint result = 0;\n> \n> +\tif (ungroup)\n> +\t\tfor (i = 0; i < pp->max_processes; i++)\n> +\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n> +\n> \twhile (pp->nr_processes > 0) {\n> \t\tfor (i = 0; i < pp->max_processes; i++)\n> \t\t\tif (pp->children[i].state == GIT_CP_WAIT_CLEANUP)\n> @@ -1697,8 +1737,8 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> \t\tcode = finish_command(&pp->children[i].process);\n> \n> \t\tcode = pp->task_finished(code,\n> -\t\t\t\t\t &pp->children[i].err, pp->data,\n> -\t\t\t\t\t pp->children[i].data);\n> +\t\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n> +\t\t\t\t\t pp->data, pp->children[i].data);\n> \n> \t\tif (code)\n> \t\t\tresult = code;\n> @@ -1707,10 +1747,13 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> \n> \t\tpp->nr_processes--;\n> \t\tpp->children[i].state = GIT_CP_FREE;\n> -\t\tpp->pfd[i].fd = -1;\n> +\t\tif (!ungroup)\n> +\t\t\tpp->pfd[i].fd = -1;\n> \t\tchild_process_init(&pp->children[i].process);\n> \n> -\t\tif (i != pp->output_owner) {\n> +\t\tif (ungroup) {\n> +\t\t\t/* no strbuf_*() work to do here */\n\nMake it a habit to keep \";\" semicolon when writing an empty\nstatement, i.e.\n\n\t\t\t; /* some comments */\n\nThis will help when other else/if body becomes shorter and we can\nlose the {} around here.\n\nI cannot quite shake the feeling that this step is doing so much\nonly because it wants to coax \"run in parallel\" infrastructure to do\nwhat it is not suited to do, i.e. drive hooks that wants to directly\nface the end-user output channel, and it might make a conceptually\ncleaner fix to simply revert the root cause, but it is getting late\nso...\n\n\n\n"},{"id":"454320","messageId":"xmqqczh84fpy.fsf@gitster.g","threadId":"57764","inReplyTo":"patch-4.6-cf62569b2e0-20220421T122108Z-avarab@gmail.com","subject":"Re: [PATCH 4/6] hook tests: fix redirection logic error in 96e7225b310","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-23T03:54:17Z","receivedAt":"2022-04-23T03:54:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> The tests added in 96e7225b310 (hook: add 'run' subcommand,\n> 2021-12-22) were redirecting to \"actual\" both in the body of the hook\n> itself and in the testing code below.\n>\n> The net result was that the \"2>>actual\" redirection later in the test\n> wasn't doing anything. Let's have those redirection do what it looks\n> like they're doing.\n\nAnd the error didn't affect the outcome of the tests?\nThis is fun.\n\nNicely spotted.\n\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> t/t1800-hook.sh | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\n> index 26ed5e11bc8..1e4adc3d53e 100755\n> --- a/t/t1800-hook.sh\n> +++ b/t/t1800-hook.sh\n> @@ -94,7 +94,7 @@ test_expect_success 'git hook run -- out-of-repo runs excluded' '\n> test_expect_success 'git -c core.hooksPath= hook run' '\n> \tmkdir my-hooks &&\n> \twrite_script my-hooks/test-hook <<-\\EOF &&\n> -\techo Hook ran $1 >>actual\n> +\techo Hook ran $1\n> \tEOF\n> \n> \tcat >expect <<-\\EOF &&\n"},{"id":"454321","messageId":"xmqq1qxo4eby.fsf@gitster.g","threadId":"57764","inReplyTo":"patch-2.6-d9c9b158130-20220421T122108Z-avarab@gmail.com","subject":"Re: [PATCH 2/6] run-command tests: test stdout of run_command_parallel()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-23T04:24:17Z","receivedAt":"2022-04-23T04:26:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> Extend the tests added in c553c72eed6 (run-command: add an\n> asynchronous parallel child processor, 2015-12-15) to test stdout in\n> addition to stderr. A subsequent commit will add additional related\n> tests for a new feature, making it obvious how the output of the two\n> compares on both stdout and stderr will make this easier to reason\n> about.\n\nOK.\n\nThe original cared only about the standard error stream, and it was\nsensible to name the actual output there \"actual\" and the correct\noutput \"expect\" to be compared.\n\nBut now we care _both_ the standard output and the standard error\nstreams, so if we call one \"out\", wouldn't we want to call the other\none \"err\", I wonder? If it makes sense, it still is OK if we are\nnot doing so inside this step or in the series, but then it would\nmake sense to leave ourselves a NEEDSWORK: note to do so later when\nthe tree is quiescent.\n\nOther than that, this one is pretty much boringly OK, and boring is\ngood.\n\n\n\n\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> t/t0061-run-command.sh | 15 ++++++++++-----\n> 1 file changed, 10 insertions(+), 5 deletions(-)\n>\n> diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\n> index ee281909bc3..131fcfda90f 100755\n> --- a/t/t0061-run-command.sh\n> +++ b/t/t0061-run-command.sh\n> @@ -130,17 +130,20 @@ World\n> EOF\n> \n> test_expect_success 'run_command runs in parallel with more jobs available than tasks' '\n> -\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n> +\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n> +\ttest_must_be_empty out &&\n> \ttest_cmp expect actual\n> '\n> \n> test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n> -\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n> +\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n> +\ttest_must_be_empty out &&\n> \ttest_cmp expect actual\n> '\n> \n> test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n> -\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n> +\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n> +\ttest_must_be_empty out &&\n> \ttest_cmp expect actual\n> '\n> \n> @@ -154,7 +157,8 @@ asking for a quick stop\n> EOF\n> \n> test_expect_success 'run_command is asked to abort gracefully' '\n> -\ttest-tool run-command run-command-abort 3 false 2>actual &&\n> +\ttest-tool run-command run-command-abort 3 false >out 2>actual &&\n> +\ttest_must_be_empty out &&\n> \ttest_cmp expect actual\n> '\n> \n> @@ -163,7 +167,8 @@ no further jobs available\n> EOF\n> \n> test_expect_success 'run_command outputs ' '\n> -\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n> +\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n> +\ttest_must_be_empty out &&\n> \ttest_cmp expect actual\n> '\n"},{"id":"454322","messageId":"xmqq7d7g4ec1.fsf@gitster.g","threadId":"57764","inReplyTo":"patch-1.6-8bf71ce63dd-20220421T122108Z-avarab@gmail.com","subject":"Re: [PATCH 1/6] run-command API: replace run_processes_parallel_tr2() with opts struct","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-23T04:24:14Z","receivedAt":"2022-04-23T04:26:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> \tif (max_children != 1 && list->nr != 1) {\n> \t\tstruct parallel_fetch_state state = { argv.v, list, 0, 0 };\n> +\t\tstruct run_process_parallel_opts run_opts = {\n> +\t\t\t.tr2_category = \"fetch\",\n> +\t\t\t.tr2_label = \"parallel/fetch\",\n> +\t\t};\n> \n> \t\tstrvec_push(&argv, \"--end-of-options\");\n> -\t\tresult = run_processes_parallel_tr2(max_children,\n> -\t\t\t\t\t\t &fetch_next_remote,\n> -\t\t\t\t\t\t &fetch_failed_to_start,\n> -\t\t\t\t\t\t &fetch_finished,\n> -\t\t\t\t\t\t &state,\n> -\t\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n> +\t\tresult = run_processes_parallel(max_children,\n> +\t\t\t\t\t\t&fetch_next_remote,\n> +\t\t\t\t\t\t&fetch_failed_to_start,\n> +\t\t\t\t\t\t&fetch_finished, &state,\n> +\t\t\t\t\t\t&run_opts);\n\nIf the idea is that with unset .tr2_* members we can silently bypass\nthe overhead to invoke trace2 machinery without changing much in the\ncaller side (or even better, instead of doing this as run_opts but\nas tr2_opts, and allow the caller to pass NULL to decline tracing at\nruntime), that would be wonderful.\n\nIf we are going to throw random other members into the struct that\nare unrelated to tr2, then it makes it unclear why we have the three\n*_fn and its callback state still passed as separate parameters,\nrather than making them members of the struct. After all, it is\nclear that this new struct is designed to be used only with the\nrun_process_parallel() API, so it is doubly dubious why these three\n*_fn and callback state are not members.\n\nSo, I dunno. \n\nEither\n\n (1) making it to very clear that this is only about trace2 and name\n the type as such, or\n\n (2) making it about run_process_parallel (and keep the name), and\n move the *_fn parameters to it (which will allow us to add more\noptional callbacks if needed),\n\nwould make it better (simply by clarifying why we have this extra\nstructure and what it is meant to be used for), but the interface as\nposted is halfway between the two, and does not look well suited for\neither purpose, making the reader feel somewhat frustrating.\n\n\n"},{"id":"454605","messageId":"YmsgWj5vPEWNyGFA@google.com","threadId":"57764","inReplyTo":"patch-1.6-8bf71ce63dd-20220421T122108Z-avarab@gmail.com","subject":"Re: [PATCH 1/6] run-command API: replace run_processes_parallel_tr2() with opts struct","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2022-04-28T23:16:42Z","receivedAt":"2022-04-28T23:16:52Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Apr 21, 2022 at 02:25:26PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> Add a new \"struct run_process_parallel_opts\" to cover the trace2\n> use-case added in ee4512ed481 (trace2: create new combined trace\n> facility, 2019-02-22). A subsequent commit will add more options, and\n> having a proliferation of new functions or extra parameters would\n> result in needless churn.\n> \n> It makes for a smaller change to make run_processes_parallel() and\n> run_processes_parallel_tr2() wrapper functions for the new \"static\"\n> run_processes_parallel_1(), which contains the main logic. We pass\n> down \"opts\" to the *_1() function even though it isn't used there\n> yet (only in the *_tr2() function), a subsequent commit will make more\n> use of it.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> builtin/fetch.c | 15 ++++++++------\n> builtin/submodule--helper.c | 12 +++++++----\n> hook.c | 13 ++++++------\n> run-command.c | 40 +++++++++++++++++++++++++++----------\n> run-command.h | 26 ++++++++++++++++--------\n> submodule.c | 13 ++++++------\n> t/helper/test-run-command.c | 13 ++++++------\n> 7 files changed, 84 insertions(+), 48 deletions(-)\n> \n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index e3791f09ed5..9bc99183191 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -1948,14 +1948,17 @@ static int fetch_multiple(struct string_list *list, int max_children)\n> \n> \tif (max_children != 1 && list->nr != 1) {\n> \t\tstruct parallel_fetch_state state = { argv.v, list, 0, 0 };\n> +\t\tstruct run_process_parallel_opts run_opts = {\n> +\t\t\t.tr2_category = \"fetch\",\n> +\t\t\t.tr2_label = \"parallel/fetch\",\n> +\t\t};\n> \n> \t\tstrvec_push(&argv, \"--end-of-options\");\n> -\t\tresult = run_processes_parallel_tr2(max_children,\n> -\t\t\t\t\t\t &fetch_next_remote,\n> -\t\t\t\t\t\t &fetch_failed_to_start,\n> -\t\t\t\t\t\t &fetch_finished,\n> -\t\t\t\t\t\t &state,\n> -\t\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n> +\t\tresult = run_processes_parallel(max_children,\n> +\t\t\t\t\t\t&fetch_next_remote,\n> +\t\t\t\t\t\t&fetch_failed_to_start,\n> +\t\t\t\t\t\t&fetch_finished, &state,\n> +\t\t\t\t\t\t&run_opts);\n> \n> \t\tif (!result)\n> \t\t\tresult = state.result;\n> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\n> index 2c87ef9364f..c3d1aace546 100644\n> --- a/builtin/submodule--helper.c\n> +++ b/builtin/submodule--helper.c\n> @@ -2652,12 +2652,16 @@ static int update_submodules(struct update_data *update_data)\n> {\n> \tint i, res = 0;\n> \tstruct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;\n> +\tstruct run_process_parallel_opts run_opts = {\n> +\t\t.tr2_category = \"submodule\",\n> +\t\t.tr2_label = \"parallel/update\",\n> +\t};\n\nHm, so now it is possible for a callsite to forget to set these, rather\nthan to grep for \"run_processes_parallel\" and notice that everybody else\nis already calling \"run_processes_parallel_tr2\". There are not any\ncallers of 'run_processes_parallel()' except for a test helper today, so\nwhy do we make this seem optional?\n\nIf I'm being honest, I'd rather see everything _but_ the trace2 stuff go\ninto an opts struct, and then see the same entry points we have today\n(run_processes_parallel that takes a struct, run_processes_parallel_tr2\nthat takes a struct and two tr2 string args). Or, I guess, a single\nrun_processes_parallel() that only takes a struct, does the right thing\nwith the trace args, and entirely removes the\nrun_processes_parallel_tr2 call.\n\n> \n> \tsuc.update_data = update_data;\n> -\trun_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,\n> -\t\t\t\t update_clone_start_failure,\n> -\t\t\t\t update_clone_task_finished, &suc, \"submodule\",\n> -\t\t\t\t \"parallel/update\");\n> +\trun_processes_parallel(suc.update_data->max_jobs,\n> +\t\t\t update_clone_get_next_task,\n> +\t\t\t update_clone_start_failure,\n> +\t\t\t update_clone_task_finished, &suc, &run_opts);\n> \n> \t/*\n> \t * We saved the output and put it out all at once now.\n> diff --git a/hook.c b/hook.c\n> index 1d51be3b77a..eadb2d58a7b 100644\n> --- a/hook.c\n> +++ b/hook.c\n> @@ -123,6 +123,10 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n> \tconst char *const hook_path = find_hook(hook_name);\n> \tint jobs = 1;\n> \tint ret = 0;\n> +\tstruct run_process_parallel_opts run_opts = {\n> +\t\t.tr2_category = \"hook\",\n> +\t\t.tr2_label = hook_name,\n> +\t};\n> \n> \tif (!options)\n> \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n> @@ -144,13 +148,8 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n> \t\tcb_data.hook_path = abs_path.buf;\n> \t}\n> \n> -\trun_processes_parallel_tr2(jobs,\n> -\t\t\t\t pick_next_hook,\n> -\t\t\t\t notify_start_failure,\n> -\t\t\t\t notify_hook_finished,\n> -\t\t\t\t &cb_data,\n> -\t\t\t\t \"hook\",\n> -\t\t\t\t hook_name);\n> +\trun_processes_parallel(jobs, pick_next_hook, notify_start_failure,\n> +\t\t\t notify_hook_finished, &cb_data, &run_opts);\n> \tret = cb_data.rc;\n> cleanup:\n> \tstrbuf_release(&abs_path);\n> diff --git a/run-command.c b/run-command.c\n> index a8501e38ceb..7b8159aa235 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1738,11 +1738,11 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> \treturn result;\n> }\n> \n> -int run_processes_parallel(int n,\n> -\t\t\t get_next_task_fn get_next_task,\n> -\t\t\t start_failure_fn start_failure,\n> -\t\t\t task_finished_fn task_finished,\n> -\t\t\t void *pp_cb)\n> +static int run_processes_parallel_1(int n, get_next_task_fn get_next_task,\n> +\t\t\t\t start_failure_fn start_failure,\n> +\t\t\t\t task_finished_fn task_finished,\n> +\t\t\t\t void *pp_cb,\n> +\t\t\t\t struct run_process_parallel_opts *opts)\n> {\n> \tint i, code;\n> \tint output_timeout = 100;\n> @@ -1780,24 +1780,42 @@ int run_processes_parallel(int n,\n> \treturn 0;\n> }\n> \n> -int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n> -\t\t\t start_failure_fn start_failure,\n> -\t\t\t task_finished_fn task_finished, void *pp_cb,\n> -\t\t\t const char *tr2_category, const char *tr2_label)\n> +static int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n> +\t\t\t\t start_failure_fn start_failure,\n> +\t\t\t\t task_finished_fn task_finished,\n> +\t\t\t\t void *pp_cb,\n> +\t\t\t\t struct run_process_parallel_opts *opts)\n> {\n> +\tconst char *tr2_category = opts->tr2_category;\n> +\tconst char *tr2_label = opts->tr2_label;\n> \tint result;\n> \n> \ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n> \t\t\t\t ((n < 1) ? online_cpus() : n));\n> \n> -\tresult = run_processes_parallel(n, get_next_task, start_failure,\n> -\t\t\t\t\ttask_finished, pp_cb);\n> +\tresult = run_processes_parallel_1(n, get_next_task, start_failure,\n> +\t\t\t\t\t task_finished, pp_cb, opts);\n> \n> \ttrace2_region_leave(tr2_category, tr2_label, NULL);\n> \n> \treturn result;\n> }\n> \n> +int run_processes_parallel(int n, get_next_task_fn get_next_task,\n> +\t\t\t start_failure_fn start_failure,\n> +\t\t\t task_finished_fn task_finished, void *pp_cb,\n> +\t\t\t struct run_process_parallel_opts *opts)\n> +{\n> +\tif (opts->tr2_category && opts->tr2_label)\n> +\t\treturn run_processes_parallel_tr2(n, get_next_task,\n> +\t\t\t\t\t\t start_failure, task_finished,\n> +\t\t\t\t\t\t pp_cb, opts);\nWhat is the point for this extra layer of indirection? I am confused why\nwe might not just change the arg list for run_processes_parallel_tr2 and\ncall it good.\n\nIf the final result was to reduce the number of run_processes_parallel.*\nfunctions available I'd be happy to see this change, but as it\nintroduces even more various entry points into run_processes_parallel.*\nI'm not so sure.\n> +\n> +\treturn run_processes_parallel_1(n, get_next_task, start_failure,\n> +\t\t\t\t\ttask_finished, pp_cb, opts);\n> +}\n> +\n> +\n> int run_auto_maintenance(int quiet)\n> {\n> \tint enabled;\n> diff --git a/run-command.h b/run-command.h\n> index 07bed6c31b4..66e7bebd88a 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -458,6 +458,19 @@ typedef int (*task_finished_fn)(int result,\n> \t\t\t\tvoid *pp_cb,\n> \t\t\t\tvoid *pp_task_cb);\n> \n> +/**\n> + * Options to pass to run_processes_parallel(), { 0 }-initialized\n> + * means no options. Fields:\n> + *\n> + * tr2_category & tr2_label: sets the trace2 category and label for\n> + * logging. These must either be unset, or both of them must be set.\n> + */\n> +struct run_process_parallel_opts\n> +{\n> +\tconst char *tr2_category;\n> +\tconst char *tr2_label;\n> +};\n> +\n> /**\n> * Runs up to n processes at the same time. Whenever a process can be\n> * started, the callback get_next_task_fn is called to obtain the data\n> @@ -469,15 +482,12 @@ typedef int (*task_finished_fn)(int result,\n> *\n> * start_failure_fn and task_finished_fn can be NULL to omit any\n> * special handling.\n> + *\n> + * Options are passed via a \"struct run_process_parallel_opts\".\n> */\n> -int run_processes_parallel(int n,\n> -\t\t\t get_next_task_fn,\n> -\t\t\t start_failure_fn,\n> -\t\t\t task_finished_fn,\n> -\t\t\t void *pp_cb);\n> -int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n> -\t\t\t task_finished_fn, void *pp_cb,\n> -\t\t\t const char *tr2_category, const char *tr2_label);\n> +int run_processes_parallel(int n, get_next_task_fn, start_failure_fn,\n> +\t\t\t task_finished_fn, void *pp_cb,\n> +\t\t\t struct run_process_parallel_opts *opts);\n> \n> /**\n> * Convenience function which prepares env_array for a command to be run in a\n> diff --git a/submodule.c b/submodule.c\n> index 86c8f0f89db..256c6bb4b8f 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -1817,6 +1817,10 @@ int fetch_submodules(struct repository *r,\n> {\n> \tint i;\n> \tstruct submodule_parallel_fetch spf = SPF_INIT;\n> +\tstruct run_process_parallel_opts run_opts = {\n> +\t\t.tr2_category = \"submodule\",\n> +\t\t.tr2_label = \"parallel/fetch\",\n> +\t};\n> \n> \tspf.r = r;\n> \tspf.command_line_option = command_line_option;\n> @@ -1838,12 +1842,9 @@ int fetch_submodules(struct repository *r,\n> \n> \tcalculate_changed_submodule_paths(r, &spf.changed_submodule_names);\n> \tstring_list_sort(&spf.changed_submodule_names);\n> -\trun_processes_parallel_tr2(max_parallel_jobs,\n> -\t\t\t\t get_next_submodule,\n> -\t\t\t\t fetch_start_failure,\n> -\t\t\t\t fetch_finish,\n> -\t\t\t\t &spf,\n> -\t\t\t\t \"submodule\", \"parallel/fetch\");\n> +\trun_processes_parallel(max_parallel_jobs, get_next_submodule,\n> +\t\t\t fetch_start_failure, fetch_finish, &spf,\n> +\t\t\t &run_opts);\n> \n> \tif (spf.submodules_with_errors.len > 0)\n> \t\tfprintf(stderr, _(\"Errors during submodule fetch:\\n%s\"),\n> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\n> index f3b90aa834a..9b21f2f9f83 100644\n> --- a/t/helper/test-run-command.c\n> +++ b/t/helper/test-run-command.c\n> @@ -183,7 +183,7 @@ static int testsuite(int argc, const char **argv)\n> \t\t(uintmax_t)suite.tests.nr, max_jobs);\n> \n> \tret = run_processes_parallel(max_jobs, next_test, test_failed,\n> -\t\t\t\t test_finished, &suite);\n> +\t\t\t\t test_finished, &suite, NULL);\n> \n> \tif (suite.failed.nr > 0) {\n> \t\tret = 1;\n> @@ -371,6 +371,7 @@ int cmd__run_command(int argc, const char **argv)\n> {\n> \tstruct child_process proc = CHILD_PROCESS_INIT;\n> \tint jobs;\n> +\tstruct run_process_parallel_opts opts = { 0 };\n> \n> \tif (argc > 1 && !strcmp(argv[1], \"testsuite\"))\n> \t\texit(testsuite(argc - 1, argv + 1));\n> @@ -413,15 +414,15 @@ int cmd__run_command(int argc, const char **argv)\n> \n> \tif (!strcmp(argv[1], \"run-command-parallel\"))\n> \t\texit(run_processes_parallel(jobs, parallel_next,\n> -\t\t\t\t\t NULL, NULL, &proc));\n> +\t\t\t\t\t NULL, NULL, &proc, &opts));\n> \n> \tif (!strcmp(argv[1], \"run-command-abort\"))\n> -\t\texit(run_processes_parallel(jobs, parallel_next,\n> -\t\t\t\t\t NULL, task_finished, &proc));\n> +\t\texit(run_processes_parallel(jobs, parallel_next, NULL,\n> +\t\t\t\t\t task_finished, &proc, &opts));\n> \n> \tif (!strcmp(argv[1], \"run-command-no-jobs\"))\n> -\t\texit(run_processes_parallel(jobs, no_job,\n> -\t\t\t\t\t NULL, task_finished, &proc));\n> +\t\texit(run_processes_parallel(jobs, no_job, NULL, task_finished,\n> +\t\t\t\t\t &proc, &opts));\n> \n> \tfprintf(stderr, \"check usage\\n\");\n> \treturn 1;\n> -- \n> 2.36.0.893.g80a51c675f6\n> \n"},{"id":"454606","messageId":"YmsisnOwekVUdDEd@google.com","threadId":"57764","inReplyTo":"patch-3.6-d76f63c2948-20220421T122108Z-avarab@gmail.com","subject":"Re: [PATCH 3/6] run-command: add an \"ungroup\" option to run_process_parallel()","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2022-04-28T23:26:42Z","receivedAt":"2022-04-28T23:26:54Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Apr 21, 2022 at 02:25:28PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> Extend the parallel execution API added in c553c72eed6 (run-command:\n> add an asynchronous parallel child processor, 2015-12-15) to support a\n> mode where the stdout and stderr of the processes isn't captured and\n> output in a deterministic order, instead we'll leave it to the kernel\n> and stdio to sort it out.\n> \n> This gives the API same functionality as GNU parallel's --ungroup\n> option. As we'll see in a subsequent commit the main reason to want\n> this is to support stdout and stderr being connected to the TTY in the\n> case of jobs=1, demonstrated here with GNU parallel:\n> \n> \t$ parallel --ungroup 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n> \tTTY\n> \tTTY\n> \t$ parallel 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n> \tNTTY\n> \tNTTY\n> \n> Another is as GNU parallel's documentation notes a potential for\n> optimization. Our results will be a bit different, but in cases where\n> you want to run processes in parallel where the exact order isn't\n> important this can be a lot faster:\n> \n> \t$ hyperfine -r 3 -L o ,--ungroup 'parallel {o} seq ::: 10000000 >/dev/null '\n> \tBenchmark 1: parallel seq ::: 10000000 >/dev/null\n> \t Time (mean ± σ): 220.2 ms ± 9.3 ms [User: 124.9 ms, System: 96.1 ms]\n> \t Range (min … max): 212.3 ms … 230.5 ms 3 runs\n> \n> \tBenchmark 2: parallel --ungroup seq ::: 10000000 >/dev/null\n> \t Time (mean ± σ): 154.7 ms ± 0.9 ms [User: 136.2 ms, System: 25.1 ms]\n> \t Range (min … max): 153.9 ms … 155.7 ms 3 runs\n> \n> \tSummary\n> \t 'parallel --ungroup seq ::: 10000000 >/dev/null ' ran\n> \t 1.42 ± 0.06 times faster than 'parallel seq ::: 10000000 >/dev/null '\n> \n> A large part of the juggling in the API is to make the API safer for\n> its maintenance and consumers alike.\n> \n> For the maintenance of the API we e.g. avoid malloc()-ing the\n> \"pp->pfd\", ensuring that SANITIZE=address and other similar tools will\n> catch any unexpected misuse.\n> \n> For API consumers we take pains to never pass the non-NULL \"out\"\n> buffer to an API user that provided the \"ungroup\" option. The\n> resulting code in t/helper/test-run-command.c isn't typical of such a\n> user, i.e. they'd typically use one mode or the other, and would know\n> whether they'd provided \"ungroup\" or not.\n\nInteresting! It is separate from whether there are >1 jobs in the task\nqueue. I like that approach - but I do also think we could set ungroup\nopportunistically if the task list has only one entry or the job number\nis 1, no? I'd like to see that, too. I guess that we can't really tell\nhow many tasks are available (because it's a callback, not a list) but\nsetting ungroup if jobs=1 sounds like a reasonable improvement to me.\n\nOtherwise, this patch's code is very straightforward and looks fine.\n\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> run-command.c | 95 ++++++++++++++++++++++++++++---------\n> run-command.h | 32 +++++++++----\n> t/helper/test-run-command.c | 31 +++++++++---\n> t/t0061-run-command.sh | 30 ++++++++++++\n> 4 files changed, 151 insertions(+), 37 deletions(-)\n> \n> diff --git a/run-command.c b/run-command.c\n> index 7b8159aa235..873de21ffaf 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1468,7 +1468,7 @@ int pipe_command(struct child_process *cmd,\n> enum child_state {\n> \tGIT_CP_FREE,\n> \tGIT_CP_WORKING,\n> -\tGIT_CP_WAIT_CLEANUP,\n> +\tGIT_CP_WAIT_CLEANUP, /* only for !ungroup */\n> };\n> \n> struct parallel_processes {\n> @@ -1494,6 +1494,7 @@ struct parallel_processes {\n> \tstruct pollfd *pfd;\n> \n> \tunsigned shutdown : 1;\n> +\tunsigned ungroup:1;\n> \n> \tint output_owner;\n> \tstruct strbuf buffered_output; /* of finished children */\n> @@ -1537,8 +1538,9 @@ static void pp_init(struct parallel_processes *pp,\n> \t\t get_next_task_fn get_next_task,\n> \t\t start_failure_fn start_failure,\n> \t\t task_finished_fn task_finished,\n> -\t\t void *data)\n> +\t\t void *data, struct run_process_parallel_opts *opts)\n> {\n> +\tconst int ungroup = opts->ungroup;\n> \tint i;\n> \n> \tif (n < 1)\n> @@ -1556,16 +1558,22 @@ static void pp_init(struct parallel_processes *pp,\n> \tpp->start_failure = start_failure ? start_failure : default_start_failure;\n> \tpp->task_finished = task_finished ? task_finished : default_task_finished;\n> \n> +\tpp->ungroup = ungroup;\n> +\n> \tpp->nr_processes = 0;\n> \tpp->output_owner = 0;\n> \tpp->shutdown = 0;\n> \tCALLOC_ARRAY(pp->children, n);\n> -\tCALLOC_ARRAY(pp->pfd, n);\n> +\tif (!ungroup)\n> +\t\tCALLOC_ARRAY(pp->pfd, n);\n> +\n> \tstrbuf_init(&pp->buffered_output, 0);\n> \n> \tfor (i = 0; i < n; i++) {\n> \t\tstrbuf_init(&pp->children[i].err, 0);\n> \t\tchild_process_init(&pp->children[i].process);\n> +\t\tif (ungroup)\n> +\t\t\tcontinue;\n> \t\tpp->pfd[i].events = POLLIN | POLLHUP;\n> \t\tpp->pfd[i].fd = -1;\n> \t}\n> @@ -1576,6 +1584,7 @@ static void pp_init(struct parallel_processes *pp,\n> \n> static void pp_cleanup(struct parallel_processes *pp)\n> {\n> +\tconst int ungroup = pp->ungroup;\n> \tint i;\n> \n> \ttrace_printf(\"run_processes_parallel: done\");\n> @@ -1585,14 +1594,17 @@ static void pp_cleanup(struct parallel_processes *pp)\n> \t}\n> \n> \tfree(pp->children);\n> -\tfree(pp->pfd);\n> +\tif (!ungroup)\n> +\t\tfree(pp->pfd);\n> \n> \t/*\n> \t * When get_next_task added messages to the buffer in its last\n> \t * iteration, the buffered output is non empty.\n> \t */\n> -\tstrbuf_write(&pp->buffered_output, stderr);\n> -\tstrbuf_release(&pp->buffered_output);\n> +\tif (!ungroup) {\n> +\t\tstrbuf_write(&pp->buffered_output, stderr);\n> +\t\tstrbuf_release(&pp->buffered_output);\n> +\t}\n> \n> \tsigchain_pop_common();\n> }\n> @@ -1606,6 +1618,7 @@ static void pp_cleanup(struct parallel_processes *pp)\n> */\n> static int pp_start_one(struct parallel_processes *pp)\n> {\n> +\tconst int ungroup = pp->ungroup;\n> \tint i, code;\n> \n> \tfor (i = 0; i < pp->max_processes; i++)\n> @@ -1615,24 +1628,31 @@ static int pp_start_one(struct parallel_processes *pp)\n> \t\tBUG(\"bookkeeping is hard\");\n> \n> \tcode = pp->get_next_task(&pp->children[i].process,\n> -\t\t\t\t &pp->children[i].err,\n> +\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n> \t\t\t\t pp->data,\n> \t\t\t\t &pp->children[i].data);\n> \tif (!code) {\n> -\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> -\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\tif (!ungroup) {\n> +\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> +\t\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\t}\n> \t\treturn 1;\n> \t}\n> -\tpp->children[i].process.err = -1;\n> -\tpp->children[i].process.stdout_to_stderr = 1;\n> -\tpp->children[i].process.no_stdin = 1;\n> +\n> +\tif (!ungroup) {\n> +\t\tpp->children[i].process.err = -1;\n> +\t\tpp->children[i].process.stdout_to_stderr = 1;\n> +\t\tpp->children[i].process.no_stdin = 1;\n> +\t}\n> \n> \tif (start_command(&pp->children[i].process)) {\n> -\t\tcode = pp->start_failure(&pp->children[i].err,\n> +\t\tcode = pp->start_failure(ungroup ? NULL : &pp->children[i].err,\n> \t\t\t\t\t pp->data,\n> \t\t\t\t\t pp->children[i].data);\n> -\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> -\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\tif (!ungroup) {\n> +\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> +\t\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\t}\n> \t\tif (code)\n> \t\t\tpp->shutdown = 1;\n> \t\treturn code;\n> @@ -1640,14 +1660,26 @@ static int pp_start_one(struct parallel_processes *pp)\n> \n> \tpp->nr_processes++;\n> \tpp->children[i].state = GIT_CP_WORKING;\n> -\tpp->pfd[i].fd = pp->children[i].process.err;\n> +\tif (!ungroup)\n> +\t\tpp->pfd[i].fd = pp->children[i].process.err;\n> \treturn 0;\n> }\n> \n> +static void pp_mark_working_for_cleanup(struct parallel_processes *pp)\n> +{\n> +\tint i;\n> +\n> +\tfor (i = 0; i < pp->max_processes; i++)\n> +\t\tif (pp->children[i].state == GIT_CP_WORKING)\n> +\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n> +}\n> +\n> static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> {\n> \tint i;\n> \n> +\tassert(!pp->ungroup);\n> +\n> \twhile ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n> \t\tif (errno == EINTR)\n> \t\t\tcontinue;\n> @@ -1674,6 +1706,9 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> static void pp_output(struct parallel_processes *pp)\n> {\n> \tint i = pp->output_owner;\n> +\n> +\tassert(!pp->ungroup);\n> +\n> \tif (pp->children[i].state == GIT_CP_WORKING &&\n> \t pp->children[i].err.len) {\n> \t\tstrbuf_write(&pp->children[i].err, stderr);\n> @@ -1683,10 +1718,15 @@ static void pp_output(struct parallel_processes *pp)\n> \n> static int pp_collect_finished(struct parallel_processes *pp)\n> {\n> +\tconst int ungroup = pp->ungroup;\n> \tint i, code;\n> \tint n = pp->max_processes;\n> \tint result = 0;\n> \n> +\tif (ungroup)\n> +\t\tfor (i = 0; i < pp->max_processes; i++)\n> +\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n> +\n> \twhile (pp->nr_processes > 0) {\n> \t\tfor (i = 0; i < pp->max_processes; i++)\n> \t\t\tif (pp->children[i].state == GIT_CP_WAIT_CLEANUP)\n> @@ -1697,8 +1737,8 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> \t\tcode = finish_command(&pp->children[i].process);\n> \n> \t\tcode = pp->task_finished(code,\n> -\t\t\t\t\t &pp->children[i].err, pp->data,\n> -\t\t\t\t\t pp->children[i].data);\n> +\t\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n> +\t\t\t\t\t pp->data, pp->children[i].data);\n> \n> \t\tif (code)\n> \t\t\tresult = code;\n> @@ -1707,10 +1747,13 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> \n> \t\tpp->nr_processes--;\n> \t\tpp->children[i].state = GIT_CP_FREE;\n> -\t\tpp->pfd[i].fd = -1;\n> +\t\tif (!ungroup)\n> +\t\t\tpp->pfd[i].fd = -1;\n> \t\tchild_process_init(&pp->children[i].process);\n> \n> -\t\tif (i != pp->output_owner) {\n> +\t\tif (ungroup) {\n> +\t\t\t/* no strbuf_*() work to do here */\n> +\t\t} else if (i != pp->output_owner) {\n> \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> \t\t\tstrbuf_reset(&pp->children[i].err);\n> \t\t} else {\n> @@ -1744,12 +1787,14 @@ static int run_processes_parallel_1(int n, get_next_task_fn get_next_task,\n> \t\t\t\t void *pp_cb,\n> \t\t\t\t struct run_process_parallel_opts *opts)\n> {\n> +\tconst int ungroup = opts->ungroup;\n> \tint i, code;\n> \tint output_timeout = 100;\n> \tint spawn_cap = 4;\n> \tstruct parallel_processes pp;\n> \n> -\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n> +\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n> +\t\topts);\n> \twhile (1) {\n> \t\tfor (i = 0;\n> \t\t i < spawn_cap && !pp.shutdown &&\n> @@ -1766,8 +1811,12 @@ static int run_processes_parallel_1(int n, get_next_task_fn get_next_task,\n> \t\t}\n> \t\tif (!pp.nr_processes)\n> \t\t\tbreak;\n> -\t\tpp_buffer_stderr(&pp, output_timeout);\n> -\t\tpp_output(&pp);\n> +\t\tif (ungroup) {\n> +\t\t\tpp_mark_working_for_cleanup(&pp);\n> +\t\t} else {\n> +\t\t\tpp_buffer_stderr(&pp, output_timeout);\n> +\t\t\tpp_output(&pp);\n> +\t\t}\n> \t\tcode = pp_collect_finished(&pp);\n> \t\tif (code) {\n> \t\t\tpp.shutdown = 1;\n> diff --git a/run-command.h b/run-command.h\n> index 66e7bebd88a..936d334eee0 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -406,6 +406,10 @@ void check_pipe(int err);\n> * pp_cb is the callback cookie as passed to run_processes_parallel.\n> * You can store a child process specific callback cookie in pp_task_cb.\n> *\n> + * The \"struct strbuf *err\" parameter is either a pointer to a string\n> + * to write errors to, or NULL if the \"ungroup\" option was\n> + * provided. See run_processes_parallel() below.\n> + *\n> * Even after returning 0 to indicate that there are no more processes,\n> * this function will be called again until there are no more running\n> * child processes.\n> @@ -424,9 +428,9 @@ typedef int (*get_next_task_fn)(struct child_process *cp,\n> * This callback is called whenever there are problems starting\n> * a new process.\n> *\n> - * You must not write to stdout or stderr in this function. Add your\n> - * message to the strbuf out instead, which will be printed without\n> - * messing up the output of the other parallel processes.\n> + * The \"struct strbuf *err\" parameter is either a pointer to a string\n> + * to write errors to, or NULL if the \"ungroup\" option was\n> + * provided. See run_processes_parallel() below.\n> *\n> * pp_cb is the callback cookie as passed into run_processes_parallel,\n> * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> @@ -442,9 +446,9 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n> /**\n> * This callback is called on every child process that finished processing.\n> *\n> - * You must not write to stdout or stderr in this function. Add your\n> - * message to the strbuf out instead, which will be printed without\n> - * messing up the output of the other parallel processes.\n> + * The \"struct strbuf *err\" parameter is either a pointer to a string\n> + * to write errors to, or NULL if the \"ungroup\" option was\n> + * provided. See run_processes_parallel() below.\n> *\n> * pp_cb is the callback cookie as passed into run_processes_parallel,\n> * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> @@ -464,11 +468,16 @@ typedef int (*task_finished_fn)(int result,\n> *\n> * tr2_category & tr2_label: sets the trace2 category and label for\n> * logging. These must either be unset, or both of them must be set.\n> + *\n> + * ungroup: Ungroup output. Output is printed as soon as possible and\n> + * bypasses run-command's internal processing. This may cause output\n> + * from different commands to be mixed.\n> */\n> struct run_process_parallel_opts\n> {\n> \tconst char *tr2_category;\n> \tconst char *tr2_label;\n> +\tunsigned int ungroup:1;\n> };\n> \n> /**\n> @@ -478,12 +487,19 @@ struct run_process_parallel_opts\n> *\n> * The children started via this function run in parallel. Their output\n> * (both stdout and stderr) is routed to stderr in a manner that output\n> - * from different tasks does not interleave.\n> + * from different tasks does not interleave (but see \"ungroup\" above).\n> *\n> * start_failure_fn and task_finished_fn can be NULL to omit any\n> * special handling.\n> *\n> - * Options are passed via a \"struct run_process_parallel_opts\".\n> + * Options are passed via a \"struct run_process_parallel_opts\". If the\n> + * \"ungroup\" option isn't specified the callbacks will get a pointer\n> + * to a \"struct strbuf *out\", and must not write to stdout or stderr\n> + * as such output will mess up the output of the other parallel\n> + * processes. If \"ungroup\" option is specified callbacks will get a\n> + * NULL \"struct strbuf *out\" parameter, and are responsible for\n> + * emitting their own output, including dealing with any race\n> + * conditions due to writing in parallel to stdout and stderr.\n> */\n> int run_processes_parallel(int n, get_next_task_fn, start_failure_fn,\n> \t\t\t task_finished_fn, void *pp_cb,\n> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\n> index 9b21f2f9f83..747e57ef536 100644\n> --- a/t/helper/test-run-command.c\n> +++ b/t/helper/test-run-command.c\n> @@ -31,7 +31,11 @@ static int parallel_next(struct child_process *cp,\n> \t\treturn 0;\n> \n> \tstrvec_pushv(&cp->args, d->args.v);\n> -\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n> +\tif (err)\n> +\t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n> +\telse\n> +\t\tfprintf(stderr, \"preloaded output of a child\\n\");\n> +\n> \tnumber_callbacks++;\n> \treturn 1;\n> }\n> @@ -41,7 +45,10 @@ static int no_job(struct child_process *cp,\n> \t\t void *cb,\n> \t\t void **task_cb)\n> {\n> -\tstrbuf_addstr(err, \"no further jobs available\\n\");\n> +\tif (err)\n> +\t\tstrbuf_addstr(err, \"no further jobs available\\n\");\n> +\telse\n> +\t\tfprintf(stderr, \"no further jobs available\\n\");\n> \treturn 0;\n> }\n> \n> @@ -50,7 +57,10 @@ static int task_finished(int result,\n> \t\t\t void *pp_cb,\n> \t\t\t void *pp_task_cb)\n> {\n> -\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n> +\tif (err)\n> +\t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n> +\telse\n> +\t\tfprintf(stderr, \"asking for a quick stop\\n\");\n> \treturn 1;\n> }\n> \n> @@ -412,17 +422,26 @@ int cmd__run_command(int argc, const char **argv)\n> \tstrvec_clear(&proc.args);\n> \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n> \n> -\tif (!strcmp(argv[1], \"run-command-parallel\"))\n> +\tif (!strcmp(argv[1], \"run-command-parallel\") ||\n> +\t !strcmp(argv[1], \"run-command-parallel-ungroup\")) {\n> +\t\topts.ungroup = !strcmp(argv[1], \"run-command-parallel-ungroup\");\n> \t\texit(run_processes_parallel(jobs, parallel_next,\n> \t\t\t\t\t NULL, NULL, &proc, &opts));\n> +\t}\n> \n> -\tif (!strcmp(argv[1], \"run-command-abort\"))\n> +\tif (!strcmp(argv[1], \"run-command-abort\") ||\n> +\t !strcmp(argv[1], \"run-command-abort-ungroup\")) {\n> +\t\topts.ungroup = !strcmp(argv[1], \"run-command-abort-ungroup\");\n> \t\texit(run_processes_parallel(jobs, parallel_next, NULL,\n> \t\t\t\t\t task_finished, &proc, &opts));\n> +\t}\n> \n> -\tif (!strcmp(argv[1], \"run-command-no-jobs\"))\n> +\tif (!strcmp(argv[1], \"run-command-no-jobs\") ||\n> +\t !strcmp(argv[1], \"run-command-no-jobs-ungroup\")) {\n> +\t\topts.ungroup = !strcmp(argv[1], \"run-command-no-jobs-ungroup\");\n> \t\texit(run_processes_parallel(jobs, no_job, NULL, task_finished,\n> \t\t\t\t\t &proc, &opts));\n> +\t}\n> \n> \tfprintf(stderr, \"check usage\\n\");\n> \treturn 1;\n> diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\n> index 131fcfda90f..0a82db965e8 100755\n> --- a/t/t0061-run-command.sh\n> +++ b/t/t0061-run-command.sh\n> @@ -135,18 +135,36 @@ test_expect_success 'run_command runs in parallel with more jobs available than\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n> +\ttest-tool run-command run-command-parallel-ungroup 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_line_count = 8 out &&\n> +\ttest_line_count = 4 err\n> +'\n> +\n> test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n> \ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n> \ttest_must_be_empty out &&\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n> +\ttest-tool run-command run-command-parallel-ungroup 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_line_count = 8 out &&\n> +\ttest_line_count = 4 err\n> +'\n> +\n> test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n> \ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n> \ttest_must_be_empty out &&\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n> +\ttest-tool run-command run-command-parallel-ungroup 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_line_count = 8 out &&\n> +\ttest_line_count = 4 err\n> +'\n> +\n> cat >expect <<-EOF\n> preloaded output of a child\n> asking for a quick stop\n> @@ -162,6 +180,12 @@ test_expect_success 'run_command is asked to abort gracefully' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n> +\ttest-tool run-command run-command-abort-ungroup 3 false >out 2>err &&\n> +\ttest_must_be_empty out &&\n> +\ttest_line_count = 6 err\n> +'\n> +\n> cat >expect <<-EOF\n> no further jobs available\n> EOF\n> @@ -172,6 +196,12 @@ test_expect_success 'run_command outputs ' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'run_command outputs (ungroup) ' '\n> +\ttest-tool run-command run-command-no-jobs-ungroup 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n> +\ttest_must_be_empty out &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> test_trace () {\n> \texpect=\"$1\"\n> \tshift\n> -- \n> 2.36.0.893.g80a51c675f6\n> \n"},{"id":"454607","messageId":"Ymsj5Lv89xqLqZxD@google.com","threadId":"57764","inReplyTo":"patch-6.6-de3664f6d2b-20220421T122108Z-avarab@gmail.com","subject":"Re: [PATCH 6/6] hook API: fix v2.36.0 regression: hooks should be connected to a TTY","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2022-04-28T23:31:48Z","receivedAt":"2022-04-28T23:31:58Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Apr 21, 2022 at 02:25:31PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> Fix a regression reported[1] in f443246b9f2 (commit: convert\n> {pre-commit,prepare-commit-msg} hook to hook.h, 2021-12-22): Due to\n> using the run_process_parallel() API in the earlier 96e7225b310 (hook:\n> add 'run' subcommand, 2021-12-22) we'd capture the hook's stderr and\n> stdout, and thus lose the connection to the TTY in the case of\n> e.g. the \"pre-commit\" hook.\n> \n> As a preceding commit notes GNU parallel's similar --ungroup option\n> also has it emit output faster. While we're unlikely to have hooks\n> that emit truly massive amounts of output (or where the performance\n> thereof matters) it's still informative to measure the overhead. In a\n> similar \"seq\" test we're now ~30% faster:\n> \n> \t$ cat .git/hooks/seq-hook; git hyperfine -L rev origin/master,HEAD~0 -s 'make CFLAGS=-O3' './git hook run seq-hook'\n> \t#!/bin/sh\n> \n> \tseq 100000000\n> \tBenchmark 1: ./git hook run seq-hook' in 'origin/master\n> \t Time (mean ± σ): 787.1 ms ± 13.6 ms [User: 701.6 ms, System: 534.4 ms]\n> \t Range (min … max): 773.2 ms … 806.3 ms 10 runs\n> \n> \tBenchmark 2: ./git hook run seq-hook' in 'HEAD~0\n> \t Time (mean ± σ): 603.4 ms ± 1.6 ms [User: 573.1 ms, System: 30.3 ms]\n> \t Range (min … max): 601.0 ms … 606.2 ms 10 runs\n> \n> \tSummary\n> \t './git hook run seq-hook' in 'HEAD~0' ran\n> \t 1.30 ± 0.02 times faster than './git hook run seq-hook' in 'origin/master'\n> \n> In the preceding commit we removed the \"no_stdin=1\" and\n> \"stdout_to_stderr=1\" assignments. This change brings them back as with\n> \".ungroup=1\" the run_process_parallel() function doesn't provide them\n> for us implicitly.\n> \n> As an aside omitting the stdout_to_stderr=1 here would have all tests\n> pass, except those that test \"git hook run\" itself in\n> t1800-hook.sh. But our tests passing is the result of another test\n> blind spot, as was the case with the regression being fixed here. The\n> \"stdout_to_stderr=1\" for hooks is long-standing behavior, see\n> e.g. 1d9e8b56fe3 (Split back out update_hook handling in receive-pack,\n> 2007-03-10) and other follow-up commits (running \"git log\" with\n> \"--reverse -p -Gstdout_to_stderr\" is a good start).\n> \n> 1. https://lore.kernel.org/git/CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com/\n> \n> Reported-by: Anthony Sottile \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> hook.c | 8 +++++++-\n> t/t1800-hook.sh | 37 +++++++++++++++++++++++++++++++++++++\n> 2 files changed, 44 insertions(+), 1 deletion(-)\n> \n> diff --git a/hook.c b/hook.c\n> index 68ee4030551..f5eef1d561b 100644\n> --- a/hook.c\n> +++ b/hook.c\n> @@ -53,7 +53,9 @@ static int pick_next_hook(struct child_process *cp,\n> \tif (!hook_path)\n> \t\treturn 0;\n> \n> +\tcp->no_stdin = 1;\n> \tstrvec_pushv(&cp->env_array, hook_cb->options->env.v);\n> +\tcp->stdout_to_stderr = 1;\n> \tcp->trace2_hook_name = hook_cb->hook_name;\n> \tcp->dir = hook_cb->options->dir;\n> \n> @@ -119,16 +121,20 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n> \t\t.options = options,\n> \t};\n> \tconst char *const hook_path = find_hook(hook_name);\n> -\tint jobs = 1;\n> +\tconst int jobs = 1;\n> \tint ret = 0;\n> \tstruct run_process_parallel_opts run_opts = {\n> \t\t.tr2_category = \"hook\",\n> \t\t.tr2_label = hook_name,\n> +\t\t.ungroup = jobs == 1,\n\nSo here we do set .ungroup based only on the job count - but we do that\nonly for hooks, which means someone else could conceivably come across\nsimilar bug in their later use of run_processes_parallel. Is the reason\nfor doing this in the context of the hook library instead of in the\ncontext of run_processes_parallel library just because we are not sure\nif it will break other parallel callers? Or some other reason?\n\n - Emily\n\n> \t};\n> \n> \tif (!options)\n> \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n> \n> +\tif (jobs != 1 || !run_opts.ungroup)\n> +\t\tBUG(\"TODO: think about & document order & interleaving of parallel hook output\");\n> +\n> \tif (options->invoked_hook)\n> \t\t*options->invoked_hook = 0;\n> \n> diff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\n> index 1e4adc3d53e..f22754deccc 100755\n> --- a/t/t1800-hook.sh\n> +++ b/t/t1800-hook.sh\n> @@ -4,6 +4,7 @@ test_description='git-hook command'\n> \n> TEST_PASSES_SANITIZE_LEAK=true\n> . ./test-lib.sh\n> +. \"$TEST_DIRECTORY\"/lib-terminal.sh\n> \n> test_expect_success 'git hook usage' '\n> \ttest_expect_code 129 git hook &&\n> @@ -120,4 +121,40 @@ test_expect_success 'git -c core.hooksPath= hook run' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_hook_tty() {\n> +\tlocal fd=\"$1\" &&\n> +\n> +\tcat >expect &&\n> +\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\n> +\ttest_hook -C repo pre-commit <<-EOF &&\n> +\t{\n> +\t\ttest -t 1 && echo >&$fd STDOUT TTY || echo >&$fd STDOUT NO TTY &&\n> +\t\ttest -t 2 && echo >&$fd STDERR TTY || echo >&$fd STDERR NO TTY\n> +\t} $fd>actual\n> +\tEOF\n> +\n> +\ttest_commit -C repo A &&\n> +\ttest_commit -C repo B &&\n> +\tgit -C repo reset --soft HEAD^ &&\n> +\ttest_terminal git -C repo commit -m\"B.new\" &&\n> +\ttest_cmp expect repo/actual\n> +}\n> +\n> +test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDOUT redirect' '\n> +\ttest_hook_tty 1 <<-\\EOF\n> +\tSTDOUT NO TTY\n> +\tSTDERR TTY\n> +\tEOF\n> +'\n> +\n> +test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDERR redirect' '\n> +\ttest_hook_tty 2 <<-\\EOF\n> +\tSTDOUT TTY\n> +\tSTDERR NO TTY\n> +\tEOF\n> +'\n> +\n> test_done\n> -- \n> 2.36.0.893.g80a51c675f6\n> \n"},{"id":"454624","messageId":"xmqqv8urdekz.fsf@gitster.g","threadId":"57764","inReplyTo":"YmsgWj5vPEWNyGFA@google.com","subject":"Re: [PATCH 1/6] run-command API: replace run_processes_parallel_tr2() with opts struct","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-29T16:44:28Z","receivedAt":"2022-04-29T16:44:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Emily Shaffer writes:\n\n> If I'm being honest, I'd rather see everything _but_ the trace2 stuff go\n> into an opts struct, and then see the same entry points we have today\n> (run_processes_parallel that takes a struct, run_processes_parallel_tr2\n> that takes a struct and two tr2 string args). Or, I guess, a single\n> run_processes_parallel() that only takes a struct, does the right thing\n> with the trace args, and entirely removes the\n> run_processes_parallel_tr2 call.\n\nYup, it was the impression I had when I saw this patch for the first\ntime.\n\n"},{"id":"454641","messageId":"xmqqlevna4b8.fsf@gitster.g","threadId":"57764","inReplyTo":"patch-5.6-98c26c9917b-20220421T122108Z-avarab@gmail.com","subject":"Re: [PATCH 5/6] hook API: don't redundantly re-set \"no_stdin\" and \"stdout_to_stderr\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-29T22:54:35Z","receivedAt":"2022-04-29T22:54:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n[jc: I stopped reviewing at 4/6 in the last batch, thought that it\nwas enough for a reroll and shifted my attention to address other\nregressions. Let me come back to this topic to finish commenting\nbefore a reroll comes.]\n\n> Amend code added in 96e7225b310 (hook: add 'run' subcommand,\n> 2021-12-22) top stop setting these two flags. We use the\n\n\"top stop\"? -ECANNOTPARSE.\n\n> run_process_parallel() API added in c553c72eed6 (run-command: add an\n> asynchronous parallel child processor, 2015-12-15), which always sets\n> these in pp_start_one() (in addition to setting .err = -1).\n>\n> Note that an assert() to check that these values are already what\n> we're setting them to here would fail. That's because in\n> pp_start_one() we'll set these after calling this \"get_next_task\"\n> callback (which we call pick_next_hook()). But the only case where we\n> weren't setting these just after returning from this function was if\n> we took the \"return 0\" path here, in which case we wouldn't have set\n> these.\n>\n> So while this code wasn't wrong, it was entirely redundant. The\n> run_process_parallel() also can't work with a generic \"struct\n> child_process\", it needs one that's behaving in a way that it expects\n> when it comes to stderr/stdout. So we shouldn't be changing these\n> values, or in this case keeping around code that gives the impression\n> that doing in the general case is OK.\n\nOK. As long as we set these two fields correctly (i.e. the hooks do\nnot read from the standard input, and stdout_to_stderr is in effect\n(i.e. dup2(2, 1) is done), by the time we pass this into run_command()\nAPI, this step would be a benign no-op. Good.\n\nNot that it is something I would expect to see in a rather urgent\npost-release regression fix, though.\n\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> hook.c | 2 --\n> 1 file changed, 2 deletions(-)\n>\n> diff --git a/hook.c b/hook.c\n> index eadb2d58a7b..68ee4030551 100644\n> --- a/hook.c\n> +++ b/hook.c\n> @@ -53,9 +53,7 @@ static int pick_next_hook(struct child_process *cp,\n> \tif (!hook_path)\n> \t\treturn 0;\n> \n> -\tcp->no_stdin = 1;\n> \tstrvec_pushv(&cp->env_array, hook_cb->options->env.v);\n> -\tcp->stdout_to_stderr = 1;\n> \tcp->trace2_hook_name = hook_cb->hook_name;\n> \tcp->dir = hook_cb->options->dir;\n"},{"id":"454642","messageId":"xmqqfslva3mx.fsf@gitster.g","threadId":"57764","inReplyTo":"patch-6.6-de3664f6d2b-20220421T122108Z-avarab@gmail.com","subject":"Re: [PATCH 6/6] hook API: fix v2.36.0 regression: hooks should be connected to a TTY","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-29T23:09:10Z","receivedAt":"2022-04-29T23:09:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> In the preceding commit we removed the \"no_stdin=1\" and\n> \"stdout_to_stderr=1\" assignments. This change brings them back as with\n> \".ungroup=1\" the run_process_parallel() function doesn't provide them\n> for us implicitly.\n\nWait.\n\nNo hunk in this step updates how pick_next_hook() is called\n(presumably \"with .ungroup=1\"), and there is no change to the code\nthat uses the cp structure prepared by pick_next_hook() further, so\nwhy does the change in the previous step need to be reverted in this\nstep? Does it mean if we apply patches 1-5 without this step, because\nof step 5, the contents of cp structure returned by pick_next_hook()\nis broken? But we clearly saw a claim that these assignments are\nredundant and unnecessary.\n\nSo I am confused as to what change in this step makes these\nassignments necessary again? There is no removal of assignments to\nthese two members that we used to have in this patch. There is no\naddition of a new caller that calls pick_next_hook and uses it\ndifferently (i.e. the other existing caller(s) made assignments to\nthese two members, which allowed 5/6 to remove the assignments, but\nif this step adds a different caller that uses the struct without\nmaking these assignments, then we do need to add them back).\n\nEither I am reading a wrong patch, or the steps 5 & 6 confused me\nbeyond repair, or perhaps a bit of both?\n\nIf [5/6] were not there, and [6/6] was added because [5/6] broke it\nby forgetting that some caller that already exists after applying\n[5/6] did not make assignments to these two members (iow, the\nassignment removed by that step were not redundant and [5/6] was\nbuggy in removing them), then I would understand it, but that does\nnot seem to be the case...\n\nPuzzled and utterly confused I am.\n\n> @@ -53,7 +53,9 @@ static int pick_next_hook(struct child_process *cp,\n> \tif (!hook_path)\n> \t\treturn 0;\n> \n> +\tcp->no_stdin = 1;\n> \tstrvec_pushv(&cp->env_array, hook_cb->options->env.v);\n> +\tcp->stdout_to_stderr = 1;\n> \tcp->trace2_hook_name = hook_cb->hook_name;\n> \tcp->dir = hook_cb->options->dir;\n> \n> @@ -119,16 +121,20 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n> \t\t.options = options,\n> \t};\n> \tconst char *const hook_path = find_hook(hook_name);\n> -\tint jobs = 1;\n> +\tconst int jobs = 1;\n> \tint ret = 0;\n> \tstruct run_process_parallel_opts run_opts = {\n> \t\t.tr2_category = \"hook\",\n> \t\t.tr2_label = hook_name,\n> +\t\t.ungroup = jobs == 1,\n> \t};\n> \n> \tif (!options)\n> \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n> \n> +\tif (jobs != 1 || !run_opts.ungroup)\n> +\t\tBUG(\"TODO: think about & document order & interleaving of parallel hook output\");\n> +\n> \tif (options->invoked_hook)\n> \t\t*options->invoked_hook = 0;\n> \n> diff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\n> index 1e4adc3d53e..f22754deccc 100755\n> --- a/t/t1800-hook.sh\n> +++ b/t/t1800-hook.sh\n> @@ -4,6 +4,7 @@ test_description='git-hook command'\n> \n> TEST_PASSES_SANITIZE_LEAK=true\n> . ./test-lib.sh\n> +. \"$TEST_DIRECTORY\"/lib-terminal.sh\n> \n> test_expect_success 'git hook usage' '\n> \ttest_expect_code 129 git hook &&\n> @@ -120,4 +121,40 @@ test_expect_success 'git -c core.hooksPath= hook run' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_hook_tty() {\n> +\tlocal fd=\"$1\" &&\n> +\n> +\tcat >expect &&\n> +\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\n> +\ttest_hook -C repo pre-commit <<-EOF &&\n> +\t{\n> +\t\ttest -t 1 && echo >&$fd STDOUT TTY || echo >&$fd STDOUT NO TTY &&\n> +\t\ttest -t 2 && echo >&$fd STDERR TTY || echo >&$fd STDERR NO TTY\n> +\t} $fd>actual\n> +\tEOF\n> +\n> +\ttest_commit -C repo A &&\n> +\ttest_commit -C repo B &&\n> +\tgit -C repo reset --soft HEAD^ &&\n> +\ttest_terminal git -C repo commit -m\"B.new\" &&\n> +\ttest_cmp expect repo/actual\n> +}\n> +\n> +test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDOUT redirect' '\n> +\ttest_hook_tty 1 <<-\\EOF\n> +\tSTDOUT NO TTY\n> +\tSTDERR TTY\n> +\tEOF\n> +'\n> +\n> +test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDERR redirect' '\n> +\ttest_hook_tty 2 <<-\\EOF\n> +\tSTDOUT TTY\n> +\tSTDERR NO TTY\n> +\tEOF\n> +'\n> +\n> test_done\n"},{"id":"455415","messageId":"patch-v2-1.8-26a81eff267-20220518T195858Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com","subject":"[PATCH v2 1/8] run-command tests: change if/if/... to if/else if/else","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-18T20:05:17Z","receivedAt":"2022-05-18T20:05:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor the code in cmd__run_command() to make a subsequent changes\nsmaller by reducing duplication.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/helper/test-run-command.c | 28 +++++++++++++++-------------\n 1 file changed, 15 insertions(+), 13 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex f3b90aa834a..bd98dd9624b 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -371,6 +371,8 @@ int cmd__run_command(int argc, const char **argv)\n {\n \tstruct child_process proc = CHILD_PROCESS_INIT;\n \tint jobs;\n+\tget_next_task_fn next_fn = NULL;\n+\ttask_finished_fn finished_fn = NULL;\n \n \tif (argc > 1 && !strcmp(argv[1], \"testsuite\"))\n \t\texit(testsuite(argc - 1, argv + 1));\n@@ -411,18 +413,18 @@ int cmd__run_command(int argc, const char **argv)\n \tstrvec_clear(&proc.args);\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n-\tif (!strcmp(argv[1], \"run-command-parallel\"))\n-\t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, NULL, &proc));\n-\n-\tif (!strcmp(argv[1], \"run-command-abort\"))\n-\t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, task_finished, &proc));\n-\n-\tif (!strcmp(argv[1], \"run-command-no-jobs\"))\n-\t\texit(run_processes_parallel(jobs, no_job,\n-\t\t\t\t\t NULL, task_finished, &proc));\n+\tif (!strcmp(argv[1], \"run-command-parallel\")) {\n+\t\tnext_fn = parallel_next;\n+\t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n+\t\tnext_fn = parallel_next;\n+\t\tfinished_fn = task_finished;\n+\t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n+\t\tnext_fn = no_job;\n+\t\tfinished_fn = task_finished;\n+\t} else {\n+\t\tfprintf(stderr, \"check usage\\n\");\n+\t\treturn 1;\n+\t}\n \n-\tfprintf(stderr, \"check usage\\n\");\n-\treturn 1;\n+\texit(run_processes_parallel(jobs, next_fn, NULL, finished_fn, &proc));\n }\n-- \n2.36.1.952.g0ae626f6cd7\n\n"},{"id":"455416","messageId":"patch-v2-2.8-5f0a6e9925f-20220518T195858Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com","subject":"[PATCH v2 2/8] run-command API: use \"opts\" struct for run_processes_parallel{,_tr2}()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-18T20:05:18Z","receivedAt":"2022-05-18T20:05:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a new \"struct run_process_parallel_opts\" to replace the growing\nrun_processes_parallel() and run_processes_parallel_tr2() argument\nlists. This refactoring makes it easier to add new options and\nparameters easier.\n\nThe *_tr2() variant of the function was added in ee4512ed481 (trace2:\ncreate new combined trace facility, 2019-02-22), and has subsequently\nbeen used by every caller except t/helper/test-run-command.c.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fetch.c | 18 +++++++++-----\n builtin/submodule--helper.c | 15 ++++++++----\n hook.c | 21 +++++++++-------\n run-command.c | 48 ++++++++++++++++---------------------\n run-command.h | 35 ++++++++++++++++++++-------\n submodule.c | 18 +++++++++-----\n t/helper/test-run-command.c | 17 ++++++++++---\n 7 files changed, 109 insertions(+), 63 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex e3791f09ed5..d85bf135e66 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1948,14 +1948,20 @@ static int fetch_multiple(struct string_list *list, int max_children)\n \n \tif (max_children != 1 && list->nr != 1) {\n \t\tstruct parallel_fetch_state state = { argv.v, list, 0, 0 };\n+\t\tstruct run_process_parallel_opts run_opts = {\n+\t\t\t.tr2_category = \"fetch\",\n+\t\t\t.tr2_label = \"parallel/fetch\",\n+\n+\t\t\t.jobs = max_children,\n+\n+\t\t\t.get_next_task = &fetch_next_remote,\n+\t\t\t.start_failure = &fetch_failed_to_start,\n+\t\t\t.task_finished = &fetch_finished,\n+\t\t\t.data = &state,\n+\t\t};\n \n \t\tstrvec_push(&argv, \"--end-of-options\");\n-\t\tresult = run_processes_parallel_tr2(max_children,\n-\t\t\t\t\t\t &fetch_next_remote,\n-\t\t\t\t\t\t &fetch_failed_to_start,\n-\t\t\t\t\t\t &fetch_finished,\n-\t\t\t\t\t\t &state,\n-\t\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n+\t\tresult = run_processes_parallel(&run_opts);\n \n \t\tif (!result)\n \t\t\tresult = state.result;\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 1a8e5d06214..756807e965d 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -2651,12 +2651,19 @@ static int update_submodules(struct update_data *update_data)\n {\n \tint i, res = 0;\n \tstruct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;\n+\tstruct run_process_parallel_opts run_opts = {\n+\t\t.tr2_category = \"submodule\",\n+\t\t.tr2_label = \"parallel/update\",\n+\n+\t\t.get_next_task = update_clone_get_next_task,\n+\t\t.start_failure = update_clone_start_failure,\n+\t\t.task_finished = update_clone_task_finished,\n+\t\t.data = &suc,\n+\t};\n \n \tsuc.update_data = update_data;\n-\trun_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,\n-\t\t\t\t update_clone_start_failure,\n-\t\t\t\t update_clone_task_finished, &suc, \"submodule\",\n-\t\t\t\t \"parallel/update\");\n+\trun_opts.jobs = suc.update_data->max_jobs;\n+\trun_processes_parallel(&run_opts);\n \n \t/*\n \t * We saved the output and put it out all at once now.\ndiff --git a/hook.c b/hook.c\nindex 1d51be3b77a..9aefccfc34a 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -121,8 +121,19 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \t\t.options = options,\n \t};\n \tconst char *const hook_path = find_hook(hook_name);\n-\tint jobs = 1;\n+\tconst int jobs = 1;\n \tint ret = 0;\n+\tstruct run_process_parallel_opts run_opts = {\n+\t\t.tr2_category = \"hook\",\n+\t\t.tr2_label = hook_name,\n+\n+\t\t.jobs = jobs,\n+\n+\t\t.get_next_task = pick_next_hook,\n+\t\t.start_failure = notify_start_failure,\n+\t\t.task_finished = notify_hook_finished,\n+\t\t.data = &cb_data,\n+\t};\n \n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n@@ -144,13 +155,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \t\tcb_data.hook_path = abs_path.buf;\n \t}\n \n-\trun_processes_parallel_tr2(jobs,\n-\t\t\t\t pick_next_hook,\n-\t\t\t\t notify_start_failure,\n-\t\t\t\t notify_hook_finished,\n-\t\t\t\t &cb_data,\n-\t\t\t\t \"hook\",\n-\t\t\t\t hook_name);\n+\trun_processes_parallel(&run_opts);\n \tret = cb_data.rc;\n cleanup:\n \tstrbuf_release(&abs_path);\ndiff --git a/run-command.c b/run-command.c\nindex a8501e38ceb..8c156fd080e 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1533,13 +1533,14 @@ static void handle_children_on_signal(int signo)\n }\n \n static void pp_init(struct parallel_processes *pp,\n-\t\t int n,\n-\t\t get_next_task_fn get_next_task,\n-\t\t start_failure_fn start_failure,\n-\t\t task_finished_fn task_finished,\n-\t\t void *data)\n+\t\t struct run_process_parallel_opts *opts)\n {\n \tint i;\n+\tint n = opts->jobs;\n+\tvoid *data = opts->data;\n+\tget_next_task_fn get_next_task = opts->get_next_task;\n+\tstart_failure_fn start_failure = opts->start_failure;\n+\ttask_finished_fn task_finished = opts->task_finished;\n \n \tif (n < 1)\n \t\tn = online_cpus();\n@@ -1738,18 +1739,23 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n-int run_processes_parallel(int n,\n-\t\t\t get_next_task_fn get_next_task,\n-\t\t\t start_failure_fn start_failure,\n-\t\t\t task_finished_fn task_finished,\n-\t\t\t void *pp_cb)\n+int run_processes_parallel(struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n \tstruct parallel_processes pp;\n+\tconst char *tr2_category = opts->tr2_category;\n+\tconst char *tr2_label = opts->tr2_label;\n+\tconst int do_trace2 = tr2_category && tr2_label;\n+\tconst int n = opts->jobs;\n \n-\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n+\tif (do_trace2)\n+\t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n+\t\t\t\t\t \"max:%d\", ((n < 1) ? online_cpus()\n+\t\t\t\t\t\t : n));\n+\n+\tpp_init(&pp, opts);\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1777,25 +1783,11 @@ int run_processes_parallel(int n,\n \t}\n \n \tpp_cleanup(&pp);\n-\treturn 0;\n-}\n-\n-int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n-\t\t\t start_failure_fn start_failure,\n-\t\t\t task_finished_fn task_finished, void *pp_cb,\n-\t\t\t const char *tr2_category, const char *tr2_label)\n-{\n-\tint result;\n \n-\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n-\t\t\t\t ((n < 1) ? online_cpus() : n));\n+\tif (do_trace2)\n+\t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n \n-\tresult = run_processes_parallel(n, get_next_task, start_failure,\n-\t\t\t\t\ttask_finished, pp_cb);\n-\n-\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n-\n-\treturn result;\n+\treturn 0;\n }\n \n int run_auto_maintenance(int quiet)\ndiff --git a/run-command.h b/run-command.h\nindex 5bd0c933e80..b0268ed3db1 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -458,6 +458,32 @@ typedef int (*task_finished_fn)(int result,\n \t\t\t\tvoid *pp_task_cb);\n \n /**\n+ * Options to pass to run_processes_parallel(), { 0 }-initialized\n+ * means no options. Fields:\n+ *\n+ * tr2_category & tr2_label: sets the trace2 category and label for\n+ * logging. These must either be unset, or both of them must be set.\n+ *\n+ * jobs: see 'n' in run_processes_parallel() below.\n+ *\n+ * *_fn & data: see run_processes_parallel() below.\n+ */\n+struct run_process_parallel_opts\n+{\n+\tconst char *tr2_category;\n+\tconst char *tr2_label;\n+\n+\tint jobs;\n+\n+\tget_next_task_fn get_next_task;\n+\tstart_failure_fn start_failure;\n+\ttask_finished_fn task_finished;\n+\tvoid *data;\n+};\n+\n+/**\n+ * Options are passed via the \"struct run_process_parallel_opts\" above.\n+\n * Runs up to n processes at the same time. Whenever a process can be\n * started, the callback get_next_task_fn is called to obtain the data\n * required to start another child process.\n@@ -469,14 +495,7 @@ typedef int (*task_finished_fn)(int result,\n * start_failure_fn and task_finished_fn can be NULL to omit any\n * special handling.\n */\n-int run_processes_parallel(int n,\n-\t\t\t get_next_task_fn,\n-\t\t\t start_failure_fn,\n-\t\t\t task_finished_fn,\n-\t\t\t void *pp_cb);\n-int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n-\t\t\t task_finished_fn, void *pp_cb,\n-\t\t\t const char *tr2_category, const char *tr2_label);\n+int run_processes_parallel(struct run_process_parallel_opts *opts);\n \n /**\n * Convenience function which prepares env_array for a command to be run in a\ndiff --git a/submodule.c b/submodule.c\nindex 86c8f0f89db..8cbcd3fce23 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1817,6 +1817,17 @@ int fetch_submodules(struct repository *r,\n {\n \tint i;\n \tstruct submodule_parallel_fetch spf = SPF_INIT;\n+\tstruct run_process_parallel_opts run_opts = {\n+\t\t.tr2_category = \"submodule\",\n+\t\t.tr2_label = \"parallel/fetch\",\n+\n+\t\t.jobs = max_parallel_jobs,\n+\n+\t\t.get_next_task = get_next_submodule,\n+\t\t.start_failure = fetch_start_failure,\n+\t\t.task_finished = fetch_finish,\n+\t\t.data = &spf,\n+\t};\n \n \tspf.r = r;\n \tspf.command_line_option = command_line_option;\n@@ -1838,12 +1849,7 @@ int fetch_submodules(struct repository *r,\n \n \tcalculate_changed_submodule_paths(r, &spf.changed_submodule_names);\n \tstring_list_sort(&spf.changed_submodule_names);\n-\trun_processes_parallel_tr2(max_parallel_jobs,\n-\t\t\t\t get_next_submodule,\n-\t\t\t\t fetch_start_failure,\n-\t\t\t\t fetch_finish,\n-\t\t\t\t &spf,\n-\t\t\t\t \"submodule\", \"parallel/fetch\");\n+\trun_processes_parallel(&run_opts);\n \n \tif (spf.submodules_with_errors.len > 0)\n \t\tfprintf(stderr, _(\"Errors during submodule fetch:\\n%s\"),\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex bd98dd9624b..56a806f228b 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -142,6 +142,12 @@ static int testsuite(int argc, const char **argv)\n \t\t\t \"write JUnit-style XML files\"),\n \t\tOPT_END()\n \t};\n+\tstruct run_process_parallel_opts run_opts = {\n+\t\t.get_next_task = next_test,\n+\t\t.start_failure = test_failed,\n+\t\t.task_finished = test_finished,\n+\t\t.data = &suite,\n+\t};\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\ttestsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);\n@@ -181,9 +187,9 @@ static int testsuite(int argc, const char **argv)\n \n \tfprintf(stderr, \"Running %\"PRIuMAX\" tests (%d at a time)\\n\",\n \t\t(uintmax_t)suite.tests.nr, max_jobs);\n+\trun_opts.jobs = max_jobs;\n \n-\tret = run_processes_parallel(max_jobs, next_test, test_failed,\n-\t\t\t\t test_finished, &suite);\n+\tret = run_processes_parallel(&run_opts);\n \n \tif (suite.failed.nr > 0) {\n \t\tret = 1;\n@@ -373,6 +379,7 @@ int cmd__run_command(int argc, const char **argv)\n \tint jobs;\n \tget_next_task_fn next_fn = NULL;\n \ttask_finished_fn finished_fn = NULL;\n+\tstruct run_process_parallel_opts opts = { 0 };\n \n \tif (argc > 1 && !strcmp(argv[1], \"testsuite\"))\n \t\texit(testsuite(argc - 1, argv + 1));\n@@ -412,6 +419,8 @@ int cmd__run_command(int argc, const char **argv)\n \tjobs = atoi(argv[2]);\n \tstrvec_clear(&proc.args);\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n+\topts.jobs = jobs;\n+\topts.data = &proc;\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n \t\tnext_fn = parallel_next;\n@@ -426,5 +435,7 @@ int cmd__run_command(int argc, const char **argv)\n \t\treturn 1;\n \t}\n \n-\texit(run_processes_parallel(jobs, next_fn, NULL, finished_fn, &proc));\n+\topts.get_next_task = next_fn;\n+\topts.task_finished = finished_fn;\n+\texit(run_processes_parallel(&opts));\n }\n-- \n2.36.1.952.g0ae626f6cd7\n\n"},{"id":"455417","messageId":"patch-v2-3.8-a8e1fc07b65-20220518T195858Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com","subject":"[PATCH v2 3/8] run-command tests: test stdout of run_command_parallel()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-18T20:05:19Z","receivedAt":"2022-05-18T20:05:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Extend the tests added in c553c72eed6 (run-command: add an\nasynchronous parallel child processor, 2015-12-15) to test stdout in\naddition to stderr. A subsequent commit will add additional related\ntests for a new feature, making it obvious how the output of the two\ncompares on both stdout and stderr will make this easier to reason\nabout.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t0061-run-command.sh | 25 +++++++++++++++----------\n 1 file changed, 15 insertions(+), 10 deletions(-)\n\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex ee281909bc3..7d00f3cc2af 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -130,18 +130,21 @@ World\n EOF\n \n test_expect_success 'run_command runs in parallel with more jobs available than tasks' '\n-\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n-\ttest_cmp expect actual\n+\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n '\n \n test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n-\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n-\ttest_cmp expect actual\n+\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n '\n \n test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n-\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n-\ttest_cmp expect actual\n+\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n '\n \n cat >expect <<-EOF\n@@ -154,8 +157,9 @@ asking for a quick stop\n EOF\n \n test_expect_success 'run_command is asked to abort gracefully' '\n-\ttest-tool run-command run-command-abort 3 false 2>actual &&\n-\ttest_cmp expect actual\n+\ttest-tool run-command run-command-abort 3 false >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n '\n \n cat >expect <<-EOF\n@@ -163,8 +167,9 @@ no further jobs available\n EOF\n \n test_expect_success 'run_command outputs ' '\n-\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n-\ttest_cmp expect actual\n+\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n '\n \n test_trace () {\n-- \n2.36.1.952.g0ae626f6cd7\n\n"},{"id":"455418","messageId":"cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-0.6-00000000000-20220421T122108Z-avarab@gmail.com","subject":"[PATCH v2 0/8] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-18T20:05:16Z","receivedAt":"2022-05-18T20:05:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A re-roll of v1[1]. I believe this addresses all comments on the v1\n(but perhaps I missed something). Changes:\n\n * The run_processes_parallel() now takes only one argument, the new\n \"opts\" struct which has options, callbacks etc. This will also make\n the subsequent config-based hooks topic less churny (it needs new\n callbacks).\n\n As a result the whole internal *_tr2() wrapper/static function are\n gone.\n\n * Replaced checks of \"ungroup\" with whether we have a NULL or not\n (e.g. for pp->pfd), also for free().\n\n * Typo/grammar fixes in commit messages.\n\n * Hopefully the 8/8 is less confusing vis-a-vis\n https://lore.kernel.org/git/xmqqfslva3mx.fsf@gitster.g/; I.e. now\n we only add \"stdout_to_stderr\".\n\n * The 01/08 and 04/08 are new: Splitting those out made subsequent\n diffs smaller.\n\n * Tweaked 5/8 a bit to make the diff smaller.\n\n * Used \"err\" and \"out\", not \"actual\" and \"out\" in tests, per Junio's\n suggestion.\n\nPassing CI for this series at: https://github.com/avar/git/actions/runs/2346571047\n\n1. https://lore.kernel.org/git/cover-0.6-00000000000-20220421T122108Z-avarab@gmail.com/\n\nÆvar Arnfjörð Bjarmason (8):\n run-command tests: change if/if/... to if/else if/else\n run-command API: use \"opts\" struct for run_processes_parallel{,_tr2}()\n run-command tests: test stdout of run_command_parallel()\n run-command.c: add an initializer for \"struct parallel_processes\"\n run-command: add an \"ungroup\" option to run_process_parallel()\n hook tests: fix redirection logic error in 96e7225b310\n hook API: don't redundantly re-set \"no_stdin\" and \"stdout_to_stderr\"\n hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n\n builtin/fetch.c | 18 +++--\n builtin/submodule--helper.c | 15 ++--\n hook.c | 28 +++++---\n run-command.c | 132 +++++++++++++++++++++++-------------\n run-command.h | 66 ++++++++++++++----\n submodule.c | 18 +++--\n t/helper/test-run-command.c | 65 ++++++++++++------\n t/t0061-run-command.sh | 55 ++++++++++++---\n t/t1800-hook.sh | 39 ++++++++++-\n 9 files changed, 316 insertions(+), 120 deletions(-)\n\nRange-diff against v1:\n-: ----------- > 1: 26a81eff267 run-command tests: change if/if/... to if/else if/else\n1: 8bf71ce63dd ! 2: 5f0a6e9925f run-command API: replace run_processes_parallel_tr2() with opts struct\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - run-command API: replace run_processes_parallel_tr2() with opts struct\n + run-command API: use \"opts\" struct for run_processes_parallel{,_tr2}()\n \n - Add a new \"struct run_process_parallel_opts\" to cover the trace2\n - use-case added in ee4512ed481 (trace2: create new combined trace\n - facility, 2019-02-22). A subsequent commit will add more options, and\n - having a proliferation of new functions or extra parameters would\n - result in needless churn.\n + Add a new \"struct run_process_parallel_opts\" to replace the growing\n + run_processes_parallel() and run_processes_parallel_tr2() argument\n + lists. This refactoring makes it easier to add new options and\n + parameters easier.\n \n - It makes for a smaller change to make run_processes_parallel() and\n - run_processes_parallel_tr2() wrapper functions for the new \"static\"\n - run_processes_parallel_1(), which contains the main logic. We pass\n - down \"opts\" to the *_1() function even though it isn't used there\n - yet (only in the *_tr2() function), a subsequent commit will make more\n - use of it.\n + The *_tr2() variant of the function was added in ee4512ed481 (trace2:\n + create new combined trace facility, 2019-02-22), and has subsequently\n + been used by every caller except t/helper/test-run-command.c.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n @@ builtin/fetch.c: static int fetch_multiple(struct string_list *list, int max_chi\n +\t\tstruct run_process_parallel_opts run_opts = {\n +\t\t\t.tr2_category = \"fetch\",\n +\t\t\t.tr2_label = \"parallel/fetch\",\n ++\n ++\t\t\t.jobs = max_children,\n ++\n ++\t\t\t.get_next_task = &fetch_next_remote,\n ++\t\t\t.start_failure = &fetch_failed_to_start,\n ++\t\t\t.task_finished = &fetch_finished,\n ++\t\t\t.data = &state,\n +\t\t};\n \n \t\tstrvec_push(&argv, \"--end-of-options\");\n @@ builtin/fetch.c: static int fetch_multiple(struct string_list *list, int max_chi\n -\t\t\t\t\t\t &fetch_finished,\n -\t\t\t\t\t\t &state,\n -\t\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n -+\t\tresult = run_processes_parallel(max_children,\n -+\t\t\t\t\t\t&fetch_next_remote,\n -+\t\t\t\t\t\t&fetch_failed_to_start,\n -+\t\t\t\t\t\t&fetch_finished, &state,\n -+\t\t\t\t\t\t&run_opts);\n ++\t\tresult = run_processes_parallel(&run_opts);\n \n \t\tif (!result)\n \t\t\tresult = state.result;\n @@ builtin/submodule--helper.c: static int update_submodules(struct update_data *up\n +\tstruct run_process_parallel_opts run_opts = {\n +\t\t.tr2_category = \"submodule\",\n +\t\t.tr2_label = \"parallel/update\",\n ++\n ++\t\t.get_next_task = update_clone_get_next_task,\n ++\t\t.start_failure = update_clone_start_failure,\n ++\t\t.task_finished = update_clone_task_finished,\n ++\t\t.data = &suc,\n +\t};\n \n \tsuc.update_data = update_data;\n @@ builtin/submodule--helper.c: static int update_submodules(struct update_data *up\n -\t\t\t\t update_clone_start_failure,\n -\t\t\t\t update_clone_task_finished, &suc, \"submodule\",\n -\t\t\t\t \"parallel/update\");\n -+\trun_processes_parallel(suc.update_data->max_jobs,\n -+\t\t\t update_clone_get_next_task,\n -+\t\t\t update_clone_start_failure,\n -+\t\t\t update_clone_task_finished, &suc, &run_opts);\n ++\trun_opts.jobs = suc.update_data->max_jobs;\n ++\trun_processes_parallel(&run_opts);\n \n \t/*\n \t * We saved the output and put it out all at once now.\n \n ## hook.c ##\n @@ hook.c: int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n + \t\t.options = options,\n + \t};\n \tconst char *const hook_path = find_hook(hook_name);\n - \tint jobs = 1;\n +-\tint jobs = 1;\n ++\tconst int jobs = 1;\n \tint ret = 0;\n +\tstruct run_process_parallel_opts run_opts = {\n +\t\t.tr2_category = \"hook\",\n +\t\t.tr2_label = hook_name,\n ++\n ++\t\t.jobs = jobs,\n ++\n ++\t\t.get_next_task = pick_next_hook,\n ++\t\t.start_failure = notify_start_failure,\n ++\t\t.task_finished = notify_hook_finished,\n ++\t\t.data = &cb_data,\n +\t};\n \n \tif (!options)\n @@ hook.c: int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n -\t\t\t\t &cb_data,\n -\t\t\t\t \"hook\",\n -\t\t\t\t hook_name);\n -+\trun_processes_parallel(jobs, pick_next_hook, notify_start_failure,\n -+\t\t\t notify_hook_finished, &cb_data, &run_opts);\n ++\trun_processes_parallel(&run_opts);\n \tret = cb_data.rc;\n cleanup:\n \tstrbuf_release(&abs_path);\n \n ## run-command.c ##\n +@@ run-command.c: static void handle_children_on_signal(int signo)\n + }\n + \n + static void pp_init(struct parallel_processes *pp,\n +-\t\t int n,\n +-\t\t get_next_task_fn get_next_task,\n +-\t\t start_failure_fn start_failure,\n +-\t\t task_finished_fn task_finished,\n +-\t\t void *data)\n ++\t\t struct run_process_parallel_opts *opts)\n + {\n + \tint i;\n ++\tint n = opts->jobs;\n ++\tvoid *data = opts->data;\n ++\tget_next_task_fn get_next_task = opts->get_next_task;\n ++\tstart_failure_fn start_failure = opts->start_failure;\n ++\ttask_finished_fn task_finished = opts->task_finished;\n + \n + \tif (n < 1)\n + \t\tn = online_cpus();\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n -\t\t\t start_failure_fn start_failure,\n -\t\t\t task_finished_fn task_finished,\n -\t\t\t void *pp_cb)\n -+static int run_processes_parallel_1(int n, get_next_task_fn get_next_task,\n -+\t\t\t\t start_failure_fn start_failure,\n -+\t\t\t\t task_finished_fn task_finished,\n -+\t\t\t\t void *pp_cb,\n -+\t\t\t\t struct run_process_parallel_opts *opts)\n ++int run_processes_parallel(struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n \tint output_timeout = 100;\n + \tint spawn_cap = 4;\n + \tstruct parallel_processes pp;\n ++\tconst char *tr2_category = opts->tr2_category;\n ++\tconst char *tr2_label = opts->tr2_label;\n ++\tconst int do_trace2 = tr2_category && tr2_label;\n ++\tconst int n = opts->jobs;\n + \n +-\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n ++\tif (do_trace2)\n ++\t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n ++\t\t\t\t\t \"max:%d\", ((n < 1) ? online_cpus()\n ++\t\t\t\t\t\t : n));\n ++\n ++\tpp_init(&pp, opts);\n + \twhile (1) {\n + \t\tfor (i = 0;\n + \t\t i < spawn_cap && !pp.shutdown &&\n @@ run-command.c: int run_processes_parallel(int n,\n - \treturn 0;\n - }\n + \t}\n \n + \tpp_cleanup(&pp);\n +-\treturn 0;\n +-}\n +-\n -int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n -\t\t\t start_failure_fn start_failure,\n -\t\t\t task_finished_fn task_finished, void *pp_cb,\n -\t\t\t const char *tr2_category, const char *tr2_label)\n -+static int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n -+\t\t\t\t start_failure_fn start_failure,\n -+\t\t\t\t task_finished_fn task_finished,\n -+\t\t\t\t void *pp_cb,\n -+\t\t\t\t struct run_process_parallel_opts *opts)\n - {\n -+\tconst char *tr2_category = opts->tr2_category;\n -+\tconst char *tr2_label = opts->tr2_label;\n - \tint result;\n +-{\n +-\tint result;\n \n - \ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n - \t\t\t\t ((n < 1) ? online_cpus() : n));\n +-\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n +-\t\t\t\t ((n < 1) ? online_cpus() : n));\n ++\tif (do_trace2)\n ++\t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n \n -\tresult = run_processes_parallel(n, get_next_task, start_failure,\n -\t\t\t\t\ttask_finished, pp_cb);\n -+\tresult = run_processes_parallel_1(n, get_next_task, start_failure,\n -+\t\t\t\t\t task_finished, pp_cb, opts);\n - \n - \ttrace2_region_leave(tr2_category, tr2_label, NULL);\n - \n - \treturn result;\n +-\n +-\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n +-\n +-\treturn result;\n ++\treturn 0;\n }\n \n -+int run_processes_parallel(int n, get_next_task_fn get_next_task,\n -+\t\t\t start_failure_fn start_failure,\n -+\t\t\t task_finished_fn task_finished, void *pp_cb,\n -+\t\t\t struct run_process_parallel_opts *opts)\n -+{\n -+\tif (opts->tr2_category && opts->tr2_label)\n -+\t\treturn run_processes_parallel_tr2(n, get_next_task,\n -+\t\t\t\t\t\t start_failure, task_finished,\n -+\t\t\t\t\t\t pp_cb, opts);\n -+\n -+\treturn run_processes_parallel_1(n, get_next_task, start_failure,\n -+\t\t\t\t\ttask_finished, pp_cb, opts);\n -+}\n -+\n -+\n int run_auto_maintenance(int quiet)\n - {\n - \tint enabled;\n \n ## run-command.h ##\n @@ run-command.h: typedef int (*task_finished_fn)(int result,\n - \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n -+/**\n + /**\n + * Options to pass to run_processes_parallel(), { 0 }-initialized\n + * means no options. Fields:\n + *\n + * tr2_category & tr2_label: sets the trace2 category and label for\n + * logging. These must either be unset, or both of them must be set.\n ++ *\n ++ * jobs: see 'n' in run_processes_parallel() below.\n ++ *\n ++ * *_fn & data: see run_processes_parallel() below.\n + */\n +struct run_process_parallel_opts\n +{\n +\tconst char *tr2_category;\n +\tconst char *tr2_label;\n ++\n ++\tint jobs;\n ++\n ++\tget_next_task_fn get_next_task;\n ++\tstart_failure_fn start_failure;\n ++\ttask_finished_fn task_finished;\n ++\tvoid *data;\n +};\n +\n - /**\n ++/**\n ++ * Options are passed via the \"struct run_process_parallel_opts\" above.\n ++\n * Runs up to n processes at the same time. Whenever a process can be\n * started, the callback get_next_task_fn is called to obtain the data\n + * required to start another child process.\n @@ run-command.h: typedef int (*task_finished_fn)(int result,\n - *\n * start_failure_fn and task_finished_fn can be NULL to omit any\n * special handling.\n -+ *\n -+ * Options are passed via a \"struct run_process_parallel_opts\".\n */\n -int run_processes_parallel(int n,\n -\t\t\t get_next_task_fn,\n @@ run-command.h: typedef int (*task_finished_fn)(int result,\n -int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n -\t\t\t task_finished_fn, void *pp_cb,\n -\t\t\t const char *tr2_category, const char *tr2_label);\n -+int run_processes_parallel(int n, get_next_task_fn, start_failure_fn,\n -+\t\t\t task_finished_fn, void *pp_cb,\n -+\t\t\t struct run_process_parallel_opts *opts);\n ++int run_processes_parallel(struct run_process_parallel_opts *opts);\n \n /**\n * Convenience function which prepares env_array for a command to be run in a\n @@ submodule.c: int fetch_submodules(struct repository *r,\n +\tstruct run_process_parallel_opts run_opts = {\n +\t\t.tr2_category = \"submodule\",\n +\t\t.tr2_label = \"parallel/fetch\",\n ++\n ++\t\t.jobs = max_parallel_jobs,\n ++\n ++\t\t.get_next_task = get_next_submodule,\n ++\t\t.start_failure = fetch_start_failure,\n ++\t\t.task_finished = fetch_finish,\n ++\t\t.data = &spf,\n +\t};\n \n \tspf.r = r;\n @@ submodule.c: int fetch_submodules(struct repository *r,\n -\t\t\t\t fetch_finish,\n -\t\t\t\t &spf,\n -\t\t\t\t \"submodule\", \"parallel/fetch\");\n -+\trun_processes_parallel(max_parallel_jobs, get_next_submodule,\n -+\t\t\t fetch_start_failure, fetch_finish, &spf,\n -+\t\t\t &run_opts);\n ++\trun_processes_parallel(&run_opts);\n \n \tif (spf.submodules_with_errors.len > 0)\n \t\tfprintf(stderr, _(\"Errors during submodule fetch:\\n%s\"),\n \n ## t/helper/test-run-command.c ##\n @@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)\n + \t\t\t \"write JUnit-style XML files\"),\n + \t\tOPT_END()\n + \t};\n ++\tstruct run_process_parallel_opts run_opts = {\n ++\t\t.get_next_task = next_test,\n ++\t\t.start_failure = test_failed,\n ++\t\t.task_finished = test_finished,\n ++\t\t.data = &suite,\n ++\t};\n + \n + \targc = parse_options(argc, argv, NULL, options,\n + \t\t\ttestsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);\n +@@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)\n + \n + \tfprintf(stderr, \"Running %\"PRIuMAX\" tests (%d at a time)\\n\",\n \t\t(uintmax_t)suite.tests.nr, max_jobs);\n ++\trun_opts.jobs = max_jobs;\n \n - \tret = run_processes_parallel(max_jobs, next_test, test_failed,\n +-\tret = run_processes_parallel(max_jobs, next_test, test_failed,\n -\t\t\t\t test_finished, &suite);\n -+\t\t\t\t test_finished, &suite, NULL);\n ++\tret = run_processes_parallel(&run_opts);\n \n \tif (suite.failed.nr > 0) {\n \t\tret = 1;\n @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)\n - {\n - \tstruct child_process proc = CHILD_PROCESS_INIT;\n \tint jobs;\n + \tget_next_task_fn next_fn = NULL;\n + \ttask_finished_fn finished_fn = NULL;\n +\tstruct run_process_parallel_opts opts = { 0 };\n \n \tif (argc > 1 && !strcmp(argv[1], \"testsuite\"))\n \t\texit(testsuite(argc - 1, argv + 1));\n @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)\n + \tjobs = atoi(argv[2]);\n + \tstrvec_clear(&proc.args);\n + \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n ++\topts.jobs = jobs;\n ++\topts.data = &proc;\n \n - \tif (!strcmp(argv[1], \"run-command-parallel\"))\n - \t\texit(run_processes_parallel(jobs, parallel_next,\n --\t\t\t\t\t NULL, NULL, &proc));\n -+\t\t\t\t\t NULL, NULL, &proc, &opts));\n - \n - \tif (!strcmp(argv[1], \"run-command-abort\"))\n --\t\texit(run_processes_parallel(jobs, parallel_next,\n --\t\t\t\t\t NULL, task_finished, &proc));\n -+\t\texit(run_processes_parallel(jobs, parallel_next, NULL,\n -+\t\t\t\t\t task_finished, &proc, &opts));\n - \n - \tif (!strcmp(argv[1], \"run-command-no-jobs\"))\n --\t\texit(run_processes_parallel(jobs, no_job,\n --\t\t\t\t\t NULL, task_finished, &proc));\n -+\t\texit(run_processes_parallel(jobs, no_job, NULL, task_finished,\n -+\t\t\t\t\t &proc, &opts));\n + \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n + \t\tnext_fn = parallel_next;\n +@@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)\n + \t\treturn 1;\n + \t}\n \n - \tfprintf(stderr, \"check usage\\n\");\n - \treturn 1;\n +-\texit(run_processes_parallel(jobs, next_fn, NULL, finished_fn, &proc));\n ++\topts.get_next_task = next_fn;\n ++\topts.task_finished = finished_fn;\n ++\texit(run_processes_parallel(&opts));\n + }\n2: d9c9b158130 ! 3: a8e1fc07b65 run-command tests: test stdout of run_command_parallel()\n @@ t/t0061-run-command.sh: World\n \n test_expect_success 'run_command runs in parallel with more jobs available than tasks' '\n -\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n -+\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n +-\ttest_cmp expect actual\n ++\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_must_be_empty out &&\n - \ttest_cmp expect actual\n ++\ttest_cmp expect err\n '\n \n test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n -\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n -+\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n +-\ttest_cmp expect actual\n ++\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_must_be_empty out &&\n - \ttest_cmp expect actual\n ++\ttest_cmp expect err\n '\n \n test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n -\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n -+\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n +-\ttest_cmp expect actual\n ++\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_must_be_empty out &&\n - \ttest_cmp expect actual\n ++\ttest_cmp expect err\n '\n \n + cat >expect <<-EOF\n @@ t/t0061-run-command.sh: asking for a quick stop\n EOF\n \n test_expect_success 'run_command is asked to abort gracefully' '\n -\ttest-tool run-command run-command-abort 3 false 2>actual &&\n -+\ttest-tool run-command run-command-abort 3 false >out 2>actual &&\n +-\ttest_cmp expect actual\n ++\ttest-tool run-command run-command-abort 3 false >out 2>err &&\n +\ttest_must_be_empty out &&\n - \ttest_cmp expect actual\n ++\ttest_cmp expect err\n '\n \n + cat >expect <<-EOF\n @@ t/t0061-run-command.sh: no further jobs available\n EOF\n \n test_expect_success 'run_command outputs ' '\n -\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n -+\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n +-\ttest_cmp expect actual\n ++\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_must_be_empty out &&\n - \ttest_cmp expect actual\n ++\ttest_cmp expect err\n '\n \n + test_trace () {\n-: ----------- > 4: 663936fb4ad run-command.c: add an initializer for \"struct parallel_processes\"\n3: d76f63c2948 ! 5: c2e015ed840 run-command: add an \"ungroup\" option to run_process_parallel()\n @@ run-command.c: struct parallel_processes {\n \tstruct pollfd *pfd;\n \n \tunsigned shutdown : 1;\n -+\tunsigned ungroup:1;\n ++\tunsigned ungroup : 1;\n \n \tint output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n - \t\t get_next_task_fn get_next_task,\n - \t\t start_failure_fn start_failure,\n - \t\t task_finished_fn task_finished,\n --\t\t void *data)\n -+\t\t void *data, struct run_process_parallel_opts *opts)\n - {\n -+\tconst int ungroup = opts->ungroup;\n - \tint i;\n - \n - \tif (n < 1)\n -@@ run-command.c: static void pp_init(struct parallel_processes *pp,\n - \tpp->start_failure = start_failure ? start_failure : default_start_failure;\n - \tpp->task_finished = task_finished ? task_finished : default_task_finished;\n - \n -+\tpp->ungroup = ungroup;\n -+\n \tpp->nr_processes = 0;\n \tpp->output_owner = 0;\n \tpp->shutdown = 0;\n ++\tpp->ungroup = opts->ungroup;\n \tCALLOC_ARRAY(pp->children, n);\n -\tCALLOC_ARRAY(pp->pfd, n);\n -+\tif (!ungroup)\n ++\tif (!pp->ungroup)\n +\t\tCALLOC_ARRAY(pp->pfd, n);\n -+\n - \tstrbuf_init(&pp->buffered_output, 0);\n \n \tfor (i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n \t\tchild_process_init(&pp->children[i].process);\n -+\t\tif (ungroup)\n ++\t\tif (!pp->pfd)\n +\t\t\tcontinue;\n \t\tpp->pfd[i].events = POLLIN | POLLHUP;\n \t\tpp->pfd[i].fd = -1;\n \t}\n -@@ run-command.c: static void pp_init(struct parallel_processes *pp,\n - \n - static void pp_cleanup(struct parallel_processes *pp)\n - {\n -+\tconst int ungroup = pp->ungroup;\n - \tint i;\n - \n - \ttrace_printf(\"run_processes_parallel: done\");\n @@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)\n - \t}\n - \n - \tfree(pp->children);\n --\tfree(pp->pfd);\n -+\tif (!ungroup)\n -+\t\tfree(pp->pfd);\n - \n - \t/*\n \t * When get_next_task added messages to the buffer in its last\n \t * iteration, the buffered output is non empty.\n \t */\n -\tstrbuf_write(&pp->buffered_output, stderr);\n --\tstrbuf_release(&pp->buffered_output);\n -+\tif (!ungroup) {\n ++\tif (!pp->ungroup)\n +\t\tstrbuf_write(&pp->buffered_output, stderr);\n -+\t\tstrbuf_release(&pp->buffered_output);\n -+\t}\n + \tstrbuf_release(&pp->buffered_output);\n \n \tsigchain_pop_common();\n - }\n @@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)\n */\n static int pp_start_one(struct parallel_processes *pp)\n @@ run-command.c: static int pp_start_one(struct parallel_processes *pp)\n \t}\n -\tpp->children[i].process.err = -1;\n -\tpp->children[i].process.stdout_to_stderr = 1;\n --\tpp->children[i].process.no_stdin = 1;\n -+\n +\tif (!ungroup) {\n +\t\tpp->children[i].process.err = -1;\n +\t\tpp->children[i].process.stdout_to_stderr = 1;\n -+\t\tpp->children[i].process.no_stdin = 1;\n +\t}\n + \tpp->children[i].process.no_stdin = 1;\n \n \tif (start_command(&pp->children[i].process)) {\n -\t\tcode = pp->start_failure(&pp->children[i].err,\n @@ run-command.c: static int pp_start_one(struct parallel_processes *pp)\n \tpp->nr_processes++;\n \tpp->children[i].state = GIT_CP_WORKING;\n -\tpp->pfd[i].fd = pp->children[i].process.err;\n -+\tif (!ungroup)\n ++\tif (pp->pfd)\n +\t\tpp->pfd[i].fd = pp->children[i].process.err;\n \treturn 0;\n }\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n \t\tpp->nr_processes--;\n \t\tpp->children[i].state = GIT_CP_FREE;\n -\t\tpp->pfd[i].fd = -1;\n -+\t\tif (!ungroup)\n ++\t\tif (pp->pfd)\n +\t\t\tpp->pfd[i].fd = -1;\n \t\tchild_process_init(&pp->children[i].process);\n \n -\t\tif (i != pp->output_owner) {\n +\t\tif (ungroup) {\n -+\t\t\t/* no strbuf_*() work to do here */\n ++\t\t\t; /* no strbuf_*() work to do here */\n +\t\t} else if (i != pp->output_owner) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t} else {\n -@@ run-command.c: static int run_processes_parallel_1(int n, get_next_task_fn get_next_task,\n - \t\t\t\t void *pp_cb,\n - \t\t\t\t struct run_process_parallel_opts *opts)\n - {\n -+\tconst int ungroup = opts->ungroup;\n - \tint i, code;\n - \tint output_timeout = 100;\n - \tint spawn_cap = 4;\n - \tstruct parallel_processes pp;\n - \n --\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n -+\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n -+\t\topts);\n - \twhile (1) {\n - \t\tfor (i = 0;\n - \t\t i < spawn_cap && !pp.shutdown &&\n -@@ run-command.c: static int run_processes_parallel_1(int n, get_next_task_fn get_next_task,\n +@@ run-command.c: int run_processes_parallel(struct run_process_parallel_opts *opts)\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n -\t\tpp_buffer_stderr(&pp, output_timeout);\n -\t\tpp_output(&pp);\n -+\t\tif (ungroup) {\n ++\t\tif (opts->ungroup) {\n +\t\t\tpp_mark_working_for_cleanup(&pp);\n +\t\t} else {\n +\t\t\tpp_buffer_stderr(&pp, output_timeout);\n @@ run-command.h: typedef int (*start_failure_fn)(struct strbuf *out,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n @@ run-command.h: typedef int (*task_finished_fn)(int result,\n *\n - * tr2_category & tr2_label: sets the trace2 category and label for\n - * logging. These must either be unset, or both of them must be set.\n -+ *\n + * jobs: see 'n' in run_processes_parallel() below.\n + *\n + * ungroup: Ungroup output. Output is printed as soon as possible and\n + * bypasses run-command's internal processing. This may cause output\n + * from different commands to be mixed.\n ++ *\n + * *_fn & data: see run_processes_parallel() below.\n */\n struct run_process_parallel_opts\n - {\n - \tconst char *tr2_category;\n +@@ run-command.h: struct run_process_parallel_opts\n \tconst char *tr2_label;\n + \n + \tint jobs;\n +\tunsigned int ungroup:1;\n - };\n \n - /**\n + \tget_next_task_fn get_next_task;\n + \tstart_failure_fn start_failure;\n @@ run-command.h: struct run_process_parallel_opts\n *\n * The children started via this function run in parallel. Their output\n @@ run-command.h: struct run_process_parallel_opts\n *\n * start_failure_fn and task_finished_fn can be NULL to omit any\n * special handling.\n - *\n -- * Options are passed via a \"struct run_process_parallel_opts\".\n -+ * Options are passed via a \"struct run_process_parallel_opts\". If the\n -+ * \"ungroup\" option isn't specified the callbacks will get a pointer\n -+ * to a \"struct strbuf *out\", and must not write to stdout or stderr\n -+ * as such output will mess up the output of the other parallel\n ++ *\n ++ * If the \"ungroup\" option isn't specified the callbacks will get a\n ++ * pointer to a \"struct strbuf *out\", and must not write to stdout or\n ++ * stderr as such output will mess up the output of the other parallel\n + * processes. If \"ungroup\" option is specified callbacks will get a\n + * NULL \"struct strbuf *out\" parameter, and are responsible for\n + * emitting their own output, including dealing with any race\n + * conditions due to writing in parallel to stdout and stderr.\n */\n - int run_processes_parallel(int n, get_next_task_fn, start_failure_fn,\n - \t\t\t task_finished_fn, void *pp_cb,\n + int run_processes_parallel(struct run_process_parallel_opts *opts);\n + \n \n ## t/helper/test-run-command.c ##\n @@ t/helper/test-run-command.c: static int parallel_next(struct child_process *cp,\n @@ t/helper/test-run-command.c: static int task_finished(int result,\n }\n \n @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)\n - \tstrvec_clear(&proc.args);\n - \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n + \topts.jobs = jobs;\n + \topts.data = &proc;\n \n --\tif (!strcmp(argv[1], \"run-command-parallel\"))\n +-\tif (!strcmp(argv[1], \"run-command-parallel\")) {\n +\tif (!strcmp(argv[1], \"run-command-parallel\") ||\n +\t !strcmp(argv[1], \"run-command-parallel-ungroup\")) {\n -+\t\topts.ungroup = !strcmp(argv[1], \"run-command-parallel-ungroup\");\n - \t\texit(run_processes_parallel(jobs, parallel_next,\n - \t\t\t\t\t NULL, NULL, &proc, &opts));\n -+\t}\n - \n --\tif (!strcmp(argv[1], \"run-command-abort\"))\n -+\tif (!strcmp(argv[1], \"run-command-abort\") ||\n -+\t !strcmp(argv[1], \"run-command-abort-ungroup\")) {\n -+\t\topts.ungroup = !strcmp(argv[1], \"run-command-abort-ungroup\");\n - \t\texit(run_processes_parallel(jobs, parallel_next, NULL,\n - \t\t\t\t\t task_finished, &proc, &opts));\n -+\t}\n - \n --\tif (!strcmp(argv[1], \"run-command-no-jobs\"))\n -+\tif (!strcmp(argv[1], \"run-command-no-jobs\") ||\n -+\t !strcmp(argv[1], \"run-command-no-jobs-ungroup\")) {\n -+\t\topts.ungroup = !strcmp(argv[1], \"run-command-no-jobs-ungroup\");\n - \t\texit(run_processes_parallel(jobs, no_job, NULL, task_finished,\n - \t\t\t\t\t &proc, &opts));\n -+\t}\n + \t\tnext_fn = parallel_next;\n +-\t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n ++\t} else if (!strcmp(argv[1], \"run-command-abort\") ||\n ++\t\t !strcmp(argv[1], \"run-command-abort-ungroup\")) {\n + \t\tnext_fn = parallel_next;\n + \t\tfinished_fn = task_finished;\n +-\t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n ++\t} else if (!strcmp(argv[1], \"run-command-no-jobs\") ||\n ++\t\t !strcmp(argv[1], \"run-command-no-jobs-ungroup\")) {\n + \t\tnext_fn = no_job;\n + \t\tfinished_fn = task_finished;\n + \t} else {\n +@@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)\n + \t\treturn 1;\n + \t}\n \n - \tfprintf(stderr, \"check usage\\n\");\n - \treturn 1;\n ++\topts.ungroup = ends_with(argv[1], \"-ungroup\");\n + \topts.get_next_task = next_fn;\n + \topts.task_finished = finished_fn;\n + \texit(run_processes_parallel(&opts));\n \n ## t/t0061-run-command.sh ##\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with more jobs available than\n - \ttest_cmp expect actual\n + \ttest_cmp expect err\n '\n \n +test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with m\n +'\n +\n test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n - \ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n + \ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_must_be_empty out &&\n - \ttest_cmp expect actual\n + \ttest_cmp expect err\n '\n \n +test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with m\n +'\n +\n test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n - \ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n + \ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_must_be_empty out &&\n - \ttest_cmp expect actual\n + \ttest_cmp expect err\n '\n \n +test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with m\n preloaded output of a child\n asking for a quick stop\n @@ t/t0061-run-command.sh: test_expect_success 'run_command is asked to abort gracefully' '\n - \ttest_cmp expect actual\n + \ttest_cmp expect err\n '\n \n +test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n @@ t/t0061-run-command.sh: test_expect_success 'run_command is asked to abort grace\n no further jobs available\n EOF\n @@ t/t0061-run-command.sh: test_expect_success 'run_command outputs ' '\n - \ttest_cmp expect actual\n + \ttest_cmp expect err\n '\n \n +test_expect_success 'run_command outputs (ungroup) ' '\n -+\ttest-tool run-command run-command-no-jobs-ungroup 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>actual &&\n ++\ttest-tool run-command run-command-no-jobs-ungroup 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_must_be_empty out &&\n -+\ttest_cmp expect actual\n ++\ttest_cmp expect err\n +'\n +\n test_trace () {\n4: cf62569b2e0 = 6: 84e92c6f7c7 hook tests: fix redirection logic error in 96e7225b310\n5: 98c26c9917b ! 7: bf7d871565f hook API: don't redundantly re-set \"no_stdin\" and \"stdout_to_stderr\"\n @@ Commit message\n hook API: don't redundantly re-set \"no_stdin\" and \"stdout_to_stderr\"\n \n Amend code added in 96e7225b310 (hook: add 'run' subcommand,\n - 2021-12-22) top stop setting these two flags. We use the\n + 2021-12-22) to stop setting these two flags. We use the\n run_process_parallel() API added in c553c72eed6 (run-command: add an\n asynchronous parallel child processor, 2015-12-15), which always sets\n these in pp_start_one() (in addition to setting .err = -1).\n6: de3664f6d2b ! 8: 238155fcb9d hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n @@ Commit message\n './git hook run seq-hook' in 'HEAD~0' ran\n 1.30 ± 0.02 times faster than './git hook run seq-hook' in 'origin/master'\n \n - In the preceding commit we removed the \"no_stdin=1\" and\n - \"stdout_to_stderr=1\" assignments. This change brings them back as with\n - \".ungroup=1\" the run_process_parallel() function doesn't provide them\n - for us implicitly.\n + In the preceding commit we removed the \"stdout_to_stderr=1\" assignment\n + as being redundant. This change brings it back as with \".ungroup=1\"\n + the run_process_parallel() function doesn't provide them for us\n + implicitly.\n \n As an aside omitting the stdout_to_stderr=1 here would have all tests\n pass, except those that test \"git hook run\" itself in\n @@ Commit message\n \n ## hook.c ##\n @@ hook.c: static int pick_next_hook(struct child_process *cp,\n - \tif (!hook_path)\n \t\treturn 0;\n \n -+\tcp->no_stdin = 1;\n \tstrvec_pushv(&cp->env_array, hook_cb->options->env.v);\n -+\tcp->stdout_to_stderr = 1;\n ++\tcp->stdout_to_stderr = 1; /* because of .ungroup = 1 */\n \tcp->trace2_hook_name = hook_cb->hook_name;\n \tcp->dir = hook_cb->options->dir;\n \n @@ hook.c: int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n - \t\t.options = options,\n - \t};\n - \tconst char *const hook_path = find_hook(hook_name);\n --\tint jobs = 1;\n -+\tconst int jobs = 1;\n - \tint ret = 0;\n - \tstruct run_process_parallel_opts run_opts = {\n - \t\t.tr2_category = \"hook\",\n \t\t.tr2_label = hook_name,\n + \n + \t\t.jobs = jobs,\n +\t\t.ungroup = jobs == 1,\n - \t};\n \n + \t\t.get_next_task = pick_next_hook,\n + \t\t.start_failure = notify_start_failure,\n +@@ hook.c: int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n \n-- \n2.36.1.952.g0ae626f6cd7\n\n"},{"id":"455419","messageId":"patch-v2-4.8-663936fb4ad-20220518T195858Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com","subject":"[PATCH v2 4/8] run-command.c: add an initializer for \"struct parallel_processes\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-18T20:05:20Z","receivedAt":"2022-05-18T20:06:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a PARALLEL_PROCESSES_INIT macro for \"struct parallel_processes\",\nthis allows us to do away with a call to strbuf_init(), in subsequent\ncommits we'll be able to rely on other fields being NULL'd.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 8c156fd080e..839c85d12e5 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1498,6 +1498,9 @@ struct parallel_processes {\n \tint output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n };\n+#define PARALLEL_PROCESSES_INIT { \\\n+\t.buffered_output = STRBUF_INIT, \\\n+}\n \n static int default_start_failure(struct strbuf *out,\n \t\t\t\t void *pp_cb,\n@@ -1562,7 +1565,6 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->shutdown = 0;\n \tCALLOC_ARRAY(pp->children, n);\n \tCALLOC_ARRAY(pp->pfd, n);\n-\tstrbuf_init(&pp->buffered_output, 0);\n \n \tfor (i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n@@ -1744,7 +1746,7 @@ int run_processes_parallel(struct run_process_parallel_opts *opts)\n \tint i, code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n-\tstruct parallel_processes pp;\n+\tstruct parallel_processes pp = PARALLEL_PROCESSES_INIT;\n \tconst char *tr2_category = opts->tr2_category;\n \tconst char *tr2_label = opts->tr2_label;\n \tconst int do_trace2 = tr2_category && tr2_label;\n-- \n2.36.1.952.g0ae626f6cd7\n\n"},{"id":"455420","messageId":"patch-v2-7.8-bf7d871565f-20220518T195858Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com","subject":"[PATCH v2 7/8] hook API: don't redundantly re-set \"no_stdin\" and \"stdout_to_stderr\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-18T20:05:23Z","receivedAt":"2022-05-18T20:06:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Amend code added in 96e7225b310 (hook: add 'run' subcommand,\n2021-12-22) to stop setting these two flags. We use the\nrun_process_parallel() API added in c553c72eed6 (run-command: add an\nasynchronous parallel child processor, 2015-12-15), which always sets\nthese in pp_start_one() (in addition to setting .err = -1).\n\nNote that an assert() to check that these values are already what\nwe're setting them to here would fail. That's because in\npp_start_one() we'll set these after calling this \"get_next_task\"\ncallback (which we call pick_next_hook()). But the only case where we\nweren't setting these just after returning from this function was if\nwe took the \"return 0\" path here, in which case we wouldn't have set\nthese.\n\nSo while this code wasn't wrong, it was entirely redundant. The\nrun_process_parallel() also can't work with a generic \"struct\nchild_process\", it needs one that's behaving in a way that it expects\nwhen it comes to stderr/stdout. So we shouldn't be changing these\nvalues, or in this case keeping around code that gives the impression\nthat doing in the general case is OK.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/hook.c b/hook.c\nindex 9aefccfc34a..dc498ef5c39 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -53,9 +53,7 @@ static int pick_next_hook(struct child_process *cp,\n \tif (!hook_path)\n \t\treturn 0;\n \n-\tcp->no_stdin = 1;\n \tstrvec_pushv(&cp->env_array, hook_cb->options->env.v);\n-\tcp->stdout_to_stderr = 1;\n \tcp->trace2_hook_name = hook_cb->hook_name;\n \tcp->dir = hook_cb->options->dir;\n \n-- \n2.36.1.952.g0ae626f6cd7\n\n"},{"id":"455421","messageId":"patch-v2-6.8-84e92c6f7c7-20220518T195858Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com","subject":"[PATCH v2 6/8] hook tests: fix redirection logic error in 96e7225b310","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-18T20:05:22Z","receivedAt":"2022-05-18T20:06:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The tests added in 96e7225b310 (hook: add 'run' subcommand,\n2021-12-22) were redirecting to \"actual\" both in the body of the hook\nitself and in the testing code below.\n\nThe net result was that the \"2>>actual\" redirection later in the test\nwasn't doing anything. Let's have those redirection do what it looks\nlike they're doing.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1800-hook.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\nindex 26ed5e11bc8..1e4adc3d53e 100755\n--- a/t/t1800-hook.sh\n+++ b/t/t1800-hook.sh\n@@ -94,7 +94,7 @@ test_expect_success 'git hook run -- out-of-repo runs excluded' '\n test_expect_success 'git -c core.hooksPath= hook run' '\n \tmkdir my-hooks &&\n \twrite_script my-hooks/test-hook <<-\\EOF &&\n-\techo Hook ran $1 >>actual\n+\techo Hook ran $1\n \tEOF\n \n \tcat >expect <<-\\EOF &&\n-- \n2.36.1.952.g0ae626f6cd7\n\n"},{"id":"455422","messageId":"patch-v2-8.8-238155fcb9d-20220518T195858Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com","subject":"[PATCH v2 8/8] hook API: fix v2.36.0 regression: hooks should be connected to a TTY","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-18T20:05:24Z","receivedAt":"2022-05-18T20:06:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a regression reported[1] in f443246b9f2 (commit: convert\n{pre-commit,prepare-commit-msg} hook to hook.h, 2021-12-22): Due to\nusing the run_process_parallel() API in the earlier 96e7225b310 (hook:\nadd 'run' subcommand, 2021-12-22) we'd capture the hook's stderr and\nstdout, and thus lose the connection to the TTY in the case of\ne.g. the \"pre-commit\" hook.\n\nAs a preceding commit notes GNU parallel's similar --ungroup option\nalso has it emit output faster. While we're unlikely to have hooks\nthat emit truly massive amounts of output (or where the performance\nthereof matters) it's still informative to measure the overhead. In a\nsimilar \"seq\" test we're now ~30% faster:\n\n\t$ cat .git/hooks/seq-hook; git hyperfine -L rev origin/master,HEAD~0 -s 'make CFLAGS=-O3' './git hook run seq-hook'\n\t#!/bin/sh\n\n\tseq 100000000\n\tBenchmark 1: ./git hook run seq-hook' in 'origin/master\n\t Time (mean ± σ): 787.1 ms ± 13.6 ms [User: 701.6 ms, System: 534.4 ms]\n\t Range (min … max): 773.2 ms … 806.3 ms 10 runs\n\n\tBenchmark 2: ./git hook run seq-hook' in 'HEAD~0\n\t Time (mean ± σ): 603.4 ms ± 1.6 ms [User: 573.1 ms, System: 30.3 ms]\n\t Range (min … max): 601.0 ms … 606.2 ms 10 runs\n\n\tSummary\n\t './git hook run seq-hook' in 'HEAD~0' ran\n\t 1.30 ± 0.02 times faster than './git hook run seq-hook' in 'origin/master'\n\nIn the preceding commit we removed the \"stdout_to_stderr=1\" assignment\nas being redundant. This change brings it back as with \".ungroup=1\"\nthe run_process_parallel() function doesn't provide them for us\nimplicitly.\n\nAs an aside omitting the stdout_to_stderr=1 here would have all tests\npass, except those that test \"git hook run\" itself in\nt1800-hook.sh. But our tests passing is the result of another test\nblind spot, as was the case with the regression being fixed here. The\n\"stdout_to_stderr=1\" for hooks is long-standing behavior, see\ne.g. 1d9e8b56fe3 (Split back out update_hook handling in receive-pack,\n2007-03-10) and other follow-up commits (running \"git log\" with\n\"--reverse -p -Gstdout_to_stderr\" is a good start).\n\n1. https://lore.kernel.org/git/CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com/\n\nReported-by: Anthony Sottile \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 5 +++++\n t/t1800-hook.sh | 37 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 42 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex dc498ef5c39..5f31b60384a 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -54,6 +54,7 @@ static int pick_next_hook(struct child_process *cp,\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->env_array, hook_cb->options->env.v);\n+\tcp->stdout_to_stderr = 1; /* because of .ungroup = 1 */\n \tcp->trace2_hook_name = hook_cb->hook_name;\n \tcp->dir = hook_cb->options->dir;\n \n@@ -126,6 +127,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \t\t.tr2_label = hook_name,\n \n \t\t.jobs = jobs,\n+\t\t.ungroup = jobs == 1,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n@@ -136,6 +138,9 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n \n+\tif (jobs != 1 || !run_opts.ungroup)\n+\t\tBUG(\"TODO: think about & document order & interleaving of parallel hook output\");\n+\n \tif (options->invoked_hook)\n \t\t*options->invoked_hook = 0;\n \ndiff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\nindex 1e4adc3d53e..f22754deccc 100755\n--- a/t/t1800-hook.sh\n+++ b/t/t1800-hook.sh\n@@ -4,6 +4,7 @@ test_description='git-hook command'\n \n TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-terminal.sh\n \n test_expect_success 'git hook usage' '\n \ttest_expect_code 129 git hook &&\n@@ -120,4 +121,40 @@ test_expect_success 'git -c core.hooksPath= hook run' '\n \ttest_cmp expect actual\n '\n \n+test_hook_tty() {\n+\tlocal fd=\"$1\" &&\n+\n+\tcat >expect &&\n+\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\n+\ttest_hook -C repo pre-commit <<-EOF &&\n+\t{\n+\t\ttest -t 1 && echo >&$fd STDOUT TTY || echo >&$fd STDOUT NO TTY &&\n+\t\ttest -t 2 && echo >&$fd STDERR TTY || echo >&$fd STDERR NO TTY\n+\t} $fd>actual\n+\tEOF\n+\n+\ttest_commit -C repo A &&\n+\ttest_commit -C repo B &&\n+\tgit -C repo reset --soft HEAD^ &&\n+\ttest_terminal git -C repo commit -m\"B.new\" &&\n+\ttest_cmp expect repo/actual\n+}\n+\n+test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDOUT redirect' '\n+\ttest_hook_tty 1 <<-\\EOF\n+\tSTDOUT NO TTY\n+\tSTDERR TTY\n+\tEOF\n+'\n+\n+test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDERR redirect' '\n+\ttest_hook_tty 2 <<-\\EOF\n+\tSTDOUT TTY\n+\tSTDERR NO TTY\n+\tEOF\n+'\n+\n test_done\n-- \n2.36.1.952.g0ae626f6cd7\n\n"},{"id":"455423","messageId":"patch-v2-5.8-c2e015ed840-20220518T195858Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com","subject":"[PATCH v2 5/8] run-command: add an \"ungroup\" option to run_process_parallel()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-18T20:05:21Z","receivedAt":"2022-05-18T20:06:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Extend the parallel execution API added in c553c72eed6 (run-command:\nadd an asynchronous parallel child processor, 2015-12-15) to support a\nmode where the stdout and stderr of the processes isn't captured and\noutput in a deterministic order, instead we'll leave it to the kernel\nand stdio to sort it out.\n\nThis gives the API same functionality as GNU parallel's --ungroup\noption. As we'll see in a subsequent commit the main reason to want\nthis is to support stdout and stderr being connected to the TTY in the\ncase of jobs=1, demonstrated here with GNU parallel:\n\n\t$ parallel --ungroup 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n\tTTY\n\tTTY\n\t$ parallel 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n\tNTTY\n\tNTTY\n\nAnother is as GNU parallel's documentation notes a potential for\noptimization. Our results will be a bit different, but in cases where\nyou want to run processes in parallel where the exact order isn't\nimportant this can be a lot faster:\n\n\t$ hyperfine -r 3 -L o ,--ungroup 'parallel {o} seq ::: 10000000 >/dev/null '\n\tBenchmark 1: parallel seq ::: 10000000 >/dev/null\n\t Time (mean ± σ): 220.2 ms ± 9.3 ms [User: 124.9 ms, System: 96.1 ms]\n\t Range (min … max): 212.3 ms … 230.5 ms 3 runs\n\n\tBenchmark 2: parallel --ungroup seq ::: 10000000 >/dev/null\n\t Time (mean ± σ): 154.7 ms ± 0.9 ms [User: 136.2 ms, System: 25.1 ms]\n\t Range (min … max): 153.9 ms … 155.7 ms 3 runs\n\n\tSummary\n\t 'parallel --ungroup seq ::: 10000000 >/dev/null ' ran\n\t 1.42 ± 0.06 times faster than 'parallel seq ::: 10000000 >/dev/null '\n\nA large part of the juggling in the API is to make the API safer for\nits maintenance and consumers alike.\n\nFor the maintenance of the API we e.g. avoid malloc()-ing the\n\"pp->pfd\", ensuring that SANITIZE=address and other similar tools will\ncatch any unexpected misuse.\n\nFor API consumers we take pains to never pass the non-NULL \"out\"\nbuffer to an API user that provided the \"ungroup\" option. The\nresulting code in t/helper/test-run-command.c isn't typical of such a\nuser, i.e. they'd typically use one mode or the other, and would know\nwhether they'd provided \"ungroup\" or not.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 76 ++++++++++++++++++++++++++++---------\n run-command.h | 31 +++++++++++----\n t/helper/test-run-command.c | 26 ++++++++++---\n t/t0061-run-command.sh | 30 +++++++++++++++\n 4 files changed, 132 insertions(+), 31 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 839c85d12e5..39e09ee39fc 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1468,7 +1468,7 @@ int pipe_command(struct child_process *cmd,\n enum child_state {\n \tGIT_CP_FREE,\n \tGIT_CP_WORKING,\n-\tGIT_CP_WAIT_CLEANUP,\n+\tGIT_CP_WAIT_CLEANUP, /* only for !ungroup */\n };\n \n struct parallel_processes {\n@@ -1494,6 +1494,7 @@ struct parallel_processes {\n \tstruct pollfd *pfd;\n \n \tunsigned shutdown : 1;\n+\tunsigned ungroup : 1;\n \n \tint output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n@@ -1563,12 +1564,16 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->nr_processes = 0;\n \tpp->output_owner = 0;\n \tpp->shutdown = 0;\n+\tpp->ungroup = opts->ungroup;\n \tCALLOC_ARRAY(pp->children, n);\n-\tCALLOC_ARRAY(pp->pfd, n);\n+\tif (!pp->ungroup)\n+\t\tCALLOC_ARRAY(pp->pfd, n);\n \n \tfor (i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n \t\tchild_process_init(&pp->children[i].process);\n+\t\tif (!pp->pfd)\n+\t\t\tcontinue;\n \t\tpp->pfd[i].events = POLLIN | POLLHUP;\n \t\tpp->pfd[i].fd = -1;\n \t}\n@@ -1594,7 +1599,8 @@ static void pp_cleanup(struct parallel_processes *pp)\n \t * When get_next_task added messages to the buffer in its last\n \t * iteration, the buffered output is non empty.\n \t */\n-\tstrbuf_write(&pp->buffered_output, stderr);\n+\tif (!pp->ungroup)\n+\t\tstrbuf_write(&pp->buffered_output, stderr);\n \tstrbuf_release(&pp->buffered_output);\n \n \tsigchain_pop_common();\n@@ -1609,6 +1615,7 @@ static void pp_cleanup(struct parallel_processes *pp)\n */\n static int pp_start_one(struct parallel_processes *pp)\n {\n+\tconst int ungroup = pp->ungroup;\n \tint i, code;\n \n \tfor (i = 0; i < pp->max_processes; i++)\n@@ -1618,24 +1625,30 @@ static int pp_start_one(struct parallel_processes *pp)\n \t\tBUG(\"bookkeeping is hard\");\n \n \tcode = pp->get_next_task(&pp->children[i].process,\n-\t\t\t\t &pp->children[i].err,\n+\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n \t\t\t\t pp->data,\n \t\t\t\t &pp->children[i].data);\n \tif (!code) {\n-\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n-\t\tstrbuf_reset(&pp->children[i].err);\n+\t\tif (!ungroup) {\n+\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n+\t\t\tstrbuf_reset(&pp->children[i].err);\n+\t\t}\n \t\treturn 1;\n \t}\n-\tpp->children[i].process.err = -1;\n-\tpp->children[i].process.stdout_to_stderr = 1;\n+\tif (!ungroup) {\n+\t\tpp->children[i].process.err = -1;\n+\t\tpp->children[i].process.stdout_to_stderr = 1;\n+\t}\n \tpp->children[i].process.no_stdin = 1;\n \n \tif (start_command(&pp->children[i].process)) {\n-\t\tcode = pp->start_failure(&pp->children[i].err,\n+\t\tcode = pp->start_failure(ungroup ? NULL : &pp->children[i].err,\n \t\t\t\t\t pp->data,\n \t\t\t\t\t pp->children[i].data);\n-\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n-\t\tstrbuf_reset(&pp->children[i].err);\n+\t\tif (!ungroup) {\n+\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n+\t\t\tstrbuf_reset(&pp->children[i].err);\n+\t\t}\n \t\tif (code)\n \t\t\tpp->shutdown = 1;\n \t\treturn code;\n@@ -1643,14 +1656,26 @@ static int pp_start_one(struct parallel_processes *pp)\n \n \tpp->nr_processes++;\n \tpp->children[i].state = GIT_CP_WORKING;\n-\tpp->pfd[i].fd = pp->children[i].process.err;\n+\tif (pp->pfd)\n+\t\tpp->pfd[i].fd = pp->children[i].process.err;\n \treturn 0;\n }\n \n+static void pp_mark_working_for_cleanup(struct parallel_processes *pp)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < pp->max_processes; i++)\n+\t\tif (pp->children[i].state == GIT_CP_WORKING)\n+\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+}\n+\n static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n {\n \tint i;\n \n+\tassert(!pp->ungroup);\n+\n \twhile ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n \t\tif (errno == EINTR)\n \t\t\tcontinue;\n@@ -1677,6 +1702,9 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n static void pp_output(struct parallel_processes *pp)\n {\n \tint i = pp->output_owner;\n+\n+\tassert(!pp->ungroup);\n+\n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n \t\tstrbuf_write(&pp->children[i].err, stderr);\n@@ -1686,10 +1714,15 @@ static void pp_output(struct parallel_processes *pp)\n \n static int pp_collect_finished(struct parallel_processes *pp)\n {\n+\tconst int ungroup = pp->ungroup;\n \tint i, code;\n \tint n = pp->max_processes;\n \tint result = 0;\n \n+\tif (ungroup)\n+\t\tfor (i = 0; i < pp->max_processes; i++)\n+\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+\n \twhile (pp->nr_processes > 0) {\n \t\tfor (i = 0; i < pp->max_processes; i++)\n \t\t\tif (pp->children[i].state == GIT_CP_WAIT_CLEANUP)\n@@ -1700,8 +1733,8 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \t\tcode = finish_command(&pp->children[i].process);\n \n \t\tcode = pp->task_finished(code,\n-\t\t\t\t\t &pp->children[i].err, pp->data,\n-\t\t\t\t\t pp->children[i].data);\n+\t\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n+\t\t\t\t\t pp->data, pp->children[i].data);\n \n \t\tif (code)\n \t\t\tresult = code;\n@@ -1710,10 +1743,13 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \n \t\tpp->nr_processes--;\n \t\tpp->children[i].state = GIT_CP_FREE;\n-\t\tpp->pfd[i].fd = -1;\n+\t\tif (pp->pfd)\n+\t\t\tpp->pfd[i].fd = -1;\n \t\tchild_process_init(&pp->children[i].process);\n \n-\t\tif (i != pp->output_owner) {\n+\t\tif (ungroup) {\n+\t\t\t; /* no strbuf_*() work to do here */\n+\t\t} else if (i != pp->output_owner) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t} else {\n@@ -1774,8 +1810,12 @@ int run_processes_parallel(struct run_process_parallel_opts *opts)\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tpp_buffer_stderr(&pp, output_timeout);\n-\t\tpp_output(&pp);\n+\t\tif (opts->ungroup) {\n+\t\t\tpp_mark_working_for_cleanup(&pp);\n+\t\t} else {\n+\t\t\tpp_buffer_stderr(&pp, output_timeout);\n+\t\t\tpp_output(&pp);\n+\t\t}\n \t\tcode = pp_collect_finished(&pp);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\ndiff --git a/run-command.h b/run-command.h\nindex b0268ed3db1..dcb6ded4b55 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -405,6 +405,10 @@ void check_pipe(int err);\n * pp_cb is the callback cookie as passed to run_processes_parallel.\n * You can store a child process specific callback cookie in pp_task_cb.\n *\n+ * The \"struct strbuf *err\" parameter is either a pointer to a string\n+ * to write errors to, or NULL if the \"ungroup\" option was\n+ * provided. See run_processes_parallel() below.\n+ *\n * Even after returning 0 to indicate that there are no more processes,\n * this function will be called again until there are no more running\n * child processes.\n@@ -423,9 +427,9 @@ typedef int (*get_next_task_fn)(struct child_process *cp,\n * This callback is called whenever there are problems starting\n * a new process.\n *\n- * You must not write to stdout or stderr in this function. Add your\n- * message to the strbuf out instead, which will be printed without\n- * messing up the output of the other parallel processes.\n+ * The \"struct strbuf *err\" parameter is either a pointer to a string\n+ * to write errors to, or NULL if the \"ungroup\" option was\n+ * provided. See run_processes_parallel() below.\n *\n * pp_cb is the callback cookie as passed into run_processes_parallel,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n@@ -441,9 +445,9 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n /**\n * This callback is called on every child process that finished processing.\n *\n- * You must not write to stdout or stderr in this function. Add your\n- * message to the strbuf out instead, which will be printed without\n- * messing up the output of the other parallel processes.\n+ * The \"struct strbuf *err\" parameter is either a pointer to a string\n+ * to write errors to, or NULL if the \"ungroup\" option was\n+ * provided. See run_processes_parallel() below.\n *\n * pp_cb is the callback cookie as passed into run_processes_parallel,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n@@ -466,6 +470,10 @@ typedef int (*task_finished_fn)(int result,\n *\n * jobs: see 'n' in run_processes_parallel() below.\n *\n+ * ungroup: Ungroup output. Output is printed as soon as possible and\n+ * bypasses run-command's internal processing. This may cause output\n+ * from different commands to be mixed.\n+ *\n * *_fn & data: see run_processes_parallel() below.\n */\n struct run_process_parallel_opts\n@@ -474,6 +482,7 @@ struct run_process_parallel_opts\n \tconst char *tr2_label;\n \n \tint jobs;\n+\tunsigned int ungroup:1;\n \n \tget_next_task_fn get_next_task;\n \tstart_failure_fn start_failure;\n@@ -490,10 +499,18 @@ struct run_process_parallel_opts\n *\n * The children started via this function run in parallel. Their output\n * (both stdout and stderr) is routed to stderr in a manner that output\n- * from different tasks does not interleave.\n+ * from different tasks does not interleave (but see \"ungroup\" above).\n *\n * start_failure_fn and task_finished_fn can be NULL to omit any\n * special handling.\n+ *\n+ * If the \"ungroup\" option isn't specified the callbacks will get a\n+ * pointer to a \"struct strbuf *out\", and must not write to stdout or\n+ * stderr as such output will mess up the output of the other parallel\n+ * processes. If \"ungroup\" option is specified callbacks will get a\n+ * NULL \"struct strbuf *out\" parameter, and are responsible for\n+ * emitting their own output, including dealing with any race\n+ * conditions due to writing in parallel to stdout and stderr.\n */\n int run_processes_parallel(struct run_process_parallel_opts *opts);\n \ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 56a806f228b..986acbce5f2 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -31,7 +31,11 @@ static int parallel_next(struct child_process *cp,\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->args.v);\n-\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n+\telse\n+\t\tfprintf(stderr, \"preloaded output of a child\\n\");\n+\n \tnumber_callbacks++;\n \treturn 1;\n }\n@@ -41,7 +45,10 @@ static int no_job(struct child_process *cp,\n \t\t void *cb,\n \t\t void **task_cb)\n {\n-\tstrbuf_addstr(err, \"no further jobs available\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"no further jobs available\\n\");\n+\telse\n+\t\tfprintf(stderr, \"no further jobs available\\n\");\n \treturn 0;\n }\n \n@@ -50,7 +57,10 @@ static int task_finished(int result,\n \t\t\t void *pp_cb,\n \t\t\t void *pp_task_cb)\n {\n-\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n+\telse\n+\t\tfprintf(stderr, \"asking for a quick stop\\n\");\n \treturn 1;\n }\n \n@@ -422,12 +432,15 @@ int cmd__run_command(int argc, const char **argv)\n \topts.jobs = jobs;\n \topts.data = &proc;\n \n-\tif (!strcmp(argv[1], \"run-command-parallel\")) {\n+\tif (!strcmp(argv[1], \"run-command-parallel\") ||\n+\t !strcmp(argv[1], \"run-command-parallel-ungroup\")) {\n \t\tnext_fn = parallel_next;\n-\t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n+\t} else if (!strcmp(argv[1], \"run-command-abort\") ||\n+\t\t !strcmp(argv[1], \"run-command-abort-ungroup\")) {\n \t\tnext_fn = parallel_next;\n \t\tfinished_fn = task_finished;\n-\t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n+\t} else if (!strcmp(argv[1], \"run-command-no-jobs\") ||\n+\t\t !strcmp(argv[1], \"run-command-no-jobs-ungroup\")) {\n \t\tnext_fn = no_job;\n \t\tfinished_fn = task_finished;\n \t} else {\n@@ -435,6 +448,7 @@ int cmd__run_command(int argc, const char **argv)\n \t\treturn 1;\n \t}\n \n+\topts.ungroup = ends_with(argv[1], \"-ungroup\");\n \topts.get_next_task = next_fn;\n \topts.task_finished = finished_fn;\n \texit(run_processes_parallel(&opts));\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 7d00f3cc2af..3628719a06d 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -135,18 +135,36 @@ test_expect_success 'run_command runs in parallel with more jobs available than\n \ttest_cmp expect err\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n+\ttest-tool run-command run-command-parallel-ungroup 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n \ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_must_be_empty out &&\n \ttest_cmp expect err\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n+\ttest-tool run-command run-command-parallel-ungroup 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n \ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_must_be_empty out &&\n \ttest_cmp expect err\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n+\ttest-tool run-command run-command-parallel-ungroup 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n@@ -162,6 +180,12 @@ test_expect_success 'run_command is asked to abort gracefully' '\n \ttest_cmp expect err\n '\n \n+test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n+\ttest-tool run-command run-command-abort-ungroup 3 false >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_line_count = 6 err\n+'\n+\n cat >expect <<-EOF\n no further jobs available\n EOF\n@@ -172,6 +196,12 @@ test_expect_success 'run_command outputs ' '\n \ttest_cmp expect err\n '\n \n+test_expect_success 'run_command outputs (ungroup) ' '\n+\ttest-tool run-command run-command-no-jobs-ungroup 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n test_trace () {\n \texpect=\"$1\"\n \tshift\n-- \n2.36.1.952.g0ae626f6cd7\n\n"},{"id":"455428","messageId":"xmqqee0qtszn.fsf@gitster.g","threadId":"57764","inReplyTo":"patch-v2-2.8-5f0a6e9925f-20220518T195858Z-avarab@gmail.com","subject":"Re: [PATCH v2 2/8] run-command API: use \"opts\" struct for run_processes_parallel{,_tr2}()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-18T21:45:32Z","receivedAt":"2022-05-18T21:56:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> \tif (max_children != 1 && list->nr != 1) {\n> \t\tstruct parallel_fetch_state state = { argv.v, list, 0, 0 };\n> +\t\tstruct run_process_parallel_opts run_opts = {\n> +\t\t\t.tr2_category = \"fetch\",\n> +\t\t\t.tr2_label = \"parallel/fetch\",\n> +\n> +\t\t\t.jobs = max_children,\n> +\n> +\t\t\t.get_next_task = &fetch_next_remote,\n> +\t\t\t.start_failure = &fetch_failed_to_start,\n> +\t\t\t.task_finished = &fetch_finished,\n> +\t\t\t.data = &state,\n> +\t\t};\n> \n> \t\tstrvec_push(&argv, \"--end-of-options\");\n> +\t\tresult = run_processes_parallel(&run_opts);\n\n;-)\n\nCan't tell if this is going overboard, but it probably is better\nthan piling more parameter on top of existing ones.\n\n> diff --git a/run-command.h b/run-command.h\n> index 5bd0c933e80..b0268ed3db1 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -458,6 +458,32 @@ typedef int (*task_finished_fn)(int result,\n> \t\t\t\tvoid *pp_task_cb);\n> \n> /**\n> + * Options to pass to run_processes_parallel(), { 0 }-initialized\n> + * means no options. Fields:\n> + *\n> + * tr2_category & tr2_label: sets the trace2 category and label for\n> + * logging. These must either be unset, or both of them must be set.\n\nI would have written \"unset\" -> \"set to NULL\" if I were writing\nthis, but it should hopefully be obvious from the context, so it is\nOK.\n\n> + * jobs: see 'n' in run_processes_parallel() below.\n> + *\n> + * *_fn & data: see run_processes_parallel() below.\n> + */\n\nOK.\n\n> +/**\n> + * Options are passed via the \"struct run_process_parallel_opts\" above.\n> +\n> * Runs up to n processes at the same time. Whenever a process can be\n> * started, the callback get_next_task_fn is called to obtain the data\n> * required to start another child process.\n\nBeyond the post context follows this text.\n\n * The children started via this function run in parallel. Their output\n * (both stdout and stderr) is routed to stderr in a manner that output\n * from different tasks does not interleave.\n *\n * start_failure_fn and task_finished_fn can be NULL to omit any\n * special handling.\n */\n int run_processes_parallel(struct run_process_parallel_opts *opts);\n\nThe forward reference of 'n' we saw earlier does have matching 'n'\nhere, but the 'n' no longer exists, so it probably is a good idea to\nrewrite the comment before this function.\n\n Runs up to opts->jobs processes at the time. Whenever a process\n can be started, the callback opts->get_next_task_fn is called to\n obtain the data required to start another child process. ...\n\nThe forward reference of 'data' we saw earlier does not have any\nmatching description here (it is a flaw in the original and not the\nproblem with this patch). The description of get_next_task_fn that\nappears much earlier in this file talks about two \"callback\ncookies\", pp_cb and pp_task_cb, but it is unclear how opts->data\n(after this patch) relates to either of these two. Presumably the\ncalling convention around the \"callback cookie\" is the same across\nget_next_task_fn, start_failure_fn, and task_finished_fn? If so,\nperhaps this is a good place to describe how opts->data is fed into\nthem.\n\n> +int run_processes_parallel(struct run_process_parallel_opts *opts);\n\n"},{"id":"455429","messageId":"xmqqa6betsqe.fsf@gitster.g","threadId":"57764","inReplyTo":"patch-v2-5.8-c2e015ed840-20220518T195858Z-avarab@gmail.com","subject":"Re: [PATCH v2 5/8] run-command: add an \"ungroup\" option to run_process_parallel()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-18T21:51:05Z","receivedAt":"2022-05-18T22:02:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> + * If the \"ungroup\" option isn't specified the callbacks will get a\n> + * pointer to a \"struct strbuf *out\", and must not write to stdout or\n> + * stderr as such output will mess up the output of the other parallel\n> + * processes. If \"ungroup\" option is specified callbacks will get a\n> + * NULL \"struct strbuf *out\" parameter, and are responsible for\n> + * emitting their own output, including dealing with any race\n> + * conditions due to writing in parallel to stdout and stderr.\n> */\n> int run_processes_parallel(struct run_process_parallel_opts *opts);\n> \n> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\n> index 56a806f228b..986acbce5f2 100644\n> --- a/t/helper/test-run-command.c\n> +++ b/t/helper/test-run-command.c\n> @@ -31,7 +31,11 @@ static int parallel_next(struct child_process *cp,\n> \t\treturn 0;\n> \n> \tstrvec_pushv(&cp->args, d->args.v);\n> -\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n> +\tif (err)\n> +\t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n> +\telse\n> +\t\tfprintf(stderr, \"preloaded output of a child\\n\");\n> +\n\nThis illustrates the intended use of !err and !!err pretty well ;-).\n\n"},{"id":"455430","messageId":"xmqq5ym2tslq.fsf@gitster.g","threadId":"57764","inReplyTo":"patch-v2-8.8-238155fcb9d-20220518T195858Z-avarab@gmail.com","subject":"Re: [PATCH v2 8/8] hook API: fix v2.36.0 regression: hooks should be connected to a TTY","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-18T21:53:53Z","receivedAt":"2022-05-18T22:03:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> In the preceding commit we removed the \"stdout_to_stderr=1\" assignment\n> as being redundant. This change brings it back as with \".ungroup=1\"\n> the run_process_parallel() function doesn't provide them for us\n> implicitly.\n\nThis part I recall commenting on the earlier round. The above\nmessage and the change in the patch makes sense.\n\nThanks.\n"},{"id":"456059","messageId":"nycvar.QRO.7.76.6.2205251308381.352@tvgsbejvaqbjf.bet","threadId":"57764","inReplyTo":"cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com","subject":"Re: [PATCH v2 0/8] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-05-25T11:30:58Z","receivedAt":"2022-05-25T11:31:39Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nas promised in the Git IRC Standup [*1*], a review.\n\nOn Wed, 18 May 2022, Ævar Arnfjörð Bjarmason wrote:\n\n> Ævar Arnfjörð Bjarmason (8):\n> run-command tests: change if/if/... to if/else if/else\n> run-command API: use \"opts\" struct for run_processes_parallel{,_tr2}()\n> run-command tests: test stdout of run_command_parallel()\n> run-command.c: add an initializer for \"struct parallel_processes\"\n> run-command: add an \"ungroup\" option to run_process_parallel()\n> hook tests: fix redirection logic error in 96e7225b310\n> hook API: don't redundantly re-set \"no_stdin\" and \"stdout_to_stderr\"\n> hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n\nI started reviewing the patches individually, but have some higher-level\nconcerns that put my per-patch review on hold.\n\nKeeping in mind that the intention is to fix a regression that was\nintroduced by way of refactoring (most of our recent regressions seem to\nshare that trait [*2*]), I strongly advise against another round of\nrefactoring [*3*], especially against tying it to fix a regression.\n\nIn this instance, it would be very easy to fix the bug without any\nrefactoring. In a nutshell, the manifestation of the bug amplifies this\npart of the commit message of 96e7225b310 (hook: add 'run' subcommand,\n2021-12-22):\n\n Some of the implementation here, such as a function being named\n run_hooks_opt() when it's tasked with running one hook, to using the\n run_processes_parallel_tr2() API to run with jobs=1 is somewhere\n between a bit odd and and an overkill for the current features of this\n \"hook run\" command and the hook.[ch] API.\n\nIt is this switch to `run_processes_parallel()` that is the root cause of\nthe regression.\n\nThe current iteration of the patch series does not fix that.\n\nIn the commit message from which I quoted, the plan is laid out to\neventually run more than one hook. If that is still the plan, we will be\npresented with the unfortunate choice to either never running them in\nparallel, or alternatively reintroducing the regression where the hooks\nrun detached from stdin/stdout/stderr.\n\nIt is pretty clear that there is no actual choice, and the hooks will\nnever be able to run in parallel. Therefore, the fix should move\n`run_hooks_opt()` away from calling `run_processes_parallel()`.\n\nIn any case, regression fixes should not be mixed with refactorings unless\nthe latter make the former easier, which is not the case here.\n\nCiao,\nJohannes\n\nFootnote *1*:\nhttps://colabti.org/irclogger/irclogger_log/git-devel?date=2022-05-23#l44\n\nFootnote *2*: I say \"seem\" because it would take a proper retro to analyze\nwhat was the reason for the uptick in regressions, and even more\nimportantly to analyze what we can learn from the experience.\n\nFootnote *3*: The refactoring, as Junio suspected, might very well go a\nbit over board. Even if a new variation of the `run_processes_parallel()`\nfunction that takes a struct should be necessary, it would be easy -- and\ndesirable -- to keep the current function signatures unchanged and simply\nturn them into shims that then call the new variant. That would make the\nrefactoring much easier to review, and in turn it would make it less\nlikely to introduce another regression.\n"},{"id":"456062","messageId":"Yo4sriq2YYtIsB0p@google.com","threadId":"57764","inReplyTo":"patch-v2-2.8-5f0a6e9925f-20220518T195858Z-avarab@gmail.com","subject":"Re: [PATCH v2 2/8] run-command API: use \"opts\" struct for run_processes_parallel{,_tr2}()","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2022-05-25T13:18:38Z","receivedAt":"2022-05-25T13:19:36Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Wed, May 18, 2022 at 10:05:18PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> Add a new \"struct run_process_parallel_opts\" to replace the growing\n> run_processes_parallel() and run_processes_parallel_tr2() argument\n> lists. This refactoring makes it easier to add new options and\n> parameters easier.\n> \n> The *_tr2() variant of the function was added in ee4512ed481 (trace2:\n> create new combined trace facility, 2019-02-22), and has subsequently\n> been used by every caller except t/helper/test-run-command.c.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n[...]\n> diff --git a/run-command.h b/run-command.h\n> index 5bd0c933e80..b0268ed3db1 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -458,6 +458,32 @@ typedef int (*task_finished_fn)(int result,\n> \t\t\t\tvoid *pp_task_cb);\n> \n> /**\n> + * Options to pass to run_processes_parallel(), { 0 }-initialized\n> + * means no options. Fields:\n> + *\n> + * tr2_category & tr2_label: sets the trace2 category and label for\n> + * logging. These must either be unset, or both of them must be set.\n\nI see this comment...\n\n> + *\n> + * jobs: see 'n' in run_processes_parallel() below.\n> + *\n> + * *_fn & data: see run_processes_parallel() below.\n> + */\n> +struct run_process_parallel_opts\n> +{\n> +\tconst char *tr2_category;\n> +\tconst char *tr2_label;\n> +\n> +\tint jobs;\n> +\n> +\tget_next_task_fn get_next_task;\n> +\tstart_failure_fn start_failure;\n> +\ttask_finished_fn task_finished;\n> +\tvoid *data;\n> +};\n\n[moved snippet]\n> -int run_processes_parallel(int n,\n> -\t\t\t get_next_task_fn get_next_task,\n> -\t\t\t start_failure_fn start_failure,\n> -\t\t\t task_finished_fn task_finished,\n> -\t\t\t void *pp_cb)\n> +int run_processes_parallel(struct run_process_parallel_opts *opts)\n> {\n> \tint i, code;\n> \tint output_timeout = 100;\n> \tint spawn_cap = 4;\n> \tstruct parallel_processes pp;\n> +\tconst char *tr2_category = opts->tr2_category;\n> +\tconst char *tr2_label = opts->tr2_label;\n> +\tconst int do_trace2 = tr2_category && tr2_label;\n\n...but it's not actually very well enforced here. That is, it seems I\ncan set one or the other but not both, and nothing bad will happen,\nexcept that I am wasting my time setting one. If you want to enforce\nthem both to be set, then why not use a BUG()? But otherwise the comment\ncould be reworded, I think.\n\n> +\tconst int n = opts->jobs;\n> \n> -\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n> +\tif (do_trace2)\n> +\t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n> +\t\t\t\t\t \"max:%d\", ((n < 1) ? online_cpus()\n> +\t\t\t\t\t\t : n));\n> +\n> +\tpp_init(&pp, opts);\n> \twhile (1) {\n> \t\tfor (i = 0;\n> \t\t i < spawn_cap && !pp.shutdown &&\n[/moved snippet]\n\n\nOtherwise, although the number of lines of code is often higher, I find\nthe named initializers in the struct much easier to read at the\ncallsites, so I like this change.\n\nReviewed-by: Emily Shaffer \n"},{"id":"456063","messageId":"220525.86fskxu4jx.gmgdl@evledraar.gmail.com","threadId":"57764","inReplyTo":"nycvar.QRO.7.76.6.2205251308381.352@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 0/8] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-25T13:00:52Z","receivedAt":"2022-05-25T13:26:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, May 25 2022, Johannes Schindelin wrote:\n\n> Hi Ævar,\n>\n> as promised in the Git IRC Standup [*1*], a review.\n>\n> On Wed, 18 May 2022, Ævar Arnfjörð Bjarmason wrote:\n>\n>> Ævar Arnfjörð Bjarmason (8):\n>> run-command tests: change if/if/... to if/else if/else\n>> run-command API: use \"opts\" struct for run_processes_parallel{,_tr2}()\n>> run-command tests: test stdout of run_command_parallel()\n>> run-command.c: add an initializer for \"struct parallel_processes\"\n>> run-command: add an \"ungroup\" option to run_process_parallel()\n>> hook tests: fix redirection logic error in 96e7225b310\n>> hook API: don't redundantly re-set \"no_stdin\" and \"stdout_to_stderr\"\n>> hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n>\n> I started reviewing the patches individually, but have some higher-level\n> concerns that put my per-patch review on hold.\n>\n> Keeping in mind that the intention is to fix a regression that was\n> introduced by way of refactoring (most of our recent regressions seem to\n> share that trait [*2*]), I strongly advise against another round of\n> refactoring [*3*], especially against tying it to fix a regression.\n>\n> In this instance, it would be very easy to fix the bug without any\n> refactoring. In a nutshell, the manifestation of the bug amplifies this\n> part of the commit message of 96e7225b310 (hook: add 'run' subcommand,\n> 2021-12-22):\n>\n> Some of the implementation here, such as a function being named\n> run_hooks_opt() when it's tasked with running one hook, to using the\n> run_processes_parallel_tr2() API to run with jobs=1 is somewhere\n> between a bit odd and and an overkill for the current features of this\n> \"hook run\" command and the hook.[ch] API.\n>\n> It is this switch to `run_processes_parallel()` that is the root cause of\n> the regression.\n\nYes, or more generally to the new hook API which makes use of it.\n\n> The current iteration of the patch series does not fix that.\n\nBecause the plan is still to continue in this direction and go for\nEmily's config-based hooks, which will run in parallel.\n\nAnd to fix that would at this point be a larger functional change,\nbecause we'd be running with more code we haven't tested before,\ni.e. hook.[ch] on some new backend. So just passing down the appropriate\nflags to have run-command.[ch] do the right thing for us seemed to be\nthe least bad option.\n\n> In the commit message from which I quoted, the plan is laid out to\n> eventually run more than one hook. If that is still the plan, we will be\n> presented with the unfortunate choice to either never running them in\n> parallel, or alternatively reintroducing the regression where the hooks\n> run detached from stdin/stdout/stderr.\n\nNo, because you can have N processes all connected to a terminal with\n\"ungroup\", what you can't do is guarantee that they won't interleave.\n\nBut as discussed in some previous threads that would be OK, since that\nwould come as an opt-in to parallel hook execution. I.e. you could pick\none of:\n\n 1. Current behavior\n 2. Our parallel hook execution (whatever \"ungroup\" etc. settings that entails)\n 3. Your own parallel hook execution\n\nIt only matters that we don't regress in #1, for #2 we could have\ndifferent behavior, but just document the caveats as such.\n\nIOW it's OK if you run parallel hooks and we decide that they won't be\nconnected to a terminal, because that's a new feature we don't have yet,\none you'd need to opt into.\n\n> It is pretty clear that there is no actual choice, and the hooks will\n> never be able to run in parallel. Therefore, the fix should move\n> `run_hooks_opt()` away from calling `run_processes_parallel()`.\n\nThey will run in parallel, see above.\n\n> In any case, regression fixes should not be mixed with refactorings unless\n> the latter make the former easier, which is not the case here.\n\nI noted upthread/side-thread (in any case, in discussions around this)\nthat I wished I'd come up with something smaller, but couldn't.\n\nIf you want to try your hand at that I'd love to see it.\n\nBut basically migrating the hook API to a new \"backend\" would also be a\nlarge change, so would making the bare minumum change in\nrun-command.[ch].\n\nBut hey, I might be wrong. So if you think it's obvious that this could\nbe much smaller I'd love to see patches for it...\n\n> Footnote *1*:\n> https://colabti.org/irclogger/irclogger_log/git-devel?date=2022-05-23#l44\n>\n> Footnote *2*: I say \"seem\" because it would take a proper retro to analyze\n> what was the reason for the uptick in regressions, and even more\n> importantly to analyze what we can learn from the experience.\n\nYes, that might be interesting.\n\nI'll only note that I think you're focusing on the wrong thing here with\n\"refactorings\".\n\nIf you look at the history of this hooks API topic it started early on\nwith a version where the config-based hooks + parallelism (currently not\nhere yet) were combined with making the existing hook users use the new\nAPI (partially here now).\n\nNow, I suggested that be split up so that we'd first re-implement all\nexisting hooks on the new API, and *then* perform any feature changes.\n\nExcept of course by doing so that alters the nature of those changes in\nyour definition, I assume, i.e. it goes from a feature series to\n\"refactorings\".\n\nWhereas I think the important thing to optimize for is to make smaller\nincremental changes. Here we had a bug, and it's relatively easy to fix\nit, it would be much harder if we had a bigger delta in v2.36 with not\nonly this bug, but some other regressions.\n\nWhich isn't hypothetical b.t.w., until 3-4 months ago nobody had seen\nthat the config-based hooks topic we had kicking around had a severe\nperformance regression. I found it and Emily & I have been kicking\naround a fix for it (mostly off-list).\n\nBut if we'd done that we'd have a more broken release, but we also\nwouldn't have \"refactorings\". I.e. the run_parallel API would actually\nbe used, but we'd have this breakage plus some others.\n\nAnyway, I think there's lots of things we could probably do better in\ndelivering more reliable software. I'm just pointing out that here that\nI think focusing on a part of a larger progression from A..B and saying\nthat it refactored something as being bad is to make a categorical\nmistake. Because a re-doing of that state to make each step not have any\nof those would result in larger change deltas.\n\n> Footnote *3*: The refactoring, as Junio suspected, might very well go a\n> bit over board. Even if a new variation of the `run_processes_parallel()`\n> function that takes a struct should be necessary, it would be easy -- and\n> desirable -- to keep the current function signatures unchanged and simply\n> turn them into shims that then call the new variant. That would make the\n> refactoring much easier to review, and in turn it would make it less\n> likely to introduce another regression.\n\nSure, we could instead add a third variant to it in addition to the two\non \"master\", instead of unifying them as is done here.\n\nBut per the v1 feedback the consensus seemed to be that this was a good\ndirection, and to the extent that there were objections it was that I\nshould add the rest of the arguments to the \"opts\" struct.\n\nBut again, I'm fully open to that, I tried that and didn't think the end\nresult was any simpler to review, but perhaps you'd like to try...\n"},{"id":"456109","messageId":"xmqqbkvl8s88.fsf@gitster.g","threadId":"57764","inReplyTo":"nycvar.QRO.7.76.6.2205251308381.352@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 0/8] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-25T16:57:59Z","receivedAt":"2022-05-25T16:58:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> Keeping in mind that the intention is to fix a regression that was\n> introduced by way of refactoring (most of our recent regressions seem to\n> share that trait [*2*]), I strongly advise against another round of\n> refactoring [*3*], especially against tying it to fix a regression.\n\nI share this sentiment.\n\n> In this instance, it would be very easy to fix the bug without any\n> refactoring. In a nutshell, the manifestation of the bug amplifies this\n> part of the commit message of 96e7225b310 (hook: add 'run' subcommand,\n> 2021-12-22):\n>\n> Some of the implementation here, such as a function being named\n> run_hooks_opt() when it's tasked with running one hook, to using the\n> run_processes_parallel_tr2() API to run with jobs=1 is somewhere\n> between a bit odd and and an overkill for the current features of this\n> \"hook run\" command and the hook.[ch] API.\n>\n> It is this switch to `run_processes_parallel()` that is the root cause of\n> the regression.\n>\n> The current iteration of the patch series does not fix that.\n\nTrue.\n\n> In the commit message from which I quoted, the plan is laid out to\n> eventually run more than one hook. If that is still the plan, we will be\n> presented with the unfortunate choice to either never running them in\n> parallel, or alternatively reintroducing the regression where the hooks\n> run detached from stdin/stdout/stderr.\n\nI had a similar impression before I reviewed the code after the\nregression report, but if I read the code before the breakage\ncorrectly, I think we didn't change the handling of the standard\ninput stream with the series from Emily/Ævar that broke the hooks.\n\nThe regression is the output streams are no longer _directly_\nconnected to the outside world, and instead to our internal relay\nthat buffers. The run_hook_ve() helper did set .no_stdin to 1\nbefore doing run_command() in Git 2.35. The series with regression\ndoes the same in pick_next_hook() callback in hook.c. Both also set\n.stdout_to_stderr to 1, so the apparent output should not change.\n\n> It is pretty clear that there is no actual choice, and the hooks will\n> never be able to run in parallel. Therefore, the fix should move\n> `run_hooks_opt()` away from calling `run_processes_parallel()`.\n\nMy take on it is slightly different.\n\nI personally do not think we should run hooks in parallel ourselves,\nbut if hook-like things, which Emily and Ævar want, want run in\nparallel, we can safely allow them to do so. No current users have\never seen such hook-like things specified in their configuration\nfiles---as long as it is clearly documented that these hook-like\nthings are not connected to the original standard output or error,\nand they may run in parallel and whatever inter-process coordination\nis their responsibility, there is no regression. It is a brand new\nfeature.\n\nThe mechanism that supports that hook-like things should have a\ncompatibility mode, if it ever wants to take responsibility of\nrunning the traditional hooks as part of its offering. I think the\nright way to do so is follows:\n\n - Unless each hook-like thing explicitly asks, it does not run in\n parallel with other hook-like things, and its output stream is\n connected directly to the original output stream. They can run\n without involving the run_processes_parallel() at all.\n\n - When the traditional on-disk hooks are treated as if it is one of\n these hook-like things, the compatibility mode should be set to\n on for them without any user interaction.\n\n - Only the new stuff written specifically to be used as these shiny\n new hook-like things would explicitly ask to run in parallel and\n emit to the output multiplexer.\n\nDoing things that way would pave the way forward to allow new stuff\nto work differently, without breaking existing stuff people have,\nwouldn't it?\n\n> In any case, regression fixes should not be mixed with refactorings unless\n> the latter make the former easier, which is not the case here.\n\nAbsolutely. I wonder how involved is would be to revert the merge\nof the whole thing from 'master'. It may give us a clean slate to\nrethink the whole mess and redo it without breaking the existing\nusers' hooks.\n"},{"id":"456141","messageId":"xmqqczg13xpy.fsf@gitster.g","threadId":"57764","inReplyTo":"xmqqbkvl8s88.fsf@gitster.g","subject":"Re: [PATCH v2 0/8] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-26T01:10:33Z","receivedAt":"2022-05-26T01:10:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Absolutely. I wonder how involved is would be to revert the merge\n> of the whole thing from 'master'. It may give us a clean slate to\n> rethink the whole mess and redo it without breaking the existing\n> users' hooks.\n\nI tried the revert, and the result compiled and tested OK, but I am\ntempted to say that it looks as if the topic was deliberately\ndesigned to make it hard to revert by taking as much stuff hostage\nas possible.\n\nAt least one fix that depends on the run_hooks_opt structure\nintroduced by c70bc338 (Merge branch 'ab/config-based-hooks-2',\n2022-02-09) needs to be discarded. 7431379a (Merge branch\n'ab/racy-hooks', 2022-03-16) did address an issue worth addressing,\nso even if we revert the whole c70bc338, we would want to redo the\nfix, possibly in some other way. But it also needed an \"oops that\nwas wrong, here is an attempt to fix it again\" by cb3b3974 (Merge\nbranch 'ab/racy-hooks', 2022-03-30). The situation is quite ugly.\n\nAs you hinted in the message I responded to in the message I am\nresponding to, if we can make a surgical fix to make the new and\nimproved run_hooks_opt() API build on top of run_command(), instead\non top of run_processes_parallel(), that would give us a cleaner way\nout than discarding everything and redoing them \"the right way\". At\nleast, the external interface into the API (read: the impression you\nwould get by \"less hook.h\") does not look too bad.\n\nThanks.\n"},{"id":"456165","messageId":"220526.86pmk060xa.gmgdl@evledraar.gmail.com","threadId":"57764","inReplyTo":"xmqqczg13xpy.fsf@gitster.g","subject":"Re: [PATCH v2 0/8] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-26T10:16:23Z","receivedAt":"2022-05-26T10:30:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, May 25 2022, Junio C Hamano wrote:\n\n> Junio C Hamano writes:\n>\n>> Absolutely. I wonder how involved is would be to revert the merge\n>> of the whole thing from 'master'. It may give us a clean slate to\n>> rethink the whole mess and redo it without breaking the existing\n>> users' hooks.\n>\n> I tried the revert, and the result compiled and tested OK, but I am\n> tempted to say that it looks as if the topic was deliberately\n> designed to make it hard to revert by taking as much stuff hostage\n> as possible.\n\nNo, it's just that...\n\n> At least one fix that depends on the run_hooks_opt structure\n> introduced by c70bc338 (Merge branch 'ab/config-based-hooks-2',\n> 2022-02-09) needs to be discarded. 7431379a (Merge branch\n> 'ab/racy-hooks', 2022-03-16) did address an issue worth addressing,\n\n...we've made some use of the API since then, including for that bug\nfix...\n\n> so even if we revert the whole c70bc338, we would want to redo the\n> fix, possibly in some other way. But it also needed an \"oops that\n> was wrong, here is an attempt to fix it again\" by cb3b3974 (Merge\n> branch 'ab/racy-hooks', 2022-03-30). The situation is quite ugly.\n\n...although for that last one if you're considering reverting that fix\ntoo to back out of the topic(s) it should be relatively easy to deal\nwith that one.\n\n> As you hinted in the message I responded to in the message I am\n> responding to, if we can make a surgical fix to make the new and\n> improved run_hooks_opt() API build on top of run_command(), instead\n> on top of run_processes_parallel(), that would give us a cleaner way\n> out than discarding everything and redoing them \"the right way\". At\n> least, the external interface into the API (read: the impression you\n> would get by \"less hook.h\") does not look too bad.\n\nI have a pending re-roll of this topic structured the way it is now (but\nwith fixes for outstanding issues).\n\nI understand your suggestion here to use the non-parallel API, and the\nreluctance to have a relatively large regression fix.\n\nI haven't come up with a patch in this direction, and I'll try before a\nre-roll, but I can't see how we wouldn't end up with code that's an even\nlarger logical change as a result.\n\nI.e. this would require rewriting a large part of hook.[ch] which is\ncurrently structured around the callback API, and carefully coming up\nwith the equivalent non-parallel API pattern for it.\n\nWhereas the current direction is more boilerplate for sure, but keeps\nall of that existing behavior, and just narrowly adjust what options we\npass down to the \"struct child_process\" in that case.\n\nI can try to come up with it (and delay the current re-roll I have\nthat's almost ready), but I really think that reviewing such a change\nwill be much harder.\n\nThe current proposal is large by line count, but it's relatively easy to\nskim it and assure oneself that a new parameter is being passed in, and\nthat all the proposed behavior change applies only to the one caller\nthat passes in that new parameter.\n\nWhereas switching to a new non-callback based API will require carefully\ngoing over the parallel API line-by-line, assuring oneself that the\nnon-callback version is really doing the same thing etc.\n"},{"id":"456190","messageId":"xmqqpmk01caj.fsf@gitster.g","threadId":"57764","inReplyTo":"220526.86pmk060xa.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 0/8] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-26T16:36:20Z","receivedAt":"2022-05-26T16:36:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> The current proposal is large by line count, but it's relatively easy to\n> skim it and assure oneself that a new parameter is being passed in, and\n> that all the proposed behavior change applies only to the one caller\n> that passes in that new parameter.\n>\n> Whereas switching to a new non-callback based API will require carefully\n> going over the parallel API line-by-line, assuring oneself that the\n> non-callback version is really doing the same thing etc.\n\nI was worried about something like that when I wrote (admittedly\nunfairly, in a somewhat frustrated state) that the series was\ndesigned to be hard to revert. The reverting itself was reasonably\neasy if the \"did we invoke the hook, really?\" topic is discarded at\nthe same time, but if was done with too much rearchitecting, it is\nunderstandable to become cumbersome to review X-<.\n\nI wonder if rebuilding from scratch is easier to review, then? The\nfirst three patches of such a series would be\n\n - Revert cb3b3974 (Merge branch 'ab/racy-hooks', 2022-03-30)\n - Revert 7431379a (Merge branch 'ab/racy-hooks', 2022-03-16)\n - Revert c70bc338 (Merge branch 'ab/config-based-hooks-2', 2022-02-09)\n\nand then the rest would rebuild what used to be in the original\nseries on top. There will be a lot of duplicate patches between\nthat \"the rest\" and the patches in the original series (e.g. I would\nimagine that the resulting hook.h would look more or less\nidentical), but \"git range-diff\" may be able to trim it down by\ncomparing between \"the rest\" and \"c70bc338^..c70bc338^2\" (aka\nab/config-based-hooks-2). I dunno.\n\nThanks.\n"},{"id":"456194","messageId":"Yo+2cbMueQyAI186@google.com","threadId":"57764","inReplyTo":"patch-v2-5.8-c2e015ed840-20220518T195858Z-avarab@gmail.com","subject":"Re: [PATCH v2 5/8] run-command: add an \"ungroup\" option to run_process_parallel()","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2022-05-26T17:18:59Z","receivedAt":"2022-05-26T17:19:15Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Wed, May 18, 2022 at 10:05:21PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> Extend the parallel execution API added in c553c72eed6 (run-command:\n> add an asynchronous parallel child processor, 2015-12-15) to support a\n> mode where the stdout and stderr of the processes isn't captured and\n> output in a deterministic order, instead we'll leave it to the kernel\n> and stdio to sort it out.\n> \n> This gives the API same functionality as GNU parallel's --ungroup\n> option. As we'll see in a subsequent commit the main reason to want\n> this is to support stdout and stderr being connected to the TTY in the\n> case of jobs=1, demonstrated here with GNU parallel:\n> \n> \t$ parallel --ungroup 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n> \tTTY\n> \tTTY\n> \t$ parallel 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n> \tNTTY\n> \tNTTY\n> \n> Another is as GNU parallel's documentation notes a potential for\n> optimization. Our results will be a bit different, but in cases where\n> you want to run processes in parallel where the exact order isn't\n> important this can be a lot faster:\n> \n> \t$ hyperfine -r 3 -L o ,--ungroup 'parallel {o} seq ::: 10000000 >/dev/null '\n> \tBenchmark 1: parallel seq ::: 10000000 >/dev/null\n> \t Time (mean ± σ): 220.2 ms ± 9.3 ms [User: 124.9 ms, System: 96.1 ms]\n> \t Range (min … max): 212.3 ms … 230.5 ms 3 runs\n> \n> \tBenchmark 2: parallel --ungroup seq ::: 10000000 >/dev/null\n> \t Time (mean ± σ): 154.7 ms ± 0.9 ms [User: 136.2 ms, System: 25.1 ms]\n> \t Range (min … max): 153.9 ms … 155.7 ms 3 runs\n> \n> \tSummary\n> \t 'parallel --ungroup seq ::: 10000000 >/dev/null ' ran\n> \t 1.42 ± 0.06 times faster than 'parallel seq ::: 10000000 >/dev/null '\n> \n> A large part of the juggling in the API is to make the API safer for\n> its maintenance and consumers alike.\n> \n> For the maintenance of the API we e.g. avoid malloc()-ing the\n> \"pp->pfd\", ensuring that SANITIZE=address and other similar tools will\n> catch any unexpected misuse.\n> \n> For API consumers we take pains to never pass the non-NULL \"out\"\n> buffer to an API user that provided the \"ungroup\" option. The\n> resulting code in t/helper/test-run-command.c isn't typical of such a\n> user, i.e. they'd typically use one mode or the other, and would know\n> whether they'd provided \"ungroup\" or not.\n\nAh, interesting, so it's a little finer grained than my suggestion of\n\"just always ungroup when jobs=1\". Since it's an option directly\navailable to the caller, that means they could use it for something\nwhere the ordering really doesn't matter as much as the rate, such as if\nthey wanted a bunch of subprocesses to split up some work in parallel or\nsomething. I like the flexibility, even though I know before I said \"why\ndon't we hide this functionality\". So I was wrong, and this looks nice\nto me :)\n\nI think we actually could even automatically set ungroup if jobs=1 as\nwell, because then there is no reason to buffer the output - it uses\nadditional memory for us, and it makes output slower to see for the end\nuser. But I do not really mind enough to want a reroll.\n\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> run-command.c | 76 ++++++++++++++++++++++++++++---------\n> run-command.h | 31 +++++++++++----\n> t/helper/test-run-command.c | 26 ++++++++++---\n> t/t0061-run-command.sh | 30 +++++++++++++++\n> 4 files changed, 132 insertions(+), 31 deletions(-)\n> \n> diff --git a/run-command.c b/run-command.c\n> index 839c85d12e5..39e09ee39fc 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1468,7 +1468,7 @@ int pipe_command(struct child_process *cmd,\n> enum child_state {\n> \tGIT_CP_FREE,\n> \tGIT_CP_WORKING,\n> -\tGIT_CP_WAIT_CLEANUP,\n> +\tGIT_CP_WAIT_CLEANUP, /* only for !ungroup */\n> };\n> \n> struct parallel_processes {\n> @@ -1494,6 +1494,7 @@ struct parallel_processes {\n> \tstruct pollfd *pfd;\n> \n> \tunsigned shutdown : 1;\n> +\tunsigned ungroup : 1;\n> \n> \tint output_owner;\n> \tstruct strbuf buffered_output; /* of finished children */\n> @@ -1563,12 +1564,16 @@ static void pp_init(struct parallel_processes *pp,\n> \tpp->nr_processes = 0;\n> \tpp->output_owner = 0;\n> \tpp->shutdown = 0;\n> +\tpp->ungroup = opts->ungroup;\n\nI was worried about what happens if the caller changes the value of\nrun_processes_parallel_opt.ungroup in the middle of execution, but since\nwe're copying the value away during init, I think it will have no\neffect. OK.\n\n> \tCALLOC_ARRAY(pp->children, n);\n> -\tCALLOC_ARRAY(pp->pfd, n);\n> +\tif (!pp->ungroup)\n> +\t\tCALLOC_ARRAY(pp->pfd, n);\n> \n> \tfor (i = 0; i < n; i++) {\n> \t\tstrbuf_init(&pp->children[i].err, 0);\n> \t\tchild_process_init(&pp->children[i].process);\n> +\t\tif (!pp->pfd)\n> +\t\t\tcontinue;\n> \t\tpp->pfd[i].events = POLLIN | POLLHUP;\n> \t\tpp->pfd[i].fd = -1;\n> \t}\n> @@ -1594,7 +1599,8 @@ static void pp_cleanup(struct parallel_processes *pp)\n> \t * When get_next_task added messages to the buffer in its last\n> \t * iteration, the buffered output is non empty.\n> \t */\n> -\tstrbuf_write(&pp->buffered_output, stderr);\n> +\tif (!pp->ungroup)\n> +\t\tstrbuf_write(&pp->buffered_output, stderr);\n> \tstrbuf_release(&pp->buffered_output);\n> \n> \tsigchain_pop_common();\n> @@ -1609,6 +1615,7 @@ static void pp_cleanup(struct parallel_processes *pp)\n> */\n> static int pp_start_one(struct parallel_processes *pp)\n> {\n> +\tconst int ungroup = pp->ungroup;\n> \tint i, code;\n> \n> \tfor (i = 0; i < pp->max_processes; i++)\n> @@ -1618,24 +1625,30 @@ static int pp_start_one(struct parallel_processes *pp)\n> \t\tBUG(\"bookkeeping is hard\");\n> \n> \tcode = pp->get_next_task(&pp->children[i].process,\n> -\t\t\t\t &pp->children[i].err,\n> +\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n> \t\t\t\t pp->data,\n> \t\t\t\t &pp->children[i].data);\n> \tif (!code) {\n> -\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> -\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\tif (!ungroup) {\n> +\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> +\t\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\t}\n> \t\treturn 1;\n> \t}\n> -\tpp->children[i].process.err = -1;\n> -\tpp->children[i].process.stdout_to_stderr = 1;\n> +\tif (!ungroup) {\n> +\t\tpp->children[i].process.err = -1;\n> +\t\tpp->children[i].process.stdout_to_stderr = 1;\n> +\t}\n\nHm, so for now if ungroup=1, we ignore the stdout entirely? It looks\nlike in patch 8 we're relying on the get_next_task callback to set these\ninstead, right? Or am I misunderstanding it, and the child's stderr goes\nto our stderr by default?\n\n> \tpp->children[i].process.no_stdin = 1;\n> \n> \tif (start_command(&pp->children[i].process)) {\n> -\t\tcode = pp->start_failure(&pp->children[i].err,\n> +\t\tcode = pp->start_failure(ungroup ? NULL : &pp->children[i].err,\n> \t\t\t\t\t pp->data,\n> \t\t\t\t\t pp->children[i].data);\n> -\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> -\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\tif (!ungroup) {\n> +\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> +\t\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\t}\n> \t\tif (code)\n> \t\t\tpp->shutdown = 1;\n> \t\treturn code;\n> @@ -1643,14 +1656,26 @@ static int pp_start_one(struct parallel_processes *pp)\n> \n> \tpp->nr_processes++;\n> \tpp->children[i].state = GIT_CP_WORKING;\n> -\tpp->pfd[i].fd = pp->children[i].process.err;\n> +\tif (pp->pfd)\n> +\t\tpp->pfd[i].fd = pp->children[i].process.err;\n> \treturn 0;\n> }\n> \n> +static void pp_mark_working_for_cleanup(struct parallel_processes *pp)\n> +{\n> +\tint i;\n> +\n> +\tfor (i = 0; i < pp->max_processes; i++)\n> +\t\tif (pp->children[i].state == GIT_CP_WORKING)\n> +\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n> +}\n> +\n> static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> {\n> \tint i;\n> \n> +\tassert(!pp->ungroup);\n> +\n> \twhile ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n> \t\tif (errno == EINTR)\n> \t\t\tcontinue;\n> @@ -1677,6 +1702,9 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> static void pp_output(struct parallel_processes *pp)\n> {\n> \tint i = pp->output_owner;\n> +\n> +\tassert(!pp->ungroup);\n> +\n> \tif (pp->children[i].state == GIT_CP_WORKING &&\n> \t pp->children[i].err.len) {\n> \t\tstrbuf_write(&pp->children[i].err, stderr);\n> @@ -1686,10 +1714,15 @@ static void pp_output(struct parallel_processes *pp)\n> \n> static int pp_collect_finished(struct parallel_processes *pp)\n> {\n> +\tconst int ungroup = pp->ungroup;\n> \tint i, code;\n> \tint n = pp->max_processes;\n> \tint result = 0;\n> \n> +\tif (ungroup)\n> +\t\tfor (i = 0; i < pp->max_processes; i++)\n> +\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n> +\n> \twhile (pp->nr_processes > 0) {\n> \t\tfor (i = 0; i < pp->max_processes; i++)\n> \t\t\tif (pp->children[i].state == GIT_CP_WAIT_CLEANUP)\n> @@ -1700,8 +1733,8 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> \t\tcode = finish_command(&pp->children[i].process);\n> \n> \t\tcode = pp->task_finished(code,\n> -\t\t\t\t\t &pp->children[i].err, pp->data,\n> -\t\t\t\t\t pp->children[i].data);\n> +\t\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n> +\t\t\t\t\t pp->data, pp->children[i].data);\n> \n> \t\tif (code)\n> \t\t\tresult = code;\n> @@ -1710,10 +1743,13 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> \n> \t\tpp->nr_processes--;\n> \t\tpp->children[i].state = GIT_CP_FREE;\n> -\t\tpp->pfd[i].fd = -1;\n> +\t\tif (pp->pfd)\n> +\t\t\tpp->pfd[i].fd = -1;\n> \t\tchild_process_init(&pp->children[i].process);\n> \n> -\t\tif (i != pp->output_owner) {\n> +\t\tif (ungroup) {\n> +\t\t\t; /* no strbuf_*() work to do here */\n> +\t\t} else if (i != pp->output_owner) {\n> \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> \t\t\tstrbuf_reset(&pp->children[i].err);\n> \t\t} else {\n> @@ -1774,8 +1810,12 @@ int run_processes_parallel(struct run_process_parallel_opts *opts)\n> \t\t}\n> \t\tif (!pp.nr_processes)\n> \t\t\tbreak;\n> -\t\tpp_buffer_stderr(&pp, output_timeout);\n> -\t\tpp_output(&pp);\n> +\t\tif (opts->ungroup) {\n> +\t\t\tpp_mark_working_for_cleanup(&pp);\n> +\t\t} else {\n> +\t\t\tpp_buffer_stderr(&pp, output_timeout);\n> +\t\t\tpp_output(&pp);\n> +\t\t}\n> \t\tcode = pp_collect_finished(&pp);\n> \t\tif (code) {\n> \t\t\tpp.shutdown = 1;\n> diff --git a/run-command.h b/run-command.h\n> index b0268ed3db1..dcb6ded4b55 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -405,6 +405,10 @@ void check_pipe(int err);\n> * pp_cb is the callback cookie as passed to run_processes_parallel.\n> * You can store a child process specific callback cookie in pp_task_cb.\n> *\n> + * The \"struct strbuf *err\" parameter is either a pointer to a string\n> + * to write errors to, or NULL if the \"ungroup\" option was\n> + * provided. See run_processes_parallel() below.\n> + *\n> * Even after returning 0 to indicate that there are no more processes,\n> * this function will be called again until there are no more running\n> * child processes.\n> @@ -423,9 +427,9 @@ typedef int (*get_next_task_fn)(struct child_process *cp,\n> * This callback is called whenever there are problems starting\n> * a new process.\n> *\n> - * You must not write to stdout or stderr in this function. Add your\n> - * message to the strbuf out instead, which will be printed without\n> - * messing up the output of the other parallel processes.\n> + * The \"struct strbuf *err\" parameter is either a pointer to a string\n> + * to write errors to, or NULL if the \"ungroup\" option was\n> + * provided. See run_processes_parallel() below.\n\nI think we are losing some useful information here (\"do not write\ndirectly to stdout or stderr from here\"). Same for below.\n\n> *\n> * pp_cb is the callback cookie as passed into run_processes_parallel,\n> * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> @@ -441,9 +445,9 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n> /**\n> * This callback is called on every child process that finished processing.\n> *\n> - * You must not write to stdout or stderr in this function. Add your\n> - * message to the strbuf out instead, which will be printed without\n> - * messing up the output of the other parallel processes.\n> + * The \"struct strbuf *err\" parameter is either a pointer to a string\n> + * to write errors to, or NULL if the \"ungroup\" option was\n> + * provided. See run_processes_parallel() below.\n> *\n> * pp_cb is the callback cookie as passed into run_processes_parallel,\n> * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> @@ -466,6 +470,10 @@ typedef int (*task_finished_fn)(int result,\n> *\n> * jobs: see 'n' in run_processes_parallel() below.\n> *\n> + * ungroup: Ungroup output. Output is printed as soon as possible and\n> + * bypasses run-command's internal processing. This may cause output\n> + * from different commands to be mixed.\n> + *\n> * *_fn & data: see run_processes_parallel() below.\n> */\n> struct run_process_parallel_opts\n> @@ -474,6 +482,7 @@ struct run_process_parallel_opts\n> \tconst char *tr2_label;\n> \n> \tint jobs;\n> +\tunsigned int ungroup:1;\n> \n> \tget_next_task_fn get_next_task;\n> \tstart_failure_fn start_failure;\n> @@ -490,10 +499,18 @@ struct run_process_parallel_opts\n> *\n> * The children started via this function run in parallel. Their output\n> * (both stdout and stderr) is routed to stderr in a manner that output\n> - * from different tasks does not interleave.\n> + * from different tasks does not interleave (but see \"ungroup\" above).\n\nHm, I think it would be more accurate to say, \"By default, their output\n(blah blah)...\" because in fact when we set ungroup, it does interleave.\nBut this is a small nit, from reading \"ungroup\" it becomes clear.\n\n> *\n> * start_failure_fn and task_finished_fn can be NULL to omit any\n> * special handling.\n> + *\n> + * If the \"ungroup\" option isn't specified the callbacks will get a\n> + * pointer to a \"struct strbuf *out\", and must not write to stdout or\n> + * stderr as such output will mess up the output of the other parallel\n> + * processes. If \"ungroup\" option is specified callbacks will get a\n> + * NULL \"struct strbuf *out\" parameter, and are responsible for\n> + * emitting their own output, including dealing with any race\n> + * conditions due to writing in parallel to stdout and stderr.\n> */\n> int run_processes_parallel(struct run_process_parallel_opts *opts);\n> \n> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\n> index 56a806f228b..986acbce5f2 100644\n> --- a/t/helper/test-run-command.c\n> +++ b/t/helper/test-run-command.c\n> @@ -31,7 +31,11 @@ static int parallel_next(struct child_process *cp,\n> \t\treturn 0;\n> \n> \tstrvec_pushv(&cp->args, d->args.v);\n> -\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n> +\tif (err)\n> +\t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n> +\telse\n> +\t\tfprintf(stderr, \"preloaded output of a child\\n\");\n> +\n> \tnumber_callbacks++;\n> \treturn 1;\n> }\n> @@ -41,7 +45,10 @@ static int no_job(struct child_process *cp,\n> \t\t void *cb,\n> \t\t void **task_cb)\n> {\n> -\tstrbuf_addstr(err, \"no further jobs available\\n\");\n> +\tif (err)\n> +\t\tstrbuf_addstr(err, \"no further jobs available\\n\");\n> +\telse\n> +\t\tfprintf(stderr, \"no further jobs available\\n\");\n> \treturn 0;\n> }\n> \n> @@ -50,7 +57,10 @@ static int task_finished(int result,\n> \t\t\t void *pp_cb,\n> \t\t\t void *pp_task_cb)\n> {\n> -\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n> +\tif (err)\n> +\t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n> +\telse\n> +\t\tfprintf(stderr, \"asking for a quick stop\\n\");\n> \treturn 1;\n> }\n> \n> @@ -422,12 +432,15 @@ int cmd__run_command(int argc, const char **argv)\n> \topts.jobs = jobs;\n> \topts.data = &proc;\n> \n> -\tif (!strcmp(argv[1], \"run-command-parallel\")) {\n> +\tif (!strcmp(argv[1], \"run-command-parallel\") ||\n> +\t !strcmp(argv[1], \"run-command-parallel-ungroup\")) {\n> \t\tnext_fn = parallel_next;\n> -\t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n> +\t} else if (!strcmp(argv[1], \"run-command-abort\") ||\n> +\t\t !strcmp(argv[1], \"run-command-abort-ungroup\")) {\n> \t\tnext_fn = parallel_next;\n> \t\tfinished_fn = task_finished;\n> -\t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n> +\t} else if (!strcmp(argv[1], \"run-command-no-jobs\") ||\n> +\t\t !strcmp(argv[1], \"run-command-no-jobs-ungroup\")) {\n> \t\tnext_fn = no_job;\n> \t\tfinished_fn = task_finished;\n> \t} else {\n> @@ -435,6 +448,7 @@ int cmd__run_command(int argc, const char **argv)\n> \t\treturn 1;\n> \t}\n> \n> +\topts.ungroup = ends_with(argv[1], \"-ungroup\");\n> \topts.get_next_task = next_fn;\n> \topts.task_finished = finished_fn;\n> \texit(run_processes_parallel(&opts));\n> diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\n> index 7d00f3cc2af..3628719a06d 100755\n> --- a/t/t0061-run-command.sh\n> +++ b/t/t0061-run-command.sh\n> @@ -135,18 +135,36 @@ test_expect_success 'run_command runs in parallel with more jobs available than\n> \ttest_cmp expect err\n> '\n> \n> +test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n> +\ttest-tool run-command run-command-parallel-ungroup 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_line_count = 8 out &&\n> +\ttest_line_count = 4 err\n> +'\n> +\n> test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n> \ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> \ttest_must_be_empty out &&\n> \ttest_cmp expect err\n> '\n> \n> +test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n> +\ttest-tool run-command run-command-parallel-ungroup 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_line_count = 8 out &&\n> +\ttest_line_count = 4 err\n> +'\n> +\n> test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n> \ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> \ttest_must_be_empty out &&\n> \ttest_cmp expect err\n> '\n> \n> +test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n> +\ttest-tool run-command run-command-parallel-ungroup 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_line_count = 8 out &&\n> +\ttest_line_count = 4 err\n> +'\n> +\n> cat >expect <<-EOF\n> preloaded output of a child\n> asking for a quick stop\n> @@ -162,6 +180,12 @@ test_expect_success 'run_command is asked to abort gracefully' '\n> \ttest_cmp expect err\n> '\n> \n> +test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n> +\ttest-tool run-command run-command-abort-ungroup 3 false >out 2>err &&\n> +\ttest_must_be_empty out &&\n> +\ttest_line_count = 6 err\n> +'\n> +\n> cat >expect <<-EOF\n> no further jobs available\n> EOF\n> @@ -172,6 +196,12 @@ test_expect_success 'run_command outputs ' '\n> \ttest_cmp expect err\n> '\n> \n> +test_expect_success 'run_command outputs (ungroup) ' '\n> +\ttest-tool run-command run-command-no-jobs-ungroup 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_must_be_empty out &&\n> +\ttest_cmp expect err\n> +'\n> +\n> test_trace () {\n> \texpect=\"$1\"\n> \tshift\n> -- \n> 2.36.1.952.g0ae626f6cd7\n> \n"},{"id":"456195","messageId":"Yo+3gmtbaARan23V@google.com","threadId":"57764","inReplyTo":"patch-v2-8.8-238155fcb9d-20220518T195858Z-avarab@gmail.com","subject":"Re: [PATCH v2 8/8] hook API: fix v2.36.0 regression: hooks should be connected to a TTY","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2022-05-26T17:23:14Z","receivedAt":"2022-05-26T17:23:23Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Wed, May 18, 2022 at 10:05:24PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> Fix a regression reported[1] in f443246b9f2 (commit: convert\n> {pre-commit,prepare-commit-msg} hook to hook.h, 2021-12-22): Due to\n> using the run_process_parallel() API in the earlier 96e7225b310 (hook:\n> add 'run' subcommand, 2021-12-22) we'd capture the hook's stderr and\n> stdout, and thus lose the connection to the TTY in the case of\n> e.g. the \"pre-commit\" hook.\n> \n> As a preceding commit notes GNU parallel's similar --ungroup option\n> also has it emit output faster. While we're unlikely to have hooks\n> that emit truly massive amounts of output (or where the performance\n> thereof matters) it's still informative to measure the overhead. In a\n> similar \"seq\" test we're now ~30% faster:\n> \n> \t$ cat .git/hooks/seq-hook; git hyperfine -L rev origin/master,HEAD~0 -s 'make CFLAGS=-O3' './git hook run seq-hook'\n> \t#!/bin/sh\n> \n> \tseq 100000000\n> \tBenchmark 1: ./git hook run seq-hook' in 'origin/master\n> \t Time (mean ± σ): 787.1 ms ± 13.6 ms [User: 701.6 ms, System: 534.4 ms]\n> \t Range (min … max): 773.2 ms … 806.3 ms 10 runs\n> \n> \tBenchmark 2: ./git hook run seq-hook' in 'HEAD~0\n> \t Time (mean ± σ): 603.4 ms ± 1.6 ms [User: 573.1 ms, System: 30.3 ms]\n> \t Range (min … max): 601.0 ms … 606.2 ms 10 runs\n> \n> \tSummary\n> \t './git hook run seq-hook' in 'HEAD~0' ran\n> \t 1.30 ± 0.02 times faster than './git hook run seq-hook' in 'origin/master'\n> \n> In the preceding commit we removed the \"stdout_to_stderr=1\" assignment\n> as being redundant. This change brings it back as with \".ungroup=1\"\n> the run_process_parallel() function doesn't provide them for us\n> implicitly.\n> \n> As an aside omitting the stdout_to_stderr=1 here would have all tests\n> pass, except those that test \"git hook run\" itself in\n> t1800-hook.sh. But our tests passing is the result of another test\n> blind spot, as was the case with the regression being fixed here. The\n> \"stdout_to_stderr=1\" for hooks is long-standing behavior, see\n> e.g. 1d9e8b56fe3 (Split back out update_hook handling in receive-pack,\n> 2007-03-10) and other follow-up commits (running \"git log\" with\n> \"--reverse -p -Gstdout_to_stderr\" is a good start).\n> \n> 1. https://lore.kernel.org/git/CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com/\n> \n> Reported-by: Anthony Sottile \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> hook.c | 5 +++++\n> t/t1800-hook.sh | 37 +++++++++++++++++++++++++++++++++++++\n> 2 files changed, 42 insertions(+)\n> \n> diff --git a/hook.c b/hook.c\n> index dc498ef5c39..5f31b60384a 100644\n> --- a/hook.c\n> +++ b/hook.c\n> @@ -54,6 +54,7 @@ static int pick_next_hook(struct child_process *cp,\n> \t\treturn 0;\n> \n> \tstrvec_pushv(&cp->env_array, hook_cb->options->env.v);\n> +\tcp->stdout_to_stderr = 1; /* because of .ungroup = 1 */\n> \tcp->trace2_hook_name = hook_cb->hook_name;\n> \tcp->dir = hook_cb->options->dir;\n> \n> @@ -126,6 +127,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n> \t\t.tr2_label = hook_name,\n> \n> \t\t.jobs = jobs,\n> +\t\t.ungroup = jobs == 1,\n\nI mentioned it on patch 5, but I actually do not see a reason why we\nshouldn't do this logic in run_processes_parallel instead of just for\nthe hooks. If someone can mention a reason we want to buffer child\nprocesses we're running in series I'm all ears, of course.\n\n> \n> \t\t.get_next_task = pick_next_hook,\n> \t\t.start_failure = notify_start_failure,\n> @@ -136,6 +138,9 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n> \tif (!options)\n> \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n> \n> +\tif (jobs != 1 || !run_opts.ungroup)\n> +\t\tBUG(\"TODO: think about & document order & interleaving of parallel hook output\");\n\nDoesn't this mean we're actually disallowing parallel hooks entirely? I\ndon't think that's necessary or desired. I guess right now when the\nconfig isn't used, there's not really a way to provide parallel hooks,\nbut I also think this will cause unnecessary conflicts for Google who is\ncarrying config hooks downstream. I know that's not such a great reason.\nBut it seems weird to be explicitly using the parallel processing\nframework, but then say, \"oh, but we actually don't want to run in\nparallel, that's a BUG()\".\n\n> +\n> \tif (options->invoked_hook)\n> \t\t*options->invoked_hook = 0;\n> \n> diff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\n> index 1e4adc3d53e..f22754deccc 100755\n> --- a/t/t1800-hook.sh\n> +++ b/t/t1800-hook.sh\n> @@ -4,6 +4,7 @@ test_description='git-hook command'\n> \n> TEST_PASSES_SANITIZE_LEAK=true\n> . ./test-lib.sh\n> +. \"$TEST_DIRECTORY\"/lib-terminal.sh\n> \n> test_expect_success 'git hook usage' '\n> \ttest_expect_code 129 git hook &&\n> @@ -120,4 +121,40 @@ test_expect_success 'git -c core.hooksPath= hook run' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_hook_tty() {\n> +\tlocal fd=\"$1\" &&\n> +\n> +\tcat >expect &&\n> +\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\n> +\ttest_hook -C repo pre-commit <<-EOF &&\n> +\t{\n> +\t\ttest -t 1 && echo >&$fd STDOUT TTY || echo >&$fd STDOUT NO TTY &&\n> +\t\ttest -t 2 && echo >&$fd STDERR TTY || echo >&$fd STDERR NO TTY\n> +\t} $fd>actual\n> +\tEOF\n> +\n> +\ttest_commit -C repo A &&\n> +\ttest_commit -C repo B &&\n> +\tgit -C repo reset --soft HEAD^ &&\n> +\ttest_terminal git -C repo commit -m\"B.new\" &&\n> +\ttest_cmp expect repo/actual\n> +}\n> +\n> +test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDOUT redirect' '\n> +\ttest_hook_tty 1 <<-\\EOF\n> +\tSTDOUT NO TTY\n> +\tSTDERR TTY\n> +\tEOF\n> +'\n> +\n> +test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDERR redirect' '\n> +\ttest_hook_tty 2 <<-\\EOF\n> +\tSTDOUT TTY\n> +\tSTDERR NO TTY\n> +\tEOF\n> +'\n> +\n> test_done\n> -- \n> 2.36.1.952.g0ae626f6cd7\n> \n"},{"id":"456204","messageId":"220526.86h75c5f01.gmgdl@evledraar.gmail.com","threadId":"57764","inReplyTo":"Yo+3gmtbaARan23V@google.com","subject":"Re: [PATCH v2 8/8] hook API: fix v2.36.0 regression: hooks should be connected to a TTY","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-26T18:23:23Z","receivedAt":"2022-05-26T18:24:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, May 26 2022, Emily Shaffer wrote:\n\n> On Wed, May 18, 2022 at 10:05:24PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> Fix a regression reported[1] in f443246b9f2 (commit: convert\n>> {pre-commit,prepare-commit-msg} hook to hook.h, 2021-12-22): Due to\n>> using the run_process_parallel() API in the earlier 96e7225b310 (hook:\n>> add 'run' subcommand, 2021-12-22) we'd capture the hook's stderr and\n>> stdout, and thus lose the connection to the TTY in the case of\n>> e.g. the \"pre-commit\" hook.\n>> \n>> As a preceding commit notes GNU parallel's similar --ungroup option\n>> also has it emit output faster. While we're unlikely to have hooks\n>> that emit truly massive amounts of output (or where the performance\n>> thereof matters) it's still informative to measure the overhead. In a\n>> similar \"seq\" test we're now ~30% faster:\n>> \n>> \t$ cat .git/hooks/seq-hook; git hyperfine -L rev origin/master,HEAD~0 -s 'make CFLAGS=-O3' './git hook run seq-hook'\n>> \t#!/bin/sh\n>> \n>> \tseq 100000000\n>> \tBenchmark 1: ./git hook run seq-hook' in 'origin/master\n>> \t Time (mean ± σ): 787.1 ms ± 13.6 ms [User: 701.6 ms, System: 534.4 ms]\n>> \t Range (min … max): 773.2 ms … 806.3 ms 10 runs\n>> \n>> \tBenchmark 2: ./git hook run seq-hook' in 'HEAD~0\n>> \t Time (mean ± σ): 603.4 ms ± 1.6 ms [User: 573.1 ms, System: 30.3 ms]\n>> \t Range (min … max): 601.0 ms … 606.2 ms 10 runs\n>> \n>> \tSummary\n>> \t './git hook run seq-hook' in 'HEAD~0' ran\n>> \t 1.30 ± 0.02 times faster than './git hook run seq-hook' in 'origin/master'\n>> \n>> In the preceding commit we removed the \"stdout_to_stderr=1\" assignment\n>> as being redundant. This change brings it back as with \".ungroup=1\"\n>> the run_process_parallel() function doesn't provide them for us\n>> implicitly.\n>> \n>> As an aside omitting the stdout_to_stderr=1 here would have all tests\n>> pass, except those that test \"git hook run\" itself in\n>> t1800-hook.sh. But our tests passing is the result of another test\n>> blind spot, as was the case with the regression being fixed here. The\n>> \"stdout_to_stderr=1\" for hooks is long-standing behavior, see\n>> e.g. 1d9e8b56fe3 (Split back out update_hook handling in receive-pack,\n>> 2007-03-10) and other follow-up commits (running \"git log\" with\n>> \"--reverse -p -Gstdout_to_stderr\" is a good start).\n>> \n>> 1. https://lore.kernel.org/git/CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com/\n>> \n>> Reported-by: Anthony Sottile \n>> Signed-off-by: Ævar Arnfjörð Bjarmason \n>> ---\n>> hook.c | 5 +++++\n>> t/t1800-hook.sh | 37 +++++++++++++++++++++++++++++++++++++\n>> 2 files changed, 42 insertions(+)\n>> \n>> diff --git a/hook.c b/hook.c\n>> index dc498ef5c39..5f31b60384a 100644\n>> --- a/hook.c\n>> +++ b/hook.c\n>> @@ -54,6 +54,7 @@ static int pick_next_hook(struct child_process *cp,\n>> \t\treturn 0;\n>> \n>> \tstrvec_pushv(&cp->env_array, hook_cb->options->env.v);\n>> +\tcp->stdout_to_stderr = 1; /* because of .ungroup = 1 */\n>> \tcp->trace2_hook_name = hook_cb->hook_name;\n>> \tcp->dir = hook_cb->options->dir;\n>> \n>> @@ -126,6 +127,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n>> \t\t.tr2_label = hook_name,\n>> \n>> \t\t.jobs = jobs,\n>> +\t\t.ungroup = jobs == 1,\n>\n> I mentioned it on patch 5, but I actually do not see a reason why we\n> shouldn't do this logic in run_processes_parallel instead of just for\n> the hooks. If someone can mention a reason we want to buffer child\n> processes we're running in series I'm all ears, of course.\n>\n>> \n>> \t\t.get_next_task = pick_next_hook,\n>> \t\t.start_failure = notify_start_failure,\n>> @@ -136,6 +138,9 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n>> \tif (!options)\n>> \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n>> \n>> +\tif (jobs != 1 || !run_opts.ungroup)\n>> +\t\tBUG(\"TODO: think about & document order & interleaving of parallel hook output\");\n>\n> Doesn't this mean we're actually disallowing parallel hooks entirely? I\n> don't think that's necessary or desired. I guess right now when the\n> config isn't used, there's not really a way to provide parallel hooks,\n> but I also think this will cause unnecessary conflicts for Google who is\n> carrying config hooks downstream. I know that's not such a great reason.\n> But it seems weird to be explicitly using the parallel processing\n> framework, but then say, \"oh, but we actually don't want to run in\n> parallel, that's a BUG()\".\n\nI can just drop this paranoia. I figured it was prudent to leave this\nlandmine in place so we'd definitely remember to re-visit this aspect of\nit, but I think there's 0% that we'll forget. So I'll make it less\nparanoid.\n"},{"id":"456211","messageId":"Yo/M86Y5jo/Yc7Nj@google.com","threadId":"57764","inReplyTo":"220526.86h75c5f01.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 8/8] hook API: fix v2.36.0 regression: hooks should be connected to a TTY","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2022-05-26T18:54:43Z","receivedAt":"2022-05-26T18:54:54Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, May 26, 2022 at 08:23:23PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> \n> On Thu, May 26 2022, Emily Shaffer wrote:\n> \n> > On Wed, May 18, 2022 at 10:05:24PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> >> \n> >> Fix a regression reported[1] in f443246b9f2 (commit: convert\n> >> {pre-commit,prepare-commit-msg} hook to hook.h, 2021-12-22): Due to\n> >> using the run_process_parallel() API in the earlier 96e7225b310 (hook:\n> >> add 'run' subcommand, 2021-12-22) we'd capture the hook's stderr and\n> >> stdout, and thus lose the connection to the TTY in the case of\n> >> e.g. the \"pre-commit\" hook.\n> >> \n> >> As a preceding commit notes GNU parallel's similar --ungroup option\n> >> also has it emit output faster. While we're unlikely to have hooks\n> >> that emit truly massive amounts of output (or where the performance\n> >> thereof matters) it's still informative to measure the overhead. In a\n> >> similar \"seq\" test we're now ~30% faster:\n> >> \n> >> \t$ cat .git/hooks/seq-hook; git hyperfine -L rev origin/master,HEAD~0 -s 'make CFLAGS=-O3' './git hook run seq-hook'\n> >> \t#!/bin/sh\n> >> \n> >> \tseq 100000000\n> >> \tBenchmark 1: ./git hook run seq-hook' in 'origin/master\n> >> \t Time (mean ± σ): 787.1 ms ± 13.6 ms [User: 701.6 ms, System: 534.4 ms]\n> >> \t Range (min … max): 773.2 ms … 806.3 ms 10 runs\n> >> \n> >> \tBenchmark 2: ./git hook run seq-hook' in 'HEAD~0\n> >> \t Time (mean ± σ): 603.4 ms ± 1.6 ms [User: 573.1 ms, System: 30.3 ms]\n> >> \t Range (min … max): 601.0 ms … 606.2 ms 10 runs\n> >> \n> >> \tSummary\n> >> \t './git hook run seq-hook' in 'HEAD~0' ran\n> >> \t 1.30 ± 0.02 times faster than './git hook run seq-hook' in 'origin/master'\n> >> \n> >> In the preceding commit we removed the \"stdout_to_stderr=1\" assignment\n> >> as being redundant. This change brings it back as with \".ungroup=1\"\n> >> the run_process_parallel() function doesn't provide them for us\n> >> implicitly.\n> >> \n> >> As an aside omitting the stdout_to_stderr=1 here would have all tests\n> >> pass, except those that test \"git hook run\" itself in\n> >> t1800-hook.sh. But our tests passing is the result of another test\n> >> blind spot, as was the case with the regression being fixed here. The\n> >> \"stdout_to_stderr=1\" for hooks is long-standing behavior, see\n> >> e.g. 1d9e8b56fe3 (Split back out update_hook handling in receive-pack,\n> >> 2007-03-10) and other follow-up commits (running \"git log\" with\n> >> \"--reverse -p -Gstdout_to_stderr\" is a good start).\n> >> \n> >> 1. https://lore.kernel.org/git/CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com/\n> >> \n> >> Reported-by: Anthony Sottile \n> >> Signed-off-by: Ævar Arnfjörð Bjarmason \n> >> ---\n> >> hook.c | 5 +++++\n> >> t/t1800-hook.sh | 37 +++++++++++++++++++++++++++++++++++++\n> >> 2 files changed, 42 insertions(+)\n> >> \n> >> diff --git a/hook.c b/hook.c\n> >> index dc498ef5c39..5f31b60384a 100644\n> >> --- a/hook.c\n> >> +++ b/hook.c\n> >> @@ -54,6 +54,7 @@ static int pick_next_hook(struct child_process *cp,\n> >> \t\treturn 0;\n> >> \n> >> \tstrvec_pushv(&cp->env_array, hook_cb->options->env.v);\n> >> +\tcp->stdout_to_stderr = 1; /* because of .ungroup = 1 */\n> >> \tcp->trace2_hook_name = hook_cb->hook_name;\n> >> \tcp->dir = hook_cb->options->dir;\n> >> \n> >> @@ -126,6 +127,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n> >> \t\t.tr2_label = hook_name,\n> >> \n> >> \t\t.jobs = jobs,\n> >> +\t\t.ungroup = jobs == 1,\n> >\n> > I mentioned it on patch 5, but I actually do not see a reason why we\n> > shouldn't do this logic in run_processes_parallel instead of just for\n> > the hooks. If someone can mention a reason we want to buffer child\n> > processes we're running in series I'm all ears, of course.\n> >\n> >> \n> >> \t\t.get_next_task = pick_next_hook,\n> >> \t\t.start_failure = notify_start_failure,\n> >> @@ -136,6 +138,9 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n> >> \tif (!options)\n> >> \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n> >> \n> >> +\tif (jobs != 1 || !run_opts.ungroup)\n> >> +\t\tBUG(\"TODO: think about & document order & interleaving of parallel hook output\");\n> >\n> > Doesn't this mean we're actually disallowing parallel hooks entirely? I\n> > don't think that's necessary or desired. I guess right now when the\n> > config isn't used, there's not really a way to provide parallel hooks,\n> > but I also think this will cause unnecessary conflicts for Google who is\n> > carrying config hooks downstream. I know that's not such a great reason.\n> > But it seems weird to be explicitly using the parallel processing\n> > framework, but then say, \"oh, but we actually don't want to run in\n> > parallel, that's a BUG()\".\n> \n> I can just drop this paranoia. I figured it was prudent to leave this\n> landmine in place so we'd definitely remember to re-visit this aspect of\n> it, but I think there's 0% that we'll forget. So I'll make it less\n> paranoid.\n\nThanks. With that change the series looks good to me otherwise, although\nif you're rerolling to drop it, maybe consider some of the other little\nnits I left elsewhere. ;)\n\n - Emily\n"},{"id":"456219","messageId":"220526.86ee0g3y1k.gmgdl@evledraar.gmail.com","threadId":"57764","inReplyTo":"xmqqpmk01caj.fsf@gitster.g","subject":"Re: [PATCH v2 0/8] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-26T19:13:50Z","receivedAt":"2022-05-26T19:15:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, May 26 2022, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> The current proposal is large by line count, but it's relatively easy to\n>> skim it and assure oneself that a new parameter is being passed in, and\n>> that all the proposed behavior change applies only to the one caller\n>> that passes in that new parameter.\n>>\n>> Whereas switching to a new non-callback based API will require carefully\n>> going over the parallel API line-by-line, assuring oneself that the\n>> non-callback version is really doing the same thing etc.\n>\n> I was worried about something like that when I wrote (admittedly\n> unfairly, in a somewhat frustrated state) that the series was\n> designed to be hard to revert. The reverting itself was reasonably\n> easy if the \"did we invoke the hook, really?\" topic is discarded at\n> the same time, but if was done with too much rearchitecting, it is\n> understandable to become cumbersome to review X-<.\n>\n> I wonder if rebuilding from scratch is easier to review, then? The\n> first three patches of such a series would be\n>\n> - Revert cb3b3974 (Merge branch 'ab/racy-hooks', 2022-03-30)\n> - Revert 7431379a (Merge branch 'ab/racy-hooks', 2022-03-16)\n> - Revert c70bc338 (Merge branch 'ab/config-based-hooks-2', 2022-02-09)\n>\n> and then the rest would rebuild what used to be in the original\n> series on top. There will be a lot of duplicate patches between\n> that \"the rest\" and the patches in the original series (e.g. I would\n> imagine that the resulting hook.h would look more or less\n> identical), but \"git range-diff\" may be able to trim it down by\n> comparing between \"the rest\" and \"c70bc338^..c70bc338^2\" (aka\n> ab/config-based-hooks-2). I dunno.\n\nI'm still happy to and planning to send a re-roll of this to try to\naddress outstanding comments/concerns, but am holding off for now\nbecause it's not clear to me if you're already planning to discard any\nsuch re-roll in favor of a revert.\n\nOr do you mean to create a point release with such revert(s) and have\nmaster free to move forward with a fix for the outstanding issue, but\nnot to use that for a point release?\n\n\n\n"},{"id":"456224","messageId":"xmqqsfowxe2p.fsf@gitster.g","threadId":"57764","inReplyTo":"220526.86ee0g3y1k.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 0/8] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-26T19:56:46Z","receivedAt":"2022-05-26T19:56:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n>> I wonder if rebuilding from scratch is easier to review, then? The\n>> first three patches of such a series would be\n>>\n>> - Revert cb3b3974 (Merge branch 'ab/racy-hooks', 2022-03-30)\n>> - Revert 7431379a (Merge branch 'ab/racy-hooks', 2022-03-16)\n>> - Revert c70bc338 (Merge branch 'ab/config-based-hooks-2', 2022-02-09)\n>>\n>> and then the rest would rebuild what used to be in the original\n>> series on top. There will be a lot of duplicate patches between\n>> that \"the rest\" and the patches in the original series (e.g. I would\n>> imagine that the resulting hook.h would look more or less\n>> identical), but \"git range-diff\" may be able to trim it down by\n>> comparing between \"the rest\" and \"c70bc338^..c70bc338^2\" (aka\n>> ab/config-based-hooks-2). I dunno.\n>\n> I'm still happy to and planning to send a re-roll of this to try to\n> address outstanding comments/concerns, but am holding off for now\n> because it's not clear to me if you're already planning to discard any\n> such re-roll in favor of a revert.\n>\n> Or do you mean to create a point release with such revert(s) and have\n> master free to move forward with a fix for the outstanding issue, but\n> not to use that for a point release?\n\nIf a maintenance release will have reverts with adjustment, then the\nsolution that will be only merged to master should still be built on\ntop. So if we were to go the route above, the early part (the first\nthree that are reverts above, and possibly a couple more directly on\ntop just to address \"did we really run hook?\") would be merged to the\nmaintenance track, while the whole thing that rebuilds on top of the\nreverted one would be merged to 'master', I would imagine.\n\nIt all depends on how involved it is to get to where we want to be,\nbetween\n\n (1) starting from 'master' and working backwards, removing the use of\n the run_parallel stuff and replacing it with the run_command API, or\n\n (2) bringing us back to pre-c70bc338 state first and then building\n up what we would have built if we didn't use run_parallel stuff\n in the original series.\n\nAs you were saying that what you would produce with the former\napproach would be, compared to the initial \"regress fix\" that still\nused the run_parallel stuff, a large and unreviewable mess, I was\nthrowing out a different approach as a potential alternative, with\nthe hope that the resulting series may make it reviewable, as long\nas the early \"straight revert\" part is straight-forward.\n\nIf we take the \"start from 'master' and fix minimally\" approach, the\nwhole thing would be both in the maintenance track and in the track\nfor the next release, I would imagine.\n\nSo, in short, either way, we would not run hooks in parallel, and we\nwould not run hooks with run_parallel API castrated to a single\nprocess usage only, in the version we will merge to the maintenance\ntrack and also to the master track. The latter may get an update to\nre-attempt reusing run_parallel API in a way that is less hostile to\nexisting users, but I do not think we should make users wait by\nspending more time on it than necessary right now, before we get the\nregression fix ready.\n\nThanks.\n"},{"id":"456269","messageId":"patch-v3-1.2-aabd99de680-20220527T090618Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v3-0.2-00000000000-20220527T090618Z-avarab@gmail.com","subject":"[PATCH v3 1/2] run-command: add an \"ungroup\" option to run_process_parallel()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-27T09:14:30Z","receivedAt":"2022-05-27T09:14:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Extend the parallel execution API added in c553c72eed6 (run-command:\nadd an asynchronous parallel child processor, 2015-12-15) to support a\nmode where the stdout and stderr of the processes isn't captured and\noutput in a deterministic order, instead we'll leave it to the kernel\nand stdio to sort it out.\n\nThis gives the API same functionality as GNU parallel's --ungroup\noption. As we'll see in a subsequent commit the main reason to want\nthis is to support stdout and stderr being connected to the TTY in the\ncase of jobs=1, demonstrated here with GNU parallel:\n\n\t$ parallel --ungroup 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n\tTTY\n\tTTY\n\t$ parallel 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n\tNTTY\n\tNTTY\n\nAnother is as GNU parallel's documentation notes a potential for\noptimization. Our results will be a bit different, but in cases where\nyou want to run processes in parallel where the exact order isn't\nimportant this can be a lot faster:\n\n\t$ hyperfine -r 3 -L o ,--ungroup 'parallel {o} seq ::: 10000000 >/dev/null '\n\tBenchmark 1: parallel seq ::: 10000000 >/dev/null\n\t Time (mean ± σ): 220.2 ms ± 9.3 ms [User: 124.9 ms, System: 96.1 ms]\n\t Range (min … max): 212.3 ms … 230.5 ms 3 runs\n\n\tBenchmark 2: parallel --ungroup seq ::: 10000000 >/dev/null\n\t Time (mean ± σ): 154.7 ms ± 0.9 ms [User: 136.2 ms, System: 25.1 ms]\n\t Range (min … max): 153.9 ms … 155.7 ms 3 runs\n\n\tSummary\n\t 'parallel --ungroup seq ::: 10000000 >/dev/null ' ran\n\t 1.42 ± 0.06 times faster than 'parallel seq ::: 10000000 >/dev/null '\n\nA large part of the juggling in the API is to make the API safer for\nits maintenance and consumers alike.\n\nFor the maintenance of the API we e.g. avoid malloc()-ing the\n\"pp->pfd\", ensuring that SANITIZE=address and other similar tools will\ncatch any unexpected misuse.\n\nFor API consumers we take pains to never pass the non-NULL \"out\"\nbuffer to an API user that provided the \"ungroup\" option. The\nresulting code in t/helper/test-run-command.c isn't typical of such a\nuser, i.e. they'd typically use one mode or the other, and would know\nwhether they'd provided \"ungroup\" or not.\n\nWe could also avoid the strbuf_init() for \"buffered_output\" by having\n\"struct parallel_processes\" use a static PARALLEL_PROCESSES_INIT\ninitializer, but let's leave that cleanup for later.\n\nUsing a global \"run_processes_parallel_ungroup\" variable to enable\nthis option is rather nasty, but is being done here to produce as\nminimal of a change as possible for a subsequent regression fix. This\nchange is extracted from a larger initial version[1] which ends up\nwith a better end-state for the API, but in doing so needed to modify\nall existing callers of the API. Let's defer that for now, and\nnarrowly focus on what we need for fixing the regression in the\nsubsequent commit.\n\nIt's safe to do this with a global variable because:\n\n A) hook.c is the only user of it that sets it to non-zero, and before\n we'll get any other API users we'll refactor away this method of\n passing in the option, i.e. re-roll [1].\n\n B) Even if hook.c wasn't the only user we don't have callers of this\n API that concurrently invoke this parallel process starting API\n itself in parallel.\n\nAs noted above \"A\" && \"B\" are rather nasty, and we don't want to live\nwith those caveats long-term, but for now they should be an acceptable\ncompromise.\n\n1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 88 ++++++++++++++++++++++++++++---------\n run-command.h | 31 ++++++++++---\n t/helper/test-run-command.c | 19 ++++++--\n t/t0061-run-command.sh | 35 +++++++++++++++\n 4 files changed, 143 insertions(+), 30 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex a8501e38ceb..b5ede8655d3 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1468,9 +1468,10 @@ int pipe_command(struct child_process *cmd,\n enum child_state {\n \tGIT_CP_FREE,\n \tGIT_CP_WORKING,\n-\tGIT_CP_WAIT_CLEANUP,\n+\tGIT_CP_WAIT_CLEANUP, /* only for !ungroup */\n };\n \n+int run_processes_parallel_ungroup;\n struct parallel_processes {\n \tvoid *data;\n \n@@ -1494,6 +1495,7 @@ struct parallel_processes {\n \tstruct pollfd *pfd;\n \n \tunsigned shutdown : 1;\n+\tunsigned ungroup : 1;\n \n \tint output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n@@ -1537,7 +1539,7 @@ static void pp_init(struct parallel_processes *pp,\n \t\t get_next_task_fn get_next_task,\n \t\t start_failure_fn start_failure,\n \t\t task_finished_fn task_finished,\n-\t\t void *data)\n+\t\t void *data, const int ungroup)\n {\n \tint i;\n \n@@ -1559,13 +1561,19 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->nr_processes = 0;\n \tpp->output_owner = 0;\n \tpp->shutdown = 0;\n+\tpp->ungroup = ungroup;\n \tCALLOC_ARRAY(pp->children, n);\n-\tCALLOC_ARRAY(pp->pfd, n);\n+\tif (pp->ungroup)\n+\t\tpp->pfd = NULL;\n+\telse\n+\t\tCALLOC_ARRAY(pp->pfd, n);\n \tstrbuf_init(&pp->buffered_output, 0);\n \n \tfor (i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n \t\tchild_process_init(&pp->children[i].process);\n+\t\tif (!pp->pfd)\n+\t\t\tcontinue;\n \t\tpp->pfd[i].events = POLLIN | POLLHUP;\n \t\tpp->pfd[i].fd = -1;\n \t}\n@@ -1591,7 +1599,8 @@ static void pp_cleanup(struct parallel_processes *pp)\n \t * When get_next_task added messages to the buffer in its last\n \t * iteration, the buffered output is non empty.\n \t */\n-\tstrbuf_write(&pp->buffered_output, stderr);\n+\tif (!pp->ungroup)\n+\t\tstrbuf_write(&pp->buffered_output, stderr);\n \tstrbuf_release(&pp->buffered_output);\n \n \tsigchain_pop_common();\n@@ -1606,6 +1615,7 @@ static void pp_cleanup(struct parallel_processes *pp)\n */\n static int pp_start_one(struct parallel_processes *pp)\n {\n+\tconst int ungroup = pp->ungroup;\n \tint i, code;\n \n \tfor (i = 0; i < pp->max_processes; i++)\n@@ -1615,24 +1625,30 @@ static int pp_start_one(struct parallel_processes *pp)\n \t\tBUG(\"bookkeeping is hard\");\n \n \tcode = pp->get_next_task(&pp->children[i].process,\n-\t\t\t\t &pp->children[i].err,\n+\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n \t\t\t\t pp->data,\n \t\t\t\t &pp->children[i].data);\n \tif (!code) {\n-\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n-\t\tstrbuf_reset(&pp->children[i].err);\n+\t\tif (!ungroup) {\n+\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n+\t\t\tstrbuf_reset(&pp->children[i].err);\n+\t\t}\n \t\treturn 1;\n \t}\n-\tpp->children[i].process.err = -1;\n-\tpp->children[i].process.stdout_to_stderr = 1;\n+\tif (!ungroup) {\n+\t\tpp->children[i].process.err = -1;\n+\t\tpp->children[i].process.stdout_to_stderr = 1;\n+\t}\n \tpp->children[i].process.no_stdin = 1;\n \n \tif (start_command(&pp->children[i].process)) {\n-\t\tcode = pp->start_failure(&pp->children[i].err,\n+\t\tcode = pp->start_failure(ungroup ? NULL : &pp->children[i].err,\n \t\t\t\t\t pp->data,\n \t\t\t\t\t pp->children[i].data);\n-\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n-\t\tstrbuf_reset(&pp->children[i].err);\n+\t\tif (!ungroup) {\n+\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n+\t\t\tstrbuf_reset(&pp->children[i].err);\n+\t\t}\n \t\tif (code)\n \t\t\tpp->shutdown = 1;\n \t\treturn code;\n@@ -1640,14 +1656,26 @@ static int pp_start_one(struct parallel_processes *pp)\n \n \tpp->nr_processes++;\n \tpp->children[i].state = GIT_CP_WORKING;\n-\tpp->pfd[i].fd = pp->children[i].process.err;\n+\tif (pp->pfd)\n+\t\tpp->pfd[i].fd = pp->children[i].process.err;\n \treturn 0;\n }\n \n+static void pp_mark_working_for_cleanup(struct parallel_processes *pp)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < pp->max_processes; i++)\n+\t\tif (pp->children[i].state == GIT_CP_WORKING)\n+\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+}\n+\n static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n {\n \tint i;\n \n+\tassert(!pp->ungroup);\n+\n \twhile ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n \t\tif (errno == EINTR)\n \t\t\tcontinue;\n@@ -1674,6 +1702,9 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n static void pp_output(struct parallel_processes *pp)\n {\n \tint i = pp->output_owner;\n+\n+\tassert(!pp->ungroup);\n+\n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n \t\tstrbuf_write(&pp->children[i].err, stderr);\n@@ -1683,10 +1714,15 @@ static void pp_output(struct parallel_processes *pp)\n \n static int pp_collect_finished(struct parallel_processes *pp)\n {\n+\tconst int ungroup = pp->ungroup;\n \tint i, code;\n \tint n = pp->max_processes;\n \tint result = 0;\n \n+\tif (ungroup)\n+\t\tfor (i = 0; i < pp->max_processes; i++)\n+\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+\n \twhile (pp->nr_processes > 0) {\n \t\tfor (i = 0; i < pp->max_processes; i++)\n \t\t\tif (pp->children[i].state == GIT_CP_WAIT_CLEANUP)\n@@ -1697,8 +1733,8 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \t\tcode = finish_command(&pp->children[i].process);\n \n \t\tcode = pp->task_finished(code,\n-\t\t\t\t\t &pp->children[i].err, pp->data,\n-\t\t\t\t\t pp->children[i].data);\n+\t\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n+\t\t\t\t\t pp->data, pp->children[i].data);\n \n \t\tif (code)\n \t\t\tresult = code;\n@@ -1707,10 +1743,13 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \n \t\tpp->nr_processes--;\n \t\tpp->children[i].state = GIT_CP_FREE;\n-\t\tpp->pfd[i].fd = -1;\n+\t\tif (pp->pfd)\n+\t\t\tpp->pfd[i].fd = -1;\n \t\tchild_process_init(&pp->children[i].process);\n \n-\t\tif (i != pp->output_owner) {\n+\t\tif (ungroup) {\n+\t\t\t; /* no strbuf_*() work to do here */\n+\t\t} else if (i != pp->output_owner) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t} else {\n@@ -1748,8 +1787,13 @@ int run_processes_parallel(int n,\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n \tstruct parallel_processes pp;\n+\tconst int ungroup = run_processes_parallel_ungroup;\n \n-\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n+\t/* unset for the next API user */\n+\trun_processes_parallel_ungroup = 0;\n+\n+\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n+\t\tungroup);\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1766,8 +1810,12 @@ int run_processes_parallel(int n,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tpp_buffer_stderr(&pp, output_timeout);\n-\t\tpp_output(&pp);\n+\t\tif (ungroup) {\n+\t\t\tpp_mark_working_for_cleanup(&pp);\n+\t\t} else {\n+\t\t\tpp_buffer_stderr(&pp, output_timeout);\n+\t\t\tpp_output(&pp);\n+\t\t}\n \t\tcode = pp_collect_finished(&pp);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\ndiff --git a/run-command.h b/run-command.h\nindex 5bd0c933e80..a44d2a6ba75 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -405,6 +405,10 @@ void check_pipe(int err);\n * pp_cb is the callback cookie as passed to run_processes_parallel.\n * You can store a child process specific callback cookie in pp_task_cb.\n *\n+ * The \"struct strbuf *err\" parameter is either a pointer to a string\n+ * to write errors to, or NULL if the \"ungroup\" option was\n+ * provided. See run_processes_parallel() below.\n+ *\n * Even after returning 0 to indicate that there are no more processes,\n * this function will be called again until there are no more running\n * child processes.\n@@ -423,9 +427,9 @@ typedef int (*get_next_task_fn)(struct child_process *cp,\n * This callback is called whenever there are problems starting\n * a new process.\n *\n- * You must not write to stdout or stderr in this function. Add your\n- * message to the strbuf out instead, which will be printed without\n- * messing up the output of the other parallel processes.\n+ * The \"struct strbuf *err\" parameter is either a pointer to a string\n+ * to write errors to, or NULL if the \"ungroup\" option was\n+ * provided. See run_processes_parallel() below.\n *\n * pp_cb is the callback cookie as passed into run_processes_parallel,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n@@ -441,9 +445,9 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n /**\n * This callback is called on every child process that finished processing.\n *\n- * You must not write to stdout or stderr in this function. Add your\n- * message to the strbuf out instead, which will be printed without\n- * messing up the output of the other parallel processes.\n+ * The \"struct strbuf *err\" parameter is either a pointer to a string\n+ * to write errors to, or NULL if the \"ungroup\" option was\n+ * provided. See run_processes_parallel() below.\n *\n * pp_cb is the callback cookie as passed into run_processes_parallel,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n@@ -464,11 +468,24 @@ typedef int (*task_finished_fn)(int result,\n *\n * The children started via this function run in parallel. Their output\n * (both stdout and stderr) is routed to stderr in a manner that output\n- * from different tasks does not interleave.\n+ * from different tasks does not interleave (but see \"ungroup\" above).\n *\n * start_failure_fn and task_finished_fn can be NULL to omit any\n * special handling.\n+ *\n+ * If the \"ungroup\" option isn't specified the callbacks will get a\n+ * pointer to a \"struct strbuf *out\", and must not write to stdout or\n+ * stderr as such output will mess up the output of the other parallel\n+ * processes. If \"ungroup\" option is specified callbacks will get a\n+ * NULL \"struct strbuf *out\" parameter, and are responsible for\n+ * emitting their own output, including dealing with any race\n+ * conditions due to writing in parallel to stdout and stderr.\n+ * The \"ungroup\" option can be enabled by setting the global\n+ * \"run_processes_parallel_ungroup\" to \"1\" before invoking\n+ * run_processes_parallel(), it will be set back to \"0\" as soon as the\n+ * API reads that setting.\n */\n+extern int run_processes_parallel_ungroup;\n int run_processes_parallel(int n,\n \t\t\t get_next_task_fn,\n \t\t\t start_failure_fn,\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex f3b90aa834a..6405c9a076a 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -31,7 +31,11 @@ static int parallel_next(struct child_process *cp,\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->args.v);\n-\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n+\telse\n+\t\tfprintf(stderr, \"preloaded output of a child\\n\");\n+\n \tnumber_callbacks++;\n \treturn 1;\n }\n@@ -41,7 +45,10 @@ static int no_job(struct child_process *cp,\n \t\t void *cb,\n \t\t void **task_cb)\n {\n-\tstrbuf_addstr(err, \"no further jobs available\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"no further jobs available\\n\");\n+\telse\n+\t\tfprintf(stderr, \"no further jobs available\\n\");\n \treturn 0;\n }\n \n@@ -50,7 +57,10 @@ static int task_finished(int result,\n \t\t\t void *pp_cb,\n \t\t\t void *pp_task_cb)\n {\n-\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n+\telse\n+\t\tfprintf(stderr, \"asking for a quick stop\\n\");\n \treturn 1;\n }\n \n@@ -411,6 +421,9 @@ int cmd__run_command(int argc, const char **argv)\n \tstrvec_clear(&proc.args);\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n+\tif (getenv(\"RUN_PROCESSES_PARALLEL_UNGROUP\"))\n+\t\trun_processes_parallel_ungroup = 1;\n+\n \tif (!strcmp(argv[1], \"run-command-parallel\"))\n \t\texit(run_processes_parallel(jobs, parallel_next,\n \t\t\t\t\t NULL, NULL, &proc));\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex ee281909bc3..69ccaa8d298 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -134,16 +134,37 @@ test_expect_success 'run_command runs in parallel with more jobs available than\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n+\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n+\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n \ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n+\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n+\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n \ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n+\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n+\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n@@ -158,6 +179,13 @@ test_expect_success 'run_command is asked to abort gracefully' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n+\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n+\ttest-tool run-command run-command-abort 3 false >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_line_count = 6 err\n+'\n+\n cat >expect <<-EOF\n no further jobs available\n EOF\n@@ -167,6 +195,13 @@ test_expect_success 'run_command outputs ' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command outputs (ungroup) ' '\n+\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n+\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n test_trace () {\n \texpect=\"$1\"\n \tshift\n-- \n2.36.1.1046.g586767a6996\n\n"},{"id":"456270","messageId":"patch-v3-2.2-ec27e3906e1-20220527T090618Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v3-0.2-00000000000-20220527T090618Z-avarab@gmail.com","subject":"[PATCH v3 2/2] hook API: fix v2.36.0 regression: hooks should be connected to a TTY","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-27T09:14:31Z","receivedAt":"2022-05-27T09:14:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a regression reported[1] in f443246b9f2 (commit: convert\n{pre-commit,prepare-commit-msg} hook to hook.h, 2021-12-22): Due to\nusing the run_process_parallel() API in the earlier 96e7225b310 (hook:\nadd 'run' subcommand, 2021-12-22) we'd capture the hook's stderr and\nstdout, and thus lose the connection to the TTY in the case of\ne.g. the \"pre-commit\" hook.\n\nAs a preceding commit notes GNU parallel's similar --ungroup option\nalso has it emit output faster. While we're unlikely to have hooks\nthat emit truly massive amounts of output (or where the performance\nthereof matters) it's still informative to measure the overhead. In a\nsimilar \"seq\" test we're now ~30% faster:\n\n\t$ cat .git/hooks/seq-hook; git hyperfine -L rev origin/master,HEAD~0 -s 'make CFLAGS=-O3' './git hook run seq-hook'\n\t#!/bin/sh\n\n\tseq 100000000\n\tBenchmark 1: ./git hook run seq-hook' in 'origin/master\n\t Time (mean ± σ): 787.1 ms ± 13.6 ms [User: 701.6 ms, System: 534.4 ms]\n\t Range (min … max): 773.2 ms … 806.3 ms 10 runs\n\n\tBenchmark 2: ./git hook run seq-hook' in 'HEAD~0\n\t Time (mean ± σ): 603.4 ms ± 1.6 ms [User: 573.1 ms, System: 30.3 ms]\n\t Range (min … max): 601.0 ms … 606.2 ms 10 runs\n\n\tSummary\n\t './git hook run seq-hook' in 'HEAD~0' ran\n\t 1.30 ± 0.02 times faster than './git hook run seq-hook' in 'origin/master'\n\n1. https://lore.kernel.org/git/CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com/\n\nReported-by: Anthony Sottile \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 1 +\n t/t1800-hook.sh | 37 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 38 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex 1d51be3b77a..7451205657a 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -144,6 +144,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \t\tcb_data.hook_path = abs_path.buf;\n \t}\n \n+\trun_processes_parallel_ungroup = 1;\n \trun_processes_parallel_tr2(jobs,\n \t\t\t\t pick_next_hook,\n \t\t\t\t notify_start_failure,\ndiff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\nindex 26ed5e11bc8..0b8370d1573 100755\n--- a/t/t1800-hook.sh\n+++ b/t/t1800-hook.sh\n@@ -4,6 +4,7 @@ test_description='git-hook command'\n \n TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-terminal.sh\n \n test_expect_success 'git hook usage' '\n \ttest_expect_code 129 git hook &&\n@@ -120,4 +121,40 @@ test_expect_success 'git -c core.hooksPath= hook run' '\n \ttest_cmp expect actual\n '\n \n+test_hook_tty() {\n+\tlocal fd=\"$1\" &&\n+\n+\tcat >expect &&\n+\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\n+\ttest_hook -C repo pre-commit <<-EOF &&\n+\t{\n+\t\ttest -t 1 && echo >&$fd STDOUT TTY || echo >&$fd STDOUT NO TTY &&\n+\t\ttest -t 2 && echo >&$fd STDERR TTY || echo >&$fd STDERR NO TTY\n+\t} $fd>actual\n+\tEOF\n+\n+\ttest_commit -C repo A &&\n+\ttest_commit -C repo B &&\n+\tgit -C repo reset --soft HEAD^ &&\n+\ttest_terminal git -C repo commit -m\"B.new\" &&\n+\ttest_cmp expect repo/actual\n+}\n+\n+test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDOUT redirect' '\n+\ttest_hook_tty 1 <<-\\EOF\n+\tSTDOUT NO TTY\n+\tSTDERR TTY\n+\tEOF\n+'\n+\n+test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDERR redirect' '\n+\ttest_hook_tty 2 <<-\\EOF\n+\tSTDOUT TTY\n+\tSTDERR NO TTY\n+\tEOF\n+'\n+\n test_done\n-- \n2.36.1.1046.g586767a6996\n\n"},{"id":"456271","messageId":"cover-v3-0.2-00000000000-20220527T090618Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com","subject":"[PATCH v3 0/2] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-27T09:14:29Z","receivedAt":"2022-05-27T09:14:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A re-roll of [1] which aims to address the concerns about the previous\n8-part series being too large to fix a release regression. \"If it\nisn't bolted down, throw it overboard!\".\n\nThe main change here is:\n\n * The new \"ungroup\" parameter is now passed via an \"extern\" parameter.\n * Tests for existing run-command.c behavior (not narrowly needed for\n the regression fix) are gone.\n * Adding an INIT macro is gone, instead we explicitly initialize to NULL.\n * Stray bugfix for existing hook test is gone.\n\netc. I think all of those still make sense, but they're something I\ncan rebase on this topic once it (hopefully) lands. In the meantime\nthe updated commit messages for the remaining two (see start of the\nrange-diff below) argue for this being a a safe API change, even if\nthe interface is a bit nasty.\n\n1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com\n\nÆvar Arnfjörð Bjarmason (2):\n run-command: add an \"ungroup\" option to run_process_parallel()\n hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n\n hook.c | 1 +\n run-command.c | 88 ++++++++++++++++++++++++++++---------\n run-command.h | 31 ++++++++++---\n t/helper/test-run-command.c | 19 ++++++--\n t/t0061-run-command.sh | 35 +++++++++++++++\n t/t1800-hook.sh | 37 ++++++++++++++++\n 6 files changed, 181 insertions(+), 30 deletions(-)\n\nRange-diff against v2:\n1: 26a81eff267 < -: ----------- run-command tests: change if/if/... to if/else if/else\n2: 5f0a6e9925f < -: ----------- run-command API: use \"opts\" struct for run_processes_parallel{,_tr2}()\n3: a8e1fc07b65 < -: ----------- run-command tests: test stdout of run_command_parallel()\n4: 663936fb4ad < -: ----------- run-command.c: add an initializer for \"struct parallel_processes\"\n5: c2e015ed840 ! 1: aabd99de680 run-command: add an \"ungroup\" option to run_process_parallel()\n @@ Commit message\n user, i.e. they'd typically use one mode or the other, and would know\n whether they'd provided \"ungroup\" or not.\n \n + We could also avoid the strbuf_init() for \"buffered_output\" by having\n + \"struct parallel_processes\" use a static PARALLEL_PROCESSES_INIT\n + initializer, but let's leave that cleanup for later.\n +\n + Using a global \"run_processes_parallel_ungroup\" variable to enable\n + this option is rather nasty, but is being done here to produce as\n + minimal of a change as possible for a subsequent regression fix. This\n + change is extracted from a larger initial version[1] which ends up\n + with a better end-state for the API, but in doing so needed to modify\n + all existing callers of the API. Let's defer that for now, and\n + narrowly focus on what we need for fixing the regression in the\n + subsequent commit.\n +\n + It's safe to do this with a global variable because:\n +\n + A) hook.c is the only user of it that sets it to non-zero, and before\n + we'll get any other API users we'll refactor away this method of\n + passing in the option, i.e. re-roll [1].\n +\n + B) Even if hook.c wasn't the only user we don't have callers of this\n + API that concurrently invoke this parallel process starting API\n + itself in parallel.\n +\n + As noted above \"A\" && \"B\" are rather nasty, and we don't want to live\n + with those caveats long-term, but for now they should be an acceptable\n + compromise.\n +\n + 1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n +\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## run-command.c ##\n @@ run-command.c: int pipe_command(struct child_process *cmd,\n +\tGIT_CP_WAIT_CLEANUP, /* only for !ungroup */\n };\n \n ++int run_processes_parallel_ungroup;\n struct parallel_processes {\n + \tvoid *data;\n + \n @@ run-command.c: struct parallel_processes {\n \tstruct pollfd *pfd;\n \n @@ run-command.c: struct parallel_processes {\n \n \tint output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n +@@ run-command.c: static void pp_init(struct parallel_processes *pp,\n + \t\t get_next_task_fn get_next_task,\n + \t\t start_failure_fn start_failure,\n + \t\t task_finished_fn task_finished,\n +-\t\t void *data)\n ++\t\t void *data, const int ungroup)\n + {\n + \tint i;\n + \n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n \tpp->nr_processes = 0;\n \tpp->output_owner = 0;\n \tpp->shutdown = 0;\n -+\tpp->ungroup = opts->ungroup;\n ++\tpp->ungroup = ungroup;\n \tCALLOC_ARRAY(pp->children, n);\n -\tCALLOC_ARRAY(pp->pfd, n);\n -+\tif (!pp->ungroup)\n ++\tif (pp->ungroup)\n ++\t\tpp->pfd = NULL;\n ++\telse\n +\t\tCALLOC_ARRAY(pp->pfd, n);\n + \tstrbuf_init(&pp->buffered_output, 0);\n \n \tfor (i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t} else {\n -@@ run-command.c: int run_processes_parallel(struct run_process_parallel_opts *opts)\n +@@ run-command.c: int run_processes_parallel(int n,\n + \tint output_timeout = 100;\n + \tint spawn_cap = 4;\n + \tstruct parallel_processes pp;\n ++\tconst int ungroup = run_processes_parallel_ungroup;\n + \n +-\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n ++\t/* unset for the next API user */\n ++\trun_processes_parallel_ungroup = 0;\n ++\n ++\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n ++\t\tungroup);\n + \twhile (1) {\n + \t\tfor (i = 0;\n + \t\t i < spawn_cap && !pp.shutdown &&\n +@@ run-command.c: int run_processes_parallel(int n,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n -\t\tpp_buffer_stderr(&pp, output_timeout);\n -\t\tpp_output(&pp);\n -+\t\tif (opts->ungroup) {\n ++\t\tif (ungroup) {\n +\t\t\tpp_mark_working_for_cleanup(&pp);\n +\t\t} else {\n +\t\t\tpp_buffer_stderr(&pp, output_timeout);\n @@ run-command.h: typedef int (*start_failure_fn)(struct strbuf *out,\n * pp_cb is the callback cookie as passed into run_processes_parallel,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n @@ run-command.h: typedef int (*task_finished_fn)(int result,\n - *\n - * jobs: see 'n' in run_processes_parallel() below.\n - *\n -+ * ungroup: Ungroup output. Output is printed as soon as possible and\n -+ * bypasses run-command's internal processing. This may cause output\n -+ * from different commands to be mixed.\n -+ *\n - * *_fn & data: see run_processes_parallel() below.\n - */\n - struct run_process_parallel_opts\n -@@ run-command.h: struct run_process_parallel_opts\n - \tconst char *tr2_label;\n - \n - \tint jobs;\n -+\tunsigned int ungroup:1;\n - \n - \tget_next_task_fn get_next_task;\n - \tstart_failure_fn start_failure;\n -@@ run-command.h: struct run_process_parallel_opts\n *\n * The children started via this function run in parallel. Their output\n * (both stdout and stderr) is routed to stderr in a manner that output\n @@ run-command.h: struct run_process_parallel_opts\n + * NULL \"struct strbuf *out\" parameter, and are responsible for\n + * emitting their own output, including dealing with any race\n + * conditions due to writing in parallel to stdout and stderr.\n ++ * The \"ungroup\" option can be enabled by setting the global\n ++ * \"run_processes_parallel_ungroup\" to \"1\" before invoking\n ++ * run_processes_parallel(), it will be set back to \"0\" as soon as the\n ++ * API reads that setting.\n */\n - int run_processes_parallel(struct run_process_parallel_opts *opts);\n - \n ++extern int run_processes_parallel_ungroup;\n + int run_processes_parallel(int n,\n + \t\t\t get_next_task_fn,\n + \t\t\t start_failure_fn,\n \n ## t/helper/test-run-command.c ##\n @@ t/helper/test-run-command.c: static int parallel_next(struct child_process *cp,\n @@ t/helper/test-run-command.c: static int task_finished(int result,\n }\n \n @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)\n - \topts.jobs = jobs;\n - \topts.data = &proc;\n + \tstrvec_clear(&proc.args);\n + \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n --\tif (!strcmp(argv[1], \"run-command-parallel\")) {\n -+\tif (!strcmp(argv[1], \"run-command-parallel\") ||\n -+\t !strcmp(argv[1], \"run-command-parallel-ungroup\")) {\n - \t\tnext_fn = parallel_next;\n --\t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n -+\t} else if (!strcmp(argv[1], \"run-command-abort\") ||\n -+\t\t !strcmp(argv[1], \"run-command-abort-ungroup\")) {\n - \t\tnext_fn = parallel_next;\n - \t\tfinished_fn = task_finished;\n --\t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n -+\t} else if (!strcmp(argv[1], \"run-command-no-jobs\") ||\n -+\t\t !strcmp(argv[1], \"run-command-no-jobs-ungroup\")) {\n - \t\tnext_fn = no_job;\n - \t\tfinished_fn = task_finished;\n - \t} else {\n -@@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)\n - \t\treturn 1;\n - \t}\n - \n -+\topts.ungroup = ends_with(argv[1], \"-ungroup\");\n - \topts.get_next_task = next_fn;\n - \topts.task_finished = finished_fn;\n - \texit(run_processes_parallel(&opts));\n ++\tif (getenv(\"RUN_PROCESSES_PARALLEL_UNGROUP\"))\n ++\t\trun_processes_parallel_ungroup = 1;\n ++\n + \tif (!strcmp(argv[1], \"run-command-parallel\"))\n + \t\texit(run_processes_parallel(jobs, parallel_next,\n + \t\t\t\t\t NULL, NULL, &proc));\n \n ## t/t0061-run-command.sh ##\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with more jobs available than\n - \ttest_cmp expect err\n + \ttest_cmp expect actual\n '\n \n +test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n -+\ttest-tool run-command run-command-parallel-ungroup 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n ++\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n ++\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_line_count = 8 out &&\n +\ttest_line_count = 4 err\n +'\n +\n test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n - \ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n - \ttest_must_be_empty out &&\n - \ttest_cmp expect err\n + \ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n + \ttest_cmp expect actual\n '\n \n +test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n -+\ttest-tool run-command run-command-parallel-ungroup 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n ++\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n ++\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_line_count = 8 out &&\n +\ttest_line_count = 4 err\n +'\n +\n test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n - \ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n - \ttest_must_be_empty out &&\n - \ttest_cmp expect err\n + \ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n + \ttest_cmp expect actual\n '\n \n +test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n -+\ttest-tool run-command run-command-parallel-ungroup 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n ++\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n ++\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_line_count = 8 out &&\n +\ttest_line_count = 4 err\n +'\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with m\n preloaded output of a child\n asking for a quick stop\n @@ t/t0061-run-command.sh: test_expect_success 'run_command is asked to abort gracefully' '\n - \ttest_cmp expect err\n + \ttest_cmp expect actual\n '\n \n +test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n -+\ttest-tool run-command run-command-abort-ungroup 3 false >out 2>err &&\n ++\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n ++\ttest-tool run-command run-command-abort 3 false >out 2>err &&\n +\ttest_must_be_empty out &&\n +\ttest_line_count = 6 err\n +'\n @@ t/t0061-run-command.sh: test_expect_success 'run_command is asked to abort grace\n no further jobs available\n EOF\n @@ t/t0061-run-command.sh: test_expect_success 'run_command outputs ' '\n - \ttest_cmp expect err\n + \ttest_cmp expect actual\n '\n \n +test_expect_success 'run_command outputs (ungroup) ' '\n -+\ttest-tool run-command run-command-no-jobs-ungroup 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n ++\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n ++\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_must_be_empty out &&\n +\ttest_cmp expect err\n +'\n6: 84e92c6f7c7 < -: ----------- hook tests: fix redirection logic error in 96e7225b310\n7: bf7d871565f < -: ----------- hook API: don't redundantly re-set \"no_stdin\" and \"stdout_to_stderr\"\n8: 238155fcb9d ! 2: ec27e3906e1 hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n @@ Commit message\n './git hook run seq-hook' in 'HEAD~0' ran\n 1.30 ± 0.02 times faster than './git hook run seq-hook' in 'origin/master'\n \n - In the preceding commit we removed the \"stdout_to_stderr=1\" assignment\n - as being redundant. This change brings it back as with \".ungroup=1\"\n - the run_process_parallel() function doesn't provide them for us\n - implicitly.\n -\n - As an aside omitting the stdout_to_stderr=1 here would have all tests\n - pass, except those that test \"git hook run\" itself in\n - t1800-hook.sh. But our tests passing is the result of another test\n - blind spot, as was the case with the regression being fixed here. The\n - \"stdout_to_stderr=1\" for hooks is long-standing behavior, see\n - e.g. 1d9e8b56fe3 (Split back out update_hook handling in receive-pack,\n - 2007-03-10) and other follow-up commits (running \"git log\" with\n - \"--reverse -p -Gstdout_to_stderr\" is a good start).\n -\n 1. https://lore.kernel.org/git/CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com/\n \n Reported-by: Anthony Sottile \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## hook.c ##\n -@@ hook.c: static int pick_next_hook(struct child_process *cp,\n - \t\treturn 0;\n - \n - \tstrvec_pushv(&cp->env_array, hook_cb->options->env.v);\n -+\tcp->stdout_to_stderr = 1; /* because of .ungroup = 1 */\n - \tcp->trace2_hook_name = hook_cb->hook_name;\n - \tcp->dir = hook_cb->options->dir;\n - \n @@ hook.c: int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n - \t\t.tr2_label = hook_name,\n - \n - \t\t.jobs = jobs,\n -+\t\t.ungroup = jobs == 1,\n - \n - \t\t.get_next_task = pick_next_hook,\n - \t\t.start_failure = notify_start_failure,\n -@@ hook.c: int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n - \tif (!options)\n - \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n - \n -+\tif (jobs != 1 || !run_opts.ungroup)\n -+\t\tBUG(\"TODO: think about & document order & interleaving of parallel hook output\");\n -+\n - \tif (options->invoked_hook)\n - \t\t*options->invoked_hook = 0;\n + \t\tcb_data.hook_path = abs_path.buf;\n + \t}\n \n ++\trun_processes_parallel_ungroup = 1;\n + \trun_processes_parallel_tr2(jobs,\n + \t\t\t\t pick_next_hook,\n + \t\t\t\t notify_start_failure,\n \n ## t/t1800-hook.sh ##\n @@ t/t1800-hook.sh: test_description='git-hook command'\n-- \n2.36.1.1046.g586767a6996\n\n"},{"id":"456300","messageId":"xmqqtu9b5576.fsf@gitster.g","threadId":"57764","inReplyTo":"Yo+2cbMueQyAI186@google.com","subject":"Re: [PATCH v2 5/8] run-command: add an \"ungroup\" option to run_process_parallel()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-27T16:08:13Z","receivedAt":"2022-05-27T16:08:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Emily Shaffer writes:\n\n> I think we actually could even automatically set ungroup if jobs=1 as\n> well, because then there is no reason to buffer the output - it uses\n> additional memory for us, and it makes output slower to see for the end\n> user. But I do not really mind enough to want a reroll.\n\nNot doing so would protect us from future end-user complaints,\nsimilar to the way that made us consider the change in 2.36 to be a\nregression. Those who are used to see their stuff run in submodules\n(which I recall was the original purpose of run_processes_parallel\nwas invented for) with their standard output and error streams not\ndirectly connected to the original end-user terminal will start seeing\nthe expectation broken but only when there is only one submodule, no?\n\nDoing it when an explicit \"ungroup\" was called for would hopefully\navoid such an inconsistent behaviour.\n"},{"id":"456301","messageId":"xmqqee0e52ur.fsf@gitster.g","threadId":"57764","inReplyTo":"patch-v3-1.2-aabd99de680-20220527T090618Z-avarab@gmail.com","subject":"Re: [PATCH v3 1/2] run-command: add an \"ungroup\" option to run_process_parallel()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-27T16:58:52Z","receivedAt":"2022-05-27T16:59:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> diff --git a/run-command.c b/run-command.c\n> index a8501e38ceb..b5ede8655d3 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1468,9 +1468,10 @@ int pipe_command(struct child_process *cmd,\n> enum child_state {\n> \tGIT_CP_FREE,\n> \tGIT_CP_WORKING,\n> -\tGIT_CP_WAIT_CLEANUP,\n> +\tGIT_CP_WAIT_CLEANUP, /* only for !ungroup */\n> };\n> \n> +int run_processes_parallel_ungroup;\n\nA few comments on these below.\n\n> @@ -1537,7 +1539,7 @@ static void pp_init(struct parallel_processes *pp,\n> \t\t get_next_task_fn get_next_task,\n> \t\t start_failure_fn start_failure,\n> \t\t task_finished_fn task_finished,\n> -\t\t void *data)\n> +\t\t void *data, const int ungroup)\n\nIt is unusual in this codebase to pass \"const\" non-pointer as a\nparameter, but OK.\n\n> @@ -1591,7 +1599,8 @@ static void pp_cleanup(struct parallel_processes *pp)\n> \t * When get_next_task added messages to the buffer in its last\n> \t * iteration, the buffered output is non empty.\n> \t */\n> -\tstrbuf_write(&pp->buffered_output, stderr);\n> +\tif (!pp->ungroup)\n> +\t\tstrbuf_write(&pp->buffered_output, stderr);\n\nMicronit. If buffered_output is empty, whether it is because we are\nin the ungroup mode and haven't buffered anything there, or because\nour subprocess didn't emit anything, we do not have to do this write.\nSo it looks to me that it would be conceptually much cleaner to do\n\n\tif (pp->buffered_output.len)\n\t\tstrbuf_write(&pp->buffered_output, stderr);\n\nor just to let the strbuf_write() worry about it, as this is an I/O\ncodepath and the overhead of a no-op function call may be negligible.\n\nOr is there a reason to believe that pp->buffered_output is in an\nundefined state when in the ungroup mode? If so, we probably should\nfix that. The fewer special rules like \"in X mode, members Y, Z and\nW are left uninitialized so do not even look at them\", the better\noff we will be, especially when Y, Z and W have their own natural\n\"initialized and untouched\" state. Allow users of Y to decide how\nthey do things with Y without having to worry about X that they do\nnot have to worry about when doing their job.\n\n> \tstrbuf_release(&pp->buffered_output);\n\nAnd this unconditinal call indicates buffered_output is never in an\nundefined state and it is safe to call _release even in the ungroup\nmode.\n\n> @@ -1606,6 +1615,7 @@ static void pp_cleanup(struct parallel_processes *pp)\n> */\n> static int pp_start_one(struct parallel_processes *pp)\n> {\n> +\tconst int ungroup = pp->ungroup;\n> \tint i, code;\n> \n> \tfor (i = 0; i < pp->max_processes; i++)\n> @@ -1615,24 +1625,30 @@ static int pp_start_one(struct parallel_processes *pp)\n> \t\tBUG(\"bookkeeping is hard\");\n> \n> \tcode = pp->get_next_task(&pp->children[i].process,\n> -\t\t\t\t &pp->children[i].err,\n> +\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n> \t\t\t\t pp->data,\n> \t\t\t\t &pp->children[i].data);\n\nOK, any process taken from the pp struct with the ungroup bit on\ndoes not get its output stolen. Makes sense.\n\n> \tif (!code) {\n> -\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> -\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\tif (!ungroup) {\n> +\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> +\t\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\t}\n> \t\treturn 1;\n> \t}\n\nOK.\n\n> -\tpp->children[i].process.err = -1;\n> -\tpp->children[i].process.stdout_to_stderr = 1;\n> +\tif (!ungroup) {\n> +\t\tpp->children[i].process.err = -1;\n> +\t\tpp->children[i].process.stdout_to_stderr = 1;\n> +\t}\n\nOK.\n\n> \tpp->children[i].process.no_stdin = 1;\n\nThis is shared between the two modes, and is unchanged from the\nrun_hook_ve() days. Good.\n\n> \tif (start_command(&pp->children[i].process)) {\n> -\t\tcode = pp->start_failure(&pp->children[i].err,\n> +\t\tcode = pp->start_failure(ungroup ? NULL : &pp->children[i].err,\n> \t\t\t\t\t pp->data,\n> \t\t\t\t\t pp->children[i].data);\n> -\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> -\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\tif (!ungroup) {\n> +\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> +\t\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\t}\n> \t\tif (code)\n> \t\t\tpp->shutdown = 1;\n> \t\treturn code;\n\nOK.\n\n> @@ -1640,14 +1656,26 @@ static int pp_start_one(struct parallel_processes *pp)\n> \n> \tpp->nr_processes++;\n> \tpp->children[i].state = GIT_CP_WORKING;\n> -\tpp->pfd[i].fd = pp->children[i].process.err;\n> +\tif (pp->pfd)\n> +\t\tpp->pfd[i].fd = pp->children[i].process.err;\n> \treturn 0;\n> }\n\nOK.\n\n> +static void pp_mark_working_for_cleanup(struct parallel_processes *pp)\n> +{\n> +\tint i;\n> +\n> +\tfor (i = 0; i < pp->max_processes; i++)\n> +\t\tif (pp->children[i].state == GIT_CP_WORKING)\n> +\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n> +}\n\nThis thing is new. I do not see a corresponding removal of a\nsimilar loop that used to be done unconditionally that was turned\ninto a call to this helper only under the non-ungroup mode, or\nanything like that, so it is a bit puzzling.\n\n> static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> {\n> \tint i;\n> \n> +\tassert(!pp->ungroup);\n> +\n\nSensible. Or even \"if (pp->ungroup) BUG()\".\n\n> \twhile ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n> \t\tif (errno == EINTR)\n> \t\t\tcontinue;\n> @@ -1674,6 +1702,9 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> static void pp_output(struct parallel_processes *pp)\n> {\n> \tint i = pp->output_owner;\n> +\n> +\tassert(!pp->ungroup);\n> +\n> \tif (pp->children[i].state == GIT_CP_WORKING &&\n> \t pp->children[i].err.len) {\n> \t\tstrbuf_write(&pp->children[i].err, stderr);\n> @@ -1683,10 +1714,15 @@ static void pp_output(struct parallel_processes *pp)\n> \n> static int pp_collect_finished(struct parallel_processes *pp)\n> {\n> +\tconst int ungroup = pp->ungroup;\n> \tint i, code;\n> \tint n = pp->max_processes;\n> \tint result = 0;\n> \n> +\tif (ungroup)\n> +\t\tfor (i = 0; i < pp->max_processes; i++)\n> +\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n\nThe new helper does this only for those in the WORKING state, but\nthis one does so unconditionally. It's not like we leave the .state\nof our subprocesses unspecified when we start them---we set to WORKING\nwhether we are in the ungroup mode or not. So it is also puzzling why\nwe are not calling the helper function here.\n\nBy the way, if we use WAIT_CLEANUP state like this in the ungroup\nmode, shouldn't we lose the \"only for !ungroup\" comment from the\nenum definition?\n\n> @@ -1697,8 +1733,8 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> \t\tcode = finish_command(&pp->children[i].process);\n> \n> \t\tcode = pp->task_finished(code,\n> -\t\t\t\t\t &pp->children[i].err, pp->data,\n> -\t\t\t\t\t pp->children[i].data);\n> +\t\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n> +\t\t\t\t\t pp->data, pp->children[i].data);\n\nOK.\n\n> @@ -1707,10 +1743,13 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> \n> \t\tpp->nr_processes--;\n> \t\tpp->children[i].state = GIT_CP_FREE;\n> -\t\tpp->pfd[i].fd = -1;\n> +\t\tif (pp->pfd)\n> +\t\t\tpp->pfd[i].fd = -1;\n> \t\tchild_process_init(&pp->children[i].process);\n> \n> -\t\tif (i != pp->output_owner) {\n> +\t\tif (ungroup) {\n> +\t\t\t; /* no strbuf_*() work to do here */\n\nOf course ;-)\n\n> +\t\t} else if (i != pp->output_owner) {\n> \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> \t\t\tstrbuf_reset(&pp->children[i].err);\n> \t\t} else {\n> @@ -1748,8 +1787,13 @@ int run_processes_parallel(int n,\n> \tint output_timeout = 100;\n> \tint spawn_cap = 4;\n> \tstruct parallel_processes pp;\n> +\tconst int ungroup = run_processes_parallel_ungroup;\n> \n> -\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n> +\t/* unset for the next API user */\n> +\trun_processes_parallel_ungroup = 0;\n\nThis way, you do not have to touch existing calls to this function\nthat do not (yet) want to know about the ungroup mode.\n\nThat makes a confusing API, but the trade-off feels OK.\n\n> +\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n> +\t\tungroup);\n> \twhile (1) {\n> \t\tfor (i = 0;\n> \t\t i < spawn_cap && !pp.shutdown &&\n> @@ -1766,8 +1810,12 @@ int run_processes_parallel(int n,\n> \t\t}\n> \t\tif (!pp.nr_processes)\n> \t\t\tbreak;\n> -\t\tpp_buffer_stderr(&pp, output_timeout);\n> -\t\tpp_output(&pp);\n> +\t\tif (ungroup) {\n> +\t\t\tpp_mark_working_for_cleanup(&pp);\n> +\t\t} else {\n> +\t\t\tpp_buffer_stderr(&pp, output_timeout);\n> +\t\t\tpp_output(&pp);\n> +\t\t}\n> \t\tcode = pp_collect_finished(&pp);\n> \t\tif (code) {\n> \t\t\tpp.shutdown = 1;\n> diff --git a/run-command.h b/run-command.h\n> index 5bd0c933e80..a44d2a6ba75 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -405,6 +405,10 @@ void check_pipe(int err);\n> * pp_cb is the callback cookie as passed to run_processes_parallel.\n> * You can store a child process specific callback cookie in pp_task_cb.\n> *\n> + * The \"struct strbuf *err\" parameter is either a pointer to a string\n> + * to write errors to, or NULL if the \"ungroup\" option was\n> + * provided. See run_processes_parallel() below.\n> + *\n> * Even after returning 0 to indicate that there are no more processes,\n> * this function will be called again until there are no more running\n> * child processes.\n\nThis comment appears just before the typedef of get_next_task_fn\nfunction type, presumably to explain the parameters involved in\ncalling such a function, and it does talk about pp_cb and\npp_task_cb. The new paragraph, however, looks out of place. There\nis no err parameter. The existing text (before the pre-context)\nmentions \"preload the error channel\" but it is left unclear what\nthat means. Does that \"err\" non-parameter the new paragraph talks\nabout have some connection to the thing that receives the preloaded\nerror channel contents? Puzzled.\n\n> @@ -423,9 +427,9 @@ typedef int (*get_next_task_fn)(struct child_process *cp,\n> * This callback is called whenever there are problems starting\n> * a new process.\n> *\n> - * You must not write to stdout or stderr in this function. Add your\n> - * message to the strbuf out instead, which will be printed without\n> - * messing up the output of the other parallel processes.\n> + * The \"struct strbuf *err\" parameter is either a pointer to a string\n> + * to write errors to, or NULL if the \"ungroup\" option was\n> + * provided. See run_processes_parallel() below.\n\nThis comment is for start_failure_fn and has the same issue. The\ntext removed gives a readable/understandable explanation for\ndevelopers who are writing for non-ungrouped mode, though.\n\n> *\n> * pp_cb is the callback cookie as passed into run_processes_parallel,\n> * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> @@ -441,9 +445,9 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n> /**\n> * This callback is called on every child process that finished processing.\n> *\n> - * You must not write to stdout or stderr in this function. Add your\n> - * message to the strbuf out instead, which will be printed without\n> - * messing up the output of the other parallel processes.\n> + * The \"struct strbuf *err\" parameter is either a pointer to a string\n> + * to write errors to, or NULL if the \"ungroup\" option was\n> + * provided. See run_processes_parallel() below.\n\nDitto for task_finished_fn.\n\n> * pp_cb is the callback cookie as passed into run_processes_parallel,\n> * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> @@ -464,11 +468,24 @@ typedef int (*task_finished_fn)(int result,\n> *\n> * The children started via this function run in parallel. Their output\n> * (both stdout and stderr) is routed to stderr in a manner that output\n> - * from different tasks does not interleave.\n> + * from different tasks does not interleave (but see \"ungroup\" above).\n\nI think you meant \"below\" here, not \"above\".\n\n> * start_failure_fn and task_finished_fn can be NULL to omit any\n> * special handling.\n> + *\n> + * If the \"ungroup\" option isn't specified the callbacks will get a\n> + * pointer to a \"struct strbuf *out\", and must not write to stdout or\n> + * stderr as such output will mess up the output of the other parallel\n> + * processes. If \"ungroup\" option is specified callbacks will get a\n\n\"specified callbacks\" -> \"specified, callbacks\"\n\n> + * NULL \"struct strbuf *out\" parameter, and are responsible for\n> + * emitting their own output, including dealing with any race\n> + * conditions due to writing in parallel to stdout and stderr.\n> + * The \"ungroup\" option can be enabled by setting the global\n> + * \"run_processes_parallel_ungroup\" to \"1\" before invoking\n> + * run_processes_parallel(), it will be set back to \"0\" as soon as the\n> + * API reads that setting.\n> */\n\nThis new paragraph is well written.\n\n> +extern int run_processes_parallel_ungroup;\n> int run_processes_parallel(int n,\n> \t\t\t get_next_task_fn,\n> \t\t\t start_failure_fn,\n\n"},{"id":"456307","messageId":"xmqqa6b2520i.fsf@gitster.g","threadId":"57764","inReplyTo":"cover-v3-0.2-00000000000-20220527T090618Z-avarab@gmail.com","subject":"Re: [PATCH v3 0/2] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-27T17:17:01Z","receivedAt":"2022-05-27T17:17:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> A re-roll of [1] which aims to address the concerns about the previous\n> 8-part series being too large to fix a release regression. \"If it\n> isn't bolted down, throw it overboard!\".\n>\n> The main change here is:\n>\n> * The new \"ungroup\" parameter is now passed via an \"extern\" parameter.\n> * Tests for existing run-command.c behavior (not narrowly needed for\n> the regression fix) are gone.\n> * Adding an INIT macro is gone, instead we explicitly initialize to NULL.\n> * Stray bugfix for existing hook test is gone.\n>\n> etc. I think all of those still make sense, but they're something I\n> can rebase on this topic once it (hopefully) lands. In the meantime\n> the updated commit messages for the remaining two (see start of the\n> range-diff below) argue for this being a a safe API change, even if\n> the interface is a bit nasty.\n\nSo the approach taken here is that we assume the reported one is the\nonly regression and keep going with run_process_parallel() API.\n\nI still share the sentiment with Dscho that it is generally a bad\nidea, when dealing with a regression, to double-down and dig in\nyour heels to keep the change that caused a regression with paper\nover patches, but too much time has passed since the release, and a\npatch or two on top does look like a quicker way forward.\n\nI left a few comments on the implementation, but modulo these small\ndetails, the code looks OK (provided that the assumption holds true,\nthat is, of course).\n\nThanks.\n\n"},{"id":"456405","messageId":"patch-v4-1.2-f1170b02553-20220531T173005Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v4-0.2-00000000000-20220531T173005Z-avarab@gmail.com","subject":"[PATCH v4 1/2] run-command: add an \"ungroup\" option to run_process_parallel()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-31T17:32:58Z","receivedAt":"2022-05-31T17:33:31Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Extend the parallel execution API added in c553c72eed6 (run-command:\nadd an asynchronous parallel child processor, 2015-12-15) to support a\nmode where the stdout and stderr of the processes isn't captured and\noutput in a deterministic order, instead we'll leave it to the kernel\nand stdio to sort it out.\n\nThis gives the API same functionality as GNU parallel's --ungroup\noption. As we'll see in a subsequent commit the main reason to want\nthis is to support stdout and stderr being connected to the TTY in the\ncase of jobs=1, demonstrated here with GNU parallel:\n\n\t$ parallel --ungroup 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n\tTTY\n\tTTY\n\t$ parallel 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n\tNTTY\n\tNTTY\n\nAnother is as GNU parallel's documentation notes a potential for\noptimization. Our results will be a bit different, but in cases where\nyou want to run processes in parallel where the exact order isn't\nimportant this can be a lot faster:\n\n\t$ hyperfine -r 3 -L o ,--ungroup 'parallel {o} seq ::: 10000000 >/dev/null '\n\tBenchmark 1: parallel seq ::: 10000000 >/dev/null\n\t Time (mean ± σ): 220.2 ms ± 9.3 ms [User: 124.9 ms, System: 96.1 ms]\n\t Range (min … max): 212.3 ms … 230.5 ms 3 runs\n\n\tBenchmark 2: parallel --ungroup seq ::: 10000000 >/dev/null\n\t Time (mean ± σ): 154.7 ms ± 0.9 ms [User: 136.2 ms, System: 25.1 ms]\n\t Range (min … max): 153.9 ms … 155.7 ms 3 runs\n\n\tSummary\n\t 'parallel --ungroup seq ::: 10000000 >/dev/null ' ran\n\t 1.42 ± 0.06 times faster than 'parallel seq ::: 10000000 >/dev/null '\n\nA large part of the juggling in the API is to make the API safer for\nits maintenance and consumers alike.\n\nFor the maintenance of the API we e.g. avoid malloc()-ing the\n\"pp->pfd\", ensuring that SANITIZE=address and other similar tools will\ncatch any unexpected misuse.\n\nFor API consumers we take pains to never pass the non-NULL \"out\"\nbuffer to an API user that provided the \"ungroup\" option. The\nresulting code in t/helper/test-run-command.c isn't typical of such a\nuser, i.e. they'd typically use one mode or the other, and would know\nwhether they'd provided \"ungroup\" or not.\n\nWe could also avoid the strbuf_init() for \"buffered_output\" by having\n\"struct parallel_processes\" use a static PARALLEL_PROCESSES_INIT\ninitializer, but let's leave that cleanup for later.\n\nUsing a global \"run_processes_parallel_ungroup\" variable to enable\nthis option is rather nasty, but is being done here to produce as\nminimal of a change as possible for a subsequent regression fix. This\nchange is extracted from a larger initial version[1] which ends up\nwith a better end-state for the API, but in doing so needed to modify\nall existing callers of the API. Let's defer that for now, and\nnarrowly focus on what we need for fixing the regression in the\nsubsequent commit.\n\nIt's safe to do this with a global variable because:\n\n A) hook.c is the only user of it that sets it to non-zero, and before\n we'll get any other API users we'll refactor away this method of\n passing in the option, i.e. re-roll [1].\n\n B) Even if hook.c wasn't the only user we don't have callers of this\n API that concurrently invoke this parallel process starting API\n itself in parallel.\n\nAs noted above \"A\" && \"B\" are rather nasty, and we don't want to live\nwith those caveats long-term, but for now they should be an acceptable\ncompromise.\n\n1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 83 +++++++++++++++++++++++++++++--------\n run-command.h | 30 ++++++++++----\n t/helper/test-run-command.c | 19 +++++++--\n t/t0061-run-command.sh | 35 ++++++++++++++++\n 4 files changed, 139 insertions(+), 28 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex a8501e38ceb..324e9548469 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1471,6 +1471,7 @@ enum child_state {\n \tGIT_CP_WAIT_CLEANUP,\n };\n \n+int run_processes_parallel_ungroup;\n struct parallel_processes {\n \tvoid *data;\n \n@@ -1494,6 +1495,7 @@ struct parallel_processes {\n \tstruct pollfd *pfd;\n \n \tunsigned shutdown : 1;\n+\tunsigned ungroup : 1;\n \n \tint output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n@@ -1537,7 +1539,7 @@ static void pp_init(struct parallel_processes *pp,\n \t\t get_next_task_fn get_next_task,\n \t\t start_failure_fn start_failure,\n \t\t task_finished_fn task_finished,\n-\t\t void *data)\n+\t\t void *data, const int ungroup)\n {\n \tint i;\n \n@@ -1559,13 +1561,19 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->nr_processes = 0;\n \tpp->output_owner = 0;\n \tpp->shutdown = 0;\n+\tpp->ungroup = ungroup;\n \tCALLOC_ARRAY(pp->children, n);\n-\tCALLOC_ARRAY(pp->pfd, n);\n+\tif (pp->ungroup)\n+\t\tpp->pfd = NULL;\n+\telse\n+\t\tCALLOC_ARRAY(pp->pfd, n);\n \tstrbuf_init(&pp->buffered_output, 0);\n \n \tfor (i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n \t\tchild_process_init(&pp->children[i].process);\n+\t\tif (!pp->pfd)\n+\t\t\tcontinue;\n \t\tpp->pfd[i].events = POLLIN | POLLHUP;\n \t\tpp->pfd[i].fd = -1;\n \t}\n@@ -1606,6 +1614,7 @@ static void pp_cleanup(struct parallel_processes *pp)\n */\n static int pp_start_one(struct parallel_processes *pp)\n {\n+\tconst int ungroup = pp->ungroup;\n \tint i, code;\n \n \tfor (i = 0; i < pp->max_processes; i++)\n@@ -1615,24 +1624,30 @@ static int pp_start_one(struct parallel_processes *pp)\n \t\tBUG(\"bookkeeping is hard\");\n \n \tcode = pp->get_next_task(&pp->children[i].process,\n-\t\t\t\t &pp->children[i].err,\n+\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n \t\t\t\t pp->data,\n \t\t\t\t &pp->children[i].data);\n \tif (!code) {\n-\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n-\t\tstrbuf_reset(&pp->children[i].err);\n+\t\tif (!ungroup) {\n+\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n+\t\t\tstrbuf_reset(&pp->children[i].err);\n+\t\t}\n \t\treturn 1;\n \t}\n-\tpp->children[i].process.err = -1;\n-\tpp->children[i].process.stdout_to_stderr = 1;\n+\tif (!ungroup) {\n+\t\tpp->children[i].process.err = -1;\n+\t\tpp->children[i].process.stdout_to_stderr = 1;\n+\t}\n \tpp->children[i].process.no_stdin = 1;\n \n \tif (start_command(&pp->children[i].process)) {\n-\t\tcode = pp->start_failure(&pp->children[i].err,\n+\t\tcode = pp->start_failure(ungroup ? NULL : &pp->children[i].err,\n \t\t\t\t\t pp->data,\n \t\t\t\t\t pp->children[i].data);\n-\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n-\t\tstrbuf_reset(&pp->children[i].err);\n+\t\tif (!ungroup) {\n+\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n+\t\t\tstrbuf_reset(&pp->children[i].err);\n+\t\t}\n \t\tif (code)\n \t\t\tpp->shutdown = 1;\n \t\treturn code;\n@@ -1640,14 +1655,29 @@ static int pp_start_one(struct parallel_processes *pp)\n \n \tpp->nr_processes++;\n \tpp->children[i].state = GIT_CP_WORKING;\n-\tpp->pfd[i].fd = pp->children[i].process.err;\n+\tif (pp->pfd)\n+\t\tpp->pfd[i].fd = pp->children[i].process.err;\n \treturn 0;\n }\n \n+static void pp_mark_ungrouped_for_cleanup(struct parallel_processes *pp)\n+{\n+\tint i;\n+\n+\tif (!pp->ungroup)\n+\t\tBUG(\"only reachable if 'ungrouped'\");\n+\n+\tfor (i = 0; i < pp->max_processes; i++)\n+\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+}\n+\n static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n {\n \tint i;\n \n+\tif (pp->ungroup)\n+\t\tBUG(\"unreachable with 'ungrouped'\");\n+\n \twhile ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n \t\tif (errno == EINTR)\n \t\t\tcontinue;\n@@ -1674,6 +1704,10 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n static void pp_output(struct parallel_processes *pp)\n {\n \tint i = pp->output_owner;\n+\n+\tif (pp->ungroup)\n+\t\tBUG(\"unreachable with 'ungrouped'\");\n+\n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n \t\tstrbuf_write(&pp->children[i].err, stderr);\n@@ -1683,6 +1717,7 @@ static void pp_output(struct parallel_processes *pp)\n \n static int pp_collect_finished(struct parallel_processes *pp)\n {\n+\tconst int ungroup = pp->ungroup;\n \tint i, code;\n \tint n = pp->max_processes;\n \tint result = 0;\n@@ -1697,8 +1732,8 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \t\tcode = finish_command(&pp->children[i].process);\n \n \t\tcode = pp->task_finished(code,\n-\t\t\t\t\t &pp->children[i].err, pp->data,\n-\t\t\t\t\t pp->children[i].data);\n+\t\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n+\t\t\t\t\t pp->data, pp->children[i].data);\n \n \t\tif (code)\n \t\t\tresult = code;\n@@ -1707,10 +1742,13 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \n \t\tpp->nr_processes--;\n \t\tpp->children[i].state = GIT_CP_FREE;\n-\t\tpp->pfd[i].fd = -1;\n+\t\tif (pp->pfd)\n+\t\t\tpp->pfd[i].fd = -1;\n \t\tchild_process_init(&pp->children[i].process);\n \n-\t\tif (i != pp->output_owner) {\n+\t\tif (ungroup) {\n+\t\t\t; /* no strbuf_*() work to do here */\n+\t\t} else if (i != pp->output_owner) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t} else {\n@@ -1748,8 +1786,13 @@ int run_processes_parallel(int n,\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n \tstruct parallel_processes pp;\n+\tconst int ungroup = run_processes_parallel_ungroup;\n \n-\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n+\t/* unset for the next API user */\n+\trun_processes_parallel_ungroup = 0;\n+\n+\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n+\t\tungroup);\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1766,8 +1809,12 @@ int run_processes_parallel(int n,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tpp_buffer_stderr(&pp, output_timeout);\n-\t\tpp_output(&pp);\n+\t\tif (ungroup) {\n+\t\t\tpp_mark_ungrouped_for_cleanup(&pp);\n+\t\t} else {\n+\t\t\tpp_buffer_stderr(&pp, output_timeout);\n+\t\t\tpp_output(&pp);\n+\t\t}\n \t\tcode = pp_collect_finished(&pp);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\ndiff --git a/run-command.h b/run-command.h\nindex 5bd0c933e80..bf4236f1164 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -405,6 +405,9 @@ void check_pipe(int err);\n * pp_cb is the callback cookie as passed to run_processes_parallel.\n * You can store a child process specific callback cookie in pp_task_cb.\n *\n+ * See run_processes_parallel() below for a discussion of the \"struct\n+ * strbuf *out\" parameter.\n+ *\n * Even after returning 0 to indicate that there are no more processes,\n * this function will be called again until there are no more running\n * child processes.\n@@ -423,9 +426,8 @@ typedef int (*get_next_task_fn)(struct child_process *cp,\n * This callback is called whenever there are problems starting\n * a new process.\n *\n- * You must not write to stdout or stderr in this function. Add your\n- * message to the strbuf out instead, which will be printed without\n- * messing up the output of the other parallel processes.\n+ * See run_processes_parallel() below for a discussion of the \"struct\n+ * strbuf *out\" parameter.\n *\n * pp_cb is the callback cookie as passed into run_processes_parallel,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n@@ -441,9 +443,8 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n /**\n * This callback is called on every child process that finished processing.\n *\n- * You must not write to stdout or stderr in this function. Add your\n- * message to the strbuf out instead, which will be printed without\n- * messing up the output of the other parallel processes.\n+ * See run_processes_parallel() below for a discussion of the \"struct\n+ * strbuf *out\" parameter.\n *\n * pp_cb is the callback cookie as passed into run_processes_parallel,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n@@ -464,11 +465,26 @@ typedef int (*task_finished_fn)(int result,\n *\n * The children started via this function run in parallel. Their output\n * (both stdout and stderr) is routed to stderr in a manner that output\n- * from different tasks does not interleave.\n+ * from different tasks does not interleave (but see \"ungroup\" below).\n *\n * start_failure_fn and task_finished_fn can be NULL to omit any\n * special handling.\n+ *\n+ * If the \"ungroup\" option isn't specified, the API will set the\n+ * \"stdout_to_stderr\" parameter in \"struct child_process\" and provide\n+ * the callbacks with a \"struct strbuf *out\" parameter to write output\n+ * to. In this case the callbacks must not write to stdout or\n+ * stderr as such output will mess up the output of the other parallel\n+ * processes. If \"ungroup\" option is specified callbacks will get a\n+ * NULL \"struct strbuf *out\" parameter, and are responsible for\n+ * emitting their own output, including dealing with any race\n+ * conditions due to writing in parallel to stdout and stderr.\n+ * The \"ungroup\" option can be enabled by setting the global\n+ * \"run_processes_parallel_ungroup\" to \"1\" before invoking\n+ * run_processes_parallel(), it will be set back to \"0\" as soon as the\n+ * API reads that setting.\n */\n+extern int run_processes_parallel_ungroup;\n int run_processes_parallel(int n,\n \t\t\t get_next_task_fn,\n \t\t\t start_failure_fn,\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex f3b90aa834a..6405c9a076a 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -31,7 +31,11 @@ static int parallel_next(struct child_process *cp,\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->args.v);\n-\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n+\telse\n+\t\tfprintf(stderr, \"preloaded output of a child\\n\");\n+\n \tnumber_callbacks++;\n \treturn 1;\n }\n@@ -41,7 +45,10 @@ static int no_job(struct child_process *cp,\n \t\t void *cb,\n \t\t void **task_cb)\n {\n-\tstrbuf_addstr(err, \"no further jobs available\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"no further jobs available\\n\");\n+\telse\n+\t\tfprintf(stderr, \"no further jobs available\\n\");\n \treturn 0;\n }\n \n@@ -50,7 +57,10 @@ static int task_finished(int result,\n \t\t\t void *pp_cb,\n \t\t\t void *pp_task_cb)\n {\n-\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n+\telse\n+\t\tfprintf(stderr, \"asking for a quick stop\\n\");\n \treturn 1;\n }\n \n@@ -411,6 +421,9 @@ int cmd__run_command(int argc, const char **argv)\n \tstrvec_clear(&proc.args);\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n+\tif (getenv(\"RUN_PROCESSES_PARALLEL_UNGROUP\"))\n+\t\trun_processes_parallel_ungroup = 1;\n+\n \tif (!strcmp(argv[1], \"run-command-parallel\"))\n \t\texit(run_processes_parallel(jobs, parallel_next,\n \t\t\t\t\t NULL, NULL, &proc));\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex ee281909bc3..69ccaa8d298 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -134,16 +134,37 @@ test_expect_success 'run_command runs in parallel with more jobs available than\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n+\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n+\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n \ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n+\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n+\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n \ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n+\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n+\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n@@ -158,6 +179,13 @@ test_expect_success 'run_command is asked to abort gracefully' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n+\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n+\ttest-tool run-command run-command-abort 3 false >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_line_count = 6 err\n+'\n+\n cat >expect <<-EOF\n no further jobs available\n EOF\n@@ -167,6 +195,13 @@ test_expect_success 'run_command outputs ' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command outputs (ungroup) ' '\n+\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n+\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n test_trace () {\n \texpect=\"$1\"\n \tshift\n-- \n2.36.1.1103.g036c05811b0\n\n"},{"id":"456406","messageId":"patch-v4-2.2-8ab09f28729-20220531T173005Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v4-0.2-00000000000-20220531T173005Z-avarab@gmail.com","subject":"[PATCH v4 2/2] hook API: fix v2.36.0 regression: hooks should be connected to a TTY","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-31T17:32:59Z","receivedAt":"2022-05-31T17:33:33Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a regression reported[1] in f443246b9f2 (commit: convert\n{pre-commit,prepare-commit-msg} hook to hook.h, 2021-12-22): Due to\nusing the run_process_parallel() API in the earlier 96e7225b310 (hook:\nadd 'run' subcommand, 2021-12-22) we'd capture the hook's stderr and\nstdout, and thus lose the connection to the TTY in the case of\ne.g. the \"pre-commit\" hook.\n\nAs a preceding commit notes GNU parallel's similar --ungroup option\nalso has it emit output faster. While we're unlikely to have hooks\nthat emit truly massive amounts of output (or where the performance\nthereof matters) it's still informative to measure the overhead. In a\nsimilar \"seq\" test we're now ~30% faster:\n\n\t$ cat .git/hooks/seq-hook; git hyperfine -L rev origin/master,HEAD~0 -s 'make CFLAGS=-O3' './git hook run seq-hook'\n\t#!/bin/sh\n\n\tseq 100000000\n\tBenchmark 1: ./git hook run seq-hook' in 'origin/master\n\t Time (mean ± σ): 787.1 ms ± 13.6 ms [User: 701.6 ms, System: 534.4 ms]\n\t Range (min … max): 773.2 ms … 806.3 ms 10 runs\n\n\tBenchmark 2: ./git hook run seq-hook' in 'HEAD~0\n\t Time (mean ± σ): 603.4 ms ± 1.6 ms [User: 573.1 ms, System: 30.3 ms]\n\t Range (min … max): 601.0 ms … 606.2 ms 10 runs\n\n\tSummary\n\t './git hook run seq-hook' in 'HEAD~0' ran\n\t 1.30 ± 0.02 times faster than './git hook run seq-hook' in 'origin/master'\n\n1. https://lore.kernel.org/git/CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com/\n\nReported-by: Anthony Sottile \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 1 +\n t/t1800-hook.sh | 37 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 38 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex 1d51be3b77a..7451205657a 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -144,6 +144,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \t\tcb_data.hook_path = abs_path.buf;\n \t}\n \n+\trun_processes_parallel_ungroup = 1;\n \trun_processes_parallel_tr2(jobs,\n \t\t\t\t pick_next_hook,\n \t\t\t\t notify_start_failure,\ndiff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\nindex 26ed5e11bc8..0b8370d1573 100755\n--- a/t/t1800-hook.sh\n+++ b/t/t1800-hook.sh\n@@ -4,6 +4,7 @@ test_description='git-hook command'\n \n TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-terminal.sh\n \n test_expect_success 'git hook usage' '\n \ttest_expect_code 129 git hook &&\n@@ -120,4 +121,40 @@ test_expect_success 'git -c core.hooksPath= hook run' '\n \ttest_cmp expect actual\n '\n \n+test_hook_tty() {\n+\tlocal fd=\"$1\" &&\n+\n+\tcat >expect &&\n+\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\n+\ttest_hook -C repo pre-commit <<-EOF &&\n+\t{\n+\t\ttest -t 1 && echo >&$fd STDOUT TTY || echo >&$fd STDOUT NO TTY &&\n+\t\ttest -t 2 && echo >&$fd STDERR TTY || echo >&$fd STDERR NO TTY\n+\t} $fd>actual\n+\tEOF\n+\n+\ttest_commit -C repo A &&\n+\ttest_commit -C repo B &&\n+\tgit -C repo reset --soft HEAD^ &&\n+\ttest_terminal git -C repo commit -m\"B.new\" &&\n+\ttest_cmp expect repo/actual\n+}\n+\n+test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDOUT redirect' '\n+\ttest_hook_tty 1 <<-\\EOF\n+\tSTDOUT NO TTY\n+\tSTDERR TTY\n+\tEOF\n+'\n+\n+test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDERR redirect' '\n+\ttest_hook_tty 2 <<-\\EOF\n+\tSTDOUT TTY\n+\tSTDERR NO TTY\n+\tEOF\n+'\n+\n test_done\n-- \n2.36.1.1103.g036c05811b0\n\n"},{"id":"456407","messageId":"cover-v4-0.2-00000000000-20220531T173005Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v3-0.2-00000000000-20220527T090618Z-avarab@gmail.com","subject":"[PATCH v4 0/2] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-31T17:32:57Z","receivedAt":"2022-05-31T17:35:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A re-roll of [1] fixing issues pointed out by Junio in the last re-roll:\n\n * A minor whitespace parameter fix.\n\n * Clear up confusion about GIT_CP_WAIT_CLEANUP, and remove the\n duplicate \"mark ungroup children as GIT_CP_WAIT_CLEANUP\" code.\n\n * Droped a conditional strbuf_write() on an always-empty string under\n \"ungroup\".\n\n * Correct \"err\" parameter name to \"out\" in the new API docs.\n\n * Replace assert() with BUG().\n\n * Address other minor issues noted by Junio.\n\nÆvar Arnfjörð Bjarmason (2):\n run-command: add an \"ungroup\" option to run_process_parallel()\n hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n\n hook.c | 1 +\n run-command.c | 83 +++++++++++++++++++++++++++++--------\n run-command.h | 30 ++++++++++----\n t/helper/test-run-command.c | 19 +++++++--\n t/t0061-run-command.sh | 35 ++++++++++++++++\n t/t1800-hook.sh | 37 +++++++++++++++++\n 6 files changed, 177 insertions(+), 28 deletions(-)\n\nRange-diff against v3:\n1: aabd99de680 ! 1: f1170b02553 run-command: add an \"ungroup\" option to run_process_parallel()\n @@ Commit message\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## run-command.c ##\n -@@ run-command.c: int pipe_command(struct child_process *cmd,\n - enum child_state {\n - \tGIT_CP_FREE,\n - \tGIT_CP_WORKING,\n --\tGIT_CP_WAIT_CLEANUP,\n -+\tGIT_CP_WAIT_CLEANUP, /* only for !ungroup */\n +@@ run-command.c: enum child_state {\n + \tGIT_CP_WAIT_CLEANUP,\n };\n \n +int run_processes_parallel_ungroup;\n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n \t\t start_failure_fn start_failure,\n \t\t task_finished_fn task_finished,\n -\t\t void *data)\n -+\t\t void *data, const int ungroup)\n ++\t\t void *data, const int ungroup)\n {\n \tint i;\n \n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n \t\tpp->pfd[i].events = POLLIN | POLLHUP;\n \t\tpp->pfd[i].fd = -1;\n \t}\n -@@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)\n - \t * When get_next_task added messages to the buffer in its last\n - \t * iteration, the buffered output is non empty.\n - \t */\n --\tstrbuf_write(&pp->buffered_output, stderr);\n -+\tif (!pp->ungroup)\n -+\t\tstrbuf_write(&pp->buffered_output, stderr);\n - \tstrbuf_release(&pp->buffered_output);\n - \n - \tsigchain_pop_common();\n @@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)\n */\n static int pp_start_one(struct parallel_processes *pp)\n @@ run-command.c: static int pp_start_one(struct parallel_processes *pp)\n \treturn 0;\n }\n \n -+static void pp_mark_working_for_cleanup(struct parallel_processes *pp)\n ++static void pp_mark_ungrouped_for_cleanup(struct parallel_processes *pp)\n +{\n +\tint i;\n +\n ++\tif (!pp->ungroup)\n ++\t\tBUG(\"only reachable if 'ungrouped'\");\n ++\n +\tfor (i = 0; i < pp->max_processes; i++)\n -+\t\tif (pp->children[i].state == GIT_CP_WORKING)\n -+\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n ++\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n +}\n +\n static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n {\n \tint i;\n \n -+\tassert(!pp->ungroup);\n ++\tif (pp->ungroup)\n ++\t\tBUG(\"unreachable with 'ungrouped'\");\n +\n \twhile ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n \t\tif (errno == EINTR)\n @@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp, int o\n {\n \tint i = pp->output_owner;\n +\n -+\tassert(!pp->ungroup);\n ++\tif (pp->ungroup)\n ++\t\tBUG(\"unreachable with 'ungrouped'\");\n +\n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n @@ run-command.c: static void pp_output(struct parallel_processes *pp)\n \tint i, code;\n \tint n = pp->max_processes;\n \tint result = 0;\n - \n -+\tif (ungroup)\n -+\t\tfor (i = 0; i < pp->max_processes; i++)\n -+\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n -+\n - \twhile (pp->nr_processes > 0) {\n - \t\tfor (i = 0; i < pp->max_processes; i++)\n - \t\t\tif (pp->children[i].state == GIT_CP_WAIT_CLEANUP)\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n \t\tcode = finish_command(&pp->children[i].process);\n \n @@ run-command.c: int run_processes_parallel(int n,\n -\t\tpp_buffer_stderr(&pp, output_timeout);\n -\t\tpp_output(&pp);\n +\t\tif (ungroup) {\n -+\t\t\tpp_mark_working_for_cleanup(&pp);\n ++\t\t\tpp_mark_ungrouped_for_cleanup(&pp);\n +\t\t} else {\n +\t\t\tpp_buffer_stderr(&pp, output_timeout);\n +\t\t\tpp_output(&pp);\n @@ run-command.h: void check_pipe(int err);\n * pp_cb is the callback cookie as passed to run_processes_parallel.\n * You can store a child process specific callback cookie in pp_task_cb.\n *\n -+ * The \"struct strbuf *err\" parameter is either a pointer to a string\n -+ * to write errors to, or NULL if the \"ungroup\" option was\n -+ * provided. See run_processes_parallel() below.\n ++ * See run_processes_parallel() below for a discussion of the \"struct\n ++ * strbuf *out\" parameter.\n + *\n * Even after returning 0 to indicate that there are no more processes,\n * this function will be called again until there are no more running\n @@ run-command.h: typedef int (*get_next_task_fn)(struct child_process *cp,\n - * You must not write to stdout or stderr in this function. Add your\n - * message to the strbuf out instead, which will be printed without\n - * messing up the output of the other parallel processes.\n -+ * The \"struct strbuf *err\" parameter is either a pointer to a string\n -+ * to write errors to, or NULL if the \"ungroup\" option was\n -+ * provided. See run_processes_parallel() below.\n ++ * See run_processes_parallel() below for a discussion of the \"struct\n ++ * strbuf *out\" parameter.\n *\n * pp_cb is the callback cookie as passed into run_processes_parallel,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n @@ run-command.h: typedef int (*start_failure_fn)(struct strbuf *out,\n - * You must not write to stdout or stderr in this function. Add your\n - * message to the strbuf out instead, which will be printed without\n - * messing up the output of the other parallel processes.\n -+ * The \"struct strbuf *err\" parameter is either a pointer to a string\n -+ * to write errors to, or NULL if the \"ungroup\" option was\n -+ * provided. See run_processes_parallel() below.\n ++ * See run_processes_parallel() below for a discussion of the \"struct\n ++ * strbuf *out\" parameter.\n *\n * pp_cb is the callback cookie as passed into run_processes_parallel,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n @@ run-command.h: typedef int (*task_finished_fn)(int result,\n * The children started via this function run in parallel. Their output\n * (both stdout and stderr) is routed to stderr in a manner that output\n - * from different tasks does not interleave.\n -+ * from different tasks does not interleave (but see \"ungroup\" above).\n ++ * from different tasks does not interleave (but see \"ungroup\" below).\n *\n * start_failure_fn and task_finished_fn can be NULL to omit any\n * special handling.\n + *\n -+ * If the \"ungroup\" option isn't specified the callbacks will get a\n -+ * pointer to a \"struct strbuf *out\", and must not write to stdout or\n ++ * If the \"ungroup\" option isn't specified, the API will set the\n ++ * \"stdout_to_stderr\" parameter in \"struct child_process\" and provide\n ++ * the callbacks with a \"struct strbuf *out\" parameter to write output\n ++ * to. In this case the callbacks must not write to stdout or\n + * stderr as such output will mess up the output of the other parallel\n + * processes. If \"ungroup\" option is specified callbacks will get a\n + * NULL \"struct strbuf *out\" parameter, and are responsible for\n2: ec27e3906e1 = 2: 8ab09f28729 hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n-- \n2.36.1.1103.g036c05811b0\n\n"},{"id":"456450","messageId":"nycvar.QRO.7.76.6.2206011827300.349@tvgsbejvaqbjf.bet","threadId":"57764","inReplyTo":"patch-v4-2.2-8ab09f28729-20220531T173005Z-avarab@gmail.com","subject":"Re: [PATCH v4 2/2] hook API: fix v2.36.0 regression: hooks should be connected to a TTY","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-06-01T16:50:06Z","receivedAt":"2022-06-01T16:50:50Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Tue, 31 May 2022, Ævar Arnfjörð Bjarmason wrote:\n\n> Fix a regression reported[1] in f443246b9f2 (commit: convert\n\nThe regression was not reported in f443247b9f2.\n\n> {pre-commit,prepare-commit-msg} hook to hook.h, 2021-12-22): Due to\n> using the run_process_parallel() API in the earlier 96e7225b310 (hook:\n> add 'run' subcommand, 2021-12-22) we'd capture the hook's stderr and\n> stdout, and thus lose the connection to the TTY in the case of\n> e.g. the \"pre-commit\" hook.\n>\n> As a preceding commit notes GNU parallel's similar --ungroup option\n> also has it emit output faster. While we're unlikely to have hooks\n> that emit truly massive amounts of output (or where the performance\n> thereof matters) it's still informative to measure the overhead. In a\n> similar \"seq\" test we're now ~30% faster:\n\nIt is an unwanted distraction to talk about the speed here, when the\nentire purpose of the patch series is to fix the regression that stdio are\nno longer connected when running hooks.\n\n>\n> \t$ cat .git/hooks/seq-hook; git hyperfine -L rev origin/master,HEAD~0 -s 'make CFLAGS=-O3' './git hook run seq-hook'\n> \t#!/bin/sh\n>\n> \tseq 100000000\n> \tBenchmark 1: ./git hook run seq-hook' in 'origin/master\n> \t Time (mean ± σ): 787.1 ms ± 13.6 ms [User: 701.6 ms, System: 534.4 ms]\n> \t Range (min … max): 773.2 ms … 806.3 ms 10 runs\n>\n> \tBenchmark 2: ./git hook run seq-hook' in 'HEAD~0\n> \t Time (mean ± σ): 603.4 ms ± 1.6 ms [User: 573.1 ms, System: 30.3 ms]\n> \t Range (min … max): 601.0 ms … 606.2 ms 10 runs\n>\n> \tSummary\n> \t './git hook run seq-hook' in 'HEAD~0' ran\n> \t 1.30 ± 0.02 times faster than './git hook run seq-hook' in 'origin/master'\n>\n> 1. https://lore.kernel.org/git/CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com/\n>\n> Reported-by: Anthony Sottile \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> hook.c | 1 +\n> t/t1800-hook.sh | 37 +++++++++++++++++++++++++++++++++++++\n> 2 files changed, 38 insertions(+)\n>\n> diff --git a/hook.c b/hook.c\n> index 1d51be3b77a..7451205657a 100644\n> --- a/hook.c\n> +++ b/hook.c\n> @@ -144,6 +144,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n> \t\tcb_data.hook_path = abs_path.buf;\n> \t}\n>\n> +\trun_processes_parallel_ungroup = 1;\n> \trun_processes_parallel_tr2(jobs,\n> \t\t\t\t pick_next_hook,\n> \t\t\t\t notify_start_failure,\n> diff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\n> index 26ed5e11bc8..0b8370d1573 100755\n> --- a/t/t1800-hook.sh\n> +++ b/t/t1800-hook.sh\n> @@ -4,6 +4,7 @@ test_description='git-hook command'\n>\n> TEST_PASSES_SANITIZE_LEAK=true\n> . ./test-lib.sh\n> +. \"$TEST_DIRECTORY\"/lib-terminal.sh\n>\n> test_expect_success 'git hook usage' '\n> \ttest_expect_code 129 git hook &&\n> @@ -120,4 +121,40 @@ test_expect_success 'git -c core.hooksPath= hook run' '\n> \ttest_cmp expect actual\n> '\n>\n> +test_hook_tty() {\n> +\tlocal fd=\"$1\" &&\n> +\n> +\tcat >expect &&\n> +\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\n> +\ttest_hook -C repo pre-commit <<-EOF &&\n> +\t{\n> +\t\ttest -t 1 && echo >&$fd STDOUT TTY || echo >&$fd STDOUT NO TTY &&\n> +\t\ttest -t 2 && echo >&$fd STDERR TTY || echo >&$fd STDERR NO TTY\n> +\t} $fd>actual\n> +\tEOF\n> +\n> +\ttest_commit -C repo A &&\n> +\ttest_commit -C repo B &&\n> +\tgit -C repo reset --soft HEAD^ &&\n> +\ttest_terminal git -C repo commit -m\"B.new\" &&\n> +\ttest_cmp expect repo/actual\n> +}\n> +\n> +test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDOUT redirect' '\n> +\ttest_hook_tty 1 <<-\\EOF\n> +\tSTDOUT NO TTY\n> +\tSTDERR TTY\n> +\tEOF\n> +'\n> +\n> +test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDERR redirect' '\n> +\ttest_hook_tty 2 <<-\\EOF\n> +\tSTDOUT TTY\n> +\tSTDERR NO TTY\n> +\tEOF\n> +'\n\nInstead of spreading the regression test out over so many lines, a single\ntest case that verifies what needs to be verified succinctly should be\nplenty sufficient. Something along these lines:\n\n\ttest_expect_success TTY 'hooks are conencted to stdio' '\n\t\ttest_when_finished \"rm .git/hooks/pre-commit\" &&\n\n\t\twrite_script .git/hooks/pre-commit <<-EOF\n\t\ttest -t 1 && echo \"stdout is a TTY\" >out\n\t\ttest -t 2 && echo \"stderr is a TTY\" >>out\n\t\tEOF\n\n\t\ttest_terminal git commit --allow-empty -m hooks-and-stdio &&\n\t\tgrep stdout out &&\n\t\tgrep stderr out\n\t'\n\nNot only is this much easier to review, not only is it more obvious what\nis being tested, it is also much quicker to debug in case it fails.\n\nCiao,\nJohannes\n\n> +\n> test_done\n> --\n> 2.36.1.1103.g036c05811b0\n>\n>\n"},{"id":"456451","messageId":"nycvar.QRO.7.76.6.2206011630400.349@tvgsbejvaqbjf.bet","threadId":"57764","inReplyTo":"patch-v4-1.2-f1170b02553-20220531T173005Z-avarab@gmail.com","subject":"Re: [PATCH v4 1/2] run-command: add an \"ungroup\" option to run_process_parallel()","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-06-01T16:49:58Z","receivedAt":"2022-06-01T16:51:01Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nwhat you work on here is important. Git v2.36.0 unfortunately had a couple\nof bugs that were quickly discovered, and this here regression is the one\nof them. The other known ones had been fixed after a little more than two\nweeks, but another four weeks later, this regression still awaits fixing.\n\nOn Tue, 31 May 2022, Ævar Arnfjörð Bjarmason wrote:\n\n> Extend the parallel execution API added in c553c72eed6 (run-command:\n> add an asynchronous parallel child processor, 2015-12-15) to support a\n> mode where the stdout and stderr of the processes isn't captured and\n> output in a deterministic order, instead we'll leave it to the kernel\n> and stdio to sort it out.\n\nIt might be worth picking a better name than \"ungroup\". Maybe something\nlike \"interleaved_output\".\n\n>\n> This gives the API same functionality as GNU parallel's --ungroup\n> option. As we'll see in a subsequent commit the main reason to want\n> this is to support stdout and stderr being connected to the TTY in the\n> case of jobs=1, demonstrated here with GNU parallel:\n>\n> \t$ parallel --ungroup 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n> \tTTY\n> \tTTY\n> \t$ parallel 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n> \tNTTY\n> \tNTTY\n>\n> Another is as GNU parallel's documentation notes a potential for\n> optimization. Our results will be a bit different, but in cases where\n> you want to run processes in parallel where the exact order isn't\n> important this can be a lot faster:\n>\n> \t$ hyperfine -r 3 -L o ,--ungroup 'parallel {o} seq ::: 10000000 >/dev/null '\n> \tBenchmark 1: parallel seq ::: 10000000 >/dev/null\n> \t Time (mean ± σ): 220.2 ms ± 9.3 ms [User: 124.9 ms, System: 96.1 ms]\n> \t Range (min … max): 212.3 ms … 230.5 ms 3 runs\n>\n> \tBenchmark 2: parallel --ungroup seq ::: 10000000 >/dev/null\n> \t Time (mean ± σ): 154.7 ms ± 0.9 ms [User: 136.2 ms, System: 25.1 ms]\n> \t Range (min … max): 153.9 ms … 155.7 ms 3 runs\n>\n> \tSummary\n> \t 'parallel --ungroup seq ::: 10000000 >/dev/null ' ran\n> \t 1.42 ± 0.06 times faster than 'parallel seq ::: 10000000 >/dev/null '\n\nThis commit message talks a lot about GNU parallel.\n\nIt would make more sense to measure Git with that new mode, though, and\ntalk about that instead.\n\n>\n> A large part of the juggling in the API is to make the API safer for\n> its maintenance and consumers alike.\n>\n> For the maintenance of the API we e.g. avoid malloc()-ing the\n> \"pp->pfd\", ensuring that SANITIZE=address and other similar tools will\n> catch any unexpected misuse.\n>\n> For API consumers we take pains to never pass the non-NULL \"out\"\n> buffer to an API user that provided the \"ungroup\" option. The\n> resulting code in t/helper/test-run-command.c isn't typical of such a\n> user, i.e. they'd typically use one mode or the other, and would know\n> whether they'd provided \"ungroup\" or not.\n>\n> We could also avoid the strbuf_init() for \"buffered_output\" by having\n> \"struct parallel_processes\" use a static PARALLEL_PROCESSES_INIT\n> initializer, but let's leave that cleanup for later.\n>\n> Using a global \"run_processes_parallel_ungroup\" variable to enable\n> this option is rather nasty, but is being done here to produce as\n> minimal of a change as possible for a subsequent regression fix. This\n> change is extracted from a larger initial version[1] which ends up\n> with a better end-state for the API, but in doing so needed to modify\n> all existing callers of the API. Let's defer that for now, and\n> narrowly focus on what we need for fixing the regression in the\n> subsequent commit.\n>\n> It's safe to do this with a global variable because:\n>\n> A) hook.c is the only user of it that sets it to non-zero, and before\n> we'll get any other API users we'll refactor away this method of\n> passing in the option, i.e. re-roll [1].\n>\n> B) Even if hook.c wasn't the only user we don't have callers of this\n> API that concurrently invoke this parallel process starting API\n> itself in parallel.\n>\n> As noted above \"A\" && \"B\" are rather nasty, and we don't want to live\n> with those caveats long-term, but for now they should be an acceptable\n> compromise.\n>\n> 1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> run-command.c | 83 +++++++++++++++++++++++++++++--------\n> run-command.h | 30 ++++++++++----\n> t/helper/test-run-command.c | 19 +++++++--\n> t/t0061-run-command.sh | 35 ++++++++++++++++\n> 4 files changed, 139 insertions(+), 28 deletions(-)\n\nThis is an uncomfortably large diffstat for a patch series that is\nsupposed to fix a regression. That makes it harder than necessary to\nreview, and hence unnecessarily blocks v2.36.2 (at least it was my\nexpectation that we would release that version relatively quickly, as the\nregression fix already missed the v2.36.1 boat).\n\n>\n> diff --git a/run-command.c b/run-command.c\n> index a8501e38ceb..324e9548469 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1471,6 +1471,7 @@ enum child_state {\n> \tGIT_CP_WAIT_CLEANUP,\n> };\n>\n> +int run_processes_parallel_ungroup;\n\nThis global variable seems to exist solely to avoid extending the\nsignature of `run_processes_parallel_tr2()`. Let's not do that.\n\n> struct parallel_processes {\n> \tvoid *data;\n>\n> @@ -1494,6 +1495,7 @@ struct parallel_processes {\n> \tstruct pollfd *pfd;\n>\n> \tunsigned shutdown : 1;\n> +\tunsigned ungroup : 1;\n>\n> \tint output_owner;\n> \tstruct strbuf buffered_output; /* of finished children */\n> @@ -1537,7 +1539,7 @@ static void pp_init(struct parallel_processes *pp,\n> \t\t get_next_task_fn get_next_task,\n> \t\t start_failure_fn start_failure,\n> \t\t task_finished_fn task_finished,\n> -\t\t void *data)\n> +\t\t void *data, const int ungroup)\n> {\n> \tint i;\n>\n> @@ -1559,13 +1561,19 @@ static void pp_init(struct parallel_processes *pp,\n> \tpp->nr_processes = 0;\n> \tpp->output_owner = 0;\n> \tpp->shutdown = 0;\n> +\tpp->ungroup = ungroup;\n> \tCALLOC_ARRAY(pp->children, n);\n> -\tCALLOC_ARRAY(pp->pfd, n);\n> +\tif (pp->ungroup)\n> +\t\tpp->pfd = NULL;\n> +\telse\n> +\t\tCALLOC_ARRAY(pp->pfd, n);\n> \tstrbuf_init(&pp->buffered_output, 0);\n>\n> \tfor (i = 0; i < n; i++) {\n> \t\tstrbuf_init(&pp->children[i].err, 0);\n> \t\tchild_process_init(&pp->children[i].process);\n> +\t\tif (!pp->pfd)\n\nIt would be more logical to test for `pp->ungroup` than for `!pp->pfd`.\nIn other instances below, the patch uses `if (ungroup)` instead. Let's not\nflip-flop between those two conditions, but the latter consistently.\n\n> +\t\t\tcontinue;\n\nThis avoids indenting the following two lines, at the price of\nreadability. The code would be more obvious if it made those two lines\ncontingent upon `!pp->ungroup`.\n\n> \t\tpp->pfd[i].events = POLLIN | POLLHUP;\n> \t\tpp->pfd[i].fd = -1;\n> \t}\n> @@ -1606,6 +1614,7 @@ static void pp_cleanup(struct parallel_processes *pp)\n> */\n> static int pp_start_one(struct parallel_processes *pp)\n> {\n> +\tconst int ungroup = pp->ungroup;\n\nIt costs readers a couple of moments when they stumble over code that is\ninconsistent with the existing code. In this instance, I find very little\nvalue in the `const` qualifier. Actually, this entire line is probably not\nworth having because `pp->ungroup` is just 4 characters longer than\n`ungroup`.\n\nThis same comment applies to another hunk below, too.\n\nThings like this do take focus away from reviewing the interesting part of\nthe contribution, which in particular in the case of a regression fix that\nmany are waiting for is something to avoid.\n\n> \tint i, code;\n>\n> \tfor (i = 0; i < pp->max_processes; i++)\n> @@ -1615,24 +1624,30 @@ static int pp_start_one(struct parallel_processes *pp)\n> \t\tBUG(\"bookkeeping is hard\");\n>\n> \tcode = pp->get_next_task(&pp->children[i].process,\n> -\t\t\t\t &pp->children[i].err,\n> +\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n> \t\t\t\t pp->data,\n> \t\t\t\t &pp->children[i].data);\n> \tif (!code) {\n> -\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> -\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\tif (!ungroup) {\n> +\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> +\t\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\t}\n\nIn contrast to the change in `pp_init()`, this hunk is good, because it\nmakes the intention and implementation quite clear.\n\n> \t\treturn 1;\n> \t}\n> -\tpp->children[i].process.err = -1;\n> -\tpp->children[i].process.stdout_to_stderr = 1;\n> +\tif (!ungroup) {\n> +\t\tpp->children[i].process.err = -1;\n> +\t\tpp->children[i].process.stdout_to_stderr = 1;\n> +\t}\n> \tpp->children[i].process.no_stdin = 1;\n>\n> \tif (start_command(&pp->children[i].process)) {\n> -\t\tcode = pp->start_failure(&pp->children[i].err,\n> +\t\tcode = pp->start_failure(ungroup ? NULL : &pp->children[i].err,\n> \t\t\t\t\t pp->data,\n> \t\t\t\t\t pp->children[i].data);\n> -\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> -\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\tif (!ungroup) {\n> +\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> +\t\t\tstrbuf_reset(&pp->children[i].err);\n> +\t\t}\n> \t\tif (code)\n> \t\t\tpp->shutdown = 1;\n> \t\treturn code;\n> @@ -1640,14 +1655,29 @@ static int pp_start_one(struct parallel_processes *pp)\n>\n> \tpp->nr_processes++;\n> \tpp->children[i].state = GIT_CP_WORKING;\n> -\tpp->pfd[i].fd = pp->children[i].process.err;\n> +\tif (pp->pfd)\n> +\t\tpp->pfd[i].fd = pp->children[i].process.err;\n\nHere, the patch uses `pp->pfd` instead of `pp->ungroup` again. It should\nuse only one of them for the many conditions that are added, not\nflip-flop between them.\n\n> \treturn 0;\n> }\n>\n> +static void pp_mark_ungrouped_for_cleanup(struct parallel_processes *pp)\n> +{\n> +\tint i;\n> +\n> +\tif (!pp->ungroup)\n> +\t\tBUG(\"only reachable if 'ungrouped'\");\n> +\n> +\tfor (i = 0; i < pp->max_processes; i++)\n> +\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n> +}\n\nThis function has but a single caller. It would improve readability to\ninsert the loop directly instead.\n\n> +\n> static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> {\n> \tint i;\n>\n> +\tif (pp->ungroup)\n> +\t\tBUG(\"unreachable with 'ungrouped'\");\n\nBetter: `BUG(\"pp_buffer_stderr() called in ungrouped mode\")`\n\nA similar issue exists in the next hunk.\n\n> +\n> \twhile ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n> \t\tif (errno == EINTR)\n> \t\t\tcontinue;\n> @@ -1674,6 +1704,10 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> static void pp_output(struct parallel_processes *pp)\n> {\n> \tint i = pp->output_owner;\n> +\n> +\tif (pp->ungroup)\n> +\t\tBUG(\"unreachable with 'ungrouped'\");\n> +\n> \tif (pp->children[i].state == GIT_CP_WORKING &&\n> \t pp->children[i].err.len) {\n> \t\tstrbuf_write(&pp->children[i].err, stderr);\n> @@ -1683,6 +1717,7 @@ static void pp_output(struct parallel_processes *pp)\n>\n> static int pp_collect_finished(struct parallel_processes *pp)\n> {\n> +\tconst int ungroup = pp->ungroup;\n> \tint i, code;\n> \tint n = pp->max_processes;\n> \tint result = 0;\n> @@ -1697,8 +1732,8 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> \t\tcode = finish_command(&pp->children[i].process);\n>\n> \t\tcode = pp->task_finished(code,\n> -\t\t\t\t\t &pp->children[i].err, pp->data,\n> -\t\t\t\t\t pp->children[i].data);\n> +\t\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n> +\t\t\t\t\t pp->data, pp->children[i].data);\n>\n> \t\tif (code)\n> \t\t\tresult = code;\n> @@ -1707,10 +1742,13 @@ static int pp_collect_finished(struct parallel_processes *pp)\n>\n> \t\tpp->nr_processes--;\n> \t\tpp->children[i].state = GIT_CP_FREE;\n> -\t\tpp->pfd[i].fd = -1;\n> +\t\tif (pp->pfd)\n> +\t\t\tpp->pfd[i].fd = -1;\n> \t\tchild_process_init(&pp->children[i].process);\n>\n> -\t\tif (i != pp->output_owner) {\n> +\t\tif (ungroup) {\n> +\t\t\t; /* no strbuf_*() work to do here */\n> +\t\t} else if (i != pp->output_owner) {\n> \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n> \t\t\tstrbuf_reset(&pp->children[i].err);\n> \t\t} else {\n> @@ -1748,8 +1786,13 @@ int run_processes_parallel(int n,\n> \tint output_timeout = 100;\n> \tint spawn_cap = 4;\n> \tstruct parallel_processes pp;\n> +\tconst int ungroup = run_processes_parallel_ungroup;\n>\n> -\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n> +\t/* unset for the next API user */\n> +\trun_processes_parallel_ungroup = 0;\n> +\n> +\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n> +\t\tungroup);\n> \twhile (1) {\n> \t\tfor (i = 0;\n> \t\t i < spawn_cap && !pp.shutdown &&\n> @@ -1766,8 +1809,12 @@ int run_processes_parallel(int n,\n> \t\t}\n> \t\tif (!pp.nr_processes)\n> \t\t\tbreak;\n> -\t\tpp_buffer_stderr(&pp, output_timeout);\n> -\t\tpp_output(&pp);\n> +\t\tif (ungroup) {\n> +\t\t\tpp_mark_ungrouped_for_cleanup(&pp);\n> +\t\t} else {\n> +\t\t\tpp_buffer_stderr(&pp, output_timeout);\n> +\t\t\tpp_output(&pp);\n> +\t\t}\n> \t\tcode = pp_collect_finished(&pp);\n> \t\tif (code) {\n> \t\t\tpp.shutdown = 1;\n> diff --git a/run-command.h b/run-command.h\n> index 5bd0c933e80..bf4236f1164 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -405,6 +405,9 @@ void check_pipe(int err);\n> * pp_cb is the callback cookie as passed to run_processes_parallel.\n> * You can store a child process specific callback cookie in pp_task_cb.\n> *\n> + * See run_processes_parallel() below for a discussion of the \"struct\n> + * strbuf *out\" parameter.\n> + *\n> * Even after returning 0 to indicate that there are no more processes,\n> * this function will be called again until there are no more running\n> * child processes.\n> @@ -423,9 +426,8 @@ typedef int (*get_next_task_fn)(struct child_process *cp,\n> * This callback is called whenever there are problems starting\n> * a new process.\n> *\n> - * You must not write to stdout or stderr in this function. Add your\n> - * message to the strbuf out instead, which will be printed without\n> - * messing up the output of the other parallel processes.\n> + * See run_processes_parallel() below for a discussion of the \"struct\n> + * strbuf *out\" parameter.\n> *\n> * pp_cb is the callback cookie as passed into run_processes_parallel,\n> * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> @@ -441,9 +443,8 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n> /**\n> * This callback is called on every child process that finished processing.\n> *\n> - * You must not write to stdout or stderr in this function. Add your\n> - * message to the strbuf out instead, which will be printed without\n> - * messing up the output of the other parallel processes.\n> + * See run_processes_parallel() below for a discussion of the \"struct\n> + * strbuf *out\" parameter.\n> *\n> * pp_cb is the callback cookie as passed into run_processes_parallel,\n> * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> @@ -464,11 +465,26 @@ typedef int (*task_finished_fn)(int result,\n> *\n> * The children started via this function run in parallel. Their output\n> * (both stdout and stderr) is routed to stderr in a manner that output\n> - * from different tasks does not interleave.\n> + * from different tasks does not interleave (but see \"ungroup\" below).\n> *\n> * start_failure_fn and task_finished_fn can be NULL to omit any\n> * special handling.\n> + *\n> + * If the \"ungroup\" option isn't specified, the API will set the\n> + * \"stdout_to_stderr\" parameter in \"struct child_process\" and provide\n> + * the callbacks with a \"struct strbuf *out\" parameter to write output\n> + * to. In this case the callbacks must not write to stdout or\n> + * stderr as such output will mess up the output of the other parallel\n> + * processes. If \"ungroup\" option is specified callbacks will get a\n> + * NULL \"struct strbuf *out\" parameter, and are responsible for\n> + * emitting their own output, including dealing with any race\n> + * conditions due to writing in parallel to stdout and stderr.\n> + * The \"ungroup\" option can be enabled by setting the global\n> + * \"run_processes_parallel_ungroup\" to \"1\" before invoking\n> + * run_processes_parallel(), it will be set back to \"0\" as soon as the\n> + * API reads that setting.\n\nA better idea would be to describe the \"interleaved_output\" mode first,\nand then say \"the rest of this comment deals with the non-interleaved\nmode\".\n\n> */\n> +extern int run_processes_parallel_ungroup;\n> int run_processes_parallel(int n,\n> \t\t\t get_next_task_fn,\n> \t\t\t start_failure_fn,\n> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\n> index f3b90aa834a..6405c9a076a 100644\n> --- a/t/helper/test-run-command.c\n> +++ b/t/helper/test-run-command.c\n> @@ -31,7 +31,11 @@ static int parallel_next(struct child_process *cp,\n> \t\treturn 0;\n>\n> \tstrvec_pushv(&cp->args, d->args.v);\n> -\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n> +\tif (err)\n> +\t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n> +\telse\n> +\t\tfprintf(stderr, \"preloaded output of a child\\n\");\n> +\n> \tnumber_callbacks++;\n> \treturn 1;\n> }\n> @@ -41,7 +45,10 @@ static int no_job(struct child_process *cp,\n> \t\t void *cb,\n> \t\t void **task_cb)\n> {\n> -\tstrbuf_addstr(err, \"no further jobs available\\n\");\n> +\tif (err)\n> +\t\tstrbuf_addstr(err, \"no further jobs available\\n\");\n> +\telse\n> +\t\tfprintf(stderr, \"no further jobs available\\n\");\n> \treturn 0;\n> }\n>\n> @@ -50,7 +57,10 @@ static int task_finished(int result,\n> \t\t\t void *pp_cb,\n> \t\t\t void *pp_task_cb)\n> {\n> -\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n> +\tif (err)\n> +\t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n> +\telse\n> +\t\tfprintf(stderr, \"asking for a quick stop\\n\");\n\nThis `if (err) strbuf_add... else fprintf...` pattern is a bit repetitive,\nbut in the interest of fixing a regression without risking to introduce\nanother regression, I agree that this should wait for later to be\naddressed.\n\nOn the other hand, this issue is indicating that the API could be designed\nbetter, e.g. by letting the `parallel_processes` struct provide a callback\nfunction for printing or buffering messages.\n\nAt this stage, it is concerning that we introduce a feature that\nneeds to be designed well and therefore needs more time to be fleshed out,\nwhen a regression fix rides on it that should be integrated swiftly and\ndoes not allow for said time to flesh things out.\n\nIt is really unfortunate that the hook changes that made it into v2.36.0\nare in such an unrevertable state. It would really make most sense, as\nJunio suggested elsewhere in this thread, to roll back the changes that\nintroduced the regression, then spend the time it actually takes to design\nthe feature properly how hooks could be run via the\n`run_processes_parallel()` API. Or spend the time to figure out that not\nusing the parallel API at all might be the best course of action, instead\nexecuting the hook directly, using the standard `run_command()` API. That\nmay very well turn out to avoid some over-engineering, as an added benefit.\n\n> \treturn 1;\n> }\n>\n> @@ -411,6 +421,9 @@ int cmd__run_command(int argc, const char **argv)\n> \tstrvec_clear(&proc.args);\n> \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n>\n> +\tif (getenv(\"RUN_PROCESSES_PARALLEL_UNGROUP\"))\n\nApart from using a naming scheme that is inconsistent with how Git does\nsimilar things elsewhere, this environment variable seems to have been\nintroduced for the sole purpose of testing the `ungroup` mode.\n\nLet's instead introduce a new `test-tool run-command` subcommand for that\nspecific purpose.\n\n> +\t\trun_processes_parallel_ungroup = 1;\n> +\n> \tif (!strcmp(argv[1], \"run-command-parallel\"))\n> \t\texit(run_processes_parallel(jobs, parallel_next,\n> \t\t\t\t\t NULL, NULL, &proc));\n> diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\n> index ee281909bc3..69ccaa8d298 100755\n> --- a/t/t0061-run-command.sh\n> +++ b/t/t0061-run-command.sh\n> @@ -134,16 +134,37 @@ test_expect_success 'run_command runs in parallel with more jobs available than\n> \ttest_cmp expect actual\n> '\n>\n> +test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n> +\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n> +\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_line_count = 8 out &&\n> +\ttest_line_count = 4 err\n\nGood. Testing for the line count avoids getting confused by interleaved\noutput.\n\nHaving said that, adding seven (!) new test cases merely to verify that\nthe `ungroup` mode does not break things is a bit excessive, in particular\nsince none of the test cases seem to _actually_ verify that the output is\ninterleaved. Remember, adding test cases is not free.\n\nCiao,\nJohannes\n\n> +'\n> +\n> test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n> \ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n> \ttest_cmp expect actual\n> '\n>\n> +test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n> +\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n> +\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_line_count = 8 out &&\n> +\ttest_line_count = 4 err\n> +'\n> +\n> test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n> \ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n> \ttest_cmp expect actual\n> '\n>\n> +test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n> +\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n> +\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_line_count = 8 out &&\n> +\ttest_line_count = 4 err\n> +'\n> +\n> cat >expect <<-EOF\n> preloaded output of a child\n> asking for a quick stop\n> @@ -158,6 +179,13 @@ test_expect_success 'run_command is asked to abort gracefully' '\n> \ttest_cmp expect actual\n> '\n>\n> +test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n> +\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n> +\ttest-tool run-command run-command-abort 3 false >out 2>err &&\n> +\ttest_must_be_empty out &&\n> +\ttest_line_count = 6 err\n> +'\n> +\n> cat >expect <<-EOF\n> no further jobs available\n> EOF\n> @@ -167,6 +195,13 @@ test_expect_success 'run_command outputs ' '\n> \ttest_cmp expect actual\n> '\n>\n> +test_expect_success 'run_command outputs (ungroup) ' '\n> +\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n> +\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_must_be_empty out &&\n> +\ttest_cmp expect err\n> +'\n> +\n> test_trace () {\n> \texpect=\"$1\"\n> \tshift\n> --\n> 2.36.1.1103.g036c05811b0\n>\n>\n"},{"id":"456452","messageId":"nycvar.QRO.7.76.6.2206011850460.349@tvgsbejvaqbjf.bet","threadId":"57764","inReplyTo":"cover-v4-0.2-00000000000-20220531T173005Z-avarab@gmail.com","subject":"Re: [PATCH v4 0/2] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-06-01T16:53:51Z","receivedAt":"2022-06-01T16:54:09Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Tue, 31 May 2022, Ævar Arnfjörð Bjarmason wrote:\n\n> Ævar Arnfjörð Bjarmason (2):\n> run-command: add an \"ungroup\" option to run_process_parallel()\n> hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n\nAs I mentioned in the review of the first patch, this introduces a feature\nwith enough code that it is quite easy for more regressions to lurk in\nthere.\n\nOne thing that is notably missing from the cover letter is a discussion\nhow the current approach compares to either reverting the patches that\nintroduced the regression or alternatively patching the code in `hook.c`\nto avoid using the `run_processes_parallel()` API altogether.\n\nCiao,\nJohannes\n"},{"id":"456454","messageId":"xmqqczfstioj.fsf@gitster.g","threadId":"57764","inReplyTo":"nycvar.QRO.7.76.6.2206011630400.349@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v4 1/2] run-command: add an \"ungroup\" option to run_process_parallel()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-06-01T17:09:00Z","receivedAt":"2022-06-01T17:09:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n>> diff --git a/run-command.c b/run-command.c\n>> index a8501e38ceb..324e9548469 100644\n>> --- a/run-command.c\n>> +++ b/run-command.c\n>> @@ -1471,6 +1471,7 @@ enum child_state {\n>> \tGIT_CP_WAIT_CLEANUP,\n>> };\n>>\n>> +int run_processes_parallel_ungroup;\n>\n> This global variable seems to exist solely to avoid extending the\n> signature of `run_processes_parallel_tr2()`. Let's not do that.\n\nIt may make the change even noisier, though.\n\n>> @@ -1537,7 +1539,7 @@ static void pp_init(struct parallel_processes *pp,\n>> \t\t get_next_task_fn get_next_task,\n>> \t\t start_failure_fn start_failure,\n>> \t\t task_finished_fn task_finished,\n>> -\t\t void *data)\n>> +\t\t void *data, const int ungroup)\n>> {\n>> \tint i;\n\nMarking incoming parameter as const is probably a misfeature in C,\nbut doing so with file-scope static would not hurt too much, so if\nthis series needs no further reroll, I'd let it pass.\n\n\n>> \tfor (i = 0; i < n; i++) {\n>> \t\tstrbuf_init(&pp->children[i].err, 0);\n>> \t\tchild_process_init(&pp->children[i].process);\n>> +\t\tif (!pp->pfd)\n>\n> It would be more logical to test for `pp->ungroup` than for `!pp->pfd`.\n> In other instances below, the patch uses `if (ungroup)` instead. Let's not\n> flip-flop between those two conditions, but the latter consistently.\n\nI'd be somewhat sympathetic to the aversion to \"flip-flop\", but I\nstrongly disagree with you here.\n\n\"ungroup\" does not have to stay to be the only reason why we do not\nallocate the pp->pfd[] array, and what we care here is \"if we are\npolling for events, then do this initialization to the array\", not\n\"if ungroup -> we must not have the pfd[] array -> so let's skip\nit\". We do not have to add more code that depends on that two step\ninference when we do not need to.\n\n>> @@ -1606,6 +1614,7 @@ static void pp_cleanup(struct parallel_processes *pp)\n>> */\n>> static int pp_start_one(struct parallel_processes *pp)\n>> {\n>> +\tconst int ungroup = pp->ungroup;\n>\n> It costs readers a couple of moments when they stumble over code that is\n> inconsistent with the existing code. In this instance, I find very little\n> value in the `const` qualifier. Actually, this entire line is probably not\n> worth having because `pp->ungroup` is just 4 characters longer than\n> `ungroup`.\n>\n> This same comment applies to another hunk below, too.\n>\n> Things like this do take focus away from reviewing the interesting part of\n> the contribution, which in particular in the case of a regression fix that\n> many are waiting for is something to avoid.\n\nOK.\n\nThanks.\n"},{"id":"456516","messageId":"cover-v5-0.2-00000000000-20220602T131858Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v4-0.2-00000000000-20220531T173005Z-avarab@gmail.com","subject":"[PATCH v5 0/2] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-06-02T14:07:55Z","receivedAt":"2022-06-02T14:08:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This series fixes a v2.36.0 regression[1]. See [2] for the v4. The\nreasons for why a regression needs this relatively large change to\nmove forward is discussed in past rounds, e.g. around [3]. CI at\nhttps://github.com/avar/git/actions/runs/2428475773\n\nChanges since v4, mainly to address comments by Johannes (thanks for\nthe review!):\n\n * First, some things like renaming \"ungroup\" to something else &\n rewriting the tests I didn't do because I thought keeping the\n inter/range-diff down in size outweighed re-arranging or changing\n the code at this late stage.\n\n In the case of the suggested shorter test in\n https://lore.kernel.org/git/nycvar.QRO.7.76.6.2206011827300.349@tvgsbejvaqbjf.bet/\n the replacement wasn't testing the same thing. I.e. we don't see\n what's connected to a TTY if we redirect one of stdout or stderr\n anymore, which is important to get right.\n\n * Ditto the suggestion to e.g. add a parameter for \"ungroup\". I agree\n that's better, but that approach was in the earlier and much larger\n round[4], here we're trying to aim for the smallest possible\n regression fix by line count & complexity.\n\n * I retained the performance test(s) for \"parallel\" and \"git hook\n run\" in 1/2 and 2/2. Yes, the former isn't ours, but I think it\n helps to explain the code, implementation and resulting performance\n with reference to existing well-known software that's doing the\n exact same thing we're doing here.\n\n * Stopped using \"const\" in \"const int ungroup\", and dropped some of\n those variables entirely.\n\n * Inlined the pp_mark_ungrouped_for_cleanup() function. I added an\n \"int i\" in the inner scope in run_processes_parallel() even though\n we have one in the outer, just to make it clear that we're not\n caring about the other one (or clobbering it).\n\n * I just got rid of the two added BUG(). It's obvious enough from the\n calling code that those two functions are !ungroup only, so we can\n do without the sprinkling of BUG() and larger resulting diff.\n\n * Passed an --ungroup parameter in the tests instead of passing a\n parameter by environment variable.\n\n * Fixed a minor s/reported in/reported against/ phrasing in the 2/2\n commit message.\n\n1. https://lore.kernel.org/git/CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com/\n2. https://lore.kernel.org/git/cover-v4-0.2-00000000000-20220531T173005Z-avarab@gmail.com/\n3. https://lore.kernel.org/git/220526.86pmk060xa.gmgdl@evledraar.gmail.com/\n4. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n\nÆvar Arnfjörð Bjarmason (2):\n run-command: add an \"ungroup\" option to run_process_parallel()\n hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n\n hook.c | 1 +\n run-command.c | 70 +++++++++++++++++++++++++++----------\n run-command.h | 30 ++++++++++++----\n t/helper/test-run-command.c | 22 ++++++++++--\n t/t0061-run-command.sh | 30 ++++++++++++++++\n t/t1800-hook.sh | 37 ++++++++++++++++++++\n 6 files changed, 161 insertions(+), 29 deletions(-)\n\nRange-diff against v4:\n1: f1170b02553 ! 1: d018b7c4441 run-command: add an \"ungroup\" option to run_process_parallel()\n @@ Commit message\n NTTY\n \n Another is as GNU parallel's documentation notes a potential for\n - optimization. Our results will be a bit different, but in cases where\n + optimization. As demonstrated in next commit our results with \"git\n + hook run\" will be similar, but generally speaking this shows that if\n you want to run processes in parallel where the exact order isn't\n important this can be a lot faster:\n \n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n \t\t start_failure_fn start_failure,\n \t\t task_finished_fn task_finished,\n -\t\t void *data)\n -+\t\t void *data, const int ungroup)\n ++\t\t void *data, int ungroup)\n {\n \tint i;\n \n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n \tfor (i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n \t\tchild_process_init(&pp->children[i].process);\n -+\t\tif (!pp->pfd)\n -+\t\t\tcontinue;\n - \t\tpp->pfd[i].events = POLLIN | POLLHUP;\n - \t\tpp->pfd[i].fd = -1;\n +-\t\tpp->pfd[i].events = POLLIN | POLLHUP;\n +-\t\tpp->pfd[i].fd = -1;\n ++\t\tif (pp->pfd) {\n ++\t\t\tpp->pfd[i].events = POLLIN | POLLHUP;\n ++\t\t\tpp->pfd[i].fd = -1;\n ++\t\t}\n \t}\n -@@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)\n - */\n - static int pp_start_one(struct parallel_processes *pp)\n - {\n -+\tconst int ungroup = pp->ungroup;\n - \tint i, code;\n \n - \tfor (i = 0; i < pp->max_processes; i++)\n + \tpp_for_signal = pp;\n @@ run-command.c: static int pp_start_one(struct parallel_processes *pp)\n \t\tBUG(\"bookkeeping is hard\");\n \n \tcode = pp->get_next_task(&pp->children[i].process,\n -\t\t\t\t &pp->children[i].err,\n -+\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n ++\t\t\t\t pp->ungroup ? NULL : &pp->children[i].err,\n \t\t\t\t pp->data,\n \t\t\t\t &pp->children[i].data);\n \tif (!code) {\n -\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n -\t\tstrbuf_reset(&pp->children[i].err);\n -+\t\tif (!ungroup) {\n ++\t\tif (!pp->ungroup) {\n +\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n +\t\t\tstrbuf_reset(&pp->children[i].err);\n +\t\t}\n @@ run-command.c: static int pp_start_one(struct parallel_processes *pp)\n \t}\n -\tpp->children[i].process.err = -1;\n -\tpp->children[i].process.stdout_to_stderr = 1;\n -+\tif (!ungroup) {\n ++\tif (!pp->ungroup) {\n +\t\tpp->children[i].process.err = -1;\n +\t\tpp->children[i].process.stdout_to_stderr = 1;\n +\t}\n @@ run-command.c: static int pp_start_one(struct parallel_processes *pp)\n \n \tif (start_command(&pp->children[i].process)) {\n -\t\tcode = pp->start_failure(&pp->children[i].err,\n -+\t\tcode = pp->start_failure(ungroup ? NULL : &pp->children[i].err,\n ++\t\tcode = pp->start_failure(pp->ungroup ? NULL :\n ++\t\t\t\t\t &pp->children[i].err,\n \t\t\t\t\t pp->data,\n \t\t\t\t\t pp->children[i].data);\n -\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n -\t\tstrbuf_reset(&pp->children[i].err);\n -+\t\tif (!ungroup) {\n ++\t\tif (!pp->ungroup) {\n +\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n +\t\t\tstrbuf_reset(&pp->children[i].err);\n +\t\t}\n @@ run-command.c: static int pp_start_one(struct parallel_processes *pp)\n \treturn 0;\n }\n \n -+static void pp_mark_ungrouped_for_cleanup(struct parallel_processes *pp)\n -+{\n -+\tint i;\n -+\n -+\tif (!pp->ungroup)\n -+\t\tBUG(\"only reachable if 'ungrouped'\");\n -+\n -+\tfor (i = 0; i < pp->max_processes; i++)\n -+\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n -+}\n -+\n - static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n - {\n - \tint i;\n - \n -+\tif (pp->ungroup)\n -+\t\tBUG(\"unreachable with 'ungrouped'\");\n -+\n - \twhile ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n - \t\tif (errno == EINTR)\n - \t\t\tcontinue;\n @@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n static void pp_output(struct parallel_processes *pp)\n {\n \tint i = pp->output_owner;\n -+\n -+\tif (pp->ungroup)\n -+\t\tBUG(\"unreachable with 'ungrouped'\");\n +\n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n \t\tstrbuf_write(&pp->children[i].err, stderr);\n -@@ run-command.c: static void pp_output(struct parallel_processes *pp)\n - \n - static int pp_collect_finished(struct parallel_processes *pp)\n - {\n -+\tconst int ungroup = pp->ungroup;\n - \tint i, code;\n - \tint n = pp->max_processes;\n - \tint result = 0;\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n + \n \t\tcode = finish_command(&pp->children[i].process);\n \n - \t\tcode = pp->task_finished(code,\n --\t\t\t\t\t &pp->children[i].err, pp->data,\n --\t\t\t\t\t pp->children[i].data);\n -+\t\t\t\t\t ungroup ? NULL : &pp->children[i].err,\n -+\t\t\t\t\t pp->data, pp->children[i].data);\n +-\t\tcode = pp->task_finished(code,\n ++\t\tcode = pp->task_finished(code, pp->ungroup ? NULL :\n + \t\t\t\t\t &pp->children[i].err, pp->data,\n + \t\t\t\t\t pp->children[i].data);\n \n - \t\tif (code)\n - \t\t\tresult = code;\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n \n \t\tpp->nr_processes--;\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n \t\tchild_process_init(&pp->children[i].process);\n \n -\t\tif (i != pp->output_owner) {\n -+\t\tif (ungroup) {\n ++\t\tif (pp->ungroup) {\n +\t\t\t; /* no strbuf_*() work to do here */\n +\t\t} else if (i != pp->output_owner) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t} else {\n @@ run-command.c: int run_processes_parallel(int n,\n + \tint i, code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n ++\tint ungroup = run_processes_parallel_ungroup;\n \tstruct parallel_processes pp;\n -+\tconst int ungroup = run_processes_parallel_ungroup;\n \n -\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n +\t/* unset for the next API user */\n @@ run-command.c: int run_processes_parallel(int n,\n -\t\tpp_buffer_stderr(&pp, output_timeout);\n -\t\tpp_output(&pp);\n +\t\tif (ungroup) {\n -+\t\t\tpp_mark_ungrouped_for_cleanup(&pp);\n ++\t\t\tint i;\n ++\n ++\t\t\tfor (i = 0; i < pp.max_processes; i++)\n ++\t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n +\t\t} else {\n +\t\t\tpp_buffer_stderr(&pp, output_timeout);\n +\t\t\tpp_output(&pp);\n @@ t/helper/test-run-command.c: static int task_finished(int result,\n }\n \n @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)\n - \tstrvec_clear(&proc.args);\n - \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n + \tif (!strcmp(argv[1], \"run-command\"))\n + \t\texit(run_command(&proc));\n \n -+\tif (getenv(\"RUN_PROCESSES_PARALLEL_UNGROUP\"))\n ++\tif (!strcmp(argv[1], \"--ungroup\")) {\n ++\t\targv += 1;\n ++\t\targc -= 1;\n +\t\trun_processes_parallel_ungroup = 1;\n ++\t}\n +\n - \tif (!strcmp(argv[1], \"run-command-parallel\"))\n - \t\texit(run_processes_parallel(jobs, parallel_next,\n - \t\t\t\t\t NULL, NULL, &proc));\n + \tjobs = atoi(argv[2]);\n + \tstrvec_clear(&proc.args);\n + \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n ## t/t0061-run-command.sh ##\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with more jobs available than\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with m\n '\n \n +test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n -+\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n -+\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n ++\ttest-tool run-command --ungroup run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_line_count = 8 out &&\n +\ttest_line_count = 4 err\n +'\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with m\n '\n \n +test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n -+\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n -+\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n ++\ttest-tool run-command --ungroup run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_line_count = 8 out &&\n +\ttest_line_count = 4 err\n +'\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with m\n '\n \n +test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n -+\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n -+\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n ++\ttest-tool run-command --ungroup run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_line_count = 8 out &&\n +\ttest_line_count = 4 err\n +'\n @@ t/t0061-run-command.sh: test_expect_success 'run_command is asked to abort grace\n '\n \n +test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n -+\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n -+\ttest-tool run-command run-command-abort 3 false >out 2>err &&\n ++\ttest-tool run-command --ungroup run-command-abort 3 false >out 2>err &&\n +\ttest_must_be_empty out &&\n +\ttest_line_count = 6 err\n +'\n @@ t/t0061-run-command.sh: test_expect_success 'run_command outputs ' '\n '\n \n +test_expect_success 'run_command outputs (ungroup) ' '\n -+\tRUN_PROCESSES_PARALLEL_UNGROUP=1 \\\n -+\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n ++\ttest-tool run-command --ungroup run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_must_be_empty out &&\n +\ttest_cmp expect err\n +'\n2: 8ab09f28729 ! 2: b0f0dc7492a hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n @@ Metadata\n ## Commit message ##\n hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n \n - Fix a regression reported[1] in f443246b9f2 (commit: convert\n + Fix a regression reported[1] against f443246b9f2 (commit: convert\n {pre-commit,prepare-commit-msg} hook to hook.h, 2021-12-22): Due to\n using the run_process_parallel() API in the earlier 96e7225b310 (hook:\n add 'run' subcommand, 2021-12-22) we'd capture the hook's stderr and\n-- \n2.36.1.1103.gb3ecdfb3e6a\n\n"},{"id":"456517","messageId":"patch-v5-1.2-d018b7c4441-20220602T131858Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v5-0.2-00000000000-20220602T131858Z-avarab@gmail.com","subject":"[PATCH v5 1/2] run-command: add an \"ungroup\" option to run_process_parallel()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-06-02T14:07:56Z","receivedAt":"2022-06-02T14:08:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Extend the parallel execution API added in c553c72eed6 (run-command:\nadd an asynchronous parallel child processor, 2015-12-15) to support a\nmode where the stdout and stderr of the processes isn't captured and\noutput in a deterministic order, instead we'll leave it to the kernel\nand stdio to sort it out.\n\nThis gives the API same functionality as GNU parallel's --ungroup\noption. As we'll see in a subsequent commit the main reason to want\nthis is to support stdout and stderr being connected to the TTY in the\ncase of jobs=1, demonstrated here with GNU parallel:\n\n\t$ parallel --ungroup 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n\tTTY\n\tTTY\n\t$ parallel 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n\tNTTY\n\tNTTY\n\nAnother is as GNU parallel's documentation notes a potential for\noptimization. As demonstrated in next commit our results with \"git\nhook run\" will be similar, but generally speaking this shows that if\nyou want to run processes in parallel where the exact order isn't\nimportant this can be a lot faster:\n\n\t$ hyperfine -r 3 -L o ,--ungroup 'parallel {o} seq ::: 10000000 >/dev/null '\n\tBenchmark 1: parallel seq ::: 10000000 >/dev/null\n\t Time (mean ± σ): 220.2 ms ± 9.3 ms [User: 124.9 ms, System: 96.1 ms]\n\t Range (min … max): 212.3 ms … 230.5 ms 3 runs\n\n\tBenchmark 2: parallel --ungroup seq ::: 10000000 >/dev/null\n\t Time (mean ± σ): 154.7 ms ± 0.9 ms [User: 136.2 ms, System: 25.1 ms]\n\t Range (min … max): 153.9 ms … 155.7 ms 3 runs\n\n\tSummary\n\t 'parallel --ungroup seq ::: 10000000 >/dev/null ' ran\n\t 1.42 ± 0.06 times faster than 'parallel seq ::: 10000000 >/dev/null '\n\nA large part of the juggling in the API is to make the API safer for\nits maintenance and consumers alike.\n\nFor the maintenance of the API we e.g. avoid malloc()-ing the\n\"pp->pfd\", ensuring that SANITIZE=address and other similar tools will\ncatch any unexpected misuse.\n\nFor API consumers we take pains to never pass the non-NULL \"out\"\nbuffer to an API user that provided the \"ungroup\" option. The\nresulting code in t/helper/test-run-command.c isn't typical of such a\nuser, i.e. they'd typically use one mode or the other, and would know\nwhether they'd provided \"ungroup\" or not.\n\nWe could also avoid the strbuf_init() for \"buffered_output\" by having\n\"struct parallel_processes\" use a static PARALLEL_PROCESSES_INIT\ninitializer, but let's leave that cleanup for later.\n\nUsing a global \"run_processes_parallel_ungroup\" variable to enable\nthis option is rather nasty, but is being done here to produce as\nminimal of a change as possible for a subsequent regression fix. This\nchange is extracted from a larger initial version[1] which ends up\nwith a better end-state for the API, but in doing so needed to modify\nall existing callers of the API. Let's defer that for now, and\nnarrowly focus on what we need for fixing the regression in the\nsubsequent commit.\n\nIt's safe to do this with a global variable because:\n\n A) hook.c is the only user of it that sets it to non-zero, and before\n we'll get any other API users we'll refactor away this method of\n passing in the option, i.e. re-roll [1].\n\n B) Even if hook.c wasn't the only user we don't have callers of this\n API that concurrently invoke this parallel process starting API\n itself in parallel.\n\nAs noted above \"A\" && \"B\" are rather nasty, and we don't want to live\nwith those caveats long-term, but for now they should be an acceptable\ncompromise.\n\n1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 70 +++++++++++++++++++++++++++----------\n run-command.h | 30 ++++++++++++----\n t/helper/test-run-command.c | 22 ++++++++++--\n t/t0061-run-command.sh | 30 ++++++++++++++++\n 4 files changed, 123 insertions(+), 29 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex a8501e38ceb..7ab2dd28f3c 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1471,6 +1471,7 @@ enum child_state {\n \tGIT_CP_WAIT_CLEANUP,\n };\n \n+int run_processes_parallel_ungroup;\n struct parallel_processes {\n \tvoid *data;\n \n@@ -1494,6 +1495,7 @@ struct parallel_processes {\n \tstruct pollfd *pfd;\n \n \tunsigned shutdown : 1;\n+\tunsigned ungroup : 1;\n \n \tint output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n@@ -1537,7 +1539,7 @@ static void pp_init(struct parallel_processes *pp,\n \t\t get_next_task_fn get_next_task,\n \t\t start_failure_fn start_failure,\n \t\t task_finished_fn task_finished,\n-\t\t void *data)\n+\t\t void *data, int ungroup)\n {\n \tint i;\n \n@@ -1559,15 +1561,21 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->nr_processes = 0;\n \tpp->output_owner = 0;\n \tpp->shutdown = 0;\n+\tpp->ungroup = ungroup;\n \tCALLOC_ARRAY(pp->children, n);\n-\tCALLOC_ARRAY(pp->pfd, n);\n+\tif (pp->ungroup)\n+\t\tpp->pfd = NULL;\n+\telse\n+\t\tCALLOC_ARRAY(pp->pfd, n);\n \tstrbuf_init(&pp->buffered_output, 0);\n \n \tfor (i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n \t\tchild_process_init(&pp->children[i].process);\n-\t\tpp->pfd[i].events = POLLIN | POLLHUP;\n-\t\tpp->pfd[i].fd = -1;\n+\t\tif (pp->pfd) {\n+\t\t\tpp->pfd[i].events = POLLIN | POLLHUP;\n+\t\t\tpp->pfd[i].fd = -1;\n+\t\t}\n \t}\n \n \tpp_for_signal = pp;\n@@ -1615,24 +1623,31 @@ static int pp_start_one(struct parallel_processes *pp)\n \t\tBUG(\"bookkeeping is hard\");\n \n \tcode = pp->get_next_task(&pp->children[i].process,\n-\t\t\t\t &pp->children[i].err,\n+\t\t\t\t pp->ungroup ? NULL : &pp->children[i].err,\n \t\t\t\t pp->data,\n \t\t\t\t &pp->children[i].data);\n \tif (!code) {\n-\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n-\t\tstrbuf_reset(&pp->children[i].err);\n+\t\tif (!pp->ungroup) {\n+\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n+\t\t\tstrbuf_reset(&pp->children[i].err);\n+\t\t}\n \t\treturn 1;\n \t}\n-\tpp->children[i].process.err = -1;\n-\tpp->children[i].process.stdout_to_stderr = 1;\n+\tif (!pp->ungroup) {\n+\t\tpp->children[i].process.err = -1;\n+\t\tpp->children[i].process.stdout_to_stderr = 1;\n+\t}\n \tpp->children[i].process.no_stdin = 1;\n \n \tif (start_command(&pp->children[i].process)) {\n-\t\tcode = pp->start_failure(&pp->children[i].err,\n+\t\tcode = pp->start_failure(pp->ungroup ? NULL :\n+\t\t\t\t\t &pp->children[i].err,\n \t\t\t\t\t pp->data,\n \t\t\t\t\t pp->children[i].data);\n-\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n-\t\tstrbuf_reset(&pp->children[i].err);\n+\t\tif (!pp->ungroup) {\n+\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n+\t\t\tstrbuf_reset(&pp->children[i].err);\n+\t\t}\n \t\tif (code)\n \t\t\tpp->shutdown = 1;\n \t\treturn code;\n@@ -1640,7 +1655,8 @@ static int pp_start_one(struct parallel_processes *pp)\n \n \tpp->nr_processes++;\n \tpp->children[i].state = GIT_CP_WORKING;\n-\tpp->pfd[i].fd = pp->children[i].process.err;\n+\tif (pp->pfd)\n+\t\tpp->pfd[i].fd = pp->children[i].process.err;\n \treturn 0;\n }\n \n@@ -1674,6 +1690,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n static void pp_output(struct parallel_processes *pp)\n {\n \tint i = pp->output_owner;\n+\n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n \t\tstrbuf_write(&pp->children[i].err, stderr);\n@@ -1696,7 +1713,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \n \t\tcode = finish_command(&pp->children[i].process);\n \n-\t\tcode = pp->task_finished(code,\n+\t\tcode = pp->task_finished(code, pp->ungroup ? NULL :\n \t\t\t\t\t &pp->children[i].err, pp->data,\n \t\t\t\t\t pp->children[i].data);\n \n@@ -1707,10 +1724,13 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \n \t\tpp->nr_processes--;\n \t\tpp->children[i].state = GIT_CP_FREE;\n-\t\tpp->pfd[i].fd = -1;\n+\t\tif (pp->pfd)\n+\t\t\tpp->pfd[i].fd = -1;\n \t\tchild_process_init(&pp->children[i].process);\n \n-\t\tif (i != pp->output_owner) {\n+\t\tif (pp->ungroup) {\n+\t\t\t; /* no strbuf_*() work to do here */\n+\t\t} else if (i != pp->output_owner) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t} else {\n@@ -1747,9 +1767,14 @@ int run_processes_parallel(int n,\n \tint i, code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n+\tint ungroup = run_processes_parallel_ungroup;\n \tstruct parallel_processes pp;\n \n-\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n+\t/* unset for the next API user */\n+\trun_processes_parallel_ungroup = 0;\n+\n+\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n+\t\tungroup);\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1766,8 +1791,15 @@ int run_processes_parallel(int n,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tpp_buffer_stderr(&pp, output_timeout);\n-\t\tpp_output(&pp);\n+\t\tif (ungroup) {\n+\t\t\tint i;\n+\n+\t\t\tfor (i = 0; i < pp.max_processes; i++)\n+\t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n+\t\t} else {\n+\t\t\tpp_buffer_stderr(&pp, output_timeout);\n+\t\t\tpp_output(&pp);\n+\t\t}\n \t\tcode = pp_collect_finished(&pp);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\ndiff --git a/run-command.h b/run-command.h\nindex 5bd0c933e80..bf4236f1164 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -405,6 +405,9 @@ void check_pipe(int err);\n * pp_cb is the callback cookie as passed to run_processes_parallel.\n * You can store a child process specific callback cookie in pp_task_cb.\n *\n+ * See run_processes_parallel() below for a discussion of the \"struct\n+ * strbuf *out\" parameter.\n+ *\n * Even after returning 0 to indicate that there are no more processes,\n * this function will be called again until there are no more running\n * child processes.\n@@ -423,9 +426,8 @@ typedef int (*get_next_task_fn)(struct child_process *cp,\n * This callback is called whenever there are problems starting\n * a new process.\n *\n- * You must not write to stdout or stderr in this function. Add your\n- * message to the strbuf out instead, which will be printed without\n- * messing up the output of the other parallel processes.\n+ * See run_processes_parallel() below for a discussion of the \"struct\n+ * strbuf *out\" parameter.\n *\n * pp_cb is the callback cookie as passed into run_processes_parallel,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n@@ -441,9 +443,8 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n /**\n * This callback is called on every child process that finished processing.\n *\n- * You must not write to stdout or stderr in this function. Add your\n- * message to the strbuf out instead, which will be printed without\n- * messing up the output of the other parallel processes.\n+ * See run_processes_parallel() below for a discussion of the \"struct\n+ * strbuf *out\" parameter.\n *\n * pp_cb is the callback cookie as passed into run_processes_parallel,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n@@ -464,11 +465,26 @@ typedef int (*task_finished_fn)(int result,\n *\n * The children started via this function run in parallel. Their output\n * (both stdout and stderr) is routed to stderr in a manner that output\n- * from different tasks does not interleave.\n+ * from different tasks does not interleave (but see \"ungroup\" below).\n *\n * start_failure_fn and task_finished_fn can be NULL to omit any\n * special handling.\n+ *\n+ * If the \"ungroup\" option isn't specified, the API will set the\n+ * \"stdout_to_stderr\" parameter in \"struct child_process\" and provide\n+ * the callbacks with a \"struct strbuf *out\" parameter to write output\n+ * to. In this case the callbacks must not write to stdout or\n+ * stderr as such output will mess up the output of the other parallel\n+ * processes. If \"ungroup\" option is specified callbacks will get a\n+ * NULL \"struct strbuf *out\" parameter, and are responsible for\n+ * emitting their own output, including dealing with any race\n+ * conditions due to writing in parallel to stdout and stderr.\n+ * The \"ungroup\" option can be enabled by setting the global\n+ * \"run_processes_parallel_ungroup\" to \"1\" before invoking\n+ * run_processes_parallel(), it will be set back to \"0\" as soon as the\n+ * API reads that setting.\n */\n+extern int run_processes_parallel_ungroup;\n int run_processes_parallel(int n,\n \t\t\t get_next_task_fn,\n \t\t\t start_failure_fn,\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex f3b90aa834a..34cce45b584 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -31,7 +31,11 @@ static int parallel_next(struct child_process *cp,\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->args.v);\n-\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n+\telse\n+\t\tfprintf(stderr, \"preloaded output of a child\\n\");\n+\n \tnumber_callbacks++;\n \treturn 1;\n }\n@@ -41,7 +45,10 @@ static int no_job(struct child_process *cp,\n \t\t void *cb,\n \t\t void **task_cb)\n {\n-\tstrbuf_addstr(err, \"no further jobs available\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"no further jobs available\\n\");\n+\telse\n+\t\tfprintf(stderr, \"no further jobs available\\n\");\n \treturn 0;\n }\n \n@@ -50,7 +57,10 @@ static int task_finished(int result,\n \t\t\t void *pp_cb,\n \t\t\t void *pp_task_cb)\n {\n-\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n+\telse\n+\t\tfprintf(stderr, \"asking for a quick stop\\n\");\n \treturn 1;\n }\n \n@@ -407,6 +417,12 @@ int cmd__run_command(int argc, const char **argv)\n \tif (!strcmp(argv[1], \"run-command\"))\n \t\texit(run_command(&proc));\n \n+\tif (!strcmp(argv[1], \"--ungroup\")) {\n+\t\targv += 1;\n+\t\targc -= 1;\n+\t\trun_processes_parallel_ungroup = 1;\n+\t}\n+\n \tjobs = atoi(argv[2]);\n \tstrvec_clear(&proc.args);\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex ee281909bc3..7b5423eebda 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -134,16 +134,34 @@ test_expect_success 'run_command runs in parallel with more jobs available than\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n+\ttest-tool run-command --ungroup run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n \ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n+\ttest-tool run-command --ungroup run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n \ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n+\ttest-tool run-command --ungroup run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n@@ -158,6 +176,12 @@ test_expect_success 'run_command is asked to abort gracefully' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n+\ttest-tool run-command --ungroup run-command-abort 3 false >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_line_count = 6 err\n+'\n+\n cat >expect <<-EOF\n no further jobs available\n EOF\n@@ -167,6 +191,12 @@ test_expect_success 'run_command outputs ' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command outputs (ungroup) ' '\n+\ttest-tool run-command --ungroup run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n test_trace () {\n \texpect=\"$1\"\n \tshift\n-- \n2.36.1.1103.gb3ecdfb3e6a\n\n"},{"id":"456518","messageId":"patch-v5-2.2-b0f0dc7492a-20220602T131858Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v5-0.2-00000000000-20220602T131858Z-avarab@gmail.com","subject":"[PATCH v5 2/2] hook API: fix v2.36.0 regression: hooks should be connected to a TTY","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-06-02T14:07:57Z","receivedAt":"2022-06-02T14:08:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a regression reported[1] against f443246b9f2 (commit: convert\n{pre-commit,prepare-commit-msg} hook to hook.h, 2021-12-22): Due to\nusing the run_process_parallel() API in the earlier 96e7225b310 (hook:\nadd 'run' subcommand, 2021-12-22) we'd capture the hook's stderr and\nstdout, and thus lose the connection to the TTY in the case of\ne.g. the \"pre-commit\" hook.\n\nAs a preceding commit notes GNU parallel's similar --ungroup option\nalso has it emit output faster. While we're unlikely to have hooks\nthat emit truly massive amounts of output (or where the performance\nthereof matters) it's still informative to measure the overhead. In a\nsimilar \"seq\" test we're now ~30% faster:\n\n\t$ cat .git/hooks/seq-hook; git hyperfine -L rev origin/master,HEAD~0 -s 'make CFLAGS=-O3' './git hook run seq-hook'\n\t#!/bin/sh\n\n\tseq 100000000\n\tBenchmark 1: ./git hook run seq-hook' in 'origin/master\n\t Time (mean ± σ): 787.1 ms ± 13.6 ms [User: 701.6 ms, System: 534.4 ms]\n\t Range (min … max): 773.2 ms … 806.3 ms 10 runs\n\n\tBenchmark 2: ./git hook run seq-hook' in 'HEAD~0\n\t Time (mean ± σ): 603.4 ms ± 1.6 ms [User: 573.1 ms, System: 30.3 ms]\n\t Range (min … max): 601.0 ms … 606.2 ms 10 runs\n\n\tSummary\n\t './git hook run seq-hook' in 'HEAD~0' ran\n\t 1.30 ± 0.02 times faster than './git hook run seq-hook' in 'origin/master'\n\n1. https://lore.kernel.org/git/CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com/\n\nReported-by: Anthony Sottile \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 1 +\n t/t1800-hook.sh | 37 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 38 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex 1d51be3b77a..7451205657a 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -144,6 +144,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \t\tcb_data.hook_path = abs_path.buf;\n \t}\n \n+\trun_processes_parallel_ungroup = 1;\n \trun_processes_parallel_tr2(jobs,\n \t\t\t\t pick_next_hook,\n \t\t\t\t notify_start_failure,\ndiff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\nindex 26ed5e11bc8..0b8370d1573 100755\n--- a/t/t1800-hook.sh\n+++ b/t/t1800-hook.sh\n@@ -4,6 +4,7 @@ test_description='git-hook command'\n \n TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-terminal.sh\n \n test_expect_success 'git hook usage' '\n \ttest_expect_code 129 git hook &&\n@@ -120,4 +121,40 @@ test_expect_success 'git -c core.hooksPath= hook run' '\n \ttest_cmp expect actual\n '\n \n+test_hook_tty() {\n+\tlocal fd=\"$1\" &&\n+\n+\tcat >expect &&\n+\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\n+\ttest_hook -C repo pre-commit <<-EOF &&\n+\t{\n+\t\ttest -t 1 && echo >&$fd STDOUT TTY || echo >&$fd STDOUT NO TTY &&\n+\t\ttest -t 2 && echo >&$fd STDERR TTY || echo >&$fd STDERR NO TTY\n+\t} $fd>actual\n+\tEOF\n+\n+\ttest_commit -C repo A &&\n+\ttest_commit -C repo B &&\n+\tgit -C repo reset --soft HEAD^ &&\n+\ttest_terminal git -C repo commit -m\"B.new\" &&\n+\ttest_cmp expect repo/actual\n+}\n+\n+test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDOUT redirect' '\n+\ttest_hook_tty 1 <<-\\EOF\n+\tSTDOUT NO TTY\n+\tSTDERR TTY\n+\tEOF\n+'\n+\n+test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDERR redirect' '\n+\ttest_hook_tty 2 <<-\\EOF\n+\tSTDOUT TTY\n+\tSTDERR NO TTY\n+\tEOF\n+'\n+\n test_done\n-- \n2.36.1.1103.gb3ecdfb3e6a\n\n"},{"id":"456541","messageId":"xmqq7d5yn85f.fsf@gitster.g","threadId":"57764","inReplyTo":"cover-v5-0.2-00000000000-20220602T131858Z-avarab@gmail.com","subject":"Re: [PATCH v5 0/2] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-06-02T20:05:16Z","receivedAt":"2022-06-02T20:06:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> This series fixes a v2.36.0 regression[1]. See [2] for the v4. The\n> reasons for why a regression needs this relatively large change to\n> move forward is discussed in past rounds, e.g. around [3]. CI at\n> https://github.com/avar/git/actions/runs/2428475773\n>\n> Changes since v4, mainly to address comments by Johannes (thanks for\n> the review!):\n\nThis version looks good to me.\n\n> @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n> \tfor (i = 0; i < n; i++) {\n> \t\tstrbuf_init(&pp->children[i].err, 0);\n> \t\tchild_process_init(&pp->children[i].process);\n> -+\t\tif (!pp->pfd)\n> -+\t\t\tcontinue;\n> - \t\tpp->pfd[i].events = POLLIN | POLLHUP;\n> - \t\tpp->pfd[i].fd = -1;\n> +-\t\tpp->pfd[i].events = POLLIN | POLLHUP;\n> +-\t\tpp->pfd[i].fd = -1;\n> ++\t\tif (pp->pfd) {\n> ++\t\t\tpp->pfd[i].events = POLLIN | POLLHUP;\n> ++\t\t\tpp->pfd[i].fd = -1;\n> ++\t\t}\n> \t}\n\nThis change is merely a personal taste---it does not match mine but\nthat is Meh ;-)\n\n> -@@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)\n> - */\n> - static int pp_start_one(struct parallel_processes *pp)\n> - {\n> -+\tconst int ungroup = pp->ungroup;\n\nIt may have made the resulting code easier to read if the local\nvariable was kept as a synonym as \"pp->\" is short enough but is\nrepeated often, but what is written is good enough and I do not see\na need to flip-flop.\n\n> -+static void pp_mark_ungrouped_for_cleanup(struct parallel_processes *pp)\n> -+{\n> -+\tint i;\n> -+\n> -+\tif (!pp->ungroup)\n> -+\t\tBUG(\"only reachable if 'ungrouped'\");\n> -+\n> -+\tfor (i = 0; i < pp->max_processes; i++)\n> -+\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n> -+}\n\nGood to see this inlined. I find the caller easier to follow\nwithout it.\n\nThanks for a quick succession of rerolling. Will queue.\n"},{"id":"456553","messageId":"6e98dfe9-5df2-caab-ed3a-81f07b0bb6bc@gmail.com","threadId":"57764","inReplyTo":"cover-v5-0.2-00000000000-20220602T131858Z-avarab@gmail.com","subject":"Re: [PATCH v5 0/2] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-06-03T08:51:55Z","receivedAt":"2022-06-03T08:52:09Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 02/06/2022 15:07, Ævar Arnfjörð Bjarmason wrote:\n> This series fixes a v2.36.0 regression[1]. See [2] for the v4. The\n> reasons for why a regression needs this relatively large change to\n> move forward is discussed in past rounds, e.g. around [3]. CI at\n> https://github.com/avar/git/actions/runs/2428475773\n> \n> Changes since v4, mainly to address comments by Johannes (thanks for\n> the review!):\n> \n> * First, some things like renaming \"ungroup\" to something else &\n> rewriting the tests I didn't do because I thought keeping the\n> inter/range-diff down in size outweighed re-arranging or changing\n> the code at this late stage.\n> \n> In the case of the suggested shorter test in\n> https://lore.kernel.org/git/nycvar.QRO.7.76.6.2206011827300.349@tvgsbejvaqbjf.bet/\n> the replacement wasn't testing the same thing. I.e. we don't see\n> what's connected to a TTY if we redirect one of stdout or stderr\n> anymore, which is important to get right.\n\nI'm a bit confused by this, the proposed test uses this hook script\n\n\twrite_script .git/hooks/pre-commit <<-EOF\n\ttest -t 1 && echo \"stdout is a TTY\" >out\n\ttest -t 2 && echo \"stderr is a TTY\" >>out\n\tEOF\n\nif either of stderr or stdout is redirected then the corresponding \"test \n-t\" should fail and so we will detect that it is not a tty.\n\nBest Wishes\n\nPhillip\n"},{"id":"456554","messageId":"220603.86o7zaxfhf.gmgdl@evledraar.gmail.com","threadId":"57764","inReplyTo":"6e98dfe9-5df2-caab-ed3a-81f07b0bb6bc@gmail.com","subject":"Re: [PATCH v5 0/2] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-06-03T09:20:02Z","receivedAt":"2022-06-03T09:29:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jun 03 2022, Phillip Wood wrote:\n\n> Hi Ævar\n>\n> On 02/06/2022 15:07, Ævar Arnfjörð Bjarmason wrote:\n>> This series fixes a v2.36.0 regression[1]. See [2] for the v4. The\n>> reasons for why a regression needs this relatively large change to\n>> move forward is discussed in past rounds, e.g. around [3]. CI at\n>> https://github.com/avar/git/actions/runs/2428475773\n>> Changes since v4, mainly to address comments by Johannes (thanks for\n>> the review!):\n>> * First, some things like renaming \"ungroup\" to something else &\n>> rewriting the tests I didn't do because I thought keeping the\n>> inter/range-diff down in size outweighed re-arranging or changing\n>> the code at this late stage.\n>> In the case of the suggested shorter test in\n>> https://lore.kernel.org/git/nycvar.QRO.7.76.6.2206011827300.349@tvgsbejvaqbjf.bet/\n>> the replacement wasn't testing the same thing. I.e. we don't see\n>> what's connected to a TTY if we redirect one of stdout or stderr\n>> anymore, which is important to get right.\n>\n> I'm a bit confused by this, the proposed test uses this hook script\n>\n> \twrite_script .git/hooks/pre-commit <<-EOF\n> \ttest -t 1 && echo \"stdout is a TTY\" >out\n> \ttest -t 2 && echo \"stderr is a TTY\" >>out\n> \tEOF\n>\n> if either of stderr or stdout is redirected then the corresponding\n> \"test -t\" should fail and so we will detect that it is not a tty.\n\nYes, exactly, but the proposed test doesn't test that, in that case both\nof them are connected, the test in 2/2 does test that case.\n\nCan that snippet bebe made to work? Sure, but I know the test I have\nworks, and that proposed replacement didn't even pass chainlint\n(i.e. hasn't been run even once in our test suite). So I didn't think\nthat trying to micro-optimize the test length was worth it in this case.\n\nIt's also getting much of that length reduction e.g. by not cleaning up\nafter itself, which the test in 2/2 does.\n\n"},{"id":"456574","messageId":"993aff66-01b7-4aa3-78ae-0027c9c04ea8@gmail.com","threadId":"57764","inReplyTo":"220603.86o7zaxfhf.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v5 0/2] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-06-03T13:21:31Z","receivedAt":"2022-06-03T13:21:39Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 03/06/2022 10:20, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Jun 03 2022, Phillip Wood wrote:\n> \n>> Hi Ævar\n>>\n>> On 02/06/2022 15:07, Ævar Arnfjörð Bjarmason wrote:\n>>> This series fixes a v2.36.0 regression[1]. See [2] for the v4. The\n>>> reasons for why a regression needs this relatively large change to\n>>> move forward is discussed in past rounds, e.g. around [3]. CI at\n>>> https://github.com/avar/git/actions/runs/2428475773\n>>> Changes since v4, mainly to address comments by Johannes (thanks for\n>>> the review!):\n>>> * First, some things like renaming \"ungroup\" to something else &\n>>> rewriting the tests I didn't do because I thought keeping the\n>>> inter/range-diff down in size outweighed re-arranging or changing\n>>> the code at this late stage.\n>>> In the case of the suggested shorter test in\n>>> https://lore.kernel.org/git/nycvar.QRO.7.76.6.2206011827300.349@tvgsbejvaqbjf.bet/\n>>> the replacement wasn't testing the same thing. I.e. we don't see\n>>> what's connected to a TTY if we redirect one of stdout or stderr\n>>> anymore, which is important to get right.\n>>\n>> I'm a bit confused by this, the proposed test uses this hook script\n>>\n>> \twrite_script .git/hooks/pre-commit <<-EOF\n>> \ttest -t 1 && echo \"stdout is a TTY\" >out\n>> \ttest -t 2 && echo \"stderr is a TTY\" >>out\n>> \tEOF\n>>\n>> if either of stderr or stdout is redirected then the corresponding\n>> \"test -t\" should fail and so we will detect that it is not a tty.\n> \n> Yes, exactly, but the proposed test doesn't test that, in that case both\n> of them are connected, the test in 2/2 does test that case.\n\nI think I must be missing something. As I understand it we want to check \nthat the hook can see a tty on stdout and stderr. In the test above \nwe'll get a line printed for each fd that is a tty. Your test always \nredirects one of stdout and stderr - why is it important to test that? - \nit feels like it is testing the shell's redirection code rather than git.\n\nI was concerned that we had also regressed the handling of stdin but \nlooking at (the now deleted) run_hook_ve() it used to set .no_stdin = 1 \nso that is unchanged in the new code.\n\nBest Wishes\n\nPhillip\n\n> Can that snippet bebe made to work? Sure, but I know the test I have\n> works, and that proposed replacement didn't even pass chainlint\n> (i.e. hasn't been run even once in our test suite). So I didn't think\n> that trying to micro-optimize the test length was worth it in this case.\n> \n> It's also getting much of that length reduction e.g. by not cleaning up\n> after itself, which the test in 2/2 does.\n> \n"},{"id":"456775","messageId":"patch-v6-2.2-503ef241a52-20220606T170356Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v6-0.2-00000000000-20220606T170356Z-avarab@gmail.com","subject":"[PATCH v6 2/2] hook API: fix v2.36.0 regression: hooks should be connected to a TTY","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-06-07T08:48:20Z","receivedAt":"2022-06-07T08:48:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a regression reported[1] against f443246b9f2 (commit: convert\n{pre-commit,prepare-commit-msg} hook to hook.h, 2021-12-22): Due to\nusing the run_process_parallel() API in the earlier 96e7225b310 (hook:\nadd 'run' subcommand, 2021-12-22) we'd capture the hook's stderr and\nstdout, and thus lose the connection to the TTY in the case of\ne.g. the \"pre-commit\" hook.\n\nAs a preceding commit notes GNU parallel's similar --ungroup option\nalso has it emit output faster. While we're unlikely to have hooks\nthat emit truly massive amounts of output (or where the performance\nthereof matters) it's still informative to measure the overhead. In a\nsimilar \"seq\" test we're now ~30% faster:\n\n\t$ cat .git/hooks/seq-hook; git hyperfine -L rev origin/master,HEAD~0 -s 'make CFLAGS=-O3' './git hook run seq-hook'\n\t#!/bin/sh\n\n\tseq 100000000\n\tBenchmark 1: ./git hook run seq-hook' in 'origin/master\n\t Time (mean ± σ): 787.1 ms ± 13.6 ms [User: 701.6 ms, System: 534.4 ms]\n\t Range (min … max): 773.2 ms … 806.3 ms 10 runs\n\n\tBenchmark 2: ./git hook run seq-hook' in 'HEAD~0\n\t Time (mean ± σ): 603.4 ms ± 1.6 ms [User: 573.1 ms, System: 30.3 ms]\n\t Range (min … max): 601.0 ms … 606.2 ms 10 runs\n\n\tSummary\n\t './git hook run seq-hook' in 'HEAD~0' ran\n\t 1.30 ± 0.02 times faster than './git hook run seq-hook' in 'origin/master'\n\n1. https://lore.kernel.org/git/CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com/\n\nReported-by: Anthony Sottile \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 1 +\n t/t1800-hook.sh | 31 +++++++++++++++++++++++++++++++\n 2 files changed, 32 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex 1d51be3b77a..7451205657a 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -144,6 +144,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \t\tcb_data.hook_path = abs_path.buf;\n \t}\n \n+\trun_processes_parallel_ungroup = 1;\n \trun_processes_parallel_tr2(jobs,\n \t\t\t\t pick_next_hook,\n \t\t\t\t notify_start_failure,\ndiff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\nindex 26ed5e11bc8..0175a0664da 100755\n--- a/t/t1800-hook.sh\n+++ b/t/t1800-hook.sh\n@@ -4,6 +4,7 @@ test_description='git-hook command'\n \n TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-terminal.sh\n \n test_expect_success 'git hook usage' '\n \ttest_expect_code 129 git hook &&\n@@ -120,4 +121,34 @@ test_expect_success 'git -c core.hooksPath= hook run' '\n \ttest_cmp expect actual\n '\n \n+test_hook_tty() {\n+\tcat >expect <<-\\EOF\n+\tSTDOUT TTY\n+\tSTDERR TTY\n+\tEOF\n+\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\n+\ttest_commit -C repo A &&\n+\ttest_commit -C repo B &&\n+\tgit -C repo reset --soft HEAD^ &&\n+\n+\ttest_hook -C repo pre-commit <<-EOF &&\n+\ttest -t 1 && echo STDOUT TTY >>actual || echo STDOUT NO TTY >>actual &&\n+\ttest -t 2 && echo STDERR TTY >>actual || echo STDERR NO TTY >>actual\n+\tEOF\n+\n+\ttest_terminal git \"$@\" &&\n+\ttest_cmp expect repo/actual\n+}\n+\n+test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY' '\n+\ttest_hook_tty -C repo hook run pre-commit\n+'\n+\n+test_expect_success TTY 'git commit: stdout and stderr are connected to a TTY' '\n+\ttest_hook_tty -C repo commit -m\"B.new\"\n+'\n+\n test_done\n-- \n2.36.1.1173.gcad22db6399\n\n"},{"id":"456776","messageId":"patch-v6-1.2-45248c786d7-20220606T170356Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v6-0.2-00000000000-20220606T170356Z-avarab@gmail.com","subject":"[PATCH v6 1/2] run-command: add an \"ungroup\" option to run_process_parallel()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-06-07T08:48:19Z","receivedAt":"2022-06-07T08:48:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Extend the parallel execution API added in c553c72eed6 (run-command:\nadd an asynchronous parallel child processor, 2015-12-15) to support a\nmode where the stdout and stderr of the processes isn't captured and\noutput in a deterministic order, instead we'll leave it to the kernel\nand stdio to sort it out.\n\nThis gives the API same functionality as GNU parallel's --ungroup\noption. As we'll see in a subsequent commit the main reason to want\nthis is to support stdout and stderr being connected to the TTY in the\ncase of jobs=1, demonstrated here with GNU parallel:\n\n\t$ parallel --ungroup 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n\tTTY\n\tTTY\n\t$ parallel 'test -t {} && echo TTY || echo NTTY' ::: 1 2\n\tNTTY\n\tNTTY\n\nAnother is as GNU parallel's documentation notes a potential for\noptimization. As demonstrated in next commit our results with \"git\nhook run\" will be similar, but generally speaking this shows that if\nyou want to run processes in parallel where the exact order isn't\nimportant this can be a lot faster:\n\n\t$ hyperfine -r 3 -L o ,--ungroup 'parallel {o} seq ::: 10000000 >/dev/null '\n\tBenchmark 1: parallel seq ::: 10000000 >/dev/null\n\t Time (mean ± σ): 220.2 ms ± 9.3 ms [User: 124.9 ms, System: 96.1 ms]\n\t Range (min … max): 212.3 ms … 230.5 ms 3 runs\n\n\tBenchmark 2: parallel --ungroup seq ::: 10000000 >/dev/null\n\t Time (mean ± σ): 154.7 ms ± 0.9 ms [User: 136.2 ms, System: 25.1 ms]\n\t Range (min … max): 153.9 ms … 155.7 ms 3 runs\n\n\tSummary\n\t 'parallel --ungroup seq ::: 10000000 >/dev/null ' ran\n\t 1.42 ± 0.06 times faster than 'parallel seq ::: 10000000 >/dev/null '\n\nA large part of the juggling in the API is to make the API safer for\nits maintenance and consumers alike.\n\nFor the maintenance of the API we e.g. avoid malloc()-ing the\n\"pp->pfd\", ensuring that SANITIZE=address and other similar tools will\ncatch any unexpected misuse.\n\nFor API consumers we take pains to never pass the non-NULL \"out\"\nbuffer to an API user that provided the \"ungroup\" option. The\nresulting code in t/helper/test-run-command.c isn't typical of such a\nuser, i.e. they'd typically use one mode or the other, and would know\nwhether they'd provided \"ungroup\" or not.\n\nWe could also avoid the strbuf_init() for \"buffered_output\" by having\n\"struct parallel_processes\" use a static PARALLEL_PROCESSES_INIT\ninitializer, but let's leave that cleanup for later.\n\nUsing a global \"run_processes_parallel_ungroup\" variable to enable\nthis option is rather nasty, but is being done here to produce as\nminimal of a change as possible for a subsequent regression fix. This\nchange is extracted from a larger initial version[1] which ends up\nwith a better end-state for the API, but in doing so needed to modify\nall existing callers of the API. Let's defer that for now, and\nnarrowly focus on what we need for fixing the regression in the\nsubsequent commit.\n\nIt's safe to do this with a global variable because:\n\n A) hook.c is the only user of it that sets it to non-zero, and before\n we'll get any other API users we'll refactor away this method of\n passing in the option, i.e. re-roll [1].\n\n B) Even if hook.c wasn't the only user we don't have callers of this\n API that concurrently invoke this parallel process starting API\n itself in parallel.\n\nAs noted above \"A\" && \"B\" are rather nasty, and we don't want to live\nwith those caveats long-term, but for now they should be an acceptable\ncompromise.\n\n1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 70 +++++++++++++++++++++++++++----------\n run-command.h | 30 ++++++++++++----\n t/helper/test-run-command.c | 22 ++++++++++--\n t/t0061-run-command.sh | 30 ++++++++++++++++\n 4 files changed, 123 insertions(+), 29 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex a8501e38ceb..7ab2dd28f3c 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1471,6 +1471,7 @@ enum child_state {\n \tGIT_CP_WAIT_CLEANUP,\n };\n \n+int run_processes_parallel_ungroup;\n struct parallel_processes {\n \tvoid *data;\n \n@@ -1494,6 +1495,7 @@ struct parallel_processes {\n \tstruct pollfd *pfd;\n \n \tunsigned shutdown : 1;\n+\tunsigned ungroup : 1;\n \n \tint output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n@@ -1537,7 +1539,7 @@ static void pp_init(struct parallel_processes *pp,\n \t\t get_next_task_fn get_next_task,\n \t\t start_failure_fn start_failure,\n \t\t task_finished_fn task_finished,\n-\t\t void *data)\n+\t\t void *data, int ungroup)\n {\n \tint i;\n \n@@ -1559,15 +1561,21 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->nr_processes = 0;\n \tpp->output_owner = 0;\n \tpp->shutdown = 0;\n+\tpp->ungroup = ungroup;\n \tCALLOC_ARRAY(pp->children, n);\n-\tCALLOC_ARRAY(pp->pfd, n);\n+\tif (pp->ungroup)\n+\t\tpp->pfd = NULL;\n+\telse\n+\t\tCALLOC_ARRAY(pp->pfd, n);\n \tstrbuf_init(&pp->buffered_output, 0);\n \n \tfor (i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n \t\tchild_process_init(&pp->children[i].process);\n-\t\tpp->pfd[i].events = POLLIN | POLLHUP;\n-\t\tpp->pfd[i].fd = -1;\n+\t\tif (pp->pfd) {\n+\t\t\tpp->pfd[i].events = POLLIN | POLLHUP;\n+\t\t\tpp->pfd[i].fd = -1;\n+\t\t}\n \t}\n \n \tpp_for_signal = pp;\n@@ -1615,24 +1623,31 @@ static int pp_start_one(struct parallel_processes *pp)\n \t\tBUG(\"bookkeeping is hard\");\n \n \tcode = pp->get_next_task(&pp->children[i].process,\n-\t\t\t\t &pp->children[i].err,\n+\t\t\t\t pp->ungroup ? NULL : &pp->children[i].err,\n \t\t\t\t pp->data,\n \t\t\t\t &pp->children[i].data);\n \tif (!code) {\n-\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n-\t\tstrbuf_reset(&pp->children[i].err);\n+\t\tif (!pp->ungroup) {\n+\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n+\t\t\tstrbuf_reset(&pp->children[i].err);\n+\t\t}\n \t\treturn 1;\n \t}\n-\tpp->children[i].process.err = -1;\n-\tpp->children[i].process.stdout_to_stderr = 1;\n+\tif (!pp->ungroup) {\n+\t\tpp->children[i].process.err = -1;\n+\t\tpp->children[i].process.stdout_to_stderr = 1;\n+\t}\n \tpp->children[i].process.no_stdin = 1;\n \n \tif (start_command(&pp->children[i].process)) {\n-\t\tcode = pp->start_failure(&pp->children[i].err,\n+\t\tcode = pp->start_failure(pp->ungroup ? NULL :\n+\t\t\t\t\t &pp->children[i].err,\n \t\t\t\t\t pp->data,\n \t\t\t\t\t pp->children[i].data);\n-\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n-\t\tstrbuf_reset(&pp->children[i].err);\n+\t\tif (!pp->ungroup) {\n+\t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n+\t\t\tstrbuf_reset(&pp->children[i].err);\n+\t\t}\n \t\tif (code)\n \t\t\tpp->shutdown = 1;\n \t\treturn code;\n@@ -1640,7 +1655,8 @@ static int pp_start_one(struct parallel_processes *pp)\n \n \tpp->nr_processes++;\n \tpp->children[i].state = GIT_CP_WORKING;\n-\tpp->pfd[i].fd = pp->children[i].process.err;\n+\tif (pp->pfd)\n+\t\tpp->pfd[i].fd = pp->children[i].process.err;\n \treturn 0;\n }\n \n@@ -1674,6 +1690,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n static void pp_output(struct parallel_processes *pp)\n {\n \tint i = pp->output_owner;\n+\n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n \t\tstrbuf_write(&pp->children[i].err, stderr);\n@@ -1696,7 +1713,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \n \t\tcode = finish_command(&pp->children[i].process);\n \n-\t\tcode = pp->task_finished(code,\n+\t\tcode = pp->task_finished(code, pp->ungroup ? NULL :\n \t\t\t\t\t &pp->children[i].err, pp->data,\n \t\t\t\t\t pp->children[i].data);\n \n@@ -1707,10 +1724,13 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \n \t\tpp->nr_processes--;\n \t\tpp->children[i].state = GIT_CP_FREE;\n-\t\tpp->pfd[i].fd = -1;\n+\t\tif (pp->pfd)\n+\t\t\tpp->pfd[i].fd = -1;\n \t\tchild_process_init(&pp->children[i].process);\n \n-\t\tif (i != pp->output_owner) {\n+\t\tif (pp->ungroup) {\n+\t\t\t; /* no strbuf_*() work to do here */\n+\t\t} else if (i != pp->output_owner) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t} else {\n@@ -1747,9 +1767,14 @@ int run_processes_parallel(int n,\n \tint i, code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n+\tint ungroup = run_processes_parallel_ungroup;\n \tstruct parallel_processes pp;\n \n-\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb);\n+\t/* unset for the next API user */\n+\trun_processes_parallel_ungroup = 0;\n+\n+\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n+\t\tungroup);\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1766,8 +1791,15 @@ int run_processes_parallel(int n,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tpp_buffer_stderr(&pp, output_timeout);\n-\t\tpp_output(&pp);\n+\t\tif (ungroup) {\n+\t\t\tint i;\n+\n+\t\t\tfor (i = 0; i < pp.max_processes; i++)\n+\t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n+\t\t} else {\n+\t\t\tpp_buffer_stderr(&pp, output_timeout);\n+\t\t\tpp_output(&pp);\n+\t\t}\n \t\tcode = pp_collect_finished(&pp);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\ndiff --git a/run-command.h b/run-command.h\nindex 5bd0c933e80..bf4236f1164 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -405,6 +405,9 @@ void check_pipe(int err);\n * pp_cb is the callback cookie as passed to run_processes_parallel.\n * You can store a child process specific callback cookie in pp_task_cb.\n *\n+ * See run_processes_parallel() below for a discussion of the \"struct\n+ * strbuf *out\" parameter.\n+ *\n * Even after returning 0 to indicate that there are no more processes,\n * this function will be called again until there are no more running\n * child processes.\n@@ -423,9 +426,8 @@ typedef int (*get_next_task_fn)(struct child_process *cp,\n * This callback is called whenever there are problems starting\n * a new process.\n *\n- * You must not write to stdout or stderr in this function. Add your\n- * message to the strbuf out instead, which will be printed without\n- * messing up the output of the other parallel processes.\n+ * See run_processes_parallel() below for a discussion of the \"struct\n+ * strbuf *out\" parameter.\n *\n * pp_cb is the callback cookie as passed into run_processes_parallel,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n@@ -441,9 +443,8 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n /**\n * This callback is called on every child process that finished processing.\n *\n- * You must not write to stdout or stderr in this function. Add your\n- * message to the strbuf out instead, which will be printed without\n- * messing up the output of the other parallel processes.\n+ * See run_processes_parallel() below for a discussion of the \"struct\n+ * strbuf *out\" parameter.\n *\n * pp_cb is the callback cookie as passed into run_processes_parallel,\n * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n@@ -464,11 +465,26 @@ typedef int (*task_finished_fn)(int result,\n *\n * The children started via this function run in parallel. Their output\n * (both stdout and stderr) is routed to stderr in a manner that output\n- * from different tasks does not interleave.\n+ * from different tasks does not interleave (but see \"ungroup\" below).\n *\n * start_failure_fn and task_finished_fn can be NULL to omit any\n * special handling.\n+ *\n+ * If the \"ungroup\" option isn't specified, the API will set the\n+ * \"stdout_to_stderr\" parameter in \"struct child_process\" and provide\n+ * the callbacks with a \"struct strbuf *out\" parameter to write output\n+ * to. In this case the callbacks must not write to stdout or\n+ * stderr as such output will mess up the output of the other parallel\n+ * processes. If \"ungroup\" option is specified callbacks will get a\n+ * NULL \"struct strbuf *out\" parameter, and are responsible for\n+ * emitting their own output, including dealing with any race\n+ * conditions due to writing in parallel to stdout and stderr.\n+ * The \"ungroup\" option can be enabled by setting the global\n+ * \"run_processes_parallel_ungroup\" to \"1\" before invoking\n+ * run_processes_parallel(), it will be set back to \"0\" as soon as the\n+ * API reads that setting.\n */\n+extern int run_processes_parallel_ungroup;\n int run_processes_parallel(int n,\n \t\t\t get_next_task_fn,\n \t\t\t start_failure_fn,\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex f3b90aa834a..34cce45b584 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -31,7 +31,11 @@ static int parallel_next(struct child_process *cp,\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->args.v);\n-\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n+\telse\n+\t\tfprintf(stderr, \"preloaded output of a child\\n\");\n+\n \tnumber_callbacks++;\n \treturn 1;\n }\n@@ -41,7 +45,10 @@ static int no_job(struct child_process *cp,\n \t\t void *cb,\n \t\t void **task_cb)\n {\n-\tstrbuf_addstr(err, \"no further jobs available\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"no further jobs available\\n\");\n+\telse\n+\t\tfprintf(stderr, \"no further jobs available\\n\");\n \treturn 0;\n }\n \n@@ -50,7 +57,10 @@ static int task_finished(int result,\n \t\t\t void *pp_cb,\n \t\t\t void *pp_task_cb)\n {\n-\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n+\tif (err)\n+\t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n+\telse\n+\t\tfprintf(stderr, \"asking for a quick stop\\n\");\n \treturn 1;\n }\n \n@@ -407,6 +417,12 @@ int cmd__run_command(int argc, const char **argv)\n \tif (!strcmp(argv[1], \"run-command\"))\n \t\texit(run_command(&proc));\n \n+\tif (!strcmp(argv[1], \"--ungroup\")) {\n+\t\targv += 1;\n+\t\targc -= 1;\n+\t\trun_processes_parallel_ungroup = 1;\n+\t}\n+\n \tjobs = atoi(argv[2]);\n \tstrvec_clear(&proc.args);\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex ee281909bc3..7b5423eebda 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -134,16 +134,34 @@ test_expect_success 'run_command runs in parallel with more jobs available than\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n+\ttest-tool run-command --ungroup run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n \ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n+\ttest-tool run-command --ungroup run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n \ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n+\ttest-tool run-command --ungroup run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_line_count = 8 out &&\n+\ttest_line_count = 4 err\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n@@ -158,6 +176,12 @@ test_expect_success 'run_command is asked to abort gracefully' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n+\ttest-tool run-command --ungroup run-command-abort 3 false >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_line_count = 6 err\n+'\n+\n cat >expect <<-EOF\n no further jobs available\n EOF\n@@ -167,6 +191,12 @@ test_expect_success 'run_command outputs ' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command outputs (ungroup) ' '\n+\ttest-tool run-command --ungroup run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n test_trace () {\n \texpect=\"$1\"\n \tshift\n-- \n2.36.1.1173.gcad22db6399\n\n"},{"id":"456777","messageId":"cover-v6-0.2-00000000000-20220606T170356Z-avarab@gmail.com","threadId":"57764","inReplyTo":"cover-v5-0.2-00000000000-20220602T131858Z-avarab@gmail.com","subject":"[PATCH v6 0/2] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-06-07T08:48:18Z","receivedAt":"2022-06-07T08:48:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This series fixes a v2.36.0 regression[1]. See [2] for the v5. The\nreasons for why a regression needs this relatively large change to\nmove forward is discussed in past rounds, e.g. around [3]. CI at\nhttps://github.com/avar/git/actions/runs/2448496389\n\nChanges since v5:\n\n * Make the hook run test more meaningful, we now test with \"-t\" in\n the hook, instead of redirecting one of STDOUT or STDERR.\n\n * Add a test for both \"git hook run\" and \"git commit\", to showh that\n the \"git hook run\" command and one \"real\" user of it agree.\n\n1. https://lore.kernel.org/git/cover-v5-0.2-00000000000-20220602T131858Z-avarab@gmail.com/\n\nÆvar Arnfjörð Bjarmason (2):\n run-command: add an \"ungroup\" option to run_process_parallel()\n hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n\nÆvar Arnfjörð Bjarmason (2):\n run-command: add an \"ungroup\" option to run_process_parallel()\n hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n\n hook.c | 1 +\n run-command.c | 70 +++++++++++++++++++++++++++----------\n run-command.h | 30 ++++++++++++----\n t/helper/test-run-command.c | 22 ++++++++++--\n t/t0061-run-command.sh | 30 ++++++++++++++++\n t/t1800-hook.sh | 31 ++++++++++++++++\n 6 files changed, 155 insertions(+), 29 deletions(-)\n\nRange-diff against v5:\n1: d018b7c4441 = 1: 45248c786d7 run-command: add an \"ungroup\" option to run_process_parallel()\n2: b0f0dc7492a ! 2: 503ef241a52 hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n @@ t/t1800-hook.sh: test_expect_success 'git -c core.hooksPath= hook run' '\n '\n \n +test_hook_tty() {\n -+\tlocal fd=\"$1\" &&\n -+\n -+\tcat >expect &&\n ++\tcat >expect <<-\\EOF\n ++\tSTDOUT TTY\n ++\tSTDERR TTY\n ++\tEOF\n +\n +\ttest_when_finished \"rm -rf repo\" &&\n +\tgit init repo &&\n +\n -+\ttest_hook -C repo pre-commit <<-EOF &&\n -+\t{\n -+\t\ttest -t 1 && echo >&$fd STDOUT TTY || echo >&$fd STDOUT NO TTY &&\n -+\t\ttest -t 2 && echo >&$fd STDERR TTY || echo >&$fd STDERR NO TTY\n -+\t} $fd>actual\n -+\tEOF\n -+\n +\ttest_commit -C repo A &&\n +\ttest_commit -C repo B &&\n +\tgit -C repo reset --soft HEAD^ &&\n -+\ttest_terminal git -C repo commit -m\"B.new\" &&\n ++\n ++\ttest_hook -C repo pre-commit <<-EOF &&\n ++\ttest -t 1 && echo STDOUT TTY >>actual || echo STDOUT NO TTY >>actual &&\n ++\ttest -t 2 && echo STDERR TTY >>actual || echo STDERR NO TTY >>actual\n ++\tEOF\n ++\n ++\ttest_terminal git \"$@\" &&\n +\ttest_cmp expect repo/actual\n +}\n +\n -+test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDOUT redirect' '\n -+\ttest_hook_tty 1 <<-\\EOF\n -+\tSTDOUT NO TTY\n -+\tSTDERR TTY\n -+\tEOF\n ++test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY' '\n ++\ttest_hook_tty -C repo hook run pre-commit\n +'\n +\n -+test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY: STDERR redirect' '\n -+\ttest_hook_tty 2 <<-\\EOF\n -+\tSTDOUT TTY\n -+\tSTDERR NO TTY\n -+\tEOF\n ++test_expect_success TTY 'git commit: stdout and stderr are connected to a TTY' '\n ++\ttest_hook_tty -C repo commit -m\"B.new\"\n +'\n +\n test_done\n-- \n2.36.1.1173.gcad22db6399\n\n"},{"id":"456778","messageId":"220607.86bkv4votc.gmgdl@evledraar.gmail.com","threadId":"57764","inReplyTo":"993aff66-01b7-4aa3-78ae-0027c9c04ea8@gmail.com","subject":"Re: [PATCH v5 0/2] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-06-07T08:49:01Z","receivedAt":"2022-06-07T08:51:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jun 03 2022, Phillip Wood wrote:\n\n> Hi Ævar\n>\n> On 03/06/2022 10:20, Ævar Arnfjörð Bjarmason wrote:\n>> On Fri, Jun 03 2022, Phillip Wood wrote:\n>> \n>>> Hi Ævar\n>>>\n>>> On 02/06/2022 15:07, Ævar Arnfjörð Bjarmason wrote:\n>>>> This series fixes a v2.36.0 regression[1]. See [2] for the v4. The\n>>>> reasons for why a regression needs this relatively large change to\n>>>> move forward is discussed in past rounds, e.g. around [3]. CI at\n>>>> https://github.com/avar/git/actions/runs/2428475773\n>>>> Changes since v4, mainly to address comments by Johannes (thanks for\n>>>> the review!):\n>>>> * First, some things like renaming \"ungroup\" to something else &\n>>>> rewriting the tests I didn't do because I thought keeping the\n>>>> inter/range-diff down in size outweighed re-arranging or changing\n>>>> the code at this late stage.\n>>>> In the case of the suggested shorter test in\n>>>> https://lore.kernel.org/git/nycvar.QRO.7.76.6.2206011827300.349@tvgsbejvaqbjf.bet/\n>>>> the replacement wasn't testing the same thing. I.e. we don't see\n>>>> what's connected to a TTY if we redirect one of stdout or stderr\n>>>> anymore, which is important to get right.\n>>>\n>>> I'm a bit confused by this, the proposed test uses this hook script\n>>>\n>>> \twrite_script .git/hooks/pre-commit <<-EOF\n>>> \ttest -t 1 && echo \"stdout is a TTY\" >out\n>>> \ttest -t 2 && echo \"stderr is a TTY\" >>out\n>>> \tEOF\n>>>\n>>> if either of stderr or stdout is redirected then the corresponding\n>>> \"test -t\" should fail and so we will detect that it is not a tty.\n>> Yes, exactly, but the proposed test doesn't test that, in that case\n>> both\n>> of them are connected, the test in 2/2 does test that case.\n>\n> I think I must be missing something. As I understand it we want to\n> check that the hook can see a tty on stdout and stderr. In the test\n> above we'll get a line printed for each fd that is a tty. Your test\n> always redirects one of stdout and stderr - why is it important to\n> test that? - it feels like it is testing the shell's redirection code\n> rather than git.\n\nYes, I think I'm the one who was missing something.\n\nI looked at this again and I thought I'd been testing that e.g. one of\nthe two not returning true from isatty() wasn't making both \"not TTY\",\ni.e. that run-command.c wasn't performing some shenanigans.\n\nBut that was probably too paranoid, and in any case I couldn't find a\ngood way to test it.\n\n> I was concerned that we had also regressed the handling of stdin but\n> looking at (the now deleted) run_hook_ve() it used to set .no_stdin =\n> 1 so that is unchanged in the new code.\n\n*nod*\n\nI re-rolled a v6 just now which I think should address your comments\nhere:\nhttps://lore.kernel.org/git/cover-v6-0.2-00000000000-20220606T170356Z-avarab@gmail.com/\n\nI've still kept the \"clean up after yourself\" etc. behavior in the test,\nand since it was easy we now test both \"git hook run\" and \"git commit\".\n\nThanks a lot for the careful review.\n"},{"id":"456796","messageId":"xmqqfskgz9sb.fsf@gitster.g","threadId":"57764","inReplyTo":"cover-v6-0.2-00000000000-20220606T170356Z-avarab@gmail.com","subject":"Re: [PATCH v6 0/2] hook API: connect hooks to the TTY again, fixes a v2.36.0 regression","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-06-07T17:02:44Z","receivedAt":"2022-06-07T17:02:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> This series fixes a v2.36.0 regression[1]. See [2] for the v5. The\n> reasons for why a regression needs this relatively large change to\n> move forward is discussed in past rounds, e.g. around [3]. CI at\n> https://github.com/avar/git/actions/runs/2448496389\n>\n> Changes since v5:\n>\n> * Make the hook run test more meaningful, we now test with \"-t\" in\n> the hook, instead of redirecting one of STDOUT or STDERR.\n>\n> * Add a test for both \"git hook run\" and \"git commit\", to showh that\n> the \"git hook run\" command and one \"real\" user of it agree.\n\nThanks for a careful review and a timely response.\n\n>\n> 1. https://lore.kernel.org/git/cover-v5-0.2-00000000000-20220602T131858Z-avarab@gmail.com/\n>\n> Ævar Arnfjörð Bjarmason (2):\n> run-command: add an \"ungroup\" option to run_process_parallel()\n> hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n>\n> Ævar Arnfjörð Bjarmason (2):\n> run-command: add an \"ungroup\" option to run_process_parallel()\n> hook API: fix v2.36.0 regression: hooks should be connected to a TTY\n\nPuzzling. Perhaps copy-and-paste mistake we can ignore.\n\n> hook.c | 1 +\n> run-command.c | 70 +++++++++++++++++++++++++++----------\n> run-command.h | 30 ++++++++++++----\n> t/helper/test-run-command.c | 22 ++++++++++--\n> t/t0061-run-command.sh | 30 ++++++++++++++++\n> t/t1800-hook.sh | 31 ++++++++++++++++\n> 6 files changed, 155 insertions(+), 29 deletions(-)\n\n"},{"id":"456797","messageId":"xmqq7d5sz9iy.fsf@gitster.g","threadId":"57764","inReplyTo":"patch-v6-2.2-503ef241a52-20220606T170356Z-avarab@gmail.com","subject":"Re: [PATCH v6 2/2] hook API: fix v2.36.0 regression: hooks should be connected to a TTY","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-06-07T17:08:21Z","receivedAt":"2022-06-07T17:08:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> @@ -120,4 +121,34 @@ test_expect_success 'git -c core.hooksPath= hook run' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_hook_tty() {\n\nStyle.\n\n> +\tcat >expect <<-\\EOF\n> +\tSTDOUT TTY\n> +\tSTDERR TTY\n> +\tEOF\n> +\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\n> +\ttest_commit -C repo A &&\n> +\ttest_commit -C repo B &&\n> +\tgit -C repo reset --soft HEAD^ &&\n> +\n> +\ttest_hook -C repo pre-commit <<-EOF &&\n> +\ttest -t 1 && echo STDOUT TTY >>actual || echo STDOUT NO TTY >>actual &&\n> +\ttest -t 2 && echo STDERR TTY >>actual || echo STDERR NO TTY >>actual\n> +\tEOF\n\nSo, when this hook is run, we'd see if STDOUT and STDERR are\nconnected to a tty in the \"actual\" file.\n\n\n> +\ttest_terminal git \"$@\" &&\n\nAnd we run the test and see \n\n> +\ttest_cmp expect repo/actual\n\nwhat happens. The test_cmp knows that the git command runs in\n\"repo\" by hardcoding repo/actual, and this helper is full of the\nsame knowledge, so it would be easier to see what is going on if\nyou removed \"-C repo\" from the two callers (below) and instead added\nit to where you run \"git\" under test_terminal (above).\n\n> +}\n> +\n> +test_expect_success TTY 'git hook run: stdout and stderr are connected to a TTY' '\n> +\ttest_hook_tty -C repo hook run pre-commit\n> +'\n> +\n> +test_expect_success TTY 'git commit: stdout and stderr are connected to a TTY' '\n> +\ttest_hook_tty -C repo commit -m\"B.new\"\n> +'\n> +\n> test_done\n\nOther than that, looking good.\n\nThanks.\n"},{"id":"457437","messageId":"YqvFpcfUqyL+SlhB@google.com","threadId":"57764","inReplyTo":"patch-v6-1.2-45248c786d7-20220606T170356Z-avarab@gmail.com","subject":"Re: [PATCH v6 1/2] run-command: add an \"ungroup\" option to run_process_parallel()","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2022-06-17T00:07:01Z","receivedAt":"2022-06-17T00:07:11Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Jun 07, 2022 at 10:48:19AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> @@ -1766,8 +1791,15 @@ int run_processes_parallel(int n,\n> \t\t}\n> \t\tif (!pp.nr_processes)\n> \t\t\tbreak;\n> -\t\tpp_buffer_stderr(&pp, output_timeout);\n> -\t\tpp_output(&pp);\n> +\t\tif (ungroup) {\n> +\t\t\tint i;\n> +\n> +\t\t\tfor (i = 0; i < pp.max_processes; i++)\n> +\t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n\nFYI, this broke for us downstream where we are carrying patches adding a\n'pp_buffer_stdin()' and friends to enable stdin buffering to parallel\nprocesses. It also appears to break when pp.max_processes exceeds the\nnumber of actual tasks provided. I needed to add something like this\n\n+ if (pp.children[i].state == GIT_CP_WORKING &&\n+ !pp.children[i].process.in)\n+ pp.children[i].state = GIT_CP_WAIT_CLEANUP;\n\nThat is, only set the WAIT_CLEANUP state if the task wasn't waiting to\nbe given work (GIT_CP_FREE -> GIT_CP_WAIT_CLEANUP leads to some \"error:\nwaitpid is confused\" errors) and if stdin is not currently being\nbuffered. In the case where .process.in is > 0, GIT_CP_WAIT_CLEANUP\ncauses pp_collect_finished() to stop spinning in this IO buffer loop,\nbut there is still stdin to pass along.\n\nAnyway, I think the first part (GIT_CP_FREE -> GIT_CP_WAIT_CLEANUP) is a\nbug that matters for the patch as it is now; the stdin bit will not\nmatter until the later config-based-hooks patches which introduce stdin\nbuffering anyway.\n\nThanks very much for this fix otherwise.\n\n - Emily\n\n> +\t\t} else {\n> +\t\t\tpp_buffer_stderr(&pp, output_timeout);\n> +\t\t\tpp_output(&pp);\n> +\t\t}\n> \t\tcode = pp_collect_finished(&pp);\n> \t\tif (code) {\n> \t\t\tpp.shutdown = 1;\n"}]}