git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty

From
Emily Shaffer <emilyshaffer@google.com>
Date
Apr 19, 2022, 23:37 UTC
Message-ID
<Yl9Hn0C0TwalASC0@google.com>
In-Reply-To
<CA+dzEBn108QoMA28f0nC8K21XT+Afua0V2Qv8XkR8rAeqUCCZw@mail.gmail.com>
On Tue, Apr 19, 2022 at 02:59:36PM -0400, Anthony Sottile wrote:
Show 132 quoted lines
> 
> here's the shortest reproduction --
> 
> ```console
> $ cat ../testrepo/.git/hooks/pre-commit
> #!/usr/bin/env bash
> if [ -t 1 ]; then
>     echo GOOD
> fi
> ```
> 
> in previous git versions:
> 
> ```
> $ git commit -q --allow-empty -m foo
> GOOD
> $
> ```
> 
> with git 2.36.0:
> 
> ````
> $ git commit -q --allow-empty -m foo
> $
> ```
> 
> why I care: I maintain a git hooks framework which uses `isatty` to
> detect whether it's appropriate to color the output.  many tools
> utilize the same check.  in 2.36.0+ isatty is false for stdout and
> stderr causing coloring to be turned off.
> 
> I bisected this (it was a little complicated, needed to force a pty):
> 
> `../testrepo`: a git repo set up with the hook above
> 
> `../bisect.sh`:
> 
> ```bash
> #!/usr/bin/env bash
> set -eux
> git clean -fxfd >& /dev/null
> make -j6 prefix="$PWD/prefix" NO_GETTEXT=1 NO_TCLTK=1 install >& /dev/null
> export PATH="$PWD/prefix/bin:$PATH"
> cd ../testrepo
> (../pty git commit -q --allow-empty -m foo || true) | grep GOOD
> ```
> 
> `../pty`:
> 
> ```python
> #!/usr/bin/env python3
> import errno
> import os
> import subprocess
> import sys
> 
> x: int = 'nope'
> 
> 
> class Pty(object):
>     def __init__(self):
>         self.r = self.w = None
> 
>     def __enter__(self):
>         self.r, self.w = os.openpty()
> 
>         return self
> 
>     def close_w(self):
>         if self.w is not None:
>             os.close(self.w)
>             self.w = None
> 
>     def close_r(self):
>         assert self.r is not None
>         os.close(self.r)
>         self.r = None
> 
>     def __exit__(self, exc_type, exc_value, traceback):
>         self.close_w()
>         self.close_r()
> 
> 
> def cmd_output_p(*cmd, **kwargs):
>     with open(os.devnull) as devnull, Pty() as pty:
>         kwargs = {'stdin': devnull, 'stdout': pty.w, 'stderr': pty.w}
>         proc = subprocess.Popen(cmd, **kwargs)
>         pty.close_w()
> 
>         buf = b''
>         while True:
>             try:
>                 bts = os.read(pty.r, 4096)
>             except OSError as e:
>                 if e.errno == errno.EIO:
>                     bts = b''
>                 else:
>                     raise
>             else:
>                 buf += bts
>             if not bts:
>                 break
> 
>     return proc.wait(), buf, None
> 
> 
> if __name__ == '__main__':
>     _, buf, _ = cmd_output_p(*sys.argv[1:])
>     sys.stdout.buffer.write(buf)
> ```
> 
> the first commit it points out:
> 
> ```
> f443246b9f29b815f0b98a07bb2d425628ae6522 is the first bad commit
> commit f443246b9f29b815f0b98a07bb2d425628ae6522
> Author: Emily Shaffer <emilyshaffer@google.com>
> 
>     commit: convert {pre-commit,prepare-commit-msg} hook to hook.h
> 
>     Move these hooks hook away from run-command.h to and over to the new
>     hook.h library.
> 
>     Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
>     Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>     Acked-by: Emily Shaffer <emilyshaffer@google.com>
>     Signed-off-by: Junio C Hamano <gitster@pobox.com>
> 
>  commit.c | 15 ++++++++-------
>  1 file changed, 8 insertions(+), 7 deletions(-)
> bisect run success
> ```

Interesting. I'm surprised to see the tty-ness of hooks changing with this patch, as the way the hook is called is pretty much the same:

run_hook_ve() ("the old way") sets no_stdin, stdout_to_stderr, args, envvars, and some trace variables, and then runs 'run_command()'; run_command() invokes start_command().

run_hooks_opt ("the new way") ultimately kicks off the hook with a callback that sets up a child_process with no_stdin, stdout_to_stderr, args, envvars, and some trace variables (hook.c:pick_next_hook); the task queue manager also sets .err to -1 on that child_process; then it calls start_command() directly (run-command.c:pp_start_one()).

I'm not sure I see why the tty-ness would change between the two. If I'm being honest, I'm actually slightly surprised that `isatty` returned true for your hook before - since the hook process is a child of Git and its output is, presumably, being consumed by Git first rather than by an interactive user shell.

I suppose that with stdout_to_stderr being set, the tty-ness of the main process's stderr would then apply to the child process's stdout (we do that by calling `dup(2)`). But that's being set in both "the old way" and "the new way", so I'm pretty surprised to see a change here.

It *is* true that run-command.c:pp_start_one() sets child_process:err=-1 for the child and run-command.c:run_hook_ve() didn't do that; that -1 means that start_command() will create a new fd for the child's stderr. Since run_hook_ve() didn't care about the child's stderr before, I wonder if that is why? Could it be that now that we're processing the child's stderr, the child no longer thinks stderr is in tty, because the parent is consuming its output?

I think if that's the case, a fix would involve run-command.c:pp_start_one() not setting .err, .stdout_to_stderr, or .no_stdin at all on its own, and relying on the 'get_next_task' callback to set those things. It's a little more painful than I initially thought because the run_processes_parallel() library depends on that err capture to run pp_buffer_stderr() unconditionally; I guess it needs a tiny bit of shim logic to deal with callers who don't care to see their children's stderr.

All that said.... I'd expect that the dup() from the child's stdout to the parent's stderr would still result in a happy isatty(1). So I'm not convinced this is actually the right solution.... From your repro script, I can't quite tell which fd the isatty call is against (to be honest, I can't find the isatty call, either). So maybe I'm going the wrong direction :)

 - Emily
Previous: Anthony SottileNext: Anthony Sottile
Message 2 of 85 in “git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty”
  1. Anthony SottileApr 19, 2022
  2. Emily ShafferApr 19, 2022
  3. Anthony SottileApr 19, 2022
  4. Phillip WoodApr 20, 2022
  5. Ævar Arnfjörð BjarmasonApr 20, 2022
  6. Emily ShafferApr 20, 2022
  7. Junio C HamanoApr 20, 2022
  8. Emily ShafferApr 20, 2022
  9. Junio C HamanoApr 20, 2022
  10. Emily ShafferApr 20, 2022
  11. Ævar Arnfjörð BjarmasonApr 21, 2022
  12. Junio C HamanoApr 21, 2022
  13. Junio C HamanoApr 21, 2022
  14. Junio C HamanoApr 20, 2022
  15. 0/6 hook API: connect hooks to the TTY again, fixes a v2.36.0 regressionÆvar Arnfjörð Bjarmason, Apr 21, 2022
  16. 1/6 run-command API: replace run_processes_parallel_tr2() with opts structÆvar Arnfjörð Bjarmason, Apr 21, 2022
  17. Junio C HamanoApr 23, 2022
  18. Emily ShafferApr 28, 2022
  19. Junio C HamanoApr 29, 2022
  20. 2/6 run-command tests: test stdout of run_command_parallel()Ævar Arnfjörð Bjarmason, Apr 21, 2022
  21. Junio C HamanoApr 23, 2022
  22. 6/6 hook API: fix v2.36.0 regression: hooks should be connected to a TTYÆvar Arnfjörð Bjarmason, Apr 21, 2022
  23. Emily ShafferApr 28, 2022
  24. Junio C HamanoApr 29, 2022
  25. 3/6 run-command: add an "ungroup" option to run_process_parallel()Ævar Arnfjörð Bjarmason, Apr 21, 2022
  26. Junio C HamanoApr 23, 2022
  27. Emily ShafferApr 28, 2022
  28. 4/6 hook tests: fix redirection logic error in 96e7225b310Ævar Arnfjörð Bjarmason, Apr 21, 2022
  29. Junio C HamanoApr 23, 2022
  30. 5/6 hook API: don't redundantly re-set "no_stdin" and "stdout_to_stderr"Ævar Arnfjörð Bjarmason, Apr 21, 2022
  31. Junio C HamanoApr 29, 2022
  32. Junio C HamanoApr 21, 2022
  33. Ævar Arnfjörð BjarmasonApr 21, 2022
  34. 0/8 hook API: connect hooks to the TTY again, fixes a v2.36.0 regressionÆvar Arnfjörð Bjarmason, May 18, 2022
  35. 1/8 run-command tests: change if/if/... to if/else if/elseÆvar Arnfjörð Bjarmason, May 18, 2022
  36. 2/8 run-command API: use "opts" struct for run_processes_parallel{,_tr2}()Ævar Arnfjörð Bjarmason, May 18, 2022
  37. Junio C HamanoMay 18, 2022
  38. Emily ShafferMay 25, 2022
  39. 3/8 run-command tests: test stdout of run_command_parallel()Ævar Arnfjörð Bjarmason, May 18, 2022
  40. 4/8 run-command.c: add an initializer for "struct parallel_processes"Ævar Arnfjörð Bjarmason, May 18, 2022
  41. 7/8 hook API: don't redundantly re-set "no_stdin" and "stdout_to_stderr"Ævar Arnfjörð Bjarmason, May 18, 2022
  42. 6/8 hook tests: fix redirection logic error in 96e7225b310Ævar Arnfjörð Bjarmason, May 18, 2022
  43. 8/8 hook API: fix v2.36.0 regression: hooks should be connected to a TTYÆvar Arnfjörð Bjarmason, May 18, 2022
  44. Junio C HamanoMay 18, 2022
  45. Emily ShafferMay 26, 2022
  46. Ævar Arnfjörð BjarmasonMay 26, 2022
  47. Emily ShafferMay 26, 2022
  48. 5/8 run-command: add an "ungroup" option to run_process_parallel()Ævar Arnfjörð Bjarmason, May 18, 2022
  49. Junio C HamanoMay 18, 2022
  50. Emily ShafferMay 26, 2022
  51. Junio C HamanoMay 27, 2022
  52. Johannes SchindelinMay 25, 2022
  53. Ævar Arnfjörð BjarmasonMay 25, 2022
  54. Junio C HamanoMay 25, 2022
  55. Junio C HamanoMay 26, 2022
  56. Ævar Arnfjörð BjarmasonMay 26, 2022
  57. Junio C HamanoMay 26, 2022
  58. Ævar Arnfjörð BjarmasonMay 26, 2022
  59. Junio C HamanoMay 26, 2022
  60. 0/2 hook API: connect hooks to the TTY again, fixes a v2.36.0 regressionÆvar Arnfjörð Bjarmason, May 27, 2022
  61. 1/2 run-command: add an "ungroup" option to run_process_parallel()Ævar Arnfjörð Bjarmason, May 27, 2022
  62. Junio C HamanoMay 27, 2022
  63. 2/2 hook API: fix v2.36.0 regression: hooks should be connected to a TTYÆvar Arnfjörð Bjarmason, May 27, 2022
  64. Junio C HamanoMay 27, 2022
  65. 0/2 hook API: connect hooks to the TTY again, fixes a v2.36.0 regressionÆvar Arnfjörð Bjarmason, May 31, 2022
  66. 1/2 run-command: add an "ungroup" option to run_process_parallel()Ævar Arnfjörð Bjarmason, May 31, 2022
  67. Johannes SchindelinJun 1, 2022
  68. Junio C HamanoJun 1, 2022
  69. 2/2 hook API: fix v2.36.0 regression: hooks should be connected to a TTYÆvar Arnfjörð Bjarmason, May 31, 2022
  70. Johannes SchindelinJun 1, 2022
  71. Johannes SchindelinJun 1, 2022
  72. 0/2 hook API: connect hooks to the TTY again, fixes a v2.36.0 regressionÆvar Arnfjörð Bjarmason, Jun 2, 2022
  73. 1/2 run-command: add an "ungroup" option to run_process_parallel()Ævar Arnfjörð Bjarmason, Jun 2, 2022
  74. 2/2 hook API: fix v2.36.0 regression: hooks should be connected to a TTYÆvar Arnfjörð Bjarmason, Jun 2, 2022
  75. Junio C HamanoJun 2, 2022
  76. Phillip WoodJun 3, 2022
  77. Ævar Arnfjörð BjarmasonJun 3, 2022
  78. Phillip WoodJun 3, 2022
  79. Ævar Arnfjörð BjarmasonJun 7, 2022
  80. 0/2 hook API: connect hooks to the TTY again, fixes a v2.36.0 regressionÆvar Arnfjörð Bjarmason, Jun 7, 2022
  81. 2/2 hook API: fix v2.36.0 regression: hooks should be connected to a TTYÆvar Arnfjörð Bjarmason, Jun 7, 2022
  82. Junio C HamanoJun 7, 2022
  83. 1/2 run-command: add an "ungroup" option to run_process_parallel()Ævar Arnfjörð Bjarmason, Jun 7, 2022
  84. Emily ShafferJun 17, 2022
  85. Junio C HamanoJun 7, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.