git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v8 00/11] Git filter protocol

From
Lars Schneider <larsxschneider@gmail.com>
Date
Oct 3, 2016, 17:35 UTC
Message-ID
<56FAEF34-6A31-4E10-9671-F890796EF5A6@gmail.com>
In-Reply-To
<xmqqvax974dl.fsf@gitster.mtv.corp.google.com>
Show 48 quoted lines
> On 03 Oct 2016, at 19:02, Junio C Hamano <gitster@pobox.com> wrote:
> 
> Lars Schneider <larsxschneider@gmail.com> writes:
> 
>>> If the filter process refuses to die forever when Git told it to
>>> shutdown (by closing the pipe to it, for example), that filter
>>> process is simply buggy.  I think we want users to become aware of
>>> that, instead of Git leaving it behind, which essentially is to
>>> sweep the problem under the rug.
>>> 
>>> I agree with what Peff said elsewhere in the thread; if a filter
>>> process wants to take time to clean things up while letting Git
>>> proceed, it can do its own process management, but I think it is
>>> sensible for Git to wait the filter process it directly spawned.
>> 
>> To realize the approach above I prototyped the run-command patch below:
>> 
>> I added an "exit_timeout" variable to the "child_process" struct.
>> On exit, Git will close the pipe to the process and wait "exit_timeout" 
>> seconds until it kills the child process. If "exit_timeout" is negative
>> then Git will wait until the process is done.
> 
>> If we use that in the long running filter process, then we could make
>> the timeout even configurable. E.g. with "filter.<driver>.process-timeout".
>> 
>> What do you think about this solution? 
> 
> Is such a configuration (or timeout in general) necessary?  I
> suspect that a need for timeout, especially needing timeout and
> needing to get killed that happens so often to require a
> configuration variable, is a sign of something else seriously wrong.
> 
> What's the justification for a filter to _require_ getting killed
> all the time when it is spawned?  Otherwise you wouldn't configure
> "this driver does not die when told, so we need a timeout" variable.
> Is it a sign of the flaw in the protocol to talk to it?  e.g. Git
> has a way to tell it to die, but it somehow is very hard to hear
> from filter's end and honor that request?
> 
> I think that we would need some timeout in the mechanism, but not to
> be used for "killing".
> 
> You would decide to "kill" an filter process in two cases: the
> filter is buggy and refuses to die when Git tells it to exit, or the
> code in Git waiting for its death is somehow miscounting its
> children, and thought it told to die one process but in fact it
> didn't (perhaps it told somebody else to die), or it thought it
> hasn't seen the child die when in fact it already did.
Agreed.
> Calling kill(2) and exiting would hide these two kind of bugs from
> end users.  Not doing so would give the end users a hung Git, which
> is a VERY GOOD thing.  Otherwise you would not notice bugs and lose
> the opportunity to diagnose and fix it.

Aha. I assumed that a hung Git because of a buggy filter would be a no-no. Thanks for this clarification.

Show 7 quoted lines
> The timeout would be good for you to give a message "filter process
> running the script '%s' is not exiting; I am waiting for it".  The
> user is still left with a hung Git, and can then see if that process
> is hanging around.  If it is, then we found a buggy filter.  Or we
> found a buggy Git.  Either needs to be fixed.  I do not think it
> would help anybody by doing a kill(2) to sweep possible bugs under
> the rug.

I could achieve that with this run-command patch: http://public-inbox.org/git/E9946E9F-6EE5-492B-B122-9078CEB88044@gmail.com/ (I'll remove the "timeout after x seconds" parts and keep the "wait until done" part with stderr output)

Thanks, Lars

Previous: Junio C HamanoNext: Jeff King
Message 63 of 71 in “Git filter protocol”
  1. 00/11 Git filter protocollarsxschneider@gmail.com, Sep 20, 2016
  2. 01/11 pkt-line: rename packet_write() to packet_write_fmt()larsxschneider@gmail.com, Sep 20, 2016
  3. Jakub NarębskiSep 24, 2016
  4. Lars SchneiderSep 26, 2016
  5. Jakub NarębskiSep 28, 2016
  6. 03/11 run-command: move check_pipe() from write_or_die to run_commandlarsxschneider@gmail.com, Sep 20, 2016
  7. Jakub NarębskiSep 24, 2016
  8. Lars SchneiderSep 26, 2016
  9. Jakub NarębskiSep 26, 2016
  10. 02/11 pkt-line: extract set_packet_header()larsxschneider@gmail.com, Sep 20, 2016
  11. Jakub NarębskiSep 24, 2016
  12. Lars SchneiderSep 26, 2016
  13. 05/11 pkt-line: add packet_flush_gently()larsxschneider@gmail.com, Sep 20, 2016
  14. Jakub NarębskiSep 24, 2016
  15. 04/11 pkt-line: add packet_write_fmt_gently()larsxschneider@gmail.com, Sep 20, 2016
  16. Jakub NarębskiSep 24, 2016
  17. 10/11 convert: make apply_filter() adhere to standard Git error handlinglarsxschneider@gmail.com, Sep 20, 2016
  18. Jakub NarębskiSep 25, 2016
  19. 06/11 pkt-line: add packet_write_gently()larsxschneider@gmail.com, Sep 20, 2016
  20. Jakub NarębskiSep 25, 2016
  21. Lars SchneiderSep 26, 2016
  22. Jeff KingSep 27, 2016
  23. Jakub NarębskiSep 27, 2016
  24. 09/11 convert: modernize testslarsxschneider@gmail.com, Sep 20, 2016
  25. Jakub NarębskiSep 25, 2016
  26. 08/11 convert: quote filter names in error messageslarsxschneider@gmail.com, Sep 20, 2016
  27. Jakub NarębskiSep 25, 2016
  28. 07/11 pkt-line: add functions to read/write flush terminated packet streamslarsxschneider@gmail.com, Sep 20, 2016
  29. Jakub NarębskiSep 25, 2016
  30. Lars SchneiderSep 26, 2016
  31. Lars SchneiderSep 27, 2016
  32. Jeff KingSep 27, 2016
  33. Lars SchneiderSep 27, 2016
  34. Jeff KingSep 27, 2016
  35. 11/11 convert: add filter.<driver>.process optionlarsxschneider@gmail.com, Sep 20, 2016
  36. Jakub NarębskiSep 26, 2016
  37. Lars SchneiderSep 30, 2016
  38. Jakub NarębskiOct 4, 2016
  39. Lars SchneiderOct 6, 2016
  40. Jakub NarębskiSep 27, 2016
  41. Lars SchneiderSep 30, 2016
  42. Jakub NarębskiOct 4, 2016
  43. Lars SchneiderOct 6, 2016
  44. Jakub NarębskiSep 28, 2016
  45. Lars SchneiderOct 1, 2016
  46. Jakub NarębskiOct 4, 2016
  47. Junio C HamanoSep 28, 2016
  48. Lars SchneiderSep 29, 2016
  49. Torsten BögershausenSep 29, 2016
  50. Junio C HamanoSep 29, 2016
  51. Lars SchneiderSep 29, 2016
  52. Torsten BögershausenSep 29, 2016
  53. Johannes SixtSep 29, 2016
  54. Junio C HamanoSep 29, 2016
  55. Lars SchneiderOct 1, 2016
  56. Jakub NarębskiOct 1, 2016
  57. Lars SchneiderOct 3, 2016
  58. Jakub NarębskiOct 4, 2016
  59. Lars SchneiderOct 6, 2016
  60. Jeff KingOct 6, 2016
  61. Junio C HamanoOct 6, 2016
  62. Junio C HamanoOct 3, 2016
  63. Lars SchneiderOct 3, 2016
  64. Jeff KingOct 4, 2016
  65. Junio C HamanoOct 4, 2016
  66. Jeff KingSep 29, 2016
  67. Junio C HamanoSep 29, 2016
  68. Lars SchneiderSep 29, 2016
  69. Junio C HamanoSep 29, 2016
  70. Jakub NarębskiSep 29, 2016
  71. Junio C HamanoSep 29, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.