git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Designing the filter process protocol (was: Re: [PATCH v3 10/10] convert: add filter.<driver>.process option)

From
Lars Schneider <larsxschneider@gmail.com>
Date
Aug 6, 2016, 18:24 UTC
Message-ID
<90A43E20-B2F5-4377-8DC3-2298BDD7CC71@gmail.com>
In-Reply-To
<607c07fe-5b6f-fd67-13e1-705020c267ee@gmail.com>
Show 26 quoted lines
> On 03 Aug 2016, at 20:30, Jakub Narębski <jnareb@gmail.com> wrote:
> 
> ...
> 
> 
> 2. HANDSHAKE (INITIALIZATION)
> 
> Next, there is deciding on and designing the handshake between Git (between
> Git command) and the filter driver process.  With the `filter.<driver>.process`
> solution the driver needs to tell which operations among (for now) "clean"
> and "smudge" it does support.  Plus it provides a way to extend protocol,
> adding new features, like support for streaming, cleaning from file or
> smudging to file, providing size upfront, perhaps even progress report.
> 
> Current handshake consist of filter driver printing a signature, version
> number and capabilities, in that order.  Git checks that it is well formed
> and matches expectations, and notes which of "clean" and "smudge" operations
> are supported by the filter.
> 
> There is no interaction from the Git side in the handshake, for example to
> set options and expectations common to all files being filtered.  Take
> one possible extension of protocol: supporting streaming.  The filter
> driver needs to know whether it needs to read all the input, or whether
> it can start printing output while input is incoming (e.g. to reduce
> memory consumption)... though we may simply decide it to be next version
> of the protocol.
I would like to change the startup sequence to this:

Git starts the filter when it encounters the first file that needs to be cleaned or smudged. After the filter started Git sends a welcome message, a list of supported protocol version numbers, and a flush packet. Git expects to read the welcome message and one protocol version number from the previously sent list. Afterwards Git sends a list of supported capabilities and a flush packet. Git expects to read a list of desired capabilities, which must be a subset of the supported capabilities list, and a flush packet as response: ------------------------ packet: git> git-filter-client packet: git> version=2 packet: git> version=42 packet: git> 0000 packet: git< git-filter-server packet: git< version=2 packet: git> clean=true packet: git> smudge=true packet: git> not-yet-invented=true packet: git> 0000 packet: git< clean=true packet: git< smudge=true packet: git< 0000 ------------------------

This would allow us to detect the case if a user configures an existing clean/smudge filter as `filter.<driver>.process`. Since Git is talking first, it would not "hang" in that case.

Would that be ok with you?

Thanks, Lars

Previous: Jeff KingNext: Lars Schneider
Message 94 of 100 in “Git filter protocol”
  1. 00/10 Git filter protocollarsxschneider@gmail.com, Jul 29, 2016
  2. 01/10 pkt-line: extract set_packet_header()larsxschneider@gmail.com, Jul 29, 2016
  3. 02/10 pkt-line: add direct_packet_write() and direct_packet_write_data()larsxschneider@gmail.com, Jul 29, 2016
  4. 03/10 pkt-line: add packet_flush_gentle()larsxschneider@gmail.com, Jul 29, 2016
  5. 04/10 pkt-line: call packet_trace() only if a packet is actually sendlarsxschneider@gmail.com, Jul 29, 2016
  6. 05/10 pack-protocol: fix maximum pkt-line sizelarsxschneider@gmail.com, Jul 29, 2016
  7. 07/10 convert: quote filter names in error messageslarsxschneider@gmail.com, Jul 29, 2016
  8. 06/10 run-command: add clean_on_exit_handlerlarsxschneider@gmail.com, Jul 29, 2016
  9. 08/10 convert: modernize testslarsxschneider@gmail.com, Jul 29, 2016
  10. 09/10 convert: generate large test files only oncelarsxschneider@gmail.com, Jul 29, 2016
  11. 10/10 convert: add filter.<driver>.process optionlarsxschneider@gmail.com, Jul 29, 2016
  12. Johannes SixtJul 30, 2016
  13. Jakub NarębskiJul 30, 2016
  14. Jakub NarębskiJul 30, 2016
  15. Jakub NarębskiJul 30, 2016
  16. Jakub NarębskiJul 30, 2016
  17. Jakub NarębskiJul 30, 2016
  18. Jakub NarębskiJul 30, 2016
  19. Jakub NarębskiJul 31, 2016
  20. Lars SchneiderJul 31, 2016
  21. Torstem BögershausenJul 31, 2016
  22. Lars SchneiderJul 31, 2016
  23. Jakub NarębskiJul 31, 2016
  24. Jakub NarębskiJul 31, 2016
  25. Lars SchneiderAug 1, 2016
  26. Lars SchneiderAug 1, 2016
  27. Lars SchneiderAug 1, 2016
  28. Lars SchneiderAug 1, 2016
  29. Lars SchneiderAug 1, 2016
  30. Lars SchneiderAug 1, 2016
  31. Lars SchneiderAug 1, 2016
  32. Lars SchneiderAug 1, 2016
  33. Johannes SixtAug 2, 2016
  34. Lars SchneiderAug 2, 2016
  35. Torsten BögershausenAug 2, 2016
  36. Lars SchneiderAug 3, 2016
  37. 01/12 pkt-line: extract set_packet_header()larsxschneider@gmail.com, Aug 3, 2016
  38. 07/12 run-command: add clean_on_exit_handlerlarsxschneider@gmail.com, Aug 3, 2016
  39. 03/12 pkt-line: add packet_flush_gentle()larsxschneider@gmail.com, Aug 3, 2016
  40. 02/12 pkt-line: add direct_packet_write() and direct_packet_write_data()larsxschneider@gmail.com, Aug 3, 2016
  41. 08/12 convert: quote filter names in error messageslarsxschneider@gmail.com, Aug 3, 2016
  42. 05/12 pkt-line: add functions to read/write flush terminated packet streamslarsxschneider@gmail.com, Aug 3, 2016
  43. 09/12 convert: modernize testslarsxschneider@gmail.com, Aug 3, 2016
  44. 00/12 Git filter protocollarsxschneider@gmail.com, Aug 3, 2016
  45. 12/12 convert: add filter.<driver>.process shutdown command optionlarsxschneider@gmail.com, Aug 3, 2016
  46. 06/12 pack-protocol: fix maximum pkt-line sizelarsxschneider@gmail.com, Aug 3, 2016
  47. 04/12 pkt-line: call packet_trace() only if a packet is actually sendlarsxschneider@gmail.com, Aug 3, 2016
  48. 11/12 convert: add filter.<driver>.process optionlarsxschneider@gmail.com, Aug 3, 2016
  49. 10/12 convert: generate large test files only oncelarsxschneider@gmail.com, Aug 3, 2016
  50. Junio C HamanoAug 3, 2016
  51. Designing the filter process protocol (was: Re: [PATCH v3 10/10] convert: add filter.<driver>.process option)Jakub Narębski, Aug 3, 2016
  52. Jakub NarębskiAug 3, 2016
  53. Junio C HamanoAug 3, 2016
  54. Jakub NarębskiAug 3, 2016
  55. Jakub NarębskiAug 3, 2016
  56. Junio C HamanoAug 3, 2016
  57. Jeff KingAug 3, 2016
  58. Jeff KingAug 3, 2016
  59. Jeff KingAug 3, 2016
  60. Lars SchneiderAug 3, 2016
  61. Lars SchneiderAug 3, 2016
  62. Jeff KingAug 3, 2016
  63. Lars SchneiderAug 3, 2016
  64. Junio C HamanoAug 3, 2016
  65. Lars SchneiderAug 3, 2016
  66. Lars SchneiderAug 3, 2016
  67. Jeff KingAug 3, 2016
  68. Jakub NarębskiAug 3, 2016
  69. Jeff KingAug 3, 2016
  70. Lars SchneiderAug 3, 2016
  71. Jeff KingAug 3, 2016
  72. Jakub NarębskiAug 4, 2016
  73. Jakub NarębskiAug 4, 2016
  74. Junio C HamanoAug 4, 2016
  75. Junio C HamanoAug 4, 2016
  76. Lars SchneiderAug 5, 2016
  77. Lars SchneiderAug 5, 2016
  78. Lars SchneiderAug 5, 2016
  79. Lars SchneiderAug 5, 2016
  80. Lars SchneiderAug 5, 2016
  81. Lars SchneiderAug 5, 2016
  82. Lars SchneiderAug 5, 2016
  83. Lars SchneiderAug 5, 2016
  84. Junio C HamanoAug 5, 2016
  85. Lars SchneiderAug 5, 2016
  86. Junio C HamanoAug 5, 2016
  87. Torsten BögershausenAug 5, 2016
  88. Torsten BögershausenAug 5, 2016
  89. Lars SchneiderAug 5, 2016
  90. Lars SchneiderAug 5, 2016
  91. Junio C HamanoAug 5, 2016
  92. Jeff KingAug 5, 2016
  93. Jeff KingAug 6, 2016
  94. Lars SchneiderAug 6, 2016
  95. Lars SchneiderAug 6, 2016
  96. Lars SchneiderAug 6, 2016
  97. Torsten BögershausenAug 6, 2016
  98. Jeff KingAug 8, 2016
  99. Lars SchneiderAug 8, 2016
  100. Jeff KingAug 8, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.