git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 8/8] fetch: introduce machine-parseable "porcelain" output format

From
Patrick Steinhardt <ps@pks.im>
Date
Apr 28, 2023, 08:51 UTC
Message-ID
<ZEuJFOBCdnCLdGc5@ncase>
In-Reply-To
<kl6lfs8knc9f.fsf@chooglen-macbookpro.roam.corp.google.com>
On Thu, Apr 27, 2023 at 04:20:12PM -0700, Glen Choo wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> > On Wed, Apr 26, 2023 at 12:52:46PM -0700, Glen Choo wrote:
> >> Patrick Steinhardt <ps@pks.im> writes:
[snip]
Show 32 quoted lines
> > So with that in mind, I'd like to continue ignoring this issue for now
> > and just not report the remote that the ref came from. But I'd also
> > argue that we don't have to restrict porcelain mode to single-remote
> > fetches: it can still be useful to do multi-remote fetches even without
> > the information where a certain reference update comes from. So any kind
> > of restriction would feel artificial to me here.
> >
> > Furthermore, I'd argue that it is not necessary to label the format as
> > experimental only because of this limitation. With the refactorings done
> > in this and the preceding patch series it is easy to add a new format in
> > case there indeed is somebody that would have a usecase for this. The
> > "porcelain" format should stay stable, and if we decide that we want to
> > also report the remote for each reference in a follow-up we can easily
> > add a "porcelain-v2" or "porcelain-with-remote" format.
> 
> I agree that the current form is useful, but it seems quite likely that
> users will want the remote name/url at some point. In the majority of
> cases, the user can parse the remote name from the updated local ref and
> if they want, they look up the url by asking "git config", but that
> breaks down quite quickly if they have a custom refspec.
> 
> My suggestion to label it experimental was meant to give us the freedom
> to change the "first version" in backwards-incompatible ways that we
> otherwise couldn't. We could say "here's porcelain-v2, use this and
> ignore porcelain-v1", but it would be even nicer if the user didn't need
> to think about porcelain-v1. The idea is that we could call it
> experimental, see what users need in the wild, then make changes without
> having to create a new "porcelain-v2" mode.
> 
> (I realize in hindsight that calling it experimental sounds a bit
> insulting. I thought your proposal was quite sensible, actually. i
> apologize if it came across that way)
No offense taken, so no need to apologize.
> Perhaps part of the reason why you'd prefer not to make it experimental
> is that users who might want this feature are too afraid of
> backwards-incompatible changes. I think that's fair.

Exactly. I'm implementing this whole feature with a concrete usecase at GitLab in mind, and if we declare this as experimental then it will become a whole lot more risky for us to actually adopt it. Because if the output format may change at any point in time, then the code that builds on top of it is likely to be broken.

Show 5 quoted lines
> A different way of
> achieving the same flexibility would be to make the "porcelain" format
> extensible (like git status --porcelain=v2). A future, multivalued
> "--porcelain-options=show-remote-url,show-remote-ref" would cover
> our bases.

Yeah, that would indeed be quite flexible. Theoretically, there is no reason why we couldn't have `--output-format=porcelain,show-remote-url` though.

> I'm happy with either approach as long as we don't have to bikeshed
> about the "perfect" porcelain output :)

Agreed, and that's why I'm currently defending the "good enough" format. It should likely work for most usecases that exist out there. The target audience is going to be quite small here as this is not a user-directed feature. Furthermore, I assume that the overlap of machines that want to parse all reference updates done in a multi-remote fetch while also being able to exactly tell which remote brought in what updates is going to be tiny.

Show 31 quoted lines
> >> > With these assumptions, the output format becomes unambiguously
> >> > parseable. Furthermore, given that this output is designed to be
> >> > consumed by scripts, the machine-readable data is printed to stdout
> >> > instead of stderr like the human-readable output is. This is mostly done
> >> > so that other data printed to stderr, like error messages or progress
> >> > meters, don't interfere with the parseable data.
> >> 
> >> Sending the 'main output' to stdout makes sense to me, but this (and
> >> possibly respecting -z) sounds like a different mode of operation, not
> >> just a matter of formats. It seems different enough that I'd prefer not
> >> to piggyback on "fetch.output" for this (even though this adds more
> >> surface to the interface...).
> >> 
> >> We could add --porcelain and say that "fetch.output" is ignored if
> >> --porcelain is also given. That also eliminates the need for
> >> --output-format, I think.
> >
> > I was thinking about this initially, as well. But ultimately I decided
> > against this especially because of your second paragraph: we'd now need
> > to think about precedence of options and mutual exclusion, and that to
> > me feels like an interface that is less obvious than a single knob that
> > works as you'd expect.
> 
> My full thoughts on this are in
> 
>   https://lore.kernel.org/git/kl6lildhlz3i.fsf@chooglen-macbookpro.roam.corp.google.com
> 
> but the short version is that I'm not sure if I expect something as
> innocuous-looking as --output-format would imply other, machine-friendly
> things (like stdout instead of stderr), and using --porcelain might make
> option precedence clearer in some situtations (like if -z is given).

I'm not even sure that `-z` makes sense in this context. If we see cases where not using `-z` can cause the machine-readable interface to become unparseable then this is a bug in the output format, if you ask me. Mostly because the whole intent of it is to be machine-parseable. So if we output data that can e.g. contain newlines, then we must not use newlines as part of the output format or alternatively escape them. Why let the author of the script shoot themselves into the foot?

Anyway, I'm digressing. It's hard for me to decide what to do right now. The thread with Junio and Jacob points into the direction of keeping the `--output-format=` interface, while this thread points into the other direction. I'm naturally more inclined to keep `--output-format=`, mostly because I personally feel like it's the more obvious interface. But I also see your point, so it's not really a choice of right-or-wrong here, but rather of style.

Patrick
Previous: Glen ChooNext: Glen Choo
Message 21 of 120 in “fetch: introduce machine-parseable output”
  1. 0/8 fetch: introduce machine-parseable outputPatrick Steinhardt, Apr 19, 2023
  2. 1/8 fetch: split out tests for output formatPatrick Steinhardt, Apr 19, 2023
  3. 2/8 fetch: add a test to exercise invalid output formatsPatrick Steinhardt, Apr 19, 2023
  4. 3/8 fetch: fix missing from-reference when fetching HEAD:fooPatrick Steinhardt, Apr 19, 2023
  5. Jacob KellerApr 26, 2023
  6. Patrick SteinhardtApr 27, 2023
  7. Jacob KellerApr 26, 2023
  8. Patrick SteinhardtApr 27, 2023
  9. Glen ChooApr 26, 2023
  10. Patrick SteinhardtApr 27, 2023
  11. 4/8 fetch: introduce `display_format` enumPatrick Steinhardt, Apr 19, 2023
  12. 5/8 fetch: move display format parsing into main functionPatrick Steinhardt, Apr 19, 2023
  13. 6/8 fetch: move option related variables into main functionPatrick Steinhardt, Apr 19, 2023
  14. 7/8 fetch: introduce new `--output-format` optionPatrick Steinhardt, Apr 19, 2023
  15. Glen ChooApr 26, 2023
  16. Patrick SteinhardtApr 27, 2023
  17. 8/8 fetch: introduce machine-parseable "porcelain" output formatPatrick Steinhardt, Apr 19, 2023
  18. Glen ChooApr 26, 2023
  19. Patrick SteinhardtApr 27, 2023
  20. Glen ChooApr 27, 2023
  21. Patrick SteinhardtApr 28, 2023
  22. Glen ChooApr 28, 2023
  23. Felipe ContrerasMay 2, 2023
  24. Felipe ContrerasApr 24, 2023
  25. Patrick SteinhardtApr 25, 2023
  26. Jacob KellerApr 26, 2023
  27. Junio C HamanoApr 26, 2023
  28. Jacob KellerApr 26, 2023
  29. Patrick SteinhardtApr 27, 2023
  30. Jacob KellerApr 27, 2023
  31. Glen ChooApr 27, 2023
  32. Junio C HamanoApr 26, 2023
  33. Glen ChooApr 26, 2023
  34. Glen ChooApr 26, 2023
  35. Jacob KellerApr 26, 2023
  36. 0/8 fetch: introduce machine-parseable outputPatrick Steinhardt, Apr 27, 2023
  37. 1/8 fetch: split out tests for output formatPatrick Steinhardt, Apr 27, 2023
  38. SZEDER GáborApr 29, 2023
  39. Patrick SteinhardtMay 3, 2023
  40. 2/8 fetch: add a test to exercise invalid output formatsPatrick Steinhardt, Apr 27, 2023
  41. 3/8 fetch: fix missing from-reference when fetching HEAD:fooPatrick Steinhardt, Apr 27, 2023
  42. Glen ChooApr 27, 2023
  43. Jacob KellerApr 27, 2023
  44. 4/8 fetch: introduce `display_format` enumPatrick Steinhardt, Apr 27, 2023
  45. 5/8 fetch: move display format parsing into main functionPatrick Steinhardt, Apr 27, 2023
  46. 6/8 fetch: move option related variables into main functionPatrick Steinhardt, Apr 27, 2023
  47. Junio C HamanoApr 27, 2023
  48. 7/8 fetch: introduce new `--output-format` optionPatrick Steinhardt, Apr 27, 2023
  49. Junio C HamanoApr 27, 2023
  50. Glen ChooApr 28, 2023
  51. Patrick SteinhardtMay 3, 2023
  52. Glen ChooApr 28, 2023
  53. Patrick SteinhardtMay 3, 2023
  54. Patrick SteinhardtMay 3, 2023
  55. 8/8 fetch: introduce machine-parseable "porcelain" output formatPatrick Steinhardt, Apr 27, 2023
  56. Jacob KellerApr 27, 2023
  57. Glen ChooApr 28, 2023
  58. 0/8 fetch: introduce machine-parseable outputPatrick Steinhardt, May 3, 2023
  59. 1/8 fetch: fix `--no-recurse-submodules` with multi-remote fetchesPatrick Steinhardt, May 3, 2023
  60. Glen ChooMay 8, 2023
  61. Patrick SteinhardtMay 9, 2023
  62. 2/8 fetch: split out tests for output formatPatrick Steinhardt, May 3, 2023
  63. 3/8 fetch: add a test to exercise invalid output formatsPatrick Steinhardt, May 3, 2023
  64. 4/8 fetch: fix missing from-reference when fetching HEAD:fooPatrick Steinhardt, May 3, 2023
  65. 5/8 fetch: introduce `display_format` enumPatrick Steinhardt, May 3, 2023
  66. 6/8 fetch: move display format parsing into main functionPatrick Steinhardt, May 3, 2023
  67. 7/8 fetch: move option related variables into main functionPatrick Steinhardt, May 3, 2023
  68. 8/8 fetch: introduce machine-parseable "porcelain" output formatPatrick Steinhardt, May 3, 2023
  69. Glen ChooMay 8, 2023
  70. Patrick SteinhardtMay 9, 2023
  71. Glen ChooMay 9, 2023
  72. Junio C HamanoMay 3, 2023
  73. Junio C HamanoMay 3, 2023
  74. Patrick SteinhardtMay 4, 2023
  75. Glen ChooMay 9, 2023
  76. Patrick SteinhardtMay 9, 2023
  77. 0/8 fetch: introduce machine-parseable outputPatrick Steinhardt, May 9, 2023
  78. 1/8 fetch: fix `--no-recurse-submodules` with multi-remote fetchesPatrick Steinhardt, May 9, 2023
  79. Junio C HamanoMay 9, 2023
  80. Glen ChooMay 9, 2023
  81. Patrick SteinhardtMay 10, 2023
  82. 2/8 fetch: split out tests for output formatPatrick Steinhardt, May 9, 2023
  83. 3/8 fetch: add a test to exercise invalid output formatsPatrick Steinhardt, May 9, 2023
  84. Junio C HamanoMay 9, 2023
  85. Patrick SteinhardtMay 10, 2023
  86. 4/8 fetch: fix missing from-reference when fetching HEAD:fooPatrick Steinhardt, May 9, 2023
  87. Junio C HamanoMay 9, 2023
  88. Patrick SteinhardtMay 10, 2023
  89. 5/8 fetch: introduce `display_format` enumPatrick Steinhardt, May 9, 2023
  90. Junio C HamanoMay 9, 2023
  91. Patrick SteinhardtMay 10, 2023
  92. 6/8 fetch: move display format parsing into main functionPatrick Steinhardt, May 9, 2023
  93. Junio C HamanoMay 9, 2023
  94. Glen ChooMay 9, 2023
  95. Patrick SteinhardtMay 10, 2023
  96. 7/8 fetch: move option related variables into main functionPatrick Steinhardt, May 9, 2023
  97. 8/8 fetch: introduce machine-parseable "porcelain" output formatPatrick Steinhardt, May 9, 2023
  98. Junio C HamanoMay 9, 2023
  99. Patrick SteinhardtMay 10, 2023
  100. 0/9 fetch: introduce machine-parseable outputPatrick Steinhardt, May 10, 2023
  101. 1/9 fetch: fix `--no-recurse-submodules` with multi-remote fetchesPatrick Steinhardt, May 10, 2023
  102. 2/9 fetch: split out tests for output formatPatrick Steinhardt, May 10, 2023
  103. 3/9 fetch: add a test to exercise invalid output formatsPatrick Steinhardt, May 10, 2023
  104. 4/9 fetch: print left-hand side when fetching HEAD:fooPatrick Steinhardt, May 10, 2023
  105. Glen ChooMay 12, 2023
  106. Jeff KingMay 13, 2023
  107. Patrick SteinhardtMay 15, 2023
  108. 5/9 fetch: refactor calculation of the display table widthPatrick Steinhardt, May 10, 2023
  109. Glen ChooMay 12, 2023
  110. 6/9 fetch: introduce `display_format` enumPatrick Steinhardt, May 10, 2023
  111. 7/9 fetch: lift up parsing of "fetch.output" config variablePatrick Steinhardt, May 10, 2023
  112. 8/9 fetch: move option related variables into main functionPatrick Steinhardt, May 10, 2023
  113. 9/9 fetch: introduce machine-parseable "porcelain" output formatPatrick Steinhardt, May 10, 2023
  114. Glen ChooMay 12, 2023
  115. Junio C HamanoMay 10, 2023
  116. Patrick SteinhardtMay 11, 2023
  117. Junio C HamanoMay 11, 2023
  118. Felipe ContrerasMay 11, 2023
  119. Glen ChooMay 12, 2023
  120. Patrick SteinhardtMay 12, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.