git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 2/3] builtin: introduce diff-pairs command

From
Justin Tobler <jltobler@gmail.com>
Date
Feb 19, 2025, 20:51 UTC
Message-ID
<wv5ziveuff7iellcmjcki372m5vp6nmltyls43e4wzslcqymog@gwczuaucpkke>
In-Reply-To
<d6d4230e-7b80-4eec-b218-37717ae2e298@gmail.com>
On 25/02/17 02:38PM, Phillip Wood wrote:
Show 34 quoted lines
> Hi Justin
> 
> On 12/02/2025 04:18, Justin Tobler wrote:
> > Through git-diff(1), a single diff can be generated from a pair of blob
> > revisions directly. Unfortunately, there is not a mechanism to compute
> > batches of specific file pair diffs in a single process. Such a feature
> > is particularly useful on the server-side where diffing between a large
> > set of changes is not feasible all at once due to timeout concerns.
> > 
> > To facilitate this, introduce git-diff-pairs(1) which takes the
> > null-terminated raw diff format as input on stdin and produces diffs in
> > other formats. As the raw diff format already contains the necessary
> > metadata, it becomes possible to progressively generate batches of diffs
> > without having to recompute rename detection or retrieve object context.
> > Something like the following:
> > 
> > 	git diff-tree -r -z -M $old $new |
> > 	git diff-pairs -p
> > 
> > should generate the same output as `git diff-tree -p -M`. Furthermore,
> > each line of raw diff formatted input can also be individually fed to a
> > separate git-diff-pairs(1) process and still produce the same output.
> 
> I like the idea of this, I've left a few comments mainly around the UI.
> 
> > +Here's an incomplete list of things that `diff-pairs` could do, but
> > +doesn't (mostly in the name of simplicity):
> > +
> > + - Only `-z` input is accepted, not normal `--raw` input.
> 
> I think only accepting NUL terminated input is fine, but if we want to
> accept other formats we should  have a plan for how to do that in a
> backwards compatible way as we cannot use `-z` to distinguish between input
> formats.

If in the future we want to support the normal format, we could introduce an `--input-format=normal` option or something along those lines.

Show 12 quoted lines
> > +	const char * const usage[] = {
> > +		N_("git diff-pairs [diff-options]"),
> 
> Normally the option summary printed by "git foo -h" is generated by the
> option parser. In this case we don't define any options and use
> setup_revisions() instead so we need to provide the option summary
> ourselves. Looking at diff-files.c we can add
> 
> 	"\n"
> 	COMMON_DIFF_OPTIONS_HELP;
> 
> to do that.

Would this be preferable even if git-diff-pairs doesn't support all of the common diff options?

> > +	argc = setup_revisions(argc, argv, &revs, NULL);
> 
> I think we should check that there are no options left on the commandline
> after setup_revisions() returns
Good call, will do in the next version.
Show 15 quoted lines
> > +	/* Don't allow pathspecs at all. */
> > +	if (revs.prune_data.nr)
> > +		usage_with_options(usage, options);
> 
> It is not just pathspecs that we want to reject but all revision related
> options. Looking at diff-files.c we can do
> 
> 	if (rev.pending.nr ||
> 	    rev.min_age != -1 || rev.max_age != -1 ||
> 	    rev.max_count != -1)
> 		usage_with_option(usage, options);
> 
> To catch some of that but it still accepts things like "--first-parent",
> "--merges" and "--ancestry-path". We may just have to live with that as I
> don't think it is worth expanding a huge amount of effort to prevent them.

Yes, we should also reject revision as well as pathspec arguments. Will update.

Show 6 quoted lines
> > +	if (!revs.diffopt.output_format)
> > +		revs.diffopt.output_format = DIFF_FORMAT_RAW;
> 
> This matches the other diff plumbing commands but I'm not sure it is the
> most helpful default for a command that is supposed to transform raw diffs
> into another format. Maybe we should default to DIFF_FORMAT_PATCH?

As you mentioned, defaulting to DIFF_FORMAT_RAW isn't the most useful behavior. I agree that it makes more sense to use DIFF_FORMAT_PATCH as the default. Will update in the next version.

> > +test_expect_success 'split input across multiple diff-pairs' '
> 
> This needs a PERL prerequisite I think. I'm a bit unsure what this test adds
> compared to the others.

This test demonstrates that the raw diff input can be split across separate git-diff-pairs(1) processes and still produce equivilant output which is one of the main usecases for the command. That being said, this test isn't really exercising different behavior of git-diff-pairs(1) itself, so maybe it would be best to drop it.

Thanks for the review :)
-Justin
Previous: Phillip WoodNext: Junio C Hamano
Message 41 of 78 in “batch blob diff generation”
  1. 0/3 batch blob diff generationJustin Tobler, Dec 13, 2024
  2. 1/3 builtin: introduce diff-blob commandJustin Tobler, Dec 13, 2024
  3. 2/3 builtin/diff-blob: add "--stdin" optionJustin Tobler, Dec 13, 2024
  4. 3/3 builtin/diff-blob: Add "-z" optionJustin Tobler, Dec 13, 2024
  5. Jeff KingDec 13, 2024
  6. Junio C HamanoDec 13, 2024
  7. Jeff KingDec 13, 2024
  8. Junio C HamanoDec 15, 2024
  9. Junio C HamanoDec 15, 2024
  10. Jeff KingDec 16, 2024
  11. Junio C HamanoDec 16, 2024
  12. Jeff KingDec 18, 2024
  13. Junio C HamanoDec 18, 2024
  14. Jeff KingDec 20, 2024
  15. Jeff KingDec 20, 2024
  16. Justin ToblerDec 13, 2024
  17. Jeff KingDec 16, 2024
  18. Junio C HamanoDec 13, 2024
  19. Junio C HamanoDec 15, 2024
  20. Jeff KingDec 16, 2024
  21. 0/3 batch blob diff generationJustin Tobler, Feb 12, 2025
  22. 1/3 diff: return diff_filepair from diff queue helpersJustin Tobler, Feb 12, 2025
  23. Karthik NayakFeb 12, 2025
  24. Justin ToblerFeb 12, 2025
  25. Patrick SteinhardtFeb 12, 2025
  26. Justin ToblerFeb 12, 2025
  27. Patrick SteinhardtFeb 13, 2025
  28. 2/3 builtin: introduce diff-pairs commandJustin Tobler, Feb 12, 2025
  29. Patrick SteinhardtFeb 12, 2025
  30. Karthik NayakFeb 12, 2025
  31. Justin ToblerFeb 25, 2025
  32. Jean-Noël AvilaFeb 12, 2025
  33. Junio C HamanoFeb 12, 2025
  34. Justin ToblerFeb 19, 2025
  35. Junio C HamanoFeb 19, 2025
  36. Junio C HamanoFeb 19, 2025
  37. Justin ToblerFeb 20, 2025
  38. Justin ToblerFeb 20, 2025
  39. Junio C HamanoFeb 20, 2025
  40. Phillip WoodFeb 17, 2025
  41. Justin ToblerFeb 19, 2025
  42. Junio C HamanoFeb 19, 2025
  43. Justin ToblerFeb 19, 2025
  44. Phillip WoodFeb 26, 2025
  45. 3/3 builtin/diff-pairs: allow explicit diff queue flushJustin Tobler, Feb 12, 2025
  46. Patrick SteinhardtFeb 12, 2025
  47. Phillip WoodFeb 17, 2025
  48. Justin ToblerFeb 19, 2025
  49. 0/3 batch blob diff generationJustin Tobler, Feb 25, 2025
  50. 1/3 diff: return diff_filepair from diff queue helpersJustin Tobler, Feb 25, 2025
  51. Junio C HamanoFeb 26, 2025
  52. 2/3 builtin: introduce diff-pairs commandJustin Tobler, Feb 25, 2025
  53. Junio C HamanoFeb 26, 2025
  54. Justin ToblerFeb 27, 2025
  55. Karthik NayakFeb 27, 2025
  56. Justin ToblerFeb 27, 2025
  57. Patrick SteinhardtFeb 27, 2025
  58. Justin ToblerFeb 27, 2025
  59. 3/3 builtin/diff-pairs: allow explicit diff queue flushJustin Tobler, Feb 25, 2025
  60. phillip.wood123@gmail.comFeb 26, 2025
  61. Justin ToblerFeb 27, 2025
  62. 0/4 batch blob diff generationJustin Tobler, Feb 28, 2025
  63. 1/4 diff: return diff_filepair from diff queue helpersJustin Tobler, Feb 28, 2025
  64. 2/4 diff: add option to skip resolving diff statusesJustin Tobler, Feb 28, 2025
  65. Patrick SteinhardtFeb 28, 2025
  66. Justin ToblerFeb 28, 2025
  67. 3/4 builtin: introduce diff-pairs commandJustin Tobler, Feb 28, 2025
  68. Patrick SteinhardtFeb 28, 2025
  69. Justin ToblerFeb 28, 2025
  70. 4/4 builtin/diff-pairs: allow explicit diff queue flushJustin Tobler, Feb 28, 2025
  71. 0/4 batch blob diff generationJustin Tobler, Feb 28, 2025
  72. 1/4 diff: return diff_filepair from diff queue helpersJustin Tobler, Feb 28, 2025
  73. Junio C HamanoMar 3, 2025
  74. 2/4 diff: add option to skip resolving diff statusesJustin Tobler, Feb 28, 2025
  75. Junio C HamanoMar 3, 2025
  76. 3/4 builtin: introduce diff-pairs commandJustin Tobler, Feb 28, 2025
  77. Junio C HamanoMar 3, 2025
  78. 4/4 builtin/diff-pairs: allow explicit diff queue flushJustin Tobler, Feb 28, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.