{"thread":{"id":"62633","subject":"[PATCH 0/3] batch blob diff generation","startedAt":"2024-12-13T04:26:13Z","lastAt":"2025-03-03T16:30:37Z","messageCount":78,"participants":["Justin Tobler","Jeff King","Junio C Hamano","Karthik Nayak","Patrick Steinhardt","Jean-Noël Avila","Phillip Wood","phillip.wood123@gmail.com"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"509042","messageId":"20241213042312.2890841-1-jltobler@gmail.com","threadId":"62633","inReplyTo":null,"subject":"[PATCH 0/3] batch blob diff generation","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-12-13T04:23:09Z","receivedAt":"2024-12-13T04:26:13Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Through git-diff(1) it is possible to generate a diff directly between\ntwo blobs. This is particularly useful when the pre-image and post-image\nblobs are known and we only care about the diff between them.\nUnfortunately, if a user has a batch of known blob pairs to compute\ndiffs for, there is currently not a way to do so via a single Git\nprocess.\n\nTo enable support for batch diffs of multiple blob pairs, this\nseries introduces a new diff plumbing command git-diff-blob(1). Similar\nto git-diff-tree(1), it provides a \"--stdin\" option that reads a pair of\nblobs on each line of input and generates the diffs. This is intended to\nbe used for scripting purposes where more fine-grained control for diff\ngeneration is desired. Below is an example for each usage:\n\n $ git diff-blob HEAD~5000:README.md HEAD:README.md\n\n $ git diff-blob --stdin <\n---\n .gitignore | 1 +\n Documentation/git-diff-blob.txt | 29 ++++++++\n Documentation/meson.build | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/diff-blob.c | 117 ++++++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n meson.build | 1 +\n t/t4063-diff-blobs.sh | 100 ++++++++++++++-------------\n 10 files changed, 205 insertions(+), 48 deletions(-)\n create mode 100644 Documentation/git-diff-blob.txt\n create mode 100644 builtin/diff-blob.c\n\ndiff --git a/.gitignore b/.gitignore\nindex e82aa19df0..e7487072bd 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -52,6 +52,7 @@\n /git-daemon\n /git-diagnose\n /git-diff\n+/git-diff-blob\n /git-diff-files\n /git-diff-index\n /git-diff-tree\ndiff --git a/Documentation/git-diff-blob.txt b/Documentation/git-diff-blob.txt\nnew file mode 100644\nindex 0000000000..732992d1d7\n--- /dev/null\n+++ b/Documentation/git-diff-blob.txt\n@@ -0,0 +1,29 @@\n+git-diff-blob(1)\n+================\n+\n+NAME\n+----\n+git-diff-blob - Compares the content and mode of two specified blobs\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git diff-blob' \n+\n+DESCRIPTION\n+-----------\n+Compare the content and mode of two specified blobs.\n+\n+OPTIONS\n+-------\n+::\n+\tThe id of a blob object or path-scoped revision that resolves to a blob.\n+\n+include::pretty-formats.txt[]\n+\n+include::diff-format.txt[]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Documentation/meson.build b/Documentation/meson.build\nindex f2426ccaa3..1bf0a80419 100644\n--- a/Documentation/meson.build\n+++ b/Documentation/meson.build\n@@ -39,6 +39,7 @@ manpages = {\n 'git-daemon.txt' : 1,\n 'git-describe.txt' : 1,\n 'git-diagnose.txt' : 1,\n+ 'git-diff-blob.txt' : 1,\n 'git-diff-files.txt' : 1,\n 'git-diff-index.txt' : 1,\n 'git-difftool.txt' : 1,\ndiff --git a/Makefile b/Makefile\nindex 06f01149ec..de2e43d4f6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1235,6 +1235,7 @@ BUILTIN_OBJS += builtin/credential-store.o\n BUILTIN_OBJS += builtin/credential.o\n BUILTIN_OBJS += builtin/describe.o\n BUILTIN_OBJS += builtin/diagnose.o\n+BUILTIN_OBJS += builtin/diff-blob.o\n BUILTIN_OBJS += builtin/diff-files.o\n BUILTIN_OBJS += builtin/diff-index.o\n BUILTIN_OBJS += builtin/diff-tree.o\ndiff --git a/builtin.h b/builtin.h\nindex f7b166b334..383e78ca99 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -152,6 +152,7 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix, struct reposit\n int cmd_diff_files(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_diff_index(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_diff(int argc, const char **argv, const char *prefix, struct repository *repo);\n+int cmd_diff_blob(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_diff_tree(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_difftool(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_env__helper(int argc, const char **argv, const char *prefix, struct repository *repo);\ndiff --git a/builtin/diff-blob.c b/builtin/diff-blob.c\nnew file mode 100644\nindex 0000000000..7cfa4eb436\n--- /dev/null\n+++ b/builtin/diff-blob.c\n@@ -0,0 +1,117 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"diff.h\"\n+#include \"diffcore.h\"\n+#include \"gettext.h\"\n+#include \"hash.h\"\n+#include \"object.h\"\n+#include \"parse-options.h\"\n+#include \"revision.h\"\n+\n+static void diff_blobs(struct object_array_entry *old_blob,\n+\t\t struct object_array_entry *new_blob,\n+\t\t struct diff_options *opts)\n+{\n+\tconst unsigned mode = canon_mode(S_IFREG | 0644);\n+\tstruct object_id old_oid = old_blob->item->oid;\n+\tstruct object_id new_oid = new_blob->item->oid;\n+\tunsigned old_mode = old_blob->mode;\n+\tunsigned new_mode = new_blob->mode;\n+\tchar *old_path = old_blob->path;\n+\tchar *new_path = new_blob->path;\n+\tstruct diff_filespec *old, *new;\n+\n+\tif (old_mode == S_IFINVALID)\n+\t\told_mode = mode;\n+\n+\tif (new_mode == S_IFINVALID)\n+\t\tnew_mode = mode;\n+\n+\tif (!old_path)\n+\t\told_path = old_blob->name;\n+\n+\tif (!new_path)\n+\t\tnew_path = new_blob->name;\n+\n+\tif (!is_null_oid(&old_oid) && !is_null_oid(&new_oid) &&\n+\t oideq(&old_oid, &new_oid) && (old_mode == new_mode))\n+\t\treturn;\n+\n+\tif (opts->flags.reverse_diff) {\n+\t\tSWAP(old_oid, new_oid);\n+\t\tSWAP(old_mode, new_mode);\n+\t\tSWAP(old_path, new_path);\n+\t}\n+\n+\tif (opts->prefix &&\n+\t (strncmp(old_path, opts->prefix, opts->prefix_length) ||\n+\t strncmp(new_path, opts->prefix, opts->prefix_length)))\n+\t\treturn;\n+\n+\told = alloc_filespec(old_path);\n+\tnew = alloc_filespec(new_path);\n+\n+\tfill_filespec(old, &old_oid, 1, old_mode);\n+\tfill_filespec(new, &new_oid, 1, new_mode);\n+\n+\tdiff_queue(&diff_queued_diff, old, new);\n+\tdiffcore_std(opts);\n+\tdiff_flush(opts);\n+}\n+\n+int cmd_diff_blob(int argc, const char **argv, const char *prefix,\n+\t\t struct repository *repo)\n+{\n+\tstruct object_array_entry *old_blob, *new_blob;\n+\tstruct rev_info revs;\n+\tint ret;\n+\n+\tconst char * const usage[] = {\n+\t\tN_(\"git diff-blob \"),\n+\t\tNULL\n+\t};\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN_OPT | PARSE_OPT_KEEP_ARGV0);\n+\n+\trepo_config(repo, git_diff_basic_config, NULL);\n+\tprepare_repo_settings(repo);\n+\trepo->settings.command_requires_full_index = 0;\n+\n+\trepo_init_revisions(repo, &revs, prefix);\n+\trevs.abbrev = 0;\n+\trevs.diff = 1;\n+\trevs.disable_stdin = 1;\n+\n+\tprefix = precompose_argv_prefix(argc, argv, prefix);\n+\targc = setup_revisions(argc, argv, &revs, NULL);\n+\n+\tif (!revs.diffopt.output_format)\n+\t\trevs.diffopt.output_format = DIFF_FORMAT_PATCH;\n+\n+\tswitch (revs.pending.nr) {\n+\tcase 2:\n+\t\told_blob = &revs.pending.objects[0];\n+\t\tnew_blob = &revs.pending.objects[1];\n+\n+\t\tif (old_blob->item->type != OBJ_BLOB)\n+\t\t\tdie(\"object %s is not a blob\", old_blob->name);\n+\n+\t\tif (new_blob->item->type != OBJ_BLOB)\n+\t\t\tdie(\"object %s is not a blob\", new_blob->name);\n+\n+\t\tdiff_blobs(old_blob, new_blob, &revs.diffopt);\n+\n+\t\tbreak;\n+\tdefault:\n+\t\tusage_with_options(usage, options);\n+\t}\n+\n+\tret = diff_result_code(&revs);\n+\trelease_revisions(&revs);\n+\n+\treturn ret;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex e0bb87b3b5..78d8308352 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -93,6 +93,7 @@ git-daemon synchingrepositories\n git-describe mainporcelain\n git-diagnose ancillaryinterrogators\n git-diff mainporcelain info\n+git-diff-blob plumbinginterrogators\n git-diff-files plumbinginterrogators\n git-diff-index plumbinginterrogators\n git-diff-tree plumbinginterrogators\ndiff --git a/git.c b/git.c\nindex 46b3c740c5..17c018ea36 100644\n--- a/git.c\n+++ b/git.c\n@@ -540,6 +540,7 @@ static struct cmd_struct commands[] = {\n \t{ \"describe\", cmd_describe, RUN_SETUP },\n \t{ \"diagnose\", cmd_diagnose, RUN_SETUP_GENTLY },\n \t{ \"diff\", cmd_diff, NO_PARSEOPT },\n+\t{ \"diff-blob\", cmd_diff_blob, RUN_SETUP | NO_PARSEOPT },\n \t{ \"diff-files\", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n \t{ \"diff-index\", cmd_diff_index, RUN_SETUP | NO_PARSEOPT },\n \t{ \"diff-tree\", cmd_diff_tree, RUN_SETUP | NO_PARSEOPT },\ndiff --git a/meson.build b/meson.build\nindex 0dccebcdf1..fefb802c27 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -503,6 +503,7 @@ builtin_sources = [\n 'builtin/credential.c',\n 'builtin/describe.c',\n 'builtin/diagnose.c',\n+ 'builtin/diff-blob.c',\n 'builtin/diff-files.c',\n 'builtin/diff-index.c',\n 'builtin/diff-tree.c',\ndiff --git a/t/t4063-diff-blobs.sh b/t/t4063-diff-blobs.sh\nindex 50fdb5ea52..23615565fe 100755\n--- a/t/t4063-diff-blobs.sh\n+++ b/t/t4063-diff-blobs.sh\n@@ -1,12 +1,14 @@\n #!/bin/sh\n \n-test_description='test direct comparison of blobs via git-diff'\n+test_description='test direct comparison of blobs via git-diff and git-diff-blob'\n \n . ./test-lib.sh\n \n+commands=\"diff diff-blob\"\n+\n run_diff () {\n \t# use full-index to make it easy to match the index line\n-\tgit diff --full-index \"$@\" >diff\n+\tgit $1 --full-index $2 $3 >diff\n }\n \n check_index () {\n@@ -37,61 +39,63 @@ test_expect_success 'create some blobs' '\n \tsha1_two=$(git rev-parse HEAD:two)\n '\n \n-test_expect_success 'diff by sha1' '\n-\trun_diff $sha1_one $sha1_two\n-'\n-test_expect_success 'index of sha1 diff' '\n-\tcheck_index $sha1_one $sha1_two\n-'\n-test_expect_success 'sha1 diff uses arguments as paths' '\n-\tcheck_paths $sha1_one $sha1_two\n-'\n-test_expect_success 'sha1 diff has no mode change' '\n-\t! grep mode diff\n-'\n-\n-test_expect_success 'diff by tree:path (run)' '\n-\trun_diff HEAD:one HEAD:two\n-'\n-test_expect_success 'index of tree:path diff' '\n-\tcheck_index $sha1_one $sha1_two\n-'\n-test_expect_success 'tree:path diff uses filenames as paths' '\n-\tcheck_paths one two\n-'\n-test_expect_success 'tree:path diff shows mode change' '\n-\tcheck_mode 100644 100755\n-'\n-\n-test_expect_success 'diff by ranged tree:path' '\n-\trun_diff HEAD:one..HEAD:two\n-'\n-test_expect_success 'index of ranged tree:path diff' '\n-\tcheck_index $sha1_one $sha1_two\n-'\n-test_expect_success 'ranged tree:path diff uses filenames as paths' '\n-\tcheck_paths one two\n-'\n-test_expect_success 'ranged tree:path diff shows mode change' '\n-\tcheck_mode 100644 100755\n-'\n-\n-test_expect_success 'diff blob against file' '\n-\trun_diff HEAD:one two\n+test_expect_success 'diff blob against file (git-diff)' '\n+\trun_diff diff HEAD:one two\n '\n-test_expect_success 'index of blob-file diff' '\n+test_expect_success 'index of blob-file diff (git-diff)' '\n \tcheck_index $sha1_one $sha1_two\n '\n-test_expect_success 'blob-file diff uses filename as paths' '\n+test_expect_success 'blob-file diff uses filename as paths (git-diff)' '\n \tcheck_paths one two\n '\n-test_expect_success FILEMODE 'blob-file diff shows mode change' '\n+test_expect_success FILEMODE 'blob-file diff shows mode change (git-diff)' '\n \tcheck_mode 100644 100755\n '\n \n-test_expect_success 'blob-file diff prefers filename to sha1' '\n-\trun_diff $sha1_one two &&\n+test_expect_success 'blob-file diff prefers filename to sha1 (git-diff)' '\n+\trun_diff diff $sha1_one two &&\n \tcheck_paths two two\n '\n \n+for cmd in $commands; do\n+\ttest_expect_success \"diff by sha1 (git-$cmd)\" '\n+\t\trun_diff $cmd $sha1_one $sha1_two\n+\t'\n+\ttest_expect_success \"index of sha1 diff (git-$cmd)\" '\n+\t\tcheck_index $sha1_one $sha1_two\n+\t'\n+\ttest_expect_success \"sha1 diff uses arguments as paths (git-$cmd)\" '\n+\t\tcheck_paths $sha1_one $sha1_two\n+\t'\n+\ttest_expect_success \"sha1 diff has no mode change (git-$cmd)\" '\n+\t\t! grep mode diff\n+\t'\n+\n+\ttest_expect_success \"diff by tree:path (run) (git-$cmd)\" '\n+\t\trun_diff $cmd HEAD:one HEAD:two\n+\t'\n+\ttest_expect_success \"index of tree:path diff (git-$cmd)\" '\n+\t\tcheck_index $sha1_one $sha1_two\n+\t'\n+\ttest_expect_success \"tree:path diff uses filenames as paths (git-$cmd)\" '\n+\t\tcheck_paths one two\n+\t'\n+\ttest_expect_success \"tree:path diff shows mode change (git-$cmd)\" '\n+\t\tcheck_mode 100644 100755\n+\t'\n+\n+\ttest_expect_success \"diff by ranged tree:path (git-$cmd)\" '\n+\t\trun_diff $cmd HEAD:one..HEAD:two\n+\t'\n+\ttest_expect_success \"index of ranged tree:path diff (git-$cmd)\" '\n+\t\tcheck_index $sha1_one $sha1_two\n+\t'\n+\ttest_expect_success \"ranged tree:path diff uses filenames as paths (git-$cmd)\" '\n+\t\tcheck_paths one two\n+\t'\n+\ttest_expect_success \"ranged tree:path diff shows mode change (git-$cmd)\" '\n+\t\tcheck_mode 100644 100755\n+\t'\n+done\n+\n test_done\n-- \n2.47.1\n\n"},{"id":"509044","messageId":"20241213042312.2890841-3-jltobler@gmail.com","threadId":"62633","inReplyTo":"20241213042312.2890841-1-jltobler@gmail.com","subject":"[PATCH 2/3] builtin/diff-blob: add \"--stdin\" option","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-12-13T04:23:11Z","receivedAt":"2024-12-13T04:26:20Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"There is not a way to generate multiple blob diffs from a single\nprocess. Similar to git-diff-tree(1) with its \"--stdin\" option, it would\nbe useful if multiple blob pairs could be provided to git-diff-blob(1)\nto compute blob diffs for.\n\nTeach git-diff-blob(1) the \"--stdin\" option to allow a pair of blobs to\nbe read from each line of stdin instead of relying on the single blob\npair provided as arguments. When this option is specified, each valid\nline of input computes a blob diff thus allowing multiple blob diffs in\na single process. A blob may be specified by its ID or a path-scoped\nrevision that resolve to a blob. When a path-scoped revision is used,\npath and mode information is also extracted and presented in the\nresulting diff header.\n\nSigned-off-by: Justin Tobler \n---\n Documentation/git-diff-blob.txt | 6 ++++\n builtin/diff-blob.c | 64 +++++++++++++++++++++++++++++++++\n t/t4063-diff-blobs.sh | 14 ++++++++\n 3 files changed, 84 insertions(+)\n\ndiff --git a/Documentation/git-diff-blob.txt b/Documentation/git-diff-blob.txt\nindex 732992d1d7..f6ecd522fa 100644\n--- a/Documentation/git-diff-blob.txt\n+++ b/Documentation/git-diff-blob.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git diff-blob' \n+'git diff-blob' --stdin\n \n DESCRIPTION\n -----------\n@@ -20,6 +21,11 @@ OPTIONS\n ::\n \tThe id of a blob object or path-scoped revision that resolves to a blob.\n \n+--stdin::\n+\tWhen `--stdin` is specified, the command does not take arguments\n+\tfrom the command line. Instead, it reads lines containing two \n+\tfrom its standard input. (Use a single space as separator.)\n+\n include::pretty-formats.txt[]\n \n include::diff-format.txt[]\ndiff --git a/builtin/diff-blob.c b/builtin/diff-blob.c\nindex 7cfa4eb436..45edfdd979 100644\n--- a/builtin/diff-blob.c\n+++ b/builtin/diff-blob.c\n@@ -4,9 +4,12 @@\n #include \"diffcore.h\"\n #include \"gettext.h\"\n #include \"hash.h\"\n+#include \"object-name.h\"\n #include \"object.h\"\n #include \"parse-options.h\"\n #include \"revision.h\"\n+#include \"strbuf.h\"\n+#include \"string-list.h\"\n \n static void diff_blobs(struct object_array_entry *old_blob,\n \t\t struct object_array_entry *new_blob,\n@@ -59,18 +62,66 @@ static void diff_blobs(struct object_array_entry *old_blob,\n \tdiff_flush(opts);\n }\n \n+static void parse_blob_stdin(struct object_array *blob_pair,\n+\t\t\t struct repository *repo, const char *name)\n+{\n+\tint flags = GET_OID_BLOB | GET_OID_RECORD_PATH;\n+\tstruct object_context oc;\n+\tstruct object_id oid;\n+\tstruct object *obj;\n+\n+\tif (get_oid_with_context(repo, name, flags, &oid, &oc))\n+\t\tdie(\"invalid object %s given\", name);\n+\n+\tobj = parse_object_or_die(&oid, name);\n+\tif (obj->type != OBJ_BLOB)\n+\t\tdie(\"object %s is not a blob\", name);\n+\n+\tadd_object_array_with_path(obj, name, blob_pair, oc.mode, oc.path);\n+\tobject_context_release(&oc);\n+}\n+\n+static void diff_blob_stdin(struct repository *repo, struct diff_options *opts)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct string_list_item *item;\n+\n+\twhile (strbuf_getline(&sb, stdin) != EOF) {\n+\t\tstruct object_array blob_pair = OBJECT_ARRAY_INIT;\n+\t\tstruct string_list list = STRING_LIST_INIT_NODUP;\n+\n+\t\tif (string_list_split_in_place(&list, sb.buf, \" \", -1) != 2)\n+\t\t\tdie(\"two blobs not provided\");\n+\n+\t\tfor_each_string_list_item(item, &list) {\n+\t\t\tparse_blob_stdin(&blob_pair, repo, item->string);\n+\t\t}\n+\n+\t\tdiff_blobs(&blob_pair.objects[0], &blob_pair.objects[1], opts);\n+\n+\t\tstring_list_clear(&list, 1);\n+\t\tobject_array_clear(&blob_pair);\n+\t}\n+\n+\tstrbuf_release(&sb);\n+}\n+\n int cmd_diff_blob(int argc, const char **argv, const char *prefix,\n \t\t struct repository *repo)\n {\n \tstruct object_array_entry *old_blob, *new_blob;\n \tstruct rev_info revs;\n+\tint read_stdin = 0;\n \tint ret;\n \n \tconst char * const usage[] = {\n \t\tN_(\"git diff-blob \"),\n+\t\tN_(\"git diff-blob --stdin\"),\n \t\tNULL\n \t};\n \tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"stdin\", &read_stdin,\n+\t\t\tN_(\"read blob pairs from stdin\")),\n \t\tOPT_END()\n \t};\n \n@@ -93,7 +144,20 @@ int cmd_diff_blob(int argc, const char **argv, const char *prefix,\n \t\trevs.diffopt.output_format = DIFF_FORMAT_PATCH;\n \n \tswitch (revs.pending.nr) {\n+\tcase 0:\n+\t\tif (!read_stdin)\n+\t\t\tusage_with_options(usage, options);\n+\n+\t\trevs.diffopt.no_free = 1;\n+\t\tdiff_blob_stdin(repo, &revs.diffopt);\n+\t\trevs.diffopt.no_free = 0;\n+\t\tdiff_free(&revs.diffopt);\n+\n+\t\tbreak;\n \tcase 2:\n+\t\tif (read_stdin)\n+\t\t\tusage_with_options(usage, options);\n+\n \t\told_blob = &revs.pending.objects[0];\n \t\tnew_blob = &revs.pending.objects[1];\n \ndiff --git a/t/t4063-diff-blobs.sh b/t/t4063-diff-blobs.sh\nindex 23615565fe..d7785d4a6e 100755\n--- a/t/t4063-diff-blobs.sh\n+++ b/t/t4063-diff-blobs.sh\n@@ -98,4 +98,18 @@ for cmd in $commands; do\n \t'\n done\n \n+test_expect_success 'diff-blob --stdin with blob ID' '\n+\techo $sha1_one $sha1_two | git diff-blob --full-index --stdin >diff &&\n+\tcheck_index $sha1_one $sha1_two &&\n+\tcheck_paths $sha1_one $sha1_two &&\n+\t! grep mode diff\n+'\n+\n+test_expect_success 'diff-blob --stdin with revision' '\n+\techo HEAD:one HEAD:two | git diff-blob --full-index --stdin >diff &&\n+\tcheck_index $sha1_one $sha1_two &&\n+\tcheck_paths one two &&\n+\tcheck_mode 100644 100755\n+'\n+\n test_done\n-- \n2.47.1\n\n"},{"id":"509045","messageId":"20241213042312.2890841-4-jltobler@gmail.com","threadId":"62633","inReplyTo":"20241213042312.2890841-1-jltobler@gmail.com","subject":"[PATCH 3/3] builtin/diff-blob: Add \"-z\" option","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-12-13T04:23:12Z","receivedAt":"2024-12-13T04:26:22Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"The \"--stdin\" option for git-diff-blob(1) reads two space separated\nblobs for each line of input. A blob may be specified by its ID or a\npath-scoped revision that resolves to a blob. It is possible for the\npath to contain whitespace or newline characters which must be escaped.\n\nTo make input more simple, teach git-diff-blob(1) the \"-z\" option which\nchanges the input delimiter for each blob to a NUL character. With this\noption, the command waits two NUL terminated blobs to read and then\ngenerates the diff. The diff output is also NUL terminated to help\ndifferentiate between outputted diffs.\n\nSigned-off-by: Justin Tobler \n---\n Documentation/git-diff-blob.txt | 6 +++++-\n builtin/diff-blob.c | 37 +++++++++++++++++++++++++--------\n 2 files changed, 33 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-diff-blob.txt b/Documentation/git-diff-blob.txt\nindex f6ecd522fa..36cd686bb1 100644\n--- a/Documentation/git-diff-blob.txt\n+++ b/Documentation/git-diff-blob.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git diff-blob' \n-'git diff-blob' --stdin\n+'git diff-blob' --stdin [-z]\n \n DESCRIPTION\n -----------\n@@ -26,6 +26,10 @@ OPTIONS\n \tfrom the command line. Instead, it reads lines containing two \n \tfrom its standard input. (Use a single space as separator.)\n \n+-z::\n+\tWhen `--stdin` has been given, use NUL characters to separate blob\n+\tinputs and diff outputs.\n+\n include::pretty-formats.txt[]\n \n include::diff-format.txt[]\ndiff --git a/builtin/diff-blob.c b/builtin/diff-blob.c\nindex 45edfdd979..60c92cec9c 100644\n--- a/builtin/diff-blob.c\n+++ b/builtin/diff-blob.c\n@@ -81,23 +81,39 @@ static void parse_blob_stdin(struct object_array *blob_pair,\n \tobject_context_release(&oc);\n }\n \n-static void diff_blob_stdin(struct repository *repo, struct diff_options *opts)\n+static void diff_blob_stdin(struct repository *repo, struct diff_options *opts,\n+\t\t\t int null_term)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct string_list_item *item;\n \n-\twhile (strbuf_getline(&sb, stdin) != EOF) {\n+\twhile (1) {\n \t\tstruct object_array blob_pair = OBJECT_ARRAY_INIT;\n \t\tstruct string_list list = STRING_LIST_INIT_NODUP;\n \n-\t\tif (string_list_split_in_place(&list, sb.buf, \" \", -1) != 2)\n-\t\t\tdie(\"two blobs not provided\");\n+\t\tif (null_term) {\n+\t\t\tif (strbuf_getline_nul(&sb, stdin) == EOF)\n+\t\t\t\tbreak;\n+\t\t\tparse_blob_stdin(&blob_pair, repo, sb.buf);\n \n-\t\tfor_each_string_list_item(item, &list) {\n-\t\t\tparse_blob_stdin(&blob_pair, repo, item->string);\n+\t\t\tif (strbuf_getline_nul(&sb, stdin) == EOF)\n+\t\t\t\tbreak;\n+\t\t\tparse_blob_stdin(&blob_pair, repo, sb.buf);\n+\t\t} else {\n+\t\t\tif (strbuf_getline(&sb, stdin) == EOF)\n+\t\t\t\tbreak;\n+\n+\t\t\tif (string_list_split_in_place(&list, sb.buf, \" \", -1) != 2)\n+\t\t\t\tdie(\"two blobs not provided\");\n+\n+\t\t\tfor_each_string_list_item(item, &list) {\n+\t\t\t\tparse_blob_stdin(&blob_pair, repo, item->string);\n+\t\t\t}\n \t\t}\n \n \t\tdiff_blobs(&blob_pair.objects[0], &blob_pair.objects[1], opts);\n+\t\tif (null_term)\n+\t\t\tprintf(\"%c\", '\\0');\n \n \t\tstring_list_clear(&list, 1);\n \t\tobject_array_clear(&blob_pair);\n@@ -112,16 +128,19 @@ int cmd_diff_blob(int argc, const char **argv, const char *prefix,\n \tstruct object_array_entry *old_blob, *new_blob;\n \tstruct rev_info revs;\n \tint read_stdin = 0;\n+\tint null_term = 0;\n \tint ret;\n \n \tconst char * const usage[] = {\n \t\tN_(\"git diff-blob \"),\n-\t\tN_(\"git diff-blob --stdin\"),\n+\t\tN_(\"git diff-blob --stdin [-z]\"),\n \t\tNULL\n \t};\n \tstruct option options[] = {\n \t\tOPT_BOOL(0, \"stdin\", &read_stdin,\n \t\t\tN_(\"read blob pairs from stdin\")),\n+\t\tOPT_BOOL('z', NULL, &null_term,\n+\t\t\tN_(\"inputed blobs and outputted diffs terminated with NUL\")),\n \t\tOPT_END()\n \t};\n \n@@ -149,13 +168,13 @@ int cmd_diff_blob(int argc, const char **argv, const char *prefix,\n \t\t\tusage_with_options(usage, options);\n \n \t\trevs.diffopt.no_free = 1;\n-\t\tdiff_blob_stdin(repo, &revs.diffopt);\n+\t\tdiff_blob_stdin(repo, &revs.diffopt, null_term);\n \t\trevs.diffopt.no_free = 0;\n \t\tdiff_free(&revs.diffopt);\n \n \t\tbreak;\n \tcase 2:\n-\t\tif (read_stdin)\n+\t\tif (read_stdin || null_term)\n \t\t\tusage_with_options(usage, options);\n \n \t\told_blob = &revs.pending.objects[0];\n-- \n2.47.1\n\n"},{"id":"509050","messageId":"20241213081211.GA1443203@coredump.intra.peff.net","threadId":"62633","inReplyTo":"20241213042312.2890841-1-jltobler@gmail.com","subject":"Re: [PATCH 0/3] batch blob diff generation","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-12-13T08:12:11Z","receivedAt":"2024-12-13T08:12:19Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 12, 2024 at 10:23:09PM -0600, Justin Tobler wrote:\n\n> To enable support for batch diffs of multiple blob pairs, this\n> series introduces a new diff plumbing command git-diff-blob(1). Similar\n> to git-diff-tree(1), it provides a \"--stdin\" option that reads a pair of\n> blobs on each line of input and generates the diffs. This is intended to\n> be used for scripting purposes where more fine-grained control for diff\n> generation is desired. Below is an example for each usage:\n> \n> $ git diff-blob HEAD~5000:README.md HEAD:README.md\n> \n> $ git diff-blob --stdin < 88f126184c52bfe4859ec189d018872902e02a84 665ce5f5a83647619fba9157fa9b0141ae8b228b\n> HEAD~5000:README.md HEAD:README.md\n> EOF\n\nIn the first example, I think just using \"git diff\" would work (though\nit is not a plumbing command). But the stdin example is what's\ninteresting here anyway, since it can handle arbitrary inputs. So let's\nfocus on that.\n\nFeeding just blob ids has a big drawback: we don't have any context! So\nyou get bogus filenames in the patch, no mode data, and so on.\n\nFeeding the paths along with their commits, as you do on the second\nline, gives you those things from the lookup context. But it also has\nsome problems. One, it's needlessly expensive; we have to traverse\nHEAD~5000, and then dig into its tree to find the blobs (which\npresumably you already did, since how else would you end up with those\noids). And two, there are parsing ambiguities, since arbitrary revision\nnames can contain spaces. E.g., are we looking for the file \"README.md\nHEAD:README.md\" in HEAD~5000?\n\nSo ideally we'd have an input format that encapsulates that extra\ncontext data and provides some mechanism for quoting. And it turns out\nwe do: the --raw diff format.\n\nIf the program takes that format, then you can manually feed it two\narbitrary blob oids if you have them (and put whatever you like for the\nmode/path context), like:\n\n git diff-blob --stdin <<\\EOF\n :100644 100644 88f126184c52bfe4859ec189d018872902e02a84 665ce5f5a83647619fba9157fa9b0141ae8b228b M\tREADME.md\n EOF\n\nOr you can get the real context yourself (though it seems to me that\nthis is a gap in what \"cat-file --batch\" should be able to do in a\nsingle process):\n\n git ls-tree HEAD~5000 README.md >out\n read mode_a blob oid_a path out\n read mode_b blob oid_b path writes:\n\n> Feeding just blob ids has a big drawback: we don't have any context! So\n> you get bogus filenames in the patch, no mode data, and so on.\n\nAnd the lack of filenames and the tree object name at the root level\nmeans you do not get anything out of the attribute subsystem, which\nin turn may affect a few more things.\n\nUnfortunately the format used in the output from \"diff --raw\" does\nnot capture this.\n\nDoes this want to be a building block for the server side diff?\nWould it be a bit too low level for each \"request\" to comparing only\ntwo blob objects? Can we place a lot more assumptions on the\nrelationship among blob pairs that appear in the --stdin request\n(e.g., they all come from the same tree and share the same attr\nstack to look up attributes applicable for them)?\n\n"},{"id":"509077","messageId":"20241213103850.GA1456983@coredump.intra.peff.net","threadId":"62633","inReplyTo":"xmqqv7vnevjc.fsf@gitster.g","subject":"Re: [PATCH 0/3] batch blob diff generation","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-12-13T10:38:50Z","receivedAt":"2024-12-13T10:38:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 13, 2024 at 07:17:59PM +0900, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > Feeding just blob ids has a big drawback: we don't have any context! So\n> > you get bogus filenames in the patch, no mode data, and so on.\n> \n> And the lack of filenames and the tree object name at the root level\n> means you do not get anything out of the attribute subsystem, which\n> in turn may affect a few more things.\n> \n> Unfortunately the format used in the output from \"diff --raw\" does\n> not capture this.\n\nDon't we just use the working tree .gitattributes by default, and ignore\nwhat's in the endpoints? In a bare repo we wouldn't have that, but I\nthink the recently added attr.tree and --attr-source options would work\nthere.\n\nYou can't use attributes from multiple trees in a single request, but I\ndoubt that would be a big drawback.\n\n-Peff\n"},{"id":"509109","messageId":"7q5dse6hbvuqwlhfceqwockouyfvvuxaekegahgelnri6nkv7h@2ph2qdfaktx2","threadId":"62633","inReplyTo":"20241213081211.GA1443203@coredump.intra.peff.net","subject":"Re: [PATCH 0/3] batch blob diff generation","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-12-13T16:41:25Z","receivedAt":"2024-12-13T16:43:38Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/12/13 03:12AM, Jeff King wrote:\n\n> In the first example, I think just using \"git diff\" would work (though\n> it is not a plumbing command). But the stdin example is what's\n> interesting here anyway, since it can handle arbitrary inputs. So let's\n> focus on that.\n> \n> Feeding just blob ids has a big drawback: we don't have any context! So\n> you get bogus filenames in the patch, no mode data, and so on.\n> \n> Feeding the paths along with their commits, as you do on the second\n> line, gives you those things from the lookup context. But it also has\n> some problems. One, it's needlessly expensive; we have to traverse\n> HEAD~5000, and then dig into its tree to find the blobs (which\n> presumably you already did, since how else would you end up with those\n> oids). And two, there are parsing ambiguities, since arbitrary revision\n> names can contain spaces. E.g., are we looking for the file \"README.md\n> HEAD:README.md\" in HEAD~5000?\n> \n> So ideally we'd have an input format that encapsulates that extra\n> context data and provides some mechanism for quoting. And it turns out\n> we do: the --raw diff format.\n\nI had not considered using the raw diff format as the input source. As\nyou pointed out, using blob IDs alone loses some of the useful context.\nBy using path-scoped revisions, we can still get this context, but at an\nadded cost of having to traverse the tree to get the underlying\ninformation.\n\nAs you also mentioned, this is potentially wasteful if, for example, the\nblobs diffs you are trying to generate are a subset of git-diff-tree(1)\noutput and thus the context is already known ahead of time. Which is\nexactly what we are hoping to accomplish.\n\n> If the program takes that format, then you can manually feed it two\n> arbitrary blob oids if you have them (and put whatever you like for the\n> mode/path context), like:\n> \n> git diff-blob --stdin <<\\EOF\n> :100644 100644 88f126184c52bfe4859ec189d018872902e02a84 665ce5f5a83647619fba9157fa9b0141ae8b228b M\tREADME.md\n> EOF\n> \n> Or you can get the real context yourself (though it seems to me that\n> this is a gap in what \"cat-file --batch\" should be able to do in a\n> single process):\n> \n> git ls-tree HEAD~5000 README.md >out\n> read mode_a blob oid_a path git ls-tree HEAD README.md >out\n> read mode_b blob oid_b path printf \":$mode_a $mode_b $oid_a $oid_b M\\tREADME.md\" |\n> git diff-blob --stdin\n> \n> But it also means you can use --raw output directly. So:\n> \n> git diff-tree --raw -r HEAD~5000 HEAD -- README.md |\n> git diff-blob --stdin\n> \n> Now that command by itself doesn't look all that useful; you could have\n> just asked for patches from diff-tree. But by splitting the two, you can\n> filter the set of paths in between (for example, to omit some entries,\n> or to batch a large diff into more manageable chunks for pagination,\n> etc).\n\nYup, this is exactly what I'm hoping to achieve! As a single commit may\ncontain an unbounded number changes, being able to control diff\ngeneration at the blob level is quite useful. Using the raw diff format\nas input looks like a rather elegant solution and I think it makes\nsense to use it here for the \"--stdin\" option over just reading two\nblobs.\n\n> The patch might look something like this:\n> \n> https://lore.kernel.org/git/20161201204042.6yslbyrg7l6ghhww@sigill.intra.peff.net/\n> \n> :) That is what has been powering the diffs at github.com since 2016 or\n> so. And continues to do so, as far as I know. I don't have access to\n> their internal repository anymore, but I've continued to rebase the\n> topic forward in my personal repo. You can fetch it from:\n> \n> https://github.com/peff/git jk/diff-pairs\n> \n> in case that is helpful.\n\nThanks Peff! From looking at the mentioned thread and branch, it looks\nlike I'm basically trying to accomplish the same thing here. Just a bit\nlate to the conversation. :)\n\nWhile the use-case is rather narrow, I think it would be nice to see\nthis functionality provided upstream. I see this as a means to faciliate\nmore fine-grained control of the blob diffs we actually want to compute\nat a given time and it seems like it would be reasonable to expose as\npart of the diff plumbing. I would certainly be interested in adapting\nthis series to instead use raw input from git-diff-tree(1) or trying to\nrevive the previous series if that is preferred. \n\nIf there is interest in continuing, some lingering questions I have:\n\nBeing that the primary purpose of git-diff-blob(1) here is to handle\ngenerating blob diffs as specified by stdin, is there any reason to have\na normal mode that accepts a blob pair as arguments? Or would it be best\nto limit the input mechanism to stdin entirely? If the user wanted to\ncompute a single blob diff they could just use git-diff(1) already so\nproviding this as a part of git-diff-blob(1) is a bit redundant. Having\nit as an option for the user does seem a bit more friendly though.\n\n-Justin\n"},{"id":"509116","messageId":"xmqqldwj9pq0.fsf@gitster.g","threadId":"62633","inReplyTo":"20241213081211.GA1443203@coredump.intra.peff.net","subject":"Re: [PATCH 0/3] batch blob diff generation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-13T22:34:47Z","receivedAt":"2024-12-13T22:34:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> So ideally we'd have an input format that encapsulates that extra\n> context data and provides some mechanism for quoting. And it turns out\n> we do: the --raw diff format.\n\nFunny. The raw diff format indeed was designed as an interchange\nformat from various \"compare two sets of things\" front-ends (like\ndiff-files, diff-cache, and diff-tree) that emits the raw format, to\nbe read by \"diff-helper\" (initially called \"diff-tree-helper\") that\ntakes the raw format and\n\n - matches removed and added paths with similar contents to detect\n renames and copies\n\n - computes the output in various formats including \"patch\".\n\nSo I guess we came a full circle, finally ;-). Looking in the archive\nfor messages exchanged between junkio@ and torvalds@ mentioning diff\nbefore 2005-05-30 finds some interesting gems.\n\nhttps://lore.kernel.org/git/7v1x8zsamn.fsf_-_@assigned-by-dhcp.cox.net/\n"},{"id":"509119","messageId":"xmqqcyhtaecy.fsf@gitster.g","threadId":"62633","inReplyTo":"20241213103850.GA1456983@coredump.intra.peff.net","subject":"Re: [PATCH 0/3] batch blob diff generation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-15T02:07:09Z","receivedAt":"2024-12-15T02:07:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> Unfortunately the format used in the output from \"diff --raw\" does\n>> not capture this.\n>\n> Don't we just use the working tree .gitattributes by default, and ignore\n> what's in the endpoints? In a bare repo we wouldn't have that, but I\n> think the recently added attr.tree and --attr-source options would work\n> there.\n\nYeah, you're right. I forgot about attr.tree (does not seem to be\ndocumented, by the way) and --attr-source & GIT_ATTR_SOURCE. I\nimagine that this feature is primarily meant to be used on the\nserver side, and in bare repositories, only \"diff-tree\" makes sense\namong the diff-* family of commands, which (as server environments\nlack \"the index\" nor \"the working tree\") would already be using\nthese mechanisms, so there is no new issues introduced here.\n\n> You can't use attributes from multiple trees in a single request, but I\n> doubt that would be a big drawback.\n\nI think it is also true with the normal diff-tree and friends; I do\nnot think it looks up attributes from each tree independently when\nyou run \"git diff-tree -r A B\" to compare the blob in tree A that is\nCRLF with the blob at the same path in tree B. So we should be OK.\n\nThanks.\n\n"},{"id":"509120","messageId":"xmqq5xnladwi.fsf@gitster.g","threadId":"62633","inReplyTo":"xmqqcyhtaecy.fsf@gitster.g","subject":"Re: [PATCH 0/3] batch blob diff generation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-15T02:17:01Z","receivedAt":"2024-12-15T02:17:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Yeah, you're right. I forgot about attr.tree (does not seem to be\n> documented, by the way)\n\nWe do have an entry in Documentation/config/attr.txt that describes\nthe three; I simply assumed it is not documented as I didn't see it\nmentioned in Documentation/git.txt where --attr-source &\nGIT_ATTR_SOURCE are described.\n\nWe may want to add something like this, perhaps?\n\n----- >8 -----\nSubject: doc: give attr.tree a bit more visibility\n\nIn \"git help config\" output, attr.tree mentions both --attr-source\nand GIT_ATTR_SOURCE, but the description of --attr-source and\nGIT_ATTR_SOURCE that appear in \"git help git\", attr.tree is missing.\n\nAdd it so that these three are described together in both places.\n\nSigned-off-by: Junio C Hamano \n---\n Documentation/git.txt | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git c/Documentation/git.txt w/Documentation/git.txt\nindex d15a869762..13f6785408 100644\n--- c/Documentation/git.txt\n+++ w/Documentation/git.txt\n@@ -228,7 +228,10 @@ If you just want to run git as if it was started in `` then use\n --attr-source=::\n \tRead gitattributes from instead of the worktree. See\n \tlinkgit:gitattributes[5]. This is equivalent to setting the\n-\t`GIT_ATTR_SOURCE` environment variable.\n+\t`GIT_ATTR_SOURCE` environment variable. The `attr.tree`\n+\tconfiguration variable is used as a fallback when this option\n+\tor the environment variable are not in use.\n+\n \n GIT COMMANDS\n ------------\n"},{"id":"509138","messageId":"xmqqa5cw8r8k.fsf@gitster.g","threadId":"62633","inReplyTo":"xmqqldwj9pq0.fsf@gitster.g","subject":"Re: [PATCH 0/3] batch blob diff generation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-15T23:24:11Z","receivedAt":"2024-12-15T23:24:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Jeff King writes:\n>\n>> So ideally we'd have an input format that encapsulates that extra\n>> context data and provides some mechanism for quoting. And it turns out\n>> we do: the --raw diff format.\n>\n> Funny. The raw diff format indeed was designed as an interchange\n> format from various \"compare two sets of things\" front-ends (like\n> diff-files, diff-cache, and diff-tree) that emits the raw format, to\n> be read by \"diff-helper\" (initially called \"diff-tree-helper\") that\n> takes the raw format and\n>\n> - matches removed and added paths with similar contents to detect\n> renames and copies\n>\n> - computes the output in various formats including \"patch\".\n>\n> So I guess we came a full circle, finally ;-). Looking in the archive\n> for messages exchanged between junkio@ and torvalds@ mentioning diff\n> before 2005-05-30 finds some interesting gems.\n>\n> https://lore.kernel.org/git/7v1x8zsamn.fsf_-_@assigned-by-dhcp.cox.net/\n\nSo, if we were to do what Justin tried to do honoring the overall\ndesign of our diff machinery, I think what we can do is as follows:\n\n * Use the \"diff --raw\" output format as the input, but with a bit\n of twist.\n\n (1) a narrow special case that takes only a single diff_filepair\n of and blobs, and immediately run diff_queue() on\n that single diff_filepair, which is Justin's use case. For\n this mode of operation, \"flush after reach record of input\"\n may be sufficient.\n\n (2) as a general \"interchange format\" to feed \"comparison between\n two sets of \" into our diff machinery, we are\n better off if we can treat the input stream as multiple\n records that describes comparison between two sets. Imagine\n \"git log --oneline --first-parent -2 --raw HEAD\", where one\n set of \"diff --raw\" records show the changed blobs with their\n paths between HEAD~1 and HEAD, and another set does so for\n HEAD~2 and HEAD~1. We need to be able to tell where the\n first set ends and the second set starts, so that rename\n detection and other things, if requested, can be done within\n each set.\n\n My recommendation is to use a single blank line as a separator,\n e.g.\n\n :100644 100644 ce31f93061 9829984b0a M\tDocumentation/git-refs.txt\n :100644 100644 8b3882cff1 4a74f7c7bd M\trefs.c\n :100755 100755 1bfff3a7af f59bc4860f M\tt/t1460-refs-migrate.sh\n\n :100644 100644 c11213f520 8953d1c6d3 M\trefs/files-backend.c\n :100644 100644 b2e3ba877d bec5962deb M\trefs/reftable-backend.c\n\n so an application that wants to compare only one diff_filepair\n at a time would issue something like\n\n :100644 100644 ce31f93061 9829984b0a M\tDocumentation/git-refs.txt\n\n :100644 100644 8b3882cff1 4a74f7c7bd M\trefs.c\n\n :100755 100755 1bfff3a7af f59bc4860f M\tt/t1460-refs-migrate.sh\n\n so the parsing machinery does not have to worry about case (1) above.\n\n * Parse and append the input into diff_queue(), until you see an\n blank line.\n\n - If at EOF you are done, but if you have something accumulated\n in diff_queue(), show them (like below) first. In any case, at\n EOF, you are done.\n\n * Run diffcore_std() followed by diff_flush() to have the contents\n of the queue nicely formatted and emptied. Go back to parsing\n more input lines.\n\n"},{"id":"509150","messageId":"20241216111112.GA2201417@coredump.intra.peff.net","threadId":"62633","inReplyTo":"xmqq5xnladwi.fsf@gitster.g","subject":"Re: [PATCH 0/3] batch blob diff generation","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-12-16T11:11:12Z","receivedAt":"2024-12-16T11:11:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Dec 14, 2024 at 06:17:01PM -0800, Junio C Hamano wrote:\n\n> Junio C Hamano writes:\n> \n> > Yeah, you're right. I forgot about attr.tree (does not seem to be\n> > documented, by the way)\n> \n> We do have an entry in Documentation/config/attr.txt that describes\n> the three; I simply assumed it is not documented as I didn't see it\n> mentioned in Documentation/git.txt where --attr-source &\n> GIT_ATTR_SOURCE are described.\n> \n> We may want to add something like this, perhaps?\n> \n> ----- >8 -----\n> Subject: doc: give attr.tree a bit more visibility\n\nYeah, that looks good to me.\n\nI recall that the performance of attr.tree is not great for _some_\ncommands (like pack-objects). So it's perhaps reasonable to use for\nsingle commands like \"git diff\" but not to set in your on-disk config.\nIt's possible we'd want to warn people about that before advertising it\nmore widely? I dunno.\n\n-Peff\n"},{"id":"509151","messageId":"20241216111831.GB2201417@coredump.intra.peff.net","threadId":"62633","inReplyTo":"7q5dse6hbvuqwlhfceqwockouyfvvuxaekegahgelnri6nkv7h@2ph2qdfaktx2","subject":"Re: [PATCH 0/3] batch blob diff generation","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-12-16T11:18:31Z","receivedAt":"2024-12-16T11:18:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 13, 2024 at 10:41:25AM -0600, Justin Tobler wrote:\n\n> > The patch might look something like this:\n> > \n> > https://lore.kernel.org/git/20161201204042.6yslbyrg7l6ghhww@sigill.intra.peff.net/\n> > \n> > :) That is what has been powering the diffs at github.com since 2016 or\n> > so. And continues to do so, as far as I know. I don't have access to\n> > their internal repository anymore, but I've continued to rebase the\n> > topic forward in my personal repo. You can fetch it from:\n> > \n> > https://github.com/peff/git jk/diff-pairs\n> > \n> > in case that is helpful.\n> \n> Thanks Peff! From looking at the mentioned thread and branch, it looks\n> like I'm basically trying to accomplish the same thing here. Just a bit\n> late to the conversation. :)\n> \n> While the use-case is rather narrow, I think it would be nice to see\n> this functionality provided upstream. I see this as a means to faciliate\n> more fine-grained control of the blob diffs we actually want to compute\n> at a given time and it seems like it would be reasonable to expose as\n> part of the diff plumbing. I would certainly be interested in adapting\n> this series to instead use raw input from git-diff-tree(1) or trying to\n> revive the previous series if that is preferred.\n\nYeah, if you want to take it in that direction, either by adapting the\nidea, or by starting with diff-pairs and polishing it up, I'm happy\neither way. GitHub folks may be happy if you keep the name \"diff-pairs\"\nand match the interface. ;)\n\n> If there is interest in continuing, some lingering questions I have:\n> \n> Being that the primary purpose of git-diff-blob(1) here is to handle\n> generating blob diffs as specified by stdin, is there any reason to have\n> a normal mode that accepts a blob pair as arguments? Or would it be best\n> to limit the input mechanism to stdin entirely? If the user wanted to\n> compute a single blob diff they could just use git-diff(1) already so\n> providing this as a part of git-diff-blob(1) is a bit redundant. Having\n> it as an option for the user does seem a bit more friendly though.\n\nI don't have a strong opinion. I agree it _could_ be more friendly, but\nit also raises questions about how that mode/path context is filled in.\nAnd also about other modes. E.g., \"git diff HEAD:foo bar\" will diff\nbetween a blob and a working tree. Should a new plumbing command support\nthat, too?\n\nIf those aren't things you immediately care about, I'd probably punt on\nit for now. I think it could be added later without losing compatibility\n(command-line arguments as appropriate for a single pair, and since the\nstdin format starts with \":mode\" there's room to extend it).\n\n-Peff\n"},{"id":"509152","messageId":"20241216113042.GC2201417@coredump.intra.peff.net","threadId":"62633","inReplyTo":"xmqqa5cw8r8k.fsf@gitster.g","subject":"Re: [PATCH 0/3] batch blob diff generation","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-12-16T11:30:42Z","receivedAt":"2024-12-16T11:30:44Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Dec 15, 2024 at 03:24:11PM -0800, Junio C Hamano wrote:\n\n> > Funny. The raw diff format indeed was designed as an interchange\n> > format from various \"compare two sets of things\" front-ends (like\n> > diff-files, diff-cache, and diff-tree) that emits the raw format, to\n> > be read by \"diff-helper\" (initially called \"diff-tree-helper\") that\n> > takes the raw format and\n> >\n> > - matches removed and added paths with similar contents to detect\n> > renames and copies\n> >\n> > - computes the output in various formats including \"patch\".\n> >\n> > So I guess we came a full circle, finally ;-). Looking in the archive\n> > for messages exchanged between junkio@ and torvalds@ mentioning diff\n> > before 2005-05-30 finds some interesting gems.\n> >\n> > https://lore.kernel.org/git/7v1x8zsamn.fsf_-_@assigned-by-dhcp.cox.net/\n\n:) That same thread was linked when I posted the original diff-pairs\nmany years ago.\n\n> So, if we were to do what Justin tried to do honoring the overall\n> design of our diff machinery, I think what we can do is as follows:\n> \n> * Use the \"diff --raw\" output format as the input, but with a bit\n> of twist.\n> \n> (1) a narrow special case that takes only a single diff_filepair\n> of and blobs, and immediately run diff_queue() on\n> that single diff_filepair, which is Justin's use case. For\n> this mode of operation, \"flush after reach record of input\"\n> may be sufficient.\n\nMy understanding was that he does not actually care about this case\n(just feeding two blobs), but is actually processing --raw output in the\nfirst place. Or did you just mean that we'd still be feeding raw output,\nbut just getting the flush behavior?\n\n> (2) as a general \"interchange format\" to feed \"comparison between\n> two sets of \" into our diff machinery, we are\n> better off if we can treat the input stream as multiple\n> records that describes comparison between two sets. Imagine\n> \"git log --oneline --first-parent -2 --raw HEAD\", where one\n> set of \"diff --raw\" records show the changed blobs with their\n> paths between HEAD~1 and HEAD, and another set does so for\n> HEAD~2 and HEAD~1. We need to be able to tell where the\n> first set ends and the second set starts, so that rename\n> detection and other things, if requested, can be done within\n> each set.\n\nSeems reasonable. For the use of diff-pairs at GitHub, I always just did\nfull-tree things like rename detection in the initial diff-tree\ninvocation. Since my goal was splitting/filtering, doing it after would\nyield wrong answers (since diff-pairs never sees the complete set).\n\nBut it's possible for somebody to want to filter the intermediate\nresults, then do full-tree stuff on the result (or even just delay the\ncost of rename detection). And certainly it's possible to want to feed a\nwhole bunch of unrelated diff segments without having to spawn a process\nfor each.\n\nSo it's not something I wanted, but I agree it's good to plan for.\n\n> My recommendation is to use a single blank line as a separator,\n> e.g.\n> \n> :100644 100644 ce31f93061 9829984b0a M\tDocumentation/git-refs.txt\n> :100644 100644 8b3882cff1 4a74f7c7bd M\trefs.c\n> :100755 100755 1bfff3a7af f59bc4860f M\tt/t1460-refs-migrate.sh\n> \n> :100644 100644 c11213f520 8953d1c6d3 M\trefs/files-backend.c\n> :100644 100644 b2e3ba877d bec5962deb M\trefs/reftable-backend.c\n> \n> so an application that wants to compare only one diff_filepair\n> at a time would issue something like\n> \n> :100644 100644 ce31f93061 9829984b0a M\tDocumentation/git-refs.txt\n> \n> :100644 100644 8b3882cff1 4a74f7c7bd M\trefs.c\n> \n> :100755 100755 1bfff3a7af f59bc4860f M\tt/t1460-refs-migrate.sh\n> \n> so the parsing machinery does not have to worry about case (1) above.\n\nYeah, that seems good. And it is backwards-compatible with the existing\ndiff-pairs format (which just barfs on a blank line). That's not a\nbig concern for the project, but it is nice that it makes life a bit\nsimpler for folks who would eventually be tasked with switching from it\nto this new tool. ;)\n\n> * Parse and append the input into diff_queue(), until you see an\n> blank line.\n> \n> - If at EOF you are done, but if you have something accumulated\n> in diff_queue(), show them (like below) first. In any case, at\n> EOF, you are done.\n\nYep, makes sense.\n\n> * Run diffcore_std() followed by diff_flush() to have the contents\n> of the queue nicely formatted and emptied. Go back to parsing\n> more input lines.\n\nThat makes sense. I don't think my diff-pairs runs diffcore_std() at\nall. The plumbing defaults mean it would always be a noop unless you\nexplicitly passed in rename, etc, options, and I never wanted to do\nthat.\n\nWe might have to check the interaction of diffcore code on a set of\nqueued diffs that already have values for renames, etc. I.e., that:\n\n git diff-tree --raw -M |\n git diff-pairs -M\n\ndoes not barf, since the input step in diff-pairs is going to set status\nto 'R', etc, in the pairs.\n\n-Peff\n"},{"id":"509170","messageId":"xmqq5xnj7fre.fsf@gitster.g","threadId":"62633","inReplyTo":"20241216111112.GA2201417@coredump.intra.peff.net","subject":"Re: [PATCH 0/3] batch blob diff generation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-16T16:29:41Z","receivedAt":"2024-12-16T16:29:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> I recall that the performance of attr.tree is not great for _some_\n> commands (like pack-objects). So it's perhaps reasonable to use for\n> single commands like \"git diff\" but not to set in your on-disk config.\n> It's possible we'd want to warn people about that before advertising it\n> more widely? I dunno.\n\nOr we disable the unusably-inefficient feature before doing so.\nWould attr.tree be much less efficient than GIT_ATTR_SOURCE?\n\n"},{"id":"509285","messageId":"20241218113927.GB594795@coredump.intra.peff.net","threadId":"62633","inReplyTo":"xmqq5xnj7fre.fsf@gitster.g","subject":"Re: [PATCH 0/3] batch blob diff generation","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-12-18T11:39:27Z","receivedAt":"2024-12-18T11:39:29Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Dec 16, 2024 at 08:29:41AM -0800, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > I recall that the performance of attr.tree is not great for _some_\n> > commands (like pack-objects). So it's perhaps reasonable to use for\n> > single commands like \"git diff\" but not to set in your on-disk config.\n> > It's possible we'd want to warn people about that before advertising it\n> > more widely? I dunno.\n> \n> Or we disable the unusably-inefficient feature before doing so.\n> Would attr.tree be much less efficient than GIT_ATTR_SOURCE?\n\nWhether it's unusably inefficient depends on what you throw at it. IIRC,\nthe performance difference for pack-objects on git.git was fairly\nnegligible. The problem in linux.git is that besides being big, it has a\ndeep(er) directory structure. So collecting all of the attributes for a\nfile like drivers/gpu/drm/foo/bar.h needs to open all of those\nintermediate trees.\n\nSo I'd be inclined to leave it in place, in case somebody is actually\nhappily using it.\n\nGIT_ATTR_SOURCE suffers all of the same problems; it's just that you'd\npresumably only use it with a few select commands (as far as I know,\npack-objects is the worst case because it's looking up one attribute on\nevery single blob in all of history).\n\n-Peff\n"},{"id":"509297","messageId":"xmqqv7vhujo4.fsf@gitster.g","threadId":"62633","inReplyTo":"20241218113927.GB594795@coredump.intra.peff.net","subject":"Re: [PATCH 0/3] batch blob diff generation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-18T14:53:31Z","receivedAt":"2024-12-18T14:53:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> > I recall that the performance of attr.tree is not great for _some_\n>> > commands (like pack-objects). So it's perhaps reasonable to use for\n>> > single commands like \"git diff\" but not to set in your on-disk config.\n>> > It's possible we'd want to warn people about that before advertising it\n>> > more widely? I dunno.\n>> \n>> Or we disable the unusably-inefficient feature before doing so.\n>> Would attr.tree be much less efficient than GIT_ATTR_SOURCE?\n>\n> Whether it's unusably inefficient depends on what you throw at it. IIRC,\n> the performance difference for pack-objects on git.git was fairly\n> negligible. The problem in linux.git is that besides being big, it has a\n> deep(er) directory structure. So collecting all of the attributes for a\n> file like drivers/gpu/drm/foo/bar.h needs to open all of those\n> intermediate trees.\n>\n> So I'd be inclined to leave it in place, in case somebody is actually\n> happily using it.\n>\n> GIT_ATTR_SOURCE suffers all of the same problems; it's just that you'd\n> presumably only use it with a few select commands (as far as I know,\n> pack-objects is the worst case because it's looking up one attribute on\n> every single blob in all of history).\n\nAh, OK. So your \"caution\" was about the underlying mechanism to\nallow attributes corrected from the specified tree, and not\nspecifically about using \"attr.tree\" to specify that tree? That was\nwhat got me confused.\n\nIf that is the case, I do not think the documentation patch that\nstarted this exchange that adds attr.tree to where GIT_ATTR_SOURCE\nand --attr-source are already mentioned makes anything worse.\n\nThanks.\n"},{"id":"509369","messageId":"20241220090908.GD133148@coredump.intra.peff.net","threadId":"62633","inReplyTo":"xmqqv7vhujo4.fsf@gitster.g","subject":"Re: [PATCH 0/3] batch blob diff generation","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-12-20T09:09:08Z","receivedAt":"2024-12-20T09:09:09Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 18, 2024 at 06:53:31AM -0800, Junio C Hamano wrote:\n\n> > Whether it's unusably inefficient depends on what you throw at it. IIRC,\n> > the performance difference for pack-objects on git.git was fairly\n> > negligible. The problem in linux.git is that besides being big, it has a\n> > deep(er) directory structure. So collecting all of the attributes for a\n> > file like drivers/gpu/drm/foo/bar.h needs to open all of those\n> > intermediate trees.\n> >\n> > So I'd be inclined to leave it in place, in case somebody is actually\n> > happily using it.\n> >\n> > GIT_ATTR_SOURCE suffers all of the same problems; it's just that you'd\n> > presumably only use it with a few select commands (as far as I know,\n> > pack-objects is the worst case because it's looking up one attribute on\n> > every single blob in all of history).\n> \n> Ah, OK. So your \"caution\" was about the underlying mechanism to\n> allow attributes corrected from the specified tree, and not\n> specifically about using \"attr.tree\" to specify that tree? That was\n> what got me confused.\n> \n> If that is the case, I do not think the documentation patch that\n> started this exchange that adds attr.tree to where GIT_ATTR_SOURCE\n> and --attr-source are already mentioned makes anything worse.\n\nYeah, I agree it's somewhat orthogonal. Your patch made me think about\nit because it is advertising the config variant more widely. Somebody\ndoing:\n\n git --attr-source=foo diff ...\n\nis probably OK, but:\n\n git --attr-source=foo pack-objects ...\n\nis less so. Using attr.tree instead means you're going to do the latter\nwhether you intended to or not.\n\n-Peff\n"},{"id":"509370","messageId":"20241220091048.GE133148@coredump.intra.peff.net","threadId":"62633","inReplyTo":"20241220090908.GD133148@coredump.intra.peff.net","subject":"Re: [PATCH 0/3] batch blob diff generation","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-12-20T09:10:48Z","receivedAt":"2024-12-20T09:10:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 20, 2024 at 04:09:08AM -0500, Jeff King wrote:\n\n> > Ah, OK. So your \"caution\" was about the underlying mechanism to\n> > allow attributes corrected from the specified tree, and not\n> > specifically about using \"attr.tree\" to specify that tree? That was\n> > what got me confused.\n> > \n> > If that is the case, I do not think the documentation patch that\n> > started this exchange that adds attr.tree to where GIT_ATTR_SOURCE\n> > and --attr-source are already mentioned makes anything worse.\n> \n> Yeah, I agree it's somewhat orthogonal. Your patch made me think about\n> it because it is advertising the config variant more widely. Somebody\n> doing:\n> \n> git --attr-source=foo diff ...\n> \n> is probably OK, but:\n> \n> git --attr-source=foo pack-objects ...\n> \n> is less so. Using attr.tree instead means you're going to do the latter\n> whether you intended to or not.\n\nRe-reading my message, I guess I didn't really give any conclusion. ;)\n\nI should add: I'm OK with leaving the performance implications\nundocumented for now. Hopefully in the long run somebody is interested\nin addressing them.\n\n-Peff\n"},{"id":"512282","messageId":"20250212041825.2455031-1-jltobler@gmail.com","threadId":"62633","inReplyTo":"20241213042312.2890841-1-jltobler@gmail.com","subject":"[PATCH v2 0/3] batch blob diff generation","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-12T04:18:22Z","receivedAt":"2025-02-12T04:22:01Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Through git-diff(1) it is possible to generate a diff directly between\ntwo blobs. This is particularly useful when the pre-image and post-image\nblobs are known and we only care about the diff between them.\nUnfortunately, if a user has a batch of known blob pairs to compute\ndiffs for, there is currently not a way to do so via a single Git\nprocess.\n\nTo enable support for batch diffs of multiple blob pairs, this\nseries introduces a new diff plumbing command git-diff-pairs(1) based on\na previous patch series submitted by Peff[1]. This command uses null\ndelimited raw diffs as its source of input to control exactly which\nfilepairs are diffed. The advantage of using the raw diff format is that\nit already has diff status type and object context information embedded\nin each line making it more efficient to generate diffs with as we can\navoid having to peel revisions to get some the same info.\n\nFor example:\n\n git diff-tree -r -z -M $old $new |\n git diff-pairs -p\n\nHere the output of git-diff-tree(1) is fed to git-diff-pairs(1) to\ngenerate the same output that would be expected from `git diff-tree -p\n-M`. While by itself not particularly useful, this means it is possible\nto split git-diff-tree(1) output across multiple git-diff-pairs(1)\nprocesses. Such a feature is useful on the server-side where diffs\nbewteen a large set of changes may not be feasible all at once due to\ntimeout concerns.\n\nThis series is structured as follows:\n\n - Patch 1 adds some new helper functions to get access to the queued\n `diff_filepair` after `diff_queue()` is invoked.\n\n - Patch 2 introduces the new git-diff-pairs(1) plumbing command.\n\n - Patch 3 teaches git-diff-pairs(1) a way to perform explicit diff\n queue flushes instead of waiting until stdin EOF to flush.\n\nIn 1f010d6bdf (doc: use .adoc extension for AsciiDoc files, 2025-01-20),\nthe extension for documentation was change from .txt to .adoc. This\nseries builds on top of that change as to avoid conflicts in next.\n\nChanges since V1:\n\n - Changed from git-diff-blob(1) to git-diff-pairs(1) based on a\n previously submitted series.\n\n - Instead of each line containing a pair of blob revisions, the raw\n diff format is used as input which already has diff status and\n object context embedded.\n\n-Justin\n\n[1]: <20161201204042.6yslbyrg7l6ghhww@sigill.intra.peff.net>\n\nJustin Tobler (3):\n diff: return diff_filepair from diff queue helpers\n builtin: introduce diff-pairs command\n builtin/diff-pairs: allow explicit diff queue flush\n\n .gitignore | 1 +\n Documentation/git-diff-pairs.adoc | 66 +++++++++++\n Documentation/meson.build | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/diff-pairs.c | 189 ++++++++++++++++++++++++++++++\n command-list.txt | 1 +\n diff.c | 66 ++++++++---\n diff.h | 15 +++\n git.c | 1 +\n meson.build | 1 +\n t/meson.build | 1 +\n t/t4070-diff-pairs.sh | 102 ++++++++++++++++\n 13 files changed, 427 insertions(+), 19 deletions(-)\n create mode 100644 Documentation/git-diff-pairs.adoc\n create mode 100644 builtin/diff-pairs.c\n create mode 100755 t/t4070-diff-pairs.sh\n\n-- \n2.48.1\n\n"},{"id":"512283","messageId":"20250212041825.2455031-2-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250212041825.2455031-1-jltobler@gmail.com","subject":"[PATCH v2 1/3] diff: return diff_filepair from diff queue helpers","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-12T04:18:23Z","receivedAt":"2025-02-12T04:22:14Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"The `diff_addremove()` and `diff_change()` functions setup and queue\ndiffs, but do not return the `diff_filepair` added to the queue. In a\nsubsequent commit, modifications to `diff_filepair` need to take place\nin certain cases after being queued.\n\nSplit out the queuing operations into `diff_filepair_addremove()` and\n`diff_filepair_change()` which also return a handle to the queued\n`diff_filepair`.\n\nSigned-off-by: Justin Tobler \n---\n diff.c | 66 +++++++++++++++++++++++++++++++++++++++++-----------------\n diff.h | 15 +++++++++++++\n 2 files changed, 62 insertions(+), 19 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 019fb893a7..afbb892c26 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -7157,16 +7157,18 @@ void compute_diffstat(struct diff_options *options,\n \toptions->found_changes = !!diffstat->nr;\n }\n \n-void diff_addremove(struct diff_options *options,\n-\t\t int addremove, unsigned mode,\n-\t\t const struct object_id *oid,\n-\t\t int oid_valid,\n-\t\t const char *concatpath, unsigned dirty_submodule)\n+struct diff_filepair *diff_filepair_addremove(struct diff_options *options,\n+\t\t\t\t\t int addremove, unsigned mode,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t int oid_valid,\n+\t\t\t\t\t const char *concatpath,\n+\t\t\t\t\t unsigned dirty_submodule)\n {\n \tstruct diff_filespec *one, *two;\n+\tstruct diff_filepair *pair;\n \n \tif (S_ISGITLINK(mode) && is_submodule_ignored(concatpath, options))\n-\t\treturn;\n+\t\treturn NULL;\n \n \t/* This may look odd, but it is a preparation for\n \t * feeding \"there are unchanged files which should\n@@ -7186,7 +7188,7 @@ void diff_addremove(struct diff_options *options,\n \n \tif (options->prefix &&\n \t strncmp(concatpath, options->prefix, options->prefix_length))\n-\t\treturn;\n+\t\treturn NULL;\n \n \tone = alloc_filespec(concatpath);\n \ttwo = alloc_filespec(concatpath);\n@@ -7198,25 +7200,28 @@ void diff_addremove(struct diff_options *options,\n \t\ttwo->dirty_submodule = dirty_submodule;\n \t}\n \n-\tdiff_queue(&diff_queued_diff, one, two);\n+\tpair = diff_queue(&diff_queued_diff, one, two);\n \tif (!options->flags.diff_from_contents)\n \t\toptions->flags.has_changes = 1;\n+\n+\treturn pair;\n }\n \n-void diff_change(struct diff_options *options,\n-\t\t unsigned old_mode, unsigned new_mode,\n-\t\t const struct object_id *old_oid,\n-\t\t const struct object_id *new_oid,\n-\t\t int old_oid_valid, int new_oid_valid,\n-\t\t const char *concatpath,\n-\t\t unsigned old_dirty_submodule, unsigned new_dirty_submodule)\n+struct diff_filepair *diff_filepair_change(struct diff_options *options,\n+\t\t\t\t\t unsigned old_mode, unsigned new_mode,\n+\t\t\t\t\t const struct object_id *old_oid,\n+\t\t\t\t\t const struct object_id *new_oid,\n+\t\t\t\t\t int old_oid_valid, int new_oid_valid,\n+\t\t\t\t\t const char *concatpath,\n+\t\t\t\t\t unsigned old_dirty_submodule,\n+\t\t\t\t\t unsigned new_dirty_submodule)\n {\n \tstruct diff_filespec *one, *two;\n \tstruct diff_filepair *p;\n \n \tif (S_ISGITLINK(old_mode) && S_ISGITLINK(new_mode) &&\n \t is_submodule_ignored(concatpath, options))\n-\t\treturn;\n+\t\treturn NULL;\n \n \tif (options->flags.reverse_diff) {\n \t\tSWAP(old_mode, new_mode);\n@@ -7227,7 +7232,7 @@ void diff_change(struct diff_options *options,\n \n \tif (options->prefix &&\n \t strncmp(concatpath, options->prefix, options->prefix_length))\n-\t\treturn;\n+\t\treturn NULL;\n \n \tone = alloc_filespec(concatpath);\n \ttwo = alloc_filespec(concatpath);\n@@ -7238,16 +7243,39 @@ void diff_change(struct diff_options *options,\n \tp = diff_queue(&diff_queued_diff, one, two);\n \n \tif (options->flags.diff_from_contents)\n-\t\treturn;\n+\t\treturn p;\n \n \tif (options->flags.quick && options->skip_stat_unmatch &&\n \t !diff_filespec_check_stat_unmatch(options->repo, p)) {\n \t\tdiff_free_filespec_data(p->one);\n \t\tdiff_free_filespec_data(p->two);\n-\t\treturn;\n+\t\treturn p;\n \t}\n \n \toptions->flags.has_changes = 1;\n+\n+\treturn p;\n+}\n+\n+void diff_addremove(struct diff_options *options, int addremove, unsigned mode,\n+\t\t const struct object_id *oid, int oid_valid,\n+\t\t const char *concatpath, unsigned dirty_submodule)\n+{\n+\tdiff_filepair_addremove(options, addremove, mode, oid, oid_valid,\n+\t\t\t\tconcatpath, dirty_submodule);\n+}\n+\n+void diff_change(struct diff_options *options,\n+\t\t unsigned old_mode, unsigned new_mode,\n+\t\t const struct object_id *old_oid,\n+\t\t const struct object_id *new_oid,\n+\t\t int old_oid_valid, int new_oid_valid,\n+\t\t const char *concatpath,\n+\t\t unsigned old_dirty_submodule, unsigned new_dirty_submodule)\n+{\n+\tdiff_filepair_change(options, old_mode, new_mode, old_oid, new_oid,\n+\t\t\t old_oid_valid, new_oid_valid, concatpath,\n+\t\t\t old_dirty_submodule, new_dirty_submodule);\n }\n \n struct diff_filepair *diff_unmerge(struct diff_options *options, const char *path)\ndiff --git a/diff.h b/diff.h\nindex 0a566f5531..6ea63f01e7 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -508,6 +508,21 @@ void diff_set_default_prefix(struct diff_options *options);\n \n int diff_can_quit_early(struct diff_options *);\n \n+struct diff_filepair *diff_filepair_addremove(struct diff_options *,\n+\t\t\t\t\t int addremove, unsigned mode,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t int oid_valid, const char *fullpath,\n+\t\t\t\t\t unsigned dirty_submodule);\n+\n+struct diff_filepair *diff_filepair_change(struct diff_options *,\n+\t\t\t\t\t unsigned mode1, unsigned mode2,\n+\t\t\t\t\t const struct object_id *old_oid,\n+\t\t\t\t\t const struct object_id *new_oid,\n+\t\t\t\t\t int old_oid_valid, int new_oid_valid,\n+\t\t\t\t\t const char *fullpath,\n+\t\t\t\t\t unsigned dirty_submodule1,\n+\t\t\t\t\t unsigned dirty_submodule2);\n+\n void diff_addremove(struct diff_options *,\n \t\t int addremove,\n \t\t unsigned mode,\n-- \n2.48.1\n\n"},{"id":"512284","messageId":"20250212041825.2455031-3-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250212041825.2455031-1-jltobler@gmail.com","subject":"[PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-12T04:18:24Z","receivedAt":"2025-02-12T04:22:18Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Through git-diff(1), a single diff can be generated from a pair of blob\nrevisions directly. Unfortunately, there is not a mechanism to compute\nbatches of specific file pair diffs in a single process. Such a feature\nis particularly useful on the server-side where diffing between a large\nset of changes is not feasible all at once due to timeout concerns.\n\nTo facilitate this, introduce git-diff-pairs(1) which takes the\nnull-terminated raw diff format as input on stdin and produces diffs in\nother formats. As the raw diff format already contains the necessary\nmetadata, it becomes possible to progressively generate batches of diffs\nwithout having to recompute rename detection or retrieve object context.\nSomething like the following:\n\n\tgit diff-tree -r -z -M $old $new |\n\tgit diff-pairs -p\n\nshould generate the same output as `git diff-tree -p -M`. Furthermore,\neach line of raw diff formatted input can also be individually fed to a\nseparate git-diff-pairs(1) process and still produce the same output.\n\nBased-on-patch-by: Jeff King \nSigned-off-by: Justin Tobler \n---\n .gitignore | 1 +\n Documentation/git-diff-pairs.adoc | 62 +++++++++++\n Documentation/meson.build | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/diff-pairs.c | 178 ++++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n meson.build | 1 +\n t/meson.build | 1 +\n t/t4070-diff-pairs.sh | 80 ++++++++++++++\n 11 files changed, 328 insertions(+)\n create mode 100644 Documentation/git-diff-pairs.adoc\n create mode 100644 builtin/diff-pairs.c\n create mode 100755 t/t4070-diff-pairs.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex e82aa19df0..03448c076a 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -54,6 +54,7 @@\n /git-diff\n /git-diff-files\n /git-diff-index\n+/git-diff-pairs\n /git-diff-tree\n /git-difftool\n /git-difftool--helper\ndiff --git a/Documentation/git-diff-pairs.adoc b/Documentation/git-diff-pairs.adoc\nnew file mode 100644\nindex 0000000000..e9ef4a6615\n--- /dev/null\n+++ b/Documentation/git-diff-pairs.adoc\n@@ -0,0 +1,62 @@\n+git-diff-pairs(1)\n+=================\n+\n+NAME\n+----\n+git-diff-pairs - Compare blob pairs generated by `diff-tree --raw`\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git diff-pairs' [diff-options]\n+\n+DESCRIPTION\n+-----------\n+\n+Given the output of `diff-tree -z` on its stdin, `diff-pairs` will\n+reformat that output into whatever format is requested on its command\n+line. For example:\n+\n+-----------------------------\n+git diff-tree -z -M $a $b |\n+git diff-pairs -p\n+-----------------------------\n+\n+will compute the tree diff in one step (including renames), and then\n+`diff-pairs` will compute and format the blob-level diffs for each pair.\n+This can be used to modify the raw diff in the middle (without having to\n+parse or re-create more complicated formats like `--patch`), or to\n+compute diffs progressively over the course of multiple invocations of\n+`diff-pairs`.\n+\n+Each blob pair is fed to the diff machinery individually queued and the output\n+is flushed on stdin EOF.\n+\n+OPTIONS\n+-------\n+\n+include::diff-options.adoc[]\n+\n+include::diff-generate-patch.adoc[]\n+\n+NOTES\n+----\n+\n+`diff-pairs` should handle any input generated by `diff-tree --raw -z`.\n+It may choke or otherwise misbehave on output from `diff-files`, etc.\n+\n+Here's an incomplete list of things that `diff-pairs` could do, but\n+doesn't (mostly in the name of simplicity):\n+\n+ - Only `-z` input is accepted, not normal `--raw` input.\n+\n+ - Abbreviated sha1s are rejected in the input from `diff-tree`; if you\n+ want to abbreviate the output, you can pass `--abbrev` to\n+ `diff-pairs`.\n+\n+ - Pathspecs are not handled by `diff-pairs`; you can limit the diff via\n+ the initial `diff-tree` invocation.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Documentation/meson.build b/Documentation/meson.build\nindex ead8e48213..e5ee177022 100644\n--- a/Documentation/meson.build\n+++ b/Documentation/meson.build\n@@ -41,6 +41,7 @@ manpages = {\n 'git-diagnose.adoc' : 1,\n 'git-diff-files.adoc' : 1,\n 'git-diff-index.adoc' : 1,\n+ 'git-diff-pairs.adoc' : 1,\n 'git-difftool.adoc' : 1,\n 'git-diff-tree.adoc' : 1,\n 'git-diff.adoc' : 1,\ndiff --git a/Makefile b/Makefile\nindex 896d02339e..3b8e1ad15e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1232,6 +1232,7 @@ BUILTIN_OBJS += builtin/describe.o\n BUILTIN_OBJS += builtin/diagnose.o\n BUILTIN_OBJS += builtin/diff-files.o\n BUILTIN_OBJS += builtin/diff-index.o\n+BUILTIN_OBJS += builtin/diff-pairs.o\n BUILTIN_OBJS += builtin/diff-tree.o\n BUILTIN_OBJS += builtin/diff.o\n BUILTIN_OBJS += builtin/difftool.o\ndiff --git a/builtin.h b/builtin.h\nindex f7b166b334..b2d2e9eb07 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -152,6 +152,7 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix, struct reposit\n int cmd_diff_files(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_diff_index(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_diff(int argc, const char **argv, const char *prefix, struct repository *repo);\n+int cmd_diff_pairs(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_diff_tree(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_difftool(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_env__helper(int argc, const char **argv, const char *prefix, struct repository *repo);\ndiff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\nnew file mode 100644\nindex 0000000000..08f3ee81e5\n--- /dev/null\n+++ b/builtin/diff-pairs.c\n@@ -0,0 +1,178 @@\n+#include \"builtin.h\"\n+#include \"commit.h\"\n+#include \"config.h\"\n+#include \"diff.h\"\n+#include \"diffcore.h\"\n+#include \"gettext.h\"\n+#include \"hex.h\"\n+#include \"object.h\"\n+#include \"parse-options.h\"\n+#include \"revision.h\"\n+#include \"strbuf.h\"\n+\n+static unsigned parse_mode_or_die(const char *mode, const char **endp)\n+{\n+\tuint16_t ret;\n+\n+\t*endp = parse_mode(mode, &ret);\n+\tif (!*endp)\n+\t\tdie(\"unable to parse mode: %s\", mode);\n+\treturn ret;\n+}\n+\n+static void parse_oid(const char *p, struct object_id *oid, const char **endp,\n+\t\t const struct git_hash_algo *algop)\n+{\n+\tif (parse_oid_hex_algop(p, oid, endp, algop) || *(*endp)++ != ' ')\n+\t\tdie(\"unable to parse object id: %s\", p);\n+}\n+\n+static unsigned short parse_score(const char *score)\n+{\n+\tunsigned long ret;\n+\tchar *endp;\n+\n+\terrno = 0;\n+\tret = strtoul(score, &endp, 10);\n+\tret *= MAX_SCORE / 100;\n+\tif (errno || endp == score || *endp || (unsigned short)ret != ret)\n+\t\tdie(\"unable to parse rename/copy score: %s\", score);\n+\treturn ret;\n+}\n+\n+static void flush_diff_queue(struct diff_options *options)\n+{\n+\t/*\n+\t * If rename detection is not requested, use rename information from the\n+\t * raw diff formatted input. Setting found_follow ensures diffcore_std()\n+\t * does not mess with rename information already present in queued\n+\t * filepairs.\n+\t */\n+\tif (!options->detect_rename)\n+\t\toptions->found_follow = 1;\n+\tdiffcore_std(options);\n+\tdiff_flush(options);\n+}\n+\n+int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n+\t\t struct repository *repo)\n+{\n+\tstruct strbuf path_dst = STRBUF_INIT;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct strbuf meta = STRBUF_INIT;\n+\tstruct rev_info revs;\n+\tint ret;\n+\n+\tconst char * const usage[] = {\n+\t\tN_(\"git diff-pairs [diff-options]\"),\n+\t\tNULL\n+\t};\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tshow_usage_with_options_if_asked(argc, argv, usage, options);\n+\n+\trepo_init_revisions(repo, &revs, prefix);\n+\trepo_config(repo, git_diff_basic_config, NULL);\n+\trevs.disable_stdin = 1;\n+\trevs.abbrev = 0;\n+\trevs.diff = 1;\n+\n+\targc = setup_revisions(argc, argv, &revs, NULL);\n+\n+\t/* Don't allow pathspecs at all. */\n+\tif (revs.prune_data.nr)\n+\t\tusage_with_options(usage, options);\n+\n+\tif (!revs.diffopt.output_format)\n+\t\trevs.diffopt.output_format = DIFF_FORMAT_RAW;\n+\n+\twhile (1) {\n+\t\tstruct object_id oid_a, oid_b;\n+\t\tstruct diff_filepair *pair;\n+\t\tunsigned mode_a, mode_b;\n+\t\tconst char *p;\n+\t\tchar status;\n+\n+\t\tif (strbuf_getline_nul(&meta, stdin) == EOF)\n+\t\t\tbreak;\n+\n+\t\tp = meta.buf;\n+\t\tif (*p != ':')\n+\t\t\tdie(\"invalid raw diff input\");\n+\t\tp++;\n+\n+\t\tmode_a = parse_mode_or_die(p, &p);\n+\t\tmode_b = parse_mode_or_die(p, &p);\n+\n+\t\tparse_oid(p, &oid_a, &p, repo->hash_algo);\n+\t\tparse_oid(p, &oid_b, &p, repo->hash_algo);\n+\n+\t\tstatus = *p++;\n+\n+\t\tif (strbuf_getline_nul(&path, stdin) == EOF)\n+\t\t\tdie(\"got EOF while reading path\");\n+\n+\t\tswitch (status) {\n+\t\tcase DIFF_STATUS_ADDED:\n+\t\t\tpair = diff_filepair_addremove(&revs.diffopt, '+',\n+\t\t\t\t\t\t mode_b, &oid_b,\n+\t\t\t\t\t\t 1, path.buf, 0);\n+\t\t\tif (pair)\n+\t\t\t\tpair->status = status;\n+\t\t\tbreak;\n+\n+\t\tcase DIFF_STATUS_DELETED:\n+\t\t\tpair = diff_filepair_addremove(&revs.diffopt, '-',\n+\t\t\t\t\t\t mode_a, &oid_a,\n+\t\t\t\t\t\t 1, path.buf, 0);\n+\t\t\tif (pair)\n+\t\t\t\tpair->status = status;\n+\t\t\tbreak;\n+\n+\t\tcase DIFF_STATUS_TYPE_CHANGED:\n+\t\tcase DIFF_STATUS_MODIFIED:\n+\t\t\tpair = diff_filepair_change(&revs.diffopt,\n+\t\t\t\t\t\t mode_a, mode_b,\n+\t\t\t\t\t\t &oid_a, &oid_b, 1, 1,\n+\t\t\t\t\t\t path.buf, 0, 0);\n+\t\t\tif (pair)\n+\t\t\t\tpair->status = status;\n+\t\t\tbreak;\n+\n+\t\tcase DIFF_STATUS_RENAMED:\n+\t\tcase DIFF_STATUS_COPIED:\n+\t\t\t{\n+\t\t\t\tstruct diff_filespec *a, *b;\n+\n+\t\t\t\tif (strbuf_getline_nul(&path_dst, stdin) == EOF)\n+\t\t\t\t\tdie(\"got EOF while reading destination path\");\n+\n+\t\t\t\ta = alloc_filespec(path.buf);\n+\t\t\t\tb = alloc_filespec(path_dst.buf);\n+\t\t\t\tfill_filespec(a, &oid_a, 1, mode_a);\n+\t\t\t\tfill_filespec(b, &oid_b, 1, mode_b);\n+\n+\t\t\t\tpair = diff_queue(&diff_queued_diff, a, b);\n+\t\t\t\tpair->status = status;\n+\t\t\t\tpair->score = parse_score(p);\n+\t\t\t\tpair->renamed_pair = 1;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tdefault:\n+\t\t\tdie(\"unknown diff status: %c\", status);\n+\t\t}\n+\t}\n+\n+\tflush_diff_queue(&revs.diffopt);\n+\tret = diff_result_code(&revs);\n+\n+\tstrbuf_release(&path_dst);\n+\tstrbuf_release(&path);\n+\tstrbuf_release(&meta);\n+\trelease_revisions(&revs);\n+\n+\treturn ret;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex e0bb87b3b5..bb8acd51d8 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -95,6 +95,7 @@ git-diagnose ancillaryinterrogators\n git-diff mainporcelain info\n git-diff-files plumbinginterrogators\n git-diff-index plumbinginterrogators\n+git-diff-pairs plumbinginterrogators\n git-diff-tree plumbinginterrogators\n git-difftool ancillaryinterrogators complete\n git-fast-export ancillarymanipulators\ndiff --git a/git.c b/git.c\nindex b23761480f..12bba872bb 100644\n--- a/git.c\n+++ b/git.c\n@@ -540,6 +540,7 @@ static struct cmd_struct commands[] = {\n \t{ \"diff\", cmd_diff, NO_PARSEOPT },\n \t{ \"diff-files\", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n \t{ \"diff-index\", cmd_diff_index, RUN_SETUP | NO_PARSEOPT },\n+\t{ \"diff-pairs\", cmd_diff_pairs, RUN_SETUP | NO_PARSEOPT },\n \t{ \"diff-tree\", cmd_diff_tree, RUN_SETUP | NO_PARSEOPT },\n \t{ \"difftool\", cmd_difftool, RUN_SETUP_GENTLY },\n \t{ \"fast-export\", cmd_fast_export, RUN_SETUP },\ndiff --git a/meson.build b/meson.build\nindex fbb8105d96..66ce3326e8 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -537,6 +537,7 @@ builtin_sources = [\n 'builtin/diagnose.c',\n 'builtin/diff-files.c',\n 'builtin/diff-index.c',\n+ 'builtin/diff-pairs.c',\n 'builtin/diff-tree.c',\n 'builtin/diff.c',\n 'builtin/difftool.c',\ndiff --git a/t/meson.build b/t/meson.build\nindex 4574280590..7ff17c6d29 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -500,6 +500,7 @@ integration_tests = [\n 't4067-diff-partial-clone.sh',\n 't4068-diff-symmetric-merge-base.sh',\n 't4069-remerge-diff.sh',\n+ 't4070-diff-pairs.sh',\n 't4100-apply-stat.sh',\n 't4101-apply-nonl.sh',\n 't4102-apply-rename.sh',\ndiff --git a/t/t4070-diff-pairs.sh b/t/t4070-diff-pairs.sh\nnew file mode 100755\nindex 0000000000..e0a8e6f0a0\n--- /dev/null\n+++ b/t/t4070-diff-pairs.sh\n@@ -0,0 +1,80 @@\n+#!/bin/sh\n+\n+test_description='basic diff-pairs tests'\n+. ./test-lib.sh\n+\n+# This creates a diff with added, modified, deleted, renamed, copied, and\n+# typechange entries. That includes one in a subdirectory for non-recursive\n+# tests, and both exact and inexact similarity scores.\n+test_expect_success 'create commit with various diffs' '\n+\techo to-be-gone >deleted &&\n+\techo original >modified &&\n+\techo now-a-file >symlink &&\n+\ttest_seq 200 >two-hundred &&\n+\ttest_seq 201 500 >five-hundred &&\n+\tgit add . &&\n+\ttest_tick &&\n+\tgit commit -m base &&\n+\tgit tag base &&\n+\n+\techo now-here >added &&\n+\techo new >modified &&\n+\trm deleted &&\n+\tmkdir subdir &&\n+\techo content >subdir/file &&\n+\tmv two-hundred renamed &&\n+\ttest_seq 201 500 | sed s/300/modified/ >copied &&\n+\trm symlink &&\n+\tgit add -A . &&\n+\ttest_ln_s_add dest symlink &&\n+\ttest_tick &&\n+\tgit commit -m new &&\n+\tgit tag new\n+'\n+\n+test_expect_success 'diff-pairs recreates --raw' '\n+\tgit diff-tree -r -M -C -C base new >expect &&\n+\tgit diff-tree -r -M -C -C -z base new |\n+\tgit diff-pairs >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'diff-pairs can create -p output' '\n+\tgit diff-tree -p -M -C -C base new >expect &&\n+\tgit diff-tree -r -M -C -C -z base new |\n+\tgit diff-pairs -p >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'non-recursive --raw retains tree entry' '\n+\tgit diff-tree base new >expect &&\n+\tgit diff-tree -z base new |\n+\tgit diff-pairs >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'split input across multiple diff-pairs' '\n+\twrite_script split-raw-diff \"$PERL_PATH\" <<-\\EOF &&\n+\t$/ = \"\\0\";\n+\twhile (<>) {\n+\t my $meta = $_;\n+\t my $path = <>;\n+\t # renames have an extra path\n+\t my $path2 = <> if $meta =~ /[RC]\\d+/;\n+\n+\t open(my $fh, \">\", sprintf \"diff%03d\", $.);\n+\t print $fh $meta, $path, $path2;\n+\t}\n+\tEOF\n+\n+\tgit diff-tree -p -M -C -C base new >expect &&\n+\n+\tgit diff-tree -r -z -M -C -C base new |\n+\t./split-raw-diff &&\n+\tfor i in diff*; do\n+\t\tgit diff-pairs -p <$i || return 1\n+\tdone >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.48.1\n\n"},{"id":"512285","messageId":"20250212041825.2455031-4-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250212041825.2455031-1-jltobler@gmail.com","subject":"[PATCH v2 3/3] builtin/diff-pairs: allow explicit diff queue flush","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-12T04:18:25Z","receivedAt":"2025-02-12T04:22:20Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"The diffs queued from git-diff-pairs(1) stdin are not flushed EOF is\nreached. To enable greater flexibility, allow control over when the diff\nqueue is flushed by writing a single nul byte on stdin between input\nfile pairs. Diff output between flushes is separated by a single line\nterminator.\n\nSigned-off-by: Justin Tobler \n---\n Documentation/git-diff-pairs.adoc | 4 ++++\n builtin/diff-pairs.c | 11 +++++++++++\n t/t4070-diff-pairs.sh | 22 ++++++++++++++++++++++\n 3 files changed, 37 insertions(+)\n\ndiff --git a/Documentation/git-diff-pairs.adoc b/Documentation/git-diff-pairs.adoc\nindex e9ef4a6615..33c0d702f0 100644\n--- a/Documentation/git-diff-pairs.adoc\n+++ b/Documentation/git-diff-pairs.adoc\n@@ -32,6 +32,10 @@ compute diffs progressively over the course of multiple invocations of\n Each blob pair is fed to the diff machinery individually queued and the output\n is flushed on stdin EOF.\n \n+To explicitly flush the diff queue, a single nul byte can be written to stdin\n+between filepairs. Diff output between flushes is separated by a single line\n+terminator.\n+\n OPTIONS\n -------\n \ndiff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\nindex 08f3ee81e5..2436ce3013 100644\n--- a/builtin/diff-pairs.c\n+++ b/builtin/diff-pairs.c\n@@ -99,6 +99,17 @@ int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n \t\t\tbreak;\n \n \t\tp = meta.buf;\n+\t\tif (!*p) {\n+\t\t\tflush_diff_queue(&revs.diffopt);\n+\t\t\t/*\n+\t\t\t * When the diff queue is explicitly flushed, append an\n+\t\t\t * additional terminator to separate batches of diffs.\n+\t\t\t */\n+\t\t\tfprintf(revs.diffopt.file, \"%c\",\n+\t\t\t\trevs.diffopt.line_termination);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (*p != ':')\n \t\t\tdie(\"invalid raw diff input\");\n \t\tp++;\ndiff --git a/t/t4070-diff-pairs.sh b/t/t4070-diff-pairs.sh\nindex e0a8e6f0a0..aca228a8fa 100755\n--- a/t/t4070-diff-pairs.sh\n+++ b/t/t4070-diff-pairs.sh\n@@ -77,4 +77,26 @@ test_expect_success 'split input across multiple diff-pairs' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'diff-pairs explicit queue flush' '\n+\tgit diff-tree -r -M -C -C -z base new >input &&\n+\tprintf \"\\0\" >>input &&\n+\tgit diff-tree -r -M -C -C -z base new >>input &&\n+\n+\tgit diff-tree -r -M -C -C base new >expect &&\n+\tprintf \"\\n\" >>expect &&\n+\tgit diff-tree -r -M -C -C base new >>expect &&\n+\n+\tgit diff-pairs actual &&\n+\ttest_cmp expect actual\n+'\n+j\n+test_expect_success 'diff-pairs explicit queue flush null terminated' '\n+\tgit diff-tree -r -M -C -C -z base new >expect &&\n+\tprintf \"\\0\" >>expect &&\n+\tgit diff-tree -r -M -C -C -z base new >>expect &&\n+\n+\tgit diff-pairs -z actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \n2.48.1\n\n"},{"id":"512297","messageId":"CAOLa=ZRssYP8U+kyTYdUY_-WGCLLk4mVxQRbgu87g7AdxPJbZw@mail.gmail.com","threadId":"62633","inReplyTo":"20250212041825.2455031-2-jltobler@gmail.com","subject":"Re: [PATCH v2 1/3] diff: return diff_filepair from diff queue helpers","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-02-12T09:06:59Z","receivedAt":"2025-02-12T09:07:01Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Justin Tobler writes:\n\n> The `diff_addremove()` and `diff_change()` functions setup and queue\n> diffs, but do not return the `diff_filepair` added to the queue. In a\n> subsequent commit, modifications to `diff_filepair` need to take place\n> in certain cases after being queued.\n>\n> Split out the queuing operations into `diff_filepair_addremove()` and\n> `diff_filepair_change()` which also return a handle to the queued\n> `diff_filepair`.\n>\n\nThis patch keeps `diff_addremove()` and `diff_change()` while\nintroducing two new functions which return the `diff_filepair`. Just a\nthought, why not replace them? The users `diff_addremove()` and\n`diff_change()` could simply call the new functions and ignore the\nreturn value?\n\nThis would be messy if there were a lot of users of `diff_addremove()`\nand `diff_change()`, but I only see a few callers. Wouldn't it be\ncleaner to just replace?\n\nThe patch looks good to me otherwise.\n\n[snip]\n\n> diff --git a/diff.h b/diff.h\n> index 0a566f5531..6ea63f01e7 100644\n> --- a/diff.h\n> +++ b/diff.h\n> @@ -508,6 +508,21 @@ void diff_set_default_prefix(struct diff_options *options);\n>\n> int diff_can_quit_early(struct diff_options *);\n>\n> +struct diff_filepair *diff_filepair_addremove(struct diff_options *,\n> +\t\t\t\t\t int addremove, unsigned mode,\n> +\t\t\t\t\t const struct object_id *oid,\n> +\t\t\t\t\t int oid_valid, const char *fullpath,\n> +\t\t\t\t\t unsigned dirty_submodule);\n> +\n> +struct diff_filepair *diff_filepair_change(struct diff_options *,\n> +\t\t\t\t\t unsigned mode1, unsigned mode2,\n> +\t\t\t\t\t const struct object_id *old_oid,\n> +\t\t\t\t\t const struct object_id *new_oid,\n> +\t\t\t\t\t int old_oid_valid, int new_oid_valid,\n> +\t\t\t\t\t const char *fullpath,\n> +\t\t\t\t\t unsigned dirty_submodule1,\n> +\t\t\t\t\t unsigned dirty_submodule2);\n> +\n\nNit: would be nice to have some comments to describe what these\nfunctions do.\n\n> void diff_addremove(struct diff_options *,\n> \t\t int addremove,\n> \t\t unsigned mode,\n> --\n> 2.48.1\n"},{"id":"512298","messageId":"Z6xomFk_Sb18UMFo@pks.im","threadId":"62633","inReplyTo":"20250212041825.2455031-2-jltobler@gmail.com","subject":"Re: [PATCH v2 1/3] diff: return diff_filepair from diff queue helpers","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-12T09:23:36Z","receivedAt":"2025-02-12T09:23:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 11, 2025 at 10:18:23PM -0600, Justin Tobler wrote:\n> The `diff_addremove()` and `diff_change()` functions setup and queue\n> diffs, but do not return the `diff_filepair` added to the queue. In a\n> subsequent commit, modifications to `diff_filepair` need to take place\n> in certain cases after being queued.\n> \n> Split out the queuing operations into `diff_filepair_addremove()` and\n> `diff_filepair_change()` which also return a handle to the queued\n> `diff_filepair`.\n\nOne of the things that puzzled me a bit is that we keep the old-style\nfunctions, where the only difference is the return value. Wouldn't it\nmake more sense to instead adapt these existing functions to reduce the\namount of duplication?\n\nAt the same time, while we're already at it, do we maybe also want to\nadapt the functions so that they get the `diff_queue` as input instead\nof relying on the global queue? That would make them more generally\nuseful and be a step into the right direction regarding libification. If\nso, it would indeed make sense to also rename the function into e.g.\n`diff_queue_addremove()`.\n\nPatrick\n"},{"id":"512299","messageId":"Z6xooEvy3w11T_px@pks.im","threadId":"62633","inReplyTo":"20250212041825.2455031-3-jltobler@gmail.com","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-12T09:23:44Z","receivedAt":"2025-02-12T09:23:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 11, 2025 at 10:18:24PM -0600, Justin Tobler wrote:\n> Through git-diff(1), a single diff can be generated from a pair of blob\n> revisions directly. Unfortunately, there is not a mechanism to compute\n> batches of specific file pair diffs in a single process. Such a feature\n> is particularly useful on the server-side where diffing between a large\n> set of changes is not feasible all at once due to timeout concerns.\n> \n> To facilitate this, introduce git-diff-pairs(1) which takes the\n> null-terminated raw diff format as input on stdin and produces diffs in\n\ns/null/NUL/\n\n> other formats. As the raw diff format already contains the necessary\n> metadata, it becomes possible to progressively generate batches of diffs\n> without having to recompute rename detection or retrieve object context.\n> Something like the following:\n> \n> \tgit diff-tree -r -z -M $old $new |\n> \tgit diff-pairs -p\n> \n> should generate the same output as `git diff-tree -p -M`. Furthermore,\n> each line of raw diff formatted input can also be individually fed to a\n> separate git-diff-pairs(1) process and still produce the same output.\n> \n> Based-on-patch-by: Jeff King \n> Signed-off-by: Justin Tobler \n\nI really like this new design, thanks for working well together!\n\n> diff --git a/Documentation/git-diff-pairs.adoc b/Documentation/git-diff-pairs.adoc\n> new file mode 100644\n> index 0000000000..e9ef4a6615\n> --- /dev/null\n> +++ b/Documentation/git-diff-pairs.adoc\n> @@ -0,0 +1,62 @@\n> +git-diff-pairs(1)\n> +=================\n> +\n> +NAME\n> +----\n> +git-diff-pairs - Compare blob pairs generated by `diff-tree --raw`\n> +\n> +SYNOPSIS\n> +--------\n> +[verse]\n> +'git diff-pairs' [diff-options]\n\nThis should use `[synopsis]`, which allows you to drop the quoting.\n\n> +DESCRIPTION\n> +-----------\n> +\n> +Given the output of `diff-tree -z` on its stdin, `diff-pairs` will\n> +reformat that output into whatever format is requested on its command\n\nReformatting from my point of view implies that we only rearrange bits a\nbit. But we're not only reformatting the input, but actually compute the\ndiffs.\n\n> +line. For example:\n> +\n> +-----------------------------\n> +git diff-tree -z -M $a $b |\n> +git diff-pairs -p\n> +-----------------------------\n> +\n> +will compute the tree diff in one step (including renames), and then\n> +`diff-pairs` will compute and format the blob-level diffs for each pair.\n> +This can be used to modify the raw diff in the middle (without having to\n> +parse or re-create more complicated formats like `--patch`), or to\n> +compute diffs progressively over the course of multiple invocations of\n> +`diff-pairs`.\n> +\n> +Each blob pair is fed to the diff machinery individually queued and the output\n> +is flushed on stdin EOF.\n\nI think the \"flushing\" part is a bit hard to understand without knowing\nanything about the command internals. As an unknowing reader I would\nassume you're talking about fflush(3p), but I think you're rather\ntalking about when we \"flush\" the internal diff queue and thus compute\nthe diffs.\n\nSo I'd rephrase this and not talk about flushing, but about the\nbehaviour observed by the user instead.\n\n> +OPTIONS\n> +-------\n> +\n> +include::diff-options.adoc[]\n> +\n> +include::diff-generate-patch.adoc[]\n> +\n> +NOTES\n> +----\n> +\n> +`diff-pairs` should handle any input generated by `diff-tree --raw -z`.\n> +It may choke or otherwise misbehave on output from `diff-files`, etc.\n\nThis reads a bit weird. The first thing that trips me is the \"should\".\nDoes it or doesn't it handle the output of git-diff-tree(1)? The second\npart is that this, at least to me, implies that other formats of course\naren't accepted, so why point that out explicitly?\n\n> +Here's an incomplete list of things that `diff-pairs` could do, but\n> +doesn't (mostly in the name of simplicity):\n> +\n> + - Only `-z` input is accepted, not normal `--raw` input.\n> +\n> + - Abbreviated sha1s are rejected in the input from `diff-tree`; if you\n\ns/sha1s/object IDs/\n\n> + want to abbreviate the output, you can pass `--abbrev` to\n> + `diff-pairs`.\n> +\n> + - Pathspecs are not handled by `diff-pairs`; you can limit the diff via\n> + the initial `diff-tree` invocation.\n\nMakes sense.\n\n> diff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\n> new file mode 100644\n> index 0000000000..08f3ee81e5\n> --- /dev/null\n> +++ b/builtin/diff-pairs.c\n> @@ -0,0 +1,178 @@\n> +#include \"builtin.h\"\n> +#include \"commit.h\"\n> +#include \"config.h\"\n> +#include \"diff.h\"\n> +#include \"diffcore.h\"\n> +#include \"gettext.h\"\n> +#include \"hex.h\"\n> +#include \"object.h\"\n> +#include \"parse-options.h\"\n> +#include \"revision.h\"\n> +#include \"strbuf.h\"\n> +\n> +static unsigned parse_mode_or_die(const char *mode, const char **endp)\n> +{\n> +\tuint16_t ret;\n> +\n> +\t*endp = parse_mode(mode, &ret);\n> +\tif (!*endp)\n> +\t\tdie(\"unable to parse mode: %s\", mode);\n\nMissing translation.\n\n> +\treturn ret;\n> +}\n> +\n> +static void parse_oid(const char *p, struct object_id *oid, const char **endp,\n> +\t\t const struct git_hash_algo *algop)\n> +{\n> +\tif (parse_oid_hex_algop(p, oid, endp, algop) || *(*endp)++ != ' ')\n> +\t\tdie(\"unable to parse object id: %s\", p);\n\nHere, too. Do we also name this `parse_oid_or_die()` to stay consistent\nwith `parse_mode_or_die()`? The same is also true for `parse_score()`.\n\n> +}\n> +\n> +static unsigned short parse_score(const char *score)\n> +{\n> +\tunsigned long ret;\n> +\tchar *endp;\n> +\n> +\terrno = 0;\n> +\tret = strtoul(score, &endp, 10);\n> +\tret *= MAX_SCORE / 100;\n> +\tif (errno || endp == score || *endp || (unsigned short)ret != ret)\n> +\t\tdie(\"unable to parse rename/copy score: %s\", score);\n> +\treturn ret;\n> +}\n\nYou can use `strtoul_ui()` instead, which does most of the error\nhandling for you.\n\n> +static void flush_diff_queue(struct diff_options *options)\n> +{\n> +\t/*\n> +\t * If rename detection is not requested, use rename information from the\n> +\t * raw diff formatted input. Setting found_follow ensures diffcore_std()\n> +\t * does not mess with rename information already present in queued\n> +\t * filepairs.\n> +\t */\n> +\tif (!options->detect_rename)\n> +\t\toptions->found_follow = 1;\n> +\tdiffcore_std(options);\n> +\tdiff_flush(options);\n> +}\n> +\n> +int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n> +\t\t struct repository *repo)\n> +{\n> +\tstruct strbuf path_dst = STRBUF_INIT;\n> +\tstruct strbuf path = STRBUF_INIT;\n> +\tstruct strbuf meta = STRBUF_INIT;\n> +\tstruct rev_info revs;\n> +\tint ret;\n> +\n> +\tconst char * const usage[] = {\n> +\t\tN_(\"git diff-pairs [diff-options]\"),\n> +\t\tNULL\n> +\t};\n> +\tstruct option options[] = {\n> +\t\tOPT_END()\n> +\t};\n> +\n> +\tshow_usage_with_options_if_asked(argc, argv, usage, options);\n> +\n> +\trepo_init_revisions(repo, &revs, prefix);\n> +\trepo_config(repo, git_diff_basic_config, NULL);\n> +\trevs.disable_stdin = 1;\n> +\trevs.abbrev = 0;\n> +\trevs.diff = 1;\n> +\n> +\targc = setup_revisions(argc, argv, &revs, NULL);\n\nWe need to check whether `argc > 0` here. Otherwise, unknown parameters\nmay simply be ignored, I think.\n\n> +\n> +\t/* Don't allow pathspecs at all. */\n> +\tif (revs.prune_data.nr)\n> +\t\tusage_with_options(usage, options);\n\nShould we give a better error in this case?\n\n> +\tif (!revs.diffopt.output_format)\n> +\t\trevs.diffopt.output_format = DIFF_FORMAT_RAW;\n> +\n> +\twhile (1) {\n> +\t\tstruct object_id oid_a, oid_b;\n> +\t\tstruct diff_filepair *pair;\n> +\t\tunsigned mode_a, mode_b;\n> +\t\tconst char *p;\n> +\t\tchar status;\n> +\n> +\t\tif (strbuf_getline_nul(&meta, stdin) == EOF)\n> +\t\t\tbreak;\n> +\n> +\t\tp = meta.buf;\n> +\t\tif (*p != ':')\n> +\t\t\tdie(\"invalid raw diff input\");\n> +\t\tp++;\n> +\n> +\t\tmode_a = parse_mode_or_die(p, &p);\n> +\t\tmode_b = parse_mode_or_die(p, &p);\n> +\n> +\t\tparse_oid(p, &oid_a, &p, repo->hash_algo);\n> +\t\tparse_oid(p, &oid_b, &p, repo->hash_algo);\n> +\n> +\t\tstatus = *p++;\n> +\n> +\t\tif (strbuf_getline_nul(&path, stdin) == EOF)\n> +\t\t\tdie(\"got EOF while reading path\");\n\nMissing translation.\n\n> +\t\tswitch (status) {\n> +\t\tcase DIFF_STATUS_ADDED:\n> +\t\t\tpair = diff_filepair_addremove(&revs.diffopt, '+',\n> +\t\t\t\t\t\t mode_b, &oid_b,\n> +\t\t\t\t\t\t 1, path.buf, 0);\n> +\t\t\tif (pair)\n> +\t\t\t\tpair->status = status;\n> +\t\t\tbreak;\n> +\n> +\t\tcase DIFF_STATUS_DELETED:\n> +\t\t\tpair = diff_filepair_addremove(&revs.diffopt, '-',\n> +\t\t\t\t\t\t mode_a, &oid_a,\n> +\t\t\t\t\t\t 1, path.buf, 0);\n> +\t\t\tif (pair)\n> +\t\t\t\tpair->status = status;\n> +\t\t\tbreak;\n> +\n> +\t\tcase DIFF_STATUS_TYPE_CHANGED:\n> +\t\tcase DIFF_STATUS_MODIFIED:\n> +\t\t\tpair = diff_filepair_change(&revs.diffopt,\n> +\t\t\t\t\t\t mode_a, mode_b,\n> +\t\t\t\t\t\t &oid_a, &oid_b, 1, 1,\n> +\t\t\t\t\t\t path.buf, 0, 0);\n> +\t\t\tif (pair)\n> +\t\t\t\tpair->status = status;\n> +\t\t\tbreak;\n> +\n> +\t\tcase DIFF_STATUS_RENAMED:\n> +\t\tcase DIFF_STATUS_COPIED:\n> +\t\t\t{\n> +\t\t\t\tstruct diff_filespec *a, *b;\n> +\n> +\t\t\t\tif (strbuf_getline_nul(&path_dst, stdin) == EOF)\n> +\t\t\t\t\tdie(\"got EOF while reading destination path\");\n\nMissing translation.\n\n> +\t\t\t\ta = alloc_filespec(path.buf);\n> +\t\t\t\tb = alloc_filespec(path_dst.buf);\n> +\t\t\t\tfill_filespec(a, &oid_a, 1, mode_a);\n> +\t\t\t\tfill_filespec(b, &oid_b, 1, mode_b);\n> +\n> +\t\t\t\tpair = diff_queue(&diff_queued_diff, a, b);\n> +\t\t\t\tpair->status = status;\n> +\t\t\t\tpair->score = parse_score(p);\n> +\t\t\t\tpair->renamed_pair = 1;\n> +\t\t\t}\n> +\t\t\tbreak;\n> +\n> +\t\tdefault:\n> +\t\t\tdie(\"unknown diff status: %c\", status);\n\nMissing translation.\n\n> diff --git a/t/t4070-diff-pairs.sh b/t/t4070-diff-pairs.sh\n> new file mode 100755\n> index 0000000000..e0a8e6f0a0\n> --- /dev/null\n> +++ b/t/t4070-diff-pairs.sh\n> @@ -0,0 +1,80 @@\n[snip]\n> +test_expect_success 'split input across multiple diff-pairs' '\n> +\twrite_script split-raw-diff \"$PERL_PATH\" <<-\\EOF &&\n> +\t$/ = \"\\0\";\n> +\twhile (<>) {\n> +\t my $meta = $_;\n> +\t my $path = <>;\n> +\t # renames have an extra path\n> +\t my $path2 = <> if $meta =~ /[RC]\\d+/;\n> +\n> +\t open(my $fh, \">\", sprintf \"diff%03d\", $.);\n> +\t print $fh $meta, $path, $path2;\n> +\t}\n> +\tEOF\n> +\n> +\tgit diff-tree -p -M -C -C base new >expect &&\n> +\n> +\tgit diff-tree -r -z -M -C -C base new |\n> +\t./split-raw-diff &&\n> +\tfor i in diff*; do\n> +\t\tgit diff-pairs -p <$i || return 1\n\nFormatting:\n\n\tfor i in diff*\n\tdo\n\t\tgit diff-pairs -p <$i ||\n\t\treturn 1\n\tdone >actual\n\n> +\tdone >actual &&\n> +\ttest_cmp expect actual\n> +'\n\nPatrick\n"},{"id":"512300","messageId":"Z6xopCvDXlX82lQ8@pks.im","threadId":"62633","inReplyTo":"20250212041825.2455031-4-jltobler@gmail.com","subject":"Re: [PATCH v2 3/3] builtin/diff-pairs: allow explicit diff queue flush","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-12T09:23:48Z","receivedAt":"2025-02-12T09:23:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 11, 2025 at 10:18:25PM -0600, Justin Tobler wrote:\n> The diffs queued from git-diff-pairs(1) stdin are not flushed EOF is\n\nI think you meant to say \"are flush when stdin is closed\" or something\nlike that.\n\n> reached. To enable greater flexibility, allow control over when the diff\n> queue is flushed by writing a single nul byte on stdin between input\n\ns/nul/NUL/\n\n> file pairs. Diff output between flushes is separated by a single line\n> terminator.\n> \n> Signed-off-by: Justin Tobler \n> ---\n> Documentation/git-diff-pairs.adoc | 4 ++++\n> builtin/diff-pairs.c | 11 +++++++++++\n> t/t4070-diff-pairs.sh | 22 ++++++++++++++++++++++\n> 3 files changed, 37 insertions(+)\n> \n> diff --git a/Documentation/git-diff-pairs.adoc b/Documentation/git-diff-pairs.adoc\n> index e9ef4a6615..33c0d702f0 100644\n> --- a/Documentation/git-diff-pairs.adoc\n> +++ b/Documentation/git-diff-pairs.adoc\n> @@ -32,6 +32,10 @@ compute diffs progressively over the course of multiple invocations of\n> Each blob pair is fed to the diff machinery individually queued and the output\n> is flushed on stdin EOF.\n> \n> +To explicitly flush the diff queue, a single nul byte can be written to stdin\n> +between filepairs. Diff output between flushes is separated by a single line\n> +terminator.\n\nThe same comment as for the previous patch applies here, I think we\nshould refrain from using jargon like \"flushing\", \"diff queue\" or\n\"filepairs\". These are internal implementation details that the user\nshouldn't need to worry about. Instead, we should be talking about the\nuser-visible effects.\n\n> diff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\n> index 08f3ee81e5..2436ce3013 100644\n> --- a/builtin/diff-pairs.c\n> +++ b/builtin/diff-pairs.c\n> @@ -99,6 +99,17 @@ int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n> \t\t\tbreak;\n> \n> \t\tp = meta.buf;\n> +\t\tif (!*p) {\n> +\t\t\tflush_diff_queue(&revs.diffopt);\n> +\t\t\t/*\n> +\t\t\t * When the diff queue is explicitly flushed, append an\n> +\t\t\t * additional terminator to separate batches of diffs.\n> +\t\t\t */\n> +\t\t\tfprintf(revs.diffopt.file, \"%c\",\n> +\t\t\t\trevs.diffopt.line_termination);\n\nYou can use `fputc(revs.diffopt.line_termination, revs.diffopt.file)`\ninstead.\n\n> diff --git a/t/t4070-diff-pairs.sh b/t/t4070-diff-pairs.sh\n> index e0a8e6f0a0..aca228a8fa 100755\n> --- a/t/t4070-diff-pairs.sh\n> +++ b/t/t4070-diff-pairs.sh\n> @@ -77,4 +77,26 @@ test_expect_success 'split input across multiple diff-pairs' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'diff-pairs explicit queue flush' '\n> +\tgit diff-tree -r -M -C -C -z base new >input &&\n> +\tprintf \"\\0\" >>input &&\n> +\tgit diff-tree -r -M -C -C -z base new >>input &&\n> +\n> +\tgit diff-tree -r -M -C -C base new >expect &&\n> +\tprintf \"\\n\" >>expect &&\n> +\tgit diff-tree -r -M -C -C base new >>expect &&\n> +\n> +\tgit diff-pairs actual &&\n> +\ttest_cmp expect actual\n> +'\n> +j\n> +test_expect_success 'diff-pairs explicit queue flush null terminated' '\n\ns/null/NUL\n\n> +\tgit diff-tree -r -M -C -C -z base new >expect &&\n> +\tprintf \"\\0\" >>expect &&\n> +\tgit diff-tree -r -M -C -C -z base new >>expect &&\n> +\n> +\tgit diff-pairs -z actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n\nPatrick\n"},{"id":"512301","messageId":"CAOLa=ZR+UFv5+=KH6R9f+JB1GstDNSiv+xohWJuWJAsDyH_DpA@mail.gmail.com","threadId":"62633","inReplyTo":"20250212041825.2455031-3-jltobler@gmail.com","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-02-12T09:51:38Z","receivedAt":"2025-02-12T09:51:40Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Justin Tobler writes:\n\n> Through git-diff(1), a single diff can be generated from a pair of blob\n> revisions directly. Unfortunately, there is not a mechanism to compute\n> batches of specific file pair diffs in a single process. Such a feature\n> is particularly useful on the server-side where diffing between a large\n> set of changes is not feasible all at once due to timeout concerns.\n>\n> To facilitate this, introduce git-diff-pairs(1) which takes the\n> null-terminated raw diff format as input on stdin and produces diffs in\n> other formats. As the raw diff format already contains the necessary\n> metadata, it becomes possible to progressively generate batches of diffs\n> without having to recompute rename detection or retrieve object context.\n> Something like the following:\n>\n> \tgit diff-tree -r -z -M $old $new |\n> \tgit diff-pairs -p\n>\n> should generate the same output as `git diff-tree -p -M`. Furthermore,\n> each line of raw diff formatted input can also be individually fed to a\n> separate git-diff-pairs(1) process and still produce the same output.\n>\n> Based-on-patch-by: Jeff King \n> Signed-off-by: Justin Tobler \n> ---\n> .gitignore | 1 +\n> Documentation/git-diff-pairs.adoc | 62 +++++++++++\n> Documentation/meson.build | 1 +\n> Makefile | 1 +\n> builtin.h | 1 +\n> builtin/diff-pairs.c | 178 ++++++++++++++++++++++++++++++\n> command-list.txt | 1 +\n> git.c | 1 +\n> meson.build | 1 +\n> t/meson.build | 1 +\n> t/t4070-diff-pairs.sh | 80 ++++++++++++++\n> 11 files changed, 328 insertions(+)\n> create mode 100644 Documentation/git-diff-pairs.adoc\n> create mode 100644 builtin/diff-pairs.c\n> create mode 100755 t/t4070-diff-pairs.sh\n>\n> diff --git a/.gitignore b/.gitignore\n> index e82aa19df0..03448c076a 100644\n> --- a/.gitignore\n> +++ b/.gitignore\n> @@ -54,6 +54,7 @@\n> /git-diff\n> /git-diff-files\n> /git-diff-index\n> +/git-diff-pairs\n> /git-diff-tree\n> /git-difftool\n> /git-difftool--helper\n> diff --git a/Documentation/git-diff-pairs.adoc b/Documentation/git-diff-pairs.adoc\n> new file mode 100644\n> index 0000000000..e9ef4a6615\n> --- /dev/null\n> +++ b/Documentation/git-diff-pairs.adoc\n> @@ -0,0 +1,62 @@\n> +git-diff-pairs(1)\n> +=================\n> +\n> +NAME\n> +----\n> +git-diff-pairs - Compare blob pairs generated by `diff-tree --raw`\n> +\n> +SYNOPSIS\n> +--------\n> +[verse]\n> +'git diff-pairs' [diff-options]\n> +\n> +DESCRIPTION\n> +-----------\n> +\n> +Given the output of `diff-tree -z` on its stdin, `diff-pairs` will\n> +reformat that output into whatever format is requested on its command\n> +line. For example:\n> +\n> +-----------------------------\n> +git diff-tree -z -M $a $b |\n> +git diff-pairs -p\n> +-----------------------------\n> +\n> +will compute the tree diff in one step (including renames), and then\n> +`diff-pairs` will compute and format the blob-level diffs for each pair.\n> +This can be used to modify the raw diff in the middle (without having to\n> +parse or re-create more complicated formats like `--patch`), or to\n> +compute diffs progressively over the course of multiple invocations of\n> +`diff-pairs`.\n> +\n> +Each blob pair is fed to the diff machinery individually queued and the output\n> +is flushed on stdin EOF.\n\nI found this hard to understand.\n\nAfter reading below, perhaps it would be easier to understand something\nsimpler which doesn't mention the internal queuing mechanism and only\ntalks about how the output is only steamed once we read EOF on stdin.\n\n> +\n> +OPTIONS\n> +-------\n> +\n> +include::diff-options.adoc[]\n> +\n> +include::diff-generate-patch.adoc[]\n> +\n> +NOTES\n> +----\n> +\n> +`diff-pairs` should handle any input generated by `diff-tree --raw -z`.\n> +It may choke or otherwise misbehave on output from `diff-files`, etc.\n> +\n> +Here's an incomplete list of things that `diff-pairs` could do, but\n> +doesn't (mostly in the name of simplicity):\n> +\n> + - Only `-z` input is accepted, not normal `--raw` input.\n> +\n> + - Abbreviated sha1s are rejected in the input from `diff-tree`; if you\n> + want to abbreviate the output, you can pass `--abbrev` to\n> + `diff-pairs`.\n> +\n> + - Pathspecs are not handled by `diff-pairs`; you can limit the diff via\n> + the initial `diff-tree` invocation.\n> +\n> +GIT\n> +---\n> +Part of the linkgit:git[1] suite\n> diff --git a/Documentation/meson.build b/Documentation/meson.build\n> index ead8e48213..e5ee177022 100644\n> --- a/Documentation/meson.build\n> +++ b/Documentation/meson.build\n> @@ -41,6 +41,7 @@ manpages = {\n> 'git-diagnose.adoc' : 1,\n> 'git-diff-files.adoc' : 1,\n> 'git-diff-index.adoc' : 1,\n> + 'git-diff-pairs.adoc' : 1,\n> 'git-difftool.adoc' : 1,\n> 'git-diff-tree.adoc' : 1,\n> 'git-diff.adoc' : 1,\n> diff --git a/Makefile b/Makefile\n> index 896d02339e..3b8e1ad15e 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1232,6 +1232,7 @@ BUILTIN_OBJS += builtin/describe.o\n> BUILTIN_OBJS += builtin/diagnose.o\n> BUILTIN_OBJS += builtin/diff-files.o\n> BUILTIN_OBJS += builtin/diff-index.o\n> +BUILTIN_OBJS += builtin/diff-pairs.o\n> BUILTIN_OBJS += builtin/diff-tree.o\n> BUILTIN_OBJS += builtin/diff.o\n> BUILTIN_OBJS += builtin/difftool.o\n> diff --git a/builtin.h b/builtin.h\n> index f7b166b334..b2d2e9eb07 100644\n> --- a/builtin.h\n> +++ b/builtin.h\n> @@ -152,6 +152,7 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix, struct reposit\n> int cmd_diff_files(int argc, const char **argv, const char *prefix, struct repository *repo);\n> int cmd_diff_index(int argc, const char **argv, const char *prefix, struct repository *repo);\n> int cmd_diff(int argc, const char **argv, const char *prefix, struct repository *repo);\n> +int cmd_diff_pairs(int argc, const char **argv, const char *prefix, struct repository *repo);\n> int cmd_diff_tree(int argc, const char **argv, const char *prefix, struct repository *repo);\n> int cmd_difftool(int argc, const char **argv, const char *prefix, struct repository *repo);\n> int cmd_env__helper(int argc, const char **argv, const char *prefix, struct repository *repo);\n> diff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\n> new file mode 100644\n> index 0000000000..08f3ee81e5\n> --- /dev/null\n> +++ b/builtin/diff-pairs.c\n> @@ -0,0 +1,178 @@\n> +#include \"builtin.h\"\n> +#include \"commit.h\"\n> +#include \"config.h\"\n> +#include \"diff.h\"\n> +#include \"diffcore.h\"\n> +#include \"gettext.h\"\n> +#include \"hex.h\"\n> +#include \"object.h\"\n> +#include \"parse-options.h\"\n> +#include \"revision.h\"\n> +#include \"strbuf.h\"\n> +\n> +static unsigned parse_mode_or_die(const char *mode, const char **endp)\n> +{\n> +\tuint16_t ret;\n> +\n> +\t*endp = parse_mode(mode, &ret);\n> +\tif (!*endp)\n> +\t\tdie(\"unable to parse mode: %s\", mode);\n> +\treturn ret;\n> +}\n> +\n> +static void parse_oid(const char *p, struct object_id *oid, const char **endp,\n> +\t\t const struct git_hash_algo *algop)\n\nNit: similar to the function above, should this be called\n`parse_oid_or_die`?\n\n> +{\n> +\tif (parse_oid_hex_algop(p, oid, endp, algop) || *(*endp)++ != ' ')\n> +\t\tdie(\"unable to parse object id: %s\", p);\n> +}\n> +\n> +static unsigned short parse_score(const char *score)\n> +{\n> +\tunsigned long ret;\n> +\tchar *endp;\n> +\n> +\terrno = 0;\n> +\tret = strtoul(score, &endp, 10);\n> +\tret *= MAX_SCORE / 100;\n> +\tif (errno || endp == score || *endp || (unsigned short)ret != ret)\n> +\t\tdie(\"unable to parse rename/copy score: %s\", score);\n> +\treturn ret;\n> +}\n> +\n> +static void flush_diff_queue(struct diff_options *options)\n> +{\n> +\t/*\n> +\t * If rename detection is not requested, use rename information from the\n> +\t * raw diff formatted input. Setting found_follow ensures diffcore_std()\n> +\t * does not mess with rename information already present in queued\n> +\t * filepairs.\n> +\t */\n> +\tif (!options->detect_rename)\n> +\t\toptions->found_follow = 1;\n> +\tdiffcore_std(options);\n> +\tdiff_flush(options);\n> +}\n> +\n> +int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n> +\t\t struct repository *repo)\n> +{\n> +\tstruct strbuf path_dst = STRBUF_INIT;\n> +\tstruct strbuf path = STRBUF_INIT;\n> +\tstruct strbuf meta = STRBUF_INIT;\n> +\tstruct rev_info revs;\n> +\tint ret;\n> +\n> +\tconst char * const usage[] = {\n> +\t\tN_(\"git diff-pairs [diff-options]\"),\n> +\t\tNULL\n> +\t};\n> +\tstruct option options[] = {\n> +\t\tOPT_END()\n> +\t};\n> +\n> +\tshow_usage_with_options_if_asked(argc, argv, usage, options);\n> +\n> +\trepo_nit_revisions(repo, &revs, prefix);\n> +\trepo_config(repo, git_diff_basic_config, NULL);\n> +\trevs.disable_stdin = 1;\n> +\trevs.abbrev = 0;\n> +\trevs.diff = 1;\n> +\n> +\targc = setup_revisions(argc, argv, &revs, NULL);\n> +\n> +\t/* Don't allow pathspecs at all. */\n> +\tif (revs.prune_data.nr)\n> +\t\tusage_with_options(usage, options);\n> +\n> +\tif (!revs.diffopt.output_format)\n> +\t\trevs.diffopt.output_format = DIFF_FORMAT_RAW;\n> +\n> +\twhile (1) {\n> +\t\tstruct object_id oid_a, oid_b;\n> +\t\tstruct diff_filepair *pair;\n> +\t\tunsigned mode_a, mode_b;\n> +\t\tconst char *p;\n> +\t\tchar status;\n> +\n> +\t\tif (strbuf_getline_nul(&meta, stdin) == EOF)\n> +\t\t\tbreak;\n> +\n> +\t\tp = meta.buf;\n> +\t\tif (*p != ':')\n> +\t\t\tdie(\"invalid raw diff input\");\n> +\t\tp++;\n> +\n> +\t\tmode_a = parse_mode_or_die(p, &p);\n> +\t\tmode_b = parse_mode_or_die(p, &p);\n> +\n> +\t\tparse_oid(p, &oid_a, &p, repo->hash_algo);\n> +\t\tparse_oid(p, &oid_b, &p, repo->hash_algo);\n> +\n> +\t\tstatus = *p++;\n> +\n> +\t\tif (strbuf_getline_nul(&path, stdin) == EOF)\n> +\t\t\tdie(\"got EOF while reading path\");\n> +\n> +\t\tswitch (status) {\n> +\t\tcase DIFF_STATUS_ADDED:\n> +\t\t\tpair = diff_filepair_addremove(&revs.diffopt, '+',\n> +\t\t\t\t\t\t mode_b, &oid_b,\n> +\t\t\t\t\t\t 1, path.buf, 0);\n> +\t\t\tif (pair)\n> +\t\t\t\tpair->status = status;\n> +\t\t\tbreak;\n> +\n> +\t\tcase DIFF_STATUS_DELETED:\n> +\t\t\tpair = diff_filepair_addremove(&revs.diffopt, '-',\n> +\t\t\t\t\t\t mode_a, &oid_a,\n> +\t\t\t\t\t\t 1, path.buf, 0);\n> +\t\t\tif (pair)\n> +\t\t\t\tpair->status = status;\n> +\t\t\tbreak;\n> +\n> +\t\tcase DIFF_STATUS_TYPE_CHANGED:\n> +\t\tcase DIFF_STATUS_MODIFIED:\n> +\t\t\tpair = diff_filepair_change(&revs.diffopt,\n> +\t\t\t\t\t\t mode_a, mode_b,\n> +\t\t\t\t\t\t &oid_a, &oid_b, 1, 1,\n> +\t\t\t\t\t\t path.buf, 0, 0);\n> +\t\t\tif (pair)\n> +\t\t\t\tpair->status = status;\n> +\t\t\tbreak;\n> +\n> +\t\tcase DIFF_STATUS_RENAMED:\n> +\t\tcase DIFF_STATUS_COPIED:\n> +\t\t\t{\n> +\t\t\t\tstruct diff_filespec *a, *b;\n> +\n> +\t\t\t\tif (strbuf_getline_nul(&path_dst, stdin) == EOF)\n> +\t\t\t\t\tdie(\"got EOF while reading destination path\");\n> +\n> +\t\t\t\ta = alloc_filespec(path.buf);\n> +\t\t\t\tb = alloc_filespec(path_dst.buf);\n> +\t\t\t\tfill_filespec(a, &oid_a, 1, mode_a);\n> +\t\t\t\tfill_filespec(b, &oid_b, 1, mode_b);\n> +\n> +\t\t\t\tpair = diff_queue(&diff_queued_diff, a, b);\n> +\t\t\t\tpair->status = status;\n> +\t\t\t\tpair->score = parse_score(p);\n> +\t\t\t\tpair->renamed_pair = 1;\n> +\t\t\t}\n> +\t\t\tbreak;\n> +\n> +\t\tdefault:\n\nThe only state I think is missing is `DIFF_STATUS_UNMERGED` (from\n'diff.h'). Is that a state we need to handle?\n\n> +\t\t\tdie(\"unknown diff status: %c\", status);\n> +\t\t}\n> +\t}\n> +\n> +\tflush_diff_queue(&revs.diffopt);\n\nNow I understand what you meant by queuing the diffs.\n\n> +\tret = diff_result_code(&revs);\n> +\n> +\tstrbuf_release(&path_dst);\n> +\tstrbuf_release(&path);\n> +\tstrbuf_release(&meta);\n> +\trelease_revisions(&revs);\n> +\n> +\treturn ret;\n> +}\n> diff --git a/command-list.txt b/command-list.txt\n> index e0bb87b3b5..bb8acd51d8 100644\n> --- a/command-list.txt\n> +++ b/command-list.txt\n> @@ -95,6 +95,7 @@ git-diagnose ancillaryinterrogators\n> git-diff mainporcelain info\n> git-diff-files plumbinginterrogators\n> git-diff-index plumbinginterrogators\n> +git-diff-pairs plumbinginterrogators\n> git-diff-tree plumbinginterrogators\n> git-difftool ancillaryinterrogators complete\n> git-fast-export ancillarymanipulators\n> diff --git a/git.c b/git.c\n> index b23761480f..12bba872bb 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -540,6 +540,7 @@ static struct cmd_struct commands[] = {\n> \t{ \"diff\", cmd_diff, NO_PARSEOPT },\n> \t{ \"diff-files\", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n> \t{ \"diff-index\", cmd_diff_index, RUN_SETUP | NO_PARSEOPT },\n> +\t{ \"diff-pairs\", cmd_diff_pairs, RUN_SETUP | NO_PARSEOPT },\n> \t{ \"diff-tree\", cmd_diff_tree, RUN_SETUP | NO_PARSEOPT },\n> \t{ \"difftool\", cmd_difftool, RUN_SETUP_GENTLY },\n> \t{ \"fast-export\", cmd_fast_export, RUN_SETUP },\n> diff --git a/meson.build b/meson.build\n> index fbb8105d96..66ce3326e8 100644\n> --- a/meson.build\n> +++ b/meson.build\n> @@ -537,6 +537,7 @@ builtin_sources = [\n> 'builtin/diagnose.c',\n> 'builtin/diff-files.c',\n> 'builtin/diff-index.c',\n> + 'builtin/diff-pairs.c',\n> 'builtin/diff-tree.c',\n> 'builtin/diff.c',\n> 'builtin/difftool.c',\n> diff --git a/t/meson.build b/t/meson.build\n> index 4574280590..7ff17c6d29 100644\n> --- a/t/meson.build\n> +++ b/t/meson.build\n> @@ -500,6 +500,7 @@ integration_tests = [\n> 't4067-diff-partial-clone.sh',\n> 't4068-diff-symmetric-merge-base.sh',\n> 't4069-remerge-diff.sh',\n> + 't4070-diff-pairs.sh',\n> 't4100-apply-stat.sh',\n> 't4101-apply-nonl.sh',\n> 't4102-apply-rename.sh',\n> diff --git a/t/t4070-diff-pairs.sh b/t/t4070-diff-pairs.sh\n> new file mode 100755\n> index 0000000000..e0a8e6f0a0\n> --- /dev/null\n> +++ b/t/t4070-diff-pairs.sh\n> @@ -0,0 +1,80 @@\n> +#!/bin/sh\n> +\n> +test_description='basic diff-pairs tests'\n> +. ./test-lib.sh\n> +\n> +# This creates a diff with added, modified, deleted, renamed, copied, and\n> +# typechange entries. That includes one in a subdirectory for non-recursive\n> +# tests, and both exact and inexact similarity scores.\n> +test_expect_success 'create commit with various diffs' '\n\nGenerally, tests for setup are named 'setup' so we can do something\nlike:\n sh ./t0050-filesystem.sh --run=setup,9-11\n\nCan we renmae this to 'setup'?\n\n> +\techo to-be-gone >deleted &&\n> +\techo original >modified &&\n> +\techo now-a-file >symlink &&\n> +\ttest_seq 200 >two-hundred &&\n> +\ttest_seq 201 500 >five-hundred &&\n> +\tgit add . &&\n> +\ttest_tick &&\n> +\tgit commit -m base &&\n> +\tgit tag base &&\n> +\n> +\techo now-here >added &&\n> +\techo new >modified &&\n> +\trm deleted &&\n> +\tmkdir subdir &&\n> +\techo content >subdir/file &&\n> +\tmv two-hundred renamed &&\n> +\ttest_seq 201 500 | sed s/300/modified/ >copied &&\n> +\trm symlink &&\n> +\tgit add -A . &&\n> +\ttest_ln_s_add dest symlink &&\n> +\ttest_tick &&\n> +\tgit commit -m new &&\n> +\tgit tag new\n> +'\n> +\n> +test_expect_success 'diff-pairs recreates --raw' '\n> +\tgit diff-tree -r -M -C -C base new >expect &&\n> +\tgit diff-tree -r -M -C -C -z base new |\n> +\tgit diff-pairs >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'diff-pairs can create -p output' '\n> +\tgit diff-tree -p -M -C -C base new >expect &&\n> +\tgit diff-tree -r -M -C -C -z base new |\n> +\tgit diff-pairs -p >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'non-recursive --raw retains tree entry' '\n> +\tgit diff-tree base new >expect &&\n> +\tgit diff-tree -z base new |\n> +\tgit diff-pairs >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'split input across multiple diff-pairs' '\n> +\twrite_script split-raw-diff \"$PERL_PATH\" <<-\\EOF &&\n> +\t$/ = \"\\0\";\n> +\twhile (<>) {\n> +\t my $meta = $_;\n> +\t my $path = <>;\n> +\t # renames have an extra path\n> +\t my $path2 = <> if $meta =~ /[RC]\\d+/;\n> +\n> +\t open(my $fh, \">\", sprintf \"diff%03d\", $.);\n> +\t print $fh $meta, $path, $path2;\n> +\t}\n> +\tEOF\n> +\n> +\tgit diff-tree -p -M -C -C base new >expect &&\n> +\n> +\tgit diff-tree -r -z -M -C -C base new |\n> +\t./split-raw-diff &&\n> +\tfor i in diff*; do\n> +\t\tgit diff-pairs -p <$i || return 1\n> +\tdone >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_done\n> --\n> 2.48.1\n"},{"id":"512312","messageId":"24dadfc4-6831-46df-b40f-98842d2865db@free.fr","threadId":"62633","inReplyTo":"20250212041825.2455031-3-jltobler@gmail.com","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Jean-Noël Avila","fromEmail":"jn.avila@free.fr","sentAt":"2025-02-12T11:40:11Z","receivedAt":"2025-02-12T11:47:10Z","isPatch":true,"sender":{"key":"jn.avila@free.fr","avatar":"https://avatars.githubusercontent.com/u/156172?v=4"},"body":"Le 12/02/2025 à 05:18, Justin Tobler a écrit :\n\n>\n> +\n> +SYNOPSIS\n> +--------\n> +[verse]\n> +'git diff-pairs' [diff-options]\n> +\n\nThis should read:\n\n[synopsis]\ngit-diff-pairs []\n\n\n\n> +DESCRIPTION\n> +-----------\n> +\n> +Given the output of `diff-tree -z` on its stdin, `diff-pairs` will\n\nPlease do not use the future form when describing the actual behavior.\n\n> +reformat that output into whatever format is requested on its command\n> +line. For example:\n> +\n> +-----------------------------\n> +git diff-tree -z -M $a $b |\n> +git diff-pairs -p\n> +-----------------------------\n> +\n> +will compute the tree diff in one step (including renames), and then\n> +`diff-pairs` will compute and format the blob-level diffs for each pair.\n> +This can be used to modify the raw diff in the middle (without having to\n> +parse or re-create more complicated formats like `--patch`), or to\n> +compute diffs progressively over the course of multiple invocations of\n> +`diff-pairs`.\n> +\n\nJN\n"},{"id":"512317","messageId":"xmqqldubumxf.fsf@gitster.g","threadId":"62633","inReplyTo":"20250212041825.2455031-3-jltobler@gmail.com","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-12T16:50:52Z","receivedAt":"2025-02-12T16:50:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Justin Tobler writes:\n\n> +NOTES\n> +----\n> +\n> +`diff-pairs` should handle any input generated by `diff-tree --raw -z`.\n> +It may choke or otherwise misbehave on output from `diff-files`, etc.\n> +\n> +Here's an incomplete list of things that `diff-pairs` could do, but\n> +doesn't (mostly in the name of simplicity):\n> +\n> + - Only `-z` input is accepted, not normal `--raw` input.\n> +\n> + - Abbreviated sha1s are rejected in the input from `diff-tree`; if you\n> + want to abbreviate the output, you can pass `--abbrev` to\n> + `diff-pairs`.\n> +\n> + - Pathspecs are not handled by `diff-pairs`; you can limit the diff via\n> + the initial `diff-tree` invocation.\n\nWhich of the above limitations are fundamental, and which are merely\ndue to incomplete implementation that could be improved in the\nfuture iterations? Without reading the code deeply, a lot of them\nlook like merely due to this iteration being at a WIP state and not\nquite ready for the general public.\n\nWhat is especially curious is the reason why it is limited to\ndiff-tree (by the way, don't you require '-r' if you are fed\n'diff-tree' output, or are you prepared to expand tree objects in\nthe input yourself?).\n\nI can guess that the 0{40} object names in the postimage to signal\npaths with working tree changes unadded to the index is something\nthis fundamentally cannot work with, but you should be able to grok\n'diff-index --cached', which does not have that issue, just fine.\n\n> diff --git a/Documentation/meson.build b/Documentation/meson.build\n> index ead8e48213..e5ee177022 100644\n> --- a/Documentation/meson.build\n> +++ b/Documentation/meson.build\n> @@ -41,6 +41,7 @@ manpages = {\n> 'git-diagnose.adoc' : 1,\n> 'git-diff-files.adoc' : 1,\n> 'git-diff-index.adoc' : 1,\n> + 'git-diff-pairs.adoc' : 1,\n> 'git-difftool.adoc' : 1,\n> 'git-diff-tree.adoc' : 1,\n> 'git-diff.adoc' : 1,\n\nThis apparently does not apply to 'master' and the base at least\nneeds to contain 1f010d6b (doc: use .adoc extension for AsciiDoc\nfiles, 2025-01-20). Please clearly mark the series as such in the\ncover letter if the series is not built on top of recent 'master'\n(or 'maint' if it is a series to fix breakage, but it does not apply\nto this series).\n\nThanks.\n"},{"id":"512321","messageId":"3vqcdil4qdam2pcb3mjruuda2kgph3arzhwswnlugtqykmc6zm@7jiwkyqty5lj","threadId":"62633","inReplyTo":"Z6xomFk_Sb18UMFo@pks.im","subject":"Re: [PATCH v2 1/3] diff: return diff_filepair from diff queue helpers","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-12T17:24:55Z","receivedAt":"2025-02-12T17:28:00Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/12 10:23AM, Patrick Steinhardt wrote:\n> On Tue, Feb 11, 2025 at 10:18:23PM -0600, Justin Tobler wrote:\n> > The `diff_addremove()` and `diff_change()` functions setup and queue\n> > diffs, but do not return the `diff_filepair` added to the queue. In a\n> > subsequent commit, modifications to `diff_filepair` need to take place\n> > in certain cases after being queued.\n> > \n> > Split out the queuing operations into `diff_filepair_addremove()` and\n> > `diff_filepair_change()` which also return a handle to the queued\n> > `diff_filepair`.\n> \n> One of the things that puzzled me a bit is that we keep the old-style\n> functions, where the only difference is the return value. Wouldn't it\n> make more sense to instead adapt these existing functions to reduce the\n> amount of duplication?\n\nThis is what I considered doing initially. I noticed though that both\n`diff_addremove()` and `diff_change()` are stored as callbacks in\n`diff_options` as types `add_remove_fn_t` and `change_fn_t`. The diff\noptions configured for pruning use `file_add_remove()` and\n`file_change()` instead. Returning `diff_filepair` doesn't seems to make\nmuch sense in the context of `file_add_remove()` and `file_change()` as\nno filepairs ever get queued, so I opted to factor out the logic into\nseparate functions instead of adapting the function signatures for all.\n\nThis may not be the best option, so I can also change it if that is\nbest.\n\n> At the same time, while we're already at it, do we maybe also want to\n> adapt the functions so that they get the `diff_queue` as input instead\n> of relying on the global queue? That would make them more generally\n> useful and be a step into the right direction regarding libification. If\n> so, it would indeed make sense to also rename the function into e.g.\n> `diff_queue_addremove()`.\n\nThanks for the suggestion. I'll adapt the next version accordingly.\n\n-Justin\n"},{"id":"512323","messageId":"6we74f5rgmhgu2gfv3x5qi6xdyad3erdgcjaxt6tb4czxbko45@b6b6iq6uo445","threadId":"62633","inReplyTo":"CAOLa=ZRssYP8U+kyTYdUY_-WGCLLk4mVxQRbgu87g7AdxPJbZw@mail.gmail.com","subject":"Re: [PATCH v2 1/3] diff: return diff_filepair from diff queue helpers","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-12T17:35:18Z","receivedAt":"2025-02-12T17:38:23Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/12 01:06AM, Karthik Nayak wrote:\n> Justin Tobler writes:\n> \n> > The `diff_addremove()` and `diff_change()` functions setup and queue\n> > diffs, but do not return the `diff_filepair` added to the queue. In a\n> > subsequent commit, modifications to `diff_filepair` need to take place\n> > in certain cases after being queued.\n> >\n> > Split out the queuing operations into `diff_filepair_addremove()` and\n> > `diff_filepair_change()` which also return a handle to the queued\n> > `diff_filepair`.\n> >\n> \n> This patch keeps `diff_addremove()` and `diff_change()` while\n> introducing two new functions which return the `diff_filepair`. Just a\n> thought, why not replace them? The users `diff_addremove()` and\n> `diff_change()` could simply call the new functions and ignore the\n> return value?\n\nThis was mostly to avoid changing the `add_remove_fn_t` and\n`change_fn_t` types that store `diff_addremove()` and `diff_change()` in\n`diff_options`. The `file_add_remove()` and `file_change()` functions,\nwhich also can be set in `diff_options`, do not ever queue file pairs so\nI don't think returning `diff_filepair` makes much sense there.\n\n> This would be messy if there were a lot of users of `diff_addremove()`\n> and `diff_change()`, but I only see a few callers. Wouldn't it be\n> cleaner to just replace?\n\nPatrick has suggested we avoid using the global `diff_queue_struct`\nimplicitly. Currently, in the next version I'm planning to keep the\nseparate functions as `diff_queue_addremove()` and\n`diff_queue_change()`, but also accept `diff_queue_struct` as an\nargument.\n\n> The patch looks good to me otherwise.\n> \n> [snip]\n> \n> > diff --git a/diff.h b/diff.h\n> > index 0a566f5531..6ea63f01e7 100644\n> > --- a/diff.h\n> > +++ b/diff.h\n> > @@ -508,6 +508,21 @@ void diff_set_default_prefix(struct diff_options *options);\n> >\n> > int diff_can_quit_early(struct diff_options *);\n> >\n> > +struct diff_filepair *diff_filepair_addremove(struct diff_options *,\n> > +\t\t\t\t\t int addremove, unsigned mode,\n> > +\t\t\t\t\t const struct object_id *oid,\n> > +\t\t\t\t\t int oid_valid, const char *fullpath,\n> > +\t\t\t\t\t unsigned dirty_submodule);\n> > +\n> > +struct diff_filepair *diff_filepair_change(struct diff_options *,\n> > +\t\t\t\t\t unsigned mode1, unsigned mode2,\n> > +\t\t\t\t\t const struct object_id *old_oid,\n> > +\t\t\t\t\t const struct object_id *new_oid,\n> > +\t\t\t\t\t int old_oid_valid, int new_oid_valid,\n> > +\t\t\t\t\t const char *fullpath,\n> > +\t\t\t\t\t unsigned dirty_submodule1,\n> > +\t\t\t\t\t unsigned dirty_submodule2);\n> > +\n> \n> Nit: would be nice to have some comments to describe what these\n> functions do.\n\nI'll add in the next version. Thanks\n\n-Justin\n"},{"id":"512343","messageId":"Z62HDg0j2qkXy0u4@pks.im","threadId":"62633","inReplyTo":"3vqcdil4qdam2pcb3mjruuda2kgph3arzhwswnlugtqykmc6zm@7jiwkyqty5lj","subject":"Re: [PATCH v2 1/3] diff: return diff_filepair from diff queue helpers","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-13T05:45:50Z","receivedAt":"2025-02-13T05:45:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Feb 12, 2025 at 11:24:55AM -0600, Justin Tobler wrote:\n> On 25/02/12 10:23AM, Patrick Steinhardt wrote:\n> > On Tue, Feb 11, 2025 at 10:18:23PM -0600, Justin Tobler wrote:\n> > > The `diff_addremove()` and `diff_change()` functions setup and queue\n> > > diffs, but do not return the `diff_filepair` added to the queue. In a\n> > > subsequent commit, modifications to `diff_filepair` need to take place\n> > > in certain cases after being queued.\n> > > \n> > > Split out the queuing operations into `diff_filepair_addremove()` and\n> > > `diff_filepair_change()` which also return a handle to the queued\n> > > `diff_filepair`.\n> > \n> > One of the things that puzzled me a bit is that we keep the old-style\n> > functions, where the only difference is the return value. Wouldn't it\n> > make more sense to instead adapt these existing functions to reduce the\n> > amount of duplication?\n> \n> This is what I considered doing initially. I noticed though that both\n> `diff_addremove()` and `diff_change()` are stored as callbacks in\n> `diff_options` as types `add_remove_fn_t` and `change_fn_t`. The diff\n> options configured for pruning use `file_add_remove()` and\n> `file_change()` instead. Returning `diff_filepair` doesn't seems to make\n> much sense in the context of `file_add_remove()` and `file_change()` as\n> no filepairs ever get queued, so I opted to factor out the logic into\n> separate functions instead of adapting the function signatures for all.\n> \n> This may not be the best option, so I can also change it if that is\n> best.\n\nOkay. This is context that should probably be part of the commit\nmessage, as it is quite an important detail to understand the\nimplementation.\n\nPatrick\n"},{"id":"512514","messageId":"d6d4230e-7b80-4eec-b218-37717ae2e298@gmail.com","threadId":"62633","inReplyTo":"20250212041825.2455031-3-jltobler@gmail.com","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2025-02-17T14:38:11Z","receivedAt":"2025-02-17T14:38:14Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Justin\n\nOn 12/02/2025 04:18, Justin Tobler wrote:\n> Through git-diff(1), a single diff can be generated from a pair of blob\n> revisions directly. Unfortunately, there is not a mechanism to compute\n> batches of specific file pair diffs in a single process. Such a feature\n> is particularly useful on the server-side where diffing between a large\n> set of changes is not feasible all at once due to timeout concerns.\n> \n> To facilitate this, introduce git-diff-pairs(1) which takes the\n> null-terminated raw diff format as input on stdin and produces diffs in\n> other formats. As the raw diff format already contains the necessary\n> metadata, it becomes possible to progressively generate batches of diffs\n> without having to recompute rename detection or retrieve object context.\n> Something like the following:\n> \n> \tgit diff-tree -r -z -M $old $new |\n> \tgit diff-pairs -p\n> \n> should generate the same output as `git diff-tree -p -M`. Furthermore,\n> each line of raw diff formatted input can also be individually fed to a\n> separate git-diff-pairs(1) process and still produce the same output.\n\nI like the idea of this, I've left a few comments mainly around the UI.\n\n > +Here's an incomplete list of things that `diff-pairs` could do, but\n > +doesn't (mostly in the name of simplicity):\n > +\n > + - Only `-z` input is accepted, not normal `--raw` input.\n\nI think only accepting NUL terminated input is fine, but if we want to \naccept other formats we should have a plan for how to do that in a \nbackwards compatible way as we cannot use `-z` to distinguish between \ninput formats.\n\n> +\tconst char * const usage[] = {\n> +\t\tN_(\"git diff-pairs [diff-options]\"),\n\nNormally the option summary printed by \"git foo -h\" is generated by the \noption parser. In this case we don't define any options and use \nsetup_revisions() instead so we need to provide the option summary \nourselves. Looking at diff-files.c we can add\n\n\t\"\\n\"\n\tCOMMON_DIFF_OPTIONS_HELP;\n\nto do that.\n\n> +\targc = setup_revisions(argc, argv, &revs, NULL);\n\nI think we should check that there are no options left on the \ncommandline after setup_revisions() returns\n\n> +\t/* Don't allow pathspecs at all. */\n> +\tif (revs.prune_data.nr)\n> +\t\tusage_with_options(usage, options);\n\nIt is not just pathspecs that we want to reject but all revision related \noptions. Looking at diff-files.c we can do\n\n\tif (rev.pending.nr ||\n\t rev.min_age != -1 || rev.max_age != -1 ||\n\t rev.max_count != -1)\n\t\tusage_with_option(usage, options);\n\nTo catch some of that but it still accepts things like \"--first-parent\", \n\"--merges\" and \"--ancestry-path\". We may just have to live with that as \nI don't think it is worth expanding a huge amount of effort to prevent them.\n\n> +\tif (!revs.diffopt.output_format)\n> +\t\trevs.diffopt.output_format = DIFF_FORMAT_RAW;\n\nThis matches the other diff plumbing commands but I'm not sure it is the \nmost helpful default for a command that is supposed to transform raw \ndiffs into another format. Maybe we should default to DIFF_FORMAT_PATCH?\n\n\n> +test_expect_success 'split input across multiple diff-pairs' '\n\nThis needs a PERL prerequisite I think. I'm a bit unsure what this test \nadds compared to the others.\n\nBest Wishes\n\nPhillip\n\n> +\twrite_script split-raw-diff \"$PERL_PATH\" <<-\\EOF &&\n> +\t$/ = \"\\0\";\n> +\twhile (<>) {\n> +\t my $meta = $_;\n> +\t my $path = <>;\n> +\t # renames have an extra path\n> +\t my $path2 = <> if $meta =~ /[RC]\\d+/;\n> +\n> +\t open(my $fh, \">\", sprintf \"diff%03d\", $.);\n> +\t print $fh $meta, $path, $path2;\n> +\t}\n> +\tEOF\n> +\n> +\tgit diff-tree -p -M -C -C base new >expect &&\n> +\n> +\tgit diff-tree -r -z -M -C -C base new |\n> +\t./split-raw-diff &&\n> +\tfor i in diff*; do\n> +\t\tgit diff-pairs -p <$i || return 1\n> +\tdone >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_done\n\n"},{"id":"512515","messageId":"30c13b14-945d-4984-bb49-9fd93a4dedc9@gmail.com","threadId":"62633","inReplyTo":"20250212041825.2455031-4-jltobler@gmail.com","subject":"Re: [PATCH v2 3/3] builtin/diff-pairs: allow explicit diff queue flush","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2025-02-17T14:38:15Z","receivedAt":"2025-02-17T14:38:18Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Justin\n\nOn 12/02/2025 04:18, Justin Tobler wrote:\n> The diffs queued from git-diff-pairs(1) stdin are not flushed EOF is\n> reached. To enable greater flexibility, allow control over when the diff\n> queue is flushed by writing a single nul byte on stdin between input\n> file pairs. Diff output between flushes is separated by a single line\n> terminator.\n\nI agree with the comments others have made about the documentation. I \nalso have some comments on the implementation below.\n\n> diff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\n> index 08f3ee81e5..2436ce3013 100644\n> --- a/builtin/diff-pairs.c\n> +++ b/builtin/diff-pairs.c\n> @@ -99,6 +99,17 @@ int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n> \t\t\tbreak;\n> \n> \t\tp = meta.buf;\n> +\t\tif (!*p) {\n> +\t\t\tflush_diff_queue(&revs.diffopt);\n> +\t\t\t/*\n> +\t\t\t * When the diff queue is explicitly flushed, append an\n> +\t\t\t * additional terminator to separate batches of diffs.\n> +\t\t\t */\n> +\t\t\tfprintf(revs.diffopt.file, \"%c\",\n> +\t\t\t\trevs.diffopt.line_termination);\n\nAs the user has requested an explicit flush we should call \nfflush(stdout) here to avoid deadlocking a caller that is waiting to \nread the terminator before writing the next batch of input. Ideally the \ntests would check that the output is flushed but I think that is quite \nhard to do with our test framework.\n\nI think it would be easier for callers to parse the output if we always \nprinted NUL here. Programming languages generally have a function that \nallows you to read all the input until a specific byte is seen. If \nflushing always used a NUL terminator the caller could use their \nequivalent of read_until(b'\\0') to hoover up the output (using '-z' to \ndo this would change the output of --numstat and embed a NUL between any \nstat data and the patch). Using a newline as the terminator here means \nthe caller needs to look for \"\\n\\n\". That string occurs in the output \nbetween the stat data and the patch and can also occur in the patch \nhunks if diff.suppressBlankEmpty is set.\n\nNow that we are calling diff_flush() in a loop we need to set .no_free \nin our diff options and call diff_free() at the end of the program (see \nthe comment in diff.h)\n\nBest Wishes\n\nPhillip\n\n\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> \t\tif (*p != ':')\n> \t\t\tdie(\"invalid raw diff input\");\n> \t\tp++;\n> diff --git a/t/t4070-diff-pairs.sh b/t/t4070-diff-pairs.sh\n> index e0a8e6f0a0..aca228a8fa 100755\n> --- a/t/t4070-diff-pairs.sh\n> +++ b/t/t4070-diff-pairs.sh\n> @@ -77,4 +77,26 @@ test_expect_success 'split input across multiple diff-pairs' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'diff-pairs explicit queue flush' '\n> +\tgit diff-tree -r -M -C -C -z base new >input &&\n> +\tprintf \"\\0\" >>input &&\n> +\tgit diff-tree -r -M -C -C -z base new >>input &&\n> +\n> +\tgit diff-tree -r -M -C -C base new >expect &&\n> +\tprintf \"\\n\" >>expect &&\n> +\tgit diff-tree -r -M -C -C base new >>expect &&\n> +\n> +\tgit diff-pairs actual &&\n> +\ttest_cmp expect actual\n> +'\n> +j\n> +test_expect_success 'diff-pairs explicit queue flush null terminated' '\n> +\tgit diff-tree -r -M -C -C -z base new >expect &&\n> +\tprintf \"\\0\" >>expect &&\n> +\tgit diff-tree -r -M -C -C -z base new >>expect &&\n> +\n> +\tgit diff-pairs -z actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> test_done\n\n"},{"id":"512743","messageId":"wv5ziveuff7iellcmjcki372m5vp6nmltyls43e4wzslcqymog@gwczuaucpkke","threadId":"62633","inReplyTo":"d6d4230e-7b80-4eec-b218-37717ae2e298@gmail.com","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-19T20:51:57Z","receivedAt":"2025-02-19T20:55:07Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/17 02:38PM, Phillip Wood wrote:\n> Hi Justin\n> \n> On 12/02/2025 04:18, Justin Tobler wrote:\n> > Through git-diff(1), a single diff can be generated from a pair of blob\n> > revisions directly. Unfortunately, there is not a mechanism to compute\n> > batches of specific file pair diffs in a single process. Such a feature\n> > is particularly useful on the server-side where diffing between a large\n> > set of changes is not feasible all at once due to timeout concerns.\n> > \n> > To facilitate this, introduce git-diff-pairs(1) which takes the\n> > null-terminated raw diff format as input on stdin and produces diffs in\n> > other formats. As the raw diff format already contains the necessary\n> > metadata, it becomes possible to progressively generate batches of diffs\n> > without having to recompute rename detection or retrieve object context.\n> > Something like the following:\n> > \n> > \tgit diff-tree -r -z -M $old $new |\n> > \tgit diff-pairs -p\n> > \n> > should generate the same output as `git diff-tree -p -M`. Furthermore,\n> > each line of raw diff formatted input can also be individually fed to a\n> > separate git-diff-pairs(1) process and still produce the same output.\n> \n> I like the idea of this, I've left a few comments mainly around the UI.\n> \n> > +Here's an incomplete list of things that `diff-pairs` could do, but\n> > +doesn't (mostly in the name of simplicity):\n> > +\n> > + - Only `-z` input is accepted, not normal `--raw` input.\n> \n> I think only accepting NUL terminated input is fine, but if we want to\n> accept other formats we should have a plan for how to do that in a\n> backwards compatible way as we cannot use `-z` to distinguish between input\n> formats.\n\nIf in the future we want to support the normal format, we could introduce\nan `--input-format=normal` option or something along those lines. \n\n> > +\tconst char * const usage[] = {\n> > +\t\tN_(\"git diff-pairs [diff-options]\"),\n> \n> Normally the option summary printed by \"git foo -h\" is generated by the\n> option parser. In this case we don't define any options and use\n> setup_revisions() instead so we need to provide the option summary\n> ourselves. Looking at diff-files.c we can add\n> \n> \t\"\\n\"\n> \tCOMMON_DIFF_OPTIONS_HELP;\n> \n> to do that.\n\nWould this be preferable even if git-diff-pairs doesn't support all of\nthe common diff options?\n\n> > +\targc = setup_revisions(argc, argv, &revs, NULL);\n> \n> I think we should check that there are no options left on the commandline\n> after setup_revisions() returns\n\nGood call, will do in the next version.\n\n> > +\t/* Don't allow pathspecs at all. */\n> > +\tif (revs.prune_data.nr)\n> > +\t\tusage_with_options(usage, options);\n> \n> It is not just pathspecs that we want to reject but all revision related\n> options. Looking at diff-files.c we can do\n> \n> \tif (rev.pending.nr ||\n> \t rev.min_age != -1 || rev.max_age != -1 ||\n> \t rev.max_count != -1)\n> \t\tusage_with_option(usage, options);\n> \n> To catch some of that but it still accepts things like \"--first-parent\",\n> \"--merges\" and \"--ancestry-path\". We may just have to live with that as I\n> don't think it is worth expanding a huge amount of effort to prevent them.\n\nYes, we should also reject revision as well as pathspec arguments. Will\nupdate.\n\n> > +\tif (!revs.diffopt.output_format)\n> > +\t\trevs.diffopt.output_format = DIFF_FORMAT_RAW;\n> \n> This matches the other diff plumbing commands but I'm not sure it is the\n> most helpful default for a command that is supposed to transform raw diffs\n> into another format. Maybe we should default to DIFF_FORMAT_PATCH?\n\nAs you mentioned, defaulting to DIFF_FORMAT_RAW isn't the most useful\nbehavior. I agree that it makes more sense to use DIFF_FORMAT_PATCH as\nthe default. Will update in the next version.\n\n> > +test_expect_success 'split input across multiple diff-pairs' '\n> \n> This needs a PERL prerequisite I think. I'm a bit unsure what this test adds\n> compared to the others.\n\nThis test demonstrates that the raw diff input can be split across\nseparate git-diff-pairs(1) processes and still produce equivilant\noutput which is one of the main usecases for the command. That being\nsaid, this test isn't really exercising different behavior of\ngit-diff-pairs(1) itself, so maybe it would be best to drop it.\n\nThanks for the review :)\n\n-Justin\n"},{"id":"512745","messageId":"xmqq7c5lfvh8.fsf@gitster.g","threadId":"62633","inReplyTo":"wv5ziveuff7iellcmjcki372m5vp6nmltyls43e4wzslcqymog@gwczuaucpkke","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-19T21:57:55Z","receivedAt":"2025-02-19T21:57:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Justin Tobler writes:\n\n>> I think only accepting NUL terminated input is fine, but if we want to\n>> accept other formats we should have a plan for how to do that in a\n>> backwards compatible way as we cannot use `-z` to distinguish between input\n>> formats.\n>\n> If in the future we want to support the normal format, we could introduce\n> an `--input-format=normal` option or something along those lines. \n\nPlease don't. Have an explicit '-z' option from the beginning, and\nif the initial version is incapable of reading from text input, then\nit is perfectly fine to have\n\n\tif (!nul_termination)\n\t\tdie(_(\"working without -z not supported (yet)\");\n\nOtherwise people have to remember that unlike everybody else that\nuses \"-z\" to signal NUL termination, this one alone wants to use a\n\"--input-format\" option that nobody else uses.\n\n>> > +\t/* Don't allow pathspecs at all. */\n>> > +\tif (revs.prune_data.nr)\n>> > +\t\tusage_with_options(usage, options);\n\nHmph, this is very unfortuate.\n\nThe \"--raw\" format was originally designed as an interchange format\nbetween the frontend and backend. \n\nThe frontend programs take two sets of contents stored in various\nplaces (like tree vs index, tree vs another tree) and express\ncomparison of corresponding paths in ( ) tuples\" (a rough equivalent to what we\ninternally have on the diff_queued_diff queue in core).\n\nThe \"--raw\" format was designed to \"dump\" what is in the\ndiff_queued_diff list.\n\nAnd then it would be passed to the single backend, that takes\n\"--raw\" format, pass them through the diffcore transform machinery\n(like matching removal and addition to detect renames), and produce\nvarious forms of output (like patch, diffstat, etc.).\n\nTo me, what you are writing is the output phase of that pipeline,\ni.e. the backend. We do want to (evantually) be able to filter with\npathspec, and all other things the current diff machinery does after\nthe existing \"all-in-one\" \"git diff\" and \"git diff-{files,index,tree}\"\ncommands do from their call to diffcore_std() and diffcore_flush().\n\nThe revisions option parsing machinery does accept options that\nwould *not* make sense to expect for them to make any difference to\nthe result of running \"diff\". Rejecting them is a nice thing to\nhave, e.g. \"git diff --no-merges HEAD^ HEAD\" does not error out, but\nsome people may want it to barf (I don't care---I am not sick enough\nto give apparently nonsense options to random commands), but it is\nperfectly fine to start your implementation with \"nonsense options\nmay be ignored\".\n\nBut in a \"git diff-* -z | git diff-pairs -z\" pipeline, I do not see\na particular reason why you would want to forbid the downstream\ncommand to further limit the paths it processes with its own\npathspec, e.g.\n\n git diff-tree -z --raw A B -- t/ | git diff-pairs -z t/helper/\n\nsounds like a perfectly sensible request to grant.\n\nMy recommendation is to avoid deciding to reject things your initial\nimplementation happens not to support (yet) too early. In the end,\nwe want this backend half just as powerful as, if not more than, the\nreal \"git diff\" machinery that has both front- and backend in the\nsame binary.\n\nThanks.\n"},{"id":"512746","messageId":"5uwp2vdm5tzv6n26fu77g4xys5ntjy2bj4hpgiuwpamxkij4zk@77jn55tynwse","threadId":"62633","inReplyTo":"xmqqldubumxf.fsf@gitster.g","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-19T22:19:44Z","receivedAt":"2025-02-19T22:22:54Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/12 08:50AM, Junio C Hamano wrote:\n> Justin Tobler writes:\n> \n> > +NOTES\n> > +----\n> > +\n> > +`diff-pairs` should handle any input generated by `diff-tree --raw -z`.\n> > +It may choke or otherwise misbehave on output from `diff-files`, etc.\n> > +\n> > +Here's an incomplete list of things that `diff-pairs` could do, but\n> > +doesn't (mostly in the name of simplicity):\n> > +\n> > + - Only `-z` input is accepted, not normal `--raw` input.\n> > +\n> > + - Abbreviated sha1s are rejected in the input from `diff-tree`; if you\n> > + want to abbreviate the output, you can pass `--abbrev` to\n> > + `diff-pairs`.\n> > +\n> > + - Pathspecs are not handled by `diff-pairs`; you can limit the diff via\n> > + the initial `diff-tree` invocation.\n> \n> Which of the above limitations are fundamental, and which are merely\n> due to incomplete implementation that could be improved in the\n> future iterations? \n\nThinking about this some more, I'm a bit unsure whether\ngit-diff-pairs(1) should support \"normal\" `--raw` input. Furthermore, if\nwe do want to support it, maybe it should be the default?\n\nFrom my perspective, ultimately I don't think there is much additional\nvalue provided by supporting multiple input options for\ngit-diff-pairs(1) since the end result would be the same and its just an\nintermediate format. As I see it, the benefit of the NUL delimited raw\ndiff ouput format is that it is a bit simpler to parse and likely a bit\nmore efficient as it wouldn't have to deal with unquoting paths with\nspecial characters. The benefit of the \"normal\" raw format is probably\nthat it is the more intuitive default option.\n\nI'm certainly interested in what folks think about this :)\n\nFor abbreviated object IDs, supporting them would make the input format\nmore flexible, but it would be simpler to just require the full OID be\nprovided thus making the input format more explicit. My current thinking\nis to leave this unless others think it would be useful to support.\n\nRegarding pathspec support, being that git-diff-pairs(1) operates solely\non the provided set of file pairs produced via some other Git operation,\nI don't think further limiting would provide much additional value\neither. If we do want this though, I think support could be added in the\nfuture.\n\n> Without reading the code deeply, a lot of them\n> look like merely due to this iteration being at a WIP state and not\n> quite ready for the general public.\n> \n> What is especially curious is the reason why it is limited to\n> diff-tree (by the way, don't you require '-r' if you are fed\n> 'diff-tree' output, or are you prepared to expand tree objects in\n> the input yourself?).\n\nThe tree objects in the input are not expanded. With `git diff-pairs\n--raw` these objects are just printed again. With the `--patch` option,\nthey are just ommitted.\n\n> I can guess that the 0{40} object names in the postimage to signal\n> paths with working tree changes unadded to the index is something\n> this fundamentally cannot work with, but you should be able to grok\n> 'diff-index --cached', which does not have that issue, just fine.\n\nI'll rework the documentation in the next version. git-diff-tree(1) is\nthe command I have in mind as the common usecase to use in combination\nwith git-diff-pairs(1), but it is not solely limited to it. As you\nmentioned, there are other commands that could be used to provide input\nhere.\n\n> > diff --git a/Documentation/meson.build b/Documentation/meson.build\n> > index ead8e48213..e5ee177022 100644\n> > --- a/Documentation/meson.build\n> > +++ b/Documentation/meson.build\n> > @@ -41,6 +41,7 @@ manpages = {\n> > 'git-diagnose.adoc' : 1,\n> > 'git-diff-files.adoc' : 1,\n> > 'git-diff-index.adoc' : 1,\n> > + 'git-diff-pairs.adoc' : 1,\n> > 'git-difftool.adoc' : 1,\n> > 'git-diff-tree.adoc' : 1,\n> > 'git-diff.adoc' : 1,\n> \n> This apparently does not apply to 'master' and the base at least\n> needs to contain 1f010d6b (doc: use .adoc extension for AsciiDoc\n> files, 2025-01-20). Please clearly mark the series as such in the\n> cover letter if the series is not built on top of recent 'master'\n> (or 'maint' if it is a series to fix breakage, but it does not apply\n> to this series).\n\nWill do\n\nThanks\n-Justin\n"},{"id":"512747","messageId":"yz74evgjbmj24cakoaseq53ief3jyonujd2pevm5exwcs3kdvl@aqhsafm4hmmm","threadId":"62633","inReplyTo":"xmqq7c5lfvh8.fsf@gitster.g","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-19T22:38:43Z","receivedAt":"2025-02-19T22:41:53Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/19 01:57PM, Junio C Hamano wrote:\n> Justin Tobler writes:\n> \n> >> I think only accepting NUL terminated input is fine, but if we want to\n> >> accept other formats we should have a plan for how to do that in a\n> >> backwards compatible way as we cannot use `-z` to distinguish between input\n> >> formats.\n> >\n> > If in the future we want to support the normal format, we could introduce\n> > an `--input-format=normal` option or something along those lines. \n> \n> Please don't. Have an explicit '-z' option from the beginning, and\n> if the initial version is incapable of reading from text input, then\n> it is perfectly fine to have\n> \n> \tif (!nul_termination)\n> \t\tdie(_(\"working without -z not supported (yet)\");\n> \n> Otherwise people have to remember that unlike everybody else that\n> uses \"-z\" to signal NUL termination, this one alone wants to use a\n> \"--input-format\" option that nobody else uses.\n\nThanks, I think this is a much better approach! :)\n\n> \n> >> > +\t/* Don't allow pathspecs at all. */\n> >> > +\tif (revs.prune_data.nr)\n> >> > +\t\tusage_with_options(usage, options);\n> \n> Hmph, this is very unfortuate.\n> \n> The \"--raw\" format was originally designed as an interchange format\n> between the frontend and backend. \n> \n> The frontend programs take two sets of contents stored in various\n> places (like tree vs index, tree vs another tree) and express\n> comparison of corresponding paths in ( mode+contents> ) tuples\" (a rough equivalent to what we\n> internally have on the diff_queued_diff queue in core).\n> \n> The \"--raw\" format was designed to \"dump\" what is in the\n> diff_queued_diff list.\n> \n> And then it would be passed to the single backend, that takes\n> \"--raw\" format, pass them through the diffcore transform machinery\n> (like matching removal and addition to detect renames), and produce\n> various forms of output (like patch, diffstat, etc.).\n> \n> To me, what you are writing is the output phase of that pipeline,\n> i.e. the backend. We do want to (evantually) be able to filter with\n> pathspec, and all other things the current diff machinery does after\n> the existing \"all-in-one\" \"git diff\" and \"git diff-{files,index,tree}\"\n> commands do from their call to diffcore_std() and diffcore_flush().\n> \n> The revisions option parsing machinery does accept options that\n> would *not* make sense to expect for them to make any difference to\n> the result of running \"diff\". Rejecting them is a nice thing to\n> have, e.g. \"git diff --no-merges HEAD^ HEAD\" does not error out, but\n> some people may want it to barf (I don't care---I am not sick enough\n> to give apparently nonsense options to random commands), but it is\n> perfectly fine to start your implementation with \"nonsense options\n> may be ignored\".\n> \n> But in a \"git diff-* -z | git diff-pairs -z\" pipeline, I do not see\n> a particular reason why you would want to forbid the downstream\n> command to further limit the paths it processes with its own\n> pathspec, e.g.\n> \n> git diff-tree -z --raw A B -- t/ | git diff-pairs -z t/helper/\n> \n> sounds like a perfectly sensible request to grant.\n> \n> My recommendation is to avoid deciding to reject things your initial\n> implementation happens not to support (yet) too early. In the end,\n> we want this backend half just as powerful as, if not more than, the\n> real \"git diff\" machinery that has both front- and backend in the\n> same binary.\n\nOk, that makes sense. I was originally thinking pathspec limiting could\njust be handled upstream, but it probably doesn't make much to\narbitrarily limit this functionality and remain more flexible. I'll do\nthis in the next version.\n\nThanks\n-Justin\n"},{"id":"512748","messageId":"ge7awzyjvgcqy4m3rixrgvizckksvc2zee37f6qpn5o43jkznp@wxur5rh5ecbt","threadId":"62633","inReplyTo":"30c13b14-945d-4984-bb49-9fd93a4dedc9@gmail.com","subject":"Re: [PATCH v2 3/3] builtin/diff-pairs: allow explicit diff queue flush","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-19T23:09:15Z","receivedAt":"2025-02-19T23:12:25Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/17 02:38PM, Phillip Wood wrote:\n> Hi Justin\n> \n> On 12/02/2025 04:18, Justin Tobler wrote:\n> > The diffs queued from git-diff-pairs(1) stdin are not flushed EOF is\n> > reached. To enable greater flexibility, allow control over when the diff\n> > queue is flushed by writing a single nul byte on stdin between input\n> > file pairs. Diff output between flushes is separated by a single line\n> > terminator.\n> \n> I agree with the comments others have made about the documentation. I also\n> have some comments on the implementation below.\n> \n> > diff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\n> > index 08f3ee81e5..2436ce3013 100644\n> > --- a/builtin/diff-pairs.c\n> > +++ b/builtin/diff-pairs.c\n> > @@ -99,6 +99,17 @@ int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n> > \t\t\tbreak;\n> > \t\tp = meta.buf;\n> > +\t\tif (!*p) {\n> > +\t\t\tflush_diff_queue(&revs.diffopt);\n> > +\t\t\t/*\n> > +\t\t\t * When the diff queue is explicitly flushed, append an\n> > +\t\t\t * additional terminator to separate batches of diffs.\n> > +\t\t\t */\n> > +\t\t\tfprintf(revs.diffopt.file, \"%c\",\n> > +\t\t\t\trevs.diffopt.line_termination);\n> \n> As the user has requested an explicit flush we should call fflush(stdout)\n> here to avoid deadlocking a caller that is waiting to read the terminator\n> before writing the next batch of input. Ideally the tests would check that\n> the output is flushed but I think that is quite hard to do with our test\n> framework.\n\nGood point, this needs to be explicitly flushed. Will fix.\n\n> I think it would be easier for callers to parse the output if we always\n> printed NUL here. Programming languages generally have a function that\n> allows you to read all the input until a specific byte is seen. If flushing\n> always used a NUL terminator the caller could use their equivalent of\n> read_until(b'\\0') to hoover up the output (using '-z' to do this would\n> change the output of --numstat and embed a NUL between any stat data and the\n> patch). Using a newline as the terminator here means the caller needs to\n> look for \"\\n\\n\". That string occurs in the output between the stat data and\n> the patch and can also occur in the patch hunks if diff.suppressBlankEmpty\n> is set.\n\nI was originally thinking that, without the -z option, a newline to\nindicate separation between queued diff batches would be more\nhuman-friendly. Always using a NUL byte would be more appropriate for\nparsing though. I'll switch to using only a NUL byte here in the next\nversion.\n\n> Now that we are calling diff_flush() in a loop we need to set .no_free in\n> our diff options and call diff_free() at the end of the program (see the\n> comment in diff.h)\n\nIndeed, will fix!\n\nThanks\n-Justin\n"},{"id":"512749","messageId":"xmqq1pvtfrq0.fsf@gitster.g","threadId":"62633","inReplyTo":"5uwp2vdm5tzv6n26fu77g4xys5ntjy2bj4hpgiuwpamxkij4zk@77jn55tynwse","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-19T23:19:03Z","receivedAt":"2025-02-19T23:19:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Justin Tobler writes:\n\n> Thinking about this some more, I'm a bit unsure whether\n> git-diff-pairs(1) should support \"normal\" `--raw` input. Furthermore, if\n> we do want to support it, maybe it should be the default?\n>\n> From my perspective, ultimately I don't think there is much additional\n> value provided by supporting multiple input options for\n> git-diff-pairs(1) since the end result would be the same and its just an\n> intermediate format. As I see it, the benefit of the NUL delimited raw\n> diff ouput format is that it is a bit simpler to parse and likely a bit\n> more efficient as it wouldn't have to deal with unquoting paths with\n> special characters. The benefit of the \"normal\" raw format is probably\n> that it is the more intuitive default option.\n>\n> I'm certainly interested in what folks think about this :)\n\nFWIW, in our toolset, \"-z\" is not the default primarily because text\nformat were chosen to help debuggability, which used to really matter\nin the early days.\n\n> For abbreviated object IDs, supporting them would make the input format\n> more flexible, but it would be simpler to just require the full OID be\n> provided thus making the input format more explicit. My current thinking\n> is to leave this unless others think it would be useful to support.\n\nAbbreviated object names would only be at \"might be nice to have\"\nlevel, I would think. We are talking about tools-to-tools\ncommunication after all.\n\n> Regarding pathspec support, being that git-diff-pairs(1) operates solely\n> on the provided set of file pairs produced via some other Git operation,\n> I don't think further limiting would provide much additional value\n> either. If we do want this though, I think support could be added in the\n> future.\n\nAnother consideration is which side of the pipeline should take the\nresponsibility to invoke the diffcore machinery. We certainly could\nmake it the job for the upstream/frontend, in which case diff-pairs\ndoes not have to call into diffcore-rename, BUT it also means the\ndownstream/backend needs to be able to parse two paths (renamed from\nand renamed to). Or we could make it the job for the downstream,\nand forbid the upstream/frontend from feeding renamed pairs (i.e.\nany input with status letter R or C are invalid), in which case\ndiff-pairs can choose to invoke rename detection or not by paying\nattention to the -M option and invoking diffcore_rename() itself\n(which should be at no-cost from coding point of view, as it should\nbe just the matter of calling diffcore_std()).\n\n> The tree objects in the input are not expanded. With `git diff-pairs\n> --raw` these objects are just printed again. With the `--patch` option,\n> they are just ommitted.\n\nInstead of getting expanded into its subpaths?\n\n>> This apparently does not apply to 'master' and the base at least\n>> needs to contain 1f010d6b (doc: use .adoc extension for AsciiDoc\n>> files, 2025-01-20). Please clearly mark the series as such in the\n>> cover letter if the series is not built on top of recent 'master'\n>> (or 'maint' if it is a series to fix breakage, but it does not apply\n>> to this series).\n>\n> Will do\n\nNo longer needed, as the tip of 'master' now lives in the .adoc\nworld. Hurrah!\n\n"},{"id":"512750","messageId":"xmqqwmdlebua.fsf@gitster.g","threadId":"62633","inReplyTo":"xmqq1pvtfrq0.fsf@gitster.g","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-19T23:47:25Z","receivedAt":"2025-02-19T23:47:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> Regarding pathspec support, being that git-diff-pairs(1) operates solely\n>> on the provided set of file pairs produced via some other Git operation,\n>> I don't think further limiting would provide much additional value\n>> either. If we do want this though, I think support could be added in the\n>> future.\n>\n> Another consideration is which side of the pipeline should take the\n> responsibility to invoke the diffcore machinery. We certainly could\n> make it the job for the upstream/frontend, in which case diff-pairs\n> does not have to call into diffcore-rename, BUT it also means the\n> downstream/backend needs to be able to parse two paths (renamed from\n> and renamed to). Or we could make it the job for the downstream,\n> and forbid the upstream/frontend from feeding renamed pairs (i.e.\n> any input with status letter R or C are invalid), in which case\n> diff-pairs can choose to invoke rename detection or not by paying\n> attention to the -M option and invoking diffcore_rename() itself\n> (which should be at no-cost from coding point of view, as it should\n> be just the matter of calling diffcore_std()).\n\nSorry, but I hit too early before finishing the most\nimportant part. We can move the features between upstream frontends\nand downstream diff-pairs. Depending on our goals, the best\ndivision of labor would be different. If we want to make it easy\nfor people to write their custom frontends, for example, it might\nmake sense to allow them to be as stupid and simple as possible and\nmake all the heavy lifting the responsibility of the diff-pairs\nbackend, which is the shared resource these frontends share and rely\non (so that they have incentive to help us make sure diff-pairs will\nstay correct and performant). If on the other hand we want to allow\npeople to do fancy processing in their custom frontends, maybe keeping\ndiff-pairs as stupid and transparent would be a better option to give\nthe people who write upstream/frontends more predictable behaviour.\n\nWhere to do the pathspec limiting is one of these things. You could\nmake it responsibility for the frontends if we assume that frontends\nmust do their own limiting. Or you could make it an optional feature\nof the backends, so that frontends that does not do its own limiting\ncan ask diff-pairs to limit. Which side to burden more really depends\non whose job we are trying to make it easier.\n\nThanks.\n\n"},{"id":"512752","messageId":"4ch3hdlv23qtwi32mgilym43eqtjefgxi4yi7577xcw6df3lbv@eiemmoap22y7","threadId":"62633","inReplyTo":"xmqqwmdlebua.fsf@gitster.g","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-20T00:32:59Z","receivedAt":"2025-02-20T00:36:09Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/19 03:47PM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> >> Regarding pathspec support, being that git-diff-pairs(1) operates solely\n> >> on the provided set of file pairs produced via some other Git operation,\n> >> I don't think further limiting would provide much additional value\n> >> either. If we do want this though, I think support could be added in the\n> >> future.\n> >\n> > Another consideration is which side of the pipeline should take the\n> > responsibility to invoke the diffcore machinery. We certainly could\n> > make it the job for the upstream/frontend, in which case diff-pairs\n> > does not have to call into diffcore-rename, BUT it also means the\n> > downstream/backend needs to be able to parse two paths (renamed from\n> > and renamed to). Or we could make it the job for the downstream,\n> > and forbid the upstream/frontend from feeding renamed pairs (i.e.\n> > any input with status letter R or C are invalid), in which case\n> > diff-pairs can choose to invoke rename detection or not by paying\n> > attention to the -M option and invoking diffcore_rename() itself\n> > (which should be at no-cost from coding point of view, as it should\n> > be just the matter of calling diffcore_std()).\n\nIn the current implementation, diff-pairs is capable of handling input\ncontaining rename/copy filepairs computed upstream. It does so by\nparsing the input line and manually setting the status, score, and paths\nfor the queued `diff_filepair`.\n\nI think diff-pairs should support rename and copy input as it would\nallow for rename/copy detection to be performed upfront in a single\npass by the upstream and the resulting output could be split up and fed\nto separate downstream diff-pairs. This is particularly useful for\nserver-side diffs to break up what would be large diffs.\n\n> Sorry, but I hit too early before finishing the most\n> important part. We can move the features between upstream frontends\n> and downstream diff-pairs. Depending on our goals, the best\n> division of labor would be different. If we want to make it easy\n> for people to write their custom frontends, for example, it might\n> make sense to allow them to be as stupid and simple as possible and\n> make all the heavy lifting the responsibility of the diff-pairs\n> backend, which is the shared resource these frontends share and rely\n> on (so that they have incentive to help us make sure diff-pairs will\n> stay correct and performant). If on the other hand we want to allow\n> people to do fancy processing in their custom frontends, maybe keeping\n> diff-pairs as stupid and transparent would be a better option to give\n> the people who write upstream/frontends more predictable behaviour.\n> \n> Where to do the pathspec limiting is one of these things. You could\n> make it responsibility for the frontends if we assume that frontends\n> must do their own limiting. Or you could make it an optional feature\n> of the backends, so that frontends that does not do its own limiting\n> can ask diff-pairs to limit. Which side to burden more really depends\n> on whose job we are trying to make it easier.\n\nFor the server-side diff usecase, I think that aligns more towards\nhaving a front-end that does more of the heavy lifting such rename/copy\ndetection and pathspec limiting, while the diff-pairs really just needs\nto compute the individual diffs for the already specified file pairs.\n\nI do see value though in keeping the door open for diff-pairs to become\nmore robust and flexible. Maybe it would be fine for now to say pathspec\nlimiting is not supported, but it could be in the future?\n\n> >> The tree objects in the input are not expanded. With `git diff-pairs\n> >> --raw` these objects are just printed again. With the `--patch` option,\n> >> they are just ommitted.\n> \n> >Instead of getting expanded into its subpaths?\n\nThe current implementation of diff-pairs is rather simple. It relies on\nthe upstream to feed it the file pairs with all the info upfront so it\ncan setup the diff queue. This means input with tree objects is also\nqueued as-is without being expanded further. I could maybe see a future\nthough where we want diff-pairs to be a more robust backend and supports\nexpanding these paths via -r option. Following previous discussion,\nmaybe it's fine to keep the initial implementation of diff-pairs on the\nsimple side for now. We could make diff-pairs die() for now if the -r\noption is explicitly set.\n\nThanks\n-Justin\n"},{"id":"512773","messageId":"3bf7llbgpb25bfiuqbsyz4iibacwnfgrumfede4nebz3vw2grs@xarsh4rzohle","threadId":"62633","inReplyTo":"4ch3hdlv23qtwi32mgilym43eqtjefgxi4yi7577xcw6df3lbv@eiemmoap22y7","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-20T14:56:02Z","receivedAt":"2025-02-20T14:59:13Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/19 06:32PM, Justin Tobler wrote:\n> > >> The tree objects in the input are not expanded. With `git diff-pairs\n> > >> --raw` these objects are just printed again. With the `--patch` option,\n> > >> they are just ommitted.\n> > \n> > >Instead of getting expanded into its subpaths?\n> \n> The current implementation of diff-pairs is rather simple. It relies on\n> the upstream to feed it the file pairs with all the info upfront so it\n> can setup the diff queue. This means input with tree objects is also\n> queued as-is without being expanded further. I could maybe see a future\n> though where we want diff-pairs to be a more robust backend and supports\n> expanding these paths via -r option. Following previous discussion,\n> maybe it's fine to keep the initial implementation of diff-pairs on the\n> simple side for now. We could make diff-pairs die() for now if the -r\n> option is explicitly set.\n\nThinking about this some more, adding support to expand trees in\ndiff-pairs would alter patch output behavior. To better enable backwards\ncompatible inclusion of this feature in the future, we may just want to\ndie() for now if any tree object is present in diff-pairs input.\n\n-Justin\n"},{"id":"512779","messageId":"xmqqr03sd257.fsf@gitster.g","threadId":"62633","inReplyTo":"3bf7llbgpb25bfiuqbsyz4iibacwnfgrumfede4nebz3vw2grs@xarsh4rzohle","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-20T16:14:28Z","receivedAt":"2025-02-20T16:14:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Justin Tobler writes:\n\n> Thinking about this some more, adding support to expand trees in\n> diff-pairs would alter patch output behavior. To better enable backwards\n> compatible inclusion of this feature in the future, we may just want to\n> die() for now if any tree object is present in diff-pairs input.\n\nSounds sensible.\n\nThanks.\n"},{"id":"513039","messageId":"4ymki6xozhiuzhbcgighwrw5rl5gt7r5yl4k5xpivb35yl3t67@6ko3l2knsc6q","threadId":"62633","inReplyTo":"CAOLa=ZR+UFv5+=KH6R9f+JB1GstDNSiv+xohWJuWJAsDyH_DpA@mail.gmail.com","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-25T23:38:57Z","receivedAt":"2025-02-25T23:42:12Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/12 01:51AM, Karthik Nayak wrote:\n> Justin Tobler writes:\n> \n> > Through git-diff(1), a single diff can be generated from a pair of blob\n> > revisions directly. Unfortunately, there is not a mechanism to compute\n> > batches of specific file pair diffs in a single process. Such a feature\n> > is particularly useful on the server-side where diffing between a large\n> > set of changes is not feasible all at once due to timeout concerns.\n> >\n> > To facilitate this, introduce git-diff-pairs(1) which takes the\n> > null-terminated raw diff format as input on stdin and produces diffs in\n> > other formats. As the raw diff format already contains the necessary\n> > metadata, it becomes possible to progressively generate batches of diffs\n> > without having to recompute rename detection or retrieve object context.\n> > Something like the following:\n> >\n> > \tgit diff-tree -r -z -M $old $new |\n> > \tgit diff-pairs -p\n> >\n> > should generate the same output as `git diff-tree -p -M`. Furthermore,\n> > each line of raw diff formatted input can also be individually fed to a\n> > separate git-diff-pairs(1) process and still produce the same output.\n> >\n> > Based-on-patch-by: Jeff King \n> > Signed-off-by: Justin Tobler \n> > ---\n> > .gitignore | 1 +\n> > Documentation/git-diff-pairs.adoc | 62 +++++++++++\n> > Documentation/meson.build | 1 +\n> > Makefile | 1 +\n> > builtin.h | 1 +\n> > builtin/diff-pairs.c | 178 ++++++++++++++++++++++++++++++\n> > command-list.txt | 1 +\n> > git.c | 1 +\n> > meson.build | 1 +\n> > t/meson.build | 1 +\n> > t/t4070-diff-pairs.sh | 80 ++++++++++++++\n> > 11 files changed, 328 insertions(+)\n> > create mode 100644 Documentation/git-diff-pairs.adoc\n> > create mode 100644 builtin/diff-pairs.c\n> > create mode 100755 t/t4070-diff-pairs.sh\n> >\n> > diff --git a/.gitignore b/.gitignore\n> > index e82aa19df0..03448c076a 100644\n> > --- a/.gitignore\n> > +++ b/.gitignore\n> > @@ -54,6 +54,7 @@\n> > /git-diff\n> > /git-diff-files\n> > /git-diff-index\n> > +/git-diff-pairs\n> > /git-diff-tree\n> > /git-difftool\n> > /git-difftool--helper\n> > diff --git a/Documentation/git-diff-pairs.adoc b/Documentation/git-diff-pairs.adoc\n> > new file mode 100644\n> > index 0000000000..e9ef4a6615\n> > --- /dev/null\n> > +++ b/Documentation/git-diff-pairs.adoc\n> > @@ -0,0 +1,62 @@\n> > +git-diff-pairs(1)\n> > +=================\n> > +\n> > +NAME\n> > +----\n> > +git-diff-pairs - Compare blob pairs generated by `diff-tree --raw`\n> > +\n> > +SYNOPSIS\n> > +--------\n> > +[verse]\n> > +'git diff-pairs' [diff-options]\n> > +\n> > +DESCRIPTION\n> > +-----------\n> > +\n> > +Given the output of `diff-tree -z` on its stdin, `diff-pairs` will\n> > +reformat that output into whatever format is requested on its command\n> > +line. For example:\n> > +\n> > +-----------------------------\n> > +git diff-tree -z -M $a $b |\n> > +git diff-pairs -p\n> > +-----------------------------\n> > +\n> > +will compute the tree diff in one step (including renames), and then\n> > +`diff-pairs` will compute and format the blob-level diffs for each pair.\n> > +This can be used to modify the raw diff in the middle (without having to\n> > +parse or re-create more complicated formats like `--patch`), or to\n> > +compute diffs progressively over the course of multiple invocations of\n> > +`diff-pairs`.\n> > +\n> > +Each blob pair is fed to the diff machinery individually queued and the output\n> > +is flushed on stdin EOF.\n> \n> I found this hard to understand.\n> \n> After reading below, perhaps it would be easier to understand something\n> simpler which doesn't mention the internal queuing mechanism and only\n> talks about how the output is only steamed once we read EOF on stdin.\n\nI've reworked the documentation in the next version to avoid discussing\ninternal details and stick to discussing user facing behavior.\n\n> > +\n> > +OPTIONS\n> > +-------\n> > +\n> > +include::diff-options.adoc[]\n> > +\n> > +include::diff-generate-patch.adoc[]\n> > +\n> > +NOTES\n> > +----\n> > +\n> > +`diff-pairs` should handle any input generated by `diff-tree --raw -z`.\n> > +It may choke or otherwise misbehave on output from `diff-files`, etc.\n> > +\n> > +Here's an incomplete list of things that `diff-pairs` could do, but\n> > +doesn't (mostly in the name of simplicity):\n> > +\n> > + - Only `-z` input is accepted, not normal `--raw` input.\n> > +\n> > + - Abbreviated sha1s are rejected in the input from `diff-tree`; if you\n> > + want to abbreviate the output, you can pass `--abbrev` to\n> > + `diff-pairs`.\n> > +\n> > + - Pathspecs are not handled by `diff-pairs`; you can limit the diff via\n> > + the initial `diff-tree` invocation.\n> > +\n> > +GIT\n> > +---\n> > +Part of the linkgit:git[1] suite\n> > diff --git a/Documentation/meson.build b/Documentation/meson.build\n> > index ead8e48213..e5ee177022 100644\n> > --- a/Documentation/meson.build\n> > +++ b/Documentation/meson.build\n> > @@ -41,6 +41,7 @@ manpages = {\n> > 'git-diagnose.adoc' : 1,\n> > 'git-diff-files.adoc' : 1,\n> > 'git-diff-index.adoc' : 1,\n> > + 'git-diff-pairs.adoc' : 1,\n> > 'git-difftool.adoc' : 1,\n> > 'git-diff-tree.adoc' : 1,\n> > 'git-diff.adoc' : 1,\n> > diff --git a/Makefile b/Makefile\n> > index 896d02339e..3b8e1ad15e 100644\n> > --- a/Makefile\n> > +++ b/Makefile\n> > @@ -1232,6 +1232,7 @@ BUILTIN_OBJS += builtin/describe.o\n> > BUILTIN_OBJS += builtin/diagnose.o\n> > BUILTIN_OBJS += builtin/diff-files.o\n> > BUILTIN_OBJS += builtin/diff-index.o\n> > +BUILTIN_OBJS += builtin/diff-pairs.o\n> > BUILTIN_OBJS += builtin/diff-tree.o\n> > BUILTIN_OBJS += builtin/diff.o\n> > BUILTIN_OBJS += builtin/difftool.o\n> > diff --git a/builtin.h b/builtin.h\n> > index f7b166b334..b2d2e9eb07 100644\n> > --- a/builtin.h\n> > +++ b/builtin.h\n> > @@ -152,6 +152,7 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix, struct reposit\n> > int cmd_diff_files(int argc, const char **argv, const char *prefix, struct repository *repo);\n> > int cmd_diff_index(int argc, const char **argv, const char *prefix, struct repository *repo);\n> > int cmd_diff(int argc, const char **argv, const char *prefix, struct repository *repo);\n> > +int cmd_diff_pairs(int argc, const char **argv, const char *prefix, struct repository *repo);\n> > int cmd_diff_tree(int argc, const char **argv, const char *prefix, struct repository *repo);\n> > int cmd_difftool(int argc, const char **argv, const char *prefix, struct repository *repo);\n> > int cmd_env__helper(int argc, const char **argv, const char *prefix, struct repository *repo);\n> > diff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\n> > new file mode 100644\n> > index 0000000000..08f3ee81e5\n> > --- /dev/null\n> > +++ b/builtin/diff-pairs.c\n> > @@ -0,0 +1,178 @@\n> > +#include \"builtin.h\"\n> > +#include \"commit.h\"\n> > +#include \"config.h\"\n> > +#include \"diff.h\"\n> > +#include \"diffcore.h\"\n> > +#include \"gettext.h\"\n> > +#include \"hex.h\"\n> > +#include \"object.h\"\n> > +#include \"parse-options.h\"\n> > +#include \"revision.h\"\n> > +#include \"strbuf.h\"\n> > +\n> > +static unsigned parse_mode_or_die(const char *mode, const char **endp)\n> > +{\n> > +\tuint16_t ret;\n> > +\n> > +\t*endp = parse_mode(mode, &ret);\n> > +\tif (!*endp)\n> > +\t\tdie(\"unable to parse mode: %s\", mode);\n> > +\treturn ret;\n> > +}\n> > +\n> > +static void parse_oid(const char *p, struct object_id *oid, const char **endp,\n> > +\t\t const struct git_hash_algo *algop)\n> \n> Nit: similar to the function above, should this be called\n> `parse_oid_or_die`?\n\nBeing consistent here is probably preferable, I've updated per your\nsuggestion in the next version.\n\n> > +{\n> > +\tif (parse_oid_hex_algop(p, oid, endp, algop) || *(*endp)++ != ' ')\n> > +\t\tdie(\"unable to parse object id: %s\", p);\n> > +}\n> > +\n> > +static unsigned short parse_score(const char *score)\n> > +{\n> > +\tunsigned long ret;\n> > +\tchar *endp;\n> > +\n> > +\terrno = 0;\n> > +\tret = strtoul(score, &endp, 10);\n> > +\tret *= MAX_SCORE / 100;\n> > +\tif (errno || endp == score || *endp || (unsigned short)ret != ret)\n> > +\t\tdie(\"unable to parse rename/copy score: %s\", score);\n> > +\treturn ret;\n> > +}\n> > +\n> > +static void flush_diff_queue(struct diff_options *options)\n> > +{\n> > +\t/*\n> > +\t * If rename detection is not requested, use rename information from the\n> > +\t * raw diff formatted input. Setting found_follow ensures diffcore_std()\n> > +\t * does not mess with rename information already present in queued\n> > +\t * filepairs.\n> > +\t */\n> > +\tif (!options->detect_rename)\n> > +\t\toptions->found_follow = 1;\n> > +\tdiffcore_std(options);\n> > +\tdiff_flush(options);\n> > +}\n> > +\n> > +int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n> > +\t\t struct repository *repo)\n> > +{\n> > +\tstruct strbuf path_dst = STRBUF_INIT;\n> > +\tstruct strbuf path = STRBUF_INIT;\n> > +\tstruct strbuf meta = STRBUF_INIT;\n> > +\tstruct rev_info revs;\n> > +\tint ret;\n> > +\n> > +\tconst char * const usage[] = {\n> > +\t\tN_(\"git diff-pairs [diff-options]\"),\n> > +\t\tNULL\n> > +\t};\n> > +\tstruct option options[] = {\n> > +\t\tOPT_END()\n> > +\t};\n> > +\n> > +\tshow_usage_with_options_if_asked(argc, argv, usage, options);\n> > +\n> > +\trepo_nit_revisions(repo, &revs, prefix);\n> > +\trepo_config(repo, git_diff_basic_config, NULL);\n> > +\trevs.disable_stdin = 1;\n> > +\trevs.abbrev = 0;\n> > +\trevs.diff = 1;\n> > +\n> > +\targc = setup_revisions(argc, argv, &revs, NULL);\n> > +\n> > +\t/* Don't allow pathspecs at all. */\n> > +\tif (revs.prune_data.nr)\n> > +\t\tusage_with_options(usage, options);\n> > +\n> > +\tif (!revs.diffopt.output_format)\n> > +\t\trevs.diffopt.output_format = DIFF_FORMAT_RAW;\n> > +\n> > +\twhile (1) {\n> > +\t\tstruct object_id oid_a, oid_b;\n> > +\t\tstruct diff_filepair *pair;\n> > +\t\tunsigned mode_a, mode_b;\n> > +\t\tconst char *p;\n> > +\t\tchar status;\n> > +\n> > +\t\tif (strbuf_getline_nul(&meta, stdin) == EOF)\n> > +\t\t\tbreak;\n> > +\n> > +\t\tp = meta.buf;\n> > +\t\tif (*p != ':')\n> > +\t\t\tdie(\"invalid raw diff input\");\n> > +\t\tp++;\n> > +\n> > +\t\tmode_a = parse_mode_or_die(p, &p);\n> > +\t\tmode_b = parse_mode_or_die(p, &p);\n> > +\n> > +\t\tparse_oid(p, &oid_a, &p, repo->hash_algo);\n> > +\t\tparse_oid(p, &oid_b, &p, repo->hash_algo);\n> > +\n> > +\t\tstatus = *p++;\n> > +\n> > +\t\tif (strbuf_getline_nul(&path, stdin) == EOF)\n> > +\t\t\tdie(\"got EOF while reading path\");\n> > +\n> > +\t\tswitch (status) {\n> > +\t\tcase DIFF_STATUS_ADDED:\n> > +\t\t\tpair = diff_filepair_addremove(&revs.diffopt, '+',\n> > +\t\t\t\t\t\t mode_b, &oid_b,\n> > +\t\t\t\t\t\t 1, path.buf, 0);\n> > +\t\t\tif (pair)\n> > +\t\t\t\tpair->status = status;\n> > +\t\t\tbreak;\n> > +\n> > +\t\tcase DIFF_STATUS_DELETED:\n> > +\t\t\tpair = diff_filepair_addremove(&revs.diffopt, '-',\n> > +\t\t\t\t\t\t mode_a, &oid_a,\n> > +\t\t\t\t\t\t 1, path.buf, 0);\n> > +\t\t\tif (pair)\n> > +\t\t\t\tpair->status = status;\n> > +\t\t\tbreak;\n> > +\n> > +\t\tcase DIFF_STATUS_TYPE_CHANGED:\n> > +\t\tcase DIFF_STATUS_MODIFIED:\n> > +\t\t\tpair = diff_filepair_change(&revs.diffopt,\n> > +\t\t\t\t\t\t mode_a, mode_b,\n> > +\t\t\t\t\t\t &oid_a, &oid_b, 1, 1,\n> > +\t\t\t\t\t\t path.buf, 0, 0);\n> > +\t\t\tif (pair)\n> > +\t\t\t\tpair->status = status;\n> > +\t\t\tbreak;\n> > +\n> > +\t\tcase DIFF_STATUS_RENAMED:\n> > +\t\tcase DIFF_STATUS_COPIED:\n> > +\t\t\t{\n> > +\t\t\t\tstruct diff_filespec *a, *b;\n> > +\n> > +\t\t\t\tif (strbuf_getline_nul(&path_dst, stdin) == EOF)\n> > +\t\t\t\t\tdie(\"got EOF while reading destination path\");\n> > +\n> > +\t\t\t\ta = alloc_filespec(path.buf);\n> > +\t\t\t\tb = alloc_filespec(path_dst.buf);\n> > +\t\t\t\tfill_filespec(a, &oid_a, 1, mode_a);\n> > +\t\t\t\tfill_filespec(b, &oid_b, 1, mode_b);\n> > +\n> > +\t\t\t\tpair = diff_queue(&diff_queued_diff, a, b);\n> > +\t\t\t\tpair->status = status;\n> > +\t\t\t\tpair->score = parse_score(p);\n> > +\t\t\t\tpair->renamed_pair = 1;\n> > +\t\t\t}\n> > +\t\t\tbreak;\n> > +\n> > +\t\tdefault:\n> \n> The only state I think is missing is `DIFF_STATUS_UNMERGED` (from\n> 'diff.h'). Is that a state we need to handle?\n\nThe `DIFF_STATUS_UNMERGED` status is present when there are unmerged\nconflicted files in the working tree that have not been added to the\nindex. I think this is a scenario where git-diff-pairs(1) would not make\nmuch sense, especially if there is no working tree present for a\nrepository. It should be fine to leave this status unhandled for now as\nit just fallback to the default case and dies.\n\n> > +\t\t\tdie(\"unknown diff status: %c\", status);\n> > +\t\t}\n> > +\t}\n> > +\n> > +\tflush_diff_queue(&revs.diffopt);\n> \n> Now I understand what you meant by queuing the diffs.\n> \n> > +\tret = diff_result_code(&revs);\n> > +\n> > +\tstrbuf_release(&path_dst);\n> > +\tstrbuf_release(&path);\n> > +\tstrbuf_release(&meta);\n> > +\trelease_revisions(&revs);\n> > +\n> > +\treturn ret;\n> > +}\n> > diff --git a/command-list.txt b/command-list.txt\n> > index e0bb87b3b5..bb8acd51d8 100644\n> > --- a/command-list.txt\n> > +++ b/command-list.txt\n> > @@ -95,6 +95,7 @@ git-diagnose ancillaryinterrogators\n> > git-diff mainporcelain info\n> > git-diff-files plumbinginterrogators\n> > git-diff-index plumbinginterrogators\n> > +git-diff-pairs plumbinginterrogators\n> > git-diff-tree plumbinginterrogators\n> > git-difftool ancillaryinterrogators complete\n> > git-fast-export ancillarymanipulators\n> > diff --git a/git.c b/git.c\n> > index b23761480f..12bba872bb 100644\n> > --- a/git.c\n> > +++ b/git.c\n> > @@ -540,6 +540,7 @@ static struct cmd_struct commands[] = {\n> > \t{ \"diff\", cmd_diff, NO_PARSEOPT },\n> > \t{ \"diff-files\", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n> > \t{ \"diff-index\", cmd_diff_index, RUN_SETUP | NO_PARSEOPT },\n> > +\t{ \"diff-pairs\", cmd_diff_pairs, RUN_SETUP | NO_PARSEOPT },\n> > \t{ \"diff-tree\", cmd_diff_tree, RUN_SETUP | NO_PARSEOPT },\n> > \t{ \"difftool\", cmd_difftool, RUN_SETUP_GENTLY },\n> > \t{ \"fast-export\", cmd_fast_export, RUN_SETUP },\n> > diff --git a/meson.build b/meson.build\n> > index fbb8105d96..66ce3326e8 100644\n> > --- a/meson.build\n> > +++ b/meson.build\n> > @@ -537,6 +537,7 @@ builtin_sources = [\n> > 'builtin/diagnose.c',\n> > 'builtin/diff-files.c',\n> > 'builtin/diff-index.c',\n> > + 'builtin/diff-pairs.c',\n> > 'builtin/diff-tree.c',\n> > 'builtin/diff.c',\n> > 'builtin/difftool.c',\n> > diff --git a/t/meson.build b/t/meson.build\n> > index 4574280590..7ff17c6d29 100644\n> > --- a/t/meson.build\n> > +++ b/t/meson.build\n> > @@ -500,6 +500,7 @@ integration_tests = [\n> > 't4067-diff-partial-clone.sh',\n> > 't4068-diff-symmetric-merge-base.sh',\n> > 't4069-remerge-diff.sh',\n> > + 't4070-diff-pairs.sh',\n> > 't4100-apply-stat.sh',\n> > 't4101-apply-nonl.sh',\n> > 't4102-apply-rename.sh',\n> > diff --git a/t/t4070-diff-pairs.sh b/t/t4070-diff-pairs.sh\n> > new file mode 100755\n> > index 0000000000..e0a8e6f0a0\n> > --- /dev/null\n> > +++ b/t/t4070-diff-pairs.sh\n> > @@ -0,0 +1,80 @@\n> > +#!/bin/sh\n> > +\n> > +test_description='basic diff-pairs tests'\n> > +. ./test-lib.sh\n> > +\n> > +# This creates a diff with added, modified, deleted, renamed, copied, and\n> > +# typechange entries. That includes one in a subdirectory for non-recursive\n> > +# tests, and both exact and inexact similarity scores.\n> > +test_expect_success 'create commit with various diffs' '\n> \n> Generally, tests for setup are named 'setup' so we can do something\n> like:\n> sh ./t0050-filesystem.sh --run=setup,9-11\n> \n> Can we renmae this to 'setup'?\n> \n\nGood suggestion, I've updated.\n\n> > +\techo to-be-gone >deleted &&\n> > +\techo original >modified &&\n> > +\techo now-a-file >symlink &&\n> > +\ttest_seq 200 >two-hundred &&\n> > +\ttest_seq 201 500 >five-hundred &&\n> > +\tgit add . &&\n> > +\ttest_tick &&\n> > +\tgit commit -m base &&\n> > +\tgit tag base &&\n> > +\n> > +\techo now-here >added &&\n> > +\techo new >modified &&\n> > +\trm deleted &&\n> > +\tmkdir subdir &&\n> > +\techo content >subdir/file &&\n> > +\tmv two-hundred renamed &&\n> > +\ttest_seq 201 500 | sed s/300/modified/ >copied &&\n> > +\trm symlink &&\n> > +\tgit add -A . &&\n> > +\ttest_ln_s_add dest symlink &&\n> > +\ttest_tick &&\n> > +\tgit commit -m new &&\n> > +\tgit tag new\n> > +'\n> > +\n> > +test_expect_success 'diff-pairs recreates --raw' '\n> > +\tgit diff-tree -r -M -C -C base new >expect &&\n> > +\tgit diff-tree -r -M -C -C -z base new |\n> > +\tgit diff-pairs >actual &&\n> > +\ttest_cmp expect actual\n> > +'\n> > +\n> > +test_expect_success 'diff-pairs can create -p output' '\n> > +\tgit diff-tree -p -M -C -C base new >expect &&\n> > +\tgit diff-tree -r -M -C -C -z base new |\n> > +\tgit diff-pairs -p >actual &&\n> > +\ttest_cmp expect actual\n> > +'\n> > +\n> > +test_expect_success 'non-recursive --raw retains tree entry' '\n> > +\tgit diff-tree base new >expect &&\n> > +\tgit diff-tree -z base new |\n> > +\tgit diff-pairs >actual &&\n> > +\ttest_cmp expect actual\n> > +'\n> > +\n> > +test_expect_success 'split input across multiple diff-pairs' '\n> > +\twrite_script split-raw-diff \"$PERL_PATH\" <<-\\EOF &&\n> > +\t$/ = \"\\0\";\n> > +\twhile (<>) {\n> > +\t my $meta = $_;\n> > +\t my $path = <>;\n> > +\t # renames have an extra path\n> > +\t my $path2 = <> if $meta =~ /[RC]\\d+/;\n> > +\n> > +\t open(my $fh, \">\", sprintf \"diff%03d\", $.);\n> > +\t print $fh $meta, $path, $path2;\n> > +\t}\n> > +\tEOF\n> > +\n> > +\tgit diff-tree -p -M -C -C base new >expect &&\n> > +\n> > +\tgit diff-tree -r -z -M -C -C base new |\n> > +\t./split-raw-diff &&\n> > +\tfor i in diff*; do\n> > +\t\tgit diff-pairs -p <$i || return 1\n> > +\tdone >actual &&\n> > +\ttest_cmp expect actual\n> > +'\n> > +\n> > +test_done\n> > --\n> > 2.48.1\n\n\n"},{"id":"513040","messageId":"20250225233925.1345086-1-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250212041825.2455031-1-jltobler@gmail.com","subject":"[PATCH v3 0/3] batch blob diff generation","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-25T23:39:22Z","receivedAt":"2025-02-25T23:42:51Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Through git-diff(1) it is possible to generate a diff directly between\ntwo blobs. This is particularly useful when the pre-image and post-image\nblobs are known and we only care about the diff between them.\nUnfortunately, if a user has a batch of known blob pairs to compute\ndiffs for, there is currently not a way to do so via a single Git\nprocess.\n\nTo enable support for batch diffs of multiple blob pairs, this series\nintroduces a new diff plumbing command git-diff-pairs(1) based on a\nprevious patch series submitted by Peff[1]. This command uses\nNUL-delimited raw diffs as its source of input to control exactly which\nfilepairs are diffed. The advantage of using the raw diff format is that\nit already has diff status type and object context information embedded\nin each line making it more efficient to generate diffs with as we can\navoid having to peel revisions to get some the same info.\n\nFor example:\n\n git diff-tree -r -z -M $old $new |\n git diff-pairs -p -z\n\nHere the output of git-diff-tree(1) is fed to git-diff-pairs(1) to\ngenerate the same output that would be expected from `git diff-tree -p\n-M`. While by itself not particularly useful, this means it is possible\nto split git-diff-tree(1) output across multiple git-diff-pairs(1)\nprocesses. Such a feature is useful on the server-side where diffs\nbewteen a large set of changes may not be feasible all at once due to\ntimeout concerns.\n\nThis command can be viewed as a backend tool that exposes Git's diff\nmachinery. In its current form, the frontend that generates the raw diff\nlines used as input is expected to most of the heavy lifting (ie.\npathspec limiting, tree object expansion).\n\nThis series is structured as follows:\n\n - Patch 1 adds some new helper functions to get access to the queued\n `diff_filepair` after `diff_queue()` is invoked.\n\n - Patch 2 introduces the new git-diff-pairs(1) plumbing command.\n\n - Patch 3 allows git-diff-pairs(1) to immediately compute diffs\n queued on stdin when a NUL-byte is written after a raw input line\n instead of waiting for stdin to close.\n\nChanges since V2:\n\n - Pathspecs are not supported and thus rejected when provided as\n arguments. It should be possible in a future series to add support\n though.\n\n - Tree objects present in `diff-pairs` input are rejected. Support\n for tree objects could be added in the future, but for now they\n are rejected to enable to future support in a backwards compatible\n manner.\n\n - The -z option is required by git-diff-pairs(1). The NUL-delimited\n raw diff format is the only accepted form of input. Consequently,\n NUL-delimited output is the only option in the `--raw` mode.\n\n - git-diff-pairs(1) defaults to patch output instead of raw output.\n This better fits the intended usecase of the command.\n\n - A NUL-byte is now always used as the delimiter between batches of\n file pair diffs when queued diffs are explicitly computed by\n writing a NUL-byte on stdin.\n\n - Several other small cleanups and fixes along with documentation\n changes.\n\nChanges since V1:\n\n - Changed from git-diff-blob(1) to git-diff-pairs(1) based on a\n previously submitted series.\n\n - Instead of each line containing a pair of blob revisions, the raw\n diff format is used as input which already has diff status and\n object context embedded.\n\n-Justin\n\n[1]: <20161201204042.6yslbyrg7l6ghhww@sigill.intra.peff.net>\n\nJustin Tobler (3):\n diff: return diff_filepair from diff queue helpers\n builtin: introduce diff-pairs command\n builtin/diff-pairs: allow explicit diff queue flush\n\n .gitignore | 1 +\n Documentation/git-diff-pairs.adoc | 60 +++++++++\n Documentation/meson.build | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/diff-pairs.c | 206 ++++++++++++++++++++++++++++++\n command-list.txt | 1 +\n diff.c | 70 +++++++---\n diff.h | 25 ++++\n git.c | 1 +\n meson.build | 1 +\n t/meson.build | 1 +\n t/t4070-diff-pairs.sh | 83 ++++++++++++\n 13 files changed, 432 insertions(+), 20 deletions(-)\n create mode 100644 Documentation/git-diff-pairs.adoc\n create mode 100644 builtin/diff-pairs.c\n create mode 100755 t/t4070-diff-pairs.sh\n\n-- \n2.48.1\n\n"},{"id":"513041","messageId":"20250225233925.1345086-2-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250225233925.1345086-1-jltobler@gmail.com","subject":"[PATCH v3 1/3] diff: return diff_filepair from diff queue helpers","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-25T23:39:23Z","receivedAt":"2025-02-25T23:42:53Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"The `diff_addremove()` and `diff_change()` functions set up and queue\ndiffs, but do not return the `diff_filepair` added to the queue. In a\nsubsequent commit, modifications to `diff_filepair` need to occur in\ncertain cases after being queued.\n\nSince the existing `diff_addremove()` and `diff_change()` are also used\nfor callbacks in `diff_options` as types `add_remove_fn_t` and\n`change_fn_t`, modifying the existing function signatures requires\nfurther changes. The diff options for pruning use `file_add_remove()`\nand `file_change()` where file pairs do not even get queued. Thus,\nseparate functions are implemented instead.\n\nSplit out the queuing operations into `diff_queue_addremove()` and\n`diff_queue_change()` which also return a handle to the queued\n`diff_filepair`.\n\nSigned-off-by: Justin Tobler \n---\n diff.c | 70 +++++++++++++++++++++++++++++++++++++++++-----------------\n diff.h | 25 +++++++++++++++++++++\n 2 files changed, 75 insertions(+), 20 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 019fb893a7..b5a779f997 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -7157,16 +7157,19 @@ void compute_diffstat(struct diff_options *options,\n \toptions->found_changes = !!diffstat->nr;\n }\n \n-void diff_addremove(struct diff_options *options,\n-\t\t int addremove, unsigned mode,\n-\t\t const struct object_id *oid,\n-\t\t int oid_valid,\n-\t\t const char *concatpath, unsigned dirty_submodule)\n+struct diff_filepair *diff_queue_addremove(struct diff_queue_struct *queue,\n+\t\t\t\t\t struct diff_options *options,\n+\t\t\t\t\t int addremove, unsigned mode,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t int oid_valid,\n+\t\t\t\t\t const char *concatpath,\n+\t\t\t\t\t unsigned dirty_submodule)\n {\n \tstruct diff_filespec *one, *two;\n+\tstruct diff_filepair *pair;\n \n \tif (S_ISGITLINK(mode) && is_submodule_ignored(concatpath, options))\n-\t\treturn;\n+\t\treturn NULL;\n \n \t/* This may look odd, but it is a preparation for\n \t * feeding \"there are unchanged files which should\n@@ -7186,7 +7189,7 @@ void diff_addremove(struct diff_options *options,\n \n \tif (options->prefix &&\n \t strncmp(concatpath, options->prefix, options->prefix_length))\n-\t\treturn;\n+\t\treturn NULL;\n \n \tone = alloc_filespec(concatpath);\n \ttwo = alloc_filespec(concatpath);\n@@ -7198,25 +7201,29 @@ void diff_addremove(struct diff_options *options,\n \t\ttwo->dirty_submodule = dirty_submodule;\n \t}\n \n-\tdiff_queue(&diff_queued_diff, one, two);\n+\tpair = diff_queue(queue, one, two);\n \tif (!options->flags.diff_from_contents)\n \t\toptions->flags.has_changes = 1;\n+\n+\treturn pair;\n }\n \n-void diff_change(struct diff_options *options,\n-\t\t unsigned old_mode, unsigned new_mode,\n-\t\t const struct object_id *old_oid,\n-\t\t const struct object_id *new_oid,\n-\t\t int old_oid_valid, int new_oid_valid,\n-\t\t const char *concatpath,\n-\t\t unsigned old_dirty_submodule, unsigned new_dirty_submodule)\n+struct diff_filepair *diff_queue_change(struct diff_queue_struct *queue,\n+\t\t\t\t\tstruct diff_options *options,\n+\t\t\t\t\tunsigned old_mode, unsigned new_mode,\n+\t\t\t\t\tconst struct object_id *old_oid,\n+\t\t\t\t\tconst struct object_id *new_oid,\n+\t\t\t\t\tint old_oid_valid, int new_oid_valid,\n+\t\t\t\t\tconst char *concatpath,\n+\t\t\t\t\tunsigned old_dirty_submodule,\n+\t\t\t\t\tunsigned new_dirty_submodule)\n {\n \tstruct diff_filespec *one, *two;\n \tstruct diff_filepair *p;\n \n \tif (S_ISGITLINK(old_mode) && S_ISGITLINK(new_mode) &&\n \t is_submodule_ignored(concatpath, options))\n-\t\treturn;\n+\t\treturn NULL;\n \n \tif (options->flags.reverse_diff) {\n \t\tSWAP(old_mode, new_mode);\n@@ -7227,7 +7234,7 @@ void diff_change(struct diff_options *options,\n \n \tif (options->prefix &&\n \t strncmp(concatpath, options->prefix, options->prefix_length))\n-\t\treturn;\n+\t\treturn NULL;\n \n \tone = alloc_filespec(concatpath);\n \ttwo = alloc_filespec(concatpath);\n@@ -7235,19 +7242,42 @@ void diff_change(struct diff_options *options,\n \tfill_filespec(two, new_oid, new_oid_valid, new_mode);\n \tone->dirty_submodule = old_dirty_submodule;\n \ttwo->dirty_submodule = new_dirty_submodule;\n-\tp = diff_queue(&diff_queued_diff, one, two);\n+\tp = diff_queue(queue, one, two);\n \n \tif (options->flags.diff_from_contents)\n-\t\treturn;\n+\t\treturn p;\n \n \tif (options->flags.quick && options->skip_stat_unmatch &&\n \t !diff_filespec_check_stat_unmatch(options->repo, p)) {\n \t\tdiff_free_filespec_data(p->one);\n \t\tdiff_free_filespec_data(p->two);\n-\t\treturn;\n+\t\treturn p;\n \t}\n \n \toptions->flags.has_changes = 1;\n+\n+\treturn p;\n+}\n+\n+void diff_addremove(struct diff_options *options, int addremove, unsigned mode,\n+\t\t const struct object_id *oid, int oid_valid,\n+\t\t const char *concatpath, unsigned dirty_submodule)\n+{\n+\tdiff_queue_addremove(&diff_queued_diff, options, addremove, mode, oid,\n+\t\t\t oid_valid, concatpath, dirty_submodule);\n+}\n+\n+void diff_change(struct diff_options *options,\n+\t\t unsigned old_mode, unsigned new_mode,\n+\t\t const struct object_id *old_oid,\n+\t\t const struct object_id *new_oid,\n+\t\t int old_oid_valid, int new_oid_valid,\n+\t\t const char *concatpath,\n+\t\t unsigned old_dirty_submodule, unsigned new_dirty_submodule)\n+{\n+\tdiff_queue_change(&diff_queued_diff, options, old_mode, new_mode,\n+\t\t\t old_oid, new_oid, old_oid_valid, new_oid_valid,\n+\t\t\t concatpath, old_dirty_submodule, new_dirty_submodule);\n }\n \n struct diff_filepair *diff_unmerge(struct diff_options *options, const char *path)\ndiff --git a/diff.h b/diff.h\nindex 0a566f5531..63afa17e84 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -508,6 +508,31 @@ void diff_set_default_prefix(struct diff_options *options);\n \n int diff_can_quit_early(struct diff_options *);\n \n+/*\n+ * Stages changes in the provided diff queue for file additions and deletions.\n+ * If a file pair gets queued, it is returned.\n+ */\n+struct diff_filepair *diff_queue_addremove(struct diff_queue_struct *queue,\n+\t\t\t\t\t struct diff_options *,\n+\t\t\t\t\t int addremove, unsigned mode,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t int oid_valid, const char *fullpath,\n+\t\t\t\t\t unsigned dirty_submodule);\n+\n+/*\n+ * Stages changes in the provided diff queue for file modifications.\n+ * If a file pair gets queued, it is returned.\n+ */\n+struct diff_filepair *diff_queue_change(struct diff_queue_struct *queue,\n+\t\t\t\t\tstruct diff_options *,\n+\t\t\t\t\tunsigned mode1, unsigned mode2,\n+\t\t\t\t\tconst struct object_id *old_oid,\n+\t\t\t\t\tconst struct object_id *new_oid,\n+\t\t\t\t\tint old_oid_valid, int new_oid_valid,\n+\t\t\t\t\tconst char *fullpath,\n+\t\t\t\t\tunsigned dirty_submodule1,\n+\t\t\t\t\tunsigned dirty_submodule2);\n+\n void diff_addremove(struct diff_options *,\n \t\t int addremove,\n \t\t unsigned mode,\n-- \n2.48.1\n\n"},{"id":"513042","messageId":"20250225233925.1345086-3-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250225233925.1345086-1-jltobler@gmail.com","subject":"[PATCH v3 2/3] builtin: introduce diff-pairs command","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-25T23:39:24Z","receivedAt":"2025-02-25T23:42:54Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Through git-diff(1), a single diff can be generated from a pair of blob\nrevisions directly. Unfortunately, there is not a mechanism to compute\nbatches of specific file pair diffs in a single process. Such a feature\nis particularly useful on the server-side where diffing between a large\nset of changes is not feasible all at once due to timeout concerns.\n\nTo facilitate this, introduce git-diff-pairs(1) which acts as a backend\npassing its NUL-terminated raw diff format input from stdin through diff\nmachinery to produce various forms of output such as patch or raw.\n\nThe raw format was originally designed as an interchange format and\nrepresents the contents of the diff_queue_diff list making it possible\nto break the diff pipeline into separate stages. For example,\ngit-diff-tree(1) can be used as a frontend to compute file pairs to\nqueue and feed its raw output to git-diff-pairs(1) to compute patches.\nWith this, batches of diffs can be progessively generated without having\nto recompute rename detection or retrieve object context. Something like\nthe following:\n\n\tgit diff-tree -r -z -M $old $new |\n\tgit diff-pairs -p -z\n\nshould generate the same output as `git diff-tree -p -M`. Furthermore,\neach line of raw diff formatted input can also be individually fed to a\nseparate git-diff-pairs(1) process and still produce the same output.\n\nBased-on-patch-by: Jeff King \nSigned-off-by: Justin Tobler \n---\n .gitignore | 1 +\n Documentation/git-diff-pairs.adoc | 56 +++++++++\n Documentation/meson.build | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/diff-pairs.c | 193 ++++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n meson.build | 1 +\n t/meson.build | 1 +\n t/t4070-diff-pairs.sh | 74 ++++++++++++\n 11 files changed, 331 insertions(+)\n create mode 100644 Documentation/git-diff-pairs.adoc\n create mode 100644 builtin/diff-pairs.c\n create mode 100755 t/t4070-diff-pairs.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 08a66ca508..04c444404e 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -55,6 +55,7 @@\n /git-diff\n /git-diff-files\n /git-diff-index\n+/git-diff-pairs\n /git-diff-tree\n /git-difftool\n /git-difftool--helper\ndiff --git a/Documentation/git-diff-pairs.adoc b/Documentation/git-diff-pairs.adoc\nnew file mode 100644\nindex 0000000000..e31f2e2fbb\n--- /dev/null\n+++ b/Documentation/git-diff-pairs.adoc\n@@ -0,0 +1,56 @@\n+git-diff-pairs(1)\n+=================\n+\n+NAME\n+----\n+git-diff-pairs - Compare the content and mode of provided blob pairs\n+\n+SYNOPSIS\n+--------\n+[synopsis]\n+git diff-pairs -z []\n+\n+DESCRIPTION\n+-----------\n+Show changes for file pairs provided on stdin. Input for this command must be\n+in the NUL-terminated raw output format as generated by commands such as `git\n+diff-tree -z -r --raw`. By default, the outputted diffs are computed and shown\n+in the patch format when stdin closes.\n+\n+Usage of this command enables the traditional diff pipeline to be broken up\n+into separate stages where `diff-pairs` acts as the output phase. Other\n+commands, such as `diff-tree`, may serve as a frontend to compute the raw\n+diff format used as input.\n+\n+Instead of computing diffs via `git diff-tree -p -M` in one step, `diff-tree`\n+can compute the file pairs and rename information without the blob diffs. This\n+output can be fed to `diff-pairs` to generate the underlying blob diffs as done\n+in the following example:\n+\n+-----------------------------\n+git diff-tree -z -r -M $a $b |\n+git diff-pairs -z\n+-----------------------------\n+\n+Computing the tree diff upfront with rename information allows patch output\n+from `diff-pairs` to be progressively computed over the course of potentially\n+multiple invocations.\n+\n+Pathspecs are not currently supported by `diff-pairs`. Pathspec limiting should\n+be performed by the upstream command generating the raw diffs used as input.\n+\n+Tree objects are not currently supported as input and are rejected.\n+\n+Abbreviated object IDs in the `diff-pairs` input are not supported. Outputted\n+object IDs can be abbreviated using the `--abbrev` option.\n+\n+OPTIONS\n+-------\n+\n+include::diff-options.adoc[]\n+\n+include::diff-generate-patch.adoc[]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Documentation/meson.build b/Documentation/meson.build\nindex 1129ce4c85..ce990e9fe5 100644\n--- a/Documentation/meson.build\n+++ b/Documentation/meson.build\n@@ -42,6 +42,7 @@ manpages = {\n 'git-diagnose.adoc' : 1,\n 'git-diff-files.adoc' : 1,\n 'git-diff-index.adoc' : 1,\n+ 'git-diff-pairs.adoc' : 1,\n 'git-difftool.adoc' : 1,\n 'git-diff-tree.adoc' : 1,\n 'git-diff.adoc' : 1,\ndiff --git a/Makefile b/Makefile\nindex bcf5ed3f85..56df7aed3f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1242,6 +1242,7 @@ BUILTIN_OBJS += builtin/describe.o\n BUILTIN_OBJS += builtin/diagnose.o\n BUILTIN_OBJS += builtin/diff-files.o\n BUILTIN_OBJS += builtin/diff-index.o\n+BUILTIN_OBJS += builtin/diff-pairs.o\n BUILTIN_OBJS += builtin/diff-tree.o\n BUILTIN_OBJS += builtin/diff.o\n BUILTIN_OBJS += builtin/difftool.o\ndiff --git a/builtin.h b/builtin.h\nindex 89928ccf92..e6aad3a6a1 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -153,6 +153,7 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix, struct reposit\n int cmd_diff_files(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_diff_index(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_diff(int argc, const char **argv, const char *prefix, struct repository *repo);\n+int cmd_diff_pairs(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_diff_tree(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_difftool(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_env__helper(int argc, const char **argv, const char *prefix, struct repository *repo);\ndiff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\nnew file mode 100644\nindex 0000000000..9472b10461\n--- /dev/null\n+++ b/builtin/diff-pairs.c\n@@ -0,0 +1,193 @@\n+#include \"builtin.h\"\n+#include \"commit.h\"\n+#include \"config.h\"\n+#include \"diff.h\"\n+#include \"diffcore.h\"\n+#include \"gettext.h\"\n+#include \"hex.h\"\n+#include \"object.h\"\n+#include \"parse-options.h\"\n+#include \"revision.h\"\n+#include \"strbuf.h\"\n+\n+static unsigned parse_mode_or_die(const char *mode, const char **endp)\n+{\n+\tuint16_t ret;\n+\n+\t*endp = parse_mode(mode, &ret);\n+\tif (!*endp)\n+\t\tdie(_(\"unable to parse mode: %s\"), mode);\n+\treturn ret;\n+}\n+\n+static void parse_oid_or_die(const char *p, struct object_id *oid,\n+\t\t\t const char **endp, const struct git_hash_algo *algop)\n+{\n+\tif (parse_oid_hex_algop(p, oid, endp, algop) || *(*endp)++ != ' ')\n+\t\tdie(_(\"unable to parse object id: %s\"), p);\n+}\n+\n+static void flush_diff_queue(struct diff_options *options)\n+{\n+\t/*\n+\t * If rename detection is not requested, use rename information from the\n+\t * raw diff formatted input. Setting found_follow ensures diffcore_std()\n+\t * does not mess with rename information already present in queued\n+\t * filepairs.\n+\t */\n+\tif (!options->detect_rename)\n+\t\toptions->found_follow = 1;\n+\tdiffcore_std(options);\n+\tdiff_flush(options);\n+}\n+\n+int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n+\t\t struct repository *repo)\n+{\n+\tstruct strbuf path_dst = STRBUF_INIT;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct strbuf meta = STRBUF_INIT;\n+\tstruct rev_info revs;\n+\tint ret;\n+\n+\tconst char * const usage[] = {\n+\t\tN_(\"git diff-pairs -z []\"),\n+\t\tNULL\n+\t};\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\tstruct option *parseopts = add_diff_options(options, &revs.diffopt);\n+\n+\tshow_usage_with_options_if_asked(argc, argv, usage, parseopts);\n+\n+\trepo_init_revisions(repo, &revs, prefix);\n+\trepo_config(repo, git_diff_basic_config, NULL);\n+\trevs.disable_stdin = 1;\n+\trevs.abbrev = 0;\n+\trevs.diff = 1;\n+\n+\tif (setup_revisions(argc, argv, &revs, NULL) > 1)\n+\t\tusage_with_options(usage, parseopts);\n+\n+\t/*\n+\t * With the -z option, both command input and raw output are\n+\t * NUL-delimited (this mode does not effect patch output). At present\n+\t * only NUL-delimited raw diff formatted input is supported.\n+\t */\n+\tif (revs.diffopt.line_termination) {\n+\t\terror(_(\"working without -z is not supported\"));\n+\t\tusage_with_options(usage, parseopts);\n+\t}\n+\n+\tif (revs.prune_data.nr) {\n+\t\terror(_(\"pathspec arguments not supported\"));\n+\t\tusage_with_options(usage, parseopts);\n+\t}\n+\n+\tif (revs.pending.nr || revs.max_count != -1 ||\n+\t revs.min_age != (timestamp_t)-1 ||\n+\t revs.max_age != (timestamp_t)-1) {\n+\t\terror(_(\"revision arguments not allowed\"));\n+\t\tusage_with_options(usage, parseopts);\n+\t}\n+\n+\tif (!revs.diffopt.output_format)\n+\t\trevs.diffopt.output_format = DIFF_FORMAT_PATCH;\n+\n+\twhile (1) {\n+\t\tstruct object_id oid_a, oid_b;\n+\t\tstruct diff_filepair *pair;\n+\t\tunsigned mode_a, mode_b;\n+\t\tconst char *p;\n+\t\tchar status;\n+\n+\t\tif (strbuf_getline_nul(&meta, stdin) == EOF)\n+\t\t\tbreak;\n+\n+\t\tp = meta.buf;\n+\t\tif (*p != ':')\n+\t\t\tdie(_(\"invalid raw diff input\"));\n+\t\tp++;\n+\n+\t\tmode_a = parse_mode_or_die(p, &p);\n+\t\tmode_b = parse_mode_or_die(p, &p);\n+\n+\t\tif (S_ISDIR(mode_a) || S_ISDIR(mode_b))\n+\t\t\tdie(_(\"tree objects not supported\"));\n+\n+\t\tparse_oid_or_die(p, &oid_a, &p, repo->hash_algo);\n+\t\tparse_oid_or_die(p, &oid_b, &p, repo->hash_algo);\n+\n+\t\tstatus = *p++;\n+\n+\t\tif (strbuf_getline_nul(&path, stdin) == EOF)\n+\t\t\tdie(_(\"got EOF while reading path\"));\n+\n+\t\tswitch (status) {\n+\t\tcase DIFF_STATUS_ADDED:\n+\t\t\tpair = diff_queue_addremove(&diff_queued_diff,\n+\t\t\t\t\t\t &revs.diffopt, '+', mode_b,\n+\t\t\t\t\t\t &oid_b, 1, path.buf, 0);\n+\t\t\tif (pair)\n+\t\t\t\tpair->status = status;\n+\t\t\tbreak;\n+\n+\t\tcase DIFF_STATUS_DELETED:\n+\t\t\tpair = diff_queue_addremove(&diff_queued_diff,\n+\t\t\t\t\t\t &revs.diffopt, '-', mode_a,\n+\t\t\t\t\t\t &oid_a, 1, path.buf, 0);\n+\t\t\tif (pair)\n+\t\t\t\tpair->status = status;\n+\t\t\tbreak;\n+\n+\t\tcase DIFF_STATUS_TYPE_CHANGED:\n+\t\tcase DIFF_STATUS_MODIFIED:\n+\t\t\tpair = diff_queue_change(&diff_queued_diff, &revs.diffopt,\n+\t\t\t\t\t\t mode_a, mode_b, &oid_a, &oid_b,\n+\t\t\t\t\t\t 1, 1, path.buf, 0, 0);\n+\t\t\tif (pair)\n+\t\t\t\tpair->status = status;\n+\t\t\tbreak;\n+\n+\t\tcase DIFF_STATUS_RENAMED:\n+\t\tcase DIFF_STATUS_COPIED:\n+\t\t\t{\n+\t\t\t\tstruct diff_filespec *a, *b;\n+\t\t\t\tunsigned int score;\n+\n+\t\t\t\tif (strbuf_getline_nul(&path_dst, stdin) == EOF)\n+\t\t\t\t\tdie(_(\"got EOF while reading destination path\"));\n+\n+\t\t\t\ta = alloc_filespec(path.buf);\n+\t\t\t\tb = alloc_filespec(path_dst.buf);\n+\t\t\t\tfill_filespec(a, &oid_a, 1, mode_a);\n+\t\t\t\tfill_filespec(b, &oid_b, 1, mode_b);\n+\n+\t\t\t\tpair = diff_queue(&diff_queued_diff, a, b);\n+\n+\t\t\t\tif (strtoul_ui(p, 10, &score))\n+\t\t\t\t\tdie(_(\"unable to parse rename/copy score: %s\"), p);\n+\n+\t\t\t\tpair->score = score * MAX_SCORE / 100;\n+\t\t\t\tpair->status = status;\n+\t\t\t\tpair->renamed_pair = 1;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tdefault:\n+\t\t\tdie(_(\"unknown diff status: %c\"), status);\n+\t\t}\n+\t}\n+\n+\tflush_diff_queue(&revs.diffopt);\n+\tret = diff_result_code(&revs);\n+\n+\tstrbuf_release(&path_dst);\n+\tstrbuf_release(&path);\n+\tstrbuf_release(&meta);\n+\trelease_revisions(&revs);\n+\tFREE_AND_NULL(parseopts);\n+\n+\treturn ret;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex c537114b46..b7ade3ab9f 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -96,6 +96,7 @@ git-diagnose ancillaryinterrogators\n git-diff mainporcelain info\n git-diff-files plumbinginterrogators\n git-diff-index plumbinginterrogators\n+git-diff-pairs plumbinginterrogators\n git-diff-tree plumbinginterrogators\n git-difftool ancillaryinterrogators complete\n git-fast-export ancillarymanipulators\ndiff --git a/git.c b/git.c\nindex 450d6aaa86..77c4359522 100644\n--- a/git.c\n+++ b/git.c\n@@ -541,6 +541,7 @@ static struct cmd_struct commands[] = {\n \t{ \"diff\", cmd_diff, NO_PARSEOPT },\n \t{ \"diff-files\", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n \t{ \"diff-index\", cmd_diff_index, RUN_SETUP | NO_PARSEOPT },\n+\t{ \"diff-pairs\", cmd_diff_pairs, RUN_SETUP | NO_PARSEOPT },\n \t{ \"diff-tree\", cmd_diff_tree, RUN_SETUP | NO_PARSEOPT },\n \t{ \"difftool\", cmd_difftool, RUN_SETUP_GENTLY },\n \t{ \"fast-export\", cmd_fast_export, RUN_SETUP },\ndiff --git a/meson.build b/meson.build\nindex bf95576f83..9e8b365d2a 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -540,6 +540,7 @@ builtin_sources = [\n 'builtin/diagnose.c',\n 'builtin/diff-files.c',\n 'builtin/diff-index.c',\n+ 'builtin/diff-pairs.c',\n 'builtin/diff-tree.c',\n 'builtin/diff.c',\n 'builtin/difftool.c',\ndiff --git a/t/meson.build b/t/meson.build\nindex 780939d49f..09c7bc2fad 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -500,6 +500,7 @@ integration_tests = [\n 't4067-diff-partial-clone.sh',\n 't4068-diff-symmetric-merge-base.sh',\n 't4069-remerge-diff.sh',\n+ 't4070-diff-pairs.sh',\n 't4100-apply-stat.sh',\n 't4101-apply-nonl.sh',\n 't4102-apply-rename.sh',\ndiff --git a/t/t4070-diff-pairs.sh b/t/t4070-diff-pairs.sh\nnew file mode 100755\nindex 0000000000..2f511cc9c9\n--- /dev/null\n+++ b/t/t4070-diff-pairs.sh\n@@ -0,0 +1,74 @@\n+#!/bin/sh\n+\n+test_description='basic diff-pairs tests'\n+. ./test-lib.sh\n+\n+# This creates a diff with added, modified, deleted, renamed, copied, and\n+# typechange entries. That includes one in a subdirectory for non-recursive\n+# tests, and both exact and inexact similarity scores.\n+test_expect_success 'setup' '\n+\techo to-be-gone >deleted &&\n+\techo original >modified &&\n+\techo now-a-file >symlink &&\n+\ttest_seq 200 >two-hundred &&\n+\ttest_seq 201 500 >five-hundred &&\n+\tgit add . &&\n+\ttest_tick &&\n+\tgit commit -m base &&\n+\tgit tag base &&\n+\n+\techo now-here >added &&\n+\techo new >modified &&\n+\trm deleted &&\n+\tmkdir subdir &&\n+\techo content >subdir/file &&\n+\tmv two-hundred renamed &&\n+\ttest_seq 201 500 | sed s/300/modified/ >copied &&\n+\trm symlink &&\n+\tgit add -A . &&\n+\ttest_ln_s_add dest symlink &&\n+\ttest_tick &&\n+\tgit commit -m new &&\n+\tgit tag new\n+'\n+\n+test_expect_success 'diff-pairs recreates --raw' '\n+\tgit diff-tree -r -M -C -C -z base new >expect &&\n+\tgit diff-tree -r -M -C -C -z base new |\n+\tgit diff-pairs --raw -z >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'diff-pairs can create -p output' '\n+\tgit diff-tree -p -M -C -C base new >expect &&\n+\tgit diff-tree -r -M -C -C -z base new |\n+\tgit diff-pairs -p -z >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'diff-pairs does not support normal raw diff input' '\n+\tgit diff-tree -r base new |\n+\ttest_must_fail git diff-pairs >out 2>err &&\n+\n+\ttest_must_be_empty out &&\n+\tgrep \"error: working without -z is not supported\" err\n+'\n+\n+test_expect_success 'diff-pairs does not support tree objects as input' '\n+\tgit diff-tree -z base new |\n+\ttest_must_fail git diff-pairs -z >out 2>err &&\n+\n+\techo \"fatal: tree objects not supported\" >expect &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'diff-pairs does not support pathspec arguments' '\n+\tgit diff-tree -r -z base new |\n+\ttest_must_fail git diff-pairs -z -- new >out 2>err &&\n+\n+\ttest_must_be_empty out &&\n+\tgrep \"error: pathspec arguments not supported\" err\n+'\n+\n+test_done\n-- \n2.48.1\n\n"},{"id":"513043","messageId":"20250225233925.1345086-4-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250225233925.1345086-1-jltobler@gmail.com","subject":"[PATCH v3 3/3] builtin/diff-pairs: allow explicit diff queue flush","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-25T23:39:25Z","receivedAt":"2025-02-25T23:42:55Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"The diffs queued from git-diff-pairs(1) are flushed when stdin is\nclosed. To enable greater flexibility, allow control over when the diff\nqueue is flushed by writing a single NUL byte on stdin between input\nfile pairs. Diff output between flushes is separated by a single NUL\nbyte.\n\nSigned-off-by: Justin Tobler \n---\n Documentation/git-diff-pairs.adoc | 4 ++++\n builtin/diff-pairs.c | 13 +++++++++++++\n t/t4070-diff-pairs.sh | 9 +++++++++\n 3 files changed, 26 insertions(+)\n\ndiff --git a/Documentation/git-diff-pairs.adoc b/Documentation/git-diff-pairs.adoc\nindex e31f2e2fbb..f99fcd1ead 100644\n--- a/Documentation/git-diff-pairs.adoc\n+++ b/Documentation/git-diff-pairs.adoc\n@@ -17,6 +17,10 @@ in the NUL-terminated raw output format as generated by commands such as `git\n diff-tree -z -r --raw`. By default, the outputted diffs are computed and shown\n in the patch format when stdin closes.\n \n+A single NUL byte may be written to stdin between raw input lines to compute\n+file pair diffs up to that point instead of waiting for stdin to close. A NUL\n+byte is also written to the output to delimit between these batches of diffs.\n+\n Usage of this command enables the traditional diff pipeline to be broken up\n into separate stages where `diff-pairs` acts as the output phase. Other\n commands, such as `diff-tree`, may serve as a frontend to compute the raw\ndiff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\nindex 9472b10461..7130569332 100644\n--- a/builtin/diff-pairs.c\n+++ b/builtin/diff-pairs.c\n@@ -63,6 +63,7 @@ int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n \n \trepo_init_revisions(repo, &revs, prefix);\n \trepo_config(repo, git_diff_basic_config, NULL);\n+\trevs.diffopt.no_free = 1;\n \trevs.disable_stdin = 1;\n \trevs.abbrev = 0;\n \trevs.diff = 1;\n@@ -106,6 +107,17 @@ int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n \t\t\tbreak;\n \n \t\tp = meta.buf;\n+\t\tif (!*p) {\n+\t\t\tflush_diff_queue(&revs.diffopt);\n+\t\t\t/*\n+\t\t\t * When the diff queue is explicitly flushed, append a\n+\t\t\t * NUL byte to separate batches of diffs.\n+\t\t\t */\n+\t\t\tfputc('\\0', revs.diffopt.file);\n+\t\t\tfflush(revs.diffopt.file);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (*p != ':')\n \t\t\tdie(_(\"invalid raw diff input\"));\n \t\tp++;\n@@ -180,6 +192,7 @@ int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n \t\t}\n \t}\n \n+\trevs.diffopt.no_free = 0;\n \tflush_diff_queue(&revs.diffopt);\n \tret = diff_result_code(&revs);\n \ndiff --git a/t/t4070-diff-pairs.sh b/t/t4070-diff-pairs.sh\nindex 2f511cc9c9..3352bfe0b9 100755\n--- a/t/t4070-diff-pairs.sh\n+++ b/t/t4070-diff-pairs.sh\n@@ -71,4 +71,13 @@ test_expect_success 'diff-pairs does not support pathspec arguments' '\n \tgrep \"error: pathspec arguments not supported\" err\n '\n \n+test_expect_success 'diff-pairs explicit queue flush' '\n+\tgit diff-tree -r -M -C -C -z base new >expect &&\n+\tprintf \"\\0\" >>expect &&\n+\tgit diff-tree -r -M -C -C -z base new >>expect &&\n+\n+\tgit diff-pairs --raw -z actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \n2.48.1\n\n"},{"id":"513092","messageId":"00ef7191-1593-454e-ba45-8223726bb37d@gmail.com","threadId":"62633","inReplyTo":"wv5ziveuff7iellcmjcki372m5vp6nmltyls43e4wzslcqymog@gwczuaucpkke","subject":"Re: [PATCH v2 2/3] builtin: introduce diff-pairs command","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2025-02-26T14:47:54Z","receivedAt":"2025-02-26T14:47:58Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Justin\n\nOn 19/02/2025 20:51, Justin Tobler wrote:\n> On 25/02/17 02:38PM, Phillip Wood wrote:\n>> Hi Justin\n>>> +\tconst char * const usage[] = {\n>>> +\t\tN_(\"git diff-pairs [diff-options]\"),\n>>\n>> Normally the option summary printed by \"git foo -h\" is generated by the\n>> option parser. In this case we don't define any options and use\n>> setup_revisions() instead so we need to provide the option summary\n>> ourselves. Looking at diff-files.c we can add\n>>\n>> \t\"\\n\"\n>> \tCOMMON_DIFF_OPTIONS_HELP;\n>>\n>> to do that.\n> \n> Would this be preferable even if git-diff-pairs doesn't support all of\n> the common diff options?\n\nWhich options are you thinking about here? I might have missed something \ndon't think that help text includes anything that's not in \ndiff-options.adoc that we include in diff-pairs.adoc. If there are \noptions in the documentation that we don't support then that is a problem.\n\nBest Wishes\n\nPhillip\n\n\n>>> +\targc = setup_revisions(argc, argv, &revs, NULL);\n>>\n>> I think we should check that there are no options left on the commandline\n>> after setup_revisions() returns\n> \n> Good call, will do in the next version.\n> \n>>> +\t/* Don't allow pathspecs at all. */\n>>> +\tif (revs.prune_data.nr)\n>>> +\t\tusage_with_options(usage, options);\n>>\n>> It is not just pathspecs that we want to reject but all revision related\n>> options. Looking at diff-files.c we can do\n>>\n>> \tif (rev.pending.nr ||\n>> \t rev.min_age != -1 || rev.max_age != -1 ||\n>> \t rev.max_count != -1)\n>> \t\tusage_with_option(usage, options);\n>>\n>> To catch some of that but it still accepts things like \"--first-parent\",\n>> \"--merges\" and \"--ancestry-path\". We may just have to live with that as I\n>> don't think it is worth expanding a huge amount of effort to prevent them.\n> \n> Yes, we should also reject revision as well as pathspec arguments. Will\n> update.\n> \n>>> +\tif (!revs.diffopt.output_format)\n>>> +\t\trevs.diffopt.output_format = DIFF_FORMAT_RAW;\n>>\n>> This matches the other diff plumbing commands but I'm not sure it is the\n>> most helpful default for a command that is supposed to transform raw diffs\n>> into another format. Maybe we should default to DIFF_FORMAT_PATCH?\n> \n> As you mentioned, defaulting to DIFF_FORMAT_RAW isn't the most useful\n> behavior. I agree that it makes more sense to use DIFF_FORMAT_PATCH as\n> the default. Will update in the next version.\n> \n>>> +test_expect_success 'split input across multiple diff-pairs' '\n>>\n>> This needs a PERL prerequisite I think. I'm a bit unsure what this test adds\n>> compared to the others.\n> \n> This test demonstrates that the raw diff input can be split across\n> separate git-diff-pairs(1) processes and still produce equivilant\n> output which is one of the main usecases for the command. That being\n> said, this test isn't really exercising different behavior of\n> git-diff-pairs(1) itself, so maybe it would be best to drop it.\n> \n> Thanks for the review :)\n> \n> -Justin\n\n"},{"id":"513093","messageId":"1e41249a-5241-4cd8-8a6a-3c9163fb0ea0@gmail.com","threadId":"62633","inReplyTo":"20250225233925.1345086-1-jltobler@gmail.com","subject":"Re: [PATCH v3 0/3] batch blob diff generation","fromName":"","fromEmail":"phillip.wood123@gmail.com","sentAt":"2025-02-26T14:58:50Z","receivedAt":"2025-02-26T14:58:54Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Justin\n\nOn 25/02/2025 23:39, Justin Tobler wrote:\n> \n> Changes since V2:\n> \n> - Pathspecs are not supported and thus rejected when provided as\n> arguments. It should be possible in a future series to add support\n> though.\n> \n> - Tree objects present in `diff-pairs` input are rejected. Support\n> for tree objects could be added in the future, but for now they\n> are rejected to enable to future support in a backwards compatible\n> manner.\n> \n> - The -z option is required by git-diff-pairs(1). The NUL-delimited\n> raw diff format is the only accepted form of input. Consequently,\n> NUL-delimited output is the only option in the `--raw` mode.\n> \n> - git-diff-pairs(1) defaults to patch output instead of raw output.\n> This better fits the intended usecase of the command.\n> \n> - A NUL-byte is now always used as the delimiter between batches of\n> file pair diffs when queued diffs are explicitly computed by\n> writing a NUL-byte on stdin.\n> \n> - Several other small cleanups and fixes along with documentation\n> changes.\n\nThis addresses all my comments on the previous version, thank you. I do \nwonder if tying the input line termination to the output line \ntermination is a good idea for a program that aims to to transform one \ndiff format into another. Having said that this series is aimed at \nmachine consumption of the output so it probably isn't a big problem.\n\nI also think we might want to massage the output in the tests so that \nwe're not running test_cmp on files containing NUL bytes. Using\n\n git diff-tree -z ... | tr '\\0' Q >actual\n\nwould get rid of the NULs but does not improve the readability of the \nraw diffs that much as everything is still on a single line. Using '\\n' \ninstead of 'Q' would give us mulit-line output but we would lose \nconfidence that the original output was actually NUL terminated.\n\nBest Wishes\n\nPhillip\n\n\n> Changes since V1:\n> \n> - Changed from git-diff-blob(1) to git-diff-pairs(1) based on a\n> previously submitted series.\n> \n> - Instead of each line containing a pair of blob revisions, the raw\n> diff format is used as input which already has diff status and\n> object context embedded.\n> \n> -Justin\n> \n> [1]: <20161201204042.6yslbyrg7l6ghhww@sigill.intra.peff.net>\n> \n> Justin Tobler (3):\n> diff: return diff_filepair from diff queue helpers\n> builtin: introduce diff-pairs command\n> builtin/diff-pairs: allow explicit diff queue flush\n> \n> .gitignore | 1 +\n> Documentation/git-diff-pairs.adoc | 60 +++++++++\n> Documentation/meson.build | 1 +\n> Makefile | 1 +\n> builtin.h | 1 +\n> builtin/diff-pairs.c | 206 ++++++++++++++++++++++++++++++\n> command-list.txt | 1 +\n> diff.c | 70 +++++++---\n> diff.h | 25 ++++\n> git.c | 1 +\n> meson.build | 1 +\n> t/meson.build | 1 +\n> t/t4070-diff-pairs.sh | 83 ++++++++++++\n> 13 files changed, 432 insertions(+), 20 deletions(-)\n> create mode 100644 Documentation/git-diff-pairs.adoc\n> create mode 100644 builtin/diff-pairs.c\n> create mode 100755 t/t4070-diff-pairs.sh\n> \n\n"},{"id":"513112","messageId":"xmqq1pvkeg5o.fsf@gitster.g","threadId":"62633","inReplyTo":"20250225233925.1345086-2-jltobler@gmail.com","subject":"Re: [PATCH v3 1/3] diff: return diff_filepair from diff queue helpers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-26T18:04:35Z","receivedAt":"2025-02-26T18:04:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Justin Tobler writes:\n\n> separate functions are implemented instead.\n\nIt would have been more assuring to explicitly say that the original\nfunctions that discarded the newly created filepair after adding\nthem to the queue are reimplemented as thin wrappers, which is the\nright thing to do and which is exactly what happens in this patch.\n\nLooking good.\n"},{"id":"513113","messageId":"xmqqjz9cd0nl.fsf@gitster.g","threadId":"62633","inReplyTo":"20250225233925.1345086-3-jltobler@gmail.com","subject":"Re: [PATCH v3 2/3] builtin: introduce diff-pairs command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-26T18:24:46Z","receivedAt":"2025-02-26T18:24:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Justin Tobler writes:\n\n> +static void flush_diff_queue(struct diff_options *options)\n> +{\n> +\t/*\n> +\t * If rename detection is not requested, use rename information from the\n> +\t * raw diff formatted input. Setting found_follow ensures diffcore_std()\n> +\t * does not mess with rename information already present in queued\n> +\t * filepairs.\n> +\t */\n> +\tif (!options->detect_rename)\n> +\t\toptions->found_follow = 1;\n\nAn ugly design decision that may be suboptimal from maintainability\npoint of view.\n\nThe parts of diffcore_std() that --follow wants to bypass may happen\nto be the same as the parts that this new caller wants to bypass,\nbut who guarantees that they will stay that way in the future?\n\n> +\tdiffcore_std(options);\n> +\tdiff_flush(options);\n> +}\n> +\n> +int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n> +\t\t struct repository *repo)\n> +{\n> +\tstruct strbuf path_dst = STRBUF_INIT;\n> +\tstruct strbuf path = STRBUF_INIT;\n> +\tstruct strbuf meta = STRBUF_INIT;\n> +\tstruct rev_info revs;\n> +\tint ret;\n> +\n> +\tconst char * const usage[] = {\n> +\t\tN_(\"git diff-pairs -z []\"),\n> +\t\tNULL\n> +\t};\n> +\tstruct option options[] = {\n> +\t\tOPT_END()\n> +\t};\n> +\tstruct option *parseopts = add_diff_options(options, &revs.diffopt);\n> +\n> +\tshow_usage_with_options_if_asked(argc, argv, usage, parseopts);\n> +\n> +\trepo_init_revisions(repo, &revs, prefix);\n> +\trepo_config(repo, git_diff_basic_config, NULL);\n> +\trevs.disable_stdin = 1;\n> +\trevs.abbrev = 0;\n> +\trevs.diff = 1;\n> +\n> +\tif (setup_revisions(argc, argv, &revs, NULL) > 1)\n> +\t\tusage_with_options(usage, parseopts);\n> +\n> +\t/*\n> +\t * With the -z option, both command input and raw output are\n> +\t * NUL-delimited (this mode does not effect patch output). At present\n\nProbably \"effect\" -> \"affect\".\n\n> +\t * only NUL-delimited raw diff formatted input is supported.\n> +\t */\n> +\tif (revs.diffopt.line_termination) {\n> +\t\terror(_(\"working without -z is not supported\"));\n> +\t\tusage_with_options(usage, parseopts);\n> +\t}\n> +\n> +\tif (revs.prune_data.nr) {\n> +\t\terror(_(\"pathspec arguments not supported\"));\n> +\t\tusage_with_options(usage, parseopts);\n> +\t}\n> +\n> +\tif (revs.pending.nr || revs.max_count != -1 ||\n> +\t revs.min_age != (timestamp_t)-1 ||\n> +\t revs.max_age != (timestamp_t)-1) {\n> +\t\terror(_(\"revision arguments not allowed\"));\n> +\t\tusage_with_options(usage, parseopts);\n> +\t}\n> +\n> +\tif (!revs.diffopt.output_format)\n> +\t\trevs.diffopt.output_format = DIFF_FORMAT_PATCH;\n> +\n> +\twhile (1) {\n> +\t\tstruct object_id oid_a, oid_b;\n> +\t\tstruct diff_filepair *pair;\n> +\t\tunsigned mode_a, mode_b;\n> +\t\tconst char *p;\n> +\t\tchar status;\n> +\n> +\t\tif (strbuf_getline_nul(&meta, stdin) == EOF)\n> +\t\t\tbreak;\n\nThere should be a variant of this function that takes delimiter\nparameter. By declaring an int variable that is initialized to '\\0'\n(because you only deal with \"-z\" input) and passing that delimiter\nvariable to strbuf_getwholeline() would future-proof this code path.\n\nHow builtin/update-ref.c:update_refs_stdin() works may be inspiring.\n\n> +\t\tswitch (status) {\n> +\t\tcase DIFF_STATUS_ADDED:\n> +\t\t\tpair = diff_queue_addremove(&diff_queued_diff,\n> +\t\t\t\t\t\t &revs.diffopt, '+', mode_b,\n> +\t\t\t\t\t\t &oid_b, 1, path.buf, 0);\n> +\t\t\tif (pair)\n> +\t\t\t\tpair->status = status;\n> +\t\t\tbreak;\n> + ...\n> +\t\tdefault:\n> +\t\t\tdie(_(\"unknown diff status: %c\"), status);\n> +\t\t}\n> +\t}\n\nAmusing; looking good.\n\n> +\tflush_diff_queue(&revs.diffopt);\n> +\tret = diff_result_code(&revs);\n> +\n> +\tstrbuf_release(&path_dst);\n> +\tstrbuf_release(&path);\n> +\tstrbuf_release(&meta);\n> +\trelease_revisions(&revs);\n> +\tFREE_AND_NULL(parseopts);\n> +\n> +\treturn ret;\n> +}\n\nNice. It is surprisingly compact and had everything I expected it\nto have ;-).\n\n> +test_expect_success 'diff-pairs recreates --raw' '\n> +\tgit diff-tree -r -M -C -C -z base new >expect &&\n> +\tgit diff-tree -r -M -C -C -z base new |\n> +\tgit diff-pairs --raw -z >actual &&\n> +\ttest_cmp expect actual\n> +'\n\nAmusing ;-) But a very obvious and important thing to test.\nI would have fed writes:\n\n> Through git-diff(1), a single diff can be generated from a pair of blob\n> revisions directly. Unfortunately, there is not a mechanism to compute\n> batches of specific file pair diffs in a single process. Such a feature\n> is particularly useful on the server-side where diffing between a large\n> set of changes is not feasible all at once due to timeout concerns.\n>\n> To facilitate this, introduce git-diff-pairs(1) which acts as a backend\n> passing its NUL-terminated raw diff format input from stdin through diff\n> machinery to produce various forms of output such as patch or raw.\n>\n> The raw format was originally designed as an interchange format and\n> represents the contents of the diff_queue_diff list making it possible\n> to break the diff pipeline into separate stages. For example,\n> git-diff-tree(1) can be used as a frontend to compute file pairs to\n> queue and feed its raw output to git-diff-pairs(1) to compute patches.\n> With this, batches of diffs can be progessively generated without having\n\ns/progessively/progressively\n\n> to recompute rename detection or retrieve object context. Something like\n> the following:\n>\n> \tgit diff-tree -r -z -M $old $new |\n> \tgit diff-pairs -p -z\n>\n> should generate the same output as `git diff-tree -p -M`. Furthermore,\n> each line of raw diff formatted input can also be individually fed to a\n> separate git-diff-pairs(1) process and still produce the same output.\n>\n> Based-on-patch-by: Jeff King \n> Signed-off-by: Justin Tobler \n\n[snip]\n\n> diff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\n> new file mode 100644\n> index 0000000000..9472b10461\n> --- /dev/null\n> +++ b/builtin/diff-pairs.c\n> @@ -0,0 +1,193 @@\n> +#include \"builtin.h\"\n> +#include \"commit.h\"\n> +#include \"config.h\"\n> +#include \"diff.h\"\n> +#include \"diffcore.h\"\n> +#include \"gettext.h\"\n> +#include \"hex.h\"\n> +#include \"object.h\"\n> +#include \"parse-options.h\"\n> +#include \"revision.h\"\n> +#include \"strbuf.h\"\n> +\n\nNit: I could also compile without some of these headers, do we still\nneed them all?\n\n diff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\n index 86e59a7e3a..1aea2ee726 100644\n --- a/builtin/diff-pairs.c\n +++ b/builtin/diff-pairs.c\n @@ -1,14 +1,9 @@\n #include \"builtin.h\"\n -#include \"commit.h\"\n #include \"config.h\"\n -#include \"diff.h\"\n #include \"diffcore.h\"\n -#include \"gettext.h\"\n #include \"hex.h\"\n -#include \"object.h\"\n #include \"parse-options.h\"\n #include \"revision.h\"\n -#include \"strbuf.h\"\n\n static unsigned parse_mode_or_die(const char *mode, const char **endp)\n {\n\n> +static unsigned parse_mode_or_die(const char *mode, const char **endp)\n> +{\n> +\tuint16_t ret;\n> +\n> +\t*endp = parse_mode(mode, &ret);\n> +\tif (!*endp)\n> +\t\tdie(_(\"unable to parse mode: %s\"), mode);\n> +\treturn ret;\n> +}\n> +\n> +static void parse_oid_or_die(const char *p, struct object_id *oid,\n> +\t\t\t const char **endp, const struct git_hash_algo *algop)\n>\n\nNit: without double checking, I couldn't tell what 'p' was, can we\nrename the variables here to be consistent with `parse_oid_hex_algop()`?\n\n> +{\n> +\tif (parse_oid_hex_algop(p, oid, endp, algop) || *(*endp)++ != ' ')\n> +\t\tdie(_(\"unable to parse object id: %s\"), p);\n> +}\n> +\n> +static void flush_diff_queue(struct diff_options *options)\n> +{\n> +\t/*\n> +\t * If rename detection is not requested, use rename information from the\n> +\t * raw diff formatted input. Setting found_follow ensures diffcore_std()\n> +\t * does not mess with rename information already present in queued\n> +\t * filepairs.\n> +\t */\n> +\tif (!options->detect_rename)\n> +\t\toptions->found_follow = 1;\n> +\tdiffcore_std(options);\n> +\tdiff_flush(options);\n> +}\n> +\n> +int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n> +\t\t struct repository *repo)\n> +{\n> +\tstruct strbuf path_dst = STRBUF_INIT;\n> +\tstruct strbuf path = STRBUF_INIT;\n> +\tstruct strbuf meta = STRBUF_INIT;\n> +\tstruct rev_info revs;\n> +\tint ret;\n> +\n> +\tconst char * const usage[] = {\n> +\t\tN_(\"git diff-pairs -z []\"),\n> +\t\tNULL\n> +\t};\n> +\tstruct option options[] = {\n> +\t\tOPT_END()\n> +\t};\n> +\tstruct option *parseopts = add_diff_options(options, &revs.diffopt);\n> +\n> +\tshow_usage_with_options_if_asked(argc, argv, usage, parseopts);\n> +\n> +\trepo_init_revisions(repo, &revs, prefix);\n> +\trepo_config(repo, git_diff_basic_config, NULL);\n> +\trevs.disable_stdin = 1;\n> +\trevs.abbrev = 0;\n> +\trevs.diff = 1;\n> +\n> +\tif (setup_revisions(argc, argv, &revs, NULL) > 1)\n> +\t\tusage_with_options(usage, parseopts);\n> +\n> +\t/*\n> +\t * With the -z option, both command input and raw output are\n> +\t * NUL-delimited (this mode does not effect patch output). At present\n> +\t * only NUL-delimited raw diff formatted input is supported.\n> +\t */\n> +\tif (revs.diffopt.line_termination) {\n> +\t\terror(_(\"working without -z is not supported\"));\n> +\t\tusage_with_options(usage, parseopts);\n> +\t}\n> +\n> +\tif (revs.prune_data.nr) {\n> +\t\terror(_(\"pathspec arguments not supported\"));\n> +\t\tusage_with_options(usage, parseopts);\n> +\t}\n> +\n> +\tif (revs.pending.nr || revs.max_count != -1 ||\n> +\t revs.min_age != (timestamp_t)-1 ||\n> +\t revs.max_age != (timestamp_t)-1) {\n> +\t\terror(_(\"revision arguments not allowed\"));\n> +\t\tusage_with_options(usage, parseopts);\n> +\t}\n> +\n> +\tif (!revs.diffopt.output_format)\n> +\t\trevs.diffopt.output_format = DIFF_FORMAT_PATCH;\n> +\n> +\twhile (1) {\n> +\t\tstruct object_id oid_a, oid_b;\n> +\t\tstruct diff_filepair *pair;\n> +\t\tunsigned mode_a, mode_b;\n> +\t\tconst char *p;\n> +\t\tchar status;\n> +\n> +\t\tif (strbuf_getline_nul(&meta, stdin) == EOF)\n> +\t\t\tbreak;\n> +\n> +\t\tp = meta.buf;\n> +\t\tif (*p != ':')\n> +\t\t\tdie(_(\"invalid raw diff input\"));\n> +\t\tp++;\n> +\n> +\t\tmode_a = parse_mode_or_die(p, &p);\n> +\t\tmode_b = parse_mode_or_die(p, &p);\n> +\n> +\t\tif (S_ISDIR(mode_a) || S_ISDIR(mode_b))\n> +\t\t\tdie(_(\"tree objects not supported\"));\n> +\n> +\t\tparse_oid_or_die(p, &oid_a, &p, repo->hash_algo);\n> +\t\tparse_oid_or_die(p, &oid_b, &p, repo->hash_algo);\n> +\n> +\t\tstatus = *p++;\n> +\n> +\t\tif (strbuf_getline_nul(&path, stdin) == EOF)\n> +\t\t\tdie(_(\"got EOF while reading path\"));\n> +\n> +\t\tswitch (status) {\n> +\t\tcase DIFF_STATUS_ADDED:\n> +\t\t\tpair = diff_queue_addremove(&diff_queued_diff,\n> +\t\t\t\t\t\t &revs.diffopt, '+', mode_b,\n> +\t\t\t\t\t\t &oid_b, 1, path.buf, 0);\n> +\t\t\tif (pair)\n> +\t\t\t\tpair->status = status;\n> +\t\t\tbreak;\n> +\n> +\t\tcase DIFF_STATUS_DELETED:\n> +\t\t\tpair = diff_queue_addremove(&diff_queued_diff,\n> +\t\t\t\t\t\t &revs.diffopt, '-', mode_a,\n> +\t\t\t\t\t\t &oid_a, 1, path.buf, 0);\n> +\t\t\tif (pair)\n> +\t\t\t\tpair->status = status;\n> +\t\t\tbreak;\n> +\n> +\t\tcase DIFF_STATUS_TYPE_CHANGED:\n> +\t\tcase DIFF_STATUS_MODIFIED:\n> +\t\t\tpair = diff_queue_change(&diff_queued_diff, &revs.diffopt,\n> +\t\t\t\t\t\t mode_a, mode_b, &oid_a, &oid_b,\n> +\t\t\t\t\t\t 1, 1, path.buf, 0, 0);\n> +\t\t\tif (pair)\n> +\t\t\t\tpair->status = status;\n> +\t\t\tbreak;\n> +\n> +\t\tcase DIFF_STATUS_RENAMED:\n> +\t\tcase DIFF_STATUS_COPIED:\n> +\t\t\t{\n\nstyle: The general rule followed is to open the braces in the same line\nas the case statement. So `case DIFF_STATUS_COPIED: {`\n\n> +\t\t\t\tstruct diff_filespec *a, *b;\n> +\t\t\t\tunsigned int score;\n> +\n> +\t\t\t\tif (strbuf_getline_nul(&path_dst, stdin) == EOF)\n> +\t\t\t\t\tdie(_(\"got EOF while reading destination path\"));\n> +\n> +\t\t\t\ta = alloc_filespec(path.buf);\n> +\t\t\t\tb = alloc_filespec(path_dst.buf);\n> +\t\t\t\tfill_filespec(a, &oid_a, 1, mode_a);\n> +\t\t\t\tfill_filespec(b, &oid_b, 1, mode_b);\n> +\n> +\t\t\t\tpair = diff_queue(&diff_queued_diff, a, b);\n> +\n> +\t\t\t\tif (strtoul_ui(p, 10, &score))\n> +\t\t\t\t\tdie(_(\"unable to parse rename/copy score: %s\"), p);\n> +\n> +\t\t\t\tpair->score = score * MAX_SCORE / 100;\n> +\t\t\t\tpair->status = status;\n> +\t\t\t\tpair->renamed_pair = 1;\n> +\t\t\t}\n> +\t\t\tbreak;\n> +\n> +\t\tdefault:\n> +\t\t\tdie(_(\"unknown diff status: %c\"), status);\n> +\t\t}\n> +\t}\n> +\n> +\tflush_diff_queue(&revs.diffopt);\n> +\tret = diff_result_code(&revs);\n> +\n> +\tstrbuf_release(&path_dst);\n> +\tstrbuf_release(&path);\n> +\tstrbuf_release(&meta);\n> +\trelease_revisions(&revs);\n> +\tFREE_AND_NULL(parseopts);\n> +\n> +\treturn ret;\n> +}\n\n[snip]\n"},{"id":"513145","messageId":"Z8Bg5EAArZVGPaAc@pks.im","threadId":"62633","inReplyTo":"20250225233925.1345086-3-jltobler@gmail.com","subject":"Re: [PATCH v3 2/3] builtin: introduce diff-pairs command","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-27T12:56:04Z","receivedAt":"2025-02-27T12:56:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 25, 2025 at 05:39:24PM -0600, Justin Tobler wrote:\n> Through git-diff(1), a single diff can be generated from a pair of blob\n> revisions directly. Unfortunately, there is not a mechanism to compute\n> batches of specific file pair diffs in a single process. Such a feature\n> is particularly useful on the server-side where diffing between a large\n> set of changes is not feasible all at once due to timeout concerns.\n> \n> To facilitate this, introduce git-diff-pairs(1) which acts as a backend\n> passing its NUL-terminated raw diff format input from stdin through diff\n> machinery to produce various forms of output such as patch or raw.\n> \n> The raw format was originally designed as an interchange format and\n> represents the contents of the diff_queue_diff list making it possible\n> to break the diff pipeline into separate stages. For example,\n> git-diff-tree(1) can be used as a frontend to compute file pairs to\n> queue and feed its raw output to git-diff-pairs(1) to compute patches.\n> With this, batches of diffs can be progessively generated without having\n> to recompute rename detection or retrieve object context. Something like\n\ns/rename detection/renames/\n\n> diff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\n> new file mode 100644\n> index 0000000000..9472b10461\n> --- /dev/null\n> +++ b/builtin/diff-pairs.c\n> @@ -0,0 +1,193 @@\n> +#include \"builtin.h\"\n> +#include \"commit.h\"\n> +#include \"config.h\"\n> +#include \"diff.h\"\n> +#include \"diffcore.h\"\n> +#include \"gettext.h\"\n> +#include \"hex.h\"\n> +#include \"object.h\"\n> +#include \"parse-options.h\"\n> +#include \"revision.h\"\n> +#include \"strbuf.h\"\n> +\n> +static unsigned parse_mode_or_die(const char *mode, const char **endp)\n> +{\n> +\tuint16_t ret;\n> +\n> +\t*endp = parse_mode(mode, &ret);\n> +\tif (!*endp)\n> +\t\tdie(_(\"unable to parse mode: %s\"), mode);\n> +\treturn ret;\n> +}\n> +\n> +static void parse_oid_or_die(const char *p, struct object_id *oid,\n> +\t\t\t const char **endp, const struct git_hash_algo *algop)\n> +{\n> +\tif (parse_oid_hex_algop(p, oid, endp, algop) || *(*endp)++ != ' ')\n> +\t\tdie(_(\"unable to parse object id: %s\"), p);\n> +}\n> +\n> +static void flush_diff_queue(struct diff_options *options)\n> +{\n> +\t/*\n> +\t * If rename detection is not requested, use rename information from the\n> +\t * raw diff formatted input. Setting found_follow ensures diffcore_std()\n> +\t * does not mess with rename information already present in queued\n> +\t * filepairs.\n> +\t */\n> +\tif (!options->detect_rename)\n> +\t\toptions->found_follow = 1;\n\nIt's a bit weird that we set this over here. Shouldn't we have set it up\nin the main function already?\n\n> +\tdiffcore_std(options);\n> +\tdiff_flush(options);\n> +}\n> +\n> +int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n> +\t\t struct repository *repo)\n> +{\n> +\tstruct strbuf path_dst = STRBUF_INIT;\n> +\tstruct strbuf path = STRBUF_INIT;\n> +\tstruct strbuf meta = STRBUF_INIT;\n> +\tstruct rev_info revs;\n> +\tint ret;\n> +\n> +\tconst char * const usage[] = {\n> +\t\tN_(\"git diff-pairs -z []\"),\n> +\t\tNULL\n> +\t};\n> +\tstruct option options[] = {\n> +\t\tOPT_END()\n> +\t};\n> +\tstruct option *parseopts = add_diff_options(options, &revs.diffopt);\n> +\n> +\tshow_usage_with_options_if_asked(argc, argv, usage, parseopts);\n\nDon't we also have to call `parse_options()` even though we don't have\nour own options yet? Or is this all handled by `setup_revisions()`?\n\n> +\trepo_init_revisions(repo, &revs, prefix);\n> +\trepo_config(repo, git_diff_basic_config, NULL);\n> +\trevs.disable_stdin = 1;\n> +\trevs.abbrev = 0;\n> +\trevs.diff = 1;\n> +\n> +\tif (setup_revisions(argc, argv, &revs, NULL) > 1)\n> +\t\tusage_with_options(usage, parseopts);\n\nI think it's discouraged nowadays to use `usage_with_options()` as it\ngenerates a ton of noise while hiding the actual error message. It is\ninstead recommended to directly call `usage()` with an error message.\n\nIn this case here we would say e.g. `usage(_(\"unrecognized argument:\n%s\"), argv[0])`, in the cases below we'd use the error messages you\nalready have.\n\n> +\n> +\t/*\n> +\t * With the -z option, both command input and raw output are\n> +\t * NUL-delimited (this mode does not effect patch output). At present\n> +\t * only NUL-delimited raw diff formatted input is supported.\n> +\t */\n> +\tif (revs.diffopt.line_termination) {\n> +\t\terror(_(\"working without -z is not supported\"));\n> +\t\tusage_with_options(usage, parseopts);\n> +\t}\n> +\n> +\tif (revs.prune_data.nr) {\n> +\t\terror(_(\"pathspec arguments not supported\"));\n> +\t\tusage_with_options(usage, parseopts);\n> +\t}\n> +\n> +\tif (revs.pending.nr || revs.max_count != -1 ||\n> +\t revs.min_age != (timestamp_t)-1 ||\n> +\t revs.max_age != (timestamp_t)-1) {\n> +\t\terror(_(\"revision arguments not allowed\"));\n> +\t\tusage_with_options(usage, parseopts);\n> +\t}\n\nOkay. We restrict a bunch of usages, which makes your job simpler right\nnow, but by dying it keeps the door open to iterate on those in the\nfuture.\n\n> +\tif (!revs.diffopt.output_format)\n> +\t\trevs.diffopt.output_format = DIFF_FORMAT_PATCH;\n\nInstead of setting this conditionally, can we already set it up as a\ndefault before calling `setup_revisions()`?\n\n> +\twhile (1) {\n> +\t\tstruct object_id oid_a, oid_b;\n> +\t\tstruct diff_filepair *pair;\n> +\t\tunsigned mode_a, mode_b;\n> +\t\tconst char *p;\n> +\t\tchar status;\n> +\n> +\t\tif (strbuf_getline_nul(&meta, stdin) == EOF)\n> +\t\t\tbreak;\n> +\n> +\t\tp = meta.buf;\n> +\t\tif (*p != ':')\n> +\t\t\tdie(_(\"invalid raw diff input\"));\n> +\t\tp++;\n> +\n> +\t\tmode_a = parse_mode_or_die(p, &p);\n> +\t\tmode_b = parse_mode_or_die(p, &p);\n> +\n> +\t\tif (S_ISDIR(mode_a) || S_ISDIR(mode_b))\n> +\t\t\tdie(_(\"tree objects not supported\"));\n\nI assume submodules aren't supported either, are they? If so, do we also\nhave to check for `S_ISGITLINK()`? It would be nice to have a test for\nthem.\n\nPatrick\n"},{"id":"513188","messageId":"b3a4lco523aodf3oqctf33utcp6372b3qp6sgekeb4dx2wbtya@huxwtr4vw5md","threadId":"62633","inReplyTo":"1e41249a-5241-4cd8-8a6a-3c9163fb0ea0@gmail.com","subject":"Re: [PATCH v3 0/3] batch blob diff generation","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-27T22:04:27Z","receivedAt":"2025-02-27T22:07:44Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/26 02:58PM, phillip.wood123@gmail.com wrote:\n> I also think we might want to massage the output in the tests so that we're\n> not running test_cmp on files containing NUL bytes. Using\n> \n> git diff-tree -z ... | tr '\\0' Q >actual\n> \n> would get rid of the NULs but does not improve the readability of the raw\n> diffs that much as everything is still on a single line. Using '\\n' instead\n> of 'Q' would give us mulit-line output but we would lose confidence that the\n> original output was actually NUL terminated.\n\nIs the underlying motivation here to provide more feedback if a test\nfails? I somewhat have a preference for the test to be validating the\noutput as it is actually expected. As you mentioned, getting rid of the\nNUL bytes wouldn't help with readability much and we probably wouldn't\nwant to replace with `\\n`, so maybe a simple \"Binary files expect and\nactual differ\" would be the most straightforward.\n\nIf this is the preferred way to handle it, I can adapt in a followup\nversion though. :)\n\nThanks for the review!\n-Justin\n"},{"id":"513189","messageId":"wxr2memiox7l4ljz4wozxlxtkd2qms7crzsztb2vim6f5jjw4s@jm5retwyxhib","threadId":"62633","inReplyTo":"xmqqjz9cd0nl.fsf@gitster.g","subject":"Re: [PATCH v3 2/3] builtin: introduce diff-pairs command","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-27T22:15:36Z","receivedAt":"2025-02-27T22:18:53Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/26 10:24AM, Junio C Hamano wrote:\n> Justin Tobler writes:\n> \n> > +static void flush_diff_queue(struct diff_options *options)\n> > +{\n> > +\t/*\n> > +\t * If rename detection is not requested, use rename information from the\n> > +\t * raw diff formatted input. Setting found_follow ensures diffcore_std()\n> > +\t * does not mess with rename information already present in queued\n> > +\t * filepairs.\n> > +\t */\n> > +\tif (!options->detect_rename)\n> > +\t\toptions->found_follow = 1;\n> \n> An ugly design decision that may be suboptimal from maintainability\n> point of view.\n> \n> The parts of diffcore_std() that --follow wants to bypass may happen\n> to be the same as the parts that this new caller wants to bypass,\n> but who guarantees that they will stay that way in the future?\n\nGood point. When invoking diffcore_std(), we really just need to be able\nto skip diff_resolve_rename_copy() as that is what is updating the diff\nfilepair statuses. In the next version, instead of relying on\n`found_follow`, I think I'll introduce a new diff_options field,\n`skip_resolving_statuses` for this specific purpose.\n\n> > +\twhile (1) {\n> > +\t\tstruct object_id oid_a, oid_b;\n> > +\t\tstruct diff_filepair *pair;\n> > +\t\tunsigned mode_a, mode_b;\n> > +\t\tconst char *p;\n> > +\t\tchar status;\n> > +\n> > +\t\tif (strbuf_getline_nul(&meta, stdin) == EOF)\n> > +\t\t\tbreak;\n> \n> There should be a variant of this function that takes delimiter\n> parameter. By declaring an int variable that is initialized to '\\0'\n> (because you only deal with \"-z\" input) and passing that delimiter\n> variable to strbuf_getwholeline() would future-proof this code path.\n> \n> How builtin/update-ref.c:update_refs_stdin() works may be inspiring.\n\nMakes sense, I'll swap to using strbuf_getwholeline() with a defined\nline terminator variable in the next version. This way it can help make\nsupporting the \"normal\" raw diff format as input easier in the future.\n\n> > +test_expect_success 'diff-pairs recreates --raw' '\n> > +\tgit diff-tree -r -M -C -C -z base new >expect &&\n> > +\tgit diff-tree -r -M -C -C -z base new |\n> > +\tgit diff-pairs --raw -z >actual &&\n> > +\ttest_cmp expect actual\n> > +'\n> \n> Amusing ;-) But a very obvious and important thing to test.\n> I would have fed Other than that, nicely done.\n\nThanks for the review!\n-Justin\n"},{"id":"513191","messageId":"z5mea66lqrhcjm2da5qyg7e4n3wvxsilhtta45qie5m6hub5o4@ftdgftsmaeul","threadId":"62633","inReplyTo":"CAOLa=ZS6yqquxAL_vCUf0FRrMbA6OLeY1VjqMv=pnxAo_17zLA@mail.gmail.com","subject":"Re: [PATCH v3 2/3] builtin: introduce diff-pairs command","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-27T22:36:21Z","receivedAt":"2025-02-27T22:39:38Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/27 01:35AM, Karthik Nayak wrote:\n> > diff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\n> > new file mode 100644\n> > index 0000000000..9472b10461\n> > --- /dev/null\n> > +++ b/builtin/diff-pairs.c\n> > @@ -0,0 +1,193 @@\n> > +#include \"builtin.h\"\n> > +#include \"commit.h\"\n> > +#include \"config.h\"\n> > +#include \"diff.h\"\n> > +#include \"diffcore.h\"\n> > +#include \"gettext.h\"\n> > +#include \"hex.h\"\n> > +#include \"object.h\"\n> > +#include \"parse-options.h\"\n> > +#include \"revision.h\"\n> > +#include \"strbuf.h\"\n> > +\n> \n> Nit: I could also compile without some of these headers, do we still\n> need them all?\n> \n> diff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\n> index 86e59a7e3a..1aea2ee726 100644\n> --- a/builtin/diff-pairs.c\n> +++ b/builtin/diff-pairs.c\n> @@ -1,14 +1,9 @@\n> #include \"builtin.h\"\n> -#include \"commit.h\"\n\nLooks like this one is unneeded. Will remove\n\n> #include \"config.h\"\n> -#include \"diff.h\"\n> #include \"diffcore.h\"\n> -#include \"gettext.h\"\n> #include \"hex.h\"\n> -#include \"object.h\"\n> #include \"parse-options.h\"\n> #include \"revision.h\"\n> -#include \"strbuf.h\"\n\nThe others are directly referenced. I think it would be preferable to\nexplicitly state them instead of relying on them being included\ntransitively.\n\n> \n> static unsigned parse_mode_or_die(const char *mode, const char **endp)\n> {\n> \n> > +static unsigned parse_mode_or_die(const char *mode, const char **endp)\n> > +{\n> > +\tuint16_t ret;\n> > +\n> > +\t*endp = parse_mode(mode, &ret);\n> > +\tif (!*endp)\n> > +\t\tdie(_(\"unable to parse mode: %s\"), mode);\n> > +\treturn ret;\n> > +}\n> > +\n> > +static void parse_oid_or_die(const char *p, struct object_id *oid,\n> > +\t\t\t const char **endp, const struct git_hash_algo *algop)\n> >\n> \n> Nit: without double checking, I couldn't tell what 'p' was, can we\n> rename the variables here to be consistent with `parse_oid_hex_algop()`?\n\nWill update\n\n> > +\t\tcase DIFF_STATUS_RENAMED:\n> > +\t\tcase DIFF_STATUS_COPIED:\n> > +\t\t\t{\n> \n> style: The general rule followed is to open the braces in the same line\n> as the case statement. So `case DIFF_STATUS_COPIED: {`\n\nWill fix in the next version.\n\nThanks\n-Justin\n"},{"id":"513194","messageId":"qnd5cou3mhareqvxhndgwkomn7t3kwdn3xd55wqn6vidly33uk@5yq4cv2yfnbh","threadId":"62633","inReplyTo":"Z8Bg5EAArZVGPaAc@pks.im","subject":"Re: [PATCH v3 2/3] builtin: introduce diff-pairs command","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-27T23:00:10Z","receivedAt":"2025-02-27T23:03:28Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/27 01:56PM, Patrick Steinhardt wrote:\n> > +static void flush_diff_queue(struct diff_options *options)\n> > +{\n> > +\t/*\n> > +\t * If rename detection is not requested, use rename information from the\n> > +\t * raw diff formatted input. Setting found_follow ensures diffcore_std()\n> > +\t * does not mess with rename information already present in queued\n> > +\t * filepairs.\n> > +\t */\n> > +\tif (!options->detect_rename)\n> > +\t\toptions->found_follow = 1;\n> \n> It's a bit weird that we set this over here. Shouldn't we have set it up\n> in the main function already?\n\nEverytime diffcore_std() is invoked found_follow gets reset. This was\nincluded here to ensure the correct value is always set.\n\nIn the next version I am going to move away from using found_follow in\nfavor of a new diff_options field to avoid some of this awkwardness\naltogether.\n\n> > +\tdiffcore_std(options);\n> > +\tdiff_flush(options);\n> > +}\n> > +\n> > +int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n> > +\t\t struct repository *repo)\n> > +{\n> > +\tstruct strbuf path_dst = STRBUF_INIT;\n> > +\tstruct strbuf path = STRBUF_INIT;\n> > +\tstruct strbuf meta = STRBUF_INIT;\n> > +\tstruct rev_info revs;\n> > +\tint ret;\n> > +\n> > +\tconst char * const usage[] = {\n> > +\t\tN_(\"git diff-pairs -z []\"),\n> > +\t\tNULL\n> > +\t};\n> > +\tstruct option options[] = {\n> > +\t\tOPT_END()\n> > +\t};\n> > +\tstruct option *parseopts = add_diff_options(options, &revs.diffopt);\n> > +\n> > +\tshow_usage_with_options_if_asked(argc, argv, usage, parseopts);\n> \n> Don't we also have to call `parse_options()` even though we don't have\n> our own options yet? Or is this all handled by `setup_revisions()`?\n\nIn the current implementation, the diff options that get appended are\nonly really used so that the usage message prints with the diff option\ninfo. It still relies on setup_revisions() to actually parse the\noptions. Since there are not any real options that need parsing,\nparse_options() was not invoked.\n\nThis is fairly confusing though. I plan to instead parse the diff\noptions upfront with parse_options(). The diff options parsing through\nsetup_revisions() becomes effectively a no-op. I think this makes more\nsense to read and still lets us print the common diff options is the\nusage message.\n\n> > +\trepo_init_revisions(repo, &revs, prefix);\n> > +\trepo_config(repo, git_diff_basic_config, NULL);\n> > +\trevs.disable_stdin = 1;\n> > +\trevs.abbrev = 0;\n> > +\trevs.diff = 1;\n> > +\n> > +\tif (setup_revisions(argc, argv, &revs, NULL) > 1)\n> > +\t\tusage_with_options(usage, parseopts);\n> \n> I think it's discouraged nowadays to use `usage_with_options()` as it\n> generates a ton of noise while hiding the actual error message. It is\n> instead recommended to directly call `usage()` with an error message.\n> \n> In this case here we would say e.g. `usage(_(\"unrecognized argument:\n> %s\"), argv[0])`, in the cases below we'd use the error messages you\n> already have.\n\nGood to know. I'll avoid printing the usage options message in all these\nfailure scenarios in favor of what you suggested.\n\n> > +\tif (!revs.diffopt.output_format)\n> > +\t\trevs.diffopt.output_format = DIFF_FORMAT_PATCH;\n> \n> Instead of setting this conditionally, can we already set it up as a\n> default before calling `setup_revisions()`?\n\nThe diff output format is set via OPT_BITOP() and thus can have multiple\nvalues at the same time. For example:\n\n $ git diff-tree --raw --patch HEAD\n\nwill render both patch and raw output. If we unconditionally set\nDIFF_FORMAT_PATCH, it will always be included in the output which is not\nwhat we want. We only want to set DIFF_FORMAT_PATCH if there is still no\nvalue after all options parsing has occurred.\n\n> > +\twhile (1) {\n> > +\t\tstruct object_id oid_a, oid_b;\n> > +\t\tstruct diff_filepair *pair;\n> > +\t\tunsigned mode_a, mode_b;\n> > +\t\tconst char *p;\n> > +\t\tchar status;\n> > +\n> > +\t\tif (strbuf_getline_nul(&meta, stdin) == EOF)\n> > +\t\t\tbreak;\n> > +\n> > +\t\tp = meta.buf;\n> > +\t\tif (*p != ':')\n> > +\t\t\tdie(_(\"invalid raw diff input\"));\n> > +\t\tp++;\n> > +\n> > +\t\tmode_a = parse_mode_or_die(p, &p);\n> > +\t\tmode_b = parse_mode_or_die(p, &p);\n> > +\n> > +\t\tif (S_ISDIR(mode_a) || S_ISDIR(mode_b))\n> > +\t\t\tdie(_(\"tree objects not supported\"));\n> \n> I assume submodules aren't supported either, are they? If so, do we also\n> have to check for `S_ISGITLINK()`? It would be nice to have a test for\n> them.\n\nSubmodules should actually be supported as I believe all the info\npresent in the raw formatted input should be enough to properly display\npatch output. I'll add a submodule to the existing test setup to\nvalidate.\n\nThanks\n-Justin\n"},{"id":"513207","messageId":"20250228002604.3859939-1-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250225233925.1345086-1-jltobler@gmail.com","subject":"[PATCH v4 0/4] batch blob diff generation","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-28T00:26:00Z","receivedAt":"2025-02-28T00:29:26Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Through git-diff(1) it is possible to generate a diff directly between\ntwo blobs. This is particularly useful when the pre-image and post-image\nblobs are known and we only care about the diff between them.\nUnfortunately, if a user has a batch of known blob pairs to compute\ndiffs for, there is currently not a way to do so via a single Git\nprocess.\n\nTo enable support for batch diffs of multiple blob pairs, this series\nintroduces a new diff plumbing command git-diff-pairs(1) based on a\nprevious patch series submitted by Peff[1]. This command uses\nNUL-delimited raw diffs as its source of input to control exactly which\nfilepairs are diffed. The advantage of using the raw diff format is that\nit already has diff status type and object context information embedded\nin each line making it more efficient to generate diffs with as we can\navoid having to peel revisions to get some the same info.\n\nFor example:\n\n git diff-tree -r -z -M $old $new |\n git diff-pairs -p -z\n\nHere the output of git-diff-tree(1) is fed to git-diff-pairs(1) to\ngenerate the same output that would be expected from `git diff-tree -p\n-M`. While by itself not particularly useful, this means it is possible\nto split git-diff-tree(1) output across multiple git-diff-pairs(1)\nprocesses. Such a feature is useful on the server-side where diffs\nbewteen a large set of changes may not be feasible all at once due to\ntimeout concerns.\n\nThis command can be viewed as a backend tool that exposes Git's diff\nmachinery. In its current form, the frontend that generates the raw diff\nlines used as input is expected to most of the heavy lifting (ie.\npathspec limiting, tree object expansion).\n\nThis series is structured as follows:\n\n - Patch 1 adds some new helper functions to get access to the queued\n `diff_filepair` after `diff_queue()` is invoked.\n\n - Patch 2 adds a new diff_options field that can be used to disable\n diff filepair status resolution. This prevents rename/copy\n statuses set from stdin from being altered when `diffcore_std()`\n is invoked.\n\n - Patch 3 introduces the new git-diff-pairs(1) plumbing command.\n\n - Patch 4 allows git-diff-pairs(1) to immediately compute diffs\n queued on stdin when a NUL-byte is written after a raw input line\n instead of waiting for stdin to close.\n\nChanges since V3:\n\n - Instead of relying on found_follow to prevent `diffcore_std()`\n from mutating diff filepair statuses, a new `diff_options` field,\n `skip_resolving_statuses` is introduced to achieve the same result\n in a more specific manner.\n\n - Parsing of diff options is now handled directly instead of going\n through `setup_revisions()`. This is done to so the diff options\n can be appended to the usage options and printed in the usage\n message.\n\n - Swapped to using `strbuf_getwholeline()` during stdin parsing to\n make the line termiantor more configurable in the future.\n\n - Stopped printing the usage message on errors to avoid masking the\n underlying error message.\n\n - Added test setup to exercise submodule change diffs.\n\n - Other small minor cleanups.\n\nChanges since V2:\n\n - Pathspecs are not supported and thus rejected when provided as\n arguments. It should be possible in a future series to add support\n though.\n\n - Tree objects present in `diff-pairs` input are rejected. Support\n for tree objects could be added in the future, but for now they\n are rejected to enable to future support in a backwards compatible\n manner.\n\n - The -z option is required by git-diff-pairs(1). The NUL-delimited\n raw diff format is the only accepted form of input. Consequently,\n NUL-delimited output is the only option in the `--raw` mode.\n\n - git-diff-pairs(1) defaults to patch output instead of raw output.\n This better fits the intended usecase of the command.\n\n - A NUL-byte is now always used as the delimiter between batches of\n file pair diffs when queued diffs are explicitly computed by\n writing a NUL-byte on stdin.\n\n - Several other small cleanups and fixes along with documentation\n changes.\n\nChanges since V1:\n\n - Changed from git-diff-blob(1) to git-diff-pairs(1) based on a\n previously submitted series.\n\n - Instead of each line containing a pair of blob revisions, the raw\n diff format is used as input which already has diff status and\n object context embedded.\n\n-Justin\n\n[1]: <20161201204042.6yslbyrg7l6ghhww@sigill.intra.peff.net>\n\nJustin Tobler (4):\n diff: return diff_filepair from diff queue helpers\n diff: add option to skip resolving diff statuses\n builtin: introduce diff-pairs command\n builtin/diff-pairs: allow explicit diff queue flush\n\n .gitignore | 1 +\n Documentation/git-diff-pairs.adoc | 60 +++++++++\n Documentation/meson.build | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/diff-pairs.c | 209 ++++++++++++++++++++++++++++++\n command-list.txt | 1 +\n diff.c | 72 +++++++---\n diff.h | 33 +++++\n git.c | 1 +\n meson.build | 1 +\n t/meson.build | 1 +\n t/t4070-diff-pairs.sh | 90 +++++++++++++\n 13 files changed, 451 insertions(+), 21 deletions(-)\n create mode 100644 Documentation/git-diff-pairs.adoc\n create mode 100644 builtin/diff-pairs.c\n create mode 100755 t/t4070-diff-pairs.sh\n\nRange-diff against v3:\n1: d19b164914 ! 1: b2e5486442 diff: return diff_filepair from diff queue helpers\n @@ Commit message\n \n Split out the queuing operations into `diff_queue_addremove()` and\n `diff_queue_change()` which also return a handle to the queued\n - `diff_filepair`.\n + `diff_filepair`. Both `diff_addremove()` and `diff_change()` are\n + reimplemented as thin wrappers around the new functions.\n \n Signed-off-by: Justin Tobler \n \n-: ---------- > 2: 31d80d99ae diff: add option to skip resolving diff statuses\n2: 991aaea3a9 ! 3: 3722c02112 builtin: introduce diff-pairs command\n @@ Commit message\n machinery to produce various forms of output such as patch or raw.\n \n The raw format was originally designed as an interchange format and\n - represents the contents of the diff_queue_diff list making it possible\n + represents the contents of the diff_queued_diff list making it possible\n to break the diff pipeline into separate stages. For example,\n git-diff-tree(1) can be used as a frontend to compute file pairs to\n queue and feed its raw output to git-diff-pairs(1) to compute patches.\n - With this, batches of diffs can be progessively generated without having\n - to recompute rename detection or retrieve object context. Something like\n + With this, batches of diffs can be progressively generated without\n + having to recompute renames or retrieve object context. Something like\n the following:\n \n git diff-tree -r -z -M $old $new |\n @@ builtin.h: int cmd_diagnose(int argc, const char **argv, const char *prefix, str\n ## builtin/diff-pairs.c (new) ##\n @@\n +#include \"builtin.h\"\n -+#include \"commit.h\"\n +#include \"config.h\"\n +#include \"diff.h\"\n +#include \"diffcore.h\"\n +#include \"gettext.h\"\n ++#include \"hash.h\"\n +#include \"hex.h\"\n +#include \"object.h\"\n +#include \"parse-options.h\"\n +#include \"revision.h\"\n +#include \"strbuf.h\"\n +\n -+static unsigned parse_mode_or_die(const char *mode, const char **endp)\n ++static unsigned parse_mode_or_die(const char *mode, const char **end)\n +{\n +\tuint16_t ret;\n +\n -+\t*endp = parse_mode(mode, &ret);\n -+\tif (!*endp)\n ++\t*end = parse_mode(mode, &ret);\n ++\tif (!*end)\n +\t\tdie(_(\"unable to parse mode: %s\"), mode);\n +\treturn ret;\n +}\n +\n -+static void parse_oid_or_die(const char *p, struct object_id *oid,\n -+\t\t\t const char **endp, const struct git_hash_algo *algop)\n ++static void parse_oid_or_die(const char *hex, struct object_id *oid,\n ++\t\t\t const char **end, const struct git_hash_algo *algop)\n +{\n -+\tif (parse_oid_hex_algop(p, oid, endp, algop) || *(*endp)++ != ' ')\n -+\t\tdie(_(\"unable to parse object id: %s\"), p);\n -+}\n -+\n -+static void flush_diff_queue(struct diff_options *options)\n -+{\n -+\t/*\n -+\t * If rename detection is not requested, use rename information from the\n -+\t * raw diff formatted input. Setting found_follow ensures diffcore_std()\n -+\t * does not mess with rename information already present in queued\n -+\t * filepairs.\n -+\t */\n -+\tif (!options->detect_rename)\n -+\t\toptions->found_follow = 1;\n -+\tdiffcore_std(options);\n -+\tdiff_flush(options);\n ++\tif (parse_oid_hex_algop(hex, oid, end, algop) || *(*end)++ != ' ')\n ++\t\tdie(_(\"unable to parse object id: %s\"), hex);\n +}\n +\n +int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n @@ builtin/diff-pairs.c (new)\n +\tstruct strbuf path_dst = STRBUF_INIT;\n +\tstruct strbuf path = STRBUF_INIT;\n +\tstruct strbuf meta = STRBUF_INIT;\n ++\tstruct option *parseopts;\n +\tstruct rev_info revs;\n ++\tint line_term = '\\0';\n +\tint ret;\n +\n -+\tconst char * const usage[] = {\n ++\tconst char * const usagestr[] = {\n +\t\tN_(\"git diff-pairs -z []\"),\n +\t\tNULL\n +\t};\n +\tstruct option options[] = {\n +\t\tOPT_END()\n +\t};\n -+\tstruct option *parseopts = add_diff_options(options, &revs.diffopt);\n -+\n -+\tshow_usage_with_options_if_asked(argc, argv, usage, parseopts);\n +\n +\trepo_init_revisions(repo, &revs, prefix);\n ++\n ++\t/*\n ++\t * Diff options are usually parsed implicitly as part of\n ++\t * setup_revisions(). Explicitly handle parsing to ensure options are\n ++\t * printed in the usage message.\n ++\t */\n ++\tparseopts = add_diff_options(options, &revs.diffopt);\n ++\tshow_usage_with_options_if_asked(argc, argv, usagestr, parseopts);\n ++\n +\trepo_config(repo, git_diff_basic_config, NULL);\n +\trevs.disable_stdin = 1;\n +\trevs.abbrev = 0;\n +\trevs.diff = 1;\n +\n ++\targc = parse_options(argc, argv, prefix, parseopts, usagestr,\n ++\t\t\t PARSE_OPT_KEEP_UNKNOWN_OPT |\n ++\t\t\t PARSE_OPT_KEEP_DASHDASH |\n ++\t\t\t PARSE_OPT_KEEP_ARGV0);\n ++\n +\tif (setup_revisions(argc, argv, &revs, NULL) > 1)\n -+\t\tusage_with_options(usage, parseopts);\n ++\t\tusagef(_(\"unrecognized argument: %s\"), argv[0]);\n +\n +\t/*\n +\t * With the -z option, both command input and raw output are\n -+\t * NUL-delimited (this mode does not effect patch output). At present\n ++\t * NUL-delimited (this mode does not affect patch output). At present\n +\t * only NUL-delimited raw diff formatted input is supported.\n +\t */\n -+\tif (revs.diffopt.line_termination) {\n -+\t\terror(_(\"working without -z is not supported\"));\n -+\t\tusage_with_options(usage, parseopts);\n -+\t}\n ++\tif (revs.diffopt.line_termination)\n ++\t\tusage(_(\"working without -z is not supported\"));\n +\n -+\tif (revs.prune_data.nr) {\n -+\t\terror(_(\"pathspec arguments not supported\"));\n -+\t\tusage_with_options(usage, parseopts);\n -+\t}\n ++\tif (revs.prune_data.nr)\n ++\t\tusage(_(\"pathspec arguments not supported\"));\n +\n +\tif (revs.pending.nr || revs.max_count != -1 ||\n +\t revs.min_age != (timestamp_t)-1 ||\n -+\t revs.max_age != (timestamp_t)-1) {\n -+\t\terror(_(\"revision arguments not allowed\"));\n -+\t\tusage_with_options(usage, parseopts);\n -+\t}\n ++\t revs.max_age != (timestamp_t)-1)\n ++\t\tusage(_(\"revision arguments not allowed\"));\n +\n +\tif (!revs.diffopt.output_format)\n +\t\trevs.diffopt.output_format = DIFF_FORMAT_PATCH;\n +\n ++\t/*\n ++\t * If rename detection is not requested, use rename information from the\n ++\t * raw diff formatted input. Setting skip_resolving_statuses ensures\n ++\t * diffcore_std() does not mess with rename information already present\n ++\t * in queued filepairs.\n ++\t */\n ++\tif (!revs.diffopt.detect_rename)\n ++\t\trevs.diffopt.skip_resolving_statuses = 1;\n ++\n +\twhile (1) {\n +\t\tstruct object_id oid_a, oid_b;\n +\t\tstruct diff_filepair *pair;\n @@ builtin/diff-pairs.c (new)\n +\t\tconst char *p;\n +\t\tchar status;\n +\n -+\t\tif (strbuf_getline_nul(&meta, stdin) == EOF)\n ++\t\tif (strbuf_getwholeline(&meta, stdin, line_term) == EOF)\n +\t\t\tbreak;\n +\n +\t\tp = meta.buf;\n @@ builtin/diff-pairs.c (new)\n +\n +\t\tstatus = *p++;\n +\n -+\t\tif (strbuf_getline_nul(&path, stdin) == EOF)\n ++\t\tif (strbuf_getwholeline(&path, stdin, line_term) == EOF)\n +\t\t\tdie(_(\"got EOF while reading path\"));\n +\n +\t\tswitch (status) {\n @@ builtin/diff-pairs.c (new)\n +\t\t\tbreak;\n +\n +\t\tcase DIFF_STATUS_RENAMED:\n -+\t\tcase DIFF_STATUS_COPIED:\n -+\t\t\t{\n ++\t\tcase DIFF_STATUS_COPIED: {\n +\t\t\t\tstruct diff_filespec *a, *b;\n +\t\t\t\tunsigned int score;\n +\n -+\t\t\t\tif (strbuf_getline_nul(&path_dst, stdin) == EOF)\n ++\t\t\t\tif (strbuf_getwholeline(&path_dst, stdin, line_term) == EOF)\n +\t\t\t\t\tdie(_(\"got EOF while reading destination path\"));\n +\n +\t\t\t\ta = alloc_filespec(path.buf);\n @@ builtin/diff-pairs.c (new)\n +\t\t}\n +\t}\n +\n -+\tflush_diff_queue(&revs.diffopt);\n ++\tdiffcore_std(&revs.diffopt);\n ++\tdiff_flush(&revs.diffopt);\n +\tret = diff_result_code(&revs);\n +\n +\tstrbuf_release(&path_dst);\n @@ t/t4070-diff-pairs.sh (new)\n +. ./test-lib.sh\n +\n +# This creates a diff with added, modified, deleted, renamed, copied, and\n -+# typechange entries. That includes one in a subdirectory for non-recursive\n -+# tests, and both exact and inexact similarity scores.\n ++# typechange entries. This includes a submodule to test submodule diff support.\n +test_expect_success 'setup' '\n ++\ttest_config_global protocol.file.allow always &&\n ++\ttest_create_repo sub &&\n ++\ttest_commit -C sub initial &&\n ++\n ++\ttest_create_repo main &&\n ++\tcd main &&\n +\techo to-be-gone >deleted &&\n +\techo original >modified &&\n +\techo now-a-file >symlink &&\n @@ t/t4070-diff-pairs.sh (new)\n +\tgit commit -m base &&\n +\tgit tag base &&\n +\n ++\tgit submodule add ../sub &&\n +\techo now-here >added &&\n +\techo new >modified &&\n +\trm deleted &&\n @@ t/t4070-diff-pairs.sh (new)\n +\n +test_expect_success 'diff-pairs recreates --raw' '\n +\tgit diff-tree -r -M -C -C -z base new >expect &&\n -+\tgit diff-tree -r -M -C -C -z base new |\n -+\tgit diff-pairs --raw -z >actual &&\n ++\tgit diff-pairs --raw -z >actual out 2>err &&\n +\n ++\techo \"usage: working without -z is not supported\" >expect &&\n +\ttest_must_be_empty out &&\n -+\tgrep \"error: working without -z is not supported\" err\n ++\ttest_cmp expect err\n +'\n +\n +test_expect_success 'diff-pairs does not support tree objects as input' '\n @@ t/t4070-diff-pairs.sh (new)\n +\tgit diff-tree -r -z base new |\n +\ttest_must_fail git diff-pairs -z -- new >out 2>err &&\n +\n ++\techo \"usage: pathspec arguments not supported\" >expect &&\n +\ttest_must_be_empty out &&\n -+\tgrep \"error: pathspec arguments not supported\" err\n ++\ttest_cmp expect err\n +'\n +\n +test_done\n3: 26c1c80b66 ! 4: a4809cbd80 builtin/diff-pairs: allow explicit diff queue flush\n @@ Documentation/git-diff-pairs.adoc: in the NUL-terminated raw output format as ge\n \n ## builtin/diff-pairs.c ##\n @@ builtin/diff-pairs.c: int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n + \tshow_usage_with_options_if_asked(argc, argv, usagestr, parseopts);\n \n - \trepo_init_revisions(repo, &revs, prefix);\n \trepo_config(repo, git_diff_basic_config, NULL);\n +\trevs.diffopt.no_free = 1;\n \trevs.disable_stdin = 1;\n @@ builtin/diff-pairs.c: int cmd_diff_pairs(int argc, const char **argv, const char\n \n \t\tp = meta.buf;\n +\t\tif (!*p) {\n -+\t\t\tflush_diff_queue(&revs.diffopt);\n ++\t\t\tdiffcore_std(&revs.diffopt);\n ++\t\t\tdiff_flush(&revs.diffopt);\n +\t\t\t/*\n +\t\t\t * When the diff queue is explicitly flushed, append a\n +\t\t\t * NUL byte to separate batches of diffs.\n @@ builtin/diff-pairs.c: int cmd_diff_pairs(int argc, const char **argv, const char\n \t}\n \n +\trevs.diffopt.no_free = 0;\n - \tflush_diff_queue(&revs.diffopt);\n + \tdiffcore_std(&revs.diffopt);\n + \tdiff_flush(&revs.diffopt);\n \tret = diff_result_code(&revs);\n - \n \n ## t/t4070-diff-pairs.sh ##\n @@ t/t4070-diff-pairs.sh: test_expect_success 'diff-pairs does not support pathspec arguments' '\n - \tgrep \"error: pathspec arguments not supported\" err\n + \ttest_cmp expect err\n '\n \n +test_expect_success 'diff-pairs explicit queue flush' '\n-- \n2.48.1\n\n"},{"id":"513208","messageId":"20250228002604.3859939-2-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250228002604.3859939-1-jltobler@gmail.com","subject":"[PATCH v4 1/4] diff: return diff_filepair from diff queue helpers","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-28T00:26:01Z","receivedAt":"2025-02-28T00:29:27Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"The `diff_addremove()` and `diff_change()` functions set up and queue\ndiffs, but do not return the `diff_filepair` added to the queue. In a\nsubsequent commit, modifications to `diff_filepair` need to occur in\ncertain cases after being queued.\n\nSince the existing `diff_addremove()` and `diff_change()` are also used\nfor callbacks in `diff_options` as types `add_remove_fn_t` and\n`change_fn_t`, modifying the existing function signatures requires\nfurther changes. The diff options for pruning use `file_add_remove()`\nand `file_change()` where file pairs do not even get queued. Thus,\nseparate functions are implemented instead.\n\nSplit out the queuing operations into `diff_queue_addremove()` and\n`diff_queue_change()` which also return a handle to the queued\n`diff_filepair`. Both `diff_addremove()` and `diff_change()` are\nreimplemented as thin wrappers around the new functions.\n\nSigned-off-by: Justin Tobler \n---\n diff.c | 70 +++++++++++++++++++++++++++++++++++++++++-----------------\n diff.h | 25 +++++++++++++++++++++\n 2 files changed, 75 insertions(+), 20 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 019fb893a7..b5a779f997 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -7157,16 +7157,19 @@ void compute_diffstat(struct diff_options *options,\n \toptions->found_changes = !!diffstat->nr;\n }\n \n-void diff_addremove(struct diff_options *options,\n-\t\t int addremove, unsigned mode,\n-\t\t const struct object_id *oid,\n-\t\t int oid_valid,\n-\t\t const char *concatpath, unsigned dirty_submodule)\n+struct diff_filepair *diff_queue_addremove(struct diff_queue_struct *queue,\n+\t\t\t\t\t struct diff_options *options,\n+\t\t\t\t\t int addremove, unsigned mode,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t int oid_valid,\n+\t\t\t\t\t const char *concatpath,\n+\t\t\t\t\t unsigned dirty_submodule)\n {\n \tstruct diff_filespec *one, *two;\n+\tstruct diff_filepair *pair;\n \n \tif (S_ISGITLINK(mode) && is_submodule_ignored(concatpath, options))\n-\t\treturn;\n+\t\treturn NULL;\n \n \t/* This may look odd, but it is a preparation for\n \t * feeding \"there are unchanged files which should\n@@ -7186,7 +7189,7 @@ void diff_addremove(struct diff_options *options,\n \n \tif (options->prefix &&\n \t strncmp(concatpath, options->prefix, options->prefix_length))\n-\t\treturn;\n+\t\treturn NULL;\n \n \tone = alloc_filespec(concatpath);\n \ttwo = alloc_filespec(concatpath);\n@@ -7198,25 +7201,29 @@ void diff_addremove(struct diff_options *options,\n \t\ttwo->dirty_submodule = dirty_submodule;\n \t}\n \n-\tdiff_queue(&diff_queued_diff, one, two);\n+\tpair = diff_queue(queue, one, two);\n \tif (!options->flags.diff_from_contents)\n \t\toptions->flags.has_changes = 1;\n+\n+\treturn pair;\n }\n \n-void diff_change(struct diff_options *options,\n-\t\t unsigned old_mode, unsigned new_mode,\n-\t\t const struct object_id *old_oid,\n-\t\t const struct object_id *new_oid,\n-\t\t int old_oid_valid, int new_oid_valid,\n-\t\t const char *concatpath,\n-\t\t unsigned old_dirty_submodule, unsigned new_dirty_submodule)\n+struct diff_filepair *diff_queue_change(struct diff_queue_struct *queue,\n+\t\t\t\t\tstruct diff_options *options,\n+\t\t\t\t\tunsigned old_mode, unsigned new_mode,\n+\t\t\t\t\tconst struct object_id *old_oid,\n+\t\t\t\t\tconst struct object_id *new_oid,\n+\t\t\t\t\tint old_oid_valid, int new_oid_valid,\n+\t\t\t\t\tconst char *concatpath,\n+\t\t\t\t\tunsigned old_dirty_submodule,\n+\t\t\t\t\tunsigned new_dirty_submodule)\n {\n \tstruct diff_filespec *one, *two;\n \tstruct diff_filepair *p;\n \n \tif (S_ISGITLINK(old_mode) && S_ISGITLINK(new_mode) &&\n \t is_submodule_ignored(concatpath, options))\n-\t\treturn;\n+\t\treturn NULL;\n \n \tif (options->flags.reverse_diff) {\n \t\tSWAP(old_mode, new_mode);\n@@ -7227,7 +7234,7 @@ void diff_change(struct diff_options *options,\n \n \tif (options->prefix &&\n \t strncmp(concatpath, options->prefix, options->prefix_length))\n-\t\treturn;\n+\t\treturn NULL;\n \n \tone = alloc_filespec(concatpath);\n \ttwo = alloc_filespec(concatpath);\n@@ -7235,19 +7242,42 @@ void diff_change(struct diff_options *options,\n \tfill_filespec(two, new_oid, new_oid_valid, new_mode);\n \tone->dirty_submodule = old_dirty_submodule;\n \ttwo->dirty_submodule = new_dirty_submodule;\n-\tp = diff_queue(&diff_queued_diff, one, two);\n+\tp = diff_queue(queue, one, two);\n \n \tif (options->flags.diff_from_contents)\n-\t\treturn;\n+\t\treturn p;\n \n \tif (options->flags.quick && options->skip_stat_unmatch &&\n \t !diff_filespec_check_stat_unmatch(options->repo, p)) {\n \t\tdiff_free_filespec_data(p->one);\n \t\tdiff_free_filespec_data(p->two);\n-\t\treturn;\n+\t\treturn p;\n \t}\n \n \toptions->flags.has_changes = 1;\n+\n+\treturn p;\n+}\n+\n+void diff_addremove(struct diff_options *options, int addremove, unsigned mode,\n+\t\t const struct object_id *oid, int oid_valid,\n+\t\t const char *concatpath, unsigned dirty_submodule)\n+{\n+\tdiff_queue_addremove(&diff_queued_diff, options, addremove, mode, oid,\n+\t\t\t oid_valid, concatpath, dirty_submodule);\n+}\n+\n+void diff_change(struct diff_options *options,\n+\t\t unsigned old_mode, unsigned new_mode,\n+\t\t const struct object_id *old_oid,\n+\t\t const struct object_id *new_oid,\n+\t\t int old_oid_valid, int new_oid_valid,\n+\t\t const char *concatpath,\n+\t\t unsigned old_dirty_submodule, unsigned new_dirty_submodule)\n+{\n+\tdiff_queue_change(&diff_queued_diff, options, old_mode, new_mode,\n+\t\t\t old_oid, new_oid, old_oid_valid, new_oid_valid,\n+\t\t\t concatpath, old_dirty_submodule, new_dirty_submodule);\n }\n \n struct diff_filepair *diff_unmerge(struct diff_options *options, const char *path)\ndiff --git a/diff.h b/diff.h\nindex 0a566f5531..63afa17e84 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -508,6 +508,31 @@ void diff_set_default_prefix(struct diff_options *options);\n \n int diff_can_quit_early(struct diff_options *);\n \n+/*\n+ * Stages changes in the provided diff queue for file additions and deletions.\n+ * If a file pair gets queued, it is returned.\n+ */\n+struct diff_filepair *diff_queue_addremove(struct diff_queue_struct *queue,\n+\t\t\t\t\t struct diff_options *,\n+\t\t\t\t\t int addremove, unsigned mode,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t int oid_valid, const char *fullpath,\n+\t\t\t\t\t unsigned dirty_submodule);\n+\n+/*\n+ * Stages changes in the provided diff queue for file modifications.\n+ * If a file pair gets queued, it is returned.\n+ */\n+struct diff_filepair *diff_queue_change(struct diff_queue_struct *queue,\n+\t\t\t\t\tstruct diff_options *,\n+\t\t\t\t\tunsigned mode1, unsigned mode2,\n+\t\t\t\t\tconst struct object_id *old_oid,\n+\t\t\t\t\tconst struct object_id *new_oid,\n+\t\t\t\t\tint old_oid_valid, int new_oid_valid,\n+\t\t\t\t\tconst char *fullpath,\n+\t\t\t\t\tunsigned dirty_submodule1,\n+\t\t\t\t\tunsigned dirty_submodule2);\n+\n void diff_addremove(struct diff_options *,\n \t\t int addremove,\n \t\t unsigned mode,\n-- \n2.48.1\n\n"},{"id":"513209","messageId":"20250228002604.3859939-3-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250228002604.3859939-1-jltobler@gmail.com","subject":"[PATCH v4 2/4] diff: add option to skip resolving diff statuses","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-28T00:26:02Z","receivedAt":"2025-02-28T00:29:28Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"By default, `diffcore_std()` resolves the statuses for queued diff file\npairs by calling `diff_resolve_rename_copy()`. If status information is\nalready manually set, invoking `diffcore_std()` may change the status\nvalue.\n\nIntroduce the `skip_resolving_statuses` diff option that prevents\n`diffcore_std()` from resolving file pair statuses when enabled.\n\nSigned-off-by: Justin Tobler \n---\n diff.c | 2 +-\n diff.h | 8 ++++++++\n 2 files changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/diff.c b/diff.c\nindex b5a779f997..37cc88c75b 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -7081,7 +7081,7 @@ void diffcore_std(struct diff_options *options)\n \t\tdiffcore_order(options->orderfile);\n \tif (options->rotate_to)\n \t\tdiffcore_rotate(options);\n-\tif (!options->found_follow)\n+\tif (!options->found_follow && !options->skip_resolving_statuses)\n \t\t/* See try_to_follow_renames() in tree-diff.c */\n \t\tdiff_resolve_rename_copy();\n \tdiffcore_apply_filter(options);\ndiff --git a/diff.h b/diff.h\nindex 63afa17e84..fc791ee2cc 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -353,6 +353,14 @@ struct diff_options {\n \t/* to support internal diff recursion by --follow hack*/\n \tint found_follow;\n \n+\t/*\n+\t * By default, diffcore_std() resolves the statuses for queued diff file\n+\t * pairs by calling diff_resolve_rename_copy(). If status information\n+\t * has already been manually set, this option prevents diffcore_std()\n+\t * from resetting statuses.\n+\t */\n+\tint skip_resolving_statuses;\n+\n \t/* Callback which allows tweaking the options in diff_setup_done(). */\n \tvoid (*set_default)(struct diff_options *);\n \n-- \n2.48.1\n\n"},{"id":"513210","messageId":"20250228002604.3859939-4-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250228002604.3859939-1-jltobler@gmail.com","subject":"[PATCH v4 3/4] builtin: introduce diff-pairs command","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-28T00:26:03Z","receivedAt":"2025-02-28T00:29:30Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Through git-diff(1), a single diff can be generated from a pair of blob\nrevisions directly. Unfortunately, there is not a mechanism to compute\nbatches of specific file pair diffs in a single process. Such a feature\nis particularly useful on the server-side where diffing between a large\nset of changes is not feasible all at once due to timeout concerns.\n\nTo facilitate this, introduce git-diff-pairs(1) which acts as a backend\npassing its NUL-terminated raw diff format input from stdin through diff\nmachinery to produce various forms of output such as patch or raw.\n\nThe raw format was originally designed as an interchange format and\nrepresents the contents of the diff_queued_diff list making it possible\nto break the diff pipeline into separate stages. For example,\ngit-diff-tree(1) can be used as a frontend to compute file pairs to\nqueue and feed its raw output to git-diff-pairs(1) to compute patches.\nWith this, batches of diffs can be progressively generated without\nhaving to recompute renames or retrieve object context. Something like\nthe following:\n\n\tgit diff-tree -r -z -M $old $new |\n\tgit diff-pairs -p -z\n\nshould generate the same output as `git diff-tree -p -M`. Furthermore,\neach line of raw diff formatted input can also be individually fed to a\nseparate git-diff-pairs(1) process and still produce the same output.\n\nBased-on-patch-by: Jeff King \nSigned-off-by: Justin Tobler \n---\n .gitignore | 1 +\n Documentation/git-diff-pairs.adoc | 56 +++++++++\n Documentation/meson.build | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/diff-pairs.c | 195 ++++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n meson.build | 1 +\n t/meson.build | 1 +\n t/t4070-diff-pairs.sh | 81 +++++++++++++\n 11 files changed, 340 insertions(+)\n create mode 100644 Documentation/git-diff-pairs.adoc\n create mode 100644 builtin/diff-pairs.c\n create mode 100755 t/t4070-diff-pairs.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 08a66ca508..04c444404e 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -55,6 +55,7 @@\n /git-diff\n /git-diff-files\n /git-diff-index\n+/git-diff-pairs\n /git-diff-tree\n /git-difftool\n /git-difftool--helper\ndiff --git a/Documentation/git-diff-pairs.adoc b/Documentation/git-diff-pairs.adoc\nnew file mode 100644\nindex 0000000000..e31f2e2fbb\n--- /dev/null\n+++ b/Documentation/git-diff-pairs.adoc\n@@ -0,0 +1,56 @@\n+git-diff-pairs(1)\n+=================\n+\n+NAME\n+----\n+git-diff-pairs - Compare the content and mode of provided blob pairs\n+\n+SYNOPSIS\n+--------\n+[synopsis]\n+git diff-pairs -z []\n+\n+DESCRIPTION\n+-----------\n+Show changes for file pairs provided on stdin. Input for this command must be\n+in the NUL-terminated raw output format as generated by commands such as `git\n+diff-tree -z -r --raw`. By default, the outputted diffs are computed and shown\n+in the patch format when stdin closes.\n+\n+Usage of this command enables the traditional diff pipeline to be broken up\n+into separate stages where `diff-pairs` acts as the output phase. Other\n+commands, such as `diff-tree`, may serve as a frontend to compute the raw\n+diff format used as input.\n+\n+Instead of computing diffs via `git diff-tree -p -M` in one step, `diff-tree`\n+can compute the file pairs and rename information without the blob diffs. This\n+output can be fed to `diff-pairs` to generate the underlying blob diffs as done\n+in the following example:\n+\n+-----------------------------\n+git diff-tree -z -r -M $a $b |\n+git diff-pairs -z\n+-----------------------------\n+\n+Computing the tree diff upfront with rename information allows patch output\n+from `diff-pairs` to be progressively computed over the course of potentially\n+multiple invocations.\n+\n+Pathspecs are not currently supported by `diff-pairs`. Pathspec limiting should\n+be performed by the upstream command generating the raw diffs used as input.\n+\n+Tree objects are not currently supported as input and are rejected.\n+\n+Abbreviated object IDs in the `diff-pairs` input are not supported. Outputted\n+object IDs can be abbreviated using the `--abbrev` option.\n+\n+OPTIONS\n+-------\n+\n+include::diff-options.adoc[]\n+\n+include::diff-generate-patch.adoc[]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Documentation/meson.build b/Documentation/meson.build\nindex 1129ce4c85..ce990e9fe5 100644\n--- a/Documentation/meson.build\n+++ b/Documentation/meson.build\n@@ -42,6 +42,7 @@ manpages = {\n 'git-diagnose.adoc' : 1,\n 'git-diff-files.adoc' : 1,\n 'git-diff-index.adoc' : 1,\n+ 'git-diff-pairs.adoc' : 1,\n 'git-difftool.adoc' : 1,\n 'git-diff-tree.adoc' : 1,\n 'git-diff.adoc' : 1,\ndiff --git a/Makefile b/Makefile\nindex bcf5ed3f85..56df7aed3f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1242,6 +1242,7 @@ BUILTIN_OBJS += builtin/describe.o\n BUILTIN_OBJS += builtin/diagnose.o\n BUILTIN_OBJS += builtin/diff-files.o\n BUILTIN_OBJS += builtin/diff-index.o\n+BUILTIN_OBJS += builtin/diff-pairs.o\n BUILTIN_OBJS += builtin/diff-tree.o\n BUILTIN_OBJS += builtin/diff.o\n BUILTIN_OBJS += builtin/difftool.o\ndiff --git a/builtin.h b/builtin.h\nindex 89928ccf92..e6aad3a6a1 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -153,6 +153,7 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix, struct reposit\n int cmd_diff_files(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_diff_index(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_diff(int argc, const char **argv, const char *prefix, struct repository *repo);\n+int cmd_diff_pairs(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_diff_tree(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_difftool(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_env__helper(int argc, const char **argv, const char *prefix, struct repository *repo);\ndiff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\nnew file mode 100644\nindex 0000000000..5a993b7c9d\n--- /dev/null\n+++ b/builtin/diff-pairs.c\n@@ -0,0 +1,195 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"diff.h\"\n+#include \"diffcore.h\"\n+#include \"gettext.h\"\n+#include \"hash.h\"\n+#include \"hex.h\"\n+#include \"object.h\"\n+#include \"parse-options.h\"\n+#include \"revision.h\"\n+#include \"strbuf.h\"\n+\n+static unsigned parse_mode_or_die(const char *mode, const char **end)\n+{\n+\tuint16_t ret;\n+\n+\t*end = parse_mode(mode, &ret);\n+\tif (!*end)\n+\t\tdie(_(\"unable to parse mode: %s\"), mode);\n+\treturn ret;\n+}\n+\n+static void parse_oid_or_die(const char *hex, struct object_id *oid,\n+\t\t\t const char **end, const struct git_hash_algo *algop)\n+{\n+\tif (parse_oid_hex_algop(hex, oid, end, algop) || *(*end)++ != ' ')\n+\t\tdie(_(\"unable to parse object id: %s\"), hex);\n+}\n+\n+int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n+\t\t struct repository *repo)\n+{\n+\tstruct strbuf path_dst = STRBUF_INIT;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct strbuf meta = STRBUF_INIT;\n+\tstruct option *parseopts;\n+\tstruct rev_info revs;\n+\tint line_term = '\\0';\n+\tint ret;\n+\n+\tconst char * const usagestr[] = {\n+\t\tN_(\"git diff-pairs -z []\"),\n+\t\tNULL\n+\t};\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\trepo_init_revisions(repo, &revs, prefix);\n+\n+\t/*\n+\t * Diff options are usually parsed implicitly as part of\n+\t * setup_revisions(). Explicitly handle parsing to ensure options are\n+\t * printed in the usage message.\n+\t */\n+\tparseopts = add_diff_options(options, &revs.diffopt);\n+\tshow_usage_with_options_if_asked(argc, argv, usagestr, parseopts);\n+\n+\trepo_config(repo, git_diff_basic_config, NULL);\n+\trevs.disable_stdin = 1;\n+\trevs.abbrev = 0;\n+\trevs.diff = 1;\n+\n+\targc = parse_options(argc, argv, prefix, parseopts, usagestr,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN_OPT |\n+\t\t\t PARSE_OPT_KEEP_DASHDASH |\n+\t\t\t PARSE_OPT_KEEP_ARGV0);\n+\n+\tif (setup_revisions(argc, argv, &revs, NULL) > 1)\n+\t\tusagef(_(\"unrecognized argument: %s\"), argv[0]);\n+\n+\t/*\n+\t * With the -z option, both command input and raw output are\n+\t * NUL-delimited (this mode does not affect patch output). At present\n+\t * only NUL-delimited raw diff formatted input is supported.\n+\t */\n+\tif (revs.diffopt.line_termination)\n+\t\tusage(_(\"working without -z is not supported\"));\n+\n+\tif (revs.prune_data.nr)\n+\t\tusage(_(\"pathspec arguments not supported\"));\n+\n+\tif (revs.pending.nr || revs.max_count != -1 ||\n+\t revs.min_age != (timestamp_t)-1 ||\n+\t revs.max_age != (timestamp_t)-1)\n+\t\tusage(_(\"revision arguments not allowed\"));\n+\n+\tif (!revs.diffopt.output_format)\n+\t\trevs.diffopt.output_format = DIFF_FORMAT_PATCH;\n+\n+\t/*\n+\t * If rename detection is not requested, use rename information from the\n+\t * raw diff formatted input. Setting skip_resolving_statuses ensures\n+\t * diffcore_std() does not mess with rename information already present\n+\t * in queued filepairs.\n+\t */\n+\tif (!revs.diffopt.detect_rename)\n+\t\trevs.diffopt.skip_resolving_statuses = 1;\n+\n+\twhile (1) {\n+\t\tstruct object_id oid_a, oid_b;\n+\t\tstruct diff_filepair *pair;\n+\t\tunsigned mode_a, mode_b;\n+\t\tconst char *p;\n+\t\tchar status;\n+\n+\t\tif (strbuf_getwholeline(&meta, stdin, line_term) == EOF)\n+\t\t\tbreak;\n+\n+\t\tp = meta.buf;\n+\t\tif (*p != ':')\n+\t\t\tdie(_(\"invalid raw diff input\"));\n+\t\tp++;\n+\n+\t\tmode_a = parse_mode_or_die(p, &p);\n+\t\tmode_b = parse_mode_or_die(p, &p);\n+\n+\t\tif (S_ISDIR(mode_a) || S_ISDIR(mode_b))\n+\t\t\tdie(_(\"tree objects not supported\"));\n+\n+\t\tparse_oid_or_die(p, &oid_a, &p, repo->hash_algo);\n+\t\tparse_oid_or_die(p, &oid_b, &p, repo->hash_algo);\n+\n+\t\tstatus = *p++;\n+\n+\t\tif (strbuf_getwholeline(&path, stdin, line_term) == EOF)\n+\t\t\tdie(_(\"got EOF while reading path\"));\n+\n+\t\tswitch (status) {\n+\t\tcase DIFF_STATUS_ADDED:\n+\t\t\tpair = diff_queue_addremove(&diff_queued_diff,\n+\t\t\t\t\t\t &revs.diffopt, '+', mode_b,\n+\t\t\t\t\t\t &oid_b, 1, path.buf, 0);\n+\t\t\tif (pair)\n+\t\t\t\tpair->status = status;\n+\t\t\tbreak;\n+\n+\t\tcase DIFF_STATUS_DELETED:\n+\t\t\tpair = diff_queue_addremove(&diff_queued_diff,\n+\t\t\t\t\t\t &revs.diffopt, '-', mode_a,\n+\t\t\t\t\t\t &oid_a, 1, path.buf, 0);\n+\t\t\tif (pair)\n+\t\t\t\tpair->status = status;\n+\t\t\tbreak;\n+\n+\t\tcase DIFF_STATUS_TYPE_CHANGED:\n+\t\tcase DIFF_STATUS_MODIFIED:\n+\t\t\tpair = diff_queue_change(&diff_queued_diff, &revs.diffopt,\n+\t\t\t\t\t\t mode_a, mode_b, &oid_a, &oid_b,\n+\t\t\t\t\t\t 1, 1, path.buf, 0, 0);\n+\t\t\tif (pair)\n+\t\t\t\tpair->status = status;\n+\t\t\tbreak;\n+\n+\t\tcase DIFF_STATUS_RENAMED:\n+\t\tcase DIFF_STATUS_COPIED: {\n+\t\t\t\tstruct diff_filespec *a, *b;\n+\t\t\t\tunsigned int score;\n+\n+\t\t\t\tif (strbuf_getwholeline(&path_dst, stdin, line_term) == EOF)\n+\t\t\t\t\tdie(_(\"got EOF while reading destination path\"));\n+\n+\t\t\t\ta = alloc_filespec(path.buf);\n+\t\t\t\tb = alloc_filespec(path_dst.buf);\n+\t\t\t\tfill_filespec(a, &oid_a, 1, mode_a);\n+\t\t\t\tfill_filespec(b, &oid_b, 1, mode_b);\n+\n+\t\t\t\tpair = diff_queue(&diff_queued_diff, a, b);\n+\n+\t\t\t\tif (strtoul_ui(p, 10, &score))\n+\t\t\t\t\tdie(_(\"unable to parse rename/copy score: %s\"), p);\n+\n+\t\t\t\tpair->score = score * MAX_SCORE / 100;\n+\t\t\t\tpair->status = status;\n+\t\t\t\tpair->renamed_pair = 1;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tdefault:\n+\t\t\tdie(_(\"unknown diff status: %c\"), status);\n+\t\t}\n+\t}\n+\n+\tdiffcore_std(&revs.diffopt);\n+\tdiff_flush(&revs.diffopt);\n+\tret = diff_result_code(&revs);\n+\n+\tstrbuf_release(&path_dst);\n+\tstrbuf_release(&path);\n+\tstrbuf_release(&meta);\n+\trelease_revisions(&revs);\n+\tFREE_AND_NULL(parseopts);\n+\n+\treturn ret;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex c537114b46..b7ade3ab9f 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -96,6 +96,7 @@ git-diagnose ancillaryinterrogators\n git-diff mainporcelain info\n git-diff-files plumbinginterrogators\n git-diff-index plumbinginterrogators\n+git-diff-pairs plumbinginterrogators\n git-diff-tree plumbinginterrogators\n git-difftool ancillaryinterrogators complete\n git-fast-export ancillarymanipulators\ndiff --git a/git.c b/git.c\nindex 450d6aaa86..77c4359522 100644\n--- a/git.c\n+++ b/git.c\n@@ -541,6 +541,7 @@ static struct cmd_struct commands[] = {\n \t{ \"diff\", cmd_diff, NO_PARSEOPT },\n \t{ \"diff-files\", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n \t{ \"diff-index\", cmd_diff_index, RUN_SETUP | NO_PARSEOPT },\n+\t{ \"diff-pairs\", cmd_diff_pairs, RUN_SETUP | NO_PARSEOPT },\n \t{ \"diff-tree\", cmd_diff_tree, RUN_SETUP | NO_PARSEOPT },\n \t{ \"difftool\", cmd_difftool, RUN_SETUP_GENTLY },\n \t{ \"fast-export\", cmd_fast_export, RUN_SETUP },\ndiff --git a/meson.build b/meson.build\nindex bf95576f83..9e8b365d2a 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -540,6 +540,7 @@ builtin_sources = [\n 'builtin/diagnose.c',\n 'builtin/diff-files.c',\n 'builtin/diff-index.c',\n+ 'builtin/diff-pairs.c',\n 'builtin/diff-tree.c',\n 'builtin/diff.c',\n 'builtin/difftool.c',\ndiff --git a/t/meson.build b/t/meson.build\nindex 780939d49f..09c7bc2fad 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -500,6 +500,7 @@ integration_tests = [\n 't4067-diff-partial-clone.sh',\n 't4068-diff-symmetric-merge-base.sh',\n 't4069-remerge-diff.sh',\n+ 't4070-diff-pairs.sh',\n 't4100-apply-stat.sh',\n 't4101-apply-nonl.sh',\n 't4102-apply-rename.sh',\ndiff --git a/t/t4070-diff-pairs.sh b/t/t4070-diff-pairs.sh\nnew file mode 100755\nindex 0000000000..8f17e55c7d\n--- /dev/null\n+++ b/t/t4070-diff-pairs.sh\n@@ -0,0 +1,81 @@\n+#!/bin/sh\n+\n+test_description='basic diff-pairs tests'\n+. ./test-lib.sh\n+\n+# This creates a diff with added, modified, deleted, renamed, copied, and\n+# typechange entries. This includes a submodule to test submodule diff support.\n+test_expect_success 'setup' '\n+\ttest_config_global protocol.file.allow always &&\n+\ttest_create_repo sub &&\n+\ttest_commit -C sub initial &&\n+\n+\ttest_create_repo main &&\n+\tcd main &&\n+\techo to-be-gone >deleted &&\n+\techo original >modified &&\n+\techo now-a-file >symlink &&\n+\ttest_seq 200 >two-hundred &&\n+\ttest_seq 201 500 >five-hundred &&\n+\tgit add . &&\n+\ttest_tick &&\n+\tgit commit -m base &&\n+\tgit tag base &&\n+\n+\tgit submodule add ../sub &&\n+\techo now-here >added &&\n+\techo new >modified &&\n+\trm deleted &&\n+\tmkdir subdir &&\n+\techo content >subdir/file &&\n+\tmv two-hundred renamed &&\n+\ttest_seq 201 500 | sed s/300/modified/ >copied &&\n+\trm symlink &&\n+\tgit add -A . &&\n+\ttest_ln_s_add dest symlink &&\n+\ttest_tick &&\n+\tgit commit -m new &&\n+\tgit tag new\n+'\n+\n+test_expect_success 'diff-pairs recreates --raw' '\n+\tgit diff-tree -r -M -C -C -z base new >expect &&\n+\tgit diff-pairs --raw -z >actual expect &&\n+\tgit diff-tree -r -M -C -C -z base new |\n+\tgit diff-pairs -p -z >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'diff-pairs does not support normal raw diff input' '\n+\tgit diff-tree -r base new |\n+\ttest_must_fail git diff-pairs >out 2>err &&\n+\n+\techo \"usage: working without -z is not supported\" >expect &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'diff-pairs does not support tree objects as input' '\n+\tgit diff-tree -z base new |\n+\ttest_must_fail git diff-pairs -z >out 2>err &&\n+\n+\techo \"fatal: tree objects not supported\" >expect &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'diff-pairs does not support pathspec arguments' '\n+\tgit diff-tree -r -z base new |\n+\ttest_must_fail git diff-pairs -z -- new >out 2>err &&\n+\n+\techo \"usage: pathspec arguments not supported\" >expect &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n+test_done\n-- \n2.48.1\n\n"},{"id":"513211","messageId":"20250228002604.3859939-5-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250228002604.3859939-1-jltobler@gmail.com","subject":"[PATCH v4 4/4] builtin/diff-pairs: allow explicit diff queue flush","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-28T00:26:04Z","receivedAt":"2025-02-28T00:29:31Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"The diffs queued from git-diff-pairs(1) are flushed when stdin is\nclosed. To enable greater flexibility, allow control over when the diff\nqueue is flushed by writing a single NUL byte on stdin between input\nfile pairs. Diff output between flushes is separated by a single NUL\nbyte.\n\nSigned-off-by: Justin Tobler \n---\n Documentation/git-diff-pairs.adoc | 4 ++++\n builtin/diff-pairs.c | 14 ++++++++++++++\n t/t4070-diff-pairs.sh | 9 +++++++++\n 3 files changed, 27 insertions(+)\n\ndiff --git a/Documentation/git-diff-pairs.adoc b/Documentation/git-diff-pairs.adoc\nindex e31f2e2fbb..f99fcd1ead 100644\n--- a/Documentation/git-diff-pairs.adoc\n+++ b/Documentation/git-diff-pairs.adoc\n@@ -17,6 +17,10 @@ in the NUL-terminated raw output format as generated by commands such as `git\n diff-tree -z -r --raw`. By default, the outputted diffs are computed and shown\n in the patch format when stdin closes.\n \n+A single NUL byte may be written to stdin between raw input lines to compute\n+file pair diffs up to that point instead of waiting for stdin to close. A NUL\n+byte is also written to the output to delimit between these batches of diffs.\n+\n Usage of this command enables the traditional diff pipeline to be broken up\n into separate stages where `diff-pairs` acts as the output phase. Other\n commands, such as `diff-tree`, may serve as a frontend to compute the raw\ndiff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\nindex 5a993b7c9d..2939d4af1d 100644\n--- a/builtin/diff-pairs.c\n+++ b/builtin/diff-pairs.c\n@@ -57,6 +57,7 @@ int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n \tshow_usage_with_options_if_asked(argc, argv, usagestr, parseopts);\n \n \trepo_config(repo, git_diff_basic_config, NULL);\n+\trevs.diffopt.no_free = 1;\n \trevs.disable_stdin = 1;\n \trevs.abbrev = 0;\n \trevs.diff = 1;\n@@ -108,6 +109,18 @@ int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n \t\t\tbreak;\n \n \t\tp = meta.buf;\n+\t\tif (!*p) {\n+\t\t\tdiffcore_std(&revs.diffopt);\n+\t\t\tdiff_flush(&revs.diffopt);\n+\t\t\t/*\n+\t\t\t * When the diff queue is explicitly flushed, append a\n+\t\t\t * NUL byte to separate batches of diffs.\n+\t\t\t */\n+\t\t\tfputc('\\0', revs.diffopt.file);\n+\t\t\tfflush(revs.diffopt.file);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (*p != ':')\n \t\t\tdie(_(\"invalid raw diff input\"));\n \t\tp++;\n@@ -181,6 +194,7 @@ int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n \t\t}\n \t}\n \n+\trevs.diffopt.no_free = 0;\n \tdiffcore_std(&revs.diffopt);\n \tdiff_flush(&revs.diffopt);\n \tret = diff_result_code(&revs);\ndiff --git a/t/t4070-diff-pairs.sh b/t/t4070-diff-pairs.sh\nindex 8f17e55c7d..c5e9972b2d 100755\n--- a/t/t4070-diff-pairs.sh\n+++ b/t/t4070-diff-pairs.sh\n@@ -78,4 +78,13 @@ test_expect_success 'diff-pairs does not support pathspec arguments' '\n \ttest_cmp expect err\n '\n \n+test_expect_success 'diff-pairs explicit queue flush' '\n+\tgit diff-tree -r -M -C -C -z base new >expect &&\n+\tprintf \"\\0\" >>expect &&\n+\tgit diff-tree -r -M -C -C -z base new >>expect &&\n+\n+\tgit diff-pairs --raw -z actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \n2.48.1\n\n"},{"id":"513239","messageId":"Z8Fz5XHtyG_m0_bX@pks.im","threadId":"62633","inReplyTo":"20250228002604.3859939-3-jltobler@gmail.com","subject":"Re: [PATCH v4 2/4] diff: add option to skip resolving diff statuses","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T08:29:25Z","receivedAt":"2025-02-28T08:29:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Feb 27, 2025 at 06:26:02PM -0600, Justin Tobler wrote:\n> By default, `diffcore_std()` resolves the statuses for queued diff file\n> pairs by calling `diff_resolve_rename_copy()`. If status information is\n> already manually set, invoking `diffcore_std()` may change the status\n> value.\n> \n> Introduce the `skip_resolving_statuses` diff option that prevents\n> `diffcore_std()` from resolving file pair statuses when enabled.\n\nYou mentioned to me that there was another user that basically abused\n`found_follow` to skip over this, which seems to be in \"tree-diff.c\".\nWould it make sense to convert that user to use the new mechanism, as\nwell, so that we don't mix up options and state?\n\nPatrick\n"},{"id":"513240","messageId":"Z8Fz7CQimWgzrvz7@pks.im","threadId":"62633","inReplyTo":"20250228002604.3859939-4-jltobler@gmail.com","subject":"Re: [PATCH v4 3/4] builtin: introduce diff-pairs command","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T08:29:32Z","receivedAt":"2025-02-28T08:29:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Feb 27, 2025 at 06:26:03PM -0600, Justin Tobler wrote:\n> diff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\n> new file mode 100644\n> index 0000000000..5a993b7c9d\n> --- /dev/null\n> +++ b/builtin/diff-pairs.c\n[snip]\n> +int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n> +\t\t struct repository *repo)\n> +{\n> +\tstruct strbuf path_dst = STRBUF_INIT;\n> +\tstruct strbuf path = STRBUF_INIT;\n> +\tstruct strbuf meta = STRBUF_INIT;\n> +\tstruct option *parseopts;\n> +\tstruct rev_info revs;\n> +\tint line_term = '\\0';\n> +\tint ret;\n> +\n> +\tconst char * const usagestr[] = {\n> +\t\tN_(\"git diff-pairs -z []\"),\n> +\t\tNULL\n> +\t};\n\nWe tend to call these `builtin_*_usage`, so in your case it would be\n`builtin_diff_pairs_usage`.\n\n> +\tstruct option options[] = {\n> +\t\tOPT_END()\n> +\t};\n> +\n> +\trepo_init_revisions(repo, &revs, prefix);\n> +\n> +\t/*\n> +\t * Diff options are usually parsed implicitly as part of\n> +\t * setup_revisions(). Explicitly handle parsing to ensure options are\n> +\t * printed in the usage message.\n> +\t */\n> +\tparseopts = add_diff_options(options, &revs.diffopt);\n> +\tshow_usage_with_options_if_asked(argc, argv, usagestr, parseopts);\n> +\n> +\trepo_config(repo, git_diff_basic_config, NULL);\n> +\trevs.disable_stdin = 1;\n> +\trevs.abbrev = 0;\n> +\trevs.diff = 1;\n> +\n> +\targc = parse_options(argc, argv, prefix, parseopts, usagestr,\n> +\t\t\t PARSE_OPT_KEEP_UNKNOWN_OPT |\n> +\t\t\t PARSE_OPT_KEEP_DASHDASH |\n> +\t\t\t PARSE_OPT_KEEP_ARGV0);\n> \n> +\tif (setup_revisions(argc, argv, &revs, NULL) > 1)\n> +\t\tusagef(_(\"unrecognized argument: %s\"), argv[0]);\n\nOkay, we now use `parse_options()` to parse stuff for us, and\n`setup_revisions()` only really does the setup for us as we know that\nall relevant diff options should've already been parsed for us. This\nlooks much nicer to me.\n\nI wonder though: we keep unknown options when calling `parse_options()`\nand then end up passing them to `setup_revisions()`. But are there even\nany options handled by `setup_revisions()` that would make sense in our\ncontext? And if not, shouldn't we rather make `parse_options()` die in\ncase it sees unknown options?\n\nIf there are, we should probably document this because it isn't obvious\nto me.\n\n> diff --git a/t/t4070-diff-pairs.sh b/t/t4070-diff-pairs.sh\n> new file mode 100755\n> index 0000000000..8f17e55c7d\n> --- /dev/null\n> +++ b/t/t4070-diff-pairs.sh\n> @@ -0,0 +1,81 @@\n> +#!/bin/sh\n> +\n> +test_description='basic diff-pairs tests'\n> +. ./test-lib.sh\n> +\n> +# This creates a diff with added, modified, deleted, renamed, copied, and\n> +# typechange entries. This includes a submodule to test submodule diff support.\n> +test_expect_success 'setup' '\n> +\ttest_config_global protocol.file.allow always &&\n> +\ttest_create_repo sub &&\n\nUse of `test_create_repo ()` is deprecated, as it is merely a wrapper\naround git-init(1).\n\nPatrick\n"},{"id":"513288","messageId":"r33nifirglhjiqhwxa37fuujvzoxefzs7g5ovjdtx43d6wu76m@mca3i66mgmtt","threadId":"62633","inReplyTo":"Z8Fz5XHtyG_m0_bX@pks.im","subject":"Re: [PATCH v4 2/4] diff: add option to skip resolving diff statuses","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-28T17:10:32Z","receivedAt":"2025-02-28T17:13:49Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/28 09:29AM, Patrick Steinhardt wrote:\n> On Thu, Feb 27, 2025 at 06:26:02PM -0600, Justin Tobler wrote:\n> > By default, `diffcore_std()` resolves the statuses for queued diff file\n> > pairs by calling `diff_resolve_rename_copy()`. If status information is\n> > already manually set, invoking `diffcore_std()` may change the status\n> > value.\n> > \n> > Introduce the `skip_resolving_statuses` diff option that prevents\n> > `diffcore_std()` from resolving file pair statuses when enabled.\n> \n> You mentioned to me that there was another user that basically abused\n> `found_follow` to skip over this, which seems to be in \"tree-diff.c\".\n> Would it make sense to convert that user to use the new mechanism, as\n> well, so that we don't mix up options and state?\n\nI was mixed up with something else and mistaken. There is only the one\nexpected existing user of the `found_follow` option. Apoligies for any\nconfusion.\n\n-Justin\n"},{"id":"513290","messageId":"yb3d32awwgdv4yp2ma6udqn5w46stemptje2ykdkgwfeqxrwac@zn7ifcssc36u","threadId":"62633","inReplyTo":"Z8Fz7CQimWgzrvz7@pks.im","subject":"Re: [PATCH v4 3/4] builtin: introduce diff-pairs command","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-28T17:26:54Z","receivedAt":"2025-02-28T17:30:12Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/28 09:29AM, Patrick Steinhardt wrote:\n> On Thu, Feb 27, 2025 at 06:26:03PM -0600, Justin Tobler wrote:\n> > diff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\n> > new file mode 100644\n> > index 0000000000..5a993b7c9d\n> > --- /dev/null\n> > +++ b/builtin/diff-pairs.c\n> [snip]\n> > +int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n> > +\t\t struct repository *repo)\n> > +{\n> > +\tstruct strbuf path_dst = STRBUF_INIT;\n> > +\tstruct strbuf path = STRBUF_INIT;\n> > +\tstruct strbuf meta = STRBUF_INIT;\n> > +\tstruct option *parseopts;\n> > +\tstruct rev_info revs;\n> > +\tint line_term = '\\0';\n> > +\tint ret;\n> > +\n> > +\tconst char * const usagestr[] = {\n> > +\t\tN_(\"git diff-pairs -z []\"),\n> > +\t\tNULL\n> > +\t};\n> \n> We tend to call these `builtin_*_usage`, so in your case it would be\n> `builtin_diff_pairs_usage`.\n\nGood to know, will adapt in a followup version.\n\n> \n> > +\tstruct option options[] = {\n> > +\t\tOPT_END()\n> > +\t};\n> > +\n> > +\trepo_init_revisions(repo, &revs, prefix);\n> > +\n> > +\t/*\n> > +\t * Diff options are usually parsed implicitly as part of\n> > +\t * setup_revisions(). Explicitly handle parsing to ensure options are\n> > +\t * printed in the usage message.\n> > +\t */\n> > +\tparseopts = add_diff_options(options, &revs.diffopt);\n> > +\tshow_usage_with_options_if_asked(argc, argv, usagestr, parseopts);\n> > +\n> > +\trepo_config(repo, git_diff_basic_config, NULL);\n> > +\trevs.disable_stdin = 1;\n> > +\trevs.abbrev = 0;\n> > +\trevs.diff = 1;\n> > +\n> > +\targc = parse_options(argc, argv, prefix, parseopts, usagestr,\n> > +\t\t\t PARSE_OPT_KEEP_UNKNOWN_OPT |\n> > +\t\t\t PARSE_OPT_KEEP_DASHDASH |\n> > +\t\t\t PARSE_OPT_KEEP_ARGV0);\n> > \n> > +\tif (setup_revisions(argc, argv, &revs, NULL) > 1)\n> > +\t\tusagef(_(\"unrecognized argument: %s\"), argv[0]);\n> \n> Okay, we now use `parse_options()` to parse stuff for us, and\n> `setup_revisions()` only really does the setup for us as we know that\n> all relevant diff options should've already been parsed for us. This\n> looks much nicer to me.\n> \n> I wonder though: we keep unknown options when calling `parse_options()`\n> and then end up passing them to `setup_revisions()`. But are there even\n> any options handled by `setup_revisions()` that would make sense in our\n> context? And if not, shouldn't we rather make `parse_options()` die in\n> case it sees unknown options?\n\nGood catch, there should not be any actaully needed options left for\n`setup_revisions()` to parse as they should all be handled by\n`parse_options()`. I'll remove the `PARSE_OPT_KEEP_UNKNOWN_OPT` flag.\n\n> If there are, we should probably document this because it isn't obvious\n> to me.\n> \n> > diff --git a/t/t4070-diff-pairs.sh b/t/t4070-diff-pairs.sh\n> > new file mode 100755\n> > index 0000000000..8f17e55c7d\n> > --- /dev/null\n> > +++ b/t/t4070-diff-pairs.sh\n> > @@ -0,0 +1,81 @@\n> > +#!/bin/sh\n> > +\n> > +test_description='basic diff-pairs tests'\n> > +. ./test-lib.sh\n> > +\n> > +# This creates a diff with added, modified, deleted, renamed, copied, and\n> > +# typechange entries. This includes a submodule to test submodule diff support.\n> > +test_expect_success 'setup' '\n> > +\ttest_config_global protocol.file.allow always &&\n> > +\ttest_create_repo sub &&\n> \n> Use of `test_create_repo ()` is deprecated, as it is merely a wrapper\n> around git-init(1).\n\nGood to know! I'll swap to using git-init(1) instead.\n\nThanks\n-Justin\n"},{"id":"513297","messageId":"20250228213346.1335224-1-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250228002604.3859939-1-jltobler@gmail.com","subject":"[PATCH v5 0/4] batch blob diff generation","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-28T21:33:42Z","receivedAt":"2025-02-28T21:37:26Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Through git-diff(1) it is possible to generate a diff directly between\ntwo blobs. This is particularly useful when the pre-image and post-image\nblobs are known and we only care about the diff between them.\nUnfortunately, if a user has a batch of known blob pairs to compute\ndiffs for, there is currently not a way to do so via a single Git\nprocess.\n\nTo enable support for batch diffs of multiple blob pairs, this series\nintroduces a new diff plumbing command git-diff-pairs(1) based on a\nprevious patch series submitted by Peff[1]. This command uses\nNUL-delimited raw diffs as its source of input to control exactly which\nfilepairs are diffed. The advantage of using the raw diff format is that\nit already has diff status type and object context information embedded\nin each line making it more efficient to generate diffs with as we can\navoid having to peel revisions to get some the same info.\n\nFor example:\n\n git diff-tree -r -z -M $old $new |\n git diff-pairs -p -z\n\nHere the output of git-diff-tree(1) is fed to git-diff-pairs(1) to\ngenerate the same output that would be expected from `git diff-tree -p\n-M`. While by itself not particularly useful, this means it is possible\nto split git-diff-tree(1) output across multiple git-diff-pairs(1)\nprocesses. Such a feature is useful on the server-side where diffs\nbewteen a large set of changes may not be feasible all at once due to\ntimeout concerns.\n\nThis command can be viewed as a backend tool that exposes Git's diff\nmachinery. In its current form, the frontend that generates the raw diff\nlines used as input is expected to most of the heavy lifting (ie.\npathspec limiting, tree object expansion).\n\nThis series is structured as follows:\n\n - Patch 1 adds some new helper functions to get access to the queued\n `diff_filepair` after `diff_queue()` is invoked.\n\n - Patch 2 adds a new diff_options field that can be used to disable\n diff filepair status resolution. This prevents rename/copy\n statuses set from stdin from being altered when `diffcore_std()`\n is invoked.\n\n - Patch 3 introduces the new git-diff-pairs(1) plumbing command.\n\n - Patch 4 allows git-diff-pairs(1) to immediately compute diffs\n queued on stdin when a NUL-byte is written after a raw input line\n instead of waiting for stdin to close.\n\nChanges since V4:\n\n - Renamed usage and options variables to better follow convention.\n\n - Removed unneeded PARSE_OPT_KEEP_UNKNOWN_OPT from\n `parse_options()`.\n\n - Instead of using the deprecated `test_create_repo ()` in the\n tests, plain git-init(1) is used.\n\nChanges since V3:\n\n - Instead of relying on found_follow to prevent `diffcore_std()`\n from mutating diff filepair statuses, a new `diff_options` field,\n `skip_resolving_statuses` is introduced to achieve the same result\n in a more specific manner.\n\n - Parsing of diff options is now handled directly instead of going\n through `setup_revisions()`. This is done to so the diff options\n can be appended to the usage options and printed in the usage\n message.\n\n - Swapped to using `strbuf_getwholeline()` during stdin parsing to\n make the line termiantor more configurable in the future.\n\n - Stopped printing the usage message on errors to avoid masking the\n underlying error message.\n\n - Added test setup to exercise submodule change diffs.\n\n - Other small minor cleanups.\n\nChanges since V2:\n\n - Pathspecs are not supported and thus rejected when provided as\n arguments. It should be possible in a future series to add support\n though.\n\n - Tree objects present in `diff-pairs` input are rejected. Support\n for tree objects could be added in the future, but for now they\n are rejected to enable to future support in a backwards compatible\n manner.\n\n - The -z option is required by git-diff-pairs(1). The NUL-delimited\n raw diff format is the only accepted form of input. Consequently,\n NUL-delimited output is the only option in the `--raw` mode.\n\n - git-diff-pairs(1) defaults to patch output instead of raw output.\n This better fits the intended usecase of the command.\n\n - A NUL-byte is now always used as the delimiter between batches of\n file pair diffs when queued diffs are explicitly computed by\n writing a NUL-byte on stdin.\n\n - Several other small cleanups and fixes along with documentation\n changes.\n\nChanges since V1:\n\n - Changed from git-diff-blob(1) to git-diff-pairs(1) based on a\n previously submitted series.\n\n - Instead of each line containing a pair of blob revisions, the raw\n diff format is used as input which already has diff status and\n object context embedded.\n\n-Justin\n\n[1]: <20161201204042.6yslbyrg7l6ghhww@sigill.intra.peff.net>\n\nJustin Tobler (4):\n diff: return diff_filepair from diff queue helpers\n diff: add option to skip resolving diff statuses\n builtin: introduce diff-pairs command\n builtin/diff-pairs: allow explicit diff queue flush\n\n .gitignore | 1 +\n Documentation/git-diff-pairs.adoc | 60 +++++++++\n Documentation/meson.build | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/diff-pairs.c | 207 ++++++++++++++++++++++++++++++\n command-list.txt | 1 +\n diff.c | 72 ++++++++---\n diff.h | 33 +++++\n git.c | 1 +\n meson.build | 1 +\n t/meson.build | 1 +\n t/t4070-diff-pairs.sh | 90 +++++++++++++\n 13 files changed, 449 insertions(+), 21 deletions(-)\n create mode 100644 Documentation/git-diff-pairs.adoc\n create mode 100644 builtin/diff-pairs.c\n create mode 100755 t/t4070-diff-pairs.sh\n\nRange-diff against v4:\n1: b2e5486442 = 1: b2e5486442 diff: return diff_filepair from diff queue helpers\n2: 31d80d99ae = 2: 31d80d99ae diff: add option to skip resolving diff statuses\n3: 3722c02112 ! 3: 1024a4290c builtin: introduce diff-pairs command\n @@ builtin/diff-pairs.c (new)\n +\tint line_term = '\\0';\n +\tint ret;\n +\n -+\tconst char * const usagestr[] = {\n ++\tconst char * const builtin_diff_pairs_usage[] = {\n +\t\tN_(\"git diff-pairs -z []\"),\n +\t\tNULL\n +\t};\n -+\tstruct option options[] = {\n ++\tstruct option builtin_diff_pairs_options[] = {\n +\t\tOPT_END()\n +\t};\n +\n @@ builtin/diff-pairs.c (new)\n +\t * setup_revisions(). Explicitly handle parsing to ensure options are\n +\t * printed in the usage message.\n +\t */\n -+\tparseopts = add_diff_options(options, &revs.diffopt);\n -+\tshow_usage_with_options_if_asked(argc, argv, usagestr, parseopts);\n ++\tparseopts = add_diff_options(builtin_diff_pairs_options, &revs.diffopt);\n ++\tshow_usage_with_options_if_asked(argc, argv, builtin_diff_pairs_usage, parseopts);\n +\n +\trepo_config(repo, git_diff_basic_config, NULL);\n +\trevs.disable_stdin = 1;\n +\trevs.abbrev = 0;\n +\trevs.diff = 1;\n +\n -+\targc = parse_options(argc, argv, prefix, parseopts, usagestr,\n -+\t\t\t PARSE_OPT_KEEP_UNKNOWN_OPT |\n -+\t\t\t PARSE_OPT_KEEP_DASHDASH |\n -+\t\t\t PARSE_OPT_KEEP_ARGV0);\n ++\targc = parse_options(argc, argv, prefix, parseopts, builtin_diff_pairs_usage,\n ++\t\t\t PARSE_OPT_KEEP_ARGV0 | PARSE_OPT_KEEP_DASHDASH);\n +\n +\tif (setup_revisions(argc, argv, &revs, NULL) > 1)\n +\t\tusagef(_(\"unrecognized argument: %s\"), argv[0]);\n @@ t/t4070-diff-pairs.sh (new)\n +# typechange entries. This includes a submodule to test submodule diff support.\n +test_expect_success 'setup' '\n +\ttest_config_global protocol.file.allow always &&\n -+\ttest_create_repo sub &&\n ++\tgit init sub &&\n +\ttest_commit -C sub initial &&\n +\n -+\ttest_create_repo main &&\n ++\tgit init main &&\n +\tcd main &&\n +\techo to-be-gone >deleted &&\n +\techo original >modified &&\n4: a4809cbd80 ! 4: 56f21b664e builtin/diff-pairs: allow explicit diff queue flush\n @@ Documentation/git-diff-pairs.adoc: in the NUL-terminated raw output format as ge\n \n ## builtin/diff-pairs.c ##\n @@ builtin/diff-pairs.c: int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n - \tshow_usage_with_options_if_asked(argc, argv, usagestr, parseopts);\n + \tshow_usage_with_options_if_asked(argc, argv, builtin_diff_pairs_usage, parseopts);\n \n \trepo_config(repo, git_diff_basic_config, NULL);\n +\trevs.diffopt.no_free = 1;\n-- \n2.49.0.rc0\n\n"},{"id":"513298","messageId":"20250228213346.1335224-2-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250228213346.1335224-1-jltobler@gmail.com","subject":"[PATCH v5 1/4] diff: return diff_filepair from diff queue helpers","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-28T21:33:43Z","receivedAt":"2025-02-28T21:37:27Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"The `diff_addremove()` and `diff_change()` functions set up and queue\ndiffs, but do not return the `diff_filepair` added to the queue. In a\nsubsequent commit, modifications to `diff_filepair` need to occur in\ncertain cases after being queued.\n\nSince the existing `diff_addremove()` and `diff_change()` are also used\nfor callbacks in `diff_options` as types `add_remove_fn_t` and\n`change_fn_t`, modifying the existing function signatures requires\nfurther changes. The diff options for pruning use `file_add_remove()`\nand `file_change()` where file pairs do not even get queued. Thus,\nseparate functions are implemented instead.\n\nSplit out the queuing operations into `diff_queue_addremove()` and\n`diff_queue_change()` which also return a handle to the queued\n`diff_filepair`. Both `diff_addremove()` and `diff_change()` are\nreimplemented as thin wrappers around the new functions.\n\nSigned-off-by: Justin Tobler \n---\n diff.c | 70 +++++++++++++++++++++++++++++++++++++++++-----------------\n diff.h | 25 +++++++++++++++++++++\n 2 files changed, 75 insertions(+), 20 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 019fb893a7..b5a779f997 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -7157,16 +7157,19 @@ void compute_diffstat(struct diff_options *options,\n \toptions->found_changes = !!diffstat->nr;\n }\n \n-void diff_addremove(struct diff_options *options,\n-\t\t int addremove, unsigned mode,\n-\t\t const struct object_id *oid,\n-\t\t int oid_valid,\n-\t\t const char *concatpath, unsigned dirty_submodule)\n+struct diff_filepair *diff_queue_addremove(struct diff_queue_struct *queue,\n+\t\t\t\t\t struct diff_options *options,\n+\t\t\t\t\t int addremove, unsigned mode,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t int oid_valid,\n+\t\t\t\t\t const char *concatpath,\n+\t\t\t\t\t unsigned dirty_submodule)\n {\n \tstruct diff_filespec *one, *two;\n+\tstruct diff_filepair *pair;\n \n \tif (S_ISGITLINK(mode) && is_submodule_ignored(concatpath, options))\n-\t\treturn;\n+\t\treturn NULL;\n \n \t/* This may look odd, but it is a preparation for\n \t * feeding \"there are unchanged files which should\n@@ -7186,7 +7189,7 @@ void diff_addremove(struct diff_options *options,\n \n \tif (options->prefix &&\n \t strncmp(concatpath, options->prefix, options->prefix_length))\n-\t\treturn;\n+\t\treturn NULL;\n \n \tone = alloc_filespec(concatpath);\n \ttwo = alloc_filespec(concatpath);\n@@ -7198,25 +7201,29 @@ void diff_addremove(struct diff_options *options,\n \t\ttwo->dirty_submodule = dirty_submodule;\n \t}\n \n-\tdiff_queue(&diff_queued_diff, one, two);\n+\tpair = diff_queue(queue, one, two);\n \tif (!options->flags.diff_from_contents)\n \t\toptions->flags.has_changes = 1;\n+\n+\treturn pair;\n }\n \n-void diff_change(struct diff_options *options,\n-\t\t unsigned old_mode, unsigned new_mode,\n-\t\t const struct object_id *old_oid,\n-\t\t const struct object_id *new_oid,\n-\t\t int old_oid_valid, int new_oid_valid,\n-\t\t const char *concatpath,\n-\t\t unsigned old_dirty_submodule, unsigned new_dirty_submodule)\n+struct diff_filepair *diff_queue_change(struct diff_queue_struct *queue,\n+\t\t\t\t\tstruct diff_options *options,\n+\t\t\t\t\tunsigned old_mode, unsigned new_mode,\n+\t\t\t\t\tconst struct object_id *old_oid,\n+\t\t\t\t\tconst struct object_id *new_oid,\n+\t\t\t\t\tint old_oid_valid, int new_oid_valid,\n+\t\t\t\t\tconst char *concatpath,\n+\t\t\t\t\tunsigned old_dirty_submodule,\n+\t\t\t\t\tunsigned new_dirty_submodule)\n {\n \tstruct diff_filespec *one, *two;\n \tstruct diff_filepair *p;\n \n \tif (S_ISGITLINK(old_mode) && S_ISGITLINK(new_mode) &&\n \t is_submodule_ignored(concatpath, options))\n-\t\treturn;\n+\t\treturn NULL;\n \n \tif (options->flags.reverse_diff) {\n \t\tSWAP(old_mode, new_mode);\n@@ -7227,7 +7234,7 @@ void diff_change(struct diff_options *options,\n \n \tif (options->prefix &&\n \t strncmp(concatpath, options->prefix, options->prefix_length))\n-\t\treturn;\n+\t\treturn NULL;\n \n \tone = alloc_filespec(concatpath);\n \ttwo = alloc_filespec(concatpath);\n@@ -7235,19 +7242,42 @@ void diff_change(struct diff_options *options,\n \tfill_filespec(two, new_oid, new_oid_valid, new_mode);\n \tone->dirty_submodule = old_dirty_submodule;\n \ttwo->dirty_submodule = new_dirty_submodule;\n-\tp = diff_queue(&diff_queued_diff, one, two);\n+\tp = diff_queue(queue, one, two);\n \n \tif (options->flags.diff_from_contents)\n-\t\treturn;\n+\t\treturn p;\n \n \tif (options->flags.quick && options->skip_stat_unmatch &&\n \t !diff_filespec_check_stat_unmatch(options->repo, p)) {\n \t\tdiff_free_filespec_data(p->one);\n \t\tdiff_free_filespec_data(p->two);\n-\t\treturn;\n+\t\treturn p;\n \t}\n \n \toptions->flags.has_changes = 1;\n+\n+\treturn p;\n+}\n+\n+void diff_addremove(struct diff_options *options, int addremove, unsigned mode,\n+\t\t const struct object_id *oid, int oid_valid,\n+\t\t const char *concatpath, unsigned dirty_submodule)\n+{\n+\tdiff_queue_addremove(&diff_queued_diff, options, addremove, mode, oid,\n+\t\t\t oid_valid, concatpath, dirty_submodule);\n+}\n+\n+void diff_change(struct diff_options *options,\n+\t\t unsigned old_mode, unsigned new_mode,\n+\t\t const struct object_id *old_oid,\n+\t\t const struct object_id *new_oid,\n+\t\t int old_oid_valid, int new_oid_valid,\n+\t\t const char *concatpath,\n+\t\t unsigned old_dirty_submodule, unsigned new_dirty_submodule)\n+{\n+\tdiff_queue_change(&diff_queued_diff, options, old_mode, new_mode,\n+\t\t\t old_oid, new_oid, old_oid_valid, new_oid_valid,\n+\t\t\t concatpath, old_dirty_submodule, new_dirty_submodule);\n }\n \n struct diff_filepair *diff_unmerge(struct diff_options *options, const char *path)\ndiff --git a/diff.h b/diff.h\nindex 0a566f5531..63afa17e84 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -508,6 +508,31 @@ void diff_set_default_prefix(struct diff_options *options);\n \n int diff_can_quit_early(struct diff_options *);\n \n+/*\n+ * Stages changes in the provided diff queue for file additions and deletions.\n+ * If a file pair gets queued, it is returned.\n+ */\n+struct diff_filepair *diff_queue_addremove(struct diff_queue_struct *queue,\n+\t\t\t\t\t struct diff_options *,\n+\t\t\t\t\t int addremove, unsigned mode,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t int oid_valid, const char *fullpath,\n+\t\t\t\t\t unsigned dirty_submodule);\n+\n+/*\n+ * Stages changes in the provided diff queue for file modifications.\n+ * If a file pair gets queued, it is returned.\n+ */\n+struct diff_filepair *diff_queue_change(struct diff_queue_struct *queue,\n+\t\t\t\t\tstruct diff_options *,\n+\t\t\t\t\tunsigned mode1, unsigned mode2,\n+\t\t\t\t\tconst struct object_id *old_oid,\n+\t\t\t\t\tconst struct object_id *new_oid,\n+\t\t\t\t\tint old_oid_valid, int new_oid_valid,\n+\t\t\t\t\tconst char *fullpath,\n+\t\t\t\t\tunsigned dirty_submodule1,\n+\t\t\t\t\tunsigned dirty_submodule2);\n+\n void diff_addremove(struct diff_options *,\n \t\t int addremove,\n \t\t unsigned mode,\n-- \n2.49.0.rc0\n\n"},{"id":"513299","messageId":"20250228213346.1335224-3-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250228213346.1335224-1-jltobler@gmail.com","subject":"[PATCH v5 2/4] diff: add option to skip resolving diff statuses","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-28T21:33:44Z","receivedAt":"2025-02-28T21:37:28Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"By default, `diffcore_std()` resolves the statuses for queued diff file\npairs by calling `diff_resolve_rename_copy()`. If status information is\nalready manually set, invoking `diffcore_std()` may change the status\nvalue.\n\nIntroduce the `skip_resolving_statuses` diff option that prevents\n`diffcore_std()` from resolving file pair statuses when enabled.\n\nSigned-off-by: Justin Tobler \n---\n diff.c | 2 +-\n diff.h | 8 ++++++++\n 2 files changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/diff.c b/diff.c\nindex b5a779f997..37cc88c75b 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -7081,7 +7081,7 @@ void diffcore_std(struct diff_options *options)\n \t\tdiffcore_order(options->orderfile);\n \tif (options->rotate_to)\n \t\tdiffcore_rotate(options);\n-\tif (!options->found_follow)\n+\tif (!options->found_follow && !options->skip_resolving_statuses)\n \t\t/* See try_to_follow_renames() in tree-diff.c */\n \t\tdiff_resolve_rename_copy();\n \tdiffcore_apply_filter(options);\ndiff --git a/diff.h b/diff.h\nindex 63afa17e84..fc791ee2cc 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -353,6 +353,14 @@ struct diff_options {\n \t/* to support internal diff recursion by --follow hack*/\n \tint found_follow;\n \n+\t/*\n+\t * By default, diffcore_std() resolves the statuses for queued diff file\n+\t * pairs by calling diff_resolve_rename_copy(). If status information\n+\t * has already been manually set, this option prevents diffcore_std()\n+\t * from resetting statuses.\n+\t */\n+\tint skip_resolving_statuses;\n+\n \t/* Callback which allows tweaking the options in diff_setup_done(). */\n \tvoid (*set_default)(struct diff_options *);\n \n-- \n2.49.0.rc0\n\n"},{"id":"513300","messageId":"20250228213346.1335224-4-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250228213346.1335224-1-jltobler@gmail.com","subject":"[PATCH v5 3/4] builtin: introduce diff-pairs command","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-28T21:33:45Z","receivedAt":"2025-02-28T21:37:29Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Through git-diff(1), a single diff can be generated from a pair of blob\nrevisions directly. Unfortunately, there is not a mechanism to compute\nbatches of specific file pair diffs in a single process. Such a feature\nis particularly useful on the server-side where diffing between a large\nset of changes is not feasible all at once due to timeout concerns.\n\nTo facilitate this, introduce git-diff-pairs(1) which acts as a backend\npassing its NUL-terminated raw diff format input from stdin through diff\nmachinery to produce various forms of output such as patch or raw.\n\nThe raw format was originally designed as an interchange format and\nrepresents the contents of the diff_queued_diff list making it possible\nto break the diff pipeline into separate stages. For example,\ngit-diff-tree(1) can be used as a frontend to compute file pairs to\nqueue and feed its raw output to git-diff-pairs(1) to compute patches.\nWith this, batches of diffs can be progressively generated without\nhaving to recompute renames or retrieve object context. Something like\nthe following:\n\n\tgit diff-tree -r -z -M $old $new |\n\tgit diff-pairs -p -z\n\nshould generate the same output as `git diff-tree -p -M`. Furthermore,\neach line of raw diff formatted input can also be individually fed to a\nseparate git-diff-pairs(1) process and still produce the same output.\n\nBased-on-patch-by: Jeff King \nSigned-off-by: Justin Tobler \n---\n .gitignore | 1 +\n Documentation/git-diff-pairs.adoc | 56 +++++++++\n Documentation/meson.build | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/diff-pairs.c | 193 ++++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n meson.build | 1 +\n t/meson.build | 1 +\n t/t4070-diff-pairs.sh | 81 +++++++++++++\n 11 files changed, 338 insertions(+)\n create mode 100644 Documentation/git-diff-pairs.adoc\n create mode 100644 builtin/diff-pairs.c\n create mode 100755 t/t4070-diff-pairs.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 08a66ca508..04c444404e 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -55,6 +55,7 @@\n /git-diff\n /git-diff-files\n /git-diff-index\n+/git-diff-pairs\n /git-diff-tree\n /git-difftool\n /git-difftool--helper\ndiff --git a/Documentation/git-diff-pairs.adoc b/Documentation/git-diff-pairs.adoc\nnew file mode 100644\nindex 0000000000..e31f2e2fbb\n--- /dev/null\n+++ b/Documentation/git-diff-pairs.adoc\n@@ -0,0 +1,56 @@\n+git-diff-pairs(1)\n+=================\n+\n+NAME\n+----\n+git-diff-pairs - Compare the content and mode of provided blob pairs\n+\n+SYNOPSIS\n+--------\n+[synopsis]\n+git diff-pairs -z []\n+\n+DESCRIPTION\n+-----------\n+Show changes for file pairs provided on stdin. Input for this command must be\n+in the NUL-terminated raw output format as generated by commands such as `git\n+diff-tree -z -r --raw`. By default, the outputted diffs are computed and shown\n+in the patch format when stdin closes.\n+\n+Usage of this command enables the traditional diff pipeline to be broken up\n+into separate stages where `diff-pairs` acts as the output phase. Other\n+commands, such as `diff-tree`, may serve as a frontend to compute the raw\n+diff format used as input.\n+\n+Instead of computing diffs via `git diff-tree -p -M` in one step, `diff-tree`\n+can compute the file pairs and rename information without the blob diffs. This\n+output can be fed to `diff-pairs` to generate the underlying blob diffs as done\n+in the following example:\n+\n+-----------------------------\n+git diff-tree -z -r -M $a $b |\n+git diff-pairs -z\n+-----------------------------\n+\n+Computing the tree diff upfront with rename information allows patch output\n+from `diff-pairs` to be progressively computed over the course of potentially\n+multiple invocations.\n+\n+Pathspecs are not currently supported by `diff-pairs`. Pathspec limiting should\n+be performed by the upstream command generating the raw diffs used as input.\n+\n+Tree objects are not currently supported as input and are rejected.\n+\n+Abbreviated object IDs in the `diff-pairs` input are not supported. Outputted\n+object IDs can be abbreviated using the `--abbrev` option.\n+\n+OPTIONS\n+-------\n+\n+include::diff-options.adoc[]\n+\n+include::diff-generate-patch.adoc[]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Documentation/meson.build b/Documentation/meson.build\nindex 1129ce4c85..ce990e9fe5 100644\n--- a/Documentation/meson.build\n+++ b/Documentation/meson.build\n@@ -42,6 +42,7 @@ manpages = {\n 'git-diagnose.adoc' : 1,\n 'git-diff-files.adoc' : 1,\n 'git-diff-index.adoc' : 1,\n+ 'git-diff-pairs.adoc' : 1,\n 'git-difftool.adoc' : 1,\n 'git-diff-tree.adoc' : 1,\n 'git-diff.adoc' : 1,\ndiff --git a/Makefile b/Makefile\nindex bcf5ed3f85..56df7aed3f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1242,6 +1242,7 @@ BUILTIN_OBJS += builtin/describe.o\n BUILTIN_OBJS += builtin/diagnose.o\n BUILTIN_OBJS += builtin/diff-files.o\n BUILTIN_OBJS += builtin/diff-index.o\n+BUILTIN_OBJS += builtin/diff-pairs.o\n BUILTIN_OBJS += builtin/diff-tree.o\n BUILTIN_OBJS += builtin/diff.o\n BUILTIN_OBJS += builtin/difftool.o\ndiff --git a/builtin.h b/builtin.h\nindex 89928ccf92..e6aad3a6a1 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -153,6 +153,7 @@ int cmd_diagnose(int argc, const char **argv, const char *prefix, struct reposit\n int cmd_diff_files(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_diff_index(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_diff(int argc, const char **argv, const char *prefix, struct repository *repo);\n+int cmd_diff_pairs(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_diff_tree(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_difftool(int argc, const char **argv, const char *prefix, struct repository *repo);\n int cmd_env__helper(int argc, const char **argv, const char *prefix, struct repository *repo);\ndiff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\nnew file mode 100644\nindex 0000000000..6be17c1abd\n--- /dev/null\n+++ b/builtin/diff-pairs.c\n@@ -0,0 +1,193 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"diff.h\"\n+#include \"diffcore.h\"\n+#include \"gettext.h\"\n+#include \"hash.h\"\n+#include \"hex.h\"\n+#include \"object.h\"\n+#include \"parse-options.h\"\n+#include \"revision.h\"\n+#include \"strbuf.h\"\n+\n+static unsigned parse_mode_or_die(const char *mode, const char **end)\n+{\n+\tuint16_t ret;\n+\n+\t*end = parse_mode(mode, &ret);\n+\tif (!*end)\n+\t\tdie(_(\"unable to parse mode: %s\"), mode);\n+\treturn ret;\n+}\n+\n+static void parse_oid_or_die(const char *hex, struct object_id *oid,\n+\t\t\t const char **end, const struct git_hash_algo *algop)\n+{\n+\tif (parse_oid_hex_algop(hex, oid, end, algop) || *(*end)++ != ' ')\n+\t\tdie(_(\"unable to parse object id: %s\"), hex);\n+}\n+\n+int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n+\t\t struct repository *repo)\n+{\n+\tstruct strbuf path_dst = STRBUF_INIT;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct strbuf meta = STRBUF_INIT;\n+\tstruct option *parseopts;\n+\tstruct rev_info revs;\n+\tint line_term = '\\0';\n+\tint ret;\n+\n+\tconst char * const builtin_diff_pairs_usage[] = {\n+\t\tN_(\"git diff-pairs -z []\"),\n+\t\tNULL\n+\t};\n+\tstruct option builtin_diff_pairs_options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\trepo_init_revisions(repo, &revs, prefix);\n+\n+\t/*\n+\t * Diff options are usually parsed implicitly as part of\n+\t * setup_revisions(). Explicitly handle parsing to ensure options are\n+\t * printed in the usage message.\n+\t */\n+\tparseopts = add_diff_options(builtin_diff_pairs_options, &revs.diffopt);\n+\tshow_usage_with_options_if_asked(argc, argv, builtin_diff_pairs_usage, parseopts);\n+\n+\trepo_config(repo, git_diff_basic_config, NULL);\n+\trevs.disable_stdin = 1;\n+\trevs.abbrev = 0;\n+\trevs.diff = 1;\n+\n+\targc = parse_options(argc, argv, prefix, parseopts, builtin_diff_pairs_usage,\n+\t\t\t PARSE_OPT_KEEP_ARGV0 | PARSE_OPT_KEEP_DASHDASH);\n+\n+\tif (setup_revisions(argc, argv, &revs, NULL) > 1)\n+\t\tusagef(_(\"unrecognized argument: %s\"), argv[0]);\n+\n+\t/*\n+\t * With the -z option, both command input and raw output are\n+\t * NUL-delimited (this mode does not affect patch output). At present\n+\t * only NUL-delimited raw diff formatted input is supported.\n+\t */\n+\tif (revs.diffopt.line_termination)\n+\t\tusage(_(\"working without -z is not supported\"));\n+\n+\tif (revs.prune_data.nr)\n+\t\tusage(_(\"pathspec arguments not supported\"));\n+\n+\tif (revs.pending.nr || revs.max_count != -1 ||\n+\t revs.min_age != (timestamp_t)-1 ||\n+\t revs.max_age != (timestamp_t)-1)\n+\t\tusage(_(\"revision arguments not allowed\"));\n+\n+\tif (!revs.diffopt.output_format)\n+\t\trevs.diffopt.output_format = DIFF_FORMAT_PATCH;\n+\n+\t/*\n+\t * If rename detection is not requested, use rename information from the\n+\t * raw diff formatted input. Setting skip_resolving_statuses ensures\n+\t * diffcore_std() does not mess with rename information already present\n+\t * in queued filepairs.\n+\t */\n+\tif (!revs.diffopt.detect_rename)\n+\t\trevs.diffopt.skip_resolving_statuses = 1;\n+\n+\twhile (1) {\n+\t\tstruct object_id oid_a, oid_b;\n+\t\tstruct diff_filepair *pair;\n+\t\tunsigned mode_a, mode_b;\n+\t\tconst char *p;\n+\t\tchar status;\n+\n+\t\tif (strbuf_getwholeline(&meta, stdin, line_term) == EOF)\n+\t\t\tbreak;\n+\n+\t\tp = meta.buf;\n+\t\tif (*p != ':')\n+\t\t\tdie(_(\"invalid raw diff input\"));\n+\t\tp++;\n+\n+\t\tmode_a = parse_mode_or_die(p, &p);\n+\t\tmode_b = parse_mode_or_die(p, &p);\n+\n+\t\tif (S_ISDIR(mode_a) || S_ISDIR(mode_b))\n+\t\t\tdie(_(\"tree objects not supported\"));\n+\n+\t\tparse_oid_or_die(p, &oid_a, &p, repo->hash_algo);\n+\t\tparse_oid_or_die(p, &oid_b, &p, repo->hash_algo);\n+\n+\t\tstatus = *p++;\n+\n+\t\tif (strbuf_getwholeline(&path, stdin, line_term) == EOF)\n+\t\t\tdie(_(\"got EOF while reading path\"));\n+\n+\t\tswitch (status) {\n+\t\tcase DIFF_STATUS_ADDED:\n+\t\t\tpair = diff_queue_addremove(&diff_queued_diff,\n+\t\t\t\t\t\t &revs.diffopt, '+', mode_b,\n+\t\t\t\t\t\t &oid_b, 1, path.buf, 0);\n+\t\t\tif (pair)\n+\t\t\t\tpair->status = status;\n+\t\t\tbreak;\n+\n+\t\tcase DIFF_STATUS_DELETED:\n+\t\t\tpair = diff_queue_addremove(&diff_queued_diff,\n+\t\t\t\t\t\t &revs.diffopt, '-', mode_a,\n+\t\t\t\t\t\t &oid_a, 1, path.buf, 0);\n+\t\t\tif (pair)\n+\t\t\t\tpair->status = status;\n+\t\t\tbreak;\n+\n+\t\tcase DIFF_STATUS_TYPE_CHANGED:\n+\t\tcase DIFF_STATUS_MODIFIED:\n+\t\t\tpair = diff_queue_change(&diff_queued_diff, &revs.diffopt,\n+\t\t\t\t\t\t mode_a, mode_b, &oid_a, &oid_b,\n+\t\t\t\t\t\t 1, 1, path.buf, 0, 0);\n+\t\t\tif (pair)\n+\t\t\t\tpair->status = status;\n+\t\t\tbreak;\n+\n+\t\tcase DIFF_STATUS_RENAMED:\n+\t\tcase DIFF_STATUS_COPIED: {\n+\t\t\t\tstruct diff_filespec *a, *b;\n+\t\t\t\tunsigned int score;\n+\n+\t\t\t\tif (strbuf_getwholeline(&path_dst, stdin, line_term) == EOF)\n+\t\t\t\t\tdie(_(\"got EOF while reading destination path\"));\n+\n+\t\t\t\ta = alloc_filespec(path.buf);\n+\t\t\t\tb = alloc_filespec(path_dst.buf);\n+\t\t\t\tfill_filespec(a, &oid_a, 1, mode_a);\n+\t\t\t\tfill_filespec(b, &oid_b, 1, mode_b);\n+\n+\t\t\t\tpair = diff_queue(&diff_queued_diff, a, b);\n+\n+\t\t\t\tif (strtoul_ui(p, 10, &score))\n+\t\t\t\t\tdie(_(\"unable to parse rename/copy score: %s\"), p);\n+\n+\t\t\t\tpair->score = score * MAX_SCORE / 100;\n+\t\t\t\tpair->status = status;\n+\t\t\t\tpair->renamed_pair = 1;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tdefault:\n+\t\t\tdie(_(\"unknown diff status: %c\"), status);\n+\t\t}\n+\t}\n+\n+\tdiffcore_std(&revs.diffopt);\n+\tdiff_flush(&revs.diffopt);\n+\tret = diff_result_code(&revs);\n+\n+\tstrbuf_release(&path_dst);\n+\tstrbuf_release(&path);\n+\tstrbuf_release(&meta);\n+\trelease_revisions(&revs);\n+\tFREE_AND_NULL(parseopts);\n+\n+\treturn ret;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex c537114b46..b7ade3ab9f 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -96,6 +96,7 @@ git-diagnose ancillaryinterrogators\n git-diff mainporcelain info\n git-diff-files plumbinginterrogators\n git-diff-index plumbinginterrogators\n+git-diff-pairs plumbinginterrogators\n git-diff-tree plumbinginterrogators\n git-difftool ancillaryinterrogators complete\n git-fast-export ancillarymanipulators\ndiff --git a/git.c b/git.c\nindex 450d6aaa86..77c4359522 100644\n--- a/git.c\n+++ b/git.c\n@@ -541,6 +541,7 @@ static struct cmd_struct commands[] = {\n \t{ \"diff\", cmd_diff, NO_PARSEOPT },\n \t{ \"diff-files\", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n \t{ \"diff-index\", cmd_diff_index, RUN_SETUP | NO_PARSEOPT },\n+\t{ \"diff-pairs\", cmd_diff_pairs, RUN_SETUP | NO_PARSEOPT },\n \t{ \"diff-tree\", cmd_diff_tree, RUN_SETUP | NO_PARSEOPT },\n \t{ \"difftool\", cmd_difftool, RUN_SETUP_GENTLY },\n \t{ \"fast-export\", cmd_fast_export, RUN_SETUP },\ndiff --git a/meson.build b/meson.build\nindex bf95576f83..9e8b365d2a 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -540,6 +540,7 @@ builtin_sources = [\n 'builtin/diagnose.c',\n 'builtin/diff-files.c',\n 'builtin/diff-index.c',\n+ 'builtin/diff-pairs.c',\n 'builtin/diff-tree.c',\n 'builtin/diff.c',\n 'builtin/difftool.c',\ndiff --git a/t/meson.build b/t/meson.build\nindex 780939d49f..09c7bc2fad 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -500,6 +500,7 @@ integration_tests = [\n 't4067-diff-partial-clone.sh',\n 't4068-diff-symmetric-merge-base.sh',\n 't4069-remerge-diff.sh',\n+ 't4070-diff-pairs.sh',\n 't4100-apply-stat.sh',\n 't4101-apply-nonl.sh',\n 't4102-apply-rename.sh',\ndiff --git a/t/t4070-diff-pairs.sh b/t/t4070-diff-pairs.sh\nnew file mode 100755\nindex 0000000000..0878ad0ad1\n--- /dev/null\n+++ b/t/t4070-diff-pairs.sh\n@@ -0,0 +1,81 @@\n+#!/bin/sh\n+\n+test_description='basic diff-pairs tests'\n+. ./test-lib.sh\n+\n+# This creates a diff with added, modified, deleted, renamed, copied, and\n+# typechange entries. This includes a submodule to test submodule diff support.\n+test_expect_success 'setup' '\n+\ttest_config_global protocol.file.allow always &&\n+\tgit init sub &&\n+\ttest_commit -C sub initial &&\n+\n+\tgit init main &&\n+\tcd main &&\n+\techo to-be-gone >deleted &&\n+\techo original >modified &&\n+\techo now-a-file >symlink &&\n+\ttest_seq 200 >two-hundred &&\n+\ttest_seq 201 500 >five-hundred &&\n+\tgit add . &&\n+\ttest_tick &&\n+\tgit commit -m base &&\n+\tgit tag base &&\n+\n+\tgit submodule add ../sub &&\n+\techo now-here >added &&\n+\techo new >modified &&\n+\trm deleted &&\n+\tmkdir subdir &&\n+\techo content >subdir/file &&\n+\tmv two-hundred renamed &&\n+\ttest_seq 201 500 | sed s/300/modified/ >copied &&\n+\trm symlink &&\n+\tgit add -A . &&\n+\ttest_ln_s_add dest symlink &&\n+\ttest_tick &&\n+\tgit commit -m new &&\n+\tgit tag new\n+'\n+\n+test_expect_success 'diff-pairs recreates --raw' '\n+\tgit diff-tree -r -M -C -C -z base new >expect &&\n+\tgit diff-pairs --raw -z >actual expect &&\n+\tgit diff-tree -r -M -C -C -z base new |\n+\tgit diff-pairs -p -z >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'diff-pairs does not support normal raw diff input' '\n+\tgit diff-tree -r base new |\n+\ttest_must_fail git diff-pairs >out 2>err &&\n+\n+\techo \"usage: working without -z is not supported\" >expect &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'diff-pairs does not support tree objects as input' '\n+\tgit diff-tree -z base new |\n+\ttest_must_fail git diff-pairs -z >out 2>err &&\n+\n+\techo \"fatal: tree objects not supported\" >expect &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'diff-pairs does not support pathspec arguments' '\n+\tgit diff-tree -r -z base new |\n+\ttest_must_fail git diff-pairs -z -- new >out 2>err &&\n+\n+\techo \"usage: pathspec arguments not supported\" >expect &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n+test_done\n-- \n2.49.0.rc0\n\n"},{"id":"513301","messageId":"20250228213346.1335224-5-jltobler@gmail.com","threadId":"62633","inReplyTo":"20250228213346.1335224-1-jltobler@gmail.com","subject":"[PATCH v5 4/4] builtin/diff-pairs: allow explicit diff queue flush","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-28T21:33:46Z","receivedAt":"2025-02-28T21:37:30Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"The diffs queued from git-diff-pairs(1) are flushed when stdin is\nclosed. To enable greater flexibility, allow control over when the diff\nqueue is flushed by writing a single NUL byte on stdin between input\nfile pairs. Diff output between flushes is separated by a single NUL\nbyte.\n\nSigned-off-by: Justin Tobler \n---\n Documentation/git-diff-pairs.adoc | 4 ++++\n builtin/diff-pairs.c | 14 ++++++++++++++\n t/t4070-diff-pairs.sh | 9 +++++++++\n 3 files changed, 27 insertions(+)\n\ndiff --git a/Documentation/git-diff-pairs.adoc b/Documentation/git-diff-pairs.adoc\nindex e31f2e2fbb..f99fcd1ead 100644\n--- a/Documentation/git-diff-pairs.adoc\n+++ b/Documentation/git-diff-pairs.adoc\n@@ -17,6 +17,10 @@ in the NUL-terminated raw output format as generated by commands such as `git\n diff-tree -z -r --raw`. By default, the outputted diffs are computed and shown\n in the patch format when stdin closes.\n \n+A single NUL byte may be written to stdin between raw input lines to compute\n+file pair diffs up to that point instead of waiting for stdin to close. A NUL\n+byte is also written to the output to delimit between these batches of diffs.\n+\n Usage of this command enables the traditional diff pipeline to be broken up\n into separate stages where `diff-pairs` acts as the output phase. Other\n commands, such as `diff-tree`, may serve as a frontend to compute the raw\ndiff --git a/builtin/diff-pairs.c b/builtin/diff-pairs.c\nindex 6be17c1abd..71c045331a 100644\n--- a/builtin/diff-pairs.c\n+++ b/builtin/diff-pairs.c\n@@ -57,6 +57,7 @@ int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n \tshow_usage_with_options_if_asked(argc, argv, builtin_diff_pairs_usage, parseopts);\n \n \trepo_config(repo, git_diff_basic_config, NULL);\n+\trevs.diffopt.no_free = 1;\n \trevs.disable_stdin = 1;\n \trevs.abbrev = 0;\n \trevs.diff = 1;\n@@ -106,6 +107,18 @@ int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n \t\t\tbreak;\n \n \t\tp = meta.buf;\n+\t\tif (!*p) {\n+\t\t\tdiffcore_std(&revs.diffopt);\n+\t\t\tdiff_flush(&revs.diffopt);\n+\t\t\t/*\n+\t\t\t * When the diff queue is explicitly flushed, append a\n+\t\t\t * NUL byte to separate batches of diffs.\n+\t\t\t */\n+\t\t\tfputc('\\0', revs.diffopt.file);\n+\t\t\tfflush(revs.diffopt.file);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (*p != ':')\n \t\t\tdie(_(\"invalid raw diff input\"));\n \t\tp++;\n@@ -179,6 +192,7 @@ int cmd_diff_pairs(int argc, const char **argv, const char *prefix,\n \t\t}\n \t}\n \n+\trevs.diffopt.no_free = 0;\n \tdiffcore_std(&revs.diffopt);\n \tdiff_flush(&revs.diffopt);\n \tret = diff_result_code(&revs);\ndiff --git a/t/t4070-diff-pairs.sh b/t/t4070-diff-pairs.sh\nindex 0878ad0ad1..70deafb860 100755\n--- a/t/t4070-diff-pairs.sh\n+++ b/t/t4070-diff-pairs.sh\n@@ -78,4 +78,13 @@ test_expect_success 'diff-pairs does not support pathspec arguments' '\n \ttest_cmp expect err\n '\n \n+test_expect_success 'diff-pairs explicit queue flush' '\n+\tgit diff-tree -r -M -C -C -z base new >expect &&\n+\tprintf \"\\0\" >>expect &&\n+\tgit diff-tree -r -M -C -C -z base new >>expect &&\n+\n+\tgit diff-pairs --raw -z actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \n2.49.0.rc0\n\n"},{"id":"513408","messageId":"xmqqpliyxf5u.fsf@gitster.g","threadId":"62633","inReplyTo":"20250228213346.1335224-2-jltobler@gmail.com","subject":"Re: [PATCH v5 1/4] diff: return diff_filepair from diff queue helpers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-03-03T16:17:01Z","receivedAt":"2025-03-03T16:17:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Justin Tobler writes:\n\n> The `diff_addremove()` and `diff_change()` functions set up and queue\n> diffs, but do not return the `diff_filepair` added to the queue. In a\n> subsequent commit, modifications to `diff_filepair` need to occur in\n> certain cases after being queued.\n>\n> Since the existing `diff_addremove()` and `diff_change()` are also used\n> for callbacks in `diff_options` as types `add_remove_fn_t` and\n> `change_fn_t`, modifying the existing function signatures requires\n> further changes.\n\nSensible. The patch presented below looks a sane and safe no-op for\nthe existing code paths, which is what we want to see in a preliminary\nrefactoring step like this one.\n\n> The diff options for pruning use `file_add_remove()`\n> and `file_change()` where file pairs do not even get queued. Thus,\n> separate functions are implemented instead.\n\nThis looked a bit confusing, but it is an explanation for the reason\nwhy we simply do not change the function signature of the callback\nmembers of diff_options structure. These addremove/change callbacks\nare designed to be a general way to allow applications to react to\ndiscovered changes to paths, and I agree that it makes sense for\nthem to be usable to perform something that has nothing to do with\nthe diff_queue structure.\n\n> Split out the queuing operations into `diff_queue_addremove()` and\n> `diff_queue_change()` which also return a handle to the queued\n> `diff_filepair`. Both `diff_addremove()` and `diff_change()` are\n> reimplemented as thin wrappers around the new functions.\n\nNice.\n"},{"id":"513409","messageId":"xmqqldtmxf0z.fsf@gitster.g","threadId":"62633","inReplyTo":"20250228213346.1335224-3-jltobler@gmail.com","subject":"Re: [PATCH v5 2/4] diff: add option to skip resolving diff statuses","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-03-03T16:19:56Z","receivedAt":"2025-03-03T16:19:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Justin Tobler writes:\n\n> By default, `diffcore_std()` resolves the statuses for queued diff file\n> pairs by calling `diff_resolve_rename_copy()`. If status information is\n> already manually set, invoking `diffcore_std()` may change the status\n> value.\n>\n> Introduce the `skip_resolving_statuses` diff option that prevents\n> `diffcore_std()` from resolving file pair statuses when enabled.\n\nMakes sense.\n\n>\n> Signed-off-by: Justin Tobler \n> ---\n> diff.c | 2 +-\n> diff.h | 8 ++++++++\n> 2 files changed, 9 insertions(+), 1 deletion(-)\n>\n> diff --git a/diff.c b/diff.c\n> index b5a779f997..37cc88c75b 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -7081,7 +7081,7 @@ void diffcore_std(struct diff_options *options)\n> \t\tdiffcore_order(options->orderfile);\n> \tif (options->rotate_to)\n> \t\tdiffcore_rotate(options);\n> -\tif (!options->found_follow)\n> +\tif (!options->found_follow && !options->skip_resolving_statuses)\n> \t\t/* See try_to_follow_renames() in tree-diff.c */\n> \t\tdiff_resolve_rename_copy();\n> \tdiffcore_apply_filter(options);\n> diff --git a/diff.h b/diff.h\n> index 63afa17e84..fc791ee2cc 100644\n> --- a/diff.h\n> +++ b/diff.h\n> @@ -353,6 +353,14 @@ struct diff_options {\n> \t/* to support internal diff recursion by --follow hack*/\n> \tint found_follow;\n> \n> +\t/*\n> +\t * By default, diffcore_std() resolves the statuses for queued diff file\n> +\t * pairs by calling diff_resolve_rename_copy(). If status information\n> +\t * has already been manually set, this option prevents diffcore_std()\n> +\t * from resetting statuses.\n> +\t */\n> +\tint skip_resolving_statuses;\n> +\n> \t/* Callback which allows tweaking the options in diff_setup_done(). */\n> \tvoid (*set_default)(struct diff_options *);\n"},{"id":"513410","messageId":"xmqq7c56xeja.fsf@gitster.g","threadId":"62633","inReplyTo":"20250228213346.1335224-4-jltobler@gmail.com","subject":"Re: [PATCH v5 3/4] builtin: introduce diff-pairs command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-03-03T16:30:33Z","receivedAt":"2025-03-03T16:30:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Justin Tobler writes:\n\n> +static unsigned parse_mode_or_die(const char *mode, const char **end)\n\nA minor naming issue, but the previous round called this endp, which\nis probably a better name; making it explicit that it is a pointer\nto receive the discovered end of converted string is in line with\nhow a similar parameter to strtol() and friends is named as\n\"endptr\".\n\nNot worth a reroll to rename this one alone, though.\n\n> +\twhile (1) {\n> +\t\tstruct object_id oid_a, oid_b;\n> +\t\tstruct diff_filepair *pair;\n> +\t\tunsigned mode_a, mode_b;\n> +\t\tconst char *p;\n> +\t\tchar status;\n> +\n> +\t\tif (strbuf_getwholeline(&meta, stdin, line_term) == EOF)\n> +\t\t\tbreak;\n\nNice.\n\n> diff --git a/git.c b/git.c\n> index 450d6aaa86..77c4359522 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -541,6 +541,7 @@ static struct cmd_struct commands[] = {\n> \t{ \"diff\", cmd_diff, NO_PARSEOPT },\n> \t{ \"diff-files\", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },\n> \t{ \"diff-index\", cmd_diff_index, RUN_SETUP | NO_PARSEOPT },\n> +\t{ \"diff-pairs\", cmd_diff_pairs, RUN_SETUP | NO_PARSEOPT },\n\nOK. We need a repository to find objects named in our input, but we\ndo not need working tree. Makes sense.\n\n"}]}