{"thread":{"id":"59197","subject":"[PATCH 0/2] Teach diff to honor diff algorithms set through git attributes","startedAt":"2023-02-05T03:46:30Z","lastAt":"2023-02-24T17:44:57Z","messageCount":78,"participants":["John Cai via GitGitGadget","Eric Sunshine","John Cai","Phillip Wood","Ævar Arnfjörð Bjarmason","Jeff King","Junio C Hamano","Elijah Newren"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"471511","messageId":"pull.1452.git.git.1675568781.gitgitgadget@gmail.com","threadId":"59197","inReplyTo":null,"subject":"[PATCH 0/2] Teach diff to honor diff algorithms set through git attributes","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-05T03:46:19Z","receivedAt":"2023-02-05T03:46:30Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"When a repository contains different kinds of files, it may be desirable to\nuse different algorithms based on file type. This is currently not feasible\nthrough the command line or using git configs. However, we can leverage the\nfact that gitattributes are path aware.\n\nTeach the diff machinery to check gitattributes when diffing files.\n\nJohn Cai (2):\n diff: consolidate diff algorithm option parsing\n diff: teach diff to read gitattribute diff-algorithm\n\n Documentation/gitattributes.txt | 23 +++++++\n diff.c | 112 ++++++++++++++++++++++----------\n diff.h | 2 +\n t/lib-diff-alternative.sh | 27 +++++++-\n 4 files changed, 130 insertions(+), 34 deletions(-)\n\n\nbase-commit: a6a323b31e2bcbac2518bddec71ea7ad558870eb\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1452%2Fjohn-cai%2Fjc%2Fattr-diff-algo-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1452/john-cai/jc/attr-diff-algo-v1\nPull-Request: https://github.com/git/git/pull/1452\n-- \ngitgitgadget\n"},{"id":"471512","messageId":"f56bd38ac3f80fb3a7e8c92cadaa57d2b0754b9f.1675568781.git.gitgitgadget@gmail.com","threadId":"59197","inReplyTo":"pull.1452.git.git.1675568781.gitgitgadget@gmail.com","subject":"[PATCH 1/2] diff: consolidate diff algorithm option parsing","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-05T03:46:20Z","receivedAt":"2023-02-05T03:46:30Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nThe diff option parsing for --minimal, --patience, --histgoram can all\nbe consolidated into one function. This is a preparatory step for the\nsubsequent commit which teaches diff to keep track of whether or not a\ndiff algorithm has been set via the command line.\n\nWhile we're at it, the logic that sets the diff algorithm in\ndiff_opt_diff_algorithm() can be refactored into a helper that will\nallow multiple callsites to set the diff algorithm.\n\nSigned-off-by: John Cai \n---\n diff.c | 87 ++++++++++++++++++++++++++++++++++++----------------------\n 1 file changed, 54 insertions(+), 33 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 329eebf16a0..a8a31c81fe7 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -3437,6 +3437,22 @@ static int diff_filepair_is_phoney(struct diff_filespec *one,\n \treturn !DIFF_FILE_VALID(one) && !DIFF_FILE_VALID(two);\n }\n \n+static int set_diff_algorithm(struct diff_options *opts,\n+\t\t\t const char *alg)\n+{\n+\tlong value = parse_algorithm_value(alg);\n+\n+\tif (value < 0)\n+\t\treturn 1;\n+\n+\t/* clear out previous settings */\n+\tDIFF_XDL_CLR(opts, NEED_MINIMAL);\n+\topts->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n+\topts->xdl_opts |= value;\n+\n+\treturn 0;\n+}\n+\n static void builtin_diff(const char *name_a,\n \t\t\t const char *name_b,\n \t\t\t struct diff_filespec *one,\n@@ -5107,17 +5123,40 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct diff_options *options = opt->value;\n-\tlong value = parse_algorithm_value(arg);\n \n \tBUG_ON_OPT_NEG(unset);\n-\tif (value < 0)\n+\n+\tif (set_diff_algorithm(options, arg))\n \t\treturn error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n \n-\t/* clear out previous settings */\n-\tDIFF_XDL_CLR(options, NEED_MINIMAL);\n-\toptions->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n-\toptions->xdl_opts |= value;\n+\treturn 0;\n+}\n+\n+static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n+\t\t\t\t const char *arg, int unset)\n+{\n+\tstruct diff_options *options = opt->value;\n+\n+\tBUG_ON_OPT_NEG(unset);\n+\tBUG_ON_OPT_ARG(arg);\n+\n+\tif (!strcmp(opt->long_name, \"patience\")) {\n+\t\tint i;\n+\t\t/*\n+\t\t * Both --patience and --anchored use PATIENCE_DIFF\n+\t\t * internally, so remove any anchors previously\n+\t\t * specified.\n+\t\t */\n+\t\tfor (i = 0; i < options->anchors_nr; i++)\n+\t\t\tfree(options->anchors[i]);\n+\t\toptions->anchors_nr = 0;\n+\t}\n+\n+\tif (set_diff_algorithm(options, opt->long_name))\n+\t\treturn error(_(\"available diff algorithms include \\\"myers\\\", \"\n+\t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n+\n \treturn 0;\n }\n \n@@ -5242,26 +5281,6 @@ static enum parse_opt_result diff_opt_output(struct parse_opt_ctx_t *ctx,\n \treturn 0;\n }\n \n-static int diff_opt_patience(const struct option *opt,\n-\t\t\t const char *arg, int unset)\n-{\n-\tstruct diff_options *options = opt->value;\n-\tint i;\n-\n-\tBUG_ON_OPT_NEG(unset);\n-\tBUG_ON_OPT_ARG(arg);\n-\toptions->xdl_opts = DIFF_WITH_ALG(options, PATIENCE_DIFF);\n-\t/*\n-\t * Both --patience and --anchored use PATIENCE_DIFF\n-\t * internally, so remove any anchors previously\n-\t * specified.\n-\t */\n-\tfor (i = 0; i < options->anchors_nr; i++)\n-\t\tfree(options->anchors[i]);\n-\toptions->anchors_nr = 0;\n-\treturn 0;\n-}\n-\n static int diff_opt_ignore_regex(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n@@ -5562,9 +5581,10 @@ struct option *add_diff_options(const struct option *opts,\n \t\t\t N_(\"prevent rename/copy detection if the number of rename/copy targets exceeds given limit\")),\n \n \t\tOPT_GROUP(N_(\"Diff algorithm options\")),\n-\t\tOPT_BIT(0, \"minimal\", &options->xdl_opts,\n-\t\t\tN_(\"produce the smallest possible diff\"),\n-\t\t\tXDF_NEED_MINIMAL),\n+\t\tOPT_CALLBACK_F(0, \"minimal\", options, NULL,\n+\t\t\t N_(\"produce the smallest possible diff\"),\n+\t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n+\t\t\t diff_opt_diff_algorithm_no_arg),\n \t\tOPT_BIT_F('w', \"ignore-all-space\", &options->xdl_opts,\n \t\t\t N_(\"ignore whitespace when comparing lines\"),\n \t\t\t XDF_IGNORE_WHITESPACE, PARSE_OPT_NONEG),\n@@ -5589,10 +5609,11 @@ struct option *add_diff_options(const struct option *opts,\n \t\tOPT_CALLBACK_F(0, \"patience\", options, NULL,\n \t\t\t N_(\"generate diff using the \\\"patience diff\\\" algorithm\"),\n \t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n-\t\t\t diff_opt_patience),\n-\t\tOPT_BITOP(0, \"histogram\", &options->xdl_opts,\n-\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n-\t\t\t XDF_HISTOGRAM_DIFF, XDF_DIFF_ALGORITHM_MASK),\n+\t\t\t diff_opt_diff_algorithm_no_arg),\n+\t\tOPT_CALLBACK_F(0, \"histogram\", options, NULL,\n+\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n+\t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n+\t\t\t diff_opt_diff_algorithm_no_arg),\n \t\tOPT_CALLBACK_F(0, \"diff-algorithm\", options, N_(\"\"),\n \t\t\t N_(\"choose a diff algorithm\"),\n \t\t\t PARSE_OPT_NONEG, diff_opt_diff_algorithm),\n-- \ngitgitgadget\n\n"},{"id":"471513","messageId":"8e73793b0db3e84366a9c6441cc0fdc04f9614a5.1675568781.git.gitgitgadget@gmail.com","threadId":"59197","inReplyTo":"pull.1452.git.git.1675568781.gitgitgadget@gmail.com","subject":"[PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-05T03:46:21Z","receivedAt":"2023-02-05T03:46:30Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nIt can be useful to specify diff algorithms per file type. For example,\none may want to use the minimal diff algorithm for .json files, another\nfor .c files, etc.\n\nTeach the diff machinery to check attributes for a diff algorithm.\nEnforce precedence by favoring the command line option, then looking at\nattributes, then finally the config.\n\nTo enforce precedence order, set the `xdl_opts_command_line` member\nduring options pasing to indicate the diff algorithm was set via command\nline args.\n\nSigned-off-by: John Cai \n---\n Documentation/gitattributes.txt | 23 +++++++++++++++++++++++\n diff.c | 25 +++++++++++++++++++++++++\n diff.h | 2 ++\n t/lib-diff-alternative.sh | 27 ++++++++++++++++++++++++++-\n 4 files changed, 76 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex c19e64ea0ef..501dd536037 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -736,6 +736,29 @@ String::\n \tby the configuration variables in the \"diff.foo\" section of the\n \tGit config file.\n \n+`diff-algorithm`\n+^^^^^^^^^^^^^^^^\n+\n+The attribute `diff-algorithm` affects which algorithm Git uses to generate\n+diffs. This allows defining diff algorithms per file extension. Precedence rules\n+are as follows, in order from highest to lowest:\n+\n+*Command line option*\n+\n+Pass in the `--diff-algorithm` command line option int git-diff(1)\n+\n+*Git attributes*\n+\n+------------------------\n+*.json\tdiff-algorithm=histogram\n+------------------------\n+\n+*Git config*\n+\n+----------------------------------------------------------------\n+[diff]\n+\talgorithm = histogram\n+----------------------------------------------------------------\n \n Defining an external diff driver\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\ndiff --git a/diff.c b/diff.c\nindex a8a31c81fe7..c78e28daeb0 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -3652,6 +3652,27 @@ static void builtin_diff(const char *name_a,\n \t\tecbdata.opt = o;\n \t\tif (header.len && !o->flags.suppress_diff_headers)\n \t\t\tecbdata.header = &header;\n+\n+\t\tif (!o->xdl_opts_command_line) {\n+\t\t\tstatic struct attr_check *check;\n+\t\t\tconst char *one_diff_algo;\n+\t\t\tconst char *two_diff_algo;\n+\n+\t\t\tcheck = attr_check_alloc();\n+\t\t\tattr_check_append(check, git_attr(\"diff-algorithm\"));\n+\n+\t\t\tgit_check_attr(the_repository->index, NULL, one->path, check);\n+\t\t\tone_diff_algo = check->items[0].value;\n+\t\t\tgit_check_attr(the_repository->index, NULL, two->path, check);\n+\t\t\ttwo_diff_algo = check->items[0].value;\n+\n+\t\t\tif (!ATTR_UNSET(one_diff_algo) && !ATTR_UNSET(two_diff_algo) &&\n+\t\t\t\t!strcmp(one_diff_algo, two_diff_algo))\n+\t\t\t\tset_diff_algorithm(o, one_diff_algo);\n+\n+\t\t\tattr_check_free(check);\n+\t\t}\n+\n \t\txpp.flags = o->xdl_opts;\n \t\txpp.ignore_regex = o->ignore_regex;\n \t\txpp.ignore_regex_nr = o->ignore_regex_nr;\n@@ -5130,6 +5151,8 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n \t\treturn error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n \n+\toptions->xdl_opts_command_line = 1;\n+\n \treturn 0;\n }\n \n@@ -5157,6 +5180,8 @@ static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n \t\treturn error(_(\"available diff algorithms include \\\"myers\\\", \"\n \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n \n+\toptions->xdl_opts_command_line = 1;\n+\n \treturn 0;\n }\n \ndiff --git a/diff.h b/diff.h\nindex 41eb2c3d428..46b565abfd4 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -333,6 +333,8 @@ struct diff_options {\n \tint prefix_length;\n \tconst char *stat_sep;\n \tint xdl_opts;\n+\t/* If xdl_opts has been set via the command line. */\n+\tint xdl_opts_command_line;\n \n \t/* see Documentation/diff-options.txt */\n \tchar **anchors;\ndiff --git a/t/lib-diff-alternative.sh b/t/lib-diff-alternative.sh\nindex 8d1e408bb58..630c98ea65a 100644\n--- a/t/lib-diff-alternative.sh\n+++ b/t/lib-diff-alternative.sh\n@@ -107,8 +107,27 @@ EOF\n \n \tSTRATEGY=$1\n \n+\ttest_expect_success \"$STRATEGY diff from attributes\" '\n+\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n+\t\ttest_must_fail git diff --no-index file1 file2 > output &&\n+\t\ttest_cmp expect output\n+\t'\n+\n \ttest_expect_success \"$STRATEGY diff\" '\n-\t\ttest_must_fail git diff --no-index \"--$STRATEGY\" file1 file2 > output &&\n+\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n+\t\ttest_cmp expect output\n+\t'\n+\n+\ttest_expect_success \"$STRATEGY diff command line precedence before attributes\" '\n+\t\techo \"file* diff-algorithm=meyers\" >.gitattributes &&\n+\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n+\t\ttest_cmp expect output\n+\t'\n+\n+\ttest_expect_success \"$STRATEGY diff attributes precedence before config\" '\n+\t\tgit config diff.algorithm default &&\n+\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n+\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n \t\ttest_cmp expect output\n \t'\n \n@@ -166,5 +185,11 @@ EOF\n \t\ttest_must_fail git diff --no-index \"--$STRATEGY\" uniq1 uniq2 > output &&\n \t\ttest_cmp expect output\n \t'\n+\n+\ttest_expect_success \"$STRATEGY diff from attributes\" '\n+\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n+\t\ttest_must_fail git diff --no-index uniq1 uniq2 > output &&\n+\t\ttest_cmp expect output\n+\t'\n }\n \n-- \ngitgitgadget\n"},{"id":"471532","messageId":"CAPig+cQNoX1-DN1=aFEX=MxEvoYhdu0NxPyhH++B4_1G2Dmp8w@mail.gmail.com","threadId":"59197","inReplyTo":"8e73793b0db3e84366a9c6441cc0fdc04f9614a5.1675568781.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2023-02-05T17:50:57Z","receivedAt":"2023-02-05T17:51:14Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sat, Feb 4, 2023 at 11:47 PM John Cai via GitGitGadget\n wrote:\n> It can be useful to specify diff algorithms per file type. For example,\n> one may want to use the minimal diff algorithm for .json files, another\n> for .c files, etc.\n>\n> Teach the diff machinery to check attributes for a diff algorithm.\n> Enforce precedence by favoring the command line option, then looking at\n> attributes, then finally the config.\n>\n> To enforce precedence order, set the `xdl_opts_command_line` member\n> during options pasing to indicate the diff algorithm was set via command\n> line args.\n>\n> Signed-off-by: John Cai \n> ---\n> diff --git a/diff.c b/diff.c\n> @@ -3652,6 +3652,27 @@ static void builtin_diff(const char *name_a,\n> + if (!o->xdl_opts_command_line) {\n> + static struct attr_check *check;\n\n`check` is declared static...\n\n> + const char *one_diff_algo;\n> + const char *two_diff_algo;\n> +\n> + check = attr_check_alloc();\n\n... is allocated here...\n\n> + attr_check_append(check, git_attr(\"diff-algorithm\"));\n> +\n> + git_check_attr(the_repository->index, NULL, one->path, check);\n> + one_diff_algo = check->items[0].value;\n> + git_check_attr(the_repository->index, NULL, two->path, check);\n> + two_diff_algo = check->items[0].value;\n> +\n> + if (!ATTR_UNSET(one_diff_algo) && !ATTR_UNSET(two_diff_algo) &&\n> + !strcmp(one_diff_algo, two_diff_algo))\n> + set_diff_algorithm(o, one_diff_algo);\n> +\n> + attr_check_free(check);\n\n... and freed here...\n\n> + }\n\n... so the reason for the `static` declaration is not clear. Am I\nmissing something obvious?\n"},{"id":"471557","messageId":"52A25839-3564-45D8-AF43-866ADDED983B@gmail.com","threadId":"59197","inReplyTo":"CAPig+cQNoX1-DN1=aFEX=MxEvoYhdu0NxPyhH++B4_1G2Dmp8w@mail.gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-06T13:10:47Z","receivedAt":"2023-02-06T13:10:52Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Eric,\n\nOn 5 Feb 2023, at 12:50, Eric Sunshine wrote:\n\n> On Sat, Feb 4, 2023 at 11:47 PM John Cai via GitGitGadget\n> wrote:\n>> It can be useful to specify diff algorithms per file type. For example,\n>> one may want to use the minimal diff algorithm for .json files, another\n>> for .c files, etc.\n>>\n>> Teach the diff machinery to check attributes for a diff algorithm.\n>> Enforce precedence by favoring the command line option, then looking at\n>> attributes, then finally the config.\n>>\n>> To enforce precedence order, set the `xdl_opts_command_line` member\n>> during options pasing to indicate the diff algorithm was set via command\n>> line args.\n>>\n>> Signed-off-by: John Cai \n>> ---\n>> diff --git a/diff.c b/diff.c\n>> @@ -3652,6 +3652,27 @@ static void builtin_diff(const char *name_a,\n>> + if (!o->xdl_opts_command_line) {\n>> + static struct attr_check *check;\n>\n> `check` is declared static...\n>\n>> + const char *one_diff_algo;\n>> + const char *two_diff_algo;\n>> +\n>> + check = attr_check_alloc();\n>\n> ... is allocated here...\n>\n>> + attr_check_append(check, git_attr(\"diff-algorithm\"));\n>> +\n>> + git_check_attr(the_repository->index, NULL, one->path, check);\n>> + one_diff_algo = check->items[0].value;\n>> + git_check_attr(the_repository->index, NULL, two->path, check);\n>> + two_diff_algo = check->items[0].value;\n>> +\n>> + if (!ATTR_UNSET(one_diff_algo) && !ATTR_UNSET(two_diff_algo) &&\n>> + !strcmp(one_diff_algo, two_diff_algo))\n>> + set_diff_algorithm(o, one_diff_algo);\n>> +\n>> + attr_check_free(check);\n>\n> ... and freed here...\n>\n>> + }\n>\n> ... so the reason for the `static` declaration is not clear. Am I\n> missing something obvious?\n\nNo, you are correct. No reason for the static declaration. `check` is not used outside of the scope of this\nconditional. I think this made it in from an earlier iteration and I didn't catch the oversight.\n\nthanks\nJohn\n\n\n"},{"id":"471560","messageId":"19a1c71c-d913-8118-a228-da72b5771421@dunelm.org.uk","threadId":"59197","inReplyTo":"f56bd38ac3f80fb3a7e8c92cadaa57d2b0754b9f.1675568781.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/2] diff: consolidate diff algorithm option parsing","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-02-06T16:20:28Z","receivedAt":"2023-02-06T16:20:58Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi John\n\nOn 05/02/2023 03:46, John Cai via GitGitGadget wrote:\n> From: John Cai \n> \n> The diff option parsing for --minimal, --patience, --histgoram can all\n> be consolidated into one function. This is a preparatory step for the\n> subsequent commit which teaches diff to keep track of whether or not a\n> diff algorithm has been set via the command line.\n> \n> While we're at it, the logic that sets the diff algorithm in\n> diff_opt_diff_algorithm() can be refactored into a helper that will\n> allow multiple callsites to set the diff algorithm.\n\nYou say \"while we're at it\" but isn't it a wholly necessary change for \nwhat you want to do?\n\nThis patch basically looks good, I've left a couple of comments below, \nthanks for separating it out as a preparatory step\n\n> Signed-off-by: John Cai \n> ---\n> diff.c | 87 ++++++++++++++++++++++++++++++++++++----------------------\n> 1 file changed, 54 insertions(+), 33 deletions(-)\n> \n> diff --git a/diff.c b/diff.c\n> index 329eebf16a0..a8a31c81fe7 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -3437,6 +3437,22 @@ static int diff_filepair_is_phoney(struct diff_filespec *one,\n> \treturn !DIFF_FILE_VALID(one) && !DIFF_FILE_VALID(two);\n> }\n> \n> +static int set_diff_algorithm(struct diff_options *opts,\n> +\t\t\t const char *alg)\n> +{\n> +\tlong value = parse_algorithm_value(alg);\n> +\n> +\tif (value < 0)\n> +\t\treturn 1;\n> +\n> +\t/* clear out previous settings */\n> +\tDIFF_XDL_CLR(opts, NEED_MINIMAL);\n> +\topts->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n> +\topts->xdl_opts |= value;\n> +\n> +\treturn 0;\n> +}\n> +\n> static void builtin_diff(const char *name_a,\n> \t\t\t const char *name_b,\n> \t\t\t struct diff_filespec *one,\n> @@ -5107,17 +5123,40 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n> \t\t\t\t const char *arg, int unset)\n> {\n> \tstruct diff_options *options = opt->value;\n> -\tlong value = parse_algorithm_value(arg);\n> \n> \tBUG_ON_OPT_NEG(unset);\n> -\tif (value < 0)\n> +\n> +\tif (set_diff_algorithm(options, arg))\n> \t\treturn error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n> \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n> \n> -\t/* clear out previous settings */\n> -\tDIFF_XDL_CLR(options, NEED_MINIMAL);\n> -\toptions->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n> -\toptions->xdl_opts |= value;\n> +\treturn 0;\n> +}\n> +\n> +static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n> +\t\t\t\t const char *arg, int unset)\n> +{\n> +\tstruct diff_options *options = opt->value;\n> +\n> +\tBUG_ON_OPT_NEG(unset);\n> +\tBUG_ON_OPT_ARG(arg);\n> +\n> +\tif (!strcmp(opt->long_name, \"patience\")) {\n> +\t\tint i;\n\nThis is copied from the existing code but as `options->anchors_nr` is a \nsize_t it is probably worth converting `i` to a size_t here.\n\n> +\t\t/*\n> +\t\t * Both --patience and --anchored use PATIENCE_DIFF\n> +\t\t * internally, so remove any anchors previously\n> +\t\t * specified.\n> +\t\t */\n> +\t\tfor (i = 0; i < options->anchors_nr; i++)\n> +\t\t\tfree(options->anchors[i]);\n> +\t\toptions->anchors_nr = 0;\n> +\t}\n> +\n> +\tif (set_diff_algorithm(options, opt->long_name))\n> +\t\treturn error(_(\"available diff algorithms include \\\"myers\\\", \"\n> +\t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n\nI think this should be a BUG() as it is a programming error if we reach \nthis point.\n\nBest Wishes\n\nPhillip\n\n> +\n> \treturn 0;\n> }\n> \n> @@ -5242,26 +5281,6 @@ static enum parse_opt_result diff_opt_output(struct parse_opt_ctx_t *ctx,\n> \treturn 0;\n> }\n> \n> -static int diff_opt_patience(const struct option *opt,\n> -\t\t\t const char *arg, int unset)\n> -{\n> -\tstruct diff_options *options = opt->value;\n> -\tint i;\n> -\n> -\tBUG_ON_OPT_NEG(unset);\n> -\tBUG_ON_OPT_ARG(arg);\n> -\toptions->xdl_opts = DIFF_WITH_ALG(options, PATIENCE_DIFF);\n> -\t/*\n> -\t * Both --patience and --anchored use PATIENCE_DIFF\n> -\t * internally, so remove any anchors previously\n> -\t * specified.\n> -\t */\n> -\tfor (i = 0; i < options->anchors_nr; i++)\n> -\t\tfree(options->anchors[i]);\n> -\toptions->anchors_nr = 0;\n> -\treturn 0;\n> -}\n> -\n> static int diff_opt_ignore_regex(const struct option *opt,\n> \t\t\t\t const char *arg, int unset)\n> {\n> @@ -5562,9 +5581,10 @@ struct option *add_diff_options(const struct option *opts,\n> \t\t\t N_(\"prevent rename/copy detection if the number of rename/copy targets exceeds given limit\")),\n> \n> \t\tOPT_GROUP(N_(\"Diff algorithm options\")),\n> -\t\tOPT_BIT(0, \"minimal\", &options->xdl_opts,\n> -\t\t\tN_(\"produce the smallest possible diff\"),\n> -\t\t\tXDF_NEED_MINIMAL),\n> +\t\tOPT_CALLBACK_F(0, \"minimal\", options, NULL,\n> +\t\t\t N_(\"produce the smallest possible diff\"),\n> +\t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n> +\t\t\t diff_opt_diff_algorithm_no_arg),\n> \t\tOPT_BIT_F('w', \"ignore-all-space\", &options->xdl_opts,\n> \t\t\t N_(\"ignore whitespace when comparing lines\"),\n> \t\t\t XDF_IGNORE_WHITESPACE, PARSE_OPT_NONEG),\n> @@ -5589,10 +5609,11 @@ struct option *add_diff_options(const struct option *opts,\n> \t\tOPT_CALLBACK_F(0, \"patience\", options, NULL,\n> \t\t\t N_(\"generate diff using the \\\"patience diff\\\" algorithm\"),\n> \t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n> -\t\t\t diff_opt_patience),\n> -\t\tOPT_BITOP(0, \"histogram\", &options->xdl_opts,\n> -\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n> -\t\t\t XDF_HISTOGRAM_DIFF, XDF_DIFF_ALGORITHM_MASK),\n> +\t\t\t diff_opt_diff_algorithm_no_arg),\n> +\t\tOPT_CALLBACK_F(0, \"histogram\", options, NULL,\n> +\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n> +\t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n> +\t\t\t diff_opt_diff_algorithm_no_arg),\n> \t\tOPT_CALLBACK_F(0, \"diff-algorithm\", options, N_(\"\"),\n> \t\t\t N_(\"choose a diff algorithm\"),\n> \t\t\t PARSE_OPT_NONEG, diff_opt_diff_algorithm),\n"},{"id":"471562","messageId":"19f6b40b-d8e8-e86c-5a9a-b35f42909118@dunelm.org.uk","threadId":"59197","inReplyTo":"8e73793b0db3e84366a9c6441cc0fdc04f9614a5.1675568781.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-02-06T16:27:53Z","receivedAt":"2023-02-06T16:27:59Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi John\n\nOn 05/02/2023 03:46, John Cai via GitGitGadget wrote:\n> From: John Cai \n> \n> It can be useful to specify diff algorithms per file type. For example,\n> one may want to use the minimal diff algorithm for .json files, another\n> for .c files, etc.\n\nHave you got any examples of why this is useful? I find myself \noccasionally changing the algorithm when the default gives a sub-optimal \ndiff but I've not really noticed any pattern with respect to file types.\n\n> Teach the diff machinery to check attributes for a diff algorithm.\n> Enforce precedence by favoring the command line option, then looking at\n> attributes, then finally the config.\n> \n> To enforce precedence order, set the `xdl_opts_command_line` member\n> during options pasing to indicate the diff algorithm was set via command\n> line args.\n\nI've only commented on the tests as it looks like Eric and had a careful \nlook at the code\n\n> diff --git a/t/lib-diff-alternative.sh b/t/lib-diff-alternative.sh\n> index 8d1e408bb58..630c98ea65a 100644\n> --- a/t/lib-diff-alternative.sh\n> +++ b/t/lib-diff-alternative.sh\n> @@ -107,8 +107,27 @@ EOF\n> \n> \tSTRATEGY=$1\n> \n> +\ttest_expect_success \"$STRATEGY diff from attributes\" '\n> +\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n> +\t\ttest_must_fail git diff --no-index file1 file2 > output &&\n> +\t\ttest_cmp expect output\n> +\t'\n> +\n> \ttest_expect_success \"$STRATEGY diff\" '\n> -\t\ttest_must_fail git diff --no-index \"--$STRATEGY\" file1 file2 > output &&\n> +\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n> +\t\ttest_cmp expect output\n> +\t'\n> +\n> +\ttest_expect_success \"$STRATEGY diff command line precedence before attributes\" '\n> +\t\techo \"file* diff-algorithm=meyers\" >.gitattributes &&\n> +\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n> +\t\ttest_cmp expect output\n> +\t'\n> +\n> +\ttest_expect_success \"$STRATEGY diff attributes precedence before config\" '\n> +\t\tgit config diff.algorithm default &&\n> +\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n> +\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n\nThis test passes with or without the code changes in this patch. I think \nyou need to drop --diff-algorithm=$STRATEGY from the diff command.\n\n> \t\ttest_cmp expect output\n> \t'\n> \n> @@ -166,5 +185,11 @@ EOF\n> \t\ttest_must_fail git diff --no-index \"--$STRATEGY\" uniq1 uniq2 > output &&\n> \t\ttest_cmp expect output\n> \t'\n> +\n> +\ttest_expect_success \"$STRATEGY diff from attributes\" '\n> +\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n> +\t\ttest_must_fail git diff --no-index uniq1 uniq2 > output &&\n> +\t\ttest_cmp expect output\n\nThis test also passes with or without the code changes in this patch. It \n is the same as the first test added above but with files that give the \nsame diff irrespective of the algorithm chosen so I don't think it is \ndoing anything useful. Unless I've missed something it should be dropped.\n\nBest Wishes\n\nPhillip\n\n> +\t'\n> }\n> \n"},{"id":"471564","messageId":"230206.865yce7n1w.gmgdl@evledraar.gmail.com","threadId":"59197","inReplyTo":"8e73793b0db3e84366a9c6441cc0fdc04f9614a5.1675568781.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-06T16:39:30Z","receivedAt":"2023-02-06T16:53:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Feb 05 2023, John Cai via GitGitGadget wrote:\n\n> From: John Cai \n> [...]\n> +\n> +\t\tif (!o->xdl_opts_command_line) {\n> +\t\t\tstatic struct attr_check *check;\n> +\t\t\tconst char *one_diff_algo;\n> +\t\t\tconst char *two_diff_algo;\n> +\n> +\t\t\tcheck = attr_check_alloc();\n> +\t\t\tattr_check_append(check, git_attr(\"diff-algorithm\"));\n> +\n> +\t\t\tgit_check_attr(the_repository->index, NULL, one->path, check);\n> +\t\t\tone_diff_algo = check->items[0].value;\n> +\t\t\tgit_check_attr(the_repository->index, NULL, two->path, check);\n> +\t\t\ttwo_diff_algo = check->items[0].value;\n> +\n> +\t\t\tif (!ATTR_UNSET(one_diff_algo) && !ATTR_UNSET(two_diff_algo) &&\n> +\t\t\t\t!strcmp(one_diff_algo, two_diff_algo))\n> +\t\t\t\tset_diff_algorithm(o, one_diff_algo);\n> +\n> +\t\t\tattr_check_free(check);\n\nThis is a bit nitpicky, but I for one would find this much easier to\nread with some shorter variables, here just with \"a\" rather than\n\"one_diff_algo\", \"b\" instead of \"two_diff_algo\", and splitting\n\"the_repository->index\" into \"istate\" (untested):\n\t\n\t+\t\tif (!o->xdl_opts_command_line) {\n\t+\t\t\tstatic struct attr_check *check;\n\t+\t\t\tconst char *a;\n\t+\t\t\tconst char *b;\n\t+\t\t\tstruct index_state *istate = the_repository->index;\n\t+\n\t+\t\t\tcheck = attr_check_alloc();\n\t+\t\t\tattr_check_append(check, git_attr(\"diff-algorithm\"));\n\t+\n\t+\t\t\tgit_check_attr(istate, NULL, one->path, check);\n\t+\t\t\ta = check->items[0].value;\n\t+\t\t\tgit_check_attr(istate, NULL, two->path, check);\n\t+\t\t\tb = check->items[0].value;\n\t+\n\t+\t\t\tif (!ATTR_UNSET(a) && !ATTR_UNSET(b) && !strcmp(a, b))\n\t+\t\t\t\tset_diff_algorithm(o, a);\n\t+\n\t+\t\t\tattr_check_free(check);\n\t+\t\t}\n\nThat also nicely keeps the line length shorter.\n\n> @@ -333,6 +333,8 @@ struct diff_options {\n> \tint prefix_length;\n> \tconst char *stat_sep;\n> \tint xdl_opts;\n> +\t/* If xdl_opts has been set via the command line. */\n> +\tint xdl_opts_command_line;\n> \n> \t/* see Documentation/diff-options.txt */\n> \tchar **anchors;\n> diff --git a/t/lib-diff-alternative.sh b/t/lib-diff-alternative.sh\n> index 8d1e408bb58..630c98ea65a 100644\n> --- a/t/lib-diff-alternative.sh\n> +++ b/t/lib-diff-alternative.sh\n> @@ -107,8 +107,27 @@ EOF\n> \n> \tSTRATEGY=$1\n> \n> +\ttest_expect_success \"$STRATEGY diff from attributes\" '\n> +\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n> +\t\ttest_must_fail git diff --no-index file1 file2 > output &&\n> +\t\ttest_cmp expect output\n> +\t'\n> +\n> \ttest_expect_success \"$STRATEGY diff\" '\n> -\t\ttest_must_fail git diff --no-index \"--$STRATEGY\" file1 file2 > output &&\n> +\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n\nNit: The usual style is \">output\", not \"> output\".\n\n> +\t\ttest_cmp expect output\n> +\t'\n> +\n> +\ttest_expect_success \"$STRATEGY diff command line precedence before attributes\" '\n> +\t\techo \"file* diff-algorithm=meyers\" >.gitattributes &&\n> +\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n> +\t\ttest_cmp expect output\n> +\t'\n> +\n> +\ttest_expect_success \"$STRATEGY diff attributes precedence before config\" '\n> +\t\tgit config diff.algorithm default &&\n> +\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n> +\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n> \t\ttest_cmp expect output\n> \t'\n> \n> @@ -166,5 +185,11 @@ EOF\n> \t\ttest_must_fail git diff --no-index \"--$STRATEGY\" uniq1 uniq2 > output &&\n> \t\ttest_cmp expect output\n> \t'\n> +\n> +\ttest_expect_success \"$STRATEGY diff from attributes\" '\n> +\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n> +\t\ttest_must_fail git diff --no-index uniq1 uniq2 > output &&\n> +\t\ttest_cmp expect output\n> +\t'\n> }\n\nFor some non-nitpicking, I do worry about exposing this as a DoS vector,\ne.g. here's a diff between two distant points in git.git with the\nvarious algorithms:\n\n\t$ hyperfine -r 1 -L a patience,minimal,histogram,myers 'git diff --diff-algorithm={a} v2.0.0 v2.28.0'\n\tBenchmark 1: git diff --diff-algorithm=patience v2.0.0 v2.28.0\n\t Time (abs ≡): 42.121 s [User: 41.879 s, System: 0.144 s]\n\t\n\tBenchmark 2: git diff --diff-algorithm=minimal v2.0.0 v2.28.0\n\t Time (abs ≡): 35.634 s [User: 35.473 s, System: 0.160 s]\n\t\n\tBenchmark 3: git diff --diff-algorithm=histogram v2.0.0 v2.28.0\n\t Time (abs ≡): 46.912 s [User: 46.657 s, System: 0.228 s]\n\t\n\tBenchmark 4: git diff --diff-algorithm=myers v2.0.0 v2.28.0\n\t Time (abs ≡): 33.233 s [User: 33.072 s, System: 0.160 s]\n\t\n\tSummary\n\t 'git diff --diff-algorithm=myers v2.0.0 v2.28.0' ran\n\t 1.07 times faster than 'git diff --diff-algorithm=minimal v2.0.0 v2.28.0'\n\t 1.27 times faster than 'git diff --diff-algorithm=patience v2.0.0 v2.28.0'\n\t 1.41 times faster than 'git diff --diff-algorithm=histogram v2.0.0 v2.28.0'\n\nNow, all of those are very slow overall, but some much more than\nothers. I seem to recall that the non-default ones also had some\npathological cases.\n\nAnother thing to think about is that we've so far considered the diff\nalgorithm to be purely about presentation, with some notable exceptions\nsuch as \"patch-id\".\n\nI've advocated for us getting to the point of having an in-repo\n.gitconfig or .gitattributes before with a whitelist of settings like\ndiff.context for certain paths, or a diff.orderFile.\n\nBut those seem easy to promise future behavior for, v.s. an entire diff\nalgorithm (which we of course had before, but now we'd have it in\nrepository data).\n\nMaybe that's not a distinction worth worrying about, just putting that\nout there.\n\nI think if others are concerned about the above something that would\nneatly side-step those is to have it opt-in via the .git/config somehow,\nsimilar to e.g. how you can commit *.gpg content, put this in\n.gitattributes:\n\n\t*.gpg diff=gpg\n\nBut not have it do anything until this is in the repo's .git/config (or\nsimilar):\n\n\t[diff \"gpg\"]\n \ttextconv = gpg --no-tty --decrypt\n\nFor that you could still keep the exact .gitattributes format you have\nhere, i.e.:\n\n\tfile* diff-algorithm=$STRATEGY\n\nBut we to pick it up we'd need either:\n\n\t[diff-algorithm]\n \thistogram = myers\n\nOr:\n\n\t[diff-algorithm \"histogram\"]\n \tallow = true\n\nThe former form being one that would allow you to map the .gitattributes\nof the repo (but maybe that would be redundant to\n.git/info/attributes)...\n"},{"id":"471572","messageId":"CAPig+cTTfsP-hTSd=1QF-zZ9WNpHMgO-+18PTmnggz4SANfmMg@mail.gmail.com","threadId":"59197","inReplyTo":"19f6b40b-d8e8-e86c-5a9a-b35f42909118@dunelm.org.uk","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2023-02-06T18:14:50Z","receivedAt":"2023-02-06T18:15:06Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Feb 6, 2023 at 11:46 AM Phillip Wood wrote:\n> On 05/02/2023 03:46, John Cai via GitGitGadget wrote:\n> > Teach the diff machinery to check attributes for a diff algorithm.\n> > Enforce precedence by favoring the command line option, then looking at\n> > attributes, then finally the config.\n> >\n> > To enforce precedence order, set the `xdl_opts_command_line` member\n> > during options pasing to indicate the diff algorithm was set via command\n> > line args.\n>\n> I've only commented on the tests as it looks like Eric and had a careful\n> look at the code\n\nI only very quickly ran my eye over the code; didn't even really read it.\n"},{"id":"471593","messageId":"7852AC7B-7A4E-4DD0-ADEA-CFFD5D16C595@gmail.com","threadId":"59197","inReplyTo":"19f6b40b-d8e8-e86c-5a9a-b35f42909118@dunelm.org.uk","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-06T19:50:42Z","receivedAt":"2023-02-06T19:50:47Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Phillip,\n\nOn 6 Feb 2023, at 11:27, Phillip Wood wrote:\n\n> Hi John\n>\n> On 05/02/2023 03:46, John Cai via GitGitGadget wrote:\n>> From: John Cai \n>>\n>> It can be useful to specify diff algorithms per file type. For example,\n>> one may want to use the minimal diff algorithm for .json files, another\n>> for .c files, etc.\n>\n> Have you got any examples of why this is useful? I find myself occasionally changing the algorithm when the default gives a sub-optimal diff but I've not really noticed any pattern with respect to file types.\n\nAt $DAYJOB, there has been a discussion and request for a feature like this [1].\nOne use case that came up was to be able to set a different diff algorithm for\n.json files.\n\n1. https://gitlab.com/gitlab-org/gitaly/-/issues/2591\n\n>\n>> Teach the diff machinery to check attributes for a diff algorithm.\n>> Enforce precedence by favoring the command line option, then looking at\n>> attributes, then finally the config.\n>>\n>> To enforce precedence order, set the `xdl_opts_command_line` member\n>> during options pasing to indicate the diff algorithm was set via command\n>> line args.\n>\n> I've only commented on the tests as it looks like Eric and had a careful look at the code\n>\n>> diff --git a/t/lib-diff-alternative.sh b/t/lib-diff-alternative.sh\n>> index 8d1e408bb58..630c98ea65a 100644\n>> --- a/t/lib-diff-alternative.sh\n>> +++ b/t/lib-diff-alternative.sh\n>> @@ -107,8 +107,27 @@ EOF\n>> \tSTRATEGY=$1\n>> +\ttest_expect_success \"$STRATEGY diff from attributes\" '\n>> +\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n>> +\t\ttest_must_fail git diff --no-index file1 file2 > output &&\n>> +\t\ttest_cmp expect output\n>> +\t'\n>> +\n>> \ttest_expect_success \"$STRATEGY diff\" '\n>> -\t\ttest_must_fail git diff --no-index \"--$STRATEGY\" file1 file2 > output &&\n>> +\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n>> +\t\ttest_cmp expect output\n>> +\t'\n>> +\n>> +\ttest_expect_success \"$STRATEGY diff command line precedence before attributes\" '\n>> +\t\techo \"file* diff-algorithm=meyers\" >.gitattributes &&\n>> +\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n>> +\t\ttest_cmp expect output\n>> +\t'\n>> +\n>> +\ttest_expect_success \"$STRATEGY diff attributes precedence before config\" '\n>> +\t\tgit config diff.algorithm default &&\n>> +\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n>> +\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n>\n> This test passes with or without the code changes in this patch. I think you need to drop --diff-algorithm=$STRATEGY from the diff command.\n\nan oversight indeed, thanks for catching this.\n\n>\n>> \t\ttest_cmp expect output\n>> \t'\n>> @@ -166,5 +185,11 @@ EOF\n>> \t\ttest_must_fail git diff --no-index \"--$STRATEGY\" uniq1 uniq2 > output &&\n>> \t\ttest_cmp expect output\n>> \t'\n>> +\n>> +\ttest_expect_success \"$STRATEGY diff from attributes\" '\n>> +\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n>> +\t\ttest_must_fail git diff --no-index uniq1 uniq2 > output &&\n>> +\t\ttest_cmp expect output\n>\n> This test also passes with or without the code changes in this patch. It is the same as the first test added above but with files that give the same diff irrespective of the algorithm chosen so I don't think it is doing anything useful. Unless I've missed something it should be dropped.\n\nI should have been more thorough with this one as well, thanks.\n\n>\n> Best Wishes\n>\n> Phillip\n>\n>> +\t'\n>> }\n>>\n"},{"id":"471597","messageId":"B544D9E8-13C4-4682-9BDA-D6E19B51C91D@gmail.com","threadId":"59197","inReplyTo":"230206.865yce7n1w.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-06T20:37:53Z","receivedAt":"2023-02-06T20:38:08Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Ævar,\n\nOn 6 Feb 2023, at 11:39, Ævar Arnfjörð Bjarmason wrote:\n\n> On Sun, Feb 05 2023, John Cai via GitGitGadget wrote:\n>\n>> From: John Cai \n>> [...]\n>> +\n>> +\t\tif (!o->xdl_opts_command_line) {\n>> +\t\t\tstatic struct attr_check *check;\n>> +\t\t\tconst char *one_diff_algo;\n>> +\t\t\tconst char *two_diff_algo;\n>> +\n>> +\t\t\tcheck = attr_check_alloc();\n>> +\t\t\tattr_check_append(check, git_attr(\"diff-algorithm\"));\n>> +\n>> +\t\t\tgit_check_attr(the_repository->index, NULL, one->path, check);\n>> +\t\t\tone_diff_algo = check->items[0].value;\n>> +\t\t\tgit_check_attr(the_repository->index, NULL, two->path, check);\n>> +\t\t\ttwo_diff_algo = check->items[0].value;\n>> +\n>> +\t\t\tif (!ATTR_UNSET(one_diff_algo) && !ATTR_UNSET(two_diff_algo) &&\n>> +\t\t\t\t!strcmp(one_diff_algo, two_diff_algo))\n>> +\t\t\t\tset_diff_algorithm(o, one_diff_algo);\n>> +\n>> +\t\t\tattr_check_free(check);\n>\n> This is a bit nitpicky, but I for one would find this much easier to\n> read with some shorter variables, here just with \"a\" rather than\n> \"one_diff_algo\", \"b\" instead of \"two_diff_algo\", and splitting\n> \"the_repository->index\" into \"istate\" (untested):\n> \t\n> \t+\t\tif (!o->xdl_opts_command_line) {\n> \t+\t\t\tstatic struct attr_check *check;\n> \t+\t\t\tconst char *a;\n> \t+\t\t\tconst char *b;\n> \t+\t\t\tstruct index_state *istate = the_repository->index;\n> \t+\n> \t+\t\t\tcheck = attr_check_alloc();\n> \t+\t\t\tattr_check_append(check, git_attr(\"diff-algorithm\"));\n> \t+\n> \t+\t\t\tgit_check_attr(istate, NULL, one->path, check);\n> \t+\t\t\ta = check->items[0].value;\n> \t+\t\t\tgit_check_attr(istate, NULL, two->path, check);\n> \t+\t\t\tb = check->items[0].value;\n> \t+\n> \t+\t\t\tif (!ATTR_UNSET(a) && !ATTR_UNSET(b) && !strcmp(a, b))\n> \t+\t\t\t\tset_diff_algorithm(o, a);\n> \t+\n> \t+\t\t\tattr_check_free(check);\n> \t+\t\t}\n>\n> That also nicely keeps the line length shorter.\n\nThanks, I think this does look better.\n\n>\n>> @@ -333,6 +333,8 @@ struct diff_options {\n>> \tint prefix_length;\n>> \tconst char *stat_sep;\n>> \tint xdl_opts;\n>> +\t/* If xdl_opts has been set via the command line. */\n>> +\tint xdl_opts_command_line;\n>>\n>> \t/* see Documentation/diff-options.txt */\n>> \tchar **anchors;\n>> diff --git a/t/lib-diff-alternative.sh b/t/lib-diff-alternative.sh\n>> index 8d1e408bb58..630c98ea65a 100644\n>> --- a/t/lib-diff-alternative.sh\n>> +++ b/t/lib-diff-alternative.sh\n>> @@ -107,8 +107,27 @@ EOF\n>>\n>> \tSTRATEGY=$1\n>>\n>> +\ttest_expect_success \"$STRATEGY diff from attributes\" '\n>> +\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n>> +\t\ttest_must_fail git diff --no-index file1 file2 > output &&\n>> +\t\ttest_cmp expect output\n>> +\t'\n>> +\n>> \ttest_expect_success \"$STRATEGY diff\" '\n>> -\t\ttest_must_fail git diff --no-index \"--$STRATEGY\" file1 file2 > output &&\n>> +\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n>\n> Nit: The usual style is \">output\", not \"> output\".\n\nnoted!\n\n>\n>> +\t\ttest_cmp expect output\n>> +\t'\n>> +\n>> +\ttest_expect_success \"$STRATEGY diff command line precedence before attributes\" '\n>> +\t\techo \"file* diff-algorithm=meyers\" >.gitattributes &&\n>> +\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n>> +\t\ttest_cmp expect output\n>> +\t'\n>> +\n>> +\ttest_expect_success \"$STRATEGY diff attributes precedence before config\" '\n>> +\t\tgit config diff.algorithm default &&\n>> +\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n>> +\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n>> \t\ttest_cmp expect output\n>> \t'\n>>\n>> @@ -166,5 +185,11 @@ EOF\n>> \t\ttest_must_fail git diff --no-index \"--$STRATEGY\" uniq1 uniq2 > output &&\n>> \t\ttest_cmp expect output\n>> \t'\n>> +\n>> +\ttest_expect_success \"$STRATEGY diff from attributes\" '\n>> +\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n>> +\t\ttest_must_fail git diff --no-index uniq1 uniq2 > output &&\n>> +\t\ttest_cmp expect output\n>> +\t'\n>> }\n>\n> For some non-nitpicking, I do worry about exposing this as a DoS vector,\n> e.g. here's a diff between two distant points in git.git with the\n> various algorithms:\n>\n> \t$ hyperfine -r 1 -L a patience,minimal,histogram,myers 'git diff --diff-algorithm={a} v2.0.0 v2.28.0'\n> \tBenchmark 1: git diff --diff-algorithm=patience v2.0.0 v2.28.0\n> \t Time (abs ≡): 42.121 s [User: 41.879 s, System: 0.144 s]\n> \t\n> \tBenchmark 2: git diff --diff-algorithm=minimal v2.0.0 v2.28.0\n> \t Time (abs ≡): 35.634 s [User: 35.473 s, System: 0.160 s]\n> \t\n> \tBenchmark 3: git diff --diff-algorithm=histogram v2.0.0 v2.28.0\n> \t Time (abs ≡): 46.912 s [User: 46.657 s, System: 0.228 s]\n> \t\n> \tBenchmark 4: git diff --diff-algorithm=myers v2.0.0 v2.28.0\n> \t Time (abs ≡): 33.233 s [User: 33.072 s, System: 0.160 s]\n> \t\n> \tSummary\n> \t 'git diff --diff-algorithm=myers v2.0.0 v2.28.0' ran\n> \t 1.07 times faster than 'git diff --diff-algorithm=minimal v2.0.0 v2.28.0'\n> \t 1.27 times faster than 'git diff --diff-algorithm=patience v2.0.0 v2.28.0'\n> \t 1.41 times faster than 'git diff --diff-algorithm=histogram v2.0.0 v2.28.0'\n\nThanks for this analysis. To clarify, .gitconfig's diff.algorithm setting is\nalready an attack vector right? I see how this would be adding another one.\n\nThat being said, here's a separate issue. I benchmarked the usage of\n.gitattributes as introduced in this patch series, and indeed it does look like\nthere is additional latency:\n\n$ echo \"* diff-algorithm=patience >> .gitattributes\n$ hyperfine -r 5 'git-bin-wrapper diff --diff-algorithm=patience v2.0.0 v2.28.0' ✭\nBenchmark 1: git-bin-wrapper diff --diff-algorithm=patience v2.0.0 v2.28.0\n Time (mean ± σ): 889.4 ms ± 113.8 ms [User: 715.7 ms, System: 65.3 ms]\n Range (min … max): 764.1 ms … 1029.3 ms 5 runs\n\n$ hyperfine -r 5 'git-bin-wrapper diff v2.0.0 v2.28.0' ✭\nBenchmark 1: git-bin-wrapper diff v2.0.0 v2.28.0\n Time (mean ± σ): 2.146 s ± 0.368 s [User: 0.827 s, System: 0.243 s]\n Range (min … max): 1.883 s … 2.795 s 5 runs\n\nand I imagine the latency scales with the size of .gitattributes. Although I'm\nnot familiar with other parts of the codebase and how it deals with the latency\nintroduced by reading attributes files.\n\n>\n> Now, all of those are very slow overall, but some much more than\n> others. I seem to recall that the non-default ones also had some\n> pathological cases.\n>\n> Another thing to think about is that we've so far considered the diff\n> algorithm to be purely about presentation, with some notable exceptions\n> such as \"patch-id\".\n>\n> I've advocated for us getting to the point of having an in-repo\n> .gitconfig or .gitattributes before with a whitelist of settings like\n> diff.context for certain paths, or a diff.orderFile.\n>\n> But those seem easy to promise future behavior for, v.s. an entire diff\n> algorithm (which we of course had before, but now we'd have it in\n> repository data).\n>\n> Maybe that's not a distinction worth worrying about, just putting that\n> out there.\n>\n> I think if others are concerned about the above something that would\n> neatly side-step those is to have it opt-in via the .git/config somehow,\n> similar to e.g. how you can commit *.gpg content, put this in\n> .gitattributes:\n>\n> \t*.gpg diff=gpg\n>\n> But not have it do anything until this is in the repo's .git/config (or\n> similar):\n>\n> \t[diff \"gpg\"]\n> \ttextconv = gpg --no-tty --decrypt\n>\n> For that you could still keep the exact .gitattributes format you have\n> here, i.e.:\n>\n> \tfile* diff-algorithm=$STRATEGY\n>\n> But we to pick it up we'd need either:\n>\n> \t[diff-algorithm]\n> \thistogram = myers\n>\n> Or:\n>\n> \t[diff-algorithm \"histogram\"]\n> \tallow = true\n\nThis would help address slowness from the diff algorithm itself. I'm not opposed\nto adding this config if this attack vector is concerning to people.\n\nHowever, it wouldn't help address the additional latency of scanning\n.gitattributes to find the diff algorithm.\n\nWould a separate config to allow gitattributes be helpful here?\n\n[diff-algorithm]\n\tattributes = true\n\n>\n> The former form being one that would allow you to map the .gitattributes\n> of the repo (but maybe that would be redundant to\n> .git/info/attributes)...\n"},{"id":"471683","messageId":"d18a5c32-2f15-93ad-ccbf-e8f048edb311@dunelm.org.uk","threadId":"59197","inReplyTo":"B544D9E8-13C4-4682-9BDA-D6E19B51C91D@gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-02-07T14:55:29Z","receivedAt":"2023-02-07T14:55:35Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi John\n\nOn 06/02/2023 20:37, John Cai wrote:\n> On 6 Feb 2023, at 11:39, Ævar Arnfjörð Bjarmason wrote:\n>> On Sun, Feb 05 2023, John Cai via GitGitGadget wrote:\n>> For some non-nitpicking, I do worry about exposing this as a DoS vector,\n>> e.g. here's a diff between two distant points in git.git with the\n>> various algorithms:\n>>\n>> \t$ hyperfine -r 1 -L a patience,minimal,histogram,myers 'git diff --diff-algorithm={a} v2.0.0 v2.28.0'\n>> \tBenchmark 1: git diff --diff-algorithm=patience v2.0.0 v2.28.0\n>> \t Time (abs ≡): 42.121 s [User: 41.879 s, System: 0.144 s]\n>> \t\n>> \tBenchmark 2: git diff --diff-algorithm=minimal v2.0.0 v2.28.0\n>> \t Time (abs ≡): 35.634 s [User: 35.473 s, System: 0.160 s]\n>> \t\n>> \tBenchmark 3: git diff --diff-algorithm=histogram v2.0.0 v2.28.0\n>> \t Time (abs ≡): 46.912 s [User: 46.657 s, System: 0.228 s]\n>> \t\n>> \tBenchmark 4: git diff --diff-algorithm=myers v2.0.0 v2.28.0\n>> \t Time (abs ≡): 33.233 s [User: 33.072 s, System: 0.160 s]\n>> \t\n>> \tSummary\n>> \t 'git diff --diff-algorithm=myers v2.0.0 v2.28.0' ran\n>> \t 1.07 times faster than 'git diff --diff-algorithm=minimal v2.0.0 v2.28.0'\n>> \t 1.27 times faster than 'git diff --diff-algorithm=patience v2.0.0 v2.28.0'\n>> \t 1.41 times faster than 'git diff --diff-algorithm=histogram v2.0.0 v2.28.0'\n> \n> Thanks for this analysis. To clarify, .gitconfig's diff.algorithm setting is\n> already an attack vector right?\n\n.gitconfig is under the user's control though whereas .gitattributes is \nattacker controlled if one clones a malicious repository. Having said \nthe worst results above are for the historgram algorithm that merge-ort \nuses internally and no one has complained about ort's performance.\n\n> I see how this would be adding another one.\n> \n> That being said, here's a separate issue. I benchmarked the usage of\n> .gitattributes as introduced in this patch series, and indeed it does look like\n> there is additional latency:\n> \n> $ echo \"* diff-algorithm=patience >> .gitattributes\n> $ hyperfine -r 5 'git-bin-wrapper diff --diff-algorithm=patience v2.0.0 v2.28.0' ✭\n> Benchmark 1: git-bin-wrapper diff --diff-algorithm=patience v2.0.0 v2.28.0\n> Time (mean ± σ): 889.4 ms ± 113.8 ms [User: 715.7 ms, System: 65.3 ms]\n> Range (min … max): 764.1 ms … 1029.3 ms 5 runs\n> \n> $ hyperfine -r 5 'git-bin-wrapper diff v2.0.0 v2.28.0' ✭\n> Benchmark 1: git-bin-wrapper diff v2.0.0 v2.28.0\n> Time (mean ± σ): 2.146 s ± 0.368 s [User: 0.827 s, System: 0.243 s]\n> Range (min … max): 1.883 s … 2.795 s 5 runs\n> \n> and I imagine the latency scales with the size of .gitattributes. Although I'm\n> not familiar with other parts of the codebase and how it deals with the latency\n> introduced by reading attributes files.\n\nOuch! Thanks for benchmarking that is a suspiciously large slow down. \nI've a feeling the attributes code parses all the .gitattribute files \nfrom scratch for each path that's queried, so there may be scope for \nmaking it a bit smarter. I see some slow down but no where near as much\n\n\n$ hyperfine -r 3 'bin-wrappers/git diff --diff-algorithm=patience \n--no-color --no-color-moved v2.0.0 v2.28.0'\nBenchmark 1: bin-wrappers/git diff --diff-algorithm=patience --no-color \n--no-color-moved v2.0.0 v2.28.0\n Time (mean ± σ): 1.996 s ± 0.008 s [User: 1.706 s, System: \n0.286 s]\n Range (min … max): 1.989 s … 2.004 s 3 runs\n\n$ hyperfine -r 5 'bin-wrappers/git diff --no-color --no-color-moved \nv2.0.0 v2.28.0'\nBenchmark 1: bin-wrappers/git diff --no-color --no-color-moved v2.0.0 \nv2.28.0\n Time (mean ± σ): 2.238 s ± 0.037 s [User: 1.880 s, System: \n0.350 s]\n Range (min … max): 2.216 s … 2.303 s 5 runs\n\n\nPerhaps I'm over simplifying but having read the issue you linked to I \ncouldn't help feeling that the majority of users might be satisfied by \njust changing gitlab to use the patience algorithm when generating \ndiffs. The idea to use .gitattributes seems to have come from Patrick \nrather than a user request.\n\nThis is slightly off topic but one thing I'd really like is a way to \ntell diff use automatically use --diff-words on some files (e.g. \nDocumentation/*)\n\nBest Wishes\n\nPhillip\n\n>> Now, all of those are very slow overall, but some much more than\n>> others. I seem to recall that the non-default ones also had some\n>> pathological cases.\n>>\n>> Another thing to think about is that we've so far considered the diff\n>> algorithm to be purely about presentation, with some notable exceptions\n>> such as \"patch-id\".\n>>\n>> I've advocated for us getting to the point of having an in-repo\n>> .gitconfig or .gitattributes before with a whitelist of settings like\n>> diff.context for certain paths, or a diff.orderFile.\n>>\n>> But those seem easy to promise future behavior for, v.s. an entire diff\n>> algorithm (which we of course had before, but now we'd have it in\n>> repository data).\n>>\n>> Maybe that's not a distinction worth worrying about, just putting that\n>> out there.\n>>\n>> I think if others are concerned about the above something that would\n>> neatly side-step those is to have it opt-in via the .git/config somehow,\n>> similar to e.g. how you can commit *.gpg content, put this in\n>> .gitattributes:\n>>\n>> \t*.gpg diff=gpg\n>>\n>> But not have it do anything until this is in the repo's .git/config (or\n>> similar):\n>>\n>> \t[diff \"gpg\"]\n>> \ttextconv = gpg --no-tty --decrypt\n>>\n>> For that you could still keep the exact .gitattributes format you have\n>> here, i.e.:\n>>\n>> \tfile* diff-algorithm=$STRATEGY\n>>\n>> But we to pick it up we'd need either:\n>>\n>> \t[diff-algorithm]\n>> \thistogram = myers\n>>\n>> Or:\n>>\n>> \t[diff-algorithm \"histogram\"]\n>> \tallow = true\n> \n> This would help address slowness from the diff algorithm itself. I'm not opposed\n> to adding this config if this attack vector is concerning to people.\n> \n> However, it wouldn't help address the additional latency of scanning\n> .gitattributes to find the diff algorithm.\n> \n> Would a separate config to allow gitattributes be helpful here?\n> \n> [diff-algorithm]\n> \tattributes = true\n> \n>>\n>> The former form being one that would allow you to map the .gitattributes\n>> of the repo (but maybe that would be redundant to\n>> .git/info/attributes)...\n"},{"id":"471697","messageId":"65129323-326F-4E4A-B6F8-06DC3BBE7B58@gmail.com","threadId":"59197","inReplyTo":"d18a5c32-2f15-93ad-ccbf-e8f048edb311@dunelm.org.uk","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-07T17:00:49Z","receivedAt":"2023-02-07T17:00:56Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Phillip,\n\nOn 7 Feb 2023, at 9:55, Phillip Wood wrote:\n\n> Hi John\n>\n> On 06/02/2023 20:37, John Cai wrote:\n>> On 6 Feb 2023, at 11:39, Ævar Arnfjörð Bjarmason wrote:\n>>> On Sun, Feb 05 2023, John Cai via GitGitGadget wrote:\n>>> For some non-nitpicking, I do worry about exposing this as a DoS vector,\n>>> e.g. here's a diff between two distant points in git.git with the\n>>> various algorithms:\n>>>\n>>> \t$ hyperfine -r 1 -L a patience,minimal,histogram,myers 'git diff --diff-algorithm={a} v2.0.0 v2.28.0'\n>>> \tBenchmark 1: git diff --diff-algorithm=patience v2.0.0 v2.28.0\n>>> \t Time (abs ≡): 42.121 s [User: 41.879 s, System: 0.144 s]\n>>> \t\n>>> \tBenchmark 2: git diff --diff-algorithm=minimal v2.0.0 v2.28.0\n>>> \t Time (abs ≡): 35.634 s [User: 35.473 s, System: 0.160 s]\n>>> \t\n>>> \tBenchmark 3: git diff --diff-algorithm=histogram v2.0.0 v2.28.0\n>>> \t Time (abs ≡): 46.912 s [User: 46.657 s, System: 0.228 s]\n>>> \t\n>>> \tBenchmark 4: git diff --diff-algorithm=myers v2.0.0 v2.28.0\n>>> \t Time (abs ≡): 33.233 s [User: 33.072 s, System: 0.160 s]\n>>> \t\n>>> \tSummary\n>>> \t 'git diff --diff-algorithm=myers v2.0.0 v2.28.0' ran\n>>> \t 1.07 times faster than 'git diff --diff-algorithm=minimal v2.0.0 v2.28.0'\n>>> \t 1.27 times faster than 'git diff --diff-algorithm=patience v2.0.0 v2.28.0'\n>>> \t 1.41 times faster than 'git diff --diff-algorithm=histogram v2.0.0 v2.28.0'\n>>\n>> Thanks for this analysis. To clarify, .gitconfig's diff.algorithm setting is\n>> already an attack vector right?\n>\n> .gitconfig is under the user's control though whereas .gitattributes is attacker controlled if one clones a malicious repository. Having said the worst results above are for the historgram algorithm that merge-ort uses internally and no one has complained about ort's performance.\n>\n>> I see how this would be adding another one.\n>>\n>> That being said, here's a separate issue. I benchmarked the usage of\n>> .gitattributes as introduced in this patch series, and indeed it does look like\n>> there is additional latency:\n>>\n>> $ echo \"* diff-algorithm=patience >> .gitattributes\n>> $ hyperfine -r 5 'git-bin-wrapper diff --diff-algorithm=patience v2.0.0 v2.28.0' ✭\n>> Benchmark 1: git-bin-wrapper diff --diff-algorithm=patience v2.0.0 v2.28.0\n>> Time (mean ± σ): 889.4 ms ± 113.8 ms [User: 715.7 ms, System: 65.3 ms]\n>> Range (min … max): 764.1 ms … 1029.3 ms 5 runs\n>>\n>> $ hyperfine -r 5 'git-bin-wrapper diff v2.0.0 v2.28.0' ✭\n>> Benchmark 1: git-bin-wrapper diff v2.0.0 v2.28.0\n>> Time (mean ± σ): 2.146 s ± 0.368 s [User: 0.827 s, System: 0.243 s]\n>> Range (min … max): 1.883 s … 2.795 s 5 runs\n>>\n>> and I imagine the latency scales with the size of .gitattributes. Although I'm\n>> not familiar with other parts of the codebase and how it deals with the latency\n>> introduced by reading attributes files.\n>\n> Ouch! Thanks for benchmarking that is a suspiciously large slow down. I've a feeling the attributes code parses all the .gitattribute files from scratch for each path that's queried, so there may be scope for making it a bit smarter. I see some slow down but no where near as much\n\nYes, definitely worth looking into how this can be sped up.\n\n>\n>\n> $ hyperfine -r 3 'bin-wrappers/git diff --diff-algorithm=patience --no-color --no-color-moved v2.0.0 v2.28.0'\n> Benchmark 1: bin-wrappers/git diff --diff-algorithm=patience --no-color --no-color-moved v2.0.0 v2.28.0\n> Time (mean ± σ): 1.996 s ± 0.008 s [User: 1.706 s, System: 0.286 s]\n> Range (min … max): 1.989 s … 2.004 s 3 runs\n>\n> $ hyperfine -r 5 'bin-wrappers/git diff --no-color --no-color-moved v2.0.0 v2.28.0'\n> Benchmark 1: bin-wrappers/git diff --no-color --no-color-moved v2.0.0 v2.28.0\n> Time (mean ± σ): 2.238 s ± 0.037 s [User: 1.880 s, System: 0.350 s]\n> Range (min … max): 2.216 s … 2.303 s 5 runs\n>\n>\n> Perhaps I'm over simplifying but having read the issue you linked to I couldn't help feeling that the majority of users might be satisfied by just changing gitlab to use the patience algorithm when generating diffs.\n\nRight, I recognize this is a judgment call that may be best left up to the list.\n\nWe don't have a way in GitLab to change the diff algorithm currently. Of course\nthat can be implemented outside of Git, but having it as part of Git may have a\nwider benefit for users who would appreciate the convenience of automatically\nhaving certain files use one diff algorithm and other files use another, without\nhaving to pass in the diff algorithm through the command line each time.\n\n>\n> The idea to use .gitattributes seems to have come from Patrick rather than a user request.\n>\n> This is slightly off topic but one thing I'd really like is a way to tell diff use automatically use --diff-words on some files (e.g. Documentation/*)\n\nI think it's a bit similar to the spirit of this desired feature.\n\nthanks\nJohn\n\n>\n> Best Wishes\n>\n> Phillip\n>\n>>> Now, all of those are very slow overall, but some much more than\n>>> others. I seem to recall that the non-default ones also had some\n>>> pathological cases.\n>>>\n>>> Another thing to think about is that we've so far considered the diff\n>>> algorithm to be purely about presentation, with some notable exceptions\n>>> such as \"patch-id\".\n>>>\n>>> I've advocated for us getting to the point of having an in-repo\n>>> .gitconfig or .gitattributes before with a whitelist of settings like\n>>> diff.context for certain paths, or a diff.orderFile.\n>>>\n>>> But those seem easy to promise future behavior for, v.s. an entire diff\n>>> algorithm (which we of course had before, but now we'd have it in\n>>> repository data).\n>>>\n>>> Maybe that's not a distinction worth worrying about, just putting that\n>>> out there.\n>>>\n>>> I think if others are concerned about the above something that would\n>>> neatly side-step those is to have it opt-in via the .git/config somehow,\n>>> similar to e.g. how you can commit *.gpg content, put this in\n>>> .gitattributes:\n>>>\n>>> \t*.gpg diff=gpg\n>>>\n>>> But not have it do anything until this is in the repo's .git/config (or\n>>> similar):\n>>>\n>>> \t[diff \"gpg\"]\n>>> \ttextconv = gpg --no-tty --decrypt\n>>>\n>>> For that you could still keep the exact .gitattributes format you have\n>>> here, i.e.:\n>>>\n>>> \tfile* diff-algorithm=$STRATEGY\n>>>\n>>> But we to pick it up we'd need either:\n>>>\n>>> \t[diff-algorithm]\n>>> \thistogram = myers\n>>>\n>>> Or:\n>>>\n>>> \t[diff-algorithm \"histogram\"]\n>>> \tallow = true\n>>\n>> This would help address slowness from the diff algorithm itself. I'm not opposed\n>> to adding this config if this attack vector is concerning to people.\n>>\n>> However, it wouldn't help address the additional latency of scanning\n>> .gitattributes to find the diff algorithm.\n>>\n>> Would a separate config to allow gitattributes be helpful here?\n>>\n>> [diff-algorithm]\n>> \tattributes = true\n>>\n>>>\n>>> The former form being one that would allow you to map the .gitattributes\n>>> of the repo (but maybe that would be redundant to\n>>> .git/info/attributes)...\n"},{"id":"471698","messageId":"230207.86sffh2xcu.gmgdl@evledraar.gmail.com","threadId":"59197","inReplyTo":"d18a5c32-2f15-93ad-ccbf-e8f048edb311@dunelm.org.uk","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-07T17:27:02Z","receivedAt":"2023-02-07T17:34:31Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 07 2023, Phillip Wood wrote:\n\n> This is slightly off topic but one thing I'd really like is a way to\n> tell diff use automatically use --diff-words on some files\n> (e.g. Documentation/*)\n\nUnlike changing the algorithm, -U options, diff.orderFile etc. doing\nthat would give you a diff that can't be applied with \"git apply\" or\nother tools that expect a valid unified diff.\n\nSo I can imagine that it would be neat in some contexts, but such a\nchange would have much wider implications than options that tweak how a\nvalid unified diff looks, or is generated.\n\nWe'd need some way to mark a diff as \"for ad-hoc viewing only\".\n\nBut as it sounds like you want this for git.git the\nDocumentation/doc-diff script is much better than anything word-diff\ncould spew out, as it diffs the resulting generated docs.\n\nI wonder (but haven't tried) whether you can't \"diff\" that using the\nsame method that can be used to diff binary files using a custom driver.\n\nHrm, except that in that case (with includes etc) there isn't really a\n1=1 mapping between files within Documentation/ and generated docs (due\nto includes etc.). But I suppose it could be used only for those files\nthat 1=1 correspond to the generated manpages.\n"},{"id":"471701","messageId":"Y+KQtqNPews3vBS8@coredump.intra.peff.net","threadId":"59197","inReplyTo":"8e73793b0db3e84366a9c6441cc0fdc04f9614a5.1675568781.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-07T17:56:06Z","receivedAt":"2023-02-07T17:56:11Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Feb 05, 2023 at 03:46:21AM +0000, John Cai via GitGitGadget wrote:\n\n> +`diff-algorithm`\n> +^^^^^^^^^^^^^^^^\n> +\n> +The attribute `diff-algorithm` affects which algorithm Git uses to generate\n> +diffs. This allows defining diff algorithms per file extension. Precedence rules\n> +are as follows, in order from highest to lowest:\n> +\n> +*Command line option*\n> +\n> +Pass in the `--diff-algorithm` command line option int git-diff(1)\n> +\n> +*Git attributes*\n> +\n> +------------------------\n> +*.json\tdiff-algorithm=histogram\n> +------------------------\n> +\n> +*Git config*\n> +\n> +----------------------------------------------------------------\n> +[diff]\n> +\talgorithm = histogram\n> +----------------------------------------------------------------\n\nFrom the user's perspective, this is weirdly inconsistent with the\nexisting diff attributes, which would be more like:\n\n # in .gitattributes\n *.json diff=json \n\n # in config\n [diff \"json\"]\n algorithm = histogram\n\nI know why one might choose the scheme you did; it kicks in if the repo\nsets the algorithm, without users having to set up any extra config.\nWhich is sort of nice, if we assume that malicious actors don't have any\nincentive to pick the algorithm. In theory they don't, though I saw Ævar\nmention possible DoS elsewhere in the thread.\n\n Side note: It's also possible that algorithm selection could be\n required to trigger a separate security bug (say, a buffer overflow in\n the patience code or something), so restricting that works in a\n belt-and-suspenders way. But that somehow feels like like the wrong\n side of the paranoia-vs-feature line.\n\nSo I dunno. I recognize that this scheme fulfills your immediate needs\nbetter, but I fear that we'll be stuck with a weird split between \"diff\"\nand \"diff-*\" attributes forever. In the long run, having a way for the\nrepo to say \"and here is some config I recommend to you\" would give you\nthe best of both, but that is a challenging topic that has been\ndiscussed and punted on for many years.\n\n-Peff\n"},{"id":"471719","messageId":"230207.86k00t2owm.gmgdl@evledraar.gmail.com","threadId":"59197","inReplyTo":"Y+KQtqNPews3vBS8@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-07T20:18:33Z","receivedAt":"2023-02-07T20:37:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 07 2023, Jeff King wrote:\n\n> On Sun, Feb 05, 2023 at 03:46:21AM +0000, John Cai via GitGitGadget wrote:\n\n> From the user's perspective, this is weirdly inconsistent with the\n> existing diff attributes, which would be more like:\n>\n> # in .gitattributes\n> *.json diff=json \n>\n> # in config\n> [diff \"json\"]\n> algorithm = histogram\n\nThat does look more elegant.\n\n> I know why one might choose the scheme you did; it kicks in if the repo\n> sets the algorithm, without users having to set up any extra config.\n> Which is sort of nice, if we assume that malicious actors don't have any\n> incentive to pick the algorithm. In theory they don't, though I saw Ævar\n> mention possible DoS elsewhere in the thread.\n>\n> Side note: It's also possible that algorithm selection could be\n> required to trigger a separate security bug (say, a buffer overflow in\n> the patience code or something), so restricting that works in a\n> belt-and-suspenders way. But that somehow feels like like the wrong\n> side of the paranoia-vs-feature line.\n>\n> So I dunno. I recognize that this scheme fulfills your immediate needs\n> better, but I fear that we'll be stuck with a weird split between \"diff\"\n> and \"diff-*\" attributes forever. In the long run, having a way for the\n> repo to say \"and here is some config I recommend to you\" would give you\n> the best of both, but that is a challenging topic that has been\n> discussed and punted on for many years.\n\nIf (and I'm not taking a stance on whether this is the case here) we\nthink that a hypothetical .gitconfig in-repo combined with\n.gitattributes would be more elegant for this or other cases, I don't\nsee why the path to some very limited version of that where we read\nliterally one whitelisted config variable from such a .gitconfig, and\nignore everything else, wouldn't be OK.\n\nI.e. the more general in-repo .gitconfig discussion (of which there was\nsome discussion this past Git Merge in Chicago) includes potentially\nmuch harder things\n\nLike what to do with changing genreal in-repo config, if and how to\ncompare that against some local whitelist of what sort of config we\nshould trust (or none) etc. etc.\n\nBut if we agreed that we'd be willing to trust the remote with some\nconfig-by-another-name unconditionally, as is being proposed here, we\ncould easily read that one thing from an in-repo .gitconfig, ignore\neverything else, and then just document that interface as a\nspecial-case.\n\nWe have several such \"limited config\" readers already, and the relevant\nbits of the config parser already have to handle arbitrary \"git config\"\nfiles in-tree, due to .gitmodules.\n\nWe even have special-snowflake config in the configspace already that\ndoesn't obey the usual rules: The trace2.* config is only read from the\n--system config, as it has an inherent bootsrtapping problem in wanting\nto log as early as possible.\n\nThe advantage of the limited in-repo .gitconfig would be that if we\nthink the interface was more elegant this would be a way to start small\nin a way that would be future-proof as far as the permanent UX goes.\n\nIf there's interest the read_early_config(), read_very_early_config()\nand gitmodules_config_oid() functions are good places to look.\n\nThe last one in particular could be generalized pretty easily to read a\nlimited in-tree .gitconfig.\n\nAlthough plugging it into the \"config order\" might be tricky, I haven't\ntried.\n\nEven a minimal implementation of such a thing would probably want a \"git\nconfig --repository\" (or whatever we call the flag) to go with\n\"--local\", \"--system\" etc, to spew out something sensible from\n\"--show-origin\" etc.\n"},{"id":"471720","messageId":"xmqqcz6lyzhw.fsf@gitster.g","threadId":"59197","inReplyTo":"230207.86k00t2owm.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-07T20:47:07Z","receivedAt":"2023-02-07T20:47:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n>> From the user's perspective, this is weirdly inconsistent with the\n>> existing diff attributes, which would be more like:\n>>\n>> # in .gitattributes\n>> *.json diff=json \n>>\n>> # in config\n>> [diff \"json\"]\n>> algorithm = histogram\n>\n> That does look more elegant.\n\nWe use attributes to define what it is, while configurations are\nused to define what to do on different things. The separation of\nattributes and configuration came not from \"elegance\" or \"security\"\nbut from a lot more practical reasons.\n\nFor a tracked file, the fact that it contains JSON text as payload\ndoes not change per user who cloned the project, or per platform the\nuser used to do so. In-tree .gitattributes that the project\ncontrols is a perfect way to define what it is for each file.\n\nOn the other hand, the diff program suitable to compare two JSON\nfiles may vary per platform (your favorite Windows program may not\nbe available to me) and per user (a platform may support more than\none and the choice becomes the matter of personal taste).\n\nThe security aspect of giving users tighter control over which exact\nprograms are to be run by not allowing the attributes or so called\nin-tree configuration mechansim is a small bonus that fell out as a\nconsequence.\n"},{"id":"471723","messageId":"230207.86bkm52n8x.gmgdl@evledraar.gmail.com","threadId":"59197","inReplyTo":"xmqqcz6lyzhw.fsf@gitster.g","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-07T21:05:29Z","receivedAt":"2023-02-07T21:12:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 07 2023, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>>> From the user's perspective, this is weirdly inconsistent with the\n>>> existing diff attributes, which would be more like:\n>>>\n>>> # in .gitattributes\n>>> *.json diff=json \n>>>\n>>> # in config\n>>> [diff \"json\"]\n>>> algorithm = histogram\n>>\n>> That does look more elegant.\n>\n> We use attributes to define what it is, while configurations are\n> used to define what to do on different things. The separation of\n> attributes and configuration came not from \"elegance\" or \"security\"\n> but from a lot more practical reasons.\n>\n> For a tracked file, the fact that it contains JSON text as payload\n> does not change per user who cloned the project, or per platform the\n> user used to do so. In-tree .gitattributes that the project\n> controls is a perfect way to define what it is for each file.\n>\n> On the other hand, the diff program suitable to compare two JSON\n> files may vary per platform (your favorite Windows program may not\n> be available to me) and per user (a platform may support more than\n> one and the choice becomes the matter of personal taste).\n>\n> The security aspect of giving users tighter control over which exact\n> programs are to be run by not allowing the attributes or so called\n> in-tree configuration mechansim is a small bonus that fell out as a\n> consequence.\n\nTo clarify, I'm not suggesting that we ever read arbitrary parts of the\n\"diff..\" config space, but that we could whitelist one set\nof \"diff..\"=\"\".\n\nThe reason to do it being that, as Jeff points out, that config\nmechanism is already established, and arguably more elegant. I.e. that\nthis is a natural fit for \"diff=\" in .gitattributes), and that\nmechanism is already tied to config as Jeff's example shows.\n\nSome of your reply seems like it assumed that I was suggesting that we\nread \"diff.algorithm\" (i.e. a config setting to apply for all paths)\nfrom an in-repo .gitconfig.\n\nI wasn't suggesting that, nor that we open Pandora's box on starting a\nlimited in-repo .gitconfig support with anything remotely to do with\nexecuting arbitrary commands (which the full \"diff..\" space\ndoes support).\n"},{"id":"471725","messageId":"xmqq4jrxyxl2.fsf@gitster.g","threadId":"59197","inReplyTo":"230207.86bkm52n8x.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-07T21:28:25Z","receivedAt":"2023-02-07T21:28:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> To clarify, I'm not suggesting that we ever read arbitrary parts of the\n> \"diff..\" config space, but that we could whitelist one set\n> of \"diff..\"=\"\".\n\nWhen the value names the path to an executable or the command line\nto invoke a program, there is no \"portable\" value that is useful.\nWhitelisting macOS only program only because its pathname is one of\nthe known values does not help me running something else.\n"},{"id":"471726","messageId":"230207.867cwt2kzh.gmgdl@evledraar.gmail.com","threadId":"59197","inReplyTo":"xmqq4jrxyxl2.fsf@gitster.g","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-07T21:44:26Z","receivedAt":"2023-02-07T22:01:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 07 2023, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> To clarify, I'm not suggesting that we ever read arbitrary parts of the\n>> \"diff..\" config space, but that we could whitelist one set\n>> of \"diff..\"=\"\".\n>\n> When the value names the path to an executable or the command line\n> to invoke a program, there is no \"portable\" value that is useful.\n> Whitelisting macOS only program only because its pathname is one of\n> the known values does not help me running something else.\n\nWe're not talking about invoking an executable of any kind, but a\nmechanism to pick up a \"diff.algorithm\" setting referring to one of our\nbuilt-in algorithms, either via the in-repo .gitattributes, as in John's\noriginal proposal:\n\n\t*.json diff-algorithm=histogram\n\nOr via some hybrid .gitattributes/.gitconfig mechanism, where (if I\nunderstand Jeff's suggestion correctly) the two would contain:\n\n\t# In .gitattributes\n\t*.json diff=json \n\t# In .gitconfig\n\t[diff \"json\"]\n\talgorithm = histogram\n\nIn terms of implementation this would be pretty much what we do with\n\"submodule..update\", where \"Allowed values here are checkout,\nrebase, merge or none.\".\n\nI don't see where you're getting this suggestion of \"[a] diff program\nsuitable to compare two JSON files\" from something I said.\n\nThat is a thing we generally support, but I've only mentioned that\narbitrary command execution (e.g. in [1]) as something we explicitly\n*don't* want to support in this context.\n\nThe \"submodule..update\" example is particularly relevant because\nit also supports arbitrary command execution with \"!\"-prefixed values\n(the rest being the arbitrary command).\n\nBut that's something we specifically exclude when reading the in-repo\nversion, it's only allowed in the user-controlled config.\n\nI'm not saying that we should be going for the read-just-the-one-key\nin-repo .gitconfig approach here, just *if* we like that interface\nbetter the implementation should be relatively straightforward, assuming\nthat we like the proposal in principle, and are just bikeshedding about\nwhat the mecanhism to enable it would be.\n\nThe advantage of that being that it more naturally slots into existing\nconfig, e.g. \"diff..binary=\", this being just another\n\"diff..\".\n\nAnd that the way to do that securely is something we're doing for\nseveral \"submodule.*\" keys, so if we pick that approach tweaking or\nstealing ideas from that code should be relatively straightforward.\n\n1. https://lore.kernel.org/git/230206.865yce7n1w.gmgdl@evledraar.gmail.com/\n"},{"id":"471837","messageId":"CABPp-BGdKjnChEp4zeCcz24wiEJVJb9Tp40MTWn1m0LRZu+M+Q@mail.gmail.com","threadId":"59197","inReplyTo":"230206.865yce7n1w.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-02-09T07:50:41Z","receivedAt":"2023-02-09T07:50:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Feb 6, 2023 at 9:24 AM Ævar Arnfjörð Bjarmason wrote:\n>\n> On Sun, Feb 05 2023, John Cai via GitGitGadget wrote:\n>\n> > From: John Cai \n> > [...]\n> > +\n> > + if (!o->xdl_opts_command_line) {\n> > + static struct attr_check *check;\n> > + const char *one_diff_algo;\n> > + const char *two_diff_algo;\n> > +\n> > + check = attr_check_alloc();\n> > + attr_check_append(check, git_attr(\"diff-algorithm\"));\n> > +\n> > + git_check_attr(the_repository->index, NULL, one->path, check);\n> > + one_diff_algo = check->items[0].value;\n> > + git_check_attr(the_repository->index, NULL, two->path, check);\n> > + two_diff_algo = check->items[0].value;\n> > +\n> > + if (!ATTR_UNSET(one_diff_algo) && !ATTR_UNSET(two_diff_algo) &&\n> > + !strcmp(one_diff_algo, two_diff_algo))\n> > + set_diff_algorithm(o, one_diff_algo);\n> > +\n> > + attr_check_free(check);\n>\n> This is a bit nitpicky, but I for one would find this much easier to\n> read with some shorter variables, here just with \"a\" rather than\n> \"one_diff_algo\", \"b\" instead of \"two_diff_algo\", and splitting\n> \"the_repository->index\" into \"istate\" (untested):\n>\n> + if (!o->xdl_opts_command_line) {\n> + static struct attr_check *check;\n> + const char *a;\n> + const char *b;\n> + struct index_state *istate = the_repository->index;\n> +\n> + check = attr_check_alloc();\n> + attr_check_append(check, git_attr(\"diff-algorithm\"));\n> +\n> + git_check_attr(istate, NULL, one->path, check);\n> + a = check->items[0].value;\n> + git_check_attr(istate, NULL, two->path, check);\n> + b = check->items[0].value;\n> +\n> + if (!ATTR_UNSET(a) && !ATTR_UNSET(b) && !strcmp(a, b))\n> + set_diff_algorithm(o, a);\n> +\n> + attr_check_free(check);\n> + }\n>\n> That also nicely keeps the line length shorter.\n>\n> > @@ -333,6 +333,8 @@ struct diff_options {\n> > int prefix_length;\n> > const char *stat_sep;\n> > int xdl_opts;\n> > + /* If xdl_opts has been set via the command line. */\n> > + int xdl_opts_command_line;\n> >\n> > /* see Documentation/diff-options.txt */\n> > char **anchors;\n> > diff --git a/t/lib-diff-alternative.sh b/t/lib-diff-alternative.sh\n> > index 8d1e408bb58..630c98ea65a 100644\n> > --- a/t/lib-diff-alternative.sh\n> > +++ b/t/lib-diff-alternative.sh\n> > @@ -107,8 +107,27 @@ EOF\n> >\n> > STRATEGY=$1\n> >\n> > + test_expect_success \"$STRATEGY diff from attributes\" '\n> > + echo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n> > + test_must_fail git diff --no-index file1 file2 > output &&\n> > + test_cmp expect output\n> > + '\n> > +\n> > test_expect_success \"$STRATEGY diff\" '\n> > - test_must_fail git diff --no-index \"--$STRATEGY\" file1 file2 > output &&\n> > + test_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n>\n> Nit: The usual style is \">output\", not \"> output\".\n>\n> > + test_cmp expect output\n> > + '\n> > +\n> > + test_expect_success \"$STRATEGY diff command line precedence before attributes\" '\n> > + echo \"file* diff-algorithm=meyers\" >.gitattributes &&\n> > + test_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n> > + test_cmp expect output\n> > + '\n> > +\n> > + test_expect_success \"$STRATEGY diff attributes precedence before config\" '\n> > + git config diff.algorithm default &&\n> > + echo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n> > + test_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n> > test_cmp expect output\n> > '\n> >\n> > @@ -166,5 +185,11 @@ EOF\n> > test_must_fail git diff --no-index \"--$STRATEGY\" uniq1 uniq2 > output &&\n> > test_cmp expect output\n> > '\n> > +\n> > + test_expect_success \"$STRATEGY diff from attributes\" '\n> > + echo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n> > + test_must_fail git diff --no-index uniq1 uniq2 > output &&\n> > + test_cmp expect output\n> > + '\n> > }\n>\n> For some non-nitpicking, I do worry about exposing this as a DoS vector,\n> e.g. here's a diff between two distant points in git.git with the\n> various algorithms:\n>\n> $ hyperfine -r 1 -L a patience,minimal,histogram,myers 'git diff --diff-algorithm={a} v2.0.0 v2.28.0'\n> Benchmark 1: git diff --diff-algorithm=patience v2.0.0 v2.28.0\n> Time (abs ≡): 42.121 s [User: 41.879 s, System: 0.144 s]\n>\n> Benchmark 2: git diff --diff-algorithm=minimal v2.0.0 v2.28.0\n> Time (abs ≡): 35.634 s [User: 35.473 s, System: 0.160 s]\n>\n> Benchmark 3: git diff --diff-algorithm=histogram v2.0.0 v2.28.0\n> Time (abs ≡): 46.912 s [User: 46.657 s, System: 0.228 s]\n>\n> Benchmark 4: git diff --diff-algorithm=myers v2.0.0 v2.28.0\n> Time (abs ≡): 33.233 s [User: 33.072 s, System: 0.160 s]\n>\n> Summary\n> 'git diff --diff-algorithm=myers v2.0.0 v2.28.0' ran\n> 1.07 times faster than 'git diff --diff-algorithm=minimal v2.0.0 v2.28.0'\n> 1.27 times faster than 'git diff --diff-algorithm=patience v2.0.0 v2.28.0'\n> 1.41 times faster than 'git diff --diff-algorithm=histogram v2.0.0 v2.28.0'\n\nI'm really surprised by these numbers; they aren't remotely close to\nwhat I compute. Am I correct in understanding you ran these in\ngit.git? Was your computer overloaded? Was your git.git in some\nserious need of repacking? Was it on a network filesystem? If you\nrun with more than 1 run, are your numbers even repeatable?\n\nUsing git compiled from current main, I see:\n\n$ hyperfine -L a patience,minimal,histogram,myers './git diff\n--diff-algorithm={a} v2.0.0 v2.28.0'\nBenchmark 1: ./git diff --diff-algorithm=patience v2.0.0 v2.28.0\n Time (mean ± σ): 1.142 s ± 0.033 s [User: 1.022 s, System: 0.114 s]\n Range (min … max): 1.117 s … 1.212 s 10 runs\n\nBenchmark 2: ./git diff --diff-algorithm=minimal v2.0.0 v2.28.0\n Time (mean ± σ): 1.959 s ± 0.011 s [User: 1.830 s, System: 0.120 s]\n Range (min … max): 1.947 s … 1.976 s 10 runs\n\nBenchmark 3: ./git diff --diff-algorithm=histogram v2.0.0 v2.28.0\n Time (mean ± σ): 1.187 s ± 0.007 s [User: 1.065 s, System: 0.115 s]\n Range (min … max): 1.175 s … 1.200 s 10 runs\n\nBenchmark 4: ./git diff --diff-algorithm=myers v2.0.0 v2.28.0\n Time (mean ± σ): 1.194 s ± 0.007 s [User: 1.068 s, System: 0.120 s]\n Range (min … max): 1.184 s … 1.206 s 10 runs\n\nSummary\n './git diff --diff-algorithm=patience v2.0.0 v2.28.0' ran\n 1.04 ± 0.03 times faster than './git diff\n--diff-algorithm=histogram v2.0.0 v2.28.0'\n 1.05 ± 0.03 times faster than './git diff --diff-algorithm=myers\nv2.0.0 v2.28.0'\n 1.71 ± 0.05 times faster than './git diff --diff-algorithm=minimal\nv2.0.0 v2.28.0'\n\nAnd this is on a kind-of low-end refurbished laptop from a few years\nago (although the repo was recently gc'ed).\n\nI'm biased towards histogram (and making it the default rather than\nmaking it configurable per file), but that's probably obvious given\nthat I made ort use it unconditionally. And when I made ort use it,\nit was actually a minor performance penalty (~1% IIRC)[*], but I\nthought it was worth it since (a) histogram diffs are more\nunderstandable to users in general, (b) the histogram diff data\nstructures provide an idea for possibly solving some ugly corner cases\nthat I don't see a way to solve with the other diffs.\n\n[*] Phillip came along and made histogram faster after my measurements\n(663c5ad035 (\"diff histogram: intern strings\", 2021-11-17)), so that\nmay not be true anymore.\n"},{"id":"471839","messageId":"CABPp-BGhuTyq_hrpMc+Ky3yt1UgO7DcAsgcYH15FK--QLdCsQw@mail.gmail.com","threadId":"59197","inReplyTo":"7852AC7B-7A4E-4DD0-ADEA-CFFD5D16C595@gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-02-09T08:26:44Z","receivedAt":"2023-02-09T08:27:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi John,\n\nOn Mon, Feb 6, 2023 at 12:02 PM John Cai wrote:\n>\n> Hi Phillip,\n>\n> On 6 Feb 2023, at 11:27, Phillip Wood wrote:\n>\n> > Hi John\n> >\n> > On 05/02/2023 03:46, John Cai via GitGitGadget wrote:\n> >> From: John Cai \n> >>\n> >> It can be useful to specify diff algorithms per file type. For example,\n> >> one may want to use the minimal diff algorithm for .json files, another\n> >> for .c files, etc.\n> >\n> > Have you got any examples of why this is useful? I find myself occasionally changing the algorithm when the default gives a sub-optimal diff but I've not really noticed any pattern with respect to file types.\n>\n> At $DAYJOB, there has been a discussion and request for a feature like this [1].\n> One use case that came up was to be able to set a different diff algorithm for\n> .json files.\n>\n> 1. https://gitlab.com/gitlab-org/gitaly/-/issues/2591\n\nA couple points:\n\nFirst, there seems to be a misunderstanding in that issue. In\nparticular, the merge algorithm does call into the xdiff library to do\nthe three-way content merge of individual files, and when it does so,\nit has to specify the diff algorithm (or take the default, currently\nmyers). merge-recursive allows the diff algorithm to be specified by\nthe user (there are\n-Xdiff-algorithm={histogram,minimal,patience,myers} flags to\nmerge/rebase for it), while merge-ort uses histogram (though it uses\nthe same parser as merge-recursive and thus gets the variables set\nfrom the -Xdiff-algorithm flag, it just ignores those values and\nhardcodes histogram).\n\nSecond, I also think the user request got converted to a particular\nsolution without looking at the wider problem space: The idea seemed\nto assume \"myers\" is default for a good reason, and thus asked for an\noption to use something else. I'm not sure the assumption is valid; I\nthink \"myers\" is default for historical reasons and histogram is\nbetter not just for special Salesforce xml files, but code files too.\nThe output makes more sense to users. So much so that even though my\nsimple testing suggested it had a 2% performance penalty compared to\nmyers, I forced ort to use it[1] even though I designed everything\nelse in that algorithm around eking out maximum performance. Others\nwho have tested the diff algorithms have also found histogram has very\nsimilar performance to myers, and oftentimes even beats it[2][3].\nAlso, worries about invalidating rerere caches[4] was real, but we\nalready paid that price when we switched to ort. And if performance\nis still a worry, [3] gives me reason to believe we can make our\nhistogram implementation faster. Finally, for the period of time when\nPalantir was letting me make an internal git distribution (mostly for\ntesting ort), I also carried a patch that changed the default diff\nalgorithm to histogram (not just for ort, but for diff/log/etc. as\nwell). Never had any complaints from the users from it. Perhaps you\ncould do the same in your local version of git used by gitaly?\n\n[1] See c8017176ac (\"merge-ort: use histogram diff\", 2020-12-13)\n[2] From 85551232b5 (\"perf: compare diff algorithms\", 2012-03-06):\n\"This does indeed show that histogram diff slightly beats Myers, while\npatience is much slower than the others.\"\n[3] https://github.com/pascalkuthe/imara-diff\n[4] https://lore.kernel.org/git/20120307114714.GA14990@sigill.intra.peff.net/\n"},{"id":"471840","messageId":"CABPp-BFKQXe-EJOd9z1TrisL64NuV9A132rf9MwV_7w79QQ9YQ@mail.gmail.com","threadId":"59197","inReplyTo":"B544D9E8-13C4-4682-9BDA-D6E19B51C91D@gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-02-09T08:44:00Z","receivedAt":"2023-02-09T08:44:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi John,\n\nOn Mon, Feb 6, 2023 at 12:47 PM John Cai wrote:\n>\n[...]\n> That being said, here's a separate issue. I benchmarked the usage of\n> .gitattributes as introduced in this patch series, and indeed it does look like\n> there is additional latency:\n>\n> $ echo \"* diff-algorithm=patience >> .gitattributes\n> $ hyperfine -r 5 'git-bin-wrapper diff --diff-algorithm=patience v2.0.0 v2.28.0' ✭\n> Benchmark 1: git-bin-wrapper diff --diff-algorithm=patience v2.0.0 v2.28.0\n> Time (mean ± σ): 889.4 ms ± 113.8 ms [User: 715.7 ms, System: 65.3 ms]\n> Range (min … max): 764.1 ms … 1029.3 ms 5 runs\n>\n> $ hyperfine -r 5 'git-bin-wrapper diff v2.0.0 v2.28.0' ✭\n> Benchmark 1: git-bin-wrapper diff v2.0.0 v2.28.0\n> Time (mean ± σ): 2.146 s ± 0.368 s [User: 0.827 s, System: 0.243 s]\n> Range (min … max): 1.883 s … 2.795 s 5 runs\n>\n> and I imagine the latency scales with the size of .gitattributes. Although I'm\n> not familiar with other parts of the codebase and how it deals with the latency\n> introduced by reading attributes files.\n\nYeah, that seems like a large relative performance penalty. I had the\nfeeling that histogram wasn't made the default over myers mostly due\nto inertia and due to a potential 2% loss in performance (since\npotentially corrected by Phillip's 663c5ad035 (\"diff histogram: intern\nstrings\", 2021-11-17)). If we had changed the default diff algorithm\nto histogram, I suspect folks wouldn't have been asking for per-file\nknobs to use a better diff algorithm. And the performance penalty for\nthis alternative is clearly much larger than 2%, which makes me think\nwe might want to just revisit the default instead of allowing per-file\ntweaks.\n\nAnd on a separate note...\n\nThere's another set of considerations we might need to include here as\nwell that I haven't seen anyone else in this thread talk about:\n\n* When trying to diff files, do we read the .gitattributes file from\nthe current checkout to determine the diff algorithm(s)? Or the\nindex? Or the commit we are diffing against?\n* If we use the current checkout or index, what about bare clones or\ndiffing between two different commits?\n* If diffing between two different commits, and the .gitattributes has\nchanged between those commits, which .gitattributes file wins?\n* If diffing between two different commits, and the .gitattributes has\nNOT changed, BUT a file has been renamed and the old and new names\nhave different rules, which rule wins?\n\n* If per-file diff algorithms are adopted widely enough, will we be\nforced to change the merge algorithm to also pay attention to them?\nIf it does, more complicated rename cases occur and we need rules for\nhow to handle those.\n* If the merge algorithm has to pay attention to .gitattributes for\nthis too, we'll have even more corner cases around what happens if\nthere are merge conflicts in .gitattributes itself (which is already\nkind of ugly and kludged)\n\n\nAnyway, I know I'm a bit animated and biased in this area, and I\napologize if I'm a bit too much so. Even if I am, hopefully my\ncomments at least provide some useful context.\n"},{"id":"471841","messageId":"CABPp-BHhhUhRqn=kKcDiV3EMckBSk2EE8TKZ-PoeqTsKWuvAng@mail.gmail.com","threadId":"59197","inReplyTo":"65129323-326F-4E4A-B6F8-06DC3BBE7B58@gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-02-09T09:09:30Z","receivedAt":"2023-02-09T09:09:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi John and Phillip,\n\nOn Tue, Feb 7, 2023 at 9:05 AM John Cai wrote:\n>\n[...]\n> > Perhaps I'm over simplifying but having read the issue you linked to I couldn't help feeling that the majority of users might be satisfied by just changing gitlab to use the patience algorithm when generating diffs.\n>\n> Right, I recognize this is a judgment call that may be best left up to the list.\n>\n> We don't have a way in GitLab to change the diff algorithm currently. Of course\n> that can be implemented outside of Git,\n\nWell, the below doesn't allow users to make diffs better for\n*individual* files of interest, but if you agree with me that we\nshould just make diffs better for all users automatically, it's a\ntwo-line change in git.git that I'd love to eventually convince the\nproject to take (though obviously doing that would also require some\ndocumentation changes and some good messaging in release notes and\nwhatnot). I've used it for a good long while, and had a few dozen\nusers using this patch too, all without complaint:\n\ndiff --git a/diff.c b/diff.c\nindex 329eebf16a..77a46d5b7d 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -55,7 +55,7 @@ static int diff_relative;\n static int diff_stat_graph_width;\n static int diff_dirstat_permille_default = 30;\n static struct diff_options default_diff_options;\n-static long diff_algorithm;\n+static long diff_algorithm = XDF_HISTOGRAM_DIFF;\n static unsigned ws_error_highlight_default = WSEH_NEW;\n\n static char diff_colors[][COLOR_MAXLEN] = {\ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex b298f220e0..2f663eab72 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1549,7 +1549,7 @@ test_expect_success 'short lines of opposite\nsign do not get marked as moved' '\n this line should be marked as oldMoved newMoved\n unchanged 4\n EOF\n- test_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n+ test_expect_code 1 git diff --diff-algorithm=myers --no-index\n--color --color-moved=zebra \\\n old.txt new.txt >output && cat output &&\n grep -v index output | test_decode_color >actual &&\n cat >expect <<-\\EOF &&\n\n\nI used histogram above rather than patience, since (a) it's what git's\nmerge backend uses, (b) it produces roughly similar results to\npatience from a user perspective, (c) past testing has shown it to be\nsomewhat faster than patience, and (d) we've potentially got some\nleads in how to speed up our histogram implementation from the README\nover at https://github.com/pascalkuthe/imara-diff. But, if you really\nwanted to use patience as the default, it'd also be an easy tweak.\n\nAnyway, just some food for thought.\n"},{"id":"471844","messageId":"230209.86lel7xi8l.gmgdl@evledraar.gmail.com","threadId":"59197","inReplyTo":"CABPp-BGdKjnChEp4zeCcz24wiEJVJb9Tp40MTWn1m0LRZu+M+Q@mail.gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-09T09:41:33Z","receivedAt":"2023-02-09T10:09:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 08 2023, Elijah Newren wrote:\n\n> On Mon, Feb 6, 2023 at 9:24 AM Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Sun, Feb 05 2023, John Cai via GitGitGadget wrote:\n>>\n>> > From: John Cai \n>> > [...]\n>> > +\n>> > + if (!o->xdl_opts_command_line) {\n>> > + static struct attr_check *check;\n>> > + const char *one_diff_algo;\n>> > + const char *two_diff_algo;\n>> > +\n>> > + check = attr_check_alloc();\n>> > + attr_check_append(check, git_attr(\"diff-algorithm\"));\n>> > +\n>> > + git_check_attr(the_repository->index, NULL, one->path, check);\n>> > + one_diff_algo = check->items[0].value;\n>> > + git_check_attr(the_repository->index, NULL, two->path, check);\n>> > + two_diff_algo = check->items[0].value;\n>> > +\n>> > + if (!ATTR_UNSET(one_diff_algo) && !ATTR_UNSET(two_diff_algo) &&\n>> > + !strcmp(one_diff_algo, two_diff_algo))\n>> > + set_diff_algorithm(o, one_diff_algo);\n>> > +\n>> > + attr_check_free(check);\n>>\n>> This is a bit nitpicky, but I for one would find this much easier to\n>> read with some shorter variables, here just with \"a\" rather than\n>> \"one_diff_algo\", \"b\" instead of \"two_diff_algo\", and splitting\n>> \"the_repository->index\" into \"istate\" (untested):\n>>\n>> + if (!o->xdl_opts_command_line) {\n>> + static struct attr_check *check;\n>> + const char *a;\n>> + const char *b;\n>> + struct index_state *istate = the_repository->index;\n>> +\n>> + check = attr_check_alloc();\n>> + attr_check_append(check, git_attr(\"diff-algorithm\"));\n>> +\n>> + git_check_attr(istate, NULL, one->path, check);\n>> + a = check->items[0].value;\n>> + git_check_attr(istate, NULL, two->path, check);\n>> + b = check->items[0].value;\n>> +\n>> + if (!ATTR_UNSET(a) && !ATTR_UNSET(b) && !strcmp(a, b))\n>> + set_diff_algorithm(o, a);\n>> +\n>> + attr_check_free(check);\n>> + }\n>>\n>> That also nicely keeps the line length shorter.\n>>\n>> > @@ -333,6 +333,8 @@ struct diff_options {\n>> > int prefix_length;\n>> > const char *stat_sep;\n>> > int xdl_opts;\n>> > + /* If xdl_opts has been set via the command line. */\n>> > + int xdl_opts_command_line;\n>> >\n>> > /* see Documentation/diff-options.txt */\n>> > char **anchors;\n>> > diff --git a/t/lib-diff-alternative.sh b/t/lib-diff-alternative.sh\n>> > index 8d1e408bb58..630c98ea65a 100644\n>> > --- a/t/lib-diff-alternative.sh\n>> > +++ b/t/lib-diff-alternative.sh\n>> > @@ -107,8 +107,27 @@ EOF\n>> >\n>> > STRATEGY=$1\n>> >\n>> > + test_expect_success \"$STRATEGY diff from attributes\" '\n>> > + echo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n>> > + test_must_fail git diff --no-index file1 file2 > output &&\n>> > + test_cmp expect output\n>> > + '\n>> > +\n>> > test_expect_success \"$STRATEGY diff\" '\n>> > - test_must_fail git diff --no-index \"--$STRATEGY\" file1 file2 > output &&\n>> > + test_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n>>\n>> Nit: The usual style is \">output\", not \"> output\".\n>>\n>> > + test_cmp expect output\n>> > + '\n>> > +\n>> > + test_expect_success \"$STRATEGY diff command line precedence before attributes\" '\n>> > + echo \"file* diff-algorithm=meyers\" >.gitattributes &&\n>> > + test_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n>> > + test_cmp expect output\n>> > + '\n>> > +\n>> > + test_expect_success \"$STRATEGY diff attributes precedence before config\" '\n>> > + git config diff.algorithm default &&\n>> > + echo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n>> > + test_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n>> > test_cmp expect output\n>> > '\n>> >\n>> > @@ -166,5 +185,11 @@ EOF\n>> > test_must_fail git diff --no-index \"--$STRATEGY\" uniq1 uniq2 > output &&\n>> > test_cmp expect output\n>> > '\n>> > +\n>> > + test_expect_success \"$STRATEGY diff from attributes\" '\n>> > + echo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n>> > + test_must_fail git diff --no-index uniq1 uniq2 > output &&\n>> > + test_cmp expect output\n>> > + '\n>> > }\n>>\n>> For some non-nitpicking, I do worry about exposing this as a DoS vector,\n>> e.g. here's a diff between two distant points in git.git with the\n>> various algorithms:\n>>\n>> $ hyperfine -r 1 -L a patience,minimal,histogram,myers 'git diff --diff-algorithm={a} v2.0.0 v2.28.0'\n>> Benchmark 1: git diff --diff-algorithm=patience v2.0.0 v2.28.0\n>> Time (abs ≡): 42.121 s [User: 41.879 s, System: 0.144 s]\n>>\n>> Benchmark 2: git diff --diff-algorithm=minimal v2.0.0 v2.28.0\n>> Time (abs ≡): 35.634 s [User: 35.473 s, System: 0.160 s]\n>>\n>> Benchmark 3: git diff --diff-algorithm=histogram v2.0.0 v2.28.0\n>> Time (abs ≡): 46.912 s [User: 46.657 s, System: 0.228 s]\n>>\n>> Benchmark 4: git diff --diff-algorithm=myers v2.0.0 v2.28.0\n>> Time (abs ≡): 33.233 s [User: 33.072 s, System: 0.160 s]\n>>\n>> Summary\n>> 'git diff --diff-algorithm=myers v2.0.0 v2.28.0' ran\n>> 1.07 times faster than 'git diff --diff-algorithm=minimal v2.0.0 v2.28.0'\n>> 1.27 times faster than 'git diff --diff-algorithm=patience v2.0.0 v2.28.0'\n>> 1.41 times faster than 'git diff --diff-algorithm=histogram v2.0.0 v2.28.0'\n\n[snip around]\n\n> If you run with more than 1 run, are your numbers even repeatable?\n\nYes, but tl;dr it's diff.colorMoved=true, sorry, see below.\n\n> I'm really surprised by these numbers; they aren't remotely close to\n> what I compute. Am I correct in understanding you ran these in\n> git.git? Was your computer overloaded? Was your git.git in some\n> serious need of repacking? Was it on a network filesystem? \n\nJust on the box I regularly hack on, which isn't too overloaded, but I\nre-did these a bit more seriously.\n\nThis is/was on a Hetzner EX41S\n(https://docs.hetzner.com/robot/dedicated-server/general-information/root-server-hardware/),\nbut I tried it again now in /dev/shm/git.git with a fresh repo from:\n\n\tgit clone --bare git@github.com:git/git.git\n\nAnd (I didn't prune out the notice here where it says it's fuzzy (not\nearlier either)).\n\nThere's some Java thing eating ~50% of 1/8 CPU cores, but otherwise the\nbox is pretty idle, and this is current \"master\" with \"make CFLAGS=-O3\":\n\t\n\t$ hyperfine -w 2 -r 10 -L a patience,minimal,histogram,myers './git -C /dev/shm/git.git diff --diff-algorithm={a} v2.0.0 v2.28.0'\n\tBenchmark 1: ./git -C /dev/shm/git.git diff --diff-algorithm=patience v2.0.0 v2.28.0\n\t Time (mean ± σ): 41.131 s ± 0.550 s [User: 40.990 s, System: 0.104 s]\n\t Range (min … max): 40.323 s … 42.172 s 10 runs\n\t\n\tBenchmark 2: ./git -C /dev/shm/git.git diff --diff-algorithm=minimal v2.0.0 v2.28.0\n\t Time (mean ± σ): 34.821 s ± 0.307 s [User: 34.707 s, System: 0.100 s]\n\t Range (min … max): 34.512 s … 35.523 s 10 runs\n\t\n\tBenchmark 3: ./git -C /dev/shm/git.git diff --diff-algorithm=histogram v2.0.0 v2.28.0\n\t Time (mean ± σ): 45.443 s ± 0.274 s [User: 45.328 s, System: 0.107 s]\n\t Range (min … max): 44.932 s … 45.810 s 10 runs\n\t\n\tBenchmark 4: ./git -C /dev/shm/git.git diff --diff-algorithm=myers v2.0.0 v2.28.0\n\t Time (mean ± σ): 33.016 s ± 0.505 s [User: 32.893 s, System: 0.094 s]\n\t Range (min … max): 32.376 s … 33.999 s 10 runs\n\t\n\tSummary\n\t './git -C /dev/shm/git.git diff --diff-algorithm=myers v2.0.0 v2.28.0' ran\n\t 1.05 ± 0.02 times faster than './git -C /dev/shm/git.git diff --diff-algorithm=minimal v2.0.0 v2.28.0'\n\t 1.25 ± 0.03 times faster than './git -C /dev/shm/git.git diff --diff-algorithm=patience v2.0.0 v2.28.0'\n\t 1.38 ± 0.02 times faster than './git -C /dev/shm/git.git diff --diff-algorithm=histogram v2.0.0 v2.28.0'\n\nSo that's pretty much the same as my earlier results, but between the\nfresh repo, ram disk, warmup & 10 measuremnets for each these should be\nmore accurate.\n\n> Using git compiled from current main, I see:\n>\n> $ hyperfine -L a patience,minimal,histogram,myers './git diff\n> --diff-algorithm={a} v2.0.0 v2.28.0'\n> Benchmark 1: ./git diff --diff-algorithm=patience v2.0.0 v2.28.0\n> Time (mean ± σ): 1.142 s ± 0.033 s [User: 1.022 s, System: 0.114 s]\n> Range (min … max): 1.117 s … 1.212 s 10 runs\n>\n> Benchmark 2: ./git diff --diff-algorithm=minimal v2.0.0 v2.28.0\n> Time (mean ± σ): 1.959 s ± 0.011 s [User: 1.830 s, System: 0.120 s]\n> Range (min … max): 1.947 s … 1.976 s 10 runs\n>\n> Benchmark 3: ./git diff --diff-algorithm=histogram v2.0.0 v2.28.0\n> Time (mean ± σ): 1.187 s ± 0.007 s [User: 1.065 s, System: 0.115 s]\n> Range (min … max): 1.175 s … 1.200 s 10 runs\n>\n> Benchmark 4: ./git diff --diff-algorithm=myers v2.0.0 v2.28.0\n> Time (mean ± σ): 1.194 s ± 0.007 s [User: 1.068 s, System: 0.120 s]\n> Range (min … max): 1.184 s … 1.206 s 10 runs\n>\n> Summary\n> './git diff --diff-algorithm=patience v2.0.0 v2.28.0' ran\n> 1.04 ± 0.03 times faster than './git diff\n> --diff-algorithm=histogram v2.0.0 v2.28.0'\n> 1.05 ± 0.03 times faster than './git diff --diff-algorithm=myers\n> v2.0.0 v2.28.0'\n> 1.71 ± 0.05 times faster than './git diff --diff-algorithm=minimal\n> v2.0.0 v2.28.0'\n\nBut without diff.colorMoved=true I see basically your results:\n\t\n\t$ hyperfine -w 2 -r 10 -L a patience,minimal,histogram,myers './git -c diff.colorMoved=false -C /dev/shm/git.git diff --diff-algorithm={a} v2.0.0 v2.28.0'\n\tBenchmark 1: ./git -c diff.colorMoved=false -C /dev/shm/git.git diff --diff-algorithm=patience v2.0.0 v2.28.0\n\t Time (mean ± σ): 760.9 ms ± 45.2 ms [User: 698.6 ms, System: 62.1 ms]\n\t Range (min … max): 719.0 ms … 862.2 ms 10 runs\n\t \n\tBenchmark 2: ./git -c diff.colorMoved=false -C /dev/shm/git.git diff --diff-algorithm=minimal v2.0.0 v2.28.0\n\t Time (mean ± σ): 1.347 s ± 0.041 s [User: 1.281 s, System: 0.065 s]\n\t Range (min … max): 1.305 s … 1.417 s 10 runs\n\t \n\tBenchmark 3: ./git -c diff.colorMoved=false -C /dev/shm/git.git diff --diff-algorithm=histogram v2.0.0 v2.28.0\n\t Time (mean ± σ): 826.3 ms ± 51.1 ms [User: 767.5 ms, System: 58.6 ms]\n\t Range (min … max): 773.7 ms … 929.8 ms 10 runs\n\t \n\tBenchmark 4: ./git -c diff.colorMoved=false -C /dev/shm/git.git diff --diff-algorithm=myers v2.0.0 v2.28.0\n\t Time (mean ± σ): 801.1 ms ± 39.4 ms [User: 736.0 ms, System: 64.9 ms]\n\t Range (min … max): 771.6 ms … 904.2 ms 10 runs\n\t \n\tSummary\n\t './git -c diff.colorMoved=false -C /dev/shm/git.git diff --diff-algorithm=patience v2.0.0 v2.28.0' ran\n\t 1.05 ± 0.08 times faster than './git -c diff.colorMoved=false -C /dev/shm/git.git diff --diff-algorithm=myers v2.0.0 v2.28.0'\n\t 1.09 ± 0.09 times faster than './git -c diff.colorMoved=false -C /dev/shm/git.git diff --diff-algorithm=histogram v2.0.0 v2.28.0'\n\t 1.77 ± 0.12 times faster than './git -c diff.colorMoved=false -C /dev/shm/git.git diff --diff-algorithm=minimal v2.0.0 v2.28.0'\n\nSo they're all within the fuzz-factor, except \"minimal\".\n\n> And this is on a kind-of low-end refurbished laptop from a few years\n> ago (although the repo was recently gc'ed).\n>\n> I'm biased towards histogram (and making it the default rather than\n> making it configurable per file), but that's probably obvious given\n> that I made ort use it unconditionally. And when I made ort use it,\n> it was actually a minor performance penalty (~1% IIRC)[*], but I\n> thought it was worth it since (a) histogram diffs are more\n> understandable to users in general, (b) the histogram diff data\n> structures provide an idea for possibly solving some ugly corner cases\n> that I don't see a way to solve with the other diffs.\n\nTo bring this all home the thing I was going for upthread was to raise\n\"is this a concern?\" I was inclined to think \"no\", but didn't\nknow. Since someone who knows way more about the diffs than I probably\never will (i.e. you :) isn't waiving their hands in panic here I think\nwe can just consider this checkbox ticked.\n\nI.e. I've seen cases (and this is from vague recollection, I've got no\nexamples in front of me) where one diff algorithm is worse than others\non performance.\n\nBut if your intent is to DoS a service provider you can also just commit\nsome very long files or whatever, so whether the user can change the\ndiff algorithm is probably not that interesting, as long as the\nperformance is within some reasonable bound.\n"},{"id":"471847","messageId":"230209.868rh7xg14.gmgdl@evledraar.gmail.com","threadId":"59197","inReplyTo":"CABPp-BGhuTyq_hrpMc+Ky3yt1UgO7DcAsgcYH15FK--QLdCsQw@mail.gmail.com","subject":"\"bad\" diffs (was: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-09T10:31:03Z","receivedAt":"2023-02-09T10:58:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Feb 09 2023, Elijah Newren wrote:\n\n> [...] I\n> think \"myers\" is default for historical reasons and histogram is\n> better not just for special Salesforce xml files, but code files too.\n> The output makes more sense to users. So much so that even though my\n> simple testing suggested it had a 2% performance penalty compared to\n> myers, I forced ort to use it[1] even though I designed everything\n> else in that algorithm around eking out maximum performance. Others\n> who have tested the diff algorithms have also found histogram has very\n> similar performance to myers, and oftentimes even beats it[2][3].\n> Also, worries about invalidating rerere caches[4] was real, but we\n> already paid that price when we switched to ort.\n\nFWIW as someone who went through that one-time pain for my many git.git\ntopics it wasn't even a price, it was paying me!\n\nMaybe it's just confirmation bias, or looking at the conflicts with\nfresh eyes, but I found I mis-solved some of them seemingly because the\noutput from the old conflicts was so confusing, but much better with\n\"ort\".\n\n> And if performance\n> is still a worry, [3] gives me reason to believe we can make our\n> histogram implementation faster. Finally, for the period of time when\n> Palantir was letting me make an internal git distribution (mostly for\n> testing ort), I also carried a patch that changed the default diff\n> algorithm to histogram (not just for ort, but for diff/log/etc. as\n> well). Never had any complaints from the users from it. Perhaps you\n> could do the same in your local version of git used by gitaly?\n\nI think that might be worth considering for GitLab, John? Although the\nbias has definitely been to go with vanilla git semantics, but it sounds\nlike you might be in favor of endorsing a patch to change the default,\nso if that happens to solve the problem...\n\n> I also think the user request got converted to a particular\n> solution without looking at the wider problem space: The idea seemed\n> to assume \"myers\" is default for a good reason, and thus asked for an\n> option to use something else. I'm not sure the assumption is valid; \n\nJust on the \"wider problem\", I've also looked at a lot of \"bad diffs\"\nand there's interesting cases where histogram does equally bad as the\nothers *from most user's POV*, but from a \"let's make a small diff\" for\na computer it's perfect.\n\nI've seen this most commonly with repetative file formats, e.g JSON-like\nones are a good example. You've probably looked at this exact thing N\ntimes, but in case it's useful consider:\n\t\n\t$ cat a\n\t{\n\t thing => 'foo',\n\t a => 'b',\n\t},\n\t{\n\t c => 'd',\n\t thing => 'bar',\n\t}\n\t$ cat b\n\t{\n\t thing => 'foo',\n\t a => 'b',\n\t},\n\t{\n\t c => 'd',\n\t},\n\t{\n\t e => 'f',\n\t thing => 'bar',\n\t}\n\nHere all of our diff algorithms generate this equally terrible diff, or\na great diff, depending on your POV :):\n\t\n\tdiff --git a/a b/b\n\tindex 186017c1f38..5afadde1800 100644\n\t--- a/a\n\t+++ b/b\n\t@@ -4,5 +4,8 @@\n\t },\n\t {\n\t c => 'd',\n\t+},\n\t+{\n\t+ e => 'f',\n\t thing => 'bar',\n\t }\n\nThe problem here is that from the user's POV they didn't add a closing\nbracket, comma, open bracket etc. They added a whole new copy/pasted\nblock, and then modified a key-value in the subsequent one.\n\nBut the diff engine doesn't know about any of that, and will \"helpfully\"\nproceed to \"steal\" parts of the previous block.\n\nAll of the cases where users have asked me (in person) about some bad\ndiffs have pretty much come down to this sort of thing.\n\nIn those cases one of the algorithms sometimes *happened* to find a\n\"better\" diff, but in my experience it's been a\nwrong-clock-is-right-twice-a-day sort of thing.\n\nI've wondered if we couldn't have some much more stupid but effective\nsolution to these common cases. All of the ones I remember could\nbasically be expressed as a hypothetical:\n\n\t[diff \"c\"]\n balanceBrackets = true\n\nWhere we'd try as hard as we could not to produce diffs that had\nun-balanced brackets. I.e. in this case (I manually produced this by\nconverting the brackets[1] to []'s, then changing them back:\n\t\n\tdiff --git a/a b/b\n\tindex 186017c1f38..05cdd03bfa4 100644\n\t--- a/a\n\t+++ b/b\n\t@@ -2,7 +2,10 @@\n\t thing => 'foo',\n\t a => 'b',\n\t },\n\t+{\n\t+ x => 'y',\n\t+},\n\t {\n\t- c => 'd',\n\t+ c => 'f',\n\t thing => 'bar',\n\t }\n\nThat's a much worse diff to a computer (now 11 lines, v.s. 8 lines\nbefore), but I'd think to most users that's *much* more understandable,\njust by knowing just a bit about the language (although I'd argue we\ncould go as far as assuming this in general, with how common balanced\nbrackets[1] are across languages).\n\nP.S.: Funny story: at I once helped a user who'd been\n struggling for hours to turn their already working change into\n something that made more sense with \"git diff\".\n\n We eventually managed to come up with something that looked\n \"right\", I can't remember how, probably some mixture of -U,\n diff algorithm etc.\n\n Their next question was \"Ok, so how do I commit this?\", referring\n to \"this particular version of the diff\".\n\n Which, having already spent more time than I'd like to admit in\n trying to \"help\" them was a good reminder to first ask what\n problem we're trying to solve :)\n\n1. By \"balanced bracket\" I'm referring not just to \"{}\", but what's\n considered a \"mirrored\" character in Unicode. I.e. not just ()[]{}<>\n etc., but also ∈∋ and the like (see\n e.g. https://www.compart.com/en/unicode/U+2208)\n\n For better or worse the Perl 6 language has this as part of its\n grammar, see e.g.:\n https://andrewshitov.com/2018/01/23/embedded-comment-delimiters-in-perl-6/\n"},{"id":"471860","messageId":"1ddac91b-7552-3e1e-9888-9e21e808104d@dunelm.org.uk","threadId":"59197","inReplyTo":"CABPp-BHhhUhRqn=kKcDiV3EMckBSk2EE8TKZ-PoeqTsKWuvAng@mail.gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-02-09T14:44:15Z","receivedAt":"2023-02-09T14:44:22Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Elijah\n\nOn 09/02/2023 09:09, Elijah Newren wrote:\n> Hi John and Phillip,\n> \n> On Tue, Feb 7, 2023 at 9:05 AM John Cai wrote:\n>>\n> [...]\n>>> Perhaps I'm over simplifying but having read the issue you linked to I couldn't help feeling that the majority of users might be satisfied by just changing gitlab to use the patience algorithm when generating diffs.\n>>\n>> Right, I recognize this is a judgment call that may be best left up to the list.\n>>\n>> We don't have a way in GitLab to change the diff algorithm currently. Of course\n>> that can be implemented outside of Git,\n> \n> Well, the below doesn't allow users to make diffs better for\n> *individual* files of interest, but if you agree with me that we\n> should just make diffs better for all users automatically, it's a\n> two-line change in git.git that I'd love to eventually convince the\n> project to take (though obviously doing that would also require some\n> documentation changes and some good messaging in release notes and\n> whatnot). I've used it for a good long while, and had a few dozen\n> users using this patch too, all without complaint:\n\nI'd support a change to either patience or histogram as the default \nalgorithm. My personal preference would be for the patience algorithm as \nI think it generally gives nicer diffs in the cases that the two \ndisagree (see below, I've tried changing diff.algorithm to histogram a \nfew times and I always end up changing it back to patience pretty \nquickly). However I can see there is an advantage in having \"diff\" and \n\"merge\" use the same algorithm as users who diffing either side to the \nmerge base will see the same diff that the merge is using. The histogram \nalgorithm is known to produce sub-optimal diffs in certain cases[1] but \nI'm not sure how much worse it is in that respect than any of the other \nalgorithms.\n\nTo see the differences between the output of patience and histogram \nalgorithms I diffed the output of \"git log -p --no-merges \n--diff-algorithm=patience\" and \"git log -p --no-merges \n--diff-algorithm=histogram\". The first three differences are\n\n- 6c065f72b8 (http: support CURLOPT_PROTOCOLS_STR, 2023-01-16)\n In get_curl_allowed_protocols() the patience algorithm shows the\n change in the return statement more clearly\n\n- 47cfc9bd7d (attr: add flag `--source` to work with tree-ish, 2023-01-14)\n The histogram algorithm shows read_attr_from_index() being moved\n whereas the patience algorithm does not making the diff easier to\n follow.\n\n- b0226007f0 (fsmonitor: eliminate call to deprecated FSEventStream \nfunction, 2022-12-14)\n In fsm_listen__stop_async() the histogram algorithm shows\n data->shutdown_style = SHUTDOWN_EVENT;\n being moved, which is not as clear as the patience output which\n shows it as a context line.\n\nI think there is a degree of personal preference when it comes to which \nout of patience or histogram is best and the user can easily select \ntheir preferred algorithm so I'd be happy with either.\n\nBest Wishes\n\nPhillip\n\n[1] \nhttps://lore.kernel.org/git/CAGZ79kZYO6hHiAM8Sfp3J=VX11c=0-7YDSx3_EAKt5-uvvt-Ew@mail.gmail.com/\n\n> diff --git a/diff.c b/diff.c\n> index 329eebf16a..77a46d5b7d 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -55,7 +55,7 @@ static int diff_relative;\n> static int diff_stat_graph_width;\n> static int diff_dirstat_permille_default = 30;\n> static struct diff_options default_diff_options;\n> -static long diff_algorithm;\n> +static long diff_algorithm = XDF_HISTOGRAM_DIFF;\n> static unsigned ws_error_highlight_default = WSEH_NEW;\n> \n> static char diff_colors[][COLOR_MAXLEN] = {\n> diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\n> index b298f220e0..2f663eab72 100755\n> --- a/t/t4015-diff-whitespace.sh\n> +++ b/t/t4015-diff-whitespace.sh\n> @@ -1549,7 +1549,7 @@ test_expect_success 'short lines of opposite\n> sign do not get marked as moved' '\n> this line should be marked as oldMoved newMoved\n> unchanged 4\n> EOF\n> - test_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n> + test_expect_code 1 git diff --diff-algorithm=myers --no-index\n> --color --color-moved=zebra \\\n> old.txt new.txt >output && cat output &&\n> grep -v index output | test_decode_color >actual &&\n> cat >expect <<-\\EOF &&\n> \n> \n> I used histogram above rather than patience, since (a) it's what git's\n> merge backend uses, (b) it produces roughly similar results to\n> patience from a user perspective, (c) past testing has shown it to be\n> somewhat faster than patience, and (d) we've potentially got some\n> leads in how to speed up our histogram implementation from the README\n> over at https://github.com/pascalkuthe/imara-diff. But, if you really\n> wanted to use patience as the default, it'd also be an easy tweak.\n> \n> Anyway, just some food for thought.\n"},{"id":"471865","messageId":"20230209163400.xi7j3ivd4k6gqoc7@jcai-ubuntu","threadId":"59197","inReplyTo":"Y+KQtqNPews3vBS8@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-09T16:34:00Z","receivedAt":"2023-02-09T16:34:07Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"On 23/02/07 12:56PM, Jeff King wrote:\n> On Sun, Feb 05, 2023 at 03:46:21AM +0000, John Cai via GitGitGadget wrote:\n> \n> > +`diff-algorithm`\n> > +^^^^^^^^^^^^^^^^\n> > +\n> > +The attribute `diff-algorithm` affects which algorithm Git uses to generate\n> > +diffs. This allows defining diff algorithms per file extension. Precedence rules\n> > +are as follows, in order from highest to lowest:\n> > +\n> > +*Command line option*\n> > +\n> > +Pass in the `--diff-algorithm` command line option int git-diff(1)\n> > +\n> > +*Git attributes*\n> > +\n> > +------------------------\n> > +*.json\tdiff-algorithm=histogram\n> > +------------------------\n> > +\n> > +*Git config*\n> > +\n> > +----------------------------------------------------------------\n> > +[diff]\n> > +\talgorithm = histogram\n> > +----------------------------------------------------------------\n> \n> From the user's perspective, this is weirdly inconsistent with the\n> existing diff attributes, which would be more like:\n> \n> # in .gitattributes\n> *.json diff=json \n> \n> # in config\n> [diff \"json\"]\n> algorithm = histogram\n\nThanks for this suggestion, Peff. What I like about this is that it builds off\nof the existing diff. scheme rather than inventing another one.\nAdditionally, we won't get hit with a performance penalty since we already read\ngitattrbitues to see if a driver has been set or not.\n\nThinking out loud, if we add \"algorithm\" as a key for diff., it would be\nmutually exclusive with \"command\" where \"command\" takes precedence, correct?\n\n> \n> I know why one might choose the scheme you did; it kicks in if the repo\n> sets the algorithm, without users having to set up any extra config.\n> Which is sort of nice, if we assume that malicious actors don't have any\n> incentive to pick the algorithm. In theory they don't, though I saw Ævar\n> mention possible DoS elsewhere in the thread.\n> \n> Side note: It's also possible that algorithm selection could be\n> required to trigger a separate security bug (say, a buffer overflow in\n> the patience code or something), so restricting that works in a\n> belt-and-suspenders way. But that somehow feels like like the wrong\n> side of the paranoia-vs-feature line.\n> \n> So I dunno. I recognize that this scheme fulfills your immediate needs\n> better, but I fear that we'll be stuck with a weird split between \"diff\"\n> and \"diff-*\" attributes forever. In the long run, having a way for the\n> repo to say \"and here is some config I recommend to you\" would give you\n> the best of both, but that is a challenging topic that has been\n> discussed and punted on for many years.\n> \n> -Peff\n"},{"id":"471866","messageId":"20230209163725.hyx62qwveut4sj5x@jcai-ubuntu","threadId":"59197","inReplyTo":"CABPp-BGhuTyq_hrpMc+Ky3yt1UgO7DcAsgcYH15FK--QLdCsQw@mail.gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-09T16:37:25Z","receivedAt":"2023-02-09T16:37:31Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"On 23/02/09 12:26AM, Elijah Newren wrote:\n> Hi John,\n> \n> On Mon, Feb 6, 2023 at 12:02 PM John Cai wrote:\n> >\n> > Hi Phillip,\n> >\n> > On 6 Feb 2023, at 11:27, Phillip Wood wrote:\n> >\n> > > Hi John\n> > >\n> > > On 05/02/2023 03:46, John Cai via GitGitGadget wrote:\n> > >> From: John Cai \n> > >>\n> > >> It can be useful to specify diff algorithms per file type. For example,\n> > >> one may want to use the minimal diff algorithm for .json files, another\n> > >> for .c files, etc.\n> > >\n> > > Have you got any examples of why this is useful? I find myself occasionally changing the algorithm when the default gives a sub-optimal diff but I've not really noticed any pattern with respect to file types.\n> >\n> > At $DAYJOB, there has been a discussion and request for a feature like this [1].\n> > One use case that came up was to be able to set a different diff algorithm for\n> > .json files.\n> >\n> > 1. https://gitlab.com/gitlab-org/gitaly/-/issues/2591\n> \n> A couple points:\n> \n> First, there seems to be a misunderstanding in that issue. In\n> particular, the merge algorithm does call into the xdiff library to do\n> the three-way content merge of individual files, and when it does so,\n> it has to specify the diff algorithm (or take the default, currently\n> myers). merge-recursive allows the diff algorithm to be specified by\n> the user (there are\n> -Xdiff-algorithm={histogram,minimal,patience,myers} flags to\n> merge/rebase for it), while merge-ort uses histogram (though it uses\n> the same parser as merge-recursive and thus gets the variables set\n> from the -Xdiff-algorithm flag, it just ignores those values and\n> hardcodes histogram).\n> \n> Second, I also think the user request got converted to a particular\n> solution without looking at the wider problem space: The idea seemed\n> to assume \"myers\" is default for a good reason, and thus asked for an\n> option to use something else. I'm not sure the assumption is valid; I\n> think \"myers\" is default for historical reasons and histogram is\n> better not just for special Salesforce xml files, but code files too.\n> The output makes more sense to users. So much so that even though my\n> simple testing suggested it had a 2% performance penalty compared to\n> myers, I forced ort to use it[1] even though I designed everything\n> else in that algorithm around eking out maximum performance. Others\n> who have tested the diff algorithms have also found histogram has very\n> similar performance to myers, and oftentimes even beats it[2][3].\n> Also, worries about invalidating rerere caches[4] was real, but we\n> already paid that price when we switched to ort. And if performance\n> is still a worry, [3] gives me reason to believe we can make our\n> histogram implementation faster. Finally, for the period of time when\n> Palantir was letting me make an internal git distribution (mostly for\n> testing ort), I also carried a patch that changed the default diff\n> algorithm to histogram (not just for ort, but for diff/log/etc. as\n> well). Never had any complaints from the users from it. Perhaps you\n> could do the same in your local version of git used by gitaly?\n\nThanks for that suggestion, Elijah. Changing the diff alg to histogram has been\nsomething we've considered doing as well. However, we've gotten customer\nrequests to also have the option to view their diffs with patience. Seeing as\nthe diff algorithm is sometimes a matter of preference, we thought giving users\nthe control through the repository would be nice.\n\nthanks\nJohn\n\n> \n> [1] See c8017176ac (\"merge-ort: use histogram diff\", 2020-12-13)\n> [2] From 85551232b5 (\"perf: compare diff algorithms\", 2012-03-06):\n> \"This does indeed show that histogram diff slightly beats Myers, while\n> patience is much slower than the others.\"\n> [3] https://github.com/pascalkuthe/imara-diff\n> [4] https://lore.kernel.org/git/20120307114714.GA14990@sigill.intra.peff.net/\n"},{"id":"471901","messageId":"CABPp-BHQDS+AzWXtk9WV4HY2QZ8UdXrWJJDr-y6VPoLB6HuAfw@mail.gmail.com","threadId":"59197","inReplyTo":"1ddac91b-7552-3e1e-9888-9e21e808104d@dunelm.org.uk","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-02-10T09:57:53Z","receivedAt":"2023-02-10T09:58:15Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Phillip,\n\nOn Thu, Feb 9, 2023 at 6:44 AM Phillip Wood wrote:\n>\n> Hi Elijah\n>\n> On 09/02/2023 09:09, Elijah Newren wrote:\n> > Hi John and Phillip,\n> >\n> > On Tue, Feb 7, 2023 at 9:05 AM John Cai wrote:\n> >>\n> > [...]\n> >>> Perhaps I'm over simplifying but having read the issue you linked to I couldn't help feeling that the majority of users might be satisfied by just changing gitlab to use the patience algorithm when generating diffs.\n> >>\n> >> Right, I recognize this is a judgment call that may be best left up to the list.\n> >>\n> >> We don't have a way in GitLab to change the diff algorithm currently. Of course\n> >> that can be implemented outside of Git,\n> >\n> > Well, the below doesn't allow users to make diffs better for\n> > *individual* files of interest, but if you agree with me that we\n> > should just make diffs better for all users automatically, it's a\n> > two-line change in git.git that I'd love to eventually convince the\n> > project to take (though obviously doing that would also require some\n> > documentation changes and some good messaging in release notes and\n> > whatnot). I've used it for a good long while, and had a few dozen\n> > users using this patch too, all without complaint:\n>\n> I'd support a change to either patience or histogram as the default\n> algorithm. My personal preference would be for the patience algorithm as\n> I think it generally gives nicer diffs in the cases that the two\n> disagree (see below, I've tried changing diff.algorithm to histogram a\n> few times and I always end up changing it back to patience pretty\n> quickly). However I can see there is an advantage in having \"diff\" and\n> \"merge\" use the same algorithm as users who diffing either side to the\n> merge base will see the same diff that the merge is using. The histogram\n> algorithm is known to produce sub-optimal diffs in certain cases[1] but\n> I'm not sure how much worse it is in that respect than any of the other\n> algorithms.\n[...]\n> [1]\n> https://lore.kernel.org/git/CAGZ79kZYO6hHiAM8Sfp3J=VX11c=0-7YDSx3_EAKt5-uvvt-Ew@mail.gmail.com/\n\nThanks, I might have a fix, though I'm a bit worried my tweaks might\ntrigger issues elsewhere or cost a bit of performance; I'll need to\ntest. Are there any other good known testcases where histogram\nproduces sub-optimal diffs?\n\n> To see the differences between the output of patience and histogram\n> algorithms I diffed the output of \"git log -p --no-merges\n> --diff-algorithm=patience\" and \"git log -p --no-merges\n> --diff-algorithm=histogram\". The first three differences are\n>\n> - 6c065f72b8 (http: support CURLOPT_PROTOCOLS_STR, 2023-01-16)\n> In get_curl_allowed_protocols() the patience algorithm shows the\n> change in the return statement more clearly\n>\n> - 47cfc9bd7d (attr: add flag `--source` to work with tree-ish, 2023-01-14)\n> The histogram algorithm shows read_attr_from_index() being moved\n> whereas the patience algorithm does not making the diff easier to\n> follow.\n>\n> - b0226007f0 (fsmonitor: eliminate call to deprecated FSEventStream\n> function, 2022-12-14)\n> In fsm_listen__stop_async() the histogram algorithm shows\n> data->shutdown_style = SHUTDOWN_EVENT;\n> being moved, which is not as clear as the patience output which\n> shows it as a context line.\n\nIf my current changes are \"good\", then they also remove the\ndifferences between patience and histogram for the second and third\ncommits above. (And the differences between the two algorithms for\nthe first commit look really minor.)\n\n> I think there is a degree of personal preference when it comes to which\n> out of patience or histogram is best and the user can easily select\n> their preferred algorithm so I'd be happy with either.\n\n:-)\n"},{"id":"471955","messageId":"Y+bxye2zOCMAEJhN@coredump.intra.peff.net","threadId":"59197","inReplyTo":"20230209163400.xi7j3ivd4k6gqoc7@jcai-ubuntu","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-11T01:39:21Z","receivedAt":"2023-02-11T01:39:26Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 09, 2023 at 11:34:00AM -0500, John Cai wrote:\n\n> > From the user's perspective, this is weirdly inconsistent with the\n> > existing diff attributes, which would be more like:\n> > \n> > # in .gitattributes\n> > *.json diff=json \n> > \n> > # in config\n> > [diff \"json\"]\n> > algorithm = histogram\n> \n> Thanks for this suggestion, Peff. What I like about this is that it builds off\n> of the existing diff. scheme rather than inventing another one.\n> Additionally, we won't get hit with a performance penalty since we already read\n> gitattrbitues to see if a driver has been set or not.\n> \n> Thinking out loud, if we add \"algorithm\" as a key for diff., it would be\n> mutually exclusive with \"command\" where \"command\" takes precedence, correct?\n\nYes. I think the documentation would be something like \"When generating\na diff internally, use to do so.\" And external diffs\nobviously skip that code path internally.\n\nI didn't check whether your patch does this or not, but should this\nfeature (however it is engaged) apply to diff-stats, too? I think\n--patience, etc, does, which makes sense. But that might be something\nworth elaborating in the description, too.\n\n-Peff\n"},{"id":"471958","messageId":"Y+b2l4Le2gTxGwO8@coredump.intra.peff.net","threadId":"59197","inReplyTo":"1ddac91b-7552-3e1e-9888-9e21e808104d@dunelm.org.uk","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-11T01:59:51Z","receivedAt":"2023-02-11T01:59:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 09, 2023 at 02:44:15PM +0000, Phillip Wood wrote:\n\n> To see the differences between the output of patience and histogram\n> algorithms I diffed the output of \"git log -p --no-merges\n> --diff-algorithm=patience\" and \"git log -p --no-merges\n> --diff-algorithm=histogram\". The first three differences are\n> \n> - 6c065f72b8 (http: support CURLOPT_PROTOCOLS_STR, 2023-01-16)\n> In get_curl_allowed_protocols() the patience algorithm shows the\n> change in the return statement more clearly\n> \n> - 47cfc9bd7d (attr: add flag `--source` to work with tree-ish, 2023-01-14)\n> The histogram algorithm shows read_attr_from_index() being moved\n> whereas the patience algorithm does not making the diff easier to\n> follow.\n> \n> - b0226007f0 (fsmonitor: eliminate call to deprecated FSEventStream\n> function, 2022-12-14)\n> In fsm_listen__stop_async() the histogram algorithm shows\n> data->shutdown_style = SHUTDOWN_EVENT;\n> being moved, which is not as clear as the patience output which\n> shows it as a context line.\n\nJust a small counter-point, since I happened to be looking at myers vs\npatience for something elsewhere in the thread, but:\n\n git show 35bd13fcd2caa4185bf3729655ca20b6a5fe9b6f builtin/add.c\n\nlooks slightly better to me with myers, even though it is 2 lines\nlonger. The issue is that patience and histogram are very eager to use\nblank lines as anchor points, so a diff like:\n\n -some words\n -\n -and some more\n +unrelated content\n +\n +but it happens to also be two paragraphs\n\nin myers becomes:\n\n -some words\n +unrelated content\n \n -and some more\n +but it happens to also be two paragraphs\n\nin patience (here I'm using single lines, but in practice these may be\nparagraphs, or stanzas of code). I think that's also the _strength_ of\npatience in many cases, but it really depends on the content. Replacing\na multi-stanza block with another one may be the best explanation for\nwhat happened. Or the two stanzas may be independent, and showing the\nchange for each one may be better.\n\nI'm not sure which one happens more often. And you'd probably want to\nweight it by how good/bad the change is. In the example I showed I don't\nfind patience very much worse, since it's already a pretty ugly diff.\nBut in cases where patience shines, it may be making things\nsignificantly more readable.\n\nI don't have a super strong opinion, but I just wanted to chime in that\nit is not clear to me that patience/histogram is always a win over myers\n(yes, I know your examples were comparing patience vs histogram, but the\nlarger thread is discussing the other).\n\n-Peff\n"},{"id":"471959","messageId":"Y+b3opa1xgnGM+VN@coredump.intra.peff.net","threadId":"59197","inReplyTo":"230209.86lel7xi8l.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-11T02:04:18Z","receivedAt":"2023-02-11T02:04:22Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 09, 2023 at 10:41:33AM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> > If you run with more than 1 run, are your numbers even repeatable?\n> \n> Yes, but tl;dr it's diff.colorMoved=true, sorry, see below.\n\nWow, that's really slow. I was slightly surprised, because I also use\ncolorMoved. But I set it to \"plain\", which is way faster:\n\n $ time git diff --color-moved=default v2.0.0 v2.28.0 >/dev/null\n real\t0m18.492s\n user\t0m18.411s\n sys\t0m0.081s\n\n $ time git diff --color-moved=plain v2.0.0 v2.28.0 >/dev/null\n real\t0m0.942s\n user\t0m0.841s\n sys\t0m0.101s\n\nI didn't dig into why, but it's possible there's some low-hanging fruit\nin the zebra/block code.\n\nI also have a mild feeling of deja vu that we may have discussed this\nbefore, but a quick search in the archive didn't yield anything. So I'll\nleave it for somebody to investigate further if they're interested.\n\n-Peff\n"},{"id":"471983","messageId":"37a89e67-23f0-83f7-aaa5-0199d5db3adb@dunelm.org.uk","threadId":"59197","inReplyTo":"CABPp-BHQDS+AzWXtk9WV4HY2QZ8UdXrWJJDr-y6VPoLB6HuAfw@mail.gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-02-11T17:39:38Z","receivedAt":"2023-02-11T17:39:47Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Elijah\n\nOn 10/02/2023 09:57, Elijah Newren wrote:\n> Hi Phillip,\n> \n> On Thu, Feb 9, 2023 at 6:44 AM Phillip Wood wrote:\n>>\n>> Hi Elijah\n>>\n>> On 09/02/2023 09:09, Elijah Newren wrote:\n>> I'd support a change to either patience or histogram as the default\n>> algorithm. My personal preference would be for the patience algorithm as\n>> I think it generally gives nicer diffs in the cases that the two\n>> disagree (see below, I've tried changing diff.algorithm to histogram a\n>> few times and I always end up changing it back to patience pretty\n>> quickly). However I can see there is an advantage in having \"diff\" and\n>> \"merge\" use the same algorithm as users who diffing either side to the\n>> merge base will see the same diff that the merge is using. The histogram\n>> algorithm is known to produce sub-optimal diffs in certain cases[1] but\n>> I'm not sure how much worse it is in that respect than any of the other\n>> algorithms.\n> [...]\n>> [1]\n>> https://lore.kernel.org/git/CAGZ79kZYO6hHiAM8Sfp3J=VX11c=0-7YDSx3_EAKt5-uvvt-Ew@mail.gmail.com/\n> \n> Thanks, I might have a fix, though I'm a bit worried my tweaks might\n> trigger issues elsewhere or cost a bit of performance; I'll need to\n> test. Are there any other good known testcases where histogram\n> produces sub-optimal diffs?\n\nNot that I'm aware of (I've a feeling there might have been something on \nthe JGit mailing list but I only managed to find a copy of Stefan's \nmessage). Loosely related is [1] which talks about hash collisions and \nI've never found the time to look at properly. I suspect any hash \ncollision problem is more likely to affect xdl_classify_record() which \nis used by all the algorithms.\n\n>> To see the differences between the output of patience and histogram\n>> algorithms I diffed the output of \"git log -p --no-merges\n>> --diff-algorithm=patience\" and \"git log -p --no-merges\n>> --diff-algorithm=histogram\". The first three differences are\n>>\n>> - 6c065f72b8 (http: support CURLOPT_PROTOCOLS_STR, 2023-01-16)\n>> In get_curl_allowed_protocols() the patience algorithm shows the\n>> change in the return statement more clearly\n>>\n>> - 47cfc9bd7d (attr: add flag `--source` to work with tree-ish, 2023-01-14)\n>> The histogram algorithm shows read_attr_from_index() being moved\n>> whereas the patience algorithm does not making the diff easier to\n>> follow.\n>>\n>> - b0226007f0 (fsmonitor: eliminate call to deprecated FSEventStream\n>> function, 2022-12-14)\n>> In fsm_listen__stop_async() the histogram algorithm shows\n>> data->shutdown_style = SHUTDOWN_EVENT;\n>> being moved, which is not as clear as the patience output which\n>> shows it as a context line.\n> \n> If my current changes are \"good\", then they also remove the\n> differences between patience and histogram for the second and third\n> commits above. (And the differences between the two algorithms for\n> the first commit look really minor.)\n\nInteresting. I agree the differences for the first commit are small. \nInterestingly I think they come from patience algorithm falling back to \nthe myers implementation because it cannot find any unique context lines \n(I have a patch that removes the fallback[2] and it gives the same \nresult as the histogram implementation).\n\nBest Wishes\n\nPhillip\n\n[1] \nhttps://lore.kernel.org/git/4e0eff48-4a3e-4f0e-9ed2-d01ec38442a5@www.fastmail.com/\n\n[2] https://github.com/phillipwood/git/commits/pure-patience-diff\n"},{"id":"472076","messageId":"AF5092D2-A561-4B56-8FB8-25DCFA28F32C@gmail.com","threadId":"59197","inReplyTo":"CABPp-BFKQXe-EJOd9z1TrisL64NuV9A132rf9MwV_7w79QQ9YQ@mail.gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-14T21:16:32Z","receivedAt":"2023-02-14T21:17:18Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Elijah,\n\nOn 9 Feb 2023, at 3:44, Elijah Newren wrote:\n\n> Hi John,\n>\n> On Mon, Feb 6, 2023 at 12:47 PM John Cai wrote:\n>>\n> [...]\n>> That being said, here's a separate issue. I benchmarked the usage of\n>> .gitattributes as introduced in this patch series, and indeed it does look like\n>> there is additional latency:\n>>\n>> $ echo \"* diff-algorithm=patience >> .gitattributes\n>> $ hyperfine -r 5 'git-bin-wrapper diff --diff-algorithm=patience v2.0.0 v2.28.0' ✭\n>> Benchmark 1: git-bin-wrapper diff --diff-algorithm=patience v2.0.0 v2.28.0\n>> Time (mean ± σ): 889.4 ms ± 113.8 ms [User: 715.7 ms, System: 65.3 ms]\n>> Range (min … max): 764.1 ms … 1029.3 ms 5 runs\n>>\n>> $ hyperfine -r 5 'git-bin-wrapper diff v2.0.0 v2.28.0' ✭\n>> Benchmark 1: git-bin-wrapper diff v2.0.0 v2.28.0\n>> Time (mean ± σ): 2.146 s ± 0.368 s [User: 0.827 s, System: 0.243 s]\n>> Range (min … max): 1.883 s … 2.795 s 5 runs\n>>\n>> and I imagine the latency scales with the size of .gitattributes. Although I'm\n>> not familiar with other parts of the codebase and how it deals with the latency\n>> introduced by reading attributes files.\n>\n> Yeah, that seems like a large relative performance penalty. I had the\n> feeling that histogram wasn't made the default over myers mostly due\n> to inertia and due to a potential 2% loss in performance (since\n> potentially corrected by Phillip's 663c5ad035 (\"diff histogram: intern\n> strings\", 2021-11-17)). If we had changed the default diff algorithm\n> to histogram, I suspect folks wouldn't have been asking for per-file\n> knobs to use a better diff algorithm. And the performance penalty for\n> this alternative is clearly much larger than 2%, which makes me think\n> we might want to just revisit the default instead of allowing per-file\n> tweaks.\n\nIt seems like the performance penalty was because I was adding calls to parse\nattribute files. Piggy backing off of the attribute parsing in userdiff.h will\nallow us to not incur this performance penalty:\n\n$ hyperfine -r 5 -L a bin-wrappers/git,git '{a} diff v2.0.0 v2.28.0'\nBenchmark 1: git-bin-wrapper diff v2.0.0 v2.28.0\n Time (mean ± σ): 1.072 s ± 0.289 s [User: 0.626 s, System: 0.081 s]\n Range (min … max): 0.772 s … 1.537 s 5 runs\n\nBenchmark 2: git diff v2.0.0 v2.28.0\n Time (mean ± σ): 1.003 s ± 0.065 s [User: 0.684 s, System: 0.067 s]\n Range (min … max): 0.914 s … 1.091 s 5 runs\n\nSummary\n 'git diff v2.0.0 v2.28.0' ran\n 1.07 ± 0.30 times faster than 'git-bin-wrapper diff v2.0.0 v2.28.0'\n\n>\n> And on a separate note...\n>\n> There's another set of considerations we might need to include here as\n> well that I haven't seen anyone else in this thread talk about:\n\nThese are some great questions. I'll do my best to answer them.\n>\n> * When trying to diff files, do we read the .gitattributes file from\n> the current checkout to determine the diff algorithm(s)? Or the\n> index? Or the commit we are diffing against?\n> * If we use the current checkout or index, what about bare clones or\n> diffing between two different commits?\n> * If diffing between two different commits, and the .gitattributes has\n> changed between those commits, which .gitattributes file wins?\n> * If diffing between two different commits, and the .gitattributes has\n> NOT changed, BUT a file has been renamed and the old and new names\n> have different rules, which rule wins?\n\nIn the next version I plan on using Peff's suggestion of utilizing the existing\ndiff driver scheme [1]. I believe these four questions are addressed if we use\nthe existing userdiff.h API, which in turn calls the attr.h API. We check the\nworktree, then fallback to the index.\n\nBy using the userdiff.h API, the behavior will match what users already expect\nwhen they for instance set an external driver.\n\n1. https://lore.kernel.org/git/Y+KQtqNPews3vBS8@coredump.intra.peff.net/\n\n>\n> * If per-file diff algorithms are adopted widely enough, will we be\n> forced to change the merge algorithm to also pay attention to them?\n> If it does, more complicated rename cases occur and we need rules for\n> how to handle those.\n> * If the merge algorithm has to pay attention to .gitattributes for\n> this too, we'll have even more corner cases around what happens if\n> there are merge conflicts in .gitattributes itself (which is already\n> kind of ugly and kludged)\n\nI see this feature as a user-experience type convenience feature, so I don't\nbelieve there's need for the merge machinery to also pay attention to the diff\nalgorithm set through gitattrbutes. We can clarify this in the documentation.\n\n>\n>\n> Anyway, I know I'm a bit animated and biased in this area, and I\n> apologize if I'm a bit too much so. Even if I am, hopefully my\n> comments at least provide some useful context.\n\nNo problem! thanks for raising these issues.\n\nthanks\nJohn\n"},{"id":"472079","messageId":"pull.1452.v2.git.git.1676410819.gitgitgadget@gmail.com","threadId":"59197","inReplyTo":"pull.1452.git.git.1675568781.gitgitgadget@gmail.com","subject":"[PATCH v2 0/2] Teach diff to honor diff algorithms set through git attributes","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-14T21:40:17Z","receivedAt":"2023-02-14T21:40:30Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"When a repository contains different kinds of files, it may be desirable to\nuse different algorithms based on file type. This is currently not feasible\nthrough the command line or using git configs. However, we can leverage the\nfact that gitattributes are path aware.\n\nTeach the diff machinery to check gitattributes when diffing files.\n\nChanges since V1:\n\n * utilize the existing diff..* scheme where the driver is defined\n in gitattributes, but the algorithm is defined in the gitconfig.\n\nTo address some of the performance concerns in the previous series, a\nbenchmark shows that a performance penalty is no longer incurred, now that\nwe are no longer adding an additional attributes parsing call:\n\n$ hyperfine -r 5 -L a bin-wrappers/git,git '{a} diff v2.0.0 v2.28.0'\nBenchmark 1: git-bin-wrapper diff v2.0.0 v2.28.0 Time (mean ± σ): 1.072 s ±\n0.289 s [User: 0.626 s, System: 0.081 s] Range (min … max): 0.772 s … 1.537\ns 5 runs\n\nBenchmark 2: git diff v2.0.0 v2.28.0 Time (mean ± σ): 1.003 s ± 0.065 s\n[User: 0.684 s, System: 0.067 s] Range (min … max): 0.914 s … 1.091 s 5 runs\n\nSummary 'git diff v2.0.0 v2.28.0' ran 1.07 ± 0.30 times faster than\n'git-bin-wrapper diff v2.0.0 v2.28.0'\n\nJohn Cai (2):\n diff: consolidate diff algorithm option parsing\n diff: teach diff to read gitattribute diff-algorithm\n\n Documentation/gitattributes.txt | 41 +++++++++++-\n diff.c | 112 ++++++++++++++++++++------------\n diff.h | 2 +\n t/lib-diff-alternative.sh | 38 ++++++++++-\n userdiff.c | 4 +-\n userdiff.h | 1 +\n 6 files changed, 154 insertions(+), 44 deletions(-)\n\n\nbase-commit: c867e4fa180bec4750e9b54eb10f459030dbebfd\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1452%2Fjohn-cai%2Fjc%2Fattr-diff-algo-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1452/john-cai/jc/attr-diff-algo-v2\nPull-Request: https://github.com/git/git/pull/1452\n\nRange-diff vs v1:\n\n 1: f56bd38ac3f ! 1: 0c5e1fc6c26 diff: consolidate diff algorithm option parsing\n @@\n ## Metadata ##\n -Author: John Cai \n +Author: John Cai \n \n ## Commit message ##\n diff: consolidate diff algorithm option parsing\n @@ Commit message\n subsequent commit which teaches diff to keep track of whether or not a\n diff algorithm has been set via the command line.\n \n - While we're at it, the logic that sets the diff algorithm in\n - diff_opt_diff_algorithm() can be refactored into a helper that will\n + Additionally, the logic that sets the diff algorithm in\n + diff_opt_diff_algorithm() can be refactored into a helper that will\n allow multiple callsites to set the diff algorithm.\n \n Signed-off-by: John Cai \n @@ diff.c: static int diff_opt_diff_algorithm(const struct option *opt,\n +\tBUG_ON_OPT_ARG(arg);\n +\n +\tif (!strcmp(opt->long_name, \"patience\")) {\n -+\t\tint i;\n ++\t\tsize_t i;\n +\t\t/*\n +\t\t * Both --patience and --anchored use PATIENCE_DIFF\n +\t\t * internally, so remove any anchors previously\n @@ diff.c: static int diff_opt_diff_algorithm(const struct option *opt,\n +\t}\n +\n +\tif (set_diff_algorithm(options, opt->long_name))\n -+\t\treturn error(_(\"available diff algorithms include \\\"myers\\\", \"\n -+\t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n ++\t\tBUG(\"available diff algorithms include \\\"myers\\\", \"\n ++\t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n +\n \treturn 0;\n }\n 2: 8e73793b0db ! 2: cb030563149 diff: teach diff to read gitattribute diff-algorithm\n @@ Commit message\n one may want to use the minimal diff algorithm for .json files, another\n for .c files, etc.\n \n - Teach the diff machinery to check attributes for a diff algorithm.\n - Enforce precedence by favoring the command line option, then looking at\n - attributes, then finally the config.\n + Teach the diff machinery to check attributes for a diff driver. Also\n + teach the diff driver parser a new type \"algorithm\" to look for in the\n + config, which will be used if a driver has been specified through the\n + attributes.\n \n - To enforce precedence order, set the `xdl_opts_command_line` member\n + Enforce precedence of diff algorithm by favoring the command line option,\n + then looking at the driver attributes & config combination, then finally\n + the diff.algorithm config.\n +\n + To enforce precedence order, use the `xdl_opts_command_line` member\n during options pasing to indicate the diff algorithm was set via command\n line args.\n \n @@ Documentation/gitattributes.txt: String::\n \tby the configuration variables in the \"diff.foo\" section of the\n \tGit config file.\n \n -+`diff-algorithm`\n -+^^^^^^^^^^^^^^^^\n -+\n -+The attribute `diff-algorithm` affects which algorithm Git uses to generate\n -+diffs. This allows defining diff algorithms per file extension. Precedence rules\n -+are as follows, in order from highest to lowest:\n +-\n + Defining an external diff driver\n + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n + \n +@@ Documentation/gitattributes.txt: with the above configuration, i.e. `j-c-diff`, with 7\n + parameters, just like `GIT_EXTERNAL_DIFF` program is called.\n + See linkgit:git[1] for details.\n + \n ++Setting the internal diff algorithm\n ++^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n +\n -+*Command line option*\n ++The diff algorithm can be set through the `diff.algorithm` config key, but\n ++sometimes it may be helpful to set the diff algorithm by path. For example, one\n ++might wish to set a diff algorithm automatically for all `.json` files such that\n ++the user would not need to pass in a separate command line `--diff-algorithm` flag each\n ++time.\n +\n -+Pass in the `--diff-algorithm` command line option int git-diff(1)\n ++First, in `.gitattributes`, you would assign the `diff` attribute for paths.\n +\n +*Git attributes*\n -+\n +------------------------\n -+*.json\tdiff-algorithm=histogram\n ++*.json diff=\n +------------------------\n +\n ++Then, you would define a \"diff..algorithm\" configuration to specify the\n ++diff algorithm, choosing from `meyers`, `patience`, `minimal`, and `histogram`.\n ++\n ++*Git config*\n ++\n ++----------------------------------------------------------------\n ++[diff \"\"]\n ++ algorithm = histogram\n ++----------------------------------------------------------------\n ++\n ++This diff algorithm applies to git-diff(1), including the `--stat` output.\n ++\n ++NOTE: If the `command` key also exists, then Git will treat this as an external\n ++diff and attempt to use the value set for `command` as an external program. For\n ++instance, the following config, combined with the above `.gitattributes` file,\n ++will result in `command` favored over `algorithm`.\n ++\n +*Git config*\n +\n +----------------------------------------------------------------\n -+[diff]\n -+\talgorithm = histogram\n ++[diff \"\"]\n ++ command = j-c-diff\n ++ algorithm = histogram\n +----------------------------------------------------------------\n \n - Defining an external diff driver\n - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n + Defining a custom hunk-header\n + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n \n ## diff.c ##\n -@@ diff.c: static void builtin_diff(const char *name_a,\n - \t\tecbdata.opt = o;\n - \t\tif (header.len && !o->flags.suppress_diff_headers)\n - \t\t\tecbdata.header = &header;\n -+\n -+\t\tif (!o->xdl_opts_command_line) {\n -+\t\t\tstatic struct attr_check *check;\n -+\t\t\tconst char *one_diff_algo;\n -+\t\t\tconst char *two_diff_algo;\n -+\n -+\t\t\tcheck = attr_check_alloc();\n -+\t\t\tattr_check_append(check, git_attr(\"diff-algorithm\"));\n -+\n -+\t\t\tgit_check_attr(the_repository->index, NULL, one->path, check);\n -+\t\t\tone_diff_algo = check->items[0].value;\n -+\t\t\tgit_check_attr(the_repository->index, NULL, two->path, check);\n -+\t\t\ttwo_diff_algo = check->items[0].value;\n -+\n -+\t\t\tif (!ATTR_UNSET(one_diff_algo) && !ATTR_UNSET(two_diff_algo) &&\n -+\t\t\t\t!strcmp(one_diff_algo, two_diff_algo))\n -+\t\t\t\tset_diff_algorithm(o, one_diff_algo);\n -+\n -+\t\t\tattr_check_free(check);\n -+\t\t}\n -+\n - \t\txpp.flags = o->xdl_opts;\n - \t\txpp.ignore_regex = o->ignore_regex;\n - \t\txpp.ignore_regex_nr = o->ignore_regex_nr;\n +@@ diff.c: static void run_diff_cmd(const char *pgm,\n + \tconst char *xfrm_msg = NULL;\n + \tint complete_rewrite = (p->status == DIFF_STATUS_MODIFIED) && p->score;\n + \tint must_show_header = 0;\n ++\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, attr_path);\n + \n +-\n +-\tif (o->flags.allow_external) {\n +-\t\tstruct userdiff_driver *drv;\n +-\n +-\t\tdrv = userdiff_find_by_path(o->repo->index, attr_path);\n ++\tif (o->flags.allow_external)\n + \t\tif (drv && drv->external)\n + \t\t\tpgm = drv->external;\n +-\t}\n + \n + \tif (msg) {\n + \t\t/*\n +@@ diff.c: static void run_diff_cmd(const char *pgm,\n + \t\trun_external_diff(pgm, name, other, one, two, xfrm_msg, o);\n + \t\treturn;\n + \t}\n +-\tif (one && two)\n ++\tif (one && two) {\n ++\t\tif (!o->xdl_opts_command_line)\n ++\t\t\tif (drv && drv->algorithm)\n ++\t\t\t\tset_diff_algorithm(o, drv->algorithm);\n ++\n + \t\tbuiltin_diff(name, other ? other : name,\n + \t\t\t one, two, xfrm_msg, must_show_header,\n + \t\t\t o, complete_rewrite);\n +-\telse\n ++\t} else {\n + \t\tfprintf(o->file, \"* Unmerged path %s\\n\", name);\n ++\t}\n + }\n + \n + static void diff_fill_oid_info(struct diff_filespec *one, struct index_state *istate)\n +@@ diff.c: static void run_diffstat(struct diff_filepair *p, struct diff_options *o,\n + \tconst char *name;\n + \tconst char *other;\n + \n ++\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, p->one->path);\n ++\tif (drv && drv->algorithm)\n ++\t\tset_diff_algorithm(o, drv->algorithm);\n ++\n + \tif (DIFF_PAIR_UNMERGED(p)) {\n + \t\t/* unmerged */\n + \t\tbuiltin_diffstat(p->one->path, NULL, NULL, NULL,\n @@ diff.c: static int diff_opt_diff_algorithm(const struct option *opt,\n \t\treturn error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n @@ diff.c: static int diff_opt_diff_algorithm(const struct option *opt,\n }\n \n @@ diff.c: static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n - \t\treturn error(_(\"available diff algorithms include \\\"myers\\\", \"\n - \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n + \t\tBUG(\"available diff algorithms include \\\"myers\\\", \"\n + \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n \n +\toptions->xdl_opts_command_line = 1;\n +\n @@ diff.h: struct diff_options {\n \tchar **anchors;\n \n ## t/lib-diff-alternative.sh ##\n -@@ t/lib-diff-alternative.sh: EOF\n +@@ t/lib-diff-alternative.sh: index $file1..$file2 100644\n + }\n + EOF\n \n ++\tcat >expect_diffstat < file2 | 21 ++++++++++-----------\n ++ 1 file changed, 10 insertions(+), 11 deletions(-)\n ++EOF\n ++\n \tSTRATEGY=$1\n \n +\ttest_expect_success \"$STRATEGY diff from attributes\" '\n -+\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n ++\t\techo \"file* diff=driver\" >.gitattributes &&\n ++\t\tgit config diff.driver.algorithm \"$STRATEGY\" &&\n +\t\ttest_must_fail git diff --no-index file1 file2 > output &&\n ++\t\tcat expect &&\n ++\t\tcat output &&\n +\t\ttest_cmp expect output\n +\t'\n ++\n ++\ttest_expect_success \"$STRATEGY diff from attributes has valid diffstat\" '\n ++\t\techo \"file* diff=driver\" >.gitattributes &&\n ++\t\tgit config diff.driver.algorithm \"$STRATEGY\" &&\n ++\t\ttest_must_fail git diff --stat --no-index file1 file2 > output &&\n ++\t\ttest_cmp expect_diffstat output\n ++\t'\n +\n \ttest_expect_success \"$STRATEGY diff\" '\n -\t\ttest_must_fail git diff --no-index \"--$STRATEGY\" file1 file2 > output &&\n @@ t/lib-diff-alternative.sh: EOF\n +\t'\n +\n +\ttest_expect_success \"$STRATEGY diff command line precedence before attributes\" '\n -+\t\techo \"file* diff-algorithm=meyers\" >.gitattributes &&\n ++\t\techo \"file* diff=driver\" >.gitattributes &&\n ++\t\tgit config diff.driver.algorithm meyers &&\n +\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n +\t\ttest_cmp expect output\n +\t'\n +\n +\ttest_expect_success \"$STRATEGY diff attributes precedence before config\" '\n +\t\tgit config diff.algorithm default &&\n -+\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n -+\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n ++\t\techo \"file* diff=driver\" >.gitattributes &&\n ++\t\tgit config diff.driver.algorithm \"$STRATEGY\" &&\n ++\t\ttest_must_fail git diff --no-index file1 file2 > output &&\n \t\ttest_cmp expect output\n \t'\n \n -@@ t/lib-diff-alternative.sh: EOF\n - \t\ttest_must_fail git diff --no-index \"--$STRATEGY\" uniq1 uniq2 > output &&\n - \t\ttest_cmp expect output\n - \t'\n -+\n -+\ttest_expect_success \"$STRATEGY diff from attributes\" '\n -+\t\techo \"file* diff-algorithm=$STRATEGY\" >.gitattributes &&\n -+\t\ttest_must_fail git diff --no-index uniq1 uniq2 > output &&\n -+\t\ttest_cmp expect output\n -+\t'\n - }\n +\n + ## userdiff.c ##\n +@@ userdiff.c: PATTERNS(\"scheme\",\n + \t \"|([^][)(}{[ \\t])+\"),\n + PATTERNS(\"tex\", \"^(\\\\\\\\((sub)*section|chapter|part)\\\\*{0,1}\\\\{.*)$\",\n + \t \"\\\\\\\\[a-zA-Z@]+|\\\\\\\\.|[a-zA-Z0-9\\x80-\\xff]+\"),\n +-{ \"default\", NULL, -1, { NULL, 0 } },\n ++{ \"default\", NULL, NULL, -1, { NULL, 0 } },\n + };\n + #undef PATTERNS\n + #undef IPATTERN\n +@@ userdiff.c: int userdiff_config(const char *k, const char *v)\n + \t\treturn parse_bool(&drv->textconv_want_cache, k, v);\n + \tif (!strcmp(type, \"wordregex\"))\n + \t\treturn git_config_string(&drv->word_regex, k, v);\n ++\tif (!strcmp(type, \"algorithm\"))\n ++\t\treturn git_config_string(&drv->algorithm, k, v);\n \n + \treturn 0;\n + }\n +\n + ## userdiff.h ##\n +@@ userdiff.h: struct userdiff_funcname {\n + struct userdiff_driver {\n + \tconst char *name;\n + \tconst char *external;\n ++\tconst char *algorithm;\n + \tint binary;\n + \tstruct userdiff_funcname funcname;\n + \tconst char *word_regex;\n\n-- \ngitgitgadget\n"},{"id":"472080","messageId":"0c5e1fc6c2651e39bcefa27ee0976c9519671969.1676410819.git.gitgitgadget@gmail.com","threadId":"59197","inReplyTo":"pull.1452.v2.git.git.1676410819.gitgitgadget@gmail.com","subject":"[PATCH v2 1/2] diff: consolidate diff algorithm option parsing","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-14T21:40:18Z","receivedAt":"2023-02-14T21:40:32Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nThe diff option parsing for --minimal, --patience, --histgoram can all\nbe consolidated into one function. This is a preparatory step for the\nsubsequent commit which teaches diff to keep track of whether or not a\ndiff algorithm has been set via the command line.\n\nAdditionally, the logic that sets the diff algorithm in\ndiff_opt_diff_algorithm() can be refactored into a helper that will\nallow multiple callsites to set the diff algorithm.\n\nSigned-off-by: John Cai \n---\n diff.c | 87 ++++++++++++++++++++++++++++++++++++----------------------\n 1 file changed, 54 insertions(+), 33 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 329eebf16a0..92a0eab942e 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -3437,6 +3437,22 @@ static int diff_filepair_is_phoney(struct diff_filespec *one,\n \treturn !DIFF_FILE_VALID(one) && !DIFF_FILE_VALID(two);\n }\n \n+static int set_diff_algorithm(struct diff_options *opts,\n+\t\t\t const char *alg)\n+{\n+\tlong value = parse_algorithm_value(alg);\n+\n+\tif (value < 0)\n+\t\treturn 1;\n+\n+\t/* clear out previous settings */\n+\tDIFF_XDL_CLR(opts, NEED_MINIMAL);\n+\topts->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n+\topts->xdl_opts |= value;\n+\n+\treturn 0;\n+}\n+\n static void builtin_diff(const char *name_a,\n \t\t\t const char *name_b,\n \t\t\t struct diff_filespec *one,\n@@ -5107,17 +5123,40 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct diff_options *options = opt->value;\n-\tlong value = parse_algorithm_value(arg);\n \n \tBUG_ON_OPT_NEG(unset);\n-\tif (value < 0)\n+\n+\tif (set_diff_algorithm(options, arg))\n \t\treturn error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n \n-\t/* clear out previous settings */\n-\tDIFF_XDL_CLR(options, NEED_MINIMAL);\n-\toptions->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n-\toptions->xdl_opts |= value;\n+\treturn 0;\n+}\n+\n+static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n+\t\t\t\t const char *arg, int unset)\n+{\n+\tstruct diff_options *options = opt->value;\n+\n+\tBUG_ON_OPT_NEG(unset);\n+\tBUG_ON_OPT_ARG(arg);\n+\n+\tif (!strcmp(opt->long_name, \"patience\")) {\n+\t\tsize_t i;\n+\t\t/*\n+\t\t * Both --patience and --anchored use PATIENCE_DIFF\n+\t\t * internally, so remove any anchors previously\n+\t\t * specified.\n+\t\t */\n+\t\tfor (i = 0; i < options->anchors_nr; i++)\n+\t\t\tfree(options->anchors[i]);\n+\t\toptions->anchors_nr = 0;\n+\t}\n+\n+\tif (set_diff_algorithm(options, opt->long_name))\n+\t\tBUG(\"available diff algorithms include \\\"myers\\\", \"\n+\t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n+\n \treturn 0;\n }\n \n@@ -5242,26 +5281,6 @@ static enum parse_opt_result diff_opt_output(struct parse_opt_ctx_t *ctx,\n \treturn 0;\n }\n \n-static int diff_opt_patience(const struct option *opt,\n-\t\t\t const char *arg, int unset)\n-{\n-\tstruct diff_options *options = opt->value;\n-\tint i;\n-\n-\tBUG_ON_OPT_NEG(unset);\n-\tBUG_ON_OPT_ARG(arg);\n-\toptions->xdl_opts = DIFF_WITH_ALG(options, PATIENCE_DIFF);\n-\t/*\n-\t * Both --patience and --anchored use PATIENCE_DIFF\n-\t * internally, so remove any anchors previously\n-\t * specified.\n-\t */\n-\tfor (i = 0; i < options->anchors_nr; i++)\n-\t\tfree(options->anchors[i]);\n-\toptions->anchors_nr = 0;\n-\treturn 0;\n-}\n-\n static int diff_opt_ignore_regex(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n@@ -5562,9 +5581,10 @@ struct option *add_diff_options(const struct option *opts,\n \t\t\t N_(\"prevent rename/copy detection if the number of rename/copy targets exceeds given limit\")),\n \n \t\tOPT_GROUP(N_(\"Diff algorithm options\")),\n-\t\tOPT_BIT(0, \"minimal\", &options->xdl_opts,\n-\t\t\tN_(\"produce the smallest possible diff\"),\n-\t\t\tXDF_NEED_MINIMAL),\n+\t\tOPT_CALLBACK_F(0, \"minimal\", options, NULL,\n+\t\t\t N_(\"produce the smallest possible diff\"),\n+\t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n+\t\t\t diff_opt_diff_algorithm_no_arg),\n \t\tOPT_BIT_F('w', \"ignore-all-space\", &options->xdl_opts,\n \t\t\t N_(\"ignore whitespace when comparing lines\"),\n \t\t\t XDF_IGNORE_WHITESPACE, PARSE_OPT_NONEG),\n@@ -5589,10 +5609,11 @@ struct option *add_diff_options(const struct option *opts,\n \t\tOPT_CALLBACK_F(0, \"patience\", options, NULL,\n \t\t\t N_(\"generate diff using the \\\"patience diff\\\" algorithm\"),\n \t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n-\t\t\t diff_opt_patience),\n-\t\tOPT_BITOP(0, \"histogram\", &options->xdl_opts,\n-\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n-\t\t\t XDF_HISTOGRAM_DIFF, XDF_DIFF_ALGORITHM_MASK),\n+\t\t\t diff_opt_diff_algorithm_no_arg),\n+\t\tOPT_CALLBACK_F(0, \"histogram\", options, NULL,\n+\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n+\t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n+\t\t\t diff_opt_diff_algorithm_no_arg),\n \t\tOPT_CALLBACK_F(0, \"diff-algorithm\", options, N_(\"\"),\n \t\t\t N_(\"choose a diff algorithm\"),\n \t\t\t PARSE_OPT_NONEG, diff_opt_diff_algorithm),\n-- \ngitgitgadget\n\n"},{"id":"472081","messageId":"cb0305631496eb4c2d51e5b586ac0ca8580c7dc1.1676410819.git.gitgitgadget@gmail.com","threadId":"59197","inReplyTo":"pull.1452.v2.git.git.1676410819.gitgitgadget@gmail.com","subject":"[PATCH v2 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-14T21:40:19Z","receivedAt":"2023-02-14T21:40:40Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nIt can be useful to specify diff algorithms per file type. For example,\none may want to use the minimal diff algorithm for .json files, another\nfor .c files, etc.\n\nTeach the diff machinery to check attributes for a diff driver. Also\nteach the diff driver parser a new type \"algorithm\" to look for in the\nconfig, which will be used if a driver has been specified through the\nattributes.\n\nEnforce precedence of diff algorithm by favoring the command line option,\nthen looking at the driver attributes & config combination, then finally\nthe diff.algorithm config.\n\nTo enforce precedence order, use the `xdl_opts_command_line` member\nduring options pasing to indicate the diff algorithm was set via command\nline args.\n\nSigned-off-by: John Cai \n---\n Documentation/gitattributes.txt | 41 ++++++++++++++++++++++++++++++++-\n diff.c | 25 +++++++++++++-------\n diff.h | 2 ++\n t/lib-diff-alternative.sh | 38 +++++++++++++++++++++++++++++-\n userdiff.c | 4 +++-\n userdiff.h | 1 +\n 6 files changed, 100 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex c19e64ea0ef..7e69f509d0a 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -736,7 +736,6 @@ String::\n \tby the configuration variables in the \"diff.foo\" section of the\n \tGit config file.\n \n-\n Defining an external diff driver\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n \n@@ -758,6 +757,46 @@ with the above configuration, i.e. `j-c-diff`, with 7\n parameters, just like `GIT_EXTERNAL_DIFF` program is called.\n See linkgit:git[1] for details.\n \n+Setting the internal diff algorithm\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+The diff algorithm can be set through the `diff.algorithm` config key, but\n+sometimes it may be helpful to set the diff algorithm by path. For example, one\n+might wish to set a diff algorithm automatically for all `.json` files such that\n+the user would not need to pass in a separate command line `--diff-algorithm` flag each\n+time.\n+\n+First, in `.gitattributes`, you would assign the `diff` attribute for paths.\n+\n+*Git attributes*\n+------------------------\n+*.json diff=\n+------------------------\n+\n+Then, you would define a \"diff..algorithm\" configuration to specify the\n+diff algorithm, choosing from `meyers`, `patience`, `minimal`, and `histogram`.\n+\n+*Git config*\n+\n+----------------------------------------------------------------\n+[diff \"\"]\n+ algorithm = histogram\n+----------------------------------------------------------------\n+\n+This diff algorithm applies to git-diff(1), including the `--stat` output.\n+\n+NOTE: If the `command` key also exists, then Git will treat this as an external\n+diff and attempt to use the value set for `command` as an external program. For\n+instance, the following config, combined with the above `.gitattributes` file,\n+will result in `command` favored over `algorithm`.\n+\n+*Git config*\n+\n+----------------------------------------------------------------\n+[diff \"\"]\n+ command = j-c-diff\n+ algorithm = histogram\n+----------------------------------------------------------------\n \n Defining a custom hunk-header\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\ndiff --git a/diff.c b/diff.c\nindex 92a0eab942e..24da439e56f 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4456,15 +4456,11 @@ static void run_diff_cmd(const char *pgm,\n \tconst char *xfrm_msg = NULL;\n \tint complete_rewrite = (p->status == DIFF_STATUS_MODIFIED) && p->score;\n \tint must_show_header = 0;\n+\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, attr_path);\n \n-\n-\tif (o->flags.allow_external) {\n-\t\tstruct userdiff_driver *drv;\n-\n-\t\tdrv = userdiff_find_by_path(o->repo->index, attr_path);\n+\tif (o->flags.allow_external)\n \t\tif (drv && drv->external)\n \t\t\tpgm = drv->external;\n-\t}\n \n \tif (msg) {\n \t\t/*\n@@ -4481,12 +4477,17 @@ static void run_diff_cmd(const char *pgm,\n \t\trun_external_diff(pgm, name, other, one, two, xfrm_msg, o);\n \t\treturn;\n \t}\n-\tif (one && two)\n+\tif (one && two) {\n+\t\tif (!o->xdl_opts_command_line)\n+\t\t\tif (drv && drv->algorithm)\n+\t\t\t\tset_diff_algorithm(o, drv->algorithm);\n+\n \t\tbuiltin_diff(name, other ? other : name,\n \t\t\t one, two, xfrm_msg, must_show_header,\n \t\t\t o, complete_rewrite);\n-\telse\n+\t} else {\n \t\tfprintf(o->file, \"* Unmerged path %s\\n\", name);\n+\t}\n }\n \n static void diff_fill_oid_info(struct diff_filespec *one, struct index_state *istate)\n@@ -4583,6 +4584,10 @@ static void run_diffstat(struct diff_filepair *p, struct diff_options *o,\n \tconst char *name;\n \tconst char *other;\n \n+\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, p->one->path);\n+\tif (drv && drv->algorithm)\n+\t\tset_diff_algorithm(o, drv->algorithm);\n+\n \tif (DIFF_PAIR_UNMERGED(p)) {\n \t\t/* unmerged */\n \t\tbuiltin_diffstat(p->one->path, NULL, NULL, NULL,\n@@ -5130,6 +5135,8 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n \t\treturn error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n \n+\toptions->xdl_opts_command_line = 1;\n+\n \treturn 0;\n }\n \n@@ -5157,6 +5164,8 @@ static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n \t\tBUG(\"available diff algorithms include \\\"myers\\\", \"\n \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n \n+\toptions->xdl_opts_command_line = 1;\n+\n \treturn 0;\n }\n \ndiff --git a/diff.h b/diff.h\nindex 41eb2c3d428..46b565abfd4 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -333,6 +333,8 @@ struct diff_options {\n \tint prefix_length;\n \tconst char *stat_sep;\n \tint xdl_opts;\n+\t/* If xdl_opts has been set via the command line. */\n+\tint xdl_opts_command_line;\n \n \t/* see Documentation/diff-options.txt */\n \tchar **anchors;\ndiff --git a/t/lib-diff-alternative.sh b/t/lib-diff-alternative.sh\nindex 8d1e408bb58..2dc02bca873 100644\n--- a/t/lib-diff-alternative.sh\n+++ b/t/lib-diff-alternative.sh\n@@ -105,10 +105,46 @@ index $file1..$file2 100644\n }\n EOF\n \n+\tcat >expect_diffstat < file2 | 21 ++++++++++-----------\n+ 1 file changed, 10 insertions(+), 11 deletions(-)\n+EOF\n+\n \tSTRATEGY=$1\n \n+\ttest_expect_success \"$STRATEGY diff from attributes\" '\n+\t\techo \"file* diff=driver\" >.gitattributes &&\n+\t\tgit config diff.driver.algorithm \"$STRATEGY\" &&\n+\t\ttest_must_fail git diff --no-index file1 file2 > output &&\n+\t\tcat expect &&\n+\t\tcat output &&\n+\t\ttest_cmp expect output\n+\t'\n+\n+\ttest_expect_success \"$STRATEGY diff from attributes has valid diffstat\" '\n+\t\techo \"file* diff=driver\" >.gitattributes &&\n+\t\tgit config diff.driver.algorithm \"$STRATEGY\" &&\n+\t\ttest_must_fail git diff --stat --no-index file1 file2 > output &&\n+\t\ttest_cmp expect_diffstat output\n+\t'\n+\n \ttest_expect_success \"$STRATEGY diff\" '\n-\t\ttest_must_fail git diff --no-index \"--$STRATEGY\" file1 file2 > output &&\n+\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n+\t\ttest_cmp expect output\n+\t'\n+\n+\ttest_expect_success \"$STRATEGY diff command line precedence before attributes\" '\n+\t\techo \"file* diff=driver\" >.gitattributes &&\n+\t\tgit config diff.driver.algorithm meyers &&\n+\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n+\t\ttest_cmp expect output\n+\t'\n+\n+\ttest_expect_success \"$STRATEGY diff attributes precedence before config\" '\n+\t\tgit config diff.algorithm default &&\n+\t\techo \"file* diff=driver\" >.gitattributes &&\n+\t\tgit config diff.driver.algorithm \"$STRATEGY\" &&\n+\t\ttest_must_fail git diff --no-index file1 file2 > output &&\n \t\ttest_cmp expect output\n \t'\n \ndiff --git a/userdiff.c b/userdiff.c\nindex d71b82feb74..ff25cfc4b4c 100644\n--- a/userdiff.c\n+++ b/userdiff.c\n@@ -293,7 +293,7 @@ PATTERNS(\"scheme\",\n \t \"|([^][)(}{[ \\t])+\"),\n PATTERNS(\"tex\", \"^(\\\\\\\\((sub)*section|chapter|part)\\\\*{0,1}\\\\{.*)$\",\n \t \"\\\\\\\\[a-zA-Z@]+|\\\\\\\\.|[a-zA-Z0-9\\x80-\\xff]+\"),\n-{ \"default\", NULL, -1, { NULL, 0 } },\n+{ \"default\", NULL, NULL, -1, { NULL, 0 } },\n };\n #undef PATTERNS\n #undef IPATTERN\n@@ -394,6 +394,8 @@ int userdiff_config(const char *k, const char *v)\n \t\treturn parse_bool(&drv->textconv_want_cache, k, v);\n \tif (!strcmp(type, \"wordregex\"))\n \t\treturn git_config_string(&drv->word_regex, k, v);\n+\tif (!strcmp(type, \"algorithm\"))\n+\t\treturn git_config_string(&drv->algorithm, k, v);\n \n \treturn 0;\n }\ndiff --git a/userdiff.h b/userdiff.h\nindex aee91bc77e6..24419db6973 100644\n--- a/userdiff.h\n+++ b/userdiff.h\n@@ -14,6 +14,7 @@ struct userdiff_funcname {\n struct userdiff_driver {\n \tconst char *name;\n \tconst char *external;\n+\tconst char *algorithm;\n \tint binary;\n \tstruct userdiff_funcname funcname;\n \tconst char *word_regex;\n-- \ngitgitgadget\n"},{"id":"472107","messageId":"CABPp-BFnCzWH6Aai0ZYv1fR7GMfXqiAE3n8q1Gcrhh-Zv_wTjA@mail.gmail.com","threadId":"59197","inReplyTo":"Y+b2l4Le2gTxGwO8@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-02-15T02:35:00Z","receivedAt":"2023-02-15T02:36:25Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Feb 10, 2023 at 5:59 PM Jeff King wrote:\n>\n> On Thu, Feb 09, 2023 at 02:44:15PM +0000, Phillip Wood wrote:\n>\n> > To see the differences between the output of patience and histogram\n> > algorithms I diffed the output of \"git log -p --no-merges\n> > --diff-algorithm=patience\" and \"git log -p --no-merges\n> > --diff-algorithm=histogram\". The first three differences are\n> >\n> > - 6c065f72b8 (http: support CURLOPT_PROTOCOLS_STR, 2023-01-16)\n> > In get_curl_allowed_protocols() the patience algorithm shows the\n> > change in the return statement more clearly\n> >\n> > - 47cfc9bd7d (attr: add flag `--source` to work with tree-ish, 2023-01-14)\n> > The histogram algorithm shows read_attr_from_index() being moved\n> > whereas the patience algorithm does not making the diff easier to\n> > follow.\n> >\n> > - b0226007f0 (fsmonitor: eliminate call to deprecated FSEventStream\n> > function, 2022-12-14)\n> > In fsm_listen__stop_async() the histogram algorithm shows\n> > data->shutdown_style = SHUTDOWN_EVENT;\n> > being moved, which is not as clear as the patience output which\n> > shows it as a context line.\n>\n> Just a small counter-point, since I happened to be looking at myers vs\n> patience for something elsewhere in the thread, but:\n>\n> git show 35bd13fcd2caa4185bf3729655ca20b6a5fe9b6f builtin/add.c\n\n\"fatal: bad object 35bd13fcd2caa4185bf3729655ca20b6a5fe9b6f\"\n\nIs that a local commit of yours?\n\n> looks slightly better to me with myers, even though it is 2 lines\n> longer. The issue is that patience and histogram are very eager to use\n> blank lines as anchor points, so a diff like:\n>\n> -some words\n> -\n> -and some more\n> +unrelated content\n> +\n> +but it happens to also be two paragraphs\n>\n> in myers becomes:\n>\n> -some words\n> +unrelated content\n>\n> -and some more\n> +but it happens to also be two paragraphs\n>\n> in patience (here I'm using single lines, but in practice these may be\n> paragraphs, or stanzas of code). I think that's also the _strength_ of\n> patience in many cases, but it really depends on the content. Replacing\n> a multi-stanza block with another one may be the best explanation for\n> what happened. Or the two stanzas may be independent, and showing the\n> change for each one may be better.\n>\n> I'm not sure which one happens more often. And you'd probably want to\n> weight it by how good/bad the change is. In the example I showed I don't\n> find patience very much worse, since it's already a pretty ugly diff.\n> But in cases where patience shines, it may be making things\n> significantly more readable.\n>\n> I don't have a super strong opinion, but I just wanted to chime in that\n> it is not clear to me that patience/histogram is always a win over myers\n> (yes, I know your examples were comparing patience vs histogram, but the\n> larger thread is discussing the other).\n\nOh, I agree histogram is not always a win over myers. I just feel it\nis the majority of the time. But if you want more than \"feels\",\nhere's some solid data to back that up...\n\nI found a study on the subject over at\nhttps://link.springer.com/article/10.1007/s10664-019-09772-z. They\nwere particularly interested in whether other academic studies could\nhave been affected by git's different diff algorithms, and came away\nwith the answer that it did. They looked at a few hundred thousand\ncommits across two dozen different repositories and found (note that\nthey only looked at myers and histogram, ignoring patience and\nminimal):\n\n * 92.4% - 98.6% of the diffs (depending on repo) are identical\nwhether you use myers or histogram\n * 93.8% - 99.2% of the diffs (depending on repo) have the same\nnumber of added/deleted lines with myers and histogram\n * Of the >20k diffs that were not the identical, they selected a\nrandom sample of 377 diffs (taking care to make sure they were\nstatistically representative)\n * They divided the 377 diffs into \"code\" and \"non-code\" diffs, i.e.\nthose modifying source code and those modifying other textual files\n * They had two people annotating the diffs and independently\nscoring them, and then checked for agreement between their answers\nafterwards. (No, they didn't always agree, but they did have\nsubstantial agreement.)\n\nFor the (again, non-identical) diffs modifying non-code, they found\n(see table 11) that:\n * 14.9% of the myers diffs are better\n * 13.4% of the histogram diffs are better\n * 71.6% of the diffs have equal quality\n\nFor the (non-identical) diffs modifying code, they found (again, see\ntable 11) that:\n * 16.9% of the myers diffs are better\n * 62.6% of the histogram diffs are better\n * 20.6% of the diffs have equal quality\n\nA ratio of 4 to 1 for histogram being better on code diffs is pretty\nweighty to me.\n\nIt's possible these results would have been even better were it not\nfor a couple of bugs in the histogram code (ported from the original\nin jgit). Phillip pointed me to a problematic testcase that Stefan\nBeller found, and in attempting to fix it (I'm on fix #4 or so), I\nbelieve I found another issue. However, I don't want to go into too\nmuch detail yet, as I found problems with some of my previous fixes\nand already invalidated things I told Phillip just last week.\n"},{"id":"472108","messageId":"xmqqk00j3b6r.fsf@gitster.g","threadId":"59197","inReplyTo":"0c5e1fc6c2651e39bcefa27ee0976c9519671969.1676410819.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/2] diff: consolidate diff algorithm option parsing","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-15T02:38:20Z","receivedAt":"2023-02-15T02:38:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\n> From: John Cai \n>\n> The diff option parsing for --minimal, --patience, --histgoram can all\n> be consolidated into one function. This is a preparatory step for the\n> subsequent commit which teaches diff to keep track of whether or not a\n> diff algorithm has been set via the command line.\n\nEverybody other than patience used to be just a bit-op but now\neverybody is a callback?\n\n> diff --git a/diff.c b/diff.c\n> index 329eebf16a0..92a0eab942e 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -3437,6 +3437,22 @@ static int diff_filepair_is_phoney(struct diff_filespec *one,\n> \treturn !DIFF_FILE_VALID(one) && !DIFF_FILE_VALID(two);\n> }\n> \n> +static int set_diff_algorithm(struct diff_options *opts,\n> +\t\t\t const char *alg)\n> +{\n> +\tlong value = parse_algorithm_value(alg);\n> +\n> +\tif (value < 0)\n> +\t\treturn 1;\n> +\n> +\t/* clear out previous settings */\n> +\tDIFF_XDL_CLR(opts, NEED_MINIMAL);\n> +\topts->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n> +\topts->xdl_opts |= value;\n> +\n> +\treturn 0;\n> +}\n\nThe above is a faithful copy of diff_opt_diff_algorithm(), except\nthat it returns 1 (not -1) on failure, which is unexpected in this\ncodebase, and should be corrected if this patch gets rerolled.\n\n> static void builtin_diff(const char *name_a,\n> \t\t\t const char *name_b,\n> \t\t\t struct diff_filespec *one,\n> @@ -5107,17 +5123,40 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n> \t\t\t\t const char *arg, int unset)\n> {\n> \tstruct diff_options *options = opt->value;\n> -\tlong value = parse_algorithm_value(arg);\n> \n> \tBUG_ON_OPT_NEG(unset);\n> -\tif (value < 0)\n> +\n> +\tif (set_diff_algorithm(options, arg))\n> \t\treturn error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n> \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n> \n> -\t/* clear out previous settings */\n> -\tDIFF_XDL_CLR(options, NEED_MINIMAL);\n> -\toptions->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n> -\toptions->xdl_opts |= value;\n> +\treturn 0;\n> +}\n\nThis version of diff_opt_diff_algorithm() behaves identically from\nthe version before this patch, which is excellent.\n\n> +static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n> +\t\t\t\t const char *arg, int unset)\n> +{\n> +\tstruct diff_options *options = opt->value;\n> +\n> +\tBUG_ON_OPT_NEG(unset);\n> +\tBUG_ON_OPT_ARG(arg);\n> +\n> +\tif (!strcmp(opt->long_name, \"patience\")) {\n> +\t\tsize_t i;\n> +\t\t/*\n> +\t\t * Both --patience and --anchored use PATIENCE_DIFF\n> +\t\t * internally, so remove any anchors previously\n> +\t\t * specified.\n> +\t\t */\n> +\t\tfor (i = 0; i < options->anchors_nr; i++)\n> +\t\t\tfree(options->anchors[i]);\n> +\t\toptions->anchors_nr = 0;\n> +\t}\n> +\n> +\tif (set_diff_algorithm(options, opt->long_name))\n> +\t\tBUG(\"available diff algorithms include \\\"myers\\\", \"\n> +\t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n> +\n> \treturn 0;\n> }\n\nCalling this instead of diff_opt_patience() would make \"--patience\"\nparsed identically as before without this patch, which is excellent.\n\n> @@ -5562,9 +5581,10 @@ struct option *add_diff_options(const struct option *opts,\n> \t\t\t N_(\"prevent rename/copy detection if the number of rename/copy targets exceeds given limit\")),\n> \n> \t\tOPT_GROUP(N_(\"Diff algorithm options\")),\n> -\t\tOPT_BIT(0, \"minimal\", &options->xdl_opts,\n> -\t\t\tN_(\"produce the smallest possible diff\"),\n> -\t\t\tXDF_NEED_MINIMAL),\n> +\t\tOPT_CALLBACK_F(0, \"minimal\", options, NULL,\n> +\t\t\t N_(\"produce the smallest possible diff\"),\n> +\t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n> +\t\t\t diff_opt_diff_algorithm_no_arg),\n\nI offhand cannot say that these two are equivalent, even though they\nought to be (otherwise this patch would break things). The callback\nseems to do much more than just a simple \"flip the NEED_MINIMAL bit\non\".\n\n> -\t\tOPT_BITOP(0, \"histogram\", &options->xdl_opts,\n> -\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n> -\t\t\t XDF_HISTOGRAM_DIFF, XDF_DIFF_ALGORITHM_MASK),\n> +\t\t\t diff_opt_diff_algorithm_no_arg),\n> +\t\tOPT_CALLBACK_F(0, \"histogram\", options, NULL,\n> +\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n> +\t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n> +\t\t\t diff_opt_diff_algorithm_no_arg),\n\nLikewise.\n\nBy nature, patience (and anchored) needs to do much more than\neverybody else, so it almost feels that it is OK (and preferable,\neven) to leave it a special case to make the distinction stand out.\nConsolidating everybody else who are much simpler to share the\nmore complex callback does not look like a good change to me, at\nleast at the first glance.\n\nThanks.\n"},{"id":"472115","messageId":"xmqq4jrn3ac7.fsf@gitster.g","threadId":"59197","inReplyTo":"cb0305631496eb4c2d51e5b586ac0ca8580c7dc1.1676410819.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-15T02:56:40Z","receivedAt":"2023-02-15T02:56:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\n> From: John Cai \n>\n> It can be useful to specify diff algorithms per file type. For example,\n> one may want to use the minimal diff algorithm for .json files, another\n> for .c files, etc.\n>\n> Teach the diff machinery to check attributes for a diff driver. Also\n> teach the diff driver parser a new type \"algorithm\" to look for in the\n> config, which will be used if a driver has been specified through the\n> attributes.\n>\n> Enforce precedence of diff algorithm by favoring the command line option,\n> then looking at the driver attributes & config combination, then finally\n> the diff.algorithm config.\n>\n> To enforce precedence order, use the `xdl_opts_command_line` member\n> during options pasing to indicate the diff algorithm was set via command\n> line args.\n>\n> Signed-off-by: John Cai \n> ---\n> Documentation/gitattributes.txt | 41 ++++++++++++++++++++++++++++++++-\n> diff.c | 25 +++++++++++++-------\n> diff.h | 2 ++\n> t/lib-diff-alternative.sh | 38 +++++++++++++++++++++++++++++-\n> userdiff.c | 4 +++-\n> userdiff.h | 1 +\n> 6 files changed, 100 insertions(+), 11 deletions(-)\n>\n> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n> index c19e64ea0ef..7e69f509d0a 100644\n> --- a/Documentation/gitattributes.txt\n> +++ b/Documentation/gitattributes.txt\n> @@ -736,7 +736,6 @@ String::\n> \tby the configuration variables in the \"diff.foo\" section of the\n> \tGit config file.\n> \n> -\n> Defining an external diff driver\n> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nUnrelated change? Wider paragraph gap between two sections than\nother inter-paragraph gaps inside a single section is what original\nhad, and I think that is a reasonable thing to keep.\n\n> @@ -758,6 +757,46 @@ with the above configuration, i.e. `j-c-diff`, with 7\n> parameters, just like `GIT_EXTERNAL_DIFF` program is called.\n> See linkgit:git[1] for details.\n\nIn other words, this new section wants another blank line before to match.\n\n> \n> +Setting the internal diff algorithm\n> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> +\n> +The diff algorithm can be set through the `diff.algorithm` config key, but\n> +sometimes it may be helpful to set the diff algorithm by path. For example, one\n> +might wish to set a diff algorithm automatically for all `.json` files such that\n> +the user would not need to pass in a separate command line `--diff-algorithm` flag each\n> +time.\n\nThat's an overly wide paragraph.\n\n> +\n> +First, in `.gitattributes`, you would assign the `diff` attribute for paths.\n> +\n> +*Git attributes*\n\nDiscard this line (mimic an existing section, like \"Defining a\ncustom hunk-header\").\n\n> +------------------------\n> +*.json diff=\n> +------------------------\n> +\n> +Then, you would define a \"diff..algorithm\" configuration to specify the\n> +diff algorithm, choosing from `meyers`, `patience`, `minimal`, and `histogram`.\n> +\n> +*Git config*\n\nLikewise, discard this line (I won't repeat but the next hunk has\nthe same issue).\n\n> diff --git a/diff.c b/diff.c\n> index 92a0eab942e..24da439e56f 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -4456,15 +4456,11 @@ static void run_diff_cmd(const char *pgm,\n> \tconst char *xfrm_msg = NULL;\n> \tint complete_rewrite = (p->status == DIFF_STATUS_MODIFIED) && p->score;\n> \tint must_show_header = 0;\n> +\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, attr_path);\n\nDo we run this look-up unconditionally, even when .allow_external\nbit is not set? Why?\n\n> -\n> -\tif (o->flags.allow_external) {\n> -\t\tstruct userdiff_driver *drv;\n> -\n> -\t\tdrv = userdiff_find_by_path(o->repo->index, attr_path);\n> +\tif (o->flags.allow_external)\n> \t\tif (drv && drv->external)\n> \t\t\tpgm = drv->external;\n> -\t}\n> \n> \tif (msg) {\n> \t\t/*\n> @@ -4481,12 +4477,17 @@ static void run_diff_cmd(const char *pgm,\n> \t\trun_external_diff(pgm, name, other, one, two, xfrm_msg, o);\n> \t\treturn;\n> \t}\n> -\tif (one && two)\n> +\tif (one && two) {\n> +\t\tif (!o->xdl_opts_command_line)\n> +\t\t\tif (drv && drv->algorithm)\n> +\t\t\t\tset_diff_algorithm(o, drv->algorithm);\n\nThe idea here seems to be \"if there is no explicit instruction, and\nif the diff driver specifies an algorithm, then use that one\", which\nis very straightforward and sensible. Can we reliably tell if we\nhad an explicit instruction to override the driver? That should\nprobably appear in other parts of the code, I guess.\n\n> \t\tbuiltin_diff(name, other ? other : name,\n> \t\t\t one, two, xfrm_msg, must_show_header,\n> \t\t\t o, complete_rewrite);\n> -\telse\n> +\t} else {\n> \t\tfprintf(o->file, \"* Unmerged path %s\\n\", name);\n> +\t}\n> }\n\n\n\n> @@ -4583,6 +4584,10 @@ static void run_diffstat(struct diff_filepair *p, struct diff_options *o,\n> \tconst char *name;\n> \tconst char *other;\n> \n> +\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, p->one->path);\n> +\tif (drv && drv->algorithm)\n> +\t\tset_diff_algorithm(o, drv->algorithm);\n\nInteresting. Does external diff play a role, like in run_diff_cmd()\nwe saw earlier?\n\n> @@ -5130,6 +5135,8 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n> \t\treturn error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n> \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n> \n> +\toptions->xdl_opts_command_line = 1;\n\nOK, calling this member \"xdl_\" anything is highly misleading, as it\nhas nothing to do with the xdiff machinery. How about calling it\nafter what it does, i.e. allowing the attribute driven diff driver\nto specify the algorithm? options.ignore_driver_algorithm or\nsomething? The options coming _from_ the command line may happen to\nbe the condition to trigger this behaviour in this current\nimplementation, but it does not have to stay that way forever.\nLosing \"command line\" from the name of the flag would make it\nclearer what is essential (i.e. this controls if the diff driver is\nallowed to affect the choice of the algorithm) and what is not (i.e.\nwe happen to let it decided based on the presence or absense of\ncommand line choice).\n\nThanks.\n"},{"id":"472116","messageId":"xmqqwn4j1uon.fsf@gitster.g","threadId":"59197","inReplyTo":"xmqq4jrn3ac7.fsf@gitster.g","subject":"Re: [PATCH v2 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-15T03:20:08Z","receivedAt":"2023-02-15T03:20:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> diff --git a/diff.c b/diff.c\n>> index 92a0eab942e..24da439e56f 100644\n>> --- a/diff.c\n>> +++ b/diff.c\n>> @@ -4456,15 +4456,11 @@ static void run_diff_cmd(const char *pgm,\n>> \tconst char *xfrm_msg = NULL;\n>> \tint complete_rewrite = (p->status == DIFF_STATUS_MODIFIED) && p->score;\n>> \tint must_show_header = 0;\n>> +\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, attr_path);\n>\n> Do we run this look-up unconditionally, even when .allow_external\n> bit is not set? Why?\n\nAh, this is perfectly fine. It used to be that this codepath can\ntell that there is no need to check the diff driver when it is told\nnever to use any external diff driver. Now, even when it is computing\nthe diff internally, it needs to check the diff driver to find out\nthe favoured algorithm for the path.\n\nStrictly speaking, if we are told NOT to use external diff driver,\nand if we are told NOT to pay attention to algorithm given by the\ndiff driver, then we know we can skip the overhead of attribute\nlook-up. I.e. we could do this to avoid attribute look-up:\n\n\tstruct userdiff_driver *drv = NULL;\n\n\tif (o->flags.allow_external || !o->ignore_driver_algorithm)\n\t\tdrv = userdiff_find_by_path(...);\n\n\tif (drv && o->flags.allow_external && drv->external)\n\t\tpgm = drv->external;\n\t...\n\tif (pgm)\n\t\t... do the external diff thing ...\n\tif (one && two) {\n\t\tif (drv && !o->ignore_driver_algorithm && drv->algorithm)\n\t\t\tset_diff_algo(...)\n\nI was not sure if it would be worth it before writing the above\ndown, but the resulting flow does not look _too_ bad.\n\n>> @@ -4583,6 +4584,10 @@ static void run_diffstat(struct diff_filepair *p, struct diff_options *o,\n>> \tconst char *name;\n>> \tconst char *other;\n>> \n>> +\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, p->one->path);\n>> +\tif (drv && drv->algorithm)\n>> +\t\tset_diff_algorithm(o, drv->algorithm);\n>\n> Interesting. Does external diff play a role, like in run_diff_cmd()\n> we saw earlier?\n\nAs whoever wrote \"diffstat\" did not think of counting output from\nexternal diff driver, of course in this codepath external diff would\nnot appear. So what we see is very much expected.\n\nJust move the blank line we see before these new lines one line\ndown, so that the variable decls are grouped together, with a blank\nline before the first executable statement. I.e.\n\n\tconst char *name;\n\tconst char *other;\n+ struct userdiff_driver *drv;\n+\n+\tdrv = userdiff_find_by_path(...);\n+\tif (drv && drv->algorithm)\n+\t\tset_diff_algorithm(o, drv->algorithm);\n\nShouldn't this function refrain from setting algorithm from the\ndriver when the algorithm was given elsewhere? E.g.\n\n\t$ git show --histogram --stat\n\t\nor something? IOW, shouldn't it also pay attention to\no->ignore_driver_algorithm bit, just like run_diff_cmd() did?\n\n\n"},{"id":"472117","messageId":"CABPp-BGmFemkiD1OFrrOdaJt9PjGRp+QHoV_azPqvTtx6CdD9Q@mail.gmail.com","threadId":"59197","inReplyTo":"AF5092D2-A561-4B56-8FB8-25DCFA28F32C@gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-02-15T03:41:00Z","receivedAt":"2023-02-15T03:42:04Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi John!\n\nOn Tue, Feb 14, 2023 at 1:16 PM John Cai wrote:\n> On 9 Feb 2023, at 3:44, Elijah Newren wrote:\n> > On Mon, Feb 6, 2023 at 12:47 PM John Cai wrote:\n> >>\n[...]\n> It seems like the performance penalty was because I was adding calls to parse\n> attribute files. Piggy backing off of the attribute parsing in userdiff.h will\n> allow us to not incur this performance penalty:\n>\n> $ hyperfine -r 5 -L a bin-wrappers/git,git '{a} diff v2.0.0 v2.28.0'\n> Benchmark 1: git-bin-wrapper diff v2.0.0 v2.28.0\n> Time (mean ± σ): 1.072 s ± 0.289 s [User: 0.626 s, System: 0.081 s]\n> Range (min … max): 0.772 s … 1.537 s 5 runs\n>\n> Benchmark 2: git diff v2.0.0 v2.28.0\n> Time (mean ± σ): 1.003 s ± 0.065 s [User: 0.684 s, System: 0.067 s]\n> Range (min … max): 0.914 s … 1.091 s 5 runs\n>\n> Summary\n> 'git diff v2.0.0 v2.28.0' ran\n> 1.07 ± 0.30 times faster than 'git-bin-wrapper diff v2.0.0 v2.28.0'\n\nYaay! Much better. :-)\n\nI'm curious, though, whether you are showing here a 7% slowdown (which\nwould still be bad), or just that the feature is correctly choosing a\ndifferent (but slower) algorithm for some files, or some kind of mix.\n\nWhat is the performance difference if you have this feature included,\nbut don't have any directives in .gitattributes selecting a different\ndiff algorithm for any files?\n\n> > And on a separate note...\n> >\n> > There's another set of considerations we might need to include here as\n> > well that I haven't seen anyone else in this thread talk about:\n>\n> These are some great questions. I'll do my best to answer them.\n> >\n> > * When trying to diff files, do we read the .gitattributes file from\n> > the current checkout to determine the diff algorithm(s)? Or the\n> > index? Or the commit we are diffing against?\n> > * If we use the current checkout or index, what about bare clones or\n> > diffing between two different commits?\n> > * If diffing between two different commits, and the .gitattributes has\n> > changed between those commits, which .gitattributes file wins?\n> > * If diffing between two different commits, and the .gitattributes has\n> > NOT changed, BUT a file has been renamed and the old and new names\n> > have different rules, which rule wins?\n>\n> In the next version I plan on using Peff's suggestion of utilizing the existing\n> diff driver scheme [1]. I believe these four questions are addressed if we use\n> the existing userdiff.h API, which in turn calls the attr.h API. We check the\n> worktree, then fallback to the index.\n\nSo...it sounds like we're just ignoring all the special cases listed\nabove, and living with bugs related to them? That's not a criticism;\nin fact, it might be okay -- after all, that's exactly what the\nexisting .gitattributes handling does and you are just hooking into\nit.\n\nI am a bit concerned, though, that we're increasing the visibility of\nthe interactions of .gitattributes with respect to these kinds of\ncases. I think external drivers are probably much less used than what\nyour feature might be, so folks are more likely to stumble into these\ncases and complain. Perhaps those cases are rare enough that we don't\ncare, but it might be at least worth documenting the issues (both to\nmanage user expectations and to give people a heads up about the\npotential issues.)\n\n(Also, it may be worth mentioning that I tend to focus on unusual\ncases for anything that might touch merging; Junio once named one of\nmy patchsets \"en/t6042-insane-merge-rename-testcases\". It's possible\nI worry about corner cases more than is justified given their real\nworld likelihood.)\n\n> By using the userdiff.h API, the behavior will match what users already expect\n> when they for instance set an external driver.\n\ns/already expect/already get/\n\nThe bugs also affect external drivers; I just suspect external drivers\naren't used enough that users have complained very loudly (yet?).\n\n> 1. https://lore.kernel.org/git/Y+KQtqNPews3vBS8@coredump.intra.peff.net/\n>\n> >\n> > * If per-file diff algorithms are adopted widely enough, will we be\n> > forced to change the merge algorithm to also pay attention to them?\n> > If it does, more complicated rename cases occur and we need rules for\n> > how to handle those.\n> > * If the merge algorithm has to pay attention to .gitattributes for\n> > this too, we'll have even more corner cases around what happens if\n> > there are merge conflicts in .gitattributes itself (which is already\n> > kind of ugly and kludged)\n>\n> I see this feature as a user-experience type convenience feature, so I don't\n> believe there's need for the merge machinery to also pay attention to the diff\n> algorithm set through gitattrbutes. We can clarify this in the documentation.\n\nThat would be awesome; *please* do this. This is my primary concern\nwith this patchset.\n\nI've spent an awful lot of time dealing with weird corner cases in the\nmerge machinery, and this appears to open a big can of worms to me.\nIt'd be a huge relief if we just agreed that the .gitattributes\nhandling here is only meant for user-facing diffs and will not be\nconsulted by the merge machinery.\n\n> > Anyway, I know I'm a bit animated and biased in this area, and I\n> > apologize if I'm a bit too much so. Even if I am, hopefully my\n> > comments at least provide some useful context.\n>\n> No problem! thanks for raising these issues.\n>\n> thanks\n> John\n"},{"id":"472119","messageId":"Y+xdvck3ZKZCewim@coredump.intra.peff.net","threadId":"59197","inReplyTo":"CABPp-BFnCzWH6Aai0ZYv1fR7GMfXqiAE3n8q1Gcrhh-Zv_wTjA@mail.gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-15T04:21:17Z","receivedAt":"2023-02-15T04:21:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Feb 14, 2023 at 06:35:00PM -0800, Elijah Newren wrote:\n\n> > Just a small counter-point, since I happened to be looking at myers vs\n> > patience for something elsewhere in the thread, but:\n> >\n> > git show 35bd13fcd2caa4185bf3729655ca20b6a5fe9b6f builtin/add.c\n> \n> \"fatal: bad object 35bd13fcd2caa4185bf3729655ca20b6a5fe9b6f\"\n> \n> Is that a local commit of yours?\n\nOh, sorry. It's not my commit, but it may have been something I picked\nup off the list. The version from Junio is 20b813d7d3d.\n\n> * They had two people annotating the diffs and independently\n> scoring them, and then checked for agreement between their answers\n> afterwards. (No, they didn't always agree, but they did have\n> substantial agreement.)\n\nKind of a weird methodology. I'd think you'd do better to put those\ndiffs in front of a lot of people to get a consensus. But OK...\n\n> For the (again, non-identical) diffs modifying non-code, they found\n> (see table 11) that:\n> * 14.9% of the myers diffs are better\n> * 13.4% of the histogram diffs are better\n> * 71.6% of the diffs have equal quality\n> \n> For the (non-identical) diffs modifying code, they found (again, see\n> table 11) that:\n> * 16.9% of the myers diffs are better\n> * 62.6% of the histogram diffs are better\n> * 20.6% of the diffs have equal quality\n> \n> A ratio of 4 to 1 for histogram being better on code diffs is pretty\n> weighty to me.\n\nInteresting. They do slightly worse on non-code, but probably not enough\nto worry about.\n\nDespite my complaint above, I do find those results compelling. Or at\nleast, it is much closer to being real data than anything I have seen\nbefore, so it seems like the best guide we have. :)\n\nLike I said, I don't have a very strong opinion myself. Mostly I wanted\nto note that while it is easy for us to remember the times we saw a\nMyers diff, said \"yuck\", and tried patience and it was better, most of\nus may be blind to the opposite case. If patience is not the default,\nmost of us would not have hit \"yuck\" case for it at all.\n\nFWIW, I coincidentally hit this case earlier today where patience does a\n_much_ better job than myers:\n\n https://lore.kernel.org/git/Y+vV8Ifkj1QV7KF0@coredump.intra.peff.net/\n\nSo don't count me as an objection, but just a curious observer. ;)\n\n-Peff\n"},{"id":"472122","messageId":"xmqqo7pv1p3o.fsf@gitster.g","threadId":"59197","inReplyTo":"Y+xdvck3ZKZCewim@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-15T05:20:43Z","receivedAt":"2023-02-15T05:20:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> > git show 35bd13fcd2caa4185bf3729655ca20b6a5fe9b6f builtin/add.c\n>> \n>> \"fatal: bad object 35bd13fcd2caa4185bf3729655ca20b6a5fe9b6f\"\n>> \n>> Is that a local commit of yours?\n>\n> Oh, sorry. It's not my commit, but it may have been something I picked\n> up off the list. The version from Junio is 20b813d7d3d.\n\n$ git show -s --notes=amlog --format='%N %s' 20b813d7d3d\nMessage-Id: \n add: remove \"add.interactive.useBuiltin\" & Perl \"git add--interactive\"\n\n> FWIW, I coincidentally hit this case earlier today where patience does a\n> _much_ better job than myers:\n>\n> https://lore.kernel.org/git/Y+vV8Ifkj1QV7KF0@coredump.intra.peff.net/\n>\n> So don't count me as an objection, but just a curious observer. ;)\n\nYeah, I know some people consider that Patience is the best thing\nsince sliced bread, and it does produce more readable diffs often.\nI haven't used it often enough to cite/remember a bad case, though.\n"},{"id":"472133","messageId":"2ca47f26-f10a-1fae-c879-f9479f7acdb1@dunelm.org.uk","threadId":"59197","inReplyTo":"Y+b2l4Le2gTxGwO8@coredump.intra.peff.net","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-02-15T14:44:59Z","receivedAt":"2023-02-15T14:45:08Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Peff\n\nThanks for the counter example\n\nOn 11/02/2023 01:59, Jeff King wrote:\n> Just a small counter-point, since I happened to be looking at myers vs\n> patience for something elsewhere in the thread, but:\n> \n> git show 35bd13fcd2caa4185bf3729655ca20b6a5fe9b6f builtin/add.c\n> \n> looks slightly better to me with myers, even though it is 2 lines\n> longer. The issue is that patience and histogram are very eager to use\n> blank lines as anchor points, so a diff like:\n> \n> -some words\n> -\n> -and some more\n> +unrelated content\n> +\n> +but it happens to also be two paragraphs\n> \n> in myers becomes:\n> \n> -some words\n> +unrelated content\n> \n> -and some more\n> +but it happens to also be two paragraphs\n> \n> in patience (here I'm using single lines, but in practice these may be\n> paragraphs, or stanzas of code). I think that's also the _strength_ of\n> patience in many cases, but it really depends on the content.\n\nIndeed. Ironically as there are no unique context lines in that example \nthe blank lines are being matched by patience implementation falling \nback to the myers algorithm. Normally the myers implementation tries to \navoid matching common context lines between two blocks of changed lines \nbut I think because in this case it is only called on a small part of \nthe file the blank lines are not common enough to trigger that \nheuristic. I've got a patch[1] that stops the patience implementation \nfalling back to the myers algorithm and just trims any leading and \ntrailing context. On the whole it I think it gives more readable diffs \nbut I've not got any systematic data to back that up. I also suspect \nthere are pathological cases such as each line in the file being \nduplicated where the falling back to the myers algorithm gives a much \nbetter result.\n\n> Replacing\n> a multi-stanza block with another one may be the best explanation for\n> what happened. Or the two stanzas may be independent, and showing the\n> change for each one may be better.\n >\n> I'm not sure which one happens more often. And you'd probably want to\n> weight it by how good/bad the change is. In the example I showed I don't\n> find patience very much worse, since it's already a pretty ugly diff.\n> But in cases where patience shines, it may be making things\n> significantly more readable.\n\nI agree that having some data would be useful if we're going to change \nthe default but collecting it would entail quite a bit of work and as \nthe scoring is subjective we'd want a few people doing it. It's great \nthat someone has done that for the histogram algorithm in the paper \nElijah cited.\n\n> I don't have a super strong opinion, but I just wanted to chime in that\n> it is not clear to me that patience/histogram is always a win over myers\n> (yes, I know your examples were comparing patience vs histogram, but the\n> larger thread is discussing the other).\n\nAgreed, there are definitely cases where myers gives more readable \ndiffs, I think if we're going to change the default the question we need \nto answer is which algorithm gives the best result most of the time.\n\nBest Wishes\n\nPhillip\n\n[1] https://github.com/phillipwood/git/commits/pure-patience-diff\n"},{"id":"472134","messageId":"159f09b1-8a69-f662-637d-8e883c31d0ec@dunelm.org.uk","threadId":"59197","inReplyTo":"230207.86sffh2xcu.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-02-15T14:47:51Z","receivedAt":"2023-02-15T14:48:57Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 07/02/2023 17:27, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Feb 07 2023, Phillip Wood wrote:\n> \n>> This is slightly off topic but one thing I'd really like is a way to\n>> tell diff use automatically use --diff-words on some files\n>> (e.g. Documentation/*)\n> \n> Unlike changing the algorithm, -U options, diff.orderFile etc. doing\n> that would give you a diff that can't be applied with \"git apply\" or\n> other tools that expect a valid unified diff.\n\nThat's a good point, we'd probably would want to guard it by checking if \nthe output is going to a tty.\n\n> So I can imagine that it would be neat in some contexts, but such a\n> change would have much wider implications than options that tweak how a\n> valid unified diff looks, or is generated.\n> \n> We'd need some way to mark a diff as \"for ad-hoc viewing only\".\n> \n> But as it sounds like you want this for git.git the\n> Documentation/doc-diff script is much better than anything word-diff\n> could spew out, as it diffs the resulting generated docs.\n\nIt's true that I should look at the doc-diff script for git, but it is a \nwider wish for text files in other projects as well.\n\nBest Wishes\n\nPhillip\n\n> I wonder (but haven't tried) whether you can't \"diff\" that using the\n> same method that can be used to diff binary files using a custom driver.\n> \n> Hrm, except that in that case (with includes etc) there isn't really a\n> 1=1 mapping between files within Documentation/ and generated docs (due\n> to includes etc.). But I suppose it could be used only for those files\n> that 1=1 correspond to the generated manpages.\n"},{"id":"472135","messageId":"Y+zzh80fybq8Tn66@coredump.intra.peff.net","threadId":"59197","inReplyTo":"2ca47f26-f10a-1fae-c879-f9479f7acdb1@dunelm.org.uk","subject":"Re: [PATCH 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-15T15:00:23Z","receivedAt":"2023-02-15T15:00:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 15, 2023 at 02:44:59PM +0000, Phillip Wood wrote:\n\n> Indeed. Ironically as there are no unique context lines in that example the\n> blank lines are being matched by patience implementation falling back to the\n> myers algorithm. Normally the myers implementation tries to avoid matching\n> common context lines between two blocks of changed lines but I think because\n> in this case it is only called on a small part of the file the blank lines\n> are not common enough to trigger that heuristic. I've got a patch[1] that\n> stops the patience implementation falling back to the myers algorithm and\n> just trims any leading and trailing context. On the whole it I think it\n> gives more readable diffs but I've not got any systematic data to back that\n> up. I also suspect there are pathological cases such as each line in the\n> file being duplicated where the falling back to the myers algorithm gives a\n> much better result.\n\nAh, I should have suspected it was something like that (since one of the\npurposes of patience is trying not to key on meaningless lines).\n\nI tried your patch on my test case, and the result is even more readable\nthan the myers output, because it really was effectively a complete\nrewrite of the function. It is, of course, not the minimal diff. I'm not\nsure if there would be cases where you'd prefer the minimal. I guess if\neach stanza's change really was independent of the others. But if there\nis no commonality except for blank lines, I find it hard to imagine that\nit's much worse to just treat the whole thing as a block.\n\nAnyway, thank you (and Elijah) for explaining. I'm getting more\ncomfortable with the idea of switching the default.\n\n-Peff\n"},{"id":"472173","messageId":"AAC53EE0-91F5-4A74-AB26-5F3F8B680CDE@gmail.com","threadId":"59197","inReplyTo":"xmqqk00j3b6r.fsf@gitster.g","subject":"Re: [PATCH v2 1/2] diff: consolidate diff algorithm option parsing","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-15T23:34:59Z","receivedAt":"2023-02-15T23:35:06Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Junio,\n\nOn 14 Feb 2023, at 21:38, Junio C Hamano wrote:\n\n> \"John Cai via GitGitGadget\" writes:\n>\n>> From: John Cai \n>>\n>> The diff option parsing for --minimal, --patience, --histgoram can all\n>> be consolidated into one function. This is a preparatory step for the\n>> subsequent commit which teaches diff to keep track of whether or not a\n>> diff algorithm has been set via the command line.\n>\n> Everybody other than patience used to be just a bit-op but now\n> everybody is a callback?\n>\n>> diff --git a/diff.c b/diff.c\n>> index 329eebf16a0..92a0eab942e 100644\n>> --- a/diff.c\n>> +++ b/diff.c\n>> @@ -3437,6 +3437,22 @@ static int diff_filepair_is_phoney(struct diff_filespec *one,\n>> \treturn !DIFF_FILE_VALID(one) && !DIFF_FILE_VALID(two);\n>> }\n>>\n>> +static int set_diff_algorithm(struct diff_options *opts,\n>> +\t\t\t const char *alg)\n>> +{\n>> +\tlong value = parse_algorithm_value(alg);\n>> +\n>> +\tif (value < 0)\n>> +\t\treturn 1;\n>> +\n>> +\t/* clear out previous settings */\n>> +\tDIFF_XDL_CLR(opts, NEED_MINIMAL);\n>> +\topts->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n>> +\topts->xdl_opts |= value;\n>> +\n>> +\treturn 0;\n>> +}\n>\n> The above is a faithful copy of diff_opt_diff_algorithm(), except\n> that it returns 1 (not -1) on failure, which is unexpected in this\n> codebase, and should be corrected if this patch gets rerolled.\n>\n>> static void builtin_diff(const char *name_a,\n>> \t\t\t const char *name_b,\n>> \t\t\t struct diff_filespec *one,\n>> @@ -5107,17 +5123,40 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n>> \t\t\t\t const char *arg, int unset)\n>> {\n>> \tstruct diff_options *options = opt->value;\n>> -\tlong value = parse_algorithm_value(arg);\n>>\n>> \tBUG_ON_OPT_NEG(unset);\n>> -\tif (value < 0)\n>> +\n>> +\tif (set_diff_algorithm(options, arg))\n>> \t\treturn error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n>> \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n>>\n>> -\t/* clear out previous settings */\n>> -\tDIFF_XDL_CLR(options, NEED_MINIMAL);\n>> -\toptions->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n>> -\toptions->xdl_opts |= value;\n>> +\treturn 0;\n>> +}\n>\n> This version of diff_opt_diff_algorithm() behaves identically from\n> the version before this patch, which is excellent.\n>\n>> +static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n>> +\t\t\t\t const char *arg, int unset)\n>> +{\n>> +\tstruct diff_options *options = opt->value;\n>> +\n>> +\tBUG_ON_OPT_NEG(unset);\n>> +\tBUG_ON_OPT_ARG(arg);\n>> +\n>> +\tif (!strcmp(opt->long_name, \"patience\")) {\n>> +\t\tsize_t i;\n>> +\t\t/*\n>> +\t\t * Both --patience and --anchored use PATIENCE_DIFF\n>> +\t\t * internally, so remove any anchors previously\n>> +\t\t * specified.\n>> +\t\t */\n>> +\t\tfor (i = 0; i < options->anchors_nr; i++)\n>> +\t\t\tfree(options->anchors[i]);\n>> +\t\toptions->anchors_nr = 0;\n>> +\t}\n>> +\n>> +\tif (set_diff_algorithm(options, opt->long_name))\n>> +\t\tBUG(\"available diff algorithms include \\\"myers\\\", \"\n>> +\t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n>> +\n>> \treturn 0;\n>> }\n>\n> Calling this instead of diff_opt_patience() would make \"--patience\"\n> parsed identically as before without this patch, which is excellent.\n>\n>> @@ -5562,9 +5581,10 @@ struct option *add_diff_options(const struct option *opts,\n>> \t\t\t N_(\"prevent rename/copy detection if the number of rename/copy targets exceeds given limit\")),\n>>\n>> \t\tOPT_GROUP(N_(\"Diff algorithm options\")),\n>> -\t\tOPT_BIT(0, \"minimal\", &options->xdl_opts,\n>> -\t\t\tN_(\"produce the smallest possible diff\"),\n>> -\t\t\tXDF_NEED_MINIMAL),\n>> +\t\tOPT_CALLBACK_F(0, \"minimal\", options, NULL,\n>> +\t\t\t N_(\"produce the smallest possible diff\"),\n>> +\t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n>> +\t\t\t diff_opt_diff_algorithm_no_arg),\n>\n> I offhand cannot say that these two are equivalent, even though they\n> ought to be (otherwise this patch would break things). The callback\n> seems to do much more than just a simple \"flip the NEED_MINIMAL bit\n> on\".\n>\n>> -\t\tOPT_BITOP(0, \"histogram\", &options->xdl_opts,\n>> -\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n>> -\t\t\t XDF_HISTOGRAM_DIFF, XDF_DIFF_ALGORITHM_MASK),\n>> +\t\t\t diff_opt_diff_algorithm_no_arg),\n>> +\t\tOPT_CALLBACK_F(0, \"histogram\", options, NULL,\n>> +\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n>> +\t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n>> +\t\t\t diff_opt_diff_algorithm_no_arg),\n>\n> Likewise.\n>\n> By nature, patience (and anchored) needs to do much more than\n> everybody else, so it almost feels that it is OK (and preferable,\n> even) to leave it a special case to make the distinction stand out.\n> Consolidating everybody else who are much simpler to share the\n> more complex callback does not look like a good change to me, at\n> least at the first glance.\n\nYeah, it's not great to pull things out from a bit flip to a callback but I\nneeded some way of setting the xdl_opts_command_line member in the next commit\nwhen we know that the diff algorithm was set via options on the command line so\nthat we can honor the precedence. If there's a different way to do that other\nthan using callbacks for command options parsing, I'd agree that keeping these\nas bit flips would be ideal.\n\n>\n> Thanks.\n\nThanks!\nJohn\n"},{"id":"472174","messageId":"xmqqmt5ev6ln.fsf@gitster.g","threadId":"59197","inReplyTo":"AAC53EE0-91F5-4A74-AB26-5F3F8B680CDE@gmail.com","subject":"Re: [PATCH v2 1/2] diff: consolidate diff algorithm option parsing","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-15T23:42:12Z","receivedAt":"2023-02-15T23:42:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"John Cai writes:\n\n> Yeah, it's not great to pull things out from a bit flip to a callback but I\n> needed some way of setting the xdl_opts_command_line member in the next commit\n\nIf that is the case (and after reading [2/2], of course, readers of\nthe series can tell that it was the reason), it would be good to be\nhonest and document _that_ as the reason why you are using callbacks\nfor all of them in the proposed log message for [1/2].\n\nAnd I think that is a reasonable way to use callback to do \"more\nthan just setting a bit\". Even in that case, I am not sure if it is\na good idea to share the same callback that has conditional code\nthat only is relevant to the \"patience\" case, though.\n\nTHanks.\n"},{"id":"472179","messageId":"Y+2RgdCDrDC1N9Y1@coredump.intra.peff.net","threadId":"59197","inReplyTo":"xmqqmt5ev6ln.fsf@gitster.g","subject":"Re: [PATCH v2 1/2] diff: consolidate diff algorithm option parsing","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-16T02:14:25Z","receivedAt":"2023-02-16T02:14:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 15, 2023 at 03:42:12PM -0800, Junio C Hamano wrote:\n\n> And I think that is a reasonable way to use callback to do \"more\n> than just setting a bit\". Even in that case, I am not sure if it is\n> a good idea to share the same callback that has conditional code\n> that only is relevant to the \"patience\" case, though.\n\nI have to admit I did a double-take at the string comparisons with\nopt->long. I guess we can rely on it, since it is coming from the\noptions struct (and is not the string that the user wrote on the command\nline!). But it still feels a little error-prone, just because there's no\nhelp from the type system or the compiler.\n\nI'd have expected it with individual callbacks like:\n\n int handle_patience(...)\n {\n\tdo_patience_specific_stuff();\n\tdo_shared_stuff(PATIENCE_DIFF);\n }\n\n int handle_histogram(...)\n {\n\tdo_shared_stuff(HISTOGRAM_DIFF);\n }\n\nand so on. That's a bit more verbose, but the call stack reflects the\nflow we expect.\n\nI can live with it either way, though.\n\n-Peff\n"},{"id":"472182","messageId":"xmqqk00itj0c.fsf@gitster.g","threadId":"59197","inReplyTo":"Y+2RgdCDrDC1N9Y1@coredump.intra.peff.net","subject":"Re: [PATCH v2 1/2] diff: consolidate diff algorithm option parsing","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-16T02:57:07Z","receivedAt":"2023-02-16T02:57:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> I'd have expected it with individual callbacks like:\n>\n> int handle_patience(...)\n> {\n> \tdo_patience_specific_stuff();\n> \tdo_shared_stuff(PATIENCE_DIFF);\n> }\n>\n> int handle_histogram(...)\n> {\n> \tdo_shared_stuff(HISTOGRAM_DIFF);\n> }\n>\n> and so on. That's a bit more verbose, but the call stack reflects the\n> flow we expect.\n\nExactly.\n\nThanks for spelling it out.\n"},{"id":"472224","messageId":"20230216203450.yej5c5dejjz6e4r7@pop-os","threadId":"59197","inReplyTo":"xmqqk00itj0c.fsf@gitster.g","subject":"Re: [PATCH v2 1/2] diff: consolidate diff algorithm option parsing","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-16T20:34:50Z","receivedAt":"2023-02-16T20:34:57Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"On 23/02/15 06:57PM, Junio C Hamano wrote:\n> Jeff King writes:\n> \n> > I'd have expected it with individual callbacks like:\n> >\n> > int handle_patience(...)\n> > {\n> > \tdo_patience_specific_stuff();\n> > \tdo_shared_stuff(PATIENCE_DIFF);\n> > }\n> >\n> > int handle_histogram(...)\n> > {\n> > \tdo_shared_stuff(HISTOGRAM_DIFF);\n> > }\n> >\n> > and so on. That's a bit more verbose, but the call stack reflects the\n> > flow we expect.\n> \n> Exactly.\n> \n> Thanks for spelling it out.\n\nThanks for this suggestion--makes sense to me. Will incorporate in the next\nversion.\n"},{"id":"472225","messageId":"20230216203731.uu3wvgs5ka5flmfv@pop-os","threadId":"59197","inReplyTo":"xmqqwn4j1uon.fsf@gitster.g","subject":"Re: [PATCH v2 2/2] diff: teach diff to read gitattribute diff-algorithm","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-16T20:37:31Z","receivedAt":"2023-02-16T20:37:36Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"On 23/02/14 07:20PM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> >> diff --git a/diff.c b/diff.c\n> >> index 92a0eab942e..24da439e56f 100644\n> >> --- a/diff.c\n> >> +++ b/diff.c\n> >> @@ -4456,15 +4456,11 @@ static void run_diff_cmd(const char *pgm,\n> >> \tconst char *xfrm_msg = NULL;\n> >> \tint complete_rewrite = (p->status == DIFF_STATUS_MODIFIED) && p->score;\n> >> \tint must_show_header = 0;\n> >> +\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, attr_path);\n> >\n> > Do we run this look-up unconditionally, even when .allow_external\n> > bit is not set? Why?\n> \n> Ah, this is perfectly fine. It used to be that this codepath can\n> tell that there is no need to check the diff driver when it is told\n> never to use any external diff driver. Now, even when it is computing\n> the diff internally, it needs to check the diff driver to find out\n> the favoured algorithm for the path.\n> \n> Strictly speaking, if we are told NOT to use external diff driver,\n> and if we are told NOT to pay attention to algorithm given by the\n> diff driver, then we know we can skip the overhead of attribute\n> look-up. I.e. we could do this to avoid attribute look-up:\n> \n> \tstruct userdiff_driver *drv = NULL;\n> \n> \tif (o->flags.allow_external || !o->ignore_driver_algorithm)\n> \t\tdrv = userdiff_find_by_path(...);\n> \n> \tif (drv && o->flags.allow_external && drv->external)\n> \t\tpgm = drv->external;\n> \t...\n> \tif (pgm)\n> \t\t... do the external diff thing ...\n> \tif (one && two) {\n> \t\tif (drv && !o->ignore_driver_algorithm && drv->algorithm)\n> \t\t\tset_diff_algo(...)\n> \n> I was not sure if it would be worth it before writing the above\n> down, but the resulting flow does not look _too_ bad.\n\nYes I think it's worth it to save on execution if we know we are not using\nexternal diff algorithm.\n\n> \n> >> @@ -4583,6 +4584,10 @@ static void run_diffstat(struct diff_filepair *p, struct diff_options *o,\n> >> \tconst char *name;\n> >> \tconst char *other;\n> >> \n> >> +\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, p->one->path);\n> >> +\tif (drv && drv->algorithm)\n> >> +\t\tset_diff_algorithm(o, drv->algorithm);\n> >\n> > Interesting. Does external diff play a role, like in run_diff_cmd()\n> > we saw earlier?\n> \n> As whoever wrote \"diffstat\" did not think of counting output from\n> external diff driver, of course in this codepath external diff would\n> not appear. So what we see is very much expected.\n> \n> Just move the blank line we see before these new lines one line\n> down, so that the variable decls are grouped together, with a blank\n> line before the first executable statement. I.e.\n> \n> \tconst char *name;\n> \tconst char *other;\n> + struct userdiff_driver *drv;\n> +\n> +\tdrv = userdiff_find_by_path(...);\n> +\tif (drv && drv->algorithm)\n> +\t\tset_diff_algorithm(o, drv->algorithm);\n\nmakes sense, thanks.\n\n> \n> Shouldn't this function refrain from setting algorithm from the\n> driver when the algorithm was given elsewhere? E.g.\n> \n> \t$ git show --histogram --stat\n> \t\n> or something? IOW, shouldn't it also pay attention to\n> o->ignore_driver_algorithm bit, just like run_diff_cmd() did?\n\nYes we should add the same guard as above.\n\n> \n> \n"},{"id":"472261","messageId":"816c47aa414586e99aa762604396bd8be4fb11f4.1676665285.git.gitgitgadget@gmail.com","threadId":"59197","inReplyTo":"pull.1452.v3.git.git.1676665285.gitgitgadget@gmail.com","subject":"[PATCH v3 1/2] diff: consolidate diff algorithm option parsing","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-17T20:21:24Z","receivedAt":"2023-02-17T20:21:32Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nA subsequent commit will need the ability to tell if the diff algorithm\nwas set through the command line through setting a new member of\ndiff_options. While this logic can be added to the\ndiff_opt_diff_algorithm() callback, the `--minimal` and `--histogram`\noptions are handled via OPT_BIT without a callback.\n\nRemedy this by consolidating the options parsing logic for --minimal and\n--histogram into one callback. This way we can modify `diff_options` in\nthat function.\n\nAs an additional refactor, the logic that sets the diff algorithm in\ndiff_opt_diff_algorithm() can be refactored into a helper that will\nallow multiple callsites to set the diff algorithm.\n\nSigned-off-by: John Cai \n---\n diff.c | 57 +++++++++++++++++++++++++++++++++++++++++++--------------\n 1 file changed, 43 insertions(+), 14 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 329eebf16a0..5efc22ca06b 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -3437,6 +3437,22 @@ static int diff_filepair_is_phoney(struct diff_filespec *one,\n \treturn !DIFF_FILE_VALID(one) && !DIFF_FILE_VALID(two);\n }\n \n+static int set_diff_algorithm(struct diff_options *opts,\n+\t\t\t const char *alg)\n+{\n+\tlong value = parse_algorithm_value(alg);\n+\n+\tif (value < 0)\n+\t\treturn -1;\n+\n+\t/* clear out previous settings */\n+\tDIFF_XDL_CLR(opts, NEED_MINIMAL);\n+\topts->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n+\topts->xdl_opts |= value;\n+\n+\treturn 0;\n+}\n+\n static void builtin_diff(const char *name_a,\n \t\t\t const char *name_b,\n \t\t\t struct diff_filespec *one,\n@@ -5107,17 +5123,28 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct diff_options *options = opt->value;\n-\tlong value = parse_algorithm_value(arg);\n \n \tBUG_ON_OPT_NEG(unset);\n-\tif (value < 0)\n+\n+\tif (set_diff_algorithm(options, arg))\n \t\treturn error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n \n-\t/* clear out previous settings */\n-\tDIFF_XDL_CLR(options, NEED_MINIMAL);\n-\toptions->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n-\toptions->xdl_opts |= value;\n+\treturn 0;\n+}\n+\n+static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n+\t\t\t\t const char *arg, int unset)\n+{\n+\tstruct diff_options *options = opt->value;\n+\n+\tBUG_ON_OPT_NEG(unset);\n+\tBUG_ON_OPT_ARG(arg);\n+\n+\tif (set_diff_algorithm(options, opt->long_name))\n+\t\tBUG(\"available diff algorithms include \\\"myers\\\", \"\n+\t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n+\n \treturn 0;\n }\n \n@@ -5250,7 +5277,6 @@ static int diff_opt_patience(const struct option *opt,\n \n \tBUG_ON_OPT_NEG(unset);\n \tBUG_ON_OPT_ARG(arg);\n-\toptions->xdl_opts = DIFF_WITH_ALG(options, PATIENCE_DIFF);\n \t/*\n \t * Both --patience and --anchored use PATIENCE_DIFF\n \t * internally, so remove any anchors previously\n@@ -5259,7 +5285,8 @@ static int diff_opt_patience(const struct option *opt,\n \tfor (i = 0; i < options->anchors_nr; i++)\n \t\tfree(options->anchors[i]);\n \toptions->anchors_nr = 0;\n-\treturn 0;\n+\n+\treturn set_diff_algorithm(options, \"patience\");\n }\n \n static int diff_opt_ignore_regex(const struct option *opt,\n@@ -5562,9 +5589,10 @@ struct option *add_diff_options(const struct option *opts,\n \t\t\t N_(\"prevent rename/copy detection if the number of rename/copy targets exceeds given limit\")),\n \n \t\tOPT_GROUP(N_(\"Diff algorithm options\")),\n-\t\tOPT_BIT(0, \"minimal\", &options->xdl_opts,\n-\t\t\tN_(\"produce the smallest possible diff\"),\n-\t\t\tXDF_NEED_MINIMAL),\n+\t\tOPT_CALLBACK_F(0, \"minimal\", options, NULL,\n+\t\t\t N_(\"produce the smallest possible diff\"),\n+\t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n+\t\t\t diff_opt_diff_algorithm_no_arg),\n \t\tOPT_BIT_F('w', \"ignore-all-space\", &options->xdl_opts,\n \t\t\t N_(\"ignore whitespace when comparing lines\"),\n \t\t\t XDF_IGNORE_WHITESPACE, PARSE_OPT_NONEG),\n@@ -5590,9 +5618,10 @@ struct option *add_diff_options(const struct option *opts,\n \t\t\t N_(\"generate diff using the \\\"patience diff\\\" algorithm\"),\n \t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n \t\t\t diff_opt_patience),\n-\t\tOPT_BITOP(0, \"histogram\", &options->xdl_opts,\n-\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n-\t\t\t XDF_HISTOGRAM_DIFF, XDF_DIFF_ALGORITHM_MASK),\n+\t\tOPT_CALLBACK_F(0, \"histogram\", options, NULL,\n+\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n+\t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n+\t\t\t diff_opt_diff_algorithm_no_arg),\n \t\tOPT_CALLBACK_F(0, \"diff-algorithm\", options, N_(\"\"),\n \t\t\t N_(\"choose a diff algorithm\"),\n \t\t\t PARSE_OPT_NONEG, diff_opt_diff_algorithm),\n-- \ngitgitgadget\n\n"},{"id":"472262","messageId":"pull.1452.v3.git.git.1676665285.gitgitgadget@gmail.com","threadId":"59197","inReplyTo":"pull.1452.v2.git.git.1676410819.gitgitgadget@gmail.com","subject":"[PATCH v3 0/2] Teach diff to honor diff algorithms set through git attributes","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-17T20:21:23Z","receivedAt":"2023-02-17T20:21:35Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"When a repository contains different kinds of files, it may be desirable to\nuse different algorithms based on file type. This is currently not feasible\nthrough the command line or using git configs. However, we can leverage the\nfact that gitattributes are path aware.\n\nTeach the diff machinery to check gitattributes when diffing files by using\nthe existing diff. scheme, and add an \"algorithm\" type to the external\ndriver config.\n\nChanges since V2:\n\n * minor clean up and variable renaming\n * avoid parsing attribute files for the driver if the diff algorithm is set\n through the command line\n\nChanges since V1:\n\n * utilize the existing diff..* scheme where the driver is defined\n in gitattributes, but the algorithm is defined in the gitconfig.\n\nTo address some of the performance concerns in the previous series, a\nbenchmark shows that a performance penalty is no longer incurred, now that\nwe are no longer adding an additional attributes parsing call:\n\n$ hyperfine -r 5 -L a bin-wrappers/git,git '{a} diff v2.0.0 v2.28.0'\nBenchmark 1: git-bin-wrapper diff v2.0.0 v2.28.0 Time (mean ± σ): 1.072 s ±\n0.289 s [User: 0.626 s, System: 0.081 s] Range (min … max): 0.772 s … 1.537\ns 5 runs\n\nBenchmark 2: git diff v2.0.0 v2.28.0 Time (mean ± σ): 1.003 s ± 0.065 s\n[User: 0.684 s, System: 0.067 s] Range (min … max): 0.914 s … 1.091 s 5 runs\n\nSummary 'git diff v2.0.0 v2.28.0' ran 1.07 ± 0.30 times faster than\n'git-bin-wrapper diff v2.0.0 v2.28.0'\n\nJohn Cai (2):\n diff: consolidate diff algorithm option parsing\n diff: teach diff to read algorithm from diff driver\n\n Documentation/gitattributes.txt | 37 ++++++++++++++\n diff.c | 90 ++++++++++++++++++++++++---------\n diff.h | 1 +\n t/lib-diff-alternative.sh | 38 +++++++++++++-\n userdiff.c | 4 +-\n userdiff.h | 1 +\n 6 files changed, 146 insertions(+), 25 deletions(-)\n\n\nbase-commit: c867e4fa180bec4750e9b54eb10f459030dbebfd\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1452%2Fjohn-cai%2Fjc%2Fattr-diff-algo-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1452/john-cai/jc/attr-diff-algo-v3\nPull-Request: https://github.com/git/git/pull/1452\n\nRange-diff vs v2:\n\n 1: 0c5e1fc6c26 ! 1: 816c47aa414 diff: consolidate diff algorithm option parsing\n @@ Metadata\n ## Commit message ##\n diff: consolidate diff algorithm option parsing\n \n - The diff option parsing for --minimal, --patience, --histgoram can all\n - be consolidated into one function. This is a preparatory step for the\n - subsequent commit which teaches diff to keep track of whether or not a\n - diff algorithm has been set via the command line.\n + A subsequent commit will need the ability to tell if the diff algorithm\n + was set through the command line through setting a new member of\n + diff_options. While this logic can be added to the\n + diff_opt_diff_algorithm() callback, the `--minimal` and `--histogram`\n + options are handled via OPT_BIT without a callback.\n \n - Additionally, the logic that sets the diff algorithm in\n - diff_opt_diff_algorithm() can be refactored into a helper that will\n + Remedy this by consolidating the options parsing logic for --minimal and\n + --histogram into one callback. This way we can modify `diff_options` in\n + that function.\n +\n + As an additional refactor, the logic that sets the diff algorithm in\n + diff_opt_diff_algorithm() can be refactored into a helper that will\n allow multiple callsites to set the diff algorithm.\n \n Signed-off-by: John Cai \n @@ diff.c: static int diff_filepair_is_phoney(struct diff_filespec *one,\n +\tlong value = parse_algorithm_value(alg);\n +\n +\tif (value < 0)\n -+\t\treturn 1;\n ++\t\treturn -1;\n +\n +\t/* clear out previous settings */\n +\tDIFF_XDL_CLR(opts, NEED_MINIMAL);\n @@ diff.c: static int diff_opt_diff_algorithm(const struct option *opt,\n +\tBUG_ON_OPT_NEG(unset);\n +\tBUG_ON_OPT_ARG(arg);\n +\n -+\tif (!strcmp(opt->long_name, \"patience\")) {\n -+\t\tsize_t i;\n -+\t\t/*\n -+\t\t * Both --patience and --anchored use PATIENCE_DIFF\n -+\t\t * internally, so remove any anchors previously\n -+\t\t * specified.\n -+\t\t */\n -+\t\tfor (i = 0; i < options->anchors_nr; i++)\n -+\t\t\tfree(options->anchors[i]);\n -+\t\toptions->anchors_nr = 0;\n -+\t}\n -+\n +\tif (set_diff_algorithm(options, opt->long_name))\n +\t\tBUG(\"available diff algorithms include \\\"myers\\\", \"\n +\t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n @@ diff.c: static int diff_opt_diff_algorithm(const struct option *opt,\n \treturn 0;\n }\n \n -@@ diff.c: static enum parse_opt_result diff_opt_output(struct parse_opt_ctx_t *ctx,\n - \treturn 0;\n - }\n +@@ diff.c: static int diff_opt_patience(const struct option *opt,\n \n --static int diff_opt_patience(const struct option *opt,\n --\t\t\t const char *arg, int unset)\n --{\n --\tstruct diff_options *options = opt->value;\n --\tint i;\n --\n --\tBUG_ON_OPT_NEG(unset);\n --\tBUG_ON_OPT_ARG(arg);\n + \tBUG_ON_OPT_NEG(unset);\n + \tBUG_ON_OPT_ARG(arg);\n -\toptions->xdl_opts = DIFF_WITH_ALG(options, PATIENCE_DIFF);\n --\t/*\n --\t * Both --patience and --anchored use PATIENCE_DIFF\n --\t * internally, so remove any anchors previously\n --\t * specified.\n --\t */\n --\tfor (i = 0; i < options->anchors_nr; i++)\n --\t\tfree(options->anchors[i]);\n --\toptions->anchors_nr = 0;\n + \t/*\n + \t * Both --patience and --anchored use PATIENCE_DIFF\n + \t * internally, so remove any anchors previously\n +@@ diff.c: static int diff_opt_patience(const struct option *opt,\n + \tfor (i = 0; i < options->anchors_nr; i++)\n + \t\tfree(options->anchors[i]);\n + \toptions->anchors_nr = 0;\n -\treturn 0;\n --}\n --\n ++\n ++\treturn set_diff_algorithm(options, \"patience\");\n + }\n + \n static int diff_opt_ignore_regex(const struct option *opt,\n - \t\t\t\t const char *arg, int unset)\n - {\n @@ diff.c: struct option *add_diff_options(const struct option *opts,\n \t\t\t N_(\"prevent rename/copy detection if the number of rename/copy targets exceeds given limit\")),\n \n @@ diff.c: struct option *add_diff_options(const struct option *opts,\n \t\t\t N_(\"ignore whitespace when comparing lines\"),\n \t\t\t XDF_IGNORE_WHITESPACE, PARSE_OPT_NONEG),\n @@ diff.c: struct option *add_diff_options(const struct option *opts,\n - \t\tOPT_CALLBACK_F(0, \"patience\", options, NULL,\n \t\t\t N_(\"generate diff using the \\\"patience diff\\\" algorithm\"),\n \t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n --\t\t\t diff_opt_patience),\n + \t\t\t diff_opt_patience),\n -\t\tOPT_BITOP(0, \"histogram\", &options->xdl_opts,\n -\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n -\t\t\t XDF_HISTOGRAM_DIFF, XDF_DIFF_ALGORITHM_MASK),\n -+\t\t\t diff_opt_diff_algorithm_no_arg),\n +\t\tOPT_CALLBACK_F(0, \"histogram\", options, NULL,\n +\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n +\t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n 2: cb030563149 ! 2: b330222ce83 diff: teach diff to read gitattribute diff-algorithm\n @@ Metadata\n Author: John Cai \n \n ## Commit message ##\n - diff: teach diff to read gitattribute diff-algorithm\n + diff: teach diff to read algorithm from diff driver\n \n It can be useful to specify diff algorithms per file type. For example,\n one may want to use the minimal diff algorithm for .json files, another\n for .c files, etc.\n \n - Teach the diff machinery to check attributes for a diff driver. Also\n - teach the diff driver parser a new type \"algorithm\" to look for in the\n + The diff machinery already checks attributes for a diff driver. Teach\n + the diff driver parser a new type \"algorithm\" to look for in the\n config, which will be used if a driver has been specified through the\n attributes.\n \n - Enforce precedence of diff algorithm by favoring the command line option,\n - then looking at the driver attributes & config combination, then finally\n - the diff.algorithm config.\n + Enforce precedence of the diff algorithm by favoring the command line\n + option, then looking at the driver attributes & config combination, then\n + finally the diff.algorithm config.\n \n - To enforce precedence order, use the `xdl_opts_command_line` member\n + To enforce precedence order, use a new `ignore_driver_algorithm` member\n during options pasing to indicate the diff algorithm was set via command\n line args.\n \n Signed-off-by: John Cai \n \n ## Documentation/gitattributes.txt ##\n -@@ Documentation/gitattributes.txt: String::\n - \tby the configuration variables in the \"diff.foo\" section of the\n - \tGit config file.\n - \n --\n - Defining an external diff driver\n - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n - \n @@ Documentation/gitattributes.txt: with the above configuration, i.e. `j-c-diff`, with 7\n parameters, just like `GIT_EXTERNAL_DIFF` program is called.\n See linkgit:git[1] for details.\n @@ Documentation/gitattributes.txt: with the above configuration, i.e. `j-c-diff`,\n +The diff algorithm can be set through the `diff.algorithm` config key, but\n +sometimes it may be helpful to set the diff algorithm by path. For example, one\n +might wish to set a diff algorithm automatically for all `.json` files such that\n -+the user would not need to pass in a separate command line `--diff-algorithm` flag each\n -+time.\n ++the user would not need to pass in a separate command line `--diff-algorithm`\n ++flag each time.\n +\n -+First, in `.gitattributes`, you would assign the `diff` attribute for paths.\n ++First, in `.gitattributes`, assign the `diff` attribute for paths.\n +\n -+*Git attributes*\n +------------------------\n +*.json diff=\n +------------------------\n +\n -+Then, you would define a \"diff..algorithm\" configuration to specify the\n -+diff algorithm, choosing from `meyers`, `patience`, `minimal`, and `histogram`.\n -+\n -+*Git config*\n ++Then, define a \"diff..algorithm\" configuration to specify the diff\n ++algorithm, choosing from `meyers`, `patience`, `minimal`, or `histogram`.\n +\n +----------------------------------------------------------------\n +[diff \"\"]\n + algorithm = histogram\n +----------------------------------------------------------------\n +\n -+This diff algorithm applies to git-diff(1), including the `--stat` output.\n ++This diff algorithm applies to user facing diff output like git-diff(1),\n ++git-show(1) and is used for the `--stat` output as well. The merge machinery\n ++will not use the diff algorithm set through this method.\n +\n +NOTE: If the `command` key also exists, then Git will treat this as an external\n +diff and attempt to use the value set for `command` as an external program. For\n +instance, the following config, combined with the above `.gitattributes` file,\n +will result in `command` favored over `algorithm`.\n +\n -+*Git config*\n -+\n +----------------------------------------------------------------\n +[diff \"\"]\n + command = j-c-diff\n @@ diff.c: static void run_diff_cmd(const char *pgm,\n \tconst char *xfrm_msg = NULL;\n \tint complete_rewrite = (p->status == DIFF_STATUS_MODIFIED) && p->score;\n \tint must_show_header = 0;\n -+\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, attr_path);\n ++\tstruct userdiff_driver *drv = NULL;\n \n -\n -\tif (o->flags.allow_external) {\n -\t\tstruct userdiff_driver *drv;\n -\n --\t\tdrv = userdiff_find_by_path(o->repo->index, attr_path);\n -+\tif (o->flags.allow_external)\n - \t\tif (drv && drv->external)\n - \t\t\tpgm = drv->external;\n ++\tif (o->flags.allow_external || !o->ignore_driver_algorithm)\n + \t\tdrv = userdiff_find_by_path(o->repo->index, attr_path);\n +-\t\tif (drv && drv->external)\n +-\t\t\tpgm = drv->external;\n -\t}\n ++\n ++\tif (o->flags.allow_external && drv && drv->external)\n ++\t\tpgm = drv->external;\n \n \tif (msg) {\n \t\t/*\n @@ diff.c: static void run_diff_cmd(const char *pgm,\n \t}\n -\tif (one && two)\n +\tif (one && two) {\n -+\t\tif (!o->xdl_opts_command_line)\n -+\t\t\tif (drv && drv->algorithm)\n -+\t\t\t\tset_diff_algorithm(o, drv->algorithm);\n ++\t\tif (drv && !o->ignore_driver_algorithm && drv->algorithm)\n ++\t\t\tset_diff_algorithm(o, drv->algorithm);\n +\n \t\tbuiltin_diff(name, other ? other : name,\n \t\t\t one, two, xfrm_msg, must_show_header,\n @@ diff.c: static void run_diffstat(struct diff_filepair *p, struct diff_options *o\n \tconst char *name;\n \tconst char *other;\n \n -+\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, p->one->path);\n -+\tif (drv && drv->algorithm)\n -+\t\tset_diff_algorithm(o, drv->algorithm);\n ++\tif (!o->ignore_driver_algorithm) {\n ++\t\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, p->one->path);\n ++\n ++\t\tif (drv && drv->algorithm) {\n ++\t\t\tset_diff_algorithm(o, drv->algorithm);\n ++\t\t}\n ++\t}\n +\n \tif (DIFF_PAIR_UNMERGED(p)) {\n \t\t/* unmerged */\n @@ diff.c: static int diff_opt_diff_algorithm(const struct option *opt,\n \t\treturn error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n \n -+\toptions->xdl_opts_command_line = 1;\n ++\toptions->ignore_driver_algorithm = 1;\n +\n \treturn 0;\n }\n @@ diff.c: static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n \t\tBUG(\"available diff algorithms include \\\"myers\\\", \"\n \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n \n -+\toptions->xdl_opts_command_line = 1;\n ++\toptions->ignore_driver_algorithm = 1;\n +\n \treturn 0;\n }\n \n +@@ diff.c: static int diff_opt_patience(const struct option *opt,\n + \tfor (i = 0; i < options->anchors_nr; i++)\n + \t\tfree(options->anchors[i]);\n + \toptions->anchors_nr = 0;\n ++\toptions->ignore_driver_algorithm = 1;\n + \n + \treturn set_diff_algorithm(options, \"patience\");\n + }\n \n ## diff.h ##\n @@ diff.h: struct diff_options {\n \tint prefix_length;\n \tconst char *stat_sep;\n \tint xdl_opts;\n -+\t/* If xdl_opts has been set via the command line. */\n -+\tint xdl_opts_command_line;\n ++\tint ignore_driver_algorithm;\n \n \t/* see Documentation/diff-options.txt */\n \tchar **anchors;\n\n-- \ngitgitgadget\n"},{"id":"472263","messageId":"b330222ce83bdf03c20085ff10fcff8a090474d5.1676665285.git.gitgitgadget@gmail.com","threadId":"59197","inReplyTo":"pull.1452.v3.git.git.1676665285.gitgitgadget@gmail.com","subject":"[PATCH v3 2/2] diff: teach diff to read algorithm from diff driver","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-17T20:21:25Z","receivedAt":"2023-02-17T20:21:36Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nIt can be useful to specify diff algorithms per file type. For example,\none may want to use the minimal diff algorithm for .json files, another\nfor .c files, etc.\n\nThe diff machinery already checks attributes for a diff driver. Teach\nthe diff driver parser a new type \"algorithm\" to look for in the\nconfig, which will be used if a driver has been specified through the\nattributes.\n\nEnforce precedence of the diff algorithm by favoring the command line\noption, then looking at the driver attributes & config combination, then\nfinally the diff.algorithm config.\n\nTo enforce precedence order, use a new `ignore_driver_algorithm` member\nduring options pasing to indicate the diff algorithm was set via command\nline args.\n\nSigned-off-by: John Cai \n---\n Documentation/gitattributes.txt | 37 ++++++++++++++++++++++++++++++++\n diff.c | 33 ++++++++++++++++++++--------\n diff.h | 1 +\n t/lib-diff-alternative.sh | 38 ++++++++++++++++++++++++++++++++-\n userdiff.c | 4 +++-\n userdiff.h | 1 +\n 6 files changed, 103 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex c19e64ea0ef..f212079a131 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -758,6 +758,43 @@ with the above configuration, i.e. `j-c-diff`, with 7\n parameters, just like `GIT_EXTERNAL_DIFF` program is called.\n See linkgit:git[1] for details.\n \n+Setting the internal diff algorithm\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+The diff algorithm can be set through the `diff.algorithm` config key, but\n+sometimes it may be helpful to set the diff algorithm by path. For example, one\n+might wish to set a diff algorithm automatically for all `.json` files such that\n+the user would not need to pass in a separate command line `--diff-algorithm`\n+flag each time.\n+\n+First, in `.gitattributes`, assign the `diff` attribute for paths.\n+\n+------------------------\n+*.json diff=\n+------------------------\n+\n+Then, define a \"diff..algorithm\" configuration to specify the diff\n+algorithm, choosing from `meyers`, `patience`, `minimal`, or `histogram`.\n+\n+----------------------------------------------------------------\n+[diff \"\"]\n+ algorithm = histogram\n+----------------------------------------------------------------\n+\n+This diff algorithm applies to user facing diff output like git-diff(1),\n+git-show(1) and is used for the `--stat` output as well. The merge machinery\n+will not use the diff algorithm set through this method.\n+\n+NOTE: If the `command` key also exists, then Git will treat this as an external\n+diff and attempt to use the value set for `command` as an external program. For\n+instance, the following config, combined with the above `.gitattributes` file,\n+will result in `command` favored over `algorithm`.\n+\n+----------------------------------------------------------------\n+[diff \"\"]\n+ command = j-c-diff\n+ algorithm = histogram\n+----------------------------------------------------------------\n \n Defining a custom hunk-header\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\ndiff --git a/diff.c b/diff.c\nindex 5efc22ca06b..04469da6d34 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4456,15 +4456,13 @@ static void run_diff_cmd(const char *pgm,\n \tconst char *xfrm_msg = NULL;\n \tint complete_rewrite = (p->status == DIFF_STATUS_MODIFIED) && p->score;\n \tint must_show_header = 0;\n+\tstruct userdiff_driver *drv = NULL;\n \n-\n-\tif (o->flags.allow_external) {\n-\t\tstruct userdiff_driver *drv;\n-\n+\tif (o->flags.allow_external || !o->ignore_driver_algorithm)\n \t\tdrv = userdiff_find_by_path(o->repo->index, attr_path);\n-\t\tif (drv && drv->external)\n-\t\t\tpgm = drv->external;\n-\t}\n+\n+\tif (o->flags.allow_external && drv && drv->external)\n+\t\tpgm = drv->external;\n \n \tif (msg) {\n \t\t/*\n@@ -4481,12 +4479,16 @@ static void run_diff_cmd(const char *pgm,\n \t\trun_external_diff(pgm, name, other, one, two, xfrm_msg, o);\n \t\treturn;\n \t}\n-\tif (one && two)\n+\tif (one && two) {\n+\t\tif (drv && !o->ignore_driver_algorithm && drv->algorithm)\n+\t\t\tset_diff_algorithm(o, drv->algorithm);\n+\n \t\tbuiltin_diff(name, other ? other : name,\n \t\t\t one, two, xfrm_msg, must_show_header,\n \t\t\t o, complete_rewrite);\n-\telse\n+\t} else {\n \t\tfprintf(o->file, \"* Unmerged path %s\\n\", name);\n+\t}\n }\n \n static void diff_fill_oid_info(struct diff_filespec *one, struct index_state *istate)\n@@ -4583,6 +4585,14 @@ static void run_diffstat(struct diff_filepair *p, struct diff_options *o,\n \tconst char *name;\n \tconst char *other;\n \n+\tif (!o->ignore_driver_algorithm) {\n+\t\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, p->one->path);\n+\n+\t\tif (drv && drv->algorithm) {\n+\t\t\tset_diff_algorithm(o, drv->algorithm);\n+\t\t}\n+\t}\n+\n \tif (DIFF_PAIR_UNMERGED(p)) {\n \t\t/* unmerged */\n \t\tbuiltin_diffstat(p->one->path, NULL, NULL, NULL,\n@@ -5130,6 +5140,8 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n \t\treturn error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n \n+\toptions->ignore_driver_algorithm = 1;\n+\n \treturn 0;\n }\n \n@@ -5145,6 +5157,8 @@ static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n \t\tBUG(\"available diff algorithms include \\\"myers\\\", \"\n \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n \n+\toptions->ignore_driver_algorithm = 1;\n+\n \treturn 0;\n }\n \n@@ -5285,6 +5299,7 @@ static int diff_opt_patience(const struct option *opt,\n \tfor (i = 0; i < options->anchors_nr; i++)\n \t\tfree(options->anchors[i]);\n \toptions->anchors_nr = 0;\n+\toptions->ignore_driver_algorithm = 1;\n \n \treturn set_diff_algorithm(options, \"patience\");\n }\ndiff --git a/diff.h b/diff.h\nindex 41eb2c3d428..8d770b1d579 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -333,6 +333,7 @@ struct diff_options {\n \tint prefix_length;\n \tconst char *stat_sep;\n \tint xdl_opts;\n+\tint ignore_driver_algorithm;\n \n \t/* see Documentation/diff-options.txt */\n \tchar **anchors;\ndiff --git a/t/lib-diff-alternative.sh b/t/lib-diff-alternative.sh\nindex 8d1e408bb58..2dc02bca873 100644\n--- a/t/lib-diff-alternative.sh\n+++ b/t/lib-diff-alternative.sh\n@@ -105,10 +105,46 @@ index $file1..$file2 100644\n }\n EOF\n \n+\tcat >expect_diffstat < file2 | 21 ++++++++++-----------\n+ 1 file changed, 10 insertions(+), 11 deletions(-)\n+EOF\n+\n \tSTRATEGY=$1\n \n+\ttest_expect_success \"$STRATEGY diff from attributes\" '\n+\t\techo \"file* diff=driver\" >.gitattributes &&\n+\t\tgit config diff.driver.algorithm \"$STRATEGY\" &&\n+\t\ttest_must_fail git diff --no-index file1 file2 > output &&\n+\t\tcat expect &&\n+\t\tcat output &&\n+\t\ttest_cmp expect output\n+\t'\n+\n+\ttest_expect_success \"$STRATEGY diff from attributes has valid diffstat\" '\n+\t\techo \"file* diff=driver\" >.gitattributes &&\n+\t\tgit config diff.driver.algorithm \"$STRATEGY\" &&\n+\t\ttest_must_fail git diff --stat --no-index file1 file2 > output &&\n+\t\ttest_cmp expect_diffstat output\n+\t'\n+\n \ttest_expect_success \"$STRATEGY diff\" '\n-\t\ttest_must_fail git diff --no-index \"--$STRATEGY\" file1 file2 > output &&\n+\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n+\t\ttest_cmp expect output\n+\t'\n+\n+\ttest_expect_success \"$STRATEGY diff command line precedence before attributes\" '\n+\t\techo \"file* diff=driver\" >.gitattributes &&\n+\t\tgit config diff.driver.algorithm meyers &&\n+\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n+\t\ttest_cmp expect output\n+\t'\n+\n+\ttest_expect_success \"$STRATEGY diff attributes precedence before config\" '\n+\t\tgit config diff.algorithm default &&\n+\t\techo \"file* diff=driver\" >.gitattributes &&\n+\t\tgit config diff.driver.algorithm \"$STRATEGY\" &&\n+\t\ttest_must_fail git diff --no-index file1 file2 > output &&\n \t\ttest_cmp expect output\n \t'\n \ndiff --git a/userdiff.c b/userdiff.c\nindex d71b82feb74..ff25cfc4b4c 100644\n--- a/userdiff.c\n+++ b/userdiff.c\n@@ -293,7 +293,7 @@ PATTERNS(\"scheme\",\n \t \"|([^][)(}{[ \\t])+\"),\n PATTERNS(\"tex\", \"^(\\\\\\\\((sub)*section|chapter|part)\\\\*{0,1}\\\\{.*)$\",\n \t \"\\\\\\\\[a-zA-Z@]+|\\\\\\\\.|[a-zA-Z0-9\\x80-\\xff]+\"),\n-{ \"default\", NULL, -1, { NULL, 0 } },\n+{ \"default\", NULL, NULL, -1, { NULL, 0 } },\n };\n #undef PATTERNS\n #undef IPATTERN\n@@ -394,6 +394,8 @@ int userdiff_config(const char *k, const char *v)\n \t\treturn parse_bool(&drv->textconv_want_cache, k, v);\n \tif (!strcmp(type, \"wordregex\"))\n \t\treturn git_config_string(&drv->word_regex, k, v);\n+\tif (!strcmp(type, \"algorithm\"))\n+\t\treturn git_config_string(&drv->algorithm, k, v);\n \n \treturn 0;\n }\ndiff --git a/userdiff.h b/userdiff.h\nindex aee91bc77e6..24419db6973 100644\n--- a/userdiff.h\n+++ b/userdiff.h\n@@ -14,6 +14,7 @@ struct userdiff_funcname {\n struct userdiff_driver {\n \tconst char *name;\n \tconst char *external;\n+\tconst char *algorithm;\n \tint binary;\n \tstruct userdiff_funcname funcname;\n \tconst char *word_regex;\n-- \ngitgitgadget\n"},{"id":"472271","messageId":"xmqq3574m183.fsf@gitster.g","threadId":"59197","inReplyTo":"816c47aa414586e99aa762604396bd8be4fb11f4.1676665285.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/2] diff: consolidate diff algorithm option parsing","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-17T21:27:40Z","receivedAt":"2023-02-17T21:28:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\n> From: John Cai \n>\n> A subsequent commit will need the ability to tell if the diff algorithm\n> was set through the command line through setting a new member of\n> diff_options. While this logic can be added to the\n> diff_opt_diff_algorithm() callback, the `--minimal` and `--histogram`\n> options are handled via OPT_BIT without a callback.\n>\n> Remedy this by consolidating the options parsing logic for --minimal and\n> --histogram into one callback. This way we can modify `diff_options` in\n> that function.\n>\n> As an additional refactor, the logic that sets the diff algorithm in\n> diff_opt_diff_algorithm() can be refactored into a helper that will\n> allow multiple callsites to set the diff algorithm.\n>\n> Signed-off-by: John Cai \n> ---\n> diff.c | 57 +++++++++++++++++++++++++++++++++++++++++++--------------\n> 1 file changed, 43 insertions(+), 14 deletions(-)\n\nNicely done. Much easier to follow. Will queue. Thanks.\n"},{"id":"472274","messageId":"xmqqfsb4klks.fsf@gitster.g","threadId":"59197","inReplyTo":"b330222ce83bdf03c20085ff10fcff8a090474d5.1676665285.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 2/2] diff: teach diff to read algorithm from diff driver","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-17T21:50:59Z","receivedAt":"2023-02-17T21:51:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\nLooking good. Some comments below. Many of them minor.\n\n> +Setting the internal diff algorithm\n> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> +\n> +The diff algorithm can be set through the `diff.algorithm` config key, but\n> +sometimes it may be helpful to set the diff algorithm by path. For example, one\n\nI would have expected \"per path\" instead of \"by path\".\n\n> +might wish to set a diff algorithm automatically for all `.json` files such that\n> +the user would not need to pass in a separate command line `--diff-algorithm`\n> +flag each time.\n\nWhile this is not incorrect per-se, I think the first paragraph of\nthe proposed commit log message was a lot more convincing. Your\nchanges may not be limited to a single kind of files, and a command\nline option is simply not enough. You may want one algorithm for\n\".json\" while using another for \".c\", which was really an excellent\nexample you gave.\n\n> +This diff algorithm applies to user facing diff output like git-diff(1),\n> +git-show(1) and is used for the `--stat` output as well. The merge machinery\n> +will not use the diff algorithm set through this method.\n\nIs \"format-patch\" considered \"user-facing\"?\n\n> +NOTE: If the `command` key also exists, then Git will treat this as an external\n> +diff and attempt to use the value set for `command` as an external program. For\n> +instance, the following config, combined with the above `.gitattributes` file,\n> +will result in `command` favored over `algorithm`.\n> +\n> +----------------------------------------------------------------\n> +[diff \"\"]\n> + command = j-c-diff\n> + algorithm = histogram\n> +----------------------------------------------------------------\n\nIsn't this a bit too verbose, given that the reader has just seen\nthe external diff driver section. I wonder something like this is\nsufficient, without any sample configuration?\n\n NOTE: If `diff..command` is defined for path with the\n `diff=` attribute, it is executed as an external diff driver\n (see above), and adding `diff..algorithm` has no effect (the\n algorithm is not passed to the external diff driver).\n\n\n> diff --git a/diff.c b/diff.c\n> index 5efc22ca06b..04469da6d34 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -4456,15 +4456,13 @@ static void run_diff_cmd(const char *pgm,\n> \tconst char *xfrm_msg = NULL;\n> \tint complete_rewrite = (p->status == DIFF_STATUS_MODIFIED) && p->score;\n> \tint must_show_header = 0;\n> +\tstruct userdiff_driver *drv = NULL;\n> \n> -\n> -\tif (o->flags.allow_external) {\n> -\t\tstruct userdiff_driver *drv;\n> -\n> +\tif (o->flags.allow_external || !o->ignore_driver_algorithm)\n> \t\tdrv = userdiff_find_by_path(o->repo->index, attr_path);\n> -\t\tif (drv && drv->external)\n> -\t\t\tpgm = drv->external;\n> -\t}\n> +\n> +\tif (o->flags.allow_external && drv && drv->external)\n> +\t\tpgm = drv->external;\n\nOK. There is no explicit \"pgm = NULL\" initialization in this\nfunction, but that is done by the caller passing NULL to the\nfunction as a parameter, so it all makes sense.\n\n> @@ -4481,12 +4479,16 @@ static void run_diff_cmd(const char *pgm,\n> \t\trun_external_diff(pgm, name, other, one, two, xfrm_msg, o);\n> \t\treturn;\n> \t}\n> -\tif (one && two)\n> +\tif (one && two) {\n> +\t\tif (drv && !o->ignore_driver_algorithm && drv->algorithm)\n> +\t\t\tset_diff_algorithm(o, drv->algorithm);\n\nFor symmetry with the above choice of pgm we just saw, the order of\nthe condition might be easier to follow if written like so:\n\n\tif (!o->ignore_driver_algorithm && drv && drv->algorithm)\n\nIt would not make any measurable difference performance-wise either way.\n\n> @@ -4583,6 +4585,14 @@ static void run_diffstat(struct diff_filepair *p, struct diff_options *o,\n> \tconst char *name;\n> \tconst char *other;\n> \n> +\tif (!o->ignore_driver_algorithm) {\n> +\t\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, p->one->path);\n\nThat's an overlong line.\n\n> +\n> +\t\tif (drv && drv->algorithm) {\n> +\t\t\tset_diff_algorithm(o, drv->algorithm);\n> +\t\t}\n\nNo need to have {} around a single statement block.\n\n> +\t}\n> +\n> \tif (DIFF_PAIR_UNMERGED(p)) {\n> \t\t/* unmerged */\n> \t\tbuiltin_diffstat(p->one->path, NULL, NULL, NULL,\n> @@ -5130,6 +5140,8 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n> \t\treturn error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n> \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n> \n> +\toptions->ignore_driver_algorithm = 1;\n> +\n> \treturn 0;\n> }\n> \n> @@ -5145,6 +5157,8 @@ static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n> \t\tBUG(\"available diff algorithms include \\\"myers\\\", \"\n> \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n> \n> +\toptions->ignore_driver_algorithm = 1;\n> +\n> \treturn 0;\n> }\n> @@ -5285,6 +5299,7 @@ static int diff_opt_patience(const struct option *opt,\n> \tfor (i = 0; i < options->anchors_nr; i++)\n> \t\tfree(options->anchors[i]);\n> \toptions->anchors_nr = 0;\n> +\toptions->ignore_driver_algorithm = 1;\n> \n> \treturn set_diff_algorithm(options, \"patience\");\n> }\n\n\nI was hoping that set_diff_algorithm() can be the shared common one\nthat signals we were told to use a specific algorithm, but it also\nis called from internal codepaths so it cannot be it.\n\nIt is probably not worth introducing an extra helper that only calls\nset_diff_algorithm() and sets ignore_driver_algorithm bit only for\nthat to reduce three-times repetition.\n\nOK.\n\n> diff --git a/userdiff.c b/userdiff.c\n> index d71b82feb74..ff25cfc4b4c 100644\n> --- a/userdiff.c\n> +++ b/userdiff.c\n> @@ -293,7 +293,7 @@ PATTERNS(\"scheme\",\n> \t \"|([^][)(}{[ \\t])+\"),\n> PATTERNS(\"tex\", \"^(\\\\\\\\((sub)*section|chapter|part)\\\\*{0,1}\\\\{.*)$\",\n> \t \"\\\\\\\\[a-zA-Z@]+|\\\\\\\\.|[a-zA-Z0-9\\x80-\\xff]+\"),\n> -{ \"default\", NULL, -1, { NULL, 0 } },\n> +{ \"default\", NULL, NULL, -1, { NULL, 0 } },\n> };\n\nI was surprised that there is so little damage to the built-in\nuserdiff driver definitions, but this is thanks to the PATTERNS()\nand IPATTERN() macro that use designated initializers. Very nice.\n\nNicely done.\n"},{"id":"472285","messageId":"CABPp-BH4042YS7yjXGpaD=1vZOLnTZQY7L2qqpdOvsGmCxO6Dw@mail.gmail.com","threadId":"59197","inReplyTo":"pull.1452.v3.git.git.1676665285.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/2] Teach diff to honor diff algorithms set through git attributes","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-02-18T01:16:04Z","receivedAt":"2023-02-18T01:16:25Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Feb 17, 2023 at 12:21 PM John Cai via GitGitGadget\n wrote:\n>\n> When a repository contains different kinds of files, it may be desirable to\n> use different algorithms based on file type. This is currently not feasible\n> through the command line or using git configs. However, we can leverage the\n> fact that gitattributes are path aware.\n>\n> Teach the diff machinery to check gitattributes when diffing files by using\n> the existing diff. scheme, and add an \"algorithm\" type to the external\n> driver config.\n[...]\n> To address some of the performance concerns in the previous series, a\n> benchmark shows that a performance penalty is no longer incurred, now that\n> we are no longer adding an additional attributes parsing call:\n>\n> $ hyperfine -r 5 -L a bin-wrappers/git,git '{a} diff v2.0.0 v2.28.0'\n> Benchmark 1: git-bin-wrapper diff v2.0.0 v2.28.0 Time (mean ± σ): 1.072 s ±\n> 0.289 s [User: 0.626 s, System: 0.081 s] Range (min … max): 0.772 s … 1.537\n> s 5 runs\n>\n> Benchmark 2: git diff v2.0.0 v2.28.0 Time (mean ± σ): 1.003 s ± 0.065 s\n> [User: 0.684 s, System: 0.067 s] Range (min … max): 0.914 s … 1.091 s 5 runs\n>\n> Summary 'git diff v2.0.0 v2.28.0' ran 1.07 ± 0.30 times faster than\n> 'git-bin-wrapper diff v2.0.0 v2.28.0'\n\nI'm sorry, I don't understand this. What are you measuring? I\npresume bin-wrappers/git refers to the version of git built with your\nchanges, but what version of git does \"git\" refer to? Also, do you\nhave any .gitattributes or .git/config changes present when you are\ntesting to trigger the new functionality you have written?\n\nAlso, doesn't this benchmark demonstrate the opposite of your claim?\nYou said there was no performance penalty, but the benchmark shows a\n7% slowdown. We've battled hard to get smaller improvements than\nthat, so this is still worrisome, even if it's no longer a factor of 2\nor whatever it was. But, again, I'm not sure what is being measured.\nIf the difference is because patience diff was used for some files,\nthen it's not an apples-to-apples comparison, and a 7% slowdown would\nbe no cause for concern.\n\nSince I was curious, I compiled both a version of git from directly\nbefore your series, and directly after, then added a '*.[ch]\ndiff=other' line to the end of .gitattributes, then ran:\n\n$ hyperfine -L a ./older-git,./newer-git '{a} -c\ndiff.other.algorithm=myers diff --numstat v2.0.0 v2.28.0'\nBenchmark 1: ./older-git -c diff.other.algorithm=myers diff --numstat\nv2.0.0 v2.28.0\n Time (mean ± σ): 870.2 ms ± 4.4 ms [User: 755.2 ms, System: 109.8 ms]\n Range (min … max): 861.0 ms … 876.8 ms 10 runs\n\nBenchmark 2: ./newer-git -c diff.other.algorithm=myers diff --numstat\nv2.0.0 v2.28.0\n Time (mean ± σ): 876.9 ms ± 4.8 ms [User: 758.0 ms, System: 113.1 ms]\n Range (min … max): 870.7 ms … 884.1 ms 10 runs\n\nSummary\n './older-git -c diff.other.algorithm=myers diff --numstat v2.0.0 v2.28.0' ran\n 1.01 ± 0.01 times faster than './newer-git -c\ndiff.other.algorithm=myers diff --numstat v2.0.0 v2.28.0'\n\nI specifically specified 'myers' to match what we'd get from the\ndefault anyway, so I would only be testing the slowdown from the\n.gitattribute parsing. So, I think the performance overhead comes out\nto just 1% rather than 7% (and further that's when I make it only\nprint overall stats about the diff rather than the full diff, since I\nknow that's faster. If I didn't do that, the perf hit might appear to\nbe less than 1%).\n"},{"id":"472286","messageId":"CABPp-BFv+=b0WN3rELib4snOnczveRcwb1b7hVJXh851BKWkvg@mail.gmail.com","threadId":"59197","inReplyTo":"816c47aa414586e99aa762604396bd8be4fb11f4.1676665285.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/2] diff: consolidate diff algorithm option parsing","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-02-18T01:36:00Z","receivedAt":"2023-02-18T01:36:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Feb 17, 2023 at 12:21 PM John Cai via GitGitGadget\n wrote:\n>\n> From: John Cai \n>\n> A subsequent commit will need the ability to tell if the diff algorithm\n> was set through the command line through setting a new member of\n> diff_options. While this logic can be added to the\n> diff_opt_diff_algorithm() callback, the `--minimal` and `--histogram`\n> options are handled via OPT_BIT without a callback.\n>\n> Remedy this by consolidating the options parsing logic for --minimal and\n> --histogram into one callback. This way we can modify `diff_options` in\n> that function.\n>\n> As an additional refactor, the logic that sets the diff algorithm in\n> diff_opt_diff_algorithm() can be refactored into a helper that will\n> allow multiple callsites to set the diff algorithm.\n>\n> Signed-off-by: John Cai \n> ---\n> diff.c | 57 +++++++++++++++++++++++++++++++++++++++++++--------------\n> 1 file changed, 43 insertions(+), 14 deletions(-)\n>\n> diff --git a/diff.c b/diff.c\n> index 329eebf16a0..5efc22ca06b 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -3437,6 +3437,22 @@ static int diff_filepair_is_phoney(struct diff_filespec *one,\n> return !DIFF_FILE_VALID(one) && !DIFF_FILE_VALID(two);\n> }\n>\n> +static int set_diff_algorithm(struct diff_options *opts,\n> + const char *alg)\n> +{\n> + long value = parse_algorithm_value(alg);\n> +\n> + if (value < 0)\n> + return -1;\n> +\n> + /* clear out previous settings */\n> + DIFF_XDL_CLR(opts, NEED_MINIMAL);\n> + opts->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n> + opts->xdl_opts |= value;\n> +\n> + return 0;\n> +}\n> +\n> static void builtin_diff(const char *name_a,\n> const char *name_b,\n> struct diff_filespec *one,\n> @@ -5107,17 +5123,28 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n> const char *arg, int unset)\n> {\n> struct diff_options *options = opt->value;\n> - long value = parse_algorithm_value(arg);\n>\n> BUG_ON_OPT_NEG(unset);\n> - if (value < 0)\n> +\n> + if (set_diff_algorithm(options, arg))\n> return error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n> \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n>\n> - /* clear out previous settings */\n> - DIFF_XDL_CLR(options, NEED_MINIMAL);\n> - options->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n> - options->xdl_opts |= value;\n> + return 0;\n> +}\n> +\n> +static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n> + const char *arg, int unset)\n> +{\n> + struct diff_options *options = opt->value;\n> +\n> + BUG_ON_OPT_NEG(unset);\n> + BUG_ON_OPT_ARG(arg);\n> +\n> + if (set_diff_algorithm(options, opt->long_name))\n> + BUG(\"available diff algorithms include \\\"myers\\\", \"\n> + \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n> +\n> return 0;\n> }\n>\n> @@ -5250,7 +5277,6 @@ static int diff_opt_patience(const struct option *opt,\n>\n> BUG_ON_OPT_NEG(unset);\n> BUG_ON_OPT_ARG(arg);\n> - options->xdl_opts = DIFF_WITH_ALG(options, PATIENCE_DIFF);\n> /*\n> * Both --patience and --anchored use PATIENCE_DIFF\n> * internally, so remove any anchors previously\n> @@ -5259,7 +5285,8 @@ static int diff_opt_patience(const struct option *opt,\n> for (i = 0; i < options->anchors_nr; i++)\n> free(options->anchors[i]);\n> options->anchors_nr = 0;\n> - return 0;\n> +\n> + return set_diff_algorithm(options, \"patience\");\n> }\n>\n> static int diff_opt_ignore_regex(const struct option *opt,\n> @@ -5562,9 +5589,10 @@ struct option *add_diff_options(const struct option *opts,\n> N_(\"prevent rename/copy detection if the number of rename/copy targets exceeds given limit\")),\n>\n> OPT_GROUP(N_(\"Diff algorithm options\")),\n> - OPT_BIT(0, \"minimal\", &options->xdl_opts,\n> - N_(\"produce the smallest possible diff\"),\n> - XDF_NEED_MINIMAL),\n> + OPT_CALLBACK_F(0, \"minimal\", options, NULL,\n> + N_(\"produce the smallest possible diff\"),\n> + PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n> + diff_opt_diff_algorithm_no_arg),\n> OPT_BIT_F('w', \"ignore-all-space\", &options->xdl_opts,\n> N_(\"ignore whitespace when comparing lines\"),\n> XDF_IGNORE_WHITESPACE, PARSE_OPT_NONEG),\n> @@ -5590,9 +5618,10 @@ struct option *add_diff_options(const struct option *opts,\n> N_(\"generate diff using the \\\"patience diff\\\" algorithm\"),\n> PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n> diff_opt_patience),\n> - OPT_BITOP(0, \"histogram\", &options->xdl_opts,\n> - N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n> - XDF_HISTOGRAM_DIFF, XDF_DIFF_ALGORITHM_MASK),\n> + OPT_CALLBACK_F(0, \"histogram\", options, NULL,\n> + N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n> + PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n> + diff_opt_diff_algorithm_no_arg),\n> OPT_CALLBACK_F(0, \"diff-algorithm\", options, N_(\"\"),\n> N_(\"choose a diff algorithm\"),\n> PARSE_OPT_NONEG, diff_opt_diff_algorithm),\n> --\n> gitgitgadget\n\nThis patch looks good to me.\n"},{"id":"472290","messageId":"CABPp-BFCMpA=nHtb5RuQL7ACbkhSEKtvmRxKwMuktcf24uQJtQ@mail.gmail.com","threadId":"59197","inReplyTo":"b330222ce83bdf03c20085ff10fcff8a090474d5.1676665285.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 2/2] diff: teach diff to read algorithm from diff driver","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-02-18T02:56:00Z","receivedAt":"2023-02-18T02:57:46Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Feb 17, 2023 at 12:21 PM John Cai via GitGitGadget\n wrote:\n>\n> From: John Cai \n>\n> It can be useful to specify diff algorithms per file type. For example,\n> one may want to use the minimal diff algorithm for .json files, another\n> for .c files, etc.\n>\n> The diff machinery already checks attributes for a diff driver. Teach\n> the diff driver parser a new type \"algorithm\" to look for in the\n> config, which will be used if a driver has been specified through the\n> attributes.\n>\n> Enforce precedence of the diff algorithm by favoring the command line\n> option, then looking at the driver attributes & config combination, then\n> finally the diff.algorithm config.\n>\n> To enforce precedence order, use a new `ignore_driver_algorithm` member\n> during options pasing to indicate the diff algorithm was set via command\n> line args.\n\ns/pasing/parsing/\n\n> Signed-off-by: John Cai \n> ---\n> Documentation/gitattributes.txt | 37 ++++++++++++++++++++++++++++++++\n> diff.c | 33 ++++++++++++++++++++--------\n> diff.h | 1 +\n> t/lib-diff-alternative.sh | 38 ++++++++++++++++++++++++++++++++-\n> userdiff.c | 4 +++-\n> userdiff.h | 1 +\n> 6 files changed, 103 insertions(+), 11 deletions(-)\n>\n> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n> index c19e64ea0ef..f212079a131 100644\n> --- a/Documentation/gitattributes.txt\n> +++ b/Documentation/gitattributes.txt\n> @@ -758,6 +758,43 @@ with the above configuration, i.e. `j-c-diff`, with 7\n> parameters, just like `GIT_EXTERNAL_DIFF` program is called.\n> See linkgit:git[1] for details.\n>\n> +Setting the internal diff algorithm\n> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> +\n> +The diff algorithm can be set through the `diff.algorithm` config key, but\n> +sometimes it may be helpful to set the diff algorithm by path. For example, one\n> +might wish to set a diff algorithm automatically for all `.json` files such that\n> +the user would not need to pass in a separate command line `--diff-algorithm`\n> +flag each time.\n> +\n> +First, in `.gitattributes`, assign the `diff` attribute for paths.\n> +\n> +------------------------\n> +*.json diff=\n> +------------------------\n> +\n> +Then, define a \"diff..algorithm\" configuration to specify the diff\n\nShould there be a link to `git-config` right after \"configuration\"?\nOtherwise, users may think that they are being told to specify\nadditional configuration within the .gitattributes file.\n\n> +algorithm, choosing from `meyers`, `patience`, `minimal`, or `histogram`.\n\ns/meyers/myers/\n\n> +\n> +----------------------------------------------------------------\n> +[diff \"\"]\n> + algorithm = histogram\n> +----------------------------------------------------------------\n\nIt's pretty easy to assume the above is meant to be part of the\n.gitattributes file instead of the .git/config file. Don't most users\nrun `git config` commands directly rather than edit the .git/config\nfile? Should we provide a sample command rather than showing what the\nconfig file will contain?\n\n> +\n> +This diff algorithm applies to user facing diff output like git-diff(1),\n> +git-show(1) and is used for the `--stat` output as well. The merge machinery\n> +will not use the diff algorithm set through this method.\n\nYaay, thanks for including this!\n\n\nI'm still curious if this should this also include warnings/caveats, such as:\n * The diff attribute specified in .gitattributes will be ignored in\na bare clone\n * The diff attribute specified in .gitattributes will be ignored if\nit is only specified in another branch (e.g. on a branch \"special-file\ndiff=patience\" recorded in .gitattributes, then checkout master but\nrun `git log -1 -p $branch`)\n * When a file is renamed, the diff attribute for the pre-image name\nis the only one the system pays attention to (thus adding \"-R\" can\nflip which diff algorithm is run for the renamed file).\n\nAlso, since I tested the three items above to verify they are valid\nwarnings, I'm a bit confused. I thought your intent was to use this\nserver-side[1], so isn't the bare clone aspect a deal-breaker for your\nintended usecase?\n\n[1] https://lore.kernel.org/git/7852AC7B-7A4E-4DD0-ADEA-CFFD5D16C595@gmail.com/\n\n> +\n> +NOTE: If the `command` key also exists, then Git will treat this as an external\n> +diff and attempt to use the value set for `command` as an external program. For\n> +instance, the following config, combined with the above `.gitattributes` file,\n> +will result in `command` favored over `algorithm`.\n> +\n> +----------------------------------------------------------------\n> +[diff \"\"]\n> + command = j-c-diff\n> + algorithm = histogram\n> +----------------------------------------------------------------\n>\n> Defining a custom hunk-header\n> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> diff --git a/diff.c b/diff.c\n> index 5efc22ca06b..04469da6d34 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -4456,15 +4456,13 @@ static void run_diff_cmd(const char *pgm,\n> const char *xfrm_msg = NULL;\n> int complete_rewrite = (p->status == DIFF_STATUS_MODIFIED) && p->score;\n> int must_show_header = 0;\n> + struct userdiff_driver *drv = NULL;\n>\n> -\n> - if (o->flags.allow_external) {\n> - struct userdiff_driver *drv;\n> -\n> + if (o->flags.allow_external || !o->ignore_driver_algorithm)\n> drv = userdiff_find_by_path(o->repo->index, attr_path);\n> - if (drv && drv->external)\n> - pgm = drv->external;\n> - }\n> +\n> + if (o->flags.allow_external && drv && drv->external)\n> + pgm = drv->external;\n>\n> if (msg) {\n> /*\n> @@ -4481,12 +4479,16 @@ static void run_diff_cmd(const char *pgm,\n> run_external_diff(pgm, name, other, one, two, xfrm_msg, o);\n> return;\n> }\n> - if (one && two)\n> + if (one && two) {\n> + if (drv && !o->ignore_driver_algorithm && drv->algorithm)\n> + set_diff_algorithm(o, drv->algorithm);\n> +\n> builtin_diff(name, other ? other : name,\n> one, two, xfrm_msg, must_show_header,\n> o, complete_rewrite);\n> - else\n> + } else {\n> fprintf(o->file, \"* Unmerged path %s\\n\", name);\n> + }\n> }\n>\n> static void diff_fill_oid_info(struct diff_filespec *one, struct index_state *istate)\n> @@ -4583,6 +4585,14 @@ static void run_diffstat(struct diff_filepair *p, struct diff_options *o,\n> const char *name;\n> const char *other;\n>\n> + if (!o->ignore_driver_algorithm) {\n> + struct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, p->one->path);\n> +\n> + if (drv && drv->algorithm) {\n> + set_diff_algorithm(o, drv->algorithm);\n> + }\n> + }\n> +\n> if (DIFF_PAIR_UNMERGED(p)) {\n> /* unmerged */\n> builtin_diffstat(p->one->path, NULL, NULL, NULL,\n> @@ -5130,6 +5140,8 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n> return error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n> \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n>\n> + options->ignore_driver_algorithm = 1;\n> +\n> return 0;\n> }\n>\n> @@ -5145,6 +5157,8 @@ static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n> BUG(\"available diff algorithms include \\\"myers\\\", \"\n> \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n>\n> + options->ignore_driver_algorithm = 1;\n> +\n> return 0;\n> }\n>\n> @@ -5285,6 +5299,7 @@ static int diff_opt_patience(const struct option *opt,\n> for (i = 0; i < options->anchors_nr; i++)\n> free(options->anchors[i]);\n> options->anchors_nr = 0;\n> + options->ignore_driver_algorithm = 1;\n>\n> return set_diff_algorithm(options, \"patience\");\n> }\n> diff --git a/diff.h b/diff.h\n> index 41eb2c3d428..8d770b1d579 100644\n> --- a/diff.h\n> +++ b/diff.h\n> @@ -333,6 +333,7 @@ struct diff_options {\n> int prefix_length;\n> const char *stat_sep;\n> int xdl_opts;\n> + int ignore_driver_algorithm;\n>\n> /* see Documentation/diff-options.txt */\n> char **anchors;\n> diff --git a/t/lib-diff-alternative.sh b/t/lib-diff-alternative.sh\n> index 8d1e408bb58..2dc02bca873 100644\n> --- a/t/lib-diff-alternative.sh\n> +++ b/t/lib-diff-alternative.sh\n> @@ -105,10 +105,46 @@ index $file1..$file2 100644\n> }\n> EOF\n>\n> + cat >expect_diffstat < + file1 => file2 | 21 ++++++++++-----------\n> + 1 file changed, 10 insertions(+), 11 deletions(-)\n> +EOF\n> +\n> STRATEGY=$1\n>\n> + test_expect_success \"$STRATEGY diff from attributes\" '\n> + echo \"file* diff=driver\" >.gitattributes &&\n> + git config diff.driver.algorithm \"$STRATEGY\" &&\n> + test_must_fail git diff --no-index file1 file2 > output &&\n> + cat expect &&\n> + cat output &&\n> + test_cmp expect output\n> + '\n> +\n> + test_expect_success \"$STRATEGY diff from attributes has valid diffstat\" '\n> + echo \"file* diff=driver\" >.gitattributes &&\n> + git config diff.driver.algorithm \"$STRATEGY\" &&\n> + test_must_fail git diff --stat --no-index file1 file2 > output &&\n> + test_cmp expect_diffstat output\n> + '\n> +\n> test_expect_success \"$STRATEGY diff\" '\n> - test_must_fail git diff --no-index \"--$STRATEGY\" file1 file2 > output &&\n> + test_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n> + test_cmp expect output\n> + '\n> +\n> + test_expect_success \"$STRATEGY diff command line precedence before attributes\" '\n> + echo \"file* diff=driver\" >.gitattributes &&\n> + git config diff.driver.algorithm meyers &&\n\nIs this misspelling of myers intentional? I think with the typo, the\ncode falls back to the default algorithm, which happens to be myers,\nso I think the test works either way, but were you intending to test\nfallback behavior in case of a typo here, or was that accidental?\n\n\n\n> + test_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n> + test_cmp expect output\n> + '\n> +\n> + test_expect_success \"$STRATEGY diff attributes precedence before config\" '\n> + git config diff.algorithm default &&\n> + echo \"file* diff=driver\" >.gitattributes &&\n> + git config diff.driver.algorithm \"$STRATEGY\" &&\n> + test_must_fail git diff --no-index file1 file2 > output &&\n> test_cmp expect output\n> '\n>\n> diff --git a/userdiff.c b/userdiff.c\n> index d71b82feb74..ff25cfc4b4c 100644\n> --- a/userdiff.c\n> +++ b/userdiff.c\n> @@ -293,7 +293,7 @@ PATTERNS(\"scheme\",\n> \"|([^][)(}{[ \\t])+\"),\n> PATTERNS(\"tex\", \"^(\\\\\\\\((sub)*section|chapter|part)\\\\*{0,1}\\\\{.*)$\",\n> \"\\\\\\\\[a-zA-Z@]+|\\\\\\\\.|[a-zA-Z0-9\\x80-\\xff]+\"),\n> -{ \"default\", NULL, -1, { NULL, 0 } },\n> +{ \"default\", NULL, NULL, -1, { NULL, 0 } },\n> };\n> #undef PATTERNS\n> #undef IPATTERN\n> @@ -394,6 +394,8 @@ int userdiff_config(const char *k, const char *v)\n> return parse_bool(&drv->textconv_want_cache, k, v);\n> if (!strcmp(type, \"wordregex\"))\n> return git_config_string(&drv->word_regex, k, v);\n> + if (!strcmp(type, \"algorithm\"))\n> + return git_config_string(&drv->algorithm, k, v);\n>\n> return 0;\n> }\n> diff --git a/userdiff.h b/userdiff.h\n> index aee91bc77e6..24419db6973 100644\n> --- a/userdiff.h\n> +++ b/userdiff.h\n> @@ -14,6 +14,7 @@ struct userdiff_funcname {\n> struct userdiff_driver {\n> const char *name;\n> const char *external;\n> + const char *algorithm;\n> int binary;\n> struct userdiff_funcname funcname;\n> const char *word_regex;\n> --\n> gitgitgadget\n"},{"id":"472331","messageId":"9C99CD32-79B4-4754-AE9A-267C29ECE19D@gmail.com","threadId":"59197","inReplyTo":"CABPp-BH4042YS7yjXGpaD=1vZOLnTZQY7L2qqpdOvsGmCxO6Dw@mail.gmail.com","subject":"Re: [PATCH v3 0/2] Teach diff to honor diff algorithms set through git attributes","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-20T13:37:10Z","receivedAt":"2023-02-20T13:37:17Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Elijah,\n\nOn 17 Feb 2023, at 20:16, Elijah Newren wrote:\n\n> On Fri, Feb 17, 2023 at 12:21 PM John Cai via GitGitGadget\n> wrote:\n>>\n>> When a repository contains different kinds of files, it may be desirable to\n>> use different algorithms based on file type. This is currently not feasible\n>> through the command line or using git configs. However, we can leverage the\n>> fact that gitattributes are path aware.\n>>\n>> Teach the diff machinery to check gitattributes when diffing files by using\n>> the existing diff. scheme, and add an \"algorithm\" type to the external\n>> driver config.\n> [...]\n>> To address some of the performance concerns in the previous series, a\n>> benchmark shows that a performance penalty is no longer incurred, now that\n>> we are no longer adding an additional attributes parsing call:\n>>\n>> $ hyperfine -r 5 -L a bin-wrappers/git,git '{a} diff v2.0.0 v2.28.0'\n>> Benchmark 1: git-bin-wrapper diff v2.0.0 v2.28.0 Time (mean ± σ): 1.072 s ±\n>> 0.289 s [User: 0.626 s, System: 0.081 s] Range (min … max): 0.772 s … 1.537\n>> s 5 runs\n>>\n>> Benchmark 2: git diff v2.0.0 v2.28.0 Time (mean ± σ): 1.003 s ± 0.065 s\n>> [User: 0.684 s, System: 0.067 s] Range (min … max): 0.914 s … 1.091 s 5 runs\n>>\n>> Summary 'git diff v2.0.0 v2.28.0' ran 1.07 ± 0.30 times faster than\n>> 'git-bin-wrapper diff v2.0.0 v2.28.0'\n>\n> I'm sorry, I don't understand this. What are you measuring? I\n> presume bin-wrappers/git refers to the version of git built with your\n> changes, but what version of git does \"git\" refer to? Also, do you\n> have any .gitattributes or .git/config changes present when you are\n> testing to trigger the new functionality you have written?\n>\n> Also, doesn't this benchmark demonstrate the opposite of your claim?\n> You said there was no performance penalty, but the benchmark shows a\n> 7% slowdown. We've battled hard to get smaller improvements than\n> that, so this is still worrisome, even if it's no longer a factor of 2\n> or whatever it was. But, again, I'm not sure what is being measured.\n> If the difference is because patience diff was used for some files,\n> then it's not an apples-to-apples comparison, and a 7% slowdown would\n> be no cause for concern.\n>\n> Since I was curious, I compiled both a version of git from directly\n> before your series, and directly after, then added a '*.[ch]\n> diff=other' line to the end of .gitattributes, then ran:\n>\n> $ hyperfine -L a ./older-git,./newer-git '{a} -c\n> diff.other.algorithm=myers diff --numstat v2.0.0 v2.28.0'\n> Benchmark 1: ./older-git -c diff.other.algorithm=myers diff --numstat\n> v2.0.0 v2.28.0\n> Time (mean ± σ): 870.2 ms ± 4.4 ms [User: 755.2 ms, System: 109.8 ms]\n> Range (min … max): 861.0 ms … 876.8 ms 10 runs\n>\n> Benchmark 2: ./newer-git -c diff.other.algorithm=myers diff --numstat\n> v2.0.0 v2.28.0\n> Time (mean ± σ): 876.9 ms ± 4.8 ms [User: 758.0 ms, System: 113.1 ms]\n> Range (min … max): 870.7 ms … 884.1 ms 10 runs\n>\n> Summary\n> './older-git -c diff.other.algorithm=myers diff --numstat v2.0.0 v2.28.0' ran\n> 1.01 ± 0.01 times faster than './newer-git -c\n> diff.other.algorithm=myers diff --numstat v2.0.0 v2.28.0'\n>\n> I specifically specified 'myers' to match what we'd get from the\n> default anyway, so I would only be testing the slowdown from the\n> .gitattribute parsing. So, I think the performance overhead comes out\n> to just 1% rather than 7% (and further that's when I make it only\n> print overall stats about the diff rather than the full diff, since I\n> know that's faster. If I didn't do that, the perf hit might appear to\n> be less than 1%).\n\nThanks for taking the time to do this! I should have been a bit more careful\nabout this benchmark, and more explicit about what it was benchmarking. I just\nran it again and made sure that the same algorithm was used, and I got results\nsimilar to you.\n\nWill update the cover letter, thanks!\n"},{"id":"472338","messageId":"EE7565DF-BE70-4C45-AF0B-95C85050DFA4@gmail.com","threadId":"59197","inReplyTo":"CABPp-BFCMpA=nHtb5RuQL7ACbkhSEKtvmRxKwMuktcf24uQJtQ@mail.gmail.com","subject":"Re: [PATCH v3 2/2] diff: teach diff to read algorithm from diff driver","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-20T15:32:52Z","receivedAt":"2023-02-20T15:33:08Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Elijah,\n\nOn 17 Feb 2023, at 21:56, Elijah Newren wrote:\n\n> On Fri, Feb 17, 2023 at 12:21 PM John Cai via GitGitGadget\n> wrote:\n>>\n>> From: John Cai \n>>\n>> It can be useful to specify diff algorithms per file type. For example,\n>> one may want to use the minimal diff algorithm for .json files, another\n>> for .c files, etc.\n>>\n>> The diff machinery already checks attributes for a diff driver. Teach\n>> the diff driver parser a new type \"algorithm\" to look for in the\n>> config, which will be used if a driver has been specified through the\n>> attributes.\n>>\n>> Enforce precedence of the diff algorithm by favoring the command line\n>> option, then looking at the driver attributes & config combination, then\n>> finally the diff.algorithm config.\n>>\n>> To enforce precedence order, use a new `ignore_driver_algorithm` member\n>> during options pasing to indicate the diff algorithm was set via command\n>> line args.\n>\n> s/pasing/parsing/\n\nthanks for noticing this!\n\n>\n>> Signed-off-by: John Cai \n>> ---\n>> Documentation/gitattributes.txt | 37 ++++++++++++++++++++++++++++++++\n>> diff.c | 33 ++++++++++++++++++++--------\n>> diff.h | 1 +\n>> t/lib-diff-alternative.sh | 38 ++++++++++++++++++++++++++++++++-\n>> userdiff.c | 4 +++-\n>> userdiff.h | 1 +\n>> 6 files changed, 103 insertions(+), 11 deletions(-)\n>>\n>> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n>> index c19e64ea0ef..f212079a131 100644\n>> --- a/Documentation/gitattributes.txt\n>> +++ b/Documentation/gitattributes.txt\n>> @@ -758,6 +758,43 @@ with the above configuration, i.e. `j-c-diff`, with 7\n>> parameters, just like `GIT_EXTERNAL_DIFF` program is called.\n>> See linkgit:git[1] for details.\n>>\n>> +Setting the internal diff algorithm\n>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n>> +\n>> +The diff algorithm can be set through the `diff.algorithm` config key, but\n>> +sometimes it may be helpful to set the diff algorithm by path. For example, one\n>> +might wish to set a diff algorithm automatically for all `.json` files such that\n>> +the user would not need to pass in a separate command line `--diff-algorithm`\n>> +flag each time.\n>> +\n>> +First, in `.gitattributes`, assign the `diff` attribute for paths.\n>> +\n>> +------------------------\n>> +*.json diff=\n>> +------------------------\n>> +\n>> +Then, define a \"diff..algorithm\" configuration to specify the diff\n>\n> Should there be a link to `git-config` right after \"configuration\"?\n> Otherwise, users may think that they are being told to specify\n> additional configuration within the .gitattributes file.\n>\n>> +algorithm, choosing from `meyers`, `patience`, `minimal`, or `histogram`.\n>\n> s/meyers/myers/\n\nlikewise.\n\n>\n>> +\n>> +----------------------------------------------------------------\n>> +[diff \"\"]\n>> + algorithm = histogram\n>> +----------------------------------------------------------------\n>\n> It's pretty easy to assume the above is meant to be part of the\n> .gitattributes file instead of the .git/config file. Don't most users\n> run `git config` commands directly rather than edit the .git/config\n> file? Should we provide a sample command rather than showing what the\n> config file will contain?\n\nNot sure--I think I was was just following what I saw in the existing\ndocumentation for driver configuration. But would be interested to see what\nother folks think.\n\n>\n>> +\n>> +This diff algorithm applies to user facing diff output like git-diff(1),\n>> +git-show(1) and is used for the `--stat` output as well. The merge machinery\n>> +will not use the diff algorithm set through this method.\n>\n> Yaay, thanks for including this!\n>\n>\n> I'm still curious if this should this also include warnings/caveats, such as:\n> * The diff attribute specified in .gitattributes will be ignored in\n> a bare clone\n> * The diff attribute specified in .gitattributes will be ignored if\n> it is only specified in another branch (e.g. on a branch \"special-file\n> diff=patience\" recorded in .gitattributes, then checkout master but\n> run `git log -1 -p $branch`)\n> * When a file is renamed, the diff attribute for the pre-image name\n> is the only one the system pays attention to (thus adding \"-R\" can\n> flip which diff algorithm is run for the renamed file).\n\nI would be fine with adding that--though originally I was thinking that these\ncan be inferred from the way that gitattributes are documented in [1]. Calling\nthese out would make it more clear though, so I could go either way.\n\n>\n> Also, since I tested the three items above to verify they are valid\n> warnings, I'm a bit confused. I thought your intent was to use this\n> server-side[1], so isn't the bare clone aspect a deal-breaker for your\n> intended usecase?\n>\n> [1] https://lore.kernel.org/git/7852AC7B-7A4E-4DD0-ADEA-CFFD5D16C595@gmail.com/\n\nyes, indeed. I was planning on adding bare repository support in a separate\npatch series, since the additions in [2] allows .gitattributes to be read from a\nbare repository.\n\n1. https://git-scm.com/docs/gitattributes\n2. https://lore.kernel.org/git/0ca8b2458921fc40269b0c43b5ec86eba77d6b54.1673684790.git.karthik.188@gmail.com/\n\nthanks!\nJohn\n\n>\n>> +\n>> +NOTE: If the `command` key also exists, then Git will treat this as an external\n>> +diff and attempt to use the value set for `command` as an external program. For\n>> +instance, the following config, combined with the above `.gitattributes` file,\n>> +will result in `command` favored over `algorithm`.\n>> +\n>> +----------------------------------------------------------------\n>> +[diff \"\"]\n>> + command = j-c-diff\n>> + algorithm = histogram\n>> +----------------------------------------------------------------\n>>\n>> Defining a custom hunk-header\n>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n>> diff --git a/diff.c b/diff.c\n>> index 5efc22ca06b..04469da6d34 100644\n>> --- a/diff.c\n>> +++ b/diff.c\n>> @@ -4456,15 +4456,13 @@ static void run_diff_cmd(const char *pgm,\n>> const char *xfrm_msg = NULL;\n>> int complete_rewrite = (p->status == DIFF_STATUS_MODIFIED) && p->score;\n>> int must_show_header = 0;\n>> + struct userdiff_driver *drv = NULL;\n>>\n>> -\n>> - if (o->flags.allow_external) {\n>> - struct userdiff_driver *drv;\n>> -\n>> + if (o->flags.allow_external || !o->ignore_driver_algorithm)\n>> drv = userdiff_find_by_path(o->repo->index, attr_path);\n>> - if (drv && drv->external)\n>> - pgm = drv->external;\n>> - }\n>> +\n>> + if (o->flags.allow_external && drv && drv->external)\n>> + pgm = drv->external;\n>>\n>> if (msg) {\n>> /*\n>> @@ -4481,12 +4479,16 @@ static void run_diff_cmd(const char *pgm,\n>> run_external_diff(pgm, name, other, one, two, xfrm_msg, o);\n>> return;\n>> }\n>> - if (one && two)\n>> + if (one && two) {\n>> + if (drv && !o->ignore_driver_algorithm && drv->algorithm)\n>> + set_diff_algorithm(o, drv->algorithm);\n>> +\n>> builtin_diff(name, other ? other : name,\n>> one, two, xfrm_msg, must_show_header,\n>> o, complete_rewrite);\n>> - else\n>> + } else {\n>> fprintf(o->file, \"* Unmerged path %s\\n\", name);\n>> + }\n>> }\n>>\n>> static void diff_fill_oid_info(struct diff_filespec *one, struct index_state *istate)\n>> @@ -4583,6 +4585,14 @@ static void run_diffstat(struct diff_filepair *p, struct diff_options *o,\n>> const char *name;\n>> const char *other;\n>>\n>> + if (!o->ignore_driver_algorithm) {\n>> + struct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, p->one->path);\n>> +\n>> + if (drv && drv->algorithm) {\n>> + set_diff_algorithm(o, drv->algorithm);\n>> + }\n>> + }\n>> +\n>> if (DIFF_PAIR_UNMERGED(p)) {\n>> /* unmerged */\n>> builtin_diffstat(p->one->path, NULL, NULL, NULL,\n>> @@ -5130,6 +5140,8 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n>> return error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n>> \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n>>\n>> + options->ignore_driver_algorithm = 1;\n>> +\n>> return 0;\n>> }\n>>\n>> @@ -5145,6 +5157,8 @@ static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n>> BUG(\"available diff algorithms include \\\"myers\\\", \"\n>> \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n>>\n>> + options->ignore_driver_algorithm = 1;\n>> +\n>> return 0;\n>> }\n>>\n>> @@ -5285,6 +5299,7 @@ static int diff_opt_patience(const struct option *opt,\n>> for (i = 0; i < options->anchors_nr; i++)\n>> free(options->anchors[i]);\n>> options->anchors_nr = 0;\n>> + options->ignore_driver_algorithm = 1;\n>>\n>> return set_diff_algorithm(options, \"patience\");\n>> }\n>> diff --git a/diff.h b/diff.h\n>> index 41eb2c3d428..8d770b1d579 100644\n>> --- a/diff.h\n>> +++ b/diff.h\n>> @@ -333,6 +333,7 @@ struct diff_options {\n>> int prefix_length;\n>> const char *stat_sep;\n>> int xdl_opts;\n>> + int ignore_driver_algorithm;\n>>\n>> /* see Documentation/diff-options.txt */\n>> char **anchors;\n>> diff --git a/t/lib-diff-alternative.sh b/t/lib-diff-alternative.sh\n>> index 8d1e408bb58..2dc02bca873 100644\n>> --- a/t/lib-diff-alternative.sh\n>> +++ b/t/lib-diff-alternative.sh\n>> @@ -105,10 +105,46 @@ index $file1..$file2 100644\n>> }\n>> EOF\n>>\n>> + cat >expect_diffstat <> + file1 => file2 | 21 ++++++++++-----------\n>> + 1 file changed, 10 insertions(+), 11 deletions(-)\n>> +EOF\n>> +\n>> STRATEGY=$1\n>>\n>> + test_expect_success \"$STRATEGY diff from attributes\" '\n>> + echo \"file* diff=driver\" >.gitattributes &&\n>> + git config diff.driver.algorithm \"$STRATEGY\" &&\n>> + test_must_fail git diff --no-index file1 file2 > output &&\n>> + cat expect &&\n>> + cat output &&\n>> + test_cmp expect output\n>> + '\n>> +\n>> + test_expect_success \"$STRATEGY diff from attributes has valid diffstat\" '\n>> + echo \"file* diff=driver\" >.gitattributes &&\n>> + git config diff.driver.algorithm \"$STRATEGY\" &&\n>> + test_must_fail git diff --stat --no-index file1 file2 > output &&\n>> + test_cmp expect_diffstat output\n>> + '\n>> +\n>> test_expect_success \"$STRATEGY diff\" '\n>> - test_must_fail git diff --no-index \"--$STRATEGY\" file1 file2 > output &&\n>> + test_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n>> + test_cmp expect output\n>> + '\n>> +\n>> + test_expect_success \"$STRATEGY diff command line precedence before attributes\" '\n>> + echo \"file* diff=driver\" >.gitattributes &&\n>> + git config diff.driver.algorithm meyers &&\n>\n> Is this misspelling of myers intentional? I think with the typo, the\n> code falls back to the default algorithm, which happens to be myers,\n> so I think the test works either way, but were you intending to test\n> fallback behavior in case of a typo here, or was that accidental?\n>\n>\n>\n>> + test_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n>> + test_cmp expect output\n>> + '\n>> +\n>> + test_expect_success \"$STRATEGY diff attributes precedence before config\" '\n>> + git config diff.algorithm default &&\n>> + echo \"file* diff=driver\" >.gitattributes &&\n>> + git config diff.driver.algorithm \"$STRATEGY\" &&\n>> + test_must_fail git diff --no-index file1 file2 > output &&\n>> test_cmp expect output\n>> '\n>>\n>> diff --git a/userdiff.c b/userdiff.c\n>> index d71b82feb74..ff25cfc4b4c 100644\n>> --- a/userdiff.c\n>> +++ b/userdiff.c\n>> @@ -293,7 +293,7 @@ PATTERNS(\"scheme\",\n>> \"|([^][)(}{[ \\t])+\"),\n>> PATTERNS(\"tex\", \"^(\\\\\\\\((sub)*section|chapter|part)\\\\*{0,1}\\\\{.*)$\",\n>> \"\\\\\\\\[a-zA-Z@]+|\\\\\\\\.|[a-zA-Z0-9\\x80-\\xff]+\"),\n>> -{ \"default\", NULL, -1, { NULL, 0 } },\n>> +{ \"default\", NULL, NULL, -1, { NULL, 0 } },\n>> };\n>> #undef PATTERNS\n>> #undef IPATTERN\n>> @@ -394,6 +394,8 @@ int userdiff_config(const char *k, const char *v)\n>> return parse_bool(&drv->textconv_want_cache, k, v);\n>> if (!strcmp(type, \"wordregex\"))\n>> return git_config_string(&drv->word_regex, k, v);\n>> + if (!strcmp(type, \"algorithm\"))\n>> + return git_config_string(&drv->algorithm, k, v);\n>>\n>> return 0;\n>> }\n>> diff --git a/userdiff.h b/userdiff.h\n>> index aee91bc77e6..24419db6973 100644\n>> --- a/userdiff.h\n>> +++ b/userdiff.h\n>> @@ -14,6 +14,7 @@ struct userdiff_funcname {\n>> struct userdiff_driver {\n>> const char *name;\n>> const char *external;\n>> + const char *algorithm;\n>> int binary;\n>> struct userdiff_funcname funcname;\n>> const char *word_regex;\n>> --\n>> gitgitgadget\n"},{"id":"472339","messageId":"CABPp-BGDi1VQXFdGw_Y8i0ZDBOoHJe9039fh4mO44qJ-nJE1ig@mail.gmail.com","threadId":"59197","inReplyTo":"EE7565DF-BE70-4C45-AF0B-95C85050DFA4@gmail.com","subject":"Re: [PATCH v3 2/2] diff: teach diff to read algorithm from diff driver","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-02-20T16:21:22Z","receivedAt":"2023-02-20T16:21:40Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Feb 20, 2023 at 7:32 AM John Cai wrote:\n[...]\n> > I'm still curious if this should this also include warnings/caveats, such as:\n> > * The diff attribute specified in .gitattributes will be ignored in\n> > a bare clone\n> > * The diff attribute specified in .gitattributes will be ignored if\n> > it is only specified in another branch (e.g. on a branch \"special-file\n> > diff=patience\" recorded in .gitattributes, then checkout master but\n> > run `git log -1 -p $branch`)\n> > * When a file is renamed, the diff attribute for the pre-image name\n> > is the only one the system pays attention to (thus adding \"-R\" can\n> > flip which diff algorithm is run for the renamed file).\n>\n> I would be fine with adding that--though originally I was thinking that these\n> can be inferred from the way that gitattributes are documented in [1]. Calling\n> these out would make it more clear though, so I could go either way.\n>\n> >\n> > Also, since I tested the three items above to verify they are valid\n> > warnings, I'm a bit confused. I thought your intent was to use this\n> > server-side[1], so isn't the bare clone aspect a deal-breaker for your\n> > intended usecase?\n> >\n> > [1] https://lore.kernel.org/git/7852AC7B-7A4E-4DD0-ADEA-CFFD5D16C595@gmail.com/\n>\n> yes, indeed. I was planning on adding bare repository support in a separate\n> patch series, since the additions in [2] allows .gitattributes to be read from a\n> bare repository.\n>\n> 1. https://git-scm.com/docs/gitattributes\n> 2. https://lore.kernel.org/git/0ca8b2458921fc40269b0c43b5ec86eba77d6b54.1673684790.git.karthik.188@gmail.com/\n>\n> thanks!\n> John\n\nOh, interesting, I didn't know about [2]. So, is the plan to take the\n--source option from that series and add it to diff (perhaps with a\ndifferent name, since log tends to consume diff options and --source\nis already taken)?\n\nAnd do you expect to get the tree-ish from the two the users are\nalready specifying to diff? If so, which one do you use (the two\ncommits being diffed might have differing .gitattributes files)? If\nnot, what does that mean for users of e.g. the GitLab UI who have to\nspecify a third tree when diffing?\n"},{"id":"472343","messageId":"47981D9E-9DC7-4C23-911D-13BA52A27040@gmail.com","threadId":"59197","inReplyTo":"CABPp-BGDi1VQXFdGw_Y8i0ZDBOoHJe9039fh4mO44qJ-nJE1ig@mail.gmail.com","subject":"Re: [PATCH v3 2/2] diff: teach diff to read algorithm from diff driver","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-20T16:49:08Z","receivedAt":"2023-02-20T16:49:14Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Elijah,\n\nOn 20 Feb 2023, at 11:21, Elijah Newren wrote:\n\n> On Mon, Feb 20, 2023 at 7:32 AM John Cai wrote:\n> [...]\n>>> I'm still curious if this should this also include warnings/caveats, such as:\n>>> * The diff attribute specified in .gitattributes will be ignored in\n>>> a bare clone\n>>> * The diff attribute specified in .gitattributes will be ignored if\n>>> it is only specified in another branch (e.g. on a branch \"special-file\n>>> diff=patience\" recorded in .gitattributes, then checkout master but\n>>> run `git log -1 -p $branch`)\n>>> * When a file is renamed, the diff attribute for the pre-image name\n>>> is the only one the system pays attention to (thus adding \"-R\" can\n>>> flip which diff algorithm is run for the renamed file).\n>>\n>> I would be fine with adding that--though originally I was thinking that these\n>> can be inferred from the way that gitattributes are documented in [1]. Calling\n>> these out would make it more clear though, so I could go either way.\n>>\n>>>\n>>> Also, since I tested the three items above to verify they are valid\n>>> warnings, I'm a bit confused. I thought your intent was to use this\n>>> server-side[1], so isn't the bare clone aspect a deal-breaker for your\n>>> intended usecase?\n>>>\n>>> [1] https://lore.kernel.org/git/7852AC7B-7A4E-4DD0-ADEA-CFFD5D16C595@gmail.com/\n>>\n>> yes, indeed. I was planning on adding bare repository support in a separate\n>> patch series, since the additions in [2] allows .gitattributes to be read from a\n>> bare repository.\n>>\n>> 1. https://git-scm.com/docs/gitattributes\n>> 2. https://lore.kernel.org/git/0ca8b2458921fc40269b0c43b5ec86eba77d6b54.1673684790.git.karthik.188@gmail.com/\n>>\n>> thanks!\n>> John\n>\n> Oh, interesting, I didn't know about [2]. So, is the plan to take the\n> --source option from that series and add it to diff (perhaps with a\n> different name, since log tends to consume diff options and --source\n> is already taken)?\n\nYep, that would be the general idea\n\n>\n> And do you expect to get the tree-ish from the two the users are\n> already specifying to diff? If so, which one do you use (the two\n> commits being diffed might have differing .gitattributes files)? If\n> not, what does that mean for users of e.g. the GitLab UI who have to\n> specify a third tree when diffing?\n\nGood question! Since it seems that when `git-diff(1)` considers diff.,\nit goes with the path of the first one. (might need some confirmation here)\n\nin diff.c:\n\n\nstatic void run_diff(struct diff_filepair *p, struct diff_options *o)\n{\n\tconst char *pgm = external_diff();\n\tstruct strbuf msg;\n\tstruct diff_filespec *one = p->one;\n\tstruct diff_filespec *two = p->two;\n\tconst char *name;\n\tconst char *other;\n\tconst char *attr_path;\n\n\tname = one->path;\n\tother = (strcmp(name, two->path) ? two->path : NULL);\n\tattr_path = name;\n\tif (o->prefix_length)\n\nI was thinking we would just use the tree-ish of the first one\n\nthanks\nJohn\n"},{"id":"472348","messageId":"CABPp-BHQn0sjAMwJ+r6uenO=nGLG1HvfnhS6tG8mu1BWt4bdOw@mail.gmail.com","threadId":"59197","inReplyTo":"47981D9E-9DC7-4C23-911D-13BA52A27040@gmail.com","subject":"Re: [PATCH v3 2/2] diff: teach diff to read algorithm from diff driver","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-02-20T17:32:52Z","receivedAt":"2023-02-20T17:33:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Feb 20, 2023 at 8:49 AM John Cai wrote:\n>\n> Hi Elijah,\n>\n> On 20 Feb 2023, at 11:21, Elijah Newren wrote:\n>\n> > On Mon, Feb 20, 2023 at 7:32 AM John Cai wrote:\n> > [...]\n> >>> I'm still curious if this should this also include warnings/caveats, such as:\n> >>> * The diff attribute specified in .gitattributes will be ignored in\n> >>> a bare clone\n> >>> * The diff attribute specified in .gitattributes will be ignored if\n> >>> it is only specified in another branch (e.g. on a branch \"special-file\n> >>> diff=patience\" recorded in .gitattributes, then checkout master but\n> >>> run `git log -1 -p $branch`)\n> >>> * When a file is renamed, the diff attribute for the pre-image name\n> >>> is the only one the system pays attention to (thus adding \"-R\" can\n> >>> flip which diff algorithm is run for the renamed file).\n> >>\n> >> I would be fine with adding that--though originally I was thinking that these\n> >> can be inferred from the way that gitattributes are documented in [1]. Calling\n> >> these out would make it more clear though, so I could go either way.\n> >>\n> >>>\n> >>> Also, since I tested the three items above to verify they are valid\n> >>> warnings, I'm a bit confused. I thought your intent was to use this\n> >>> server-side[1], so isn't the bare clone aspect a deal-breaker for your\n> >>> intended usecase?\n> >>>\n> >>> [1] https://lore.kernel.org/git/7852AC7B-7A4E-4DD0-ADEA-CFFD5D16C595@gmail.com/\n> >>\n> >> yes, indeed. I was planning on adding bare repository support in a separate\n> >> patch series, since the additions in [2] allows .gitattributes to be read from a\n> >> bare repository.\n> >>\n> >> 1. https://git-scm.com/docs/gitattributes\n> >> 2. https://lore.kernel.org/git/0ca8b2458921fc40269b0c43b5ec86eba77d6b54.1673684790.git.karthik.188@gmail.com/\n> >>\n> >> thanks!\n> >> John\n> >\n> > Oh, interesting, I didn't know about [2]. So, is the plan to take the\n> > --source option from that series and add it to diff (perhaps with a\n> > different name, since log tends to consume diff options and --source\n> > is already taken)?\n>\n> Yep, that would be the general idea\n>\n> >\n> > And do you expect to get the tree-ish from the two the users are\n> > already specifying to diff? If so, which one do you use (the two\n> > commits being diffed might have differing .gitattributes files)? If\n> > not, what does that mean for users of e.g. the GitLab UI who have to\n> > specify a third tree when diffing?\n>\n> Good question! Since it seems that when `git-diff(1)` considers diff.,\n> it goes with the path of the first one. (might need some confirmation here)\n>\n> in diff.c:\n>\n>\n> static void run_diff(struct diff_filepair *p, struct diff_options *o)\n> {\n> const char *pgm = external_diff();\n> struct strbuf msg;\n> struct diff_filespec *one = p->one;\n> struct diff_filespec *two = p->two;\n> const char *name;\n> const char *other;\n> const char *attr_path;\n>\n> name = one->path;\n> other = (strcmp(name, two->path) ? two->path : NULL);\n> attr_path = name;\n> if (o->prefix_length)\n>\n> I was thinking we would just use the tree-ish of the first one\n\nThat would certainly simplify, but it'd be pretty important to\ndocument. (Incidentally, this kind of decision was my reason for\nasking about all those special cases earlier, i.e. how to handle diff\nbetween different commits, how to handle renames, how to handle bare\nrepositories, etc.)\n\nThis kind of decision probably also means you'd need a variety of\ntestcases where .gitattributes is different in every commit & the\nindex & the working tree, and then you start testing several of the\npossible pairings to make sure the right .gitattributes file is used\n(e.g. (commit, commit), (commit, index), (index, commit), (worktree,\nindex), etc.)\n\nHowever, I'm curious again. You brought this up because you want to\nuse it in GitLab, yet configuration of using this option as it appears\nin this series requires changing _both_ .gitattributes and\n.git/config. How will users of the GitLab UI do the configuration\nnecessary for the git-config side to take effect?\n"},{"id":"472351","messageId":"647D3D49-B85B-4B66-A857-695CFF9685EE@gmail.com","threadId":"59197","inReplyTo":"CABPp-BHQn0sjAMwJ+r6uenO=nGLG1HvfnhS6tG8mu1BWt4bdOw@mail.gmail.com","subject":"Re: [PATCH v3 2/2] diff: teach diff to read algorithm from diff driver","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-20T20:53:09Z","receivedAt":"2023-02-20T20:53:16Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Elijah,\n\nOn 20 Feb 2023, at 12:32, Elijah Newren wrote:\n\n> On Mon, Feb 20, 2023 at 8:49 AM John Cai wrote:\n>>\n>> Hi Elijah,\n>>\n>> On 20 Feb 2023, at 11:21, Elijah Newren wrote:\n>>\n>>> On Mon, Feb 20, 2023 at 7:32 AM John Cai wrote:\n>>> [...]\n>>>>> I'm still curious if this should this also include warnings/caveats, such as:\n>>>>> * The diff attribute specified in .gitattributes will be ignored in\n>>>>> a bare clone\n>>>>> * The diff attribute specified in .gitattributes will be ignored if\n>>>>> it is only specified in another branch (e.g. on a branch \"special-file\n>>>>> diff=patience\" recorded in .gitattributes, then checkout master but\n>>>>> run `git log -1 -p $branch`)\n>>>>> * When a file is renamed, the diff attribute for the pre-image name\n>>>>> is the only one the system pays attention to (thus adding \"-R\" can\n>>>>> flip which diff algorithm is run for the renamed file).\n>>>>\n>>>> I would be fine with adding that--though originally I was thinking that these\n>>>> can be inferred from the way that gitattributes are documented in [1]. Calling\n>>>> these out would make it more clear though, so I could go either way.\n>>>>\n>>>>>\n>>>>> Also, since I tested the three items above to verify they are valid\n>>>>> warnings, I'm a bit confused. I thought your intent was to use this\n>>>>> server-side[1], so isn't the bare clone aspect a deal-breaker for your\n>>>>> intended usecase?\n>>>>>\n>>>>> [1] https://lore.kernel.org/git/7852AC7B-7A4E-4DD0-ADEA-CFFD5D16C595@gmail.com/\n>>>>\n>>>> yes, indeed. I was planning on adding bare repository support in a separate\n>>>> patch series, since the additions in [2] allows .gitattributes to be read from a\n>>>> bare repository.\n>>>>\n>>>> 1. https://git-scm.com/docs/gitattributes\n>>>> 2. https://lore.kernel.org/git/0ca8b2458921fc40269b0c43b5ec86eba77d6b54.1673684790.git.karthik.188@gmail.com/\n>>>>\n>>>> thanks!\n>>>> John\n>>>\n>>> Oh, interesting, I didn't know about [2]. So, is the plan to take the\n>>> --source option from that series and add it to diff (perhaps with a\n>>> different name, since log tends to consume diff options and --source\n>>> is already taken)?\n>>\n>> Yep, that would be the general idea\n>>\n>>>\n>>> And do you expect to get the tree-ish from the two the users are\n>>> already specifying to diff? If so, which one do you use (the two\n>>> commits being diffed might have differing .gitattributes files)? If\n>>> not, what does that mean for users of e.g. the GitLab UI who have to\n>>> specify a third tree when diffing?\n>>\n>> Good question! Since it seems that when `git-diff(1)` considers diff.,\n>> it goes with the path of the first one. (might need some confirmation here)\n>>\n>> in diff.c:\n>>\n>>\n>> static void run_diff(struct diff_filepair *p, struct diff_options *o)\n>> {\n>> const char *pgm = external_diff();\n>> struct strbuf msg;\n>> struct diff_filespec *one = p->one;\n>> struct diff_filespec *two = p->two;\n>> const char *name;\n>> const char *other;\n>> const char *attr_path;\n>>\n>> name = one->path;\n>> other = (strcmp(name, two->path) ? two->path : NULL);\n>> attr_path = name;\n>> if (o->prefix_length)\n>>\n>> I was thinking we would just use the tree-ish of the first one\n>\n> That would certainly simplify, but it'd be pretty important to\n> document. (Incidentally, this kind of decision was my reason for\n> asking about all those special cases earlier, i.e. how to handle diff\n> between different commits, how to handle renames, how to handle bare\n> repositories, etc.)\n\nGood point--that would be good to document.\n\n>\n> This kind of decision probably also means you'd need a variety of\n> testcases where .gitattributes is different in every commit & the\n> index & the working tree, and then you start testing several of the\n> possible pairings to make sure the right .gitattributes file is used\n> (e.g. (commit, commit), (commit, index), (index, commit), (worktree,\n> index), etc.)\n>\n> However, I'm curious again. You brought this up because you want to\n> use it in GitLab, yet configuration of using this option as it appears\n> in this series requires changing _both_ .gitattributes and\n> .git/config. How will users of the GitLab UI do the configuration\n> necessary for the git-config side to take effect?\n\nGood question--we would likely need to add some pre-baked configuration server\nside with a driver config that users could then tap into with their\ngitattributes files.\n"},{"id":"472352","messageId":"pull.1452.v4.git.git.1676927082.gitgitgadget@gmail.com","threadId":"59197","inReplyTo":"pull.1452.v3.git.git.1676665285.gitgitgadget@gmail.com","subject":"[PATCH v4 0/2] Teach diff to honor diff algorithms set through git attributes","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-20T21:04:40Z","receivedAt":"2023-02-20T21:04:49Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"When a repository contains different kinds of files, it may be desirable to\nuse different algorithms based on file type. This is currently not feasible\nthrough the command line or using git configs. However, we can leverage the\nfact that gitattributes are path aware.\n\nTeach the diff machinery to check gitattributes when diffing files by using\nthe existing diff. scheme, and add an \"algorithm\" type to the external\ndriver config.\n\nChange since V3:\n\n * cleaned up documentation, typos\n * minor cleanup such as if statement ordering, and overly long lines\n\nChanges since V2:\n\n * minor clean up and variable renaming\n * avoid parsing attribute files for the driver if the diff algorithm is set\n through the command line\n\nChanges since V1:\n\n * utilize the existing diff..* scheme where the driver is defined\n in gitattributes, but the algorithm is defined in the gitconfig.\n\nTo address some of the performance concerns in the previous series, a\nbenchmark shows that now only a minor performance penalty is incurred, now\nthat we are no longer adding an additional attributes parsing call:\n\n$ echo \"*.[ch] diff=other\" >> .gitattributes $ hyperfine -r 10 -L a\ngit-bin-wrapper,git '{a} -c diff.other.algorithm=myers diff v2.0.0 v2.28.0'\nBenchmark 1: git-bin-wrapper -c diff.other.algorithm=myers diff v2.0.0\nv2.28.0 Time (mean ± σ): 716.3 ms ± 3.8 ms [User: 660.2 ms, System: 50.8 ms]\nRange (min … max): 709.8 ms … 720.6 ms 10 runs\n\nBenchmark 2: git -c diff.other.algorithm=myers diff v2.0.0 v2.28.0 Time\n(mean ± σ): 704.3 ms ± 2.9 ms [User: 656.6 ms, System: 44.3 ms] Range (min …\nmax): 700.1 ms … 708.6 ms 10 runs\n\nSummary 'git -c diff.other.algorithm=myers diff v2.0.0 v2.28.0' ran 1.02 ±\n0.01 times faster than 'git-bin-wrapper -c diff.other.algorithm=myers diff\nv2.0.0 v2.28.0'\n\nJohn Cai (2):\n diff: consolidate diff algorithm option parsing\n diff: teach diff to read algorithm from diff driver\n\n Documentation/gitattributes.txt | 31 ++++++++++++\n diff.c | 90 ++++++++++++++++++++++++---------\n diff.h | 1 +\n t/lib-diff-alternative.sh | 38 +++++++++++++-\n userdiff.c | 4 +-\n userdiff.h | 1 +\n 6 files changed, 140 insertions(+), 25 deletions(-)\n\n\nbase-commit: c867e4fa180bec4750e9b54eb10f459030dbebfd\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1452%2Fjohn-cai%2Fjc%2Fattr-diff-algo-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1452/john-cai/jc/attr-diff-algo-v4\nPull-Request: https://github.com/git/git/pull/1452\n\nRange-diff vs v3:\n\n 1: 816c47aa414 = 1: 816c47aa414 diff: consolidate diff algorithm option parsing\n 2: b330222ce83 ! 2: 77e66ab98fc diff: teach diff to read algorithm from diff driver\n @@ Commit message\n finally the diff.algorithm config.\n \n To enforce precedence order, use a new `ignore_driver_algorithm` member\n - during options pasing to indicate the diff algorithm was set via command\n + during options parsing to indicate the diff algorithm was set via command\n line args.\n \n Signed-off-by: John Cai \n @@ Documentation/gitattributes.txt: with the above configuration, i.e. `j-c-diff`,\n +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n +\n +The diff algorithm can be set through the `diff.algorithm` config key, but\n -+sometimes it may be helpful to set the diff algorithm by path. For example, one\n -+might wish to set a diff algorithm automatically for all `.json` files such that\n -+the user would not need to pass in a separate command line `--diff-algorithm`\n -+flag each time.\n ++sometimes it may be helpful to set the diff algorithm per path. For example,\n ++one may want to use the `minimal` diff algorithm for .json files, and the\n ++`histogram` for .c files, and so on without having to pass in the algorithm\n ++through the command line each time.\n +\n +First, in `.gitattributes`, assign the `diff` attribute for paths.\n +\n @@ Documentation/gitattributes.txt: with the above configuration, i.e. `j-c-diff`,\n +------------------------\n +\n +Then, define a \"diff..algorithm\" configuration to specify the diff\n -+algorithm, choosing from `meyers`, `patience`, `minimal`, or `histogram`.\n ++algorithm, choosing from `myers`, `patience`, `minimal`, or `histogram`.\n +\n +----------------------------------------------------------------\n +[diff \"\"]\n @@ Documentation/gitattributes.txt: with the above configuration, i.e. `j-c-diff`,\n +git-show(1) and is used for the `--stat` output as well. The merge machinery\n +will not use the diff algorithm set through this method.\n +\n -+NOTE: If the `command` key also exists, then Git will treat this as an external\n -+diff and attempt to use the value set for `command` as an external program. For\n -+instance, the following config, combined with the above `.gitattributes` file,\n -+will result in `command` favored over `algorithm`.\n -+\n -+----------------------------------------------------------------\n -+[diff \"\"]\n -+ command = j-c-diff\n -+ algorithm = histogram\n -+----------------------------------------------------------------\n ++NOTE: If `diff..command` is defined for path with the\n ++`diff=` attribute, it is executed as an external diff driver\n ++(see above), and adding `diff..algorithm` has no effect, as the\n ++algorithm is not passed to the external diff driver.\n \n Defining a custom hunk-header\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n @@ diff.c: static void run_diff_cmd(const char *pgm,\n \t}\n -\tif (one && two)\n +\tif (one && two) {\n -+\t\tif (drv && !o->ignore_driver_algorithm && drv->algorithm)\n ++\t\tif (!o->ignore_driver_algorithm && drv && drv->algorithm)\n +\t\t\tset_diff_algorithm(o, drv->algorithm);\n +\n \t\tbuiltin_diff(name, other ? other : name,\n @@ diff.c: static void run_diffstat(struct diff_filepair *p, struct diff_options *o\n \tconst char *other;\n \n +\tif (!o->ignore_driver_algorithm) {\n -+\t\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index, p->one->path);\n ++\t\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index,\n ++\t\t\t\t\t\t\t\t p->one->path);\n +\n -+\t\tif (drv && drv->algorithm) {\n ++\t\tif (drv && drv->algorithm)\n +\t\t\tset_diff_algorithm(o, drv->algorithm);\n -+\t\t}\n +\t}\n +\n \tif (DIFF_PAIR_UNMERGED(p)) {\n @@ t/lib-diff-alternative.sh: index $file1..$file2 100644\n +\n +\ttest_expect_success \"$STRATEGY diff command line precedence before attributes\" '\n +\t\techo \"file* diff=driver\" >.gitattributes &&\n -+\t\tgit config diff.driver.algorithm meyers &&\n ++\t\tgit config diff.driver.algorithm myers &&\n +\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n +\t\ttest_cmp expect output\n +\t'\n\n-- \ngitgitgadget\n"},{"id":"472353","messageId":"816c47aa414586e99aa762604396bd8be4fb11f4.1676927082.git.gitgitgadget@gmail.com","threadId":"59197","inReplyTo":"pull.1452.v4.git.git.1676927082.gitgitgadget@gmail.com","subject":"[PATCH v4 1/2] diff: consolidate diff algorithm option parsing","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-20T21:04:41Z","receivedAt":"2023-02-20T21:04:52Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nA subsequent commit will need the ability to tell if the diff algorithm\nwas set through the command line through setting a new member of\ndiff_options. While this logic can be added to the\ndiff_opt_diff_algorithm() callback, the `--minimal` and `--histogram`\noptions are handled via OPT_BIT without a callback.\n\nRemedy this by consolidating the options parsing logic for --minimal and\n--histogram into one callback. This way we can modify `diff_options` in\nthat function.\n\nAs an additional refactor, the logic that sets the diff algorithm in\ndiff_opt_diff_algorithm() can be refactored into a helper that will\nallow multiple callsites to set the diff algorithm.\n\nSigned-off-by: John Cai \n---\n diff.c | 57 +++++++++++++++++++++++++++++++++++++++++++--------------\n 1 file changed, 43 insertions(+), 14 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 329eebf16a0..5efc22ca06b 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -3437,6 +3437,22 @@ static int diff_filepair_is_phoney(struct diff_filespec *one,\n \treturn !DIFF_FILE_VALID(one) && !DIFF_FILE_VALID(two);\n }\n \n+static int set_diff_algorithm(struct diff_options *opts,\n+\t\t\t const char *alg)\n+{\n+\tlong value = parse_algorithm_value(alg);\n+\n+\tif (value < 0)\n+\t\treturn -1;\n+\n+\t/* clear out previous settings */\n+\tDIFF_XDL_CLR(opts, NEED_MINIMAL);\n+\topts->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n+\topts->xdl_opts |= value;\n+\n+\treturn 0;\n+}\n+\n static void builtin_diff(const char *name_a,\n \t\t\t const char *name_b,\n \t\t\t struct diff_filespec *one,\n@@ -5107,17 +5123,28 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct diff_options *options = opt->value;\n-\tlong value = parse_algorithm_value(arg);\n \n \tBUG_ON_OPT_NEG(unset);\n-\tif (value < 0)\n+\n+\tif (set_diff_algorithm(options, arg))\n \t\treturn error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n \n-\t/* clear out previous settings */\n-\tDIFF_XDL_CLR(options, NEED_MINIMAL);\n-\toptions->xdl_opts &= ~XDF_DIFF_ALGORITHM_MASK;\n-\toptions->xdl_opts |= value;\n+\treturn 0;\n+}\n+\n+static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n+\t\t\t\t const char *arg, int unset)\n+{\n+\tstruct diff_options *options = opt->value;\n+\n+\tBUG_ON_OPT_NEG(unset);\n+\tBUG_ON_OPT_ARG(arg);\n+\n+\tif (set_diff_algorithm(options, opt->long_name))\n+\t\tBUG(\"available diff algorithms include \\\"myers\\\", \"\n+\t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n+\n \treturn 0;\n }\n \n@@ -5250,7 +5277,6 @@ static int diff_opt_patience(const struct option *opt,\n \n \tBUG_ON_OPT_NEG(unset);\n \tBUG_ON_OPT_ARG(arg);\n-\toptions->xdl_opts = DIFF_WITH_ALG(options, PATIENCE_DIFF);\n \t/*\n \t * Both --patience and --anchored use PATIENCE_DIFF\n \t * internally, so remove any anchors previously\n@@ -5259,7 +5285,8 @@ static int diff_opt_patience(const struct option *opt,\n \tfor (i = 0; i < options->anchors_nr; i++)\n \t\tfree(options->anchors[i]);\n \toptions->anchors_nr = 0;\n-\treturn 0;\n+\n+\treturn set_diff_algorithm(options, \"patience\");\n }\n \n static int diff_opt_ignore_regex(const struct option *opt,\n@@ -5562,9 +5589,10 @@ struct option *add_diff_options(const struct option *opts,\n \t\t\t N_(\"prevent rename/copy detection if the number of rename/copy targets exceeds given limit\")),\n \n \t\tOPT_GROUP(N_(\"Diff algorithm options\")),\n-\t\tOPT_BIT(0, \"minimal\", &options->xdl_opts,\n-\t\t\tN_(\"produce the smallest possible diff\"),\n-\t\t\tXDF_NEED_MINIMAL),\n+\t\tOPT_CALLBACK_F(0, \"minimal\", options, NULL,\n+\t\t\t N_(\"produce the smallest possible diff\"),\n+\t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n+\t\t\t diff_opt_diff_algorithm_no_arg),\n \t\tOPT_BIT_F('w', \"ignore-all-space\", &options->xdl_opts,\n \t\t\t N_(\"ignore whitespace when comparing lines\"),\n \t\t\t XDF_IGNORE_WHITESPACE, PARSE_OPT_NONEG),\n@@ -5590,9 +5618,10 @@ struct option *add_diff_options(const struct option *opts,\n \t\t\t N_(\"generate diff using the \\\"patience diff\\\" algorithm\"),\n \t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n \t\t\t diff_opt_patience),\n-\t\tOPT_BITOP(0, \"histogram\", &options->xdl_opts,\n-\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n-\t\t\t XDF_HISTOGRAM_DIFF, XDF_DIFF_ALGORITHM_MASK),\n+\t\tOPT_CALLBACK_F(0, \"histogram\", options, NULL,\n+\t\t\t N_(\"generate diff using the \\\"histogram diff\\\" algorithm\"),\n+\t\t\t PARSE_OPT_NONEG | PARSE_OPT_NOARG,\n+\t\t\t diff_opt_diff_algorithm_no_arg),\n \t\tOPT_CALLBACK_F(0, \"diff-algorithm\", options, N_(\"\"),\n \t\t\t N_(\"choose a diff algorithm\"),\n \t\t\t PARSE_OPT_NONEG, diff_opt_diff_algorithm),\n-- \ngitgitgadget\n\n"},{"id":"472354","messageId":"77e66ab98fc53d8089a53f537427b5cf74b9ed2c.1676927082.git.gitgitgadget@gmail.com","threadId":"59197","inReplyTo":"pull.1452.v4.git.git.1676927082.gitgitgadget@gmail.com","subject":"[PATCH v4 2/2] diff: teach diff to read algorithm from diff driver","fromName":"John Cai via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-20T21:04:42Z","receivedAt":"2023-02-20T21:04:54Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"From: John Cai \n\nIt can be useful to specify diff algorithms per file type. For example,\none may want to use the minimal diff algorithm for .json files, another\nfor .c files, etc.\n\nThe diff machinery already checks attributes for a diff driver. Teach\nthe diff driver parser a new type \"algorithm\" to look for in the\nconfig, which will be used if a driver has been specified through the\nattributes.\n\nEnforce precedence of the diff algorithm by favoring the command line\noption, then looking at the driver attributes & config combination, then\nfinally the diff.algorithm config.\n\nTo enforce precedence order, use a new `ignore_driver_algorithm` member\nduring options parsing to indicate the diff algorithm was set via command\nline args.\n\nSigned-off-by: John Cai \n---\n Documentation/gitattributes.txt | 31 +++++++++++++++++++++++++++\n diff.c | 33 ++++++++++++++++++++--------\n diff.h | 1 +\n t/lib-diff-alternative.sh | 38 ++++++++++++++++++++++++++++++++-\n userdiff.c | 4 +++-\n userdiff.h | 1 +\n 6 files changed, 97 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex c19e64ea0ef..39bfbca1ffe 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -758,6 +758,37 @@ with the above configuration, i.e. `j-c-diff`, with 7\n parameters, just like `GIT_EXTERNAL_DIFF` program is called.\n See linkgit:git[1] for details.\n \n+Setting the internal diff algorithm\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+The diff algorithm can be set through the `diff.algorithm` config key, but\n+sometimes it may be helpful to set the diff algorithm per path. For example,\n+one may want to use the `minimal` diff algorithm for .json files, and the\n+`histogram` for .c files, and so on without having to pass in the algorithm\n+through the command line each time.\n+\n+First, in `.gitattributes`, assign the `diff` attribute for paths.\n+\n+------------------------\n+*.json diff=\n+------------------------\n+\n+Then, define a \"diff..algorithm\" configuration to specify the diff\n+algorithm, choosing from `myers`, `patience`, `minimal`, or `histogram`.\n+\n+----------------------------------------------------------------\n+[diff \"\"]\n+ algorithm = histogram\n+----------------------------------------------------------------\n+\n+This diff algorithm applies to user facing diff output like git-diff(1),\n+git-show(1) and is used for the `--stat` output as well. The merge machinery\n+will not use the diff algorithm set through this method.\n+\n+NOTE: If `diff..command` is defined for path with the\n+`diff=` attribute, it is executed as an external diff driver\n+(see above), and adding `diff..algorithm` has no effect, as the\n+algorithm is not passed to the external diff driver.\n \n Defining a custom hunk-header\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\ndiff --git a/diff.c b/diff.c\nindex 5efc22ca06b..469e18aed20 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4456,15 +4456,13 @@ static void run_diff_cmd(const char *pgm,\n \tconst char *xfrm_msg = NULL;\n \tint complete_rewrite = (p->status == DIFF_STATUS_MODIFIED) && p->score;\n \tint must_show_header = 0;\n+\tstruct userdiff_driver *drv = NULL;\n \n-\n-\tif (o->flags.allow_external) {\n-\t\tstruct userdiff_driver *drv;\n-\n+\tif (o->flags.allow_external || !o->ignore_driver_algorithm)\n \t\tdrv = userdiff_find_by_path(o->repo->index, attr_path);\n-\t\tif (drv && drv->external)\n-\t\t\tpgm = drv->external;\n-\t}\n+\n+\tif (o->flags.allow_external && drv && drv->external)\n+\t\tpgm = drv->external;\n \n \tif (msg) {\n \t\t/*\n@@ -4481,12 +4479,16 @@ static void run_diff_cmd(const char *pgm,\n \t\trun_external_diff(pgm, name, other, one, two, xfrm_msg, o);\n \t\treturn;\n \t}\n-\tif (one && two)\n+\tif (one && two) {\n+\t\tif (!o->ignore_driver_algorithm && drv && drv->algorithm)\n+\t\t\tset_diff_algorithm(o, drv->algorithm);\n+\n \t\tbuiltin_diff(name, other ? other : name,\n \t\t\t one, two, xfrm_msg, must_show_header,\n \t\t\t o, complete_rewrite);\n-\telse\n+\t} else {\n \t\tfprintf(o->file, \"* Unmerged path %s\\n\", name);\n+\t}\n }\n \n static void diff_fill_oid_info(struct diff_filespec *one, struct index_state *istate)\n@@ -4583,6 +4585,14 @@ static void run_diffstat(struct diff_filepair *p, struct diff_options *o,\n \tconst char *name;\n \tconst char *other;\n \n+\tif (!o->ignore_driver_algorithm) {\n+\t\tstruct userdiff_driver *drv = userdiff_find_by_path(o->repo->index,\n+\t\t\t\t\t\t\t\t p->one->path);\n+\n+\t\tif (drv && drv->algorithm)\n+\t\t\tset_diff_algorithm(o, drv->algorithm);\n+\t}\n+\n \tif (DIFF_PAIR_UNMERGED(p)) {\n \t\t/* unmerged */\n \t\tbuiltin_diffstat(p->one->path, NULL, NULL, NULL,\n@@ -5130,6 +5140,8 @@ static int diff_opt_diff_algorithm(const struct option *opt,\n \t\treturn error(_(\"option diff-algorithm accepts \\\"myers\\\", \"\n \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\"));\n \n+\toptions->ignore_driver_algorithm = 1;\n+\n \treturn 0;\n }\n \n@@ -5145,6 +5157,8 @@ static int diff_opt_diff_algorithm_no_arg(const struct option *opt,\n \t\tBUG(\"available diff algorithms include \\\"myers\\\", \"\n \t\t\t \"\\\"minimal\\\", \\\"patience\\\" and \\\"histogram\\\"\");\n \n+\toptions->ignore_driver_algorithm = 1;\n+\n \treturn 0;\n }\n \n@@ -5285,6 +5299,7 @@ static int diff_opt_patience(const struct option *opt,\n \tfor (i = 0; i < options->anchors_nr; i++)\n \t\tfree(options->anchors[i]);\n \toptions->anchors_nr = 0;\n+\toptions->ignore_driver_algorithm = 1;\n \n \treturn set_diff_algorithm(options, \"patience\");\n }\ndiff --git a/diff.h b/diff.h\nindex 41eb2c3d428..8d770b1d579 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -333,6 +333,7 @@ struct diff_options {\n \tint prefix_length;\n \tconst char *stat_sep;\n \tint xdl_opts;\n+\tint ignore_driver_algorithm;\n \n \t/* see Documentation/diff-options.txt */\n \tchar **anchors;\ndiff --git a/t/lib-diff-alternative.sh b/t/lib-diff-alternative.sh\nindex 8d1e408bb58..a8f5d3274a5 100644\n--- a/t/lib-diff-alternative.sh\n+++ b/t/lib-diff-alternative.sh\n@@ -105,10 +105,46 @@ index $file1..$file2 100644\n }\n EOF\n \n+\tcat >expect_diffstat < file2 | 21 ++++++++++-----------\n+ 1 file changed, 10 insertions(+), 11 deletions(-)\n+EOF\n+\n \tSTRATEGY=$1\n \n+\ttest_expect_success \"$STRATEGY diff from attributes\" '\n+\t\techo \"file* diff=driver\" >.gitattributes &&\n+\t\tgit config diff.driver.algorithm \"$STRATEGY\" &&\n+\t\ttest_must_fail git diff --no-index file1 file2 > output &&\n+\t\tcat expect &&\n+\t\tcat output &&\n+\t\ttest_cmp expect output\n+\t'\n+\n+\ttest_expect_success \"$STRATEGY diff from attributes has valid diffstat\" '\n+\t\techo \"file* diff=driver\" >.gitattributes &&\n+\t\tgit config diff.driver.algorithm \"$STRATEGY\" &&\n+\t\ttest_must_fail git diff --stat --no-index file1 file2 > output &&\n+\t\ttest_cmp expect_diffstat output\n+\t'\n+\n \ttest_expect_success \"$STRATEGY diff\" '\n-\t\ttest_must_fail git diff --no-index \"--$STRATEGY\" file1 file2 > output &&\n+\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n+\t\ttest_cmp expect output\n+\t'\n+\n+\ttest_expect_success \"$STRATEGY diff command line precedence before attributes\" '\n+\t\techo \"file* diff=driver\" >.gitattributes &&\n+\t\tgit config diff.driver.algorithm myers &&\n+\t\ttest_must_fail git diff --no-index \"--diff-algorithm=$STRATEGY\" file1 file2 > output &&\n+\t\ttest_cmp expect output\n+\t'\n+\n+\ttest_expect_success \"$STRATEGY diff attributes precedence before config\" '\n+\t\tgit config diff.algorithm default &&\n+\t\techo \"file* diff=driver\" >.gitattributes &&\n+\t\tgit config diff.driver.algorithm \"$STRATEGY\" &&\n+\t\ttest_must_fail git diff --no-index file1 file2 > output &&\n \t\ttest_cmp expect output\n \t'\n \ndiff --git a/userdiff.c b/userdiff.c\nindex d71b82feb74..ff25cfc4b4c 100644\n--- a/userdiff.c\n+++ b/userdiff.c\n@@ -293,7 +293,7 @@ PATTERNS(\"scheme\",\n \t \"|([^][)(}{[ \\t])+\"),\n PATTERNS(\"tex\", \"^(\\\\\\\\((sub)*section|chapter|part)\\\\*{0,1}\\\\{.*)$\",\n \t \"\\\\\\\\[a-zA-Z@]+|\\\\\\\\.|[a-zA-Z0-9\\x80-\\xff]+\"),\n-{ \"default\", NULL, -1, { NULL, 0 } },\n+{ \"default\", NULL, NULL, -1, { NULL, 0 } },\n };\n #undef PATTERNS\n #undef IPATTERN\n@@ -394,6 +394,8 @@ int userdiff_config(const char *k, const char *v)\n \t\treturn parse_bool(&drv->textconv_want_cache, k, v);\n \tif (!strcmp(type, \"wordregex\"))\n \t\treturn git_config_string(&drv->word_regex, k, v);\n+\tif (!strcmp(type, \"algorithm\"))\n+\t\treturn git_config_string(&drv->algorithm, k, v);\n \n \treturn 0;\n }\ndiff --git a/userdiff.h b/userdiff.h\nindex aee91bc77e6..24419db6973 100644\n--- a/userdiff.h\n+++ b/userdiff.h\n@@ -14,6 +14,7 @@ struct userdiff_funcname {\n struct userdiff_driver {\n \tconst char *name;\n \tconst char *external;\n+\tconst char *algorithm;\n \tint binary;\n \tstruct userdiff_funcname funcname;\n \tconst char *word_regex;\n-- \ngitgitgadget\n"},{"id":"472386","messageId":"xmqq5ybuhqhc.fsf@gitster.g","threadId":"59197","inReplyTo":"pull.1452.v4.git.git.1676927082.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 0/2] Teach diff to honor diff algorithms set through git attributes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-21T17:34:39Z","receivedAt":"2023-02-21T17:34:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"John Cai via GitGitGadget\" writes:\n\n> When a repository contains different kinds of files, it may be desirable to\n> use different algorithms based on file type. This is currently not feasible\n> through the command line or using git configs. However, we can leverage the\n> fact that gitattributes are path aware.\n> ...\n> To address some of the performance concerns in the previous series, a\n> benchmark shows that now only a minor performance penalty is incurred, now\n> that we are no longer adding an additional attributes parsing call:\n>\n> $ echo \"*.[ch] diff=other\" >> .gitattributes $ hyperfine -r 10 -L a\n> git-bin-wrapper,git '{a} -c diff.other.algorithm=myers diff v2.0.0 v2.28.0'\n> Benchmark 1: git-bin-wrapper -c diff.other.algorithm=myers diff v2.0.0\n> v2.28.0 Time (mean ± σ): 716.3 ms ± 3.8 ms [User: 660.2 ms, System: 50.8 ms]\n> Range (min … max): 709.8 ms … 720.6 ms 10 runs\n>\n> Benchmark 2: git -c diff.other.algorithm=myers diff v2.0.0 v2.28.0 Time\n> (mean ± σ): 704.3 ms ± 2.9 ms [User: 656.6 ms, System: 44.3 ms] Range (min …\n> max): 700.1 ms … 708.6 ms 10 runs\n>\n> Summary 'git -c diff.other.algorithm=myers diff v2.0.0 v2.28.0' ran 1.02 ±\n> 0.01 times faster than 'git-bin-wrapper -c diff.other.algorithm=myers diff\n> v2.0.0 v2.28.0'\n\nHopefully this round can immediately be merged down to 'next'?\nThanks.\n\n"},{"id":"472388","messageId":"CABPp-BEAB6jKXeo=DHiimQUDj=5i7U5MQPXTEsAagLVAuk7FOQ@mail.gmail.com","threadId":"59197","inReplyTo":"xmqq5ybuhqhc.fsf@gitster.g","subject":"Re: [PATCH v4 0/2] Teach diff to honor diff algorithms set through git attributes","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-02-21T18:05:50Z","receivedAt":"2023-02-21T18:06:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Feb 21, 2023 at 9:34 AM Junio C Hamano wrote:\n>\n> \"John Cai via GitGitGadget\" writes:\n>\n> > When a repository contains different kinds of files, it may be desirable to\n> > use different algorithms based on file type. This is currently not feasible\n> > through the command line or using git configs. However, we can leverage the\n> > fact that gitattributes are path aware.\n> > ...\n> > To address some of the performance concerns in the previous series, a\n> > benchmark shows that now only a minor performance penalty is incurred, now\n> > that we are no longer adding an additional attributes parsing call:\n> >\n> > $ echo \"*.[ch] diff=other\" >> .gitattributes $ hyperfine -r 10 -L a\n> > git-bin-wrapper,git '{a} -c diff.other.algorithm=myers diff v2.0.0 v2.28.0'\n> > Benchmark 1: git-bin-wrapper -c diff.other.algorithm=myers diff v2.0.0\n> > v2.28.0 Time (mean ± σ): 716.3 ms ± 3.8 ms [User: 660.2 ms, System: 50.8 ms]\n> > Range (min … max): 709.8 ms … 720.6 ms 10 runs\n> >\n> > Benchmark 2: git -c diff.other.algorithm=myers diff v2.0.0 v2.28.0 Time\n> > (mean ± σ): 704.3 ms ± 2.9 ms [User: 656.6 ms, System: 44.3 ms] Range (min …\n> > max): 700.1 ms … 708.6 ms 10 runs\n> >\n> > Summary 'git -c diff.other.algorithm=myers diff v2.0.0 v2.28.0' ran 1.02 ±\n> > 0.01 times faster than 'git-bin-wrapper -c diff.other.algorithm=myers diff\n> > v2.0.0 v2.28.0'\n>\n> Hopefully this round can immediately be merged down to 'next'?\n> Thanks.\n\nI'll leave that up to you and John, but are we risking merging code\nthat could go unused or that we need to fundamentally change? I don't\nsee how to handle the issues over at\nhttps://lore.kernel.org/git/647D3D49-B85B-4B66-A857-695CFF9685EE@gmail.com/\n(not that I've spent much thought on it besides asking how things are\ngoing to work), and would be worried that we'd end up needing to\nchange the mechanism somehow to cover the things John ultimately\nwants.\n"},{"id":"472391","messageId":"xmqq1qmig8c4.fsf@gitster.g","threadId":"59197","inReplyTo":"CABPp-BEAB6jKXeo=DHiimQUDj=5i7U5MQPXTEsAagLVAuk7FOQ@mail.gmail.com","subject":"Re: [PATCH v4 0/2] Teach diff to honor diff algorithms set through git attributes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-21T18:51:55Z","receivedAt":"2023-02-21T18:51:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> I'll leave that up to you and John, but are we risking merging code\n> that could go unused or that we need to fundamentally change? I don't\n> see how to handle the issues over at\n> https://lore.kernel.org/git/647D3D49-B85B-4B66-A857-695CFF9685EE@gmail.com/\n\nIf this is useful enough for desktop users already, then that is a\ngood enough reason to take it, I would say.\n\nGitLab can easily add WebUI that says \"You can define what diff\nalgorithm is used for files with which suffix\" to allow you to\nconfigure a table like this:\n\n\tjson | histogram\n\tc | patience\n\t* | myers\n\nand populate the server-side equivalent of .git/info/attributes and\n.git/config based on that, and without anything further than the\nposted patches, the result should just work, no?\n"},{"id":"472393","messageId":"3F1016EE-07BD-4F7C-9C75-6995A1EB211F@gmail.com","threadId":"59197","inReplyTo":"xmqq1qmig8c4.fsf@gitster.g","subject":"Re: [PATCH v4 0/2] Teach diff to honor diff algorithms set through git attributes","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-21T19:36:12Z","receivedAt":"2023-02-21T19:36:43Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"\n\nOn 21 Feb 2023, at 13:51, Junio C Hamano wrote:\n\n> Elijah Newren writes:\n>\n>> I'll leave that up to you and John, but are we risking merging code\n>> that could go unused or that we need to fundamentally change? I don't\n>> see how to handle the issues over at\n>> https://lore.kernel.org/git/647D3D49-B85B-4B66-A857-695CFF9685EE@gmail.com/\n>\n> If this is useful enough for desktop users already, then that is a\n> good enough reason to take it, I would say.\n>\n> GitLab can easily add WebUI that says \"You can define what diff\n> algorithm is used for files with which suffix\" to allow you to\n> configure a table like this:\n>\n> \tjson | histogram\n> \tc | patience\n> \t* | myers\n>\n> and populate the server-side equivalent of .git/info/attributes and\n> .git/config based on that, and without anything further than the\n> posted patches, the result should just work, no?\n\nYes, that is my thinking too. The goal was to make this user friendly for\neveryone on the Git command line, and then GitLab can hook into this accordingly\nto make it ussable with the GitLab UI.\n\nthanks\nJohn\n"},{"id":"472399","messageId":"CABPp-BHp3wxJYkWV9i7Qwp4idseRehgf_hXRHdAkuoozi5Zh0A@mail.gmail.com","threadId":"59197","inReplyTo":"3F1016EE-07BD-4F7C-9C75-6995A1EB211F@gmail.com","subject":"Re: [PATCH v4 0/2] Teach diff to honor diff algorithms set through git attributes","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-02-21T20:16:30Z","receivedAt":"2023-02-21T20:16:47Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Feb 21, 2023 at 11:36 AM John Cai wrote:\n>\n> On 21 Feb 2023, at 13:51, Junio C Hamano wrote:\n>\n> > Elijah Newren writes:\n> >\n> >> I'll leave that up to you and John, but are we risking merging code\n> >> that could go unused or that we need to fundamentally change? I don't\n> >> see how to handle the issues over at\n> >> https://lore.kernel.org/git/647D3D49-B85B-4B66-A857-695CFF9685EE@gmail.com/\n> >\n> > If this is useful enough for desktop users already, then that is a\n> > good enough reason to take it, I would say.\n> >\n> > GitLab can easily add WebUI that says \"You can define what diff\n> > algorithm is used for files with which suffix\" to allow you to\n> > configure a table like this:\n> >\n> > json | histogram\n> > c | patience\n> > * | myers\n> >\n> > and populate the server-side equivalent of .git/info/attributes and\n> > .git/config based on that, and without anything further than the\n> > posted patches, the result should just work, no?\n>\n> Yes, that is my thinking too. The goal was to make this user friendly for\n> everyone on the Git command line, and then GitLab can hook into this accordingly\n> to make it ussable with the GitLab UI.\n\nOk, sounds good. Merge away. :-)\n"},{"id":"472454","messageId":"Y/ZxT2ByET4BxGVt@coredump.intra.peff.net","threadId":"59197","inReplyTo":"CABPp-BHQn0sjAMwJ+r6uenO=nGLG1HvfnhS6tG8mu1BWt4bdOw@mail.gmail.com","subject":"Re: [PATCH v3 2/2] diff: teach diff to read algorithm from diff driver","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-22T19:47:27Z","receivedAt":"2023-02-22T19:47:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Feb 20, 2023 at 09:32:52AM -0800, Elijah Newren wrote:\n\n> > I was thinking we would just use the tree-ish of the first one\n> \n> That would certainly simplify, but it'd be pretty important to\n> document. (Incidentally, this kind of decision was my reason for\n> asking about all those special cases earlier, i.e. how to handle diff\n> between different commits, how to handle renames, how to handle bare\n> repositories, etc.)\n> \n> This kind of decision probably also means you'd need a variety of\n> testcases where .gitattributes is different in every commit & the\n> index & the working tree, and then you start testing several of the\n> possible pairings to make sure the right .gitattributes file is used\n> (e.g. (commit, commit), (commit, index), (index, commit), (worktree,\n> index), etc.)\n\nThere may be some prior art here in how we handle mailmaps in a bare\nrepository. In that case, we pull them from HEAD (or really any commit\nof your choosing, but the default is HEAD). That may seem a bit weird,\nbut it matches how non-bare repositories work, which read the mailmap\nfrom the working tree.\n\nSo likewise, even looking at an old commit like \"git show HEAD~1000\", in\na non-bare repository we will read .gitattributes from the working tree,\nwhich means it is (roughly) coming from HEAD. In some ways that is good\n(there may be improvements to the attributes) and in some ways it is\nweird and confusing (the meaning of the attributes may have been\ndifferent back then, or the two histories may even be somewhat\nunrelated!). So I think you can make an argument either way on what is\nuseful, but harmonizing the non-bare and bare cases seems like the best\nplace to start.\n\nAnd then that machinery would probably be enough to let people ask for\nspecific things on top (like \"git show --attributes-from=HEAD~1000\nHEAD~1000\" if they really wanted).\n\n-Peff\n"},{"id":"472658","messageId":"95D30379-4B31-4197-8D2F-71369E72514D@gmail.com","threadId":"59197","inReplyTo":"Y/ZxT2ByET4BxGVt@coredump.intra.peff.net","subject":"Re: [PATCH v3 2/2] diff: teach diff to read algorithm from diff driver","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2023-02-24T17:44:47Z","receivedAt":"2023-02-24T17:44:57Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hey Peff,\n\nOn 22 Feb 2023, at 14:47, Jeff King wrote:\n\n> On Mon, Feb 20, 2023 at 09:32:52AM -0800, Elijah Newren wrote:\n>\n>>> I was thinking we would just use the tree-ish of the first one\n>>\n>> That would certainly simplify, but it'd be pretty important to\n>> document. (Incidentally, this kind of decision was my reason for\n>> asking about all those special cases earlier, i.e. how to handle diff\n>> between different commits, how to handle renames, how to handle bare\n>> repositories, etc.)\n>>\n>> This kind of decision probably also means you'd need a variety of\n>> testcases where .gitattributes is different in every commit & the\n>> index & the working tree, and then you start testing several of the\n>> possible pairings to make sure the right .gitattributes file is used\n>> (e.g. (commit, commit), (commit, index), (index, commit), (worktree,\n>> index), etc.)\n>\n> There may be some prior art here in how we handle mailmaps in a bare\n> repository. In that case, we pull them from HEAD (or really any commit\n> of your choosing, but the default is HEAD). That may seem a bit weird,\n> but it matches how non-bare repositories work, which read the mailmap\n> from the working tree.\n>\n> So likewise, even looking at an old commit like \"git show HEAD~1000\", in\n> a non-bare repository we will read .gitattributes from the working tree,\n> which means it is (roughly) coming from HEAD. In some ways that is good\n> (there may be improvements to the attributes) and in some ways it is\n> weird and confusing (the meaning of the attributes may have been\n> different back then, or the two histories may even be somewhat\n> unrelated!). So I think you can make an argument either way on what is\n> useful, but harmonizing the non-bare and bare cases seems like the best\n> place to start.\n\nThanks for the historical context and suggestion here\n>\n> And then that machinery would probably be enough to let people ask for\n> specific things on top (like \"git show --attributes-from=HEAD~1000\n> HEAD~1000\" if they really wanted).\n\nMakes sense to me\n\nthanks\nJohn\n\n>\n> -Peff\n"}]}