{"thread":{"id":"45958","subject":"[RFC PATCH 00/19] Diff machine: highlight moved lines.","startedAt":"2017-05-14T04:01:31Z","lastAt":"2017-05-30T21:38:26Z","messageCount":128,"participants":["Stefan Beller","Jeff King","Junio C Hamano","Jonathan Tan","Brandon Williams","Jacob Keller"],"isPatch":true,"patchVersion":1,"patchTotal":19},"messages":[{"id":"319699","messageId":"20170514040117.25865-1-sbeller@google.com","threadId":"45958","inReplyTo":null,"subject":"[RFC PATCH 00/19] Diff machine: highlight moved lines.","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:00:58Z","receivedAt":"2017-05-14T04:01:31Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"For details on *why* see the commit message of the last commit.\n\nThe first five patches are slight refactorings to get into good\nshape, the next patches are funneling all output through emit_line_*.\n\nThe second last patch introduces an option to buffer up all output\nbefore printing, and then the last patch can color up moved lines\nof code.\n\nAny feedback welcome.\n\nThanks,\nStefan \n\n\nStefan Beller (19):\n diff: readability fix\n diff: move line ending check into emit_hunk_header\n diff.c: drop 'nofirst' from emit_line_0\n diff.c: factor out diff_flush_patch_all_file_pairs\n diff.c: emit_line_0 can handle no color setting\n diff: add emit_line_fmt\n diff.c: convert fn_out_consume to use emit_line_*\n diff.c: convert builtin_diff to use emit_line_*\n diff.c: convert emit_rewrite_diff to use emit_line_*\n diff.c: convert emit_rewrite_lines to use emit_line_*\n submodule.c: convert show_submodule_summary to use emit_line_fmt\n diff.c: convert emit_binary_diff_body to use emit_line_*\n diff.c: convert show_stats to use emit_line_*\n diff.c: convert word diffing to use emit_line_*\n diff.c: convert diff_flush to use emit_line_*\n diff.c: convert diff_summary to use emit_line_*\n diff.c: factor out emit_line_ws for coloring whitespaces\n diff: buffer all output if asked to\n diff.c: color moved lines differently\n\n Documentation/config.txt | 12 +-\n diff.c | 815 ++++++++++++++++++++++++++++++++++-----------\n diff.h | 69 +++-\n submodule.c | 78 ++---\n submodule.h | 9 +-\n t/t4015-diff-whitespace.sh | 229 +++++++++++++\n 6 files changed, 960 insertions(+), 252 deletions(-)\n\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319700","messageId":"20170514040117.25865-2-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 01/19] diff: readability fix","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:00:59Z","receivedAt":"2017-05-14T04:01:33Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"We already have dereferenced 'p->two' into a local variable 'two'. Use\nthat.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 74283d9001..3f5bf8b5a4 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -3283,8 +3283,8 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)\n \tconst char *other;\n \tconst char *attr_path;\n \n-\tname = p->one->path;\n-\tother = (strcmp(name, p->two->path) ? p->two->path : NULL);\n+\tname = one->path;\n+\tother = (strcmp(name, two->path) ? two->path : NULL);\n \tattr_path = name;\n \tif (o->prefix_length)\n \t\tstrip_prefix(o->prefix_length, &name, &other);\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319701","messageId":"20170514040117.25865-3-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 02/19] diff: move line ending check into emit_hunk_header","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:00Z","receivedAt":"2017-05-14T04:01:36Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis patch moves code that is conceptually part of\nemit_hunk_header, but was using output in fn_out_consume,\nback to emit_hunk_header.\n\nMeanwhile simplify it by using a function that is designed for it.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 3f5bf8b5a4..c2ed605cd0 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -677,6 +677,8 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \t}\n \n \tstrbuf_add(&msgbuf, line + len, org_len - len);\n+\tstrbuf_complete_line(&msgbuf);\n+\n \temit_line(ecbdata->opt, \"\", \"\", msgbuf.buf, msgbuf.len);\n \tstrbuf_release(&msgbuf);\n }\n@@ -1315,8 +1317,6 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\tlen = sane_truncate_line(ecbdata, line, len);\n \t\tfind_lno(line, ecbdata);\n \t\temit_hunk_header(ecbdata, line, len);\n-\t\tif (line[len-1] != '\\n')\n-\t\t\tputc('\\n', o->file);\n \t\treturn;\n \t}\n \n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319702","messageId":"20170514040117.25865-7-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 06/19] diff: add emit_line_fmt","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:04Z","receivedAt":"2017-05-14T04:01:41Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In the following patches we'll convert all printing functions to use\nthe emit_line_* family of functions.\n\nMany of the printing functions to be converted are formatted. So offer\na formatted function in the emit_line function family as well.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 14 ++++++++++++++\n 1 file changed, 14 insertions(+)\n\ndiff --git a/diff.c b/diff.c\nindex 48f0fb98dc..aef159a919 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -552,6 +552,20 @@ static void emit_line(struct diff_options *o, const char *set, const char *reset\n \temit_line_0(o, set, reset, 0, line, len);\n }\n \n+static void emit_line_fmt(struct diff_options *o,\n+\t\t\t const char *set, const char *reset,\n+\t\t\t const char *fmt, ...)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tva_list ap;\n+\tva_start(ap, fmt);\n+\tstrbuf_vaddf(&sb, fmt, ap);\n+\tva_end(ap);\n+\n+\temit_line(o, set, reset, sb.buf, sb.len);\n+\tstrbuf_release(&sb);\n+}\n+\n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n {\n \tif (!((ecbdata->ws_rule & WS_BLANK_AT_EOF) &&\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319703","messageId":"20170514040117.25865-4-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 03/19] diff.c: drop 'nofirst' from emit_line_0","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:01Z","receivedAt":"2017-05-14T04:01:43Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In 250f79930d (diff.c: split emit_line() from the first char and the rest\nof the line, 2009-09-14) we introduced the local variable 'nofirst' that\nindicates if we have no first sign character. With the given implementation\nwe had to use an extra variable unlike reusing 'first' because the lines\nfirst character could be '\\0'.\n\nChange the meaning of the 'first' argument to not mean the first character\nof the line, but rather just containing the sign that is prepended to the\nline. Refactor emit_line to not include the lines first character, but pass\nthe complete line as well as a '\\0' sign, which now serves as an indication\nnot to print a sign.\n\nWith this patch other callers hard code the sign (which are '+', '-',\n' ' and '\\\\') such that we do not run into unexpectedly emitting an\nerror-nous '\\0'.\n\nThe audit of the caller revealed that the sign cannot be '\\n' or '\\r',\nso remove that condition for trailing newline or carriage return in the\nsign; the else part of the condition handles the len==0 perfectly,\nso we can drop the if/else construct.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 40 +++++++++++++++++-----------------------\n 1 file changed, 17 insertions(+), 23 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex c2ed605cd0..4269b8dccf 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -517,33 +517,24 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n }\n \n static void emit_line_0(struct diff_options *o, const char *set, const char *reset,\n-\t\t\tint first, const char *line, int len)\n+\t\t\tint sign, const char *line, int len)\n {\n \tint has_trailing_newline, has_trailing_carriage_return;\n-\tint nofirst;\n \tFILE *file = o->file;\n \n \tfputs(diff_line_prefix(o), file);\n \n-\tif (len == 0) {\n-\t\thas_trailing_newline = (first == '\\n');\n-\t\thas_trailing_carriage_return = (!has_trailing_newline &&\n-\t\t\t\t\t\t(first == '\\r'));\n-\t\tnofirst = has_trailing_newline || has_trailing_carriage_return;\n-\t} else {\n-\t\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n-\t\tif (has_trailing_newline)\n-\t\t\tlen--;\n-\t\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n-\t\tif (has_trailing_carriage_return)\n-\t\t\tlen--;\n-\t\tnofirst = 0;\n-\t}\n+\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n+\tif (has_trailing_newline)\n+\t\tlen--;\n+\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n+\tif (has_trailing_carriage_return)\n+\t\tlen--;\n \n-\tif (len || !nofirst) {\n+\tif (len || sign) {\n \t\tfputs(set, file);\n-\t\tif (!nofirst)\n-\t\t\tfputc(first, file);\n+\t\tif (sign)\n+\t\t\tfputc(sign, file);\n \t\tfwrite(line, len, 1, file);\n \t\tfputs(reset, file);\n \t}\n@@ -556,7 +547,7 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n static void emit_line(struct diff_options *o, const char *set, const char *reset,\n \t\t const char *line, int len)\n {\n-\temit_line_0(o, set, reset, line[0], line+1, len-1);\n+\temit_line_0(o, set, reset, 0, line, len);\n }\n \n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n@@ -4822,9 +4813,12 @@ void diff_flush(struct diff_options *options)\n \n \tif (output_format & DIFF_FORMAT_PATCH) {\n \t\tif (separator) {\n-\t\t\tfprintf(options->file, \"%s%c\",\n-\t\t\t\tdiff_line_prefix(options),\n-\t\t\t\toptions->line_termination);\n+\t\t\tchar term[2];\n+\t\t\tterm[0] = options->line_termination;\n+\t\t\tterm[1] = '\\0';\n+\n+\t\t\temit_line(options, NULL, NULL,\n+\t\t\t\t term, 1);\n \t\t\tif (options->stat_sep) {\n \t\t\t\t/* attach patch instead of inline */\n \t\t\t\tfputs(options->stat_sep, options->file);\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319704","messageId":"20170514040117.25865-9-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 08/19] diff.c: convert builtin_diff to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:06Z","receivedAt":"2017-05-14T04:01:45Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers builtin_diff.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 43 ++++++++++++++++++++++++++-----------------\n 1 file changed, 26 insertions(+), 17 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 93343a9ccc..8e00010bf4 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1293,8 +1293,9 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \to->found_changes = 1;\n \n \tif (ecbdata->header) {\n-\t\tfprintf(o->file, \"%s\", ecbdata->header->buf);\n-\t\tstrbuf_reset(ecbdata->header);\n+\t\temit_line(o, NULL, NULL,\n+\t\t\t ecbdata->header->buf, ecbdata->header->len);\n+\t\tstrbuf_release(ecbdata->header);\n \t\tecbdata->header = NULL;\n \t}\n \n@@ -2407,7 +2408,7 @@ static void builtin_diff(const char *name_a,\n \tb_two = quote_two(b_prefix, name_b + (*name_b == '/'));\n \tlbl[0] = DIFF_FILE_VALID(one) ? a_one : \"/dev/null\";\n \tlbl[1] = DIFF_FILE_VALID(two) ? b_two : \"/dev/null\";\n-\tstrbuf_addf(&header, \"%s%sdiff --git %s %s%s\\n\", line_prefix, meta, a_one, b_two, reset);\n+\tstrbuf_addf(&header, \"%sdiff --git %s %s%s\\n\", meta, a_one, b_two, reset);\n \tif (lbl[0][0] == '/') {\n \t\t/* /dev/null */\n \t\tstrbuf_addf(&header, \"%s%snew file mode %06o%s\\n\", line_prefix, meta, two->mode, reset);\n@@ -2439,7 +2440,7 @@ static void builtin_diff(const char *name_a,\n \t\tif (complete_rewrite &&\n \t\t (textconv_one || !diff_filespec_is_binary(one)) &&\n \t\t (textconv_two || !diff_filespec_is_binary(two))) {\n-\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\t\temit_line(o, NULL, NULL, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t\temit_rewrite_diff(name_a, name_b, one, two,\n \t\t\t\t\t\ttextconv_one, textconv_two, o);\n@@ -2449,7 +2450,8 @@ static void builtin_diff(const char *name_a,\n \t}\n \n \tif (o->irreversible_delete && lbl[1][0] == '/') {\n-\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\tif (header.len)\n+\t\t\temit_line(o, NULL, NULL, header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tgoto free_ab_and_return;\n \t} else if (!DIFF_OPT_TST(o, TEXT) &&\n@@ -2459,13 +2461,16 @@ static void builtin_diff(const char *name_a,\n \t\t S_ISREG(one->mode) && S_ISREG(two->mode) &&\n \t\t !DIFF_OPT_TST(o, BINARY)) {\n \t\t\tif (!oidcmp(&one->oid, &two->oid)) {\n-\t\t\t\tif (must_show_header)\n-\t\t\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\t\t\tif (must_show_header && header.len)\n+\t\t\t\t\temit_line(o, NULL, NULL,\n+\t\t\t\t\t\t header.buf, header.len);\n \t\t\t\tgoto free_ab_and_return;\n \t\t\t}\n-\t\t\tfprintf(o->file, \"%s\", header.buf);\n-\t\t\tfprintf(o->file, \"%sBinary files %s and %s differ\\n\",\n-\t\t\t\tline_prefix, lbl[0], lbl[1]);\n+\t\t\tif (header.len)\n+\t\t\t\temit_line(o, NULL, NULL,\n+\t\t\t\t\t header.buf, header.len);\n+\t\t\temit_line_fmt(o, 0, 0, \"Binary files %s and %s differ\\n\",\n+\t\t\t\t lbl[0], lbl[1]);\n \t\t\tgoto free_ab_and_return;\n \t\t}\n \t\tif (fill_mmfile(&mf1, one) < 0 || fill_mmfile(&mf2, two) < 0)\n@@ -2473,17 +2478,21 @@ static void builtin_diff(const char *name_a,\n \t\t/* Quite common confusing case */\n \t\tif (mf1.size == mf2.size &&\n \t\t !memcmp(mf1.ptr, mf2.ptr, mf1.size)) {\n-\t\t\tif (must_show_header)\n-\t\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\t\tif (must_show_header && header.len)\n+\t\t\t\temit_line(o, NULL, NULL,\n+\t\t\t\t\t header.buf, header.len);\n \t\t\tgoto free_ab_and_return;\n \t\t}\n-\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\tif (header.len)\n+\t\t\temit_line(o, NULL, NULL,\n+\t\t\t\t header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tif (DIFF_OPT_TST(o, BINARY))\n \t\t\temit_binary_diff(o->file, &mf1, &mf2, line_prefix);\n \t\telse\n-\t\t\tfprintf(o->file, \"%sBinary files %s and %s differ\\n\",\n-\t\t\t\tline_prefix, lbl[0], lbl[1]);\n+\t\t\temit_line_fmt(o, NULL, NULL,\n+\t\t\t\t \"Binary files %s and %s differ\\n\",\n+\t\t\t\t lbl[0], lbl[1]);\n \t\to->found_changes = 1;\n \t} else {\n \t\t/* Crazy xdl interfaces.. */\n@@ -2494,8 +2503,8 @@ static void builtin_diff(const char *name_a,\n \t\tstruct emit_callback ecbdata;\n \t\tconst struct userdiff_funcname *pe;\n \n-\t\tif (must_show_header) {\n-\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\tif (must_show_header && header.len) {\n+\t\t\temit_line(o, NULL, NULL, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t}\n \n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319705","messageId":"20170514040117.25865-11-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 10/19] diff.c: convert emit_rewrite_lines to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:08Z","receivedAt":"2017-05-14T04:01:46Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers emit_rewrite_lines.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/diff.c b/diff.c\nindex e4b46fee4f..369c804f03 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -748,7 +748,7 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n \tif (!endp) {\n \t\tconst char *context = diff_get_color(ecb->color_diff,\n \t\t\t\t\t\t DIFF_CONTEXT);\n-\t\tputc('\\n', ecb->opt->file);\n+\t\temit_line(ecb->opt, NULL, NULL, \"\\n\", 1);\n \t\temit_line_0(ecb->opt, context, reset, '\\\\',\n \t\t\t nneof, strlen(nneof));\n \t}\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319706","messageId":"20170514040117.25865-8-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 07/19] diff.c: convert fn_out_consume to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:05Z","receivedAt":"2017-05-14T04:01:49Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers the parts of fn_out_consume.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 13 +++++--------\n 1 file changed, 5 insertions(+), 8 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex aef159a919..93343a9ccc 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1289,7 +1289,6 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \tconst char *context = diff_get_color(ecbdata->color_diff, DIFF_CONTEXT);\n \tconst char *reset = diff_get_color(ecbdata->color_diff, DIFF_RESET);\n \tstruct diff_options *o = ecbdata->opt;\n-\tconst char *line_prefix = diff_line_prefix(o);\n \n \to->found_changes = 1;\n \n@@ -1301,14 +1300,12 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \n \tif (ecbdata->label_path[0]) {\n \t\tconst char *name_a_tab, *name_b_tab;\n-\n \t\tname_a_tab = strchr(ecbdata->label_path[0], ' ') ? \"\\t\" : \"\";\n \t\tname_b_tab = strchr(ecbdata->label_path[1], ' ') ? \"\\t\" : \"\";\n-\n-\t\tfprintf(o->file, \"%s%s--- %s%s%s\\n\",\n-\t\t\tline_prefix, meta, ecbdata->label_path[0], reset, name_a_tab);\n-\t\tfprintf(o->file, \"%s%s+++ %s%s%s\\n\",\n-\t\t\tline_prefix, meta, ecbdata->label_path[1], reset, name_b_tab);\n+\t\temit_line_fmt(o, meta, reset, \"--- %s%s\\n\",\n+\t\t\t ecbdata->label_path[0], name_a_tab);\n+\t\temit_line_fmt(o, meta, reset, \"+++ %s%s\\n\",\n+\t\t\t ecbdata->label_path[1], name_b_tab);\n \t\tecbdata->label_path[0] = ecbdata->label_path[1] = NULL;\n \t}\n \n@@ -1349,7 +1346,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\tdiff_words_flush(ecbdata);\n \t\tif (ecbdata->diff_words->type == DIFF_WORDS_PORCELAIN) {\n \t\t\temit_line(o, context, reset, line, len);\n-\t\t\tfputs(\"~\\n\", o->file);\n+\t\t\temit_line(o, NULL, NULL, \"~\\n\", 2);\n \t\t} else {\n \t\t\t/*\n \t\t\t * Skip the prefix character, if any. With\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319707","messageId":"20170514040117.25865-12-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 11/19] submodule.c: convert show_submodule_summary to use emit_line_fmt","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:09Z","receivedAt":"2017-05-14T04:01:52Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis prepares the code for submodules to go through the\nemit_line function.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 18 +++++++-------\n diff.h | 5 ++++\n submodule.c | 78 ++++++++++++++++++++++++++++++-------------------------------\n submodule.h | 9 +++----\n 4 files changed, 55 insertions(+), 55 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 369c804f03..45ec311828 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -546,15 +546,15 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n \t\tfputc('\\n', file);\n }\n \n-static void emit_line(struct diff_options *o, const char *set, const char *reset,\n-\t\t const char *line, int len)\n+void emit_line(struct diff_options *o, const char *set, const char *reset,\n+\t const char *line, int len)\n {\n \temit_line_0(o, set, reset, 0, line, len);\n }\n \n-static void emit_line_fmt(struct diff_options *o,\n-\t\t\t const char *set, const char *reset,\n-\t\t\t const char *fmt, ...)\n+void emit_line_fmt(struct diff_options *o,\n+\t\t const char *set, const char *reset,\n+\t\t const char *fmt, ...)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tva_list ap;\n@@ -2379,8 +2379,7 @@ static void builtin_diff(const char *name_a,\n \t (!two->mode || S_ISGITLINK(two->mode))) {\n \t\tconst char *del = diff_get_color_opt(o, DIFF_FILE_OLD);\n \t\tconst char *add = diff_get_color_opt(o, DIFF_FILE_NEW);\n-\t\tshow_submodule_summary(o->file, one->path ? one->path : two->path,\n-\t\t\t\tline_prefix,\n+\t\tshow_submodule_summary(o, one->path ? one->path : two->path,\n \t\t\t\t&one->oid, &two->oid,\n \t\t\t\ttwo->dirty_submodule,\n \t\t\t\tmeta, del, add, reset);\n@@ -2390,11 +2389,10 @@ static void builtin_diff(const char *name_a,\n \t\t (!two->mode || S_ISGITLINK(two->mode))) {\n \t\tconst char *del = diff_get_color_opt(o, DIFF_FILE_OLD);\n \t\tconst char *add = diff_get_color_opt(o, DIFF_FILE_NEW);\n-\t\tshow_submodule_inline_diff(o->file, one->path ? one->path : two->path,\n-\t\t\t\tline_prefix,\n+\t\tshow_submodule_inline_diff(o, one->path ? one->path : two->path,\n \t\t\t\t&one->oid, &two->oid,\n \t\t\t\ttwo->dirty_submodule,\n-\t\t\t\tmeta, del, add, reset, o);\n+\t\t\t\tmeta, del, add, reset);\n \t\treturn;\n \t}\n \ndiff --git a/diff.h b/diff.h\nindex 5be1ee77a7..addebd5a0f 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -188,6 +188,11 @@ struct diff_options {\n \tint diff_path_counter;\n };\n \n+void emit_line_fmt(struct diff_options *o, const char *set, const char *reset,\n+\t\t const char *fmt, ...);\n+void emit_line(struct diff_options *o, const char *set, const char *reset,\n+\t const char *line, int len);\n+\n enum color_diff {\n \tDIFF_RESET = 0,\n \tDIFF_CONTEXT = 1,\ndiff --git a/submodule.c b/submodule.c\nindex d3299e29c0..cfad469a2f 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -362,8 +362,8 @@ static int prepare_submodule_summary(struct rev_info *rev, const char *path,\n \treturn prepare_revision_walk(rev);\n }\n \n-static void print_submodule_summary(struct rev_info *rev, FILE *f,\n-\t\tconst char *line_prefix,\n+static void print_submodule_summary(struct rev_info *rev,\n+\t\tstruct diff_options *o,\n \t\tconst char *del, const char *add, const char *reset)\n {\n \tstatic const char format[] = \" %m %s\";\n@@ -375,18 +375,12 @@ static void print_submodule_summary(struct rev_info *rev, FILE *f,\n \t\tctx.date_mode = rev->date_mode;\n \t\tctx.output_encoding = get_log_output_encoding();\n \t\tstrbuf_setlen(&sb, 0);\n-\t\tstrbuf_addstr(&sb, line_prefix);\n-\t\tif (commit->object.flags & SYMMETRIC_LEFT) {\n-\t\t\tif (del)\n-\t\t\t\tstrbuf_addstr(&sb, del);\n-\t\t}\n-\t\telse if (add)\n-\t\t\tstrbuf_addstr(&sb, add);\n \t\tformat_commit_message(commit, format, &sb, &ctx);\n-\t\tif (reset)\n-\t\t\tstrbuf_addstr(&sb, reset);\n \t\tstrbuf_addch(&sb, '\\n');\n-\t\tfprintf(f, \"%s\", sb.buf);\n+\t\tif (commit->object.flags & SYMMETRIC_LEFT)\n+\t\t\temit_line(o, del, reset, sb.buf, sb.len);\n+\t\telse if (add)\n+\t\t\temit_line(o, add, reset, sb.buf, sb.len);\n \t}\n \tstrbuf_release(&sb);\n }\n@@ -413,8 +407,7 @@ void prepare_submodule_repo_env(struct argv_array *out)\n * attempt to lookup both the left and right commits and put them into the\n * left and right pointers.\n */\n-static void show_submodule_header(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+static void show_submodule_header(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n \t\tconst char *reset,\n@@ -426,11 +419,11 @@ static void show_submodule_header(FILE *f, const char *path,\n \tint fast_forward = 0, fast_backward = 0;\n \n \tif (dirty_submodule & DIRTY_SUBMODULE_UNTRACKED)\n-\t\tfprintf(f, \"%sSubmodule %s contains untracked content\\n\",\n-\t\t\tline_prefix, path);\n+\t\temit_line_fmt(o, NULL, NULL,\n+\t\t\t \"Submodule %s contains untracked content\\n\", path);\n \tif (dirty_submodule & DIRTY_SUBMODULE_MODIFIED)\n-\t\tfprintf(f, \"%sSubmodule %s contains modified content\\n\",\n-\t\t\tline_prefix, path);\n+\t\temit_line_fmt(o, NULL, NULL,\n+\t\t\t \"Submodule %s contains modified content\\n\", path);\n \n \tif (is_null_oid(one))\n \t\tmessage = \"(new submodule)\";\n@@ -472,21 +465,20 @@ static void show_submodule_header(FILE *f, const char *path,\n \t}\n \n output_header:\n-\tstrbuf_addf(&sb, \"%s%sSubmodule %s \", line_prefix, meta, path);\n+\tstrbuf_addf(&sb, \"Submodule %s \", path);\n \tstrbuf_add_unique_abbrev(&sb, one->hash, DEFAULT_ABBREV);\n \tstrbuf_addstr(&sb, (fast_backward || fast_forward) ? \"..\" : \"...\");\n \tstrbuf_add_unique_abbrev(&sb, two->hash, DEFAULT_ABBREV);\n \tif (message)\n-\t\tstrbuf_addf(&sb, \" %s%s\\n\", message, reset);\n+\t\tstrbuf_addf(&sb, \" %s\\n\", message);\n \telse\n-\t\tstrbuf_addf(&sb, \"%s:%s\\n\", fast_backward ? \" (rewind)\" : \"\", reset);\n-\tfwrite(sb.buf, sb.len, 1, f);\n+\t\tstrbuf_addf(&sb, \"%s:\\n\", fast_backward ? \" (rewind)\" : \"\");\n+\temit_line(o, meta, reset, sb.buf, sb.len);\n \n \tstrbuf_release(&sb);\n }\n \n-void show_submodule_summary(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+void show_submodule_summary(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n \t\tconst char *del, const char *add, const char *reset)\n@@ -495,7 +487,7 @@ void show_submodule_summary(FILE *f, const char *path,\n \tstruct commit *left = NULL, *right = NULL;\n \tstruct commit_list *merge_bases = NULL;\n \n-\tshow_submodule_header(f, path, line_prefix, one, two, dirty_submodule,\n+\tshow_submodule_header(o, path, one, two, dirty_submodule,\n \t\t\t meta, reset, &left, &right, &merge_bases);\n \n \t/*\n@@ -508,11 +500,12 @@ void show_submodule_summary(FILE *f, const char *path,\n \n \t/* Treat revision walker failure the same as missing commits */\n \tif (prepare_submodule_summary(&rev, path, left, right, merge_bases)) {\n-\t\tfprintf(f, \"%s(revision walker failed)\\n\", line_prefix);\n+\t\tconst char *error = \"(revision walker failed)\\n\";\n+\t\temit_line(o, NULL, NULL, error, strlen(error));\n \t\tgoto out;\n \t}\n \n-\tprint_submodule_summary(&rev, f, line_prefix, del, add, reset);\n+\tprint_submodule_summary(&rev, o, del, add, reset);\n \n out:\n \tif (merge_bases)\n@@ -521,20 +514,18 @@ void show_submodule_summary(FILE *f, const char *path,\n \tclear_commit_marks(right, ~0);\n }\n \n-void show_submodule_inline_diff(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+void show_submodule_inline_diff(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n-\t\tconst char *del, const char *add, const char *reset,\n-\t\tconst struct diff_options *o)\n+\t\tconst char *del, const char *add, const char *reset)\n {\n \tconst struct object_id *old = &empty_tree_oid, *new = &empty_tree_oid;\n \tstruct commit *left = NULL, *right = NULL;\n \tstruct commit_list *merge_bases = NULL;\n-\tstruct strbuf submodule_dir = STRBUF_INIT;\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tstruct strbuf sb = STRBUF_INIT;\n \n-\tshow_submodule_header(f, path, line_prefix, one, two, dirty_submodule,\n+\tshow_submodule_header(o, path, one, two, dirty_submodule,\n \t\t\t meta, reset, &left, &right, &merge_bases);\n \n \t/* We need a valid left and right commit to display a difference */\n@@ -547,15 +538,14 @@ void show_submodule_inline_diff(FILE *f, const char *path,\n \tif (right)\n \t\tnew = two;\n \n-\tfflush(f);\n \tcp.git_cmd = 1;\n \tcp.dir = path;\n-\tcp.out = dup(fileno(f));\n+\tcp.out = -1;\n \tcp.no_stdin = 1;\n \n \t/* TODO: other options may need to be passed here. */\n \targv_array_push(&cp.args, \"diff\");\n-\targv_array_pushf(&cp.args, \"--line-prefix=%s\", line_prefix);\n+\targv_array_pushf(&cp.args, \"--line-prefix=%s\", diff_line_prefix(o));\n \tif (DIFF_OPT_TST(o, REVERSE_DIFF)) {\n \t\targv_array_pushf(&cp.args, \"--src-prefix=%s%s/\",\n \t\t\t\t o->b_prefix, path);\n@@ -578,11 +568,21 @@ void show_submodule_inline_diff(FILE *f, const char *path,\n \t\targv_array_push(&cp.args, oid_to_hex(new));\n \n \tprepare_submodule_repo_env(&cp.env_array);\n-\tif (run_command(&cp))\n-\t\tfprintf(f, \"(diff failed)\\n\");\n+\tif (start_command(&cp)) {\n+\t\tconst char *error = \"(diff failed)\\n\";\n+\t\temit_line(o, NULL, NULL, error, strlen(error));\n+\t}\n+\n+\twhile (strbuf_getwholeline_fd(&sb, cp.out, '\\n') != EOF)\n+\t\temit_line(o, NULL, NULL, sb.buf, sb.len);\n+\n+\tif (finish_command(&cp)) {\n+\t\tconst char *error = \"(diff failed)\\n\";\n+\t\temit_line(o, NULL, NULL, error, strlen(error));\n+\t}\n \n done:\n-\tstrbuf_release(&submodule_dir);\n+\tstrbuf_release(&sb);\n \tif (merge_bases)\n \t\tfree_commit_list(merge_bases);\n \tif (left)\ndiff --git a/submodule.h b/submodule.h\nindex 1277480add..9df0a3aea2 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -53,17 +53,14 @@ extern int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst);\n extern const char *submodule_strategy_to_string(const struct submodule_update_strategy *s);\n extern void handle_ignore_submodules_arg(struct diff_options *, const char *);\n-extern void show_submodule_summary(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+extern void show_submodule_summary(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n \t\tconst char *del, const char *add, const char *reset);\n-extern void show_submodule_inline_diff(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+extern void show_submodule_inline_diff(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n-\t\tconst char *del, const char *add, const char *reset,\n-\t\tconst struct diff_options *opt);\n+\t\tconst char *del, const char *add, const char *reset);\n extern void set_config_fetch_recurse_submodules(int value);\n extern void set_config_update_recurse_submodules(int value);\n /* Check if we want to update any submodule.*/\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319708","messageId":"20170514040117.25865-18-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 17/19] diff.c: factor out emit_line_ws for coloring whitespaces","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:15Z","receivedAt":"2017-05-14T04:01:56Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Introduce a helper that calls ws_check_emit. We'll have it as a helper\nas in a later patch we'll add buffering in there.\n\nIn a later patch we want to buffer up all output and to do that\nwe'll need to keep around information for outputting a line such\nas the whitespace information. We choose to put this information into\na new struct 'buffered_filepair', which later will be persisting longer\nthan builtin_diff.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 18 +++++++++++++-----\n 1 file changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 899dc69dff..08dcc56bb9 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -552,6 +552,16 @@ void emit_line(struct diff_options *o, const char *set, const char *reset,\n \temit_line_0(o, set, reset, 0, line, len);\n }\n \n+static void emit_line_ws(struct diff_options *o,\n+\t\t\t const char *set, const char *reset, int sign,\n+\t\t\t const char *line, int len,\n+\t\t\t const char *ws, unsigned ws_rule)\n+{\n+\temit_line_0(o, set, reset, sign, \"\", 0);\n+\tws_check_emit(line, len, ws_rule,\n+\t\t o->file, set, reset, ws);\n+}\n+\n void emit_line_fmt(struct diff_options *o,\n \t\t const char *set, const char *reset,\n \t\t const char *fmt, ...)\n@@ -598,12 +608,10 @@ static void emit_line_checked(const char *reset,\n \telse if (sign == '+' && new_blank_line_at_eof(ecbdata, line, len))\n \t\t/* Blank line at EOF - paint '+' as well */\n \t\temit_line_0(ecbdata->opt, ws, reset, sign, line, len);\n-\telse {\n+\telse\n \t\t/* Emit just the prefix, then the rest. */\n-\t\temit_line_0(ecbdata->opt, set, reset, sign, \"\", 0);\n-\t\tws_check_emit(line, len, ecbdata->ws_rule,\n-\t\t\t ecbdata->opt->file, set, reset, ws);\n-\t}\n+\t\temit_line_ws(ecbdata->opt, set, reset, sign, line, len,\n+\t\t\t ws, ecbdata->ws_rule);\n }\n \n static void emit_add_line(const char *reset,\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319709","messageId":"20170514040117.25865-15-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 14/19] diff.c: convert word diffing to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:12Z","receivedAt":"2017-05-14T04:01:58Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers all code related to diffing words.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 56 ++++++++++++++++++++++++++++----------------------------\n 1 file changed, 28 insertions(+), 28 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 91dc045a45..07041a35ad 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -886,37 +886,38 @@ struct diff_words_data {\n \tstruct diff_words_style *style;\n };\n \n-static int fn_out_diff_words_write_helper(FILE *fp,\n+static int fn_out_diff_words_write_helper(struct diff_options *o,\n \t\t\t\t\t struct diff_words_style_elem *st_el,\n \t\t\t\t\t const char *newline,\n \t\t\t\t\t size_t count, const char *buf,\n \t\t\t\t\t const char *line_prefix)\n {\n-\tint print = 0;\n+\tstruct strbuf sb = STRBUF_INIT;\n \n \twhile (count) {\n \t\tchar *p = memchr(buf, '\\n', count);\n-\t\tif (print)\n-\t\t\tfputs(line_prefix, fp);\n+\n \t\tif (p != buf) {\n-\t\t\tif (st_el->color && fputs(st_el->color, fp) < 0)\n-\t\t\t\treturn -1;\n-\t\t\tif (fputs(st_el->prefix, fp) < 0 ||\n-\t\t\t fwrite(buf, p ? p - buf : count, 1, fp) != 1 ||\n-\t\t\t fputs(st_el->suffix, fp) < 0)\n-\t\t\t\treturn -1;\n-\t\t\tif (st_el->color && *st_el->color\n-\t\t\t && fputs(GIT_COLOR_RESET, fp) < 0)\n-\t\t\t\treturn -1;\n+\t\t\tif (st_el->color)\n+\t\t\t\tstrbuf_addstr(&sb, st_el->color);\n+\t\t\tstrbuf_addstr(&sb, st_el->prefix);\n+\t\t\tstrbuf_add(&sb, buf, p ? p - buf : count);\n+\t\t\tstrbuf_addstr(&sb, st_el->suffix);\n+\t\t\tif (st_el->color && *st_el->color)\n+\t\t\t strbuf_addstr(&sb, GIT_COLOR_RESET);\n \t\t}\n \t\tif (!p)\n-\t\t\treturn 0;\n-\t\tif (fputs(newline, fp) < 0)\n-\t\t\treturn -1;\n+\t\t\tgoto out;\n+\t\tstrbuf_addstr(&sb, newline);\n+\t\temit_line(o, NULL, NULL, sb.buf, sb.len);\n+\t\tstrbuf_reset(&sb);\n \t\tcount -= p + 1 - buf;\n \t\tbuf = p + 1;\n-\t\tprint = 1;\n \t}\n+out:\n+\tif (sb.len)\n+\t\temit_line(o, NULL, NULL, sb.buf, sb.len);\n+\tstrbuf_release(&sb);\n \treturn 0;\n }\n \n@@ -994,25 +995,25 @@ static void fn_out_diff_words_aux(void *priv, char *line, unsigned long len)\n \t} else\n \t\tplus_begin = plus_end = diff_words->plus.orig[plus_first].end;\n \n-\tif (color_words_output_graph_prefix(diff_words)) {\n-\t\tfputs(line_prefix, diff_words->opt->file);\n-\t}\n+\tif (color_words_output_graph_prefix(diff_words))\n+\t\temit_line(diff_words->opt, NULL, NULL, \"\", 0);\n+\n \tif (diff_words->current_plus != plus_begin) {\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t\t&style->ctx, style->newline,\n \t\t\t\tplus_begin - diff_words->current_plus,\n \t\t\t\tdiff_words->current_plus, line_prefix);\n \t\tif (*(plus_begin - 1) == '\\n')\n-\t\t\tfputs(line_prefix, diff_words->opt->file);\n+\t\t\temit_line(diff_words->opt, NULL, NULL, \"\", 0);\n \t}\n \tif (minus_begin != minus_end) {\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t\t&style->old, style->newline,\n \t\t\t\tminus_end - minus_begin, minus_begin,\n \t\t\t\tline_prefix);\n \t}\n \tif (plus_begin != plus_end) {\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t\t&style->new, style->newline,\n \t\t\t\tplus_end - plus_begin, plus_begin,\n \t\t\t\tline_prefix);\n@@ -1109,8 +1110,7 @@ static void diff_words_show(struct diff_words_data *diff_words)\n \n \t/* special case: only removal */\n \tif (!diff_words->plus.text.size) {\n-\t\tfputs(line_prefix, diff_words->opt->file);\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t&style->old, style->newline,\n \t\t\tdiff_words->minus.text.size,\n \t\t\tdiff_words->minus.text.ptr, line_prefix);\n@@ -1136,8 +1136,8 @@ static void diff_words_show(struct diff_words_data *diff_words)\n \tif (diff_words->current_plus != diff_words->plus.text.ptr +\n \t\t\tdiff_words->plus.text.size) {\n \t\tif (color_words_output_graph_prefix(diff_words))\n-\t\t\tfputs(line_prefix, diff_words->opt->file);\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\t\temit_line(diff_words->opt, NULL, NULL, \"\", 0);\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t&style->ctx, style->newline,\n \t\t\tdiff_words->plus.text.ptr + diff_words->plus.text.size\n \t\t\t- diff_words->current_plus, diff_words->current_plus,\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319710","messageId":"20170514040117.25865-19-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 18/19] diff: buffer all output if asked to","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:16Z","receivedAt":"2017-05-14T04:02:01Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Introduce a new option 'use_buffer' in the struct diff_options which\ncontrols whether all output is buffered up until all output is available.\n\nWe'll have two new structs in diff.h, one of them 'buffered_patch_line'\nwill be used to buffer each line, and the other 'buffered_patch_file_pair'\nwill store information about information relevant on a per file basis.\nThe buffered_patch_line will duplicate the memory of the line to buffer\nas that is easiest to reason about for now. In a future patch we may want\nto decrease the memory usage by not duplicating all output for buffering\nbut rather we may want to store offsets into the file or in case of hunk\ndescriptions such as the similarity score, we could just store the relevant\nnumber and reproduce the text later on.\n\nThis approach was chosen as a first step because it is quite simple\ncompared to the alternative with less memory footprint.\n\nemit_line_0 factors out the emission part into emit_line_emission,\nand depending on the diff_options->use_buffer the emission\nwill be performed directly when calling emit_line_0 or after the\nwhole process is done, i.e. by buffering we have add the possibility\nfor a second pass over the whole output before doing the actual\noutput.\n\nIn 6440d34 (2012-03-14, diff: tweak a _copy_ of diff_options with\nword-diff) we introduced a duplicate diff options struct for word\nemissions as we may have different regex settings in there.\n\nWhen buffering the output, we need to operate on just one buffer,\nso we have to copy back the emissions of the word buffer into the main\nbuffer.\n\nUnconditionally enable output via buffer in this patch as it yields\na great opportunity for testing, i.e. all the diff tests from the\ntest suite pass without having reordering issues (i.e. only parts\nof the output got buffered, and we forgot to buffer other parts).\nThe test suite passes, which gives confidence that we converted all\nfunctions to use emit_line_* for output.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 158 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------\n diff.h | 39 ++++++++++++++++\n 2 files changed, 181 insertions(+), 16 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 08dcc56bb9..dbab7fb44e 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -516,29 +516,29 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n-static void emit_line_0(struct diff_options *o, const char *set, const char *reset,\n-\t\t\tint sign, const char *line, int len)\n+static void emit_buffered_patch_line(struct diff_options *o,\n+\t\t\t\t struct buffered_patch_line *e)\n {\n-\tint has_trailing_newline, has_trailing_carriage_return;\n+\tint has_trailing_newline, has_trailing_carriage_return, len = e->len;\n \tFILE *file = o->file;\n \n \tfputs(diff_line_prefix(o), file);\n \n-\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n+\thas_trailing_newline = (len > 0 && e->line[len-1] == '\\n');\n \tif (has_trailing_newline)\n \t\tlen--;\n-\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n+\thas_trailing_carriage_return = (len > 0 && e->line[len-1] == '\\r');\n \tif (has_trailing_carriage_return)\n \t\tlen--;\n \n-\tif (len || sign) {\n-\t\tif (set)\n-\t\t\tfputs(set, file);\n-\t\tif (sign)\n-\t\t\tfputc(sign, file);\n-\t\tfwrite(line, len, 1, file);\n-\t\tif (reset)\n-\t\t\tfputs(reset, file);\n+\tif (len || e->sign) {\n+\t\tif (e->set)\n+\t\t\tfputs(e->set, file);\n+\t\tif (e->sign)\n+\t\t\tfputc(e->sign, file);\n+\t\tfwrite(e->line, len, 1, file);\n+\t\tif (e->reset)\n+\t\t\tfputs(e->reset, file);\n \t}\n \tif (has_trailing_carriage_return)\n \t\tfputc('\\r', file);\n@@ -546,6 +546,65 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n \t\tfputc('\\n', file);\n }\n \n+static void emit_buffered_patch_line_ws(struct diff_options *o,\n+\t\t\t\t\tstruct buffered_patch_line *e,\n+\t\t\t\t\tconst char *ws, unsigned ws_rule)\n+{\n+\tstruct buffered_patch_line s = {e->set, e->reset, \"\", 0, e->sign};\n+\n+\temit_buffered_patch_line(o, &s);\n+\tws_check_emit(e->line, e->len, ws_rule,\n+\t\t o->file, e->set, e->reset, ws);\n+}\n+\n+static void process_next_buffered_patch_line(struct diff_options *o, int line_no)\n+{\n+\tstruct buffered_patch_line *e = &o->line_buffer[line_no];\n+\n+\tconst char *ws = o->current_filepair->ws;\n+\tunsigned ws_rule = o->current_filepair->ws_rule;\n+\n+\tswitch (e->state) {\n+\t\tcase BPL_EMIT_LINE_ASIS:\n+\t\t\temit_buffered_patch_line(o, e);\n+\t\t\tbreak;\n+\t\tcase BPL_EMIT_LINE_WS:\n+\t\t\temit_buffered_patch_line_ws(o, e, ws, ws_rule);\n+\t\t\tbreak;\n+\t\tcase BPL_HANDOVER:\n+\t\t\to->current_filepair++;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tdie(\"BUG: malformatted buffered patch line: '%d'\", e->state);\n+\t}\n+}\n+\n+static void append_buffered_patch_line(struct diff_options *o,\n+\t\t\t\t struct buffered_patch_line *e)\n+{\n+\tstruct buffered_patch_line *f;\n+\tALLOC_GROW(o->line_buffer,\n+\t\t o->line_buffer_nr + 1,\n+\t\t o->line_buffer_alloc);\n+\tf = &o->line_buffer[o->line_buffer_nr];\n+\to->line_buffer_nr++;\n+\n+\tmemcpy(f, e, sizeof(struct buffered_patch_line));\n+\tf->line = e->line ? xmemdupz(e->line, e->len) : NULL;\n+}\n+\n+static void emit_line_0(struct diff_options *o,\n+\t\t\tconst char *set, const char *reset,\n+\t\t\tint sign, const char *line, int len)\n+{\n+\tstruct buffered_patch_line e = {set, reset, line, len, sign, BPL_EMIT_LINE_ASIS};\n+\n+\tif (o->use_buffer)\n+\t\tappend_buffered_patch_line(o, &e);\n+\telse\n+\t\temit_buffered_patch_line(o, &e);\n+}\n+\n void emit_line(struct diff_options *o, const char *set, const char *reset,\n \t const char *line, int len)\n {\n@@ -557,9 +616,12 @@ static void emit_line_ws(struct diff_options *o,\n \t\t\t const char *line, int len,\n \t\t\t const char *ws, unsigned ws_rule)\n {\n-\temit_line_0(o, set, reset, sign, \"\", 0);\n-\tws_check_emit(line, len, ws_rule,\n-\t\t o->file, set, reset, ws);\n+\tstruct buffered_patch_line e = {set, reset, line, len, sign, BPL_EMIT_LINE_WS};\n+\n+\tif (o->use_buffer)\n+\t\tappend_buffered_patch_line(o, &e);\n+\telse\n+\t\temit_buffered_patch_line_ws(o, &e, ws, ws_rule);\n }\n \n void emit_line_fmt(struct diff_options *o,\n@@ -1160,6 +1222,16 @@ static void diff_words_flush(struct emit_callback *ecbdata)\n \tif (ecbdata->diff_words->minus.text.size ||\n \t ecbdata->diff_words->plus.text.size)\n \t\tdiff_words_show(ecbdata->diff_words);\n+\n+\tif (ecbdata->diff_words->opt->line_buffer_nr) {\n+\t\tint i;\n+\t\tfor (i = 0; i < ecbdata->diff_words->opt->line_buffer_nr; i++)\n+\t\t\tappend_buffered_patch_line(ecbdata->opt,\n+\t\t\t\t&ecbdata->diff_words->opt->line_buffer[i]);\n+\n+\t\tecbdata->diff_words->opt->line_buffer_nr = 0;\n+\t\t/* TODO: free memory as well */\n+\t}\n }\n \n static void diff_filespec_load_driver(struct diff_filespec *one)\n@@ -1195,6 +1267,11 @@ static void init_diff_words_data(struct emit_callback *ecbdata,\n \t\txcalloc(1, sizeof(struct diff_words_data));\n \tecbdata->diff_words->type = o->word_diff;\n \tecbdata->diff_words->opt = o;\n+\n+\to->line_buffer = NULL;\n+\to->line_buffer_nr = 0;\n+\to->line_buffer_alloc = 0;\n+\n \tif (!o->word_regex)\n \t\to->word_regex = userdiff_word_regex(one);\n \tif (!o->word_regex)\n@@ -2568,9 +2645,25 @@ static void builtin_diff(const char *name_a,\n \t\t\txecfg.ctxlen = strtoul(v, NULL, 10);\n \t\tif (o->word_diff)\n \t\t\tinit_diff_words_data(&ecbdata, o, one, two);\n+\t\tif (o->use_buffer) {\n+\t\t\tALLOC_GROW(o->filepair_buffer,\n+\t\t\t\t o->filepair_buffer_nr + 1,\n+\t\t\t\t o->filepair_buffer_alloc);\n+\t\t\to->current_filepair =\n+\t\t\t\t&o->filepair_buffer[o->filepair_buffer_nr++];\n+\n+\t\t\to->current_filepair->ws_rule = ecbdata.ws_rule;\n+\t\t\to->current_filepair->ws =\n+\t\t\t\tdiff_get_color(ecbdata.color_diff, DIFF_WHITESPACE);\n+\t\t}\n \t\tif (xdi_diff_outf(&mf1, &mf2, fn_out_consume, &ecbdata,\n \t\t\t\t &xpp, &xecfg))\n \t\t\tdie(\"unable to generate diff for %s\", one->path);\n+\t\tif (o->use_buffer) {\n+\t\t\tstruct buffered_patch_line e = BUFFERED_PATCH_LINE_INIT;\n+\t\t\te.state = BPL_HANDOVER; /* handover to next file pair */\n+\t\t\tappend_buffered_patch_line(o, &e);\n+\t\t}\n \t\tif (o->word_diff)\n \t\t\tfree_diff_words_data(&ecbdata);\n \t\tif (textconv_one)\n@@ -4785,11 +4878,44 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n {\n \tint i;\n \tstruct diff_queue_struct *q = &diff_queued_diff;\n+\t/*\n+\t * For testing purposes we want to make sure the diff machinery\n+\t * works completely with the buffer. If there is anything emitted\n+\t * outside the emit_buffered_patch_line, then the order is screwed\n+\t * up and the tests will fail.\n+\t *\n+\t * TODO (later in this series):\n+\t * We'll unset this flag in a later patch.\n+\t */\n+\to->use_buffer = 1;\n+\n+\tif (o->use_buffer) {\n+\t\tALLOC_GROW(o->filepair_buffer,\n+\t\t\t o->filepair_buffer_nr + 1,\n+\t\t\t o->filepair_buffer_alloc);\n+\t\to->current_filepair = &o->filepair_buffer[o->filepair_buffer_nr];\n+\t}\n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n \t\tif (check_pair_status(p))\n \t\t\tdiff_flush_patch(p, o);\n \t}\n+\n+\tif (o->use_buffer) {\n+\t\to->current_filepair = &o->filepair_buffer[0];\n+\t\tfor (i = 0; i < o->line_buffer_nr; i++)\n+\t\t\tprocess_next_buffered_patch_line(o, i);\n+\n+\t\tfor (i = 0; i < o->line_buffer_nr; i++);\n+\t\t\tfree((void*)o->line_buffer[i].line);\n+\n+\t\to->line_buffer = NULL;\n+\t\to->line_buffer_nr = 0;\n+\t\tfree(o->line_buffer);\n+\t\to->filepair_buffer = NULL;\n+\t\to->filepair_buffer_nr = 0;\n+\t\tfree(o->filepair_buffer);\n+\t}\n }\n \n void diff_flush(struct diff_options *options)\ndiff --git a/diff.h b/diff.h\nindex 5e89481769..c334aac02e 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -115,6 +115,36 @@ enum diff_submodule_format {\n \tDIFF_SUBMODULE_INLINE_DIFF\n };\n \n+/*\n+ * This struct is used when we need to buffer the output of the diff output.\n+ *\n+ * NEEDSWORK: Instead of storing a copy of the line, add an offset pointer\n+ * into the pre/post image file. This pointer could be a union with the\n+ * line pointer. By storing an offset into the file instead of the literal line,\n+ * we can decrease the memory footprint for the buffered output. At first we\n+ * may want to only have indirection for the content lines, but we could\n+ * also have an enum (based on sign?) that stores prefabricated lines, e.g.\n+ * the similarity score line or hunk/file headers.\n+ */\n+struct buffered_patch_line {\n+\tconst char *set;\n+\tconst char *reset;\n+\tconst char *line;\n+\tint len;\n+\tint sign;\n+\tenum {\n+\t\tBPL_EMIT_LINE_WS,\n+\t\tBPL_EMIT_LINE_ASIS,\n+\t\tBPL_HANDOVER\n+\t} state;\n+};\n+#define BUFFERED_PATCH_LINE_INIT {NULL, NULL, NULL, 0, 0, 0}\n+\n+struct buffered_filepair {\n+\tconst char *ws;\n+\tunsigned ws_rule;\n+};\n+\n struct diff_options {\n \tconst char *orderfile;\n \tconst char *pickaxe;\n@@ -186,6 +216,15 @@ struct diff_options {\n \tvoid *output_prefix_data;\n \n \tint diff_path_counter;\n+\n+\tint use_buffer;\n+\n+\tstruct buffered_patch_line *line_buffer;\n+\tint line_buffer_nr, line_buffer_alloc;\n+\n+\tstruct buffered_filepair *filepair_buffer;\n+\tint filepair_buffer_nr, filepair_buffer_alloc;\n+\tstruct buffered_filepair *current_filepair;\n };\n \n void emit_line_fmt(struct diff_options *o, const char *set, const char *reset,\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319711","messageId":"20170514040117.25865-20-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 19/19] diff.c: color moved lines differently","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:17Z","receivedAt":"2017-05-14T04:02:07Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"When there is a lot of code moved around such as in 11979b9 (2005-11-18,\n\"http.c: reorder to avoid compilation failure.\") for example, the review\nprocess is quite hard, as it is not mentally challenging. It is a rather\ntedious process, that gets boring quickly. However you still need to read\nthrough all of the code to make sure the moved lines are there as supposed.\n\nWhile it is trivial to color up a patch like the following\n\n $ git diff\n diff --git a/file2.c b/file2.c\n index 9163a0f..8e66dc0 100644\n --- a/file2.c\n +++ b/file2.c\n @@ -3,13 +3,6 @@ void *xmemdupz(const void *data, size_t len)\n return memcpy(xmallocz(len), data, len);\n }\n\n -int secure_foo(struct user *u)\n -{\n - if (!u->is_allowed_foo)\n - return;\n - foo(u);\n -}\n -\n char *xstrndup(const char *str, size_t len)\n {\n char *p = memchr(str, '\\0', len);\n diff --git a/test.c b/test.c\n index a95e6fe..81eb0eb 100644\n --- a/test.c\n +++ b/test.c\n @@ -18,6 +18,13 @@ ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset)\n return total;\n }\n\n +int secure_foo(struct user *u)\n +{\n + if (!u->is_allowed_foo)\n + return;\n + foo(u);\n +}\n +\n int xdup(int fd)\n {\n int ret = dup(fd);\n\nas in this patch all lines that add or remove lines\nshould be colored in the new color that indicates moved\nlines.\n\nHowever the intention of this patch is to aid reviewers\nto spotting permutations in the moved code. So consider the\nfollowing malicious move:\n\n diff --git a/file2.c b/file2.c\n index 9163a0f..8e66dc0 100644\n --- a/file2.c\n +++ b/file2.c\n @@ -3,13 +3,6 @@ void *xmemdupz(const void *data, size_t len)\n return memcpy(xmallocz(len), data, len);\n }\n\n -int secure_foo(struct user *u)\n -{\n - if (!u->is_allowed_foo)\n - return;\n - foo(u);\n -}\n -\n char *xstrndup(const char *str, size_t len)\n {\n char *p = memchr(str, '\\0', len);\n diff --git a/test.c b/test.c\n index a95e6fe..a679c40 100644\n --- a/test.c\n +++ b/test.c\n @@ -18,6 +18,13 @@ ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset)\n return total;\n }\n\n +int secure_foo(struct user *u)\n +{\n + foo(u);\n + if (!u->is_allowed_foo)\n + return;\n +}\n +\n int xdup(int fd)\n {\n int ret = dup(fd);\n\nIf the moved code is larger, it is easier to hide some permutation in the\ncode, which is why we would not want to color all lines as \"moved\" in this\ncase. So we do not just need to color lines differently that are added and\nremoved in the same diff, we need to tweak the algorithm a bit more.\n\nAs the reviewers attention should be brought to the places, where the\ndifference is introduced to the moved code, we cannot just have one new\ncolor for all of moved code.\n\nFirst I implemented an alternative design, which would show a moved hunk\nin one color, and its boundaries in another color. This idea was error\nprone as it inspected each line and its neighboring lines to determine\nif the line was (a) moved and (b) if was deep inside a hunk by having\nmatching neighboring lines. This is unreliable as the we can construct\nhunks which have equal neighbors that just exceed the number of lines\ninspected. See one of the tests as an example.\n\nInstead this provides a dynamic programming greedy algorithm that finds\nthe largest moved hunk and then switches color to the alternative color\nfor the next hunk. By doing this any permutation is recognized and\ndisplayed. That implies that there is no dedicated boundary or\ninside-hunk color, but instead we'll have just two colors alternating\nfor hunks.\n\nIt would be a bit more UX friendly if the two corresponding hunks\n(of added and deleted lines) for one move would get the same color id.\n(Both get \"regular moved\" or \"alternative moved\"). This problem is\ndeferred to a later patch for now.\n\nAlgorithm-by: Jonathan Tan \nSigned-off-by: Stefan Beller \n---\n Documentation/config.txt | 12 +-\n diff.c | 265 ++++++++++++++++++++++++++++++++++++++++++---\n diff.h | 21 +++-\n t/t4015-diff-whitespace.sh | 229 +++++++++++++++++++++++++++++++++++++++\n 4 files changed, 506 insertions(+), 21 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 475e874d51..90403c06e3 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1051,14 +1051,22 @@ This does not affect linkgit:git-format-patch[1] or the\n 'git-diff-{asterisk}' plumbing commands. Can be overridden on the\n command line with the `--color[=]` option.\n \n+color.moved::\n+\tA boolean value, whether a diff should color moved lines\n+\tdifferently. The moved lines are searched for in the diff only.\n+\tDuplicated lines from somewhere in the project that are not\n+\tpart of the diff are not colored as moved.\n+\tDefaults to false.\n+\n color.diff.::\n \tUse customized color for diff colorization. `` specifies\n \twhich part of the patch to use the specified color, and is one\n \tof `context` (context text - `plain` is a historical synonym),\n \t`meta` (metainformation), `frag`\n \t(hunk header), 'func' (function in hunk header), `old` (removed lines),\n-\t`new` (added lines), `commit` (commit headers), or `whitespace`\n-\t(highlighting whitespace errors).\n+\t`new` (added lines), `commit` (commit headers), `whitespace`\n+\t(highlighting whitespace errors), `movedFrom` (removed lines that\n+\treappear), `movedTo` (added lines that were removed elsewhere).\n \n color.decorate.::\n \tUse customized color for 'git log --decorate' output. `` is one\ndiff --git a/diff.c b/diff.c\nindex dbab7fb44e..6372e0eb25 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -31,6 +31,7 @@ static int diff_indent_heuristic; /* experimental */\n static int diff_rename_limit_default = 400;\n static int diff_suppress_blank_empty;\n static int diff_use_color_default = -1;\n+static int diff_color_moved_default;\n static int diff_context_default = 3;\n static int diff_interhunk_context_default;\n static const char *diff_word_regex_cfg;\n@@ -55,6 +56,10 @@ static char diff_colors[][COLOR_MAXLEN] = {\n \tGIT_COLOR_YELLOW,\t/* COMMIT */\n \tGIT_COLOR_BG_RED,\t/* WHITESPACE */\n \tGIT_COLOR_NORMAL,\t/* FUNCINFO */\n+\tGIT_COLOR_BOLD_RED,\t/* OLD_MOVED_A */\n+\tGIT_COLOR_BG_RED,\t/* OLD_MOVED_B */\n+\tGIT_COLOR_BOLD_GREEN,\t/* NEW_MOVED_A */\n+\tGIT_COLOR_BG_GREEN,\t/* NEW_MOVED_B */\n };\n \n static NORETURN void die_want_option(const char *option_name)\n@@ -80,6 +85,14 @@ static int parse_diff_color_slot(const char *var)\n \t\treturn DIFF_WHITESPACE;\n \tif (!strcasecmp(var, \"func\"))\n \t\treturn DIFF_FUNCINFO;\n+\tif (!strcasecmp(var, \"oldmoved\"))\n+\t\treturn DIFF_FILE_OLD_MOVED;\n+\tif (!strcasecmp(var, \"oldmovedalternative\"))\n+\t\treturn DIFF_FILE_OLD_MOVED_ALT;\n+\tif (!strcasecmp(var, \"newmoved\"))\n+\t\treturn DIFF_FILE_NEW_MOVED;\n+\tif (!strcasecmp(var, \"newmovedalternative\"))\n+\t\treturn DIFF_FILE_NEW_MOVED_ALT;\n \treturn -1;\n }\n \n@@ -234,6 +247,10 @@ int git_diff_ui_config(const char *var, const char *value, void *cb)\n \t\tdiff_use_color_default = git_config_colorbool(var, value);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(var, \"color.moved\")) {\n+\t\tdiff_color_moved_default = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(var, \"diff.context\")) {\n \t\tdiff_context_default = git_config_int(var, value);\n \t\tif (diff_context_default < 0)\n@@ -354,6 +371,81 @@ int git_diff_basic_config(const char *var, const char *value, void *cb)\n \treturn git_default_config(var, value, cb);\n }\n \n+struct moved_entry {\n+\tstruct hashmap_entry ent;\n+\tconst struct buffered_patch_line *line;\n+\tstruct moved_entry *next_line;\n+};\n+\n+static void get_ws_cleaned_string(const struct buffered_patch_line *l,\n+\t\t\t\t struct strbuf *out)\n+{\n+\tint i;\n+\tfor (i = 0; i < l->len; i++) {\n+\t\tif (isspace(l->line[i]))\n+\t\t\tcontinue;\n+\t\tstrbuf_addch(out, l->line[i]);\n+\t}\n+}\n+\n+static int buffered_patch_line_cmp_no_ws(const struct buffered_patch_line *a,\n+\t\t\t\t\t const struct buffered_patch_line *b,\n+\t\t\t\t\t const void *keydata)\n+{\n+\tstruct strbuf sba = STRBUF_INIT;\n+\tstruct strbuf sbb = STRBUF_INIT;\n+\tget_ws_cleaned_string(a, &sba);\n+\tget_ws_cleaned_string(b, &sbb);\n+\treturn sba.len != sbb.len || strncmp(sba.buf, sbb.buf, sba.len);\n+}\n+\n+static int buffered_patch_line_cmp(const struct buffered_patch_line *a,\n+\t\t\t\t const struct buffered_patch_line *b,\n+\t\t\t\t const void *keydata)\n+{\n+\treturn a->len != b->len || strncmp(a->line, b->line, a->len);\n+}\n+\n+static int moved_entry_cmp(const struct moved_entry *a,\n+\t\t\t const struct moved_entry *b,\n+\t\t\t const void *keydata)\n+{\n+\treturn buffered_patch_line_cmp(a->line, b->line, keydata);\n+}\n+\n+static int moved_entry_cmp_no_ws(const struct moved_entry *a,\n+\t\t\t\t const struct moved_entry *b,\n+\t\t\t\t const void *keydata)\n+{\n+\treturn buffered_patch_line_cmp_no_ws(a->line, b->line, keydata);\n+}\n+\n+static unsigned get_line_hash(struct buffered_patch_line *line, unsigned ignore_ws)\n+{\n+\tstatic struct strbuf sb = STRBUF_INIT;\n+\n+\tif (ignore_ws) {\n+\t\tstrbuf_reset(&sb);\n+\t\tget_ws_cleaned_string(line, &sb);\n+\t\treturn memhash(sb.buf, sb.len);\n+\t} else\n+\t\treturn memhash(line->line, line->len);\n+}\n+\n+static struct moved_entry *prepare_entry(struct diff_options *o,\n+\t\t\t\t\t int line_no)\n+{\n+\tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n+\tunsigned ignore_ws = DIFF_XDL_TST(o, IGNORE_WHITESPACE);\n+\tstruct buffered_patch_line *l = &o->line_buffer[line_no];\n+\n+\tret->ent.hash = get_line_hash(l, ignore_ws);\n+\tret->line = l;\n+\tret->next_line = NULL;\n+\n+\treturn ret;\n+}\n+\n static char *quote_two(const char *one, const char *two)\n {\n \tint need_one = quote_c_style(one, NULL, NULL, 1);\n@@ -516,8 +608,98 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n+static void mark_color_as_moved(struct diff_options *o, int line_no)\n+{\n+\tstruct hashmap *hm = NULL;\n+\tstruct moved_entry *key = prepare_entry(o, line_no);\n+\tstruct moved_entry *match = NULL;\n+\tstruct buffered_patch_line *l = &o->line_buffer[line_no];\n+\tint alt_flag;\n+\tint i, lp, rp;\n+\n+\tswitch (l->sign) {\n+\tcase '+':\n+\t\thm = o->deleted_lines;\n+\t\tbreak;\n+\tcase '-':\n+\t\thm = o->added_lines;\n+\t\tbreak;\n+\tdefault:\n+\t\t/* reset to standard, on-alt move color */\n+\t\to->color_moved = 1;\n+\t\tbreak;\n+\t}\n+\n+\t/* Check for any match to color it as a move. */\n+\tif (!hm)\n+\t\treturn;\n+\tmatch = hashmap_get(hm, key, o);\n+\tfree(key);\n+\tif (!match)\n+\t\treturn;\n+\n+\t/* Check any potential block runs, advance each or nullify */\n+\tfor (i = 0; i < o->pmb_nr; i++) {\n+\t\tstruct moved_entry *p = o->pmb[i];\n+\t\tif (p && p->next_line &&\n+\t\t !buffered_patch_line_cmp(p->next_line->line, l, o)) {\n+\t\t\to->pmb[i] = p->next_line;\n+\t\t} else {\n+\t\t\to->pmb[i] = NULL;\n+\t\t}\n+\t}\n+\n+\t/* Shrink the set to the remaining runs */\n+\tfor (lp = 0, rp = o->pmb_nr - 1; lp <= rp;) {\n+\t\twhile (lp < o->pmb_nr && o->pmb[lp])\n+\t\t\tlp ++;\n+\t\t/* lp points at the first NULL now */\n+\n+\t\twhile (rp > -1 && !o->pmb[rp])\n+\t\t\trp--;\n+\t\t/* rp points at the last non-NULL */\n+\n+\t\tif (lp < o->pmb_nr && rp > -1 && lp < rp) {\n+\t\t\to->pmb[lp] = o->pmb[rp];\n+\t\t\to->pmb[rp] = NULL;\n+\t\t\trp--;\n+\t\t\tlp++;\n+\t\t}\n+\t}\n+\n+\tif (rp > -1) {\n+\t\t/* Remember the number of running sets */\n+\t\to->pmb_nr = rp + 1;\n+\t} else {\n+\t\t/* Toggle color */\n+\t\to->color_moved = o->color_moved == 2 ? 1 : 2;\n+\n+\t\t/* Build up a new set */\n+\t\ti = 0;\n+\t\tfor (; match; match = hashmap_get_next(hm, match)) {\n+\t\t\tALLOC_GROW(o->pmb, i + 1, o->pmb_alloc);\n+\t\t\to->pmb[i] = match;\n+\t\t\ti++;\n+\t\t}\n+\t\to->pmb_nr = i;\n+\t}\n+\n+\talt_flag = o->color_moved - 1;\n+\tswitch (l->sign) {\n+\tcase '+':\n+\t\tl->set = diff_get_color_opt(o, DIFF_FILE_NEW_MOVED + alt_flag);\n+\t\tbreak;\n+\tcase '-':\n+\t\tl->set = diff_get_color_opt(o, DIFF_FILE_OLD_MOVED + alt_flag);\n+\t\tbreak;\n+\tdefault:\n+\t\t; /* nothing */\n+\t}\n+}\n+\n static void emit_buffered_patch_line(struct diff_options *o,\n-\t\t\t\t struct buffered_patch_line *e)\n+\t\t\t\t struct buffered_patch_line *e,\n+\t\t\t\t int pass)\n {\n \tint has_trailing_newline, has_trailing_carriage_return, len = e->len;\n \tFILE *file = o->file;\n@@ -548,11 +730,11 @@ static void emit_buffered_patch_line(struct diff_options *o,\n \n static void emit_buffered_patch_line_ws(struct diff_options *o,\n \t\t\t\t\tstruct buffered_patch_line *e,\n-\t\t\t\t\tconst char *ws, unsigned ws_rule)\n+\t\t\t\t\tconst char *ws, unsigned ws_rule,\n+\t\t\t\t\tint pass)\n {\n \tstruct buffered_patch_line s = {e->set, e->reset, \"\", 0, e->sign};\n-\n-\temit_buffered_patch_line(o, &s);\n+\temit_buffered_patch_line(o, &s, 0);\n \tws_check_emit(e->line, e->len, ws_rule,\n \t\t o->file, e->set, e->reset, ws);\n }\n@@ -564,12 +746,14 @@ static void process_next_buffered_patch_line(struct diff_options *o, int line_no\n \tconst char *ws = o->current_filepair->ws;\n \tunsigned ws_rule = o->current_filepair->ws_rule;\n \n+\tmark_color_as_moved(o, line_no);\n+\n \tswitch (e->state) {\n \t\tcase BPL_EMIT_LINE_ASIS:\n-\t\t\temit_buffered_patch_line(o, e);\n+\t\t\temit_buffered_patch_line(o, e, 1);\n \t\t\tbreak;\n \t\tcase BPL_EMIT_LINE_WS:\n-\t\t\temit_buffered_patch_line_ws(o, e, ws, ws_rule);\n+\t\t\temit_buffered_patch_line_ws(o, e, ws, ws_rule, 1);\n \t\t\tbreak;\n \t\tcase BPL_HANDOVER:\n \t\t\to->current_filepair++;\n@@ -602,7 +786,7 @@ static void emit_line_0(struct diff_options *o,\n \tif (o->use_buffer)\n \t\tappend_buffered_patch_line(o, &e);\n \telse\n-\t\temit_buffered_patch_line(o, &e);\n+\t\temit_buffered_patch_line(o, &e, 0);\n }\n \n void emit_line(struct diff_options *o, const char *set, const char *reset,\n@@ -621,7 +805,7 @@ static void emit_line_ws(struct diff_options *o,\n \tif (o->use_buffer)\n \t\tappend_buffered_patch_line(o, &e);\n \telse\n-\t\temit_buffered_patch_line_ws(o, &e, ws, ws_rule);\n+\t\temit_buffered_patch_line_ws(o, &e, ws, ws_rule, 0);\n }\n \n void emit_line_fmt(struct diff_options *o,\n@@ -676,6 +860,36 @@ static void emit_line_checked(const char *reset,\n \t\t\t ws, ecbdata->ws_rule);\n }\n \n+static void add_line_to_move_detection(struct diff_options *o, int line_idx)\n+{\n+\tint sign = 0;\n+\tstruct hashmap *hm;\n+\tstruct moved_entry *key;\n+\n+\tswitch (o->line_buffer[line_idx].sign) {\n+\tcase '+':\n+\t\tsign = '+';\n+\t\thm = o->added_lines;\n+\t\tbreak;\n+\tcase '-':\n+\t\tsign = '-';\n+\t\thm = o->deleted_lines;\n+\t\tbreak;\n+\tcase ' ':\n+\tdefault:\n+\t\to->prev_line = NULL;\n+\t\treturn;\n+\t}\n+\n+\tkey = prepare_entry(o, line_idx);\n+\tif (o->prev_line &&\n+\t o->prev_line->line->sign == sign)\n+\t\to->prev_line->next_line = key;\n+\n+\thashmap_add(hm, key);\n+\to->prev_line = key;\n+}\n+\n static void emit_add_line(const char *reset,\n \t\t\t struct emit_callback *ecbdata,\n \t\t\t const char *line, int len)\n@@ -3649,6 +3863,9 @@ void diff_setup_done(struct diff_options *options)\n \n \tif (DIFF_OPT_TST(options, FOLLOW_RENAMES) && options->pathspec.nr != 1)\n \t\tdie(_(\"--follow requires exactly one pathspec\"));\n+\n+\tif (!options->use_color || external_diff())\n+\t\toptions->color_moved = 0;\n }\n \n static int opt_arg(const char *arg, int arg_short, const char *arg_long, int *val)\n@@ -4073,6 +4290,10 @@ int diff_opt_parse(struct diff_options *options,\n \t}\n \telse if (!strcmp(arg, \"--no-color\"))\n \t\toptions->use_color = 0;\n+\telse if (!strcmp(arg, \"--color-moved\"))\n+\t\toptions->color_moved = 1;\n+\telse if (!strcmp(arg, \"--no-color-moved\"))\n+\t\toptions->color_moved = 0;\n \telse if (!strcmp(arg, \"--color-words\")) {\n \t\toptions->use_color = 1;\n \t\toptions->word_diff = DIFF_WORDS_COLOR;\n@@ -4878,16 +5099,19 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n {\n \tint i;\n \tstruct diff_queue_struct *q = &diff_queued_diff;\n-\t/*\n-\t * For testing purposes we want to make sure the diff machinery\n-\t * works completely with the buffer. If there is anything emitted\n-\t * outside the emit_buffered_patch_line, then the order is screwed\n-\t * up and the tests will fail.\n-\t *\n-\t * TODO (later in this series):\n-\t * We'll unset this flag in a later patch.\n-\t */\n-\to->use_buffer = 1;\n+\n+\tif (o->color_moved) {\n+\t\tunsigned ignore_ws = DIFF_XDL_TST(o, IGNORE_WHITESPACE);\n+\t\to->use_buffer = 1;\n+\t\to->deleted_lines = xmallocz(sizeof(*o->deleted_lines));\n+\t\to->added_lines = xmallocz(sizeof(*o->added_lines));\n+\t\thashmap_init(o->deleted_lines, ignore_ws ?\n+\t\t\t(hashmap_cmp_fn)moved_entry_cmp_no_ws :\n+\t\t\t(hashmap_cmp_fn)moved_entry_cmp, 0);\n+\t\thashmap_init(o->added_lines, ignore_ws ?\n+\t\t\t(hashmap_cmp_fn)moved_entry_cmp_no_ws :\n+\t\t\t(hashmap_cmp_fn)moved_entry_cmp, 0);\n+\t}\n \n \tif (o->use_buffer) {\n \t\tALLOC_GROW(o->filepair_buffer,\n@@ -4902,6 +5126,10 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \t}\n \n \tif (o->use_buffer) {\n+\t\to->current_filepair = &o->filepair_buffer[0];\n+\t\tfor (i = 0; i < o->line_buffer_nr; i++)\n+\t\t\tadd_line_to_move_detection(o, i);\n+\n \t\to->current_filepair = &o->filepair_buffer[0];\n \t\tfor (i = 0; i < o->line_buffer_nr; i++)\n \t\t\tprocess_next_buffered_patch_line(o, i);\n@@ -4992,6 +5220,7 @@ void diff_flush(struct diff_options *options)\n \t\tif (!options->file)\n \t\t\tdie_errno(\"Could not open /dev/null\");\n \t\toptions->close_file = 1;\n+\t\toptions->color_moved = 0;\n \t\tfor (i = 0; i < q->nr; i++) {\n \t\t\tstruct diff_filepair *p = q->queue[i];\n \t\t\tif (check_pair_status(p))\ndiff --git a/diff.h b/diff.h\nindex c334aac02e..b83d6fefcc 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -7,6 +7,7 @@\n #include \"tree-walk.h\"\n #include \"pathspec.h\"\n #include \"object.h\"\n+#include \"hashmap.h\"\n \n struct rev_info;\n struct diff_options;\n@@ -145,6 +146,8 @@ struct buffered_filepair {\n \tunsigned ws_rule;\n };\n \n+struct moved_entry;\n+\n struct diff_options {\n \tconst char *orderfile;\n \tconst char *pickaxe;\n@@ -217,6 +220,8 @@ struct diff_options {\n \n \tint diff_path_counter;\n \n+\t/* Determines color moved code. Flipped between 1, 2 for alt. color. */\n+\tint color_moved;\n \tint use_buffer;\n \n \tstruct buffered_patch_line *line_buffer;\n@@ -225,6 +230,16 @@ struct diff_options {\n \tstruct buffered_filepair *filepair_buffer;\n \tint filepair_buffer_nr, filepair_buffer_alloc;\n \tstruct buffered_filepair *current_filepair;\n+\n+\t/* built up in the first pass: */\n+\tstruct hashmap *deleted_lines;\n+\tstruct hashmap *added_lines;\n+\t/* needed for building up */\n+\tstruct moved_entry *prev_line;\n+\n+\t/* state in the second pass */\n+\tstruct moved_entry **pmb; /* potentially moved blocks */\n+\tint pmb_nr, pmb_alloc;\n };\n \n void emit_line_fmt(struct diff_options *o, const char *set, const char *reset,\n@@ -241,7 +256,11 @@ enum color_diff {\n \tDIFF_FILE_NEW = 5,\n \tDIFF_COMMIT = 6,\n \tDIFF_WHITESPACE = 7,\n-\tDIFF_FUNCINFO = 8\n+\tDIFF_FUNCINFO = 8,\n+\tDIFF_FILE_OLD_MOVED = 9,\n+\tDIFF_FILE_OLD_MOVED_ALT = 10,\n+\tDIFF_FILE_NEW_MOVED = 11,\n+\tDIFF_FILE_NEW_MOVED_ALT = 12\n };\n const char *diff_get_color(int diff_use_color, enum color_diff ix);\n #define diff_get_color_opt(o, ix) \\\ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 289806d0c7..232d9ad55e 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -972,4 +972,233 @@ test_expect_success 'option overrides diff.wsErrorHighlight' '\n \n '\n \n+test_expect_success 'detect moved code, complete file' '\n+\tgit reset --hard &&\n+\tcat <<-\\EOF >test.c &&\n+\t#include\n+\tmain()\n+\t{\n+\tprintf(\"Hello World\");\n+\t}\n+\tEOF\n+\tgit add test.c &&\n+\tgit commit -m \"add main function\" &&\n+\tgit mv test.c main.c &&\n+\tgit diff HEAD --color-moved --no-renames | test_decode_color >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tdiff --git a/main.c b/main.c\n+\tnew file mode 100644\n+\tindex 0000000..a986c57\n+\t--- /dev/null\n+\t+++ b/main.c\n+\t@@ -0,0 +1,5 @@\n+\t+#include\n+\t+main()\n+\t+{\n+\t+printf(\"Hello World\");\n+\t+}\n+\tdiff --git a/test.c b/test.c\n+\tdeleted file mode 100644\n+\tindex a986c57..0000000\n+\t--- a/test.c\n+\t+++ /dev/null\n+\t@@ -1,5 +0,0 @@\n+\t-#include\n+\t-main()\n+\t-{\n+\t-printf(\"Hello World\");\n+\t-}\n+\tEOF\n+\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'detect moved code, inside file' '\n+\tgit reset --hard &&\n+\tcat <<-\\EOF >main.c &&\n+\t\t#include\n+\t\tint stuff()\n+\t\t{\n+\t\t\tprintf(\"Hello \");\n+\t\t\tprintf(\"World\\n\");\n+\t\t}\n+\n+\t\tint secure_foo(struct user *u)\n+\t\t{\n+\t\t\tif (!u->is_allowed_foo)\n+\t\t\t\treturn;\n+\t\t\tfoo(u);\n+\t\t}\n+\n+\t\tint main()\n+\t\t{\n+\t\t\tfoo();\n+\t\t}\n+\tEOF\n+\tcat <<-\\EOF >test.c &&\n+\t\t#include\n+\t\tint bar()\n+\t\t{\n+\t\t\tprintf(\"Hello World, but different\\n\");\n+\t\t}\n+\n+\t\tint another_function()\n+\t\t{\n+\t\t\tbar();\n+\t\t}\n+\tEOF\n+\tgit add main.c test.c &&\n+\tgit commit -m \"add main and test file\" &&\n+\tcat <<-\\EOF >main.c &&\n+\t\t#include\n+\t\tint stuff()\n+\t\t{\n+\t\t\tprintf(\"Hello \");\n+\t\t\tprintf(\"World\\n\");\n+\t\t}\n+\n+\t\tint main()\n+\t\t{\n+\t\t\tfoo();\n+\t\t}\n+\tEOF\n+\tcat <<-\\EOF >test.c &&\n+\t\t#include\n+\t\tint bar()\n+\t\t{\n+\t\t\tprintf(\"Hello World, but different\\n\");\n+\t\t}\n+\n+\t\tint secure_foo(struct user *u)\n+\t\t{\n+\t\t\tif (!u->is_allowed_foo)\n+\t\t\t\treturn;\n+\t\t\tfoo(u);\n+\t\t}\n+\n+\t\tint another_function()\n+\t\t{\n+\t\t\tbar();\n+\t\t}\n+\tEOF\n+\tgit diff HEAD --no-renames --color-moved| test_decode_color >actual &&\n+\tcat <<-\\EOF >expected &&\n+\tdiff --git a/main.c b/main.c\n+\tindex 27a619c..7cf9336 100644\n+\t--- a/main.c\n+\t+++ b/main.c\n+\t@@ -5,13 +5,6 @@ printf(\"Hello \");\n+\t printf(\"World\\n\");\n+\t }\n+\t \n+\t-int secure_foo(struct user *u)\n+\t-{\n+\t-if (!u->is_allowed_foo)\n+\t-return;\n+\t-foo(u);\n+\t-}\n+\t-\n+\t int main()\n+\t {\n+\t foo();\n+\tdiff --git a/test.c b/test.c\n+\tindex 1dc1d85..e34eb69 100644\n+\t--- a/test.c\n+\t+++ b/test.c\n+\t@@ -4,6 +4,13 @@ int bar()\n+\t printf(\"Hello World, but different\\n\");\n+\t }\n+\t \n+\t+int secure_foo(struct user *u)\n+\t+{\n+\t+if (!u->is_allowed_foo)\n+\t+return;\n+\t+foo(u);\n+\t+}\n+\t+\n+\t int another_function()\n+\t {\n+\t bar();\n+\tEOF\n+\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'detect permutations inside moved code, ' '\n+\t# reusing the move example from last test:\n+\tcat <<-\\EOF >main.c &&\n+\t\t#include\n+\t\tint stuff()\n+\t\t{\n+\t\t\tprintf(\"Hello \");\n+\t\t\tprintf(\"World\\n\");\n+\t\t}\n+\n+\t\tint main()\n+\t\t{\n+\t\t\tfoo();\n+\t\t}\n+\tEOF\n+\tcat <<-\\EOF >test.c &&\n+\t\t#include\n+\t\tint bar()\n+\t\t{\n+\t\t\tprintf(\"Hello World, but different\\n\");\n+\t\t}\n+\n+\t\tint secure_foo(struct user *u)\n+\t\t{\n+\t\t\tfoo(u);\n+\t\t\tif (!u->is_allowed_foo)\n+\t\t\t\treturn;\n+\t\t}\n+\n+\t\tint another_function()\n+\t\t{\n+\t\t\tbar();\n+\t\t}\n+\tEOF\n+\tgit diff HEAD --no-renames --color-moved| test_decode_color >actual &&\n+\tcat <<-\\EOF >expected &&\n+\tdiff --git a/main.c b/main.c\n+\tindex 27a619c..7cf9336 100644\n+\t--- a/main.c\n+\t+++ b/main.c\n+\t@@ -5,13 +5,6 @@ printf(\"Hello \");\n+\t printf(\"World\\n\");\n+\t }\n+\t \n+\t-int secure_foo(struct user *u)\n+\t-{\n+\t-if (!u->is_allowed_foo)\n+\t-return;\n+\t-foo(u);\n+\t-}\n+\t-\n+\t int main()\n+\t {\n+\t foo();\n+\tdiff --git a/test.c b/test.c\n+\tindex 1dc1d85..2bedec9 100644\n+\t--- a/test.c\n+\t+++ b/test.c\n+\t@@ -4,6 +4,13 @@ int bar()\n+\t printf(\"Hello World, but different\\n\");\n+\t }\n+\t \n+\t+int secure_foo(struct user *u)\n+\t+{\n+\t+foo(u);\n+\t+if (!u->is_allowed_foo)\n+\t+return;\n+\t+}\n+\t+\n+\t int another_function()\n+\t {\n+\t bar();\n+\tEOF\n+\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319712","messageId":"20170514040117.25865-13-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 12/19] diff.c: convert emit_binary_diff_body to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:10Z","receivedAt":"2017-05-14T04:02:08Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers emit_binary_diff_body.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 39 ++++++++++++++++++++++-----------------\n 1 file changed, 22 insertions(+), 17 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 45ec311828..ffdb728810 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -2233,8 +2233,8 @@ static unsigned char *deflate_it(char *data,\n \treturn deflated;\n }\n \n-static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n-\t\t\t\t const char *prefix)\n+static void emit_binary_diff_body(struct diff_options *o,\n+\t\t\t\t mmfile_t *one, mmfile_t *two)\n {\n \tvoid *cp;\n \tvoid *delta;\n@@ -2263,13 +2263,12 @@ static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n \t}\n \n \tif (delta && delta_size < deflate_size) {\n-\t\tfprintf(file, \"%sdelta %lu\\n\", prefix, orig_size);\n+\t\temit_line_fmt(o, NULL, NULL, \"delta %lu\\n\", orig_size);\n \t\tfree(deflated);\n \t\tdata = delta;\n \t\tdata_size = delta_size;\n-\t}\n-\telse {\n-\t\tfprintf(file, \"%sliteral %lu\\n\", prefix, two->size);\n+\t} else {\n+\t\temit_line_fmt(o, NULL, NULL, \"literal %lu\\n\", two->size);\n \t\tfree(delta);\n \t\tdata = deflated;\n \t\tdata_size = deflate_size;\n@@ -2278,8 +2277,9 @@ static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n \t/* emit data encoded in base85 */\n \tcp = data;\n \twhile (data_size) {\n+\t\tint len;\n \t\tint bytes = (52 < data_size) ? 52 : data_size;\n-\t\tchar line[70];\n+\t\tchar line[71];\n \t\tdata_size -= bytes;\n \t\tif (bytes <= 26)\n \t\t\tline[0] = bytes + 'A' - 1;\n@@ -2287,20 +2287,25 @@ static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n \t\t\tline[0] = bytes - 26 + 'a' - 1;\n \t\tencode_85(line + 1, cp, bytes);\n \t\tcp = (char *) cp + bytes;\n-\t\tfprintf(file, \"%s\", prefix);\n-\t\tfputs(line, file);\n-\t\tfputc('\\n', file);\n+\n+\t\tlen = strlen(line);\n+\t\tline[len++] = '\\n';\n+\t\tline[len] = '\\0';\n+\n+\t\temit_line(o, NULL, NULL, line, len);\n \t}\n-\tfprintf(file, \"%s\\n\", prefix);\n+\temit_line(o, NULL, NULL, \"\\n\", 1);\n \tfree(data);\n }\n \n-static void emit_binary_diff(FILE *file, mmfile_t *one, mmfile_t *two,\n-\t\t\t const char *prefix)\n+static void emit_binary_diff(struct diff_options *o,\n+\t\t\t mmfile_t *one, mmfile_t *two)\n {\n-\tfprintf(file, \"%sGIT binary patch\\n\", prefix);\n-\temit_binary_diff_body(file, one, two, prefix);\n-\temit_binary_diff_body(file, two, one, prefix);\n+\tconst char *s = \"GIT binary patch\\n\";\n+\tconst int len = strlen(s);\n+\temit_line(o, NULL, NULL, s, len);\n+\temit_binary_diff_body(o, one, two);\n+\temit_binary_diff_body(o, two, one);\n }\n \n int diff_filespec_is_binary(struct diff_filespec *one)\n@@ -2489,7 +2494,7 @@ static void builtin_diff(const char *name_a,\n \t\t\t\t header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tif (DIFF_OPT_TST(o, BINARY))\n-\t\t\temit_binary_diff(o->file, &mf1, &mf2, line_prefix);\n+\t\t\temit_binary_diff(o, &mf1, &mf2);\n \t\telse\n \t\t\temit_line_fmt(o, NULL, NULL,\n \t\t\t\t \"Binary files %s and %s differ\\n\",\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319713","messageId":"20170514040117.25865-17-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 16/19] diff.c: convert diff_summary to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:14Z","receivedAt":"2017-05-14T04:02:10Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers diff_summary.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 62 +++++++++++++++++++++++++++++++++-----------------------------\n 1 file changed, 33 insertions(+), 29 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 386b28cf47..899dc69dff 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4504,67 +4504,71 @@ static void flush_one_pair(struct diff_filepair *p, struct diff_options *opt)\n \t}\n }\n \n-static void show_file_mode_name(FILE *file, const char *newdelete, struct diff_filespec *fs)\n+static void show_file_mode_name(struct diff_options *opt, const char *newdelete, struct diff_filespec *fs)\n {\n+\tstruct strbuf sb = STRBUF_INIT;\n \tif (fs->mode)\n-\t\tfprintf(file, \" %s mode %06o \", newdelete, fs->mode);\n+\t\tstrbuf_addf(&sb, \" %s mode %06o \", newdelete, fs->mode);\n \telse\n-\t\tfprintf(file, \" %s \", newdelete);\n-\twrite_name_quoted(fs->path, file, '\\n');\n+\t\tstrbuf_addf(&sb, \" %s \", newdelete);\n+\n+\tquote_c_style(fs->path, &sb, NULL, 0);\n+\tstrbuf_addch(&sb, '\\n');\n+\temit_line(opt, NULL, NULL, sb.buf, sb.len);\n+\tstrbuf_release(&sb);\n }\n \n \n-static void show_mode_change(FILE *file, struct diff_filepair *p, int show_name,\n-\t\tconst char *line_prefix)\n+static void show_mode_change(struct diff_options *opt, struct diff_filepair *p,\n+\t\tint show_name)\n {\n \tif (p->one->mode && p->two->mode && p->one->mode != p->two->mode) {\n-\t\tfprintf(file, \"%s mode change %06o => %06o%c\", line_prefix, p->one->mode,\n-\t\t\tp->two->mode, show_name ? ' ' : '\\n');\n+\t\tstruct strbuf sb = STRBUF_INIT;\n \t\tif (show_name) {\n-\t\t\twrite_name_quoted(p->two->path, file, '\\n');\n+\t\t\tstrbuf_addch(&sb, ' ');\n+\t\t\tquote_c_style(p->two->path, &sb, NULL, 0);\n \t\t}\n+\t\temit_line_fmt(opt, NULL, NULL,\n+\t\t\t \" mode change %06o => %06o%s\\n\",\n+\t\t\t p->one->mode, p->two->mode,\n+\t\t\t show_name ? sb.buf : \"\");\n+\t\tstrbuf_release(&sb);\n \t}\n }\n \n-static void show_rename_copy(FILE *file, const char *renamecopy, struct diff_filepair *p,\n-\t\t\tconst char *line_prefix)\n+static void show_rename_copy(struct diff_options *opt, const char *renamecopy,\n+\t\tstruct diff_filepair *p)\n {\n \tchar *names = pprint_rename(p->one->path, p->two->path);\n-\n-\tfprintf(file, \" %s %s (%d%%)\\n\", renamecopy, names, similarity_index(p));\n+\temit_line_fmt(opt, NULL, NULL, \" %s %s (%d%%)\\n\", renamecopy, names, similarity_index(p));\n \tfree(names);\n-\tshow_mode_change(file, p, 0, line_prefix);\n+\tshow_mode_change(opt, p, 0);\n }\n \n static void diff_summary(struct diff_options *opt, struct diff_filepair *p)\n {\n-\tFILE *file = opt->file;\n-\tconst char *line_prefix = diff_line_prefix(opt);\n-\n \tswitch(p->status) {\n \tcase DIFF_STATUS_DELETED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_file_mode_name(file, \"delete\", p->one);\n+\t\tshow_file_mode_name(opt, \"delete\", p->one);\n \t\tbreak;\n \tcase DIFF_STATUS_ADDED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_file_mode_name(file, \"create\", p->two);\n+\t\tshow_file_mode_name(opt, \"create\", p->two);\n \t\tbreak;\n \tcase DIFF_STATUS_COPIED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_rename_copy(file, \"copy\", p, line_prefix);\n+\t\tshow_rename_copy(opt, \"copy\", p);\n \t\tbreak;\n \tcase DIFF_STATUS_RENAMED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_rename_copy(file, \"rename\", p, line_prefix);\n+\t\tshow_rename_copy(opt, \"rename\", p);\n \t\tbreak;\n \tdefault:\n \t\tif (p->score) {\n-\t\t\tfprintf(file, \"%s rewrite \", line_prefix);\n-\t\t\twrite_name_quoted(p->two->path, file, ' ');\n-\t\t\tfprintf(file, \"(%d%%)\\n\", similarity_index(p));\n+\t\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\t\tstrbuf_addstr(&sb, \" rewrite \");\n+\t\t\tquote_c_style(p->two->path, &sb, NULL, 0);\n+\t\t\tstrbuf_addf(&sb, \" (%d%%)\\n\", similarity_index(p));\n+\t\t\temit_line(opt, NULL, NULL, sb.buf, sb.len);\n \t\t}\n-\t\tshow_mode_change(file, p, !p->score, line_prefix);\n+\t\tshow_mode_change(opt, p, !p->score);\n \t\tbreak;\n \t}\n }\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319714","messageId":"20170514040117.25865-14-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 13/19] diff.c: convert show_stats to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:11Z","receivedAt":"2017-05-14T04:02:18Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nWe call print_stat_summary from builtin/apply, so we still\nneed the version with a file pointer, so introduce\nprint_stat_summary_0 that uses emit_line_* machinery and\nkeep print_stat_summary with the same arguments around.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 89 ++++++++++++++++++++++++++++++++++++++----------------------------\n diff.h | 4 +--\n 2 files changed, 53 insertions(+), 40 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex ffdb728810..91dc045a45 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1529,20 +1529,19 @@ static int scale_linear(int it, int width, int max_change)\n \treturn 1 + (it * (width - 1) / max_change);\n }\n \n-static void show_name(FILE *file,\n+static void show_name(struct strbuf *out,\n \t\t const char *prefix, const char *name, int len)\n {\n-\tfprintf(file, \" %s%-*s |\", prefix, len, name);\n+\tstrbuf_addf(out, \" %s%-*s |\", prefix, len, name);\n }\n \n-static void show_graph(FILE *file, char ch, int cnt, const char *set, const char *reset)\n+static void show_graph(struct strbuf *out, char ch, int cnt, const char *set, const char *reset)\n {\n \tif (cnt <= 0)\n \t\treturn;\n-\tfprintf(file, \"%s\", set);\n-\twhile (cnt--)\n-\t\tputc(ch, file);\n-\tfprintf(file, \"%s\", reset);\n+\tstrbuf_addstr(out, set);\n+\tstrbuf_addchars(out, ch, cnt);\n+\tstrbuf_addstr(out, reset);\n }\n \n static void fill_print_name(struct diffstat_file *file)\n@@ -1566,14 +1565,16 @@ static void fill_print_name(struct diffstat_file *file)\n \tfile->print_name = pname;\n }\n \n-int print_stat_summary(FILE *fp, int files, int insertions, int deletions)\n+void print_stat_summary_0(struct diff_options *options, int files,\n+\t\t\t int insertions, int deletions)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n-\tint ret;\n \n \tif (!files) {\n \t\tassert(insertions == 0 && deletions == 0);\n-\t\treturn fprintf(fp, \"%s\\n\", \" 0 files changed\");\n+\t\tstrbuf_addstr(&sb, \" 0 files changed\");\n+\t\temit_line(options, NULL, NULL, sb.buf, sb.len);\n+\t\treturn;\n \t}\n \n \tstrbuf_addf(&sb,\n@@ -1600,9 +1601,17 @@ int print_stat_summary(FILE *fp, int files, int insertions, int deletions)\n \t\t\t deletions);\n \t}\n \tstrbuf_addch(&sb, '\\n');\n-\tret = fputs(sb.buf, fp);\n+\temit_line(options, NULL, NULL, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n-\treturn ret;\n+}\n+\n+void print_stat_summary(FILE *fp, int files,\n+\t\t\tint insertions, int deletions)\n+{\n+\tstruct diff_options o;\n+\tmemset(&o, 0, sizeof(o));\n+\to.file = fp;\n+\tprint_stat_summary_0(&o, files, insertions, deletions);\n }\n \n static void show_stats(struct diffstat_t *data, struct diff_options *options)\n@@ -1612,13 +1621,13 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \tint total_files = data->nr, count;\n \tint width, name_width, graph_width, number_width = 0, bin_width = 0;\n \tconst char *reset, *add_c, *del_c;\n-\tconst char *line_prefix = \"\";\n \tint extra_shown = 0;\n+\tconst char *line_prefix = diff_line_prefix(options);\n+\tstruct strbuf out = STRBUF_INIT;\n \n \tif (data->nr == 0)\n \t\treturn;\n \n-\tline_prefix = diff_line_prefix(options);\n \tcount = options->stat_count ? options->stat_count : data->nr;\n \n \treset = diff_get_color_opt(options, DIFF_RESET);\n@@ -1772,26 +1781,29 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t}\n \n \t\tif (file->is_binary) {\n-\t\t\tfprintf(options->file, \"%s\", line_prefix);\n-\t\t\tshow_name(options->file, prefix, name, len);\n-\t\t\tfprintf(options->file, \" %*s\", number_width, \"Bin\");\n+\t\t\tshow_name(&out, prefix, name, len);\n+\t\t\tstrbuf_addf(&out, \" %*s\", number_width, \"Bin\");\n \t\t\tif (!added && !deleted) {\n-\t\t\t\tputc('\\n', options->file);\n+\t\t\t\tstrbuf_addch(&out, '\\n');\n+\t\t\t\temit_line(options, NULL, NULL, out.buf, out.len);\n+\t\t\t\tstrbuf_reset(&out);\n \t\t\t\tcontinue;\n \t\t\t}\n-\t\t\tfprintf(options->file, \" %s%\"PRIuMAX\"%s\",\n+\t\t\tstrbuf_addf(&out, \" %s%\"PRIuMAX\"%s\",\n \t\t\t\tdel_c, deleted, reset);\n-\t\t\tfprintf(options->file, \" -> \");\n-\t\t\tfprintf(options->file, \"%s%\"PRIuMAX\"%s\",\n+\t\t\tstrbuf_addstr(&out, \" -> \");\n+\t\t\tstrbuf_addf(&out, \"%s%\"PRIuMAX\"%s\",\n \t\t\t\tadd_c, added, reset);\n-\t\t\tfprintf(options->file, \" bytes\");\n-\t\t\tfprintf(options->file, \"\\n\");\n+\t\t\tstrbuf_addstr(&out, \" bytes\\n\");\n+\t\t\temit_line(options, NULL, NULL, out.buf, out.len);\n+\t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n \t\telse if (file->is_unmerged) {\n-\t\t\tfprintf(options->file, \"%s\", line_prefix);\n-\t\t\tshow_name(options->file, prefix, name, len);\n-\t\t\tfprintf(options->file, \" Unmerged\\n\");\n+\t\t\tshow_name(&out, prefix, name, len);\n+\t\t\tstrbuf_addstr(&out, \" Unmerged\\n\");\n+\t\t\temit_line(options, NULL, NULL, out.buf, out.len);\n+\t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n \n@@ -1814,14 +1826,15 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t\t\tadd = total - del;\n \t\t\t}\n \t\t}\n-\t\tfprintf(options->file, \"%s\", line_prefix);\n-\t\tshow_name(options->file, prefix, name, len);\n-\t\tfprintf(options->file, \" %*\"PRIuMAX\"%s\",\n+\t\tshow_name(&out, prefix, name, len);\n+\t\tstrbuf_addf(&out, \" %*\"PRIuMAX\"%s\",\n \t\t\tnumber_width, added + deleted,\n \t\t\tadded + deleted ? \" \" : \"\");\n-\t\tshow_graph(options->file, '+', add, add_c, reset);\n-\t\tshow_graph(options->file, '-', del, del_c, reset);\n-\t\tfprintf(options->file, \"\\n\");\n+\t\tshow_graph(&out, '+', add, add_c, reset);\n+\t\tshow_graph(&out, '-', del, del_c, reset);\n+\t\tstrbuf_addch(&out, '\\n');\n+\t\temit_line(options, NULL, NULL, out.buf, out.len);\n+\t\tstrbuf_reset(&out);\n \t}\n \n \tfor (i = 0; i < data->nr; i++) {\n@@ -1842,11 +1855,12 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\tif (i < count)\n \t\t\tcontinue;\n \t\tif (!extra_shown)\n-\t\t\tfprintf(options->file, \"%s ...\\n\", line_prefix);\n+\t\t\temit_line(options, NULL, NULL,\n+\t\t\t\t \" ...\\n\", strlen(\" ...\\n\"));\n \t\textra_shown = 1;\n \t}\n-\tfprintf(options->file, \"%s\", line_prefix);\n-\tprint_stat_summary(options->file, total_files, adds, dels);\n+\n+\tprint_stat_summary_0(options, total_files, adds, dels);\n }\n \n static void show_shortstats(struct diffstat_t *data, struct diff_options *options)\n@@ -1858,7 +1872,7 @@ static void show_shortstats(struct diffstat_t *data, struct diff_options *option\n \n \tfor (i = 0; i < data->nr; i++) {\n \t\tint added = data->files[i]->added;\n-\t\tint deleted= data->files[i]->deleted;\n+\t\tint deleted = data->files[i]->deleted;\n \n \t\tif (data->files[i]->is_unmerged ||\n \t\t (!data->files[i]->is_interesting && (added + deleted == 0))) {\n@@ -1868,8 +1882,7 @@ static void show_shortstats(struct diffstat_t *data, struct diff_options *option\n \t\t\tdels += deleted;\n \t\t}\n \t}\n-\tfprintf(options->file, \"%s\", diff_line_prefix(options));\n-\tprint_stat_summary(options->file, total_files, adds, dels);\n+\tprint_stat_summary_0(options, total_files, adds, dels);\n }\n \n static void show_numstat(struct diffstat_t *data, struct diff_options *options)\ndiff --git a/diff.h b/diff.h\nindex addebd5a0f..5e89481769 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -394,8 +394,8 @@ extern int parse_rename_score(const char **cp_p);\n \n extern long parse_algorithm_value(const char *value);\n \n-extern int print_stat_summary(FILE *fp, int files,\n-\t\t\t int insertions, int deletions);\n+extern void print_stat_summary(FILE *fp, int files,\n+\t\t\t int insertions, int deletions);\n extern void setup_diff_pager(struct diff_options *);\n \n #endif /* DIFF_H */\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319715","messageId":"20170514040117.25865-10-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 09/19] diff.c: convert emit_rewrite_diff to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:07Z","receivedAt":"2017-05-14T04:02:20Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers emit_rewrite_diff.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 33 ++++++++++++++++++---------------\n 1 file changed, 18 insertions(+), 15 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 8e00010bf4..e4b46fee4f 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -708,17 +708,17 @@ static void remove_tempfile(void)\n \t}\n }\n \n-static void print_line_count(FILE *file, int count)\n+static void add_line_count(struct strbuf *out, int count)\n {\n \tswitch (count) {\n \tcase 0:\n-\t\tfprintf(file, \"0,0\");\n+\t\tstrbuf_addstr(out, \"0,0\");\n \t\tbreak;\n \tcase 1:\n-\t\tfprintf(file, \"1\");\n+\t\tstrbuf_addstr(out, \"1\");\n \t\tbreak;\n \tdefault:\n-\t\tfprintf(file, \"1,%d\", count);\n+\t\tstrbuf_addf(out, \"1,%d\", count);\n \t\tbreak;\n \t}\n }\n@@ -772,7 +772,7 @@ static void emit_rewrite_diff(const char *name_a,\n \tchar *data_one, *data_two;\n \tsize_t size_one, size_two;\n \tstruct emit_callback ecbdata;\n-\tconst char *line_prefix = diff_line_prefix(o);\n+\tstruct strbuf out = STRBUF_INIT;\n \n \tif (diff_mnemonic_prefix && DIFF_OPT_TST(o, REVERSE_DIFF)) {\n \t\ta_prefix = o->b_prefix;\n@@ -810,20 +810,23 @@ static void emit_rewrite_diff(const char *name_a,\n \tecbdata.lno_in_preimage = 1;\n \tecbdata.lno_in_postimage = 1;\n \n+\temit_line_fmt(o, metainfo, reset, \"--- %s%s\\n\", a_name.buf, name_a_tab);\n+\temit_line_fmt(o, metainfo, reset, \"+++ %s%s\\n\", b_name.buf, name_b_tab);\n+\n \tlc_a = count_lines(data_one, size_one);\n \tlc_b = count_lines(data_two, size_two);\n-\tfprintf(o->file,\n-\t\t\"%s%s--- %s%s%s\\n%s%s+++ %s%s%s\\n%s%s@@ -\",\n-\t\tline_prefix, metainfo, a_name.buf, name_a_tab, reset,\n-\t\tline_prefix, metainfo, b_name.buf, name_b_tab, reset,\n-\t\tline_prefix, fraginfo);\n+\n+\tstrbuf_addstr(&out, \"@@ -\");\n \tif (!o->irreversible_delete)\n-\t\tprint_line_count(o->file, lc_a);\n+\t\tadd_line_count(&out, lc_a);\n \telse\n-\t\tfprintf(o->file, \"?,?\");\n-\tfprintf(o->file, \" +\");\n-\tprint_line_count(o->file, lc_b);\n-\tfprintf(o->file, \" @@%s\\n\", reset);\n+\t\tstrbuf_addstr(&out, \"?,?\");\n+\tstrbuf_addstr(&out, \" +\");\n+\tadd_line_count(&out, lc_b);\n+\tstrbuf_addstr(&out, \" @@\\n\");\n+\temit_line(o, fraginfo, reset, out.buf, out.len);\n+\tstrbuf_release(&out);\n+\n \tif (lc_a && !o->irreversible_delete)\n \t\temit_rewrite_lines(&ecbdata, '-', data_one, size_one);\n \tif (lc_b)\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319716","messageId":"20170514040117.25865-16-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 15/19] diff.c: convert diff_flush to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:13Z","receivedAt":"2017-05-14T04:02:21Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers diff_flush.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/diff.c b/diff.c\nindex 07041a35ad..386b28cf47 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4873,7 +4873,9 @@ void diff_flush(struct diff_options *options)\n \t\t\t\t term, 1);\n \t\t\tif (options->stat_sep) {\n \t\t\t\t/* attach patch instead of inline */\n-\t\t\t\tfputs(options->stat_sep, options->file);\n+\t\t\t\temit_line(options, NULL, NULL,\n+\t\t\t\t\t options->stat_sep,\n+\t\t\t\t\t strlen(options->stat_sep));\n \t\t\t}\n \t\t}\n \n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319717","messageId":"20170514040117.25865-5-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 04/19] diff.c: factor out diff_flush_patch_all_file_pairs","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:02Z","receivedAt":"2017-05-14T04:02:22Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch we want to do more things before and after all filepairs\nare flushed. So factor flushing out all file pairs into its own function\nthat the new code can be plugged in easily.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 17 ++++++++++++-----\n 1 file changed, 12 insertions(+), 5 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 4269b8dccf..381b572d76 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4728,6 +4728,17 @@ void diff_warn_rename_limit(const char *varname, int needed, int degraded_cc)\n \t\twarning(_(rename_limit_advice), varname, needed);\n }\n \n+static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n+{\n+\tint i;\n+\tstruct diff_queue_struct *q = &diff_queued_diff;\n+\tfor (i = 0; i < q->nr; i++) {\n+\t\tstruct diff_filepair *p = q->queue[i];\n+\t\tif (check_pair_status(p))\n+\t\t\tdiff_flush_patch(p, o);\n+\t}\n+}\n+\n void diff_flush(struct diff_options *options)\n {\n \tstruct diff_queue_struct *q = &diff_queued_diff;\n@@ -4825,11 +4836,7 @@ void diff_flush(struct diff_options *options)\n \t\t\t}\n \t\t}\n \n-\t\tfor (i = 0; i < q->nr; i++) {\n-\t\t\tstruct diff_filepair *p = q->queue[i];\n-\t\t\tif (check_pair_status(p))\n-\t\t\t\tdiff_flush_patch(p, options);\n-\t\t}\n+\t\tdiff_flush_patch_all_file_pairs(options);\n \t}\n \n \tif (output_format & DIFF_FORMAT_CALLBACK)\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319718","messageId":"20170514040117.25865-6-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCH 05/19] diff.c: emit_line_0 can handle no color setting","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:01:03Z","receivedAt":"2017-05-14T04:02:23Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nIn later patches we may pass lines that are not colored to\nthe central function emit_line_0, so we\nneed to emit the color only when it is non-NULL.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 381b572d76..48f0fb98dc 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -532,11 +532,13 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n \t\tlen--;\n \n \tif (len || sign) {\n-\t\tfputs(set, file);\n+\t\tif (set)\n+\t\t\tfputs(set, file);\n \t\tif (sign)\n \t\t\tfputc(sign, file);\n \t\tfwrite(line, len, 1, file);\n-\t\tfputs(reset, file);\n+\t\tif (reset)\n+\t\t\tfputs(reset, file);\n \t}\n \tif (has_trailing_carriage_return)\n \t\tfputc('\\r', file);\n-- \n2.13.0.18.g183880de0a\n\n"},{"id":"319719","messageId":"20170514040559.uvs6c2dnd6aelrln@sigill.intra.peff.net","threadId":"45958","inReplyTo":"20170514040117.25865-19-sbeller@google.com","subject":"Re: [PATCH 18/19] diff: buffer all output if asked to","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-05-14T04:06:00Z","receivedAt":"2017-05-14T04:06:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, May 13, 2017 at 09:01:16PM -0700, Stefan Beller wrote:\n\n> +\t\tfor (i = 0; i < o->line_buffer_nr; i++);\n> +\t\t\tfree((void*)o->line_buffer[i].line);\n\nI haven't looked at the patches yet, but this \";\" on the for line is\nalmost certainly a typo (gcc catches it due to the misleading\nindentation of the next line).\n\n-Peff\n"},{"id":"319720","messageId":"CAGZ79kaWQJ-UJxdVfkaXVdpXzHET+8xg9U9Pj9geE4kchcR1zA@mail.gmail.com","threadId":"45958","inReplyTo":"20170514040559.uvs6c2dnd6aelrln@sigill.intra.peff.net","subject":"Re: [PATCH 18/19] diff: buffer all output if asked to","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-14T04:25:59Z","receivedAt":"2017-05-14T04:26:05Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Sat, May 13, 2017 at 9:06 PM, Jeff King wrote:\n> On Sat, May 13, 2017 at 09:01:16PM -0700, Stefan Beller wrote:\n>\n>> + for (i = 0; i < o->line_buffer_nr; i++);\n>> + free((void*)o->line_buffer[i].line);\n>\n> I haven't looked at the patches yet, but this \";\" on the for line is\n> almost certainly a typo (gcc catches it due to the misleading\n> indentation of the next line).\n\nGrr :/\n\nI have spent hours trying to figure out why this does not work,\nquestioning the design, my mental model of how pointers work\nand programming in general.\n\n/me should get gcc 6 and set -Wmisleading-indentation\n\nThanks,\nStefan\n\n>\n> -Peff\n"},{"id":"319759","messageId":"xmqqk25ivcnb.fsf@gitster.mtv.corp.google.com","threadId":"45958","inReplyTo":"20170514040117.25865-3-sbeller@google.com","subject":"Re: [PATCH 02/19] diff: move line ending check into emit_hunk_header","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-15T06:48:24Z","receivedAt":"2017-05-15T06:48:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> In a later patch, I want to propose an option to detect&color\n> moved lines in a diff, which cannot be done in a one-pass over\n> the diff. Instead we need to go over the whole diff twice,\n> because we cannot detect the first line of the two corresponding\n> lines (+ and -) that got moved.\n>\n> So to prepare the diff machinery for two pass algorithms\n> (i.e. buffer it all up and then operate on the result),\n> move all emissions to places, such that the only emitting\n> function is emit_line_0.\n>\n> This patch moves code that is conceptually part of\n> emit_hunk_header, but was using output in fn_out_consume,\n> back to emit_hunk_header.\n\nMakes sort-of sense. If I were selling this patch, I'd remove the\nfirst two paragraph and stress on how completing the line inside\nemit_hunk_header() is conceptually cleaner than doing it outside.\n\n\temit_hunk_header() function is responsible for assembling a\n\thunk header and calling emit_line() to send the hunk header\n\tto the output file. Its only caller fn_out_consume() needs\n\tto prepare for a case where the function emits an incomplete\n\tline and add the terminating LF. \n\n\tInstead make sure emit_hunk_header() to always send a\n\tcompleted line to emit_line().\n\nor something like that.\n\nNote that I am not saying \"buffering the entire diff in-core? why\nshould we support such a use case?\". I am saying that this change\nis a clean-up that is justifiable, without having to answer such a\nquestion.\n\n>\n> Meanwhile simplify it by using a function that is designed for it.\n>\n> Signed-off-by: Stefan Beller \n> ---\n> diff.c | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/diff.c b/diff.c\n> index 3f5bf8b5a4..c2ed605cd0 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -677,6 +677,8 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n> \t}\n> \n> \tstrbuf_add(&msgbuf, line + len, org_len - len);\n> +\tstrbuf_complete_line(&msgbuf);\n> +\n> \temit_line(ecbdata->opt, \"\", \"\", msgbuf.buf, msgbuf.len);\n> \tstrbuf_release(&msgbuf);\n> }\n> @@ -1315,8 +1317,6 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n> \t\tlen = sane_truncate_line(ecbdata, line, len);\n> \t\tfind_lno(line, ecbdata);\n> \t\temit_hunk_header(ecbdata, line, len);\n> -\t\tif (line[len-1] != '\\n')\n> -\t\t\tputc('\\n', o->file);\n> \t\treturn;\n> \t}\n"},{"id":"319780","messageId":"xmqq37c6uw7n.fsf@gitster.mtv.corp.google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"Re: [RFC PATCH 00/19] Diff machine: highlight moved lines.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-15T12:43:24Z","receivedAt":"2017-05-15T12:43:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> For details on *why* see the commit message of the last commit.\n\nLuckily, we have a good test case to see how effective this approach\nis in the flight. Running\n\n $ git diff master...'pu^{/Merge branch .js/blame-lib}'^2\n\nwith your new feature should tell us that bulk of blame.[ch] that\nare new files came from builtin/blame.c with some symbols renamed.\n\n;-)\n"},{"id":"319795","messageId":"CAGZ79kYLNojFHE7mhgmy84-tRbNN66Kt=DFQoMAKz_p4=_m8fA@mail.gmail.com","threadId":"45958","inReplyTo":"xmqqk25ivcnb.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 02/19] diff: move line ending check into emit_hunk_header","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-15T16:13:29Z","receivedAt":"2017-05-15T16:13:35Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Sun, May 14, 2017 at 11:48 PM, Junio C Hamano wrote:\n> Stefan Beller writes:\n>\n>> In a later patch, I want to propose an option to detect&color\n>> moved lines in a diff, which cannot be done in a one-pass over\n>> the diff. Instead we need to go over the whole diff twice,\n>> because we cannot detect the first line of the two corresponding\n>> lines (+ and -) that got moved.\n>>\n>> So to prepare the diff machinery for two pass algorithms\n>> (i.e. buffer it all up and then operate on the result),\n>> move all emissions to places, such that the only emitting\n>> function is emit_line_0.\n>>\n>> This patch moves code that is conceptually part of\n>> emit_hunk_header, but was using output in fn_out_consume,\n>> back to emit_hunk_header.\n>\n> Makes sort-of sense. If I were selling this patch, I'd remove the\n> first two paragraph and stress on how completing the line inside\n> emit_hunk_header() is conceptually cleaner than doing it outside.\n>\n> emit_hunk_header() function is responsible for assembling a\n> hunk header and calling emit_line() to send the hunk header\n> to the output file. Its only caller fn_out_consume() needs\n> to prepare for a case where the function emits an incomplete\n> line and add the terminating LF.\n>\n> Instead make sure emit_hunk_header() to always send a\n> completed line to emit_line().\n>\n> or something like that.\n>\n> Note that I am not saying \"buffering the entire diff in-core? why\n> should we support such a use case?\". I am saying that this change\n> is a clean-up that is justifiable, without having to answer such a\n> question.\n\nRight, the first couple patches are more cleanup than preparation.\nI considered sending them on their own, but then decided to rather\ninclude it in this series.\n\nI'll reword the commit message for a resend.\n\nThanks,\nStefan\n"},{"id":"319798","messageId":"CAGZ79kaJeeXvcwj-Jc+KckLLzBvtOShp=bXjqHNQ83NBtzpaqA@mail.gmail.com","threadId":"45958","inReplyTo":"xmqq37c6uw7n.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC PATCH 00/19] Diff machine: highlight moved lines.","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-15T16:33:56Z","receivedAt":"2017-05-15T16:34:06Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, May 15, 2017 at 5:43 AM, Junio C Hamano wrote:\n> Stefan Beller writes:\n>\n>> For details on *why* see the commit message of the last commit.\n>\n> Luckily, we have a good test case to see how effective this approach\n> is in the flight. Running\n>\n> $ git diff master...'pu^{/Merge branch .js/blame-lib}'^2\n>\n> with your new feature should tell us that bulk of blame.[ch] that\n> are new files came from builtin/blame.c with some symbols renamed.\n\nOh! Yeah, that looks nice. Though looking through that[1] it seems not yet\noptimal to me.\n\nSo we have 2 additional colors for moved code to differentiate between\nadjacent moved blocks. However in this implementation we toggle between\nthese two colors even if we're not adjacent, such that the moved_alternative\ncolor shows up quite frequently.\n\nSo if we have normal lines in between, we may want to reset to \"default\"\nmoved color.\n\n[1] http://i.imgur.com/djAoTf8.png\n\nThanks for pointing out this series,\nStefan\n"},{"id":"319810","messageId":"115f4d30-5c97-9ee9-e56b-3cf955a5454e@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-4-sbeller@google.com","subject":"Re: [PATCH 03/19] diff.c: drop 'nofirst' from emit_line_0","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-05-15T18:26:33Z","receivedAt":"2017-05-15T18:26:41Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 05/13/2017 09:01 PM, Stefan Beller wrote:\n> In 250f79930d (diff.c: split emit_line() from the first char and the rest\n> of the line, 2009-09-14) we introduced the local variable 'nofirst' that\n> indicates if we have no first sign character. With the given implementation\n> we had to use an extra variable unlike reusing 'first' because the lines\n> first character could be '\\0'.\n>\n> Change the meaning of the 'first' argument to not mean the first character\n> of the line, but rather just containing the sign that is prepended to the\n> line. Refactor emit_line to not include the lines first character, but pass\n> the complete line as well as a '\\0' sign, which now serves as an indication\n> not to print a sign.\n>\n> With this patch other callers hard code the sign (which are '+', '-',\n> ' ' and '\\\\') such that we do not run into unexpectedly emitting an\n> error-nous '\\0'.\n\n\"erroneous\"?\n\nI also don't understand the meaning of this paragraph - if you mean that \nthis patch teaches other callers to hardcode the sign, I don't see any \nsuch changes in the diff below.\n\nAfter reading the patch below, would this commit message be better:\n\n[begin]\ndiff.c: teach emit_line_0 to accept sign parameter\n\nInstead of a separate \"first\" parameter representing the first character \nof the line to be printed, make emit_line_0 take an optional \"sign\" \nparameter specifically intended to hold the sign of the line. Callers \nthat store the sign and line separately can use the \"sign\" parameter \nlike they used the \"first\" parameter previously, and callers that store \nthe sign and line together (or do not have a sign) no longer need to \nmanipulate their arguments to fit the requirements of emit_line_0.\n\n(And then mention that you have checked all the callers and that none of \nthem send '\\n' or '\\r' as the sign, as you have done in this version.)\n[end]\n\n>\n> The audit of the caller revealed that the sign cannot be '\\n' or '\\r',\n> so remove that condition for trailing newline or carriage return in the\n> sign; the else part of the condition handles the len==0 perfectly,\n> so we can drop the if/else construct.\n>\n> Signed-off-by: Stefan Beller \n> ---\n> diff.c | 40 +++++++++++++++++-----------------------\n> 1 file changed, 17 insertions(+), 23 deletions(-)\n>\n> diff --git a/diff.c b/diff.c\n> index c2ed605cd0..4269b8dccf 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -517,33 +517,24 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n> }\n>\n> static void emit_line_0(struct diff_options *o, const char *set, const char *reset,\n> -\t\t\tint first, const char *line, int len)\n> +\t\t\tint sign, const char *line, int len)\n> {\n> \tint has_trailing_newline, has_trailing_carriage_return;\n> -\tint nofirst;\n> \tFILE *file = o->file;\n>\n> \tfputs(diff_line_prefix(o), file);\n>\n> -\tif (len == 0) {\n> -\t\thas_trailing_newline = (first == '\\n');\n> -\t\thas_trailing_carriage_return = (!has_trailing_newline &&\n> -\t\t\t\t\t\t(first == '\\r'));\n> -\t\tnofirst = has_trailing_newline || has_trailing_carriage_return;\n> -\t} else {\n> -\t\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n> -\t\tif (has_trailing_newline)\n> -\t\t\tlen--;\n> -\t\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n> -\t\tif (has_trailing_carriage_return)\n> -\t\t\tlen--;\n> -\t\tnofirst = 0;\n> -\t}\n> +\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n> +\tif (has_trailing_newline)\n> +\t\tlen--;\n> +\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n> +\tif (has_trailing_carriage_return)\n> +\t\tlen--;\n>\n> -\tif (len || !nofirst) {\n> +\tif (len || sign) {\n> \t\tfputs(set, file);\n> -\t\tif (!nofirst)\n> -\t\t\tfputc(first, file);\n> +\t\tif (sign)\n> +\t\t\tfputc(sign, file);\n> \t\tfwrite(line, len, 1, file);\n> \t\tfputs(reset, file);\n> \t}\n> @@ -556,7 +547,7 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n> static void emit_line(struct diff_options *o, const char *set, const char *reset,\n> \t\t const char *line, int len)\n> {\n> -\temit_line_0(o, set, reset, line[0], line+1, len-1);\n> +\temit_line_0(o, set, reset, 0, line, len);\n> }\n\nMaybe this function is unnecessary now that emit_line_0 can take the \nline directly.\n\n>\n> static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n> @@ -4822,9 +4813,12 @@ void diff_flush(struct diff_options *options)\n>\n> \tif (output_format & DIFF_FORMAT_PATCH) {\n> \t\tif (separator) {\n> -\t\t\tfprintf(options->file, \"%s%c\",\n> -\t\t\t\tdiff_line_prefix(options),\n> -\t\t\t\toptions->line_termination);\n> +\t\t\tchar term[2];\n> +\t\t\tterm[0] = options->line_termination;\n> +\t\t\tterm[1] = '\\0';\n> +\n> +\t\t\temit_line(options, NULL, NULL,\n> +\t\t\t\t term, 1);\n\nIf options->line_termination is 0, this is actually a zero-length string \n(not 1).\n\n> \t\t\tif (options->stat_sep) {\n> \t\t\t\t/* attach patch instead of inline */\n> \t\t\t\tfputs(options->stat_sep, options->file);\n>\n"},{"id":"319812","messageId":"847ecb20-ebbe-7f7a-e532-4e75e69c05c2@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-6-sbeller@google.com","subject":"Re: [PATCH 05/19] diff.c: emit_line_0 can handle no color setting","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-05-15T18:31:26Z","receivedAt":"2017-05-15T18:31:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 05/13/2017 09:01 PM, Stefan Beller wrote:\n> In a later patch, I want to propose an option to detect&color\n> moved lines in a diff, which cannot be done in a one-pass over\n> the diff. Instead we need to go over the whole diff twice,\n> because we cannot detect the first line of the two corresponding\n> lines (+ and -) that got moved.\n>\n> So to prepare the diff machinery for two pass algorithms\n> (i.e. buffer it all up and then operate on the result),\n> move all emissions to places, such that the only emitting\n> function is emit_line_0.\n>\n> In later patches we may pass lines that are not colored to\n> the central function emit_line_0, so we\n> need to emit the color only when it is non-NULL.\n\nThe diff below seems to just make emit_line_0 allow NULL for set and \nreset, unlike what the commit message above describes. (And is that \nnecessary? Couldn't the caller just pass \"\" if they don't want any \nsetting and/or resetting?)\n\n>\n> Signed-off-by: Stefan Beller \n> ---\n> diff.c | 6 ++++--\n> 1 file changed, 4 insertions(+), 2 deletions(-)\n>\n> diff --git a/diff.c b/diff.c\n> index 381b572d76..48f0fb98dc 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -532,11 +532,13 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n> \t\tlen--;\n>\n> \tif (len || sign) {\n> -\t\tfputs(set, file);\n> +\t\tif (set)\n> +\t\t\tfputs(set, file);\n> \t\tif (sign)\n> \t\t\tfputc(sign, file);\n> \t\tfwrite(line, len, 1, file);\n> -\t\tfputs(reset, file);\n> +\t\tif (reset)\n> +\t\t\tfputs(reset, file);\n> \t}\n> \tif (has_trailing_carriage_return)\n> \t\tfputc('\\r', file);\n>\n"},{"id":"319813","messageId":"CAGZ79kZSixaG2CoO-WV47=-eE96yhtoCm82KFbZ=-=8c-VPXcA@mail.gmail.com","threadId":"45958","inReplyTo":"115f4d30-5c97-9ee9-e56b-3cf955a5454e@google.com","subject":"Re: [PATCH 03/19] diff.c: drop 'nofirst' from emit_line_0","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-15T18:33:03Z","receivedAt":"2017-05-15T18:33:10Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, May 15, 2017 at 11:26 AM, Jonathan Tan wrote:\n\n> \"erroneous\"?\n\nyep, words are hard.\n\n>\n> I also don't understand the meaning of this paragraph - if you mean that\n> this patch teaches other callers to hardcode the sign, I don't see any such\n> changes in the diff below.\n\nThe last two hunks of the patch switch two callers that call with a sign\nthat is hard to reason about.\n\n> After reading the patch below, would this commit message be better:\n>\n> [begin]\n> diff.c: teach emit_line_0 to accept sign parameter\n>\n> Instead of a separate \"first\" parameter representing the first character of\n> the line to be printed, make emit_line_0 take an optional \"sign\" parameter\n> specifically intended to hold the sign of the line. Callers that store the\n> sign and line separately can use the \"sign\" parameter like they used the\n> \"first\" parameter previously, and callers that store the sign and line\n> together (or do not have a sign) no longer need to manipulate their\n> arguments to fit the requirements of emit_line_0.\n>\n> (And then mention that you have checked all the callers and that none of\n> them send '\\n' or '\\r' as the sign, as you have done in this version.)\n> [end]\n\nThat describes the situation better, indeed.\n\n>> @@ -556,7 +547,7 @@ static void emit_line_0(struct diff_options *o, const\n>> char *set, const char *res\n>> static void emit_line(struct diff_options *o, const char *set, const char\n>> *reset,\n>> const char *line, int len)\n>> {\n>> - emit_line_0(o, set, reset, line[0], line+1, len-1);\n>> + emit_line_0(o, set, reset, 0, line, len);\n>> }\n>\n>\n> Maybe this function is unnecessary now that emit_line_0 can take the line\n> directly.\n\nThat would produce a lot of code churn, specifically in later patches;\nbut I can remove that function if anyone feels strongly about it.\n\n>> + char term[2];\n>> + term[0] = options->line_termination;\n>> + term[1] = '\\0';\n>> +\n>> + emit_line(options, NULL, NULL,\n>> + term, 1);\n>\n>\n> If options->line_termination is 0, this is actually a zero-length string\n> (not 1).\n\nSo passing in !!options->line_termination should be fine?\n\nThanks,\nStefan\n"},{"id":"319815","messageId":"c88f7f39-b0c1-d437-3786-6c470bfe7d00@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-9-sbeller@google.com","subject":"Re: [PATCH 08/19] diff.c: convert builtin_diff to use emit_line_*","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-05-15T18:42:10Z","receivedAt":"2017-05-15T18:42:22Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 05/13/2017 09:01 PM, Stefan Beller wrote:\n> In a later patch, I want to propose an option to detect&color\n> moved lines in a diff, which cannot be done in a one-pass over\n> the diff. Instead we need to go over the whole diff twice,\n> because we cannot detect the first line of the two corresponding\n> lines (+ and -) that got moved.\n>\n> So to prepare the diff machinery for two pass algorithms\n> (i.e. buffer it all up and then operate on the result),\n> move all emissions to places, such that the only emitting\n> function is emit_line_0.\n>\n> This covers builtin_diff.\n>\n> Signed-off-by: Stefan Beller \n> ---\n> diff.c | 43 ++++++++++++++++++++++++++-----------------\n> 1 file changed, 26 insertions(+), 17 deletions(-)\n>\n> diff --git a/diff.c b/diff.c\n> index 93343a9ccc..8e00010bf4 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -1293,8 +1293,9 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n> \to->found_changes = 1;\n>\n> \tif (ecbdata->header) {\n> -\t\tfprintf(o->file, \"%s\", ecbdata->header->buf);\n> -\t\tstrbuf_reset(ecbdata->header);\n> +\t\temit_line(o, NULL, NULL,\n> +\t\t\t ecbdata->header->buf, ecbdata->header->len);\n> +\t\tstrbuf_release(ecbdata->header);\n> \t\tecbdata->header = NULL;\n> \t}\n>\n> @@ -2407,7 +2408,7 @@ static void builtin_diff(const char *name_a,\n> \tb_two = quote_two(b_prefix, name_b + (*name_b == '/'));\n> \tlbl[0] = DIFF_FILE_VALID(one) ? a_one : \"/dev/null\";\n> \tlbl[1] = DIFF_FILE_VALID(two) ? b_two : \"/dev/null\";\n> -\tstrbuf_addf(&header, \"%s%sdiff --git %s %s%s\\n\", line_prefix, meta, a_one, b_two, reset);\n> +\tstrbuf_addf(&header, \"%sdiff --git %s %s%s\\n\", meta, a_one, b_two, reset);\n> \tif (lbl[0][0] == '/') {\n> \t\t/* /dev/null */\n> \t\tstrbuf_addf(&header, \"%s%snew file mode %06o%s\\n\", line_prefix, meta, two->mode, reset);\n> @@ -2439,7 +2440,7 @@ static void builtin_diff(const char *name_a,\n> \t\tif (complete_rewrite &&\n> \t\t (textconv_one || !diff_filespec_is_binary(one)) &&\n> \t\t (textconv_two || !diff_filespec_is_binary(two))) {\n> -\t\t\tfprintf(o->file, \"%s\", header.buf);\n> +\t\t\temit_line(o, NULL, NULL, header.buf, header.len);\n> \t\t\tstrbuf_reset(&header);\n> \t\t\temit_rewrite_diff(name_a, name_b, one, two,\n> \t\t\t\t\t\ttextconv_one, textconv_two, o);\n> @@ -2449,7 +2450,8 @@ static void builtin_diff(const char *name_a,\n> \t}\n>\n> \tif (o->irreversible_delete && lbl[1][0] == '/') {\n> -\t\tfprintf(o->file, \"%s\", header.buf);\n> +\t\tif (header.len)\n> +\t\t\temit_line(o, NULL, NULL, header.buf, header.len);\n\nThis used to unconditionally output diff_line_prefix(0), but now outputs \nit only if the non-prefix part is blank. Is that expected? (Same \ncomments below.)\n\n> \t\tstrbuf_reset(&header);\n> \t\tgoto free_ab_and_return;\n> \t} else if (!DIFF_OPT_TST(o, TEXT) &&\n> @@ -2459,13 +2461,16 @@ static void builtin_diff(const char *name_a,\n> \t\t S_ISREG(one->mode) && S_ISREG(two->mode) &&\n> \t\t !DIFF_OPT_TST(o, BINARY)) {\n> \t\t\tif (!oidcmp(&one->oid, &two->oid)) {\n> -\t\t\t\tif (must_show_header)\n> -\t\t\t\t\tfprintf(o->file, \"%s\", header.buf);\n> +\t\t\t\tif (must_show_header && header.len)\n> +\t\t\t\t\temit_line(o, NULL, NULL,\n> +\t\t\t\t\t\t header.buf, header.len);\n> \t\t\t\tgoto free_ab_and_return;\n> \t\t\t}\n> -\t\t\tfprintf(o->file, \"%s\", header.buf);\n> -\t\t\tfprintf(o->file, \"%sBinary files %s and %s differ\\n\",\n> -\t\t\t\tline_prefix, lbl[0], lbl[1]);\n> +\t\t\tif (header.len)\n> +\t\t\t\temit_line(o, NULL, NULL,\n> +\t\t\t\t\t header.buf, header.len);\n> +\t\t\temit_line_fmt(o, 0, 0, \"Binary files %s and %s differ\\n\",\n> +\t\t\t\t lbl[0], lbl[1]);\n> \t\t\tgoto free_ab_and_return;\n> \t\t}\n> \t\tif (fill_mmfile(&mf1, one) < 0 || fill_mmfile(&mf2, two) < 0)\n> @@ -2473,17 +2478,21 @@ static void builtin_diff(const char *name_a,\n> \t\t/* Quite common confusing case */\n> \t\tif (mf1.size == mf2.size &&\n> \t\t !memcmp(mf1.ptr, mf2.ptr, mf1.size)) {\n> -\t\t\tif (must_show_header)\n> -\t\t\t\tfprintf(o->file, \"%s\", header.buf);\n> +\t\t\tif (must_show_header && header.len)\n> +\t\t\t\temit_line(o, NULL, NULL,\n> +\t\t\t\t\t header.buf, header.len);\n> \t\t\tgoto free_ab_and_return;\n> \t\t}\n> -\t\tfprintf(o->file, \"%s\", header.buf);\n> +\t\tif (header.len)\n> +\t\t\temit_line(o, NULL, NULL,\n> +\t\t\t\t header.buf, header.len);\n> \t\tstrbuf_reset(&header);\n> \t\tif (DIFF_OPT_TST(o, BINARY))\n> \t\t\temit_binary_diff(o->file, &mf1, &mf2, line_prefix);\n> \t\telse\n> -\t\t\tfprintf(o->file, \"%sBinary files %s and %s differ\\n\",\n> -\t\t\t\tline_prefix, lbl[0], lbl[1]);\n> +\t\t\temit_line_fmt(o, NULL, NULL,\n> +\t\t\t\t \"Binary files %s and %s differ\\n\",\n> +\t\t\t\t lbl[0], lbl[1]);\n> \t\to->found_changes = 1;\n> \t} else {\n> \t\t/* Crazy xdl interfaces.. */\n> @@ -2494,8 +2503,8 @@ static void builtin_diff(const char *name_a,\n> \t\tstruct emit_callback ecbdata;\n> \t\tconst struct userdiff_funcname *pe;\n>\n> -\t\tif (must_show_header) {\n> -\t\t\tfprintf(o->file, \"%s\", header.buf);\n> +\t\tif (must_show_header && header.len) {\n> +\t\t\temit_line(o, NULL, NULL, header.buf, header.len);\n> \t\t\tstrbuf_reset(&header);\n> \t\t}\n>\n>\n"},{"id":"319817","messageId":"d67fa7ed-fbde-7bce-a691-01dafaa93732@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-11-sbeller@google.com","subject":"Re: [PATCH 10/19] diff.c: convert emit_rewrite_lines to use emit_line_*","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-05-15T19:09:40Z","receivedAt":"2017-05-15T19:09:48Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"\n\nOn 05/13/2017 09:01 PM, Stefan Beller wrote:\n> In a later patch, I want to propose an option to detect&color\n> moved lines in a diff, which cannot be done in a one-pass over\n> the diff. Instead we need to go over the whole diff twice,\n> because we cannot detect the first line of the two corresponding\n> lines (+ and -) that got moved.\n>\n> So to prepare the diff machinery for two pass algorithms\n> (i.e. buffer it all up and then operate on the result),\n> move all emissions to places, such that the only emitting\n> function is emit_line_0.\n>\n> This covers emit_rewrite_lines.\n>\n> Signed-off-by: Stefan Beller \n> ---\n> diff.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/diff.c b/diff.c\n> index e4b46fee4f..369c804f03 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -748,7 +748,7 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n> \tif (!endp) {\n> \t\tconst char *context = diff_get_color(ecb->color_diff,\n> \t\t\t\t\t\t DIFF_CONTEXT);\n> -\t\tputc('\\n', ecb->opt->file);\n> +\t\temit_line(ecb->opt, NULL, NULL, \"\\n\", 1);\n\nThis outputs diff_line_prefix(ecb->opt) - is that OK?\n\n> \t\temit_line_0(ecb->opt, context, reset, '\\\\',\n> \t\t\t nneof, strlen(nneof));\n> \t}\n>\n"},{"id":"319825","messageId":"20170515192242.GB79147@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-4-sbeller@google.com","subject":"Re: [PATCH 03/19] diff.c: drop 'nofirst' from emit_line_0","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-05-15T19:22:42Z","receivedAt":"2017-05-15T19:22:53Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 05/13, Stefan Beller wrote:\n> In 250f79930d (diff.c: split emit_line() from the first char and the rest\n> of the line, 2009-09-14) we introduced the local variable 'nofirst' that\n> indicates if we have no first sign character. With the given implementation\n> we had to use an extra variable unlike reusing 'first' because the lines\n> first character could be '\\0'.\n> \n> Change the meaning of the 'first' argument to not mean the first character\n> of the line, but rather just containing the sign that is prepended to the\n> line. Refactor emit_line to not include the lines first character, but pass\n> the complete line as well as a '\\0' sign, which now serves as an indication\n> not to print a sign.\n> \n> With this patch other callers hard code the sign (which are '+', '-',\n> ' ' and '\\\\') such that we do not run into unexpectedly emitting an\n> error-nous '\\0'.\n> \n> The audit of the caller revealed that the sign cannot be '\\n' or '\\r',\n> so remove that condition for trailing newline or carriage return in the\n> sign; the else part of the condition handles the len==0 perfectly,\n> so we can drop the if/else construct.\n> \n> Signed-off-by: Stefan Beller \n> ---\n> diff.c | 40 +++++++++++++++++-----------------------\n> 1 file changed, 17 insertions(+), 23 deletions(-)\n> \n> diff --git a/diff.c b/diff.c\n> index c2ed605cd0..4269b8dccf 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -517,33 +517,24 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n> }\n> \n> static void emit_line_0(struct diff_options *o, const char *set, const char *reset,\n> -\t\t\tint first, const char *line, int len)\n> +\t\t\tint sign, const char *line, int len)\n> {\n> \tint has_trailing_newline, has_trailing_carriage_return;\n> -\tint nofirst;\n> \tFILE *file = o->file;\n> \n> \tfputs(diff_line_prefix(o), file);\n> \n> -\tif (len == 0) {\n> -\t\thas_trailing_newline = (first == '\\n');\n> -\t\thas_trailing_carriage_return = (!has_trailing_newline &&\n> -\t\t\t\t\t\t(first == '\\r'));\n> -\t\tnofirst = has_trailing_newline || has_trailing_carriage_return;\n> -\t} else {\n> -\t\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n> -\t\tif (has_trailing_newline)\n> -\t\t\tlen--;\n> -\t\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n> -\t\tif (has_trailing_carriage_return)\n> -\t\t\tlen--;\n> -\t\tnofirst = 0;\n> -\t}\n> +\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n> +\tif (has_trailing_newline)\n> +\t\tlen--;\n> +\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n> +\tif (has_trailing_carriage_return)\n> +\t\tlen--;\n\nDoes the order of newline/carriage return always the same?\n\n> \n> -\tif (len || !nofirst) {\n> +\tif (len || sign) {\n> \t\tfputs(set, file);\n> -\t\tif (!nofirst)\n> -\t\t\tfputc(first, file);\n> +\t\tif (sign)\n> +\t\t\tfputc(sign, file);\n> \t\tfwrite(line, len, 1, file);\n> \t\tfputs(reset, file);\n> \t}\n> @@ -556,7 +547,7 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n> static void emit_line(struct diff_options *o, const char *set, const char *reset,\n> \t\t const char *line, int len)\n> {\n> -\temit_line_0(o, set, reset, line[0], line+1, len-1);\n> +\temit_line_0(o, set, reset, 0, line, len);\n> }\n> \n> static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n> @@ -4822,9 +4813,12 @@ void diff_flush(struct diff_options *options)\n> \n> \tif (output_format & DIFF_FORMAT_PATCH) {\n> \t\tif (separator) {\n> -\t\t\tfprintf(options->file, \"%s%c\",\n> -\t\t\t\tdiff_line_prefix(options),\n> -\t\t\t\toptions->line_termination);\n> +\t\t\tchar term[2];\n> +\t\t\tterm[0] = options->line_termination;\n> +\t\t\tterm[1] = '\\0';\n> +\n> +\t\t\temit_line(options, NULL, NULL,\n> +\t\t\t\t term, 1);\n> \t\t\tif (options->stat_sep) {\n> \t\t\t\t/* attach patch instead of inline */\n> \t\t\t\tfputs(options->stat_sep, options->file);\n> -- \n> 2.13.0.18.g183880de0a\n> \n\n-- \nBrandon Williams\n"},{"id":"319826","messageId":"CAGZ79kafiLjKbMSz5qpm6uqEJqHKJkn0rQQ--OA7g0w5=5S3jw@mail.gmail.com","threadId":"45958","inReplyTo":"d67fa7ed-fbde-7bce-a691-01dafaa93732@google.com","subject":"Re: [PATCH 10/19] diff.c: convert emit_rewrite_lines to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-15T19:31:05Z","receivedAt":"2017-05-15T19:31:12Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, May 15, 2017 at 12:09 PM, Jonathan Tan wrote:\n>\n>\n> On 05/13/2017 09:01 PM, Stefan Beller wrote:\n>>\n>> In a later patch, I want to propose an option to detect&color\n>> moved lines in a diff, which cannot be done in a one-pass over\n>> the diff. Instead we need to go over the whole diff twice,\n>> because we cannot detect the first line of the two corresponding\n>> lines (+ and -) that got moved.\n>>\n>> So to prepare the diff machinery for two pass algorithms\n>> (i.e. buffer it all up and then operate on the result),\n>> move all emissions to places, such that the only emitting\n>> function is emit_line_0.\n>>\n>> This covers emit_rewrite_lines.\n>>\n>> Signed-off-by: Stefan Beller \n>> ---\n>> diff.c | 2 +-\n>> 1 file changed, 1 insertion(+), 1 deletion(-)\n>>\n>> diff --git a/diff.c b/diff.c\n>> index e4b46fee4f..369c804f03 100644\n>> --- a/diff.c\n>> +++ b/diff.c\n>> @@ -748,7 +748,7 @@ static void emit_rewrite_lines(struct emit_callback\n>> *ecb,\n>> if (!endp) {\n>> const char *context = diff_get_color(ecb->color_diff,\n>> DIFF_CONTEXT);\n>> - putc('\\n', ecb->opt->file);\n>> + emit_line(ecb->opt, NULL, NULL, \"\\n\", 1);\n>\n>\n> This outputs diff_line_prefix(ecb->opt) - is that OK?\n\nIt shows this area is not covered well by our test suite.\n\nMy first reaction was that this is not ok, but we'd have\nto inspect the situation. It was introduced in\n35e2d03c2c (Fix '\\ No newline...' annotation in rewrite diffs,\n2012-08-04).\n\nAnd looking at the code of the function I think this is broken.\n\nI wonder what the best way forward is for this patch series here,\nas we'd need to buffer the last line. That should be fine as it is\na corner case, maybe:\n\ndiff --git a/diff.c b/diff.c\nindex 0f10736ee6..f46e52135d 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1011,15 +1011,27 @@ static void add_line_count(struct strbuf *out,\nint count)\n static void emit_rewrite_lines(struct emit_callback *ecb,\n int prefix, const char *data, int size)\n {\n- const char *endp = NULL;\n- static const char *nneof = \" No newline at end of file\\n\";\n+ static const char *nneof = \"\\\\ No newline at end of file\\n\";\n const char *reset = diff_get_color(ecb->color_diff, DIFF_RESET);\n+ struct strbuf sb = STRBUF_INIT;\n\n while (0 < size) {\n int len;\n\n endp = memchr(data, '\\n', size);\n- len = endp ? (endp - data + 1) : size;\n+ if (endp)\n+ len = endp - data + 1;\n+ else {\n+ /* last line has no \\n */\n+ while (0 < size) {\n+ strbuf_addch(&sb, *data);\n+ size -= len;\n+ data += len;\n+ }\n+ strbuf_addch(&sb, '\\n');\n+ data = sb.buf;\n+ len = sb.len;\n+ }\n if (prefix != '+') {\n ecb->lno_in_preimage++;\n emit_del_line(reset, ecb, data, len);\n@@ -1030,12 +1042,12 @@ static void emit_rewrite_lines(struct\nemit_callback *ecb,\n size -= len;\n data += len;\n }\n- if (!endp) {\n+ if (sb.len) {\n const char *context = diff_get_color(ecb->color_diff,\n DIFF_CONTEXT);\n- emit_line(ecb->opt, NULL, NULL, \"\\n\", 1);\n- emit_line_0(ecb->opt, context, reset, '\\\\',\n+ emit_line(ecb->opt, context, reset,\n nneof, strlen(nneof));\n+ strbuf_release(&sb);\n }\n }\n"},{"id":"319827","messageId":"20170515193124.GC79147@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-7-sbeller@google.com","subject":"Re: [PATCH 06/19] diff: add emit_line_fmt","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-05-15T19:31:24Z","receivedAt":"2017-05-15T19:31:32Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 05/13, Stefan Beller wrote:\n> In the following patches we'll convert all printing functions to use\n> the emit_line_* family of functions.\n> \n> Many of the printing functions to be converted are formatted. So offer\n> a formatted function in the emit_line function family as well.\n> \n> Signed-off-by: Stefan Beller \n> ---\n> diff.c | 14 ++++++++++++++\n> 1 file changed, 14 insertions(+)\n> \n> diff --git a/diff.c b/diff.c\n> index 48f0fb98dc..aef159a919 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -552,6 +552,20 @@ static void emit_line(struct diff_options *o, const char *set, const char *reset\n> \temit_line_0(o, set, reset, 0, line, len);\n> }\n> \n> +static void emit_line_fmt(struct diff_options *o,\n> +\t\t\t const char *set, const char *reset,\n> +\t\t\t const char *fmt, ...)\n> +{\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\tva_list ap;\n> +\tva_start(ap, fmt);\n> +\tstrbuf_vaddf(&sb, fmt, ap);\n> +\tva_end(ap);\n> +\n> +\temit_line(o, set, reset, sb.buf, sb.len);\n> +\tstrbuf_release(&sb);\n> +}\n> +\n> static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n> {\n> \tif (!((ecbdata->ws_rule & WS_BLANK_AT_EOF) &&\n> -- \n> 2.13.0.18.g183880de0a\n> \n\nSince this is a new function, and it is marked static, this patch\nshouldn't compile or the compiler should throw a warning or something.\n\n-- \nBrandon Williams\n"},{"id":"319828","messageId":"CAGZ79kZMHuk8g7wwwoDxBOG17Vo3P0s9v9bwzpwy5WnuNUogtw@mail.gmail.com","threadId":"45958","inReplyTo":"20170515192242.GB79147@google.com","subject":"Re: [PATCH 03/19] diff.c: drop 'nofirst' from emit_line_0","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-15T19:35:15Z","receivedAt":"2017-05-15T19:35:29Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, May 15, 2017 at 12:22 PM, Brandon Williams wrote:\n\n> Does the order of newline/carriage return always the same?\n\nhttps://en.wikipedia.org/wiki/Newline\n\nThere are operating systems that like it the other way round.\nThe BBC micro is no longer relevant (IMHO), but RISC OS\nspooled text output *may* be relevant as they released a stable\nversion not that long ago.\n\nBut I would think this code would have issues with RISC OS\ntext spooling without this patch as well.\n\nStefan\n"},{"id":"319829","messageId":"20170515194549.GD79147@google.com","threadId":"45958","inReplyTo":"CAGZ79kZMHuk8g7wwwoDxBOG17Vo3P0s9v9bwzpwy5WnuNUogtw@mail.gmail.com","subject":"Re: [PATCH 03/19] diff.c: drop 'nofirst' from emit_line_0","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-05-15T19:45:49Z","receivedAt":"2017-05-15T19:45:56Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 05/15, Stefan Beller wrote:\n> On Mon, May 15, 2017 at 12:22 PM, Brandon Williams wrote:\n> \n> > Does the order of newline/carriage return always the same?\n> \n> https://en.wikipedia.org/wiki/Newline\n> \n> There are operating systems that like it the other way round.\n> The BBC micro is no longer relevant (IMHO), but RISC OS\n> spooled text output *may* be relevant as they released a stable\n> version not that long ago.\n> \n> But I would think this code would have issues with RISC OS\n> text spooling without this patch as well.\n> \n> Stefan\n\nFair enough, its not relevant to the series. I was just pointing it\nout.\n\n-- \nBrandon Williams\n"},{"id":"319833","messageId":"1b85a773-fb44-9a88-63d3-36c642d07c22@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-16-sbeller@google.com","subject":"Re: [PATCH 15/19] diff.c: convert diff_flush to use emit_line_*","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-05-15T20:21:49Z","receivedAt":"2017-05-15T20:21:57Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 05/13/2017 09:01 PM, Stefan Beller wrote:\n> In a later patch, I want to propose an option to detect&color\n> moved lines in a diff, which cannot be done in a one-pass over\n> the diff. Instead we need to go over the whole diff twice,\n> because we cannot detect the first line of the two corresponding\n> lines (+ and -) that got moved.\n>\n> So to prepare the diff machinery for two pass algorithms\n> (i.e. buffer it all up and then operate on the result),\n> move all emissions to places, such that the only emitting\n> function is emit_line_0.\n>\n> This covers diff_flush.\n>\n> Signed-off-by: Stefan Beller \n> ---\n> diff.c | 4 +++-\n> 1 file changed, 3 insertions(+), 1 deletion(-)\n>\n> diff --git a/diff.c b/diff.c\n> index 07041a35ad..386b28cf47 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -4873,7 +4873,9 @@ void diff_flush(struct diff_options *options)\n> \t\t\t\t term, 1);\n> \t\t\tif (options->stat_sep) {\n> \t\t\t\t/* attach patch instead of inline */\n> -\t\t\t\tfputs(options->stat_sep, options->file);\n> +\t\t\t\temit_line(options, NULL, NULL,\n> +\t\t\t\t\t options->stat_sep,\n> +\t\t\t\t\t strlen(options->stat_sep));\n\nSame comment as patch 10 - is it OK that we now output a prefix too?\n\n> \t\t\t}\n> \t\t}\n>\n>\n"},{"id":"319839","messageId":"CAGZ79kaWAMjzJUkqWOOmUrQcQZzT1RxqYE9tTr4ouy9Ax33t9g@mail.gmail.com","threadId":"45958","inReplyTo":"1b85a773-fb44-9a88-63d3-36c642d07c22@google.com","subject":"Re: [PATCH 15/19] diff.c: convert diff_flush to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-15T22:08:14Z","receivedAt":"2017-05-15T22:08:20Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, May 15, 2017 at 1:21 PM, Jonathan Tan wrote:\n> On 05/13/2017 09:01 PM, Stefan Beller wrote:\n>>\n>> In a later patch, I want to propose an option to detect&color\n>> moved lines in a diff, which cannot be done in a one-pass over\n>> the diff. Instead we need to go over the whole diff twice,\n>> because we cannot detect the first line of the two corresponding\n>> lines (+ and -) that got moved.\n>>\n>> So to prepare the diff machinery for two pass algorithms\n>> (i.e. buffer it all up and then operate on the result),\n>> move all emissions to places, such that the only emitting\n>> function is emit_line_0.\n>>\n>> This covers diff_flush.\n>>\n>> Signed-off-by: Stefan Beller \n>> ---\n>> diff.c | 4 +++-\n>> 1 file changed, 3 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/diff.c b/diff.c\n>> index 07041a35ad..386b28cf47 100644\n>> --- a/diff.c\n>> +++ b/diff.c\n>> @@ -4873,7 +4873,9 @@ void diff_flush(struct diff_options *options)\n>> term, 1);\n>> if (options->stat_sep) {\n>> /* attach patch instead of inline */\n>> - fputs(options->stat_sep, options->file);\n>> + emit_line(options, NULL, NULL,\n>> + options->stat_sep,\n>> + strlen(options->stat_sep));\n>\n>\n> Same comment as patch 10 - is it OK that we now output a prefix too?\n\nIn this case, I would think it is ok. The stat_sep is only used in\n\"format-patch --attach\" for example, which makes NO sense to\nuse in combination with --line-prefix.\n\n(That is already broken; at least the line-prefix part, as we\ndo *not* prefix all the lines with the given prefix. That is because\nstat_sep is a multiline string emitted, starting with '\\n'.)\n\nThanks,\nStefan\n"},{"id":"319842","messageId":"CAGZ79kbyt_PsGGphWJbCs-4MqmGRh+mZaxdsNMg0iyK3P-MAJg@mail.gmail.com","threadId":"45958","inReplyTo":"847ecb20-ebbe-7f7a-e532-4e75e69c05c2@google.com","subject":"Re: [PATCH 05/19] diff.c: emit_line_0 can handle no color setting","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-15T22:11:14Z","receivedAt":"2017-05-15T22:11:21Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, May 15, 2017 at 11:31 AM, Jonathan Tan wrote:\n> On 05/13/2017 09:01 PM, Stefan Beller wrote:\n>>\n>> In a later patch, I want to propose an option to detect&color\n>> moved lines in a diff, which cannot be done in a one-pass over\n>> the diff. Instead we need to go over the whole diff twice,\n>> because we cannot detect the first line of the two corresponding\n>> lines (+ and -) that got moved.\n>>\n>> So to prepare the diff machinery for two pass algorithms\n>> (i.e. buffer it all up and then operate on the result),\n>> move all emissions to places, such that the only emitting\n>> function is emit_line_0.\n>>\n>> In later patches we may pass lines that are not colored to\n>> the central function emit_line_0, so we\n>> need to emit the color only when it is non-NULL.\n>\n>\n> The diff below seems to just make emit_line_0 allow NULL for set and reset,\n> unlike what the commit message above describes. (And is that necessary?\n> Couldn't the caller just pass \"\" if they don't want any setting and/or\n> resetting?)\n>\n\nThey could just give \"\"; but instead of having an empty system\ncall I thought about this short cut.\n\nI'll reword the commit message.\n\nThanks,\nStefan\n"},{"id":"319844","messageId":"f49dc4f8-cc83-934f-e582-79efdc613f70@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-15-sbeller@google.com","subject":"Re: [PATCH 14/19] diff.c: convert word diffing to use emit_line_*","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-05-15T22:40:56Z","receivedAt":"2017-05-15T22:41:06Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 05/13/2017 09:01 PM, Stefan Beller wrote:\n> In a later patch, I want to propose an option to detect&color\n> moved lines in a diff, which cannot be done in a one-pass over\n> the diff. Instead we need to go over the whole diff twice,\n> because we cannot detect the first line of the two corresponding\n> lines (+ and -) that got moved.\n>\n> So to prepare the diff machinery for two pass algorithms\n> (i.e. buffer it all up and then operate on the result),\n> move all emissions to places, such that the only emitting\n> function is emit_line_0.\n>\n> This covers all code related to diffing words.\n>\n> Signed-off-by: Stefan Beller \n> ---\n> diff.c | 56 ++++++++++++++++++++++++++++----------------------------\n> 1 file changed, 28 insertions(+), 28 deletions(-)\n>\n> diff --git a/diff.c b/diff.c\n> index 91dc045a45..07041a35ad 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -886,37 +886,38 @@ struct diff_words_data {\n> \tstruct diff_words_style *style;\n> };\n>\n> -static int fn_out_diff_words_write_helper(FILE *fp,\n> +static int fn_out_diff_words_write_helper(struct diff_options *o,\n> \t\t\t\t\t struct diff_words_style_elem *st_el,\n> \t\t\t\t\t const char *newline,\n> \t\t\t\t\t size_t count, const char *buf,\n> \t\t\t\t\t const char *line_prefix)\n> {\n> -\tint print = 0;\n> +\tstruct strbuf sb = STRBUF_INIT;\n>\n> \twhile (count) {\n> \t\tchar *p = memchr(buf, '\\n', count);\n> -\t\tif (print)\n> -\t\t\tfputs(line_prefix, fp);\n> +\n> \t\tif (p != buf) {\n> -\t\t\tif (st_el->color && fputs(st_el->color, fp) < 0)\n> -\t\t\t\treturn -1;\n> -\t\t\tif (fputs(st_el->prefix, fp) < 0 ||\n> -\t\t\t fwrite(buf, p ? p - buf : count, 1, fp) != 1 ||\n> -\t\t\t fputs(st_el->suffix, fp) < 0)\n> -\t\t\t\treturn -1;\n> -\t\t\tif (st_el->color && *st_el->color\n> -\t\t\t && fputs(GIT_COLOR_RESET, fp) < 0)\n> -\t\t\t\treturn -1;\n> +\t\t\tif (st_el->color)\n> +\t\t\t\tstrbuf_addstr(&sb, st_el->color);\n> +\t\t\tstrbuf_addstr(&sb, st_el->prefix);\n> +\t\t\tstrbuf_add(&sb, buf, p ? p - buf : count);\n> +\t\t\tstrbuf_addstr(&sb, st_el->suffix);\n> +\t\t\tif (st_el->color && *st_el->color)\n> +\t\t\t strbuf_addstr(&sb, GIT_COLOR_RESET);\n> \t\t}\n> \t\tif (!p)\n> -\t\t\treturn 0;\n> -\t\tif (fputs(newline, fp) < 0)\n> -\t\t\treturn -1;\n> +\t\t\tgoto out;\n> +\t\tstrbuf_addstr(&sb, newline);\n> +\t\temit_line(o, NULL, NULL, sb.buf, sb.len);\n\nI suspect that this will need to be refactored more thoroughly. Here, \nfor example, emit_line (which prints the prefix) is printed nearly \nunconditionally, whereas in the original version, \"fputs(line_prefix, \nfp)\" is only printed when \"print\" is true.\n\n> +\t\tstrbuf_reset(&sb);\n> \t\tcount -= p + 1 - buf;\n> \t\tbuf = p + 1;\n> -\t\tprint = 1;\n> \t}\n> +out:\n> +\tif (sb.len)\n> +\t\temit_line(o, NULL, NULL, sb.buf, sb.len);\n> +\tstrbuf_release(&sb);\n> \treturn 0;\n> }\n>\n> @@ -994,25 +995,25 @@ static void fn_out_diff_words_aux(void *priv, char *line, unsigned long len)\n> \t} else\n> \t\tplus_begin = plus_end = diff_words->plus.orig[plus_first].end;\n>\n> -\tif (color_words_output_graph_prefix(diff_words)) {\n> -\t\tfputs(line_prefix, diff_words->opt->file);\n> -\t}\n> +\tif (color_words_output_graph_prefix(diff_words))\n> +\t\temit_line(diff_words->opt, NULL, NULL, \"\", 0);\n> +\n> \tif (diff_words->current_plus != plus_begin) {\n> -\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n> +\t\tfn_out_diff_words_write_helper(diff_words->opt,\n> \t\t\t\t&style->ctx, style->newline,\n> \t\t\t\tplus_begin - diff_words->current_plus,\n> \t\t\t\tdiff_words->current_plus, line_prefix);\n> \t\tif (*(plus_begin - 1) == '\\n')\n> -\t\t\tfputs(line_prefix, diff_words->opt->file);\n> +\t\t\temit_line(diff_words->opt, NULL, NULL, \"\", 0);\n> \t}\n> \tif (minus_begin != minus_end) {\n> -\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n> +\t\tfn_out_diff_words_write_helper(diff_words->opt,\n> \t\t\t\t&style->old, style->newline,\n> \t\t\t\tminus_end - minus_begin, minus_begin,\n> \t\t\t\tline_prefix);\n> \t}\n> \tif (plus_begin != plus_end) {\n> -\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n> +\t\tfn_out_diff_words_write_helper(diff_words->opt,\n> \t\t\t\t&style->new, style->newline,\n> \t\t\t\tplus_end - plus_begin, plus_begin,\n> \t\t\t\tline_prefix);\n> @@ -1109,8 +1110,7 @@ static void diff_words_show(struct diff_words_data *diff_words)\n>\n> \t/* special case: only removal */\n> \tif (!diff_words->plus.text.size) {\n> -\t\tfputs(line_prefix, diff_words->opt->file);\n> -\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n> +\t\tfn_out_diff_words_write_helper(diff_words->opt,\n> \t\t\t&style->old, style->newline,\n> \t\t\tdiff_words->minus.text.size,\n> \t\t\tdiff_words->minus.text.ptr, line_prefix);\n> @@ -1136,8 +1136,8 @@ static void diff_words_show(struct diff_words_data *diff_words)\n> \tif (diff_words->current_plus != diff_words->plus.text.ptr +\n> \t\t\tdiff_words->plus.text.size) {\n> \t\tif (color_words_output_graph_prefix(diff_words))\n> -\t\t\tfputs(line_prefix, diff_words->opt->file);\n> -\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n> +\t\t\temit_line(diff_words->opt, NULL, NULL, \"\", 0);\n> +\t\tfn_out_diff_words_write_helper(diff_words->opt,\n> \t\t\t&style->ctx, style->newline,\n> \t\t\tdiff_words->plus.text.ptr + diff_words->plus.text.size\n> \t\t\t- diff_words->current_plus, diff_words->current_plus,\n>\n"},{"id":"319845","messageId":"20170515224226.GE79147@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-20-sbeller@google.com","subject":"Re: [PATCH 19/19] diff.c: color moved lines differently","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-05-15T22:42:26Z","receivedAt":"2017-05-15T22:42:36Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 05/13, Stefan Beller wrote:\n> When there is a lot of code moved around such as in 11979b9 (2005-11-18,\n> \"http.c: reorder to avoid compilation failure.\") for example, the review\n> process is quite hard, as it is not mentally challenging. It is a rather\n> tedious process, that gets boring quickly. However you still need to read\n> through all of the code to make sure the moved lines are there as supposed.\n> \n> While it is trivial to color up a patch like the following\n> \n> $ git diff\n> diff --git a/file2.c b/file2.c\n> index 9163a0f..8e66dc0 100644\n> --- a/file2.c\n> +++ b/file2.c\n> @@ -3,13 +3,6 @@ void *xmemdupz(const void *data, size_t len)\n> return memcpy(xmallocz(len), data, len);\n> }\n> \n> -int secure_foo(struct user *u)\n> -{\n> - if (!u->is_allowed_foo)\n> - return;\n> - foo(u);\n> -}\n> -\n> char *xstrndup(const char *str, size_t len)\n> {\n> char *p = memchr(str, '\\0', len);\n> diff --git a/test.c b/test.c\n> index a95e6fe..81eb0eb 100644\n> --- a/test.c\n> +++ b/test.c\n> @@ -18,6 +18,13 @@ ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset)\n> return total;\n> }\n> \n> +int secure_foo(struct user *u)\n> +{\n> + if (!u->is_allowed_foo)\n> + return;\n> + foo(u);\n> +}\n> +\n> int xdup(int fd)\n> {\n> int ret = dup(fd);\n> \n> as in this patch all lines that add or remove lines\n> should be colored in the new color that indicates moved\n> lines.\n> \n> However the intention of this patch is to aid reviewers\n> to spotting permutations in the moved code. So consider the\n> following malicious move:\n> \n> diff --git a/file2.c b/file2.c\n> index 9163a0f..8e66dc0 100644\n> --- a/file2.c\n> +++ b/file2.c\n> @@ -3,13 +3,6 @@ void *xmemdupz(const void *data, size_t len)\n> return memcpy(xmallocz(len), data, len);\n> }\n> \n> -int secure_foo(struct user *u)\n> -{\n> - if (!u->is_allowed_foo)\n> - return;\n> - foo(u);\n> -}\n> -\n> char *xstrndup(const char *str, size_t len)\n> {\n> char *p = memchr(str, '\\0', len);\n> diff --git a/test.c b/test.c\n> index a95e6fe..a679c40 100644\n> --- a/test.c\n> +++ b/test.c\n> @@ -18,6 +18,13 @@ ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset)\n> return total;\n> }\n> \n> +int secure_foo(struct user *u)\n> +{\n> + foo(u);\n> + if (!u->is_allowed_foo)\n> + return;\n> +}\n> +\n> int xdup(int fd)\n> {\n> int ret = dup(fd);\n> \n> If the moved code is larger, it is easier to hide some permutation in the\n> code, which is why we would not want to color all lines as \"moved\" in this\n> case. So we do not just need to color lines differently that are added and\n> removed in the same diff, we need to tweak the algorithm a bit more.\n> \n> As the reviewers attention should be brought to the places, where the\n> difference is introduced to the moved code, we cannot just have one new\n> color for all of moved code.\n> \n> First I implemented an alternative design, which would show a moved hunk\n> in one color, and its boundaries in another color. This idea was error\n> prone as it inspected each line and its neighboring lines to determine\n> if the line was (a) moved and (b) if was deep inside a hunk by having\n> matching neighboring lines. This is unreliable as the we can construct\n> hunks which have equal neighbors that just exceed the number of lines\n> inspected. See one of the tests as an example.\n> \n> Instead this provides a dynamic programming greedy algorithm that finds\n> the largest moved hunk and then switches color to the alternative color\n> for the next hunk. By doing this any permutation is recognized and\n> displayed. That implies that there is no dedicated boundary or\n> inside-hunk color, but instead we'll have just two colors alternating\n> for hunks.\n> \n> It would be a bit more UX friendly if the two corresponding hunks\n> (of added and deleted lines) for one move would get the same color id.\n> (Both get \"regular moved\" or \"alternative moved\"). This problem is\n> deferred to a later patch for now.\n> \n> Algorithm-by: Jonathan Tan \n> Signed-off-by: Stefan Beller \n> ---\n> Documentation/config.txt | 12 +-\n> diff.c | 265 ++++++++++++++++++++++++++++++++++++++++++---\n> diff.h | 21 +++-\n> t/t4015-diff-whitespace.sh | 229 +++++++++++++++++++++++++++++++++++++++\n> 4 files changed, 506 insertions(+), 21 deletions(-)\n> \n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 475e874d51..90403c06e3 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -1051,14 +1051,22 @@ This does not affect linkgit:git-format-patch[1] or the\n> 'git-diff-{asterisk}' plumbing commands. Can be overridden on the\n> command line with the `--color[=]` option.\n> \n> +color.moved::\n> +\tA boolean value, whether a diff should color moved lines\n> +\tdifferently. The moved lines are searched for in the diff only.\n> +\tDuplicated lines from somewhere in the project that are not\n> +\tpart of the diff are not colored as moved.\n> +\tDefaults to false.\n> +\n> color.diff.::\n> \tUse customized color for diff colorization. `` specifies\n> \twhich part of the patch to use the specified color, and is one\n> \tof `context` (context text - `plain` is a historical synonym),\n> \t`meta` (metainformation), `frag`\n> \t(hunk header), 'func' (function in hunk header), `old` (removed lines),\n> -\t`new` (added lines), `commit` (commit headers), or `whitespace`\n> -\t(highlighting whitespace errors).\n> +\t`new` (added lines), `commit` (commit headers), `whitespace`\n> +\t(highlighting whitespace errors), `movedFrom` (removed lines that\n> +\treappear), `movedTo` (added lines that were removed elsewhere).\n> \n> color.decorate.::\n> \tUse customized color for 'git log --decorate' output. `` is one\n> diff --git a/diff.c b/diff.c\n> index dbab7fb44e..6372e0eb25 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -31,6 +31,7 @@ static int diff_indent_heuristic; /* experimental */\n> static int diff_rename_limit_default = 400;\n> static int diff_suppress_blank_empty;\n> static int diff_use_color_default = -1;\n> +static int diff_color_moved_default;\n> static int diff_context_default = 3;\n> static int diff_interhunk_context_default;\n> static const char *diff_word_regex_cfg;\n> @@ -55,6 +56,10 @@ static char diff_colors[][COLOR_MAXLEN] = {\n> \tGIT_COLOR_YELLOW,\t/* COMMIT */\n> \tGIT_COLOR_BG_RED,\t/* WHITESPACE */\n> \tGIT_COLOR_NORMAL,\t/* FUNCINFO */\n> +\tGIT_COLOR_BOLD_RED,\t/* OLD_MOVED_A */\n> +\tGIT_COLOR_BG_RED,\t/* OLD_MOVED_B */\n> +\tGIT_COLOR_BOLD_GREEN,\t/* NEW_MOVED_A */\n> +\tGIT_COLOR_BG_GREEN,\t/* NEW_MOVED_B */\n> };\n> \n> static NORETURN void die_want_option(const char *option_name)\n> @@ -80,6 +85,14 @@ static int parse_diff_color_slot(const char *var)\n> \t\treturn DIFF_WHITESPACE;\n> \tif (!strcasecmp(var, \"func\"))\n> \t\treturn DIFF_FUNCINFO;\n> +\tif (!strcasecmp(var, \"oldmoved\"))\n> +\t\treturn DIFF_FILE_OLD_MOVED;\n> +\tif (!strcasecmp(var, \"oldmovedalternative\"))\n> +\t\treturn DIFF_FILE_OLD_MOVED_ALT;\n> +\tif (!strcasecmp(var, \"newmoved\"))\n> +\t\treturn DIFF_FILE_NEW_MOVED;\n> +\tif (!strcasecmp(var, \"newmovedalternative\"))\n> +\t\treturn DIFF_FILE_NEW_MOVED_ALT;\n> \treturn -1;\n> }\n> \n> @@ -234,6 +247,10 @@ int git_diff_ui_config(const char *var, const char *value, void *cb)\n> \t\tdiff_use_color_default = git_config_colorbool(var, value);\n> \t\treturn 0;\n> \t}\n> +\tif (!strcmp(var, \"color.moved\")) {\n> +\t\tdiff_color_moved_default = git_config_bool(var, value);\n> +\t\treturn 0;\n> +\t}\n> \tif (!strcmp(var, \"diff.context\")) {\n> \t\tdiff_context_default = git_config_int(var, value);\n> \t\tif (diff_context_default < 0)\n> @@ -354,6 +371,81 @@ int git_diff_basic_config(const char *var, const char *value, void *cb)\n> \treturn git_default_config(var, value, cb);\n> }\n> \n> +struct moved_entry {\n> +\tstruct hashmap_entry ent;\n> +\tconst struct buffered_patch_line *line;\n> +\tstruct moved_entry *next_line;\n> +};\n> +\n> +static void get_ws_cleaned_string(const struct buffered_patch_line *l,\n> +\t\t\t\t struct strbuf *out)\n> +{\n> +\tint i;\n> +\tfor (i = 0; i < l->len; i++) {\n> +\t\tif (isspace(l->line[i]))\n> +\t\t\tcontinue;\n> +\t\tstrbuf_addch(out, l->line[i]);\n> +\t}\n> +}\n> +\n> +static int buffered_patch_line_cmp_no_ws(const struct buffered_patch_line *a,\n> +\t\t\t\t\t const struct buffered_patch_line *b,\n> +\t\t\t\t\t const void *keydata)\n> +{\n> +\tstruct strbuf sba = STRBUF_INIT;\n> +\tstruct strbuf sbb = STRBUF_INIT;\n> +\tget_ws_cleaned_string(a, &sba);\n> +\tget_ws_cleaned_string(b, &sbb);\n> +\treturn sba.len != sbb.len || strncmp(sba.buf, sbb.buf, sba.len);\n\nYou have a memory leak here, the strbufs need to be released before\nreturning. Also this seems very computationally heavy in that you would\nneed to reconstruct a string every time you perform this comparison.\nMaybe that's ok because the alternative (storing the string w/o\nwhitespace would be too memory intensive).\n\n> +}\n> +\n> +static int buffered_patch_line_cmp(const struct buffered_patch_line *a,\n> +\t\t\t\t const struct buffered_patch_line *b,\n> +\t\t\t\t const void *keydata)\n> +{\n> +\treturn a->len != b->len || strncmp(a->line, b->line, a->len);\n> +}\n> +\n> +static int moved_entry_cmp(const struct moved_entry *a,\n> +\t\t\t const struct moved_entry *b,\n> +\t\t\t const void *keydata)\n> +{\n> +\treturn buffered_patch_line_cmp(a->line, b->line, keydata);\n> +}\n> +\n> +static int moved_entry_cmp_no_ws(const struct moved_entry *a,\n> +\t\t\t\t const struct moved_entry *b,\n> +\t\t\t\t const void *keydata)\n> +{\n> +\treturn buffered_patch_line_cmp_no_ws(a->line, b->line, keydata);\n> +}\n> +\n> +static unsigned get_line_hash(struct buffered_patch_line *line, unsigned ignore_ws)\n> +{\n> +\tstatic struct strbuf sb = STRBUF_INIT;\n> +\n> +\tif (ignore_ws) {\n> +\t\tstrbuf_reset(&sb);\n> +\t\tget_ws_cleaned_string(line, &sb);\n> +\t\treturn memhash(sb.buf, sb.len);\n> +\t} else\n> +\t\treturn memhash(line->line, line->len);\n\nnit: braces on the else block.\nI know other may not agree but I will still point it out when its\nmismatched with a block with braces.\n\n> +}\n> +\n> +static struct moved_entry *prepare_entry(struct diff_options *o,\n> +\t\t\t\t\t int line_no)\n> +{\n> +\tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n> +\tunsigned ignore_ws = DIFF_XDL_TST(o, IGNORE_WHITESPACE);\n> +\tstruct buffered_patch_line *l = &o->line_buffer[line_no];\n> +\n> +\tret->ent.hash = get_line_hash(l, ignore_ws);\n> +\tret->line = l;\n> +\tret->next_line = NULL;\n> +\n> +\treturn ret;\n> +}\n> +\n> static char *quote_two(const char *one, const char *two)\n> {\n> \tint need_one = quote_c_style(one, NULL, NULL, 1);\n> @@ -516,8 +608,98 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n> \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n> }\n> \n> +static void mark_color_as_moved(struct diff_options *o, int line_no)\n> +{\n> +\tstruct hashmap *hm = NULL;\n> +\tstruct moved_entry *key = prepare_entry(o, line_no);\n> +\tstruct moved_entry *match = NULL;\n> +\tstruct buffered_patch_line *l = &o->line_buffer[line_no];\n> +\tint alt_flag;\n> +\tint i, lp, rp;\n> +\n> +\tswitch (l->sign) {\n> +\tcase '+':\n> +\t\thm = o->deleted_lines;\n> +\t\tbreak;\n> +\tcase '-':\n> +\t\thm = o->added_lines;\n> +\t\tbreak;\n> +\tdefault:\n> +\t\t/* reset to standard, on-alt move color */\n> +\t\to->color_moved = 1;\n> +\t\tbreak;\n> +\t}\n> +\n> +\t/* Check for any match to color it as a move. */\n> +\tif (!hm)\n> +\t\treturn;\n> +\tmatch = hashmap_get(hm, key, o);\n> +\tfree(key);\n\nmemory leak? if (!hm), then 'key' is never freed.\n\n> +\tif (!match)\n> +\t\treturn;\n> +\n> +\t/* Check any potential block runs, advance each or nullify */\n> +\tfor (i = 0; i < o->pmb_nr; i++) {\n> +\t\tstruct moved_entry *p = o->pmb[i];\n> +\t\tif (p && p->next_line &&\n> +\t\t !buffered_patch_line_cmp(p->next_line->line, l, o)) {\n> +\t\t\to->pmb[i] = p->next_line;\n> +\t\t} else {\n> +\t\t\to->pmb[i] = NULL;\n> +\t\t}\n> +\t}\n> +\n> +\t/* Shrink the set to the remaining runs */\n> +\tfor (lp = 0, rp = o->pmb_nr - 1; lp <= rp;) {\n> +\t\twhile (lp < o->pmb_nr && o->pmb[lp])\n> +\t\t\tlp ++;\n> +\t\t/* lp points at the first NULL now */\n> +\n> +\t\twhile (rp > -1 && !o->pmb[rp])\n> +\t\t\trp--;\n> +\t\t/* rp points at the last non-NULL */\n> +\n> +\t\tif (lp < o->pmb_nr && rp > -1 && lp < rp) {\n> +\t\t\to->pmb[lp] = o->pmb[rp];\n> +\t\t\to->pmb[rp] = NULL;\n> +\t\t\trp--;\n> +\t\t\tlp++;\n> +\t\t}\n> +\t}\n> +\n> +\tif (rp > -1) {\n> +\t\t/* Remember the number of running sets */\n> +\t\to->pmb_nr = rp + 1;\n> +\t} else {\n> +\t\t/* Toggle color */\n> +\t\to->color_moved = o->color_moved == 2 ? 1 : 2;\n> +\n> +\t\t/* Build up a new set */\n> +\t\ti = 0;\n> +\t\tfor (; match; match = hashmap_get_next(hm, match)) {\n> +\t\t\tALLOC_GROW(o->pmb, i + 1, o->pmb_alloc);\n> +\t\t\to->pmb[i] = match;\n> +\t\t\ti++;\n> +\t\t}\n> +\t\to->pmb_nr = i;\n\nwhy not just use o->pmb_nr directly instead of using 'i'? I was\nimmediately concerned when 'i' was used in the ALLOC_GROW macro.\n\n> +\t}\n> +\n> +\talt_flag = o->color_moved - 1;\n> +\tswitch (l->sign) {\n> +\tcase '+':\n> +\t\tl->set = diff_get_color_opt(o, DIFF_FILE_NEW_MOVED + alt_flag);\n> +\t\tbreak;\n> +\tcase '-':\n> +\t\tl->set = diff_get_color_opt(o, DIFF_FILE_OLD_MOVED + alt_flag);\n> +\t\tbreak;\n> +\tdefault:\n> +\t\t; /* nothing */\n> +\t}\n> +}\n> +\n> static void emit_buffered_patch_line(struct diff_options *o,\n> -\t\t\t\t struct buffered_patch_line *e)\n> +\t\t\t\t struct buffered_patch_line *e,\n> +\t\t\t\t int pass)\n> {\n> \tint has_trailing_newline, has_trailing_carriage_return, len = e->len;\n> \tFILE *file = o->file;\n> @@ -548,11 +730,11 @@ static void emit_buffered_patch_line(struct diff_options *o,\n> \n> static void emit_buffered_patch_line_ws(struct diff_options *o,\n> \t\t\t\t\tstruct buffered_patch_line *e,\n> -\t\t\t\t\tconst char *ws, unsigned ws_rule)\n> +\t\t\t\t\tconst char *ws, unsigned ws_rule,\n> +\t\t\t\t\tint pass)\n> {\n> \tstruct buffered_patch_line s = {e->set, e->reset, \"\", 0, e->sign};\n> -\n> -\temit_buffered_patch_line(o, &s);\n> +\temit_buffered_patch_line(o, &s, 0);\n> \tws_check_emit(e->line, e->len, ws_rule,\n> \t\t o->file, e->set, e->reset, ws);\n> }\n> @@ -564,12 +746,14 @@ static void process_next_buffered_patch_line(struct diff_options *o, int line_no\n> \tconst char *ws = o->current_filepair->ws;\n> \tunsigned ws_rule = o->current_filepair->ws_rule;\n> \n> +\tmark_color_as_moved(o, line_no);\n> +\n> \tswitch (e->state) {\n> \t\tcase BPL_EMIT_LINE_ASIS:\n> -\t\t\temit_buffered_patch_line(o, e);\n> +\t\t\temit_buffered_patch_line(o, e, 1);\n> \t\t\tbreak;\n> \t\tcase BPL_EMIT_LINE_WS:\n> -\t\t\temit_buffered_patch_line_ws(o, e, ws, ws_rule);\n> +\t\t\temit_buffered_patch_line_ws(o, e, ws, ws_rule, 1);\n> \t\t\tbreak;\n> \t\tcase BPL_HANDOVER:\n> \t\t\to->current_filepair++;\n> @@ -602,7 +786,7 @@ static void emit_line_0(struct diff_options *o,\n> \tif (o->use_buffer)\n> \t\tappend_buffered_patch_line(o, &e);\n> \telse\n> -\t\temit_buffered_patch_line(o, &e);\n> +\t\temit_buffered_patch_line(o, &e, 0);\n> }\n> \n> void emit_line(struct diff_options *o, const char *set, const char *reset,\n> @@ -621,7 +805,7 @@ static void emit_line_ws(struct diff_options *o,\n> \tif (o->use_buffer)\n> \t\tappend_buffered_patch_line(o, &e);\n> \telse\n> -\t\temit_buffered_patch_line_ws(o, &e, ws, ws_rule);\n> +\t\temit_buffered_patch_line_ws(o, &e, ws, ws_rule, 0);\n> }\n> \n> void emit_line_fmt(struct diff_options *o,\n> @@ -676,6 +860,36 @@ static void emit_line_checked(const char *reset,\n> \t\t\t ws, ecbdata->ws_rule);\n> }\n> \n> +static void add_line_to_move_detection(struct diff_options *o, int line_idx)\n> +{\n> +\tint sign = 0;\n> +\tstruct hashmap *hm;\n> +\tstruct moved_entry *key;\n> +\n> +\tswitch (o->line_buffer[line_idx].sign) {\n> +\tcase '+':\n> +\t\tsign = '+';\n> +\t\thm = o->added_lines;\n> +\t\tbreak;\n> +\tcase '-':\n> +\t\tsign = '-';\n> +\t\thm = o->deleted_lines;\n> +\t\tbreak;\n> +\tcase ' ':\n> +\tdefault:\n> +\t\to->prev_line = NULL;\n> +\t\treturn;\n> +\t}\n> +\n> +\tkey = prepare_entry(o, line_idx);\n> +\tif (o->prev_line &&\n> +\t o->prev_line->line->sign == sign)\n> +\t\to->prev_line->next_line = key;\n> +\n> +\thashmap_add(hm, key);\n> +\to->prev_line = key;\n\n'key' was freed in another function, does it need to be freed here?\n\n> +}\n> +\n> static void emit_add_line(const char *reset,\n> \t\t\t struct emit_callback *ecbdata,\n> \t\t\t const char *line, int len)\n> @@ -3649,6 +3863,9 @@ void diff_setup_done(struct diff_options *options)\n> \n> \tif (DIFF_OPT_TST(options, FOLLOW_RENAMES) && options->pathspec.nr != 1)\n> \t\tdie(_(\"--follow requires exactly one pathspec\"));\n> +\n> +\tif (!options->use_color || external_diff())\n> +\t\toptions->color_moved = 0;\n> }\n> \n> static int opt_arg(const char *arg, int arg_short, const char *arg_long, int *val)\n> @@ -4073,6 +4290,10 @@ int diff_opt_parse(struct diff_options *options,\n> \t}\n> \telse if (!strcmp(arg, \"--no-color\"))\n> \t\toptions->use_color = 0;\n> +\telse if (!strcmp(arg, \"--color-moved\"))\n> +\t\toptions->color_moved = 1;\n> +\telse if (!strcmp(arg, \"--no-color-moved\"))\n> +\t\toptions->color_moved = 0;\n> \telse if (!strcmp(arg, \"--color-words\")) {\n> \t\toptions->use_color = 1;\n> \t\toptions->word_diff = DIFF_WORDS_COLOR;\n> @@ -4878,16 +5099,19 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n> {\n> \tint i;\n> \tstruct diff_queue_struct *q = &diff_queued_diff;\n> -\t/*\n> -\t * For testing purposes we want to make sure the diff machinery\n> -\t * works completely with the buffer. If there is anything emitted\n> -\t * outside the emit_buffered_patch_line, then the order is screwed\n> -\t * up and the tests will fail.\n> -\t *\n> -\t * TODO (later in this series):\n> -\t * We'll unset this flag in a later patch.\n> -\t */\n> -\to->use_buffer = 1;\n> +\n> +\tif (o->color_moved) {\n> +\t\tunsigned ignore_ws = DIFF_XDL_TST(o, IGNORE_WHITESPACE);\n> +\t\to->use_buffer = 1;\n> +\t\to->deleted_lines = xmallocz(sizeof(*o->deleted_lines));\n> +\t\to->added_lines = xmallocz(sizeof(*o->added_lines));\n> +\t\thashmap_init(o->deleted_lines, ignore_ws ?\n> +\t\t\t(hashmap_cmp_fn)moved_entry_cmp_no_ws :\n> +\t\t\t(hashmap_cmp_fn)moved_entry_cmp, 0);\n> +\t\thashmap_init(o->added_lines, ignore_ws ?\n> +\t\t\t(hashmap_cmp_fn)moved_entry_cmp_no_ws :\n> +\t\t\t(hashmap_cmp_fn)moved_entry_cmp, 0);\n> +\t}\n> \n> \tif (o->use_buffer) {\n> \t\tALLOC_GROW(o->filepair_buffer,\n> @@ -4902,6 +5126,10 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n> \t}\n> \n> \tif (o->use_buffer) {\n> +\t\to->current_filepair = &o->filepair_buffer[0];\n> +\t\tfor (i = 0; i < o->line_buffer_nr; i++)\n> +\t\t\tadd_line_to_move_detection(o, i);\n> +\n> \t\to->current_filepair = &o->filepair_buffer[0];\n> \t\tfor (i = 0; i < o->line_buffer_nr; i++)\n> \t\t\tprocess_next_buffered_patch_line(o, i);\n> @@ -4992,6 +5220,7 @@ void diff_flush(struct diff_options *options)\n> \t\tif (!options->file)\n> \t\t\tdie_errno(\"Could not open /dev/null\");\n> \t\toptions->close_file = 1;\n> +\t\toptions->color_moved = 0;\n> \t\tfor (i = 0; i < q->nr; i++) {\n> \t\t\tstruct diff_filepair *p = q->queue[i];\n> \t\t\tif (check_pair_status(p))\n> diff --git a/diff.h b/diff.h\n> index c334aac02e..b83d6fefcc 100644\n> --- a/diff.h\n> +++ b/diff.h\n> @@ -7,6 +7,7 @@\n> #include \"tree-walk.h\"\n> #include \"pathspec.h\"\n> #include \"object.h\"\n> +#include \"hashmap.h\"\n> \n> struct rev_info;\n> struct diff_options;\n> @@ -145,6 +146,8 @@ struct buffered_filepair {\n> \tunsigned ws_rule;\n> };\n> \n> +struct moved_entry;\n> +\n> struct diff_options {\n> \tconst char *orderfile;\n> \tconst char *pickaxe;\n> @@ -217,6 +220,8 @@ struct diff_options {\n> \n> \tint diff_path_counter;\n> \n> +\t/* Determines color moved code. Flipped between 1, 2 for alt. color. */\n> +\tint color_moved;\n> \tint use_buffer;\n> \n> \tstruct buffered_patch_line *line_buffer;\n> @@ -225,6 +230,16 @@ struct diff_options {\n> \tstruct buffered_filepair *filepair_buffer;\n> \tint filepair_buffer_nr, filepair_buffer_alloc;\n> \tstruct buffered_filepair *current_filepair;\n> +\n> +\t/* built up in the first pass: */\n> +\tstruct hashmap *deleted_lines;\n> +\tstruct hashmap *added_lines;\n> +\t/* needed for building up */\n> +\tstruct moved_entry *prev_line;\n> +\n> +\t/* state in the second pass */\n> +\tstruct moved_entry **pmb; /* potentially moved blocks */\n> +\tint pmb_nr, pmb_alloc;\n> };\n> \n> void emit_line_fmt(struct diff_options *o, const char *set, const char *reset,\n> @@ -241,7 +256,11 @@ enum color_diff {\n> \tDIFF_FILE_NEW = 5,\n> \tDIFF_COMMIT = 6,\n> \tDIFF_WHITESPACE = 7,\n> -\tDIFF_FUNCINFO = 8\n> +\tDIFF_FUNCINFO = 8,\n> +\tDIFF_FILE_OLD_MOVED = 9,\n> +\tDIFF_FILE_OLD_MOVED_ALT = 10,\n> +\tDIFF_FILE_NEW_MOVED = 11,\n> +\tDIFF_FILE_NEW_MOVED_ALT = 12\n> };\n> const char *diff_get_color(int diff_use_color, enum color_diff ix);\n> #define diff_get_color_opt(o, ix) \\\n> diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\n> index 289806d0c7..232d9ad55e 100755\n> --- a/t/t4015-diff-whitespace.sh\n> +++ b/t/t4015-diff-whitespace.sh\n> @@ -972,4 +972,233 @@ test_expect_success 'option overrides diff.wsErrorHighlight' '\n> \n> '\n> \n> +test_expect_success 'detect moved code, complete file' '\n> +\tgit reset --hard &&\n> +\tcat <<-\\EOF >test.c &&\n> +\t#include\n> +\tmain()\n> +\t{\n> +\tprintf(\"Hello World\");\n> +\t}\n> +\tEOF\n> +\tgit add test.c &&\n> +\tgit commit -m \"add main function\" &&\n> +\tgit mv test.c main.c &&\n> +\tgit diff HEAD --color-moved --no-renames | test_decode_color >actual &&\n> +\tcat >expected <<-\\EOF &&\n> +\tdiff --git a/main.c b/main.c\n> +\tnew file mode 100644\n> +\tindex 0000000..a986c57\n> +\t--- /dev/null\n> +\t+++ b/main.c\n> +\t@@ -0,0 +1,5 @@\n> +\t+#include\n> +\t+main()\n> +\t+{\n> +\t+printf(\"Hello World\");\n> +\t+}\n> +\tdiff --git a/test.c b/test.c\n> +\tdeleted file mode 100644\n> +\tindex a986c57..0000000\n> +\t--- a/test.c\n> +\t+++ /dev/null\n> +\t@@ -1,5 +0,0 @@\n> +\t-#include\n> +\t-main()\n> +\t-{\n> +\t-printf(\"Hello World\");\n> +\t-}\n> +\tEOF\n> +\n> +\ttest_cmp expected actual\n> +'\n> +\n> +test_expect_success 'detect moved code, inside file' '\n> +\tgit reset --hard &&\n> +\tcat <<-\\EOF >main.c &&\n> +\t\t#include\n> +\t\tint stuff()\n> +\t\t{\n> +\t\t\tprintf(\"Hello \");\n> +\t\t\tprintf(\"World\\n\");\n> +\t\t}\n> +\n> +\t\tint secure_foo(struct user *u)\n> +\t\t{\n> +\t\t\tif (!u->is_allowed_foo)\n> +\t\t\t\treturn;\n> +\t\t\tfoo(u);\n> +\t\t}\n> +\n> +\t\tint main()\n> +\t\t{\n> +\t\t\tfoo();\n> +\t\t}\n> +\tEOF\n> +\tcat <<-\\EOF >test.c &&\n> +\t\t#include\n> +\t\tint bar()\n> +\t\t{\n> +\t\t\tprintf(\"Hello World, but different\\n\");\n> +\t\t}\n> +\n> +\t\tint another_function()\n> +\t\t{\n> +\t\t\tbar();\n> +\t\t}\n> +\tEOF\n> +\tgit add main.c test.c &&\n> +\tgit commit -m \"add main and test file\" &&\n> +\tcat <<-\\EOF >main.c &&\n> +\t\t#include\n> +\t\tint stuff()\n> +\t\t{\n> +\t\t\tprintf(\"Hello \");\n> +\t\t\tprintf(\"World\\n\");\n> +\t\t}\n> +\n> +\t\tint main()\n> +\t\t{\n> +\t\t\tfoo();\n> +\t\t}\n> +\tEOF\n> +\tcat <<-\\EOF >test.c &&\n> +\t\t#include\n> +\t\tint bar()\n> +\t\t{\n> +\t\t\tprintf(\"Hello World, but different\\n\");\n> +\t\t}\n> +\n> +\t\tint secure_foo(struct user *u)\n> +\t\t{\n> +\t\t\tif (!u->is_allowed_foo)\n> +\t\t\t\treturn;\n> +\t\t\tfoo(u);\n> +\t\t}\n> +\n> +\t\tint another_function()\n> +\t\t{\n> +\t\t\tbar();\n> +\t\t}\n> +\tEOF\n> +\tgit diff HEAD --no-renames --color-moved| test_decode_color >actual &&\n> +\tcat <<-\\EOF >expected &&\n> +\tdiff --git a/main.c b/main.c\n> +\tindex 27a619c..7cf9336 100644\n> +\t--- a/main.c\n> +\t+++ b/main.c\n> +\t@@ -5,13 +5,6 @@ printf(\"Hello \");\n> +\t printf(\"World\\n\");\n> +\t }\n> +\t \n> +\t-int secure_foo(struct user *u)\n> +\t-{\n> +\t-if (!u->is_allowed_foo)\n> +\t-return;\n> +\t-foo(u);\n> +\t-}\n> +\t-\n> +\t int main()\n> +\t {\n> +\t foo();\n> +\tdiff --git a/test.c b/test.c\n> +\tindex 1dc1d85..e34eb69 100644\n> +\t--- a/test.c\n> +\t+++ b/test.c\n> +\t@@ -4,6 +4,13 @@ int bar()\n> +\t printf(\"Hello World, but different\\n\");\n> +\t }\n> +\t \n> +\t+int secure_foo(struct user *u)\n> +\t+{\n> +\t+if (!u->is_allowed_foo)\n> +\t+return;\n> +\t+foo(u);\n> +\t+}\n> +\t+\n> +\t int another_function()\n> +\t {\n> +\t bar();\n> +\tEOF\n> +\n> +\ttest_cmp expected actual\n> +'\n> +\n> +test_expect_success 'detect permutations inside moved code, ' '\n> +\t# reusing the move example from last test:\n> +\tcat <<-\\EOF >main.c &&\n> +\t\t#include\n> +\t\tint stuff()\n> +\t\t{\n> +\t\t\tprintf(\"Hello \");\n> +\t\t\tprintf(\"World\\n\");\n> +\t\t}\n> +\n> +\t\tint main()\n> +\t\t{\n> +\t\t\tfoo();\n> +\t\t}\n> +\tEOF\n> +\tcat <<-\\EOF >test.c &&\n> +\t\t#include\n> +\t\tint bar()\n> +\t\t{\n> +\t\t\tprintf(\"Hello World, but different\\n\");\n> +\t\t}\n> +\n> +\t\tint secure_foo(struct user *u)\n> +\t\t{\n> +\t\t\tfoo(u);\n> +\t\t\tif (!u->is_allowed_foo)\n> +\t\t\t\treturn;\n> +\t\t}\n> +\n> +\t\tint another_function()\n> +\t\t{\n> +\t\t\tbar();\n> +\t\t}\n> +\tEOF\n> +\tgit diff HEAD --no-renames --color-moved| test_decode_color >actual &&\n> +\tcat <<-\\EOF >expected &&\n> +\tdiff --git a/main.c b/main.c\n> +\tindex 27a619c..7cf9336 100644\n> +\t--- a/main.c\n> +\t+++ b/main.c\n> +\t@@ -5,13 +5,6 @@ printf(\"Hello \");\n> +\t printf(\"World\\n\");\n> +\t }\n> +\t \n> +\t-int secure_foo(struct user *u)\n> +\t-{\n> +\t-if (!u->is_allowed_foo)\n> +\t-return;\n> +\t-foo(u);\n> +\t-}\n> +\t-\n> +\t int main()\n> +\t {\n> +\t foo();\n> +\tdiff --git a/test.c b/test.c\n> +\tindex 1dc1d85..2bedec9 100644\n> +\t--- a/test.c\n> +\t+++ b/test.c\n> +\t@@ -4,6 +4,13 @@ int bar()\n> +\t printf(\"Hello World, but different\\n\");\n> +\t }\n> +\t \n> +\t+int secure_foo(struct user *u)\n> +\t+{\n> +\t+foo(u);\n> +\t+if (!u->is_allowed_foo)\n> +\t+return;\n> +\t+}\n> +\t+\n> +\t int another_function()\n> +\t {\n> +\t bar();\n> +\tEOF\n> +\n> +\ttest_cmp expected actual\n> +'\n> +\n> test_done\n> -- \n> 2.13.0.18.g183880de0a\n> \n\n-- \nBrandon Williams\n"},{"id":"319851","messageId":"CAGZ79kYx18HZGOLfN7jGzoG0DAEb7O+=L=7uZ07fDsNRK0ic_g@mail.gmail.com","threadId":"45958","inReplyTo":"f49dc4f8-cc83-934f-e582-79efdc613f70@google.com","subject":"Re: [PATCH 14/19] diff.c: convert word diffing to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-15T23:12:30Z","receivedAt":"2017-05-15T23:12:36Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, May 15, 2017 at 3:40 PM, Jonathan Tan wrote:\n>\n> I suspect that this will need to be refactored more thoroughly. Here, for\n> example, emit_line (which prints the prefix) is printed nearly\n> unconditionally, whereas in the original version, \"fputs(line_prefix, fp)\"\n> is only printed when \"print\" is true.\n\nYes, manual testing confirms this is the case. Maybe I should to add a test\nfor \"git diff --word-diff --line-prefix\"\n\nThanks,\nStefan\n"},{"id":"319858","messageId":"xmqqtw4l4nuk.fsf@gitster.mtv.corp.google.com","threadId":"45958","inReplyTo":"20170514040117.25865-8-sbeller@google.com","subject":"Re: [PATCH 07/19] diff.c: convert fn_out_consume to use emit_line_*","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-16T01:00:51Z","receivedAt":"2017-05-16T01:01:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> In a later patch, I want to propose an option to detect&color\n> moved lines in a diff, which cannot be done in a one-pass over\n> the diff. Instead we need to go over the whole diff twice,\n> because we cannot detect the first line of the two corresponding\n> lines (+ and -) that got moved.\n>\n> So to prepare the diff machinery for two pass algorithms\n> (i.e. buffer it all up and then operate on the result),\n> move all emissions to places, such that the only emitting\n> function is emit_line_0.\n>\n> This covers the parts of fn_out_consume.\n>\n> Signed-off-by: Stefan Beller \n> ---\n> diff.c | 13 +++++--------\n> 1 file changed, 5 insertions(+), 8 deletions(-)\n>\n> diff --git a/diff.c b/diff.c\n> index aef159a919..93343a9ccc 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -1289,7 +1289,6 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n> \tconst char *context = diff_get_color(ecbdata->color_diff, DIFF_CONTEXT);\n> \tconst char *reset = diff_get_color(ecbdata->color_diff, DIFF_RESET);\n> \tstruct diff_options *o = ecbdata->opt;\n> -\tconst char *line_prefix = diff_line_prefix(o);\n> \n> \to->found_changes = 1;\n> \n> @@ -1301,14 +1300,12 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n> \n> \tif (ecbdata->label_path[0]) {\n> \t\tconst char *name_a_tab, *name_b_tab;\n> -\n> \t\tname_a_tab = strchr(ecbdata->label_path[0], ' ') ? \"\\t\" : \"\";\n> \t\tname_b_tab = strchr(ecbdata->label_path[1], ' ') ? \"\\t\" : \"\";\n> -\n> -\t\tfprintf(o->file, \"%s%s--- %s%s%s\\n\",\n> -\t\t\tline_prefix, meta, ecbdata->label_path[0], reset, name_a_tab);\n> -\t\tfprintf(o->file, \"%s%s+++ %s%s%s\\n\",\n> -\t\t\tline_prefix, meta, ecbdata->label_path[1], reset, name_b_tab);\n> +\t\temit_line_fmt(o, meta, reset, \"--- %s%s\\n\",\n> +\t\t\t ecbdata->label_path[0], name_a_tab);\n> +\t\temit_line_fmt(o, meta, reset, \"+++ %s%s\\n\",\n> +\t\t\t ecbdata->label_path[1], name_b_tab);\n\nHow is the loss of line_prefix from this call site compensated?\nemit_line_fmt() receives o so it is possible diff_line_prefix(o)\nmay be called there and prepended to the output over there, but I\nsomehow do not think that is the case---in fact 06/19 does not seem\nto teach emit_line_fmt() to do something like that.\n\nUnless emit_line() is now taught to do the line_prefix thing, that\nis.\n\nBut then the hunk we see below, which didn't add line_prefix to the\noutput, would add an unwanted prefix, so that is not likely.\n\nHmph...\n\n> \t\tecbdata->label_path[0] = ecbdata->label_path[1] = NULL;\n> \t}\n> \n> @@ -1349,7 +1346,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n> \t\tdiff_words_flush(ecbdata);\n> \t\tif (ecbdata->diff_words->type == DIFF_WORDS_PORCELAIN) {\n> \t\t\temit_line(o, context, reset, line, len);\n> -\t\t\tfputs(\"~\\n\", o->file);\n> +\t\t\temit_line(o, NULL, NULL, \"~\\n\", 2);\n> \t\t} else {\n> \t\t\t/*\n> \t\t\t * Skip the prefix character, if any. With\n"},{"id":"319859","messageId":"xmqqpof94nn2.fsf@gitster.mtv.corp.google.com","threadId":"45958","inReplyTo":"xmqqtw4l4nuk.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 07/19] diff.c: convert fn_out_consume to use emit_line_*","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-16T01:05:21Z","receivedAt":"2017-05-16T01:05:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> -\t\tfprintf(o->file, \"%s%s--- %s%s%s\\n\",\n>> -\t\t\tline_prefix, meta, ecbdata->label_path[0], reset, name_a_tab);\n>> -\t\tfprintf(o->file, \"%s%s+++ %s%s%s\\n\",\n>> -\t\t\tline_prefix, meta, ecbdata->label_path[1], reset, name_b_tab);\n>> +\t\temit_line_fmt(o, meta, reset, \"--- %s%s\\n\",\n>> +\t\t\t ecbdata->label_path[0], name_a_tab);\n>> +\t\temit_line_fmt(o, meta, reset, \"+++ %s%s\\n\",\n>> +\t\t\t ecbdata->label_path[1], name_b_tab);\n>\n> How is the loss of line_prefix from this call site compensated?\n\nOK, emit_line_0() has already been aware of line_prefix, so that is\nhow the loss of line_prefix in the above is accounted for. We are\ngood here.\n\n>> \t\tecbdata->label_path[0] = ecbdata->label_path[1] = NULL;\n>> \t}\n>> \n>> @@ -1349,7 +1346,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n>> \t\tdiff_words_flush(ecbdata);\n>> \t\tif (ecbdata->diff_words->type == DIFF_WORDS_PORCELAIN) {\n>> \t\t\temit_line(o, context, reset, line, len);\n>> -\t\t\tfputs(\"~\\n\", o->file);\n>> +\t\t\temit_line(o, NULL, NULL, \"~\\n\", 2);\n\nSo unless we have some magic here, we would see an extra line-prefix\nbefore that \"~\\n\" thing, no?\n\n\n>> \t\t} else {\n>> \t\t\t/*\n>> \t\t\t * Skip the prefix character, if any. With\n"},{"id":"319884","messageId":"CAGf8dgJP+i5RL3FaGSYZyVKkt1ttSnnPd924ebs=4xJb4Fhc6w@mail.gmail.com","threadId":"45958","inReplyTo":"20170514040117.25865-19-sbeller@google.com","subject":"Re: [PATCH 18/19] diff: buffer all output if asked to","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-05-16T04:14:28Z","receivedAt":"2017-05-16T04:14:37Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Overall, this patch seems larger than it should to me, although there\nmight be good reasons for that that I don't know. I'll remark on what\nI find unexpected.\n\nOn Sat, May 13, 2017 at 9:01 PM, Stefan Beller wrote:\n> diff --git a/diff.c b/diff.c\n> index 08dcc56bb9..dbab7fb44e 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -516,29 +516,29 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n> ecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n> }\n>\n> -static void emit_line_0(struct diff_options *o, const char *set, const char *reset,\n> - int sign, const char *line, int len)\n> +static void emit_buffered_patch_line(struct diff_options *o,\n> + struct buffered_patch_line *e)\n> {\n> - int has_trailing_newline, has_trailing_carriage_return;\n> + int has_trailing_newline, has_trailing_carriage_return, len = e->len;\n> FILE *file = o->file;\n>\n> fputs(diff_line_prefix(o), file);\n>\n> - has_trailing_newline = (len > 0 && line[len-1] == '\\n');\n> + has_trailing_newline = (len > 0 && e->line[len-1] == '\\n');\n> if (has_trailing_newline)\n> len--;\n> - has_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n> + has_trailing_carriage_return = (len > 0 && e->line[len-1] == '\\r');\n> if (has_trailing_carriage_return)\n> len--;\n>\n> - if (len || sign) {\n> - if (set)\n> - fputs(set, file);\n> - if (sign)\n> - fputc(sign, file);\n> - fwrite(line, len, 1, file);\n> - if (reset)\n> - fputs(reset, file);\n> + if (len || e->sign) {\n> + if (e->set)\n> + fputs(e->set, file);\n> + if (e->sign)\n> + fputc(e->sign, file);\n> + fwrite(e->line, len, 1, file);\n> + if (e->reset)\n> + fputs(e->reset, file);\n> }\n> if (has_trailing_carriage_return)\n> fputc('\\r', file);\n> @@ -546,6 +546,65 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n> fputc('\\n', file);\n> }\n>\n> +static void emit_buffered_patch_line_ws(struct diff_options *o,\n> + struct buffered_patch_line *e,\n> + const char *ws, unsigned ws_rule)\n\nThis introduces a new _ws emission function - how is this used and how\nis this different from the non-ws one? I see BPL_EMIT_LINE_WS, but I\ndon't see the caller that introduces that constant in this patch.\n\n> +{\n> + struct buffered_patch_line s = {e->set, e->reset, \"\", 0, e->sign};\n> +\n> + emit_buffered_patch_line(o, &s);\n> + ws_check_emit(e->line, e->len, ws_rule,\n> + o->file, e->set, e->reset, ws);\n> +}\n> +\n> +static void process_next_buffered_patch_line(struct diff_options *o, int line_no)\n> +{\n> + struct buffered_patch_line *e = &o->line_buffer[line_no];\n> +\n> + const char *ws = o->current_filepair->ws;\n> + unsigned ws_rule = o->current_filepair->ws_rule;\n> +\n> + switch (e->state) {\n> + case BPL_EMIT_LINE_ASIS:\n> + emit_buffered_patch_line(o, e);\n> + break;\n> + case BPL_EMIT_LINE_WS:\n> + emit_buffered_patch_line_ws(o, e, ws, ws_rule);\n> + break;\n> + case BPL_HANDOVER:\n> + o->current_filepair++;\n\nIf we're just buffering the diff output, do we need to store\nper-file-pair metadata? (I assume that's why you need a special\nhandover constant.) Clients can already read what they need from the\ndiff output.\n\n> + break;\n> + default:\n> + die(\"BUG: malformatted buffered patch line: '%d'\", e->state);\n> + }\n> +}\n> +\n> +static void append_buffered_patch_line(struct diff_options *o,\n> + struct buffered_patch_line *e)\n> +{\n> + struct buffered_patch_line *f;\n> + ALLOC_GROW(o->line_buffer,\n> + o->line_buffer_nr + 1,\n> + o->line_buffer_alloc);\n> + f = &o->line_buffer[o->line_buffer_nr];\n> + o->line_buffer_nr++;\n> +\n> + memcpy(f, e, sizeof(struct buffered_patch_line));\n> + f->line = e->line ? xmemdupz(e->line, e->len) : NULL;\n> +}\n> +\n> +static void emit_line_0(struct diff_options *o,\n> + const char *set, const char *reset,\n> + int sign, const char *line, int len)\n> +{\n> + struct buffered_patch_line e = {set, reset, line, len, sign, BPL_EMIT_LINE_ASIS};\n> +\n> + if (o->use_buffer)\n> + append_buffered_patch_line(o, &e);\n> + else\n> + emit_buffered_patch_line(o, &e);\n> +}\n> +\n> void emit_line(struct diff_options *o, const char *set, const char *reset,\n> const char *line, int len)\n> {\n> @@ -557,9 +616,12 @@ static void emit_line_ws(struct diff_options *o,\n> const char *line, int len,\n> const char *ws, unsigned ws_rule)\n> {\n> - emit_line_0(o, set, reset, sign, \"\", 0);\n> - ws_check_emit(line, len, ws_rule,\n> - o->file, set, reset, ws);\n> + struct buffered_patch_line e = {set, reset, line, len, sign, BPL_EMIT_LINE_WS};\n> +\n> + if (o->use_buffer)\n> + append_buffered_patch_line(o, &e);\n> + else\n> + emit_buffered_patch_line_ws(o, &e, ws, ws_rule);\n> }\n>\n> void emit_line_fmt(struct diff_options *o,\n> @@ -1160,6 +1222,16 @@ static void diff_words_flush(struct emit_callback *ecbdata)\n> if (ecbdata->diff_words->minus.text.size ||\n> ecbdata->diff_words->plus.text.size)\n> diff_words_show(ecbdata->diff_words);\n> +\n> + if (ecbdata->diff_words->opt->line_buffer_nr) {\n> + int i;\n> + for (i = 0; i < ecbdata->diff_words->opt->line_buffer_nr; i++)\n> + append_buffered_patch_line(ecbdata->opt,\n> + &ecbdata->diff_words->opt->line_buffer[i]);\n> +\n> + ecbdata->diff_words->opt->line_buffer_nr = 0;\n> + /* TODO: free memory as well */\n> + }\n> }\n>\n> static void diff_filespec_load_driver(struct diff_filespec *one)\n> @@ -1195,6 +1267,11 @@ static void init_diff_words_data(struct emit_callback *ecbdata,\n> xcalloc(1, sizeof(struct diff_words_data));\n> ecbdata->diff_words->type = o->word_diff;\n> ecbdata->diff_words->opt = o;\n> +\n> + o->line_buffer = NULL;\n> + o->line_buffer_nr = 0;\n> + o->line_buffer_alloc = 0;\n> +\n> if (!o->word_regex)\n> o->word_regex = userdiff_word_regex(one);\n> if (!o->word_regex)\n> @@ -2568,9 +2645,25 @@ static void builtin_diff(const char *name_a,\n> xecfg.ctxlen = strtoul(v, NULL, 10);\n> if (o->word_diff)\n> init_diff_words_data(&ecbdata, o, one, two);\n> + if (o->use_buffer) {\n> + ALLOC_GROW(o->filepair_buffer,\n> + o->filepair_buffer_nr + 1,\n> + o->filepair_buffer_alloc);\n> + o->current_filepair =\n> + &o->filepair_buffer[o->filepair_buffer_nr++];\n> +\n> + o->current_filepair->ws_rule = ecbdata.ws_rule;\n> + o->current_filepair->ws =\n> + diff_get_color(ecbdata.color_diff, DIFF_WHITESPACE);\n> + }\n> if (xdi_diff_outf(&mf1, &mf2, fn_out_consume, &ecbdata,\n> &xpp, &xecfg))\n> die(\"unable to generate diff for %s\", one->path);\n> + if (o->use_buffer) {\n> + struct buffered_patch_line e = BUFFERED_PATCH_LINE_INIT;\n> + e.state = BPL_HANDOVER; /* handover to next file pair */\n> + append_buffered_patch_line(o, &e);\n> + }\n> if (o->word_diff)\n> free_diff_words_data(&ecbdata);\n> if (textconv_one)\n> @@ -4785,11 +4878,44 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n> {\n> int i;\n> struct diff_queue_struct *q = &diff_queued_diff;\n> + /*\n> + * For testing purposes we want to make sure the diff machinery\n> + * works completely with the buffer. If there is anything emitted\n> + * outside the emit_buffered_patch_line, then the order is screwed\n> + * up and the tests will fail.\n> + *\n> + * TODO (later in this series):\n> + * We'll unset this flag in a later patch.\n> + */\n> + o->use_buffer = 1;\n\nWhat I would do is to add a demonstration patch at the end of the\npatch series (which is not supposed to be queued) to avoid such churn\nin history, but I'm not sure how the Git project prefers to do this.\n\n> +\n> + if (o->use_buffer) {\n> + ALLOC_GROW(o->filepair_buffer,\n> + o->filepair_buffer_nr + 1,\n> + o->filepair_buffer_alloc);\n> + o->current_filepair = &o->filepair_buffer[o->filepair_buffer_nr];\n> + }\n> for (i = 0; i < q->nr; i++) {\n> struct diff_filepair *p = q->queue[i];\n> if (check_pair_status(p))\n> diff_flush_patch(p, o);\n> }\n> +\n> + if (o->use_buffer) {\n> + o->current_filepair = &o->filepair_buffer[0];\n> + for (i = 0; i < o->line_buffer_nr; i++)\n> + process_next_buffered_patch_line(o, i);\n> +\n> + for (i = 0; i < o->line_buffer_nr; i++);\n> + free((void*)o->line_buffer[i].line);\n> +\n> + o->line_buffer = NULL;\n> + o->line_buffer_nr = 0;\n> + free(o->line_buffer);\n> + o->filepair_buffer = NULL;\n> + o->filepair_buffer_nr = 0;\n> + free(o->filepair_buffer);\n> + }\n> }\n>\n> void diff_flush(struct diff_options *options)\n> diff --git a/diff.h b/diff.h\n> index 5e89481769..c334aac02e 100644\n> --- a/diff.h\n> +++ b/diff.h\n> @@ -115,6 +115,36 @@ enum diff_submodule_format {\n> DIFF_SUBMODULE_INLINE_DIFF\n> };\n>\n> +/*\n> + * This struct is used when we need to buffer the output of the diff output.\n> + *\n> + * NEEDSWORK: Instead of storing a copy of the line, add an offset pointer\n> + * into the pre/post image file. This pointer could be a union with the\n> + * line pointer. By storing an offset into the file instead of the literal line,\n> + * we can decrease the memory footprint for the buffered output. At first we\n> + * may want to only have indirection for the content lines, but we could\n> + * also have an enum (based on sign?) that stores prefabricated lines, e.g.\n> + * the similarity score line or hunk/file headers.\n\nThis would be nice, but come to think of it, might not be possible.\nWhen requesting --word-diff, control characters (or others) might\nappear in the output, right?\n\n> + */\n> +struct buffered_patch_line {\n> + const char *set;\n> + const char *reset;\n> + const char *line;\n> + int len;\n> + int sign;\n> + enum {\n> + BPL_EMIT_LINE_WS,\n> + BPL_EMIT_LINE_ASIS,\n> + BPL_HANDOVER\n> + } state;\n\nIt might be better, for simplicity, just to have one big buffer\nincluding everything (if we decide that we really can't add pointers\nto input later).\n\n> +};\n> +#define BUFFERED_PATCH_LINE_INIT {NULL, NULL, NULL, 0, 0, 0}\n> +\n> +struct buffered_filepair {\n> + const char *ws;\n> + unsigned ws_rule;\n> +};\n> +\n> struct diff_options {\n> const char *orderfile;\n> const char *pickaxe;\n> @@ -186,6 +216,15 @@ struct diff_options {\n> void *output_prefix_data;\n>\n> int diff_path_counter;\n> +\n> + int use_buffer;\n> +\n> + struct buffered_patch_line *line_buffer;\n> + int line_buffer_nr, line_buffer_alloc;\n> +\n> + struct buffered_filepair *filepair_buffer;\n> + int filepair_buffer_nr, filepair_buffer_alloc;\n> + struct buffered_filepair *current_filepair;\n> };\n>\n> void emit_line_fmt(struct diff_options *o, const char *set, const char *reset,\n> --\n> 2.13.0.18.g183880de0a\n>\n"},{"id":"319885","messageId":"CAGf8dgLuJLn=X4gmVGQ0tfTo2=py4XhBw-WSYtw2acy9x3NSJQ@mail.gmail.com","threadId":"45958","inReplyTo":"20170514040117.25865-20-sbeller@google.com","subject":"Re: [PATCH 19/19] diff.c: color moved lines differently","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-05-16T04:34:04Z","receivedAt":"2017-05-16T04:34:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"I expected there to be one main function that takes in a diff options\nand returns the appropriate output without much (if any) changes in\nother functions...but (as with the previous patch) maybe there are\nsome complications that I didn't foresee.\n\nOn Sat, May 13, 2017 at 9:01 PM, Stefan Beller wrote:\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 475e874d51..90403c06e3 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -1051,14 +1051,22 @@ This does not affect linkgit:git-format-patch[1] or the\n> 'git-diff-{asterisk}' plumbing commands. Can be overridden on the\n> command line with the `--color[=]` option.\n>\n> +color.moved::\n> + A boolean value, whether a diff should color moved lines\n> + differently. The moved lines are searched for in the diff only.\n> + Duplicated lines from somewhere in the project that are not\n> + part of the diff are not colored as moved.\n> + Defaults to false.\n> +\n> color.diff.::\n> Use customized color for diff colorization. `` specifies\n> which part of the patch to use the specified color, and is one\n> of `context` (context text - `plain` is a historical synonym),\n> `meta` (metainformation), `frag`\n> (hunk header), 'func' (function in hunk header), `old` (removed lines),\n> - `new` (added lines), `commit` (commit headers), or `whitespace`\n> - (highlighting whitespace errors).\n> + `new` (added lines), `commit` (commit headers), `whitespace`\n> + (highlighting whitespace errors), `movedFrom` (removed lines that\n> + reappear), `movedTo` (added lines that were removed elsewhere).\n\nThere should be 4 \"moved\" colors. I think the code below is correct\n(oldmoved, oldmovedalternate, etc.) but the documentation above is\nwrong.\n\n>\n> color.decorate.::\n> Use customized color for 'git log --decorate' output. `` is one\n> diff --git a/diff.c b/diff.c\n> index dbab7fb44e..6372e0eb25 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -31,6 +31,7 @@ static int diff_indent_heuristic; /* experimental */\n> static int diff_rename_limit_default = 400;\n> static int diff_suppress_blank_empty;\n> static int diff_use_color_default = -1;\n> +static int diff_color_moved_default;\n> static int diff_context_default = 3;\n> static int diff_interhunk_context_default;\n> static const char *diff_word_regex_cfg;\n> @@ -55,6 +56,10 @@ static char diff_colors[][COLOR_MAXLEN] = {\n> GIT_COLOR_YELLOW, /* COMMIT */\n> GIT_COLOR_BG_RED, /* WHITESPACE */\n> GIT_COLOR_NORMAL, /* FUNCINFO */\n> + GIT_COLOR_BOLD_RED, /* OLD_MOVED_A */\n> + GIT_COLOR_BG_RED, /* OLD_MOVED_B */\n> + GIT_COLOR_BOLD_GREEN, /* NEW_MOVED_A */\n> + GIT_COLOR_BG_GREEN, /* NEW_MOVED_B */\n> };\n>\n> static NORETURN void die_want_option(const char *option_name)\n> @@ -80,6 +85,14 @@ static int parse_diff_color_slot(const char *var)\n> return DIFF_WHITESPACE;\n> if (!strcasecmp(var, \"func\"))\n> return DIFF_FUNCINFO;\n> + if (!strcasecmp(var, \"oldmoved\"))\n> + return DIFF_FILE_OLD_MOVED;\n> + if (!strcasecmp(var, \"oldmovedalternative\"))\n> + return DIFF_FILE_OLD_MOVED_ALT;\n> + if (!strcasecmp(var, \"newmoved\"))\n> + return DIFF_FILE_NEW_MOVED;\n> + if (!strcasecmp(var, \"newmovedalternative\"))\n> + return DIFF_FILE_NEW_MOVED_ALT;\n> return -1;\n> }\n>\n> @@ -234,6 +247,10 @@ int git_diff_ui_config(const char *var, const char *value, void *cb)\n> diff_use_color_default = git_config_colorbool(var, value);\n> return 0;\n> }\n> + if (!strcmp(var, \"color.moved\")) {\n> + diff_color_moved_default = git_config_bool(var, value);\n> + return 0;\n> + }\n> if (!strcmp(var, \"diff.context\")) {\n> diff_context_default = git_config_int(var, value);\n> if (diff_context_default < 0)\n> @@ -354,6 +371,81 @@ int git_diff_basic_config(const char *var, const char *value, void *cb)\n> return git_default_config(var, value, cb);\n> }\n>\n> +struct moved_entry {\n> + struct hashmap_entry ent;\n> + const struct buffered_patch_line *line;\n> + struct moved_entry *next_line;\n> +};\n> +\n> +static void get_ws_cleaned_string(const struct buffered_patch_line *l,\n> + struct strbuf *out)\n> +{\n> + int i;\n> + for (i = 0; i < l->len; i++) {\n> + if (isspace(l->line[i]))\n> + continue;\n> + strbuf_addch(out, l->line[i]);\n> + }\n> +}\n> +\n> +static int buffered_patch_line_cmp_no_ws(const struct buffered_patch_line *a,\n> + const struct buffered_patch_line *b,\n> + const void *keydata)\n> +{\n> + struct strbuf sba = STRBUF_INIT;\n> + struct strbuf sbb = STRBUF_INIT;\n> + get_ws_cleaned_string(a, &sba);\n> + get_ws_cleaned_string(b, &sbb);\n> + return sba.len != sbb.len || strncmp(sba.buf, sbb.buf, sba.len);\n> +}\n> +\n> +static int buffered_patch_line_cmp(const struct buffered_patch_line *a,\n> + const struct buffered_patch_line *b,\n> + const void *keydata)\n> +{\n> + return a->len != b->len || strncmp(a->line, b->line, a->len);\n> +}\n> +\n> +static int moved_entry_cmp(const struct moved_entry *a,\n> + const struct moved_entry *b,\n> + const void *keydata)\n> +{\n> + return buffered_patch_line_cmp(a->line, b->line, keydata);\n> +}\n> +\n> +static int moved_entry_cmp_no_ws(const struct moved_entry *a,\n> + const struct moved_entry *b,\n> + const void *keydata)\n> +{\n> + return buffered_patch_line_cmp_no_ws(a->line, b->line, keydata);\n> +}\n> +\n> +static unsigned get_line_hash(struct buffered_patch_line *line, unsigned ignore_ws)\n> +{\n> + static struct strbuf sb = STRBUF_INIT;\n> +\n> + if (ignore_ws) {\n> + strbuf_reset(&sb);\n> + get_ws_cleaned_string(line, &sb);\n> + return memhash(sb.buf, sb.len);\n> + } else\n> + return memhash(line->line, line->len);\n> +}\n> +\n> +static struct moved_entry *prepare_entry(struct diff_options *o,\n> + int line_no)\n> +{\n> + struct moved_entry *ret = xmalloc(sizeof(*ret));\n> + unsigned ignore_ws = DIFF_XDL_TST(o, IGNORE_WHITESPACE);\n> + struct buffered_patch_line *l = &o->line_buffer[line_no];\n> +\n> + ret->ent.hash = get_line_hash(l, ignore_ws);\n> + ret->line = l;\n> + ret->next_line = NULL;\n> +\n> + return ret;\n> +}\n> +\n> static char *quote_two(const char *one, const char *two)\n> {\n> int need_one = quote_c_style(one, NULL, NULL, 1);\n> @@ -516,8 +608,98 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n> ecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n> }\n>\n> +static void mark_color_as_moved(struct diff_options *o, int line_no)\n> +{\n> + struct hashmap *hm = NULL;\n> + struct moved_entry *key = prepare_entry(o, line_no);\n> + struct moved_entry *match = NULL;\n> + struct buffered_patch_line *l = &o->line_buffer[line_no];\n> + int alt_flag;\n> + int i, lp, rp;\n> +\n> + switch (l->sign) {\n> + case '+':\n> + hm = o->deleted_lines;\n> + break;\n> + case '-':\n> + hm = o->added_lines;\n> + break;\n> + default:\n> + /* reset to standard, on-alt move color */\n> + o->color_moved = 1;\n> + break;\n> + }\n> +\n> + /* Check for any match to color it as a move. */\n> + if (!hm)\n> + return;\n> + match = hashmap_get(hm, key, o);\n> + free(key);\n> + if (!match)\n> + return;\n> +\n> + /* Check any potential block runs, advance each or nullify */\n> + for (i = 0; i < o->pmb_nr; i++) {\n> + struct moved_entry *p = o->pmb[i];\n> + if (p && p->next_line &&\n> + !buffered_patch_line_cmp(p->next_line->line, l, o)) {\n> + o->pmb[i] = p->next_line;\n> + } else {\n> + o->pmb[i] = NULL;\n> + }\n> + }\n> +\n> + /* Shrink the set to the remaining runs */\n> + for (lp = 0, rp = o->pmb_nr - 1; lp <= rp;) {\n> + while (lp < o->pmb_nr && o->pmb[lp])\n> + lp ++;\n> + /* lp points at the first NULL now */\n> +\n> + while (rp > -1 && !o->pmb[rp])\n> + rp--;\n> + /* rp points at the last non-NULL */\n> +\n> + if (lp < o->pmb_nr && rp > -1 && lp < rp) {\n> + o->pmb[lp] = o->pmb[rp];\n> + o->pmb[rp] = NULL;\n> + rp--;\n> + lp++;\n> + }\n> + }\n> +\n> + if (rp > -1) {\n> + /* Remember the number of running sets */\n> + o->pmb_nr = rp + 1;\n> + } else {\n> + /* Toggle color */\n> + o->color_moved = o->color_moved == 2 ? 1 : 2;\n> +\n> + /* Build up a new set */\n> + i = 0;\n> + for (; match; match = hashmap_get_next(hm, match)) {\n> + ALLOC_GROW(o->pmb, i + 1, o->pmb_alloc);\n> + o->pmb[i] = match;\n> + i++;\n> + }\n> + o->pmb_nr = i;\n> + }\n> +\n> + alt_flag = o->color_moved - 1;\n> + switch (l->sign) {\n> + case '+':\n> + l->set = diff_get_color_opt(o, DIFF_FILE_NEW_MOVED + alt_flag);\n> + break;\n> + case '-':\n> + l->set = diff_get_color_opt(o, DIFF_FILE_OLD_MOVED + alt_flag);\n> + break;\n> + default:\n> + ; /* nothing */\n> + }\n> +}\n> +\n> static void emit_buffered_patch_line(struct diff_options *o,\n> - struct buffered_patch_line *e)\n> + struct buffered_patch_line *e,\n> + int pass)\n\n1. I didn't expect such a function to need to be modified in this patch.\n2. What does \"pass\" do?\n\n> {\n> int has_trailing_newline, has_trailing_carriage_return, len = e->len;\n> FILE *file = o->file;\n> @@ -548,11 +730,11 @@ static void emit_buffered_patch_line(struct diff_options *o,\n>\n> static void emit_buffered_patch_line_ws(struct diff_options *o,\n> struct buffered_patch_line *e,\n> - const char *ws, unsigned ws_rule)\n> + const char *ws, unsigned ws_rule,\n> + int pass)\n> {\n> struct buffered_patch_line s = {e->set, e->reset, \"\", 0, e->sign};\n> -\n> - emit_buffered_patch_line(o, &s);\n> + emit_buffered_patch_line(o, &s, 0);\n> ws_check_emit(e->line, e->len, ws_rule,\n> o->file, e->set, e->reset, ws);\n> }\n> @@ -564,12 +746,14 @@ static void process_next_buffered_patch_line(struct diff_options *o, int line_no\n> const char *ws = o->current_filepair->ws;\n> unsigned ws_rule = o->current_filepair->ws_rule;\n>\n> + mark_color_as_moved(o, line_no);\n> +\n> switch (e->state) {\n> case BPL_EMIT_LINE_ASIS:\n> - emit_buffered_patch_line(o, e);\n> + emit_buffered_patch_line(o, e, 1);\n> break;\n> case BPL_EMIT_LINE_WS:\n> - emit_buffered_patch_line_ws(o, e, ws, ws_rule);\n> + emit_buffered_patch_line_ws(o, e, ws, ws_rule, 1);\n> break;\n> case BPL_HANDOVER:\n> o->current_filepair++;\n> @@ -602,7 +786,7 @@ static void emit_line_0(struct diff_options *o,\n> if (o->use_buffer)\n> append_buffered_patch_line(o, &e);\n> else\n> - emit_buffered_patch_line(o, &e);\n> + emit_buffered_patch_line(o, &e, 0);\n> }\n>\n> void emit_line(struct diff_options *o, const char *set, const char *reset,\n> @@ -621,7 +805,7 @@ static void emit_line_ws(struct diff_options *o,\n> if (o->use_buffer)\n> append_buffered_patch_line(o, &e);\n> else\n> - emit_buffered_patch_line_ws(o, &e, ws, ws_rule);\n> + emit_buffered_patch_line_ws(o, &e, ws, ws_rule, 0);\n> }\n>\n> void emit_line_fmt(struct diff_options *o,\n> @@ -676,6 +860,36 @@ static void emit_line_checked(const char *reset,\n> ws, ecbdata->ws_rule);\n> }\n>\n> +static void add_line_to_move_detection(struct diff_options *o, int line_idx)\n> +{\n> + int sign = 0;\n> + struct hashmap *hm;\n> + struct moved_entry *key;\n> +\n> + switch (o->line_buffer[line_idx].sign) {\n> + case '+':\n> + sign = '+';\n> + hm = o->added_lines;\n> + break;\n> + case '-':\n> + sign = '-';\n> + hm = o->deleted_lines;\n> + break;\n> + case ' ':\n> + default:\n> + o->prev_line = NULL;\n> + return;\n> + }\n> +\n> + key = prepare_entry(o, line_idx);\n> + if (o->prev_line &&\n> + o->prev_line->line->sign == sign)\n> + o->prev_line->next_line = key;\n> +\n> + hashmap_add(hm, key);\n> + o->prev_line = key;\n> +}\n> +\n> static void emit_add_line(const char *reset,\n> struct emit_callback *ecbdata,\n> const char *line, int len)\n> @@ -3649,6 +3863,9 @@ void diff_setup_done(struct diff_options *options)\n>\n> if (DIFF_OPT_TST(options, FOLLOW_RENAMES) && options->pathspec.nr != 1)\n> die(_(\"--follow requires exactly one pathspec\"));\n> +\n> + if (!options->use_color || external_diff())\n> + options->color_moved = 0;\n> }\n>\n> static int opt_arg(const char *arg, int arg_short, const char *arg_long, int *val)\n> @@ -4073,6 +4290,10 @@ int diff_opt_parse(struct diff_options *options,\n> }\n> else if (!strcmp(arg, \"--no-color\"))\n> options->use_color = 0;\n> + else if (!strcmp(arg, \"--color-moved\"))\n> + options->color_moved = 1;\n> + else if (!strcmp(arg, \"--no-color-moved\"))\n> + options->color_moved = 0;\n> else if (!strcmp(arg, \"--color-words\")) {\n> options->use_color = 1;\n> options->word_diff = DIFF_WORDS_COLOR;\n> @@ -4878,16 +5099,19 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n> {\n> int i;\n> struct diff_queue_struct *q = &diff_queued_diff;\n> - /*\n> - * For testing purposes we want to make sure the diff machinery\n> - * works completely with the buffer. If there is anything emitted\n> - * outside the emit_buffered_patch_line, then the order is screwed\n> - * up and the tests will fail.\n> - *\n> - * TODO (later in this series):\n> - * We'll unset this flag in a later patch.\n> - */\n> - o->use_buffer = 1;\n> +\n> + if (o->color_moved) {\n> + unsigned ignore_ws = DIFF_XDL_TST(o, IGNORE_WHITESPACE);\n> + o->use_buffer = 1;\n> + o->deleted_lines = xmallocz(sizeof(*o->deleted_lines));\n> + o->added_lines = xmallocz(sizeof(*o->added_lines));\n> + hashmap_init(o->deleted_lines, ignore_ws ?\n> + (hashmap_cmp_fn)moved_entry_cmp_no_ws :\n> + (hashmap_cmp_fn)moved_entry_cmp, 0);\n> + hashmap_init(o->added_lines, ignore_ws ?\n> + (hashmap_cmp_fn)moved_entry_cmp_no_ws :\n> + (hashmap_cmp_fn)moved_entry_cmp, 0);\n> + }\n>\n> if (o->use_buffer) {\n> ALLOC_GROW(o->filepair_buffer,\n> @@ -4902,6 +5126,10 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n> }\n>\n> if (o->use_buffer) {\n> + o->current_filepair = &o->filepair_buffer[0];\n> + for (i = 0; i < o->line_buffer_nr; i++)\n> + add_line_to_move_detection(o, i);\n> +\n> o->current_filepair = &o->filepair_buffer[0];\n> for (i = 0; i < o->line_buffer_nr; i++)\n> process_next_buffered_patch_line(o, i);\n> @@ -4992,6 +5220,7 @@ void diff_flush(struct diff_options *options)\n> if (!options->file)\n> die_errno(\"Could not open /dev/null\");\n> options->close_file = 1;\n> + options->color_moved = 0;\n> for (i = 0; i < q->nr; i++) {\n> struct diff_filepair *p = q->queue[i];\n> if (check_pair_status(p))\n> diff --git a/diff.h b/diff.h\n> index c334aac02e..b83d6fefcc 100644\n> --- a/diff.h\n> +++ b/diff.h\n> @@ -7,6 +7,7 @@\n> #include \"tree-walk.h\"\n> #include \"pathspec.h\"\n> #include \"object.h\"\n> +#include \"hashmap.h\"\n>\n> struct rev_info;\n> struct diff_options;\n> @@ -145,6 +146,8 @@ struct buffered_filepair {\n> unsigned ws_rule;\n> };\n>\n> +struct moved_entry;\n> +\n> struct diff_options {\n> const char *orderfile;\n> const char *pickaxe;\n> @@ -217,6 +220,8 @@ struct diff_options {\n>\n> int diff_path_counter;\n>\n> + /* Determines color moved code. Flipped between 1, 2 for alt. color. */\n> + int color_moved;\n> int use_buffer;\n>\n> struct buffered_patch_line *line_buffer;\n> @@ -225,6 +230,16 @@ struct diff_options {\n> struct buffered_filepair *filepair_buffer;\n> int filepair_buffer_nr, filepair_buffer_alloc;\n> struct buffered_filepair *current_filepair;\n> +\n> + /* built up in the first pass: */\n> + struct hashmap *deleted_lines;\n> + struct hashmap *added_lines;\n> + /* needed for building up */\n> + struct moved_entry *prev_line;\n> +\n> + /* state in the second pass */\n> + struct moved_entry **pmb; /* potentially moved blocks */\n> + int pmb_nr, pmb_alloc;\n\nPlacing these in the public API makes the scope unnecessarily large -\ncould these be stored in a struct (or better, in local variables)\nprivate to the .c file? In particular, prev_line should not need a\nscope greater than a function - a function could just loop through all\nthe buffered lines and construct the two hash maps, and prev_line\nwould not be needed elsewhere.\n\n> };\n>\n> void emit_line_fmt(struct diff_options *o, const char *set, const char *reset,\n> @@ -241,7 +256,11 @@ enum color_diff {\n> DIFF_FILE_NEW = 5,\n> DIFF_COMMIT = 6,\n> DIFF_WHITESPACE = 7,\n> - DIFF_FUNCINFO = 8\n> + DIFF_FUNCINFO = 8,\n> + DIFF_FILE_OLD_MOVED = 9,\n> + DIFF_FILE_OLD_MOVED_ALT = 10,\n> + DIFF_FILE_NEW_MOVED = 11,\n> + DIFF_FILE_NEW_MOVED_ALT = 12\n> };\n> const char *diff_get_color(int diff_use_color, enum color_diff ix);\n> #define diff_get_color_opt(o, ix) \\\n> diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\n> index 289806d0c7..232d9ad55e 100755\n> --- a/t/t4015-diff-whitespace.sh\n> +++ b/t/t4015-diff-whitespace.sh\n[snip]\n"},{"id":"319917","messageId":"20170516123132.gekwt6fjbmadubnp@sigill.intra.peff.net","threadId":"45958","inReplyTo":"20170514040117.25865-20-sbeller@google.com","subject":"Re: [PATCH 19/19] diff.c: color moved lines differently","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-05-16T12:31:32Z","receivedAt":"2017-05-16T12:32:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, May 13, 2017 at 09:01:17PM -0700, Stefan Beller wrote:\n\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 475e874d51..90403c06e3 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -1051,14 +1051,22 @@ This does not affect linkgit:git-format-patch[1] or the\n> 'git-diff-{asterisk}' plumbing commands. Can be overridden on the\n> command line with the `--color[=]` option.\n> \n> +color.moved::\n> +\tA boolean value, whether a diff should color moved lines\n> +\tdifferently. The moved lines are searched for in the diff only.\n> +\tDuplicated lines from somewhere in the project that are not\n> +\tpart of the diff are not colored as moved.\n> +\tDefaults to false.\n\nI wanted to play with this series to see how it looked on a few commits.\nSince this was the only documentation change, I tried \"git -c\ncolor.moved=true diff ...\". But it doesn't seem to work.\n\nIf we grep for diff_color_moved_default, we can see it declared:\n\n> diff --git a/diff.c b/diff.c\n> index dbab7fb44e..6372e0eb25 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -31,6 +31,7 @@ static int diff_indent_heuristic; /* experimental */\n> static int diff_rename_limit_default = 400;\n> static int diff_suppress_blank_empty;\n> static int diff_use_color_default = -1;\n> +static int diff_color_moved_default;\n> static int diff_context_default = 3;\n> static int diff_interhunk_context_default;\n> static const char *diff_word_regex_cfg;\n\nand we can see it parsed:\n\n> @@ -234,6 +247,10 @@ int git_diff_ui_config(const char *var, const char *value, void *cb)\n> \t\tdiff_use_color_default = git_config_colorbool(var, value);\n> \t\treturn 0;\n> \t}\n> +\tif (!strcmp(var, \"color.moved\")) {\n> +\t\tdiff_color_moved_default = git_config_bool(var, value);\n> +\t\treturn 0;\n> +\t}\n\nBut then nobody uses it. ;)\n\nI suspect diff_setup() needs to copy the default into the diff_options\nstruct?\n\nBy reading the code, I found --color-moved, which did work. Yay. So then\nI wanted to see how it looked on a wide variety of commits. So I ran\n\"git log -p --color-moved\", but it segfaulted. :-/\n\nI didn't dig, but here's what valgrind says, in case it helps:\n\n $ valgrind ./git --no-pager log --oneline -p --color-moved\n [...]\n fcdb8874d diff: buffer all output if asked to\n ==8801== Invalid write of size 4\n ==8801== at 0x1FD521: builtin_diff (diff.c:2869)\n ==8801== by 0x1FF3CB: run_diff_cmd (diff.c:3582)\n ==8801== by 0x1FF7F9: run_diff (diff.c:3670)\n ==8801== by 0x202CC8: diff_flush_patch (diff.c:4664)\n ==8801== by 0x204123: diff_flush_patch_all_file_pairs (diff.c:5125)\n ==8801== by 0x204682: diff_flush (diff.c:5249)\n ==8801== by 0x22CE18: log_tree_diff_flush (log-tree.c:775)\n ==8801== by 0x22D063: log_tree_diff (log-tree.c:842)\n ==8801== by 0x22D195: log_tree_commit (log-tree.c:871)\n ==8801== by 0x1648CE: cmd_log_walk (log.c:365)\n ==8801== by 0x1659B0: cmd_log (log.c:689)\n ==8801== by 0x11A434: run_builtin (git.c:371)\n ==8801== Address 0x8 is not stack'd, malloc'd or (recently) free'd\n ==8801== \n ==8801== \n ==8801== Process terminating with default action of signal 11\n (SIGSEGV)\n ==8801== Access not within mapped region at address 0x8\n ==8801== at 0x1FD521: builtin_diff (diff.c:2869)\n ==8801== by 0x1FF3CB: run_diff_cmd (diff.c:3582)\n ==8801== by 0x1FF7F9: run_diff (diff.c:3670)\n ==8801== by 0x202CC8: diff_flush_patch (diff.c:4664)\n ==8801== by 0x204123: diff_flush_patch_all_file_pairs (diff.c:5125)\n ==8801== by 0x204682: diff_flush (diff.c:5249)\n ==8801== by 0x22CE18: log_tree_diff_flush (log-tree.c:775)\n ==8801== by 0x22D063: log_tree_diff (log-tree.c:842)\n ==8801== by 0x22D195: log_tree_commit (log-tree.c:871)\n ==8801== by 0x1648CE: cmd_log_walk (log.c:365)\n ==8801== by 0x1659B0: cmd_log (log.c:689)\n ==8801== by 0x11A434: run_builtin (git.c:371)\n\nIt does show one commit correctly; fcdb8874d (your commit, but picked up\nfrom the list, so my commit sha1) is the second one. I don't know if\nthat's coincidence, or maybe there's something that's not properly\nbetween diff runs.\n\n-Peff\n"},{"id":"319923","messageId":"CAGf8dgJY0=hr9_M3LijZnEhv7rEad8XHAjcpjG425JtKNjweHQ@mail.gmail.com","threadId":"45958","inReplyTo":"CAGZ79kZSixaG2CoO-WV47=-eE96yhtoCm82KFbZ=-=8c-VPXcA@mail.gmail.com","subject":"Re: [PATCH 03/19] diff.c: drop 'nofirst' from emit_line_0","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-05-16T16:05:27Z","receivedAt":"2017-05-16T16:05:36Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Mon, May 15, 2017 at 11:33 AM, Stefan Beller wrote:\n> On Mon, May 15, 2017 at 11:26 AM, Jonathan Tan wrote:\n>> I also don't understand the meaning of this paragraph - if you mean that\n>> this patch teaches other callers to hardcode the sign, I don't see any such\n>> changes in the diff below.\n>\n> The last two hunks of the patch switch two callers that call with a sign\n> that is hard to reason about.\n\nThe last two hunks don't hardcode any signs, as far as I can see. They\ndo pass in a \"first\" character that may or may not be a sign, if that\nis what you mean.\n\nIn any case, can you reword that paragraph into an imperative\nstatement (e.g. \"teach X to...\", \"make X...\")?\n\n>>> + char term[2];\n>>> + term[0] = options->line_termination;\n>>> + term[1] = '\\0';\n>>> +\n>>> + emit_line(options, NULL, NULL,\n>>> + term, 1);\n>>\n>>\n>> If options->line_termination is 0, this is actually a zero-length string\n>> (not 1).\n>\n> So passing in !!options->line_termination should be fine?\n\nYes, that would work. I slightly prefer !!term[0].\n"},{"id":"319926","messageId":"CAGZ79ka3JU=7WgF1tYTTam32QhZVokJxdqyOi33P-sd=XrUecw@mail.gmail.com","threadId":"45958","inReplyTo":"xmqqpof94nn2.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 07/19] diff.c: convert fn_out_consume to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-16T16:23:39Z","receivedAt":"2017-05-16T16:23:45Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, May 15, 2017 at 6:05 PM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n>\n>>> - fprintf(o->file, \"%s%s--- %s%s%s\\n\",\n>>> - line_prefix, meta, ecbdata->label_path[0], reset, name_a_tab);\n>>> - fprintf(o->file, \"%s%s+++ %s%s%s\\n\",\n>>> - line_prefix, meta, ecbdata->label_path[1], reset, name_b_tab);\n>>> + emit_line_fmt(o, meta, reset, \"--- %s%s\\n\",\n>>> + ecbdata->label_path[0], name_a_tab);\n>>> + emit_line_fmt(o, meta, reset, \"+++ %s%s\\n\",\n>>> + ecbdata->label_path[1], name_b_tab);\n>>\n>> How is the loss of line_prefix from this call site compensated?\n>\n> OK, emit_line_0() has already been aware of line_prefix, so that is\n> how the loss of line_prefix in the above is accounted for. We are\n> good here.\n>\n>>> ecbdata->label_path[0] = ecbdata->label_path[1] = NULL;\n>>> }\n>>>\n>>> @@ -1349,7 +1346,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n>>> diff_words_flush(ecbdata);\n>>> if (ecbdata->diff_words->type == DIFF_WORDS_PORCELAIN) {\n>>> emit_line(o, context, reset, line, len);\n>>> - fputs(\"~\\n\", o->file);\n>>> + emit_line(o, NULL, NULL, \"~\\n\", 2);\n>\n> So unless we have some magic here, we would see an extra line-prefix\n> before that \"~\\n\" thing, no?\n\nRight.\n\nWith all the discussion on the the convert* patches, I consider just introducing\nanother parameter to emit_line, which specifies whether we want\nemit_line to output\na line_prefix. Then the conversion becomes a lot easier as well.\n\nThanks,\nStefan\n"},{"id":"319928","messageId":"CAGZ79kZQBG0AmADWCQ5yj+sRMtwiy8HOnD3SLFS_-yPPoRbfTA@mail.gmail.com","threadId":"45958","inReplyTo":"CAGf8dgJP+i5RL3FaGSYZyVKkt1ttSnnPd924ebs=4xJb4Fhc6w@mail.gmail.com","subject":"Re: [PATCH 18/19] diff: buffer all output if asked to","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-16T16:42:23Z","receivedAt":"2017-05-16T16:42:29Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, May 15, 2017 at 9:14 PM, Jonathan Tan wrote:\n> Overall, this patch seems larger than it should to me, although there\n> might be good reasons for that that I don't know. I'll remark on what\n> I find unexpected.\n>\n\n\n>>\n>> +static void emit_buffered_patch_line_ws(struct diff_options *o,\n>> + struct buffered_patch_line *e,\n>> + const char *ws, unsigned ws_rule)\n>\n> This introduces a new _ws emission function - how is this used and how\n> is this different from the non-ws one? I see BPL_EMIT_LINE_WS, but I\n> don't see the caller that introduces that constant in this patch.\n\nSee emit_line_ws, which makes use of BPL_EMIT_LINE_WS.\nThe difference between BPL_EMIT_LINE_WS and BPL_EMIT_LINE_AS_IS\nis that _WS is emitted marking up whitespace differently (e.g. 8 continuous\nspaces instead of a tab or such), see the core.whitespace option.\nRelevant hunk:\n\n@@ -557,9 +616,12 @@ static void emit_line_ws(struct diff_options *o,\n const char *line, int len,\n const char *ws, unsigned ws_rule)\n {\n- emit_line_0(o, set, reset, sign, \"\", 0);\n- ws_check_emit(line, len, ws_rule,\n- o->file, set, reset, ws);\n+ struct buffered_patch_line e = {set, reset, line, len, sign,\nBPL_EMIT_LINE_WS};\n+\n+ if (o->use_buffer)\n+ append_buffered_patch_line(o, &e);\n+ else\n+ emit_buffered_patch_line_ws(o, &e, ws, ws_rule);\n }\n\n>> + switch (e->state) {\n>> + case BPL_EMIT_LINE_ASIS:\n>> + emit_buffered_patch_line(o, e);\n>> + break;\n>> + case BPL_EMIT_LINE_WS:\n>> + emit_buffered_patch_line_ws(o, e, ws, ws_rule);\n>> + break;\n>> + case BPL_HANDOVER:\n>> + o->current_filepair++;\n>\n> If we're just buffering the diff output, do we need to store\n> per-file-pair metadata? (I assume that's why you need a special\n> handover constant.) Clients can already read what they need from the\n> diff output.\n\nCurrently we keep only whitespace settings per-file separate as they are\ndefined per path (via attributes).\n\nSo I read this comment as if you consider the per-file buffer unneeded and\nwe could just detect the next file via the line\n\n diff --git a/ b/\n\nand then re-read the attributes and remember only the current files settings in\nthe output pass. I'll look into that.\n\n>> + *\n>> + * TODO (later in this series):\n>> + * We'll unset this flag in a later patch.\n>> + */\n>> + o->use_buffer = 1;\n>\n> What I would do is to add a demonstration patch at the end of the\n> patch series (which is not supposed to be queued) to avoid such churn\n> in history, but I'm not sure how the Git project prefers to do this.\n\nok, I can omit this part in a reroll.\n\n>> + *\n>> + * NEEDSWORK: Instead of storing a copy of the line, add an offset pointer\n>> + * into the pre/post image file. This pointer could be a union with the\n>> + * line pointer. By storing an offset into the file instead of the literal line,\n>> + * we can decrease the memory footprint for the buffered output. At first we\n>> + * may want to only have indirection for the content lines, but we could\n>> + * also have an enum (based on sign?) that stores prefabricated lines, e.g.\n>> + * the similarity score line or hunk/file headers.\n>\n> This would be nice, but come to think of it, might not be possible.\n> When requesting --word-diff, control characters (or others) might\n> appear in the output, right?\n\nThat is why we'd have even more states. ;)\nOr duplicate word diff still, but lines left intact are referenced via offsets.\nIt's a hard problem to get right, so I defer it via this comment.\n\n>\n>> + */\n>> +struct buffered_patch_line {\n>> + const char *set;\n>> + const char *reset;\n>> + const char *line;\n>> + int len;\n>> + int sign;\n>> + enum {\n>> + BPL_EMIT_LINE_WS,\n>> + BPL_EMIT_LINE_ASIS,\n>> + BPL_HANDOVER\n>> + } state;\n>\n> It might be better, for simplicity, just to have one big buffer\n> including everything (if we decide that we really can't add pointers\n> to input later).\n\nWhat do you mean here? (Drop the other structs such as the file pair?)\n\nThanks for the review!\nStefan\n"},{"id":"319992","messageId":"20170517025857.32320-1-sbeller@google.com","threadId":"45958","inReplyTo":"20170514040117.25865-1-sbeller@google.com","subject":"[PATCHv2 00/20] Diff machine: highlight moved lines.","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:37Z","receivedAt":"2017-05-17T02:59:13Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"v2:\n* emit_line now takes an argument that indicates if we want it\n to emit the line prefix as well. This should allow for a more faithful\n refactoring in the beginning. (Thanks Jonathan!)\n* fixed memleaks (Thanks Brandon!)\n* \"git -c color.moved=true log -p\" works now! (Thanks Jeff)\n* interdiff below, though it is large.\n* less intrusive than v1 (Thanks Jonathan!)\n\nv1:\n\nFor details on *why* see the commit message of the last commit.\n\nThe first five patches are slight refactorings to get into good\nshape, the next patches are funneling all output through emit_line_*.\n\nThe second last patch introduces an option to buffer up all output\nbefore printing, and then the last patch can color up moved lines\nof code.\n\nAny feedback welcome.\n\nThanks,\nStefan\n\nStefan Beller (20):\n diff: readability fix\n diff: move line ending check into emit_hunk_header\n diff.c: factor out diff_flush_patch_all_file_pairs\n diff.c: teach emit_line_0 to accept sign parameter\n diff.c: emit_line_0 can handle no color setting\n diff.c: emit_line_0 takes parameter whether to output line prefix\n diff.c: inline emit_line_0 into emit_line\n diff.c: convert fn_out_consume to use emit_line\n diff.c: convert builtin_diff to use emit_line_*\n diff.c: convert emit_rewrite_diff to use emit_line_*\n diff.c: convert emit_rewrite_lines to use emit_line_*\n submodule.c: convert show_submodule_summary to use emit_line_fmt\n diff.c: convert emit_binary_diff_body to use emit_line_*\n diff.c: convert show_stats to use emit_line_*\n diff.c: convert word diffing to use emit_line_*\n diff.c: convert diff_flush to use emit_line_*\n diff.c: convert diff_summary to use emit_line_*\n diff.c: emit_line includes whitespace highlighting\n diff: buffer all output if asked to\n diff.c: color moved lines differently\n\n Documentation/config.txt | 14 +-\n diff.c | 845 +++++++++++++++++++++++++++++++++------------\n diff.h | 61 +++-\n submodule.c | 78 ++---\n submodule.h | 9 +-\n t/t4015-diff-whitespace.sh | 229 ++++++++++++\n 6 files changed, 960 insertions(+), 276 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 90403c06e3..902d017c3b 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1065,8 +1065,10 @@ color.diff.::\n \t`meta` (metainformation), `frag`\n \t(hunk header), 'func' (function in hunk header), `old` (removed lines),\n \t`new` (added lines), `commit` (commit headers), `whitespace`\n-\t(highlighting whitespace errors), `movedFrom` (removed lines that\n-\treappear), `movedTo` (added lines that were removed elsewhere).\n+\t(highlighting whitespace errors), `oldMoved` (removed lines that\n+\treappear), `newMoved` (added lines that were removed elsewhere),\n+\t`oldMovedAlternative` and `newMovedAlternative` (as a fallback to\n+\tcover adjacent blocks of moved code)\n \n color.decorate.::\n \tUse customized color for 'git log --decorate' output. `` is one\ndiff --git a/diff.c b/diff.c\nindex 5dfd582084..15cf322b50 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -392,11 +392,17 @@ static int buffered_patch_line_cmp_no_ws(const struct buffered_patch_line *a,\n \t\t\t\t\t const struct buffered_patch_line *b,\n \t\t\t\t\t const void *keydata)\n {\n+\tint ret;\n \tstruct strbuf sba = STRBUF_INIT;\n \tstruct strbuf sbb = STRBUF_INIT;\n+\n \tget_ws_cleaned_string(a, &sba);\n \tget_ws_cleaned_string(b, &sbb);\n-\treturn sba.len != sbb.len || strncmp(sba.buf, sbb.buf, sba.len);\n+\tret = sba.len != sbb.len || strncmp(sba.buf, sbb.buf, sba.len);\n+\n+\tstrbuf_release(&sba);\n+\tstrbuf_release(&sbb);\n+\treturn ret;\n }\n \n static int buffered_patch_line_cmp(const struct buffered_patch_line *a,\n@@ -428,8 +434,9 @@ static unsigned get_line_hash(struct buffered_patch_line *line, unsigned ignore_\n \t\tstrbuf_reset(&sb);\n \t\tget_ws_cleaned_string(line, &sb);\n \t\treturn memhash(sb.buf, sb.len);\n-\t} else\n+\t} else {\n \t\treturn memhash(line->line, line->len);\n+\t}\n }\n \n static struct moved_entry *prepare_entry(struct diff_options *o,\n@@ -608,159 +615,185 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n-static void mark_color_as_moved(struct diff_options *o, int line_no)\n+static void add_lines_to_move_detection(struct diff_options *o)\n {\n-\tstruct hashmap *hm = NULL;\n-\tstruct moved_entry *key = prepare_entry(o, line_no);\n-\tstruct moved_entry *match = NULL;\n-\tstruct buffered_patch_line *l = &o->line_buffer[line_no];\n-\tint alt_flag;\n-\tint i, lp, rp;\n+\tstruct moved_entry *prev_line;\n \n-\tswitch (l->sign) {\n-\tcase '+':\n-\t\thm = o->deleted_lines;\n-\t\tbreak;\n-\tcase '-':\n-\t\thm = o->added_lines;\n-\t\tbreak;\n-\tdefault:\n-\t\t/* reset to standard, on-alt move color */\n-\t\to->color_moved = 1;\n-\t\tbreak;\n+\tint n;\n+\tfor (n = 0; n < o->line_buffer_nr; n++) {\n+\t\tint sign = 0;\n+\t\tstruct hashmap *hm;\n+\t\tstruct moved_entry *key;\n+\n+\t\tswitch (o->line_buffer[n].sign) {\n+\t\tcase '+':\n+\t\t\tsign = '+';\n+\t\t\thm = o->added_lines;\n+\t\t\tbreak;\n+\t\tcase '-':\n+\t\t\tsign = '-';\n+\t\t\thm = o->deleted_lines;\n+\t\t\tbreak;\n+\t\tcase ' ':\n+\t\tdefault:\n+\t\t\tprev_line = NULL;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tkey = prepare_entry(o, n);\n+\t\tif (prev_line &&\n+\t\t prev_line->line->sign == sign)\n+\t\t\tprev_line->next_line = key;\n+\n+\t\thashmap_add(hm, key);\n+\t\tprev_line = key;\n \t}\n+}\n \n-\t/* Check for any match to color it as a move. */\n-\tif (!hm)\n-\t\treturn;\n-\tmatch = hashmap_get(hm, key, o);\n-\tfree(key);\n-\tif (!match)\n-\t\treturn;\n+static void mark_color_as_moved(struct diff_options *o)\n+{\n+\tstruct moved_entry **pmb = NULL; /* potentially moved blocks */\n+\tint pmb_nr = 0, pmb_alloc = 0;\n+\tint alt_flag = 0;\n+\tint n;\n \n-\t/* Check any potential block runs, advance each or nullify */\n-\tfor (i = 0; i < o->pmb_nr; i++) {\n-\t\tstruct moved_entry *p = o->pmb[i];\n-\t\tif (p && p->next_line &&\n-\t\t !buffered_patch_line_cmp(p->next_line->line, l, o)) {\n-\t\t\to->pmb[i] = p->next_line;\n-\t\t} else {\n-\t\t\to->pmb[i] = NULL;\n+\tfor (n = 0; n < o->line_buffer_nr; n++) {\n+\t\tstruct hashmap *hm = NULL;\n+\t\tstruct moved_entry *key;\n+\t\tstruct moved_entry *match = NULL;\n+\t\tstruct buffered_patch_line *l = &o->line_buffer[n];\n+\t\tint i, lp, rp;\n+\n+\t\tswitch (l->sign) {\n+\t\tcase '+':\n+\t\t\thm = o->deleted_lines;\n+\t\t\tbreak;\n+\t\tcase '-':\n+\t\t\thm = o->added_lines;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\talt_flag = 0; /* reset to standard, no-alt move color */\n+\t\t\tpmb_nr = 0; /* no running sets */\n+\t\t\tcontinue;\n \t\t}\n-\t}\n \n-\t/* Shrink the set to the remaining runs */\n-\tfor (lp = 0, rp = o->pmb_nr - 1; lp <= rp;) {\n-\t\twhile (lp < o->pmb_nr && o->pmb[lp])\n-\t\t\tlp ++;\n-\t\t/* lp points at the first NULL now */\n+\t\t/* Check for any match to color it as a move. */\n+\t\tkey = prepare_entry(o, n);\n+\t\tmatch = hashmap_get(hm, key, o);\n+\t\tfree(key);\n+\t\tif (!match)\n+\t\t\tcontinue;\n \n-\t\twhile (rp > -1 && !o->pmb[rp])\n-\t\t\trp--;\n-\t\t/* rp points at the last non-NULL */\n+\t\t/* Check any potential block runs, advance each or nullify */\n+\t\tfor (i = 0; i < pmb_nr; i++) {\n+\t\t\tstruct moved_entry *p = pmb[i];\n+\t\t\tstruct moved_entry *pnext = (p && p->next_line) ?\n+\t\t\t\t\tp->next_line : NULL;\n+\t\t\tif (pnext &&\n+\t\t\t !buffered_patch_line_cmp(pnext->line, l, o)) {\n+\t\t\t\tpmb[i] = p->next_line;\n+\t\t\t} else {\n+\t\t\t\tpmb[i] = NULL;\n+\t\t\t}\n+\t\t}\n \n-\t\tif (lp < o->pmb_nr && rp > -1 && lp < rp) {\n-\t\t\to->pmb[lp] = o->pmb[rp];\n-\t\t\to->pmb[rp] = NULL;\n-\t\t\trp--;\n-\t\t\tlp++;\n+\t\t/* Shrink the set to the remaining runs */\n+\t\tfor (lp = 0, rp = pmb_nr - 1; lp <= rp;) {\n+\t\t\twhile (lp < pmb_nr && pmb[lp])\n+\t\t\t\tlp ++;\n+\t\t\t/* lp points at the first NULL now */\n+\n+\t\t\twhile (rp > -1 && !pmb[rp])\n+\t\t\t\trp--;\n+\t\t\t/* rp points at the last non-NULL */\n+\n+\t\t\tif (lp < pmb_nr && rp > -1 && lp < rp) {\n+\t\t\t\tpmb[lp] = pmb[rp];\n+\t\t\t\tpmb[rp] = NULL;\n+\t\t\t\trp--;\n+\t\t\t\tlp++;\n+\t\t\t}\n \t\t}\n-\t}\n \n-\tif (rp > -1) {\n-\t\t/* Remember the number of running sets */\n-\t\to->pmb_nr = rp + 1;\n-\t} else {\n-\t\t/* Toggle color */\n-\t\to->color_moved = o->color_moved == 2 ? 1 : 2;\n-\n-\t\t/* Build up a new set */\n-\t\ti = 0;\n-\t\tfor (; match; match = hashmap_get_next(hm, match)) {\n-\t\t\tALLOC_GROW(o->pmb, i + 1, o->pmb_alloc);\n-\t\t\to->pmb[i] = match;\n-\t\t\ti++;\n+\t\tif (rp > -1) {\n+\t\t\t/* Remember the number of running sets */\n+\t\t\tpmb_nr = rp + 1;\n+\t\t} else {\n+\t\t\t/* Toggle color */\n+\t\t\talt_flag = (alt_flag + 1) % 2;\n+\n+\t\t\t/* Build up a new set */\n+\t\t\tpmb_nr = 0;\n+\t\t\tfor (; match; match = hashmap_get_next(hm, match)) {\n+\t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n+\t\t\t\tpmb[pmb_nr++] = match;\n+\t\t\t}\n \t\t}\n-\t\to->pmb_nr = i;\n-\t}\n \n-\talt_flag = o->color_moved - 1;\n-\tswitch (l->sign) {\n-\tcase '+':\n-\t\tl->set = diff_get_color_opt(o, DIFF_FILE_NEW_MOVED + alt_flag);\n-\t\tbreak;\n-\tcase '-':\n-\t\tl->set = diff_get_color_opt(o, DIFF_FILE_OLD_MOVED + alt_flag);\n-\t\tbreak;\n-\tdefault:\n-\t\t/* reset to standard, on-alt move color */\n-\t\to->color_moved = 1;\n+\t\tswitch (l->sign) {\n+\t\tcase '+':\n+\t\t\tl->set = diff_get_color_opt(o, DIFF_FILE_NEW_MOVED + alt_flag);\n+\t\t\tbreak;\n+\t\tcase '-':\n+\t\t\tl->set = diff_get_color_opt(o, DIFF_FILE_OLD_MOVED + alt_flag);\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tdie(\"BUG: we should have continued earlier?\");\n+\t\t}\n \t}\n+\tfree(pmb);\n }\n \n static void emit_buffered_patch_line(struct diff_options *o,\n-\t\t\t\t struct buffered_patch_line *e,\n-\t\t\t\t int pass)\n+\t\t\t\t struct buffered_patch_line *e)\n {\n-\tint has_trailing_newline, has_trailing_carriage_return, len = e->len;\n+\tconst char *ws;\n+\tint has_trailing_newline, has_trailing_carriage_return;\n+\tint len = e->len;\n \tFILE *file = o->file;\n \n-\tfputs(diff_line_prefix(o), file);\n-\n-\thas_trailing_newline = (len > 0 && e->line[len-1] == '\\n');\n-\tif (has_trailing_newline)\n-\t\tlen--;\n-\thas_trailing_carriage_return = (len > 0 && e->line[len-1] == '\\r');\n-\tif (has_trailing_carriage_return)\n-\t\tlen--;\n+\tif (e->add_line_prefix)\n+\t\tfputs(diff_line_prefix(o), file);\n \n-\tif (len || e->sign) {\n+\tswitch (e->state) {\n+\tcase BPL_EMIT_LINE_WS:\n+\t\tws = diff_get_color(o->use_color, DIFF_WHITESPACE);\n \t\tif (e->set)\n \t\t\tfputs(e->set, file);\n \t\tif (e->sign)\n \t\t\tfputc(e->sign, file);\n-\t\tfwrite(e->line, len, 1, file);\n \t\tif (e->reset)\n \t\t\tfputs(e->reset, file);\n-\t}\n-\tif (has_trailing_carriage_return)\n-\t\tfputc('\\r', file);\n-\tif (has_trailing_newline)\n-\t\tfputc('\\n', file);\n-}\n-\n-static void emit_buffered_patch_line_ws(struct diff_options *o,\n-\t\t\t\t\tstruct buffered_patch_line *e,\n-\t\t\t\t\tconst char *ws, unsigned ws_rule,\n-\t\t\t\t\tint pass)\n-{\n-\tstruct buffered_patch_line s = {e->set, e->reset, \"\", 0, e->sign};\n-\temit_buffered_patch_line(o, &s, 0);\n-\tws_check_emit(e->line, e->len, ws_rule,\n-\t\t o->file, e->set, e->reset, ws);\n-}\n-\n-static void process_next_buffered_patch_line(struct diff_options *o, int line_no)\n-{\n-\tstruct buffered_patch_line *e = &o->line_buffer[line_no];\n-\n-\tconst char *ws = o->current_filepair->ws;\n-\tunsigned ws_rule = o->current_filepair->ws_rule;\n-\n-\tmark_color_as_moved(o, line_no);\n+\t\tws_check_emit(e->line, e->len, o->ws_rule,\n+\t\t\t file, e->set, e->reset, ws);\n+\t\treturn;\n+\tcase BPL_EMIT_LINE_ASIS:\n+\t\thas_trailing_newline = (len > 0 && e->line[len-1] == '\\n');\n+\t\tif (has_trailing_newline)\n+\t\t\tlen--;\n+\t\thas_trailing_carriage_return = (len > 0 && e->line[len-1] == '\\r');\n+\t\tif (has_trailing_carriage_return)\n+\t\t\tlen--;\n \n-\tswitch (e->state) {\n-\t\tcase BPL_EMIT_LINE_ASIS:\n-\t\t\temit_buffered_patch_line(o, e, 1);\n-\t\t\tbreak;\n-\t\tcase BPL_EMIT_LINE_WS:\n-\t\t\temit_buffered_patch_line_ws(o, e, ws, ws_rule, 1);\n-\t\t\tbreak;\n-\t\tcase BPL_HANDOVER:\n-\t\t\to->current_filepair++;\n-\t\t\tbreak;\n-\t\tdefault:\n-\t\t\tdie(\"BUG: malformatted buffered patch line: '%d'\", e->state);\n+\t\tif (len || e->sign) {\n+\t\t\tif (e->set)\n+\t\t\t\tfputs(e->set, file);\n+\t\t\tif (e->sign)\n+\t\t\t\tfputc(e->sign, file);\n+\t\t\tfwrite(e->line, len, 1, file);\n+\t\t\tif (e->reset)\n+\t\t\t\tfputs(e->reset, file);\n+\t\t}\n+\t\tif (has_trailing_carriage_return)\n+\t\t\tfputc('\\r', file);\n+\t\tif (has_trailing_newline)\n+\t\t\tfputc('\\n', file);\n+\t\treturn;\n+\tcase BPL_HANDOVER:\n+\t\to->ws_rule = whitespace_rule(e->line); /*read from file, stored in line?*/\n+\t\treturn;\n+\tdefault:\n+\t\tdie(\"BUG: malformatted buffered patch line: '%d'\", e->state);\n \t}\n }\n \n@@ -771,46 +804,30 @@ static void append_buffered_patch_line(struct diff_options *o,\n \tALLOC_GROW(o->line_buffer,\n \t\t o->line_buffer_nr + 1,\n \t\t o->line_buffer_alloc);\n-\tf = &o->line_buffer[o->line_buffer_nr];\n-\to->line_buffer_nr++;\n+\tf = &o->line_buffer[o->line_buffer_nr++];\n \n \tmemcpy(f, e, sizeof(struct buffered_patch_line));\n \tf->line = e->line ? xmemdupz(e->line, e->len) : NULL;\n }\n \n-static void emit_line_0(struct diff_options *o,\n-\t\t\tconst char *set, const char *reset,\n-\t\t\tint sign, const char *line, int len)\n+void emit_line(struct diff_options *o,\n+\t const char *set, const char *reset,\n+\t int add_line_prefix, int markup_ws,\n+\t int sign, const char *line, int len)\n {\n-\tstruct buffered_patch_line e = {set, reset, line, len, sign, BPL_EMIT_LINE_ASIS};\n+\tstruct buffered_patch_line e = {set, reset, line,\n+\t\tlen, sign, add_line_prefix,\n+\t\tmarkup_ws ? BPL_EMIT_LINE_WS : BPL_EMIT_LINE_ASIS};\n \n \tif (o->use_buffer)\n \t\tappend_buffered_patch_line(o, &e);\n \telse\n-\t\temit_buffered_patch_line(o, &e, 0);\n-}\n-\n-void emit_line(struct diff_options *o, const char *set, const char *reset,\n-\t const char *line, int len)\n-{\n-\temit_line_0(o, set, reset, 0, line, len);\n-}\n-\n-static void emit_line_ws(struct diff_options *o,\n-\t\t\t const char *set, const char *reset, int sign,\n-\t\t\t const char *line, int len,\n-\t\t\t const char *ws, unsigned ws_rule)\n-{\n-\tstruct buffered_patch_line e = {set, reset, line, len, sign, BPL_EMIT_LINE_WS};\n-\n-\tif (o->use_buffer)\n-\t\tappend_buffered_patch_line(o, &e);\n-\telse\n-\t\temit_buffered_patch_line_ws(o, &e, ws, ws_rule, 0);\n+\t\temit_buffered_patch_line(o, &e);\n }\n \n void emit_line_fmt(struct diff_options *o,\n \t\t const char *set, const char *reset,\n+\t\t int add_line_prefix,\n \t\t const char *fmt, ...)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n@@ -819,7 +836,7 @@ void emit_line_fmt(struct diff_options *o,\n \tstrbuf_vaddf(&sb, fmt, ap);\n \tva_end(ap);\n \n-\temit_line(o, set, reset, sb.buf, sb.len);\n+\temit_line(o, set, reset, add_line_prefix, 0, 0, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n }\n \n@@ -851,44 +868,15 @@ static void emit_line_checked(const char *reset,\n \t}\n \n \tif (!ws)\n-\t\temit_line_0(ecbdata->opt, set, reset, sign, line, len);\n+\t\temit_line(ecbdata->opt, set, reset, 1, 0, sign, line, len);\n \telse if (sign == '+' && new_blank_line_at_eof(ecbdata, line, len))\n \t\t/* Blank line at EOF - paint '+' as well */\n-\t\temit_line_0(ecbdata->opt, ws, reset, sign, line, len);\n-\telse\n+\t\temit_line(ecbdata->opt, ws, reset, 1, 1, sign, line, len);\n+\telse {\n \t\t/* Emit just the prefix, then the rest. */\n-\t\temit_line_ws(ecbdata->opt, set, reset, sign, line, len,\n-\t\t\t ws, ecbdata->ws_rule);\n-}\n-\n-static void add_line_to_move_detection(struct diff_options *o, int line_idx)\n-{\n-\tint sign = 0;\n-\tstruct hashmap *hm;\n-\tstruct moved_entry *key;\n-\n-\tswitch (o->line_buffer[line_idx].sign) {\n-\tcase '+':\n-\t\tsign = '+';\n-\t\thm = o->added_lines;\n-\t\tbreak;\n-\tcase '-':\n-\t\tsign = '-';\n-\t\thm = o->deleted_lines;\n-\t\tbreak;\n-\tcase ' ':\n-\tdefault:\n-\t\to->prev_line = NULL;\n-\t\treturn;\n+\t\temit_line(ecbdata->opt, set, reset, 1, 1, sign, line, len);\n \t}\n \n-\tkey = prepare_entry(o, line_idx);\n-\tif (o->prev_line &&\n-\t o->prev_line->line->sign == sign)\n-\t\to->prev_line->next_line = key;\n-\n-\thashmap_add(hm, key);\n-\to->prev_line = key;\n }\n \n static void emit_add_line(const char *reset,\n@@ -935,7 +923,7 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \tif (len < 10 ||\n \t memcmp(line, atat, 2) ||\n \t !(ep = memmem(line + 2, len - 2, atat, 2))) {\n-\t\temit_line(ecbdata->opt, context, reset, line, len);\n+\t\temit_line(ecbdata->opt, context, reset, 1, 0, 0, line, len);\n \t\treturn;\n \t}\n \tep += 2; /* skip over @@ */\n@@ -971,7 +959,7 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \tstrbuf_add(&msgbuf, line + len, org_len - len);\n \tstrbuf_complete_line(&msgbuf);\n \n-\temit_line(ecbdata->opt, \"\", \"\", msgbuf.buf, msgbuf.len);\n+\temit_line(ecbdata->opt, \"\", \"\", 1, 0, 0, msgbuf.buf, msgbuf.len);\n \tstrbuf_release(&msgbuf);\n }\n \n@@ -1011,15 +999,25 @@ static void add_line_count(struct strbuf *out, int count)\n static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\t\t int prefix, const char *data, int size)\n {\n-\tconst char *endp = NULL;\n-\tstatic const char *nneof = \" No newline at end of file\\n\";\n \tconst char *reset = diff_get_color(ecb->color_diff, DIFF_RESET);\n+\tstruct strbuf sb = STRBUF_INIT;\n \n \twhile (0 < size) {\n \t\tint len;\n \n-\t\tendp = memchr(data, '\\n', size);\n-\t\tlen = endp ? (endp - data + 1) : size;\n+\t\tconst char *endp = memchr(data, '\\n', size);\n+\t\tif (endp)\n+\t\t\tlen = endp - data + 1;\n+\t\telse {\n+\t\t\twhile (0 < size) {\n+\t\t\t\tstrbuf_addch(&sb, *data);\n+\t\t\t\tsize -= len;\n+\t\t\t\tdata += len;\n+\t\t\t}\n+\t\t\tstrbuf_addch(&sb, '\\n');\n+\t\t\tdata = sb.buf;\n+\t\t\tlen = sb.len;\n+\t\t}\n \t\tif (prefix != '+') {\n \t\t\tecb->lno_in_preimage++;\n \t\t\temit_del_line(reset, ecb, data, len);\n@@ -1030,12 +1028,13 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\tsize -= len;\n \t\tdata += len;\n \t}\n-\tif (!endp) {\n+\tif (sb.len) {\n+\t\tstatic const char *nneof = \"\\\\ No newline at end of file\\n\";\n \t\tconst char *context = diff_get_color(ecb->color_diff,\n \t\t\t\t\t\t DIFF_CONTEXT);\n-\t\temit_line(ecb->opt, NULL, NULL, \"\\n\", 1);\n-\t\temit_line_0(ecb->opt, context, reset, '\\\\',\n+\t\temit_line(ecb->opt, context, reset, 1, 0, 0,\n \t\t\t nneof, strlen(nneof));\n+\t\tstrbuf_release(&sb);\n \t}\n }\n \n@@ -1095,8 +1094,8 @@ static void emit_rewrite_diff(const char *name_a,\n \tecbdata.lno_in_preimage = 1;\n \tecbdata.lno_in_postimage = 1;\n \n-\temit_line_fmt(o, metainfo, reset, \"--- %s%s\\n\", a_name.buf, name_a_tab);\n-\temit_line_fmt(o, metainfo, reset, \"+++ %s%s\\n\", b_name.buf, name_b_tab);\n+\temit_line_fmt(o, metainfo, reset, 1, \"--- %s%s\\n\", a_name.buf, name_a_tab);\n+\temit_line_fmt(o, metainfo, reset, 1, \"+++ %s%s\\n\", b_name.buf, name_b_tab);\n \n \tlc_a = count_lines(data_one, size_one);\n \tlc_b = count_lines(data_two, size_two);\n@@ -1109,7 +1108,7 @@ static void emit_rewrite_diff(const char *name_a,\n \tstrbuf_addstr(&out, \" +\");\n \tadd_line_count(&out, lc_b);\n \tstrbuf_addstr(&out, \" @@\\n\");\n-\temit_line(o, fraginfo, reset, out.buf, out.len);\n+\temit_line(o, fraginfo, reset, 1, 0, 0, out.buf, out.len);\n \tstrbuf_release(&out);\n \n \tif (lc_a && !o->irreversible_delete)\n@@ -1174,34 +1173,38 @@ struct diff_words_data {\n static int fn_out_diff_words_write_helper(struct diff_options *o,\n \t\t\t\t\t struct diff_words_style_elem *st_el,\n \t\t\t\t\t const char *newline,\n-\t\t\t\t\t size_t count, const char *buf,\n-\t\t\t\t\t const char *line_prefix)\n+\t\t\t\t\t size_t count, const char *buf)\n {\n+\tint print = 0;\n \tstruct strbuf sb = STRBUF_INIT;\n \n \twhile (count) {\n \t\tchar *p = memchr(buf, '\\n', count);\n+\t\tif (print)\n+\t\t\temit_line(o, NULL, NULL, 1, 0, 0, \"\", 0);\n \n \t\tif (p != buf) {\n-\t\t\tif (st_el->color)\n-\t\t\t\tstrbuf_addstr(&sb, st_el->color);\n+\t\t\tconst char *reset = st_el->color && *st_el->color ?\n+\t\t\t\t\t GIT_COLOR_RESET : NULL;\n \t\t\tstrbuf_addstr(&sb, st_el->prefix);\n \t\t\tstrbuf_add(&sb, buf, p ? p - buf : count);\n \t\t\tstrbuf_addstr(&sb, st_el->suffix);\n-\t\t\tif (st_el->color && *st_el->color)\n-\t\t\t strbuf_addstr(&sb, GIT_COLOR_RESET);\n+\t\t\temit_line(o, st_el->color, reset,\n+\t\t\t\t 0, 0, 0, sb.buf, sb.len);\n+\t\t\tstrbuf_reset(&sb);\n \t\t}\n \t\tif (!p)\n \t\t\tgoto out;\n+\n \t\tstrbuf_addstr(&sb, newline);\n-\t\temit_line(o, NULL, NULL, sb.buf, sb.len);\n+\t\temit_line(o, NULL, NULL, 0, 0, 0, sb.buf, sb.len);\n \t\tstrbuf_reset(&sb);\n \t\tcount -= p + 1 - buf;\n \t\tbuf = p + 1;\n+\t\tprint = 1;\n \t}\n+\n out:\n-\tif (sb.len)\n-\t\temit_line(o, NULL, NULL, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n \treturn 0;\n }\n@@ -1256,14 +1259,12 @@ static void fn_out_diff_words_aux(void *priv, char *line, unsigned long len)\n \tint minus_first, minus_len, plus_first, plus_len;\n \tconst char *minus_begin, *minus_end, *plus_begin, *plus_end;\n \tstruct diff_options *opt = diff_words->opt;\n-\tconst char *line_prefix;\n \n \tif (line[0] != '@' || parse_hunk_header(line, len,\n \t\t\t&minus_first, &minus_len, &plus_first, &plus_len))\n \t\treturn;\n \n \tassert(opt);\n-\tline_prefix = diff_line_prefix(opt);\n \n \t/* POSIX requires that first be decremented by one if len == 0... */\n \tif (minus_len) {\n@@ -1280,28 +1281,21 @@ static void fn_out_diff_words_aux(void *priv, char *line, unsigned long len)\n \t} else\n \t\tplus_begin = plus_end = diff_words->plus.orig[plus_first].end;\n \n-\tif (color_words_output_graph_prefix(diff_words))\n-\t\temit_line(diff_words->opt, NULL, NULL, \"\", 0);\n-\n \tif (diff_words->current_plus != plus_begin) {\n \t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t\t&style->ctx, style->newline,\n \t\t\t\tplus_begin - diff_words->current_plus,\n-\t\t\t\tdiff_words->current_plus, line_prefix);\n-\t\tif (*(plus_begin - 1) == '\\n')\n-\t\t\temit_line(diff_words->opt, NULL, NULL, \"\", 0);\n+\t\t\t\tdiff_words->current_plus);\n \t}\n \tif (minus_begin != minus_end) {\n \t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t\t&style->old, style->newline,\n-\t\t\t\tminus_end - minus_begin, minus_begin,\n-\t\t\t\tline_prefix);\n+\t\t\t\tminus_end - minus_begin, minus_begin);\n \t}\n \tif (plus_begin != plus_end) {\n \t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t\t&style->new, style->newline,\n-\t\t\t\tplus_end - plus_begin, plus_begin,\n-\t\t\t\tline_prefix);\n+\t\t\t\tplus_end - plus_begin, plus_begin);\n \t}\n \n \tdiff_words->current_plus = plus_end;\n@@ -1388,17 +1382,14 @@ static void diff_words_show(struct diff_words_data *diff_words)\n \tstruct diff_words_style *style = diff_words->style;\n \n \tstruct diff_options *opt = diff_words->opt;\n-\tconst char *line_prefix;\n-\n \tassert(opt);\n-\tline_prefix = diff_line_prefix(opt);\n \n \t/* special case: only removal */\n \tif (!diff_words->plus.text.size) {\n \t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t&style->old, style->newline,\n \t\t\tdiff_words->minus.text.size,\n-\t\t\tdiff_words->minus.text.ptr, line_prefix);\n+\t\t\tdiff_words->minus.text.ptr);\n \t\tdiff_words->minus.text.size = 0;\n \t\treturn;\n \t}\n@@ -1421,12 +1412,11 @@ static void diff_words_show(struct diff_words_data *diff_words)\n \tif (diff_words->current_plus != diff_words->plus.text.ptr +\n \t\t\tdiff_words->plus.text.size) {\n \t\tif (color_words_output_graph_prefix(diff_words))\n-\t\t\temit_line(diff_words->opt, NULL, NULL, \"\", 0);\n+\t\t\temit_line(diff_words->opt, NULL, NULL, 1, 0, 0, \"\", 0);\n \t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t&style->ctx, style->newline,\n \t\t\tdiff_words->plus.text.ptr + diff_words->plus.text.size\n-\t\t\t- diff_words->current_plus, diff_words->current_plus,\n-\t\t\tline_prefix);\n+\t\t\t- diff_words->current_plus, diff_words->current_plus);\n \t}\n \tdiff_words->minus.text.size = diff_words->plus.text.size = 0;\n }\n@@ -1444,8 +1434,10 @@ static void diff_words_flush(struct emit_callback *ecbdata)\n \t\t\tappend_buffered_patch_line(ecbdata->opt,\n \t\t\t\t&ecbdata->diff_words->opt->line_buffer[i]);\n \n+\t\tfor (i = 0; i < ecbdata->diff_words->opt->line_buffer_nr; i++)\n+\t\t\tfree((void*) ecbdata->diff_words->opt->line_buffer[i].line);\n+\n \t\tecbdata->diff_words->opt->line_buffer_nr = 0;\n-\t\t/* TODO: free memory as well */\n \t}\n }\n \n@@ -1521,6 +1513,7 @@ static void free_diff_words_data(struct emit_callback *ecbdata)\n {\n \tif (ecbdata->diff_words) {\n \t\tdiff_words_flush(ecbdata);\n+\t\tfree (ecbdata->diff_words->opt->line_buffer);\n \t\tfree (ecbdata->diff_words->opt);\n \t\tfree (ecbdata->diff_words->minus.text.ptr);\n \t\tfree (ecbdata->diff_words->minus.orig);\n@@ -1596,7 +1589,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \to->found_changes = 1;\n \n \tif (ecbdata->header) {\n-\t\temit_line(o, NULL, NULL,\n+\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t ecbdata->header->buf, ecbdata->header->len);\n \t\tstrbuf_release(ecbdata->header);\n \t\tecbdata->header = NULL;\n@@ -1606,9 +1599,9 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\tconst char *name_a_tab, *name_b_tab;\n \t\tname_a_tab = strchr(ecbdata->label_path[0], ' ') ? \"\\t\" : \"\";\n \t\tname_b_tab = strchr(ecbdata->label_path[1], ' ') ? \"\\t\" : \"\";\n-\t\temit_line_fmt(o, meta, reset, \"--- %s%s\\n\",\n+\t\temit_line_fmt(o, meta, reset, 1, \"--- %s%s\\n\",\n \t\t\t ecbdata->label_path[0], name_a_tab);\n-\t\temit_line_fmt(o, meta, reset, \"+++ %s%s\\n\",\n+\t\temit_line_fmt(o, meta, reset, 1, \"+++ %s%s\\n\",\n \t\t\t ecbdata->label_path[1], name_b_tab);\n \t\tecbdata->label_path[0] = ecbdata->label_path[1] = NULL;\n \t}\n@@ -1649,8 +1642,8 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t}\n \t\tdiff_words_flush(ecbdata);\n \t\tif (ecbdata->diff_words->type == DIFF_WORDS_PORCELAIN) {\n-\t\t\temit_line(o, context, reset, line, len);\n-\t\t\temit_line(o, NULL, NULL, \"~\\n\", 2);\n+\t\t\temit_line(o, context, reset, 1, 0, 0, line, len);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0, \"~\\n\", 2);\n \t\t} else {\n \t\t\t/*\n \t\t\t * Skip the prefix character, if any. With\n@@ -1661,7 +1654,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t\t line++;\n \t\t\t len--;\n \t\t\t}\n-\t\t\temit_line(o, context, reset, line, len);\n+\t\t\temit_line(o, context, reset, 1, 0, 0, line, len);\n \t\t}\n \t\treturn;\n \t}\n@@ -1684,7 +1677,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t/* incomplete line at the end */\n \t\tecbdata->lno_in_preimage++;\n \t\temit_line(o, diff_get_color(ecbdata->color_diff, DIFF_CONTEXT),\n-\t\t\t reset, line, len);\n+\t\t\t reset, 1, 0, 0, line, len);\n \t\tbreak;\n \t}\n }\n@@ -1873,7 +1866,7 @@ void print_stat_summary_0(struct diff_options *options, int files,\n \tif (!files) {\n \t\tassert(insertions == 0 && deletions == 0);\n \t\tstrbuf_addstr(&sb, \" 0 files changed\");\n-\t\temit_line(options, NULL, NULL, sb.buf, sb.len);\n+\t\temit_line(options, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \t\treturn;\n \t}\n \n@@ -1901,7 +1894,7 @@ void print_stat_summary_0(struct diff_options *options, int files,\n \t\t\t deletions);\n \t}\n \tstrbuf_addch(&sb, '\\n');\n-\temit_line(options, NULL, NULL, sb.buf, sb.len);\n+\temit_line(options, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n }\n \n@@ -2085,7 +2078,7 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t\tstrbuf_addf(&out, \" %*s\", number_width, \"Bin\");\n \t\t\tif (!added && !deleted) {\n \t\t\t\tstrbuf_addch(&out, '\\n');\n-\t\t\t\temit_line(options, NULL, NULL, out.buf, out.len);\n+\t\t\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\t\t\tstrbuf_reset(&out);\n \t\t\t\tcontinue;\n \t\t\t}\n@@ -2095,14 +2088,14 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t\tstrbuf_addf(&out, \"%s%\"PRIuMAX\"%s\",\n \t\t\t\tadd_c, added, reset);\n \t\t\tstrbuf_addstr(&out, \" bytes\\n\");\n-\t\t\temit_line(options, NULL, NULL, out.buf, out.len);\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n \t\telse if (file->is_unmerged) {\n \t\t\tshow_name(&out, prefix, name, len);\n \t\t\tstrbuf_addstr(&out, \" Unmerged\\n\");\n-\t\t\temit_line(options, NULL, NULL, out.buf, out.len);\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n@@ -2133,7 +2126,7 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\tshow_graph(&out, '+', add, add_c, reset);\n \t\tshow_graph(&out, '-', del, del_c, reset);\n \t\tstrbuf_addch(&out, '\\n');\n-\t\temit_line(options, NULL, NULL, out.buf, out.len);\n+\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\tstrbuf_reset(&out);\n \t}\n \n@@ -2155,7 +2148,7 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\tif (i < count)\n \t\t\tcontinue;\n \t\tif (!extra_shown)\n-\t\t\temit_line(options, NULL, NULL,\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0,\n \t\t\t\t \" ...\\n\", strlen(\" ...\\n\"));\n \t\textra_shown = 1;\n \t}\n@@ -2509,7 +2502,7 @@ static void checkdiff_consume(void *priv, char *line, unsigned long len)\n \t\tfprintf(data->o->file, \"%s%s:%d: %s.\\n\",\n \t\t\tline_prefix, data->filename, data->lineno, err);\n \t\tfree(err);\n-\t\temit_line(data->o, set, reset, line, 1);\n+\t\temit_line(data->o, set, reset, 1, 0, 0, line, 1);\n \t\tws_check_emit(line + 1, len - 1, data->ws_rule,\n \t\t\t data->o->file, set, reset, ws);\n \t} else if (line[0] == ' ') {\n@@ -2576,12 +2569,12 @@ static void emit_binary_diff_body(struct diff_options *o,\n \t}\n \n \tif (delta && delta_size < deflate_size) {\n-\t\temit_line_fmt(o, NULL, NULL, \"delta %lu\\n\", orig_size);\n+\t\temit_line_fmt(o, NULL, NULL, 1, \"delta %lu\\n\", orig_size);\n \t\tfree(deflated);\n \t\tdata = delta;\n \t\tdata_size = delta_size;\n \t} else {\n-\t\temit_line_fmt(o, NULL, NULL, \"literal %lu\\n\", two->size);\n+\t\temit_line_fmt(o, NULL, NULL, 1, \"literal %lu\\n\", two->size);\n \t\tfree(delta);\n \t\tdata = deflated;\n \t\tdata_size = deflate_size;\n@@ -2605,9 +2598,9 @@ static void emit_binary_diff_body(struct diff_options *o,\n \t\tline[len++] = '\\n';\n \t\tline[len] = '\\0';\n \n-\t\temit_line(o, NULL, NULL, line, len);\n+\t\temit_line(o, NULL, NULL, 1, 0, 0, line, len);\n \t}\n-\temit_line(o, NULL, NULL, \"\\n\", 1);\n+\temit_line(o, NULL, NULL, 1, 0, 0, \"\\n\", 1);\n \tfree(data);\n }\n \n@@ -2616,7 +2609,7 @@ static void emit_binary_diff(struct diff_options *o,\n {\n \tconst char *s = \"GIT binary patch\\n\";\n \tconst int len = strlen(s);\n-\temit_line(o, NULL, NULL, s, len);\n+\temit_line(o, NULL, NULL, 1, 0, 0, s, len);\n \temit_binary_diff_body(o, one, two);\n \temit_binary_diff_body(o, two, one);\n }\n@@ -2727,7 +2720,7 @@ static void builtin_diff(const char *name_a,\n \tb_two = quote_two(b_prefix, name_b + (*name_b == '/'));\n \tlbl[0] = DIFF_FILE_VALID(one) ? a_one : \"/dev/null\";\n \tlbl[1] = DIFF_FILE_VALID(two) ? b_two : \"/dev/null\";\n-\tstrbuf_addf(&header, \"%sdiff --git %s %s%s\\n\", meta, a_one, b_two, reset);\n+\tstrbuf_addf(&header, \"%s%sdiff --git %s %s%s\\n\", line_prefix, meta, a_one, b_two, reset);\n \tif (lbl[0][0] == '/') {\n \t\t/* /dev/null */\n \t\tstrbuf_addf(&header, \"%s%snew file mode %06o%s\\n\", line_prefix, meta, two->mode, reset);\n@@ -2759,7 +2752,7 @@ static void builtin_diff(const char *name_a,\n \t\tif (complete_rewrite &&\n \t\t (textconv_one || !diff_filespec_is_binary(one)) &&\n \t\t (textconv_two || !diff_filespec_is_binary(two))) {\n-\t\t\temit_line(o, NULL, NULL, header.buf, header.len);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t\temit_rewrite_diff(name_a, name_b, one, two,\n \t\t\t\t\t\ttextconv_one, textconv_two, o);\n@@ -2769,8 +2762,7 @@ static void builtin_diff(const char *name_a,\n \t}\n \n \tif (o->irreversible_delete && lbl[1][0] == '/') {\n-\t\tif (header.len)\n-\t\t\temit_line(o, NULL, NULL, header.buf, header.len);\n+\t\temit_line(o, NULL, NULL, 0, 0, 0, header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tgoto free_ab_and_return;\n \t} else if (!DIFF_OPT_TST(o, TEXT) &&\n@@ -2780,15 +2772,15 @@ static void builtin_diff(const char *name_a,\n \t\t S_ISREG(one->mode) && S_ISREG(two->mode) &&\n \t\t !DIFF_OPT_TST(o, BINARY)) {\n \t\t\tif (!oidcmp(&one->oid, &two->oid)) {\n-\t\t\t\tif (must_show_header && header.len)\n-\t\t\t\t\temit_line(o, NULL, NULL,\n+\t\t\t\tif (must_show_header)\n+\t\t\t\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t\t\t\t header.buf, header.len);\n \t\t\t\tgoto free_ab_and_return;\n \t\t\t}\n-\t\t\tif (header.len)\n-\t\t\t\temit_line(o, NULL, NULL,\n-\t\t\t\t\t header.buf, header.len);\n-\t\t\temit_line_fmt(o, 0, 0, \"Binary files %s and %s differ\\n\",\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0,\n+\t\t\t\t header.buf, header.len);\n+\t\t\temit_line_fmt(o, NULL, NULL, 1,\n+\t\t\t\t \"Binary files %s and %s differ\\n\",\n \t\t\t\t lbl[0], lbl[1]);\n \t\t\tgoto free_ab_and_return;\n \t\t}\n@@ -2797,19 +2789,18 @@ static void builtin_diff(const char *name_a,\n \t\t/* Quite common confusing case */\n \t\tif (mf1.size == mf2.size &&\n \t\t !memcmp(mf1.ptr, mf2.ptr, mf1.size)) {\n-\t\t\tif (must_show_header && header.len)\n-\t\t\t\temit_line(o, NULL, NULL,\n+\t\t\tif (must_show_header)\n+\t\t\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t\t\t header.buf, header.len);\n \t\t\tgoto free_ab_and_return;\n \t\t}\n-\t\tif (header.len)\n-\t\t\temit_line(o, NULL, NULL,\n-\t\t\t\t header.buf, header.len);\n+\t\temit_line(o, NULL, NULL, 0, 0, 0,\n+\t\t\t header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tif (DIFF_OPT_TST(o, BINARY))\n \t\t\temit_binary_diff(o, &mf1, &mf2);\n \t\telse\n-\t\t\temit_line_fmt(o, NULL, NULL,\n+\t\t\temit_line_fmt(o, NULL, NULL, 1,\n \t\t\t\t \"Binary files %s and %s differ\\n\",\n \t\t\t\t lbl[0], lbl[1]);\n \t\to->found_changes = 1;\n@@ -2822,8 +2813,8 @@ static void builtin_diff(const char *name_a,\n \t\tstruct emit_callback ecbdata;\n \t\tconst struct userdiff_funcname *pe;\n \n-\t\tif (must_show_header && header.len) {\n-\t\t\temit_line(o, NULL, NULL, header.buf, header.len);\n+\t\tif (must_show_header) {\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t}\n \n@@ -2840,6 +2831,7 @@ static void builtin_diff(const char *name_a,\n \t\tecbdata.label_path = lbl;\n \t\tecbdata.color_diff = want_color(o->use_color);\n \t\tecbdata.ws_rule = whitespace_rule(name_b);\n+\t\to->ws_rule = ecbdata.ws_rule;\n \t\tif (ecbdata.ws_rule & WS_BLANK_AT_EOF)\n \t\t\tcheck_blank_at_eof(&mf1, &mf2, &ecbdata);\n \t\tecbdata.opt = o;\n@@ -2861,24 +2853,15 @@ static void builtin_diff(const char *name_a,\n \t\tif (o->word_diff)\n \t\t\tinit_diff_words_data(&ecbdata, o, one, two);\n \t\tif (o->use_buffer) {\n-\t\t\tALLOC_GROW(o->filepair_buffer,\n-\t\t\t\t o->filepair_buffer_nr + 1,\n-\t\t\t\t o->filepair_buffer_alloc);\n-\t\t\to->current_filepair =\n-\t\t\t\t&o->filepair_buffer[o->filepair_buffer_nr++];\n-\n-\t\t\to->current_filepair->ws_rule = ecbdata.ws_rule;\n-\t\t\to->current_filepair->ws =\n-\t\t\t\tdiff_get_color(ecbdata.color_diff, DIFF_WHITESPACE);\n+\t\t\tstruct buffered_patch_line e = BUFFERED_PATCH_LINE_INIT;\n+\t\t\te.state = BPL_HANDOVER;\n+\t\t\te.line = name_b;\n+\t\t\te.len = strlen(name_b);\n+\t\t\tappend_buffered_patch_line(o, &e);\n \t\t}\n \t\tif (xdi_diff_outf(&mf1, &mf2, fn_out_consume, &ecbdata,\n \t\t\t\t &xpp, &xecfg))\n \t\t\tdie(\"unable to generate diff for %s\", one->path);\n-\t\tif (o->use_buffer) {\n-\t\t\tstruct buffered_patch_line e = BUFFERED_PATCH_LINE_INIT;\n-\t\t\te.state = BPL_HANDOVER; /* handover to next file pair */\n-\t\t\tappend_buffered_patch_line(o, &e);\n-\t\t}\n \t\tif (o->word_diff)\n \t\t\tfree_diff_words_data(&ecbdata);\n \t\tif (textconv_one)\n@@ -3755,6 +3738,12 @@ void diff_setup(struct diff_options *options)\n \t\toptions->a_prefix = \"a/\";\n \t\toptions->b_prefix = \"b/\";\n \t}\n+\n+\toptions->line_buffer = NULL;\n+\toptions->line_buffer_nr = 0;\n+\toptions->line_buffer_alloc = 0;\n+\n+\toptions->color_moved = diff_color_moved_default;\n }\n \n void diff_setup_done(struct diff_options *options)\n@@ -4837,11 +4826,10 @@ static void show_file_mode_name(struct diff_options *opt, const char *newdelete,\n \n \tquote_c_style(fs->path, &sb, NULL, 0);\n \tstrbuf_addch(&sb, '\\n');\n-\temit_line(opt, NULL, NULL, sb.buf, sb.len);\n+\temit_line(opt, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n }\n \n-\n static void show_mode_change(struct diff_options *opt, struct diff_filepair *p,\n \t\tint show_name)\n {\n@@ -4851,7 +4839,7 @@ static void show_mode_change(struct diff_options *opt, struct diff_filepair *p,\n \t\t\tstrbuf_addch(&sb, ' ');\n \t\t\tquote_c_style(p->two->path, &sb, NULL, 0);\n \t\t}\n-\t\temit_line_fmt(opt, NULL, NULL,\n+\t\temit_line_fmt(opt, NULL, NULL, 1,\n \t\t\t \" mode change %06o => %06o%s\\n\",\n \t\t\t p->one->mode, p->two->mode,\n \t\t\t show_name ? sb.buf : \"\");\n@@ -4863,7 +4851,8 @@ static void show_rename_copy(struct diff_options *opt, const char *renamecopy,\n \t\tstruct diff_filepair *p)\n {\n \tchar *names = pprint_rename(p->one->path, p->two->path);\n-\temit_line_fmt(opt, NULL, NULL, \" %s %s (%d%%)\\n\", renamecopy, names, similarity_index(p));\n+\temit_line_fmt(opt, NULL, NULL, 1, \" %s %s (%d%%)\\n\",\n+\t\t renamecopy, names, similarity_index(p));\n \tfree(names);\n \tshow_mode_change(opt, p, 0);\n }\n@@ -4889,7 +4878,7 @@ static void diff_summary(struct diff_options *opt, struct diff_filepair *p)\n \t\t\tstrbuf_addstr(&sb, \" rewrite \");\n \t\t\tquote_c_style(p->two->path, &sb, NULL, 0);\n \t\t\tstrbuf_addf(&sb, \" (%d%%)\\n\", similarity_index(p));\n-\t\t\temit_line(opt, NULL, NULL, sb.buf, sb.len);\n+\t\t\temit_line(opt, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \t\t}\n \t\tshow_mode_change(opt, p, !p->score);\n \t\tbreak;\n@@ -5114,12 +5103,6 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \t\t\t(hashmap_cmp_fn)moved_entry_cmp, 0);\n \t}\n \n-\tif (o->use_buffer) {\n-\t\tALLOC_GROW(o->filepair_buffer,\n-\t\t\t o->filepair_buffer_nr + 1,\n-\t\t\t o->filepair_buffer_alloc);\n-\t\to->current_filepair = &o->filepair_buffer[o->filepair_buffer_nr];\n-\t}\n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n \t\tif (check_pair_status(p))\n@@ -5127,23 +5110,22 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \t}\n \n \tif (o->use_buffer) {\n-\t\to->current_filepair = &o->filepair_buffer[0];\n-\t\tfor (i = 0; i < o->line_buffer_nr; i++)\n-\t\t\tadd_line_to_move_detection(o, i);\n+\t\tif (o->color_moved) {\n+\t\t\tadd_lines_to_move_detection(o);\n+\t\t\tmark_color_as_moved(o);\n+\t\t}\n \n-\t\to->current_filepair = &o->filepair_buffer[0];\n \t\tfor (i = 0; i < o->line_buffer_nr; i++)\n-\t\t\tprocess_next_buffered_patch_line(o, i);\n+\t\t\temit_buffered_patch_line(o, &o->line_buffer[i]);\n \n-\t\tfor (i = 0; i < o->line_buffer_nr; i++);\n+\t\tfor (i = 0; i < o->line_buffer_nr; i++)\n \t\t\tfree((void*)o->line_buffer[i].line);\n \n+\t\tfree(o->line_buffer);\n+\n \t\to->line_buffer = NULL;\n \t\to->line_buffer_nr = 0;\n-\t\tfree(o->line_buffer);\n-\t\to->filepair_buffer = NULL;\n-\t\to->filepair_buffer_nr = 0;\n-\t\tfree(o->filepair_buffer);\n+\t\to->line_buffer_alloc = 0;\n \t}\n }\n \n@@ -5237,11 +5219,10 @@ void diff_flush(struct diff_options *options)\n \t\t\tterm[0] = options->line_termination;\n \t\t\tterm[1] = '\\0';\n \n-\t\t\temit_line(options, NULL, NULL,\n-\t\t\t\t term, 1);\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0, term, !!term[0]);\n \t\t\tif (options->stat_sep) {\n \t\t\t\t/* attach patch instead of inline */\n-\t\t\t\temit_line(options, NULL, NULL,\n+\t\t\t\temit_line(options, NULL, NULL, 0, 0, 0,\n \t\t\t\t\t options->stat_sep,\n \t\t\t\t\t strlen(options->stat_sep));\n \t\t\t}\ndiff --git a/diff.h b/diff.h\nindex b83d6fefcc..b8b2a33ccc 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -133,21 +133,24 @@ struct buffered_patch_line {\n \tconst char *line;\n \tint len;\n \tint sign;\n+\tint add_line_prefix;\n \tenum {\n+\t\t/*\n+\t\t * Emits [lineprefix][set][sign][reset] and then calls\n+\t\t * ws_check_emit which will output \"line\", marked up\n+\t\t * according to ws_rule.\n+\t\t */\n \t\tBPL_EMIT_LINE_WS,\n+\n+\t\t/* Emits [lineprefix][set][sign] line [reset] */\n \t\tBPL_EMIT_LINE_ASIS,\n+\n+\t\t/* Reloads the ws_rule; line contains the file name */\n \t\tBPL_HANDOVER\n \t} state;\n };\n #define BUFFERED_PATCH_LINE_INIT {NULL, NULL, NULL, 0, 0, 0}\n \n-struct buffered_filepair {\n-\tconst char *ws;\n-\tunsigned ws_rule;\n-};\n-\n-struct moved_entry;\n-\n struct diff_options {\n \tconst char *orderfile;\n \tconst char *pickaxe;\n@@ -220,32 +223,21 @@ struct diff_options {\n \n \tint diff_path_counter;\n \n-\t/* Determines color moved code. Flipped between 1, 2 for alt. color. */\n-\tint color_moved;\n+\tunsigned ws_rule;\n \tint use_buffer;\n \n \tstruct buffered_patch_line *line_buffer;\n \tint line_buffer_nr, line_buffer_alloc;\n \n-\tstruct buffered_filepair *filepair_buffer;\n-\tint filepair_buffer_nr, filepair_buffer_alloc;\n-\tstruct buffered_filepair *current_filepair;\n-\n-\t/* built up in the first pass: */\n+\tint color_moved;\n \tstruct hashmap *deleted_lines;\n \tstruct hashmap *added_lines;\n-\t/* needed for building up */\n-\tstruct moved_entry *prev_line;\n-\n-\t/* state in the second pass */\n-\tstruct moved_entry **pmb; /* potentially moved blocks */\n-\tint pmb_nr, pmb_alloc;\n };\n \n void emit_line_fmt(struct diff_options *o, const char *set, const char *reset,\n-\t\t const char *fmt, ...);\n+\t\t int add_line_prefix, const char *fmt, ...);\n void emit_line(struct diff_options *o, const char *set, const char *reset,\n-\t const char *line, int len);\n+\t int add_line_prefix, int markup_ws, int sign, const char *line, int len);\n \n enum color_diff {\n \tDIFF_RESET = 0,\ndiff --git a/submodule.c b/submodule.c\nindex cfad469a2f..868f913971 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -378,9 +378,9 @@ static void print_submodule_summary(struct rev_info *rev,\n \t\tformat_commit_message(commit, format, &sb, &ctx);\n \t\tstrbuf_addch(&sb, '\\n');\n \t\tif (commit->object.flags & SYMMETRIC_LEFT)\n-\t\t\temit_line(o, del, reset, sb.buf, sb.len);\n+\t\t\temit_line(o, del, reset, 1, 0, 0, sb.buf, sb.len);\n \t\telse if (add)\n-\t\t\temit_line(o, add, reset, sb.buf, sb.len);\n+\t\t\temit_line(o, add, reset, 1, 0, 0, sb.buf, sb.len);\n \t}\n \tstrbuf_release(&sb);\n }\n@@ -419,10 +419,10 @@ static void show_submodule_header(struct diff_options *o, const char *path,\n \tint fast_forward = 0, fast_backward = 0;\n \n \tif (dirty_submodule & DIRTY_SUBMODULE_UNTRACKED)\n-\t\temit_line_fmt(o, NULL, NULL,\n+\t\temit_line_fmt(o, NULL, NULL, 1,\n \t\t\t \"Submodule %s contains untracked content\\n\", path);\n \tif (dirty_submodule & DIRTY_SUBMODULE_MODIFIED)\n-\t\temit_line_fmt(o, NULL, NULL,\n+\t\temit_line_fmt(o, NULL, NULL, 1,\n \t\t\t \"Submodule %s contains modified content\\n\", path);\n \n \tif (is_null_oid(one))\n@@ -473,7 +473,7 @@ static void show_submodule_header(struct diff_options *o, const char *path,\n \t\tstrbuf_addf(&sb, \" %s\\n\", message);\n \telse\n \t\tstrbuf_addf(&sb, \"%s:\\n\", fast_backward ? \" (rewind)\" : \"\");\n-\temit_line(o, meta, reset, sb.buf, sb.len);\n+\temit_line(o, meta, reset, 1, 0, 0, sb.buf, sb.len);\n \n \tstrbuf_release(&sb);\n }\n@@ -501,7 +501,7 @@ void show_submodule_summary(struct diff_options *o, const char *path,\n \t/* Treat revision walker failure the same as missing commits */\n \tif (prepare_submodule_summary(&rev, path, left, right, merge_bases)) {\n \t\tconst char *error = \"(revision walker failed)\\n\";\n-\t\temit_line(o, NULL, NULL, error, strlen(error));\n+\t\temit_line(o, NULL, NULL, 1, 0, 0, error, strlen(error));\n \t\tgoto out;\n \t}\n \n@@ -570,15 +570,15 @@ void show_submodule_inline_diff(struct diff_options *o, const char *path,\n \tprepare_submodule_repo_env(&cp.env_array);\n \tif (start_command(&cp)) {\n \t\tconst char *error = \"(diff failed)\\n\";\n-\t\temit_line(o, NULL, NULL, error, strlen(error));\n+\t\temit_line(o, NULL, NULL, 1, 0, 0, error, strlen(error));\n \t}\n \n \twhile (strbuf_getwholeline_fd(&sb, cp.out, '\\n') != EOF)\n-\t\temit_line(o, NULL, NULL, sb.buf, sb.len);\n+\t\temit_line(o, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \n \tif (finish_command(&cp)) {\n \t\tconst char *error = \"(diff failed)\\n\";\n-\t\temit_line(o, NULL, NULL, error, strlen(error));\n+\t\temit_line(o, NULL, NULL, 1, 0, 0, error, strlen(error));\n \t}\n \n done:\n\n\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"319993","messageId":"20170517025857.32320-2-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 01/20] diff: readability fix","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:38Z","receivedAt":"2017-05-17T02:59:15Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"We already have dereferenced 'p->two' into a local variable 'two'. Use\nthat.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 74283d9001..3f5bf8b5a4 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -3283,8 +3283,8 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)\n \tconst char *other;\n \tconst char *attr_path;\n \n-\tname = p->one->path;\n-\tother = (strcmp(name, p->two->path) ? p->two->path : NULL);\n+\tname = one->path;\n+\tother = (strcmp(name, two->path) ? two->path : NULL);\n \tattr_path = name;\n \tif (o->prefix_length)\n \t\tstrip_prefix(o->prefix_length, &name, &other);\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"319994","messageId":"20170517025857.32320-3-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 02/20] diff: move line ending check into emit_hunk_header","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:39Z","receivedAt":"2017-05-17T02:59:16Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"The emit_hunk_header() function is responsible for assembling a\nhunk header and calling emit_line() to send the hunk header\nto the output file. Its only caller fn_out_consume() needs\nto prepare for a case where the function emits an incomplete\nline and add the terminating LF.\n\nInstead make sure emit_hunk_header() to always send a\ncompleted line to emit_line().\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 3f5bf8b5a4..c2ed605cd0 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -677,6 +677,8 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \t}\n \n \tstrbuf_add(&msgbuf, line + len, org_len - len);\n+\tstrbuf_complete_line(&msgbuf);\n+\n \temit_line(ecbdata->opt, \"\", \"\", msgbuf.buf, msgbuf.len);\n \tstrbuf_release(&msgbuf);\n }\n@@ -1315,8 +1317,6 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\tlen = sane_truncate_line(ecbdata, line, len);\n \t\tfind_lno(line, ecbdata);\n \t\temit_hunk_header(ecbdata, line, len);\n-\t\tif (line[len-1] != '\\n')\n-\t\t\tputc('\\n', o->file);\n \t\treturn;\n \t}\n \n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"319995","messageId":"20170517025857.32320-4-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 03/20] diff.c: factor out diff_flush_patch_all_file_pairs","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:40Z","receivedAt":"2017-05-17T02:59:17Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch we want to do more things before and after all filepairs\nare flushed. So factor flushing out all file pairs into its own function\nthat the new code can be plugged in easily.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 17 ++++++++++++-----\n 1 file changed, 12 insertions(+), 5 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex c2ed605cd0..2f9722b382 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4737,6 +4737,17 @@ void diff_warn_rename_limit(const char *varname, int needed, int degraded_cc)\n \t\twarning(_(rename_limit_advice), varname, needed);\n }\n \n+static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n+{\n+\tint i;\n+\tstruct diff_queue_struct *q = &diff_queued_diff;\n+\tfor (i = 0; i < q->nr; i++) {\n+\t\tstruct diff_filepair *p = q->queue[i];\n+\t\tif (check_pair_status(p))\n+\t\t\tdiff_flush_patch(p, o);\n+\t}\n+}\n+\n void diff_flush(struct diff_options *options)\n {\n \tstruct diff_queue_struct *q = &diff_queued_diff;\n@@ -4831,11 +4842,7 @@ void diff_flush(struct diff_options *options)\n \t\t\t}\n \t\t}\n \n-\t\tfor (i = 0; i < q->nr; i++) {\n-\t\t\tstruct diff_filepair *p = q->queue[i];\n-\t\t\tif (check_pair_status(p))\n-\t\t\t\tdiff_flush_patch(p, options);\n-\t\t}\n+\t\tdiff_flush_patch_all_file_pairs(options);\n \t}\n \n \tif (output_format & DIFF_FORMAT_CALLBACK)\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"319996","messageId":"20170517025857.32320-5-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 04/20] diff.c: teach emit_line_0 to accept sign parameter","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:41Z","receivedAt":"2017-05-17T02:59:19Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Teach emit_line_0 take an optional \"sign\" parameter specifically intended\nto hold the sign of the line instead of a separate \"first\" parameter\nrepresenting the first character of the line to be printed. Callers\nthat store the sign and line separately can use the \"sign\" parameter\nlike they used the \"first\" parameter previously, and callers that store\nthe sign and line together (or do not have a sign) no longer need to\nmanipulate their arguments to fit the requirements of emit_line_0.\n\nWith this patch other callers hard code the sign (which are '+', '-',\n' ' and '\\\\') such that we do not run into unexpectedly emitting an\nerroneous '\\0'.\n\nThe audit of the caller revealed that the sign cannot be '\\n' or '\\r',\nso remove that condition for trailing newline or carriage return in the\nsign; the else part of the condition handles the len==0 perfectly,\nso we can drop the if/else construct.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 39 ++++++++++++++++-----------------------\n 1 file changed, 16 insertions(+), 23 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 2f9722b382..73e55b0c10 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -517,33 +517,24 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n }\n \n static void emit_line_0(struct diff_options *o, const char *set, const char *reset,\n-\t\t\tint first, const char *line, int len)\n+\t\t\tint sign, const char *line, int len)\n {\n \tint has_trailing_newline, has_trailing_carriage_return;\n-\tint nofirst;\n \tFILE *file = o->file;\n \n \tfputs(diff_line_prefix(o), file);\n \n-\tif (len == 0) {\n-\t\thas_trailing_newline = (first == '\\n');\n-\t\thas_trailing_carriage_return = (!has_trailing_newline &&\n-\t\t\t\t\t\t(first == '\\r'));\n-\t\tnofirst = has_trailing_newline || has_trailing_carriage_return;\n-\t} else {\n-\t\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n-\t\tif (has_trailing_newline)\n-\t\t\tlen--;\n-\t\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n-\t\tif (has_trailing_carriage_return)\n-\t\t\tlen--;\n-\t\tnofirst = 0;\n-\t}\n+\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n+\tif (has_trailing_newline)\n+\t\tlen--;\n+\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n+\tif (has_trailing_carriage_return)\n+\t\tlen--;\n \n-\tif (len || !nofirst) {\n+\tif (len || sign) {\n \t\tfputs(set, file);\n-\t\tif (!nofirst)\n-\t\t\tfputc(first, file);\n+\t\tif (sign)\n+\t\t\tfputc(sign, file);\n \t\tfwrite(line, len, 1, file);\n \t\tfputs(reset, file);\n \t}\n@@ -556,7 +547,7 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n static void emit_line(struct diff_options *o, const char *set, const char *reset,\n \t\t const char *line, int len)\n {\n-\temit_line_0(o, set, reset, line[0], line+1, len-1);\n+\temit_line_0(o, set, reset, 0, line, len);\n }\n \n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n@@ -4833,9 +4824,11 @@ void diff_flush(struct diff_options *options)\n \n \tif (output_format & DIFF_FORMAT_PATCH) {\n \t\tif (separator) {\n-\t\t\tfprintf(options->file, \"%s%c\",\n-\t\t\t\tdiff_line_prefix(options),\n-\t\t\t\toptions->line_termination);\n+\t\t\tchar term[2];\n+\t\t\tterm[0] = options->line_termination;\n+\t\t\tterm[1] = '\\0';\n+\n+\t\t\temit_line(options, NULL, NULL, term, !!term[0]);\n \t\t\tif (options->stat_sep) {\n \t\t\t\t/* attach patch instead of inline */\n \t\t\t\tfputs(options->stat_sep, options->file);\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"319997","messageId":"20170517025857.32320-8-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 07/20] diff.c: inline emit_line_0 into emit_line","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:44Z","receivedAt":"2017-05-17T02:59:22Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"The argument list of emit_line_0 is just 2 more arguments that are\nhard-coded in emit_line. Eliminate this intermediate function and\nrename the remaining function by dropping the '_0'.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 34 ++++++++++++++--------------------\n 1 file changed, 14 insertions(+), 20 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 25735f03d2..3569857818 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -516,8 +516,8 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n-static void emit_line_0(struct diff_options *o, const char *set, const char *reset,\n-\t\t\tint add_line_prefix, int sign, const char *line, int len)\n+static void emit_line(struct diff_options *o, const char *set, const char *reset,\n+\t\t int add_line_prefix, int sign, const char *line, int len)\n {\n \tint has_trailing_newline, has_trailing_carriage_return;\n \tFILE *file = o->file;\n@@ -547,12 +547,6 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n \t\tfputc('\\n', file);\n }\n \n-static void emit_line(struct diff_options *o, const char *set, const char *reset,\n-\t\t const char *line, int len)\n-{\n-\temit_line_0(o, set, reset, 1, 0, line, len);\n-}\n-\n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n {\n \tif (!((ecbdata->ws_rule & WS_BLANK_AT_EOF) &&\n@@ -581,13 +575,13 @@ static void emit_line_checked(const char *reset,\n \t}\n \n \tif (!ws)\n-\t\temit_line_0(ecbdata->opt, set, reset, 1, sign, line, len);\n+\t\temit_line(ecbdata->opt, set, reset, 1, sign, line, len);\n \telse if (sign == '+' && new_blank_line_at_eof(ecbdata, line, len))\n \t\t/* Blank line at EOF - paint '+' as well */\n-\t\temit_line_0(ecbdata->opt, ws, reset, 1, sign, line, len);\n+\t\temit_line(ecbdata->opt, ws, reset, 1, sign, line, len);\n \telse {\n \t\t/* Emit just the prefix, then the rest. */\n-\t\temit_line_0(ecbdata->opt, set, reset, 1, sign, \"\", 0);\n+\t\temit_line(ecbdata->opt, set, reset, 1, sign, \"\", 0);\n \t\tws_check_emit(line, len, ecbdata->ws_rule,\n \t\t\t ecbdata->opt->file, set, reset, ws);\n \t}\n@@ -637,7 +631,7 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \tif (len < 10 ||\n \t memcmp(line, atat, 2) ||\n \t !(ep = memmem(line + 2, len - 2, atat, 2))) {\n-\t\temit_line(ecbdata->opt, context, reset, line, len);\n+\t\temit_line(ecbdata->opt, context, reset, 1, 0, line, len);\n \t\treturn;\n \t}\n \tep += 2; /* skip over @@ */\n@@ -673,7 +667,7 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \tstrbuf_add(&msgbuf, line + len, org_len - len);\n \tstrbuf_complete_line(&msgbuf);\n \n-\temit_line(ecbdata->opt, \"\", \"\", msgbuf.buf, msgbuf.len);\n+\temit_line(ecbdata->opt, \"\", \"\", 1, 0, msgbuf.buf, msgbuf.len);\n \tstrbuf_release(&msgbuf);\n }\n \n@@ -736,8 +730,8 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\tconst char *context = diff_get_color(ecb->color_diff,\n \t\t\t\t\t\t DIFF_CONTEXT);\n \t\tputc('\\n', ecb->opt->file);\n-\t\temit_line_0(ecb->opt, context, reset, 1, '\\\\',\n-\t\t\t nneof, strlen(nneof));\n+\t\temit_line(ecb->opt, context, reset, 1, '\\\\',\n+\t\t\t nneof, strlen(nneof));\n \t}\n }\n \n@@ -1335,7 +1329,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t}\n \t\tdiff_words_flush(ecbdata);\n \t\tif (ecbdata->diff_words->type == DIFF_WORDS_PORCELAIN) {\n-\t\t\temit_line(o, context, reset, line, len);\n+\t\t\temit_line(o, context, reset, 1, 0, line, len);\n \t\t\tfputs(\"~\\n\", o->file);\n \t\t} else {\n \t\t\t/*\n@@ -1347,7 +1341,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t\t line++;\n \t\t\t len--;\n \t\t\t}\n-\t\t\temit_line(o, context, reset, line, len);\n+\t\t\temit_line(o, context, reset, 1, 0, line, len);\n \t\t}\n \t\treturn;\n \t}\n@@ -1370,7 +1364,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t/* incomplete line at the end */\n \t\tecbdata->lno_in_preimage++;\n \t\temit_line(o, diff_get_color(ecbdata->color_diff, DIFF_CONTEXT),\n-\t\t\t reset, line, len);\n+\t\t\t reset, 1, 0, line, len);\n \t\tbreak;\n \t}\n }\n@@ -2182,7 +2176,7 @@ static void checkdiff_consume(void *priv, char *line, unsigned long len)\n \t\tfprintf(data->o->file, \"%s%s:%d: %s.\\n\",\n \t\t\tline_prefix, data->filename, data->lineno, err);\n \t\tfree(err);\n-\t\temit_line(data->o, set, reset, line, 1);\n+\t\temit_line(data->o, set, reset, 1, 0, line, 1);\n \t\tws_check_emit(line + 1, len - 1, data->ws_rule,\n \t\t\t data->o->file, set, reset, ws);\n \t} else if (line[0] == ' ') {\n@@ -4831,7 +4825,7 @@ void diff_flush(struct diff_options *options)\n \t\t\tterm[0] = options->line_termination;\n \t\t\tterm[1] = '\\0';\n \n-\t\t\temit_line(options, NULL, NULL, term, !!term[0]);\n+\t\t\temit_line(options, NULL, NULL, 1, 0, term, !!term[0]);\n \t\t\tif (options->stat_sep) {\n \t\t\t\t/* attach patch instead of inline */\n \t\t\t\tfputs(options->stat_sep, options->file);\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"319998","messageId":"20170517025857.32320-12-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 11/20] diff.c: convert emit_rewrite_lines to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:48Z","receivedAt":"2017-05-17T02:59:28Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers emit_rewrite_lines.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 27 +++++++++++++++++++--------\n 1 file changed, 19 insertions(+), 8 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 3dda9f3c8e..690794aeb8 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -722,15 +722,25 @@ static void add_line_count(struct strbuf *out, int count)\n static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\t\t int prefix, const char *data, int size)\n {\n-\tconst char *endp = NULL;\n-\tstatic const char *nneof = \" No newline at end of file\\n\";\n \tconst char *reset = diff_get_color(ecb->color_diff, DIFF_RESET);\n+\tstruct strbuf sb = STRBUF_INIT;\n \n \twhile (0 < size) {\n \t\tint len;\n \n-\t\tendp = memchr(data, '\\n', size);\n-\t\tlen = endp ? (endp - data + 1) : size;\n+\t\tconst char *endp = memchr(data, '\\n', size);\n+\t\tif (endp)\n+\t\t\tlen = endp - data + 1;\n+\t\telse {\n+\t\t\twhile (0 < size) {\n+\t\t\t\tstrbuf_addch(&sb, *data);\n+\t\t\t\tsize -= len;\n+\t\t\t\tdata += len;\n+\t\t\t}\n+\t\t\tstrbuf_addch(&sb, '\\n');\n+\t\t\tdata = sb.buf;\n+\t\t\tlen = sb.len;\n+\t\t}\n \t\tif (prefix != '+') {\n \t\t\tecb->lno_in_preimage++;\n \t\t\temit_del_line(reset, ecb, data, len);\n@@ -741,12 +751,13 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\tsize -= len;\n \t\tdata += len;\n \t}\n-\tif (!endp) {\n+\tif (sb.len) {\n+\t\tstatic const char *nneof = \"\\\\ No newline at end of file\\n\";\n \t\tconst char *context = diff_get_color(ecb->color_diff,\n \t\t\t\t\t\t DIFF_CONTEXT);\n-\t\tputc('\\n', ecb->opt->file);\n-\t\temit_line(ecb->opt, context, reset, 1, '\\\\',\n-\t\t\t nneof, strlen(nneof));\n+\t\temit_line(ecb->opt, context, reset, 1, 0,\n+\t\t\t nneof, strlen(nneof));\n+\t\tstrbuf_release(&sb);\n \t}\n }\n \n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"319999","messageId":"20170517025857.32320-15-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 14/20] diff.c: convert show_stats to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:51Z","receivedAt":"2017-05-17T02:59:31Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nWe call print_stat_summary from builtin/apply, so we still\nneed the version with a file pointer, so introduce\nprint_stat_summary_0 that uses emit_line_* machinery and\nkeep print_stat_summary with the same arguments around.\n\nThe responsibility to print the line prefix moves from the callers\nof print_stat_summary_0 into the function itself.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 89 ++++++++++++++++++++++++++++++++++++++----------------------------\n diff.h | 4 +--\n 2 files changed, 53 insertions(+), 40 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 2dd10fa16a..ccd28953d7 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1536,20 +1536,19 @@ static int scale_linear(int it, int width, int max_change)\n \treturn 1 + (it * (width - 1) / max_change);\n }\n \n-static void show_name(FILE *file,\n+static void show_name(struct strbuf *out,\n \t\t const char *prefix, const char *name, int len)\n {\n-\tfprintf(file, \" %s%-*s |\", prefix, len, name);\n+\tstrbuf_addf(out, \" %s%-*s |\", prefix, len, name);\n }\n \n-static void show_graph(FILE *file, char ch, int cnt, const char *set, const char *reset)\n+static void show_graph(struct strbuf *out, char ch, int cnt, const char *set, const char *reset)\n {\n \tif (cnt <= 0)\n \t\treturn;\n-\tfprintf(file, \"%s\", set);\n-\twhile (cnt--)\n-\t\tputc(ch, file);\n-\tfprintf(file, \"%s\", reset);\n+\tstrbuf_addstr(out, set);\n+\tstrbuf_addchars(out, ch, cnt);\n+\tstrbuf_addstr(out, reset);\n }\n \n static void fill_print_name(struct diffstat_file *file)\n@@ -1573,14 +1572,16 @@ static void fill_print_name(struct diffstat_file *file)\n \tfile->print_name = pname;\n }\n \n-int print_stat_summary(FILE *fp, int files, int insertions, int deletions)\n+void print_stat_summary_0(struct diff_options *options, int files,\n+\t\t\t int insertions, int deletions)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n-\tint ret;\n \n \tif (!files) {\n \t\tassert(insertions == 0 && deletions == 0);\n-\t\treturn fprintf(fp, \"%s\\n\", \" 0 files changed\");\n+\t\tstrbuf_addstr(&sb, \" 0 files changed\");\n+\t\temit_line(options, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\t\treturn;\n \t}\n \n \tstrbuf_addf(&sb,\n@@ -1607,9 +1608,17 @@ int print_stat_summary(FILE *fp, int files, int insertions, int deletions)\n \t\t\t deletions);\n \t}\n \tstrbuf_addch(&sb, '\\n');\n-\tret = fputs(sb.buf, fp);\n+\temit_line(options, NULL, NULL, 1, 0, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n-\treturn ret;\n+}\n+\n+void print_stat_summary(FILE *fp, int files,\n+\t\t\tint insertions, int deletions)\n+{\n+\tstruct diff_options o;\n+\tmemset(&o, 0, sizeof(o));\n+\to.file = fp;\n+\tprint_stat_summary_0(&o, files, insertions, deletions);\n }\n \n static void show_stats(struct diffstat_t *data, struct diff_options *options)\n@@ -1619,13 +1628,13 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \tint total_files = data->nr, count;\n \tint width, name_width, graph_width, number_width = 0, bin_width = 0;\n \tconst char *reset, *add_c, *del_c;\n-\tconst char *line_prefix = \"\";\n \tint extra_shown = 0;\n+\tconst char *line_prefix = diff_line_prefix(options);\n+\tstruct strbuf out = STRBUF_INIT;\n \n \tif (data->nr == 0)\n \t\treturn;\n \n-\tline_prefix = diff_line_prefix(options);\n \tcount = options->stat_count ? options->stat_count : data->nr;\n \n \treset = diff_get_color_opt(options, DIFF_RESET);\n@@ -1779,26 +1788,29 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t}\n \n \t\tif (file->is_binary) {\n-\t\t\tfprintf(options->file, \"%s\", line_prefix);\n-\t\t\tshow_name(options->file, prefix, name, len);\n-\t\t\tfprintf(options->file, \" %*s\", number_width, \"Bin\");\n+\t\t\tshow_name(&out, prefix, name, len);\n+\t\t\tstrbuf_addf(&out, \" %*s\", number_width, \"Bin\");\n \t\t\tif (!added && !deleted) {\n-\t\t\t\tputc('\\n', options->file);\n+\t\t\t\tstrbuf_addch(&out, '\\n');\n+\t\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\t\tstrbuf_reset(&out);\n \t\t\t\tcontinue;\n \t\t\t}\n-\t\t\tfprintf(options->file, \" %s%\"PRIuMAX\"%s\",\n+\t\t\tstrbuf_addf(&out, \" %s%\"PRIuMAX\"%s\",\n \t\t\t\tdel_c, deleted, reset);\n-\t\t\tfprintf(options->file, \" -> \");\n-\t\t\tfprintf(options->file, \"%s%\"PRIuMAX\"%s\",\n+\t\t\tstrbuf_addstr(&out, \" -> \");\n+\t\t\tstrbuf_addf(&out, \"%s%\"PRIuMAX\"%s\",\n \t\t\t\tadd_c, added, reset);\n-\t\t\tfprintf(options->file, \" bytes\");\n-\t\t\tfprintf(options->file, \"\\n\");\n+\t\t\tstrbuf_addstr(&out, \" bytes\\n\");\n+\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n \t\telse if (file->is_unmerged) {\n-\t\t\tfprintf(options->file, \"%s\", line_prefix);\n-\t\t\tshow_name(options->file, prefix, name, len);\n-\t\t\tfprintf(options->file, \" Unmerged\\n\");\n+\t\t\tshow_name(&out, prefix, name, len);\n+\t\t\tstrbuf_addstr(&out, \" Unmerged\\n\");\n+\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n \n@@ -1821,14 +1833,15 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t\t\tadd = total - del;\n \t\t\t}\n \t\t}\n-\t\tfprintf(options->file, \"%s\", line_prefix);\n-\t\tshow_name(options->file, prefix, name, len);\n-\t\tfprintf(options->file, \" %*\"PRIuMAX\"%s\",\n+\t\tshow_name(&out, prefix, name, len);\n+\t\tstrbuf_addf(&out, \" %*\"PRIuMAX\"%s\",\n \t\t\tnumber_width, added + deleted,\n \t\t\tadded + deleted ? \" \" : \"\");\n-\t\tshow_graph(options->file, '+', add, add_c, reset);\n-\t\tshow_graph(options->file, '-', del, del_c, reset);\n-\t\tfprintf(options->file, \"\\n\");\n+\t\tshow_graph(&out, '+', add, add_c, reset);\n+\t\tshow_graph(&out, '-', del, del_c, reset);\n+\t\tstrbuf_addch(&out, '\\n');\n+\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\tstrbuf_reset(&out);\n \t}\n \n \tfor (i = 0; i < data->nr; i++) {\n@@ -1849,11 +1862,12 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\tif (i < count)\n \t\t\tcontinue;\n \t\tif (!extra_shown)\n-\t\t\tfprintf(options->file, \"%s ...\\n\", line_prefix);\n+\t\t\temit_line(options, NULL, NULL, 1, 0,\n+\t\t\t\t \" ...\\n\", strlen(\" ...\\n\"));\n \t\textra_shown = 1;\n \t}\n-\tfprintf(options->file, \"%s\", line_prefix);\n-\tprint_stat_summary(options->file, total_files, adds, dels);\n+\n+\tprint_stat_summary_0(options, total_files, adds, dels);\n }\n \n static void show_shortstats(struct diffstat_t *data, struct diff_options *options)\n@@ -1865,7 +1879,7 @@ static void show_shortstats(struct diffstat_t *data, struct diff_options *option\n \n \tfor (i = 0; i < data->nr; i++) {\n \t\tint added = data->files[i]->added;\n-\t\tint deleted= data->files[i]->deleted;\n+\t\tint deleted = data->files[i]->deleted;\n \n \t\tif (data->files[i]->is_unmerged ||\n \t\t (!data->files[i]->is_interesting && (added + deleted == 0))) {\n@@ -1875,8 +1889,7 @@ static void show_shortstats(struct diffstat_t *data, struct diff_options *option\n \t\t\tdels += deleted;\n \t\t}\n \t}\n-\tfprintf(options->file, \"%s\", diff_line_prefix(options));\n-\tprint_stat_summary(options->file, total_files, adds, dels);\n+\tprint_stat_summary_0(options, total_files, adds, dels);\n }\n \n static void show_numstat(struct diffstat_t *data, struct diff_options *options)\ndiff --git a/diff.h b/diff.h\nindex 6e14100102..b75b0d7283 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -394,8 +394,8 @@ extern int parse_rename_score(const char **cp_p);\n \n extern long parse_algorithm_value(const char *value);\n \n-extern int print_stat_summary(FILE *fp, int files,\n-\t\t\t int insertions, int deletions);\n+extern void print_stat_summary(FILE *fp, int files,\n+\t\t\t int insertions, int deletions);\n extern void setup_diff_pager(struct diff_options *);\n \n #endif /* DIFF_H */\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320000","messageId":"20170517025857.32320-7-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 06/20] diff.c: emit_line_0 takes parameter whether to output line prefix","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:43Z","receivedAt":"2017-05-17T02:59:34Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In later patches we'll make extensive use of emit_line_0, as we'd want\nto funnel all output through this function such that we can add buffering\nthere.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 15 ++++++++-------\n 1 file changed, 8 insertions(+), 7 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 6c1886d495..25735f03d2 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -517,12 +517,13 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n }\n \n static void emit_line_0(struct diff_options *o, const char *set, const char *reset,\n-\t\t\tint sign, const char *line, int len)\n+\t\t\tint add_line_prefix, int sign, const char *line, int len)\n {\n \tint has_trailing_newline, has_trailing_carriage_return;\n \tFILE *file = o->file;\n \n-\tfputs(diff_line_prefix(o), file);\n+\tif (add_line_prefix)\n+\t\tfputs(diff_line_prefix(o), file);\n \n \thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n \tif (has_trailing_newline)\n@@ -549,7 +550,7 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n static void emit_line(struct diff_options *o, const char *set, const char *reset,\n \t\t const char *line, int len)\n {\n-\temit_line_0(o, set, reset, 0, line, len);\n+\temit_line_0(o, set, reset, 1, 0, line, len);\n }\n \n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n@@ -580,13 +581,13 @@ static void emit_line_checked(const char *reset,\n \t}\n \n \tif (!ws)\n-\t\temit_line_0(ecbdata->opt, set, reset, sign, line, len);\n+\t\temit_line_0(ecbdata->opt, set, reset, 1, sign, line, len);\n \telse if (sign == '+' && new_blank_line_at_eof(ecbdata, line, len))\n \t\t/* Blank line at EOF - paint '+' as well */\n-\t\temit_line_0(ecbdata->opt, ws, reset, sign, line, len);\n+\t\temit_line_0(ecbdata->opt, ws, reset, 1, sign, line, len);\n \telse {\n \t\t/* Emit just the prefix, then the rest. */\n-\t\temit_line_0(ecbdata->opt, set, reset, sign, \"\", 0);\n+\t\temit_line_0(ecbdata->opt, set, reset, 1, sign, \"\", 0);\n \t\tws_check_emit(line, len, ecbdata->ws_rule,\n \t\t\t ecbdata->opt->file, set, reset, ws);\n \t}\n@@ -735,7 +736,7 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\tconst char *context = diff_get_color(ecb->color_diff,\n \t\t\t\t\t\t DIFF_CONTEXT);\n \t\tputc('\\n', ecb->opt->file);\n-\t\temit_line_0(ecb->opt, context, reset, '\\\\',\n+\t\temit_line_0(ecb->opt, context, reset, 1, '\\\\',\n \t\t\t nneof, strlen(nneof));\n \t}\n }\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320001","messageId":"20170517025857.32320-9-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 08/20] diff.c: convert fn_out_consume to use emit_line","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:45Z","receivedAt":"2017-05-17T02:59:35Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line.\n\nThis covers the parts of fn_out_consume. In the next\npatches we'll convert more functions that want to emit\nformatted output, so we'd want to have a formatted emit\nfunction. Add it here.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 28 ++++++++++++++++++++--------\n 1 file changed, 20 insertions(+), 8 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 3569857818..8186289734 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -547,6 +547,21 @@ static void emit_line(struct diff_options *o, const char *set, const char *reset\n \t\tfputc('\\n', file);\n }\n \n+static void emit_line_fmt(struct diff_options *o,\n+\t\t\t const char *set, const char *reset,\n+\t\t\t int add_line_prefix,\n+\t\t\t const char *fmt, ...)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tva_list ap;\n+\tva_start(ap, fmt);\n+\tstrbuf_vaddf(&sb, fmt, ap);\n+\tva_end(ap);\n+\n+\temit_line(o, set, reset, add_line_prefix, 0, sb.buf, sb.len);\n+\tstrbuf_release(&sb);\n+}\n+\n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n {\n \tif (!((ecbdata->ws_rule & WS_BLANK_AT_EOF) &&\n@@ -1270,7 +1285,6 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \tconst char *context = diff_get_color(ecbdata->color_diff, DIFF_CONTEXT);\n \tconst char *reset = diff_get_color(ecbdata->color_diff, DIFF_RESET);\n \tstruct diff_options *o = ecbdata->opt;\n-\tconst char *line_prefix = diff_line_prefix(o);\n \n \to->found_changes = 1;\n \n@@ -1282,14 +1296,12 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \n \tif (ecbdata->label_path[0]) {\n \t\tconst char *name_a_tab, *name_b_tab;\n-\n \t\tname_a_tab = strchr(ecbdata->label_path[0], ' ') ? \"\\t\" : \"\";\n \t\tname_b_tab = strchr(ecbdata->label_path[1], ' ') ? \"\\t\" : \"\";\n-\n-\t\tfprintf(o->file, \"%s%s--- %s%s%s\\n\",\n-\t\t\tline_prefix, meta, ecbdata->label_path[0], reset, name_a_tab);\n-\t\tfprintf(o->file, \"%s%s+++ %s%s%s\\n\",\n-\t\t\tline_prefix, meta, ecbdata->label_path[1], reset, name_b_tab);\n+\t\temit_line_fmt(o, meta, reset, 1, \"--- %s%s\\n\",\n+\t\t\t ecbdata->label_path[0], name_a_tab);\n+\t\temit_line_fmt(o, meta, reset, 1, \"+++ %s%s\\n\",\n+\t\t\t ecbdata->label_path[1], name_b_tab);\n \t\tecbdata->label_path[0] = ecbdata->label_path[1] = NULL;\n \t}\n \n@@ -1330,7 +1342,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\tdiff_words_flush(ecbdata);\n \t\tif (ecbdata->diff_words->type == DIFF_WORDS_PORCELAIN) {\n \t\t\temit_line(o, context, reset, 1, 0, line, len);\n-\t\t\tfputs(\"~\\n\", o->file);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, \"~\\n\", 2);\n \t\t} else {\n \t\t\t/*\n \t\t\t * Skip the prefix character, if any. With\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320002","messageId":"20170517025857.32320-18-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 17/20] diff.c: convert diff_summary to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:54Z","receivedAt":"2017-05-17T02:59:40Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers diff_summary.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 64 ++++++++++++++++++++++++++++++++++------------------------------\n 1 file changed, 34 insertions(+), 30 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 4cd1b3c520..964b5cb5a7 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4500,67 +4500,71 @@ static void flush_one_pair(struct diff_filepair *p, struct diff_options *opt)\n \t}\n }\n \n-static void show_file_mode_name(FILE *file, const char *newdelete, struct diff_filespec *fs)\n+static void show_file_mode_name(struct diff_options *opt, const char *newdelete, struct diff_filespec *fs)\n {\n+\tstruct strbuf sb = STRBUF_INIT;\n \tif (fs->mode)\n-\t\tfprintf(file, \" %s mode %06o \", newdelete, fs->mode);\n+\t\tstrbuf_addf(&sb, \" %s mode %06o \", newdelete, fs->mode);\n \telse\n-\t\tfprintf(file, \" %s \", newdelete);\n-\twrite_name_quoted(fs->path, file, '\\n');\n-}\n+\t\tstrbuf_addf(&sb, \" %s \", newdelete);\n \n+\tquote_c_style(fs->path, &sb, NULL, 0);\n+\tstrbuf_addch(&sb, '\\n');\n+\temit_line(opt, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\tstrbuf_release(&sb);\n+}\n \n-static void show_mode_change(FILE *file, struct diff_filepair *p, int show_name,\n-\t\tconst char *line_prefix)\n+static void show_mode_change(struct diff_options *opt, struct diff_filepair *p,\n+\t\tint show_name)\n {\n \tif (p->one->mode && p->two->mode && p->one->mode != p->two->mode) {\n-\t\tfprintf(file, \"%s mode change %06o => %06o%c\", line_prefix, p->one->mode,\n-\t\t\tp->two->mode, show_name ? ' ' : '\\n');\n+\t\tstruct strbuf sb = STRBUF_INIT;\n \t\tif (show_name) {\n-\t\t\twrite_name_quoted(p->two->path, file, '\\n');\n+\t\t\tstrbuf_addch(&sb, ' ');\n+\t\t\tquote_c_style(p->two->path, &sb, NULL, 0);\n \t\t}\n+\t\temit_line_fmt(opt, NULL, NULL, 1,\n+\t\t\t \" mode change %06o => %06o%s\\n\",\n+\t\t\t p->one->mode, p->two->mode,\n+\t\t\t show_name ? sb.buf : \"\");\n+\t\tstrbuf_release(&sb);\n \t}\n }\n \n-static void show_rename_copy(FILE *file, const char *renamecopy, struct diff_filepair *p,\n-\t\t\tconst char *line_prefix)\n+static void show_rename_copy(struct diff_options *opt, const char *renamecopy,\n+\t\tstruct diff_filepair *p)\n {\n \tchar *names = pprint_rename(p->one->path, p->two->path);\n-\n-\tfprintf(file, \" %s %s (%d%%)\\n\", renamecopy, names, similarity_index(p));\n+\temit_line_fmt(opt, NULL, NULL, 1, \" %s %s (%d%%)\\n\",\n+\t\t renamecopy, names, similarity_index(p));\n \tfree(names);\n-\tshow_mode_change(file, p, 0, line_prefix);\n+\tshow_mode_change(opt, p, 0);\n }\n \n static void diff_summary(struct diff_options *opt, struct diff_filepair *p)\n {\n-\tFILE *file = opt->file;\n-\tconst char *line_prefix = diff_line_prefix(opt);\n-\n \tswitch(p->status) {\n \tcase DIFF_STATUS_DELETED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_file_mode_name(file, \"delete\", p->one);\n+\t\tshow_file_mode_name(opt, \"delete\", p->one);\n \t\tbreak;\n \tcase DIFF_STATUS_ADDED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_file_mode_name(file, \"create\", p->two);\n+\t\tshow_file_mode_name(opt, \"create\", p->two);\n \t\tbreak;\n \tcase DIFF_STATUS_COPIED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_rename_copy(file, \"copy\", p, line_prefix);\n+\t\tshow_rename_copy(opt, \"copy\", p);\n \t\tbreak;\n \tcase DIFF_STATUS_RENAMED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_rename_copy(file, \"rename\", p, line_prefix);\n+\t\tshow_rename_copy(opt, \"rename\", p);\n \t\tbreak;\n \tdefault:\n \t\tif (p->score) {\n-\t\t\tfprintf(file, \"%s rewrite \", line_prefix);\n-\t\t\twrite_name_quoted(p->two->path, file, ' ');\n-\t\t\tfprintf(file, \"(%d%%)\\n\", similarity_index(p));\n+\t\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\t\tstrbuf_addstr(&sb, \" rewrite \");\n+\t\t\tquote_c_style(p->two->path, &sb, NULL, 0);\n+\t\t\tstrbuf_addf(&sb, \" (%d%%)\\n\", similarity_index(p));\n+\t\t\temit_line(opt, NULL, NULL, 1, 0, sb.buf, sb.len);\n \t\t}\n-\t\tshow_mode_change(file, p, !p->score, line_prefix);\n+\t\tshow_mode_change(opt, p, !p->score);\n \t\tbreak;\n \t}\n }\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320003","messageId":"20170517025857.32320-17-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 16/20] diff.c: convert diff_flush to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:53Z","receivedAt":"2017-05-17T02:59:44Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers diff_flush.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/diff.c b/diff.c\nindex f1cb0b7799..4cd1b3c520 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4868,7 +4868,9 @@ void diff_flush(struct diff_options *options)\n \t\t\temit_line(options, NULL, NULL, 1, 0, term, !!term[0]);\n \t\t\tif (options->stat_sep) {\n \t\t\t\t/* attach patch instead of inline */\n-\t\t\t\tfputs(options->stat_sep, options->file);\n+\t\t\t\temit_line(options, NULL, NULL, 0, 0,\n+\t\t\t\t\t options->stat_sep,\n+\t\t\t\t\t strlen(options->stat_sep));\n \t\t\t}\n \t\t}\n \n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320004","messageId":"20170517025857.32320-21-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 20/20] diff.c: color moved lines differently","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:57Z","receivedAt":"2017-05-17T02:59:47Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"When there is a lot of code moved around such as in 11979b9 (2005-11-18,\n\"http.c: reorder to avoid compilation failure.\") for example, the review\nprocess is quite hard, as it is not mentally challenging. It is a rather\ntedious process, that gets boring quickly. However you still need to read\nthrough all of the code to make sure the moved lines are there as supposed.\n\nWhile it is trivial to color up a patch like the following\n\n $ git diff\n diff --git a/file2.c b/file2.c\n index 9163a0f..8e66dc0 100644\n --- a/file2.c\n +++ b/file2.c\n @@ -3,13 +3,6 @@ void *xmemdupz(const void *data, size_t len)\n return memcpy(xmallocz(len), data, len);\n }\n\n -int secure_foo(struct user *u)\n -{\n - if (!u->is_allowed_foo)\n - return;\n - foo(u);\n -}\n -\n char *xstrndup(const char *str, size_t len)\n {\n char *p = memchr(str, '\\0', len);\n diff --git a/test.c b/test.c\n index a95e6fe..81eb0eb 100644\n --- a/test.c\n +++ b/test.c\n @@ -18,6 +18,13 @@ ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset)\n return total;\n }\n\n +int secure_foo(struct user *u)\n +{\n + if (!u->is_allowed_foo)\n + return;\n + foo(u);\n +}\n +\n int xdup(int fd)\n {\n int ret = dup(fd);\n\nas in this patch all lines that add or remove lines\nshould be colored in the new color that indicates moved\nlines.\n\nHowever the intention of this patch is to aid reviewers\nto spotting permutations in the moved code. So consider the\nfollowing malicious move:\n\n diff --git a/file2.c b/file2.c\n index 9163a0f..8e66dc0 100644\n --- a/file2.c\n +++ b/file2.c\n @@ -3,13 +3,6 @@ void *xmemdupz(const void *data, size_t len)\n return memcpy(xmallocz(len), data, len);\n }\n\n -int secure_foo(struct user *u)\n -{\n - if (!u->is_allowed_foo)\n - return;\n - foo(u);\n -}\n -\n char *xstrndup(const char *str, size_t len)\n {\n char *p = memchr(str, '\\0', len);\n diff --git a/test.c b/test.c\n index a95e6fe..a679c40 100644\n --- a/test.c\n +++ b/test.c\n @@ -18,6 +18,13 @@ ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset)\n return total;\n }\n\n +int secure_foo(struct user *u)\n +{\n + foo(u);\n + if (!u->is_allowed_foo)\n + return;\n +}\n +\n int xdup(int fd)\n {\n int ret = dup(fd);\n\nIf the moved code is larger, it is easier to hide some permutation in the\ncode, which is why we would not want to color all lines as \"moved\" in this\ncase. So we do not just need to color lines differently that are added and\nremoved in the same diff, we need to tweak the algorithm a bit more.\n\nAs the reviewers attention should be brought to the places, where the\ndifference is introduced to the moved code, we cannot just have one new\ncolor for all of moved code.\n\nFirst I implemented an alternative design, which would show a moved hunk\nin one color, and its boundaries in another color. This idea was error\nprone as it inspected each line and its neighboring lines to determine\nif the line was (a) moved and (b) if was deep inside a hunk by having\nmatching neighboring lines. This is unreliable as the we can construct\nhunks which have equal neighbors that just exceed the number of lines\ninspected. (Think of 'AXYZBXYZCXYZD..' with each letter as a line, that\nis permutated to AXYZCXYZBXYZD..').\n\nInstead this provides a dynamic programming greedy algorithm that finds\nthe largest moved hunk and then switches color to the alternative color\nfor the next hunk. By doing this any permutation is recognized and\ndisplayed. That implies that there is no dedicated boundary or\ninside-hunk color, but instead we'll have just two colors alternating\nfor hunks.\n\nIt would be a bit more UX friendly if the two corresponding hunks\n(of added and deleted lines) for one move would get the same color id.\n(Both get \"regular moved\" or \"alternative moved\"). This problem is\ndeferred to a later patch for now.\n\nAlgorithm-by: Jonathan Tan \nSigned-off-by: Stefan Beller \n---\n Documentation/config.txt | 14 ++-\n diff.c | 266 +++++++++++++++++++++++++++++++++++++++++++--\n diff.h | 11 +-\n t/t4015-diff-whitespace.sh | 229 ++++++++++++++++++++++++++++++++++++++\n 4 files changed, 507 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 475e874d51..902d017c3b 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1051,14 +1051,24 @@ This does not affect linkgit:git-format-patch[1] or the\n 'git-diff-{asterisk}' plumbing commands. Can be overridden on the\n command line with the `--color[=]` option.\n \n+color.moved::\n+\tA boolean value, whether a diff should color moved lines\n+\tdifferently. The moved lines are searched for in the diff only.\n+\tDuplicated lines from somewhere in the project that are not\n+\tpart of the diff are not colored as moved.\n+\tDefaults to false.\n+\n color.diff.::\n \tUse customized color for diff colorization. `` specifies\n \twhich part of the patch to use the specified color, and is one\n \tof `context` (context text - `plain` is a historical synonym),\n \t`meta` (metainformation), `frag`\n \t(hunk header), 'func' (function in hunk header), `old` (removed lines),\n-\t`new` (added lines), `commit` (commit headers), or `whitespace`\n-\t(highlighting whitespace errors).\n+\t`new` (added lines), `commit` (commit headers), `whitespace`\n+\t(highlighting whitespace errors), `oldMoved` (removed lines that\n+\treappear), `newMoved` (added lines that were removed elsewhere),\n+\t`oldMovedAlternative` and `newMovedAlternative` (as a fallback to\n+\tcover adjacent blocks of moved code)\n \n color.decorate.::\n \tUse customized color for 'git log --decorate' output. `` is one\ndiff --git a/diff.c b/diff.c\nindex 1d8d1786fd..15cf322b50 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -31,6 +31,7 @@ static int diff_indent_heuristic; /* experimental */\n static int diff_rename_limit_default = 400;\n static int diff_suppress_blank_empty;\n static int diff_use_color_default = -1;\n+static int diff_color_moved_default;\n static int diff_context_default = 3;\n static int diff_interhunk_context_default;\n static const char *diff_word_regex_cfg;\n@@ -55,6 +56,10 @@ static char diff_colors[][COLOR_MAXLEN] = {\n \tGIT_COLOR_YELLOW,\t/* COMMIT */\n \tGIT_COLOR_BG_RED,\t/* WHITESPACE */\n \tGIT_COLOR_NORMAL,\t/* FUNCINFO */\n+\tGIT_COLOR_BOLD_RED,\t/* OLD_MOVED_A */\n+\tGIT_COLOR_BG_RED,\t/* OLD_MOVED_B */\n+\tGIT_COLOR_BOLD_GREEN,\t/* NEW_MOVED_A */\n+\tGIT_COLOR_BG_GREEN,\t/* NEW_MOVED_B */\n };\n \n static NORETURN void die_want_option(const char *option_name)\n@@ -80,6 +85,14 @@ static int parse_diff_color_slot(const char *var)\n \t\treturn DIFF_WHITESPACE;\n \tif (!strcasecmp(var, \"func\"))\n \t\treturn DIFF_FUNCINFO;\n+\tif (!strcasecmp(var, \"oldmoved\"))\n+\t\treturn DIFF_FILE_OLD_MOVED;\n+\tif (!strcasecmp(var, \"oldmovedalternative\"))\n+\t\treturn DIFF_FILE_OLD_MOVED_ALT;\n+\tif (!strcasecmp(var, \"newmoved\"))\n+\t\treturn DIFF_FILE_NEW_MOVED;\n+\tif (!strcasecmp(var, \"newmovedalternative\"))\n+\t\treturn DIFF_FILE_NEW_MOVED_ALT;\n \treturn -1;\n }\n \n@@ -234,6 +247,10 @@ int git_diff_ui_config(const char *var, const char *value, void *cb)\n \t\tdiff_use_color_default = git_config_colorbool(var, value);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(var, \"color.moved\")) {\n+\t\tdiff_color_moved_default = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(var, \"diff.context\")) {\n \t\tdiff_context_default = git_config_int(var, value);\n \t\tif (diff_context_default < 0)\n@@ -354,6 +371,88 @@ int git_diff_basic_config(const char *var, const char *value, void *cb)\n \treturn git_default_config(var, value, cb);\n }\n \n+struct moved_entry {\n+\tstruct hashmap_entry ent;\n+\tconst struct buffered_patch_line *line;\n+\tstruct moved_entry *next_line;\n+};\n+\n+static void get_ws_cleaned_string(const struct buffered_patch_line *l,\n+\t\t\t\t struct strbuf *out)\n+{\n+\tint i;\n+\tfor (i = 0; i < l->len; i++) {\n+\t\tif (isspace(l->line[i]))\n+\t\t\tcontinue;\n+\t\tstrbuf_addch(out, l->line[i]);\n+\t}\n+}\n+\n+static int buffered_patch_line_cmp_no_ws(const struct buffered_patch_line *a,\n+\t\t\t\t\t const struct buffered_patch_line *b,\n+\t\t\t\t\t const void *keydata)\n+{\n+\tint ret;\n+\tstruct strbuf sba = STRBUF_INIT;\n+\tstruct strbuf sbb = STRBUF_INIT;\n+\n+\tget_ws_cleaned_string(a, &sba);\n+\tget_ws_cleaned_string(b, &sbb);\n+\tret = sba.len != sbb.len || strncmp(sba.buf, sbb.buf, sba.len);\n+\n+\tstrbuf_release(&sba);\n+\tstrbuf_release(&sbb);\n+\treturn ret;\n+}\n+\n+static int buffered_patch_line_cmp(const struct buffered_patch_line *a,\n+\t\t\t\t const struct buffered_patch_line *b,\n+\t\t\t\t const void *keydata)\n+{\n+\treturn a->len != b->len || strncmp(a->line, b->line, a->len);\n+}\n+\n+static int moved_entry_cmp(const struct moved_entry *a,\n+\t\t\t const struct moved_entry *b,\n+\t\t\t const void *keydata)\n+{\n+\treturn buffered_patch_line_cmp(a->line, b->line, keydata);\n+}\n+\n+static int moved_entry_cmp_no_ws(const struct moved_entry *a,\n+\t\t\t\t const struct moved_entry *b,\n+\t\t\t\t const void *keydata)\n+{\n+\treturn buffered_patch_line_cmp_no_ws(a->line, b->line, keydata);\n+}\n+\n+static unsigned get_line_hash(struct buffered_patch_line *line, unsigned ignore_ws)\n+{\n+\tstatic struct strbuf sb = STRBUF_INIT;\n+\n+\tif (ignore_ws) {\n+\t\tstrbuf_reset(&sb);\n+\t\tget_ws_cleaned_string(line, &sb);\n+\t\treturn memhash(sb.buf, sb.len);\n+\t} else {\n+\t\treturn memhash(line->line, line->len);\n+\t}\n+}\n+\n+static struct moved_entry *prepare_entry(struct diff_options *o,\n+\t\t\t\t\t int line_no)\n+{\n+\tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n+\tunsigned ignore_ws = DIFF_XDL_TST(o, IGNORE_WHITESPACE);\n+\tstruct buffered_patch_line *l = &o->line_buffer[line_no];\n+\n+\tret->ent.hash = get_line_hash(l, ignore_ws);\n+\tret->line = l;\n+\tret->next_line = NULL;\n+\n+\treturn ret;\n+}\n+\n static char *quote_two(const char *one, const char *two)\n {\n \tint need_one = quote_c_style(one, NULL, NULL, 1);\n@@ -516,6 +615,135 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n+static void add_lines_to_move_detection(struct diff_options *o)\n+{\n+\tstruct moved_entry *prev_line;\n+\n+\tint n;\n+\tfor (n = 0; n < o->line_buffer_nr; n++) {\n+\t\tint sign = 0;\n+\t\tstruct hashmap *hm;\n+\t\tstruct moved_entry *key;\n+\n+\t\tswitch (o->line_buffer[n].sign) {\n+\t\tcase '+':\n+\t\t\tsign = '+';\n+\t\t\thm = o->added_lines;\n+\t\t\tbreak;\n+\t\tcase '-':\n+\t\t\tsign = '-';\n+\t\t\thm = o->deleted_lines;\n+\t\t\tbreak;\n+\t\tcase ' ':\n+\t\tdefault:\n+\t\t\tprev_line = NULL;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tkey = prepare_entry(o, n);\n+\t\tif (prev_line &&\n+\t\t prev_line->line->sign == sign)\n+\t\t\tprev_line->next_line = key;\n+\n+\t\thashmap_add(hm, key);\n+\t\tprev_line = key;\n+\t}\n+}\n+\n+static void mark_color_as_moved(struct diff_options *o)\n+{\n+\tstruct moved_entry **pmb = NULL; /* potentially moved blocks */\n+\tint pmb_nr = 0, pmb_alloc = 0;\n+\tint alt_flag = 0;\n+\tint n;\n+\n+\tfor (n = 0; n < o->line_buffer_nr; n++) {\n+\t\tstruct hashmap *hm = NULL;\n+\t\tstruct moved_entry *key;\n+\t\tstruct moved_entry *match = NULL;\n+\t\tstruct buffered_patch_line *l = &o->line_buffer[n];\n+\t\tint i, lp, rp;\n+\n+\t\tswitch (l->sign) {\n+\t\tcase '+':\n+\t\t\thm = o->deleted_lines;\n+\t\t\tbreak;\n+\t\tcase '-':\n+\t\t\thm = o->added_lines;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\talt_flag = 0; /* reset to standard, no-alt move color */\n+\t\t\tpmb_nr = 0; /* no running sets */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* Check for any match to color it as a move. */\n+\t\tkey = prepare_entry(o, n);\n+\t\tmatch = hashmap_get(hm, key, o);\n+\t\tfree(key);\n+\t\tif (!match)\n+\t\t\tcontinue;\n+\n+\t\t/* Check any potential block runs, advance each or nullify */\n+\t\tfor (i = 0; i < pmb_nr; i++) {\n+\t\t\tstruct moved_entry *p = pmb[i];\n+\t\t\tstruct moved_entry *pnext = (p && p->next_line) ?\n+\t\t\t\t\tp->next_line : NULL;\n+\t\t\tif (pnext &&\n+\t\t\t !buffered_patch_line_cmp(pnext->line, l, o)) {\n+\t\t\t\tpmb[i] = p->next_line;\n+\t\t\t} else {\n+\t\t\t\tpmb[i] = NULL;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/* Shrink the set to the remaining runs */\n+\t\tfor (lp = 0, rp = pmb_nr - 1; lp <= rp;) {\n+\t\t\twhile (lp < pmb_nr && pmb[lp])\n+\t\t\t\tlp ++;\n+\t\t\t/* lp points at the first NULL now */\n+\n+\t\t\twhile (rp > -1 && !pmb[rp])\n+\t\t\t\trp--;\n+\t\t\t/* rp points at the last non-NULL */\n+\n+\t\t\tif (lp < pmb_nr && rp > -1 && lp < rp) {\n+\t\t\t\tpmb[lp] = pmb[rp];\n+\t\t\t\tpmb[rp] = NULL;\n+\t\t\t\trp--;\n+\t\t\t\tlp++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (rp > -1) {\n+\t\t\t/* Remember the number of running sets */\n+\t\t\tpmb_nr = rp + 1;\n+\t\t} else {\n+\t\t\t/* Toggle color */\n+\t\t\talt_flag = (alt_flag + 1) % 2;\n+\n+\t\t\t/* Build up a new set */\n+\t\t\tpmb_nr = 0;\n+\t\t\tfor (; match; match = hashmap_get_next(hm, match)) {\n+\t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n+\t\t\t\tpmb[pmb_nr++] = match;\n+\t\t\t}\n+\t\t}\n+\n+\t\tswitch (l->sign) {\n+\t\tcase '+':\n+\t\t\tl->set = diff_get_color_opt(o, DIFF_FILE_NEW_MOVED + alt_flag);\n+\t\t\tbreak;\n+\t\tcase '-':\n+\t\t\tl->set = diff_get_color_opt(o, DIFF_FILE_OLD_MOVED + alt_flag);\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tdie(\"BUG: we should have continued earlier?\");\n+\t\t}\n+\t}\n+\tfree(pmb);\n+}\n+\n static void emit_buffered_patch_line(struct diff_options *o,\n \t\t\t\t struct buffered_patch_line *e)\n {\n@@ -3514,6 +3742,8 @@ void diff_setup(struct diff_options *options)\n \toptions->line_buffer = NULL;\n \toptions->line_buffer_nr = 0;\n \toptions->line_buffer_alloc = 0;\n+\n+\toptions->color_moved = diff_color_moved_default;\n }\n \n void diff_setup_done(struct diff_options *options)\n@@ -3623,6 +3853,9 @@ void diff_setup_done(struct diff_options *options)\n \n \tif (DIFF_OPT_TST(options, FOLLOW_RENAMES) && options->pathspec.nr != 1)\n \t\tdie(_(\"--follow requires exactly one pathspec\"));\n+\n+\tif (!options->use_color || external_diff())\n+\t\toptions->color_moved = 0;\n }\n \n static int opt_arg(const char *arg, int arg_short, const char *arg_long, int *val)\n@@ -4047,6 +4280,10 @@ int diff_opt_parse(struct diff_options *options,\n \t}\n \telse if (!strcmp(arg, \"--no-color\"))\n \t\toptions->use_color = 0;\n+\telse if (!strcmp(arg, \"--color-moved\"))\n+\t\toptions->color_moved = 1;\n+\telse if (!strcmp(arg, \"--no-color-moved\"))\n+\t\toptions->color_moved = 0;\n \telse if (!strcmp(arg, \"--color-words\")) {\n \t\toptions->use_color = 1;\n \t\toptions->word_diff = DIFF_WORDS_COLOR;\n@@ -4852,16 +5089,19 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n {\n \tint i;\n \tstruct diff_queue_struct *q = &diff_queued_diff;\n-\t/*\n-\t * For testing purposes we want to make sure the diff machinery\n-\t * works completely with the buffer. If there is anything emitted\n-\t * outside the emit_buffered_patch_line, then the order is screwed\n-\t * up and the tests will fail.\n-\t *\n-\t * TODO (later in this series):\n-\t * We'll unset this flag in a later patch.\n-\t */\n-\to->use_buffer = 1;\n+\n+\tif (o->color_moved) {\n+\t\tunsigned ignore_ws = DIFF_XDL_TST(o, IGNORE_WHITESPACE);\n+\t\to->use_buffer = 1;\n+\t\to->deleted_lines = xmallocz(sizeof(*o->deleted_lines));\n+\t\to->added_lines = xmallocz(sizeof(*o->added_lines));\n+\t\thashmap_init(o->deleted_lines, ignore_ws ?\n+\t\t\t(hashmap_cmp_fn)moved_entry_cmp_no_ws :\n+\t\t\t(hashmap_cmp_fn)moved_entry_cmp, 0);\n+\t\thashmap_init(o->added_lines, ignore_ws ?\n+\t\t\t(hashmap_cmp_fn)moved_entry_cmp_no_ws :\n+\t\t\t(hashmap_cmp_fn)moved_entry_cmp, 0);\n+\t}\n \n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n@@ -4870,6 +5110,11 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \t}\n \n \tif (o->use_buffer) {\n+\t\tif (o->color_moved) {\n+\t\t\tadd_lines_to_move_detection(o);\n+\t\t\tmark_color_as_moved(o);\n+\t\t}\n+\n \t\tfor (i = 0; i < o->line_buffer_nr; i++)\n \t\t\temit_buffered_patch_line(o, &o->line_buffer[i]);\n \n@@ -4958,6 +5203,7 @@ void diff_flush(struct diff_options *options)\n \t\tif (!options->file)\n \t\t\tdie_errno(\"Could not open /dev/null\");\n \t\toptions->close_file = 1;\n+\t\toptions->color_moved = 0;\n \t\tfor (i = 0; i < q->nr; i++) {\n \t\t\tstruct diff_filepair *p = q->queue[i];\n \t\t\tif (check_pair_status(p))\ndiff --git a/diff.h b/diff.h\nindex 64bc9bcd8c..b8b2a33ccc 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -7,6 +7,7 @@\n #include \"tree-walk.h\"\n #include \"pathspec.h\"\n #include \"object.h\"\n+#include \"hashmap.h\"\n \n struct rev_info;\n struct diff_options;\n@@ -227,6 +228,10 @@ struct diff_options {\n \n \tstruct buffered_patch_line *line_buffer;\n \tint line_buffer_nr, line_buffer_alloc;\n+\n+\tint color_moved;\n+\tstruct hashmap *deleted_lines;\n+\tstruct hashmap *added_lines;\n };\n \n void emit_line_fmt(struct diff_options *o, const char *set, const char *reset,\n@@ -243,7 +248,11 @@ enum color_diff {\n \tDIFF_FILE_NEW = 5,\n \tDIFF_COMMIT = 6,\n \tDIFF_WHITESPACE = 7,\n-\tDIFF_FUNCINFO = 8\n+\tDIFF_FUNCINFO = 8,\n+\tDIFF_FILE_OLD_MOVED = 9,\n+\tDIFF_FILE_OLD_MOVED_ALT = 10,\n+\tDIFF_FILE_NEW_MOVED = 11,\n+\tDIFF_FILE_NEW_MOVED_ALT = 12\n };\n const char *diff_get_color(int diff_use_color, enum color_diff ix);\n #define diff_get_color_opt(o, ix) \\\ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 289806d0c7..232d9ad55e 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -972,4 +972,233 @@ test_expect_success 'option overrides diff.wsErrorHighlight' '\n \n '\n \n+test_expect_success 'detect moved code, complete file' '\n+\tgit reset --hard &&\n+\tcat <<-\\EOF >test.c &&\n+\t#include\n+\tmain()\n+\t{\n+\tprintf(\"Hello World\");\n+\t}\n+\tEOF\n+\tgit add test.c &&\n+\tgit commit -m \"add main function\" &&\n+\tgit mv test.c main.c &&\n+\tgit diff HEAD --color-moved --no-renames | test_decode_color >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tdiff --git a/main.c b/main.c\n+\tnew file mode 100644\n+\tindex 0000000..a986c57\n+\t--- /dev/null\n+\t+++ b/main.c\n+\t@@ -0,0 +1,5 @@\n+\t+#include\n+\t+main()\n+\t+{\n+\t+printf(\"Hello World\");\n+\t+}\n+\tdiff --git a/test.c b/test.c\n+\tdeleted file mode 100644\n+\tindex a986c57..0000000\n+\t--- a/test.c\n+\t+++ /dev/null\n+\t@@ -1,5 +0,0 @@\n+\t-#include\n+\t-main()\n+\t-{\n+\t-printf(\"Hello World\");\n+\t-}\n+\tEOF\n+\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'detect moved code, inside file' '\n+\tgit reset --hard &&\n+\tcat <<-\\EOF >main.c &&\n+\t\t#include\n+\t\tint stuff()\n+\t\t{\n+\t\t\tprintf(\"Hello \");\n+\t\t\tprintf(\"World\\n\");\n+\t\t}\n+\n+\t\tint secure_foo(struct user *u)\n+\t\t{\n+\t\t\tif (!u->is_allowed_foo)\n+\t\t\t\treturn;\n+\t\t\tfoo(u);\n+\t\t}\n+\n+\t\tint main()\n+\t\t{\n+\t\t\tfoo();\n+\t\t}\n+\tEOF\n+\tcat <<-\\EOF >test.c &&\n+\t\t#include\n+\t\tint bar()\n+\t\t{\n+\t\t\tprintf(\"Hello World, but different\\n\");\n+\t\t}\n+\n+\t\tint another_function()\n+\t\t{\n+\t\t\tbar();\n+\t\t}\n+\tEOF\n+\tgit add main.c test.c &&\n+\tgit commit -m \"add main and test file\" &&\n+\tcat <<-\\EOF >main.c &&\n+\t\t#include\n+\t\tint stuff()\n+\t\t{\n+\t\t\tprintf(\"Hello \");\n+\t\t\tprintf(\"World\\n\");\n+\t\t}\n+\n+\t\tint main()\n+\t\t{\n+\t\t\tfoo();\n+\t\t}\n+\tEOF\n+\tcat <<-\\EOF >test.c &&\n+\t\t#include\n+\t\tint bar()\n+\t\t{\n+\t\t\tprintf(\"Hello World, but different\\n\");\n+\t\t}\n+\n+\t\tint secure_foo(struct user *u)\n+\t\t{\n+\t\t\tif (!u->is_allowed_foo)\n+\t\t\t\treturn;\n+\t\t\tfoo(u);\n+\t\t}\n+\n+\t\tint another_function()\n+\t\t{\n+\t\t\tbar();\n+\t\t}\n+\tEOF\n+\tgit diff HEAD --no-renames --color-moved| test_decode_color >actual &&\n+\tcat <<-\\EOF >expected &&\n+\tdiff --git a/main.c b/main.c\n+\tindex 27a619c..7cf9336 100644\n+\t--- a/main.c\n+\t+++ b/main.c\n+\t@@ -5,13 +5,6 @@ printf(\"Hello \");\n+\t printf(\"World\\n\");\n+\t }\n+\t \n+\t-int secure_foo(struct user *u)\n+\t-{\n+\t-if (!u->is_allowed_foo)\n+\t-return;\n+\t-foo(u);\n+\t-}\n+\t-\n+\t int main()\n+\t {\n+\t foo();\n+\tdiff --git a/test.c b/test.c\n+\tindex 1dc1d85..e34eb69 100644\n+\t--- a/test.c\n+\t+++ b/test.c\n+\t@@ -4,6 +4,13 @@ int bar()\n+\t printf(\"Hello World, but different\\n\");\n+\t }\n+\t \n+\t+int secure_foo(struct user *u)\n+\t+{\n+\t+if (!u->is_allowed_foo)\n+\t+return;\n+\t+foo(u);\n+\t+}\n+\t+\n+\t int another_function()\n+\t {\n+\t bar();\n+\tEOF\n+\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'detect permutations inside moved code, ' '\n+\t# reusing the move example from last test:\n+\tcat <<-\\EOF >main.c &&\n+\t\t#include\n+\t\tint stuff()\n+\t\t{\n+\t\t\tprintf(\"Hello \");\n+\t\t\tprintf(\"World\\n\");\n+\t\t}\n+\n+\t\tint main()\n+\t\t{\n+\t\t\tfoo();\n+\t\t}\n+\tEOF\n+\tcat <<-\\EOF >test.c &&\n+\t\t#include\n+\t\tint bar()\n+\t\t{\n+\t\t\tprintf(\"Hello World, but different\\n\");\n+\t\t}\n+\n+\t\tint secure_foo(struct user *u)\n+\t\t{\n+\t\t\tfoo(u);\n+\t\t\tif (!u->is_allowed_foo)\n+\t\t\t\treturn;\n+\t\t}\n+\n+\t\tint another_function()\n+\t\t{\n+\t\t\tbar();\n+\t\t}\n+\tEOF\n+\tgit diff HEAD --no-renames --color-moved| test_decode_color >actual &&\n+\tcat <<-\\EOF >expected &&\n+\tdiff --git a/main.c b/main.c\n+\tindex 27a619c..7cf9336 100644\n+\t--- a/main.c\n+\t+++ b/main.c\n+\t@@ -5,13 +5,6 @@ printf(\"Hello \");\n+\t printf(\"World\\n\");\n+\t }\n+\t \n+\t-int secure_foo(struct user *u)\n+\t-{\n+\t-if (!u->is_allowed_foo)\n+\t-return;\n+\t-foo(u);\n+\t-}\n+\t-\n+\t int main()\n+\t {\n+\t foo();\n+\tdiff --git a/test.c b/test.c\n+\tindex 1dc1d85..2bedec9 100644\n+\t--- a/test.c\n+\t+++ b/test.c\n+\t@@ -4,6 +4,13 @@ int bar()\n+\t printf(\"Hello World, but different\\n\");\n+\t }\n+\t \n+\t+int secure_foo(struct user *u)\n+\t+{\n+\t+foo(u);\n+\t+if (!u->is_allowed_foo)\n+\t+return;\n+\t+}\n+\t+\n+\t int another_function()\n+\t {\n+\t bar();\n+\tEOF\n+\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320005","messageId":"20170517025857.32320-13-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 12/20] submodule.c: convert show_submodule_summary to use emit_line_fmt","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:49Z","receivedAt":"2017-05-17T02:59:49Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis prepares the code for submodules to go through the\nemit_line function.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 20 +++++++---------\n diff.h | 5 ++++\n submodule.c | 78 ++++++++++++++++++++++++++++++-------------------------------\n submodule.h | 9 +++----\n 4 files changed, 56 insertions(+), 56 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 690794aeb8..7c8d6a5d12 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -516,8 +516,8 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n-static void emit_line(struct diff_options *o, const char *set, const char *reset,\n-\t\t int add_line_prefix, int sign, const char *line, int len)\n+void emit_line(struct diff_options *o, const char *set, const char *reset,\n+\t int add_line_prefix, int sign, const char *line, int len)\n {\n \tint has_trailing_newline, has_trailing_carriage_return;\n \tFILE *file = o->file;\n@@ -547,10 +547,10 @@ static void emit_line(struct diff_options *o, const char *set, const char *reset\n \t\tfputc('\\n', file);\n }\n \n-static void emit_line_fmt(struct diff_options *o,\n-\t\t\t const char *set, const char *reset,\n-\t\t\t int add_line_prefix,\n-\t\t\t const char *fmt, ...)\n+void emit_line_fmt(struct diff_options *o,\n+\t\t const char *set, const char *reset,\n+\t\t int add_line_prefix,\n+\t\t const char *fmt, ...)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tva_list ap;\n@@ -2386,8 +2386,7 @@ static void builtin_diff(const char *name_a,\n \t (!two->mode || S_ISGITLINK(two->mode))) {\n \t\tconst char *del = diff_get_color_opt(o, DIFF_FILE_OLD);\n \t\tconst char *add = diff_get_color_opt(o, DIFF_FILE_NEW);\n-\t\tshow_submodule_summary(o->file, one->path ? one->path : two->path,\n-\t\t\t\tline_prefix,\n+\t\tshow_submodule_summary(o, one->path ? one->path : two->path,\n \t\t\t\t&one->oid, &two->oid,\n \t\t\t\ttwo->dirty_submodule,\n \t\t\t\tmeta, del, add, reset);\n@@ -2397,11 +2396,10 @@ static void builtin_diff(const char *name_a,\n \t\t (!two->mode || S_ISGITLINK(two->mode))) {\n \t\tconst char *del = diff_get_color_opt(o, DIFF_FILE_OLD);\n \t\tconst char *add = diff_get_color_opt(o, DIFF_FILE_NEW);\n-\t\tshow_submodule_inline_diff(o->file, one->path ? one->path : two->path,\n-\t\t\t\tline_prefix,\n+\t\tshow_submodule_inline_diff(o, one->path ? one->path : two->path,\n \t\t\t\t&one->oid, &two->oid,\n \t\t\t\ttwo->dirty_submodule,\n-\t\t\t\tmeta, del, add, reset, o);\n+\t\t\t\tmeta, del, add, reset);\n \t\treturn;\n \t}\n \ndiff --git a/diff.h b/diff.h\nindex 5be1ee77a7..6e14100102 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -188,6 +188,11 @@ struct diff_options {\n \tint diff_path_counter;\n };\n \n+void emit_line_fmt(struct diff_options *o, const char *set, const char *reset,\n+\t\t int add_line_prefix, const char *fmt, ...);\n+void emit_line(struct diff_options *o, const char *set, const char *reset,\n+\t int add_line_prefix, int sign, const char *line, int len);\n+\n enum color_diff {\n \tDIFF_RESET = 0,\n \tDIFF_CONTEXT = 1,\ndiff --git a/submodule.c b/submodule.c\nindex d3299e29c0..5996ebca44 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -362,8 +362,8 @@ static int prepare_submodule_summary(struct rev_info *rev, const char *path,\n \treturn prepare_revision_walk(rev);\n }\n \n-static void print_submodule_summary(struct rev_info *rev, FILE *f,\n-\t\tconst char *line_prefix,\n+static void print_submodule_summary(struct rev_info *rev,\n+\t\tstruct diff_options *o,\n \t\tconst char *del, const char *add, const char *reset)\n {\n \tstatic const char format[] = \" %m %s\";\n@@ -375,18 +375,12 @@ static void print_submodule_summary(struct rev_info *rev, FILE *f,\n \t\tctx.date_mode = rev->date_mode;\n \t\tctx.output_encoding = get_log_output_encoding();\n \t\tstrbuf_setlen(&sb, 0);\n-\t\tstrbuf_addstr(&sb, line_prefix);\n-\t\tif (commit->object.flags & SYMMETRIC_LEFT) {\n-\t\t\tif (del)\n-\t\t\t\tstrbuf_addstr(&sb, del);\n-\t\t}\n-\t\telse if (add)\n-\t\t\tstrbuf_addstr(&sb, add);\n \t\tformat_commit_message(commit, format, &sb, &ctx);\n-\t\tif (reset)\n-\t\t\tstrbuf_addstr(&sb, reset);\n \t\tstrbuf_addch(&sb, '\\n');\n-\t\tfprintf(f, \"%s\", sb.buf);\n+\t\tif (commit->object.flags & SYMMETRIC_LEFT)\n+\t\t\temit_line(o, del, reset, 1, 0, sb.buf, sb.len);\n+\t\telse if (add)\n+\t\t\temit_line(o, add, reset, 1, 0, sb.buf, sb.len);\n \t}\n \tstrbuf_release(&sb);\n }\n@@ -413,8 +407,7 @@ void prepare_submodule_repo_env(struct argv_array *out)\n * attempt to lookup both the left and right commits and put them into the\n * left and right pointers.\n */\n-static void show_submodule_header(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+static void show_submodule_header(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n \t\tconst char *reset,\n@@ -426,11 +419,11 @@ static void show_submodule_header(FILE *f, const char *path,\n \tint fast_forward = 0, fast_backward = 0;\n \n \tif (dirty_submodule & DIRTY_SUBMODULE_UNTRACKED)\n-\t\tfprintf(f, \"%sSubmodule %s contains untracked content\\n\",\n-\t\t\tline_prefix, path);\n+\t\temit_line_fmt(o, NULL, NULL, 1,\n+\t\t\t \"Submodule %s contains untracked content\\n\", path);\n \tif (dirty_submodule & DIRTY_SUBMODULE_MODIFIED)\n-\t\tfprintf(f, \"%sSubmodule %s contains modified content\\n\",\n-\t\t\tline_prefix, path);\n+\t\temit_line_fmt(o, NULL, NULL, 1,\n+\t\t\t \"Submodule %s contains modified content\\n\", path);\n \n \tif (is_null_oid(one))\n \t\tmessage = \"(new submodule)\";\n@@ -472,21 +465,20 @@ static void show_submodule_header(FILE *f, const char *path,\n \t}\n \n output_header:\n-\tstrbuf_addf(&sb, \"%s%sSubmodule %s \", line_prefix, meta, path);\n+\tstrbuf_addf(&sb, \"Submodule %s \", path);\n \tstrbuf_add_unique_abbrev(&sb, one->hash, DEFAULT_ABBREV);\n \tstrbuf_addstr(&sb, (fast_backward || fast_forward) ? \"..\" : \"...\");\n \tstrbuf_add_unique_abbrev(&sb, two->hash, DEFAULT_ABBREV);\n \tif (message)\n-\t\tstrbuf_addf(&sb, \" %s%s\\n\", message, reset);\n+\t\tstrbuf_addf(&sb, \" %s\\n\", message);\n \telse\n-\t\tstrbuf_addf(&sb, \"%s:%s\\n\", fast_backward ? \" (rewind)\" : \"\", reset);\n-\tfwrite(sb.buf, sb.len, 1, f);\n+\t\tstrbuf_addf(&sb, \"%s:\\n\", fast_backward ? \" (rewind)\" : \"\");\n+\temit_line(o, meta, reset, 1, 0, sb.buf, sb.len);\n \n \tstrbuf_release(&sb);\n }\n \n-void show_submodule_summary(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+void show_submodule_summary(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n \t\tconst char *del, const char *add, const char *reset)\n@@ -495,7 +487,7 @@ void show_submodule_summary(FILE *f, const char *path,\n \tstruct commit *left = NULL, *right = NULL;\n \tstruct commit_list *merge_bases = NULL;\n \n-\tshow_submodule_header(f, path, line_prefix, one, two, dirty_submodule,\n+\tshow_submodule_header(o, path, one, two, dirty_submodule,\n \t\t\t meta, reset, &left, &right, &merge_bases);\n \n \t/*\n@@ -508,11 +500,12 @@ void show_submodule_summary(FILE *f, const char *path,\n \n \t/* Treat revision walker failure the same as missing commits */\n \tif (prepare_submodule_summary(&rev, path, left, right, merge_bases)) {\n-\t\tfprintf(f, \"%s(revision walker failed)\\n\", line_prefix);\n+\t\tconst char *error = \"(revision walker failed)\\n\";\n+\t\temit_line(o, NULL, NULL, 1, 0, error, strlen(error));\n \t\tgoto out;\n \t}\n \n-\tprint_submodule_summary(&rev, f, line_prefix, del, add, reset);\n+\tprint_submodule_summary(&rev, o, del, add, reset);\n \n out:\n \tif (merge_bases)\n@@ -521,20 +514,18 @@ void show_submodule_summary(FILE *f, const char *path,\n \tclear_commit_marks(right, ~0);\n }\n \n-void show_submodule_inline_diff(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+void show_submodule_inline_diff(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n-\t\tconst char *del, const char *add, const char *reset,\n-\t\tconst struct diff_options *o)\n+\t\tconst char *del, const char *add, const char *reset)\n {\n \tconst struct object_id *old = &empty_tree_oid, *new = &empty_tree_oid;\n \tstruct commit *left = NULL, *right = NULL;\n \tstruct commit_list *merge_bases = NULL;\n-\tstruct strbuf submodule_dir = STRBUF_INIT;\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tstruct strbuf sb = STRBUF_INIT;\n \n-\tshow_submodule_header(f, path, line_prefix, one, two, dirty_submodule,\n+\tshow_submodule_header(o, path, one, two, dirty_submodule,\n \t\t\t meta, reset, &left, &right, &merge_bases);\n \n \t/* We need a valid left and right commit to display a difference */\n@@ -547,15 +538,14 @@ void show_submodule_inline_diff(FILE *f, const char *path,\n \tif (right)\n \t\tnew = two;\n \n-\tfflush(f);\n \tcp.git_cmd = 1;\n \tcp.dir = path;\n-\tcp.out = dup(fileno(f));\n+\tcp.out = -1;\n \tcp.no_stdin = 1;\n \n \t/* TODO: other options may need to be passed here. */\n \targv_array_push(&cp.args, \"diff\");\n-\targv_array_pushf(&cp.args, \"--line-prefix=%s\", line_prefix);\n+\targv_array_pushf(&cp.args, \"--line-prefix=%s\", diff_line_prefix(o));\n \tif (DIFF_OPT_TST(o, REVERSE_DIFF)) {\n \t\targv_array_pushf(&cp.args, \"--src-prefix=%s%s/\",\n \t\t\t\t o->b_prefix, path);\n@@ -578,11 +568,21 @@ void show_submodule_inline_diff(FILE *f, const char *path,\n \t\targv_array_push(&cp.args, oid_to_hex(new));\n \n \tprepare_submodule_repo_env(&cp.env_array);\n-\tif (run_command(&cp))\n-\t\tfprintf(f, \"(diff failed)\\n\");\n+\tif (start_command(&cp)) {\n+\t\tconst char *error = \"(diff failed)\\n\";\n+\t\temit_line(o, NULL, NULL, 1, 0, error, strlen(error));\n+\t}\n+\n+\twhile (strbuf_getwholeline_fd(&sb, cp.out, '\\n') != EOF)\n+\t\temit_line(o, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\n+\tif (finish_command(&cp)) {\n+\t\tconst char *error = \"(diff failed)\\n\";\n+\t\temit_line(o, NULL, NULL, 1, 0, error, strlen(error));\n+\t}\n \n done:\n-\tstrbuf_release(&submodule_dir);\n+\tstrbuf_release(&sb);\n \tif (merge_bases)\n \t\tfree_commit_list(merge_bases);\n \tif (left)\ndiff --git a/submodule.h b/submodule.h\nindex 1277480add..9df0a3aea2 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -53,17 +53,14 @@ extern int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst);\n extern const char *submodule_strategy_to_string(const struct submodule_update_strategy *s);\n extern void handle_ignore_submodules_arg(struct diff_options *, const char *);\n-extern void show_submodule_summary(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+extern void show_submodule_summary(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n \t\tconst char *del, const char *add, const char *reset);\n-extern void show_submodule_inline_diff(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+extern void show_submodule_inline_diff(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n-\t\tconst char *del, const char *add, const char *reset,\n-\t\tconst struct diff_options *opt);\n+\t\tconst char *del, const char *add, const char *reset);\n extern void set_config_fetch_recurse_submodules(int value);\n extern void set_config_update_recurse_submodules(int value);\n /* Check if we want to update any submodule.*/\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320006","messageId":"20170517025857.32320-19-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 18/20] diff.c: emit_line includes whitespace highlighting","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:55Z","receivedAt":"2017-05-17T02:59:52Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Currently any whitespace highlighting happens outside the emit_line\nfunction. Teach the highlighting to emit_line, triggered by a new\nparameter.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 104 +++++++++++++++++++++++++++++++++++-------------------------\n diff.h | 4 ++-\n submodule.c | 14 ++++----\n 3 files changed, 71 insertions(+), 51 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 964b5cb5a7..34482a6a09 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -516,15 +516,33 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n-void emit_line(struct diff_options *o, const char *set, const char *reset,\n-\t int add_line_prefix, int sign, const char *line, int len)\n+void emit_line(struct diff_options *o,\n+\t const char *set, const char *reset,\n+\t int add_line_prefix, int markup_ws,\n+\t int sign, const char *line, int len)\n {\n+\tconst char *ws;\n \tint has_trailing_newline, has_trailing_carriage_return;\n \tFILE *file = o->file;\n \n \tif (add_line_prefix)\n \t\tfputs(diff_line_prefix(o), file);\n \n+\tif (markup_ws) {\n+\t\tws = diff_get_color(o->use_color, DIFF_WHITESPACE);\n+\n+\t\tif (set)\n+\t\t\tfputs(set, file);\n+\t\tif (sign)\n+\t\t\tfputc(sign, file);\n+\t\tif (reset)\n+\t\t\tfputs(reset, file);\n+\t\tws = diff_get_color(o->use_color, DIFF_WHITESPACE);\n+\t\tws_check_emit(line, len, o->ws_rule,\n+\t\t\t file, set, reset, ws);\n+\t\treturn;\n+\t}\n+\n \thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n \tif (has_trailing_newline)\n \t\tlen--;\n@@ -558,7 +576,7 @@ void emit_line_fmt(struct diff_options *o,\n \tstrbuf_vaddf(&sb, fmt, ap);\n \tva_end(ap);\n \n-\temit_line(o, set, reset, add_line_prefix, 0, sb.buf, sb.len);\n+\temit_line(o, set, reset, add_line_prefix, 0, 0, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n }\n \n@@ -590,16 +608,15 @@ static void emit_line_checked(const char *reset,\n \t}\n \n \tif (!ws)\n-\t\temit_line(ecbdata->opt, set, reset, 1, sign, line, len);\n+\t\temit_line(ecbdata->opt, set, reset, 1, 0, sign, line, len);\n \telse if (sign == '+' && new_blank_line_at_eof(ecbdata, line, len))\n \t\t/* Blank line at EOF - paint '+' as well */\n-\t\temit_line(ecbdata->opt, ws, reset, 1, sign, line, len);\n+\t\temit_line(ecbdata->opt, ws, reset, 1, 1, sign, line, len);\n \telse {\n \t\t/* Emit just the prefix, then the rest. */\n-\t\temit_line(ecbdata->opt, set, reset, 1, sign, \"\", 0);\n-\t\tws_check_emit(line, len, ecbdata->ws_rule,\n-\t\t\t ecbdata->opt->file, set, reset, ws);\n+\t\temit_line(ecbdata->opt, set, reset, 1, 1, sign, line, len);\n \t}\n+\n }\n \n static void emit_add_line(const char *reset,\n@@ -646,7 +663,7 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \tif (len < 10 ||\n \t memcmp(line, atat, 2) ||\n \t !(ep = memmem(line + 2, len - 2, atat, 2))) {\n-\t\temit_line(ecbdata->opt, context, reset, 1, 0, line, len);\n+\t\temit_line(ecbdata->opt, context, reset, 1, 0, 0, line, len);\n \t\treturn;\n \t}\n \tep += 2; /* skip over @@ */\n@@ -682,7 +699,7 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \tstrbuf_add(&msgbuf, line + len, org_len - len);\n \tstrbuf_complete_line(&msgbuf);\n \n-\temit_line(ecbdata->opt, \"\", \"\", 1, 0, msgbuf.buf, msgbuf.len);\n+\temit_line(ecbdata->opt, \"\", \"\", 1, 0, 0, msgbuf.buf, msgbuf.len);\n \tstrbuf_release(&msgbuf);\n }\n \n@@ -755,7 +772,7 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\tstatic const char *nneof = \"\\\\ No newline at end of file\\n\";\n \t\tconst char *context = diff_get_color(ecb->color_diff,\n \t\t\t\t\t\t DIFF_CONTEXT);\n-\t\temit_line(ecb->opt, context, reset, 1, 0,\n+\t\temit_line(ecb->opt, context, reset, 1, 0, 0,\n \t\t\t nneof, strlen(nneof));\n \t\tstrbuf_release(&sb);\n \t}\n@@ -831,7 +848,7 @@ static void emit_rewrite_diff(const char *name_a,\n \tstrbuf_addstr(&out, \" +\");\n \tadd_line_count(&out, lc_b);\n \tstrbuf_addstr(&out, \" @@\\n\");\n-\temit_line(o, fraginfo, reset, 1, 0, out.buf, out.len);\n+\temit_line(o, fraginfo, reset, 1, 0, 0, out.buf, out.len);\n \tstrbuf_release(&out);\n \n \tif (lc_a && !o->irreversible_delete)\n@@ -904,7 +921,7 @@ static int fn_out_diff_words_write_helper(struct diff_options *o,\n \twhile (count) {\n \t\tchar *p = memchr(buf, '\\n', count);\n \t\tif (print)\n-\t\t\temit_line(o, NULL, NULL, 1, 0, \"\", 0);\n+\t\t\temit_line(o, NULL, NULL, 1, 0, 0, \"\", 0);\n \n \t\tif (p != buf) {\n \t\t\tconst char *reset = st_el->color && *st_el->color ?\n@@ -913,14 +930,14 @@ static int fn_out_diff_words_write_helper(struct diff_options *o,\n \t\t\tstrbuf_add(&sb, buf, p ? p - buf : count);\n \t\t\tstrbuf_addstr(&sb, st_el->suffix);\n \t\t\temit_line(o, st_el->color, reset,\n-\t\t\t\t 0, 0, sb.buf, sb.len);\n+\t\t\t\t 0, 0, 0, sb.buf, sb.len);\n \t\t\tstrbuf_reset(&sb);\n \t\t}\n \t\tif (!p)\n \t\t\tgoto out;\n \n \t\tstrbuf_addstr(&sb, newline);\n-\t\temit_line(o, NULL, NULL, 0, 0, sb.buf, sb.len);\n+\t\temit_line(o, NULL, NULL, 0, 0, 0, sb.buf, sb.len);\n \t\tstrbuf_reset(&sb);\n \t\tcount -= p + 1 - buf;\n \t\tbuf = p + 1;\n@@ -1135,7 +1152,7 @@ static void diff_words_show(struct diff_words_data *diff_words)\n \tif (diff_words->current_plus != diff_words->plus.text.ptr +\n \t\t\tdiff_words->plus.text.size) {\n \t\tif (color_words_output_graph_prefix(diff_words))\n-\t\t\temit_line(diff_words->opt, NULL, NULL, 1, 0, \"\", 0);\n+\t\t\temit_line(diff_words->opt, NULL, NULL, 1, 0, 0, \"\", 0);\n \t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t&style->ctx, style->newline,\n \t\t\tdiff_words->plus.text.ptr + diff_words->plus.text.size\n@@ -1294,7 +1311,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \to->found_changes = 1;\n \n \tif (ecbdata->header) {\n-\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t ecbdata->header->buf, ecbdata->header->len);\n \t\tstrbuf_release(ecbdata->header);\n \t\tecbdata->header = NULL;\n@@ -1347,8 +1364,8 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t}\n \t\tdiff_words_flush(ecbdata);\n \t\tif (ecbdata->diff_words->type == DIFF_WORDS_PORCELAIN) {\n-\t\t\temit_line(o, context, reset, 1, 0, line, len);\n-\t\t\temit_line(o, NULL, NULL, 0, 0, \"~\\n\", 2);\n+\t\t\temit_line(o, context, reset, 1, 0, 0, line, len);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0, \"~\\n\", 2);\n \t\t} else {\n \t\t\t/*\n \t\t\t * Skip the prefix character, if any. With\n@@ -1359,7 +1376,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t\t line++;\n \t\t\t len--;\n \t\t\t}\n-\t\t\temit_line(o, context, reset, 1, 0, line, len);\n+\t\t\temit_line(o, context, reset, 1, 0, 0, line, len);\n \t\t}\n \t\treturn;\n \t}\n@@ -1382,7 +1399,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t/* incomplete line at the end */\n \t\tecbdata->lno_in_preimage++;\n \t\temit_line(o, diff_get_color(ecbdata->color_diff, DIFF_CONTEXT),\n-\t\t\t reset, 1, 0, line, len);\n+\t\t\t reset, 1, 0, 0, line, len);\n \t\tbreak;\n \t}\n }\n@@ -1571,7 +1588,7 @@ void print_stat_summary_0(struct diff_options *options, int files,\n \tif (!files) {\n \t\tassert(insertions == 0 && deletions == 0);\n \t\tstrbuf_addstr(&sb, \" 0 files changed\");\n-\t\temit_line(options, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\t\temit_line(options, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \t\treturn;\n \t}\n \n@@ -1599,7 +1616,7 @@ void print_stat_summary_0(struct diff_options *options, int files,\n \t\t\t deletions);\n \t}\n \tstrbuf_addch(&sb, '\\n');\n-\temit_line(options, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\temit_line(options, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n }\n \n@@ -1783,7 +1800,7 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t\tstrbuf_addf(&out, \" %*s\", number_width, \"Bin\");\n \t\t\tif (!added && !deleted) {\n \t\t\t\tstrbuf_addch(&out, '\\n');\n-\t\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\t\t\tstrbuf_reset(&out);\n \t\t\t\tcontinue;\n \t\t\t}\n@@ -1793,14 +1810,14 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t\tstrbuf_addf(&out, \"%s%\"PRIuMAX\"%s\",\n \t\t\t\tadd_c, added, reset);\n \t\t\tstrbuf_addstr(&out, \" bytes\\n\");\n-\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n \t\telse if (file->is_unmerged) {\n \t\t\tshow_name(&out, prefix, name, len);\n \t\t\tstrbuf_addstr(&out, \" Unmerged\\n\");\n-\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n@@ -1831,7 +1848,7 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\tshow_graph(&out, '+', add, add_c, reset);\n \t\tshow_graph(&out, '-', del, del_c, reset);\n \t\tstrbuf_addch(&out, '\\n');\n-\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\tstrbuf_reset(&out);\n \t}\n \n@@ -1853,7 +1870,7 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\tif (i < count)\n \t\t\tcontinue;\n \t\tif (!extra_shown)\n-\t\t\temit_line(options, NULL, NULL, 1, 0,\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0,\n \t\t\t\t \" ...\\n\", strlen(\" ...\\n\"));\n \t\textra_shown = 1;\n \t}\n@@ -2207,7 +2224,7 @@ static void checkdiff_consume(void *priv, char *line, unsigned long len)\n \t\tfprintf(data->o->file, \"%s%s:%d: %s.\\n\",\n \t\t\tline_prefix, data->filename, data->lineno, err);\n \t\tfree(err);\n-\t\temit_line(data->o, set, reset, 1, 0, line, 1);\n+\t\temit_line(data->o, set, reset, 1, 0, 0, line, 1);\n \t\tws_check_emit(line + 1, len - 1, data->ws_rule,\n \t\t\t data->o->file, set, reset, ws);\n \t} else if (line[0] == ' ') {\n@@ -2303,9 +2320,9 @@ static void emit_binary_diff_body(struct diff_options *o,\n \t\tline[len++] = '\\n';\n \t\tline[len] = '\\0';\n \n-\t\temit_line(o, NULL, NULL, 1, 0, line, len);\n+\t\temit_line(o, NULL, NULL, 1, 0, 0, line, len);\n \t}\n-\temit_line(o, NULL, NULL, 1, 0, \"\\n\", 1);\n+\temit_line(o, NULL, NULL, 1, 0, 0, \"\\n\", 1);\n \tfree(data);\n }\n \n@@ -2314,7 +2331,7 @@ static void emit_binary_diff(struct diff_options *o,\n {\n \tconst char *s = \"GIT binary patch\\n\";\n \tconst int len = strlen(s);\n-\temit_line(o, NULL, NULL, 1, 0, s, len);\n+\temit_line(o, NULL, NULL, 1, 0, 0, s, len);\n \temit_binary_diff_body(o, one, two);\n \temit_binary_diff_body(o, two, one);\n }\n@@ -2457,7 +2474,7 @@ static void builtin_diff(const char *name_a,\n \t\tif (complete_rewrite &&\n \t\t (textconv_one || !diff_filespec_is_binary(one)) &&\n \t\t (textconv_two || !diff_filespec_is_binary(two))) {\n-\t\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t\temit_rewrite_diff(name_a, name_b, one, two,\n \t\t\t\t\t\ttextconv_one, textconv_two, o);\n@@ -2467,7 +2484,7 @@ static void builtin_diff(const char *name_a,\n \t}\n \n \tif (o->irreversible_delete && lbl[1][0] == '/') {\n-\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n+\t\temit_line(o, NULL, NULL, 0, 0, 0, header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tgoto free_ab_and_return;\n \t} else if (!DIFF_OPT_TST(o, TEXT) &&\n@@ -2478,11 +2495,11 @@ static void builtin_diff(const char *name_a,\n \t\t !DIFF_OPT_TST(o, BINARY)) {\n \t\t\tif (!oidcmp(&one->oid, &two->oid)) {\n \t\t\t\tif (must_show_header)\n-\t\t\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t\t\t\t header.buf, header.len);\n \t\t\t\tgoto free_ab_and_return;\n \t\t\t}\n-\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t\t header.buf, header.len);\n \t\t\temit_line_fmt(o, NULL, NULL, 1,\n \t\t\t\t \"Binary files %s and %s differ\\n\",\n@@ -2495,11 +2512,11 @@ static void builtin_diff(const char *name_a,\n \t\tif (mf1.size == mf2.size &&\n \t\t !memcmp(mf1.ptr, mf2.ptr, mf1.size)) {\n \t\t\tif (must_show_header)\n-\t\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t\t\t header.buf, header.len);\n \t\t\tgoto free_ab_and_return;\n \t\t}\n-\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tif (DIFF_OPT_TST(o, BINARY))\n@@ -2519,7 +2536,7 @@ static void builtin_diff(const char *name_a,\n \t\tconst struct userdiff_funcname *pe;\n \n \t\tif (must_show_header) {\n-\t\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t}\n \n@@ -2536,6 +2553,7 @@ static void builtin_diff(const char *name_a,\n \t\tecbdata.label_path = lbl;\n \t\tecbdata.color_diff = want_color(o->use_color);\n \t\tecbdata.ws_rule = whitespace_rule(name_b);\n+\t\to->ws_rule = ecbdata.ws_rule;\n \t\tif (ecbdata.ws_rule & WS_BLANK_AT_EOF)\n \t\t\tcheck_blank_at_eof(&mf1, &mf2, &ecbdata);\n \t\tecbdata.opt = o;\n@@ -4510,7 +4528,7 @@ static void show_file_mode_name(struct diff_options *opt, const char *newdelete,\n \n \tquote_c_style(fs->path, &sb, NULL, 0);\n \tstrbuf_addch(&sb, '\\n');\n-\temit_line(opt, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\temit_line(opt, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n }\n \n@@ -4562,7 +4580,7 @@ static void diff_summary(struct diff_options *opt, struct diff_filepair *p)\n \t\t\tstrbuf_addstr(&sb, \" rewrite \");\n \t\t\tquote_c_style(p->two->path, &sb, NULL, 0);\n \t\t\tstrbuf_addf(&sb, \" (%d%%)\\n\", similarity_index(p));\n-\t\t\temit_line(opt, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\t\t\temit_line(opt, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \t\t}\n \t\tshow_mode_change(opt, p, !p->score);\n \t\tbreak;\n@@ -4869,10 +4887,10 @@ void diff_flush(struct diff_options *options)\n \t\t\tterm[0] = options->line_termination;\n \t\t\tterm[1] = '\\0';\n \n-\t\t\temit_line(options, NULL, NULL, 1, 0, term, !!term[0]);\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0, term, !!term[0]);\n \t\t\tif (options->stat_sep) {\n \t\t\t\t/* attach patch instead of inline */\n-\t\t\t\temit_line(options, NULL, NULL, 0, 0,\n+\t\t\t\temit_line(options, NULL, NULL, 0, 0, 0,\n \t\t\t\t\t options->stat_sep,\n \t\t\t\t\t strlen(options->stat_sep));\n \t\t\t}\ndiff --git a/diff.h b/diff.h\nindex b75b0d7283..267acf1980 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -186,12 +186,14 @@ struct diff_options {\n \tvoid *output_prefix_data;\n \n \tint diff_path_counter;\n+\n+\tunsigned ws_rule;\n };\n \n void emit_line_fmt(struct diff_options *o, const char *set, const char *reset,\n \t\t int add_line_prefix, const char *fmt, ...);\n void emit_line(struct diff_options *o, const char *set, const char *reset,\n-\t int add_line_prefix, int sign, const char *line, int len);\n+\t int add_line_prefix, int markup_ws, int sign, const char *line, int len);\n \n enum color_diff {\n \tDIFF_RESET = 0,\ndiff --git a/submodule.c b/submodule.c\nindex 5996ebca44..868f913971 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -378,9 +378,9 @@ static void print_submodule_summary(struct rev_info *rev,\n \t\tformat_commit_message(commit, format, &sb, &ctx);\n \t\tstrbuf_addch(&sb, '\\n');\n \t\tif (commit->object.flags & SYMMETRIC_LEFT)\n-\t\t\temit_line(o, del, reset, 1, 0, sb.buf, sb.len);\n+\t\t\temit_line(o, del, reset, 1, 0, 0, sb.buf, sb.len);\n \t\telse if (add)\n-\t\t\temit_line(o, add, reset, 1, 0, sb.buf, sb.len);\n+\t\t\temit_line(o, add, reset, 1, 0, 0, sb.buf, sb.len);\n \t}\n \tstrbuf_release(&sb);\n }\n@@ -473,7 +473,7 @@ static void show_submodule_header(struct diff_options *o, const char *path,\n \t\tstrbuf_addf(&sb, \" %s\\n\", message);\n \telse\n \t\tstrbuf_addf(&sb, \"%s:\\n\", fast_backward ? \" (rewind)\" : \"\");\n-\temit_line(o, meta, reset, 1, 0, sb.buf, sb.len);\n+\temit_line(o, meta, reset, 1, 0, 0, sb.buf, sb.len);\n \n \tstrbuf_release(&sb);\n }\n@@ -501,7 +501,7 @@ void show_submodule_summary(struct diff_options *o, const char *path,\n \t/* Treat revision walker failure the same as missing commits */\n \tif (prepare_submodule_summary(&rev, path, left, right, merge_bases)) {\n \t\tconst char *error = \"(revision walker failed)\\n\";\n-\t\temit_line(o, NULL, NULL, 1, 0, error, strlen(error));\n+\t\temit_line(o, NULL, NULL, 1, 0, 0, error, strlen(error));\n \t\tgoto out;\n \t}\n \n@@ -570,15 +570,15 @@ void show_submodule_inline_diff(struct diff_options *o, const char *path,\n \tprepare_submodule_repo_env(&cp.env_array);\n \tif (start_command(&cp)) {\n \t\tconst char *error = \"(diff failed)\\n\";\n-\t\temit_line(o, NULL, NULL, 1, 0, error, strlen(error));\n+\t\temit_line(o, NULL, NULL, 1, 0, 0, error, strlen(error));\n \t}\n \n \twhile (strbuf_getwholeline_fd(&sb, cp.out, '\\n') != EOF)\n-\t\temit_line(o, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\t\temit_line(o, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \n \tif (finish_command(&cp)) {\n \t\tconst char *error = \"(diff failed)\\n\";\n-\t\temit_line(o, NULL, NULL, 1, 0, error, strlen(error));\n+\t\temit_line(o, NULL, NULL, 1, 0, 0, error, strlen(error));\n \t}\n \n done:\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320007","messageId":"20170517025857.32320-6-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 05/20] diff.c: emit_line_0 can handle no color setting","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:42Z","receivedAt":"2017-05-17T02:59:54Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In later patches we may pass lines that are not colored to the central\nfunction emit_line_0, so we need to emit the color only when it is\nnon-NULL.\n\nWe could have chosen to pass \"\" instead of NULL, but that would be more\nwork.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 73e55b0c10..6c1886d495 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -532,11 +532,13 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n \t\tlen--;\n \n \tif (len || sign) {\n-\t\tfputs(set, file);\n+\t\tif (set)\n+\t\t\tfputs(set, file);\n \t\tif (sign)\n \t\t\tfputc(sign, file);\n \t\tfwrite(line, len, 1, file);\n-\t\tfputs(reset, file);\n+\t\tif (reset)\n+\t\t\tfputs(reset, file);\n \t}\n \tif (has_trailing_carriage_return)\n \t\tfputc('\\r', file);\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320008","messageId":"20170517025857.32320-20-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 19/20] diff: buffer all output if asked to","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:56Z","receivedAt":"2017-05-17T02:59:58Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Introduce a new option 'use_buffer' in the struct diff_options which\ncontrols whether all output is buffered up until all output is available.\n\nWe'll have a new struct 'buffered_patch_line' in diff.h which will be\nused to buffer each line. The buffered_patch_line will duplicate the\nmemory of the line to buffer as that is easiest to reason about for now.\nIn a future patch we may want to decrease the memory usage by not\nduplicating all output for buffering but rather we may want to store\noffsets into the file or in case of hunk descriptions such as the\nsimilarity score, we could just store the relevant number and\nreproduce the text later on.\n\nThis approach was chosen as a first step because it is quite simple\ncompared to the alternative with less memory footprint.\n\nemit_line factors out the emission part into emit_line_emission,\nand depending on the diff_options->use_buffer the emission\nwill be performed directly when calling emit_line or after the\nwhole process is done, i.e. by buffering we have add the possibility\nfor a second pass over the whole output before doing the actual\noutput.\n\nIn 6440d34 (2012-03-14, diff: tweak a _copy_ of diff_options with\nword-diff) we introduced a duplicate diff options struct for word\nemissions as we may have different regex settings in there.\nWhen buffering the output, we need to operate on just one buffer,\nso we have to copy back the emissions of the word buffer into the\nmain buffer.\n\nUnconditionally enable output via buffer in this patch as it yields\na great opportunity for testing, i.e. all the diff tests from the\ntest suite pass without having reordering issues (i.e. only parts\nof the output got buffered, and we forgot to buffer other parts).\nThe test suite passes, which gives confidence that we converted all\nfunctions to use emit_line for output.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 154 ++++++++++++++++++++++++++++++++++++++++++++++++++---------------\n diff.h | 39 +++++++++++++++++\n 2 files changed, 159 insertions(+), 34 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 34482a6a09..1d8d1786fd 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -516,53 +516,85 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n-void emit_line(struct diff_options *o,\n-\t const char *set, const char *reset,\n-\t int add_line_prefix, int markup_ws,\n-\t int sign, const char *line, int len)\n+static void emit_buffered_patch_line(struct diff_options *o,\n+\t\t\t\t struct buffered_patch_line *e)\n {\n \tconst char *ws;\n \tint has_trailing_newline, has_trailing_carriage_return;\n+\tint len = e->len;\n \tFILE *file = o->file;\n \n-\tif (add_line_prefix)\n+\tif (e->add_line_prefix)\n \t\tfputs(diff_line_prefix(o), file);\n \n-\tif (markup_ws) {\n+\tswitch (e->state) {\n+\tcase BPL_EMIT_LINE_WS:\n \t\tws = diff_get_color(o->use_color, DIFF_WHITESPACE);\n+\t\tif (e->set)\n+\t\t\tfputs(e->set, file);\n+\t\tif (e->sign)\n+\t\t\tfputc(e->sign, file);\n+\t\tif (e->reset)\n+\t\t\tfputs(e->reset, file);\n+\t\tws_check_emit(e->line, e->len, o->ws_rule,\n+\t\t\t file, e->set, e->reset, ws);\n+\t\treturn;\n+\tcase BPL_EMIT_LINE_ASIS:\n+\t\thas_trailing_newline = (len > 0 && e->line[len-1] == '\\n');\n+\t\tif (has_trailing_newline)\n+\t\t\tlen--;\n+\t\thas_trailing_carriage_return = (len > 0 && e->line[len-1] == '\\r');\n+\t\tif (has_trailing_carriage_return)\n+\t\t\tlen--;\n \n-\t\tif (set)\n-\t\t\tfputs(set, file);\n-\t\tif (sign)\n-\t\t\tfputc(sign, file);\n-\t\tif (reset)\n-\t\t\tfputs(reset, file);\n-\t\tws = diff_get_color(o->use_color, DIFF_WHITESPACE);\n-\t\tws_check_emit(line, len, o->ws_rule,\n-\t\t\t file, set, reset, ws);\n+\t\tif (len || e->sign) {\n+\t\t\tif (e->set)\n+\t\t\t\tfputs(e->set, file);\n+\t\t\tif (e->sign)\n+\t\t\t\tfputc(e->sign, file);\n+\t\t\tfwrite(e->line, len, 1, file);\n+\t\t\tif (e->reset)\n+\t\t\t\tfputs(e->reset, file);\n+\t\t}\n+\t\tif (has_trailing_carriage_return)\n+\t\t\tfputc('\\r', file);\n+\t\tif (has_trailing_newline)\n+\t\t\tfputc('\\n', file);\n+\t\treturn;\n+\tcase BPL_HANDOVER:\n+\t\to->ws_rule = whitespace_rule(e->line); /*read from file, stored in line?*/\n \t\treturn;\n+\tdefault:\n+\t\tdie(\"BUG: malformatted buffered patch line: '%d'\", e->state);\n \t}\n+}\n \n-\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n-\tif (has_trailing_newline)\n-\t\tlen--;\n-\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n-\tif (has_trailing_carriage_return)\n-\t\tlen--;\n+static void append_buffered_patch_line(struct diff_options *o,\n+\t\t\t\t struct buffered_patch_line *e)\n+{\n+\tstruct buffered_patch_line *f;\n+\tALLOC_GROW(o->line_buffer,\n+\t\t o->line_buffer_nr + 1,\n+\t\t o->line_buffer_alloc);\n+\tf = &o->line_buffer[o->line_buffer_nr++];\n \n-\tif (len || sign) {\n-\t\tif (set)\n-\t\t\tfputs(set, file);\n-\t\tif (sign)\n-\t\t\tfputc(sign, file);\n-\t\tfwrite(line, len, 1, file);\n-\t\tif (reset)\n-\t\t\tfputs(reset, file);\n-\t}\n-\tif (has_trailing_carriage_return)\n-\t\tfputc('\\r', file);\n-\tif (has_trailing_newline)\n-\t\tfputc('\\n', file);\n+\tmemcpy(f, e, sizeof(struct buffered_patch_line));\n+\tf->line = e->line ? xmemdupz(e->line, e->len) : NULL;\n+}\n+\n+void emit_line(struct diff_options *o,\n+\t const char *set, const char *reset,\n+\t int add_line_prefix, int markup_ws,\n+\t int sign, const char *line, int len)\n+{\n+\tstruct buffered_patch_line e = {set, reset, line,\n+\t\tlen, sign, add_line_prefix,\n+\t\tmarkup_ws ? BPL_EMIT_LINE_WS : BPL_EMIT_LINE_ASIS};\n+\n+\tif (o->use_buffer)\n+\t\tappend_buffered_patch_line(o, &e);\n+\telse\n+\t\temit_buffered_patch_line(o, &e);\n }\n \n void emit_line_fmt(struct diff_options *o,\n@@ -1167,6 +1199,18 @@ static void diff_words_flush(struct emit_callback *ecbdata)\n \tif (ecbdata->diff_words->minus.text.size ||\n \t ecbdata->diff_words->plus.text.size)\n \t\tdiff_words_show(ecbdata->diff_words);\n+\n+\tif (ecbdata->diff_words->opt->line_buffer_nr) {\n+\t\tint i;\n+\t\tfor (i = 0; i < ecbdata->diff_words->opt->line_buffer_nr; i++)\n+\t\t\tappend_buffered_patch_line(ecbdata->opt,\n+\t\t\t\t&ecbdata->diff_words->opt->line_buffer[i]);\n+\n+\t\tfor (i = 0; i < ecbdata->diff_words->opt->line_buffer_nr; i++)\n+\t\t\tfree((void*) ecbdata->diff_words->opt->line_buffer[i].line);\n+\n+\t\tecbdata->diff_words->opt->line_buffer_nr = 0;\n+\t}\n }\n \n static void diff_filespec_load_driver(struct diff_filespec *one)\n@@ -1202,6 +1246,11 @@ static void init_diff_words_data(struct emit_callback *ecbdata,\n \t\txcalloc(1, sizeof(struct diff_words_data));\n \tecbdata->diff_words->type = o->word_diff;\n \tecbdata->diff_words->opt = o;\n+\n+\to->line_buffer = NULL;\n+\to->line_buffer_nr = 0;\n+\to->line_buffer_alloc = 0;\n+\n \tif (!o->word_regex)\n \t\to->word_regex = userdiff_word_regex(one);\n \tif (!o->word_regex)\n@@ -1236,6 +1285,7 @@ static void free_diff_words_data(struct emit_callback *ecbdata)\n {\n \tif (ecbdata->diff_words) {\n \t\tdiff_words_flush(ecbdata);\n+\t\tfree (ecbdata->diff_words->opt->line_buffer);\n \t\tfree (ecbdata->diff_words->opt);\n \t\tfree (ecbdata->diff_words->minus.text.ptr);\n \t\tfree (ecbdata->diff_words->minus.orig);\n@@ -2574,6 +2624,13 @@ static void builtin_diff(const char *name_a,\n \t\t\txecfg.ctxlen = strtoul(v, NULL, 10);\n \t\tif (o->word_diff)\n \t\t\tinit_diff_words_data(&ecbdata, o, one, two);\n+\t\tif (o->use_buffer) {\n+\t\t\tstruct buffered_patch_line e = BUFFERED_PATCH_LINE_INIT;\n+\t\t\te.state = BPL_HANDOVER;\n+\t\t\te.line = name_b;\n+\t\t\te.len = strlen(name_b);\n+\t\t\tappend_buffered_patch_line(o, &e);\n+\t\t}\n \t\tif (xdi_diff_outf(&mf1, &mf2, fn_out_consume, &ecbdata,\n \t\t\t\t &xpp, &xecfg))\n \t\t\tdie(\"unable to generate diff for %s\", one->path);\n@@ -3453,6 +3510,10 @@ void diff_setup(struct diff_options *options)\n \t\toptions->a_prefix = \"a/\";\n \t\toptions->b_prefix = \"b/\";\n \t}\n+\n+\toptions->line_buffer = NULL;\n+\toptions->line_buffer_nr = 0;\n+\toptions->line_buffer_alloc = 0;\n }\n \n void diff_setup_done(struct diff_options *options)\n@@ -4791,11 +4852,36 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n {\n \tint i;\n \tstruct diff_queue_struct *q = &diff_queued_diff;\n+\t/*\n+\t * For testing purposes we want to make sure the diff machinery\n+\t * works completely with the buffer. If there is anything emitted\n+\t * outside the emit_buffered_patch_line, then the order is screwed\n+\t * up and the tests will fail.\n+\t *\n+\t * TODO (later in this series):\n+\t * We'll unset this flag in a later patch.\n+\t */\n+\to->use_buffer = 1;\n+\n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n \t\tif (check_pair_status(p))\n \t\t\tdiff_flush_patch(p, o);\n \t}\n+\n+\tif (o->use_buffer) {\n+\t\tfor (i = 0; i < o->line_buffer_nr; i++)\n+\t\t\temit_buffered_patch_line(o, &o->line_buffer[i]);\n+\n+\t\tfor (i = 0; i < o->line_buffer_nr; i++)\n+\t\t\tfree((void*)o->line_buffer[i].line);\n+\n+\t\tfree(o->line_buffer);\n+\n+\t\to->line_buffer = NULL;\n+\t\to->line_buffer_nr = 0;\n+\t\to->line_buffer_alloc = 0;\n+\t}\n }\n \n void diff_flush(struct diff_options *options)\ndiff --git a/diff.h b/diff.h\nindex 267acf1980..64bc9bcd8c 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -115,6 +115,41 @@ enum diff_submodule_format {\n \tDIFF_SUBMODULE_INLINE_DIFF\n };\n \n+/*\n+ * This struct is used when we need to buffer the output of the diff output.\n+ *\n+ * NEEDSWORK: Instead of storing a copy of the line, add an offset pointer\n+ * into the pre/post image file. This pointer could be a union with the\n+ * line pointer. By storing an offset into the file instead of the literal line,\n+ * we can decrease the memory footprint for the buffered output. At first we\n+ * may want to only have indirection for the content lines, but we could\n+ * also have an enum (based on sign?) that stores prefabricated lines, e.g.\n+ * the similarity score line or hunk/file headers.\n+ */\n+struct buffered_patch_line {\n+\tconst char *set;\n+\tconst char *reset;\n+\tconst char *line;\n+\tint len;\n+\tint sign;\n+\tint add_line_prefix;\n+\tenum {\n+\t\t/*\n+\t\t * Emits [lineprefix][set][sign][reset] and then calls\n+\t\t * ws_check_emit which will output \"line\", marked up\n+\t\t * according to ws_rule.\n+\t\t */\n+\t\tBPL_EMIT_LINE_WS,\n+\n+\t\t/* Emits [lineprefix][set][sign] line [reset] */\n+\t\tBPL_EMIT_LINE_ASIS,\n+\n+\t\t/* Reloads the ws_rule; line contains the file name */\n+\t\tBPL_HANDOVER\n+\t} state;\n+};\n+#define BUFFERED_PATCH_LINE_INIT {NULL, NULL, NULL, 0, 0, 0}\n+\n struct diff_options {\n \tconst char *orderfile;\n \tconst char *pickaxe;\n@@ -188,6 +223,10 @@ struct diff_options {\n \tint diff_path_counter;\n \n \tunsigned ws_rule;\n+\tint use_buffer;\n+\n+\tstruct buffered_patch_line *line_buffer;\n+\tint line_buffer_nr, line_buffer_alloc;\n };\n \n void emit_line_fmt(struct diff_options *o, const char *set, const char *reset,\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320009","messageId":"20170517025857.32320-14-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 13/20] diff.c: convert emit_binary_diff_body to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:50Z","receivedAt":"2017-05-17T03:00:00Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers emit_binary_diff_body.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 39 ++++++++++++++++++++++-----------------\n 1 file changed, 22 insertions(+), 17 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 7c8d6a5d12..2dd10fa16a 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -2240,8 +2240,8 @@ static unsigned char *deflate_it(char *data,\n \treturn deflated;\n }\n \n-static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n-\t\t\t\t const char *prefix)\n+static void emit_binary_diff_body(struct diff_options *o,\n+\t\t\t\t mmfile_t *one, mmfile_t *two)\n {\n \tvoid *cp;\n \tvoid *delta;\n@@ -2270,13 +2270,12 @@ static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n \t}\n \n \tif (delta && delta_size < deflate_size) {\n-\t\tfprintf(file, \"%sdelta %lu\\n\", prefix, orig_size);\n+\t\temit_line_fmt(o, NULL, NULL, 1, \"delta %lu\\n\", orig_size);\n \t\tfree(deflated);\n \t\tdata = delta;\n \t\tdata_size = delta_size;\n-\t}\n-\telse {\n-\t\tfprintf(file, \"%sliteral %lu\\n\", prefix, two->size);\n+\t} else {\n+\t\temit_line_fmt(o, NULL, NULL, 1, \"literal %lu\\n\", two->size);\n \t\tfree(delta);\n \t\tdata = deflated;\n \t\tdata_size = deflate_size;\n@@ -2285,8 +2284,9 @@ static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n \t/* emit data encoded in base85 */\n \tcp = data;\n \twhile (data_size) {\n+\t\tint len;\n \t\tint bytes = (52 < data_size) ? 52 : data_size;\n-\t\tchar line[70];\n+\t\tchar line[71];\n \t\tdata_size -= bytes;\n \t\tif (bytes <= 26)\n \t\t\tline[0] = bytes + 'A' - 1;\n@@ -2294,20 +2294,25 @@ static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n \t\t\tline[0] = bytes - 26 + 'a' - 1;\n \t\tencode_85(line + 1, cp, bytes);\n \t\tcp = (char *) cp + bytes;\n-\t\tfprintf(file, \"%s\", prefix);\n-\t\tfputs(line, file);\n-\t\tfputc('\\n', file);\n+\n+\t\tlen = strlen(line);\n+\t\tline[len++] = '\\n';\n+\t\tline[len] = '\\0';\n+\n+\t\temit_line(o, NULL, NULL, 1, 0, line, len);\n \t}\n-\tfprintf(file, \"%s\\n\", prefix);\n+\temit_line(o, NULL, NULL, 1, 0, \"\\n\", 1);\n \tfree(data);\n }\n \n-static void emit_binary_diff(FILE *file, mmfile_t *one, mmfile_t *two,\n-\t\t\t const char *prefix)\n+static void emit_binary_diff(struct diff_options *o,\n+\t\t\t mmfile_t *one, mmfile_t *two)\n {\n-\tfprintf(file, \"%sGIT binary patch\\n\", prefix);\n-\temit_binary_diff_body(file, one, two, prefix);\n-\temit_binary_diff_body(file, two, one, prefix);\n+\tconst char *s = \"GIT binary patch\\n\";\n+\tconst int len = strlen(s);\n+\temit_line(o, NULL, NULL, 1, 0, s, len);\n+\temit_binary_diff_body(o, one, two);\n+\temit_binary_diff_body(o, two, one);\n }\n \n int diff_filespec_is_binary(struct diff_filespec *one)\n@@ -2494,7 +2499,7 @@ static void builtin_diff(const char *name_a,\n \t\t\t header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tif (DIFF_OPT_TST(o, BINARY))\n-\t\t\temit_binary_diff(o->file, &mf1, &mf2, line_prefix);\n+\t\t\temit_binary_diff(o, &mf1, &mf2);\n \t\telse\n \t\t\temit_line_fmt(o, NULL, NULL, 1,\n \t\t\t\t \"Binary files %s and %s differ\\n\",\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320010","messageId":"20170517025857.32320-16-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 15/20] diff.c: convert word diffing to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:52Z","receivedAt":"2017-05-17T03:00:04Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers all code related to diffing words.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 73 +++++++++++++++++++++++++++++-------------------------------------\n 1 file changed, 32 insertions(+), 41 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex ccd28953d7..f1cb0b7799 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -893,37 +893,42 @@ struct diff_words_data {\n \tstruct diff_words_style *style;\n };\n \n-static int fn_out_diff_words_write_helper(FILE *fp,\n+static int fn_out_diff_words_write_helper(struct diff_options *o,\n \t\t\t\t\t struct diff_words_style_elem *st_el,\n \t\t\t\t\t const char *newline,\n-\t\t\t\t\t size_t count, const char *buf,\n-\t\t\t\t\t const char *line_prefix)\n+\t\t\t\t\t size_t count, const char *buf)\n {\n \tint print = 0;\n+\tstruct strbuf sb = STRBUF_INIT;\n \n \twhile (count) {\n \t\tchar *p = memchr(buf, '\\n', count);\n \t\tif (print)\n-\t\t\tfputs(line_prefix, fp);\n+\t\t\temit_line(o, NULL, NULL, 1, 0, \"\", 0);\n+\n \t\tif (p != buf) {\n-\t\t\tif (st_el->color && fputs(st_el->color, fp) < 0)\n-\t\t\t\treturn -1;\n-\t\t\tif (fputs(st_el->prefix, fp) < 0 ||\n-\t\t\t fwrite(buf, p ? p - buf : count, 1, fp) != 1 ||\n-\t\t\t fputs(st_el->suffix, fp) < 0)\n-\t\t\t\treturn -1;\n-\t\t\tif (st_el->color && *st_el->color\n-\t\t\t && fputs(GIT_COLOR_RESET, fp) < 0)\n-\t\t\t\treturn -1;\n+\t\t\tconst char *reset = st_el->color && *st_el->color ?\n+\t\t\t\t\t GIT_COLOR_RESET : NULL;\n+\t\t\tstrbuf_addstr(&sb, st_el->prefix);\n+\t\t\tstrbuf_add(&sb, buf, p ? p - buf : count);\n+\t\t\tstrbuf_addstr(&sb, st_el->suffix);\n+\t\t\temit_line(o, st_el->color, reset,\n+\t\t\t\t 0, 0, sb.buf, sb.len);\n+\t\t\tstrbuf_reset(&sb);\n \t\t}\n \t\tif (!p)\n-\t\t\treturn 0;\n-\t\tif (fputs(newline, fp) < 0)\n-\t\t\treturn -1;\n+\t\t\tgoto out;\n+\n+\t\tstrbuf_addstr(&sb, newline);\n+\t\temit_line(o, NULL, NULL, 0, 0, sb.buf, sb.len);\n+\t\tstrbuf_reset(&sb);\n \t\tcount -= p + 1 - buf;\n \t\tbuf = p + 1;\n \t\tprint = 1;\n \t}\n+\n+out:\n+\tstrbuf_release(&sb);\n \treturn 0;\n }\n \n@@ -977,14 +982,12 @@ static void fn_out_diff_words_aux(void *priv, char *line, unsigned long len)\n \tint minus_first, minus_len, plus_first, plus_len;\n \tconst char *minus_begin, *minus_end, *plus_begin, *plus_end;\n \tstruct diff_options *opt = diff_words->opt;\n-\tconst char *line_prefix;\n \n \tif (line[0] != '@' || parse_hunk_header(line, len,\n \t\t\t&minus_first, &minus_len, &plus_first, &plus_len))\n \t\treturn;\n \n \tassert(opt);\n-\tline_prefix = diff_line_prefix(opt);\n \n \t/* POSIX requires that first be decremented by one if len == 0... */\n \tif (minus_len) {\n@@ -1001,28 +1004,21 @@ static void fn_out_diff_words_aux(void *priv, char *line, unsigned long len)\n \t} else\n \t\tplus_begin = plus_end = diff_words->plus.orig[plus_first].end;\n \n-\tif (color_words_output_graph_prefix(diff_words)) {\n-\t\tfputs(line_prefix, diff_words->opt->file);\n-\t}\n \tif (diff_words->current_plus != plus_begin) {\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t\t&style->ctx, style->newline,\n \t\t\t\tplus_begin - diff_words->current_plus,\n-\t\t\t\tdiff_words->current_plus, line_prefix);\n-\t\tif (*(plus_begin - 1) == '\\n')\n-\t\t\tfputs(line_prefix, diff_words->opt->file);\n+\t\t\t\tdiff_words->current_plus);\n \t}\n \tif (minus_begin != minus_end) {\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t\t&style->old, style->newline,\n-\t\t\t\tminus_end - minus_begin, minus_begin,\n-\t\t\t\tline_prefix);\n+\t\t\t\tminus_end - minus_begin, minus_begin);\n \t}\n \tif (plus_begin != plus_end) {\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t\t&style->new, style->newline,\n-\t\t\t\tplus_end - plus_begin, plus_begin,\n-\t\t\t\tline_prefix);\n+\t\t\t\tplus_end - plus_begin, plus_begin);\n \t}\n \n \tdiff_words->current_plus = plus_end;\n@@ -1109,18 +1105,14 @@ static void diff_words_show(struct diff_words_data *diff_words)\n \tstruct diff_words_style *style = diff_words->style;\n \n \tstruct diff_options *opt = diff_words->opt;\n-\tconst char *line_prefix;\n-\n \tassert(opt);\n-\tline_prefix = diff_line_prefix(opt);\n \n \t/* special case: only removal */\n \tif (!diff_words->plus.text.size) {\n-\t\tfputs(line_prefix, diff_words->opt->file);\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t&style->old, style->newline,\n \t\t\tdiff_words->minus.text.size,\n-\t\t\tdiff_words->minus.text.ptr, line_prefix);\n+\t\t\tdiff_words->minus.text.ptr);\n \t\tdiff_words->minus.text.size = 0;\n \t\treturn;\n \t}\n@@ -1143,12 +1135,11 @@ static void diff_words_show(struct diff_words_data *diff_words)\n \tif (diff_words->current_plus != diff_words->plus.text.ptr +\n \t\t\tdiff_words->plus.text.size) {\n \t\tif (color_words_output_graph_prefix(diff_words))\n-\t\t\tfputs(line_prefix, diff_words->opt->file);\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\t\temit_line(diff_words->opt, NULL, NULL, 1, 0, \"\", 0);\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t&style->ctx, style->newline,\n \t\t\tdiff_words->plus.text.ptr + diff_words->plus.text.size\n-\t\t\t- diff_words->current_plus, diff_words->current_plus,\n-\t\t\tline_prefix);\n+\t\t\t- diff_words->current_plus, diff_words->current_plus);\n \t}\n \tdiff_words->minus.text.size = diff_words->plus.text.size = 0;\n }\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320011","messageId":"20170517025857.32320-10-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 09/20] diff.c: convert builtin_diff to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:46Z","receivedAt":"2017-05-17T03:00:05Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers builtin_diff.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 33 ++++++++++++++++++++-------------\n 1 file changed, 20 insertions(+), 13 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 8186289734..4fa976d43c 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1289,8 +1289,9 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \to->found_changes = 1;\n \n \tif (ecbdata->header) {\n-\t\tfprintf(o->file, \"%s\", ecbdata->header->buf);\n-\t\tstrbuf_reset(ecbdata->header);\n+\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t ecbdata->header->buf, ecbdata->header->len);\n+\t\tstrbuf_release(ecbdata->header);\n \t\tecbdata->header = NULL;\n \t}\n \n@@ -2435,7 +2436,7 @@ static void builtin_diff(const char *name_a,\n \t\tif (complete_rewrite &&\n \t\t (textconv_one || !diff_filespec_is_binary(one)) &&\n \t\t (textconv_two || !diff_filespec_is_binary(two))) {\n-\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t\temit_rewrite_diff(name_a, name_b, one, two,\n \t\t\t\t\t\ttextconv_one, textconv_two, o);\n@@ -2445,7 +2446,7 @@ static void builtin_diff(const char *name_a,\n \t}\n \n \tif (o->irreversible_delete && lbl[1][0] == '/') {\n-\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tgoto free_ab_and_return;\n \t} else if (!DIFF_OPT_TST(o, TEXT) &&\n@@ -2456,12 +2457,15 @@ static void builtin_diff(const char *name_a,\n \t\t !DIFF_OPT_TST(o, BINARY)) {\n \t\t\tif (!oidcmp(&one->oid, &two->oid)) {\n \t\t\t\tif (must_show_header)\n-\t\t\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\t\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t\t\t\t header.buf, header.len);\n \t\t\t\tgoto free_ab_and_return;\n \t\t\t}\n-\t\t\tfprintf(o->file, \"%s\", header.buf);\n-\t\t\tfprintf(o->file, \"%sBinary files %s and %s differ\\n\",\n-\t\t\t\tline_prefix, lbl[0], lbl[1]);\n+\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t\t header.buf, header.len);\n+\t\t\temit_line_fmt(o, NULL, NULL, 1,\n+\t\t\t\t \"Binary files %s and %s differ\\n\",\n+\t\t\t\t lbl[0], lbl[1]);\n \t\t\tgoto free_ab_and_return;\n \t\t}\n \t\tif (fill_mmfile(&mf1, one) < 0 || fill_mmfile(&mf2, two) < 0)\n@@ -2470,16 +2474,19 @@ static void builtin_diff(const char *name_a,\n \t\tif (mf1.size == mf2.size &&\n \t\t !memcmp(mf1.ptr, mf2.ptr, mf1.size)) {\n \t\t\tif (must_show_header)\n-\t\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t\t\t header.buf, header.len);\n \t\t\tgoto free_ab_and_return;\n \t\t}\n-\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tif (DIFF_OPT_TST(o, BINARY))\n \t\t\temit_binary_diff(o->file, &mf1, &mf2, line_prefix);\n \t\telse\n-\t\t\tfprintf(o->file, \"%sBinary files %s and %s differ\\n\",\n-\t\t\t\tline_prefix, lbl[0], lbl[1]);\n+\t\t\temit_line_fmt(o, NULL, NULL, 1,\n+\t\t\t\t \"Binary files %s and %s differ\\n\",\n+\t\t\t\t lbl[0], lbl[1]);\n \t\to->found_changes = 1;\n \t} else {\n \t\t/* Crazy xdl interfaces.. */\n@@ -2491,7 +2498,7 @@ static void builtin_diff(const char *name_a,\n \t\tconst struct userdiff_funcname *pe;\n \n \t\tif (must_show_header) {\n-\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t}\n \n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320012","messageId":"20170517025857.32320-11-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv2 10/20] diff.c: convert emit_rewrite_diff to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T02:58:47Z","receivedAt":"2017-05-17T03:00:10Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers emit_rewrite_diff.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 33 ++++++++++++++++++---------------\n 1 file changed, 18 insertions(+), 15 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 4fa976d43c..3dda9f3c8e 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -704,17 +704,17 @@ static void remove_tempfile(void)\n \t}\n }\n \n-static void print_line_count(FILE *file, int count)\n+static void add_line_count(struct strbuf *out, int count)\n {\n \tswitch (count) {\n \tcase 0:\n-\t\tfprintf(file, \"0,0\");\n+\t\tstrbuf_addstr(out, \"0,0\");\n \t\tbreak;\n \tcase 1:\n-\t\tfprintf(file, \"1\");\n+\t\tstrbuf_addstr(out, \"1\");\n \t\tbreak;\n \tdefault:\n-\t\tfprintf(file, \"1,%d\", count);\n+\t\tstrbuf_addf(out, \"1,%d\", count);\n \t\tbreak;\n \t}\n }\n@@ -768,7 +768,7 @@ static void emit_rewrite_diff(const char *name_a,\n \tchar *data_one, *data_two;\n \tsize_t size_one, size_two;\n \tstruct emit_callback ecbdata;\n-\tconst char *line_prefix = diff_line_prefix(o);\n+\tstruct strbuf out = STRBUF_INIT;\n \n \tif (diff_mnemonic_prefix && DIFF_OPT_TST(o, REVERSE_DIFF)) {\n \t\ta_prefix = o->b_prefix;\n@@ -806,20 +806,23 @@ static void emit_rewrite_diff(const char *name_a,\n \tecbdata.lno_in_preimage = 1;\n \tecbdata.lno_in_postimage = 1;\n \n+\temit_line_fmt(o, metainfo, reset, 1, \"--- %s%s\\n\", a_name.buf, name_a_tab);\n+\temit_line_fmt(o, metainfo, reset, 1, \"+++ %s%s\\n\", b_name.buf, name_b_tab);\n+\n \tlc_a = count_lines(data_one, size_one);\n \tlc_b = count_lines(data_two, size_two);\n-\tfprintf(o->file,\n-\t\t\"%s%s--- %s%s%s\\n%s%s+++ %s%s%s\\n%s%s@@ -\",\n-\t\tline_prefix, metainfo, a_name.buf, name_a_tab, reset,\n-\t\tline_prefix, metainfo, b_name.buf, name_b_tab, reset,\n-\t\tline_prefix, fraginfo);\n+\n+\tstrbuf_addstr(&out, \"@@ -\");\n \tif (!o->irreversible_delete)\n-\t\tprint_line_count(o->file, lc_a);\n+\t\tadd_line_count(&out, lc_a);\n \telse\n-\t\tfprintf(o->file, \"?,?\");\n-\tfprintf(o->file, \" +\");\n-\tprint_line_count(o->file, lc_b);\n-\tfprintf(o->file, \" @@%s\\n\", reset);\n+\t\tstrbuf_addstr(&out, \"?,?\");\n+\tstrbuf_addstr(&out, \" +\");\n+\tadd_line_count(&out, lc_b);\n+\tstrbuf_addstr(&out, \" @@\\n\");\n+\temit_line(o, fraginfo, reset, 1, 0, out.buf, out.len);\n+\tstrbuf_release(&out);\n+\n \tif (lc_a && !o->irreversible_delete)\n \t\temit_rewrite_lines(&ecbdata, '-', data_one, size_one);\n \tif (lc_b)\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320020","messageId":"xmqqefvoxeg7.fsf@gitster.mtv.corp.google.com","threadId":"45958","inReplyTo":"20170517025857.32320-12-sbeller@google.com","subject":"Re: [PATCHv2 11/20] diff.c: convert emit_rewrite_lines to use emit_line_*","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-17T05:03:20Z","receivedAt":"2017-05-17T05:03:27Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> In a later patch, I want to propose an option to detect&color\n> moved lines in a diff, which cannot be done in a one-pass over\n> the diff. Instead we need to go over the whole diff twice,\n> because we cannot detect the first line of the two corresponding\n> lines (+ and -) that got moved.\n>\n> So to prepare the diff machinery for two pass algorithms\n> (i.e. buffer it all up and then operate on the result),\n> move all emissions to places, such that the only emitting\n> function is emit_line_0.\n>\n> This covers emit_rewrite_lines.\n>\n> Signed-off-by: Stefan Beller \n> ---\n> diff.c | 27 +++++++++++++++++++--------\n> 1 file changed, 19 insertions(+), 8 deletions(-)\n>\n> diff --git a/diff.c b/diff.c\n> index 3dda9f3c8e..690794aeb8 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -722,15 +722,25 @@ static void add_line_count(struct strbuf *out, int count)\n> static void emit_rewrite_lines(struct emit_callback *ecb,\n> \t\t\t int prefix, const char *data, int size)\n> {\n> -\tconst char *endp = NULL;\n> -\tstatic const char *nneof = \" No newline at end of file\\n\";\n> \tconst char *reset = diff_get_color(ecb->color_diff, DIFF_RESET);\n> +\tstruct strbuf sb = STRBUF_INIT;\n> \n> \twhile (0 < size) {\n> \t\tint len;\n> \n> -\t\tendp = memchr(data, '\\n', size);\n> -\t\tlen = endp ? (endp - data + 1) : size;\n> +\t\tconst char *endp = memchr(data, '\\n', size);\n> +\t\tif (endp)\n> +\t\t\tlen = endp - data + 1;\n> +\t\telse {\n> +\t\t\twhile (0 < size) {\n> +\t\t\t\tstrbuf_addch(&sb, *data);\n> +\t\t\t\tsize -= len;\n> +\t\t\t\tdata += len;\n> +\t\t\t}\n> +\t\t\tstrbuf_addch(&sb, '\\n');\n> +\t\t\tdata = sb.buf;\n> +\t\t\tlen = sb.len;\n> +\t\t}\n> \t\tif (prefix != '+') {\n> \t\t\tecb->lno_in_preimage++;\n> \t\t\temit_del_line(reset, ecb, data, len);\n> @@ -741,12 +751,13 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n> \t\tsize -= len;\n> \t\tdata += len;\n> \t}\n> -\tif (!endp) {\n> +\tif (sb.len) {\n> +\t\tstatic const char *nneof = \"\\\\ No newline at end of file\\n\";\n> \t\tconst char *context = diff_get_color(ecb->color_diff,\n> \t\t\t\t\t\t DIFF_CONTEXT);\n> -\t\tputc('\\n', ecb->opt->file);\n> -\t\temit_line(ecb->opt, context, reset, 1, '\\\\',\n> -\t\t\t nneof, strlen(nneof));\n> +\t\temit_line(ecb->opt, context, reset, 1, 0,\n> +\t\t\t nneof, strlen(nneof));\n> +\t\tstrbuf_release(&sb);\n\nThe reason why we can lose the LF immediately after the incomplete\nline we found in the above loop is because the updated emit_line_0()\nadds LF if its input is an incomplete line? Even before this series\nstarted, emit_line_0() was already prepared to see a complete or\nincomplete line and emit the \"reset\" color after the optional EOL\nbytes at the end, so emit_line() and emit_{add,del}_line() calls\nthroughout the code can pass the body of the line with or without\nthe EOL and right things will happen. Sounds about right.\n\n\n\n\n> \t}\n> }\n"},{"id":"320021","messageId":"xmqq8tlwxdp3.fsf@gitster.mtv.corp.google.com","threadId":"45958","inReplyTo":"20170517025857.32320-13-sbeller@google.com","subject":"Re: [PATCHv2 12/20] submodule.c: convert show_submodule_summary to use emit_line_fmt","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-17T05:19:36Z","receivedAt":"2017-05-17T05:19:43Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> In a later patch, I want to propose an option to detect&color\n> moved lines in a diff, which cannot be done in a one-pass over\n> the diff. Instead we need to go over the whole diff twice,\n> because we cannot detect the first line of the two corresponding\n> lines (+ and -) that got moved.\n>\n> So to prepare the diff machinery for two pass algorithms\n> (i.e. buffer it all up and then operate on the result),\n> move all emissions to places, such that the only emitting\n> function is emit_line_0.\n>\n> This prepares the code for submodules to go through the\n> emit_line function.\n>\n> Signed-off-by: Stefan Beller \n> ---\n> diff.c | 20 +++++++---------\n> diff.h | 5 ++++\n> submodule.c | 78 ++++++++++++++++++++++++++++++-------------------------------\n> submodule.h | 9 +++----\n> 4 files changed, 56 insertions(+), 56 deletions(-)\n>\n> diff --git a/diff.c b/diff.c\n> index 690794aeb8..7c8d6a5d12 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -516,8 +516,8 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n> \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n> }\n> \n> -static void emit_line(struct diff_options *o, const char *set, const char *reset,\n> -\t\t int add_line_prefix, int sign, const char *line, int len)\n> +void emit_line(struct diff_options *o, const char *set, const char *reset,\n> +\t int add_line_prefix, int sign, const char *line, int len)\n> {\n> \tint has_trailing_newline, has_trailing_carriage_return;\n> \tFILE *file = o->file;\n> @@ -547,10 +547,10 @@ static void emit_line(struct diff_options *o, const char *set, const char *reset\n> \t\tfputc('\\n', file);\n> }\n> \n> -static void emit_line_fmt(struct diff_options *o,\n> -\t\t\t const char *set, const char *reset,\n> -\t\t\t int add_line_prefix,\n> -\t\t\t const char *fmt, ...)\n> +void emit_line_fmt(struct diff_options *o,\n> +\t\t const char *set, const char *reset,\n> +\t\t int add_line_prefix,\n> +\t\t const char *fmt, ...)\n\nInteresting...\n\n> -static void show_submodule_header(FILE *f, const char *path,\n> -\t\tconst char *line_prefix,\n> +static void show_submodule_header(struct diff_options *o, const char *path,\n> \t\tstruct object_id *one, struct object_id *two,\n> \t\tunsigned dirty_submodule, const char *meta,\n> \t\tconst char *reset,\n\nIs this ONLY called when the caller wants its output inserted to the\n\"diff\" (or \"log -p\") output? If so, I think it makes sense to pass\n'o', but if the function is oblivious that it is driven to produce\npart of a \"diff\", it feels wrong to pass 'o'. The original was\ntaking a \"FILE *\" and line_prefix, so it is rather clear that the\nanswer to the question is \"yes, this is very closely tied to diff\noutput\". Now you have access to 'o', so you do not need to pass\nthem separately. Good.\n\nEach line in its output, when incorporated in \"diff\" or \"log -p\"\noutput, must be prefixed with the line-prefix to accomodate users of\n\"log --graph\", so I guess it cannot be helped. Your calls to\nemit_line_fmt() below seems to ask the line-prefix to be added,\nwhich is good, too.\n\nHow does capturing these lines help moved line detection, by the\nway? These must never be matched with any other added or removed\nline in the real patch output.\n\n> @@ -426,11 +419,11 @@ static void show_submodule_header(FILE *f, const char *path,\n> \tint fast_forward = 0, fast_backward = 0;\n> \n> \tif (dirty_submodule & DIRTY_SUBMODULE_UNTRACKED)\n> -\t\tfprintf(f, \"%sSubmodule %s contains untracked content\\n\",\n> -\t\t\tline_prefix, path);\n> +\t\temit_line_fmt(o, NULL, NULL, 1,\n> +\t\t\t \"Submodule %s contains untracked content\\n\", path);\n> \tif (dirty_submodule & DIRTY_SUBMODULE_MODIFIED)\n> -\t\tfprintf(f, \"%sSubmodule %s contains modified content\\n\",\n> -\t\t\tline_prefix, path);\n> +\t\temit_line_fmt(o, NULL, NULL, 1,\n> +\t\t\t \"Submodule %s contains modified content\\n\", path);\n> \n> \tif (is_null_oid(one))\n> \t\tmessage = \"(new submodule)\";\n\nemit_line() and emit_line_fmt() are both inappropriate names for a\nglobal function. These are very closely tied to diff generation, so\nwe probably want to see some form of \"diff\" in their names. \n\nThe fact that it is clear because its first parameter is \"struct\ndiff_options\" is insufficient---\"you cannot tell what context the\nfunction is meant to be used by only looking at its name\" is\ncertainly solved by its function signature, but the other issue with\nan overly generic name is that other codepaths in different contexts\nmay want to use such a short and sweet name.\n\nThanks.\n"},{"id":"320104","messageId":"CAGZ79kbCYiDRbf2+uXANqBiR9sG4ir4m6VK5+t1LS-kE7T1aQg@mail.gmail.com","threadId":"45958","inReplyTo":"xmqq8tlwxdp3.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCHv2 12/20] submodule.c: convert show_submodule_summary to use emit_line_fmt","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T21:05:10Z","receivedAt":"2017-05-17T21:05:17Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, May 16, 2017 at 10:19 PM, Junio C Hamano wrote:\n> Stefan Beller writes:\n>\n>> In a later patch, I want to propose an option to detect&color\n>> moved lines in a diff, which cannot be done in a one-pass over\n>> the diff. Instead we need to go over the whole diff twice,\n>> because we cannot detect the first line of the two corresponding\n>> lines (+ and -) that got moved.\n>>\n>> So to prepare the diff machinery for two pass algorithms\n>> (i.e. buffer it all up and then operate on the result),\n>> move all emissions to places, such that the only emitting\n>> function is emit_line_0.\n>>\n>> This prepares the code for submodules to go through the\n>> emit_line function.\n>>\n>> Signed-off-by: Stefan Beller \n>> ---\n>> diff.c | 20 +++++++---------\n>> diff.h | 5 ++++\n>> submodule.c | 78 ++++++++++++++++++++++++++++++-------------------------------\n>> submodule.h | 9 +++----\n>> 4 files changed, 56 insertions(+), 56 deletions(-)\n>>\n>> diff --git a/diff.c b/diff.c\n>> index 690794aeb8..7c8d6a5d12 100644\n>> --- a/diff.c\n>> +++ b/diff.c\n>> @@ -516,8 +516,8 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n>> ecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n>> }\n>>\n>> -static void emit_line(struct diff_options *o, const char *set, const char *reset,\n>> - int add_line_prefix, int sign, const char *line, int len)\n>> +void emit_line(struct diff_options *o, const char *set, const char *reset,\n>> + int add_line_prefix, int sign, const char *line, int len)\n>> {\n>> int has_trailing_newline, has_trailing_carriage_return;\n>> FILE *file = o->file;\n>> @@ -547,10 +547,10 @@ static void emit_line(struct diff_options *o, const char *set, const char *reset\n>> fputc('\\n', file);\n>> }\n>>\n>> -static void emit_line_fmt(struct diff_options *o,\n>> - const char *set, const char *reset,\n>> - int add_line_prefix,\n>> - const char *fmt, ...)\n>> +void emit_line_fmt(struct diff_options *o,\n>> + const char *set, const char *reset,\n>> + int add_line_prefix,\n>> + const char *fmt, ...)\n>\n> Interesting...\n>\n>> -static void show_submodule_header(FILE *f, const char *path,\n>> - const char *line_prefix,\n>> +static void show_submodule_header(struct diff_options *o, const char *path,\n>> struct object_id *one, struct object_id *two,\n>> unsigned dirty_submodule, const char *meta,\n>> const char *reset,\n>\n> Is this ONLY called when the caller wants its output inserted to the\n> \"diff\" (or \"log -p\") output?\n\nYes.\n\n> If so, I think it makes sense to pass\n> 'o', but if the function is oblivious that it is driven to produce\n> part of a \"diff\", it feels wrong to pass 'o'. The original was\n> taking a \"FILE *\" and line_prefix, so it is rather clear that the\n> answer to the question is \"yes, this is very closely tied to diff\n> output\". Now you have access to 'o', so you do not need to pass\n> them separately. Good.\n\nok.\n\n> Each line in its output, when incorporated in \"diff\" or \"log -p\"\n> output, must be prefixed with the line-prefix to accomodate users of\n> \"log --graph\", so I guess it cannot be helped. Your calls to\n> emit_line_fmt() below seems to ask the line-prefix to be added,\n> which is good, too.\n>\n> How does capturing these lines help moved line detection, by the\n> way? These must never be matched with any other added or removed\n> line in the real patch output.\n\nWhy?\n\nActually I think it has some value if it can match across\n(submodule-)repository boundaries, e.g. think of Ævars RFC to put\nSHA1DC into a submodule. If reviewing that commit later on, a user\nmay be interested in \"what is the difference between what we carried so\nfar in this repo compared to what we point at now in the submodule\".\nMost of the code should be the same, but anchored at a different\npath/repo, so a move detection would be super helpful.\n\nI do understand that you may not want to see a move crossing\na repo boundary, but I would prefer that to a later patch, once we\nhave a better understanding on the use cases of this new feature.\n\n>> if (is_null_oid(one))\n>> message = \"(new submodule)\";\n>\n> emit_line() and emit_line_fmt() are both inappropriate names for a\n> global function. These are very closely tied to diff generation, so\n> we probably want to see some form of \"diff\" in their names.\n\nOh, uh. You're right.\n\nI would think inside of diff.c we'd still want to keep the short name,\nso maybe I'd expose a wrapper to the outside world.\n\nMaybe\n\n diff_emit_strbuf(diff_options *, strbuf *)\n\nwould be fine for all use cases from outside.\n\n\n> The fact that it is clear because its first parameter is \"struct\n> diff_options\" is insufficient---\"you cannot tell what context the\n> function is meant to be used by only looking at its name\" is\n> certainly solved by its function signature, but the other issue with\n> an overly generic name is that other codepaths in different contexts\n> may want to use such a short and sweet name.\n>\n> Thanks.\n\nI am bad at naming, so if you have a better idea for names,\nfeel free to mention them.\n\nThanks,\nStefan\n"},{"id":"320105","messageId":"CAGZ79kbrFacBvGsHo6A1o94sA=zB7g7q1z3hKGNi-msXnQVjVw@mail.gmail.com","threadId":"45958","inReplyTo":"xmqqefvoxeg7.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCHv2 11/20] diff.c: convert emit_rewrite_lines to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-17T21:16:47Z","receivedAt":"2017-05-17T21:16:59Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, May 16, 2017 at 10:03 PM, Junio C Hamano wrote:\n> Stefan Beller writes:\n\n> The reason why we can lose the LF immediately after the incomplete\n> line we found in the above loop is because the updated emit_line_0()\n> adds LF if its input is an incomplete line?\n\nNo. Because there are no incomplete lines any more, as we complete\nthe line above via strbuf_addch(&sb, '\\n');\n\nIn a buffered world, we need to think about what to buffer, and I think\nwe rather want to buffer all lines with the same line ending otherwise\nthe comparison function is harder. So in that case we'll rather\n\n line1=\"last line, but we added EOL\\n\"\n line2=\"\\\\ No newline at end of file\\n\"\n\nbecause line1 could occur somewhere else as is with the \\n.\n\n> Even before this series\n> started, emit_line_0() was already prepared to see a complete or\n> incomplete line and emit the \"reset\" color after the optional EOL\n> bytes at the end, so emit_line() and emit_{add,del}_line() calls\n> throughout the code can pass the body of the line with or without\n> the EOL and right things will happen. Sounds about right.\n\nYes the emit_line_0 will strip off \\n if there is and output it after a\npotential color reset.\n"},{"id":"320129","messageId":"xmqqinkyvobp.fsf@gitster.mtv.corp.google.com","threadId":"45958","inReplyTo":"CAGZ79kbCYiDRbf2+uXANqBiR9sG4ir4m6VK5+t1LS-kE7T1aQg@mail.gmail.com","subject":"Re: [PATCHv2 12/20] submodule.c: convert show_submodule_summary to use emit_line_fmt","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-18T03:25:14Z","receivedAt":"2017-05-18T03:25:40Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n>>> +static void show_submodule_header(struct diff_options *o, const char *path,\n>>> struct object_id *one, struct object_id *two,\n>>> unsigned dirty_submodule, const char *meta,\n>>> const char *reset,\n>> ...\n>> How does capturing these lines help moved line detection, by the\n>> way? These must never be matched with any other added or removed\n>> line in the real patch output.\n>\n> Why?\n\nWhat are buffered are not patch text, but informational text like\n\"Submodule X contains untracked content\", etc. When a text file is\nmodified elsewhere and lost a line that happened to say the same\ncontents, we do not want to consider that such a line was moved to\nwhere a submodule had an untracked file.\n\nI have a suspicion that the two-pass buffering is done at too high a\nlevel in this series. Doesn't the code (I haven't reached the end\nof the series) update emit_line() to buffer the patch text and these\nnon-patch text with all the coloring and resetting sequences?\nBecause the \"ah, this old line removed corresponds to that new line\nthat appears elsewhere?\" logic do not want to see these color/reset\nsequence, the buffering code needs to become quite specific to how\nthe current diff code is colored (e.g. a line must be painted in a\nsingle color and have reset at the end) and makes future change to\ncolor things differently almost impossible (e.g. imagine how you\nwould add a \"feature\" that paints certain words on added lines\ndifferently?).\n\nAhh, yes, I see NEEDSWORK comment in patch 19/20. \n\nYes, I agree that this code really should be working in terms of\noffsets into pre/post images when finding matching changes, which\nprobably should happen without letting fn_out_consume produce fully\ncolored textual diff output in the first pass. For the purpose of\n\"moved lines detection\", the logic to match a stretch of preimage\nlines with postimage lines do not want to bother with \"--- a/$path\"\nheaders, and it does not want to care if a line that begins with '+'\nneeds to be added by calling emit_add_line() that knows how to check\nws errors or the payload needs to be painted in green. After the\nfirst pass determines which added lines are not true addition but\nmerely moved, the second pass would know how that '+' line needs to\nbe painted a lot better (e.g. it may not be painted in green).\nLetting fn_out_consume() call emit_add_line() only to compute\ninformation (e.g. \"'+'? ok, green\") that the first pass does not\nwant to see and the second pass will compute better is probably not\na good longer term direction to go in.\n\nHaving said that, we need to start somewhere, and I think it is a\nreasonable first-cut attempt to work on top of the textual output\nlike this series does (IOW, while I do agree with the NEEDSWORK and\nthe way this series currently does things must be revamped in the\nlonger term, I do not think we should wait until that happens to\nstart playing with this topic).\n\nThanks.\n\n\n\n"},{"id":"320130","messageId":"xmqqefvmvnu5.fsf@gitster.mtv.corp.google.com","threadId":"45958","inReplyTo":"20170517025857.32320-12-sbeller@google.com","subject":"Re: [PATCHv2 11/20] diff.c: convert emit_rewrite_lines to use emit_line_*","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-18T03:35:46Z","receivedAt":"2017-05-18T03:35:52Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> In a later patch, I want to propose an option to detect&color\n> moved lines in a diff, which cannot be done in a one-pass over\n> the diff. Instead we need to go over the whole diff twice,\n> because we cannot detect the first line of the two corresponding\n> lines (+ and -) that got moved.\n>\n> So to prepare the diff machinery for two pass algorithms\n> (i.e. buffer it all up and then operate on the result),\n> move all emissions to places, such that the only emitting\n> function is emit_line_0.\n>\n> This covers emit_rewrite_lines.\n>\n> Signed-off-by: Stefan Beller \n> ---\n> diff.c | 27 +++++++++++++++++++--------\n> 1 file changed, 19 insertions(+), 8 deletions(-)\n>\n> diff --git a/diff.c b/diff.c\n> index 3dda9f3c8e..690794aeb8 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -722,15 +722,25 @@ static void add_line_count(struct strbuf *out, int count)\n> static void emit_rewrite_lines(struct emit_callback *ecb,\n> \t\t\t int prefix, const char *data, int size)\n> {\n> -\tconst char *endp = NULL;\n> -\tstatic const char *nneof = \" No newline at end of file\\n\";\n> \tconst char *reset = diff_get_color(ecb->color_diff, DIFF_RESET);\n> +\tstruct strbuf sb = STRBUF_INIT;\n> \n> \twhile (0 < size) {\n> \t\tint len;\n> \n> -\t\tendp = memchr(data, '\\n', size);\n> -\t\tlen = endp ? (endp - data + 1) : size;\n> +\t\tconst char *endp = memchr(data, '\\n', size);\n> +\t\tif (endp)\n> +\t\t\tlen = endp - data + 1;\n> +\t\telse {\n\nThis side does not initialize \"len\" at all, which means ...\n\n> +\t\t\twhile (0 < size) {\n> +\t\t\t\tstrbuf_addch(&sb, *data);\n> +\t\t\t\tsize -= len;\n> +\t\t\t\tdata += len;\n\n... we do random computation here.\n\n> +\t\t\t}\n> +\t\t\tstrbuf_addch(&sb, '\\n');\n> +\t\t\tdata = sb.buf;\n> +\t\t\tlen = sb.len;\n> +\t\t}\n> \t\tif (prefix != '+') {\n> \t\t\tecb->lno_in_preimage++;\n> \t\t\temit_del_line(reset, ecb, data, len);\n> @@ -741,12 +751,13 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n> \t\tsize -= len;\n> \t\tdata += len;\n> \t}\n> -\tif (!endp) {\n> +\tif (sb.len) {\n> +\t\tstatic const char *nneof = \"\\\\ No newline at end of file\\n\";\n> \t\tconst char *context = diff_get_color(ecb->color_diff,\n> \t\t\t\t\t\t DIFF_CONTEXT);\n> -\t\tputc('\\n', ecb->opt->file);\n> -\t\temit_line(ecb->opt, context, reset, 1, '\\\\',\n> -\t\t\t nneof, strlen(nneof));\n> +\t\temit_line(ecb->opt, context, reset, 1, 0,\n> +\t\t\t nneof, strlen(nneof));\n> +\t\tstrbuf_release(&sb);\n> \t}\n> }\n"},{"id":"320173","messageId":"CAGZ79kbXyjTaE=F7YU8yu8=eNzLS96sA5r5N4xrhmOfw+T4Lwg@mail.gmail.com","threadId":"45958","inReplyTo":"xmqqinkyvobp.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCHv2 12/20] submodule.c: convert show_submodule_summary to use emit_line_fmt","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T17:12:20Z","receivedAt":"2017-05-18T17:12:28Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, May 17, 2017 at 8:25 PM, Junio C Hamano wrote:\n> Stefan Beller writes:\n>\n>>>> +static void show_submodule_header(struct diff_options *o, const char *path,\n>>>> struct object_id *one, struct object_id *two,\n>>>> unsigned dirty_submodule, const char *meta,\n>>>> const char *reset,\n>>> ...\n>>> How does capturing these lines help moved line detection, by the\n>>> way? These must never be matched with any other added or removed\n>>> line in the real patch output.\n>>\n>> Why?\n>\n> What are buffered are not patch text, but informational text like\n> \"Submodule X contains untracked content\", etc. When a text file is\n> modified elsewhere and lost a line that happened to say the same\n> contents, we do not want to consider that such a line was moved to\n> where a submodule had an untracked file.\n>\n> I have a suspicion that the two-pass buffering is done at too high a\n> level in this series. Doesn't the code (I haven't reached the end\n> of the series) update emit_line() to buffer the patch text and these\n> non-patch text with all the coloring and resetting sequences?\n> Because the \"ah, this old line removed corresponds to that new line\n> that appears elsewhere?\" logic do not want to see these color/reset\n> sequence, the buffering code needs to become quite specific to how\n> the current diff code is colored (e.g. a line must be painted in a\n> single color and have reset at the end) and makes future change to\n> color things differently almost impossible (e.g. imagine how you\n> would add a \"feature\" that paints certain words on added lines\n> differently?).\n\nThat could be added in ws.c:ws_check_emit, as these certain words\nare similar to coloring whitespace.\n\nIt depends on the precedence of such a future feature, is the move\ndetection or the word highlighting more important to keep its color?\n\n> Ahh, yes, I see NEEDSWORK comment in patch 19/20.\n>\n> Yes, I agree that this code really should be working in terms of\n> offsets into pre/post images when finding matching changes, which\n> probably should happen without letting fn_out_consume produce fully\n> colored textual diff output in the first pass. For the purpose of\n> \"moved lines detection\", the logic to match a stretch of preimage\n> lines with postimage lines do not want to bother with \"--- a/$path\"\n> headers, and it does not want to care if a line that begins with '+'\n> needs to be added by calling emit_add_line() that knows how to check\n> ws errors or the payload needs to be painted in green. After the\n> first pass determines which added lines are not true addition but\n> merely moved, the second pass would know how that '+' line needs to\n> be painted a lot better (e.g. it may not be painted in green).\n> Letting fn_out_consume() call emit_add_line() only to compute\n> information (e.g. \"'+'? ok, green\") that the first pass does not\n> want to see and the second pass will compute better is probably not\n> a good longer term direction to go in.\n>\n> Having said that, we need to start somewhere, and I think it is a\n> reasonable first-cut attempt to work on top of the textual output\n> like this series does (IOW, while I do agree with the NEEDSWORK and\n> the way this series currently does things must be revamped in the\n> longer term, I do not think we should wait until that happens to\n> start playing with this topic).\n\nOk. I share a similar reaction to submodule diffs that we discuss above\nand word coloring, that Jonathan Tan brought up off list.\n\nBoth of them are broken in this implementation, but the NEEDSWORK\nwould hint at how to fix them.\n\nFor word coloring, we'd invent a new state BPL_EMIT_WITH_BRACES,\nthat would only store the word and at output time we'd have to add\nsign, braces, colors. Then a block movement detection is possible.\n(and this would also work with offset/len into the files longer term)\n\nFor the submodule diffs, I am really looking forward how Brandons\ncurrent work is coming along to have a repository struct such that we\ncan process submodules in the same process. For this diffing the\nrepo object would need to learn about the attribute system of\nthe submodules, such that we can obtain the whitespace coloring\nrules, as well as the config (submodule may be configured to use\ndifferent colors for diffs).\n"},{"id":"320181","messageId":"20170518193746.486-1-sbeller@google.com","threadId":"45958","inReplyTo":"20170517025857.32320-1-sbeller@google.com","subject":"[PATCHv3 00/20] Diff machine: highlight moved lines.","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:26Z","receivedAt":"2017-05-18T19:38:02Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"v3:\n* see interdiff below.\n* fixing one invalid computation (Thanks Junio!)\n* I reasoned more about submodule and word diffing, see the commit message\n of the last patch:\n \n A note on the options '--submodule=diff' and '--color-words/--word-diff':\n In the conversion to use emit_line in the prior patches both submodules\n as well as word diff output carefully chose to call emit_line with sign=0.\n All output with sign=0 is ignored for move detection purposes in this\n patch, such that no weird looking output will be generated for these\n cases. This leads to another thought: We could pass on '--color-moved' to\n submodules such that they color up moved lines for themselves. If we'd do\n so only line moves within a repository boundary are marked up.\n\n* better name for emit_line outside of diff.[ch]\n\nv2:\n* emit_line now takes an argument that indicates if we want it\n to emit the line prefix as well. This should allow for a more faithful\n refactoring in the beginning. (Thanks Jonathan!)\n* fixed memleaks (Thanks Brandon!)\n* \"git -c color.moved=true log -p\" works now! (Thanks Jeff)\n* interdiff below, though it is large.\n* less intrusive than v1 (Thanks Jonathan!)\n\nv1:\n\nFor details on *why* see the commit message of the last commit.\n\nThe first five patches are slight refactorings to get into good\nshape, the next patches are funneling all output through emit_line_*.\n\nThe second last patch introduces an option to buffer up all output\nbefore printing, and then the last patch can color up moved lines\nof code.\n\nAny feedback welcome.\n\nThanks,\nStefan\n\nStefan Beller (20):\n diff: readability fix\n diff: move line ending check into emit_hunk_header\n diff.c: factor out diff_flush_patch_all_file_pairs\n diff.c: teach emit_line_0 to accept sign parameter\n diff.c: emit_line_0 can handle no color setting\n diff.c: emit_line_0 takes parameter whether to output line prefix\n diff.c: inline emit_line_0 into emit_line\n diff.c: convert fn_out_consume to use emit_line\n diff.c: convert builtin_diff to use emit_line_*\n diff.c: convert emit_rewrite_diff to use emit_line_*\n diff.c: convert emit_rewrite_lines to use emit_line_*\n submodule.c: convert show_submodule_summary to use emit_line_fmt\n diff.c: convert emit_binary_diff_body to use emit_line_*\n diff.c: convert show_stats to use emit_line_*\n diff.c: convert word diffing to use emit_line_*\n diff.c: convert diff_flush to use emit_line_*\n diff.c: convert diff_summary to use emit_line_*\n diff.c: emit_line includes whitespace highlighting\n diff: buffer all output if asked to\n diff.c: color moved lines differently\n\n Documentation/config.txt | 14 +-\n diff.c | 849 +++++++++++++++++++++++++++++++++------------\n diff.h | 59 +++-\n submodule.c | 87 ++---\n submodule.h | 9 +-\n t/t4015-diff-whitespace.sh | 229 ++++++++++++\n 6 files changed, 969 insertions(+), 278 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 15cf322b50..451cab2875 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -840,6 +840,12 @@ void emit_line_fmt(struct diff_options *o,\n \tstrbuf_release(&sb);\n }\n \n+void diff_emit_line(struct diff_options *o, const char *set, const char *reset,\n+\t\t const char *line, int len)\n+{\n+\temit_line(o, set, reset, 1, 0, 0, line, len);\n+}\n+\n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n {\n \tif (!((ecbdata->ws_rule & WS_BLANK_AT_EOF) &&\n@@ -1009,12 +1015,10 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\tif (endp)\n \t\t\tlen = endp - data + 1;\n \t\telse {\n-\t\t\twhile (0 < size) {\n-\t\t\t\tstrbuf_addch(&sb, *data);\n-\t\t\t\tsize -= len;\n-\t\t\t\tdata += len;\n-\t\t\t}\n+\t\t\tstrbuf_add(&sb, data, size);\n \t\t\tstrbuf_addch(&sb, '\\n');\n+\t\t\tsize = 0; /* to exit the loop. */\n+\n \t\t\tdata = sb.buf;\n \t\t\tlen = sb.len;\n \t\t}\ndiff --git a/diff.h b/diff.h\nindex b8b2a33ccc..2d86e3a012 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -234,10 +234,8 @@ struct diff_options {\n \tstruct hashmap *added_lines;\n };\n \n-void emit_line_fmt(struct diff_options *o, const char *set, const char *reset,\n-\t\t int add_line_prefix, const char *fmt, ...);\n-void emit_line(struct diff_options *o, const char *set, const char *reset,\n-\t int add_line_prefix, int markup_ws, int sign, const char *line, int len);\n+void diff_emit_line(struct diff_options *o, const char *set, const char *reset,\n+\t\t const char *line, int len);\n \n enum color_diff {\n \tDIFF_RESET = 0,\ndiff --git a/submodule.c b/submodule.c\nindex 868f913971..19c63197fb 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -378,9 +378,9 @@ static void print_submodule_summary(struct rev_info *rev,\n \t\tformat_commit_message(commit, format, &sb, &ctx);\n \t\tstrbuf_addch(&sb, '\\n');\n \t\tif (commit->object.flags & SYMMETRIC_LEFT)\n-\t\t\temit_line(o, del, reset, 1, 0, 0, sb.buf, sb.len);\n+\t\t\tdiff_emit_line(o, del, reset, sb.buf, sb.len);\n \t\telse if (add)\n-\t\t\temit_line(o, add, reset, 1, 0, 0, sb.buf, sb.len);\n+\t\t\tdiff_emit_line(o, add, reset, sb.buf, sb.len);\n \t}\n \tstrbuf_release(&sb);\n }\n@@ -418,12 +418,17 @@ static void show_submodule_header(struct diff_options *o, const char *path,\n \tstruct strbuf sb = STRBUF_INIT;\n \tint fast_forward = 0, fast_backward = 0;\n \n-\tif (dirty_submodule & DIRTY_SUBMODULE_UNTRACKED)\n-\t\temit_line_fmt(o, NULL, NULL, 1,\n-\t\t\t \"Submodule %s contains untracked content\\n\", path);\n-\tif (dirty_submodule & DIRTY_SUBMODULE_MODIFIED)\n-\t\temit_line_fmt(o, NULL, NULL, 1,\n-\t\t\t \"Submodule %s contains modified content\\n\", path);\n+\tif (dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) {\n+\t\tstrbuf_addf(&sb, \"Submodule %s contains untracked content\\n\", path);\n+\t\tdiff_emit_line(o, NULL, NULL, sb.buf, sb.len);\n+\t\tstrbuf_reset(&sb);\n+\t}\n+\n+\tif (dirty_submodule & DIRTY_SUBMODULE_MODIFIED) {\n+\t\tstrbuf_addf(&sb, \"Submodule %s contains modified content\\n\", path);\n+\t\tdiff_emit_line(o, NULL, NULL, sb.buf, sb.len);\n+\t\tstrbuf_reset(&sb);\n+\t}\n \n \tif (is_null_oid(one))\n \t\tmessage = \"(new submodule)\";\n@@ -473,7 +478,7 @@ static void show_submodule_header(struct diff_options *o, const char *path,\n \t\tstrbuf_addf(&sb, \" %s\\n\", message);\n \telse\n \t\tstrbuf_addf(&sb, \"%s:\\n\", fast_backward ? \" (rewind)\" : \"\");\n-\temit_line(o, meta, reset, 1, 0, 0, sb.buf, sb.len);\n+\tdiff_emit_line(o, meta, reset, sb.buf, sb.len);\n \n \tstrbuf_release(&sb);\n }\n@@ -501,7 +506,7 @@ void show_submodule_summary(struct diff_options *o, const char *path,\n \t/* Treat revision walker failure the same as missing commits */\n \tif (prepare_submodule_summary(&rev, path, left, right, merge_bases)) {\n \t\tconst char *error = \"(revision walker failed)\\n\";\n-\t\temit_line(o, NULL, NULL, 1, 0, 0, error, strlen(error));\n+\t\tdiff_emit_line(o, NULL, NULL, error, strlen(error));\n \t\tgoto out;\n \t}\n \n@@ -570,15 +575,15 @@ void show_submodule_inline_diff(struct diff_options *o, const char *path,\n \tprepare_submodule_repo_env(&cp.env_array);\n \tif (start_command(&cp)) {\n \t\tconst char *error = \"(diff failed)\\n\";\n-\t\temit_line(o, NULL, NULL, 1, 0, 0, error, strlen(error));\n+\t\tdiff_emit_line(o, NULL, NULL, error, strlen(error));\n \t}\n \n \twhile (strbuf_getwholeline_fd(&sb, cp.out, '\\n') != EOF)\n-\t\temit_line(o, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n+\t\tdiff_emit_line(o, NULL, NULL, sb.buf, sb.len);\n \n \tif (finish_command(&cp)) {\n \t\tconst char *error = \"(diff failed)\\n\";\n-\t\temit_line(o, NULL, NULL, 1, 0, 0, error, strlen(error));\n+\t\tdiff_emit_line(o, NULL, NULL, error, strlen(error));\n \t}\n \n done:\n\n"},{"id":"320182","messageId":"20170518193746.486-2-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 01/20] diff: readability fix","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:27Z","receivedAt":"2017-05-18T19:38:06Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"We already have dereferenced 'p->two' into a local variable 'two'. Use\nthat.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 74283d9001..3f5bf8b5a4 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -3283,8 +3283,8 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)\n \tconst char *other;\n \tconst char *attr_path;\n \n-\tname = p->one->path;\n-\tother = (strcmp(name, p->two->path) ? p->two->path : NULL);\n+\tname = one->path;\n+\tother = (strcmp(name, two->path) ? two->path : NULL);\n \tattr_path = name;\n \tif (o->prefix_length)\n \t\tstrip_prefix(o->prefix_length, &name, &other);\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320183","messageId":"20170518193746.486-4-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 03/20] diff.c: factor out diff_flush_patch_all_file_pairs","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:29Z","receivedAt":"2017-05-18T19:38:07Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch we want to do more things before and after all filepairs\nare flushed. So factor flushing out all file pairs into its own function\nthat the new code can be plugged in easily.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 17 ++++++++++++-----\n 1 file changed, 12 insertions(+), 5 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex c2ed605cd0..2f9722b382 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4737,6 +4737,17 @@ void diff_warn_rename_limit(const char *varname, int needed, int degraded_cc)\n \t\twarning(_(rename_limit_advice), varname, needed);\n }\n \n+static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n+{\n+\tint i;\n+\tstruct diff_queue_struct *q = &diff_queued_diff;\n+\tfor (i = 0; i < q->nr; i++) {\n+\t\tstruct diff_filepair *p = q->queue[i];\n+\t\tif (check_pair_status(p))\n+\t\t\tdiff_flush_patch(p, o);\n+\t}\n+}\n+\n void diff_flush(struct diff_options *options)\n {\n \tstruct diff_queue_struct *q = &diff_queued_diff;\n@@ -4831,11 +4842,7 @@ void diff_flush(struct diff_options *options)\n \t\t\t}\n \t\t}\n \n-\t\tfor (i = 0; i < q->nr; i++) {\n-\t\t\tstruct diff_filepair *p = q->queue[i];\n-\t\t\tif (check_pair_status(p))\n-\t\t\t\tdiff_flush_patch(p, options);\n-\t\t}\n+\t\tdiff_flush_patch_all_file_pairs(options);\n \t}\n \n \tif (output_format & DIFF_FORMAT_CALLBACK)\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320184","messageId":"20170518193746.486-3-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 02/20] diff: move line ending check into emit_hunk_header","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:28Z","receivedAt":"2017-05-18T19:38:10Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"The emit_hunk_header() function is responsible for assembling a\nhunk header and calling emit_line() to send the hunk header\nto the output file. Its only caller fn_out_consume() needs\nto prepare for a case where the function emits an incomplete\nline and add the terminating LF.\n\nInstead make sure emit_hunk_header() to always send a\ncompleted line to emit_line().\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 3f5bf8b5a4..c2ed605cd0 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -677,6 +677,8 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \t}\n \n \tstrbuf_add(&msgbuf, line + len, org_len - len);\n+\tstrbuf_complete_line(&msgbuf);\n+\n \temit_line(ecbdata->opt, \"\", \"\", msgbuf.buf, msgbuf.len);\n \tstrbuf_release(&msgbuf);\n }\n@@ -1315,8 +1317,6 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\tlen = sane_truncate_line(ecbdata, line, len);\n \t\tfind_lno(line, ecbdata);\n \t\temit_hunk_header(ecbdata, line, len);\n-\t\tif (line[len-1] != '\\n')\n-\t\t\tputc('\\n', o->file);\n \t\treturn;\n \t}\n \n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320185","messageId":"20170518193746.486-12-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 11/20] diff.c: convert emit_rewrite_lines to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:37Z","receivedAt":"2017-05-18T19:38:16Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers emit_rewrite_lines.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 25 +++++++++++++++++--------\n 1 file changed, 17 insertions(+), 8 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 3dda9f3c8e..ca6b48cf49 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -722,15 +722,23 @@ static void add_line_count(struct strbuf *out, int count)\n static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\t\t int prefix, const char *data, int size)\n {\n-\tconst char *endp = NULL;\n-\tstatic const char *nneof = \" No newline at end of file\\n\";\n \tconst char *reset = diff_get_color(ecb->color_diff, DIFF_RESET);\n+\tstruct strbuf sb = STRBUF_INIT;\n \n \twhile (0 < size) {\n \t\tint len;\n \n-\t\tendp = memchr(data, '\\n', size);\n-\t\tlen = endp ? (endp - data + 1) : size;\n+\t\tconst char *endp = memchr(data, '\\n', size);\n+\t\tif (endp)\n+\t\t\tlen = endp - data + 1;\n+\t\telse {\n+\t\t\tstrbuf_add(&sb, data, size);\n+\t\t\tstrbuf_addch(&sb, '\\n');\n+\t\t\tsize = 0; /* to exit the loop. */\n+\n+\t\t\tdata = sb.buf;\n+\t\t\tlen = sb.len;\n+\t\t}\n \t\tif (prefix != '+') {\n \t\t\tecb->lno_in_preimage++;\n \t\t\temit_del_line(reset, ecb, data, len);\n@@ -741,12 +749,13 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\tsize -= len;\n \t\tdata += len;\n \t}\n-\tif (!endp) {\n+\tif (sb.len) {\n+\t\tstatic const char *nneof = \"\\\\ No newline at end of file\\n\";\n \t\tconst char *context = diff_get_color(ecb->color_diff,\n \t\t\t\t\t\t DIFF_CONTEXT);\n-\t\tputc('\\n', ecb->opt->file);\n-\t\temit_line(ecb->opt, context, reset, 1, '\\\\',\n-\t\t\t nneof, strlen(nneof));\n+\t\temit_line(ecb->opt, context, reset, 1, 0,\n+\t\t\t nneof, strlen(nneof));\n+\t\tstrbuf_release(&sb);\n \t}\n }\n \n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320186","messageId":"20170518193746.486-10-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 09/20] diff.c: convert builtin_diff to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:35Z","receivedAt":"2017-05-18T19:38:19Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers builtin_diff.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 33 ++++++++++++++++++++-------------\n 1 file changed, 20 insertions(+), 13 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 8186289734..4fa976d43c 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1289,8 +1289,9 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \to->found_changes = 1;\n \n \tif (ecbdata->header) {\n-\t\tfprintf(o->file, \"%s\", ecbdata->header->buf);\n-\t\tstrbuf_reset(ecbdata->header);\n+\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t ecbdata->header->buf, ecbdata->header->len);\n+\t\tstrbuf_release(ecbdata->header);\n \t\tecbdata->header = NULL;\n \t}\n \n@@ -2435,7 +2436,7 @@ static void builtin_diff(const char *name_a,\n \t\tif (complete_rewrite &&\n \t\t (textconv_one || !diff_filespec_is_binary(one)) &&\n \t\t (textconv_two || !diff_filespec_is_binary(two))) {\n-\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t\temit_rewrite_diff(name_a, name_b, one, two,\n \t\t\t\t\t\ttextconv_one, textconv_two, o);\n@@ -2445,7 +2446,7 @@ static void builtin_diff(const char *name_a,\n \t}\n \n \tif (o->irreversible_delete && lbl[1][0] == '/') {\n-\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tgoto free_ab_and_return;\n \t} else if (!DIFF_OPT_TST(o, TEXT) &&\n@@ -2456,12 +2457,15 @@ static void builtin_diff(const char *name_a,\n \t\t !DIFF_OPT_TST(o, BINARY)) {\n \t\t\tif (!oidcmp(&one->oid, &two->oid)) {\n \t\t\t\tif (must_show_header)\n-\t\t\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\t\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t\t\t\t header.buf, header.len);\n \t\t\t\tgoto free_ab_and_return;\n \t\t\t}\n-\t\t\tfprintf(o->file, \"%s\", header.buf);\n-\t\t\tfprintf(o->file, \"%sBinary files %s and %s differ\\n\",\n-\t\t\t\tline_prefix, lbl[0], lbl[1]);\n+\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t\t header.buf, header.len);\n+\t\t\temit_line_fmt(o, NULL, NULL, 1,\n+\t\t\t\t \"Binary files %s and %s differ\\n\",\n+\t\t\t\t lbl[0], lbl[1]);\n \t\t\tgoto free_ab_and_return;\n \t\t}\n \t\tif (fill_mmfile(&mf1, one) < 0 || fill_mmfile(&mf2, two) < 0)\n@@ -2470,16 +2474,19 @@ static void builtin_diff(const char *name_a,\n \t\tif (mf1.size == mf2.size &&\n \t\t !memcmp(mf1.ptr, mf2.ptr, mf1.size)) {\n \t\t\tif (must_show_header)\n-\t\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t\t\t header.buf, header.len);\n \t\t\tgoto free_ab_and_return;\n \t\t}\n-\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tif (DIFF_OPT_TST(o, BINARY))\n \t\t\temit_binary_diff(o->file, &mf1, &mf2, line_prefix);\n \t\telse\n-\t\t\tfprintf(o->file, \"%sBinary files %s and %s differ\\n\",\n-\t\t\t\tline_prefix, lbl[0], lbl[1]);\n+\t\t\temit_line_fmt(o, NULL, NULL, 1,\n+\t\t\t\t \"Binary files %s and %s differ\\n\",\n+\t\t\t\t lbl[0], lbl[1]);\n \t\to->found_changes = 1;\n \t} else {\n \t\t/* Crazy xdl interfaces.. */\n@@ -2491,7 +2498,7 @@ static void builtin_diff(const char *name_a,\n \t\tconst struct userdiff_funcname *pe;\n \n \t\tif (must_show_header) {\n-\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t}\n \n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320187","messageId":"20170518193746.486-11-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 10/20] diff.c: convert emit_rewrite_diff to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:36Z","receivedAt":"2017-05-18T19:38:22Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers emit_rewrite_diff.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 33 ++++++++++++++++++---------------\n 1 file changed, 18 insertions(+), 15 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 4fa976d43c..3dda9f3c8e 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -704,17 +704,17 @@ static void remove_tempfile(void)\n \t}\n }\n \n-static void print_line_count(FILE *file, int count)\n+static void add_line_count(struct strbuf *out, int count)\n {\n \tswitch (count) {\n \tcase 0:\n-\t\tfprintf(file, \"0,0\");\n+\t\tstrbuf_addstr(out, \"0,0\");\n \t\tbreak;\n \tcase 1:\n-\t\tfprintf(file, \"1\");\n+\t\tstrbuf_addstr(out, \"1\");\n \t\tbreak;\n \tdefault:\n-\t\tfprintf(file, \"1,%d\", count);\n+\t\tstrbuf_addf(out, \"1,%d\", count);\n \t\tbreak;\n \t}\n }\n@@ -768,7 +768,7 @@ static void emit_rewrite_diff(const char *name_a,\n \tchar *data_one, *data_two;\n \tsize_t size_one, size_two;\n \tstruct emit_callback ecbdata;\n-\tconst char *line_prefix = diff_line_prefix(o);\n+\tstruct strbuf out = STRBUF_INIT;\n \n \tif (diff_mnemonic_prefix && DIFF_OPT_TST(o, REVERSE_DIFF)) {\n \t\ta_prefix = o->b_prefix;\n@@ -806,20 +806,23 @@ static void emit_rewrite_diff(const char *name_a,\n \tecbdata.lno_in_preimage = 1;\n \tecbdata.lno_in_postimage = 1;\n \n+\temit_line_fmt(o, metainfo, reset, 1, \"--- %s%s\\n\", a_name.buf, name_a_tab);\n+\temit_line_fmt(o, metainfo, reset, 1, \"+++ %s%s\\n\", b_name.buf, name_b_tab);\n+\n \tlc_a = count_lines(data_one, size_one);\n \tlc_b = count_lines(data_two, size_two);\n-\tfprintf(o->file,\n-\t\t\"%s%s--- %s%s%s\\n%s%s+++ %s%s%s\\n%s%s@@ -\",\n-\t\tline_prefix, metainfo, a_name.buf, name_a_tab, reset,\n-\t\tline_prefix, metainfo, b_name.buf, name_b_tab, reset,\n-\t\tline_prefix, fraginfo);\n+\n+\tstrbuf_addstr(&out, \"@@ -\");\n \tif (!o->irreversible_delete)\n-\t\tprint_line_count(o->file, lc_a);\n+\t\tadd_line_count(&out, lc_a);\n \telse\n-\t\tfprintf(o->file, \"?,?\");\n-\tfprintf(o->file, \" +\");\n-\tprint_line_count(o->file, lc_b);\n-\tfprintf(o->file, \" @@%s\\n\", reset);\n+\t\tstrbuf_addstr(&out, \"?,?\");\n+\tstrbuf_addstr(&out, \" +\");\n+\tadd_line_count(&out, lc_b);\n+\tstrbuf_addstr(&out, \" @@\\n\");\n+\temit_line(o, fraginfo, reset, 1, 0, out.buf, out.len);\n+\tstrbuf_release(&out);\n+\n \tif (lc_a && !o->irreversible_delete)\n \t\temit_rewrite_lines(&ecbdata, '-', data_one, size_one);\n \tif (lc_b)\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320188","messageId":"20170518193746.486-14-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 13/20] diff.c: convert emit_binary_diff_body to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:39Z","receivedAt":"2017-05-18T19:38:25Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers emit_binary_diff_body.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 39 ++++++++++++++++++++++-----------------\n 1 file changed, 22 insertions(+), 17 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 2950455e49..126038696d 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -2244,8 +2244,8 @@ static unsigned char *deflate_it(char *data,\n \treturn deflated;\n }\n \n-static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n-\t\t\t\t const char *prefix)\n+static void emit_binary_diff_body(struct diff_options *o,\n+\t\t\t\t mmfile_t *one, mmfile_t *two)\n {\n \tvoid *cp;\n \tvoid *delta;\n@@ -2274,13 +2274,12 @@ static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n \t}\n \n \tif (delta && delta_size < deflate_size) {\n-\t\tfprintf(file, \"%sdelta %lu\\n\", prefix, orig_size);\n+\t\temit_line_fmt(o, NULL, NULL, 1, \"delta %lu\\n\", orig_size);\n \t\tfree(deflated);\n \t\tdata = delta;\n \t\tdata_size = delta_size;\n-\t}\n-\telse {\n-\t\tfprintf(file, \"%sliteral %lu\\n\", prefix, two->size);\n+\t} else {\n+\t\temit_line_fmt(o, NULL, NULL, 1, \"literal %lu\\n\", two->size);\n \t\tfree(delta);\n \t\tdata = deflated;\n \t\tdata_size = deflate_size;\n@@ -2289,8 +2288,9 @@ static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n \t/* emit data encoded in base85 */\n \tcp = data;\n \twhile (data_size) {\n+\t\tint len;\n \t\tint bytes = (52 < data_size) ? 52 : data_size;\n-\t\tchar line[70];\n+\t\tchar line[71];\n \t\tdata_size -= bytes;\n \t\tif (bytes <= 26)\n \t\t\tline[0] = bytes + 'A' - 1;\n@@ -2298,20 +2298,25 @@ static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n \t\t\tline[0] = bytes - 26 + 'a' - 1;\n \t\tencode_85(line + 1, cp, bytes);\n \t\tcp = (char *) cp + bytes;\n-\t\tfprintf(file, \"%s\", prefix);\n-\t\tfputs(line, file);\n-\t\tfputc('\\n', file);\n+\n+\t\tlen = strlen(line);\n+\t\tline[len++] = '\\n';\n+\t\tline[len] = '\\0';\n+\n+\t\temit_line(o, NULL, NULL, 1, 0, line, len);\n \t}\n-\tfprintf(file, \"%s\\n\", prefix);\n+\temit_line(o, NULL, NULL, 1, 0, \"\\n\", 1);\n \tfree(data);\n }\n \n-static void emit_binary_diff(FILE *file, mmfile_t *one, mmfile_t *two,\n-\t\t\t const char *prefix)\n+static void emit_binary_diff(struct diff_options *o,\n+\t\t\t mmfile_t *one, mmfile_t *two)\n {\n-\tfprintf(file, \"%sGIT binary patch\\n\", prefix);\n-\temit_binary_diff_body(file, one, two, prefix);\n-\temit_binary_diff_body(file, two, one, prefix);\n+\tconst char *s = \"GIT binary patch\\n\";\n+\tconst int len = strlen(s);\n+\temit_line(o, NULL, NULL, 1, 0, s, len);\n+\temit_binary_diff_body(o, one, two);\n+\temit_binary_diff_body(o, two, one);\n }\n \n int diff_filespec_is_binary(struct diff_filespec *one)\n@@ -2498,7 +2503,7 @@ static void builtin_diff(const char *name_a,\n \t\t\t header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tif (DIFF_OPT_TST(o, BINARY))\n-\t\t\temit_binary_diff(o->file, &mf1, &mf2, line_prefix);\n+\t\t\temit_binary_diff(o, &mf1, &mf2);\n \t\telse\n \t\t\temit_line_fmt(o, NULL, NULL, 1,\n \t\t\t\t \"Binary files %s and %s differ\\n\",\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320189","messageId":"20170518193746.486-13-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 12/20] submodule.c: convert show_submodule_summary to use emit_line_fmt","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:38Z","receivedAt":"2017-05-18T19:38:28Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis prepares the code for submodules to go through the\nemit_line function.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 26 ++++++++++--------\n diff.h | 3 +++\n submodule.c | 87 ++++++++++++++++++++++++++++++++-----------------------------\n submodule.h | 9 +++----\n 4 files changed, 67 insertions(+), 58 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex ca6b48cf49..2950455e49 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -516,8 +516,8 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n-static void emit_line(struct diff_options *o, const char *set, const char *reset,\n-\t\t int add_line_prefix, int sign, const char *line, int len)\n+void emit_line(struct diff_options *o, const char *set, const char *reset,\n+\t int add_line_prefix, int sign, const char *line, int len)\n {\n \tint has_trailing_newline, has_trailing_carriage_return;\n \tFILE *file = o->file;\n@@ -547,10 +547,10 @@ static void emit_line(struct diff_options *o, const char *set, const char *reset\n \t\tfputc('\\n', file);\n }\n \n-static void emit_line_fmt(struct diff_options *o,\n-\t\t\t const char *set, const char *reset,\n-\t\t\t int add_line_prefix,\n-\t\t\t const char *fmt, ...)\n+void emit_line_fmt(struct diff_options *o,\n+\t\t const char *set, const char *reset,\n+\t\t int add_line_prefix,\n+\t\t const char *fmt, ...)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tva_list ap;\n@@ -562,6 +562,12 @@ static void emit_line_fmt(struct diff_options *o,\n \tstrbuf_release(&sb);\n }\n \n+void diff_emit_line(struct diff_options *o, const char *set, const char *reset,\n+\t\t const char *line, int len)\n+{\n+\temit_line(o, set, reset, 1, 0, line, len);\n+}\n+\n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n {\n \tif (!((ecbdata->ws_rule & WS_BLANK_AT_EOF) &&\n@@ -2384,8 +2390,7 @@ static void builtin_diff(const char *name_a,\n \t (!two->mode || S_ISGITLINK(two->mode))) {\n \t\tconst char *del = diff_get_color_opt(o, DIFF_FILE_OLD);\n \t\tconst char *add = diff_get_color_opt(o, DIFF_FILE_NEW);\n-\t\tshow_submodule_summary(o->file, one->path ? one->path : two->path,\n-\t\t\t\tline_prefix,\n+\t\tshow_submodule_summary(o, one->path ? one->path : two->path,\n \t\t\t\t&one->oid, &two->oid,\n \t\t\t\ttwo->dirty_submodule,\n \t\t\t\tmeta, del, add, reset);\n@@ -2395,11 +2400,10 @@ static void builtin_diff(const char *name_a,\n \t\t (!two->mode || S_ISGITLINK(two->mode))) {\n \t\tconst char *del = diff_get_color_opt(o, DIFF_FILE_OLD);\n \t\tconst char *add = diff_get_color_opt(o, DIFF_FILE_NEW);\n-\t\tshow_submodule_inline_diff(o->file, one->path ? one->path : two->path,\n-\t\t\t\tline_prefix,\n+\t\tshow_submodule_inline_diff(o, one->path ? one->path : two->path,\n \t\t\t\t&one->oid, &two->oid,\n \t\t\t\ttwo->dirty_submodule,\n-\t\t\t\tmeta, del, add, reset, o);\n+\t\t\t\tmeta, del, add, reset);\n \t\treturn;\n \t}\n \ndiff --git a/diff.h b/diff.h\nindex 5be1ee77a7..9ad546361a 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -188,6 +188,9 @@ struct diff_options {\n \tint diff_path_counter;\n };\n \n+void diff_emit_line(struct diff_options *o, const char *set, const char *reset,\n+\t\t const char *line, int len);\n+\n enum color_diff {\n \tDIFF_RESET = 0,\n \tDIFF_CONTEXT = 1,\ndiff --git a/submodule.c b/submodule.c\nindex d3299e29c0..19c63197fb 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -362,8 +362,8 @@ static int prepare_submodule_summary(struct rev_info *rev, const char *path,\n \treturn prepare_revision_walk(rev);\n }\n \n-static void print_submodule_summary(struct rev_info *rev, FILE *f,\n-\t\tconst char *line_prefix,\n+static void print_submodule_summary(struct rev_info *rev,\n+\t\tstruct diff_options *o,\n \t\tconst char *del, const char *add, const char *reset)\n {\n \tstatic const char format[] = \" %m %s\";\n@@ -375,18 +375,12 @@ static void print_submodule_summary(struct rev_info *rev, FILE *f,\n \t\tctx.date_mode = rev->date_mode;\n \t\tctx.output_encoding = get_log_output_encoding();\n \t\tstrbuf_setlen(&sb, 0);\n-\t\tstrbuf_addstr(&sb, line_prefix);\n-\t\tif (commit->object.flags & SYMMETRIC_LEFT) {\n-\t\t\tif (del)\n-\t\t\t\tstrbuf_addstr(&sb, del);\n-\t\t}\n-\t\telse if (add)\n-\t\t\tstrbuf_addstr(&sb, add);\n \t\tformat_commit_message(commit, format, &sb, &ctx);\n-\t\tif (reset)\n-\t\t\tstrbuf_addstr(&sb, reset);\n \t\tstrbuf_addch(&sb, '\\n');\n-\t\tfprintf(f, \"%s\", sb.buf);\n+\t\tif (commit->object.flags & SYMMETRIC_LEFT)\n+\t\t\tdiff_emit_line(o, del, reset, sb.buf, sb.len);\n+\t\telse if (add)\n+\t\t\tdiff_emit_line(o, add, reset, sb.buf, sb.len);\n \t}\n \tstrbuf_release(&sb);\n }\n@@ -413,8 +407,7 @@ void prepare_submodule_repo_env(struct argv_array *out)\n * attempt to lookup both the left and right commits and put them into the\n * left and right pointers.\n */\n-static void show_submodule_header(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+static void show_submodule_header(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n \t\tconst char *reset,\n@@ -425,12 +418,17 @@ static void show_submodule_header(FILE *f, const char *path,\n \tstruct strbuf sb = STRBUF_INIT;\n \tint fast_forward = 0, fast_backward = 0;\n \n-\tif (dirty_submodule & DIRTY_SUBMODULE_UNTRACKED)\n-\t\tfprintf(f, \"%sSubmodule %s contains untracked content\\n\",\n-\t\t\tline_prefix, path);\n-\tif (dirty_submodule & DIRTY_SUBMODULE_MODIFIED)\n-\t\tfprintf(f, \"%sSubmodule %s contains modified content\\n\",\n-\t\t\tline_prefix, path);\n+\tif (dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) {\n+\t\tstrbuf_addf(&sb, \"Submodule %s contains untracked content\\n\", path);\n+\t\tdiff_emit_line(o, NULL, NULL, sb.buf, sb.len);\n+\t\tstrbuf_reset(&sb);\n+\t}\n+\n+\tif (dirty_submodule & DIRTY_SUBMODULE_MODIFIED) {\n+\t\tstrbuf_addf(&sb, \"Submodule %s contains modified content\\n\", path);\n+\t\tdiff_emit_line(o, NULL, NULL, sb.buf, sb.len);\n+\t\tstrbuf_reset(&sb);\n+\t}\n \n \tif (is_null_oid(one))\n \t\tmessage = \"(new submodule)\";\n@@ -472,21 +470,20 @@ static void show_submodule_header(FILE *f, const char *path,\n \t}\n \n output_header:\n-\tstrbuf_addf(&sb, \"%s%sSubmodule %s \", line_prefix, meta, path);\n+\tstrbuf_addf(&sb, \"Submodule %s \", path);\n \tstrbuf_add_unique_abbrev(&sb, one->hash, DEFAULT_ABBREV);\n \tstrbuf_addstr(&sb, (fast_backward || fast_forward) ? \"..\" : \"...\");\n \tstrbuf_add_unique_abbrev(&sb, two->hash, DEFAULT_ABBREV);\n \tif (message)\n-\t\tstrbuf_addf(&sb, \" %s%s\\n\", message, reset);\n+\t\tstrbuf_addf(&sb, \" %s\\n\", message);\n \telse\n-\t\tstrbuf_addf(&sb, \"%s:%s\\n\", fast_backward ? \" (rewind)\" : \"\", reset);\n-\tfwrite(sb.buf, sb.len, 1, f);\n+\t\tstrbuf_addf(&sb, \"%s:\\n\", fast_backward ? \" (rewind)\" : \"\");\n+\tdiff_emit_line(o, meta, reset, sb.buf, sb.len);\n \n \tstrbuf_release(&sb);\n }\n \n-void show_submodule_summary(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+void show_submodule_summary(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n \t\tconst char *del, const char *add, const char *reset)\n@@ -495,7 +492,7 @@ void show_submodule_summary(FILE *f, const char *path,\n \tstruct commit *left = NULL, *right = NULL;\n \tstruct commit_list *merge_bases = NULL;\n \n-\tshow_submodule_header(f, path, line_prefix, one, two, dirty_submodule,\n+\tshow_submodule_header(o, path, one, two, dirty_submodule,\n \t\t\t meta, reset, &left, &right, &merge_bases);\n \n \t/*\n@@ -508,11 +505,12 @@ void show_submodule_summary(FILE *f, const char *path,\n \n \t/* Treat revision walker failure the same as missing commits */\n \tif (prepare_submodule_summary(&rev, path, left, right, merge_bases)) {\n-\t\tfprintf(f, \"%s(revision walker failed)\\n\", line_prefix);\n+\t\tconst char *error = \"(revision walker failed)\\n\";\n+\t\tdiff_emit_line(o, NULL, NULL, error, strlen(error));\n \t\tgoto out;\n \t}\n \n-\tprint_submodule_summary(&rev, f, line_prefix, del, add, reset);\n+\tprint_submodule_summary(&rev, o, del, add, reset);\n \n out:\n \tif (merge_bases)\n@@ -521,20 +519,18 @@ void show_submodule_summary(FILE *f, const char *path,\n \tclear_commit_marks(right, ~0);\n }\n \n-void show_submodule_inline_diff(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+void show_submodule_inline_diff(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n-\t\tconst char *del, const char *add, const char *reset,\n-\t\tconst struct diff_options *o)\n+\t\tconst char *del, const char *add, const char *reset)\n {\n \tconst struct object_id *old = &empty_tree_oid, *new = &empty_tree_oid;\n \tstruct commit *left = NULL, *right = NULL;\n \tstruct commit_list *merge_bases = NULL;\n-\tstruct strbuf submodule_dir = STRBUF_INIT;\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tstruct strbuf sb = STRBUF_INIT;\n \n-\tshow_submodule_header(f, path, line_prefix, one, two, dirty_submodule,\n+\tshow_submodule_header(o, path, one, two, dirty_submodule,\n \t\t\t meta, reset, &left, &right, &merge_bases);\n \n \t/* We need a valid left and right commit to display a difference */\n@@ -547,15 +543,14 @@ void show_submodule_inline_diff(FILE *f, const char *path,\n \tif (right)\n \t\tnew = two;\n \n-\tfflush(f);\n \tcp.git_cmd = 1;\n \tcp.dir = path;\n-\tcp.out = dup(fileno(f));\n+\tcp.out = -1;\n \tcp.no_stdin = 1;\n \n \t/* TODO: other options may need to be passed here. */\n \targv_array_push(&cp.args, \"diff\");\n-\targv_array_pushf(&cp.args, \"--line-prefix=%s\", line_prefix);\n+\targv_array_pushf(&cp.args, \"--line-prefix=%s\", diff_line_prefix(o));\n \tif (DIFF_OPT_TST(o, REVERSE_DIFF)) {\n \t\targv_array_pushf(&cp.args, \"--src-prefix=%s%s/\",\n \t\t\t\t o->b_prefix, path);\n@@ -578,11 +573,21 @@ void show_submodule_inline_diff(FILE *f, const char *path,\n \t\targv_array_push(&cp.args, oid_to_hex(new));\n \n \tprepare_submodule_repo_env(&cp.env_array);\n-\tif (run_command(&cp))\n-\t\tfprintf(f, \"(diff failed)\\n\");\n+\tif (start_command(&cp)) {\n+\t\tconst char *error = \"(diff failed)\\n\";\n+\t\tdiff_emit_line(o, NULL, NULL, error, strlen(error));\n+\t}\n+\n+\twhile (strbuf_getwholeline_fd(&sb, cp.out, '\\n') != EOF)\n+\t\tdiff_emit_line(o, NULL, NULL, sb.buf, sb.len);\n+\n+\tif (finish_command(&cp)) {\n+\t\tconst char *error = \"(diff failed)\\n\";\n+\t\tdiff_emit_line(o, NULL, NULL, error, strlen(error));\n+\t}\n \n done:\n-\tstrbuf_release(&submodule_dir);\n+\tstrbuf_release(&sb);\n \tif (merge_bases)\n \t\tfree_commit_list(merge_bases);\n \tif (left)\ndiff --git a/submodule.h b/submodule.h\nindex 1277480add..9df0a3aea2 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -53,17 +53,14 @@ extern int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst);\n extern const char *submodule_strategy_to_string(const struct submodule_update_strategy *s);\n extern void handle_ignore_submodules_arg(struct diff_options *, const char *);\n-extern void show_submodule_summary(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+extern void show_submodule_summary(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n \t\tconst char *del, const char *add, const char *reset);\n-extern void show_submodule_inline_diff(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+extern void show_submodule_inline_diff(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n-\t\tconst char *del, const char *add, const char *reset,\n-\t\tconst struct diff_options *opt);\n+\t\tconst char *del, const char *add, const char *reset);\n extern void set_config_fetch_recurse_submodules(int value);\n extern void set_config_update_recurse_submodules(int value);\n /* Check if we want to update any submodule.*/\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320190","messageId":"20170518193746.486-18-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 17/20] diff.c: convert diff_summary to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:43Z","receivedAt":"2017-05-18T19:38:31Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers diff_summary.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 64 ++++++++++++++++++++++++++++++++++------------------------------\n 1 file changed, 34 insertions(+), 30 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 774f1acdd3..0945802ebf 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4504,67 +4504,71 @@ static void flush_one_pair(struct diff_filepair *p, struct diff_options *opt)\n \t}\n }\n \n-static void show_file_mode_name(FILE *file, const char *newdelete, struct diff_filespec *fs)\n+static void show_file_mode_name(struct diff_options *opt, const char *newdelete, struct diff_filespec *fs)\n {\n+\tstruct strbuf sb = STRBUF_INIT;\n \tif (fs->mode)\n-\t\tfprintf(file, \" %s mode %06o \", newdelete, fs->mode);\n+\t\tstrbuf_addf(&sb, \" %s mode %06o \", newdelete, fs->mode);\n \telse\n-\t\tfprintf(file, \" %s \", newdelete);\n-\twrite_name_quoted(fs->path, file, '\\n');\n-}\n+\t\tstrbuf_addf(&sb, \" %s \", newdelete);\n \n+\tquote_c_style(fs->path, &sb, NULL, 0);\n+\tstrbuf_addch(&sb, '\\n');\n+\temit_line(opt, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\tstrbuf_release(&sb);\n+}\n \n-static void show_mode_change(FILE *file, struct diff_filepair *p, int show_name,\n-\t\tconst char *line_prefix)\n+static void show_mode_change(struct diff_options *opt, struct diff_filepair *p,\n+\t\tint show_name)\n {\n \tif (p->one->mode && p->two->mode && p->one->mode != p->two->mode) {\n-\t\tfprintf(file, \"%s mode change %06o => %06o%c\", line_prefix, p->one->mode,\n-\t\t\tp->two->mode, show_name ? ' ' : '\\n');\n+\t\tstruct strbuf sb = STRBUF_INIT;\n \t\tif (show_name) {\n-\t\t\twrite_name_quoted(p->two->path, file, '\\n');\n+\t\t\tstrbuf_addch(&sb, ' ');\n+\t\t\tquote_c_style(p->two->path, &sb, NULL, 0);\n \t\t}\n+\t\temit_line_fmt(opt, NULL, NULL, 1,\n+\t\t\t \" mode change %06o => %06o%s\\n\",\n+\t\t\t p->one->mode, p->two->mode,\n+\t\t\t show_name ? sb.buf : \"\");\n+\t\tstrbuf_release(&sb);\n \t}\n }\n \n-static void show_rename_copy(FILE *file, const char *renamecopy, struct diff_filepair *p,\n-\t\t\tconst char *line_prefix)\n+static void show_rename_copy(struct diff_options *opt, const char *renamecopy,\n+\t\tstruct diff_filepair *p)\n {\n \tchar *names = pprint_rename(p->one->path, p->two->path);\n-\n-\tfprintf(file, \" %s %s (%d%%)\\n\", renamecopy, names, similarity_index(p));\n+\temit_line_fmt(opt, NULL, NULL, 1, \" %s %s (%d%%)\\n\",\n+\t\t renamecopy, names, similarity_index(p));\n \tfree(names);\n-\tshow_mode_change(file, p, 0, line_prefix);\n+\tshow_mode_change(opt, p, 0);\n }\n \n static void diff_summary(struct diff_options *opt, struct diff_filepair *p)\n {\n-\tFILE *file = opt->file;\n-\tconst char *line_prefix = diff_line_prefix(opt);\n-\n \tswitch(p->status) {\n \tcase DIFF_STATUS_DELETED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_file_mode_name(file, \"delete\", p->one);\n+\t\tshow_file_mode_name(opt, \"delete\", p->one);\n \t\tbreak;\n \tcase DIFF_STATUS_ADDED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_file_mode_name(file, \"create\", p->two);\n+\t\tshow_file_mode_name(opt, \"create\", p->two);\n \t\tbreak;\n \tcase DIFF_STATUS_COPIED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_rename_copy(file, \"copy\", p, line_prefix);\n+\t\tshow_rename_copy(opt, \"copy\", p);\n \t\tbreak;\n \tcase DIFF_STATUS_RENAMED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_rename_copy(file, \"rename\", p, line_prefix);\n+\t\tshow_rename_copy(opt, \"rename\", p);\n \t\tbreak;\n \tdefault:\n \t\tif (p->score) {\n-\t\t\tfprintf(file, \"%s rewrite \", line_prefix);\n-\t\t\twrite_name_quoted(p->two->path, file, ' ');\n-\t\t\tfprintf(file, \"(%d%%)\\n\", similarity_index(p));\n+\t\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\t\tstrbuf_addstr(&sb, \" rewrite \");\n+\t\t\tquote_c_style(p->two->path, &sb, NULL, 0);\n+\t\t\tstrbuf_addf(&sb, \" (%d%%)\\n\", similarity_index(p));\n+\t\t\temit_line(opt, NULL, NULL, 1, 0, sb.buf, sb.len);\n \t\t}\n-\t\tshow_mode_change(file, p, !p->score, line_prefix);\n+\t\tshow_mode_change(opt, p, !p->score);\n \t\tbreak;\n \t}\n }\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320191","messageId":"20170518193746.486-21-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 20/20] diff.c: color moved lines differently","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:46Z","receivedAt":"2017-05-18T19:38:35Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"When there is a lot of code moved around such as in 11979b9 (2005-11-18,\n\"http.c: reorder to avoid compilation failure.\") for example, the review\nprocess is quite hard, as it is not mentally challenging. It is a rather\ntedious process, that gets boring quickly. However you still need to read\nthrough all of the code to make sure the moved lines are there as supposed.\n\nWhile it is trivial to color up a patch like the following\n\n $ git diff\n diff --git a/file2.c b/file2.c\n index 9163a0f..8e66dc0 100644\n --- a/file2.c\n +++ b/file2.c\n @@ -3,13 +3,6 @@ void *xmemdupz(const void *data, size_t len)\n return memcpy(xmallocz(len), data, len);\n }\n\n -int secure_foo(struct user *u)\n -{\n - if (!u->is_allowed_foo)\n - return;\n - foo(u);\n -}\n -\n char *xstrndup(const char *str, size_t len)\n {\n char *p = memchr(str, '\\0', len);\n diff --git a/test.c b/test.c\n index a95e6fe..81eb0eb 100644\n --- a/test.c\n +++ b/test.c\n @@ -18,6 +18,13 @@ ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset)\n return total;\n }\n\n +int secure_foo(struct user *u)\n +{\n + if (!u->is_allowed_foo)\n + return;\n + foo(u);\n +}\n +\n int xdup(int fd)\n {\n int ret = dup(fd);\n\nas in this patch all lines that add or remove lines\nshould be colored in the new color that indicates moved\nlines.\n\nHowever the intention of this patch is to aid reviewers\nto spotting permutations in the moved code. So consider the\nfollowing malicious move:\n\n diff --git a/file2.c b/file2.c\n index 9163a0f..8e66dc0 100644\n --- a/file2.c\n +++ b/file2.c\n @@ -3,13 +3,6 @@ void *xmemdupz(const void *data, size_t len)\n return memcpy(xmallocz(len), data, len);\n }\n\n -int secure_foo(struct user *u)\n -{\n - if (!u->is_allowed_foo)\n - return;\n - foo(u);\n -}\n -\n char *xstrndup(const char *str, size_t len)\n {\n char *p = memchr(str, '\\0', len);\n diff --git a/test.c b/test.c\n index a95e6fe..a679c40 100644\n --- a/test.c\n +++ b/test.c\n @@ -18,6 +18,13 @@ ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset)\n return total;\n }\n\n +int secure_foo(struct user *u)\n +{\n + foo(u);\n + if (!u->is_allowed_foo)\n + return;\n +}\n +\n int xdup(int fd)\n {\n int ret = dup(fd);\n\nIf the moved code is larger, it is easier to hide some permutation in the\ncode, which is why we would not want to color all lines as \"moved\" in this\ncase. So we do not just need to color lines differently that are added and\nremoved in the same diff, we need to tweak the algorithm a bit more.\n\nAs the reviewers attention should be brought to the places, where the\ndifference is introduced to the moved code, we cannot just have one new\ncolor for all of moved code.\n\nFirst I implemented an alternative design, which would show a moved hunk\nin one color, and its boundaries in another color. This idea was error\nprone as it inspected each line and its neighboring lines to determine\nif the line was (a) moved and (b) if was deep inside a hunk by having\nmatching neighboring lines. This is unreliable as the we can construct\nhunks which have equal neighbors that just exceed the number of lines\ninspected. (Think of 'AXYZBXYZCXYZD..' with each letter as a line, that\nis permutated to AXYZCXYZBXYZD..').\n\nInstead this provides a dynamic programming greedy algorithm that finds\nthe largest moved hunk and then switches color to the alternative color\nfor the next hunk. By doing this any permutation is recognized and\ndisplayed. That implies that there is no dedicated boundary or\ninside-hunk color, but instead we'll have just two colors alternating\nfor hunks.\n\nIt would be a bit more UX friendly if the two corresponding hunks\n(of added and deleted lines) for one move would get the same color id.\n(Both get \"regular moved\" or \"alternative moved\"). This problem is\ndeferred to a later patch for now.\n\nA note on the options '--submodule=diff' and '--color-words/--word-diff':\nIn the conversion to use emit_line in the prior patches both submodules\nas well as word diff output carefully chose to call emit_line with sign=0.\nAll output with sign=0 is ignored for move detection purposes in this\npatch, such that no weird looking output will be generated for these\ncases. This leads to another thought: We could pass on '--color-moved' to\nsubmodules such that they color up moved lines for themselves. If we'd do\nso only line moves within a repository boundary are marked up.\n\nAlgorithm-by: Jonathan Tan \nSigned-off-by: Stefan Beller \n---\n Documentation/config.txt | 14 ++-\n diff.c | 266 +++++++++++++++++++++++++++++++++++++++++++--\n diff.h | 11 +-\n t/t4015-diff-whitespace.sh | 229 ++++++++++++++++++++++++++++++++++++++\n 4 files changed, 507 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 475e874d51..902d017c3b 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1051,14 +1051,24 @@ This does not affect linkgit:git-format-patch[1] or the\n 'git-diff-{asterisk}' plumbing commands. Can be overridden on the\n command line with the `--color[=]` option.\n \n+color.moved::\n+\tA boolean value, whether a diff should color moved lines\n+\tdifferently. The moved lines are searched for in the diff only.\n+\tDuplicated lines from somewhere in the project that are not\n+\tpart of the diff are not colored as moved.\n+\tDefaults to false.\n+\n color.diff.::\n \tUse customized color for diff colorization. `` specifies\n \twhich part of the patch to use the specified color, and is one\n \tof `context` (context text - `plain` is a historical synonym),\n \t`meta` (metainformation), `frag`\n \t(hunk header), 'func' (function in hunk header), `old` (removed lines),\n-\t`new` (added lines), `commit` (commit headers), or `whitespace`\n-\t(highlighting whitespace errors).\n+\t`new` (added lines), `commit` (commit headers), `whitespace`\n+\t(highlighting whitespace errors), `oldMoved` (removed lines that\n+\treappear), `newMoved` (added lines that were removed elsewhere),\n+\t`oldMovedAlternative` and `newMovedAlternative` (as a fallback to\n+\tcover adjacent blocks of moved code)\n \n color.decorate.::\n \tUse customized color for 'git log --decorate' output. `` is one\ndiff --git a/diff.c b/diff.c\nindex 2ccf93cd09..451cab2875 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -31,6 +31,7 @@ static int diff_indent_heuristic; /* experimental */\n static int diff_rename_limit_default = 400;\n static int diff_suppress_blank_empty;\n static int diff_use_color_default = -1;\n+static int diff_color_moved_default;\n static int diff_context_default = 3;\n static int diff_interhunk_context_default;\n static const char *diff_word_regex_cfg;\n@@ -55,6 +56,10 @@ static char diff_colors[][COLOR_MAXLEN] = {\n \tGIT_COLOR_YELLOW,\t/* COMMIT */\n \tGIT_COLOR_BG_RED,\t/* WHITESPACE */\n \tGIT_COLOR_NORMAL,\t/* FUNCINFO */\n+\tGIT_COLOR_BOLD_RED,\t/* OLD_MOVED_A */\n+\tGIT_COLOR_BG_RED,\t/* OLD_MOVED_B */\n+\tGIT_COLOR_BOLD_GREEN,\t/* NEW_MOVED_A */\n+\tGIT_COLOR_BG_GREEN,\t/* NEW_MOVED_B */\n };\n \n static NORETURN void die_want_option(const char *option_name)\n@@ -80,6 +85,14 @@ static int parse_diff_color_slot(const char *var)\n \t\treturn DIFF_WHITESPACE;\n \tif (!strcasecmp(var, \"func\"))\n \t\treturn DIFF_FUNCINFO;\n+\tif (!strcasecmp(var, \"oldmoved\"))\n+\t\treturn DIFF_FILE_OLD_MOVED;\n+\tif (!strcasecmp(var, \"oldmovedalternative\"))\n+\t\treturn DIFF_FILE_OLD_MOVED_ALT;\n+\tif (!strcasecmp(var, \"newmoved\"))\n+\t\treturn DIFF_FILE_NEW_MOVED;\n+\tif (!strcasecmp(var, \"newmovedalternative\"))\n+\t\treturn DIFF_FILE_NEW_MOVED_ALT;\n \treturn -1;\n }\n \n@@ -234,6 +247,10 @@ int git_diff_ui_config(const char *var, const char *value, void *cb)\n \t\tdiff_use_color_default = git_config_colorbool(var, value);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(var, \"color.moved\")) {\n+\t\tdiff_color_moved_default = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(var, \"diff.context\")) {\n \t\tdiff_context_default = git_config_int(var, value);\n \t\tif (diff_context_default < 0)\n@@ -354,6 +371,88 @@ int git_diff_basic_config(const char *var, const char *value, void *cb)\n \treturn git_default_config(var, value, cb);\n }\n \n+struct moved_entry {\n+\tstruct hashmap_entry ent;\n+\tconst struct buffered_patch_line *line;\n+\tstruct moved_entry *next_line;\n+};\n+\n+static void get_ws_cleaned_string(const struct buffered_patch_line *l,\n+\t\t\t\t struct strbuf *out)\n+{\n+\tint i;\n+\tfor (i = 0; i < l->len; i++) {\n+\t\tif (isspace(l->line[i]))\n+\t\t\tcontinue;\n+\t\tstrbuf_addch(out, l->line[i]);\n+\t}\n+}\n+\n+static int buffered_patch_line_cmp_no_ws(const struct buffered_patch_line *a,\n+\t\t\t\t\t const struct buffered_patch_line *b,\n+\t\t\t\t\t const void *keydata)\n+{\n+\tint ret;\n+\tstruct strbuf sba = STRBUF_INIT;\n+\tstruct strbuf sbb = STRBUF_INIT;\n+\n+\tget_ws_cleaned_string(a, &sba);\n+\tget_ws_cleaned_string(b, &sbb);\n+\tret = sba.len != sbb.len || strncmp(sba.buf, sbb.buf, sba.len);\n+\n+\tstrbuf_release(&sba);\n+\tstrbuf_release(&sbb);\n+\treturn ret;\n+}\n+\n+static int buffered_patch_line_cmp(const struct buffered_patch_line *a,\n+\t\t\t\t const struct buffered_patch_line *b,\n+\t\t\t\t const void *keydata)\n+{\n+\treturn a->len != b->len || strncmp(a->line, b->line, a->len);\n+}\n+\n+static int moved_entry_cmp(const struct moved_entry *a,\n+\t\t\t const struct moved_entry *b,\n+\t\t\t const void *keydata)\n+{\n+\treturn buffered_patch_line_cmp(a->line, b->line, keydata);\n+}\n+\n+static int moved_entry_cmp_no_ws(const struct moved_entry *a,\n+\t\t\t\t const struct moved_entry *b,\n+\t\t\t\t const void *keydata)\n+{\n+\treturn buffered_patch_line_cmp_no_ws(a->line, b->line, keydata);\n+}\n+\n+static unsigned get_line_hash(struct buffered_patch_line *line, unsigned ignore_ws)\n+{\n+\tstatic struct strbuf sb = STRBUF_INIT;\n+\n+\tif (ignore_ws) {\n+\t\tstrbuf_reset(&sb);\n+\t\tget_ws_cleaned_string(line, &sb);\n+\t\treturn memhash(sb.buf, sb.len);\n+\t} else {\n+\t\treturn memhash(line->line, line->len);\n+\t}\n+}\n+\n+static struct moved_entry *prepare_entry(struct diff_options *o,\n+\t\t\t\t\t int line_no)\n+{\n+\tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n+\tunsigned ignore_ws = DIFF_XDL_TST(o, IGNORE_WHITESPACE);\n+\tstruct buffered_patch_line *l = &o->line_buffer[line_no];\n+\n+\tret->ent.hash = get_line_hash(l, ignore_ws);\n+\tret->line = l;\n+\tret->next_line = NULL;\n+\n+\treturn ret;\n+}\n+\n static char *quote_two(const char *one, const char *two)\n {\n \tint need_one = quote_c_style(one, NULL, NULL, 1);\n@@ -516,6 +615,135 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n+static void add_lines_to_move_detection(struct diff_options *o)\n+{\n+\tstruct moved_entry *prev_line;\n+\n+\tint n;\n+\tfor (n = 0; n < o->line_buffer_nr; n++) {\n+\t\tint sign = 0;\n+\t\tstruct hashmap *hm;\n+\t\tstruct moved_entry *key;\n+\n+\t\tswitch (o->line_buffer[n].sign) {\n+\t\tcase '+':\n+\t\t\tsign = '+';\n+\t\t\thm = o->added_lines;\n+\t\t\tbreak;\n+\t\tcase '-':\n+\t\t\tsign = '-';\n+\t\t\thm = o->deleted_lines;\n+\t\t\tbreak;\n+\t\tcase ' ':\n+\t\tdefault:\n+\t\t\tprev_line = NULL;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tkey = prepare_entry(o, n);\n+\t\tif (prev_line &&\n+\t\t prev_line->line->sign == sign)\n+\t\t\tprev_line->next_line = key;\n+\n+\t\thashmap_add(hm, key);\n+\t\tprev_line = key;\n+\t}\n+}\n+\n+static void mark_color_as_moved(struct diff_options *o)\n+{\n+\tstruct moved_entry **pmb = NULL; /* potentially moved blocks */\n+\tint pmb_nr = 0, pmb_alloc = 0;\n+\tint alt_flag = 0;\n+\tint n;\n+\n+\tfor (n = 0; n < o->line_buffer_nr; n++) {\n+\t\tstruct hashmap *hm = NULL;\n+\t\tstruct moved_entry *key;\n+\t\tstruct moved_entry *match = NULL;\n+\t\tstruct buffered_patch_line *l = &o->line_buffer[n];\n+\t\tint i, lp, rp;\n+\n+\t\tswitch (l->sign) {\n+\t\tcase '+':\n+\t\t\thm = o->deleted_lines;\n+\t\t\tbreak;\n+\t\tcase '-':\n+\t\t\thm = o->added_lines;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\talt_flag = 0; /* reset to standard, no-alt move color */\n+\t\t\tpmb_nr = 0; /* no running sets */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* Check for any match to color it as a move. */\n+\t\tkey = prepare_entry(o, n);\n+\t\tmatch = hashmap_get(hm, key, o);\n+\t\tfree(key);\n+\t\tif (!match)\n+\t\t\tcontinue;\n+\n+\t\t/* Check any potential block runs, advance each or nullify */\n+\t\tfor (i = 0; i < pmb_nr; i++) {\n+\t\t\tstruct moved_entry *p = pmb[i];\n+\t\t\tstruct moved_entry *pnext = (p && p->next_line) ?\n+\t\t\t\t\tp->next_line : NULL;\n+\t\t\tif (pnext &&\n+\t\t\t !buffered_patch_line_cmp(pnext->line, l, o)) {\n+\t\t\t\tpmb[i] = p->next_line;\n+\t\t\t} else {\n+\t\t\t\tpmb[i] = NULL;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/* Shrink the set to the remaining runs */\n+\t\tfor (lp = 0, rp = pmb_nr - 1; lp <= rp;) {\n+\t\t\twhile (lp < pmb_nr && pmb[lp])\n+\t\t\t\tlp ++;\n+\t\t\t/* lp points at the first NULL now */\n+\n+\t\t\twhile (rp > -1 && !pmb[rp])\n+\t\t\t\trp--;\n+\t\t\t/* rp points at the last non-NULL */\n+\n+\t\t\tif (lp < pmb_nr && rp > -1 && lp < rp) {\n+\t\t\t\tpmb[lp] = pmb[rp];\n+\t\t\t\tpmb[rp] = NULL;\n+\t\t\t\trp--;\n+\t\t\t\tlp++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (rp > -1) {\n+\t\t\t/* Remember the number of running sets */\n+\t\t\tpmb_nr = rp + 1;\n+\t\t} else {\n+\t\t\t/* Toggle color */\n+\t\t\talt_flag = (alt_flag + 1) % 2;\n+\n+\t\t\t/* Build up a new set */\n+\t\t\tpmb_nr = 0;\n+\t\t\tfor (; match; match = hashmap_get_next(hm, match)) {\n+\t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n+\t\t\t\tpmb[pmb_nr++] = match;\n+\t\t\t}\n+\t\t}\n+\n+\t\tswitch (l->sign) {\n+\t\tcase '+':\n+\t\t\tl->set = diff_get_color_opt(o, DIFF_FILE_NEW_MOVED + alt_flag);\n+\t\t\tbreak;\n+\t\tcase '-':\n+\t\t\tl->set = diff_get_color_opt(o, DIFF_FILE_OLD_MOVED + alt_flag);\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tdie(\"BUG: we should have continued earlier?\");\n+\t\t}\n+\t}\n+\tfree(pmb);\n+}\n+\n static void emit_buffered_patch_line(struct diff_options *o,\n \t\t\t\t struct buffered_patch_line *e)\n {\n@@ -3518,6 +3746,8 @@ void diff_setup(struct diff_options *options)\n \toptions->line_buffer = NULL;\n \toptions->line_buffer_nr = 0;\n \toptions->line_buffer_alloc = 0;\n+\n+\toptions->color_moved = diff_color_moved_default;\n }\n \n void diff_setup_done(struct diff_options *options)\n@@ -3627,6 +3857,9 @@ void diff_setup_done(struct diff_options *options)\n \n \tif (DIFF_OPT_TST(options, FOLLOW_RENAMES) && options->pathspec.nr != 1)\n \t\tdie(_(\"--follow requires exactly one pathspec\"));\n+\n+\tif (!options->use_color || external_diff())\n+\t\toptions->color_moved = 0;\n }\n \n static int opt_arg(const char *arg, int arg_short, const char *arg_long, int *val)\n@@ -4051,6 +4284,10 @@ int diff_opt_parse(struct diff_options *options,\n \t}\n \telse if (!strcmp(arg, \"--no-color\"))\n \t\toptions->use_color = 0;\n+\telse if (!strcmp(arg, \"--color-moved\"))\n+\t\toptions->color_moved = 1;\n+\telse if (!strcmp(arg, \"--no-color-moved\"))\n+\t\toptions->color_moved = 0;\n \telse if (!strcmp(arg, \"--color-words\")) {\n \t\toptions->use_color = 1;\n \t\toptions->word_diff = DIFF_WORDS_COLOR;\n@@ -4856,16 +5093,19 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n {\n \tint i;\n \tstruct diff_queue_struct *q = &diff_queued_diff;\n-\t/*\n-\t * For testing purposes we want to make sure the diff machinery\n-\t * works completely with the buffer. If there is anything emitted\n-\t * outside the emit_buffered_patch_line, then the order is screwed\n-\t * up and the tests will fail.\n-\t *\n-\t * TODO (later in this series):\n-\t * We'll unset this flag in a later patch.\n-\t */\n-\to->use_buffer = 1;\n+\n+\tif (o->color_moved) {\n+\t\tunsigned ignore_ws = DIFF_XDL_TST(o, IGNORE_WHITESPACE);\n+\t\to->use_buffer = 1;\n+\t\to->deleted_lines = xmallocz(sizeof(*o->deleted_lines));\n+\t\to->added_lines = xmallocz(sizeof(*o->added_lines));\n+\t\thashmap_init(o->deleted_lines, ignore_ws ?\n+\t\t\t(hashmap_cmp_fn)moved_entry_cmp_no_ws :\n+\t\t\t(hashmap_cmp_fn)moved_entry_cmp, 0);\n+\t\thashmap_init(o->added_lines, ignore_ws ?\n+\t\t\t(hashmap_cmp_fn)moved_entry_cmp_no_ws :\n+\t\t\t(hashmap_cmp_fn)moved_entry_cmp, 0);\n+\t}\n \n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n@@ -4874,6 +5114,11 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \t}\n \n \tif (o->use_buffer) {\n+\t\tif (o->color_moved) {\n+\t\t\tadd_lines_to_move_detection(o);\n+\t\t\tmark_color_as_moved(o);\n+\t\t}\n+\n \t\tfor (i = 0; i < o->line_buffer_nr; i++)\n \t\t\temit_buffered_patch_line(o, &o->line_buffer[i]);\n \n@@ -4962,6 +5207,7 @@ void diff_flush(struct diff_options *options)\n \t\tif (!options->file)\n \t\t\tdie_errno(\"Could not open /dev/null\");\n \t\toptions->close_file = 1;\n+\t\toptions->color_moved = 0;\n \t\tfor (i = 0; i < q->nr; i++) {\n \t\t\tstruct diff_filepair *p = q->queue[i];\n \t\t\tif (check_pair_status(p))\ndiff --git a/diff.h b/diff.h\nindex f9fd0ea3ae..2d86e3a012 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -7,6 +7,7 @@\n #include \"tree-walk.h\"\n #include \"pathspec.h\"\n #include \"object.h\"\n+#include \"hashmap.h\"\n \n struct rev_info;\n struct diff_options;\n@@ -227,6 +228,10 @@ struct diff_options {\n \n \tstruct buffered_patch_line *line_buffer;\n \tint line_buffer_nr, line_buffer_alloc;\n+\n+\tint color_moved;\n+\tstruct hashmap *deleted_lines;\n+\tstruct hashmap *added_lines;\n };\n \n void diff_emit_line(struct diff_options *o, const char *set, const char *reset,\n@@ -241,7 +246,11 @@ enum color_diff {\n \tDIFF_FILE_NEW = 5,\n \tDIFF_COMMIT = 6,\n \tDIFF_WHITESPACE = 7,\n-\tDIFF_FUNCINFO = 8\n+\tDIFF_FUNCINFO = 8,\n+\tDIFF_FILE_OLD_MOVED = 9,\n+\tDIFF_FILE_OLD_MOVED_ALT = 10,\n+\tDIFF_FILE_NEW_MOVED = 11,\n+\tDIFF_FILE_NEW_MOVED_ALT = 12\n };\n const char *diff_get_color(int diff_use_color, enum color_diff ix);\n #define diff_get_color_opt(o, ix) \\\ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 289806d0c7..232d9ad55e 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -972,4 +972,233 @@ test_expect_success 'option overrides diff.wsErrorHighlight' '\n \n '\n \n+test_expect_success 'detect moved code, complete file' '\n+\tgit reset --hard &&\n+\tcat <<-\\EOF >test.c &&\n+\t#include\n+\tmain()\n+\t{\n+\tprintf(\"Hello World\");\n+\t}\n+\tEOF\n+\tgit add test.c &&\n+\tgit commit -m \"add main function\" &&\n+\tgit mv test.c main.c &&\n+\tgit diff HEAD --color-moved --no-renames | test_decode_color >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tdiff --git a/main.c b/main.c\n+\tnew file mode 100644\n+\tindex 0000000..a986c57\n+\t--- /dev/null\n+\t+++ b/main.c\n+\t@@ -0,0 +1,5 @@\n+\t+#include\n+\t+main()\n+\t+{\n+\t+printf(\"Hello World\");\n+\t+}\n+\tdiff --git a/test.c b/test.c\n+\tdeleted file mode 100644\n+\tindex a986c57..0000000\n+\t--- a/test.c\n+\t+++ /dev/null\n+\t@@ -1,5 +0,0 @@\n+\t-#include\n+\t-main()\n+\t-{\n+\t-printf(\"Hello World\");\n+\t-}\n+\tEOF\n+\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'detect moved code, inside file' '\n+\tgit reset --hard &&\n+\tcat <<-\\EOF >main.c &&\n+\t\t#include\n+\t\tint stuff()\n+\t\t{\n+\t\t\tprintf(\"Hello \");\n+\t\t\tprintf(\"World\\n\");\n+\t\t}\n+\n+\t\tint secure_foo(struct user *u)\n+\t\t{\n+\t\t\tif (!u->is_allowed_foo)\n+\t\t\t\treturn;\n+\t\t\tfoo(u);\n+\t\t}\n+\n+\t\tint main()\n+\t\t{\n+\t\t\tfoo();\n+\t\t}\n+\tEOF\n+\tcat <<-\\EOF >test.c &&\n+\t\t#include\n+\t\tint bar()\n+\t\t{\n+\t\t\tprintf(\"Hello World, but different\\n\");\n+\t\t}\n+\n+\t\tint another_function()\n+\t\t{\n+\t\t\tbar();\n+\t\t}\n+\tEOF\n+\tgit add main.c test.c &&\n+\tgit commit -m \"add main and test file\" &&\n+\tcat <<-\\EOF >main.c &&\n+\t\t#include\n+\t\tint stuff()\n+\t\t{\n+\t\t\tprintf(\"Hello \");\n+\t\t\tprintf(\"World\\n\");\n+\t\t}\n+\n+\t\tint main()\n+\t\t{\n+\t\t\tfoo();\n+\t\t}\n+\tEOF\n+\tcat <<-\\EOF >test.c &&\n+\t\t#include\n+\t\tint bar()\n+\t\t{\n+\t\t\tprintf(\"Hello World, but different\\n\");\n+\t\t}\n+\n+\t\tint secure_foo(struct user *u)\n+\t\t{\n+\t\t\tif (!u->is_allowed_foo)\n+\t\t\t\treturn;\n+\t\t\tfoo(u);\n+\t\t}\n+\n+\t\tint another_function()\n+\t\t{\n+\t\t\tbar();\n+\t\t}\n+\tEOF\n+\tgit diff HEAD --no-renames --color-moved| test_decode_color >actual &&\n+\tcat <<-\\EOF >expected &&\n+\tdiff --git a/main.c b/main.c\n+\tindex 27a619c..7cf9336 100644\n+\t--- a/main.c\n+\t+++ b/main.c\n+\t@@ -5,13 +5,6 @@ printf(\"Hello \");\n+\t printf(\"World\\n\");\n+\t }\n+\t \n+\t-int secure_foo(struct user *u)\n+\t-{\n+\t-if (!u->is_allowed_foo)\n+\t-return;\n+\t-foo(u);\n+\t-}\n+\t-\n+\t int main()\n+\t {\n+\t foo();\n+\tdiff --git a/test.c b/test.c\n+\tindex 1dc1d85..e34eb69 100644\n+\t--- a/test.c\n+\t+++ b/test.c\n+\t@@ -4,6 +4,13 @@ int bar()\n+\t printf(\"Hello World, but different\\n\");\n+\t }\n+\t \n+\t+int secure_foo(struct user *u)\n+\t+{\n+\t+if (!u->is_allowed_foo)\n+\t+return;\n+\t+foo(u);\n+\t+}\n+\t+\n+\t int another_function()\n+\t {\n+\t bar();\n+\tEOF\n+\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'detect permutations inside moved code, ' '\n+\t# reusing the move example from last test:\n+\tcat <<-\\EOF >main.c &&\n+\t\t#include\n+\t\tint stuff()\n+\t\t{\n+\t\t\tprintf(\"Hello \");\n+\t\t\tprintf(\"World\\n\");\n+\t\t}\n+\n+\t\tint main()\n+\t\t{\n+\t\t\tfoo();\n+\t\t}\n+\tEOF\n+\tcat <<-\\EOF >test.c &&\n+\t\t#include\n+\t\tint bar()\n+\t\t{\n+\t\t\tprintf(\"Hello World, but different\\n\");\n+\t\t}\n+\n+\t\tint secure_foo(struct user *u)\n+\t\t{\n+\t\t\tfoo(u);\n+\t\t\tif (!u->is_allowed_foo)\n+\t\t\t\treturn;\n+\t\t}\n+\n+\t\tint another_function()\n+\t\t{\n+\t\t\tbar();\n+\t\t}\n+\tEOF\n+\tgit diff HEAD --no-renames --color-moved| test_decode_color >actual &&\n+\tcat <<-\\EOF >expected &&\n+\tdiff --git a/main.c b/main.c\n+\tindex 27a619c..7cf9336 100644\n+\t--- a/main.c\n+\t+++ b/main.c\n+\t@@ -5,13 +5,6 @@ printf(\"Hello \");\n+\t printf(\"World\\n\");\n+\t }\n+\t \n+\t-int secure_foo(struct user *u)\n+\t-{\n+\t-if (!u->is_allowed_foo)\n+\t-return;\n+\t-foo(u);\n+\t-}\n+\t-\n+\t int main()\n+\t {\n+\t foo();\n+\tdiff --git a/test.c b/test.c\n+\tindex 1dc1d85..2bedec9 100644\n+\t--- a/test.c\n+\t+++ b/test.c\n+\t@@ -4,6 +4,13 @@ int bar()\n+\t printf(\"Hello World, but different\\n\");\n+\t }\n+\t \n+\t+int secure_foo(struct user *u)\n+\t+{\n+\t+foo(u);\n+\t+if (!u->is_allowed_foo)\n+\t+return;\n+\t+}\n+\t+\n+\t int another_function()\n+\t {\n+\t bar();\n+\tEOF\n+\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320192","messageId":"20170518193746.486-20-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 19/20] diff: buffer all output if asked to","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:45Z","receivedAt":"2017-05-18T19:38:41Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Introduce a new option 'use_buffer' in the struct diff_options which\ncontrols whether all output is buffered up until all output is available.\n\nWe'll have a new struct 'buffered_patch_line' in diff.h which will be\nused to buffer each line. The buffered_patch_line will duplicate the\nmemory of the line to buffer as that is easiest to reason about for now.\nIn a future patch we may want to decrease the memory usage by not\nduplicating all output for buffering but rather we may want to store\noffsets into the file or in case of hunk descriptions such as the\nsimilarity score, we could just store the relevant number and\nreproduce the text later on.\n\nThis approach was chosen as a first step because it is quite simple\ncompared to the alternative with less memory footprint.\n\nemit_line factors out the emission part into emit_line_emission,\nand depending on the diff_options->use_buffer the emission\nwill be performed directly when calling emit_line or after the\nwhole process is done, i.e. by buffering we have add the possibility\nfor a second pass over the whole output before doing the actual\noutput.\n\nIn 6440d34 (2012-03-14, diff: tweak a _copy_ of diff_options with\nword-diff) we introduced a duplicate diff options struct for word\nemissions as we may have different regex settings in there.\nWhen buffering the output, we need to operate on just one buffer,\nso we have to copy back the emissions of the word buffer into the\nmain buffer.\n\nUnconditionally enable output via buffer in this patch as it yields\na great opportunity for testing, i.e. all the diff tests from the\ntest suite pass without having reordering issues (i.e. only parts\nof the output got buffered, and we forgot to buffer other parts).\nThe test suite passes, which gives confidence that we converted all\nfunctions to use emit_line for output.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 154 ++++++++++++++++++++++++++++++++++++++++++++++++++---------------\n diff.h | 39 +++++++++++++++++\n 2 files changed, 159 insertions(+), 34 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 50d91643b6..2ccf93cd09 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -516,53 +516,85 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n-void emit_line(struct diff_options *o,\n-\t const char *set, const char *reset,\n-\t int add_line_prefix, int markup_ws,\n-\t int sign, const char *line, int len)\n+static void emit_buffered_patch_line(struct diff_options *o,\n+\t\t\t\t struct buffered_patch_line *e)\n {\n \tconst char *ws;\n \tint has_trailing_newline, has_trailing_carriage_return;\n+\tint len = e->len;\n \tFILE *file = o->file;\n \n-\tif (add_line_prefix)\n+\tif (e->add_line_prefix)\n \t\tfputs(diff_line_prefix(o), file);\n \n-\tif (markup_ws) {\n+\tswitch (e->state) {\n+\tcase BPL_EMIT_LINE_WS:\n \t\tws = diff_get_color(o->use_color, DIFF_WHITESPACE);\n+\t\tif (e->set)\n+\t\t\tfputs(e->set, file);\n+\t\tif (e->sign)\n+\t\t\tfputc(e->sign, file);\n+\t\tif (e->reset)\n+\t\t\tfputs(e->reset, file);\n+\t\tws_check_emit(e->line, e->len, o->ws_rule,\n+\t\t\t file, e->set, e->reset, ws);\n+\t\treturn;\n+\tcase BPL_EMIT_LINE_ASIS:\n+\t\thas_trailing_newline = (len > 0 && e->line[len-1] == '\\n');\n+\t\tif (has_trailing_newline)\n+\t\t\tlen--;\n+\t\thas_trailing_carriage_return = (len > 0 && e->line[len-1] == '\\r');\n+\t\tif (has_trailing_carriage_return)\n+\t\t\tlen--;\n \n-\t\tif (set)\n-\t\t\tfputs(set, file);\n-\t\tif (sign)\n-\t\t\tfputc(sign, file);\n-\t\tif (reset)\n-\t\t\tfputs(reset, file);\n-\t\tws = diff_get_color(o->use_color, DIFF_WHITESPACE);\n-\t\tws_check_emit(line, len, o->ws_rule,\n-\t\t\t file, set, reset, ws);\n+\t\tif (len || e->sign) {\n+\t\t\tif (e->set)\n+\t\t\t\tfputs(e->set, file);\n+\t\t\tif (e->sign)\n+\t\t\t\tfputc(e->sign, file);\n+\t\t\tfwrite(e->line, len, 1, file);\n+\t\t\tif (e->reset)\n+\t\t\t\tfputs(e->reset, file);\n+\t\t}\n+\t\tif (has_trailing_carriage_return)\n+\t\t\tfputc('\\r', file);\n+\t\tif (has_trailing_newline)\n+\t\t\tfputc('\\n', file);\n+\t\treturn;\n+\tcase BPL_HANDOVER:\n+\t\to->ws_rule = whitespace_rule(e->line); /*read from file, stored in line?*/\n \t\treturn;\n+\tdefault:\n+\t\tdie(\"BUG: malformatted buffered patch line: '%d'\", e->state);\n \t}\n+}\n \n-\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n-\tif (has_trailing_newline)\n-\t\tlen--;\n-\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n-\tif (has_trailing_carriage_return)\n-\t\tlen--;\n+static void append_buffered_patch_line(struct diff_options *o,\n+\t\t\t\t struct buffered_patch_line *e)\n+{\n+\tstruct buffered_patch_line *f;\n+\tALLOC_GROW(o->line_buffer,\n+\t\t o->line_buffer_nr + 1,\n+\t\t o->line_buffer_alloc);\n+\tf = &o->line_buffer[o->line_buffer_nr++];\n \n-\tif (len || sign) {\n-\t\tif (set)\n-\t\t\tfputs(set, file);\n-\t\tif (sign)\n-\t\t\tfputc(sign, file);\n-\t\tfwrite(line, len, 1, file);\n-\t\tif (reset)\n-\t\t\tfputs(reset, file);\n-\t}\n-\tif (has_trailing_carriage_return)\n-\t\tfputc('\\r', file);\n-\tif (has_trailing_newline)\n-\t\tfputc('\\n', file);\n+\tmemcpy(f, e, sizeof(struct buffered_patch_line));\n+\tf->line = e->line ? xmemdupz(e->line, e->len) : NULL;\n+}\n+\n+void emit_line(struct diff_options *o,\n+\t const char *set, const char *reset,\n+\t int add_line_prefix, int markup_ws,\n+\t int sign, const char *line, int len)\n+{\n+\tstruct buffered_patch_line e = {set, reset, line,\n+\t\tlen, sign, add_line_prefix,\n+\t\tmarkup_ws ? BPL_EMIT_LINE_WS : BPL_EMIT_LINE_ASIS};\n+\n+\tif (o->use_buffer)\n+\t\tappend_buffered_patch_line(o, &e);\n+\telse\n+\t\temit_buffered_patch_line(o, &e);\n }\n \n void emit_line_fmt(struct diff_options *o,\n@@ -1171,6 +1203,18 @@ static void diff_words_flush(struct emit_callback *ecbdata)\n \tif (ecbdata->diff_words->minus.text.size ||\n \t ecbdata->diff_words->plus.text.size)\n \t\tdiff_words_show(ecbdata->diff_words);\n+\n+\tif (ecbdata->diff_words->opt->line_buffer_nr) {\n+\t\tint i;\n+\t\tfor (i = 0; i < ecbdata->diff_words->opt->line_buffer_nr; i++)\n+\t\t\tappend_buffered_patch_line(ecbdata->opt,\n+\t\t\t\t&ecbdata->diff_words->opt->line_buffer[i]);\n+\n+\t\tfor (i = 0; i < ecbdata->diff_words->opt->line_buffer_nr; i++)\n+\t\t\tfree((void*) ecbdata->diff_words->opt->line_buffer[i].line);\n+\n+\t\tecbdata->diff_words->opt->line_buffer_nr = 0;\n+\t}\n }\n \n static void diff_filespec_load_driver(struct diff_filespec *one)\n@@ -1206,6 +1250,11 @@ static void init_diff_words_data(struct emit_callback *ecbdata,\n \t\txcalloc(1, sizeof(struct diff_words_data));\n \tecbdata->diff_words->type = o->word_diff;\n \tecbdata->diff_words->opt = o;\n+\n+\to->line_buffer = NULL;\n+\to->line_buffer_nr = 0;\n+\to->line_buffer_alloc = 0;\n+\n \tif (!o->word_regex)\n \t\to->word_regex = userdiff_word_regex(one);\n \tif (!o->word_regex)\n@@ -1240,6 +1289,7 @@ static void free_diff_words_data(struct emit_callback *ecbdata)\n {\n \tif (ecbdata->diff_words) {\n \t\tdiff_words_flush(ecbdata);\n+\t\tfree (ecbdata->diff_words->opt->line_buffer);\n \t\tfree (ecbdata->diff_words->opt);\n \t\tfree (ecbdata->diff_words->minus.text.ptr);\n \t\tfree (ecbdata->diff_words->minus.orig);\n@@ -2578,6 +2628,13 @@ static void builtin_diff(const char *name_a,\n \t\t\txecfg.ctxlen = strtoul(v, NULL, 10);\n \t\tif (o->word_diff)\n \t\t\tinit_diff_words_data(&ecbdata, o, one, two);\n+\t\tif (o->use_buffer) {\n+\t\t\tstruct buffered_patch_line e = BUFFERED_PATCH_LINE_INIT;\n+\t\t\te.state = BPL_HANDOVER;\n+\t\t\te.line = name_b;\n+\t\t\te.len = strlen(name_b);\n+\t\t\tappend_buffered_patch_line(o, &e);\n+\t\t}\n \t\tif (xdi_diff_outf(&mf1, &mf2, fn_out_consume, &ecbdata,\n \t\t\t\t &xpp, &xecfg))\n \t\t\tdie(\"unable to generate diff for %s\", one->path);\n@@ -3457,6 +3514,10 @@ void diff_setup(struct diff_options *options)\n \t\toptions->a_prefix = \"a/\";\n \t\toptions->b_prefix = \"b/\";\n \t}\n+\n+\toptions->line_buffer = NULL;\n+\toptions->line_buffer_nr = 0;\n+\toptions->line_buffer_alloc = 0;\n }\n \n void diff_setup_done(struct diff_options *options)\n@@ -4795,11 +4856,36 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n {\n \tint i;\n \tstruct diff_queue_struct *q = &diff_queued_diff;\n+\t/*\n+\t * For testing purposes we want to make sure the diff machinery\n+\t * works completely with the buffer. If there is anything emitted\n+\t * outside the emit_buffered_patch_line, then the order is screwed\n+\t * up and the tests will fail.\n+\t *\n+\t * TODO (later in this series):\n+\t * We'll unset this flag in a later patch.\n+\t */\n+\to->use_buffer = 1;\n+\n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n \t\tif (check_pair_status(p))\n \t\t\tdiff_flush_patch(p, o);\n \t}\n+\n+\tif (o->use_buffer) {\n+\t\tfor (i = 0; i < o->line_buffer_nr; i++)\n+\t\t\temit_buffered_patch_line(o, &o->line_buffer[i]);\n+\n+\t\tfor (i = 0; i < o->line_buffer_nr; i++)\n+\t\t\tfree((void*)o->line_buffer[i].line);\n+\n+\t\tfree(o->line_buffer);\n+\n+\t\to->line_buffer = NULL;\n+\t\to->line_buffer_nr = 0;\n+\t\to->line_buffer_alloc = 0;\n+\t}\n }\n \n void diff_flush(struct diff_options *options)\ndiff --git a/diff.h b/diff.h\nindex 85948ed65a..f9fd0ea3ae 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -115,6 +115,41 @@ enum diff_submodule_format {\n \tDIFF_SUBMODULE_INLINE_DIFF\n };\n \n+/*\n+ * This struct is used when we need to buffer the output of the diff output.\n+ *\n+ * NEEDSWORK: Instead of storing a copy of the line, add an offset pointer\n+ * into the pre/post image file. This pointer could be a union with the\n+ * line pointer. By storing an offset into the file instead of the literal line,\n+ * we can decrease the memory footprint for the buffered output. At first we\n+ * may want to only have indirection for the content lines, but we could\n+ * also have an enum (based on sign?) that stores prefabricated lines, e.g.\n+ * the similarity score line or hunk/file headers.\n+ */\n+struct buffered_patch_line {\n+\tconst char *set;\n+\tconst char *reset;\n+\tconst char *line;\n+\tint len;\n+\tint sign;\n+\tint add_line_prefix;\n+\tenum {\n+\t\t/*\n+\t\t * Emits [lineprefix][set][sign][reset] and then calls\n+\t\t * ws_check_emit which will output \"line\", marked up\n+\t\t * according to ws_rule.\n+\t\t */\n+\t\tBPL_EMIT_LINE_WS,\n+\n+\t\t/* Emits [lineprefix][set][sign] line [reset] */\n+\t\tBPL_EMIT_LINE_ASIS,\n+\n+\t\t/* Reloads the ws_rule; line contains the file name */\n+\t\tBPL_HANDOVER\n+\t} state;\n+};\n+#define BUFFERED_PATCH_LINE_INIT {NULL, NULL, NULL, 0, 0, 0}\n+\n struct diff_options {\n \tconst char *orderfile;\n \tconst char *pickaxe;\n@@ -188,6 +223,10 @@ struct diff_options {\n \tint diff_path_counter;\n \n \tunsigned ws_rule;\n+\tint use_buffer;\n+\n+\tstruct buffered_patch_line *line_buffer;\n+\tint line_buffer_nr, line_buffer_alloc;\n };\n \n void diff_emit_line(struct diff_options *o, const char *set, const char *reset,\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320193","messageId":"20170518193746.486-19-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 18/20] diff.c: emit_line includes whitespace highlighting","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:44Z","receivedAt":"2017-05-18T19:38:44Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Currently any whitespace highlighting happens outside the emit_line\nfunction. Teach the highlighting to emit_line, triggered by a new\nparameter.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 106 ++++++++++++++++++++++++++++++++++++++---------------------------\n diff.h | 2 ++\n 2 files changed, 64 insertions(+), 44 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 0945802ebf..50d91643b6 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -516,15 +516,33 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n-void emit_line(struct diff_options *o, const char *set, const char *reset,\n-\t int add_line_prefix, int sign, const char *line, int len)\n+void emit_line(struct diff_options *o,\n+\t const char *set, const char *reset,\n+\t int add_line_prefix, int markup_ws,\n+\t int sign, const char *line, int len)\n {\n+\tconst char *ws;\n \tint has_trailing_newline, has_trailing_carriage_return;\n \tFILE *file = o->file;\n \n \tif (add_line_prefix)\n \t\tfputs(diff_line_prefix(o), file);\n \n+\tif (markup_ws) {\n+\t\tws = diff_get_color(o->use_color, DIFF_WHITESPACE);\n+\n+\t\tif (set)\n+\t\t\tfputs(set, file);\n+\t\tif (sign)\n+\t\t\tfputc(sign, file);\n+\t\tif (reset)\n+\t\t\tfputs(reset, file);\n+\t\tws = diff_get_color(o->use_color, DIFF_WHITESPACE);\n+\t\tws_check_emit(line, len, o->ws_rule,\n+\t\t\t file, set, reset, ws);\n+\t\treturn;\n+\t}\n+\n \thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n \tif (has_trailing_newline)\n \t\tlen--;\n@@ -558,14 +576,14 @@ void emit_line_fmt(struct diff_options *o,\n \tstrbuf_vaddf(&sb, fmt, ap);\n \tva_end(ap);\n \n-\temit_line(o, set, reset, add_line_prefix, 0, sb.buf, sb.len);\n+\temit_line(o, set, reset, add_line_prefix, 0, 0, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n }\n \n void diff_emit_line(struct diff_options *o, const char *set, const char *reset,\n \t\t const char *line, int len)\n {\n-\temit_line(o, set, reset, 1, 0, line, len);\n+\temit_line(o, set, reset, 1, 0, 0, line, len);\n }\n \n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n@@ -596,16 +614,15 @@ static void emit_line_checked(const char *reset,\n \t}\n \n \tif (!ws)\n-\t\temit_line(ecbdata->opt, set, reset, 1, sign, line, len);\n+\t\temit_line(ecbdata->opt, set, reset, 1, 0, sign, line, len);\n \telse if (sign == '+' && new_blank_line_at_eof(ecbdata, line, len))\n \t\t/* Blank line at EOF - paint '+' as well */\n-\t\temit_line(ecbdata->opt, ws, reset, 1, sign, line, len);\n+\t\temit_line(ecbdata->opt, ws, reset, 1, 1, sign, line, len);\n \telse {\n \t\t/* Emit just the prefix, then the rest. */\n-\t\temit_line(ecbdata->opt, set, reset, 1, sign, \"\", 0);\n-\t\tws_check_emit(line, len, ecbdata->ws_rule,\n-\t\t\t ecbdata->opt->file, set, reset, ws);\n+\t\temit_line(ecbdata->opt, set, reset, 1, 1, sign, line, len);\n \t}\n+\n }\n \n static void emit_add_line(const char *reset,\n@@ -652,7 +669,7 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \tif (len < 10 ||\n \t memcmp(line, atat, 2) ||\n \t !(ep = memmem(line + 2, len - 2, atat, 2))) {\n-\t\temit_line(ecbdata->opt, context, reset, 1, 0, line, len);\n+\t\temit_line(ecbdata->opt, context, reset, 1, 0, 0, line, len);\n \t\treturn;\n \t}\n \tep += 2; /* skip over @@ */\n@@ -688,7 +705,7 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \tstrbuf_add(&msgbuf, line + len, org_len - len);\n \tstrbuf_complete_line(&msgbuf);\n \n-\temit_line(ecbdata->opt, \"\", \"\", 1, 0, msgbuf.buf, msgbuf.len);\n+\temit_line(ecbdata->opt, \"\", \"\", 1, 0, 0, msgbuf.buf, msgbuf.len);\n \tstrbuf_release(&msgbuf);\n }\n \n@@ -759,7 +776,7 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\tstatic const char *nneof = \"\\\\ No newline at end of file\\n\";\n \t\tconst char *context = diff_get_color(ecb->color_diff,\n \t\t\t\t\t\t DIFF_CONTEXT);\n-\t\temit_line(ecb->opt, context, reset, 1, 0,\n+\t\temit_line(ecb->opt, context, reset, 1, 0, 0,\n \t\t\t nneof, strlen(nneof));\n \t\tstrbuf_release(&sb);\n \t}\n@@ -835,7 +852,7 @@ static void emit_rewrite_diff(const char *name_a,\n \tstrbuf_addstr(&out, \" +\");\n \tadd_line_count(&out, lc_b);\n \tstrbuf_addstr(&out, \" @@\\n\");\n-\temit_line(o, fraginfo, reset, 1, 0, out.buf, out.len);\n+\temit_line(o, fraginfo, reset, 1, 0, 0, out.buf, out.len);\n \tstrbuf_release(&out);\n \n \tif (lc_a && !o->irreversible_delete)\n@@ -908,7 +925,7 @@ static int fn_out_diff_words_write_helper(struct diff_options *o,\n \twhile (count) {\n \t\tchar *p = memchr(buf, '\\n', count);\n \t\tif (print)\n-\t\t\temit_line(o, NULL, NULL, 1, 0, \"\", 0);\n+\t\t\temit_line(o, NULL, NULL, 1, 0, 0, \"\", 0);\n \n \t\tif (p != buf) {\n \t\t\tconst char *reset = st_el->color && *st_el->color ?\n@@ -917,14 +934,14 @@ static int fn_out_diff_words_write_helper(struct diff_options *o,\n \t\t\tstrbuf_add(&sb, buf, p ? p - buf : count);\n \t\t\tstrbuf_addstr(&sb, st_el->suffix);\n \t\t\temit_line(o, st_el->color, reset,\n-\t\t\t\t 0, 0, sb.buf, sb.len);\n+\t\t\t\t 0, 0, 0, sb.buf, sb.len);\n \t\t\tstrbuf_reset(&sb);\n \t\t}\n \t\tif (!p)\n \t\t\tgoto out;\n \n \t\tstrbuf_addstr(&sb, newline);\n-\t\temit_line(o, NULL, NULL, 0, 0, sb.buf, sb.len);\n+\t\temit_line(o, NULL, NULL, 0, 0, 0, sb.buf, sb.len);\n \t\tstrbuf_reset(&sb);\n \t\tcount -= p + 1 - buf;\n \t\tbuf = p + 1;\n@@ -1139,7 +1156,7 @@ static void diff_words_show(struct diff_words_data *diff_words)\n \tif (diff_words->current_plus != diff_words->plus.text.ptr +\n \t\t\tdiff_words->plus.text.size) {\n \t\tif (color_words_output_graph_prefix(diff_words))\n-\t\t\temit_line(diff_words->opt, NULL, NULL, 1, 0, \"\", 0);\n+\t\t\temit_line(diff_words->opt, NULL, NULL, 1, 0, 0, \"\", 0);\n \t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t&style->ctx, style->newline,\n \t\t\tdiff_words->plus.text.ptr + diff_words->plus.text.size\n@@ -1298,7 +1315,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \to->found_changes = 1;\n \n \tif (ecbdata->header) {\n-\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t ecbdata->header->buf, ecbdata->header->len);\n \t\tstrbuf_release(ecbdata->header);\n \t\tecbdata->header = NULL;\n@@ -1351,8 +1368,8 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t}\n \t\tdiff_words_flush(ecbdata);\n \t\tif (ecbdata->diff_words->type == DIFF_WORDS_PORCELAIN) {\n-\t\t\temit_line(o, context, reset, 1, 0, line, len);\n-\t\t\temit_line(o, NULL, NULL, 0, 0, \"~\\n\", 2);\n+\t\t\temit_line(o, context, reset, 1, 0, 0, line, len);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0, \"~\\n\", 2);\n \t\t} else {\n \t\t\t/*\n \t\t\t * Skip the prefix character, if any. With\n@@ -1363,7 +1380,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t\t line++;\n \t\t\t len--;\n \t\t\t}\n-\t\t\temit_line(o, context, reset, 1, 0, line, len);\n+\t\t\temit_line(o, context, reset, 1, 0, 0, line, len);\n \t\t}\n \t\treturn;\n \t}\n@@ -1386,7 +1403,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t/* incomplete line at the end */\n \t\tecbdata->lno_in_preimage++;\n \t\temit_line(o, diff_get_color(ecbdata->color_diff, DIFF_CONTEXT),\n-\t\t\t reset, 1, 0, line, len);\n+\t\t\t reset, 1, 0, 0, line, len);\n \t\tbreak;\n \t}\n }\n@@ -1575,7 +1592,7 @@ void print_stat_summary_0(struct diff_options *options, int files,\n \tif (!files) {\n \t\tassert(insertions == 0 && deletions == 0);\n \t\tstrbuf_addstr(&sb, \" 0 files changed\");\n-\t\temit_line(options, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\t\temit_line(options, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \t\treturn;\n \t}\n \n@@ -1603,7 +1620,7 @@ void print_stat_summary_0(struct diff_options *options, int files,\n \t\t\t deletions);\n \t}\n \tstrbuf_addch(&sb, '\\n');\n-\temit_line(options, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\temit_line(options, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n }\n \n@@ -1787,7 +1804,7 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t\tstrbuf_addf(&out, \" %*s\", number_width, \"Bin\");\n \t\t\tif (!added && !deleted) {\n \t\t\t\tstrbuf_addch(&out, '\\n');\n-\t\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\t\t\tstrbuf_reset(&out);\n \t\t\t\tcontinue;\n \t\t\t}\n@@ -1797,14 +1814,14 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t\tstrbuf_addf(&out, \"%s%\"PRIuMAX\"%s\",\n \t\t\t\tadd_c, added, reset);\n \t\t\tstrbuf_addstr(&out, \" bytes\\n\");\n-\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n \t\telse if (file->is_unmerged) {\n \t\t\tshow_name(&out, prefix, name, len);\n \t\t\tstrbuf_addstr(&out, \" Unmerged\\n\");\n-\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n@@ -1835,7 +1852,7 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\tshow_graph(&out, '+', add, add_c, reset);\n \t\tshow_graph(&out, '-', del, del_c, reset);\n \t\tstrbuf_addch(&out, '\\n');\n-\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\tstrbuf_reset(&out);\n \t}\n \n@@ -1857,7 +1874,7 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\tif (i < count)\n \t\t\tcontinue;\n \t\tif (!extra_shown)\n-\t\t\temit_line(options, NULL, NULL, 1, 0,\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0,\n \t\t\t\t \" ...\\n\", strlen(\" ...\\n\"));\n \t\textra_shown = 1;\n \t}\n@@ -2211,7 +2228,7 @@ static void checkdiff_consume(void *priv, char *line, unsigned long len)\n \t\tfprintf(data->o->file, \"%s%s:%d: %s.\\n\",\n \t\t\tline_prefix, data->filename, data->lineno, err);\n \t\tfree(err);\n-\t\temit_line(data->o, set, reset, 1, 0, line, 1);\n+\t\temit_line(data->o, set, reset, 1, 0, 0, line, 1);\n \t\tws_check_emit(line + 1, len - 1, data->ws_rule,\n \t\t\t data->o->file, set, reset, ws);\n \t} else if (line[0] == ' ') {\n@@ -2307,9 +2324,9 @@ static void emit_binary_diff_body(struct diff_options *o,\n \t\tline[len++] = '\\n';\n \t\tline[len] = '\\0';\n \n-\t\temit_line(o, NULL, NULL, 1, 0, line, len);\n+\t\temit_line(o, NULL, NULL, 1, 0, 0, line, len);\n \t}\n-\temit_line(o, NULL, NULL, 1, 0, \"\\n\", 1);\n+\temit_line(o, NULL, NULL, 1, 0, 0, \"\\n\", 1);\n \tfree(data);\n }\n \n@@ -2318,7 +2335,7 @@ static void emit_binary_diff(struct diff_options *o,\n {\n \tconst char *s = \"GIT binary patch\\n\";\n \tconst int len = strlen(s);\n-\temit_line(o, NULL, NULL, 1, 0, s, len);\n+\temit_line(o, NULL, NULL, 1, 0, 0, s, len);\n \temit_binary_diff_body(o, one, two);\n \temit_binary_diff_body(o, two, one);\n }\n@@ -2461,7 +2478,7 @@ static void builtin_diff(const char *name_a,\n \t\tif (complete_rewrite &&\n \t\t (textconv_one || !diff_filespec_is_binary(one)) &&\n \t\t (textconv_two || !diff_filespec_is_binary(two))) {\n-\t\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t\temit_rewrite_diff(name_a, name_b, one, two,\n \t\t\t\t\t\ttextconv_one, textconv_two, o);\n@@ -2471,7 +2488,7 @@ static void builtin_diff(const char *name_a,\n \t}\n \n \tif (o->irreversible_delete && lbl[1][0] == '/') {\n-\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n+\t\temit_line(o, NULL, NULL, 0, 0, 0, header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tgoto free_ab_and_return;\n \t} else if (!DIFF_OPT_TST(o, TEXT) &&\n@@ -2482,11 +2499,11 @@ static void builtin_diff(const char *name_a,\n \t\t !DIFF_OPT_TST(o, BINARY)) {\n \t\t\tif (!oidcmp(&one->oid, &two->oid)) {\n \t\t\t\tif (must_show_header)\n-\t\t\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t\t\t\t header.buf, header.len);\n \t\t\t\tgoto free_ab_and_return;\n \t\t\t}\n-\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t\t header.buf, header.len);\n \t\t\temit_line_fmt(o, NULL, NULL, 1,\n \t\t\t\t \"Binary files %s and %s differ\\n\",\n@@ -2499,11 +2516,11 @@ static void builtin_diff(const char *name_a,\n \t\tif (mf1.size == mf2.size &&\n \t\t !memcmp(mf1.ptr, mf2.ptr, mf1.size)) {\n \t\t\tif (must_show_header)\n-\t\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t\t\t header.buf, header.len);\n \t\t\tgoto free_ab_and_return;\n \t\t}\n-\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tif (DIFF_OPT_TST(o, BINARY))\n@@ -2523,7 +2540,7 @@ static void builtin_diff(const char *name_a,\n \t\tconst struct userdiff_funcname *pe;\n \n \t\tif (must_show_header) {\n-\t\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t}\n \n@@ -2540,6 +2557,7 @@ static void builtin_diff(const char *name_a,\n \t\tecbdata.label_path = lbl;\n \t\tecbdata.color_diff = want_color(o->use_color);\n \t\tecbdata.ws_rule = whitespace_rule(name_b);\n+\t\to->ws_rule = ecbdata.ws_rule;\n \t\tif (ecbdata.ws_rule & WS_BLANK_AT_EOF)\n \t\t\tcheck_blank_at_eof(&mf1, &mf2, &ecbdata);\n \t\tecbdata.opt = o;\n@@ -4514,7 +4532,7 @@ static void show_file_mode_name(struct diff_options *opt, const char *newdelete,\n \n \tquote_c_style(fs->path, &sb, NULL, 0);\n \tstrbuf_addch(&sb, '\\n');\n-\temit_line(opt, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\temit_line(opt, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n }\n \n@@ -4566,7 +4584,7 @@ static void diff_summary(struct diff_options *opt, struct diff_filepair *p)\n \t\t\tstrbuf_addstr(&sb, \" rewrite \");\n \t\t\tquote_c_style(p->two->path, &sb, NULL, 0);\n \t\t\tstrbuf_addf(&sb, \" (%d%%)\\n\", similarity_index(p));\n-\t\t\temit_line(opt, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\t\t\temit_line(opt, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \t\t}\n \t\tshow_mode_change(opt, p, !p->score);\n \t\tbreak;\n@@ -4873,10 +4891,10 @@ void diff_flush(struct diff_options *options)\n \t\t\tterm[0] = options->line_termination;\n \t\t\tterm[1] = '\\0';\n \n-\t\t\temit_line(options, NULL, NULL, 1, 0, term, !!term[0]);\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0, term, !!term[0]);\n \t\t\tif (options->stat_sep) {\n \t\t\t\t/* attach patch instead of inline */\n-\t\t\t\temit_line(options, NULL, NULL, 0, 0,\n+\t\t\t\temit_line(options, NULL, NULL, 0, 0, 0,\n \t\t\t\t\t options->stat_sep,\n \t\t\t\t\t strlen(options->stat_sep));\n \t\t\t}\ndiff --git a/diff.h b/diff.h\nindex 56d8dd036e..85948ed65a 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -186,6 +186,8 @@ struct diff_options {\n \tvoid *output_prefix_data;\n \n \tint diff_path_counter;\n+\n+\tunsigned ws_rule;\n };\n \n void diff_emit_line(struct diff_options *o, const char *set, const char *reset,\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320194","messageId":"20170518193746.486-16-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 15/20] diff.c: convert word diffing to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:41Z","receivedAt":"2017-05-18T19:38:46Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers all code related to diffing words.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 73 +++++++++++++++++++++++++++++-------------------------------------\n 1 file changed, 32 insertions(+), 41 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex f45f034036..383c8c4b52 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -897,37 +897,42 @@ struct diff_words_data {\n \tstruct diff_words_style *style;\n };\n \n-static int fn_out_diff_words_write_helper(FILE *fp,\n+static int fn_out_diff_words_write_helper(struct diff_options *o,\n \t\t\t\t\t struct diff_words_style_elem *st_el,\n \t\t\t\t\t const char *newline,\n-\t\t\t\t\t size_t count, const char *buf,\n-\t\t\t\t\t const char *line_prefix)\n+\t\t\t\t\t size_t count, const char *buf)\n {\n \tint print = 0;\n+\tstruct strbuf sb = STRBUF_INIT;\n \n \twhile (count) {\n \t\tchar *p = memchr(buf, '\\n', count);\n \t\tif (print)\n-\t\t\tfputs(line_prefix, fp);\n+\t\t\temit_line(o, NULL, NULL, 1, 0, \"\", 0);\n+\n \t\tif (p != buf) {\n-\t\t\tif (st_el->color && fputs(st_el->color, fp) < 0)\n-\t\t\t\treturn -1;\n-\t\t\tif (fputs(st_el->prefix, fp) < 0 ||\n-\t\t\t fwrite(buf, p ? p - buf : count, 1, fp) != 1 ||\n-\t\t\t fputs(st_el->suffix, fp) < 0)\n-\t\t\t\treturn -1;\n-\t\t\tif (st_el->color && *st_el->color\n-\t\t\t && fputs(GIT_COLOR_RESET, fp) < 0)\n-\t\t\t\treturn -1;\n+\t\t\tconst char *reset = st_el->color && *st_el->color ?\n+\t\t\t\t\t GIT_COLOR_RESET : NULL;\n+\t\t\tstrbuf_addstr(&sb, st_el->prefix);\n+\t\t\tstrbuf_add(&sb, buf, p ? p - buf : count);\n+\t\t\tstrbuf_addstr(&sb, st_el->suffix);\n+\t\t\temit_line(o, st_el->color, reset,\n+\t\t\t\t 0, 0, sb.buf, sb.len);\n+\t\t\tstrbuf_reset(&sb);\n \t\t}\n \t\tif (!p)\n-\t\t\treturn 0;\n-\t\tif (fputs(newline, fp) < 0)\n-\t\t\treturn -1;\n+\t\t\tgoto out;\n+\n+\t\tstrbuf_addstr(&sb, newline);\n+\t\temit_line(o, NULL, NULL, 0, 0, sb.buf, sb.len);\n+\t\tstrbuf_reset(&sb);\n \t\tcount -= p + 1 - buf;\n \t\tbuf = p + 1;\n \t\tprint = 1;\n \t}\n+\n+out:\n+\tstrbuf_release(&sb);\n \treturn 0;\n }\n \n@@ -981,14 +986,12 @@ static void fn_out_diff_words_aux(void *priv, char *line, unsigned long len)\n \tint minus_first, minus_len, plus_first, plus_len;\n \tconst char *minus_begin, *minus_end, *plus_begin, *plus_end;\n \tstruct diff_options *opt = diff_words->opt;\n-\tconst char *line_prefix;\n \n \tif (line[0] != '@' || parse_hunk_header(line, len,\n \t\t\t&minus_first, &minus_len, &plus_first, &plus_len))\n \t\treturn;\n \n \tassert(opt);\n-\tline_prefix = diff_line_prefix(opt);\n \n \t/* POSIX requires that first be decremented by one if len == 0... */\n \tif (minus_len) {\n@@ -1005,28 +1008,21 @@ static void fn_out_diff_words_aux(void *priv, char *line, unsigned long len)\n \t} else\n \t\tplus_begin = plus_end = diff_words->plus.orig[plus_first].end;\n \n-\tif (color_words_output_graph_prefix(diff_words)) {\n-\t\tfputs(line_prefix, diff_words->opt->file);\n-\t}\n \tif (diff_words->current_plus != plus_begin) {\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t\t&style->ctx, style->newline,\n \t\t\t\tplus_begin - diff_words->current_plus,\n-\t\t\t\tdiff_words->current_plus, line_prefix);\n-\t\tif (*(plus_begin - 1) == '\\n')\n-\t\t\tfputs(line_prefix, diff_words->opt->file);\n+\t\t\t\tdiff_words->current_plus);\n \t}\n \tif (minus_begin != minus_end) {\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t\t&style->old, style->newline,\n-\t\t\t\tminus_end - minus_begin, minus_begin,\n-\t\t\t\tline_prefix);\n+\t\t\t\tminus_end - minus_begin, minus_begin);\n \t}\n \tif (plus_begin != plus_end) {\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t\t&style->new, style->newline,\n-\t\t\t\tplus_end - plus_begin, plus_begin,\n-\t\t\t\tline_prefix);\n+\t\t\t\tplus_end - plus_begin, plus_begin);\n \t}\n \n \tdiff_words->current_plus = plus_end;\n@@ -1113,18 +1109,14 @@ static void diff_words_show(struct diff_words_data *diff_words)\n \tstruct diff_words_style *style = diff_words->style;\n \n \tstruct diff_options *opt = diff_words->opt;\n-\tconst char *line_prefix;\n-\n \tassert(opt);\n-\tline_prefix = diff_line_prefix(opt);\n \n \t/* special case: only removal */\n \tif (!diff_words->plus.text.size) {\n-\t\tfputs(line_prefix, diff_words->opt->file);\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t&style->old, style->newline,\n \t\t\tdiff_words->minus.text.size,\n-\t\t\tdiff_words->minus.text.ptr, line_prefix);\n+\t\t\tdiff_words->minus.text.ptr);\n \t\tdiff_words->minus.text.size = 0;\n \t\treturn;\n \t}\n@@ -1147,12 +1139,11 @@ static void diff_words_show(struct diff_words_data *diff_words)\n \tif (diff_words->current_plus != diff_words->plus.text.ptr +\n \t\t\tdiff_words->plus.text.size) {\n \t\tif (color_words_output_graph_prefix(diff_words))\n-\t\t\tfputs(line_prefix, diff_words->opt->file);\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\t\temit_line(diff_words->opt, NULL, NULL, 1, 0, \"\", 0);\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t&style->ctx, style->newline,\n \t\t\tdiff_words->plus.text.ptr + diff_words->plus.text.size\n-\t\t\t- diff_words->current_plus, diff_words->current_plus,\n-\t\t\tline_prefix);\n+\t\t\t- diff_words->current_plus, diff_words->current_plus);\n \t}\n \tdiff_words->minus.text.size = diff_words->plus.text.size = 0;\n }\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320195","messageId":"20170518193746.486-17-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 16/20] diff.c: convert diff_flush to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:42Z","receivedAt":"2017-05-18T19:38:49Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers diff_flush.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/diff.c b/diff.c\nindex 383c8c4b52..774f1acdd3 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4872,7 +4872,9 @@ void diff_flush(struct diff_options *options)\n \t\t\temit_line(options, NULL, NULL, 1, 0, term, !!term[0]);\n \t\t\tif (options->stat_sep) {\n \t\t\t\t/* attach patch instead of inline */\n-\t\t\t\tfputs(options->stat_sep, options->file);\n+\t\t\t\temit_line(options, NULL, NULL, 0, 0,\n+\t\t\t\t\t options->stat_sep,\n+\t\t\t\t\t strlen(options->stat_sep));\n \t\t\t}\n \t\t}\n \n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320196","messageId":"20170518193746.486-9-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 08/20] diff.c: convert fn_out_consume to use emit_line","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:34Z","receivedAt":"2017-05-18T19:38:55Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line.\n\nThis covers the parts of fn_out_consume. In the next\npatches we'll convert more functions that want to emit\nformatted output, so we'd want to have a formatted emit\nfunction. Add it here.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 28 ++++++++++++++++++++--------\n 1 file changed, 20 insertions(+), 8 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 3569857818..8186289734 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -547,6 +547,21 @@ static void emit_line(struct diff_options *o, const char *set, const char *reset\n \t\tfputc('\\n', file);\n }\n \n+static void emit_line_fmt(struct diff_options *o,\n+\t\t\t const char *set, const char *reset,\n+\t\t\t int add_line_prefix,\n+\t\t\t const char *fmt, ...)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tva_list ap;\n+\tva_start(ap, fmt);\n+\tstrbuf_vaddf(&sb, fmt, ap);\n+\tva_end(ap);\n+\n+\temit_line(o, set, reset, add_line_prefix, 0, sb.buf, sb.len);\n+\tstrbuf_release(&sb);\n+}\n+\n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n {\n \tif (!((ecbdata->ws_rule & WS_BLANK_AT_EOF) &&\n@@ -1270,7 +1285,6 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \tconst char *context = diff_get_color(ecbdata->color_diff, DIFF_CONTEXT);\n \tconst char *reset = diff_get_color(ecbdata->color_diff, DIFF_RESET);\n \tstruct diff_options *o = ecbdata->opt;\n-\tconst char *line_prefix = diff_line_prefix(o);\n \n \to->found_changes = 1;\n \n@@ -1282,14 +1296,12 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \n \tif (ecbdata->label_path[0]) {\n \t\tconst char *name_a_tab, *name_b_tab;\n-\n \t\tname_a_tab = strchr(ecbdata->label_path[0], ' ') ? \"\\t\" : \"\";\n \t\tname_b_tab = strchr(ecbdata->label_path[1], ' ') ? \"\\t\" : \"\";\n-\n-\t\tfprintf(o->file, \"%s%s--- %s%s%s\\n\",\n-\t\t\tline_prefix, meta, ecbdata->label_path[0], reset, name_a_tab);\n-\t\tfprintf(o->file, \"%s%s+++ %s%s%s\\n\",\n-\t\t\tline_prefix, meta, ecbdata->label_path[1], reset, name_b_tab);\n+\t\temit_line_fmt(o, meta, reset, 1, \"--- %s%s\\n\",\n+\t\t\t ecbdata->label_path[0], name_a_tab);\n+\t\temit_line_fmt(o, meta, reset, 1, \"+++ %s%s\\n\",\n+\t\t\t ecbdata->label_path[1], name_b_tab);\n \t\tecbdata->label_path[0] = ecbdata->label_path[1] = NULL;\n \t}\n \n@@ -1330,7 +1342,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\tdiff_words_flush(ecbdata);\n \t\tif (ecbdata->diff_words->type == DIFF_WORDS_PORCELAIN) {\n \t\t\temit_line(o, context, reset, 1, 0, line, len);\n-\t\t\tfputs(\"~\\n\", o->file);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, \"~\\n\", 2);\n \t\t} else {\n \t\t\t/*\n \t\t\t * Skip the prefix character, if any. With\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320197","messageId":"20170518193746.486-15-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 14/20] diff.c: convert show_stats to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:40Z","receivedAt":"2017-05-18T19:38:59Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nWe call print_stat_summary from builtin/apply, so we still\nneed the version with a file pointer, so introduce\nprint_stat_summary_0 that uses emit_line_* machinery and\nkeep print_stat_summary with the same arguments around.\n\nThe responsibility to print the line prefix moves from the callers\nof print_stat_summary_0 into the function itself.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 89 ++++++++++++++++++++++++++++++++++++++----------------------------\n diff.h | 4 +--\n 2 files changed, 53 insertions(+), 40 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 126038696d..f45f034036 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1540,20 +1540,19 @@ static int scale_linear(int it, int width, int max_change)\n \treturn 1 + (it * (width - 1) / max_change);\n }\n \n-static void show_name(FILE *file,\n+static void show_name(struct strbuf *out,\n \t\t const char *prefix, const char *name, int len)\n {\n-\tfprintf(file, \" %s%-*s |\", prefix, len, name);\n+\tstrbuf_addf(out, \" %s%-*s |\", prefix, len, name);\n }\n \n-static void show_graph(FILE *file, char ch, int cnt, const char *set, const char *reset)\n+static void show_graph(struct strbuf *out, char ch, int cnt, const char *set, const char *reset)\n {\n \tif (cnt <= 0)\n \t\treturn;\n-\tfprintf(file, \"%s\", set);\n-\twhile (cnt--)\n-\t\tputc(ch, file);\n-\tfprintf(file, \"%s\", reset);\n+\tstrbuf_addstr(out, set);\n+\tstrbuf_addchars(out, ch, cnt);\n+\tstrbuf_addstr(out, reset);\n }\n \n static void fill_print_name(struct diffstat_file *file)\n@@ -1577,14 +1576,16 @@ static void fill_print_name(struct diffstat_file *file)\n \tfile->print_name = pname;\n }\n \n-int print_stat_summary(FILE *fp, int files, int insertions, int deletions)\n+void print_stat_summary_0(struct diff_options *options, int files,\n+\t\t\t int insertions, int deletions)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n-\tint ret;\n \n \tif (!files) {\n \t\tassert(insertions == 0 && deletions == 0);\n-\t\treturn fprintf(fp, \"%s\\n\", \" 0 files changed\");\n+\t\tstrbuf_addstr(&sb, \" 0 files changed\");\n+\t\temit_line(options, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\t\treturn;\n \t}\n \n \tstrbuf_addf(&sb,\n@@ -1611,9 +1612,17 @@ int print_stat_summary(FILE *fp, int files, int insertions, int deletions)\n \t\t\t deletions);\n \t}\n \tstrbuf_addch(&sb, '\\n');\n-\tret = fputs(sb.buf, fp);\n+\temit_line(options, NULL, NULL, 1, 0, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n-\treturn ret;\n+}\n+\n+void print_stat_summary(FILE *fp, int files,\n+\t\t\tint insertions, int deletions)\n+{\n+\tstruct diff_options o;\n+\tmemset(&o, 0, sizeof(o));\n+\to.file = fp;\n+\tprint_stat_summary_0(&o, files, insertions, deletions);\n }\n \n static void show_stats(struct diffstat_t *data, struct diff_options *options)\n@@ -1623,13 +1632,13 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \tint total_files = data->nr, count;\n \tint width, name_width, graph_width, number_width = 0, bin_width = 0;\n \tconst char *reset, *add_c, *del_c;\n-\tconst char *line_prefix = \"\";\n \tint extra_shown = 0;\n+\tconst char *line_prefix = diff_line_prefix(options);\n+\tstruct strbuf out = STRBUF_INIT;\n \n \tif (data->nr == 0)\n \t\treturn;\n \n-\tline_prefix = diff_line_prefix(options);\n \tcount = options->stat_count ? options->stat_count : data->nr;\n \n \treset = diff_get_color_opt(options, DIFF_RESET);\n@@ -1783,26 +1792,29 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t}\n \n \t\tif (file->is_binary) {\n-\t\t\tfprintf(options->file, \"%s\", line_prefix);\n-\t\t\tshow_name(options->file, prefix, name, len);\n-\t\t\tfprintf(options->file, \" %*s\", number_width, \"Bin\");\n+\t\t\tshow_name(&out, prefix, name, len);\n+\t\t\tstrbuf_addf(&out, \" %*s\", number_width, \"Bin\");\n \t\t\tif (!added && !deleted) {\n-\t\t\t\tputc('\\n', options->file);\n+\t\t\t\tstrbuf_addch(&out, '\\n');\n+\t\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\t\tstrbuf_reset(&out);\n \t\t\t\tcontinue;\n \t\t\t}\n-\t\t\tfprintf(options->file, \" %s%\"PRIuMAX\"%s\",\n+\t\t\tstrbuf_addf(&out, \" %s%\"PRIuMAX\"%s\",\n \t\t\t\tdel_c, deleted, reset);\n-\t\t\tfprintf(options->file, \" -> \");\n-\t\t\tfprintf(options->file, \"%s%\"PRIuMAX\"%s\",\n+\t\t\tstrbuf_addstr(&out, \" -> \");\n+\t\t\tstrbuf_addf(&out, \"%s%\"PRIuMAX\"%s\",\n \t\t\t\tadd_c, added, reset);\n-\t\t\tfprintf(options->file, \" bytes\");\n-\t\t\tfprintf(options->file, \"\\n\");\n+\t\t\tstrbuf_addstr(&out, \" bytes\\n\");\n+\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n \t\telse if (file->is_unmerged) {\n-\t\t\tfprintf(options->file, \"%s\", line_prefix);\n-\t\t\tshow_name(options->file, prefix, name, len);\n-\t\t\tfprintf(options->file, \" Unmerged\\n\");\n+\t\t\tshow_name(&out, prefix, name, len);\n+\t\t\tstrbuf_addstr(&out, \" Unmerged\\n\");\n+\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n \n@@ -1825,14 +1837,15 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t\t\tadd = total - del;\n \t\t\t}\n \t\t}\n-\t\tfprintf(options->file, \"%s\", line_prefix);\n-\t\tshow_name(options->file, prefix, name, len);\n-\t\tfprintf(options->file, \" %*\"PRIuMAX\"%s\",\n+\t\tshow_name(&out, prefix, name, len);\n+\t\tstrbuf_addf(&out, \" %*\"PRIuMAX\"%s\",\n \t\t\tnumber_width, added + deleted,\n \t\t\tadded + deleted ? \" \" : \"\");\n-\t\tshow_graph(options->file, '+', add, add_c, reset);\n-\t\tshow_graph(options->file, '-', del, del_c, reset);\n-\t\tfprintf(options->file, \"\\n\");\n+\t\tshow_graph(&out, '+', add, add_c, reset);\n+\t\tshow_graph(&out, '-', del, del_c, reset);\n+\t\tstrbuf_addch(&out, '\\n');\n+\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\tstrbuf_reset(&out);\n \t}\n \n \tfor (i = 0; i < data->nr; i++) {\n@@ -1853,11 +1866,12 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\tif (i < count)\n \t\t\tcontinue;\n \t\tif (!extra_shown)\n-\t\t\tfprintf(options->file, \"%s ...\\n\", line_prefix);\n+\t\t\temit_line(options, NULL, NULL, 1, 0,\n+\t\t\t\t \" ...\\n\", strlen(\" ...\\n\"));\n \t\textra_shown = 1;\n \t}\n-\tfprintf(options->file, \"%s\", line_prefix);\n-\tprint_stat_summary(options->file, total_files, adds, dels);\n+\n+\tprint_stat_summary_0(options, total_files, adds, dels);\n }\n \n static void show_shortstats(struct diffstat_t *data, struct diff_options *options)\n@@ -1869,7 +1883,7 @@ static void show_shortstats(struct diffstat_t *data, struct diff_options *option\n \n \tfor (i = 0; i < data->nr; i++) {\n \t\tint added = data->files[i]->added;\n-\t\tint deleted= data->files[i]->deleted;\n+\t\tint deleted = data->files[i]->deleted;\n \n \t\tif (data->files[i]->is_unmerged ||\n \t\t (!data->files[i]->is_interesting && (added + deleted == 0))) {\n@@ -1879,8 +1893,7 @@ static void show_shortstats(struct diffstat_t *data, struct diff_options *option\n \t\t\tdels += deleted;\n \t\t}\n \t}\n-\tfprintf(options->file, \"%s\", diff_line_prefix(options));\n-\tprint_stat_summary(options->file, total_files, adds, dels);\n+\tprint_stat_summary_0(options, total_files, adds, dels);\n }\n \n static void show_numstat(struct diffstat_t *data, struct diff_options *options)\ndiff --git a/diff.h b/diff.h\nindex 9ad546361a..56d8dd036e 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -392,8 +392,8 @@ extern int parse_rename_score(const char **cp_p);\n \n extern long parse_algorithm_value(const char *value);\n \n-extern int print_stat_summary(FILE *fp, int files,\n-\t\t\t int insertions, int deletions);\n+extern void print_stat_summary(FILE *fp, int files,\n+\t\t\t int insertions, int deletions);\n extern void setup_diff_pager(struct diff_options *);\n \n #endif /* DIFF_H */\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320198","messageId":"20170518193746.486-8-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 07/20] diff.c: inline emit_line_0 into emit_line","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:33Z","receivedAt":"2017-05-18T19:39:00Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"The argument list of emit_line_0 is just 2 more arguments that are\nhard-coded in emit_line. Eliminate this intermediate function and\nrename the remaining function by dropping the '_0'.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 34 ++++++++++++++--------------------\n 1 file changed, 14 insertions(+), 20 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 25735f03d2..3569857818 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -516,8 +516,8 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n-static void emit_line_0(struct diff_options *o, const char *set, const char *reset,\n-\t\t\tint add_line_prefix, int sign, const char *line, int len)\n+static void emit_line(struct diff_options *o, const char *set, const char *reset,\n+\t\t int add_line_prefix, int sign, const char *line, int len)\n {\n \tint has_trailing_newline, has_trailing_carriage_return;\n \tFILE *file = o->file;\n@@ -547,12 +547,6 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n \t\tfputc('\\n', file);\n }\n \n-static void emit_line(struct diff_options *o, const char *set, const char *reset,\n-\t\t const char *line, int len)\n-{\n-\temit_line_0(o, set, reset, 1, 0, line, len);\n-}\n-\n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n {\n \tif (!((ecbdata->ws_rule & WS_BLANK_AT_EOF) &&\n@@ -581,13 +575,13 @@ static void emit_line_checked(const char *reset,\n \t}\n \n \tif (!ws)\n-\t\temit_line_0(ecbdata->opt, set, reset, 1, sign, line, len);\n+\t\temit_line(ecbdata->opt, set, reset, 1, sign, line, len);\n \telse if (sign == '+' && new_blank_line_at_eof(ecbdata, line, len))\n \t\t/* Blank line at EOF - paint '+' as well */\n-\t\temit_line_0(ecbdata->opt, ws, reset, 1, sign, line, len);\n+\t\temit_line(ecbdata->opt, ws, reset, 1, sign, line, len);\n \telse {\n \t\t/* Emit just the prefix, then the rest. */\n-\t\temit_line_0(ecbdata->opt, set, reset, 1, sign, \"\", 0);\n+\t\temit_line(ecbdata->opt, set, reset, 1, sign, \"\", 0);\n \t\tws_check_emit(line, len, ecbdata->ws_rule,\n \t\t\t ecbdata->opt->file, set, reset, ws);\n \t}\n@@ -637,7 +631,7 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \tif (len < 10 ||\n \t memcmp(line, atat, 2) ||\n \t !(ep = memmem(line + 2, len - 2, atat, 2))) {\n-\t\temit_line(ecbdata->opt, context, reset, line, len);\n+\t\temit_line(ecbdata->opt, context, reset, 1, 0, line, len);\n \t\treturn;\n \t}\n \tep += 2; /* skip over @@ */\n@@ -673,7 +667,7 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \tstrbuf_add(&msgbuf, line + len, org_len - len);\n \tstrbuf_complete_line(&msgbuf);\n \n-\temit_line(ecbdata->opt, \"\", \"\", msgbuf.buf, msgbuf.len);\n+\temit_line(ecbdata->opt, \"\", \"\", 1, 0, msgbuf.buf, msgbuf.len);\n \tstrbuf_release(&msgbuf);\n }\n \n@@ -736,8 +730,8 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\tconst char *context = diff_get_color(ecb->color_diff,\n \t\t\t\t\t\t DIFF_CONTEXT);\n \t\tputc('\\n', ecb->opt->file);\n-\t\temit_line_0(ecb->opt, context, reset, 1, '\\\\',\n-\t\t\t nneof, strlen(nneof));\n+\t\temit_line(ecb->opt, context, reset, 1, '\\\\',\n+\t\t\t nneof, strlen(nneof));\n \t}\n }\n \n@@ -1335,7 +1329,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t}\n \t\tdiff_words_flush(ecbdata);\n \t\tif (ecbdata->diff_words->type == DIFF_WORDS_PORCELAIN) {\n-\t\t\temit_line(o, context, reset, line, len);\n+\t\t\temit_line(o, context, reset, 1, 0, line, len);\n \t\t\tfputs(\"~\\n\", o->file);\n \t\t} else {\n \t\t\t/*\n@@ -1347,7 +1341,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t\t line++;\n \t\t\t len--;\n \t\t\t}\n-\t\t\temit_line(o, context, reset, line, len);\n+\t\t\temit_line(o, context, reset, 1, 0, line, len);\n \t\t}\n \t\treturn;\n \t}\n@@ -1370,7 +1364,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t/* incomplete line at the end */\n \t\tecbdata->lno_in_preimage++;\n \t\temit_line(o, diff_get_color(ecbdata->color_diff, DIFF_CONTEXT),\n-\t\t\t reset, line, len);\n+\t\t\t reset, 1, 0, line, len);\n \t\tbreak;\n \t}\n }\n@@ -2182,7 +2176,7 @@ static void checkdiff_consume(void *priv, char *line, unsigned long len)\n \t\tfprintf(data->o->file, \"%s%s:%d: %s.\\n\",\n \t\t\tline_prefix, data->filename, data->lineno, err);\n \t\tfree(err);\n-\t\temit_line(data->o, set, reset, line, 1);\n+\t\temit_line(data->o, set, reset, 1, 0, line, 1);\n \t\tws_check_emit(line + 1, len - 1, data->ws_rule,\n \t\t\t data->o->file, set, reset, ws);\n \t} else if (line[0] == ' ') {\n@@ -4831,7 +4825,7 @@ void diff_flush(struct diff_options *options)\n \t\t\tterm[0] = options->line_termination;\n \t\t\tterm[1] = '\\0';\n \n-\t\t\temit_line(options, NULL, NULL, term, !!term[0]);\n+\t\t\temit_line(options, NULL, NULL, 1, 0, term, !!term[0]);\n \t\t\tif (options->stat_sep) {\n \t\t\t\t/* attach patch instead of inline */\n \t\t\t\tfputs(options->stat_sep, options->file);\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320199","messageId":"20170518193746.486-7-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 06/20] diff.c: emit_line_0 takes parameter whether to output line prefix","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:32Z","receivedAt":"2017-05-18T19:39:05Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In later patches we'll make extensive use of emit_line_0, as we'd want\nto funnel all output through this function such that we can add buffering\nthere.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 15 ++++++++-------\n 1 file changed, 8 insertions(+), 7 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 6c1886d495..25735f03d2 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -517,12 +517,13 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n }\n \n static void emit_line_0(struct diff_options *o, const char *set, const char *reset,\n-\t\t\tint sign, const char *line, int len)\n+\t\t\tint add_line_prefix, int sign, const char *line, int len)\n {\n \tint has_trailing_newline, has_trailing_carriage_return;\n \tFILE *file = o->file;\n \n-\tfputs(diff_line_prefix(o), file);\n+\tif (add_line_prefix)\n+\t\tfputs(diff_line_prefix(o), file);\n \n \thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n \tif (has_trailing_newline)\n@@ -549,7 +550,7 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n static void emit_line(struct diff_options *o, const char *set, const char *reset,\n \t\t const char *line, int len)\n {\n-\temit_line_0(o, set, reset, 0, line, len);\n+\temit_line_0(o, set, reset, 1, 0, line, len);\n }\n \n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n@@ -580,13 +581,13 @@ static void emit_line_checked(const char *reset,\n \t}\n \n \tif (!ws)\n-\t\temit_line_0(ecbdata->opt, set, reset, sign, line, len);\n+\t\temit_line_0(ecbdata->opt, set, reset, 1, sign, line, len);\n \telse if (sign == '+' && new_blank_line_at_eof(ecbdata, line, len))\n \t\t/* Blank line at EOF - paint '+' as well */\n-\t\temit_line_0(ecbdata->opt, ws, reset, sign, line, len);\n+\t\temit_line_0(ecbdata->opt, ws, reset, 1, sign, line, len);\n \telse {\n \t\t/* Emit just the prefix, then the rest. */\n-\t\temit_line_0(ecbdata->opt, set, reset, sign, \"\", 0);\n+\t\temit_line_0(ecbdata->opt, set, reset, 1, sign, \"\", 0);\n \t\tws_check_emit(line, len, ecbdata->ws_rule,\n \t\t\t ecbdata->opt->file, set, reset, ws);\n \t}\n@@ -735,7 +736,7 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\tconst char *context = diff_get_color(ecb->color_diff,\n \t\t\t\t\t\t DIFF_CONTEXT);\n \t\tputc('\\n', ecb->opt->file);\n-\t\temit_line_0(ecb->opt, context, reset, '\\\\',\n+\t\temit_line_0(ecb->opt, context, reset, 1, '\\\\',\n \t\t\t nneof, strlen(nneof));\n \t}\n }\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320200","messageId":"20170518193746.486-6-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 05/20] diff.c: emit_line_0 can handle no color setting","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:31Z","receivedAt":"2017-05-18T19:39:08Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In later patches we may pass lines that are not colored to the central\nfunction emit_line_0, so we need to emit the color only when it is\nnon-NULL.\n\nWe could have chosen to pass \"\" instead of NULL, but that would be more\nwork.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 73e55b0c10..6c1886d495 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -532,11 +532,13 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n \t\tlen--;\n \n \tif (len || sign) {\n-\t\tfputs(set, file);\n+\t\tif (set)\n+\t\t\tfputs(set, file);\n \t\tif (sign)\n \t\t\tfputc(sign, file);\n \t\tfwrite(line, len, 1, file);\n-\t\tfputs(reset, file);\n+\t\tif (reset)\n+\t\t\tfputs(reset, file);\n \t}\n \tif (has_trailing_carriage_return)\n \t\tfputc('\\r', file);\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320201","messageId":"20170518193746.486-5-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv3 04/20] diff.c: teach emit_line_0 to accept sign parameter","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-18T19:37:30Z","receivedAt":"2017-05-18T19:39:12Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Teach emit_line_0 take a \"sign\" parameter specifically intended\nto hold the sign of the line instead of a separate \"first\" parameter\nrepresenting the first character of the line to be printed. Callers\nthat store the sign and line separately can use the \"sign\" parameter\nlike they used the \"first\" parameter previously, and callers that store\nthe sign and line together (or do not have a sign) no longer need to\nmanipulate their arguments to fit the requirements of emit_line_0.\n\nWith this patch other callers hard code the sign (which are '+', '-',\n' ' and '\\\\') such that we do not run into unexpectedly emitting an\nerroneous '\\0'.\n\nThe audit of the caller revealed that the sign cannot be '\\n' or '\\r',\nso remove that condition for trailing newline or carriage return in the\nsign; the else part of the condition handles the len==0 perfectly,\nso we can drop the if/else construct.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 39 ++++++++++++++++-----------------------\n 1 file changed, 16 insertions(+), 23 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 2f9722b382..73e55b0c10 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -517,33 +517,24 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n }\n \n static void emit_line_0(struct diff_options *o, const char *set, const char *reset,\n-\t\t\tint first, const char *line, int len)\n+\t\t\tint sign, const char *line, int len)\n {\n \tint has_trailing_newline, has_trailing_carriage_return;\n-\tint nofirst;\n \tFILE *file = o->file;\n \n \tfputs(diff_line_prefix(o), file);\n \n-\tif (len == 0) {\n-\t\thas_trailing_newline = (first == '\\n');\n-\t\thas_trailing_carriage_return = (!has_trailing_newline &&\n-\t\t\t\t\t\t(first == '\\r'));\n-\t\tnofirst = has_trailing_newline || has_trailing_carriage_return;\n-\t} else {\n-\t\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n-\t\tif (has_trailing_newline)\n-\t\t\tlen--;\n-\t\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n-\t\tif (has_trailing_carriage_return)\n-\t\t\tlen--;\n-\t\tnofirst = 0;\n-\t}\n+\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n+\tif (has_trailing_newline)\n+\t\tlen--;\n+\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n+\tif (has_trailing_carriage_return)\n+\t\tlen--;\n \n-\tif (len || !nofirst) {\n+\tif (len || sign) {\n \t\tfputs(set, file);\n-\t\tif (!nofirst)\n-\t\t\tfputc(first, file);\n+\t\tif (sign)\n+\t\t\tfputc(sign, file);\n \t\tfwrite(line, len, 1, file);\n \t\tfputs(reset, file);\n \t}\n@@ -556,7 +547,7 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n static void emit_line(struct diff_options *o, const char *set, const char *reset,\n \t\t const char *line, int len)\n {\n-\temit_line_0(o, set, reset, line[0], line+1, len-1);\n+\temit_line_0(o, set, reset, 0, line, len);\n }\n \n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n@@ -4833,9 +4824,11 @@ void diff_flush(struct diff_options *options)\n \n \tif (output_format & DIFF_FORMAT_PATCH) {\n \t\tif (separator) {\n-\t\t\tfprintf(options->file, \"%s%c\",\n-\t\t\t\tdiff_line_prefix(options),\n-\t\t\t\toptions->line_termination);\n+\t\t\tchar term[2];\n+\t\t\tterm[0] = options->line_termination;\n+\t\t\tterm[1] = '\\0';\n+\n+\t\t\temit_line(options, NULL, NULL, term, !!term[0]);\n \t\t\tif (options->stat_sep) {\n \t\t\t\t/* attach patch instead of inline */\n \t\t\t\tfputs(options->stat_sep, options->file);\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320242","messageId":"20170518163307.657caedc@twelve2.svl.corp.google.com","threadId":"45958","inReplyTo":"20170518193746.486-5-sbeller@google.com","subject":"Re: [PATCHv3 04/20] diff.c: teach emit_line_0 to accept sign parameter","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-05-18T23:33:07Z","receivedAt":"2017-05-18T23:33:31Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 18 May 2017 12:37:30 -0700\nStefan Beller wrote:\n\n> Teach emit_line_0 take a \"sign\" parameter specifically intended\n> to hold the sign of the line instead of a separate \"first\" parameter\n> representing the first character of the line to be printed. Callers\n> that store the sign and line separately can use the \"sign\" parameter\n> like they used the \"first\" parameter previously, and callers that\n> store the sign and line together (or do not have a sign) no longer\n> need to manipulate their arguments to fit the requirements of\n> emit_line_0.\n\nI know I suggested the paragraph above, but after rereading your patch\nset, I think I finally understand what you're trying to accomplish.\nI think it's better to combine patches 4/20, 5/20, and 6/20, with the\nfollowing commit message:\n\n diff: introduce more flexible emit function\n\n Currently, diff output is written either through the emit_line_0\n function or through the FILE * in struct diff_options directly. To\n make it easier to teach diff to buffer its output (which will be done\n in a subsequent commit), introduce a more flexible emit() function. In\n this commit, direct usages of emit_line_0() are replaced with emit();\n subsequent commits will also replace usages of the FILE * with emit().\n\nAnd the function itself can be documented this way (with the appropriate\nformatting):\n\n /*\n Emits the following line or part of line. It is expected that \"set\"\n and \"reset\", if not NULL, should contain terminal color codes (or\n markup denoting color) and that \"sign\", if not 0, should contain '+',\n '-', or ' '. But those are not requirements.\n */\n\nIf you do all that, then the buffering patch (19/20) can be improved by\nadding this comment somewhere in the file:\n\n Buffer the diff output into ??? instead of immediately writing it to\n \"file\". \n\n NEEDSWORK: The contents of the ??? array - in particular, how the diff\n output is divided into array elements - is not precisely defined; some\n functions may emit a line all at once (resulting in one element)\n whereas some others may emit a line piecemeal (resulting in more than\n one element). Ideally, the code in this file should be structured so\n that we do not have such imprecision, but in the meantime, callers\n that request buffering should ensure that the diff output is divided\n the way they expect (and have tests to ensure that it remains so).\n\n> With this patch other callers hard code the sign (which are '+', '-',\n> ' ' and '\\\\') such that we do not run into unexpectedly emitting an\n> erroneous '\\0'.\n\nI still don't understand this paragraph - can you rewrite this in the\nimperative tense?\n \n> The audit of the caller revealed that the sign cannot be '\\n' or '\\r',\n> so remove that condition for trailing newline or carriage return in\n> the sign; the else part of the condition handles the len==0 perfectly,\n> so we can drop the if/else construct.\n> \n> Signed-off-by: Stefan Beller \n> ---\n> diff.c | 39 ++++++++++++++++-----------------------\n> 1 file changed, 16 insertions(+), 23 deletions(-)\n[snip]\n"},{"id":"320278","messageId":"20170519112330.731ecc66@twelve2.svl.corp.google.com","threadId":"45958","inReplyTo":"20170518193746.486-21-sbeller@google.com","subject":"Re: [PATCHv3 20/20] diff.c: color moved lines differently","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-05-19T18:23:30Z","receivedAt":"2017-05-19T18:26:53Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 18 May 2017 12:37:46 -0700\nStefan Beller wrote:\n\n[snip]\n\n> Instead this provides a dynamic programming greedy algorithm that\n\nNot sure if this is called \"dynamic programming\".\n\n> finds the largest moved hunk and then switches color to the\n> alternative color for the next hunk. By doing this any permutation is\n> recognized and displayed. That implies that there is no dedicated\n> boundary or inside-hunk color, but instead we'll have just two colors\n> alternating for hunks.\n\n[snip]\n\nI would title this \"color moved blocks differently\" to emphasize that we\nare treating the moves in terms of blocks, not just lines.\n\nThe first part of the commit message could probably be written more\nconcisely, like the following:\n\n When a patch consists mostly of moving blocks of code around, it can\n be quite tedious to ensure that the blocks are moved verbatim, and not\n undesirably modified in the move. To that end, color blocks that are\n moved within the same patch differently. For example (OM, del, add,\n and NM are different colors):\n\n [OM] -void sensitive_stuff(void)\n [OM] -{\n [OM] - if (!is_authorized_user())\n [OM] - die(\"unauthorized\");\n [OM] - sensitive_stuff(spanning,\n [OM] - multiple,\n [OM] - lines);\n [OM] -}\n\n void another_function()\n {\n [del] - printf(\"foo\");\n [add] + printf(\"bar\");\n }\n\n [NM] +void sensitive_stuff(void)\n [NM] +{\n [NM] + if (!is_authorized_user())\n [NM] + die(\"unauthorized\");\n [NM] + sensitive_stuff(spanning,\n [NM] + multiple,\n [NM] + lines);\n [NM] +}\n\n Adjacent blocks are colored differently. For example, in this\n potentially malicious patch, the swapping of blocks can be spotted:\n\n [OM] -void sensitive_stuff(void)\n [OM] -{\n [OMA] - if (!is_authorized_user())\n [OMA] - die(\"unauthorized\");\n [OM] - sensitive_stuff(spanning,\n [OM] - multiple,\n [OM] - lines);\n [OMA] -}\n\n void another_function()\n {\n [del] - printf(\"foo\");\n [add] + printf(\"bar\");\n }\n\n [NM] +void sensitive_stuff(void)\n [NM] +{\n [NMA] + sensitive_stuff(spanning,\n [NMA] + multiple,\n [NMA] + lines);\n [NM] + if (!is_authorized_user())\n [NM] + die(\"unauthorized\");\n [NMA] +}\n\nHaving said that, thanks - this version is much more like what I would\nexpect.\n\n> +static int buffered_patch_line_cmp_no_ws(const struct buffered_patch_line *a,\n> +\t\t\t\t\t const struct buffered_patch_line *b,\n> +\t\t\t\t\t const void *keydata)\n> +{\n> +\tint ret;\n> +\tstruct strbuf sba = STRBUF_INIT;\n> +\tstruct strbuf sbb = STRBUF_INIT;\n> +\n> +\tget_ws_cleaned_string(a, &sba);\n> +\tget_ws_cleaned_string(b, &sbb);\n> +\tret = sba.len != sbb.len || strncmp(sba.buf, sbb.buf, sba.len);\n> +\tstrbuf_release(&sba);\n> +\tstrbuf_release(&sbb);\n> +\treturn ret;\n> +}\n> +\n> +static int buffered_patch_line_cmp(const struct buffered_patch_line *a,\n> +\t\t\t\t const struct buffered_patch_line *b,\n> +\t\t\t\t const void *keydata)\n> +{\n> +\treturn a->len != b->len || strncmp(a->line, b->line, a->len);\n> +}\n\nInstead of having 2 versions of all the comparison functions, could the\nws-ness be passed as the keydata?\n\n> +static unsigned get_line_hash(struct buffered_patch_line *line, unsigned ignore_ws)\n> +{\n> +\tstatic struct strbuf sb = STRBUF_INIT;\n> +\n> +\tif (ignore_ws) {\n> +\t\tstrbuf_reset(&sb);\n> +\t\tget_ws_cleaned_string(line, &sb);\n\nMemory leak here, I think.\n\n> +\t\treturn memhash(sb.buf, sb.len);\n> +\t} else {\n> +\t\treturn memhash(line->line, line->len);\n> +\t}\n> +}\n\n[snip]\n\n> +static void add_lines_to_move_detection(struct diff_options *o)\n> +{\n> +\tstruct moved_entry *prev_line;\n\ngcc says (rightly) that this must be initialized.\n\n> +\n> +\tint n;\n> +\tfor (n = 0; n < o->line_buffer_nr; n++) {\n> +\t\tint sign = 0;\n> +\t\tstruct hashmap *hm;\n> +\t\tstruct moved_entry *key;\n> +\n> +\t\tswitch (o->line_buffer[n].sign) {\n> +\t\tcase '+':\n> +\t\t\tsign = '+';\n> +\t\t\thm = o->added_lines;\n> +\t\t\tbreak;\n> +\t\tcase '-':\n> +\t\t\tsign = '-';\n> +\t\t\thm = o->deleted_lines;\n> +\t\t\tbreak;\n> +\t\tcase ' ':\n> +\t\tdefault:\n> +\t\t\tprev_line = NULL;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tkey = prepare_entry(o, n);\n> +\t\tif (prev_line &&\n> +\t\t prev_line->line->sign == sign)\n> +\t\t\tprev_line->next_line = key;\n> +\n> +\t\thashmap_add(hm, key);\n> +\t\tprev_line = key;\n> +\t}\n> +}\n> +\n> +static void mark_color_as_moved(struct diff_options *o)\n> +{\n> +\tstruct moved_entry **pmb = NULL; /* potentially moved blocks */\n> +\tint pmb_nr = 0, pmb_alloc = 0;\n> +\tint alt_flag = 0;\n\nProbably call this \"use_alt_color\" or something similar.\n\n> +\tint n;\n> +\n> +\tfor (n = 0; n < o->line_buffer_nr; n++) {\n> +\t\tstruct hashmap *hm = NULL;\n> +\t\tstruct moved_entry *key;\n> +\t\tstruct moved_entry *match = NULL;\n> +\t\tstruct buffered_patch_line *l = &o->line_buffer[n];\n> +\t\tint i, lp, rp;\n> +\n> +\t\tswitch (l->sign) {\n> +\t\tcase '+':\n> +\t\t\thm = o->deleted_lines;\n> +\t\t\tbreak;\n> +\t\tcase '-':\n> +\t\t\thm = o->added_lines;\n> +\t\t\tbreak;\n> +\t\tdefault:\n> +\t\t\talt_flag = 0; /* reset to standard, no-alt move color */\n> +\t\t\tpmb_nr = 0; /* no running sets */\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\t/* Check for any match to color it as a move. */\n> +\t\tkey = prepare_entry(o, n);\n> +\t\tmatch = hashmap_get(hm, key, o);\n> +\t\tfree(key);\n> +\t\tif (!match)\n> +\t\t\tcontinue;\n> +\n> +\t\t/* Check any potential block runs, advance each or nullify */\n> +\t\tfor (i = 0; i < pmb_nr; i++) {\n> +\t\t\tstruct moved_entry *p = pmb[i];\n> +\t\t\tstruct moved_entry *pnext = (p && p->next_line) ?\n> +\t\t\t\t\tp->next_line : NULL;\n> +\t\t\tif (pnext &&\n> +\t\t\t !buffered_patch_line_cmp(pnext->line, l, o)) {\n> +\t\t\t\tpmb[i] = p->next_line;\n> +\t\t\t} else {\n> +\t\t\t\tpmb[i] = NULL;\n> +\t\t\t}\n\nMemory leak of pmb[i] somewhere here?\n\n> +\t\t}\n> +\n> +\t\t/* Shrink the set to the remaining runs */\n> +\t\tfor (lp = 0, rp = pmb_nr - 1; lp <= rp;) {\n> +\t\t\twhile (lp < pmb_nr && pmb[lp])\n> +\t\t\t\tlp ++;\n> +\t\t\t/* lp points at the first NULL now */\n> +\n> +\t\t\twhile (rp > -1 && !pmb[rp])\n> +\t\t\t\trp--;\n> +\t\t\t/* rp points at the last non-NULL */\n> +\n> +\t\t\tif (lp < pmb_nr && rp > -1 && lp < rp) {\n> +\t\t\t\tpmb[lp] = pmb[rp];\n> +\t\t\t\tpmb[rp] = NULL;\n> +\t\t\t\trp--;\n> +\t\t\t\tlp++;\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\tif (rp > -1) {\n> +\t\t\t/* Remember the number of running sets */\n> +\t\t\tpmb_nr = rp + 1;\n> +\t\t} else {\n> +\t\t\t/* Toggle color */\n> +\t\t\talt_flag = (alt_flag + 1) % 2;\n> +\n> +\t\t\t/* Build up a new set */\n> +\t\t\tpmb_nr = 0;\n> +\t\t\tfor (; match; match = hashmap_get_next(hm, match)) {\n> +\t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n> +\t\t\t\tpmb[pmb_nr++] = match;\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\tswitch (l->sign) {\n> +\t\tcase '+':\n> +\t\t\tl->set = diff_get_color_opt(o, DIFF_FILE_NEW_MOVED + alt_flag);\n> +\t\t\tbreak;\n> +\t\tcase '-':\n> +\t\t\tl->set = diff_get_color_opt(o, DIFF_FILE_OLD_MOVED + alt_flag);\n> +\t\t\tbreak;\n> +\t\tdefault:\n> +\t\t\tdie(\"BUG: we should have continued earlier?\");\n> +\t\t}\n> +\t}\n> +\tfree(pmb);\n> +}\n\n[snip]\n\n> @@ -4874,6 +5114,11 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n> \n> \tif (o->use_buffer) {\n> +\t\tif (o->color_moved) {\n\nCan you just declare the two hashmaps here, so that we do not need to\nput them in o? They don't seem to be used outside this block anyway.\n\n> +\t\t\tadd_lines_to_move_detection(o);\n> +\t\t\tmark_color_as_moved(o);\n> +\t\t}\n> +\n> \t\tfor (i = 0; i < o->line_buffer_nr; i++)\n> \t\t\temit_buffered_patch_line(o, &o->line_buffer[i]); \n\n[snip]\n\n> diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\n> index 289806d0c7..232d9ad55e 100755\n> --- a/t/t4015-diff-whitespace.sh\n> +++ b/t/t4015-diff-whitespace.sh\n\nAs for the tests, also add a test checking the interaction with\nwhitespace highlighting, and a test showing that diff errors out if we\nask for both move coloring and word-by-word diffing.\n"},{"id":"320281","messageId":"CAGZ79kYzgZeAnYsEODEYAEavvABS89oSq9Zj80=eS2ObtnMmNg@mail.gmail.com","threadId":"45958","inReplyTo":"20170519112330.731ecc66@twelve2.svl.corp.google.com","subject":"Re: [PATCHv3 20/20] diff.c: color moved lines differently","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-19T18:40:34Z","receivedAt":"2017-05-19T18:40:41Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, May 19, 2017 at 11:23 AM, Jonathan Tan wrote:\n> On Thu, 18 May 2017 12:37:46 -0700\n> Stefan Beller wrote:\n>\n> [snip]\n>\n>> Instead this provides a dynamic programming greedy algorithm that\n>\n> Not sure if this is called \"dynamic programming\".\n\nhttps://loveforprogramming.quora.com/Backtracking-Memoization-Dynamic-Programming\nhttp://stackoverflow.com/questions/3592943/difference-between-back-tracking-and-dynamic-programming\n\nInstead of doing backtracking (finding the lengthiest hunk for\neach line), we keep a set of potential hunks around, this sounds\nvery much like the examples given in these links.\n\n\n> The first part of the commit message could probably be written more\n> concisely, like the following:\n...\n> Having said that, thanks - this version is much more like what I would\n> expect.\n\nThanks for giving a more concise commit message, will fix in a reroll.\n\n\n>\n>> +static int buffered_patch_line_cmp_no_ws(const struct buffered_patch_line *a,\n>\n>> +static int buffered_patch_line_cmp(const struct buffered_patch_line *a,\n>\n> Instead of having 2 versions of all the comparison functions, could the\n> ws-ness be passed as the keydata?\n\nNo, this is misuse use of the API, peff explains:\n\nhttps://public-inbox.org/git/20170513085050.plmau5ffvzn6ibfp@sigill.intra.peff.net/\n\n\n\n>\n>> +static unsigned get_line_hash(struct buffered_patch_line *line, unsigned ignore_ws)\n>> +{\n>> + static struct strbuf sb = STRBUF_INIT;\n>> +\n>> + if (ignore_ws) {\n>> + strbuf_reset(&sb);\n>> + get_ws_cleaned_string(line, &sb);\n>\n> Memory leak here, I think.\n\nIt's static, so we don't care.\nI can make it non-static and release the memory in a resend.\n\n>\n>> + return memhash(sb.buf, sb.len);\n>> + } else {\n>> + return memhash(line->line, line->len);\n>> + }\n>> +}\n>\n> [snip]\n>\n>> +static void add_lines_to_move_detection(struct diff_options *o)\n>> +{\n>> + struct moved_entry *prev_line;\n>\n> gcc says (rightly) that this must be initialized.\n\nThis is one of the last refactorings I did on this patch, moving\nthe prev_line out of the diff_options struct (which is memset in its\ninit), forgot to init it here. will fix.\n\n>> + int alt_flag = 0;\n>\n> Probably call this \"use_alt_color\" or something similar.\n\nSounds better than alt_flag.\n\n>> + struct moved_entry *p = pmb[i];\n>> + struct moved_entry *pnext = (p && p->next_line) ?\n>> + p->next_line : NULL;\n>> + if (pnext &&\n>> + !buffered_patch_line_cmp(pnext->line, l, o)) {\n>> + pmb[i] = p->next_line;\n>> + } else {\n>> + pmb[i] = NULL;\n>> + }\n>\n> Memory leak of pmb[i] somewhere here?\n\npmb[] holds pointers into moved)entry elements that\nare obtained via hashmap_get_next(hm, match), such that\nany pmb[] element is also part of a hashmap.\n\nWhen freeing the hashmap, we'll free the memory. This\narray doesn't own the underlying memory.\n\n>> @@ -4874,6 +5114,11 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n>>\n>> if (o->use_buffer) {\n>> + if (o->color_moved) {\n>\n> Can you just declare the two hashmaps here, so that we do not need to\n> put them in o? They don't seem to be used outside this block anyway.\n\nObviously. Thanks for that pointer as well.\n\n\n>> diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\n>> index 289806d0c7..232d9ad55e 100755\n>> --- a/t/t4015-diff-whitespace.sh\n>> +++ b/t/t4015-diff-whitespace.sh\n>\n> As for the tests, also add a test checking the interaction with\n> whitespace highlighting, and a test showing that diff errors out if we\n> ask for both move coloring and word-by-word diffing.\n\nWe do not error out, but ignore the move heuristic doesn't find any\nblocks. I can make it error out, instead. (and add tests)\n\nThanks,\nStefan\n"},{"id":"320282","messageId":"CAGf8dgLRsHPdvxza6YJQXi4C8Q7QLw5A6LfFARSZ9639cGvb+w@mail.gmail.com","threadId":"45958","inReplyTo":"CAGZ79kYzgZeAnYsEODEYAEavvABS89oSq9Zj80=eS2ObtnMmNg@mail.gmail.com","subject":"Re: [PATCHv3 20/20] diff.c: color moved lines differently","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-05-19T19:34:57Z","receivedAt":"2017-05-19T19:35:05Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Fri, May 19, 2017 at 11:40 AM, Stefan Beller wrote:\n>>> +static unsigned get_line_hash(struct buffered_patch_line *line, unsigned ignore_ws)\n>>> +{\n>>> + static struct strbuf sb = STRBUF_INIT;\n>>> +\n>>> + if (ignore_ws) {\n>>> + strbuf_reset(&sb);\n>>> + get_ws_cleaned_string(line, &sb);\n>>\n>> Memory leak here, I think.\n>\n> It's static, so we don't care.\n> I can make it non-static and release the memory in a resend.\n\nAh, I missed the \"static\". It seems that \"static\" is used elsewhere\ntoo, so these functions are not reentrant anyway, so this is fine.\n"},{"id":"320293","messageId":"xmqqk25ct9lh.fsf@gitster.mtv.corp.google.com","threadId":"45958","inReplyTo":"CAGZ79kbXyjTaE=F7YU8yu8=eNzLS96sA5r5N4xrhmOfw+T4Lwg@mail.gmail.com","subject":"Re: [PATCHv2 12/20] submodule.c: convert show_submodule_summary to use emit_line_fmt","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-20T04:50:50Z","receivedAt":"2017-05-20T04:51:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> That could be added in ws.c:ws_check_emit, as these certain words\n> are similar to coloring whitespace.\n\nI actually was envisioning of highlighting a part of a line, like\n\n -Very poor SCM\n +Very nice SCM\n\nwhich would be done by finding semi-matching removed and added lines\nin the same hunk (i.e. local buffering) and makes a coloring decision.\nThat does not have any place in ws.c.\n\n>> Having said that, we need to start somewhere, and I think it is a\n>> reasonable first-cut attempt to work on top of the textual output\n>> like this series does (IOW, while I do agree with the NEEDSWORK and\n>> the way this series currently does things must be revamped in the\n>> longer term, I do not think we should wait until that happens to\n>> start playing with this topic).\n>\n> Ok. I share a similar reaction to submodule diffs that we discuss above\n> and word coloring, that Jonathan Tan brought up off list.\n>\n> Both of them are broken in this implementation, but the NEEDSWORK\n> would hint at how to fix them.\n\nYes, but if NEEDSWORK has to say \"the current hack is working at a\nwrong level, we need to do all of this before producing textual\ndiffs that are passed to the layer that colors lines\", that wouldn't\nhelp that much as a hint X-<.\n"},{"id":"320358","messageId":"CAGZ79kaHS2O90M8BXj_fEouSRgO+top7mkiawjqM7Xd=tUiX9g@mail.gmail.com","threadId":"45958","inReplyTo":"xmqqk25ct9lh.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCHv2 12/20] submodule.c: convert show_submodule_summary to use emit_line_fmt","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-20T22:00:13Z","receivedAt":"2017-05-20T22:00:21Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, May 19, 2017 at 9:50 PM, Junio C Hamano wrote:\n> Stefan Beller writes:\n>\n>> That could be added in ws.c:ws_check_emit, as these certain words\n>> are similar to coloring whitespace.\n>\n> I actually was envisioning of highlighting a part of a line, like\n>\n> -Very poor SCM\n> +Very nice SCM\n>\n> which would be done by finding semi-matching removed and added lines\n> in the same hunk (i.e. local buffering) and makes a coloring decision.\n> That does not have any place in ws.c.\n\nYes such a feature would not want to be in ws.c\n\nFor the problem above, we're still fine with the given data structures IMO.\nThough it may hint at bad naming of the struct 'buffered_patch_line'\nas it is not a complete line.\n\nAssuming the example above, running \"show --word-diff\" would produce\nthe following \"buffered_patch_lines\"\n\n{show_prefix=1, sign='\\0', line=\"-Very \", set=NULL, reset=NULL}\n{show_prefix=0, sign='\\0', line=\"{- poor}\", set='red', reset='normal'}\n{show_prefix=0, sign='\\0', line=\"{+ nice}\", set='green', reset='normal'}\n{show_prefix=0, sign='\\0', line=\" SCM\\n\", set=NULL, reset=NULL}\n\nso in the future, we may want to produce\n\n{show_prefix=1, sign='-', line=\"Very \", set=NULL, reset=NULL}\n{show_prefix=0, sign='\\0', line=\"poor\", set='red', reset='normal'}\n{show_prefix=0, sign='\\0', line=\" SCM\\n\", set=NULL, reset=NULL}\n{show_prefix=1, sign='+', line=\"Very \", set=NULL, reset=NULL}\n{show_prefix=0, sign='\\0', line=\"nice\", set='gren', reset='normal'}\n{show_prefix=0, sign='\\0', line=\" SCM\\n\", set=NULL, reset=NULL}\n\ninstead.\n\nI think the data structure can be used as-is, but is just mis-named.\nI'll fix that in a resend.\n\nThe algorithm to highlight what changed on a word level after a hunk\nwould have to be put into the current hunk finding\n(\"mark_color_as_moved\"), and then split up the actual lines into pieces\nof a line and add different colors like we see above.\n\n>\n>>> Having said that, we need to start somewhere, and I think it is a\n>>> reasonable first-cut attempt to work on top of the textual output\n>>> like this series does (IOW, while I do agree with the NEEDSWORK and\n>>> the way this series currently does things must be revamped in the\n>>> longer term, I do not think we should wait until that happens to\n>>> start playing with this topic).\n>>\n>> Ok. I share a similar reaction to submodule diffs that we discuss above\n>> and word coloring, that Jonathan Tan brought up off list.\n>>\n>> Both of them are broken in this implementation, but the NEEDSWORK\n>> would hint at how to fix them.\n>\n> Yes, but if NEEDSWORK has to say \"the current hack is working at a\n> wrong level, we need to do all of this before producing textual\n> diffs that are passed to the layer that colors lines\", that wouldn't\n> help that much as a hint X-<.\n\nFor the word coloring, I think we'd just need better post processing.\n\nFor cros-submodule move detection, we may want to wait in Brandons\nwork to be able to run submodule stuff in-process and then we\nhave the lines buffered directly there, so we can operate on them as well.\n\nI agree that \"NEEDSWORK: the current hack is working at a wrong level\"\nis not useful. But I thought we're in agreement that it is not? I must have\nmisunderstood parts of what you were saying.\n\nBy saying it could go to ws.c in my reply I rather meant:\nit could go into some post-processing function of the \"buffered_patch_line\"\nstruct. Currently we only have ws_emit_line that does it, but we can do\nit either similarly or just split up one buffered_patch_line into more\nthan one and color up each individually.\n\nThis would also be possible for us now, too. Instead of running it through\nws_emit_line we could split up the line and color each piece differently.\nHowever that could be a problem in the algorithm to find similar hunks\n(as we do have different rules for added and deleted text).\n\nThanks,\nStefan\n"},{"id":"320489","messageId":"CAGZ79kYQj-RECTiGkSV9PS5eD=6_s27CSpaF1tX6Ek34eajH=w@mail.gmail.com","threadId":"45958","inReplyTo":"20170518163307.657caedc@twelve2.svl.corp.google.com","subject":"Re: [PATCHv3 04/20] diff.c: teach emit_line_0 to accept sign parameter","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-22T23:36:34Z","receivedAt":"2017-05-22T23:36:58Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, May 18, 2017 at 4:33 PM, Jonathan Tan wrote:\n\n> I know I suggested the paragraph above, but after rereading your patch\n> set, I think I finally understand what you're trying to accomplish.\n> I think it's better to combine patches 4/20, 5/20, and 6/20, with the\n> following commit message:\n\nand patch 7/20 (diff.c: inline emit_line_0 into emit_line)\n\n> diff: introduce more flexible emit function\n>\n> Currently, diff output is written either through the emit_line_0\n> function or through the FILE * in struct diff_options directly. To\n> make it easier to teach diff to buffer its output (which will be done\n> in a subsequent commit), introduce a more flexible emit() function. In\n> this commit, direct usages of emit_line_0() are replaced with emit();\n> subsequent commits will also replace usages of the FILE * with emit().\n\nok. sounds reasonable to me. I kept them separate for easier review\noriginally, but I can certainly combine them again.\n\n>\n> And the function itself can be documented this way (with the appropriate\n> formatting):\n\nI have some documentation in 19/20 in diff.h for the data structure stored.\nWe'd want to discuss where to put the documentation. I'd not like to add\nsuch documentation to some static function in diff.c but rather only document\nthe data structure, and then (after in-lining <...>_0 to emit_line) keep the\nfunction documentation rather short as:\n\n/* see struct buffered_patch_line */\nstatic void emit_line(...lots of args...)\n{\n /* work on said data structure */\n ...\n\n\n> If you do all that, then the buffering patch (19/20) can be improved by\n> adding this comment somewhere in the file:\n>\n> Buffer the diff output into ??? instead of immediately writing it to\n> \"file\".\n>\n> NEEDSWORK: The contents of the ??? array - in particular, how the diff\n> output is divided into array elements - is not precisely defined; some\n> functions may emit a line all at once (resulting in one element)\n> whereas some others may emit a line piecemeal (resulting in more than\n> one element). Ideally, the code in this file should be structured so\n> that we do not have such imprecision, but in the meantime, callers\n> that request buffering should ensure that the diff output is divided\n> the way they expect (and have tests to ensure that it remains so).\n\nI want to rename that struct to \"buffered_diff_piece\" or\n\"partial_diff_output\", not implying we have a line or other another\nspecific piece.\n\nRethinking the discussion with Junio, what the correct abstraction level is,\nwe could even name it \"one_color_piece\", or \"colored_diff_part\", to\nimply we'd want to have as much content as possible in the same\ncolor. (And line endings always being in RESET color, we'd have a\nnatural separation at EOL)\n\n>\n>> With this patch other callers hard code the sign (which are '+', '-',\n>> ' ' and '\\\\') such that we do not run into unexpectedly emitting an\n>> erroneous '\\0'.\n>\n> I still don't understand this paragraph - can you rewrite this in the\n> imperative tense?\n\nAs in this patch the function signature is the same, but just changes\nmeaning of one of the arguments, one could imagine that there\ncould be a caller with first='\\0' given some interesting data/mode of operation,\nwhich would have instructed to literally output a '\\0'. We would not do this\nany more as the meaning changed.\n\nHowever this hypothetical subtle bug was not introduced, because\nthere are no callers that call the function with a 'first' depending on\nuser provided data or otherwise hard-coded expectation of a '\\0' output.\n\nMaybe I'll just drop this part.\n"},{"id":"320494","messageId":"20170523024048.16879-1-sbeller@google.com","threadId":"45958","inReplyTo":"20170518193746.486-1-sbeller@google.com","subject":"[PATCHv4 00/17] Diff machine: highlight moved lines.","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:31Z","receivedAt":"2017-05-23T02:41:10Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"v4:\n* interdiff to v3 (what is currently origin/sb/diff-color-move) below.\n* renamed the \"buffered_patch_line\" to \"diff_line\". Originally I planned\n to not carry the \"line\" part as it can be a piece of a line as well.\n But for the intended functionality it is best to keep the name.\n If we'd want to add more functionality to say have a move detection\n for words as well, we'd rename the struct to have a better name then.\n For now diff_line is the best. (Thanks Jonathan Nieder!)\n* tests to demonstrate it doesn't mess with --color-words as well as\n submodules. (Thanks Jonathan Tan!)\n* added in the statics (Thanks Ramsay!)\n* smaller scope for the hashmaps (Thanks Jonathan Tan!)\n* some commit messages were updated, prior patch 4-7 is squashed into one\n (Thanks Jonathan Tan!)\n* the tests added revealed an actual fault: now that the submodule process\n is not attached to a dupe of our stdout, it would stop coloring the\n output. We need to pass on use-color explicitly.\n* updated the NEEDSWORK comment in the second last patch.\n\nThanks for bearing,\nStefan\n\nv3:\n* see interdiff below.\n* fixing one invalid computation (Thanks Junio!)\n* I reasoned more about submodule and word diffing, see the commit message\n of the last patch:\n \n A note on the options '--submodule=diff' and '--color-words/--word-diff':\n In the conversion to use emit_line in the prior patches both submodules\n as well as word diff output carefully chose to call emit_line with sign=0.\n All output with sign=0 is ignored for move detection purposes in this\n patch, such that no weird looking output will be generated for these\n cases. This leads to another thought: We could pass on '--color-moved' to\n submodules such that they color up moved lines for themselves. If we'd do\n so only line moves within a repository boundary are marked up.\n\n* better name for emit_line outside of diff.[ch]\n\nv2:\n* emit_line now takes an argument that indicates if we want it\n to emit the line prefix as well. This should allow for a more faithful\n refactoring in the beginning. (Thanks Jonathan!)\n* fixed memleaks (Thanks Brandon!)\n* \"git -c color.moved=true log -p\" works now! (Thanks Jeff)\n* interdiff below, though it is large.\n* less intrusive than v1 (Thanks Jonathan!)\n\nv1:\n\nFor details on *why* see the commit message of the last commit.\n\nThe first five patches are slight refactorings to get into good\nshape, the next patches are funneling all output through emit_line_*.\n\nThe second last patch introduces an option to buffer up all output\nbefore printing, and then the last patch can color up moved lines\nof code.\n\nAny feedback welcome.\n\nThanks,\nStefan\n\nStefan Beller (17):\n diff: readability fix\n diff: move line ending check into emit_hunk_header\n diff.c: factor out diff_flush_patch_all_file_pairs\n diff: introduce more flexible emit function\n diff.c: convert fn_out_consume to use emit_line\n diff.c: convert builtin_diff to use emit_line_*\n diff.c: convert emit_rewrite_diff to use emit_line_*\n diff.c: convert emit_rewrite_lines to use emit_line_*\n submodule.c: convert show_submodule_summary to use emit_line_fmt\n diff.c: convert emit_binary_diff_body to use emit_line_*\n diff.c: convert show_stats to use emit_line_*\n diff.c: convert word diffing to use emit_line_*\n diff.c: convert diff_flush to use emit_line_*\n diff.c: convert diff_summary to use emit_line_*\n diff.c: emit_line includes whitespace highlighting\n diff: buffer all output if asked to\n diff.c: color moved lines differently\n\n Documentation/config.txt | 14 +-\n diff.c | 858 +++++++++++++++++++++++++++++++++------------\n diff.h | 59 +++-\n submodule.c | 89 ++---\n submodule.h | 9 +-\n t/t4015-diff-whitespace.sh | 267 ++++++++++++++\n 6 files changed, 1018 insertions(+), 278 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex b977a5d95b..23e70d348e 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -373,11 +373,11 @@ int git_diff_basic_config(const char *var, const char *value, void *cb)\n \n struct moved_entry {\n \tstruct hashmap_entry ent;\n-\tconst struct buffered_patch_line *line;\n+\tconst struct diff_line *line;\n \tstruct moved_entry *next_line;\n };\n \n-static void get_ws_cleaned_string(const struct buffered_patch_line *l,\n+static void get_ws_cleaned_string(const struct diff_line *l,\n \t\t\t\t struct strbuf *out)\n {\n \tint i;\n@@ -388,8 +388,8 @@ static void get_ws_cleaned_string(const struct buffered_patch_line *l,\n \t}\n }\n \n-static int buffered_patch_line_cmp_no_ws(const struct buffered_patch_line *a,\n-\t\t\t\t\t const struct buffered_patch_line *b,\n+static int diff_line_cmp_no_ws(const struct diff_line *a,\n+\t\t\t\t\t const struct diff_line *b,\n \t\t\t\t\t const void *keydata)\n {\n \tint ret;\n@@ -405,8 +405,8 @@ static int buffered_patch_line_cmp_no_ws(const struct buffered_patch_line *a,\n \treturn ret;\n }\n \n-static int buffered_patch_line_cmp(const struct buffered_patch_line *a,\n-\t\t\t\t const struct buffered_patch_line *b,\n+static int diff_line_cmp(const struct diff_line *a,\n+\t\t\t\t const struct diff_line *b,\n \t\t\t\t const void *keydata)\n {\n \treturn a->len != b->len || strncmp(a->line, b->line, a->len);\n@@ -416,17 +416,17 @@ static int moved_entry_cmp(const struct moved_entry *a,\n \t\t\t const struct moved_entry *b,\n \t\t\t const void *keydata)\n {\n-\treturn buffered_patch_line_cmp(a->line, b->line, keydata);\n+\treturn diff_line_cmp(a->line, b->line, keydata);\n }\n \n static int moved_entry_cmp_no_ws(const struct moved_entry *a,\n \t\t\t\t const struct moved_entry *b,\n \t\t\t\t const void *keydata)\n {\n-\treturn buffered_patch_line_cmp_no_ws(a->line, b->line, keydata);\n+\treturn diff_line_cmp_no_ws(a->line, b->line, keydata);\n }\n \n-static unsigned get_line_hash(struct buffered_patch_line *line, unsigned ignore_ws)\n+static unsigned get_line_hash(struct diff_line *line, unsigned ignore_ws)\n {\n \tstatic struct strbuf sb = STRBUF_INIT;\n \n@@ -444,7 +444,7 @@ static struct moved_entry *prepare_entry(struct diff_options *o,\n {\n \tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n \tunsigned ignore_ws = DIFF_XDL_TST(o, IGNORE_WHITESPACE);\n-\tstruct buffered_patch_line *l = &o->line_buffer[line_no];\n+\tstruct diff_line *l = &o->line_buffer[line_no];\n \n \tret->ent.hash = get_line_hash(l, ignore_ws);\n \tret->line = l;\n@@ -615,7 +615,9 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n-static void add_lines_to_move_detection(struct diff_options *o)\n+static void add_lines_to_move_detection(struct diff_options *o,\n+\t\t\t\t\tstruct hashmap *add_lines,\n+\t\t\t\t\tstruct hashmap *del_lines)\n {\n \tstruct moved_entry *prev_line = NULL;\n \n@@ -628,11 +630,11 @@ static void add_lines_to_move_detection(struct diff_options *o)\n \t\tswitch (o->line_buffer[n].sign) {\n \t\tcase '+':\n \t\t\tsign = '+';\n-\t\t\thm = o->added_lines;\n+\t\t\thm = add_lines;\n \t\t\tbreak;\n \t\tcase '-':\n \t\t\tsign = '-';\n-\t\t\thm = o->deleted_lines;\n+\t\t\thm = del_lines;\n \t\t\tbreak;\n \t\tcase ' ':\n \t\tdefault:\n@@ -650,29 +652,31 @@ static void add_lines_to_move_detection(struct diff_options *o)\n \t}\n }\n \n-static void mark_color_as_moved(struct diff_options *o)\n+static void mark_color_as_moved(struct diff_options *o,\n+\t\t\t\tstruct hashmap *add_lines,\n+\t\t\t\tstruct hashmap *del_lines)\n {\n \tstruct moved_entry **pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n-\tint alt_flag = 0;\n+\tint use_alt_color = 0;\n \tint n;\n \n \tfor (n = 0; n < o->line_buffer_nr; n++) {\n \t\tstruct hashmap *hm = NULL;\n \t\tstruct moved_entry *key;\n \t\tstruct moved_entry *match = NULL;\n-\t\tstruct buffered_patch_line *l = &o->line_buffer[n];\n+\t\tstruct diff_line *l = &o->line_buffer[n];\n \t\tint i, lp, rp;\n \n \t\tswitch (l->sign) {\n \t\tcase '+':\n-\t\t\thm = o->deleted_lines;\n+\t\t\thm = del_lines;\n \t\t\tbreak;\n \t\tcase '-':\n-\t\t\thm = o->added_lines;\n+\t\t\thm = add_lines;\n \t\t\tbreak;\n \t\tdefault:\n-\t\t\talt_flag = 0; /* reset to standard, no-alt move color */\n+\t\t\tuse_alt_color = 0;\n \t\t\tpmb_nr = 0; /* no running sets */\n \t\t\tcontinue;\n \t\t}\n@@ -690,7 +694,7 @@ static void mark_color_as_moved(struct diff_options *o)\n \t\t\tstruct moved_entry *pnext = (p && p->next_line) ?\n \t\t\t\t\tp->next_line : NULL;\n \t\t\tif (pnext &&\n-\t\t\t !buffered_patch_line_cmp(pnext->line, l, o)) {\n+\t\t\t !diff_line_cmp(pnext->line, l, o)) {\n \t\t\t\tpmb[i] = p->next_line;\n \t\t\t} else {\n \t\t\t\tpmb[i] = NULL;\n@@ -720,7 +724,7 @@ static void mark_color_as_moved(struct diff_options *o)\n \t\t\tpmb_nr = rp + 1;\n \t\t} else {\n \t\t\t/* Toggle color */\n-\t\t\talt_flag = (alt_flag + 1) % 2;\n+\t\t\tuse_alt_color = (use_alt_color + 1) % 2;\n \n \t\t\t/* Build up a new set */\n \t\t\tpmb_nr = 0;\n@@ -732,10 +736,12 @@ static void mark_color_as_moved(struct diff_options *o)\n \n \t\tswitch (l->sign) {\n \t\tcase '+':\n-\t\t\tl->set = diff_get_color_opt(o, DIFF_FILE_NEW_MOVED + alt_flag);\n+\t\t\tl->set = diff_get_color_opt(o,\n+\t\t\t\tDIFF_FILE_NEW_MOVED + use_alt_color);\n \t\t\tbreak;\n \t\tcase '-':\n-\t\t\tl->set = diff_get_color_opt(o, DIFF_FILE_OLD_MOVED + alt_flag);\n+\t\t\tl->set = diff_get_color_opt(o,\n+\t\t\t\tDIFF_FILE_OLD_MOVED + use_alt_color);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tdie(\"BUG: we should have continued earlier?\");\n@@ -744,8 +750,8 @@ static void mark_color_as_moved(struct diff_options *o)\n \tfree(pmb);\n }\n \n-static void emit_buffered_patch_line(struct diff_options *o,\n-\t\t\t\t struct buffered_patch_line *e)\n+static void emit_diff_line(struct diff_options *o,\n+\t\t\t\t struct diff_line *e)\n {\n \tconst char *ws;\n \tint has_trailing_newline, has_trailing_carriage_return;\n@@ -756,7 +762,7 @@ static void emit_buffered_patch_line(struct diff_options *o,\n \t\tfputs(diff_line_prefix(o), file);\n \n \tswitch (e->state) {\n-\tcase BPL_EMIT_LINE_WS:\n+\tcase DIFF_LINE_WS:\n \t\tws = diff_get_color(o->use_color, DIFF_WHITESPACE);\n \t\tif (e->set)\n \t\t\tfputs(e->set, file);\n@@ -767,7 +773,7 @@ static void emit_buffered_patch_line(struct diff_options *o,\n \t\tws_check_emit(e->line, e->len, o->ws_rule,\n \t\t\t file, e->set, e->reset, ws);\n \t\treturn;\n-\tcase BPL_EMIT_LINE_ASIS:\n+\tcase DIFF_LINE_ASIS:\n \t\thas_trailing_newline = (len > 0 && e->line[len-1] == '\\n');\n \t\tif (has_trailing_newline)\n \t\t\tlen--;\n@@ -789,46 +795,46 @@ static void emit_buffered_patch_line(struct diff_options *o,\n \t\tif (has_trailing_newline)\n \t\t\tfputc('\\n', file);\n \t\treturn;\n-\tcase BPL_HANDOVER:\n-\t\to->ws_rule = whitespace_rule(e->line); /*read from file, stored in line?*/\n+\tcase DIFF_LINE_RELOAD_WS_RULE:\n+\t\to->ws_rule = whitespace_rule(e->line);\n \t\treturn;\n \tdefault:\n \t\tdie(\"BUG: malformatted buffered patch line: '%d'\", e->state);\n \t}\n }\n \n-static void append_buffered_patch_line(struct diff_options *o,\n-\t\t\t\t struct buffered_patch_line *e)\n+static void append_diff_line(struct diff_options *o,\n+\t\t\t\t struct diff_line *e)\n {\n-\tstruct buffered_patch_line *f;\n+\tstruct diff_line *f;\n \tALLOC_GROW(o->line_buffer,\n \t\t o->line_buffer_nr + 1,\n \t\t o->line_buffer_alloc);\n \tf = &o->line_buffer[o->line_buffer_nr++];\n \n-\tmemcpy(f, e, sizeof(struct buffered_patch_line));\n+\tmemcpy(f, e, sizeof(struct diff_line));\n \tf->line = e->line ? xmemdupz(e->line, e->len) : NULL;\n }\n \n-void emit_line(struct diff_options *o,\n-\t const char *set, const char *reset,\n-\t int add_line_prefix, int markup_ws,\n-\t int sign, const char *line, int len)\n+static void emit_line(struct diff_options *o,\n+\t\t const char *set, const char *reset,\n+\t\t int add_line_prefix, int markup_ws,\n+\t\t int sign, const char *line, int len)\n {\n-\tstruct buffered_patch_line e = {set, reset, line,\n+\tstruct diff_line e = {set, reset, line,\n \t\tlen, sign, add_line_prefix,\n-\t\tmarkup_ws ? BPL_EMIT_LINE_WS : BPL_EMIT_LINE_ASIS};\n+\t\tmarkup_ws ? DIFF_LINE_WS : DIFF_LINE_ASIS};\n \n \tif (o->use_buffer)\n-\t\tappend_buffered_patch_line(o, &e);\n+\t\tappend_diff_line(o, &e);\n \telse\n-\t\temit_buffered_patch_line(o, &e);\n+\t\temit_diff_line(o, &e);\n }\n \n-void emit_line_fmt(struct diff_options *o,\n-\t\t const char *set, const char *reset,\n-\t\t int add_line_prefix,\n-\t\t const char *fmt, ...)\n+static void emit_line_fmt(struct diff_options *o,\n+\t\t\t const char *set, const char *reset,\n+\t\t\t int add_line_prefix,\n+\t\t\t const char *fmt, ...)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tva_list ap;\n@@ -1435,7 +1441,7 @@ static void diff_words_flush(struct emit_callback *ecbdata)\n \tif (ecbdata->diff_words->opt->line_buffer_nr) {\n \t\tint i;\n \t\tfor (i = 0; i < ecbdata->diff_words->opt->line_buffer_nr; i++)\n-\t\t\tappend_buffered_patch_line(ecbdata->opt,\n+\t\t\tappend_diff_line(ecbdata->opt,\n \t\t\t\t&ecbdata->diff_words->opt->line_buffer[i]);\n \n \t\tfor (i = 0; i < ecbdata->diff_words->opt->line_buffer_nr; i++)\n@@ -1862,8 +1868,8 @@ static void fill_print_name(struct diffstat_file *file)\n \tfile->print_name = pname;\n }\n \n-void print_stat_summary_0(struct diff_options *options, int files,\n-\t\t\t int insertions, int deletions)\n+static void print_stat_summary_0(struct diff_options *options, int files,\n+\t\t\t\t int insertions, int deletions)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \n@@ -2857,11 +2863,11 @@ static void builtin_diff(const char *name_a,\n \t\tif (o->word_diff)\n \t\t\tinit_diff_words_data(&ecbdata, o, one, two);\n \t\tif (o->use_buffer) {\n-\t\t\tstruct buffered_patch_line e = BUFFERED_PATCH_LINE_INIT;\n-\t\t\te.state = BPL_HANDOVER;\n+\t\t\tstruct diff_line e = diff_line_INIT;\n+\t\t\te.state = DIFF_LINE_RELOAD_WS_RULE;\n \t\t\te.line = name_b;\n \t\t\te.len = strlen(name_b);\n-\t\t\tappend_buffered_patch_line(o, &e);\n+\t\t\tappend_diff_line(o, &e);\n \t\t}\n \t\tif (xdi_diff_outf(&mf1, &mf2, fn_out_consume, &ecbdata,\n \t\t\t\t &xpp, &xecfg))\n@@ -5094,18 +5100,8 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \tint i;\n \tstruct diff_queue_struct *q = &diff_queued_diff;\n \n-\tif (o->color_moved) {\n-\t\tunsigned ignore_ws = DIFF_XDL_TST(o, IGNORE_WHITESPACE);\n+\tif (o->color_moved)\n \t\to->use_buffer = 1;\n-\t\to->deleted_lines = xmallocz(sizeof(*o->deleted_lines));\n-\t\to->added_lines = xmallocz(sizeof(*o->added_lines));\n-\t\thashmap_init(o->deleted_lines, ignore_ws ?\n-\t\t\t(hashmap_cmp_fn)moved_entry_cmp_no_ws :\n-\t\t\t(hashmap_cmp_fn)moved_entry_cmp, 0);\n-\t\thashmap_init(o->added_lines, ignore_ws ?\n-\t\t\t(hashmap_cmp_fn)moved_entry_cmp_no_ws :\n-\t\t\t(hashmap_cmp_fn)moved_entry_cmp, 0);\n-\t}\n \n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n@@ -5115,12 +5111,25 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \n \tif (o->use_buffer) {\n \t\tif (o->color_moved) {\n-\t\t\tadd_lines_to_move_detection(o);\n-\t\t\tmark_color_as_moved(o);\n+\t\t\tstruct hashmap add_lines, del_lines;\n+\t\t\tunsigned ignore_ws = DIFF_XDL_TST(o, IGNORE_WHITESPACE);\n+\n+\t\t\thashmap_init(&del_lines, ignore_ws ?\n+\t\t\t\t(hashmap_cmp_fn)moved_entry_cmp_no_ws :\n+\t\t\t\t(hashmap_cmp_fn)moved_entry_cmp, 0);\n+\t\t\thashmap_init(&add_lines, ignore_ws ?\n+\t\t\t\t(hashmap_cmp_fn)moved_entry_cmp_no_ws :\n+\t\t\t\t(hashmap_cmp_fn)moved_entry_cmp, 0);\n+\n+\t\t\tadd_lines_to_move_detection(o, &add_lines, &del_lines);\n+\t\t\tmark_color_as_moved(o, &add_lines, &del_lines);\n+\n+\t\t\thashmap_free(&add_lines, 0);\n+\t\t\thashmap_free(&del_lines, 0);\n \t\t}\n \n \t\tfor (i = 0; i < o->line_buffer_nr; i++)\n-\t\t\temit_buffered_patch_line(o, &o->line_buffer[i]);\n+\t\t\temit_diff_line(o, &o->line_buffer[i]);\n \n \t\tfor (i = 0; i < o->line_buffer_nr; i++)\n \t\t\tfree((void *)o->line_buffer[i].line);\ndiff --git a/diff.h b/diff.h\nindex 2d86e3a012..445259ebf7 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -123,11 +123,12 @@ enum diff_submodule_format {\n * into the pre/post image file. This pointer could be a union with the\n * line pointer. By storing an offset into the file instead of the literal line,\n * we can decrease the memory footprint for the buffered output. At first we\n- * may want to only have indirection for the content lines, but we could\n- * also have an enum (based on sign?) that stores prefabricated lines, e.g.\n- * the similarity score line or hunk/file headers.\n+ * may want to only have indirection for the content lines, but we could also\n+ * enhance the state for emitting prefabricated lines, e.g. the similarity\n+ * score line or hunk/file headers would only need to store a number or path\n+ * and then the output can be constructed later on depending on state.\n */\n-struct buffered_patch_line {\n+struct diff_line {\n \tconst char *set;\n \tconst char *reset;\n \tconst char *line;\n@@ -140,16 +141,16 @@ struct buffered_patch_line {\n \t\t * ws_check_emit which will output \"line\", marked up\n \t\t * according to ws_rule.\n \t\t */\n-\t\tBPL_EMIT_LINE_WS,\n+\t\tDIFF_LINE_WS,\n \n \t\t/* Emits [lineprefix][set][sign] line [reset] */\n-\t\tBPL_EMIT_LINE_ASIS,\n+\t\tDIFF_LINE_ASIS,\n \n \t\t/* Reloads the ws_rule; line contains the file name */\n-\t\tBPL_HANDOVER\n+\t\tDIFF_LINE_RELOAD_WS_RULE\n \t} state;\n };\n-#define BUFFERED_PATCH_LINE_INIT {NULL, NULL, NULL, 0, 0, 0}\n+#define diff_line_INIT {NULL, NULL, NULL, 0, 0, 0}\n \n struct diff_options {\n \tconst char *orderfile;\n@@ -226,14 +227,13 @@ struct diff_options {\n \tunsigned ws_rule;\n \tint use_buffer;\n \n-\tstruct buffered_patch_line *line_buffer;\n+\tstruct diff_line *line_buffer;\n \tint line_buffer_nr, line_buffer_alloc;\n \n \tint color_moved;\n-\tstruct hashmap *deleted_lines;\n-\tstruct hashmap *added_lines;\n };\n \n+/* Emit [line_prefix] [set] line [reset] */\n void diff_emit_line(struct diff_options *o, const char *set, const char *reset,\n \t\t const char *line, int len);\n \ndiff --git a/submodule.c b/submodule.c\nindex 19c63197fb..428c996c97 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -550,6 +550,8 @@ void show_submodule_inline_diff(struct diff_options *o, const char *path,\n \n \t/* TODO: other options may need to be passed here. */\n \targv_array_push(&cp.args, \"diff\");\n+\tif (o->use_color)\n+\t\targv_array_push(&cp.args, \"--color=always\");\n \targv_array_pushf(&cp.args, \"--line-prefix=%s\", diff_line_prefix(o));\n \tif (DIFF_OPT_TST(o, REVERSE_DIFF)) {\n \t\targv_array_pushf(&cp.args, \"--src-prefix=%s%s/\",\ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 232d9ad55e..0e92bf94bf 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1124,7 +1124,7 @@ test_expect_success 'detect moved code, inside file' '\n \ttest_cmp expected actual\n '\n \n-test_expect_success 'detect permutations inside moved code, ' '\n+test_expect_success 'detect permutations inside moved code' '\n \t# reusing the move example from last test:\n \tcat <<-\\EOF >main.c &&\n \t\t#include\n@@ -1201,4 +1201,42 @@ test_expect_success 'detect permutations inside moved code, ' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'move detection does not mess up colored words' '\n+\tcat <<-\\EOF >text.txt &&\n+\tLorem Ipsum is simply dummy text of the printing and typesetting industry.\n+\tEOF\n+\tgit add text.txt &&\n+\tgit commit -a -m \"clean state\" &&\n+\tcat <<-\\EOF >text.txt &&\n+\tsimply Lorem Ipsum dummy is text of the typesetting and printing industry.\n+\tEOF\n+\tgit diff --color-moved --word-diff >actual &&\n+\tgit diff --word-diff >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'move detection with submodules' '\n+\ttest_create_repo bananas &&\n+\techo ripe >bananas/recipe &&\n+\tgit -C bananas add recipe &&\n+\ttest_commit fruit &&\n+\ttest_commit -C bananas recipe &&\n+\tgit submodule add ./bananas &&\n+\tgit add bananas &&\n+\tgit commit -a -m \"bananas are like a heavy library?\" &&\n+\techo foul >bananas/recipe &&\n+\techo ripe >fruit.t &&\n+\n+\tgit diff --submodule=diff --color-moved >actual &&\n+\n+\t# no move detection as the moved line is across repository boundaries.\n+\ttest_decode_color decoded_actual &&\n+\t! grep BGREEN decoded_actual &&\n+\t! grep BRED decoded_actual &&\n+\n+\t# nor did we mess with it another way\n+\tgit diff --submodule=diff | test_decode_color >expect &&\n+\ttest_cmp expect decoded_actual\n+'\n+\n test_done\n\n\n"},{"id":"320495","messageId":"20170523024048.16879-2-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 01/17] diff: readability fix","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:32Z","receivedAt":"2017-05-23T02:41:12Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"We already have dereferenced 'p->two' into a local variable 'two'. Use\nthat.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n diff.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 74283d9001..3f5bf8b5a4 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -3283,8 +3283,8 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)\n \tconst char *other;\n \tconst char *attr_path;\n \n-\tname = p->one->path;\n-\tother = (strcmp(name, p->two->path) ? p->two->path : NULL);\n+\tname = one->path;\n+\tother = (strcmp(name, two->path) ? two->path : NULL);\n \tattr_path = name;\n \tif (o->prefix_length)\n \t\tstrip_prefix(o->prefix_length, &name, &other);\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320496","messageId":"20170523024048.16879-3-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 02/17] diff: move line ending check into emit_hunk_header","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:33Z","receivedAt":"2017-05-23T02:41:14Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"The emit_hunk_header() function is responsible for assembling a\nhunk header and calling emit_line() to send the hunk header\nto the output file. Its only caller fn_out_consume() needs\nto prepare for a case where the function emits an incomplete\nline and add the terminating LF.\n\nInstead make sure emit_hunk_header() to always send a\ncompleted line to emit_line().\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n diff.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 3f5bf8b5a4..c2ed605cd0 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -677,6 +677,8 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \t}\n \n \tstrbuf_add(&msgbuf, line + len, org_len - len);\n+\tstrbuf_complete_line(&msgbuf);\n+\n \temit_line(ecbdata->opt, \"\", \"\", msgbuf.buf, msgbuf.len);\n \tstrbuf_release(&msgbuf);\n }\n@@ -1315,8 +1317,6 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\tlen = sane_truncate_line(ecbdata, line, len);\n \t\tfind_lno(line, ecbdata);\n \t\temit_hunk_header(ecbdata, line, len);\n-\t\tif (line[len-1] != '\\n')\n-\t\t\tputc('\\n', o->file);\n \t\treturn;\n \t}\n \n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320497","messageId":"20170523024048.16879-6-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 05/17] diff.c: convert fn_out_consume to use emit_line","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:36Z","receivedAt":"2017-05-23T02:41:19Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line.\n\nThis covers the parts of fn_out_consume. In the next\npatches we'll convert more functions that want to emit\nformatted output, so we'd want to have a formatted emit\nfunction. Add it here.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n diff.c | 28 ++++++++++++++++++++--------\n 1 file changed, 20 insertions(+), 8 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 3569857818..8186289734 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -547,6 +547,21 @@ static void emit_line(struct diff_options *o, const char *set, const char *reset\n \t\tfputc('\\n', file);\n }\n \n+static void emit_line_fmt(struct diff_options *o,\n+\t\t\t const char *set, const char *reset,\n+\t\t\t int add_line_prefix,\n+\t\t\t const char *fmt, ...)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tva_list ap;\n+\tva_start(ap, fmt);\n+\tstrbuf_vaddf(&sb, fmt, ap);\n+\tva_end(ap);\n+\n+\temit_line(o, set, reset, add_line_prefix, 0, sb.buf, sb.len);\n+\tstrbuf_release(&sb);\n+}\n+\n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n {\n \tif (!((ecbdata->ws_rule & WS_BLANK_AT_EOF) &&\n@@ -1270,7 +1285,6 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \tconst char *context = diff_get_color(ecbdata->color_diff, DIFF_CONTEXT);\n \tconst char *reset = diff_get_color(ecbdata->color_diff, DIFF_RESET);\n \tstruct diff_options *o = ecbdata->opt;\n-\tconst char *line_prefix = diff_line_prefix(o);\n \n \to->found_changes = 1;\n \n@@ -1282,14 +1296,12 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \n \tif (ecbdata->label_path[0]) {\n \t\tconst char *name_a_tab, *name_b_tab;\n-\n \t\tname_a_tab = strchr(ecbdata->label_path[0], ' ') ? \"\\t\" : \"\";\n \t\tname_b_tab = strchr(ecbdata->label_path[1], ' ') ? \"\\t\" : \"\";\n-\n-\t\tfprintf(o->file, \"%s%s--- %s%s%s\\n\",\n-\t\t\tline_prefix, meta, ecbdata->label_path[0], reset, name_a_tab);\n-\t\tfprintf(o->file, \"%s%s+++ %s%s%s\\n\",\n-\t\t\tline_prefix, meta, ecbdata->label_path[1], reset, name_b_tab);\n+\t\temit_line_fmt(o, meta, reset, 1, \"--- %s%s\\n\",\n+\t\t\t ecbdata->label_path[0], name_a_tab);\n+\t\temit_line_fmt(o, meta, reset, 1, \"+++ %s%s\\n\",\n+\t\t\t ecbdata->label_path[1], name_b_tab);\n \t\tecbdata->label_path[0] = ecbdata->label_path[1] = NULL;\n \t}\n \n@@ -1330,7 +1342,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\tdiff_words_flush(ecbdata);\n \t\tif (ecbdata->diff_words->type == DIFF_WORDS_PORCELAIN) {\n \t\t\temit_line(o, context, reset, 1, 0, line, len);\n-\t\t\tfputs(\"~\\n\", o->file);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, \"~\\n\", 2);\n \t\t} else {\n \t\t\t/*\n \t\t\t * Skip the prefix character, if any. With\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320498","messageId":"20170523024048.16879-7-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 06/17] diff.c: convert builtin_diff to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:37Z","receivedAt":"2017-05-23T02:41:20Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers builtin_diff.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n diff.c | 33 ++++++++++++++++++++-------------\n 1 file changed, 20 insertions(+), 13 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 8186289734..4fa976d43c 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1289,8 +1289,9 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \to->found_changes = 1;\n \n \tif (ecbdata->header) {\n-\t\tfprintf(o->file, \"%s\", ecbdata->header->buf);\n-\t\tstrbuf_reset(ecbdata->header);\n+\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t ecbdata->header->buf, ecbdata->header->len);\n+\t\tstrbuf_release(ecbdata->header);\n \t\tecbdata->header = NULL;\n \t}\n \n@@ -2435,7 +2436,7 @@ static void builtin_diff(const char *name_a,\n \t\tif (complete_rewrite &&\n \t\t (textconv_one || !diff_filespec_is_binary(one)) &&\n \t\t (textconv_two || !diff_filespec_is_binary(two))) {\n-\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t\temit_rewrite_diff(name_a, name_b, one, two,\n \t\t\t\t\t\ttextconv_one, textconv_two, o);\n@@ -2445,7 +2446,7 @@ static void builtin_diff(const char *name_a,\n \t}\n \n \tif (o->irreversible_delete && lbl[1][0] == '/') {\n-\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tgoto free_ab_and_return;\n \t} else if (!DIFF_OPT_TST(o, TEXT) &&\n@@ -2456,12 +2457,15 @@ static void builtin_diff(const char *name_a,\n \t\t !DIFF_OPT_TST(o, BINARY)) {\n \t\t\tif (!oidcmp(&one->oid, &two->oid)) {\n \t\t\t\tif (must_show_header)\n-\t\t\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\t\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t\t\t\t header.buf, header.len);\n \t\t\t\tgoto free_ab_and_return;\n \t\t\t}\n-\t\t\tfprintf(o->file, \"%s\", header.buf);\n-\t\t\tfprintf(o->file, \"%sBinary files %s and %s differ\\n\",\n-\t\t\t\tline_prefix, lbl[0], lbl[1]);\n+\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t\t header.buf, header.len);\n+\t\t\temit_line_fmt(o, NULL, NULL, 1,\n+\t\t\t\t \"Binary files %s and %s differ\\n\",\n+\t\t\t\t lbl[0], lbl[1]);\n \t\t\tgoto free_ab_and_return;\n \t\t}\n \t\tif (fill_mmfile(&mf1, one) < 0 || fill_mmfile(&mf2, two) < 0)\n@@ -2470,16 +2474,19 @@ static void builtin_diff(const char *name_a,\n \t\tif (mf1.size == mf2.size &&\n \t\t !memcmp(mf1.ptr, mf2.ptr, mf1.size)) {\n \t\t\tif (must_show_header)\n-\t\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t\t\t header.buf, header.len);\n \t\t\tgoto free_ab_and_return;\n \t\t}\n-\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tif (DIFF_OPT_TST(o, BINARY))\n \t\t\temit_binary_diff(o->file, &mf1, &mf2, line_prefix);\n \t\telse\n-\t\t\tfprintf(o->file, \"%sBinary files %s and %s differ\\n\",\n-\t\t\t\tline_prefix, lbl[0], lbl[1]);\n+\t\t\temit_line_fmt(o, NULL, NULL, 1,\n+\t\t\t\t \"Binary files %s and %s differ\\n\",\n+\t\t\t\t lbl[0], lbl[1]);\n \t\to->found_changes = 1;\n \t} else {\n \t\t/* Crazy xdl interfaces.. */\n@@ -2491,7 +2498,7 @@ static void builtin_diff(const char *name_a,\n \t\tconst struct userdiff_funcname *pe;\n \n \t\tif (must_show_header) {\n-\t\t\tfprintf(o->file, \"%s\", header.buf);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t}\n \n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320499","messageId":"20170523024048.16879-8-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 07/17] diff.c: convert emit_rewrite_diff to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:38Z","receivedAt":"2017-05-23T02:41:24Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers emit_rewrite_diff.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n diff.c | 33 ++++++++++++++++++---------------\n 1 file changed, 18 insertions(+), 15 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 4fa976d43c..3dda9f3c8e 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -704,17 +704,17 @@ static void remove_tempfile(void)\n \t}\n }\n \n-static void print_line_count(FILE *file, int count)\n+static void add_line_count(struct strbuf *out, int count)\n {\n \tswitch (count) {\n \tcase 0:\n-\t\tfprintf(file, \"0,0\");\n+\t\tstrbuf_addstr(out, \"0,0\");\n \t\tbreak;\n \tcase 1:\n-\t\tfprintf(file, \"1\");\n+\t\tstrbuf_addstr(out, \"1\");\n \t\tbreak;\n \tdefault:\n-\t\tfprintf(file, \"1,%d\", count);\n+\t\tstrbuf_addf(out, \"1,%d\", count);\n \t\tbreak;\n \t}\n }\n@@ -768,7 +768,7 @@ static void emit_rewrite_diff(const char *name_a,\n \tchar *data_one, *data_two;\n \tsize_t size_one, size_two;\n \tstruct emit_callback ecbdata;\n-\tconst char *line_prefix = diff_line_prefix(o);\n+\tstruct strbuf out = STRBUF_INIT;\n \n \tif (diff_mnemonic_prefix && DIFF_OPT_TST(o, REVERSE_DIFF)) {\n \t\ta_prefix = o->b_prefix;\n@@ -806,20 +806,23 @@ static void emit_rewrite_diff(const char *name_a,\n \tecbdata.lno_in_preimage = 1;\n \tecbdata.lno_in_postimage = 1;\n \n+\temit_line_fmt(o, metainfo, reset, 1, \"--- %s%s\\n\", a_name.buf, name_a_tab);\n+\temit_line_fmt(o, metainfo, reset, 1, \"+++ %s%s\\n\", b_name.buf, name_b_tab);\n+\n \tlc_a = count_lines(data_one, size_one);\n \tlc_b = count_lines(data_two, size_two);\n-\tfprintf(o->file,\n-\t\t\"%s%s--- %s%s%s\\n%s%s+++ %s%s%s\\n%s%s@@ -\",\n-\t\tline_prefix, metainfo, a_name.buf, name_a_tab, reset,\n-\t\tline_prefix, metainfo, b_name.buf, name_b_tab, reset,\n-\t\tline_prefix, fraginfo);\n+\n+\tstrbuf_addstr(&out, \"@@ -\");\n \tif (!o->irreversible_delete)\n-\t\tprint_line_count(o->file, lc_a);\n+\t\tadd_line_count(&out, lc_a);\n \telse\n-\t\tfprintf(o->file, \"?,?\");\n-\tfprintf(o->file, \" +\");\n-\tprint_line_count(o->file, lc_b);\n-\tfprintf(o->file, \" @@%s\\n\", reset);\n+\t\tstrbuf_addstr(&out, \"?,?\");\n+\tstrbuf_addstr(&out, \" +\");\n+\tadd_line_count(&out, lc_b);\n+\tstrbuf_addstr(&out, \" @@\\n\");\n+\temit_line(o, fraginfo, reset, 1, 0, out.buf, out.len);\n+\tstrbuf_release(&out);\n+\n \tif (lc_a && !o->irreversible_delete)\n \t\temit_rewrite_lines(&ecbdata, '-', data_one, size_one);\n \tif (lc_b)\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320500","messageId":"20170523024048.16879-9-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 08/17] diff.c: convert emit_rewrite_lines to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:39Z","receivedAt":"2017-05-23T02:41:25Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers emit_rewrite_lines.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n diff.c | 25 +++++++++++++++++--------\n 1 file changed, 17 insertions(+), 8 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 3dda9f3c8e..ca6b48cf49 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -722,15 +722,23 @@ static void add_line_count(struct strbuf *out, int count)\n static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\t\t int prefix, const char *data, int size)\n {\n-\tconst char *endp = NULL;\n-\tstatic const char *nneof = \" No newline at end of file\\n\";\n \tconst char *reset = diff_get_color(ecb->color_diff, DIFF_RESET);\n+\tstruct strbuf sb = STRBUF_INIT;\n \n \twhile (0 < size) {\n \t\tint len;\n \n-\t\tendp = memchr(data, '\\n', size);\n-\t\tlen = endp ? (endp - data + 1) : size;\n+\t\tconst char *endp = memchr(data, '\\n', size);\n+\t\tif (endp)\n+\t\t\tlen = endp - data + 1;\n+\t\telse {\n+\t\t\tstrbuf_add(&sb, data, size);\n+\t\t\tstrbuf_addch(&sb, '\\n');\n+\t\t\tsize = 0; /* to exit the loop. */\n+\n+\t\t\tdata = sb.buf;\n+\t\t\tlen = sb.len;\n+\t\t}\n \t\tif (prefix != '+') {\n \t\t\tecb->lno_in_preimage++;\n \t\t\temit_del_line(reset, ecb, data, len);\n@@ -741,12 +749,13 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\tsize -= len;\n \t\tdata += len;\n \t}\n-\tif (!endp) {\n+\tif (sb.len) {\n+\t\tstatic const char *nneof = \"\\\\ No newline at end of file\\n\";\n \t\tconst char *context = diff_get_color(ecb->color_diff,\n \t\t\t\t\t\t DIFF_CONTEXT);\n-\t\tputc('\\n', ecb->opt->file);\n-\t\temit_line(ecb->opt, context, reset, 1, '\\\\',\n-\t\t\t nneof, strlen(nneof));\n+\t\temit_line(ecb->opt, context, reset, 1, 0,\n+\t\t\t nneof, strlen(nneof));\n+\t\tstrbuf_release(&sb);\n \t}\n }\n \n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320501","messageId":"20170523024048.16879-10-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 09/17] submodule.c: convert show_submodule_summary to use emit_line_fmt","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:40Z","receivedAt":"2017-05-23T02:41:29Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis prepares the code for submodules to go through the\nemit_line function.\n\nAs the submodule process is no longer attached to the\nsame stdout as the superprojects process we need to\npass on the usage of colors explicitly.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 14 ++++++----\n diff.h | 3 +++\n submodule.c | 89 +++++++++++++++++++++++++++++++++----------------------------\n submodule.h | 9 +++----\n 4 files changed, 63 insertions(+), 52 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex ca6b48cf49..3357c0fca3 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -562,6 +562,12 @@ static void emit_line_fmt(struct diff_options *o,\n \tstrbuf_release(&sb);\n }\n \n+void diff_emit_line(struct diff_options *o, const char *set, const char *reset,\n+\t\t const char *line, int len)\n+{\n+\temit_line(o, set, reset, 1, 0, line, len);\n+}\n+\n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n {\n \tif (!((ecbdata->ws_rule & WS_BLANK_AT_EOF) &&\n@@ -2384,8 +2390,7 @@ static void builtin_diff(const char *name_a,\n \t (!two->mode || S_ISGITLINK(two->mode))) {\n \t\tconst char *del = diff_get_color_opt(o, DIFF_FILE_OLD);\n \t\tconst char *add = diff_get_color_opt(o, DIFF_FILE_NEW);\n-\t\tshow_submodule_summary(o->file, one->path ? one->path : two->path,\n-\t\t\t\tline_prefix,\n+\t\tshow_submodule_summary(o, one->path ? one->path : two->path,\n \t\t\t\t&one->oid, &two->oid,\n \t\t\t\ttwo->dirty_submodule,\n \t\t\t\tmeta, del, add, reset);\n@@ -2395,11 +2400,10 @@ static void builtin_diff(const char *name_a,\n \t\t (!two->mode || S_ISGITLINK(two->mode))) {\n \t\tconst char *del = diff_get_color_opt(o, DIFF_FILE_OLD);\n \t\tconst char *add = diff_get_color_opt(o, DIFF_FILE_NEW);\n-\t\tshow_submodule_inline_diff(o->file, one->path ? one->path : two->path,\n-\t\t\t\tline_prefix,\n+\t\tshow_submodule_inline_diff(o, one->path ? one->path : two->path,\n \t\t\t\t&one->oid, &two->oid,\n \t\t\t\ttwo->dirty_submodule,\n-\t\t\t\tmeta, del, add, reset, o);\n+\t\t\t\tmeta, del, add, reset);\n \t\treturn;\n \t}\n \ndiff --git a/diff.h b/diff.h\nindex 5be1ee77a7..9ad546361a 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -188,6 +188,9 @@ struct diff_options {\n \tint diff_path_counter;\n };\n \n+void diff_emit_line(struct diff_options *o, const char *set, const char *reset,\n+\t\t const char *line, int len);\n+\n enum color_diff {\n \tDIFF_RESET = 0,\n \tDIFF_CONTEXT = 1,\ndiff --git a/submodule.c b/submodule.c\nindex d3299e29c0..428c996c97 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -362,8 +362,8 @@ static int prepare_submodule_summary(struct rev_info *rev, const char *path,\n \treturn prepare_revision_walk(rev);\n }\n \n-static void print_submodule_summary(struct rev_info *rev, FILE *f,\n-\t\tconst char *line_prefix,\n+static void print_submodule_summary(struct rev_info *rev,\n+\t\tstruct diff_options *o,\n \t\tconst char *del, const char *add, const char *reset)\n {\n \tstatic const char format[] = \" %m %s\";\n@@ -375,18 +375,12 @@ static void print_submodule_summary(struct rev_info *rev, FILE *f,\n \t\tctx.date_mode = rev->date_mode;\n \t\tctx.output_encoding = get_log_output_encoding();\n \t\tstrbuf_setlen(&sb, 0);\n-\t\tstrbuf_addstr(&sb, line_prefix);\n-\t\tif (commit->object.flags & SYMMETRIC_LEFT) {\n-\t\t\tif (del)\n-\t\t\t\tstrbuf_addstr(&sb, del);\n-\t\t}\n-\t\telse if (add)\n-\t\t\tstrbuf_addstr(&sb, add);\n \t\tformat_commit_message(commit, format, &sb, &ctx);\n-\t\tif (reset)\n-\t\t\tstrbuf_addstr(&sb, reset);\n \t\tstrbuf_addch(&sb, '\\n');\n-\t\tfprintf(f, \"%s\", sb.buf);\n+\t\tif (commit->object.flags & SYMMETRIC_LEFT)\n+\t\t\tdiff_emit_line(o, del, reset, sb.buf, sb.len);\n+\t\telse if (add)\n+\t\t\tdiff_emit_line(o, add, reset, sb.buf, sb.len);\n \t}\n \tstrbuf_release(&sb);\n }\n@@ -413,8 +407,7 @@ void prepare_submodule_repo_env(struct argv_array *out)\n * attempt to lookup both the left and right commits and put them into the\n * left and right pointers.\n */\n-static void show_submodule_header(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+static void show_submodule_header(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n \t\tconst char *reset,\n@@ -425,12 +418,17 @@ static void show_submodule_header(FILE *f, const char *path,\n \tstruct strbuf sb = STRBUF_INIT;\n \tint fast_forward = 0, fast_backward = 0;\n \n-\tif (dirty_submodule & DIRTY_SUBMODULE_UNTRACKED)\n-\t\tfprintf(f, \"%sSubmodule %s contains untracked content\\n\",\n-\t\t\tline_prefix, path);\n-\tif (dirty_submodule & DIRTY_SUBMODULE_MODIFIED)\n-\t\tfprintf(f, \"%sSubmodule %s contains modified content\\n\",\n-\t\t\tline_prefix, path);\n+\tif (dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) {\n+\t\tstrbuf_addf(&sb, \"Submodule %s contains untracked content\\n\", path);\n+\t\tdiff_emit_line(o, NULL, NULL, sb.buf, sb.len);\n+\t\tstrbuf_reset(&sb);\n+\t}\n+\n+\tif (dirty_submodule & DIRTY_SUBMODULE_MODIFIED) {\n+\t\tstrbuf_addf(&sb, \"Submodule %s contains modified content\\n\", path);\n+\t\tdiff_emit_line(o, NULL, NULL, sb.buf, sb.len);\n+\t\tstrbuf_reset(&sb);\n+\t}\n \n \tif (is_null_oid(one))\n \t\tmessage = \"(new submodule)\";\n@@ -472,21 +470,20 @@ static void show_submodule_header(FILE *f, const char *path,\n \t}\n \n output_header:\n-\tstrbuf_addf(&sb, \"%s%sSubmodule %s \", line_prefix, meta, path);\n+\tstrbuf_addf(&sb, \"Submodule %s \", path);\n \tstrbuf_add_unique_abbrev(&sb, one->hash, DEFAULT_ABBREV);\n \tstrbuf_addstr(&sb, (fast_backward || fast_forward) ? \"..\" : \"...\");\n \tstrbuf_add_unique_abbrev(&sb, two->hash, DEFAULT_ABBREV);\n \tif (message)\n-\t\tstrbuf_addf(&sb, \" %s%s\\n\", message, reset);\n+\t\tstrbuf_addf(&sb, \" %s\\n\", message);\n \telse\n-\t\tstrbuf_addf(&sb, \"%s:%s\\n\", fast_backward ? \" (rewind)\" : \"\", reset);\n-\tfwrite(sb.buf, sb.len, 1, f);\n+\t\tstrbuf_addf(&sb, \"%s:\\n\", fast_backward ? \" (rewind)\" : \"\");\n+\tdiff_emit_line(o, meta, reset, sb.buf, sb.len);\n \n \tstrbuf_release(&sb);\n }\n \n-void show_submodule_summary(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+void show_submodule_summary(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n \t\tconst char *del, const char *add, const char *reset)\n@@ -495,7 +492,7 @@ void show_submodule_summary(FILE *f, const char *path,\n \tstruct commit *left = NULL, *right = NULL;\n \tstruct commit_list *merge_bases = NULL;\n \n-\tshow_submodule_header(f, path, line_prefix, one, two, dirty_submodule,\n+\tshow_submodule_header(o, path, one, two, dirty_submodule,\n \t\t\t meta, reset, &left, &right, &merge_bases);\n \n \t/*\n@@ -508,11 +505,12 @@ void show_submodule_summary(FILE *f, const char *path,\n \n \t/* Treat revision walker failure the same as missing commits */\n \tif (prepare_submodule_summary(&rev, path, left, right, merge_bases)) {\n-\t\tfprintf(f, \"%s(revision walker failed)\\n\", line_prefix);\n+\t\tconst char *error = \"(revision walker failed)\\n\";\n+\t\tdiff_emit_line(o, NULL, NULL, error, strlen(error));\n \t\tgoto out;\n \t}\n \n-\tprint_submodule_summary(&rev, f, line_prefix, del, add, reset);\n+\tprint_submodule_summary(&rev, o, del, add, reset);\n \n out:\n \tif (merge_bases)\n@@ -521,20 +519,18 @@ void show_submodule_summary(FILE *f, const char *path,\n \tclear_commit_marks(right, ~0);\n }\n \n-void show_submodule_inline_diff(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+void show_submodule_inline_diff(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n-\t\tconst char *del, const char *add, const char *reset,\n-\t\tconst struct diff_options *o)\n+\t\tconst char *del, const char *add, const char *reset)\n {\n \tconst struct object_id *old = &empty_tree_oid, *new = &empty_tree_oid;\n \tstruct commit *left = NULL, *right = NULL;\n \tstruct commit_list *merge_bases = NULL;\n-\tstruct strbuf submodule_dir = STRBUF_INIT;\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tstruct strbuf sb = STRBUF_INIT;\n \n-\tshow_submodule_header(f, path, line_prefix, one, two, dirty_submodule,\n+\tshow_submodule_header(o, path, one, two, dirty_submodule,\n \t\t\t meta, reset, &left, &right, &merge_bases);\n \n \t/* We need a valid left and right commit to display a difference */\n@@ -547,15 +543,16 @@ void show_submodule_inline_diff(FILE *f, const char *path,\n \tif (right)\n \t\tnew = two;\n \n-\tfflush(f);\n \tcp.git_cmd = 1;\n \tcp.dir = path;\n-\tcp.out = dup(fileno(f));\n+\tcp.out = -1;\n \tcp.no_stdin = 1;\n \n \t/* TODO: other options may need to be passed here. */\n \targv_array_push(&cp.args, \"diff\");\n-\targv_array_pushf(&cp.args, \"--line-prefix=%s\", line_prefix);\n+\tif (o->use_color)\n+\t\targv_array_push(&cp.args, \"--color=always\");\n+\targv_array_pushf(&cp.args, \"--line-prefix=%s\", diff_line_prefix(o));\n \tif (DIFF_OPT_TST(o, REVERSE_DIFF)) {\n \t\targv_array_pushf(&cp.args, \"--src-prefix=%s%s/\",\n \t\t\t\t o->b_prefix, path);\n@@ -578,11 +575,21 @@ void show_submodule_inline_diff(FILE *f, const char *path,\n \t\targv_array_push(&cp.args, oid_to_hex(new));\n \n \tprepare_submodule_repo_env(&cp.env_array);\n-\tif (run_command(&cp))\n-\t\tfprintf(f, \"(diff failed)\\n\");\n+\tif (start_command(&cp)) {\n+\t\tconst char *error = \"(diff failed)\\n\";\n+\t\tdiff_emit_line(o, NULL, NULL, error, strlen(error));\n+\t}\n+\n+\twhile (strbuf_getwholeline_fd(&sb, cp.out, '\\n') != EOF)\n+\t\tdiff_emit_line(o, NULL, NULL, sb.buf, sb.len);\n+\n+\tif (finish_command(&cp)) {\n+\t\tconst char *error = \"(diff failed)\\n\";\n+\t\tdiff_emit_line(o, NULL, NULL, error, strlen(error));\n+\t}\n \n done:\n-\tstrbuf_release(&submodule_dir);\n+\tstrbuf_release(&sb);\n \tif (merge_bases)\n \t\tfree_commit_list(merge_bases);\n \tif (left)\ndiff --git a/submodule.h b/submodule.h\nindex 1277480add..9df0a3aea2 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -53,17 +53,14 @@ extern int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst);\n extern const char *submodule_strategy_to_string(const struct submodule_update_strategy *s);\n extern void handle_ignore_submodules_arg(struct diff_options *, const char *);\n-extern void show_submodule_summary(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+extern void show_submodule_summary(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n \t\tconst char *del, const char *add, const char *reset);\n-extern void show_submodule_inline_diff(FILE *f, const char *path,\n-\t\tconst char *line_prefix,\n+extern void show_submodule_inline_diff(struct diff_options *o, const char *path,\n \t\tstruct object_id *one, struct object_id *two,\n \t\tunsigned dirty_submodule, const char *meta,\n-\t\tconst char *del, const char *add, const char *reset,\n-\t\tconst struct diff_options *opt);\n+\t\tconst char *del, const char *add, const char *reset);\n extern void set_config_fetch_recurse_submodules(int value);\n extern void set_config_update_recurse_submodules(int value);\n /* Check if we want to update any submodule.*/\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320502","messageId":"20170523024048.16879-12-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 11/17] diff.c: convert show_stats to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:42Z","receivedAt":"2017-05-23T02:41:32Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nWe call print_stat_summary from builtin/apply, so we still\nneed the version with a file pointer, so introduce\nprint_stat_summary_0 that uses emit_line_* machinery and\nkeep print_stat_summary with the same arguments around.\n\nThe responsibility to print the line prefix moves from the callers\nof print_stat_summary_0 into the function itself.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n diff.c | 89 ++++++++++++++++++++++++++++++++++++++----------------------------\n diff.h | 4 +--\n 2 files changed, 53 insertions(+), 40 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex b5a5261a4e..a76abf5f69 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1540,20 +1540,19 @@ static int scale_linear(int it, int width, int max_change)\n \treturn 1 + (it * (width - 1) / max_change);\n }\n \n-static void show_name(FILE *file,\n+static void show_name(struct strbuf *out,\n \t\t const char *prefix, const char *name, int len)\n {\n-\tfprintf(file, \" %s%-*s |\", prefix, len, name);\n+\tstrbuf_addf(out, \" %s%-*s |\", prefix, len, name);\n }\n \n-static void show_graph(FILE *file, char ch, int cnt, const char *set, const char *reset)\n+static void show_graph(struct strbuf *out, char ch, int cnt, const char *set, const char *reset)\n {\n \tif (cnt <= 0)\n \t\treturn;\n-\tfprintf(file, \"%s\", set);\n-\twhile (cnt--)\n-\t\tputc(ch, file);\n-\tfprintf(file, \"%s\", reset);\n+\tstrbuf_addstr(out, set);\n+\tstrbuf_addchars(out, ch, cnt);\n+\tstrbuf_addstr(out, reset);\n }\n \n static void fill_print_name(struct diffstat_file *file)\n@@ -1577,14 +1576,16 @@ static void fill_print_name(struct diffstat_file *file)\n \tfile->print_name = pname;\n }\n \n-int print_stat_summary(FILE *fp, int files, int insertions, int deletions)\n+static void print_stat_summary_0(struct diff_options *options, int files,\n+\t\t\t\t int insertions, int deletions)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n-\tint ret;\n \n \tif (!files) {\n \t\tassert(insertions == 0 && deletions == 0);\n-\t\treturn fprintf(fp, \"%s\\n\", \" 0 files changed\");\n+\t\tstrbuf_addstr(&sb, \" 0 files changed\");\n+\t\temit_line(options, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\t\treturn;\n \t}\n \n \tstrbuf_addf(&sb,\n@@ -1611,9 +1612,17 @@ int print_stat_summary(FILE *fp, int files, int insertions, int deletions)\n \t\t\t deletions);\n \t}\n \tstrbuf_addch(&sb, '\\n');\n-\tret = fputs(sb.buf, fp);\n+\temit_line(options, NULL, NULL, 1, 0, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n-\treturn ret;\n+}\n+\n+void print_stat_summary(FILE *fp, int files,\n+\t\t\tint insertions, int deletions)\n+{\n+\tstruct diff_options o;\n+\tmemset(&o, 0, sizeof(o));\n+\to.file = fp;\n+\tprint_stat_summary_0(&o, files, insertions, deletions);\n }\n \n static void show_stats(struct diffstat_t *data, struct diff_options *options)\n@@ -1623,13 +1632,13 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \tint total_files = data->nr, count;\n \tint width, name_width, graph_width, number_width = 0, bin_width = 0;\n \tconst char *reset, *add_c, *del_c;\n-\tconst char *line_prefix = \"\";\n \tint extra_shown = 0;\n+\tconst char *line_prefix = diff_line_prefix(options);\n+\tstruct strbuf out = STRBUF_INIT;\n \n \tif (data->nr == 0)\n \t\treturn;\n \n-\tline_prefix = diff_line_prefix(options);\n \tcount = options->stat_count ? options->stat_count : data->nr;\n \n \treset = diff_get_color_opt(options, DIFF_RESET);\n@@ -1783,26 +1792,29 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t}\n \n \t\tif (file->is_binary) {\n-\t\t\tfprintf(options->file, \"%s\", line_prefix);\n-\t\t\tshow_name(options->file, prefix, name, len);\n-\t\t\tfprintf(options->file, \" %*s\", number_width, \"Bin\");\n+\t\t\tshow_name(&out, prefix, name, len);\n+\t\t\tstrbuf_addf(&out, \" %*s\", number_width, \"Bin\");\n \t\t\tif (!added && !deleted) {\n-\t\t\t\tputc('\\n', options->file);\n+\t\t\t\tstrbuf_addch(&out, '\\n');\n+\t\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\t\tstrbuf_reset(&out);\n \t\t\t\tcontinue;\n \t\t\t}\n-\t\t\tfprintf(options->file, \" %s%\"PRIuMAX\"%s\",\n+\t\t\tstrbuf_addf(&out, \" %s%\"PRIuMAX\"%s\",\n \t\t\t\tdel_c, deleted, reset);\n-\t\t\tfprintf(options->file, \" -> \");\n-\t\t\tfprintf(options->file, \"%s%\"PRIuMAX\"%s\",\n+\t\t\tstrbuf_addstr(&out, \" -> \");\n+\t\t\tstrbuf_addf(&out, \"%s%\"PRIuMAX\"%s\",\n \t\t\t\tadd_c, added, reset);\n-\t\t\tfprintf(options->file, \" bytes\");\n-\t\t\tfprintf(options->file, \"\\n\");\n+\t\t\tstrbuf_addstr(&out, \" bytes\\n\");\n+\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n \t\telse if (file->is_unmerged) {\n-\t\t\tfprintf(options->file, \"%s\", line_prefix);\n-\t\t\tshow_name(options->file, prefix, name, len);\n-\t\t\tfprintf(options->file, \" Unmerged\\n\");\n+\t\t\tshow_name(&out, prefix, name, len);\n+\t\t\tstrbuf_addstr(&out, \" Unmerged\\n\");\n+\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n \n@@ -1825,14 +1837,15 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t\t\tadd = total - del;\n \t\t\t}\n \t\t}\n-\t\tfprintf(options->file, \"%s\", line_prefix);\n-\t\tshow_name(options->file, prefix, name, len);\n-\t\tfprintf(options->file, \" %*\"PRIuMAX\"%s\",\n+\t\tshow_name(&out, prefix, name, len);\n+\t\tstrbuf_addf(&out, \" %*\"PRIuMAX\"%s\",\n \t\t\tnumber_width, added + deleted,\n \t\t\tadded + deleted ? \" \" : \"\");\n-\t\tshow_graph(options->file, '+', add, add_c, reset);\n-\t\tshow_graph(options->file, '-', del, del_c, reset);\n-\t\tfprintf(options->file, \"\\n\");\n+\t\tshow_graph(&out, '+', add, add_c, reset);\n+\t\tshow_graph(&out, '-', del, del_c, reset);\n+\t\tstrbuf_addch(&out, '\\n');\n+\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\tstrbuf_reset(&out);\n \t}\n \n \tfor (i = 0; i < data->nr; i++) {\n@@ -1853,11 +1866,12 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\tif (i < count)\n \t\t\tcontinue;\n \t\tif (!extra_shown)\n-\t\t\tfprintf(options->file, \"%s ...\\n\", line_prefix);\n+\t\t\temit_line(options, NULL, NULL, 1, 0,\n+\t\t\t\t \" ...\\n\", strlen(\" ...\\n\"));\n \t\textra_shown = 1;\n \t}\n-\tfprintf(options->file, \"%s\", line_prefix);\n-\tprint_stat_summary(options->file, total_files, adds, dels);\n+\n+\tprint_stat_summary_0(options, total_files, adds, dels);\n }\n \n static void show_shortstats(struct diffstat_t *data, struct diff_options *options)\n@@ -1869,7 +1883,7 @@ static void show_shortstats(struct diffstat_t *data, struct diff_options *option\n \n \tfor (i = 0; i < data->nr; i++) {\n \t\tint added = data->files[i]->added;\n-\t\tint deleted= data->files[i]->deleted;\n+\t\tint deleted = data->files[i]->deleted;\n \n \t\tif (data->files[i]->is_unmerged ||\n \t\t (!data->files[i]->is_interesting && (added + deleted == 0))) {\n@@ -1879,8 +1893,7 @@ static void show_shortstats(struct diffstat_t *data, struct diff_options *option\n \t\t\tdels += deleted;\n \t\t}\n \t}\n-\tfprintf(options->file, \"%s\", diff_line_prefix(options));\n-\tprint_stat_summary(options->file, total_files, adds, dels);\n+\tprint_stat_summary_0(options, total_files, adds, dels);\n }\n \n static void show_numstat(struct diffstat_t *data, struct diff_options *options)\ndiff --git a/diff.h b/diff.h\nindex 9ad546361a..56d8dd036e 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -392,8 +392,8 @@ extern int parse_rename_score(const char **cp_p);\n \n extern long parse_algorithm_value(const char *value);\n \n-extern int print_stat_summary(FILE *fp, int files,\n-\t\t\t int insertions, int deletions);\n+extern void print_stat_summary(FILE *fp, int files,\n+\t\t\t int insertions, int deletions);\n extern void setup_diff_pager(struct diff_options *);\n \n #endif /* DIFF_H */\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320503","messageId":"20170523024048.16879-11-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 10/17] diff.c: convert emit_binary_diff_body to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:41Z","receivedAt":"2017-05-23T02:41:34Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers emit_binary_diff_body.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n diff.c | 39 ++++++++++++++++++++++-----------------\n 1 file changed, 22 insertions(+), 17 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 3357c0fca3..b5a5261a4e 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -2244,8 +2244,8 @@ static unsigned char *deflate_it(char *data,\n \treturn deflated;\n }\n \n-static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n-\t\t\t\t const char *prefix)\n+static void emit_binary_diff_body(struct diff_options *o,\n+\t\t\t\t mmfile_t *one, mmfile_t *two)\n {\n \tvoid *cp;\n \tvoid *delta;\n@@ -2274,13 +2274,12 @@ static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n \t}\n \n \tif (delta && delta_size < deflate_size) {\n-\t\tfprintf(file, \"%sdelta %lu\\n\", prefix, orig_size);\n+\t\temit_line_fmt(o, NULL, NULL, 1, \"delta %lu\\n\", orig_size);\n \t\tfree(deflated);\n \t\tdata = delta;\n \t\tdata_size = delta_size;\n-\t}\n-\telse {\n-\t\tfprintf(file, \"%sliteral %lu\\n\", prefix, two->size);\n+\t} else {\n+\t\temit_line_fmt(o, NULL, NULL, 1, \"literal %lu\\n\", two->size);\n \t\tfree(delta);\n \t\tdata = deflated;\n \t\tdata_size = deflate_size;\n@@ -2289,8 +2288,9 @@ static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n \t/* emit data encoded in base85 */\n \tcp = data;\n \twhile (data_size) {\n+\t\tint len;\n \t\tint bytes = (52 < data_size) ? 52 : data_size;\n-\t\tchar line[70];\n+\t\tchar line[71];\n \t\tdata_size -= bytes;\n \t\tif (bytes <= 26)\n \t\t\tline[0] = bytes + 'A' - 1;\n@@ -2298,20 +2298,25 @@ static void emit_binary_diff_body(FILE *file, mmfile_t *one, mmfile_t *two,\n \t\t\tline[0] = bytes - 26 + 'a' - 1;\n \t\tencode_85(line + 1, cp, bytes);\n \t\tcp = (char *) cp + bytes;\n-\t\tfprintf(file, \"%s\", prefix);\n-\t\tfputs(line, file);\n-\t\tfputc('\\n', file);\n+\n+\t\tlen = strlen(line);\n+\t\tline[len++] = '\\n';\n+\t\tline[len] = '\\0';\n+\n+\t\temit_line(o, NULL, NULL, 1, 0, line, len);\n \t}\n-\tfprintf(file, \"%s\\n\", prefix);\n+\temit_line(o, NULL, NULL, 1, 0, \"\\n\", 1);\n \tfree(data);\n }\n \n-static void emit_binary_diff(FILE *file, mmfile_t *one, mmfile_t *two,\n-\t\t\t const char *prefix)\n+static void emit_binary_diff(struct diff_options *o,\n+\t\t\t mmfile_t *one, mmfile_t *two)\n {\n-\tfprintf(file, \"%sGIT binary patch\\n\", prefix);\n-\temit_binary_diff_body(file, one, two, prefix);\n-\temit_binary_diff_body(file, two, one, prefix);\n+\tconst char *s = \"GIT binary patch\\n\";\n+\tconst int len = strlen(s);\n+\temit_line(o, NULL, NULL, 1, 0, s, len);\n+\temit_binary_diff_body(o, one, two);\n+\temit_binary_diff_body(o, two, one);\n }\n \n int diff_filespec_is_binary(struct diff_filespec *one)\n@@ -2498,7 +2503,7 @@ static void builtin_diff(const char *name_a,\n \t\t\t header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tif (DIFF_OPT_TST(o, BINARY))\n-\t\t\temit_binary_diff(o->file, &mf1, &mf2, line_prefix);\n+\t\t\temit_binary_diff(o, &mf1, &mf2);\n \t\telse\n \t\t\temit_line_fmt(o, NULL, NULL, 1,\n \t\t\t\t \"Binary files %s and %s differ\\n\",\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320504","messageId":"20170523024048.16879-17-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 16/17] diff: buffer all output if asked to","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:47Z","receivedAt":"2017-05-23T02:41:36Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Introduce a new option 'use_buffer' in the struct diff_options which\ncontrols whether all output is buffered up until all output is available.\n\nWe'll have a new struct 'diff_line' in diff.h which will be used to buffer\neach line. The diff_line will duplicate the memory of the line to buffer\nas that is easiest to reason about for now. In a future patch we may want\nto decrease the memory usage by not duplicating all output for buffering\nbut rather we may want to store offsets into the file or in case of hunk\ndescriptions such as the similarity score, we could just store the\nrelevant number and reproduce the text later on.\n\nThis approach was chosen as a first step because it is quite simple\ncompared to the alternative with less memory footprint.\n\nemit_line factors out the emission part into emit_line_emission,\nand depending on the diff_options->use_buffer the emission\nwill be performed directly when calling emit_line or after the\nwhole process is done, i.e. by buffering we have add the possibility\nfor a second pass over the whole output before doing the actual\noutput.\n\nIn 6440d34 (2012-03-14, diff: tweak a _copy_ of diff_options with\nword-diff) we introduced a duplicate diff options struct for word\nemissions as we may have different regex settings in there.\nWhen buffering the output, we need to operate on just one buffer,\nso we have to copy back the emissions of the word buffer into the\nmain buffer.\n\nUnconditionally enable output via buffer in this patch as it yields\na great opportunity for testing, i.e. all the diff tests from the\ntest suite pass without having reordering issues (i.e. only parts\nof the output got buffered, and we forgot to buffer other parts).\nThe test suite passes, which gives confidence that we converted all\nfunctions to use emit_line for output.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n diff.c | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++---------------\n diff.h | 41 +++++++++++++++++\n 2 files changed, 161 insertions(+), 35 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 514c5facd7..c0b8afa38f 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -516,54 +516,85 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n-\n-static void emit_line(struct diff_options *o,\n-\t\t const char *set, const char *reset,\n-\t\t int add_line_prefix, int markup_ws,\n-\t\t int sign, const char *line, int len)\n+static void emit_diff_line(struct diff_options *o,\n+\t\t\t\t struct diff_line *e)\n {\n \tconst char *ws;\n \tint has_trailing_newline, has_trailing_carriage_return;\n+\tint len = e->len;\n \tFILE *file = o->file;\n \n-\tif (add_line_prefix)\n+\tif (e->add_line_prefix)\n \t\tfputs(diff_line_prefix(o), file);\n \n-\tif (markup_ws) {\n+\tswitch (e->state) {\n+\tcase DIFF_LINE_WS:\n \t\tws = diff_get_color(o->use_color, DIFF_WHITESPACE);\n+\t\tif (e->set)\n+\t\t\tfputs(e->set, file);\n+\t\tif (e->sign)\n+\t\t\tfputc(e->sign, file);\n+\t\tif (e->reset)\n+\t\t\tfputs(e->reset, file);\n+\t\tws_check_emit(e->line, e->len, o->ws_rule,\n+\t\t\t file, e->set, e->reset, ws);\n+\t\treturn;\n+\tcase DIFF_LINE_ASIS:\n+\t\thas_trailing_newline = (len > 0 && e->line[len-1] == '\\n');\n+\t\tif (has_trailing_newline)\n+\t\t\tlen--;\n+\t\thas_trailing_carriage_return = (len > 0 && e->line[len-1] == '\\r');\n+\t\tif (has_trailing_carriage_return)\n+\t\t\tlen--;\n \n-\t\tif (set)\n-\t\t\tfputs(set, file);\n-\t\tif (sign)\n-\t\t\tfputc(sign, file);\n-\t\tif (reset)\n-\t\t\tfputs(reset, file);\n-\t\tws = diff_get_color(o->use_color, DIFF_WHITESPACE);\n-\t\tws_check_emit(line, len, o->ws_rule,\n-\t\t\t file, set, reset, ws);\n+\t\tif (len || e->sign) {\n+\t\t\tif (e->set)\n+\t\t\t\tfputs(e->set, file);\n+\t\t\tif (e->sign)\n+\t\t\t\tfputc(e->sign, file);\n+\t\t\tfwrite(e->line, len, 1, file);\n+\t\t\tif (e->reset)\n+\t\t\t\tfputs(e->reset, file);\n+\t\t}\n+\t\tif (has_trailing_carriage_return)\n+\t\t\tfputc('\\r', file);\n+\t\tif (has_trailing_newline)\n+\t\t\tfputc('\\n', file);\n+\t\treturn;\n+\tcase DIFF_LINE_RELOAD_WS_RULE:\n+\t\to->ws_rule = whitespace_rule(e->line);\n \t\treturn;\n+\tdefault:\n+\t\tdie(\"BUG: malformatted buffered patch line: '%d'\", e->state);\n \t}\n+}\n \n-\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n-\tif (has_trailing_newline)\n-\t\tlen--;\n-\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n-\tif (has_trailing_carriage_return)\n-\t\tlen--;\n+static void append_diff_line(struct diff_options *o,\n+\t\t\t\t struct diff_line *e)\n+{\n+\tstruct diff_line *f;\n+\tALLOC_GROW(o->line_buffer,\n+\t\t o->line_buffer_nr + 1,\n+\t\t o->line_buffer_alloc);\n+\tf = &o->line_buffer[o->line_buffer_nr++];\n \n-\tif (len || sign) {\n-\t\tif (set)\n-\t\t\tfputs(set, file);\n-\t\tif (sign)\n-\t\t\tfputc(sign, file);\n-\t\tfwrite(line, len, 1, file);\n-\t\tif (reset)\n-\t\t\tfputs(reset, file);\n-\t}\n-\tif (has_trailing_carriage_return)\n-\t\tfputc('\\r', file);\n-\tif (has_trailing_newline)\n-\t\tfputc('\\n', file);\n+\tmemcpy(f, e, sizeof(struct diff_line));\n+\tf->line = e->line ? xmemdupz(e->line, e->len) : NULL;\n+}\n+\n+static void emit_line(struct diff_options *o,\n+\t\t const char *set, const char *reset,\n+\t\t int add_line_prefix, int markup_ws,\n+\t\t int sign, const char *line, int len)\n+{\n+\tstruct diff_line e = {set, reset, line,\n+\t\tlen, sign, add_line_prefix,\n+\t\tmarkup_ws ? DIFF_LINE_WS : DIFF_LINE_ASIS};\n+\n+\tif (o->use_buffer)\n+\t\tappend_diff_line(o, &e);\n+\telse\n+\t\temit_diff_line(o, &e);\n }\n \n static void emit_line_fmt(struct diff_options *o,\n@@ -1172,6 +1203,18 @@ static void diff_words_flush(struct emit_callback *ecbdata)\n \tif (ecbdata->diff_words->minus.text.size ||\n \t ecbdata->diff_words->plus.text.size)\n \t\tdiff_words_show(ecbdata->diff_words);\n+\n+\tif (ecbdata->diff_words->opt->line_buffer_nr) {\n+\t\tint i;\n+\t\tfor (i = 0; i < ecbdata->diff_words->opt->line_buffer_nr; i++)\n+\t\t\tappend_diff_line(ecbdata->opt,\n+\t\t\t\t&ecbdata->diff_words->opt->line_buffer[i]);\n+\n+\t\tfor (i = 0; i < ecbdata->diff_words->opt->line_buffer_nr; i++)\n+\t\t\tfree((void *)ecbdata->diff_words->opt->line_buffer[i].line);\n+\n+\t\tecbdata->diff_words->opt->line_buffer_nr = 0;\n+\t}\n }\n \n static void diff_filespec_load_driver(struct diff_filespec *one)\n@@ -1207,6 +1250,11 @@ static void init_diff_words_data(struct emit_callback *ecbdata,\n \t\txcalloc(1, sizeof(struct diff_words_data));\n \tecbdata->diff_words->type = o->word_diff;\n \tecbdata->diff_words->opt = o;\n+\n+\to->line_buffer = NULL;\n+\to->line_buffer_nr = 0;\n+\to->line_buffer_alloc = 0;\n+\n \tif (!o->word_regex)\n \t\to->word_regex = userdiff_word_regex(one);\n \tif (!o->word_regex)\n@@ -1241,6 +1289,7 @@ static void free_diff_words_data(struct emit_callback *ecbdata)\n {\n \tif (ecbdata->diff_words) {\n \t\tdiff_words_flush(ecbdata);\n+\t\tfree (ecbdata->diff_words->opt->line_buffer);\n \t\tfree (ecbdata->diff_words->opt);\n \t\tfree (ecbdata->diff_words->minus.text.ptr);\n \t\tfree (ecbdata->diff_words->minus.orig);\n@@ -2579,6 +2628,13 @@ static void builtin_diff(const char *name_a,\n \t\t\txecfg.ctxlen = strtoul(v, NULL, 10);\n \t\tif (o->word_diff)\n \t\t\tinit_diff_words_data(&ecbdata, o, one, two);\n+\t\tif (o->use_buffer) {\n+\t\t\tstruct diff_line e = diff_line_INIT;\n+\t\t\te.state = DIFF_LINE_RELOAD_WS_RULE;\n+\t\t\te.line = name_b;\n+\t\t\te.len = strlen(name_b);\n+\t\t\tappend_diff_line(o, &e);\n+\t\t}\n \t\tif (xdi_diff_outf(&mf1, &mf2, fn_out_consume, &ecbdata,\n \t\t\t\t &xpp, &xecfg))\n \t\t\tdie(\"unable to generate diff for %s\", one->path);\n@@ -3458,6 +3514,10 @@ void diff_setup(struct diff_options *options)\n \t\toptions->a_prefix = \"a/\";\n \t\toptions->b_prefix = \"b/\";\n \t}\n+\n+\toptions->line_buffer = NULL;\n+\toptions->line_buffer_nr = 0;\n+\toptions->line_buffer_alloc = 0;\n }\n \n void diff_setup_done(struct diff_options *options)\n@@ -4796,11 +4856,36 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n {\n \tint i;\n \tstruct diff_queue_struct *q = &diff_queued_diff;\n+\t/*\n+\t * For testing purposes we want to make sure the diff machinery\n+\t * works completely with the buffer. If there is anything emitted\n+\t * outside the emit_diff_line, then the order is screwed\n+\t * up and the tests will fail.\n+\t *\n+\t * TODO (later in this series):\n+\t * We'll unset this flag in a later patch.\n+\t */\n+\to->use_buffer = 1;\n+\n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n \t\tif (check_pair_status(p))\n \t\t\tdiff_flush_patch(p, o);\n \t}\n+\n+\tif (o->use_buffer) {\n+\t\tfor (i = 0; i < o->line_buffer_nr; i++)\n+\t\t\temit_diff_line(o, &o->line_buffer[i]);\n+\n+\t\tfor (i = 0; i < o->line_buffer_nr; i++)\n+\t\t\tfree((void *)o->line_buffer[i].line);\n+\n+\t\tfree(o->line_buffer);\n+\n+\t\to->line_buffer = NULL;\n+\t\to->line_buffer_nr = 0;\n+\t\to->line_buffer_alloc = 0;\n+\t}\n }\n \n void diff_flush(struct diff_options *options)\ndiff --git a/diff.h b/diff.h\nindex 85948ed65a..fad1258556 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -115,6 +115,42 @@ enum diff_submodule_format {\n \tDIFF_SUBMODULE_INLINE_DIFF\n };\n \n+/*\n+ * This struct is used when we need to buffer the output of the diff output.\n+ *\n+ * NEEDSWORK: Instead of storing a copy of the line, add an offset pointer\n+ * into the pre/post image file. This pointer could be a union with the\n+ * line pointer. By storing an offset into the file instead of the literal line,\n+ * we can decrease the memory footprint for the buffered output. At first we\n+ * may want to only have indirection for the content lines, but we could also\n+ * enhance the state for emitting prefabricated lines, e.g. the similarity\n+ * score line or hunk/file headers would only need to store a number or path\n+ * and then the output can be constructed later on depending on state.\n+ */\n+struct diff_line {\n+\tconst char *set;\n+\tconst char *reset;\n+\tconst char *line;\n+\tint len;\n+\tint sign;\n+\tint add_line_prefix;\n+\tenum {\n+\t\t/*\n+\t\t * Emits [lineprefix][set][sign][reset] and then calls\n+\t\t * ws_check_emit which will output \"line\", marked up\n+\t\t * according to ws_rule.\n+\t\t */\n+\t\tDIFF_LINE_WS,\n+\n+\t\t/* Emits [lineprefix][set][sign] line [reset] */\n+\t\tDIFF_LINE_ASIS,\n+\n+\t\t/* Reloads the ws_rule; line contains the file name */\n+\t\tDIFF_LINE_RELOAD_WS_RULE\n+\t} state;\n+};\n+#define diff_line_INIT {NULL, NULL, NULL, 0, 0, 0}\n+\n struct diff_options {\n \tconst char *orderfile;\n \tconst char *pickaxe;\n@@ -188,8 +224,13 @@ struct diff_options {\n \tint diff_path_counter;\n \n \tunsigned ws_rule;\n+\tint use_buffer;\n+\n+\tstruct diff_line *line_buffer;\n+\tint line_buffer_nr, line_buffer_alloc;\n };\n \n+/* Emit [line_prefix] [set] line [reset] */\n void diff_emit_line(struct diff_options *o, const char *set, const char *reset,\n \t\t const char *line, int len);\n \n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320505","messageId":"20170523024048.16879-15-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 14/17] diff.c: convert diff_summary to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:45Z","receivedAt":"2017-05-23T02:41:40Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers diff_summary.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n diff.c | 64 ++++++++++++++++++++++++++++++++++------------------------------\n 1 file changed, 34 insertions(+), 30 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 8ebe673331..76cafde4be 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4504,67 +4504,71 @@ static void flush_one_pair(struct diff_filepair *p, struct diff_options *opt)\n \t}\n }\n \n-static void show_file_mode_name(FILE *file, const char *newdelete, struct diff_filespec *fs)\n+static void show_file_mode_name(struct diff_options *opt, const char *newdelete, struct diff_filespec *fs)\n {\n+\tstruct strbuf sb = STRBUF_INIT;\n \tif (fs->mode)\n-\t\tfprintf(file, \" %s mode %06o \", newdelete, fs->mode);\n+\t\tstrbuf_addf(&sb, \" %s mode %06o \", newdelete, fs->mode);\n \telse\n-\t\tfprintf(file, \" %s \", newdelete);\n-\twrite_name_quoted(fs->path, file, '\\n');\n-}\n+\t\tstrbuf_addf(&sb, \" %s \", newdelete);\n \n+\tquote_c_style(fs->path, &sb, NULL, 0);\n+\tstrbuf_addch(&sb, '\\n');\n+\temit_line(opt, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\tstrbuf_release(&sb);\n+}\n \n-static void show_mode_change(FILE *file, struct diff_filepair *p, int show_name,\n-\t\tconst char *line_prefix)\n+static void show_mode_change(struct diff_options *opt, struct diff_filepair *p,\n+\t\tint show_name)\n {\n \tif (p->one->mode && p->two->mode && p->one->mode != p->two->mode) {\n-\t\tfprintf(file, \"%s mode change %06o => %06o%c\", line_prefix, p->one->mode,\n-\t\t\tp->two->mode, show_name ? ' ' : '\\n');\n+\t\tstruct strbuf sb = STRBUF_INIT;\n \t\tif (show_name) {\n-\t\t\twrite_name_quoted(p->two->path, file, '\\n');\n+\t\t\tstrbuf_addch(&sb, ' ');\n+\t\t\tquote_c_style(p->two->path, &sb, NULL, 0);\n \t\t}\n+\t\temit_line_fmt(opt, NULL, NULL, 1,\n+\t\t\t \" mode change %06o => %06o%s\\n\",\n+\t\t\t p->one->mode, p->two->mode,\n+\t\t\t show_name ? sb.buf : \"\");\n+\t\tstrbuf_release(&sb);\n \t}\n }\n \n-static void show_rename_copy(FILE *file, const char *renamecopy, struct diff_filepair *p,\n-\t\t\tconst char *line_prefix)\n+static void show_rename_copy(struct diff_options *opt, const char *renamecopy,\n+\t\tstruct diff_filepair *p)\n {\n \tchar *names = pprint_rename(p->one->path, p->two->path);\n-\n-\tfprintf(file, \" %s %s (%d%%)\\n\", renamecopy, names, similarity_index(p));\n+\temit_line_fmt(opt, NULL, NULL, 1, \" %s %s (%d%%)\\n\",\n+\t\t renamecopy, names, similarity_index(p));\n \tfree(names);\n-\tshow_mode_change(file, p, 0, line_prefix);\n+\tshow_mode_change(opt, p, 0);\n }\n \n static void diff_summary(struct diff_options *opt, struct diff_filepair *p)\n {\n-\tFILE *file = opt->file;\n-\tconst char *line_prefix = diff_line_prefix(opt);\n-\n \tswitch(p->status) {\n \tcase DIFF_STATUS_DELETED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_file_mode_name(file, \"delete\", p->one);\n+\t\tshow_file_mode_name(opt, \"delete\", p->one);\n \t\tbreak;\n \tcase DIFF_STATUS_ADDED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_file_mode_name(file, \"create\", p->two);\n+\t\tshow_file_mode_name(opt, \"create\", p->two);\n \t\tbreak;\n \tcase DIFF_STATUS_COPIED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_rename_copy(file, \"copy\", p, line_prefix);\n+\t\tshow_rename_copy(opt, \"copy\", p);\n \t\tbreak;\n \tcase DIFF_STATUS_RENAMED:\n-\t\tfputs(line_prefix, file);\n-\t\tshow_rename_copy(file, \"rename\", p, line_prefix);\n+\t\tshow_rename_copy(opt, \"rename\", p);\n \t\tbreak;\n \tdefault:\n \t\tif (p->score) {\n-\t\t\tfprintf(file, \"%s rewrite \", line_prefix);\n-\t\t\twrite_name_quoted(p->two->path, file, ' ');\n-\t\t\tfprintf(file, \"(%d%%)\\n\", similarity_index(p));\n+\t\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\t\tstrbuf_addstr(&sb, \" rewrite \");\n+\t\t\tquote_c_style(p->two->path, &sb, NULL, 0);\n+\t\t\tstrbuf_addf(&sb, \" (%d%%)\\n\", similarity_index(p));\n+\t\t\temit_line(opt, NULL, NULL, 1, 0, sb.buf, sb.len);\n \t\t}\n-\t\tshow_mode_change(file, p, !p->score, line_prefix);\n+\t\tshow_mode_change(opt, p, !p->score);\n \t\tbreak;\n \t}\n }\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320506","messageId":"20170523024048.16879-16-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 15/17] diff.c: emit_line includes whitespace highlighting","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:46Z","receivedAt":"2017-05-23T02:41:42Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Currently any whitespace highlighting happens outside the emit_line\nfunction. Teach the highlighting to emit_line, triggered by a new\nparameter.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n diff.c | 107 ++++++++++++++++++++++++++++++++++++++---------------------------\n diff.h | 2 ++\n 2 files changed, 65 insertions(+), 44 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 76cafde4be..514c5facd7 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -516,15 +516,34 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n-static void emit_line(struct diff_options *o, const char *set, const char *reset,\n-\t\t int add_line_prefix, int sign, const char *line, int len)\n+\n+static void emit_line(struct diff_options *o,\n+\t\t const char *set, const char *reset,\n+\t\t int add_line_prefix, int markup_ws,\n+\t\t int sign, const char *line, int len)\n {\n+\tconst char *ws;\n \tint has_trailing_newline, has_trailing_carriage_return;\n \tFILE *file = o->file;\n \n \tif (add_line_prefix)\n \t\tfputs(diff_line_prefix(o), file);\n \n+\tif (markup_ws) {\n+\t\tws = diff_get_color(o->use_color, DIFF_WHITESPACE);\n+\n+\t\tif (set)\n+\t\t\tfputs(set, file);\n+\t\tif (sign)\n+\t\t\tfputc(sign, file);\n+\t\tif (reset)\n+\t\t\tfputs(reset, file);\n+\t\tws = diff_get_color(o->use_color, DIFF_WHITESPACE);\n+\t\tws_check_emit(line, len, o->ws_rule,\n+\t\t\t file, set, reset, ws);\n+\t\treturn;\n+\t}\n+\n \thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n \tif (has_trailing_newline)\n \t\tlen--;\n@@ -558,14 +577,14 @@ static void emit_line_fmt(struct diff_options *o,\n \tstrbuf_vaddf(&sb, fmt, ap);\n \tva_end(ap);\n \n-\temit_line(o, set, reset, add_line_prefix, 0, sb.buf, sb.len);\n+\temit_line(o, set, reset, add_line_prefix, 0, 0, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n }\n \n void diff_emit_line(struct diff_options *o, const char *set, const char *reset,\n \t\t const char *line, int len)\n {\n-\temit_line(o, set, reset, 1, 0, line, len);\n+\temit_line(o, set, reset, 1, 0, 0, line, len);\n }\n \n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n@@ -596,16 +615,15 @@ static void emit_line_checked(const char *reset,\n \t}\n \n \tif (!ws)\n-\t\temit_line(ecbdata->opt, set, reset, 1, sign, line, len);\n+\t\temit_line(ecbdata->opt, set, reset, 1, 0, sign, line, len);\n \telse if (sign == '+' && new_blank_line_at_eof(ecbdata, line, len))\n \t\t/* Blank line at EOF - paint '+' as well */\n-\t\temit_line(ecbdata->opt, ws, reset, 1, sign, line, len);\n+\t\temit_line(ecbdata->opt, ws, reset, 1, 1, sign, line, len);\n \telse {\n \t\t/* Emit just the prefix, then the rest. */\n-\t\temit_line(ecbdata->opt, set, reset, 1, sign, \"\", 0);\n-\t\tws_check_emit(line, len, ecbdata->ws_rule,\n-\t\t\t ecbdata->opt->file, set, reset, ws);\n+\t\temit_line(ecbdata->opt, set, reset, 1, 1, sign, line, len);\n \t}\n+\n }\n \n static void emit_add_line(const char *reset,\n@@ -652,7 +670,7 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \tif (len < 10 ||\n \t memcmp(line, atat, 2) ||\n \t !(ep = memmem(line + 2, len - 2, atat, 2))) {\n-\t\temit_line(ecbdata->opt, context, reset, 1, 0, line, len);\n+\t\temit_line(ecbdata->opt, context, reset, 1, 0, 0, line, len);\n \t\treturn;\n \t}\n \tep += 2; /* skip over @@ */\n@@ -688,7 +706,7 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \tstrbuf_add(&msgbuf, line + len, org_len - len);\n \tstrbuf_complete_line(&msgbuf);\n \n-\temit_line(ecbdata->opt, \"\", \"\", 1, 0, msgbuf.buf, msgbuf.len);\n+\temit_line(ecbdata->opt, \"\", \"\", 1, 0, 0, msgbuf.buf, msgbuf.len);\n \tstrbuf_release(&msgbuf);\n }\n \n@@ -759,7 +777,7 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\tstatic const char *nneof = \"\\\\ No newline at end of file\\n\";\n \t\tconst char *context = diff_get_color(ecb->color_diff,\n \t\t\t\t\t\t DIFF_CONTEXT);\n-\t\temit_line(ecb->opt, context, reset, 1, 0,\n+\t\temit_line(ecb->opt, context, reset, 1, 0, 0,\n \t\t\t nneof, strlen(nneof));\n \t\tstrbuf_release(&sb);\n \t}\n@@ -835,7 +853,7 @@ static void emit_rewrite_diff(const char *name_a,\n \tstrbuf_addstr(&out, \" +\");\n \tadd_line_count(&out, lc_b);\n \tstrbuf_addstr(&out, \" @@\\n\");\n-\temit_line(o, fraginfo, reset, 1, 0, out.buf, out.len);\n+\temit_line(o, fraginfo, reset, 1, 0, 0, out.buf, out.len);\n \tstrbuf_release(&out);\n \n \tif (lc_a && !o->irreversible_delete)\n@@ -908,7 +926,7 @@ static int fn_out_diff_words_write_helper(struct diff_options *o,\n \twhile (count) {\n \t\tchar *p = memchr(buf, '\\n', count);\n \t\tif (print)\n-\t\t\temit_line(o, NULL, NULL, 1, 0, \"\", 0);\n+\t\t\temit_line(o, NULL, NULL, 1, 0, 0, \"\", 0);\n \n \t\tif (p != buf) {\n \t\t\tconst char *reset = st_el->color && *st_el->color ?\n@@ -917,14 +935,14 @@ static int fn_out_diff_words_write_helper(struct diff_options *o,\n \t\t\tstrbuf_add(&sb, buf, p ? p - buf : count);\n \t\t\tstrbuf_addstr(&sb, st_el->suffix);\n \t\t\temit_line(o, st_el->color, reset,\n-\t\t\t\t 0, 0, sb.buf, sb.len);\n+\t\t\t\t 0, 0, 0, sb.buf, sb.len);\n \t\t\tstrbuf_reset(&sb);\n \t\t}\n \t\tif (!p)\n \t\t\tgoto out;\n \n \t\tstrbuf_addstr(&sb, newline);\n-\t\temit_line(o, NULL, NULL, 0, 0, sb.buf, sb.len);\n+\t\temit_line(o, NULL, NULL, 0, 0, 0, sb.buf, sb.len);\n \t\tstrbuf_reset(&sb);\n \t\tcount -= p + 1 - buf;\n \t\tbuf = p + 1;\n@@ -1139,7 +1157,7 @@ static void diff_words_show(struct diff_words_data *diff_words)\n \tif (diff_words->current_plus != diff_words->plus.text.ptr +\n \t\t\tdiff_words->plus.text.size) {\n \t\tif (color_words_output_graph_prefix(diff_words))\n-\t\t\temit_line(diff_words->opt, NULL, NULL, 1, 0, \"\", 0);\n+\t\t\temit_line(diff_words->opt, NULL, NULL, 1, 0, 0, \"\", 0);\n \t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t&style->ctx, style->newline,\n \t\t\tdiff_words->plus.text.ptr + diff_words->plus.text.size\n@@ -1298,7 +1316,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \to->found_changes = 1;\n \n \tif (ecbdata->header) {\n-\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t ecbdata->header->buf, ecbdata->header->len);\n \t\tstrbuf_release(ecbdata->header);\n \t\tecbdata->header = NULL;\n@@ -1351,8 +1369,8 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t}\n \t\tdiff_words_flush(ecbdata);\n \t\tif (ecbdata->diff_words->type == DIFF_WORDS_PORCELAIN) {\n-\t\t\temit_line(o, context, reset, 1, 0, line, len);\n-\t\t\temit_line(o, NULL, NULL, 0, 0, \"~\\n\", 2);\n+\t\t\temit_line(o, context, reset, 1, 0, 0, line, len);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0, \"~\\n\", 2);\n \t\t} else {\n \t\t\t/*\n \t\t\t * Skip the prefix character, if any. With\n@@ -1363,7 +1381,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t\t line++;\n \t\t\t len--;\n \t\t\t}\n-\t\t\temit_line(o, context, reset, 1, 0, line, len);\n+\t\t\temit_line(o, context, reset, 1, 0, 0, line, len);\n \t\t}\n \t\treturn;\n \t}\n@@ -1386,7 +1404,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t/* incomplete line at the end */\n \t\tecbdata->lno_in_preimage++;\n \t\temit_line(o, diff_get_color(ecbdata->color_diff, DIFF_CONTEXT),\n-\t\t\t reset, 1, 0, line, len);\n+\t\t\t reset, 1, 0, 0, line, len);\n \t\tbreak;\n \t}\n }\n@@ -1575,7 +1593,7 @@ static void print_stat_summary_0(struct diff_options *options, int files,\n \tif (!files) {\n \t\tassert(insertions == 0 && deletions == 0);\n \t\tstrbuf_addstr(&sb, \" 0 files changed\");\n-\t\temit_line(options, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\t\temit_line(options, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \t\treturn;\n \t}\n \n@@ -1603,7 +1621,7 @@ static void print_stat_summary_0(struct diff_options *options, int files,\n \t\t\t deletions);\n \t}\n \tstrbuf_addch(&sb, '\\n');\n-\temit_line(options, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\temit_line(options, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n }\n \n@@ -1787,7 +1805,7 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t\tstrbuf_addf(&out, \" %*s\", number_width, \"Bin\");\n \t\t\tif (!added && !deleted) {\n \t\t\t\tstrbuf_addch(&out, '\\n');\n-\t\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\t\t\tstrbuf_reset(&out);\n \t\t\t\tcontinue;\n \t\t\t}\n@@ -1797,14 +1815,14 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\t\tstrbuf_addf(&out, \"%s%\"PRIuMAX\"%s\",\n \t\t\t\tadd_c, added, reset);\n \t\t\tstrbuf_addstr(&out, \" bytes\\n\");\n-\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n \t\telse if (file->is_unmerged) {\n \t\t\tshow_name(&out, prefix, name, len);\n \t\t\tstrbuf_addstr(&out, \" Unmerged\\n\");\n-\t\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\t\tstrbuf_reset(&out);\n \t\t\tcontinue;\n \t\t}\n@@ -1835,7 +1853,7 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\tshow_graph(&out, '+', add, add_c, reset);\n \t\tshow_graph(&out, '-', del, del_c, reset);\n \t\tstrbuf_addch(&out, '\\n');\n-\t\temit_line(options, NULL, NULL, 1, 0, out.buf, out.len);\n+\t\temit_line(options, NULL, NULL, 1, 0, 0, out.buf, out.len);\n \t\tstrbuf_reset(&out);\n \t}\n \n@@ -1857,7 +1875,7 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)\n \t\tif (i < count)\n \t\t\tcontinue;\n \t\tif (!extra_shown)\n-\t\t\temit_line(options, NULL, NULL, 1, 0,\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0,\n \t\t\t\t \" ...\\n\", strlen(\" ...\\n\"));\n \t\textra_shown = 1;\n \t}\n@@ -2211,7 +2229,7 @@ static void checkdiff_consume(void *priv, char *line, unsigned long len)\n \t\tfprintf(data->o->file, \"%s%s:%d: %s.\\n\",\n \t\t\tline_prefix, data->filename, data->lineno, err);\n \t\tfree(err);\n-\t\temit_line(data->o, set, reset, 1, 0, line, 1);\n+\t\temit_line(data->o, set, reset, 1, 0, 0, line, 1);\n \t\tws_check_emit(line + 1, len - 1, data->ws_rule,\n \t\t\t data->o->file, set, reset, ws);\n \t} else if (line[0] == ' ') {\n@@ -2307,9 +2325,9 @@ static void emit_binary_diff_body(struct diff_options *o,\n \t\tline[len++] = '\\n';\n \t\tline[len] = '\\0';\n \n-\t\temit_line(o, NULL, NULL, 1, 0, line, len);\n+\t\temit_line(o, NULL, NULL, 1, 0, 0, line, len);\n \t}\n-\temit_line(o, NULL, NULL, 1, 0, \"\\n\", 1);\n+\temit_line(o, NULL, NULL, 1, 0, 0, \"\\n\", 1);\n \tfree(data);\n }\n \n@@ -2318,7 +2336,7 @@ static void emit_binary_diff(struct diff_options *o,\n {\n \tconst char *s = \"GIT binary patch\\n\";\n \tconst int len = strlen(s);\n-\temit_line(o, NULL, NULL, 1, 0, s, len);\n+\temit_line(o, NULL, NULL, 1, 0, 0, s, len);\n \temit_binary_diff_body(o, one, two);\n \temit_binary_diff_body(o, two, one);\n }\n@@ -2461,7 +2479,7 @@ static void builtin_diff(const char *name_a,\n \t\tif (complete_rewrite &&\n \t\t (textconv_one || !diff_filespec_is_binary(one)) &&\n \t\t (textconv_two || !diff_filespec_is_binary(two))) {\n-\t\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t\temit_rewrite_diff(name_a, name_b, one, two,\n \t\t\t\t\t\ttextconv_one, textconv_two, o);\n@@ -2471,7 +2489,7 @@ static void builtin_diff(const char *name_a,\n \t}\n \n \tif (o->irreversible_delete && lbl[1][0] == '/') {\n-\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n+\t\temit_line(o, NULL, NULL, 0, 0, 0, header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tgoto free_ab_and_return;\n \t} else if (!DIFF_OPT_TST(o, TEXT) &&\n@@ -2482,11 +2500,11 @@ static void builtin_diff(const char *name_a,\n \t\t !DIFF_OPT_TST(o, BINARY)) {\n \t\t\tif (!oidcmp(&one->oid, &two->oid)) {\n \t\t\t\tif (must_show_header)\n-\t\t\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t\t\t\t header.buf, header.len);\n \t\t\t\tgoto free_ab_and_return;\n \t\t\t}\n-\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t\t header.buf, header.len);\n \t\t\temit_line_fmt(o, NULL, NULL, 1,\n \t\t\t\t \"Binary files %s and %s differ\\n\",\n@@ -2499,11 +2517,11 @@ static void builtin_diff(const char *name_a,\n \t\tif (mf1.size == mf2.size &&\n \t\t !memcmp(mf1.ptr, mf2.ptr, mf1.size)) {\n \t\t\tif (must_show_header)\n-\t\t\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\t\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t\t\t header.buf, header.len);\n \t\t\tgoto free_ab_and_return;\n \t\t}\n-\t\temit_line(o, NULL, NULL, 0, 0,\n+\t\temit_line(o, NULL, NULL, 0, 0, 0,\n \t\t\t header.buf, header.len);\n \t\tstrbuf_reset(&header);\n \t\tif (DIFF_OPT_TST(o, BINARY))\n@@ -2523,7 +2541,7 @@ static void builtin_diff(const char *name_a,\n \t\tconst struct userdiff_funcname *pe;\n \n \t\tif (must_show_header) {\n-\t\t\temit_line(o, NULL, NULL, 0, 0, header.buf, header.len);\n+\t\t\temit_line(o, NULL, NULL, 0, 0, 0, header.buf, header.len);\n \t\t\tstrbuf_reset(&header);\n \t\t}\n \n@@ -2540,6 +2558,7 @@ static void builtin_diff(const char *name_a,\n \t\tecbdata.label_path = lbl;\n \t\tecbdata.color_diff = want_color(o->use_color);\n \t\tecbdata.ws_rule = whitespace_rule(name_b);\n+\t\to->ws_rule = ecbdata.ws_rule;\n \t\tif (ecbdata.ws_rule & WS_BLANK_AT_EOF)\n \t\t\tcheck_blank_at_eof(&mf1, &mf2, &ecbdata);\n \t\tecbdata.opt = o;\n@@ -4514,7 +4533,7 @@ static void show_file_mode_name(struct diff_options *opt, const char *newdelete,\n \n \tquote_c_style(fs->path, &sb, NULL, 0);\n \tstrbuf_addch(&sb, '\\n');\n-\temit_line(opt, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\temit_line(opt, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \tstrbuf_release(&sb);\n }\n \n@@ -4566,7 +4585,7 @@ static void diff_summary(struct diff_options *opt, struct diff_filepair *p)\n \t\t\tstrbuf_addstr(&sb, \" rewrite \");\n \t\t\tquote_c_style(p->two->path, &sb, NULL, 0);\n \t\t\tstrbuf_addf(&sb, \" (%d%%)\\n\", similarity_index(p));\n-\t\t\temit_line(opt, NULL, NULL, 1, 0, sb.buf, sb.len);\n+\t\t\temit_line(opt, NULL, NULL, 1, 0, 0, sb.buf, sb.len);\n \t\t}\n \t\tshow_mode_change(opt, p, !p->score);\n \t\tbreak;\n@@ -4873,10 +4892,10 @@ void diff_flush(struct diff_options *options)\n \t\t\tterm[0] = options->line_termination;\n \t\t\tterm[1] = '\\0';\n \n-\t\t\temit_line(options, NULL, NULL, 1, 0, term, !!term[0]);\n+\t\t\temit_line(options, NULL, NULL, 1, 0, 0, term, !!term[0]);\n \t\t\tif (options->stat_sep) {\n \t\t\t\t/* attach patch instead of inline */\n-\t\t\t\temit_line(options, NULL, NULL, 0, 0,\n+\t\t\t\temit_line(options, NULL, NULL, 0, 0, 0,\n \t\t\t\t\t options->stat_sep,\n \t\t\t\t\t strlen(options->stat_sep));\n \t\t\t}\ndiff --git a/diff.h b/diff.h\nindex 56d8dd036e..85948ed65a 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -186,6 +186,8 @@ struct diff_options {\n \tvoid *output_prefix_data;\n \n \tint diff_path_counter;\n+\n+\tunsigned ws_rule;\n };\n \n void diff_emit_line(struct diff_options *o, const char *set, const char *reset,\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320507","messageId":"20170523024048.16879-14-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 13/17] diff.c: convert diff_flush to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:44Z","receivedAt":"2017-05-23T02:41:54Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers diff_flush.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n diff.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/diff.c b/diff.c\nindex 8317824963..8ebe673331 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4872,7 +4872,9 @@ void diff_flush(struct diff_options *options)\n \t\t\temit_line(options, NULL, NULL, 1, 0, term, !!term[0]);\n \t\t\tif (options->stat_sep) {\n \t\t\t\t/* attach patch instead of inline */\n-\t\t\t\tfputs(options->stat_sep, options->file);\n+\t\t\t\temit_line(options, NULL, NULL, 0, 0,\n+\t\t\t\t\t options->stat_sep,\n+\t\t\t\t\t strlen(options->stat_sep));\n \t\t\t}\n \t\t}\n \n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320508","messageId":"20170523024048.16879-13-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 12/17] diff.c: convert word diffing to use emit_line_*","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:43Z","receivedAt":"2017-05-23T02:41:57Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch, I want to propose an option to detect&color\nmoved lines in a diff, which cannot be done in a one-pass over\nthe diff. Instead we need to go over the whole diff twice,\nbecause we cannot detect the first line of the two corresponding\nlines (+ and -) that got moved.\n\nSo to prepare the diff machinery for two pass algorithms\n(i.e. buffer it all up and then operate on the result),\nmove all emissions to places, such that the only emitting\nfunction is emit_line_0.\n\nThis covers all code related to diffing words.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n diff.c | 73 +++++++++++++++++++++++++++++-------------------------------------\n 1 file changed, 32 insertions(+), 41 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex a76abf5f69..8317824963 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -897,37 +897,42 @@ struct diff_words_data {\n \tstruct diff_words_style *style;\n };\n \n-static int fn_out_diff_words_write_helper(FILE *fp,\n+static int fn_out_diff_words_write_helper(struct diff_options *o,\n \t\t\t\t\t struct diff_words_style_elem *st_el,\n \t\t\t\t\t const char *newline,\n-\t\t\t\t\t size_t count, const char *buf,\n-\t\t\t\t\t const char *line_prefix)\n+\t\t\t\t\t size_t count, const char *buf)\n {\n \tint print = 0;\n+\tstruct strbuf sb = STRBUF_INIT;\n \n \twhile (count) {\n \t\tchar *p = memchr(buf, '\\n', count);\n \t\tif (print)\n-\t\t\tfputs(line_prefix, fp);\n+\t\t\temit_line(o, NULL, NULL, 1, 0, \"\", 0);\n+\n \t\tif (p != buf) {\n-\t\t\tif (st_el->color && fputs(st_el->color, fp) < 0)\n-\t\t\t\treturn -1;\n-\t\t\tif (fputs(st_el->prefix, fp) < 0 ||\n-\t\t\t fwrite(buf, p ? p - buf : count, 1, fp) != 1 ||\n-\t\t\t fputs(st_el->suffix, fp) < 0)\n-\t\t\t\treturn -1;\n-\t\t\tif (st_el->color && *st_el->color\n-\t\t\t && fputs(GIT_COLOR_RESET, fp) < 0)\n-\t\t\t\treturn -1;\n+\t\t\tconst char *reset = st_el->color && *st_el->color ?\n+\t\t\t\t\t GIT_COLOR_RESET : NULL;\n+\t\t\tstrbuf_addstr(&sb, st_el->prefix);\n+\t\t\tstrbuf_add(&sb, buf, p ? p - buf : count);\n+\t\t\tstrbuf_addstr(&sb, st_el->suffix);\n+\t\t\temit_line(o, st_el->color, reset,\n+\t\t\t\t 0, 0, sb.buf, sb.len);\n+\t\t\tstrbuf_reset(&sb);\n \t\t}\n \t\tif (!p)\n-\t\t\treturn 0;\n-\t\tif (fputs(newline, fp) < 0)\n-\t\t\treturn -1;\n+\t\t\tgoto out;\n+\n+\t\tstrbuf_addstr(&sb, newline);\n+\t\temit_line(o, NULL, NULL, 0, 0, sb.buf, sb.len);\n+\t\tstrbuf_reset(&sb);\n \t\tcount -= p + 1 - buf;\n \t\tbuf = p + 1;\n \t\tprint = 1;\n \t}\n+\n+out:\n+\tstrbuf_release(&sb);\n \treturn 0;\n }\n \n@@ -981,14 +986,12 @@ static void fn_out_diff_words_aux(void *priv, char *line, unsigned long len)\n \tint minus_first, minus_len, plus_first, plus_len;\n \tconst char *minus_begin, *minus_end, *plus_begin, *plus_end;\n \tstruct diff_options *opt = diff_words->opt;\n-\tconst char *line_prefix;\n \n \tif (line[0] != '@' || parse_hunk_header(line, len,\n \t\t\t&minus_first, &minus_len, &plus_first, &plus_len))\n \t\treturn;\n \n \tassert(opt);\n-\tline_prefix = diff_line_prefix(opt);\n \n \t/* POSIX requires that first be decremented by one if len == 0... */\n \tif (minus_len) {\n@@ -1005,28 +1008,21 @@ static void fn_out_diff_words_aux(void *priv, char *line, unsigned long len)\n \t} else\n \t\tplus_begin = plus_end = diff_words->plus.orig[plus_first].end;\n \n-\tif (color_words_output_graph_prefix(diff_words)) {\n-\t\tfputs(line_prefix, diff_words->opt->file);\n-\t}\n \tif (diff_words->current_plus != plus_begin) {\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t\t&style->ctx, style->newline,\n \t\t\t\tplus_begin - diff_words->current_plus,\n-\t\t\t\tdiff_words->current_plus, line_prefix);\n-\t\tif (*(plus_begin - 1) == '\\n')\n-\t\t\tfputs(line_prefix, diff_words->opt->file);\n+\t\t\t\tdiff_words->current_plus);\n \t}\n \tif (minus_begin != minus_end) {\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t\t&style->old, style->newline,\n-\t\t\t\tminus_end - minus_begin, minus_begin,\n-\t\t\t\tline_prefix);\n+\t\t\t\tminus_end - minus_begin, minus_begin);\n \t}\n \tif (plus_begin != plus_end) {\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t\t&style->new, style->newline,\n-\t\t\t\tplus_end - plus_begin, plus_begin,\n-\t\t\t\tline_prefix);\n+\t\t\t\tplus_end - plus_begin, plus_begin);\n \t}\n \n \tdiff_words->current_plus = plus_end;\n@@ -1113,18 +1109,14 @@ static void diff_words_show(struct diff_words_data *diff_words)\n \tstruct diff_words_style *style = diff_words->style;\n \n \tstruct diff_options *opt = diff_words->opt;\n-\tconst char *line_prefix;\n-\n \tassert(opt);\n-\tline_prefix = diff_line_prefix(opt);\n \n \t/* special case: only removal */\n \tif (!diff_words->plus.text.size) {\n-\t\tfputs(line_prefix, diff_words->opt->file);\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t&style->old, style->newline,\n \t\t\tdiff_words->minus.text.size,\n-\t\t\tdiff_words->minus.text.ptr, line_prefix);\n+\t\t\tdiff_words->minus.text.ptr);\n \t\tdiff_words->minus.text.size = 0;\n \t\treturn;\n \t}\n@@ -1147,12 +1139,11 @@ static void diff_words_show(struct diff_words_data *diff_words)\n \tif (diff_words->current_plus != diff_words->plus.text.ptr +\n \t\t\tdiff_words->plus.text.size) {\n \t\tif (color_words_output_graph_prefix(diff_words))\n-\t\t\tfputs(line_prefix, diff_words->opt->file);\n-\t\tfn_out_diff_words_write_helper(diff_words->opt->file,\n+\t\t\temit_line(diff_words->opt, NULL, NULL, 1, 0, \"\", 0);\n+\t\tfn_out_diff_words_write_helper(diff_words->opt,\n \t\t\t&style->ctx, style->newline,\n \t\t\tdiff_words->plus.text.ptr + diff_words->plus.text.size\n-\t\t\t- diff_words->current_plus, diff_words->current_plus,\n-\t\t\tline_prefix);\n+\t\t\t- diff_words->current_plus, diff_words->current_plus);\n \t}\n \tdiff_words->minus.text.size = diff_words->plus.text.size = 0;\n }\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320509","messageId":"20170523024048.16879-5-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 04/17] diff: introduce more flexible emit function","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:35Z","receivedAt":"2017-05-23T02:41:59Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Currently, diff output is written either through the emit_line_0\nfunction or through the FILE * in struct diff_options directly. To\nmake it easier to teach diff to buffer its output (which will be done\nin a subsequent commit), introduce a more flexible emit_line() function.\nIn this commit, direct usages of emit_line_0() are replaced with\nemit_line(); subsequent commits will also replace usages of the\nFILE * with emit().\n\nInstead of having a 'first' parameter containing the first character\nof a line, have a dedicated 'sign' parameter that is just set when\nthe first character of the line is part of the actual content, i.e.\n' ', '+', '-'.\n\nSigned-off-by: Stefan Beller \n---\n diff.c | 76 +++++++++++++++++++++++++++++-------------------------------------\n 1 file changed, 33 insertions(+), 43 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 2f9722b382..3569857818 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -516,36 +516,30 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n-static void emit_line_0(struct diff_options *o, const char *set, const char *reset,\n-\t\t\tint first, const char *line, int len)\n+static void emit_line(struct diff_options *o, const char *set, const char *reset,\n+\t\t int add_line_prefix, int sign, const char *line, int len)\n {\n \tint has_trailing_newline, has_trailing_carriage_return;\n-\tint nofirst;\n \tFILE *file = o->file;\n \n-\tfputs(diff_line_prefix(o), file);\n+\tif (add_line_prefix)\n+\t\tfputs(diff_line_prefix(o), file);\n \n-\tif (len == 0) {\n-\t\thas_trailing_newline = (first == '\\n');\n-\t\thas_trailing_carriage_return = (!has_trailing_newline &&\n-\t\t\t\t\t\t(first == '\\r'));\n-\t\tnofirst = has_trailing_newline || has_trailing_carriage_return;\n-\t} else {\n-\t\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n-\t\tif (has_trailing_newline)\n-\t\t\tlen--;\n-\t\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n-\t\tif (has_trailing_carriage_return)\n-\t\t\tlen--;\n-\t\tnofirst = 0;\n-\t}\n+\thas_trailing_newline = (len > 0 && line[len-1] == '\\n');\n+\tif (has_trailing_newline)\n+\t\tlen--;\n+\thas_trailing_carriage_return = (len > 0 && line[len-1] == '\\r');\n+\tif (has_trailing_carriage_return)\n+\t\tlen--;\n \n-\tif (len || !nofirst) {\n-\t\tfputs(set, file);\n-\t\tif (!nofirst)\n-\t\t\tfputc(first, file);\n+\tif (len || sign) {\n+\t\tif (set)\n+\t\t\tfputs(set, file);\n+\t\tif (sign)\n+\t\t\tfputc(sign, file);\n \t\tfwrite(line, len, 1, file);\n-\t\tfputs(reset, file);\n+\t\tif (reset)\n+\t\t\tfputs(reset, file);\n \t}\n \tif (has_trailing_carriage_return)\n \t\tfputc('\\r', file);\n@@ -553,12 +547,6 @@ static void emit_line_0(struct diff_options *o, const char *set, const char *res\n \t\tfputc('\\n', file);\n }\n \n-static void emit_line(struct diff_options *o, const char *set, const char *reset,\n-\t\t const char *line, int len)\n-{\n-\temit_line_0(o, set, reset, line[0], line+1, len-1);\n-}\n-\n static int new_blank_line_at_eof(struct emit_callback *ecbdata, const char *line, int len)\n {\n \tif (!((ecbdata->ws_rule & WS_BLANK_AT_EOF) &&\n@@ -587,13 +575,13 @@ static void emit_line_checked(const char *reset,\n \t}\n \n \tif (!ws)\n-\t\temit_line_0(ecbdata->opt, set, reset, sign, line, len);\n+\t\temit_line(ecbdata->opt, set, reset, 1, sign, line, len);\n \telse if (sign == '+' && new_blank_line_at_eof(ecbdata, line, len))\n \t\t/* Blank line at EOF - paint '+' as well */\n-\t\temit_line_0(ecbdata->opt, ws, reset, sign, line, len);\n+\t\temit_line(ecbdata->opt, ws, reset, 1, sign, line, len);\n \telse {\n \t\t/* Emit just the prefix, then the rest. */\n-\t\temit_line_0(ecbdata->opt, set, reset, sign, \"\", 0);\n+\t\temit_line(ecbdata->opt, set, reset, 1, sign, \"\", 0);\n \t\tws_check_emit(line, len, ecbdata->ws_rule,\n \t\t\t ecbdata->opt->file, set, reset, ws);\n \t}\n@@ -643,7 +631,7 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \tif (len < 10 ||\n \t memcmp(line, atat, 2) ||\n \t !(ep = memmem(line + 2, len - 2, atat, 2))) {\n-\t\temit_line(ecbdata->opt, context, reset, line, len);\n+\t\temit_line(ecbdata->opt, context, reset, 1, 0, line, len);\n \t\treturn;\n \t}\n \tep += 2; /* skip over @@ */\n@@ -679,7 +667,7 @@ static void emit_hunk_header(struct emit_callback *ecbdata,\n \tstrbuf_add(&msgbuf, line + len, org_len - len);\n \tstrbuf_complete_line(&msgbuf);\n \n-\temit_line(ecbdata->opt, \"\", \"\", msgbuf.buf, msgbuf.len);\n+\temit_line(ecbdata->opt, \"\", \"\", 1, 0, msgbuf.buf, msgbuf.len);\n \tstrbuf_release(&msgbuf);\n }\n \n@@ -742,8 +730,8 @@ static void emit_rewrite_lines(struct emit_callback *ecb,\n \t\tconst char *context = diff_get_color(ecb->color_diff,\n \t\t\t\t\t\t DIFF_CONTEXT);\n \t\tputc('\\n', ecb->opt->file);\n-\t\temit_line_0(ecb->opt, context, reset, '\\\\',\n-\t\t\t nneof, strlen(nneof));\n+\t\temit_line(ecb->opt, context, reset, 1, '\\\\',\n+\t\t\t nneof, strlen(nneof));\n \t}\n }\n \n@@ -1341,7 +1329,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t}\n \t\tdiff_words_flush(ecbdata);\n \t\tif (ecbdata->diff_words->type == DIFF_WORDS_PORCELAIN) {\n-\t\t\temit_line(o, context, reset, line, len);\n+\t\t\temit_line(o, context, reset, 1, 0, line, len);\n \t\t\tfputs(\"~\\n\", o->file);\n \t\t} else {\n \t\t\t/*\n@@ -1353,7 +1341,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t\t line++;\n \t\t\t len--;\n \t\t\t}\n-\t\t\temit_line(o, context, reset, line, len);\n+\t\t\temit_line(o, context, reset, 1, 0, line, len);\n \t\t}\n \t\treturn;\n \t}\n@@ -1376,7 +1364,7 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t/* incomplete line at the end */\n \t\tecbdata->lno_in_preimage++;\n \t\temit_line(o, diff_get_color(ecbdata->color_diff, DIFF_CONTEXT),\n-\t\t\t reset, line, len);\n+\t\t\t reset, 1, 0, line, len);\n \t\tbreak;\n \t}\n }\n@@ -2188,7 +2176,7 @@ static void checkdiff_consume(void *priv, char *line, unsigned long len)\n \t\tfprintf(data->o->file, \"%s%s:%d: %s.\\n\",\n \t\t\tline_prefix, data->filename, data->lineno, err);\n \t\tfree(err);\n-\t\temit_line(data->o, set, reset, line, 1);\n+\t\temit_line(data->o, set, reset, 1, 0, line, 1);\n \t\tws_check_emit(line + 1, len - 1, data->ws_rule,\n \t\t\t data->o->file, set, reset, ws);\n \t} else if (line[0] == ' ') {\n@@ -4833,9 +4821,11 @@ void diff_flush(struct diff_options *options)\n \n \tif (output_format & DIFF_FORMAT_PATCH) {\n \t\tif (separator) {\n-\t\t\tfprintf(options->file, \"%s%c\",\n-\t\t\t\tdiff_line_prefix(options),\n-\t\t\t\toptions->line_termination);\n+\t\t\tchar term[2];\n+\t\t\tterm[0] = options->line_termination;\n+\t\t\tterm[1] = '\\0';\n+\n+\t\t\temit_line(options, NULL, NULL, 1, 0, term, !!term[0]);\n \t\t\tif (options->stat_sep) {\n \t\t\t\t/* attach patch instead of inline */\n \t\t\t\tfputs(options->stat_sep, options->file);\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320510","messageId":"20170523024048.16879-18-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 17/17] diff.c: color moved lines differently","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:48Z","receivedAt":"2017-05-23T02:42:02Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"When a patch consists mostly of moving blocks of code around, it can\nbe quite tedious to ensure that the blocks are moved verbatim, and not\nundesirably modified in the move. To that end, color blocks that are\nmoved within the same patch differently. For example (OM, del, add,\nand NM are different colors):\n\n [OM] -void sensitive_stuff(void)\n [OM] -{\n [OM] - if (!is_authorized_user())\n [OM] - die(\"unauthorized\");\n [OM] - sensitive_stuff(spanning,\n [OM] - multiple,\n [OM] - lines);\n [OM] -}\n\n void another_function()\n {\n [del] - printf(\"foo\");\n [add] + printf(\"bar\");\n }\n\n [NM] +void sensitive_stuff(void)\n [NM] +{\n [NM] + if (!is_authorized_user())\n [NM] + die(\"unauthorized\");\n [NM] + sensitive_stuff(spanning,\n [NM] + multiple,\n [NM] + lines);\n [NM] +}\n\nAdjacent blocks are colored differently. For example, in this\npotentially malicious patch, the swapping of blocks can be spotted:\n\n [OM] -void sensitive_stuff(void)\n [OM] -{\n [OMA] - if (!is_authorized_user())\n [OMA] - die(\"unauthorized\");\n [OM] - sensitive_stuff(spanning,\n [OM] - multiple,\n [OM] - lines);\n [OMA] -}\n\n void another_function()\n {\n [del] - printf(\"foo\");\n [add] + printf(\"bar\");\n }\n\n [NM] +void sensitive_stuff(void)\n [NM] +{\n [NMA] + sensitive_stuff(spanning,\n [NMA] + multiple,\n [NMA] + lines);\n [NM] + if (!is_authorized_user())\n [NM] + die(\"unauthorized\");\n [NMA] +}\n\nIf the moved code is larger, it is easier to hide some permutation in the\ncode, which is why the alternative coloring is really needed.\n\nAs the reviewers attention should be brought to the places, where the\ndifference is introduced to the moved code, we cannot just have one new\ncolor for all of moved code.\n\nFirst I implemented an alternative design, which would show a moved hunk\nin one color, and its boundaries in another color. This idea was error\nprone as it inspected each line and its neighboring lines to determine\nif the line was (a) moved and (b) if was deep inside a hunk by having\nmatching neighboring lines. This is unreliable as the we can construct\nhunks which have equal neighbors that just exceed the number of lines\ninspected. (Think of 'AXYZBXYZCXYZD..' with each letter as a line, that\nis permutated to AXYZCXYZBXYZD..').\n\nInstead this provides a dynamic programming greedy algorithm that finds\nthe largest moved hunk and then switches color to the alternative color\nfor the next hunk. By doing this any permutation is recognized and\ndisplayed. That implies that there is no dedicated boundary or\ninside-hunk color, but instead we'll have just two colors alternating\nfor hunks.\n\nIt would be a bit more UX friendly if the two corresponding hunks\n(of added and deleted lines) for one move would get the same color id.\n(Both get \"regular moved\" or \"alternative moved\"). This problem is\ndeferred to a later patch for now.\n\nA note on the options '--submodule=diff' and '--color-words/--word-diff':\nIn the conversion to use emit_line in the prior patches both submodules\nas well as word diff output carefully chose to call emit_line with sign=0.\nAll output with sign=0 is ignored for move detection purposes in this\npatch, such that no weird looking output will be generated for these\ncases. This leads to another thought: We could pass on '--color-moved' to\nsubmodules such that they color up moved lines for themselves. If we'd do\nso only line moves within a repository boundary are marked up.\n\nHelped-by: Jonathan Tan \nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n\n# Conflicts:\n#\tdiff.c\n---\n Documentation/config.txt | 14 ++-\n diff.c | 275 +++++++++++++++++++++++++++++++++++++++++++--\n diff.h | 9 +-\n t/t4015-diff-whitespace.sh | 267 +++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 552 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 475e874d51..902d017c3b 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1051,14 +1051,24 @@ This does not affect linkgit:git-format-patch[1] or the\n 'git-diff-{asterisk}' plumbing commands. Can be overridden on the\n command line with the `--color[=]` option.\n \n+color.moved::\n+\tA boolean value, whether a diff should color moved lines\n+\tdifferently. The moved lines are searched for in the diff only.\n+\tDuplicated lines from somewhere in the project that are not\n+\tpart of the diff are not colored as moved.\n+\tDefaults to false.\n+\n color.diff.::\n \tUse customized color for diff colorization. `` specifies\n \twhich part of the patch to use the specified color, and is one\n \tof `context` (context text - `plain` is a historical synonym),\n \t`meta` (metainformation), `frag`\n \t(hunk header), 'func' (function in hunk header), `old` (removed lines),\n-\t`new` (added lines), `commit` (commit headers), or `whitespace`\n-\t(highlighting whitespace errors).\n+\t`new` (added lines), `commit` (commit headers), `whitespace`\n+\t(highlighting whitespace errors), `oldMoved` (removed lines that\n+\treappear), `newMoved` (added lines that were removed elsewhere),\n+\t`oldMovedAlternative` and `newMovedAlternative` (as a fallback to\n+\tcover adjacent blocks of moved code)\n \n color.decorate.::\n \tUse customized color for 'git log --decorate' output. `` is one\ndiff --git a/diff.c b/diff.c\nindex c0b8afa38f..23e70d348e 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -31,6 +31,7 @@ static int diff_indent_heuristic; /* experimental */\n static int diff_rename_limit_default = 400;\n static int diff_suppress_blank_empty;\n static int diff_use_color_default = -1;\n+static int diff_color_moved_default;\n static int diff_context_default = 3;\n static int diff_interhunk_context_default;\n static const char *diff_word_regex_cfg;\n@@ -55,6 +56,10 @@ static char diff_colors[][COLOR_MAXLEN] = {\n \tGIT_COLOR_YELLOW,\t/* COMMIT */\n \tGIT_COLOR_BG_RED,\t/* WHITESPACE */\n \tGIT_COLOR_NORMAL,\t/* FUNCINFO */\n+\tGIT_COLOR_BOLD_RED,\t/* OLD_MOVED_A */\n+\tGIT_COLOR_BG_RED,\t/* OLD_MOVED_B */\n+\tGIT_COLOR_BOLD_GREEN,\t/* NEW_MOVED_A */\n+\tGIT_COLOR_BG_GREEN,\t/* NEW_MOVED_B */\n };\n \n static NORETURN void die_want_option(const char *option_name)\n@@ -80,6 +85,14 @@ static int parse_diff_color_slot(const char *var)\n \t\treturn DIFF_WHITESPACE;\n \tif (!strcasecmp(var, \"func\"))\n \t\treturn DIFF_FUNCINFO;\n+\tif (!strcasecmp(var, \"oldmoved\"))\n+\t\treturn DIFF_FILE_OLD_MOVED;\n+\tif (!strcasecmp(var, \"oldmovedalternative\"))\n+\t\treturn DIFF_FILE_OLD_MOVED_ALT;\n+\tif (!strcasecmp(var, \"newmoved\"))\n+\t\treturn DIFF_FILE_NEW_MOVED;\n+\tif (!strcasecmp(var, \"newmovedalternative\"))\n+\t\treturn DIFF_FILE_NEW_MOVED_ALT;\n \treturn -1;\n }\n \n@@ -234,6 +247,10 @@ int git_diff_ui_config(const char *var, const char *value, void *cb)\n \t\tdiff_use_color_default = git_config_colorbool(var, value);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(var, \"color.moved\")) {\n+\t\tdiff_color_moved_default = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(var, \"diff.context\")) {\n \t\tdiff_context_default = git_config_int(var, value);\n \t\tif (diff_context_default < 0)\n@@ -354,6 +371,88 @@ int git_diff_basic_config(const char *var, const char *value, void *cb)\n \treturn git_default_config(var, value, cb);\n }\n \n+struct moved_entry {\n+\tstruct hashmap_entry ent;\n+\tconst struct diff_line *line;\n+\tstruct moved_entry *next_line;\n+};\n+\n+static void get_ws_cleaned_string(const struct diff_line *l,\n+\t\t\t\t struct strbuf *out)\n+{\n+\tint i;\n+\tfor (i = 0; i < l->len; i++) {\n+\t\tif (isspace(l->line[i]))\n+\t\t\tcontinue;\n+\t\tstrbuf_addch(out, l->line[i]);\n+\t}\n+}\n+\n+static int diff_line_cmp_no_ws(const struct diff_line *a,\n+\t\t\t\t\t const struct diff_line *b,\n+\t\t\t\t\t const void *keydata)\n+{\n+\tint ret;\n+\tstruct strbuf sba = STRBUF_INIT;\n+\tstruct strbuf sbb = STRBUF_INIT;\n+\n+\tget_ws_cleaned_string(a, &sba);\n+\tget_ws_cleaned_string(b, &sbb);\n+\tret = sba.len != sbb.len || strncmp(sba.buf, sbb.buf, sba.len);\n+\n+\tstrbuf_release(&sba);\n+\tstrbuf_release(&sbb);\n+\treturn ret;\n+}\n+\n+static int diff_line_cmp(const struct diff_line *a,\n+\t\t\t\t const struct diff_line *b,\n+\t\t\t\t const void *keydata)\n+{\n+\treturn a->len != b->len || strncmp(a->line, b->line, a->len);\n+}\n+\n+static int moved_entry_cmp(const struct moved_entry *a,\n+\t\t\t const struct moved_entry *b,\n+\t\t\t const void *keydata)\n+{\n+\treturn diff_line_cmp(a->line, b->line, keydata);\n+}\n+\n+static int moved_entry_cmp_no_ws(const struct moved_entry *a,\n+\t\t\t\t const struct moved_entry *b,\n+\t\t\t\t const void *keydata)\n+{\n+\treturn diff_line_cmp_no_ws(a->line, b->line, keydata);\n+}\n+\n+static unsigned get_line_hash(struct diff_line *line, unsigned ignore_ws)\n+{\n+\tstatic struct strbuf sb = STRBUF_INIT;\n+\n+\tif (ignore_ws) {\n+\t\tstrbuf_reset(&sb);\n+\t\tget_ws_cleaned_string(line, &sb);\n+\t\treturn memhash(sb.buf, sb.len);\n+\t} else {\n+\t\treturn memhash(line->line, line->len);\n+\t}\n+}\n+\n+static struct moved_entry *prepare_entry(struct diff_options *o,\n+\t\t\t\t\t int line_no)\n+{\n+\tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n+\tunsigned ignore_ws = DIFF_XDL_TST(o, IGNORE_WHITESPACE);\n+\tstruct diff_line *l = &o->line_buffer[line_no];\n+\n+\tret->ent.hash = get_line_hash(l, ignore_ws);\n+\tret->line = l;\n+\tret->next_line = NULL;\n+\n+\treturn ret;\n+}\n+\n static char *quote_two(const char *one, const char *two)\n {\n \tint need_one = quote_c_style(one, NULL, NULL, 1);\n@@ -516,6 +615,141 @@ static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,\n \tecbdata->blank_at_eof_in_postimage = (at - l2) + 1;\n }\n \n+static void add_lines_to_move_detection(struct diff_options *o,\n+\t\t\t\t\tstruct hashmap *add_lines,\n+\t\t\t\t\tstruct hashmap *del_lines)\n+{\n+\tstruct moved_entry *prev_line = NULL;\n+\n+\tint n;\n+\tfor (n = 0; n < o->line_buffer_nr; n++) {\n+\t\tint sign = 0;\n+\t\tstruct hashmap *hm;\n+\t\tstruct moved_entry *key;\n+\n+\t\tswitch (o->line_buffer[n].sign) {\n+\t\tcase '+':\n+\t\t\tsign = '+';\n+\t\t\thm = add_lines;\n+\t\t\tbreak;\n+\t\tcase '-':\n+\t\t\tsign = '-';\n+\t\t\thm = del_lines;\n+\t\t\tbreak;\n+\t\tcase ' ':\n+\t\tdefault:\n+\t\t\tprev_line = NULL;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tkey = prepare_entry(o, n);\n+\t\tif (prev_line &&\n+\t\t prev_line->line->sign == sign)\n+\t\t\tprev_line->next_line = key;\n+\n+\t\thashmap_add(hm, key);\n+\t\tprev_line = key;\n+\t}\n+}\n+\n+static void mark_color_as_moved(struct diff_options *o,\n+\t\t\t\tstruct hashmap *add_lines,\n+\t\t\t\tstruct hashmap *del_lines)\n+{\n+\tstruct moved_entry **pmb = NULL; /* potentially moved blocks */\n+\tint pmb_nr = 0, pmb_alloc = 0;\n+\tint use_alt_color = 0;\n+\tint n;\n+\n+\tfor (n = 0; n < o->line_buffer_nr; n++) {\n+\t\tstruct hashmap *hm = NULL;\n+\t\tstruct moved_entry *key;\n+\t\tstruct moved_entry *match = NULL;\n+\t\tstruct diff_line *l = &o->line_buffer[n];\n+\t\tint i, lp, rp;\n+\n+\t\tswitch (l->sign) {\n+\t\tcase '+':\n+\t\t\thm = del_lines;\n+\t\t\tbreak;\n+\t\tcase '-':\n+\t\t\thm = add_lines;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tuse_alt_color = 0;\n+\t\t\tpmb_nr = 0; /* no running sets */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* Check for any match to color it as a move. */\n+\t\tkey = prepare_entry(o, n);\n+\t\tmatch = hashmap_get(hm, key, o);\n+\t\tfree(key);\n+\t\tif (!match)\n+\t\t\tcontinue;\n+\n+\t\t/* Check any potential block runs, advance each or nullify */\n+\t\tfor (i = 0; i < pmb_nr; i++) {\n+\t\t\tstruct moved_entry *p = pmb[i];\n+\t\t\tstruct moved_entry *pnext = (p && p->next_line) ?\n+\t\t\t\t\tp->next_line : NULL;\n+\t\t\tif (pnext &&\n+\t\t\t !diff_line_cmp(pnext->line, l, o)) {\n+\t\t\t\tpmb[i] = p->next_line;\n+\t\t\t} else {\n+\t\t\t\tpmb[i] = NULL;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/* Shrink the set to the remaining runs */\n+\t\tfor (lp = 0, rp = pmb_nr - 1; lp <= rp;) {\n+\t\t\twhile (lp < pmb_nr && pmb[lp])\n+\t\t\t\tlp++;\n+\t\t\t/* lp points at the first NULL now */\n+\n+\t\t\twhile (rp > -1 && !pmb[rp])\n+\t\t\t\trp--;\n+\t\t\t/* rp points at the last non-NULL */\n+\n+\t\t\tif (lp < pmb_nr && rp > -1 && lp < rp) {\n+\t\t\t\tpmb[lp] = pmb[rp];\n+\t\t\t\tpmb[rp] = NULL;\n+\t\t\t\trp--;\n+\t\t\t\tlp++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (rp > -1) {\n+\t\t\t/* Remember the number of running sets */\n+\t\t\tpmb_nr = rp + 1;\n+\t\t} else {\n+\t\t\t/* Toggle color */\n+\t\t\tuse_alt_color = (use_alt_color + 1) % 2;\n+\n+\t\t\t/* Build up a new set */\n+\t\t\tpmb_nr = 0;\n+\t\t\tfor (; match; match = hashmap_get_next(hm, match)) {\n+\t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n+\t\t\t\tpmb[pmb_nr++] = match;\n+\t\t\t}\n+\t\t}\n+\n+\t\tswitch (l->sign) {\n+\t\tcase '+':\n+\t\t\tl->set = diff_get_color_opt(o,\n+\t\t\t\tDIFF_FILE_NEW_MOVED + use_alt_color);\n+\t\t\tbreak;\n+\t\tcase '-':\n+\t\t\tl->set = diff_get_color_opt(o,\n+\t\t\t\tDIFF_FILE_OLD_MOVED + use_alt_color);\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tdie(\"BUG: we should have continued earlier?\");\n+\t\t}\n+\t}\n+\tfree(pmb);\n+}\n+\n static void emit_diff_line(struct diff_options *o,\n \t\t\t\t struct diff_line *e)\n {\n@@ -3518,6 +3752,8 @@ void diff_setup(struct diff_options *options)\n \toptions->line_buffer = NULL;\n \toptions->line_buffer_nr = 0;\n \toptions->line_buffer_alloc = 0;\n+\n+\toptions->color_moved = diff_color_moved_default;\n }\n \n void diff_setup_done(struct diff_options *options)\n@@ -3627,6 +3863,9 @@ void diff_setup_done(struct diff_options *options)\n \n \tif (DIFF_OPT_TST(options, FOLLOW_RENAMES) && options->pathspec.nr != 1)\n \t\tdie(_(\"--follow requires exactly one pathspec\"));\n+\n+\tif (!options->use_color || external_diff())\n+\t\toptions->color_moved = 0;\n }\n \n static int opt_arg(const char *arg, int arg_short, const char *arg_long, int *val)\n@@ -4051,6 +4290,10 @@ int diff_opt_parse(struct diff_options *options,\n \t}\n \telse if (!strcmp(arg, \"--no-color\"))\n \t\toptions->use_color = 0;\n+\telse if (!strcmp(arg, \"--color-moved\"))\n+\t\toptions->color_moved = 1;\n+\telse if (!strcmp(arg, \"--no-color-moved\"))\n+\t\toptions->color_moved = 0;\n \telse if (!strcmp(arg, \"--color-words\")) {\n \t\toptions->use_color = 1;\n \t\toptions->word_diff = DIFF_WORDS_COLOR;\n@@ -4856,16 +5099,9 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n {\n \tint i;\n \tstruct diff_queue_struct *q = &diff_queued_diff;\n-\t/*\n-\t * For testing purposes we want to make sure the diff machinery\n-\t * works completely with the buffer. If there is anything emitted\n-\t * outside the emit_diff_line, then the order is screwed\n-\t * up and the tests will fail.\n-\t *\n-\t * TODO (later in this series):\n-\t * We'll unset this flag in a later patch.\n-\t */\n-\to->use_buffer = 1;\n+\n+\tif (o->color_moved)\n+\t\to->use_buffer = 1;\n \n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n@@ -4874,6 +5110,24 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \t}\n \n \tif (o->use_buffer) {\n+\t\tif (o->color_moved) {\n+\t\t\tstruct hashmap add_lines, del_lines;\n+\t\t\tunsigned ignore_ws = DIFF_XDL_TST(o, IGNORE_WHITESPACE);\n+\n+\t\t\thashmap_init(&del_lines, ignore_ws ?\n+\t\t\t\t(hashmap_cmp_fn)moved_entry_cmp_no_ws :\n+\t\t\t\t(hashmap_cmp_fn)moved_entry_cmp, 0);\n+\t\t\thashmap_init(&add_lines, ignore_ws ?\n+\t\t\t\t(hashmap_cmp_fn)moved_entry_cmp_no_ws :\n+\t\t\t\t(hashmap_cmp_fn)moved_entry_cmp, 0);\n+\n+\t\t\tadd_lines_to_move_detection(o, &add_lines, &del_lines);\n+\t\t\tmark_color_as_moved(o, &add_lines, &del_lines);\n+\n+\t\t\thashmap_free(&add_lines, 0);\n+\t\t\thashmap_free(&del_lines, 0);\n+\t\t}\n+\n \t\tfor (i = 0; i < o->line_buffer_nr; i++)\n \t\t\temit_diff_line(o, &o->line_buffer[i]);\n \n@@ -4962,6 +5216,7 @@ void diff_flush(struct diff_options *options)\n \t\tif (!options->file)\n \t\t\tdie_errno(\"Could not open /dev/null\");\n \t\toptions->close_file = 1;\n+\t\toptions->color_moved = 0;\n \t\tfor (i = 0; i < q->nr; i++) {\n \t\t\tstruct diff_filepair *p = q->queue[i];\n \t\t\tif (check_pair_status(p))\ndiff --git a/diff.h b/diff.h\nindex fad1258556..445259ebf7 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -7,6 +7,7 @@\n #include \"tree-walk.h\"\n #include \"pathspec.h\"\n #include \"object.h\"\n+#include \"hashmap.h\"\n \n struct rev_info;\n struct diff_options;\n@@ -228,6 +229,8 @@ struct diff_options {\n \n \tstruct diff_line *line_buffer;\n \tint line_buffer_nr, line_buffer_alloc;\n+\n+\tint color_moved;\n };\n \n /* Emit [line_prefix] [set] line [reset] */\n@@ -243,7 +246,11 @@ enum color_diff {\n \tDIFF_FILE_NEW = 5,\n \tDIFF_COMMIT = 6,\n \tDIFF_WHITESPACE = 7,\n-\tDIFF_FUNCINFO = 8\n+\tDIFF_FUNCINFO = 8,\n+\tDIFF_FILE_OLD_MOVED = 9,\n+\tDIFF_FILE_OLD_MOVED_ALT = 10,\n+\tDIFF_FILE_NEW_MOVED = 11,\n+\tDIFF_FILE_NEW_MOVED_ALT = 12\n };\n const char *diff_get_color(int diff_use_color, enum color_diff ix);\n #define diff_get_color_opt(o, ix) \\\ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 289806d0c7..0e92bf94bf 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -972,4 +972,271 @@ test_expect_success 'option overrides diff.wsErrorHighlight' '\n \n '\n \n+test_expect_success 'detect moved code, complete file' '\n+\tgit reset --hard &&\n+\tcat <<-\\EOF >test.c &&\n+\t#include\n+\tmain()\n+\t{\n+\tprintf(\"Hello World\");\n+\t}\n+\tEOF\n+\tgit add test.c &&\n+\tgit commit -m \"add main function\" &&\n+\tgit mv test.c main.c &&\n+\tgit diff HEAD --color-moved --no-renames | test_decode_color >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tdiff --git a/main.c b/main.c\n+\tnew file mode 100644\n+\tindex 0000000..a986c57\n+\t--- /dev/null\n+\t+++ b/main.c\n+\t@@ -0,0 +1,5 @@\n+\t+#include\n+\t+main()\n+\t+{\n+\t+printf(\"Hello World\");\n+\t+}\n+\tdiff --git a/test.c b/test.c\n+\tdeleted file mode 100644\n+\tindex a986c57..0000000\n+\t--- a/test.c\n+\t+++ /dev/null\n+\t@@ -1,5 +0,0 @@\n+\t-#include\n+\t-main()\n+\t-{\n+\t-printf(\"Hello World\");\n+\t-}\n+\tEOF\n+\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'detect moved code, inside file' '\n+\tgit reset --hard &&\n+\tcat <<-\\EOF >main.c &&\n+\t\t#include\n+\t\tint stuff()\n+\t\t{\n+\t\t\tprintf(\"Hello \");\n+\t\t\tprintf(\"World\\n\");\n+\t\t}\n+\n+\t\tint secure_foo(struct user *u)\n+\t\t{\n+\t\t\tif (!u->is_allowed_foo)\n+\t\t\t\treturn;\n+\t\t\tfoo(u);\n+\t\t}\n+\n+\t\tint main()\n+\t\t{\n+\t\t\tfoo();\n+\t\t}\n+\tEOF\n+\tcat <<-\\EOF >test.c &&\n+\t\t#include\n+\t\tint bar()\n+\t\t{\n+\t\t\tprintf(\"Hello World, but different\\n\");\n+\t\t}\n+\n+\t\tint another_function()\n+\t\t{\n+\t\t\tbar();\n+\t\t}\n+\tEOF\n+\tgit add main.c test.c &&\n+\tgit commit -m \"add main and test file\" &&\n+\tcat <<-\\EOF >main.c &&\n+\t\t#include\n+\t\tint stuff()\n+\t\t{\n+\t\t\tprintf(\"Hello \");\n+\t\t\tprintf(\"World\\n\");\n+\t\t}\n+\n+\t\tint main()\n+\t\t{\n+\t\t\tfoo();\n+\t\t}\n+\tEOF\n+\tcat <<-\\EOF >test.c &&\n+\t\t#include\n+\t\tint bar()\n+\t\t{\n+\t\t\tprintf(\"Hello World, but different\\n\");\n+\t\t}\n+\n+\t\tint secure_foo(struct user *u)\n+\t\t{\n+\t\t\tif (!u->is_allowed_foo)\n+\t\t\t\treturn;\n+\t\t\tfoo(u);\n+\t\t}\n+\n+\t\tint another_function()\n+\t\t{\n+\t\t\tbar();\n+\t\t}\n+\tEOF\n+\tgit diff HEAD --no-renames --color-moved| test_decode_color >actual &&\n+\tcat <<-\\EOF >expected &&\n+\tdiff --git a/main.c b/main.c\n+\tindex 27a619c..7cf9336 100644\n+\t--- a/main.c\n+\t+++ b/main.c\n+\t@@ -5,13 +5,6 @@ printf(\"Hello \");\n+\t printf(\"World\\n\");\n+\t }\n+\t \n+\t-int secure_foo(struct user *u)\n+\t-{\n+\t-if (!u->is_allowed_foo)\n+\t-return;\n+\t-foo(u);\n+\t-}\n+\t-\n+\t int main()\n+\t {\n+\t foo();\n+\tdiff --git a/test.c b/test.c\n+\tindex 1dc1d85..e34eb69 100644\n+\t--- a/test.c\n+\t+++ b/test.c\n+\t@@ -4,6 +4,13 @@ int bar()\n+\t printf(\"Hello World, but different\\n\");\n+\t }\n+\t \n+\t+int secure_foo(struct user *u)\n+\t+{\n+\t+if (!u->is_allowed_foo)\n+\t+return;\n+\t+foo(u);\n+\t+}\n+\t+\n+\t int another_function()\n+\t {\n+\t bar();\n+\tEOF\n+\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'detect permutations inside moved code' '\n+\t# reusing the move example from last test:\n+\tcat <<-\\EOF >main.c &&\n+\t\t#include\n+\t\tint stuff()\n+\t\t{\n+\t\t\tprintf(\"Hello \");\n+\t\t\tprintf(\"World\\n\");\n+\t\t}\n+\n+\t\tint main()\n+\t\t{\n+\t\t\tfoo();\n+\t\t}\n+\tEOF\n+\tcat <<-\\EOF >test.c &&\n+\t\t#include\n+\t\tint bar()\n+\t\t{\n+\t\t\tprintf(\"Hello World, but different\\n\");\n+\t\t}\n+\n+\t\tint secure_foo(struct user *u)\n+\t\t{\n+\t\t\tfoo(u);\n+\t\t\tif (!u->is_allowed_foo)\n+\t\t\t\treturn;\n+\t\t}\n+\n+\t\tint another_function()\n+\t\t{\n+\t\t\tbar();\n+\t\t}\n+\tEOF\n+\tgit diff HEAD --no-renames --color-moved| test_decode_color >actual &&\n+\tcat <<-\\EOF >expected &&\n+\tdiff --git a/main.c b/main.c\n+\tindex 27a619c..7cf9336 100644\n+\t--- a/main.c\n+\t+++ b/main.c\n+\t@@ -5,13 +5,6 @@ printf(\"Hello \");\n+\t printf(\"World\\n\");\n+\t }\n+\t \n+\t-int secure_foo(struct user *u)\n+\t-{\n+\t-if (!u->is_allowed_foo)\n+\t-return;\n+\t-foo(u);\n+\t-}\n+\t-\n+\t int main()\n+\t {\n+\t foo();\n+\tdiff --git a/test.c b/test.c\n+\tindex 1dc1d85..2bedec9 100644\n+\t--- a/test.c\n+\t+++ b/test.c\n+\t@@ -4,6 +4,13 @@ int bar()\n+\t printf(\"Hello World, but different\\n\");\n+\t }\n+\t \n+\t+int secure_foo(struct user *u)\n+\t+{\n+\t+foo(u);\n+\t+if (!u->is_allowed_foo)\n+\t+return;\n+\t+}\n+\t+\n+\t int another_function()\n+\t {\n+\t bar();\n+\tEOF\n+\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'move detection does not mess up colored words' '\n+\tcat <<-\\EOF >text.txt &&\n+\tLorem Ipsum is simply dummy text of the printing and typesetting industry.\n+\tEOF\n+\tgit add text.txt &&\n+\tgit commit -a -m \"clean state\" &&\n+\tcat <<-\\EOF >text.txt &&\n+\tsimply Lorem Ipsum dummy is text of the typesetting and printing industry.\n+\tEOF\n+\tgit diff --color-moved --word-diff >actual &&\n+\tgit diff --word-diff >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'move detection with submodules' '\n+\ttest_create_repo bananas &&\n+\techo ripe >bananas/recipe &&\n+\tgit -C bananas add recipe &&\n+\ttest_commit fruit &&\n+\ttest_commit -C bananas recipe &&\n+\tgit submodule add ./bananas &&\n+\tgit add bananas &&\n+\tgit commit -a -m \"bananas are like a heavy library?\" &&\n+\techo foul >bananas/recipe &&\n+\techo ripe >fruit.t &&\n+\n+\tgit diff --submodule=diff --color-moved >actual &&\n+\n+\t# no move detection as the moved line is across repository boundaries.\n+\ttest_decode_color decoded_actual &&\n+\t! grep BGREEN decoded_actual &&\n+\t! grep BRED decoded_actual &&\n+\n+\t# nor did we mess with it another way\n+\tgit diff --submodule=diff | test_decode_color >expect &&\n+\ttest_cmp expect decoded_actual\n+'\n+\n test_done\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320511","messageId":"20170523024048.16879-4-sbeller@google.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"[PATCHv4 03/17] diff.c: factor out diff_flush_patch_all_file_pairs","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T02:40:34Z","receivedAt":"2017-05-23T02:42:04Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"In a later patch we want to do more things before and after all filepairs\nare flushed. So factor flushing out all file pairs into its own function\nthat the new code can be plugged in easily.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n diff.c | 17 ++++++++++++-----\n 1 file changed, 12 insertions(+), 5 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex c2ed605cd0..2f9722b382 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4737,6 +4737,17 @@ void diff_warn_rename_limit(const char *varname, int needed, int degraded_cc)\n \t\twarning(_(rename_limit_advice), varname, needed);\n }\n \n+static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n+{\n+\tint i;\n+\tstruct diff_queue_struct *q = &diff_queued_diff;\n+\tfor (i = 0; i < q->nr; i++) {\n+\t\tstruct diff_filepair *p = q->queue[i];\n+\t\tif (check_pair_status(p))\n+\t\t\tdiff_flush_patch(p, o);\n+\t}\n+}\n+\n void diff_flush(struct diff_options *options)\n {\n \tstruct diff_queue_struct *q = &diff_queued_diff;\n@@ -4831,11 +4842,7 @@ void diff_flush(struct diff_options *options)\n \t\t\t}\n \t\t}\n \n-\t\tfor (i = 0; i < q->nr; i++) {\n-\t\t\tstruct diff_filepair *p = q->queue[i];\n-\t\t\tif (check_pair_status(p))\n-\t\t\t\tdiff_flush_patch(p, options);\n-\t\t}\n+\t\tdiff_flush_patch_all_file_pairs(options);\n \t}\n \n \tif (output_format & DIFF_FORMAT_CALLBACK)\n-- \n2.13.0.18.g7d86cc8ba0\n\n"},{"id":"320524","messageId":"xmqqinksktaj.fsf@gitster.mtv.corp.google.com","threadId":"45958","inReplyTo":"20170523024048.16879-10-sbeller@google.com","subject":"Re: [PATCHv4 09/17] submodule.c: convert show_submodule_summary to use emit_line_fmt","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-23T05:59:16Z","receivedAt":"2017-05-23T05:59:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> diff --git a/submodule.c b/submodule.c\n> index d3299e29c0..428c996c97 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> ...\n> @@ -547,15 +543,16 @@ void show_submodule_inline_diff(FILE *f, const char *path,\n> \tif (right)\n> \t\tnew = two;\n> \n> -\tfflush(f);\n> \tcp.git_cmd = 1;\n> \tcp.dir = path;\n> -\tcp.out = dup(fileno(f));\n> +\tcp.out = -1;\n> \tcp.no_stdin = 1;\n> \n> \t/* TODO: other options may need to be passed here. */\n> \targv_array_push(&cp.args, \"diff\");\n> -\targv_array_pushf(&cp.args, \"--line-prefix=%s\", line_prefix);\n> +\tif (o->use_color)\n> +\t\targv_array_push(&cp.args, \"--color=always\");\n> +\targv_array_pushf(&cp.args, \"--line-prefix=%s\", diff_line_prefix(o));\n\nThis makes me wonder if we also need to explicitly decline coloring\nwhen o->use_color is not set. After all, even if configuration in\nthe submodule's config file says diff.color=never, we will enable\nthe color with this codepath (because the user explicitly asked to\nuse the color in the top-level), so we should do the same for the\nopposite case where the config says yes/auto if the user said no at\nthe top-level, no?\n"},{"id":"320566","messageId":"CAGZ79kZ4_giLxXeRLZRDbCzNK754QthscthmzaU02oSpv3nw8g@mail.gmail.com","threadId":"45958","inReplyTo":"xmqqinksktaj.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCHv4 09/17] submodule.c: convert show_submodule_summary to use emit_line_fmt","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-23T18:14:04Z","receivedAt":"2017-05-23T18:14:20Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, May 22, 2017 at 10:59 PM, Junio C Hamano wrote:\n> Stefan Beller writes:\n>\n>> diff --git a/submodule.c b/submodule.c\n>> index d3299e29c0..428c996c97 100644\n>> --- a/submodule.c\n>> +++ b/submodule.c\n>> ...\n>> @@ -547,15 +543,16 @@ void show_submodule_inline_diff(FILE *f, const char *path,\n>> if (right)\n>> new = two;\n>>\n>> - fflush(f);\n>> cp.git_cmd = 1;\n>> cp.dir = path;\n>> - cp.out = dup(fileno(f));\n>> + cp.out = -1;\n>> cp.no_stdin = 1;\n>>\n>> /* TODO: other options may need to be passed here. */\n>> argv_array_push(&cp.args, \"diff\");\n>> - argv_array_pushf(&cp.args, \"--line-prefix=%s\", line_prefix);\n>> + if (o->use_color)\n>> + argv_array_push(&cp.args, \"--color=always\");\n>> + argv_array_pushf(&cp.args, \"--line-prefix=%s\", diff_line_prefix(o));\n>\n> This makes me wonder if we also need to explicitly decline coloring\n> when o->use_color is not set. After all, even if configuration in\n> the submodule's config file says diff.color=never, we will enable\n> the color with this codepath (because the user explicitly asked to\n> use the color in the top-level), so we should do the same for the\n> opposite case where the config says yes/auto if the user said no at\n> the top-level, no?\n\nThat makes sense, so instead we'd do\n\n argv_array_push(&cp.args, \"--color=%s\", o->use_color ?\n\"always\" : \"never\");\n\nto override the submodule config in all cases.\n\nHowever that changes from current behavior.\n\nYou could imagine that you want to see the superproject colored\nand the submodule non-colored to easily spot that it is a submodule change.\nCurrently this can be made to work via setting color=never in the\nsubmodule and then run the diff from the superproject.\n\nWhat we really want here is a switch that influences the automatic detection\nand say: pretend \"dup(fileno(f));\" was your stdout, now run your auto-detection\nto decide for yourself.\n\nI am not sure if it worth the effort to fix this hypothetical situation, though.\n\nThanks,\nStefan\n"},{"id":"320853","messageId":"CA+P7+xrFjwUWNA9BQ3V4ad6mizrfXTcU4td+u1-i9GCBshPjfQ@mail.gmail.com","threadId":"45958","inReplyTo":"20170523024048.16879-1-sbeller@google.com","subject":"Re: [PATCHv4 00/17] Diff machine: highlight moved lines.","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2017-05-27T01:04:23Z","receivedAt":"2017-05-27T01:48:26Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Mon, May 22, 2017 at 7:40 PM, Stefan Beller wrote:\n> v4:\n> * interdiff to v3 (what is currently origin/sb/diff-color-move) below.\n> * renamed the \"buffered_patch_line\" to \"diff_line\". Originally I planned\n> to not carry the \"line\" part as it can be a piece of a line as well.\n> But for the intended functionality it is best to keep the name.\n> If we'd want to add more functionality to say have a move detection\n> for words as well, we'd rename the struct to have a better name then.\n> For now diff_line is the best. (Thanks Jonathan Nieder!)\n> * tests to demonstrate it doesn't mess with --color-words as well as\n> submodules. (Thanks Jonathan Tan!)\n> * added in the statics (Thanks Ramsay!)\n> * smaller scope for the hashmaps (Thanks Jonathan Tan!)\n> * some commit messages were updated, prior patch 4-7 is squashed into one\n> (Thanks Jonathan Tan!)\n> * the tests added revealed an actual fault: now that the submodule process\n> is not attached to a dupe of our stdout, it would stop coloring the\n> output. We need to pass on use-color explicitly.\n> * updated the NEEDSWORK comment in the second last patch.\n>\n> Thanks for bearing,\n> Stefan\n>\n\nOne thing to note when I was playing around with what's on pu right\nnow, I noticed that the oldMovedAlternative and newMovedAlternative\nare the first moved colors to be used if there is only one move. (Ie:\na simple case of literally one section moved) This is a bit weird that\nthe alternative colors are used before the \"main\" colors. I would have\nthought it would be the other way.\n\nI noticed this because the default colors do not work well for my\nterminal color scheme and I had to configure but realized that I\nneeded to configure the alternative ones to make a difference in the\nsimple diff I was viewing.\n\nThanks,\nJake\n"},{"id":"321084","messageId":"CAGZ79kYiv_t+EEGoPGE5-ZRCXbTEyeodjpNCLMXVMq04sW224Q@mail.gmail.com","threadId":"45958","inReplyTo":"CA+P7+xrFjwUWNA9BQ3V4ad6mizrfXTcU4td+u1-i9GCBshPjfQ@mail.gmail.com","subject":"Re: [PATCHv4 00/17] Diff machine: highlight moved lines.","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-05-30T21:38:21Z","receivedAt":"2017-05-30T21:38:26Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, May 26, 2017 at 6:04 PM, Jacob Keller wrote:\n> On Mon, May 22, 2017 at 7:40 PM, Stefan Beller wrote:\n>> v4:\n>> * interdiff to v3 (what is currently origin/sb/diff-color-move) below.\n>> * renamed the \"buffered_patch_line\" to \"diff_line\". Originally I planned\n>> to not carry the \"line\" part as it can be a piece of a line as well.\n>> But for the intended functionality it is best to keep the name.\n>> If we'd want to add more functionality to say have a move detection\n>> for words as well, we'd rename the struct to have a better name then.\n>> For now diff_line is the best. (Thanks Jonathan Nieder!)\n>> * tests to demonstrate it doesn't mess with --color-words as well as\n>> submodules. (Thanks Jonathan Tan!)\n>> * added in the statics (Thanks Ramsay!)\n>> * smaller scope for the hashmaps (Thanks Jonathan Tan!)\n>> * some commit messages were updated, prior patch 4-7 is squashed into one\n>> (Thanks Jonathan Tan!)\n>> * the tests added revealed an actual fault: now that the submodule process\n>> is not attached to a dupe of our stdout, it would stop coloring the\n>> output. We need to pass on use-color explicitly.\n>> * updated the NEEDSWORK comment in the second last patch.\n>>\n>> Thanks for bearing,\n>> Stefan\n>>\n>\n> One thing to note when I was playing around with what's on pu right\n> now, I noticed that the oldMovedAlternative and newMovedAlternative\n> are the first moved colors to be used if there is only one move. (Ie:\n> a simple case of literally one section moved) This is a bit weird that\n> the alternative colors are used before the \"main\" colors. I would have\n> thought it would be the other way.\n\nWhile pu is not up-to-date, I double checked with the most recent\nimplementation and that is no longer the case.\n\n> I noticed this because the default colors do not work well for my\n> terminal color scheme and I had to configure but realized that I\n> needed to configure the alternative ones to make a difference in the\n> simple diff I was viewing.\n\nThe v4 that you tested, is the \"alternate\" scheme in the resend\nhttps://public-inbox.org/git/20170527001820.25214-2-sbeller@google.com/\n\nThanks,\nStefan\n"}]}