{"thread":{"id":"55926","subject":"[PATCH 00/10] diff --color-moved[-ws] speedups","startedAt":"2021-06-14T13:04:54Z","lastAt":"2021-12-09T10:30:43Z","messageCount":92,"participants":["Phillip Wood via GitGitGadget","Junio C Hamano","Phillip Wood","Ævar Arnfjörð Bjarmason","Elijah Newren","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":10},"messages":[{"id":"427338","messageId":"pull.981.git.1623675888.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":null,"subject":"[PATCH 00/10] diff --color-moved[-ws] speedups","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-14T13:04:38Z","receivedAt":"2021-06-14T13:04:54Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"The current implementation of diff --color-moved-ws=allow-indentation-change\nis considerably slower that the implementation of diff --color-moved which\nis in turn slower than a regular diff. This patch series starts with a\ncouple of bug fixes and then reworks the implementation of diff\n--color-moved and diff --color-moved-ws=allow-indentation-change to speed\nthem up on large diffs. The time to run git diff --color-moved\n--no-color-moved-ws v2.28.0 v2.29.0 is reduced by 33% and the time to run\ngit diff --color-moved --color-moved-ws=allow-indentation-change v2.28.0\nv2.29.0 is reduced by 88%. There is a small slowdown for commit sized diffs\nwith --color-moved - the time to run git log -p --color-moved\n--no-color-moved-ws --no-merges -n1000 v2.29.0 is increased by 2% on recent\nprocessors. On older processors these patches reduce the running time in all\ncases that I've tested. In general the larger the diff the larger the speed\nup. As an extreme example the time to run diff --color-moved\n--color-moved-ws=allow-indentation-change v2.25.0 v2.30.0 goes down from 8\nminutes to 6 seconds.\n\nPhillip Wood (10):\n diff --color-moved=zerba: fix alternate coloring\n diff --color-moved: avoid false short line matches and bad zerba\n coloring\n diff: simplify allow-indentation-change delta calculation\n diff --color-moved-ws=allow-indentation-change: simplify and optimize\n diff --color-moved: call comparison function directly\n diff --color-moved: unify moved block growth functions\n diff --color-moved: shrink potential moved blocks as we go\n diff --color-moved: stop clearing potential moved blocks\n diff --color-moved-ws=allow-indentation-change: improve hash lookups\n diff --color-moved: intern strings\n\n diff.c | 375 ++++++++++++++-----------------------\n t/t4015-diff-whitespace.sh | 137 ++++++++++++++\n 2 files changed, 276 insertions(+), 236 deletions(-)\n\n\nbase-commit: 211eca0895794362184da2be2a2d812d070719d3\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-981%2Fphillipwood%2Fwip%2Fdiff-color-moved-tweaks-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-981/phillipwood/wip/diff-color-moved-tweaks-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/981\n-- \ngitgitgadget\n"},{"id":"427339","messageId":"374dbebcbf29b686508e51205b2f7a4e72104950.1623675888.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.git.1623675888.gitgitgadget@gmail.com","subject":"[PATCH 01/10] diff --color-moved=zerba: fix alternate coloring","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-14T13:04:39Z","receivedAt":"2021-06-14T13:04:55Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nb0a2ba4776 (\"diff --color-moved=zebra: be stricter with color\nalternation\", 2018-11-23) sought to avoid using the alternate colors\nunless there are two adjacent moved blocks of the same\nsign. Unfortunately it contains two bugs that prevented it from fixing\nthe problem properly. Firstly `last_symbol` is reset at the start of\neach iteration of the loop losing the symbol of the last line and\nsecondly when deciding whether to use the alternate color it should be\nchecking if the current line is the same sign of the last line, not a\ndifferent sign. The combination of the two errors means that we still\nuse the alternate color when we should do but we also use it when we\nshouldn't. This is most noticable when using\n--color-moved-ws=allow-indentation-change with hunks like\n\n-this line gets indented\n+ this line gets indented\n\nwhere the post image is colored with newMovedAlternate rather than\nnewMoved. While this does not matter much, the next commit will change\nthe coloring to be correct in this case, so lets fix the bug here to\nmake it clear why the output is changing and add a regression test.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 4 +--\n t/t4015-diff-whitespace.sh | 72 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 74 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 52c791574b71..cb068f8258c0 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1142,6 +1142,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n \tint n, flipped_block = 0, block_length = 0;\n+\tenum diff_symbol last_symbol = 0;\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n@@ -1149,7 +1150,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\tstruct moved_entry *key;\n \t\tstruct moved_entry *match = NULL;\n \t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n-\t\tenum diff_symbol last_symbol = 0;\n \n \t\tswitch (l->s) {\n \t\tcase DIFF_SYMBOL_PLUS:\n@@ -1214,7 +1214,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t}\n \n \t\t\tif (adjust_last_block(o, n, block_length) &&\n-\t\t\t pmb_nr && last_symbol != l->s)\n+\t\t\t pmb_nr && last_symbol == l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n \t\t\telse\n \t\t\t\tflipped_block = 0;\ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 2c13b62d3c65..920114cd795c 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1442,6 +1442,78 @@ test_expect_success 'detect permutations inside moved code -- dimmed-zebra' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'zebra alternate color is only used when necessary' '\n+\tcat >old.txt <<-\\EOF &&\n+\tline 1A should be marked as oldMoved newMovedAlternate\n+\tline 1B should be marked as oldMoved newMovedAlternate\n+\tunchanged\n+\tline 2A should be marked as oldMoved newMovedAlternate\n+\tline 2B should be marked as oldMoved newMovedAlternate\n+\tline 3A should be marked as oldMovedAlternate newMoved\n+\tline 3B should be marked as oldMovedAlternate newMoved\n+\tunchanged\n+\tline 4A should be marked as oldMoved newMovedAlternate\n+\tline 4B should be marked as oldMoved newMovedAlternate\n+\tline 5A should be marked as oldMovedAlternate newMoved\n+\tline 5B should be marked as oldMovedAlternate newMoved\n+\tline 6A should be marked as oldMoved newMoved\n+\tline 6B should be marked as oldMoved newMoved\n+\tEOF\n+\tcat >new.txt <<-\\EOF &&\n+\t line 1A should be marked as oldMoved newMovedAlternate\n+\t line 1B should be marked as oldMoved newMovedAlternate\n+\tunchanged\n+\t line 3A should be marked as oldMovedAlternate newMoved\n+\t line 3B should be marked as oldMovedAlternate newMoved\n+\t line 2A should be marked as oldMoved newMovedAlternate\n+\t line 2B should be marked as oldMoved newMovedAlternate\n+\tunchanged\n+\t line 6A should be marked as oldMoved newMoved\n+\t line 6B should be marked as oldMoved newMoved\n+\t line 4A should be marked as oldMoved newMovedAlternate\n+\t line 4B should be marked as oldMoved newMovedAlternate\n+\t line 5A should be marked as oldMovedAlternate newMoved\n+\t line 5B should be marked as oldMovedAlternate newMoved\n+\tEOF\n+\ttest_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n+\t\t --color-moved-ws=allow-indentation-change \\\n+\t\t old.txt new.txt >output &&\n+\tgrep -v index output | test_decode_color >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tdiff --git a/old.txt b/new.txt\n+\t--- a/old.txt\n+\t+++ b/new.txt\n+\t@@ -1,14 +1,14 @@\n+\t-line 1A should be marked as oldMoved newMovedAlternate\n+\t-line 1B should be marked as oldMoved newMovedAlternate\n+\t+ line 1A should be marked as oldMoved newMovedAlternate\n+\t+ line 1B should be marked as oldMoved newMovedAlternate\n+\t unchanged\n+\t-line 2A should be marked as oldMoved newMovedAlternate\n+\t-line 2B should be marked as oldMoved newMovedAlternate\n+\t-line 3A should be marked as oldMovedAlternate newMoved\n+\t-line 3B should be marked as oldMovedAlternate newMoved\n+\t+ line 3A should be marked as oldMovedAlternate newMoved\n+\t+ line 3B should be marked as oldMovedAlternate newMoved\n+\t+ line 2A should be marked as oldMoved newMovedAlternate\n+\t+ line 2B should be marked as oldMoved newMovedAlternate\n+\t unchanged\n+\t-line 4A should be marked as oldMoved newMovedAlternate\n+\t-line 4B should be marked as oldMoved newMovedAlternate\n+\t-line 5A should be marked as oldMovedAlternate newMoved\n+\t-line 5B should be marked as oldMovedAlternate newMoved\n+\t-line 6A should be marked as oldMoved newMoved\n+\t-line 6B should be marked as oldMoved newMoved\n+\t+ line 6A should be marked as oldMoved newMoved\n+\t+ line 6B should be marked as oldMoved newMoved\n+\t+ line 4A should be marked as oldMoved newMovedAlternate\n+\t+ line 4B should be marked as oldMoved newMovedAlternate\n+\t+ line 5A should be marked as oldMovedAlternate newMoved\n+\t+ line 5B should be marked as oldMovedAlternate newMoved\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'cmd option assumes configured colored-moved' '\n \ttest_config color.diff.oldMoved \"magenta\" &&\n \ttest_config color.diff.newMoved \"cyan\" &&\n-- \ngitgitgadget\n\n"},{"id":"427340","messageId":"9390e9a66ebf3c2ccd2b47958134dee8c3c9c553.1623675889.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.git.1623675888.gitgitgadget@gmail.com","subject":"[PATCH 07/10] diff --color-moved: shrink potential moved blocks as we go","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-14T13:04:45Z","receivedAt":"2021-06-14T13:05:05Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nRather than setting `match` to NULL and then looping over the list of\npotential matched blocks for a second time to remove blocks with no\nmatches just filter out the blocks with no matches as we go.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 42 ++++++------------------------------------\n 1 file changed, 6 insertions(+), 36 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex de6522a3a860..f60cce654c14 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -997,12 +997,12 @@ static void add_lines_to_move_detection(struct diff_options *o,\n static void pmb_advance_or_null(struct diff_options *o,\n \t\t\t\tstruct emitted_diff_symbol *l,\n \t\t\t\tstruct moved_block *pmb,\n-\t\t\t\tint pmb_nr)\n+\t\t\t\tint *pmb_nr)\n {\n-\tint i;\n+\tint i, j;\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \n-\tfor (i = 0; i < pmb_nr; i++) {\n+\tfor (i = 0, j = 0; i < *pmb_nr; i++) {\n \t\tint match;\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n@@ -1017,37 +1017,9 @@ static void pmb_advance_or_null(struct diff_options *o,\n \t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n \t\t\t\t\t\t l->line, l->len, flags);\n \t\tif (match)\n-\t\t\tpmb[i].match = cur;\n-\t\telse\n-\t\t\tmoved_block_clear(&pmb[i]);\n+\t\t\tpmb[j++].match = cur;\n \t}\n-}\n-\n-static int shrink_potential_moved_blocks(struct moved_block *pmb,\n-\t\t\t\t\t int pmb_nr)\n-{\n-\tint lp, rp;\n-\n-\t/* Shrink the set of potential block to the remaining running */\n-\tfor (lp = 0, rp = pmb_nr - 1; lp <= rp;) {\n-\t\twhile (lp < pmb_nr && pmb[lp].match)\n-\t\t\tlp++;\n-\t\t/* lp points at the first NULL now */\n-\n-\t\twhile (rp > -1 && !pmb[rp].match)\n-\t\t\trp--;\n-\t\t/* rp points at the last non-NULL */\n-\n-\t\tif (lp < pmb_nr && rp > -1 && lp < rp) {\n-\t\t\tpmb[lp] = pmb[rp];\n-\t\t\tmemset(&pmb[rp], 0, sizeof(pmb[rp]));\n-\t\t\trp--;\n-\t\t\tlp++;\n-\t\t}\n-\t}\n-\n-\t/* Remember the number of running sets */\n-\treturn rp + 1;\n+\t*pmb_nr = j;\n }\n \n /*\n@@ -1140,9 +1112,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n-\n-\t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n+\t\tpmb_advance_or_null(o, l, pmb, &pmb_nr);\n \n \t\tif (pmb_nr == 0) {\n \t\t\t/*\n-- \ngitgitgadget\n\n"},{"id":"427341","messageId":"41cdedd60907b966dffa6cf0c9825ffb448f4971.1623675889.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.git.1623675888.gitgitgadget@gmail.com","subject":"[PATCH 09/10] diff --color-moved-ws=allow-indentation-change: improve hash lookups","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-14T13:04:47Z","receivedAt":"2021-06-14T13:05:07Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nAs libxdiff does not have a whitespace flag to ignore the indentation\nthe code for --color-moved-ws=allow-indentation-change uses\nXDF_IGNORE_WHITESPACE and then filters out any hash lookups where\nthere are non-indentation changes. This is filtering is inefficient as\nwe have to perform another string comparison.\n\nBy using the offset data that we have already computed to skip the\nindentation we can avoid using XDF_IGNORE_WHITESPACE and safely remove\nthe extra checks which improves the performance by 14% and paves the\nway for the elimination of string comparisons in the next commit.\n\nThis change slightly increases the runtime of other --color-moved\nmodes. This could be avoided by using different comparison functions\nfor the different modes but after the changes in the next commit there\nis no measurable benefit.\n\nBefore this change\nBenchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n Time (mean ± σ): 1.116 s ± 0.005 s [User: 1.057 s, System: 0.056 s]\n Range (min … max): 1.109 s … 1.123 s 10 runs\n\nBenchmark #2: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n Time (mean ± σ): 1.216 s ± 0.005 s [User: 1.155 s, System: 0.059 s]\n Range (min … max): 1.206 s … 1.223 s 10 runs\n\nAfter this change\nBenchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n Time (mean ± σ): 1.147 s ± 0.005 s [User: 1.085 s, System: 0.059 s]\n Range (min … max): 1.140 s … 1.154 s 10 runs\n\nBenchmark #2: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n Time (mean ± σ): 1.048 s ± 0.005 s [User: 987.4 ms, System: 58.8 ms]\n Range (min … max): 1.043 s … 1.056 s 10 runs\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 66 +++++++++++++++++-----------------------------------------\n 1 file changed, 19 insertions(+), 47 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex ee58373f55f8..e6f3586b39bf 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -850,28 +850,15 @@ static void fill_es_indent_data(struct emitted_diff_symbol *es)\n }\n \n static int compute_ws_delta(const struct emitted_diff_symbol *a,\n-\t\t\t const struct emitted_diff_symbol *b,\n-\t\t\t int *out)\n-{\n-\tint a_len = a->len,\n-\t b_len = b->len,\n-\t a_off = a->indent_off,\n-\t a_width = a->indent_width,\n-\t b_off = b->indent_off,\n+\t\t\t const struct emitted_diff_symbol *b)\n+{\n+\tint a_width = a->indent_width,\n \t b_width = b->indent_width;\n \n-\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE) {\n-\t\t*out = INDENT_BLANKLINE;\n-\t\treturn 1;\n-\t}\n-\n-\tif (a_len - a_off != b_len - b_off ||\n-\t memcmp(a->line + a_off, b->line + b_off, a_len - a_off))\n-\t\treturn 0;\n-\n-\t*out = a_width - b_width;\n+\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n+\t\treturn INDENT_BLANKLINE;\n \n-\treturn 1;\n+\treturn a_width - b_width;\n }\n \n static int cmp_in_block_with_wsd(const struct diff_options *o,\n@@ -917,26 +904,17 @@ static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n \t\t\t const void *keydata)\n {\n \tconst struct diff_options *diffopt = hashmap_cmp_fn_data;\n-\tconst struct moved_entry *a, *b;\n+\tconst struct emitted_diff_symbol *a, *b;\n \tunsigned flags = diffopt->color_moved_ws_handling\n \t\t\t & XDF_WHITESPACE_FLAGS;\n \n-\ta = container_of(eptr, const struct moved_entry, ent);\n-\tb = container_of(entry_or_key, const struct moved_entry, ent);\n-\n-\tif (diffopt->color_moved_ws_handling &\n-\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t/*\n-\t\t * As there is not specific white space config given,\n-\t\t * we'd need to check for a new block, so ignore all\n-\t\t * white space. The setup of the white space\n-\t\t * configuration for the next block is done else where\n-\t\t */\n-\t\tflags |= XDF_IGNORE_WHITESPACE;\n+\ta = container_of(eptr, const struct moved_entry, ent)->es;\n+\tb = container_of(entry_or_key, const struct moved_entry, ent)->es;\n \n-\treturn !xdiff_compare_lines(a->es->line, a->es->len,\n-\t\t\t\t b->es->line, b->es->len,\n-\t\t\t\t flags);\n+\treturn !xdiff_compare_lines(a->line + a->indent_off,\n+\t\t\t\t a->len - a->indent_off,\n+\t\t\t\t b->line + b->indent_off,\n+\t\t\t\t b->len - b->indent_off, flags);\n }\n \n static struct moved_entry *prepare_entry(struct diff_options *o,\n@@ -945,7 +923,8 @@ static struct moved_entry *prepare_entry(struct diff_options *o,\n \tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n \tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[line_no];\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n-\tunsigned int hash = xdiff_hash_string(l->line, l->len, flags);\n+\tunsigned int hash = xdiff_hash_string(l->line + l->indent_off,\n+\t\t\t\t\t l->len - l->indent_off, flags);\n \n \thashmap_entry_init(&ret->ent, hash);\n \tret->es = l;\n@@ -1113,14 +1092,11 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\thashmap_for_each_entry_from(hm, match, ent) {\n \t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n \t\t\t\tif (o->color_moved_ws_handling &\n-\t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n-\t\t\t\t\tif (compute_ws_delta(l, match->es,\n-\t\t\t\t\t\t\t &pmb[pmb_nr].wsd))\n-\t\t\t\t\t\tpmb[pmb_nr++].match = match;\n-\t\t\t\t} else {\n+\t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n+\t\t\t\t\tpmb[pmb_nr].wsd = compute_ws_delta(l, match->es);\n+\t\t\t\telse\n \t\t\t\t\tpmb[pmb_nr].wsd = 0;\n-\t\t\t\t\tpmb[pmb_nr++].match = match;\n-\t\t\t\t}\n+\t\t\t\tpmb[pmb_nr++].match = match;\n \t\t\t}\n \n \t\t\tif (adjust_last_block(o, n, block_length) &&\n@@ -6240,10 +6216,6 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \t\tif (o->color_moved) {\n \t\t\tstruct hashmap add_lines, del_lines;\n \n-\t\t\tif (o->color_moved_ws_handling &\n-\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\t\to->color_moved_ws_handling |= XDF_IGNORE_WHITESPACE;\n-\n \t\t\thashmap_init(&del_lines, moved_entry_cmp, o, 0);\n \t\t\thashmap_init(&add_lines, moved_entry_cmp, o, 0);\n \n-- \ngitgitgadget\n\n"},{"id":"427342","messageId":"3d02a0a91a086417f9dec4823255f50644e3aa87.1623675889.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.git.1623675888.gitgitgadget@gmail.com","subject":"[PATCH 02/10] diff --color-moved: avoid false short line matches and bad zerba coloring","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-14T13:04:40Z","receivedAt":"2021-06-14T13:05:17Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nWhen marking moved lines it is possible for a block of potential\nmatched lines to extend past a change in sign when there is a sequence\nof added lines whose text matches the text of a sequence of deleted\nand added lines. Most of the time either `match` will be NULL or\n`pmb_advance_or_null()` will fail when the loop encounters a change of\nsign but there are corner cases where `match` is non-NULL and\n`pmb_advance_or_null()` successfully advances the moved block despite\nthe change in sign.\n\nOne consequence of this is highlighting a short line as moved when it\nshould not be. For example\n\n-moved line # Correctly highlighted as moved\n+short line # Wrongly highlighted as moved\n context\n+moved line # Correctly highlighted as moved\n+short line\n context\n-short line\n\nThe other consequence is coloring a moved addition following a moved\ndeletion in the wrong color. In the example below the first \"+moved\nline 3\" should be highlighted as newMoved not newMovedAlternate.\n\n-moved line 1 # Correctly highlighted as oldMoved\n-moved line 2 # Correctly highlighted as oldMovedAlternate\n+moved line 3 # Wrongly highlighted as newMovedAlternate\n context # Everything else is highlighted correctly\n+moved line 2\n+moved line 3\n context\n+moved line 1\n-moved line 3\n\nThese false matches are more likely when using --color-moved-ws with\nthe exception of --color-moved-ws=allow-indentation-change which ties\nthe sign of the current whitespace delta to the sign of the line to\navoid this problem. The fix is to check that the sign of the new line\nbeing matched is the same as the sign of the line that started the\nblock of potential matches.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 17 ++++++----\n t/t4015-diff-whitespace.sh | 65 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 76 insertions(+), 6 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex cb068f8258c0..a0c43a104768 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1142,7 +1142,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n \tint n, flipped_block = 0, block_length = 0;\n-\tenum diff_symbol last_symbol = 0;\n+\tenum diff_symbol moved_symbol = 0;\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n@@ -1168,7 +1168,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tflipped_block = 0;\n \t\t}\n \n-\t\tif (!match) {\n+\t\tif (pmb_nr && (!match || l->s != moved_symbol)) {\n \t\t\tint i;\n \n \t\t\tadjust_last_block(o, n, block_length);\n@@ -1177,12 +1177,13 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tpmb_nr = 0;\n \t\t\tblock_length = 0;\n \t\t\tflipped_block = 0;\n-\t\t\tlast_symbol = l->s;\n+\t\t}\n+\t\tif (!match) {\n+\t\t\tmoved_symbol = 0;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (o->color_moved == COLOR_MOVED_PLAIN) {\n-\t\t\tlast_symbol = l->s;\n \t\t\tl->flags |= DIFF_SYMBOL_MOVED_LINE;\n \t\t\tcontinue;\n \t\t}\n@@ -1214,11 +1215,16 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t}\n \n \t\t\tif (adjust_last_block(o, n, block_length) &&\n-\t\t\t pmb_nr && last_symbol == l->s)\n+\t\t\t pmb_nr && moved_symbol == l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n \t\t\telse\n \t\t\t\tflipped_block = 0;\n \n+\t\t\tif (pmb_nr)\n+\t\t\t\tmoved_symbol = l->s;\n+\t\t\telse\n+\t\t\t\tmoved_symbol = 0;\n+\n \t\t\tblock_length = 0;\n \t\t}\n \n@@ -1228,7 +1234,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tif (flipped_block && o->color_moved != COLOR_MOVED_BLOCKS)\n \t\t\t\tl->flags |= DIFF_SYMBOL_MOVED_LINE_ALT;\n \t\t}\n-\t\tlast_symbol = l->s;\n \t}\n \tadjust_last_block(o, n, block_length);\n \ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 920114cd795c..3119a59f071d 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1514,6 +1514,71 @@ test_expect_success 'zebra alternate color is only used when necessary' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'short lines of opposite sign do not get marked as moved' '\n+\tcat >old.txt <<-\\EOF &&\n+\tthis line should be marked as moved\n+\tunchanged\n+\tunchanged\n+\tunchanged\n+\tunchanged\n+\ttoo short\n+\tthis line should be marked as oldMoved newMoved\n+\tthis line should be marked as oldMovedAlternate newMoved\n+\tunchanged 1\n+\tunchanged 2\n+\tunchanged 3\n+\tunchanged 4\n+\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n+\tEOF\n+\tcat >new.txt <<-\\EOF &&\n+\ttoo short\n+\tunchanged\n+\tunchanged\n+\tthis line should be marked as moved\n+\ttoo short\n+\tunchanged\n+\tunchanged\n+\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n+\tunchanged 1\n+\tunchanged 2\n+\tthis line should be marked as oldMovedAlternate newMoved\n+\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n+\tunchanged 3\n+\tthis line should be marked as oldMoved newMoved\n+\tunchanged 4\n+\tEOF\n+\ttest_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n+\t\told.txt new.txt >output && cat output &&\n+\tgrep -v index output | test_decode_color >actual &&\n+\tcat >expect <<-\\EOF &&\n+\tdiff --git a/old.txt b/new.txt\n+\t--- a/old.txt\n+\t+++ b/new.txt\n+\t@@ -1,13 +1,15 @@\n+\t-this line should be marked as moved\n+\t+too short\n+\t unchanged\n+\t unchanged\n+\t+this line should be marked as moved\n+\t+too short\n+\t unchanged\n+\t unchanged\n+\t-too short\n+\t-this line should be marked as oldMoved newMoved\n+\t-this line should be marked as oldMovedAlternate newMoved\n+\t+this line should be marked as oldMoved newMoved/newMovedAlternate\n+\t unchanged 1\n+\t unchanged 2\n+\t+this line should be marked as oldMovedAlternate newMoved\n+\t+this line should be marked as oldMoved newMoved/newMovedAlternate\n+\t unchanged 3\n+\t+this line should be marked as oldMoved newMoved\n+\t unchanged 4\n+\t-this line should be marked as oldMoved newMoved/newMovedAlternate\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'cmd option assumes configured colored-moved' '\n \ttest_config color.diff.oldMoved \"magenta\" &&\n \ttest_config color.diff.newMoved \"cyan\" &&\n-- \ngitgitgadget\n\n"},{"id":"427343","messageId":"30f0ed447683506735ecc37f553b655f23769385.1623675889.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.git.1623675888.gitgitgadget@gmail.com","subject":"[PATCH 03/10] diff: simplify allow-indentation-change delta calculation","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-14T13:04:41Z","receivedAt":"2021-06-14T13:05:54Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nNow that we reliably end a block when the sign changes we don't need\nthe whitespace delta calculation to rely on the sign.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 13 ++-----------\n 1 file changed, 2 insertions(+), 11 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex a0c43a104768..19c8954ec546 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -864,23 +864,17 @@ static int compute_ws_delta(const struct emitted_diff_symbol *a,\n \t a_width = a->indent_width,\n \t b_off = b->indent_off,\n \t b_width = b->indent_width;\n-\tint delta;\n \n \tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE) {\n \t\t*out = INDENT_BLANKLINE;\n \t\treturn 1;\n \t}\n \n-\tif (a->s == DIFF_SYMBOL_PLUS)\n-\t\tdelta = a_width - b_width;\n-\telse\n-\t\tdelta = b_width - a_width;\n-\n \tif (a_len - a_off != b_len - b_off ||\n \t memcmp(a->line + a_off, b->line + b_off, a_len - a_off))\n \t\treturn 0;\n \n-\t*out = delta;\n+\t*out = a_width - b_width;\n \n \treturn 1;\n }\n@@ -924,10 +918,7 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \t * match those of the current block and that the text of 'l' and 'cur'\n \t * after the indentation match.\n \t */\n-\tif (cur->es->s == DIFF_SYMBOL_PLUS)\n-\t\tdelta = a_width - c_width;\n-\telse\n-\t\tdelta = c_width - a_width;\n+\tdelta = c_width - a_width;\n \n \t/*\n \t * If the previous lines of this block were all blank then set its\n-- \ngitgitgadget\n\n"},{"id":"427344","messageId":"ebb6eec1d928e02c9d85afad8b18551ec245fc71.1623675889.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.git.1623675888.gitgitgadget@gmail.com","subject":"[PATCH 04/10] diff --color-moved-ws=allow-indentation-change: simplify and optimize","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-14T13:04:42Z","receivedAt":"2021-06-14T13:05:55Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nIf we already have a block of potentially moved lines then as we move\ndown the diff we need to check if the next line of each potentially\nmoved line matches the current line of the diff. The implementation of\n--color-moved-ws=allow-indentation-change was needlessly performing\nthis check on all the lines in the diff that matched the current line\nrather than just the current line. To exacerbate the problem finding\nall the other lines in the diff that match the current line involves a\nfuzzy lookup so we were wasting even more time performing a second\ncomparison to filter out the non-matching lines. Fixing this reduces\ntime to run\n git diff --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\nby 88% and simplifies the code.\n\nBefore this change\nBenchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n Time (mean ± σ): 9.978 s ± 0.042 s [User: 9.905 s, System: 0.057 s]\n Range (min … max): 9.917 s … 10.037 s 10 runs\n\nAfter this change\nBenchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n Time (mean ± σ): 1.220 s ± 0.004 s [User: 1.160 s, System: 0.058 s]\n Range (min … max): 1.214 s … 1.226 s 10 runs\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 65 ++++++++++++++++------------------------------------------\n 1 file changed, 18 insertions(+), 47 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 19c8954ec546..5d5d168107a6 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -881,35 +881,20 @@ static int compute_ws_delta(const struct emitted_diff_symbol *a,\n \n static int cmp_in_block_with_wsd(const struct diff_options *o,\n \t\t\t\t const struct moved_entry *cur,\n-\t\t\t\t const struct moved_entry *match,\n-\t\t\t\t struct moved_block *pmb,\n-\t\t\t\t int n)\n+\t\t\t\t const struct emitted_diff_symbol *l,\n+\t\t\t\t struct moved_block *pmb)\n {\n-\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n-\tint al = cur->es->len, bl = match->es->len, cl = l->len;\n+\tint al = cur->es->len, bl = l->len;\n \tconst char *a = cur->es->line,\n-\t\t *b = match->es->line,\n-\t\t *c = l->line;\n+\t\t *b = l->line;\n \tint a_off = cur->es->indent_off,\n \t a_width = cur->es->indent_width,\n-\t c_off = l->indent_off,\n-\t c_width = l->indent_width;\n+\t b_off = l->indent_off,\n+\t b_width = l->indent_width;\n \tint delta;\n \n-\t/*\n-\t * We need to check if 'cur' is equal to 'match'. As those\n-\t * are from the same (+/-) side, we do not need to adjust for\n-\t * indent changes. However these were found using fuzzy\n-\t * matching so we do have to check if they are equal. Here we\n-\t * just check the lengths. We delay calling memcmp() to check\n-\t * the contents until later as if the length comparison for a\n-\t * and c fails we can avoid the call all together.\n-\t */\n-\tif (al != bl)\n-\t\treturn 1;\n-\n \t/* If 'l' and 'cur' are both blank then they match. */\n-\tif (a_width == INDENT_BLANKLINE && c_width == INDENT_BLANKLINE)\n+\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n \t\treturn 0;\n \n \t/*\n@@ -918,7 +903,7 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \t * match those of the current block and that the text of 'l' and 'cur'\n \t * after the indentation match.\n \t */\n-\tdelta = c_width - a_width;\n+\tdelta = b_width - a_width;\n \n \t/*\n \t * If the previous lines of this block were all blank then set its\n@@ -927,9 +912,8 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \tif (pmb->wsd == INDENT_BLANKLINE)\n \t\tpmb->wsd = delta;\n \n-\treturn !(delta == pmb->wsd && al - a_off == cl - c_off &&\n-\t\t !memcmp(a, b, al) && !\n-\t\t memcmp(a + a_off, c + c_off, al - a_off));\n+\treturn !(delta == pmb->wsd && al - a_off == bl - b_off &&\n+\t\t !memcmp(a + a_off, b + b_off, al - a_off));\n }\n \n static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n@@ -1030,36 +1014,23 @@ static void pmb_advance_or_null(struct diff_options *o,\n }\n \n static void pmb_advance_or_null_multi_match(struct diff_options *o,\n-\t\t\t\t\t struct moved_entry *match,\n-\t\t\t\t\t struct hashmap *hm,\n+\t\t\t\t\t struct emitted_diff_symbol *l,\n \t\t\t\t\t struct moved_block *pmb,\n-\t\t\t\t\t int pmb_nr, int n)\n+\t\t\t\t\t int pmb_nr)\n {\n \tint i;\n-\tchar *got_match = xcalloc(1, pmb_nr);\n-\n-\thashmap_for_each_entry_from(hm, match, ent) {\n-\t\tfor (i = 0; i < pmb_nr; i++) {\n-\t\t\tstruct moved_entry *prev = pmb[i].match;\n-\t\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n-\t\t\t\t\tprev->next_line : NULL;\n-\t\t\tif (!cur)\n-\t\t\t\tcontinue;\n-\t\t\tif (!cmp_in_block_with_wsd(o, cur, match, &pmb[i], n))\n-\t\t\t\tgot_match[i] |= 1;\n-\t\t}\n-\t}\n \n \tfor (i = 0; i < pmb_nr; i++) {\n-\t\tif (got_match[i]) {\n+\t\tstruct moved_entry *prev = pmb[i].match;\n+\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n+\t\t\tprev->next_line : NULL;\n+\t\tif (cur && !cmp_in_block_with_wsd(o, cur, l, &pmb[i])) {\n \t\t\t/* Advance to the next line */\n-\t\t\tpmb[i].match = pmb[i].match->next_line;\n+\t\t\tpmb[i].match = cur;\n \t\t} else {\n \t\t\tmoved_block_clear(&pmb[i]);\n \t\t}\n \t}\n-\n-\tfree(got_match);\n }\n \n static int shrink_potential_moved_blocks(struct moved_block *pmb,\n@@ -1181,7 +1152,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \n \t\tif (o->color_moved_ws_handling &\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\tpmb_advance_or_null_multi_match(o, match, hm, pmb, pmb_nr, n);\n+\t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n \t\telse\n \t\t\tpmb_advance_or_null(o, match, hm, pmb, pmb_nr);\n \n-- \ngitgitgadget\n\n"},{"id":"427345","messageId":"cec0c2d04d704a0b969eda13a1c1858a7ba6fbc0.1623675889.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.git.1623675888.gitgitgadget@gmail.com","subject":"[PATCH 05/10] diff --color-moved: call comparison function directly","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-14T13:04:43Z","receivedAt":"2021-06-14T13:05:55Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nCalling xdiff_compare_lines() directly rather than using a function\npointer from the hash map reduces the time very slightly but more\nimportantly it will allow us to easily combine pmb_advance_or_null()\nand pmb_advance_or_null_multi_match() in the next commit.\n\nBefore this change\nBenchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n Time (mean ± σ): 1.136 s ± 0.004 s [User: 1.079 s, System: 0.053 s]\n Range (min … max): 1.130 s … 1.141 s 10 runs\n\nAfter this change\nBenchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n Time (mean ± σ): 1.118 s ± 0.003 s [User: 1.062 s, System: 0.053 s]\n Range (min … max): 1.114 s … 1.121 s 10 runs\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 11 +++++++----\n 1 file changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 5d5d168107a6..c8fdfc9049bb 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -995,17 +995,20 @@ static void add_lines_to_move_detection(struct diff_options *o,\n }\n \n static void pmb_advance_or_null(struct diff_options *o,\n-\t\t\t\tstruct moved_entry *match,\n-\t\t\t\tstruct hashmap *hm,\n+\t\t\t\tstruct emitted_diff_symbol *l,\n \t\t\t\tstruct moved_block *pmb,\n \t\t\t\tint pmb_nr)\n {\n \tint i;\n+\tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n+\n \tfor (i = 0; i < pmb_nr; i++) {\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n \t\t\t\tprev->next_line : NULL;\n-\t\tif (cur && !hm->cmpfn(o, &cur->ent, &match->ent, NULL)) {\n+\t\tif (cur && xdiff_compare_lines(cur->es->line, cur->es->len,\n+\t\t\t\t\t\tl->line, l->len,\n+\t\t\t\t\t\tflags)) {\n \t\t\tpmb[i].match = cur;\n \t\t} else {\n \t\t\tpmb[i].match = NULL;\n@@ -1154,7 +1157,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n \t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n \t\telse\n-\t\t\tpmb_advance_or_null(o, match, hm, pmb, pmb_nr);\n+\t\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n \n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n-- \ngitgitgadget\n\n"},{"id":"427346","messageId":"050cef0081dc8252b55a4d69cca3bd08ae4eff98.1623675889.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.git.1623675888.gitgitgadget@gmail.com","subject":"[PATCH 06/10] diff --color-moved: unify moved block growth functions","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-14T13:04:44Z","receivedAt":"2021-06-14T13:06:00Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nAfter the last two commits pmb_advance_or_null() and\npmb_advance_or_null_multi_match() differ only in the comparison they\nperform. Lets simplify the code by combining them into a single\nfunction.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 41 ++++++++++++-----------------------------\n 1 file changed, 12 insertions(+), 29 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex c8fdfc9049bb..de6522a3a860 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1003,36 +1003,23 @@ static void pmb_advance_or_null(struct diff_options *o,\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \n \tfor (i = 0; i < pmb_nr; i++) {\n+\t\tint match;\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n \t\t\t\tprev->next_line : NULL;\n-\t\tif (cur && xdiff_compare_lines(cur->es->line, cur->es->len,\n-\t\t\t\t\t\tl->line, l->len,\n-\t\t\t\t\t\tflags)) {\n-\t\t\tpmb[i].match = cur;\n-\t\t} else {\n-\t\t\tpmb[i].match = NULL;\n-\t\t}\n-\t}\n-}\n \n-static void pmb_advance_or_null_multi_match(struct diff_options *o,\n-\t\t\t\t\t struct emitted_diff_symbol *l,\n-\t\t\t\t\t struct moved_block *pmb,\n-\t\t\t\t\t int pmb_nr)\n-{\n-\tint i;\n-\n-\tfor (i = 0; i < pmb_nr; i++) {\n-\t\tstruct moved_entry *prev = pmb[i].match;\n-\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n-\t\t\tprev->next_line : NULL;\n-\t\tif (cur && !cmp_in_block_with_wsd(o, cur, l, &pmb[i])) {\n-\t\t\t/* Advance to the next line */\n+\t\tif (o->color_moved_ws_handling &\n+\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n+\t\t\tmatch = cur &&\n+\t\t\t\t!cmp_in_block_with_wsd(o, cur, l, &pmb[i]);\n+\t\telse\n+\t\t\tmatch = cur &&\n+\t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n+\t\t\t\t\t\t l->line, l->len, flags);\n+\t\tif (match)\n \t\t\tpmb[i].match = cur;\n-\t\t} else {\n+\t\telse\n \t\t\tmoved_block_clear(&pmb[i]);\n-\t\t}\n \t}\n }\n \n@@ -1153,11 +1140,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tif (o->color_moved_ws_handling &\n-\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n-\t\telse\n-\t\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n+\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n \n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n-- \ngitgitgadget\n\n"},{"id":"427347","messageId":"220664dd907ed5e2183722fa2e1877f62c7d762a.1623675889.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.git.1623675888.gitgitgadget@gmail.com","subject":"[PATCH 10/10] diff --color-moved: intern strings","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-14T13:04:48Z","receivedAt":"2021-06-14T13:06:01Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nTaking inspiration from xdl_classify_record() assign an id to each\naddition and deletion such that lines that match for the current\n--color-moved-ws mode share the same unique id. This reduces the\nnumber of hash lookups a little (calculating the ids still involves\none hash lookup per line) but the main benefit is that when growing\nblocks of potentially moved lines we can replace string comparisons\nwhich involve chasing a pointer with a simple integer comparison. On\na large diff this commit reduces the time to run 'diff --color-moved'\nby 33% and 'diff --color-moved-ws=allow-indentation-change' by 20%.\n\nCompared to master the time to run 'git log --patch --color-moved' is\nincreased by 2% and 'git log --patch\n--color-moved-ws=allow-indentation-change' in reduced by 14%. These\ntimings were performed on an i5-7200U, on an i5-3470 both commands are\nfaster than master. The small speed decrease on commit sized diffs is\nunfortunate but I think it is small enough to be worth it for the\ngains on larger diffs.\n\nLarge diff before this change:\nBenchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n Time (mean ± σ): 1.147 s ± 0.005 s [User: 1.085 s, System: 0.059 s]\n Range (min … max): 1.140 s … 1.154 s 10 runs\n\nBenchmark #2: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n Time (mean ± σ): 1.048 s ± 0.005 s [User: 987.4 ms, System: 58.8 ms]\n Range (min … max): 1.043 s … 1.056 s 10 runs\n\nLarge diff after this change\nBenchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n Time (mean ± σ): 762.7 ms ± 2.8 ms [User: 707.5 ms, System: 53.7 ms]\n Range (min … max): 758.0 ms … 767.0 ms 10 runs\n\nBenchmark #2: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n Time (mean ± σ): 831.7 ms ± 1.7 ms [User: 776.5 ms, System: 53.3 ms]\n Range (min … max): 829.2 ms … 835.1 ms 10 runs\n\nSmall diffs on master\nBenchmark #1: bin-wrappers/git log -p --diff-algorithm=myers --color-moved --no-color-moved-ws --no-merges -n1000 v2.29.0\n Time (mean ± σ): 1.567 s ± 0.001 s [User: 1.443 s, System: 0.121 s]\n Range (min … max): 1.566 s … 1.571 s 10 runs\n\nBenchmark #2: bin-wrappers/git log -p --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change -n1000 --no-merges v2.29.0\n Time (mean ± σ): 1.865 s ± 0.008 s [User: 1.748 s, System: 0.112 s]\n Range (min … max): 1.857 s … 1.881 s 10 runs\n\nSmall diffs after this change\nBenchmark #1: bin-wrappers/git log -p --diff-algorithm=myers --color-moved --no-color-moved-ws --no-merges -n1000 v2.29.0\n Time (mean ± σ): 1.597 s ± 0.003 s [User: 1.413 s, System: 0.179 s]\n Range (min … max): 1.591 s … 1.601 s 10 runs\n\nBenchmark #2: bin-wrappers/git log -p --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change -n1000 --no-merges v2.29.0\n Time (mean ± σ): 1.606 s ± 0.006 s [User: 1.420 s, System: 0.181 s]\n Range (min … max): 1.601 s … 1.622 s 10 runs\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 173 ++++++++++++++++++++++++++++++++-------------------------\n 1 file changed, 96 insertions(+), 77 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex e6f3586b39bf..3260e2c60591 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -18,6 +18,7 @@\n #include \"submodule-config.h\"\n #include \"submodule.h\"\n #include \"hashmap.h\"\n+#include \"mem-pool.h\"\n #include \"ll-merge.h\"\n #include \"string-list.h\"\n #include \"strvec.h\"\n@@ -772,6 +773,7 @@ struct emitted_diff_symbol {\n \tint flags;\n \tint indent_off; /* Offset to first non-whitespace character */\n \tint indent_width; /* The visual width of the indentation */\n+\tunsigned id;\n \tenum diff_symbol s;\n };\n #define EMITTED_DIFF_SYMBOL_INIT {NULL}\n@@ -797,9 +799,9 @@ static void append_emitted_diff_symbol(struct diff_options *o,\n }\n \n struct moved_entry {\n-\tstruct hashmap_entry ent;\n \tconst struct emitted_diff_symbol *es;\n \tstruct moved_entry *next_line;\n+\tstruct moved_entry *next_match;\n };\n \n struct moved_block {\n@@ -866,24 +868,24 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \t\t\t\t const struct emitted_diff_symbol *l,\n \t\t\t\t struct moved_block *pmb)\n {\n-\tint al = cur->es->len, bl = l->len;\n-\tconst char *a = cur->es->line,\n-\t\t *b = l->line;\n-\tint a_off = cur->es->indent_off,\n-\t a_width = cur->es->indent_width,\n-\t b_off = l->indent_off,\n-\t b_width = l->indent_width;\n+\tint a_width = cur->es->indent_width, b_width = l->indent_width;\n \tint delta;\n \n-\t/* If 'l' and 'cur' are both blank then they match. */\n-\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n+\t/* The text of each line must match */\n+\tif (cur->es->id != l->id)\n+\t\treturn 1;\n+\n+\t/*\n+\t * If 'l' and 'cur' are both blank then we don't need to check the\n+\t * indent. We only need to check cur as we know the strings match.\n+\t * */\n+\tif (a_width == INDENT_BLANKLINE)\n \t\treturn 0;\n \n \t/*\n \t * The indent changes of the block are known and stored in pmb->wsd;\n \t * however we need to check if the indent changes of the current line\n-\t * match those of the current block and that the text of 'l' and 'cur'\n-\t * after the indentation match.\n+\t * match those of the current block.\n \t */\n \tdelta = b_width - a_width;\n \n@@ -894,22 +896,26 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \tif (pmb->wsd == INDENT_BLANKLINE)\n \t\tpmb->wsd = delta;\n \n-\treturn !(delta == pmb->wsd && al - a_off == bl - b_off &&\n-\t\t !memcmp(a + a_off, b + b_off, al - a_off));\n+\treturn delta != pmb->wsd;\n }\n \n-static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n-\t\t\t const struct hashmap_entry *eptr,\n-\t\t\t const struct hashmap_entry *entry_or_key,\n-\t\t\t const void *keydata)\n+struct interned_diff_symbol {\n+\tstruct hashmap_entry ent;\n+\tstruct emitted_diff_symbol *es;\n+};\n+\n+static int interned_diff_symbol_cmp(const void *hashmap_cmp_fn_data,\n+\t\t\t\t const struct hashmap_entry *eptr,\n+\t\t\t\t const struct hashmap_entry *entry_or_key,\n+\t\t\t\t const void *keydata)\n {\n \tconst struct diff_options *diffopt = hashmap_cmp_fn_data;\n \tconst struct emitted_diff_symbol *a, *b;\n \tunsigned flags = diffopt->color_moved_ws_handling\n \t\t\t & XDF_WHITESPACE_FLAGS;\n \n-\ta = container_of(eptr, const struct moved_entry, ent)->es;\n-\tb = container_of(entry_or_key, const struct moved_entry, ent)->es;\n+\ta = container_of(eptr, const struct interned_diff_symbol, ent)->es;\n+\tb = container_of(entry_or_key, const struct interned_diff_symbol, ent)->es;\n \n \treturn !xdiff_compare_lines(a->line + a->indent_off,\n \t\t\t\t a->len - a->indent_off,\n@@ -917,55 +923,81 @@ static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n \t\t\t\t b->len - b->indent_off, flags);\n }\n \n-static struct moved_entry *prepare_entry(struct diff_options *o,\n-\t\t\t\t\t int line_no)\n+static void prepare_entry(struct diff_options *o, struct emitted_diff_symbol *l,\n+\t\t\t struct interned_diff_symbol *s)\n {\n-\tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n-\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[line_no];\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \tunsigned int hash = xdiff_hash_string(l->line + l->indent_off,\n \t\t\t\t\t l->len - l->indent_off, flags);\n \n-\thashmap_entry_init(&ret->ent, hash);\n-\tret->es = l;\n-\tret->next_line = NULL;\n-\n-\treturn ret;\n+\thashmap_entry_init(&s->ent, hash);\n+\ts->es = l;\n }\n \n-static void add_lines_to_move_detection(struct diff_options *o,\n-\t\t\t\t\tstruct hashmap *add_lines,\n-\t\t\t\t\tstruct hashmap *del_lines)\n+struct moved_entry_list {\n+\tstruct moved_entry *add, *del;\n+};\n+\n+static struct moved_entry_list *add_lines_to_move_detection(struct diff_options *o,\n+\t\t\t\t\t\t\t struct mem_pool *entry_mem_pool)\n {\n \tstruct moved_entry *prev_line = NULL;\n-\n+\tstruct mem_pool interned_pool;\n+\tstruct hashmap interned_map;\n+\tstruct moved_entry_list *entry_list = NULL;\n+\tsize_t entry_list_alloc = 0;\n+\tunsigned id = 0;\n \tint n;\n+\n+\thashmap_init(&interned_map, interned_diff_symbol_cmp, o, 8096);\n+\tmem_pool_init(&interned_pool, 1024 * 1024);\n+\n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n-\t\tstruct hashmap *hm;\n-\t\tstruct moved_entry *key;\n+\t\tstruct interned_diff_symbol key;\n+\t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n+\t\tstruct interned_diff_symbol *s;\n+\t\tstruct moved_entry *entry;\n \n-\t\tswitch (o->emitted_symbols->buf[n].s) {\n-\t\tcase DIFF_SYMBOL_PLUS:\n-\t\t\thm = add_lines;\n-\t\t\tbreak;\n-\t\tcase DIFF_SYMBOL_MINUS:\n-\t\t\thm = del_lines;\n-\t\t\tbreak;\n-\t\tdefault:\n+\t\tif (l->s != DIFF_SYMBOL_PLUS && l->s != DIFF_SYMBOL_MINUS) {\n \t\t\tprev_line = NULL;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (o->color_moved_ws_handling &\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\tfill_es_indent_data(&o->emitted_symbols->buf[n]);\n-\t\tkey = prepare_entry(o, n);\n-\t\tif (prev_line && prev_line->es->s == o->emitted_symbols->buf[n].s)\n-\t\t\tprev_line->next_line = key;\n+\t\t\tfill_es_indent_data(l);\n \n-\t\thashmap_add(hm, &key->ent);\n-\t\tprev_line = key;\n+\t\tprepare_entry(o, l, &key);\n+\t\ts = hashmap_get_entry(&interned_map, &key, ent, &key.ent);\n+\t\tif (s) {\n+\t\t\tl->id = s->es->id;\n+\t\t} else {\n+\t\t\tl->id = id;\n+\t\t\tALLOC_GROW_BY(entry_list, id, 1, entry_list_alloc);\n+\t\t\thashmap_add(&interned_map,\n+\t\t\t\t memcpy(mem_pool_alloc(&interned_pool,\n+\t\t\t\t\t\t\t sizeof(key)),\n+\t\t\t\t\t &key, sizeof(key)));\n+\t\t}\n+\t\tentry = mem_pool_alloc(entry_mem_pool, sizeof(*entry));\n+\t\tentry->es = l;\n+\t\tentry->next_line = NULL;\n+\t\tif (prev_line && prev_line->es->s == l->s)\n+\t\t\tprev_line->next_line = entry;\n+\t\tprev_line = entry;\n+\t\tif (l->s == DIFF_SYMBOL_PLUS) {\n+\t\t\tentry->next_match = entry_list[l->id].add;\n+\t\t\tentry_list[l->id].add = entry;\n+\t\t} else {\n+\t\t\tentry->next_match = entry_list[l->id].del;\n+\t\t\tentry_list[l->id].del = entry;\n+\t\t}\n \t}\n+\n+\thashmap_clear(&interned_map);\n+\tmem_pool_discard(&interned_pool, 0);\n+\n+\treturn entry_list;\n }\n \n static void pmb_advance_or_null(struct diff_options *o,\n@@ -974,7 +1006,6 @@ static void pmb_advance_or_null(struct diff_options *o,\n \t\t\t\tint *pmb_nr)\n {\n \tint i, j;\n-\tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \n \tfor (i = 0, j = 0; i < *pmb_nr; i++) {\n \t\tint match;\n@@ -987,9 +1018,8 @@ static void pmb_advance_or_null(struct diff_options *o,\n \t\t\tmatch = cur &&\n \t\t\t\t!cmp_in_block_with_wsd(o, cur, l, &pmb[i]);\n \t\telse\n-\t\t\tmatch = cur &&\n-\t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n-\t\t\t\t\t\t l->line, l->len, flags);\n+\t\t\tmatch = cur && cur->es->id == l->id;\n+\n \t\tif (match)\n \t\t\tpmb[j++].match = cur;\n \t}\n@@ -1034,8 +1064,7 @@ static int adjust_last_block(struct diff_options *o, int n, int block_length)\n \n /* Find blocks of moved code, delegate actual coloring decision to helper */\n static void mark_color_as_moved(struct diff_options *o,\n-\t\t\t\tstruct hashmap *add_lines,\n-\t\t\t\tstruct hashmap *del_lines)\n+\t\t\t\tstruct moved_entry_list *entry_list)\n {\n \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n@@ -1044,23 +1073,15 @@ static void mark_color_as_moved(struct diff_options *o,\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n-\t\tstruct hashmap *hm = NULL;\n-\t\tstruct moved_entry *key;\n \t\tstruct moved_entry *match = NULL;\n \t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n \n \t\tswitch (l->s) {\n \t\tcase DIFF_SYMBOL_PLUS:\n-\t\t\thm = del_lines;\n-\t\t\tkey = prepare_entry(o, n);\n-\t\t\tmatch = hashmap_get_entry(hm, key, ent, NULL);\n-\t\t\tfree(key);\n+\t\t\tmatch = entry_list[l->id].del;\n \t\t\tbreak;\n \t\tcase DIFF_SYMBOL_MINUS:\n-\t\t\thm = add_lines;\n-\t\t\tkey = prepare_entry(o, n);\n-\t\t\tmatch = hashmap_get_entry(hm, key, ent, NULL);\n-\t\t\tfree(key);\n+\t\t\tmatch = entry_list[l->id].add;\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tflipped_block = 0;\n@@ -1089,7 +1110,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t * The current line is the start of a new block.\n \t\t\t * Setup the set of potential blocks.\n \t\t\t */\n-\t\t\thashmap_for_each_entry_from(hm, match, ent) {\n+\t\t\tfor (; match; match = match->next_match) {\n \t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n \t\t\t\tif (o->color_moved_ws_handling &\n \t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n@@ -1460,7 +1481,7 @@ static void emit_diff_symbol_from_struct(struct diff_options *o,\n static void emit_diff_symbol(struct diff_options *o, enum diff_symbol s,\n \t\t\t const char *line, int len, unsigned flags)\n {\n-\tstruct emitted_diff_symbol e = {line, len, flags, 0, 0, s};\n+\tstruct emitted_diff_symbol e = {line, len, flags, 0, 0, 0, s};\n \n \tif (o->emitted_symbols)\n \t\tappend_emitted_diff_symbol(o, &e);\n@@ -6214,20 +6235,18 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \n \tif (o->emitted_symbols) {\n \t\tif (o->color_moved) {\n-\t\t\tstruct hashmap add_lines, del_lines;\n-\n-\t\t\thashmap_init(&del_lines, moved_entry_cmp, o, 0);\n-\t\t\thashmap_init(&add_lines, moved_entry_cmp, o, 0);\n+\t\t\tstruct mem_pool entry_pool;\n+\t\t\tstruct moved_entry_list *entry_list;\n \n-\t\t\tadd_lines_to_move_detection(o, &add_lines, &del_lines);\n-\t\t\tmark_color_as_moved(o, &add_lines, &del_lines);\n+\t\t\tmem_pool_init(&entry_pool, 1024 * 1024);\n+\t\t\tentry_list = add_lines_to_move_detection(o,\n+\t\t\t\t\t\t\t\t &entry_pool);\n+\t\t\tmark_color_as_moved(o, entry_list);\n \t\t\tif (o->color_moved == COLOR_MOVED_ZEBRA_DIM)\n \t\t\t\tdim_moved_lines(o);\n \n-\t\t\thashmap_clear_and_free(&add_lines, struct moved_entry,\n-\t\t\t\t\t\tent);\n-\t\t\thashmap_clear_and_free(&del_lines, struct moved_entry,\n-\t\t\t\t\t\tent);\n+\t\t\tmem_pool_discard(&entry_pool, 0);\n+\t\t\tfree(entry_list);\n \t\t}\n \n \t\tfor (i = 0; i < esm.nr; i++)\n-- \ngitgitgadget\n"},{"id":"427348","messageId":"1de99ac2bc3c60bc4639687d5ad2e2638aa9cadb.1623675889.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.git.1623675888.gitgitgadget@gmail.com","subject":"[PATCH 08/10] diff --color-moved: stop clearing potential moved blocks","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-14T13:04:46Z","receivedAt":"2021-06-14T13:06:17Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nmoved_block_clear() was introduced in 74d156f4a1 (\"diff\n--color-moved-ws: fix double free crash\", 2018-10-04) to free the\nmemory that was allocated when initializing a potential moved\nblock. However since 21536d077f (\"diff --color-moved-ws: modify\nallow-indentation-change\", 2018-11-23) initializing a potential moved\nblock no longer allocates any memory. Up until the last commit we were\nrelying on moved_block_clear() to set the `match` pointer to NULL when\na block stopped matching, but since that commit we do not clear a\nmoved block that does not match so it does not make sense to clear\nthem elsewhere.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 11 -----------\n 1 file changed, 11 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex f60cce654c14..ee58373f55f8 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -807,11 +807,6 @@ struct moved_block {\n \tint wsd; /* The whitespace delta of this block */\n };\n \n-static void moved_block_clear(struct moved_block *b)\n-{\n-\tmemset(b, 0, sizeof(*b));\n-}\n-\n #define INDENT_BLANKLINE INT_MIN\n \n static void fill_es_indent_data(struct emitted_diff_symbol *es)\n@@ -1093,11 +1088,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t}\n \n \t\tif (pmb_nr && (!match || l->s != moved_symbol)) {\n-\t\t\tint i;\n-\n \t\t\tadjust_last_block(o, n, block_length);\n-\t\t\tfor(i = 0; i < pmb_nr; i++)\n-\t\t\t\tmoved_block_clear(&pmb[i]);\n \t\t\tpmb_nr = 0;\n \t\t\tblock_length = 0;\n \t\t\tflipped_block = 0;\n@@ -1155,8 +1146,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t}\n \tadjust_last_block(o, n, block_length);\n \n-\tfor(n = 0; n < pmb_nr; n++)\n-\t\tmoved_block_clear(&pmb[n]);\n \tfree(pmb);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"427405","messageId":"xmqq4kdzyfm0.fsf@gitster.g","threadId":"55926","inReplyTo":"374dbebcbf29b686508e51205b2f7a4e72104950.1623675888.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 01/10] diff --color-moved=zerba: fix alternate coloring","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-15T03:24:39Z","receivedAt":"2021-06-15T03:24:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Phillip Wood via GitGitGadget\" writes:\n\n> Subject: Re: [PATCH 01/10] diff --color-moved=zerba: fix alternate coloring\n\nZerba?\n\n> From: Phillip Wood \n>\n> b0a2ba4776 (\"diff --color-moved=zebra: be stricter with color\n> alternation\", 2018-11-23) sought to avoid using the alternate colors\n> unless there are two adjacent moved blocks of the same\n> sign. Unfortunately it contains two bugs that prevented it from fixing\n> the problem properly. Firstly `last_symbol` is reset at the start of\n> each iteration of the loop losing the symbol of the last line and\n> secondly when deciding whether to use the alternate color it should be\n> checking if the current line is the same sign of the last line, not a\n> different sign. The combination of the two errors means that we still\n> use the alternate color when we should do but we also use it when we\n> shouldn't. This is most noticable when using\n> --color-moved-ws=allow-indentation-change with hunks like\n>\n> -this line gets indented\n> + this line gets indented\n>\n> where the post image is colored with newMovedAlternate rather than\n> newMoved. While this does not matter much, the next commit will change\n> the coloring to be correct in this case, so lets fix the bug here to\n> make it clear why the output is changing and add a regression test.\n>\n> Signed-off-by: Phillip Wood \n> ---\n> diff.c | 4 +--\n> t/t4015-diff-whitespace.sh | 72 ++++++++++++++++++++++++++++++++++++++\n> 2 files changed, 74 insertions(+), 2 deletions(-)\n>\n> diff --git a/diff.c b/diff.c\n> index 52c791574b71..cb068f8258c0 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -1142,6 +1142,7 @@ static void mark_color_as_moved(struct diff_options *o,\n> \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n> \tint pmb_nr = 0, pmb_alloc = 0;\n> \tint n, flipped_block = 0, block_length = 0;\n> +\tenum diff_symbol last_symbol = 0;\n> \n> \n> \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n> @@ -1149,7 +1150,6 @@ static void mark_color_as_moved(struct diff_options *o,\n> \t\tstruct moved_entry *key;\n> \t\tstruct moved_entry *match = NULL;\n> \t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n> -\t\tenum diff_symbol last_symbol = 0;\n> \n> \t\tswitch (l->s) {\n> \t\tcase DIFF_SYMBOL_PLUS:\n> @@ -1214,7 +1214,7 @@ static void mark_color_as_moved(struct diff_options *o,\n> \t\t\t}\n> \n> \t\t\tif (adjust_last_block(o, n, block_length) &&\n> -\t\t\t pmb_nr && last_symbol != l->s)\n> +\t\t\t pmb_nr && last_symbol == l->s)\n> \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n> \t\t\telse\n> \t\t\t\tflipped_block = 0;\n> diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\n> index 2c13b62d3c65..920114cd795c 100755\n> --- a/t/t4015-diff-whitespace.sh\n> +++ b/t/t4015-diff-whitespace.sh\n> @@ -1442,6 +1442,78 @@ test_expect_success 'detect permutations inside moved code -- dimmed-zebra' '\n> \ttest_cmp expected actual\n> '\n> \n> +test_expect_success 'zebra alternate color is only used when necessary' '\n> +\tcat >old.txt <<-\\EOF &&\n> +\tline 1A should be marked as oldMoved newMovedAlternate\n> +\tline 1B should be marked as oldMoved newMovedAlternate\n> +\tunchanged\n> +\tline 2A should be marked as oldMoved newMovedAlternate\n> +\tline 2B should be marked as oldMoved newMovedAlternate\n> +\tline 3A should be marked as oldMovedAlternate newMoved\n> +\tline 3B should be marked as oldMovedAlternate newMoved\n> +\tunchanged\n> +\tline 4A should be marked as oldMoved newMovedAlternate\n> +\tline 4B should be marked as oldMoved newMovedAlternate\n> +\tline 5A should be marked as oldMovedAlternate newMoved\n> +\tline 5B should be marked as oldMovedAlternate newMoved\n> +\tline 6A should be marked as oldMoved newMoved\n> +\tline 6B should be marked as oldMoved newMoved\n> +\tEOF\n> +\tcat >new.txt <<-\\EOF &&\n> +\t line 1A should be marked as oldMoved newMovedAlternate\n> +\t line 1B should be marked as oldMoved newMovedAlternate\n> +\tunchanged\n> +\t line 3A should be marked as oldMovedAlternate newMoved\n> +\t line 3B should be marked as oldMovedAlternate newMoved\n> +\t line 2A should be marked as oldMoved newMovedAlternate\n> +\t line 2B should be marked as oldMoved newMovedAlternate\n> +\tunchanged\n> +\t line 6A should be marked as oldMoved newMoved\n> +\t line 6B should be marked as oldMoved newMoved\n> +\t line 4A should be marked as oldMoved newMovedAlternate\n> +\t line 4B should be marked as oldMoved newMovedAlternate\n> +\t line 5A should be marked as oldMovedAlternate newMoved\n> +\t line 5B should be marked as oldMovedAlternate newMoved\n> +\tEOF\n> +\ttest_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n> +\t\t --color-moved-ws=allow-indentation-change \\\n> +\t\t old.txt new.txt >output &&\n> +\tgrep -v index output | test_decode_color >actual &&\n> +\tcat >expected <<-\\EOF &&\n> +\tdiff --git a/old.txt b/new.txt\n> +\t--- a/old.txt\n> +\t+++ b/new.txt\n> +\t@@ -1,14 +1,14 @@\n> +\t-line 1A should be marked as oldMoved newMovedAlternate\n> +\t-line 1B should be marked as oldMoved newMovedAlternate\n> +\t+ line 1A should be marked as oldMoved newMovedAlternate\n> +\t+ line 1B should be marked as oldMoved newMovedAlternate\n> +\t unchanged\n> +\t-line 2A should be marked as oldMoved newMovedAlternate\n> +\t-line 2B should be marked as oldMoved newMovedAlternate\n> +\t-line 3A should be marked as oldMovedAlternate newMoved\n> +\t-line 3B should be marked as oldMovedAlternate newMoved\n> +\t+ line 3A should be marked as oldMovedAlternate newMoved\n> +\t+ line 3B should be marked as oldMovedAlternate newMoved\n> +\t+ line 2A should be marked as oldMoved newMovedAlternate\n> +\t+ line 2B should be marked as oldMoved newMovedAlternate\n> +\t unchanged\n> +\t-line 4A should be marked as oldMoved newMovedAlternate\n> +\t-line 4B should be marked as oldMoved newMovedAlternate\n> +\t-line 5A should be marked as oldMovedAlternate newMoved\n> +\t-line 5B should be marked as oldMovedAlternate newMoved\n> +\t-line 6A should be marked as oldMoved newMoved\n> +\t-line 6B should be marked as oldMoved newMoved\n> +\t+ line 6A should be marked as oldMoved newMoved\n> +\t+ line 6B should be marked as oldMoved newMoved\n> +\t+ line 4A should be marked as oldMoved newMovedAlternate\n> +\t+ line 4B should be marked as oldMoved newMovedAlternate\n> +\t+ line 5A should be marked as oldMovedAlternate newMoved\n> +\t+ line 5B should be marked as oldMovedAlternate newMoved\n> +\tEOF\n> +\ttest_cmp expected actual\n> +'\n> +\n> test_expect_success 'cmd option assumes configured colored-moved' '\n> \ttest_config color.diff.oldMoved \"magenta\" &&\n> \ttest_config color.diff.newMoved \"cyan\" &&\n"},{"id":"427452","messageId":"e74e3750-b43d-1010-17df-6c821bb7b090@gmail.com","threadId":"55926","inReplyTo":"xmqq4kdzyfm0.fsf@gitster.g","subject":"Re: [PATCH 01/10] diff --color-moved=zerba: fix alternate coloring","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-06-15T11:22:51Z","receivedAt":"2021-06-15T11:22:57Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 15/06/2021 04:24, Junio C Hamano wrote:\n> \"Phillip Wood via GitGitGadget\" writes:\n> \n>> Subject: Re: [PATCH 01/10] diff --color-moved=zerba: fix alternate coloring\n> \n> Zerba?\n\nThey're quite rare in the wild :-/ Thanks for pointing that out I'll fix \nit when I re-roll\n\nBest Wishes\n\nPhillip\n\n>> From: Phillip Wood \n>>\n>> b0a2ba4776 (\"diff --color-moved=zebra: be stricter with color\n>> alternation\", 2018-11-23) sought to avoid using the alternate colors\n>> unless there are two adjacent moved blocks of the same\n>> sign. Unfortunately it contains two bugs that prevented it from fixing\n>> the problem properly. Firstly `last_symbol` is reset at the start of\n>> each iteration of the loop losing the symbol of the last line and\n>> secondly when deciding whether to use the alternate color it should be\n>> checking if the current line is the same sign of the last line, not a\n>> different sign. The combination of the two errors means that we still\n>> use the alternate color when we should do but we also use it when we\n>> shouldn't. This is most noticable when using\n>> --color-moved-ws=allow-indentation-change with hunks like\n>>\n>> -this line gets indented\n>> + this line gets indented\n>>\n>> where the post image is colored with newMovedAlternate rather than\n>> newMoved. While this does not matter much, the next commit will change\n>> the coloring to be correct in this case, so lets fix the bug here to\n>> make it clear why the output is changing and add a regression test.\n>>\n>> Signed-off-by: Phillip Wood \n>> ---\n>> diff.c | 4 +--\n>> t/t4015-diff-whitespace.sh | 72 ++++++++++++++++++++++++++++++++++++++\n>> 2 files changed, 74 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/diff.c b/diff.c\n>> index 52c791574b71..cb068f8258c0 100644\n>> --- a/diff.c\n>> +++ b/diff.c\n>> @@ -1142,6 +1142,7 @@ static void mark_color_as_moved(struct diff_options *o,\n>> \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n>> \tint pmb_nr = 0, pmb_alloc = 0;\n>> \tint n, flipped_block = 0, block_length = 0;\n>> +\tenum diff_symbol last_symbol = 0;\n>> \n>> \n>> \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n>> @@ -1149,7 +1150,6 @@ static void mark_color_as_moved(struct diff_options *o,\n>> \t\tstruct moved_entry *key;\n>> \t\tstruct moved_entry *match = NULL;\n>> \t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n>> -\t\tenum diff_symbol last_symbol = 0;\n>> \n>> \t\tswitch (l->s) {\n>> \t\tcase DIFF_SYMBOL_PLUS:\n>> @@ -1214,7 +1214,7 @@ static void mark_color_as_moved(struct diff_options *o,\n>> \t\t\t}\n>> \n>> \t\t\tif (adjust_last_block(o, n, block_length) &&\n>> -\t\t\t pmb_nr && last_symbol != l->s)\n>> +\t\t\t pmb_nr && last_symbol == l->s)\n>> \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n>> \t\t\telse\n>> \t\t\t\tflipped_block = 0;\n>> diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\n>> index 2c13b62d3c65..920114cd795c 100755\n>> --- a/t/t4015-diff-whitespace.sh\n>> +++ b/t/t4015-diff-whitespace.sh\n>> @@ -1442,6 +1442,78 @@ test_expect_success 'detect permutations inside moved code -- dimmed-zebra' '\n>> \ttest_cmp expected actual\n>> '\n>> \n>> +test_expect_success 'zebra alternate color is only used when necessary' '\n>> +\tcat >old.txt <<-\\EOF &&\n>> +\tline 1A should be marked as oldMoved newMovedAlternate\n>> +\tline 1B should be marked as oldMoved newMovedAlternate\n>> +\tunchanged\n>> +\tline 2A should be marked as oldMoved newMovedAlternate\n>> +\tline 2B should be marked as oldMoved newMovedAlternate\n>> +\tline 3A should be marked as oldMovedAlternate newMoved\n>> +\tline 3B should be marked as oldMovedAlternate newMoved\n>> +\tunchanged\n>> +\tline 4A should be marked as oldMoved newMovedAlternate\n>> +\tline 4B should be marked as oldMoved newMovedAlternate\n>> +\tline 5A should be marked as oldMovedAlternate newMoved\n>> +\tline 5B should be marked as oldMovedAlternate newMoved\n>> +\tline 6A should be marked as oldMoved newMoved\n>> +\tline 6B should be marked as oldMoved newMoved\n>> +\tEOF\n>> +\tcat >new.txt <<-\\EOF &&\n>> +\t line 1A should be marked as oldMoved newMovedAlternate\n>> +\t line 1B should be marked as oldMoved newMovedAlternate\n>> +\tunchanged\n>> +\t line 3A should be marked as oldMovedAlternate newMoved\n>> +\t line 3B should be marked as oldMovedAlternate newMoved\n>> +\t line 2A should be marked as oldMoved newMovedAlternate\n>> +\t line 2B should be marked as oldMoved newMovedAlternate\n>> +\tunchanged\n>> +\t line 6A should be marked as oldMoved newMoved\n>> +\t line 6B should be marked as oldMoved newMoved\n>> +\t line 4A should be marked as oldMoved newMovedAlternate\n>> +\t line 4B should be marked as oldMoved newMovedAlternate\n>> +\t line 5A should be marked as oldMovedAlternate newMoved\n>> +\t line 5B should be marked as oldMovedAlternate newMoved\n>> +\tEOF\n>> +\ttest_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n>> +\t\t --color-moved-ws=allow-indentation-change \\\n>> +\t\t old.txt new.txt >output &&\n>> +\tgrep -v index output | test_decode_color >actual &&\n>> +\tcat >expected <<-\\EOF &&\n>> +\tdiff --git a/old.txt b/new.txt\n>> +\t--- a/old.txt\n>> +\t+++ b/new.txt\n>> +\t@@ -1,14 +1,14 @@\n>> +\t-line 1A should be marked as oldMoved newMovedAlternate\n>> +\t-line 1B should be marked as oldMoved newMovedAlternate\n>> +\t+ line 1A should be marked as oldMoved newMovedAlternate\n>> +\t+ line 1B should be marked as oldMoved newMovedAlternate\n>> +\t unchanged\n>> +\t-line 2A should be marked as oldMoved newMovedAlternate\n>> +\t-line 2B should be marked as oldMoved newMovedAlternate\n>> +\t-line 3A should be marked as oldMovedAlternate newMoved\n>> +\t-line 3B should be marked as oldMovedAlternate newMoved\n>> +\t+ line 3A should be marked as oldMovedAlternate newMoved\n>> +\t+ line 3B should be marked as oldMovedAlternate newMoved\n>> +\t+ line 2A should be marked as oldMoved newMovedAlternate\n>> +\t+ line 2B should be marked as oldMoved newMovedAlternate\n>> +\t unchanged\n>> +\t-line 4A should be marked as oldMoved newMovedAlternate\n>> +\t-line 4B should be marked as oldMoved newMovedAlternate\n>> +\t-line 5A should be marked as oldMovedAlternate newMoved\n>> +\t-line 5B should be marked as oldMovedAlternate newMoved\n>> +\t-line 6A should be marked as oldMoved newMoved\n>> +\t-line 6B should be marked as oldMoved newMoved\n>> +\t+ line 6A should be marked as oldMoved newMoved\n>> +\t+ line 6B should be marked as oldMoved newMoved\n>> +\t+ line 4A should be marked as oldMoved newMovedAlternate\n>> +\t+ line 4B should be marked as oldMoved newMovedAlternate\n>> +\t+ line 5A should be marked as oldMovedAlternate newMoved\n>> +\t+ line 5B should be marked as oldMovedAlternate newMoved\n>> +\tEOF\n>> +\ttest_cmp expected actual\n>> +'\n>> +\n>> test_expect_success 'cmd option assumes configured colored-moved' '\n>> \ttest_config color.diff.oldMoved \"magenta\" &&\n>> \ttest_config color.diff.newMoved \"cyan\" &&\n"},{"id":"427618","messageId":"87eed1yjcz.fsf@evledraar.gmail.com","threadId":"55926","inReplyTo":"pull.981.git.1623675888.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/10] diff --color-moved[-ws] speedups","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-16T14:24:48Z","receivedAt":"2021-06-16T14:28:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 14 2021, Phillip Wood via GitGitGadget wrote:\n\n> The current implementation of diff --color-moved-ws=allow-indentation-change\n> is considerably slower that the implementation of diff --color-moved which\n> is in turn slower than a regular diff. This patch series starts with a\n> couple of bug fixes and then reworks the implementation of diff\n> --color-moved and diff --color-moved-ws=allow-indentation-change to speed\n> them up on large diffs. The time to run git diff --color-moved\n> --no-color-moved-ws v2.28.0 v2.29.0 is reduced by 33% and the time to run\n> git diff --color-moved --color-moved-ws=allow-indentation-change v2.28.0\n> v2.29.0 is reduced by 88%. There is a small slowdown for commit sized diffs\n> with --color-moved - the time to run git log -p --color-moved\n> --no-color-moved-ws --no-merges -n1000 v2.29.0 is increased by 2% on recent\n> processors. On older processors these patches reduce the running time in all\n> cases that I've tested. In general the larger the diff the larger the speed\n> up. As an extreme example the time to run diff --color-moved\n> --color-moved-ws=allow-indentation-change v2.25.0 v2.30.0 goes down from 8\n> minutes to 6 seconds.\n>\n> Phillip Wood (10):\n> diff --color-moved=zerba: fix alternate coloring\n> diff --color-moved: avoid false short line matches and bad zerba\n> coloring\n> diff: simplify allow-indentation-change delta calculation\n> diff --color-moved-ws=allow-indentation-change: simplify and optimize\n> diff --color-moved: call comparison function directly\n> diff --color-moved: unify moved block growth functions\n> diff --color-moved: shrink potential moved blocks as we go\n> diff --color-moved: stop clearing potential moved blocks\n> diff --color-moved-ws=allow-indentation-change: improve hash lookups\n> diff --color-moved: intern strings\n\nNice to see these land after the earlier on-list reference to them.\n\nI skimmed these mostly, and am not familiar with this code, but didn't\nsee any glaring things missing. There was one existing oddity with\nassigning a 0 to an \"enum diff_symbol\", don't we want\nDIFF_SYMBOL_BINARY_DIFF_HEADER? In any case, it's just a line you touch\nin 02/10, and pre-dates these changes.\n\nOne thing I would very much like to see here is a conversion of the\nexisting ad-hoc benchmarks you note in commit messages to something that\nlives in t/perf/, it really helps future maintenance of perf-sensitive\ncode to be able to re-run those, and I for one find the output much\neasier to read than whatever tool you're using to produce your\nbenchmarks.\n"},{"id":"428027","messageId":"094f5e5f-d447-8867-a9a7-be5c8827bba6@gmail.com","threadId":"55926","inReplyTo":"87eed1yjcz.fsf@evledraar.gmail.com","subject":"Re: [PATCH 00/10] diff --color-moved[-ws] speedups","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-06-21T10:03:50Z","receivedAt":"2021-06-21T10:03:57Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 16/06/2021 15:24, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Jun 14 2021, Phillip Wood via GitGitGadget wrote:\n> \n>> The current implementation of diff --color-moved-ws=allow-indentation-change\n>> is considerably slower that the implementation of diff --color-moved which\n>> is in turn slower than a regular diff. This patch series starts with a\n>> couple of bug fixes and then reworks the implementation of diff\n>> --color-moved and diff --color-moved-ws=allow-indentation-change to speed\n>> them up on large diffs. The time to run git diff --color-moved\n>> --no-color-moved-ws v2.28.0 v2.29.0 is reduced by 33% and the time to run\n>> git diff --color-moved --color-moved-ws=allow-indentation-change v2.28.0\n>> v2.29.0 is reduced by 88%. There is a small slowdown for commit sized diffs\n>> with --color-moved - the time to run git log -p --color-moved\n>> --no-color-moved-ws --no-merges -n1000 v2.29.0 is increased by 2% on recent\n>> processors. On older processors these patches reduce the running time in all\n>> cases that I've tested. In general the larger the diff the larger the speed\n>> up. As an extreme example the time to run diff --color-moved\n>> --color-moved-ws=allow-indentation-change v2.25.0 v2.30.0 goes down from 8\n>> minutes to 6 seconds.\n>>\n>> Phillip Wood (10):\n>> diff --color-moved=zerba: fix alternate coloring\n>> diff --color-moved: avoid false short line matches and bad zerba\n>> coloring\n>> diff: simplify allow-indentation-change delta calculation\n>> diff --color-moved-ws=allow-indentation-change: simplify and optimize\n>> diff --color-moved: call comparison function directly\n>> diff --color-moved: unify moved block growth functions\n>> diff --color-moved: shrink potential moved blocks as we go\n>> diff --color-moved: stop clearing potential moved blocks\n>> diff --color-moved-ws=allow-indentation-change: improve hash lookups\n>> diff --color-moved: intern strings\n> \n> Nice to see these land after the earlier on-list reference to them.\n> \n> I skimmed these mostly, and am not familiar with this code, but didn't\n> see any glaring things missing. There was one existing oddity with\n> assigning a 0 to an \"enum diff_symbol\", don't we want\n> DIFF_SYMBOL_BINARY_DIFF_HEADER? In any case, it's just a line you touch\n> in 02/10, and pre-dates these changes.\n\nThanks for taking a look at these patches. I take your point about the \nassignment, I don't think the actual value matters so long as it's not \nDIFF_SYMBOL_PLUS or DIFF_SYMBOL_MINUS.\n\n> One thing I would very much like to see here is a conversion of the\n> existing ad-hoc benchmarks you note in commit messages to something that\n> lives in t/perf/, it really helps future maintenance of perf-sensitive\n> code to be able to re-run those, and I for one find the output much\n> easier to read than whatever tool you're using to produce your\n> benchmarks.\n\nAdding some perf tests is a good idea, I'll do that when I reroll which \nmay take a couple of weeks as I'm going offline for a while at the end \nof the week. The tool I have been using is hyperfine[1], it has been \nused by a few other contributors (see `git log --grep σ` if you're \ninterested)\n\n[1] https://github.com/sharkdp/hyperfine\n\nBest Wishes\n\nPhillip\n\n"},{"id":"429555","messageId":"CABPp-BEYT+uobXqBB2JZsZCtCm-5XcwqEWUYCf54sgB6hZX4DA@mail.gmail.com","threadId":"55926","inReplyTo":"41cdedd60907b966dffa6cf0c9825ffb448f4971.1623675889.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 09/10] diff --color-moved-ws=allow-indentation-change: improve hash lookups","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-09T15:36:06Z","receivedAt":"2021-07-09T15:36:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jun 14, 2021 at 6:06 AM Phillip Wood via GitGitGadget\n wrote:\n>\n> From: Phillip Wood \n>\n> As libxdiff does not have a whitespace flag to ignore the indentation\n> the code for --color-moved-ws=allow-indentation-change uses\n> XDF_IGNORE_WHITESPACE and then filters out any hash lookups where\n> there are non-indentation changes. This is filtering is inefficient as\n\ns/This is filtering is/This filtering is/\n"},{"id":"430596","messageId":"pull.981.v2.git.1626777393.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.git.1623675888.gitgitgadget@gmail.com","subject":"[PATCH v2 00/12] diff --color-moved[-ws] speedups","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T10:36:21Z","receivedAt":"2021-07-20T10:39:02Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Thanks to Ævar and Elijah for their comments, I've reworded the commit\nmessages, addressed the enum initialization issue in patch 2 (now 3) and\nadded some perf tests.\n\nThere are two new patches in this round. The first patch is new and adds the\nperf tests suggested by Ævar, the penultimate patch is also new and coverts\nthe existing code to use a designated initializer.\n\nI've converted the benchmark results in the commit messages to use the new\ntests, the percentage changes are broadly similar to the previous results\nthough I ended up running them on a different computer this time.\n\nV1 cover letter:\n\nThe current implementation of diff --color-moved-ws=allow-indentation-change\nis considerably slower that the implementation of diff --color-moved which\nis in turn slower than a regular diff. This patch series starts with a\ncouple of bug fixes and then reworks the implementation of diff\n--color-moved and diff --color-moved-ws=allow-indentation-change to speed\nthem up on large diffs. The time to run git diff --color-moved\n--no-color-moved-ws v2.28.0 v2.29.0 is reduced by 33% and the time to run\ngit diff --color-moved --color-moved-ws=allow-indentation-change v2.28.0\nv2.29.0 is reduced by 88%. There is a small slowdown for commit sized diffs\nwith --color-moved - the time to run git log -p --color-moved\n--no-color-moved-ws --no-merges -n1000 v2.29.0 is increased by 2% on recent\nprocessors. On older processors these patches reduce the running time in all\ncases that I've tested. In general the larger the diff the larger the speed\nup. As an extreme example the time to run diff --color-moved\n--color-moved-ws=allow-indentation-change v2.25.0 v2.30.0 goes down from 8\nminutes to 6 seconds.\n\nPhillip Wood (12):\n diff --color-moved: add perf tests\n diff --color-moved=zebra: fix alternate coloring\n diff --color-moved: avoid false short line matches and bad zerba\n coloring\n diff: simplify allow-indentation-change delta calculation\n diff --color-moved-ws=allow-indentation-change: simplify and optimize\n diff --color-moved: call comparison function directly\n diff --color-moved: unify moved block growth functions\n diff --color-moved: shrink potential moved blocks as we go\n diff --color-moved: stop clearing potential moved blocks\n diff --color-moved-ws=allow-indentation-change: improve hash lookups\n diff: use designated initializers for emitted_diff_symbol\n diff --color-moved: intern strings\n\n diff.c | 377 ++++++++++++-------------------\n t/perf/p4002-diff-color-moved.sh | 45 ++++\n t/t4015-diff-whitespace.sh | 137 +++++++++++\n 3 files changed, 323 insertions(+), 236 deletions(-)\n create mode 100755 t/perf/p4002-diff-color-moved.sh\n\n\nbase-commit: 211eca0895794362184da2be2a2d812d070719d3\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-981%2Fphillipwood%2Fwip%2Fdiff-color-moved-tweaks-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-981/phillipwood/wip/diff-color-moved-tweaks-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/981\n\nRange-diff vs v1:\n\n -: ----------- > 1: 8fc8914a37b diff --color-moved: add perf tests\n 1: 374dbebcbf2 ! 2: 9b4e4d2674a diff --color-moved=zerba: fix alternate coloring\n @@ Metadata\n Author: Phillip Wood \n \n ## Commit message ##\n - diff --color-moved=zerba: fix alternate coloring\n + diff --color-moved=zebra: fix alternate coloring\n \n b0a2ba4776 (\"diff --color-moved=zebra: be stricter with color\n alternation\", 2018-11-23) sought to avoid using the alternate colors\n 2: 3d02a0a91a0 ! 3: 5512145c70f diff --color-moved: avoid false short line matches and bad zerba coloring\n @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n \tint pmb_nr = 0, pmb_alloc = 0;\n \tint n, flipped_block = 0, block_length = 0;\n -\tenum diff_symbol last_symbol = 0;\n -+\tenum diff_symbol moved_symbol = 0;\n ++\tenum diff_symbol moved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n -\t\t\tlast_symbol = l->s;\n +\t\t}\n +\t\tif (!match) {\n -+\t\t\tmoved_symbol = 0;\n ++\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n \t\t\tcontinue;\n \t\t}\n \n @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n +\t\t\tif (pmb_nr)\n +\t\t\t\tmoved_symbol = l->s;\n +\t\t\telse\n -+\t\t\t\tmoved_symbol = 0;\n ++\t\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n +\n \t\t\tblock_length = 0;\n \t\t}\n 3: 30f0ed44768 = 4: 93fdef30d64 diff: simplify allow-indentation-change delta calculation\n 4: ebb6eec1d92 ! 5: 6b7a8aed4ec diff --color-moved-ws=allow-indentation-change: simplify and optimize\n @@ Commit message\n comparison to filter out the non-matching lines. Fixing this reduces\n time to run\n git diff --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n - by 88% and simplifies the code.\n + by 93% compared to master and simplifies the code.\n \n - Before this change\n - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n - Time (mean ± σ): 9.978 s ± 0.042 s [User: 9.905 s, System: 0.057 s]\n - Range (min … max): 9.917 s … 10.037 s 10 runs\n -\n - After this change\n - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n - Time (mean ± σ): 1.220 s ± 0.004 s [User: 1.160 s, System: 0.058 s]\n - Range (min … max): 1.214 s … 1.226 s 10 runs\n + Test HEAD^ HEAD\n + ---------------------------------------------------------------------------------------------------------------\n + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41( 0.38+0.03) 0.41(0.37+0.04) +0.0%\n + 4002.2: diff --color-moved --no-color-moved-ws large change 0.83( 0.79+0.04) 0.82(0.79+0.02) -1.2%\n + 4002.3: diff --color-moved-ws=allow-indentation-change large change 13.68(13.59+0.07) 0.92(0.89+0.03) -93.3%\n + 4002.4: log --no-color-moved --no-color-moved-ws 1.31( 1.22+0.08) 1.31(1.21+0.10) +0.0%\n + 4002.5: log --color-moved --no-color-moved-ws 1.47( 1.40+0.07) 1.47(1.36+0.10) +0.0%\n + 4002.6: log --color-moved-ws=allow-indentation-change 1.87( 1.77+0.09) 1.50(1.41+0.09) -19.8%\n \n Signed-off-by: Phillip Wood \n \n 5: cec0c2d04d7 ! 6: cfbdd447eee diff --color-moved: call comparison function directly\n @@ Metadata\n ## Commit message ##\n diff --color-moved: call comparison function directly\n \n - Calling xdiff_compare_lines() directly rather than using a function\n - pointer from the hash map reduces the time very slightly but more\n - importantly it will allow us to easily combine pmb_advance_or_null()\n - and pmb_advance_or_null_multi_match() in the next commit.\n + This change will allow us to easily combine pmb_advance_or_null() and\n + pmb_advance_or_null_multi_match() in the next commit. Calling\n + xdiff_compare_lines() directly rather than using a function pointer\n + from the hash map has little effect on the run time.\n \n - Before this change\n - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n - Time (mean ± σ): 1.136 s ± 0.004 s [User: 1.079 s, System: 0.053 s]\n - Range (min … max): 1.130 s … 1.141 s 10 runs\n -\n - After this change\n - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n - Time (mean ± σ): 1.118 s ± 0.003 s [User: 1.062 s, System: 0.053 s]\n - Range (min … max): 1.114 s … 1.121 s 10 runs\n + Test HEAD^ HEAD\n + -------------------------------------------------------------------------------------------------------------\n + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41(0.37+0.04) 0.41(0.39+0.02) +0.0%\n + 4002.2: diff --color-moved --no-color-moved-ws large change 0.82(0.79+0.02) 0.83(0.79+0.03) +1.2%\n + 4002.3: diff --color-moved-ws=allow-indentation-change large change 0.92(0.89+0.03) 0.91(0.85+0.05) -1.1%\n + 4002.4: log --no-color-moved --no-color-moved-ws 1.31(1.21+0.10) 1.33(1.22+0.10) +1.5%\n + 4002.5: log --color-moved --no-color-moved-ws 1.47(1.36+0.10) 1.47(1.39+0.08) +0.0%\n + 4002.6: log --color-moved-ws=allow-indentation-change 1.50(1.41+0.09) 1.51(1.42+0.09) +0.7%\n \n Signed-off-by: Phillip Wood \n \n 6: 050cef0081d = 7: 73ce9b54e86 diff --color-moved: unify moved block growth functions\n 7: 9390e9a66eb = 8: ef8ce0e6ebc diff --color-moved: shrink potential moved blocks as we go\n 8: 1de99ac2bc3 = 9: 9d0a042eae1 diff --color-moved: stop clearing potential moved blocks\n 9: 41cdedd6090 ! 10: dd365ad115f diff --color-moved-ws=allow-indentation-change: improve hash lookups\n @@ Commit message\n As libxdiff does not have a whitespace flag to ignore the indentation\n the code for --color-moved-ws=allow-indentation-change uses\n XDF_IGNORE_WHITESPACE and then filters out any hash lookups where\n - there are non-indentation changes. This is filtering is inefficient as\n + there are non-indentation changes. This filtering is inefficient as\n we have to perform another string comparison.\n \n By using the offset data that we have already computed to skip the\n indentation we can avoid using XDF_IGNORE_WHITESPACE and safely remove\n - the extra checks which improves the performance by 14% and paves the\n + the extra checks which improves the performance by 11% and paves the\n way for the elimination of string comparisons in the next commit.\n \n - This change slightly increases the runtime of other --color-moved\n + This change slightly increases the run time of other --color-moved\n modes. This could be avoided by using different comparison functions\n - for the different modes but after the changes in the next commit there\n - is no measurable benefit.\n + for the different modes but after the next two commits there is no\n + measurable benefit in doing so.\n \n - Before this change\n - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n - Time (mean ± σ): 1.116 s ± 0.005 s [User: 1.057 s, System: 0.056 s]\n - Range (min … max): 1.109 s … 1.123 s 10 runs\n -\n - Benchmark #2: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n - Time (mean ± σ): 1.216 s ± 0.005 s [User: 1.155 s, System: 0.059 s]\n - Range (min … max): 1.206 s … 1.223 s 10 runs\n -\n - After this change\n - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n - Time (mean ± σ): 1.147 s ± 0.005 s [User: 1.085 s, System: 0.059 s]\n - Range (min … max): 1.140 s … 1.154 s 10 runs\n -\n - Benchmark #2: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n - Time (mean ± σ): 1.048 s ± 0.005 s [User: 987.4 ms, System: 58.8 ms]\n - Range (min … max): 1.043 s … 1.056 s 10 runs\n + Test HEAD^ HEAD\n + --------------------------------------------------------------------------------------------------------------\n + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41(0.38+0.03) 0.41(0.36+0.04) +0.0%\n + 4002.2: diff --color-moved --no-color-moved-ws large change 0.82(0.76+0.05) 0.84(0.79+0.04) +2.4%\n + 4002.3: diff --color-moved-ws=allow-indentation-change large change 0.91(0.88+0.03) 0.81(0.74+0.06) -11.0%\n + 4002.4: log --no-color-moved --no-color-moved-ws 1.32(1.21+0.10) 1.31(1.19+0.11) -0.8%\n + 4002.5: log --color-moved --no-color-moved-ws 1.47(1.37+0.10) 1.47(1.36+0.11) +0.0%\n + 4002.6: log --color-moved-ws=allow-indentation-change 1.51(1.42+0.09) 1.48(1.37+0.10) -2.0%\n \n Signed-off-by: Phillip Wood \n \n -: ----------- > 11: c160222ab3c diff: use designated initializers for emitted_diff_symbol\n 10: 220664dd907 ! 12: 753554587f9 diff --color-moved: intern strings\n @@ Commit message\n number of hash lookups a little (calculating the ids still involves\n one hash lookup per line) but the main benefit is that when growing\n blocks of potentially moved lines we can replace string comparisons\n - which involve chasing a pointer with a simple integer comparison. On\n - a large diff this commit reduces the time to run 'diff --color-moved'\n - by 33% and 'diff --color-moved-ws=allow-indentation-change' by 20%.\n + which involve chasing a pointer with a simple integer comparison.\n \n - Compared to master the time to run 'git log --patch --color-moved' is\n - increased by 2% and 'git log --patch\n - --color-moved-ws=allow-indentation-change' in reduced by 14%. These\n - timings were performed on an i5-7200U, on an i5-3470 both commands are\n - faster than master. The small speed decrease on commit sized diffs is\n - unfortunate but I think it is small enough to be worth it for the\n - gains on larger diffs.\n + On a large diff this commit reduces the time to run\n + diff --color-moved\n + by 33% and\n + diff --color-moved-ws=allow-indentation-change\n + by 26%. Compared to master the time to run\n + diff --color-moved-ws=allow-indentation-change\n + is now reduced by 95% and the overhead compared to --no-color-moved is\n + reduced to 50%.\n \n - Large diff before this change:\n - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n - Time (mean ± σ): 1.147 s ± 0.005 s [User: 1.085 s, System: 0.059 s]\n - Range (min … max): 1.140 s … 1.154 s 10 runs\n + Compared to the previous commit the time to run\n + git log --patch --color-moved\n + is increased slightly, but compared to master there is no change in\n + run time.\n \n - Benchmark #2: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n - Time (mean ± σ): 1.048 s ± 0.005 s [User: 987.4 ms, System: 58.8 ms]\n - Range (min … max): 1.043 s … 1.056 s 10 runs\n + Test HEAD^ HEAD\n + --------------------------------------------------------------------------------------------------------------\n + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41(0.36+0.04) 0.41(0.37+0.03) +0.0%\n + 4002.2: diff --color-moved --no-color-moved-ws large change 0.83(0.79+0.03) 0.55(0.52+0.03) -33.7%\n + 4002.3: diff --color-moved-ws=allow-indentation-change large change 0.81(0.77+0.04) 0.60(0.55+0.05) -25.9%\n + 4002.4: log --no-color-moved --no-color-moved-ws 1.30(1.20+0.09) 1.31(1.22+0.08) +0.8%\n + 4002.5: log --color-moved --no-color-moved-ws 1.46(1.35+0.11) 1.47(1.30+0.16) +0.7%\n + 4002.6: log --color-moved-ws=allow-indentation-change 1.46(1.38+0.07) 1.47(1.34+0.13) +0.7%\n \n - Large diff after this change\n - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n - Time (mean ± σ): 762.7 ms ± 2.8 ms [User: 707.5 ms, System: 53.7 ms]\n - Range (min … max): 758.0 ms … 767.0 ms 10 runs\n -\n - Benchmark #2: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n - Time (mean ± σ): 831.7 ms ± 1.7 ms [User: 776.5 ms, System: 53.3 ms]\n - Range (min … max): 829.2 ms … 835.1 ms 10 runs\n -\n - Small diffs on master\n - Benchmark #1: bin-wrappers/git log -p --diff-algorithm=myers --color-moved --no-color-moved-ws --no-merges -n1000 v2.29.0\n - Time (mean ± σ): 1.567 s ± 0.001 s [User: 1.443 s, System: 0.121 s]\n - Range (min … max): 1.566 s … 1.571 s 10 runs\n -\n - Benchmark #2: bin-wrappers/git log -p --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change -n1000 --no-merges v2.29.0\n - Time (mean ± σ): 1.865 s ± 0.008 s [User: 1.748 s, System: 0.112 s]\n - Range (min … max): 1.857 s … 1.881 s 10 runs\n -\n - Small diffs after this change\n - Benchmark #1: bin-wrappers/git log -p --diff-algorithm=myers --color-moved --no-color-moved-ws --no-merges -n1000 v2.29.0\n - Time (mean ± σ): 1.597 s ± 0.003 s [User: 1.413 s, System: 0.179 s]\n - Range (min … max): 1.591 s … 1.601 s 10 runs\n -\n - Benchmark #2: bin-wrappers/git log -p --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change -n1000 --no-merges v2.29.0\n - Time (mean ± σ): 1.606 s ± 0.006 s [User: 1.420 s, System: 0.181 s]\n - Range (min … max): 1.601 s … 1.622 s 10 runs\n + Test master HEAD\n + --------------------------------------------------------------------------------------------------------------\n + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.40( 0.36+0.03) 0.41(0.37+0.03) +2.5%\n + 4002.2: diff --color-moved --no-color-moved-ws large change 0.82( 0.77+0.04) 0.55(0.52+0.03) -32.9%\n + 4002.3: diff --color-moved-ws=allow-indentation-change large change 14.10(14.04+0.04) 0.60(0.55+0.05) -95.7%\n + 4002.4: log --no-color-moved --no-color-moved-ws 1.31( 1.21+0.09) 1.31(1.22+0.08) +0.0%\n + 4002.5: log --color-moved --no-color-moved-ws 1.47( 1.37+0.09) 1.47(1.30+0.16) +0.0%\n + 4002.6: log --color-moved-ws=allow-indentation-change 1.86( 1.76+0.10) 1.47(1.34+0.13) -21.0%\n \n Signed-off-by: Phillip Wood \n \n @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n \t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n \t\t\t\tif (o->color_moved_ws_handling &\n \t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n -@@ diff.c: static void emit_diff_symbol_from_struct(struct diff_options *o,\n - static void emit_diff_symbol(struct diff_options *o, enum diff_symbol s,\n - \t\t\t const char *line, int len, unsigned flags)\n - {\n --\tstruct emitted_diff_symbol e = {line, len, flags, 0, 0, s};\n -+\tstruct emitted_diff_symbol e = {line, len, flags, 0, 0, 0, s};\n - \n - \tif (o->emitted_symbols)\n - \t\tappend_emitted_diff_symbol(o, &e);\n @@ diff.c: static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \n \tif (o->emitted_symbols) {\n\n-- \ngitgitgadget\n"},{"id":"430597","messageId":"6b7a8aed4ec005b12e041846ba6c4251a2490554.1626777394.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v2.git.1626777393.gitgitgadget@gmail.com","subject":"[PATCH v2 05/12] diff --color-moved-ws=allow-indentation-change: simplify and optimize","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T10:36:26Z","receivedAt":"2021-07-20T10:39:17Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nIf we already have a block of potentially moved lines then as we move\ndown the diff we need to check if the next line of each potentially\nmoved line matches the current line of the diff. The implementation of\n--color-moved-ws=allow-indentation-change was needlessly performing\nthis check on all the lines in the diff that matched the current line\nrather than just the current line. To exacerbate the problem finding\nall the other lines in the diff that match the current line involves a\nfuzzy lookup so we were wasting even more time performing a second\ncomparison to filter out the non-matching lines. Fixing this reduces\ntime to run\n git diff --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\nby 93% compared to master and simplifies the code.\n\nTest HEAD^ HEAD\n---------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41( 0.38+0.03) 0.41(0.37+0.04) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.83( 0.79+0.04) 0.82(0.79+0.02) -1.2%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 13.68(13.59+0.07) 0.92(0.89+0.03) -93.3%\n4002.4: log --no-color-moved --no-color-moved-ws 1.31( 1.22+0.08) 1.31(1.21+0.10) +0.0%\n4002.5: log --color-moved --no-color-moved-ws 1.47( 1.40+0.07) 1.47(1.36+0.10) +0.0%\n4002.6: log --color-moved-ws=allow-indentation-change 1.87( 1.77+0.09) 1.50(1.41+0.09) -19.8%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 65 ++++++++++++++++------------------------------------------\n 1 file changed, 18 insertions(+), 47 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 77c893b266a..55384449170 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -881,35 +881,20 @@ static int compute_ws_delta(const struct emitted_diff_symbol *a,\n \n static int cmp_in_block_with_wsd(const struct diff_options *o,\n \t\t\t\t const struct moved_entry *cur,\n-\t\t\t\t const struct moved_entry *match,\n-\t\t\t\t struct moved_block *pmb,\n-\t\t\t\t int n)\n+\t\t\t\t const struct emitted_diff_symbol *l,\n+\t\t\t\t struct moved_block *pmb)\n {\n-\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n-\tint al = cur->es->len, bl = match->es->len, cl = l->len;\n+\tint al = cur->es->len, bl = l->len;\n \tconst char *a = cur->es->line,\n-\t\t *b = match->es->line,\n-\t\t *c = l->line;\n+\t\t *b = l->line;\n \tint a_off = cur->es->indent_off,\n \t a_width = cur->es->indent_width,\n-\t c_off = l->indent_off,\n-\t c_width = l->indent_width;\n+\t b_off = l->indent_off,\n+\t b_width = l->indent_width;\n \tint delta;\n \n-\t/*\n-\t * We need to check if 'cur' is equal to 'match'. As those\n-\t * are from the same (+/-) side, we do not need to adjust for\n-\t * indent changes. However these were found using fuzzy\n-\t * matching so we do have to check if they are equal. Here we\n-\t * just check the lengths. We delay calling memcmp() to check\n-\t * the contents until later as if the length comparison for a\n-\t * and c fails we can avoid the call all together.\n-\t */\n-\tif (al != bl)\n-\t\treturn 1;\n-\n \t/* If 'l' and 'cur' are both blank then they match. */\n-\tif (a_width == INDENT_BLANKLINE && c_width == INDENT_BLANKLINE)\n+\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n \t\treturn 0;\n \n \t/*\n@@ -918,7 +903,7 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \t * match those of the current block and that the text of 'l' and 'cur'\n \t * after the indentation match.\n \t */\n-\tdelta = c_width - a_width;\n+\tdelta = b_width - a_width;\n \n \t/*\n \t * If the previous lines of this block were all blank then set its\n@@ -927,9 +912,8 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \tif (pmb->wsd == INDENT_BLANKLINE)\n \t\tpmb->wsd = delta;\n \n-\treturn !(delta == pmb->wsd && al - a_off == cl - c_off &&\n-\t\t !memcmp(a, b, al) && !\n-\t\t memcmp(a + a_off, c + c_off, al - a_off));\n+\treturn !(delta == pmb->wsd && al - a_off == bl - b_off &&\n+\t\t !memcmp(a + a_off, b + b_off, al - a_off));\n }\n \n static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n@@ -1030,36 +1014,23 @@ static void pmb_advance_or_null(struct diff_options *o,\n }\n \n static void pmb_advance_or_null_multi_match(struct diff_options *o,\n-\t\t\t\t\t struct moved_entry *match,\n-\t\t\t\t\t struct hashmap *hm,\n+\t\t\t\t\t struct emitted_diff_symbol *l,\n \t\t\t\t\t struct moved_block *pmb,\n-\t\t\t\t\t int pmb_nr, int n)\n+\t\t\t\t\t int pmb_nr)\n {\n \tint i;\n-\tchar *got_match = xcalloc(1, pmb_nr);\n-\n-\thashmap_for_each_entry_from(hm, match, ent) {\n-\t\tfor (i = 0; i < pmb_nr; i++) {\n-\t\t\tstruct moved_entry *prev = pmb[i].match;\n-\t\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n-\t\t\t\t\tprev->next_line : NULL;\n-\t\t\tif (!cur)\n-\t\t\t\tcontinue;\n-\t\t\tif (!cmp_in_block_with_wsd(o, cur, match, &pmb[i], n))\n-\t\t\t\tgot_match[i] |= 1;\n-\t\t}\n-\t}\n \n \tfor (i = 0; i < pmb_nr; i++) {\n-\t\tif (got_match[i]) {\n+\t\tstruct moved_entry *prev = pmb[i].match;\n+\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n+\t\t\tprev->next_line : NULL;\n+\t\tif (cur && !cmp_in_block_with_wsd(o, cur, l, &pmb[i])) {\n \t\t\t/* Advance to the next line */\n-\t\t\tpmb[i].match = pmb[i].match->next_line;\n+\t\t\tpmb[i].match = cur;\n \t\t} else {\n \t\t\tmoved_block_clear(&pmb[i]);\n \t\t}\n \t}\n-\n-\tfree(got_match);\n }\n \n static int shrink_potential_moved_blocks(struct moved_block *pmb,\n@@ -1181,7 +1152,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \n \t\tif (o->color_moved_ws_handling &\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\tpmb_advance_or_null_multi_match(o, match, hm, pmb, pmb_nr, n);\n+\t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n \t\telse\n \t\t\tpmb_advance_or_null(o, match, hm, pmb, pmb_nr);\n \n-- \ngitgitgadget\n\n"},{"id":"430598","messageId":"93fdef30d64dd86493733c28f67ee22bcf5e6f58.1626777394.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v2.git.1626777393.gitgitgadget@gmail.com","subject":"[PATCH v2 04/12] diff: simplify allow-indentation-change delta calculation","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T10:36:25Z","receivedAt":"2021-07-20T10:39:25Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nNow that we reliably end a block when the sign changes we don't need\nthe whitespace delta calculation to rely on the sign.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 13 ++-----------\n 1 file changed, 2 insertions(+), 11 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 2b51b77fd20..77c893b266a 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -864,23 +864,17 @@ static int compute_ws_delta(const struct emitted_diff_symbol *a,\n \t a_width = a->indent_width,\n \t b_off = b->indent_off,\n \t b_width = b->indent_width;\n-\tint delta;\n \n \tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE) {\n \t\t*out = INDENT_BLANKLINE;\n \t\treturn 1;\n \t}\n \n-\tif (a->s == DIFF_SYMBOL_PLUS)\n-\t\tdelta = a_width - b_width;\n-\telse\n-\t\tdelta = b_width - a_width;\n-\n \tif (a_len - a_off != b_len - b_off ||\n \t memcmp(a->line + a_off, b->line + b_off, a_len - a_off))\n \t\treturn 0;\n \n-\t*out = delta;\n+\t*out = a_width - b_width;\n \n \treturn 1;\n }\n@@ -924,10 +918,7 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \t * match those of the current block and that the text of 'l' and 'cur'\n \t * after the indentation match.\n \t */\n-\tif (cur->es->s == DIFF_SYMBOL_PLUS)\n-\t\tdelta = a_width - c_width;\n-\telse\n-\t\tdelta = c_width - a_width;\n+\tdelta = c_width - a_width;\n \n \t/*\n \t * If the previous lines of this block were all blank then set its\n-- \ngitgitgadget\n\n"},{"id":"430599","messageId":"5512145c70ff47e9ab239f4f896498edf4c16745.1626777393.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v2.git.1626777393.gitgitgadget@gmail.com","subject":"[PATCH v2 03/12] diff --color-moved: avoid false short line matches and bad zerba coloring","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T10:36:24Z","receivedAt":"2021-07-20T10:39:33Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nWhen marking moved lines it is possible for a block of potential\nmatched lines to extend past a change in sign when there is a sequence\nof added lines whose text matches the text of a sequence of deleted\nand added lines. Most of the time either `match` will be NULL or\n`pmb_advance_or_null()` will fail when the loop encounters a change of\nsign but there are corner cases where `match` is non-NULL and\n`pmb_advance_or_null()` successfully advances the moved block despite\nthe change in sign.\n\nOne consequence of this is highlighting a short line as moved when it\nshould not be. For example\n\n-moved line # Correctly highlighted as moved\n+short line # Wrongly highlighted as moved\n context\n+moved line # Correctly highlighted as moved\n+short line\n context\n-short line\n\nThe other consequence is coloring a moved addition following a moved\ndeletion in the wrong color. In the example below the first \"+moved\nline 3\" should be highlighted as newMoved not newMovedAlternate.\n\n-moved line 1 # Correctly highlighted as oldMoved\n-moved line 2 # Correctly highlighted as oldMovedAlternate\n+moved line 3 # Wrongly highlighted as newMovedAlternate\n context # Everything else is highlighted correctly\n+moved line 2\n+moved line 3\n context\n+moved line 1\n-moved line 3\n\nThese false matches are more likely when using --color-moved-ws with\nthe exception of --color-moved-ws=allow-indentation-change which ties\nthe sign of the current whitespace delta to the sign of the line to\navoid this problem. The fix is to check that the sign of the new line\nbeing matched is the same as the sign of the line that started the\nblock of potential matches.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 17 ++++++----\n t/t4015-diff-whitespace.sh | 65 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 76 insertions(+), 6 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex cb068f8258c..2b51b77fd20 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1142,7 +1142,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n \tint n, flipped_block = 0, block_length = 0;\n-\tenum diff_symbol last_symbol = 0;\n+\tenum diff_symbol moved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n@@ -1168,7 +1168,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tflipped_block = 0;\n \t\t}\n \n-\t\tif (!match) {\n+\t\tif (pmb_nr && (!match || l->s != moved_symbol)) {\n \t\t\tint i;\n \n \t\t\tadjust_last_block(o, n, block_length);\n@@ -1177,12 +1177,13 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tpmb_nr = 0;\n \t\t\tblock_length = 0;\n \t\t\tflipped_block = 0;\n-\t\t\tlast_symbol = l->s;\n+\t\t}\n+\t\tif (!match) {\n+\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (o->color_moved == COLOR_MOVED_PLAIN) {\n-\t\t\tlast_symbol = l->s;\n \t\t\tl->flags |= DIFF_SYMBOL_MOVED_LINE;\n \t\t\tcontinue;\n \t\t}\n@@ -1214,11 +1215,16 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t}\n \n \t\t\tif (adjust_last_block(o, n, block_length) &&\n-\t\t\t pmb_nr && last_symbol == l->s)\n+\t\t\t pmb_nr && moved_symbol == l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n \t\t\telse\n \t\t\t\tflipped_block = 0;\n \n+\t\t\tif (pmb_nr)\n+\t\t\t\tmoved_symbol = l->s;\n+\t\t\telse\n+\t\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n+\n \t\t\tblock_length = 0;\n \t\t}\n \n@@ -1228,7 +1234,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tif (flipped_block && o->color_moved != COLOR_MOVED_BLOCKS)\n \t\t\t\tl->flags |= DIFF_SYMBOL_MOVED_LINE_ALT;\n \t\t}\n-\t\tlast_symbol = l->s;\n \t}\n \tadjust_last_block(o, n, block_length);\n \ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 920114cd795..3119a59f071 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1514,6 +1514,71 @@ test_expect_success 'zebra alternate color is only used when necessary' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'short lines of opposite sign do not get marked as moved' '\n+\tcat >old.txt <<-\\EOF &&\n+\tthis line should be marked as moved\n+\tunchanged\n+\tunchanged\n+\tunchanged\n+\tunchanged\n+\ttoo short\n+\tthis line should be marked as oldMoved newMoved\n+\tthis line should be marked as oldMovedAlternate newMoved\n+\tunchanged 1\n+\tunchanged 2\n+\tunchanged 3\n+\tunchanged 4\n+\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n+\tEOF\n+\tcat >new.txt <<-\\EOF &&\n+\ttoo short\n+\tunchanged\n+\tunchanged\n+\tthis line should be marked as moved\n+\ttoo short\n+\tunchanged\n+\tunchanged\n+\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n+\tunchanged 1\n+\tunchanged 2\n+\tthis line should be marked as oldMovedAlternate newMoved\n+\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n+\tunchanged 3\n+\tthis line should be marked as oldMoved newMoved\n+\tunchanged 4\n+\tEOF\n+\ttest_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n+\t\told.txt new.txt >output && cat output &&\n+\tgrep -v index output | test_decode_color >actual &&\n+\tcat >expect <<-\\EOF &&\n+\tdiff --git a/old.txt b/new.txt\n+\t--- a/old.txt\n+\t+++ b/new.txt\n+\t@@ -1,13 +1,15 @@\n+\t-this line should be marked as moved\n+\t+too short\n+\t unchanged\n+\t unchanged\n+\t+this line should be marked as moved\n+\t+too short\n+\t unchanged\n+\t unchanged\n+\t-too short\n+\t-this line should be marked as oldMoved newMoved\n+\t-this line should be marked as oldMovedAlternate newMoved\n+\t+this line should be marked as oldMoved newMoved/newMovedAlternate\n+\t unchanged 1\n+\t unchanged 2\n+\t+this line should be marked as oldMovedAlternate newMoved\n+\t+this line should be marked as oldMoved newMoved/newMovedAlternate\n+\t unchanged 3\n+\t+this line should be marked as oldMoved newMoved\n+\t unchanged 4\n+\t-this line should be marked as oldMoved newMoved/newMovedAlternate\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'cmd option assumes configured colored-moved' '\n \ttest_config color.diff.oldMoved \"magenta\" &&\n \ttest_config color.diff.newMoved \"cyan\" &&\n-- \ngitgitgadget\n\n"},{"id":"430600","messageId":"9b4e4d2674a38b6bed793e0f5661edefa6df5d23.1626777393.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v2.git.1626777393.gitgitgadget@gmail.com","subject":"[PATCH v2 02/12] diff --color-moved=zebra: fix alternate coloring","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T10:36:23Z","receivedAt":"2021-07-20T10:39:45Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nb0a2ba4776 (\"diff --color-moved=zebra: be stricter with color\nalternation\", 2018-11-23) sought to avoid using the alternate colors\nunless there are two adjacent moved blocks of the same\nsign. Unfortunately it contains two bugs that prevented it from fixing\nthe problem properly. Firstly `last_symbol` is reset at the start of\neach iteration of the loop losing the symbol of the last line and\nsecondly when deciding whether to use the alternate color it should be\nchecking if the current line is the same sign of the last line, not a\ndifferent sign. The combination of the two errors means that we still\nuse the alternate color when we should do but we also use it when we\nshouldn't. This is most noticable when using\n--color-moved-ws=allow-indentation-change with hunks like\n\n-this line gets indented\n+ this line gets indented\n\nwhere the post image is colored with newMovedAlternate rather than\nnewMoved. While this does not matter much, the next commit will change\nthe coloring to be correct in this case, so lets fix the bug here to\nmake it clear why the output is changing and add a regression test.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 4 +--\n t/t4015-diff-whitespace.sh | 72 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 74 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 52c791574b7..cb068f8258c 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1142,6 +1142,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n \tint n, flipped_block = 0, block_length = 0;\n+\tenum diff_symbol last_symbol = 0;\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n@@ -1149,7 +1150,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\tstruct moved_entry *key;\n \t\tstruct moved_entry *match = NULL;\n \t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n-\t\tenum diff_symbol last_symbol = 0;\n \n \t\tswitch (l->s) {\n \t\tcase DIFF_SYMBOL_PLUS:\n@@ -1214,7 +1214,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t}\n \n \t\t\tif (adjust_last_block(o, n, block_length) &&\n-\t\t\t pmb_nr && last_symbol != l->s)\n+\t\t\t pmb_nr && last_symbol == l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n \t\t\telse\n \t\t\t\tflipped_block = 0;\ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 2c13b62d3c6..920114cd795 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1442,6 +1442,78 @@ test_expect_success 'detect permutations inside moved code -- dimmed-zebra' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'zebra alternate color is only used when necessary' '\n+\tcat >old.txt <<-\\EOF &&\n+\tline 1A should be marked as oldMoved newMovedAlternate\n+\tline 1B should be marked as oldMoved newMovedAlternate\n+\tunchanged\n+\tline 2A should be marked as oldMoved newMovedAlternate\n+\tline 2B should be marked as oldMoved newMovedAlternate\n+\tline 3A should be marked as oldMovedAlternate newMoved\n+\tline 3B should be marked as oldMovedAlternate newMoved\n+\tunchanged\n+\tline 4A should be marked as oldMoved newMovedAlternate\n+\tline 4B should be marked as oldMoved newMovedAlternate\n+\tline 5A should be marked as oldMovedAlternate newMoved\n+\tline 5B should be marked as oldMovedAlternate newMoved\n+\tline 6A should be marked as oldMoved newMoved\n+\tline 6B should be marked as oldMoved newMoved\n+\tEOF\n+\tcat >new.txt <<-\\EOF &&\n+\t line 1A should be marked as oldMoved newMovedAlternate\n+\t line 1B should be marked as oldMoved newMovedAlternate\n+\tunchanged\n+\t line 3A should be marked as oldMovedAlternate newMoved\n+\t line 3B should be marked as oldMovedAlternate newMoved\n+\t line 2A should be marked as oldMoved newMovedAlternate\n+\t line 2B should be marked as oldMoved newMovedAlternate\n+\tunchanged\n+\t line 6A should be marked as oldMoved newMoved\n+\t line 6B should be marked as oldMoved newMoved\n+\t line 4A should be marked as oldMoved newMovedAlternate\n+\t line 4B should be marked as oldMoved newMovedAlternate\n+\t line 5A should be marked as oldMovedAlternate newMoved\n+\t line 5B should be marked as oldMovedAlternate newMoved\n+\tEOF\n+\ttest_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n+\t\t --color-moved-ws=allow-indentation-change \\\n+\t\t old.txt new.txt >output &&\n+\tgrep -v index output | test_decode_color >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tdiff --git a/old.txt b/new.txt\n+\t--- a/old.txt\n+\t+++ b/new.txt\n+\t@@ -1,14 +1,14 @@\n+\t-line 1A should be marked as oldMoved newMovedAlternate\n+\t-line 1B should be marked as oldMoved newMovedAlternate\n+\t+ line 1A should be marked as oldMoved newMovedAlternate\n+\t+ line 1B should be marked as oldMoved newMovedAlternate\n+\t unchanged\n+\t-line 2A should be marked as oldMoved newMovedAlternate\n+\t-line 2B should be marked as oldMoved newMovedAlternate\n+\t-line 3A should be marked as oldMovedAlternate newMoved\n+\t-line 3B should be marked as oldMovedAlternate newMoved\n+\t+ line 3A should be marked as oldMovedAlternate newMoved\n+\t+ line 3B should be marked as oldMovedAlternate newMoved\n+\t+ line 2A should be marked as oldMoved newMovedAlternate\n+\t+ line 2B should be marked as oldMoved newMovedAlternate\n+\t unchanged\n+\t-line 4A should be marked as oldMoved newMovedAlternate\n+\t-line 4B should be marked as oldMoved newMovedAlternate\n+\t-line 5A should be marked as oldMovedAlternate newMoved\n+\t-line 5B should be marked as oldMovedAlternate newMoved\n+\t-line 6A should be marked as oldMoved newMoved\n+\t-line 6B should be marked as oldMoved newMoved\n+\t+ line 6A should be marked as oldMoved newMoved\n+\t+ line 6B should be marked as oldMoved newMoved\n+\t+ line 4A should be marked as oldMoved newMovedAlternate\n+\t+ line 4B should be marked as oldMoved newMovedAlternate\n+\t+ line 5A should be marked as oldMovedAlternate newMoved\n+\t+ line 5B should be marked as oldMovedAlternate newMoved\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'cmd option assumes configured colored-moved' '\n \ttest_config color.diff.oldMoved \"magenta\" &&\n \ttest_config color.diff.newMoved \"cyan\" &&\n-- \ngitgitgadget\n\n"},{"id":"430601","messageId":"8fc8914a37b3c343cd92bb0255088f7b000ff7f7.1626777393.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v2.git.1626777393.gitgitgadget@gmail.com","subject":"[PATCH v2 01/12] diff --color-moved: add perf tests","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T10:36:22Z","receivedAt":"2021-07-20T10:39:48Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nAdd some tests so we can monitor changes to the performance of the\nmove detection code. The tests record the performance of a single\nlarge diff and a sequence of smaller diffs.\n\nSigned-off-by: Phillip Wood \n---\n t/perf/p4002-diff-color-moved.sh | 45 ++++++++++++++++++++++++++++++++\n 1 file changed, 45 insertions(+)\n create mode 100755 t/perf/p4002-diff-color-moved.sh\n\ndiff --git a/t/perf/p4002-diff-color-moved.sh b/t/perf/p4002-diff-color-moved.sh\nnew file mode 100755\nindex 00000000000..ad56bcb71e4\n--- /dev/null\n+++ b/t/perf/p4002-diff-color-moved.sh\n@@ -0,0 +1,45 @@\n+#!/bin/sh\n+\n+test_description='Tests diff --color-moved performance'\n+. ./perf-lib.sh\n+\n+test_perf_default_repo\n+\n+if ! git rev-parse --verify v2.29.0^{commit} >/dev/null\n+then\n+\tskip_all='skipping because tag v2.29.0 was not found'\n+\ttest_done\n+fi\n+\n+GIT_PAGER_IN_USE=1\n+test_export GIT_PAGER_IN_USE\n+\n+test_perf 'diff --no-color-moved --no-color-moved-ws large change' '\n+\tgit diff --no-color-moved --no-color-moved-ws v2.28.0 v2.29.0\n+'\n+\n+test_perf 'diff --color-moved --no-color-moved-ws large change' '\n+\tgit diff --color-moved=zebra --no-color-moved-ws v2.28.0 v2.29.0\n+'\n+\n+test_perf 'diff --color-moved-ws=allow-indentation-change large change' '\n+\tgit diff --color-moved=zebra --color-moved-ws=allow-indentation-change \\\n+\t\tv2.28.0 v2.29.0\n+'\n+\n+test_perf 'log --no-color-moved --no-color-moved-ws' '\n+\tgit log --no-color-moved --no-color-moved-ws --no-merges --patch \\\n+\t\t-n1000 v2.29.0\n+'\n+\n+test_perf 'log --color-moved --no-color-moved-ws' '\n+\tgit log --color-moved=zebra --no-color-moved-ws --no-merges --patch \\\n+\t\t-n1000 v2.29.0\n+'\n+\n+test_perf 'log --color-moved-ws=allow-indentation-change' '\n+\tgit log --color-moved=zebra --color-moved-ws=allow-indentation-change \\\n+\t\t--no-merges --patch -n1000 v2.29.0\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"430602","messageId":"73ce9b54e869740ff23253d4153405444c673276.1626777394.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v2.git.1626777393.gitgitgadget@gmail.com","subject":"[PATCH v2 07/12] diff --color-moved: unify moved block growth functions","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T10:36:28Z","receivedAt":"2021-07-20T10:40:34Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nAfter the last two commits pmb_advance_or_null() and\npmb_advance_or_null_multi_match() differ only in the comparison they\nperform. Lets simplify the code by combining them into a single\nfunction.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 41 ++++++++++++-----------------------------\n 1 file changed, 12 insertions(+), 29 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex c056d917d0d..b03f79b626c 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1003,36 +1003,23 @@ static void pmb_advance_or_null(struct diff_options *o,\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \n \tfor (i = 0; i < pmb_nr; i++) {\n+\t\tint match;\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n \t\t\t\tprev->next_line : NULL;\n-\t\tif (cur && xdiff_compare_lines(cur->es->line, cur->es->len,\n-\t\t\t\t\t\tl->line, l->len,\n-\t\t\t\t\t\tflags)) {\n-\t\t\tpmb[i].match = cur;\n-\t\t} else {\n-\t\t\tpmb[i].match = NULL;\n-\t\t}\n-\t}\n-}\n \n-static void pmb_advance_or_null_multi_match(struct diff_options *o,\n-\t\t\t\t\t struct emitted_diff_symbol *l,\n-\t\t\t\t\t struct moved_block *pmb,\n-\t\t\t\t\t int pmb_nr)\n-{\n-\tint i;\n-\n-\tfor (i = 0; i < pmb_nr; i++) {\n-\t\tstruct moved_entry *prev = pmb[i].match;\n-\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n-\t\t\tprev->next_line : NULL;\n-\t\tif (cur && !cmp_in_block_with_wsd(o, cur, l, &pmb[i])) {\n-\t\t\t/* Advance to the next line */\n+\t\tif (o->color_moved_ws_handling &\n+\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n+\t\t\tmatch = cur &&\n+\t\t\t\t!cmp_in_block_with_wsd(o, cur, l, &pmb[i]);\n+\t\telse\n+\t\t\tmatch = cur &&\n+\t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n+\t\t\t\t\t\t l->line, l->len, flags);\n+\t\tif (match)\n \t\t\tpmb[i].match = cur;\n-\t\t} else {\n+\t\telse\n \t\t\tmoved_block_clear(&pmb[i]);\n-\t\t}\n \t}\n }\n \n@@ -1153,11 +1140,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tif (o->color_moved_ws_handling &\n-\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n-\t\telse\n-\t\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n+\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n \n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n-- \ngitgitgadget\n\n"},{"id":"430603","messageId":"cfbdd447eeeff41a7cd32ba513ae302588438753.1626777394.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v2.git.1626777393.gitgitgadget@gmail.com","subject":"[PATCH v2 06/12] diff --color-moved: call comparison function directly","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T10:36:27Z","receivedAt":"2021-07-20T10:40:34Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nThis change will allow us to easily combine pmb_advance_or_null() and\npmb_advance_or_null_multi_match() in the next commit. Calling\nxdiff_compare_lines() directly rather than using a function pointer\nfrom the hash map has little effect on the run time.\n\nTest HEAD^ HEAD\n-------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41(0.37+0.04) 0.41(0.39+0.02) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.82(0.79+0.02) 0.83(0.79+0.03) +1.2%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 0.92(0.89+0.03) 0.91(0.85+0.05) -1.1%\n4002.4: log --no-color-moved --no-color-moved-ws 1.31(1.21+0.10) 1.33(1.22+0.10) +1.5%\n4002.5: log --color-moved --no-color-moved-ws 1.47(1.36+0.10) 1.47(1.39+0.08) +0.0%\n4002.6: log --color-moved-ws=allow-indentation-change 1.50(1.41+0.09) 1.51(1.42+0.09) +0.7%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 11 +++++++----\n 1 file changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 55384449170..c056d917d0d 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -995,17 +995,20 @@ static void add_lines_to_move_detection(struct diff_options *o,\n }\n \n static void pmb_advance_or_null(struct diff_options *o,\n-\t\t\t\tstruct moved_entry *match,\n-\t\t\t\tstruct hashmap *hm,\n+\t\t\t\tstruct emitted_diff_symbol *l,\n \t\t\t\tstruct moved_block *pmb,\n \t\t\t\tint pmb_nr)\n {\n \tint i;\n+\tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n+\n \tfor (i = 0; i < pmb_nr; i++) {\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n \t\t\t\tprev->next_line : NULL;\n-\t\tif (cur && !hm->cmpfn(o, &cur->ent, &match->ent, NULL)) {\n+\t\tif (cur && xdiff_compare_lines(cur->es->line, cur->es->len,\n+\t\t\t\t\t\tl->line, l->len,\n+\t\t\t\t\t\tflags)) {\n \t\t\tpmb[i].match = cur;\n \t\t} else {\n \t\t\tpmb[i].match = NULL;\n@@ -1154,7 +1157,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n \t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n \t\telse\n-\t\t\tpmb_advance_or_null(o, match, hm, pmb, pmb_nr);\n+\t\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n \n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n-- \ngitgitgadget\n\n"},{"id":"430604","messageId":"9d0a042eae17e449d6fd5a4cb7df31a8a02d972c.1626777394.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v2.git.1626777393.gitgitgadget@gmail.com","subject":"[PATCH v2 09/12] diff --color-moved: stop clearing potential moved blocks","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T10:36:30Z","receivedAt":"2021-07-20T10:40:46Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nmoved_block_clear() was introduced in 74d156f4a1 (\"diff\n--color-moved-ws: fix double free crash\", 2018-10-04) to free the\nmemory that was allocated when initializing a potential moved\nblock. However since 21536d077f (\"diff --color-moved-ws: modify\nallow-indentation-change\", 2018-11-23) initializing a potential moved\nblock no longer allocates any memory. Up until the last commit we were\nrelying on moved_block_clear() to set the `match` pointer to NULL when\na block stopped matching, but since that commit we do not clear a\nmoved block that does not match so it does not make sense to clear\nthem elsewhere.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 11 -----------\n 1 file changed, 11 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 068473c0be3..4b5776a5a0a 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -807,11 +807,6 @@ struct moved_block {\n \tint wsd; /* The whitespace delta of this block */\n };\n \n-static void moved_block_clear(struct moved_block *b)\n-{\n-\tmemset(b, 0, sizeof(*b));\n-}\n-\n #define INDENT_BLANKLINE INT_MIN\n \n static void fill_es_indent_data(struct emitted_diff_symbol *es)\n@@ -1093,11 +1088,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t}\n \n \t\tif (pmb_nr && (!match || l->s != moved_symbol)) {\n-\t\t\tint i;\n-\n \t\t\tadjust_last_block(o, n, block_length);\n-\t\t\tfor(i = 0; i < pmb_nr; i++)\n-\t\t\t\tmoved_block_clear(&pmb[i]);\n \t\t\tpmb_nr = 0;\n \t\t\tblock_length = 0;\n \t\t\tflipped_block = 0;\n@@ -1155,8 +1146,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t}\n \tadjust_last_block(o, n, block_length);\n \n-\tfor(n = 0; n < pmb_nr; n++)\n-\t\tmoved_block_clear(&pmb[n]);\n \tfree(pmb);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"430605","messageId":"ef8ce0e6ebceb715e75c567a3caf5a64052b5312.1626777394.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v2.git.1626777393.gitgitgadget@gmail.com","subject":"[PATCH v2 08/12] diff --color-moved: shrink potential moved blocks as we go","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T10:36:29Z","receivedAt":"2021-07-20T10:40:52Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nRather than setting `match` to NULL and then looping over the list of\npotential matched blocks for a second time to remove blocks with no\nmatches just filter out the blocks with no matches as we go.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 42 ++++++------------------------------------\n 1 file changed, 6 insertions(+), 36 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex b03f79b626c..068473c0be3 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -997,12 +997,12 @@ static void add_lines_to_move_detection(struct diff_options *o,\n static void pmb_advance_or_null(struct diff_options *o,\n \t\t\t\tstruct emitted_diff_symbol *l,\n \t\t\t\tstruct moved_block *pmb,\n-\t\t\t\tint pmb_nr)\n+\t\t\t\tint *pmb_nr)\n {\n-\tint i;\n+\tint i, j;\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \n-\tfor (i = 0; i < pmb_nr; i++) {\n+\tfor (i = 0, j = 0; i < *pmb_nr; i++) {\n \t\tint match;\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n@@ -1017,37 +1017,9 @@ static void pmb_advance_or_null(struct diff_options *o,\n \t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n \t\t\t\t\t\t l->line, l->len, flags);\n \t\tif (match)\n-\t\t\tpmb[i].match = cur;\n-\t\telse\n-\t\t\tmoved_block_clear(&pmb[i]);\n+\t\t\tpmb[j++].match = cur;\n \t}\n-}\n-\n-static int shrink_potential_moved_blocks(struct moved_block *pmb,\n-\t\t\t\t\t int pmb_nr)\n-{\n-\tint lp, rp;\n-\n-\t/* Shrink the set of potential block to the remaining running */\n-\tfor (lp = 0, rp = pmb_nr - 1; lp <= rp;) {\n-\t\twhile (lp < pmb_nr && pmb[lp].match)\n-\t\t\tlp++;\n-\t\t/* lp points at the first NULL now */\n-\n-\t\twhile (rp > -1 && !pmb[rp].match)\n-\t\t\trp--;\n-\t\t/* rp points at the last non-NULL */\n-\n-\t\tif (lp < pmb_nr && rp > -1 && lp < rp) {\n-\t\t\tpmb[lp] = pmb[rp];\n-\t\t\tmemset(&pmb[rp], 0, sizeof(pmb[rp]));\n-\t\t\trp--;\n-\t\t\tlp++;\n-\t\t}\n-\t}\n-\n-\t/* Remember the number of running sets */\n-\treturn rp + 1;\n+\t*pmb_nr = j;\n }\n \n /*\n@@ -1140,9 +1112,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n-\n-\t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n+\t\tpmb_advance_or_null(o, l, pmb, &pmb_nr);\n \n \t\tif (pmb_nr == 0) {\n \t\t\t/*\n-- \ngitgitgadget\n\n"},{"id":"430606","messageId":"c160222ab3cee77c88cb947f4bf18993a923d53b.1626777394.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v2.git.1626777393.gitgitgadget@gmail.com","subject":"[PATCH v2 11/12] diff: use designated initializers for emitted_diff_symbol","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T10:36:32Z","receivedAt":"2021-07-20T10:41:11Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nThis makes it clearer which fields are being explicitly initialized\nand will simplify the next commit where we add a new field to the\nstruct.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/diff.c b/diff.c\nindex f899083d028..31a20a34240 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1460,7 +1460,9 @@ static void emit_diff_symbol_from_struct(struct diff_options *o,\n static void emit_diff_symbol(struct diff_options *o, enum diff_symbol s,\n \t\t\t const char *line, int len, unsigned flags)\n {\n-\tstruct emitted_diff_symbol e = {line, len, flags, 0, 0, s};\n+\tstruct emitted_diff_symbol e = {\n+\t\t.line = line, .len = len, .flags = flags, .s = s\n+\t};\n \n \tif (o->emitted_symbols)\n \t\tappend_emitted_diff_symbol(o, &e);\n-- \ngitgitgadget\n\n"},{"id":"430607","messageId":"dd365ad115ffcf3ce5b4b14864ac3ff0ac292214.1626777394.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v2.git.1626777393.gitgitgadget@gmail.com","subject":"[PATCH v2 10/12] diff --color-moved-ws=allow-indentation-change: improve hash lookups","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T10:36:31Z","receivedAt":"2021-07-20T10:41:33Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nAs libxdiff does not have a whitespace flag to ignore the indentation\nthe code for --color-moved-ws=allow-indentation-change uses\nXDF_IGNORE_WHITESPACE and then filters out any hash lookups where\nthere are non-indentation changes. This filtering is inefficient as\nwe have to perform another string comparison.\n\nBy using the offset data that we have already computed to skip the\nindentation we can avoid using XDF_IGNORE_WHITESPACE and safely remove\nthe extra checks which improves the performance by 11% and paves the\nway for the elimination of string comparisons in the next commit.\n\nThis change slightly increases the run time of other --color-moved\nmodes. This could be avoided by using different comparison functions\nfor the different modes but after the next two commits there is no\nmeasurable benefit in doing so.\n\nTest HEAD^ HEAD\n--------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41(0.38+0.03) 0.41(0.36+0.04) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.82(0.76+0.05) 0.84(0.79+0.04) +2.4%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 0.91(0.88+0.03) 0.81(0.74+0.06) -11.0%\n4002.4: log --no-color-moved --no-color-moved-ws 1.32(1.21+0.10) 1.31(1.19+0.11) -0.8%\n4002.5: log --color-moved --no-color-moved-ws 1.47(1.37+0.10) 1.47(1.36+0.11) +0.0%\n4002.6: log --color-moved-ws=allow-indentation-change 1.51(1.42+0.09) 1.48(1.37+0.10) -2.0%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 66 +++++++++++++++++-----------------------------------------\n 1 file changed, 19 insertions(+), 47 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 4b5776a5a0a..f899083d028 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -850,28 +850,15 @@ static void fill_es_indent_data(struct emitted_diff_symbol *es)\n }\n \n static int compute_ws_delta(const struct emitted_diff_symbol *a,\n-\t\t\t const struct emitted_diff_symbol *b,\n-\t\t\t int *out)\n-{\n-\tint a_len = a->len,\n-\t b_len = b->len,\n-\t a_off = a->indent_off,\n-\t a_width = a->indent_width,\n-\t b_off = b->indent_off,\n+\t\t\t const struct emitted_diff_symbol *b)\n+{\n+\tint a_width = a->indent_width,\n \t b_width = b->indent_width;\n \n-\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE) {\n-\t\t*out = INDENT_BLANKLINE;\n-\t\treturn 1;\n-\t}\n-\n-\tif (a_len - a_off != b_len - b_off ||\n-\t memcmp(a->line + a_off, b->line + b_off, a_len - a_off))\n-\t\treturn 0;\n-\n-\t*out = a_width - b_width;\n+\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n+\t\treturn INDENT_BLANKLINE;\n \n-\treturn 1;\n+\treturn a_width - b_width;\n }\n \n static int cmp_in_block_with_wsd(const struct diff_options *o,\n@@ -917,26 +904,17 @@ static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n \t\t\t const void *keydata)\n {\n \tconst struct diff_options *diffopt = hashmap_cmp_fn_data;\n-\tconst struct moved_entry *a, *b;\n+\tconst struct emitted_diff_symbol *a, *b;\n \tunsigned flags = diffopt->color_moved_ws_handling\n \t\t\t & XDF_WHITESPACE_FLAGS;\n \n-\ta = container_of(eptr, const struct moved_entry, ent);\n-\tb = container_of(entry_or_key, const struct moved_entry, ent);\n-\n-\tif (diffopt->color_moved_ws_handling &\n-\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t/*\n-\t\t * As there is not specific white space config given,\n-\t\t * we'd need to check for a new block, so ignore all\n-\t\t * white space. The setup of the white space\n-\t\t * configuration for the next block is done else where\n-\t\t */\n-\t\tflags |= XDF_IGNORE_WHITESPACE;\n+\ta = container_of(eptr, const struct moved_entry, ent)->es;\n+\tb = container_of(entry_or_key, const struct moved_entry, ent)->es;\n \n-\treturn !xdiff_compare_lines(a->es->line, a->es->len,\n-\t\t\t\t b->es->line, b->es->len,\n-\t\t\t\t flags);\n+\treturn !xdiff_compare_lines(a->line + a->indent_off,\n+\t\t\t\t a->len - a->indent_off,\n+\t\t\t\t b->line + b->indent_off,\n+\t\t\t\t b->len - b->indent_off, flags);\n }\n \n static struct moved_entry *prepare_entry(struct diff_options *o,\n@@ -945,7 +923,8 @@ static struct moved_entry *prepare_entry(struct diff_options *o,\n \tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n \tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[line_no];\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n-\tunsigned int hash = xdiff_hash_string(l->line, l->len, flags);\n+\tunsigned int hash = xdiff_hash_string(l->line + l->indent_off,\n+\t\t\t\t\t l->len - l->indent_off, flags);\n \n \thashmap_entry_init(&ret->ent, hash);\n \tret->es = l;\n@@ -1113,14 +1092,11 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\thashmap_for_each_entry_from(hm, match, ent) {\n \t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n \t\t\t\tif (o->color_moved_ws_handling &\n-\t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n-\t\t\t\t\tif (compute_ws_delta(l, match->es,\n-\t\t\t\t\t\t\t &pmb[pmb_nr].wsd))\n-\t\t\t\t\t\tpmb[pmb_nr++].match = match;\n-\t\t\t\t} else {\n+\t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n+\t\t\t\t\tpmb[pmb_nr].wsd = compute_ws_delta(l, match->es);\n+\t\t\t\telse\n \t\t\t\t\tpmb[pmb_nr].wsd = 0;\n-\t\t\t\t\tpmb[pmb_nr++].match = match;\n-\t\t\t\t}\n+\t\t\t\tpmb[pmb_nr++].match = match;\n \t\t\t}\n \n \t\t\tif (adjust_last_block(o, n, block_length) &&\n@@ -6240,10 +6216,6 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \t\tif (o->color_moved) {\n \t\t\tstruct hashmap add_lines, del_lines;\n \n-\t\t\tif (o->color_moved_ws_handling &\n-\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\t\to->color_moved_ws_handling |= XDF_IGNORE_WHITESPACE;\n-\n \t\t\thashmap_init(&del_lines, moved_entry_cmp, o, 0);\n \t\t\thashmap_init(&add_lines, moved_entry_cmp, o, 0);\n \n-- \ngitgitgadget\n\n"},{"id":"430608","messageId":"753554587f9bbe22e9f32a245b551ab1f38ea1bd.1626777394.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v2.git.1626777393.gitgitgadget@gmail.com","subject":"[PATCH v2 12/12] diff --color-moved: intern strings","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-20T10:36:33Z","receivedAt":"2021-07-20T10:41:44Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nTaking inspiration from xdl_classify_record() assign an id to each\naddition and deletion such that lines that match for the current\n--color-moved-ws mode share the same unique id. This reduces the\nnumber of hash lookups a little (calculating the ids still involves\none hash lookup per line) but the main benefit is that when growing\nblocks of potentially moved lines we can replace string comparisons\nwhich involve chasing a pointer with a simple integer comparison.\n\nOn a large diff this commit reduces the time to run\n diff --color-moved\nby 33% and\n diff --color-moved-ws=allow-indentation-change\nby 26%. Compared to master the time to run\n diff --color-moved-ws=allow-indentation-change\nis now reduced by 95% and the overhead compared to --no-color-moved is\nreduced to 50%.\n\nCompared to the previous commit the time to run\n git log --patch --color-moved\nis increased slightly, but compared to master there is no change in\nrun time.\n\nTest HEAD^ HEAD\n--------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41(0.36+0.04) 0.41(0.37+0.03) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.83(0.79+0.03) 0.55(0.52+0.03) -33.7%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 0.81(0.77+0.04) 0.60(0.55+0.05) -25.9%\n4002.4: log --no-color-moved --no-color-moved-ws 1.30(1.20+0.09) 1.31(1.22+0.08) +0.8%\n4002.5: log --color-moved --no-color-moved-ws 1.46(1.35+0.11) 1.47(1.30+0.16) +0.7%\n4002.6: log --color-moved-ws=allow-indentation-change 1.46(1.38+0.07) 1.47(1.34+0.13) +0.7%\n\nTest master HEAD\n--------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.40( 0.36+0.03) 0.41(0.37+0.03) +2.5%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.82( 0.77+0.04) 0.55(0.52+0.03) -32.9%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 14.10(14.04+0.04) 0.60(0.55+0.05) -95.7%\n4002.4: log --no-color-moved --no-color-moved-ws 1.31( 1.21+0.09) 1.31(1.22+0.08) +0.0%\n4002.5: log --color-moved --no-color-moved-ws 1.47( 1.37+0.09) 1.47(1.30+0.16) +0.0%\n4002.6: log --color-moved-ws=allow-indentation-change 1.86( 1.76+0.10) 1.47(1.34+0.13) -21.0%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 171 ++++++++++++++++++++++++++++++++-------------------------\n 1 file changed, 95 insertions(+), 76 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 31a20a34240..2956c8f7103 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -18,6 +18,7 @@\n #include \"submodule-config.h\"\n #include \"submodule.h\"\n #include \"hashmap.h\"\n+#include \"mem-pool.h\"\n #include \"ll-merge.h\"\n #include \"string-list.h\"\n #include \"strvec.h\"\n@@ -772,6 +773,7 @@ struct emitted_diff_symbol {\n \tint flags;\n \tint indent_off; /* Offset to first non-whitespace character */\n \tint indent_width; /* The visual width of the indentation */\n+\tunsigned id;\n \tenum diff_symbol s;\n };\n #define EMITTED_DIFF_SYMBOL_INIT {NULL}\n@@ -797,9 +799,9 @@ static void append_emitted_diff_symbol(struct diff_options *o,\n }\n \n struct moved_entry {\n-\tstruct hashmap_entry ent;\n \tconst struct emitted_diff_symbol *es;\n \tstruct moved_entry *next_line;\n+\tstruct moved_entry *next_match;\n };\n \n struct moved_block {\n@@ -866,24 +868,24 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \t\t\t\t const struct emitted_diff_symbol *l,\n \t\t\t\t struct moved_block *pmb)\n {\n-\tint al = cur->es->len, bl = l->len;\n-\tconst char *a = cur->es->line,\n-\t\t *b = l->line;\n-\tint a_off = cur->es->indent_off,\n-\t a_width = cur->es->indent_width,\n-\t b_off = l->indent_off,\n-\t b_width = l->indent_width;\n+\tint a_width = cur->es->indent_width, b_width = l->indent_width;\n \tint delta;\n \n-\t/* If 'l' and 'cur' are both blank then they match. */\n-\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n+\t/* The text of each line must match */\n+\tif (cur->es->id != l->id)\n+\t\treturn 1;\n+\n+\t/*\n+\t * If 'l' and 'cur' are both blank then we don't need to check the\n+\t * indent. We only need to check cur as we know the strings match.\n+\t * */\n+\tif (a_width == INDENT_BLANKLINE)\n \t\treturn 0;\n \n \t/*\n \t * The indent changes of the block are known and stored in pmb->wsd;\n \t * however we need to check if the indent changes of the current line\n-\t * match those of the current block and that the text of 'l' and 'cur'\n-\t * after the indentation match.\n+\t * match those of the current block.\n \t */\n \tdelta = b_width - a_width;\n \n@@ -894,22 +896,26 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \tif (pmb->wsd == INDENT_BLANKLINE)\n \t\tpmb->wsd = delta;\n \n-\treturn !(delta == pmb->wsd && al - a_off == bl - b_off &&\n-\t\t !memcmp(a + a_off, b + b_off, al - a_off));\n+\treturn delta != pmb->wsd;\n }\n \n-static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n-\t\t\t const struct hashmap_entry *eptr,\n-\t\t\t const struct hashmap_entry *entry_or_key,\n-\t\t\t const void *keydata)\n+struct interned_diff_symbol {\n+\tstruct hashmap_entry ent;\n+\tstruct emitted_diff_symbol *es;\n+};\n+\n+static int interned_diff_symbol_cmp(const void *hashmap_cmp_fn_data,\n+\t\t\t\t const struct hashmap_entry *eptr,\n+\t\t\t\t const struct hashmap_entry *entry_or_key,\n+\t\t\t\t const void *keydata)\n {\n \tconst struct diff_options *diffopt = hashmap_cmp_fn_data;\n \tconst struct emitted_diff_symbol *a, *b;\n \tunsigned flags = diffopt->color_moved_ws_handling\n \t\t\t & XDF_WHITESPACE_FLAGS;\n \n-\ta = container_of(eptr, const struct moved_entry, ent)->es;\n-\tb = container_of(entry_or_key, const struct moved_entry, ent)->es;\n+\ta = container_of(eptr, const struct interned_diff_symbol, ent)->es;\n+\tb = container_of(entry_or_key, const struct interned_diff_symbol, ent)->es;\n \n \treturn !xdiff_compare_lines(a->line + a->indent_off,\n \t\t\t\t a->len - a->indent_off,\n@@ -917,55 +923,81 @@ static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n \t\t\t\t b->len - b->indent_off, flags);\n }\n \n-static struct moved_entry *prepare_entry(struct diff_options *o,\n-\t\t\t\t\t int line_no)\n+static void prepare_entry(struct diff_options *o, struct emitted_diff_symbol *l,\n+\t\t\t struct interned_diff_symbol *s)\n {\n-\tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n-\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[line_no];\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \tunsigned int hash = xdiff_hash_string(l->line + l->indent_off,\n \t\t\t\t\t l->len - l->indent_off, flags);\n \n-\thashmap_entry_init(&ret->ent, hash);\n-\tret->es = l;\n-\tret->next_line = NULL;\n-\n-\treturn ret;\n+\thashmap_entry_init(&s->ent, hash);\n+\ts->es = l;\n }\n \n-static void add_lines_to_move_detection(struct diff_options *o,\n-\t\t\t\t\tstruct hashmap *add_lines,\n-\t\t\t\t\tstruct hashmap *del_lines)\n+struct moved_entry_list {\n+\tstruct moved_entry *add, *del;\n+};\n+\n+static struct moved_entry_list *add_lines_to_move_detection(struct diff_options *o,\n+\t\t\t\t\t\t\t struct mem_pool *entry_mem_pool)\n {\n \tstruct moved_entry *prev_line = NULL;\n-\n+\tstruct mem_pool interned_pool;\n+\tstruct hashmap interned_map;\n+\tstruct moved_entry_list *entry_list = NULL;\n+\tsize_t entry_list_alloc = 0;\n+\tunsigned id = 0;\n \tint n;\n+\n+\thashmap_init(&interned_map, interned_diff_symbol_cmp, o, 8096);\n+\tmem_pool_init(&interned_pool, 1024 * 1024);\n+\n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n-\t\tstruct hashmap *hm;\n-\t\tstruct moved_entry *key;\n+\t\tstruct interned_diff_symbol key;\n+\t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n+\t\tstruct interned_diff_symbol *s;\n+\t\tstruct moved_entry *entry;\n \n-\t\tswitch (o->emitted_symbols->buf[n].s) {\n-\t\tcase DIFF_SYMBOL_PLUS:\n-\t\t\thm = add_lines;\n-\t\t\tbreak;\n-\t\tcase DIFF_SYMBOL_MINUS:\n-\t\t\thm = del_lines;\n-\t\t\tbreak;\n-\t\tdefault:\n+\t\tif (l->s != DIFF_SYMBOL_PLUS && l->s != DIFF_SYMBOL_MINUS) {\n \t\t\tprev_line = NULL;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (o->color_moved_ws_handling &\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\tfill_es_indent_data(&o->emitted_symbols->buf[n]);\n-\t\tkey = prepare_entry(o, n);\n-\t\tif (prev_line && prev_line->es->s == o->emitted_symbols->buf[n].s)\n-\t\t\tprev_line->next_line = key;\n+\t\t\tfill_es_indent_data(l);\n \n-\t\thashmap_add(hm, &key->ent);\n-\t\tprev_line = key;\n+\t\tprepare_entry(o, l, &key);\n+\t\ts = hashmap_get_entry(&interned_map, &key, ent, &key.ent);\n+\t\tif (s) {\n+\t\t\tl->id = s->es->id;\n+\t\t} else {\n+\t\t\tl->id = id;\n+\t\t\tALLOC_GROW_BY(entry_list, id, 1, entry_list_alloc);\n+\t\t\thashmap_add(&interned_map,\n+\t\t\t\t memcpy(mem_pool_alloc(&interned_pool,\n+\t\t\t\t\t\t\t sizeof(key)),\n+\t\t\t\t\t &key, sizeof(key)));\n+\t\t}\n+\t\tentry = mem_pool_alloc(entry_mem_pool, sizeof(*entry));\n+\t\tentry->es = l;\n+\t\tentry->next_line = NULL;\n+\t\tif (prev_line && prev_line->es->s == l->s)\n+\t\t\tprev_line->next_line = entry;\n+\t\tprev_line = entry;\n+\t\tif (l->s == DIFF_SYMBOL_PLUS) {\n+\t\t\tentry->next_match = entry_list[l->id].add;\n+\t\t\tentry_list[l->id].add = entry;\n+\t\t} else {\n+\t\t\tentry->next_match = entry_list[l->id].del;\n+\t\t\tentry_list[l->id].del = entry;\n+\t\t}\n \t}\n+\n+\thashmap_clear(&interned_map);\n+\tmem_pool_discard(&interned_pool, 0);\n+\n+\treturn entry_list;\n }\n \n static void pmb_advance_or_null(struct diff_options *o,\n@@ -974,7 +1006,6 @@ static void pmb_advance_or_null(struct diff_options *o,\n \t\t\t\tint *pmb_nr)\n {\n \tint i, j;\n-\tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \n \tfor (i = 0, j = 0; i < *pmb_nr; i++) {\n \t\tint match;\n@@ -987,9 +1018,8 @@ static void pmb_advance_or_null(struct diff_options *o,\n \t\t\tmatch = cur &&\n \t\t\t\t!cmp_in_block_with_wsd(o, cur, l, &pmb[i]);\n \t\telse\n-\t\t\tmatch = cur &&\n-\t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n-\t\t\t\t\t\t l->line, l->len, flags);\n+\t\t\tmatch = cur && cur->es->id == l->id;\n+\n \t\tif (match)\n \t\t\tpmb[j++].match = cur;\n \t}\n@@ -1034,8 +1064,7 @@ static int adjust_last_block(struct diff_options *o, int n, int block_length)\n \n /* Find blocks of moved code, delegate actual coloring decision to helper */\n static void mark_color_as_moved(struct diff_options *o,\n-\t\t\t\tstruct hashmap *add_lines,\n-\t\t\t\tstruct hashmap *del_lines)\n+\t\t\t\tstruct moved_entry_list *entry_list)\n {\n \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n@@ -1044,23 +1073,15 @@ static void mark_color_as_moved(struct diff_options *o,\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n-\t\tstruct hashmap *hm = NULL;\n-\t\tstruct moved_entry *key;\n \t\tstruct moved_entry *match = NULL;\n \t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n \n \t\tswitch (l->s) {\n \t\tcase DIFF_SYMBOL_PLUS:\n-\t\t\thm = del_lines;\n-\t\t\tkey = prepare_entry(o, n);\n-\t\t\tmatch = hashmap_get_entry(hm, key, ent, NULL);\n-\t\t\tfree(key);\n+\t\t\tmatch = entry_list[l->id].del;\n \t\t\tbreak;\n \t\tcase DIFF_SYMBOL_MINUS:\n-\t\t\thm = add_lines;\n-\t\t\tkey = prepare_entry(o, n);\n-\t\t\tmatch = hashmap_get_entry(hm, key, ent, NULL);\n-\t\t\tfree(key);\n+\t\t\tmatch = entry_list[l->id].add;\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tflipped_block = 0;\n@@ -1089,7 +1110,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t * The current line is the start of a new block.\n \t\t\t * Setup the set of potential blocks.\n \t\t\t */\n-\t\t\thashmap_for_each_entry_from(hm, match, ent) {\n+\t\t\tfor (; match; match = match->next_match) {\n \t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n \t\t\t\tif (o->color_moved_ws_handling &\n \t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n@@ -6216,20 +6237,18 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \n \tif (o->emitted_symbols) {\n \t\tif (o->color_moved) {\n-\t\t\tstruct hashmap add_lines, del_lines;\n-\n-\t\t\thashmap_init(&del_lines, moved_entry_cmp, o, 0);\n-\t\t\thashmap_init(&add_lines, moved_entry_cmp, o, 0);\n+\t\t\tstruct mem_pool entry_pool;\n+\t\t\tstruct moved_entry_list *entry_list;\n \n-\t\t\tadd_lines_to_move_detection(o, &add_lines, &del_lines);\n-\t\t\tmark_color_as_moved(o, &add_lines, &del_lines);\n+\t\t\tmem_pool_init(&entry_pool, 1024 * 1024);\n+\t\t\tentry_list = add_lines_to_move_detection(o,\n+\t\t\t\t\t\t\t\t &entry_pool);\n+\t\t\tmark_color_as_moved(o, entry_list);\n \t\t\tif (o->color_moved == COLOR_MOVED_ZEBRA_DIM)\n \t\t\t\tdim_moved_lines(o);\n \n-\t\t\thashmap_clear_and_free(&add_lines, struct moved_entry,\n-\t\t\t\t\t\tent);\n-\t\t\thashmap_clear_and_free(&del_lines, struct moved_entry,\n-\t\t\t\t\t\tent);\n+\t\t\tmem_pool_discard(&entry_pool, 0);\n+\t\t\tfree(entry_list);\n \t\t}\n \n \t\tfor (i = 0; i < esm.nr; i++)\n-- \ngitgitgadget\n"},{"id":"430623","messageId":"c9a14175-a20a-72fa-bd63-b047a14f9b2f@gmail.com","threadId":"55926","inReplyTo":"pull.981.v2.git.1626777393.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/12] diff --color-moved[-ws] speedups","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-07-20T13:38:07Z","receivedAt":"2021-07-20T13:39:40Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Sorry Elijah I forgot to add you to the CC list on GitGitGadget\n\nBest Wishes\n\nPhillip\n\nOn 20/07/2021 11:36, Phillip Wood via GitGitGadget wrote:\n> Thanks to Ævar and Elijah for their comments, I've reworded the commit\n> messages, addressed the enum initialization issue in patch 2 (now 3) and\n> added some perf tests.\n> \n> There are two new patches in this round. The first patch is new and adds the\n> perf tests suggested by Ævar, the penultimate patch is also new and coverts\n> the existing code to use a designated initializer.\n> \n> I've converted the benchmark results in the commit messages to use the new\n> tests, the percentage changes are broadly similar to the previous results\n> though I ended up running them on a different computer this time.\n> \n> V1 cover letter:\n> \n> The current implementation of diff --color-moved-ws=allow-indentation-change\n> is considerably slower that the implementation of diff --color-moved which\n> is in turn slower than a regular diff. This patch series starts with a\n> couple of bug fixes and then reworks the implementation of diff\n> --color-moved and diff --color-moved-ws=allow-indentation-change to speed\n> them up on large diffs. The time to run git diff --color-moved\n> --no-color-moved-ws v2.28.0 v2.29.0 is reduced by 33% and the time to run\n> git diff --color-moved --color-moved-ws=allow-indentation-change v2.28.0\n> v2.29.0 is reduced by 88%. There is a small slowdown for commit sized diffs\n> with --color-moved - the time to run git log -p --color-moved\n> --no-color-moved-ws --no-merges -n1000 v2.29.0 is increased by 2% on recent\n> processors. On older processors these patches reduce the running time in all\n> cases that I've tested. In general the larger the diff the larger the speed\n> up. As an extreme example the time to run diff --color-moved\n> --color-moved-ws=allow-indentation-change v2.25.0 v2.30.0 goes down from 8\n> minutes to 6 seconds.\n> \n> Phillip Wood (12):\n> diff --color-moved: add perf tests\n> diff --color-moved=zebra: fix alternate coloring\n> diff --color-moved: avoid false short line matches and bad zerba\n> coloring\n> diff: simplify allow-indentation-change delta calculation\n> diff --color-moved-ws=allow-indentation-change: simplify and optimize\n> diff --color-moved: call comparison function directly\n> diff --color-moved: unify moved block growth functions\n> diff --color-moved: shrink potential moved blocks as we go\n> diff --color-moved: stop clearing potential moved blocks\n> diff --color-moved-ws=allow-indentation-change: improve hash lookups\n> diff: use designated initializers for emitted_diff_symbol\n> diff --color-moved: intern strings\n> \n> diff.c | 377 ++++++++++++-------------------\n> t/perf/p4002-diff-color-moved.sh | 45 ++++\n> t/t4015-diff-whitespace.sh | 137 +++++++++++\n> 3 files changed, 323 insertions(+), 236 deletions(-)\n> create mode 100755 t/perf/p4002-diff-color-moved.sh\n> \n> \n> base-commit: 211eca0895794362184da2be2a2d812d070719d3\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-981%2Fphillipwood%2Fwip%2Fdiff-color-moved-tweaks-v2\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-981/phillipwood/wip/diff-color-moved-tweaks-v2\n> Pull-Request: https://github.com/gitgitgadget/git/pull/981\n> \n> Range-diff vs v1:\n> \n> -: ----------- > 1: 8fc8914a37b diff --color-moved: add perf tests\n> 1: 374dbebcbf2 ! 2: 9b4e4d2674a diff --color-moved=zerba: fix alternate coloring\n> @@ Metadata\n> Author: Phillip Wood \n> \n> ## Commit message ##\n> - diff --color-moved=zerba: fix alternate coloring\n> + diff --color-moved=zebra: fix alternate coloring\n> \n> b0a2ba4776 (\"diff --color-moved=zebra: be stricter with color\n> alternation\", 2018-11-23) sought to avoid using the alternate colors\n> 2: 3d02a0a91a0 ! 3: 5512145c70f diff --color-moved: avoid false short line matches and bad zerba coloring\n> @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n> \tint pmb_nr = 0, pmb_alloc = 0;\n> \tint n, flipped_block = 0, block_length = 0;\n> -\tenum diff_symbol last_symbol = 0;\n> -+\tenum diff_symbol moved_symbol = 0;\n> ++\tenum diff_symbol moved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n> \n> \n> \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n> @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n> -\t\t\tlast_symbol = l->s;\n> +\t\t}\n> +\t\tif (!match) {\n> -+\t\t\tmoved_symbol = 0;\n> ++\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n> \t\t\tcontinue;\n> \t\t}\n> \n> @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n> +\t\t\tif (pmb_nr)\n> +\t\t\t\tmoved_symbol = l->s;\n> +\t\t\telse\n> -+\t\t\t\tmoved_symbol = 0;\n> ++\t\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n> +\n> \t\t\tblock_length = 0;\n> \t\t}\n> 3: 30f0ed44768 = 4: 93fdef30d64 diff: simplify allow-indentation-change delta calculation\n> 4: ebb6eec1d92 ! 5: 6b7a8aed4ec diff --color-moved-ws=allow-indentation-change: simplify and optimize\n> @@ Commit message\n> comparison to filter out the non-matching lines. Fixing this reduces\n> time to run\n> git diff --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n> - by 88% and simplifies the code.\n> + by 93% compared to master and simplifies the code.\n> \n> - Before this change\n> - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n> - Time (mean ± σ): 9.978 s ± 0.042 s [User: 9.905 s, System: 0.057 s]\n> - Range (min … max): 9.917 s … 10.037 s 10 runs\n> -\n> - After this change\n> - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n> - Time (mean ± σ): 1.220 s ± 0.004 s [User: 1.160 s, System: 0.058 s]\n> - Range (min … max): 1.214 s … 1.226 s 10 runs\n> + Test HEAD^ HEAD\n> + ---------------------------------------------------------------------------------------------------------------\n> + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41( 0.38+0.03) 0.41(0.37+0.04) +0.0%\n> + 4002.2: diff --color-moved --no-color-moved-ws large change 0.83( 0.79+0.04) 0.82(0.79+0.02) -1.2%\n> + 4002.3: diff --color-moved-ws=allow-indentation-change large change 13.68(13.59+0.07) 0.92(0.89+0.03) -93.3%\n> + 4002.4: log --no-color-moved --no-color-moved-ws 1.31( 1.22+0.08) 1.31(1.21+0.10) +0.0%\n> + 4002.5: log --color-moved --no-color-moved-ws 1.47( 1.40+0.07) 1.47(1.36+0.10) +0.0%\n> + 4002.6: log --color-moved-ws=allow-indentation-change 1.87( 1.77+0.09) 1.50(1.41+0.09) -19.8%\n> \n> Signed-off-by: Phillip Wood \n> \n> 5: cec0c2d04d7 ! 6: cfbdd447eee diff --color-moved: call comparison function directly\n> @@ Metadata\n> ## Commit message ##\n> diff --color-moved: call comparison function directly\n> \n> - Calling xdiff_compare_lines() directly rather than using a function\n> - pointer from the hash map reduces the time very slightly but more\n> - importantly it will allow us to easily combine pmb_advance_or_null()\n> - and pmb_advance_or_null_multi_match() in the next commit.\n> + This change will allow us to easily combine pmb_advance_or_null() and\n> + pmb_advance_or_null_multi_match() in the next commit. Calling\n> + xdiff_compare_lines() directly rather than using a function pointer\n> + from the hash map has little effect on the run time.\n> \n> - Before this change\n> - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n> - Time (mean ± σ): 1.136 s ± 0.004 s [User: 1.079 s, System: 0.053 s]\n> - Range (min … max): 1.130 s … 1.141 s 10 runs\n> -\n> - After this change\n> - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n> - Time (mean ± σ): 1.118 s ± 0.003 s [User: 1.062 s, System: 0.053 s]\n> - Range (min … max): 1.114 s … 1.121 s 10 runs\n> + Test HEAD^ HEAD\n> + -------------------------------------------------------------------------------------------------------------\n> + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41(0.37+0.04) 0.41(0.39+0.02) +0.0%\n> + 4002.2: diff --color-moved --no-color-moved-ws large change 0.82(0.79+0.02) 0.83(0.79+0.03) +1.2%\n> + 4002.3: diff --color-moved-ws=allow-indentation-change large change 0.92(0.89+0.03) 0.91(0.85+0.05) -1.1%\n> + 4002.4: log --no-color-moved --no-color-moved-ws 1.31(1.21+0.10) 1.33(1.22+0.10) +1.5%\n> + 4002.5: log --color-moved --no-color-moved-ws 1.47(1.36+0.10) 1.47(1.39+0.08) +0.0%\n> + 4002.6: log --color-moved-ws=allow-indentation-change 1.50(1.41+0.09) 1.51(1.42+0.09) +0.7%\n> \n> Signed-off-by: Phillip Wood \n> \n> 6: 050cef0081d = 7: 73ce9b54e86 diff --color-moved: unify moved block growth functions\n> 7: 9390e9a66eb = 8: ef8ce0e6ebc diff --color-moved: shrink potential moved blocks as we go\n> 8: 1de99ac2bc3 = 9: 9d0a042eae1 diff --color-moved: stop clearing potential moved blocks\n> 9: 41cdedd6090 ! 10: dd365ad115f diff --color-moved-ws=allow-indentation-change: improve hash lookups\n> @@ Commit message\n> As libxdiff does not have a whitespace flag to ignore the indentation\n> the code for --color-moved-ws=allow-indentation-change uses\n> XDF_IGNORE_WHITESPACE and then filters out any hash lookups where\n> - there are non-indentation changes. This is filtering is inefficient as\n> + there are non-indentation changes. This filtering is inefficient as\n> we have to perform another string comparison.\n> \n> By using the offset data that we have already computed to skip the\n> indentation we can avoid using XDF_IGNORE_WHITESPACE and safely remove\n> - the extra checks which improves the performance by 14% and paves the\n> + the extra checks which improves the performance by 11% and paves the\n> way for the elimination of string comparisons in the next commit.\n> \n> - This change slightly increases the runtime of other --color-moved\n> + This change slightly increases the run time of other --color-moved\n> modes. This could be avoided by using different comparison functions\n> - for the different modes but after the changes in the next commit there\n> - is no measurable benefit.\n> + for the different modes but after the next two commits there is no\n> + measurable benefit in doing so.\n> \n> - Before this change\n> - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n> - Time (mean ± σ): 1.116 s ± 0.005 s [User: 1.057 s, System: 0.056 s]\n> - Range (min … max): 1.109 s … 1.123 s 10 runs\n> -\n> - Benchmark #2: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n> - Time (mean ± σ): 1.216 s ± 0.005 s [User: 1.155 s, System: 0.059 s]\n> - Range (min … max): 1.206 s … 1.223 s 10 runs\n> -\n> - After this change\n> - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n> - Time (mean ± σ): 1.147 s ± 0.005 s [User: 1.085 s, System: 0.059 s]\n> - Range (min … max): 1.140 s … 1.154 s 10 runs\n> -\n> - Benchmark #2: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n> - Time (mean ± σ): 1.048 s ± 0.005 s [User: 987.4 ms, System: 58.8 ms]\n> - Range (min … max): 1.043 s … 1.056 s 10 runs\n> + Test HEAD^ HEAD\n> + --------------------------------------------------------------------------------------------------------------\n> + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41(0.38+0.03) 0.41(0.36+0.04) +0.0%\n> + 4002.2: diff --color-moved --no-color-moved-ws large change 0.82(0.76+0.05) 0.84(0.79+0.04) +2.4%\n> + 4002.3: diff --color-moved-ws=allow-indentation-change large change 0.91(0.88+0.03) 0.81(0.74+0.06) -11.0%\n> + 4002.4: log --no-color-moved --no-color-moved-ws 1.32(1.21+0.10) 1.31(1.19+0.11) -0.8%\n> + 4002.5: log --color-moved --no-color-moved-ws 1.47(1.37+0.10) 1.47(1.36+0.11) +0.0%\n> + 4002.6: log --color-moved-ws=allow-indentation-change 1.51(1.42+0.09) 1.48(1.37+0.10) -2.0%\n> \n> Signed-off-by: Phillip Wood \n> \n> -: ----------- > 11: c160222ab3c diff: use designated initializers for emitted_diff_symbol\n> 10: 220664dd907 ! 12: 753554587f9 diff --color-moved: intern strings\n> @@ Commit message\n> number of hash lookups a little (calculating the ids still involves\n> one hash lookup per line) but the main benefit is that when growing\n> blocks of potentially moved lines we can replace string comparisons\n> - which involve chasing a pointer with a simple integer comparison. On\n> - a large diff this commit reduces the time to run 'diff --color-moved'\n> - by 33% and 'diff --color-moved-ws=allow-indentation-change' by 20%.\n> + which involve chasing a pointer with a simple integer comparison.\n> \n> - Compared to master the time to run 'git log --patch --color-moved' is\n> - increased by 2% and 'git log --patch\n> - --color-moved-ws=allow-indentation-change' in reduced by 14%. These\n> - timings were performed on an i5-7200U, on an i5-3470 both commands are\n> - faster than master. The small speed decrease on commit sized diffs is\n> - unfortunate but I think it is small enough to be worth it for the\n> - gains on larger diffs.\n> + On a large diff this commit reduces the time to run\n> + diff --color-moved\n> + by 33% and\n> + diff --color-moved-ws=allow-indentation-change\n> + by 26%. Compared to master the time to run\n> + diff --color-moved-ws=allow-indentation-change\n> + is now reduced by 95% and the overhead compared to --no-color-moved is\n> + reduced to 50%.\n> \n> - Large diff before this change:\n> - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n> - Time (mean ± σ): 1.147 s ± 0.005 s [User: 1.085 s, System: 0.059 s]\n> - Range (min … max): 1.140 s … 1.154 s 10 runs\n> + Compared to the previous commit the time to run\n> + git log --patch --color-moved\n> + is increased slightly, but compared to master there is no change in\n> + run time.\n> \n> - Benchmark #2: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n> - Time (mean ± σ): 1.048 s ± 0.005 s [User: 987.4 ms, System: 58.8 ms]\n> - Range (min … max): 1.043 s … 1.056 s 10 runs\n> + Test HEAD^ HEAD\n> + --------------------------------------------------------------------------------------------------------------\n> + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41(0.36+0.04) 0.41(0.37+0.03) +0.0%\n> + 4002.2: diff --color-moved --no-color-moved-ws large change 0.83(0.79+0.03) 0.55(0.52+0.03) -33.7%\n> + 4002.3: diff --color-moved-ws=allow-indentation-change large change 0.81(0.77+0.04) 0.60(0.55+0.05) -25.9%\n> + 4002.4: log --no-color-moved --no-color-moved-ws 1.30(1.20+0.09) 1.31(1.22+0.08) +0.8%\n> + 4002.5: log --color-moved --no-color-moved-ws 1.46(1.35+0.11) 1.47(1.30+0.16) +0.7%\n> + 4002.6: log --color-moved-ws=allow-indentation-change 1.46(1.38+0.07) 1.47(1.34+0.13) +0.7%\n> \n> - Large diff after this change\n> - Benchmark #1: bin-wrappers/git diff --diff-algorithm=myers --color-moved --no-color-moved-ws v2.28.0 v2.29.0\n> - Time (mean ± σ): 762.7 ms ± 2.8 ms [User: 707.5 ms, System: 53.7 ms]\n> - Range (min … max): 758.0 ms … 767.0 ms 10 runs\n> -\n> - Benchmark #2: bin-wrappers/git diff --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n> - Time (mean ± σ): 831.7 ms ± 1.7 ms [User: 776.5 ms, System: 53.3 ms]\n> - Range (min … max): 829.2 ms … 835.1 ms 10 runs\n> -\n> - Small diffs on master\n> - Benchmark #1: bin-wrappers/git log -p --diff-algorithm=myers --color-moved --no-color-moved-ws --no-merges -n1000 v2.29.0\n> - Time (mean ± σ): 1.567 s ± 0.001 s [User: 1.443 s, System: 0.121 s]\n> - Range (min … max): 1.566 s … 1.571 s 10 runs\n> -\n> - Benchmark #2: bin-wrappers/git log -p --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change -n1000 --no-merges v2.29.0\n> - Time (mean ± σ): 1.865 s ± 0.008 s [User: 1.748 s, System: 0.112 s]\n> - Range (min … max): 1.857 s … 1.881 s 10 runs\n> -\n> - Small diffs after this change\n> - Benchmark #1: bin-wrappers/git log -p --diff-algorithm=myers --color-moved --no-color-moved-ws --no-merges -n1000 v2.29.0\n> - Time (mean ± σ): 1.597 s ± 0.003 s [User: 1.413 s, System: 0.179 s]\n> - Range (min … max): 1.591 s … 1.601 s 10 runs\n> -\n> - Benchmark #2: bin-wrappers/git log -p --diff-algorithm=myers --color-moved --color-moved-ws=allow-indentation-change -n1000 --no-merges v2.29.0\n> - Time (mean ± σ): 1.606 s ± 0.006 s [User: 1.420 s, System: 0.181 s]\n> - Range (min … max): 1.601 s … 1.622 s 10 runs\n> + Test master HEAD\n> + --------------------------------------------------------------------------------------------------------------\n> + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.40( 0.36+0.03) 0.41(0.37+0.03) +2.5%\n> + 4002.2: diff --color-moved --no-color-moved-ws large change 0.82( 0.77+0.04) 0.55(0.52+0.03) -32.9%\n> + 4002.3: diff --color-moved-ws=allow-indentation-change large change 14.10(14.04+0.04) 0.60(0.55+0.05) -95.7%\n> + 4002.4: log --no-color-moved --no-color-moved-ws 1.31( 1.21+0.09) 1.31(1.22+0.08) +0.0%\n> + 4002.5: log --color-moved --no-color-moved-ws 1.47( 1.37+0.09) 1.47(1.30+0.16) +0.0%\n> + 4002.6: log --color-moved-ws=allow-indentation-change 1.86( 1.76+0.10) 1.47(1.34+0.13) -21.0%\n> \n> Signed-off-by: Phillip Wood \n> \n> @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n> \t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n> \t\t\t\tif (o->color_moved_ws_handling &\n> \t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n> -@@ diff.c: static void emit_diff_symbol_from_struct(struct diff_options *o,\n> - static void emit_diff_symbol(struct diff_options *o, enum diff_symbol s,\n> - \t\t\t const char *line, int len, unsigned flags)\n> - {\n> --\tstruct emitted_diff_symbol e = {line, len, flags, 0, 0, s};\n> -+\tstruct emitted_diff_symbol e = {line, len, flags, 0, 0, 0, s};\n> -\n> - \tif (o->emitted_symbols)\n> - \t\tappend_emitted_diff_symbol(o, &e);\n> @@ diff.c: static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n> \n> \tif (o->emitted_symbols) {\n> \n"},{"id":"439754","messageId":"8fc8914a37b3c343cd92bb0255088f7b000ff7f7.1635336262.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v3 01/15] diff --color-moved: add perf tests","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:08Z","receivedAt":"2021-10-27T12:04:28Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nAdd some tests so we can monitor changes to the performance of the\nmove detection code. The tests record the performance of a single\nlarge diff and a sequence of smaller diffs.\n\nSigned-off-by: Phillip Wood \n---\n t/perf/p4002-diff-color-moved.sh | 45 ++++++++++++++++++++++++++++++++\n 1 file changed, 45 insertions(+)\n create mode 100755 t/perf/p4002-diff-color-moved.sh\n\ndiff --git a/t/perf/p4002-diff-color-moved.sh b/t/perf/p4002-diff-color-moved.sh\nnew file mode 100755\nindex 00000000000..ad56bcb71e4\n--- /dev/null\n+++ b/t/perf/p4002-diff-color-moved.sh\n@@ -0,0 +1,45 @@\n+#!/bin/sh\n+\n+test_description='Tests diff --color-moved performance'\n+. ./perf-lib.sh\n+\n+test_perf_default_repo\n+\n+if ! git rev-parse --verify v2.29.0^{commit} >/dev/null\n+then\n+\tskip_all='skipping because tag v2.29.0 was not found'\n+\ttest_done\n+fi\n+\n+GIT_PAGER_IN_USE=1\n+test_export GIT_PAGER_IN_USE\n+\n+test_perf 'diff --no-color-moved --no-color-moved-ws large change' '\n+\tgit diff --no-color-moved --no-color-moved-ws v2.28.0 v2.29.0\n+'\n+\n+test_perf 'diff --color-moved --no-color-moved-ws large change' '\n+\tgit diff --color-moved=zebra --no-color-moved-ws v2.28.0 v2.29.0\n+'\n+\n+test_perf 'diff --color-moved-ws=allow-indentation-change large change' '\n+\tgit diff --color-moved=zebra --color-moved-ws=allow-indentation-change \\\n+\t\tv2.28.0 v2.29.0\n+'\n+\n+test_perf 'log --no-color-moved --no-color-moved-ws' '\n+\tgit log --no-color-moved --no-color-moved-ws --no-merges --patch \\\n+\t\t-n1000 v2.29.0\n+'\n+\n+test_perf 'log --color-moved --no-color-moved-ws' '\n+\tgit log --color-moved=zebra --no-color-moved-ws --no-merges --patch \\\n+\t\t-n1000 v2.29.0\n+'\n+\n+test_perf 'log --color-moved-ws=allow-indentation-change' '\n+\tgit log --color-moved=zebra --color-moved-ws=allow-indentation-change \\\n+\t\t--no-merges --patch -n1000 v2.29.0\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"439755","messageId":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v2.git.1626777393.gitgitgadget@gmail.com","subject":"[PATCH v3 00/15] diff --color-moved[-ws] speedups","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:07Z","receivedAt":"2021-10-27T12:04:28Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Sorry it has taken so long to post this re-roll. Changes since V2:\n\n * Patches 1-3 are new and fix an existing bug.\n * Patch 8 includes Peff's unused parameter fix.\n * Patch 11 has been updated to fix a bug fix in V2.\n * Patch 13 has an expanded commit message explaining a change in behavior\n for lines starting with a form-feed.\n * Updated benchmark results.\n\nThe bug fix in patch 3 degrades the performance, but by the end of the\nseries the timings are the same as V2 - see the range diff.\n\nV2 Cover Letter: Thanks to Ævar and Elijah for their comments, I've reworded\nthe commit messages, addressed the enum initialization issue in patch 2 (now\n3) and added some perf tests.\n\nThere are two new patches in this round. The first patch is new and adds the\nperf tests suggested by Ævar, the penultimate patch is also new and coverts\nthe existing code to use a designated initializer.\n\nI've converted the benchmark results in the commit messages to use the new\ntests, the percentage changes are broadly similar to the previous results\nthough I ended up running them on a different computer this time.\n\nV1 cover letter:\n\nThe current implementation of diff --color-moved-ws=allow-indentation-change\nis considerably slower that the implementation of diff --color-moved which\nis in turn slower than a regular diff. This patch series starts with a\ncouple of bug fixes and then reworks the implementation of diff\n--color-moved and diff --color-moved-ws=allow-indentation-change to speed\nthem up on large diffs. The time to run git diff --color-moved\n--no-color-moved-ws v2.28.0 v2.29.0 is reduced by 33% and the time to run\ngit diff --color-moved --color-moved-ws=allow-indentation-change v2.28.0\nv2.29.0 is reduced by 88%. There is a small slowdown for commit sized diffs\nwith --color-moved - the time to run git log -p --color-moved\n--no-color-moved-ws --no-merges -n1000 v2.29.0 is increased by 2% on recent\nprocessors. On older processors these patches reduce the running time in all\ncases that I've tested. In general the larger the diff the larger the speed\nup. As an extreme example the time to run diff --color-moved\n--color-moved-ws=allow-indentation-change v2.25.0 v2.30.0 goes down from 8\nminutes to 6 seconds.\n\nPhillip Wood (15):\n diff --color-moved: add perf tests\n diff --color-moved: clear all flags on blocks that are too short\n diff --color-moved: factor out function\n diff --color-moved: rewind when discarding pmb\n diff --color-moved=zebra: fix alternate coloring\n diff --color-moved: avoid false short line matches and bad zerba\n coloring\n diff: simplify allow-indentation-change delta calculation\n diff --color-moved-ws=allow-indentation-change: simplify and optimize\n diff --color-moved: call comparison function directly\n diff --color-moved: unify moved block growth functions\n diff --color-moved: shrink potential moved blocks as we go\n diff --color-moved: stop clearing potential moved blocks\n diff --color-moved-ws=allow-indentation-change: improve hash lookups\n diff: use designated initializers for emitted_diff_symbol\n diff --color-moved: intern strings\n\n diff.c | 431 +++++++++++++------------------\n t/perf/p4002-diff-color-moved.sh | 45 ++++\n t/t4015-diff-whitespace.sh | 205 ++++++++++++++-\n 3 files changed, 425 insertions(+), 256 deletions(-)\n create mode 100755 t/perf/p4002-diff-color-moved.sh\n\n\nbase-commit: 211eca0895794362184da2be2a2d812d070719d3\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-981%2Fphillipwood%2Fwip%2Fdiff-color-moved-tweaks-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-981/phillipwood/wip/diff-color-moved-tweaks-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/981\n\nRange-diff vs v2:\n\n 1: 8fc8914a37b = 1: 8fc8914a37b diff --color-moved: add perf tests\n -: ----------- > 2: e9daed2360c diff --color-moved: clear all flags on blocks that are too short\n -: ----------- > 3: 658aec2670c diff --color-moved: factor out function\n -: ----------- > 4: a30f52d7f15 diff --color-moved: rewind when discarding pmb\n 2: 9b4e4d2674a ! 5: 1dde206b7b1 diff --color-moved=zebra: fix alternate coloring\n @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n \t\tswitch (l->s) {\n \t\tcase DIFF_SYMBOL_PLUS:\n @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n - \t\t\t}\n + \t\t\t\t\t\t\t &pmb, &pmb_alloc,\n + \t\t\t\t\t\t\t &pmb_nr);\n \n - \t\t\tif (adjust_last_block(o, n, block_length) &&\n --\t\t\t pmb_nr && last_symbol != l->s)\n -+\t\t\t pmb_nr && last_symbol == l->s)\n +-\t\t\tif (contiguous && pmb_nr && last_symbol != l->s)\n ++\t\t\tif (contiguous && pmb_nr && last_symbol == l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n \t\t\telse\n \t\t\t\tflipped_block = 0;\n 3: 5512145c70f ! 6: 2717ff500d2 diff --color-moved: avoid false short line matches and bad zerba coloring\n @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n +\t\tif (pmb_nr && (!match || l->s != moved_symbol)) {\n \t\t\tint i;\n \n - \t\t\tadjust_last_block(o, n, block_length);\n + \t\t\tif (!adjust_last_block(o, n, block_length) &&\n @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n \t\t\tpmb_nr = 0;\n \t\t\tblock_length = 0;\n @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n \t\t\tcontinue;\n \t\t}\n @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n - \t\t\t}\n + \t\t\t\t\t\t\t &pmb, &pmb_alloc,\n + \t\t\t\t\t\t\t &pmb_nr);\n \n - \t\t\tif (adjust_last_block(o, n, block_length) &&\n --\t\t\t pmb_nr && last_symbol == l->s)\n -+\t\t\t pmb_nr && moved_symbol == l->s)\n +-\t\t\tif (contiguous && pmb_nr && last_symbol == l->s)\n ++\t\t\tif (contiguous && pmb_nr && moved_symbol == l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n \t\t\telse\n \t\t\t\tflipped_block = 0;\n 4: 93fdef30d64 = 7: f96fa71d53c diff: simplify allow-indentation-change delta calculation\n 5: 6b7a8aed4ec ! 8: 324b689c915 diff --color-moved-ws=allow-indentation-change: simplify and optimize\n @@ Commit message\n git diff --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n by 93% compared to master and simplifies the code.\n \n - Test HEAD^ HEAD\n + Test HEAD^ HEAD\n ---------------------------------------------------------------------------------------------------------------\n - 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41( 0.38+0.03) 0.41(0.37+0.04) +0.0%\n - 4002.2: diff --color-moved --no-color-moved-ws large change 0.83( 0.79+0.04) 0.82(0.79+0.02) -1.2%\n - 4002.3: diff --color-moved-ws=allow-indentation-change large change 13.68(13.59+0.07) 0.92(0.89+0.03) -93.3%\n - 4002.4: log --no-color-moved --no-color-moved-ws 1.31( 1.22+0.08) 1.31(1.21+0.10) +0.0%\n - 4002.5: log --color-moved --no-color-moved-ws 1.47( 1.40+0.07) 1.47(1.36+0.10) +0.0%\n - 4002.6: log --color-moved-ws=allow-indentation-change 1.87( 1.77+0.09) 1.50(1.41+0.09) -19.8%\n + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.35+0.03) 0.38(0.35+0.03) +0.0%\n + 4002.2: diff --color-moved --no-color-moved-ws large change 0.86 (0.80+0.06) 0.87(0.83+0.04) +1.2%\n + 4002.3: diff --color-moved-ws=allow-indentation-change large change 19.01(18.93+0.06) 0.97(0.92+0.04) -94.9%\n + 4002.4: log --no-color-moved --no-color-moved-ws 1.16 (1.06+0.09) 1.17(1.06+0.10) +0.9%\n + 4002.5: log --color-moved --no-color-moved-ws 1.32 (1.25+0.07) 1.32(1.24+0.08) +0.0%\n + 4002.6: log --color-moved-ws=allow-indentation-change 1.71 (1.64+0.06) 1.36(1.25+0.10) -20.5%\n \n + Test master HEAD\n + ---------------------------------------------------------------------------------------------------------------\n + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.33+0.05) 0.38(0.35+0.03) +0.0%\n + 4002.2: diff --color-moved --no-color-moved-ws large change 0.80 (0.75+0.04) 0.87(0.83+0.04) +8.7%\n + 4002.3: diff --color-moved-ws=allow-indentation-change large change 14.20(14.15+0.05) 0.97(0.92+0.04) -93.2%\n + 4002.4: log --no-color-moved --no-color-moved-ws 1.15 (1.05+0.09) 1.17(1.06+0.10) +1.7%\n + 4002.5: log --color-moved --no-color-moved-ws 1.30 (1.19+0.11) 1.32(1.24+0.08) +1.5%\n + 4002.6: log --color-moved-ws=allow-indentation-change 1.70 (1.63+0.06) 1.36(1.25+0.10) -20.0%\n +\n + Helped-by: Jeff King \n Signed-off-by: Phillip Wood \n \n ## diff.c ##\n @@ diff.c: static int compute_ws_delta(const struct emitted_diff_symbol *a,\n + \treturn 1;\n + }\n \n - static int cmp_in_block_with_wsd(const struct diff_options *o,\n - \t\t\t\t const struct moved_entry *cur,\n +-static int cmp_in_block_with_wsd(const struct diff_options *o,\n +-\t\t\t\t const struct moved_entry *cur,\n -\t\t\t\t const struct moved_entry *match,\n -\t\t\t\t struct moved_block *pmb,\n -\t\t\t\t int n)\n -+\t\t\t\t const struct emitted_diff_symbol *l,\n -+\t\t\t\t struct moved_block *pmb)\n - {\n +-{\n -\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n -\tint al = cur->es->len, bl = match->es->len, cl = l->len;\n ++static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n ++\t\t\t\t const struct emitted_diff_symbol *l,\n ++\t\t\t\t struct moved_block *pmb)\n ++{\n +\tint al = cur->es->len, bl = l->len;\n \tconst char *a = cur->es->line,\n -\t\t *b = match->es->line,\n @@ diff.c: static void pmb_advance_or_null(struct diff_options *o,\n +\t\tstruct moved_entry *prev = pmb[i].match;\n +\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n +\t\t\tprev->next_line : NULL;\n -+\t\tif (cur && !cmp_in_block_with_wsd(o, cur, l, &pmb[i])) {\n ++\t\tif (cur && !cmp_in_block_with_wsd(cur, l, &pmb[i])) {\n \t\t\t/* Advance to the next line */\n -\t\t\tpmb[i].match = pmb[i].match->next_line;\n +\t\t\tpmb[i].match = cur;\n 6: cfbdd447eee ! 9: f142f33276a diff --color-moved: call comparison function directly\n @@ Commit message\n \n Test HEAD^ HEAD\n -------------------------------------------------------------------------------------------------------------\n - 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41(0.37+0.04) 0.41(0.39+0.02) +0.0%\n - 4002.2: diff --color-moved --no-color-moved-ws large change 0.82(0.79+0.02) 0.83(0.79+0.03) +1.2%\n - 4002.3: diff --color-moved-ws=allow-indentation-change large change 0.92(0.89+0.03) 0.91(0.85+0.05) -1.1%\n - 4002.4: log --no-color-moved --no-color-moved-ws 1.31(1.21+0.10) 1.33(1.22+0.10) +1.5%\n - 4002.5: log --color-moved --no-color-moved-ws 1.47(1.36+0.10) 1.47(1.39+0.08) +0.0%\n - 4002.6: log --color-moved-ws=allow-indentation-change 1.50(1.41+0.09) 1.51(1.42+0.09) +0.7%\n + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38(0.35+0.03) 0.38(0.32+0.06) +0.0%\n + 4002.2: diff --color-moved --no-color-moved-ws large change 0.87(0.83+0.04) 0.87(0.80+0.06) +0.0%\n + 4002.3: diff --color-moved-ws=allow-indentation-change large change 0.97(0.92+0.04) 0.97(0.93+0.04) +0.0%\n + 4002.4: log --no-color-moved --no-color-moved-ws 1.17(1.06+0.10) 1.16(1.10+0.05) -0.9%\n + 4002.5: log --color-moved --no-color-moved-ws 1.32(1.24+0.08) 1.31(1.22+0.09) -0.8%\n + 4002.6: log --color-moved-ws=allow-indentation-change 1.36(1.25+0.10) 1.35(1.25+0.10) -0.7%\n \n Signed-off-by: Phillip Wood \n \n 7: 73ce9b54e86 ! 10: 8f3ea865dd3 diff --color-moved: unify moved block growth functions\n @@ diff.c: static void pmb_advance_or_null(struct diff_options *o,\n -\t\tstruct moved_entry *prev = pmb[i].match;\n -\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n -\t\t\tprev->next_line : NULL;\n --\t\tif (cur && !cmp_in_block_with_wsd(o, cur, l, &pmb[i])) {\n +-\t\tif (cur && !cmp_in_block_with_wsd(cur, l, &pmb[i])) {\n -\t\t\t/* Advance to the next line */\n +\t\tif (o->color_moved_ws_handling &\n +\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n +\t\t\tmatch = cur &&\n -+\t\t\t\t!cmp_in_block_with_wsd(o, cur, l, &pmb[i]);\n ++\t\t\t\t!cmp_in_block_with_wsd(cur, l, &pmb[i]);\n +\t\telse\n +\t\t\tmatch = cur &&\n +\t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n 8: ef8ce0e6ebc ! 11: 078c04d4a66 diff --color-moved: shrink potential moved blocks as we go\n @@ diff.c: static void add_lines_to_move_detection(struct diff_options *o,\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n @@ diff.c: static void pmb_advance_or_null(struct diff_options *o,\n + \t\t\tmatch = cur &&\n \t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n \t\t\t\t\t\t l->line, l->len, flags);\n - \t\tif (match)\n +-\t\tif (match)\n -\t\t\tpmb[i].match = cur;\n -\t\telse\n -\t\t\tmoved_block_clear(&pmb[i]);\n -+\t\t\tpmb[j++].match = cur;\n - \t}\n +-\t}\n -}\n -\n -static int shrink_potential_moved_blocks(struct moved_block *pmb,\n @@ diff.c: static void pmb_advance_or_null(struct diff_options *o,\n -\t\t\tmemset(&pmb[rp], 0, sizeof(pmb[rp]));\n -\t\t\trp--;\n -\t\t\tlp++;\n --\t\t}\n --\t}\n ++\t\tif (match) {\n ++\t\t\tpmb[j] = pmb[i];\n ++\t\t\tpmb[j++].match = cur;\n + \t\t}\n + \t}\n -\n -\t/* Remember the number of running sets */\n -\treturn rp + 1;\n +\t*pmb_nr = j;\n }\n \n - /*\n + static void fill_potential_moved_blocks(struct diff_options *o,\n @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n \t\t\tcontinue;\n \t\t}\n @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n +\t\tpmb_advance_or_null(o, l, pmb, &pmb_nr);\n \n \t\tif (pmb_nr == 0) {\n - \t\t\t/*\n + \t\t\tint contiguous = adjust_last_block(o, n, block_length);\n 9: 9d0a042eae1 ! 12: 618371471a0 diff --color-moved: stop clearing potential moved blocks\n @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n \t\tif (pmb_nr && (!match || l->s != moved_symbol)) {\n -\t\t\tint i;\n -\n - \t\t\tadjust_last_block(o, n, block_length);\n + \t\t\tif (!adjust_last_block(o, n, block_length) &&\n + \t\t\t block_length > 1) {\n + \t\t\t\t/*\n +@@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n + \t\t\t\tmatch = NULL;\n + \t\t\t\tn -= block_length;\n + \t\t\t}\n -\t\t\tfor(i = 0; i < pmb_nr; i++)\n -\t\t\t\tmoved_block_clear(&pmb[i]);\n \t\t\tpmb_nr = 0;\n 10: dd365ad115f ! 13: 6a8e9a2724d diff --color-moved-ws=allow-indentation-change: improve hash lookups\n @@ Commit message\n for the different modes but after the next two commits there is no\n measurable benefit in doing so.\n \n + There is a change in behavior for lines that begin with a form-feed or\n + vertical-tab character. Since b46054b374 (\"xdiff: use\n + git-compat-util\", 2019-04-11) xdiff does not treat '\\f' or '\\v' as\n + whitespace characters. This means that lines starting with those\n + characters are never considered to be blank and never match a line\n + that does not start with the same character. After this patch a line\n + matching \"^[\\f\\v\\r]*[ \\t]*$\" is considered to be blank by\n + --color-moved-ws=allow-indentation-change and lines beginning\n + \"^[\\f\\v\\r]*[ \\t]*\" can match another line if the suffixes match. This\n + changes the output of git show for d18f76dccf (\"compat/regex: use the\n + regex engine from gawk for compat\", 2010-08-17) as some lines in the\n + pre-image before a moved block that contain '\\f' are now considered\n + moved as well as they match a blank line before the moved lines in the\n + post-image. This commit updates one of the tests to reflect this\n + change.\n +\n Test HEAD^ HEAD\n --------------------------------------------------------------------------------------------------------------\n - 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41(0.38+0.03) 0.41(0.36+0.04) +0.0%\n - 4002.2: diff --color-moved --no-color-moved-ws large change 0.82(0.76+0.05) 0.84(0.79+0.04) +2.4%\n - 4002.3: diff --color-moved-ws=allow-indentation-change large change 0.91(0.88+0.03) 0.81(0.74+0.06) -11.0%\n - 4002.4: log --no-color-moved --no-color-moved-ws 1.32(1.21+0.10) 1.31(1.19+0.11) -0.8%\n - 4002.5: log --color-moved --no-color-moved-ws 1.47(1.37+0.10) 1.47(1.36+0.11) +0.0%\n - 4002.6: log --color-moved-ws=allow-indentation-change 1.51(1.42+0.09) 1.48(1.37+0.10) -2.0%\n + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38(0.33+0.05) 0.38(0.33+0.05) +0.0%\n + 4002.2: diff --color-moved --no-color-moved-ws large change 0.86(0.82+0.04) 0.88(0.84+0.04) +2.3%\n + 4002.3: diff --color-moved-ws=allow-indentation-change large change 0.97(0.94+0.03) 0.86(0.81+0.05) -11.3%\n + 4002.4: log --no-color-moved --no-color-moved-ws 1.16(1.07+0.09) 1.16(1.06+0.09) +0.0%\n + 4002.5: log --color-moved --no-color-moved-ws 1.32(1.26+0.06) 1.33(1.27+0.05) +0.8%\n + 4002.6: log --color-moved-ws=allow-indentation-change 1.35(1.29+0.06) 1.33(1.24+0.08) -1.5%\n \n Signed-off-by: Phillip Wood \n \n @@ diff.c: static void fill_es_indent_data(struct emitted_diff_symbol *es)\n +\treturn a_width - b_width;\n }\n \n - static int cmp_in_block_with_wsd(const struct diff_options *o,\n + static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n @@ diff.c: static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n \t\t\t const void *keydata)\n {\n @@ diff.c: static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n \n -\ta = container_of(eptr, const struct moved_entry, ent);\n -\tb = container_of(entry_or_key, const struct moved_entry, ent);\n --\n ++\ta = container_of(eptr, const struct moved_entry, ent)->es;\n ++\tb = container_of(entry_or_key, const struct moved_entry, ent)->es;\n + \n -\tif (diffopt->color_moved_ws_handling &\n -\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n -\t\t/*\n @@ diff.c: static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n -\t\t * configuration for the next block is done else where\n -\t\t */\n -\t\tflags |= XDF_IGNORE_WHITESPACE;\n -+\ta = container_of(eptr, const struct moved_entry, ent)->es;\n -+\tb = container_of(entry_or_key, const struct moved_entry, ent)->es;\n - \n +-\n -\treturn !xdiff_compare_lines(a->es->line, a->es->len,\n -\t\t\t\t b->es->line, b->es->len,\n -\t\t\t\t flags);\n @@ diff.c: static struct moved_entry *prepare_entry(struct diff_options *o,\n \n \thashmap_entry_init(&ret->ent, hash);\n \tret->es = l;\n -@@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n - \t\t\thashmap_for_each_entry_from(hm, match, ent) {\n - \t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n - \t\t\t\tif (o->color_moved_ws_handling &\n --\t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n --\t\t\t\t\tif (compute_ws_delta(l, match->es,\n --\t\t\t\t\t\t\t &pmb[pmb_nr].wsd))\n --\t\t\t\t\t\tpmb[pmb_nr++].match = match;\n --\t\t\t\t} else {\n -+\t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n -+\t\t\t\t\tpmb[pmb_nr].wsd = compute_ws_delta(l, match->es);\n -+\t\t\t\telse\n - \t\t\t\t\tpmb[pmb_nr].wsd = 0;\n --\t\t\t\t\tpmb[pmb_nr++].match = match;\n --\t\t\t\t}\n -+\t\t\t\tpmb[pmb_nr++].match = match;\n - \t\t\t}\n +@@ diff.c: static void fill_potential_moved_blocks(struct diff_options *o,\n + \thashmap_for_each_entry_from(hm, match, ent) {\n + \t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n + \t\tif (o->color_moved_ws_handling &\n +-\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n +-\t\t\tif (compute_ws_delta(l, match->es, &(pmb[pmb_nr]).wsd))\n +-\t\t\t\tpmb[pmb_nr++].match = match;\n +-\t\t} else {\n ++\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n ++\t\t\tpmb[pmb_nr].wsd = compute_ws_delta(l, match->es);\n ++\t\telse\n + \t\t\tpmb[pmb_nr].wsd = 0;\n +-\t\t\tpmb[pmb_nr++].match = match;\n +-\t\t}\n ++\t\tpmb[pmb_nr++].match = match;\n + \t}\n \n - \t\t\tif (adjust_last_block(o, n, block_length) &&\n + \t*pmb_p = pmb;\n @@ diff.c: static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \t\tif (o->color_moved) {\n \t\t\tstruct hashmap add_lines, del_lines;\n @@ diff.c: static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \t\t\thashmap_init(&del_lines, moved_entry_cmp, o, 0);\n \t\t\thashmap_init(&add_lines, moved_entry_cmp, o, 0);\n \n +\n + ## t/t4015-diff-whitespace.sh ##\n +@@ t/t4015-diff-whitespace.sh: EMPTY=''\n + test_expect_success 'compare mixed whitespace delta across moved blocks' '\n + \n + \tgit reset --hard &&\n +-\ttr Q_ \"\\t \" <<-EOF >text.txt &&\n +-\t${EMPTY}\n +-\t____too short without\n +-\t${EMPTY}\n ++\ttr \"^|Q_\" \"\\f\\v\\t \" <<-EOF >text.txt &&\n ++\t^__\n ++\t|____too short without\n ++\t^\n + \t___being grouped across blank line\n + \t${EMPTY}\n + \tcontext\n +@@ t/t4015-diff-whitespace.sh: test_expect_success 'compare mixed whitespace delta across moved blocks' '\n + \tgit add text.txt &&\n + \tgit commit -m \"add text.txt\" &&\n + \n +-\ttr Q_ \"\\t \" <<-EOF >text.txt &&\n ++\ttr \"^|Q_\" \"\\f\\v\\t \" <<-EOF >text.txt &&\n + \tcontext\n + \tlines\n + \tto\n +@@ t/t4015-diff-whitespace.sh: test_expect_success 'compare mixed whitespace delta across moved blocks' '\n + \t${EMPTY}\n + \tQQtoo short without\n + \t${EMPTY}\n +-\tQ_______being grouped across blank line\n ++\t^Q_______being grouped across blank line\n + \t${EMPTY}\n + \tQ_QThese two lines have had their\n + \tindentation reduced by four spaces\n +@@ t/t4015-diff-whitespace.sh: test_expect_success 'compare mixed whitespace delta across moved blocks' '\n + \t\t-c core.whitespace=space-before-tab \\\n + \t\tdiff --color --color-moved --ws-error-highlight=all \\\n + \t\t--color-moved-ws=allow-indentation-change >actual.raw &&\n +-\tgrep -v \"index\" actual.raw | test_decode_color >actual &&\n ++\tgrep -v \"index\" actual.raw | tr \"\\f\\v\" \"^|\" | test_decode_color >actual &&\n + \n + \tcat <<-\\EOF >expected &&\n + \tdiff --git a/text.txt b/text.txt\n + \t--- a/text.txt\n + \t+++ b/text.txt\n + \t@@ -1,16 +1,16 @@\n +-\t-\n +-\t- too short without\n +-\t-\n ++\t-^ \n ++\t-| too short without\n ++\t-^\n + \t- being grouped across blank line\n + \t-\n + \t context\n +@@ t/t4015-diff-whitespace.sh: test_expect_success 'compare mixed whitespace delta across moved blocks' '\n + \t+\n + \t+\t\ttoo short without\n + \t+\n +-\t+\t being grouped across blank line\n ++\t+^\t being grouped across blank line\n + \t+\n + \t+\t \tThese two lines have had their\n + \t+indentation reduced by four spaces\n 11: c160222ab3c = 14: ef98a6e7015 diff: use designated initializers for emitted_diff_symbol\n 12: 753554587f9 ! 15: ae78c05f08d diff --color-moved: intern strings\n @@ Commit message\n number of hash lookups a little (calculating the ids still involves\n one hash lookup per line) but the main benefit is that when growing\n blocks of potentially moved lines we can replace string comparisons\n - which involve chasing a pointer with a simple integer comparison.\n + which involve chasing a pointer with a simple integer comparison. On a\n + large diff this commit reduces the time to run 'diff --color-moved' by\n + 37% compared to the previous commit and 31% compared to master, for\n + 'diff --color-moved-ws=allow-indentation-change' the reduction is 28%\n + compared to the previous commit and 96% compared to master. There is\n + little change in the performance of 'git log --patch' as the diffs are\n + smaller.\n \n - On a large diff this commit reduces the time to run\n - diff --color-moved\n - by 33% and\n - diff --color-moved-ws=allow-indentation-change\n - by 26%. Compared to master the time to run\n - diff --color-moved-ws=allow-indentation-change\n - is now reduced by 95% and the overhead compared to --no-color-moved is\n - reduced to 50%.\n + Test HEAD^ HEAD\n + ---------------------------------------------------------------------------------------------------------------\n + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38(0.33+0.05) 0.38(0.33+0.05) +0.0%\n + 4002.2: diff --color-moved --no-color-moved-ws large change 0.88(0.81+0.06) 0.55(0.50+0.04) -37.5%\n + 4002.3: diff --color-moved-ws=allow-indentation-change large change 0.85(0.79+0.06) 0.61(0.54+0.06) -28.2%\n + 4002.4: log --no-color-moved --no-color-moved-ws 1.16(1.07+0.08) 1.15(1.09+0.05) -0.9%\n + 4002.5: log --color-moved --no-color-moved-ws 1.31(1.22+0.08) 1.29(1.19+0.09) -1.5%\n + 4002.6: log --color-moved-ws=allow-indentation-change 1.32(1.24+0.08) 1.31(1.18+0.13) -0.8%\n \n - Compared to the previous commit the time to run\n - git log --patch --color-moved\n - is increased slightly, but compared to master there is no change in\n - run time.\n -\n - Test HEAD^ HEAD\n - --------------------------------------------------------------------------------------------------------------\n - 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.41(0.36+0.04) 0.41(0.37+0.03) +0.0%\n - 4002.2: diff --color-moved --no-color-moved-ws large change 0.83(0.79+0.03) 0.55(0.52+0.03) -33.7%\n - 4002.3: diff --color-moved-ws=allow-indentation-change large change 0.81(0.77+0.04) 0.60(0.55+0.05) -25.9%\n - 4002.4: log --no-color-moved --no-color-moved-ws 1.30(1.20+0.09) 1.31(1.22+0.08) +0.8%\n - 4002.5: log --color-moved --no-color-moved-ws 1.46(1.35+0.11) 1.47(1.30+0.16) +0.7%\n - 4002.6: log --color-moved-ws=allow-indentation-change 1.46(1.38+0.07) 1.47(1.34+0.13) +0.7%\n -\n - Test master HEAD\n - --------------------------------------------------------------------------------------------------------------\n - 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.40( 0.36+0.03) 0.41(0.37+0.03) +2.5%\n - 4002.2: diff --color-moved --no-color-moved-ws large change 0.82( 0.77+0.04) 0.55(0.52+0.03) -32.9%\n - 4002.3: diff --color-moved-ws=allow-indentation-change large change 14.10(14.04+0.04) 0.60(0.55+0.05) -95.7%\n - 4002.4: log --no-color-moved --no-color-moved-ws 1.31( 1.21+0.09) 1.31(1.22+0.08) +0.0%\n - 4002.5: log --color-moved --no-color-moved-ws 1.47( 1.37+0.09) 1.47(1.30+0.16) +0.0%\n - 4002.6: log --color-moved-ws=allow-indentation-change 1.86( 1.76+0.10) 1.47(1.34+0.13) -21.0%\n + Test master HEAD\n + ---------------------------------------------------------------------------------------------------------------\n + 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.33+0.05) 0.38(0.33+0.05) +0.0%\n + 4002.2: diff --color-moved --no-color-moved-ws large change 0.80 (0.75+0.04) 0.55(0.50+0.04) -31.2%\n + 4002.3: diff --color-moved-ws=allow-indentation-change large change 14.20(14.15+0.05) 0.61(0.54+0.06) -95.7%\n + 4002.4: log --no-color-moved --no-color-moved-ws 1.15 (1.05+0.09) 1.15(1.09+0.05) +0.0%\n + 4002.5: log --color-moved --no-color-moved-ws 1.30 (1.19+0.11) 1.29(1.19+0.09) -0.8%\n + 4002.6: log --color-moved-ws=allow-indentation-change 1.70 (1.63+0.06) 1.31(1.18+0.13) -22.9%\n \n Signed-off-by: Phillip Wood \n \n @@ diff.c: static void append_emitted_diff_symbol(struct diff_options *o,\n };\n \n struct moved_block {\n -@@ diff.c: static int cmp_in_block_with_wsd(const struct diff_options *o,\n +@@ diff.c: static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n \t\t\t\t const struct emitted_diff_symbol *l,\n \t\t\t\t struct moved_block *pmb)\n {\n @@ diff.c: static int cmp_in_block_with_wsd(const struct diff_options *o,\n \t */\n \tdelta = b_width - a_width;\n \n -@@ diff.c: static int cmp_in_block_with_wsd(const struct diff_options *o,\n +@@ diff.c: static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n \tif (pmb->wsd == INDENT_BLANKLINE)\n \t\tpmb->wsd = delta;\n \n @@ diff.c: static void pmb_advance_or_null(struct diff_options *o,\n \t\tint match;\n @@ diff.c: static void pmb_advance_or_null(struct diff_options *o,\n \t\t\tmatch = cur &&\n - \t\t\t\t!cmp_in_block_with_wsd(o, cur, l, &pmb[i]);\n + \t\t\t\t!cmp_in_block_with_wsd(cur, l, &pmb[i]);\n \t\telse\n -\t\t\tmatch = cur &&\n -\t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n -\t\t\t\t\t\t l->line, l->len, flags);\n +\t\t\tmatch = cur && cur->es->id == l->id;\n +\n - \t\tif (match)\n + \t\tif (match) {\n + \t\t\tpmb[j] = pmb[i];\n \t\t\tpmb[j++].match = cur;\n - \t}\n +@@ diff.c: static void pmb_advance_or_null(struct diff_options *o,\n + }\n + \n + static void fill_potential_moved_blocks(struct diff_options *o,\n +-\t\t\t\t\tstruct hashmap *hm,\n + \t\t\t\t\tstruct moved_entry *match,\n + \t\t\t\t\tstruct emitted_diff_symbol *l,\n + \t\t\t\t\tstruct moved_block **pmb_p,\n +@@ diff.c: static void fill_potential_moved_blocks(struct diff_options *o,\n + \t * The current line is the start of a new block.\n + \t * Setup the set of potential blocks.\n + \t */\n +-\thashmap_for_each_entry_from(hm, match, ent) {\n ++\tfor (; match; match = match->next_match) {\n + \t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n + \t\tif (o->color_moved_ws_handling &\n + \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n @@ diff.c: static int adjust_last_block(struct diff_options *o, int n, int block_length)\n \n /* Find blocks of moved code, delegate actual coloring decision to helper */\n @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n \t\tdefault:\n \t\t\tflipped_block = 0;\n @@ diff.c: static void mark_color_as_moved(struct diff_options *o,\n - \t\t\t * The current line is the start of a new block.\n - \t\t\t * Setup the set of potential blocks.\n - \t\t\t */\n --\t\t\thashmap_for_each_entry_from(hm, match, ent) {\n -+\t\t\tfor (; match; match = match->next_match) {\n - \t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n - \t\t\t\tif (o->color_moved_ws_handling &\n - \t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n + \t\t\t\t */\n + \t\t\t\tn -= block_length;\n + \t\t\telse\n +-\t\t\t\tfill_potential_moved_blocks(o, hm, match, l,\n ++\t\t\t\tfill_potential_moved_blocks(o, match, l,\n + \t\t\t\t\t\t\t &pmb, &pmb_alloc,\n + \t\t\t\t\t\t\t &pmb_nr);\n + \n @@ diff.c: static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \n \tif (o->emitted_symbols) {\n\n-- \ngitgitgadget\n"},{"id":"439756","messageId":"e9daed2360c8047b1a21cc21dc975f54d9f4b256.1635336262.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v3 02/15] diff --color-moved: clear all flags on blocks that are too short","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:09Z","receivedAt":"2021-10-27T12:04:31Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nIf a block of potentially moved lines is not long enough then the\nDIFF_SYMBOL_MOVED_LINE flag is cleared on the matching lines so they\nare not marked as moved. To avoid problems when we start rewinding\nafter an unsuccessful match in a couple of commits time make sure all\nthe move related flags are cleared, not just DIFF_SYMBOL_MOVED_LINE.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 52c791574b7..bd8e4ec9757 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1114,6 +1114,8 @@ static int shrink_potential_moved_blocks(struct moved_block *pmb,\n * NEEDSWORK: This uses the same heuristic as blame_entry_score() in blame.c.\n * Think of a way to unify them.\n */\n+#define DIFF_SYMBOL_MOVED_LINE_ZEBRA_MASK \\\n+ (DIFF_SYMBOL_MOVED_LINE | DIFF_SYMBOL_MOVED_LINE_ALT)\n static int adjust_last_block(struct diff_options *o, int n, int block_length)\n {\n \tint i, alnum_count = 0;\n@@ -1130,7 +1132,7 @@ static int adjust_last_block(struct diff_options *o, int n, int block_length)\n \t\t}\n \t}\n \tfor (i = 1; i < block_length + 1; i++)\n-\t\to->emitted_symbols->buf[n - i].flags &= ~DIFF_SYMBOL_MOVED_LINE;\n+\t\to->emitted_symbols->buf[n - i].flags &= ~DIFF_SYMBOL_MOVED_LINE_ZEBRA_MASK;\n \treturn 0;\n }\n \n@@ -1237,8 +1239,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \tfree(pmb);\n }\n \n-#define DIFF_SYMBOL_MOVED_LINE_ZEBRA_MASK \\\n- (DIFF_SYMBOL_MOVED_LINE | DIFF_SYMBOL_MOVED_LINE_ALT)\n static void dim_moved_lines(struct diff_options *o)\n {\n \tint n;\n-- \ngitgitgadget\n\n"},{"id":"439757","messageId":"658aec2670c78f9753a5acccab20d3a1741403e6.1635336262.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v3 03/15] diff --color-moved: factor out function","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:10Z","receivedAt":"2021-10-27T12:04:33Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nThis code is quite heavily indented and having it in its own function\nsimplifies an upcoming change.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 51 ++++++++++++++++++++++++++++++++++-----------------\n 1 file changed, 34 insertions(+), 17 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex bd8e4ec9757..09af94e018c 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1098,6 +1098,38 @@ static int shrink_potential_moved_blocks(struct moved_block *pmb,\n \treturn rp + 1;\n }\n \n+static void fill_potential_moved_blocks(struct diff_options *o,\n+\t\t\t\t\tstruct hashmap *hm,\n+\t\t\t\t\tstruct moved_entry *match,\n+\t\t\t\t\tstruct emitted_diff_symbol *l,\n+\t\t\t\t\tstruct moved_block **pmb_p,\n+\t\t\t\t\tint *pmb_alloc_p, int *pmb_nr_p)\n+\n+{\n+\tstruct moved_block *pmb = *pmb_p;\n+\tint pmb_alloc = *pmb_alloc_p, pmb_nr = *pmb_nr_p;\n+\n+\t/*\n+\t * The current line is the start of a new block.\n+\t * Setup the set of potential blocks.\n+\t */\n+\thashmap_for_each_entry_from(hm, match, ent) {\n+\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n+\t\tif (o->color_moved_ws_handling &\n+\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n+\t\t\tif (compute_ws_delta(l, match->es, &(pmb[pmb_nr]).wsd))\n+\t\t\t\tpmb[pmb_nr++].match = match;\n+\t\t} else {\n+\t\t\tpmb[pmb_nr].wsd = 0;\n+\t\t\tpmb[pmb_nr++].match = match;\n+\t\t}\n+\t}\n+\n+\t*pmb_p = pmb;\n+\t*pmb_alloc_p = pmb_alloc;\n+\t*pmb_nr_p = pmb_nr;\n+}\n+\n /*\n * If o->color_moved is COLOR_MOVED_PLAIN, this function does nothing.\n *\n@@ -1198,23 +1230,8 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n \t\tif (pmb_nr == 0) {\n-\t\t\t/*\n-\t\t\t * The current line is the start of a new block.\n-\t\t\t * Setup the set of potential blocks.\n-\t\t\t */\n-\t\t\thashmap_for_each_entry_from(hm, match, ent) {\n-\t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n-\t\t\t\tif (o->color_moved_ws_handling &\n-\t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n-\t\t\t\t\tif (compute_ws_delta(l, match->es,\n-\t\t\t\t\t\t\t &pmb[pmb_nr].wsd))\n-\t\t\t\t\t\tpmb[pmb_nr++].match = match;\n-\t\t\t\t} else {\n-\t\t\t\t\tpmb[pmb_nr].wsd = 0;\n-\t\t\t\t\tpmb[pmb_nr++].match = match;\n-\t\t\t\t}\n-\t\t\t}\n-\n+\t\t\tfill_potential_moved_blocks(\n+\t\t\t\to, hm, match, l, &pmb, &pmb_alloc, &pmb_nr);\n \t\t\tif (adjust_last_block(o, n, block_length) &&\n \t\t\t pmb_nr && last_symbol != l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n-- \ngitgitgadget\n\n"},{"id":"439758","messageId":"a30f52d7f15e387836eb943ba081826b12a475a2.1635336263.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v3 04/15] diff --color-moved: rewind when discarding pmb","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:11Z","receivedAt":"2021-10-27T12:04:34Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\ndiff --color-moved colors the two sides of the diff separately. It\nwalks through the diff and tries to find matches on the other side of\nthe diff for the current line. When it finds one or more matches it\nstarts a \"potential moved block\" (pmb) and marks the current line as\nmoved. Then as it walks through the diff it only looks for matches for\nthe current line in the lines following those in the pmb. When none of\nthe lines in the pmb match it checks how long the match is and if it\nis too short it unmarks the lines as matched and goes back to finding\nall the lines that match the current line. As the process of finding\nmatching lines restarts from the end of the block that was too short\nit is possible to miss the start of a matching block on on side but\nnot the other. In the test added here \"-two\" would not be colored as\nmoved but \"+two\" would be.\n\nFix this by rewinding the current line when we reach the end of a\nblock that is too short. This is quadratic in the length of the\ndiscarded block. While the discarded blocks are quite short on a large\ndiff this still has a significant impact on the performance of\n--color-moved-ws=allow-indentation-change. The following commits\noptimize the performance of the --color-moved machinery which\nmitigates the performance impact of this commit. After the\noptimization this commit has a negligible impact on performance.\n\nTest HEAD^ HEAD\n------------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.33+0.05) 0.39 (0.34+0.04) +2.6%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.80 (0.76+0.03) 0.86 (0.82+0.04) +7.5%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 14.22(14.17+0.04) 19.01(18.93+0.05) +33.7%\n4002.4: log --no-color-moved --no-color-moved-ws 1.16 (1.06+0.09) 1.16 (1.07+0.07) +0.0%\n4002.5: log --color-moved --no-color-moved-ws 1.31 (1.22+0.09) 1.32 (1.22+0.09) +0.8%\n4002.6: log --color-moved-ws=allow-indentation-change 1.71 (1.61+0.09) 1.72 (1.63+0.08) +0.6%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 28 ++++++++++++++++++-----\n t/t4015-diff-whitespace.sh | 46 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 69 insertions(+), 5 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 09af94e018c..1e1b5127d15 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1205,7 +1205,15 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\tif (!match) {\n \t\t\tint i;\n \n-\t\t\tadjust_last_block(o, n, block_length);\n+\t\t\tif (!adjust_last_block(o, n, block_length) &&\n+\t\t\t block_length > 1) {\n+\t\t\t\t/*\n+\t\t\t\t * Rewind in case there is another match\n+\t\t\t\t * starting at the second line of the block\n+\t\t\t\t */\n+\t\t\t\tmatch = NULL;\n+\t\t\t\tn -= block_length;\n+\t\t\t}\n \t\t\tfor(i = 0; i < pmb_nr; i++)\n \t\t\t\tmoved_block_clear(&pmb[i]);\n \t\t\tpmb_nr = 0;\n@@ -1230,10 +1238,20 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n \t\tif (pmb_nr == 0) {\n-\t\t\tfill_potential_moved_blocks(\n-\t\t\t\to, hm, match, l, &pmb, &pmb_alloc, &pmb_nr);\n-\t\t\tif (adjust_last_block(o, n, block_length) &&\n-\t\t\t pmb_nr && last_symbol != l->s)\n+\t\t\tint contiguous = adjust_last_block(o, n, block_length);\n+\n+\t\t\tif (!contiguous && block_length > 1)\n+\t\t\t\t/*\n+\t\t\t\t * Rewind in case there is another match\n+\t\t\t\t * starting at the second line of the block\n+\t\t\t\t */\n+\t\t\t\tn -= block_length;\n+\t\t\telse\n+\t\t\t\tfill_potential_moved_blocks(o, hm, match, l,\n+\t\t\t\t\t\t\t &pmb, &pmb_alloc,\n+\t\t\t\t\t\t\t &pmb_nr);\n+\n+\t\t\tif (contiguous && pmb_nr && last_symbol != l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n \t\t\telse\n \t\t\t\tflipped_block = 0;\ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 2c13b62d3c6..308dc136596 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1833,6 +1833,52 @@ test_expect_success '--color-moved treats adjacent blocks as separate for MIN_AL\n \ttest_cmp expected actual\n '\n \n+test_expect_success '--color-moved rewinds for MIN_ALNUM_COUNT' '\n+\tgit reset --hard &&\n+\ttest_write_lines >file \\\n+\t\tA B C one two three four five six seven D E F G H I J &&\n+\tgit add file &&\n+\ttest_write_lines >file \\\n+\t\tone two A B C D E F G H I J two three four five six seven &&\n+\tgit diff --color-moved=zebra -- file &&\n+\n+\tgit diff --color-moved=zebra --color -- file >actual.raw &&\n+\tgrep -v \"index\" actual.raw | test_decode_color >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tdiff --git a/file b/file\n+\t--- a/file\n+\t+++ b/file\n+\t@@ -1,13 +1,8 @@\n+\t+one\n+\t+two\n+\t A\n+\t B\n+\t C\n+\t-one\n+\t-two\n+\t-three\n+\t-four\n+\t-five\n+\t-six\n+\t-seven\n+\t D\n+\t E\n+\t F\n+\t@@ -15,3 +10,9 @@ G\n+\t H\n+\t I\n+\t J\n+\t+two\n+\t+three\n+\t+four\n+\t+five\n+\t+six\n+\t+seven\n+\tEOF\n+\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'move detection with submodules' '\n \ttest_create_repo bananas &&\n \techo ripe >bananas/recipe &&\n-- \ngitgitgadget\n\n"},{"id":"439759","messageId":"1dde206b7b11bd04ed48336f4879370dbcd65671.1635336263.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v3 05/15] diff --color-moved=zebra: fix alternate coloring","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:12Z","receivedAt":"2021-10-27T12:04:35Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nb0a2ba4776 (\"diff --color-moved=zebra: be stricter with color\nalternation\", 2018-11-23) sought to avoid using the alternate colors\nunless there are two adjacent moved blocks of the same\nsign. Unfortunately it contains two bugs that prevented it from fixing\nthe problem properly. Firstly `last_symbol` is reset at the start of\neach iteration of the loop losing the symbol of the last line and\nsecondly when deciding whether to use the alternate color it should be\nchecking if the current line is the same sign of the last line, not a\ndifferent sign. The combination of the two errors means that we still\nuse the alternate color when we should do but we also use it when we\nshouldn't. This is most noticable when using\n--color-moved-ws=allow-indentation-change with hunks like\n\n-this line gets indented\n+ this line gets indented\n\nwhere the post image is colored with newMovedAlternate rather than\nnewMoved. While this does not matter much, the next commit will change\nthe coloring to be correct in this case, so lets fix the bug here to\nmake it clear why the output is changing and add a regression test.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 4 +--\n t/t4015-diff-whitespace.sh | 72 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 74 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 1e1b5127d15..53f0df75329 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1176,6 +1176,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n \tint n, flipped_block = 0, block_length = 0;\n+\tenum diff_symbol last_symbol = 0;\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n@@ -1183,7 +1184,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\tstruct moved_entry *key;\n \t\tstruct moved_entry *match = NULL;\n \t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n-\t\tenum diff_symbol last_symbol = 0;\n \n \t\tswitch (l->s) {\n \t\tcase DIFF_SYMBOL_PLUS:\n@@ -1251,7 +1251,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t\t\t\t\t &pmb, &pmb_alloc,\n \t\t\t\t\t\t\t &pmb_nr);\n \n-\t\t\tif (contiguous && pmb_nr && last_symbol != l->s)\n+\t\t\tif (contiguous && pmb_nr && last_symbol == l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n \t\t\telse\n \t\t\t\tflipped_block = 0;\ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 308dc136596..4e0fd76c6c5 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1442,6 +1442,78 @@ test_expect_success 'detect permutations inside moved code -- dimmed-zebra' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'zebra alternate color is only used when necessary' '\n+\tcat >old.txt <<-\\EOF &&\n+\tline 1A should be marked as oldMoved newMovedAlternate\n+\tline 1B should be marked as oldMoved newMovedAlternate\n+\tunchanged\n+\tline 2A should be marked as oldMoved newMovedAlternate\n+\tline 2B should be marked as oldMoved newMovedAlternate\n+\tline 3A should be marked as oldMovedAlternate newMoved\n+\tline 3B should be marked as oldMovedAlternate newMoved\n+\tunchanged\n+\tline 4A should be marked as oldMoved newMovedAlternate\n+\tline 4B should be marked as oldMoved newMovedAlternate\n+\tline 5A should be marked as oldMovedAlternate newMoved\n+\tline 5B should be marked as oldMovedAlternate newMoved\n+\tline 6A should be marked as oldMoved newMoved\n+\tline 6B should be marked as oldMoved newMoved\n+\tEOF\n+\tcat >new.txt <<-\\EOF &&\n+\t line 1A should be marked as oldMoved newMovedAlternate\n+\t line 1B should be marked as oldMoved newMovedAlternate\n+\tunchanged\n+\t line 3A should be marked as oldMovedAlternate newMoved\n+\t line 3B should be marked as oldMovedAlternate newMoved\n+\t line 2A should be marked as oldMoved newMovedAlternate\n+\t line 2B should be marked as oldMoved newMovedAlternate\n+\tunchanged\n+\t line 6A should be marked as oldMoved newMoved\n+\t line 6B should be marked as oldMoved newMoved\n+\t line 4A should be marked as oldMoved newMovedAlternate\n+\t line 4B should be marked as oldMoved newMovedAlternate\n+\t line 5A should be marked as oldMovedAlternate newMoved\n+\t line 5B should be marked as oldMovedAlternate newMoved\n+\tEOF\n+\ttest_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n+\t\t --color-moved-ws=allow-indentation-change \\\n+\t\t old.txt new.txt >output &&\n+\tgrep -v index output | test_decode_color >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tdiff --git a/old.txt b/new.txt\n+\t--- a/old.txt\n+\t+++ b/new.txt\n+\t@@ -1,14 +1,14 @@\n+\t-line 1A should be marked as oldMoved newMovedAlternate\n+\t-line 1B should be marked as oldMoved newMovedAlternate\n+\t+ line 1A should be marked as oldMoved newMovedAlternate\n+\t+ line 1B should be marked as oldMoved newMovedAlternate\n+\t unchanged\n+\t-line 2A should be marked as oldMoved newMovedAlternate\n+\t-line 2B should be marked as oldMoved newMovedAlternate\n+\t-line 3A should be marked as oldMovedAlternate newMoved\n+\t-line 3B should be marked as oldMovedAlternate newMoved\n+\t+ line 3A should be marked as oldMovedAlternate newMoved\n+\t+ line 3B should be marked as oldMovedAlternate newMoved\n+\t+ line 2A should be marked as oldMoved newMovedAlternate\n+\t+ line 2B should be marked as oldMoved newMovedAlternate\n+\t unchanged\n+\t-line 4A should be marked as oldMoved newMovedAlternate\n+\t-line 4B should be marked as oldMoved newMovedAlternate\n+\t-line 5A should be marked as oldMovedAlternate newMoved\n+\t-line 5B should be marked as oldMovedAlternate newMoved\n+\t-line 6A should be marked as oldMoved newMoved\n+\t-line 6B should be marked as oldMoved newMoved\n+\t+ line 6A should be marked as oldMoved newMoved\n+\t+ line 6B should be marked as oldMoved newMoved\n+\t+ line 4A should be marked as oldMoved newMovedAlternate\n+\t+ line 4B should be marked as oldMoved newMovedAlternate\n+\t+ line 5A should be marked as oldMovedAlternate newMoved\n+\t+ line 5B should be marked as oldMovedAlternate newMoved\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'cmd option assumes configured colored-moved' '\n \ttest_config color.diff.oldMoved \"magenta\" &&\n \ttest_config color.diff.newMoved \"cyan\" &&\n-- \ngitgitgadget\n\n"},{"id":"439760","messageId":"2717ff500d2ebf82179f89c90d690718221e9fa8.1635336263.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v3 06/15] diff --color-moved: avoid false short line matches and bad zerba coloring","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:13Z","receivedAt":"2021-10-27T12:04:37Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nWhen marking moved lines it is possible for a block of potential\nmatched lines to extend past a change in sign when there is a sequence\nof added lines whose text matches the text of a sequence of deleted\nand added lines. Most of the time either `match` will be NULL or\n`pmb_advance_or_null()` will fail when the loop encounters a change of\nsign but there are corner cases where `match` is non-NULL and\n`pmb_advance_or_null()` successfully advances the moved block despite\nthe change in sign.\n\nOne consequence of this is highlighting a short line as moved when it\nshould not be. For example\n\n-moved line # Correctly highlighted as moved\n+short line # Wrongly highlighted as moved\n context\n+moved line # Correctly highlighted as moved\n+short line\n context\n-short line\n\nThe other consequence is coloring a moved addition following a moved\ndeletion in the wrong color. In the example below the first \"+moved\nline 3\" should be highlighted as newMoved not newMovedAlternate.\n\n-moved line 1 # Correctly highlighted as oldMoved\n-moved line 2 # Correctly highlighted as oldMovedAlternate\n+moved line 3 # Wrongly highlighted as newMovedAlternate\n context # Everything else is highlighted correctly\n+moved line 2\n+moved line 3\n context\n+moved line 1\n-moved line 3\n\nThese false matches are more likely when using --color-moved-ws with\nthe exception of --color-moved-ws=allow-indentation-change which ties\nthe sign of the current whitespace delta to the sign of the line to\navoid this problem. The fix is to check that the sign of the new line\nbeing matched is the same as the sign of the line that started the\nblock of potential matches.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 17 ++++++----\n t/t4015-diff-whitespace.sh | 65 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 76 insertions(+), 6 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 53f0df75329..efba2789354 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1176,7 +1176,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n \tint n, flipped_block = 0, block_length = 0;\n-\tenum diff_symbol last_symbol = 0;\n+\tenum diff_symbol moved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n@@ -1202,7 +1202,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tflipped_block = 0;\n \t\t}\n \n-\t\tif (!match) {\n+\t\tif (pmb_nr && (!match || l->s != moved_symbol)) {\n \t\t\tint i;\n \n \t\t\tif (!adjust_last_block(o, n, block_length) &&\n@@ -1219,12 +1219,13 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tpmb_nr = 0;\n \t\t\tblock_length = 0;\n \t\t\tflipped_block = 0;\n-\t\t\tlast_symbol = l->s;\n+\t\t}\n+\t\tif (!match) {\n+\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (o->color_moved == COLOR_MOVED_PLAIN) {\n-\t\t\tlast_symbol = l->s;\n \t\t\tl->flags |= DIFF_SYMBOL_MOVED_LINE;\n \t\t\tcontinue;\n \t\t}\n@@ -1251,11 +1252,16 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t\t\t\t\t &pmb, &pmb_alloc,\n \t\t\t\t\t\t\t &pmb_nr);\n \n-\t\t\tif (contiguous && pmb_nr && last_symbol == l->s)\n+\t\t\tif (contiguous && pmb_nr && moved_symbol == l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n \t\t\telse\n \t\t\t\tflipped_block = 0;\n \n+\t\t\tif (pmb_nr)\n+\t\t\t\tmoved_symbol = l->s;\n+\t\t\telse\n+\t\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n+\n \t\t\tblock_length = 0;\n \t\t}\n \n@@ -1265,7 +1271,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tif (flipped_block && o->color_moved != COLOR_MOVED_BLOCKS)\n \t\t\t\tl->flags |= DIFF_SYMBOL_MOVED_LINE_ALT;\n \t\t}\n-\t\tlast_symbol = l->s;\n \t}\n \tadjust_last_block(o, n, block_length);\n \ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 4e0fd76c6c5..15782c879d2 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1514,6 +1514,71 @@ test_expect_success 'zebra alternate color is only used when necessary' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'short lines of opposite sign do not get marked as moved' '\n+\tcat >old.txt <<-\\EOF &&\n+\tthis line should be marked as moved\n+\tunchanged\n+\tunchanged\n+\tunchanged\n+\tunchanged\n+\ttoo short\n+\tthis line should be marked as oldMoved newMoved\n+\tthis line should be marked as oldMovedAlternate newMoved\n+\tunchanged 1\n+\tunchanged 2\n+\tunchanged 3\n+\tunchanged 4\n+\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n+\tEOF\n+\tcat >new.txt <<-\\EOF &&\n+\ttoo short\n+\tunchanged\n+\tunchanged\n+\tthis line should be marked as moved\n+\ttoo short\n+\tunchanged\n+\tunchanged\n+\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n+\tunchanged 1\n+\tunchanged 2\n+\tthis line should be marked as oldMovedAlternate newMoved\n+\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n+\tunchanged 3\n+\tthis line should be marked as oldMoved newMoved\n+\tunchanged 4\n+\tEOF\n+\ttest_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n+\t\told.txt new.txt >output && cat output &&\n+\tgrep -v index output | test_decode_color >actual &&\n+\tcat >expect <<-\\EOF &&\n+\tdiff --git a/old.txt b/new.txt\n+\t--- a/old.txt\n+\t+++ b/new.txt\n+\t@@ -1,13 +1,15 @@\n+\t-this line should be marked as moved\n+\t+too short\n+\t unchanged\n+\t unchanged\n+\t+this line should be marked as moved\n+\t+too short\n+\t unchanged\n+\t unchanged\n+\t-too short\n+\t-this line should be marked as oldMoved newMoved\n+\t-this line should be marked as oldMovedAlternate newMoved\n+\t+this line should be marked as oldMoved newMoved/newMovedAlternate\n+\t unchanged 1\n+\t unchanged 2\n+\t+this line should be marked as oldMovedAlternate newMoved\n+\t+this line should be marked as oldMoved newMoved/newMovedAlternate\n+\t unchanged 3\n+\t+this line should be marked as oldMoved newMoved\n+\t unchanged 4\n+\t-this line should be marked as oldMoved newMoved/newMovedAlternate\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'cmd option assumes configured colored-moved' '\n \ttest_config color.diff.oldMoved \"magenta\" &&\n \ttest_config color.diff.newMoved \"cyan\" &&\n-- \ngitgitgadget\n\n"},{"id":"439761","messageId":"f96fa71d53c491a2fd0a4dc61e9747d73b624329.1635336263.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v3 07/15] diff: simplify allow-indentation-change delta calculation","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:14Z","receivedAt":"2021-10-27T12:04:39Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nNow that we reliably end a block when the sign changes we don't need\nthe whitespace delta calculation to rely on the sign.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 13 ++-----------\n 1 file changed, 2 insertions(+), 11 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex efba2789354..9aff167be27 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -864,23 +864,17 @@ static int compute_ws_delta(const struct emitted_diff_symbol *a,\n \t a_width = a->indent_width,\n \t b_off = b->indent_off,\n \t b_width = b->indent_width;\n-\tint delta;\n \n \tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE) {\n \t\t*out = INDENT_BLANKLINE;\n \t\treturn 1;\n \t}\n \n-\tif (a->s == DIFF_SYMBOL_PLUS)\n-\t\tdelta = a_width - b_width;\n-\telse\n-\t\tdelta = b_width - a_width;\n-\n \tif (a_len - a_off != b_len - b_off ||\n \t memcmp(a->line + a_off, b->line + b_off, a_len - a_off))\n \t\treturn 0;\n \n-\t*out = delta;\n+\t*out = a_width - b_width;\n \n \treturn 1;\n }\n@@ -924,10 +918,7 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \t * match those of the current block and that the text of 'l' and 'cur'\n \t * after the indentation match.\n \t */\n-\tif (cur->es->s == DIFF_SYMBOL_PLUS)\n-\t\tdelta = a_width - c_width;\n-\telse\n-\t\tdelta = c_width - a_width;\n+\tdelta = c_width - a_width;\n \n \t/*\n \t * If the previous lines of this block were all blank then set its\n-- \ngitgitgadget\n\n"},{"id":"439762","messageId":"324b689c915ddcb006907eec72a10257186c9e17.1635336263.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v3 08/15] diff --color-moved-ws=allow-indentation-change: simplify and optimize","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:15Z","receivedAt":"2021-10-27T12:04:40Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nIf we already have a block of potentially moved lines then as we move\ndown the diff we need to check if the next line of each potentially\nmoved line matches the current line of the diff. The implementation of\n--color-moved-ws=allow-indentation-change was needlessly performing\nthis check on all the lines in the diff that matched the current line\nrather than just the current line. To exacerbate the problem finding\nall the other lines in the diff that match the current line involves a\nfuzzy lookup so we were wasting even more time performing a second\ncomparison to filter out the non-matching lines. Fixing this reduces\ntime to run\n git diff --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\nby 93% compared to master and simplifies the code.\n\nTest HEAD^ HEAD\n---------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.35+0.03) 0.38(0.35+0.03) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.86 (0.80+0.06) 0.87(0.83+0.04) +1.2%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 19.01(18.93+0.06) 0.97(0.92+0.04) -94.9%\n4002.4: log --no-color-moved --no-color-moved-ws 1.16 (1.06+0.09) 1.17(1.06+0.10) +0.9%\n4002.5: log --color-moved --no-color-moved-ws 1.32 (1.25+0.07) 1.32(1.24+0.08) +0.0%\n4002.6: log --color-moved-ws=allow-indentation-change 1.71 (1.64+0.06) 1.36(1.25+0.10) -20.5%\n\nTest master HEAD\n---------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.33+0.05) 0.38(0.35+0.03) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.80 (0.75+0.04) 0.87(0.83+0.04) +8.7%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 14.20(14.15+0.05) 0.97(0.92+0.04) -93.2%\n4002.4: log --no-color-moved --no-color-moved-ws 1.15 (1.05+0.09) 1.17(1.06+0.10) +1.7%\n4002.5: log --color-moved --no-color-moved-ws 1.30 (1.19+0.11) 1.32(1.24+0.08) +1.5%\n4002.6: log --color-moved-ws=allow-indentation-change 1.70 (1.63+0.06) 1.36(1.25+0.10) -20.0%\n\nHelped-by: Jeff King \nSigned-off-by: Phillip Wood \n---\n diff.c | 70 +++++++++++++++++-----------------------------------------\n 1 file changed, 20 insertions(+), 50 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 9aff167be27..78a486021ab 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -879,37 +879,21 @@ static int compute_ws_delta(const struct emitted_diff_symbol *a,\n \treturn 1;\n }\n \n-static int cmp_in_block_with_wsd(const struct diff_options *o,\n-\t\t\t\t const struct moved_entry *cur,\n-\t\t\t\t const struct moved_entry *match,\n-\t\t\t\t struct moved_block *pmb,\n-\t\t\t\t int n)\n-{\n-\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n-\tint al = cur->es->len, bl = match->es->len, cl = l->len;\n+static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n+\t\t\t\t const struct emitted_diff_symbol *l,\n+\t\t\t\t struct moved_block *pmb)\n+{\n+\tint al = cur->es->len, bl = l->len;\n \tconst char *a = cur->es->line,\n-\t\t *b = match->es->line,\n-\t\t *c = l->line;\n+\t\t *b = l->line;\n \tint a_off = cur->es->indent_off,\n \t a_width = cur->es->indent_width,\n-\t c_off = l->indent_off,\n-\t c_width = l->indent_width;\n+\t b_off = l->indent_off,\n+\t b_width = l->indent_width;\n \tint delta;\n \n-\t/*\n-\t * We need to check if 'cur' is equal to 'match'. As those\n-\t * are from the same (+/-) side, we do not need to adjust for\n-\t * indent changes. However these were found using fuzzy\n-\t * matching so we do have to check if they are equal. Here we\n-\t * just check the lengths. We delay calling memcmp() to check\n-\t * the contents until later as if the length comparison for a\n-\t * and c fails we can avoid the call all together.\n-\t */\n-\tif (al != bl)\n-\t\treturn 1;\n-\n \t/* If 'l' and 'cur' are both blank then they match. */\n-\tif (a_width == INDENT_BLANKLINE && c_width == INDENT_BLANKLINE)\n+\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n \t\treturn 0;\n \n \t/*\n@@ -918,7 +902,7 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \t * match those of the current block and that the text of 'l' and 'cur'\n \t * after the indentation match.\n \t */\n-\tdelta = c_width - a_width;\n+\tdelta = b_width - a_width;\n \n \t/*\n \t * If the previous lines of this block were all blank then set its\n@@ -927,9 +911,8 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \tif (pmb->wsd == INDENT_BLANKLINE)\n \t\tpmb->wsd = delta;\n \n-\treturn !(delta == pmb->wsd && al - a_off == cl - c_off &&\n-\t\t !memcmp(a, b, al) && !\n-\t\t memcmp(a + a_off, c + c_off, al - a_off));\n+\treturn !(delta == pmb->wsd && al - a_off == bl - b_off &&\n+\t\t !memcmp(a + a_off, b + b_off, al - a_off));\n }\n \n static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n@@ -1030,36 +1013,23 @@ static void pmb_advance_or_null(struct diff_options *o,\n }\n \n static void pmb_advance_or_null_multi_match(struct diff_options *o,\n-\t\t\t\t\t struct moved_entry *match,\n-\t\t\t\t\t struct hashmap *hm,\n+\t\t\t\t\t struct emitted_diff_symbol *l,\n \t\t\t\t\t struct moved_block *pmb,\n-\t\t\t\t\t int pmb_nr, int n)\n+\t\t\t\t\t int pmb_nr)\n {\n \tint i;\n-\tchar *got_match = xcalloc(1, pmb_nr);\n-\n-\thashmap_for_each_entry_from(hm, match, ent) {\n-\t\tfor (i = 0; i < pmb_nr; i++) {\n-\t\t\tstruct moved_entry *prev = pmb[i].match;\n-\t\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n-\t\t\t\t\tprev->next_line : NULL;\n-\t\t\tif (!cur)\n-\t\t\t\tcontinue;\n-\t\t\tif (!cmp_in_block_with_wsd(o, cur, match, &pmb[i], n))\n-\t\t\t\tgot_match[i] |= 1;\n-\t\t}\n-\t}\n \n \tfor (i = 0; i < pmb_nr; i++) {\n-\t\tif (got_match[i]) {\n+\t\tstruct moved_entry *prev = pmb[i].match;\n+\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n+\t\t\tprev->next_line : NULL;\n+\t\tif (cur && !cmp_in_block_with_wsd(cur, l, &pmb[i])) {\n \t\t\t/* Advance to the next line */\n-\t\t\tpmb[i].match = pmb[i].match->next_line;\n+\t\t\tpmb[i].match = cur;\n \t\t} else {\n \t\t\tmoved_block_clear(&pmb[i]);\n \t\t}\n \t}\n-\n-\tfree(got_match);\n }\n \n static int shrink_potential_moved_blocks(struct moved_block *pmb,\n@@ -1223,7 +1193,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \n \t\tif (o->color_moved_ws_handling &\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\tpmb_advance_or_null_multi_match(o, match, hm, pmb, pmb_nr, n);\n+\t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n \t\telse\n \t\t\tpmb_advance_or_null(o, match, hm, pmb, pmb_nr);\n \n-- \ngitgitgadget\n\n"},{"id":"439763","messageId":"f142f33276a17b5ad74a60c26b54f099b25ba5d6.1635336263.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v3 09/15] diff --color-moved: call comparison function directly","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:16Z","receivedAt":"2021-10-27T12:04:45Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nThis change will allow us to easily combine pmb_advance_or_null() and\npmb_advance_or_null_multi_match() in the next commit. Calling\nxdiff_compare_lines() directly rather than using a function pointer\nfrom the hash map has little effect on the run time.\n\nTest HEAD^ HEAD\n-------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38(0.35+0.03) 0.38(0.32+0.06) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.87(0.83+0.04) 0.87(0.80+0.06) +0.0%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 0.97(0.92+0.04) 0.97(0.93+0.04) +0.0%\n4002.4: log --no-color-moved --no-color-moved-ws 1.17(1.06+0.10) 1.16(1.10+0.05) -0.9%\n4002.5: log --color-moved --no-color-moved-ws 1.32(1.24+0.08) 1.31(1.22+0.09) -0.8%\n4002.6: log --color-moved-ws=allow-indentation-change 1.36(1.25+0.10) 1.35(1.25+0.10) -0.7%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 11 +++++++----\n 1 file changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 78a486021ab..22e0edac173 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -994,17 +994,20 @@ static void add_lines_to_move_detection(struct diff_options *o,\n }\n \n static void pmb_advance_or_null(struct diff_options *o,\n-\t\t\t\tstruct moved_entry *match,\n-\t\t\t\tstruct hashmap *hm,\n+\t\t\t\tstruct emitted_diff_symbol *l,\n \t\t\t\tstruct moved_block *pmb,\n \t\t\t\tint pmb_nr)\n {\n \tint i;\n+\tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n+\n \tfor (i = 0; i < pmb_nr; i++) {\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n \t\t\t\tprev->next_line : NULL;\n-\t\tif (cur && !hm->cmpfn(o, &cur->ent, &match->ent, NULL)) {\n+\t\tif (cur && xdiff_compare_lines(cur->es->line, cur->es->len,\n+\t\t\t\t\t\tl->line, l->len,\n+\t\t\t\t\t\tflags)) {\n \t\t\tpmb[i].match = cur;\n \t\t} else {\n \t\t\tpmb[i].match = NULL;\n@@ -1195,7 +1198,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n \t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n \t\telse\n-\t\t\tpmb_advance_or_null(o, match, hm, pmb, pmb_nr);\n+\t\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n \n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n-- \ngitgitgadget\n\n"},{"id":"439764","messageId":"8f3ea865dd33047397207c9909f114601b0a2dda.1635336263.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v3 10/15] diff --color-moved: unify moved block growth functions","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:17Z","receivedAt":"2021-10-27T12:04:47Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nAfter the last two commits pmb_advance_or_null() and\npmb_advance_or_null_multi_match() differ only in the comparison they\nperform. Lets simplify the code by combining them into a single\nfunction.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 41 ++++++++++++-----------------------------\n 1 file changed, 12 insertions(+), 29 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 22e0edac173..51f092e724e 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1002,36 +1002,23 @@ static void pmb_advance_or_null(struct diff_options *o,\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \n \tfor (i = 0; i < pmb_nr; i++) {\n+\t\tint match;\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n \t\t\t\tprev->next_line : NULL;\n-\t\tif (cur && xdiff_compare_lines(cur->es->line, cur->es->len,\n-\t\t\t\t\t\tl->line, l->len,\n-\t\t\t\t\t\tflags)) {\n-\t\t\tpmb[i].match = cur;\n-\t\t} else {\n-\t\t\tpmb[i].match = NULL;\n-\t\t}\n-\t}\n-}\n \n-static void pmb_advance_or_null_multi_match(struct diff_options *o,\n-\t\t\t\t\t struct emitted_diff_symbol *l,\n-\t\t\t\t\t struct moved_block *pmb,\n-\t\t\t\t\t int pmb_nr)\n-{\n-\tint i;\n-\n-\tfor (i = 0; i < pmb_nr; i++) {\n-\t\tstruct moved_entry *prev = pmb[i].match;\n-\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n-\t\t\tprev->next_line : NULL;\n-\t\tif (cur && !cmp_in_block_with_wsd(cur, l, &pmb[i])) {\n-\t\t\t/* Advance to the next line */\n+\t\tif (o->color_moved_ws_handling &\n+\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n+\t\t\tmatch = cur &&\n+\t\t\t\t!cmp_in_block_with_wsd(cur, l, &pmb[i]);\n+\t\telse\n+\t\t\tmatch = cur &&\n+\t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n+\t\t\t\t\t\t l->line, l->len, flags);\n+\t\tif (match)\n \t\t\tpmb[i].match = cur;\n-\t\t} else {\n+\t\telse\n \t\t\tmoved_block_clear(&pmb[i]);\n-\t\t}\n \t}\n }\n \n@@ -1194,11 +1181,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tif (o->color_moved_ws_handling &\n-\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n-\t\telse\n-\t\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n+\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n \n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n-- \ngitgitgadget\n\n"},{"id":"439765","messageId":"078c04d4a66c51acdb3477b49813a9dc1c144f15.1635336263.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v3 11/15] diff --color-moved: shrink potential moved blocks as we go","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:18Z","receivedAt":"2021-10-27T12:04:49Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nRather than setting `match` to NULL and then looping over the list of\npotential matched blocks for a second time to remove blocks with no\nmatches just filter out the blocks with no matches as we go.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 44 ++++++++------------------------------------\n 1 file changed, 8 insertions(+), 36 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 51f092e724e..626fd47aa0e 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -996,12 +996,12 @@ static void add_lines_to_move_detection(struct diff_options *o,\n static void pmb_advance_or_null(struct diff_options *o,\n \t\t\t\tstruct emitted_diff_symbol *l,\n \t\t\t\tstruct moved_block *pmb,\n-\t\t\t\tint pmb_nr)\n+\t\t\t\tint *pmb_nr)\n {\n-\tint i;\n+\tint i, j;\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \n-\tfor (i = 0; i < pmb_nr; i++) {\n+\tfor (i = 0, j = 0; i < *pmb_nr; i++) {\n \t\tint match;\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n@@ -1015,38 +1015,12 @@ static void pmb_advance_or_null(struct diff_options *o,\n \t\t\tmatch = cur &&\n \t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n \t\t\t\t\t\t l->line, l->len, flags);\n-\t\tif (match)\n-\t\t\tpmb[i].match = cur;\n-\t\telse\n-\t\t\tmoved_block_clear(&pmb[i]);\n-\t}\n-}\n-\n-static int shrink_potential_moved_blocks(struct moved_block *pmb,\n-\t\t\t\t\t int pmb_nr)\n-{\n-\tint lp, rp;\n-\n-\t/* Shrink the set of potential block to the remaining running */\n-\tfor (lp = 0, rp = pmb_nr - 1; lp <= rp;) {\n-\t\twhile (lp < pmb_nr && pmb[lp].match)\n-\t\t\tlp++;\n-\t\t/* lp points at the first NULL now */\n-\n-\t\twhile (rp > -1 && !pmb[rp].match)\n-\t\t\trp--;\n-\t\t/* rp points at the last non-NULL */\n-\n-\t\tif (lp < pmb_nr && rp > -1 && lp < rp) {\n-\t\t\tpmb[lp] = pmb[rp];\n-\t\t\tmemset(&pmb[rp], 0, sizeof(pmb[rp]));\n-\t\t\trp--;\n-\t\t\tlp++;\n+\t\tif (match) {\n+\t\t\tpmb[j] = pmb[i];\n+\t\t\tpmb[j++].match = cur;\n \t\t}\n \t}\n-\n-\t/* Remember the number of running sets */\n-\treturn rp + 1;\n+\t*pmb_nr = j;\n }\n \n static void fill_potential_moved_blocks(struct diff_options *o,\n@@ -1181,9 +1155,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n-\n-\t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n+\t\tpmb_advance_or_null(o, l, pmb, &pmb_nr);\n \n \t\tif (pmb_nr == 0) {\n \t\t\tint contiguous = adjust_last_block(o, n, block_length);\n-- \ngitgitgadget\n\n"},{"id":"439766","messageId":"618371471a0effd4ad2d69105a7e4495ad46c350.1635336263.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v3 12/15] diff --color-moved: stop clearing potential moved blocks","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:19Z","receivedAt":"2021-10-27T12:04:50Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nmoved_block_clear() was introduced in 74d156f4a1 (\"diff\n--color-moved-ws: fix double free crash\", 2018-10-04) to free the\nmemory that was allocated when initializing a potential moved\nblock. However since 21536d077f (\"diff --color-moved-ws: modify\nallow-indentation-change\", 2018-11-23) initializing a potential moved\nblock no longer allocates any memory. Up until the last commit we were\nrelying on moved_block_clear() to set the `match` pointer to NULL when\na block stopped matching, but since that commit we do not clear a\nmoved block that does not match so it does not make sense to clear\nthem elsewhere.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 11 -----------\n 1 file changed, 11 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 626fd47aa0e..ffbe09937bc 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -807,11 +807,6 @@ struct moved_block {\n \tint wsd; /* The whitespace delta of this block */\n };\n \n-static void moved_block_clear(struct moved_block *b)\n-{\n-\tmemset(b, 0, sizeof(*b));\n-}\n-\n #define INDENT_BLANKLINE INT_MIN\n \n static void fill_es_indent_data(struct emitted_diff_symbol *es)\n@@ -1128,8 +1123,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t}\n \n \t\tif (pmb_nr && (!match || l->s != moved_symbol)) {\n-\t\t\tint i;\n-\n \t\t\tif (!adjust_last_block(o, n, block_length) &&\n \t\t\t block_length > 1) {\n \t\t\t\t/*\n@@ -1139,8 +1132,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t\tmatch = NULL;\n \t\t\t\tn -= block_length;\n \t\t\t}\n-\t\t\tfor(i = 0; i < pmb_nr; i++)\n-\t\t\t\tmoved_block_clear(&pmb[i]);\n \t\t\tpmb_nr = 0;\n \t\t\tblock_length = 0;\n \t\t\tflipped_block = 0;\n@@ -1193,8 +1184,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t}\n \tadjust_last_block(o, n, block_length);\n \n-\tfor(n = 0; n < pmb_nr; n++)\n-\t\tmoved_block_clear(&pmb[n]);\n \tfree(pmb);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"439767","messageId":"6a8e9a2724d638cca04e3ac7d52995d9e2b0b990.1635336263.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v3 13/15] diff --color-moved-ws=allow-indentation-change: improve hash lookups","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:20Z","receivedAt":"2021-10-27T12:04:52Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nAs libxdiff does not have a whitespace flag to ignore the indentation\nthe code for --color-moved-ws=allow-indentation-change uses\nXDF_IGNORE_WHITESPACE and then filters out any hash lookups where\nthere are non-indentation changes. This filtering is inefficient as\nwe have to perform another string comparison.\n\nBy using the offset data that we have already computed to skip the\nindentation we can avoid using XDF_IGNORE_WHITESPACE and safely remove\nthe extra checks which improves the performance by 11% and paves the\nway for the elimination of string comparisons in the next commit.\n\nThis change slightly increases the run time of other --color-moved\nmodes. This could be avoided by using different comparison functions\nfor the different modes but after the next two commits there is no\nmeasurable benefit in doing so.\n\nThere is a change in behavior for lines that begin with a form-feed or\nvertical-tab character. Since b46054b374 (\"xdiff: use\ngit-compat-util\", 2019-04-11) xdiff does not treat '\\f' or '\\v' as\nwhitespace characters. This means that lines starting with those\ncharacters are never considered to be blank and never match a line\nthat does not start with the same character. After this patch a line\nmatching \"^[\\f\\v\\r]*[ \\t]*$\" is considered to be blank by\n--color-moved-ws=allow-indentation-change and lines beginning\n\"^[\\f\\v\\r]*[ \\t]*\" can match another line if the suffixes match. This\nchanges the output of git show for d18f76dccf (\"compat/regex: use the\nregex engine from gawk for compat\", 2010-08-17) as some lines in the\npre-image before a moved block that contain '\\f' are now considered\nmoved as well as they match a blank line before the moved lines in the\npost-image. This commit updates one of the tests to reflect this\nchange.\n\nTest HEAD^ HEAD\n--------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38(0.33+0.05) 0.38(0.33+0.05) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.86(0.82+0.04) 0.88(0.84+0.04) +2.3%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 0.97(0.94+0.03) 0.86(0.81+0.05) -11.3%\n4002.4: log --no-color-moved --no-color-moved-ws 1.16(1.07+0.09) 1.16(1.06+0.09) +0.0%\n4002.5: log --color-moved --no-color-moved-ws 1.32(1.26+0.06) 1.33(1.27+0.05) +0.8%\n4002.6: log --color-moved-ws=allow-indentation-change 1.35(1.29+0.06) 1.33(1.24+0.08) -1.5%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 65 +++++++++++---------------------------\n t/t4015-diff-whitespace.sh | 22 ++++++-------\n 2 files changed, 30 insertions(+), 57 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex ffbe09937bc..2085c063675 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -850,28 +850,15 @@ static void fill_es_indent_data(struct emitted_diff_symbol *es)\n }\n \n static int compute_ws_delta(const struct emitted_diff_symbol *a,\n-\t\t\t const struct emitted_diff_symbol *b,\n-\t\t\t int *out)\n-{\n-\tint a_len = a->len,\n-\t b_len = b->len,\n-\t a_off = a->indent_off,\n-\t a_width = a->indent_width,\n-\t b_off = b->indent_off,\n+\t\t\t const struct emitted_diff_symbol *b)\n+{\n+\tint a_width = a->indent_width,\n \t b_width = b->indent_width;\n \n-\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE) {\n-\t\t*out = INDENT_BLANKLINE;\n-\t\treturn 1;\n-\t}\n-\n-\tif (a_len - a_off != b_len - b_off ||\n-\t memcmp(a->line + a_off, b->line + b_off, a_len - a_off))\n-\t\treturn 0;\n-\n-\t*out = a_width - b_width;\n+\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n+\t\treturn INDENT_BLANKLINE;\n \n-\treturn 1;\n+\treturn a_width - b_width;\n }\n \n static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n@@ -916,26 +903,17 @@ static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n \t\t\t const void *keydata)\n {\n \tconst struct diff_options *diffopt = hashmap_cmp_fn_data;\n-\tconst struct moved_entry *a, *b;\n+\tconst struct emitted_diff_symbol *a, *b;\n \tunsigned flags = diffopt->color_moved_ws_handling\n \t\t\t & XDF_WHITESPACE_FLAGS;\n \n-\ta = container_of(eptr, const struct moved_entry, ent);\n-\tb = container_of(entry_or_key, const struct moved_entry, ent);\n+\ta = container_of(eptr, const struct moved_entry, ent)->es;\n+\tb = container_of(entry_or_key, const struct moved_entry, ent)->es;\n \n-\tif (diffopt->color_moved_ws_handling &\n-\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t/*\n-\t\t * As there is not specific white space config given,\n-\t\t * we'd need to check for a new block, so ignore all\n-\t\t * white space. The setup of the white space\n-\t\t * configuration for the next block is done else where\n-\t\t */\n-\t\tflags |= XDF_IGNORE_WHITESPACE;\n-\n-\treturn !xdiff_compare_lines(a->es->line, a->es->len,\n-\t\t\t\t b->es->line, b->es->len,\n-\t\t\t\t flags);\n+\treturn !xdiff_compare_lines(a->line + a->indent_off,\n+\t\t\t\t a->len - a->indent_off,\n+\t\t\t\t b->line + b->indent_off,\n+\t\t\t\t b->len - b->indent_off, flags);\n }\n \n static struct moved_entry *prepare_entry(struct diff_options *o,\n@@ -944,7 +922,8 @@ static struct moved_entry *prepare_entry(struct diff_options *o,\n \tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n \tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[line_no];\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n-\tunsigned int hash = xdiff_hash_string(l->line, l->len, flags);\n+\tunsigned int hash = xdiff_hash_string(l->line + l->indent_off,\n+\t\t\t\t\t l->len - l->indent_off, flags);\n \n \thashmap_entry_init(&ret->ent, hash);\n \tret->es = l;\n@@ -1036,13 +1015,11 @@ static void fill_potential_moved_blocks(struct diff_options *o,\n \thashmap_for_each_entry_from(hm, match, ent) {\n \t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n \t\tif (o->color_moved_ws_handling &\n-\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n-\t\t\tif (compute_ws_delta(l, match->es, &(pmb[pmb_nr]).wsd))\n-\t\t\t\tpmb[pmb_nr++].match = match;\n-\t\t} else {\n+\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n+\t\t\tpmb[pmb_nr].wsd = compute_ws_delta(l, match->es);\n+\t\telse\n \t\t\tpmb[pmb_nr].wsd = 0;\n-\t\t\tpmb[pmb_nr++].match = match;\n-\t\t}\n+\t\tpmb[pmb_nr++].match = match;\n \t}\n \n \t*pmb_p = pmb;\n@@ -6276,10 +6253,6 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \t\tif (o->color_moved) {\n \t\t\tstruct hashmap add_lines, del_lines;\n \n-\t\t\tif (o->color_moved_ws_handling &\n-\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\t\to->color_moved_ws_handling |= XDF_IGNORE_WHITESPACE;\n-\n \t\t\thashmap_init(&del_lines, moved_entry_cmp, o, 0);\n \t\t\thashmap_init(&add_lines, moved_entry_cmp, o, 0);\n \ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 15782c879d2..50d0cf486be 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -2206,10 +2206,10 @@ EMPTY=''\n test_expect_success 'compare mixed whitespace delta across moved blocks' '\n \n \tgit reset --hard &&\n-\ttr Q_ \"\\t \" <<-EOF >text.txt &&\n-\t${EMPTY}\n-\t____too short without\n-\t${EMPTY}\n+\ttr \"^|Q_\" \"\\f\\v\\t \" <<-EOF >text.txt &&\n+\t^__\n+\t|____too short without\n+\t^\n \t___being grouped across blank line\n \t${EMPTY}\n \tcontext\n@@ -2228,7 +2228,7 @@ test_expect_success 'compare mixed whitespace delta across moved blocks' '\n \tgit add text.txt &&\n \tgit commit -m \"add text.txt\" &&\n \n-\ttr Q_ \"\\t \" <<-EOF >text.txt &&\n+\ttr \"^|Q_\" \"\\f\\v\\t \" <<-EOF >text.txt &&\n \tcontext\n \tlines\n \tto\n@@ -2239,7 +2239,7 @@ test_expect_success 'compare mixed whitespace delta across moved blocks' '\n \t${EMPTY}\n \tQQtoo short without\n \t${EMPTY}\n-\tQ_______being grouped across blank line\n+\t^Q_______being grouped across blank line\n \t${EMPTY}\n \tQ_QThese two lines have had their\n \tindentation reduced by four spaces\n@@ -2251,16 +2251,16 @@ test_expect_success 'compare mixed whitespace delta across moved blocks' '\n \t\t-c core.whitespace=space-before-tab \\\n \t\tdiff --color --color-moved --ws-error-highlight=all \\\n \t\t--color-moved-ws=allow-indentation-change >actual.raw &&\n-\tgrep -v \"index\" actual.raw | test_decode_color >actual &&\n+\tgrep -v \"index\" actual.raw | tr \"\\f\\v\" \"^|\" | test_decode_color >actual &&\n \n \tcat <<-\\EOF >expected &&\n \tdiff --git a/text.txt b/text.txt\n \t--- a/text.txt\n \t+++ b/text.txt\n \t@@ -1,16 +1,16 @@\n-\t-\n-\t- too short without\n-\t-\n+\t-^ \n+\t-| too short without\n+\t-^\n \t- being grouped across blank line\n \t-\n \t context\n@@ -2280,7 +2280,7 @@ test_expect_success 'compare mixed whitespace delta across moved blocks' '\n \t+\n \t+\t\ttoo short without\n \t+\n-\t+\t being grouped across blank line\n+\t+^\t being grouped across blank line\n \t+\n \t+\t \tThese two lines have had their\n \t+indentation reduced by four spaces\n-- \ngitgitgadget\n\n"},{"id":"439768","messageId":"ef98a6e70153a2de9b4d63b5ba170e1c69e00d03.1635336263.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v3 14/15] diff: use designated initializers for emitted_diff_symbol","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:21Z","receivedAt":"2021-10-27T12:05:02Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nThis makes it clearer which fields are being explicitly initialized\nand will simplify the next commit where we add a new field to the\nstruct.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/diff.c b/diff.c\nindex 2085c063675..9ef88d7665a 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1497,7 +1497,9 @@ static void emit_diff_symbol_from_struct(struct diff_options *o,\n static void emit_diff_symbol(struct diff_options *o, enum diff_symbol s,\n \t\t\t const char *line, int len, unsigned flags)\n {\n-\tstruct emitted_diff_symbol e = {line, len, flags, 0, 0, s};\n+\tstruct emitted_diff_symbol e = {\n+\t\t.line = line, .len = len, .flags = flags, .s = s\n+\t};\n \n \tif (o->emitted_symbols)\n \t\tappend_emitted_diff_symbol(o, &e);\n-- \ngitgitgadget\n\n"},{"id":"439769","messageId":"ae78c05f08dfe0ce8d2870fdabeafceecbf6784c.1635336263.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v3 15/15] diff --color-moved: intern strings","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T12:04:22Z","receivedAt":"2021-10-27T12:05:03Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nTaking inspiration from xdl_classify_record() assign an id to each\naddition and deletion such that lines that match for the current\n--color-moved-ws mode share the same unique id. This reduces the\nnumber of hash lookups a little (calculating the ids still involves\none hash lookup per line) but the main benefit is that when growing\nblocks of potentially moved lines we can replace string comparisons\nwhich involve chasing a pointer with a simple integer comparison. On a\nlarge diff this commit reduces the time to run 'diff --color-moved' by\n37% compared to the previous commit and 31% compared to master, for\n'diff --color-moved-ws=allow-indentation-change' the reduction is 28%\ncompared to the previous commit and 96% compared to master. There is\nlittle change in the performance of 'git log --patch' as the diffs are\nsmaller.\n\nTest HEAD^ HEAD\n---------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38(0.33+0.05) 0.38(0.33+0.05) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.88(0.81+0.06) 0.55(0.50+0.04) -37.5%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 0.85(0.79+0.06) 0.61(0.54+0.06) -28.2%\n4002.4: log --no-color-moved --no-color-moved-ws 1.16(1.07+0.08) 1.15(1.09+0.05) -0.9%\n4002.5: log --color-moved --no-color-moved-ws 1.31(1.22+0.08) 1.29(1.19+0.09) -1.5%\n4002.6: log --color-moved-ws=allow-indentation-change 1.32(1.24+0.08) 1.31(1.18+0.13) -0.8%\n\nTest master HEAD\n---------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.33+0.05) 0.38(0.33+0.05) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.80 (0.75+0.04) 0.55(0.50+0.04) -31.2%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 14.20(14.15+0.05) 0.61(0.54+0.06) -95.7%\n4002.4: log --no-color-moved --no-color-moved-ws 1.15 (1.05+0.09) 1.15(1.09+0.05) +0.0%\n4002.5: log --color-moved --no-color-moved-ws 1.30 (1.19+0.11) 1.29(1.19+0.09) -0.8%\n4002.6: log --color-moved-ws=allow-indentation-change 1.70 (1.63+0.06) 1.31(1.18+0.13) -22.9%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 174 +++++++++++++++++++++++++++++++--------------------------\n 1 file changed, 96 insertions(+), 78 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 9ef88d7665a..c28c56c1283 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -18,6 +18,7 @@\n #include \"submodule-config.h\"\n #include \"submodule.h\"\n #include \"hashmap.h\"\n+#include \"mem-pool.h\"\n #include \"ll-merge.h\"\n #include \"string-list.h\"\n #include \"strvec.h\"\n@@ -772,6 +773,7 @@ struct emitted_diff_symbol {\n \tint flags;\n \tint indent_off; /* Offset to first non-whitespace character */\n \tint indent_width; /* The visual width of the indentation */\n+\tunsigned id;\n \tenum diff_symbol s;\n };\n #define EMITTED_DIFF_SYMBOL_INIT {NULL}\n@@ -797,9 +799,9 @@ static void append_emitted_diff_symbol(struct diff_options *o,\n }\n \n struct moved_entry {\n-\tstruct hashmap_entry ent;\n \tconst struct emitted_diff_symbol *es;\n \tstruct moved_entry *next_line;\n+\tstruct moved_entry *next_match;\n };\n \n struct moved_block {\n@@ -865,24 +867,24 @@ static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n \t\t\t\t const struct emitted_diff_symbol *l,\n \t\t\t\t struct moved_block *pmb)\n {\n-\tint al = cur->es->len, bl = l->len;\n-\tconst char *a = cur->es->line,\n-\t\t *b = l->line;\n-\tint a_off = cur->es->indent_off,\n-\t a_width = cur->es->indent_width,\n-\t b_off = l->indent_off,\n-\t b_width = l->indent_width;\n+\tint a_width = cur->es->indent_width, b_width = l->indent_width;\n \tint delta;\n \n-\t/* If 'l' and 'cur' are both blank then they match. */\n-\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n+\t/* The text of each line must match */\n+\tif (cur->es->id != l->id)\n+\t\treturn 1;\n+\n+\t/*\n+\t * If 'l' and 'cur' are both blank then we don't need to check the\n+\t * indent. We only need to check cur as we know the strings match.\n+\t * */\n+\tif (a_width == INDENT_BLANKLINE)\n \t\treturn 0;\n \n \t/*\n \t * The indent changes of the block are known and stored in pmb->wsd;\n \t * however we need to check if the indent changes of the current line\n-\t * match those of the current block and that the text of 'l' and 'cur'\n-\t * after the indentation match.\n+\t * match those of the current block.\n \t */\n \tdelta = b_width - a_width;\n \n@@ -893,22 +895,26 @@ static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n \tif (pmb->wsd == INDENT_BLANKLINE)\n \t\tpmb->wsd = delta;\n \n-\treturn !(delta == pmb->wsd && al - a_off == bl - b_off &&\n-\t\t !memcmp(a + a_off, b + b_off, al - a_off));\n+\treturn delta != pmb->wsd;\n }\n \n-static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n-\t\t\t const struct hashmap_entry *eptr,\n-\t\t\t const struct hashmap_entry *entry_or_key,\n-\t\t\t const void *keydata)\n+struct interned_diff_symbol {\n+\tstruct hashmap_entry ent;\n+\tstruct emitted_diff_symbol *es;\n+};\n+\n+static int interned_diff_symbol_cmp(const void *hashmap_cmp_fn_data,\n+\t\t\t\t const struct hashmap_entry *eptr,\n+\t\t\t\t const struct hashmap_entry *entry_or_key,\n+\t\t\t\t const void *keydata)\n {\n \tconst struct diff_options *diffopt = hashmap_cmp_fn_data;\n \tconst struct emitted_diff_symbol *a, *b;\n \tunsigned flags = diffopt->color_moved_ws_handling\n \t\t\t & XDF_WHITESPACE_FLAGS;\n \n-\ta = container_of(eptr, const struct moved_entry, ent)->es;\n-\tb = container_of(entry_or_key, const struct moved_entry, ent)->es;\n+\ta = container_of(eptr, const struct interned_diff_symbol, ent)->es;\n+\tb = container_of(entry_or_key, const struct interned_diff_symbol, ent)->es;\n \n \treturn !xdiff_compare_lines(a->line + a->indent_off,\n \t\t\t\t a->len - a->indent_off,\n@@ -916,55 +922,81 @@ static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n \t\t\t\t b->len - b->indent_off, flags);\n }\n \n-static struct moved_entry *prepare_entry(struct diff_options *o,\n-\t\t\t\t\t int line_no)\n+static void prepare_entry(struct diff_options *o, struct emitted_diff_symbol *l,\n+\t\t\t struct interned_diff_symbol *s)\n {\n-\tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n-\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[line_no];\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \tunsigned int hash = xdiff_hash_string(l->line + l->indent_off,\n \t\t\t\t\t l->len - l->indent_off, flags);\n \n-\thashmap_entry_init(&ret->ent, hash);\n-\tret->es = l;\n-\tret->next_line = NULL;\n-\n-\treturn ret;\n+\thashmap_entry_init(&s->ent, hash);\n+\ts->es = l;\n }\n \n-static void add_lines_to_move_detection(struct diff_options *o,\n-\t\t\t\t\tstruct hashmap *add_lines,\n-\t\t\t\t\tstruct hashmap *del_lines)\n+struct moved_entry_list {\n+\tstruct moved_entry *add, *del;\n+};\n+\n+static struct moved_entry_list *add_lines_to_move_detection(struct diff_options *o,\n+\t\t\t\t\t\t\t struct mem_pool *entry_mem_pool)\n {\n \tstruct moved_entry *prev_line = NULL;\n-\n+\tstruct mem_pool interned_pool;\n+\tstruct hashmap interned_map;\n+\tstruct moved_entry_list *entry_list = NULL;\n+\tsize_t entry_list_alloc = 0;\n+\tunsigned id = 0;\n \tint n;\n+\n+\thashmap_init(&interned_map, interned_diff_symbol_cmp, o, 8096);\n+\tmem_pool_init(&interned_pool, 1024 * 1024);\n+\n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n-\t\tstruct hashmap *hm;\n-\t\tstruct moved_entry *key;\n+\t\tstruct interned_diff_symbol key;\n+\t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n+\t\tstruct interned_diff_symbol *s;\n+\t\tstruct moved_entry *entry;\n \n-\t\tswitch (o->emitted_symbols->buf[n].s) {\n-\t\tcase DIFF_SYMBOL_PLUS:\n-\t\t\thm = add_lines;\n-\t\t\tbreak;\n-\t\tcase DIFF_SYMBOL_MINUS:\n-\t\t\thm = del_lines;\n-\t\t\tbreak;\n-\t\tdefault:\n+\t\tif (l->s != DIFF_SYMBOL_PLUS && l->s != DIFF_SYMBOL_MINUS) {\n \t\t\tprev_line = NULL;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (o->color_moved_ws_handling &\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\tfill_es_indent_data(&o->emitted_symbols->buf[n]);\n-\t\tkey = prepare_entry(o, n);\n-\t\tif (prev_line && prev_line->es->s == o->emitted_symbols->buf[n].s)\n-\t\t\tprev_line->next_line = key;\n+\t\t\tfill_es_indent_data(l);\n \n-\t\thashmap_add(hm, &key->ent);\n-\t\tprev_line = key;\n+\t\tprepare_entry(o, l, &key);\n+\t\ts = hashmap_get_entry(&interned_map, &key, ent, &key.ent);\n+\t\tif (s) {\n+\t\t\tl->id = s->es->id;\n+\t\t} else {\n+\t\t\tl->id = id;\n+\t\t\tALLOC_GROW_BY(entry_list, id, 1, entry_list_alloc);\n+\t\t\thashmap_add(&interned_map,\n+\t\t\t\t memcpy(mem_pool_alloc(&interned_pool,\n+\t\t\t\t\t\t\t sizeof(key)),\n+\t\t\t\t\t &key, sizeof(key)));\n+\t\t}\n+\t\tentry = mem_pool_alloc(entry_mem_pool, sizeof(*entry));\n+\t\tentry->es = l;\n+\t\tentry->next_line = NULL;\n+\t\tif (prev_line && prev_line->es->s == l->s)\n+\t\t\tprev_line->next_line = entry;\n+\t\tprev_line = entry;\n+\t\tif (l->s == DIFF_SYMBOL_PLUS) {\n+\t\t\tentry->next_match = entry_list[l->id].add;\n+\t\t\tentry_list[l->id].add = entry;\n+\t\t} else {\n+\t\t\tentry->next_match = entry_list[l->id].del;\n+\t\t\tentry_list[l->id].del = entry;\n+\t\t}\n \t}\n+\n+\thashmap_clear(&interned_map);\n+\tmem_pool_discard(&interned_pool, 0);\n+\n+\treturn entry_list;\n }\n \n static void pmb_advance_or_null(struct diff_options *o,\n@@ -973,7 +1005,6 @@ static void pmb_advance_or_null(struct diff_options *o,\n \t\t\t\tint *pmb_nr)\n {\n \tint i, j;\n-\tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \n \tfor (i = 0, j = 0; i < *pmb_nr; i++) {\n \t\tint match;\n@@ -986,9 +1017,8 @@ static void pmb_advance_or_null(struct diff_options *o,\n \t\t\tmatch = cur &&\n \t\t\t\t!cmp_in_block_with_wsd(cur, l, &pmb[i]);\n \t\telse\n-\t\t\tmatch = cur &&\n-\t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n-\t\t\t\t\t\t l->line, l->len, flags);\n+\t\t\tmatch = cur && cur->es->id == l->id;\n+\n \t\tif (match) {\n \t\t\tpmb[j] = pmb[i];\n \t\t\tpmb[j++].match = cur;\n@@ -998,7 +1028,6 @@ static void pmb_advance_or_null(struct diff_options *o,\n }\n \n static void fill_potential_moved_blocks(struct diff_options *o,\n-\t\t\t\t\tstruct hashmap *hm,\n \t\t\t\t\tstruct moved_entry *match,\n \t\t\t\t\tstruct emitted_diff_symbol *l,\n \t\t\t\t\tstruct moved_block **pmb_p,\n@@ -1012,7 +1041,7 @@ static void fill_potential_moved_blocks(struct diff_options *o,\n \t * The current line is the start of a new block.\n \t * Setup the set of potential blocks.\n \t */\n-\thashmap_for_each_entry_from(hm, match, ent) {\n+\tfor (; match; match = match->next_match) {\n \t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n \t\tif (o->color_moved_ws_handling &\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n@@ -1067,8 +1096,7 @@ static int adjust_last_block(struct diff_options *o, int n, int block_length)\n \n /* Find blocks of moved code, delegate actual coloring decision to helper */\n static void mark_color_as_moved(struct diff_options *o,\n-\t\t\t\tstruct hashmap *add_lines,\n-\t\t\t\tstruct hashmap *del_lines)\n+\t\t\t\tstruct moved_entry_list *entry_list)\n {\n \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n@@ -1077,23 +1105,15 @@ static void mark_color_as_moved(struct diff_options *o,\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n-\t\tstruct hashmap *hm = NULL;\n-\t\tstruct moved_entry *key;\n \t\tstruct moved_entry *match = NULL;\n \t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n \n \t\tswitch (l->s) {\n \t\tcase DIFF_SYMBOL_PLUS:\n-\t\t\thm = del_lines;\n-\t\t\tkey = prepare_entry(o, n);\n-\t\t\tmatch = hashmap_get_entry(hm, key, ent, NULL);\n-\t\t\tfree(key);\n+\t\t\tmatch = entry_list[l->id].del;\n \t\t\tbreak;\n \t\tcase DIFF_SYMBOL_MINUS:\n-\t\t\thm = add_lines;\n-\t\t\tkey = prepare_entry(o, n);\n-\t\t\tmatch = hashmap_get_entry(hm, key, ent, NULL);\n-\t\t\tfree(key);\n+\t\t\tmatch = entry_list[l->id].add;\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tflipped_block = 0;\n@@ -1135,7 +1155,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t\t */\n \t\t\t\tn -= block_length;\n \t\t\telse\n-\t\t\t\tfill_potential_moved_blocks(o, hm, match, l,\n+\t\t\t\tfill_potential_moved_blocks(o, match, l,\n \t\t\t\t\t\t\t &pmb, &pmb_alloc,\n \t\t\t\t\t\t\t &pmb_nr);\n \n@@ -6253,20 +6273,18 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \n \tif (o->emitted_symbols) {\n \t\tif (o->color_moved) {\n-\t\t\tstruct hashmap add_lines, del_lines;\n-\n-\t\t\thashmap_init(&del_lines, moved_entry_cmp, o, 0);\n-\t\t\thashmap_init(&add_lines, moved_entry_cmp, o, 0);\n+\t\t\tstruct mem_pool entry_pool;\n+\t\t\tstruct moved_entry_list *entry_list;\n \n-\t\t\tadd_lines_to_move_detection(o, &add_lines, &del_lines);\n-\t\t\tmark_color_as_moved(o, &add_lines, &del_lines);\n+\t\t\tmem_pool_init(&entry_pool, 1024 * 1024);\n+\t\t\tentry_list = add_lines_to_move_detection(o,\n+\t\t\t\t\t\t\t\t &entry_pool);\n+\t\t\tmark_color_as_moved(o, entry_list);\n \t\t\tif (o->color_moved == COLOR_MOVED_ZEBRA_DIM)\n \t\t\t\tdim_moved_lines(o);\n \n-\t\t\thashmap_clear_and_free(&add_lines, struct moved_entry,\n-\t\t\t\t\t\tent);\n-\t\t\thashmap_clear_and_free(&del_lines, struct moved_entry,\n-\t\t\t\t\t\tent);\n+\t\t\tmem_pool_discard(&entry_pool, 0);\n+\t\t\tfree(entry_list);\n \t\t}\n \n \t\tfor (i = 0; i < esm.nr; i++)\n-- \ngitgitgadget\n"},{"id":"439775","messageId":"d359c8a6-412f-98eb-8509-15267cdb0988@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/15] diff --color-moved[-ws] speedups","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-10-27T13:28:24Z","receivedAt":"2021-10-27T13:28:30Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 27/10/2021 13:04, Phillip Wood via GitGitGadget wrote:\n > [...]\n> * Patches 1-3 are new and fix an existing bug.\n\nSorry that should be \"Patches 2-4\"\n\n>[...]\n> The bug fix in patch 3 degrades the performance, but by the end of the\n> series the timings are the same as V2 - see the range diff.\n\nIt is patch 4 that degrades the performance, not patch 3\n\nBest Wishes\n\nPhillip\n"},{"id":"439950","messageId":"xmqqsfwkq1h4.fsf@gitster.g","threadId":"55926","inReplyTo":"8fc8914a37b3c343cd92bb0255088f7b000ff7f7.1635336262.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 01/15] diff --color-moved: add perf tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-28T21:32:55Z","receivedAt":"2021-10-28T21:33:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Phillip Wood via GitGitGadget\" writes:\n\n> From: Phillip Wood \n>\n> Add some tests so we can monitor changes to the performance of the\n> move detection code. The tests record the performance of a single\n> large diff and a sequence of smaller diffs.\n\n\"A single large diff\" meaning...?\n\n> +if ! git rev-parse --verify v2.29.0^{commit} >/dev/null\n> +then\n> +\tskip_all='skipping because tag v2.29.0 was not found'\n> +\ttest_done\n> +fi\n\nHmph. So this is designed only to be run in a clone of git.git with\nthat tag (and a bit of history, at least to v2.28.0 and 1000 commits)?\n\nI am asking primarily because this seems to be the first instance of\na test that hardcodes the dependency on our history, instead of\nallowing the tester to use their favourite history by using the\nGIT_PERF_LARGE_REPO and GIT_PERF_REPO environment variables.\n\nThe intention of the tests themselves looks quite clear. Thanks.\n\n> +GIT_PAGER_IN_USE=1\n> +test_export GIT_PAGER_IN_USE\n> +\n> +test_perf 'diff --no-color-moved --no-color-moved-ws large change' '\n> +\tgit diff --no-color-moved --no-color-moved-ws v2.28.0 v2.29.0\n> +'\n> +\n> +test_perf 'diff --color-moved --no-color-moved-ws large change' '\n> +\tgit diff --color-moved=zebra --no-color-moved-ws v2.28.0 v2.29.0\n> +'\n> +\n> +test_perf 'diff --color-moved-ws=allow-indentation-change large change' '\n> +\tgit diff --color-moved=zebra --color-moved-ws=allow-indentation-change \\\n> +\t\tv2.28.0 v2.29.0\n> +'\n> +\n> +test_perf 'log --no-color-moved --no-color-moved-ws' '\n> +\tgit log --no-color-moved --no-color-moved-ws --no-merges --patch \\\n> +\t\t-n1000 v2.29.0\n> +'\n> +\n> +test_perf 'log --color-moved --no-color-moved-ws' '\n> +\tgit log --color-moved=zebra --no-color-moved-ws --no-merges --patch \\\n> +\t\t-n1000 v2.29.0\n> +'\n> +\n> +test_perf 'log --color-moved-ws=allow-indentation-change' '\n> +\tgit log --color-moved=zebra --color-moved-ws=allow-indentation-change \\\n> +\t\t--no-merges --patch -n1000 v2.29.0\n> +'\n> +\n> +test_done\n"},{"id":"439953","messageId":"xmqqmtmsq0lg.fsf@gitster.g","threadId":"55926","inReplyTo":"658aec2670c78f9753a5acccab20d3a1741403e6.1635336262.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 03/15] diff --color-moved: factor out function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-28T21:51:55Z","receivedAt":"2021-10-28T21:52:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Phillip Wood via GitGitGadget\" writes:\n\n> From: Phillip Wood \n>\n> This code is quite heavily indented and having it in its own function\n> simplifies an upcoming change.\n\nAnd this should show as \"moved\" lines correctly in the output from\n\"log -p --color-moved -w\"?\n\n... not really. There is an unfortunate artificial line wrapping in\nthe original, which was unwrapped by this move, so the blocks do not\nexactly match.\n\n> Signed-off-by: Phillip Wood \n> ---\n> diff.c | 51 ++++++++++++++++++++++++++++++++++-----------------\n> 1 file changed, 34 insertions(+), 17 deletions(-)\n>\n> diff --git a/diff.c b/diff.c\n> index bd8e4ec9757..09af94e018c 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -1098,6 +1098,38 @@ static int shrink_potential_moved_blocks(struct moved_block *pmb,\n> \treturn rp + 1;\n> }\n> \n> +static void fill_potential_moved_blocks(struct diff_options *o,\n> +\t\t\t\t\tstruct hashmap *hm,\n> +\t\t\t\t\tstruct moved_entry *match,\n> +\t\t\t\t\tstruct emitted_diff_symbol *l,\n> +\t\t\t\t\tstruct moved_block **pmb_p,\n> +\t\t\t\t\tint *pmb_alloc_p, int *pmb_nr_p)\n> +\n> +{\n> +\tstruct moved_block *pmb = *pmb_p;\n> +\tint pmb_alloc = *pmb_alloc_p, pmb_nr = *pmb_nr_p;\n> +\n> +\t/*\n> +\t * The current line is the start of a new block.\n> +\t * Setup the set of potential blocks.\n> +\t */\n> +\thashmap_for_each_entry_from(hm, match, ent) {\n> +\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n> +\t\tif (o->color_moved_ws_handling &\n> +\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n> +\t\t\tif (compute_ws_delta(l, match->es, &(pmb[pmb_nr]).wsd))\n> +\t\t\t\tpmb[pmb_nr++].match = match;\n> +\t\t} else {\n> +\t\t\tpmb[pmb_nr].wsd = 0;\n> +\t\t\tpmb[pmb_nr++].match = match;\n> +\t\t}\n> +\t}\n> +\n> +\t*pmb_p = pmb;\n> +\t*pmb_alloc_p = pmb_alloc;\n> +\t*pmb_nr_p = pmb_nr;\n> +}\n> +\n> /*\n> * If o->color_moved is COLOR_MOVED_PLAIN, this function does nothing.\n> *\n> @@ -1198,23 +1230,8 @@ static void mark_color_as_moved(struct diff_options *o,\n> \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n> \n> \t\tif (pmb_nr == 0) {\n> -\t\t\t/*\n> -\t\t\t * The current line is the start of a new block.\n> -\t\t\t * Setup the set of potential blocks.\n> -\t\t\t */\n> -\t\t\thashmap_for_each_entry_from(hm, match, ent) {\n> -\t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n> -\t\t\t\tif (o->color_moved_ws_handling &\n> -\t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n> -\t\t\t\t\tif (compute_ws_delta(l, match->es,\n> -\t\t\t\t\t\t\t &pmb[pmb_nr].wsd))\n> -\t\t\t\t\t\tpmb[pmb_nr++].match = match;\n> -\t\t\t\t} else {\n> -\t\t\t\t\tpmb[pmb_nr].wsd = 0;\n> -\t\t\t\t\tpmb[pmb_nr++].match = match;\n> -\t\t\t\t}\n> -\t\t\t}\n> -\n> +\t\t\tfill_potential_moved_blocks(\n> +\t\t\t\to, hm, match, l, &pmb, &pmb_alloc, &pmb_nr);\n> \t\t\tif (adjust_last_block(o, n, block_length) &&\n> \t\t\t pmb_nr && last_symbol != l->s)\n> \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n"},{"id":"439990","messageId":"b6f57fc3-75d9-d7d5-7153-28dde066a101@gmail.com","threadId":"55926","inReplyTo":"xmqqsfwkq1h4.fsf@gitster.g","subject":"Re: [PATCH v3 01/15] diff --color-moved: add perf tests","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-10-29T10:24:00Z","receivedAt":"2021-10-29T10:24:05Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Junio\n\nOn 28/10/2021 22:32, Junio C Hamano wrote:\n> \"Phillip Wood via GitGitGadget\" writes:\n> \n>> From: Phillip Wood \n>>\n>> Add some tests so we can monitor changes to the performance of the\n>> move detection code. The tests record the performance of a single\n>> large diff and a sequence of smaller diffs.\n> \n> \"A single large diff\" meaning...?\n\nThe diff of two commits that are far apart in the history so have lots \nof changes between them\n\n>> +if ! git rev-parse --verify v2.29.0^{commit} >/dev/null\n>> +then\n>> +\tskip_all='skipping because tag v2.29.0 was not found'\n>> +\ttest_done\n>> +fi\n> \n> Hmph. So this is designed only to be run in a clone of git.git with\n> that tag (and a bit of history, at least to v2.28.0 and 1000 commits)?\n> \n> I am asking primarily because this seems to be the first instance of\n> a test that hardcodes the dependency on our history, instead of\n> allowing the tester to use their favourite history by using the\n> GIT_PERF_LARGE_REPO and GIT_PERF_REPO environment variables.\n\np3404-rebase-interactive does the same thing. The aim is to have a \nrepeatable test rather than just using whatever commit HEAD happens to \nbe pointing at when the test is run as the starting point, if you have \nany ideas for doing that another way I'm happy to change it.\n\n> The intention of the tests themselves looks quite clear. Thanks.\n\nThanks\n\nPhillip\n\n>> +GIT_PAGER_IN_USE=1\n>> +test_export GIT_PAGER_IN_USE\n>> +\n>> +test_perf 'diff --no-color-moved --no-color-moved-ws large change' '\n>> +\tgit diff --no-color-moved --no-color-moved-ws v2.28.0 v2.29.0\n>> +'\n>> +\n>> +test_perf 'diff --color-moved --no-color-moved-ws large change' '\n>> +\tgit diff --color-moved=zebra --no-color-moved-ws v2.28.0 v2.29.0\n>> +'\n>> +\n>> +test_perf 'diff --color-moved-ws=allow-indentation-change large change' '\n>> +\tgit diff --color-moved=zebra --color-moved-ws=allow-indentation-change \\\n>> +\t\tv2.28.0 v2.29.0\n>> +'\n>> +\n>> +test_perf 'log --no-color-moved --no-color-moved-ws' '\n>> +\tgit log --no-color-moved --no-color-moved-ws --no-merges --patch \\\n>> +\t\t-n1000 v2.29.0\n>> +'\n>> +\n>> +test_perf 'log --color-moved --no-color-moved-ws' '\n>> +\tgit log --color-moved=zebra --no-color-moved-ws --no-merges --patch \\\n>> +\t\t-n1000 v2.29.0\n>> +'\n>> +\n>> +test_perf 'log --color-moved-ws=allow-indentation-change' '\n>> +\tgit log --color-moved=zebra --color-moved-ws=allow-indentation-change \\\n>> +\t\t--no-merges --patch -n1000 v2.29.0\n>> +'\n>> +\n>> +test_done\n\n"},{"id":"439991","messageId":"e911836c-1212-5d89-5e2a-78cc6c8f81be@gmail.com","threadId":"55926","inReplyTo":"xmqqmtmsq0lg.fsf@gitster.g","subject":"Re: [PATCH v3 03/15] diff --color-moved: factor out function","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-10-29T10:35:37Z","receivedAt":"2021-10-29T10:35:41Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Junio\n\nOn 28/10/2021 22:51, Junio C Hamano wrote:\n> \"Phillip Wood via GitGitGadget\" writes:\n> \n>> From: Phillip Wood \n>>\n>> This code is quite heavily indented and having it in its own function\n>> simplifies an upcoming change.\n> \n> And this should show as \"moved\" lines correctly in the output from\n> \"log -p --color-moved -w\"?\n\nrather that \"-w\" one can use \"--color-moved-ws=allow-indentation-change\" \nto highlight moved lines where the indentation has changed. It took me a \nwhile to realize why \"-w\" does not do anything here but it is because \nthe lines are moved as well as having their indentation changed.\n\n> ... not really. There is an unfortunate artificial line wrapping in\n> the original, which was unwrapped by this move, so the blocks do not\n> exactly match.\n\nYes that's a shame, it seemed overkill to have one commit moving the \ncode as is and then another reformatting it. All of the moved lines \napart from the one that is unwrapped are highlighted with \n\"--color-moved-ws=allow-indentation-change\".\n\nBest Wishes\n\nPhillip\n\n>> Signed-off-by: Phillip Wood \n>> ---\n>> diff.c | 51 ++++++++++++++++++++++++++++++++++-----------------\n>> 1 file changed, 34 insertions(+), 17 deletions(-)\n>>\n>> diff --git a/diff.c b/diff.c\n>> index bd8e4ec9757..09af94e018c 100644\n>> --- a/diff.c\n>> +++ b/diff.c\n>> @@ -1098,6 +1098,38 @@ static int shrink_potential_moved_blocks(struct moved_block *pmb,\n>> \treturn rp + 1;\n>> }\n>> \n>> +static void fill_potential_moved_blocks(struct diff_options *o,\n>> +\t\t\t\t\tstruct hashmap *hm,\n>> +\t\t\t\t\tstruct moved_entry *match,\n>> +\t\t\t\t\tstruct emitted_diff_symbol *l,\n>> +\t\t\t\t\tstruct moved_block **pmb_p,\n>> +\t\t\t\t\tint *pmb_alloc_p, int *pmb_nr_p)\n>> +\n>> +{\n>> +\tstruct moved_block *pmb = *pmb_p;\n>> +\tint pmb_alloc = *pmb_alloc_p, pmb_nr = *pmb_nr_p;\n>> +\n>> +\t/*\n>> +\t * The current line is the start of a new block.\n>> +\t * Setup the set of potential blocks.\n>> +\t */\n>> +\thashmap_for_each_entry_from(hm, match, ent) {\n>> +\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n>> +\t\tif (o->color_moved_ws_handling &\n>> +\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n>> +\t\t\tif (compute_ws_delta(l, match->es, &(pmb[pmb_nr]).wsd))\n>> +\t\t\t\tpmb[pmb_nr++].match = match;\n>> +\t\t} else {\n>> +\t\t\tpmb[pmb_nr].wsd = 0;\n>> +\t\t\tpmb[pmb_nr++].match = match;\n>> +\t\t}\n>> +\t}\n>> +\n>> +\t*pmb_p = pmb;\n>> +\t*pmb_alloc_p = pmb_alloc;\n>> +\t*pmb_nr_p = pmb_nr;\n>> +}\n>> +\n>> /*\n>> * If o->color_moved is COLOR_MOVED_PLAIN, this function does nothing.\n>> *\n>> @@ -1198,23 +1230,8 @@ static void mark_color_as_moved(struct diff_options *o,\n>> \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n>> \n>> \t\tif (pmb_nr == 0) {\n>> -\t\t\t/*\n>> -\t\t\t * The current line is the start of a new block.\n>> -\t\t\t * Setup the set of potential blocks.\n>> -\t\t\t */\n>> -\t\t\thashmap_for_each_entry_from(hm, match, ent) {\n>> -\t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n>> -\t\t\t\tif (o->color_moved_ws_handling &\n>> -\t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n>> -\t\t\t\t\tif (compute_ws_delta(l, match->es,\n>> -\t\t\t\t\t\t\t &pmb[pmb_nr].wsd))\n>> -\t\t\t\t\t\tpmb[pmb_nr++].match = match;\n>> -\t\t\t\t} else {\n>> -\t\t\t\t\tpmb[pmb_nr].wsd = 0;\n>> -\t\t\t\t\tpmb[pmb_nr++].match = match;\n>> -\t\t\t\t}\n>> -\t\t\t}\n>> -\n>> +\t\t\tfill_potential_moved_blocks(\n>> +\t\t\t\to, hm, match, l, &pmb, &pmb_alloc, &pmb_nr);\n>> \t\t\tif (adjust_last_block(o, n, block_length) &&\n>> \t\t\t pmb_nr && last_symbol != l->s)\n>> \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n\n"},{"id":"439994","messageId":"211029.86zgqs3wpx.gmgdl@evledraar.gmail.com","threadId":"55926","inReplyTo":"b6f57fc3-75d9-d7d5-7153-28dde066a101@gmail.com","subject":"Re: [PATCH v3 01/15] diff --color-moved: add perf tests","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-29T11:06:15Z","receivedAt":"2021-10-29T11:18:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 29 2021, Phillip Wood wrote:\n\n> Hi Junio\n>\n> On 28/10/2021 22:32, Junio C Hamano wrote:\n>> \"Phillip Wood via GitGitGadget\" writes:\n>> \n>>> From: Phillip Wood \n>>>\n>>> Add some tests so we can monitor changes to the performance of the\n>>> move detection code. The tests record the performance of a single\n>>> large diff and a sequence of smaller diffs.\n>> \"A single large diff\" meaning...?\n>\n> The diff of two commits that are far apart in the history so have lots\n> of changes between them\n>\n>>> +if ! git rev-parse --verify v2.29.0^{commit} >/dev/null\n>>> +then\n>>> +\tskip_all='skipping because tag v2.29.0 was not found'\n>>> +\ttest_done\n>>> +fi\n>> Hmph. So this is designed only to be run in a clone of git.git with\n>> that tag (and a bit of history, at least to v2.28.0 and 1000 commits)?\n>> I am asking primarily because this seems to be the first instance of\n>> a test that hardcodes the dependency on our history, instead of\n>> allowing the tester to use their favourite history by using the\n>> GIT_PERF_LARGE_REPO and GIT_PERF_REPO environment variables.\n>\n> p3404-rebase-interactive does the same thing. The aim is to have a\n> repeatable test rather than just using whatever commit HEAD happens to \n> be pointing at when the test is run as the starting point, if you have\n> any ideas for doing that another way I'm happy to change it.\n\nI don't know if it's worth it here, but the following would work:\n\n 1. List all tags in the repository, sorted in reverse order, so e.g.:\n\n git tag -l 'v*.0' --sort=version:refname\n\n (The glob can be configurable as an env variable, or we could fall\n back)\n\n 2. Go down that list and find the first pair that matches some limit, I\n think say the first \"major\" release with 500 commits would qualify\n\n 3. Make it a GIT_PERF_LARGE_REPO test\n\nWe've got some perf tests that do similar things. I think you'd find\nthat with something like this you should able to hand the perf test a\npath to git.git, or linux.git, and probably any \"major\" repository\" as\nlong as it follows a common \"we tag our releases at some interval\"\npattern.\n\nOr perhaps more simply:\n\n 1. Note the number of commits in the history, per \"git rev-list HEAD |\n wc -l\" 2.\n\n 2. Then round that down to the nearest 10^x, so for a 250k commit\n repository round down to 100k and diff say the 90k..100kth commits,\n for git.git which has 60k that would be 10k, and the diff is commits\n 9k..10k..\n\nIt means you'll get a \"bump\" eventually when say git.git crosses 100k\ncommits, but it will prorably be stable for any measurement anyone cares\nto do, and means that you can get \"realistic\" measurements for diffing a\nbig chuck on of history from anything from a tiny repository with >=10\ncommits, to something truly gargantuan where you'd end up diffing say\n900k..1m.\n \n\n"},{"id":"440824","messageId":"33c63811-b048-8227-6148-16d5bd908564@gmail.com","threadId":"55926","inReplyTo":"211029.86zgqs3wpx.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 01/15] diff --color-moved: add perf tests","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-11-10T11:05:25Z","receivedAt":"2021-11-10T11:05:42Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 29/10/2021 12:06, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Oct 29 2021, Phillip Wood wrote:\n> \n>> Hi Junio\n>>\n>> On 28/10/2021 22:32, Junio C Hamano wrote:\n>>> \"Phillip Wood via GitGitGadget\" writes:\n>>>\n>>>> From: Phillip Wood \n>>>>\n>>>> Add some tests so we can monitor changes to the performance of the\n>>>> move detection code. The tests record the performance of a single\n>>>> large diff and a sequence of smaller diffs.\n>>> \"A single large diff\" meaning...?\n>>\n>> The diff of two commits that are far apart in the history so have lots\n>> of changes between them\n>>\n>>>> +if ! git rev-parse --verify v2.29.0^{commit} >/dev/null\n>>>> +then\n>>>> +\tskip_all='skipping because tag v2.29.0 was not found'\n>>>> +\ttest_done\n>>>> +fi\n>>> Hmph. So this is designed only to be run in a clone of git.git with\n>>> that tag (and a bit of history, at least to v2.28.0 and 1000 commits)?\n>>> I am asking primarily because this seems to be the first instance of\n>>> a test that hardcodes the dependency on our history, instead of\n>>> allowing the tester to use their favourite history by using the\n>>> GIT_PERF_LARGE_REPO and GIT_PERF_REPO environment variables.\n>>\n>> p3404-rebase-interactive does the same thing. The aim is to have a\n>> repeatable test rather than just using whatever commit HEAD happens to\n>> be pointing at when the test is run as the starting point, if you have\n>> any ideas for doing that another way I'm happy to change it.\n> \n> I don't know if it's worth it here, but the following would work:\n> \n> 1. List all tags in the repository, sorted in reverse order, so e.g.:\n> \n> git tag -l 'v*.0' --sort=version:refname\n> \n> (The glob can be configurable as an env variable, or we could fall\n> back)\n> \n> 2. Go down that list and find the first pair that matches some limit, I\n> think say the first \"major\" release with 500 commits would qualify\n> \n> 3. Make it a GIT_PERF_LARGE_REPO test\n> \n> We've got some perf tests that do similar things. I think you'd find\n> that with something like this you should able to hand the perf test a\n> path to git.git, or linux.git, and probably any \"major\" repository\" as\n> long as it follows a common \"we tag our releases at some interval\"\n> pattern.\n> \n> Or perhaps more simply:\n> \n> 1. Note the number of commits in the history, per \"git rev-list HEAD |\n> wc -l\" 2.\n> \n> 2. Then round that down to the nearest 10^x, so for a 250k commit\n> repository round down to 100k and diff say the 90k..100kth commits,\n> for git.git which has 60k that would be 10k, and the diff is commits\n> 9k..10k..\n> \n> It means you'll get a \"bump\" eventually when say git.git crosses 100k\n> commits, but it will prorably be stable for any measurement anyone cares\n> to do, and means that you can get \"realistic\" measurements for diffing a\n> big chuck on of history from anything from a tiny repository with >=10\n> commits, to something truly gargantuan where you'd end up diffing say\n> 900k..1m.\n\nThanks for the suggestions, I was quite tempted by the second idea, but \nin the end I couldn't face rerunning the pref tests and updating all the \ncommit messages again. I've added a couple of environment variables to \nallow the revs in the diff commands to be customized.\n\nBest Wishes\n\nPhillip\n\n\n"},{"id":"441271","messageId":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v3.git.1635336262.gitgitgadget@gmail.com","subject":"[PATCH v4 00/15] diff --color-moved[-ws] speedups","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:23Z","receivedAt":"2021-11-16T09:49:50Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Changes since V3:\n\n * Patch 1 now allows the user to choose different endpoints for the diff\n perf tests to facilitate testing with different repositories.\n * Fixed the alignment of the perf results column headers in a couple of\n patches.\n\nChanges since V2:\n\n * Patches 1-3 are new and fix an existing bug.\n * Patch 8 includes Peff's unused parameter fix.\n * Patch 11 has been updated to fix a bug fix in V2.\n * Patch 13 has an expanded commit message explaining a change in behavior\n for lines starting with a form-feed.\n * Updated benchmark results.\n\nThe bug fix in patch 3 degrades the performance, but by the end of the\nseries the timings are the same as V2 - see the range diff.\n\nV2 Cover Letter: Thanks to Ævar and Elijah for their comments, I've reworded\nthe commit messages, addressed the enum initialization issue in patch 2 (now\n3) and added some perf tests.\n\nThere are two new patches in this round. The first patch is new and adds the\nperf tests suggested by Ævar, the penultimate patch is also new and coverts\nthe existing code to use a designated initializer.\n\nI've converted the benchmark results in the commit messages to use the new\ntests, the percentage changes are broadly similar to the previous results\nthough I ended up running them on a different computer this time.\n\nV1 cover letter:\n\nThe current implementation of diff --color-moved-ws=allow-indentation-change\nis considerably slower that the implementation of diff --color-moved which\nis in turn slower than a regular diff. This patch series starts with a\ncouple of bug fixes and then reworks the implementation of diff\n--color-moved and diff --color-moved-ws=allow-indentation-change to speed\nthem up on large diffs. The time to run git diff --color-moved\n--no-color-moved-ws v2.28.0 v2.29.0 is reduced by 33% and the time to run\ngit diff --color-moved --color-moved-ws=allow-indentation-change v2.28.0\nv2.29.0 is reduced by 88%. There is a small slowdown for commit sized diffs\nwith --color-moved - the time to run git log -p --color-moved\n--no-color-moved-ws --no-merges -n1000 v2.29.0 is increased by 2% on recent\nprocessors. On older processors these patches reduce the running time in all\ncases that I've tested. In general the larger the diff the larger the speed\nup. As an extreme example the time to run diff --color-moved\n--color-moved-ws=allow-indentation-change v2.25.0 v2.30.0 goes down from 8\nminutes to 6 seconds.\n\nPhillip Wood (15):\n diff --color-moved: add perf tests\n diff --color-moved: clear all flags on blocks that are too short\n diff --color-moved: factor out function\n diff --color-moved: rewind when discarding pmb\n diff --color-moved=zebra: fix alternate coloring\n diff --color-moved: avoid false short line matches and bad zerba\n coloring\n diff: simplify allow-indentation-change delta calculation\n diff --color-moved-ws=allow-indentation-change: simplify and optimize\n diff --color-moved: call comparison function directly\n diff --color-moved: unify moved block growth functions\n diff --color-moved: shrink potential moved blocks as we go\n diff --color-moved: stop clearing potential moved blocks\n diff --color-moved-ws=allow-indentation-change: improve hash lookups\n diff: use designated initializers for emitted_diff_symbol\n diff --color-moved: intern strings\n\n diff.c | 431 +++++++++++++------------------\n t/perf/p4002-diff-color-moved.sh | 57 ++++\n t/t4015-diff-whitespace.sh | 205 ++++++++++++++-\n 3 files changed, 437 insertions(+), 256 deletions(-)\n create mode 100755 t/perf/p4002-diff-color-moved.sh\n\n\nbase-commit: 211eca0895794362184da2be2a2d812d070719d3\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-981%2Fphillipwood%2Fwip%2Fdiff-color-moved-tweaks-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-981/phillipwood/wip/diff-color-moved-tweaks-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/981\n\nRange-diff vs v3:\n\n 1: 8fc8914a37b ! 1: 48ee03cf52a diff --color-moved: add perf tests\n @@ Commit message\n diff --color-moved: add perf tests\n \n Add some tests so we can monitor changes to the performance of the\n - move detection code. The tests record the performance of a single\n - large diff and a sequence of smaller diffs.\n + move detection code. The tests record the performance --color-moved\n + and --color-moved-ws=allow-indentation-change for a large diff and a\n + sequence of smaller diffs. The range of commits used for the large\n + diff can be customized by exporting TEST_REV_A and TEST_REV_B when\n + running the test.\n \n Signed-off-by: Phillip Wood \n \n @@ t/perf/p4002-diff-color-moved.sh (new)\n +\n +test_perf_default_repo\n +\n -+if ! git rev-parse --verify v2.29.0^{commit} >/dev/null\n ++# The endpoints of the diff can be customized by setting TEST_REV_A\n ++# and TEST_REV_B in the environment when running this test.\n ++\n ++rev=\"${TEST_REV_A:-v2.28.0}\"\n ++if ! rev_a=\"$(git rev-parse --quiet --verify \"$rev\")\"\n ++then\n ++\tskip_all=\"skipping because '$rev' was not found. \\\n ++\t\t Use TEST_REV_A and TEST_REV_B to set the revs to use\"\n ++\ttest_done\n ++fi\n ++rev=\"${TEST_REV_B:-v2.29.0}\"\n ++if ! rev_b=\"$(git rev-parse --quiet --verify \"$rev\")\"\n +then\n -+\tskip_all='skipping because tag v2.29.0 was not found'\n ++\tskip_all=\"skipping because '$rev' was not found. \\\n ++\t\t Use TEST_REV_A and TEST_REV_B to set the revs to use\"\n +\ttest_done\n +fi\n +\n +GIT_PAGER_IN_USE=1\n -+test_export GIT_PAGER_IN_USE\n ++test_export GIT_PAGER_IN_USE rev_a rev_b\n +\n +test_perf 'diff --no-color-moved --no-color-moved-ws large change' '\n -+\tgit diff --no-color-moved --no-color-moved-ws v2.28.0 v2.29.0\n ++\tgit diff --no-color-moved --no-color-moved-ws $rev_a $rev_b\n +'\n +\n +test_perf 'diff --color-moved --no-color-moved-ws large change' '\n -+\tgit diff --color-moved=zebra --no-color-moved-ws v2.28.0 v2.29.0\n ++\tgit diff --color-moved=zebra --no-color-moved-ws $rev_a $rev_b\n +'\n +\n +test_perf 'diff --color-moved-ws=allow-indentation-change large change' '\n +\tgit diff --color-moved=zebra --color-moved-ws=allow-indentation-change \\\n -+\t\tv2.28.0 v2.29.0\n ++\t\t$rev_a $rev_b\n +'\n +\n +test_perf 'log --no-color-moved --no-color-moved-ws' '\n +\tgit log --no-color-moved --no-color-moved-ws --no-merges --patch \\\n -+\t\t-n1000 v2.29.0\n ++\t\t-n1000 $rev_b\n +'\n +\n +test_perf 'log --color-moved --no-color-moved-ws' '\n +\tgit log --color-moved=zebra --no-color-moved-ws --no-merges --patch \\\n -+\t\t-n1000 v2.29.0\n ++\t\t-n1000 $rev_b\n +'\n +\n +test_perf 'log --color-moved-ws=allow-indentation-change' '\n +\tgit log --color-moved=zebra --color-moved-ws=allow-indentation-change \\\n -+\t\t--no-merges --patch -n1000 v2.29.0\n ++\t\t--no-merges --patch -n1000 $rev_b\n +'\n +\n +test_done\n 2: e9daed2360c = 2: 47c652716e8 diff --color-moved: clear all flags on blocks that are too short\n 3: 658aec2670c = 3: 99e38ba9de9 diff --color-moved: factor out function\n 4: a30f52d7f15 ! 4: 9ca71db61ae diff --color-moved: rewind when discarding pmb\n @@ Commit message\n mitigates the performance impact of this commit. After the\n optimization this commit has a negligible impact on performance.\n \n - Test HEAD^ HEAD\n - ------------------------------------------------------------------------------------------------------------------\n + Test HEAD^ HEAD\n + -----------------------------------------------------------------------------------------------------------------\n 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.33+0.05) 0.39 (0.34+0.04) +2.6%\n 4002.2: diff --color-moved --no-color-moved-ws large change 0.80 (0.76+0.03) 0.86 (0.82+0.04) +7.5%\n 4002.3: diff --color-moved-ws=allow-indentation-change large change 14.22(14.17+0.04) 19.01(18.93+0.05) +33.7%\n 5: 1dde206b7b1 = 5: 56bb69af36e diff --color-moved=zebra: fix alternate coloring\n 6: 2717ff500d2 = 6: 10b11526206 diff --color-moved: avoid false short line matches and bad zerba coloring\n 7: f96fa71d53c = 7: c2e7b347257 diff: simplify allow-indentation-change delta calculation\n 8: 324b689c915 ! 8: d7bbc0041e0 diff --color-moved-ws=allow-indentation-change: simplify and optimize\n @@ Commit message\n git diff --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n by 93% compared to master and simplifies the code.\n \n - Test HEAD^ HEAD\n + Test HEAD^ HEAD\n ---------------------------------------------------------------------------------------------------------------\n 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.35+0.03) 0.38(0.35+0.03) +0.0%\n 4002.2: diff --color-moved --no-color-moved-ws large change 0.86 (0.80+0.06) 0.87(0.83+0.04) +1.2%\n @@ Commit message\n 4002.5: log --color-moved --no-color-moved-ws 1.32 (1.25+0.07) 1.32(1.24+0.08) +0.0%\n 4002.6: log --color-moved-ws=allow-indentation-change 1.71 (1.64+0.06) 1.36(1.25+0.10) -20.5%\n \n - Test master HEAD\n + Test master HEAD\n ---------------------------------------------------------------------------------------------------------------\n 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.33+0.05) 0.38(0.35+0.03) +0.0%\n 4002.2: diff --color-moved --no-color-moved-ws large change 0.80 (0.75+0.04) 0.87(0.83+0.04) +8.7%\n 9: f142f33276a = 9: c3e5dce1910 diff --color-moved: call comparison function directly\n 10: 8f3ea865dd3 = 10: 9eb8cecd52a diff --color-moved: unify moved block growth functions\n 11: 078c04d4a66 = 11: 35e204e1578 diff --color-moved: shrink potential moved blocks as we go\n 12: 618371471a0 = 12: ec329e7946d diff --color-moved: stop clearing potential moved blocks\n 13: 6a8e9a2724d = 13: 6ec94134aaf diff --color-moved-ws=allow-indentation-change: improve hash lookups\n 14: ef98a6e7015 = 14: d44c5d734c3 diff: use designated initializers for emitted_diff_symbol\n 15: ae78c05f08d ! 15: 5177f669423 diff --color-moved: intern strings\n @@ Commit message\n little change in the performance of 'git log --patch' as the diffs are\n smaller.\n \n - Test HEAD^ HEAD\n + Test HEAD^ HEAD\n ---------------------------------------------------------------------------------------------------------------\n 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38(0.33+0.05) 0.38(0.33+0.05) +0.0%\n 4002.2: diff --color-moved --no-color-moved-ws large change 0.88(0.81+0.06) 0.55(0.50+0.04) -37.5%\n @@ Commit message\n 4002.5: log --color-moved --no-color-moved-ws 1.31(1.22+0.08) 1.29(1.19+0.09) -1.5%\n 4002.6: log --color-moved-ws=allow-indentation-change 1.32(1.24+0.08) 1.31(1.18+0.13) -0.8%\n \n - Test master HEAD\n + Test master HEAD\n ---------------------------------------------------------------------------------------------------------------\n 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.33+0.05) 0.38(0.33+0.05) +0.0%\n 4002.2: diff --color-moved --no-color-moved-ws large change 0.80 (0.75+0.04) 0.55(0.50+0.04) -31.2%\n\n-- \ngitgitgadget\n"},{"id":"441272","messageId":"48ee03cf52a619d22d7ee7b1958dd59872bf721f.1637056178.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v4 01/15] diff --color-moved: add perf tests","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:24Z","receivedAt":"2021-11-16T09:49:51Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nAdd some tests so we can monitor changes to the performance of the\nmove detection code. The tests record the performance --color-moved\nand --color-moved-ws=allow-indentation-change for a large diff and a\nsequence of smaller diffs. The range of commits used for the large\ndiff can be customized by exporting TEST_REV_A and TEST_REV_B when\nrunning the test.\n\nSigned-off-by: Phillip Wood \n---\n t/perf/p4002-diff-color-moved.sh | 57 ++++++++++++++++++++++++++++++++\n 1 file changed, 57 insertions(+)\n create mode 100755 t/perf/p4002-diff-color-moved.sh\n\ndiff --git a/t/perf/p4002-diff-color-moved.sh b/t/perf/p4002-diff-color-moved.sh\nnew file mode 100755\nindex 00000000000..ab2af931c04\n--- /dev/null\n+++ b/t/perf/p4002-diff-color-moved.sh\n@@ -0,0 +1,57 @@\n+#!/bin/sh\n+\n+test_description='Tests diff --color-moved performance'\n+. ./perf-lib.sh\n+\n+test_perf_default_repo\n+\n+# The endpoints of the diff can be customized by setting TEST_REV_A\n+# and TEST_REV_B in the environment when running this test.\n+\n+rev=\"${TEST_REV_A:-v2.28.0}\"\n+if ! rev_a=\"$(git rev-parse --quiet --verify \"$rev\")\"\n+then\n+\tskip_all=\"skipping because '$rev' was not found. \\\n+\t\t Use TEST_REV_A and TEST_REV_B to set the revs to use\"\n+\ttest_done\n+fi\n+rev=\"${TEST_REV_B:-v2.29.0}\"\n+if ! rev_b=\"$(git rev-parse --quiet --verify \"$rev\")\"\n+then\n+\tskip_all=\"skipping because '$rev' was not found. \\\n+\t\t Use TEST_REV_A and TEST_REV_B to set the revs to use\"\n+\ttest_done\n+fi\n+\n+GIT_PAGER_IN_USE=1\n+test_export GIT_PAGER_IN_USE rev_a rev_b\n+\n+test_perf 'diff --no-color-moved --no-color-moved-ws large change' '\n+\tgit diff --no-color-moved --no-color-moved-ws $rev_a $rev_b\n+'\n+\n+test_perf 'diff --color-moved --no-color-moved-ws large change' '\n+\tgit diff --color-moved=zebra --no-color-moved-ws $rev_a $rev_b\n+'\n+\n+test_perf 'diff --color-moved-ws=allow-indentation-change large change' '\n+\tgit diff --color-moved=zebra --color-moved-ws=allow-indentation-change \\\n+\t\t$rev_a $rev_b\n+'\n+\n+test_perf 'log --no-color-moved --no-color-moved-ws' '\n+\tgit log --no-color-moved --no-color-moved-ws --no-merges --patch \\\n+\t\t-n1000 $rev_b\n+'\n+\n+test_perf 'log --color-moved --no-color-moved-ws' '\n+\tgit log --color-moved=zebra --no-color-moved-ws --no-merges --patch \\\n+\t\t-n1000 $rev_b\n+'\n+\n+test_perf 'log --color-moved-ws=allow-indentation-change' '\n+\tgit log --color-moved=zebra --color-moved-ws=allow-indentation-change \\\n+\t\t--no-merges --patch -n1000 $rev_b\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"441273","messageId":"47c652716e88af7199c44a555780eafb643a1f9e.1637056178.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v4 02/15] diff --color-moved: clear all flags on blocks that are too short","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:25Z","receivedAt":"2021-11-16T09:49:55Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nIf a block of potentially moved lines is not long enough then the\nDIFF_SYMBOL_MOVED_LINE flag is cleared on the matching lines so they\nare not marked as moved. To avoid problems when we start rewinding\nafter an unsuccessful match in a couple of commits time make sure all\nthe move related flags are cleared, not just DIFF_SYMBOL_MOVED_LINE.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 52c791574b7..bd8e4ec9757 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1114,6 +1114,8 @@ static int shrink_potential_moved_blocks(struct moved_block *pmb,\n * NEEDSWORK: This uses the same heuristic as blame_entry_score() in blame.c.\n * Think of a way to unify them.\n */\n+#define DIFF_SYMBOL_MOVED_LINE_ZEBRA_MASK \\\n+ (DIFF_SYMBOL_MOVED_LINE | DIFF_SYMBOL_MOVED_LINE_ALT)\n static int adjust_last_block(struct diff_options *o, int n, int block_length)\n {\n \tint i, alnum_count = 0;\n@@ -1130,7 +1132,7 @@ static int adjust_last_block(struct diff_options *o, int n, int block_length)\n \t\t}\n \t}\n \tfor (i = 1; i < block_length + 1; i++)\n-\t\to->emitted_symbols->buf[n - i].flags &= ~DIFF_SYMBOL_MOVED_LINE;\n+\t\to->emitted_symbols->buf[n - i].flags &= ~DIFF_SYMBOL_MOVED_LINE_ZEBRA_MASK;\n \treturn 0;\n }\n \n@@ -1237,8 +1239,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \tfree(pmb);\n }\n \n-#define DIFF_SYMBOL_MOVED_LINE_ZEBRA_MASK \\\n- (DIFF_SYMBOL_MOVED_LINE | DIFF_SYMBOL_MOVED_LINE_ALT)\n static void dim_moved_lines(struct diff_options *o)\n {\n \tint n;\n-- \ngitgitgadget\n\n"},{"id":"441274","messageId":"99e38ba9de9bcd7b767908ec218937ae8c2ccf36.1637056178.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v4 03/15] diff --color-moved: factor out function","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:26Z","receivedAt":"2021-11-16T09:49:59Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nThis code is quite heavily indented and having it in its own function\nsimplifies an upcoming change.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 51 ++++++++++++++++++++++++++++++++++-----------------\n 1 file changed, 34 insertions(+), 17 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex bd8e4ec9757..09af94e018c 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1098,6 +1098,38 @@ static int shrink_potential_moved_blocks(struct moved_block *pmb,\n \treturn rp + 1;\n }\n \n+static void fill_potential_moved_blocks(struct diff_options *o,\n+\t\t\t\t\tstruct hashmap *hm,\n+\t\t\t\t\tstruct moved_entry *match,\n+\t\t\t\t\tstruct emitted_diff_symbol *l,\n+\t\t\t\t\tstruct moved_block **pmb_p,\n+\t\t\t\t\tint *pmb_alloc_p, int *pmb_nr_p)\n+\n+{\n+\tstruct moved_block *pmb = *pmb_p;\n+\tint pmb_alloc = *pmb_alloc_p, pmb_nr = *pmb_nr_p;\n+\n+\t/*\n+\t * The current line is the start of a new block.\n+\t * Setup the set of potential blocks.\n+\t */\n+\thashmap_for_each_entry_from(hm, match, ent) {\n+\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n+\t\tif (o->color_moved_ws_handling &\n+\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n+\t\t\tif (compute_ws_delta(l, match->es, &(pmb[pmb_nr]).wsd))\n+\t\t\t\tpmb[pmb_nr++].match = match;\n+\t\t} else {\n+\t\t\tpmb[pmb_nr].wsd = 0;\n+\t\t\tpmb[pmb_nr++].match = match;\n+\t\t}\n+\t}\n+\n+\t*pmb_p = pmb;\n+\t*pmb_alloc_p = pmb_alloc;\n+\t*pmb_nr_p = pmb_nr;\n+}\n+\n /*\n * If o->color_moved is COLOR_MOVED_PLAIN, this function does nothing.\n *\n@@ -1198,23 +1230,8 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n \t\tif (pmb_nr == 0) {\n-\t\t\t/*\n-\t\t\t * The current line is the start of a new block.\n-\t\t\t * Setup the set of potential blocks.\n-\t\t\t */\n-\t\t\thashmap_for_each_entry_from(hm, match, ent) {\n-\t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n-\t\t\t\tif (o->color_moved_ws_handling &\n-\t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n-\t\t\t\t\tif (compute_ws_delta(l, match->es,\n-\t\t\t\t\t\t\t &pmb[pmb_nr].wsd))\n-\t\t\t\t\t\tpmb[pmb_nr++].match = match;\n-\t\t\t\t} else {\n-\t\t\t\t\tpmb[pmb_nr].wsd = 0;\n-\t\t\t\t\tpmb[pmb_nr++].match = match;\n-\t\t\t\t}\n-\t\t\t}\n-\n+\t\t\tfill_potential_moved_blocks(\n+\t\t\t\to, hm, match, l, &pmb, &pmb_alloc, &pmb_nr);\n \t\t\tif (adjust_last_block(o, n, block_length) &&\n \t\t\t pmb_nr && last_symbol != l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n-- \ngitgitgadget\n\n"},{"id":"441275","messageId":"9ca71db61ae92d99fceba112593999596b65cd4c.1637056178.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v4 04/15] diff --color-moved: rewind when discarding pmb","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:27Z","receivedAt":"2021-11-16T09:50:02Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\ndiff --color-moved colors the two sides of the diff separately. It\nwalks through the diff and tries to find matches on the other side of\nthe diff for the current line. When it finds one or more matches it\nstarts a \"potential moved block\" (pmb) and marks the current line as\nmoved. Then as it walks through the diff it only looks for matches for\nthe current line in the lines following those in the pmb. When none of\nthe lines in the pmb match it checks how long the match is and if it\nis too short it unmarks the lines as matched and goes back to finding\nall the lines that match the current line. As the process of finding\nmatching lines restarts from the end of the block that was too short\nit is possible to miss the start of a matching block on on side but\nnot the other. In the test added here \"-two\" would not be colored as\nmoved but \"+two\" would be.\n\nFix this by rewinding the current line when we reach the end of a\nblock that is too short. This is quadratic in the length of the\ndiscarded block. While the discarded blocks are quite short on a large\ndiff this still has a significant impact on the performance of\n--color-moved-ws=allow-indentation-change. The following commits\noptimize the performance of the --color-moved machinery which\nmitigates the performance impact of this commit. After the\noptimization this commit has a negligible impact on performance.\n\nTest HEAD^ HEAD\n-----------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.33+0.05) 0.39 (0.34+0.04) +2.6%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.80 (0.76+0.03) 0.86 (0.82+0.04) +7.5%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 14.22(14.17+0.04) 19.01(18.93+0.05) +33.7%\n4002.4: log --no-color-moved --no-color-moved-ws 1.16 (1.06+0.09) 1.16 (1.07+0.07) +0.0%\n4002.5: log --color-moved --no-color-moved-ws 1.31 (1.22+0.09) 1.32 (1.22+0.09) +0.8%\n4002.6: log --color-moved-ws=allow-indentation-change 1.71 (1.61+0.09) 1.72 (1.63+0.08) +0.6%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 28 ++++++++++++++++++-----\n t/t4015-diff-whitespace.sh | 46 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 69 insertions(+), 5 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 09af94e018c..1e1b5127d15 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1205,7 +1205,15 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\tif (!match) {\n \t\t\tint i;\n \n-\t\t\tadjust_last_block(o, n, block_length);\n+\t\t\tif (!adjust_last_block(o, n, block_length) &&\n+\t\t\t block_length > 1) {\n+\t\t\t\t/*\n+\t\t\t\t * Rewind in case there is another match\n+\t\t\t\t * starting at the second line of the block\n+\t\t\t\t */\n+\t\t\t\tmatch = NULL;\n+\t\t\t\tn -= block_length;\n+\t\t\t}\n \t\t\tfor(i = 0; i < pmb_nr; i++)\n \t\t\t\tmoved_block_clear(&pmb[i]);\n \t\t\tpmb_nr = 0;\n@@ -1230,10 +1238,20 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n \t\tif (pmb_nr == 0) {\n-\t\t\tfill_potential_moved_blocks(\n-\t\t\t\to, hm, match, l, &pmb, &pmb_alloc, &pmb_nr);\n-\t\t\tif (adjust_last_block(o, n, block_length) &&\n-\t\t\t pmb_nr && last_symbol != l->s)\n+\t\t\tint contiguous = adjust_last_block(o, n, block_length);\n+\n+\t\t\tif (!contiguous && block_length > 1)\n+\t\t\t\t/*\n+\t\t\t\t * Rewind in case there is another match\n+\t\t\t\t * starting at the second line of the block\n+\t\t\t\t */\n+\t\t\t\tn -= block_length;\n+\t\t\telse\n+\t\t\t\tfill_potential_moved_blocks(o, hm, match, l,\n+\t\t\t\t\t\t\t &pmb, &pmb_alloc,\n+\t\t\t\t\t\t\t &pmb_nr);\n+\n+\t\t\tif (contiguous && pmb_nr && last_symbol != l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n \t\t\telse\n \t\t\t\tflipped_block = 0;\ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 2c13b62d3c6..308dc136596 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1833,6 +1833,52 @@ test_expect_success '--color-moved treats adjacent blocks as separate for MIN_AL\n \ttest_cmp expected actual\n '\n \n+test_expect_success '--color-moved rewinds for MIN_ALNUM_COUNT' '\n+\tgit reset --hard &&\n+\ttest_write_lines >file \\\n+\t\tA B C one two three four five six seven D E F G H I J &&\n+\tgit add file &&\n+\ttest_write_lines >file \\\n+\t\tone two A B C D E F G H I J two three four five six seven &&\n+\tgit diff --color-moved=zebra -- file &&\n+\n+\tgit diff --color-moved=zebra --color -- file >actual.raw &&\n+\tgrep -v \"index\" actual.raw | test_decode_color >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tdiff --git a/file b/file\n+\t--- a/file\n+\t+++ b/file\n+\t@@ -1,13 +1,8 @@\n+\t+one\n+\t+two\n+\t A\n+\t B\n+\t C\n+\t-one\n+\t-two\n+\t-three\n+\t-four\n+\t-five\n+\t-six\n+\t-seven\n+\t D\n+\t E\n+\t F\n+\t@@ -15,3 +10,9 @@ G\n+\t H\n+\t I\n+\t J\n+\t+two\n+\t+three\n+\t+four\n+\t+five\n+\t+six\n+\t+seven\n+\tEOF\n+\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'move detection with submodules' '\n \ttest_create_repo bananas &&\n \techo ripe >bananas/recipe &&\n-- \ngitgitgadget\n\n"},{"id":"441276","messageId":"56bb69af36e5e3180d53586d30048d5033a01d14.1637056178.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v4 05/15] diff --color-moved=zebra: fix alternate coloring","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:28Z","receivedAt":"2021-11-16T09:50:04Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nb0a2ba4776 (\"diff --color-moved=zebra: be stricter with color\nalternation\", 2018-11-23) sought to avoid using the alternate colors\nunless there are two adjacent moved blocks of the same\nsign. Unfortunately it contains two bugs that prevented it from fixing\nthe problem properly. Firstly `last_symbol` is reset at the start of\neach iteration of the loop losing the symbol of the last line and\nsecondly when deciding whether to use the alternate color it should be\nchecking if the current line is the same sign of the last line, not a\ndifferent sign. The combination of the two errors means that we still\nuse the alternate color when we should do but we also use it when we\nshouldn't. This is most noticable when using\n--color-moved-ws=allow-indentation-change with hunks like\n\n-this line gets indented\n+ this line gets indented\n\nwhere the post image is colored with newMovedAlternate rather than\nnewMoved. While this does not matter much, the next commit will change\nthe coloring to be correct in this case, so lets fix the bug here to\nmake it clear why the output is changing and add a regression test.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 4 +--\n t/t4015-diff-whitespace.sh | 72 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 74 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 1e1b5127d15..53f0df75329 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1176,6 +1176,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n \tint n, flipped_block = 0, block_length = 0;\n+\tenum diff_symbol last_symbol = 0;\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n@@ -1183,7 +1184,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\tstruct moved_entry *key;\n \t\tstruct moved_entry *match = NULL;\n \t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n-\t\tenum diff_symbol last_symbol = 0;\n \n \t\tswitch (l->s) {\n \t\tcase DIFF_SYMBOL_PLUS:\n@@ -1251,7 +1251,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t\t\t\t\t &pmb, &pmb_alloc,\n \t\t\t\t\t\t\t &pmb_nr);\n \n-\t\t\tif (contiguous && pmb_nr && last_symbol != l->s)\n+\t\t\tif (contiguous && pmb_nr && last_symbol == l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n \t\t\telse\n \t\t\t\tflipped_block = 0;\ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 308dc136596..4e0fd76c6c5 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1442,6 +1442,78 @@ test_expect_success 'detect permutations inside moved code -- dimmed-zebra' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'zebra alternate color is only used when necessary' '\n+\tcat >old.txt <<-\\EOF &&\n+\tline 1A should be marked as oldMoved newMovedAlternate\n+\tline 1B should be marked as oldMoved newMovedAlternate\n+\tunchanged\n+\tline 2A should be marked as oldMoved newMovedAlternate\n+\tline 2B should be marked as oldMoved newMovedAlternate\n+\tline 3A should be marked as oldMovedAlternate newMoved\n+\tline 3B should be marked as oldMovedAlternate newMoved\n+\tunchanged\n+\tline 4A should be marked as oldMoved newMovedAlternate\n+\tline 4B should be marked as oldMoved newMovedAlternate\n+\tline 5A should be marked as oldMovedAlternate newMoved\n+\tline 5B should be marked as oldMovedAlternate newMoved\n+\tline 6A should be marked as oldMoved newMoved\n+\tline 6B should be marked as oldMoved newMoved\n+\tEOF\n+\tcat >new.txt <<-\\EOF &&\n+\t line 1A should be marked as oldMoved newMovedAlternate\n+\t line 1B should be marked as oldMoved newMovedAlternate\n+\tunchanged\n+\t line 3A should be marked as oldMovedAlternate newMoved\n+\t line 3B should be marked as oldMovedAlternate newMoved\n+\t line 2A should be marked as oldMoved newMovedAlternate\n+\t line 2B should be marked as oldMoved newMovedAlternate\n+\tunchanged\n+\t line 6A should be marked as oldMoved newMoved\n+\t line 6B should be marked as oldMoved newMoved\n+\t line 4A should be marked as oldMoved newMovedAlternate\n+\t line 4B should be marked as oldMoved newMovedAlternate\n+\t line 5A should be marked as oldMovedAlternate newMoved\n+\t line 5B should be marked as oldMovedAlternate newMoved\n+\tEOF\n+\ttest_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n+\t\t --color-moved-ws=allow-indentation-change \\\n+\t\t old.txt new.txt >output &&\n+\tgrep -v index output | test_decode_color >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tdiff --git a/old.txt b/new.txt\n+\t--- a/old.txt\n+\t+++ b/new.txt\n+\t@@ -1,14 +1,14 @@\n+\t-line 1A should be marked as oldMoved newMovedAlternate\n+\t-line 1B should be marked as oldMoved newMovedAlternate\n+\t+ line 1A should be marked as oldMoved newMovedAlternate\n+\t+ line 1B should be marked as oldMoved newMovedAlternate\n+\t unchanged\n+\t-line 2A should be marked as oldMoved newMovedAlternate\n+\t-line 2B should be marked as oldMoved newMovedAlternate\n+\t-line 3A should be marked as oldMovedAlternate newMoved\n+\t-line 3B should be marked as oldMovedAlternate newMoved\n+\t+ line 3A should be marked as oldMovedAlternate newMoved\n+\t+ line 3B should be marked as oldMovedAlternate newMoved\n+\t+ line 2A should be marked as oldMoved newMovedAlternate\n+\t+ line 2B should be marked as oldMoved newMovedAlternate\n+\t unchanged\n+\t-line 4A should be marked as oldMoved newMovedAlternate\n+\t-line 4B should be marked as oldMoved newMovedAlternate\n+\t-line 5A should be marked as oldMovedAlternate newMoved\n+\t-line 5B should be marked as oldMovedAlternate newMoved\n+\t-line 6A should be marked as oldMoved newMoved\n+\t-line 6B should be marked as oldMoved newMoved\n+\t+ line 6A should be marked as oldMoved newMoved\n+\t+ line 6B should be marked as oldMoved newMoved\n+\t+ line 4A should be marked as oldMoved newMovedAlternate\n+\t+ line 4B should be marked as oldMoved newMovedAlternate\n+\t+ line 5A should be marked as oldMovedAlternate newMoved\n+\t+ line 5B should be marked as oldMovedAlternate newMoved\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'cmd option assumes configured colored-moved' '\n \ttest_config color.diff.oldMoved \"magenta\" &&\n \ttest_config color.diff.newMoved \"cyan\" &&\n-- \ngitgitgadget\n\n"},{"id":"441277","messageId":"10b11526206d3b515ba08ac80ccf09ecb7a03420.1637056178.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v4 06/15] diff --color-moved: avoid false short line matches and bad zerba coloring","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:29Z","receivedAt":"2021-11-16T09:50:06Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nWhen marking moved lines it is possible for a block of potential\nmatched lines to extend past a change in sign when there is a sequence\nof added lines whose text matches the text of a sequence of deleted\nand added lines. Most of the time either `match` will be NULL or\n`pmb_advance_or_null()` will fail when the loop encounters a change of\nsign but there are corner cases where `match` is non-NULL and\n`pmb_advance_or_null()` successfully advances the moved block despite\nthe change in sign.\n\nOne consequence of this is highlighting a short line as moved when it\nshould not be. For example\n\n-moved line # Correctly highlighted as moved\n+short line # Wrongly highlighted as moved\n context\n+moved line # Correctly highlighted as moved\n+short line\n context\n-short line\n\nThe other consequence is coloring a moved addition following a moved\ndeletion in the wrong color. In the example below the first \"+moved\nline 3\" should be highlighted as newMoved not newMovedAlternate.\n\n-moved line 1 # Correctly highlighted as oldMoved\n-moved line 2 # Correctly highlighted as oldMovedAlternate\n+moved line 3 # Wrongly highlighted as newMovedAlternate\n context # Everything else is highlighted correctly\n+moved line 2\n+moved line 3\n context\n+moved line 1\n-moved line 3\n\nThese false matches are more likely when using --color-moved-ws with\nthe exception of --color-moved-ws=allow-indentation-change which ties\nthe sign of the current whitespace delta to the sign of the line to\navoid this problem. The fix is to check that the sign of the new line\nbeing matched is the same as the sign of the line that started the\nblock of potential matches.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 17 ++++++----\n t/t4015-diff-whitespace.sh | 65 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 76 insertions(+), 6 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 53f0df75329..efba2789354 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1176,7 +1176,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n \tint n, flipped_block = 0, block_length = 0;\n-\tenum diff_symbol last_symbol = 0;\n+\tenum diff_symbol moved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n@@ -1202,7 +1202,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tflipped_block = 0;\n \t\t}\n \n-\t\tif (!match) {\n+\t\tif (pmb_nr && (!match || l->s != moved_symbol)) {\n \t\t\tint i;\n \n \t\t\tif (!adjust_last_block(o, n, block_length) &&\n@@ -1219,12 +1219,13 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tpmb_nr = 0;\n \t\t\tblock_length = 0;\n \t\t\tflipped_block = 0;\n-\t\t\tlast_symbol = l->s;\n+\t\t}\n+\t\tif (!match) {\n+\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (o->color_moved == COLOR_MOVED_PLAIN) {\n-\t\t\tlast_symbol = l->s;\n \t\t\tl->flags |= DIFF_SYMBOL_MOVED_LINE;\n \t\t\tcontinue;\n \t\t}\n@@ -1251,11 +1252,16 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t\t\t\t\t &pmb, &pmb_alloc,\n \t\t\t\t\t\t\t &pmb_nr);\n \n-\t\t\tif (contiguous && pmb_nr && last_symbol == l->s)\n+\t\t\tif (contiguous && pmb_nr && moved_symbol == l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n \t\t\telse\n \t\t\t\tflipped_block = 0;\n \n+\t\t\tif (pmb_nr)\n+\t\t\t\tmoved_symbol = l->s;\n+\t\t\telse\n+\t\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n+\n \t\t\tblock_length = 0;\n \t\t}\n \n@@ -1265,7 +1271,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tif (flipped_block && o->color_moved != COLOR_MOVED_BLOCKS)\n \t\t\t\tl->flags |= DIFF_SYMBOL_MOVED_LINE_ALT;\n \t\t}\n-\t\tlast_symbol = l->s;\n \t}\n \tadjust_last_block(o, n, block_length);\n \ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 4e0fd76c6c5..15782c879d2 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1514,6 +1514,71 @@ test_expect_success 'zebra alternate color is only used when necessary' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'short lines of opposite sign do not get marked as moved' '\n+\tcat >old.txt <<-\\EOF &&\n+\tthis line should be marked as moved\n+\tunchanged\n+\tunchanged\n+\tunchanged\n+\tunchanged\n+\ttoo short\n+\tthis line should be marked as oldMoved newMoved\n+\tthis line should be marked as oldMovedAlternate newMoved\n+\tunchanged 1\n+\tunchanged 2\n+\tunchanged 3\n+\tunchanged 4\n+\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n+\tEOF\n+\tcat >new.txt <<-\\EOF &&\n+\ttoo short\n+\tunchanged\n+\tunchanged\n+\tthis line should be marked as moved\n+\ttoo short\n+\tunchanged\n+\tunchanged\n+\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n+\tunchanged 1\n+\tunchanged 2\n+\tthis line should be marked as oldMovedAlternate newMoved\n+\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n+\tunchanged 3\n+\tthis line should be marked as oldMoved newMoved\n+\tunchanged 4\n+\tEOF\n+\ttest_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n+\t\told.txt new.txt >output && cat output &&\n+\tgrep -v index output | test_decode_color >actual &&\n+\tcat >expect <<-\\EOF &&\n+\tdiff --git a/old.txt b/new.txt\n+\t--- a/old.txt\n+\t+++ b/new.txt\n+\t@@ -1,13 +1,15 @@\n+\t-this line should be marked as moved\n+\t+too short\n+\t unchanged\n+\t unchanged\n+\t+this line should be marked as moved\n+\t+too short\n+\t unchanged\n+\t unchanged\n+\t-too short\n+\t-this line should be marked as oldMoved newMoved\n+\t-this line should be marked as oldMovedAlternate newMoved\n+\t+this line should be marked as oldMoved newMoved/newMovedAlternate\n+\t unchanged 1\n+\t unchanged 2\n+\t+this line should be marked as oldMovedAlternate newMoved\n+\t+this line should be marked as oldMoved newMoved/newMovedAlternate\n+\t unchanged 3\n+\t+this line should be marked as oldMoved newMoved\n+\t unchanged 4\n+\t-this line should be marked as oldMoved newMoved/newMovedAlternate\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'cmd option assumes configured colored-moved' '\n \ttest_config color.diff.oldMoved \"magenta\" &&\n \ttest_config color.diff.newMoved \"cyan\" &&\n-- \ngitgitgadget\n\n"},{"id":"441278","messageId":"c2e7b3472575f2d3c4143d68777dfb45663d68c4.1637056178.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v4 07/15] diff: simplify allow-indentation-change delta calculation","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:30Z","receivedAt":"2021-11-16T09:50:08Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nNow that we reliably end a block when the sign changes we don't need\nthe whitespace delta calculation to rely on the sign.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 13 ++-----------\n 1 file changed, 2 insertions(+), 11 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex efba2789354..9aff167be27 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -864,23 +864,17 @@ static int compute_ws_delta(const struct emitted_diff_symbol *a,\n \t a_width = a->indent_width,\n \t b_off = b->indent_off,\n \t b_width = b->indent_width;\n-\tint delta;\n \n \tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE) {\n \t\t*out = INDENT_BLANKLINE;\n \t\treturn 1;\n \t}\n \n-\tif (a->s == DIFF_SYMBOL_PLUS)\n-\t\tdelta = a_width - b_width;\n-\telse\n-\t\tdelta = b_width - a_width;\n-\n \tif (a_len - a_off != b_len - b_off ||\n \t memcmp(a->line + a_off, b->line + b_off, a_len - a_off))\n \t\treturn 0;\n \n-\t*out = delta;\n+\t*out = a_width - b_width;\n \n \treturn 1;\n }\n@@ -924,10 +918,7 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \t * match those of the current block and that the text of 'l' and 'cur'\n \t * after the indentation match.\n \t */\n-\tif (cur->es->s == DIFF_SYMBOL_PLUS)\n-\t\tdelta = a_width - c_width;\n-\telse\n-\t\tdelta = c_width - a_width;\n+\tdelta = c_width - a_width;\n \n \t/*\n \t * If the previous lines of this block were all blank then set its\n-- \ngitgitgadget\n\n"},{"id":"441279","messageId":"d7bbc0041e076077d3c3299858799cc9907d9831.1637056179.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v4 08/15] diff --color-moved-ws=allow-indentation-change: simplify and optimize","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:31Z","receivedAt":"2021-11-16T09:50:26Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nIf we already have a block of potentially moved lines then as we move\ndown the diff we need to check if the next line of each potentially\nmoved line matches the current line of the diff. The implementation of\n--color-moved-ws=allow-indentation-change was needlessly performing\nthis check on all the lines in the diff that matched the current line\nrather than just the current line. To exacerbate the problem finding\nall the other lines in the diff that match the current line involves a\nfuzzy lookup so we were wasting even more time performing a second\ncomparison to filter out the non-matching lines. Fixing this reduces\ntime to run\n git diff --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\nby 93% compared to master and simplifies the code.\n\nTest HEAD^ HEAD\n---------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.35+0.03) 0.38(0.35+0.03) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.86 (0.80+0.06) 0.87(0.83+0.04) +1.2%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 19.01(18.93+0.06) 0.97(0.92+0.04) -94.9%\n4002.4: log --no-color-moved --no-color-moved-ws 1.16 (1.06+0.09) 1.17(1.06+0.10) +0.9%\n4002.5: log --color-moved --no-color-moved-ws 1.32 (1.25+0.07) 1.32(1.24+0.08) +0.0%\n4002.6: log --color-moved-ws=allow-indentation-change 1.71 (1.64+0.06) 1.36(1.25+0.10) -20.5%\n\nTest master HEAD\n---------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.33+0.05) 0.38(0.35+0.03) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.80 (0.75+0.04) 0.87(0.83+0.04) +8.7%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 14.20(14.15+0.05) 0.97(0.92+0.04) -93.2%\n4002.4: log --no-color-moved --no-color-moved-ws 1.15 (1.05+0.09) 1.17(1.06+0.10) +1.7%\n4002.5: log --color-moved --no-color-moved-ws 1.30 (1.19+0.11) 1.32(1.24+0.08) +1.5%\n4002.6: log --color-moved-ws=allow-indentation-change 1.70 (1.63+0.06) 1.36(1.25+0.10) -20.0%\n\nHelped-by: Jeff King \nSigned-off-by: Phillip Wood \n---\n diff.c | 70 +++++++++++++++++-----------------------------------------\n 1 file changed, 20 insertions(+), 50 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 9aff167be27..78a486021ab 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -879,37 +879,21 @@ static int compute_ws_delta(const struct emitted_diff_symbol *a,\n \treturn 1;\n }\n \n-static int cmp_in_block_with_wsd(const struct diff_options *o,\n-\t\t\t\t const struct moved_entry *cur,\n-\t\t\t\t const struct moved_entry *match,\n-\t\t\t\t struct moved_block *pmb,\n-\t\t\t\t int n)\n-{\n-\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n-\tint al = cur->es->len, bl = match->es->len, cl = l->len;\n+static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n+\t\t\t\t const struct emitted_diff_symbol *l,\n+\t\t\t\t struct moved_block *pmb)\n+{\n+\tint al = cur->es->len, bl = l->len;\n \tconst char *a = cur->es->line,\n-\t\t *b = match->es->line,\n-\t\t *c = l->line;\n+\t\t *b = l->line;\n \tint a_off = cur->es->indent_off,\n \t a_width = cur->es->indent_width,\n-\t c_off = l->indent_off,\n-\t c_width = l->indent_width;\n+\t b_off = l->indent_off,\n+\t b_width = l->indent_width;\n \tint delta;\n \n-\t/*\n-\t * We need to check if 'cur' is equal to 'match'. As those\n-\t * are from the same (+/-) side, we do not need to adjust for\n-\t * indent changes. However these were found using fuzzy\n-\t * matching so we do have to check if they are equal. Here we\n-\t * just check the lengths. We delay calling memcmp() to check\n-\t * the contents until later as if the length comparison for a\n-\t * and c fails we can avoid the call all together.\n-\t */\n-\tif (al != bl)\n-\t\treturn 1;\n-\n \t/* If 'l' and 'cur' are both blank then they match. */\n-\tif (a_width == INDENT_BLANKLINE && c_width == INDENT_BLANKLINE)\n+\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n \t\treturn 0;\n \n \t/*\n@@ -918,7 +902,7 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \t * match those of the current block and that the text of 'l' and 'cur'\n \t * after the indentation match.\n \t */\n-\tdelta = c_width - a_width;\n+\tdelta = b_width - a_width;\n \n \t/*\n \t * If the previous lines of this block were all blank then set its\n@@ -927,9 +911,8 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \tif (pmb->wsd == INDENT_BLANKLINE)\n \t\tpmb->wsd = delta;\n \n-\treturn !(delta == pmb->wsd && al - a_off == cl - c_off &&\n-\t\t !memcmp(a, b, al) && !\n-\t\t memcmp(a + a_off, c + c_off, al - a_off));\n+\treturn !(delta == pmb->wsd && al - a_off == bl - b_off &&\n+\t\t !memcmp(a + a_off, b + b_off, al - a_off));\n }\n \n static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n@@ -1030,36 +1013,23 @@ static void pmb_advance_or_null(struct diff_options *o,\n }\n \n static void pmb_advance_or_null_multi_match(struct diff_options *o,\n-\t\t\t\t\t struct moved_entry *match,\n-\t\t\t\t\t struct hashmap *hm,\n+\t\t\t\t\t struct emitted_diff_symbol *l,\n \t\t\t\t\t struct moved_block *pmb,\n-\t\t\t\t\t int pmb_nr, int n)\n+\t\t\t\t\t int pmb_nr)\n {\n \tint i;\n-\tchar *got_match = xcalloc(1, pmb_nr);\n-\n-\thashmap_for_each_entry_from(hm, match, ent) {\n-\t\tfor (i = 0; i < pmb_nr; i++) {\n-\t\t\tstruct moved_entry *prev = pmb[i].match;\n-\t\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n-\t\t\t\t\tprev->next_line : NULL;\n-\t\t\tif (!cur)\n-\t\t\t\tcontinue;\n-\t\t\tif (!cmp_in_block_with_wsd(o, cur, match, &pmb[i], n))\n-\t\t\t\tgot_match[i] |= 1;\n-\t\t}\n-\t}\n \n \tfor (i = 0; i < pmb_nr; i++) {\n-\t\tif (got_match[i]) {\n+\t\tstruct moved_entry *prev = pmb[i].match;\n+\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n+\t\t\tprev->next_line : NULL;\n+\t\tif (cur && !cmp_in_block_with_wsd(cur, l, &pmb[i])) {\n \t\t\t/* Advance to the next line */\n-\t\t\tpmb[i].match = pmb[i].match->next_line;\n+\t\t\tpmb[i].match = cur;\n \t\t} else {\n \t\t\tmoved_block_clear(&pmb[i]);\n \t\t}\n \t}\n-\n-\tfree(got_match);\n }\n \n static int shrink_potential_moved_blocks(struct moved_block *pmb,\n@@ -1223,7 +1193,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \n \t\tif (o->color_moved_ws_handling &\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\tpmb_advance_or_null_multi_match(o, match, hm, pmb, pmb_nr, n);\n+\t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n \t\telse\n \t\t\tpmb_advance_or_null(o, match, hm, pmb, pmb_nr);\n \n-- \ngitgitgadget\n\n"},{"id":"441280","messageId":"35e204e157896ea1bd99535a89051dee245a6600.1637056179.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v4 11/15] diff --color-moved: shrink potential moved blocks as we go","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:34Z","receivedAt":"2021-11-16T09:50:28Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nRather than setting `match` to NULL and then looping over the list of\npotential matched blocks for a second time to remove blocks with no\nmatches just filter out the blocks with no matches as we go.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 44 ++++++++------------------------------------\n 1 file changed, 8 insertions(+), 36 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 51f092e724e..626fd47aa0e 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -996,12 +996,12 @@ static void add_lines_to_move_detection(struct diff_options *o,\n static void pmb_advance_or_null(struct diff_options *o,\n \t\t\t\tstruct emitted_diff_symbol *l,\n \t\t\t\tstruct moved_block *pmb,\n-\t\t\t\tint pmb_nr)\n+\t\t\t\tint *pmb_nr)\n {\n-\tint i;\n+\tint i, j;\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \n-\tfor (i = 0; i < pmb_nr; i++) {\n+\tfor (i = 0, j = 0; i < *pmb_nr; i++) {\n \t\tint match;\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n@@ -1015,38 +1015,12 @@ static void pmb_advance_or_null(struct diff_options *o,\n \t\t\tmatch = cur &&\n \t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n \t\t\t\t\t\t l->line, l->len, flags);\n-\t\tif (match)\n-\t\t\tpmb[i].match = cur;\n-\t\telse\n-\t\t\tmoved_block_clear(&pmb[i]);\n-\t}\n-}\n-\n-static int shrink_potential_moved_blocks(struct moved_block *pmb,\n-\t\t\t\t\t int pmb_nr)\n-{\n-\tint lp, rp;\n-\n-\t/* Shrink the set of potential block to the remaining running */\n-\tfor (lp = 0, rp = pmb_nr - 1; lp <= rp;) {\n-\t\twhile (lp < pmb_nr && pmb[lp].match)\n-\t\t\tlp++;\n-\t\t/* lp points at the first NULL now */\n-\n-\t\twhile (rp > -1 && !pmb[rp].match)\n-\t\t\trp--;\n-\t\t/* rp points at the last non-NULL */\n-\n-\t\tif (lp < pmb_nr && rp > -1 && lp < rp) {\n-\t\t\tpmb[lp] = pmb[rp];\n-\t\t\tmemset(&pmb[rp], 0, sizeof(pmb[rp]));\n-\t\t\trp--;\n-\t\t\tlp++;\n+\t\tif (match) {\n+\t\t\tpmb[j] = pmb[i];\n+\t\t\tpmb[j++].match = cur;\n \t\t}\n \t}\n-\n-\t/* Remember the number of running sets */\n-\treturn rp + 1;\n+\t*pmb_nr = j;\n }\n \n static void fill_potential_moved_blocks(struct diff_options *o,\n@@ -1181,9 +1155,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n-\n-\t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n+\t\tpmb_advance_or_null(o, l, pmb, &pmb_nr);\n \n \t\tif (pmb_nr == 0) {\n \t\t\tint contiguous = adjust_last_block(o, n, block_length);\n-- \ngitgitgadget\n\n"},{"id":"441281","messageId":"9eb8cecd52a5eef7510d59d7750f225eb5f8d64c.1637056179.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v4 10/15] diff --color-moved: unify moved block growth functions","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:33Z","receivedAt":"2021-11-16T09:50:28Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nAfter the last two commits pmb_advance_or_null() and\npmb_advance_or_null_multi_match() differ only in the comparison they\nperform. Lets simplify the code by combining them into a single\nfunction.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 41 ++++++++++++-----------------------------\n 1 file changed, 12 insertions(+), 29 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 22e0edac173..51f092e724e 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1002,36 +1002,23 @@ static void pmb_advance_or_null(struct diff_options *o,\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \n \tfor (i = 0; i < pmb_nr; i++) {\n+\t\tint match;\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n \t\t\t\tprev->next_line : NULL;\n-\t\tif (cur && xdiff_compare_lines(cur->es->line, cur->es->len,\n-\t\t\t\t\t\tl->line, l->len,\n-\t\t\t\t\t\tflags)) {\n-\t\t\tpmb[i].match = cur;\n-\t\t} else {\n-\t\t\tpmb[i].match = NULL;\n-\t\t}\n-\t}\n-}\n \n-static void pmb_advance_or_null_multi_match(struct diff_options *o,\n-\t\t\t\t\t struct emitted_diff_symbol *l,\n-\t\t\t\t\t struct moved_block *pmb,\n-\t\t\t\t\t int pmb_nr)\n-{\n-\tint i;\n-\n-\tfor (i = 0; i < pmb_nr; i++) {\n-\t\tstruct moved_entry *prev = pmb[i].match;\n-\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n-\t\t\tprev->next_line : NULL;\n-\t\tif (cur && !cmp_in_block_with_wsd(cur, l, &pmb[i])) {\n-\t\t\t/* Advance to the next line */\n+\t\tif (o->color_moved_ws_handling &\n+\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n+\t\t\tmatch = cur &&\n+\t\t\t\t!cmp_in_block_with_wsd(cur, l, &pmb[i]);\n+\t\telse\n+\t\t\tmatch = cur &&\n+\t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n+\t\t\t\t\t\t l->line, l->len, flags);\n+\t\tif (match)\n \t\t\tpmb[i].match = cur;\n-\t\t} else {\n+\t\telse\n \t\t\tmoved_block_clear(&pmb[i]);\n-\t\t}\n \t}\n }\n \n@@ -1194,11 +1181,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tif (o->color_moved_ws_handling &\n-\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n-\t\telse\n-\t\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n+\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n \n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n-- \ngitgitgadget\n\n"},{"id":"441282","messageId":"c3e5dce1910b3d640757e0845d646c3b040a8e28.1637056179.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v4 09/15] diff --color-moved: call comparison function directly","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:32Z","receivedAt":"2021-11-16T09:50:28Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nThis change will allow us to easily combine pmb_advance_or_null() and\npmb_advance_or_null_multi_match() in the next commit. Calling\nxdiff_compare_lines() directly rather than using a function pointer\nfrom the hash map has little effect on the run time.\n\nTest HEAD^ HEAD\n-------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38(0.35+0.03) 0.38(0.32+0.06) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.87(0.83+0.04) 0.87(0.80+0.06) +0.0%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 0.97(0.92+0.04) 0.97(0.93+0.04) +0.0%\n4002.4: log --no-color-moved --no-color-moved-ws 1.17(1.06+0.10) 1.16(1.10+0.05) -0.9%\n4002.5: log --color-moved --no-color-moved-ws 1.32(1.24+0.08) 1.31(1.22+0.09) -0.8%\n4002.6: log --color-moved-ws=allow-indentation-change 1.36(1.25+0.10) 1.35(1.25+0.10) -0.7%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 11 +++++++----\n 1 file changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 78a486021ab..22e0edac173 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -994,17 +994,20 @@ static void add_lines_to_move_detection(struct diff_options *o,\n }\n \n static void pmb_advance_or_null(struct diff_options *o,\n-\t\t\t\tstruct moved_entry *match,\n-\t\t\t\tstruct hashmap *hm,\n+\t\t\t\tstruct emitted_diff_symbol *l,\n \t\t\t\tstruct moved_block *pmb,\n \t\t\t\tint pmb_nr)\n {\n \tint i;\n+\tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n+\n \tfor (i = 0; i < pmb_nr; i++) {\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n \t\t\t\tprev->next_line : NULL;\n-\t\tif (cur && !hm->cmpfn(o, &cur->ent, &match->ent, NULL)) {\n+\t\tif (cur && xdiff_compare_lines(cur->es->line, cur->es->len,\n+\t\t\t\t\t\tl->line, l->len,\n+\t\t\t\t\t\tflags)) {\n \t\t\tpmb[i].match = cur;\n \t\t} else {\n \t\t\tpmb[i].match = NULL;\n@@ -1195,7 +1198,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n \t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n \t\telse\n-\t\t\tpmb_advance_or_null(o, match, hm, pmb, pmb_nr);\n+\t\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n \n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n-- \ngitgitgadget\n\n"},{"id":"441283","messageId":"ec329e7946df27e978f6966c6ab23d7db6414ede.1637056179.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v4 12/15] diff --color-moved: stop clearing potential moved blocks","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:35Z","receivedAt":"2021-11-16T09:50:28Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nmoved_block_clear() was introduced in 74d156f4a1 (\"diff\n--color-moved-ws: fix double free crash\", 2018-10-04) to free the\nmemory that was allocated when initializing a potential moved\nblock. However since 21536d077f (\"diff --color-moved-ws: modify\nallow-indentation-change\", 2018-11-23) initializing a potential moved\nblock no longer allocates any memory. Up until the last commit we were\nrelying on moved_block_clear() to set the `match` pointer to NULL when\na block stopped matching, but since that commit we do not clear a\nmoved block that does not match so it does not make sense to clear\nthem elsewhere.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 11 -----------\n 1 file changed, 11 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 626fd47aa0e..ffbe09937bc 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -807,11 +807,6 @@ struct moved_block {\n \tint wsd; /* The whitespace delta of this block */\n };\n \n-static void moved_block_clear(struct moved_block *b)\n-{\n-\tmemset(b, 0, sizeof(*b));\n-}\n-\n #define INDENT_BLANKLINE INT_MIN\n \n static void fill_es_indent_data(struct emitted_diff_symbol *es)\n@@ -1128,8 +1123,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t}\n \n \t\tif (pmb_nr && (!match || l->s != moved_symbol)) {\n-\t\t\tint i;\n-\n \t\t\tif (!adjust_last_block(o, n, block_length) &&\n \t\t\t block_length > 1) {\n \t\t\t\t/*\n@@ -1139,8 +1132,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t\tmatch = NULL;\n \t\t\t\tn -= block_length;\n \t\t\t}\n-\t\t\tfor(i = 0; i < pmb_nr; i++)\n-\t\t\t\tmoved_block_clear(&pmb[i]);\n \t\t\tpmb_nr = 0;\n \t\t\tblock_length = 0;\n \t\t\tflipped_block = 0;\n@@ -1193,8 +1184,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t}\n \tadjust_last_block(o, n, block_length);\n \n-\tfor(n = 0; n < pmb_nr; n++)\n-\t\tmoved_block_clear(&pmb[n]);\n \tfree(pmb);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"441284","messageId":"6ec94134aaf6f6114adc5984722f99acb359ac56.1637056179.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v4 13/15] diff --color-moved-ws=allow-indentation-change: improve hash lookups","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:36Z","receivedAt":"2021-11-16T09:50:28Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nAs libxdiff does not have a whitespace flag to ignore the indentation\nthe code for --color-moved-ws=allow-indentation-change uses\nXDF_IGNORE_WHITESPACE and then filters out any hash lookups where\nthere are non-indentation changes. This filtering is inefficient as\nwe have to perform another string comparison.\n\nBy using the offset data that we have already computed to skip the\nindentation we can avoid using XDF_IGNORE_WHITESPACE and safely remove\nthe extra checks which improves the performance by 11% and paves the\nway for the elimination of string comparisons in the next commit.\n\nThis change slightly increases the run time of other --color-moved\nmodes. This could be avoided by using different comparison functions\nfor the different modes but after the next two commits there is no\nmeasurable benefit in doing so.\n\nThere is a change in behavior for lines that begin with a form-feed or\nvertical-tab character. Since b46054b374 (\"xdiff: use\ngit-compat-util\", 2019-04-11) xdiff does not treat '\\f' or '\\v' as\nwhitespace characters. This means that lines starting with those\ncharacters are never considered to be blank and never match a line\nthat does not start with the same character. After this patch a line\nmatching \"^[\\f\\v\\r]*[ \\t]*$\" is considered to be blank by\n--color-moved-ws=allow-indentation-change and lines beginning\n\"^[\\f\\v\\r]*[ \\t]*\" can match another line if the suffixes match. This\nchanges the output of git show for d18f76dccf (\"compat/regex: use the\nregex engine from gawk for compat\", 2010-08-17) as some lines in the\npre-image before a moved block that contain '\\f' are now considered\nmoved as well as they match a blank line before the moved lines in the\npost-image. This commit updates one of the tests to reflect this\nchange.\n\nTest HEAD^ HEAD\n--------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38(0.33+0.05) 0.38(0.33+0.05) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.86(0.82+0.04) 0.88(0.84+0.04) +2.3%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 0.97(0.94+0.03) 0.86(0.81+0.05) -11.3%\n4002.4: log --no-color-moved --no-color-moved-ws 1.16(1.07+0.09) 1.16(1.06+0.09) +0.0%\n4002.5: log --color-moved --no-color-moved-ws 1.32(1.26+0.06) 1.33(1.27+0.05) +0.8%\n4002.6: log --color-moved-ws=allow-indentation-change 1.35(1.29+0.06) 1.33(1.24+0.08) -1.5%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 65 +++++++++++---------------------------\n t/t4015-diff-whitespace.sh | 22 ++++++-------\n 2 files changed, 30 insertions(+), 57 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex ffbe09937bc..2085c063675 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -850,28 +850,15 @@ static void fill_es_indent_data(struct emitted_diff_symbol *es)\n }\n \n static int compute_ws_delta(const struct emitted_diff_symbol *a,\n-\t\t\t const struct emitted_diff_symbol *b,\n-\t\t\t int *out)\n-{\n-\tint a_len = a->len,\n-\t b_len = b->len,\n-\t a_off = a->indent_off,\n-\t a_width = a->indent_width,\n-\t b_off = b->indent_off,\n+\t\t\t const struct emitted_diff_symbol *b)\n+{\n+\tint a_width = a->indent_width,\n \t b_width = b->indent_width;\n \n-\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE) {\n-\t\t*out = INDENT_BLANKLINE;\n-\t\treturn 1;\n-\t}\n-\n-\tif (a_len - a_off != b_len - b_off ||\n-\t memcmp(a->line + a_off, b->line + b_off, a_len - a_off))\n-\t\treturn 0;\n-\n-\t*out = a_width - b_width;\n+\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n+\t\treturn INDENT_BLANKLINE;\n \n-\treturn 1;\n+\treturn a_width - b_width;\n }\n \n static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n@@ -916,26 +903,17 @@ static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n \t\t\t const void *keydata)\n {\n \tconst struct diff_options *diffopt = hashmap_cmp_fn_data;\n-\tconst struct moved_entry *a, *b;\n+\tconst struct emitted_diff_symbol *a, *b;\n \tunsigned flags = diffopt->color_moved_ws_handling\n \t\t\t & XDF_WHITESPACE_FLAGS;\n \n-\ta = container_of(eptr, const struct moved_entry, ent);\n-\tb = container_of(entry_or_key, const struct moved_entry, ent);\n+\ta = container_of(eptr, const struct moved_entry, ent)->es;\n+\tb = container_of(entry_or_key, const struct moved_entry, ent)->es;\n \n-\tif (diffopt->color_moved_ws_handling &\n-\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t/*\n-\t\t * As there is not specific white space config given,\n-\t\t * we'd need to check for a new block, so ignore all\n-\t\t * white space. The setup of the white space\n-\t\t * configuration for the next block is done else where\n-\t\t */\n-\t\tflags |= XDF_IGNORE_WHITESPACE;\n-\n-\treturn !xdiff_compare_lines(a->es->line, a->es->len,\n-\t\t\t\t b->es->line, b->es->len,\n-\t\t\t\t flags);\n+\treturn !xdiff_compare_lines(a->line + a->indent_off,\n+\t\t\t\t a->len - a->indent_off,\n+\t\t\t\t b->line + b->indent_off,\n+\t\t\t\t b->len - b->indent_off, flags);\n }\n \n static struct moved_entry *prepare_entry(struct diff_options *o,\n@@ -944,7 +922,8 @@ static struct moved_entry *prepare_entry(struct diff_options *o,\n \tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n \tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[line_no];\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n-\tunsigned int hash = xdiff_hash_string(l->line, l->len, flags);\n+\tunsigned int hash = xdiff_hash_string(l->line + l->indent_off,\n+\t\t\t\t\t l->len - l->indent_off, flags);\n \n \thashmap_entry_init(&ret->ent, hash);\n \tret->es = l;\n@@ -1036,13 +1015,11 @@ static void fill_potential_moved_blocks(struct diff_options *o,\n \thashmap_for_each_entry_from(hm, match, ent) {\n \t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n \t\tif (o->color_moved_ws_handling &\n-\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n-\t\t\tif (compute_ws_delta(l, match->es, &(pmb[pmb_nr]).wsd))\n-\t\t\t\tpmb[pmb_nr++].match = match;\n-\t\t} else {\n+\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n+\t\t\tpmb[pmb_nr].wsd = compute_ws_delta(l, match->es);\n+\t\telse\n \t\t\tpmb[pmb_nr].wsd = 0;\n-\t\t\tpmb[pmb_nr++].match = match;\n-\t\t}\n+\t\tpmb[pmb_nr++].match = match;\n \t}\n \n \t*pmb_p = pmb;\n@@ -6276,10 +6253,6 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \t\tif (o->color_moved) {\n \t\t\tstruct hashmap add_lines, del_lines;\n \n-\t\t\tif (o->color_moved_ws_handling &\n-\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\t\to->color_moved_ws_handling |= XDF_IGNORE_WHITESPACE;\n-\n \t\t\thashmap_init(&del_lines, moved_entry_cmp, o, 0);\n \t\t\thashmap_init(&add_lines, moved_entry_cmp, o, 0);\n \ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 15782c879d2..50d0cf486be 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -2206,10 +2206,10 @@ EMPTY=''\n test_expect_success 'compare mixed whitespace delta across moved blocks' '\n \n \tgit reset --hard &&\n-\ttr Q_ \"\\t \" <<-EOF >text.txt &&\n-\t${EMPTY}\n-\t____too short without\n-\t${EMPTY}\n+\ttr \"^|Q_\" \"\\f\\v\\t \" <<-EOF >text.txt &&\n+\t^__\n+\t|____too short without\n+\t^\n \t___being grouped across blank line\n \t${EMPTY}\n \tcontext\n@@ -2228,7 +2228,7 @@ test_expect_success 'compare mixed whitespace delta across moved blocks' '\n \tgit add text.txt &&\n \tgit commit -m \"add text.txt\" &&\n \n-\ttr Q_ \"\\t \" <<-EOF >text.txt &&\n+\ttr \"^|Q_\" \"\\f\\v\\t \" <<-EOF >text.txt &&\n \tcontext\n \tlines\n \tto\n@@ -2239,7 +2239,7 @@ test_expect_success 'compare mixed whitespace delta across moved blocks' '\n \t${EMPTY}\n \tQQtoo short without\n \t${EMPTY}\n-\tQ_______being grouped across blank line\n+\t^Q_______being grouped across blank line\n \t${EMPTY}\n \tQ_QThese two lines have had their\n \tindentation reduced by four spaces\n@@ -2251,16 +2251,16 @@ test_expect_success 'compare mixed whitespace delta across moved blocks' '\n \t\t-c core.whitespace=space-before-tab \\\n \t\tdiff --color --color-moved --ws-error-highlight=all \\\n \t\t--color-moved-ws=allow-indentation-change >actual.raw &&\n-\tgrep -v \"index\" actual.raw | test_decode_color >actual &&\n+\tgrep -v \"index\" actual.raw | tr \"\\f\\v\" \"^|\" | test_decode_color >actual &&\n \n \tcat <<-\\EOF >expected &&\n \tdiff --git a/text.txt b/text.txt\n \t--- a/text.txt\n \t+++ b/text.txt\n \t@@ -1,16 +1,16 @@\n-\t-\n-\t- too short without\n-\t-\n+\t-^ \n+\t-| too short without\n+\t-^\n \t- being grouped across blank line\n \t-\n \t context\n@@ -2280,7 +2280,7 @@ test_expect_success 'compare mixed whitespace delta across moved blocks' '\n \t+\n \t+\t\ttoo short without\n \t+\n-\t+\t being grouped across blank line\n+\t+^\t being grouped across blank line\n \t+\n \t+\t \tThese two lines have had their\n \t+indentation reduced by four spaces\n-- \ngitgitgadget\n\n"},{"id":"441285","messageId":"d44c5d734c3743c7468876e90e94bc6304ad3542.1637056179.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v4 14/15] diff: use designated initializers for emitted_diff_symbol","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:37Z","receivedAt":"2021-11-16T09:50:28Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nThis makes it clearer which fields are being explicitly initialized\nand will simplify the next commit where we add a new field to the\nstruct.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/diff.c b/diff.c\nindex 2085c063675..9ef88d7665a 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1497,7 +1497,9 @@ static void emit_diff_symbol_from_struct(struct diff_options *o,\n static void emit_diff_symbol(struct diff_options *o, enum diff_symbol s,\n \t\t\t const char *line, int len, unsigned flags)\n {\n-\tstruct emitted_diff_symbol e = {line, len, flags, 0, 0, s};\n+\tstruct emitted_diff_symbol e = {\n+\t\t.line = line, .len = len, .flags = flags, .s = s\n+\t};\n \n \tif (o->emitted_symbols)\n \t\tappend_emitted_diff_symbol(o, &e);\n-- \ngitgitgadget\n\n"},{"id":"441286","messageId":"5177f669423b6c17d574c17e0da660055ef7ea59.1637056179.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v4 15/15] diff --color-moved: intern strings","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T09:49:38Z","receivedAt":"2021-11-16T09:50:32Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nTaking inspiration from xdl_classify_record() assign an id to each\naddition and deletion such that lines that match for the current\n--color-moved-ws mode share the same unique id. This reduces the\nnumber of hash lookups a little (calculating the ids still involves\none hash lookup per line) but the main benefit is that when growing\nblocks of potentially moved lines we can replace string comparisons\nwhich involve chasing a pointer with a simple integer comparison. On a\nlarge diff this commit reduces the time to run 'diff --color-moved' by\n37% compared to the previous commit and 31% compared to master, for\n'diff --color-moved-ws=allow-indentation-change' the reduction is 28%\ncompared to the previous commit and 96% compared to master. There is\nlittle change in the performance of 'git log --patch' as the diffs are\nsmaller.\n\nTest HEAD^ HEAD\n---------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38(0.33+0.05) 0.38(0.33+0.05) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.88(0.81+0.06) 0.55(0.50+0.04) -37.5%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 0.85(0.79+0.06) 0.61(0.54+0.06) -28.2%\n4002.4: log --no-color-moved --no-color-moved-ws 1.16(1.07+0.08) 1.15(1.09+0.05) -0.9%\n4002.5: log --color-moved --no-color-moved-ws 1.31(1.22+0.08) 1.29(1.19+0.09) -1.5%\n4002.6: log --color-moved-ws=allow-indentation-change 1.32(1.24+0.08) 1.31(1.18+0.13) -0.8%\n\nTest master HEAD\n---------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.33+0.05) 0.38(0.33+0.05) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.80 (0.75+0.04) 0.55(0.50+0.04) -31.2%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 14.20(14.15+0.05) 0.61(0.54+0.06) -95.7%\n4002.4: log --no-color-moved --no-color-moved-ws 1.15 (1.05+0.09) 1.15(1.09+0.05) +0.0%\n4002.5: log --color-moved --no-color-moved-ws 1.30 (1.19+0.11) 1.29(1.19+0.09) -0.8%\n4002.6: log --color-moved-ws=allow-indentation-change 1.70 (1.63+0.06) 1.31(1.18+0.13) -22.9%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 174 +++++++++++++++++++++++++++++++--------------------------\n 1 file changed, 96 insertions(+), 78 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 9ef88d7665a..c28c56c1283 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -18,6 +18,7 @@\n #include \"submodule-config.h\"\n #include \"submodule.h\"\n #include \"hashmap.h\"\n+#include \"mem-pool.h\"\n #include \"ll-merge.h\"\n #include \"string-list.h\"\n #include \"strvec.h\"\n@@ -772,6 +773,7 @@ struct emitted_diff_symbol {\n \tint flags;\n \tint indent_off; /* Offset to first non-whitespace character */\n \tint indent_width; /* The visual width of the indentation */\n+\tunsigned id;\n \tenum diff_symbol s;\n };\n #define EMITTED_DIFF_SYMBOL_INIT {NULL}\n@@ -797,9 +799,9 @@ static void append_emitted_diff_symbol(struct diff_options *o,\n }\n \n struct moved_entry {\n-\tstruct hashmap_entry ent;\n \tconst struct emitted_diff_symbol *es;\n \tstruct moved_entry *next_line;\n+\tstruct moved_entry *next_match;\n };\n \n struct moved_block {\n@@ -865,24 +867,24 @@ static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n \t\t\t\t const struct emitted_diff_symbol *l,\n \t\t\t\t struct moved_block *pmb)\n {\n-\tint al = cur->es->len, bl = l->len;\n-\tconst char *a = cur->es->line,\n-\t\t *b = l->line;\n-\tint a_off = cur->es->indent_off,\n-\t a_width = cur->es->indent_width,\n-\t b_off = l->indent_off,\n-\t b_width = l->indent_width;\n+\tint a_width = cur->es->indent_width, b_width = l->indent_width;\n \tint delta;\n \n-\t/* If 'l' and 'cur' are both blank then they match. */\n-\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n+\t/* The text of each line must match */\n+\tif (cur->es->id != l->id)\n+\t\treturn 1;\n+\n+\t/*\n+\t * If 'l' and 'cur' are both blank then we don't need to check the\n+\t * indent. We only need to check cur as we know the strings match.\n+\t * */\n+\tif (a_width == INDENT_BLANKLINE)\n \t\treturn 0;\n \n \t/*\n \t * The indent changes of the block are known and stored in pmb->wsd;\n \t * however we need to check if the indent changes of the current line\n-\t * match those of the current block and that the text of 'l' and 'cur'\n-\t * after the indentation match.\n+\t * match those of the current block.\n \t */\n \tdelta = b_width - a_width;\n \n@@ -893,22 +895,26 @@ static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n \tif (pmb->wsd == INDENT_BLANKLINE)\n \t\tpmb->wsd = delta;\n \n-\treturn !(delta == pmb->wsd && al - a_off == bl - b_off &&\n-\t\t !memcmp(a + a_off, b + b_off, al - a_off));\n+\treturn delta != pmb->wsd;\n }\n \n-static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n-\t\t\t const struct hashmap_entry *eptr,\n-\t\t\t const struct hashmap_entry *entry_or_key,\n-\t\t\t const void *keydata)\n+struct interned_diff_symbol {\n+\tstruct hashmap_entry ent;\n+\tstruct emitted_diff_symbol *es;\n+};\n+\n+static int interned_diff_symbol_cmp(const void *hashmap_cmp_fn_data,\n+\t\t\t\t const struct hashmap_entry *eptr,\n+\t\t\t\t const struct hashmap_entry *entry_or_key,\n+\t\t\t\t const void *keydata)\n {\n \tconst struct diff_options *diffopt = hashmap_cmp_fn_data;\n \tconst struct emitted_diff_symbol *a, *b;\n \tunsigned flags = diffopt->color_moved_ws_handling\n \t\t\t & XDF_WHITESPACE_FLAGS;\n \n-\ta = container_of(eptr, const struct moved_entry, ent)->es;\n-\tb = container_of(entry_or_key, const struct moved_entry, ent)->es;\n+\ta = container_of(eptr, const struct interned_diff_symbol, ent)->es;\n+\tb = container_of(entry_or_key, const struct interned_diff_symbol, ent)->es;\n \n \treturn !xdiff_compare_lines(a->line + a->indent_off,\n \t\t\t\t a->len - a->indent_off,\n@@ -916,55 +922,81 @@ static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n \t\t\t\t b->len - b->indent_off, flags);\n }\n \n-static struct moved_entry *prepare_entry(struct diff_options *o,\n-\t\t\t\t\t int line_no)\n+static void prepare_entry(struct diff_options *o, struct emitted_diff_symbol *l,\n+\t\t\t struct interned_diff_symbol *s)\n {\n-\tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n-\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[line_no];\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \tunsigned int hash = xdiff_hash_string(l->line + l->indent_off,\n \t\t\t\t\t l->len - l->indent_off, flags);\n \n-\thashmap_entry_init(&ret->ent, hash);\n-\tret->es = l;\n-\tret->next_line = NULL;\n-\n-\treturn ret;\n+\thashmap_entry_init(&s->ent, hash);\n+\ts->es = l;\n }\n \n-static void add_lines_to_move_detection(struct diff_options *o,\n-\t\t\t\t\tstruct hashmap *add_lines,\n-\t\t\t\t\tstruct hashmap *del_lines)\n+struct moved_entry_list {\n+\tstruct moved_entry *add, *del;\n+};\n+\n+static struct moved_entry_list *add_lines_to_move_detection(struct diff_options *o,\n+\t\t\t\t\t\t\t struct mem_pool *entry_mem_pool)\n {\n \tstruct moved_entry *prev_line = NULL;\n-\n+\tstruct mem_pool interned_pool;\n+\tstruct hashmap interned_map;\n+\tstruct moved_entry_list *entry_list = NULL;\n+\tsize_t entry_list_alloc = 0;\n+\tunsigned id = 0;\n \tint n;\n+\n+\thashmap_init(&interned_map, interned_diff_symbol_cmp, o, 8096);\n+\tmem_pool_init(&interned_pool, 1024 * 1024);\n+\n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n-\t\tstruct hashmap *hm;\n-\t\tstruct moved_entry *key;\n+\t\tstruct interned_diff_symbol key;\n+\t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n+\t\tstruct interned_diff_symbol *s;\n+\t\tstruct moved_entry *entry;\n \n-\t\tswitch (o->emitted_symbols->buf[n].s) {\n-\t\tcase DIFF_SYMBOL_PLUS:\n-\t\t\thm = add_lines;\n-\t\t\tbreak;\n-\t\tcase DIFF_SYMBOL_MINUS:\n-\t\t\thm = del_lines;\n-\t\t\tbreak;\n-\t\tdefault:\n+\t\tif (l->s != DIFF_SYMBOL_PLUS && l->s != DIFF_SYMBOL_MINUS) {\n \t\t\tprev_line = NULL;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (o->color_moved_ws_handling &\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\tfill_es_indent_data(&o->emitted_symbols->buf[n]);\n-\t\tkey = prepare_entry(o, n);\n-\t\tif (prev_line && prev_line->es->s == o->emitted_symbols->buf[n].s)\n-\t\t\tprev_line->next_line = key;\n+\t\t\tfill_es_indent_data(l);\n \n-\t\thashmap_add(hm, &key->ent);\n-\t\tprev_line = key;\n+\t\tprepare_entry(o, l, &key);\n+\t\ts = hashmap_get_entry(&interned_map, &key, ent, &key.ent);\n+\t\tif (s) {\n+\t\t\tl->id = s->es->id;\n+\t\t} else {\n+\t\t\tl->id = id;\n+\t\t\tALLOC_GROW_BY(entry_list, id, 1, entry_list_alloc);\n+\t\t\thashmap_add(&interned_map,\n+\t\t\t\t memcpy(mem_pool_alloc(&interned_pool,\n+\t\t\t\t\t\t\t sizeof(key)),\n+\t\t\t\t\t &key, sizeof(key)));\n+\t\t}\n+\t\tentry = mem_pool_alloc(entry_mem_pool, sizeof(*entry));\n+\t\tentry->es = l;\n+\t\tentry->next_line = NULL;\n+\t\tif (prev_line && prev_line->es->s == l->s)\n+\t\t\tprev_line->next_line = entry;\n+\t\tprev_line = entry;\n+\t\tif (l->s == DIFF_SYMBOL_PLUS) {\n+\t\t\tentry->next_match = entry_list[l->id].add;\n+\t\t\tentry_list[l->id].add = entry;\n+\t\t} else {\n+\t\t\tentry->next_match = entry_list[l->id].del;\n+\t\t\tentry_list[l->id].del = entry;\n+\t\t}\n \t}\n+\n+\thashmap_clear(&interned_map);\n+\tmem_pool_discard(&interned_pool, 0);\n+\n+\treturn entry_list;\n }\n \n static void pmb_advance_or_null(struct diff_options *o,\n@@ -973,7 +1005,6 @@ static void pmb_advance_or_null(struct diff_options *o,\n \t\t\t\tint *pmb_nr)\n {\n \tint i, j;\n-\tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \n \tfor (i = 0, j = 0; i < *pmb_nr; i++) {\n \t\tint match;\n@@ -986,9 +1017,8 @@ static void pmb_advance_or_null(struct diff_options *o,\n \t\t\tmatch = cur &&\n \t\t\t\t!cmp_in_block_with_wsd(cur, l, &pmb[i]);\n \t\telse\n-\t\t\tmatch = cur &&\n-\t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n-\t\t\t\t\t\t l->line, l->len, flags);\n+\t\t\tmatch = cur && cur->es->id == l->id;\n+\n \t\tif (match) {\n \t\t\tpmb[j] = pmb[i];\n \t\t\tpmb[j++].match = cur;\n@@ -998,7 +1028,6 @@ static void pmb_advance_or_null(struct diff_options *o,\n }\n \n static void fill_potential_moved_blocks(struct diff_options *o,\n-\t\t\t\t\tstruct hashmap *hm,\n \t\t\t\t\tstruct moved_entry *match,\n \t\t\t\t\tstruct emitted_diff_symbol *l,\n \t\t\t\t\tstruct moved_block **pmb_p,\n@@ -1012,7 +1041,7 @@ static void fill_potential_moved_blocks(struct diff_options *o,\n \t * The current line is the start of a new block.\n \t * Setup the set of potential blocks.\n \t */\n-\thashmap_for_each_entry_from(hm, match, ent) {\n+\tfor (; match; match = match->next_match) {\n \t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n \t\tif (o->color_moved_ws_handling &\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n@@ -1067,8 +1096,7 @@ static int adjust_last_block(struct diff_options *o, int n, int block_length)\n \n /* Find blocks of moved code, delegate actual coloring decision to helper */\n static void mark_color_as_moved(struct diff_options *o,\n-\t\t\t\tstruct hashmap *add_lines,\n-\t\t\t\tstruct hashmap *del_lines)\n+\t\t\t\tstruct moved_entry_list *entry_list)\n {\n \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n@@ -1077,23 +1105,15 @@ static void mark_color_as_moved(struct diff_options *o,\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n-\t\tstruct hashmap *hm = NULL;\n-\t\tstruct moved_entry *key;\n \t\tstruct moved_entry *match = NULL;\n \t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n \n \t\tswitch (l->s) {\n \t\tcase DIFF_SYMBOL_PLUS:\n-\t\t\thm = del_lines;\n-\t\t\tkey = prepare_entry(o, n);\n-\t\t\tmatch = hashmap_get_entry(hm, key, ent, NULL);\n-\t\t\tfree(key);\n+\t\t\tmatch = entry_list[l->id].del;\n \t\t\tbreak;\n \t\tcase DIFF_SYMBOL_MINUS:\n-\t\t\thm = add_lines;\n-\t\t\tkey = prepare_entry(o, n);\n-\t\t\tmatch = hashmap_get_entry(hm, key, ent, NULL);\n-\t\t\tfree(key);\n+\t\t\tmatch = entry_list[l->id].add;\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tflipped_block = 0;\n@@ -1135,7 +1155,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t\t */\n \t\t\t\tn -= block_length;\n \t\t\telse\n-\t\t\t\tfill_potential_moved_blocks(o, hm, match, l,\n+\t\t\t\tfill_potential_moved_blocks(o, match, l,\n \t\t\t\t\t\t\t &pmb, &pmb_alloc,\n \t\t\t\t\t\t\t &pmb_nr);\n \n@@ -6253,20 +6273,18 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \n \tif (o->emitted_symbols) {\n \t\tif (o->color_moved) {\n-\t\t\tstruct hashmap add_lines, del_lines;\n-\n-\t\t\thashmap_init(&del_lines, moved_entry_cmp, o, 0);\n-\t\t\thashmap_init(&add_lines, moved_entry_cmp, o, 0);\n+\t\t\tstruct mem_pool entry_pool;\n+\t\t\tstruct moved_entry_list *entry_list;\n \n-\t\t\tadd_lines_to_move_detection(o, &add_lines, &del_lines);\n-\t\t\tmark_color_as_moved(o, &add_lines, &del_lines);\n+\t\t\tmem_pool_init(&entry_pool, 1024 * 1024);\n+\t\t\tentry_list = add_lines_to_move_detection(o,\n+\t\t\t\t\t\t\t\t &entry_pool);\n+\t\t\tmark_color_as_moved(o, entry_list);\n \t\t\tif (o->color_moved == COLOR_MOVED_ZEBRA_DIM)\n \t\t\t\tdim_moved_lines(o);\n \n-\t\t\thashmap_clear_and_free(&add_lines, struct moved_entry,\n-\t\t\t\t\t\tent);\n-\t\t\thashmap_clear_and_free(&del_lines, struct moved_entry,\n-\t\t\t\t\t\tent);\n+\t\t\tmem_pool_discard(&entry_pool, 0);\n+\t\t\tfree(entry_list);\n \t\t}\n \n \t\tfor (i = 0; i < esm.nr; i++)\n-- \ngitgitgadget\n"},{"id":"441946","messageId":"nycvar.QRO.7.76.6.2111221431350.63@tvgsbejvaqbjf.bet","threadId":"55926","inReplyTo":"56bb69af36e5e3180d53586d30048d5033a01d14.1637056178.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 05/15] diff --color-moved=zebra: fix alternate coloring","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-11-22T13:34:52Z","receivedAt":"2021-11-22T13:35:04Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Phillip,\n\nOn Tue, 16 Nov 2021, Phillip Wood via GitGitGadget wrote:\n\n> From: Phillip Wood \n>\n> b0a2ba4776 (\"diff --color-moved=zebra: be stricter with color\n> alternation\", 2018-11-23) sought to avoid using the alternate colors\n> unless there are two adjacent moved blocks of the same\n> sign. Unfortunately it contains two bugs that prevented it from fixing\n> the problem properly. Firstly `last_symbol` is reset at the start of\n> each iteration of the loop losing the symbol of the last line and\n> secondly when deciding whether to use the alternate color it should be\n> checking if the current line is the same sign of the last line, not a\n> different sign. The combination of the two errors means that we still\n> use the alternate color when we should do but we also use it when we\n> shouldn't. This is most noticable when using\n> --color-moved-ws=allow-indentation-change with hunks like\n>\n> -this line gets indented\n> + this line gets indented\n>\n> where the post image is colored with newMovedAlternate rather than\n> newMoved. While this does not matter much, the next commit will change\n> the coloring to be correct in this case, so lets fix the bug here to\n> make it clear why the output is changing and add a regression test.\n\nWhat an excellent commit message!\n\nThank you,\nDscho\n\n>\n> Signed-off-by: Phillip Wood \n> ---\n> diff.c | 4 +--\n> t/t4015-diff-whitespace.sh | 72 ++++++++++++++++++++++++++++++++++++++\n> 2 files changed, 74 insertions(+), 2 deletions(-)\n>\n> diff --git a/diff.c b/diff.c\n> index 1e1b5127d15..53f0df75329 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -1176,6 +1176,7 @@ static void mark_color_as_moved(struct diff_options *o,\n> \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n> \tint pmb_nr = 0, pmb_alloc = 0;\n> \tint n, flipped_block = 0, block_length = 0;\n> +\tenum diff_symbol last_symbol = 0;\n>\n>\n> \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n> @@ -1183,7 +1184,6 @@ static void mark_color_as_moved(struct diff_options *o,\n> \t\tstruct moved_entry *key;\n> \t\tstruct moved_entry *match = NULL;\n> \t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n> -\t\tenum diff_symbol last_symbol = 0;\n>\n> \t\tswitch (l->s) {\n> \t\tcase DIFF_SYMBOL_PLUS:\n> @@ -1251,7 +1251,7 @@ static void mark_color_as_moved(struct diff_options *o,\n> \t\t\t\t\t\t\t &pmb, &pmb_alloc,\n> \t\t\t\t\t\t\t &pmb_nr);\n>\n> -\t\t\tif (contiguous && pmb_nr && last_symbol != l->s)\n> +\t\t\tif (contiguous && pmb_nr && last_symbol == l->s)\n> \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n> \t\t\telse\n> \t\t\t\tflipped_block = 0;\n> diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\n> index 308dc136596..4e0fd76c6c5 100755\n> --- a/t/t4015-diff-whitespace.sh\n> +++ b/t/t4015-diff-whitespace.sh\n> @@ -1442,6 +1442,78 @@ test_expect_success 'detect permutations inside moved code -- dimmed-zebra' '\n> \ttest_cmp expected actual\n> '\n>\n> +test_expect_success 'zebra alternate color is only used when necessary' '\n> +\tcat >old.txt <<-\\EOF &&\n> +\tline 1A should be marked as oldMoved newMovedAlternate\n> +\tline 1B should be marked as oldMoved newMovedAlternate\n> +\tunchanged\n> +\tline 2A should be marked as oldMoved newMovedAlternate\n> +\tline 2B should be marked as oldMoved newMovedAlternate\n> +\tline 3A should be marked as oldMovedAlternate newMoved\n> +\tline 3B should be marked as oldMovedAlternate newMoved\n> +\tunchanged\n> +\tline 4A should be marked as oldMoved newMovedAlternate\n> +\tline 4B should be marked as oldMoved newMovedAlternate\n> +\tline 5A should be marked as oldMovedAlternate newMoved\n> +\tline 5B should be marked as oldMovedAlternate newMoved\n> +\tline 6A should be marked as oldMoved newMoved\n> +\tline 6B should be marked as oldMoved newMoved\n> +\tEOF\n> +\tcat >new.txt <<-\\EOF &&\n> +\t line 1A should be marked as oldMoved newMovedAlternate\n> +\t line 1B should be marked as oldMoved newMovedAlternate\n> +\tunchanged\n> +\t line 3A should be marked as oldMovedAlternate newMoved\n> +\t line 3B should be marked as oldMovedAlternate newMoved\n> +\t line 2A should be marked as oldMoved newMovedAlternate\n> +\t line 2B should be marked as oldMoved newMovedAlternate\n> +\tunchanged\n> +\t line 6A should be marked as oldMoved newMoved\n> +\t line 6B should be marked as oldMoved newMoved\n> +\t line 4A should be marked as oldMoved newMovedAlternate\n> +\t line 4B should be marked as oldMoved newMovedAlternate\n> +\t line 5A should be marked as oldMovedAlternate newMoved\n> +\t line 5B should be marked as oldMovedAlternate newMoved\n> +\tEOF\n> +\ttest_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n> +\t\t --color-moved-ws=allow-indentation-change \\\n> +\t\t old.txt new.txt >output &&\n> +\tgrep -v index output | test_decode_color >actual &&\n> +\tcat >expected <<-\\EOF &&\n> +\tdiff --git a/old.txt b/new.txt\n> +\t--- a/old.txt\n> +\t+++ b/new.txt\n> +\t@@ -1,14 +1,14 @@\n> +\t-line 1A should be marked as oldMoved newMovedAlternate\n> +\t-line 1B should be marked as oldMoved newMovedAlternate\n> +\t+ line 1A should be marked as oldMoved newMovedAlternate\n> +\t+ line 1B should be marked as oldMoved newMovedAlternate\n> +\t unchanged\n> +\t-line 2A should be marked as oldMoved newMovedAlternate\n> +\t-line 2B should be marked as oldMoved newMovedAlternate\n> +\t-line 3A should be marked as oldMovedAlternate newMoved\n> +\t-line 3B should be marked as oldMovedAlternate newMoved\n> +\t+ line 3A should be marked as oldMovedAlternate newMoved\n> +\t+ line 3B should be marked as oldMovedAlternate newMoved\n> +\t+ line 2A should be marked as oldMoved newMovedAlternate\n> +\t+ line 2B should be marked as oldMoved newMovedAlternate\n> +\t unchanged\n> +\t-line 4A should be marked as oldMoved newMovedAlternate\n> +\t-line 4B should be marked as oldMoved newMovedAlternate\n> +\t-line 5A should be marked as oldMovedAlternate newMoved\n> +\t-line 5B should be marked as oldMovedAlternate newMoved\n> +\t-line 6A should be marked as oldMoved newMoved\n> +\t-line 6B should be marked as oldMoved newMoved\n> +\t+ line 6A should be marked as oldMoved newMoved\n> +\t+ line 6B should be marked as oldMoved newMoved\n> +\t+ line 4A should be marked as oldMoved newMovedAlternate\n> +\t+ line 4B should be marked as oldMoved newMovedAlternate\n> +\t+ line 5A should be marked as oldMovedAlternate newMoved\n> +\t+ line 5B should be marked as oldMovedAlternate newMoved\n> +\tEOF\n> +\ttest_cmp expected actual\n> +'\n> +\n> test_expect_success 'cmd option assumes configured colored-moved' '\n> \ttest_config color.diff.oldMoved \"magenta\" &&\n> \ttest_config color.diff.newMoved \"cyan\" &&\n> --\n> gitgitgadget\n>\n>\n"},{"id":"441947","messageId":"nycvar.QRO.7.76.6.2111221435140.63@tvgsbejvaqbjf.bet","threadId":"55926","inReplyTo":"10b11526206d3b515ba08ac80ccf09ecb7a03420.1637056178.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 06/15] diff --color-moved: avoid false short line matches and bad zerba coloring","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-11-22T14:18:34Z","receivedAt":"2021-11-22T14:18:47Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Phillip,\n\nThe commit's oneline has a typo: zerba instead of zebra.\n\nOn Tue, 16 Nov 2021, Phillip Wood via GitGitGadget wrote:\n\n> From: Phillip Wood \n>\n> When marking moved lines it is possible for a block of potential\n> matched lines to extend past a change in sign when there is a sequence\n> of added lines whose text matches the text of a sequence of deleted\n> and added lines. Most of the time either `match` will be NULL or\n> `pmb_advance_or_null()` will fail when the loop encounters a change of\n> sign but there are corner cases where `match` is non-NULL and\n> `pmb_advance_or_null()` successfully advances the moved block despite\n> the change in sign.\n>\n> One consequence of this is highlighting a short line as moved when it\n> should not be. For example\n>\n> -moved line # Correctly highlighted as moved\n> +short line # Wrongly highlighted as moved\n> context\n> +moved line # Correctly highlighted as moved\n> +short line\n> context\n> -short line\n>\n> The other consequence is coloring a moved addition following a moved\n> deletion in the wrong color. In the example below the first \"+moved\n> line 3\" should be highlighted as newMoved not newMovedAlternate.\n>\n> -moved line 1 # Correctly highlighted as oldMoved\n> -moved line 2 # Correctly highlighted as oldMovedAlternate\n> +moved line 3 # Wrongly highlighted as newMovedAlternate\n> context # Everything else is highlighted correctly\n> +moved line 2\n> +moved line 3\n> context\n> +moved line 1\n> -moved line 3\n>\n> These false matches are more likely when using --color-moved-ws with\n> the exception of --color-moved-ws=allow-indentation-change which ties\n> the sign of the current whitespace delta to the sign of the line to\n> avoid this problem. The fix is to check that the sign of the new line\n> being matched is the same as the sign of the line that started the\n> block of potential matches.\n>\n> Signed-off-by: Phillip Wood \n> ---\n> diff.c | 17 ++++++----\n> t/t4015-diff-whitespace.sh | 65 ++++++++++++++++++++++++++++++++++++++\n> 2 files changed, 76 insertions(+), 6 deletions(-)\n>\n> diff --git a/diff.c b/diff.c\n> index 53f0df75329..efba2789354 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -1176,7 +1176,7 @@ static void mark_color_as_moved(struct diff_options *o,\n> \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n> \tint pmb_nr = 0, pmb_alloc = 0;\n> \tint n, flipped_block = 0, block_length = 0;\n> -\tenum diff_symbol last_symbol = 0;\n> +\tenum diff_symbol moved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n\nThe exact value does not matter, as long as it is different from whatever\nthe next line will have, of course.\n\n>\n>\n> \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n> @@ -1202,7 +1202,7 @@ static void mark_color_as_moved(struct diff_options *o,\n> \t\t\tflipped_block = 0;\n> \t\t}\n>\n> -\t\tif (!match) {\n> +\t\tif (pmb_nr && (!match || l->s != moved_symbol)) {\n> \t\t\tint i;\n>\n> \t\t\tif (!adjust_last_block(o, n, block_length) &&\n> @@ -1219,12 +1219,13 @@ static void mark_color_as_moved(struct diff_options *o,\n> \t\t\tpmb_nr = 0;\n> \t\t\tblock_length = 0;\n> \t\t\tflipped_block = 0;\n> -\t\t\tlast_symbol = l->s;\n> +\t\t}\n\nThis is one of those instances where I dislike having the patch in a\nstatic mail. I so want to have a _convenient_ way to expand the diff\ncontext, to have a look around.\n\nSo I went over to\nhttps://github.com/gitgitgadget/git/pull/981/commits/10b11526206d3b515ba08ac80ccf09ecb7a03420\nto get the convenience I need for a pleasant reviewing experience.\n\nIn this instance, the `continue` that dropped out of that conditional\nblock gave me pause.\n\nMy understanding is that the diff makes it essentially a lot harder to\nunderstand what is done here: this conditional block did two things, it\nre-set the possibly-moved-block, and it skipped to the next loop\niteration. With this patch, we now re-set the possibly-moved-block in more\ncases, but still skip to the next loop iteration under the same condition\nas before:\n\n> +\t\tif (!match) {\n> +\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n> \t\t\tcontinue;\n> \t\t}\n\nHowever, after reading the commit message, I would have expected the\ncondition above to read `if (!match || l->s != moved_symbol)` instead of\n`if (!match)`. Could you help me understand what I am missing?\n\n>\n> \t\tif (o->color_moved == COLOR_MOVED_PLAIN) {\n> -\t\t\tlast_symbol = l->s;\n> \t\t\tl->flags |= DIFF_SYMBOL_MOVED_LINE;\n> \t\t\tcontinue;\n> \t\t}\n\nI want to make sure that I understand why the `last_symbol` assignment\ncould be removed without any `moved_symbol` assignment in its place. But I\ndon't, I still do not see why we do not need a `moved_symbol = l->s;`\nassignment here.\n\nUnless, that is, we extended the `!match` condition above to also cover\nthe case where `l->s != moved_symbol`.\n\n> @@ -1251,11 +1252,16 @@ static void mark_color_as_moved(struct diff_options *o,\n> \t\t\t\t\t\t\t &pmb, &pmb_alloc,\n> \t\t\t\t\t\t\t &pmb_nr);\n>\n> -\t\t\tif (contiguous && pmb_nr && last_symbol == l->s)\n> +\t\t\tif (contiguous && pmb_nr && moved_symbol == l->s)\n> \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n\nThis is totally not your fault, but I really wish we could have the much\nsimpler and much easier to understand `flipped_block = !flipped_block`\nhere.\n\n> \t\t\telse\n> \t\t\t\tflipped_block = 0;\n>\n> +\t\t\tif (pmb_nr)\n> +\t\t\t\tmoved_symbol = l->s;\n> +\t\t\telse\n> +\t\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n> +\n> \t\t\tblock_length = 0;\n> \t\t}\n>\n> @@ -1265,7 +1271,6 @@ static void mark_color_as_moved(struct diff_options *o,\n> \t\t\tif (flipped_block && o->color_moved != COLOR_MOVED_BLOCKS)\n> \t\t\t\tl->flags |= DIFF_SYMBOL_MOVED_LINE_ALT;\n> \t\t}\n> -\t\tlast_symbol = l->s;\n\nThat makes sense: we only set `moved_symbol` when `pmb_nr` had been 0 now,\nand don't want it to be overridden.\n\nAs I said, I do not quite understand this patch yet, and am looking for\nyour guidance to wrap my head around it.\n\nThank you for working on this!\nDscho\n\n> \t}\n> \tadjust_last_block(o, n, block_length);\n>\n> diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\n> index 4e0fd76c6c5..15782c879d2 100755\n> --- a/t/t4015-diff-whitespace.sh\n> +++ b/t/t4015-diff-whitespace.sh\n> @@ -1514,6 +1514,71 @@ test_expect_success 'zebra alternate color is only used when necessary' '\n> \ttest_cmp expected actual\n> '\n>\n> +test_expect_success 'short lines of opposite sign do not get marked as moved' '\n> +\tcat >old.txt <<-\\EOF &&\n> +\tthis line should be marked as moved\n> +\tunchanged\n> +\tunchanged\n> +\tunchanged\n> +\tunchanged\n> +\ttoo short\n> +\tthis line should be marked as oldMoved newMoved\n> +\tthis line should be marked as oldMovedAlternate newMoved\n> +\tunchanged 1\n> +\tunchanged 2\n> +\tunchanged 3\n> +\tunchanged 4\n> +\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n> +\tEOF\n> +\tcat >new.txt <<-\\EOF &&\n> +\ttoo short\n> +\tunchanged\n> +\tunchanged\n> +\tthis line should be marked as moved\n> +\ttoo short\n> +\tunchanged\n> +\tunchanged\n> +\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n> +\tunchanged 1\n> +\tunchanged 2\n> +\tthis line should be marked as oldMovedAlternate newMoved\n> +\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n> +\tunchanged 3\n> +\tthis line should be marked as oldMoved newMoved\n> +\tunchanged 4\n> +\tEOF\n> +\ttest_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n> +\t\told.txt new.txt >output && cat output &&\n> +\tgrep -v index output | test_decode_color >actual &&\n> +\tcat >expect <<-\\EOF &&\n> +\tdiff --git a/old.txt b/new.txt\n> +\t--- a/old.txt\n> +\t+++ b/new.txt\n> +\t@@ -1,13 +1,15 @@\n> +\t-this line should be marked as moved\n> +\t+too short\n> +\t unchanged\n> +\t unchanged\n> +\t+this line should be marked as moved\n> +\t+too short\n> +\t unchanged\n> +\t unchanged\n> +\t-too short\n> +\t-this line should be marked as oldMoved newMoved\n> +\t-this line should be marked as oldMovedAlternate newMoved\n> +\t+this line should be marked as oldMoved newMoved/newMovedAlternate\n> +\t unchanged 1\n> +\t unchanged 2\n> +\t+this line should be marked as oldMovedAlternate newMoved\n> +\t+this line should be marked as oldMoved newMoved/newMovedAlternate\n> +\t unchanged 3\n> +\t+this line should be marked as oldMoved newMoved\n> +\t unchanged 4\n> +\t-this line should be marked as oldMoved newMoved/newMovedAlternate\n> +\tEOF\n> +\ttest_cmp expect actual\n> +'\n> +\n> test_expect_success 'cmd option assumes configured colored-moved' '\n> \ttest_config color.diff.oldMoved \"magenta\" &&\n> \ttest_config color.diff.newMoved \"cyan\" &&\n> --\n> gitgitgadget\n>\n>\n"},{"id":"442011","messageId":"a6ff589e-c968-8a96-a8ec-1a982d71f6be@gmail.com","threadId":"55926","inReplyTo":"nycvar.QRO.7.76.6.2111221435140.63@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v4 06/15] diff --color-moved: avoid false short line matches and bad zerba coloring","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-11-22T19:00:07Z","receivedAt":"2021-11-22T19:00:28Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Dscho\n\nThanks ever so much for taking a detailed look at this series.\n\nOn 22/11/2021 14:18, Johannes Schindelin wrote:\n> Hi Phillip,\n> \n> The commit's oneline has a typo: zerba instead of zebra.\n\nSigh, I thought I'd fixed that\n\n> On Tue, 16 Nov 2021, Phillip Wood via GitGitGadget wrote:\n> \n>> From: Phillip Wood \n>>\n>> When marking moved lines it is possible for a block of potential\n>> matched lines to extend past a change in sign when there is a sequence\n>> of added lines whose text matches the text of a sequence of deleted\n>> and added lines. Most of the time either `match` will be NULL or\n>> `pmb_advance_or_null()` will fail when the loop encounters a change of\n>> sign but there are corner cases where `match` is non-NULL and\n>> `pmb_advance_or_null()` successfully advances the moved block despite\n>> the change in sign.\n>>\n>> One consequence of this is highlighting a short line as moved when it\n>> should not be. For example\n>>\n>> -moved line # Correctly highlighted as moved\n>> +short line # Wrongly highlighted as moved\n>> context\n>> +moved line # Correctly highlighted as moved\n>> +short line\n>> context\n>> -short line\n>>\n>> The other consequence is coloring a moved addition following a moved\n>> deletion in the wrong color. In the example below the first \"+moved\n>> line 3\" should be highlighted as newMoved not newMovedAlternate.\n>>\n>> -moved line 1 # Correctly highlighted as oldMoved\n>> -moved line 2 # Correctly highlighted as oldMovedAlternate\n>> +moved line 3 # Wrongly highlighted as newMovedAlternate\n>> context # Everything else is highlighted correctly\n>> +moved line 2\n>> +moved line 3\n>> context\n>> +moved line 1\n>> -moved line 3\n>>\n>> These false matches are more likely when using --color-moved-ws with\n>> the exception of --color-moved-ws=allow-indentation-change which ties\n>> the sign of the current whitespace delta to the sign of the line to\n>> avoid this problem. The fix is to check that the sign of the new line\n>> being matched is the same as the sign of the line that started the\n>> block of potential matches.\n>>\n>> Signed-off-by: Phillip Wood \n>> ---\n>> diff.c | 17 ++++++----\n>> t/t4015-diff-whitespace.sh | 65 ++++++++++++++++++++++++++++++++++++++\n>> 2 files changed, 76 insertions(+), 6 deletions(-)\n>>\n>> diff --git a/diff.c b/diff.c\n>> index 53f0df75329..efba2789354 100644\n>> --- a/diff.c\n>> +++ b/diff.c\n>> @@ -1176,7 +1176,7 @@ static void mark_color_as_moved(struct diff_options *o,\n>> \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n>> \tint pmb_nr = 0, pmb_alloc = 0;\n>> \tint n, flipped_block = 0, block_length = 0;\n>> -\tenum diff_symbol last_symbol = 0;\n>> +\tenum diff_symbol moved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n> \n> The exact value does not matter, as long as it is different from whatever\n> the next line will have, of course.\n> \n>>\n>>\n>> \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n>> @@ -1202,7 +1202,7 @@ static void mark_color_as_moved(struct diff_options *o,\n>> \t\t\tflipped_block = 0;\n>> \t\t}\n>>\n>> -\t\tif (!match) {\n>> +\t\tif (pmb_nr && (!match || l->s != moved_symbol)) {\n>> \t\t\tint i;\n>>\n>> \t\t\tif (!adjust_last_block(o, n, block_length) &&\n>> @@ -1219,12 +1219,13 @@ static void mark_color_as_moved(struct diff_options *o,\n>> \t\t\tpmb_nr = 0;\n>> \t\t\tblock_length = 0;\n>> \t\t\tflipped_block = 0;\n>> -\t\t\tlast_symbol = l->s;\n>> +\t\t}\n> \n> This is one of those instances where I dislike having the patch in a\n> static mail. I so want to have a _convenient_ way to expand the diff\n> context, to have a look around.\n> \n> So I went over to\n> https://github.com/gitgitgadget/git/pull/981/commits/10b11526206d3b515ba08ac80ccf09ecb7a03420\n> to get the convenience I need for a pleasant reviewing experience.\n> \n> In this instance, the `continue` that dropped out of that conditional\n> block gave me pause.\n> \n> My understanding is that the diff makes it essentially a lot harder to\n> understand what is done here: this conditional block did two things, it\n> re-set the possibly-moved-block, and it skipped to the next loop\n> iteration. With this patch, we now re-set the possibly-moved-block in more\n> cases, but still skip to the next loop iteration under the same condition\n> as before:\n> \n>> +\t\tif (!match) {\n>> +\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n>> \t\t\tcontinue;\n>> \t\t}\n> \n> However, after reading the commit message, I would have expected the\n> condition above to read `if (!match || l->s != moved_symbol)` instead of\n> `if (!match)`. Could you help me understand what I am missing?\n\nIf there is a match we want to carry on executing the body of the loop \nto start a new block of moved lines. moved_symbol will be updated at the \nend of the loop.\n\n>>\n>> \t\tif (o->color_moved == COLOR_MOVED_PLAIN) {\n>> -\t\t\tlast_symbol = l->s;\n>> \t\t\tl->flags |= DIFF_SYMBOL_MOVED_LINE;\n>> \t\t\tcontinue;\n>> \t\t}\n> \n> I want to make sure that I understand why the `last_symbol` assignment\n> could be removed without any `moved_symbol` assignment in its place. But I\n> don't, I still do not see why we do not need a `moved_symbol = l->s;`\n> assignment here.\n\nI had to think about it but I think the answer is that COLOR_MOVED_PLAIN \ndoes not care about moved_symbol - it is only used by the zebra coloring \nmodes.\n\n> Unless, that is, we extended the `!match` condition above to also cover\n> the case where `l->s != moved_symbol`.\n> \n>> @@ -1251,11 +1252,16 @@ static void mark_color_as_moved(struct diff_options *o,\n>> \t\t\t\t\t\t\t &pmb, &pmb_alloc,\n>> \t\t\t\t\t\t\t &pmb_nr);\n>>\n>> -\t\t\tif (contiguous && pmb_nr && last_symbol == l->s)\n>> +\t\t\tif (contiguous && pmb_nr && moved_symbol == l->s)\n>> \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n> \n> This is totally not your fault, but I really wish we could have the much\n> simpler and much easier to understand `flipped_block = !flipped_block`\n> here.\n\nIt's partially my fault - I should have simplified it when I moved that \nline in b0a2ba4776 (\"diff --color-moved=zebra: be stricter with color \nalternation\", 2018-11-23)\n\n>> \t\t\telse\n>> \t\t\t\tflipped_block = 0;\n>>\n>> +\t\t\tif (pmb_nr)\n>> +\t\t\t\tmoved_symbol = l->s;\n>> +\t\t\telse\n>> +\t\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n>> +\n\nThis is where we update moved_symbol when it did not match l->s above.\n\n \t\t\tblock_length = 0;\n>> \t\t}\n>>\n>> @@ -1265,7 +1271,6 @@ static void mark_color_as_moved(struct diff_options *o,\n>> \t\t\tif (flipped_block && o->color_moved != COLOR_MOVED_BLOCKS)\n>> \t\t\t\tl->flags |= DIFF_SYMBOL_MOVED_LINE_ALT;\n>> \t\t}\n>> -\t\tlast_symbol = l->s;\n> \n> That makes sense: we only set `moved_symbol` when `pmb_nr` had been 0 now,\n> and don't want it to be overridden.\n >\n> As I said, I do not quite understand this patch yet, and am looking for\n> your guidance to wrap my head around it.\n> \n> Thank you for working on this!\n\nThanks for looking at it, I hope these comments help, let me know if \nI've failed to explain well enough.\n\nBest Wishes\n\nPhillip\n\n> Dscho\n> \n>> \t}\n>> \tadjust_last_block(o, n, block_length);\n>>\n>> diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\n>> index 4e0fd76c6c5..15782c879d2 100755\n>> --- a/t/t4015-diff-whitespace.sh\n>> +++ b/t/t4015-diff-whitespace.sh\n>> @@ -1514,6 +1514,71 @@ test_expect_success 'zebra alternate color is only used when necessary' '\n>> \ttest_cmp expected actual\n>> '\n>>\n>> +test_expect_success 'short lines of opposite sign do not get marked as moved' '\n>> +\tcat >old.txt <<-\\EOF &&\n>> +\tthis line should be marked as moved\n>> +\tunchanged\n>> +\tunchanged\n>> +\tunchanged\n>> +\tunchanged\n>> +\ttoo short\n>> +\tthis line should be marked as oldMoved newMoved\n>> +\tthis line should be marked as oldMovedAlternate newMoved\n>> +\tunchanged 1\n>> +\tunchanged 2\n>> +\tunchanged 3\n>> +\tunchanged 4\n>> +\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n>> +\tEOF\n>> +\tcat >new.txt <<-\\EOF &&\n>> +\ttoo short\n>> +\tunchanged\n>> +\tunchanged\n>> +\tthis line should be marked as moved\n>> +\ttoo short\n>> +\tunchanged\n>> +\tunchanged\n>> +\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n>> +\tunchanged 1\n>> +\tunchanged 2\n>> +\tthis line should be marked as oldMovedAlternate newMoved\n>> +\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n>> +\tunchanged 3\n>> +\tthis line should be marked as oldMoved newMoved\n>> +\tunchanged 4\n>> +\tEOF\n>> +\ttest_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n>> +\t\told.txt new.txt >output && cat output &&\n>> +\tgrep -v index output | test_decode_color >actual &&\n>> +\tcat >expect <<-\\EOF &&\n>> +\tdiff --git a/old.txt b/new.txt\n>> +\t--- a/old.txt\n>> +\t+++ b/new.txt\n>> +\t@@ -1,13 +1,15 @@\n>> +\t-this line should be marked as moved\n>> +\t+too short\n>> +\t unchanged\n>> +\t unchanged\n>> +\t+this line should be marked as moved\n>> +\t+too short\n>> +\t unchanged\n>> +\t unchanged\n>> +\t-too short\n>> +\t-this line should be marked as oldMoved newMoved\n>> +\t-this line should be marked as oldMovedAlternate newMoved\n>> +\t+this line should be marked as oldMoved newMoved/newMovedAlternate\n>> +\t unchanged 1\n>> +\t unchanged 2\n>> +\t+this line should be marked as oldMovedAlternate newMoved\n>> +\t+this line should be marked as oldMoved newMoved/newMovedAlternate\n>> +\t unchanged 3\n>> +\t+this line should be marked as oldMoved newMoved\n>> +\t unchanged 4\n>> +\t-this line should be marked as oldMoved newMoved/newMovedAlternate\n>> +\tEOF\n>> +\ttest_cmp expect actual\n>> +'\n>> +\n>> test_expect_success 'cmd option assumes configured colored-moved' '\n>> \ttest_config color.diff.oldMoved \"magenta\" &&\n>> \ttest_config color.diff.newMoved \"cyan\" &&\n>> --\n>> gitgitgadget\n>>\n>>\n\n"},{"id":"442028","messageId":"nycvar.QRO.7.76.6.2111222252450.63@tvgsbejvaqbjf.bet","threadId":"55926","inReplyTo":"a6ff589e-c968-8a96-a8ec-1a982d71f6be@gmail.com","subject":"Re: [PATCH v4 06/15] diff --color-moved: avoid false short line matches and bad zerba coloring","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-11-22T21:54:03Z","receivedAt":"2021-11-22T21:54:10Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Phillip,\n\nOn Mon, 22 Nov 2021, Phillip Wood wrote:\n\n[... a good explanation...]\n\n> On 22/11/2021 14:18, Johannes Schindelin wrote:\n>\n> > As I said, I do not quite understand this patch yet, and am looking\n> > for your guidance to wrap my head around it.\n>\n> Thanks for looking at it, I hope these comments help, let me know if I've\n> failed to explain well enough.\n\nYes, thank you, I think I understand enough now to say that the patch\nlooks good to me.\n\nThanks,\nDscho\n"},{"id":"442151","messageId":"nycvar.QRO.7.76.6.2111231540280.63@tvgsbejvaqbjf.bet","threadId":"55926","inReplyTo":"d7bbc0041e076077d3c3299858799cc9907d9831.1637056179.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 08/15] diff --color-moved-ws=allow-indentation-change: simplify and optimize","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-11-23T14:51:42Z","receivedAt":"2021-11-23T14:51:50Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Phillip,\n\ntl;dr: the patch looks good to me (it is a bit tricky to review, though,\nbut that is not your fault, it is our code review process' fault).\n\nOn Tue, 16 Nov 2021, Phillip Wood via GitGitGadget wrote:\n\n> git diff --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\n> by 93% compared to master and simplifies the code.\n\nNice!\n\n> diff --git a/diff.c b/diff.c\n> index 9aff167be27..78a486021ab 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -879,37 +879,21 @@ static int compute_ws_delta(const struct emitted_diff_symbol *a,\n> \treturn 1;\n> }\n>\n> -static int cmp_in_block_with_wsd(const struct diff_options *o,\n> -\t\t\t\t const struct moved_entry *cur,\n> -\t\t\t\t const struct moved_entry *match,\n> -\t\t\t\t struct moved_block *pmb,\n> -\t\t\t\t int n)\n> -{\n> -\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n> -\tint al = cur->es->len, bl = match->es->len, cl = l->len;\n> +static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n> +\t\t\t\t const struct emitted_diff_symbol *l,\n> +\t\t\t\t struct moved_block *pmb)\n> +{\n> +\tint al = cur->es->len, bl = l->len;\n\nOnce I realized that the old `b` was removed and the old `c` became the\nnew `b`, it was a breeze to validate this hunk.\n\n> \tconst char *a = cur->es->line,\n> -\t\t *b = match->es->line,\n> -\t\t *c = l->line;\n> +\t\t *b = l->line;\n> \tint a_off = cur->es->indent_off,\n> \t a_width = cur->es->indent_width,\n> -\t c_off = l->indent_off,\n> -\t c_width = l->indent_width;\n> +\t b_off = l->indent_off,\n> +\t b_width = l->indent_width;\n> \tint delta;\n>\n> -\t/*\n> -\t * We need to check if 'cur' is equal to 'match'. As those\n> -\t * are from the same (+/-) side, we do not need to adjust for\n> -\t * indent changes. However these were found using fuzzy\n> -\t * matching so we do have to check if they are equal. Here we\n> -\t * just check the lengths. We delay calling memcmp() to check\n> -\t * the contents until later as if the length comparison for a\n> -\t * and c fails we can avoid the call all together.\n> -\t */\n> -\tif (al != bl)\n> -\t\treturn 1;\n\nThe commit message really helped understanding why this is not needed.\nThank you!\n\n> -\n> \t/* If 'l' and 'cur' are both blank then they match. */\n> -\tif (a_width == INDENT_BLANKLINE && c_width == INDENT_BLANKLINE)\n> +\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n> \t\treturn 0;\n>\n> \t/*\n> @@ -918,7 +902,7 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n> \t * match those of the current block and that the text of 'l' and 'cur'\n> \t * after the indentation match.\n> \t */\n> -\tdelta = c_width - a_width;\n> +\tdelta = b_width - a_width;\n>\n> \t/*\n> \t * If the previous lines of this block were all blank then set its\n> @@ -927,9 +911,8 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n> \tif (pmb->wsd == INDENT_BLANKLINE)\n> \t\tpmb->wsd = delta;\n>\n> -\treturn !(delta == pmb->wsd && al - a_off == cl - c_off &&\n> -\t\t !memcmp(a, b, al) && !\n> -\t\t memcmp(a + a_off, c + c_off, al - a_off));\n> +\treturn !(delta == pmb->wsd && al - a_off == bl - b_off &&\n> +\t\t !memcmp(a + a_off, b + b_off, al - a_off));\n> }\n\nOnce again, I am sad that we have no better platform to do our code\ncontribution and review. Whatever you can say about GitHub's UI, it is\nbetter than static diffs in mails.\n\nBut you used GitGitGadget, and I finally broke down and wrote a script\nthat allows me to magic my way from the mail into the correct commit in\nthe GitGitGadget PR in the browser. It is still shell script (at some\nstage, I will need to extend the script to be much smarter than any shell\nscript can be, and probably convert it to node.js, but not today).\n\nThis helped me verify that there are no left-over references to the old\n`b`. So all is good!\n\n>\n> static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n> @@ -1030,36 +1013,23 @@ static void pmb_advance_or_null(struct diff_options *o,\n> }\n>\n> static void pmb_advance_or_null_multi_match(struct diff_options *o,\n> -\t\t\t\t\t struct moved_entry *match,\n> -\t\t\t\t\t struct hashmap *hm,\n> +\t\t\t\t\t struct emitted_diff_symbol *l,\n> \t\t\t\t\t struct moved_block *pmb,\n> -\t\t\t\t\t int pmb_nr, int n)\n> +\t\t\t\t\t int pmb_nr)\n> {\n> \tint i;\n> -\tchar *got_match = xcalloc(1, pmb_nr);\n> -\n> -\thashmap_for_each_entry_from(hm, match, ent) {\n> -\t\tfor (i = 0; i < pmb_nr; i++) {\n> -\t\t\tstruct moved_entry *prev = pmb[i].match;\n> -\t\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n> -\t\t\t\t\tprev->next_line : NULL;\n> -\t\t\tif (!cur)\n> -\t\t\t\tcontinue;\n> -\t\t\tif (!cmp_in_block_with_wsd(o, cur, match, &pmb[i], n))\n> -\t\t\t\tgot_match[i] |= 1;\n> -\t\t}\n> -\t}\n>\n> \tfor (i = 0; i < pmb_nr; i++) {\n> -\t\tif (got_match[i]) {\n> +\t\tstruct moved_entry *prev = pmb[i].match;\n> +\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n> +\t\t\tprev->next_line : NULL;\n> +\t\tif (cur && !cmp_in_block_with_wsd(cur, l, &pmb[i])) {\n> \t\t\t/* Advance to the next line */\n> -\t\t\tpmb[i].match = pmb[i].match->next_line;\n> +\t\t\tpmb[i].match = cur;\n> \t\t} else {\n> \t\t\tmoved_block_clear(&pmb[i]);\n> \t\t}\n> \t}\n> -\n> -\tfree(got_match);\n\nEven got rid of an allocation. Very nice.\n\n> }\n>\n> static int shrink_potential_moved_blocks(struct moved_block *pmb,\n> @@ -1223,7 +1193,7 @@ static void mark_color_as_moved(struct diff_options *o,\n>\n> \t\tif (o->color_moved_ws_handling &\n> \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n> -\t\t\tpmb_advance_or_null_multi_match(o, match, hm, pmb, pmb_nr, n);\n> +\t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n\nAgain, magic button to the rescue! And I can verify that `l` is assigned\nto `&o->emitted_symbols->buf[n]`, so: the patch does the correct thing.\n\nThank you,\nDscho\n\n> \t\telse\n> \t\t\tpmb_advance_or_null(o, match, hm, pmb, pmb_nr);\n>\n> --\n> gitgitgadget\n>\n>\n"},{"id":"442155","messageId":"nycvar.QRO.7.76.6.2111231605520.63@tvgsbejvaqbjf.bet","threadId":"55926","inReplyTo":"c3e5dce1910b3d640757e0845d646c3b040a8e28.1637056179.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 09/15] diff --color-moved: call comparison function directly","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-11-23T15:09:29Z","receivedAt":"2021-11-23T15:09:38Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Phillip,\n\nOn Tue, 16 Nov 2021, Phillip Wood via GitGitGadget wrote:\n\n> From: Phillip Wood \n>\n> This change will allow us to easily combine pmb_advance_or_null() and\n> pmb_advance_or_null_multi_match() in the next commit. Calling\n> xdiff_compare_lines() directly rather than using a function pointer\n> from the hash map has little effect on the run time.\n\nGood. I verified that the function `moved_entry_cmp()`\n(https://github.com/gitgitgadget/git/blob/c3e5dce191/diff.c#L918-L944)\ncalls `xdiff_compare_lines()`, and it is this function that is used for\nboth `del_lines` and `add_lines` which would have been passed as `hm`:\nhttps://github.com/gitgitgadget/git/blob/c3e5dce191/diff.c#L6339-L6340\n\n>\n> Test HEAD^ HEAD\n> -------------------------------------------------------------------------------------------------------------\n> 4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38(0.35+0.03) 0.38(0.32+0.06) +0.0%\n> 4002.2: diff --color-moved --no-color-moved-ws large change 0.87(0.83+0.04) 0.87(0.80+0.06) +0.0%\n> 4002.3: diff --color-moved-ws=allow-indentation-change large change 0.97(0.92+0.04) 0.97(0.93+0.04) +0.0%\n> 4002.4: log --no-color-moved --no-color-moved-ws 1.17(1.06+0.10) 1.16(1.10+0.05) -0.9%\n> 4002.5: log --color-moved --no-color-moved-ws 1.32(1.24+0.08) 1.31(1.22+0.09) -0.8%\n> 4002.6: log --color-moved-ws=allow-indentation-change 1.36(1.25+0.10) 1.35(1.25+0.10) -0.7%\n\nHonestly, I would have expected an improvement, given that\n`moved_entry_cmp()` has to do a few things before it can call\n`xdiff_compare_lines()`.\n\nI love your attention to detail, providing performance numbers in the\ncommit message to prove that it at least has no negative impact on the\nspeed.\n\nThanks,\nDscho\n\n>\n> Signed-off-by: Phillip Wood \n> ---\n> diff.c | 11 +++++++----\n> 1 file changed, 7 insertions(+), 4 deletions(-)\n>\n> diff --git a/diff.c b/diff.c\n> index 78a486021ab..22e0edac173 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -994,17 +994,20 @@ static void add_lines_to_move_detection(struct diff_options *o,\n> }\n>\n> static void pmb_advance_or_null(struct diff_options *o,\n> -\t\t\t\tstruct moved_entry *match,\n> -\t\t\t\tstruct hashmap *hm,\n> +\t\t\t\tstruct emitted_diff_symbol *l,\n> \t\t\t\tstruct moved_block *pmb,\n> \t\t\t\tint pmb_nr)\n> {\n> \tint i;\n> +\tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n> +\n> \tfor (i = 0; i < pmb_nr; i++) {\n> \t\tstruct moved_entry *prev = pmb[i].match;\n> \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n> \t\t\t\tprev->next_line : NULL;\n> -\t\tif (cur && !hm->cmpfn(o, &cur->ent, &match->ent, NULL)) {\n> +\t\tif (cur && xdiff_compare_lines(cur->es->line, cur->es->len,\n> +\t\t\t\t\t\tl->line, l->len,\n> +\t\t\t\t\t\tflags)) {\n> \t\t\tpmb[i].match = cur;\n> \t\t} else {\n> \t\t\tpmb[i].match = NULL;\n> @@ -1195,7 +1198,7 @@ static void mark_color_as_moved(struct diff_options *o,\n> \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n> \t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n> \t\telse\n> -\t\t\tpmb_advance_or_null(o, match, hm, pmb, pmb_nr);\n> +\t\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n>\n> \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n>\n> --\n> gitgitgadget\n>\n>\n"},{"id":"443423","messageId":"nycvar.QRO.7.76.6.2112081329030.90@tvgsbejvaqbjf.bet","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 00/15] diff --color-moved[-ws] speedups","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-12-08T12:30:33Z","receivedAt":"2021-12-08T12:30:44Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Phillip,\n\nOn Tue, 16 Nov 2021, Phillip Wood via GitGitGadget wrote:\n\n> Changes since V3:\n>\n> * Patch 1 now allows the user to choose different endpoints for the diff\n> perf tests to facilitate testing with different repositories.\n> * Fixed the alignment of the perf results column headers in a couple of\n> patches.\n\nI finished reading over the patches in this iteration. Although I did not\nhave the mental bandwidth to review in particular the last patch in the\nseries in detail, the regression tests seem comprehensive enough for me to\nbe confident in the correctness.\n\nSo: from my side, this patch series is good to go!\n\nThank you,\nDscho\n"},{"id":"443627","messageId":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v4.git.1637056178.gitgitgadget@gmail.com","subject":"[PATCH v5 00/15] diff --color-moved[-ws] speedups","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:29:54Z","receivedAt":"2021-12-09T10:30:14Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Thanks to Dscho for his comments on V3. Changes since V4:\n\n * Fixed a typo in the commit message to patch 6\n\nChanges since V3:\n\n * Patch 1 now allows the user to choose different endpoints for the diff\n perf tests to facilitate testing with different repositories.\n * Fixed the alignment of the perf results column headers in a couple of\n patches.\n\nChanges since V2:\n\n * Patches 1-3 are new and fix an existing bug.\n * Patch 8 includes Peff's unused parameter fix.\n * Patch 11 has been updated to fix a bug fix in V2.\n * Patch 13 has an expanded commit message explaining a change in behavior\n for lines starting with a form-feed.\n * Updated benchmark results.\n\nThe bug fix in patch 3 degrades the performance, but by the end of the\nseries the timings are the same as V2 - see the range diff.\n\nV2 Cover Letter: Thanks to Ævar and Elijah for their comments, I've reworded\nthe commit messages, addressed the enum initialization issue in patch 2 (now\n3) and added some perf tests.\n\nThere are two new patches in this round. The first patch is new and adds the\nperf tests suggested by Ævar, the penultimate patch is also new and coverts\nthe existing code to use a designated initializer.\n\nI've converted the benchmark results in the commit messages to use the new\ntests, the percentage changes are broadly similar to the previous results\nthough I ended up running them on a different computer this time.\n\nV1 cover letter:\n\nThe current implementation of diff --color-moved-ws=allow-indentation-change\nis considerably slower that the implementation of diff --color-moved which\nis in turn slower than a regular diff. This patch series starts with a\ncouple of bug fixes and then reworks the implementation of diff\n--color-moved and diff --color-moved-ws=allow-indentation-change to speed\nthem up on large diffs. The time to run git diff --color-moved\n--no-color-moved-ws v2.28.0 v2.29.0 is reduced by 33% and the time to run\ngit diff --color-moved --color-moved-ws=allow-indentation-change v2.28.0\nv2.29.0 is reduced by 88%. There is a small slowdown for commit sized diffs\nwith --color-moved - the time to run git log -p --color-moved\n--no-color-moved-ws --no-merges -n1000 v2.29.0 is increased by 2% on recent\nprocessors. On older processors these patches reduce the running time in all\ncases that I've tested. In general the larger the diff the larger the speed\nup. As an extreme example the time to run diff --color-moved\n--color-moved-ws=allow-indentation-change v2.25.0 v2.30.0 goes down from 8\nminutes to 6 seconds.\n\nPhillip Wood (15):\n diff --color-moved: add perf tests\n diff --color-moved: clear all flags on blocks that are too short\n diff --color-moved: factor out function\n diff --color-moved: rewind when discarding pmb\n diff --color-moved=zebra: fix alternate coloring\n diff --color-moved: avoid false short line matches and bad zebra\n coloring\n diff: simplify allow-indentation-change delta calculation\n diff --color-moved-ws=allow-indentation-change: simplify and optimize\n diff --color-moved: call comparison function directly\n diff --color-moved: unify moved block growth functions\n diff --color-moved: shrink potential moved blocks as we go\n diff --color-moved: stop clearing potential moved blocks\n diff --color-moved-ws=allow-indentation-change: improve hash lookups\n diff: use designated initializers for emitted_diff_symbol\n diff --color-moved: intern strings\n\n diff.c | 431 +++++++++++++------------------\n t/perf/p4002-diff-color-moved.sh | 57 ++++\n t/t4015-diff-whitespace.sh | 205 ++++++++++++++-\n 3 files changed, 437 insertions(+), 256 deletions(-)\n create mode 100755 t/perf/p4002-diff-color-moved.sh\n\n\nbase-commit: 211eca0895794362184da2be2a2d812d070719d3\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-981%2Fphillipwood%2Fwip%2Fdiff-color-moved-tweaks-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-981/phillipwood/wip/diff-color-moved-tweaks-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/981\n\nRange-diff vs v4:\n\n 1: 48ee03cf52a = 1: 48ee03cf52a diff --color-moved: add perf tests\n 2: 47c652716e8 = 2: 47c652716e8 diff --color-moved: clear all flags on blocks that are too short\n 3: 99e38ba9de9 = 3: 99e38ba9de9 diff --color-moved: factor out function\n 4: 9ca71db61ae = 4: 9ca71db61ae diff --color-moved: rewind when discarding pmb\n 5: 56bb69af36e = 5: 56bb69af36e diff --color-moved=zebra: fix alternate coloring\n 6: 10b11526206 ! 6: ed62b980225 diff --color-moved: avoid false short line matches and bad zerba coloring\n @@ Metadata\n Author: Phillip Wood \n \n ## Commit message ##\n - diff --color-moved: avoid false short line matches and bad zerba coloring\n + diff --color-moved: avoid false short line matches and bad zebra coloring\n \n When marking moved lines it is possible for a block of potential\n matched lines to extend past a change in sign when there is a sequence\n 7: c2e7b347257 = 7: b8db6a1af7d diff: simplify allow-indentation-change delta calculation\n 8: d7bbc0041e0 = 8: eeb633063b7 diff --color-moved-ws=allow-indentation-change: simplify and optimize\n 9: c3e5dce1910 = 9: fb413cab3a8 diff --color-moved: call comparison function directly\n 10: 9eb8cecd52a = 10: ec8764082d5 diff --color-moved: unify moved block growth functions\n 11: 35e204e1578 = 11: 6199a014547 diff --color-moved: shrink potential moved blocks as we go\n 12: ec329e7946d = 12: 1db84490ee4 diff --color-moved: stop clearing potential moved blocks\n 13: 6ec94134aaf = 13: 3e769bab78c diff --color-moved-ws=allow-indentation-change: improve hash lookups\n 14: d44c5d734c3 = 14: b8869659664 diff: use designated initializers for emitted_diff_symbol\n 15: 5177f669423 = 15: 350fa55ce5e diff --color-moved: intern strings\n\n-- \ngitgitgadget\n"},{"id":"443628","messageId":"48ee03cf52a619d22d7ee7b1958dd59872bf721f.1639045809.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","subject":"[PATCH v5 01/15] diff --color-moved: add perf tests","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:29:55Z","receivedAt":"2021-12-09T10:30:21Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nAdd some tests so we can monitor changes to the performance of the\nmove detection code. The tests record the performance --color-moved\nand --color-moved-ws=allow-indentation-change for a large diff and a\nsequence of smaller diffs. The range of commits used for the large\ndiff can be customized by exporting TEST_REV_A and TEST_REV_B when\nrunning the test.\n\nSigned-off-by: Phillip Wood \n---\n t/perf/p4002-diff-color-moved.sh | 57 ++++++++++++++++++++++++++++++++\n 1 file changed, 57 insertions(+)\n create mode 100755 t/perf/p4002-diff-color-moved.sh\n\ndiff --git a/t/perf/p4002-diff-color-moved.sh b/t/perf/p4002-diff-color-moved.sh\nnew file mode 100755\nindex 00000000000..ab2af931c04\n--- /dev/null\n+++ b/t/perf/p4002-diff-color-moved.sh\n@@ -0,0 +1,57 @@\n+#!/bin/sh\n+\n+test_description='Tests diff --color-moved performance'\n+. ./perf-lib.sh\n+\n+test_perf_default_repo\n+\n+# The endpoints of the diff can be customized by setting TEST_REV_A\n+# and TEST_REV_B in the environment when running this test.\n+\n+rev=\"${TEST_REV_A:-v2.28.0}\"\n+if ! rev_a=\"$(git rev-parse --quiet --verify \"$rev\")\"\n+then\n+\tskip_all=\"skipping because '$rev' was not found. \\\n+\t\t Use TEST_REV_A and TEST_REV_B to set the revs to use\"\n+\ttest_done\n+fi\n+rev=\"${TEST_REV_B:-v2.29.0}\"\n+if ! rev_b=\"$(git rev-parse --quiet --verify \"$rev\")\"\n+then\n+\tskip_all=\"skipping because '$rev' was not found. \\\n+\t\t Use TEST_REV_A and TEST_REV_B to set the revs to use\"\n+\ttest_done\n+fi\n+\n+GIT_PAGER_IN_USE=1\n+test_export GIT_PAGER_IN_USE rev_a rev_b\n+\n+test_perf 'diff --no-color-moved --no-color-moved-ws large change' '\n+\tgit diff --no-color-moved --no-color-moved-ws $rev_a $rev_b\n+'\n+\n+test_perf 'diff --color-moved --no-color-moved-ws large change' '\n+\tgit diff --color-moved=zebra --no-color-moved-ws $rev_a $rev_b\n+'\n+\n+test_perf 'diff --color-moved-ws=allow-indentation-change large change' '\n+\tgit diff --color-moved=zebra --color-moved-ws=allow-indentation-change \\\n+\t\t$rev_a $rev_b\n+'\n+\n+test_perf 'log --no-color-moved --no-color-moved-ws' '\n+\tgit log --no-color-moved --no-color-moved-ws --no-merges --patch \\\n+\t\t-n1000 $rev_b\n+'\n+\n+test_perf 'log --color-moved --no-color-moved-ws' '\n+\tgit log --color-moved=zebra --no-color-moved-ws --no-merges --patch \\\n+\t\t-n1000 $rev_b\n+'\n+\n+test_perf 'log --color-moved-ws=allow-indentation-change' '\n+\tgit log --color-moved=zebra --color-moved-ws=allow-indentation-change \\\n+\t\t--no-merges --patch -n1000 $rev_b\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"443629","messageId":"47c652716e88af7199c44a555780eafb643a1f9e.1639045809.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","subject":"[PATCH v5 02/15] diff --color-moved: clear all flags on blocks that are too short","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:29:56Z","receivedAt":"2021-12-09T10:30:23Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nIf a block of potentially moved lines is not long enough then the\nDIFF_SYMBOL_MOVED_LINE flag is cleared on the matching lines so they\nare not marked as moved. To avoid problems when we start rewinding\nafter an unsuccessful match in a couple of commits time make sure all\nthe move related flags are cleared, not just DIFF_SYMBOL_MOVED_LINE.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 52c791574b7..bd8e4ec9757 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1114,6 +1114,8 @@ static int shrink_potential_moved_blocks(struct moved_block *pmb,\n * NEEDSWORK: This uses the same heuristic as blame_entry_score() in blame.c.\n * Think of a way to unify them.\n */\n+#define DIFF_SYMBOL_MOVED_LINE_ZEBRA_MASK \\\n+ (DIFF_SYMBOL_MOVED_LINE | DIFF_SYMBOL_MOVED_LINE_ALT)\n static int adjust_last_block(struct diff_options *o, int n, int block_length)\n {\n \tint i, alnum_count = 0;\n@@ -1130,7 +1132,7 @@ static int adjust_last_block(struct diff_options *o, int n, int block_length)\n \t\t}\n \t}\n \tfor (i = 1; i < block_length + 1; i++)\n-\t\to->emitted_symbols->buf[n - i].flags &= ~DIFF_SYMBOL_MOVED_LINE;\n+\t\to->emitted_symbols->buf[n - i].flags &= ~DIFF_SYMBOL_MOVED_LINE_ZEBRA_MASK;\n \treturn 0;\n }\n \n@@ -1237,8 +1239,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \tfree(pmb);\n }\n \n-#define DIFF_SYMBOL_MOVED_LINE_ZEBRA_MASK \\\n- (DIFF_SYMBOL_MOVED_LINE | DIFF_SYMBOL_MOVED_LINE_ALT)\n static void dim_moved_lines(struct diff_options *o)\n {\n \tint n;\n-- \ngitgitgadget\n\n"},{"id":"443630","messageId":"56bb69af36e5e3180d53586d30048d5033a01d14.1639045809.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","subject":"[PATCH v5 05/15] diff --color-moved=zebra: fix alternate coloring","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:29:59Z","receivedAt":"2021-12-09T10:30:23Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nb0a2ba4776 (\"diff --color-moved=zebra: be stricter with color\nalternation\", 2018-11-23) sought to avoid using the alternate colors\nunless there are two adjacent moved blocks of the same\nsign. Unfortunately it contains two bugs that prevented it from fixing\nthe problem properly. Firstly `last_symbol` is reset at the start of\neach iteration of the loop losing the symbol of the last line and\nsecondly when deciding whether to use the alternate color it should be\nchecking if the current line is the same sign of the last line, not a\ndifferent sign. The combination of the two errors means that we still\nuse the alternate color when we should do but we also use it when we\nshouldn't. This is most noticable when using\n--color-moved-ws=allow-indentation-change with hunks like\n\n-this line gets indented\n+ this line gets indented\n\nwhere the post image is colored with newMovedAlternate rather than\nnewMoved. While this does not matter much, the next commit will change\nthe coloring to be correct in this case, so lets fix the bug here to\nmake it clear why the output is changing and add a regression test.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 4 +--\n t/t4015-diff-whitespace.sh | 72 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 74 insertions(+), 2 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 1e1b5127d15..53f0df75329 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1176,6 +1176,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n \tint n, flipped_block = 0, block_length = 0;\n+\tenum diff_symbol last_symbol = 0;\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n@@ -1183,7 +1184,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\tstruct moved_entry *key;\n \t\tstruct moved_entry *match = NULL;\n \t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n-\t\tenum diff_symbol last_symbol = 0;\n \n \t\tswitch (l->s) {\n \t\tcase DIFF_SYMBOL_PLUS:\n@@ -1251,7 +1251,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t\t\t\t\t &pmb, &pmb_alloc,\n \t\t\t\t\t\t\t &pmb_nr);\n \n-\t\t\tif (contiguous && pmb_nr && last_symbol != l->s)\n+\t\t\tif (contiguous && pmb_nr && last_symbol == l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n \t\t\telse\n \t\t\t\tflipped_block = 0;\ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 308dc136596..4e0fd76c6c5 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1442,6 +1442,78 @@ test_expect_success 'detect permutations inside moved code -- dimmed-zebra' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'zebra alternate color is only used when necessary' '\n+\tcat >old.txt <<-\\EOF &&\n+\tline 1A should be marked as oldMoved newMovedAlternate\n+\tline 1B should be marked as oldMoved newMovedAlternate\n+\tunchanged\n+\tline 2A should be marked as oldMoved newMovedAlternate\n+\tline 2B should be marked as oldMoved newMovedAlternate\n+\tline 3A should be marked as oldMovedAlternate newMoved\n+\tline 3B should be marked as oldMovedAlternate newMoved\n+\tunchanged\n+\tline 4A should be marked as oldMoved newMovedAlternate\n+\tline 4B should be marked as oldMoved newMovedAlternate\n+\tline 5A should be marked as oldMovedAlternate newMoved\n+\tline 5B should be marked as oldMovedAlternate newMoved\n+\tline 6A should be marked as oldMoved newMoved\n+\tline 6B should be marked as oldMoved newMoved\n+\tEOF\n+\tcat >new.txt <<-\\EOF &&\n+\t line 1A should be marked as oldMoved newMovedAlternate\n+\t line 1B should be marked as oldMoved newMovedAlternate\n+\tunchanged\n+\t line 3A should be marked as oldMovedAlternate newMoved\n+\t line 3B should be marked as oldMovedAlternate newMoved\n+\t line 2A should be marked as oldMoved newMovedAlternate\n+\t line 2B should be marked as oldMoved newMovedAlternate\n+\tunchanged\n+\t line 6A should be marked as oldMoved newMoved\n+\t line 6B should be marked as oldMoved newMoved\n+\t line 4A should be marked as oldMoved newMovedAlternate\n+\t line 4B should be marked as oldMoved newMovedAlternate\n+\t line 5A should be marked as oldMovedAlternate newMoved\n+\t line 5B should be marked as oldMovedAlternate newMoved\n+\tEOF\n+\ttest_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n+\t\t --color-moved-ws=allow-indentation-change \\\n+\t\t old.txt new.txt >output &&\n+\tgrep -v index output | test_decode_color >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tdiff --git a/old.txt b/new.txt\n+\t--- a/old.txt\n+\t+++ b/new.txt\n+\t@@ -1,14 +1,14 @@\n+\t-line 1A should be marked as oldMoved newMovedAlternate\n+\t-line 1B should be marked as oldMoved newMovedAlternate\n+\t+ line 1A should be marked as oldMoved newMovedAlternate\n+\t+ line 1B should be marked as oldMoved newMovedAlternate\n+\t unchanged\n+\t-line 2A should be marked as oldMoved newMovedAlternate\n+\t-line 2B should be marked as oldMoved newMovedAlternate\n+\t-line 3A should be marked as oldMovedAlternate newMoved\n+\t-line 3B should be marked as oldMovedAlternate newMoved\n+\t+ line 3A should be marked as oldMovedAlternate newMoved\n+\t+ line 3B should be marked as oldMovedAlternate newMoved\n+\t+ line 2A should be marked as oldMoved newMovedAlternate\n+\t+ line 2B should be marked as oldMoved newMovedAlternate\n+\t unchanged\n+\t-line 4A should be marked as oldMoved newMovedAlternate\n+\t-line 4B should be marked as oldMoved newMovedAlternate\n+\t-line 5A should be marked as oldMovedAlternate newMoved\n+\t-line 5B should be marked as oldMovedAlternate newMoved\n+\t-line 6A should be marked as oldMoved newMoved\n+\t-line 6B should be marked as oldMoved newMoved\n+\t+ line 6A should be marked as oldMoved newMoved\n+\t+ line 6B should be marked as oldMoved newMoved\n+\t+ line 4A should be marked as oldMoved newMovedAlternate\n+\t+ line 4B should be marked as oldMoved newMovedAlternate\n+\t+ line 5A should be marked as oldMovedAlternate newMoved\n+\t+ line 5B should be marked as oldMovedAlternate newMoved\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'cmd option assumes configured colored-moved' '\n \ttest_config color.diff.oldMoved \"magenta\" &&\n \ttest_config color.diff.newMoved \"cyan\" &&\n-- \ngitgitgadget\n\n"},{"id":"443631","messageId":"99e38ba9de9bcd7b767908ec218937ae8c2ccf36.1639045809.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","subject":"[PATCH v5 03/15] diff --color-moved: factor out function","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:29:57Z","receivedAt":"2021-12-09T10:30:25Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nThis code is quite heavily indented and having it in its own function\nsimplifies an upcoming change.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 51 ++++++++++++++++++++++++++++++++++-----------------\n 1 file changed, 34 insertions(+), 17 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex bd8e4ec9757..09af94e018c 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1098,6 +1098,38 @@ static int shrink_potential_moved_blocks(struct moved_block *pmb,\n \treturn rp + 1;\n }\n \n+static void fill_potential_moved_blocks(struct diff_options *o,\n+\t\t\t\t\tstruct hashmap *hm,\n+\t\t\t\t\tstruct moved_entry *match,\n+\t\t\t\t\tstruct emitted_diff_symbol *l,\n+\t\t\t\t\tstruct moved_block **pmb_p,\n+\t\t\t\t\tint *pmb_alloc_p, int *pmb_nr_p)\n+\n+{\n+\tstruct moved_block *pmb = *pmb_p;\n+\tint pmb_alloc = *pmb_alloc_p, pmb_nr = *pmb_nr_p;\n+\n+\t/*\n+\t * The current line is the start of a new block.\n+\t * Setup the set of potential blocks.\n+\t */\n+\thashmap_for_each_entry_from(hm, match, ent) {\n+\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n+\t\tif (o->color_moved_ws_handling &\n+\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n+\t\t\tif (compute_ws_delta(l, match->es, &(pmb[pmb_nr]).wsd))\n+\t\t\t\tpmb[pmb_nr++].match = match;\n+\t\t} else {\n+\t\t\tpmb[pmb_nr].wsd = 0;\n+\t\t\tpmb[pmb_nr++].match = match;\n+\t\t}\n+\t}\n+\n+\t*pmb_p = pmb;\n+\t*pmb_alloc_p = pmb_alloc;\n+\t*pmb_nr_p = pmb_nr;\n+}\n+\n /*\n * If o->color_moved is COLOR_MOVED_PLAIN, this function does nothing.\n *\n@@ -1198,23 +1230,8 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n \t\tif (pmb_nr == 0) {\n-\t\t\t/*\n-\t\t\t * The current line is the start of a new block.\n-\t\t\t * Setup the set of potential blocks.\n-\t\t\t */\n-\t\t\thashmap_for_each_entry_from(hm, match, ent) {\n-\t\t\t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n-\t\t\t\tif (o->color_moved_ws_handling &\n-\t\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n-\t\t\t\t\tif (compute_ws_delta(l, match->es,\n-\t\t\t\t\t\t\t &pmb[pmb_nr].wsd))\n-\t\t\t\t\t\tpmb[pmb_nr++].match = match;\n-\t\t\t\t} else {\n-\t\t\t\t\tpmb[pmb_nr].wsd = 0;\n-\t\t\t\t\tpmb[pmb_nr++].match = match;\n-\t\t\t\t}\n-\t\t\t}\n-\n+\t\t\tfill_potential_moved_blocks(\n+\t\t\t\to, hm, match, l, &pmb, &pmb_alloc, &pmb_nr);\n \t\t\tif (adjust_last_block(o, n, block_length) &&\n \t\t\t pmb_nr && last_symbol != l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n-- \ngitgitgadget\n\n"},{"id":"443632","messageId":"ed62b9802259e875c8fa1b3404143ef5a382f968.1639045809.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","subject":"[PATCH v5 06/15] diff --color-moved: avoid false short line matches and bad zebra coloring","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:30:00Z","receivedAt":"2021-12-09T10:30:26Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nWhen marking moved lines it is possible for a block of potential\nmatched lines to extend past a change in sign when there is a sequence\nof added lines whose text matches the text of a sequence of deleted\nand added lines. Most of the time either `match` will be NULL or\n`pmb_advance_or_null()` will fail when the loop encounters a change of\nsign but there are corner cases where `match` is non-NULL and\n`pmb_advance_or_null()` successfully advances the moved block despite\nthe change in sign.\n\nOne consequence of this is highlighting a short line as moved when it\nshould not be. For example\n\n-moved line # Correctly highlighted as moved\n+short line # Wrongly highlighted as moved\n context\n+moved line # Correctly highlighted as moved\n+short line\n context\n-short line\n\nThe other consequence is coloring a moved addition following a moved\ndeletion in the wrong color. In the example below the first \"+moved\nline 3\" should be highlighted as newMoved not newMovedAlternate.\n\n-moved line 1 # Correctly highlighted as oldMoved\n-moved line 2 # Correctly highlighted as oldMovedAlternate\n+moved line 3 # Wrongly highlighted as newMovedAlternate\n context # Everything else is highlighted correctly\n+moved line 2\n+moved line 3\n context\n+moved line 1\n-moved line 3\n\nThese false matches are more likely when using --color-moved-ws with\nthe exception of --color-moved-ws=allow-indentation-change which ties\nthe sign of the current whitespace delta to the sign of the line to\navoid this problem. The fix is to check that the sign of the new line\nbeing matched is the same as the sign of the line that started the\nblock of potential matches.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 17 ++++++----\n t/t4015-diff-whitespace.sh | 65 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 76 insertions(+), 6 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 53f0df75329..efba2789354 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1176,7 +1176,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n \tint n, flipped_block = 0, block_length = 0;\n-\tenum diff_symbol last_symbol = 0;\n+\tenum diff_symbol moved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n@@ -1202,7 +1202,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tflipped_block = 0;\n \t\t}\n \n-\t\tif (!match) {\n+\t\tif (pmb_nr && (!match || l->s != moved_symbol)) {\n \t\t\tint i;\n \n \t\t\tif (!adjust_last_block(o, n, block_length) &&\n@@ -1219,12 +1219,13 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tpmb_nr = 0;\n \t\t\tblock_length = 0;\n \t\t\tflipped_block = 0;\n-\t\t\tlast_symbol = l->s;\n+\t\t}\n+\t\tif (!match) {\n+\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (o->color_moved == COLOR_MOVED_PLAIN) {\n-\t\t\tlast_symbol = l->s;\n \t\t\tl->flags |= DIFF_SYMBOL_MOVED_LINE;\n \t\t\tcontinue;\n \t\t}\n@@ -1251,11 +1252,16 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t\t\t\t\t &pmb, &pmb_alloc,\n \t\t\t\t\t\t\t &pmb_nr);\n \n-\t\t\tif (contiguous && pmb_nr && last_symbol == l->s)\n+\t\t\tif (contiguous && pmb_nr && moved_symbol == l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n \t\t\telse\n \t\t\t\tflipped_block = 0;\n \n+\t\t\tif (pmb_nr)\n+\t\t\t\tmoved_symbol = l->s;\n+\t\t\telse\n+\t\t\t\tmoved_symbol = DIFF_SYMBOL_BINARY_DIFF_HEADER;\n+\n \t\t\tblock_length = 0;\n \t\t}\n \n@@ -1265,7 +1271,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tif (flipped_block && o->color_moved != COLOR_MOVED_BLOCKS)\n \t\t\t\tl->flags |= DIFF_SYMBOL_MOVED_LINE_ALT;\n \t\t}\n-\t\tlast_symbol = l->s;\n \t}\n \tadjust_last_block(o, n, block_length);\n \ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 4e0fd76c6c5..15782c879d2 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1514,6 +1514,71 @@ test_expect_success 'zebra alternate color is only used when necessary' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'short lines of opposite sign do not get marked as moved' '\n+\tcat >old.txt <<-\\EOF &&\n+\tthis line should be marked as moved\n+\tunchanged\n+\tunchanged\n+\tunchanged\n+\tunchanged\n+\ttoo short\n+\tthis line should be marked as oldMoved newMoved\n+\tthis line should be marked as oldMovedAlternate newMoved\n+\tunchanged 1\n+\tunchanged 2\n+\tunchanged 3\n+\tunchanged 4\n+\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n+\tEOF\n+\tcat >new.txt <<-\\EOF &&\n+\ttoo short\n+\tunchanged\n+\tunchanged\n+\tthis line should be marked as moved\n+\ttoo short\n+\tunchanged\n+\tunchanged\n+\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n+\tunchanged 1\n+\tunchanged 2\n+\tthis line should be marked as oldMovedAlternate newMoved\n+\tthis line should be marked as oldMoved newMoved/newMovedAlternate\n+\tunchanged 3\n+\tthis line should be marked as oldMoved newMoved\n+\tunchanged 4\n+\tEOF\n+\ttest_expect_code 1 git diff --no-index --color --color-moved=zebra \\\n+\t\told.txt new.txt >output && cat output &&\n+\tgrep -v index output | test_decode_color >actual &&\n+\tcat >expect <<-\\EOF &&\n+\tdiff --git a/old.txt b/new.txt\n+\t--- a/old.txt\n+\t+++ b/new.txt\n+\t@@ -1,13 +1,15 @@\n+\t-this line should be marked as moved\n+\t+too short\n+\t unchanged\n+\t unchanged\n+\t+this line should be marked as moved\n+\t+too short\n+\t unchanged\n+\t unchanged\n+\t-too short\n+\t-this line should be marked as oldMoved newMoved\n+\t-this line should be marked as oldMovedAlternate newMoved\n+\t+this line should be marked as oldMoved newMoved/newMovedAlternate\n+\t unchanged 1\n+\t unchanged 2\n+\t+this line should be marked as oldMovedAlternate newMoved\n+\t+this line should be marked as oldMoved newMoved/newMovedAlternate\n+\t unchanged 3\n+\t+this line should be marked as oldMoved newMoved\n+\t unchanged 4\n+\t-this line should be marked as oldMoved newMoved/newMovedAlternate\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'cmd option assumes configured colored-moved' '\n \ttest_config color.diff.oldMoved \"magenta\" &&\n \ttest_config color.diff.newMoved \"cyan\" &&\n-- \ngitgitgadget\n\n"},{"id":"443633","messageId":"b8db6a1af7da5ff7cee0a86f08a5a163965d965b.1639045809.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","subject":"[PATCH v5 07/15] diff: simplify allow-indentation-change delta calculation","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:30:01Z","receivedAt":"2021-12-09T10:30:27Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nNow that we reliably end a block when the sign changes we don't need\nthe whitespace delta calculation to rely on the sign.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 13 ++-----------\n 1 file changed, 2 insertions(+), 11 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex efba2789354..9aff167be27 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -864,23 +864,17 @@ static int compute_ws_delta(const struct emitted_diff_symbol *a,\n \t a_width = a->indent_width,\n \t b_off = b->indent_off,\n \t b_width = b->indent_width;\n-\tint delta;\n \n \tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE) {\n \t\t*out = INDENT_BLANKLINE;\n \t\treturn 1;\n \t}\n \n-\tif (a->s == DIFF_SYMBOL_PLUS)\n-\t\tdelta = a_width - b_width;\n-\telse\n-\t\tdelta = b_width - a_width;\n-\n \tif (a_len - a_off != b_len - b_off ||\n \t memcmp(a->line + a_off, b->line + b_off, a_len - a_off))\n \t\treturn 0;\n \n-\t*out = delta;\n+\t*out = a_width - b_width;\n \n \treturn 1;\n }\n@@ -924,10 +918,7 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \t * match those of the current block and that the text of 'l' and 'cur'\n \t * after the indentation match.\n \t */\n-\tif (cur->es->s == DIFF_SYMBOL_PLUS)\n-\t\tdelta = a_width - c_width;\n-\telse\n-\t\tdelta = c_width - a_width;\n+\tdelta = c_width - a_width;\n \n \t/*\n \t * If the previous lines of this block were all blank then set its\n-- \ngitgitgadget\n\n"},{"id":"443634","messageId":"9ca71db61ae92d99fceba112593999596b65cd4c.1639045809.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","subject":"[PATCH v5 04/15] diff --color-moved: rewind when discarding pmb","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:29:58Z","receivedAt":"2021-12-09T10:30:28Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\ndiff --color-moved colors the two sides of the diff separately. It\nwalks through the diff and tries to find matches on the other side of\nthe diff for the current line. When it finds one or more matches it\nstarts a \"potential moved block\" (pmb) and marks the current line as\nmoved. Then as it walks through the diff it only looks for matches for\nthe current line in the lines following those in the pmb. When none of\nthe lines in the pmb match it checks how long the match is and if it\nis too short it unmarks the lines as matched and goes back to finding\nall the lines that match the current line. As the process of finding\nmatching lines restarts from the end of the block that was too short\nit is possible to miss the start of a matching block on on side but\nnot the other. In the test added here \"-two\" would not be colored as\nmoved but \"+two\" would be.\n\nFix this by rewinding the current line when we reach the end of a\nblock that is too short. This is quadratic in the length of the\ndiscarded block. While the discarded blocks are quite short on a large\ndiff this still has a significant impact on the performance of\n--color-moved-ws=allow-indentation-change. The following commits\noptimize the performance of the --color-moved machinery which\nmitigates the performance impact of this commit. After the\noptimization this commit has a negligible impact on performance.\n\nTest HEAD^ HEAD\n-----------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.33+0.05) 0.39 (0.34+0.04) +2.6%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.80 (0.76+0.03) 0.86 (0.82+0.04) +7.5%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 14.22(14.17+0.04) 19.01(18.93+0.05) +33.7%\n4002.4: log --no-color-moved --no-color-moved-ws 1.16 (1.06+0.09) 1.16 (1.07+0.07) +0.0%\n4002.5: log --color-moved --no-color-moved-ws 1.31 (1.22+0.09) 1.32 (1.22+0.09) +0.8%\n4002.6: log --color-moved-ws=allow-indentation-change 1.71 (1.61+0.09) 1.72 (1.63+0.08) +0.6%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 28 ++++++++++++++++++-----\n t/t4015-diff-whitespace.sh | 46 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 69 insertions(+), 5 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 09af94e018c..1e1b5127d15 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1205,7 +1205,15 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\tif (!match) {\n \t\t\tint i;\n \n-\t\t\tadjust_last_block(o, n, block_length);\n+\t\t\tif (!adjust_last_block(o, n, block_length) &&\n+\t\t\t block_length > 1) {\n+\t\t\t\t/*\n+\t\t\t\t * Rewind in case there is another match\n+\t\t\t\t * starting at the second line of the block\n+\t\t\t\t */\n+\t\t\t\tmatch = NULL;\n+\t\t\t\tn -= block_length;\n+\t\t\t}\n \t\t\tfor(i = 0; i < pmb_nr; i++)\n \t\t\t\tmoved_block_clear(&pmb[i]);\n \t\t\tpmb_nr = 0;\n@@ -1230,10 +1238,20 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n \t\tif (pmb_nr == 0) {\n-\t\t\tfill_potential_moved_blocks(\n-\t\t\t\to, hm, match, l, &pmb, &pmb_alloc, &pmb_nr);\n-\t\t\tif (adjust_last_block(o, n, block_length) &&\n-\t\t\t pmb_nr && last_symbol != l->s)\n+\t\t\tint contiguous = adjust_last_block(o, n, block_length);\n+\n+\t\t\tif (!contiguous && block_length > 1)\n+\t\t\t\t/*\n+\t\t\t\t * Rewind in case there is another match\n+\t\t\t\t * starting at the second line of the block\n+\t\t\t\t */\n+\t\t\t\tn -= block_length;\n+\t\t\telse\n+\t\t\t\tfill_potential_moved_blocks(o, hm, match, l,\n+\t\t\t\t\t\t\t &pmb, &pmb_alloc,\n+\t\t\t\t\t\t\t &pmb_nr);\n+\n+\t\t\tif (contiguous && pmb_nr && last_symbol != l->s)\n \t\t\t\tflipped_block = (flipped_block + 1) % 2;\n \t\t\telse\n \t\t\t\tflipped_block = 0;\ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 2c13b62d3c6..308dc136596 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1833,6 +1833,52 @@ test_expect_success '--color-moved treats adjacent blocks as separate for MIN_AL\n \ttest_cmp expected actual\n '\n \n+test_expect_success '--color-moved rewinds for MIN_ALNUM_COUNT' '\n+\tgit reset --hard &&\n+\ttest_write_lines >file \\\n+\t\tA B C one two three four five six seven D E F G H I J &&\n+\tgit add file &&\n+\ttest_write_lines >file \\\n+\t\tone two A B C D E F G H I J two three four five six seven &&\n+\tgit diff --color-moved=zebra -- file &&\n+\n+\tgit diff --color-moved=zebra --color -- file >actual.raw &&\n+\tgrep -v \"index\" actual.raw | test_decode_color >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tdiff --git a/file b/file\n+\t--- a/file\n+\t+++ b/file\n+\t@@ -1,13 +1,8 @@\n+\t+one\n+\t+two\n+\t A\n+\t B\n+\t C\n+\t-one\n+\t-two\n+\t-three\n+\t-four\n+\t-five\n+\t-six\n+\t-seven\n+\t D\n+\t E\n+\t F\n+\t@@ -15,3 +10,9 @@ G\n+\t H\n+\t I\n+\t J\n+\t+two\n+\t+three\n+\t+four\n+\t+five\n+\t+six\n+\t+seven\n+\tEOF\n+\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'move detection with submodules' '\n \ttest_create_repo bananas &&\n \techo ripe >bananas/recipe &&\n-- \ngitgitgadget\n\n"},{"id":"443635","messageId":"eeb633063b7c08aae0f8375bcf41466ad33bcec3.1639045809.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","subject":"[PATCH v5 08/15] diff --color-moved-ws=allow-indentation-change: simplify and optimize","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:30:02Z","receivedAt":"2021-12-09T10:30:30Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nIf we already have a block of potentially moved lines then as we move\ndown the diff we need to check if the next line of each potentially\nmoved line matches the current line of the diff. The implementation of\n--color-moved-ws=allow-indentation-change was needlessly performing\nthis check on all the lines in the diff that matched the current line\nrather than just the current line. To exacerbate the problem finding\nall the other lines in the diff that match the current line involves a\nfuzzy lookup so we were wasting even more time performing a second\ncomparison to filter out the non-matching lines. Fixing this reduces\ntime to run\n git diff --color-moved-ws=allow-indentation-change v2.28.0 v2.29.0\nby 93% compared to master and simplifies the code.\n\nTest HEAD^ HEAD\n---------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.35+0.03) 0.38(0.35+0.03) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.86 (0.80+0.06) 0.87(0.83+0.04) +1.2%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 19.01(18.93+0.06) 0.97(0.92+0.04) -94.9%\n4002.4: log --no-color-moved --no-color-moved-ws 1.16 (1.06+0.09) 1.17(1.06+0.10) +0.9%\n4002.5: log --color-moved --no-color-moved-ws 1.32 (1.25+0.07) 1.32(1.24+0.08) +0.0%\n4002.6: log --color-moved-ws=allow-indentation-change 1.71 (1.64+0.06) 1.36(1.25+0.10) -20.5%\n\nTest master HEAD\n---------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.33+0.05) 0.38(0.35+0.03) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.80 (0.75+0.04) 0.87(0.83+0.04) +8.7%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 14.20(14.15+0.05) 0.97(0.92+0.04) -93.2%\n4002.4: log --no-color-moved --no-color-moved-ws 1.15 (1.05+0.09) 1.17(1.06+0.10) +1.7%\n4002.5: log --color-moved --no-color-moved-ws 1.30 (1.19+0.11) 1.32(1.24+0.08) +1.5%\n4002.6: log --color-moved-ws=allow-indentation-change 1.70 (1.63+0.06) 1.36(1.25+0.10) -20.0%\n\nHelped-by: Jeff King \nSigned-off-by: Phillip Wood \n---\n diff.c | 70 +++++++++++++++++-----------------------------------------\n 1 file changed, 20 insertions(+), 50 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 9aff167be27..78a486021ab 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -879,37 +879,21 @@ static int compute_ws_delta(const struct emitted_diff_symbol *a,\n \treturn 1;\n }\n \n-static int cmp_in_block_with_wsd(const struct diff_options *o,\n-\t\t\t\t const struct moved_entry *cur,\n-\t\t\t\t const struct moved_entry *match,\n-\t\t\t\t struct moved_block *pmb,\n-\t\t\t\t int n)\n-{\n-\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n-\tint al = cur->es->len, bl = match->es->len, cl = l->len;\n+static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n+\t\t\t\t const struct emitted_diff_symbol *l,\n+\t\t\t\t struct moved_block *pmb)\n+{\n+\tint al = cur->es->len, bl = l->len;\n \tconst char *a = cur->es->line,\n-\t\t *b = match->es->line,\n-\t\t *c = l->line;\n+\t\t *b = l->line;\n \tint a_off = cur->es->indent_off,\n \t a_width = cur->es->indent_width,\n-\t c_off = l->indent_off,\n-\t c_width = l->indent_width;\n+\t b_off = l->indent_off,\n+\t b_width = l->indent_width;\n \tint delta;\n \n-\t/*\n-\t * We need to check if 'cur' is equal to 'match'. As those\n-\t * are from the same (+/-) side, we do not need to adjust for\n-\t * indent changes. However these were found using fuzzy\n-\t * matching so we do have to check if they are equal. Here we\n-\t * just check the lengths. We delay calling memcmp() to check\n-\t * the contents until later as if the length comparison for a\n-\t * and c fails we can avoid the call all together.\n-\t */\n-\tif (al != bl)\n-\t\treturn 1;\n-\n \t/* If 'l' and 'cur' are both blank then they match. */\n-\tif (a_width == INDENT_BLANKLINE && c_width == INDENT_BLANKLINE)\n+\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n \t\treturn 0;\n \n \t/*\n@@ -918,7 +902,7 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \t * match those of the current block and that the text of 'l' and 'cur'\n \t * after the indentation match.\n \t */\n-\tdelta = c_width - a_width;\n+\tdelta = b_width - a_width;\n \n \t/*\n \t * If the previous lines of this block were all blank then set its\n@@ -927,9 +911,8 @@ static int cmp_in_block_with_wsd(const struct diff_options *o,\n \tif (pmb->wsd == INDENT_BLANKLINE)\n \t\tpmb->wsd = delta;\n \n-\treturn !(delta == pmb->wsd && al - a_off == cl - c_off &&\n-\t\t !memcmp(a, b, al) && !\n-\t\t memcmp(a + a_off, c + c_off, al - a_off));\n+\treturn !(delta == pmb->wsd && al - a_off == bl - b_off &&\n+\t\t !memcmp(a + a_off, b + b_off, al - a_off));\n }\n \n static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n@@ -1030,36 +1013,23 @@ static void pmb_advance_or_null(struct diff_options *o,\n }\n \n static void pmb_advance_or_null_multi_match(struct diff_options *o,\n-\t\t\t\t\t struct moved_entry *match,\n-\t\t\t\t\t struct hashmap *hm,\n+\t\t\t\t\t struct emitted_diff_symbol *l,\n \t\t\t\t\t struct moved_block *pmb,\n-\t\t\t\t\t int pmb_nr, int n)\n+\t\t\t\t\t int pmb_nr)\n {\n \tint i;\n-\tchar *got_match = xcalloc(1, pmb_nr);\n-\n-\thashmap_for_each_entry_from(hm, match, ent) {\n-\t\tfor (i = 0; i < pmb_nr; i++) {\n-\t\t\tstruct moved_entry *prev = pmb[i].match;\n-\t\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n-\t\t\t\t\tprev->next_line : NULL;\n-\t\t\tif (!cur)\n-\t\t\t\tcontinue;\n-\t\t\tif (!cmp_in_block_with_wsd(o, cur, match, &pmb[i], n))\n-\t\t\t\tgot_match[i] |= 1;\n-\t\t}\n-\t}\n \n \tfor (i = 0; i < pmb_nr; i++) {\n-\t\tif (got_match[i]) {\n+\t\tstruct moved_entry *prev = pmb[i].match;\n+\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n+\t\t\tprev->next_line : NULL;\n+\t\tif (cur && !cmp_in_block_with_wsd(cur, l, &pmb[i])) {\n \t\t\t/* Advance to the next line */\n-\t\t\tpmb[i].match = pmb[i].match->next_line;\n+\t\t\tpmb[i].match = cur;\n \t\t} else {\n \t\t\tmoved_block_clear(&pmb[i]);\n \t\t}\n \t}\n-\n-\tfree(got_match);\n }\n \n static int shrink_potential_moved_blocks(struct moved_block *pmb,\n@@ -1223,7 +1193,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \n \t\tif (o->color_moved_ws_handling &\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\tpmb_advance_or_null_multi_match(o, match, hm, pmb, pmb_nr, n);\n+\t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n \t\telse\n \t\t\tpmb_advance_or_null(o, match, hm, pmb, pmb_nr);\n \n-- \ngitgitgadget\n\n"},{"id":"443636","messageId":"fb413cab3a86a8a880ce3a5b10769764bb0e49e6.1639045809.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","subject":"[PATCH v5 09/15] diff --color-moved: call comparison function directly","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:30:03Z","receivedAt":"2021-12-09T10:30:31Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nThis change will allow us to easily combine pmb_advance_or_null() and\npmb_advance_or_null_multi_match() in the next commit. Calling\nxdiff_compare_lines() directly rather than using a function pointer\nfrom the hash map has little effect on the run time.\n\nTest HEAD^ HEAD\n-------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38(0.35+0.03) 0.38(0.32+0.06) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.87(0.83+0.04) 0.87(0.80+0.06) +0.0%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 0.97(0.92+0.04) 0.97(0.93+0.04) +0.0%\n4002.4: log --no-color-moved --no-color-moved-ws 1.17(1.06+0.10) 1.16(1.10+0.05) -0.9%\n4002.5: log --color-moved --no-color-moved-ws 1.32(1.24+0.08) 1.31(1.22+0.09) -0.8%\n4002.6: log --color-moved-ws=allow-indentation-change 1.36(1.25+0.10) 1.35(1.25+0.10) -0.7%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 11 +++++++----\n 1 file changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 78a486021ab..22e0edac173 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -994,17 +994,20 @@ static void add_lines_to_move_detection(struct diff_options *o,\n }\n \n static void pmb_advance_or_null(struct diff_options *o,\n-\t\t\t\tstruct moved_entry *match,\n-\t\t\t\tstruct hashmap *hm,\n+\t\t\t\tstruct emitted_diff_symbol *l,\n \t\t\t\tstruct moved_block *pmb,\n \t\t\t\tint pmb_nr)\n {\n \tint i;\n+\tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n+\n \tfor (i = 0; i < pmb_nr; i++) {\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n \t\t\t\tprev->next_line : NULL;\n-\t\tif (cur && !hm->cmpfn(o, &cur->ent, &match->ent, NULL)) {\n+\t\tif (cur && xdiff_compare_lines(cur->es->line, cur->es->len,\n+\t\t\t\t\t\tl->line, l->len,\n+\t\t\t\t\t\tflags)) {\n \t\t\tpmb[i].match = cur;\n \t\t} else {\n \t\t\tpmb[i].match = NULL;\n@@ -1195,7 +1198,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n \t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n \t\telse\n-\t\t\tpmb_advance_or_null(o, match, hm, pmb, pmb_nr);\n+\t\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n \n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n-- \ngitgitgadget\n\n"},{"id":"443637","messageId":"ec8764082d5080284214b2c639f99984bad166ea.1639045809.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","subject":"[PATCH v5 10/15] diff --color-moved: unify moved block growth functions","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:30:04Z","receivedAt":"2021-12-09T10:30:33Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nAfter the last two commits pmb_advance_or_null() and\npmb_advance_or_null_multi_match() differ only in the comparison they\nperform. Lets simplify the code by combining them into a single\nfunction.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 41 ++++++++++++-----------------------------\n 1 file changed, 12 insertions(+), 29 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 22e0edac173..51f092e724e 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1002,36 +1002,23 @@ static void pmb_advance_or_null(struct diff_options *o,\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \n \tfor (i = 0; i < pmb_nr; i++) {\n+\t\tint match;\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n \t\t\t\tprev->next_line : NULL;\n-\t\tif (cur && xdiff_compare_lines(cur->es->line, cur->es->len,\n-\t\t\t\t\t\tl->line, l->len,\n-\t\t\t\t\t\tflags)) {\n-\t\t\tpmb[i].match = cur;\n-\t\t} else {\n-\t\t\tpmb[i].match = NULL;\n-\t\t}\n-\t}\n-}\n \n-static void pmb_advance_or_null_multi_match(struct diff_options *o,\n-\t\t\t\t\t struct emitted_diff_symbol *l,\n-\t\t\t\t\t struct moved_block *pmb,\n-\t\t\t\t\t int pmb_nr)\n-{\n-\tint i;\n-\n-\tfor (i = 0; i < pmb_nr; i++) {\n-\t\tstruct moved_entry *prev = pmb[i].match;\n-\t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n-\t\t\tprev->next_line : NULL;\n-\t\tif (cur && !cmp_in_block_with_wsd(cur, l, &pmb[i])) {\n-\t\t\t/* Advance to the next line */\n+\t\tif (o->color_moved_ws_handling &\n+\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n+\t\t\tmatch = cur &&\n+\t\t\t\t!cmp_in_block_with_wsd(cur, l, &pmb[i]);\n+\t\telse\n+\t\t\tmatch = cur &&\n+\t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n+\t\t\t\t\t\t l->line, l->len, flags);\n+\t\tif (match)\n \t\t\tpmb[i].match = cur;\n-\t\t} else {\n+\t\telse\n \t\t\tmoved_block_clear(&pmb[i]);\n-\t\t}\n \t}\n }\n \n@@ -1194,11 +1181,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tif (o->color_moved_ws_handling &\n-\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\tpmb_advance_or_null_multi_match(o, l, pmb, pmb_nr);\n-\t\telse\n-\t\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n+\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n \n \t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n \n-- \ngitgitgadget\n\n"},{"id":"443638","messageId":"6199a01454710f23bb96be9c73564365b77b40e5.1639045809.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","subject":"[PATCH v5 11/15] diff --color-moved: shrink potential moved blocks as we go","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:30:05Z","receivedAt":"2021-12-09T10:30:34Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nRather than setting `match` to NULL and then looping over the list of\npotential matched blocks for a second time to remove blocks with no\nmatches just filter out the blocks with no matches as we go.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 44 ++++++++------------------------------------\n 1 file changed, 8 insertions(+), 36 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 51f092e724e..626fd47aa0e 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -996,12 +996,12 @@ static void add_lines_to_move_detection(struct diff_options *o,\n static void pmb_advance_or_null(struct diff_options *o,\n \t\t\t\tstruct emitted_diff_symbol *l,\n \t\t\t\tstruct moved_block *pmb,\n-\t\t\t\tint pmb_nr)\n+\t\t\t\tint *pmb_nr)\n {\n-\tint i;\n+\tint i, j;\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \n-\tfor (i = 0; i < pmb_nr; i++) {\n+\tfor (i = 0, j = 0; i < *pmb_nr; i++) {\n \t\tint match;\n \t\tstruct moved_entry *prev = pmb[i].match;\n \t\tstruct moved_entry *cur = (prev && prev->next_line) ?\n@@ -1015,38 +1015,12 @@ static void pmb_advance_or_null(struct diff_options *o,\n \t\t\tmatch = cur &&\n \t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n \t\t\t\t\t\t l->line, l->len, flags);\n-\t\tif (match)\n-\t\t\tpmb[i].match = cur;\n-\t\telse\n-\t\t\tmoved_block_clear(&pmb[i]);\n-\t}\n-}\n-\n-static int shrink_potential_moved_blocks(struct moved_block *pmb,\n-\t\t\t\t\t int pmb_nr)\n-{\n-\tint lp, rp;\n-\n-\t/* Shrink the set of potential block to the remaining running */\n-\tfor (lp = 0, rp = pmb_nr - 1; lp <= rp;) {\n-\t\twhile (lp < pmb_nr && pmb[lp].match)\n-\t\t\tlp++;\n-\t\t/* lp points at the first NULL now */\n-\n-\t\twhile (rp > -1 && !pmb[rp].match)\n-\t\t\trp--;\n-\t\t/* rp points at the last non-NULL */\n-\n-\t\tif (lp < pmb_nr && rp > -1 && lp < rp) {\n-\t\t\tpmb[lp] = pmb[rp];\n-\t\t\tmemset(&pmb[rp], 0, sizeof(pmb[rp]));\n-\t\t\trp--;\n-\t\t\tlp++;\n+\t\tif (match) {\n+\t\t\tpmb[j] = pmb[i];\n+\t\t\tpmb[j++].match = cur;\n \t\t}\n \t}\n-\n-\t/* Remember the number of running sets */\n-\treturn rp + 1;\n+\t*pmb_nr = j;\n }\n \n static void fill_potential_moved_blocks(struct diff_options *o,\n@@ -1181,9 +1155,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tpmb_advance_or_null(o, l, pmb, pmb_nr);\n-\n-\t\tpmb_nr = shrink_potential_moved_blocks(pmb, pmb_nr);\n+\t\tpmb_advance_or_null(o, l, pmb, &pmb_nr);\n \n \t\tif (pmb_nr == 0) {\n \t\t\tint contiguous = adjust_last_block(o, n, block_length);\n-- \ngitgitgadget\n\n"},{"id":"443639","messageId":"1db84490ee42bf8ad681d217de00a902ce9734c2.1639045809.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","subject":"[PATCH v5 12/15] diff --color-moved: stop clearing potential moved blocks","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:30:06Z","receivedAt":"2021-12-09T10:30:38Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nmoved_block_clear() was introduced in 74d156f4a1 (\"diff\n--color-moved-ws: fix double free crash\", 2018-10-04) to free the\nmemory that was allocated when initializing a potential moved\nblock. However since 21536d077f (\"diff --color-moved-ws: modify\nallow-indentation-change\", 2018-11-23) initializing a potential moved\nblock no longer allocates any memory. Up until the last commit we were\nrelying on moved_block_clear() to set the `match` pointer to NULL when\na block stopped matching, but since that commit we do not clear a\nmoved block that does not match so it does not make sense to clear\nthem elsewhere.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 11 -----------\n 1 file changed, 11 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 626fd47aa0e..ffbe09937bc 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -807,11 +807,6 @@ struct moved_block {\n \tint wsd; /* The whitespace delta of this block */\n };\n \n-static void moved_block_clear(struct moved_block *b)\n-{\n-\tmemset(b, 0, sizeof(*b));\n-}\n-\n #define INDENT_BLANKLINE INT_MIN\n \n static void fill_es_indent_data(struct emitted_diff_symbol *es)\n@@ -1128,8 +1123,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t}\n \n \t\tif (pmb_nr && (!match || l->s != moved_symbol)) {\n-\t\t\tint i;\n-\n \t\t\tif (!adjust_last_block(o, n, block_length) &&\n \t\t\t block_length > 1) {\n \t\t\t\t/*\n@@ -1139,8 +1132,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t\tmatch = NULL;\n \t\t\t\tn -= block_length;\n \t\t\t}\n-\t\t\tfor(i = 0; i < pmb_nr; i++)\n-\t\t\t\tmoved_block_clear(&pmb[i]);\n \t\t\tpmb_nr = 0;\n \t\t\tblock_length = 0;\n \t\t\tflipped_block = 0;\n@@ -1193,8 +1184,6 @@ static void mark_color_as_moved(struct diff_options *o,\n \t}\n \tadjust_last_block(o, n, block_length);\n \n-\tfor(n = 0; n < pmb_nr; n++)\n-\t\tmoved_block_clear(&pmb[n]);\n \tfree(pmb);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"443640","messageId":"3e769bab78cdef61be4b3e06dbe15445f1983a1f.1639045809.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","subject":"[PATCH v5 13/15] diff --color-moved-ws=allow-indentation-change: improve hash lookups","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:30:07Z","receivedAt":"2021-12-09T10:30:39Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nAs libxdiff does not have a whitespace flag to ignore the indentation\nthe code for --color-moved-ws=allow-indentation-change uses\nXDF_IGNORE_WHITESPACE and then filters out any hash lookups where\nthere are non-indentation changes. This filtering is inefficient as\nwe have to perform another string comparison.\n\nBy using the offset data that we have already computed to skip the\nindentation we can avoid using XDF_IGNORE_WHITESPACE and safely remove\nthe extra checks which improves the performance by 11% and paves the\nway for the elimination of string comparisons in the next commit.\n\nThis change slightly increases the run time of other --color-moved\nmodes. This could be avoided by using different comparison functions\nfor the different modes but after the next two commits there is no\nmeasurable benefit in doing so.\n\nThere is a change in behavior for lines that begin with a form-feed or\nvertical-tab character. Since b46054b374 (\"xdiff: use\ngit-compat-util\", 2019-04-11) xdiff does not treat '\\f' or '\\v' as\nwhitespace characters. This means that lines starting with those\ncharacters are never considered to be blank and never match a line\nthat does not start with the same character. After this patch a line\nmatching \"^[\\f\\v\\r]*[ \\t]*$\" is considered to be blank by\n--color-moved-ws=allow-indentation-change and lines beginning\n\"^[\\f\\v\\r]*[ \\t]*\" can match another line if the suffixes match. This\nchanges the output of git show for d18f76dccf (\"compat/regex: use the\nregex engine from gawk for compat\", 2010-08-17) as some lines in the\npre-image before a moved block that contain '\\f' are now considered\nmoved as well as they match a blank line before the moved lines in the\npost-image. This commit updates one of the tests to reflect this\nchange.\n\nTest HEAD^ HEAD\n--------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38(0.33+0.05) 0.38(0.33+0.05) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.86(0.82+0.04) 0.88(0.84+0.04) +2.3%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 0.97(0.94+0.03) 0.86(0.81+0.05) -11.3%\n4002.4: log --no-color-moved --no-color-moved-ws 1.16(1.07+0.09) 1.16(1.06+0.09) +0.0%\n4002.5: log --color-moved --no-color-moved-ws 1.32(1.26+0.06) 1.33(1.27+0.05) +0.8%\n4002.6: log --color-moved-ws=allow-indentation-change 1.35(1.29+0.06) 1.33(1.24+0.08) -1.5%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 65 +++++++++++---------------------------\n t/t4015-diff-whitespace.sh | 22 ++++++-------\n 2 files changed, 30 insertions(+), 57 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex ffbe09937bc..2085c063675 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -850,28 +850,15 @@ static void fill_es_indent_data(struct emitted_diff_symbol *es)\n }\n \n static int compute_ws_delta(const struct emitted_diff_symbol *a,\n-\t\t\t const struct emitted_diff_symbol *b,\n-\t\t\t int *out)\n-{\n-\tint a_len = a->len,\n-\t b_len = b->len,\n-\t a_off = a->indent_off,\n-\t a_width = a->indent_width,\n-\t b_off = b->indent_off,\n+\t\t\t const struct emitted_diff_symbol *b)\n+{\n+\tint a_width = a->indent_width,\n \t b_width = b->indent_width;\n \n-\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE) {\n-\t\t*out = INDENT_BLANKLINE;\n-\t\treturn 1;\n-\t}\n-\n-\tif (a_len - a_off != b_len - b_off ||\n-\t memcmp(a->line + a_off, b->line + b_off, a_len - a_off))\n-\t\treturn 0;\n-\n-\t*out = a_width - b_width;\n+\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n+\t\treturn INDENT_BLANKLINE;\n \n-\treturn 1;\n+\treturn a_width - b_width;\n }\n \n static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n@@ -916,26 +903,17 @@ static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n \t\t\t const void *keydata)\n {\n \tconst struct diff_options *diffopt = hashmap_cmp_fn_data;\n-\tconst struct moved_entry *a, *b;\n+\tconst struct emitted_diff_symbol *a, *b;\n \tunsigned flags = diffopt->color_moved_ws_handling\n \t\t\t & XDF_WHITESPACE_FLAGS;\n \n-\ta = container_of(eptr, const struct moved_entry, ent);\n-\tb = container_of(entry_or_key, const struct moved_entry, ent);\n+\ta = container_of(eptr, const struct moved_entry, ent)->es;\n+\tb = container_of(entry_or_key, const struct moved_entry, ent)->es;\n \n-\tif (diffopt->color_moved_ws_handling &\n-\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t/*\n-\t\t * As there is not specific white space config given,\n-\t\t * we'd need to check for a new block, so ignore all\n-\t\t * white space. The setup of the white space\n-\t\t * configuration for the next block is done else where\n-\t\t */\n-\t\tflags |= XDF_IGNORE_WHITESPACE;\n-\n-\treturn !xdiff_compare_lines(a->es->line, a->es->len,\n-\t\t\t\t b->es->line, b->es->len,\n-\t\t\t\t flags);\n+\treturn !xdiff_compare_lines(a->line + a->indent_off,\n+\t\t\t\t a->len - a->indent_off,\n+\t\t\t\t b->line + b->indent_off,\n+\t\t\t\t b->len - b->indent_off, flags);\n }\n \n static struct moved_entry *prepare_entry(struct diff_options *o,\n@@ -944,7 +922,8 @@ static struct moved_entry *prepare_entry(struct diff_options *o,\n \tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n \tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[line_no];\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n-\tunsigned int hash = xdiff_hash_string(l->line, l->len, flags);\n+\tunsigned int hash = xdiff_hash_string(l->line + l->indent_off,\n+\t\t\t\t\t l->len - l->indent_off, flags);\n \n \thashmap_entry_init(&ret->ent, hash);\n \tret->es = l;\n@@ -1036,13 +1015,11 @@ static void fill_potential_moved_blocks(struct diff_options *o,\n \thashmap_for_each_entry_from(hm, match, ent) {\n \t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n \t\tif (o->color_moved_ws_handling &\n-\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE) {\n-\t\t\tif (compute_ws_delta(l, match->es, &(pmb[pmb_nr]).wsd))\n-\t\t\t\tpmb[pmb_nr++].match = match;\n-\t\t} else {\n+\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n+\t\t\tpmb[pmb_nr].wsd = compute_ws_delta(l, match->es);\n+\t\telse\n \t\t\tpmb[pmb_nr].wsd = 0;\n-\t\t\tpmb[pmb_nr++].match = match;\n-\t\t}\n+\t\tpmb[pmb_nr++].match = match;\n \t}\n \n \t*pmb_p = pmb;\n@@ -6276,10 +6253,6 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \t\tif (o->color_moved) {\n \t\t\tstruct hashmap add_lines, del_lines;\n \n-\t\t\tif (o->color_moved_ws_handling &\n-\t\t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\t\to->color_moved_ws_handling |= XDF_IGNORE_WHITESPACE;\n-\n \t\t\thashmap_init(&del_lines, moved_entry_cmp, o, 0);\n \t\t\thashmap_init(&add_lines, moved_entry_cmp, o, 0);\n \ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 15782c879d2..50d0cf486be 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -2206,10 +2206,10 @@ EMPTY=''\n test_expect_success 'compare mixed whitespace delta across moved blocks' '\n \n \tgit reset --hard &&\n-\ttr Q_ \"\\t \" <<-EOF >text.txt &&\n-\t${EMPTY}\n-\t____too short without\n-\t${EMPTY}\n+\ttr \"^|Q_\" \"\\f\\v\\t \" <<-EOF >text.txt &&\n+\t^__\n+\t|____too short without\n+\t^\n \t___being grouped across blank line\n \t${EMPTY}\n \tcontext\n@@ -2228,7 +2228,7 @@ test_expect_success 'compare mixed whitespace delta across moved blocks' '\n \tgit add text.txt &&\n \tgit commit -m \"add text.txt\" &&\n \n-\ttr Q_ \"\\t \" <<-EOF >text.txt &&\n+\ttr \"^|Q_\" \"\\f\\v\\t \" <<-EOF >text.txt &&\n \tcontext\n \tlines\n \tto\n@@ -2239,7 +2239,7 @@ test_expect_success 'compare mixed whitespace delta across moved blocks' '\n \t${EMPTY}\n \tQQtoo short without\n \t${EMPTY}\n-\tQ_______being grouped across blank line\n+\t^Q_______being grouped across blank line\n \t${EMPTY}\n \tQ_QThese two lines have had their\n \tindentation reduced by four spaces\n@@ -2251,16 +2251,16 @@ test_expect_success 'compare mixed whitespace delta across moved blocks' '\n \t\t-c core.whitespace=space-before-tab \\\n \t\tdiff --color --color-moved --ws-error-highlight=all \\\n \t\t--color-moved-ws=allow-indentation-change >actual.raw &&\n-\tgrep -v \"index\" actual.raw | test_decode_color >actual &&\n+\tgrep -v \"index\" actual.raw | tr \"\\f\\v\" \"^|\" | test_decode_color >actual &&\n \n \tcat <<-\\EOF >expected &&\n \tdiff --git a/text.txt b/text.txt\n \t--- a/text.txt\n \t+++ b/text.txt\n \t@@ -1,16 +1,16 @@\n-\t-\n-\t- too short without\n-\t-\n+\t-^ \n+\t-| too short without\n+\t-^\n \t- being grouped across blank line\n \t-\n \t context\n@@ -2280,7 +2280,7 @@ test_expect_success 'compare mixed whitespace delta across moved blocks' '\n \t+\n \t+\t\ttoo short without\n \t+\n-\t+\t being grouped across blank line\n+\t+^\t being grouped across blank line\n \t+\n \t+\t \tThese two lines have had their\n \t+indentation reduced by four spaces\n-- \ngitgitgadget\n\n"},{"id":"443642","messageId":"350fa55ce5eaf92f000f52668dfa7c2485b8660a.1639045810.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","subject":"[PATCH v5 15/15] diff --color-moved: intern strings","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:30:09Z","receivedAt":"2021-12-09T10:30:42Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nTaking inspiration from xdl_classify_record() assign an id to each\naddition and deletion such that lines that match for the current\n--color-moved-ws mode share the same unique id. This reduces the\nnumber of hash lookups a little (calculating the ids still involves\none hash lookup per line) but the main benefit is that when growing\nblocks of potentially moved lines we can replace string comparisons\nwhich involve chasing a pointer with a simple integer comparison. On a\nlarge diff this commit reduces the time to run 'diff --color-moved' by\n37% compared to the previous commit and 31% compared to master, for\n'diff --color-moved-ws=allow-indentation-change' the reduction is 28%\ncompared to the previous commit and 96% compared to master. There is\nlittle change in the performance of 'git log --patch' as the diffs are\nsmaller.\n\nTest HEAD^ HEAD\n---------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38(0.33+0.05) 0.38(0.33+0.05) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.88(0.81+0.06) 0.55(0.50+0.04) -37.5%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 0.85(0.79+0.06) 0.61(0.54+0.06) -28.2%\n4002.4: log --no-color-moved --no-color-moved-ws 1.16(1.07+0.08) 1.15(1.09+0.05) -0.9%\n4002.5: log --color-moved --no-color-moved-ws 1.31(1.22+0.08) 1.29(1.19+0.09) -1.5%\n4002.6: log --color-moved-ws=allow-indentation-change 1.32(1.24+0.08) 1.31(1.18+0.13) -0.8%\n\nTest master HEAD\n---------------------------------------------------------------------------------------------------------------\n4002.1: diff --no-color-moved --no-color-moved-ws large change 0.38 (0.33+0.05) 0.38(0.33+0.05) +0.0%\n4002.2: diff --color-moved --no-color-moved-ws large change 0.80 (0.75+0.04) 0.55(0.50+0.04) -31.2%\n4002.3: diff --color-moved-ws=allow-indentation-change large change 14.20(14.15+0.05) 0.61(0.54+0.06) -95.7%\n4002.4: log --no-color-moved --no-color-moved-ws 1.15 (1.05+0.09) 1.15(1.09+0.05) +0.0%\n4002.5: log --color-moved --no-color-moved-ws 1.30 (1.19+0.11) 1.29(1.19+0.09) -0.8%\n4002.6: log --color-moved-ws=allow-indentation-change 1.70 (1.63+0.06) 1.31(1.18+0.13) -22.9%\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 174 +++++++++++++++++++++++++++++++--------------------------\n 1 file changed, 96 insertions(+), 78 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 9ef88d7665a..c28c56c1283 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -18,6 +18,7 @@\n #include \"submodule-config.h\"\n #include \"submodule.h\"\n #include \"hashmap.h\"\n+#include \"mem-pool.h\"\n #include \"ll-merge.h\"\n #include \"string-list.h\"\n #include \"strvec.h\"\n@@ -772,6 +773,7 @@ struct emitted_diff_symbol {\n \tint flags;\n \tint indent_off; /* Offset to first non-whitespace character */\n \tint indent_width; /* The visual width of the indentation */\n+\tunsigned id;\n \tenum diff_symbol s;\n };\n #define EMITTED_DIFF_SYMBOL_INIT {NULL}\n@@ -797,9 +799,9 @@ static void append_emitted_diff_symbol(struct diff_options *o,\n }\n \n struct moved_entry {\n-\tstruct hashmap_entry ent;\n \tconst struct emitted_diff_symbol *es;\n \tstruct moved_entry *next_line;\n+\tstruct moved_entry *next_match;\n };\n \n struct moved_block {\n@@ -865,24 +867,24 @@ static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n \t\t\t\t const struct emitted_diff_symbol *l,\n \t\t\t\t struct moved_block *pmb)\n {\n-\tint al = cur->es->len, bl = l->len;\n-\tconst char *a = cur->es->line,\n-\t\t *b = l->line;\n-\tint a_off = cur->es->indent_off,\n-\t a_width = cur->es->indent_width,\n-\t b_off = l->indent_off,\n-\t b_width = l->indent_width;\n+\tint a_width = cur->es->indent_width, b_width = l->indent_width;\n \tint delta;\n \n-\t/* If 'l' and 'cur' are both blank then they match. */\n-\tif (a_width == INDENT_BLANKLINE && b_width == INDENT_BLANKLINE)\n+\t/* The text of each line must match */\n+\tif (cur->es->id != l->id)\n+\t\treturn 1;\n+\n+\t/*\n+\t * If 'l' and 'cur' are both blank then we don't need to check the\n+\t * indent. We only need to check cur as we know the strings match.\n+\t * */\n+\tif (a_width == INDENT_BLANKLINE)\n \t\treturn 0;\n \n \t/*\n \t * The indent changes of the block are known and stored in pmb->wsd;\n \t * however we need to check if the indent changes of the current line\n-\t * match those of the current block and that the text of 'l' and 'cur'\n-\t * after the indentation match.\n+\t * match those of the current block.\n \t */\n \tdelta = b_width - a_width;\n \n@@ -893,22 +895,26 @@ static int cmp_in_block_with_wsd(const struct moved_entry *cur,\n \tif (pmb->wsd == INDENT_BLANKLINE)\n \t\tpmb->wsd = delta;\n \n-\treturn !(delta == pmb->wsd && al - a_off == bl - b_off &&\n-\t\t !memcmp(a + a_off, b + b_off, al - a_off));\n+\treturn delta != pmb->wsd;\n }\n \n-static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n-\t\t\t const struct hashmap_entry *eptr,\n-\t\t\t const struct hashmap_entry *entry_or_key,\n-\t\t\t const void *keydata)\n+struct interned_diff_symbol {\n+\tstruct hashmap_entry ent;\n+\tstruct emitted_diff_symbol *es;\n+};\n+\n+static int interned_diff_symbol_cmp(const void *hashmap_cmp_fn_data,\n+\t\t\t\t const struct hashmap_entry *eptr,\n+\t\t\t\t const struct hashmap_entry *entry_or_key,\n+\t\t\t\t const void *keydata)\n {\n \tconst struct diff_options *diffopt = hashmap_cmp_fn_data;\n \tconst struct emitted_diff_symbol *a, *b;\n \tunsigned flags = diffopt->color_moved_ws_handling\n \t\t\t & XDF_WHITESPACE_FLAGS;\n \n-\ta = container_of(eptr, const struct moved_entry, ent)->es;\n-\tb = container_of(entry_or_key, const struct moved_entry, ent)->es;\n+\ta = container_of(eptr, const struct interned_diff_symbol, ent)->es;\n+\tb = container_of(entry_or_key, const struct interned_diff_symbol, ent)->es;\n \n \treturn !xdiff_compare_lines(a->line + a->indent_off,\n \t\t\t\t a->len - a->indent_off,\n@@ -916,55 +922,81 @@ static int moved_entry_cmp(const void *hashmap_cmp_fn_data,\n \t\t\t\t b->len - b->indent_off, flags);\n }\n \n-static struct moved_entry *prepare_entry(struct diff_options *o,\n-\t\t\t\t\t int line_no)\n+static void prepare_entry(struct diff_options *o, struct emitted_diff_symbol *l,\n+\t\t\t struct interned_diff_symbol *s)\n {\n-\tstruct moved_entry *ret = xmalloc(sizeof(*ret));\n-\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[line_no];\n \tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \tunsigned int hash = xdiff_hash_string(l->line + l->indent_off,\n \t\t\t\t\t l->len - l->indent_off, flags);\n \n-\thashmap_entry_init(&ret->ent, hash);\n-\tret->es = l;\n-\tret->next_line = NULL;\n-\n-\treturn ret;\n+\thashmap_entry_init(&s->ent, hash);\n+\ts->es = l;\n }\n \n-static void add_lines_to_move_detection(struct diff_options *o,\n-\t\t\t\t\tstruct hashmap *add_lines,\n-\t\t\t\t\tstruct hashmap *del_lines)\n+struct moved_entry_list {\n+\tstruct moved_entry *add, *del;\n+};\n+\n+static struct moved_entry_list *add_lines_to_move_detection(struct diff_options *o,\n+\t\t\t\t\t\t\t struct mem_pool *entry_mem_pool)\n {\n \tstruct moved_entry *prev_line = NULL;\n-\n+\tstruct mem_pool interned_pool;\n+\tstruct hashmap interned_map;\n+\tstruct moved_entry_list *entry_list = NULL;\n+\tsize_t entry_list_alloc = 0;\n+\tunsigned id = 0;\n \tint n;\n+\n+\thashmap_init(&interned_map, interned_diff_symbol_cmp, o, 8096);\n+\tmem_pool_init(&interned_pool, 1024 * 1024);\n+\n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n-\t\tstruct hashmap *hm;\n-\t\tstruct moved_entry *key;\n+\t\tstruct interned_diff_symbol key;\n+\t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n+\t\tstruct interned_diff_symbol *s;\n+\t\tstruct moved_entry *entry;\n \n-\t\tswitch (o->emitted_symbols->buf[n].s) {\n-\t\tcase DIFF_SYMBOL_PLUS:\n-\t\t\thm = add_lines;\n-\t\t\tbreak;\n-\t\tcase DIFF_SYMBOL_MINUS:\n-\t\t\thm = del_lines;\n-\t\t\tbreak;\n-\t\tdefault:\n+\t\tif (l->s != DIFF_SYMBOL_PLUS && l->s != DIFF_SYMBOL_MINUS) {\n \t\t\tprev_line = NULL;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (o->color_moved_ws_handling &\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n-\t\t\tfill_es_indent_data(&o->emitted_symbols->buf[n]);\n-\t\tkey = prepare_entry(o, n);\n-\t\tif (prev_line && prev_line->es->s == o->emitted_symbols->buf[n].s)\n-\t\t\tprev_line->next_line = key;\n+\t\t\tfill_es_indent_data(l);\n \n-\t\thashmap_add(hm, &key->ent);\n-\t\tprev_line = key;\n+\t\tprepare_entry(o, l, &key);\n+\t\ts = hashmap_get_entry(&interned_map, &key, ent, &key.ent);\n+\t\tif (s) {\n+\t\t\tl->id = s->es->id;\n+\t\t} else {\n+\t\t\tl->id = id;\n+\t\t\tALLOC_GROW_BY(entry_list, id, 1, entry_list_alloc);\n+\t\t\thashmap_add(&interned_map,\n+\t\t\t\t memcpy(mem_pool_alloc(&interned_pool,\n+\t\t\t\t\t\t\t sizeof(key)),\n+\t\t\t\t\t &key, sizeof(key)));\n+\t\t}\n+\t\tentry = mem_pool_alloc(entry_mem_pool, sizeof(*entry));\n+\t\tentry->es = l;\n+\t\tentry->next_line = NULL;\n+\t\tif (prev_line && prev_line->es->s == l->s)\n+\t\t\tprev_line->next_line = entry;\n+\t\tprev_line = entry;\n+\t\tif (l->s == DIFF_SYMBOL_PLUS) {\n+\t\t\tentry->next_match = entry_list[l->id].add;\n+\t\t\tentry_list[l->id].add = entry;\n+\t\t} else {\n+\t\t\tentry->next_match = entry_list[l->id].del;\n+\t\t\tentry_list[l->id].del = entry;\n+\t\t}\n \t}\n+\n+\thashmap_clear(&interned_map);\n+\tmem_pool_discard(&interned_pool, 0);\n+\n+\treturn entry_list;\n }\n \n static void pmb_advance_or_null(struct diff_options *o,\n@@ -973,7 +1005,6 @@ static void pmb_advance_or_null(struct diff_options *o,\n \t\t\t\tint *pmb_nr)\n {\n \tint i, j;\n-\tunsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;\n \n \tfor (i = 0, j = 0; i < *pmb_nr; i++) {\n \t\tint match;\n@@ -986,9 +1017,8 @@ static void pmb_advance_or_null(struct diff_options *o,\n \t\t\tmatch = cur &&\n \t\t\t\t!cmp_in_block_with_wsd(cur, l, &pmb[i]);\n \t\telse\n-\t\t\tmatch = cur &&\n-\t\t\t\txdiff_compare_lines(cur->es->line, cur->es->len,\n-\t\t\t\t\t\t l->line, l->len, flags);\n+\t\t\tmatch = cur && cur->es->id == l->id;\n+\n \t\tif (match) {\n \t\t\tpmb[j] = pmb[i];\n \t\t\tpmb[j++].match = cur;\n@@ -998,7 +1028,6 @@ static void pmb_advance_or_null(struct diff_options *o,\n }\n \n static void fill_potential_moved_blocks(struct diff_options *o,\n-\t\t\t\t\tstruct hashmap *hm,\n \t\t\t\t\tstruct moved_entry *match,\n \t\t\t\t\tstruct emitted_diff_symbol *l,\n \t\t\t\t\tstruct moved_block **pmb_p,\n@@ -1012,7 +1041,7 @@ static void fill_potential_moved_blocks(struct diff_options *o,\n \t * The current line is the start of a new block.\n \t * Setup the set of potential blocks.\n \t */\n-\thashmap_for_each_entry_from(hm, match, ent) {\n+\tfor (; match; match = match->next_match) {\n \t\tALLOC_GROW(pmb, pmb_nr + 1, pmb_alloc);\n \t\tif (o->color_moved_ws_handling &\n \t\t COLOR_MOVED_WS_ALLOW_INDENTATION_CHANGE)\n@@ -1067,8 +1096,7 @@ static int adjust_last_block(struct diff_options *o, int n, int block_length)\n \n /* Find blocks of moved code, delegate actual coloring decision to helper */\n static void mark_color_as_moved(struct diff_options *o,\n-\t\t\t\tstruct hashmap *add_lines,\n-\t\t\t\tstruct hashmap *del_lines)\n+\t\t\t\tstruct moved_entry_list *entry_list)\n {\n \tstruct moved_block *pmb = NULL; /* potentially moved blocks */\n \tint pmb_nr = 0, pmb_alloc = 0;\n@@ -1077,23 +1105,15 @@ static void mark_color_as_moved(struct diff_options *o,\n \n \n \tfor (n = 0; n < o->emitted_symbols->nr; n++) {\n-\t\tstruct hashmap *hm = NULL;\n-\t\tstruct moved_entry *key;\n \t\tstruct moved_entry *match = NULL;\n \t\tstruct emitted_diff_symbol *l = &o->emitted_symbols->buf[n];\n \n \t\tswitch (l->s) {\n \t\tcase DIFF_SYMBOL_PLUS:\n-\t\t\thm = del_lines;\n-\t\t\tkey = prepare_entry(o, n);\n-\t\t\tmatch = hashmap_get_entry(hm, key, ent, NULL);\n-\t\t\tfree(key);\n+\t\t\tmatch = entry_list[l->id].del;\n \t\t\tbreak;\n \t\tcase DIFF_SYMBOL_MINUS:\n-\t\t\thm = add_lines;\n-\t\t\tkey = prepare_entry(o, n);\n-\t\t\tmatch = hashmap_get_entry(hm, key, ent, NULL);\n-\t\t\tfree(key);\n+\t\t\tmatch = entry_list[l->id].add;\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tflipped_block = 0;\n@@ -1135,7 +1155,7 @@ static void mark_color_as_moved(struct diff_options *o,\n \t\t\t\t */\n \t\t\t\tn -= block_length;\n \t\t\telse\n-\t\t\t\tfill_potential_moved_blocks(o, hm, match, l,\n+\t\t\t\tfill_potential_moved_blocks(o, match, l,\n \t\t\t\t\t\t\t &pmb, &pmb_alloc,\n \t\t\t\t\t\t\t &pmb_nr);\n \n@@ -6253,20 +6273,18 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o)\n \n \tif (o->emitted_symbols) {\n \t\tif (o->color_moved) {\n-\t\t\tstruct hashmap add_lines, del_lines;\n-\n-\t\t\thashmap_init(&del_lines, moved_entry_cmp, o, 0);\n-\t\t\thashmap_init(&add_lines, moved_entry_cmp, o, 0);\n+\t\t\tstruct mem_pool entry_pool;\n+\t\t\tstruct moved_entry_list *entry_list;\n \n-\t\t\tadd_lines_to_move_detection(o, &add_lines, &del_lines);\n-\t\t\tmark_color_as_moved(o, &add_lines, &del_lines);\n+\t\t\tmem_pool_init(&entry_pool, 1024 * 1024);\n+\t\t\tentry_list = add_lines_to_move_detection(o,\n+\t\t\t\t\t\t\t\t &entry_pool);\n+\t\t\tmark_color_as_moved(o, entry_list);\n \t\t\tif (o->color_moved == COLOR_MOVED_ZEBRA_DIM)\n \t\t\t\tdim_moved_lines(o);\n \n-\t\t\thashmap_clear_and_free(&add_lines, struct moved_entry,\n-\t\t\t\t\t\tent);\n-\t\t\thashmap_clear_and_free(&del_lines, struct moved_entry,\n-\t\t\t\t\t\tent);\n+\t\t\tmem_pool_discard(&entry_pool, 0);\n+\t\t\tfree(entry_list);\n \t\t}\n \n \t\tfor (i = 0; i < esm.nr; i++)\n-- \ngitgitgadget\n"},{"id":"443641","messageId":"b88696596641eba8f2bd5c09869972874e642023.1639045810.git.gitgitgadget@gmail.com","threadId":"55926","inReplyTo":"pull.981.v5.git.1639045809.gitgitgadget@gmail.com","subject":"[PATCH v5 14/15] diff: use designated initializers for emitted_diff_symbol","fromName":"Phillip Wood via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-09T10:30:08Z","receivedAt":"2021-12-09T10:30:43Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"From: Phillip Wood \n\nThis makes it clearer which fields are being explicitly initialized\nand will simplify the next commit where we add a new field to the\nstruct.\n\nSigned-off-by: Phillip Wood \n---\n diff.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/diff.c b/diff.c\nindex 2085c063675..9ef88d7665a 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1497,7 +1497,9 @@ static void emit_diff_symbol_from_struct(struct diff_options *o,\n static void emit_diff_symbol(struct diff_options *o, enum diff_symbol s,\n \t\t\t const char *line, int len, unsigned flags)\n {\n-\tstruct emitted_diff_symbol e = {line, len, flags, 0, 0, s};\n+\tstruct emitted_diff_symbol e = {\n+\t\t.line = line, .len = len, .flags = flags, .s = s\n+\t};\n \n \tif (o->emitted_symbols)\n \t\tappend_emitted_diff_symbol(o, &e);\n-- \ngitgitgadget\n\n"}]}