{"thread":{"id":"47259","subject":"[RFC PATCH v2 0/9] Improve merge recursive performance","startedAt":"2017-11-20T22:19:52Z","lastAt":"2017-11-20T22:20:19Z","messageCount":10,"participants":["Elijah Newren"],"isPatch":true,"patchVersion":2,"patchTotal":9},"messages":[{"id":"333000","messageId":"20171120221944.15431-1-newren@gmail.com","threadId":"47259","inReplyTo":null,"subject":"[RFC PATCH v2 0/9] Improve merge recursive performance","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-20T22:19:35Z","receivedAt":"2017-11-20T22:19:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"You probably want to wait to review this series until my rename detection\nseries lands so that I can clean up any more conflicts, but I'm posting\nthis new series in case anyone wants to take an early look. It includes\nfixes identified by the reviews of my other patch series, and has been\nrebased on top of newer versions with conflicts resolved.\n\nFor the adventerous, though...\n\nThis patch series improves merge recursive performance, particularly when\none side of history only makes a small number of changes, and the other has\nlots of renames (or lots of adds and deletes); I saw a speedup factor of\nover 30 on one particular real world repository.\n\nStill RFC for the same reasons that the first series was (see\nhttps://public-inbox.org/git/20171110222156.23221-1-newren@gmail.com/), I\nhaven't yet had time to push this further other than rebasing and cleaning\nup issues shared with my other series.\n\nIf you just want to test it out, it's available as the\n'big-repo-small-cherry-pick' branch of https://github.com/newren/git .\n\nElijah Newren (9):\n diffcore-rename: no point trying to find a match better than exact\n merge-recursive: avoid unnecessary string list lookups\n merge-recursive: new function for better colliding conflict\n resolutions\n Add testcases for improved file collision conflict handling\n merge-recursive: fix rename/add conflict handling\n merge-recursive: improve handling for rename/rename(2to1) conflicts\n merge-recursive: improve handling for add/add conflicts\n merge-recursive: accelerate rename detection\n diffcore-rename: filter rename_src list when possible\n\n diff.c | 1 +\n diff.h | 7 +\n diffcore-rename.c | 85 ++++++-\n merge-recursive.c | 452 ++++++++++++++++++++++++-----------\n t/t2023-checkout-m.sh | 2 +-\n t/t3418-rebase-continue.sh | 27 ++-\n t/t3504-cherry-pick-rerere.sh | 19 +-\n t/t4200-rerere.sh | 12 +-\n t/t6020-merge-df.sh | 4 +-\n t/t6024-recursive-merge.sh | 35 +--\n t/t6025-merge-symlinks.sh | 9 +-\n t/t6031-merge-filemode.sh | 4 +-\n t/t6036-recursive-corner-cases.sh | 19 +-\n t/t6042-merge-rename-corner-cases.sh | 212 +++++++++++++++-\n t/t6043-merge-rename-directories.sh | 15 +-\n t/t7060-wtstatus.sh | 1 +\n t/t7064-wtstatus-pv2.sh | 4 +-\n t/t7506-status-submodule.sh | 11 +-\n t/t7610-mergetool.sh | 28 +--\n 19 files changed, 723 insertions(+), 224 deletions(-)\n\n-- \n2.15.0.323.g31fe956618\n\n"},{"id":"333001","messageId":"20171120221944.15431-3-newren@gmail.com","threadId":"47259","inReplyTo":"20171120221944.15431-1-newren@gmail.com","subject":"[RFC PATCH v2 2/9] merge-recursive: avoid unnecessary string list lookups","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-20T22:19:37Z","receivedAt":"2017-11-20T22:19:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Since we're taking entries from active_cache, which is already in sorted\norder with same-named entries adjacent, we can skip a lookup. Also, we can\njust use append instead of insert (avoiding the need to find where to put\nthe new item) and still end up with a sorted list.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 14 ++++++++++----\n 1 file changed, 10 insertions(+), 4 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex cf8986be82..09b6092abb 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -467,22 +467,28 @@ static struct stage_data *insert_stage_data(const char *path,\n static struct string_list *get_unmerged(void)\n {\n \tstruct string_list *unmerged = xcalloc(1, sizeof(struct string_list));\n+\tstruct string_list_item *item;\n+\tconst char *last = NULL;\n \tint i;\n \n \tunmerged->strdup_strings = 1;\n \n \tfor (i = 0; i < active_nr; i++) {\n-\t\tstruct string_list_item *item;\n \t\tstruct stage_data *e;\n \t\tconst struct cache_entry *ce = active_cache[i];\n \t\tif (!ce_stage(ce))\n \t\t\tcontinue;\n \n-\t\titem = string_list_lookup(unmerged, ce->name);\n-\t\tif (!item) {\n-\t\t\titem = string_list_insert(unmerged, ce->name);\n+\t\tif (last == NULL || strcmp(last, ce->name)) {\n+\t\t\t/*\n+\t\t\t * active_cache is in sorted order, so we can just call\n+\t\t\t * string_list_append instead of string_list_insert and\n+\t\t\t * still end up with a sorted list.\n+\t\t\t */\n+\t\t\titem = string_list_append(unmerged, ce->name);\n \t\t\titem->util = xcalloc(1, sizeof(struct stage_data));\n \t\t}\n+\t\tlast = ce->name;\n \t\te = item->util;\n \t\te->stages[ce_stage(ce)].mode = ce->ce_mode;\n \t\toidcpy(&e->stages[ce_stage(ce)].oid, &ce->oid);\n-- \n2.15.0.323.g31fe956618\n\n"},{"id":"333002","messageId":"20171120221944.15431-4-newren@gmail.com","threadId":"47259","inReplyTo":"20171120221944.15431-1-newren@gmail.com","subject":"[RFC PATCH v2 3/9] merge-recursive: new function for better colliding conflict resolutions","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-20T22:19:38Z","receivedAt":"2017-11-20T22:19:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"There are three conflict types that represent two (possibly entirely\nunrelated) files colliding at the same location:\n * add/add\n * rename/add\n * rename/rename(2to1)\n\nThese three conflict types already share more similarity than might be\nimmediately apparent from their description: (1) the handling of the\nrename variants already involves removing any entries from the index\ncorresponding to the original file names[*], thus only leaving entries\nin the index for the colliding path; (2) likewise, any trace of the\noriginal file name in the working tree is also removed. So, in all\nthree cases we're left with how to represent two colliding files in both\nthe index and the working copy.\n\n[*] Technically, this isn't quite true because rename/rename(2to1)\nconflicts in the recursive (o->call_depth > 0) case do an \"unrename\"\nsince about seven years ago. But even in that case, Junio felt\ncompelled to explain that my decision to \"unrename\" wasn't necessarily\nthe only or right answer -- search for \"Comment from Junio\" in t6036 for\ndetails.\n\nMy initial motivation for looking at these three conflict types was that\nif the handling of these three conflict types is the \"same\" (defined\nmore precisely in a later commit), or is at least the \"same\" in the\nlimited set of cases where a renamed file is unmodified on the side of\nhistory where the file is not renamed, then a significant performance\nimprovement for rename detection during merges is possible. However,\nwhile that served as motivation to look at these three types of\nconflicts, the actual goal of this new function is to try to improve the\nhandling for all three cases, not to merely make them the same.\n\nThe previous behavior for these conflict types in regards to the\nworking tree (assuming the file collision occurs at 'foo') was:\n * add/add does a two-way merge of the two files and records it as 'foo'.\n * rename/rename(2to1) records the two different files into two new\n uniquely named files (foo~HEAD and foo~$MERGE), while removing 'foo'\n from the working tree.\n * rename/add records the two different files into two different\n locations, recording the add at foo~$SIDE and, oddly, recording\n the rename at foo (why is the rename more important than the add?)\n\nSo, the biggest question is whether the two colliding files should be\ntwo-way merged and recorded in place, or recorded into separate files.\nIf the files are similar enough, the two-way merge is probably\npreferable, but if they're not similar, recording as separate files is\nprobably preferable. (The same logic that applies for the working\ndirectory here would also apply to the recursive (i.e. o->call_depth >\n0) case as well.) The code handling the different types of conflicts\nappear to have been written with different assumptions about whether the\ncolliding files would be similar.\n\nBut, rather than make an assumption about whether the two files will be\nsimilar, why not just check? If we simply call estimate_similarity(),\nwe can two-way merge the files if they are similar, and otherwise record\nthe two files at different locations.\n\nChecking similarity in order to optimize worktree handling is the\nprimary improvement for these three conflict types. Further\nimprovements will be discussed in subsequent commits that modify the\ncode to take advantage of this new shared function.\n\nSigned-off-by: Elijah Newren \n---\n diff.h | 4 ++\n diffcore-rename.c | 6 +--\n merge-recursive.c | 121 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 128 insertions(+), 3 deletions(-)\n\ndiff --git a/diff.h b/diff.h\nindex 0fb18dd735..34dbc3cc05 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -438,4 +438,8 @@ extern void print_stat_summary(FILE *fp, int files,\n \t\t\t int insertions, int deletions);\n extern void setup_diff_pager(struct diff_options *);\n \n+extern int estimate_similarity(struct diff_filespec *src,\n+\t\t\t struct diff_filespec *dst,\n+\t\t\t int minimum_score);\n+\n #endif /* DIFF_H */\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex fe37e3b861..4fe5d0471c 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -127,9 +127,9 @@ struct diff_score {\n \tshort name_score;\n };\n \n-static int estimate_similarity(struct diff_filespec *src,\n-\t\t\t struct diff_filespec *dst,\n-\t\t\t int minimum_score)\n+int estimate_similarity(struct diff_filespec *src,\n+\t\t\tstruct diff_filespec *dst,\n+\t\t\tint minimum_score)\n {\n \t/* src points at a file that existed in the original tree (or\n \t * optionally a file in the destination tree) and dst points\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 09b6092abb..9dda13bf4f 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -1293,6 +1293,127 @@ static struct diff_filespec *filespec_from_entry(struct diff_filespec *target,\n \treturn target;\n }\n \n+static int handle_file_collision(struct merge_options *o,\n+\t\t\t\t const char *collide_path,\n+\t\t\t\t const char *prev_path1,\n+\t\t\t\t const char *prev_path2,\n+\t\t\t\t const char *branch1, const char *branch2,\n+\t\t\t\t const struct object_id *a_oid,\n+\t\t\t\t unsigned int a_mode,\n+\t\t\t\t const struct object_id *b_oid,\n+\t\t\t\t unsigned int b_mode,\n+\t\t\t\t unsigned int conflict_markers_already_present)\n+{\n+\tstruct merge_file_info mfi;\n+\tstruct diff_filespec null, a, b;\n+\tint minimum_score;\n+\n+\t/* Remove rename sources if rename/add or rename/rename(2to1) */\n+\tif (prev_path1)\n+\t\tremove_file(o, 1, prev_path1,\n+\t\t\t o->call_depth || would_lose_untracked(prev_path1));\n+\tif (prev_path2)\n+\t\tremove_file(o, 1, prev_path2,\n+\t\t\t o->call_depth || would_lose_untracked(prev_path2));\n+\n+\t/*\n+\t * Remove the collision path, if it wouldn't cause dirty contents\n+\t * or an untracked file to get lost. We'll either overwrite with\n+\t * merged contents, or just write out to differently named files.\n+\t */\n+\tif (was_dirty(o, collide_path))\n+\t\toutput(o, 1, _(\"Refusing to lose dirty file at %s\"),\n+\t\t collide_path);\n+\telse if (would_lose_untracked(collide_path))\n+\t\t/*\n+\t\t * Only way we get here is if both renames were from\n+\t\t * a directory rename AND user had an untracked file\n+\t\t * at the location where both files end up after the\n+\t\t * two directory renames. See testcase 10d of t6043.\n+\t\t */\n+\t\toutput(o, 1, _(\"Refusing to lose untracked file at \"\n+\t\t\t \"%s, even though it's in the way.\"),\n+\t\t collide_path);\n+\telse\n+\t\t/*\n+\t\t * FIXME: It's possible that neither of the two files have\n+\t\t * conflict markers already present, and that they're\n+\t\t * identical, and that the current working copy happens to\n+\t\t * match, in which case we are unnecessarily touching the\n+\t\t * working tree file. It's not a likely enough scenario\n+\t\t * that I want to code up the checks for it and a better\n+\t\t * fix is available if we restructure how unpack_trees()\n+\t\t * and merge-recursive interoperate anyway, so punting for\n+\t\t * now...\n+\t\t */\n+\t\tremove_file(o, 0, collide_path, 0);\n+\n+\t/* Store things in diff_filespecs for functions that need it */\n+\tmemset(&a, 0, sizeof(struct diff_filespec));\n+\tmemset(&b, 0, sizeof(struct diff_filespec));\n+\tnull.path = a.path = b.path = (char *)collide_path;\n+\toidcpy(&null.oid, &null_oid);\n+\tnull.mode = 0;\n+\toidcpy(&a.oid, a_oid);\n+\ta.mode = a_mode;\n+\ta.oid_valid = 1;\n+\toidcpy(&b.oid, b_oid);\n+\tb.mode = b_mode;\n+\tb.oid_valid = 1;\n+\n+\t/*\n+\t * If the colliding files are similar enough, we can simply merge\n+\t * them. But we don't want to merge files that have conflict\n+\t * markers in them already, because nested conflict markers are\n+\t * too confusing.\n+\t */\n+\tminimum_score = o->rename_score ? o->rename_score\n+\t\t\t\t\t: DEFAULT_RENAME_SCORE;\n+\tif (!conflict_markers_already_present && minimum_score <=\n+\t estimate_similarity(&a, &b, o->rename_score)) {\n+\t\tif (merge_file_1(o, &null, &a, &b, branch1, branch2, &mfi))\n+\t\t\treturn -1;\n+\t\tif (update_file(o, mfi.clean, &mfi.oid, mfi.mode, collide_path))\n+\t\t\treturn -1;\n+\t\tif (!mfi.clean && !o->call_depth &&\n+\t\t update_stages(o, collide_path, NULL, &a, &b))\n+\t\t\treturn -1;\n+\t\treturn mfi.clean;\n+\t}\n+\n+\t/*\n+\t * Put the colliding files into different paths, and record the\n+\t * updated sha1sums in the index\n+\t */\n+\tchar *new_path1 = (o->call_depth && prev_path1) ? strdup(prev_path1) :\n+\t\t\t unique_path(o, collide_path, branch1);\n+\tchar *new_path2 = (o->call_depth && prev_path2) ? strdup(prev_path2) :\n+\t\t\t unique_path(o, collide_path, branch2);\n+\toutput(o, 1, _(\"Renaming collisions at %s to %s and %s instead\"),\n+\t collide_path, new_path1, new_path2);\n+\n+\tif (update_file(o, 0, a_oid, a_mode, new_path1))\n+\t\treturn -1;\n+\tif (update_file(o, 0, b_oid, b_mode, new_path2))\n+\t\treturn -1;\n+\n+\t/* Update index too, making sure to get stage order correct. */\n+\tif (!o->call_depth) {\n+\t\tif (o->branch1 == branch1) {\n+\t\t\tif (update_stages(o, collide_path, NULL, &a, &b))\n+\t\t\t\treturn -1;\n+\t\t} else {\n+\t\t\tif (update_stages(o, collide_path, NULL, &b, &a))\n+\t\t\t\treturn -1;\n+\t\t}\n+\t}\n+\n+\tfree(new_path2);\n+\tfree(new_path1);\n+\n+\treturn 0; /* not clean */\n+}\n+\n static int handle_file(struct merge_options *o,\n \t\t\tstruct diff_filespec *rename,\n \t\t\tint stage,\n-- \n2.15.0.323.g31fe956618\n\n"},{"id":"333003","messageId":"20171120221944.15431-2-newren@gmail.com","threadId":"47259","inReplyTo":"20171120221944.15431-1-newren@gmail.com","subject":"[RFC PATCH v2 1/9] diffcore-rename: no point trying to find a match better than exact","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-20T22:19:36Z","receivedAt":"2017-11-20T22:19:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"diffcore_rename() had some code to avoid having destination paths that\nalready had an exact rename detected from being re-checked for other\nrenames. Source paths, however, were re-checked because we wanted to\nallow the possibility of detecting copies. But if copy detection isn't\nturned on, then this merely amounts to attempting to find a\nbetter-than-exact match, which naturally ends up being an expensive\nno-op. In particular, copy detection is never turned on by the merge\nrecursive machinery.\n\nIn a large repository (~50k files, about 60% of which was java) that had\na number of high level directories involved in renames, this cut the time\nnecessary for a cherry-pick down by about 50% (from around 9 minutes to\n4.5 minutes).\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 17 +++++++++++------\n 1 file changed, 11 insertions(+), 6 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 9ca0eaec70..fe37e3b861 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -377,11 +377,10 @@ static void record_if_better(struct diff_score m[], struct diff_score *o)\n * 1 if we need to disable inexact rename detection;\n * 2 if we would be under the limit if we were given -C instead of -C -C.\n */\n-static int too_many_rename_candidates(int num_create,\n+static int too_many_rename_candidates(int num_create, int num_src,\n \t\t\t\t struct diff_options *options)\n {\n \tint rename_limit = options->rename_limit;\n-\tint num_src = rename_src_nr;\n \tint i;\n \n \toptions->needed_rename_limit = 0;\n@@ -446,7 +445,7 @@ void diffcore_rename(struct diff_options *options)\n \tstruct diff_queue_struct outq;\n \tstruct diff_score *mx;\n \tint i, j, rename_count, skip_unmodified = 0;\n-\tint num_create, dst_cnt;\n+\tint num_create, dst_cnt, num_src;\n \tstruct progress *progress = NULL;\n \n \tif (!minimum_score)\n@@ -512,12 +511,14 @@ void diffcore_rename(struct diff_options *options)\n \t * files still remain as options for rename/copies!)\n \t */\n \tnum_create = (rename_dst_nr - rename_count);\n+\tnum_src = (detect_rename == DIFF_DETECT_COPY ?\n+\t\t rename_src_nr : rename_src_nr - rename_count);\n \n \t/* All done? */\n \tif (!num_create)\n \t\tgoto cleanup;\n \n-\tswitch (too_many_rename_candidates(num_create, options)) {\n+\tswitch (too_many_rename_candidates(num_create, num_src, options)) {\n \tcase 1:\n \t\tgoto cleanup;\n \tcase 2:\n@@ -531,7 +532,7 @@ void diffcore_rename(struct diff_options *options)\n \tif (options->show_rename_progress) {\n \t\tprogress = start_delayed_progress(\n \t\t\t\t_(\"Performing inexact rename detection\"),\n-\t\t\t\t(uint64_t)rename_dst_nr * (uint64_t)rename_src_nr);\n+\t\t\t\t(uint64_t)num_create * (uint64_t)num_src);\n \t}\n \n \tmx = xcalloc(st_mult(NUM_CANDIDATE_PER_DST, num_create), sizeof(*mx));\n@@ -550,6 +551,10 @@ void diffcore_rename(struct diff_options *options)\n \t\t\tstruct diff_filespec *one = rename_src[j].p->one;\n \t\t\tstruct diff_score this_src;\n \n+\t\t\tif (one->rename_used &&\n+\t\t\t detect_rename != DIFF_DETECT_COPY)\n+\t\t\t\tcontinue;\n+\n \t\t\tif (skip_unmodified &&\n \t\t\t diff_unmodified_pair(rename_src[j].p))\n \t\t\t\tcontinue;\n@@ -568,7 +573,7 @@ void diffcore_rename(struct diff_options *options)\n \t\t\tdiff_free_filespec_blob(two);\n \t\t}\n \t\tdst_cnt++;\n-\t\tdisplay_progress(progress, (uint64_t)(i+1)*(uint64_t)rename_src_nr);\n+\t\tdisplay_progress(progress, (uint64_t)dst_cnt*(uint64_t)num_src);\n \t}\n \tstop_progress(&progress);\n \n-- \n2.15.0.323.g31fe956618\n\n"},{"id":"333004","messageId":"20171120221944.15431-8-newren@gmail.com","threadId":"47259","inReplyTo":"20171120221944.15431-1-newren@gmail.com","subject":"[RFC PATCH v2 7/9] merge-recursive: improve handling for add/add conflicts","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-20T22:19:42Z","receivedAt":"2017-11-20T22:20:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This makes add/add conflicts use the new handle_file_collision()\nfunction. This leaves the handling of the index the same, but\nmodifies how the working tree is handled: instead of always doing\na two-way merge of the file contents and recording them at the\ncollision path name, we instead first estimate the similarity of the\ntwo files involved. If they are not similar, we instead record the\nfile contents into two separate files for the user to inspect.\n\nSeveral testcases had to be modified to either expect files to be\nwritten to different locations, or for the two test colliding files\nto be modified so that they were similar.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 25 ++++++++++++++++++++-----\n t/t2023-checkout-m.sh | 2 +-\n t/t3418-rebase-continue.sh | 27 +++++++++++++++++++++++----\n t/t3504-cherry-pick-rerere.sh | 19 ++++++++++++++-----\n t/t4200-rerere.sh | 12 ++++++------\n t/t6020-merge-df.sh | 4 ++--\n t/t6024-recursive-merge.sh | 35 +++++++++++++++++++++--------------\n t/t6025-merge-symlinks.sh | 9 ++++++---\n t/t6031-merge-filemode.sh | 4 ++--\n t/t6042-merge-rename-corner-cases.sh | 2 +-\n t/t6043-merge-rename-directories.sh | 15 ++++++++-------\n t/t7060-wtstatus.sh | 1 +\n t/t7064-wtstatus-pv2.sh | 4 ++--\n t/t7506-status-submodule.sh | 11 +++++++----\n t/t7610-mergetool.sh | 28 ++++++++++++++--------------\n 15 files changed, 128 insertions(+), 70 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex d818830098..e2d3b0fb4a 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -3051,11 +3051,26 @@ static int process_entry(struct merge_options *o,\n \t\t\t\tclean_merge = -1;\n \t\t}\n \t} else if (a_oid && b_oid) {\n-\t\t/* Case C: Added in both (check for same permissions) and */\n-\t\t/* case D: Modified in both, but differently. */\n-\t\tclean_merge = merge_content(o, path, 0 /* file_in_way */,\n-\t\t\t\t\t o_oid, o_mode, a_oid, a_mode, b_oid, b_mode,\n-\t\t\t\t\t NULL);\n+\t\tif (!o_oid) {\n+\t\t\t/* Case C: Added in both (check for same permissions) */\n+\t\t\toutput(o, 1,\n+\t\t\t _(\"CONFLICT (add/add): Merge conflict in %s\"),\n+\t\t\t path);\n+\t\t\tclean_merge = handle_file_collision(o,\n+\t\t\t\t\t\t\t path, NULL, NULL,\n+\t\t\t\t\t\t\t o->branch1,\n+\t\t\t\t\t\t\t o->branch2,\n+\t\t\t\t\t\t\t a_oid, a_mode,\n+\t\t\t\t\t\t\t b_oid, b_mode,\n+\t\t\t\t\t\t\t 0);\n+\t\t} else\n+\t\t\t/* case D: Modified in both, but differently. */\n+\t\t\tclean_merge = merge_content(o, path,\n+\t\t\t\t\t\t 0 /* file_in_way */,\n+\t\t\t\t\t\t o_oid, o_mode,\n+\t\t\t\t\t\t a_oid, a_mode,\n+\t\t\t\t\t\t b_oid, b_mode,\n+\t\t\t\t\t\t NULL);\n \t} else if (!o_oid && !a_oid && !b_oid) {\n \t\t/*\n \t\t * this entry was deleted altogether. a_mode == 0 means\ndiff --git a/t/t2023-checkout-m.sh b/t/t2023-checkout-m.sh\nindex 7e18985134..2f8ea52318 100755\n--- a/t/t2023-checkout-m.sh\n+++ b/t/t2023-checkout-m.sh\n@@ -27,7 +27,7 @@ clean_branchnames () {\n }\n \n test_expect_success '-m restores 2-way conflicted+resolved file' '\n-\tcp each.txt each.txt.conflicted &&\n+\ttest_must_fail git merge-file -p each.txt~HEAD /dev/null each.txt~master >each.txt.conflicted &&\n \techo resolved >each.txt &&\n \tgit add each.txt &&\n \tgit checkout -m -- each.txt &&\ndiff --git a/t/t3418-rebase-continue.sh b/t/t3418-rebase-continue.sh\nindex fcfdd197bd..3523558421 100755\n--- a/t/t3418-rebase-continue.sh\n+++ b/t/t3418-rebase-continue.sh\n@@ -10,10 +10,17 @@ set_fake_editor\n \n test_expect_success 'setup' '\n \ttest_commit \"commit-new-file-F1\" F1 1 &&\n-\ttest_commit \"commit-new-file-F2\" F2 2 &&\n+\tprintf \"1\\n2\\n2\\n\" >F2 &&\n+\tgit add F2 &&\n+\ttest_tick &&\n+\tgit commit -m \"commit-new-file-F2\" &&\n \n \tgit checkout -b topic HEAD^ &&\n-\ttest_commit \"commit-new-file-F2-on-topic-branch\" F2 22 &&\n+\tprintf \"1\\n2\\n22\\n\" >F2 &&\n+\tgit add F2 &&\n+\ttest_tick &&\n+\tgit commit -m \"commit-new-file-F2-on-topic-branch\" &&\n+\tgit tag \"commit-new-file-F2-on-topic-branch\" &&\n \n \tgit checkout master\n '\n@@ -48,7 +55,13 @@ test_expect_success 'rebase --continue can not be used with other options' '\n test_expect_success 'rebase --continue remembers merge strategy and options' '\n \trm -fr .git/rebase-* &&\n \tgit reset --hard commit-new-file-F2-on-topic-branch &&\n-\ttest_commit \"commit-new-file-F3-on-topic-branch\" F3 32 &&\n+\n+\tprintf \"1\\n2\\n32\\n\" >F3 &&\n+\tgit add F3 &&\n+\ttest_tick &&\n+\tgit commit -m \"commit-new-file-F3-on-topic-branch\" &&\n+\tgit tag \"commit-new-file-F3-on-topic-branch\" &&\n+\n \ttest_when_finished \"rm -fr test-bin funny.was.run\" &&\n \tmkdir test-bin &&\n \tcat >test-bin/git-merge-funny <<-EOF &&\n@@ -78,7 +91,13 @@ test_expect_success 'setup rerere database' '\n \trm -fr .git/rebase-* &&\n \tgit reset --hard commit-new-file-F3-on-topic-branch &&\n \tgit checkout master &&\n-\ttest_commit \"commit-new-file-F3\" F3 3 &&\n+\n+\tprintf \"1\\n2\\n3\\n\" >F3 &&\n+\tgit add F3 &&\n+\ttest_tick &&\n+\tgit commit -m \"commit-new-file-F3\" &&\n+\tgit tag \"commit-new-file-F3\" &&\n+\n \ttest_config rerere.enabled true &&\n \ttest_must_fail git rebase -m master topic &&\n \techo \"Resolved\" >F2 &&\ndiff --git a/t/t3504-cherry-pick-rerere.sh b/t/t3504-cherry-pick-rerere.sh\nindex a267b2d144..444ab06ad1 100755\n--- a/t/t3504-cherry-pick-rerere.sh\n+++ b/t/t3504-cherry-pick-rerere.sh\n@@ -4,14 +4,23 @@ test_description='cherry-pick should rerere for conflicts'\n \n . ./test-lib.sh\n \n+test_basic_commit () {\n+\tfile=$2 &&\n+\tprintf \"1\\n2\\n3\\n4\\n5\\n6\\n7\\n8\\n9\\n10\\n$1\" >\"$2\" &&\n+\tgit add \"$file\" &&\n+\ttest_tick\n+\tgit commit -m \"$1\" &&\n+\tgit tag \"$1\"\n+}\n+\n test_expect_success setup '\n-\ttest_commit foo &&\n-\ttest_commit foo-master foo &&\n-\ttest_commit bar-master bar &&\n+\ttest_basic_commit foo foo &&\n+\ttest_basic_commit foo-master foo &&\n+\ttest_basic_commit bar-master bar &&\n \n \tgit checkout -b dev foo &&\n-\ttest_commit foo-dev foo &&\n-\ttest_commit bar-dev bar &&\n+\ttest_basic_commit foo-dev foo &&\n+\ttest_basic_commit bar-dev bar &&\n \tgit config rerere.enabled true\n '\n \ndiff --git a/t/t4200-rerere.sh b/t/t4200-rerere.sh\nindex d97d2bebc9..80451907ce 100755\n--- a/t/t4200-rerere.sh\n+++ b/t/t4200-rerere.sh\n@@ -280,36 +280,36 @@ test_expect_success 'setup: file2 added differently in two branches' '\n \tgit reset --hard &&\n \n \tgit checkout -b fourth &&\n-\techo Hallo >file2 &&\n+\tprintf \"1\\n2\\n3\\n4\\n5\\n6\\n7\\n8\\n9\\nHallo\" >file2 &&\n \tgit add file2 &&\n \ttest_tick &&\n \tgit commit -m version1 &&\n \n \tgit checkout third &&\n-\techo Bello >file2 &&\n+\tprintf \"1\\n2\\n3\\n4\\n5\\n6\\n7\\n8\\n9\\nBello\" >file2 &&\n \tgit add file2 &&\n \ttest_tick &&\n \tgit commit -m version2 &&\n \n \ttest_must_fail git merge fourth &&\n-\techo Cello >file2 &&\n+\tprintf \"1\\n2\\n3\\n4\\n5\\n6\\n7\\n8\\n9\\nCello\" >file2 &&\n \tgit add file2 &&\n \tgit commit -m resolution\n '\n \n test_expect_success 'resolution was recorded properly' '\n-\techo Cello >expected &&\n+\tprintf \"1\\n2\\n3\\n4\\n5\\n6\\n7\\n8\\n9\\nCello\" >expected &&\n \n \tgit reset --hard HEAD~2 &&\n \tgit checkout -b fifth &&\n \n-\techo Hallo >file3 &&\n+\tprintf \"1\\n2\\n3\\n4\\n5\\n6\\n7\\n8\\n9\\nHallo\" >file3 &&\n \tgit add file3 &&\n \ttest_tick &&\n \tgit commit -m version1 &&\n \n \tgit checkout third &&\n-\techo Bello >file3 &&\n+\tprintf \"1\\n2\\n3\\n4\\n5\\n6\\n7\\n8\\n9\\nBello\" >file3 &&\n \tgit add file3 &&\n \ttest_tick &&\n \tgit commit -m version2 &&\ndiff --git a/t/t6020-merge-df.sh b/t/t6020-merge-df.sh\nindex 2af1beec5f..4ee27d2e20 100755\n--- a/t/t6020-merge-df.sh\n+++ b/t/t6020-merge-df.sh\n@@ -62,7 +62,7 @@ test_expect_success 'setup modify/delete + directory/file conflict' '\n \t# Throw in letters.txt for sorting order fun\n \t# (\"letters.txt\" sorts between \"letters\" and \"letters/file\")\n \techo i >>letters &&\n-\techo \"version 2\" >letters.txt &&\n+\tprintf \"1\\n2\\n3\\n4\\n5\\nversion 2\" >letters.txt &&\n \tgit add letters letters.txt &&\n \tgit commit -m modified &&\n \n@@ -70,7 +70,7 @@ test_expect_success 'setup modify/delete + directory/file conflict' '\n \tgit rm letters &&\n \tmkdir letters &&\n \t>letters/file &&\n-\techo \"version 1\" >letters.txt &&\n+\tprintf \"1\\n2\\n3\\n4\\n5\\nversion 1\" >letters.txt &&\n \tgit add letters letters.txt &&\n \tgit commit -m deleted\n '\ndiff --git a/t/t6024-recursive-merge.sh b/t/t6024-recursive-merge.sh\nindex 3f59e58dfb..354f70a66f 100755\n--- a/t/t6024-recursive-merge.sh\n+++ b/t/t6024-recursive-merge.sh\n@@ -15,47 +15,47 @@ GIT_COMMITTER_DATE=\"2006-12-12 23:28:00 +0100\"\n export GIT_COMMITTER_DATE\n \n test_expect_success \"setup tests\" '\n-echo 1 > a1 &&\n+printf \"1\\n2\\n3\\n4\\n1\\n\" > a1 &&\n git add a1 &&\n GIT_AUTHOR_DATE=\"2006-12-12 23:00:00\" git commit -m 1 a1 &&\n \n git checkout -b A master &&\n-echo A > a1 &&\n+printf \"1\\n2\\n3\\n4\\nA\\n\" > a1 &&\n GIT_AUTHOR_DATE=\"2006-12-12 23:00:01\" git commit -m A a1 &&\n \n git checkout -b B master &&\n-echo B > a1 &&\n+printf \"1\\n2\\n3\\n4\\nB\\n\" > a1 &&\n GIT_AUTHOR_DATE=\"2006-12-12 23:00:02\" git commit -m B a1 &&\n \n git checkout -b D A &&\n git rev-parse B > .git/MERGE_HEAD &&\n-echo D > a1 &&\n+printf \"1\\n2\\n3\\n4\\nD\\n\" > a1 &&\n git update-index a1 &&\n GIT_AUTHOR_DATE=\"2006-12-12 23:00:03\" git commit -m D &&\n \n git symbolic-ref HEAD refs/heads/other &&\n-echo 2 > a1 &&\n+printf \"1\\n2\\n3\\n4\\n2\\n\" > a1 &&\n GIT_AUTHOR_DATE=\"2006-12-12 23:00:04\" git commit -m 2 a1 &&\n \n git checkout -b C &&\n-echo C > a1 &&\n+printf \"1\\n2\\n3\\n4\\nC\\n\" > a1 &&\n GIT_AUTHOR_DATE=\"2006-12-12 23:00:05\" git commit -m C a1 &&\n \n git checkout -b E C &&\n git rev-parse B > .git/MERGE_HEAD &&\n-echo E > a1 &&\n+printf \"1\\n2\\n3\\n4\\nE\\n\" > a1 &&\n git update-index a1 &&\n GIT_AUTHOR_DATE=\"2006-12-12 23:00:06\" git commit -m E &&\n \n git checkout -b G E &&\n git rev-parse A > .git/MERGE_HEAD &&\n-echo G > a1 &&\n+printf \"1\\n2\\n3\\n4\\nG\\n\" > a1 &&\n git update-index a1 &&\n GIT_AUTHOR_DATE=\"2006-12-12 23:00:07\" git commit -m G &&\n \n git checkout -b F D &&\n git rev-parse C > .git/MERGE_HEAD &&\n-echo F > a1 &&\n+printf \"1\\n2\\n3\\n4\\nF\\n\" > a1 &&\n git update-index a1 &&\n GIT_AUTHOR_DATE=\"2006-12-12 23:00:08\" git commit -m F\n '\n@@ -65,6 +65,10 @@ test_expect_success \"combined merge conflicts\" \"\n \"\n \n cat > expect << EOF\n+1\n+2\n+3\n+4\n <<<<<<< HEAD\n F\n =======\n@@ -76,9 +80,9 @@ test_expect_success \"result contains a conflict\" \"test_cmp expect a1\"\n \n git ls-files --stage > out\n cat > expect << EOF\n-100644 ec3fe2a791706733f2d8fa7ad45d9a9672031f5e 1\ta1\n-100644 cf84443e49e1b366fac938711ddf4be2d4d1d9e9 2\ta1\n-100644 fd7923529855d0b274795ae3349c5e0438333979 3\ta1\n+100644 a2849e152d46797b83870a479775e980d9a138b1 1\ta1\n+100644 b1a886a67c3f471cec4d4112a91dc6caa6e3b709 2\ta1\n+100644 e9d143c719e446df2bafb4b6aba8ed511ecd63b1 3\ta1\n EOF\n \n test_expect_success \"virtual trees were processed\" \"test_cmp expect out\"\n@@ -87,10 +91,13 @@ test_expect_success 'refuse to merge binary files' '\n \tgit reset --hard &&\n \tprintf \"\\0\" > binary-file &&\n \tgit add binary-file &&\n-\tgit commit -m binary &&\n-\tgit checkout G &&\n+\tgit commit -m initial-binary &&\n \tprintf \"\\0\\0\" > binary-file &&\n \tgit add binary-file &&\n+\tgit commit -m binary1 &&\n+\tgit checkout -b H HEAD~1 &&\n+\tprintf \"\\0\\0\\0\" > binary-file &&\n+\tgit add binary-file &&\n \tgit commit -m binary2 &&\n \ttest_must_fail git merge F > merge.out 2> merge.err &&\n \tgrep \"Cannot merge binary files: binary-file (HEAD vs. F)\" merge.err\ndiff --git a/t/t6025-merge-symlinks.sh b/t/t6025-merge-symlinks.sh\nindex 433c4de08f..08eb85e703 100755\n--- a/t/t6025-merge-symlinks.sh\n+++ b/t/t6025-merge-symlinks.sh\n@@ -37,7 +37,8 @@ test_must_fail git merge master'\n \n test_expect_success \\\n 'the merge result must be a file' '\n-test -f symlink'\n+test -f symlink~HEAD &&\n+test -f symlink~master'\n \n test_expect_success \\\n 'merge master into b-file, which has a file instead of a symbolic link' '\n@@ -46,7 +47,8 @@ test_must_fail git merge master'\n \n test_expect_success \\\n 'the merge result must be a file' '\n-test -f symlink'\n+test -f symlink~HEAD &&\n+test -f symlink~master'\n \n test_expect_success \\\n 'merge b-file, which has a file instead of a symbolic link, into master' '\n@@ -56,6 +58,7 @@ test_must_fail git merge b-file'\n \n test_expect_success \\\n 'the merge result must be a file' '\n-test -f symlink'\n+test -f symlink~HEAD &&\n+test -f symlink~master'\n \n test_done\ndiff --git a/t/t6031-merge-filemode.sh b/t/t6031-merge-filemode.sh\nindex 7d06461f13..79e77f391f 100755\n--- a/t/t6031-merge-filemode.sh\n+++ b/t/t6031-merge-filemode.sh\n@@ -40,12 +40,12 @@ do_one_mode resolve b1 a1\n test_expect_success 'set up mode change in both branches' '\n \tgit reset --hard HEAD &&\n \tgit checkout -b a2 master &&\n-\t: >file2 &&\n+\techo true >file2 &&\n \tH=$(git hash-object file2) &&\n \ttest_chmod +x file2 &&\n \tgit commit -m a2 &&\n \tgit checkout -b b2 master &&\n-\t: >file2 &&\n+\techo true >file2 &&\n \tgit add file2 &&\n \tgit commit -m b2 &&\n \t{\ndiff --git a/t/t6042-merge-rename-corner-cases.sh b/t/t6042-merge-rename-corner-cases.sh\nindex b0b840223b..f06147e6ce 100755\n--- a/t/t6042-merge-rename-corner-cases.sh\n+++ b/t/t6042-merge-rename-corner-cases.sh\n@@ -730,7 +730,7 @@ test_conflicts_with_adds_and_renames() {\n \t\ttest_tick && git commit -m R\n \t'\n \n-\ttest_expect_failure \"check simple $side1/$side2 conflict\" '\n+\ttest_expect_success \"check simple $side1/$side2 conflict\" '\n \t\tgit reset --hard &&\n \t\tgit checkout L^0 &&\n \ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex f1538087ca..50506d951b 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -961,7 +961,7 @@ test_expect_success '5b-check: Rename/delete in order to get add/add/add conflic\n \n \t\ttest 5 -eq $(git ls-files -s | wc -l) &&\n \t\ttest 2 -eq $(git ls-files -u | wc -l) &&\n-\t\ttest 1 -eq $(git ls-files -o | wc -l) &&\n+\t\ttest 3 -eq $(git ls-files -o | wc -l) &&\n \n \t\tgit rev-parse >actual \\\n \t\t\t:0:y/b :0:y/c :0:y/e :2:y/d :3:y/d &&\n@@ -970,7 +970,9 @@ test_expect_success '5b-check: Rename/delete in order to get add/add/add conflic\n \t\ttest_cmp expect actual &&\n \n \t\ttest_must_fail git rev-parse :1:y/d &&\n-\t\ttest -f y/d\n+\t\ttest -f y/d~HEAD &&\n+\t\ttest -f y/d~B^0 &&\n+\t\ttest ! -f y/d\n \t)\n '\n \n@@ -1039,7 +1041,7 @@ test_expect_success '5c-check: Transitive rename would cause rename/rename/renam\n \n \t\ttest 9 -eq $(git ls-files -s | wc -l) &&\n \t\ttest 6 -eq $(git ls-files -u | wc -l) &&\n-\t\ttest 3 -eq $(git ls-files -o | wc -l) &&\n+\t\ttest 5 -eq $(git ls-files -o | wc -l) &&\n \n \t\tgit rev-parse >actual \\\n \t\t\t:0:y/b :0:y/c :0:y/e &&\n@@ -1055,13 +1057,12 @@ test_expect_success '5c-check: Transitive rename would cause rename/rename/renam\n \t\ttest_cmp expect actual &&\n \n \t\tgit hash-object >actual \\\n-\t\t\tw/d~HEAD w/d~B^0 &&\n+\t\t\tw/d~HEAD w/d~B^0 y/d~HEAD y/d~B^0 &&\n \t\tgit rev-parse >expect \\\n-\t\t\tO:x/d B:w/d &&\n+\t\t\tO:x/d B:w/d A:y/d B:y/d &&\n \t\ttest_cmp expect actual &&\n \t\ttest ! -f x/d &&\n-\t\ttest -f y/d &&\n-\t\tgrep -q \"<<<<\" y/d && # conflict markers should be present\n+\t\ttest ! -f y/d &&\n \t\ttest $(git hash-object z/d) = $(git rev-parse O:x/d)\n \t)\n '\ndiff --git a/t/t7060-wtstatus.sh b/t/t7060-wtstatus.sh\nindex 53cf42fac1..a7888f9fc3 100755\n--- a/t/t7060-wtstatus.sh\n+++ b/t/t7060-wtstatus.sh\n@@ -214,6 +214,7 @@ EOF\n \tgit status --untracked-files=no >actual &&\n \ttest_i18ncmp expected actual &&\n \tgit reset --hard &&\n+\tgit clean -f *~* &&\n \tgit checkout master\n '\n \ndiff --git a/t/t7064-wtstatus-pv2.sh b/t/t7064-wtstatus-pv2.sh\nindex e319fa2e84..c4d3b1d7e4 100755\n--- a/t/t7064-wtstatus-pv2.sh\n+++ b/t/t7064-wtstatus-pv2.sh\n@@ -259,14 +259,14 @@ test_expect_success 'verify AA (add-add) conflict' '\n \n \tgit branch AA_A master &&\n \tgit checkout AA_A &&\n-\techo \"Branch AA_A\" >conflict.txt &&\n+\tprintf \"1\\n2\\n3\\n4\\n5\\n6\\nBranch AA_A\" >conflict.txt &&\n \tOID_AA_A=$(git hash-object -t blob -- conflict.txt) &&\n \tgit add conflict.txt &&\n \tgit commit -m \"branch aa_a\" &&\n \n \tgit branch AA_B master &&\n \tgit checkout AA_B &&\n-\techo \"Branch AA_B\" >conflict.txt &&\n+\tprintf \"1\\n2\\n3\\n4\\n5\\n6\\nBranch AA_B\" >conflict.txt &&\n \tOID_AA_B=$(git hash-object -t blob -- conflict.txt) &&\n \tgit add conflict.txt &&\n \tgit commit -m \"branch aa_b\" &&\ndiff --git a/t/t7506-status-submodule.sh b/t/t7506-status-submodule.sh\nindex 9edf6572ed..0514a019be 100755\n--- a/t/t7506-status-submodule.sh\n+++ b/t/t7506-status-submodule.sh\n@@ -240,6 +240,8 @@ test_expect_success 'status -a clean (empty submodule dir)' '\n cat >status_expect <<\\EOF\n AA .gitmodules\n A sub1\n+?? .gitmodules~HEAD\n+?? .gitmodules~add_sub1\n EOF\n \n test_expect_success 'status with merge conflict in .gitmodules' '\n@@ -273,7 +275,7 @@ index badaa4c,44f999a..0000000\n --- a/.gitmodules\n +++ b/.gitmodules\n @@@ -1,3 -1,3 +1,9 @@@\n-++<<<<<<< HEAD\n+++<<<<<<< ours\n +[submodule \"sub2\"]\n +\tpath = sub2\n +\turl = ../sub2\n@@ -281,7 +283,7 @@ index badaa4c,44f999a..0000000\n + [submodule \"sub1\"]\n + \tpath = sub1\n + \turl = ../sub1\n-++>>>>>>> add_sub1\n+++>>>>>>> theirs\n EOF\n \n cat >diff_submodule_expect <<\\EOF\n@@ -290,7 +292,7 @@ index badaa4c,44f999a..0000000\n --- a/.gitmodules\n +++ b/.gitmodules\n @@@ -1,3 -1,3 +1,9 @@@\n-++<<<<<<< HEAD\n+++<<<<<<< ours\n +[submodule \"sub2\"]\n +\tpath = sub2\n +\turl = ../sub2\n@@ -298,12 +300,13 @@ index badaa4c,44f999a..0000000\n + [submodule \"sub1\"]\n + \tpath = sub1\n + \turl = ../sub1\n-++>>>>>>> add_sub1\n+++>>>>>>> theirs\n EOF\n \n test_expect_success 'diff with merge conflict in .gitmodules' '\n \t(\n \t\tcd super &&\n+\t\tgit checkout -m .gitmodules &&\n \t\tgit diff >../diff_actual 2>&1\n \t) &&\n \ttest_cmp diff_expect diff_actual\ndiff --git a/t/t7610-mergetool.sh b/t/t7610-mergetool.sh\nindex 1a430b9c40..71f41fc9a8 100755\n--- a/t/t7610-mergetool.sh\n+++ b/t/t7610-mergetool.sh\n@@ -37,9 +37,9 @@ test_expect_success 'setup' '\n \tgit checkout -b branch1 master &&\n \tgit submodule update -N &&\n \techo branch1 change >file1 &&\n-\techo branch1 newfile >file2 &&\n+\tprintf \"1\\n2\\n3\\n4\\n5\\n6\\n7\\n8\\n9\\nbranch1 newfile\\n\" >file2 &&\n \techo branch1 spaced >\"spaced name\" &&\n-\techo branch1 both added >both &&\n+\tprintf \"1\\n2\\n3\\n4\\n5\\n6\\n7\\n8\\n9\\n10\\nbranch1 both added\\n\" >both &&\n \techo branch1 change file11 >file11 &&\n \techo branch1 change file13 >file13 &&\n \techo branch1 sub >subdir/file3 &&\n@@ -84,9 +84,9 @@ test_expect_success 'setup' '\n \tgit checkout master &&\n \tgit submodule update -N &&\n \techo master updated >file1 &&\n-\techo master new >file2 &&\n+\tprintf \"1\\n2\\n3\\n4\\n5\\n6\\n7\\n8\\n9\\nmaster new\\n\" >file2 &&\n \techo master updated spaced >\"spaced name\" &&\n-\techo master both added >both &&\n+\tprintf \"1\\n2\\n3\\n4\\n5\\n6\\n7\\n8\\n9\\n10\\nmaster both added\\n\" >both &&\n \techo master updated file12 >file12 &&\n \techo master updated file14 >file14 &&\n \techo master new sub >subdir/file3 &&\n@@ -139,7 +139,7 @@ test_expect_success 'custom mergetool' '\n \t( yes \"d\" | git mergetool file12 >/dev/null 2>&1 ) &&\n \t( yes \"l\" | git mergetool submod >/dev/null 2>&1 ) &&\n \ttest \"$(cat file1)\" = \"master updated\" &&\n-\ttest \"$(cat file2)\" = \"master new\" &&\n+\ttest \"$(git hash-object file2)\" = \"$(git rev-parse master:file2)\" &&\n \ttest \"$(cat subdir/file3)\" = \"master new sub\" &&\n \ttest \"$(cat submod/bar)\" = \"branch1 submodule\" &&\n \tgit commit -m \"branch1 resolved with mergetool\"\n@@ -163,7 +163,7 @@ test_expect_success 'mergetool crlf' '\n \t( yes \"d\" | git mergetool file12 >/dev/null 2>&1 ) &&\n \t( yes \"r\" | git mergetool submod >/dev/null 2>&1 ) &&\n \ttest \"$(printf x | cat file1 -)\" = \"$(printf \"master updated\\r\\nx\")\" &&\n-\ttest \"$(printf x | cat file2 -)\" = \"$(printf \"master new\\r\\nx\")\" &&\n+\ttest \"$(printf x | cat file2 -)\" = \"$(printf \"1\\r\\n2\\r\\n3\\r\\n4\\r\\n5\\r\\n6\\r\\n7\\r\\n8\\r\\n9\\r\\nmaster new\\r\\nx\")\" &&\n \ttest \"$(printf x | cat subdir/file3 -)\" = \"$(printf \"master new sub\\r\\nx\")\" &&\n \tgit submodule update -N &&\n \ttest \"$(cat submod/bar)\" = \"master submodule\" &&\n@@ -171,7 +171,7 @@ test_expect_success 'mergetool crlf' '\n '\n \n test_expect_success 'mergetool in subdir' '\n-\ttest_when_finished \"git reset --hard\" &&\n+\ttest_when_finished \"git reset --hard && git clean -f\" &&\n \tgit checkout -b test$test_count branch1 &&\n \tgit submodule update -N &&\n \t(\n@@ -183,7 +183,7 @@ test_expect_success 'mergetool in subdir' '\n '\n \n test_expect_success 'mergetool on file in parent dir' '\n-\ttest_when_finished \"git reset --hard\" &&\n+\ttest_when_finished \"git reset --hard && git clean -f\" &&\n \tgit checkout -b test$test_count branch1 &&\n \tgit submodule update -N &&\n \t(\n@@ -197,14 +197,14 @@ test_expect_success 'mergetool on file in parent dir' '\n \t\t( yes \"d\" | git mergetool ../file12 >/dev/null 2>&1 ) &&\n \t\t( yes \"l\" | git mergetool ../submod >/dev/null 2>&1 ) &&\n \t\ttest \"$(cat ../file1)\" = \"master updated\" &&\n-\t\ttest \"$(cat ../file2)\" = \"master new\" &&\n+\t\ttest \"$(git hash-object ../file2)\" = \"$(git rev-parse master:file2)\" &&\n \t\ttest \"$(cat ../submod/bar)\" = \"branch1 submodule\" &&\n \t\tgit commit -m \"branch1 resolved with mergetool - subdir\"\n \t)\n '\n \n test_expect_success 'mergetool skips autoresolved' '\n-\ttest_when_finished \"git reset --hard\" &&\n+\ttest_when_finished \"git reset --hard && git clean -f\" &&\n \tgit checkout -b test$test_count branch1 &&\n \tgit submodule update -N &&\n \ttest_must_fail git merge master &&\n@@ -217,7 +217,7 @@ test_expect_success 'mergetool skips autoresolved' '\n '\n \n test_expect_success 'mergetool merges all from subdir (rerere disabled)' '\n-\ttest_when_finished \"git reset --hard\" &&\n+\ttest_when_finished \"git reset --hard && git clean -f\" &&\n \tgit checkout -b test$test_count branch1 &&\n \ttest_config rerere.enabled false &&\n \t(\n@@ -226,7 +226,7 @@ test_expect_success 'mergetool merges all from subdir (rerere disabled)' '\n \t\t( yes \"r\" | git mergetool ../submod ) &&\n \t\t( yes \"d\" \"d\" | git mergetool --no-prompt ) &&\n \t\ttest \"$(cat ../file1)\" = \"master updated\" &&\n-\t\ttest \"$(cat ../file2)\" = \"master new\" &&\n+\t\ttest \"$(git hash-object ../file2)\" = \"$(git rev-parse master:file2)\" &&\n \t\ttest \"$(cat file3)\" = \"master new sub\" &&\n \t\t( cd .. && git submodule update -N ) &&\n \t\ttest \"$(cat ../submod/bar)\" = \"master submodule\" &&\n@@ -245,7 +245,7 @@ test_expect_success 'mergetool merges all from subdir (rerere enabled)' '\n \t\t( yes \"r\" | git mergetool ../submod ) &&\n \t\t( yes \"d\" \"d\" | git mergetool --no-prompt ) &&\n \t\ttest \"$(cat ../file1)\" = \"master updated\" &&\n-\t\ttest \"$(cat ../file2)\" = \"master new\" &&\n+\t\ttest \"$(git hash-object ../file2)\" = \"$(git rev-parse master:file2)\" &&\n \t\ttest \"$(cat file3)\" = \"master new sub\" &&\n \t\t( cd .. && git submodule update -N ) &&\n \t\ttest \"$(cat ../submod/bar)\" = \"master submodule\" &&\n@@ -631,7 +631,7 @@ test_expect_success 'custom commands override built-ins' '\n \ttest_config mergetool.defaults.trustExitCode true &&\n \ttest_must_fail git merge master &&\n \tgit mergetool --no-prompt --tool defaults -- both &&\n-\techo master both added >expected &&\n+\tgit cat-file -p master:both >expected &&\n \ttest_cmp expected both\n '\n \n-- \n2.15.0.323.g31fe956618\n\n"},{"id":"333006","messageId":"20171120221944.15431-7-newren@gmail.com","threadId":"47259","inReplyTo":"20171120221944.15431-1-newren@gmail.com","subject":"[RFC PATCH v2 6/9] merge-recursive: improve handling for rename/rename(2to1) conflicts","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-20T22:19:41Z","receivedAt":"2017-11-20T22:20:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This makes the rename/rename(2to1) conflicts use the new\nhandle_file_collision() function. Since that function was based\noriginally on the rename/rename(2to1) handling code, the main\ndifferences here are in what was added. In particular:\n\n * If the two colliding files are similar, instead of being stored\n at collide_path~HEAD and collide_path~MERGE, the files are two-way\n merged and recorded at collide_path.\n * Instead of recording the version of the renamed file that existed\n on the renamed side in the index (thus ignoring any changes that\n were made to the file on the side of history without the rename),\n we do a three-way content merge on the renamed path, then store\n that at either stage 2 or stage 3.\n * Note that if either of the three-way content merges done for each\n rename have conflicts, we do NOT try to estimate the similarity of\n the resulting two files and just automatically consider them to be\n dissimilar. This is done to avoid foisting conflicts-of-conflicts\n on the user.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 101 +++++------------------------------\n t/t6042-merge-rename-corner-cases.sh | 2 +-\n 2 files changed, 14 insertions(+), 89 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex bae50258d7..d818830098 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -660,27 +660,6 @@ static int update_stages(struct merge_options *opt, const char *path,\n \treturn 0;\n }\n \n-static int update_stages_for_stage_data(struct merge_options *opt,\n-\t\t\t\t\tconst char *path,\n-\t\t\t\t\tconst struct stage_data *stage_data)\n-{\n-\tstruct diff_filespec o, a, b;\n-\n-\to.mode = stage_data->stages[1].mode;\n-\toidcpy(&o.oid, &stage_data->stages[1].oid);\n-\n-\ta.mode = stage_data->stages[2].mode;\n-\toidcpy(&a.oid, &stage_data->stages[2].oid);\n-\n-\tb.mode = stage_data->stages[3].mode;\n-\toidcpy(&b.oid, &stage_data->stages[3].oid);\n-\n-\treturn update_stages(opt, path,\n-\t\t\t is_null_sha1(o.oid.hash) ? NULL : &o,\n-\t\t\t is_null_sha1(a.oid.hash) ? NULL : &a,\n-\t\t\t is_null_sha1(b.oid.hash) ? NULL : &b);\n-}\n-\n static void update_entry(struct stage_data *entry,\n \t\t\t struct diff_filespec *o,\n \t\t\t struct diff_filespec *a,\n@@ -1618,7 +1597,6 @@ static int conflict_rename_rename_2to1(struct merge_options *o,\n \tchar *path = c1->path; /* == c2->path */\n \tstruct merge_file_info mfi_c1;\n \tstruct merge_file_info mfi_c2;\n-\tint ret;\n \n \toutput(o, 1, _(\"CONFLICT (rename/rename): \"\n \t \"Rename %s->%s in %s. \"\n@@ -1626,9 +1604,6 @@ static int conflict_rename_rename_2to1(struct merge_options *o,\n \t a->path, c1->path, ci->branch1,\n \t b->path, c2->path, ci->branch2);\n \n-\tremove_file(o, 1, a->path, o->call_depth || would_lose_untracked(a->path));\n-\tremove_file(o, 1, b->path, o->call_depth || would_lose_untracked(b->path));\n-\n \tif (merge_file_special_markers(o, a, c1, &ci->ren1_other,\n \t\t\t\t o->branch1, c1->path,\n \t\t\t\t o->branch2, ci->ren1_other.path, &mfi_c1) ||\n@@ -1637,66 +1612,11 @@ static int conflict_rename_rename_2to1(struct merge_options *o,\n \t\t\t\t o->branch2, c2->path, &mfi_c2))\n \t\treturn -1;\n \n-\tif (o->call_depth) {\n-\t\t/*\n-\t\t * If mfi_c1.clean && mfi_c2.clean, then it might make\n-\t\t * sense to do a two-way merge of those results. But, I\n-\t\t * think in all cases, it makes sense to have the virtual\n-\t\t * merge base just undo the renames; they can be detected\n-\t\t * again later for the non-recursive merge.\n-\t\t */\n-\t\tremove_file(o, 0, path, 0);\n-\t\tret = update_file(o, 0, &mfi_c1.oid, mfi_c1.mode, a->path);\n-\t\tif (!ret)\n-\t\t\tret = update_file(o, 0, &mfi_c2.oid, mfi_c2.mode,\n-\t\t\t\t\t b->path);\n-\t} else {\n-\t\tchar *new_path1 = unique_path(o, path, ci->branch1);\n-\t\tchar *new_path2 = unique_path(o, path, ci->branch2);\n-\t\toutput(o, 1, _(\"Renaming %s to %s and %s to %s instead\"),\n-\t\t a->path, new_path1, b->path, new_path2);\n-\t\tif (was_dirty(o, path))\n-\t\t\toutput(o, 1, _(\"Refusing to lose dirty file at %s\"),\n-\t\t\t path);\n-\t\telse if (would_lose_untracked(path))\n-\t\t\t/*\n-\t\t\t * Only way we get here is if both renames were from\n-\t\t\t * a directory rename AND user had an untracked file\n-\t\t\t * at the location where both files end up after the\n-\t\t\t * two directory renames. See testcase 10d of t6043.\n-\t\t\t */\n-\t\t\toutput(o, 1, _(\"Refusing to lose untracked file at \"\n-\t\t\t\t \"%s, even though it's in the way.\"),\n-\t\t\t path);\n-\t\telse\n-\t\t\tremove_file(o, 0, path, 0);\n-\t\tret = update_file(o, 0, &mfi_c1.oid, mfi_c1.mode, new_path1);\n-\t\tif (!ret)\n-\t\t\tret = update_file(o, 0, &mfi_c2.oid, mfi_c2.mode,\n-\t\t\t\t\t new_path2);\n-\t\t/*\n-\t\t * unpack_trees() actually populates the index for us for\n-\t\t * \"normal\" rename/rename(2to1) situtations so that the\n-\t\t * correct entries are at the higher stages, which would\n-\t\t * make the call below to update_stages_for_stage_data\n-\t\t * unnecessary. However, if either of the renames came\n-\t\t * from a directory rename, then unpack_trees() will not\n-\t\t * have gotten the right data loaded into the index, so we\n-\t\t * need to do so now. (While it'd be tempting to move this\n-\t\t * call to update_stages_for_stage_data() to\n-\t\t * apply_directory_rename_modifications(), that would break\n-\t\t * our intermediate calls to would_lose_untracked() since\n-\t\t * those rely on the current in-memory index. See also the\n-\t\t * big \"NOTE\" in update_stages()).\n-\t\t */\n-\t\tif (update_stages_for_stage_data(o, path, ci->dst_entry1))\n-\t\t\tret = -1;\n-\n-\t\tfree(new_path2);\n-\t\tfree(new_path1);\n-\t}\n-\n-\treturn ret;\n+\treturn handle_file_collision(o, path, a->path, b->path,\n+\t\t\t\t ci->branch1, ci->branch2,\n+\t\t\t\t &mfi_c1.oid, mfi_c1.mode,\n+\t\t\t\t &mfi_c2.oid, mfi_c2.mode,\n+\t\t\t\t !mfi_c1.clean || !mfi_c2.clean);\n }\n \n /*\n@@ -3057,9 +2977,14 @@ static int process_entry(struct merge_options *o,\n \t\t\t\tclean_merge = -1;\n \t\t\tbreak;\n \t\tcase RENAME_TWO_FILES_TO_ONE:\n-\t\t\tclean_merge = 0;\n-\t\t\tif (conflict_rename_rename_2to1(o, conflict_info))\n-\t\t\t\tclean_merge = -1;\n+\t\t\t/*\n+\t\t\t * Probably unclean merge, but if the two renamed\n+\t\t\t * files merge cleanly and the two resulting files\n+\t\t\t * can then be two-way merged cleanly, I guess it's\n+\t\t\t * a clean merge?\n+\t\t\t */\n+\t\t\tclean_merge = conflict_rename_rename_2to1(o,\n+\t\t\t\t\t\t\t\t conflict_info);\n \t\t\tbreak;\n \t\tdefault:\n \t\t\tentry->processed = 0;\ndiff --git a/t/t6042-merge-rename-corner-cases.sh b/t/t6042-merge-rename-corner-cases.sh\nindex d8fe797f0d..b0b840223b 100755\n--- a/t/t6042-merge-rename-corner-cases.sh\n+++ b/t/t6042-merge-rename-corner-cases.sh\n@@ -358,7 +358,7 @@ test_expect_success 'setup rename/rename (2to1) + modify/modify' '\n \tgit init &&\n \n \tprintf \"1\\n2\\n3\\n4\\n5\\n\" >a &&\n-\tprintf \"5\\n4\\n3\\n2\\n1\\n\" >b &&\n+\tprintf \"9\\n8\\n7\\n6\\n5\\n\" >b &&\n \tgit add a b &&\n \tgit commit -m A &&\n \tgit tag A &&\n-- \n2.15.0.323.g31fe956618\n\n"},{"id":"333007","messageId":"20171120221944.15431-5-newren@gmail.com","threadId":"47259","inReplyTo":"20171120221944.15431-1-newren@gmail.com","subject":"[RFC PATCH v2 4/9] Add testcases for improved file collision conflict handling","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-20T22:19:39Z","receivedAt":"2017-11-20T22:20:12Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Adds testcases dealing with file collisions for the following types of\nconflicts:\n * add/add\n * rename/add\n * rename/rename(2to1)\nThese tests include expectations for new, smarter behavior provided by\nhandle_file_collision(). Since that function is not in use yet, the\ntests are currently expected to fail.\n\nSigned-off-by: Elijah Newren \n---\n t/t6042-merge-rename-corner-cases.sh | 210 +++++++++++++++++++++++++++++++++++\n 1 file changed, 210 insertions(+)\n\ndiff --git a/t/t6042-merge-rename-corner-cases.sh b/t/t6042-merge-rename-corner-cases.sh\nindex 411550d2b6..d8fe797f0d 100755\n--- a/t/t6042-merge-rename-corner-cases.sh\n+++ b/t/t6042-merge-rename-corner-cases.sh\n@@ -575,4 +575,214 @@ test_expect_success 'rename/rename/add-dest merge still knows about conflicting\n \ttest ! -f c\n '\n \n+test_conflicts_with_adds_and_renames() {\n+\ttest $1 != 0 && side1=rename || side1=add\n+\ttest $2 != 0 && side2=rename || side2=add\n+\n+\t# Setup:\n+\t# L\n+\t# / \\\n+\t# master ?\n+\t# \\ /\n+\t# R\n+\t#\n+\t# Where:\n+\t# Both L and R have files named 'three-unrelated' and\n+\t# 'three-related' which collide. Each of the colliding files\n+\t# could have been involved in a rename, in which case there was a\n+\t# file named 'one-[un]related' or 'two-[un]related' that was\n+\t# modified on the opposite side of history and renamed into the\n+\t# collision on this side of history.\n+\t#\n+\t# Questions for both three-unrelated and three-related:\n+\t# 1) The index should contain both a stage 2 and stage 3 entry\n+\t# for the colliding file. Does it?\n+\t# 2) When renames are involved, the content merges are clean, so\n+\t# the index should reflect the content merges, not merely the\n+\t# version of the colliding file from the prior commit. Does\n+\t# it?\n+\t#\n+\t# Questions for three-unrelated:\n+\t# 3) There should be files in the worktree named\n+\t# 'three-unrelated~HEAD' and 'three-unrelated~R^0' with the\n+\t# (content-merged) version of 'three-unrelated' from the\n+\t# appropriate side of the merge. Are they present?\n+\t# 4) There should be no file named 'three-unrelated' in the\n+\t# working tree. That'd make it too likely that users would\n+\t# use it instead of carefully looking at both\n+\t# three-unrelated~HEAD and three-unrelated~R^0. Is it\n+\t# correctly missing?\n+\t#\n+\t# Questions for three-related:\n+\t# 3) There should be a file in the worktree named three-related\n+\t# containing the two-way merged contents of the content-merged\n+\t# versions of three-related from each of the two colliding\n+\t# files. Is it present?\n+\t# 4) There should not be any three-related~* files in the working\n+\t# tree.\n+\ttest_expect_success \"setup simple $side1/$side2 conflict\" '\n+\t\tgit rm -rf . &&\n+\t\tgit clean -fdqx &&\n+\t\trm -rf .git &&\n+\t\tgit init &&\n+\n+\t\t# Create a simple file with 10 lines\n+\t\tten=\"0 1 2 3 4 5 6 7 8 9\" &&\n+\t\tfor i in $ten\n+\t\tdo\n+\t\t\techo line $i in a sample file\n+\t\tdone >unrelated1_v1 &&\n+\t\t# Create a second version of same file with one more line\n+\t\tcat unrelated1_v1 >unrelated1_v2 &&\n+\t\techo another line >>unrelated1_v2 &&\n+\n+\t\t# Create an unrelated simple file with 10 lines\n+\t\tfor i in $ten\n+\t\tdo\n+\t\t\techo line $i in another sample file\n+\t\tdone >unrelated2_v1 &&\n+\t\t# Create a second version of same file with one more line\n+\t\tcat unrelated2_v1 >unrelated2_v2 &&\n+\t\techo another line >>unrelated2_v2 &&\n+\n+\t\t# Create some related files now\n+\t\tfor i in $ten\n+\t\tdo\n+\t\t\techo Random base content line $i\n+\t\tdone >related1_v1 &&\n+\t\tcp -a related1_v1 related1_v2 &&\n+\t\techo modification >>related1_v2 &&\n+\n+\t\tcp -a related1_v1 related2_v1 &&\n+\t\techo more stuff >>related2_v1 &&\n+\t\tcp -a related2_v1 related2_v2 &&\n+\t\techo yet more stuff >>related2_v2 &&\n+\n+\t\t# Use a tag to record both these files for simple access,\n+\t\t# and clean out these untracked files\n+\t\tgit tag unrelated1_v1 `git hash-object -w unrelated1_v1` &&\n+\t\tgit tag unrelated1_v2 `git hash-object -w unrelated1_v2` &&\n+\t\tgit tag unrelated2_v1 `git hash-object -w unrelated2_v1` &&\n+\t\tgit tag unrelated2_v2 `git hash-object -w unrelated2_v2` &&\n+\t\tgit tag related1_v1 `git hash-object -w related1_v1` &&\n+\t\tgit tag related1_v2 `git hash-object -w related1_v2` &&\n+\t\tgit tag related2_v1 `git hash-object -w related2_v1` &&\n+\t\tgit tag related2_v2 `git hash-object -w related2_v2` &&\n+\t\tgit clean -f &&\n+\n+\t\t# Setup merge-base, consisting of files named \"one-*\"\n+\t\t# and \"two-*\" if renames were involved.\n+\t\ttouch irrelevant_file &&\n+\t\tgit add irrelevant_file &&\n+\t\tif [ $side1 == \"rename\" ]; then\n+\t\t\tgit show unrelated1_v1 >one-unrelated &&\n+\t\t\tgit add one-unrelated\n+\t\t\tgit show related1_v1 >one-related &&\n+\t\t\tgit add one-related\n+\t\tfi &&\n+\t\tif [ $side2 == \"rename\" ]; then\n+\t\t\tgit show unrelated2_v1 >two-unrelated &&\n+\t\t\tgit add two-unrelated\n+\t\t\tgit show related2_v1 >two-related &&\n+\t\t\tgit add two-related\n+\t\tfi &&\n+\t\ttest_tick && git commit -m initial &&\n+\n+\t\tgit branch L &&\n+\t\tgit branch R &&\n+\n+\t\t# Handle the left side\n+\t\tgit checkout L &&\n+\t\tif [ $side1 == \"rename\" ]; then\n+\t\t\tgit mv one-unrelated three-unrelated\n+\t\t\tgit mv one-related three-related\n+\t\telse\n+\t\t\tgit show unrelated1_v2 >three-unrelated &&\n+\t\t\tgit add three-unrelated\n+\t\t\tgit show related1_v2 >three-related &&\n+\t\t\tgit add three-related\n+\t\tfi &&\n+\t\tif [ $side2 == \"rename\" ]; then\n+\t\t\tgit show unrelated2_v2 >two-unrelated &&\n+\t\t\tgit add two-unrelated\n+\t\t\tgit show related2_v2 >two-related &&\n+\t\t\tgit add two-related\n+\t\tfi &&\n+\t\ttest_tick && git commit -m L &&\n+\n+\t\t# Handle the right side\n+\t\tgit checkout R &&\n+\t\tif [ $side1 == \"rename\" ]; then\n+\t\t\tgit show unrelated1_v2 >one-unrelated &&\n+\t\t\tgit add one-unrelated\n+\t\t\tgit show related1_v2 >one-related &&\n+\t\t\tgit add one-related\n+\t\tfi &&\n+\t\tif [ $side2 == \"rename\" ]; then\n+\t\t\tgit mv two-unrelated three-unrelated\n+\t\t\tgit mv two-related three-related\n+\t\telse\n+\t\t\tgit show unrelated2_v2 >three-unrelated &&\n+\t\t\tgit add three-unrelated\n+\t\t\tgit show related2_v2 >three-related &&\n+\t\t\tgit add three-related\n+\t\tfi &&\n+\t\ttest_tick && git commit -m R\n+\t'\n+\n+\ttest_expect_failure \"check simple $side1/$side2 conflict\" '\n+\t\tgit reset --hard &&\n+\t\tgit checkout L^0 &&\n+\n+\t\t# Merge must fail; there is a conflict\n+\t\ttest_must_fail git merge -s recursive R^0 &&\n+\n+\t\t# Make sure the index has the right number of entries\n+\t\ttest 5 = $(git ls-files -s | wc -l) &&\n+\t\ttest 4 = $(git ls-files -u | wc -l) &&\n+\n+\t\t# Nothing should have touched irrelevant_file\n+\t\ttest $(git rev-parse :0:irrelevant_file) = $(git rev-parse master:irrelevant_file) &&\n+\n+\t\t# Even for renames, make sure the index contains the MERGED\n+\t\t# version of the file, not the version of the file that existed\n+\t\t# on the given side.\n+\t\ttest $(git rev-parse :2:three-unrelated) = $(git rev-parse unrelated1_v2) &&\n+\t\ttest $(git rev-parse :3:three-unrelated) = $(git rev-parse unrelated2_v2) &&\n+\t\ttest $(git rev-parse :2:three-related) = $(git rev-parse related1_v2) &&\n+\t\ttest $(git rev-parse :3:three-related) = $(git rev-parse related2_v2) &&\n+\n+\t\t# Make sure we have the correct number of untracked files\n+\t\ttest 2 = $(git ls-files -o | wc -l) &&\n+\n+\t\t# Make sure each file (with merging if rename involved) is\n+\t\t# present in the working tree for the user to work with.\n+\t\ttest $(git hash-object three-unrelated~HEAD) = $(git rev-parse unrelated1_v2) &&\n+\t\ttest $(git hash-object three-unrelated~R^0) = $(git rev-parse unrelated2_v2) &&\n+\n+\t\t# \"three-unrelated\" should not exist because there is no\n+\t\t# reason to give preference to either three-unrelated~HEAD\n+\t\t# or three-unrelated~R^0\n+\t\ttest ! -f three-unrelated &&\n+\n+\t\t# Make sure we have the correct merged contents for three-related\n+\t\tgit show related1_v1 >expected &&\n+\t\tcat <>expected &&\n+<<<<<<< HEAD\n+modification\n+=======\n+more stuff\n+yet more stuff\n+>>>>>>> R^0\n+EOF\n+\n+\t\ttest_cmp expected three-related\n+\t'\n+}\n+\n+test_conflicts_with_adds_and_renames 1 1\n+test_conflicts_with_adds_and_renames 1 0\n+test_conflicts_with_adds_and_renames 0 1\n+test_conflicts_with_adds_and_renames 0 0\n+\n test_done\n-- \n2.15.0.323.g31fe956618\n\n"},{"id":"333008","messageId":"20171120221944.15431-10-newren@gmail.com","threadId":"47259","inReplyTo":"20171120221944.15431-1-newren@gmail.com","subject":"[RFC PATCH v2 9/9] diffcore-rename: filter rename_src list when possible","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-20T22:19:44Z","receivedAt":"2017-11-20T22:20:14Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"We have to look at each entry in rename_src a total of rename_dst_nr\ntimes. When we're not detecting copies, any exact renames or ignorable\nrename paths will just be skipped over. While checking that these can\nbe skipped over is a relatively cheap check, it's still a waste of time\nto do that check more than once, let alone rename_dst_nr times. When\nrename_src_nr is a few thousand times bigger than the number of relevant\nsources (such as when cherry-picking a commit that only touched a\nhandful of files, but from a side of history that has different names\nfor some high level directories), this time can add up.\n\nFirst make an initial pass over the rename_src array and move all the\nrelevant entries to the front, so that we can iterate over just those\nrelevant entries.\n\nIn one particular testcase involving a large repository and some\nhigh-level directories having been renamed, this cut the time necessary\nfor a cherry-pick down by a factor of about 2 (from around 34 seconds\ndown to just under 16 seconds)\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 47 +++++++++++++++++++++++++++++++----------------\n 1 file changed, 31 insertions(+), 16 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 5bf5bf7379..e60abb5980 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -437,16 +437,14 @@ static int find_renames(struct diff_score *mx, int dst_cnt, int minimum_score, i\n \treturn count;\n }\n \n-static int handle_rename_ignores(struct diff_options *options)\n+static void handle_rename_ignores(struct diff_options *options)\n {\n-\tint detect_rename = options->detect_rename;\n \tstruct string_list *ignores = options->ignore_for_renames;\n-\tint ignored = 0;\n \tint i, j;\n \n \t/* rename_ignores onlhy relevant when we're not detecting copies */\n-\tif (ignores == NULL || detect_rename == DIFF_DETECT_COPY)\n-\t\treturn 0;\n+\tif (ignores == NULL)\n+\t\treturn;\n \n \tfor (i = 0, j = 0; i < ignores->nr && j < rename_src_nr;) {\n \t\tstruct diff_filespec *one = rename_src[j].p->one;\n@@ -464,11 +462,27 @@ static int handle_rename_ignores(struct diff_options *options)\n \t\t\tj++;\n \t\telse {\n \t\t\tone->rename_used++;\n-\t\t\tignored++;\n+\t\t\ti++;\n+\t\t\tj++;\n \t\t}\n \t}\n+}\n+\n+static int remove_renames_from_src(void)\n+{\n+\tint j, new_j;\n+\n+\tfor (j = 0, new_j = 0; j < rename_src_nr; j++) {\n+\t\tif (rename_src[j].p->one->rename_used)\n+\t\t\tcontinue;\n+\n+\t\tif (new_j < j)\n+\t\t\tmemcpy(&rename_src[new_j], &rename_src[j],\n+\t\t\t sizeof(struct diff_rename_src));\n+\t\tnew_j++;\n+\t}\n \n-\treturn ignored;\n+\treturn new_j;\n }\n \n void diffcore_rename(struct diff_options *options)\n@@ -479,7 +493,7 @@ void diffcore_rename(struct diff_options *options)\n \tstruct diff_queue_struct outq;\n \tstruct diff_score *mx;\n \tint i, j, rename_count, skip_unmodified = 0;\n-\tint num_create, dst_cnt, num_src, ignore_count;\n+\tint num_create, dst_cnt, num_src;\n \tstruct progress *progress = NULL;\n \n \tif (!minimum_score)\n@@ -542,18 +556,19 @@ void diffcore_rename(struct diff_options *options)\n \n \t/*\n \t * Mark source files as used if they are found in the\n-\t * ignore_for_renames list.\n+\t * ignore_for_renames list, and clean out files from rename_src\n+\t * that we don't need to continue considering.\n \t */\n-\tignore_count = handle_rename_ignores(options);\n+\tnum_src = rename_src_nr;\n+\tif (detect_rename != DIFF_DETECT_COPY) {\n+\t\thandle_rename_ignores(options);\n+\t\tnum_src = remove_renames_from_src();\n+\t}\n \n \t/*\n-\t * Calculate how many renames are left (but all the source\n-\t * files still remain as options for rename/copies!)\n+\t * Calculate how many renames are left\n \t */\n \tnum_create = (rename_dst_nr - rename_count);\n-\tnum_src = (detect_rename == DIFF_DETECT_COPY ?\n-\t\t rename_src_nr : rename_src_nr - rename_count);\n-\tnum_src -= ignore_count;\n \n \t/* All done? */\n \tif (!num_create)\n@@ -588,7 +603,7 @@ void diffcore_rename(struct diff_options *options)\n \t\tfor (j = 0; j < NUM_CANDIDATE_PER_DST; j++)\n \t\t\tm[j].dst = -1;\n \n-\t\tfor (j = 0; j < rename_src_nr; j++) {\n+\t\tfor (j = 0; j < num_src; j++) {\n \t\t\tstruct diff_filespec *one = rename_src[j].p->one;\n \t\t\tstruct diff_score this_src;\n \n-- \n2.15.0.323.g31fe956618\n\n"},{"id":"333009","messageId":"20171120221944.15431-6-newren@gmail.com","threadId":"47259","inReplyTo":"20171120221944.15431-1-newren@gmail.com","subject":"[RFC PATCH v2 5/9] merge-recursive: fix rename/add conflict handling","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-20T22:19:40Z","receivedAt":"2017-11-20T22:20:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This makes the rename/add conflict handling make use of the new\nhandle_file_collision() function, which fixes several bugs and improves\nthings for the rename/add case significantly. Previously, rename/add\nwould:\n\n * Not leave any higher order stage entries in the index, making it\n appear as if there were no conflict.\n * Would place the rename file at the colliding path, and move the\n added file elsewhere, which combined with the lack of higher order\n stage entries felt really odd. It's not clear to me why the\n rename should take precedence over the add; if one should be moved\n out of the way, they both probably should.\n * In the recursive case, it would do a two way merge of the added\n file and the version of the renamed file on the renamed side,\n completely excluding modifications to the renamed file on the\n unrenamed side of history.\n\nUsing the new handle_file_collision() fixes all of these issues, and\nadds smarts to allow two-way merge OR move colliding files to separate\npaths depending on the similarity of the colliding files.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 135 +++++++++++++++++++++++++++-----------\n t/t6036-recursive-corner-cases.sh | 19 +++---\n 2 files changed, 108 insertions(+), 46 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 9dda13bf4f..bae50258d7 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -178,6 +178,7 @@ static int oid_eq(const struct object_id *a, const struct object_id *b)\n enum rename_type {\n \tRENAME_NORMAL = 0,\n \tRENAME_DIR,\n+\tRENAME_ADD,\n \tRENAME_DELETE,\n \tRENAME_ONE_FILE_TO_ONE,\n \tRENAME_ONE_FILE_TO_TWO,\n@@ -220,6 +221,7 @@ static inline void setup_rename_conflict_info(enum rename_type rename_type,\n \t\t\t\t\t struct stage_data *src_entry1,\n \t\t\t\t\t struct stage_data *src_entry2)\n {\n+\tint ostage1, ostage2;\n \tstruct rename_conflict_info *ci = xcalloc(1, sizeof(struct rename_conflict_info));\n \tci->rename_type = rename_type;\n \tci->pair1 = pair1;\n@@ -237,18 +239,22 @@ static inline void setup_rename_conflict_info(enum rename_type rename_type,\n \t\tdst_entry2->rename_conflict_info = ci;\n \t}\n \n-\tif (rename_type == RENAME_TWO_FILES_TO_ONE) {\n-\t\t/*\n-\t\t * For each rename, there could have been\n-\t\t * modifications on the side of history where that\n-\t\t * file was not renamed.\n-\t\t */\n-\t\tint ostage1 = o->branch1 == branch1 ? 3 : 2;\n-\t\tint ostage2 = ostage1 ^ 1;\n+\t/*\n+\t * For each rename, there could have been\n+\t * modifications on the side of history where that\n+\t * file was not renamed.\n+\t */\n+\tif (rename_type == RENAME_ADD ||\n+\t rename_type == RENAME_TWO_FILES_TO_ONE) {\n+\t\tostage1 = o->branch1 == branch1 ? 3 : 2;\n \n \t\tci->ren1_other.path = pair1->one->path;\n \t\toidcpy(&ci->ren1_other.oid, &src_entry1->stages[ostage1].oid);\n \t\tci->ren1_other.mode = src_entry1->stages[ostage1].mode;\n+\t}\n+\n+\tif (rename_type == RENAME_TWO_FILES_TO_ONE) {\n+\t\tostage2 = ostage1 ^ 1;\n \n \t\tci->ren2_other.path = pair2->one->path;\n \t\toidcpy(&ci->ren2_other.oid, &src_entry2->stages[ostage2].oid);\n@@ -1118,6 +1124,18 @@ static int merge_file_special_markers(struct merge_options *o,\n \t\t\t const char *filename2,\n \t\t\t struct merge_file_info *mfi)\n {\n+\tif (o->branch1 != branch1) {\n+\t\t/*\n+\t\t * It's weird getting a reverse merge with HEAD on the bottom\n+\t\t * and the other branch on the top. Fix that.\n+\t\t */\n+\t\treturn merge_file_special_markers(o,\n+\t\t\t\t\t\t one, b, a,\n+\t\t\t\t\t\t branch2, filename2,\n+\t\t\t\t\t\t branch1, filename1,\n+\t\t\t\t\t\t mfi);\n+\t}\n+\n \tchar *side1 = NULL;\n \tchar *side2 = NULL;\n \tint ret;\n@@ -1308,6 +1326,21 @@ static int handle_file_collision(struct merge_options *o,\n \tstruct diff_filespec null, a, b;\n \tint minimum_score;\n \n+\t/*\n+\t * It's easiest to get the correct things into stage 2 and 3, and\n+\t * to make sure that the content merge puts HEAD before the other\n+\t * branch if we just ensure that branch1 == o->branch1. So, simply\n+\t * flip arguments around if we don't have that.\n+\t */\n+\tif (branch1 != o->branch1) {\n+\t\treturn handle_file_collision(o, collide_path,\n+\t\t\t\t\t prev_path2, prev_path1,\n+\t\t\t\t\t branch2, branch1,\n+\t\t\t\t\t b_oid, b_mode,\n+\t\t\t\t\t a_oid, a_mode,\n+\t\t\t\t\t conflict_markers_already_present);\n+\t}\n+\n \t/* Remove rename sources if rename/add or rename/rename(2to1) */\n \tif (prev_path1)\n \t\tremove_file(o, 1, prev_path1,\n@@ -1414,6 +1447,36 @@ static int handle_file_collision(struct merge_options *o,\n \treturn 0; /* not clean */\n }\n \n+static int conflict_rename_add(struct merge_options *o,\n+\t\t\t struct rename_conflict_info *ci)\n+{\n+\t/* a was renamed to c, and a separate c was added. */\n+\tstruct diff_filespec *a = ci->pair1->one;\n+\tstruct diff_filespec *c = ci->pair1->two;\n+\tchar *path = c->path;\n+\tstruct merge_file_info mfi;\n+\n+\tint other_stage = (ci->branch1 == o->branch1 ? 3 : 2);\n+\n+\toutput(o, 1, _(\"CONFLICT (rename/add): \"\n+\t \"Rename %s->%s in %s. Added %s in %s\"),\n+\t a->path, c->path, ci->branch1,\n+\t c->path, ci->branch2);\n+\n+\tif (merge_file_special_markers(o, a, c, &ci->ren1_other,\n+\t\t\t\t o->branch1, path,\n+\t\t\t\t o->branch2, ci->ren1_other.path, &mfi))\n+\t\treturn -1;\n+\n+\treturn handle_file_collision(o,\n+\t\t\t\t c->path, a->path, NULL,\n+\t\t\t\t ci->branch1, ci->branch2,\n+\t\t\t\t &mfi.oid, mfi.mode,\n+\t\t\t\t &ci->dst_entry1->stages[other_stage].oid,\n+\t\t\t\t ci->dst_entry1->stages[other_stage].mode,\n+\t\t\t\t !mfi.clean);\n+}\n+\n static int handle_file(struct merge_options *o,\n \t\t\tstruct diff_filespec *rename,\n \t\t\tint stage,\n@@ -2556,36 +2619,23 @@ static int process_renames(struct merge_options *o,\n \t\t\t\t\t\t 0 /* update_wd */))\n \t\t\t\t\tclean_merge = -1;\n \t\t\t} else if (!oid_eq(&dst_other.oid, &null_oid)) {\n-\t\t\t\tclean_merge = 0;\n-\t\t\t\ttry_merge = 1;\n-\t\t\t\toutput(o, 1, _(\"CONFLICT (rename/add): Rename %s->%s in %s. \"\n-\t\t\t\t \"%s added in %s\"),\n-\t\t\t\t ren1_src, ren1_dst, branch1,\n-\t\t\t\t ren1_dst, branch2);\n-\t\t\t\tif (o->call_depth) {\n-\t\t\t\t\tstruct merge_file_info mfi;\n-\t\t\t\t\tif (merge_file_one(o, ren1_dst, &null_oid, 0,\n-\t\t\t\t\t\t\t &ren1->pair->two->oid,\n-\t\t\t\t\t\t\t ren1->pair->two->mode,\n-\t\t\t\t\t\t\t &dst_other.oid,\n-\t\t\t\t\t\t\t dst_other.mode,\n-\t\t\t\t\t\t\t branch1, branch2, &mfi)) {\n-\t\t\t\t\t\tclean_merge = -1;\n-\t\t\t\t\t\tgoto cleanup_and_return;\n-\t\t\t\t\t}\n-\t\t\t\t\toutput(o, 1, _(\"Adding merged %s\"), ren1_dst);\n-\t\t\t\t\tif (update_file(o, 0, &mfi.oid,\n-\t\t\t\t\t\t\tmfi.mode, ren1_dst))\n-\t\t\t\t\t\tclean_merge = -1;\n-\t\t\t\t\ttry_merge = 0;\n-\t\t\t\t} else {\n-\t\t\t\t\tchar *new_path = unique_path(o, ren1_dst, branch2);\n-\t\t\t\t\toutput(o, 1, _(\"Adding as %s instead\"), new_path);\n-\t\t\t\t\tif (update_file(o, 0, &dst_other.oid,\n-\t\t\t\t\t\t\tdst_other.mode, new_path))\n-\t\t\t\t\t\tclean_merge = -1;\n-\t\t\t\t\tfree(new_path);\n-\t\t\t\t}\n+\t\t\t\t/*\n+\t\t\t\t * Probably not a clean merge, but it's\n+\t\t\t\t * premature to set clean_merge to 0 here,\n+\t\t\t\t * because if the rename merges cleanly and\n+\t\t\t\t * the merge exactly matches the newly added\n+\t\t\t\t * file, then the merge will be clean.\n+\t\t\t\t */\n+\t\t\t\tsetup_rename_conflict_info(RENAME_ADD,\n+\t\t\t\t\t\t\t ren1->pair,\n+\t\t\t\t\t\t\t NULL,\n+\t\t\t\t\t\t\t branch1,\n+\t\t\t\t\t\t\t branch2,\n+\t\t\t\t\t\t\t ren1->dst_entry,\n+\t\t\t\t\t\t\t NULL,\n+\t\t\t\t\t\t\t o,\n+\t\t\t\t\t\t\t ren1->src_entry,\n+\t\t\t\t\t\t\t NULL);\n \t\t\t} else\n \t\t\t\ttry_merge = 1;\n \n@@ -2984,6 +3034,15 @@ static int process_entry(struct merge_options *o,\n \t\t\t\t\t\tconflict_info->branch2))\n \t\t\t\tclean_merge = -1;\n \t\t\tbreak;\n+\t\tcase RENAME_ADD:\n+\t\t\t/*\n+\t\t\t * Probably unclean merge, but if the renamed file\n+\t\t\t * merges cleanly and the result can then be\n+\t\t\t * two-way merged cleanly with the added file, I\n+\t\t\t * guess it's a clean merge?\n+\t\t\t */\n+\t\t\tclean_merge = conflict_rename_add(o, conflict_info);\n+\t\t\tbreak;\n \t\tcase RENAME_DELETE:\n \t\t\tclean_merge = 0;\n \t\t\tif (conflict_rename_delete(o,\ndiff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh\nindex 18aa88b5c0..09accbc62a 100755\n--- a/t/t6036-recursive-corner-cases.sh\n+++ b/t/t6036-recursive-corner-cases.sh\n@@ -168,7 +168,7 @@ test_expect_success 'setup differently handled merges of rename/add conflict' '\n \tgit branch B &&\n \tgit checkout -b C &&\n \techo 10 >>a &&\n-\techo \"other content\" >>new_a &&\n+\tprintf \"0\\n1\\n2\\n3\\n4\\n5\\n6\\n7\\nfoobar\" >new_a &&\n \tgit add a new_a &&\n \ttest_tick && git commit -m C &&\n \n@@ -179,13 +179,13 @@ test_expect_success 'setup differently handled merges of rename/add conflict' '\n \tgit checkout B^0 &&\n \ttest_must_fail git merge C &&\n \tgit clean -f &&\n+\tgit add new_a &&\n \ttest_tick && git commit -m D &&\n \tgit tag D &&\n \n \tgit checkout C^0 &&\n \ttest_must_fail git merge B &&\n-\trm new_a~HEAD new_a &&\n-\tprintf \"Incorrectly merged content\" >>new_a &&\n+\tprintf \"0\\n1\\n2\\n3\\n4\\n5\\n6\\n7\\nbad merge\" >new_a &&\n \tgit add -u &&\n \ttest_tick && git commit -m E &&\n \tgit tag E\n@@ -204,15 +204,18 @@ test_expect_success 'git detects differently handled merges conflict' '\n \ttest $(git rev-parse :2:new_a) = $(git rev-parse D:new_a) &&\n \ttest $(git rev-parse :3:new_a) = $(git rev-parse E:new_a) &&\n \n-\tgit cat-file -p B:new_a >>merged &&\n+\t# Since A:a == B:new_a, the three-way merge of A:a, B:new_a, and\n+\t# C:a is just C:a. Then we do a two-way merge of that with\n+\t# C:new_a.\n+\tgit cat-file -p C:a >>merged &&\n \tgit cat-file -p C:new_a >>merge-me &&\n \t>empty &&\n \ttest_must_fail git merge-file \\\n-\t\t-L \"Temporary merge branch 2\" \\\n-\t\t-L \"\" \\\n \t\t-L \"Temporary merge branch 1\" \\\n-\t\tmerged empty merge-me &&\n-\tsed -e \"s/^\\([<=>]\\)/\\1\\1\\1/\" merged >merged-internal &&\n+\t\t-L \"\" \\\n+\t\t-L \"Temporary merge branch 2\" \\\n+\t\tmerge-me empty merged &&\n+\tsed -e \"s/^\\([<=>]\\)/\\1\\1\\1/\" merge-me >merged-internal &&\n \ttest $(git rev-parse :1:new_a) = $(git hash-object merged-internal)\n '\n \n-- \n2.15.0.323.g31fe956618\n\n"},{"id":"333010","messageId":"20171120221944.15431-9-newren@gmail.com","threadId":"47259","inReplyTo":"20171120221944.15431-1-newren@gmail.com","subject":"[RFC PATCH v2 8/9] merge-recursive: accelerate rename detection","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-20T22:19:43Z","receivedAt":"2017-11-20T22:20:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"If a file is unmodified on one side of history (no content changes, no\nname change, and no mode change) and is renamed on the other side, then\nthe correct merge result is to take both the file name and the file\ncontents (and file mode) of the renamed file. merge-recursive detects\nthis rename and gets the correct merge result.\n\nNote that if no rename detection is done, then this will appear to the\nmerge machinery as two files: one that was unmodified on one side of\nhistory and deleted on the other (thus the merge should delete it), and\none which was newly added on one side of history (thus the merge should\ninclude it). Thus, even if the rename wasn't detected, we still would\nhave ended up with the correct result.\n\nIn other words, rename detection is a waste of time for files that were\nunmodified on the OTHER side of history. We can accelerate rename\ndetection for merges by providing information about the other side of\nhistory, which will allow us to remove all such rename sources from the\nlist of candidates we care about.\n\nThere are two gotchas:\n\n 1) Not trying to detect renames for these types of files can result in\n rename/add conflicts being instead detected as add/add conflicts,\n and can result in rename/rename(2to1) conflicts being instead\n detected as either rename/add or add/add conflicts. Luckily for\n us, these three types of conflicts happen to make the same changes\n to the index and working tree (what a coincidence...), so this\n isn't a significant issue; the only annoyance is that the stdout\n from the merge command will include a \"CONFLICT($type)\" message for\n a related conflict type instead of the precise conflict type.\n\n 2) If there is a directory rename on one side of history AND all files\n within the directory are not merely renamed but are modified as\n well AND none of the original files in the directory are modified\n on the other side of history AND there are new files added (or\n moved into) to the original directory on that other side of\n history, then this change will prevent us from being able to detect\n that directory rename and placing the new file(s) into the\n appropriate directory. A subsequent commit will correct this\n downside.\n\nIn one particular testcase involving a large repository and some\nhigh-level directories having been renamed, this cut the time necessary\nfor a cherry-pick down by a factor of about 8 (from around 4.5 minutes\ndown to around 34 seconds)\n\nSigned-off-by: Elijah Newren \n---\n diff.c | 1 +\n diff.h | 3 +++\n diffcore-rename.c | 43 +++++++++++++++++++++++++++++++++++++++++-\n merge-recursive.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++----\n 4 files changed, 98 insertions(+), 5 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 40054070bd..b4717df4a4 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -4102,6 +4102,7 @@ void diff_setup(struct diff_options *options)\n \t}\n \n \toptions->color_moved = diff_color_moved_default;\n+\toptions->ignore_for_renames = NULL;\n }\n \n void diff_setup_done(struct diff_options *options)\ndiff --git a/diff.h b/diff.h\nindex 34dbc3cc05..4d7318c12c 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -206,6 +206,9 @@ struct diff_options {\n \t} color_moved;\n \t#define COLOR_MOVED_DEFAULT COLOR_MOVED_ZEBRA\n \t#define COLOR_MOVED_MIN_ALNUM_COUNT 20\n+\n+\t/* Paths we should ignore for rename purposes */\n+\tstruct string_list *ignore_for_renames;\n };\n \n void diff_emit_submodule_del(struct diff_options *o, const char *line);\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 4fe5d0471c..5bf5bf7379 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -437,6 +437,40 @@ static int find_renames(struct diff_score *mx, int dst_cnt, int minimum_score, i\n \treturn count;\n }\n \n+static int handle_rename_ignores(struct diff_options *options)\n+{\n+\tint detect_rename = options->detect_rename;\n+\tstruct string_list *ignores = options->ignore_for_renames;\n+\tint ignored = 0;\n+\tint i, j;\n+\n+\t/* rename_ignores onlhy relevant when we're not detecting copies */\n+\tif (ignores == NULL || detect_rename == DIFF_DETECT_COPY)\n+\t\treturn 0;\n+\n+\tfor (i = 0, j = 0; i < ignores->nr && j < rename_src_nr;) {\n+\t\tstruct diff_filespec *one = rename_src[j].p->one;\n+\t\tint cmp;\n+\n+\t\tif (one->rename_used) {\n+\t\t\tj++;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tcmp = strcmp(ignores->items[i].string, one->path);\n+\t\tif (cmp < 0)\n+\t\t\ti++;\n+\t\telse if (cmp > 0)\n+\t\t\tj++;\n+\t\telse {\n+\t\t\tone->rename_used++;\n+\t\t\tignored++;\n+\t\t}\n+\t}\n+\n+\treturn ignored;\n+}\n+\n void diffcore_rename(struct diff_options *options)\n {\n \tint detect_rename = options->detect_rename;\n@@ -445,7 +479,7 @@ void diffcore_rename(struct diff_options *options)\n \tstruct diff_queue_struct outq;\n \tstruct diff_score *mx;\n \tint i, j, rename_count, skip_unmodified = 0;\n-\tint num_create, dst_cnt, num_src;\n+\tint num_create, dst_cnt, num_src, ignore_count;\n \tstruct progress *progress = NULL;\n \n \tif (!minimum_score)\n@@ -506,6 +540,12 @@ void diffcore_rename(struct diff_options *options)\n \tif (minimum_score == MAX_SCORE)\n \t\tgoto cleanup;\n \n+\t/*\n+\t * Mark source files as used if they are found in the\n+\t * ignore_for_renames list.\n+\t */\n+\tignore_count = handle_rename_ignores(options);\n+\n \t/*\n \t * Calculate how many renames are left (but all the source\n \t * files still remain as options for rename/copies!)\n@@ -513,6 +553,7 @@ void diffcore_rename(struct diff_options *options)\n \tnum_create = (rename_dst_nr - rename_count);\n \tnum_src = (detect_rename == DIFF_DETECT_COPY ?\n \t\t rename_src_nr : rename_src_nr - rename_count);\n+\tnum_src -= ignore_count;\n \n \t/* All done? */\n \tif (!num_create)\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex e2d3b0fb4a..231d2d6a66 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -503,6 +503,48 @@ static struct string_list *get_unmerged(void)\n \treturn unmerged;\n }\n \n+static void get_rename_ignore(struct string_list *unmerged,\n+\t\t\t struct string_list *rename_ignore)\n+{\n+\t/*\n+\t * If a file is unmodified on one side of history (no content\n+\t * changes, no mode change, and no name change) and is renamed on\n+\t * the other side, then the correct merge result is to take both\n+\t * the file name and the file contents (and file mode) of the\n+\t * renamed file. merge-recursive detects this rename and gets the\n+\t * correct merge result.\n+\t *\n+\t * Note that if no rename detection is done, then this will appear\n+\t * to the merge machinery as two files: one that was unmodified on\n+\t * one side of history and deleted on the other (thus the merge\n+\t * should delete it), and one which was newly added on one side of\n+\t * history (thus the merge should include it). Thus, even if the\n+\t * rename wasn't detected, we still would have ended up with the\n+\t * correct result.\n+\t *\n+\t * In other words, rename detection is a waste of time for files\n+\t * that were unmodified on the OTHER side of history. We can\n+\t * accelerate rename detection for merges by find these sets of\n+\t * unmodified files, and feeding them to get_renames so it can\n+\t * omit using those files as rename sources.\n+\t */\n+\tint i;\n+\n+\tfor (i = 0; i < unmerged->nr; i++) {\n+\t\tconst char *path = unmerged->items[i].string;\n+\t\tstruct stage_data *e = unmerged->items[i].util;\n+\t\tunsigned int ign_head = is_null_oid(&e->stages[2].oid) &&\n+\t\t oid_eq(&e->stages[1].oid, &e->stages[3].oid) &&\n+\t\t e->stages[1].mode == e->stages[3].mode;\n+\t\tunsigned int ign_merge = is_null_oid(&e->stages[3].oid) &&\n+\t\t oid_eq(&e->stages[1].oid, &e->stages[2].oid) &&\n+\t\t e->stages[1].mode == e->stages[2].mode;\n+\t\tif (ign_head || ign_merge)\n+\t\t\tstring_list_append(rename_ignore, path);\n+\t}\n+}\n+\n+\n static int string_list_df_name_compare(const char *one, const char *two)\n {\n \tint onelen = strlen(one);\n@@ -1624,7 +1666,8 @@ static int conflict_rename_rename_2to1(struct merge_options *o,\n */\n static struct diff_queue_struct *get_diffpairs(struct merge_options *o,\n \t\t\t\t\t struct tree *o_tree,\n-\t\t\t\t\t struct tree *tree)\n+\t\t\t\t\t struct tree *tree,\n+\t\t\t\t\t struct string_list *rename_ignore)\n {\n \tstruct diff_queue_struct *ret;\n \tstruct diff_options opts;\n@@ -1633,6 +1676,7 @@ static struct diff_queue_struct *get_diffpairs(struct merge_options *o,\n \topts.flags.recursive = 1;\n \topts.flags.rename_empty = 0;\n \topts.detect_rename = DIFF_DETECT_RENAME;\n+\topts.ignore_for_renames = rename_ignore;\n \topts.rename_limit = o->merge_rename_limit >= 0 ? o->merge_rename_limit :\n \t\t\t o->diff_rename_limit >= 0 ? o->diff_rename_limit :\n \t\t\t 1000;\n@@ -2604,6 +2648,7 @@ static int handle_renames(struct merge_options *o,\n \t\t\t struct tree *head,\n \t\t\t struct tree *merge,\n \t\t\t struct string_list *entries,\n+\t\t\t struct string_list *rename_ignore,\n \t\t\t struct rename_info *ri)\n {\n \tstruct diff_queue_struct *head_pairs, *merge_pairs;\n@@ -2618,8 +2663,8 @@ static int handle_renames(struct merge_options *o,\n \t\treturn 1;\n \t}\n \n-\thead_pairs = get_diffpairs(o, common, head);\n-\tmerge_pairs = get_diffpairs(o, common, merge);\n+\thead_pairs = get_diffpairs(o, common, head, rename_ignore);\n+\tmerge_pairs = get_diffpairs(o, common, merge, rename_ignore);\n \n \tdir_re_head = get_directory_renames(head_pairs, head);\n \tdir_re_merge = get_directory_renames(merge_pairs, merge);\n@@ -3114,6 +3159,7 @@ int merge_trees(struct merge_options *o,\n \n \tif (unmerged_cache()) {\n \t\tstruct string_list *entries;\n+\t\tstruct string_list rename_ignore = STRING_LIST_INIT_NODUP;\n \t\tstruct rename_info re_info;\n \t\tint i;\n \t\t/*\n@@ -3128,8 +3174,9 @@ int merge_trees(struct merge_options *o,\n \t\tget_files_dirs(o, merge);\n \n \t\tentries = get_unmerged();\n+\t\tget_rename_ignore(entries, &rename_ignore);\n \t\tclean = handle_renames(o, common, head, merge, entries,\n-\t\t\t\t &re_info);\n+\t\t\t\t &rename_ignore, &re_info);\n \t\trecord_df_conflict_files(o, entries);\n \t\tif (clean < 0)\n \t\t\tgoto cleanup;\n@@ -3156,6 +3203,7 @@ int merge_trees(struct merge_options *o,\n cleanup:\n \t\tcleanup_renames(&re_info);\n \n+\t\tstring_list_clear(&rename_ignore, 0);\n \t\tstring_list_clear(entries, 1);\n \t\tfree(entries);\n \n-- \n2.15.0.323.g31fe956618\n\n"}]}