{"thread":{"id":"47162","subject":"[PATCH 00/30] Add directory rename detection to git","startedAt":"2017-11-10T19:06:06Z","lastAt":"2018-06-14T17:36:54Z","messageCount":81,"participants":["Elijah Newren","Philip Oakley","Stefan Beller","Junio C Hamano","René Scharfe","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":30},"messages":[{"id":"332213","messageId":"20171110190550.27059-1-newren@gmail.com","threadId":"47162","inReplyTo":null,"subject":"[PATCH 00/30] Add directory rename detection to git","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:20Z","receivedAt":"2017-11-10T19:06:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"[This series is entirely independent of my rename detection limits series.\nHowever, I have a separate rename detection performance series that depends\non both this series and the rename detection limits series.]\n\nIn this patchset, I introduce directory rename detection to merge-recursive,\npredominantly so that when files are added to directories on one side of\nhistory and those directories are renamed on the other side of history, the\nfiles will end up in the proper location after a merge or cherry-pick.\n\nHowever, this isn't limited to that simplistic case. More interesting\npossibilities exist, such as:\n\n * a file being renamed into a directory which is renamed on the other\n side of history, causing the need for a transitive rename.\n\n * two (or three or N) directories being merged (with no conflicts so\n long as files/directories within the merged directory have different\n names), and the \"merging\" being detected as a directory rename for\n each original directory.\n\n * not all files in a directory being renamed to the same location;\n i.e. perhaps the directory was renamed, but some files within it were\n renamed to a different location\n\n * a directory being renamed, which also contained a subdirectory that\n was renamed to some entirely different location. (And perhaps the\n inner directory itself contained inner directories that were renamed\n to yet other locations).\n\nAlso, I found it useful to allow all files within the directory being\nrenamed to themselves be renamed and still detect the directory rename.\nFor example, if goal/a and goal/b are renamed to priority/alpha and\npriority/bravo, we can detect that goal/ was renamed to priority/, so that\nif someone adds goal/c on the other side of history, after the merge we'll\nend up with priority/c. (In the absence of a readily available\nlibmindread.so library that I can link to, we can't rename directly from\ngoal/c to priority/charlie automatically, and will need to have priority/c\nsuffice.)\n\nNaturally, an attempt to do all of the above brings up all kinds of\ninteresting edge and corner cases, some of which result in conflicts\nthat cannot be represented in the index, and others of which might be\nconsidered too complex for users to understand and resolve. For\nexample:\n\n * An add/add/add/.../add conflict, all on one side of history (see\n testcase 9e in the new t6043, or any of the testcases in section 5)\n\n * Doubly, triply, or N-fold transitive renames (testcases 9c & 9d)\n\nIn order to prevent such problems, I introduce a couple basic rules that\nlimit when directory rename detection applies:\n\n 1) If a subset of to-be-renamed files have a file or directory in the\n way (or would be in the way of each other), \"turn off\" the directory\n rename for those specific sub-paths and report the conflict to the\n user.\n\n 2) If the other side of history did a directory rename to a path that\n your side of history renamed away, then ignore that particular\n rename from the other side of history for any implicit directory\n renames (but warn the user).\n\nFurther, there's a basic question about when directory rename detection\nshould be applied at all. I have a simple rule:\n\n 3) If a given directory still exists on both sides of a merge, we do\n not consider it to have been renamed.\n\nRule 3 may sound obvious at first, but it will probably arise as a\nquestion for some users -- what if someone \"mostly\" moved a directory but\nstill left some files around, or, equivalently (from the perspective of the\nthree-way merge that merge-recursive performs), fully renamed a directory\nin one commmit and then recreated that directory in a later commit adding\nsome new files and then tried to merge? See the big comment in section 4\nof the new t6043 for further discussion of this rule.\n\nThis set of rules seems to be reasonably easy to explain, is\nself-consistent, allows all conflict cases to be represented without\nchanging any on-disk data structures or introducing new terminology or\ncommands for users, prevents excessively complex conflicts that users\nmight struggle to understand, and brings peace to the middle east.\nActually, maybe not that last one.\n\nWhile I feel that this directory rename detection reduces the number of\nsuboptimal merges and cherry-picks that git performs, there are sadly\nstill a number of cases that remain suboptimal, or that even newly appear\nto be not-quite-consistent with other cases. The fact that one file\nlayout might trigger some of the rules above while another \"slightly\"\ndifferent file layout doesn't might occasionally cause some user\ngrumblings. I've tried to explore and document these cases in section 8\nof the new t6043-merge-rename-directories.sh\n\nFinally, from an implementation perspective, there's another strong\nadvantage to the ruleset above: it means that any path to which we want\nto apply an implicit directory rename will have a free and open spot\nfor us to move it into. Thus, we can just adjust the diff_filepair\nfrom an add or modify into a rename (or adjust a rename diff_filepair\nto change the target a little more), and then let process_renames and\nprocess_entry do all their magic. That allows us to rely on all the\nheavy testing already done for those code paths to handle a large\nvariety of edge and corner cases (e.g. D/F, rename/rename, criss-cross\nmerges, etc.) The big trick is just making sure to do all the\nnecessary checks that we can apply directory rename detection, and then\nfixing things up to put it in the expected format, with enough test\ncases to make sure we actually got it into the right format.\n\nOkay, the last paragraph had a small lie (though I didn't know that when\nI originally wrote it): the fact that unpack_trees() aborts early if it\ndetects an untracked or dirty file would be overwritten by a merge, and\nif not it immediately proceeds to start modifying the working tree before\npassing control back to merge-recursive, causes some problems. Not only\nhas it always made the code more complex, but the fact that\nunpack_trees() doesn't understand renames means that it can't\nappropriately abort early if a path involved in a rename has untracked\nor dirty contents in the way of the merge. But by the time we detect\nrenames, it's too late to abort early. So we have to instead figure out\nways of emitting warnings messages and writing something sensible to the\nworking copy without overwriting any of their data. This was a problem\nbefore directory rename detection, but directory rename detection\nincreases the number of places where we have to worry about this.\n\nElijah Newren (30):\n Tighten and correct a few testcases for merging and cherry-picking\n merge-recursive: Fix logic ordering issue\n merge-recursive: Add explanation for src_entry and dst_entry\n\nThese three patches provide a few miscellaneous fixups that could be\nsubmitted independent of this series, though the series partially\ndepends on the fixes in the first one, and the second fix becomes more\nimportant with the rest of the changes in this series.\n\n directory rename detection: basic testcases\n directory rename detection: directory splitting testcases\n directory rename detection: testcases to avoid taking detection too\n far\n directory rename detection: partially renamed directory\n testcase/discussion\n directory rename detection: files/directories in the way of some\n renames\n directory rename detection: testcases checking which side did the\n rename\n directory rename detection: more involved edge/corner testcases\n directory rename detection: testcases exploring possibly suboptimal\n merges\n directory rename detection: miscellaneous testcases to complete\n coverage\n directory rename detection: tests for handling overwriting untracked\n files\n directory rename detection: tests for handling overwriting dirty files\n\nThese patches add testcases for directory rename detection, trying to\ncover the space of possibilities as exhaustively as I can while trying\nto avoid excessive overlap in testcases\n\n merge-recursive: Move the get_renames() function\n merge-recursive: Introduce new functions to handle rename logic\n merge-recursive: Fix leaks of allocated renames and diff_filepairs\n merge-recursive: Make !o->detect_rename codepath more obvious\n merge-recursive: Split out code for determining diff_filepairs\n\nThese four patches make small code reorganizations in preparation for\nfurther changes, though they include some memory leak fixes.\n\n merge-recursive: Add a new hashmap for storing directory renames\n merge-recursive: Add get_directory_renames()\n merge-recursive: Check for directory level conflicts\n merge-recursive: Add a new hashmap for storing file collisions\n merge-recursive: Add computation of collisions due to dir rename &\n merging\n merge-recursive: Check for file level conflicts then get new name\n merge-recursive: When comparing files, don't include trees\n merge-recursive: Apply necessary modifications for directory renames\n\nThese eight patches implement the directory rename detection logic.\n \n merge-recursive: Avoid clobbering untracked files with directory\n renames\n merge-recursive: Fix overwriting dirty files involved in renames\n merge-recursive: Fix remaining directory rename + dirty overwrite\n cases\n\nThese last three deal with untracked and dirty file overwriting\nheadaches. The middle patch in particular, isn't just a fix for\ndirectory rename detection but fixes a bug in current versions of git\nin overwriting dirty files that are involved in a rename. That patch\ncould be backported and submitted independent of this series, but the\nfinal patch depends heavily on it.\n\n merge-recursive.c | 1212 +++++++++++--\n merge-recursive.h | 17 +\n t/t3501-revert-cherry-pick.sh | 5 +-\n t/t6043-merge-rename-directories.sh | 3277 +++++++++++++++++++++++++++++++++++\n t/t7607-merge-overwrite.sh | 7 +-\n unpack-trees.c | 4 +-\n unpack-trees.h | 4 +\n 7 files changed, 4413 insertions(+), 113 deletions(-)\n create mode 100755 t/t6043-merge-rename-directories.sh\n\n-- \n2.15.0.5.g9567be9905\n"},{"id":"332214","messageId":"20171110190550.27059-2-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 01/30] Tighten and correct a few testcases for merging and cherry-picking","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:21Z","receivedAt":"2017-11-10T19:06:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"t3501 had a testcase originally added to ensure cherry-pick wouldn't\nsegfault when working with a dirty file involved in a rename. While\nthe segfault was fixed, there was another problem this test demonstrated:\nnamely, that git would overwrite a dirty file involved in a rename.\nFurther, the test encoded a \"successful merge\" and overwriting of this\nfile as correct behavior. Modify the test so that it would still catch\nthe segfault, but to require the correct behavior.\n\nt7607 had a test specific to looking for a merge overwriting a dirty file\ninvolved in a rename, but it too actually encoded what I would term\nincorrect behavior: it expected the merge to succeed. Fix that, and add\na few more checks to make sure that the merge really does produce the\nexpected results.\n\nSigned-off-by: Elijah Newren \n---\n t/t3501-revert-cherry-pick.sh | 7 +++++--\n t/t7607-merge-overwrite.sh | 5 ++++-\n 2 files changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t3501-revert-cherry-pick.sh b/t/t3501-revert-cherry-pick.sh\nindex 4f2a263b63..783bdbf59d 100755\n--- a/t/t3501-revert-cherry-pick.sh\n+++ b/t/t3501-revert-cherry-pick.sh\n@@ -141,7 +141,7 @@ test_expect_success 'cherry-pick \"-\" works with arguments' '\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'cherry-pick works with dirty renamed file' '\n+test_expect_failure 'cherry-pick works with dirty renamed file' '\n \ttest_commit to-rename &&\n \tgit checkout -b unrelated &&\n \ttest_commit unrelated &&\n@@ -150,7 +150,10 @@ test_expect_success 'cherry-pick works with dirty renamed file' '\n \ttest_tick &&\n \tgit commit -m renamed &&\n \techo modified >renamed &&\n-\tgit cherry-pick refs/heads/unrelated\n+\ttest_must_fail git cherry-pick refs/heads/unrelated >out &&\n+\ttest_i18ngrep \"Refusing to lose dirty file at renamed\" out &&\n+\ttest $(git rev-parse :0:renamed) = $(git rev-parse HEAD^:to-rename.t) &&\n+\tgrep -q \"^modified$\" renamed\n '\n \n test_done\ndiff --git a/t/t7607-merge-overwrite.sh b/t/t7607-merge-overwrite.sh\nindex 9444d6a9b9..00617dadf8 100755\n--- a/t/t7607-merge-overwrite.sh\n+++ b/t/t7607-merge-overwrite.sh\n@@ -97,7 +97,10 @@ test_expect_failure 'will not overwrite unstaged changes in renamed file' '\n \tgit mv c1.c other.c &&\n \tgit commit -m rename &&\n \tcp important other.c &&\n-\tgit merge c1a &&\n+\ttest_must_fail git merge c1a >out &&\n+\ttest_i18ngrep \"Refusing to lose dirty file at other.c\" out &&\n+\ttest -f other.c~HEAD &&\n+\ttest $(git hash-object other.c~HEAD) = $(git rev-parse c1a:c1.c) &&\n \ttest_cmp important other.c\n '\n \n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332215","messageId":"20171110190550.27059-25-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 24/30] merge-recursive: Add computation of collisions due to dir rename & merging","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:44Z","receivedAt":"2017-11-10T19:06:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"directory renaming and merging can cause one or more files to be moved to\nwhere an existing file is, or to cause several files to all be moved to\nthe same (otherwise vacant) location. Add checking and reporting for such\ncases, falling back to no-directory-rename handling for such paths.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 118 +++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 116 insertions(+), 2 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 1858686c35..251c4cc7fa 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -1408,6 +1408,30 @@ static int tree_has_path(struct tree *tree, const char *path)\n \t\t\t hashy, &mode_o);\n }\n \n+/*\n+ * Return a new string that replaces the beginning portion (which matches\n+ * entry->dir), with entry->new_dir. In perl-speak:\n+ * new_path_name = (old_path =~ s/entry->dir/entry->new_dir/);\n+ */\n+static char *apply_dir_rename(struct dir_rename_entry *entry,\n+\t\t\t const char *old_path) {\n+\tchar *new_path;\n+\tint oldlen, newlen;\n+\n+\tif (entry->non_unique_new_dir)\n+\t\treturn NULL;\n+\n+\toldlen = strlen(entry->dir);\n+\tassert(strncmp(entry->dir, old_path, oldlen) == 0 &&\n+\t old_path[oldlen] == '/');\n+\tnewlen = strlen(entry->new_dir) + (strlen(old_path) - oldlen) + 1;\n+\tnew_path = malloc(newlen);\n+\tstrcpy(new_path, entry->new_dir);\n+\tstrcpy(&new_path[strlen(new_path)], &old_path[oldlen]);\n+\n+\treturn new_path;\n+}\n+\n static void get_renamed_dir_portion(const char *old_path, const char *new_path,\n \t\t\t\t char **old_dir, char **new_dir) {\n \t*old_dir = NULL;\n@@ -1625,6 +1649,82 @@ static struct hashmap *get_directory_renames(struct diff_queue_struct *pairs,\n \treturn dir_renames;\n }\n \n+static struct dir_rename_entry *check_dir_renamed(const char *path,\n+\t\t\t\t\t\t struct hashmap *dir_renames) {\n+\tchar temp[PATH_MAX];\n+\tchar *end;\n+\tstruct dir_rename_entry *entry;\n+\n+\tstrcpy(temp, path);\n+\twhile ((end = strrchr(temp, '/'))) {\n+\t\t*end = '\\0';\n+\t\tentry = dir_rename_find_entry(dir_renames, temp);\n+\t\tif (entry)\n+\t\t\treturn entry;\n+\t}\n+\treturn NULL;\n+}\n+\n+static void compute_collisions(struct hashmap *collisions,\n+\t\t\t struct hashmap *dir_renames,\n+\t\t\t struct diff_queue_struct *pairs)\n+{\n+\tint i;\n+\n+\t/*\n+\t * Multiple files can be mapped to the same path due to directory\n+\t * renames done by the other side of history. Since that other\n+\t * side of history could have merged multiple directories into one,\n+\t * if our side of history added the same file basename to each of\n+\t * those directories, then all N of them would get implicitly\n+\t * renamed by the directory rename detection into the same path,\n+\t * and we'd get an add/add/.../add conflict, and all those adds\n+\t * from *this* side of history. This is not representable in the\n+\t * index, and users aren't going to easily be able to make sense of\n+\t * it. So we need to provide a good warning about what's\n+\t * happening, and fall back to no-directory-rename detection\n+\t * behavior for those paths.\n+\t *\n+\t * See testcases 9e and all of section 5 from t6043 for examples.\n+\t */\n+\tcollision_init(collisions);\n+\n+\tfor (i = 0; i < pairs->nr; ++i) {\n+\t\tstruct dir_rename_entry *dir_rename_ent;\n+\t\tstruct collision_entry *collision_ent;\n+\t\tchar *new_path;\n+\t\tstruct diff_filepair *pair = pairs->queue[i];\n+\n+\t\tif (pair->status == 'D')\n+\t\t\tcontinue;\n+\t\tdir_rename_ent = check_dir_renamed(pair->two->path, dir_renames);\n+\t\tif (!dir_rename_ent)\n+\t\t\tcontinue;\n+\n+\t\tnew_path = apply_dir_rename(dir_rename_ent, pair->two->path);\n+\t\tif (!new_path)\n+\t\t\t/*\n+\t\t\t * dir_rename_ent->non_unique_new_path is true, which\n+\t\t\t * means there is no directory rename for us to use,\n+\t\t\t * which means it won't cause us any additional\n+\t\t\t * collisions.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\tcollision_ent = collision_find_entry(collisions, new_path);\n+\t\tif (!collision_ent) {\n+\t\t\tcollision_ent = xcalloc(1,\n+\t\t\t\t\t\tsizeof(struct collision_entry));\n+\t\t\thashmap_entry_init(collision_ent, strhash(new_path));\n+\t\t\thashmap_put(collisions, collision_ent);\n+\t\t\tcollision_ent->target_file = new_path;\n+\t\t} else {\n+\t\t\tfree(new_path);\n+\t\t}\n+\t\tstring_list_insert(&collision_ent->source_files,\n+\t\t\t\t pair->two->path);\n+\t}\n+}\n+\n /*\n * Get information of all renames which occurred in 'pairs', making use of\n * any implicit directory renames inferred from the other side of history.\n@@ -1634,6 +1734,7 @@ static struct hashmap *get_directory_renames(struct diff_queue_struct *pairs,\n */\n static struct string_list *get_renames(struct merge_options *o,\n \t\t\t\t struct diff_queue_struct *pairs,\n+\t\t\t\t struct hashmap *dir_renames,\n \t\t\t\t struct tree *tree,\n \t\t\t\t struct tree *o_tree,\n \t\t\t\t struct tree *a_tree,\n@@ -1641,8 +1742,12 @@ static struct string_list *get_renames(struct merge_options *o,\n \t\t\t\t struct string_list *entries)\n {\n \tint i;\n+\tstruct hashmap collisions;\n+\tstruct hashmap_iter iter;\n+\tstruct collision_entry *e;\n \tstruct string_list *renames;\n \n+\tcompute_collisions(&collisions, dir_renames, pairs);\n \trenames = xcalloc(1, sizeof(struct string_list));\n \n \tfor (i = 0; i < pairs->nr; ++i) {\n@@ -1672,6 +1777,13 @@ static struct string_list *get_renames(struct merge_options *o,\n \t\titem = string_list_insert(renames, pair->one->path);\n \t\titem->util = re;\n \t}\n+\n+\thashmap_iter_init(&collisions, &iter);\n+\twhile ((e = hashmap_iter_next(&iter))) {\n+\t\tfree(e->target_file);\n+\t\tstring_list_clear(&e->source_files, 0);\n+\t}\n+\thashmap_free(&collisions, 1);\n \treturn renames;\n }\n \n@@ -1962,9 +2074,11 @@ static struct rename_info *handle_renames(struct merge_options *o,\n \t\t\t\t\t dir_re_head, head,\n \t\t\t\t\t dir_re_merge, merge);\n \n-\trei->head_renames = get_renames(o, head_pairs, head,\n+\trei->head_renames = get_renames(o, head_pairs,\n+\t\t\t\t\t dir_re_merge, head,\n \t\t\t\t\t common, head, merge, entries);\n-\trei->merge_renames = get_renames(o, merge_pairs, merge,\n+\trei->merge_renames = get_renames(o, merge_pairs,\n+\t\t\t\t\t dir_re_head, merge,\n \t\t\t\t\t common, head, merge, entries);\n \t*clean = process_renames(o, rei->head_renames, rei->merge_renames);\n \n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332216","messageId":"20171110190550.27059-24-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 23/30] merge-recursive: Add a new hashmap for storing file collisions","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:43Z","receivedAt":"2017-11-10T19:06:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Directory renames with the ability to merge directories opens up the\npossibility of add/add/add/.../add conflicts, if each of the N\ndirectories being merged into one target directory all had a file with\nthe same name. We need a way to check for and report on such\ncollisions; this hashmap will be used for this purpose.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 23 +++++++++++++++++++++++\n merge-recursive.h | 7 +++++++\n 2 files changed, 30 insertions(+)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 3633be0123..1858686c35 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -73,6 +73,29 @@ static void dir_rename_init(struct hashmap *map)\n \thashmap_init(map, (hashmap_cmp_fn) dir_rename_cmp, NULL, 0);\n }\n \n+static struct collision_entry *collision_find_entry(struct hashmap *hashmap,\n+\t\t\t\t\t\t char *target_file)\n+{\n+\tstruct collision_entry key;\n+\n+\thashmap_entry_init(&key, strhash(target_file));\n+\tkey.target_file = target_file;\n+\treturn hashmap_get(hashmap, &key, NULL);\n+}\n+\n+static int collision_cmp(void *unused_cmp_data,\n+\t\t\t const struct collision_entry *e1,\n+\t\t\t const struct collision_entry *e2,\n+\t\t\t const void *unused_keydata)\n+{\n+\treturn strcmp(e1->target_file, e2->target_file);\n+}\n+\n+static void collision_init(struct hashmap *map)\n+{\n+\thashmap_init(map, (hashmap_cmp_fn) collision_cmp, NULL, 0);\n+}\n+\n static void flush_output(struct merge_options *o)\n {\n \tif (o->buffer_output < 2 && o->obuf.len) {\ndiff --git a/merge-recursive.h b/merge-recursive.h\nindex a024949739..e02c1e1243 100644\n--- a/merge-recursive.h\n+++ b/merge-recursive.h\n@@ -37,6 +37,13 @@ struct dir_rename_entry {\n \tstruct string_list possible_new_dirs;\n };\n \n+struct collision_entry {\n+\tstruct hashmap_entry ent; /* must be the first member! */\n+\tchar *target_file;\n+\tstruct string_list source_files;\n+\tunsigned reported_already:1;\n+};\n+\n /* merge_trees() but with recursive ancestor consolidation */\n int merge_recursive(struct merge_options *o,\n \t\t struct commit *h1,\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332217","messageId":"20171110190550.27059-12-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 11/30] directory rename detection: testcases exploring possibly suboptimal merges","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:31Z","receivedAt":"2017-11-10T19:06:23Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n t/t6043-merge-rename-directories.sh | 371 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 371 insertions(+)\n\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex 115d0d2622..bdfd943c88 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -1683,4 +1683,375 @@ test_expect_failure '7e-check: transitive rename in rename/delete AND dirs in th\n \ttest $(git hash-object y/d~C^0) = $(git rev-parse A:x/d)\n '\n \n+\n+###########################################################################\n+# SECTION 8: Suboptimal merges\n+#\n+# As alluded to in the last section, the ruleset we have built up for\n+# detecting directory renames unfortunately has some special cases where it\n+# results in slightly suboptimal or non-intuitive behavior. This section\n+# explores these cases.\n+#\n+# To be fair, we already had non-intuitive or suboptimal behavior for most\n+# of these cases in git before introducing implicit directory rename\n+# detection, but it'd be nice if there was a modified ruleset out there\n+# that handled these cases a bit better.\n+###########################################################################\n+\n+# Testcase 8a, Dual-directory rename, one into the others' way\n+# Commit A. x/{a,b}, y/{c,d}\n+# Commit B. x/{a,b,e}, y/{c,d,f}\n+# Commit C. y/{a,b}, z/{c,d}\n+#\n+# Possible Resolutions:\n+# Previous git: y/{a,b,f}, z/{c,d}, x/e\n+# Expected: y/{a,b,e,f}, z/{c,d}\n+# Preferred: y/{a,b,e}, z/{c,d,f}\n+#\n+# Note: Both x and y got renamed and it'd be nice to detect both, and we do\n+# better with directory rename detection than git did previously, but the\n+# simple rule from section 5 prevents me from handling this as optimally as\n+# we potentially could.\n+\n+test_expect_success '8a-setup: Dual-directory rename, one into the others way' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir x &&\n+\tmkdir y &&\n+\techo a >x/a &&\n+\techo b >x/b &&\n+\techo c >y/c &&\n+\techo d >y/d &&\n+\tgit add x y &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\techo e >x/e &&\n+\techo f >y/f &&\n+\tgit add x/e y/f &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv y z &&\n+\tgit mv x y &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '8a-check: Dual-directory rename, one into the others way' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 6 -eq $(git ls-files -s | wc -l) &&\n+\ttest 0 -eq $(git ls-files -u | wc -l) &&\n+\ttest 0 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/a) = $(git rev-parse A:x/a) &&\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:x/b) &&\n+\ttest $(git rev-parse HEAD:y/e) = $(git rev-parse B:x/e) &&\n+\ttest $(git rev-parse HEAD:y/f) = $(git rev-parse B:y/f) &&\n+\ttest $(git rev-parse HEAD:z/c) = $(git rev-parse A:y/c) &&\n+\ttest $(git rev-parse HEAD:z/d) = $(git rev-parse A:y/d)\n+'\n+\n+# Testcase 8b, Dual-directory rename, one into the others' way, with conflicting filenames\n+# Commit A. x/{a_1,b_1}, y/{a_2,b_2}\n+# Commit B. x/{a_1,b_1,e_1}, y/{a_2,b_2,e_2}\n+# Commit C. y/{a_1,b_1}, z/{a_2,b_2}\n+#\n+# Possible Resolutions:\n+# Previous git: y/{a_1,b_1,e_2}, z/{a_2,b_2}, x/e_1\n+# Scary: y/{a_1,b_1}, z/{a_2,b_2}, CONFLICT(add/add, e_1 vs. e_2)\n+# Preferred: y/{a_1,b_1,e_1}, z/{a_2,b_2,e_2}\n+#\n+# Note: Very similar to 8a, except instead of 'e' and 'f' in directories x and\n+# y, both are named 'e'. Without directory rename detection, neither file\n+# moves directories. Implment directory rename detection suboptimally, and\n+# you get an add/add conflict, but both files were added in commit B, so this\n+# is an add/add conflict where one side of history added both files --\n+# something we can't represent in the index. Obviously, we'd prefer the last\n+# resolution, but our previous rules are too coarse to allow it. Using both\n+# the rules from section 4 and section 5 save us from the Scary resolution,\n+# making us fall back to pre-directory-rename-detection behavior for both\n+# e_1 and e_2.\n+\n+test_expect_success '8b-setup: Dual-directory rename, one into the others way, with conflicting filenames' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir x &&\n+\tmkdir y &&\n+\techo a1 >x/a &&\n+\techo b1 >x/b &&\n+\techo a2 >y/a &&\n+\techo b2 >y/b &&\n+\tgit add x y &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\techo e1 >x/e &&\n+\techo e2 >y/e &&\n+\tgit add x/e y/e &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv y z &&\n+\tgit mv x y &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_success '8b-check: Dual-directory rename, one into the others way, with conflicting filenames' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 6 -eq $(git ls-files -s | wc -l) &&\n+\ttest 0 -eq $(git ls-files -u | wc -l) &&\n+\ttest 0 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/a) = $(git rev-parse A:x/a) &&\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:x/b) &&\n+\ttest $(git rev-parse HEAD:z/a) = $(git rev-parse A:y/a) &&\n+\ttest $(git rev-parse HEAD:z/b) = $(git rev-parse A:y/b) &&\n+\ttest $(git rev-parse HEAD:x/e) = $(git rev-parse B:x/e) &&\n+\ttest $(git rev-parse HEAD:y/e) = $(git rev-parse B:y/e)\n+'\n+\n+# Testcase 8c, rename+modify/delete\n+# (Related to testcases 5b and 8d)\n+# Commit A: z/{b,c,d}\n+# Commit B: y/{b,c}\n+# Commit C: z/{b,c,d_modified,e}\n+# Expected: y/{b,c,e}, CONFLICT(rename+modify/delete: x/d -> y/d or deleted)\n+#\n+# Note: This testcase doesn't present any concerns for me...until you\n+# compare it with testcases 5b and 8d. See notes in 8d for more\n+# details.\n+\n+test_expect_success '8c-setup: rename+modify/delete' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\ttest_seq 1 10 >z/d &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit rm z/d &&\n+\tgit mv z y &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\techo 11 >z/d &&\n+\ttest_chmod +x z/d &&\n+\techo e >z/e &&\n+\tgit add z/d z/e &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '8c-check: rename+modify/delete' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out &&\n+\ttest_i18ngrep \"CONFLICT (rename/delete).* z/d.*y/d\" out &&\n+\n+\ttest 4 -eq $(git ls-files -s | wc -l) &&\n+\ttest 1 -eq $(git ls-files -u | wc -l) &&\n+\ttest 1 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse :0:y/e) = $(git rev-parse C:z/e) &&\n+\n+\ttest_must_fail git rev-parse :1:y/d &&\n+\ttest_must_fail git rev-parse :2:y/d &&\n+\ttest $(git rev-parse :3:y/d) = $(git rev-parse C:z/d) &&\n+\tgit ls-files -s y/d | grep ^100755 &&\n+\ttest -f y/d\n+'\n+\n+# Testcase 8d, rename/delete...or not?\n+# (Related to testcase 5b; these may appear slightly inconsistent to users;\n+# Also related to testcases 7d and 7e)\n+# Commit A: z/{b,c,d}\n+# Commit B: y/{b,c}\n+# Commit C: z/{b,c,d,e}\n+# Expected: y/{b,c,e}\n+#\n+# Note: It would also be somewhat reasonable to resolve this as\n+# y/{b,c,e}, CONFLICT(rename/delete: x/d -> y/d or deleted)\n+# The logic being that the only difference between this testcase and 8c\n+# is that there is no modification to d. That suggests that instead of a\n+# rename/modify vs. delete conflict, we should just have a rename/delete\n+# conflict, otherwise we are being inconsistent.\n+#\n+# However...as far as consistency goes, we didn't report a conflict for\n+# path d_1 in testcase 5b due to a different file being in the way. So,\n+# we seem to be forced to have cases where users can change things\n+# slightly and get what they may perceive as inconsistent results. It\n+# would be nice to avoid that, but I'm not sure I see how.\n+#\n+# In this case, I'm leaning towards: commit B was the one that deleted z/d\n+# and it did the rename of z to y, so the two \"conflicts\" (rename vs.\n+# delete) are both coming from commit B, which is non-sensical. Conflicts\n+# during merging are supposed to be about opposite sides doing things\n+# differently.\n+\n+test_expect_success '8d-setup: rename/delete...or not?' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\ttest_seq 1 10 >z/d &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit rm z/d &&\n+\tgit mv z y &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\techo e >z/e &&\n+\tgit add z/e &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '8d-check: rename/delete...or not?' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 3 -eq $(git ls-files -s | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse HEAD:y/e) = $(git rev-parse C:z/e)\n+'\n+\n+# Testcase 8e, Both sides rename, one side adds to original directory\n+# Commit A: z/{b,c}\n+# Commit B: y/{b,c}\n+# Commit C: w/{b,c}, z/d\n+#\n+# Possible Resolutions:\n+# Previous git: z/d, CONFLICT(z/b -> y/b vs. w/b), CONFLICT(z/c -> y/c vs. w/c)\n+# Expected: y/d, CONFLICT(z/b -> y/b vs. w/b), CONFLICT(z/c -> y/c vs. w/c)\n+# Preferred: ??\n+#\n+# Notes: In commit B, directory z got renamed to y. In commit C, directory z\n+# did NOT get renamed; the directory is still present; instead it is\n+# considered to have just renamed a subset of paths in directory z\n+# elsewhere. Therefore, the directory rename done in commit B to z/\n+# applies to z/d and maps it to y/d.\n+#\n+# It's possible that users would get confused about this, but what\n+# should we do instead? Silently leaving at z/d seems just as bad or\n+# maybe even worse. Perhaps we could print a big warning about z/d\n+# and how we're moving to y/d in this case, but when I started thinking\n+# abouty the ramifications of doing that, I didn't know how to rule out\n+# that opening other weird edge and corner cases so I just punted.\n+\n+test_expect_success '8e-setup: Both sides rename, one side adds to original directory' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z y &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv z w &&\n+\tmkdir z &&\n+\techo d >z/d &&\n+\tgit add z/d &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '8e-check: Both sides rename, one side adds to original directory' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out 2>err &&\n+\n+\ttest 7 -eq $(git ls-files -s | wc -l) &&\n+\ttest 6 -eq $(git ls-files -u | wc -l) &&\n+\ttest 2 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/d) = $(git rev-parse C:z/d) &&\n+\n+\ttest $(git rev-parse :1:z/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :2:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :3:w/b) = $(git rev-parse A:z/b) &&\n+\ttest ! -f z/b &&\n+\ttest $(git hash-object y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git hash-object w/b) = $(git rev-parse A:z/b) &&\n+\n+\ttest $(git rev-parse :1:z/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse :2:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse :3:w/c) = $(git rev-parse A:z/c) &&\n+\ttest ! -f z/c &&\n+\ttest $(git hash-object y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git hash-object w/c) = $(git rev-parse A:z/c) &&\n+\n+\ttest_i18ngrep CONFLICT.*rename/rename.*z/c.*y/c.*w/c out &&\n+\ttest_i18ngrep CONFLICT.*rename/rename.*z/b.*y/b.*w/b out\n+'\n+\n test_done\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332218","messageId":"20171110190550.27059-17-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 16/30] merge-recursive: Introduce new functions to handle rename logic","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:36Z","receivedAt":"2017-11-10T19:06:25Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"The amount of logic in merge_trees() relative to renames was just a few\nlines, but split it out into new handle_renames() and cleanup_renames()\nfunctions to prepare for additional logic to be added to each. No code\nor logic changes, just a new place to put stuff for when the rename\ndetection gains additional checks.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 48 ++++++++++++++++++++++++++++++++++++++----------\n 1 file changed, 38 insertions(+), 10 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 382016508b..49710c0964 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -1638,6 +1638,38 @@ static int process_renames(struct merge_options *o,\n \treturn clean_merge;\n }\n \n+struct rename_info {\n+\tstruct string_list *head_renames;\n+\tstruct string_list *merge_renames;\n+};\n+\n+static struct rename_info *handle_renames(struct merge_options *o,\n+\t\t\t\t\t struct tree *common,\n+\t\t\t\t\t struct tree *head,\n+\t\t\t\t\t struct tree *merge,\n+\t\t\t\t\t struct string_list *entries,\n+\t\t\t\t\t int *clean)\n+{\n+\tstruct rename_info *rei = xcalloc(1, sizeof(struct rename_info));\n+\n+\trei->head_renames = get_renames(o, head, common, head, merge, entries);\n+\trei->merge_renames = get_renames(o, merge, common, head, merge, entries);\n+\t*clean = process_renames(o, rei->head_renames, rei->merge_renames);\n+\n+\treturn rei;\n+}\n+\n+static void cleanup_renames(struct rename_info *re_info)\n+{\n+\tstring_list_clear(re_info->head_renames, 0);\n+\tstring_list_clear(re_info->merge_renames, 0);\n+\n+\tfree(re_info->head_renames);\n+\tfree(re_info->merge_renames);\n+\n+\tfree(re_info);\n+}\n+\n static struct object_id *stage_oid(const struct object_id *oid, unsigned mode)\n {\n \treturn (is_null_oid(oid) || mode == 0) ? NULL: (struct object_id *)oid;\n@@ -1989,7 +2021,8 @@ int merge_trees(struct merge_options *o,\n \t}\n \n \tif (unmerged_cache()) {\n-\t\tstruct string_list *entries, *re_head, *re_merge;\n+\t\tstruct string_list *entries;\n+\t\tstruct rename_info *re_info;\n \t\tint i;\n \t\t/*\n \t\t * Only need the hashmap while processing entries, so\n@@ -2003,9 +2036,7 @@ int merge_trees(struct merge_options *o,\n \t\tget_files_dirs(o, merge);\n \n \t\tentries = get_unmerged();\n-\t\tre_head = get_renames(o, head, common, head, merge, entries);\n-\t\tre_merge = get_renames(o, merge, common, head, merge, entries);\n-\t\tclean = process_renames(o, re_head, re_merge);\n+\t\tre_info = handle_renames(o, common, head, merge, entries, &clean);\n \t\trecord_df_conflict_files(o, entries);\n \t\tif (clean < 0)\n \t\t\tgoto cleanup;\n@@ -2030,16 +2061,13 @@ int merge_trees(struct merge_options *o,\n \t\t}\n \n cleanup:\n-\t\tstring_list_clear(re_merge, 0);\n-\t\tstring_list_clear(re_head, 0);\n+\t\tcleanup_renames(re_info);\n+\n \t\tstring_list_clear(entries, 1);\n+\t\tfree(entries);\n \n \t\thashmap_free(&o->current_file_dir_set, 1);\n \n-\t\tfree(re_merge);\n-\t\tfree(re_head);\n-\t\tfree(entries);\n-\n \t\tif (clean < 0)\n \t\t\treturn clean;\n \t}\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332219","messageId":"20171110190550.27059-10-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 09/30] directory rename detection: testcases checking which side did the rename","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:29Z","receivedAt":"2017-11-10T19:06:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n t/t6043-merge-rename-directories.sh | 283 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 283 insertions(+)\n\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex d15153c652..157299105f 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -1053,4 +1053,287 @@ test_expect_failure '5d-check: Directory/file/file conflict due to directory ren\n # back to old handling. But, sadly, see testcases 8a and 8b.\n ###########################################################################\n \n+\n+###########################################################################\n+# SECTION 6: Same side of the merge was the one that did the rename\n+#\n+# It may sound obvious that you only want to apply implicit directory\n+# renames to directories if the _other_ side of history did the renaming.\n+# If you did make an implementation that didn't explicitly enforce this\n+# rule, the majority of cases that would fall under this section would\n+# also be solved by following the rules from the above sections. But\n+# there are still a few that stick out, so this section covers them just\n+# to make sure we also get them right.\n+###########################################################################\n+\n+# Testcase 6a, Tricky rename/delete\n+# Commit A: z/{b,c,d}\n+# Commit B: z/b\n+# Commit C: y/{b,c}, z/d\n+# Expected: y/b, CONFLICT(rename/delete, z/c -> y/c vs. NULL)\n+# Note: We're just checking here that the rename of z/b and z/c to put\n+# them under y/ doesn't accidentally catch z/d and make it look like\n+# it is also involved in a rename/delete conflict.\n+\n+test_expect_success '6a-setup: Tricky rename/delete' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\techo d >z/d &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit rm z/c &&\n+\tgit rm z/d &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tmkdir y &&\n+\tgit mv z/b y/ &&\n+\tgit mv z/c y/ &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_success '6a-check: Tricky rename/delete' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out &&\n+\ttest_i18ngrep \"CONFLICT (rename/delete).*z/c.*y/c\" out &&\n+\n+\ttest 2 -eq $(git ls-files -s | wc -l) &&\n+\ttest 1 -eq $(git ls-files -u | wc -l) &&\n+\ttest 1 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :3:y/c) = $(git rev-parse A:z/c)\n+'\n+\n+# Testcase 6b, Same rename done on both sides\n+# (Related to testcases 6c and 8e)\n+# Commit A: z/{b,c}\n+# Commit B: y/{b,c}\n+# Commit C: y/{b,c}, z/d\n+# Note: If we did directory rename detection here, we'd move z/d into y/,\n+# but C did that rename and still decided to put the file into z/,\n+# so we probably shouldn't apply directory rename detection for it.\n+\n+test_expect_success '6b-setup: Same rename done on both sides' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z y &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv z y &&\n+\tmkdir z &&\n+\techo d >z/d &&\n+\tgit add z/d &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_success '6b-check: Same rename done on both sides' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 3 -eq $(git ls-files -s | wc -l) &&\n+\ttest 0 -eq $(git ls-files -u | wc -l) &&\n+\ttest 0 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse HEAD:z/d) = $(git rev-parse C:z/d)\n+'\n+\n+# Testcase 6c, Rename only done on same side\n+# (Related to testcases 6b and 8e)\n+# Commit A: z/{b,c}\n+# Commit B: z/{b,c} (no change)\n+# Commit C: y/{b,c}, z/d\n+# Expected: y/{b,c}, z/d\n+# NOTE: Seems obvious, but just checking that the implementation doesn't\n+# \"accidentally detect a rename\" and give us y/{b,c,d}.\n+\n+test_expect_success '6c-setup: Rename only done on same side' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\ttest_tick &&\n+\tgit commit --allow-empty -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv z y &&\n+\tmkdir z &&\n+\techo d >z/d &&\n+\tgit add z/d &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_success '6c-check: Rename only done on same side' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 3 -eq $(git ls-files -s | wc -l) &&\n+\ttest 0 -eq $(git ls-files -u | wc -l) &&\n+\ttest 0 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse HEAD:z/d) = $(git rev-parse C:z/d)\n+'\n+\n+# Testcase 6d, We don't always want transitive renaming\n+# (Related to testcase 1c)\n+# Commit A: z/{b,c}, x/d\n+# Commit B: z/{b,c}, x/d (no change)\n+# Commit C: y/{b,c}, z/d\n+# Expected: y/{b,c}, z/d\n+# NOTE: Again, this seems obvious but just checking that the implementation\n+# doesn't \"accidentally detect a rename\" and give us y/{b,c,d}.\n+\n+test_expect_success '6d-setup: We do not always want transitive renaming' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tmkdir x &&\n+\techo d >x/d &&\n+\tgit add z x &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\ttest_tick &&\n+\tgit commit --allow-empty -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv z y &&\n+\tgit mv x z &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_success '6d-check: We do not always want transitive renaming' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 3 -eq $(git ls-files -s | wc -l) &&\n+\ttest 0 -eq $(git ls-files -u | wc -l) &&\n+\ttest 0 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse HEAD:z/d) = $(git rev-parse A:x/d)\n+'\n+\n+# Testcase 6e, Add/add from one-side\n+# Commit A: z/{b,c}\n+# Commit B: z/{b,c} (no change)\n+# Commit C: y/{b,c,d_1}, z/d_2\n+# Expected: y/{b,c,d_1}, z/d_2\n+# NOTE: Again, this seems obvious but just checking that the implementation\n+# doesn't \"accidentally detect a rename\" and give us y/{b,c} +\n+# add/add conflict on y/d_1 vs y/d_2.\n+\n+test_expect_success '6e-setup: Add/add from one side' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\ttest_tick &&\n+\tgit commit --allow-empty -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv z y &&\n+\techo d1 > y/d &&\n+\tmkdir z &&\n+\techo d2 > z/d &&\n+\tgit add y/d z/d &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_success '6e-check: Add/add from one side' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 4 -eq $(git ls-files -s | wc -l) &&\n+\ttest 0 -eq $(git ls-files -u | wc -l) &&\n+\ttest 0 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse HEAD:y/d) = $(git rev-parse C:y/d) &&\n+\ttest $(git rev-parse HEAD:z/d) = $(git rev-parse C:z/d)\n+'\n+\n test_done\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332220","messageId":"20171110190550.27059-7-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 06/30] directory rename detection: testcases to avoid taking detection too far","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:26Z","receivedAt":"2017-11-10T19:06:32Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n t/t6043-merge-rename-directories.sh | 137 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 137 insertions(+)\n\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex 00811f512a..021513ec00 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -513,4 +513,141 @@ test_expect_success '2b-check: Directory split into two on one side, with equal\n # messages are handled correctly.\n ###########################################################################\n \n+\n+###########################################################################\n+# SECTION 3: Path in question is the source path for some rename already\n+#\n+# Combining cases from Section 1 and trying to handle them could lead to\n+# directory renaming detection being over-applied. So, this section\n+# provides some good testcases to check that the implementation doesn't go\n+# too far.\n+###########################################################################\n+\n+# Testcase 3a, Avoid implicit rename if involved as source on other side\n+# (Related to testcases 1c and 1f)\n+# Commit A: z/{b,c,d}\n+# Commit B: z/{b,c,d} (no change)\n+# Commit C: y/{b,c}, x/d\n+# Expected: y/{b,c}, x/d\n+test_expect_success '3a-setup: Avoid implicit rename if involved as source on other side' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\techo d >z/d &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\ttest_tick &&\n+\tgit commit --allow-empty -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tmkdir y &&\n+\tmkdir x &&\n+\tgit mv z/b y/ &&\n+\tgit mv z/c y/ &&\n+\tgit mv z/d x/ &&\n+\trmdir z &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_success '3a-check: Avoid implicit rename if involved as source on other side' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 3 -eq $(git ls-files -s | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse HEAD:x/d) = $(git rev-parse A:z/d)\n+'\n+\n+# Testcase 3b, Avoid implicit rename if involved as source on other side\n+# (Related to testcases 5c and 7c, also kind of 1e and 1f)\n+# Commit A: z/{b,c,d}\n+# Commit B: y/{b,c}, x/d\n+# Commit C: z/{b,c}, w/d\n+# Expected: y/{b,c}, CONFLICT:(z/d -> x/d vs. w/d)\n+# NOTE: We're particularly checking that since z/d is already involved as\n+# a source in a file rename on the same side of history, that we don't\n+# get it involved in directory rename detection. If it were, we might\n+# end up with CONFLICT:(z/d -> y/d vs. x/d vs. w/d), i.e. a\n+# rename/rename/rename(1to3) conflict, which is just weird.\n+test_expect_success '3b-setup: Avoid implicit rename if involved as source on current side' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\techo d >z/d &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tmkdir y &&\n+\tmkdir x &&\n+\tgit mv z/b y/ &&\n+\tgit mv z/c y/ &&\n+\tgit mv z/d x/ &&\n+\trmdir z &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tmkdir w &&\n+\tgit mv z/d w/ &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_success '3b-check: Avoid implicit rename if involved as source on current side' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out &&\n+\n+\ttest 5 -eq $(git ls-files -s | wc -l) &&\n+\ttest 3 -eq $(git ls-files -u | wc -l) &&\n+\ttest 1 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:y/c) = $(git rev-parse A:z/c) &&\n+\n+\ttest $(git rev-parse :1:z/d) = $(git rev-parse A:z/d) &&\n+\ttest $(git rev-parse :2:x/d) = $(git rev-parse A:z/d) &&\n+\ttest $(git rev-parse :3:w/d) = $(git rev-parse A:z/d) &&\n+\ttest ! -f z/d &&\n+\ttest $(git hash-object x/d) = $(git rev-parse A:z/d) &&\n+\ttest $(git hash-object w/d) = $(git rev-parse A:z/d) &&\n+\n+\ttest_i18ngrep CONFLICT.*rename/rename.*z/d.*x/d.*w/d out &&\n+\t! test_i18ngrep CONFLICT.*rename/rename.*y/d\n+'\n+\n+###########################################################################\n+# Rules suggested by section 3:\n+#\n+# Avoid directory-rename-detection for a path, if that path is the source\n+# of a rename on either side of a merge.\n+###########################################################################\n+\n test_done\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332221","messageId":"20171110190550.27059-9-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 08/30] directory rename detection: files/directories in the way of some renames","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:28Z","receivedAt":"2017-11-10T19:06:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n t/t6043-merge-rename-directories.sh | 303 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 303 insertions(+)\n\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex ec054b210a..d15153c652 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -750,4 +750,307 @@ test_expect_success '4a-check: Directory split, with original directory still pr\n # detection.) But, sadly, see testcase 8b.\n ###########################################################################\n \n+\n+###########################################################################\n+# SECTION 5: Files/directories in the way of subset of to-be-renamed paths\n+#\n+# Implicitly renaming files due to a detected directory rename could run\n+# into problems if there are files or directories in the way of the paths\n+# we want to rename. Explore such cases in this section.\n+###########################################################################\n+\n+# Testcase 5a, Merge directories, other side adds files to original and target\n+# Commit A: z/{b,c}, y/d\n+# Commit B: z/{b,c,e_1,f}, y/{d,e_2}\n+# Commit C: y/{b,c,d}\n+# Expected: z/e_1, y/{b,c,d,e_2,f} + CONFLICT warning\n+# NOTE: While directory rename detection is active here causing z/f to\n+# become y/f, we did not apply this for z/e_1 because that would\n+# give us an add/add conflict for y/e_1 vs y/e_2. This problem with\n+# this add/add, is that both versions of y/e are from the same side\n+# of history, giving us no way to represent this conflict in the\n+# index.\n+\n+test_expect_success '5a-setup: Merge directories, other side adds files to original and target' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tmkdir y &&\n+\techo d >y/d &&\n+\tgit add z y &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\techo e1 >z/e &&\n+\techo f >z/f &&\n+\techo e2 >y/e &&\n+\tgit add z/e z/f y/e &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv z/b y/ &&\n+\tgit mv z/c y/ &&\n+\trmdir z &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '5a-check: Merge directories, other side adds files to original and target' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out &&\n+\n+\ttest 6 -eq $(git ls-files -s | wc -l) &&\n+\ttest 0 -eq $(git ls-files -u | wc -l) &&\n+\ttest 1 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse :0:y/d) = $(git rev-parse A:y/d) &&\n+\n+\ttest $(git rev-parse :0:y/e) = $(git rev-parse B:y/e) &&\n+\ttest $(git rev-parse :0:z/e) = $(git rev-parse B:z/e) &&\n+\n+\ttest $(git rev-parse :0:y/f) = $(git rev-parse B:z/f) &&\n+\n+\ttest_i18ngrep \"CONFLICT.*implicit dir rename\" out\n+'\n+\n+# Testcase 5b, Rename/delete in order to get add/add/add conflict\n+# (Related to testcase 8d; these may appear slightly inconsistent to users;\n+# Also related to testcases 7d and 7e)\n+# Commit A: z/{b,c,d_1}\n+# Commit B: y/{b,c,d_2}\n+# Commit C: z/{b,c,d_1,e}, y/d_3\n+# Expected: y/{b,c,e}, CONFLICT(add/add: y/d_2 vs. y/d_3)\n+# NOTE: If z/d_1 in commit C were to be involved in dir rename detection, as\n+# we normaly would since z/ is being renamed to y/, then this would be\n+# a rename/delete (z/d_1 -> y/d_1 vs. deleted) AND an add/add/add\n+# conflict of y/d_1 vs. y/d_2 vs. y/d_3. Add/add/add is not\n+# representable in the index, so the existence of y/d_3 needs to\n+# cause us to bail on directory rename detection for that path, falling\n+# back to git behavior without the directory rename detection.\n+\n+test_expect_success '5b-setup: Rename/delete in order to get add/add/add conflict' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\techo d1 >z/d &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit rm z/d &&\n+\tgit mv z y &&\n+\techo d2 >y/d &&\n+\tgit add y/d &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tmkdir y &&\n+\techo d3 >y/d &&\n+\techo e >z/e &&\n+\tgit add y/d z/e &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '5b-check: Rename/delete in order to get add/add/add conflict' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out &&\n+\ttest_i18ngrep \"CONFLICT (add/add).* y/d\" out &&\n+\n+\ttest 5 -eq $(git ls-files -s | wc -l) &&\n+\ttest 2 -eq $(git ls-files -u | wc -l) &&\n+\ttest 1 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse :0:y/e) = $(git rev-parse C:z/e) &&\n+\n+\ttest_must_fail git rev-parse :1:y/d &&\n+\ttest $(git rev-parse :2:y/d) = $(git rev-parse B:y/d) &&\n+\ttest $(git rev-parse :3:y/d) = $(git rev-parse C:y/d) &&\n+\ttest -f y/d\n+'\n+\n+# Testcase 5c, Transitive rename would cause rename/rename/rename/add/add/add\n+# (Directory rename detection would result in transitive rename vs.\n+# rename/rename(1to2) and turn it into a rename/rename(1to3). Further,\n+# rename paths conflict with separate adds on the other side)\n+# (Related to testcases 3b and 7c)\n+# Commit A: z/{b,c}, x/d_1\n+# Commit B: y/{b,c,d_2}, w/d_1\n+# Commit C: z/{b,c,d_1,e}, w/d_3, y/d_4\n+# Expected: A mess, but only a rename/rename(1to2)/add/add mess. Use the\n+# presence of y/d_4 in C to avoid doing transitive rename of\n+# x/d_1 -> z/d_1 -> y/d_1, so that the only paths we have at\n+# y/d are y/d_2 and y/d_4. We still do the move from z/e to y/e,\n+# though, because it doesn't have anything in the way.\n+\n+test_expect_success '5c-setup: Transitive rename would cause rename/rename/rename/add/add/add' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tmkdir x &&\n+\techo d1 >x/d &&\n+\tgit add z x &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z y &&\n+\techo d2 >y/d &&\n+\tgit add y/d &&\n+\tgit mv x w &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv x/d z/ &&\n+\tmkdir w &&\n+\tmkdir y &&\n+\techo d3 >w/d &&\n+\techo d4 >y/d &&\n+\techo e >z/e &&\n+\tgit add w/ y/ z/e &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '5c-check: Transitive rename would cause rename/rename/rename/add/add/add' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out &&\n+\ttest_i18ngrep \"CONFLICT (rename/rename).*x/d.*w/d.*z/d\" out &&\n+\ttest_i18ngrep \"CONFLICT (add/add).* y/d\" out &&\n+\n+\ttest 9 -eq $(git ls-files -s | wc -l) &&\n+\ttest 6 -eq $(git ls-files -u | wc -l) &&\n+\ttest 3 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse :0:y/e) = $(git rev-parse C:z/e) &&\n+\n+\ttest_must_fail git rev-parse :1:y/d &&\n+\ttest $(git rev-parse :2:w/d) = $(git rev-parse A:x/d) &&\n+\ttest $(git rev-parse :3:w/d) = $(git rev-parse C:w/d) &&\n+\ttest $(git rev-parse :1:x/d) = $(git rev-parse A:x/d) &&\n+\ttest $(git rev-parse :2:y/d) = $(git rev-parse B:y/d) &&\n+\ttest $(git rev-parse :3:y/d) = $(git rev-parse C:y/d) &&\n+\ttest $(git rev-parse :3:z/d) = $(git rev-parse A:x/d) &&\n+\n+\ttest $(git hash-object w/d~HEAD) = $(git rev-parse A:x/d) &&\n+\ttest $(git hash-object w/d~C^0) = $(git rev-parse C:w/d) &&\n+\ttest ! -f x/d &&\n+\ttest -f y/d &&\n+\tgrep -q \"<<<<\" y/d && # conflict markers should be present\n+\ttest $(git hash-object z/d) = $(git rev-parse A:x/d)\n+'\n+\n+# Testcase 5d, Directory/file/file conflict due to directory rename\n+# Commit A: z/{b,c}\n+# Commit B: y/{b,c,d_1}\n+# Commit C: z/{b,c,d_2,f}, y/d/e\n+# Expected: y/{b,c,d/e,f}, z/d_2, CONFLICT(file/directory), y/d_1~HEAD\n+# Note: The fact that y/d/ exists in C makes us bail on directory rename\n+# detection for z/d_2, but that doesn't prevent us from applying the\n+# directory rename detection for z/f -> y/f.\n+\n+test_expect_success '5d-setup: Directory/file/file conflict due to directory rename' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z y &&\n+\techo d1 >y/d &&\n+\tgit add y/d &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tmkdir -p y/d &&\n+\techo e >y/d/e &&\n+\techo d2 >z/d &&\n+\techo f >z/f &&\n+\tgit add y/d/e z/d z/f &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '5d-check: Directory/file/file conflict due to directory rename' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out &&\n+\ttest_i18ngrep \"CONFLICT (file/directory).*y/d\" out &&\n+\n+\ttest 6 -eq $(git ls-files -s | wc -l) &&\n+\ttest 1 -eq $(git ls-files -u | wc -l) &&\n+\ttest 2 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse :0:z/d) = $(git rev-parse C:z/d) &&\n+\ttest $(git rev-parse :0:y/f) = $(git rev-parse C:z/f) &&\n+\n+\ttest $(git rev-parse :2:y/d) = $(git rev-parse B:y/d) &&\n+\ttest $(git rev-parse :0:y/d/e) = $(git rev-parse C:y/d/e) &&\n+\n+\ttest $(git hash-object y/d~HEAD) = $(git rev-parse B:y/d)\n+'\n+\n+###########################################################################\n+# Rules suggested by section 5:\n+#\n+# If a subset of to-be-renamed files have a file or directory in the way,\n+# \"turn off\" the directory rename for those specific sub-paths, falling\n+# back to old handling. But, sadly, see testcases 8a and 8b.\n+###########################################################################\n+\n test_done\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332222","messageId":"20171110190550.27059-6-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 05/30] directory rename detection: directory splitting testcases","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:25Z","receivedAt":"2017-11-10T19:06:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n t/t6043-merge-rename-directories.sh | 125 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 125 insertions(+)\n\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex b737b0a105..00811f512a 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -388,4 +388,129 @@ test_expect_failure '1f-check: Split a directory into two other directories' '\n # in section 2, plus testcases 3a and 4a.\n ###########################################################################\n \n+\n+###########################################################################\n+# SECTION 2: Split into multiple directories, with equal number of paths\n+#\n+# Explore the splitting-a-directory rules a bit; what happens in the\n+# edge cases?\n+#\n+# Note that there is a closely related case of a directory not being\n+# split on either side of history, but being renamed differently on\n+# each side. See testcase 8e for that.\n+###########################################################################\n+\n+# Testcase 2a, Directory split into two on one side, with equal numbers of paths\n+# Commit A: z/{b,c}\n+# Commit B: y/b, w/c\n+# Commit C: z/{b,c,d}\n+# Expected: y/b, w/c, z/d, with warning about z/ -> (y/ vs. w/) conflict\n+test_expect_success '2a-setup: Directory split into two on one side, with equal numbers of paths' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tmkdir y &&\n+\tmkdir w &&\n+\tgit mv z/b y/ &&\n+\tgit mv z/c w/ &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\techo d >z/d &&\n+\tgit add z/d &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '2a-check: Directory split into two on one side, with equal numbers of paths' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out &&\n+\n+\ttest 3 -eq $(git ls-files -s | wc -l) &&\n+\ttest 0 -eq $(git ls-files -u | wc -l) &&\n+\ttest 1 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:w/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse :0:z/d) = $(git rev-parse C:z/d) &&\n+\ttest_i18ngrep \"CONFLICT.*directory rename split\" out\n+'\n+\n+# Testcase 2b, Directory split into two on one side, with equal numbers of paths\n+# Commit A: z/{b,c}\n+# Commit B: y/b, w/c\n+# Commit C: z/{b,c}, x/d\n+# Expected: y/b, w/c, x/d; No warning about z/ -> (y/ vs. w/) conflict\n+test_expect_success '2b-setup: Directory split into two on one side, with equal numbers of paths' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tmkdir y &&\n+\tmkdir w &&\n+\tgit mv z/b y/ &&\n+\tgit mv z/c w/ &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tmkdir x &&\n+\techo d >x/d &&\n+\tgit add x/d &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_success '2b-check: Directory split into two on one side, with equal numbers of paths' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 >out &&\n+\n+\ttest 3 -eq $(git ls-files -s | wc -l) &&\n+\ttest 0 -eq $(git ls-files -u | wc -l) &&\n+\ttest 1 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:w/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse :0:x/d) = $(git rev-parse C:x/d) &&\n+\t! test_i18ngrep \"CONFLICT.*directory rename split\" out\n+'\n+\n+###########################################################################\n+# Rules suggested by section 2:\n+#\n+# None; the rule was already covered in section 1. These testcases are\n+# here just to make sure the conflict resolution and necessary warning\n+# messages are handled correctly.\n+###########################################################################\n+\n test_done\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332223","messageId":"20171110190550.27059-8-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 07/30] directory rename detection: partially renamed directory testcase/discussion","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:27Z","receivedAt":"2017-11-10T19:06:39Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n t/t6043-merge-rename-directories.sh | 100 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 100 insertions(+)\n\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex 021513ec00..ec054b210a 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -650,4 +650,104 @@ test_expect_success '3b-check: Avoid implicit rename if involved as source on cu\n # of a rename on either side of a merge.\n ###########################################################################\n \n+\n+###########################################################################\n+# SECTION 4: Partially renamed directory; still exists on both sides of merge\n+#\n+# What if we were to attempt to do directory rename detection when someone\n+# \"mostly\" moved a directory but still left some files around, or,\n+# equivalently, fully renamed a directory in one commmit and then recreated\n+# that directory in a later commit adding some new files and then tried to\n+# merge?\n+#\n+# It's hard to divine user intent in these cases, because you can make an\n+# argument that, depending on the intermediate history of the side being\n+# merged, that some users will want files in that directory to\n+# automatically be detected and renamed, while users with a different\n+# intermediate history wouldn't want that rename to happen.\n+#\n+# I think that it is best to simply not have directory rename detection\n+# apply to such cases. My reasoning for this is four-fold: (1) it's\n+# easiest for users in general to figure out what happened if we don't\n+# apply directory rename detection in any such case, (2) it's an easy rule\n+# to explain [\"We don't do directory rename detection if the directory\n+# still exists on both sides of the merge\"], (3) we can get some hairy\n+# edge/corner cases that would be really confusing and possibly not even\n+# representable in the index if we were to even try, and [related to 3] (4)\n+# attempting to resolve this issue of divining user intent by examining\n+# intermediate history goes against the spirit of three-way merges and is a\n+# path towards crazy corner cases that are far more complex than what we're\n+# already dealing with.\n+#\n+# This section contains a test for this partially-renamed-directory case.\n+###########################################################################\n+\n+# Testcase 4a, Directory split, with original directory still present\n+# (Related to testcase 1f)\n+# Commit A: z/{b,c,d,e}\n+# Commit B: y/{b,c,d}, z/e\n+# Commit C: z/{b,c,d,e,f}\n+# Expected: y/{b,c,d}, z/{e,f}\n+# NOTE: Even though most files from z moved to y, we don't want f to follow.\n+\n+test_expect_success '4a-setup: Directory split, with original directory still present' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\techo d >z/d &&\n+\techo e >z/e &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tmkdir y &&\n+\tgit mv z/b y/ &&\n+\tgit mv z/c y/ &&\n+\tgit mv z/d y/ &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\techo f >z/f &&\n+\tgit add z/f &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_success '4a-check: Directory split, with original directory still present' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 5 -eq $(git ls-files -s | wc -l) &&\n+\ttest 0 -eq $(git ls-files -u | wc -l) &&\n+\ttest 0 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse HEAD:y/d) = $(git rev-parse A:z/d) &&\n+\ttest $(git rev-parse HEAD:z/e) = $(git rev-parse A:z/e) &&\n+\ttest $(git rev-parse HEAD:z/f) = $(git rev-parse C:z/f)\n+'\n+\n+###########################################################################\n+# Rules suggested by section 4:\n+#\n+# Directory-rename-detection should be turned off for any directories (as\n+# a source for renames) that exist on both sides of the merge. (The \"as\n+# a source for renames\" clarification is due to cases like 1c where\n+# the target directory exists on both sides and we do want the rename\n+# detection.) But, sadly, see testcase 8b.\n+###########################################################################\n+\n test_done\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332224","messageId":"20171110190550.27059-27-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 26/30] merge-recursive: When comparing files, don't include trees","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:46Z","receivedAt":"2017-11-10T19:06:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"get_renames() would look up stage data that already existed (populated\nin get_unmerged(), taken from whatever unpack_trees() created), and if\nit didn't exist, would call insert_stage_data() to create the necessary\nentry for the given file. The insert_stage_data() fallback becomes\nmuch more important for directory rename detection, because that creates\na mechanism to have a file in the resulting merge that didn't exist on\neither side of history. However, insert_stage_data(), due to calling\nget_tree_entry() loaded up trees as readily as files. We aren't\ninterested in comparing trees to files; the D/F conflict handling is\ndone elsewhere. This code is just concerned with what entries existed\nfor a given path on the different sides of the merge, so create a\nget_tree_entry_if_blob() helper function and use it.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 27 +++++++++++++++++++++------\n 1 file changed, 21 insertions(+), 6 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 7c2c29bb51..2a7258f6bb 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -407,6 +407,21 @@ static void get_files_dirs(struct merge_options *o, struct tree *tree)\n \tread_tree_recursive(tree, \"\", 0, 0, &match_all, save_files_dirs, o);\n }\n \n+static int get_tree_entry_if_blob(const unsigned char *tree,\n+\t\t\t\t const char *path,\n+\t\t\t\t unsigned char *hashy,\n+\t\t\t\t unsigned *mode_o)\n+{\n+\tint ret;\n+\n+\tret = get_tree_entry(tree, path, hashy, mode_o);\n+\tif (S_ISDIR(*mode_o)) {\n+\t\thashcpy(hashy, null_sha1);\n+\t\t*mode_o = 0;\n+\t}\n+\treturn ret;\n+}\n+\n /*\n * Returns an index_entry instance which doesn't have to correspond to\n * a real cache entry in Git's index.\n@@ -417,12 +432,12 @@ static struct stage_data *insert_stage_data(const char *path,\n {\n \tstruct string_list_item *item;\n \tstruct stage_data *e = xcalloc(1, sizeof(struct stage_data));\n-\tget_tree_entry(o->object.oid.hash, path,\n-\t\t\te->stages[1].oid.hash, &e->stages[1].mode);\n-\tget_tree_entry(a->object.oid.hash, path,\n-\t\t\te->stages[2].oid.hash, &e->stages[2].mode);\n-\tget_tree_entry(b->object.oid.hash, path,\n-\t\t\te->stages[3].oid.hash, &e->stages[3].mode);\n+\tget_tree_entry_if_blob(o->object.oid.hash, path,\n+\t\t\t e->stages[1].oid.hash, &e->stages[1].mode);\n+\tget_tree_entry_if_blob(a->object.oid.hash, path,\n+\t\t\t e->stages[2].oid.hash, &e->stages[2].mode);\n+\tget_tree_entry_if_blob(b->object.oid.hash, path,\n+\t\t\t e->stages[3].oid.hash, &e->stages[3].mode);\n \titem = string_list_insert(entries, path);\n \titem->util = e;\n \treturn e;\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332225","messageId":"20171110190550.27059-30-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[RFC PATCH 29/30] merge-recursive: Fix overwriting dirty files involved in renames","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:49Z","receivedAt":"2017-11-10T19:06:45Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This fixes an issue that existed before my directory rename detection\npatches that affects both normal renames and renames implied by\ndirectory rename detection. Additional codepaths that only affect\noverwriting of directy files that are involved in directory rename\ndetection will be added in a subsequent commit.\n\nSigned-off-by: Elijah Newren \n---\nSeems kinda hacky, in multiple different ways. It seems like there\nshould be a better way to handle lots of different things about this\npatch, though I have a hard time seeing how without doing a bigger\nrewrite of the whole interface between unpack_trees and\nmerge-recursive (possibly rewriting them so there isn't an interface\nbut some piece of code that does both functions). Alternate simpler\nsuggestions?\n\n\n merge-recursive.c | 81 ++++++++++++++++++++++++++++---------\n merge-recursive.h | 2 +\n t/t3501-revert-cherry-pick.sh | 2 +-\n t/t6043-merge-rename-directories.sh | 2 +-\n t/t7607-merge-overwrite.sh | 2 +-\n unpack-trees.c | 4 +-\n unpack-trees.h | 4 ++\n 7 files changed, 73 insertions(+), 24 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 1b3ee5b9fb..86ddb89727 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -323,32 +323,32 @@ static void init_tree_desc_from_tree(struct tree_desc *desc, struct tree *tree)\n \tinit_tree_desc(desc, tree->buffer, tree->size);\n }\n \n-static int git_merge_trees(int index_only,\n+static int git_merge_trees(struct merge_options *o,\n \t\t\t struct tree *common,\n \t\t\t struct tree *head,\n \t\t\t struct tree *merge)\n {\n \tint rc;\n \tstruct tree_desc t[3];\n-\tstruct unpack_trees_options opts;\n \n-\tmemset(&opts, 0, sizeof(opts));\n-\tif (index_only)\n-\t\topts.index_only = 1;\n+\tmemset(&o->unpack_opts, 0, sizeof(o->unpack_opts));\n+\tif (o->call_depth)\n+\t\to->unpack_opts.index_only = 1;\n \telse\n-\t\topts.update = 1;\n-\topts.merge = 1;\n-\topts.head_idx = 2;\n-\topts.fn = threeway_merge;\n-\topts.src_index = &the_index;\n-\topts.dst_index = &the_index;\n-\tsetup_unpack_trees_porcelain(&opts, \"merge\");\n+\t\to->unpack_opts.update = 1;\n+\to->unpack_opts.merge = 1;\n+\to->unpack_opts.head_idx = 2;\n+\to->unpack_opts.fn = threeway_merge;\n+\to->unpack_opts.src_index = &the_index;\n+\to->unpack_opts.dst_index = &the_index;\n+\tsetup_unpack_trees_porcelain(&o->unpack_opts, \"merge\");\n \n \tinit_tree_desc_from_tree(t+0, common);\n \tinit_tree_desc_from_tree(t+1, head);\n \tinit_tree_desc_from_tree(t+2, merge);\n \n-\trc = unpack_trees(3, t, &opts);\n+\trc = unpack_trees(3, t, &o->unpack_opts);\n+\to->unpack_opts.src_index = &the_index; // unpack_trees NULLifies src_index, but it's used in verify_uptodate, so set it back\n \tcache_tree_free(&active_cache_tree);\n \treturn rc;\n }\n@@ -783,6 +783,20 @@ static int would_lose_untracked(const char *path)\n \treturn !was_tracked(path) && file_exists(path);\n }\n \n+static int was_dirty(struct merge_options *o, const char *path)\n+{\n+\tstruct cache_entry *ce;\n+\n+\tint dirty = 1;\n+\tif (o->call_depth || !was_tracked(path))\n+\t return !dirty;\n+\n+\tce = cache_file_exists(path, strlen(path), ignore_case);\n+\tdirty = (ce->ce_stat_data.sd_mtime.sec > 0 &&\n+\t\t verify_uptodate(ce, &o->unpack_opts) != 0);\n+\treturn dirty;\n+}\n+\n static int make_room_for_path(struct merge_options *o, const char *path)\n {\n \tint status, i;\n@@ -2635,6 +2649,7 @@ static int handle_modify_delete(struct merge_options *o,\n \n static int merge_content(struct merge_options *o,\n \t\t\t const char *path,\n+\t\t\t int file_in_way,\n \t\t\t struct object_id *o_oid, int o_mode,\n \t\t\t struct object_id *a_oid, int a_mode,\n \t\t\t struct object_id *b_oid, int b_mode,\n@@ -2709,7 +2724,7 @@ static int merge_content(struct merge_options *o,\n \t\t\t\treturn -1;\n \t}\n \n-\tif (df_conflict_remains) {\n+\tif (df_conflict_remains || file_in_way) {\n \t\tchar *new_path;\n \t\tif (o->call_depth) {\n \t\t\tremove_file_from_cache(path);\n@@ -2743,6 +2758,31 @@ static int merge_content(struct merge_options *o,\n \treturn mfi.clean;\n }\n \n+static int conflict_rename_normal(struct merge_options *o,\n+\t\t\t\t const char *path,\n+\t\t\t\t struct object_id *o_oid, unsigned o_mode,\n+\t\t\t\t struct object_id *a_oid, unsigned a_mode,\n+\t\t\t\t struct object_id *b_oid, unsigned b_mode,\n+\t\t\t\t struct rename_conflict_info *ci)\n+{\n+\tint clean_merge;\n+\tint file_in_the_way = 0;\n+\n+\tif (was_dirty(o, path)) {\n+\t\t\tfile_in_the_way = 1;\n+\t\t\toutput(o, 1, _(\"Refusing to lose dirty file at %s\"),\n+\t\t\t path);\n+\t}\n+\n+\t/* Merge the content and write it out */\n+\tclean_merge = merge_content(o, path, file_in_the_way,\n+\t\t\t\t o_oid, o_mode, a_oid, a_mode, b_oid, b_mode,\n+\t\t\t\t ci);\n+\tif (clean_merge > 0 && file_in_the_way)\n+\t\tclean_merge = 0;\n+\treturn clean_merge;\n+}\n+\n /* Per entry merge function */\n static int process_entry(struct merge_options *o,\n \t\t\t const char *path, struct stage_data *entry)\n@@ -2762,9 +2802,12 @@ static int process_entry(struct merge_options *o,\n \t\tswitch (conflict_info->rename_type) {\n \t\tcase RENAME_NORMAL:\n \t\tcase RENAME_ONE_FILE_TO_ONE:\n-\t\t\tclean_merge = merge_content(o, path,\n-\t\t\t\t\t\t o_oid, o_mode, a_oid, a_mode, b_oid, b_mode,\n-\t\t\t\t\t\t conflict_info);\n+\t\t\tclean_merge = conflict_rename_normal(o,\n+\t\t\t\t\t\t\t path,\n+\t\t\t\t\t\t\t o_oid, o_mode,\n+\t\t\t\t\t\t\t a_oid, a_mode,\n+\t\t\t\t\t\t\t b_oid, b_mode,\n+\t\t\t\t\t\t\t conflict_info);\n \t\t\tbreak;\n \t\tcase RENAME_DIR:\n \t\t\tclean_merge = 1;\n@@ -2859,7 +2902,7 @@ static int process_entry(struct merge_options *o,\n \t} else if (a_oid && b_oid) {\n \t\t/* Case C: Added in both (check for same permissions) and */\n \t\t/* case D: Modified in both, but differently. */\n-\t\tclean_merge = merge_content(o, path,\n+\t\tclean_merge = merge_content(o, path, 0 /* file_in_way */,\n \t\t\t\t\t o_oid, o_mode, a_oid, a_mode, b_oid, b_mode,\n \t\t\t\t\t NULL);\n \t} else if (!o_oid && !a_oid && !b_oid) {\n@@ -2893,7 +2936,7 @@ int merge_trees(struct merge_options *o,\n \t\treturn 1;\n \t}\n \n-\tcode = git_merge_trees(o->call_depth, common, head, merge);\n+\tcode = git_merge_trees(o, common, head, merge);\n \n \tif (code != 0) {\n \t\tif (show(o, 4) || o->call_depth)\ndiff --git a/merge-recursive.h b/merge-recursive.h\nindex e02c1e1243..591b824a98 100644\n--- a/merge-recursive.h\n+++ b/merge-recursive.h\n@@ -1,6 +1,7 @@\n #ifndef MERGE_RECURSIVE_H\n #define MERGE_RECURSIVE_H\n \n+#include \"unpack-trees.h\"\n #include \"string-list.h\"\n \n struct merge_options {\n@@ -27,6 +28,7 @@ struct merge_options {\n \tstruct strbuf obuf;\n \tstruct hashmap current_file_dir_set;\n \tstruct string_list df_conflict_file_set;\n+\tstruct unpack_trees_options unpack_opts;\n };\n \n struct dir_rename_entry {\ndiff --git a/t/t3501-revert-cherry-pick.sh b/t/t3501-revert-cherry-pick.sh\nindex 783bdbf59d..0d89f6d0f6 100755\n--- a/t/t3501-revert-cherry-pick.sh\n+++ b/t/t3501-revert-cherry-pick.sh\n@@ -141,7 +141,7 @@ test_expect_success 'cherry-pick \"-\" works with arguments' '\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'cherry-pick works with dirty renamed file' '\n+test_expect_success 'cherry-pick works with dirty renamed file' '\n \ttest_commit to-rename &&\n \tgit checkout -b unrelated &&\n \ttest_commit unrelated &&\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex 6db764a1b6..02c97c9823 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -2925,7 +2925,7 @@ test_expect_success '11a-setup: Avoid losing dirty contents with simple rename'\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '11a-check: Avoid losing dirty contents with simple rename' '\n+test_expect_success '11a-check: Avoid losing dirty contents with simple rename' '\n \tgit checkout B^0 &&\n \techo stuff >>z/c &&\n \ndiff --git a/t/t7607-merge-overwrite.sh b/t/t7607-merge-overwrite.sh\nindex 00617dadf8..e44fb50173 100755\n--- a/t/t7607-merge-overwrite.sh\n+++ b/t/t7607-merge-overwrite.sh\n@@ -92,7 +92,7 @@ test_expect_success 'will not overwrite removed file with staged changes' '\n \ttest_cmp important c1.c\n '\n \n-test_expect_failure 'will not overwrite unstaged changes in renamed file' '\n+test_expect_success 'will not overwrite unstaged changes in renamed file' '\n \tgit reset --hard c1 &&\n \tgit mv c1.c other.c &&\n \tgit commit -m rename &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 71b70ccb12..71826edfb6 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1484,8 +1484,8 @@ static int verify_uptodate_1(const struct cache_entry *ce,\n \t\tadd_rejected_path(o, error_type, ce->name);\n }\n \n-static int verify_uptodate(const struct cache_entry *ce,\n-\t\t\t struct unpack_trees_options *o)\n+int verify_uptodate(const struct cache_entry *ce,\n+\t\t struct unpack_trees_options *o)\n {\n \tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n \t\treturn 0;\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 6c48117b84..41178ada94 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -1,6 +1,7 @@\n #ifndef UNPACK_TREES_H\n #define UNPACK_TREES_H\n \n+#include \"tree-walk.h\"\n #include \"string-list.h\"\n \n #define MAX_UNPACK_TREES 8\n@@ -78,6 +79,9 @@ struct unpack_trees_options {\n extern int unpack_trees(unsigned n, struct tree_desc *t,\n \t\tstruct unpack_trees_options *options);\n \n+int verify_uptodate(const struct cache_entry *ce,\n+\t\t struct unpack_trees_options *o);\n+\n int threeway_merge(const struct cache_entry * const *stages,\n \t\t struct unpack_trees_options *o);\n int twoway_merge(const struct cache_entry * const *src,\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332226","messageId":"20171110190550.27059-13-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 12/30] directory rename detection: miscellaneous testcases to complete coverage","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:32Z","receivedAt":"2017-11-10T19:06:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n t/t6043-merge-rename-directories.sh | 505 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 505 insertions(+)\n\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex bdfd943c88..bb179b16c8 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -265,6 +265,7 @@ test_expect_failure '1d-check: Directory renames cause a rename/rename(2to1) con\n '\n \n # Testcase 1e, Renamed directory, with all filenames being renamed too\n+# (Related to testcases 9f & 9g)\n # Commit A: z/{oldb,oldc}\n # Commit B: y/{newb,newc}\n # Commit C: z/{oldb,oldc,d}\n@@ -2054,4 +2055,508 @@ test_expect_failure '8e-check: Both sides rename, one side adds to original dire\n \ttest_i18ngrep CONFLICT.*rename/rename.*z/b.*y/b.*w/b out\n '\n \n+###########################################################################\n+# SECTION 9: Other testcases\n+#\n+# I came up with the testcases in the first eight sections before coding up\n+# the implementation. The testcases in this section were mostly ones I\n+# thought of while coding/debugging, and which I was too lazy to insert\n+# into the previous sections because I didn't want to re-label with all the\n+# testcase references. :-)\n+###########################################################################\n+\n+# Testcase 9a, Inner renamed directory within outer renamed directory\n+# (Related to testcase 1f)\n+# Commit A: z/{b,c,d/{e,f,g}}\n+# Commit B: y/{b,c}, x/w/{e,f,g}\n+# Commit C: z/{b,c,d/{e,f,g,h},i}\n+# Expected: y/{b,c,i}, x/w/{e,f,g,h}\n+# NOTE: The only reason this one is interesting is because when a directory\n+# is split into multiple other directories, we determine by the weight\n+# of which one had the most paths going to it. A naive implementation\n+# of that could take the new file in commit C at z/i to x/w/i or x/i.\n+\n+test_expect_success '9a-setup: Inner renamed directory within outer renamed directory' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir -p z/d &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\techo e >z/d/e &&\n+\techo f >z/d/f &&\n+\techo g >z/d/g &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tmkdir x &&\n+\tgit mv z/d x/w &&\n+\tgit mv z y &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\techo h >z/d/h &&\n+\techo i >z/i &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '9a-check: Inner renamed directory within outer renamed directory' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 7 -eq $(git ls-files -s | wc -l) &&\n+\ttest 0 -eq $(git ls-files -u | wc -l) &&\n+\ttest 0 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse HEAD:y/i) = $(git rev-parse C:z/i) &&\n+\n+\ttest $(git rev-parse HEAD:x/w/e) = $(git rev-parse A:z/d/e) &&\n+\ttest $(git rev-parse HEAD:x/w/f) = $(git rev-parse A:z/d/f) &&\n+\ttest $(git rev-parse HEAD:x/w/g) = $(git rev-parse A:z/d/g) &&\n+\ttest $(git rev-parse HEAD:x/w/h) = $(git rev-parse C:z/d/h)\n+'\n+\n+# Testcase 9b, Transitive rename with content merge\n+# (Related to testcase 1c)\n+# Commit A: z/{b,c}, x/d_1\n+# Commit B: y/{b,c}, x/d_2\n+# Commit C: z/{b,c,d_3}\n+# Expected: y/{b,c,d_merged}\n+\n+test_expect_success '9b-setup: Transitive rename with content merge' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tmkdir x &&\n+\ttest_seq 1 10 >x/d &&\n+\tgit add z x &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z y &&\n+\ttest_seq 1 11 >x/d &&\n+\tgit add x/d &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\ttest_seq 0 10 >x/d &&\n+\tgit mv x/d z/d &&\n+\tgit add z/d &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '9b-check: Transitive rename with content merge' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 3 -eq $(git ls-files -s | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n+\ttest_must_fail git rev-parse HEAD:x/d &&\n+\ttest_must_fail git rev-parse HEAD:z/d &&\n+\ttest ! -f z/d &&\n+\n+\ttest $(git rev-parse HEAD:y/d) != $(git rev-parse A:x/d) &&\n+\ttest $(git rev-parse HEAD:y/d) != $(git rev-parse B:x/d) &&\n+\ttest $(git rev-parse HEAD:y/d) != $(git rev-parse C:z/d) &&\n+\ttest_seq 0 11 >expected &&\n+\tgit add expected &&\n+\ttest $(git rev-parse HEAD:y/d) = $(git rev-parse :0:expected) &&\n+\ttest_cmp expected y/d\n+'\n+\n+# Testcase 9c, Doubly transitive rename?\n+# (Related to testcase 1c, 7e, and 9d)\n+# Commit A: z/{b,c}, x/{d,e}, w/f\n+# Commit B: y/{b,c}, x/{d,e,f,g}\n+# Commit C: z/{b,c,d,e}, w/f\n+# Expected: y/{b,c,d,e}, x/{f,g}\n+#\n+# NOTE: x/f and x/g may be slightly confusing here. The rename from w/f to\n+# x/f is clear. Let's look beyond that. Here's the logic:\n+# Commit C renamed x/ -> z/\n+# Commit B renamed z/ -> y/\n+# So, we could possibly further rename x/f to z/f to y/f, a doubly\n+# transient rename. However, where does it end? We can chain these\n+# indefinitely (see testcase 9d). What if there is a D/F conflict\n+# at z/f/ or y/f/? Or just another file conflict at one of those\n+# paths? In the case of an N-long chain of transient renamings,\n+# where do we \"abort\" the rename at? Can the user make sense of\n+# the resulting conflict and resolve it?\n+#\n+# To avoid this confusion I use the simple rule that if the other side\n+# of history did a directory rename to a path that your side renamed\n+# away, then ignore that particular rename from the other side of\n+# history for any implicit directory renames.\n+\n+test_expect_success '9c-setup: Doubly transitive rename?' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tmkdir x &&\n+\techo d >x/d &&\n+\techo e >x/e &&\n+\tmkdir w &&\n+\techo f >w/f &&\n+\tgit add z x w &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z y &&\n+\tgit mv w/f x/ &&\n+\techo g >x/g &&\n+\tgit add x/g &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv x/d z/d &&\n+\tgit mv x/e z/e &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '9c-check: Doubly transitive rename?' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 >out &&\n+\ttest_i18ngrep \"WARNING: Avoiding applying x -> z rename to x/f\" out &&\n+\n+\ttest 6 -eq $(git ls-files -s | wc -l) &&\n+\ttest 1 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse HEAD:y/d) = $(git rev-parse A:x/d) &&\n+\ttest $(git rev-parse HEAD:y/e) = $(git rev-parse A:x/e) &&\n+\ttest $(git rev-parse HEAD:x/f) = $(git rev-parse A:w/f) &&\n+\ttest $(git rev-parse HEAD:x/g) = $(git rev-parse B:x/g)\n+'\n+\n+# Testcase 9d, N-fold transitive rename?\n+# (Related to testcase 9c...and 1c and 7e)\n+# Commit A: z/a, y/b, x/c, w/d, v/e, u/f\n+# Commit B: y/{a,b}, w/{c,d}, u/{e,f}\n+# Commit C: z/{a,t}, x/{b,c}, v/{d,e}, u/f\n+# Expected: \n+#\n+# NOTE: z/ -> y/ (in commit B)\n+# y/ -> x/ (in commit C)\n+# x/ -> w/ (in commit B)\n+# w/ -> v/ (in commit C)\n+# v/ -> u/ (in commit B)\n+# So, if we add a file to z, say z/t, where should it end up? In u?\n+# What if there's another file or directory named 't' in one of the\n+# intervening directories and/or in u itself? Also, shouldn't the\n+# same logic that places 't' in u/ also move ALL other files to u/?\n+# What if there are file or directory conflicts in any of them? If\n+# we attempted to do N-way (N-fold? N-ary? N-uple?) transitive renames\n+# like this, would the user have any hope of understanding any\n+# conflicts or how their working tree ended up? I think not, so I'm\n+# ruling out N-ary transitive renames for N>1.\n+#\n+# Therefore our expected result is:\n+# z/t, y/a, x/b, w/c, u/d, u/e, u/f\n+# The reason that v/d DOES get transitively renamed to u/d is that u/ isn't\n+# renamed somewhere. A slightly sub-optimal result, but it uses fairly\n+# simple rules that are consistent with what we need for all the other\n+# testcases and simplifies things for the user.\n+\n+test_expect_success '9d-setup: N-way transitive rename?' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z y x w v u &&\n+\techo a >z/a &&\n+\techo b >y/b &&\n+\techo c >x/c &&\n+\techo d >w/d &&\n+\techo e >v/e &&\n+\techo f >u/f &&\n+\tgit add z y x w v u &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z/a y/ &&\n+\tgit mv x/c w/ &&\n+\tgit mv v/e u/ &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\techo t >z/t &&\n+\tgit mv y/b x/ &&\n+\tgit mv w/d v/ &&\n+\tgit add z/t &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '9d-check: N-way transitive rename?' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 >out &&\n+\ttest_i18ngrep \"WARNING: Avoiding applying z -> y rename to z/t\" out &&\n+\ttest_i18ngrep \"WARNING: Avoiding applying y -> x rename to y/a\" out &&\n+\ttest_i18ngrep \"WARNING: Avoiding applying x -> w rename to x/b\" out &&\n+\ttest_i18ngrep \"WARNING: Avoiding applying w -> v rename to w/c\" out &&\n+\n+\ttest 7 -eq $(git ls-files -s | wc -l) &&\n+\ttest 1 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:z/t) = $(git rev-parse C:z/t) &&\n+\ttest $(git rev-parse HEAD:y/a) = $(git rev-parse A:z/a) &&\n+\ttest $(git rev-parse HEAD:x/b) = $(git rev-parse A:y/b) &&\n+\ttest $(git rev-parse HEAD:w/c) = $(git rev-parse A:x/c) &&\n+\ttest $(git rev-parse HEAD:u/d) = $(git rev-parse A:w/d) &&\n+\ttest $(git rev-parse HEAD:u/e) = $(git rev-parse A:v/e) &&\n+\ttest $(git rev-parse HEAD:u/f) = $(git rev-parse B:u/f)\n+'\n+\n+# Testcase 9e, N-to-1 whammo\n+# (Related to testcase 9c...and 1c and 7e)\n+# Commit A: dir1/{a,b}, dir2/{d,e}, dir3/{g,h}, dirN/{j,k}\n+# Commit B: dir1/{a,b,c,yo}, dir2/{d,e,f,yo}, dir3/{g,h,i,yo}, dirN/{j,k,l,yo}\n+# Commit C: combined/{a,b,d,e,g,h,j,k}\n+# Expected: combined/{a,b,c,d,e,f,g,h,i,j,k,l}, CONFLICT(Nto1) warnings,\n+# dir1/yo, dir2/yo, dir3/yo, dirN/yo\n+\n+test_expect_success '9e-setup: N-to-1 whammo' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir dir1 dir2 dir3 dirN &&\n+\techo a >dir1/a &&\n+\techo b >dir1/b &&\n+\techo d >dir2/d &&\n+\techo e >dir2/e &&\n+\techo g >dir3/g &&\n+\techo h >dir3/h &&\n+\techo j >dirN/j &&\n+\techo k >dirN/k &&\n+\tgit add dir* &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\techo c >dir1/c &&\n+\techo yo >dir1/yo &&\n+\techo f >dir2/f &&\n+\techo yo >dir2/yo &&\n+\techo i >dir3/i &&\n+\techo yo >dir3/yo &&\n+\techo l >dirN/l &&\n+\techo yo >dirN/yo &&\n+\tgit add dir* &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv dir1 combined &&\n+\tgit mv dir2/* combined/ &&\n+\tgit mv dir3/* combined/ &&\n+\tgit mv dirN/* combined/ &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '9e-check: N-to-1 whammo' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out &&\n+\ttest_i18ngrep \"CONFLICT (implicit dir rename): Cannot map more than one path to combined/yo\" out >error_line &&\n+\tgrep -q dir1/yo error_line &&\n+\tgrep -q dir2/yo error_line &&\n+\tgrep -q dir3/yo error_line &&\n+\tgrep -q dirN/yo error_line &&\n+\n+\ttest 16 -eq $(git ls-files -s | wc -l) &&\n+\ttest 0 -eq $(git ls-files -u | wc -l) &&\n+\ttest 2 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:combined/a) = $(git rev-parse A:dir1/a) &&\n+\ttest $(git rev-parse :0:combined/b) = $(git rev-parse A:dir1/b) &&\n+\ttest $(git rev-parse :0:combined/c) = $(git rev-parse B:dir1/c) &&\n+\ttest $(git rev-parse :0:combined/d) = $(git rev-parse A:dir2/d) &&\n+\ttest $(git rev-parse :0:combined/e) = $(git rev-parse A:dir2/e) &&\n+\ttest $(git rev-parse :0:combined/f) = $(git rev-parse B:dir2/f) &&\n+\ttest $(git rev-parse :0:combined/g) = $(git rev-parse A:dir3/g) &&\n+\ttest $(git rev-parse :0:combined/h) = $(git rev-parse A:dir3/h) &&\n+\ttest $(git rev-parse :0:combined/i) = $(git rev-parse B:dir3/i) &&\n+\ttest $(git rev-parse :0:combined/j) = $(git rev-parse A:dirN/j) &&\n+\ttest $(git rev-parse :0:combined/k) = $(git rev-parse A:dirN/k) &&\n+\ttest $(git rev-parse :0:combined/l) = $(git rev-parse B:dirN/l) &&\n+\n+\ttest $(git rev-parse :0:dir1/yo) = $(git rev-parse B:dir1/yo) &&\n+\ttest $(git rev-parse :0:dir2/yo) = $(git rev-parse B:dir2/yo) &&\n+\ttest $(git rev-parse :0:dir3/yo) = $(git rev-parse B:dir3/yo) &&\n+\ttest $(git rev-parse :0:dirN/yo) = $(git rev-parse B:dirN/yo)\n+'\n+\n+# Testcase 9f, Renamed directory that only contained immediate subdirs\n+# (Related to testcases 1e & 9g)\n+# Commit A: goal/{a,b}/$more_files\n+# Commit B: priority/{a,b}/$more_files\n+# Commit C: goal/{a,b}/$more_files, goal/c\n+# Expected: priority/{a,b}/$more_files, priority/c\n+\n+test_expect_success '9f-setup: Renamed directory that only contained immediate subdirs' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir -p goal/a &&\n+\tmkdir -p goal/b &&\n+\techo foo >goal/a/foo &&\n+\techo bar >goal/b/bar &&\n+\techo baz >goal/b/baz &&\n+\tgit add goal &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv goal/ priority &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\techo c >goal/c &&\n+\tgit add goal/c &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '9f-check: Renamed directory that only contained immediate subdirs' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 4 -eq $(git ls-files -s | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:priority/a/foo) = $(git rev-parse A:goal/a/foo) &&\n+\ttest $(git rev-parse HEAD:priority/b/bar) = $(git rev-parse A:goal/b/bar) &&\n+\ttest $(git rev-parse HEAD:priority/b/baz) = $(git rev-parse A:goal/b/baz) &&\n+\ttest $(git rev-parse HEAD:priority/c) = $(git rev-parse C:goal/c) &&\n+\ttest_must_fail git rev-parse HEAD:goal/c\n+'\n+\n+# Testcase 9g, Renamed directory that only contained immediate subdirs, immediate subdirs renamed\n+# (Related to testcases 1e & 9f)\n+# Commit A: goal/{a,b}/$more_files\n+# Commit B: priority/{alpha,bravo}/$more_files\n+# Commit C: goal/{a,b}/$more_files, goal/c\n+# Expected: priority/{alpha,bravo}/$more_files, priority/c\n+\n+test_expect_success '9g-setup: Renamed directory that only contained immediate subdirs, immediate subdirs renamed' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir -p goal/a &&\n+\tmkdir -p goal/b &&\n+\techo foo >goal/a/foo &&\n+\techo bar >goal/b/bar &&\n+\techo baz >goal/b/baz &&\n+\tgit add goal &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tmkdir priority &&\n+\tgit mv goal/a/ priority/alpha &&\n+\tgit mv goal/b/ priority/beta &&\n+\trmdir goal/ &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\techo c >goal/c &&\n+\tgit add goal/c &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '9g-check: Renamed directory that only contained immediate subdirs, immediate subdirs renamed' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 4 -eq $(git ls-files -s | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:priority/alpha/foo) = $(git rev-parse A:goal/a/foo) &&\n+\ttest $(git rev-parse HEAD:priority/beta/bar) = $(git rev-parse A:goal/b/bar) &&\n+\ttest $(git rev-parse HEAD:priority/beta/baz) = $(git rev-parse A:goal/b/baz) &&\n+\ttest $(git rev-parse HEAD:priority/c) = $(git rev-parse C:goal/c) &&\n+\ttest_must_fail git rev-parse HEAD:goal/c\n+'\n+\n+###########################################################################\n+# Rules suggested by section 9:\n+#\n+# If the other side of history did a directory rename to a path that your\n+# side renamed away, then ignore that particular rename from the other\n+# side of history for any implicit directory renames.\n+###########################################################################\n+\n test_done\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332227","messageId":"20171110190550.27059-11-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 10/30] directory rename detection: more involved edge/corner testcases","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:30Z","receivedAt":"2017-11-10T19:06:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n t/t6043-merge-rename-directories.sh | 347 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 347 insertions(+)\n\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex 157299105f..115d0d2622 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -1336,4 +1336,351 @@ test_expect_success '6e-check: Add/add from one side' '\n \ttest $(git rev-parse HEAD:z/d) = $(git rev-parse C:z/d)\n '\n \n+\n+###########################################################################\n+# SECTION 7: More involved Edge/Corner cases\n+#\n+# The ruleset we have generated in the above sections seems to provide\n+# well-defined merges. But can we find edge/corner cases that either (a)\n+# are harder for users to understand, or (b) have a resolution that is\n+# non-intuitive or suboptimal?\n+#\n+# The testcases in this section dive into cases that I've tried to craft in\n+# a way to find some that might be surprising to users or difficult for\n+# them to understand (the next section will look at non-intuitive or\n+# suboptimal merge results). Some of the testcases are similar to ones\n+# from past sections, but have been simplified to try to highlight error\n+# messages using a \"modified\" path (due to the directory rename). Are\n+# users okay with these?\n+#\n+# In my opinion, testcases that are difficult to understand from this\n+# section is due to difficulty in the testcase rather than the directory\n+# renaming (similar to how t6042 and t6036 have difficult resolutions due\n+# to the problem setup itself being complex). And I don't think the\n+# error messages are a problem.\n+#\n+# On the other hand, the testcases in section 8 worry me slightly more...\n+###########################################################################\n+\n+# Testcase 7a, rename-dir vs. rename-dir (NOT split evenly) PLUS add-other-file\n+# Commit A: z/{b,c}\n+# Commit B: y/{b,c}\n+# Commit C: w/b, x/c, z/d\n+# Expected: y/d, CONFLICT(rename/rename for both z/b and z/c)\n+# NOTE: There's a rename of z/ here, y/ has more renames, so z/d -> y/d.\n+\n+test_expect_success '7a-setup: rename-dir vs. rename-dir (NOT split evenly) PLUS add-other-file' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z y &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tmkdir w &&\n+\tmkdir x &&\n+\tgit mv z/b w/ &&\n+\tgit mv z/c x/ &&\n+\techo d > z/d &&\n+\tgit add z/d &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '7a-check: rename-dir vs. rename-dir (NOT split evenly) PLUS add-other-file' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out &&\n+\ttest_i18ngrep \"CONFLICT (rename/rename).*z/b.*y/b.*w/b\" out &&\n+\ttest_i18ngrep \"CONFLICT (rename/rename).*z/c.*y/c.*x/c\" out &&\n+\n+\ttest 7 -eq $(git ls-files -s | wc -l) &&\n+\ttest 6 -eq $(git ls-files -u | wc -l) &&\n+\ttest 1 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/d) = $(git rev-parse C:z/d) &&\n+\n+\ttest $(git rev-parse :1:z/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :2:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :3:w/b) = $(git rev-parse A:z/b) &&\n+\n+\ttest $(git rev-parse :1:z/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse :2:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse :3:x/c) = $(git rev-parse A:z/c) &&\n+\n+\ttest $(git hash-object y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git hash-object w/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git hash-object y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git hash-object x/c) = $(git rev-parse A:z/c)\n+'\n+\n+# Testcase 7b, rename/rename(2to1), but only due to transitive rename\n+# (Related to testcase 1d)\n+# Commit A: z/{b,c}, x/d_1, w/d_2\n+# Commit B: y/{b,c,d_2}, x/d_1\n+# Commit C: z/{b,c,d_1}, w/d_2\n+# Expected: y/{b,c}, CONFLICT(rename/rename(2to1): x/d_1, w/d_2 -> y_d)\n+\n+test_expect_success '7b-setup: rename/rename(2to1), but only due to transitive rename' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\tmkdir x &&\n+\tmkdir w &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\techo d1 > x/d &&\n+\techo d2 > w/d &&\n+\tgit add z x w &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z y &&\n+\tgit mv w/d y/ &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv x/d z/ &&\n+\trmdir x &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '7b-check: rename/rename(2to1), but only due to transitive rename' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out &&\n+\ttest_i18ngrep \"CONFLICT (rename/rename)\" out &&\n+\n+\ttest 4 -eq $(git ls-files -s | wc -l) &&\n+\ttest 2 -eq $(git ls-files -u | wc -l) &&\n+\ttest 3 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:y/c) = $(git rev-parse A:z/c) &&\n+\n+\ttest $(git rev-parse :2:y/d) = $(git rev-parse A:w/d) &&\n+\ttest $(git rev-parse :3:y/d) = $(git rev-parse A:x/d) &&\n+\n+\ttest ! -f y/d &&\n+\ttest -f y/d~HEAD &&\n+\ttest -f y/d~C^0 &&\n+\n+\ttest $(git hash-object y/d~HEAD) = $(git rev-parse A:w/d) &&\n+\ttest $(git hash-object y/d~C^0) = $(git rev-parse A:x/d)\n+'\n+\n+# Testcase 7c, rename/rename(1to...2or3); transitive rename may add complexity\n+# (Related to testcases 3b and 5c)\n+# Commit A: z/{b,c}, x/d\n+# Commit B: y/{b,c}, w/d\n+# Commit C: z/{b,c,d}\n+# Expected: y/{b,c}, CONFLICT(x/d -> w/d vs. y/d)\n+# NOTE: z/ was renamed to y/ so we do not want to report\n+# either CONFLICT(x/d -> w/d vs. z/d)\n+# or CONFLiCT x/d -> w/d vs. y/d vs. z/d)\n+\n+test_expect_success '7c-setup: rename/rename(1to...2or3); transitive rename may add complexity' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tmkdir x &&\n+\techo d >x/d &&\n+\tgit add z x &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z y &&\n+\tgit mv x w &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv x/d z/ &&\n+\trmdir x &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '7c-check: rename/rename(1to...2or3); transitive rename may add complexity' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out &&\n+\ttest_i18ngrep \"CONFLICT (rename/rename).*x/d.*w/d.*y/d\" out &&\n+\n+\ttest 5 -eq $(git ls-files -s | wc -l) &&\n+\ttest 3 -eq $(git ls-files -u | wc -l) &&\n+\ttest 1 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:y/c) = $(git rev-parse A:z/c) &&\n+\n+\ttest $(git rev-parse :1:x/d) = $(git rev-parse A:x/d) &&\n+\ttest $(git rev-parse :2:w/d) = $(git rev-parse A:x/d) &&\n+\ttest $(git rev-parse :3:y/d) = $(git rev-parse A:x/d)\n+'\n+\n+# Testcase 7d, transitive rename involved in rename/delete; how is it reported?\n+# (Related somewhat to testcases 5b and 8d)\n+# Commit A: z/{b,c}, x/d\n+# Commit B: y/{b,c}\n+# Commit C: z/{b,c,d}\n+# Expected: y/{b,c}, CONFLICT(delete x/d vs rename to y/d)\n+# NOTE: z->y so NOT CONFLICT(delete x/d vs rename to z/d)\n+\n+test_expect_success '7d-setup: transitive rename involved in rename/delete; how is it reported?' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tmkdir x &&\n+\techo d >x/d &&\n+\tgit add z x &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z y &&\n+\tgit rm -rf x &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv x/d z/ &&\n+\trmdir x &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '7d-check: transitive rename involved in rename/delete; how is it reported?' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out &&\n+\ttest_i18ngrep \"CONFLICT (rename/delete).*x/d.*y/d\" out &&\n+\n+\ttest 3 -eq $(git ls-files -s | wc -l) &&\n+\ttest 1 -eq $(git ls-files -u | wc -l) &&\n+\ttest 1 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse :3:y/d) = $(git rev-parse A:x/d)\n+'\n+\n+# Testcase 7e, transitive rename in rename/delete AND dirs in the way\n+# (Very similar to 'both rename source and destination involved in D/F conflict' from t6022-merge-rename.sh)\n+# (Also related to testcases 9c and 9d)\n+# Commit A: z/{b,c}, x/d_1\n+# Commit B: y/{b,c,d/g}, x/d/f\n+# Commit C: z/{b,c,d_1}\n+# Expected: rename/delete(x/d_1->y/d_1 vs. None) + D/F conflict on y/d\n+# y/{b,c,d/g}, y/d_1~C^0, x/d/f\n+# NOTE: x/d/f may be slightly confusing here. x/d_1 -> z/d_1 implies\n+# there is a directory rename from x/ -> z/, performed by commit C.\n+# However, on the side of commit B, it renamed z/ -> y/, thus\n+# making a rename from x/ -> z/ when it was getting rid of z/ seems\n+# non-sensical. Further, putting x/d/f into y/d/f also doesn't\n+# make a lot of sense because commit B did the renaming of z to y\n+# and it created x/d/f, and it clearly made these things separate,\n+# so it doesn't make much sense to push these together.\n+\n+test_expect_success '7e-setup: transitive rename in rename/delete AND dirs in the way' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tmkdir x &&\n+\techo d1 >x/d &&\n+\tgit add z x &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z y &&\n+\tgit rm x/d &&\n+\tmkdir -p x/d &&\n+\tmkdir -p y/d &&\n+\techo f >x/d/f &&\n+\techo g >y/d/g &&\n+\tgit add x/d/f y/d/g &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv x/d z/ &&\n+\trmdir x &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '7e-check: transitive rename in rename/delete AND dirs in the way' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out &&\n+\ttest_i18ngrep \"CONFLICT (rename/delete).*x/d.*y/d\" out &&\n+\n+\ttest 5 -eq $(git ls-files -s | wc -l) &&\n+\ttest 1 -eq $(git ls-files -u | wc -l) &&\n+\ttest 2 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:x/d/f) = $(git rev-parse B:x/d/f) &&\n+\ttest $(git rev-parse :0:y/d/g) = $(git rev-parse B:y/d/g) &&\n+\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse :3:y/d) = $(git rev-parse A:x/d) &&\n+\n+\ttest $(git hash-object y/d~C^0) = $(git rev-parse A:x/d)\n+'\n+\n test_done\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332228","messageId":"20171110190550.27059-15-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 14/30] directory rename detection: tests for handling overwriting dirty files","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:34Z","receivedAt":"2017-11-10T19:06:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n t/t6043-merge-rename-directories.sh | 401 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 401 insertions(+)\n\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex 7af8962512..4066b08767 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -2873,4 +2873,405 @@ test_expect_failure '10e-check: Does git complain about untracked file that is n\n \ttest \"random\" = \"$(cat z/c)\"\n '\n \n+###########################################################################\n+# SECTION 11: Handling dirty (not up-to-date) files\n+#\n+# unpack_trees(), upon which the recursive merge algorithm is based, aborts\n+# the operation if untracked or dirty files would be deleted or overwritten\n+# by the merge. Unfortunately, unpack_trees() does not understand renames,\n+# and if it doesn't abort, then it muddies up the working directory before\n+# we even get to the point of detecting renames, so we need some special\n+# handling. This was true even of normal renames, but there are additional\n+# codepaths that need special handling with directory renames. Add\n+# testcases for both renamed-by-directory-rename-detection and standard\n+# rename cases.\n+###########################################################################\n+\n+# Testcase 11a, Avoid losing dirty contents with simple rename\n+# Commit A: z/{a,b_v1},\n+# Commit B: z/{a,c_v1}, and z/c_v1 has uncommitted mods\n+# Commit C: z/{a,b_v2}\n+# Expected: ERROR_MSG(Refusing to lose dirty file at z/c) +\n+# z/a, staged version of z/c has sha1sum matching C:z/b_v2,\n+# z/c~HEAD with contents of C:z/b_v2,\n+# z/c with uncommitted mods on top of B:z/c_v1\n+\n+test_expect_success '11a-setup: Avoid losing dirty contents with simple rename' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo a >z/a &&\n+\ttest_seq 1 10 >z/b &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z/b z/c &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\techo 11 >>z/b &&\n+\tgit add z/b &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '11a-check: Avoid losing dirty contents with simple rename' '\n+\tgit checkout B^0 &&\n+\techo stuff >>z/c &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out 2>err &&\n+\ttest_i18ngrep \"Refusing to lose dirty file at z/c\" out &&\n+\n+\ttest_seq 1 10 >expected &&\n+\techo stuff >>expected &&\n+\n+\ttest 2 -eq $(git ls-files -s | wc -l) &&\n+\ttest 1 -eq $(git ls-files -u | wc -l) &&\n+\ttest 4 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:z/a) = $(git rev-parse A:z/a) &&\n+\ttest $(git rev-parse :2:z/c) = $(git rev-parse C:z/b) &&\n+\n+\ttest \"$(git hash-object z/c~HEAD)\" = $(git rev-parse C:z/b) &&\n+\ttest_cmp expected z/c\n+'\n+\n+# Testcase 11b, Avoid losing dirty file involved in directory rename\n+# Commit A: z/a, x/{b,c_v1}\n+# Commit B: z/{a,c_v1}, x/b, and z/c_v1 has uncommitted mods\n+# Commit C: y/a, x/{b,c_v2}\n+# Expected: y/{a,c_v2}, x/b, z/c_v1 with uncommited mods untracked,\n+# ERROR_MSG(Refusing to lose dirty file at z/c)\n+\n+\n+test_expect_success '11b-setup: Avoid losing dirty file involved in directory rename' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z x &&\n+\techo a >z/a &&\n+\techo b >x/b &&\n+\ttest_seq 1 10 >x/c &&\n+\tgit add z x &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv x/c z/c &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv z y &&\n+\techo 11 >>x/c &&\n+\tgit add x/c &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '11b-check: Avoid losing dirty file involved in directory rename' '\n+\tgit checkout B^0 &&\n+\techo stuff >>z/c &&\n+\n+\tgit merge -s recursive C^0 >out 2>err &&\n+\ttest_i18ngrep \"Refusing to lose dirty file at z/c\" out &&\n+\n+\tgrep -q stuff */* &&\n+\ttest_seq 1 10 >expected &&\n+\techo stuff >>expected &&\n+\n+\ttest 3 -eq $(git ls-files -s | wc -l) &&\n+\ttest 0 -eq $(git ls-files -u | wc -l) &&\n+\ttest 0 -eq $(git ls-files -m | wc -l) &&\n+\ttest 4 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:x/b) = $(git rev-parse A:x/b) &&\n+\ttest $(git rev-parse :0:y/a) = $(git rev-parse A:z/a) &&\n+\ttest $(git rev-parse :0:y/c) = $(git rev-parse C:x/c) &&\n+\n+\ttest \"$(git hash-object y/c)\" = $(git rev-parse C:x/c) &&\n+\ttest_cmp expected z/c\n+'\n+\n+# Testcase 11c, Avoid losing not-up-to-date with rename + D/F conflict\n+# Commit A: y/a, x/{b,c_v1}\n+# Commit B: y/{a,c_v1}, x/b, and y/c_v1 has uncommitted mods\n+# Commit C: y/{a,c/d}, x/{b,c_v2}\n+# Expected: Abort_msg(\"following files would be overwritten by merge\") +\n+# y/c left untouched (still has uncommitted mods)\n+\n+test_expect_success '11c-setup: Avoid losing not-uptodate with rename + D/F conflict' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir y x &&\n+\techo a >y/a &&\n+\techo b >x/b &&\n+\ttest_seq 1 10 >x/c &&\n+\tgit add y x &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv x/c y/c &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tmkdir y/c &&\n+\techo d >y/c/d &&\n+\techo 11 >>x/c &&\n+\tgit add x/c y/c/d &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_success '11c-check: Avoid losing not-uptodate with rename + D/F conflict' '\n+\tgit checkout B^0 &&\n+\techo stuff >>y/c &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out 2>err &&\n+\ttest_i18ngrep \"following files would be overwritten by merge\" err &&\n+\n+\tgrep -q stuff */* &&\n+\ttest_seq 1 10 >expected &&\n+\techo stuff >>expected &&\n+\n+\ttest 3 -eq $(git ls-files -s | wc -l) &&\n+\ttest 0 -eq $(git ls-files -u | wc -l) &&\n+\ttest 1 -eq $(git ls-files -m | wc -l) &&\n+\ttest 3 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD) = $(git rev-parse B) &&\n+\ttest_cmp expected y/c\n+'\n+\n+# Testcase 11d, Avoid losing not-up-to-date with rename + D/F conflict\n+# Commit A: z/a, x/{b,c_v1}\n+# Commit B: z/{a,c_v1}, x/b, and z/c_v1 has uncommitted mods\n+# Commit C: y/{a,c/d}, x/{b,c_v2}\n+# Expected: D/F: y/c_v2 vs y/c/d) +\n+# Warning_Msg(\"Refusing to lose dirty file at z/c) +\n+# y/{a,c~HEAD,c/d}, x/b, now-untracked z/c_v1 with uncommited mods\n+\n+test_expect_success '11d-setup: Avoid losing not-uptodate with rename + D/F conflict' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z x &&\n+\techo a >z/a &&\n+\techo b >x/b &&\n+\ttest_seq 1 10 >x/c &&\n+\tgit add z x &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv x/c z/c &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv z y &&\n+\tmkdir y/c &&\n+\techo d >y/c/d &&\n+\techo 11 >>x/c &&\n+\tgit add x/c y/c/d &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '11d-check: Avoid losing not-uptodate with rename + D/F conflict' '\n+\tgit checkout B^0 &&\n+\techo stuff >>z/c &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out 2>err &&\n+\ttest_i18ngrep \"Refusing to lose dirty file at z/c\" out &&\n+\n+\tgrep -q stuff */* &&\n+\ttest_seq 1 10 >expected &&\n+\techo stuff >>expected &&\n+\n+\ttest 4 -eq $(git ls-files -s | wc -l) &&\n+\ttest 1 -eq $(git ls-files -u | wc -l) &&\n+\ttest 5 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:x/b) = $(git rev-parse A:x/b) &&\n+\ttest $(git rev-parse :0:y/a) = $(git rev-parse A:z/a) &&\n+\ttest $(git rev-parse :0:y/c/d) = $(git rev-parse C:y/c/d) &&\n+\ttest $(git rev-parse :3:y/c) = $(git rev-parse C:x/c) &&\n+\n+\ttest \"$(git hash-object y/c~HEAD)\" = $(git rev-parse C:x/c) &&\n+\ttest_cmp expected z/c\n+'\n+\n+# Testcase 11e, Avoid deleting not-up-to-date with dir rename/rename(1to2)/add\n+# Commit A: z/{a,b}, x/{c_1,d}\n+# Commit B: y/{a,b,c_2}, x/d, w/c_1, and y/c_2 has uncommitted mods\n+# Commit C: z/{a,b,c_1}, x/d\n+# Expected: Failed Merge; y/{a,b} + x/d +\n+# CONFLICT(rename/rename) x/c_1 -> w/c_1 vs y/c_1 +\n+# ERROR_MSG(Refusing to lose dirty file at y/c)\n+# y/c~C^0 has A:x/c_1 contents\n+# y/c~HEAD has B:y/c_2 contents\n+# y/c has dirty file from before merge\n+\n+test_expect_success '11e-setup: Avoid deleting not-uptodate with dir rename/rename(1to2)/add' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z x &&\n+\techo a >z/a &&\n+\techo b >z/b &&\n+\techo c >x/c &&\n+\techo d >x/d &&\n+\tgit add z x &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z/ y/ &&\n+\techo different >y/c &&\n+\tmkdir w &&\n+\tgit mv x/c w/ &&\n+\tgit add y/c &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv x/c z/ &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '11e-check: Avoid deleting not-uptodate with dir rename/rename(1to2)/add' '\n+\tgit checkout B^0 &&\n+\techo mods >>y/c &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out 2>err &&\n+\ttest_i18ngrep \"CONFLICT (rename/rename)\" out &&\n+\ttest_i18ngrep \"Refusing to lose dirty file at y/c\" out &&\n+\n+\ttest 7 -eq $(git ls-files -s | wc -l) &&\n+\ttest 4 -eq $(git ls-files -u | wc -l) &&\n+\ttest 4 -eq $(git ls-files -o | wc -l) &&\n+\n+\techo different >expected &&\n+\techo mods >>expected &&\n+\n+\ttest $(git rev-parse :0:y/a) = $(git rev-parse A:z/a) &&\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:x/d) = $(git rev-parse A:x/d) &&\n+\n+\ttest $(git rev-parse :1:x/c) = $(git rev-parse A:x/c) &&\n+\ttest $(git rev-parse :2:w/c) = $(git rev-parse A:x/c) &&\n+\ttest $(git rev-parse :2:y/c) = $(git rev-parse B:y/c) &&\n+\ttest $(git rev-parse :3:y/c) = $(git rev-parse A:x/c) &&\n+\n+\ttest \"$(git hash-object y/c~C^0)\" = $(git rev-parse A:x/c) &&\n+\ttest \"$(git hash-object y/c~HEAD)\" = $(git rev-parse B:y/c) &&\n+\ttest_cmp expected y/c\n+'\n+\n+# Testcase 11f, Avoid deleting not-up-to-date w/ dir rename/rename(2to1)\n+# Commit A: z/{a,b}, x/{c_1,d_2}\n+# Commit B: y/{a,b,wham_1}, x/d_2, except y/wham has uncommitted mods\n+# Commit C: z/{a,b,wham_2}, x/c_1\n+# Expected: Failed Merge; y/{a,b} + untracked y/{wham~C^0,wham~C^HEAD} +\n+# y/wham with dirty changes from before merge +\n+# CONFLICT(rename/rename) x/c vs x/d -> y/wham\n+# ERROR_MSG(Refusing to lose dirty file at y/wham)\n+\n+test_expect_success '11f-setup: Avoid deleting not-uptodate with dir rename/rename(2to1)' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z x &&\n+\techo a >z/a &&\n+\techo b >z/b &&\n+\ttest_seq 1 10 >x/c &&\n+\techo d >x/d &&\n+\tgit add z x &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z/ y/ &&\n+\tgit mv x/c y/wham &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv x/d z/wham &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '11f-check: Avoid deleting not-uptodate with dir rename/rename(2to1)' '\n+\tgit checkout B^0 &&\n+\techo important >>y/wham &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out 2>err &&\n+\ttest_i18ngrep \"CONFLICT (rename/rename)\" out &&\n+\ttest_i18ngrep \"Refusing to lose dirty file at y/wham\" out &&\n+\n+\ttest 4 -eq $(git ls-files -s | wc -l) &&\n+\ttest 2 -eq $(git ls-files -u | wc -l) &&\n+\ttest 4 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest_seq 1 10 >expected &&\n+\techo important >>expected &&\n+\n+\ttest $(git rev-parse :0:y/a) = $(git rev-parse A:z/a) &&\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\n+\ttest_must_fail git rev-parse :1:y/wham &&\n+\ttest $(git rev-parse :2:y/wham) = $(git rev-parse A:x/c) &&\n+\ttest $(git rev-parse :3:y/wham) = $(git rev-parse A:x/d) &&\n+\n+\ttest_cmp expected y/wham &&\n+\ttest $(git hash-object y/wham~C^0) = $(git rev-parse A:x/d) &&\n+\ttest $(git hash-object y/wham~HEAD) = $(git rev-parse A:x/c)\n+'\n+\n test_done\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332229","messageId":"20171110190550.27059-16-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 15/30] merge-recursive: Move the get_renames() function","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:35Z","receivedAt":"2017-11-10T19:06:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"I want to re-use some other functions in the file without moving those other\nfunctions or dealing with a handful of annoying split function declarations\nand definitions.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 138 +++++++++++++++++++++++++++---------------------------\n 1 file changed, 69 insertions(+), 69 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 3526c8d0b8..382016508b 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -540,75 +540,6 @@ struct rename {\n \tunsigned processed:1;\n };\n \n-/*\n- * Get information of all renames which occurred between 'o_tree' and\n- * 'tree'. We need the three trees in the merge ('o_tree', 'a_tree' and\n- * 'b_tree') to be able to associate the correct cache entries with\n- * the rename information. 'tree' is always equal to either a_tree or b_tree.\n- */\n-static struct string_list *get_renames(struct merge_options *o,\n-\t\t\t\t struct tree *tree,\n-\t\t\t\t struct tree *o_tree,\n-\t\t\t\t struct tree *a_tree,\n-\t\t\t\t struct tree *b_tree,\n-\t\t\t\t struct string_list *entries)\n-{\n-\tint i;\n-\tstruct string_list *renames;\n-\tstruct diff_options opts;\n-\n-\trenames = xcalloc(1, sizeof(struct string_list));\n-\tif (!o->detect_rename)\n-\t\treturn renames;\n-\n-\tdiff_setup(&opts);\n-\tDIFF_OPT_SET(&opts, RECURSIVE);\n-\tDIFF_OPT_CLR(&opts, RENAME_EMPTY);\n-\topts.detect_rename = DIFF_DETECT_RENAME;\n-\topts.rename_limit = o->merge_rename_limit >= 0 ? o->merge_rename_limit :\n-\t\t\t o->diff_rename_limit >= 0 ? o->diff_rename_limit :\n-\t\t\t 1000;\n-\topts.rename_score = o->rename_score;\n-\topts.show_rename_progress = o->show_rename_progress;\n-\topts.output_format = DIFF_FORMAT_NO_OUTPUT;\n-\tdiff_setup_done(&opts);\n-\tdiff_tree_oid(&o_tree->object.oid, &tree->object.oid, \"\", &opts);\n-\tdiffcore_std(&opts);\n-\tif (opts.needed_rename_limit > o->needed_rename_limit)\n-\t\to->needed_rename_limit = opts.needed_rename_limit;\n-\tfor (i = 0; i < diff_queued_diff.nr; ++i) {\n-\t\tstruct string_list_item *item;\n-\t\tstruct rename *re;\n-\t\tstruct diff_filepair *pair = diff_queued_diff.queue[i];\n-\t\tif (pair->status != 'R') {\n-\t\t\tdiff_free_filepair(pair);\n-\t\t\tcontinue;\n-\t\t}\n-\t\tre = xmalloc(sizeof(*re));\n-\t\tre->processed = 0;\n-\t\tre->pair = pair;\n-\t\titem = string_list_lookup(entries, re->pair->one->path);\n-\t\tif (!item)\n-\t\t\tre->src_entry = insert_stage_data(re->pair->one->path,\n-\t\t\t\t\to_tree, a_tree, b_tree, entries);\n-\t\telse\n-\t\t\tre->src_entry = item->util;\n-\n-\t\titem = string_list_lookup(entries, re->pair->two->path);\n-\t\tif (!item)\n-\t\t\tre->dst_entry = insert_stage_data(re->pair->two->path,\n-\t\t\t\t\to_tree, a_tree, b_tree, entries);\n-\t\telse\n-\t\t\tre->dst_entry = item->util;\n-\t\titem = string_list_insert(renames, pair->one->path);\n-\t\titem->util = re;\n-\t}\n-\topts.output_format = DIFF_FORMAT_NO_OUTPUT;\n-\tdiff_queued_diff.nr = 0;\n-\tdiff_flush(&opts);\n-\treturn renames;\n-}\n-\n static int update_stages(struct merge_options *opt, const char *path,\n \t\t\t const struct diff_filespec *o,\n \t\t\t const struct diff_filespec *a,\n@@ -1383,6 +1314,75 @@ static int conflict_rename_rename_2to1(struct merge_options *o,\n \treturn ret;\n }\n \n+/*\n+ * Get information of all renames which occurred between 'o_tree' and\n+ * 'tree'. We need the three trees in the merge ('o_tree', 'a_tree' and\n+ * 'b_tree') to be able to associate the correct cache entries with\n+ * the rename information. 'tree' is always equal to either a_tree or b_tree.\n+ */\n+static struct string_list *get_renames(struct merge_options *o,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct tree *o_tree,\n+\t\t\t\t struct tree *a_tree,\n+\t\t\t\t struct tree *b_tree,\n+\t\t\t\t struct string_list *entries)\n+{\n+\tint i;\n+\tstruct string_list *renames;\n+\tstruct diff_options opts;\n+\n+\trenames = xcalloc(1, sizeof(struct string_list));\n+\tif (!o->detect_rename)\n+\t\treturn renames;\n+\n+\tdiff_setup(&opts);\n+\tDIFF_OPT_SET(&opts, RECURSIVE);\n+\tDIFF_OPT_CLR(&opts, RENAME_EMPTY);\n+\topts.detect_rename = DIFF_DETECT_RENAME;\n+\topts.rename_limit = o->merge_rename_limit >= 0 ? o->merge_rename_limit :\n+\t\t\t o->diff_rename_limit >= 0 ? o->diff_rename_limit :\n+\t\t\t 1000;\n+\topts.rename_score = o->rename_score;\n+\topts.show_rename_progress = o->show_rename_progress;\n+\topts.output_format = DIFF_FORMAT_NO_OUTPUT;\n+\tdiff_setup_done(&opts);\n+\tdiff_tree_oid(&o_tree->object.oid, &tree->object.oid, \"\", &opts);\n+\tdiffcore_std(&opts);\n+\tif (opts.needed_rename_limit > o->needed_rename_limit)\n+\t\to->needed_rename_limit = opts.needed_rename_limit;\n+\tfor (i = 0; i < diff_queued_diff.nr; ++i) {\n+\t\tstruct string_list_item *item;\n+\t\tstruct rename *re;\n+\t\tstruct diff_filepair *pair = diff_queued_diff.queue[i];\n+\t\tif (pair->status != 'R') {\n+\t\t\tdiff_free_filepair(pair);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tre = xmalloc(sizeof(*re));\n+\t\tre->processed = 0;\n+\t\tre->pair = pair;\n+\t\titem = string_list_lookup(entries, re->pair->one->path);\n+\t\tif (!item)\n+\t\t\tre->src_entry = insert_stage_data(re->pair->one->path,\n+\t\t\t\t\to_tree, a_tree, b_tree, entries);\n+\t\telse\n+\t\t\tre->src_entry = item->util;\n+\n+\t\titem = string_list_lookup(entries, re->pair->two->path);\n+\t\tif (!item)\n+\t\t\tre->dst_entry = insert_stage_data(re->pair->two->path,\n+\t\t\t\t\to_tree, a_tree, b_tree, entries);\n+\t\telse\n+\t\t\tre->dst_entry = item->util;\n+\t\titem = string_list_insert(renames, pair->one->path);\n+\t\titem->util = re;\n+\t}\n+\topts.output_format = DIFF_FORMAT_NO_OUTPUT;\n+\tdiff_queued_diff.nr = 0;\n+\tdiff_flush(&opts);\n+\treturn renames;\n+}\n+\n static int process_renames(struct merge_options *o,\n \t\t\t struct string_list *a_renames,\n \t\t\t struct string_list *b_renames)\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332230","messageId":"20171110190550.27059-23-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 22/30] merge-recursive: Check for directory level conflicts","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:42Z","receivedAt":"2017-11-10T19:07:01Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Before trying to apply directory renames to paths within the given\ndirectories, we want to make sure that there aren't conflicts at the\ndirectory level. There will be additional checks at the individual\nfile level too, which will be added later.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 112 insertions(+)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex b5770d3d7f..3633be0123 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -1376,6 +1376,15 @@ static struct diff_queue_struct *get_diffpairs(struct merge_options *o,\n \treturn ret;\n }\n \n+static int tree_has_path(struct tree *tree, const char *path)\n+{\n+\tunsigned char hashy[20];\n+\tunsigned mode_o;\n+\n+\treturn !get_tree_entry(tree->object.oid.hash, path,\n+\t\t\t hashy, &mode_o);\n+}\n+\n static void get_renamed_dir_portion(const char *old_path, const char *new_path,\n \t\t\t\t char **old_dir, char **new_dir) {\n \t*old_dir = NULL;\n@@ -1425,6 +1434,105 @@ static void get_renamed_dir_portion(const char *old_path, const char *new_path,\n \t}\n }\n \n+/*\n+ * There are a couple things we want to do at the directory level:\n+ * 1. Check for both sides renaming to the same thing, in order to avoid\n+ * implicit renaming of files that should be left in place. (See\n+ * testcase 6b in t6043 for details.)\n+ * 2. Prune directory renames if there are still files left in the\n+ * the original directory. These represent a partial directory rename,\n+ * i.e. a rename where only some of the files within the directory\n+ * were renamed elsewhere. (Technically, this could be done earlier\n+ * in get_directory_renames(), except that would prevent us from\n+ * doing the previous check and thus failing testcase 6b.)\n+ * 3. Check for rename/rename(1to2) conflicts (at the directory level).\n+ * In the future, we could potentially record this info as well and\n+ * omit reporting rename/rename(1to2) conflicts for each path within\n+ * the affected directories, thus cleaning up the merge output.\n+ * NOTE: We do NOT check for rename/rename(2to1) conflicts at the\n+ * directory level, because merging directories is fine. If it\n+ * causes conflicts for files within those merged directories, then\n+ * that should be detected at the individual path level.\n+ */\n+static void handle_directory_level_conflicts(struct merge_options *o,\n+\t\t\t\t\t struct hashmap *dir_re_head,\n+\t\t\t\t\t struct tree *head,\n+\t\t\t\t\t struct hashmap *dir_re_merge,\n+\t\t\t\t\t struct tree *merge)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct dir_rename_entry *head_ent;\n+\tstruct dir_rename_entry *merge_ent;\n+\tint i;\n+\n+\tstruct string_list remove_from_head = STRING_LIST_INIT_NODUP;\n+\tstruct string_list remove_from_merge = STRING_LIST_INIT_NODUP;\n+\n+\thashmap_iter_init(dir_re_head, &iter);\n+\twhile ((head_ent = hashmap_iter_next(&iter))) {\n+\t\tmerge_ent = dir_rename_find_entry(dir_re_merge, head_ent->dir);\n+\t\tif (merge_ent &&\n+\t\t !head_ent->non_unique_new_dir &&\n+\t\t !merge_ent->non_unique_new_dir &&\n+\t\t !strcmp(head_ent->new_dir, merge_ent->new_dir)) {\n+\t\t\t/* 1. Renamed identically; remove it from both sides */\n+\t\t\tstring_list_append(&remove_from_head,\n+\t\t\t\t\t head_ent->dir)->util = head_ent;\n+\t\t\tfree(head_ent->new_dir);\n+\t\t\tstring_list_append(&remove_from_merge,\n+\t\t\t\t\t merge_ent->dir)->util = merge_ent;\n+\t\t\tfree(merge_ent->new_dir);\n+\t\t} else if (tree_has_path(head, head_ent->dir)) {\n+\t\t\t/* 2. This wasn't a directory rename after all */\n+\t\t\tstring_list_append(&remove_from_head,\n+\t\t\t\t\t head_ent->dir)->util = head_ent;\n+\t\t\tfree(head_ent->new_dir);\n+\t\t}\n+\t}\n+\n+\thashmap_iter_init(dir_re_merge, &iter);\n+\twhile ((merge_ent = hashmap_iter_next(&iter))) {\n+\t\thead_ent = dir_rename_find_entry(dir_re_head, merge_ent->dir);\n+\t\tif (tree_has_path(merge, merge_ent->dir)) {\n+\t\t\t/* 2. This wasn't a directory rename after all */\n+\t\t\tstring_list_append(&remove_from_merge,\n+\t\t\t\t\t merge_ent->dir)->util = merge_ent;\n+\t\t} else if (head_ent &&\n+\t\t\t !head_ent->non_unique_new_dir &&\n+\t\t\t !merge_ent->non_unique_new_dir) {\n+\t\t\t/* 3. rename/rename(1to2) */\n+\t\t\t/* We can assume it's not rename/rename(1to1) because\n+\t\t\t * that was case (1), already checked above. But\n+\t\t\t * quickly test that assertion, just because.\n+\t\t\t */\n+\t\t\tassert(strcmp(head_ent->new_dir, merge_ent->new_dir));\n+\t\t\toutput(o, 1, _(\"CONFLICT (rename/rename): \"\n+\t\t\t\t \"Rename directory %s->%s in %s. \"\n+\t\t\t\t \"Rename directory %s->%s in %s\"),\n+\t\t\t head_ent->dir, head_ent->new_dir, o->branch1,\n+\t\t\t head_ent->dir, merge_ent->new_dir, o->branch2);\n+\t\t\tstring_list_append(&remove_from_head,\n+\t\t\t\t\t head_ent->dir)->util = head_ent;\n+\t\t\tfree(head_ent->new_dir);\n+\t\t\tstring_list_append(&remove_from_merge,\n+\t\t\t\t\t merge_ent->dir)->util = merge_ent;\n+\t\t\tfree(merge_ent->new_dir);\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < remove_from_head.nr; i++) {\n+\t\thead_ent = remove_from_head.items[i].util;\n+\t\thashmap_remove(dir_re_head, head_ent, NULL);\n+\t}\n+\tfor (i = 0; i < remove_from_merge.nr; i++) {\n+\t\tmerge_ent = remove_from_merge.items[i].util;\n+\t\thashmap_remove(dir_re_merge, merge_ent, NULL);\n+\t}\n+\n+\tstring_list_clear(&remove_from_head, 0);\n+\tstring_list_clear(&remove_from_merge, 0);\n+}\n+\n static struct hashmap *get_directory_renames(struct diff_queue_struct *pairs,\n \t\t\t\t\t struct tree *tree) {\n \tstruct hashmap *dir_renames;\n@@ -1827,6 +1935,10 @@ static struct rename_info *handle_renames(struct merge_options *o,\n \tdir_re_head = get_directory_renames(head_pairs, head);\n \tdir_re_merge = get_directory_renames(merge_pairs, merge);\n \n+\thandle_directory_level_conflicts(o,\n+\t\t\t\t\t dir_re_head, head,\n+\t\t\t\t\t dir_re_merge, merge);\n+\n \trei->head_renames = get_renames(o, head_pairs, head,\n \t\t\t\t\t common, head, merge, entries);\n \trei->merge_renames = get_renames(o, merge_pairs, merge,\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332231","messageId":"20171110190550.27059-20-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 19/30] merge-recursive: Split out code for determining diff_filepairs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:39Z","receivedAt":"2017-11-10T19:07:05Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Create a new function, get_diffpairs() to compute the diff_filepairs\nbetween two trees. While these are currently only used in\nget_renames(), I want them to be available to some new functions. No\nactual logic changes yet.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 81 ++++++++++++++++++++++++++++++++++++++++---------------\n 1 file changed, 60 insertions(+), 21 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex f40c70990c..8c9543d85c 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -1315,24 +1315,15 @@ static int conflict_rename_rename_2to1(struct merge_options *o,\n }\n \n /*\n- * Get information of all renames which occurred between 'o_tree' and\n- * 'tree'. We need the three trees in the merge ('o_tree', 'a_tree' and\n- * 'b_tree') to be able to associate the correct cache entries with\n- * the rename information. 'tree' is always equal to either a_tree or b_tree.\n+ * Get the diff_filepairs changed between o_tree and tree.\n */\n-static struct string_list *get_renames(struct merge_options *o,\n-\t\t\t\t struct tree *tree,\n-\t\t\t\t struct tree *o_tree,\n-\t\t\t\t struct tree *a_tree,\n-\t\t\t\t struct tree *b_tree,\n-\t\t\t\t struct string_list *entries)\n+static struct diff_queue_struct *get_diffpairs(struct merge_options *o,\n+\t\t\t\t\t struct tree *o_tree,\n+\t\t\t\t\t struct tree *tree)\n {\n-\tint i;\n-\tstruct string_list *renames;\n+\tstruct diff_queue_struct *ret;\n \tstruct diff_options opts;\n \n-\trenames = xcalloc(1, sizeof(struct string_list));\n-\n \tdiff_setup(&opts);\n \tDIFF_OPT_SET(&opts, RECURSIVE);\n \tDIFF_OPT_CLR(&opts, RENAME_EMPTY);\n@@ -1348,10 +1339,43 @@ static struct string_list *get_renames(struct merge_options *o,\n \tdiffcore_std(&opts);\n \tif (opts.needed_rename_limit > o->needed_rename_limit)\n \t\to->needed_rename_limit = opts.needed_rename_limit;\n-\tfor (i = 0; i < diff_queued_diff.nr; ++i) {\n+\n+\tret = malloc(sizeof(struct diff_queue_struct));\n+\tret->queue = diff_queued_diff.queue;\n+\tret->nr = diff_queued_diff.nr;\n+\t// Ignore diff_queued_diff.alloc; we won't be changing the size at all\n+\n+\topts.output_format = DIFF_FORMAT_NO_OUTPUT;\n+\tdiff_queued_diff.nr = 0;\n+\tdiff_queued_diff.queue = NULL;\n+\tdiff_flush(&opts);\n+\treturn ret;\n+}\n+\n+/*\n+ * Get information of all renames which occurred in 'pairs', making use of\n+ * any implicit directory renames inferred from the other side of history.\n+ * We need the three trees in the merge ('o_tree', 'a_tree' and 'b_tree')\n+ * to be able to associate the correct cache entries with the rename\n+ * information; tree is always equal to either a_tree or b_tree.\n+ */\n+static struct string_list *get_renames(struct merge_options *o,\n+\t\t\t\t struct diff_queue_struct *pairs,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct tree *o_tree,\n+\t\t\t\t struct tree *a_tree,\n+\t\t\t\t struct tree *b_tree,\n+\t\t\t\t struct string_list *entries)\n+{\n+\tint i;\n+\tstruct string_list *renames;\n+\n+\trenames = xcalloc(1, sizeof(struct string_list));\n+\n+\tfor (i = 0; i < pairs->nr; ++i) {\n \t\tstruct string_list_item *item;\n \t\tstruct rename *re;\n-\t\tstruct diff_filepair *pair = diff_queued_diff.queue[i];\n+\t\tstruct diff_filepair *pair = pairs->queue[i];\n \t\tif (pair->status != 'R') {\n \t\t\tdiff_free_filepair(pair);\n \t\t\tcontinue;\n@@ -1375,9 +1399,6 @@ static struct string_list *get_renames(struct merge_options *o,\n \t\titem = string_list_insert(renames, pair->one->path);\n \t\titem->util = re;\n \t}\n-\topts.output_format = DIFF_FORMAT_NO_OUTPUT;\n-\tdiff_queued_diff.nr = 0;\n-\tdiff_flush(&opts);\n \treturn renames;\n }\n \n@@ -1649,15 +1670,33 @@ static struct rename_info *handle_renames(struct merge_options *o,\n \t\t\t\t\t int *clean)\n {\n \tstruct rename_info *rei = xcalloc(1, sizeof(struct rename_info));\n+\tstruct diff_queue_struct *head_pairs, *merge_pairs;\n \n \t*clean = 1;\n \tif (!o->detect_rename)\n \t\treturn NULL;\n \n-\trei->head_renames = get_renames(o, head, common, head, merge, entries);\n-\trei->merge_renames = get_renames(o, merge, common, head, merge, entries);\n+\thead_pairs = get_diffpairs(o, common, head);\n+\tmerge_pairs = get_diffpairs(o, common, merge);\n+\n+\trei->head_renames = get_renames(o, head_pairs, head,\n+\t\t\t\t\t common, head, merge, entries);\n+\trei->merge_renames = get_renames(o, merge_pairs, merge,\n+\t\t\t\t\t common, head, merge, entries);\n \t*clean = process_renames(o, rei->head_renames, rei->merge_renames);\n \n+cleanup:\n+\t/*\n+\t * Some cleanup is deferred until cleanup_renames() because the\n+\t * data structures are still needed and referenced in\n+\t * process_entry(). But there are a few things we can free now.\n+\t */\n+\n+\tfree(head_pairs->queue);\n+\tfree(head_pairs);\n+\tfree(merge_pairs->queue);\n+\tfree(merge_pairs);\n+\n \treturn rei;\n }\n \n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332232","messageId":"20171110190550.27059-14-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 13/30] directory rename detection: tests for handling overwriting untracked files","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:33Z","receivedAt":"2017-11-10T19:07:07Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n t/t6043-merge-rename-directories.sh | 314 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 314 insertions(+)\n\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex bb179b16c8..7af8962512 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -2559,4 +2559,318 @@ test_expect_failure '9g-check: Renamed directory that only contained immediate s\n # side of history for any implicit directory renames.\n ###########################################################################\n \n+###########################################################################\n+# SECTION 10: Handling untracked files\n+#\n+# unpack_trees(), upon which the recursive merge algorithm is based, aborts\n+# the operation if untracked or dirty files would be deleted or overwritten\n+# by the merge. Unfortunately, unpack_trees() does not understand renames,\n+# and if it doesn't abort, then it muddies up the working directory before\n+# we even get to the point of detecting renames, so we need some special\n+# handling, at least in the case of directory renames.\n+###########################################################################\n+\n+# Testcase 10a, Overwrite untracked: normal rename/delete\n+# Commit A: z/{b,c_1}\n+# Commit B: z/b + untracked z/c + untracked z/d\n+# Commit C: z/{b,d_1}\n+# Expected: Aborted Merge +\n+# ERROR_MSG(untracked working tree files would be overwritten by merge)\n+\n+test_expect_success '10a-setup: Overwrite untracked with normal rename/delete' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit rm z/c &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv z/c z/d &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_success '10a-check: Overwrite untracked with normal rename/delete' '\n+\tgit checkout B^0 &&\n+\techo very >z/c &&\n+\techo important >z/d &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out 2>err &&\n+\ttest_i18ngrep \"The following untracked working tree files would be overwritten by merge\" err &&\n+\n+\ttest 1 -eq $(git ls-files -s | wc -l) &&\n+\ttest 4 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest \"very\" = \"$(cat z/c)\" &&\n+\ttest \"important\" = \"$(cat z/d)\" &&\n+\ttest $(git rev-parse HEAD:z/b) = $(git rev-parse A:z/b)\n+'\n+\n+# Testcase 10b, Overwrite untracked: dir rename + delete\n+# Commit A: z/{b,c_1}\n+# Commit B: y/b + untracked y/{c,d,e}\n+# Commit C: z/{b,d_1,e}\n+# Expected: Failed Merge; y/b + untracked y/c + untracked y/d on disk +\n+# z/c_1 -> z/d_1 rename recorded at stage 3 for y/d +\n+# ERROR_MSG(refusing to lose untracked file at 'y/d')\n+\n+test_expect_success '10b-setup: Overwrite untracked with dir rename + delete' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit rm z/c &&\n+\tgit mv z/ y/ &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv z/c z/d &&\n+\techo e >z/e &&\n+\tgit add z/e &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '10b-check: Overwrite untracked with dir rename + delete' '\n+\tgit checkout B^0 &&\n+\techo very >y/c &&\n+\techo important >y/d &&\n+\techo contents >y/e &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out 2>err &&\n+\ttest_i18ngrep \"CONFLICT (rename/delete).*Version C^0 of y/d left in tree at y/d~C^0\" out &&\n+\ttest_i18ngrep \"Error: Refusing to lose untracked file at y/e; writing to y/e~C^0 instead\" out &&\n+\n+\ttest 3 -eq $(git ls-files -s | wc -l) &&\n+\ttest 2 -eq $(git ls-files -u | wc -l) &&\n+\ttest 5 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest \"very\" = \"$(cat y/c)\" &&\n+\n+\ttest \"important\" = \"$(cat y/d)\" &&\n+\ttest \"important\" != \"$(git rev-parse :3:y/d)\" &&\n+\ttest $(git rev-parse :3:y/d) = $(git rev-parse A:z/c) &&\n+\n+\ttest \"contents\" = \"$(cat y/e)\" &&\n+\ttest \"contents\" != \"$(git rev-parse :3:y/e)\" &&\n+\ttest $(git rev-parse :3:y/e) = $(git rev-parse C:z/e)\n+'\n+\n+# Testcase 10c, Overwrite untracked: dir rename/rename(1to2)\n+# Commit A: z/{a,b}, x/{c,d}\n+# Commit B: y/{a,b}, w/c, x/d + different untracked y/c\n+# Commit C: z/{a,b,c}, x/d\n+# Expected: Failed Merge; y/{a,b} + x/d + untracked y/c +\n+# CONFLICT(rename/rename) x/c -> w/c vs y/c +\n+# y/c~C^0 +\n+# ERROR_MSG(Refusing to lose untracked file at y/c)\n+\n+test_expect_success '10c-setup: Overwrite untracked with dir rename/rename(1to2)' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z x &&\n+\techo a >z/a &&\n+\techo b >z/b &&\n+\techo c >x/c &&\n+\techo d >x/d &&\n+\tgit add z x &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tmkdir w &&\n+\tgit mv x/c w/c &&\n+\tgit mv z/ y/ &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv x/c z/ &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '10c-check: Overwrite untracked with dir rename/rename(1to2)' '\n+\tgit checkout B^0 &&\n+\techo important >y/c &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out 2>err &&\n+\ttest_i18ngrep \"CONFLICT (rename/rename)\" out &&\n+\ttest_i18ngrep \"Refusing to lose untracked file at y/c; adding as y/c~C^0 instead\" out &&\n+\n+\ttest 6 -eq $(git ls-files -s | wc -l) &&\n+\ttest 3 -eq $(git ls-files -u | wc -l) &&\n+\ttest 3 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/a) = $(git rev-parse A:z/a) &&\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:x/d) = $(git rev-parse A:x/d) &&\n+\n+\ttest \"important\" = \"$(cat y/c)\" &&\n+\ttest \"important\" != \"$(git rev-parse :3:y/c)\" &&\n+\ttest $(git rev-parse :1:x/c) = $(git rev-parse A:x/c) &&\n+\ttest $(git rev-parse :2:w/c) = $(git rev-parse A:x/c) &&\n+\ttest $(git rev-parse :3:y/c) = $(git rev-parse A:x/c) &&\n+\ttest $(git hash-object y/c~C^0) = $(git rev-parse A:x/c)\n+'\n+\n+# Testcase 10d, Delete untracked w/ dir rename/rename(2to1)\n+# Commit A: z/{a,b,c_1}, x/{d,e,f_2}\n+# Commit B: y/{a,b}, x/{d,e,f_2,wham_1} + untracked y/wham\n+# Commit C: z/{a,b,c_1,wham_2}, y/{d,e}\n+# Expected: Failed Merge; y/{a,b,d,e} + untracked y/{wham,wham~C^0,wham~HEAD}+\n+# CONFLICT(rename/rename) z/c_1 vs x/f_2 -> y/wham\n+# ERROR_MSG(Refusing to lose untracked file at y/wham)\n+\n+test_expect_success '10d-setup: Delete untracked with dir rename/rename(2to1)' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z x &&\n+\techo a >z/a &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\techo d >x/d &&\n+\techo e >x/e &&\n+\techo f >x/f &&\n+\tgit add z x &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z/c x/wham &&\n+\tgit mv z/ y/ &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv x/f z/wham &&\n+\tgit mv x/ y/ &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '10d-check: Delete untracked with dir rename/rename(2to1)' '\n+\tgit checkout B^0 &&\n+\techo important >y/wham &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out 2>err &&\n+\ttest_i18ngrep \"CONFLICT (rename/rename)\" out &&\n+\ttest_i18ngrep \"Refusing to lose untracked file at y/wham\" out &&\n+\n+\ttest 6 -eq $(git ls-files -s | wc -l) &&\n+\ttest 2 -eq $(git ls-files -u | wc -l) &&\n+\ttest 4 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/a) = $(git rev-parse A:z/a) &&\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:y/d) = $(git rev-parse A:x/d) &&\n+\ttest $(git rev-parse :0:y/e) = $(git rev-parse A:x/e) &&\n+\n+\ttest_must_fail git rev-parse :1:y/wham &&\n+\ttest $(git rev-parse :2:y/wham) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse :3:y/wham) = $(git rev-parse A:x/f) &&\n+\n+\ttest \"important\" = \"$(cat y/wham)\" &&\n+\ttest $(git hash-object y/wham~C^0) = $(git rev-parse A:x/f) &&\n+\ttest $(git hash-object y/wham~HEAD) = $(git rev-parse A:z/c)\n+'\n+\n+# Testcase 10e, Does git complain about untracked file that's not in the way?\n+# Commit A: z/{a,b}\n+# Commit B: y/{a,b} + untracked z/c\n+# Commit C: z/{a,b,c}\n+# Expected: y/{a,b,c} + untracked z/c\n+\n+test_expect_success '10e-setup: Does git complain about untracked file that is not really in the way?' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo a >z/a &&\n+\techo b >z/b &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z/ y/ &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\techo c >z/c &&\n+\tgit add z/c &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '10e-check: Does git complain about untracked file that is not really in the way?' '\n+\tgit checkout B^0 &&\n+\tmkdir z &&\n+\techo random >z/c &&\n+\n+\tgit merge -s recursive C^0 >out 2>err &&\n+\t! test_i18ngrep \"following untracked working tree files would be overwritten by merge\" err &&\n+\n+\ttest 3 -eq $(git ls-files -s | wc -l) &&\n+\ttest 0 -eq $(git ls-files -u | wc -l) &&\n+\ttest 3 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:y/a) = $(git rev-parse A:z/a) &&\n+\ttest $(git rev-parse :0:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:y/c) = $(git rev-parse C:z/c) &&\n+\n+\ttest \"random\" = \"$(cat z/c)\"\n+'\n+\n test_done\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332233","messageId":"20171110190550.27059-28-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 27/30] merge-recursive: Apply necessary modifications for directory renames","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:47Z","receivedAt":"2017-11-10T19:07:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This commit hooks together all the directory rename logic by making the\nnecessary changes to the rename struct, it's dst_entry, and the\ndiff_filepair under consideration.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 195 +++++++++++++++++++++++++++++++++++-\n t/t6043-merge-rename-directories.sh | 50 ++++-----\n 2 files changed, 219 insertions(+), 26 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 2a7258f6bb..838bfd32ec 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -166,6 +166,7 @@ static int oid_eq(const struct object_id *a, const struct object_id *b)\n \n enum rename_type {\n \tRENAME_NORMAL = 0,\n+\tRENAME_DIR,\n \tRENAME_DELETE,\n \tRENAME_ONE_FILE_TO_ONE,\n \tRENAME_ONE_FILE_TO_TWO,\n@@ -599,6 +600,7 @@ struct rename {\n \t */\n \tstruct stage_data *src_entry;\n \tstruct stage_data *dst_entry;\n+\tunsigned add_turned_into_rename:1;\n \tunsigned processed:1;\n };\n \n@@ -633,6 +635,26 @@ static int update_stages(struct merge_options *opt, const char *path,\n \treturn 0;\n }\n \n+static int update_stages_for_stage_data(struct merge_options *opt,\n+\t\t\t\t\tconst char *path,\n+\t\t\t\t\tconst struct stage_data *stage_data)\n+{\n+\tstruct diff_filespec o, a, b;\n+\to.mode = stage_data->stages[1].mode;\n+\toidcpy(&o.oid, &stage_data->stages[1].oid);\n+\n+\ta.mode = stage_data->stages[2].mode;\n+\toidcpy(&a.oid, &stage_data->stages[2].oid);\n+\n+\tb.mode = stage_data->stages[3].mode;\n+\toidcpy(&b.oid, &stage_data->stages[3].oid);\n+\n+\treturn update_stages(opt, path,\n+\t\t\t is_null_sha1(o.oid.hash) ? NULL : &o,\n+\t\t\t is_null_sha1(a.oid.hash) ? NULL : &a,\n+\t\t\t is_null_sha1(b.oid.hash) ? NULL : &b);\n+}\n+\n static void update_entry(struct stage_data *entry,\n \t\t\t struct diff_filespec *o,\n \t\t\t struct diff_filespec *a,\n@@ -1100,6 +1122,18 @@ static int merge_file_one(struct merge_options *o,\n \treturn merge_file_1(o, &one, &a, &b, branch1, branch2, mfi);\n }\n \n+static int conflict_rename_dir(struct merge_options *o,\n+\t\t\t struct diff_filepair *pair,\n+\t\t\t const char *rename_branch,\n+\t\t\t const char *other_branch)\n+{\n+\tconst struct diff_filespec *dest = pair->two;\n+\n+\tif (update_file(o, 1, &dest->oid, dest->mode, dest->path))\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n static int handle_change_delete(struct merge_options *o,\n \t\t\t\t const char *path, const char *old_path,\n \t\t\t\t const struct object_id *o_oid, int o_mode,\n@@ -1369,6 +1403,24 @@ static int conflict_rename_rename_2to1(struct merge_options *o,\n \t\tif (!ret)\n \t\t\tret = update_file(o, 0, &mfi_c2.oid, mfi_c2.mode,\n \t\t\t\t\t new_path2);\n+\t\t/*\n+\t\t * unpack_trees() actually populates the index for us for\n+\t\t * \"normal\" rename/rename(2to1) situtations so that the\n+\t\t * correct entries are at the higher stages, which would\n+\t\t * make the call below to update_stages_for_stage_data\n+\t\t * unnecessary. However, if either of the renames came\n+\t\t * from a directory rename, then unpack_trees() will not\n+\t\t * have gotten the right data loaded into the index, so we\n+\t\t * need to do so now. (While it'd be tempting to move this\n+\t\t * call to update_stages_for_stage_data() to\n+\t\t * apply_directory_rename_modifications(), that would break\n+\t\t * our intermediate calls to would_lose_untracked() since\n+\t\t * those rely on the current in-memory index. See also the\n+\t\t * big \"NOTE\" in update_stages()).\n+\t\t */\n+\t\tif (update_stages_for_stage_data(o, path, ci->dst_entry1))\n+\t\t\tret = -1;\n+\n \t\tfree(new_path2);\n \t\tfree(new_path1);\n \t}\n@@ -1888,6 +1940,120 @@ static char *check_for_directory_rename(struct merge_options *o,\n \treturn new_path;\n }\n \n+static void apply_directory_rename_modifications(struct merge_options *o,\n+\t\t\t\t\t\t struct diff_filepair *pair,\n+\t\t\t\t\t\t char *new_path,\n+\t\t\t\t\t\t struct rename *re,\n+\t\t\t\t\t\t struct tree *tree,\n+\t\t\t\t\t\t struct tree *o_tree,\n+\t\t\t\t\t\t struct tree *a_tree,\n+\t\t\t\t\t\t struct tree *b_tree,\n+\t\t\t\t\t\t struct string_list *entries,\n+\t\t\t\t\t\t int *clean)\n+{\n+\tstruct string_list_item *item;\n+\tint stage = (tree == a_tree ? 2 : 3);\n+\n+\t/*\n+\t * In all cases where we can do directory rename detection,\n+\t * unpack_trees() will have read pair->two->path into the\n+\t * index and the working copy. We need to remove it so that\n+\t * we can instead place it at new_path. It is guaranteed to\n+\t * not be untracked (unpack_trees() would have errored out\n+\t * saying the file would have been overwritten), but it might\n+\t * be dirty, though.\n+\t */\n+\tremove_file(o, 1, pair->two->path, 0 /* no_wd */);\n+\n+\t/* Find or create a new re->dst_entry */\n+\titem = string_list_lookup(entries, new_path);\n+\tif (item) {\n+\t\t/*\n+\t\t * Since we're renaming on this side of history, and it's\n+\t\t * due to a directory rename on the other side of history\n+\t\t * (which we only allow when the directory in question no\n+\t\t * longer exists on the other side of history), the\n+\t\t * original entry for re->dst_entry is no longer\n+\t\t * necessary...\n+\t\t */\n+\t\tre->dst_entry->processed = 1;\n+\n+\t\t/*\n+\t\t * ...because we'll be using this new one.\n+\t\t */\n+\t\tre->dst_entry = item->util;\n+\t} else {\n+\t\t/*\n+\t\t * re->dst_entry is for the before-dir-rename path, and we\n+\t\t * need it to hold information for the after-dir-rename\n+\t\t * path. Before creating a new entry, we need to mark the\n+\t\t * old one as unnecessary (...unless it is shared by\n+\t\t * src_entry, i.e. this didn't use to be a rename, in which\n+\t\t * case we can just allow the normal processing to happen\n+\t\t * for it).\n+\t\t */\n+\t\tif (!strcmp(pair->one->path, pair->two->path)) {\n+\t\t\t/*\n+\t\t\t * Paths should only match if this was initially a\n+\t\t\t * non-rename that is being turned into one by\n+\t\t\t * directory rename detection.\n+\t\t\t */\n+\t\t\tassert(pair->status != 'R');\n+\t\t} else {\n+\t\t\tassert(pair->status == 'R');\n+\t\t\tre->dst_entry->processed = 1;\n+\t\t\t//string_list_remove(entries, pair->two->path, 0);\n+\t\t}\n+\n+\t\tre->dst_entry = insert_stage_data(new_path,\n+\t\t\t\t\t\t o_tree, a_tree, b_tree,\n+\t\t\t\t\t\t entries);\n+\t\titem = string_list_insert(entries, new_path);\n+\t\titem->util = re->dst_entry;\n+\t}\n+\n+\t/*\n+\t * Update the stage_data with the information about the path we are\n+\t * moving into place. That slot will be empty and available for us\n+\t * to write to because of the collision checks in\n+\t * handle_path_level_conflicts().\n+\t *\n+\t * It may be tempting to actually update the index at this point as\n+\t * well, using update_stages_for_stage_data(), but as per the big\n+\t * \"NOTE\" in update_stages(), doing so will modify the current\n+\t * in-memory index which will break calls to would_lose_untracked()\n+\t * that we need to make. Instead, we need to just make sure that\n+\t * the various conflict_rename_*() functions update the index\n+\t * explicitly rather than relying on unpack_trees() to have done it.\n+\t */\n+\tassert(is_null_oid(&re->dst_entry->stages[stage].oid));\n+\tget_tree_entry(tree->object.oid.hash,\n+\t\t pair->two->path,\n+\t\t re->dst_entry->stages[stage].oid.hash,\n+\t\t &re->dst_entry->stages[stage].mode);\n+\n+\t/* Update pair status */\n+\tif (pair->status == 'A') {\n+\t\t/*\n+\t\t * Recording rename information for this add makes it look\n+\t\t * like a rename/delete conflict. Make sure we can\n+\t\t * correctly handle this as an add that was moved to a new\n+\t\t * directory instead of reporting a rename/delete conflict.\n+\t\t */\n+\t\tre->add_turned_into_rename = 1;\n+\t}\n+\t/*\n+\t * We don't actually look at pair->status again, but it seems\n+\t * pedagogically correct to adjust it.\n+\t */\n+\tpair->status = 'R';\n+\n+\t/*\n+\t * Finally, record the new location.\n+\t */\n+\tpair->two->path = new_path;\n+}\n+\n /*\n * Get information of all renames which occurred in 'pairs', making use of\n * any implicit directory renames inferred from the other side of history.\n@@ -1937,6 +2103,7 @@ static struct string_list *get_renames(struct merge_options *o,\n \n \t\tre = xmalloc(sizeof(*re));\n \t\tre->processed = 0;\n+\t\tre->add_turned_into_rename = 0;\n \t\tre->pair = pair;\n \t\titem = string_list_lookup(entries, re->pair->one->path);\n \t\tif (!item)\n@@ -1953,6 +2120,12 @@ static struct string_list *get_renames(struct merge_options *o,\n \t\t\tre->dst_entry = item->util;\n \t\titem = string_list_insert(renames, pair->one->path);\n \t\titem->util = re;\n+\t\tif (new_path)\n+\t\t\t apply_directory_rename_modifications(o, pair, new_path,\n+\t\t\t\t\t\t\t re, tree, o_tree,\n+\t\t\t\t\t\t\t a_tree, b_tree,\n+\t\t\t\t\t\t\t entries,\n+\t\t\t\t\t\t\t clean_merge);\n \t}\n \n \thashmap_iter_init(&collisions, &iter);\n@@ -2122,7 +2295,19 @@ static int process_renames(struct merge_options *o,\n \t\t\tdst_other.mode = ren1->dst_entry->stages[other_stage].mode;\n \t\t\ttry_merge = 0;\n \n-\t\t\tif (oid_eq(&src_other.oid, &null_oid)) {\n+\t\t\tif (oid_eq(&src_other.oid, &null_oid) &&\n+\t\t\t ren1->add_turned_into_rename) {\n+\t\t\t\tsetup_rename_conflict_info(RENAME_DIR,\n+\t\t\t\t\t\t\t ren1->pair,\n+\t\t\t\t\t\t\t NULL,\n+\t\t\t\t\t\t\t branch1,\n+\t\t\t\t\t\t\t branch2,\n+\t\t\t\t\t\t\t ren1->dst_entry,\n+\t\t\t\t\t\t\t NULL,\n+\t\t\t\t\t\t\t o,\n+\t\t\t\t\t\t\t NULL,\n+\t\t\t\t\t\t\t NULL);\n+\t\t\t} else if (oid_eq(&src_other.oid, &null_oid)) {\n \t\t\t\tsetup_rename_conflict_info(RENAME_DELETE,\n \t\t\t\t\t\t\t ren1->pair,\n \t\t\t\t\t\t\t NULL,\n@@ -2546,6 +2731,14 @@ static int process_entry(struct merge_options *o,\n \t\t\t\t\t\t o_oid, o_mode, a_oid, a_mode, b_oid, b_mode,\n \t\t\t\t\t\t conflict_info);\n \t\t\tbreak;\n+\t\tcase RENAME_DIR:\n+\t\t\tclean_merge = 1;\n+\t\t\tif (conflict_rename_dir(o,\n+\t\t\t\t\t\tconflict_info->pair1,\n+\t\t\t\t\t\tconflict_info->branch1,\n+\t\t\t\t\t\tconflict_info->branch2))\n+\t\t\t\tclean_merge = -1;\n+\t\t\tbreak;\n \t\tcase RENAME_DELETE:\n \t\t\tclean_merge = 0;\n \t\t\tif (conflict_rename_delete(o,\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex 858d83016a..e737bad2c5 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -64,7 +64,7 @@ test_expect_success '1a-setup: Simple directory rename detection' '\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '1a-check: Simple directory rename detection' '\n+test_expect_success '1a-check: Simple directory rename detection' '\n \tgit checkout B^0 &&\n \n \tgit merge -s recursive C^0 &&\n@@ -121,7 +121,7 @@ test_expect_success '1b-setup: Merge a directory with another' '\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '1b-check: Merge a directory with another' '\n+test_expect_success '1b-check: Merge a directory with another' '\n \tgit checkout B^0 &&\n \n \tgit merge -s recursive C^0 &&\n@@ -173,7 +173,7 @@ test_expect_success '1c-setup: Transitive renaming' '\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '1c-check: Transitive renaming' '\n+test_expect_success '1c-check: Transitive renaming' '\n \tgit checkout B^0 &&\n \n \tgit merge -s recursive C^0 &&\n@@ -235,7 +235,7 @@ test_expect_success '1d-setup: Directory renames cause a rename/rename(2to1) con\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '1d-check: Directory renames cause a rename/rename(2to1) conflict' '\n+test_expect_success '1d-check: Directory renames cause a rename/rename(2to1) conflict' '\n \tgit checkout B^0 &&\n \n \ttest_must_fail git merge -s recursive C^0 >out &&\n@@ -302,7 +302,7 @@ test_expect_success '1e-setup: Renamed directory, with all files being renamed t\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '1e-check: Renamed directory, with all files being renamed too' '\n+test_expect_success '1e-check: Renamed directory, with all files being renamed too' '\n \tgit checkout B^0 &&\n \n \tgit merge -s recursive C^0 &&\n@@ -361,7 +361,7 @@ test_expect_success '1f-setup: Split a directory into two other directories' '\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '1f-check: Split a directory into two other directories' '\n+test_expect_success '1f-check: Split a directory into two other directories' '\n \tgit checkout B^0 &&\n \n \tgit merge -s recursive C^0 &&\n@@ -807,7 +807,7 @@ test_expect_success '5a-setup: Merge directories, other side adds files to origi\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '5a-check: Merge directories, other side adds files to original and target' '\n+test_expect_success '5a-check: Merge directories, other side adds files to original and target' '\n \tgit checkout B^0 &&\n \n \ttest_must_fail git merge -s recursive C^0 >out &&\n@@ -878,7 +878,7 @@ test_expect_success '5b-setup: Rename/delete in order to get add/add/add conflic\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '5b-check: Rename/delete in order to get add/add/add conflict' '\n+test_expect_success '5b-check: Rename/delete in order to get add/add/add conflict' '\n \tgit checkout B^0 &&\n \n \ttest_must_fail git merge -s recursive C^0 >out &&\n@@ -951,7 +951,7 @@ test_expect_success '5c-setup: Transitive rename would cause rename/rename/renam\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '5c-check: Transitive rename would cause rename/rename/rename/add/add/add' '\n+test_expect_success '5c-check: Transitive rename would cause rename/rename/rename/add/add/add' '\n \tgit checkout B^0 &&\n \n \ttest_must_fail git merge -s recursive C^0 >out &&\n@@ -1025,7 +1025,7 @@ test_expect_success '5d-setup: Directory/file/file conflict due to directory ren\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '5d-check: Directory/file/file conflict due to directory rename' '\n+test_expect_success '5d-check: Directory/file/file conflict due to directory rename' '\n \tgit checkout B^0 &&\n \n \ttest_must_fail git merge -s recursive C^0 >out &&\n@@ -1403,7 +1403,7 @@ test_expect_success '7a-setup: rename-dir vs. rename-dir (NOT split evenly) PLUS\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '7a-check: rename-dir vs. rename-dir (NOT split evenly) PLUS add-other-file' '\n+test_expect_success '7a-check: rename-dir vs. rename-dir (NOT split evenly) PLUS add-other-file' '\n \tgit checkout B^0 &&\n \n \ttest_must_fail git merge -s recursive C^0 >out &&\n@@ -1471,7 +1471,7 @@ test_expect_success '7b-setup: rename/rename(2to1), but only due to transitive r\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '7b-check: rename/rename(2to1), but only due to transitive rename' '\n+test_expect_success '7b-check: rename/rename(2to1), but only due to transitive rename' '\n \tgit checkout B^0 &&\n \n \ttest_must_fail git merge -s recursive C^0 >out &&\n@@ -1537,7 +1537,7 @@ test_expect_success '7c-setup: rename/rename(1to...2or3); transitive rename may\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '7c-check: rename/rename(1to...2or3); transitive rename may add complexity' '\n+test_expect_success '7c-check: rename/rename(1to...2or3); transitive rename may add complexity' '\n \tgit checkout B^0 &&\n \n \ttest_must_fail git merge -s recursive C^0 >out &&\n@@ -1595,7 +1595,7 @@ test_expect_success '7d-setup: transitive rename involved in rename/delete; how\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '7d-check: transitive rename involved in rename/delete; how is it reported?' '\n+test_expect_success '7d-check: transitive rename involved in rename/delete; how is it reported?' '\n \tgit checkout B^0 &&\n \n \ttest_must_fail git merge -s recursive C^0 >out &&\n@@ -1664,7 +1664,7 @@ test_expect_success '7e-setup: transitive rename in rename/delete AND dirs in th\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '7e-check: transitive rename in rename/delete AND dirs in the way' '\n+test_expect_success '7e-check: transitive rename in rename/delete AND dirs in the way' '\n \tgit checkout B^0 &&\n \n \ttest_must_fail git merge -s recursive C^0 >out &&\n@@ -1748,7 +1748,7 @@ test_expect_success '8a-setup: Dual-directory rename, one into the others way' '\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '8a-check: Dual-directory rename, one into the others way' '\n+test_expect_success '8a-check: Dual-directory rename, one into the others way' '\n \tgit checkout B^0 &&\n \n \tgit merge -s recursive C^0 &&\n@@ -1881,7 +1881,7 @@ test_expect_success '8c-setup: rename+modify/delete' '\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '8c-check: rename+modify/delete' '\n+test_expect_success '8c-check: rename+modify/delete' '\n \tgit checkout B^0 &&\n \n \ttest_must_fail git merge -s recursive C^0 >out &&\n@@ -1960,7 +1960,7 @@ test_expect_success '8d-setup: rename/delete...or not?' '\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '8d-check: rename/delete...or not?' '\n+test_expect_success '8d-check: rename/delete...or not?' '\n \tgit checkout B^0 &&\n \n \tgit merge -s recursive C^0 &&\n@@ -2026,7 +2026,7 @@ test_expect_success '8e-setup: Both sides rename, one side adds to original dire\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '8e-check: Both sides rename, one side adds to original directory' '\n+test_expect_success '8e-check: Both sides rename, one side adds to original directory' '\n \tgit checkout B^0 &&\n \n \ttest_must_fail git merge -s recursive C^0 >out 2>err &&\n@@ -2111,7 +2111,7 @@ test_expect_success '9a-setup: Inner renamed directory within outer renamed dire\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '9a-check: Inner renamed directory within outer renamed directory' '\n+test_expect_success '9a-check: Inner renamed directory within outer renamed directory' '\n \tgit checkout B^0 &&\n \n \tgit merge -s recursive C^0 &&\n@@ -2171,7 +2171,7 @@ test_expect_success '9b-setup: Transitive rename with content merge' '\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '9b-check: Transitive rename with content merge' '\n+test_expect_success '9b-check: Transitive rename with content merge' '\n \tgit checkout B^0 &&\n \n \tgit merge -s recursive C^0 &&\n@@ -2254,7 +2254,7 @@ test_expect_success '9c-setup: Doubly transitive rename?' '\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '9c-check: Doubly transitive rename?' '\n+test_expect_success '9c-check: Doubly transitive rename?' '\n \tgit checkout B^0 &&\n \n \tgit merge -s recursive C^0 >out &&\n@@ -2337,7 +2337,7 @@ test_expect_success '9d-setup: N-way transitive rename?' '\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '9d-check: N-way transitive rename?' '\n+test_expect_success '9d-check: N-way transitive rename?' '\n \tgit checkout B^0 &&\n \n \tgit merge -s recursive C^0 >out &&\n@@ -2411,7 +2411,7 @@ test_expect_success '9e-setup: N-to-1 whammo' '\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '9e-check: N-to-1 whammo' '\n+test_expect_success '9e-check: N-to-1 whammo' '\n \tgit checkout B^0 &&\n \n \ttest_must_fail git merge -s recursive C^0 >out &&\n@@ -2482,7 +2482,7 @@ test_expect_success '9f-setup: Renamed directory that only contained immediate s\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '9f-check: Renamed directory that only contained immediate subdirs' '\n+test_expect_success '9f-check: Renamed directory that only contained immediate subdirs' '\n \tgit checkout B^0 &&\n \n \tgit merge -s recursive C^0 &&\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332234","messageId":"20171110190550.27059-19-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 18/30] merge-recursive: Make !o->detect_rename codepath more obvious","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:38Z","receivedAt":"2017-11-10T19:07:12Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Previously, if !o->detect_rename then get_renames() would return an\nempty string_list, and then process_renames() would have nothing to\niterate over. It seems more straightforward to simply avoid calling\neither function in that case.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 9 +++++++--\n 1 file changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 7a3402e50c..f40c70990c 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -1332,8 +1332,6 @@ static struct string_list *get_renames(struct merge_options *o,\n \tstruct diff_options opts;\n \n \trenames = xcalloc(1, sizeof(struct string_list));\n-\tif (!o->detect_rename)\n-\t\treturn renames;\n \n \tdiff_setup(&opts);\n \tDIFF_OPT_SET(&opts, RECURSIVE);\n@@ -1652,6 +1650,10 @@ static struct rename_info *handle_renames(struct merge_options *o,\n {\n \tstruct rename_info *rei = xcalloc(1, sizeof(struct rename_info));\n \n+\t*clean = 1;\n+\tif (!o->detect_rename)\n+\t\treturn NULL;\n+\n \trei->head_renames = get_renames(o, head, common, head, merge, entries);\n \trei->merge_renames = get_renames(o, merge, common, head, merge, entries);\n \t*clean = process_renames(o, rei->head_renames, rei->merge_renames);\n@@ -1664,6 +1666,9 @@ static void cleanup_renames(struct rename_info *re_info)\n \tconst struct rename *re;\n \tint i;\n \n+\tif (!re_info)\n+\t\treturn;\n+\n \tfor (i = 0; i < re_info->head_renames->nr; i++) {\n \t\tre = re_info->head_renames->items[i].util;\n \t\tdiff_free_filepair(re->pair);\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332235","messageId":"20171110190550.27059-18-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 17/30] merge-recursive: Fix leaks of allocated renames and diff_filepairs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:37Z","receivedAt":"2017-11-10T19:07:15Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"get_renames() has always zero'ed out diff_queued_diff.nr while only\nmanually free'ing diff_filepairs that did not correspond to renames.\nFurther, it allocated struct renames that were tucked away in the\nreturn string_list. Make sure all of these are deallocated when we\nare done with them.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 15 +++++++++++++--\n 1 file changed, 13 insertions(+), 2 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 49710c0964..7a3402e50c 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -1661,10 +1661,21 @@ static struct rename_info *handle_renames(struct merge_options *o,\n \n static void cleanup_renames(struct rename_info *re_info)\n {\n-\tstring_list_clear(re_info->head_renames, 0);\n-\tstring_list_clear(re_info->merge_renames, 0);\n+\tconst struct rename *re;\n+\tint i;\n \n+\tfor (i = 0; i < re_info->head_renames->nr; i++) {\n+\t\tre = re_info->head_renames->items[i].util;\n+\t\tdiff_free_filepair(re->pair);\n+\t}\n+\tstring_list_clear(re_info->head_renames, 1);\n \tfree(re_info->head_renames);\n+\n+\tfor (i = 0; i < re_info->merge_renames->nr; i++) {\n+\t\tre = re_info->merge_renames->items[i].util;\n+\t\tdiff_free_filepair(re->pair);\n+\t}\n+\tstring_list_clear(re_info->merge_renames, 1);\n \tfree(re_info->merge_renames);\n \n \tfree(re_info);\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332236","messageId":"20171110190550.27059-29-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 28/30] merge-recursive: Avoid clobbering untracked files with directory renames","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:48Z","receivedAt":"2017-11-10T19:07:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n merge-recursive.c | 39 +++++++++++++++++++++++++++++++++++--\n t/t6043-merge-rename-directories.sh | 6 +++---\n 2 files changed, 40 insertions(+), 5 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 838bfd32ec..1b3ee5b9fb 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -1129,6 +1129,25 @@ static int conflict_rename_dir(struct merge_options *o,\n {\n \tconst struct diff_filespec *dest = pair->two;\n \n+\tif (!o->call_depth && would_lose_untracked(dest->path)) {\n+\t\tchar *alt_path = unique_path(o, dest->path, rename_branch);\n+\t\toutput(o, 1, _(\"Error: Refusing to lose untracked file at %s; \"\n+\t\t\t \"writing to %s instead.\"),\n+\t\t dest->path, alt_path);\n+\t\t/*\n+\t\t * Write the file in worktree at alt_path, but not in the\n+\t\t * index. Instead, write to dest->path for the index but\n+\t\t * only at the higher appropriate stage.\n+\t\t */\n+\t\tif (update_file(o, 0, &dest->oid, dest->mode, alt_path))\n+\t\t\treturn -1;\n+\t\tfree(alt_path);\n+\t\treturn update_stages(o, dest->path, NULL,\n+\t\t\t\t rename_branch == o->branch1 ? dest : NULL,\n+\t\t\t\t rename_branch == o->branch1 ? NULL : dest);\n+\t}\n+\n+\t/* Update dest->path both in index and in worktree */\n \tif (update_file(o, 1, &dest->oid, dest->mode, dest->path))\n \t\treturn -1;\n \treturn 0;\n@@ -1147,7 +1166,8 @@ static int handle_change_delete(struct merge_options *o,\n \tconst char *update_path = path;\n \tint ret = 0;\n \n-\tif (dir_in_way(path, !o->call_depth, 0)) {\n+\tif (dir_in_way(path, !o->call_depth, 0) ||\n+\t (!o->call_depth && would_lose_untracked(path))) {\n \t\tupdate_path = alt_path = unique_path(o, path, change_branch);\n \t}\n \n@@ -1273,6 +1293,10 @@ static int handle_file(struct merge_options *o,\n \t\t\tdst_name = unique_path(o, rename->path, cur_branch);\n \t\t\toutput(o, 1, _(\"%s is a directory in %s adding as %s instead\"),\n \t\t\t rename->path, other_branch, dst_name);\n+\t\t} else if (!o->call_depth && would_lose_untracked(rename->path)) {\n+\t\t\tdst_name = unique_path(o, rename->path, cur_branch);\n+\t\t\toutput(o, 1, _(\"Refusing to lose untracked file at %s; adding as %s instead\"),\n+\t\t\t rename->path, dst_name);\n \t\t}\n \t}\n \tif ((ret = update_file(o, 0, &rename->oid, rename->mode, dst_name)))\n@@ -1398,7 +1422,18 @@ static int conflict_rename_rename_2to1(struct merge_options *o,\n \t\tchar *new_path2 = unique_path(o, path, ci->branch2);\n \t\toutput(o, 1, _(\"Renaming %s to %s and %s to %s instead\"),\n \t\t a->path, new_path1, b->path, new_path2);\n-\t\tremove_file(o, 0, path, 0);\n+\t\tif (would_lose_untracked(path))\n+\t\t\t/*\n+\t\t\t * Only way we get here is if both renames were from\n+\t\t\t * a directory rename AND user had an untracked file\n+\t\t\t * at the location where both files end up after the\n+\t\t\t * two directory renames. See testcase 10d of t6043.\n+\t\t\t */\n+\t\t\toutput(o, 1, _(\"Refusing to lose untracked file at \"\n+\t\t\t\t \"%s, even though it's in the way.\"),\n+\t\t\t path);\n+\t\telse\n+\t\t\tremove_file(o, 0, path, 0);\n \t\tret = update_file(o, 0, &mfi_c1.oid, mfi_c1.mode, new_path1);\n \t\tif (!ret)\n \t\t\tret = update_file(o, 0, &mfi_c2.oid, mfi_c2.mode,\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex e737bad2c5..6db764a1b6 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -2660,7 +2660,7 @@ test_expect_success '10b-setup: Overwrite untracked with dir rename + delete' '\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '10b-check: Overwrite untracked with dir rename + delete' '\n+test_expect_success '10b-check: Overwrite untracked with dir rename + delete' '\n \tgit checkout B^0 &&\n \techo very >y/c &&\n \techo important >y/d &&\n@@ -2727,7 +2727,7 @@ test_expect_success '10c-setup: Overwrite untracked with dir rename/rename(1to2)\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '10c-check: Overwrite untracked with dir rename/rename(1to2)' '\n+test_expect_success '10c-check: Overwrite untracked with dir rename/rename(1to2)' '\n \tgit checkout B^0 &&\n \techo important >y/c &&\n \n@@ -2793,7 +2793,7 @@ test_expect_success '10d-setup: Delete untracked with dir rename/rename(2to1)' '\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '10d-check: Delete untracked with dir rename/rename(2to1)' '\n+test_expect_success '10d-check: Delete untracked with dir rename/rename(2to1)' '\n \tgit checkout B^0 &&\n \techo important >y/wham &&\n \n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332237","messageId":"20171110190550.27059-31-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 30/30] merge-recursive: Fix remaining directory rename + dirty overwrite cases","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:50Z","receivedAt":"2017-11-10T19:07:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n merge-recursive.c | 26 +++++++++++++++++++++++---\n t/t6043-merge-rename-directories.sh | 8 ++++----\n 2 files changed, 27 insertions(+), 7 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 86ddb89727..6ef1d52f0a 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -1296,11 +1296,23 @@ static int handle_file(struct merge_options *o,\n \n \tadd = filespec_from_entry(&other, dst_entry, stage ^ 1);\n \tif (add) {\n+\t\tint ren_src_was_dirty = was_dirty(o, rename->path);\n \t\tchar *add_name = unique_path(o, rename->path, other_branch);\n \t\tif (update_file(o, 0, &add->oid, add->mode, add_name))\n \t\t\treturn -1;\n \n-\t\tremove_file(o, 0, rename->path, 0);\n+\t\tif (ren_src_was_dirty) {\n+\t\t\toutput(o, 1, _(\"Refusing to lose dirty file at %s\"),\n+\t\t\t rename->path);\n+\t\t}\n+\t\t/*\n+\t\t * Stupid double negatives in remove_file; it somehow manages\n+\t\t * to repeatedly mess me up. So, just for myself:\n+\t\t * 1) update_wd iff !ren_src_was_dirty.\n+\t\t * 2) no_wd iff !update_wd\n+\t\t * 3) so, no_wd == !!ren_src_was_dirty == ren_src_was_dirty\n+\t\t */\n+\t\tremove_file(o, 0, rename->path, ren_src_was_dirty);\n \t\tdst_name = unique_path(o, rename->path, cur_branch);\n \t} else {\n \t\tif (dir_in_way(rename->path, !o->call_depth, 0)) {\n@@ -1436,7 +1448,10 @@ static int conflict_rename_rename_2to1(struct merge_options *o,\n \t\tchar *new_path2 = unique_path(o, path, ci->branch2);\n \t\toutput(o, 1, _(\"Renaming %s to %s and %s to %s instead\"),\n \t\t a->path, new_path1, b->path, new_path2);\n-\t\tif (would_lose_untracked(path))\n+\t\tif (was_dirty(o, path))\n+\t\t\toutput(o, 1, _(\"Refusing to lose dirty file at %s\"),\n+\t\t\t path);\n+\t\telse if (would_lose_untracked(path))\n \t\t\t/*\n \t\t\t * Only way we get here is if both renames were from\n \t\t\t * a directory rename AND user had an untracked file\n@@ -2002,6 +2017,7 @@ static void apply_directory_rename_modifications(struct merge_options *o,\n {\n \tstruct string_list_item *item;\n \tint stage = (tree == a_tree ? 2 : 3);\n+\tint update_wd;\n \n \t/*\n \t * In all cases where we can do directory rename detection,\n@@ -2012,7 +2028,11 @@ static void apply_directory_rename_modifications(struct merge_options *o,\n \t * saying the file would have been overwritten), but it might\n \t * be dirty, though.\n \t */\n-\tremove_file(o, 1, pair->two->path, 0 /* no_wd */);\n+\tupdate_wd = !was_dirty(o, pair->two->path);\n+\tif (!update_wd)\n+\t\toutput(o, 1, _(\"Refusing to lose dirty file at %s\"),\n+\t\t pair->two->path);\n+\tremove_file(o, 1, pair->two->path, !update_wd);\n \n \t/* Find or create a new re->dst_entry */\n \titem = string_list_lookup(entries, new_path);\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex 02c97c9823..676e72e9e6 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -2985,7 +2985,7 @@ test_expect_success '11b-setup: Avoid losing dirty file involved in directory re\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '11b-check: Avoid losing dirty file involved in directory rename' '\n+test_expect_success '11b-check: Avoid losing dirty file involved in directory rename' '\n \tgit checkout B^0 &&\n \techo stuff >>z/c &&\n \n@@ -3109,7 +3109,7 @@ test_expect_success '11d-setup: Avoid losing not-uptodate with rename + D/F conf\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '11d-check: Avoid losing not-uptodate with rename + D/F conflict' '\n+test_expect_success '11d-check: Avoid losing not-uptodate with rename + D/F conflict' '\n \tgit checkout B^0 &&\n \techo stuff >>z/c &&\n \n@@ -3178,7 +3178,7 @@ test_expect_success '11e-setup: Avoid deleting not-uptodate with dir rename/rena\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '11e-check: Avoid deleting not-uptodate with dir rename/rename(1to2)/add' '\n+test_expect_success '11e-check: Avoid deleting not-uptodate with dir rename/rename(1to2)/add' '\n \tgit checkout B^0 &&\n \techo mods >>y/c &&\n \n@@ -3247,7 +3247,7 @@ test_expect_success '11f-setup: Avoid deleting not-uptodate with dir rename/rena\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '11f-check: Avoid deleting not-uptodate with dir rename/rename(2to1)' '\n+test_expect_success '11f-check: Avoid deleting not-uptodate with dir rename/rename(2to1)' '\n \tgit checkout B^0 &&\n \techo important >>y/wham &&\n \n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332238","messageId":"20171110190550.27059-22-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 21/30] merge-recursive: Add get_directory_renames()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:41Z","receivedAt":"2017-11-10T19:07:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This populates a list of directory renames for us. The list of\ndirectory renames is not yet used, but will be in subsequent commits.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 146 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 146 insertions(+)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 89a9b32635..b5770d3d7f 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -1376,6 +1376,124 @@ static struct diff_queue_struct *get_diffpairs(struct merge_options *o,\n \treturn ret;\n }\n \n+static void get_renamed_dir_portion(const char *old_path, const char *new_path,\n+\t\t\t\t char **old_dir, char **new_dir) {\n+\t*old_dir = NULL;\n+\t*new_dir = NULL;\n+\n+\t/* For\n+\t * \"a/b/c/d/foo.c\" -> \"a/b/something-else/d/foo.c\"\n+\t * the \"d/foo.c\" part is the same, we just want to know that\n+\t * \"a/b/c\" was renamed to \"a/b/something-else\"\n+\t * so, for this example, this function returns \"a/b/c\" in\n+\t * *old_dir and \"a/b/something-else\" in *new_dir.\n+\t *\n+\t * Also, if the basename of the file changed, we don't care. We\n+\t * want to know which portion of the directory, if any, changed.\n+\t */\n+\tchar *end_of_old = strrchr(old_path, '/');\n+\tchar *end_of_new = strrchr(new_path, '/');\n+\tif (end_of_old == NULL || end_of_new == NULL)\n+\t\treturn;\n+\twhile (*--end_of_new == *--end_of_old &&\n+\t end_of_old != old_path &&\n+\t end_of_new != new_path)\n+\t\t; // Do nothing; all in the while loop\n+\t/*\n+\t * We've found the first non-matching character in the directory\n+\t * paths. That means the current directory we were comparing\n+\t * represents the rename. Move end_of_old and end_of_new back\n+\t * to the full directory name.\n+\t */\n+\tif (*end_of_old == '/')\n+\t\tend_of_old++;\n+\tif (*end_of_old != '/')\n+\t\tend_of_new++;\n+\tend_of_old = strchr(end_of_old, '/');\n+\tend_of_new = strchr(end_of_new, '/');\n+\n+\t/*\n+\t * It may have been the case that old_path and new_path were the same\n+\t * directory all along. Don't claim a rename if they're the same.\n+\t */\n+\tint old_len = end_of_old - old_path;\n+\tint new_len = end_of_new - new_path;\n+\n+\tif (old_len != new_len || strncmp(old_path, new_path, old_len)) {\n+\t\t*old_dir = strndup(old_path, old_len);\n+\t\t*new_dir = strndup(new_path, new_len);\n+\t}\n+}\n+\n+static struct hashmap *get_directory_renames(struct diff_queue_struct *pairs,\n+\t\t\t\t\t struct tree *tree) {\n+\tstruct hashmap *dir_renames;\n+\tstruct hashmap_iter iter;\n+\tstruct dir_rename_entry *entry;\n+\tint i;\n+\n+\tdir_renames = malloc(sizeof(struct hashmap));\n+\tdir_rename_init(dir_renames);\n+\tfor (i = 0; i < pairs->nr; ++i) {\n+\t\tstruct string_list_item *item;\n+\t\tint *count;\n+\t\tstruct diff_filepair *pair = pairs->queue[i];\n+\n+\t\tchar *old_dir, *new_dir;\n+\t\tget_renamed_dir_portion(pair->one->path, pair->two->path,\n+\t\t\t\t\t&old_dir, &new_dir);\n+\t\tif (!old_dir)\n+\t\t\t// Directory didn't change at all; ignore this one.\n+\t\t\tcontinue;\n+\n+\t\tentry = dir_rename_find_entry(dir_renames, old_dir);\n+\t\tif (!entry) {\n+\t\t\tentry = xcalloc(1, sizeof(struct dir_rename_entry));\n+\t\t\thashmap_entry_init(entry, strhash(old_dir));\n+\t\t\thashmap_put(dir_renames, entry);\n+\t\t\tentry->dir = old_dir;\n+\t\t} else {\n+\t\t\tfree(old_dir);\n+\t\t}\n+\t\titem = string_list_lookup(&entry->possible_new_dirs, new_dir);\n+\t\tif (!item) {\n+\t\t\titem = string_list_insert(&entry->possible_new_dirs, new_dir);\n+\t\t\titem->util = xcalloc(1, sizeof(int));\n+\t\t} else {\n+\t\t\tfree(new_dir);\n+\t\t}\n+\t\tcount = item->util;\n+\t\t*count += 1;\n+\t}\n+\n+\thashmap_iter_init(dir_renames, &iter);\n+\twhile ((entry = hashmap_iter_next(&iter))) {\n+\t\tint max = 0;\n+\t\tint bad_max = 0;\n+\t\tchar *best = NULL;\n+\t\tfor (i = 0; i < entry->possible_new_dirs.nr; i++) {\n+\t\t\tint *count = entry->possible_new_dirs.items[i].util;\n+\t\t\tif (*count == max)\n+\t\t\t\tbad_max = max;\n+\t\t\telse if (*count > max) {\n+\t\t\t\tmax = *count;\n+\t\t\t\tbest = entry->possible_new_dirs.items[i].string;\n+\t\t\t}\n+\t\t}\n+\t\tif (bad_max == max)\n+\t\t\tentry->non_unique_new_dir = 1;\n+\t\telse\n+\t\t\tentry->new_dir = strdup(best);\n+\t\t/* Strings were strndup'ed before inserting into string-list,\n+\t\t * so ask string_list to remove the entries for us.\n+\t\t */\n+\t\tentry->possible_new_dirs.strdup_strings = 1;\n+\t\tstring_list_clear(&entry->possible_new_dirs, 1);\n+\t}\n+\n+\treturn dir_renames;\n+}\n+\n /*\n * Get information of all renames which occurred in 'pairs', making use of\n * any implicit directory renames inferred from the other side of history.\n@@ -1695,6 +1813,9 @@ static struct rename_info *handle_renames(struct merge_options *o,\n {\n \tstruct rename_info *rei = xcalloc(1, sizeof(struct rename_info));\n \tstruct diff_queue_struct *head_pairs, *merge_pairs;\n+\tstruct hashmap *dir_re_head, *dir_re_merge;\n+\tstruct hashmap_iter iter;\n+\tstruct dir_rename_entry *e;\n \n \t*clean = 1;\n \tif (!o->detect_rename)\n@@ -1703,6 +1824,9 @@ static struct rename_info *handle_renames(struct merge_options *o,\n \thead_pairs = get_diffpairs(o, common, head);\n \tmerge_pairs = get_diffpairs(o, common, merge);\n \n+\tdir_re_head = get_directory_renames(head_pairs, head);\n+\tdir_re_merge = get_directory_renames(merge_pairs, merge);\n+\n \trei->head_renames = get_renames(o, head_pairs, head,\n \t\t\t\t\t common, head, merge, entries);\n \trei->merge_renames = get_renames(o, merge_pairs, merge,\n@@ -1716,6 +1840,28 @@ static struct rename_info *handle_renames(struct merge_options *o,\n \t * process_entry(). But there are a few things we can free now.\n \t */\n \n+\thashmap_iter_init(dir_re_head, &iter);\n+\twhile ((e = hashmap_iter_next(&iter))) {\n+\t\tfree(e->dir);\n+\t\tif (e->new_dir)\n+\t\t\tfree(e->new_dir);\n+\t\t/* possible_new_dirs already cleared in get_directory_renames */\n+\t\t//string_list_clear(&e->possible_new_dirs, 1);\n+\t}\n+\thashmap_free(dir_re_head, 1);\n+\tfree(dir_re_head);\n+\n+\thashmap_iter_init(dir_re_merge, &iter);\n+\twhile ((e = hashmap_iter_next(&iter))) {\n+\t\tfree(e->dir);\n+\t\tif (e->new_dir)\n+\t\t\tfree(e->new_dir);\n+\t\t/* possible_new_dirs already cleared in get_directory_renames */\n+\t\t//string_list_clear(&e->possible_new_dirs, 1);\n+\t}\n+\thashmap_free(dir_re_merge, 1);\n+\tfree(dir_re_merge);\n+\n \tfree(head_pairs->queue);\n \tfree(head_pairs);\n \tfree(merge_pairs->queue);\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332239","messageId":"20171110190550.27059-26-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 25/30] merge-recursive: Check for file level conflicts then get new name","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:45Z","receivedAt":"2017-11-10T19:07:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Before trying to apply directory renames to paths within the given\ndirectories, we want to make sure that there aren't conflicts at the\nfile level either. If there aren't any, then get the new name from\nany directory renames.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 184 ++++++++++++++++++++++++++++++++++--\n t/t6043-merge-rename-directories.sh | 2 +-\n 2 files changed, 176 insertions(+), 10 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 251c4cc7fa..7c2c29bb51 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -1481,6 +1481,102 @@ static void get_renamed_dir_portion(const char *old_path, const char *new_path,\n \t}\n }\n \n+/*\n+ * Write:\n+ * element1, element2, element3, ..., elementN\n+ * to str. If only one element, just write \"element1\" to str.\n+ */\n+static void comma_separated_list(char *str, struct string_list *slist) {\n+\tint i;\n+\tfor (i=0; inr; i++) {\n+\t\tstr += sprintf(str, \"%s\", slist->items[i].string);\n+\t\tif (i < slist->nr-1)\n+\t\t\tstr += sprintf(str, \", \");\n+\t}\n+}\n+\n+/*\n+ * See if there is a directory rename for path, and if there are any file\n+ * level conflicts for the renamed location. If there is a rename and\n+ * there are no conflicts, return the new name. Otherwise, return NULL.\n+ */\n+static char* handle_path_level_conflicts(struct merge_options *o,\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t struct dir_rename_entry *entry,\n+\t\t\t\t\t struct hashmap *collisions,\n+\t\t\t\t\t struct tree *tree)\n+{\n+\tchar *new_path = NULL;\n+\tstruct collision_entry *collision_ent;\n+\tint clean = 1;\n+\n+\t/*\n+\t * entry has the mapping of old directory name to new directory name\n+\t * that we want to apply to path.\n+\t */\n+\tnew_path = apply_dir_rename(entry, path);\n+\n+\tif (!new_path) {\n+\t\t/* This should only happen when entry->non_unique_new_dir set */\n+\t\tassert(entry->non_unique_new_dir);\n+\t\toutput(o, 1, _(\"CONFLICT (directory rename split): \"\n+\t\t\t \"Unclear where to place %s because directory \"\n+\t\t\t \"%s was renamed to multiple other directories, \"\n+\t\t\t \"with no destination getting a majority of the \"\n+\t\t\t \"files.\"),\n+\t\t path, entry->dir);\n+\t\tclean = 0;\n+\t\treturn NULL;\n+\t}\n+\n+\t/*\n+\t * The caller needs to have ensured that it has pre-populated\n+\t * collisions with all paths that map to new_path. Do a quick check\n+\t * to ensure that's the case.\n+\t */\n+\tcollision_ent = collision_find_entry(collisions, new_path);\n+\tassert(collision_ent != NULL);\n+\n+\t/*\n+\t * Check for one-sided add/add/.../add conflicts, i.e.\n+\t * where implicit renames from the other side doing\n+\t * directory rename(s) can affect this side of history\n+\t * to put multiple paths into the same location. Warn\n+\t * and bail on directory renames for such paths.\n+\t */\n+\tchar collision_paths[(PATH_MAX+2) * collision_ent->source_files.nr];\n+\tif (collision_ent->reported_already) {\n+\t\tclean = 0;\n+\t} else if (tree_has_path(tree, new_path)) {\n+\t\tcollision_ent->reported_already = 1;\n+\t\tcomma_separated_list(collision_paths,\n+\t\t\t\t &collision_ent->source_files);\n+\t\toutput(o, 1, _(\"CONFLICT (implicit dir rename): Existing \"\n+\t\t\t \"file/dir at %s in the way of implicit \"\n+\t\t\t \"directory rename(s) putting the following \"\n+\t\t\t \"path(s) there: %s.\"),\n+\t\t new_path, collision_paths);\n+\t\tclean = 0;\n+\t} else if (collision_ent->source_files.nr > 1) {\n+\t\tcollision_ent->reported_already = 1;\n+\t\tcomma_separated_list(collision_paths,\n+\t\t\t\t &collision_ent->source_files);\n+\t\toutput(o, 1, _(\"CONFLICT (implicit dir rename): Cannot map \"\n+\t\t\t \"more than one path to %s; implicit directory \"\n+\t\t\t \"renames tried to put these paths there: %s\"),\n+\t\t new_path, collision_paths);\n+\t\tclean = 0;\n+\t}\n+\n+\t/* Free memory we no longer need */\n+\tif (!clean && new_path) {\n+\t\tfree(new_path);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn new_path;\n+}\n+\n /*\n * There are a couple things we want to do at the directory level:\n * 1. Check for both sides renaming to the same thing, in order to avoid\n@@ -1725,6 +1821,58 @@ static void compute_collisions(struct hashmap *collisions,\n \t}\n }\n \n+static char *check_for_directory_rename(struct merge_options *o,\n+\t\t\t\t\tconst char *path,\n+\t\t\t\t\tstruct tree *tree,\n+\t\t\t\t\tstruct hashmap *dir_renames,\n+\t\t\t\t\tstruct hashmap *dir_rename_exclusions,\n+\t\t\t\t\tstruct hashmap *collisions,\n+\t\t\t\t\tint *clean_merge)\n+{\n+\tchar *new_path = NULL;\n+\tstruct dir_rename_entry *entry = check_dir_renamed(path, dir_renames);\n+\tif (!entry)\n+\t\treturn new_path;\n+\n+\t/*\n+\t * This next part is a little weird. We do not want to do an\n+\t * implicit rename into a directory we renamed on our side, because\n+\t * that will result in a spurious rename/rename(1to2) conflict. An\n+\t * example:\n+\t * Base commit: dumbdir/afile, otherdir/bfile\n+\t * Side 1: smrtdir/afile, otherdir/bfile\n+\t * Side 2: dumbdir/afile, dumbdir/bfile\n+\t * Here, while working on Side 1, we could notice that otherdir was\n+\t * renamed/merged to dumbdir, and change the diff_filepair for\n+\t * otherdir/bfile into a rename into dumbdir/bfile. However, Side\n+\t * 2 will notice the rename from dumbdir to smrtdir, and do the\n+\t * transitive rename to move it from dumbdir/bfile to\n+\t * smrtdir/bfile. That gives us bfile in dumbdir vs being in\n+\t * smrtdir, a rename/rename(1to2) conflict. We really just want\n+\t * the file to end up in smrtdir. And the way to achieve that is\n+\t * to not let Side1 do the rename to dumbdir, since we know that is\n+\t * the source of one of our directory renames.\n+\t *\n+\t * That's why oentry and dir_rename_exclusions is here.\n+\t *\n+\t * As it turns out, this also prevents N-way transient rename\n+\t * confusion; See testcases 9c and 9d of t6043.\n+\t */\n+\tstruct dir_rename_entry *oentry = NULL;\n+\toentry = dir_rename_find_entry(dir_rename_exclusions, entry->new_dir);\n+\tif (oentry) {\n+\t\toutput(o, 1, _(\"WARNING: Avoiding applying %s -> %s rename \"\n+\t\t\t \"to %s, because %s itself was renamed.\"),\n+\t\t entry->dir, entry->new_dir, path, entry->new_dir);\n+\t} else {\n+\t\tnew_path = handle_path_level_conflicts(o, path, entry,\n+\t\t\t\t\t\t collisions, tree);\n+\t\t*clean_merge &= (new_path != NULL);\n+\t}\n+\n+\treturn new_path;\n+}\n+\n /*\n * Get information of all renames which occurred in 'pairs', making use of\n * any implicit directory renames inferred from the other side of history.\n@@ -1735,11 +1883,13 @@ static void compute_collisions(struct hashmap *collisions,\n static struct string_list *get_renames(struct merge_options *o,\n \t\t\t\t struct diff_queue_struct *pairs,\n \t\t\t\t struct hashmap *dir_renames,\n+\t\t\t\t struct hashmap *dir_rename_exclusions,\n \t\t\t\t struct tree *tree,\n \t\t\t\t struct tree *o_tree,\n \t\t\t\t struct tree *a_tree,\n \t\t\t\t struct tree *b_tree,\n-\t\t\t\t struct string_list *entries)\n+\t\t\t\t struct string_list *entries,\n+\t\t\t\t int *clean_merge)\n {\n \tint i;\n \tstruct hashmap collisions;\n@@ -1754,10 +1904,22 @@ static struct string_list *get_renames(struct merge_options *o,\n \t\tstruct string_list_item *item;\n \t\tstruct rename *re;\n \t\tstruct diff_filepair *pair = pairs->queue[i];\n-\t\tif (pair->status != 'R') {\n+\t\tchar *new_path; // non-NULL only with directory renames\n+\n+\t\tif (pair->status == 'D') {\n \t\t\tdiff_free_filepair(pair);\n \t\t\tcontinue;\n \t\t}\n+\t\tnew_path = check_for_directory_rename(o, pair->two->path, tree,\n+\t\t\t\t\t\t dir_renames,\n+\t\t\t\t\t\t dir_rename_exclusions,\n+\t\t\t\t\t\t &collisions,\n+\t\t\t\t\t\t clean_merge);\n+\t\tif (pair->status != 'R' && !new_path) {\n+\t\t\tdiff_free_filepair(pair);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tre = xmalloc(sizeof(*re));\n \t\tre->processed = 0;\n \t\tre->pair = pair;\n@@ -2075,12 +2237,18 @@ static struct rename_info *handle_renames(struct merge_options *o,\n \t\t\t\t\t dir_re_merge, merge);\n \n \trei->head_renames = get_renames(o, head_pairs,\n-\t\t\t\t\t dir_re_merge, head,\n-\t\t\t\t\t common, head, merge, entries);\n+\t\t\t\t\t dir_re_merge, dir_re_head, head,\n+\t\t\t\t\t common, head, merge, entries,\n+\t\t\t\t\t clean);\n+\tif (*clean < 0)\n+\t\tgoto cleanup;\n \trei->merge_renames = get_renames(o, merge_pairs,\n-\t\t\t\t\t dir_re_head, merge,\n-\t\t\t\t\t common, head, merge, entries);\n-\t*clean = process_renames(o, rei->head_renames, rei->merge_renames);\n+\t\t\t\t\t dir_re_head, dir_re_merge, merge,\n+\t\t\t\t\t common, head, merge, entries,\n+\t\t\t\t\t clean);\n+\tif (*clean < 0)\n+\t\tgoto cleanup;\n+\t*clean &= process_renames(o, rei->head_renames, rei->merge_renames);\n \n cleanup:\n \t/*\n@@ -2095,7 +2263,6 @@ static struct rename_info *handle_renames(struct merge_options *o,\n \t\tif (e->new_dir)\n \t\t\tfree(e->new_dir);\n \t\t/* possible_new_dirs already cleared in get_directory_renames */\n-\t\t//string_list_clear(&e->possible_new_dirs, 1);\n \t}\n \thashmap_free(dir_re_head, 1);\n \tfree(dir_re_head);\n@@ -2106,7 +2273,6 @@ static struct rename_info *handle_renames(struct merge_options *o,\n \t\tif (e->new_dir)\n \t\t\tfree(e->new_dir);\n \t\t/* possible_new_dirs already cleared in get_directory_renames */\n-\t\t//string_list_clear(&e->possible_new_dirs, 1);\n \t}\n \thashmap_free(dir_re_merge, 1);\n \tfree(dir_re_merge);\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nindex 4066b08767..858d83016a 100755\n--- a/t/t6043-merge-rename-directories.sh\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -438,7 +438,7 @@ test_expect_success '2a-setup: Directory split into two on one side, with equal\n \tgit commit -m \"C\"\n '\n \n-test_expect_failure '2a-check: Directory split into two on one side, with equal numbers of paths' '\n+test_expect_success '2a-check: Directory split into two on one side, with equal numbers of paths' '\n \tgit checkout B^0 &&\n \n \ttest_must_fail git merge -s recursive C^0 >out &&\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332240","messageId":"20171110190550.27059-21-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 20/30] merge-recursive: Add a new hashmap for storing directory renames","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:40Z","receivedAt":"2017-11-10T19:07:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This just adds dir_rename_entry and the associated functions; code using\nthese will be added in subsequent commits.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 24 ++++++++++++++++++++++++\n merge-recursive.h | 8 ++++++++\n 2 files changed, 32 insertions(+)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 8c9543d85c..89a9b32635 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -49,6 +49,30 @@ static unsigned int path_hash(const char *path)\n \treturn ignore_case ? strihash(path) : strhash(path);\n }\n \n+static struct dir_rename_entry *dir_rename_find_entry(struct hashmap *hashmap, char *dir)\n+{\n+\tstruct dir_rename_entry key;\n+\n+\tif (dir == NULL)\n+\t\treturn NULL;\n+\thashmap_entry_init(&key, strhash(dir));\n+\tkey.dir = dir;\n+\treturn hashmap_get(hashmap, &key, NULL);\n+}\n+\n+static int dir_rename_cmp(void *unused_cmp_data,\n+\t\t\t const struct dir_rename_entry *e1,\n+\t\t\t const struct dir_rename_entry *e2,\n+\t\t\t const void *unused_keydata)\n+{\n+\treturn strcmp(e1->dir, e2->dir);\n+}\n+\n+static void dir_rename_init(struct hashmap *map)\n+{\n+\thashmap_init(map, (hashmap_cmp_fn) dir_rename_cmp, NULL, 0);\n+}\n+\n static void flush_output(struct merge_options *o)\n {\n \tif (o->buffer_output < 2 && o->obuf.len) {\ndiff --git a/merge-recursive.h b/merge-recursive.h\nindex 80d69d1401..a024949739 100644\n--- a/merge-recursive.h\n+++ b/merge-recursive.h\n@@ -29,6 +29,14 @@ struct merge_options {\n \tstruct string_list df_conflict_file_set;\n };\n \n+struct dir_rename_entry {\n+\tstruct hashmap_entry ent; /* must be the first member! */\n+\tchar *dir;\n+\tunsigned non_unique_new_dir:1;\n+\tchar *new_dir;\n+\tstruct string_list possible_new_dirs;\n+};\n+\n /* merge_trees() but with recursive ancestor consolidation */\n int merge_recursive(struct merge_options *o,\n \t\t struct commit *h1,\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332241","messageId":"20171110190550.27059-3-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 02/30] merge-recursive: Fix logic ordering issue","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:22Z","receivedAt":"2017-11-10T19:07:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"merge_trees() did a variety of work, including:\n * Calling get_unmerged() to get unmerged entries\n * Calling record_df_conflict_files() with all unmerged entries to\n do some work to ensure we could handle D/F conflicts correctly\n * Calling get_renames() to check for renames.\n\nAn easily overlooked issue is that get_renames() can create more\nunmerged entries and add them to the list, which have the possibility of\nbeing involved in D/F conflicts. So the call to\nrecord_df_conflict_files() should really be moved after all the rename\ndetection. I didn't come up with any testcases demonstrating any bugs\nwith the old ordering, but I suspect there were some for both normal\nrenames and for directory renames. Fix the ordering.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 1d3f8f0d22..52521faf09 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -1981,10 +1981,10 @@ int merge_trees(struct merge_options *o,\n \t\tget_files_dirs(o, merge);\n \n \t\tentries = get_unmerged();\n-\t\trecord_df_conflict_files(o, entries);\n \t\tre_head = get_renames(o, head, common, head, merge, entries);\n \t\tre_merge = get_renames(o, merge, common, head, merge, entries);\n \t\tclean = process_renames(o, re_head, re_merge);\n+\t\trecord_df_conflict_files(o, entries);\n \t\tif (clean < 0)\n \t\t\tgoto cleanup;\n \t\tfor (i = entries->nr-1; 0 <= i; i--) {\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332242","messageId":"20171110190550.27059-5-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 04/30] directory rename detection: basic testcases","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:24Z","receivedAt":"2017-11-10T19:07:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n t/t6043-merge-rename-directories.sh | 391 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 391 insertions(+)\n create mode 100755 t/t6043-merge-rename-directories.sh\n\ndiff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\nnew file mode 100755\nindex 0000000000..b737b0a105\n--- /dev/null\n+++ b/t/t6043-merge-rename-directories.sh\n@@ -0,0 +1,391 @@\n+#!/bin/sh\n+\n+test_description=\"recursive merge with directory renames\"\n+# includes checking of many corner cases, with a similar methodology to:\n+# t6042: corner cases with renames but not criss-cross merges\n+# t6036: corner cases with both renames and criss-cross merges\n+#\n+# The setup for all of them, pictorially, is:\n+#\n+# B\n+# o\n+# / \\\n+# A o ?\n+# \\ /\n+# o\n+# C\n+#\n+# To help make it easier to follow the flow of tests, they have been\n+# divided into sections and each test will start with a quick explanation\n+# of what commits A, B, and C contain.\n+#\n+# Notation:\n+# z/{b,c} means files z/b and z/c both exist\n+# x/d_1 means file x/d exists with content d1. (Purpose of the\n+# underscore notation is to differentiate different\n+# files that might be renamed into each other's paths.)\n+\n+. ./test-lib.sh\n+\n+\n+###########################################################################\n+# SECTION 1: Basic cases we should be able to handle\n+###########################################################################\n+\n+# Testcase 1a, Basic directory rename.\n+# Commit A: z/{b,c}\n+# Commit B: y/{b,c}\n+# Commit C: z/{b,c,d,e/f}\n+# Expected: y/{b,c,d,e/f}\n+\n+test_expect_success '1a-setup: Simple directory rename detection' '\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z y &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\techo d >z/d &&\n+\tmkdir z/e &&\n+\techo f >z/e/f &&\n+\tgit add z/d z/e/f &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '1a-check: Simple directory rename detection' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 4 -eq $(git ls-files -s | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse HEAD:y/d) = $(git rev-parse C:z/d) &&\n+\ttest \"$(git hash-object y/d)\" = $(git rev-parse C:z/d) &&\n+\ttest $(git rev-parse HEAD:y/e/f) = $(git rev-parse C:z/e/f) &&\n+\ttest_must_fail git rev-parse HEAD:z/d &&\n+\ttest_must_fail git rev-parse HEAD:z/e/f &&\n+\ttest ! -d z/d &&\n+\ttest ! -d z/e/f\n+'\n+\n+# Testcase 1b, Merge a directory with another\n+# Commit A: z/{b,c}, y/d\n+# Commit B: z/{b,c,e}, y/d\n+# Commit C: y/{b,c,d}\n+# Expected: y/{b,c,d,e}\n+\n+test_expect_success '1b-setup: Merge a directory with another' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tmkdir y &&\n+\techo d >y/d &&\n+\tgit add z y &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\techo e >z/e &&\n+\tgit add z/e &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv z/b y &&\n+\tgit mv z/c y &&\n+\trmdir z &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '1b-check: Merge a directory with another' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 4 -eq $(git ls-files -s | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse HEAD:y/d) = $(git rev-parse A:y/d) &&\n+\ttest $(git rev-parse HEAD:y/e) = $(git rev-parse B:z/e) &&\n+\ttest_must_fail git rev-parse HEAD:z/e\n+'\n+\n+# Testcase 1c, Transitive renaming\n+# (Related to testcases 3a and 6d -- when should a transitive rename apply?)\n+# (Related to testcases 9c and 9d -- can transitivity repeat?)\n+# Commit A: z/{b,c}, x/d\n+# Commit B: y/{b,c}, x/d\n+# Commit C: z/{b,c,d}\n+# Expected: y/{b,c,d} (because x/d -> z/d -> y/d)\n+\n+test_expect_success '1c-setup: Transitive renaming' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tmkdir x &&\n+\techo d >x/d &&\n+\tgit add z x &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z y &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv x/d z/d &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '1c-check: Transitive renaming' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 3 -eq $(git ls-files -s | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse HEAD:y/d) = $(git rev-parse A:x/d) &&\n+\ttest_must_fail git rev-parse HEAD:x/d &&\n+\ttest_must_fail git rev-parse HEAD:z/d &&\n+\ttest ! -f z/d\n+'\n+\n+# Testcase 1d, Directory renames (merging two directories into one new one)\n+# cause a rename/rename(2to1) conflict\n+# (Related to testcases 1c and 7b)\n+# Commit A. z/{b,c}, y/{d,e}\n+# Commit B. x/{b,c}, y/{d,e,m,wham}\n+# Commit C. z/{b,c,n,wham}, x/{d,e}\n+# Expected: x/{b,c,d,e,m,n}, CONFLICT:(y/wham & z/wham -> x/wham)\n+# Note: y/m & z/n should definitely move into x. By the same token, both\n+# y/wham & z/wham should to...giving us a conflict.\n+\n+test_expect_success '1d-setup: Directory renames cause a rename/rename(2to1) conflict' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\tmkdir y &&\n+\techo d >y/d &&\n+\techo e >y/e &&\n+\tgit add z y &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tgit mv z x &&\n+\techo m >y/m &&\n+\techo wham1 >y/wham &&\n+\tgit add y &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tgit mv y x &&\n+\techo n >z/n &&\n+\techo wham2 >z/wham &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '1d-check: Directory renames cause a rename/rename(2to1) conflict' '\n+\tgit checkout B^0 &&\n+\n+\ttest_must_fail git merge -s recursive C^0 >out &&\n+\ttest_i18ngrep \"CONFLICT (rename/rename)\" out &&\n+\n+\ttest 8 -eq $(git ls-files -s | wc -l) &&\n+\ttest 2 -eq $(git ls-files -u | wc -l) &&\n+\ttest 3 -eq $(git ls-files -o | wc -l) &&\n+\n+\ttest $(git rev-parse :0:x/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse :0:x/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse :0:x/d) = $(git rev-parse A:y/d) &&\n+\ttest $(git rev-parse :0:x/e) = $(git rev-parse A:y/e) &&\n+\ttest $(git rev-parse :0:x/m) = $(git rev-parse B:y/m) &&\n+\ttest $(git rev-parse :0:x/n) = $(git rev-parse C:z/n) &&\n+\n+\ttest_must_fail git rev-parse :0:x/wham &&\n+\ttest $(git rev-parse :2:x/wham) = $(git rev-parse B:y/wham) &&\n+\ttest $(git rev-parse :3:x/wham) = $(git rev-parse C:z/wham) &&\n+\n+\ttest ! -f x/wham &&\n+\ttest -f x/wham~HEAD &&\n+\ttest -f x/wham~C^0 &&\n+\n+\ttest $(git hash-object x/wham~HEAD) = $(git rev-parse B:y/wham) &&\n+\ttest $(git hash-object x/wham~C^0) = $(git rev-parse C:z/wham)\n+'\n+\n+# Testcase 1e, Renamed directory, with all filenames being renamed too\n+# Commit A: z/{oldb,oldc}\n+# Commit B: y/{newb,newc}\n+# Commit C: z/{oldb,oldc,d}\n+# Expected: y/{newb,newc,d}\n+\n+test_expect_success '1e-setup: Renamed directory, with all files being renamed too' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/oldb &&\n+\techo c >z/oldc &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\tmkdir y &&\n+\tgit mv z/oldb y/newb &&\n+\tgit mv z/oldc y/newc &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\techo d >z/d &&\n+\tgit add z/d &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '1e-check: Renamed directory, with all files being renamed too' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 3 -eq $(git ls-files -s | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/newb) = $(git rev-parse A:z/oldb) &&\n+\ttest $(git rev-parse HEAD:y/newc) = $(git rev-parse A:z/oldc) &&\n+\ttest $(git rev-parse HEAD:y/d) = $(git rev-parse C:z/d) &&\n+\ttest_must_fail git rev-parse HEAD:z/d\n+'\n+\n+# Testcase 1f, Split a directory into two other directories\n+# (Related to testcases 3a, all of section 2, and all of section 4)\n+# Commit A: z/{b,c,d,e,f}\n+# Commit B: z/{b,c,d,e,f,g}\n+# Commit C: y/{b,c}, x/{d,e,f}\n+# Expected: y/{b,c}, x/{d,e,f,g}\n+\n+test_expect_success '1f-setup: Split a directory into two other directories' '\n+\tgit rm -rf . &&\n+\tgit clean -fdqx &&\n+\trm -rf .git &&\n+\tgit init &&\n+\n+\tmkdir z &&\n+\techo b >z/b &&\n+\techo c >z/c &&\n+\techo d >z/d &&\n+\techo e >z/e &&\n+\techo f >z/f &&\n+\tgit add z &&\n+\ttest_tick &&\n+\tgit commit -m \"A\" &&\n+\n+\tgit branch A &&\n+\tgit branch B &&\n+\tgit branch C &&\n+\n+\tgit checkout B &&\n+\techo g >z/g &&\n+\tgit add z/g &&\n+\ttest_tick &&\n+\tgit commit -m \"B\" &&\n+\n+\tgit checkout C &&\n+\tmkdir y &&\n+\tmkdir x &&\n+\tgit mv z/b y/ &&\n+\tgit mv z/c y/ &&\n+\tgit mv z/d x/ &&\n+\tgit mv z/e x/ &&\n+\tgit mv z/f x/ &&\n+\trmdir z &&\n+\ttest_tick &&\n+\tgit commit -m \"C\"\n+'\n+\n+test_expect_failure '1f-check: Split a directory into two other directories' '\n+\tgit checkout B^0 &&\n+\n+\tgit merge -s recursive C^0 &&\n+\n+\ttest 6 -eq $(git ls-files -s | wc -l) &&\n+\n+\ttest $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n+\ttest $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n+\ttest $(git rev-parse HEAD:x/d) = $(git rev-parse A:z/d) &&\n+\ttest $(git rev-parse HEAD:x/e) = $(git rev-parse A:z/e) &&\n+\ttest $(git rev-parse HEAD:x/f) = $(git rev-parse A:z/f) &&\n+\ttest $(git rev-parse HEAD:x/g) = $(git rev-parse B:z/g) &&\n+\ttest ! -f z/g &&\n+\ttest_must_fail git rev-parse HEAD:z/g\n+'\n+\n+###########################################################################\n+# Rules suggested by testcases in section 1:\n+#\n+# We should still detect the directory rename even if it wasn't just\n+# the directory renamed, but the files within it. (see 1b)\n+#\n+# If renames split a directory into two or more others, the directory\n+# with the most renames, \"wins\" (see 1c). However, see the testcases\n+# in section 2, plus testcases 3a and 4a.\n+###########################################################################\n+\n+test_done\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332243","messageId":"20171110190550.27059-4-newren@gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"[PATCH 03/30] merge-recursive: Add explanation for src_entry and dst_entry","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T19:05:23Z","receivedAt":"2017-11-10T19:07:40Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"If I have to walk through the debugger and inspect the values found in\nhere in order to figure out their meaning, despite having known these\nthings inside and out some years back, then they probably need a comment\nfor the casual reader to explain their purpose.\n\nSigned-off-by: Elijah Newren \n---\n merge-recursive.c | 22 ++++++++++++++++++++++\n 1 file changed, 22 insertions(+)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 52521faf09..3526c8d0b8 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -513,6 +513,28 @@ static void record_df_conflict_files(struct merge_options *o,\n \n struct rename {\n \tstruct diff_filepair *pair;\n+\t/*\n+\t * Because I keep forgetting every few years what src_entry and\n+\t * dst_entry are and have to walk through a debugger and puzzle\n+\t * through it to remind myself...\n+\t *\n+\t * If 'before' is renamed to 'after' then src_entry will contain\n+\t * the versions of 'before' from the merge_base, HEAD, and MERGE in\n+\t * stages 1, 2, and 3; dst_entry will contain the versions of\n+\t * 'after' from the merge_base, HEAD, and MERGE in stages 1, 2, and\n+\t * 3. Thus, we have a total of six modes and oids, though some\n+\t * will be null. (Stage 0 is ignored; we're interested in handling\n+\t * conflicts.)\n+\t *\n+\t * Since we don't turn on break-rewrites by default, neither\n+\t * src_entry nor dst_entry can have all three of their stages have\n+\t * non-null oids, meaning at most four of the six will be non-null.\n+\t * Also, since this is a rename, both src_entry and dst_entry will\n+\t * have at least one non-null oid, meaning at least two will be\n+\t * non-null. Of the six oids, a typical rename will have three be\n+\t * non-null. Only two implies a rename/delete, and four implies a\n+\t * rename/add.\n+\t */\n \tstruct stage_data *src_entry;\n \tstruct stage_data *dst_entry;\n \tunsigned processed:1;\n-- \n2.15.0.5.g9567be9905\n\n"},{"id":"332261","messageId":"3076EE0E678D43B286F86E622CB9F06E@PhilipOakley","threadId":"47162","inReplyTo":"20171110190550.27059-1-newren@gmail.com","subject":"Re: [PATCH 00/30] Add directory rename detection to git","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-11-10T22:27:19Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Elijah Newren\" \n> [This series is entirely independent of my rename detection limits series.\n> However, I have a separate rename detection performance series that \n> depends\n> on both this series and the rename detection limits series.]\n>\n> In this patchset, I introduce directory rename detection to \n> merge-recursive,\n> predominantly so that when files are added to directories on one side of\n> history and those directories are renamed on the other side of history, \n> the\n> files will end up in the proper location after a merge or cherry-pick.\n>\n> However, this isn't limited to that simplistic case. More interesting\n> possibilities exist, such as:\n>\n> * a file being renamed into a directory which is renamed on the other\n> side of history, causing the need for a transitive rename.\n>\n\nHow does this cope with the case insensitive case preserving file systems on \nMac and Windows, esp when core.ignorecase is true. If it's a bigger problem \nthat the series already covers, would the likely changes be reasonably \nlocalised?\n\nThis came up recently on GfW for `git checkout` of a branch where the case \nchanged (\"Test\" <-> \"test\"), but git didn't notice that it needed to rename \nthe directories on such an file system. \nhttps://github.com/git-for-windows/git/issues/1333\n\n\n\n--\nPhilip \n\n"},{"id":"332267","messageId":"CABPp-BE1+32fQQdYuwP-7m+GXqyhoUpHKgMM6SCcW4e9geutvw@mail.gmail.com","threadId":"47162","inReplyTo":"3076EE0E678D43B286F86E622CB9F06E@PhilipOakley","subject":"Re: [PATCH 00/30] Add directory rename detection to git","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-10T23:26:24Z","receivedAt":"2017-11-10T23:26:31Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Nov 10, 2017 at 2:27 PM, Philip Oakley wrote:\n> From: \"Elijah Newren\" \n>>\n>> In this patchset, I introduce directory rename detection to\n>> merge-recursive,\n>> predominantly so that when files are added to directories on one side of\n>> history and those directories are renamed on the other side of history,\n>> the\n>> files will end up in the proper location after a merge or cherry-pick.\n>>\n>> However, this isn't limited to that simplistic case. More interesting\n>> possibilities exist, such as:\n>>\n>> * a file being renamed into a directory which is renamed on the other\n>> side of history, causing the need for a transitive rename.\n>>\n>\n> How does this cope with the case insensitive case preserving file systems on\n> Mac and Windows, esp when core.ignorecase is true. If it's a bigger problem\n> that the series already covers, would the likely changes be reasonably\n> localised?\n>\n> This came up recently on GfW for `git checkout` of a branch where the case\n> changed (\"Test\" <-> \"test\"), but git didn't notice that it needed to rename\n> the directories on such an file system.\n> https://github.com/git-for-windows/git/issues/1333\n\nI wasn't aware there were problems with git on case insensitive case\npreserving filesystems; fixing them wasn't something I had in mind\nwhen writing this series. However, the particular bug you mention is\nactually completely orthogonal to this series; it talks about\ngit-checkout without the -m/--merge option, which doesn't touch any\ncode path I modified in my series, so my series can't really fix or\nworsen that particular issue.\n\nBut, if there are further issues with such filesystems that also\naffect merges/cherry-picks/rebases, then I don't think my series will\neither help or hurt there either. The recursive merge machinery\nalready has remove_file() and update_file() wrappers that it uses\nwhenever it needs to remove/add/update a file in the working directory\nand/or index, and I have simply continued using those, so the number\nof places you'd need to modify to fix issues would remain just as\nlocalized as before. Also, I continue to depend on the reading of the\nindex & trees that unpack_trees() does, which I haven't modified, so\nagain it'd be the same number of places that someone would need to\nfix. (However, the whole design to have unpack_trees() do the initial\nwork and then have recursive merge try to \"fix it up\" is really\nstarting to strain. I'm starting to think, again, that merge\nrecursive needs a redesign, and have some arguments I wanted to float\nout there...but I've dumped enough on the list for a day.)\n\nIt's possible that this series fixes one particular issue -- namely\nwhen merging, if the merge-base contained a \"Test\" directory, one side\nadded a file to that directory, and the other side renamed \"Test\" to\n\"test\", and if the presence of both \"Test\" and \"test\" directories in\nthe merge result is problematic, then at least with my fixes you\nwouldn't end up with both directories and could thus avoid that\nproblem in a narrow set of cases.\n\nSorry that I don't have any better news than that for you.\n\nElijah\n"},{"id":"332417","messageId":"0CA45D65AEDF405C987769D865EE0836@PhilipOakley","threadId":"47162","inReplyTo":"CABPp-BE1+32fQQdYuwP-7m+GXqyhoUpHKgMM6SCcW4e9geutvw@mail.gmail.com","subject":"Re: [PATCH 00/30] Add directory rename detection to git","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-11-13T15:04:52Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Elijah Newren\" \n: Friday, November 10, 2017 11:26 PM\n> On Fri, Nov 10, 2017 at 2:27 PM, Philip Oakley \n> wrote:\n>> From: \"Elijah Newren\" \n>>>\n>>> In this patchset, I introduce directory rename detection to\n>>> merge-recursive,\n>>> predominantly so that when files are added to directories on one side of\n>>> history and those directories are renamed on the other side of history,\n>>> the\n>>> files will end up in the proper location after a merge or cherry-pick.\n>>>\n>>> However, this isn't limited to that simplistic case. More interesting\n>>> possibilities exist, such as:\n>>>\n>>> * a file being renamed into a directory which is renamed on the other\n>>> side of history, causing the need for a transitive rename.\n>>>\n>>\n>> How does this cope with the case insensitive case preserving file systems \n>> on\n>> Mac and Windows, esp when core.ignorecase is true. If it's a bigger \n>> problem\n>> that the series already covers, would the likely changes be reasonably\n>> localised?\n>>\n>> This came up recently on GfW for `git checkout` of a branch where the \n>> case\n>> changed (\"Test\" <-> \"test\"), but git didn't notice that it needed to \n>> rename\n>> the directories on such an file system.\n>> https://github.com/git-for-windows/git/issues/1333\n>\n> I wasn't aware there were problems with git on case insensitive case\n> preserving filesystems; fixing them wasn't something I had in mind\n> when writing this series.\n\nI was mainly ensuring awareness of the potential issue, as it's not easy to \nsolve.\n\n> However, the particular bug you mention is\n> actually completely orthogonal to this series; it talks about\n> git-checkout without the -m/--merge option, which doesn't touch any\n> code path I modified in my series, so my series can't really fix or\n> worsen that particular issue.\n\nThat's good.\n>\n> But, if there are further issues with such filesystems that also\n> affect merges/cherry-picks/rebases, then I don't think my series will\n> either help or hurt there either. The recursive merge machinery\n> already has remove_file() and update_file() wrappers that it uses\n> whenever it needs to remove/add/update a file in the working directory\n> and/or index, and I have simply continued using those, so the number\n> of places you'd need to modify to fix issues would remain just as\n> localized as before.\n\nIt's when the working directory path/filename has a case change that goes \nundetected (one way or another) that can cause issues. I think that part of \nthe problem (after awareness) is not having a cannonical expectation of \nwhich way is 'right', and what options there may be. E,g. if a project is \nwholly on a case insensitive system then the filenames in the worktree never \nmatter, but aligning the path/filenames in the repository would still be a \nproblem.\n\n> Also, I continue to depend on the reading of the\n> index & trees that unpack_trees() does, which I haven't modified, so\n> again it'd be the same number of places that someone would need to\n> fix. (However, the whole design to have unpack_trees() do the initial\n> work and then have recursive merge try to \"fix it up\" is really\n> starting to strain.\n\nInteresting point.\n\n> I'm starting to think, again, that merge\n> recursive needs a redesign, and have some arguments I wanted to float\n> out there...but I've dumped enough on the list for a day.)\n>\n> It's possible that this series fixes one particular issue -- namely\n> when merging, if the merge-base contained a \"Test\" directory, one side\n> added a file to that directory, and the other side renamed \"Test\" to\n> \"test\", and if the presence of both \"Test\" and \"test\" directories in\n> the merge result is problematic, then at least with my fixes you\n> wouldn't end up with both directories and could thus avoid that\n> problem in a narrow set of cases.\n\nI'll think on that. It may provide extra clues as to what the right \nsolutions could be!\n>\n> Sorry that I don't have any better news than that for you.\n>\n> Elijah\n\nThanks\n--\nPhilip \n\n"},{"id":"332432","messageId":"CAGZ79kZtOjB43d9kT8yjnwA9ii60VsyzRxM7XHQBtMs70S2zGQ@mail.gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-2-newren@gmail.com","subject":"Re: [PATCH 01/30] Tighten and correct a few testcases for merging and cherry-picking","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-13T19:32:42Z","receivedAt":"2017-11-13T19:32:48Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n> t3501 had a testcase originally added\n\n... goes and looks ...\n\n \"in 05f2dfb965 (cherry-pick: demonstrate a segmentation fault, 2016-11-26)\"\n\nwould have helped me here in the commit message.\n\n\n> to ensure cherry-pick wouldn't\n> segfault when working with a dirty file involved in a rename. While\n> the segfault was fixed, there was another problem this test demonstrated:\n> namely, that git would overwrite a dirty file involved in a rename.\n> Further, the test encoded a \"successful merge\" and overwriting of this\n> file as correct behavior. Modify the test so that it would still catch\n> the segfault, but to require the correct behavior.\n\nAs the correct behavior is not yet implemented, mark it as\ntest_expect_failure, too. (probably this reads implicit)\n\n\n>\n> t7607 had a test\n\n... added in 30fd3a5425 (merge overwrites unstaged changes in renamed file,\n2012-04-15) ...\n\n> specific to looking for a merge overwriting a dirty file\n> involved in a rename, but it too actually encoded what I would term\n> incorrect behavior: it expected the merge to succeed. Fix that, and add\n> a few more checks to make sure that the merge really does produce the\n> expected results.\n>\n> Signed-off-by: Elijah Newren \n> ---\n> t/t3501-revert-cherry-pick.sh | 7 +++++--\n> t/t7607-merge-overwrite.sh | 5 ++++-\n> 2 files changed, 9 insertions(+), 3 deletions(-)\n>\n> diff --git a/t/t3501-revert-cherry-pick.sh b/t/t3501-revert-cherry-pick.sh\n> index 4f2a263b63..783bdbf59d 100755\n> --- a/t/t3501-revert-cherry-pick.sh\n> +++ b/t/t3501-revert-cherry-pick.sh\n> @@ -141,7 +141,7 @@ test_expect_success 'cherry-pick \"-\" works with arguments' '\n> test_cmp expect actual\n> '\n>\n> -test_expect_success 'cherry-pick works with dirty renamed file' '\n> +test_expect_failure 'cherry-pick works with dirty renamed file' '\n> test_commit to-rename &&\n> git checkout -b unrelated &&\n> test_commit unrelated &&\n> @@ -150,7 +150,10 @@ test_expect_success 'cherry-pick works with dirty renamed file' '\n> test_tick &&\n> git commit -m renamed &&\n> echo modified >renamed &&\n> - git cherry-pick refs/heads/unrelated\n> + test_must_fail git cherry-pick refs/heads/unrelated >out &&\n> + test_i18ngrep \"Refusing to lose dirty file at renamed\" out &&\n> + test $(git rev-parse :0:renamed) = $(git rev-parse HEAD^:to-rename.t) &&\n> + grep -q \"^modified$\" renamed\n> '\n>\n> test_done\n> diff --git a/t/t7607-merge-overwrite.sh b/t/t7607-merge-overwrite.sh\n> index 9444d6a9b9..00617dadf8 100755\n> --- a/t/t7607-merge-overwrite.sh\n> +++ b/t/t7607-merge-overwrite.sh\n> @@ -97,7 +97,10 @@ test_expect_failure 'will not overwrite unstaged changes in renamed file' '\n> git mv c1.c other.c &&\n> git commit -m rename &&\n> cp important other.c &&\n> - git merge c1a &&\n> + test_must_fail git merge c1a >out &&\n> + test_i18ngrep \"Refusing to lose dirty file at other.c\" out &&\n> + test -f other.c~HEAD &&\n> + test $(git hash-object other.c~HEAD) = $(git rev-parse c1a:c1.c) &&\n> test_cmp important other.c\n\nCode looks good,\n\nThanks,\nStefan\n"},{"id":"332434","messageId":"CAGZ79kYXbvDL_SoEENf60DsDFA-yEWfvgv8bNv_v+mw042ZH=w@mail.gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-3-newren@gmail.com","subject":"Re: [PATCH 02/30] merge-recursive: Fix logic ordering issue","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-13T19:48:04Z","receivedAt":"2017-11-13T19:48:16Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n> merge_trees() did a variety of work, including:\n> * Calling get_unmerged() to get unmerged entries\n> * Calling record_df_conflict_files() with all unmerged entries to\n> do some work to ensure we could handle D/F conflicts correctly\n> * Calling get_renames() to check for renames.\n>\n> An easily overlooked issue is that get_renames() can create more\n> unmerged entries and add them to the list, which have the possibility of\n> being involved in D/F conflicts.\n\nI presume these are created via insert_stage_data called in\nget_renames, when the path entry is not found?\n\n> So the call to\n> record_df_conflict_files() should really be moved after all the rename\n> detection. I didn't come up with any testcases demonstrating any bugs\n> with the old ordering, but I suspect there were some for both normal\n> renames and for directory renames. Fix the ordering.\n\nIt is hard to trace this down, though looking at\n3af244caa8 (Cumulative update of merge-recursive in C, 2006-07-27)\nmay help us reason about it.\n\nHow would a bug look like?\n\n>\n> Signed-off-by: Elijah Newren \n> ---\n> merge-recursive.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/merge-recursive.c b/merge-recursive.c\n> index 1d3f8f0d22..52521faf09 100644\n> --- a/merge-recursive.c\n> +++ b/merge-recursive.c\n> @@ -1981,10 +1981,10 @@ int merge_trees(struct merge_options *o,\n> get_files_dirs(o, merge);\n>\n> entries = get_unmerged();\n> - record_df_conflict_files(o, entries);\n> re_head = get_renames(o, head, common, head, merge, entries);\n> re_merge = get_renames(o, merge, common, head, merge, entries);\n> clean = process_renames(o, re_head, re_merge);\n> + record_df_conflict_files(o, entries);\n> if (clean < 0)\n> goto cleanup;\n> for (i = entries->nr-1; 0 <= i; i--) {\n> --\n> 2.15.0.5.g9567be9905\n>\n"},{"id":"332445","messageId":"CAGZ79kZ7d7fMp3Cw3Yy4bq9-Mh5Z_jhM5XDQbkVLoqzm6ONzDg@mail.gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-4-newren@gmail.com","subject":"Re: [PATCH 03/30] merge-recursive: Add explanation for src_entry and dst_entry","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-13T21:06:37Z","receivedAt":"2017-11-13T21:06:45Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n> If I have to walk through the debugger and inspect the values found in\n> here in order to figure out their meaning, despite having known these\n> things inside and out some years back, then they probably need a comment\n> for the casual reader to explain their purpose.\n>\n> Signed-off-by: Elijah Newren \n> ---\n> merge-recursive.c | 22 ++++++++++++++++++++++\n> 1 file changed, 22 insertions(+)\n>\n> diff --git a/merge-recursive.c b/merge-recursive.c\n> index 52521faf09..3526c8d0b8 100644\n> --- a/merge-recursive.c\n> +++ b/merge-recursive.c\n> @@ -513,6 +513,28 @@ static void record_df_conflict_files(struct merge_options *o,\n>\n> struct rename {\n> struct diff_filepair *pair;\n> + /*\n> + * Because I keep forgetting every few years what src_entry and\n> + * dst_entry are and have to walk through a debugger and puzzle\n> + * through it to remind myself...\n\nThis repeats the commit message; and doesn't help me understanding the\n{src/dst}_entry. (Maybe drop the first part here?) I'll read on.\n\n> + *\n> + * If 'before' is renamed to 'after' then src_entry will contain\n> + * the versions of 'before' from the merge_base, HEAD, and MERGE in\n> + * stages 1, 2, and 3; dst_entry will contain the versions of\n> + * 'after' from the merge_base, HEAD, and MERGE in stages 1, 2, and\n> + * 3.\n\nSo src == before, dst = after; no trickery with the stages (the same\nstage number\nbefore and after; only the order needs to be conveyed:\nbase, HEAD (ours?), MERGE (theirs?)\n\nI can understand that, so I wonder if we can phrase it to mention (base,\nHEAD, MERGE) just once.\n\n> Thus, we have a total of six modes and oids, though some\n> + * will be null. (Stage 0 is ignored; we're interested in handling\n\ns/will be/may be/ or /can be/?\n\n> + * conflicts.)\n> + *\n> + * Since we don't turn on break-rewrites by default, neither\n> + * src_entry nor dst_entry can have all three of their stages have\n> + * non-null oids, meaning at most four of the six will be non-null.\n\nOh. That explains the choice of /will be/ above. Thanks!\n\n> + * Also, since this is a rename, both src_entry and dst_entry will\n> + * have at least one non-null oid, meaning at least two will be\n> + * non-null. Of the six oids, a typical rename will have three be\n> + * non-null. Only two implies a rename/delete, and four implies a\n> + * rename/add.\n\nThat makes sense.\n\nThanks,\nStefan\n\n> + */\n> struct stage_data *src_entry;\n> struct stage_data *dst_entry;\n> unsigned processed:1;\n> --\n> 2.15.0.5.g9567be9905\n>\n"},{"id":"332455","messageId":"CAGZ79kbmt7jt13D-HY90+LBaCHsqvDOYnrmJ41hR3YsgEceirg@mail.gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-5-newren@gmail.com","subject":"Re: [PATCH 04/30] directory rename detection: basic testcases","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-13T22:04:35Z","receivedAt":"2017-11-13T22:04:41Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n> Signed-off-by: Elijah Newren \n> ---\n> t/t6043-merge-rename-directories.sh | 391 ++++++++++++++++++++++++++++++++++++\n> 1 file changed, 391 insertions(+)\n> create mode 100755 t/t6043-merge-rename-directories.sh\n>\n> diff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\n> new file mode 100755\n> index 0000000000..b737b0a105\n> --- /dev/null\n> +++ b/t/t6043-merge-rename-directories.sh\n> @@ -0,0 +1,391 @@\n> +#!/bin/sh\n> +\n> +test_description=\"recursive merge with directory renames\"\n> +# includes checking of many corner cases, with a similar methodology to:\n> +# t6042: corner cases with renames but not criss-cross merges\n> +# t6036: corner cases with both renames and criss-cross merges\n> +#\n> +# The setup for all of them, pictorially, is:\n> +#\n> +# B\n> +# o\n> +# / \\\n> +# A o ?\n> +# \\ /\n> +# o\n> +# C\n> +#\n> +# To help make it easier to follow the flow of tests, they have been\n> +# divided into sections and each test will start with a quick explanation\n> +# of what commits A, B, and C contain.\n> +#\n> +# Notation:\n> +# z/{b,c} means files z/b and z/c both exist\n> +# x/d_1 means file x/d exists with content d1. (Purpose of the\n> +# underscore notation is to differentiate different\n> +# files that might be renamed into each other's paths.)\n> +\n> +. ./test-lib.sh\n> +\n> +\n> +###########################################################################\n> +# SECTION 1: Basic cases we should be able to handle\n> +###########################################################################\n> +\n> +# Testcase 1a, Basic directory rename.\n> +# Commit A: z/{b,c}\n> +# Commit B: y/{b,c}\n> +# Commit C: z/{b,c,d,e/f}\n\n(minor thought:)\nAfter rereading the docs above this is clear; I wonder if instead of A, B, C\na notation of Base, ours, theirs would be easier to understand?\n\n\n> +test_expect_success '1a-setup: Simple directory rename detection' '\n> +test_expect_failure '1a-check: Simple directory rename detection' '\n\nThanks for splitting the setup and the check into two different test cases!\n\n\n> + git checkout B^0 &&\n\nAny reason for ^0 ? (to make clear it is a branch?)\n\n> +# Testcase 1b, Merge a directory with another\n> +# Commit A: z/{b,c}, y/d\n> +# Commit B: z/{b,c,e}, y/d\n> +# Commit C: y/{b,c,d}\n> +# Expected: y/{b,c,d,e}\n> +\n> +test_expect_success '1b-setup: Merge a directory with another' '\n> + git rm -rf . &&\n> + git clean -fdqx &&\n> + rm -rf .git &&\n> + git init &&\n\nThis is quite a strong statement to start a test with.\nNowadays we rather do\n\n test_when_finished \"some command\" &&\n\nthan polluting the follow up tests. But as you split up the previous test\ninto 2 tests, it is not clear if this would bring any good.\n\nAlso these are four cleanup commands, I'd have expected fewer.\nMaybe just \"rm -rf .\" ? Or as we make a new repo for each test case,\n\n test_create_repo 1a &&\n cd 1a\n\nin the first setup, and here we do\n test_create_repo 1b &&\n cd 1b\n\nrelying on test_done to cleanup everything afterwards?\n\n\n> +# Testcase 1c, Transitive renaming\n> +# (Related to testcases 3a and 6d -- when should a transitive rename apply?)\n> +# (Related to testcases 9c and 9d -- can transitivity repeat?)\n> +# Commit A: z/{b,c}, x/d\n> +# Commit B: y/{b,c}, x/d\n> +# Commit C: z/{b,c,d}\n\nSo B is \"Rename z to y\" and C is \"Move x/d to z/d\"\n\n> +# Expected: y/{b,c,d} (because x/d -> z/d -> y/d)\n\nThis is an excellent expectation for a clean case like this.\nI have not reached 3, 9 yet, so I'll remember these questions.\n\n> +test_expect_success '1c-setup: Transitive renaming' '\n> + git rm -rf . &&\n> + git clean -fdqx &&\n> + rm -rf .git &&\n> + git init &&\n> +\n> + mkdir z &&\n> + echo b >z/b &&\n> + echo c >z/c &&\n> + mkdir x &&\n> + echo d >x/d &&\n> + git add z x &&\n> + test_tick &&\n> + git commit -m \"A\" &&\n> +\n> + git branch A &&\n> + git branch B &&\n> + git branch C &&\n> +\n> + git checkout B &&\n> + git mv z y &&\n> + test_tick &&\n> + git commit -m \"B\" &&\n> +\n> + git checkout C &&\n> + git mv x/d z/d &&\n> + test_tick &&\n> + git commit -m \"C\"\n> +'\n> +\n> +test_expect_failure '1c-check: Transitive renaming' '\n> + git checkout B^0 &&\n> +\n> + git merge -s recursive C^0 &&\n> +\n> + test 3 -eq $(git ls-files -s | wc -l) &&\n\n git ls-files >out &&\n test_line_count = 3 out &&\n\nmaybe? Piping output of git commands somewhere is an\nanti-pattern as we cannot examine the return code of ls-files in this case.\n\n> + test $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n> + test $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n> + test $(git rev-parse HEAD:y/d) = $(git rev-parse A:x/d) &&\n\nSpeaking of these, there are quite a lot of invocations of rev-parse,\nthough it looks clean; recently Junio had some good ideas regarding a\nsimilar test[1].\nSo maybe\n\n git rev-parse >actual \\\n HEAD:y/b HEAD:y/c HEAD:y/d &&\n git rev-parse >expect \\\n A:z/b A:z/c A:x/d &&\n test_cmp expect actual\n\nNot sure if that is any nicer, but has fewer calls.\n\n[1] https://public-inbox.org/git/xmqqa807ztx4.fsf@gitster.mtv.corp.google.com/\n\n\n> + test_must_fail git rev-parse HEAD:x/d &&\n> + test_must_fail git rev-parse HEAD:z/d &&\n> + test ! -f z/d\n> +'\n> +\n> +# Testcase 1d, Directory renames (merging two directories into one new one)\n> +# cause a rename/rename(2to1) conflict\n> +# (Related to testcases 1c and 7b)\n> +# Commit A. z/{b,c}, y/{d,e}\n> +# Commit B. x/{b,c}, y/{d,e,m,wham}\n> +# Commit C. z/{b,c,n,wham}, x/{d,e}\n> +# Expected: x/{b,c,d,e,m,n}, CONFLICT:(y/wham & z/wham -> x/wham)\n> +# Note: y/m & z/n should definitely move into x. By the same token, both\n> +# y/wham & z/wham should to...giving us a conflict.\n\nIf wham are equal (in oid), shouldn't this not conflict and only conflict\nwhen z/wham and x/wham differ in oid, but have the same sub-path?\n\n> +\n> +# Testcase 1e, Renamed directory, with all filenames being renamed too\n> +# Commit A: z/{oldb,oldc}\n> +# Commit B: y/{newb,newc}\n> +# Commit C: z/{oldb,oldc,d}\n\nWhat about oldd ?\n(expecting a pattern across many files of s/old/new/ isn't unreasonable,\nbut maybe too much for now?)\nBy having no \"old\" prefix there is only one thing to do, which is y/d\n\n> +# Expected: y/{newb,newc,d}\n\nok.\n\n> +# Testcase 1f, Split a directory into two other directories\n> +# (Related to testcases 3a, all of section 2, and all of section 4)\n> +# Commit A: z/{b,c,d,e,f}\n> +# Commit B: z/{b,c,d,e,f,g}\n> +# Commit C: y/{b,c}, x/{d,e,f}\n> +# Expected: y/{b,c}, x/{d,e,f,g}\n\nWhy not y/g ? Because there are more files in x?\nI can come up with a reasonable counter example:\n\nA: src/{main.c, foo.c, bar.c, magic.py}\nB: src/{main.c, foo.c, bar.c, magic.py, magic-helper.py}\nC: src/{main.c, foo.c, bar.c} py/{magic.py}\nExpect: src/{main.c, foo.c, bar.c} py/{magic.py, magic-helper.py}\n\n\n> +\n> +###########################################################################\n> +# Rules suggested by testcases in section 1:\n> +#\n> +# We should still detect the directory rename even if it wasn't just\n> +# the directory renamed, but the files within it. (see 1b)\n> +#\n> +# If renames split a directory into two or more others, the directory\n> +# with the most renames, \"wins\" (see 1c). However, see the testcases\n> +# in section 2, plus testcases 3a and 4a.\n\noh, ok. I presume testcases 2 follows in a later patch.\nI'll go looking.\n\nThanks,\nStefan\n"},{"id":"332456","messageId":"CABPp-BFX-xZf962vuCjJSaB0=sPMT_zNF8+mGPG04dFi5PE6SQ@mail.gmail.com","threadId":"47162","inReplyTo":"CAGZ79kYXbvDL_SoEENf60DsDFA-yEWfvgv8bNv_v+mw042ZH=w@mail.gmail.com","subject":"Re: [PATCH 02/30] merge-recursive: Fix logic ordering issue","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-13T22:04:53Z","receivedAt":"2017-11-13T22:04:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Thanks for the reviews!\n\nOn Mon, Nov 13, 2017 at 11:48 AM, Stefan Beller wrote:\n> On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n>> merge_trees() did a variety of work, including:\n>> * Calling get_unmerged() to get unmerged entries\n>> * Calling record_df_conflict_files() with all unmerged entries to\n>> do some work to ensure we could handle D/F conflicts correctly\n>> * Calling get_renames() to check for renames.\n>>\n>> An easily overlooked issue is that get_renames() can create more\n>> unmerged entries and add them to the list, which have the possibility of\n>> being involved in D/F conflicts.\n>\n> I presume these are created via insert_stage_data called in\n> get_renames, when the path entry is not found?\n\nYes.\n\n>> So the call to\n>> record_df_conflict_files() should really be moved after all the rename\n>> detection. I didn't come up with any testcases demonstrating any bugs\n>> with the old ordering, but I suspect there were some for both normal\n>> renames and for directory renames. Fix the ordering.\n>\n> It is hard to trace this down, though looking at\n> 3af244caa8 (Cumulative update of merge-recursive in C, 2006-07-27)\n> may help us reason about it.\n\nIt doesn't really go back that far. I added the\nrecord_df_conflict_files() function (originally named\nmake_room_for_directories_of_df_conflicts()) in commit ef02b31721\n(merge-recursive: Make room for directories in D/F conflicts\n2010-09-20); the rename happened in commit 70cc3d36eb\n(merge-recursive: Save D/F conflict filenames instead of unlinking\nthem 2011-08-11).\n\n> How would a bug look like?\n\nSome of these corner cases sometimes get confusing to try to reason\nabout and duplicate, so I was trying to avoid that....oh, well. :-)\nI mostly wanted to use the simple logic that:\nrecord_df_conflict_files() exists to take an inventory of all unmerged\nfiles to make sure that D/F conflicts can be handled appropriately.\nget_renames() has the potential for adding more unmerged files, thus I\nshould have placed record_df_conflict_files() after get_renames() when\nI introduced it.\n\nBut since you asked...\n\nA bug here would essentially mean that a git merge fails to handle\nfiles in directories under a D/F conflict; when trying to process such\nfiles and write out their conflict state to disk, it would fail to\ncreate the necessary directory because a file is in the way.\n\nIn order to trigger it, you'd need to have a D/F conflict where the\nfile involved in the D/F conflict wasn't unmerged after unpack_trees()\nbut only \"shows up\" due to the rename detection (i.e. added by the\ninsert_stage_data() call as you mention above). I think reading\nthrough Documentation/technical/trivial-merge.txt, that this actually\nisn't possible with what I'm calling \"normal\" renames; it's actually\nsomething newly possible only due to directory rename detection. But\nyou may have to get the merge direction just right, you might have to\nworry about files that sort between a file with the same name as a\ndirectory and the files within the directory (e.g. \"path.txt\" in the\nlist \"path\", then \"path.txt\", then \"path/foo\").\n\nDo you feel it's important that I come up with a demonstration case\nhere? If so, I'll see if I can generate one.\n"},{"id":"332457","messageId":"CAGZ79kYEwOhjMfNH3ovfEnRdU_OYWnGnai8HrZWxoBEC67xKVw@mail.gmail.com","threadId":"47162","inReplyTo":"CABPp-BFX-xZf962vuCjJSaB0=sPMT_zNF8+mGPG04dFi5PE6SQ@mail.gmail.com","subject":"Re: [PATCH 02/30] merge-recursive: Fix logic ordering issue","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-13T22:12:19Z","receivedAt":"2017-11-13T22:12:25Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Nov 13, 2017 at 2:04 PM, Elijah Newren wrote:\n\n> Do you feel it's important that I come up with a demonstration case\n> here? If so, I'll see if I can generate one.\n\nI was mostly \"just curious\" on how you'd construct it theoretically.\n\n> it's actually something newly possible only due to directory rename detection.\n\nSo something like {rename/delete} on a directory in the merge, but\nalso an addition instead of the delete of another file?\n\nI wanted to debug a very similar issue today just after reviewing this\nseries, see\nhttps://public-inbox.org/git/743acc29-85bb-3773-b6a0-68d4a0b8fd63@ispras.ru/\n\nThanks,\nStefan\n"},{"id":"332468","messageId":"CABPp-BFSt8NGSzkoJdhst_NHqx_hVGMzO-688SOdYRB413Zd9g@mail.gmail.com","threadId":"47162","inReplyTo":"CAGZ79kZ7d7fMp3Cw3Yy4bq9-Mh5Z_jhM5XDQbkVLoqzm6ONzDg@mail.gmail.com","subject":"Re: [PATCH 03/30] merge-recursive: Add explanation for src_entry and dst_entry","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-13T22:57:02Z","receivedAt":"2017-11-13T22:57:08Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Nov 13, 2017 at 1:06 PM, Stefan Beller wrote:\n> On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n>> + /*\n>> + * Because I keep forgetting every few years what src_entry and\n>> + * dst_entry are and have to walk through a debugger and puzzle\n>> + * through it to remind myself...\n>\n> This repeats the commit message; and doesn't help me understanding the\n> {src/dst}_entry. (Maybe drop the first part here?) I'll read on.\n\nYep, I'll toss it.\n\n>> + *\n>> + * If 'before' is renamed to 'after' then src_entry will contain\n>> + * the versions of 'before' from the merge_base, HEAD, and MERGE in\n>> + * stages 1, 2, and 3; dst_entry will contain the versions of\n>> + * 'after' from the merge_base, HEAD, and MERGE in stages 1, 2, and\n>> + * 3.\n>\n> So src == before, dst = after; no trickery with the stages (the same\n> stage number\n> before and after; only the order needs to be conveyed:\n> base, HEAD (ours?), MERGE (theirs?)\n>\n> I can understand that, so I wonder if we can phrase it to mention (base,\n> HEAD, MERGE) just once.\n\nPerhaps:\n\n If 'before' is renamed to 'after' then src_entry will contain\n the versions of 'before' from the merge_base, HEAD, and MERGE in\n stages 1, 2, and 3; and dst_entry will contain the respective versions of\n 'after' in corresponding locations. Thus, we have a total of six modes\n and oids, though some will be null. (Stage 0 is ignored; we're interested\n in handling conflicts.)\n\n?\n"},{"id":"332471","messageId":"CAGZ79kYD8QjLDsF8APBP8P-WvAz7NBhnh6vbkr8xMnzW7XAm9g@mail.gmail.com","threadId":"47162","inReplyTo":"CABPp-BFSt8NGSzkoJdhst_NHqx_hVGMzO-688SOdYRB413Zd9g@mail.gmail.com","subject":"Re: [PATCH 03/30] merge-recursive: Add explanation for src_entry and dst_entry","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-13T23:11:25Z","receivedAt":"2017-11-13T23:11:32Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Nov 13, 2017 at 2:57 PM, Elijah Newren wrote:\n>\n> Perhaps:\n>\n> If 'before' is renamed to 'after' then src_entry will contain\n> the versions of 'before' from the merge_base, HEAD, and MERGE in\n> stages 1, 2, and 3; and dst_entry will contain the respective versions of\n> 'after' in corresponding locations. Thus, we have a total of six modes\n> and oids, though some will be null. (Stage 0 is ignored; we're interested\n> in handling conflicts.)\n\nI find that much easier to read, though I am biased with prior knowledge now. ;)\n"},{"id":"332472","messageId":"CAGZ79kY=JWXs3ASmWQPt__REMbcWQinkJt8ZcZWQEM6V5S7x+w@mail.gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-6-newren@gmail.com","subject":"Re: [PATCH 05/30] directory rename detection: directory splitting testcases","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-13T23:20:58Z","receivedAt":"2017-11-13T23:21:07Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n> Signed-off-by: Elijah Newren \n> ---\n> t/t6043-merge-rename-directories.sh | 125 ++++++++++++++++++++++++++++++++++++\n> 1 file changed, 125 insertions(+)\n>\n> diff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\n> index b737b0a105..00811f512a 100755\n> --- a/t/t6043-merge-rename-directories.sh\n> +++ b/t/t6043-merge-rename-directories.sh\n> @@ -388,4 +388,129 @@ test_expect_failure '1f-check: Split a directory into two other directories' '\n> # in section 2, plus testcases 3a and 4a.\n> ###########################################################################\n>\n> +\n> +###########################################################################\n> +# SECTION 2: Split into multiple directories, with equal number of paths\n> +#\n> +# Explore the splitting-a-directory rules a bit; what happens in the\n> +# edge cases?\n> +#\n> +# Note that there is a closely related case of a directory not being\n> +# split on either side of history, but being renamed differently on\n> +# each side. See testcase 8e for that.\n> +###########################################################################\n> +\n> +# Testcase 2a, Directory split into two on one side, with equal numbers of paths\n> +# Commit A: z/{b,c}\n> +# Commit B: y/b, w/c\n> +# Commit C: z/{b,c,d}\n> +# Expected: y/b, w/c, z/d, with warning about z/ -> (y/ vs. w/) conflict\n\n> + test_i18ngrep \"CONFLICT.*directory rename split\" out\n\n\n\n> +# Testcase 2b, Directory split into two on one side, with equal numbers of paths\n> +# Commit A: z/{b,c}\n> +# Commit B: y/b, w/c\n> +# Commit C: z/{b,c}, x/d\n> +# Expected: y/b, w/c, x/d; No warning about z/ -> (y/ vs. w/) conflict\n\nThis makes sense.\n\n> +\n> +###########################################################################\n> +# Rules suggested by section 2:\n> +#\n> +# None; the rule was already covered in section 1. These testcases are\n> +# here just to make sure the conflict resolution and necessary warning\n> +# messages are handled correctly.\n> +###########################################################################\n\nokay, then I'll go back to 1. and discuss \"the number of files as a\nhint where to rename it to\" there\n"},{"id":"332473","messageId":"CAGZ79kYOW9+pP5ApKDHR2_VwPXyiLEEbbO0-3ka-SDCP6ryLPw@mail.gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-7-newren@gmail.com","subject":"Re: [PATCH 06/30] directory rename detection: testcases to avoid taking detection too far","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-13T23:25:39Z","receivedAt":"2017-11-13T23:25:45Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n> Signed-off-by: Elijah Newren \n> ---\n> t/t6043-merge-rename-directories.sh | 137 ++++++++++++++++++++++++++++++++++++\n> 1 file changed, 137 insertions(+)\n>\n> diff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\n> index 00811f512a..021513ec00 100755\n> --- a/t/t6043-merge-rename-directories.sh\n> +++ b/t/t6043-merge-rename-directories.sh\n> @@ -513,4 +513,141 @@ test_expect_success '2b-check: Directory split into two on one side, with equal\n> # messages are handled correctly.\n> ###########################################################################\n>\n> +\n> +###########################################################################\n> +# SECTION 3: Path in question is the source path for some rename already\n> +#\n> +# Combining cases from Section 1 and trying to handle them could lead to\n> +# directory renaming detection being over-applied. So, this section\n> +# provides some good testcases to check that the implementation doesn't go\n> +# too far.\n> +###########################################################################\n> +\n> +# Testcase 3a, Avoid implicit rename if involved as source on other side\n> +# (Related to testcases 1c and 1f)\n> +# Commit A: z/{b,c,d}\n> +# Commit B: z/{b,c,d} (no change)\n\nThis could also be an unrelated change such as adding w/e?\n\n> +# Commit C: y/{b,c}, x/d\n> +# Expected: y/{b,c}, x/d\n\n\n> +\n> +# Testcase 3b, Avoid implicit rename if involved as source on other side\n> +# (Related to testcases 5c and 7c, also kind of 1e and 1f)\n> +# Commit A: z/{b,c,d}\n> +# Commit B: y/{b,c}, x/d\n> +# Commit C: z/{b,c}, w/d\n> +# Expected: y/{b,c}, CONFLICT:(z/d -> x/d vs. w/d)\n> +# NOTE: We're particularly checking that since z/d is already involved as\n> +# a source in a file rename on the same side of history, that we don't\n> +# get it involved in directory rename detection. If it were, we might\n> +# end up with CONFLICT:(z/d -> y/d vs. x/d vs. w/d), i.e. a\n> +# rename/rename/rename(1to3) conflict, which is just weird.\n\nMakes sense.\n\n>\n"},{"id":"332474","messageId":"CABPp-BFynaf6abbd2ywzjuQnus5BFp=DQmKd80qHj9vFOqEmig@mail.gmail.com","threadId":"47162","inReplyTo":"CAGZ79kYEwOhjMfNH3ovfEnRdU_OYWnGnai8HrZWxoBEC67xKVw@mail.gmail.com","subject":"Re: [PATCH 02/30] merge-recursive: Fix logic ordering issue","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-13T23:39:04Z","receivedAt":"2017-11-13T23:39:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Nov 13, 2017 at 2:12 PM, Stefan Beller wrote:\n> I wanted to debug a very similar issue today just after reviewing this\n> series, see\n> https://public-inbox.org/git/743acc29-85bb-3773-b6a0-68d4a0b8fd63@ispras.ru/\n\nOh, bleh. That's not a D/F conflict at all, it's the code assuming\nthere's a D/F conflict because the entry it is processing (\"sub\") is a\nsubmodule rather than a file, and it panics when it sees \"a directory\nin the way\" -- a directory that just so happens to be named \"sub\" and\nwhich is in fact the desired submodule, meaning that the working\ndirectory is already good and needs no changes.\n\nIn this case, the relevant code from merge-recursive.c is the following:\n\n /* Case B: Added in one. */\n /* [nothing|directory] -> ([nothing|directory], file) */\n\n if (dir_in_way(path, !o->call_depth,\n S_ISGITLINK(a_mode))) {\n char *new_path = unique_path(o, path, add_branch);\n clean_merge = 0;\n output(o, 1, _(\"CONFLICT (%s): There is a directory with\nname %s in %s. \"\n \"Adding %s as %s\"),\n conf, path, other_branch, path, new_path);\n\nNote that the comment even explicitly assumes the newly added entry is\na file. We should expect there to be a directory present (the\nsubmodule being added), but the code doesn't have a check for that.\nThe S_ISGITLINK(a_mode) makes you think it has special handling for\nthe submodule case, but that's for the reverse situation (the\nsubmodule isn't yet present in the working copy, it came from the\nother side of history, but there is an empty directory present).\n"},{"id":"332475","messageId":"CAGZ79ka=7NPU7vm5Onx89+qR3wU6UqDuPVbWsmwpk2Q_b1pjGA@mail.gmail.com","threadId":"47162","inReplyTo":"CABPp-BFynaf6abbd2ywzjuQnus5BFp=DQmKd80qHj9vFOqEmig@mail.gmail.com","subject":"Re: [PATCH 02/30] merge-recursive: Fix logic ordering issue","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-13T23:46:11Z","receivedAt":"2017-11-13T23:46:21Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Nov 13, 2017 at 3:39 PM, Elijah Newren wrote:\n> On Mon, Nov 13, 2017 at 2:12 PM, Stefan Beller wrote:\n>> I wanted to debug a very similar issue today just after reviewing this\n>> series, see\n>> https://public-inbox.org/git/743acc29-85bb-3773-b6a0-68d4a0b8fd63@ispras.ru/\n>\n> Oh, bleh. That's not a D/F conflict at all, it's the code assuming\n> there's a D/F conflict because the entry it is processing (\"sub\") is a\n> submodule rather than a file, and it panics when it sees \"a directory\n> in the way\" -- a directory that just so happens to be named \"sub\" and\n> which is in fact the desired submodule, meaning that the working\n> directory is already good and needs no changes.\n\nyup, I came to find the same snippet of code to be the offender,\nI just haven't figured out how to fix this bug.\n\nThanks for taking a look!\n\nAs you have a lot of fresh knowledge in the merge-recursive case\ncurrently, how would we approach the fix here?\n\n(there is a test available at\nremotes/origin/sb/test-cherry-pick-submodule-getting-in-a-way)\n\n> In this case, the relevant code from merge-recursive.c is the following:\n>\n> /* Case B: Added in one. */\n> /* [nothing|directory] -> ([nothing|directory], file) */\n> \n> if (dir_in_way(path, !o->call_depth,\n> S_ISGITLINK(a_mode))) {\n> char *new_path = unique_path(o, path, add_branch);\n> clean_merge = 0;\n> output(o, 1, _(\"CONFLICT (%s): There is a directory with\n> name %s in %s. \"\n> \"Adding %s as %s\"),\n> conf, path, other_branch, path, new_path);\n>\n> Note that the comment even explicitly assumes the newly added entry is\n> a file. We should expect there to be a directory present (the\n> submodule being added), but the code doesn't have a check for that.\n> The S_ISGITLINK(a_mode) makes you think it has special handling for\n> the submodule case, but that's for the reverse situation (the\n> submodule isn't yet present in the working copy, it came from the\n> other side of history, but there is an empty directory present).\n\noh :/\n"},{"id":"332476","messageId":"CAGZ79kaR5yfpopa+oB13ZftY_Qi37OXaATjeL21gyAP6abpxAw@mail.gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-8-newren@gmail.com","subject":"Re: [PATCH 07/30] directory rename detection: partially renamed directory testcase/discussion","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-14T00:07:06Z","receivedAt":"2017-11-14T00:07:12Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n> Signed-off-by: Elijah Newren \n> ---\n> t/t6043-merge-rename-directories.sh | 100 ++++++++++++++++++++++++++++++++++++\n> 1 file changed, 100 insertions(+)\n>\n> diff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\n> index 021513ec00..ec054b210a 100755\n> --- a/t/t6043-merge-rename-directories.sh\n> +++ b/t/t6043-merge-rename-directories.sh\n> @@ -650,4 +650,104 @@ test_expect_success '3b-check: Avoid implicit rename if involved as source on cu\n> # of a rename on either side of a merge.\n> ###########################################################################\n>\n> +\n> +###########################################################################\n> +# SECTION 4: Partially renamed directory; still exists on both sides of merge\n> +#\n> +# What if we were to attempt to do directory rename detection when someone\n> +# \"mostly\" moved a directory but still left some files around, or,\n> +# equivalently, fully renamed a directory in one commmit and then recreated\n> +# that directory in a later commit adding some new files and then tried to\n> +# merge?\n> +#\n> +# It's hard to divine user intent in these cases, because you can make an\n> +# argument that, depending on the intermediate history of the side being\n> +# merged, that some users will want files in that directory to\n> +# automatically be detected and renamed, while users with a different\n> +# intermediate history wouldn't want that rename to happen.\n> +#\n> +# I think that it is best to simply not have directory rename detection\n> +# apply to such cases. My reasoning for this is four-fold: (1) it's\n> +# easiest for users in general to figure out what happened if we don't\n> +# apply directory rename detection in any such case, (2) it's an easy rule\n> +# to explain [\"We don't do directory rename detection if the directory\n> +# still exists on both sides of the merge\"], (3) we can get some hairy\n> +# edge/corner cases that would be really confusing and possibly not even\n> +# representable in the index if we were to even try, and [related to 3] (4)\n> +# attempting to resolve this issue of divining user intent by examining\n> +# intermediate history goes against the spirit of three-way merges and is a\n> +# path towards crazy corner cases that are far more complex than what we're\n> +# already dealing with.\n\nThis last paragraph (\"I think\") sounds like part of a commit message,\nrather than\na note inside a testing script. Not sure if I recommend moving this\ntext into the\ncommit message.\n\n> +# This section contains a test for this partially-renamed-directory case.\n> +###########################################################################\n> +\n> +# Testcase 4a, Directory split, with original directory still present\n> +# (Related to testcase 1f)\n> +# Commit A: z/{b,c,d,e}\n> +# Commit B: y/{b,c,d}, z/e\n> +# Commit C: z/{b,c,d,e,f}\n> +# Expected: y/{b,c,d}, z/{e,f}\n> +# NOTE: Even though most files from z moved to y, we don't want f to follow.\n\nMakes sense.\n\n> +###########################################################################\n> +# Rules suggested by section 4:\n> +#\n> +# Directory-rename-detection should be turned off for any directories (as\n> +# a source for renames) that exist on both sides of the merge. (The \"as\n> +# a source for renames\" clarification is due to cases like 1c where\n> +# the target directory exists on both sides and we do want the rename\n> +# detection.) But, sadly, see testcase 8b.\n\nLooking forward for that test case.\n"},{"id":"332477","messageId":"CAGZ79kana0_9N+gG+m2BT5Tnu_q4MC=WSTuvtwkdeD_Wn70ydA@mail.gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-9-newren@gmail.com","subject":"Re: [PATCH 08/30] directory rename detection: files/directories in the way of some renames","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-14T00:15:27Z","receivedAt":"2017-11-14T00:15:34Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n> Signed-off-by: Elijah Newren \n> ---\n> t/t6043-merge-rename-directories.sh | 303 ++++++++++++++++++++++++++++++++++++\n> 1 file changed, 303 insertions(+)\n>\n> diff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\n> index ec054b210a..d15153c652 100755\n> --- a/t/t6043-merge-rename-directories.sh\n> +++ b/t/t6043-merge-rename-directories.sh\n> @@ -750,4 +750,307 @@ test_expect_success '4a-check: Directory split, with original directory still pr\n> # detection.) But, sadly, see testcase 8b.\n> ###########################################################################\n>\n> +\n> +###########################################################################\n> +# SECTION 5: Files/directories in the way of subset of to-be-renamed paths\n> +#\n> +# Implicitly renaming files due to a detected directory rename could run\n> +# into problems if there are files or directories in the way of the paths\n> +# we want to rename. Explore such cases in this section.\n> +###########################################################################\n> +\n> +# Testcase 5a, Merge directories, other side adds files to original and target\n> +# Commit A: z/{b,c}, y/d\n> +# Commit B: z/{b,c,e_1,f}, y/{d,e_2}\n> +# Commit C: y/{b,c,d}\n> +# Expected: z/e_1, y/{b,c,d,e_2,f} + CONFLICT warning\n> +# NOTE: While directory rename detection is active here causing z/f to\n> +# become y/f, we did not apply this for z/e_1 because that would\n> +# give us an add/add conflict for y/e_1 vs y/e_2. This problem with\n> +# this add/add, is that both versions of y/e are from the same side\n> +# of history, giving us no way to represent this conflict in the\n> +# index.\n\nMakes sense.\n\n> +# Testcase 5b, Rename/delete in order to get add/add/add conflict\n> +# (Related to testcase 8d; these may appear slightly inconsistent to users;\n> +# Also related to testcases 7d and 7e)\n> +# Commit A: z/{b,c,d_1}\n> +# Commit B: y/{b,c,d_2}\n> +# Commit C: z/{b,c,d_1,e}, y/d_3\n> +# Expected: y/{b,c,e}, CONFLICT(add/add: y/d_2 vs. y/d_3)\n> +# NOTE: If z/d_1 in commit C were to be involved in dir rename detection, as\n> +# we normaly would since z/ is being renamed to y/, then this would be\n> +# a rename/delete (z/d_1 -> y/d_1 vs. deleted) AND an add/add/add\n> +# conflict of y/d_1 vs. y/d_2 vs. y/d_3. Add/add/add is not\n> +# representable in the index, so the existence of y/d_3 needs to\n> +# cause us to bail on directory rename detection for that path, falling\n> +# back to git behavior without the directory rename detection.\n\n\n> +\n> +# Testcase 5c, Transitive rename would cause rename/rename/rename/add/add/add\n> +# (Directory rename detection would result in transitive rename vs.\n> +# rename/rename(1to2) and turn it into a rename/rename(1to3). Further,\n> +# rename paths conflict with separate adds on the other side)\n> +# (Related to testcases 3b and 7c)\n> +# Commit A: z/{b,c}, x/d_1\n> +# Commit B: y/{b,c,d_2}, w/d_1\n> +# Commit C: z/{b,c,d_1,e}, w/d_3, y/d_4\n> +# Expected: A mess, but only a rename/rename(1to2)/add/add mess. Use the\n> +# presence of y/d_4 in C to avoid doing transitive rename of\n> +# x/d_1 -> z/d_1 -> y/d_1, so that the only paths we have at\n> +# y/d are y/d_2 and y/d_4. We still do the move from z/e to y/e,\n> +# though, because it doesn't have anything in the way.\n\nMissing the expected state, only the explanation is given.\n\n\n> +# Testcase 5d, Directory/file/file conflict due to directory rename\n> +# Commit A: z/{b,c}\n> +# Commit B: y/{b,c,d_1}\n> +# Commit C: z/{b,c,d_2,f}, y/d/e\n> +# Expected: y/{b,c,d/e,f}, z/d_2, CONFLICT(file/directory), y/d_1~HEAD\n> +# Note: The fact that y/d/ exists in C makes us bail on directory rename\n> +# detection for z/d_2, but that doesn't prevent us from applying the\n> +# directory rename detection for z/f -> y/f.\n\nMakes sense.\n\n> +\n> +###########################################################################\n> +# Rules suggested by section 5:\n> +#\n> +# If a subset of to-be-renamed files have a file or directory in the way,\n> +# \"turn off\" the directory rename for those specific sub-paths,\n\nMakes sense.\n\n> falling\n> +# back to old handling. But, sadly, see testcases 8a and 8b.\n\nYou seem to be hinting at these all the time.\n"},{"id":"332478","messageId":"CAGZ79kbsQqAFGFVN3dOnSAFuPishF3PGS7_go+53T5Oa-4qWiA@mail.gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-10-newren@gmail.com","subject":"Re: [PATCH 09/30] directory rename detection: testcases checking which side did the rename","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-14T00:25:51Z","receivedAt":"2017-11-14T00:25:58Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n> Signed-off-by: Elijah Newren \n> ---\n> t/t6043-merge-rename-directories.sh | 283 ++++++++++++++++++++++++++++++++++++\n> 1 file changed, 283 insertions(+)\n>\n> diff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\n> index d15153c652..157299105f 100755\n> --- a/t/t6043-merge-rename-directories.sh\n> +++ b/t/t6043-merge-rename-directories.sh\n> @@ -1053,4 +1053,287 @@ test_expect_failure '5d-check: Directory/file/file conflict due to directory ren\n> # back to old handling. But, sadly, see testcases 8a and 8b.\n> ###########################################################################\n>\n> +\n> +###########################################################################\n> +# SECTION 6: Same side of the merge was the one that did the rename\n> +#\n> +# It may sound obvious that you only want to apply implicit directory\n> +# renames to directories if the _other_ side of history did the renaming.\n> +# If you did make an implementation that didn't explicitly enforce this\n> +# rule, the majority of cases that would fall under this section would\n> +# also be solved by following the rules from the above sections. But\n> +# there are still a few that stick out, so this section covers them just\n> +# to make sure we also get them right.\n> +###########################################################################\n> +\n> +# Testcase 6a, Tricky rename/delete\n> +# Commit A: z/{b,c,d}\n> +# Commit B: z/b\n> +# Commit C: y/{b,c}, z/d\n> +# Expected: y/b, CONFLICT(rename/delete, z/c -> y/c vs. NULL)\n> +# Note: We're just checking here that the rename of z/b and z/c to put\n> +# them under y/ doesn't accidentally catch z/d and make it look like\n> +# it is also involved in a rename/delete conflict.\n> +\n\n> +\n> +# Testcase 6b, Same rename done on both sides\n> +# (Related to testcases 6c and 8e)\n> +# Commit A: z/{b,c}\n> +# Commit B: y/{b,c}\n> +# Commit C: y/{b,c}, z/d\n\nMissing expected state\n\n> +# Note: If we did directory rename detection here, we'd move z/d into y/,\n> +# but C did that rename and still decided to put the file into z/,\n> +# so we probably shouldn't apply directory rename detection for it.\n\ncorrect. Also we don't want to see a rename/rename conflict (obviously).\n\nIf we have\n\n Commit A: z/{b_1,c}\n Commit B: y/{b_2,c}\n Commit C: y/{b_3,c}, z/d\n\nthen we'd produce a standard file merge (which may or may not result\nin conflict,\ndepending on touched lines) for y/b_{try-resolve}\n\n> +\n> +# Testcase 6c, Rename only done on same side\n> +# (Related to testcases 6b and 8e)\n> +# Commit A: z/{b,c}\n> +# Commit B: z/{b,c} (no change)\n> +# Commit C: y/{b,c}, z/d\n> +# Expected: y/{b,c}, z/d\n> +# NOTE: Seems obvious, but just checking that the implementation doesn't\n> +# \"accidentally detect a rename\" and give us y/{b,c,d}.\n\nmakes sense.\n\n> +\n> +# Testcase 6d, We don't always want transitive renaming\n> +# (Related to testcase 1c)\n> +# Commit A: z/{b,c}, x/d\n> +# Commit B: z/{b,c}, x/d (no change)\n> +# Commit C: y/{b,c}, z/d\n> +# Expected: y/{b,c}, z/d\n> +# NOTE: Again, this seems obvious but just checking that the implementation\n> +# doesn't \"accidentally detect a rename\" and give us y/{b,c,d}.\n\nmakes sense, too.\n\n> +# Testcase 6e, Add/add from one-side\n> +# Commit A: z/{b,c}\n> +# Commit B: z/{b,c} (no change)\n> +# Commit C: y/{b,c,d_1}, z/d_2\n> +# Expected: y/{b,c,d_1}, z/d_2\n> +# NOTE: Again, this seems obvious but just checking that the implementation\n> +# doesn't \"accidentally detect a rename\" and give us y/{b,c} +\n> +# add/add conflict on y/d_1 vs y/d_2.\n\nWhat is less obvious in all these cases is the \"(no change)\" part to me.\nI would think that at least *something* changes in B in all cases above, maybe\nadd file u/r (un-related) to have the tree ids changed?\n(\"Less obvious\" as in: we don't rely on the \"no changes\" part to make\nthe decision,\nwhich sounds tempting so far)\n\n> test_done\n\nNo conclusion box here, so my (misguided) suggestion:\n\n If \"No change\" occurs, just take the other side. ;)\n"},{"id":"332479","messageId":"CAGZ79kaDWxx2g2OMY-VKJZD0-0t7V33644aqPLHq5Hppo_Be+w@mail.gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-11-newren@gmail.com","subject":"Re: [PATCH 10/30] directory rename detection: more involved edge/corner testcases","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-14T00:42:38Z","receivedAt":"2017-11-14T00:44:37Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n\n> +# In my opinion, testcases that are difficult to understand from this\n> +# section is due to difficulty in the testcase rather than the directory\n> +# renaming (similar to how t6042 and t6036 have difficult resolutions due\n> +# to the problem setup itself being complex). And I don't think the\n> +# error messages are a problem.\n\n\"In my opinion\" ... sounds like commit message?\n\n> +# On the other hand, the testcases in section 8 worry me slightly more...\n\nThe aforementioned section 8... :)\n\n> +# Testcase 7a, rename-dir vs. rename-dir (NOT split evenly) PLUS add-other-file\n> +# Commit A: z/{b,c}\n> +# Commit B: y/{b,c}\n> +# Commit C: w/b, x/c, z/d\n> +# Expected: y/d, CONFLICT(rename/rename for both z/b and z/c)\n> +# NOTE: There's a rename of z/ here, y/ has more renames, so z/d -> y/d.\n\nBut the creator of C intended to have z/d, not {w,x}/d, and as {w,x} == y,\nI am not sure I like this result. (I have no concrete counter example, just\nmessy logic)\n\n\n> +# Testcase 7b, rename/rename(2to1), but only due to transitive rename\n> +# (Related to testcase 1d)\n> +# Commit A: z/{b,c}, x/d_1, w/d_2\n> +# Commit B: y/{b,c,d_2}, x/d_1\n> +# Commit C: z/{b,c,d_1}, w/d_2\n> +# Expected: y/{b,c}, CONFLICT(rename/rename(2to1): x/d_1, w/d_2 -> y_d)\n\nMakes sense.\n\n> +# Testcase 7c, rename/rename(1to...2or3); transitive rename may add complexity\n> +# (Related to testcases 3b and 5c)\n> +# Commit A: z/{b,c}, x/d\n> +# Commit B: y/{b,c}, w/d\n> +# Commit C: z/{b,c,d}\n> +# Expected: y/{b,c}, CONFLICT(x/d -> w/d vs. y/d)\n\nCONFLICT(x/d -> y/d vs w/d) ?\n\n> +# NOTE: z/ was renamed to y/ so we do not want to report\n> +# either CONFLICT(x/d -> w/d vs. z/d)\n> +# or CONFLiCT x/d -> w/d vs. y/d vs. z/d)\n\n\"neither ... nor\" instead of \"not either or\"?\n\n> +# Testcase 7d, transitive rename involved in rename/delete; how is it reported?\n> +# (Related somewhat to testcases 5b and 8d)\n> +# Commit A: z/{b,c}, x/d\n> +# Commit B: y/{b,c}\n> +# Commit C: z/{b,c,d}\n> +# Expected: y/{b,c}, CONFLICT(delete x/d vs rename to y/d)\n> +# NOTE: z->y so NOT CONFLICT(delete x/d vs rename to z/d)\n\n\n> +# Testcase 7e, transitive rename in rename/delete AND dirs in the way\n> +# (Very similar to 'both rename source and destination involved in D/F conflict' from t6022-merge-rename.sh)\n> +# (Also related to testcases 9c and 9d)\n> +# Commit A: z/{b,c}, x/d_1\n> +# Commit B: y/{b,c,d/g}, x/d/f\n> +# Commit C: z/{b,c,d_1}\n> +# Expected: rename/delete(x/d_1->y/d_1 vs. None) + D/F conflict on y/d\n> +# y/{b,c,d/g}, y/d_1~C^0, x/d/f\n> +# NOTE: x/d/f may be slightly confusing here. x/d_1 -> z/d_1 implies\n> +# there is a directory rename from x/ -> z/, performed by commit C.\n> +# However, on the side of commit B, it renamed z/ -> y/, thus\n> +# making a rename from x/ -> z/ when it was getting rid of z/ seems\n> +# non-sensical. Further, putting x/d/f into y/d/f also doesn't\n> +# make a lot of sense because commit B did the renaming of z to y\n> +# and it created x/d/f, and it clearly made these things separate,\n> +# so it doesn't make much sense to push these together.\n\nThis is confusing.\n"},{"id":"332480","messageId":"CABPp-BFKiam6AK-Gg_RzaLuLur-jz0kvv3TqsHNHg5+HTv_uzA@mail.gmail.com","threadId":"47162","inReplyTo":"CAGZ79kbmt7jt13D-HY90+LBaCHsqvDOYnrmJ41hR3YsgEceirg@mail.gmail.com","subject":"Re: [PATCH 04/30] directory rename detection: basic testcases","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-14T00:57:31Z","receivedAt":"2017-11-14T00:57:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Nov 13, 2017 at 2:04 PM, Stefan Beller wrote:\n> On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n> (minor thought:)\n> After rereading the docs above this is clear; I wonder if instead of A, B, C\n> a notation of Base, ours, theirs would be easier to understand?\n\nSure, that'd be an easy change.\n\n>> +test_expect_success '1a-setup: Simple directory rename detection' '\n>> +test_expect_failure '1a-check: Simple directory rename detection' '\n>\n> Thanks for splitting the setup and the check into two different test cases!\n>\n>\n>> + git checkout B^0 &&\n>\n> Any reason for ^0 ? (to make clear it is a branch?)\n\nPartially force-of-habit (did the same with t6036 and t6042), but it\nseemed to have the nice feature of making debugging easier through\nimproved reproducability. In particular, if I had checked out B\nrather than B^0 in the testcase and a merge succeeded when I didn't\nexpect it, then attempting to run the same commands gives me a\ndifferent starting point for the merge. By instead explicitly\nchecking out B^0, then even if the merge succeeds, someone who again\nruns checkout B^0 will have the same starting point.\n\n>> +test_expect_success '1b-setup: Merge a directory with another' '\n>> + git rm -rf . &&\n>> + git clean -fdqx &&\n>> + rm -rf .git &&\n>> + git init &&\n>\n> This is quite a strong statement to start a test with.\n\nIt was actually copy-paste from t6036, and achieved two purposes:\n 1) Even if one test fails, subsequent ones continue running. (Had\nlots of problems with this in t6036 years ago and just ended up with\nthose four steps)\n 2) Someone who runs into a failing testcase has a _much_ easier time\nfiguring out what is going on with the testcase. I find it takes a\nfair amount of time to figure out what's going on with other tests in\ngit's testsuite because of the presence of so many files completely\nunrelated to the given test, which have simply accumulated from\nprevious tests. For many tests, that may be fine, but in particular\nfor t6036, t6042, and now t6043, since these are mostly about corner\ncases that are hard enough to reason about, I didn't want the extra\ndistractions.\n\nbut...\n\n> Nowadays we rather do\n>\n> test_when_finished \"some command\" &&\n>\n> than polluting the follow up tests. But as you split up the previous test\n> into 2 tests, it is not clear if this would bring any good.\n\ntest_when_finished looks pretty cool; I didn't know about it. Thanks\nfor the pointer. Not sure if we want to use it here (if we do, we'd\nonly do so in the check tests rather than the setup ones).\n\n> Also these are four cleanup commands, I'd have expected fewer.\n> Maybe just \"rm -rf .\" ? Or as we make a new repo for each test case,\n>\n> test_create_repo 1a &&\n> cd 1a\n>\n> in the first setup, and here we do\n> test_create_repo 1b &&\n> cd 1b\n>\n> relying on test_done to cleanup everything afterwards?\n\nrm aborts for me with 'rm -rf .', but I could possibly make it 'rm -rf\n* .* && git init .'\n\nThe test_create_repo might not be so bad as long as every test used it\nand put all files under it's own little repo. It does create some\nclutter, but it's at least somewhat managed. I'm still a bit partial\nto just totally cleaning it out, but if others would prefer, I can\nswitch.\n\n>> + test 3 -eq $(git ls-files -s | wc -l) &&\n>\n> git ls-files >out &&\n> test_line_count = 3 out &&\n>\n> maybe? Piping output of git commands somewhere is an\n> anti-pattern as we cannot examine the return code of ls-files in this case.\n\nUm...I guess that makes sense, if you assumed that I cared about the\nreturn code of ls-files. But it seems to make the tests with multiple\ncalls to ls-files in a row (of which there are many) considerably\nuglier, so I'd personally prefer to avoid that. Also, why would I\ncare about the return code of ls-files?\n\nYour suggestion made me curious, so I went looking. As far as I can\ntell, the return code of ls-files is always 0 unless you both specify\nboth --error-unmatch and one or more paths, neither of which I did.\nAm I missing something?\n\nIf you feel the return code of ls-files is important, perhaps I could\njust have a separate\n git ls-files -s >/dev/null &&\ncall before the others?\n\n>> + test $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n>> + test $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n>> + test $(git rev-parse HEAD:y/d) = $(git rev-parse A:x/d) &&\n>\n> Speaking of these, there are quite a lot of invocations of rev-parse,\n> though it looks clean; recently Junio had some good ideas regarding a\n> similar test[1].\n> So maybe\n>\n> git rev-parse >actual \\\n> HEAD:y/b HEAD:y/c HEAD:y/d &&\n> git rev-parse >expect \\\n> A:z/b A:z/c A:x/d &&\n> test_cmp expect actual\n>\n> Not sure if that is any nicer, but has fewer calls.\n\nSure, I can switch it over.\n\n>> + test_must_fail git rev-parse HEAD:x/d &&\n>> + test_must_fail git rev-parse HEAD:z/d &&\n>> + test ! -f z/d\n>> +'\n>> +\n>> +# Testcase 1d, Directory renames (merging two directories into one new one)\n>> +# cause a rename/rename(2to1) conflict\n>> +# (Related to testcases 1c and 7b)\n>> +# Commit A. z/{b,c}, y/{d,e}\n>> +# Commit B. x/{b,c}, y/{d,e,m,wham}\n>> +# Commit C. z/{b,c,n,wham}, x/{d,e}\n>> +# Expected: x/{b,c,d,e,m,n}, CONFLICT:(y/wham & z/wham -> x/wham)\n>> +# Note: y/m & z/n should definitely move into x. By the same token, both\n>> +# y/wham & z/wham should to...giving us a conflict.\n>\n> If wham are equal (in oid), shouldn't this not conflict and only conflict\n> when z/wham and x/wham differ in oid, but have the same sub-path?\n\nGood eyes, I should have labelled these as wham_1 and wham_2, since\nthe testcase did explicitly make them different (having contents of\n\"wham1\\n\" and \"wham2\\n\"), but it could make sense to add a test to the\ntestsuite where two such colliding files are identical.\n\nI thought about that, figured it wasn't worth it, and didn't add the\ncode to handle it. I could add a testcase for it, though I'd be very\ntempted to just leave it as test_expect_failure and let someone else\ncome along and implement it if someone feels so inclined. Note,\nthough, that the situation is made slightly more complex due to the\nfact that it might be N colliding files rather than just 2 (due to the\npossibility of multiple directories being merged into one on either\nside of history).\n\n>> +\n>> +# Testcase 1e, Renamed directory, with all filenames being renamed too\n>> +# Commit A: z/{oldb,oldc}\n>> +# Commit B: y/{newb,newc}\n>> +# Commit C: z/{oldb,oldc,d}\n>\n> What about oldd ?\n> (expecting a pattern across many files of s/old/new/ isn't unreasonable,\n> but maybe too much for now?)\n> By having no \"old\" prefix there is only one thing to do, which is y/d\n\nI'm not following. The \"old\" and \"new\" in the filenames were just\nthere so a human reading the testcase could easily tell which\nfilenames were related and involved in renames. There is no rename\nassociated with d, so why would I have called it \"old\" or \"new\"?\n\n>> +# Testcase 1f, Split a directory into two other directories\n>> +# (Related to testcases 3a, all of section 2, and all of section 4)\n>> +# Commit A: z/{b,c,d,e,f}\n>> +# Commit B: z/{b,c,d,e,f,g}\n>> +# Commit C: y/{b,c}, x/{d,e,f}\n>> +# Expected: y/{b,c}, x/{d,e,f,g}\n>\n> Why not y/g ? Because there are more files in x?\n\nYes.\n\n> I can come up with a reasonable counter example:\n\nOh sure, of course. You should also be able to come up with\ncounter-examples to the \"correctness\" of git's content merging of\nfiles when it fails to report any conflicts (e.g. one side added\nanother call to a function, the other side changed the number of\nparameters the function took). In short, this is just a case of where\nwe need to come up with simple predictable rules since we can't read\nminds. Here's the situation we're faced with, and my line of thinking\nin coming up with this simple, predictable, but definitely coarse\nrule:\n\n * Users sometimes rename directories on one side of history and add\nfiles to the original directory on the other side.\n * We would like to \"detect\" the directory rename, and move the new\nfiles into the correct directory.\n * We don't really have any hints for detecting directory renames\nother by comparing content, i.e. basing it on the file rename\ndetection we already have.\n * There is a possibility that not all files in a certain directory\nwent to the same location. It's possible that a few may have gone\nelsewhere.\n * Only in weird corner cases would I expect renamed-file location to\nbe split nearly 50-50 (as in this contrived testcase) -- although for\nsuch cases, as you point out, there is a much higher chance of us\ngetting the merge \"wrong\". Thus, our rule should be simple so people\ncan understand and expect what we did and have an easy time fixing it.\n\nSo, what to do? There are a few options:\n 1) Don't do directory rename detection at all. Just declare it to\nbe an anti-feature.\n 2) Try to guess why the user moved different files to different\ndirectories and try to mimic that somehow.\n 3) Only allow directory rename detection to happen when ALL renamed\nfiles in the directory went to the same directory.\n 4) Use a simple predictable rule like majority wins.\n\nI think 2 is insanity. Choices 1 and 3 don't have much appeal to me;\nI'm strongly against them. I'm unware of any remaining choices other\nthan 4, but 4 seems pretty reasonable to me. It won't always be\nright, but neither is simple content merging.\n"},{"id":"332481","messageId":"CABPp-BH2eGzTMzHvF9TD9uw_n8k_kbgvSF-ct8B9HMJRFAr6Ug@mail.gmail.com","threadId":"47162","inReplyTo":"CAGZ79kYOW9+pP5ApKDHR2_VwPXyiLEEbbO0-3ka-SDCP6ryLPw@mail.gmail.com","subject":"Re: [PATCH 06/30] directory rename detection: testcases to avoid taking detection too far","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-14T01:02:44Z","receivedAt":"2017-11-14T01:02:50Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Nov 13, 2017 at 3:25 PM, Stefan Beller wrote:\n> On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n\n>> +# Testcase 3a, Avoid implicit rename if involved as source on other side\n>> +# (Related to testcases 1c and 1f)\n>> +# Commit A: z/{b,c,d}\n>> +# Commit B: z/{b,c,d} (no change)\n>\n> This could also be an unrelated change such as adding w/e?\n\nYes, precisely. Whenever I have a \"no change\" commit, the intent is\nthat there may be unrelated changes involved, they've just been\nexcluded from the testcase in order to make it minimal.\n\n>> +# Commit C: y/{b,c}, x/d\n>> +# Expected: y/{b,c}, x/d\n"},{"id":"332485","messageId":"CABPp-BEVJ3c-Y+9buqh-XE8K5ETCH_iGpVxCDXdrdHN5iuAebQ@mail.gmail.com","threadId":"47162","inReplyTo":"CAGZ79kana0_9N+gG+m2BT5Tnu_q4MC=WSTuvtwkdeD_Wn70ydA@mail.gmail.com","subject":"Re: [PATCH 08/30] directory rename detection: files/directories in the way of some renames","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-14T01:19:02Z","receivedAt":"2017-11-14T01:19:08Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Nov 13, 2017 at 4:15 PM, Stefan Beller wrote:\n> On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n>> +# Testcase 5c, Transitive rename would cause rename/rename/rename/add/add/add\n>> +# (Directory rename detection would result in transitive rename vs.\n>> +# rename/rename(1to2) and turn it into a rename/rename(1to3). Further,\n>> +# rename paths conflict with separate adds on the other side)\n>> +# (Related to testcases 3b and 7c)\n>> +# Commit A: z/{b,c}, x/d_1\n>> +# Commit B: y/{b,c,d_2}, w/d_1\n>> +# Commit C: z/{b,c,d_1,e}, w/d_3, y/d_4\n>> +# Expected: A mess, but only a rename/rename(1to2)/add/add mess. Use the\n>> +# presence of y/d_4 in C to avoid doing transitive rename of\n>> +# x/d_1 -> z/d_1 -> y/d_1, so that the only paths we have at\n>> +# y/d are y/d_2 and y/d_4. We still do the move from z/e to y/e,\n>> +# though, because it doesn't have anything in the way.\n>\n> Missing the expected state, only the explanation is given.\n\nYeah...it seemed so ugly to try to write down. As a possible\nsidenote, this testcase was actually guided by the final test of\nt6042, which is messy enough, but directory rename detection provides\na little extra freedom to get a higher order conflict and make things\na bit messier. It felt like it was a case where just leaving the\nexpectation in code in the 5c-check was just easier and maybe even\nclearer. Should I add a comment to that effect, or would you really\njust prefer to see it spelled out?\n\n>> falling\n>> +# back to old handling. But, sadly, see testcases 8a and 8b.\n>\n> You seem to be hinting at these all the time.\n\nI think there were just multiple angles at which to approach those\ntestcases. *shrug*\n"},{"id":"332486","messageId":"CAGZ79kbK-deKq4KR-G5nq8MoCNK77hbr_B32W6Uo4i5Kq4_ojg@mail.gmail.com","threadId":"47162","inReplyTo":"CABPp-BFKiam6AK-Gg_RzaLuLur-jz0kvv3TqsHNHg5+HTv_uzA@mail.gmail.com","subject":"Re: [PATCH 04/30] directory rename detection: basic testcases","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-14T01:21:49Z","receivedAt":"2017-11-14T01:21:56Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Nov 13, 2017 at 4:57 PM, Elijah Newren wrote:\n> On Mon, Nov 13, 2017 at 2:04 PM, Stefan Beller wrote:\n>> On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n>> (minor thought:)\n>> After rereading the docs above this is clear; I wonder if instead of A, B, C\n>> a notation of Base, ours, theirs would be easier to understand?\n>\n> Sure, that'd be an easy change.\n>\n>>> +test_expect_success '1a-setup: Simple directory rename detection' '\n>>> +test_expect_failure '1a-check: Simple directory rename detection' '\n>>\n>> Thanks for splitting the setup and the check into two different test cases!\n>>\n>>\n>>> + git checkout B^0 &&\n>>\n>> Any reason for ^0 ? (to make clear it is a branch?)\n>\n> Partially force-of-habit (did the same with t6036 and t6042), but it\n> seemed to have the nice feature of making debugging easier through\n> improved reproducability. In particular, if I had checked out B\n> rather than B^0 in the testcase and a merge succeeded when I didn't\n> expect it, then attempting to run the same commands gives me a\n> different starting point for the merge. By instead explicitly\n> checking out B^0, then even if the merge succeeds, someone who again\n> runs checkout B^0 will have the same starting point.\n\nThanks for enlightening me. Makes sense.\n\n>\n>>> +test_expect_success '1b-setup: Merge a directory with another' '\n>>> + git rm -rf . &&\n>>> + git clean -fdqx &&\n>>> + rm -rf .git &&\n>>> + git init &&\n>>\n>> This is quite a strong statement to start a test with.\n>\n> It was actually copy-paste from t6036, and achieved two purposes:\n> 1) Even if one test fails, subsequent ones continue running. (Had\n> lots of problems with this in t6036 years ago and just ended up with\n> those four steps)\n> 2) Someone who runs into a failing testcase has a _much_ easier time\n> figuring out what is going on with the testcase. I find it takes a\n> fair amount of time to figure out what's going on with other tests in\n> git's testsuite because of the presence of so many files completely\n> unrelated to the given test, which have simply accumulated from\n> previous tests. For many tests, that may be fine, but in particular\n> for t6036, t6042, and now t6043, since these are mostly about corner\n> cases that are hard enough to reason about, I didn't want the extra\n> distractions.\n\nI agree with these reasons to strongly want a clean slate.\n\n>> Nowadays we rather do\n>>\n>> test_when_finished \"some command\" &&\n>>\n>> than polluting the follow up tests. But as you split up the previous test\n>> into 2 tests, it is not clear if this would bring any good.\n>\n> test_when_finished looks pretty cool; I didn't know about it. Thanks\n> for the pointer. Not sure if we want to use it here (if we do, we'd\n> only do so in the check tests rather than the setup ones).\n>\n>> Also these are four cleanup commands, I'd have expected fewer.\n>> Maybe just \"rm -rf .\" ? Or as we make a new repo for each test case,\n>>\n>> test_create_repo 1a &&\n>> cd 1a\n>>\n>> in the first setup, and here we do\n>> test_create_repo 1b &&\n>> cd 1b\n>>\n>> relying on test_done to cleanup everything afterwards?\n>\n> rm aborts for me with 'rm -rf .', but I could possibly make it 'rm -rf\n> * .* && git init .'\n>\n> The test_create_repo might not be so bad as long as every test used it\n> and put all files under it's own little repo.\n\nThat is my current favorite, I would think.\n\n\n> It does create some\n> clutter, but it's at least somewhat managed.\n\nBut the clutter is outside the repository under test, which\nmay help with the situation.\n\n> I'm still a bit partial\n> to just totally cleaning it out, but if others would prefer, I can\n> switch.\n\n(slightly dreaming:)\nI wonder if we could teach our test suite to accept multiple test_done\nor restart_tests or such to resurrect the clean slate.\n\n>\n>>> + test 3 -eq $(git ls-files -s | wc -l) &&\n>>\n>> git ls-files >out &&\n>> test_line_count = 3 out &&\n>>\n>> maybe? Piping output of git commands somewhere is an\n>> anti-pattern as we cannot examine the return code of ls-files in this case.\n>\n> Um...I guess that makes sense, if you assumed that I cared about the\n> return code of ls-files.\n\nAs this is the test suite, we care about the return code of any git\ncommand, for current git as well as future-gits.\n\n> But it seems to make the tests with multiple\n> calls to ls-files in a row (of which there are many) considerably\n> uglier, so I'd personally prefer to avoid that. Also, why would I\n> care about the return code of ls-files?\n\nWhile I understand the rationale here and your examination of ls-files\nseems to indicate that we are ok doing it here, a lot of (test) code\nis taken for inspiration (copied, adapted) to other test cases.\nAnd most of the time we actually care if the return code is correct\nadditionally to the actions performed, so I was shooting from the hip\nhere.\n\n> Your suggestion made me curious, so I went looking. As far as I can\n> tell, the return code of ls-files is always 0 unless you both specify\n> both --error-unmatch and one or more paths, neither of which I did.\n> Am I missing something?\n\nI am not saying it was a cargo-culting reaction, but rather relaying\na well discussed style issue to you. ;)\n\n\n> If you feel the return code of ls-files is important, perhaps I could\n> just have a separate\n> git ls-files -s >/dev/null &&\n> call before the others?\n\nI would prefer to not add any further calls; also (speaking generically)\nthis would bring in potential racing issues (what if the second ls-files\nbehaves differently than the first?)\n\n>>> + test $(git rev-parse HEAD:y/b) = $(git rev-parse A:z/b) &&\n>>> + test $(git rev-parse HEAD:y/c) = $(git rev-parse A:z/c) &&\n>>> + test $(git rev-parse HEAD:y/d) = $(git rev-parse A:x/d) &&\n>>\n>> Speaking of these, there are quite a lot of invocations of rev-parse,\n>> though it looks clean; recently Junio had some good ideas regarding a\n>> similar test[1].\n>> So maybe\n>>\n>> git rev-parse >actual \\\n>> HEAD:y/b HEAD:y/c HEAD:y/d &&\n>> git rev-parse >expect \\\n>> A:z/b A:z/c A:x/d &&\n>> test_cmp expect actual\n>>\n>> Not sure if that is any nicer, but has fewer calls.\n>\n> Sure, I can switch it over.\n\nWell, that was also just a quick suggestion, maybe we'll find an\neven nicer way that has also very few invocations.\n\n>>> +\n>>> +# Testcase 1e, Renamed directory, with all filenames being renamed too\n>>> +# Commit A: z/{oldb,oldc}\n>>> +# Commit B: y/{newb,newc}\n>>> +# Commit C: z/{oldb,oldc,d}\n>>\n>> What about oldd ?\n>> (expecting a pattern across many files of s/old/new/ isn't unreasonable,\n>> but maybe too much for now?)\n>> By having no \"old\" prefix there is only one thing to do, which is y/d\n>\n> I'm not following. The \"old\" and \"new\" in the filenames were just\n> there so a human reading the testcase could easily tell which\n> filenames were related and involved in renames. There is no rename\n> associated with d, so why would I have called it \"old\" or \"new\"?\n\nbecause a user may be impressed by gits pattern matching in the\nrename detection:\n\n A: z/{oldb,oldc}\n B: z/{newb,newc}\n C: z/{oldb, oldc, oldd}\n\nObviously A->B is z/{old->new}-* (not a directory rename,\nbut just patterns), so one might be tempted to expect newd\nas the expectation. But that is nonsense(!?)\n\n>\n> * Users sometimes rename directories on one side of history and add\n> files to the original directory on the other side.\n> * We would like to \"detect\" the directory rename, and move the new\n> files into the correct directory.\n> * We don't really have any hints for detecting directory renames\n> other by comparing content, i.e. basing it on the file rename\n> detection we already have.\n> * There is a possibility that not all files in a certain directory\n> went to the same location. It's possible that a few may have gone\n> elsewhere.\n> * Only in weird corner cases would I expect renamed-file location to\n> be split nearly 50-50 (as in this contrived testcase) -- although for\n> such cases, as you point out, there is a much higher chance of us\n> getting the merge \"wrong\". Thus, our rule should be simple so people\n> can understand and expect what we did and have an easy time fixing it.\n\nMakes sense\n\n>\n> So, what to do? There are a few options:\n> 1) Don't do directory rename detection at all. Just declare it to\n> be an anti-feature.\n> 2) Try to guess why the user moved different files to different\n> directories and try to mimic that somehow.\n> 3) Only allow directory rename detection to happen when ALL renamed\n> files in the directory went to the same directory.\n> 4) Use a simple predictable rule like majority wins.\n\nI wonder if eventually (down the road, not now) we would want to\nalso teach custom merge/diff drivers about potential dir renaming.\n(Well not these drivers itself, but rather a method by which a custom diff\ndriver can tell us to use another custom rule for thees rename detections)\n\n>\n> I think 2 is insanity.\n\nor the place where hooks/custom code shines. :)\n\n> Choices 1 and 3 don't have much appeal to me;\n> I'm strongly against them. I'm unware of any remaining choices other\n> than 4, but 4 seems pretty reasonable to me. It won't always be\n> right, but neither is simple content merging.\n\nThat makes sense, thanks!\nStefan\n"},{"id":"332487","messageId":"xmqqshdh7iqw.fsf@gitster.mtv.corp.google.com","threadId":"47162","inReplyTo":"20171110190550.27059-4-newren@gmail.com","subject":"Re: [PATCH 03/30] merge-recursive: Add explanation for src_entry and dst_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-14T01:26:47Z","receivedAt":"2017-11-14T01:26:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> If I have to walk through the debugger and inspect the values found in\n> here in order to figure out their meaning, despite having known these\n> things inside and out some years back, then they probably need a comment\n> for the casual reader to explain their purpose.\n>\n> Signed-off-by: Elijah Newren \n> ---\n> merge-recursive.c | 22 ++++++++++++++++++++++\n> 1 file changed, 22 insertions(+)\n>\n> diff --git a/merge-recursive.c b/merge-recursive.c\n> index 52521faf09..3526c8d0b8 100644\n> --- a/merge-recursive.c\n> +++ b/merge-recursive.c\n> @@ -513,6 +513,28 @@ static void record_df_conflict_files(struct merge_options *o,\n> \n> struct rename {\n> \tstruct diff_filepair *pair;\n> +\t/*\n> +\t * Because I keep forgetting every few years what src_entry and\n> +\t * dst_entry are and have to walk through a debugger and puzzle\n> +\t * through it to remind myself...\n\nVery much appreciated. I recall having trouble reasoning about\nthem myself, too (even though I admit I wasn't involved in the\nimplementation of this part very much and know this code a lot less\nintimately than you do in the first place).\n\n> +\t *\n> +\t * If 'before' is renamed to 'after' then src_entry will contain\n> +\t * the versions of 'before' from the merge_base, HEAD, and MERGE in\n> +\t * stages 1, 2, and 3; dst_entry will contain the versions of\n> +\t * 'after' from the merge_base, HEAD, and MERGE in stages 1, 2, and\n> +\t * 3. Thus, we have a total of six modes and oids, though some\n> +\t * will be null. (Stage 0 is ignored; we're interested in handling\n> +\t * conflicts.)\n> +\t *\n> +\t * Since we don't turn on break-rewrites by default, neither\n> +\t * src_entry nor dst_entry can have all three of their stages have\n> +\t * non-null oids, meaning at most four of the six will be non-null.\n> +\t * Also, since this is a rename, both src_entry and dst_entry will\n> +\t * have at least one non-null oid, meaning at least two will be\n> +\t * non-null. Of the six oids, a typical rename will have three be\n> +\t * non-null. Only two implies a rename/delete, and four implies a\n> +\t * rename/add.\n> +\t */\n> \tstruct stage_data *src_entry;\n> \tstruct stage_data *dst_entry;\n> \tunsigned processed:1;\n"},{"id":"332488","messageId":"CABPp-BFfWTRvaQrBn+gOKsu8s9fzT2eiTVeSw93p6SvzdQBPjw@mail.gmail.com","threadId":"47162","inReplyTo":"CAGZ79kbsQqAFGFVN3dOnSAFuPishF3PGS7_go+53T5Oa-4qWiA@mail.gmail.com","subject":"Re: [PATCH 09/30] directory rename detection: testcases checking which side did the rename","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-14T01:30:43Z","receivedAt":"2017-11-14T01:30:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Nov 13, 2017 at 4:25 PM, Stefan Beller wrote:\n> On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n>> +# Testcase 6b, Same rename done on both sides\n>> +# (Related to testcases 6c and 8e)\n>> +# Commit A: z/{b,c}\n>> +# Commit B: y/{b,c}\n>> +# Commit C: y/{b,c}, z/d\n>\n> Missing expected state\n\nOops!\n\n>> +# Note: If we did directory rename detection here, we'd move z/d into y/,\n>> +# but C did that rename and still decided to put the file into z/,\n>> +# so we probably shouldn't apply directory rename detection for it.\n>\n> correct. Also we don't want to see a rename/rename conflict (obviously).\n\nInterestingly, (and this is unrelated to directory rename detection)\nmerge-recursive.c has a RENAME_ONE_FILE_TO_ONE value exactly for the\ncase where one file was renamed on both sides of history, but was\nrenamed to the exact same thing on both sides. And in those cases, it\nturns what would otherwise be a rename/rename(1to2) conflict into\nessentially a RENAME_NORMAL case. So, we wouldn't have to worry about\na rename/rename conflict anyway.\n\n>> +# Testcase 6e, Add/add from one-side\n>> +# Commit A: z/{b,c}\n>> +# Commit B: z/{b,c} (no change)\n>> +# Commit C: y/{b,c,d_1}, z/d_2\n>> +# Expected: y/{b,c,d_1}, z/d_2\n>> +# NOTE: Again, this seems obvious but just checking that the implementation\n>> +# doesn't \"accidentally detect a rename\" and give us y/{b,c} +\n>> +# add/add conflict on y/d_1 vs y/d_2.\n>\n> What is less obvious in all these cases is the \"(no change)\" part to me.\n> I would think that at least *something* changes in B in all cases above, maybe\n> add file u/r (un-related) to have the tree ids changed?\n> (\"Less obvious\" as in: we don't rely on the \"no changes\" part to make\n> the decision,\n> which sounds tempting so far)\n\nYes, I could have introduced unrelated changes into the test, and my\nassumption is that the real world testcase would have such, it's just\nthat in paring down to a minimal testcase I end up with a \"no change\"\ncommit.\n\n>> test_done\n>\n> No conclusion box here, so my (misguided) suggestion:\n\nYeah, the conclusion was actually in the summary. I could potentially\nrestate it here:\n\n\"Only apply implicit directory renames to directories if the _other_\nside of history is the one doing the renaming\"\n\nI can add that.\n"},{"id":"332489","messageId":"CABPp-BGhxK4r_jhniOTT=UsFtu4C9yYPMF3azASQTiY=fVVDrA@mail.gmail.com","threadId":"47162","inReplyTo":"CAGZ79kbK-deKq4KR-G5nq8MoCNK77hbr_B32W6Uo4i5Kq4_ojg@mail.gmail.com","subject":"Re: [PATCH 04/30] directory rename detection: basic testcases","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-14T01:40:58Z","receivedAt":"2017-11-14T01:41:04Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Nov 13, 2017 at 5:21 PM, Stefan Beller wrote:\n> On Mon, Nov 13, 2017 at 4:57 PM, Elijah Newren wrote:\n\n> (slightly dreaming:)\n> I wonder if we could teach our test suite to accept multiple test_done\n> or restart_tests or such to resurrect the clean slate.\n\nI'm dreaming now too; I would like that a lot more, although the\nseparate test_create_repo for each case isn't too bad and should be a\npretty mechanical switch.\n\n>>>> + test 3 -eq $(git ls-files -s | wc -l) &&\n>>>\n>>> git ls-files >out &&\n>>> test_line_count = 3 out &&\n>>>\n\n> I am not saying it was a cargo-culting reaction, but rather relaying\n> a well discussed style issue to you. ;)\n\nAh, gotcha.\n\n>> If you feel the return code of ls-files is important, perhaps I could\n>> just have a separate\n>> git ls-files -s >/dev/null &&\n>> call before the others?\n>\n> I would prefer to not add any further calls; also (speaking generically)\n> this would bring in potential racing issues (what if the second ls-files\n> behaves differently than the first?)\n\nMakes sense.\n\n>> I'm not following. The \"old\" and \"new\" in the filenames were just\n>> there so a human reading the testcase could easily tell which\n>> filenames were related and involved in renames. There is no rename\n>> associated with d, so why would I have called it \"old\" or \"new\"?\n>\n> because a user may be impressed by gits pattern matching in the\n> rename detection:\n>\n> A: z/{oldb,oldc}\n> B: z/{newb,newc}\n> C: z/{oldb, oldc, oldd}\n>\n> Obviously A->B is z/{old->new}-* (not a directory rename,\n> but just patterns), so one might be tempted to expect newd\n> as the expectation. But that is nonsense(!?)\n\nAh, now I see where you were going. Thanks for explaining.\n\n>> I think 2 is insanity.\n>\n> or the place where hooks/custom code shines. :)\n\n:)\n"},{"id":"332490","messageId":"xmqqmv3p7h1p.fsf@gitster.mtv.corp.google.com","threadId":"47162","inReplyTo":"CAGZ79kbmt7jt13D-HY90+LBaCHsqvDOYnrmJ41hR3YsgEceirg@mail.gmail.com","subject":"Re: [PATCH 04/30] directory rename detection: basic testcases","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-14T02:03:30Z","receivedAt":"2017-11-14T02:03:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n>> Signed-off-by: Elijah Newren \n>> ...\n>> +# B\n>> +# o\n>> +# / \\\n>> +# A o ?\n>> +# \\ /\n>> +# o\n>> +# C\n>> + ...\n>> +# Testcase 1a, Basic directory rename.\n>> +# Commit A: z/{b,c}\n>> +# Commit B: y/{b,c}\n>> +# Commit C: z/{b,c,d,e/f}\n>\n> (minor thought:)\n> After rereading the docs above this is clear; I wonder if instead of A, B, C\n> a notation of Base, ours, theirs would be easier to understand?\n\nI had a similar thought, but as long as everything in this file is\nconsistent, as we have that picture upfront, I am OK with it. FWIW,\nt1000 uses O (original--common ancestor) A and B, which was the\nnotation commonly used in our codebase since the early days when we\nneeded to call them with single letters.\n\n>> +test_expect_success '1a-setup: Simple directory rename detection' '\n>> +test_expect_failure '1a-check: Simple directory rename detection' '\n>\n> Thanks for splitting the setup and the check into two different test cases!\n>\n>\n>> + git checkout B^0 &&\n>\n> Any reason for ^0 ? (to make clear it is a branch?)\n\nI think it is to make it clear that no matter what this test does\n(or fails to do), the branch B is *not* affected by it because we'd\nbe playing on a detached head.\n\n>> +test_expect_success '1b-setup: Merge a directory with another' '\n>> + git rm -rf . &&\n>> + git clean -fdqx &&\n>> + rm -rf .git &&\n>> + git init &&\n>\n> This is quite a strong statement to start a test with.\n\nYes. If a test before this one did cd ../.. and forgot to come\nback, we'd be in trouble. If we want a fresh repository perhaps\ntest-create-repo inside the trash repository may be a less evil\noption.\n"},{"id":"332502","messageId":"xmqqh8tx5ux8.fsf@gitster.mtv.corp.google.com","threadId":"47162","inReplyTo":"20171110190550.27059-16-newren@gmail.com","subject":"Re: [PATCH 15/30] merge-recursive: Move the get_renames() function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-14T04:46:43Z","receivedAt":"2017-11-14T04:47:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> I want to re-use some other functions in the file without moving those other\n> functions or dealing with a handful of annoying split function declarations\n> and definitions.\n>\n> Signed-off-by: Elijah Newren \n> ---\n\nIt took me a while to figure out that you are basing this on top of\na slightly older tip of 'master'. When rebasing on, or merging to,\na newer codebase, these two lines\n\n> -\tdiff_setup(&opts);\n> -\tDIFF_OPT_SET(&opts, RECURSIVE);\n> -\tDIFF_OPT_CLR(&opts, RENAME_EMPTY);\n> -\topts.detect_rename = DIFF_DETECT_RENAME;\n> ...\n> +\tdiff_setup(&opts);\n> +\tDIFF_OPT_SET(&opts, RECURSIVE);\n> +\tDIFF_OPT_CLR(&opts, RENAME_EMPTY);\n> +\topts.detect_rename = DIFF_DETECT_RENAME;\n\nwould need a bit of adjustment.\n\nBy the way, checkpatch.pl complains about // C99 comments and binary\noperators missing SP on both ends, etc., on the entire series [*1*].\nThese look like small issues, but they are distracting enough to\nbreak concentration while reading the changes to spot places with\nreal issues and places that can be improved, so cleaning them up\nearly would help the final result to get better reviews.\n\nI won't reproduce all of them here, but here are a representable\nfew.\n\nERROR: spaces required around that '=' (ctx:VxV)\n#30: FILE: merge-recursive.c:1491:\n+ for (i=0; inr; i++) {\n ^\n\nERROR: spaces required around that '<' (ctx:VxV)\n#30: FILE: merge-recursive.c:1491:\n+ for (i=0; inr; i++) {\n ^\n\nERROR: \"foo* bar\" should be \"foo *bar\"\n#42: FILE: merge-recursive.c:1503:\n+static char* handle_path_level_conflicts(struct merge_options *o,\n\nWARNING: suspect code indent for conditional statements (8, 10)\n#80: FILE: merge-recursive.c:791:\n+ if (o->call_depth || !was_tracked(path))\n+ return !dirty;\n\nThanks.\n\n[Footnote]\n\nJust FYI, checkpatch.pl also notices these but it seems that our\nexisting codebase already violates them in a major way, so I usually\ndo not pay attention to these classes of complaints:\n\nERROR: spaces required around that ':' (ctx:VxV)\n#30: FILE: merge-recursive.c:603:\n+ unsigned add_turned_into_rename:1;\n ^\n\nWARNING: quoted string split across lines\n#74: FILE: merge-recursive.c:1433:\n+ output(o, 1, _(\"Refusing to lose untracked file at \"\n+ \"%s, even though it's in the way.\"),\n"},{"id":"332503","messageId":"xmqqd14l5ugx.fsf@gitster.mtv.corp.google.com","threadId":"47162","inReplyTo":"20171110190550.27059-17-newren@gmail.com","subject":"Re: [PATCH 16/30] merge-recursive: Introduce new functions to handle rename logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-14T04:56:30Z","receivedAt":"2017-11-14T04:56:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> +struct rename_info {\n> +\tstruct string_list *head_renames;\n> +\tstruct string_list *merge_renames;\n> +};\n\nThis type is added in order to allow the caller and the helper to\ncommunicate the findings in a single logical structure, instead of\nhaving to pass them as separate parameters, etc. If we anticipate\nthat the information that needs to be passed will grow richer in\nlater steps (or a follow-up series), such encapsulation makes a lot\nof sence.\n\n> +static struct rename_info *handle_renames(struct merge_options *o,\n> +\t\t\t\t\t struct tree *common,\n> +\t\t\t\t\t struct tree *head,\n> +\t\t\t\t\t struct tree *merge,\n> +\t\t\t\t\t struct string_list *entries,\n> +\t\t\t\t\t int *clean)\n> +{\n> +\tstruct rename_info *rei = xcalloc(1, sizeof(struct rename_info));\n\nI however notice that there is only one caller of this helper at\nthis step, and also at the end of this series. I suspect that it\nwould probably be a better design to make \"clean\" the return value\nof this helper, and instead have the caller pass an uninitialised\nrename_info structure on its stack by address to be filled by the\nhelper.\n\n> +\trei->head_renames = get_renames(o, head, common, head, merge, entries);\n> +\trei->merge_renames = get_renames(o, merge, common, head, merge, entries);\n> +\t*clean = process_renames(o, rei->head_renames, rei->merge_renames);\n> +\n> +\treturn rei;\n> +}\n> +\n> +static void cleanup_renames(struct rename_info *re_info)\n> +{\n> +\tstring_list_clear(re_info->head_renames, 0);\n> +\tstring_list_clear(re_info->merge_renames, 0);\n> +\n> +\tfree(re_info->head_renames);\n> +\tfree(re_info->merge_renames);\n> +\n> +\tfree(re_info);\n> +}\n> static struct object_id *stage_oid(const struct object_id *oid, unsigned mode)\n> {\n> \treturn (is_null_oid(oid) || mode == 0) ? NULL: (struct object_id *)oid;\n> @@ -1989,7 +2021,8 @@ int merge_trees(struct merge_options *o,\n> \t}\n> \n> \tif (unmerged_cache()) {\n> -\t\tstruct string_list *entries, *re_head, *re_merge;\n> +\t\tstruct string_list *entries;\n> +\t\tstruct rename_info *re_info;\n> \t\tint i;\n> \t\t/*\n> \t\t * Only need the hashmap while processing entries, so\n> @@ -2003,9 +2036,7 @@ int merge_trees(struct merge_options *o,\n> \t\tget_files_dirs(o, merge);\n> \n> \t\tentries = get_unmerged();\n> -\t\tre_head = get_renames(o, head, common, head, merge, entries);\n> -\t\tre_merge = get_renames(o, merge, common, head, merge, entries);\n> -\t\tclean = process_renames(o, re_head, re_merge);\n> +\t\tre_info = handle_renames(o, common, head, merge, entries, &clean);\n> \t\trecord_df_conflict_files(o, entries);\n> \t\tif (clean < 0)\n> \t\t\tgoto cleanup;\n> @@ -2030,16 +2061,13 @@ int merge_trees(struct merge_options *o,\n> \t\t}\n> \n> cleanup:\n> -\t\tstring_list_clear(re_merge, 0);\n> -\t\tstring_list_clear(re_head, 0);\n> +\t\tcleanup_renames(re_info);\n> +\n> \t\tstring_list_clear(entries, 1);\n> +\t\tfree(entries);\n> \n> \t\thashmap_free(&o->current_file_dir_set, 1);\n> \n> -\t\tfree(re_merge);\n> -\t\tfree(re_head);\n> -\t\tfree(entries);\n> -\n> \t\tif (clean < 0)\n> \t\t\treturn clean;\n> \t}\n"},{"id":"332504","messageId":"xmqq60ad5udx.fsf@gitster.mtv.corp.google.com","threadId":"47162","inReplyTo":"20171110190550.27059-18-newren@gmail.com","subject":"Re: [PATCH 17/30] merge-recursive: Fix leaks of allocated renames and diff_filepairs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-14T04:58:18Z","receivedAt":"2017-11-14T04:58:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> get_renames() has always zero'ed out diff_queued_diff.nr while only\n> manually free'ing diff_filepairs that did not correspond to renames.\n> Further, it allocated struct renames that were tucked away in the\n> return string_list. Make sure all of these are deallocated when we\n> are done with them.\n>\n> Signed-off-by: Elijah Newren \n> ---\n> merge-recursive.c | 15 +++++++++++++--\n> 1 file changed, 13 insertions(+), 2 deletions(-)\n>\n> diff --git a/merge-recursive.c b/merge-recursive.c\n> index 49710c0964..7a3402e50c 100644\n> --- a/merge-recursive.c\n> +++ b/merge-recursive.c\n> @@ -1661,10 +1661,21 @@ static struct rename_info *handle_renames(struct merge_options *o,\n> \n> static void cleanup_renames(struct rename_info *re_info)\n> {\n> -\tstring_list_clear(re_info->head_renames, 0);\n> -\tstring_list_clear(re_info->merge_renames, 0);\n> +\tconst struct rename *re;\n> +\tint i;\n> \n> +\tfor (i = 0; i < re_info->head_renames->nr; i++) {\n> +\t\tre = re_info->head_renames->items[i].util;\n> +\t\tdiff_free_filepair(re->pair);\n> +\t}\n> +\tstring_list_clear(re_info->head_renames, 1);\n> \tfree(re_info->head_renames);\n> +\n> +\tfor (i = 0; i < re_info->merge_renames->nr; i++) {\n> +\t\tre = re_info->merge_renames->items[i].util;\n> +\t\tdiff_free_filepair(re->pair);\n> +\t}\n> +\tstring_list_clear(re_info->merge_renames, 1);\n\nAnd this obviously will be helped by having another helper \"cleanup_rename()\"\nthat does one of them, and call it twice from here.\n"},{"id":"332505","messageId":"xmqqy3n94f36.fsf@gitster.mtv.corp.google.com","threadId":"47162","inReplyTo":"xmqqd14l5ugx.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 16/30] merge-recursive: Introduce new functions to handle rename logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-14T05:14:05Z","receivedAt":"2017-11-14T05:14:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Elijah Newren writes:\n>\n>> +struct rename_info {\n>> +\tstruct string_list *head_renames;\n>> +\tstruct string_list *merge_renames;\n>> +};\n>\n> This type is added in order to allow the caller and the helper to\n> communicate the findings in a single logical structure, instead of\n> having to pass them as separate parameters, etc. If we anticipate\n> that the information that needs to be passed will grow richer in\n> later steps (or a follow-up series), such encapsulation makes a lot\n\nHmph, I actually am quite confused with the existing code.\n\nThe caller (originally in merge_trees(), now in handle_renames())\ncalls get_renames() twice and have the list of renamed paths in\nthese two string lists. get_renames() mostly works with the\nelements in the \"entries\" list and adds the \"struct rename\" to the\nstring list that is to be returned. And the caller uses these two\nstring lists get_renames() returns when calling process_renames(),\nbut once process_renames() is done with them, these two string lists\nare never looked at by anybody.\n\nSo do we really need to pass this structure around in the first\nplace? I am wondering if we can do the cleanup_rename() on both of\nthese lists after handle_renames() calls process_renames() before\nreturning, which will eliminate the need for this structure and a\nseparate cleanup_renames() helper that clears the structure and the\ntwo string lists in it.\n\n"},{"id":"332507","messageId":"xmqqtvxx4esv.fsf@gitster.mtv.corp.google.com","threadId":"47162","inReplyTo":"20171110190550.27059-20-newren@gmail.com","subject":"Re: [PATCH 19/30] merge-recursive: Split out code for determining diff_filepairs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-14T05:20:16Z","receivedAt":"2017-11-14T05:20:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> Create a new function, get_diffpairs() to compute the diff_filepairs\n> between two trees. While these are currently only used in\n> get_renames(), I want them to be available to some new functions. No\n> actual logic changes yet.\n\nOK. \n\nThis refactors an easy-to-use (in the context of merge-recursive\ncode) wrapper to diff-tree out of the existing code, which makes\nsense.\n\n"},{"id":"332509","messageId":"xmqqmv3p4ebo.fsf@gitster.mtv.corp.google.com","threadId":"47162","inReplyTo":"20171110190550.27059-22-newren@gmail.com","subject":"Re: [PATCH 21/30] merge-recursive: Add get_directory_renames()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-14T05:30:35Z","receivedAt":"2017-11-14T05:30:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> +\t\tentry = dir_rename_find_entry(dir_renames, old_dir);\n> +\t\tif (!entry) {\n> +\t\t\tentry = xcalloc(1, sizeof(struct dir_rename_entry));\n> +\t\t\thashmap_entry_init(entry, strhash(old_dir));\n\nPlease make these two lines into its own dir_rename_entry_init()\nhelper. Because the structure is defined as\n\n+struct dir_rename_entry {\n+\tstruct hashmap_entry ent; /* must be the first member! */\n+\tchar *dir;\n+\tunsigned non_unique_new_dir:1;\n+\tchar *new_dir;\n+\tstruct string_list possible_new_dirs;\n+};\n+\n\nin the previous patch, we'd want to see its string_list member to be\ninitialised explicitly (we do not want to depend on \"filling with\nNUL happens to make it a NODUP kind of string_list, which suits our\npurpose\"). The definition of _init() function may belong to the\nprevious step.\n\n"},{"id":"332557","messageId":"CABPp-BE8Jc-ZmW2bt3Fe3Azd6wEiSTUjsmhNNNdoip_LXHC_Yg@mail.gmail.com","threadId":"47162","inReplyTo":"xmqqh8tx5ux8.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 15/30] merge-recursive: Move the get_renames() function","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-14T17:41:32Z","receivedAt":"2017-11-14T17:41:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Nov 13, 2017 at 8:46 PM, Junio C Hamano wrote:\n\n> It took me a while to figure out that you are basing this on top of\n> a slightly older tip of 'master'. When rebasing on, or merging to,\n> a newer codebase\n\nSorry about that. Yes, I worked on the series over time and rebased a\ncouple times up to v2.15.0. I assumed that was new enough, but\nclearly I was wrong.\n\n> By the way, checkpatch.pl complains about // C99 comments and binary\n> operators missing SP on both ends, etc., on the entire series [*1*].\n> These look like small issues, but they are distracting enough to\n> break concentration while reading the changes to spot places with\n> real issues and places that can be improved, so cleaning them up\n> early would help the final result to get better reviews.\n>\n> I won't reproduce all of them here, but here are a representable\n> few.\n\nEek! My apologies. I will go through and fix them up. I see no\nreference to checkpatch.pl in git, but a google search shows there's\none in the linux source tree. Is that were I get it from, or is there\na different one?\n\nAlso, would you like me to make a separate commit that cleans up\npre-existing issues in merge-recursive.c so that it runs clean, or\njust remove the problems I added?\n\n\nThanks for all the reviews!\n"},{"id":"332559","messageId":"CABPp-BFJm9C1zTKVLeFf6fnhxrUhTd+XZsD17DezqFKwwzsEBQ@mail.gmail.com","threadId":"47162","inReplyTo":"xmqqy3n94f36.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 16/30] merge-recursive: Introduce new functions to handle rename logic","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-14T18:24:37Z","receivedAt":"2017-11-14T18:24:47Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Nov 13, 2017 at 9:14 PM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n>\n>> Elijah Newren writes:\n>>\n>>> +struct rename_info {\n>>> + struct string_list *head_renames;\n>>> + struct string_list *merge_renames;\n>>> +};\n>>\n>> This type is added in order to allow the caller and the helper to\n>> communicate the findings in a single logical structure, instead of\n>> having to pass them as separate parameters, etc. If we anticipate\n>> that the information that needs to be passed will grow richer in\n>> later steps (or a follow-up series), such encapsulation makes a lot\n>\n> Hmph, I actually am quite confused with the existing code.\n>\n> The caller (originally in merge_trees(), now in handle_renames())\n> calls get_renames() twice and have the list of renamed paths in\n> these two string lists. get_renames() mostly works with the\n> elements in the \"entries\" list and adds the \"struct rename\" to the\n> string list that is to be returned. And the caller uses these two\n> string lists get_renames() returns when calling process_renames(),\n> but once process_renames() is done with them, these two string lists\n> are never looked at by anybody.\n\nActually, if I remember correctly, my first stab was to do all the\ncleanup at the end of handle_renames(), but then I ran into\nuse-after-free errors. I'm not sure if I remember all the details,\nbut I'll try to lay out the path:\n\nprocess_renames() can't handle conflicts immediately because of D/F\nconcerns (if all entries in the competing directory resolve away, then\nthere's no more D/F conflict, but we have to wait until each of those\nentries is processed to find out if that happens or if a D/F conflict\nremains). Because of that, process_renames() needs to store\ninformation into a rename_conflict_info struct for process_entry() to\nlook at later. Included in rename_conflict_info are things like\ndiff_filepair and stage_data entries, both taken from the rename\nlists. If the rename lists are freed at the end of handle_renames(),\nthen this information is freed before process_entry() runs and thus we\nget a use-after-free error.\n\nSince both you and I thought to push this cleanup to the end of\nhandle_renames(), though, I should probably add that explanation to\nthe commit message. Granted, it isn't actually needed for this\nparticular commit, because up to this point all the information used\nin rename_conflict_info was leaked anyway. But it becomes an issue\nwith patch 17 when we start freeing that info.\n"},{"id":"332561","messageId":"CABPp-BEsEwqAZWtHVe9zhn_KtnLo9yv55666FAiDsMEw62yZoA@mail.gmail.com","threadId":"47162","inReplyTo":"xmqqmv3p4ebo.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 21/30] merge-recursive: Add get_directory_renames()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-14T18:38:09Z","receivedAt":"2017-11-14T18:38:15Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Nov 13, 2017 at 9:30 PM, Junio C Hamano wrote:\n> Elijah Newren writes:\n>\n>> + entry = dir_rename_find_entry(dir_renames, old_dir);\n>> + if (!entry) {\n>> + entry = xcalloc(1, sizeof(struct dir_rename_entry));\n>> + hashmap_entry_init(entry, strhash(old_dir));\n>\n> Please make these two lines into its own dir_rename_entry_init()\n> helper.\n\n> we'd want to see its string_list member to be\n> initialised explicitly\n\nWill do.\n"},{"id":"332571","messageId":"CAGZ79kYCDBFwMhW4Qgwhomiv4kKRWS53cjMLuRG1NxrzULYwgg@mail.gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-12-newren@gmail.com","subject":"Re: [PATCH 11/30] directory rename detection: testcases exploring possibly suboptimal merges","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-14T20:33:54Z","receivedAt":"2017-11-14T20:34:01Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n> Signed-off-by: Elijah Newren \n> ---\n> t/t6043-merge-rename-directories.sh | 371 ++++++++++++++++++++++++++++++++++++\n> 1 file changed, 371 insertions(+)\n>\n> diff --git a/t/t6043-merge-rename-directories.sh b/t/t6043-merge-rename-directories.sh\n> index 115d0d2622..bdfd943c88 100755\n> --- a/t/t6043-merge-rename-directories.sh\n> +++ b/t/t6043-merge-rename-directories.sh\n> @@ -1683,4 +1683,375 @@ test_expect_failure '7e-check: transitive rename in rename/delete AND dirs in th\n> test $(git hash-object y/d~C^0) = $(git rev-parse A:x/d)\n> '\n>\n> +\n> +###########################################################################\n> +# SECTION 8: Suboptimal merges\n> +#\n> +# As alluded to in the last section, the ruleset we have built up for\n> +# detecting directory renames unfortunately has some special cases where it\n> +# results in slightly suboptimal or non-intuitive behavior. This section\n> +# explores these cases.\n> +#\n> +# To be fair, we already had non-intuitive or suboptimal behavior for most\n> +# of these cases in git before introducing implicit directory rename\n> +# detection, but it'd be nice if there was a modified ruleset out there\n> +# that handled these cases a bit better.\n> +###########################################################################\n> +\n> +# Testcase 8a, Dual-directory rename, one into the others' way\n> +# Commit A. x/{a,b}, y/{c,d}\n> +# Commit B. x/{a,b,e}, y/{c,d,f}\n> +# Commit C. y/{a,b}, z/{c,d}\n> +#\n> +# Possible Resolutions:\n> +# Previous git: y/{a,b,f}, z/{c,d}, x/e\n> +# Expected: y/{a,b,e,f}, z/{c,d}\n> +# Preferred: y/{a,b,e}, z/{c,d,f}\n\nit might be tricky in the future to know what \"previous git\" is;\n\"Previous git\" means without directory renames enabled;\n\n\"expected\" means we expect the algorithm presented in this series to produce\nthis output, preferred is what we actually expect.\n\n\n> +#\n> +# Note: Both x and y got renamed and it'd be nice to detect both, and we do\n> +# better with directory rename detection than git did previously, but the\n> +# simple rule from section 5 prevents me from handling this as optimally as\n> +# we potentially could.\n\nwhich were:\n\n If a subset of to-be-renamed files have a file or directory in the way,\n \"turn off\" the directory rename for those specific sub-paths, falling\n back to old handling. But, sadly, see testcases 8a and 8b.\n\nThe tricky part is y in this example as x,y \"swapped\" its content in C,\nand moved 'old y content' to the new z/.\n\nMakes sense, but I agree it might be painful to debug such a case\nfrom a users point of view.\n\n> +\n> +# Testcase 8b, Dual-directory rename, one into the others' way, with conflicting filenames\n> +# Commit A. x/{a_1,b_1}, y/{a_2,b_2}\n> +# Commit B. x/{a_1,b_1,e_1}, y/{a_2,b_2,e_2}\n> +# Commit C. y/{a_1,b_1}, z/{a_2,b_2}\n> +#\n> +# Possible Resolutions:\n> +# Previous git: y/{a_1,b_1,e_2}, z/{a_2,b_2}, x/e_1\n> +# Scary: y/{a_1,b_1}, z/{a_2,b_2}, CONFLICT(add/add, e_1 vs. e_2)\n> +# Preferred: y/{a_1,b_1,e_1}, z/{a_2,b_2,e_2}\n\nIt may be common to have sub directories with the same path having different\nblobs, e.g. when having say multiple hardware configurations in different sub\ndirectories configured. Then renaming becomes a pain when they overlap.\n\n> +# Note: Very similar to 8a, except instead of 'e' and 'f' in directories x and\n> +# y, both are named 'e'. Without directory rename detection, neither file\n> +# moves directories. Implment directory rename detection suboptimally, and\n\nImplement\n\n> +# you get an add/add conflict, but both files were added in commit B, so this\n> +# is an add/add conflict where one side of history added both files --\n> +# something we can't represent in the index. Obviously, we'd prefer the last\n> +# resolution, but our previous rules are too coarse to allow it. Using both\n> +# the rules from section 4 and section 5 save us from the Scary resolution,\n> +# making us fall back to pre-directory-rename-detection behavior for both\n> +# e_1 and e_2.\n\nok, so add \"Expected\" as well? (repeating \"Previous git\", or so?)\n\n> +\n> +# Testcase 8c, rename+modify/delete\n> +# (Related to testcases 5b and 8d)\n> +# Commit A: z/{b,c,d}\n> +# Commit B: y/{b,c}\n> +# Commit C: z/{b,c,d_modified,e}\n> +# Expected: y/{b,c,e}, CONFLICT(rename+modify/delete: x/d -> y/d or deleted)\n> +#\n> +# Note: This testcase doesn't present any concerns for me...until you\n> +# compare it with testcases 5b and 8d. See notes in 8d for more\n> +# details.\n\nMakes sense.\n\n> +# Testcase 8d, rename/delete...or not?\n> +# (Related to testcase 5b; these may appear slightly inconsistent to users;\n> +# Also related to testcases 7d and 7e)\n\n> +# Commit A: z/{b,c,d}\n> +# Commit B: y/{b,c}\n> +# Commit C: z/{b,c,d,e}\n> +# Expected: y/{b,c,e}\n\nWhy this?\n* d is deleted in B and not found in the result\n* the rename detection also worked well in z->y for adding e\n\nI do not see the confusion, yet.\n\n> +# Note: It would also be somewhat reasonable to resolve this as\n> +# y/{b,c,e}, CONFLICT(rename/delete: x/d -> y/d or deleted)\n> +# The logic being that the only difference between this testcase and 8c\n> +# is that there is no modification to d. That suggests that instead of a\n> +# rename/modify vs. delete conflict, we should just have a rename/delete\n> +# conflict, otherwise we are being inconsistent.\n> +#\n> +# However...as far as consistency goes, we didn't report a conflict for\n> +# path d_1 in testcase 5b due to a different file being in the way. So,\n> +# we seem to be forced to have cases where users can change things\n> +# slightly and get what they may perceive as inconsistent results. It\n> +# would be nice to avoid that, but I'm not sure I see how.\n> +#\n> +# In this case, I'm leaning towards: commit B was the one that deleted z/d\n> +# and it did the rename of z to y, so the two \"conflicts\" (rename vs.\n> +# delete) are both coming from commit B, which is non-sensical. Conflicts\n> +# during merging are supposed to be about opposite sides doing things\n> +# differently.\n\n \"Sensical has not yet become an \"official\" word in the English language, which\n would be why you can't use it. Nonsense is a word, therefore nonsensical can\n used to describe something of nonsense. However, sense has different meanings\n and doesn't have an adjective for something of sense\"\n\nfrom https://english.stackexchange.com/questions/38582/antonym-of-nonsensical\nI don't mind it, the spell checker just made me go on a detour. Maybe illogical?\n\n> +# Testcase 8e, Both sides rename, one side adds to original directory\n> +# Commit A: z/{b,c}\n> +# Commit B: y/{b,c}\n> +# Commit C: w/{b,c}, z/d\n> +#\n> +# Possible Resolutions:\n> +# Previous git: z/d, CONFLICT(z/b -> y/b vs. w/b), CONFLICT(z/c -> y/c vs. w/c)\n> +# Expected: y/d, CONFLICT(z/b -> y/b vs. w/b), CONFLICT(z/c -> y/c vs. w/c)\n> +# Preferred: ??\n> +#\n> +# Notes: In commit B, directory z got renamed to y. In commit C, directory z\n> +# did NOT get renamed; the directory is still present; instead it is\n> +# considered to have just renamed a subset of paths in directory z\n> +# elsewhere. Therefore, the directory rename done in commit B to z/\n> +# applies to z/d and maps it to y/d.\n> +#\n> +# It's possible that users would get confused about this, but what\n> +# should we do instead? Silently leaving at z/d seems just as bad or\n> +# maybe even worse. Perhaps we could print a big warning about z/d\n> +# and how we're moving to y/d in this case, but when I started thinking\n> +# abouty the ramifications of doing that, I didn't know how to rule out\n> +# that opening other weird edge and corner cases so I just punted.\n\ns/about/abouty\n\nIt sort of makes sense from a users POV.\n"},{"id":"332574","messageId":"CABPp-BE7qKiYBNxMquXRv+Gr=sA3jGs8JyKuqNSzAwKbOWOQ=Q@mail.gmail.com","threadId":"47162","inReplyTo":"CAGZ79kaDWxx2g2OMY-VKJZD0-0t7V33644aqPLHq5Hppo_Be+w@mail.gmail.com","subject":"Re: [PATCH 10/30] directory rename detection: more involved edge/corner testcases","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-14T21:11:44Z","receivedAt":"2017-11-14T21:11:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Nov 13, 2017 at 4:42 PM, Stefan Beller wrote:\n> On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n\n> \"In my opinion\" ... sounds like commit message?\n\nSure, I can move it there.\n\n\n>> +# Testcase 7a, rename-dir vs. rename-dir (NOT split evenly) PLUS add-other-file\n>> +# Commit A: z/{b,c}\n>> +# Commit B: y/{b,c}\n>> +# Commit C: w/b, x/c, z/d\n>> +# Expected: y/d, CONFLICT(rename/rename for both z/b and z/c)\n>> +# NOTE: There's a rename of z/ here, y/ has more renames, so z/d -> y/d.\n>\n> But the creator of C intended to have z/d, not {w,x}/d, and as {w,x} == y,\n> I am not sure I like this result. (I have no concrete counter example, just\n> messy logic)\n\nI'm open to alternative interpretations here. The biggest issue for\nme -- going back our discussion at the end of\nhttps://public-inbox.org/git/CABPp-BFKiam6AK-Gg_RzaLuLur-jz0kvv3TqsHNHg5+HTv_uzA@mail.gmail.com/\n-- is \"simple, predictable rule\", which is consistent with the other\nrules and limits the number of nasty corner cases as much as possible.\nPerhaps you think this is one of those nasty corner cases, and that's\nfair, but I think it'd be hard to do much better.\n\nAfter spending quite a while trying to think of any other alternative\nrules or ways of looking at this, I could only come up with two\npoints:\n\n 1) One could view this as a case where commit C didn't in fact do\nany directory rename -- note that directory z/ still exists in that\ncommit. Thus, only B did a rename, it renamed z/ -> y/, thus C's z/d\nshould be moved to y/d. So, this choice is consistent with the other\nrules we've got.\n\n 2) An alternate (or maybe additional?) rule: We could decide that if\na source path is renamed on both sides of history, then we'll just\nignore both renames for consideration of directory rename detection.\n\nThe new rule idea would \"fix\" this testcase to your liking, although\nnow we'd be somewhat inconsistent with the \"directory still exists\nimplies no directory rename occurred rule\". But what other weirdness\ncould entail? Here's a few I've thought of:\n\nCommit O: z/{b,c,d}\nCommit A: y/{b,c}\nCommit B: z/{newb, newc, e}\n\nHere, A renamed z/ -> y/. Except B renamed z/b and z/c differently,\nso all paths used to detect the z/ -> y/ rename are ignored, so there\nisn't a rename after all. I'm not so sure I like that decision.\nLet's keep looking though, and change it up a bit more:\n\nCommit O: z/{b,c,d}\nCommit A: y/{b,c}, x/d\nCommit B: z/{newb, newc, d, e}\n\nHere, A has a split rename. Since B renames z/b and z/c differently,\nwe have to ignore the z/ -> y/ rename, and thus the only rename left\nimplies z/ -> x/. Thus we'd end up with z/e getting moved into x/e.\nSeems weird to me, and less likely that a user would understand this\nrule than the \"majority wins\" one.\n\n\n>> +# Testcase 7c, rename/rename(1to...2or3); transitive rename may add complexity\n>> +# (Related to testcases 3b and 5c)\n>> +# Commit A: z/{b,c}, x/d\n>> +# Commit B: y/{b,c}, w/d\n>> +# Commit C: z/{b,c,d}\n>> +# Expected: y/{b,c}, CONFLICT(x/d -> w/d vs. y/d)\n>\n> CONFLICT(x/d -> y/d vs w/d) ?\n\nI'm afraid I'm not following the question.\n\n>\n>> +# NOTE: z/ was renamed to y/ so we do not want to report\n>> +# either CONFLICT(x/d -> w/d vs. z/d)\n>> +# or CONFLiCT x/d -> w/d vs. y/d vs. z/d)\n>\n> \"neither ... nor\" instead of \"not either or\"?\n\nYes, thanks.\n\n>> +# Testcase 7e, transitive rename in rename/delete AND dirs in the way\n>> +# (Very similar to 'both rename source and destination involved in D/F conflict' from t6022-merge-rename.sh)\n>> +# (Also related to testcases 9c and 9d)\n>> +# Commit A: z/{b,c}, x/d_1\n>> +# Commit B: y/{b,c,d/g}, x/d/f\n>> +# Commit C: z/{b,c,d_1}\n>> +# Expected: rename/delete(x/d_1->y/d_1 vs. None) + D/F conflict on y/d\n>> +# y/{b,c,d/g}, y/d_1~C^0, x/d/f\n>> +# NOTE: x/d/f may be slightly confusing here. x/d_1 -> z/d_1 implies\n>> +# there is a directory rename from x/ -> z/, performed by commit C.\n>> +# However, on the side of commit B, it renamed z/ -> y/, thus\n>> +# making a rename from x/ -> z/ when it was getting rid of z/ seems\n>> +# non-sensical. Further, putting x/d/f into y/d/f also doesn't\n>> +# make a lot of sense because commit B did the renaming of z to y\n>> +# and it created x/d/f, and it clearly made these things separate,\n>> +# so it doesn't make much sense to push these together.\n>\n> This is confusing.\n\nIndeed it is. When I first wrote this testcase, I didn't realize that\nI actually had two potentially directory renames involved and a\ndoubly-transitive rename from it, on top of the D/F conflict. I can\nsee two ways to resolve this.\n\n1) Leave the testcase alone, just try to make the NOTE more clear:\n\nNOTE: The main path of interest here is d_1 and where it ends up, but\nthis is actually a case that has two potential directory renames\ninvolved and D/F conflict(s), so it makes sense to walk through each\nstep. Commit B renames z/ -> y/. Thus everything that C adds to z/\nshould be instead moved to y/. This gives us the D/F conflict on y/d\nbecause x/d_1 -> z/d_1 -> y/d_1 conflicts with y/d/g. Further, commit\nC renames x/ -> z/, thus everything B adds to x/ should instead be\nmoved to z/...BUT we removed z/ and renamed it to y/, so maybe\neverything should move not from x/ to z/, but from x/ to z/ to y/.\nDoing so might make sense from the logic so far, but note that commit\nB had both an x/ and a y/; it did the renaming of z/ to y/ and created\nx/d/f and it clearly made these things separate, so it doesn't make\nmuch sense to push these together. Doing so is what I'd call a doubly\ntransitive rename; see testcases 9c and 9d for further discussion of\nthis issue and how it's resolved.\n\n2) Modify the testcase so it doesn't have two potential directory\nrenames involved. Just add another unrelated file under x/ that\ndoesn't change on either side, thus removing the x/ -> z/ rename from\nthe mix. That wouldn't actually change the expected result (other\nthan the new file should remain around), but it would change the\nreasoning and simplify it:\n\nNOTE: Commit B renames z/ -> y/. Thus everything that C adds to z/\nshould be instead moved to y/. This gives us the D/F conflict on y/d\nbecause x/d_1 -> z/d_1 -> y/d_1 conflicts with y/d/g. As a side note,\none could imagine an alternative implementation trying to resolve D/F\nconflicts caused by renames by just undoing the rename, but in this\ncase that would end up with us needing to write an x/d_1, which would\nstill be a D/F conflict with x/d/f.\n"},{"id":"332576","messageId":"CABPp-BHhf37R1LuHceXvuf=9nW4-EDaNLHkOBbgpsQhMPP2O_g@mail.gmail.com","threadId":"47162","inReplyTo":"CAGZ79kYCDBFwMhW4Qgwhomiv4kKRWS53cjMLuRG1NxrzULYwgg@mail.gmail.com","subject":"Re: [PATCH 11/30] directory rename detection: testcases exploring possibly suboptimal merges","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-14T21:42:46Z","receivedAt":"2017-11-14T21:43:07Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Nov 14, 2017 at 12:33 PM, Stefan Beller wrote:\n> On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n\n>> +# Possible Resolutions:\n>> +# Previous git: y/{a,b,f}, z/{c,d}, x/e\n>> +# Expected: y/{a,b,e,f}, z/{c,d}\n>> +# Preferred: y/{a,b,e}, z/{c,d,f}\n>\n> it might be tricky in the future to know what \"previous git\" is;\n> \"Previous git\" means without directory renames enabled;\n>\n> \"expected\" means we expect the algorithm presented in this series to produce\n> this output, preferred is what we actually expect.\n\nYes, how about using:\n \"Without dir rename detection:\"\n \"Currently expected:\"\nand\n \"Optimal:\"\n?\n\n>> +# Testcase 8b, Dual-directory rename, one into the others' way, with conflicting filenames\n>> +# Commit A. x/{a_1,b_1}, y/{a_2,b_2}\n>> +# Commit B. x/{a_1,b_1,e_1}, y/{a_2,b_2,e_2}\n>> +# Commit C. y/{a_1,b_1}, z/{a_2,b_2}\n>> +#\n>> +# Possible Resolutions:\n>> +# Previous git: y/{a_1,b_1,e_2}, z/{a_2,b_2}, x/e_1\n>> +# Scary: y/{a_1,b_1}, z/{a_2,b_2}, CONFLICT(add/add, e_1 vs. e_2)\n>> +# Preferred: y/{a_1,b_1,e_1}, z/{a_2,b_2,e_2}\n>\n> It may be common to have sub directories with the same path having different\n> blobs, e.g. when having say multiple hardware configurations in different sub\n> directories configured. Then renaming becomes a pain when they overlap.\n\nSure, agreed. Although, the one nice thing about this particular\ntestcase is that despite showing suboptimal merge behavior, it's at\nleast the exact same suboptimal behavior as before when we didn't have\ndirectory rename detection.\n\n>> +# moves directories. Implment directory rename detection suboptimally, and\n>\n> Implement\n\nThanks.\n\n> ok, so add \"Expected\" as well? (repeating \"Previous git\", or so?)\n\nYeah, I should make that more explicit.\n\n>> +# Testcase 8d, rename/delete...or not?\n>> +# (Related to testcase 5b; these may appear slightly inconsistent to users;\n>> +# Also related to testcases 7d and 7e)\n>\n>> +# Commit A: z/{b,c,d}\n>> +# Commit B: y/{b,c}\n>> +# Commit C: z/{b,c,d,e}\n>> +# Expected: y/{b,c,e}\n>\n> Why this?\n> * d is deleted in B and not found in the result\n> * the rename detection also worked well in z->y for adding e\n>\n> I do not see the confusion, yet.\n\nUm...yaay? If you don't see it as confusing, then maybe others don't?\n I was wondering if folks would expect a rename/delete conflict (x/d\neither deleted or renamed to y/d via directory rename detection), and\nbe annoyed if the merge succeeded and didn't even give so much as a\nwarning about what happened to 'd'.\n\n>> +# In this case, I'm leaning towards: commit B was the one that deleted z/d\n>> +# and it did the rename of z to y, so the two \"conflicts\" (rename vs.\n>> +# delete) are both coming from commit B, which is non-sensical. Conflicts\n>> +# during merging are supposed to be about opposite sides doing things\n>> +# differently.\n>\n> \"Sensical has not yet become an \"official\" word in the English language, which\n> would be why you can't use it. Nonsense is a word, therefore nonsensical can\n> used to describe something of nonsense. However, sense has different meanings\n> and doesn't have an adjective for something of sense\"\n>\n> from https://english.stackexchange.com/questions/38582/antonym-of-nonsensical\n> I don't mind it, the spell checker just made me go on a detour. Maybe illogical?\n\nIllogical works for me.\n\n>> +# Testcase 8e, Both sides rename, one side adds to original directory\n>> +# Commit A: z/{b,c}\n>> +# Commit B: y/{b,c}\n>> +# Commit C: w/{b,c}, z/d\n>> +#\n>> +# Possible Resolutions:\n>> +# Previous git: z/d, CONFLICT(z/b -> y/b vs. w/b), CONFLICT(z/c -> y/c vs. w/c)\n>> +# Expected: y/d, CONFLICT(z/b -> y/b vs. w/b), CONFLICT(z/c -> y/c vs. w/c)\n>> +# Preferred: ??\n>> +#\n>> +# Notes: In commit B, directory z got renamed to y. In commit C, directory z\n>> +# did NOT get renamed; the directory is still present; instead it is\n>> +# considered to have just renamed a subset of paths in directory z\n>> +# elsewhere. Therefore, the directory rename done in commit B to z/\n>> +# applies to z/d and maps it to y/d.\n>> +#\n>> +# It's possible that users would get confused about this, but what\n>> +# should we do instead? Silently leaving at z/d seems just as bad or\n>> +# maybe even worse. Perhaps we could print a big warning about z/d\n>> +# and how we're moving to y/d in this case, but when I started thinking\n>> +# abouty the ramifications of doing that, I didn't know how to rule out\n>> +# that opening other weird edge and corner cases so I just punted.\n>\n> s/about/abouty\n\nI think you mean the other direction? Thanks for catching, I'll fix that up.\n\n> It sort of makes sense from a users POV.\n\nI'm afraid I'm unsure what the antecedent of \"It\" is here. (Are you\njust saying that my rationale for what I listed as \"Expected\" makes\nsense, or something else?)\n"},{"id":"332577","messageId":"CAGZ79kY+DcQkSBjz-=_pUzkpLk+yE2b+Nqi07a-WC4fwzsSCyA@mail.gmail.com","threadId":"47162","inReplyTo":"CABPp-BE7qKiYBNxMquXRv+Gr=sA3jGs8JyKuqNSzAwKbOWOQ=Q@mail.gmail.com","subject":"Re: [PATCH 10/30] directory rename detection: more involved edge/corner testcases","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-14T22:47:55Z","receivedAt":"2017-11-14T22:48:17Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Nov 14, 2017 at 1:11 PM, Elijah Newren wrote:\n> On Mon, Nov 13, 2017 at 4:42 PM, Stefan Beller wrote:\n>> On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n>\n>> \"In my opinion\" ... sounds like commit message?\n>\n> Sure, I can move it there.\n>\n>\n>>> +# Testcase 7a, rename-dir vs. rename-dir (NOT split evenly) PLUS add-other-file\n>>> +# Commit A: z/{b,c}\n>>> +# Commit B: y/{b,c}\n>>> +# Commit C: w/b, x/c, z/d\n>>> +# Expected: y/d, CONFLICT(rename/rename for both z/b and z/c)\n>>> +# NOTE: There's a rename of z/ here, y/ has more renames, so z/d -> y/d.\n>>\n>> But the creator of C intended to have z/d, not {w,x}/d, and as {w,x} == y,\n>> I am not sure I like this result. (I have no concrete counter example, just\n>> messy logic)\n>\n> I'm open to alternative interpretations here. The biggest issue for\n> me -- going back our discussion at the end of\n> https://public-inbox.org/git/CABPp-BFKiam6AK-Gg_RzaLuLur-jz0kvv3TqsHNHg5+HTv_uzA@mail.gmail.com/\n> -- is \"simple, predictable rule\", which is consistent with the other\n> rules and limits the number of nasty corner cases as much as possible.\n> Perhaps you think this is one of those nasty corner cases, and that's\n> fair, but I think it'd be hard to do much better.\n\nyup, I agree that a simple, predictable rule is better than optimizing\nfor the corner cases we can come up now. (I sent this email before\nreading your reply, so thanks for re-iterating the answer.)\n\n> After spending quite a while trying to think of any other alternative\n> rules or ways of looking at this, I could only come up with two\n> points:\n>\n> 1) One could view this as a case where commit C didn't in fact do\n> any directory rename -- note that directory z/ still exists in that\n> commit. Thus, only B did a rename, it renamed z/ -> y/, thus C's z/d\n> should be moved to y/d. So, this choice is consistent with the other\n> rules we've got.\n\nI wonder if we can do a data driven approach, i.e. mine some history\n(linux, git, other large projects), that would tell us which of these cases\nhappens very often, and which of these corner cases can be \"safely\nignored because it never happens\". My gut feeling tells me that\nsplitting up a directory into two or three (potential sub-)directories\nis a common thing, whereas double renames are not as often.\n(But that's just my view, I have no data to back it up; the selection\nof the data would open the next debate as it will be very specific to\na given community. But as linux has had such a huge impact on git,\nI'd be tempted to claim any study on linux.git is fruitful for gits defaults)\n\n> 2) An alternate (or maybe additional?) rule: We could decide that if\n> a source path is renamed on both sides of history, then we'll just\n> ignore both renames for consideration of directory rename detection.\n>\n> The new rule idea would \"fix\" this testcase to your liking, although\n> now we'd be somewhat inconsistent with the \"directory still exists\n> implies no directory rename occurred rule\". But what other weirdness\n> could entail? Here's a few I've thought of:\n>\n> Commit O: z/{b,c,d}\n> Commit A: y/{b,c}\n> Commit B: z/{newb, newc, e}\n>\n> Here, A renamed z/ -> y/.\n\n.. while deleting d ...\n\n\n> Except B renamed z/b and z/c differently,\n\n... and z/d -> z/e, maybe(?).\n\n> so all paths used to detect the z/ -> y/ rename are ignored, so there\n> isn't a rename after all. I'm not so sure I like that decision.\n> Let's keep looking though, and change it up a bit more:\n>\n> Commit O: z/{b,c,d}\n> Commit A: y/{b,c}, x/d\n> Commit B: z/{newb, newc, d, e}\n>\n> Here, A has a split rename. Since B renames z/b and z/c differently,\n> we have to ignore the z/ -> y/ rename, and thus the only rename left\n> implies z/ -> x/. Thus we'd end up with z/e getting moved into x/e.\n> Seems weird to me, and less likely that a user would understand this\n> rule than the \"majority wins\" one.\n\nIt still is \"majority wins\" except the set of inspected files is filtered first.\n\nIn an *ideal, but expensive* algorithm, we might give different\nweights to files, e.g.\n* large files have more weight than smaller files,\n* files with interesting names have more weight (c.f. Makefile vs. xstrbuf.c)\n* similar files have more weight than files that are rewritten, or rather\n the more rewrite is done the less impact one file has.\n* how unique file content is (LICENSE.txt that exists 23 times in the\n tree has less weight than the sekret-algorithm.c)\n\nand depending on these weights we have a \"majority of content\" moved\nto y/ or x/.\n\n>>> +# Testcase 7c, rename/rename(1to...2or3); transitive rename may add complexity\n>>> +# (Related to testcases 3b and 5c)\n>>> +# Commit A: z/{b,c}, x/d\n>>> +# Commit B: y/{b,c}, w/d\n>>> +# Commit C: z/{b,c,d}\n>>> +# Expected: y/{b,c}, CONFLICT(x/d -> w/d vs. y/d)\n>>\n>> CONFLICT(x/d -> y/d vs w/d) ?\n>\n> I'm afraid I'm not following the question.\n\nYesterday I had the impression the renaming perspective changed,\nnote the difference in order of y/ and w/ inside the CONFLICT.\nI might have been confused already, though.\n\n>>> +# Testcase 7e, transitive rename in rename/delete AND dirs in the way\n>>> +# (Very similar to 'both rename source and destination involved in D/F conflict' from t6022-merge-rename.sh)\n>>> +# (Also related to testcases 9c and 9d)\n>>> +# Commit A: z/{b,c}, x/d_1\n>>> +# Commit B: y/{b,c,d/g}, x/d/f\n>>> +# Commit C: z/{b,c,d_1}\n>>> +# Expected: rename/delete(x/d_1->y/d_1 vs. None) + D/F conflict on y/d\n>>> +# y/{b,c,d/g}, y/d_1~C^0, x/d/f\n>>> +# NOTE: x/d/f may be slightly confusing here. x/d_1 -> z/d_1 implies\n>>> +# there is a directory rename from x/ -> z/, performed by commit C.\n>>> +# However, on the side of commit B, it renamed z/ -> y/, thus\n>>> +# making a rename from x/ -> z/ when it was getting rid of z/ seems\n>>> +# non-sensical. Further, putting x/d/f into y/d/f also doesn't\n>>> +# make a lot of sense because commit B did the renaming of z to y\n>>> +# and it created x/d/f, and it clearly made these things separate,\n>>> +# so it doesn't make much sense to push these together.\n>>\n>> This is confusing.\n>\n> Indeed it is. When I first wrote this testcase, I didn't realize that\n> I actually had two potentially directory renames involved and a\n> doubly-transitive rename from it, on top of the D/F conflict. I can\n> see two ways to resolve this.\n>\n> 1) Leave the testcase alone, just try to make the NOTE more clear:\n>\n> NOTE: The main path of interest here is d_1 and where it ends up, but\n> this is actually a case that has two potential directory renames\n> involved and D/F conflict(s), so it makes sense to walk through each\n> step. Commit B renames z/ -> y/. Thus everything that C adds to z/\n> should be instead moved to y/. This gives us the D/F conflict on y/d\n> because x/d_1 -> z/d_1 -> y/d_1 conflicts with y/d/g. Further, commit\n> C renames x/ -> z/, thus everything B adds to x/ should instead be\n> moved to z/...BUT we removed z/ and renamed it to y/, so maybe\n> everything should move not from x/ to z/, but from x/ to z/ to y/.\n> Doing so might make sense from the logic so far, but note that commit\n> B had both an x/ and a y/; it did the renaming of z/ to y/ and created\n> x/d/f and it clearly made these things separate, so it doesn't make\n> much sense to push these together. Doing so is what I'd call a doubly\n> transitive rename; see testcases 9c and 9d for further discussion of\n> this issue and how it's resolved.\n>\n> 2) Modify the testcase so it doesn't have two potential directory\n> renames involved. Just add another unrelated file under x/ that\n> doesn't change on either side, thus removing the x/ -> z/ rename from\n> the mix. That wouldn't actually change the expected result (other\n> than the new file should remain around), but it would change the\n> reasoning and simplify it:\n>\n> NOTE: Commit B renames z/ -> y/. Thus everything that C adds to z/\n> should be instead moved to y/. This gives us the D/F conflict on y/d\n> because x/d_1 -> z/d_1 -> y/d_1 conflicts with y/d/g. As a side note,\n> one could imagine an alternative implementation trying to resolve D/F\n> conflicts caused by renames by just undoing the rename, but in this\n> case that would end up with us needing to write an x/d_1, which would\n> still be a D/F conflict with x/d/f.\n\nWhat do you want to test in 7e? AFAICT section 7 is about\n\"More involved Edge/Corner cases\", so keeping it edge sounds fine.\n(hence I'd vote for (1), just adjusting the notes)\n"},{"id":"332589","messageId":"xmqqshdg1goa.fsf@gitster.mtv.corp.google.com","threadId":"47162","inReplyTo":"CABPp-BE8Jc-ZmW2bt3Fe3Azd6wEiSTUjsmhNNNdoip_LXHC_Yg@mail.gmail.com","subject":"Re: [PATCH 15/30] merge-recursive: Move the get_renames() function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-15T01:20:21Z","receivedAt":"2017-11-15T01:20:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> Eek! My apologies. I will go through and fix them up. I see no\n> reference to checkpatch.pl in git, but a google search shows there's\n> one in the linux source tree. Is that were I get it from, or is there\n> a different one?\n\n\n\n> Also, would you like me to make a separate commit that cleans up\n> pre-existing issues in merge-recursive.c so that it runs clean, or\n> just remove the problems I added?\n\nThat's optional. These three patch series are already sufficiently\nlarge, so I do not mind a clean-up after dust settles down, instead\nof preliminary clean-up.\n\n"},{"id":"332616","messageId":"CAGZ79kb13gGJ6V9d08evHKLvTdTQwcp8VAyzi36BnGn-m5pTXQ@mail.gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-13-newren@gmail.com","subject":"Re: [PATCH 12/30] directory rename detection: miscellaneous testcases to complete coverage","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-15T20:03:53Z","receivedAt":"2017-11-15T20:04:00Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n\n> +###########################################################################\n> +# SECTION 9: Other testcases\n> +#\n> +# I came up with the testcases in the first eight sections before coding up\n> +# the implementation. The testcases in this section were mostly ones I\n> +# thought of while coding/debugging, and which I was too lazy to insert\n> +# into the previous sections because I didn't want to re-label with all the\n> +# testcase references. :-)\n\nThis might also be commit message material, as it describes the workflow,\nnot the 'misc' aspect of these test cases.\n\n> +###########################################################################\n> +\n> +# Testcase 9a, Inner renamed directory within outer renamed directory\n> +# (Related to testcase 1f)\n> +# Commit A: z/{b,c,d/{e,f,g}}\n> +# Commit B: y/{b,c}, x/w/{e,f,g}\n> +# Commit C: z/{b,c,d/{e,f,g,h},i}\n> +# Expected: y/{b,c,i}, x/w/{e,f,g,h}\n> +# NOTE: The only reason this one is interesting is because when a directory\n> +# is split into multiple other directories, we determine by the weight\n> +# of which one had the most paths going to it. A naive implementation\n> +# of that could take the new file in commit C at z/i to x/w/i or x/i.\n\nMakes sense.\n\n> +# Testcase 9b, Transitive rename with content merge\n> +# (Related to testcase 1c)\n> +# Commit A: z/{b,c}, x/d_1\n> +# Commit B: y/{b,c}, x/d_2\n> +# Commit C: z/{b,c,d_3}\n> +# Expected: y/{b,c,d_merged}\n\nMakes sense.\n\n> +# Testcase 9c, Doubly transitive rename?\n> +# (Related to testcase 1c, 7e, and 9d)\n> +# Commit A: z/{b,c}, x/{d,e}, w/f\n> +# Commit B: y/{b,c}, x/{d,e,f,g}\n> +# Commit C: z/{b,c,d,e}, w/f\n> +# Expected: y/{b,c,d,e}, x/{f,g}\n> +#\n> +# NOTE: x/f and x/g may be slightly confusing here. The rename from w/f to\n> +# x/f is clear. Let's look beyond that. Here's the logic:\n> +# Commit C renamed x/ -> z/\n> +# Commit B renamed z/ -> y/\n> +# So, we could possibly further rename x/f to z/f to y/f, a doubly\n> +# transient rename. However, where does it end? We can chain these\n> +# indefinitely (see testcase 9d). What if there is a D/F conflict\n> +# at z/f/ or y/f/? Or just another file conflict at one of those\n> +# paths? In the case of an N-long chain of transient renamings,\n> +# where do we \"abort\" the rename at? Can the user make sense of\n> +# the resulting conflict and resolve it?\n> +#\n> +# To avoid this confusion I use the simple rule that if the other side\n> +# of history did a directory rename to a path that your side renamed\n> +# away, then ignore that particular rename from the other side of\n> +# history for any implicit directory renames.\n\nThis is repeated in the rule of section 9 below.\nMakes sense.\n\n> +# Testcase 9d, N-fold transitive rename?\n> +# (Related to testcase 9c...and 1c and 7e)\n> +# Commit A: z/a, y/b, x/c, w/d, v/e, u/f\n> +# Commit B: y/{a,b}, w/{c,d}, u/{e,f}\n> +# Commit C: z/{a,t}, x/{b,c}, v/{d,e}, u/f\n> +# Expected: \n> +#\n> +# NOTE: z/ -> y/ (in commit B)\n> +# y/ -> x/ (in commit C)\n> +# x/ -> w/ (in commit B)\n> +# w/ -> v/ (in commit C)\n> +# v/ -> u/ (in commit B)\n> +# So, if we add a file to z, say z/t, where should it end up? In u?\n> +# What if there's another file or directory named 't' in one of the\n> +# intervening directories and/or in u itself? Also, shouldn't the\n> +# same logic that places 't' in u/ also move ALL other files to u/?\n> +# What if there are file or directory conflicts in any of them? If\n> +# we attempted to do N-way (N-fold? N-ary? N-uple?) transitive renames\n> +# like this, would the user have any hope of understanding any\n> +# conflicts or how their working tree ended up? I think not, so I'm\n> +# ruling out N-ary transitive renames for N>1.\n> +#\n> +# Therefore our expected result is:\n> +# z/t, y/a, x/b, w/c, u/d, u/e, u/f\n> +# The reason that v/d DOES get transitively renamed to u/d is that u/ isn't\n> +# renamed somewhere. A slightly sub-optimal result, but it uses fairly\n> +# simple rules that are consistent with what we need for all the other\n> +# testcases and simplifies things for the user.\n\nDoes the merge order matter here?\nIf B and C were swapped, applying the same logic presented in the NOTE,\none could argue that we expect:\n\n z/t y/a x/b w/c v/d v/e u/f\n\nI can make a strong point for y/a here, but the v/{d,e} also seem to deviate.\n\n> +# Testcase 9e, N-to-1 whammo\n> +# (Related to testcase 9c...and 1c and 7e)\n> +# Commit A: dir1/{a,b}, dir2/{d,e}, dir3/{g,h}, dirN/{j,k}\n> +# Commit B: dir1/{a,b,c,yo}, dir2/{d,e,f,yo}, dir3/{g,h,i,yo}, dirN/{j,k,l,yo}\n> +# Commit C: combined/{a,b,d,e,g,h,j,k}\n> +# Expected: combined/{a,b,c,d,e,f,g,h,i,j,k,l}, CONFLICT(Nto1) warnings,\n> +# dir1/yo, dir2/yo, dir3/yo, dirN/yo\n\nVery neat!\n\n> +# Testcase 9f, Renamed directory that only contained immediate subdirs\n> +# (Related to testcases 1e & 9g)\n> +# Commit A: goal/{a,b}/$more_files\n> +# Commit B: priority/{a,b}/$more_files\n> +# Commit C: goal/{a,b}/$more_files, goal/c\n> +# Expected: priority/{a,b}/$more_files, priority/c\n\n> +# Testcase 9g, Renamed directory that only contained immediate subdirs, immediate subdirs renamed\n> +# (Related to testcases 1e & 9f)\n> +# Commit A: goal/{a,b}/$more_files\n> +# Commit B: priority/{alpha,bravo}/$more_files\n> +# Commit C: goal/{a,b}/$more_files, goal/c\n> +# Expected: priority/{alpha,bravo}/$more_files, priority/c\n\nand if C also added goal/a/another_file, we'd expect it to\nbecome priority/alpha/another_file.\n\nWhat happens in moving dir hierarchies?\n\nA: root/node1/{leaf1, leaf2}, root/node2/{leaf3, leaf4}\nB: \"Move node2 one layer down into node1\"\n root/node1/{leaf1, leaf2, node2/{leaf3, leaf4}}\nC: \"Add more leaves\"\n root/node1/{leaf1, leaf2, leaf5}, root/node2/{leaf3, leaf4, leaf6}\n\nOr chaining putting things in one another:\n(Same A)\nB: \"Move node2 one layer down into node1\"\n root/node1/{leaf1, leaf2, node2/{leaf3, leaf4}}\nC: \"Move node1 one layer down into node2\"\n root/node2/{leaf3, leaf4, node1/{leaf1, leaf2}}\n\nJust food for thought.\n\n> +# Rules suggested by section 9:\n> +#\n> +# If the other side of history did a directory rename to a path that your\n> +# side renamed away, then ignore that particular rename from the other\n> +# side of history for any implicit directory renames.\n"},{"id":"332617","messageId":"CAGZ79kanxn7+bgMxbEfSPZtSr4KBWzk9NHE9F-g4J+2TXbz_mQ@mail.gmail.com","threadId":"47162","inReplyTo":"20171110190550.27059-28-newren@gmail.com","subject":"Re: [PATCH 27/30] merge-recursive: Apply necessary modifications for directory renames","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-11-15T20:23:50Z","receivedAt":"2017-11-15T20:25:45Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> + if (!strcmp(pair->one->path, pair->two->path)) {\n> + /*\n> + * Paths should only match if this was initially a\n> + * non-rename that is being turned into one by\n> + * directory rename detection.\n> + */\n> + assert(pair->status != 'R');\n> + } else {\n> + assert(pair->status == 'R');\n\nassert() is compiled conditionally depending on whether\nNDEBUG is set, which makes potential error reports more interesting and\nhead-scratching. But we'd rather prefer easy bug reports, therefore\nwe'd want to (a) either have the condition checked always, when\nyou know this could occur, e.g. via\n\n if ()\n BUG(\"Git is broken, because...\");\n\nor (b) you could omit the asserts if they are more of a developer guideline?\n\nI wonder if we want to introduce a BUG_ON(, ) macro\nthat contains (a).\n\n\n> + re->dst_entry->processed = 1;\n> + //string_list_remove(entries, pair->two->path, 0);\n\ncommented code?\n"},{"id":"332648","messageId":"CABPp-BHfRtfOhD5bt-DCBgLW8NjzKsohaf4C5ZatWwPS-adYfQ@mail.gmail.com","threadId":"47162","inReplyTo":"CAGZ79kanxn7+bgMxbEfSPZtSr4KBWzk9NHE9F-g4J+2TXbz_mQ@mail.gmail.com","subject":"Re: [PATCH 27/30] merge-recursive: Apply necessary modifications for directory renames","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-16T03:54:58Z","receivedAt":"2017-11-16T03:55:04Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Nov 15, 2017 at 12:23 PM, Stefan Beller wrote:\n>> + if (!strcmp(pair->one->path, pair->two->path)) {\n>> + /*\n>> + * Paths should only match if this was initially a\n>> + * non-rename that is being turned into one by\n>> + * directory rename detection.\n>> + */\n>> + assert(pair->status != 'R');\n>> + } else {\n>> + assert(pair->status == 'R');\n>\n> assert() is compiled conditionally depending on whether\n> NDEBUG is set, which makes potential error reports more interesting and\n> head-scratching. But we'd rather prefer easy bug reports, therefore\n> we'd want to (a) either have the condition checked always, when\n> you know this could occur, e.g. via\n>\n> if ()\n> BUG(\"Git is broken, because...\");\n>\n> or (b) you could omit the asserts if they are more of a developer guideline?\n>\n> I wonder if we want to introduce a BUG_ON(, ) macro\n> that contains (a).\n\nYeah, I added a few other asserts in other commits too. None of these\nwere written with the expectation that they should or could ever occur\nfor a user; it was just a developer guideline to make sure I (and\nfuture others) didn't break certain invariants during the\nimplementation or while making modifications to it.\n\nSo that makes it more like (b), but I feel that there is something to\nbe said for having a convenient syntax for expressing pre-conditions\nthat others shouldn't violate when changing the code, and which will\nbe given more weight than a comment. For that, something that is\ncompiled out on many users systems seemed just fine.\n\nBut, I have certainly seen abuses of asserts in my time as well (e.g.\nfunction calls with important side-effects being placed inside\nasserts), so if folks have decided it's against git's style, then I\nunderstand. I'll remove some, and switch the cheaper checks over to\nBUG().\n\n>> + re->dst_entry->processed = 1;\n>> + //string_list_remove(entries, pair->two->path, 0);\n>\n> commented code?\n\nUgh, that's embarrassing. I'll clean that out.\n"},{"id":"332724","messageId":"CABPp-BF5=vKUhGGrzkV=FZ_4Syuer+WEZT36kvEHbaxtHVdTsQ@mail.gmail.com","threadId":"47162","inReplyTo":"CAGZ79kb13gGJ6V9d08evHKLvTdTQwcp8VAyzi36BnGn-m5pTXQ@mail.gmail.com","subject":"Re: [PATCH 12/30] directory rename detection: miscellaneous testcases to complete coverage","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2017-11-16T21:17:52Z","receivedAt":"2017-11-16T21:18:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Nov 15, 2017 at 12:03 PM, Stefan Beller wrote:\n> On Fri, Nov 10, 2017 at 11:05 AM, Elijah Newren wrote:\n>\n>> +# Testcase 9d, N-fold transitive rename?\n>> +# (Related to testcase 9c...and 1c and 7e)\n>> +# Commit A: z/a, y/b, x/c, w/d, v/e, u/f\n>> +# Commit B: y/{a,b}, w/{c,d}, u/{e,f}\n>> +# Commit C: z/{a,t}, x/{b,c}, v/{d,e}, u/f\n>> +# Expected: \n>> +#\n>> +# NOTE: z/ -> y/ (in commit B)\n>> +# y/ -> x/ (in commit C)\n>> +# x/ -> w/ (in commit B)\n>> +# w/ -> v/ (in commit C)\n>> +# v/ -> u/ (in commit B)\n>> +# So, if we add a file to z, say z/t, where should it end up? In u?\n>> +# What if there's another file or directory named 't' in one of the\n>> +# intervening directories and/or in u itself? Also, shouldn't the\n>> +# same logic that places 't' in u/ also move ALL other files to u/?\n>> +# What if there are file or directory conflicts in any of them? If\n>> +# we attempted to do N-way (N-fold? N-ary? N-uple?) transitive renames\n>> +# like this, would the user have any hope of understanding any\n>> +# conflicts or how their working tree ended up? I think not, so I'm\n>> +# ruling out N-ary transitive renames for N>1.\n>> +#\n>> +# Therefore our expected result is:\n>> +# z/t, y/a, x/b, w/c, u/d, u/e, u/f\n>> +# The reason that v/d DOES get transitively renamed to u/d is that u/ isn't\n>> +# renamed somewhere. A slightly sub-optimal result, but it uses fairly\n>> +# simple rules that are consistent with what we need for all the other\n>> +# testcases and simplifies things for the user.\n>\n> Does the merge order matter here?\n\nNo.\n\n> If B and C were swapped, applying the same logic presented in the NOTE,\n> one could argue that we expect:\n>\n> z/t y/a x/b w/c v/d v/e u/f\n>\n> I can make a strong point for y/a here, but the v/{d,e} also seem to deviate.\n\nI don't understand; I thought my argument as presented was agnostic of\ndirection. Perhaps I have an unstated assumption I'm not realizing or\nsomething; could you explain how my logic above could lead to this\nconclusion?\n\nAlso, let me try a different tack to see if it's clearer than the\nabove argument I made. Looking at each path:\n\n* z/t from commit C does not get renamed to y/t despite B's rename of\nz/ -> y/ because C renamed y/ elsewhere.\n* z/a from commit A was renamed to y/a in commit B. We do not\ntransitively rename further from y/a to x/a (despite C's rename of y/\nto x/) because B renamed x/ elsewhere.\n* y/b from commit A was renamed to x/b in commit C. We do not\ntransitively rename further from x/b to w/b (despite B's rename of x/\nto w/) because C renamed w/ elsewhere.\n* x/c from commit A was renamed to w/c in commit B. We do not\ntransitively rename further from w/c to v/c (despite C's rename from\nw/ to v/) because B renamed v/ elsewhere.\n* w/d from commit A was renamed to v/d in commit C. We DO\ntransitively rename from v/d to u/d because of B's rename of v/ to u/\nand because C did not rename u/ to somewhere else.\n\n(And, to complete the list, e and f are simple: v/e is renamed to u/e\nin commit B, and there's no directory name on u on either side, so\nthere's no special logic needed at all. u/f is even simpler; there's\nno renames or directory renames or anything affecting it.)\n\n\n>> +# Testcase 9e, N-to-1 whammo\n>> +# (Related to testcase 9c...and 1c and 7e)\n>> +# Commit A: dir1/{a,b}, dir2/{d,e}, dir3/{g,h}, dirN/{j,k}\n>> +# Commit B: dir1/{a,b,c,yo}, dir2/{d,e,f,yo}, dir3/{g,h,i,yo}, dirN/{j,k,l,yo}\n>> +# Commit C: combined/{a,b,d,e,g,h,j,k}\n>> +# Expected: combined/{a,b,c,d,e,f,g,h,i,j,k,l}, CONFLICT(Nto1) warnings,\n>> +# dir1/yo, dir2/yo, dir3/yo, dirN/yo\n>\n> Very neat!\n\n:-)\n\n>> +# Testcase 9f, Renamed directory that only contained immediate subdirs\n>> +# (Related to testcases 1e & 9g)\n>> +# Commit A: goal/{a,b}/$more_files\n>> +# Commit B: priority/{a,b}/$more_files\n>> +# Commit C: goal/{a,b}/$more_files, goal/c\n>> +# Expected: priority/{a,b}/$more_files, priority/c\n>\n>> +# Testcase 9g, Renamed directory that only contained immediate subdirs, immediate subdirs renamed\n>> +# (Related to testcases 1e & 9f)\n>> +# Commit A: goal/{a,b}/$more_files\n>> +# Commit B: priority/{alpha,bravo}/$more_files\n>> +# Commit C: goal/{a,b}/$more_files, goal/c\n>> +# Expected: priority/{alpha,bravo}/$more_files, priority/c\n>\n> and if C also added goal/a/another_file, we'd expect it to\n> become priority/alpha/another_file.\n\nYep! I thought that was covered enough by other tests, but do you\nfeel I should add that to this testcase?\n\n> What happens in moving dir hierarchies?\n>\n> A: root/node1/{leaf1, leaf2}, root/node2/{leaf3, leaf4}\n> B: \"Move node2 one layer down into node1\"\n> root/node1/{leaf1, leaf2, node2/{leaf3, leaf4}}\n> C: \"Add more leaves\"\n> root/node1/{leaf1, leaf2, leaf5}, root/node2/{leaf3, leaf4, leaf6}\n\nWorks just fine; similar to testcase 9a. Do you feel this one is\ndifferent enough to add to the testsuite? I'm happy to do so.\n\n> Or chaining putting things in one another:\n> (Same A)\n> B: \"Move node2 one layer down into node1\"\n> root/node1/{leaf1, leaf2, node2/{leaf3, leaf4}}\n> C: \"Move node1 one layer down into node2\"\n> root/node2/{leaf3, leaf4, node1/{leaf1, leaf2}}\n>\n> Just food for thought.\n\nThat's evil. I mean, it's a brilliant testcase designed to really\nmess things up. I'm not entirely sure what the right answer should\nbe, but I am confident saying my current implementation handles it\nwrong. I'm digging into why.\n"},{"id":"349854","messageId":"7d1237c7-5a83-d766-7d93-5f0d59166067@web.de","threadId":"47162","inReplyTo":"20171110190550.27059-25-newren@gmail.com","subject":"Re: [PATCH 24/30] merge-recursive: Add computation of collisions due to dir rename & merging","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-06-10T10:56:31Z","receivedAt":"2018-06-10T10:56:50Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 10.11.2017 um 20:05 schrieb Elijah Newren:\n> +static struct dir_rename_entry *check_dir_renamed(const char *path,\n> +\t\t\t\t\t\t struct hashmap *dir_renames) {\n> +\tchar temp[PATH_MAX];\n> +\tchar *end;\n> +\tstruct dir_rename_entry *entry;\n> +\n> +\tstrcpy(temp, path);\n> +\twhile ((end = strrchr(temp, '/'))) {\n> +\t\t*end = '\\0';\n> +\t\tentry = dir_rename_find_entry(dir_renames, temp);\n> +\t\tif (entry)\n> +\t\t\treturn entry;\n> +\t}\n> +\treturn NULL;\n> +}\n\nThe value of PATH_MAX is platform-dependent, so it's easy to exceed when\ndoing cross-platform development. It's also not a hard limit on most\noperating systems, not even on Windows. Further reading:\n\n https://insanecoding.blogspot.com/2007/11/pathmax-simply-isnt.html\n\nSo using a fixed buffer is not a good idea, and writing to it without\nchecking is dangerous. Here's a fix:\n\n-- >8 --\nSubject: [PATCH] merge-recursive: use xstrdup() instead of fixed buffer\n\nPaths can be longer than PATH_MAX. Avoid a buffer overrun in\ncheck_dir_renamed() by using xstrdup() to make a private copy safely.\n\nSigned-off-by: Rene Scharfe \n---\n merge-recursive.c | 10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex ac27abbd4c..db708176c5 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -2211,18 +2211,18 @@ static struct hashmap *get_directory_renames(struct diff_queue_struct *pairs,\n static struct dir_rename_entry *check_dir_renamed(const char *path,\n \t\t\t\t\t\t struct hashmap *dir_renames)\n {\n-\tchar temp[PATH_MAX];\n+\tchar *temp = xstrdup(path);\n \tchar *end;\n-\tstruct dir_rename_entry *entry;\n+\tstruct dir_rename_entry *entry = NULL;;\n \n-\tstrcpy(temp, path);\n \twhile ((end = strrchr(temp, '/'))) {\n \t\t*end = '\\0';\n \t\tentry = dir_rename_find_entry(dir_renames, temp);\n \t\tif (entry)\n-\t\t\treturn entry;\n+\t\t\tbreak;\n \t}\n-\treturn NULL;\n+\tfree(temp);\n+\treturn entry;\n }\n \n static void compute_collisions(struct hashmap *collisions,\n-- \n2.17.1\n"},{"id":"349855","messageId":"e702fb82-2dd2-5060-3f34-0f60882e05c2@web.de","threadId":"47162","inReplyTo":"7d1237c7-5a83-d766-7d93-5f0d59166067@web.de","subject":"Re: [PATCH 24/30] merge-recursive: Add computation of collisions due to dir rename & merging","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-06-10T11:03:12Z","receivedAt":"2018-06-10T11:03:25Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 10.06.2018 um 12:56 schrieb René Scharfe:\n> Am 10.11.2017 um 20:05 schrieb Elijah Newren:\n>> +static struct dir_rename_entry *check_dir_renamed(const char *path,\n>> +\t\t\t\t\t\t struct hashmap *dir_renames) {\n>> +\tchar temp[PATH_MAX];\n>> +\tchar *end;\n>> +\tstruct dir_rename_entry *entry;\n>> +\n>> +\tstrcpy(temp, path);\n>> +\twhile ((end = strrchr(temp, '/'))) {\n>> +\t\t*end = '\\0';\n>> +\t\tentry = dir_rename_find_entry(dir_renames, temp);\n>> +\t\tif (entry)\n>> +\t\t\treturn entry;\n>> +\t}\n>> +\treturn NULL;\n>> +}\n> \n> The value of PATH_MAX is platform-dependent, so it's easy to exceed when\n> doing cross-platform development. It's also not a hard limit on most\n> operating systems, not even on Windows. Further reading:\n> \n> https://insanecoding.blogspot.com/2007/11/pathmax-simply-isnt.html\n> \n> So using a fixed buffer is not a good idea, and writing to it without\n> checking is dangerous. Here's a fix:\n\nArgh, I meant to reply to v10 of that patch, i.e. this:\n\n https://public-inbox.org/git/20180419175823.7946-21-newren@gmail.com/\n\nThe cited code wasn't changed and is in current master, though, so both\nthat part and my patch are still relevant.\n\nRené\n"},{"id":"349868","messageId":"20180610204419.GA11273@sigill.intra.peff.net","threadId":"47162","inReplyTo":"7d1237c7-5a83-d766-7d93-5f0d59166067@web.de","subject":"Re: [PATCH 24/30] merge-recursive: Add computation of collisions due to dir rename & merging","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-06-10T20:44:19Z","receivedAt":"2018-06-10T20:44:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jun 10, 2018 at 12:56:31PM +0200, René Scharfe wrote:\n\n> The value of PATH_MAX is platform-dependent, so it's easy to exceed when\n> doing cross-platform development. It's also not a hard limit on most\n> operating systems, not even on Windows. Further reading:\n> \n> https://insanecoding.blogspot.com/2007/11/pathmax-simply-isnt.html\n> \n> So using a fixed buffer is not a good idea, and writing to it without\n> checking is dangerous. Here's a fix:\n\nEven on platforms where it _is_ a hard-limit, we are quite often dealing\nwith paths that come from tree objects (so even if the OS would\neventually complain about our path, it is small consolation when we\nsmash the stack before we get there).\n\nYour patch looks good to me, and we definitely should address this\nbefore v2.18-final.\n\n> -\tchar temp[PATH_MAX];\n> +\tchar *temp = xstrdup(path);\n> \tchar *end;\n> -\tstruct dir_rename_entry *entry;\n> +\tstruct dir_rename_entry *entry = NULL;;\n> \n> -\tstrcpy(temp, path);\n\nI'm sad that this strcpy() wasn't caught in review. IMHO we should avoid\nthat function altogether, even when we _think_ it can't trigger an\noverflow. That's easier to reason about (and makes auditing easier).\n\nIt looks like another one has crept in recently, too.\n\n-- >8 --\nSubject: [PATCH] blame: prefer xsnprintf to strcpy for colors\n\nOur color buffers are all COLOR_MAXLEN, which fits the\nlargest possible color. So we can never overflow the buffer\nby copying an existing color. However, using strcpy() makes\nit harder to audit the code-base for calls that _are_\nproblems. We should use something like xsnprintf(), which\nshows the reader that we expect this never to fail (and\nprovides a run-time assertion if it does, just in case).\n\nSigned-off-by: Jeff King \n---\nAnother option would just be color_parse(repeated_meta_color, \"cyan\").\nThe run-time cost is slightly higher, but it probably doesn't matter\nhere, and perhaps it's more readable.\n\nThis one is less critical for v2.18.\n\n builtin/blame.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/blame.c b/builtin/blame.c\nindex 4202584f97..45770c5a8c 100644\n--- a/builtin/blame.c\n+++ b/builtin/blame.c\n@@ -1068,7 +1068,9 @@ int cmd_blame(int argc, const char **argv, const char *prefix)\n \t\tfind_alignment(&sb, &output_option);\n \t\tif (!*repeated_meta_color &&\n \t\t (output_option & OUTPUT_COLOR_LINE))\n-\t\t\tstrcpy(repeated_meta_color, GIT_COLOR_CYAN);\n+\t\t\txsnprintf(repeated_meta_color,\n+\t\t\t\t sizeof(repeated_meta_color),\n+\t\t\t\t \"%s\", GIT_COLOR_CYAN);\n \t}\n \tif (output_option & OUTPUT_ANNOTATE_COMPAT)\n \t\toutput_option &= ~(OUTPUT_COLOR_LINE | OUTPUT_SHOW_AGE_WITH_COLOR);\n-- \n2.18.0.rc1.446.g4486251e51\n\n"},{"id":"349899","messageId":"CABPp-BF3gUf_hzUNXtXdyv01R1hQzLCKdRgb+iKPnBgLYi6d+Q@mail.gmail.com","threadId":"47162","inReplyTo":"7d1237c7-5a83-d766-7d93-5f0d59166067@web.de","subject":"Re: [PATCH 24/30] merge-recursive: Add computation of collisions due to dir rename & merging","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2018-06-11T15:03:00Z","receivedAt":"2018-06-11T15:03:04Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Jun 10, 2018 at 3:56 AM, René Scharfe wrote:\n> Am 10.11.2017 um 20:05 schrieb Elijah Newren:\n>> +static struct dir_rename_entry *check_dir_renamed(const char *path,\n>> + struct hashmap *dir_renames) {\n>> + char temp[PATH_MAX];\n>> + char *end;\n>> + struct dir_rename_entry *entry;\n>> +\n>> + strcpy(temp, path);\n>> + while ((end = strrchr(temp, '/'))) {\n>> + *end = '\\0';\n>> + entry = dir_rename_find_entry(dir_renames, temp);\n>> + if (entry)\n>> + return entry;\n>> + }\n>> + return NULL;\n>> +}\n>\n> The value of PATH_MAX is platform-dependent, so it's easy to exceed when\n> doing cross-platform development. It's also not a hard limit on most\n> operating systems, not even on Windows. Further reading:\n>\n> https://insanecoding.blogspot.com/2007/11/pathmax-simply-isnt.html\n>\n> So using a fixed buffer is not a good idea, and writing to it without\n> checking is dangerous. Here's a fix:\n\nThanks for the pointers, and for providing a fix.\n\n> -- >8 --\n> Subject: [PATCH] merge-recursive: use xstrdup() instead of fixed buffer\n>\n> Paths can be longer than PATH_MAX. Avoid a buffer overrun in\n> check_dir_renamed() by using xstrdup() to make a private copy safely.\n>\n> Signed-off-by: Rene Scharfe \n> ---\n> merge-recursive.c | 10 +++++-----\n> 1 file changed, 5 insertions(+), 5 deletions(-)\n>\n> diff --git a/merge-recursive.c b/merge-recursive.c\n> index ac27abbd4c..db708176c5 100644\n> --- a/merge-recursive.c\n> +++ b/merge-recursive.c\n> @@ -2211,18 +2211,18 @@ static struct hashmap *get_directory_renames(struct diff_queue_struct *pairs,\n> static struct dir_rename_entry *check_dir_renamed(const char *path,\n> struct hashmap *dir_renames)\n> {\n> - char temp[PATH_MAX];\n> + char *temp = xstrdup(path);\n> char *end;\n> - struct dir_rename_entry *entry;\n> + struct dir_rename_entry *entry = NULL;;\n>\n> - strcpy(temp, path);\n> while ((end = strrchr(temp, '/'))) {\n> *end = '\\0';\n> entry = dir_rename_find_entry(dir_renames, temp);\n> if (entry)\n> - return entry;\n> + break;\n> }\n> - return NULL;\n> + free(temp);\n> + return entry;\n> }\n>\n> static void compute_collisions(struct hashmap *collisions,\n> --\n> 2.17.1\n\nReviewed-by: Elijah Newren \n"},{"id":"350169","messageId":"xmqqr2l946dr.fsf@gitster-ct.c.googlers.com","threadId":"47162","inReplyTo":"7d1237c7-5a83-d766-7d93-5f0d59166067@web.de","subject":"Re: [PATCH 24/30] merge-recursive: Add computation of collisions due to dir rename & merging","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-14T17:36:48Z","receivedAt":"2018-06-14T17:36:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n> The value of PATH_MAX is platform-dependent, so it's easy to exceed when\n> doing cross-platform development. It's also not a hard limit on most\n> operating systems, not even on Windows. Further reading:\n>\n> https://insanecoding.blogspot.com/2007/11/pathmax-simply-isnt.html\n>\n> So using a fixed buffer is not a good idea, and writing to it without\n> checking is dangerous. Here's a fix:\n>\n> -- >8 --\n> Subject: [PATCH] merge-recursive: use xstrdup() instead of fixed buffer\n>\n> Paths can be longer than PATH_MAX. Avoid a buffer overrun in\n> check_dir_renamed() by using xstrdup() to make a private copy safely.\n>\n> Signed-off-by: Rene Scharfe \n> ---\n\nThanks. Makes sense.\n\n> merge-recursive.c | 10 +++++-----\n> 1 file changed, 5 insertions(+), 5 deletions(-)\n>\n> diff --git a/merge-recursive.c b/merge-recursive.c\n> index ac27abbd4c..db708176c5 100644\n> --- a/merge-recursive.c\n> +++ b/merge-recursive.c\n> @@ -2211,18 +2211,18 @@ static struct hashmap *get_directory_renames(struct diff_queue_struct *pairs,\n> static struct dir_rename_entry *check_dir_renamed(const char *path,\n> \t\t\t\t\t\t struct hashmap *dir_renames)\n> {\n> -\tchar temp[PATH_MAX];\n> +\tchar *temp = xstrdup(path);\n> \tchar *end;\n> -\tstruct dir_rename_entry *entry;\n> +\tstruct dir_rename_entry *entry = NULL;;\n> \n> -\tstrcpy(temp, path);\n> \twhile ((end = strrchr(temp, '/'))) {\n> \t\t*end = '\\0';\n> \t\tentry = dir_rename_find_entry(dir_renames, temp);\n> \t\tif (entry)\n> -\t\t\treturn entry;\n> +\t\t\tbreak;\n> \t}\n> -\treturn NULL;\n> +\tfree(temp);\n> +\treturn entry;\n> }\n> \n> static void compute_collisions(struct hashmap *collisions,\n"}]}