git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 17/17] merge-ort: fix a directory rename detection bug

From
Elijah Newren via GitGitGadget <gitgitgadget@gmail.com>
Date
Jan 19, 2021, 19:53 UTC
Message-ID
<7c24f9f7aedc57c08cc1d12141590d8a83f1c72e.1611086033.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.835.v3.git.1611086033.gitgitgadget@gmail.com>
From: Elijah Newren <newren@gmail.com>

As noted in commit 902c521a35 ("t6423: more involved directory rename test", 2020-10-15), when we have a case where

  * dir/subdir/ has several files
  * almost all files in dir/subdir/ are renamed to folder/subdir/
  * one of the files in dir/subdir/ is renamed to folder/subdir/newsubdir/
  * the other side of history (that doesn't do the renames) adds a
    new file to dir/subdir/
Then for the majority of the file renames, the directory rename of
   dir/subdir/ -> folder/subdir/
is actually not represented that way but as
   dir/ -> folder/
We also had one rename that was represented as
   dir/subdir/ -> folder/subdir/newsubdir/

Now, since there's a new file in dir/subdir/, where does it go? Well, there's only one rule for dir/subdir/, so the code previously noted that this rule had the "majority" of the one "relevant" rename and thus erroneously used it to place the file in folder/subdir/newsubdir/. We really want the heavy weight associated with dir/ -> folder/ to also be treated as dir/subdir/ -> folder/subdir/, so that we correctly place the file in folder/subdir/.

Add a bunch of logic to make sure that we use all relevant renamings in directory rename detection.

Note that testcase 12f of t6423 still fails after this, but it gets further than merge-recursive does. There are some performance related bits in that testcase (the region_enter messages) that do not yet succeed, but the rest of the testcase works after this patch. Subsequent patch series will fix up the performance side.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-ort.c | 198 +++++++++++++++++++++-------------------------------
 1 file changed, 81 insertions(+), 117 deletions(-)
diff --git a/merge-ort.c b/merge-ort.c
index 7314f9c69c2..3192b27625e 100644
--- a/merge-ort.c
+++ b/merge-ort.c
@@ -764,109 +764,6 @@ static char *apply_dir_rename(struct strmap_entry *rename_info,
 	return strbuf_detach(&new_path, NULL);
 }
 
-static void get_renamed_dir_portion(const char *old_path, const char *new_path,
-				    char **old_dir, char **new_dir)
-{
-	char *end_of_old, *end_of_new;
-
-	/* Default return values: NULL, meaning no rename */
-	*old_dir = NULL;
-	*new_dir = NULL;
-
-	/*
-	 * For
-	 *    "a/b/c/d/e/foo.c" -> "a/b/some/thing/else/e/foo.c"
-	 * the "e/foo.c" part is the same, we just want to know that
-	 *    "a/b/c/d" was renamed to "a/b/some/thing/else"
-	 * so, for this example, this function returns "a/b/c/d" in
-	 * *old_dir and "a/b/some/thing/else" in *new_dir.
-	 */
-
-	/*
-	 * If the basename of the file changed, we don't care.  We want
-	 * to know which portion of the directory, if any, changed.
-	 */
-	end_of_old = strrchr(old_path, '/');
-	end_of_new = strrchr(new_path, '/');
-
-	/*
-	 * If end_of_old is NULL, old_path wasn't in a directory, so there
-	 * could not be a directory rename (our rule elsewhere that a
-	 * directory which still exists is not considered to have been
-	 * renamed means the root directory can never be renamed -- because
-	 * the root directory always exists).
-	 */
-	if (end_of_old == NULL)
-		return; /* Note: *old_dir and *new_dir are still NULL */
-
-	/*
-	 * If new_path contains no directory (end_of_new is NULL), then we
-	 * have a rename of old_path's directory to the root directory.
-	 */
-	if (end_of_new == NULL) {
-		*old_dir = xstrndup(old_path, end_of_old - old_path);
-		*new_dir = xstrdup("");
-		return;
-	}
-
-	/* Find the first non-matching character traversing backwards */
-	while (*--end_of_new == *--end_of_old &&
-	       end_of_old != old_path &&
-	       end_of_new != new_path)
-		; /* Do nothing; all in the while loop */
-
-	/*
-	 * If both got back to the beginning of their strings, then the
-	 * directory didn't change at all, only the basename did.
-	 */
-	if (end_of_old == old_path && end_of_new == new_path &&
-	    *end_of_old == *end_of_new)
-		return; /* Note: *old_dir and *new_dir are still NULL */
-
-	/*
-	 * If end_of_new got back to the beginning of its string, and
-	 * end_of_old got back to the beginning of some subdirectory, then
-	 * we have a rename/merge of a subdirectory into the root, which
-	 * needs slightly special handling.
-	 *
-	 * Note: There is no need to consider the opposite case, with a
-	 * rename/merge of the root directory into some subdirectory
-	 * because as noted above the root directory always exists so it
-	 * cannot be considered to be renamed.
-	 */
-	if (end_of_new == new_path &&
-	    end_of_old != old_path && end_of_old[-1] == '/') {
-		*old_dir = xstrndup(old_path, --end_of_old - old_path);
-		*new_dir = xstrdup("");
-		return;
-	}
-
-	/*
-	 * We've found the first non-matching character in the directory
-	 * paths.  That means the current characters we were looking at
-	 * were part of the first non-matching subdir name going back from
-	 * the end of the strings.  Get the whole name by advancing both
-	 * end_of_old and end_of_new to the NEXT '/' character.  That will
-	 * represent the entire directory rename.
-	 *
-	 * The reason for the increment is cases like
-	 *    a/b/star/foo/whatever.c -> a/b/tar/foo/random.c
-	 * After dropping the basename and going back to the first
-	 * non-matching character, we're now comparing:
-	 *    a/b/s          and         a/b/
-	 * and we want to be comparing:
-	 *    a/b/star/      and         a/b/tar/
-	 * but without the pre-increment, the one on the right would stay
-	 * a/b/.
-	 */
-	end_of_old = strchr(++end_of_old, '/');
-	end_of_new = strchr(++end_of_new, '/');
-
-	/* Copy the old and new directories into *old_dir and *new_dir. */
-	*old_dir = xstrndup(old_path, end_of_old - old_path);
-	*new_dir = xstrndup(new_path, end_of_new - new_path);
-}
-
 static int path_in_way(struct strmap *paths, const char *path, unsigned side_mask)
 {
 	struct merged_info *mi = strmap_get(paths, path);
@@ -952,6 +849,14 @@ static char *handle_path_level_conflicts(struct merge_options *opt,
 	return new_path;
 }
 
+static void dirname_munge(char *filename)
+{
+	char *slash = strrchr(filename, '/');
+	if (!slash)
+		slash = filename;
+	*slash = '\0';
+}
+
 static void increment_count(struct strmap *dir_rename_count,
 			    char *old_dir,
 			    char *new_dir)
@@ -973,6 +878,76 @@ static void increment_count(struct strmap *dir_rename_count,
 	strintmap_incr(counts, new_dir, 1);
 }
 
+static void update_dir_rename_counts(struct strmap *dir_rename_count,
+				     struct strset *dirs_removed,
+				     const char *oldname,
+				     const char *newname)
+{
+	char *old_dir = xstrdup(oldname);
+	char *new_dir = xstrdup(newname);
+	char new_dir_first_char = new_dir[0];
+	int first_time_in_loop = 1;
+
+	while (1) {
+		dirname_munge(old_dir);
+		dirname_munge(new_dir);
+
+		/*
+		 * When renaming
+		 *   "a/b/c/d/e/foo.c" -> "a/b/some/thing/else/e/foo.c"
+		 * then this suggests that both
+		 *   a/b/c/d/e/ => a/b/some/thing/else/e/
+		 *   a/b/c/d/   => a/b/some/thing/else/
+		 * so we want to increment counters for both.  We do NOT,
+		 * however, also want to suggest that there was the following
+		 * rename:
+		 *   a/b/c/ => a/b/some/thing/
+		 * so we need to quit at that point.
+		 *
+		 * Note the when first_time_in_loop, we only strip off the
+		 * basename, and we don't care if that's different.
+		 */
+		if (!first_time_in_loop) {
+			char *old_sub_dir = strchr(old_dir, '\0')+1;
+			char *new_sub_dir = strchr(new_dir, '\0')+1;
+			if (!*new_dir) {
+				/*
+				 * Special case when renaming to root directory,
+				 * i.e. when new_dir == "".  In this case, we had
+				 * something like
+				 *    a/b/subdir => subdir
+				 * and so dirname_munge() sets things up so that
+				 *    old_dir = "a/b\0subdir\0"
+				 *    new_dir = "\0ubdir\0"
+				 * We didn't have a '/' to overwrite a '\0' onto
+				 * in new_dir, so we have to compare differently.
+				 */
+				if (new_dir_first_char != old_sub_dir[0] ||
+				    strcmp(old_sub_dir+1, new_sub_dir))
+					break;
+			} else {
+				if (strcmp(old_sub_dir, new_sub_dir))
+					break;
+			}
+		}
+
+		if (strset_contains(dirs_removed, old_dir))
+			increment_count(dir_rename_count, old_dir, new_dir);
+		else
+			break;
+
+		/* If we hit toplevel directory ("") for old or new dir, quit */
+		if (!*old_dir || !*new_dir)
+			break;
+
+		first_time_in_loop = 0;
+	}
+
+	/* Free resources we don't need anymore */
+	free(old_dir);
+	free(new_dir);
+}
+
 static void compute_rename_counts(struct diff_queue_struct *pairs,
 				  struct strmap *dir_rename_count,
 				  struct strset *dirs_removed)
@@ -980,20 +955,12 @@ static void compute_rename_counts(struct diff_queue_struct *pairs,
 	int i;
 
 	for (i = 0; i < pairs->nr; ++i) {
-		char *old_dir, *new_dir;
 		struct diff_filepair *pair = pairs->queue[i];
 
 		/* File not part of directory rename if it wasn't renamed */
 		if (pair->status != 'R')
 			continue;
 
-		/* Get the old and new directory names */
-		get_renamed_dir_portion(pair->one->path, pair->two->path,
-					&old_dir,        &new_dir);
-		if (!old_dir)
-			/* Directory didn't change at all; ignore this one. */
-			continue;
-
 		/*
 		 * Make dir_rename_count contain a map of a map:
 		 *   old_directory -> {new_directory -> count}
@@ -1001,12 +968,9 @@ static void compute_rename_counts(struct diff_queue_struct *pairs,
 		 * the old filename and the new filename and count how many
 		 * times that pairing occurs.
 		 */
-		if (strset_contains(dirs_removed, old_dir))
-			increment_count(dir_rename_count, old_dir, new_dir);
-
-		/* Free resources we don't need anymore */
-		free(old_dir);
-		free(new_dir);
+		update_dir_rename_counts(dir_rename_count, dirs_removed,
+					 pair->one->path,
+					 pair->two->path);
 	}
 }
 
-- 
gitgitgadget
Previous: Elijah Newren via GitGitGadgetNext: Elijah Newren via GitGitGadget
Message 56 of 66 in “Add directory rename detection to merge-ort”
  1. 00/18 Add directory rename detection to merge-ortElijah Newren via GitGitGadget, Jan 7, 2021
  2. 10/18 merge-ort: implement compute_collisions()Elijah Newren via GitGitGadget, Jan 7, 2021
  3. 12/18 merge-ort: implement check_for_directory_rename()Elijah Newren via GitGitGadget, Jan 7, 2021
  4. 02/18 merge-ort: initialize and free new directory rename data structuresElijah Newren via GitGitGadget, Jan 7, 2021
  5. 01/18 merge-ort: add new data structures for directory rename detectionElijah Newren via GitGitGadget, Jan 7, 2021
  6. 07/18 merge-ort: implement compute_rename_counts()Elijah Newren via GitGitGadget, Jan 7, 2021
  7. 14/18 merge-ort: add a new toplevel_dir fieldElijah Newren via GitGitGadget, Jan 7, 2021
  8. 03/18 merge-ort: collect which directories are removed in dirs_removedElijah Newren via GitGitGadget, Jan 7, 2021
  9. 05/18 merge-ort: add outline of get_provisional_directory_renames()Elijah Newren via GitGitGadget, Jan 7, 2021
  10. 06/18 merge-ort: copy get_renamed_dir_portion() from merge-recursive.cElijah Newren via GitGitGadget, Jan 7, 2021
  11. 08/18 merge-ort: implement handle_directory_level_conflicts()Elijah Newren via GitGitGadget, Jan 7, 2021
  12. 09/18 merge-ort: modify collect_renames() for directory rename handlingElijah Newren via GitGitGadget, Jan 7, 2021
  13. 04/18 merge-ort: add outline for computing directory renamesElijah Newren via GitGitGadget, Jan 7, 2021
  14. 16/18 merge-ort: process_renames() now needs more defensivenessElijah Newren via GitGitGadget, Jan 7, 2021
  15. 11/18 merge-ort: implement apply_dir_rename() and check_dir_renamed()Elijah Newren via GitGitGadget, Jan 7, 2021
  16. 18/18 DO NOT SUBMIT: directory rename stuff for redo_after_renamesElijah Newren via GitGitGadget, Jan 7, 2021
  17. Elijah NewrenJan 7, 2021
  18. 13/18 merge-ort: implement handle_path_level_conflicts()Elijah Newren via GitGitGadget, Jan 7, 2021
  19. 15/18 merge-ort: implement apply_directory_rename_modifications()Elijah Newren via GitGitGadget, Jan 7, 2021
  20. 17/18 merge-ort: fix a directory rename detection bugElijah Newren via GitGitGadget, Jan 7, 2021
  21. 00/17 Add directory rename detection to merge-ortElijah Newren via GitGitGadget, Jan 7, 2021
  22. 02/17 merge-ort: initialize and free new directory rename data structuresElijah Newren via GitGitGadget, Jan 7, 2021
  23. 01/17 merge-ort: add new data structures for directory rename detectionElijah Newren via GitGitGadget, Jan 7, 2021
  24. 05/17 merge-ort: add outline of get_provisional_directory_renames()Elijah Newren via GitGitGadget, Jan 7, 2021
  25. Taylor BlauJan 18, 2021
  26. Elijah NewrenJan 18, 2021
  27. 09/17 merge-ort: modify collect_renames() for directory rename handlingElijah Newren via GitGitGadget, Jan 7, 2021
  28. 15/17 merge-ort: implement apply_directory_rename_modifications()Elijah Newren via GitGitGadget, Jan 7, 2021
  29. 03/17 merge-ort: collect which directories are removed in dirs_removedElijah Newren via GitGitGadget, Jan 7, 2021
  30. 16/17 merge-ort: process_renames() now needs more defensivenessElijah Newren via GitGitGadget, Jan 7, 2021
  31. 04/17 merge-ort: add outline for computing directory renamesElijah Newren via GitGitGadget, Jan 7, 2021
  32. Taylor BlauJan 18, 2021
  33. Elijah NewrenJan 18, 2021
  34. Taylor BlauJan 18, 2021
  35. 10/17 merge-ort: implement compute_collisions()Elijah Newren via GitGitGadget, Jan 7, 2021
  36. 13/17 merge-ort: implement handle_path_level_conflicts()Elijah Newren via GitGitGadget, Jan 7, 2021
  37. 11/17 merge-ort: implement apply_dir_rename() and check_dir_renamed()Elijah Newren via GitGitGadget, Jan 7, 2021
  38. 14/17 merge-ort: add a new toplevel_dir fieldElijah Newren via GitGitGadget, Jan 7, 2021
  39. 08/17 merge-ort: implement handle_directory_level_conflicts()Elijah Newren via GitGitGadget, Jan 7, 2021
  40. Taylor BlauJan 18, 2021
  41. Elijah NewrenJan 18, 2021
  42. 06/17 merge-ort: copy get_renamed_dir_portion() from merge-recursive.cElijah Newren via GitGitGadget, Jan 7, 2021
  43. 12/17 merge-ort: implement check_for_directory_rename()Elijah Newren via GitGitGadget, Jan 7, 2021
  44. 07/17 merge-ort: implement compute_rename_counts()Elijah Newren via GitGitGadget, Jan 7, 2021
  45. Taylor BlauJan 18, 2021
  46. Elijah NewrenJan 18, 2021
  47. 17/17 merge-ort: fix a directory rename detection bugElijah Newren via GitGitGadget, Jan 7, 2021
  48. 00/17 Add directory rename detection to merge-ortElijah Newren via GitGitGadget, Jan 19, 2021
  49. 02/17 merge-ort: initialize and free new directory rename data structuresElijah Newren via GitGitGadget, Jan 19, 2021
  50. 03/17 merge-ort: collect which directories are removed in dirs_removedElijah Newren via GitGitGadget, Jan 19, 2021
  51. 06/17 merge-ort: copy get_renamed_dir_portion() from merge-recursive.cElijah Newren via GitGitGadget, Jan 19, 2021
  52. 04/17 merge-ort: add outline for computing directory renamesElijah Newren via GitGitGadget, Jan 19, 2021
  53. 05/17 merge-ort: add outline of get_provisional_directory_renames()Elijah Newren via GitGitGadget, Jan 19, 2021
  54. 10/17 merge-ort: implement compute_collisions()Elijah Newren via GitGitGadget, Jan 19, 2021
  55. 15/17 merge-ort: implement apply_directory_rename_modifications()Elijah Newren via GitGitGadget, Jan 19, 2021
  56. 17/17 merge-ort: fix a directory rename detection bugElijah Newren via GitGitGadget, Jan 19, 2021
  57. 11/17 merge-ort: implement apply_dir_rename() and check_dir_renamed()Elijah Newren via GitGitGadget, Jan 19, 2021
  58. 16/17 merge-ort: process_renames() now needs more defensivenessElijah Newren via GitGitGadget, Jan 19, 2021
  59. 14/17 merge-ort: add a new toplevel_dir fieldElijah Newren via GitGitGadget, Jan 19, 2021
  60. 13/17 merge-ort: implement handle_path_level_conflicts()Elijah Newren via GitGitGadget, Jan 19, 2021
  61. 12/17 merge-ort: implement check_for_directory_rename()Elijah Newren via GitGitGadget, Jan 19, 2021
  62. 08/17 merge-ort: implement handle_directory_level_conflicts()Elijah Newren via GitGitGadget, Jan 19, 2021
  63. 09/17 merge-ort: modify collect_renames() for directory rename handlingElijah Newren via GitGitGadget, Jan 19, 2021
  64. 01/17 merge-ort: add new data structures for directory rename detectionElijah Newren via GitGitGadget, Jan 19, 2021
  65. 07/17 merge-ort: implement compute_rename_counts()Elijah Newren via GitGitGadget, Jan 19, 2021
  66. Taylor BlauJan 19, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.