{"thread":{"id":"54556","subject":"[PATCH v2 10/20] merge-ort: avoid recursing into identical trees","startedAt":"2020-11-02T20:44:09Z","lastAt":"2020-11-30T18:42:43Z","messageCount":83,"participants":["Elijah Newren","Derrick Stolee","Jonathan Tan","Eric Sunshine"],"isPatch":true,"patchVersion":2,"patchTotal":20},"messages":[{"id":"408905","messageId":"20201102204344.342633-11-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 10/20] merge-ort: avoid recursing into identical trees","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:34Z","receivedAt":"2020-11-02T20:44:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"When all three trees have the same oid, there is no need to recurse into\nthese trees to find that all files within them happen to match. We can\njust record any one of the trees as the resolution of merging that\nparticular path.\n\nImmediately resolving trees for other types of trivial tree merges (such\nas one side matches the merge base, or the two sides match each other)\nwould prevent us from detecting renames for some paths, and thus prevent\nus from doing three-way content merges for those paths whose renames we\ndid not detect.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex bef3c648a0..9900fa1bf8 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -203,6 +203,19 @@ static int collect_merge_info_callback(int n,\n \tfullpath = xmalloc(len+1);\n \tmake_traverse_path(fullpath, len+1, info, p->path, p->pathlen);\n \n+\t/*\n+\t * If mbase, side1, and side2 all match, we can resolve early. Even\n+\t * if these are trees, there will be no renames or anything\n+\t * underneath.\n+\t */\n+\tif (side1_matches_mbase && side2_matches_mbase) {\n+\t\t/* mbase, side1, & side2 all match; use mbase as resolution */\n+\t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n+\t\t\t\tnames, names+0, mbase_null, 0,\n+\t\t\t\tfilemask, dirmask, 1);\n+\t\treturn mask;\n+\t}\n+\n \t/*\n \t * Record information about the path so we can resolve later in\n \t * process_entries.\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408906","messageId":"20201102204344.342633-9-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 08/20] merge-ort: compute a few more useful fields for collect_merge_info","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:32Z","receivedAt":"2020-11-02T20:44:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n merge-ort.c | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex d3c1d00fc6..0ff90981cf 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -96,6 +96,7 @@ static int collect_merge_info_callback(int n,\n \tsize_t len;\n \tchar *fullpath;\n \tunsigned filemask = mask & ~dirmask;\n+\tunsigned match_mask = 0; /* will be updated below */\n \tunsigned mbase_null = !(mask & 1);\n \tunsigned side1_null = !(mask & 2);\n \tunsigned side2_null = !(mask & 4);\n@@ -108,6 +109,13 @@ static int collect_merge_info_callback(int n,\n \tunsigned sides_match = (!side1_null && !side2_null &&\n \t\t\t\tnames[1].mode == names[2].mode &&\n \t\t\t\toideq(&names[1].oid, &names[2].oid));\n+\t/*\n+\t * Note: We only label files with df_conflict, not directories.\n+\t * Since directories stay where they are, and files move out of the\n+\t * way to make room for a directory, we don't care if there was a\n+\t * directory/file conflict for a parent directory of the current path.\n+\t */\n+\tunsigned df_conflict = (filemask != 0) && (dirmask != 0);\n \n \t/* n = 3 is a fundamental assumption. */\n \tif (n != 3)\n@@ -127,6 +135,14 @@ static int collect_merge_info_callback(int n,\n \t/* Other invariant checks, mostly for documentation purposes. */\n \tassert(mask == (dirmask | filemask));\n \n+\t/* Determine match_mask */\n+\tif (side1_matches_mbase)\n+\t\tmatch_mask = (side2_matches_mbase ? 7 : 3);\n+\telse if (side2_matches_mbase)\n+\t\tmatch_mask = 5;\n+\telse if (sides_match)\n+\t\tmatch_mask = 6;\n+\n \t/*\n \t * Get the name of the relevant filepath, which we'll pass to\n \t * setup_path_info() for tracking.\n@@ -145,6 +161,8 @@ static int collect_merge_info_callback(int n,\n \t * so we can resolve later in process_entries.\n \t */\n \tci = xcalloc(1, sizeof(struct conflict_info));\n+\tci->df_conflict = df_conflict;\n+\tci->match_mask = match_mask;\n \tstrmap_put(&opti->paths, fullpath, ci);\n \n \t/* If dirmask, recurse into subdirectories */\n@@ -161,6 +179,13 @@ static int collect_merge_info_callback(int n,\n \t\tnewinfo.name = p->path;\n \t\tnewinfo.namelen = p->pathlen;\n \t\tnewinfo.pathlen = st_add3(newinfo.pathlen, p->pathlen, 1);\n+\t\t/*\n+\t\t * If we did care about parent directories having a D/F\n+\t\t * conflict, then we'd include\n+\t\t * newinfo.df_conflicts |= (mask & ~dirmask);\n+\t\t * here. But we don't. (See comment near setting of local\n+\t\t * df_conflict variable near the beginning of this function).\n+\t\t */\n \n \t\tfor (i = 0; i < 3; i++, dirmask >>= 1) {\n \t\t\tif (i == 1 && side1_matches_mbase)\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408907","messageId":"20201102204344.342633-12-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 11/20] merge-ort: add a preliminary simple process_entries() implementation","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:35Z","receivedAt":"2020-11-02T20:44:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Add a process_entries() implementation that just loops over the paths\nand processes each one individually with an auxiliary process_entry()\ncall. Add a basic process_entry() as well, which handles several cases\nbut leaves a few of the more involved ones with die-not-implemented\nmessages. Also, although process_entries() is supposed to create a\ntree, it does not yet have code to do so -- except in the special case\nof merging completely empty trees.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 106 +++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 105 insertions(+), 1 deletion(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 9900fa1bf8..92bbdc7255 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -323,10 +323,114 @@ static int detect_and_process_renames(struct merge_options *opt,\n \treturn clean;\n }\n \n+/* Per entry merge function */\n+static void process_entry(struct merge_options *opt,\n+\t\t\t const char *path,\n+\t\t\t struct conflict_info *ci)\n+{\n+\tassert(!ci->merged.clean);\n+\tassert(ci->filemask >= 0 && ci->filemask <= 7);\n+\n+\tif (ci->filemask == 0) {\n+\t\t/*\n+\t\t * This is a placeholder for directories that were recursed\n+\t\t * into; nothing to do in this case.\n+\t\t */\n+\t\treturn;\n+\t}\n+\n+\tif (ci->df_conflict) {\n+\t\tdie(\"Not yet implemented.\");\n+\t}\n+\n+\t/*\n+\t * NOTE: Below there is a long switch-like if-elseif-elseif... block\n+\t * which the code goes through even for the df_conflict cases\n+\t * above. Well, it will once we don't die-not-implemented above.\n+\t */\n+\tif (ci->match_mask) {\n+\t\tci->merged.clean = 1;\n+\t\tif (ci->match_mask == 6) {\n+\t\t\t/* stages[1] == stages[2] */\n+\t\t\tci->merged.result.mode = ci->stages[1].mode;\n+\t\t\toidcpy(&ci->merged.result.oid, &ci->stages[1].oid);\n+\t\t} else {\n+\t\t\t/* determine the mask of the side that didn't match */\n+\t\t\tunsigned int othermask = 7 & ~ci->match_mask;\n+\t\t\tint side = (othermask == 4) ? 2 : 1;\n+\n+\t\t\tci->merged.is_null = (ci->filemask == ci->match_mask);\n+\t\t\tci->merged.result.mode = ci->stages[side].mode;\n+\t\t\toidcpy(&ci->merged.result.oid, &ci->stages[side].oid);\n+\n+\t\t\tassert(othermask == 2 || othermask == 4);\n+\t\t\tassert(ci->merged.is_null == !ci->merged.result.mode);\n+\t\t}\n+\t} else if (ci->filemask >= 6 &&\n+\t\t (S_IFMT & ci->stages[1].mode) !=\n+\t\t (S_IFMT & ci->stages[2].mode)) {\n+\t\t/*\n+\t\t * Two different items from (file/submodule/symlink)\n+\t\t */\n+\t\tdie(\"Not yet implemented.\");\n+\t} else if (ci->filemask >= 6) {\n+\t\t/*\n+\t\t * TODO: Needs a two-way or three-way content merge, but we're\n+\t\t * just being lazy and copying the version from HEAD and\n+\t\t * leaving it as conflicted.\n+\t\t */\n+\t\tci->merged.clean = 0;\n+\t\tci->merged.result.mode = ci->stages[1].mode;\n+\t\toidcpy(&ci->merged.result.oid, &ci->stages[1].oid);\n+\t} else if (ci->filemask == 3 || ci->filemask == 5) {\n+\t\t/* Modify/delete */\n+\t\tdie(\"Not yet implemented.\");\n+\t} else if (ci->filemask == 2 || ci->filemask == 4) {\n+\t\t/* Added on one side */\n+\t\tint side = (ci->filemask == 4) ? 2 : 1;\n+\t\tci->merged.result.mode = ci->stages[side].mode;\n+\t\toidcpy(&ci->merged.result.oid, &ci->stages[side].oid);\n+\t\tci->merged.clean = !ci->df_conflict && !ci->path_conflict;\n+\t} else if (ci->filemask == 1) {\n+\t\t/* Deleted on both sides */\n+\t\tci->merged.is_null = 1;\n+\t\tci->merged.result.mode = 0;\n+\t\toidcpy(&ci->merged.result.oid, &null_oid);\n+\t\tci->merged.clean = !ci->path_conflict;\n+\t}\n+\n+\t/*\n+\t * If still unmerged, record it separately. This allows us to later\n+\t * iterate over just unmerged entries when updating the index instead\n+\t * of iterating over all entries.\n+\t */\n+\tif (!ci->merged.clean)\n+\t\tstrmap_put(&opt->priv->unmerged, path, ci);\n+}\n+\n static void process_entries(struct merge_options *opt,\n \t\t\t struct object_id *result_oid)\n {\n-\tdie(\"Not yet implemented.\");\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *e;\n+\n+\tif (strmap_empty(&opt->priv->paths)) {\n+\t\toidcpy(result_oid, opt->repo->hash_algo->empty_tree);\n+\t\treturn;\n+\t}\n+\n+\tstrmap_for_each_entry(&opt->priv->paths, &iter, e) {\n+\t\t/*\n+\t\t * WARNING: If ci->merged.clean is true, then ci does not\n+\t\t * actually point to a conflict_info but a struct merge_info.\n+\t\t */\n+\t\tstruct conflict_info *ci = e->value;\n+\n+\t\tif (!ci->merged.clean)\n+\t\t\tprocess_entry(opt, e->key, e->value);\n+\t}\n+\n+\tdie(\"Tree creation not yet implemented\");\n }\n \n void merge_switch_to_result(struct merge_options *opt,\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408908","messageId":"20201102204344.342633-10-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 09/20] merge-ort: record stage and auxiliary info for every path","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:33Z","receivedAt":"2020-11-02T20:44:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Create a helper function, setup_path_info(), which can be used to record\nall the information we want in a merged_info or conflict_info. While\nthere is currently only one caller of this new function, and some of its\nparticular parameters are fixed, future callers of this function will be\nadded later.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++------\n 1 file changed, 54 insertions(+), 7 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 0ff90981cf..bef3c648a0 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -77,6 +77,51 @@ static int err(struct merge_options *opt, const char *err, ...)\n \treturn -1;\n }\n \n+static void setup_path_info(struct merge_options *opt,\n+\t\t\t struct string_list_item *result,\n+\t\t\t const char *current_dir_name,\n+\t\t\t int current_dir_name_len,\n+\t\t\t char *fullpath, /* we'll take over ownership */\n+\t\t\t struct name_entry *names,\n+\t\t\t struct name_entry *merged_version,\n+\t\t\t unsigned is_null, /* boolean */\n+\t\t\t unsigned df_conflict, /* boolean */\n+\t\t\t unsigned filemask,\n+\t\t\t unsigned dirmask,\n+\t\t\t int resolved /* boolean */)\n+{\n+\tstruct conflict_info *path_info;\n+\n+\tassert(!is_null || resolved);\n+\tassert(!df_conflict || !resolved); /* df_conflict implies !resolved */\n+\tassert(resolved == (merged_version != NULL));\n+\n+\tpath_info = xcalloc(1, resolved ? sizeof(struct merged_info) :\n+\t\t\t\t\t sizeof(struct conflict_info));\n+\tpath_info->merged.directory_name = current_dir_name;\n+\tpath_info->merged.basename_offset = current_dir_name_len;\n+\tpath_info->merged.clean = !!resolved;\n+\tif (resolved) {\n+\t\tpath_info->merged.result.mode = merged_version->mode;\n+\t\toidcpy(&path_info->merged.result.oid, &merged_version->oid);\n+\t\tpath_info->merged.is_null = !!is_null;\n+\t} else {\n+\t\tint i;\n+\n+\t\tfor (i = 0; i < 3; i++) {\n+\t\t\tpath_info->pathnames[i] = fullpath;\n+\t\t\tpath_info->stages[i].mode = names[i].mode;\n+\t\t\toidcpy(&path_info->stages[i].oid, &names[i].oid);\n+\t\t}\n+\t\tpath_info->filemask = filemask;\n+\t\tpath_info->dirmask = dirmask;\n+\t\tpath_info->df_conflict = !!df_conflict;\n+\t}\n+\tstrmap_put(&opt->priv->paths, fullpath, path_info);\n+\tresult->string = fullpath;\n+\tresult->util = path_info;\n+}\n+\n static int collect_merge_info_callback(int n,\n \t\t\t\t unsigned long mask,\n \t\t\t\t unsigned long dirmask,\n@@ -91,10 +136,12 @@ static int collect_merge_info_callback(int n,\n \t */\n \tstruct merge_options *opt = info->data;\n \tstruct merge_options_internal *opti = opt->priv;\n-\tstruct conflict_info *ci;\n+\tstruct string_list_item pi; /* Path Info */\n+\tstruct conflict_info *ci; /* pi.util when there's a conflict */\n \tstruct name_entry *p;\n \tsize_t len;\n \tchar *fullpath;\n+\tconst char *dirname = opti->current_dir_name;\n \tunsigned filemask = mask & ~dirmask;\n \tunsigned match_mask = 0; /* will be updated below */\n \tunsigned mbase_null = !(mask & 1);\n@@ -157,13 +204,13 @@ static int collect_merge_info_callback(int n,\n \tmake_traverse_path(fullpath, len+1, info, p->path, p->pathlen);\n \n \t/*\n-\t * TODO: record information about the path other than all zeros,\n-\t * so we can resolve later in process_entries.\n+\t * Record information about the path so we can resolve later in\n+\t * process_entries.\n \t */\n-\tci = xcalloc(1, sizeof(struct conflict_info));\n-\tci->df_conflict = df_conflict;\n+\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n+\t\t\tnames, NULL, 0, df_conflict, filemask, dirmask, 0);\n+\tci = pi.util;\n \tci->match_mask = match_mask;\n-\tstrmap_put(&opti->paths, fullpath, ci);\n \n \t/* If dirmask, recurse into subdirectories */\n \tif (dirmask) {\n@@ -204,7 +251,7 @@ static int collect_merge_info_callback(int n,\n \t\t}\n \n \t\toriginal_dir_name = opti->current_dir_name;\n-\t\topti->current_dir_name = fullpath;\n+\t\topti->current_dir_name = pi.string;\n \t\tret = traverse_trees(NULL, 3, t, &newinfo);\n \t\topti->current_dir_name = original_dir_name;\n \n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408909","messageId":"20201102204344.342633-16-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 15/20] merge-ort: step 3 of tree writing -- handling subdirectories as we go","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:39Z","receivedAt":"2020-11-02T20:44:14Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Our order for processing of entries means that if we have a tree of\nfiles that looks like\n Makefile\n src/moduleA/foo.c\n src/moduleA/bar.c\n src/moduleB/baz.c\n src/moduleB/umm.c\n tokens.txt\n\nThen we will process paths in the order of the leftmost column below. I\nhave added two additional columns that help explain the algorithm that\nfollows; the 2nd column is there to remind us we have oid & mode info we\nare tracking for each of these paths (which differs between the paths\nwhich I'm not representing well here), and the third column annotates\nthe parent directory of the entry:\n tokens.txt \"\"\n src/moduleB/umm.c src/moduleB\n src/moduleB/baz.c src/moduleB\n src/moduleB src\n src/moduleA/foo.c src/moduleA\n src/moduleA/bar.c src/moduleA\n src/moduleA src\n src \"\"\n Makefile \"\"\n\nWhen the parent directory changes, if it's a subdirectory of the previous\nparent directory (e.g. \"\" -> src/moduleB) then we can just keep appending.\nIf the parent directory differs from the previous parent directory and is\nnot a subdirectory, then we should process that directory.\n\nSo, for example, when we get to this point:\n tokens.txt \"\"\n src/moduleB/umm.c src/moduleB\n src/moduleB/baz.c src/moduleB\n\nand note that the next entry (src/moduleB) has a different parent than\nthe last one that isn't a subdirectory, we should write out a tree for it\n 100644 blob umm.c\n 100644 blob baz.c\n\nthen pop all the entries under that directory while recording the new\nhash for that directory, leaving us with\n tokens.txt \"\"\n src/moduleB src\n\nThis process repeats until at the end we get to\n tokens.txt \"\"\n src \"\"\n Makefile \"\"\n\nand then we can write out the toplevel tree. Since we potentially have\nentries in our string_list corresponding to multiple different toplevel\ndirectories, e.g. a slightly different repository might have:\n whizbang.txt \"\"\n tokens.txt \"\"\n src/moduleD src\n src/moduleC src\n src/moduleB src\n src/moduleA/foo.c src/moduleA\n src/moduleA/bar.c src/moduleA\n\nWhen src/moduleA is popped off, we need to know that the \"last\ndirectory\" reverts back to src, and how many entries in our string_list\nare associated with that parent directory. So I use an auxiliary\noffsets string_list which would have (parent_directory,offset)\ninformation of the form\n \"\" 0\n src 2\n src/moduleA 5\n\nWhenever I write out a tree for a subdirectory, I set versions.nr to\nthe final offset value and then decrement offsets.nr...and then add\nan entry to versions with a hash for the new directory.\n\nThe idea is relatively simple, there's just a lot of accounting to\nimplement this.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 113 ++++++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 106 insertions(+), 7 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex c560dd1634..20b7c0d8b0 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -353,6 +353,9 @@ static int string_list_df_name_compare(const char *one, const char *two)\n \n struct directory_versions {\n \tstruct string_list versions;\n+\tstruct string_list offsets;\n+\tconst char *last_directory;\n+\tunsigned last_directory_len;\n };\n \n static void write_tree(struct object_id *result_oid,\n@@ -409,12 +412,100 @@ static void record_entry_for_tree(struct directory_versions *dir_metadata,\n \t\t/* nothing to record */\n \t\treturn;\n \n+\t/*\n+\t * Note: write_completed_directories() already added\n+\t * entries for directories to dir_metadata->versions,\n+\t * so no need to handle ci->filemask == 0 again.\n+\t */\n+\tif (!ci->merged.clean && !ci->filemask)\n+\t\treturn;\n+\n \tbasename = path + ci->merged.basename_offset;\n \tassert(strchr(basename, '/') == NULL);\n \tstring_list_append(&dir_metadata->versions,\n \t\t\t basename)->util = &ci->merged.result;\n }\n \n+static void write_completed_directories(struct merge_options *opt,\n+\t\t\t\t\tconst char *new_directory_name,\n+\t\t\t\t\tstruct directory_versions *info)\n+{\n+\tconst char *prev_dir;\n+\tstruct merged_info *dir_info = NULL;\n+\tunsigned int offset;\n+\tint wrote_a_new_tree = 0;\n+\n+\tif (new_directory_name == info->last_directory)\n+\t\treturn;\n+\n+\t/*\n+\t * If we are just starting (last_directory is NULL), or last_directory\n+\t * is a prefix of the current directory, then we can just update\n+\t * last_directory and record the offset where we started this directory.\n+\t */\n+\tif (info->last_directory == NULL ||\n+\t !strncmp(new_directory_name, info->last_directory,\n+\t\t info->last_directory_len)) {\n+\t\tuintptr_t offset = info->versions.nr;\n+\n+\t\tinfo->last_directory = new_directory_name;\n+\t\tinfo->last_directory_len = strlen(info->last_directory);\n+\t\tstring_list_append(&info->offsets,\n+\t\t\t\t info->last_directory)->util = (void*)offset;\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * At this point, ne (next entry) is within a different directory\n+\t * than the last entry, so we need to create a tree object for all\n+\t * the entries in info->versions that are under info->last_directory.\n+\t */\n+\tdir_info = strmap_get(&opt->priv->paths, info->last_directory);\n+\tassert(dir_info);\n+\toffset = (uintptr_t)info->offsets.items[info->offsets.nr-1].util;\n+\tif (offset == info->versions.nr) {\n+\t\tdir_info->is_null = 1;\n+\t} else {\n+\t\tdir_info->result.mode = S_IFDIR;\n+\t\twrite_tree(&dir_info->result.oid, &info->versions, offset);\n+\t\twrote_a_new_tree = 1;\n+\t}\n+\n+\t/*\n+\t * We've now used several entries from info->versions and one entry\n+\t * from info->offsets, so we get rid of those values.\n+\t */\n+\tinfo->offsets.nr--;\n+\tinfo->versions.nr = offset;\n+\n+\t/*\n+\t * Now we've got an OID for last_directory in dir_info. We need to\n+\t * add it to info->versions for it to be part of the computation of\n+\t * its parent directories' OID. But first, we have to find out what\n+\t * its' parent name was and whether that matches the previous\n+\t * info->offsets or we need to set up a new one.\n+\t */\n+\tprev_dir = info->offsets.nr == 0 ? NULL :\n+\t\t info->offsets.items[info->offsets.nr-1].string;\n+\tif (new_directory_name != prev_dir) {\n+\t\tuintptr_t c = info->versions.nr;\n+\t\tstring_list_append(&info->offsets,\n+\t\t\t\t new_directory_name)->util = (void*)c;\n+\t}\n+\n+\t/*\n+\t * Okay, finally record OID for last_directory in info->versions,\n+\t * and update last_directory.\n+\t */\n+\tif (wrote_a_new_tree) {\n+\t\tconst char *dir_name = strrchr(info->last_directory, '/');\n+\t\tdir_name = dir_name ? dir_name+1 : info->last_directory;\n+\t\tstring_list_append(&info->versions, dir_name)->util = dir_info;\n+\t}\n+\tinfo->last_directory = new_directory_name;\n+\tinfo->last_directory_len = strlen(info->last_directory);\n+}\n+\n /* Per entry merge function */\n static void process_entry(struct merge_options *opt,\n \t\t\t const char *path,\n@@ -528,6 +619,9 @@ static void process_entries(struct merge_options *opt,\n \n \t/* other setup */\n \tstring_list_init(&dir_metadata.versions, 0);\n+\tstring_list_init(&dir_metadata.offsets, 0);\n+\tdir_metadata.last_directory = NULL;\n+\tdir_metadata.last_directory_len = 0;\n \n \t/*\n \t * Iterate over the items in reverse order, so we can handle paths\n@@ -541,22 +635,27 @@ static void process_entries(struct merge_options *opt,\n \t\t */\n \t\tstruct conflict_info *ci = entry->util;\n \n+\t\twrite_completed_directories(opt, ci->merged.directory_name,\n+\t\t\t\t\t &dir_metadata);\n \t\tif (ci->merged.clean)\n \t\t\trecord_entry_for_tree(&dir_metadata, path, ci);\n \t\telse\n \t\t\tprocess_entry(opt, path, ci, &dir_metadata);\n \t}\n \n-\t/*\n-\t * TODO: We can't actually write a tree yet, because dir_metadata just\n-\t * contains all basenames of all files throughout the tree with their\n-\t * mode and hash. Not only is that a nonsensical tree, it will have\n-\t * lots of duplicates for paths such as \"Makefile\" or \".gitignore\".\n-\t */\n-\tdie(\"Not yet implemented; need to process subtrees separately\");\n+\tif (dir_metadata.offsets.nr != 1 ||\n+\t (uintptr_t)dir_metadata.offsets.items[0].util != 0) {\n+\t\tprintf(\"dir_metadata.offsets.nr = %d (should be 1)\\n\",\n+\t\t dir_metadata.offsets.nr);\n+\t\tprintf(\"dir_metadata.offsets.items[0].util = %u (should be 0)\\n\",\n+\t\t (unsigned)(uintptr_t)dir_metadata.offsets.items[0].util);\n+\t\tfflush(stdout);\n+\t\tBUG(\"dir_metadata accounting completely off; shouldn't happen\");\n+\t}\n \twrite_tree(result_oid, &dir_metadata.versions, 0);\n \tstring_list_clear(&plist, 0);\n \tstring_list_clear(&dir_metadata.versions, 0);\n+\tstring_list_clear(&dir_metadata.offsets, 0);\n }\n \n void merge_switch_to_result(struct merge_options *opt,\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408910","messageId":"20201102204344.342633-18-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 17/20] merge-ort: add implementation of checkout()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:41Z","receivedAt":"2020-11-02T20:44:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Since merge-ort creates a tree for its output, when there are no\nconflicts, updating the working tree and index is as simple as using the\nunpack_trees() machinery with a twoway_merge (i.e. doing the equivalent\nof a \"checkout\" operation).\n\nIf there were conflicts in the merge, then since the tree we created\nincluded all the conflict markers, then using the unpack_trees machinery\nin this manner will still update the working tree correctly. Further,\nall index entries corresponding to cleanly merged files will also be\nupdated correctly by this procedure. Index entries corresponding to\nunmerged entries will appear as though the user had run \"git add -u\"\nafter the merge to accept all files as-is with conflict markers.\n\nThus, after running unpack_trees(), there needs to be a separate step\nfor updating the entries in the index corresponding to unmerged files.\nThis will be the job for the function record_unmerged_index_entries(),\nwhich will be implemented in a subsequent commit.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 45 ++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 44 insertions(+), 1 deletion(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 2a60d84f1d..b7c5973d4d 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -19,9 +19,11 @@\n \n #include \"diff.h\"\n #include \"diffcore.h\"\n+#include \"dir.h\"\n #include \"object-store.h\"\n #include \"strmap.h\"\n #include \"tree.h\"\n+#include \"unpack-trees.h\"\n #include \"xdiff-interface.h\"\n \n struct merge_options_internal {\n@@ -662,7 +664,48 @@ static int checkout(struct merge_options *opt,\n \t\t struct tree *prev,\n \t\t struct tree *next)\n {\n-\tdie(\"Not yet implemented.\");\n+\t/* Switch the index/working copy from old to new */\n+\tint ret;\n+\tstruct tree_desc trees[2];\n+\tstruct unpack_trees_options unpack_opts;\n+\n+\tmemset(&unpack_opts, 0, sizeof(unpack_opts));\n+\tunpack_opts.head_idx = -1;\n+\tunpack_opts.src_index = opt->repo->index;\n+\tunpack_opts.dst_index = opt->repo->index;\n+\n+\tsetup_unpack_trees_porcelain(&unpack_opts, \"merge\");\n+\n+\t/*\n+\t * NOTE: if this were just \"git checkout\" code, we would probably\n+\t * read or refresh the cache and check for an unmerged index, but\n+\t * builtin/merge.c or sequencer.c really needs to read the index\n+\t * and check for unmerged entries before starting merging for a\n+\t * good user experience (no sense waiting for merges/rebases before\n+\t * erroring out), so there's no reason to duplicate that work here.\n+\t */\n+\n+\t/* 2-way merge to the new branch */\n+\tunpack_opts.update = 1;\n+\tunpack_opts.merge = 1;\n+\tunpack_opts.quiet = 0; /* FIXME: sequencer might want quiet? */\n+\tunpack_opts.verbose_update = (opt->verbosity > 2);\n+\tunpack_opts.fn = twoway_merge;\n+\tif (1/* FIXME: opts->overwrite_ignore*/) {\n+\t\tunpack_opts.dir = xcalloc(1, sizeof(*unpack_opts.dir));\n+\t\tunpack_opts.dir->flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(unpack_opts.dir);\n+\t}\n+\tparse_tree(prev);\n+\tinit_tree_desc(&trees[0], prev->buffer, prev->size);\n+\tparse_tree(next);\n+\tinit_tree_desc(&trees[1], next->buffer, next->size);\n+\n+\tret = unpack_trees(2, trees, &unpack_opts);\n+\tclear_unpack_trees_porcelain(&unpack_opts);\n+\tdir_clear(unpack_opts.dir);\n+\tFREE_AND_NULL(unpack_opts.dir);\n+\treturn ret;\n }\n \n static int record_unmerged_index_entries(struct merge_options *opt,\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408911","messageId":"20201102204344.342633-19-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 18/20] tree: enable cmp_cache_name_compare() to be used elsewhere","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:42Z","receivedAt":"2020-11-02T20:44:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n tree.c | 2 +-\n tree.h | 2 ++\n 2 files changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/tree.c b/tree.c\nindex e76517f6b1..a52479812c 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -144,7 +144,7 @@ int read_tree_recursive(struct repository *r,\n \treturn ret;\n }\n \n-static int cmp_cache_name_compare(const void *a_, const void *b_)\n+int cmp_cache_name_compare(const void *a_, const void *b_)\n {\n \tconst struct cache_entry *ce1, *ce2;\n \ndiff --git a/tree.h b/tree.h\nindex 9383745073..3eb0484cbf 100644\n--- a/tree.h\n+++ b/tree.h\n@@ -28,6 +28,8 @@ void free_tree_buffer(struct tree *tree);\n /* Parses and returns the tree in the given ent, chasing tags and commits. */\n struct tree *parse_tree_indirect(const struct object_id *oid);\n \n+int cmp_cache_name_compare(const void *a_, const void *b_);\n+\n #define READ_TREE_RECURSIVE 1\n typedef int (*read_tree_fn_t)(const struct object_id *, struct strbuf *, const char *, unsigned int, int, void *);\n \n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408912","messageId":"20201102204344.342633-15-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 14/20] merge-ort: step 2 of tree writing -- function to create tree object","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:38Z","receivedAt":"2020-11-02T20:44:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Create a new function, write_tree(), which will take a list of\nbasenames, modes, and oids for a single directory and create a tree\nobject in the object-store. We do not yet have just basenames, modes,\nand oids for just a single directory (we have a mixture of entries from\nall directory levels in the hierarchy) so we still die() before the\ncurrent call to write_tree(), but the next patch will rectify that.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 53 insertions(+), 1 deletion(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex ff4d455dce..c560dd1634 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -19,6 +19,7 @@\n \n #include \"diff.h\"\n #include \"diffcore.h\"\n+#include \"object-store.h\"\n #include \"strmap.h\"\n #include \"tree.h\"\n #include \"xdiff-interface.h\"\n@@ -354,6 +355,50 @@ struct directory_versions {\n \tstruct string_list versions;\n };\n \n+static void write_tree(struct object_id *result_oid,\n+\t\t struct string_list *versions,\n+\t\t unsigned int offset)\n+{\n+\tsize_t maxlen = 0;\n+\tunsigned int nr = versions->nr - offset;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct string_list relevant_entries = STRING_LIST_INIT_NODUP;\n+\tint i;\n+\n+\t/*\n+\t * We want to sort the last (versions->nr-offset) entries in versions.\n+\t * Do so by abusing the string_list API a bit: make another string_list\n+\t * that contains just those entries and then sort them.\n+\t *\n+\t * We won't use relevant_entries again and will let it just pop off the\n+\t * stack, so there won't be allocation worries or anything.\n+\t */\n+\trelevant_entries.items = versions->items + offset;\n+\trelevant_entries.nr = versions->nr - offset;\n+\tstring_list_sort(&relevant_entries);\n+\n+\t/* Pre-allocate some space in buf */\n+\tfor (i = 0; i < nr; i++) {\n+\t\tmaxlen += strlen(versions->items[offset+i].string) + 34;\n+\t}\n+\tstrbuf_reset(&buf);\n+\tstrbuf_grow(&buf, maxlen);\n+\n+\t/* Write each entry out to buf */\n+\tfor (i = 0; i < nr; i++) {\n+\t\tstruct merged_info *mi = versions->items[offset+i].util;\n+\t\tstruct version_info *ri = &mi->result;\n+\t\tstrbuf_addf(&buf, \"%o %s%c\",\n+\t\t\t ri->mode,\n+\t\t\t versions->items[offset+i].string, '\\0');\n+\t\tstrbuf_add(&buf, ri->oid.hash, the_hash_algo->rawsz);\n+\t}\n+\n+\t/* Write this object file out, and record in result_oid */\n+\twrite_object_file(buf.buf, buf.len, tree_type, result_oid);\n+\tstrbuf_release(&buf);\n+}\n+\n static void record_entry_for_tree(struct directory_versions *dir_metadata,\n \t\t\t\t const char *path,\n \t\t\t\t struct conflict_info *ci)\n@@ -502,9 +547,16 @@ static void process_entries(struct merge_options *opt,\n \t\t\tprocess_entry(opt, path, ci, &dir_metadata);\n \t}\n \n+\t/*\n+\t * TODO: We can't actually write a tree yet, because dir_metadata just\n+\t * contains all basenames of all files throughout the tree with their\n+\t * mode and hash. Not only is that a nonsensical tree, it will have\n+\t * lots of duplicates for paths such as \"Makefile\" or \".gitignore\".\n+\t */\n+\tdie(\"Not yet implemented; need to process subtrees separately\");\n+\twrite_tree(result_oid, &dir_metadata.versions, 0);\n \tstring_list_clear(&plist, 0);\n \tstring_list_clear(&dir_metadata.versions, 0);\n-\tdie(\"Tree creation not yet implemented\");\n }\n \n void merge_switch_to_result(struct merge_options *opt,\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408913","messageId":"20201102204344.342633-20-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 19/20] merge-ort: add implementation of record_unmerged_index_entries()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:43Z","receivedAt":"2020-11-02T20:44:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"After checkout(), the working tree has the appropriate contents, and the\nindex matches the working copy. That means that all unmodified and\ncleanly merged files have correct index entries, but unmerged entries\nneed to be updated.\n\nWe do this by looping over the unmerged entries, marking the existing\nindex entry for the path with CE_REMOVE, adding new higher order staged\nfor the path at the end of the index (ignoring normal index sort order),\nand then at the end of the loop removing the CE_REMOVED-marked cache\nentries and sorting the index.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 86 insertions(+), 1 deletion(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex b7c5973d4d..19c30117b0 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -17,6 +17,7 @@\n #include \"cache.h\"\n #include \"merge-ort.h\"\n \n+#include \"cache-tree.h\"\n #include \"diff.h\"\n #include \"diffcore.h\"\n #include \"dir.h\"\n@@ -713,10 +714,94 @@ static int record_unmerged_index_entries(struct merge_options *opt,\n \t\t\t\t\t struct strmap *paths,\n \t\t\t\t\t struct strmap *unmerged)\n {\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *e;\n+\tint errs = 0;\n+\tint original_cache_nr;\n+\n \tif (strmap_empty(unmerged))\n \t\treturn 0;\n \n-\tdie(\"Not yet implemented.\");\n+\toriginal_cache_nr = index->cache_nr;\n+\n+\t/* Put every entry from paths into plist, then sort */\n+\tstrmap_for_each_entry(unmerged, &iter, e) {\n+\t\tconst char *path = e->key;\n+\t\tstruct conflict_info *ci = e->value;\n+\t\tint pos;\n+\t\tstruct cache_entry *ce;\n+\t\tint i;\n+\n+\t\t/*\n+\t\t * The index will already have a stage=0 entry for this path,\n+\t\t * because we created an as-merged-as-possible version of the\n+\t\t * file and checkout() moved the working copy and index over\n+\t\t * to that version.\n+\t\t *\n+\t\t * However, previous iterations through this loop will have\n+\t\t * added unstaged entries to the end of the cache which\n+\t\t * ignore the standard alphabetical ordering of cache\n+\t\t * entries and break invariants needed for index_name_pos()\n+\t\t * to work. However, we know the entry we want is before\n+\t\t * those appended cache entries, so do a temporary swap on\n+\t\t * cache_nr to only look through entries of interest.\n+\t\t */\n+\t\tSWAP(index->cache_nr, original_cache_nr);\n+\t\tpos = index_name_pos(index, path, strlen(path));\n+\t\tSWAP(index->cache_nr, original_cache_nr);\n+\t\tif (pos < 0) {\n+\t\t\tif (ci->filemask == 1)\n+\t\t\t\tcache_tree_invalidate_path(index, path);\n+\t\t\telse\n+\t\t\t\tBUG(\"Unmerged %s but nothing in basic working tree or index; this shouldn't happen\", path);\n+\t\t} else {\n+\t\t\tce = index->cache[pos];\n+\n+\t\t\t/*\n+\t\t\t * Clean paths with CE_SKIP_WORKTREE set will not be\n+\t\t\t * written to the working tree by the unpack_trees()\n+\t\t\t * call in checkout(). Our unmerged entries would\n+\t\t\t * have appeared clean to that code since we ignored\n+\t\t\t * the higher order stages. Thus, we need override\n+\t\t\t * the CE_SKIP_WORKTREE bit and manually write those\n+\t\t\t * files to the working disk here.\n+\t\t\t *\n+\t\t\t * TODO: Implement this CE_SKIP_WORKTREE fixup.\n+\t\t\t */\n+\n+\t\t\t/*\n+\t\t\t * Mark this cache entry for removal and instead add\n+\t\t\t * new stage>0 entries corresponding to the\n+\t\t\t * conflicts. If there are many unmerged entries, we\n+\t\t\t * want to avoid memmove'ing O(NM) entries by\n+\t\t\t * inserting the new entries one at a time. So,\n+\t\t\t * instead, we just add the new cache entries to the\n+\t\t\t * end (ignoring normal index requirements on sort\n+\t\t\t * order) and sort the index once we're all done.\n+\t\t\t */\n+\t\t\tce->ce_flags |= CE_REMOVE;\n+\t\t}\n+\n+\t\tfor (i = 0; i < 3; i++) {\n+\t\t\tstruct version_info *vi;\n+\t\t\tif (!(ci->filemask & (1ul << i)))\n+\t\t\t\tcontinue;\n+\t\t\tvi = &ci->stages[i];\n+\t\t\tce = make_cache_entry(index, vi->mode, &vi->oid,\n+\t\t\t\t\t path, i+1, 0);\n+\t\t\tadd_index_entry(index, ce, ADD_CACHE_JUST_APPEND);\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Remove the unused cache entries (and invalidate the relevant\n+\t * cache-trees), then sort the index entries to get the unmerged\n+\t * entries we added to the end into their right locations.\n+\t */\n+\tremove_marked_cache_entries(index, 1);\n+\tQSORT(index->cache, index->cache_nr, cmp_cache_name_compare);\n+\n+\treturn errs;\n }\n \n void merge_switch_to_result(struct merge_options *opt,\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408914","messageId":"20201102204344.342633-21-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 20/20] merge-ort: free data structures in merge_finalize()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:44Z","receivedAt":"2020-11-02T20:44:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n merge-ort.c | 32 +++++++++++++++++++++++++++++++-\n 1 file changed, 31 insertions(+), 1 deletion(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 19c30117b0..c6a0fc388f 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -81,6 +81,16 @@ static int err(struct merge_options *opt, const char *err, ...)\n \treturn -1;\n }\n \n+static void free_strmap_strings(struct strmap *map)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\n+\tstrmap_for_each_entry(map, &iter, entry) {\n+\t\tfree((char*)entry->key);\n+\t}\n+}\n+\n static void setup_path_info(struct merge_options *opt,\n \t\t\t struct string_list_item *result,\n \t\t\t const char *current_dir_name,\n@@ -839,7 +849,27 @@ void merge_switch_to_result(struct merge_options *opt,\n void merge_finalize(struct merge_options *opt,\n \t\t struct merge_result *result)\n {\n-\tdie(\"Not yet implemented\");\n+\tstruct merge_options_internal *opti = result->priv;\n+\n+\tassert(opt->priv == NULL);\n+\n+\t/*\n+\t * We marked opti->paths with strdup_strings = 0, so that we\n+\t * wouldn't have to make another copy of the fullpath created by\n+\t * make_traverse_path from setup_path_info(). But, now that we've\n+\t * used it and have no other references to these strings, it is time\n+\t * to deallocate them.\n+\t */\n+\tfree_strmap_strings(&opti->paths);\n+\tstrmap_clear(&opti->paths, 1);\n+\n+\t/*\n+\t * All strings and util fields in opti->unmerged are a subset of\n+\t * those in opti->paths. We don't want to deallocate anything\n+\t * twice, so we don't free the strings we pass 0 for free_util.\n+\t */\n+\tstrmap_clear(&opti->unmerged, 0);\n+\tFREE_AND_NULL(opti);\n }\n \n static void merge_start(struct merge_options *opt, struct merge_result *result)\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408915","messageId":"20201102204344.342633-17-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 16/20] merge-ort: basic outline for merge_switch_to_result()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:40Z","receivedAt":"2020-11-02T20:44:23Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This adds a basic implementation for merge_switch_to_result(), though\njust in terms of a few new empty functions that will be defined in\nsubsequent commits.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 42 +++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 41 insertions(+), 1 deletion(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 20b7c0d8b0..2a60d84f1d 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -658,13 +658,53 @@ static void process_entries(struct merge_options *opt,\n \tstring_list_clear(&dir_metadata.offsets, 0);\n }\n \n+static int checkout(struct merge_options *opt,\n+\t\t struct tree *prev,\n+\t\t struct tree *next)\n+{\n+\tdie(\"Not yet implemented.\");\n+}\n+\n+static int record_unmerged_index_entries(struct merge_options *opt,\n+\t\t\t\t\t struct index_state *index,\n+\t\t\t\t\t struct strmap *paths,\n+\t\t\t\t\t struct strmap *unmerged)\n+{\n+\tif (strmap_empty(unmerged))\n+\t\treturn 0;\n+\n+\tdie(\"Not yet implemented.\");\n+}\n+\n void merge_switch_to_result(struct merge_options *opt,\n \t\t\t struct tree *head,\n \t\t\t struct merge_result *result,\n \t\t\t int update_worktree_and_index,\n \t\t\t int display_update_msgs)\n {\n-\tdie(\"Not yet implemented\");\n+\tassert(opt->priv == NULL);\n+\tif (result->clean >= 0 && update_worktree_and_index) {\n+\t\tstruct merge_options_internal *opti = result->priv;\n+\n+\t\tif (checkout(opt, head, result->tree)) {\n+\t\t\t/* failure to function */\n+\t\t\tresult->clean = -1;\n+\t\t\treturn;\n+\t\t}\n+\n+\t\tif (record_unmerged_index_entries(opt, opt->repo->index,\n+\t\t\t\t\t\t &opti->paths,\n+\t\t\t\t\t\t &opti->unmerged)) {\n+\t\t\t/* failure to function */\n+\t\t\tresult->clean = -1;\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tif (display_update_msgs) {\n+\t\t/* TODO: print out CONFLICT and other informational messages. */\n+\t}\n+\n \tmerge_finalize(opt, result);\n }\n \n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408916","messageId":"20201102204344.342633-13-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 12/20] merge-ort: have process_entries operate in a defined order","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:36Z","receivedAt":"2020-11-02T20:44:24Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"We want to handle paths below a directory before needing to handle the\ndirectory itself. Also, we want to handle the directory immediately\nafter the paths below it, so we can't use simple lexicographic ordering\nfrom strcmp (which would insert foo.txt between foo and foo/file.c).\nCopy string_list_df_name_compare() from merge-recursive.c, and set up a\nstring list of paths sorted by that function so that we can iterate in\nthe desired order.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 47 insertions(+), 2 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 92bbdc7255..3d46d62ed3 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -323,6 +323,33 @@ static int detect_and_process_renames(struct merge_options *opt,\n \treturn clean;\n }\n \n+static int string_list_df_name_compare(const char *one, const char *two)\n+{\n+\tint onelen = strlen(one);\n+\tint twolen = strlen(two);\n+\t/*\n+\t * Here we only care that entries for D/F conflicts are\n+\t * adjacent, in particular with the file of the D/F conflict\n+\t * appearing before files below the corresponding directory.\n+\t * The order of the rest of the list is irrelevant for us.\n+\t *\n+\t * To achieve this, we sort with df_name_compare and provide\n+\t * the mode S_IFDIR so that D/F conflicts will sort correctly.\n+\t * We use the mode S_IFDIR for everything else for simplicity,\n+\t * since in other cases any changes in their order due to\n+\t * sorting cause no problems for us.\n+\t */\n+\tint cmp = df_name_compare(one, onelen, S_IFDIR,\n+\t\t\t\t two, twolen, S_IFDIR);\n+\t/*\n+\t * Now that 'foo' and 'foo/bar' compare equal, we have to make sure\n+\t * that 'foo' comes before 'foo/bar'.\n+\t */\n+\tif (cmp)\n+\t\treturn cmp;\n+\treturn onelen - twolen;\n+}\n+\n /* Per entry merge function */\n static void process_entry(struct merge_options *opt,\n \t\t\t const char *path,\n@@ -413,23 +440,41 @@ static void process_entries(struct merge_options *opt,\n {\n \tstruct hashmap_iter iter;\n \tstruct strmap_entry *e;\n+\tstruct string_list plist = STRING_LIST_INIT_NODUP;\n+\tstruct string_list_item *entry;\n \n \tif (strmap_empty(&opt->priv->paths)) {\n \t\toidcpy(result_oid, opt->repo->hash_algo->empty_tree);\n \t\treturn;\n \t}\n \n+\t/* Hack to pre-allocate plist to the desired size */\n+\tALLOC_GROW(plist.items, strmap_get_size(&opt->priv->paths), plist.alloc);\n+\n+\t/* Put every entry from paths into plist, then sort */\n \tstrmap_for_each_entry(&opt->priv->paths, &iter, e) {\n+\t\tstring_list_append(&plist, e->key)->util = e->value;\n+\t}\n+\tplist.cmp = string_list_df_name_compare;\n+\tstring_list_sort(&plist);\n+\n+\t/*\n+\t * Iterate over the items in reverse order, so we can handle paths\n+\t * below a directory before needing to handle the directory itself.\n+\t */\n+\tfor (entry = &plist.items[plist.nr-1]; entry >= plist.items; --entry) {\n+\t\tchar *path = entry->string;\n \t\t/*\n \t\t * WARNING: If ci->merged.clean is true, then ci does not\n \t\t * actually point to a conflict_info but a struct merge_info.\n \t\t */\n-\t\tstruct conflict_info *ci = e->value;\n+\t\tstruct conflict_info *ci = entry->util;\n \n \t\tif (!ci->merged.clean)\n-\t\t\tprocess_entry(opt, e->key, e->value);\n+\t\t\tprocess_entry(opt, path, ci);\n \t}\n \n+\tstring_list_clear(&plist, 0);\n \tdie(\"Tree creation not yet implemented\");\n }\n \n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408917","messageId":"20201102204344.342633-14-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 13/20] merge-ort: step 1 of tree writing -- record basenames, modes, and oids","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:37Z","receivedAt":"2020-11-02T20:44:25Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"As a step towards transforming the processed path->conflict_info entries\ninto an actual tree object, start recording basenames, modes, and oids\nin a dir_metadata structure. Subsequent commits will make use of this\nto actually write a tree.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 35 ++++++++++++++++++++++++++++++++---\n 1 file changed, 32 insertions(+), 3 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 3d46d62ed3..ff4d455dce 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -350,10 +350,31 @@ static int string_list_df_name_compare(const char *one, const char *two)\n \treturn onelen - twolen;\n }\n \n+struct directory_versions {\n+\tstruct string_list versions;\n+};\n+\n+static void record_entry_for_tree(struct directory_versions *dir_metadata,\n+\t\t\t\t const char *path,\n+\t\t\t\t struct conflict_info *ci)\n+{\n+\tconst char *basename;\n+\n+\tif (ci->merged.is_null)\n+\t\t/* nothing to record */\n+\t\treturn;\n+\n+\tbasename = path + ci->merged.basename_offset;\n+\tassert(strchr(basename, '/') == NULL);\n+\tstring_list_append(&dir_metadata->versions,\n+\t\t\t basename)->util = &ci->merged.result;\n+}\n+\n /* Per entry merge function */\n static void process_entry(struct merge_options *opt,\n \t\t\t const char *path,\n-\t\t\t struct conflict_info *ci)\n+\t\t\t struct conflict_info *ci,\n+\t\t\t struct directory_versions *dir_metadata)\n {\n \tassert(!ci->merged.clean);\n \tassert(ci->filemask >= 0 && ci->filemask <= 7);\n@@ -433,6 +454,7 @@ static void process_entry(struct merge_options *opt,\n \t */\n \tif (!ci->merged.clean)\n \t\tstrmap_put(&opt->priv->unmerged, path, ci);\n+\trecord_entry_for_tree(dir_metadata, path, ci);\n }\n \n static void process_entries(struct merge_options *opt,\n@@ -442,6 +464,7 @@ static void process_entries(struct merge_options *opt,\n \tstruct strmap_entry *e;\n \tstruct string_list plist = STRING_LIST_INIT_NODUP;\n \tstruct string_list_item *entry;\n+\tstruct directory_versions dir_metadata;\n \n \tif (strmap_empty(&opt->priv->paths)) {\n \t\toidcpy(result_oid, opt->repo->hash_algo->empty_tree);\n@@ -458,6 +481,9 @@ static void process_entries(struct merge_options *opt,\n \tplist.cmp = string_list_df_name_compare;\n \tstring_list_sort(&plist);\n \n+\t/* other setup */\n+\tstring_list_init(&dir_metadata.versions, 0);\n+\n \t/*\n \t * Iterate over the items in reverse order, so we can handle paths\n \t * below a directory before needing to handle the directory itself.\n@@ -470,11 +496,14 @@ static void process_entries(struct merge_options *opt,\n \t\t */\n \t\tstruct conflict_info *ci = entry->util;\n \n-\t\tif (!ci->merged.clean)\n-\t\t\tprocess_entry(opt, path, ci);\n+\t\tif (ci->merged.clean)\n+\t\t\trecord_entry_for_tree(&dir_metadata, path, ci);\n+\t\telse\n+\t\t\tprocess_entry(opt, path, ci, &dir_metadata);\n \t}\n \n \tstring_list_clear(&plist, 0);\n+\tstring_list_clear(&dir_metadata.versions, 0);\n \tdie(\"Tree creation not yet implemented\");\n }\n \n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408918","messageId":"20201102204344.342633-8-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 07/20] merge-ort: avoid repeating fill_tree_descriptor() on the same tree","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:31Z","receivedAt":"2020-11-02T20:44:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Three-way merges, by their nature, are going to often have two or more\ntrees match at a given subdirectory. We can avoid calling\nfill_tree_descriptor() on the same tree by checking when these trees\nmatch. Noting when various oids match will also be useful in other\ncalculations and optimizations as well.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 26 ++++++++++++++++++++++----\n 1 file changed, 22 insertions(+), 4 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 626eb9713e..d3c1d00fc6 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -99,6 +99,15 @@ static int collect_merge_info_callback(int n,\n \tunsigned mbase_null = !(mask & 1);\n \tunsigned side1_null = !(mask & 2);\n \tunsigned side2_null = !(mask & 4);\n+\tunsigned side1_matches_mbase = (!side1_null && !mbase_null &&\n+\t\t\t\t\tnames[0].mode == names[1].mode &&\n+\t\t\t\t\toideq(&names[0].oid, &names[1].oid));\n+\tunsigned side2_matches_mbase = (!side2_null && !mbase_null &&\n+\t\t\t\t\tnames[0].mode == names[2].mode &&\n+\t\t\t\t\toideq(&names[0].oid, &names[2].oid));\n+\tunsigned sides_match = (!side1_null && !side2_null &&\n+\t\t\t\tnames[1].mode == names[2].mode &&\n+\t\t\t\toideq(&names[1].oid, &names[2].oid));\n \n \t/* n = 3 is a fundamental assumption. */\n \tif (n != 3)\n@@ -154,10 +163,19 @@ static int collect_merge_info_callback(int n,\n \t\tnewinfo.pathlen = st_add3(newinfo.pathlen, p->pathlen, 1);\n \n \t\tfor (i = 0; i < 3; i++, dirmask >>= 1) {\n-\t\t\tconst struct object_id *oid = NULL;\n-\t\t\tif (dirmask & 1)\n-\t\t\t\toid = &names[i].oid;\n-\t\t\tbuf[i] = fill_tree_descriptor(opt->repo, t + i, oid);\n+\t\t\tif (i == 1 && side1_matches_mbase)\n+\t\t\t\tt[1] = t[0];\n+\t\t\telse if (i == 2 && side2_matches_mbase)\n+\t\t\t\tt[2] = t[0];\n+\t\t\telse if (i == 2 && sides_match)\n+\t\t\t\tt[2] = t[1];\n+\t\t\telse {\n+\t\t\t\tconst struct object_id *oid = NULL;\n+\t\t\t\tif (dirmask & 1)\n+\t\t\t\t\toid = &names[i].oid;\n+\t\t\t\tbuf[i] = fill_tree_descriptor(opt->repo,\n+\t\t\t\t\t\t\t t + i, oid);\n+\t\t\t}\n \t\t}\n \n \t\toriginal_dir_name = opti->current_dir_name;\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408919","messageId":"20201102204344.342633-6-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 05/20] merge-ort: add an err() function similar to one from merge-recursive","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:29Z","receivedAt":"2020-11-02T20:44:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Various places in merge-recursive used an err() function when it hit\nsome kind of unrecoverable error. That code was from the reusable bits\nof merge-recursive.c that we liked, such as merge_3way, writing object\nfiles to the object store, reading blobs from the object store, etc. So\ncreate a similar function to allow us to port that code over, and use it\nfor when we detect problems returned from collect_merge_info()'s\ntraverse_trees() call, which we will be adding next.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 27 ++++++++++++++++++++++++++-\n 1 file changed, 26 insertions(+), 1 deletion(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex df97a54773..537da9f6df 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -61,11 +61,28 @@ struct conflict_info {\n \tunsigned match_mask:3;\n };\n \n+static int err(struct merge_options *opt, const char *err, ...)\n+{\n+\tva_list params;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&sb, \"error: \");\n+\tva_start(params, err);\n+\tstrbuf_vaddf(&sb, err, params);\n+\tva_end(params);\n+\n+\terror(\"%s\", sb.buf);\n+\tstrbuf_release(&sb);\n+\n+\treturn -1;\n+}\n+\n static int collect_merge_info(struct merge_options *opt,\n \t\t\t struct tree *merge_base,\n \t\t\t struct tree *side1,\n \t\t\t struct tree *side2)\n {\n+\t/* TODO: Implement this using traverse_trees() */\n \tdie(\"Not yet implemented.\");\n }\n \n@@ -167,7 +184,15 @@ static void merge_ort_nonrecursive_internal(struct merge_options *opt,\n {\n \tstruct object_id working_tree_oid;\n \n-\tcollect_merge_info(opt, merge_base, side1, side2);\n+\tif (collect_merge_info(opt, merge_base, side1, side2) != 0) {\n+\t\terr(opt, _(\"collecting merge info failed for trees %s, %s, %s\"),\n+\t\t oid_to_hex(&merge_base->object.oid),\n+\t\t oid_to_hex(&side1->object.oid),\n+\t\t oid_to_hex(&side2->object.oid));\n+\t\tresult->clean = -1;\n+\t\treturn;\n+\t}\n+\n \tresult->clean = detect_and_process_renames(opt, merge_base,\n \t\t\t\t\t\t side1, side2);\n \tprocess_entries(opt, &working_tree_oid);\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408920","messageId":"20201102204344.342633-2-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 01/20] merge-ort: setup basic internal data structures","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:25Z","receivedAt":"2020-11-02T20:46:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Set up some basic internal data structures. The only carry-over from\nmerge-recursive.c is call_depth, though needed_rename_limit will be\nadded later.\n\nThe central piece of data will definitely be the strmap \"paths\", which\nwill map every relevant pathname under consideration to either a\nmerged_info or a conflict_info. (\"unmerged\" is a strmap that is a\nsubset of \"paths\".)\n\nmerged_info contains all relevant information for a non-conflicted\nentry. conflict_info contains a merged_info, plus any additional\ninformation about a conflict such as the higher orders stages involved\nand the names of the paths those came from (handy once renames get\ninvolved). If an entry remains conflicted, the merged_info portion of a\nconflict_info will later be filled with whatever version of the file\nshould be placed in the working directory (e.g. an as-merged-as-possible\nvariation that contains conflict markers).\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 40 ++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 40 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex b487901d3e..9d5ea0930d 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -17,6 +17,46 @@\n #include \"cache.h\"\n #include \"merge-ort.h\"\n \n+#include \"strmap.h\"\n+\n+struct merge_options_internal {\n+\tstruct strmap paths; /* maps path -> (merged|conflict)_info */\n+\tstruct strmap unmerged; /* maps path -> conflict_info */\n+\tconst char *current_dir_name;\n+\tint call_depth;\n+};\n+\n+struct version_info {\n+\tstruct object_id oid;\n+\tunsigned short mode;\n+};\n+\n+struct merged_info {\n+\tstruct version_info result;\n+\tunsigned is_null:1;\n+\tunsigned clean:1;\n+\tsize_t basename_offset;\n+\t /*\n+\t * Containing directory name. Note that we assume directory_name is\n+\t * constructed such that\n+\t * strcmp(dir1_name, dir2_name) == 0 iff dir1_name == dir2_name,\n+\t * i.e. string equality is equivalent to pointer equality. For this\n+\t * to hold, we have to be careful setting directory_name.\n+\t */\n+\tconst char *directory_name;\n+};\n+\n+struct conflict_info {\n+\tstruct merged_info merged;\n+\tstruct version_info stages[3];\n+\tconst char *pathnames[3];\n+\tunsigned df_conflict:1;\n+\tunsigned path_conflict:1;\n+\tunsigned filemask:3;\n+\tunsigned dirmask:3;\n+\tunsigned match_mask:3;\n+};\n+\n void merge_switch_to_result(struct merge_options *opt,\n \t\t\t struct tree *head,\n \t\t\t struct merge_result *result,\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408921","messageId":"20201102204344.342633-3-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 02/20] merge-ort: add some high-level algorithm structure","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:26Z","receivedAt":"2020-11-02T20:46:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"merge_ort_nonrecursive_internal() will be used by both\nmerge_inmemory_nonrecursive() and merge_inmemory_recursive(); let's\nfocus on it for now. It involves some setup -- merge_start() --\nfollowed by the following chain of functions:\n\n collect_merge_info()\n This function will populate merge_options_internal's paths field,\n via a call to traverse_trees() and a new callback that will be added\n later.\n\n detect_and_process_renames()\n This function will detect renames, and then adjust entries in paths\n to move conflict stages from old pathnames into those for new\n pathnames, so that the next step doesn't have to think about renames\n and just can do three-way content merging and such.\n\n process_entries()\n This function determines how to take the various stages (versions of\n a file from the three different sides) and merge them, and whether\n to mark the result as conflicted or cleanly merged. It also writes\n out these merged file versions as it goes to create a tree.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 66 insertions(+), 1 deletion(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 9d5ea0930d..b53cd80104 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -18,6 +18,7 @@\n #include \"merge-ort.h\"\n \n #include \"strmap.h\"\n+#include \"tree.h\"\n \n struct merge_options_internal {\n \tstruct strmap paths; /* maps path -> (merged|conflict)_info */\n@@ -57,6 +58,37 @@ struct conflict_info {\n \tunsigned match_mask:3;\n };\n \n+static int collect_merge_info(struct merge_options *opt,\n+\t\t\t struct tree *merge_base,\n+\t\t\t struct tree *side1,\n+\t\t\t struct tree *side2)\n+{\n+\tdie(\"Not yet implemented.\");\n+}\n+\n+static int detect_and_process_renames(struct merge_options *opt,\n+\t\t\t\t struct tree *merge_base,\n+\t\t\t\t struct tree *side1,\n+\t\t\t\t struct tree *side2)\n+{\n+\tint clean = 1;\n+\n+\t/*\n+\t * Rename detection works by detecting file similarity. Here we use\n+\t * a really easy-to-implement scheme: files are similar IFF they have\n+\t * the same filename. Therefore, by this scheme, there are no renames.\n+\t *\n+\t * TODO: Actually implement a real rename detection scheme.\n+\t */\n+\treturn clean;\n+}\n+\n+static void process_entries(struct merge_options *opt,\n+\t\t\t struct object_id *result_oid)\n+{\n+\tdie(\"Not yet implemented.\");\n+}\n+\n void merge_switch_to_result(struct merge_options *opt,\n \t\t\t struct tree *head,\n \t\t\t struct merge_result *result,\n@@ -73,13 +105,46 @@ void merge_finalize(struct merge_options *opt,\n \tdie(\"Not yet implemented\");\n }\n \n+static void merge_start(struct merge_options *opt, struct merge_result *result)\n+{\n+\tdie(\"Not yet implemented.\");\n+}\n+\n+/*\n+ * Originally from merge_trees_internal(); heavily adapted, though.\n+ */\n+static void merge_ort_nonrecursive_internal(struct merge_options *opt,\n+\t\t\t\t\t struct tree *merge_base,\n+\t\t\t\t\t struct tree *side1,\n+\t\t\t\t\t struct tree *side2,\n+\t\t\t\t\t struct merge_result *result)\n+{\n+\tstruct object_id working_tree_oid;\n+\n+\tcollect_merge_info(opt, merge_base, side1, side2);\n+\tresult->clean = detect_and_process_renames(opt, merge_base,\n+\t\t\t\t\t\t side1, side2);\n+\tprocess_entries(opt, &working_tree_oid);\n+\n+\t/* Set return values */\n+\tresult->tree = parse_tree_indirect(&working_tree_oid);\n+\t/* existence of unmerged entries implies unclean */\n+\tresult->clean &= strmap_empty(&opt->priv->unmerged);\n+\tif (!opt->priv->call_depth) {\n+\t\tresult->priv = opt->priv;\n+\t\topt->priv = NULL;\n+\t}\n+}\n+\n void merge_incore_nonrecursive(struct merge_options *opt,\n \t\t\t struct tree *merge_base,\n \t\t\t struct tree *side1,\n \t\t\t struct tree *side2,\n \t\t\t struct merge_result *result)\n {\n-\tdie(\"Not yet implemented\");\n+\tassert(opt->ancestor != NULL);\n+\tmerge_start(opt, result);\n+\tmerge_ort_nonrecursive_internal(opt, merge_base, side1, side2, result);\n }\n \n void merge_incore_recursive(struct merge_options *opt,\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408922","messageId":"20201102204344.342633-4-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 03/20] merge-ort: port merge_start() from merge-recursive","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:27Z","receivedAt":"2020-11-02T20:46:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"merge_start() basically does a bunch of sanity checks, then allocates\nand initializes opt->priv -- a struct merge_options_internal.\n\nMost the sanity checks are usable as-is. The allocation/intialization\nis a bit different since merge-ort has a very different\nmerge_options_internal than merge-recursive, but the idea is the same.\n\nThe weirdest part here is that merge-ort and merge-recursive use the\nsame struct merge_options, even though merge_options has a number of\nfields that are oddly specific to merge-recursive's internal\nimplementation and don't even make sense with merge-ort's high-level\ndesign (e.g. buffer_output, which merge-ort has to always do). I reused\nthe same data structure because:\n * most the fields made sense to both merge algorithms\n * making a new struct would have required making new enums or somehow\n externalizing them, and that was getting messy.\n * it simplifies converting the existing callers by not having to\n have different code paths for merge_options setup.\n\nI also marked detect_renames as ignored. We can revisit that later, but\nin short: merge-recursive allowed turning off rename detection because\nit was sometimes glacially slow. When you speed something up by a few\norders of magnitude, it's worth revisiting whether that justification is\nstill relevant. Besides, if folks find it's still too slow, perhaps\nthey have a better scaling case than I could find and maybe it turns up\nsome more optimizations we can add. If it still is needed as an option,\nit is easy to add later.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 44 +++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 43 insertions(+), 1 deletion(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex b53cd80104..f5460a8a52 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -17,6 +17,8 @@\n #include \"cache.h\"\n #include \"merge-ort.h\"\n \n+#include \"diff.h\"\n+#include \"diffcore.h\"\n #include \"strmap.h\"\n #include \"tree.h\"\n \n@@ -107,7 +109,47 @@ void merge_finalize(struct merge_options *opt,\n \n static void merge_start(struct merge_options *opt, struct merge_result *result)\n {\n-\tdie(\"Not yet implemented.\");\n+\t/* Sanity checks on opt */\n+\tassert(opt->repo);\n+\n+\tassert(opt->branch1 && opt->branch2);\n+\n+\tassert(opt->detect_directory_renames >= MERGE_DIRECTORY_RENAMES_NONE &&\n+\t opt->detect_directory_renames <= MERGE_DIRECTORY_RENAMES_TRUE);\n+\tassert(opt->rename_limit >= -1);\n+\tassert(opt->rename_score >= 0 && opt->rename_score <= MAX_SCORE);\n+\tassert(opt->show_rename_progress >= 0 && opt->show_rename_progress <= 1);\n+\n+\tassert(opt->xdl_opts >= 0);\n+\tassert(opt->recursive_variant >= MERGE_VARIANT_NORMAL &&\n+\t opt->recursive_variant <= MERGE_VARIANT_THEIRS);\n+\n+\t/*\n+\t * detect_renames, verbosity, buffer_output, and obuf are ignored\n+\t * fields that were used by \"recursive\" rather than \"ort\" -- but\n+\t * sanity check them anyway.\n+\t */\n+\tassert(opt->detect_renames >= -1 &&\n+\t opt->detect_renames <= DIFF_DETECT_COPY);\n+\tassert(opt->verbosity >= 0 && opt->verbosity <= 5);\n+\tassert(opt->buffer_output <= 2);\n+\tassert(opt->obuf.len == 0);\n+\n+\tassert(opt->priv == NULL);\n+\n+\t/* Initialization of opt->priv, our internal merge data */\n+\topt->priv = xcalloc(1, sizeof(*opt->priv));\n+\t/*\n+\t * Although we initialize opt->priv->paths with strdup_strings=0,\n+\t * that's just to avoid making yet another copy of an allocated\n+\t * string. Putting the entry into paths means we are taking\n+\t * ownership, so we will later free it.\n+\t *\n+\t * In contrast, unmerged just has a subset of keys from paths, so\n+\t * we don't want to free those (it'd be a duplicate free).\n+\t */\n+\tstrmap_init_with_options(&opt->priv->paths, NULL, 0);\n+\tstrmap_init_with_options(&opt->priv->unmerged, NULL, 0);\n }\n \n /*\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408923","messageId":"20201102204344.342633-5-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 04/20] merge-ort: use histogram diff","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:28Z","receivedAt":"2020-11-02T20:46:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"I have some ideas for using a histogram diff to improve content merges,\nwhich fundamentally relies on the idea of a histogram. Since the diffs\nare never displayed to the user but just used internally for merging,\nthe typical user preference shouldn't matter anyway, and I want to make\nsure that all my testing works with this algorithm.\n\nGranted, I don't yet know if those ideas will pan out and I haven't even\ntried any of them out yet, but it's easy to change the diff algorithm in\nthe future if needed or wanted. For now, just set it to histogram.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex f5460a8a52..df97a54773 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -21,6 +21,7 @@\n #include \"diffcore.h\"\n #include \"strmap.h\"\n #include \"tree.h\"\n+#include \"xdiff-interface.h\"\n \n struct merge_options_internal {\n \tstruct strmap paths; /* maps path -> (merged|conflict)_info */\n@@ -137,6 +138,9 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \n \tassert(opt->priv == NULL);\n \n+\t/* Default to histogram diff. Actually, just hardcode it...for now. */\n+\topt->xdl_opts = DIFF_WITH_ALG(opt, HISTOGRAM_DIFF);\n+\n \t/* Initialization of opt->priv, our internal merge data */\n \topt->priv = xcalloc(1, sizeof(*opt->priv));\n \t/*\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408925","messageId":"20201102204344.342633-7-newren@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"[PATCH v2 06/20] merge-ort: implement a very basic collect_merge_info()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:30Z","receivedAt":"2020-11-02T20:50:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This does not actually collect any necessary info other than the\npathnames involved, since it just allocates an all-zero conflict_info\nand stuffs that into paths. However, it invokes the traverse_trees()\nmachinery to walk over all the paths and sets up the basic\ninfrastructure we need.\n\nI have left out a few obvious optimizations to try to make this patch as\nshort and obvious as possible. A subsequent patch will add some of\nthose back in with some more useful data fields before we introduce a\npatch that actually sets up the conflict_info fields.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 121 +++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 119 insertions(+), 2 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 537da9f6df..626eb9713e 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -77,13 +77,130 @@ static int err(struct merge_options *opt, const char *err, ...)\n \treturn -1;\n }\n \n+static int collect_merge_info_callback(int n,\n+\t\t\t\t unsigned long mask,\n+\t\t\t\t unsigned long dirmask,\n+\t\t\t\t struct name_entry *names,\n+\t\t\t\t struct traverse_info *info)\n+{\n+\t/*\n+\t * n is 3. Always.\n+\t * common ancestor (mbase) has mask 1, and stored in index 0 of names\n+\t * head of side 1 (side1) has mask 2, and stored in index 1 of names\n+\t * head of side 2 (side2) has mask 4, and stored in index 2 of names\n+\t */\n+\tstruct merge_options *opt = info->data;\n+\tstruct merge_options_internal *opti = opt->priv;\n+\tstruct conflict_info *ci;\n+\tstruct name_entry *p;\n+\tsize_t len;\n+\tchar *fullpath;\n+\tunsigned filemask = mask & ~dirmask;\n+\tunsigned mbase_null = !(mask & 1);\n+\tunsigned side1_null = !(mask & 2);\n+\tunsigned side2_null = !(mask & 4);\n+\n+\t/* n = 3 is a fundamental assumption. */\n+\tif (n != 3)\n+\t\tBUG(\"Called collect_merge_info_callback wrong\");\n+\n+\t/*\n+\t * A bunch of sanity checks verifying that traverse_trees() calls\n+\t * us the way I expect. Could just remove these at some point,\n+\t * though maybe they are helpful to future code readers.\n+\t */\n+\tassert(mbase_null == is_null_oid(&names[0].oid));\n+\tassert(side1_null == is_null_oid(&names[1].oid));\n+\tassert(side2_null == is_null_oid(&names[2].oid));\n+\tassert(!mbase_null || !side1_null || !side2_null);\n+\tassert(mask > 0 && mask < 8);\n+\n+\t/* Other invariant checks, mostly for documentation purposes. */\n+\tassert(mask == (dirmask | filemask));\n+\n+\t/*\n+\t * Get the name of the relevant filepath, which we'll pass to\n+\t * setup_path_info() for tracking.\n+\t */\n+\tp = names;\n+\twhile (!p->mode)\n+\t\tp++;\n+\tlen = traverse_path_len(info, p->pathlen);\n+\n+\t/* +1 in both of the following lines to include the NUL byte */\n+\tfullpath = xmalloc(len+1);\n+\tmake_traverse_path(fullpath, len+1, info, p->path, p->pathlen);\n+\n+\t/*\n+\t * TODO: record information about the path other than all zeros,\n+\t * so we can resolve later in process_entries.\n+\t */\n+\tci = xcalloc(1, sizeof(struct conflict_info));\n+\tstrmap_put(&opti->paths, fullpath, ci);\n+\n+\t/* If dirmask, recurse into subdirectories */\n+\tif (dirmask) {\n+\t\tstruct traverse_info newinfo;\n+\t\tstruct tree_desc t[3];\n+\t\tvoid *buf[3] = {NULL,};\n+\t\tconst char *original_dir_name;\n+\t\tint i, ret;\n+\n+\t\tci->match_mask &= filemask;\n+\t\tnewinfo = *info;\n+\t\tnewinfo.prev = info;\n+\t\tnewinfo.name = p->path;\n+\t\tnewinfo.namelen = p->pathlen;\n+\t\tnewinfo.pathlen = st_add3(newinfo.pathlen, p->pathlen, 1);\n+\n+\t\tfor (i = 0; i < 3; i++, dirmask >>= 1) {\n+\t\t\tconst struct object_id *oid = NULL;\n+\t\t\tif (dirmask & 1)\n+\t\t\t\toid = &names[i].oid;\n+\t\t\tbuf[i] = fill_tree_descriptor(opt->repo, t + i, oid);\n+\t\t}\n+\n+\t\toriginal_dir_name = opti->current_dir_name;\n+\t\topti->current_dir_name = fullpath;\n+\t\tret = traverse_trees(NULL, 3, t, &newinfo);\n+\t\topti->current_dir_name = original_dir_name;\n+\n+\t\tfor (i = 0; i < 3; i++)\n+\t\t\tfree(buf[i]);\n+\n+\t\tif (ret < 0)\n+\t\t\treturn -1;\n+\t}\n+\n+\treturn mask;\n+}\n+\n static int collect_merge_info(struct merge_options *opt,\n \t\t\t struct tree *merge_base,\n \t\t\t struct tree *side1,\n \t\t\t struct tree *side2)\n {\n-\t/* TODO: Implement this using traverse_trees() */\n-\tdie(\"Not yet implemented.\");\n+\tint ret;\n+\tstruct tree_desc t[3];\n+\tstruct traverse_info info;\n+\tchar *toplevel_dir_placeholder = \"\";\n+\n+\topt->priv->current_dir_name = toplevel_dir_placeholder;\n+\tsetup_traverse_info(&info, toplevel_dir_placeholder);\n+\tinfo.fn = collect_merge_info_callback;\n+\tinfo.data = opt;\n+\tinfo.show_all_errors = 1;\n+\n+\tparse_tree(merge_base);\n+\tparse_tree(side1);\n+\tparse_tree(side2);\n+\tinit_tree_desc(t+0, merge_base->buffer, merge_base->size);\n+\tinit_tree_desc(t+1, side1->buffer, side1->size);\n+\tinit_tree_desc(t+2, side2->buffer, side2->size);\n+\n+\tret = traverse_trees(NULL, 3, t, &info);\n+\n+\treturn ret;\n }\n \n static int detect_and_process_renames(struct merge_options *opt,\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"408926","messageId":"20201102204344.342633-1-newren@gmail.com","threadId":"54556","inReplyTo":null,"subject":"[PATCH v2 00/20] fundamentals of merge-ort implementation","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-02T20:43:24Z","receivedAt":"2020-11-02T20:51:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series depends on a merge of en/strmap (after updating to v3) and\nen/merge-ort-api-null-impl.\n\nAs promised, here's the update of the series due to the strmap\nupdates...and two other tiny updates.\n\nChanges since v1:\n * updates needed based on changes to made in v3 of strmap series\n * fixed a typo in a comment\n * tiny tweak to move a strmap_put() into setup_paths()\n\nElijah Newren (20):\n merge-ort: setup basic internal data structures\n merge-ort: add some high-level algorithm structure\n merge-ort: port merge_start() from merge-recursive\n merge-ort: use histogram diff\n merge-ort: add an err() function similar to one from merge-recursive\n merge-ort: implement a very basic collect_merge_info()\n merge-ort: avoid repeating fill_tree_descriptor() on the same tree\n merge-ort: compute a few more useful fields for collect_merge_info\n merge-ort: record stage and auxiliary info for every path\n merge-ort: avoid recursing into identical trees\n merge-ort: add a preliminary simple process_entries() implementation\n merge-ort: have process_entries operate in a defined order\n merge-ort: step 1 of tree writing -- record basenames, modes, and oids\n merge-ort: step 2 of tree writing -- function to create tree object\n merge-ort: step 3 of tree writing -- handling subdirectories as we go\n merge-ort: basic outline for merge_switch_to_result()\n merge-ort: add implementation of checkout()\n tree: enable cmp_cache_name_compare() to be used elsewhere\n merge-ort: add implementation of record_unmerged_index_entries()\n merge-ort: free data structures in merge_finalize()\n\n merge-ort.c | 929 +++++++++++++++++++++++++++++++++++++++++++++++++++-\n tree.c | 2 +-\n tree.h | 2 +\n 3 files changed, 929 insertions(+), 4 deletions(-)\n\n-- \n2.29.0.471.ga4f56089c0\n\n"},{"id":"409002","messageId":"0197d698-e966-f0bb-4d77-0183e93d9bef@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"Re: [PATCH v2 00/20] fundamentals of merge-ort implementation","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-03T14:49:58Z","receivedAt":"2020-11-03T14:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> This series depends on a merge of en/strmap (after updating to v3) and\n> en/merge-ort-api-null-impl.\n> \n> As promised, here's the update of the series due to the strmap\n> updates...and two other tiny updates.\n\nHi Elijah,\n\nI'm sorry that I've been unavailable to read and review your series\non this topic. I'm very excited about the opportunities here, and I\nwanted to take your topic and merge it with our microsoft/git fork\nso I could test the performance in a Scalar-enabled monorepo. My\nbranch is available in my fork [1]\n\n[1] https://github.com/derrickstolee/git/tree/merge-ort-vfs\n\nHowever, I'm unable to discover how to trigger your ort strategy,\neven for a simple rebase. Perhaps you could supply a recommended\ncommand for testing?\n\nThanks,\n-Stolee\n"},{"id":"409015","messageId":"CABPp-BESfpqjrskz-UQikEfk_rV_QpQybo_hStVw=K8jXHYmfA@mail.gmail.com","threadId":"54556","inReplyTo":"0197d698-e966-f0bb-4d77-0183e93d9bef@gmail.com","subject":"Re: [PATCH v2 00/20] fundamentals of merge-ort implementation","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-03T16:36:47Z","receivedAt":"2020-11-03T16:37:01Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Nov 3, 2020 at 6:50 AM Derrick Stolee wrote:\n>\n> On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> > This series depends on a merge of en/strmap (after updating to v3) and\n> > en/merge-ort-api-null-impl.\n> >\n> > As promised, here's the update of the series due to the strmap\n> > updates...and two other tiny updates.\n>\n> Hi Elijah,\n>\n> I'm sorry that I've been unavailable to read and review your series\n> on this topic. I'm very excited about the opportunities here, and I\n> wanted to take your topic and merge it with our microsoft/git fork\n> so I could test the performance in a Scalar-enabled monorepo. My\n> branch is available in my fork [1]\n>\n> [1] https://github.com/derrickstolee/git/tree/merge-ort-vfs\n>\n> However, I'm unable to discover how to trigger your ort strategy,\n> even for a simple rebase. Perhaps you could supply a recommended\n> command for testing?\n>\n> Thanks,\n> -Stolee\n\nIf you want to test performance, you shouldn't test this particular\nsubmission, you should test the end result which exists as the 'ort'\nbranch of my repo. It actually passes all the tests rather than just\ntrivial cherry-picks and rebases, and has lots (and lots) of\nperformance work that hasn't even begun at the point of the\n'ort-basics' branch. (However, it also contains some unrelated memory\ncleanup in revision.c, chdir-notify.c, and a number of other places\nbecause I was annoyed that a rebase wouldn't run valgrind-free and\nmade it harder to spot my memory leaks. And the day I went hunting\nthose memory \"leaks\", I went and grabbed some unrelated memory leaks\ntoo. If it causes you merge conflicts, let me know and I'll try to\ncreate a branch for you that hash the minimal changes outside of\nmerge-ort*.[ch] and diffcore*.[ch])\n\nAll that said, for testing either branch you just need to first set\npull.twohead=ort in your git config (see\nhttps://lore.kernel.org/git/61217a83bd7ff0ce9016eb4df9ded4fdf29a506c.1604360734.git.gitgitgadget@gmail.com/),\nor, if running regression tests, set GIT_TEST_MERGE_ALGORITHM=ort.\n"},{"id":"409301","messageId":"20201106220521.757698-1-jonathantanmy@google.com","threadId":"54556","inReplyTo":"20201102204344.342633-2-newren@gmail.com","subject":"Re: [PATCH v2 01/20] merge-ort: setup basic internal data structures","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-06T22:05:21Z","receivedAt":"2020-11-06T22:05:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"I'm not very familiar with the merge machinery, but I'll attempt a\nreview of at least the first 10 patches.\n\n> merged_info contains all relevant information for a non-conflicted\n> entry. conflict_info contains a merged_info, plus any additional\n> information about a conflict such as the higher orders stages involved\n> and the names of the paths those came from (handy once renames get\n> involved). If an entry remains conflicted, the merged_info portion of a\n> conflict_info will later be filled with whatever version of the file\n> should be placed in the working directory (e.g. an as-merged-as-possible\n> variation that contains conflict markers).\n\nI think that this information should be in the .c file.\n\n> diff --git a/merge-ort.c b/merge-ort.c\n> index b487901d3e..9d5ea0930d 100644\n> --- a/merge-ort.c\n> +++ b/merge-ort.c\n> @@ -17,6 +17,46 @@\n> #include \"cache.h\"\n> #include \"merge-ort.h\"\n> \n> +#include \"strmap.h\"\n> +\n> +struct merge_options_internal {\n> +\tstruct strmap paths; /* maps path -> (merged|conflict)_info */\n> +\tstruct strmap unmerged; /* maps path -> conflict_info */\n> +\tconst char *current_dir_name;\n> +\tint call_depth;\n> +};\n\nMaybe document if the path is from the root of the directory or just the\nfilename as it appears in a tree object?\n\nI would have expected \"unmerged\" to be a \"strset\", but I guess it's more\nconvenient for it to be a map itself. Maybe just document it as \"all\nmappings in paths wherein the value is a struct conflict_info\".\n\nThere seems to be 2 ways of referring to something that we couldn't\nmerge - \"conflicted\" (or \"having a conflict\") and \"unmerged\". Should we\nstick to one of them?\n\nAlso, looking ahead, I see that current_dir_name is used as a temporary\nvariable in the recursive calls to collect_merge_info_callback(). I\nwould prefer if current_dir_name went in the cbdata to that function\ninstead, but if that's not possible, maybe document here that\ncurrent_dir_name is only used in collect_merge_info_callback(), and\ntemporarily at that.\n\n> +struct version_info {\n> +\tstruct object_id oid;\n> +\tunsigned short mode;\n> +};\n\nOK.\n\n> +struct merged_info {\n> +\tstruct version_info result;\n> +\tunsigned is_null:1;\n> +\tunsigned clean:1;\n> +\tsize_t basename_offset;\n> +\t /*\n> +\t * Containing directory name. Note that we assume directory_name is\n> +\t * constructed such that\n> +\t * strcmp(dir1_name, dir2_name) == 0 iff dir1_name == dir2_name,\n> +\t * i.e. string equality is equivalent to pointer equality. For this\n> +\t * to hold, we have to be careful setting directory_name.\n> +\t */\n> +\tconst char *directory_name;\n> +};\n\nI'm not sure how most of the fields in this struct are to be used, but\nperhaps that will be clearer once I read the subsequent code.\n\n> +struct conflict_info {\n> +\tstruct merged_info merged;\n> +\tstruct version_info stages[3];\n> +\tconst char *pathnames[3];\n\nWhy would these be different across stages? (Rename detection?)\n\n> +\tunsigned df_conflict:1;\n\nOK.\n\n> +\tunsigned path_conflict:1;\n\nThis doesn't seem to be assigned anywhere in this patch set?\n\n> +\tunsigned filemask:3;\n> +\tunsigned dirmask:3;\n\nI wonder if this needs to be documented that the least significant bit\ncorresponds to stages[0], and so forth.\n\n> +\tunsigned match_mask:3;\n\nI think this can be derived by just looking at the stages array? Maybe\ndocument as:\n\n Optimization to track which stages match. Either 0 or at least 2 bits\n are set; if at least 2 bits are set, their corresponding stages match.\n\n> +};\n"},{"id":"409303","messageId":"20201106221934.764672-1-jonathantanmy@google.com","threadId":"54556","inReplyTo":"20201102204344.342633-7-newren@gmail.com","subject":"Re: [PATCH v2 06/20] merge-ort: implement a very basic collect_merge_info()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-06T22:19:34Z","receivedAt":"2020-11-06T22:19:43Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> diff --git a/merge-ort.c b/merge-ort.c\n> index 537da9f6df..626eb9713e 100644\n> --- a/merge-ort.c\n> +++ b/merge-ort.c\n> @@ -77,13 +77,130 @@ static int err(struct merge_options *opt, const char *err, ...)\n> \treturn -1;\n> }\n> \n> +static int collect_merge_info_callback(int n,\n> +\t\t\t\t unsigned long mask,\n> +\t\t\t\t unsigned long dirmask,\n> +\t\t\t\t struct name_entry *names,\n> +\t\t\t\t struct traverse_info *info)\n> +{\n\n[snip]\n\n> +\tunsigned mbase_null = !(mask & 1);\n> +\tunsigned side1_null = !(mask & 2);\n> +\tunsigned side2_null = !(mask & 4);\n\nShould these be \"int\"?\n\n> +\t/*\n> +\t * A bunch of sanity checks verifying that traverse_trees() calls\n> +\t * us the way I expect. Could just remove these at some point,\n> +\t * though maybe they are helpful to future code readers.\n> +\t */\n> +\tassert(mbase_null == is_null_oid(&names[0].oid));\n> +\tassert(side1_null == is_null_oid(&names[1].oid));\n> +\tassert(side2_null == is_null_oid(&names[2].oid));\n> +\tassert(!mbase_null || !side1_null || !side2_null);\n> +\tassert(mask > 0 && mask < 8);\n\nThese were helpful to me.\n\n> +\t/* Other invariant checks, mostly for documentation purposes. */\n> +\tassert(mask == (dirmask | filemask));\n\nBut not this - filemask was computed in this function, so I need not\nlook elsewhere to see that this is correct.\n\n> +\t/*\n> +\t * TODO: record information about the path other than all zeros,\n> +\t * so we can resolve later in process_entries.\n> +\t */\n> +\tci = xcalloc(1, sizeof(struct conflict_info));\n> +\tstrmap_put(&opti->paths, fullpath, ci);\n\nOK - so each entry is a full-size conflict_info to store all relevant\ninformation. Presumably some of these will be converted later into what\nis effectively a struct merged_info (so, the extra struct conflict_info\nfields are unused but memory is still occupied).\n\nI do see that in patch 10, there is an optimization that directly\nallocates the smaller struct merged_info when it is known at this point\nthat there is no conflict.\n\n[snip rest of function]\n\n> static int collect_merge_info(struct merge_options *opt,\n> \t\t\t struct tree *merge_base,\n> \t\t\t struct tree *side1,\n> \t\t\t struct tree *side2)\n> {\n> -\t/* TODO: Implement this using traverse_trees() */\n> -\tdie(\"Not yet implemented.\");\n> +\tint ret;\n> +\tstruct tree_desc t[3];\n> +\tstruct traverse_info info;\n> +\tchar *toplevel_dir_placeholder = \"\";\n> +\n> +\topt->priv->current_dir_name = toplevel_dir_placeholder;\n> +\tsetup_traverse_info(&info, toplevel_dir_placeholder);\n\nI thought that this was written like this (instead of inlining the 2\ndouble-quotes) to ensure that the string-equality-is-pointer-equality\ncharacteristic holds, but I see that that characteristic is for\ndirectory_name in struct merged_info, not current_dir_name in struct\nmerge_options_internal. Any reason for not inlining \"\"?\n\n[snip rest of function]\n"},{"id":"409305","messageId":"CABPp-BHx4w1_JAen0fupCzosu0NfoTsT9d8dEve7n5VJZ0EvBQ@mail.gmail.com","threadId":"54556","inReplyTo":"20201106220521.757698-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 01/20] merge-ort: setup basic internal data structures","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-06T22:45:02Z","receivedAt":"2020-11-06T22:45:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Nov 6, 2020 at 2:05 PM Jonathan Tan wrote:\n>\n> I'm not very familiar with the merge machinery, but I'll attempt a\n> review of at least the first 10 patches.\n\nThanks for taking a look!\n\n> > merged_info contains all relevant information for a non-conflicted\n> > entry. conflict_info contains a merged_info, plus any additional\n> > information about a conflict such as the higher orders stages involved\n> > and the names of the paths those came from (handy once renames get\n> > involved). If an entry remains conflicted, the merged_info portion of a\n> > conflict_info will later be filled with whatever version of the file\n> > should be placed in the working directory (e.g. an as-merged-as-possible\n> > variation that contains conflict markers).\n>\n> I think that this information should be in the .c file.\n\nOkay.\n\n> > diff --git a/merge-ort.c b/merge-ort.c\n> > index b487901d3e..9d5ea0930d 100644\n> > --- a/merge-ort.c\n> > +++ b/merge-ort.c\n> > @@ -17,6 +17,46 @@\n> > #include \"cache.h\"\n> > #include \"merge-ort.h\"\n> >\n> > +#include \"strmap.h\"\n> > +\n> > +struct merge_options_internal {\n> > + struct strmap paths; /* maps path -> (merged|conflict)_info */\n> > + struct strmap unmerged; /* maps path -> conflict_info */\n> > + const char *current_dir_name;\n> > + int call_depth;\n> > +};\n>\n> Maybe document if the path is from the root of the directory or just the\n> filename as it appears in a tree object?\n\nYeah, full relative path from toplevel. I'll try to add some\ndocumentation to that effect.\n\n> I would have expected \"unmerged\" to be a \"strset\", but I guess it's more\n> convenient for it to be a map itself. Maybe just document it as \"all\n> mappings in paths wherein the value is a struct conflict_info\".\n\nMakes sense. And yeah, it's not a strset just because of the simple\noptimization to avoid needing to do another lookup in paths afterwards\nto get the actual conflict_info; the only time it is used is as a loop\nover the still-unmerged entries to try to three-way merge them and\nsuch.\n\n> There seems to be 2 ways of referring to something that we couldn't\n> merge - \"conflicted\" (or \"having a conflict\") and \"unmerged\". Should we\n> stick to one of them?\n\nUhm...perhaps, but it feels like I'm going to miss some while looking\nover it. Also, there are some semantic differences at play. Since\nthe whole algorithm is divided around multiple stages --\ncollect_merge_info(), detect_and_process_renames(), process_entries(),\nas of a given early stage we might just know that we couldn't merge it\n*yet*. For example, both sides modified the entry, or one side\nmodified and the other side is missing (\"did they delete it or rename\nit?\"). After rename detection and three-way content merge, something\nthat had not been automatically mergeable as of an earlier step might\nend up being so. But we need names for stuff in the interim state.\nBut it's also possible for us to know at an early state that thing are\ndefinitely going to be a conflict regardless of what later stages do\n(e.g. both sides rename a path, but rename it differently).\n\n> Also, looking ahead, I see that current_dir_name is used as a temporary\n> variable in the recursive calls to collect_merge_info_callback(). I\n> would prefer if current_dir_name went in the cbdata to that function\n> instead, but if that's not possible, maybe document here that\n> current_dir_name is only used in collect_merge_info_callback(), and\n> temporarily at that.\n\nYeah, not possible. collect_merge_info_callback() has to be of\ntraverse_callback_t type (from tree-walk.h), which provides no extra\nparameters for extra callback data. I can add a documentation\ncomment.\n\n> > +struct version_info {\n> > + struct object_id oid;\n> > + unsigned short mode;\n> > +};\n>\n> OK.\n>\n> > +struct merged_info {\n> > + struct version_info result;\n> > + unsigned is_null:1;\n> > + unsigned clean:1;\n> > + size_t basename_offset;\n> > + /*\n> > + * Containing directory name. Note that we assume directory_name is\n> > + * constructed such that\n> > + * strcmp(dir1_name, dir2_name) == 0 iff dir1_name == dir2_name,\n> > + * i.e. string equality is equivalent to pointer equality. For this\n> > + * to hold, we have to be careful setting directory_name.\n> > + */\n> > + const char *directory_name;\n> > +};\n>\n> I'm not sure how most of the fields in this struct are to be used, but\n> perhaps that will be clearer once I read the subsequent code.\n>\n> > +struct conflict_info {\n> > + struct merged_info merged;\n> > + struct version_info stages[3];\n> > + const char *pathnames[3];\n>\n> Why would these be different across stages? (Rename detection?)\n\nYes, as noted in the portion of the commit message that you said you\nwanted in the .c file.\n\n>\n> > + unsigned df_conflict:1;\n>\n> OK.\n>\n> > + unsigned path_conflict:1;\n>\n> This doesn't seem to be assigned anywhere in this patch set?\n\nOh, right, I could drop it for now and add it back later when it is\nused. It's basically non-content conflict other than the specialized\nD/F conflict. So, things like rename/delete, moved by directory\nrename, rename/rename(1to2), and rename/add/delete. I could have\npotentially lumped it in with df_conflict or made df_conflict a\nsubset, but df_conflict is different enough from the others that it\ngot a special flag.\n\nBut yeah, since it's all rename-related stuff and this patchset\ndoesn't have any rename handling yet, I probably should have left it\nout until I added that stuff later.\n\n> > + unsigned filemask:3;\n> > + unsigned dirmask:3;\n>\n> I wonder if this needs to be documented that the least significant bit\n> corresponds to stages[0], and so forth.\n\nMaybe I should just put a comment to look at tree-walk.h? The struct\ntraverse_info has a \"fn\" member with a big comment above it describing\nmask & dirmask; filemask is just mask & ~dirmask.\n\n> > + unsigned match_mask:3;\n>\n> I think this can be derived by just looking at the stages array? Maybe\n> document as:\n>\n> Optimization to track which stages match. Either 0 or at least 2 bits\n> are set; if at least 2 bits are set, their corresponding stages match.\n\nYep, I only wanted to compute the match_mask once (I always got\nannoyed in merge-recursive.c that we were re-comparing what had\nalready been compared and computed within unpack_trees()). I like\nyour suggested comment; will add it.\n"},{"id":"409306","messageId":"20201106225211.772384-1-jonathantanmy@google.com","threadId":"54556","inReplyTo":"20201102204344.342633-9-newren@gmail.com","subject":"Re: [PATCH v2 08/20] merge-ort: compute a few more useful fields for collect_merge_info","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-06T22:52:11Z","receivedAt":"2020-11-06T22:52:17Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> +\t/*\n> +\t * Note: We only label files with df_conflict, not directories.\n> +\t * Since directories stay where they are, and files move out of the\n> +\t * way to make room for a directory, we don't care if there was a\n> +\t * directory/file conflict for a parent directory of the current path.\n> +\t */\n> +\tunsigned df_conflict = (filemask != 0) && (dirmask != 0);\n\nSuppose you have:\n\n [ours]\n foo/\n bar/\n baz\n quux\n [theirs]\n foo\n\nBy \"we only label files with df_conflict, not directories\", are you\nreferring to not labelling \"foo/\" in [ours], or to \"bar/\", \"baz\", and\n\"quux\" (so, the files and directories within a directory)? At first I\nthought you were referring to the former, but perhaps you are referring\nto the latter.\n\n> @@ -161,6 +179,13 @@ static int collect_merge_info_callback(int n,\n> \t\tnewinfo.name = p->path;\n> \t\tnewinfo.namelen = p->pathlen;\n> \t\tnewinfo.pathlen = st_add3(newinfo.pathlen, p->pathlen, 1);\n> +\t\t/*\n> +\t\t * If we did care about parent directories having a D/F\n> +\t\t * conflict, then we'd include\n> +\t\t * newinfo.df_conflicts |= (mask & ~dirmask);\n> +\t\t * here. But we don't. (See comment near setting of local\n> +\t\t * df_conflict variable near the beginning of this function).\n> +\t\t */\n\nI'm not sure how \"mask\" and \"dirmask\" contains information about parent\ndirectories. \"mask\" represents the available entries, and \"dirmask\"\nrepresents which of them are directories, as far as I know. So we can\nnotice when something is missing, but I don't see how this distinguishes\nbetween the case that something is missing because it was in a parent\ndirectory that got deleted, vs something is missing because it itself\ngot deleted.\n"},{"id":"409307","messageId":"20201106225828.774616-1-jonathantanmy@google.com","threadId":"54556","inReplyTo":"20201102204344.342633-10-newren@gmail.com","subject":"Re: [PATCH v2 09/20] merge-ort: record stage and auxiliary info for every path","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-06T22:58:28Z","receivedAt":"2020-11-06T22:58:34Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> +static void setup_path_info(struct merge_options *opt,\n> +\t\t\t struct string_list_item *result,\n> +\t\t\t const char *current_dir_name,\n> +\t\t\t int current_dir_name_len,\n> +\t\t\t char *fullpath, /* we'll take over ownership */\n> +\t\t\t struct name_entry *names,\n> +\t\t\t struct name_entry *merged_version,\n> +\t\t\t unsigned is_null, /* boolean */\n> +\t\t\t unsigned df_conflict, /* boolean */\n\nBooleans could be int, I think?\n\n> +\t\t\t unsigned filemask,\n> +\t\t\t unsigned dirmask,\n> +\t\t\t int resolved /* boolean */)\n> +{\n> +\tstruct conflict_info *path_info;\n> +\n> +\tassert(!is_null || resolved);\n> +\tassert(!df_conflict || !resolved); /* df_conflict implies !resolved */\n> +\tassert(resolved == (merged_version != NULL));\n> +\n> +\tpath_info = xcalloc(1, resolved ? sizeof(struct merged_info) :\n> +\t\t\t\t\t sizeof(struct conflict_info));\n> +\tpath_info->merged.directory_name = current_dir_name;\n> +\tpath_info->merged.basename_offset = current_dir_name_len;\n> +\tpath_info->merged.clean = !!resolved;\n> +\tif (resolved) {\n> +\t\tpath_info->merged.result.mode = merged_version->mode;\n> +\t\toidcpy(&path_info->merged.result.oid, &merged_version->oid);\n> +\t\tpath_info->merged.is_null = !!is_null;\n> +\t} else {\n> +\t\tint i;\n> +\n> +\t\tfor (i = 0; i < 3; i++) {\n> +\t\t\tpath_info->pathnames[i] = fullpath;\n> +\t\t\tpath_info->stages[i].mode = names[i].mode;\n> +\t\t\toidcpy(&path_info->stages[i].oid, &names[i].oid);\n> +\t\t}\n> +\t\tpath_info->filemask = filemask;\n> +\t\tpath_info->dirmask = dirmask;\n> +\t\tpath_info->df_conflict = !!df_conflict;\n> +\t}\n> +\tstrmap_put(&opt->priv->paths, fullpath, path_info);\n\nSo these are placed in paths but not unmerged. I'm starting to wonder if\nstruct merge_options_internal should be called merge_options_state or\nsomething, and each field having documentation about when they're used\n(or better yet, have functions like collect_merge_info() return their\ncalculations in return values (which may be \"out\" parameters) instead of\nin this struct).\n\n> +\tresult->string = fullpath;\n> +\tresult->util = path_info;\n> +}\n> +\n> static int collect_merge_info_callback(int n,\n> \t\t\t\t unsigned long mask,\n> \t\t\t\t unsigned long dirmask,\n> @@ -91,10 +136,12 @@ static int collect_merge_info_callback(int n,\n> \t */\n> \tstruct merge_options *opt = info->data;\n> \tstruct merge_options_internal *opti = opt->priv;\n> -\tstruct conflict_info *ci;\n> +\tstruct string_list_item pi; /* Path Info */\n> +\tstruct conflict_info *ci; /* pi.util when there's a conflict */\n\nLooking ahead to patch 10, this seems more like \"pi.util unless we know\nfor sure that there's no conflict\".\n"},{"id":"409308","messageId":"CABPp-BFkAu6oO5V0jRh=ExuPxBXma1F17zaq-zEaqMagc72Ccw@mail.gmail.com","threadId":"54556","inReplyTo":"20201106221934.764672-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 06/20] merge-ort: implement a very basic collect_merge_info()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-06T23:10:07Z","receivedAt":"2020-11-06T23:10:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Nov 6, 2020 at 2:19 PM Jonathan Tan wrote:\n>\n> > diff --git a/merge-ort.c b/merge-ort.c\n> > index 537da9f6df..626eb9713e 100644\n> > --- a/merge-ort.c\n> > +++ b/merge-ort.c\n> > @@ -77,13 +77,130 @@ static int err(struct merge_options *opt, const char *err, ...)\n> > return -1;\n> > }\n> >\n> > +static int collect_merge_info_callback(int n,\n> > + unsigned long mask,\n> > + unsigned long dirmask,\n> > + struct name_entry *names,\n> > + struct traverse_info *info)\n> > +{\n>\n> [snip]\n>\n> > + unsigned mbase_null = !(mask & 1);\n> > + unsigned side1_null = !(mask & 2);\n> > + unsigned side2_null = !(mask & 4);\n>\n> Should these be \"int\"?\n\nDoes the type matter, particularly since \"boolean\" isn't available?\n\n> > + /*\n> > + * A bunch of sanity checks verifying that traverse_trees() calls\n> > + * us the way I expect. Could just remove these at some point,\n> > + * though maybe they are helpful to future code readers.\n> > + */\n> > + assert(mbase_null == is_null_oid(&names[0].oid));\n> > + assert(side1_null == is_null_oid(&names[1].oid));\n> > + assert(side2_null == is_null_oid(&names[2].oid));\n> > + assert(!mbase_null || !side1_null || !side2_null);\n> > + assert(mask > 0 && mask < 8);\n>\n> These were helpful to me.\n>\n> > + /* Other invariant checks, mostly for documentation purposes. */\n> > + assert(mask == (dirmask | filemask));\n>\n> But not this - filemask was computed in this function, so I need not\n> look elsewhere to see that this is correct.\n>\n> > + /*\n> > + * TODO: record information about the path other than all zeros,\n> > + * so we can resolve later in process_entries.\n> > + */\n> > + ci = xcalloc(1, sizeof(struct conflict_info));\n> > + strmap_put(&opti->paths, fullpath, ci);\n>\n> OK - so each entry is a full-size conflict_info to store all relevant\n> information. Presumably some of these will be converted later into what\n> is effectively a struct merged_info (so, the extra struct conflict_info\n> fields are unused but memory is still occupied).\n>\n> I do see that in patch 10, there is an optimization that directly\n> allocates the smaller struct merged_info when it is known at this point\n> that there is no conflict.\n\nYep. :-)\n\n> [snip rest of function]\n>\n> > static int collect_merge_info(struct merge_options *opt,\n> > struct tree *merge_base,\n> > struct tree *side1,\n> > struct tree *side2)\n> > {\n> > - /* TODO: Implement this using traverse_trees() */\n> > - die(\"Not yet implemented.\");\n> > + int ret;\n> > + struct tree_desc t[3];\n> > + struct traverse_info info;\n> > + char *toplevel_dir_placeholder = \"\";\n> > +\n> > + opt->priv->current_dir_name = toplevel_dir_placeholder;\n> > + setup_traverse_info(&info, toplevel_dir_placeholder);\n>\n> I thought that this was written like this (instead of inlining the 2\n> double-quotes) to ensure that the string-equality-is-pointer-equality\n> characteristic holds, but I see that that characteristic is for\n> directory_name in struct merged_info, not current_dir_name in struct\n> merge_options_internal. Any reason for not inlining \"\"?\n\nYou're really digging in; I love it. From setup_path_info(), the\ndirectory_name is set from the current_dir_name:\n path_info->merged.directory_name = current_dir_name;\n(and if you follow where the current_dir_name parameter gets its value\nfrom, you find that it came indirectly from\nopt->priv->current_dir_name), so current_dir_name must meet all the\nrequirements on merge_info's directory_name field.\n\nPerhaps there's still some kind of additional simplification possible\nhere, but directory rename detection is an area that has to take some\nspecial care around this requirement. I simplified the code a little\nbit in this area as I was trying to break off a good first 20 patches\nto submit, but even if we can simplify it more, the structure is just\ngoing to come back later.\n"},{"id":"409310","messageId":"CABPp-BHOvk+W0u_MYWtH8OC-Uh5bEfsRO6QuXZKWfDQ=8fyasA@mail.gmail.com","threadId":"54556","inReplyTo":"20201106225211.772384-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 08/20] merge-ort: compute a few more useful fields for collect_merge_info","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-06T23:41:12Z","receivedAt":"2020-11-06T23:41:29Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Nov 6, 2020 at 2:52 PM Jonathan Tan wrote:\n>\n> > + /*\n> > + * Note: We only label files with df_conflict, not directories.\n> > + * Since directories stay where they are, and files move out of the\n> > + * way to make room for a directory, we don't care if there was a\n> > + * directory/file conflict for a parent directory of the current path.\n> > + */\n> > + unsigned df_conflict = (filemask != 0) && (dirmask != 0);\n>\n> Suppose you have:\n>\n> [ours]\n> foo/\n> bar/\n> baz\n> quux\n> [theirs]\n> foo\n>\n> By \"we only label files with df_conflict, not directories\", are you\n> referring to not labelling \"foo/\" in [ours], or to \"bar/\", \"baz\", and\n> \"quux\" (so, the files and directories within a directory)? At first I\n> thought you were referring to the former, but perhaps you are referring\n> to the latter.\n\nThe former. I was drawing a distinction between how this code\noperates, and how unpack_trees() operates, which probably only matters\nto those familiar with unpack_trees() or who have been reading through\nit recently. unpack_trees() will note when there is a directory/file\nconflict, and propagates that information to all subtrees, with every\npath specially checking for the o->df_conflict_entry and then handling\nit specially (e.g. keeping higher order stages instead of using an\naggressive or trivial resolutions). However, leaving both a file and\na directory at the same path, while allowed in the index, makes for\nugliness and difficulty for users to resolve. Plus it isn't allowed\nin the working tree anyway. We decided a while ago that it'd be\nbetter to represent these conflicts differently[1], [2].\n\nAlso, at the time you are unpacking or traversing trees, you only know\nif one side had a directory where the other side had a file. You\ndon't know if the final merge result will actually have a\ndirectory/file conflict. If the file existed in both the base version\nand unmodified on one side, for example, then the file will be removed\nas part of the merge. It is similarly possible that the entire\ndirectory of files all need to be deleted or are all renamed\nelsewhere. So, you have to keep track of a df_conflict bit, but you\ncan't act on it until you've processed several other things first.\n\nSince I already know I'm not going to move a whole directory of files\nout of the way so that a file can be placed in the working tree\ninstead of that whole directory, the directory doesn't need to be\ntweaked. I'm not going to propagate any information about a\ndirectory/file conflict at some path down to all subpaths of the\ndirectory. I only track it for the file that immediately conflicts,\nand then only take action on it after resolving all the paths under\nthe corresponding directory to see if the directory/file conflict\nremains.\n\n[1] https://lore.kernel.org/git/xmqqbmabcuhf.fsf@gitster-ct.c.googlers.com/\nand the thread surrounding it\n[2] https://lore.kernel.org/git/f27f12e8e50e56c010c29caa00296475d4de205b.1603731704.git.gitgitgadget@gmail.com/,\nwhich is now commit ef52778708 (\"merge tests: expect improved\ndirectory/file conflict handling in ort\", 2020-10-26)\n\n>\n> > @@ -161,6 +179,13 @@ static int collect_merge_info_callback(int n,\n> > newinfo.name = p->path;\n> > newinfo.namelen = p->pathlen;\n> > newinfo.pathlen = st_add3(newinfo.pathlen, p->pathlen, 1);\n> > + /*\n> > + * If we did care about parent directories having a D/F\n> > + * conflict, then we'd include\n> > + * newinfo.df_conflicts |= (mask & ~dirmask);\n> > + * here. But we don't. (See comment near setting of local\n> > + * df_conflict variable near the beginning of this function).\n> > + */\n>\n> I'm not sure how \"mask\" and \"dirmask\" contains information about parent\n> directories. \"mask\" represents the available entries, and \"dirmask\"\n> represents which of them are directories, as far as I know. So we can\n> notice when something is missing, but I don't see how this distinguishes\n> between the case that something is missing because it was in a parent\n> directory that got deleted, vs something is missing because it itself\n> got deleted.\n\nYeah, this is more comparisons to unpack_trees. This code is about to\nset up a recursive call into subdirectories. newinfo is set based on\nthe mask and dirmask of the current entry, and then subdirectories can\nconsult newinfo.df_conflicts to see if that path is within a directory\nthat was involved in a directory/file conflict. For example:\n\nTree in base version:\n foo/\n bar\n stuff.txt\nTree on side 1: (adds foo/baz)\n foo/\n bar\n baz\n stuff.txt\nTree on side 2: (deletes foo/, adds new file foo)\n foo\n stuff.txt\n\nWhen processing 'foo', we have mask=7, dirmask = 3. So, here\nunpack_trees() would have set newinfo.df_conflicts = (mask & ~dirmask)\n= 4. Then when we process foo/bar or foo/baz, we have mask=2,\ndirmask=0, which looks like there are no directory/file conflicts.\nHowever, we can note that these paths are under a directory involved\nin a directory/file conflict via info.df_conflicts whose value is 4.\nunpack_trees() cared about paths under a directory that was involved\nin a directory/file conflict, and someone familiar with that code\nmight ask why I don't also track the same information. The answer is\nthat I don't track it, even though I thought about it, because it's\nuseless overhead since I'm going to leave the directory alone and move\nthe file out of the way.\n\nDoes that make sense?\n"},{"id":"409311","messageId":"CABPp-BEyosjCpBr-8B19YXZV0mpn3oYAXoaaROKFNZQ+p4ZMnQ@mail.gmail.com","threadId":"54556","inReplyTo":"20201106225828.774616-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 09/20] merge-ort: record stage and auxiliary info for every path","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-07T00:26:54Z","receivedAt":"2020-11-07T00:27:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Nov 6, 2020 at 2:58 PM Jonathan Tan wrote:\n>\n> > +static void setup_path_info(struct merge_options *opt,\n> > + struct string_list_item *result,\n> > + const char *current_dir_name,\n> > + int current_dir_name_len,\n> > + char *fullpath, /* we'll take over ownership */\n> > + struct name_entry *names,\n> > + struct name_entry *merged_version,\n> > + unsigned is_null, /* boolean */\n> > + unsigned df_conflict, /* boolean */\n>\n> Booleans could be int, I think?\n\nI guess this goes back to the question on patch 6 where you suggested\nI mark some unsigned variables (derived from bit math on other\nunsigned quantities) instead be int. I guess it could, but I have the\nsame question; since \"boolean\" isn't available in C, does int vs.\nunsigned matter?\n\n> > + unsigned filemask,\n> > + unsigned dirmask,\n> > + int resolved /* boolean */)\n> > +{\n> > + struct conflict_info *path_info;\n> > +\n> > + assert(!is_null || resolved);\n> > + assert(!df_conflict || !resolved); /* df_conflict implies !resolved */\n> > + assert(resolved == (merged_version != NULL));\n> > +\n> > + path_info = xcalloc(1, resolved ? sizeof(struct merged_info) :\n> > + sizeof(struct conflict_info));\n> > + path_info->merged.directory_name = current_dir_name;\n> > + path_info->merged.basename_offset = current_dir_name_len;\n> > + path_info->merged.clean = !!resolved;\n> > + if (resolved) {\n> > + path_info->merged.result.mode = merged_version->mode;\n> > + oidcpy(&path_info->merged.result.oid, &merged_version->oid);\n> > + path_info->merged.is_null = !!is_null;\n> > + } else {\n> > + int i;\n> > +\n> > + for (i = 0; i < 3; i++) {\n> > + path_info->pathnames[i] = fullpath;\n> > + path_info->stages[i].mode = names[i].mode;\n> > + oidcpy(&path_info->stages[i].oid, &names[i].oid);\n> > + }\n> > + path_info->filemask = filemask;\n> > + path_info->dirmask = dirmask;\n> > + path_info->df_conflict = !!df_conflict;\n> > + }\n> > + strmap_put(&opt->priv->paths, fullpath, path_info);\n>\n> So these are placed in paths but not unmerged. I'm starting to wonder if\n> struct merge_options_internal should be called merge_options_state or\n> something, and each field having documentation about when they're used\n> (or better yet, have functions like collect_merge_info() return their\n> calculations in return values (which may be \"out\" parameters) instead of\n> in this struct).\n\nRight, unmerged is only those paths that remain unmerged after all\nsteps. record_unmerged_index_entries() could simply walk over all\nentries in paths and pick out the ones that were unmerged, but\nprocess_entries() has to walk over all paths, determine whether they\ncan be merged, and determine what to record for the resulting tree for\neach path. So, having it stash away the unmerged stuff is a simple\noptimization.\n\nRenaming to merge_options_state or even just merge_state would be fine\n-- but any renaming done here will also affect merge-recursive.[ch].\nSee the definition of merge_options in merge-recursive. (For history,\nmerge-recursive.h stuffed state into merge_options, which risked funny\nmisusage patterns and made the API unnecessarily complex...and made it\nsuggest that alternative algorithms needed to have the same state.\nSo, the state was moved to a merge_options_internal struct. That's\nnot to say we can't rename, but it does need to be done in\nmerge-recursive as well.)\n\nAs for having collect_merge_info() return their calculations in return\nvalues, would that just end with me returning a struct\nmerge_options_internal? Or did you want each return value added to\nthe function signature? Each return value in the function signature\nmakes sense right now for this super-simplified initial 20 patches,\nbut what about when this data structure gains all kind of\nrename-related state that is collected, updated, and passed between\nthese areas? I'd have a huge number of \"out\" and \"in\" fields to every\nfunction. Eventually, merge_options_internal (or whatever it might be\nrenamed to) expands to the following, where I have to first define an\nextra enum and two extra structs so that you know the definitions of\nnew types that show up in merge_options_internal:\n\nenum relevance {\n RELEVANT_NO_MORE = 0,\n RELEVANT_CONTENT = 1,\n RELEVANT_LOCATION = 2,\n RELEVANT_BOTH = 3\n};\n\nstruct traversal_callback_data {\n unsigned long mask;\n unsigned long dirmask;\n struct name_entry names[3];\n};\n\nstruct rename_info {\n /* For the next six vars, the 0th entry is ignored and unused */\n struct diff_queue_struct pairs[3]; /* input to & output from\ndiffcore_rename */\n struct strintmap relevant_sources[3]; /* filepath => enum relevance */\n struct strintmap dirs_removed[3]; /* directory => bool */\n struct strmap dir_rename_count[3]; /* old_dir => {new_dir => int} */\n struct strintmap possible_trivial_merges[3]; /* dirname->dir_rename_mask */\n struct strset target_dirs[3]; /* set of directory paths */\n unsigned trivial_merges_okay[3]; /* 0 = no, 1 = maybe */\n /*\n * dir_rename_mask:\n * 0: optimization removing unmodified potential rename source okay\n * 2 or 4: optimization okay, but must check for files added to dir\n * 7: optimization forbidden; need rename source in case of dir rename\n */\n unsigned dir_rename_mask:3;\n\n /*\n * dir_rename_mask needs to be coupled with a traversal through trees\n * that iterates over all files in a given tree before all immediate\n * subdirectories within that tree. Since traverse_trees() doesn't do\n * that naturally, we have a traverse_trees_wrapper() that stores any\n * immediate subdirectories while traversing files, then traverses the\n * immediate subdirectories later.\n */\n struct traversal_callback_data *callback_data;\n int callback_data_nr, callback_data_alloc;\n char *callback_data_traverse_path;\n\n /*\n * When doing repeated merges, we can re-use renaming information from\n * previous merges under special circumstances;\n */\n struct tree *merge_trees[3];\n int cached_pairs_valid_side;\n struct strmap cached_pairs[3]; /* fullnames -> {rename_path or NULL} */\n struct strset cached_irrelevant[3]; /* fullnames */\n struct strset cached_target_names[3]; /* set of target fullnames */\n /*\n * And sometimes it pays to detect renames, and then restart the merge\n * with the renames cached so that we can do trivial tree merging.\n * Values: 0 = don't bother, 1 = let's do it, 2 = we already did it.\n */\n unsigned redo_after_renames;\n};\n\nstruct merge_options_internal {\n struct strmap paths; /* maps path -> (merged|conflict)_info */\n struct strmap unmerged; /* maps path -> conflict_info */\n#if USE_MEMORY_POOL\n struct mem_pool pool;\n#else\n struct string_list paths_to_free; /* list of strings to free */\n#endif\n struct rename_info *renames;\n struct index_state attr_index; /* renormalization weirdly needs one... */\n struct strmap output; /* maps path -> conflict messages */\n const char *current_dir_name;\n char *toplevel_dir; /* see merge_info.directory_name comment */\n int call_depth;\n int needed_rename_limit;\n};\n\n\n> > + result->string = fullpath;\n> > + result->util = path_info;\n> > +}\n> > +\n> > static int collect_merge_info_callback(int n,\n> > unsigned long mask,\n> > unsigned long dirmask,\n> > @@ -91,10 +136,12 @@ static int collect_merge_info_callback(int n,\n> > */\n> > struct merge_options *opt = info->data;\n> > struct merge_options_internal *opti = opt->priv;\n> > - struct conflict_info *ci;\n> > + struct string_list_item pi; /* Path Info */\n> > + struct conflict_info *ci; /* pi.util when there's a conflict */\n>\n> Looking ahead to patch 10, this seems more like \"pi.util unless we know\n> for sure that there's no conflict\".\n\nThat's too long for the line to remain at 80 characters; it's 16\ncharacters over the limit. ;-)\n"},{"id":"409317","messageId":"CABPp-BGvgdK8vEu5+mHjy9=GPg-aVtforC4-AAYgs3RWkBwQJA@mail.gmail.com","threadId":"54556","inReplyTo":"CABPp-BESfpqjrskz-UQikEfk_rV_QpQybo_hStVw=K8jXHYmfA@mail.gmail.com","subject":"Re: [PATCH v2 00/20] fundamentals of merge-ort implementation","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-07T06:06:14Z","receivedAt":"2020-11-07T06:26:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Derrick,\n\nOn Tue, Nov 3, 2020 at 8:36 AM Elijah Newren wrote:\n>\n> On Tue, Nov 3, 2020 at 6:50 AM Derrick Stolee wrote:\n> >\n> > On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> > > This series depends on a merge of en/strmap (after updating to v3) and\n> > > en/merge-ort-api-null-impl.\n> > >\n> > > As promised, here's the update of the series due to the strmap\n> > > updates...and two other tiny updates.\n> >\n> > Hi Elijah,\n> >\n> > I'm sorry that I've been unavailable to read and review your series\n> > on this topic. I'm very excited about the opportunities here, and I\n> > wanted to take your topic and merge it with our microsoft/git fork\n> > so I could test the performance in a Scalar-enabled monorepo. My\n> > branch is available in my fork [1]\n> >\n> > [1] https://github.com/derrickstolee/git/tree/merge-ort-vfs\n> >\n> > However, I'm unable to discover how to trigger your ort strategy,\n> > even for a simple rebase. Perhaps you could supply a recommended\n> > command for testing?\n> >\n> > Thanks,\n> > -Stolee\n>\n> If you want to test performance, you shouldn't test this particular\n> submission, you should test the end result which exists as the 'ort'\n> branch of my repo. It actually passes all the tests rather than just\n> trivial cherry-picks and rebases, and has lots (and lots) of\n> performance work that hasn't even begun at the point of the\n> 'ort-basics' branch. (However, it also contains some unrelated memory\n> cleanup in revision.c, chdir-notify.c, and a number of other places\n> because I was annoyed that a rebase wouldn't run valgrind-free and\n> made it harder to spot my memory leaks. And the day I went hunting\n> those memory \"leaks\", I went and grabbed some unrelated memory leaks\n> too. If it causes you merge conflicts, let me know and I'll try to\n> create a branch for you that hash the minimal changes outside of\n> merge-ort*.[ch] and diffcore*.[ch])\n>\n> All that said, for testing either branch you just need to first set\n> pull.twohead=ort in your git config (see\n> https://lore.kernel.org/git/61217a83bd7ff0ce9016eb4df9ded4fdf29a506c.1604360734.git.gitgitgadget@gmail.com/),\n> or, if running regression tests, set GIT_TEST_MERGE_ALGORITHM=ort.\n\nI probably also should have mentioned that merge-ort does not (yet?)\nheed merge.renames configuration setting; it always detects renames.\nI know you run with merge.renames=false, so you won't quite get an\napples-to-apples comparison. However, part of my point was I wanted\nto make renames fast enough that they could be left turned on, even\nfor the large scale repos, so I'm very interested in your experience.\nIf you need an escape hatch, though, just put a \"return 1\" at the top\nof detect_and_process_renames() to turn it off.\n\nOh, and I went through and re-merged all the merge commits in the\nlinux kernel and found a bug in merge-ort while doing that (causing it\nto die, not to merge badly). I'm kind of surprised that none of my\ntestcases triggered that failure earlier; if you're testing it out,\nyou might want to update to get the fix (commit 067e5c1a38,\n\"merge-ort: fix bug with cached_target_names not being initialized in\nredos\", 2020-11-06).\n"},{"id":"409322","messageId":"ca61acbf-1757-5ddc-49f7-6da0cba4a647@gmail.com","threadId":"54556","inReplyTo":"CABPp-BGvgdK8vEu5+mHjy9=GPg-aVtforC4-AAYgs3RWkBwQJA@mail.gmail.com","subject":"Re: [PATCH v2 00/20] fundamentals of merge-ort implementation","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-07T15:02:35Z","receivedAt":"2020-11-07T15:02:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/7/20 1:06 AM, Elijah Newren wrote:\n> Hi Derrick,\n> \n> On Tue, Nov 3, 2020 at 8:36 AM Elijah Newren wrote:\n>> All that said, for testing either branch you just need to first set\n>> pull.twohead=ort in your git config (see\n>> https://lore.kernel.org/git/61217a83bd7ff0ce9016eb4df9ded4fdf29a506c.1604360734.git.gitgitgadget@gmail.com/),\n>> or, if running regression tests, set GIT_TEST_MERGE_ALGORITHM=ort.\n> \n> I probably also should have mentioned that merge-ort does not (yet?)\n> heed merge.renames configuration setting; it always detects renames.\n> I know you run with merge.renames=false, so you won't quite get an\n> apples-to-apples comparison. However, part of my point was I wanted\n> to make renames fast enough that they could be left turned on, even\n> for the large scale repos, so I'm very interested in your experience.\n> If you need an escape hatch, though, just put a \"return 1\" at the top\n> of detect_and_process_renames() to turn it off.\n> \n> Oh, and I went through and re-merged all the merge commits in the\n> linux kernel and found a bug in merge-ort while doing that (causing it\n> to die, not to merge badly). I'm kind of surprised that none of my\n> testcases triggered that failure earlier; if you're testing it out,\n> you might want to update to get the fix (commit 067e5c1a38,\n> \"merge-ort: fix bug with cached_target_names not being initialized in\n> redos\", 2020-11-06).\n\nI did manage to do some testing to see what happens with\na large repo under a small sparse-checkout. And using\ntrace2, I was able to see that your code is being exercised.\nUnfortunately, I didn't see any performance improvement, and\nthat is likely due to needing to expand the index entirely\nwhen checking out the merge commit.\n\nIs there a command to construct a merge commit without\nactually checking it out? That would reduce the time spent\nexpanding the index, which would allow your algorithm to\nreally show its benefits!\n\nThanks,\n-Stolee\n"},{"id":"409324","messageId":"CABPp-BEzZ18KfX0kgOja+yeXbuEZmMORWjGePUMTdyzvqXLkFg@mail.gmail.com","threadId":"54556","inReplyTo":"ca61acbf-1757-5ddc-49f7-6da0cba4a647@gmail.com","subject":"Re: [PATCH v2 00/20] fundamentals of merge-ort implementation","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-07T19:39:39Z","receivedAt":"2020-11-07T19:39:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Nov 7, 2020 at 7:02 AM Derrick Stolee wrote:\n>\n> On 11/7/20 1:06 AM, Elijah Newren wrote:\n> > Hi Derrick,\n> >\n> > On Tue, Nov 3, 2020 at 8:36 AM Elijah Newren wrote:\n> >> All that said, for testing either branch you just need to first set\n> >> pull.twohead=ort in your git config (see\n> >> https://lore.kernel.org/git/61217a83bd7ff0ce9016eb4df9ded4fdf29a506c.1604360734.git.gitgitgadget@gmail.com/),\n> >> or, if running regression tests, set GIT_TEST_MERGE_ALGORITHM=ort.\n> >\n> > I probably also should have mentioned that merge-ort does not (yet?)\n> > heed merge.renames configuration setting; it always detects renames.\n> > I know you run with merge.renames=false, so you won't quite get an\n> > apples-to-apples comparison. However, part of my point was I wanted\n> > to make renames fast enough that they could be left turned on, even\n> > for the large scale repos, so I'm very interested in your experience.\n> > If you need an escape hatch, though, just put a \"return 1\" at the top\n> > of detect_and_process_renames() to turn it off.\n> >\n> > Oh, and I went through and re-merged all the merge commits in the\n> > linux kernel and found a bug in merge-ort while doing that (causing it\n> > to die, not to merge badly). I'm kind of surprised that none of my\n> > testcases triggered that failure earlier; if you're testing it out,\n> > you might want to update to get the fix (commit 067e5c1a38,\n> > \"merge-ort: fix bug with cached_target_names not being initialized in\n> > redos\", 2020-11-06).\n>\n> I did manage to do some testing to see what happens with\n> a large repo under a small sparse-checkout. And using\n> trace2, I was able to see that your code is being exercised.\n> Unfortunately, I didn't see any performance improvement, and\n> that is likely due to needing to expand the index entirely\n> when checking out the merge commit.\n>\n> Is there a command to construct a merge commit without\n> actually checking it out? That would reduce the time spent\n> expanding the index, which would allow your algorithm to\n> really show its benefits!\n\nWow, very interesting. I am working on a --remerge-diff option for\nlog, which implies -p and is similar to -c or --cc in that it makes\nmerge commits show a diff, but which in particular remerges the two\nparent commits complete with conflict markers and such and then diffs\nthe merge commit against that intermediate remerge. That's a case\nthat constructs a merge commit without ever touching the index (or\nworking tree)...but there's no equivalent comparison point for\nmerge-recursive. So, it doesn't provide something to compare against\n(and while the code can be used I don't actually have a --remerge-diff\noption yet -- it just hardcodes the behavior on if wanted or not), so\nI'm not sure if you'd be interested in it. If you are, let me know\nthough, and I'll send details.\n\nHowever, I'm really surprised here, because merge-recursive always\nreads and writes the index too (the index is the basis for its whole\nalgorithm). In fact, merge-recursive always reads the index at least\n*twice* (it unconditionally discards and re-reads the index), so you\nmust have some kind of specialized tweaking of merge-recursive if it\nsomehow avoids a full index read/write. In order to do an\napples-to-apples comparison, we'd need to make those same tweaks to\nmerge-ort, but I don't have a clue what kind of tweaks you've made\nhere. So, some investigation points:\n\n*1*. Could you give me the accumulated times from the trace2_regions\nso we can verify where the time is spent? The 'summarize-perf' script\nat the toplevel of the repo in my ort branch might be helpful for\nthis; just prefix any git command with that script and it accumulates\nthe trace2 region times and prints them out. For example, I could run\n'summarize-perf git merge --no-edit B^0' or 'summarize-perf test-tool\nfast-rebase --onto HEAD ca76bea9 myfeature'. Here's an example:\n\n=== BEGIN OUTPUT ===\n$ /home/newren/floss/git/summarize-perf test-tool fast-rebase --onto\nHEAD 4703d9119972bf586d2cca76ec6438f819ffa30e hwmon-updates\nRebasing fd8bdb23b91876ac1e624337bb88dc1dcc21d67e...\nDone.\nAccumulated times:\n 0.031 : ( 3.2%)\n 0.837 : 35 : label:incore_nonrecursive\n 0.003 : ( 0.4%)\n 0.476 : 41 : ..label:collect_merge_info\n 0.001 : ( 0.2%)\n 0.475 : 41 : ....label:traverse_trees\n 0.298 : 41 : ..label:renames\n 0.015 : ( 5.1%)\n 0.280 : 41 : ....label:regular renames\n 0.036 : (12.7%)\n 0.244 : 6 : ......label:diffcore_rename\n 0.001 : ( 0.4%)\n 0.078 : 6 : ........label:dir rename setup\n 0.055 : 6 : ........label:basename matches\n 0.051 : 6 : ........label:exact renames\n 0.031 : 6 : ........label:write back to queue\n 0.017 : 6 : ........label:setup\n 0.009 : 6 : ........label:cull basename\n 0.003 : 6 : ........label:cull exact\n 0.002 : 35 : ....label:directory renames\n 0.001 : 35 : ....label:process renames\n 0.052 : 35 : ..label:process_entries\n 0.001 : ( 1.7%)\n 0.033 : 35 : ....label:processing\n 0.017 : 35 : ....label:process_entries setup\n 0.001 : ( 5.8%)\n 0.008 : 35 : ......label:plist copy\n 0.008 : 35 : ......label:plist sort\n 0.000 : 35 : ......label:plist grow\n 0.001 : 35 : ....label:finalize\n 0.005 : 35 : ..label:merge_start\n 0.001 : (18.8%)\n 0.004 : 34 : ....label:reset_maps\n 0.000 : 35 : ....label:sanity checks\n 0.000 : 1 : ....label:allocate/init\n 0.003 : 6 : ..label:reset_maps\n 0.035 : 1 : label:do_write_index\n/home/newren/floss/linux-stable/.git/index.lock\n 0.034 : 1 : label:checkout\n 0.034 : (99.9%)\n 0.000 : 1 : ..label:Filtering content\n 0.009 : 1 : label:do_read_index .git/index\n 0.000 : 1 : label:write_auto_merge\n 0.000 : 1 : label:record_unmerged\nEstimated measurement overhead (.010 ms/region-measure * 679):\n0.006790000000000001\nTiming including forking: 0.960 (0.013 additional seconds)\n=== END OUTPUT ===\nThis was a run that took just under 1s (and was a hot-cache case; I\nhad just done the same rebase before to warm the caches), and the\ncombination of index/working tree bits (everything at and after\ndo_write_index in the output) was 0.035+0.034+0.009+0+0=0.078 seconds,\ncorresponding to just over 8.1% of overall time. I'm curious where\nthat lands for your repository testcase; if the larger time ends up\nsomewhere under the indented label:incore_nonrecursive region, then\nit's due to something other than index reading/updating/writing.\n\n*2*. If it really is due to index reading/updating/writing, then index\nhandling in merge-ort is confined to two functions: checkout() and\nrecord_unmerged_index_entries(). Both functions aren't too long, and\nneither one calls into any other function within merge-ort.c.\n(Further, checkout() is a near copy of code from merge_working_tree()\nin builtin/checkout.c, or at least a copy of that function from a year\nor so ago.) As such, it's possible you can go in and make whatever\nspecial tweaks you have for partial index reading/writing to those\nfunctions.\n\nI'm curious to hear back more on this.\n\nElijah\n"},{"id":"409391","messageId":"ee5cb3fe-4983-deba-2dba-b11a74adfbde@gmail.com","threadId":"54556","inReplyTo":"CABPp-BEzZ18KfX0kgOja+yeXbuEZmMORWjGePUMTdyzvqXLkFg@mail.gmail.com","subject":"Re: [PATCH v2 00/20] fundamentals of merge-ort implementation","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-09T12:30:33Z","receivedAt":"2020-11-09T12:30:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/7/2020 2:39 PM, Elijah Newren wrote:\n> On Sat, Nov 7, 2020 at 7:02 AM Derrick Stolee wrote:\n>>\n>> On 11/7/20 1:06 AM, Elijah Newren wrote:\n>>> Hi Derrick,\n>>>\n>>> On Tue, Nov 3, 2020 at 8:36 AM Elijah Newren wrote:\n>>>> All that said, for testing either branch you just need to first set\n>>>> pull.twohead=ort in your git config (see\n>>>> https://lore.kernel.org/git/61217a83bd7ff0ce9016eb4df9ded4fdf29a506c.1604360734.git.gitgitgadget@gmail.com/),\n>>>> or, if running regression tests, set GIT_TEST_MERGE_ALGORITHM=ort.\n>>>\n>>> I probably also should have mentioned that merge-ort does not (yet?)\n>>> heed merge.renames configuration setting; it always detects renames.\n>>> I know you run with merge.renames=false, so you won't quite get an\n>>> apples-to-apples comparison. However, part of my point was I wanted\n>>> to make renames fast enough that they could be left turned on, even\n>>> for the large scale repos, so I'm very interested in your experience.\n>>> If you need an escape hatch, though, just put a \"return 1\" at the top\n>>> of detect_and_process_renames() to turn it off.\n>>>\n>>> Oh, and I went through and re-merged all the merge commits in the\n>>> linux kernel and found a bug in merge-ort while doing that (causing it\n>>> to die, not to merge badly). I'm kind of surprised that none of my\n>>> testcases triggered that failure earlier; if you're testing it out,\n>>> you might want to update to get the fix (commit 067e5c1a38,\n>>> \"merge-ort: fix bug with cached_target_names not being initialized in\n>>> redos\", 2020-11-06).\n>>\n>> I did manage to do some testing to see what happens with\n>> a large repo under a small sparse-checkout. And using\n>> trace2, I was able to see that your code is being exercised.\n>> Unfortunately, I didn't see any performance improvement, and\n>> that is likely due to needing to expand the index entirely\n>> when checking out the merge commit.\n>>\n>> Is there a command to construct a merge commit without\n>> actually checking it out? That would reduce the time spent\n>> expanding the index, which would allow your algorithm to\n>> really show its benefits!\n> \n> Wow, very interesting. I am working on a --remerge-diff option for\n> log, which implies -p and is similar to -c or --cc in that it makes\n> merge commits show a diff, but which in particular remerges the two\n> parent commits complete with conflict markers and such and then diffs\n> the merge commit against that intermediate remerge. That's a case\n> that constructs a merge commit without ever touching the index (or\n> working tree)...but there's no equivalent comparison point for\n> merge-recursive. So, it doesn't provide something to compare against\n> (and while the code can be used I don't actually have a --remerge-diff\n> option yet -- it just hardcodes the behavior on if wanted or not), so\n> I'm not sure if you'd be interested in it. If you are, let me know\n> though, and I'll send details.\n> \n> However, I'm really surprised here, because merge-recursive always\n> reads and writes the index too (the index is the basis for its whole\n> algorithm). In fact, merge-recursive always reads the index at least\n> *twice* (it unconditionally discards and re-reads the index), so you\n> must have some kind of specialized tweaking of merge-recursive if it\n> somehow avoids a full index read/write. In order to do an\n> apples-to-apples comparison, we'd need to make those same tweaks to\n> merge-ort, but I don't have a clue what kind of tweaks you've made\n> here. So, some investigation points:\n> \n> *1*. Could you give me the accumulated times from the trace2_regions\n> so we can verify where the time is spent? The 'summarize-perf' script\n> at the toplevel of the repo in my ort branch might be helpful for\n> this; just prefix any git command with that script and it accumulates\n> the trace2 region times and prints them out. For example, I could run\n> 'summarize-perf git merge --no-edit B^0' or 'summarize-perf test-tool\n> fast-rebase --onto HEAD ca76bea9 myfeature'. Here's an example:\n> \n> === BEGIN OUTPUT ===\n> $ /home/newren/floss/git/summarize-perf test-tool fast-rebase --onto\n> HEAD 4703d9119972bf586d2cca76ec6438f819ffa30e hwmon-updates\n> Rebasing fd8bdb23b91876ac1e624337bb88dc1dcc21d67e...\n> Done.\n> Accumulated times:\n> 0.031 : ( 3.2%)\n> 0.837 : 35 : label:incore_nonrecursive\n> 0.003 : ( 0.4%)\n> 0.476 : 41 : ..label:collect_merge_info\n> 0.001 : ( 0.2%)\n> 0.475 : 41 : ....label:traverse_trees\n> 0.298 : 41 : ..label:renames\n> 0.015 : ( 5.1%)\n> 0.280 : 41 : ....label:regular renames\n> 0.036 : (12.7%)\n> 0.244 : 6 : ......label:diffcore_rename\n> 0.001 : ( 0.4%)\n> 0.078 : 6 : ........label:dir rename setup\n> 0.055 : 6 : ........label:basename matches\n> 0.051 : 6 : ........label:exact renames\n> 0.031 : 6 : ........label:write back to queue\n> 0.017 : 6 : ........label:setup\n> 0.009 : 6 : ........label:cull basename\n> 0.003 : 6 : ........label:cull exact\n> 0.002 : 35 : ....label:directory renames\n> 0.001 : 35 : ....label:process renames\n> 0.052 : 35 : ..label:process_entries\n> 0.001 : ( 1.7%)\n> 0.033 : 35 : ....label:processing\n> 0.017 : 35 : ....label:process_entries setup\n> 0.001 : ( 5.8%)\n> 0.008 : 35 : ......label:plist copy\n> 0.008 : 35 : ......label:plist sort\n> 0.000 : 35 : ......label:plist grow\n> 0.001 : 35 : ....label:finalize\n> 0.005 : 35 : ..label:merge_start\n> 0.001 : (18.8%)\n> 0.004 : 34 : ....label:reset_maps\n> 0.000 : 35 : ....label:sanity checks\n> 0.000 : 1 : ....label:allocate/init\n> 0.003 : 6 : ..label:reset_maps\n> 0.035 : 1 : label:do_write_index\n> /home/newren/floss/linux-stable/.git/index.lock\n> 0.034 : 1 : label:checkout\n> 0.034 : (99.9%)\n> 0.000 : 1 : ..label:Filtering content\n> 0.009 : 1 : label:do_read_index .git/index\n> 0.000 : 1 : label:write_auto_merge\n> 0.000 : 1 : label:record_unmerged\n> Estimated measurement overhead (.010 ms/region-measure * 679):\n> 0.006790000000000001\n> Timing including forking: 0.960 (0.013 additional seconds)\n> === END OUTPUT ===\n> This was a run that took just under 1s (and was a hot-cache case; I\n> had just done the same rebase before to warm the caches), and the\n> combination of index/working tree bits (everything at and after\n> do_write_index in the output) was 0.035+0.034+0.009+0+0=0.078 seconds,\n> corresponding to just over 8.1% of overall time. I'm curious where\n> that lands for your repository testcase; if the larger time ends up\n> somewhere under the indented label:incore_nonrecursive region, then\n> it's due to something other than index reading/updating/writing.\n> \n> *2*. If it really is due to index reading/updating/writing, then index\n> handling in merge-ort is confined to two functions: checkout() and\n> record_unmerged_index_entries(). Both functions aren't too long, and\n> neither one calls into any other function within merge-ort.c.\n> (Further, checkout() is a near copy of code from merge_working_tree()\n> in builtin/checkout.c, or at least a copy of that function from a year\n> or so ago.) As such, it's possible you can go in and make whatever\n> special tweaks you have for partial index reading/writing to those\n> functions.\n> \n> I'm curious to hear back more on this.\n\nI don't have a lot of time to dig into this right now, but here are\nthe stats for my rebases and merges with and without your option.\n\nThe first thing I notice for each is that there is a significant\namount of \"unmeasured\" time at the beginning of each, and that\ncould possibly be improved separately.\n\nFirst, try a rebase forward and backward.\n\n$ /_git/git/summarize-perf git rebase --onto to from test\nSuccessfully rebased and updated refs/heads/test.\nAccumulated times:\n 8.511 : (74.9%)\n 1.331 : 1 : ......label:unpack_trees\n 0.200 : (15.1%)\n 0.580 : 1 : ........label:traverse_trees\n 0.403 : 1 : ........label:clear_ce_flags/0x00000000_0x02000000\n 0.126 : 1 : ........label:check_updates\n 0.126 : (100.0%)\n 0.000 : 1 : ..........label:Filtering content\n 0.021 : 1 : ........label:clear_ce_flags/0x00080000_0x42000000\n 0.000 : 1 : ........label:fully_valid\n 1.059 : 1 : ......label:do_write_index /_git/office/src/.git/index.lock\n 0.930 : (87.9%)\n 0.128 : 1 : ........label:write/extension/cache_tree\n 0.455 : 2 : ......label:fully_valid\n 0.001 : 1 : ......label:traverse_trees\n 0.000 : 1 : ......label:check_updates\nEstimated measurement overhead (.010 ms/region-measure * 41): 0.00041000000000000005\nTiming including forking: 11.382 (0.026 additional seconds)\n\n$ /_git/git/summarize-perf git rebase --onto from to test\nSuccessfully rebased and updated refs/heads/test.\nAccumulated times:\n 8.556 : (75.2%)\n 1.315 : 1 : ......label:unpack_trees\n 0.197 : (15.0%)\n 0.580 : 1 : ........label:traverse_trees\n 0.391 : 1 : ........label:clear_ce_flags/0x00000000_0x02000000\n 0.126 : 1 : ........label:check_updates\n 0.126 : (100.0%)\n 0.000 : 1 : ..........label:Filtering content\n 0.021 : 1 : ........label:clear_ce_flags/0x00080000_0x42000000\n 0.000 : 1 : ........label:fully_valid\n 1.071 : 1 : ......label:do_write_index /_git/office/src/.git/index.lock\n 0.942 : (88.0%)\n 0.129 : 1 : ........label:write/extension/cache_tree\n 0.431 : 2 : ......label:fully_valid\n 0.001 : 1 : ......label:traverse_trees\n 0.000 : 1 : ......label:check_updates\nEstimated measurement overhead (.010 ms/region-measure * 41): 0.00041000000000000005\nTiming including forking: 11.399 (0.026 additional seconds)\n\nThen do the same with the ort strategy.\n\n$ /_git/git/summarize-perf git -c pull.twohead=ort rebase --onto to from test\nSuccessfully rebased and updated refs/heads/test.\nAccumulated times:\n 8.350 : (73.2%)\n 1.403 : 1 : ....label:checkout \n 0.000 : ( 0.0%)\n 1.403 : 1 : ......label:unpack_trees\n 0.312 : (22.3%)\n 0.539 : 1 : ........label:traverse_trees\n 0.401 : 1 : ........label:clear_ce_flags/0x00000000_0x02000000\n 0.128 : 1 : ........label:check_updates\n 0.128 : (100.0%)\n 0.000 : 1 : ..........label:Filtering content\n 0.021 : 1 : ........label:clear_ce_flags/0x00080000_0x42000000\n 0.000 : 1 : ........label:fully_valid\n 1.081 : 1 : ....label:do_write_index /_git/office/src/.git/index.lock\n 0.951 : (88.1%)\n 0.129 : 1 : ......label:write/extension/cache_tree\n 0.432 : 2 : ....label:fully_valid\n 0.143 : 1 : ....label:do_read_index .git/index\n 0.019 : (13.1%)\n 0.125 : 1 : label:read/extension/cache_tree\n 0.004 : 1 : ....label:incore_nonrecursive\n 0.001 : (25.8%)\n 0.002 : 1 : ......label:process_entries\n 0.000 : ( 2.6%)\n 0.001 : 1 : ........label:finalize\n 0.001 : 1 : ........label:process_entries setup\n 0.000 : ( 6.7%)\n 0.001 : 1 : ..........label:plist sort\n 0.000 : 1 : ..........label:plist copy\n 0.000 : 1 : ..........label:plist grow\n 0.000 : 1 : ........label:processing\n 0.001 : 1 : ......label:collect_merge_info\n 0.000 : (35.3%)\n 0.001 : 1 : ........label:traverse_trees\n 0.000 : 1 : ......label:merge_start\n 0.000 : (42.3%)\n 0.000 : 1 : ........label:allocate/init\n 0.000 : 1 : ........label:sanity checks\n 0.000 : 1 : ......label:renames \n 0.001 : 1 : ....label:traverse_trees\n 0.000 : 1 : ....label:write_auto_merge\n 0.000 : 1 : ....label:check_updates\n 0.000 : 1 : ....label:record_unmerged\nEstimated measurement overhead (.010 ms/region-measure * 56): 0.0005600000000000001\nTiming including forking: 11.442 (0.027 additional seconds)\n\n$ /_git/git/summarize-perf git -c pull.twohead=ort rebase --onto from to test\nSuccessfully rebased and updated refs/heads/test.\nAccumulated times:\n 8.337 : (73.2%)\n 1.395 : 1 : ....label:checkout \n 0.000 : ( 0.0%)\n 1.395 : 1 : ......label:unpack_trees\n 0.309 : (22.1%)\n 0.537 : 1 : ........label:traverse_trees\n 0.403 : 1 : ........label:clear_ce_flags/0x00000000_0x02000000\n 0.124 : 1 : ........label:check_updates\n 0.124 : (100.0%)\n 0.000 : 1 : ..........label:Filtering content\n 0.021 : 1 : ........label:clear_ce_flags/0x00080000_0x42000000\n 0.000 : 1 : ........label:fully_valid\n 1.084 : 1 : ....label:do_write_index /_git/office/src/.git/index.lock\n 0.955 : (88.1%)\n 0.129 : 1 : ......label:write/extension/cache_tree\n 0.436 : 2 : ....label:fully_valid\n 0.137 : 1 : ....label:do_read_index .git/index\n 0.013 : ( 9.3%)\n 0.125 : 1 : label:read/extension/cache_tree\n 0.004 : 1 : ....label:incore_nonrecursive\n 0.001 : (24.5%)\n 0.002 : 1 : ......label:process_entries\n 0.000 : ( 2.5%)\n 0.001 : 1 : ........label:finalize\n 0.001 : 1 : ........label:process_entries setup\n 0.000 : ( 6.5%)\n 0.001 : 1 : ..........label:plist sort\n 0.000 : 1 : ..........label:plist copy\n 0.000 : 1 : ..........label:plist grow\n 0.000 : 1 : ........label:processing\n 0.001 : 1 : ......label:collect_merge_info\n 0.000 : (26.5%)\n 0.001 : 1 : ........label:traverse_trees\n 0.000 : 1 : ......label:merge_start\n 0.000 : (43.1%)\n 0.000 : 1 : ........label:allocate/init\n 0.000 : 1 : ........label:sanity checks\n 0.000 : 1 : ......label:renames \n 0.001 : 1 : ....label:traverse_trees\n 0.000 : 1 : ....label:write_auto_merge\n 0.000 : 1 : ....label:check_updates\n 0.000 : 1 : ....label:record_unmerged\nEstimated measurement overhead (.010 ms/region-measure * 56): 0.0005600000000000001\nTiming including forking: 11.418 (0.024 additional seconds)\n\nAnd here are timings for a simple merge. Two files at root were changed in the\ncommits I made, but there are also some larger changes from the commit history.\nThese should all be seen as \"this tree updated in one of the two, so take that\ntree\".\n\n$ git reset --hard test2 && /_git/git/summarize-perf git merge test -m test\nMerge made by the 'recursive' strategy.\nAccumulated times:\n 2.647 : (48.6%)\n 1.384 : 1 : ..label:unpack_trees\n 0.267 : (19.3%)\n 0.582 : 1 : ....label:traverse_trees\n 0.391 : 1 : ....label:clear_ce_flags/0x00000000_0x02000000\n 0.124 : 1 : ....label:check_updates\n 0.124 : (100.0%)\n 0.000 : 1 : ......label:Filtering content\n 0.021 : 1 : ....label:clear_ce_flags/0x00080000_0x42000000\n 0.000 : 1 : ....label:fully_valid\n 1.060 : 1 : ..label:do_write_index /_git/office/src/.git/index.lock\n 0.931 : (87.9%)\n 0.128 : 1 : ....label:write/extension/cache_tree\n 0.226 : 1 : ..label:fully_valid \n 0.134 : 1 : ..label:do_read_index .git/index\n 0.008 : ( 5.8%)\n 0.126 : 1 : label:read/extension/cache_tree\n 0.001 : 1 : ..label:traverse_trees\n 0.000 : 1 : ..label:check_updates\n 0.000 : 1 : ..label:setup \n 0.000 : 1 : ..label:write back to queue\nEstimated measurement overhead (.010 ms/region-measure * 20): 0.0002\nTiming including forking: 5.466 (0.015 additional seconds)\n\n$ git reset --hard test2 && /_git/git/summarize-perf git -c pull.twohead=ort merge test -m test\nMerge made by the 'ort' strategy.\nAccumulated times:\n 2.531 : (49.1%)\n 1.328 : 1 : ..label:checkout \n 0.000 : ( 0.0%)\n 1.328 : 1 : ....label:unpack_trees\n 0.228 : (17.2%)\n 0.566 : 1 : ......label:traverse_trees\n 0.388 : 1 : ......label:clear_ce_flags/0x00000000_0x02000000\n 0.125 : 1 : ......label:check_updates\n 0.125 : (100.0%)\n 0.000 : 1 : ........label:Filtering content\n 0.021 : 1 : ......label:clear_ce_flags/0x00080000_0x42000000\n 0.000 : 1 : ......label:fully_valid\n 1.067 : 1 : ..label:do_write_index /_git/office/src/.git/index.lock\n 0.938 : (87.9%)\n 0.129 : 1 : ....label:write/extension/cache_tree\n 0.230 : 1 : ..label:fully_valid \n 0.002 : 1 : ..label:incore_recursive\n 0.001 : (22.3%)\n 0.001 : 1 : ....label:collect_merge_info\n 0.001 : (60.2%)\n 0.000 : 1 : ......label:traverse_trees\n 0.001 : 1 : ....label:process_entries\n 0.000 : ( 2.8%)\n 0.001 : 1 : ......label:finalize\n 0.000 : 1 : ......label:process_entries setup\n 0.000 : ( 6.9%)\n 0.000 : 1 : ........label:plist sort\n 0.000 : 1 : ........label:plist copy\n 0.000 : 1 : ........label:plist grow\n 0.000 : 1 : ......label:processing\n 0.000 : 1 : ....label:merge_start\n 0.000 : (50.0%)\n 0.000 : 1 : ......label:allocate/init\n 0.000 : 1 : ......label:sanity checks\n 0.000 : 1 : ....label:renames \n 0.001 : 1 : ..label:traverse_trees\n 0.000 : 1 : ..label:write_auto_merge\n 0.000 : 1 : ..label:check_updates\n 0.000 : 1 : ..label:setup \n 0.000 : 1 : ..label:display messages\n 0.000 : 1 : ..label:write back to queue\n 0.000 : 1 : ..label:record_unmerged\nEstimated measurement overhead (.010 ms/region-measure * 36): 0.00036\nTiming including forking: 5.174 (0.015 additional seconds)\n\nThanks,\n-Stolee\n"},{"id":"409400","messageId":"CABPp-BHO7bZ3H7A=E9TudhvBoNfwPvRiDMm8S9kq3mYeSXrpXw@mail.gmail.com","threadId":"54556","inReplyTo":"ee5cb3fe-4983-deba-2dba-b11a74adfbde@gmail.com","subject":"Re: [PATCH v2 00/20] fundamentals of merge-ort implementation","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-09T17:13:24Z","receivedAt":"2020-11-09T17:13:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Derrick,\n\nOn Mon, Nov 9, 2020 at 4:30 AM Derrick Stolee wrote:\n>\n> On 11/7/2020 2:39 PM, Elijah Newren wrote:\n\n> > *1*. Could you give me the accumulated times from the trace2_regions\n> > so we can verify where the time is spent? The 'summarize-perf' script\n> > at the toplevel of the repo in my ort branch might be helpful for\n> > this; just prefix any git command with that script and it accumulates\n> > the trace2 region times and prints them out. For example, I could run\n> > 'summarize-perf git merge --no-edit B^0' or 'summarize-perf test-tool\n> > fast-rebase --onto HEAD ca76bea9 myfeature'. Here's an example:\n> >\n> > === BEGIN OUTPUT ===\n> > $ /home/newren/floss/git/summarize-perf test-tool fast-rebase --onto\n> > HEAD 4703d9119972bf586d2cca76ec6438f819ffa30e hwmon-updates\n> > Rebasing fd8bdb23b91876ac1e624337bb88dc1dcc21d67e...\n> > Done.\n> > Accumulated times:\n> > 0.031 : ( 3.2%)\n> > 0.837 : 35 : label:incore_nonrecursive\n> > 0.003 : ( 0.4%)\n> > 0.476 : 41 : ..label:collect_merge_info\n> > 0.001 : ( 0.2%)\n> > 0.475 : 41 : ....label:traverse_trees\n> > 0.298 : 41 : ..label:renames\n> > 0.015 : ( 5.1%)\n> > 0.280 : 41 : ....label:regular renames\n> > 0.036 : (12.7%)\n> > 0.244 : 6 : ......label:diffcore_rename\n> > 0.001 : ( 0.4%)\n> > 0.078 : 6 : ........label:dir rename setup\n> > 0.055 : 6 : ........label:basename matches\n> > 0.051 : 6 : ........label:exact renames\n> > 0.031 : 6 : ........label:write back to queue\n> > 0.017 : 6 : ........label:setup\n> > 0.009 : 6 : ........label:cull basename\n> > 0.003 : 6 : ........label:cull exact\n> > 0.002 : 35 : ....label:directory renames\n> > 0.001 : 35 : ....label:process renames\n> > 0.052 : 35 : ..label:process_entries\n> > 0.001 : ( 1.7%)\n> > 0.033 : 35 : ....label:processing\n> > 0.017 : 35 : ....label:process_entries setup\n> > 0.001 : ( 5.8%)\n> > 0.008 : 35 : ......label:plist copy\n> > 0.008 : 35 : ......label:plist sort\n> > 0.000 : 35 : ......label:plist grow\n> > 0.001 : 35 : ....label:finalize\n> > 0.005 : 35 : ..label:merge_start\n> > 0.001 : (18.8%)\n> > 0.004 : 34 : ....label:reset_maps\n> > 0.000 : 35 : ....label:sanity checks\n> > 0.000 : 1 : ....label:allocate/init\n> > 0.003 : 6 : ..label:reset_maps\n> > 0.035 : 1 : label:do_write_index\n> > /home/newren/floss/linux-stable/.git/index.lock\n> > 0.034 : 1 : label:checkout\n> > 0.034 : (99.9%)\n> > 0.000 : 1 : ..label:Filtering content\n> > 0.009 : 1 : label:do_read_index .git/index\n> > 0.000 : 1 : label:write_auto_merge\n> > 0.000 : 1 : label:record_unmerged\n> > Estimated measurement overhead (.010 ms/region-measure * 679):\n> > 0.006790000000000001\n> > Timing including forking: 0.960 (0.013 additional seconds)\n> > === END OUTPUT ===\n> > This was a run that took just under 1s (and was a hot-cache case; I\n> > had just done the same rebase before to warm the caches), and the\n> > combination of index/working tree bits (everything at and after\n> > do_write_index in the output) was 0.035+0.034+0.009+0+0=0.078 seconds,\n> > corresponding to just over 8.1% of overall time. I'm curious where\n> > that lands for your repository testcase; if the larger time ends up\n> > somewhere under the indented label:incore_nonrecursive region, then\n> > it's due to something other than index reading/updating/writing.\n> >\n> > *2*. If it really is due to index reading/updating/writing, then index\n> > handling in merge-ort is confined to two functions: checkout() and\n> > record_unmerged_index_entries(). Both functions aren't too long, and\n> > neither one calls into any other function within merge-ort.c.\n> > (Further, checkout() is a near copy of code from merge_working_tree()\n> > in builtin/checkout.c, or at least a copy of that function from a year\n> > or so ago.) As such, it's possible you can go in and make whatever\n> > special tweaks you have for partial index reading/writing to those\n> > functions.\n> >\n> > I'm curious to hear back more on this.\n>\n> I don't have a lot of time to dig into this right now, but here are\n> the stats for my rebases and merges with and without your option.\n\nActually, this was pretty enlightening. I think I know about what's\nhappening...\n\nFirst, a few years ago, Ben said that merges in the Microsoft repos\ntook about an hour[1]:\n\"For the repro that I have been using this drops the merge time from ~1 hour to\n~5 minutes and the unmerged entries goes down from ~40,000 to 1.\"\nThe change he made to drop it that far was to turn off rename detection.\n\n[1] https://lore.kernel.org/git/20180426205202.23056-1-benpeart@microsoft.com/\n\nKeep that in mind, especially since your times are actually\nsignificantly less than 5 minutes...\n\n> The first thing I notice for each is that there is a significant\n> amount of \"unmeasured\" time at the beginning of each, and that\n> could possibly be improved separately.\n>\n> First, try a rebase forward and backward.\n>\n> $ /_git/git/summarize-perf git rebase --onto to from test\n> Successfully rebased and updated refs/heads/test.\n> Accumulated times:\n> 8.511 : (74.9%)\n\nWild guess: This is setup_git_directory() loading your ~3 million entry index.\n\n> 1.331 : 1 : ......label:unpack_trees\n> 0.200 : (15.1%)\n> 0.580 : 1 : ........label:traverse_trees\n> 0.403 : 1 : ........label:clear_ce_flags/0x00000000_0x02000000\n> 0.126 : 1 : ........label:check_updates\n> 0.126 : (100.0%)\n> 0.000 : 1 : ..........label:Filtering content\n> 0.021 : 1 : ........label:clear_ce_flags/0x00080000_0x42000000\n> 0.000 : 1 : ........label:fully_valid\n> 1.059 : 1 : ......label:do_write_index /_git/office/src/.git/index.lock\n> 0.930 : (87.9%)\n> 0.128 : 1 : ........label:write/extension/cache_tree\n> 0.455 : 2 : ......label:fully_valid\n> 0.001 : 1 : ......label:traverse_trees\n> 0.000 : 1 : ......label:check_updates\n> Estimated measurement overhead (.010 ms/region-measure * 41): 0.00041000000000000005\n> Timing including forking: 11.382 (0.026 additional seconds)\n>\n> $ /_git/git/summarize-perf git rebase --onto from to test\n> Successfully rebased and updated refs/heads/test.\n> Accumulated times:\n> 8.556 : (75.2%)\n> 1.315 : 1 : ......label:unpack_trees\n> 0.197 : (15.0%)\n> 0.580 : 1 : ........label:traverse_trees\n> 0.391 : 1 : ........label:clear_ce_flags/0x00000000_0x02000000\n> 0.126 : 1 : ........label:check_updates\n> 0.126 : (100.0%)\n> 0.000 : 1 : ..........label:Filtering content\n> 0.021 : 1 : ........label:clear_ce_flags/0x00080000_0x42000000\n> 0.000 : 1 : ........label:fully_valid\n> 1.071 : 1 : ......label:do_write_index /_git/office/src/.git/index.lock\n> 0.942 : (88.0%)\n> 0.129 : 1 : ........label:write/extension/cache_tree\n> 0.431 : 2 : ......label:fully_valid\n> 0.001 : 1 : ......label:traverse_trees\n> 0.000 : 1 : ......label:check_updates\n> Estimated measurement overhead (.010 ms/region-measure * 41): 0.00041000000000000005\n> Timing including forking: 11.399 (0.026 additional seconds)\n\nDid you include two runs of recursive and two runs of ort just to show\nthat the timings were stable and thus there wasn't warm or cold disk\ncache issues affecting things? If so, good plan. (If there was\nanother reason, let me know; I missed it.)\n\n> Then do the same with the ort strategy.\n>\n> $ /_git/git/summarize-perf git -c pull.twohead=ort rebase --onto to from test\n> Successfully rebased and updated refs/heads/test.\n> Accumulated times:\n> 8.350 : (73.2%)\n> 1.403 : 1 : ....label:checkout\n> 0.000 : ( 0.0%)\n> 1.403 : 1 : ......label:unpack_trees\n> 0.312 : (22.3%)\n> 0.539 : 1 : ........label:traverse_trees\n> 0.401 : 1 : ........label:clear_ce_flags/0x00000000_0x02000000\n> 0.128 : 1 : ........label:check_updates\n> 0.128 : (100.0%)\n> 0.000 : 1 : ..........label:Filtering content\n> 0.021 : 1 : ........label:clear_ce_flags/0x00080000_0x42000000\n> 0.000 : 1 : ........label:fully_valid\n> 1.081 : 1 : ....label:do_write_index /_git/office/src/.git/index.lock\n> 0.951 : (88.1%)\n> 0.129 : 1 : ......label:write/extension/cache_tree\n> 0.432 : 2 : ....label:fully_valid\n> 0.143 : 1 : ....label:do_read_index .git/index\n> 0.019 : (13.1%)\n> 0.125 : 1 : label:read/extension/cache_tree\n> 0.004 : 1 : ....label:incore_nonrecursive\n> 0.001 : (25.8%)\n> 0.002 : 1 : ......label:process_entries\n> 0.000 : ( 2.6%)\n> 0.001 : 1 : ........label:finalize\n> 0.001 : 1 : ........label:process_entries setup\n> 0.000 : ( 6.7%)\n> 0.001 : 1 : ..........label:plist sort\n> 0.000 : 1 : ..........label:plist copy\n> 0.000 : 1 : ..........label:plist grow\n> 0.000 : 1 : ........label:processing\n> 0.001 : 1 : ......label:collect_merge_info\n> 0.000 : (35.3%)\n> 0.001 : 1 : ........label:traverse_trees\n> 0.000 : 1 : ......label:merge_start\n> 0.000 : (42.3%)\n> 0.000 : 1 : ........label:allocate/init\n> 0.000 : 1 : ........label:sanity checks\n> 0.000 : 1 : ......label:renames\n> 0.001 : 1 : ....label:traverse_trees\n> 0.000 : 1 : ....label:write_auto_merge\n> 0.000 : 1 : ....label:check_updates\n> 0.000 : 1 : ....label:record_unmerged\n> Estimated measurement overhead (.010 ms/region-measure * 56): 0.0005600000000000001\n> Timing including forking: 11.442 (0.027 additional seconds)\n\n.004s on label:incore_nonrecursive -- that's the actual merge\noperation. This was a trivial rebase, and the merging took just 4\nmilliseconds. But the overall run took 11.442 seconds because working\nwith 3M+ entries in the index just takes forever, and my code didn't\ntouch any on-disk formats, certainly not the index format.\n\n_All_ of my optimization work was on the merging piece, not the stuff\noutside. But for what you're testing here, it appears to be\nirrelevant compared to the overhead.\n\n> $ /_git/git/summarize-perf git -c pull.twohead=ort rebase --onto from to test\n> Successfully rebased and updated refs/heads/test.\n> Accumulated times:\n> 8.337 : (73.2%)\n> 1.395 : 1 : ....label:checkout\n> 0.000 : ( 0.0%)\n> 1.395 : 1 : ......label:unpack_trees\n> 0.309 : (22.1%)\n> 0.537 : 1 : ........label:traverse_trees\n> 0.403 : 1 : ........label:clear_ce_flags/0x00000000_0x02000000\n> 0.124 : 1 : ........label:check_updates\n> 0.124 : (100.0%)\n> 0.000 : 1 : ..........label:Filtering content\n> 0.021 : 1 : ........label:clear_ce_flags/0x00080000_0x42000000\n> 0.000 : 1 : ........label:fully_valid\n> 1.084 : 1 : ....label:do_write_index /_git/office/src/.git/index.lock\n> 0.955 : (88.1%)\n> 0.129 : 1 : ......label:write/extension/cache_tree\n> 0.436 : 2 : ....label:fully_valid\n> 0.137 : 1 : ....label:do_read_index .git/index\n> 0.013 : ( 9.3%)\n> 0.125 : 1 : label:read/extension/cache_tree\n> 0.004 : 1 : ....label:incore_nonrecursive\n> 0.001 : (24.5%)\n> 0.002 : 1 : ......label:process_entries\n> 0.000 : ( 2.5%)\n> 0.001 : 1 : ........label:finalize\n> 0.001 : 1 : ........label:process_entries setup\n> 0.000 : ( 6.5%)\n> 0.001 : 1 : ..........label:plist sort\n> 0.000 : 1 : ..........label:plist copy\n> 0.000 : 1 : ..........label:plist grow\n> 0.000 : 1 : ........label:processing\n> 0.001 : 1 : ......label:collect_merge_info\n> 0.000 : (26.5%)\n> 0.001 : 1 : ........label:traverse_trees\n> 0.000 : 1 : ......label:merge_start\n> 0.000 : (43.1%)\n> 0.000 : 1 : ........label:allocate/init\n> 0.000 : 1 : ........label:sanity checks\n> 0.000 : 1 : ......label:renames\n> 0.001 : 1 : ....label:traverse_trees\n> 0.000 : 1 : ....label:write_auto_merge\n> 0.000 : 1 : ....label:check_updates\n> 0.000 : 1 : ....label:record_unmerged\n> Estimated measurement overhead (.010 ms/region-measure * 56): 0.0005600000000000001\n> Timing including forking: 11.418 (0.024 additional seconds)\n\nAh, you included two copies for merge-ort too. I'm guessing you did\nthat just to show there wasn't some cold cache issues or something and\nthat the runs showed consistent timings?\n\n\n> And here are timings for a simple merge. Two files at root were changed in the\n> commits I made, but there are also some larger changes from the commit history.\n> These should all be seen as \"this tree updated in one of the two, so take that\n> tree\".\n\nAhah! That's a microsoft-specific optimization you guys made in the\nrecursive strategy, yes? It does NOT exist in upstream git. It's\nalso one that is nearly incompatible with rename detection; it turns\nout you can only do that optimization in the face of rename detection\nif you do a HUGE amount of specialized work and tracking in order to\ndetermine when it's safe _despite_ needing to detect renames. I\nthought that optimization was totally incompatible with rename\ndetection for a long time; I tried it a couple times while working on\nort and watched it break all kinds of rename tests...but I eventually\ndiscovered some tricks involving a lot of work to be able to run that\noptimization.\n\nSo, you aren't comparing upstream \"recursive\" to \"ort\", you're\ncomparing a tweaked version of recursive, and one that is incompatible\nwith how recursive's rename detection work. In fact, just to be clear\nin case you go looking, I suspect that this tweak is to be found\nwithin unpack_trees.c (which recursive relies on heavily).\n\nFurther, you've set it up so there are only a few files changed after\nunpack_trees returns.\n\nIn total, you have: (1) turned off rename detection (most my\noptimizations are for improving this factor, meaning I can't show an\nadvantage), (2) you took advantage of no rename detection to implement\ntrivial-tree merges (thus killing the main second advantage my\nalgorithm has), and (3) you are looking at a case with a tiny number\nof changes for the merge algorithm to process (thus killing a third\noptimization that removes quadratic performance). Those are my three\nbig optimizations, and you've made them all irrelevant. In fact,\nyou're in an area I would have been worried that ort would do _worse_\nthan recursive. I track an awful lot of things and there is overhead\nin checking and filling all that information in; if there are only a\nfew entries to merge, then all that information was a waste to collect\nand ort might be slower than recursive. But then again, that should\nbe a case where both algorithms are \"nearly instantaneous\" (or would\nbe if it weren't for your 3M+ index entry repo causing run_builtin()'s\ncall to setup_git_directory() in git.c to take a huge amount of time\nbefore the builtin is even called.)\n\n\n> $ git reset --hard test2 && /_git/git/summarize-perf git merge test -m test\n> Merge made by the 'recursive' strategy.\n> Accumulated times:\n> 2.647 : (48.6%)\n> 1.384 : 1 : ..label:unpack_trees\n> 0.267 : (19.3%)\n> 0.582 : 1 : ....label:traverse_trees\n> 0.391 : 1 : ....label:clear_ce_flags/0x00000000_0x02000000\n> 0.124 : 1 : ....label:check_updates\n> 0.124 : (100.0%)\n> 0.000 : 1 : ......label:Filtering content\n> 0.021 : 1 : ....label:clear_ce_flags/0x00080000_0x42000000\n> 0.000 : 1 : ....label:fully_valid\n> 1.060 : 1 : ..label:do_write_index /_git/office/src/.git/index.lock\n> 0.931 : (87.9%)\n> 0.128 : 1 : ....label:write/extension/cache_tree\n> 0.226 : 1 : ..label:fully_valid\n> 0.134 : 1 : ..label:do_read_index .git/index\n> 0.008 : ( 5.8%)\n> 0.126 : 1 : label:read/extension/cache_tree\n> 0.001 : 1 : ..label:traverse_trees\n> 0.000 : 1 : ..label:check_updates\n> 0.000 : 1 : ..label:setup\n> 0.000 : 1 : ..label:write back to queue\n> Estimated measurement overhead (.010 ms/region-measure * 20): 0.0002\n> Timing including forking: 5.466 (0.015 additional seconds)\n\n5 seconds. I do have to hand it to Ben and anyone else involved,\nthough. From 1 hour down to 5 seconds is pretty good, even if it was\ndone by hacks (turning off rename detection, and then implementing\ntrivial-tree merging that would have broken rename detection). I\nsuspect that whoever did that work might have found the unconditional\ndiscarding and re-reading of the index and fixed it as well?\n\n> $ git reset --hard test2 && /_git/git/summarize-perf git -c pull.twohead=ort merge test -m test\n> Merge made by the 'ort' strategy.\n> Accumulated times:\n> 2.531 : (49.1%)\n> 1.328 : 1 : ..label:checkout\n> 0.000 : ( 0.0%)\n> 1.328 : 1 : ....label:unpack_trees\n> 0.228 : (17.2%)\n> 0.566 : 1 : ......label:traverse_trees\n> 0.388 : 1 : ......label:clear_ce_flags/0x00000000_0x02000000\n> 0.125 : 1 : ......label:check_updates\n> 0.125 : (100.0%)\n> 0.000 : 1 : ........label:Filtering content\n> 0.021 : 1 : ......label:clear_ce_flags/0x00080000_0x42000000\n> 0.000 : 1 : ......label:fully_valid\n> 1.067 : 1 : ..label:do_write_index /_git/office/src/.git/index.lock\n> 0.938 : (87.9%)\n> 0.129 : 1 : ....label:write/extension/cache_tree\n> 0.230 : 1 : ..label:fully_valid\n> 0.002 : 1 : ..label:incore_recursive\n> 0.001 : (22.3%)\n> 0.001 : 1 : ....label:collect_merge_info\n> 0.001 : (60.2%)\n> 0.000 : 1 : ......label:traverse_trees\n> 0.001 : 1 : ....label:process_entries\n> 0.000 : ( 2.8%)\n> 0.001 : 1 : ......label:finalize\n> 0.000 : 1 : ......label:process_entries setup\n> 0.000 : ( 6.9%)\n> 0.000 : 1 : ........label:plist sort\n> 0.000 : 1 : ........label:plist copy\n> 0.000 : 1 : ........label:plist grow\n> 0.000 : 1 : ......label:processing\n> 0.000 : 1 : ....label:merge_start\n> 0.000 : (50.0%)\n> 0.000 : 1 : ......label:allocate/init\n> 0.000 : 1 : ......label:sanity checks\n> 0.000 : 1 : ....label:renames\n> 0.001 : 1 : ..label:traverse_trees\n> 0.000 : 1 : ..label:write_auto_merge\n> 0.000 : 1 : ..label:check_updates\n> 0.000 : 1 : ..label:setup\n> 0.000 : 1 : ..label:display messages\n> 0.000 : 1 : ..label:write back to queue\n> 0.000 : 1 : ..label:record_unmerged\n> Estimated measurement overhead (.010 ms/region-measure * 36): 0.00036\n> Timing including forking: 5.174 (0.015 additional seconds)\n\nHeh, yeah 0.002 seconds for ..label:incore_recursive. Only 2\nmilliseconds to create the actual merge tree. That does suggest you\nmight have fun with 'git log -p --remerge-diff'; if you can redo\nmerges in 2 milliseconds, showing them in git log output is very\nreasonable. :-)\n\n\nCould we have some fun, though? What if you have some merge or rebase\ninvolving lots of changes, and you turn rename detection back on, and\nyou disable that trivial-tree resolution optimization that breaks\nrecursive's rename detection handling...and then compare recursive and\nort? (It might be easiest to just compare upstream recursive rather\nthan the one with all the microsoft changes to make sure you undid\nwhatever trivial tree handling work exists.)\n\nFor example, my testcase in the linux kernel was finding a series of a\nfew dozen patches I could rebase back to an older version, but\ntweaking the \"older\" version by renaming drivers/ -> pilots/ (with\nabout 26K files under that directory, that meant about 26K renames).\nSo, I got to see rebasing of dozens of real changes across a massive\nrename boundary -- and the massive rename boundary also guaranteed\nthere were lots of entries for the merge algorithm to deal with.\n\n\nIn the end, though, 4 milliseconds for the rebase and 2 milliseconds\nfor the merge, with the rest all being overhead of interfacing to the\nindex and working tree actually seems pretty good to me. I'm just\ncurious if we can check how things work for more involved cases.\n"},{"id":"409406","messageId":"8b33b42f-2939-6fec-ef01-892ac5a0f581@gmail.com","threadId":"54556","inReplyTo":"CABPp-BHO7bZ3H7A=E9TudhvBoNfwPvRiDMm8S9kq3mYeSXrpXw@mail.gmail.com","subject":"Re: [PATCH v2 00/20] fundamentals of merge-ort implementation","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-09T19:51:36Z","receivedAt":"2020-11-09T19:51:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/9/20 12:13 PM, Elijah Newren wrote:> Actually, this was pretty enlightening. I think I know about what's\n> happening...\n> \n> First, a few years ago, Ben said that merges in the Microsoft repos\n> took about an hour[1]:\n> \"For the repro that I have been using this drops the merge time from ~1 hour to\n> ~5 minutes and the unmerged entries goes down from ~40,000 to 1.\"\n> The change he made to drop it that far was to turn off rename detection.\n> \n> [1] https://lore.kernel.org/git/20180426205202.23056-1-benpeart@microsoft.com/\n> \n> Keep that in mind, especially since your times are actually\n> significantly less than 5 minutes...\n\nYes, the other thing to keep in mind is that this is\na Scalar repo with the default cone-mode sparse-checkout\nof only the files at root. For this repo, that means that\nthere are only ~10 files actually present.\n\nI wanted to remove any working directory updates/checks\nfrom the performance check as possible.\n\n>> $ /_git/git/summarize-perf git rebase --onto to from test\n>> Successfully rebased and updated refs/heads/test.\n>> Accumulated times:\n>> 8.511 : (74.9%)\n> \n> Wild guess: This is setup_git_directory() loading your ~3 million entry index.\n\nI think there is also some commit walking happening, but\nit shouldn't be too much. 'from' and 'to' are not very\nfar away.\n\n> Did you include two runs of recursive and two runs of ort just to show\n> that the timings were stable and thus there wasn't warm or cold disk\n> cache issues affecting things? If so, good plan. (If there was\n> another reason, let me know; I missed it.)\n\nFor the rebase, I did \"--onto to from test\" and \"--onto from to test\"\nto show both directions of the rebase. The merge I did twice for the\ncache issues ;)\n\n> .004s on label:incore_nonrecursive -- that's the actual merge\n> operation. This was a trivial rebase, and the merging took just 4\n> milliseconds. But the overall run took 11.442 seconds because working\n> with 3M+ entries in the index just takes forever, and my code didn't\n> touch any on-disk formats, certainly not the index format.\n> \n> _All_ of my optimization work was on the merging piece, not the stuff\n> outside. But for what you're testing here, it appears to be\n> irrelevant compared to the overhead.\n\nOK, so since we already disable rename detection through config,\nthe machinery that you are changing is already fast with the old\nalgorithm in these trivial cases.\n\nTo actually show any benefits, we would need to disable rename\ndetection or use a larger change.\n>> And here are timings for a simple merge. Two files at root were changed in the\n>> commits I made, but there are also some larger changes from the commit history.\n>> These should all be seen as \"this tree updated in one of the two, so take that\n>> tree\".\n> \n> Ahah! That's a microsoft-specific optimization you guys made in the\n> recursive strategy, yes? \n\nI'm not aware of any logic we have that's different from core Git.\nThe config we use [1] includes \"merge.stat = false\" and \"merge.renames\n= false\" but otherwise seems to be using stock Git.\n\n[1] https://github.com/microsoft/scalar/blob/1d7938d2df6921f7a3b4f3f1cce56a00929adc40/Scalar.Common/Maintenance/ConfigStep.cs#L100-L127\n\nI'm CC'ing Jeff Hostetler to see if he knows anything about a custom\nmerge algorithm in microsoft/git.\n\n> It does NOT exist in upstream git. It's\n> also one that is nearly incompatible with rename detection; it turns\n> out you can only do that optimization in the face of rename detection\n> if you do a HUGE amount of specialized work and tracking in order to\n> determine when it's safe _despite_ needing to detect renames. \n\nPerhaps merge.renames=false is enough to trigger this logic already?\n\n> I\n> thought that optimization was totally incompatible with rename\n> detection for a long time; I tried it a couple times while working on\n> ort and watched it break all kinds of rename tests...but I eventually\n> discovered some tricks involving a lot of work to be able to run that\n> optimization.\n\nI will try to keep this in mind.\n\n> So, you aren't comparing upstream \"recursive\" to \"ort\", you're\n> comparing a tweaked version of recursive, and one that is incompatible\n> with how recursive's rename detection work. In fact, just to be clear\n> in case you go looking, I suspect that this tweak is to be found\n> within unpack_trees.c (which recursive relies on heavily).\n> \n> Further, you've set it up so there are only a few files changed after\n> unpack_trees returns.\n> \n> In total, you have: (1) turned off rename detection (most my\n> optimizations are for improving this factor, meaning I can't show an\n> advantage), (2) you took advantage of no rename detection to implement\n> trivial-tree merges (thus killing the main second advantage my\n> algorithm has), and (3) you are looking at a case with a tiny number\n> of changes for the merge algorithm to process (thus killing a third\n> optimization that removes quadratic performance). Those are my three\n> big optimizations, and you've made them all irrelevant. In fact,\n> you're in an area I would have been worried that ort would do _worse_\n> than recursive. I track an awful lot of things and there is overhead\n> in checking and filling all that information in; if there are only a\n> few entries to merge, then all that information was a waste to collect\n> and ort might be slower than recursive. But then again, that should\n> be a case where both algorithms are \"nearly instantaneous\" (or would\n> be if it weren't for your 3M+ index entry repo causing run_builtin()'s\n> call to setup_git_directory() in git.c to take a huge amount of time\n> before the builtin is even called.)\n\nThanks for your time isolating this case. I appreciate knowing exactly\nwhich portions of the merge algorithm are being touched and which are\nnot.\n> 5 seconds. I do have to hand it to Ben and anyone else involved,\n> though. From 1 hour down to 5 seconds is pretty good, even if it was\n> done by hacks (turning off rename detection, and then implementing\n> trivial-tree merging that would have broken rename detection). I\n> suspect that whoever did that work might have found the unconditional\n> discarding and re-reading of the index and fixed it as well?\n\nAs you can probably tell from my general confusion, I had nothing\nto do with it. ;)\n\n> Heh, yeah 0.002 seconds for ..label:incore_recursive. Only 2\n> milliseconds to create the actual merge tree. That does suggest you\n> might have fun with 'git log -p --remerge-diff'; if you can redo\n> merges in 2 milliseconds, showing them in git log output is very\n> reasonable. :-)\n\nYeah, 'git merge-tree' is very fast for these cases, so I assumed\nthat something else was going on for that command.\n\n> Could we have some fun, though? What if you have some merge or rebase\n> involving lots of changes, and you turn rename detection back on, and\n> you disable that trivial-tree resolution optimization that breaks\n> recursive's rename detection handling...and then compare recursive and\n> ort? (It might be easiest to just compare upstream recursive rather\n> than the one with all the microsoft changes to make sure you undid\n> whatever trivial tree handling work exists.)\n\nI can try these kinds of cases, but it won't be today. I'm on kid duty\ntoday, and answering emails in between running around with them.\n\n> For example, my testcase in the linux kernel was finding a series of a\n> few dozen patches I could rebase back to an older version, but\n> tweaking the \"older\" version by renaming drivers/ -> pilots/ (with\n> about 26K files under that directory, that meant about 26K renames).\n> So, I got to see rebasing of dozens of real changes across a massive\n> rename boundary -- and the massive rename boundary also guaranteed\n> there were lots of entries for the merge algorithm to deal with.\n> \n> In the end, though, 4 milliseconds for the rebase and 2 milliseconds\n> for the merge, with the rest all being overhead of interfacing to the\n> index and working tree actually seems pretty good to me. I'm just\n> curious if we can check how things work for more involved cases.\n\nI'm definitely interested in identifying how your algorithm improves\nover the previous cases, and perhaps re-enabling rename detection for\nmerges is enough of a benefit to justify the new one.\n\nEventually, I hope to actually engage with your patches in the form\nof review. Just trying to build a mental model for what's going on\nfirst.\n\nThanks,\n-Stolee\n"},{"id":"409409","messageId":"20201109205525.2521806-1-jonathantanmy@google.com","threadId":"54556","inReplyTo":"CABPp-BHx4w1_JAen0fupCzosu0NfoTsT9d8dEve7n5VJZ0EvBQ@mail.gmail.com","subject":"Re: [PATCH v2 01/20] merge-ort: setup basic internal data structures","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-09T20:55:25Z","receivedAt":"2020-11-09T20:55:30Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > There seems to be 2 ways of referring to something that we couldn't\n> > merge - \"conflicted\" (or \"having a conflict\") and \"unmerged\". Should we\n> > stick to one of them?\n> \n> Uhm...perhaps, but it feels like I'm going to miss some while looking\n> over it. Also, there are some semantic differences at play. Since\n> the whole algorithm is divided around multiple stages --\n> collect_merge_info(), detect_and_process_renames(), process_entries(),\n> as of a given early stage we might just know that we couldn't merge it\n> *yet*. For example, both sides modified the entry, or one side\n> modified and the other side is missing (\"did they delete it or rename\n> it?\"). After rename detection and three-way content merge, something\n> that had not been automatically mergeable as of an earlier step might\n> end up being so. But we need names for stuff in the interim state.\n> But it's also possible for us to know at an early state that thing are\n> definitely going to be a conflict regardless of what later stages do\n> (e.g. both sides rename a path, but rename it differently).\n\nIn that case, maybe \"possibly conflicted\" and \"unconflicted\"? Or maybe\nsomeone else will come up with a better name.\n\n> > Also, looking ahead, I see that current_dir_name is used as a temporary\n> > variable in the recursive calls to collect_merge_info_callback(). I\n> > would prefer if current_dir_name went in the cbdata to that function\n> > instead, but if that's not possible, maybe document here that\n> > current_dir_name is only used in collect_merge_info_callback(), and\n> > temporarily at that.\n> \n> Yeah, not possible. collect_merge_info_callback() has to be of\n> traverse_callback_t type (from tree-walk.h), which provides no extra\n> parameters for extra callback data. I can add a documentation\n> comment.\n\nBut traverse_callback_t provides a data field, which you use to store a\nstruct merge_options in patch 6 - in theory, you could instead create\nanother struct that contains a pointer to struct merge_options and store\nthat in data instead. That does sound like it will make the code more\ncomplicated, though, so maybe current_dir_name here is the way.\n\n> > I wonder if this needs to be documented that the least significant bit\n> > corresponds to stages[0], and so forth.\n> \n> Maybe I should just put a comment to look at tree-walk.h? The struct\n> traverse_info has a \"fn\" member with a big comment above it describing\n> mask & dirmask; filemask is just mask & ~dirmask.\n\nThat makes sense. I understood a lot more once I saw that it was\niterating over a variable number of trees - hence the arrays.\n"},{"id":"409411","messageId":"20201109205955.2523889-1-jonathantanmy@google.com","threadId":"54556","inReplyTo":"CABPp-BFkAu6oO5V0jRh=ExuPxBXma1F17zaq-zEaqMagc72Ccw@mail.gmail.com","subject":"Re: [PATCH v2 06/20] merge-ort: implement a very basic collect_merge_info()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-09T20:59:55Z","receivedAt":"2020-11-09T20:59:59Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > > + unsigned mbase_null = !(mask & 1);\n> > > + unsigned side1_null = !(mask & 2);\n> > > + unsigned side2_null = !(mask & 4);\n> >\n> > Should these be \"int\"?\n> \n> Does the type matter, particularly since \"boolean\" isn't available?\n\nIt doesn't, which is why I would expect the most generic type - if I see\nsomething else, I would be led to think that there was a specific reason\nfor choosing that. But if I'm in the minority, that's fine.\n\n> > I thought that this was written like this (instead of inlining the 2\n> > double-quotes) to ensure that the string-equality-is-pointer-equality\n> > characteristic holds, but I see that that characteristic is for\n> > directory_name in struct merged_info, not current_dir_name in struct\n> > merge_options_internal. Any reason for not inlining \"\"?\n> \n> You're really digging in; I love it. From setup_path_info(), the\n> directory_name is set from the current_dir_name:\n> path_info->merged.directory_name = current_dir_name;\n> (and if you follow where the current_dir_name parameter gets its value\n> from, you find that it came indirectly from\n> opt->priv->current_dir_name), so current_dir_name must meet all the\n> requirements on merge_info's directory_name field.\n> \n> Perhaps there's still some kind of additional simplification possible\n> here, but directory rename detection is an area that has to take some\n> special care around this requirement. I simplified the code a little\n> bit in this area as I was trying to break off a good first 20 patches\n> to submit, but even if we can simplify it more, the structure is just\n> going to come back later.\n\nAh, I see.\n"},{"id":"409420","messageId":"20201109220431.2534786-1-jonathantanmy@google.com","threadId":"54556","inReplyTo":"CABPp-BHOvk+W0u_MYWtH8OC-Uh5bEfsRO6QuXZKWfDQ=8fyasA@mail.gmail.com","subject":"Re: [PATCH v2 08/20] merge-ort: compute a few more useful fields for collect_merge_info","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-09T22:04:31Z","receivedAt":"2020-11-09T22:04:37Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On Fri, Nov 6, 2020 at 2:52 PM Jonathan Tan wrote:\n> >\n> > > + /*\n> > > + * Note: We only label files with df_conflict, not directories.\n> > > + * Since directories stay where they are, and files move out of the\n> > > + * way to make room for a directory, we don't care if there was a\n> > > + * directory/file conflict for a parent directory of the current path.\n> > > + */\n> > > + unsigned df_conflict = (filemask != 0) && (dirmask != 0);\n> >\n> > Suppose you have:\n> >\n> > [ours]\n> > foo/\n> > bar/\n> > baz\n> > quux\n> > [theirs]\n> > foo\n> >\n> > By \"we only label files with df_conflict, not directories\", are you\n> > referring to not labelling \"foo/\" in [ours], or to \"bar/\", \"baz\", and\n> > \"quux\" (so, the files and directories within a directory)? At first I\n> > thought you were referring to the former, but perhaps you are referring\n> > to the latter.\n> \n> The former. I was drawing a distinction between how this code\n> operates, and how unpack_trees() operates, which probably only matters\n> to those familiar with unpack_trees() or who have been reading through\n> it recently.\n\nJust for clarification: do you mean \"the latter\"? (The \"not\" in my\nquestion might be confusing.)\n\nTo be more illustrative in what I meant, at first I thought that you\nwere NOT labelling \"foo/\" in [ours], hence:\n\n [ours]\n foo/ <- unlabeled\n [theirs]\n foo <- labeled\n\nIn this way, in a sense, you are indeed labelling only the file, not the\ndirectory.\n\nBut instead what you seem to be doing is this:\n\n [ours]\n foo/ <- labeled\n bar/ <- unlabeled\n baz <- unlabeled\n quux <- unlabeled\n [theirs]\n foo <- labeled\n\nwhich is what I meant by NOT labelling \"bar/\", \"baz\", and \"quux\".\n\n> unpack_trees() will note when there is a directory/file\n> conflict, and propagates that information to all subtrees, with every\n> path specially checking for the o->df_conflict_entry and then handling\n> it specially (e.g. keeping higher order stages instead of using an\n> aggressive or trivial resolutions).\n\nAnd here it seems like you're describing that unpack_trees() would label\nit in this way:\n\n [ours]\n foo/ <- labeled\n bar/ <- labeled\n baz <- labeled\n quux <- labeled\n [theirs]\n foo <- labeled\n\n(and you're emphasizing by contrast that merge-ort is NOT doing this).\n\n> However, leaving both a file and\n> a directory at the same path, while allowed in the index, makes for\n> ugliness and difficulty for users to resolve. Plus it isn't allowed\n> in the working tree anyway. We decided a while ago that it'd be\n> better to represent these conflicts differently[1], [2].\n> \n> Also, at the time you are unpacking or traversing trees, you only know\n> if one side had a directory where the other side had a file. You\n> don't know if the final merge result will actually have a\n> directory/file conflict. If the file existed in both the base version\n> and unmodified on one side, for example, then the file will be removed\n> as part of the merge. It is similarly possible that the entire\n> directory of files all need to be deleted or are all renamed\n> elsewhere. So, you have to keep track of a df_conflict bit, but you\n> can't act on it until you've processed several other things first.\n> \n> Since I already know I'm not going to move a whole directory of files\n> out of the way so that a file can be placed in the working tree\n> instead of that whole directory, the directory doesn't need to be\n> tweaked. I'm not going to propagate any information about a\n> directory/file conflict at some path down to all subpaths of the\n> directory. I only track it for the file that immediately conflicts,\n> and then only take action on it after resolving all the paths under\n> the corresponding directory to see if the directory/file conflict\n> remains.\n> \n> [1] https://lore.kernel.org/git/xmqqbmabcuhf.fsf@gitster-ct.c.googlers.com/\n> and the thread surrounding it\n> [2] https://lore.kernel.org/git/f27f12e8e50e56c010c29caa00296475d4de205b.1603731704.git.gitgitgadget@gmail.com/,\n> which is now commit ef52778708 (\"merge tests: expect improved\n> directory/file conflict handling in ort\", 2020-10-26)\n\nMakes sense.\n\n> > > @@ -161,6 +179,13 @@ static int collect_merge_info_callback(int n,\n> > > newinfo.name = p->path;\n> > > newinfo.namelen = p->pathlen;\n> > > newinfo.pathlen = st_add3(newinfo.pathlen, p->pathlen, 1);\n> > > + /*\n> > > + * If we did care about parent directories having a D/F\n> > > + * conflict, then we'd include\n> > > + * newinfo.df_conflicts |= (mask & ~dirmask);\n> > > + * here. But we don't. (See comment near setting of local\n> > > + * df_conflict variable near the beginning of this function).\n> > > + */\n> >\n> > I'm not sure how \"mask\" and \"dirmask\" contains information about parent\n> > directories. \"mask\" represents the available entries, and \"dirmask\"\n> > represents which of them are directories, as far as I know. So we can\n> > notice when something is missing, but I don't see how this distinguishes\n> > between the case that something is missing because it was in a parent\n> > directory that got deleted, vs something is missing because it itself\n> > got deleted.\n> \n> Yeah, this is more comparisons to unpack_trees. This code is about to\n> set up a recursive call into subdirectories. newinfo is set based on\n> the mask and dirmask of the current entry, and then subdirectories can\n> consult newinfo.df_conflicts to see if that path is within a directory\n> that was involved in a directory/file conflict. For example:\n> \n> Tree in base version:\n> foo/\n> bar\n> stuff.txt\n> Tree on side 1: (adds foo/baz)\n> foo/\n> bar\n> baz\n> stuff.txt\n> Tree on side 2: (deletes foo/, adds new file foo)\n> foo\n> stuff.txt\n> \n> When processing 'foo', we have mask=7, dirmask = 3. So, here\n> unpack_trees() would have set newinfo.df_conflicts = (mask & ~dirmask)\n> = 4. Then when we process foo/bar or foo/baz, we have mask=2,\n> dirmask=0, which looks like there are no directory/file conflicts.\n> However, we can note that these paths are under a directory involved\n> in a directory/file conflict via info.df_conflicts whose value is 4.\n> unpack_trees() cared about paths under a directory that was involved\n> in a directory/file conflict, and someone familiar with that code\n> might ask why I don't also track the same information. The answer is\n> that I don't track it, even though I thought about it, because it's\n> useless overhead since I'm going to leave the directory alone and move\n> the file out of the way.\n> \n> Does that make sense?\n\nAh...yes, that makes sense. I think I didn't notice the \"newinfo\", so I\ndidn't realize that we were setting the info of our children based on\nourselves. Perhaps I would have noticed it sooner if the comment had\nread \"If this file/directory cared about its parent directory (the\ncurrent directory) having a D/F conflict, then we'd propagate the masks\nin this way:\" instead of \"If we did care about parent directories having\na D/F conflict\", but perhaps the point is already obvious enough.\n"},{"id":"409421","messageId":"20201109220903.2536706-1-jonathantanmy@google.com","threadId":"54556","inReplyTo":"CABPp-BEyosjCpBr-8B19YXZV0mpn3oYAXoaaROKFNZQ+p4ZMnQ@mail.gmail.com","subject":"Re: [PATCH v2 09/20] merge-ort: record stage and auxiliary info for every path","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-09T22:09:03Z","receivedAt":"2020-11-09T22:09:08Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > So these are placed in paths but not unmerged. I'm starting to wonder if\n> > struct merge_options_internal should be called merge_options_state or\n> > something, and each field having documentation about when they're used\n> > (or better yet, have functions like collect_merge_info() return their\n> > calculations in return values (which may be \"out\" parameters) instead of\n> > in this struct).\n> \n> Right, unmerged is only those paths that remain unmerged after all\n> steps. record_unmerged_index_entries() could simply walk over all\n> entries in paths and pick out the ones that were unmerged, but\n> process_entries() has to walk over all paths, determine whether they\n> can be merged, and determine what to record for the resulting tree for\n> each path. So, having it stash away the unmerged stuff is a simple\n> optimization.\n> \n> Renaming to merge_options_state or even just merge_state would be fine\n> -- but any renaming done here will also affect merge-recursive.[ch].\n> See the definition of merge_options in merge-recursive. (For history,\n> merge-recursive.h stuffed state into merge_options, which risked funny\n> misusage patterns and made the API unnecessarily complex...and made it\n> suggest that alternative algorithms needed to have the same state.\n> So, the state was moved to a merge_options_internal struct. That's\n> not to say we can't rename, but it does need to be done in\n> merge-recursive as well.)\n\nAh, I see.\n\n> As for having collect_merge_info() return their calculations in return\n> values, would that just end with me returning a struct\n> merge_options_internal? Or did you want each return value added to\n> the function signature? Each return value in the function signature\n> makes sense right now for this super-simplified initial 20 patches,\n> but what about when this data structure gains all kind of\n> rename-related state that is collected, updated, and passed between\n> these areas? I'd have a huge number of \"out\" and \"in\" fields to every\n> function. Eventually, merge_options_internal (or whatever it might be\n> renamed to) expands to the following, where I have to first define an\n> extra enum and two extra structs so that you know the definitions of\n> new types that show up in merge_options_internal:\n\n[snip enums and structs]\n\nGood point. I should have realized that there would be much more to\ntrack.\n\n> > > + result->string = fullpath;\n> > > + result->util = path_info;\n> > > +}\n> > > +\n> > > static int collect_merge_info_callback(int n,\n> > > unsigned long mask,\n> > > unsigned long dirmask,\n> > > @@ -91,10 +136,12 @@ static int collect_merge_info_callback(int n,\n> > > */\n> > > struct merge_options *opt = info->data;\n> > > struct merge_options_internal *opti = opt->priv;\n> > > - struct conflict_info *ci;\n> > > + struct string_list_item pi; /* Path Info */\n> > > + struct conflict_info *ci; /* pi.util when there's a conflict */\n> >\n> > Looking ahead to patch 10, this seems more like \"pi.util unless we know\n> > for sure that there's no conflict\".\n> \n> That's too long for the line to remain at 80 characters; it's 16\n> characters over the limit. ;-)\n\nWell, you could move the description onto its own line :-)\n"},{"id":"409429","messageId":"CABPp-BEdcqaP4Le_31Mgo+u4-b+ZFVPw_HsEMKpNuHknc8pCqQ@mail.gmail.com","threadId":"54556","inReplyTo":"8b33b42f-2939-6fec-ef01-892ac5a0f581@gmail.com","subject":"Re: [PATCH v2 00/20] fundamentals of merge-ort implementation","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-09T22:44:45Z","receivedAt":"2020-11-09T22:44:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Derrick,\n\nOn Mon, Nov 9, 2020 at 11:51 AM Derrick Stolee wrote:\n>\n> On 11/9/20 12:13 PM, Elijah Newren wrote:> Actually, this was pretty enlightening. I think I know about what's\n> > happening...\n> >\n> > First, a few years ago, Ben said that merges in the Microsoft repos\n> > took about an hour[1]:\n> > \"For the repro that I have been using this drops the merge time from ~1 hour to\n> > ~5 minutes and the unmerged entries goes down from ~40,000 to 1.\"\n> > The change he made to drop it that far was to turn off rename detection.\n> >\n> > [1] https://lore.kernel.org/git/20180426205202.23056-1-benpeart@microsoft.com/\n> >\n> > Keep that in mind, especially since your times are actually\n> > significantly less than 5 minutes...\n>\n> Yes, the other thing to keep in mind is that this is\n> a Scalar repo with the default cone-mode sparse-checkout\n> of only the files at root. For this repo, that means that\n> there are only ~10 files actually present.\n>\n> I wanted to remove any working directory updates/checks\n> from the performance check as possible.\n\nAh, that explains how you got under 20s. I remember elsewhere on the\nlist someone (I think it was Ben again) mentioned that a \"git checkout\n-b \" took 20s, despite no need to update the working tree\nor index.\n\nI have only done one cursory test of merge-ort with sparse-checkouts;\nI should do more. There might be a bug somewhere, though it does at\nleast pass the regression tests and I think for the most part it's\nactually better: there are cases where merge-recursive will vivify\nfiles outside the sparse-checkout which were not conflicted (see e.g.\nhttps://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/);\nin contrast, merge-ort shouldn't have any such cases -- it'll only add\nfiles to the working copy if they match the sparsity patterns or the\npath has conflicts.\n\n> >> $ /_git/git/summarize-perf git rebase --onto to from test\n> >> Successfully rebased and updated refs/heads/test.\n> >> Accumulated times:\n> >> 8.511 : (74.9%)\n> >\n> > Wild guess: This is setup_git_directory() loading your ~3 million entry index.\n>\n> I think there is also some commit walking happening, but\n> it shouldn't be too much. 'from' and 'to' are not very\n> far away.\n\nMakes sense. I suspect that with your commit-graphs this ends up\nbeing fast enough that you might have difficulty even measuring it,\nthough.\n\n> > Did you include two runs of recursive and two runs of ort just to show\n> > that the timings were stable and thus there wasn't warm or cold disk\n> > cache issues affecting things? If so, good plan. (If there was\n> > another reason, let me know; I missed it.)\n>\n> For the rebase, I did \"--onto to from test\" and \"--onto from to test\"\n> to show both directions of the rebase. The merge I did twice for the\n> cache issues ;)\n\nOh, good call. Thanks for pointing it out, I missed that on first reading.\n\n> > .004s on label:incore_nonrecursive -- that's the actual merge\n> > operation. This was a trivial rebase, and the merging took just 4\n> > milliseconds. But the overall run took 11.442 seconds because working\n> > with 3M+ entries in the index just takes forever, and my code didn't\n> > touch any on-disk formats, certainly not the index format.\n> >\n> > _All_ of my optimization work was on the merging piece, not the stuff\n> > outside. But for what you're testing here, it appears to be\n> > irrelevant compared to the overhead.\n>\n> OK, so since we already disable rename detection through config,\n> the machinery that you are changing is already fast with the old\n> algorithm in these trivial cases.\n>\n> To actually show any benefits, we would need to disable rename\n> detection or use a larger change.\n\n...or both. :-)\n\n> >> And here are timings for a simple merge. Two files at root were changed in the\n> >> commits I made, but there are also some larger changes from the commit history.\n> >> These should all be seen as \"this tree updated in one of the two, so take that\n> >> tree\".\n> >\n> > Ahah! That's a microsoft-specific optimization you guys made in the\n> > recursive strategy, yes?\n>\n> I'm not aware of any logic we have that's different from core Git.\n> The config we use [1] includes \"merge.stat = false\" and \"merge.renames\n> = false\" but otherwise seems to be using stock Git.\n>\n> [1] https://github.com/microsoft/scalar/blob/1d7938d2df6921f7a3b4f3f1cce56a00929adc40/Scalar.Common/Maintenance/ConfigStep.cs#L100-L127\n>\n> I'm CC'ing Jeff Hostetler to see if he knows anything about a custom\n> merge algorithm in microsoft/git.\n\nOh, I took your wording that 'These should all be seen as \"this tree\nupdated in one of the two, so take that tree\"' as an implication that\nyou had a special merge tweak and wanted to verify it didn't regress.\nI think I read too much into your wording.\n\nAlso, thinking over it more, I remember now that Ben also turned on\nunpack_opts.aggressive when rename detection was turned off -- see\ncommit 6f10a09e0a (\"merge: pass aggressive when rename detection is\nturned off\", 2018-05-02). That isn't quite as advantageous as doing a\ntrivial tree merge, but if the algorithm that does the trivial tree\nmerge has to end up updating a complete index later anyway via the\ncheckout logic of unpack_trees, then the differences are basically a\nwash.\n\n> > It does NOT exist in upstream git. It's\n> > also one that is nearly incompatible with rename detection; it turns\n> > out you can only do that optimization in the face of rename detection\n> > if you do a HUGE amount of specialized work and tracking in order to\n> > determine when it's safe _despite_ needing to detect renames.\n>\n> Perhaps merge.renames=false is enough to trigger this logic already?\n\nYeah, since I read too much into what you wrote and know that I\nremember the \"if (no_renames) o.aggressive = 1\" bit, then yeah this\nwould be enough.\n\n> > I\n> > thought that optimization was totally incompatible with rename\n> > detection for a long time; I tried it a couple times while working on\n> > ort and watched it break all kinds of rename tests...but I eventually\n> > discovered some tricks involving a lot of work to be able to run that\n> > optimization.\n>\n> I will try to keep this in mind.\n>\n> > So, you aren't comparing upstream \"recursive\" to \"ort\", you're\n> > comparing a tweaked version of recursive, and one that is incompatible\n> > with how recursive's rename detection work. In fact, just to be clear\n> > in case you go looking, I suspect that this tweak is to be found\n> > within unpack_trees.c (which recursive relies on heavily).\n> >\n> > Further, you've set it up so there are only a few files changed after\n> > unpack_trees returns.\n> >\n> > In total, you have: (1) turned off rename detection (most my\n> > optimizations are for improving this factor, meaning I can't show an\n> > advantage), (2) you took advantage of no rename detection to implement\n> > trivial-tree merges (thus killing the main second advantage my\n> > algorithm has), and (3) you are looking at a case with a tiny number\n> > of changes for the merge algorithm to process (thus killing a third\n> > optimization that removes quadratic performance). Those are my three\n> > big optimizations, and you've made them all irrelevant. In fact,\n> > you're in an area I would have been worried that ort would do _worse_\n> > than recursive. I track an awful lot of things and there is overhead\n> > in checking and filling all that information in; if there are only a\n> > few entries to merge, then all that information was a waste to collect\n> > and ort might be slower than recursive. But then again, that should\n> > be a case where both algorithms are \"nearly instantaneous\" (or would\n> > be if it weren't for your 3M+ index entry repo causing run_builtin()'s\n> > call to setup_git_directory() in git.c to take a huge amount of time\n> > before the builtin is even called.)\n>\n> Thanks for your time isolating this case. I appreciate knowing exactly\n> which portions of the merge algorithm are being touched and which are\n> not.\n> > 5 seconds. I do have to hand it to Ben and anyone else involved,\n> > though. From 1 hour down to 5 seconds is pretty good, even if it was\n> > done by hacks (turning off rename detection, and then implementing\n> > trivial-tree merging that would have broken rename detection). I\n> > suspect that whoever did that work might have found the unconditional\n> > discarding and re-reading of the index and fixed it as well?\n>\n> As you can probably tell from my general confusion, I had nothing\n> to do with it. ;)\n>\n> > Heh, yeah 0.002 seconds for ..label:incore_recursive. Only 2\n> > milliseconds to create the actual merge tree. That does suggest you\n> > might have fun with 'git log -p --remerge-diff'; if you can redo\n> > merges in 2 milliseconds, showing them in git log output is very\n> > reasonable. :-)\n>\n> Yeah, 'git merge-tree' is very fast for these cases, so I assumed\n> that something else was going on for that command.\n\nOh, interesting. I forgot about merge-tree. Maybe I should make a\nversion based on merge-ort (and then it'd handle rename detection too,\nsomething it doesn't currently do.)? However, that wouldn't be\ncomparing merge algorithms, because builtin/merge-tree.c doesn't use\nmerge-recursive.[ch]. (It would be easy to get confused into thinking\nit does, since merge-recursive.[ch] defines a function called\nmerge_trees(), but builtin/merge-tree.c doesn't use it despite the\nname similarity.)\n\n> > Could we have some fun, though? What if you have some merge or rebase\n> > involving lots of changes, and you turn rename detection back on, and\n> > you disable that trivial-tree resolution optimization that breaks\n> > recursive's rename detection handling...and then compare recursive and\n> > ort? (It might be easiest to just compare upstream recursive rather\n> > than the one with all the microsoft changes to make sure you undid\n> > whatever trivial tree handling work exists.)\n>\n> I can try these kinds of cases, but it won't be today. I'm on kid duty\n> today, and answering emails in between running around with them.\n\nOne word of caution: merge.renameLimit may get in your way. The\ndefault of 1000 means that you're likely to hit that limit on your\nfirst run, and get a warning message like the following printed out:\n\nwarning: inexact rename detection was skipped due to too many files.\nwarning: you may want to set your merge.renamelimit variable to at\nleast 27328 and retry the command.\n\nYou then need to undo your rebase or merge, bump the limit, and\nre-run. Also, you will need a higher limit for merge-recursive than\nyou do for merge-ort. The default of 1000 is enough for merge-ort to\ndetect all the renames in my 26K-files-in-a-directory rename testcase\nof the linux kernel, but the value needs to be bumped to 27328 for\nmerge-recursive. And if you don't have the limit high enough, then\none algorithm is doing the work to detect renames and the other is\nbailing and skipping it, so it's not an apples-to-apples comparison.\nIf that warning doesn't appear for either backend, then you have an\napples-to-apples comparison.\n\n> > For example, my testcase in the linux kernel was finding a series of a\n> > few dozen patches I could rebase back to an older version, but\n> > tweaking the \"older\" version by renaming drivers/ -> pilots/ (with\n> > about 26K files under that directory, that meant about 26K renames).\n> > So, I got to see rebasing of dozens of real changes across a massive\n> > rename boundary -- and the massive rename boundary also guaranteed\n> > there were lots of entries for the merge algorithm to deal with.\n> >\n> > In the end, though, 4 milliseconds for the rebase and 2 milliseconds\n> > for the merge, with the rest all being overhead of interfacing to the\n> > index and working tree actually seems pretty good to me. I'm just\n> > curious if we can check how things work for more involved cases.\n>\n> I'm definitely interested in identifying how your algorithm improves\n> over the previous cases, and perhaps re-enabling rename detection for\n> merges is enough of a benefit to justify the new one.\n>\n> Eventually, I hope to actually engage with your patches in the form\n> of review. Just trying to build a mental model for what's going on\n> first.\n\nOoh, I can help with that; here's what's going on: *** Magic ***\n\n(Black, evil magic in the case of merge-recurisve. Good magic in the\ncase of merge-ort.)\n\nGlad I could help clear things up for you. :-)\n"},{"id":"409432","messageId":"CABPp-BFNk5ZQLyo5G-7vq3XD3s6z_rGUNM3QL14hXvMYTzK_Ug@mail.gmail.com","threadId":"54556","inReplyTo":"20201109220431.2534786-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 08/20] merge-ort: compute a few more useful fields for collect_merge_info","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-09T23:05:48Z","receivedAt":"2020-11-09T23:06:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Nov 9, 2020 at 2:04 PM Jonathan Tan wrote:\n>\n> > On Fri, Nov 6, 2020 at 2:52 PM Jonathan Tan wrote:\n> > >\n> > > > + /*\n> > > > + * Note: We only label files with df_conflict, not directories.\n> > > > + * Since directories stay where they are, and files move out of the\n> > > > + * way to make room for a directory, we don't care if there was a\n> > > > + * directory/file conflict for a parent directory of the current path.\n> > > > + */\n> > > > + unsigned df_conflict = (filemask != 0) && (dirmask != 0);\n> > >\n> > > Suppose you have:\n> > >\n> > > [ours]\n> > > foo/\n> > > bar/\n> > > baz\n> > > quux\n> > > [theirs]\n> > > foo\n> > >\n> > > By \"we only label files with df_conflict, not directories\", are you\n> > > referring to not labelling \"foo/\" in [ours], or to \"bar/\", \"baz\", and\n> > > \"quux\" (so, the files and directories within a directory)? At first I\n> > > thought you were referring to the former, but perhaps you are referring\n> > > to the latter.\n> >\n> > The former. I was drawing a distinction between how this code\n> > operates, and how unpack_trees() operates, which probably only matters\n> > to those familiar with unpack_trees() or who have been reading through\n> > it recently.\n>\n> Just for clarification: do you mean \"the latter\"? (The \"not\" in my\n> question might be confusing.)\n\nYeah, probably was confusing, so let me just state where you are\nalmost right below.\n\n> To be more illustrative in what I meant, at first I thought that you\n> were NOT labelling \"foo/\" in [ours], hence:\n>\n> [ours]\n> foo/ <- unlabeled\n> [theirs]\n> foo <- labeled\n>\n> In this way, in a sense, you are indeed labelling only the file, not the\n> directory.\n>\n> But instead what you seem to be doing is this:\n>\n> [ours]\n> foo/ <- labeled\n> bar/ <- unlabeled\n> baz <- unlabeled\n> quux <- unlabeled\n> [theirs]\n> foo <- labeled\n>\n> which is what I meant by NOT labelling \"bar/\", \"baz\", and \"quux\".\n\nI'm doing something /really/ close to this, yes. However, just to be\npedantic, there is no \"foo/\". '/' is an illegal character in a\nfilename to record in a tree. One side has a \"foo\" whose mode and\nobject_id happen to reflect a tree rather than a blob. But I only\nhave one conflict_info per pathname, not 3 (can't have three since\nstrmaps don't allow duplicate keys, and wouldn't want it if I could).\nThat one conflict_info stores 3 (mode, object_id) pairs, and also has\na single df_conflict bit. So, I label \"foo\" by setting that\ndf_conflict bit. But I only pay attention to it for the pairs\nrepresenting a blob, not the ones representing a tree. And I don't\npropagate the information down to paths below the foo directory.\n\n> > unpack_trees() will note when there is a directory/file\n> > conflict, and propagates that information to all subtrees, with every\n> > path specially checking for the o->df_conflict_entry and then handling\n> > it specially (e.g. keeping higher order stages instead of using an\n> > aggressive or trivial resolutions).\n>\n> And here it seems like you're describing that unpack_trees() would label\n> it in this way:\n>\n> [ours]\n> foo/ <- labeled\n> bar/ <- labeled\n> baz <- labeled\n> quux <- labeled\n> [theirs]\n> foo <- labeled\n>\n> (and you're emphasizing by contrast that merge-ort is NOT doing this).\n\nCorrect.\n\n> > However, leaving both a file and\n> > a directory at the same path, while allowed in the index, makes for\n> > ugliness and difficulty for users to resolve. Plus it isn't allowed\n> > in the working tree anyway. We decided a while ago that it'd be\n> > better to represent these conflicts differently[1], [2].\n> >\n> > Also, at the time you are unpacking or traversing trees, you only know\n> > if one side had a directory where the other side had a file. You\n> > don't know if the final merge result will actually have a\n> > directory/file conflict. If the file existed in both the base version\n> > and unmodified on one side, for example, then the file will be removed\n> > as part of the merge. It is similarly possible that the entire\n> > directory of files all need to be deleted or are all renamed\n> > elsewhere. So, you have to keep track of a df_conflict bit, but you\n> > can't act on it until you've processed several other things first.\n> >\n> > Since I already know I'm not going to move a whole directory of files\n> > out of the way so that a file can be placed in the working tree\n> > instead of that whole directory, the directory doesn't need to be\n> > tweaked. I'm not going to propagate any information about a\n> > directory/file conflict at some path down to all subpaths of the\n> > directory. I only track it for the file that immediately conflicts,\n> > and then only take action on it after resolving all the paths under\n> > the corresponding directory to see if the directory/file conflict\n> > remains.\n> >\n> > [1] https://lore.kernel.org/git/xmqqbmabcuhf.fsf@gitster-ct.c.googlers.com/\n> > and the thread surrounding it\n> > [2] https://lore.kernel.org/git/f27f12e8e50e56c010c29caa00296475d4de205b.1603731704.git.gitgitgadget@gmail.com/,\n> > which is now commit ef52778708 (\"merge tests: expect improved\n> > directory/file conflict handling in ort\", 2020-10-26)\n>\n> Makes sense.\n>\n> > > > @@ -161,6 +179,13 @@ static int collect_merge_info_callback(int n,\n> > > > newinfo.name = p->path;\n> > > > newinfo.namelen = p->pathlen;\n> > > > newinfo.pathlen = st_add3(newinfo.pathlen, p->pathlen, 1);\n> > > > + /*\n> > > > + * If we did care about parent directories having a D/F\n> > > > + * conflict, then we'd include\n> > > > + * newinfo.df_conflicts |= (mask & ~dirmask);\n> > > > + * here. But we don't. (See comment near setting of local\n> > > > + * df_conflict variable near the beginning of this function).\n> > > > + */\n> > >\n> > > I'm not sure how \"mask\" and \"dirmask\" contains information about parent\n> > > directories. \"mask\" represents the available entries, and \"dirmask\"\n> > > represents which of them are directories, as far as I know. So we can\n> > > notice when something is missing, but I don't see how this distinguishes\n> > > between the case that something is missing because it was in a parent\n> > > directory that got deleted, vs something is missing because it itself\n> > > got deleted.\n> >\n> > Yeah, this is more comparisons to unpack_trees. This code is about to\n> > set up a recursive call into subdirectories. newinfo is set based on\n> > the mask and dirmask of the current entry, and then subdirectories can\n> > consult newinfo.df_conflicts to see if that path is within a directory\n> > that was involved in a directory/file conflict. For example:\n> >\n> > Tree in base version:\n> > foo/\n> > bar\n> > stuff.txt\n> > Tree on side 1: (adds foo/baz)\n> > foo/\n> > bar\n> > baz\n> > stuff.txt\n> > Tree on side 2: (deletes foo/, adds new file foo)\n> > foo\n> > stuff.txt\n> >\n> > When processing 'foo', we have mask=7, dirmask = 3. So, here\n> > unpack_trees() would have set newinfo.df_conflicts = (mask & ~dirmask)\n> > = 4. Then when we process foo/bar or foo/baz, we have mask=2,\n> > dirmask=0, which looks like there are no directory/file conflicts.\n> > However, we can note that these paths are under a directory involved\n> > in a directory/file conflict via info.df_conflicts whose value is 4.\n> > unpack_trees() cared about paths under a directory that was involved\n> > in a directory/file conflict, and someone familiar with that code\n> > might ask why I don't also track the same information. The answer is\n> > that I don't track it, even though I thought about it, because it's\n> > useless overhead since I'm going to leave the directory alone and move\n> > the file out of the way.\n> >\n> > Does that make sense?\n>\n> Ah...yes, that makes sense. I think I didn't notice the \"newinfo\", so I\n> didn't realize that we were setting the info of our children based on\n> ourselves. Perhaps I would have noticed it sooner if the comment had\n> read \"If this file/directory cared about its parent directory (the\n> current directory) having a D/F conflict, then we'd propagate the masks\n> in this way:\" instead of \"If we did care about parent directories having\n> a D/F conflict\", but perhaps the point is already obvious enough.\n\nI'm happy to reword it if that makes it clearer. Thanks for the suggestion.\n"},{"id":"409433","messageId":"CABPp-BE=4uTRyxsuBUz7iB7+kqKjtM1-J3qiSi4q1nvwC1Odvg@mail.gmail.com","threadId":"54556","inReplyTo":"20201109220903.2536706-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 09/20] merge-ort: record stage and auxiliary info for every path","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-09T23:08:58Z","receivedAt":"2020-11-09T23:09:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Nov 9, 2020 at 2:09 PM Jonathan Tan wrote:\n>\n> > > So these are placed in paths but not unmerged. I'm starting to wonder if\n> > > struct merge_options_internal should be called merge_options_state or\n> > > something, and each field having documentation about when they're used\n> > > (or better yet, have functions like collect_merge_info() return their\n> > > calculations in return values (which may be \"out\" parameters) instead of\n> > > in this struct).\n> >\n> > Right, unmerged is only those paths that remain unmerged after all\n> > steps. record_unmerged_index_entries() could simply walk over all\n> > entries in paths and pick out the ones that were unmerged, but\n> > process_entries() has to walk over all paths, determine whether they\n> > can be merged, and determine what to record for the resulting tree for\n> > each path. So, having it stash away the unmerged stuff is a simple\n> > optimization.\n> >\n> > Renaming to merge_options_state or even just merge_state would be fine\n> > -- but any renaming done here will also affect merge-recursive.[ch].\n> > See the definition of merge_options in merge-recursive. (For history,\n> > merge-recursive.h stuffed state into merge_options, which risked funny\n> > misusage patterns and made the API unnecessarily complex...and made it\n> > suggest that alternative algorithms needed to have the same state.\n> > So, the state was moved to a merge_options_internal struct. That's\n> > not to say we can't rename, but it does need to be done in\n> > merge-recursive as well.)\n>\n> Ah, I see.\n>\n> > As for having collect_merge_info() return their calculations in return\n> > values, would that just end with me returning a struct\n> > merge_options_internal? Or did you want each return value added to\n> > the function signature? Each return value in the function signature\n> > makes sense right now for this super-simplified initial 20 patches,\n> > but what about when this data structure gains all kind of\n> > rename-related state that is collected, updated, and passed between\n> > these areas? I'd have a huge number of \"out\" and \"in\" fields to every\n> > function. Eventually, merge_options_internal (or whatever it might be\n> > renamed to) expands to the following, where I have to first define an\n> > extra enum and two extra structs so that you know the definitions of\n> > new types that show up in merge_options_internal:\n>\n> [snip enums and structs]\n>\n> Good point. I should have realized that there would be much more to\n> track.\n>\n> > > > + result->string = fullpath;\n> > > > + result->util = path_info;\n> > > > +}\n> > > > +\n> > > > static int collect_merge_info_callback(int n,\n> > > > unsigned long mask,\n> > > > unsigned long dirmask,\n> > > > @@ -91,10 +136,12 @@ static int collect_merge_info_callback(int n,\n> > > > */\n> > > > struct merge_options *opt = info->data;\n> > > > struct merge_options_internal *opti = opt->priv;\n> > > > - struct conflict_info *ci;\n> > > > + struct string_list_item pi; /* Path Info */\n> > > > + struct conflict_info *ci; /* pi.util when there's a conflict */\n> > >\n> > > Looking ahead to patch 10, this seems more like \"pi.util unless we know\n> > > for sure that there's no conflict\".\n> >\n> > That's too long for the line to remain at 80 characters; it's 16\n> > characters over the limit. ;-)\n>\n> Well, you could move the description onto its own line :-)\n\n:-)\n"},{"id":"409621","messageId":"3f1cd4b6-695d-7524-4cb7-7c31f370fe36@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-4-newren@gmail.com","subject":"Re: [PATCH v2 03/20] merge-ort: port merge_start() from merge-recursive","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-11T13:52:59Z","receivedAt":"2020-11-11T13:53:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> merge_start() basically does a bunch of sanity checks, then allocates\n> and initializes opt->priv -- a struct merge_options_internal.\n> \n> Most the sanity checks are usable as-is. The allocation/intialization\n\ns/Most the/Most of the/\n\n> The weirdest part here is that merge-ort and merge-recursive use the\n> same struct merge_options, even though merge_options has a number of\n> fields that are oddly specific to merge-recursive's internal\n> implementation and don't even make sense with merge-ort's high-level\n> design (e.g. buffer_output, which merge-ort has to always do). I reused\n> the same data structure because:\n> * most the fields made sense to both merge algorithms\n> * making a new struct would have required making new enums or somehow\n> externalizing them, and that was getting messy.\n> * it simplifies converting the existing callers by not having to\n> have different code paths for merge_options setup.\n\nI think this is appropriate. The other option would be to split the\nstruct into \"common options\" and \"specific options\" but that starts\nto get messy if we add yet another merge strategy that changes what\nshould be \"common\". Hopefully we can group options within the struct\nmerge_options definition to assist with this?\n\nFor now, the assertions are a good approach.\n\n> I also marked detect_renames as ignored. We can revisit that later, but\n> in short: merge-recursive allowed turning off rename detection because\n> it was sometimes glacially slow. When you speed something up by a few\n> orders of magnitude, it's worth revisiting whether that justification is\n> still relevant. Besides, if folks find it's still too slow, perhaps\n> they have a better scaling case than I could find and maybe it turns up\n> some more optimizations we can add. If it still is needed as an option,\n> it is easy to add later.\n\nAs long as it is easy to add later, I don't see much of a problem. Usually\nadding a knob to disable a feature is necessary to mitigate risk, and here\nwe can simply adjust config to use the non-ort algorithm if we notice a data\nshape where rename detection makes the algorithm slow/unusable.\n\n> static void merge_start(struct merge_options *opt, struct merge_result *result)\n> {\n> -\tdie(\"Not yet implemented.\");\n> +\t/* Sanity checks on opt */\n> +\tassert(opt->repo);\n> +\n> +\tassert(opt->branch1 && opt->branch2);\n> +\n> +\tassert(opt->detect_directory_renames >= MERGE_DIRECTORY_RENAMES_NONE &&\n> +\t opt->detect_directory_renames <= MERGE_DIRECTORY_RENAMES_TRUE);\n> +\tassert(opt->rename_limit >= -1);\n> +\tassert(opt->rename_score >= 0 && opt->rename_score <= MAX_SCORE);\n> +\tassert(opt->show_rename_progress >= 0 && opt->show_rename_progress <= 1);\n> +\n> +\tassert(opt->xdl_opts >= 0);\n> +\tassert(opt->recursive_variant >= MERGE_VARIANT_NORMAL &&\n> +\t opt->recursive_variant <= MERGE_VARIANT_THEIRS);\n> +\n> +\t/*\n> +\t * detect_renames, verbosity, buffer_output, and obuf are ignored\n> +\t * fields that were used by \"recursive\" rather than \"ort\" -- but\n> +\t * sanity check them anyway.\n> +\t */\n> +\tassert(opt->detect_renames >= -1 &&\n> +\t opt->detect_renames <= DIFF_DETECT_COPY);\n> +\tassert(opt->verbosity >= 0 && opt->verbosity <= 5);\n> +\tassert(opt->buffer_output <= 2);\n> +\tassert(opt->obuf.len == 0);\n> +\n> +\tassert(opt->priv == NULL);\n> +\n> +\t/* Initialization of opt->priv, our internal merge data */\n> +\topt->priv = xcalloc(1, sizeof(*opt->priv));\n\nnit: I would insert an empty line between this code and the\nmulti-line comment below.\n\n> +\t/*\n> +\t * Although we initialize opt->priv->paths with strdup_strings=0,\n> +\t * that's just to avoid making yet another copy of an allocated\n> +\t * string. Putting the entry into paths means we are taking\n> +\t * ownership, so we will later free it.\n> +\t *\n> +\t * In contrast, unmerged just has a subset of keys from paths, so\n> +\t * we don't want to free those (it'd be a duplicate free).\n> +\t */\n> +\tstrmap_init_with_options(&opt->priv->paths, NULL, 0);\n> +\tstrmap_init_with_options(&opt->priv->unmerged, NULL, 0);\n> }\n\nThis approach looks fine to me.\n\nThanks,\n-Stolee\n\n"},{"id":"409622","messageId":"cd05d258-a526-9843-f646-d2d42f1b8cbc@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-5-newren@gmail.com","subject":"Re: [PATCH v2 04/20] merge-ort: use histogram diff","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-11T13:54:23Z","receivedAt":"2020-11-11T13:54:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> I have some ideas for using a histogram diff to improve content merges,\n> which fundamentally relies on the idea of a histogram. Since the diffs\n> are never displayed to the user but just used internally for merging,\n> the typical user preference shouldn't matter anyway, and I want to make\n> sure that all my testing works with this algorithm.\n> \n> Granted, I don't yet know if those ideas will pan out and I haven't even\n> tried any of them out yet, but it's easy to change the diff algorithm in\n> the future if needed or wanted. For now, just set it to histogram.\n\nIf you are not making use of the histogram yet, then could you set this\npatch aside until you _do_ use it? Or are there performance implications\nthat are also a side benefit?\n\nThanks,\n-Stolee\n"},{"id":"409623","messageId":"267dbe58-e319-0904-3552-bf56530181c3@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-6-newren@gmail.com","subject":"Re: [PATCH v2 05/20] merge-ort: add an err() function similar to one from merge-recursive","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-11T13:58:31Z","receivedAt":"2020-11-11T13:58:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> Various places in merge-recursive used an err() function when it hit\n> some kind of unrecoverable error. That code was from the reusable bits\n> of merge-recursive.c that we liked, such as merge_3way, writing object\n> files to the object store, reading blobs from the object store, etc. So\n> create a similar function to allow us to port that code over, and use it\n> for when we detect problems returned from collect_merge_info()'s\n> traverse_trees() call, which we will be adding next.\n> \n> Signed-off-by: Elijah Newren \n> ---\n> merge-ort.c | 27 ++++++++++++++++++++++++++-\n> 1 file changed, 26 insertions(+), 1 deletion(-)\n> \n> diff --git a/merge-ort.c b/merge-ort.c\n> index df97a54773..537da9f6df 100644\n> --- a/merge-ort.c\n> +++ b/merge-ort.c\n> @@ -61,11 +61,28 @@ struct conflict_info {\n> \tunsigned match_mask:3;\n> };\n> \n> +static int err(struct merge_options *opt, const char *err, ...)\n> +{\n> +\tva_list params;\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\n> +\tstrbuf_addstr(&sb, \"error: \");\n> +\tva_start(params, err);\n> +\tstrbuf_vaddf(&sb, err, params);\n> +\tva_end(params);\n> +\n> +\terror(\"%s\", sb.buf);\n> +\tstrbuf_release(&sb);\n> +\n> +\treturn -1;\n> +}\n> +\n\nThis seems simple enough to have a duplicate copy lying\naround. Do you anticipate that all common code will live\nin the same file eventually? Or will these two static err()\nmethod always be duplicated?\n\nAside: I wonder if these errors could be logged using trace2\nprimitives, to assist diagnosing problems with merges. CC'ing\nJeff Hostetler if he has an opinion.\n\n> static int collect_merge_info(struct merge_options *opt,\n> \t\t\t struct tree *merge_base,\n> \t\t\t struct tree *side1,\n> \t\t\t struct tree *side2)\n> {\n> +\t/* TODO: Implement this using traverse_trees() */\n> \tdie(\"Not yet implemented.\");\n> }\n\nThis comment looks to be applied to the wrong patch.\n\nThanks,\n-Stolee\n"},{"id":"409624","messageId":"42037753-c8c1-629c-3d99-d54842686b2e@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-7-newren@gmail.com","subject":"Re: [PATCH v2 06/20] merge-ort: implement a very basic collect_merge_info()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-11T14:38:01Z","receivedAt":"2020-11-11T14:38:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> +\t/* +1 in both of the following lines to include the NUL byte */\n> +\tfullpath = xmalloc(len+1);\n> +\tmake_traverse_path(fullpath, len+1, info, p->path, p->pathlen);\n\nnit: s/len+1/len + 1/g\n\n> +\t\tvoid *buf[3] = {NULL,};\n\nThis \"{NULL,}\" seems odd to me. I suppose there is a reason why it\nisn't \"{ NULL, NULL, NULL }\"?\n\n> +\t\tconst char *original_dir_name;\n> +\t\tint i, ret;\n> +\n> +\t\tci->match_mask &= filemask;\n> +\t\tnewinfo = *info;\n> +\t\tnewinfo.prev = info;\n> +\t\tnewinfo.name = p->path;\n> +\t\tnewinfo.namelen = p->pathlen;\n> +\t\tnewinfo.pathlen = st_add3(newinfo.pathlen, p->pathlen, 1);\n> +\n> +\t\tfor (i = 0; i < 3; i++, dirmask >>= 1) {\n\nThis multi-action iterator borders on \"too clever\". It seems like\nplacing \"dirmask >>= 1;\" or \"dirmask = dirmask >> 1;\" at the end\nof the block would be equivalent and less jarring to a reader.\n\nI was thinking it doesn't really matter, except that dirmask is not\nin the initializer or sentinel of the for(), so having it here does\nnot immediately make sense.\n\n(This has been too much writing for such an inconsequential line\nof code. Sorry.)\n\n> +\t\t\tconst struct object_id *oid = NULL;\n> +\t\t\tif (dirmask & 1)\n> +\t\t\t\toid = &names[i].oid;\n> +\t\t\tbuf[i] = fill_tree_descriptor(opt->repo, t + i, oid);\n> +\t\t}\n\n\n> static int collect_merge_info(struct merge_options *opt,\n> \t\t\t struct tree *merge_base,\n> \t\t\t struct tree *side1,\n> \t\t\t struct tree *side2)\n> {\n> -\t/* TODO: Implement this using traverse_trees() */\n> -\tdie(\"Not yet implemented.\");\n> +\tint ret;\n> +\tstruct tree_desc t[3];\n> +\tstruct traverse_info info;\n> +\tchar *toplevel_dir_placeholder = \"\";\n\nIt seems like this should be \"const char *\"\n\n> +\tinit_tree_desc(t+0, merge_base->buffer, merge_base->size);\n> +\tinit_tree_desc(t+1, side1->buffer, side1->size);\n> +\tinit_tree_desc(t+2, side2->buffer, side2->size);\n\nMore space issues: s/t+/t + /g\n\nI'm only really able to engage in this at a surface level, it\nseems, but maybe I'll have more to say as the implementation\ngrows.\n\nThanks,\n-Stolee\n"},{"id":"409625","messageId":"c4b9b64c-f0a0-5824-44b9-072370cfc465@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-8-newren@gmail.com","subject":"Re: [PATCH v2 07/20] merge-ort: avoid repeating fill_tree_descriptor() on the same tree","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-11T14:51:35Z","receivedAt":"2020-11-11T14:51:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> @@ -99,6 +99,15 @@ static int collect_merge_info_callback(int n,\n> \tunsigned mbase_null = !(mask & 1);\n> \tunsigned side1_null = !(mask & 2);\n> \tunsigned side2_null = !(mask & 4);\n> +\tunsigned side1_matches_mbase = (!side1_null && !mbase_null &&\n> +\t\t\t\t\tnames[0].mode == names[1].mode &&\n> +\t\t\t\t\toideq(&names[0].oid, &names[1].oid));\n> +\tunsigned side2_matches_mbase = (!side2_null && !mbase_null &&\n> +\t\t\t\t\tnames[0].mode == names[2].mode &&\n> +\t\t\t\t\toideq(&names[0].oid, &names[2].oid));\n> +\tunsigned sides_match = (!side1_null && !side2_null &&\n> +\t\t\t\tnames[1].mode == names[2].mode &&\n> +\t\t\t\toideq(&names[1].oid, &names[2].oid));\n\nIf the *_null values were in an array, instead, then all of these\nlines could be grouped as a macro:\n\n\tunsigned null_oid[3] = {\n\t\t!(mask & 1),\n\t\t!(mask & 2),\n\t\t!(mask & 4)\n\t};\n\n\t#define trivial_merge(i,j) (!null_oid[i] && !null_oid[j] && \\\n\t\t\t\t names[i].mode == names[j].mode && \\\n\t\t\t\t oideq(&names[i].oid, &names[j].oid))\n\n\tunsigned side1_matches_mbase = trivial_merge(0, 1);\n\tunsigned side2_matches_mbase = trivial_merge(0, 2);\n\tunsigned sides_match = trivial_merge(1, 2);\n\nI briefly considered making these last three an array, as well,\nexcept the loop below doesn't use 'i' in a symmetrical way:\n\n> +\t\t\tif (i == 1 && side1_matches_mbase)\n> +\t\t\t\tt[1] = t[0];\n> +\t\t\telse if (i == 2 && side2_matches_mbase)\n> +\t\t\t\tt[2] = t[0];\n> +\t\t\telse if (i == 2 && sides_match)\n> +\t\t\t\tt[2] = t[1];\n\nSince the 'i == 2' case has two possible options, it wouldn't be\npossible to just have 'side_matches[i]' here.\n\n> +\t\t\telse {\n> +\t\t\t\tconst struct object_id *oid = NULL;\n> +\t\t\t\tif (dirmask & 1)\n> +\t\t\t\t\toid = &names[i].oid;\n> +\t\t\t\tbuf[i] = fill_tree_descriptor(opt->repo,\n> +\t\t\t\t\t\t\t t + i, oid);\n> +\t\t\t}\n\nI do appreciate the reduced recursion here!\n\nThanks,\n-Stolee\n"},{"id":"409632","messageId":"94bf9b69-5d13-c914-fb1a-bce912018a63@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-10-newren@gmail.com","subject":"Re: [PATCH v2 09/20] merge-ort: record stage and auxiliary info for every path","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-11T15:26:43Z","receivedAt":"2020-11-11T15:26:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> +static void setup_path_info(struct merge_options *opt,\n> +\t\t\t struct string_list_item *result,\n> +\t\t\t const char *current_dir_name,\n> +\t\t\t int current_dir_name_len,\n> +\t\t\t char *fullpath, /* we'll take over ownership */\n> +\t\t\t struct name_entry *names,\n> +\t\t\t struct name_entry *merged_version,\n> +\t\t\t unsigned is_null, /* boolean */\n> +\t\t\t unsigned df_conflict, /* boolean */\n> +\t\t\t unsigned filemask,\n> +\t\t\t unsigned dirmask,\n> +\t\t\t int resolved /* boolean */)\n> +{\n> +\tstruct conflict_info *path_info;\n\nIn addition to my concerns below about 'conflict_info' versus\n'merged_info', I was doubly confused that 'result' in the parameter\nlist is given a variable named 'pi' for \"path info\" and result->util\neventually is equal to this path_info. What if we renamed 'result'\nto 'pi' for \"path info\" here, then operated on 'pi->util' in this\nmethod?\n\n> +\tpath_info = xcalloc(1, resolved ? sizeof(struct merged_info) :\n> +\t\t\t\t\t sizeof(struct conflict_info));\n\nHm. I'm happy to have a `struct merged_info *` pointing to a\n`struct conflict_info`, but the opposite seems very dangerous.\nPerhaps we should always use sizeof(struct conflict_info)?\n\nWe can use path_info->merged.clean to detect whether the rest of\nthe data is worth looking at. (Or, in your case, whether or not\nit is allocated.)\n\nI imagine that in a large repo we will need many of these structs,\nbut very few of them will actually need to be conflicts, so using\n'struct conflict_info' always will lead to memory bloat. But in\nthat case, would we not be better off with an array instead of a\nscattering of data across the heap?\n\nPerhaps 'struct conflict_info' shouldn't contain a 'struct merged_info'\nand instead be just the \"extra\" data. Then we could have a contiguous\narray of 'struct merged_info' values for most of the paths, but heap\npointers for 'struct conflict_info' as necessary.\n\nIt's also true that I haven't fully formed a mental model for how these\nare used in your algorithm, so I'll keep reading.\n\n> +\tpath_info->merged.directory_name = current_dir_name;\n> +\tpath_info->merged.basename_offset = current_dir_name_len;\n> +\tpath_info->merged.clean = !!resolved;\n> +\tif (resolved) {\n> +\t\tpath_info->merged.result.mode = merged_version->mode;\n> +\t\toidcpy(&path_info->merged.result.oid, &merged_version->oid);\n> +\t\tpath_info->merged.is_null = !!is_null;\n> +\t} else {\n> +\t\tint i;\n> +\n> +\t\tfor (i = 0; i < 3; i++) {\n> +\t\t\tpath_info->pathnames[i] = fullpath;\n> +\t\t\tpath_info->stages[i].mode = names[i].mode;\n> +\t\t\toidcpy(&path_info->stages[i].oid, &names[i].oid);\n> +\t\t}\n> +\t\tpath_info->filemask = filemask;\n> +\t\tpath_info->dirmask = dirmask;\n> +\t\tpath_info->df_conflict = !!df_conflict;\n> +\t}\n> +\tstrmap_put(&opt->priv->paths, fullpath, path_info);\n> +\tresult->string = fullpath;\n> +\tresult->util = path_info;\n\nThis is set in all cases, so should we use it everywhere? Naturally,\nthere might be a cost to the extra pointer indirection, so maybe we\ncreate a 'struct conflict_info *util' to operate on during this\nmethod, but set 'result->util = util' right after allocating so we\nknow how it should behave?\n\n> @@ -91,10 +136,12 @@ static int collect_merge_info_callback(int n,\n> \t */\n> \tstruct merge_options *opt = info->data;\n> \tstruct merge_options_internal *opti = opt->priv;\n> -\tstruct conflict_info *ci;\n> +\tstruct string_list_item pi; /* Path Info */\n> +\tstruct conflict_info *ci; /* pi.util when there's a conflict */\n\n...\n\n> +\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n> +\t\t\tnames, NULL, 0, df_conflict, filemask, dirmask, 0);\n> +\tci = pi.util;\n\nHere is the use of 'pi' that I was talking about earlier.\n\nThanks,\n-Stolee\n"},{"id":"409633","messageId":"88ff65b6-1529-3471-00c3-c91233d75ad9@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-11-newren@gmail.com","subject":"Re: [PATCH v2 10/20] merge-ort: avoid recursing into identical trees","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-11T15:31:44Z","receivedAt":"2020-11-11T15:31:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> +\t/*\n> +\t * If mbase, side1, and side2 all match, we can resolve early. Even\n> +\t * if these are trees, there will be no renames or anything\n> +\t * underneath.\n> +\t */\n> +\tif (side1_matches_mbase && side2_matches_mbase) {\n\nHere is a case where if you were not caring about renames you could prevent\nrecursion here when \"!renames && sides_match\". Something to think about.\n\n> +\t\t/* mbase, side1, & side2 all match; use mbase as resolution */\n> +\t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n> +\t\t\t\tnames, names+0, mbase_null, 0,\n> +\t\t\t\tfilemask, dirmask, 1);\n> +\t\treturn mask;\n> +\t}\n> +\n\n-Stolee\n\n"},{"id":"409637","messageId":"4432ae6c-90ae-90d9-218f-15856395efac@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-13-newren@gmail.com","subject":"Re: [PATCH v2 12/20] merge-ort: have process_entries operate in a defined order","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-11T16:09:38Z","receivedAt":"2020-11-11T16:09:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> We want to handle paths below a directory before needing to handle the\n> directory itself. Also, we want to handle the directory immediately\n> after the paths below it, so we can't use simple lexicographic ordering\n> from strcmp (which would insert foo.txt between foo and foo/file.c).\n> Copy string_list_df_name_compare() from merge-recursive.c, and set up a\n> string list of paths sorted by that function so that we can iterate in\n> the desired order.\n\nThis is at least the second time we've copied something from\nmerge-recursive.c. Should we be starting a merge-utils.[c|h] to group\nthese together under a common implementation?\n\n> +\t/* Put every entry from paths into plist, then sort */\n> \tstrmap_for_each_entry(&opt->priv->paths, &iter, e) {\n> +\t\tstring_list_append(&plist, e->key)->util = e->value;\n> +\t}\n\nnit: are braces required here?\n\n-Stolee\n"},{"id":"409638","messageId":"CABPp-BGtpXRSz+ngFz20j8W4qgpb8juogsLf6HF7b0-Pt=s6=g@mail.gmail.com","threadId":"54556","inReplyTo":"3f1cd4b6-695d-7524-4cb7-7c31f370fe36@gmail.com","subject":"Re: [PATCH v2 03/20] merge-ort: port merge_start() from merge-recursive","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-11T16:22:36Z","receivedAt":"2020-11-11T16:22:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Thanks for all the reviews and suggestions!\n\nI'll avoid commenting on the simple fixes that I'm just going to\napply, and instead concentrate on the bigger questions you have in my\nreply.\n\nOn Wed, Nov 11, 2020 at 5:53 AM Derrick Stolee wrote:\n>\n> On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> > The weirdest part here is that merge-ort and merge-recursive use the\n> > same struct merge_options, even though merge_options has a number of\n> > fields that are oddly specific to merge-recursive's internal\n> > implementation and don't even make sense with merge-ort's high-level\n> > design (e.g. buffer_output, which merge-ort has to always do). I reused\n> > the same data structure because:\n> > * most the fields made sense to both merge algorithms\n> > * making a new struct would have required making new enums or somehow\n> > externalizing them, and that was getting messy.\n> > * it simplifies converting the existing callers by not having to\n> > have different code paths for merge_options setup.\n>\n> I think this is appropriate. The other option would be to split the\n> struct into \"common options\" and \"specific options\" but that starts\n> to get messy if we add yet another merge strategy that changes what\n> should be \"common\". Hopefully we can group options within the struct\n> merge_options definition to assist with this?\n\nI think we should plan on merge-recursive.[ch] being deleted before a\nthird merge strategy comes along, so the common options might make\nsense. But then again, it sounds like work towards simultaneously\nsupporting two backends in perpetuity, which isn't at all the current\nplan[1].\n\nAs far as grouping and other cleanups, see the series included with\nmerge commit 280bd44551 (\"Merge branch 'en/merge-recursive-cleanup'\",\n2019-10-15), particularly commits ff1bfa2cd5 (\"merge-recursive: use\ncommon name for ancestors/common/base_list\", 2019-08-17), a779fb829b\n(\"merge-recursive: comment and reorder the merge_options fields\",\n2019-08-17), and 8599ab4574 (\"merge-recursive: consolidate unnecessary\nfields in merge_options\", 2019-08-17). I guess we could change from\ngrouping by option similarity and instead group by which are in use by\nwhich merge backends, but since the plan is to [eventually] kill\nmerge-recursive and then to just drop the unused or ignored fields, I\nthink helping users understand the purpose of the options (which\ngrouping-by-similarity aids with) is more important than grouping for\nthe purpose of reminding myself which ones to remove later.\n\n[1] https://lore.kernel.org/git/xmqqk1ydkbx0.fsf@gitster.mtv.corp.google.com/\n\n> For now, the assertions are a good approach.\n>\n> > I also marked detect_renames as ignored. We can revisit that later, but\n> > in short: merge-recursive allowed turning off rename detection because\n> > it was sometimes glacially slow. When you speed something up by a few\n> > orders of magnitude, it's worth revisiting whether that justification is\n> > still relevant. Besides, if folks find it's still too slow, perhaps\n> > they have a better scaling case than I could find and maybe it turns up\n> > some more optimizations we can add. If it still is needed as an option,\n> > it is easy to add later.\n>\n> As long as it is easy to add later, I don't see much of a problem. Usually\n> adding a knob to disable a feature is necessary to mitigate risk, and here\n> we can simply adjust config to use the non-ort algorithm if we notice a data\n> shape where rename detection makes the algorithm slow/unusable.\n\nYes, it should be pretty easy to add later.\n\n> > static void merge_start(struct merge_options *opt, struct merge_result *result)\n> > {\n> > - die(\"Not yet implemented.\");\n> > + /* Sanity checks on opt */\n> > + assert(opt->repo);\n> > +\n> > + assert(opt->branch1 && opt->branch2);\n> > +\n> > + assert(opt->detect_directory_renames >= MERGE_DIRECTORY_RENAMES_NONE &&\n> > + opt->detect_directory_renames <= MERGE_DIRECTORY_RENAMES_TRUE);\n> > + assert(opt->rename_limit >= -1);\n> > + assert(opt->rename_score >= 0 && opt->rename_score <= MAX_SCORE);\n> > + assert(opt->show_rename_progress >= 0 && opt->show_rename_progress <= 1);\n> > +\n> > + assert(opt->xdl_opts >= 0);\n> > + assert(opt->recursive_variant >= MERGE_VARIANT_NORMAL &&\n> > + opt->recursive_variant <= MERGE_VARIANT_THEIRS);\n> > +\n> > + /*\n> > + * detect_renames, verbosity, buffer_output, and obuf are ignored\n> > + * fields that were used by \"recursive\" rather than \"ort\" -- but\n> > + * sanity check them anyway.\n> > + */\n> > + assert(opt->detect_renames >= -1 &&\n> > + opt->detect_renames <= DIFF_DETECT_COPY);\n> > + assert(opt->verbosity >= 0 && opt->verbosity <= 5);\n> > + assert(opt->buffer_output <= 2);\n> > + assert(opt->obuf.len == 0);\n> > +\n> > + assert(opt->priv == NULL);\n> > +\n> > + /* Initialization of opt->priv, our internal merge data */\n> > + opt->priv = xcalloc(1, sizeof(*opt->priv));\n>\n> nit: I would insert an empty line between this code and the\n> multi-line comment below.\n>\n> > + /*\n> > + * Although we initialize opt->priv->paths with strdup_strings=0,\n> > + * that's just to avoid making yet another copy of an allocated\n> > + * string. Putting the entry into paths means we are taking\n> > + * ownership, so we will later free it.\n> > + *\n> > + * In contrast, unmerged just has a subset of keys from paths, so\n> > + * we don't want to free those (it'd be a duplicate free).\n> > + */\n> > + strmap_init_with_options(&opt->priv->paths, NULL, 0);\n> > + strmap_init_with_options(&opt->priv->unmerged, NULL, 0);\n> > }\n>\n> This approach looks fine to me.\n>\n> Thanks,\n> -Stolee\n"},{"id":"409646","messageId":"CABPp-BFd2Wg-9h9+Gx20szX=YfKLqvBMWGg-eZU0yL8_DPh6kw@mail.gmail.com","threadId":"54556","inReplyTo":"cd05d258-a526-9843-f646-d2d42f1b8cbc@gmail.com","subject":"Re: [PATCH v2 04/20] merge-ort: use histogram diff","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-11T16:47:20Z","receivedAt":"2020-11-11T16:47:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Nov 11, 2020 at 5:54 AM Derrick Stolee wrote:\n>\n> On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> > I have some ideas for using a histogram diff to improve content merges,\n> > which fundamentally relies on the idea of a histogram. Since the diffs\n> > are never displayed to the user but just used internally for merging,\n> > the typical user preference shouldn't matter anyway, and I want to make\n> > sure that all my testing works with this algorithm.\n> >\n> > Granted, I don't yet know if those ideas will pan out and I haven't even\n> > tried any of them out yet, but it's easy to change the diff algorithm in\n> > the future if needed or wanted. For now, just set it to histogram.\n>\n> If you are not making use of the histogram yet, then could you set this\n> patch aside until you _do_ use it? Or are there performance implications\n> that are also a side benefit?\n\nLong story...\n\ngit folks tend to value performance pretty strongly -- including\nsometimes valuing it OVER correctness. For example, if fast-export\ncompletely munges some merge commits and you send your first ever\npatch in to the list to fix it (by turning on topo_order), you might\nrun into folks asking if it should be made an option so that we don't\nslow down exports except for people who happen to know they need it\n(and thus risk breaking exports for people who happen to be unaware\nthat they need it...). Luckily, Peff bailed me out in that situation\nby doing some timings and finding that topo_order actually made\nfast-export _faster_, to everyone's surprise at the time. Or, to take\nanother example, perhaps someone will introduce some commit-date\ncutoff on the revision walking machinery that sometimes breaks the\nanswer but makes things faster...and then causes a bunch of headaches\nyears down the road when someone tries to introduce commit-graphs to\nget us always correct _and_ fast answers.\n\nIn this case, histogram diffs in my cursory investigation are about 2%\nslower than Myers diffs. I think others may have done even more\ndetailed benchmarks. They've been around for years, but haven't been\nmade the default, despite giving obviously better looking diffs to\nusers in a number of cases where Myers diffs are unintelligible. But,\nfar more importantly, there are real merge bugs we know about that are\neven affecting git.git and linux.git that I don't have a clue how to\naddress without the additional information that I believe is provided\nby histogram diffs. See the following:\n\nhttps://lore.kernel.org/git/20190816184051.GB13894@sigill.intra.peff.net/\nhttps://lore.kernel.org/git/CABPp-BHvJHpSJT7sdFwfNcPn_sOXwJi3=o14qjZS3M8Rzcxe2A@mail.gmail.com/\nhttps://lore.kernel.org/git/CABPp-BGtez4qjbtFT1hQoREfcJPmk9MzjhY5eEq1QhXT23tFOw@mail.gmail.com/\n\nI don't like mismerges. I really don't like silent mismerges. While\nI am sometimes willing to make performance and correctness tradeoff,\nI'm much more interested in correctness in general. I want to fix the\nabove bugs. I have not yet started doing so, but I believe histogram\ndiff at least gives me an angle. But I can't rely on using the\ninformation from histogram diff unless it's in use. And it hasn't\nbeen used because of a few percentage performance hit.\n\nBut, since I happen to be speeding up typical non-trivial\nmerges/rebases/cherry-picks by factors of 10 or more (at least,\nanywhere that read-and-update-the-index is a trivial percentage of\noverall time instead of 99.9% of overall time like it is for you), now\nis golden opportunity to switch out the underlying diff algorithm so\nthat I can get the data I need to fix the bugs I know are there.\nWhether I can actually fix them is yet to be seen; I won't even start\nuntil merge-ort is complete and merged.\n\nDoes that help?\n"},{"id":"409647","messageId":"91898d2a-174f-a9c6-f174-3c7809f79b34@gmail.com","threadId":"54556","inReplyTo":"CABPp-BFd2Wg-9h9+Gx20szX=YfKLqvBMWGg-eZU0yL8_DPh6kw@mail.gmail.com","subject":"Re: [PATCH v2 04/20] merge-ort: use histogram diff","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-11T16:51:19Z","receivedAt":"2020-11-11T16:51:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/11/2020 11:47 AM, Elijah Newren wrote:\n> On Wed, Nov 11, 2020 at 5:54 AM Derrick Stolee wrote:\n>>\n>> On 11/2/2020 3:43 PM, Elijah Newren wrote:\n>>> I have some ideas for using a histogram diff to improve content merges,\n>>> which fundamentally relies on the idea of a histogram. Since the diffs\n>>> are never displayed to the user but just used internally for merging,\n>>> the typical user preference shouldn't matter anyway, and I want to make\n>>> sure that all my testing works with this algorithm.\n>>>\n>>> Granted, I don't yet know if those ideas will pan out and I haven't even\n>>> tried any of them out yet, but it's easy to change the diff algorithm in\n>>> the future if needed or wanted. For now, just set it to histogram.\n>>\n>> If you are not making use of the histogram yet, then could you set this\n>> patch aside until you _do_ use it? Or are there performance implications\n>> that are also a side benefit?\n> \n> Long story...\n\n...\n\n> Does that help?\n\nIn summary, you have some concrete reasons to prefer the histogram\ndiff other than just \"I have some ideas that might pan out later\" so\nthis code change is a good one but could be better justified in the\ncommit message. Does that sound correct?\n\nThanks,\n-Stolee\n\n"},{"id":"409648","messageId":"CABPp-BFGs-uMNT3Ge6ZttMTxNEu-S3cdwU74XjNYYent3PQi6g@mail.gmail.com","threadId":"54556","inReplyTo":"42037753-c8c1-629c-3d99-d54842686b2e@gmail.com","subject":"Re: [PATCH v2 06/20] merge-ort: implement a very basic collect_merge_info()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-11T17:02:27Z","receivedAt":"2020-11-11T17:02:45Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Nov 11, 2020 at 6:38 AM Derrick Stolee wrote:\n>\n> On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> > + /* +1 in both of the following lines to include the NUL byte */\n> > + fullpath = xmalloc(len+1);\n> > + make_traverse_path(fullpath, len+1, info, p->path, p->pathlen);\n>\n> nit: s/len+1/len + 1/g\n>\n> > + void *buf[3] = {NULL,};\n>\n> This \"{NULL,}\" seems odd to me. I suppose there is a reason why it\n> isn't \"{ NULL, NULL, NULL }\"?\n\nProbably because I was copying from unpack-trees.c, which deals with a\nvariable number of trees instead of always exactly 3. But yeah, it'd\nprobably be more straightforward as { NULL, NULL, NULL }.\n\n> > + const char *original_dir_name;\n> > + int i, ret;\n> > +\n> > + ci->match_mask &= filemask;\n> > + newinfo = *info;\n> > + newinfo.prev = info;\n> > + newinfo.name = p->path;\n> > + newinfo.namelen = p->pathlen;\n> > + newinfo.pathlen = st_add3(newinfo.pathlen, p->pathlen, 1);\n> > +\n> > + for (i = 0; i < 3; i++, dirmask >>= 1) {\n>\n> This multi-action iterator borders on \"too clever\". It seems like\n> placing \"dirmask >>= 1;\" or \"dirmask = dirmask >> 1;\" at the end\n> of the block would be equivalent and less jarring to a reader.\n>\n> I was thinking it doesn't really matter, except that dirmask is not\n> in the initializer or sentinel of the for(), so having it here does\n> not immediately make sense.\n>\n> (This has been too much writing for such an inconsequential line\n> of code. Sorry.)\n\nYeah, copied from unpack-trees.c:traverse_trees_recursive(). The\nnewinfo variable name and a bunch of the surrounding lines were copied\nfrom there too. I can switch it, though, if it makes it easier.\n\n> > + const struct object_id *oid = NULL;\n> > + if (dirmask & 1)\n> > + oid = &names[i].oid;\n> > + buf[i] = fill_tree_descriptor(opt->repo, t + i, oid);\n> > + }\n>\n>\n> > static int collect_merge_info(struct merge_options *opt,\n> > struct tree *merge_base,\n> > struct tree *side1,\n> > struct tree *side2)\n> > {\n> > - /* TODO: Implement this using traverse_trees() */\n> > - die(\"Not yet implemented.\");\n> > + int ret;\n> > + struct tree_desc t[3];\n> > + struct traverse_info info;\n> > + char *toplevel_dir_placeholder = \"\";\n>\n> It seems like this should be \"const char *\"\n>\n> > + init_tree_desc(t+0, merge_base->buffer, merge_base->size);\n> > + init_tree_desc(t+1, side1->buffer, side1->size);\n> > + init_tree_desc(t+2, side2->buffer, side2->size);\n>\n> More space issues: s/t+/t + /g\n\nIn my defense:\n\n$ git grep init_tree_desc.*t.*\\+ | grep -v merge-ort\nbuiltin/merge.c: init_tree_desc(t+i, trees[i]->buffer, trees[i]->size);\nbuiltin/read-tree.c: init_tree_desc(t+i, tree->buffer, tree->size);\nmerge-recursive.c: init_tree_desc_from_tree(t+0, common);\nmerge-recursive.c: init_tree_desc_from_tree(t+1, head);\nmerge-recursive.c: init_tree_desc_from_tree(t+2, merge);\nmerge.c: init_tree_desc(t+i, trees[i]->buffer, trees[i]->size);\n\nNone of which blames to me. :-)\n\nI can fix it up, though...at least the merge-ort one. Someone else\ncan go through existing code if they so desire.\n\n> I'm only really able to engage in this at a surface level, it\n> seems, but maybe I'll have more to say as the implementation\n> grows.\n\nIt _might_ be helpful to compare to unpack-trees.c's unpack_callback()\nand traverse_trees_recursive(), but there's so much unrelated stuff\nthere that it's possible that just gets in the way more than it helps.\nRegardless, thanks for taking a look and spotting little fixes; every\nbit helps.\n"},{"id":"409649","messageId":"CABPp-BENqoXL9yPW1DnoEa89oGxM2J1H4Vz3VE5r2-D+-m=PKA@mail.gmail.com","threadId":"54556","inReplyTo":"91898d2a-174f-a9c6-f174-3c7809f79b34@gmail.com","subject":"Re: [PATCH v2 04/20] merge-ort: use histogram diff","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-11T17:03:50Z","receivedAt":"2020-11-11T17:04:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Nov 11, 2020 at 8:51 AM Derrick Stolee wrote:\n>\n> On 11/11/2020 11:47 AM, Elijah Newren wrote:\n> > On Wed, Nov 11, 2020 at 5:54 AM Derrick Stolee wrote:\n> >>\n> >> On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> >>> I have some ideas for using a histogram diff to improve content merges,\n> >>> which fundamentally relies on the idea of a histogram. Since the diffs\n> >>> are never displayed to the user but just used internally for merging,\n> >>> the typical user preference shouldn't matter anyway, and I want to make\n> >>> sure that all my testing works with this algorithm.\n> >>>\n> >>> Granted, I don't yet know if those ideas will pan out and I haven't even\n> >>> tried any of them out yet, but it's easy to change the diff algorithm in\n> >>> the future if needed or wanted. For now, just set it to histogram.\n> >>\n> >> If you are not making use of the histogram yet, then could you set this\n> >> patch aside until you _do_ use it? Or are there performance implications\n> >> that are also a side benefit?\n> >\n> > Long story...\n>\n> ...\n>\n> > Does that help?\n>\n> In summary, you have some concrete reasons to prefer the histogram\n> diff other than just \"I have some ideas that might pan out later\" so\n> this code change is a good one but could be better justified in the\n> commit message. Does that sound correct?\n\nSure, I can add some additional wording about those ideas and the\nconcrete issues I want to fix.\n"},{"id":"409650","messageId":"CABPp-BER+nzGb42fB2__tF=a8Y-JQ-Fzi8PZ9T6nWG-5RZfzhA@mail.gmail.com","threadId":"54556","inReplyTo":"267dbe58-e319-0904-3552-bf56530181c3@gmail.com","subject":"Re: [PATCH v2 05/20] merge-ort: add an err() function similar to one from merge-recursive","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-11T17:07:18Z","receivedAt":"2020-11-11T17:07:32Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Nov 11, 2020 at 5:58 AM Derrick Stolee wrote:\n\n> > +static int err(struct merge_options *opt, const char *err, ...)\n> > +{\n> > + va_list params;\n> > + struct strbuf sb = STRBUF_INIT;\n> > +\n> > + strbuf_addstr(&sb, \"error: \");\n> > + va_start(params, err);\n> > + strbuf_vaddf(&sb, err, params);\n> > + va_end(params);\n> > +\n> > + error(\"%s\", sb.buf);\n> > + strbuf_release(&sb);\n> > +\n> > + return -1;\n> > +}\n> > +\n>\n> This seems simple enough to have a duplicate copy lying\n> around. Do you anticipate that all common code will live\n> in the same file eventually? Or will these two static err()\n> method always be duplicated?\n\nI anticipate that merge-recursive.[ch] will be deleted.\n\nmerge-ort was what I wanted to modify merge-recursive.[ch] to be, but\nJunio suggested doing it as a different merge backend since it had\ninvasive changes, so that we could have an easy way to try it out and\nfallback to the known good algorithm until we had sufficient comfort\nwith the new algorithm to switch over to it.\n\n> Aside: I wonder if these errors could be logged using trace2\n> primitives, to assist diagnosing problems with merges. CC'ing\n> Jeff Hostetler if he has an opinion.\n>\n> > static int collect_merge_info(struct merge_options *opt,\n> > struct tree *merge_base,\n> > struct tree *side1,\n> > struct tree *side2)\n> > {\n> > + /* TODO: Implement this using traverse_trees() */\n> > die(\"Not yet implemented.\");\n> > }\n>\n> This comment looks to be applied to the wrong patch.\n\nOops! You are correct; will fix.\n"},{"id":"409651","messageId":"75170ee7-525e-31fc-f6bd-6dfac12b00c8@gmail.com","threadId":"54556","inReplyTo":"20201102204344.342633-1-newren@gmail.com","subject":"Re: [PATCH v2 00/20] fundamentals of merge-ort implementation","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-11T17:08:58Z","receivedAt":"2020-11-11T17:09:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> Elijah Newren (20):\n> merge-ort: setup basic internal data structures\n> merge-ort: add some high-level algorithm structure\n> merge-ort: port merge_start() from merge-recursive\n> merge-ort: use histogram diff\n> merge-ort: add an err() function similar to one from merge-recursive\n> merge-ort: implement a very basic collect_merge_info()\n> merge-ort: avoid repeating fill_tree_descriptor() on the same tree\n> merge-ort: compute a few more useful fields for collect_merge_info\n> merge-ort: record stage and auxiliary info for every path\n> merge-ort: avoid recursing into identical trees\n> merge-ort: add a preliminary simple process_entries() implementation\n> merge-ort: have process_entries operate in a defined order\n\nI got this far before my attention to detail really started slipping.\n\n> merge-ort: step 1 of tree writing -- record basenames, modes, and oids\n> merge-ort: step 2 of tree writing -- function to create tree object\n> merge-ort: step 3 of tree writing -- handling subdirectories as we go\n> merge-ort: basic outline for merge_switch_to_result()\n> merge-ort: add implementation of checkout()\n> tree: enable cmp_cache_name_compare() to be used elsewhere\n> merge-ort: add implementation of record_unmerged_index_entries()\n> merge-ort: free data structures in merge_finalize()\n\nI'll try to take another pass on these commits tomorrow.\n\nFor the series as a whole I'd love to see at least one test that\ndemonstrates that this code does something, if even only for a very\nnarrow case.\n\nThere's a lot of code being moved here, and it would be nice to have\neven a very simple test case that can check that we didn't leave any\nimportant die(\"not implemented\") calls lying around or worse accessing\nan uninitialized pointer or something.\n\nThanks,\n-Stolee\n"},{"id":"409652","messageId":"acb6291f-a9a4-6c74-61ab-77aa6e7bfc77@gmail.com","threadId":"54556","inReplyTo":"CABPp-BER+nzGb42fB2__tF=a8Y-JQ-Fzi8PZ9T6nWG-5RZfzhA@mail.gmail.com","subject":"Re: [PATCH v2 05/20] merge-ort: add an err() function similar to one from merge-recursive","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-11T17:10:03Z","receivedAt":"2020-11-11T17:10:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/11/2020 12:07 PM, Elijah Newren wrote:\n> On Wed, Nov 11, 2020 at 5:58 AM Derrick Stolee wrote:\n>> This seems simple enough to have a duplicate copy lying\n>> around. Do you anticipate that all common code will live\n>> in the same file eventually? Or will these two static err()\n>> method always be duplicated?\n> \n> I anticipate that merge-recursive.[ch] will be deleted.\n> \n> merge-ort was what I wanted to modify merge-recursive.[ch] to be, but\n> Junio suggested doing it as a different merge backend since it had\n> invasive changes, so that we could have an easy way to try it out and\n> fallback to the known good algorithm until we had sufficient comfort\n> with the new algorithm to switch over to it.\n\nOK, I missed that context. Your approach is fine as long as these\nare not going to both exist forever.\n\nThanks,\n-Stolee\n"},{"id":"409653","messageId":"CABPp-BGk8FBt8WWw_h8t2jiE4quBoMeaco19F-fJ6V_vOnsNww@mail.gmail.com","threadId":"54556","inReplyTo":"c4b9b64c-f0a0-5824-44b9-072370cfc465@gmail.com","subject":"Re: [PATCH v2 07/20] merge-ort: avoid repeating fill_tree_descriptor() on the same tree","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-11T17:13:02Z","receivedAt":"2020-11-11T17:13:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Nov 11, 2020 at 6:51 AM Derrick Stolee wrote:\n>\n> On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> > @@ -99,6 +99,15 @@ static int collect_merge_info_callback(int n,\n> > unsigned mbase_null = !(mask & 1);\n> > unsigned side1_null = !(mask & 2);\n> > unsigned side2_null = !(mask & 4);\n> > + unsigned side1_matches_mbase = (!side1_null && !mbase_null &&\n> > + names[0].mode == names[1].mode &&\n> > + oideq(&names[0].oid, &names[1].oid));\n> > + unsigned side2_matches_mbase = (!side2_null && !mbase_null &&\n> > + names[0].mode == names[2].mode &&\n> > + oideq(&names[0].oid, &names[2].oid));\n> > + unsigned sides_match = (!side1_null && !side2_null &&\n> > + names[1].mode == names[2].mode &&\n> > + oideq(&names[1].oid, &names[2].oid));\n>\n> If the *_null values were in an array, instead, then all of these\n> lines could be grouped as a macro:\n>\n> unsigned null_oid[3] = {\n> !(mask & 1),\n> !(mask & 2),\n> !(mask & 4)\n> };\n>\n> #define trivial_merge(i,j) (!null_oid[i] && !null_oid[j] && \\\n> names[i].mode == names[j].mode && \\\n> oideq(&names[i].oid, &names[j].oid))\n>\n> unsigned side1_matches_mbase = trivial_merge(0, 1);\n> unsigned side2_matches_mbase = trivial_merge(0, 2);\n> unsigned sides_match = trivial_merge(1, 2);\n\nHmm, I like it. I think I'll rename trivial_merge() to\nnon_null_match() (trivial merge suggests it can immediately be\nresolved which is not necessarily true if rename detection is on), but\notherwise I'll use this.\n\n> I briefly considered making these last three an array, as well,\n> except the loop below doesn't use 'i' in a symmetrical way:\n>\n> > + if (i == 1 && side1_matches_mbase)\n> > + t[1] = t[0];\n> > + else if (i == 2 && side2_matches_mbase)\n> > + t[2] = t[0];\n> > + else if (i == 2 && sides_match)\n> > + t[2] = t[1];\n>\n> Since the 'i == 2' case has two possible options, it wouldn't be\n> possible to just have 'side_matches[i]' here.\n>\n> > + else {\n> > + const struct object_id *oid = NULL;\n> > + if (dirmask & 1)\n> > + oid = &names[i].oid;\n> > + buf[i] = fill_tree_descriptor(opt->repo,\n> > + t + i, oid);\n> > + }\n>\n> I do appreciate the reduced recursion here!\n\nTechnically, not my own optimization; I just copied from\nunpack-trees.c:traverse_trees_recursive() -- though the code looks\nslightly different because I didn't want to compare oids multiple\ntimes (I use the side match variables earlier in the function as\nwell).\n"},{"id":"409654","messageId":"CAPig+cSrcodAJu9S82vogt05priRDj1vBpaZUTFfhJraEzKvZA@mail.gmail.com","threadId":"54556","inReplyTo":"CABPp-BGk8FBt8WWw_h8t2jiE4quBoMeaco19F-fJ6V_vOnsNww@mail.gmail.com","subject":"Re: [PATCH v2 07/20] merge-ort: avoid repeating fill_tree_descriptor() on the same tree","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2020-11-11T17:21:13Z","receivedAt":"2020-11-11T17:21:27Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Nov 11, 2020 at 12:13 PM Elijah Newren wrote:\n> On Wed, Nov 11, 2020 at 6:51 AM Derrick Stolee wrote:\n> > If the *_null values were in an array, instead, then all of these\n> > lines could be grouped as a macro:\n> >\n> > unsigned null_oid[3] = {\n> > !(mask & 1),\n> > !(mask & 2),\n> > !(mask & 4)\n> > };\n>\n> Hmm, I like it. I think I'll rename trivial_merge() to\n> non_null_match() (trivial merge suggests it can immediately be\n> resolved which is not necessarily true if rename detection is on), but\n> otherwise I'll use this.\n\nAre we allowing non-constant array initializers in the codebase these\ndays? I don't see anything in CodingGuidelines suggesting the use of\nthem.\n"},{"id":"409660","messageId":"CABPp-BFRPuxLYz_n6jbr=j7Gu1GhsV95nKPE1=HxUcsvimrz0g@mail.gmail.com","threadId":"54556","inReplyTo":"94bf9b69-5d13-c914-fb1a-bce912018a63@gmail.com","subject":"Re: [PATCH v2 09/20] merge-ort: record stage and auxiliary info for every path","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-11T18:16:40Z","receivedAt":"2020-11-11T18:16:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Nov 11, 2020 at 7:26 AM Derrick Stolee wrote:\n>\n> On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> > +static void setup_path_info(struct merge_options *opt,\n> > + struct string_list_item *result,\n> > + const char *current_dir_name,\n> > + int current_dir_name_len,\n> > + char *fullpath, /* we'll take over ownership */\n> > + struct name_entry *names,\n> > + struct name_entry *merged_version,\n> > + unsigned is_null, /* boolean */\n> > + unsigned df_conflict, /* boolean */\n> > + unsigned filemask,\n> > + unsigned dirmask,\n> > + int resolved /* boolean */)\n> > +{\n> > + struct conflict_info *path_info;\n>\n> In addition to my concerns below about 'conflict_info' versus\n> 'merged_info', I was doubly confused that 'result' in the parameter\n> list is given a variable named 'pi' for \"path info\" and result->util\n> eventually is equal to this path_info. What if we renamed 'result'\n> to 'pi' for \"path info\" here, then operated on 'pi->util' in this\n> method?\n\nresult->util (or pi->util if you rename) is void *, making it hard to\noperate on; you'd have to typecast at every usage. Since it is used a\n*lot*, it makes sense to have a typed pointer, and then just set\nresult->util to a copy of that value at the end. That is what\npath_info is for.\n\n>\n> > + path_info = xcalloc(1, resolved ? sizeof(struct merged_info) :\n> > + sizeof(struct conflict_info));\n>\n> Hm. I'm happy to have a `struct merged_info *` pointing to a\n> `struct conflict_info`, but the opposite seems very dangerous.\n\nYeah, this is perhaps the scariest bit, and if it were a side data\nstructure rather than the fundamental main one that was central to the\nalgorithm, then safety would trump performance concerns. But since it\nis the main data structure and likely the biggest (once you count the\nvarious copies for each relevant path), then it might be worth the\nextra care needed to shave off the extra memory. Maybe we can still\ntweak things to get some safety back without killing performance so\nlet me consider each of your suggestions/questions.\n\nIf I define it as a merged_info*, the compiler will only let me modify\nfields within the merged_info portion of the struct. Should I\ntypecast every line that touches the bits in the resolved==0 path\nwhere I need to set fields within the conflict_info portion?\nAlternatively, would a code flow like the following make you happier?\n\n struct conflict_info *ci = NULL;\n struct merge_info *mi = xcalloc(...);\n result->util = mi;\n /* Operate on mi */\n ...\n if (resolved)\n return;\n ci = mi;\n /* Operate on ci */\n ...\n\nIn either case, the returned item has potentially different sizes, so\nthe caller will still have to take care so I'm not sure how much extra\nthis structure within setup_path_info() buys us.\n\n> Perhaps we should always use sizeof(struct conflict_info)?\n\nWe could do that; it'd certainly waste memory as I expect many entries\nto be unmodified (on one or both sides of history). But I'd way\nrather go this route than splitting or re-arranging this data\nstructure.\n\n> We can use path_info->merged.clean to detect whether the rest of\n> the data is worth looking at. (Or, in your case, whether or not\n> it is allocated.)\n\nci->merged.clean is used to determine whether to look at the rest of\nthe data, yes -- and that's an enforced assumption throughout the code\n(as alluded to by the comment in the merge_options_internal data\nstructure that \"paths\" maps pathanemes to merge_info and conflict_info\ntypes). However, that is not quite the same as using the clean bit to\ndetermine if more data is allocated; something can be allocated as a\nconflict_info rather than a merged_info due to both sides making\nmodifying the same path, but then a threeway content merge comes back\nclean and ci->merged.clean is updated from 0 to 1. The extra data\nremains allocated, but nothing in the algorithm ever needs to use\nanything outside the merged bits for that path again. (Actually, let\nme state that more forcefully: nothing is *allowed* to look outside\nthe merged bits for that path once the clean bit is updated to 1).\n\n> I imagine that in a large repo we will need many of these structs,\n> but very few of them will actually need to be conflicts, so using\n> 'struct conflict_info' always will lead to memory bloat. But in\n> that case, would we not be better off with an array instead of a\n> scattering of data across the heap?\n\nNot sure what you're trying to solve here. Putting them in an array\nwould mean copying every single one of them every time the array is\nresized. It would also make insertion or deletion very expensive.\nAnd it'd prevent O(1) lookup. It'd be a horrible data structure all\naround. Maybe you're assuming you know exactly how many entries you\nneed and what they are before the merge algorithm starts? I don't.\nIn fact, I can't even give a good magnitude approximation of how many\nit'll be before a merge starts. (Even if you assume it's a case where\nyou have an index loaded and that index is related to the merge being\ndone, the number can be and often is much smaller than the number of\nentries in the index. And just to cover the extremes, in unusual\ncases the number might be much larger than the number of index entries\nif the merge base and side being merged in has far more paths).\n\nThis was the whole point of the strmap API[1] I recently added --\nprovide a hashmap specialized for the case where the key is a string.\nThat way I get fast lookup, and relatively fast resize as the hash\nonly contains pointers to the values, not a copy of the values.\n\nIs your concern that allocating many small structs is more expensive\nthan allocating a huge block of them? If so, yes that matters, but\nsee the mem_pool related patches of the strmap API[1].\n\n[1] https://lore.kernel.org/git/pull.835.v5.git.git.1604622298.gitgitgadget@gmail.com/\n\n> Perhaps 'struct conflict_info' shouldn't contain a 'struct merged_info'\n> and instead be just the \"extra\" data. Then we could have a contiguous\n> array of 'struct merged_info' values for most of the paths, but heap\n> pointers for 'struct conflict_info' as necessary.\n>\n> It's also true that I haven't fully formed a mental model for how these\n> are used in your algorithm, so I'll keep reading.\n\nI don't understand how contiguous arrays are practical or desirable\n(I'm close to saying they're not possible, but one could employ some\nextremes to get them, as mentioned above).\n\nI could possibly have two strmaps; one mapping paths to a merge_info,\nand another (with fewer entries) mapping paths to a conflict_info.\nSeems like a royal pain, and would make for some pretty ugly code (I\nhave other places that had to use two strmaps and I've hated it every\ntime -- but those were cases of strmaps that were used much, much less\nthan the \"paths\" one). Might also slightly hurt perf\n\n> > + path_info->merged.directory_name = current_dir_name;\n> > + path_info->merged.basename_offset = current_dir_name_len;\n> > + path_info->merged.clean = !!resolved;\n> > + if (resolved) {\n> > + path_info->merged.result.mode = merged_version->mode;\n> > + oidcpy(&path_info->merged.result.oid, &merged_version->oid);\n> > + path_info->merged.is_null = !!is_null;\n> > + } else {\n> > + int i;\n> > +\n> > + for (i = 0; i < 3; i++) {\n> > + path_info->pathnames[i] = fullpath;\n> > + path_info->stages[i].mode = names[i].mode;\n> > + oidcpy(&path_info->stages[i].oid, &names[i].oid);\n> > + }\n> > + path_info->filemask = filemask;\n> > + path_info->dirmask = dirmask;\n> > + path_info->df_conflict = !!df_conflict;\n> > + }\n> > + strmap_put(&opt->priv->paths, fullpath, path_info);\n> > + result->string = fullpath;\n> > + result->util = path_info;\n>\n> This is set in all cases, so should we use it everywhere? Naturally,\n> there might be a cost to the extra pointer indirection, so maybe we\n> create a 'struct conflict_info *util' to operate on during this\n> method, but set 'result->util = util' right after allocating so we\n> know how it should behave?\n\nresult->util is void*, so it's not just an extra pointer indirection,\nit's also the need to cast it to the appropriate type every time you\nwant to use it. It's easier to have that done via another copy of the\npointer with the correct type, which is the reason for path_info. So,\nessentially, I did use util everywhere, it's just that I spelled it as\n\"path_info\". If I had named \"path_info\" \"util\" as you suggest,\nwouldn't everyone be annoyed that I used a lame name that didn't name\nthe variable's purpose?\n\nPerhaps I should just add a comment saying that path_util is a typed\nalias/copy of result->util when I define it?\n\n> > @@ -91,10 +136,12 @@ static int collect_merge_info_callback(int n,\n> > */\n> > struct merge_options *opt = info->data;\n> > struct merge_options_internal *opti = opt->priv;\n> > - struct conflict_info *ci;\n> > + struct string_list_item pi; /* Path Info */\n> > + struct conflict_info *ci; /* pi.util when there's a conflict */\n\nPerhaps here I should mention that ci is just a typed copy of pi.util\n(since pi.util is a void*).\n\n> ...\n>\n> > + setup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n> > + names, NULL, 0, df_conflict, filemask, dirmask, 0);\n> > + ci = pi.util;\n>\n> Here is the use of 'pi' that I was talking about earlier.\n\n...although, to be fair, I don't actually have all that many uses of\nci (at least not anymore) in this function. So maybe typecasting\npi.util each of the three-or-so times it is used isn't so bad?\n"},{"id":"409661","messageId":"CABPp-BF=+-ieP8fCxWWuvhe2dTTMKVGnYSTJuV6H9Z+zYs0SNg@mail.gmail.com","threadId":"54556","inReplyTo":"75170ee7-525e-31fc-f6bd-6dfac12b00c8@gmail.com","subject":"Re: [PATCH v2 00/20] fundamentals of merge-ort implementation","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-11T18:35:21Z","receivedAt":"2020-11-11T18:35:39Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Nov 11, 2020 at 9:09 AM Derrick Stolee wrote:\n>\n> On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> > Elijah Newren (20):\n> > merge-ort: setup basic internal data structures\n> > merge-ort: add some high-level algorithm structure\n> > merge-ort: port merge_start() from merge-recursive\n> > merge-ort: use histogram diff\n> > merge-ort: add an err() function similar to one from merge-recursive\n> > merge-ort: implement a very basic collect_merge_info()\n> > merge-ort: avoid repeating fill_tree_descriptor() on the same tree\n> > merge-ort: compute a few more useful fields for collect_merge_info\n> > merge-ort: record stage and auxiliary info for every path\n> > merge-ort: avoid recursing into identical trees\n> > merge-ort: add a preliminary simple process_entries() implementation\n> > merge-ort: have process_entries operate in a defined order\n>\n> I got this far before my attention to detail really started slipping.\n>\n> > merge-ort: step 1 of tree writing -- record basenames, modes, and oids\n> > merge-ort: step 2 of tree writing -- function to create tree object\n> > merge-ort: step 3 of tree writing -- handling subdirectories as we go\n> > merge-ort: basic outline for merge_switch_to_result()\n> > merge-ort: add implementation of checkout()\n> > tree: enable cmp_cache_name_compare() to be used elsewhere\n> > merge-ort: add implementation of record_unmerged_index_entries()\n> > merge-ort: free data structures in merge_finalize()\n>\n> I'll try to take another pass on these commits tomorrow.\n>\n> For the series as a whole I'd love to see at least one test that\n> demonstrates that this code does something, if even only for a very\n> narrow case.\n>\n> There's a lot of code being moved here, and it would be nice to have\n> even a very simple test case that can check that we didn't leave any\n> important die(\"not implemented\") calls lying around or worse accessing\n> an uninitialized pointer or something.\n\nWe absolutely left several die(\"not implemented\") calls lying around.\nThe series was long enough at 20 patches; reviewers lose steam at 10\n(at least both you and Jonathan have), so maybe I should have left\neven more in there as an attempt to split up this series more.\n\nHowever, if you run the testsuite with GIT_TEST_MERGE_ALGORITHM=ort,\nthen this series drops the number of failures in the testsuite from\naround 2200, down to 1500. So, there's about 700 testcases for you.\n\nAlso, there were several preparatory series all designed for getting\nthe testsuite in order for this new merge algorithm. See the\nfollowing currently cooking topics:\n * en/merge-tests topic\n * en/dir-rename-tests\nand the following topics that were previously merged:\n * 36d225c7d4 (\"Merge branch 'en/merge-tests'\", 2020-08-19)\n * cf372dc815 (\"Merge branch 'en/test-cleanup'\", 2020-03-09)\n * ac193e0e0a (\"Merge branch 'en/merge-path-collision'\", 2019-01-04)\n * c99033060f (\"Merge branch\n'en/t7405-recursive-submodule-conflicts'\", 2018-08-02)\n * e6da45c7cd (\"Merge branch 'en/t6036-merge-recursive-tests'\", 2018-08-02)\n * 84e74c6403 (\"Merge branch\n'en/t6042-insane-merge-rename-testcases'\", 2018-08-02)\n * bba1a5559c (\"Merge branch 'en/t6036-recursive-corner-cases'\", 2018-08-02)\n * 93b74a7cfa (\"Merge branch 'en/merge-recursive-tests'\", 2018-06-25)\nand maybe others I missed.\n"},{"id":"409664","messageId":"CABPp-BEEoqOer11BYrqSVE9E4HcT5MNWcRm7ZHBQ7MVZRUDVXw@mail.gmail.com","threadId":"54556","inReplyTo":"4432ae6c-90ae-90d9-218f-15856395efac@gmail.com","subject":"Re: [PATCH v2 12/20] merge-ort: have process_entries operate in a defined order","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-11T18:58:06Z","receivedAt":"2020-11-11T18:58:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Nov 11, 2020 at 8:09 AM Derrick Stolee wrote:\n>\n> On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> > We want to handle paths below a directory before needing to handle the\n> > directory itself. Also, we want to handle the directory immediately\n> > after the paths below it, so we can't use simple lexicographic ordering\n> > from strcmp (which would insert foo.txt between foo and foo/file.c).\n> > Copy string_list_df_name_compare() from merge-recursive.c, and set up a\n> > string list of paths sorted by that function so that we can iterate in\n> > the desired order.\n>\n> This is at least the second time we've copied something from\n> merge-recursive.c. Should we be starting a merge-utils.[c|h] to group\n> these together under a common implementation?\n\nI'm actually going to replace the function later for performance\nreasons, but trying to make the series as simple as possible prompted\nme to \"just copy something for a starting point\".\n\nThere will be more functions that I copy, yes, but since I sometimes\nalso tweak and since the goal is to delete merge-recursive.[ch], I\ndidn't really want to set up an infrastructure to share stuff.\n\n> > + /* Put every entry from paths into plist, then sort */\n> > strmap_for_each_entry(&opt->priv->paths, &iter, e) {\n> > + string_list_append(&plist, e->key)->util = e->value;\n> > + }\n>\n> nit: are braces required here?\n\nIt might not be with the current macro definition of\nstrmap_for_each_entry(), but I think at one point it was (the macro\nhas undergone some changes over time). Given the difficulty of\ndigging through the layers of macros (and the possible risk of it\nchanging in the future with hashmap or strmap changes), I wonder if\nit's simpler for readers to just keep them?\n"},{"id":"409694","messageId":"20201111195111.3115407-1-jonathantanmy@google.com","threadId":"54556","inReplyTo":"20201102204344.342633-12-newren@gmail.com","subject":"Re: [PATCH v2 11/20] merge-ort: add a preliminary simple process_entries() implementation","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-11T19:51:11Z","receivedAt":"2020-11-11T19:51:18Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Okay...let me review patches 11-15. (Patches 16-20 deal with checkout\nand might be better reviewed by someone who is already familiar with how\nthe existing merge performs checkout. If no one reviews it, I might come\nback to it if I have time.)\n\n> +/* Per entry merge function */\n> +static void process_entry(struct merge_options *opt,\n> +\t\t\t const char *path,\n> +\t\t\t struct conflict_info *ci)\n> +{\n> +\tassert(!ci->merged.clean);\n> +\tassert(ci->filemask >= 0 && ci->filemask <= 7);\n\nI see below that this function doesn't handle ci->match_mask == 7 (and\nit doesn't need to because, I believe, there is a function in one of the\nearlier patches that optimizes the case wherein all 3 match with each\nother). Maybe add an assert here for that too.\n\n> +\n> +\tif (ci->filemask == 0) {\n> +\t\t/*\n> +\t\t * This is a placeholder for directories that were recursed\n> +\t\t * into; nothing to do in this case.\n> +\t\t */\n> +\t\treturn;\n> +\t}\n> +\n> +\tif (ci->df_conflict) {\n> +\t\tdie(\"Not yet implemented.\");\n> +\t}\n> +\n> +\t/*\n> +\t * NOTE: Below there is a long switch-like if-elseif-elseif... block\n> +\t * which the code goes through even for the df_conflict cases\n> +\t * above. Well, it will once we don't die-not-implemented above.\n> +\t */\n> +\tif (ci->match_mask) {\n> +\t\tci->merged.clean = 1;\n\nOK, looks straightforward so far. It's a clean merge if 2 match. (As I\nsaid earlier, at this point in the code, it is not possible for 3 to\nmatch.)\n\n> +\t\tif (ci->match_mask == 6) {\n> +\t\t\t/* stages[1] == stages[2] */\n> +\t\t\tci->merged.result.mode = ci->stages[1].mode;\n> +\t\t\toidcpy(&ci->merged.result.oid, &ci->stages[1].oid);\n\nIf OURS and THEIRS match, use one of them arbitrarily (because they are\nthe same anyway). OK.\n\n> +\t\t} else {\n> +\t\t\t/* determine the mask of the side that didn't match */\n> +\t\t\tunsigned int othermask = 7 & ~ci->match_mask;\n> +\t\t\tint side = (othermask == 4) ? 2 : 1;\n\nBASE matches with either OURS or THEIRS, so use the side that doesn't\nmatch. OK.\n\n> +\n> +\t\t\tci->merged.is_null = (ci->filemask == ci->match_mask);\n\nThis works (if the non-matching bit in filemask is set, the file exists;\nthe comparison will be false and therefore is_null is false - and\ncorrectly false because the file exists), but seems unnecessarily\nclever. Couldn't you just check nullness of the OID (or through the\nmode, like the line below it) and set it here?\n\nAdmittedly, the way you wrote it also verifies that filemask is what we\nexpect. I don't think it is important to verify it, but if you think it\nis important, I think it is this verification that should go in the\nassert statement.\n\n> +\t\t\tci->merged.result.mode = ci->stages[side].mode;\n> +\t\t\toidcpy(&ci->merged.result.oid, &ci->stages[side].oid);\n> +\n> +\t\t\tassert(othermask == 2 || othermask == 4);\n> +\t\t\tassert(ci->merged.is_null == !ci->merged.result.mode);\n> +\t\t}\n> +\t} else if (ci->filemask >= 6 &&\n> +\t\t (S_IFMT & ci->stages[1].mode) !=\n> +\t\t (S_IFMT & ci->stages[2].mode)) {\n> +\t\t/*\n> +\t\t * Two different items from (file/submodule/symlink)\n> +\t\t */\n> +\t\tdie(\"Not yet implemented.\");\n\nThere are no matches, and OURS and THEIRS have different types. OK.\n\n> +\t} else if (ci->filemask >= 6) {\n> +\t\t/*\n> +\t\t * TODO: Needs a two-way or three-way content merge, but we're\n> +\t\t * just being lazy and copying the version from HEAD and\n> +\t\t * leaving it as conflicted.\n> +\t\t */\n> +\t\tci->merged.clean = 0;\n> +\t\tci->merged.result.mode = ci->stages[1].mode;\n> +\t\toidcpy(&ci->merged.result.oid, &ci->stages[1].oid);\n\nOK.\n\n> +\t} else if (ci->filemask == 3 || ci->filemask == 5) {\n> +\t\t/* Modify/delete */\n> +\t\tdie(\"Not yet implemented.\");\n> +\t} else if (ci->filemask == 2 || ci->filemask == 4) {\n> +\t\t/* Added on one side */\n> +\t\tint side = (ci->filemask == 4) ? 2 : 1;\n> +\t\tci->merged.result.mode = ci->stages[side].mode;\n> +\t\toidcpy(&ci->merged.result.oid, &ci->stages[side].oid);\n> +\t\tci->merged.clean = !ci->df_conflict && !ci->path_conflict;\n> +\t} else if (ci->filemask == 1) {\n> +\t\t/* Deleted on both sides */\n> +\t\tci->merged.is_null = 1;\n> +\t\tci->merged.result.mode = 0;\n> +\t\toidcpy(&ci->merged.result.oid, &null_oid);\n> +\t\tci->merged.clean = !ci->path_conflict;\n> +\t}\n\nThe rest is OK.\n\n> +\n> +\t/*\n> +\t * If still unmerged, record it separately. This allows us to later\n> +\t * iterate over just unmerged entries when updating the index instead\n> +\t * of iterating over all entries.\n> +\t */\n> +\tif (!ci->merged.clean)\n> +\t\tstrmap_put(&opt->priv->unmerged, path, ci);\n> +}\n> +\n> static void process_entries(struct merge_options *opt,\n> \t\t\t struct object_id *result_oid)\n> {\n> -\tdie(\"Not yet implemented.\");\n> +\tstruct hashmap_iter iter;\n> +\tstruct strmap_entry *e;\n> +\n> +\tif (strmap_empty(&opt->priv->paths)) {\n> +\t\toidcpy(result_oid, opt->repo->hash_algo->empty_tree);\n> +\t\treturn;\n> +\t}\n> +\n> +\tstrmap_for_each_entry(&opt->priv->paths, &iter, e) {\n> +\t\t/*\n> +\t\t * WARNING: If ci->merged.clean is true, then ci does not\n> +\t\t * actually point to a conflict_info but a struct merge_info.\n> +\t\t */\n> +\t\tstruct conflict_info *ci = e->value;\n> +\n> +\t\tif (!ci->merged.clean)\n> +\t\t\tprocess_entry(opt, e->key, e->value);\n> +\t}\n> +\n> +\tdie(\"Tree creation not yet implemented\");\n\nThe rest looks straightforward.\n"},{"id":"409695","messageId":"20201111200157.3118955-1-jonathantanmy@google.com","threadId":"54556","inReplyTo":"20201102204344.342633-14-newren@gmail.com","subject":"Re: [PATCH v2 13/20] merge-ort: step 1 of tree writing -- record basenames, modes, and oids","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-11T20:01:57Z","receivedAt":"2020-11-11T20:02:03Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> +struct directory_versions {\n> +\tstruct string_list versions;\n\nMaybe comment that this is an unordered list of basenames to merged.result is>.\n\n> @@ -442,6 +464,7 @@ static void process_entries(struct merge_options *opt,\n> \tstruct strmap_entry *e;\n> \tstruct string_list plist = STRING_LIST_INIT_NODUP;\n> \tstruct string_list_item *entry;\n> +\tstruct directory_versions dir_metadata;\n> \n> \tif (strmap_empty(&opt->priv->paths)) {\n> \t\toidcpy(result_oid, opt->repo->hash_algo->empty_tree);\n> @@ -458,6 +481,9 @@ static void process_entries(struct merge_options *opt,\n> \tplist.cmp = string_list_df_name_compare;\n> \tstring_list_sort(&plist);\n> \n> +\t/* other setup */\n> +\tstring_list_init(&dir_metadata.versions, 0);\n> +\n\nMight be clearer to just initialize dir_metadata as {\nSTRING_LIST_INIT_NODUP }.\n\nThe rest makes sense.\n"},{"id":"409713","messageId":"CABPp-BGr4kpjyZc3Rbh+8StPbWWqo36Kg0P=dgqctwX0SCGPxw@mail.gmail.com","threadId":"54556","inReplyTo":"20201111200157.3118955-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 13/20] merge-ort: step 1 of tree writing -- record basenames, modes, and oids","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-11T20:24:12Z","receivedAt":"2020-11-11T20:24:26Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Nov 11, 2020 at 12:01 PM Jonathan Tan wrote:\n>\n> > +struct directory_versions {\n> > + struct string_list versions;\n>\n> Maybe comment that this is an unordered list of basenames to the type of ci->merged.result is>.\n\nThere actually is an order, and it's important. It's reverse\nlexicographic order of full pathnames (the ordering comes from the\nfact that process_entries() iterates paths in that order). The\nreasons for that ordering are (1) all the basenames within a directory\nare adjacent so that I can write out a tree for a directory as soon as\nit is done, and (2) paths within a directory are listed before the\ndirectory itself so that I get the necessary info for subtrees before\ntrying to write out their parent trees.\n\nIt's not until later patches that I take advantage of this ordering\n(and when I do I have a very long commit message to describe it all),\nbut I can add a comment that this is a list of basenames to\nmerge_info.\n\n>\n> > @@ -442,6 +464,7 @@ static void process_entries(struct merge_options *opt,\n> > struct strmap_entry *e;\n> > struct string_list plist = STRING_LIST_INIT_NODUP;\n> > struct string_list_item *entry;\n> > + struct directory_versions dir_metadata;\n> >\n> > if (strmap_empty(&opt->priv->paths)) {\n> > oidcpy(result_oid, opt->repo->hash_algo->empty_tree);\n> > @@ -458,6 +481,9 @@ static void process_entries(struct merge_options *opt,\n> > plist.cmp = string_list_df_name_compare;\n> > string_list_sort(&plist);\n> >\n> > + /* other setup */\n> > + string_list_init(&dir_metadata.versions, 0);\n> > +\n>\n> Might be clearer to just initialize dir_metadata as {\n> STRING_LIST_INIT_NODUP }.\n\nIt'll eventually grow to { STRING_LIST_INIT_NODUP,\nSTRING_LIST_INIT_NODUP, NULL, 0 }, which is a tad long, but if the\ninitializer is clearer I'm happy to switch over to it.\n\n> The rest makes sense.\n"},{"id":"409718","messageId":"20201111204707.3126694-1-jonathantanmy@google.com","threadId":"54556","inReplyTo":"20201102204344.342633-15-newren@gmail.com","subject":"Re: [PATCH v2 14/20] merge-ort: step 2 of tree writing -- function to create tree object","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-11T20:47:07Z","receivedAt":"2020-11-11T20:47:13Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> +static void write_tree(struct object_id *result_oid,\n> +\t\t struct string_list *versions,\n> +\t\t unsigned int offset)\n> +{\n> +\tsize_t maxlen = 0;\n> +\tunsigned int nr = versions->nr - offset;\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\tstruct string_list relevant_entries = STRING_LIST_INIT_NODUP;\n> +\tint i;\n> +\n> +\t/*\n> +\t * We want to sort the last (versions->nr-offset) entries in versions.\n> +\t * Do so by abusing the string_list API a bit: make another string_list\n> +\t * that contains just those entries and then sort them.\n> +\t *\n> +\t * We won't use relevant_entries again and will let it just pop off the\n> +\t * stack, so there won't be allocation worries or anything.\n> +\t */\n> +\trelevant_entries.items = versions->items + offset;\n> +\trelevant_entries.nr = versions->nr - offset;\n> +\tstring_list_sort(&relevant_entries);\n> +\n> +\t/* Pre-allocate some space in buf */\n> +\tfor (i = 0; i < nr; i++) {\n> +\t\tmaxlen += strlen(versions->items[offset+i].string) + 34;\n\nProbably should include the_hash_algo->rawsz instead of hardcoding 34.\n\nThe rest looks straightforward.\n"},{"id":"409719","messageId":"e3fcaead-8bee-bcda-aee0-459ad9774f9e@gmail.com","threadId":"54556","inReplyTo":"CABPp-BF=+-ieP8fCxWWuvhe2dTTMKVGnYSTJuV6H9Z+zYs0SNg@mail.gmail.com","subject":"Re: [PATCH v2 00/20] fundamentals of merge-ort implementation","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-11T20:48:01Z","receivedAt":"2020-11-11T20:48:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/11/2020 1:35 PM, Elijah Newren wrote:\n> On Wed, Nov 11, 2020 at 9:09 AM Derrick Stolee wrote:\n>> For the series as a whole I'd love to see at least one test that\n>> demonstrates that this code does something, if even only for a very\n>> narrow case.\n>>\n>> There's a lot of code being moved here, and it would be nice to have\n>> even a very simple test case that can check that we didn't leave any\n>> important die(\"not implemented\") calls lying around or worse accessing\n>> an uninitialized pointer or something.\n> \n> We absolutely left several die(\"not implemented\") calls lying around.\n> The series was long enough at 20 patches; reviewers lose steam at 10\n> (at least both you and Jonathan have), so maybe I should have left\n> even more in there as an attempt to split up this series more.\n> \n> However, if you run the testsuite with GIT_TEST_MERGE_ALGORITHM=ort,\n> then this series drops the number of failures in the testsuite from\n> around 2200, down to 1500. So, there's about 700 testcases for you.\n\nSorry that I'm jumping in to the series-of-series in the middle, so\nI am unfamiliar with the previous progress and testing strategy. This\n\"number of test failures\" metric is sufficient to demonstrate the\nprogress provided in this series. Perhaps it was even in your v1 cover\nletter.\n\nThanks,\n-Stolee\n\n"},{"id":"409723","messageId":"CABPp-BE=XzueTJ18dZXXXGEkJ0b6R-TjrgK5SAjDo2oPzTLETA@mail.gmail.com","threadId":"54556","inReplyTo":"e3fcaead-8bee-bcda-aee0-459ad9774f9e@gmail.com","subject":"Re: [PATCH v2 00/20] fundamentals of merge-ort implementation","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-11T21:18:34Z","receivedAt":"2020-11-11T21:18:48Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Derrick,\n\nOn Wed, Nov 11, 2020 at 12:48 PM Derrick Stolee wrote:\n>\n> On 11/11/2020 1:35 PM, Elijah Newren wrote:\n> > On Wed, Nov 11, 2020 at 9:09 AM Derrick Stolee wrote:\n> >> For the series as a whole I'd love to see at least one test that\n> >> demonstrates that this code does something, if even only for a very\n> >> narrow case.\n> >>\n> >> There's a lot of code being moved here, and it would be nice to have\n> >> even a very simple test case that can check that we didn't leave any\n> >> important die(\"not implemented\") calls lying around or worse accessing\n> >> an uninitialized pointer or something.\n> >\n> > We absolutely left several die(\"not implemented\") calls lying around.\n> > The series was long enough at 20 patches; reviewers lose steam at 10\n> > (at least both you and Jonathan have), so maybe I should have left\n> > even more in there as an attempt to split up this series more.\n> >\n> > However, if you run the testsuite with GIT_TEST_MERGE_ALGORITHM=ort,\n> > then this series drops the number of failures in the testsuite from\n> > around 2200, down to 1500. So, there's about 700 testcases for you.\n>\n> Sorry that I'm jumping in to the series-of-series in the middle, so\n> I am unfamiliar with the previous progress and testing strategy. This\n\nNot a problem at all. Thanks much for jumping in and taking a look!\nYou always provide some good feedback and suggestions.\n\n(Besides, those testcase changes have been spread over two and a half\nyears...hard to stay on top of all of them.)\n\n> \"number of test failures\" metric is sufficient to demonstrate the\n> progress provided in this series. Perhaps it was even in your v1 cover\n> letter.\n\nUm, oops; it's not. I did mention there were still some \"not\nimplemented\" messages left, but didn't mention the testcase counts.\nBut even that mention is apparently in the v1 cover letter rather than\nv2, and v2 wasn't sent in-reply-to v1, so it's harder to catch that.\nSorry about that; I'll include the testcase counts in the v3 cover\nletter.\n"},{"id":"409724","messageId":"CABPp-BF+wX4nRnVNJLFKqKhjop1nL948SyhYRs2Tx3rErDa8sw@mail.gmail.com","threadId":"54556","inReplyTo":"20201111204707.3126694-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 14/20] merge-ort: step 2 of tree writing -- function to create tree object","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-11T21:21:54Z","receivedAt":"2020-11-11T21:22:08Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Nov 11, 2020 at 12:47 PM Jonathan Tan wrote:\n>\n> > +static void write_tree(struct object_id *result_oid,\n> > + struct string_list *versions,\n> > + unsigned int offset)\n> > +{\n> > + size_t maxlen = 0;\n> > + unsigned int nr = versions->nr - offset;\n> > + struct strbuf buf = STRBUF_INIT;\n> > + struct string_list relevant_entries = STRING_LIST_INIT_NODUP;\n> > + int i;\n> > +\n> > + /*\n> > + * We want to sort the last (versions->nr-offset) entries in versions.\n> > + * Do so by abusing the string_list API a bit: make another string_list\n> > + * that contains just those entries and then sort them.\n> > + *\n> > + * We won't use relevant_entries again and will let it just pop off the\n> > + * stack, so there won't be allocation worries or anything.\n> > + */\n> > + relevant_entries.items = versions->items + offset;\n> > + relevant_entries.nr = versions->nr - offset;\n> > + string_list_sort(&relevant_entries);\n> > +\n> > + /* Pre-allocate some space in buf */\n> > + for (i = 0; i < nr; i++) {\n> > + maxlen += strlen(versions->items[offset+i].string) + 34;\n>\n> Probably should include the_hash_algo->rawsz instead of hardcoding 34.\n\nAh, indeed. And I should submit a patch for fast-import.c to update\nit to not hardcode 34 either (though I'll submit the fast-import\nchange separate from this series).\n"},{"id":"409727","messageId":"CABPp-BHCqD8yS_4t7ztL2+FxjPCZ9Meq97VfRhnDRwCjLpdSzg@mail.gmail.com","threadId":"54556","inReplyTo":"CABPp-BFRPuxLYz_n6jbr=j7Gu1GhsV95nKPE1=HxUcsvimrz0g@mail.gmail.com","subject":"Re: [PATCH v2 09/20] merge-ort: record stage and auxiliary info for every path","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-11T22:06:13Z","receivedAt":"2020-11-11T22:06:29Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Nov 11, 2020 at 10:16 AM Elijah Newren wrote:\n>\n> On Wed, Nov 11, 2020 at 7:26 AM Derrick Stolee wrote:\n> >\n> > On 11/2/2020 3:43 PM, Elijah Newren wrote:\n> > > +static void setup_path_info(struct merge_options *opt,\n> > > + struct string_list_item *result,\n> > > + const char *current_dir_name,\n> > > + int current_dir_name_len,\n> > > + char *fullpath, /* we'll take over ownership */\n> > > + struct name_entry *names,\n> > > + struct name_entry *merged_version,\n> > > + unsigned is_null, /* boolean */\n> > > + unsigned df_conflict, /* boolean */\n> > > + unsigned filemask,\n> > > + unsigned dirmask,\n> > > + int resolved /* boolean */)\n> > > +{\n> > > + struct conflict_info *path_info;\n> >\n> > In addition to my concerns below about 'conflict_info' versus\n> > 'merged_info', I was doubly confused that 'result' in the parameter\n> > list is given a variable named 'pi' for \"path info\" and result->util\n> > eventually is equal to this path_info. What if we renamed 'result'\n> > to 'pi' for \"path info\" here, then operated on 'pi->util' in this\n> > method?\n>\n> result->util (or pi->util if you rename) is void *, making it hard to\n> operate on; you'd have to typecast at every usage. Since it is used a\n> *lot*, it makes sense to have a typed pointer, and then just set\n> result->util to a copy of that value at the end. That is what\n> path_info is for.\n>\n> >\n> > > + path_info = xcalloc(1, resolved ? sizeof(struct merged_info) :\n> > > + sizeof(struct conflict_info));\n> >\n> > Hm. I'm happy to have a `struct merged_info *` pointing to a\n> > `struct conflict_info`, but the opposite seems very dangerous.\n>\n> Yeah, this is perhaps the scariest bit, and if it were a side data\n> structure rather than the fundamental main one that was central to the\n> algorithm, then safety would trump performance concerns. But since it\n> is the main data structure and likely the biggest (once you count the\n> various copies for each relevant path), then it might be worth the\n> extra care needed to shave off the extra memory. Maybe we can still\n> tweak things to get some safety back without killing performance so\n> let me consider each of your suggestions/questions.\n>\n> If I define it as a merged_info*, the compiler will only let me modify\n> fields within the merged_info portion of the struct. Should I\n> typecast every line that touches the bits in the resolved==0 path\n> where I need to set fields within the conflict_info portion?\n> Alternatively, would a code flow like the following make you happier?\n>\n> struct conflict_info *ci = NULL;\n> struct merge_info *mi = xcalloc(...);\n> result->util = mi;\n> /* Operate on mi */\n> ...\n> if (resolved)\n> return;\n> ci = mi;\n> /* Operate on ci */\n> ...\n>\n> In either case, the returned item has potentially different sizes, so\n> the caller will still have to take care so I'm not sure how much extra\n> this structure within setup_path_info() buys us.\n>\n> > Perhaps we should always use sizeof(struct conflict_info)?\n>\n> We could do that; it'd certainly waste memory as I expect many entries\n> to be unmodified (on one or both sides of history). But I'd way\n> rather go this route than splitting or re-arranging this data\n> structure.\n>\n> > We can use path_info->merged.clean to detect whether the rest of\n> > the data is worth looking at. (Or, in your case, whether or not\n> > it is allocated.)\n>\n> ci->merged.clean is used to determine whether to look at the rest of\n> the data, yes -- and that's an enforced assumption throughout the code\n> (as alluded to by the comment in the merge_options_internal data\n> structure that \"paths\" maps pathanemes to merge_info and conflict_info\n> types). However, that is not quite the same as using the clean bit to\n> determine if more data is allocated; something can be allocated as a\n> conflict_info rather than a merged_info due to both sides making\n> modifying the same path, but then a threeway content merge comes back\n> clean and ci->merged.clean is updated from 0 to 1. The extra data\n> remains allocated, but nothing in the algorithm ever needs to use\n> anything outside the merged bits for that path again. (Actually, let\n> me state that more forcefully: nothing is *allowed* to look outside\n> the merged bits for that path once the clean bit is updated to 1).\n>\n> > I imagine that in a large repo we will need many of these structs,\n> > but very few of them will actually need to be conflicts, so using\n> > 'struct conflict_info' always will lead to memory bloat. But in\n> > that case, would we not be better off with an array instead of a\n> > scattering of data across the heap?\n>\n> Not sure what you're trying to solve here. Putting them in an array\n> would mean copying every single one of them every time the array is\n> resized. It would also make insertion or deletion very expensive.\n> And it'd prevent O(1) lookup. It'd be a horrible data structure all\n> around. Maybe you're assuming you know exactly how many entries you\n> need and what they are before the merge algorithm starts? I don't.\n> In fact, I can't even give a good magnitude approximation of how many\n> it'll be before a merge starts. (Even if you assume it's a case where\n> you have an index loaded and that index is related to the merge being\n> done, the number can be and often is much smaller than the number of\n> entries in the index. And just to cover the extremes, in unusual\n> cases the number might be much larger than the number of index entries\n> if the merge base and side being merged in has far more paths).\n>\n> This was the whole point of the strmap API[1] I recently added --\n> provide a hashmap specialized for the case where the key is a string.\n> That way I get fast lookup, and relatively fast resize as the hash\n> only contains pointers to the values, not a copy of the values.\n>\n> Is your concern that allocating many small structs is more expensive\n> than allocating a huge block of them? If so, yes that matters, but\n> see the mem_pool related patches of the strmap API[1].\n>\n> [1] https://lore.kernel.org/git/pull.835.v5.git.git.1604622298.gitgitgadget@gmail.com/\n\n\nI just re-read what I wrote, here and below...and I need to apologize.\nI tend to write, edit, revise, and repeat while composing emails and\nthe end result of my emails doesn't tend to reflect the path to get\nthere; I looped through that cycle more times than most on this email.\nBut, even worse, I added in a sentence or two that just shouldn't be\nincluded regardless. I think in particular this one sounds extremely\naggressive and dismissive which was not at all my intent.\n\nI find your reviews to be very helpful, and I don't want to discourage\nthem. Hopefully my comments didn't come across anywhere near as\nstrongly as they did to me on a second reading, but if they did, I'm\nsorry.\n\n> > Perhaps 'struct conflict_info' shouldn't contain a 'struct merged_info'\n> > and instead be just the \"extra\" data. Then we could have a contiguous\n> > array of 'struct merged_info' values for most of the paths, but heap\n> > pointers for 'struct conflict_info' as necessary.\n> >\n> > It's also true that I haven't fully formed a mental model for how these\n> > are used in your algorithm, so I'll keep reading.\n>\n> I don't understand how contiguous arrays are practical or desirable\n> (I'm close to saying they're not possible, but one could employ some\n> extremes to get them, as mentioned above).\n>\n> I could possibly have two strmaps; one mapping paths to a merge_info,\n> and another (with fewer entries) mapping paths to a conflict_info.\n> Seems like a royal pain, and would make for some pretty ugly code (I\n> have other places that had to use two strmaps and I've hated it every\n> time -- but those were cases of strmaps that were used much, much less\n> than the \"paths\" one). Might also slightly hurt perf\n>\n> > > + path_info->merged.directory_name = current_dir_name;\n> > > + path_info->merged.basename_offset = current_dir_name_len;\n> > > + path_info->merged.clean = !!resolved;\n> > > + if (resolved) {\n> > > + path_info->merged.result.mode = merged_version->mode;\n> > > + oidcpy(&path_info->merged.result.oid, &merged_version->oid);\n> > > + path_info->merged.is_null = !!is_null;\n> > > + } else {\n> > > + int i;\n> > > +\n> > > + for (i = 0; i < 3; i++) {\n> > > + path_info->pathnames[i] = fullpath;\n> > > + path_info->stages[i].mode = names[i].mode;\n> > > + oidcpy(&path_info->stages[i].oid, &names[i].oid);\n> > > + }\n> > > + path_info->filemask = filemask;\n> > > + path_info->dirmask = dirmask;\n> > > + path_info->df_conflict = !!df_conflict;\n> > > + }\n> > > + strmap_put(&opt->priv->paths, fullpath, path_info);\n> > > + result->string = fullpath;\n> > > + result->util = path_info;\n> >\n> > This is set in all cases, so should we use it everywhere? Naturally,\n> > there might be a cost to the extra pointer indirection, so maybe we\n> > create a 'struct conflict_info *util' to operate on during this\n> > method, but set 'result->util = util' right after allocating so we\n> > know how it should behave?\n>\n> result->util is void*, so it's not just an extra pointer indirection,\n> it's also the need to cast it to the appropriate type every time you\n> want to use it. It's easier to have that done via another copy of the\n> pointer with the correct type, which is the reason for path_info. So,\n> essentially, I did use util everywhere, it's just that I spelled it as\n> \"path_info\". If I had named \"path_info\" \"util\" as you suggest,\n> wouldn't everyone be annoyed that I used a lame name that didn't name\n> the variable's purpose?\n>\n> Perhaps I should just add a comment saying that path_util is a typed\n> alias/copy of result->util when I define it?\n>\n> > > @@ -91,10 +136,12 @@ static int collect_merge_info_callback(int n,\n> > > */\n> > > struct merge_options *opt = info->data;\n> > > struct merge_options_internal *opti = opt->priv;\n> > > - struct conflict_info *ci;\n> > > + struct string_list_item pi; /* Path Info */\n> > > + struct conflict_info *ci; /* pi.util when there's a conflict */\n>\n> Perhaps here I should mention that ci is just a typed copy of pi.util\n> (since pi.util is a void*).\n>\n> > ...\n> >\n> > > + setup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n> > > + names, NULL, 0, df_conflict, filemask, dirmask, 0);\n> > > + ci = pi.util;\n> >\n> > Here is the use of 'pi' that I was talking about earlier.\n>\n> ...although, to be fair, I don't actually have all that many uses of\n> ci (at least not anymore) in this function. So maybe typecasting\n> pi.util each of the three-or-so times it is used isn't so bad?\n"},{"id":"409747","messageId":"CABPp-BFgQX6Ash03x7z+RfE3ytbw3x0DzDSBrGddgMr_soODoA@mail.gmail.com","threadId":"54556","inReplyTo":"20201111195111.3115407-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 11/20] merge-ort: add a preliminary simple process_entries() implementation","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-12T01:48:17Z","receivedAt":"2020-11-12T05:37:23Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Nov 11, 2020 at 11:51 AM Jonathan Tan wrote:\n>\n> Okay...let me review patches 11-15. (Patches 16-20 deal with checkout\n> and might be better reviewed by someone who is already familiar with how\n> the existing merge performs checkout. If no one reviews it, I might come\n> back to it if I have time.)\n\nThanks for the reviews! I was hoping to see some comments on patch\n15, as it's possibly the gnarliest. It's a relatively straightforward\nalgorithm, just lots of bookkeeping.\n\nAnd I think you took on the harder part of the remaining reviews. :-)\n The checkout stuff is much easier, IMO -- and knowledge of how the\nexisting merge performs checkout wouldn't help at all with reviewing\nthat; it's just too different.\n\nIf you do find the time to look at the last five patches, or parts of\nthem here's some tips on the reviewing:\n * Patches 16, 18, and 20 are very straightforward; patches 17 and 19\nare the ones that would benefit more from review.\n * Patch 17 is basically the twoway_merge subset of\nmerge_working_tree() from builtin/checkout.c. Find that bit of code\nand it's a direct comparison.\n * Patch 19 amounts to \"how do I remove stage 0 entries in the index\nand replace them with 1-3 higher order stages?\".\n\n> > +/* Per entry merge function */\n> > +static void process_entry(struct merge_options *opt,\n> > + const char *path,\n> > + struct conflict_info *ci)\n> > +{\n> > + assert(!ci->merged.clean);\n> > + assert(ci->filemask >= 0 && ci->filemask <= 7);\n>\n> I see below that this function doesn't handle ci->match_mask == 7 (and\n> it doesn't need to because, I believe, there is a function in one of the\n> earlier patches that optimizes the case wherein all 3 match with each\n> other). Maybe add an assert here for that too.\n>\n> > +\n> > + if (ci->filemask == 0) {\n> > + /*\n> > + * This is a placeholder for directories that were recursed\n> > + * into; nothing to do in this case.\n> > + */\n> > + return;\n> > + }\n> > +\n> > + if (ci->df_conflict) {\n> > + die(\"Not yet implemented.\");\n> > + }\n> > +\n> > + /*\n> > + * NOTE: Below there is a long switch-like if-elseif-elseif... block\n> > + * which the code goes through even for the df_conflict cases\n> > + * above. Well, it will once we don't die-not-implemented above.\n> > + */\n> > + if (ci->match_mask) {\n> > + ci->merged.clean = 1;\n>\n> OK, looks straightforward so far. It's a clean merge if 2 match. (As I\n> said earlier, at this point in the code, it is not possible for 3 to\n> match.)\n>\n> > + if (ci->match_mask == 6) {\n> > + /* stages[1] == stages[2] */\n> > + ci->merged.result.mode = ci->stages[1].mode;\n> > + oidcpy(&ci->merged.result.oid, &ci->stages[1].oid);\n>\n> If OURS and THEIRS match, use one of them arbitrarily (because they are\n> the same anyway). OK.\n>\n> > + } else {\n> > + /* determine the mask of the side that didn't match */\n> > + unsigned int othermask = 7 & ~ci->match_mask;\n> > + int side = (othermask == 4) ? 2 : 1;\n>\n> BASE matches with either OURS or THEIRS, so use the side that doesn't\n> match. OK.\n>\n> > +\n> > + ci->merged.is_null = (ci->filemask == ci->match_mask);\n>\n> This works (if the non-matching bit in filemask is set, the file exists;\n> the comparison will be false and therefore is_null is false - and\n> correctly false because the file exists), but seems unnecessarily\n> clever. Couldn't you just check nullness of the OID (or through the\n> mode, like the line below it) and set it here?\n>\n> Admittedly, the way you wrote it also verifies that filemask is what we\n> expect. I don't think it is important to verify it, but if you think it\n> is important, I think it is this verification that should go in the\n> assert statement.\n\nThese points and the others earlier in this file and other points I\ndidn't comment on are all good points; thanks for all the suggestions.\n\n> > + ci->merged.result.mode = ci->stages[side].mode;\n> > + oidcpy(&ci->merged.result.oid, &ci->stages[side].oid);\n> > +\n> > + assert(othermask == 2 || othermask == 4);\n> > + assert(ci->merged.is_null == !ci->merged.result.mode);\n> > + }\n> > + } else if (ci->filemask >= 6 &&\n> > + (S_IFMT & ci->stages[1].mode) !=\n> > + (S_IFMT & ci->stages[2].mode)) {\n> > + /*\n> > + * Two different items from (file/submodule/symlink)\n> > + */\n> > + die(\"Not yet implemented.\");\n>\n> There are no matches, and OURS and THEIRS have different types. OK.\n>\n> > + } else if (ci->filemask >= 6) {\n> > + /*\n> > + * TODO: Needs a two-way or three-way content merge, but we're\n> > + * just being lazy and copying the version from HEAD and\n> > + * leaving it as conflicted.\n> > + */\n> > + ci->merged.clean = 0;\n> > + ci->merged.result.mode = ci->stages[1].mode;\n> > + oidcpy(&ci->merged.result.oid, &ci->stages[1].oid);\n>\n> OK.\n>\n> > + } else if (ci->filemask == 3 || ci->filemask == 5) {\n> > + /* Modify/delete */\n> > + die(\"Not yet implemented.\");\n> > + } else if (ci->filemask == 2 || ci->filemask == 4) {\n> > + /* Added on one side */\n> > + int side = (ci->filemask == 4) ? 2 : 1;\n> > + ci->merged.result.mode = ci->stages[side].mode;\n> > + oidcpy(&ci->merged.result.oid, &ci->stages[side].oid);\n> > + ci->merged.clean = !ci->df_conflict && !ci->path_conflict;\n> > + } else if (ci->filemask == 1) {\n> > + /* Deleted on both sides */\n> > + ci->merged.is_null = 1;\n> > + ci->merged.result.mode = 0;\n> > + oidcpy(&ci->merged.result.oid, &null_oid);\n> > + ci->merged.clean = !ci->path_conflict;\n> > + }\n>\n> The rest is OK.\n>\n> > +\n> > + /*\n> > + * If still unmerged, record it separately. This allows us to later\n> > + * iterate over just unmerged entries when updating the index instead\n> > + * of iterating over all entries.\n> > + */\n> > + if (!ci->merged.clean)\n> > + strmap_put(&opt->priv->unmerged, path, ci);\n> > +}\n> > +\n> > static void process_entries(struct merge_options *opt,\n> > struct object_id *result_oid)\n> > {\n> > - die(\"Not yet implemented.\");\n> > + struct hashmap_iter iter;\n> > + struct strmap_entry *e;\n> > +\n> > + if (strmap_empty(&opt->priv->paths)) {\n> > + oidcpy(result_oid, opt->repo->hash_algo->empty_tree);\n> > + return;\n> > + }\n> > +\n> > + strmap_for_each_entry(&opt->priv->paths, &iter, e) {\n> > + /*\n> > + * WARNING: If ci->merged.clean is true, then ci does not\n> > + * actually point to a conflict_info but a struct merge_info.\n> > + */\n> > + struct conflict_info *ci = e->value;\n> > +\n> > + if (!ci->merged.clean)\n> > + process_entry(opt, e->key, e->value);\n> > + }\n> > +\n> > + die(\"Tree creation not yet implemented\");\n>\n> The rest looks straightforward.\n"},{"id":"409779","messageId":"dbc8eddf-a6ef-b087-7714-0dc03f5b32a7@gmail.com","threadId":"54556","inReplyTo":"CABPp-BHCqD8yS_4t7ztL2+FxjPCZ9Meq97VfRhnDRwCjLpdSzg@mail.gmail.com","subject":"Re: [PATCH v2 09/20] merge-ort: record stage and auxiliary info for every path","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-12T18:23:09Z","receivedAt":"2020-11-12T18:23:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/11/2020 5:06 PM, Elijah Newren wrote:\n> On Wed, Nov 11, 2020 at 10:16 AM Elijah Newren wrote:\n>> This was the whole point of the strmap API[1] I recently added --\n>> provide a hashmap specialized for the case where the key is a string.\n>> That way I get fast lookup, and relatively fast resize as the hash\n>> only contains pointers to the values, not a copy of the values.\n>>\n>> Is your concern that allocating many small structs is more expensive\n>> than allocating a huge block of them? If so, yes that matters, but\n>> see the mem_pool related patches of the strmap API[1].\n>>\n>> [1] https://lore.kernel.org/git/pull.835.v5.git.git.1604622298.gitgitgadget@gmail.com/\n> \n> \n> I just re-read what I wrote, here and below...and I need to apologize.\n> I tend to write, edit, revise, and repeat while composing emails and\n> the end result of my emails doesn't tend to reflect the path to get\n> there; I looped through that cycle more times than most on this email.\n> But, even worse, I added in a sentence or two that just shouldn't be\n> included regardless. I think in particular this one sounds extremely\n> aggressive and dismissive which was not at all my intent.\n> \n> I find your reviews to be very helpful, and I don't want to discourage\n> them. Hopefully my comments didn't come across anywhere near as\n> strongly as they did to me on a second reading, but if they did, I'm\n> sorry.\n\nI did not feel any animosity. Your response was just so involved that\nI didn't have time to respond in kind. I hope to do so later today.\n\nThe short version of my reply is \"You know more about this than me,\nand I appreciate the additional detail.\"\n\nThanks,\n-Stolee\n"},{"id":"409783","messageId":"306958b5-030a-8653-a997-0bf6ad1d7d4b@gmail.com","threadId":"54556","inReplyTo":"CABPp-BFRPuxLYz_n6jbr=j7Gu1GhsV95nKPE1=HxUcsvimrz0g@mail.gmail.com","subject":"Re: [PATCH v2 09/20] merge-ort: record stage and auxiliary info for every path","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-11-12T18:39:26Z","receivedAt":"2020-11-12T18:39:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/11/2020 1:16 PM, Elijah Newren wrote:\n> On Wed, Nov 11, 2020 at 7:26 AM Derrick Stolee wrote:\n>>\n>> On 11/2/2020 3:43 PM, Elijah Newren wrote:\n>>> +static void setup_path_info(struct merge_options *opt,\n>>> + struct string_list_item *result,\n>>> + const char *current_dir_name,\n>>> + int current_dir_name_len,\n>>> + char *fullpath, /* we'll take over ownership */\n>>> + struct name_entry *names,\n>>> + struct name_entry *merged_version,\n>>> + unsigned is_null, /* boolean */\n>>> + unsigned df_conflict, /* boolean */\n>>> + unsigned filemask,\n>>> + unsigned dirmask,\n>>> + int resolved /* boolean */)\n>>> +{\n>>> + struct conflict_info *path_info;\n>>\n>> In addition to my concerns below about 'conflict_info' versus\n>> 'merged_info', I was doubly confused that 'result' in the parameter\n>> list is given a variable named 'pi' for \"path info\" and result->util\n>> eventually is equal to this path_info. What if we renamed 'result'\n>> to 'pi' for \"path info\" here, then operated on 'pi->util' in this\n>> method?\n> \n> result->util (or pi->util if you rename) is void *, making it hard to\n> operate on; you'd have to typecast at every usage. Since it is used a\n> *lot*, it makes sense to have a typed pointer, and then just set\n> result->util to a copy of that value at the end. That is what\n> path_info is for.\n\nGood point. I need to be more careful thinking about types during\nreview.\n\n>>\n>>> + path_info = xcalloc(1, resolved ? sizeof(struct merged_info) :\n>>> + sizeof(struct conflict_info));\n>>\n>> Hm. I'm happy to have a `struct merged_info *` pointing to a\n>> `struct conflict_info`, but the opposite seems very dangerous.\n> \n> Yeah, this is perhaps the scariest bit, and if it were a side data\n> structure rather than the fundamental main one that was central to the\n> algorithm, then safety would trump performance concerns. But since it\n> is the main data structure and likely the biggest (once you count the\n> various copies for each relevant path), then it might be worth the\n> extra care needed to shave off the extra memory. Maybe we can still\n> tweak things to get some safety back without killing performance so\n> let me consider each of your suggestions/questions.\n> \n> If I define it as a merged_info*, the compiler will only let me modify\n> fields within the merged_info portion of the struct. Should I\n> typecast every line that touches the bits in the resolved==0 path\n> where I need to set fields within the conflict_info portion?\n> Alternatively, would a code flow like the following make you happier?\n> \n> struct conflict_info *ci = NULL;\n> struct merge_info *mi = xcalloc(...);\n> result->util = mi;\n> /* Operate on mi */\n> ...\n> if (resolved)\n> return;\n> ci = mi;\n> /* Operate on ci */\n> ...\n> \n> In either case, the returned item has potentially different sizes, so\n> the caller will still have to take care so I'm not sure how much extra\n> this structure within setup_path_info() buys us.\n\nThere might be good reason to use this example. Specifically,\nalways first cast into a 'struct merge_info *mi' then check\n'mi->clean' before casting into 'struct conflict_info *ci'. It\ndefinitely helps that something within the smaller memory\nfootprint gives an indicator as to whether the larger struct\nshould exist.\n \n>> Perhaps we should always use sizeof(struct conflict_info)?\n> \n> We could do that; it'd certainly waste memory as I expect many entries\n> to be unmodified (on one or both sides of history). But I'd way\n> rather go this route than splitting or re-arranging this data\n> structure.\n\nYes, I agree exactly on this.\n\n>> We can use path_info->merged.clean to detect whether the rest of\n>> the data is worth looking at. (Or, in your case, whether or not\n>> it is allocated.)\n> \n> ci->merged.clean is used to determine whether to look at the rest of\n> the data, yes -- and that's an enforced assumption throughout the code\n> (as alluded to by the comment in the merge_options_internal data\n> structure that \"paths\" maps pathanemes to merge_info and conflict_info\n> types). However, that is not quite the same as using the clean bit to\n> determine if more data is allocated; something can be allocated as a\n> conflict_info rather than a merged_info due to both sides making\n> modifying the same path, but then a threeway content merge comes back\n> clean and ci->merged.clean is updated from 0 to 1. The extra data\n> remains allocated, but nothing in the algorithm ever needs to use\n> anything outside the merged bits for that path again. (Actually, let\n> me state that more forcefully: nothing is *allowed* to look outside\n> the merged bits for that path once the clean bit is updated to 1).\n\nOk, so the two-stage casting (merge_info then conflict_info) would\nstill work even after the clean bit is enabled eventually. This assumes\nthat the threeway content merge data is cleaned up before losing\nthe conflict_info pointer.\n\n>> I imagine that in a large repo we will need many of these structs,\n>> but very few of them will actually need to be conflicts, so using\n>> 'struct conflict_info' always will lead to memory bloat. But in\n>> that case, would we not be better off with an array instead of a\n>> scattering of data across the heap?\n> \n> Not sure what you're trying to solve here. Putting them in an array\n> would mean copying every single one of them every time the array is\n> resized. It would also make insertion or deletion very expensive.\n> And it'd prevent O(1) lookup. It'd be a horrible data structure all\n> around. Maybe you're assuming you know exactly how many entries you\n> need and what they are before the merge algorithm starts? I don't.\n> In fact, I can't even give a good magnitude approximation of how many\n> it'll be before a merge starts. (Even if you assume it's a case where\n> you have an index loaded and that index is related to the merge being\n> done, the number can be and often is much smaller than the number of\n> entries in the index. And just to cover the extremes, in unusual\n> cases the number might be much larger than the number of index entries\n> if the merge base and side being merged in has far more paths).\n\nYeah, I was mostly thinking about pooling allocations to reduce memory\nfragmentation. But its likely that we don't need to do that, or rather\nyou are already doing some of that in the strmap structure.\n\n> This was the whole point of the strmap API[1] I recently added --\n> provide a hashmap specialized for the case where the key is a string.\n> That way I get fast lookup, and relatively fast resize as the hash\n> only contains pointers to the values, not a copy of the values.\n> \n> Is your concern that allocating many small structs is more expensive\n> than allocating a huge block of them? If so, yes that matters, but\n> see the mem_pool related patches of the strmap API[1].\n> \n> [1] https://lore.kernel.org/git/pull.835.v5.git.git.1604622298.gitgitgadget@gmail.com/\n\nIt appears that you already do some mempool stuff there [2], so I'm sure\nyou know more about how to optimize memory here. My initial reaction of\n\"that's a lot of calloc()s\" could easily be punted to a later improvement\n_if_ it is valuable at all.\n\n[2] https://lore.kernel.org/git/3926c4c97bd08aac93d3f521273db9d76b4d5cd3.1605124942.git.gitgitgadget@gmail.com/\n\n>> Perhaps 'struct conflict_info' shouldn't contain a 'struct merged_info'\n>> and instead be just the \"extra\" data. Then we could have a contiguous\n>> array of 'struct merged_info' values for most of the paths, but heap\n>> pointers for 'struct conflict_info' as necessary.\n>>\n>> It's also true that I haven't fully formed a mental model for how these\n>> are used in your algorithm, so I'll keep reading.\n> \n> I don't understand how contiguous arrays are practical or desirable\n> (I'm close to saying they're not possible, but one could employ some\n> extremes to get them, as mentioned above).\n> \n> I could possibly have two strmaps; one mapping paths to a merge_info,\n> and another (with fewer entries) mapping paths to a conflict_info.\n> Seems like a royal pain, and would make for some pretty ugly code (I\n> have other places that had to use two strmaps and I've hated it every\n> time -- but those were cases of strmaps that were used much, much less\n> than the \"paths\" one). Might also slightly hurt perf\n\nI am convinced that the alternatives are harder to implement with no\nclear benefit.\n\n>>> + path_info->merged.directory_name = current_dir_name;\n>>> + path_info->merged.basename_offset = current_dir_name_len;\n>>> + path_info->merged.clean = !!resolved;\n>>> + if (resolved) {\n>>> + path_info->merged.result.mode = merged_version->mode;\n>>> + oidcpy(&path_info->merged.result.oid, &merged_version->oid);\n>>> + path_info->merged.is_null = !!is_null;\n>>> + } else {\n>>> + int i;\n>>> +\n>>> + for (i = 0; i < 3; i++) {\n>>> + path_info->pathnames[i] = fullpath;\n>>> + path_info->stages[i].mode = names[i].mode;\n>>> + oidcpy(&path_info->stages[i].oid, &names[i].oid);\n>>> + }\n>>> + path_info->filemask = filemask;\n>>> + path_info->dirmask = dirmask;\n>>> + path_info->df_conflict = !!df_conflict;\n>>> + }\n>>> + strmap_put(&opt->priv->paths, fullpath, path_info);\n>>> + result->string = fullpath;\n>>> + result->util = path_info;\n>>\n>> This is set in all cases, so should we use it everywhere? Naturally,\n>> there might be a cost to the extra pointer indirection, so maybe we\n>> create a 'struct conflict_info *util' to operate on during this\n>> method, but set 'result->util = util' right after allocating so we\n>> know how it should behave?\n> \n> result->util is void*, so it's not just an extra pointer indirection,\n> it's also the need to cast it to the appropriate type every time you\n> want to use it. It's easier to have that done via another copy of the\n> pointer with the correct type, which is the reason for path_info. So,\n> essentially, I did use util everywhere, it's just that I spelled it as\n> \"path_info\". If I had named \"path_info\" \"util\" as you suggest,\n> wouldn't everyone be annoyed that I used a lame name that didn't name\n> the variable's purpose?\n> \n> Perhaps I should just add a comment saying that path_util is a typed\n> alias/copy of result->util when I define it?\n\nA comment wouldn't hurt.\n\n>>> @@ -91,10 +136,12 @@ static int collect_merge_info_callback(int n,\n>>> */\n>>> struct merge_options *opt = info->data;\n>>> struct merge_options_internal *opti = opt->priv;\n>>> - struct conflict_info *ci;\n>>> + struct string_list_item pi; /* Path Info */\n>>> + struct conflict_info *ci; /* pi.util when there's a conflict */\n> \n> Perhaps here I should mention that ci is just a typed copy of pi.util\n> (since pi.util is a void*).\n> \n>> ...\n>>\n>>> + setup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n>>> + names, NULL, 0, df_conflict, filemask, dirmask, 0);\n>>> + ci = pi.util;\n>>\n>> Here is the use of 'pi' that I was talking about earlier.\n> \n> ...although, to be fair, I don't actually have all that many uses of\n> ci (at least not anymore) in this function. So maybe typecasting\n> pi.util each of the three-or-so times it is used isn't so bad?\n \nWhile I usually use \"three is many\" for justifying extracting\nduplicated code, for casting issues I usually think \"two is\ntoo many\" is appropriate. Keep 'ci'.\n\nThanks,\n-Stolee\n"},{"id":"409791","messageId":"20201112201524.3438029-1-jonathantanmy@google.com","threadId":"54556","inReplyTo":"20201102204344.342633-16-newren@gmail.com","subject":"Re: [PATCH v2 15/20] merge-ort: step 3 of tree writing -- handling subdirectories as we go","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-12T20:15:24Z","receivedAt":"2020-11-12T20:15:32Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Firstly, from [1]:\n\n> Thanks for the reviews! I was hoping to see some comments on patch\n> 15, as it's possibly the gnarliest. It's a relatively straightforward\n> algorithm, just lots of bookkeeping.\n\nI was planning to send this out yesterday, but couldn't finish it. :-P\nIndeed, a lot of things to think about.\n\n[1] https://lore.kernel.org/git/CABPp-BFgQX6Ash03x7z+RfE3ytbw3x0DzDSBrGddgMr_soODoA@mail.gmail.com/\n\n[snip commit message]\n\nThanks for the thorough explanation.\n\n> @@ -353,6 +353,9 @@ static int string_list_df_name_compare(const char *one, const char *two)\n> \n> struct directory_versions {\n> \tstruct string_list versions;\n> +\tstruct string_list offsets;\n\nLooking below (and at the explanation in the commit message), this is a\nmapping from full paths to integers casted to the pointer type.\n\n> +\tconst char *last_directory;\n> +\tunsigned last_directory_len;\n\nIs this just the last entry in \"versions\"?\n\n> static void write_tree(struct object_id *result_oid,\n> @@ -409,12 +412,100 @@ static void record_entry_for_tree(struct directory_versions *dir_metadata,\n> \t\t/* nothing to record */\n> \t\treturn;\n> \n> +\t/*\n> +\t * Note: write_completed_directories() already added\n> +\t * entries for directories to dir_metadata->versions,\n> +\t * so no need to handle ci->filemask == 0 again.\n> +\t */\n> +\tif (!ci->merged.clean && !ci->filemask)\n> +\t\treturn;\n> +\n> \tbasename = path + ci->merged.basename_offset;\n> \tassert(strchr(basename, '/') == NULL);\n> \tstring_list_append(&dir_metadata->versions,\n> \t\t\t basename)->util = &ci->merged.result;\n> }\n\nConceptually, I can see how the algorithm below inserts directories, but\nI don't understand the significance of \"!ci->merged.clean\" in the change\nabove.\n\n> +static void write_completed_directories(struct merge_options *opt,\n> +\t\t\t\t\tconst char *new_directory_name,\n> +\t\t\t\t\tstruct directory_versions *info)\n> +{\n> +\tconst char *prev_dir;\n> +\tstruct merged_info *dir_info = NULL;\n> +\tunsigned int offset;\n> +\tint wrote_a_new_tree = 0;\n> +\n> +\tif (new_directory_name == info->last_directory)\n> +\t\treturn;\n\nPointer equality is OK here presumably because of the string interning\nof directory names.\n\nI'm starting to think that it might be too difficult to keep track of\nwhere strings are interned. Maybe there should be a data structure\ncontaining all interned strings, and make the path a struct or something\nlike that (to clearly indicate that the string inside comes from the\ninterned string data structure).\n\n> +\t/*\n> +\t * If we are just starting (last_directory is NULL), or last_directory\n> +\t * is a prefix of the current directory, then we can just update\n> +\t * last_directory and record the offset where we started this directory.\n> +\t */\n> +\tif (info->last_directory == NULL ||\n> +\t !strncmp(new_directory_name, info->last_directory,\n> +\t\t info->last_directory_len)) {\n\nGit has starts_with() for prefix checking. (May not be as optimized as\nthis one, though.)\n\n> +\t\tuintptr_t offset = info->versions.nr;\n> +\n> +\t\tinfo->last_directory = new_directory_name;\n> +\t\tinfo->last_directory_len = strlen(info->last_directory);\n> +\t\tstring_list_append(&info->offsets,\n> +\t\t\t\t info->last_directory)->util = (void*)offset;\n> +\t\treturn;\n> +\t}\n\nDue to the way this is sorted, there might be a jump of 2 or more\ndirectories. (The commit message also provides such an example - from \"\"\nto \"src/moduleB\", without going through \"src\".)\n\n> +\t/*\n> +\t * At this point, ne (next entry) is within a different directory\n> +\t * than the last entry, so we need to create a tree object for all\n> +\t * the entries in info->versions that are under info->last_directory.\n> +\t */\n\nThere's no \"ne\" below.\n\n> +\tdir_info = strmap_get(&opt->priv->paths, info->last_directory);\n> +\tassert(dir_info);\n> +\toffset = (uintptr_t)info->offsets.items[info->offsets.nr-1].util;\n> +\tif (offset == info->versions.nr) {\n> +\t\tdir_info->is_null = 1;\n> +\t} else {\n> +\t\tdir_info->result.mode = S_IFDIR;\n> +\t\twrite_tree(&dir_info->result.oid, &info->versions, offset);\n> +\t\twrote_a_new_tree = 1;\n> +\t}\n\nI was trying to figure out the cases in which offset would be\ninfo->versions.nr - if such a case existed, and if yes, would it be\nincorrect to skip creating such a tree because presumably this offset\nexists in info->offsets for a reason. Do you know in which situation\noffset would equal info->versions.nr?\n\n> +\t/*\n> +\t * We've now used several entries from info->versions and one entry\n> +\t * from info->offsets, so we get rid of those values.\n> +\t */\n> +\tinfo->offsets.nr--;\n> +\tinfo->versions.nr = offset;\n\nOK.\n\n> +\t/*\n> +\t * Now we've got an OID for last_directory in dir_info. We need to\n> +\t * add it to info->versions for it to be part of the computation of\n> +\t * its parent directories' OID. But first, we have to find out what\n> +\t * its' parent name was and whether that matches the previous\n> +\t * info->offsets or we need to set up a new one.\n> +\t */\n> +\tprev_dir = info->offsets.nr == 0 ? NULL :\n> +\t\t info->offsets.items[info->offsets.nr-1].string;\n> +\tif (new_directory_name != prev_dir) {\n> +\t\tuintptr_t c = info->versions.nr;\n> +\t\tstring_list_append(&info->offsets,\n> +\t\t\t\t new_directory_name)->util = (void*)c;\n> +\t}\n\nBecause of the possible jump of 2 or more directories that I mentioned\nearlier, there may be gaps in the offsets. So it makes sense that we\nsometimes need to insert an intermediate one.\n\nI wonder if the code would be clearer if we had explicit \"begin tree\"\nand \"end tree\" steps just like in list-objects-filter.c (LOFS_BEGIN_TREE\nand LOFS_END_TREE). Here we have \"end tree\" (because of the way the\nentries were sorted) but not \"begin tree\". If we had \"begin tree\", we\nprobably would be able to create the necessary offsets in a loop at that\nstage, and the reasoning about the contents of the offsets would not be\nso complicated.\n\nIf we really only want one side (i.e. you don't want to introduce a\nsynthetic entry just to mark the end or the beginning), then my personal\nexperience is that having the \"begin\" side is easier to understand, as\nthe state is more natural and easier to reason about. (Unlike here,\nwhere there could be gaps in the offsets and the reader has to\nunderstand that the gaps will be filled just in time.) But that may just\nbe my own experience.\n\n> +\t/*\n> +\t * Okay, finally record OID for last_directory in info->versions,\n> +\t * and update last_directory.\n> +\t */\n> +\tif (wrote_a_new_tree) {\n> +\t\tconst char *dir_name = strrchr(info->last_directory, '/');\n> +\t\tdir_name = dir_name ? dir_name+1 : info->last_directory;\n> +\t\tstring_list_append(&info->versions, dir_name)->util = dir_info;\n> +\t}\n> +\tinfo->last_directory = new_directory_name;\n> +\tinfo->last_directory_len = strlen(info->last_directory);\n> +}\n\nOK - several entries in info->versions were deleted earlier (through\ninfo->versions.nr = offset), and we add one here to represent the tree\ncontaining all those deleted versions.\n\n> @@ -541,22 +635,27 @@ static void process_entries(struct merge_options *opt,\n> \t\t */\n> \t\tstruct conflict_info *ci = entry->util;\n> \n> +\t\twrite_completed_directories(opt, ci->merged.directory_name,\n> +\t\t\t\t\t &dir_metadata);\n> \t\tif (ci->merged.clean)\n> \t\t\trecord_entry_for_tree(&dir_metadata, path, ci);\n> \t\telse\n> \t\t\tprocess_entry(opt, path, ci, &dir_metadata);\n> \t}\n\nTrying to make sense of this: we pass in the directory name of the\ncurrent entry so that if the last directory is *not* a prefix of the\ncurrent directory (so we either went up a directory or went sideways),\nthen we write a tree (unless offset == info->versions.nr, which as I\nstated above, I still don't fully understand - I thought we would always\nhave to write a tree). So maybe the name of the function should be\n\"write_completed_directory\" (and document it as \"write a tree if\n???\"), since we write at most one.\n\nIn this kind of algorithm (where intermediate accumulated results are\nbeing written), there needs to be a last write after the loop that\nwrites whatever's left in the accumulation buffer. I do see it below\n(\"write_tree\"), so that's good.\n\n> -\t/*\n> -\t * TODO: We can't actually write a tree yet, because dir_metadata just\n> -\t * contains all basenames of all files throughout the tree with their\n> -\t * mode and hash. Not only is that a nonsensical tree, it will have\n> -\t * lots of duplicates for paths such as \"Makefile\" or \".gitignore\".\n> -\t */\n> -\tdie(\"Not yet implemented; need to process subtrees separately\");\n> +\tif (dir_metadata.offsets.nr != 1 ||\n> +\t (uintptr_t)dir_metadata.offsets.items[0].util != 0) {\n> +\t\tprintf(\"dir_metadata.offsets.nr = %d (should be 1)\\n\",\n> +\t\t dir_metadata.offsets.nr);\n> +\t\tprintf(\"dir_metadata.offsets.items[0].util = %u (should be 0)\\n\",\n> +\t\t (unsigned)(uintptr_t)dir_metadata.offsets.items[0].util);\n> +\t\tfflush(stdout);\n> +\t\tBUG(\"dir_metadata accounting completely off; shouldn't happen\");\n> +\t}\n\nSanity check, OK.\n\n[snip rest]\n\nIn summary, I think that the code would be easier to understand (for\neveryone) if there were both BEGIN_TREE and END_TREE entries. And for me\npersonally, once the offset == info->versions.nr part is clarified\n(perhaps there is something obvious that I'm missing).\n"},{"id":"409795","messageId":"20201112203904.3443874-1-jonathantanmy@google.com","threadId":"54556","inReplyTo":"CABPp-BGr4kpjyZc3Rbh+8StPbWWqo36Kg0P=dgqctwX0SCGPxw@mail.gmail.com","subject":"Re: [PATCH v2 13/20] merge-ort: step 1 of tree writing -- record basenames, modes, and oids","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-12T20:39:04Z","receivedAt":"2020-11-12T20:39:13Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On Wed, Nov 11, 2020 at 12:01 PM Jonathan Tan wrote:\n> >\n> > > +struct directory_versions {\n> > > + struct string_list versions;\n> >\n> > Maybe comment that this is an unordered list of basenames to > the type of ci->merged.result is>.\n> \n> There actually is an order, and it's important. It's reverse\n> lexicographic order of full pathnames (the ordering comes from the\n> fact that process_entries() iterates paths in that order). The\n> reasons for that ordering are (1) all the basenames within a directory\n> are adjacent so that I can write out a tree for a directory as soon as\n> it is done, and (2) paths within a directory are listed before the\n> directory itself so that I get the necessary info for subtrees before\n> trying to write out their parent trees.\n> \n> It's not until later patches that I take advantage of this ordering\n> (and when I do I have a very long commit message to describe it all),\n> but I can add a comment that this is a list of basenames to\n> merge_info.\n\nAh, yes you're right. I'm not sure what I was thinking of.\n"},{"id":"409804","messageId":"CABPp-BFwn5+S_swrtZEq6DmZx6kbB0xO_d=ZJkqs4H6FG1vo1Q@mail.gmail.com","threadId":"54556","inReplyTo":"20201112201524.3438029-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 15/20] merge-ort: step 3 of tree writing -- handling subdirectories as we go","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-12T22:30:43Z","receivedAt":"2020-11-12T22:30:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Nov 12, 2020 at 12:15 PM Jonathan Tan wrote:\n>\n> Firstly, from [1]:\n>\n> > Thanks for the reviews! I was hoping to see some comments on patch\n> > 15, as it's possibly the gnarliest. It's a relatively straightforward\n> > algorithm, just lots of bookkeeping.\n>\n> I was planning to send this out yesterday, but couldn't finish it. :-P\n> Indeed, a lot of things to think about.\n>\n> [1] https://lore.kernel.org/git/CABPp-BFgQX6Ash03x7z+RfE3ytbw3x0DzDSBrGddgMr_soODoA@mail.gmail.com/\n>\n> [snip commit message]\n>\n> Thanks for the thorough explanation.\n>\n> > @@ -353,6 +353,9 @@ static int string_list_df_name_compare(const char *one, const char *two)\n> >\n> > struct directory_versions {\n> > struct string_list versions;\n> > + struct string_list offsets;\n>\n> Looking below (and at the explanation in the commit message), this is a\n> mapping from full paths to integers casted to the pointer type.\n>\n> > + const char *last_directory;\n> > + unsigned last_directory_len;\n>\n> Is this just the last entry in \"versions\"?\n\nNo, it's a simple cache of strlen(info->last_directory), so I don't\nhave to recompute that length multiple times. Perhaps I should add a\ncomment to that effect.\n\n> > static void write_tree(struct object_id *result_oid,\n> > @@ -409,12 +412,100 @@ static void record_entry_for_tree(struct directory_versions *dir_metadata,\n> > /* nothing to record */\n> > return;\n> >\n> > + /*\n> > + * Note: write_completed_directories() already added\n> > + * entries for directories to dir_metadata->versions,\n> > + * so no need to handle ci->filemask == 0 again.\n> > + */\n> > + if (!ci->merged.clean && !ci->filemask)\n> > + return;\n> > +\n> > basename = path + ci->merged.basename_offset;\n> > assert(strchr(basename, '/') == NULL);\n> > string_list_append(&dir_metadata->versions,\n> > basename)->util = &ci->merged.result;\n> > }\n>\n> Conceptually, I can see how the algorithm below inserts directories, but\n> I don't understand the significance of \"!ci->merged.clean\" in the change\n> above.\n\nChecking ci->filemask is likely an out-of-bounds memory read if\nci->merged.clean is true. (ci may point to something that was\nallocated with the size of a merge_info or a conflict_info.) Perhaps\nI could extend the comment to say that conflicted directories, i.e.\npaths that are unclean with ci->filemask == 0 can be skipped because\nthey were already handled.\n\n> > +static void write_completed_directories(struct merge_options *opt,\n> > + const char *new_directory_name,\n> > + struct directory_versions *info)\n> > +{\n> > + const char *prev_dir;\n> > + struct merged_info *dir_info = NULL;\n> > + unsigned int offset;\n> > + int wrote_a_new_tree = 0;\n> > +\n> > + if (new_directory_name == info->last_directory)\n> > + return;\n>\n> Pointer equality is OK here presumably because of the string interning\n> of directory names.\n\nYes, precisely.\n\n> I'm starting to think that it might be too difficult to keep track of\n> where strings are interned. Maybe there should be a data structure\n> containing all interned strings, and make the path a struct or something\n> like that (to clearly indicate that the string inside comes from the\n> interned string data structure).\n\nGood news: Interned strings already are stored as the keys of our\nprimary data structure -- the strmap known as opt->priv->paths. All\nrelative paths from the root of the repository that are relevant to\nthe merge at all -- both for files and directories -- are interned by\ncollect_merge_info() inside that \"paths\" strmap.\n\nI guess I should note that it eventually gets _slightly_ more\ncomplicated. Due to renames and directory renames, I might need to\nremove a path from opt->priv->paths. In such a case there will be an\nauxiliary string_list named \"paths_to_free\" that stores the interned\nstrings which are no longer part of opt->priv->paths.\n\n> > + /*\n> > + * If we are just starting (last_directory is NULL), or last_directory\n> > + * is a prefix of the current directory, then we can just update\n> > + * last_directory and record the offset where we started this directory.\n> > + */\n> > + if (info->last_directory == NULL ||\n> > + !strncmp(new_directory_name, info->last_directory,\n> > + info->last_directory_len)) {\n>\n> Git has starts_with() for prefix checking. (May not be as optimized as\n> this one, though.)\n>\n> > + uintptr_t offset = info->versions.nr;\n> > +\n> > + info->last_directory = new_directory_name;\n> > + info->last_directory_len = strlen(info->last_directory);\n> > + string_list_append(&info->offsets,\n> > + info->last_directory)->util = (void*)offset;\n> > + return;\n> > + }\n>\n> Due to the way this is sorted, there might be a jump of 2 or more\n> directories. (The commit message also provides such an example - from \"\"\n> to \"src/moduleB\", without going through \"src\".)\n>\n> > + /*\n> > + * At this point, ne (next entry) is within a different directory\n> > + * than the last entry, so we need to create a tree object for all\n> > + * the entries in info->versions that are under info->last_directory.\n> > + */\n>\n> There's no \"ne\" below.\n\nOops, that code has been heavily refactored since that comment.\nSomething like this would be more up-to-date:\n /*\n * The next entry will be within new_directory_name. Since at this\n * point we know that new_directory_name is within a different\n * directory than info->last_directory, we have all entries for\n * info->last_directory in info->versions and we need to create a\n * tree object for them.\n */\n\n> > + dir_info = strmap_get(&opt->priv->paths, info->last_directory);\n> > + assert(dir_info);\n> > + offset = (uintptr_t)info->offsets.items[info->offsets.nr-1].util;\n> > + if (offset == info->versions.nr) {\n> > + dir_info->is_null = 1;\n> > + } else {\n> > + dir_info->result.mode = S_IFDIR;\n> > + write_tree(&dir_info->result.oid, &info->versions, offset);\n> > + wrote_a_new_tree = 1;\n> > + }\n>\n> I was trying to figure out the cases in which offset would be\n> info->versions.nr - if such a case existed, and if yes, would it be\n> incorrect to skip creating such a tree because presumably this offset\n> exists in info->offsets for a reason. Do you know in which situation\n> offset would equal info->versions.nr?\n\nYes, it is possible that all files within the directory become empty\nas a result of merging[1], and in such cases this line of logic will\ntrigger (note that record_entry_for_tree(), which is what adds more\nitems to info->versions, returns early if ci->merged.is_null). We do\nnot want to write out an empty tree for the directory or record the\ntree's hash for its parent directory, we simply want to omit it\nentirely. Omitting it entirely is handled by the line\n\"dir_info->is_null = 1\".\n\n[1] The simplest example is when one side doesn't touch anything\nwithin a directory but the other side deletes the whole directory.\nFiles can also disappear in a merge for other reasons, such as being\ndeleted on both sides, or being renamed. If _all_ files within the\ndirectory are removed by the merge logic, the directory has no\nentries.\n\n>\n> > + /*\n> > + * We've now used several entries from info->versions and one entry\n> > + * from info->offsets, so we get rid of those values.\n> > + */\n> > + info->offsets.nr--;\n> > + info->versions.nr = offset;\n>\n> OK.\n>\n> > + /*\n> > + * Now we've got an OID for last_directory in dir_info. We need to\n> > + * add it to info->versions for it to be part of the computation of\n> > + * its parent directories' OID. But first, we have to find out what\n> > + * its' parent name was and whether that matches the previous\n> > + * info->offsets or we need to set up a new one.\n> > + */\n> > + prev_dir = info->offsets.nr == 0 ? NULL :\n> > + info->offsets.items[info->offsets.nr-1].string;\n> > + if (new_directory_name != prev_dir) {\n> > + uintptr_t c = info->versions.nr;\n> > + string_list_append(&info->offsets,\n> > + new_directory_name)->util = (void*)c;\n> > + }\n>\n> Because of the possible jump of 2 or more directories that I mentioned\n> earlier, there may be gaps in the offsets. So it makes sense that we\n> sometimes need to insert an intermediate one.\n>\n> I wonder if the code would be clearer if we had explicit \"begin tree\"\n> and \"end tree\" steps just like in list-objects-filter.c (LOFS_BEGIN_TREE\n> and LOFS_END_TREE). Here we have \"end tree\" (because of the way the\n> entries were sorted) but not \"begin tree\". If we had \"begin tree\", we\n> probably would be able to create the necessary offsets in a loop at that\n> stage, and the reasoning about the contents of the offsets would not be\n> so complicated.\n>\n> If we really only want one side (i.e. you don't want to introduce a\n> synthetic entry just to mark the end or the beginning), then my personal\n> experience is that having the \"begin\" side is easier to understand, as\n> the state is more natural and easier to reason about. (Unlike here,\n> where there could be gaps in the offsets and the reader has to\n> understand that the gaps will be filled just in time.) But that may just\n> be my own experience.\n\nInteresting, I'll take a look into it.\n\n>\n> > + /*\n> > + * Okay, finally record OID for last_directory in info->versions,\n> > + * and update last_directory.\n> > + */\n> > + if (wrote_a_new_tree) {\n> > + const char *dir_name = strrchr(info->last_directory, '/');\n> > + dir_name = dir_name ? dir_name+1 : info->last_directory;\n> > + string_list_append(&info->versions, dir_name)->util = dir_info;\n> > + }\n> > + info->last_directory = new_directory_name;\n> > + info->last_directory_len = strlen(info->last_directory);\n> > +}\n>\n> OK - several entries in info->versions were deleted earlier (through\n> info->versions.nr = offset), and we add one here to represent the tree\n> containing all those deleted versions.\n>\n> > @@ -541,22 +635,27 @@ static void process_entries(struct merge_options *opt,\n> > */\n> > struct conflict_info *ci = entry->util;\n> >\n> > + write_completed_directories(opt, ci->merged.directory_name,\n> > + &dir_metadata);\n> > if (ci->merged.clean)\n> > record_entry_for_tree(&dir_metadata, path, ci);\n> > else\n> > process_entry(opt, path, ci, &dir_metadata);\n> > }\n>\n> Trying to make sense of this: we pass in the directory name of the\n> current entry so that if the last directory is *not* a prefix of the\n> current directory (so we either went up a directory or went sideways),\n> then we write a tree (unless offset == info->versions.nr, which as I\n> stated above, I still don't fully understand - I thought we would always\n> have to write a tree). So maybe the name of the function should be\n> \"write_completed_directory\" (and document it as \"write a tree if\n> ???\"), since we write at most one.\n\nYeah, write_completed_directory() would be better. And just to\nreiterate on the offset == info->versions.nr thing, we do not want to\nwrite a tree if it turns out that the merged result of all files\nwithin the directory is to delete them all.\n\n> In this kind of algorithm (where intermediate accumulated results are\n> being written), there needs to be a last write after the loop that\n> writes whatever's left in the accumulation buffer. I do see it below\n> (\"write_tree\"), so that's good.\n>\n> > - /*\n> > - * TODO: We can't actually write a tree yet, because dir_metadata just\n> > - * contains all basenames of all files throughout the tree with their\n> > - * mode and hash. Not only is that a nonsensical tree, it will have\n> > - * lots of duplicates for paths such as \"Makefile\" or \".gitignore\".\n> > - */\n> > - die(\"Not yet implemented; need to process subtrees separately\");\n> > + if (dir_metadata.offsets.nr != 1 ||\n> > + (uintptr_t)dir_metadata.offsets.items[0].util != 0) {\n> > + printf(\"dir_metadata.offsets.nr = %d (should be 1)\\n\",\n> > + dir_metadata.offsets.nr);\n> > + printf(\"dir_metadata.offsets.items[0].util = %u (should be 0)\\n\",\n> > + (unsigned)(uintptr_t)dir_metadata.offsets.items[0].util);\n> > + fflush(stdout);\n> > + BUG(\"dir_metadata accounting completely off; shouldn't happen\");\n> > + }\n>\n> Sanity check, OK.\n>\n> [snip rest]\n>\n> In summary, I think that the code would be easier to understand (for\n> everyone) if there were both BEGIN_TREE and END_TREE entries. And for me\n> personally, once the offset == info->versions.nr part is clarified\n> (perhaps there is something obvious that I'm missing).\n\nI'm not sure how the BEGIN_TREE/END_TREE entries would look, but I'll\ninvestigate.\n"},{"id":"410703","messageId":"CABPp-BHa21aeXwzykyQAtGd3rzOiE++Gozu7umOSYihc_145iA@mail.gmail.com","threadId":"54556","inReplyTo":"CABPp-BFwn5+S_swrtZEq6DmZx6kbB0xO_d=ZJkqs4H6FG1vo1Q@mail.gmail.com","subject":"Re: [PATCH v2 15/20] merge-ort: step 3 of tree writing -- handling subdirectories as we go","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-24T20:19:35Z","receivedAt":"2020-11-24T20:20:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Nov 12, 2020 at 2:30 PM Elijah Newren wrote:\n>\n> On Thu, Nov 12, 2020 at 12:15 PM Jonathan Tan wrote:\n> >\n> > > + /*\n> > > + * Now we've got an OID for last_directory in dir_info. We need to\n> > > + * add it to info->versions for it to be part of the computation of\n> > > + * its parent directories' OID. But first, we have to find out what\n> > > + * its' parent name was and whether that matches the previous\n> > > + * info->offsets or we need to set up a new one.\n> > > + */\n> > > + prev_dir = info->offsets.nr == 0 ? NULL :\n> > > + info->offsets.items[info->offsets.nr-1].string;\n> > > + if (new_directory_name != prev_dir) {\n> > > + uintptr_t c = info->versions.nr;\n> > > + string_list_append(&info->offsets,\n> > > + new_directory_name)->util = (void*)c;\n> > > + }\n> >\n> > Because of the possible jump of 2 or more directories that I mentioned\n> > earlier, there may be gaps in the offsets. So it makes sense that we\n> > sometimes need to insert an intermediate one.\n> >\n> > I wonder if the code would be clearer if we had explicit \"begin tree\"\n> > and \"end tree\" steps just like in list-objects-filter.c (LOFS_BEGIN_TREE\n> > and LOFS_END_TREE). Here we have \"end tree\" (because of the way the\n> > entries were sorted) but not \"begin tree\". If we had \"begin tree\", we\n> > probably would be able to create the necessary offsets in a loop at that\n> > stage, and the reasoning about the contents of the offsets would not be\n> > so complicated.\n> >\n> > If we really only want one side (i.e. you don't want to introduce a\n> > synthetic entry just to mark the end or the beginning), then my personal\n> > experience is that having the \"begin\" side is easier to understand, as\n> > the state is more natural and easier to reason about. (Unlike here,\n> > where there could be gaps in the offsets and the reader has to\n> > understand that the gaps will be filled just in time.) But that may just\n> > be my own experience.\n>\n> Interesting, I'll take a look into it.\n>\n\nSo, I've been going through making all the changes you and Derrick\nsuggested or highlighted...but I don't see how to tackle this one.\nPerhaps I'm missing something.\n\nUsing your example of LOFS_BEGIN_TREE and LOFS_END_TREE from\nlist-objects-filter.c, I note that you handle it as part of\ntraverse_trees(), and thus you have a very natural \"I'm going to\nprocess this tree\" point and \"I'm done processing this tree\" point.\nThere is no equivalent mapping to merge-ort that I can figure out.\n\nmerge-ort does use traverse_trees() in collect_merge_info(), and fills\nopt->priv->paths with all full pathnames (both files and directories)\nfound in any of the three trees. But I cannot process\nfiles/directories at that time; rename detection needs\ntraverse_trees() to be finished to have all paths so far. Further,\nthe list of pathnames from traverse_trees is not necessarily complete;\nadditional paths could be added by any of\n * Directory/file conflicts (need to move the file to a different location)\n * Directory/submodule conflicts (need to move something to a\ndifferent location)\n * Add/add conflicts of files of different types (e.g.\nsymlink/regular file; can't just content merge them with conflict\nmarkers)\n * Directory rename detection (can move new files or even directories\non one side of history into a new directory on other side)\n\nThus, after traverse_trees() ends, my rename detection stuff can add\npaths (including new directories), then process_entries() can add\npaths -- and remove some when the resolution is to delete. And the\ncode here in question runs as part of the process_entries() loop.\n\nNow, we'd still be able to create synthetic BEGIN_TREE markers if we\noperated in lexicographic ordering, but process_entries() *must*\noperate in _reverse_ lexicographic ordering because:\n 1) subtrees need to be written out before trees are; hashes of those\nsubtrees are used in the parent tree\n 2) it's the only sane way to handle directory/file conflicts; I need\nto know if all entries under the directory resolve to nothing; if not,\nthe directory is still in the way when it comes time to process the\nfile.\n\nGranted, I could do some tricky calculations based on the reverse\nlexicographic ordering of fullpaths (and their containing directories)\nto figure out where trees begin and end -- but that takes us to\nexactly what I *did* do. It was precisely this step that you thought\nshould be made simpler, but I'm not seeing how to avoid it.\n\nFor now, I'll keep the code as-is, but add more comments to both the\ndata structure and the code. If I've missed something about how I\ncould make use of your BEGIN_TREE idea, let me know and I'll look at\nit again.\n"},{"id":"410746","messageId":"20201125020701.956393-1-jonathantanmy@google.com","threadId":"54556","inReplyTo":"CABPp-BHa21aeXwzykyQAtGd3rzOiE++Gozu7umOSYihc_145iA@mail.gmail.com","subject":"Re: [PATCH v2 15/20] merge-ort: step 3 of tree writing -- handling subdirectories as we go","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-25T02:07:01Z","receivedAt":"2020-11-25T02:07:27Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> For now, I'll keep the code as-is, but add more comments to both the\n> data structure and the code. If I've missed something about how I\n> could make use of your BEGIN_TREE idea, let me know and I'll look at\n> it again.\n\nIn collect_merge_info_callback(), you call setup_path_info() to add to\nopt->priv->paths, then call traverse_trees() (which recursively calls\ncollect_merge_info_callback()). I was thinking that in addition to doing\nthat, you could call setup_path_info() a second time, but teach it to\nadd a synthetic path (maybe have a special bit in struct conflict_info\nor something like that) that indicates \"this is the end of the tree\".\nSubsequent code can notice that bit and not do the normal processing,\nbut instead do end-of-tree processing.\n\nHaving said that, maybe it will turn out that your additional comments\nin v3 will be clearer, and we wouldn't need the synthetic entry.\n"},{"id":"410881","messageId":"CABPp-BEfqrT7B=Mu=LWpi3GYkP0SRLFFjPVkca8krairry3iew@mail.gmail.com","threadId":"54556","inReplyTo":"20201125020701.956393-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 15/20] merge-ort: step 3 of tree writing -- handling subdirectories as we go","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-11-26T18:13:27Z","receivedAt":"2020-11-26T18:13:41Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Nov 24, 2020 at 6:07 PM Jonathan Tan wrote:\n>\n> > For now, I'll keep the code as-is, but add more comments to both the\n> > data structure and the code. If I've missed something about how I\n> > could make use of your BEGIN_TREE idea, let me know and I'll look at\n> > it again.\n>\n> In collect_merge_info_callback(), you call setup_path_info() to add to\n> opt->priv->paths, then call traverse_trees() (which recursively calls\n> collect_merge_info_callback()). I was thinking that in addition to doing\n> that, you could call setup_path_info() a second time, but teach it to\n> add a synthetic path (maybe have a special bit in struct conflict_info\n> or something like that) that indicates \"this is the end of the tree\".\n> Subsequent code can notice that bit and not do the normal processing,\n> but instead do end-of-tree processing.\n\nSo, I realized that I already had end-of-tree markers -- the\ndirectories themselves. But due to some other weirdness in how I had\nbuilt up the processing, the existence of those markers was both\nobscured, and deliberately side-stepped. So, I did a little\nrestructuring so we can use these as actual end-of-tree markers more\ndirectly.\n\n> Having said that, maybe it will turn out that your additional comments\n> in v3 will be clearer, and we wouldn't need the synthetic entry.\n\nHopefully it's clearer now, but the entries aren't synthetic. My big\nopt->priv->paths strmap with all full relative paths contained all\nfiles _and_ directories already, and now I just use the directory\nmarkers more directly. Hopefully the extra comments help too.\n"},{"id":"411006","messageId":"20201130184147.3086836-1-jonathantanmy@google.com","threadId":"54556","inReplyTo":"CABPp-BEfqrT7B=Mu=LWpi3GYkP0SRLFFjPVkca8krairry3iew@mail.gmail.com","subject":"Re: [PATCH v2 15/20] merge-ort: step 3 of tree writing -- handling subdirectories as we go","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-11-30T18:41:47Z","receivedAt":"2020-11-30T18:42:43Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On Tue, Nov 24, 2020 at 6:07 PM Jonathan Tan wrote:\n> >\n> > > For now, I'll keep the code as-is, but add more comments to both the\n> > > data structure and the code. If I've missed something about how I\n> > > could make use of your BEGIN_TREE idea, let me know and I'll look at\n> > > it again.\n> >\n> > In collect_merge_info_callback(), you call setup_path_info() to add to\n> > opt->priv->paths, then call traverse_trees() (which recursively calls\n> > collect_merge_info_callback()). I was thinking that in addition to doing\n> > that, you could call setup_path_info() a second time, but teach it to\n> > add a synthetic path (maybe have a special bit in struct conflict_info\n> > or something like that) that indicates \"this is the end of the tree\".\n> > Subsequent code can notice that bit and not do the normal processing,\n> > but instead do end-of-tree processing.\n> \n> So, I realized that I already had end-of-tree markers -- the\n> directories themselves. But due to some other weirdness in how I had\n> built up the processing, the existence of those markers was both\n> obscured, and deliberately side-stepped. So, I did a little\n> restructuring so we can use these as actual end-of-tree markers more\n> directly.\n\nAh sorry...what I meant was to have both begin-of-tree and end-of-tree\nelements in the path list, so one of them is real and the other\nsynthetic. Right now you have an end-of-tree real path in the list of\npaths, yes.\n\n> > Having said that, maybe it will turn out that your additional comments\n> > in v3 will be clearer, and we wouldn't need the synthetic entry.\n> \n> Hopefully it's clearer now, but the entries aren't synthetic. My big\n> opt->priv->paths strmap with all full relative paths contained all\n> files _and_ directories already, and now I just use the directory\n> markers more directly. Hopefully the extra comments help too.\n\nOK - I see that you have a new version [1] and hopefully I'll be able to\ntake a look soon.\n\n[1] https://lore.kernel.org/git/pull.923.git.git.1606635803.gitgitgadget@gmail.com/\n"}]}