{"thread":{"id":"56152","subject":"[PATCH 0/7] Final optimization batch (#15): use memory pools","startedAt":"2021-07-23T12:55:02Z","lastAt":"2021-08-03T15:45:31Z","messageCount":65,"participants":["Elijah Newren via GitGitGadget","Eric Sunshine","Elijah Newren","Derrick Stolee","Jeff King","Junio C Hamano","Ævar Arnfjörð Bjarmason"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"431030","messageId":"pull.990.git.1627044897.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":null,"subject":"[PATCH 0/7] Final optimization batch (#15): use memory pools","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-23T12:54:50Z","receivedAt":"2021-07-23T12:55:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series textually depends on en/ort-perf-batch-14, but the ideas are\northogonal to it and orthogonal to previous series. It can be reviewed\nindependently.\n\nThis series is more about strmaps & memory pools than merge logic. CC'ing\nPeff since he reviewed the strmap work[1], and that work included a number\nof decisions that specifically had this series in mind.\n\n[1]\nhttps://lore.kernel.org/git/20201111200701.GB39046@coredump.intra.peff.net/\n\n=== Basic Optimization idea ===\n\nIn this series, I make use of memory pools to get faster allocations and\ndeallocations for many data structures that tend to all be deallocated at\nthe same time anyway.\n\n=== Results ===\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28), the\nchanges in just this series improves the performance as follows:\n\n Before Series After Series\nno-renames: 204.2 ms ± 3.0 ms 198.3 ms ± 2.9 ms\nmega-renames: 1.076 s ± 0.015 s 661.8 ms ± 5.9 ms\njust-one-mega: 364.1 ms ± 7.0 ms 264.6 ms ± 2.5 ms\n\n\nAs a reminder, before any merge-ort/diffcore-rename performance work, the\nperformance results we started with were:\n\nno-renames-am: 6.940 s ± 0.485 s\nno-renames: 18.912 s ± 0.174 s\nmega-renames: 5964.031 s ± 10.459 s\njust-one-mega: 149.583 s ± 0.751 s\n\n\n=== Overall Results across all optimization work ===\n\nThis is my final prepared optimization series. It might be worth reviewing\nhow my optimizations fared overall, comparing the original merge-recursive\ntimings with three things: how much merge-recursive improved (as a\nside-effect of optimizing merge-ort), how much improvement we would have\ngotten from a hypothetical infinite parallelization of rename detection, and\nwhat I achieved at the end with merge-ort:\n\n Timings\n\n Infinite\n merge- merge- Parallelism\n recursive recursive of rename merge-ort\n v2.30.0 current detection current\n ---------- --------- ----------- ---------\nno-renames: 18.912 s 18.030 s 11.699 s 198.3 ms\nmega-renames: 5964.031 s 361.281 s 203.886 s 661.8 ms\njust-one-mega: 149.583 s 11.009 s 7.553 s 264.6 ms\n\n Speedup factors\n\n Infinite\n merge- merge- Parallelism\n recursive recursive of rename\n v2.30.0 current detection merge-ort\n ---------- --------- ----------- ---------\nno-renames: 1 1.05 1.6 95\nmega-renames: 1 16.5 29 9012\njust-one-mega: 1 13.6 20 565\n\n\nAnd, for partial clone users:\n\n Factor reduction in number of objects needed\n\n Infinite\n merge- merge- Parallelism\n recursive recursive of rename\n v2.30.0 current detection merge-ort\n ---------- --------- ----------- ---------\nmega-renames: 1 1 1 181.3\n\n\n=== Caveat ===\n\nIt may be worth noting, though, that my optimization numbers above for\nmerge-ort use test-tool fast-rebase. git rebase -s ort on the three\ntestcases above is 5-20 times slower (taking 3.835s, 6.798s, and 1.235s,\nrespectively). At this point, any further optimization work should go into\nmaking a faster full-featured rebase by copying the ideas from fast-rebase:\navoid unnecessary process forking, avoid updating the index and working copy\nuntil either the rebase is finished or you hit a conflict (and don't write\nrebase metadata to disk until that point either), get rid of the glacially\nslow revision walking of the upstream side of history (nuke\ncan_fast_forward(), make --reapply-cherry-picks the default) or at least\ndon't revision walk so many times (multiple calls to get_merge_bases in\ncan_fast_forward() plus a is_linear_history() walk, checking for upstream\ncherry-picks, probably more), turn off per-commit hooks that probably should\nhave never been on anyway, etc.\n\nElijah Newren (7):\n diffcore-rename: use a mem_pool for exact rename detection's hashmap\n merge-ort: set up a memory pool\n merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers\n merge-ort: switch our strmaps over to using memory pools\n diffcore-rename, merge-ort: add wrapper functions for filepair\n alloc/dealloc\n merge-ort: store filepairs and filespecs in our mem_pool\n merge-ort: reuse path strings in pool_alloc_filespec\n\n diffcore-rename.c | 66 +++++++++++--\n diffcore.h | 3 +\n merge-ort.c | 247 +++++++++++++++++++++++++++++++++++-----------\n 3 files changed, 251 insertions(+), 65 deletions(-)\n\n\nbase-commit: c9ada8369e6575be488028aae0f654422a9b1410\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-990%2Fnewren%2Fort-perf-batch-15-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-990/newren/ort-perf-batch-15-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/990\n-- \ngitgitgadget\n"},{"id":"431031","messageId":"9f8ab62b84256be6d7d984d576ff4fda09d88a1d.1627044897.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.git.1627044897.gitgitgadget@gmail.com","subject":"[PATCH 1/7] diffcore-rename: use a mem_pool for exact rename detection's hashmap","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-23T12:54:51Z","receivedAt":"2021-07-23T12:55:04Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nExact rename detection, via insert_file_table(), uses a hashmap to store\nfiles by oid. Use a mem_pool for the hashmap entries so these can all be\nallocated and deallocated together.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 204.2 ms ± 3.0 ms 202.5 ms ± 3.2 ms\n mega-renames: 1.076 s ± 0.015 s 1.072 s ± 0.012 s\n just-one-mega: 364.1 ms ± 7.0 ms 357.3 ms ± 3.9 ms\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 20 +++++++++++++++-----\n 1 file changed, 15 insertions(+), 5 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 4ef0459cfb5..23b917eca42 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -317,10 +317,11 @@ static int find_identical_files(struct hashmap *srcs,\n }\n \n static void insert_file_table(struct repository *r,\n+\t\t\t struct mem_pool *pool,\n \t\t\t struct hashmap *table, int index,\n \t\t\t struct diff_filespec *filespec)\n {\n-\tstruct file_similarity *entry = xmalloc(sizeof(*entry));\n+\tstruct file_similarity *entry = mem_pool_alloc(pool, sizeof(*entry));\n \n \tentry->index = index;\n \tentry->filespec = filespec;\n@@ -336,7 +337,8 @@ static void insert_file_table(struct repository *r,\n * and then during the second round we try to match\n * cache-dirty entries as well.\n */\n-static int find_exact_renames(struct diff_options *options)\n+static int find_exact_renames(struct diff_options *options,\n+\t\t\t struct mem_pool *pool)\n {\n \tint i, renames = 0;\n \tstruct hashmap file_table;\n@@ -346,7 +348,7 @@ static int find_exact_renames(struct diff_options *options)\n \t */\n \thashmap_init(&file_table, NULL, NULL, rename_src_nr);\n \tfor (i = rename_src_nr-1; i >= 0; i--)\n-\t\tinsert_file_table(options->repo,\n+\t\tinsert_file_table(options->repo, pool,\n \t\t\t\t &file_table, i,\n \t\t\t\t rename_src[i].p->one);\n \n@@ -355,7 +357,7 @@ static int find_exact_renames(struct diff_options *options)\n \t\trenames += find_identical_files(&file_table, i, options);\n \n \t/* Free the hash data structure and entries */\n-\thashmap_clear_and_free(&file_table, struct file_similarity, entry);\n+\thashmap_clear(&file_table);\n \n \treturn renames;\n }\n@@ -1341,6 +1343,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \tint num_destinations, dst_cnt;\n \tint num_sources, want_copies;\n \tstruct progress *progress = NULL;\n+\tstruct mem_pool local_pool;\n \tstruct dir_rename_info info;\n \tstruct diff_populate_filespec_options dpf_options = {\n \t\t.check_binary = 0,\n@@ -1409,11 +1412,18 @@ void diffcore_rename_extended(struct diff_options *options,\n \t\tgoto cleanup; /* nothing to do */\n \n \ttrace2_region_enter(\"diff\", \"exact renames\", options->repo);\n+\tmem_pool_init(&local_pool, 32*1024);\n \t/*\n \t * We really want to cull the candidates list early\n \t * with cheap tests in order to avoid doing deltas.\n \t */\n-\trename_count = find_exact_renames(options);\n+\trename_count = find_exact_renames(options, &local_pool);\n+\t/*\n+\t * Discard local_pool immediately instead of at \"cleanup:\" in order\n+\t * to reduce maximum memory usage; inexact rename detection uses up\n+\t * a fair amount of memory, and mem_pools can too.\n+\t */\n+\tmem_pool_discard(&local_pool, 0);\n \ttrace2_region_leave(\"diff\", \"exact renames\", options->repo);\n \n \t/* Did we only want exact renames? */\n-- \ngitgitgadget\n\n"},{"id":"431032","messageId":"77367a69daad042d2a6ed56b922ade892337cf16.1627044897.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.git.1627044897.gitgitgadget@gmail.com","subject":"[PATCH 2/7] merge-ort: set up a memory pool","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-23T12:54:52Z","receivedAt":"2021-07-23T12:55:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nmerge-ort has a lot of data structures, and they all tend to be freed\ntogether in clear_or_reinit_internal_opts(). Set up a memory pool to\nallow us to make these allocations and deallocations faster. Future\ncommits will adjust various callers to make use of this memory pool.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex e361443087a..cb33c76760f 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -37,6 +37,8 @@\n #include \"unpack-trees.h\"\n #include \"xdiff-interface.h\"\n \n+#define USE_MEMORY_POOL 1 /* faster, but obscures memory leak hunting */\n+\n /*\n * We have many arrays of size 3. Whenever we have such an array, the\n * indices refer to one of the sides of the three-way merge. This is so\n@@ -339,6 +341,17 @@ struct merge_options_internal {\n \t */\n \tstruct strmap conflicted;\n \n+\t/*\n+\t * pool: memory pool for fast allocation/deallocation\n+\t *\n+\t * We allocate room for lots of filenames and auxiliary data\n+\t * structures in merge_options_internal, and it tends to all be\n+\t * freed together too. Using a memory pool for these provides a\n+\t * nice speedup.\n+\t */\n+\tstruct mem_pool internal_pool;\n+\tstruct mem_pool *pool; /* NULL, or pointer to internal_pool */\n+\n \t/*\n \t * paths_to_free: additional list of strings to free\n \t *\n@@ -603,6 +616,12 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\tstrmap_clear(&opti->output, 0);\n \t}\n \n+#if USE_MEMORY_POOL\n+\tmem_pool_discard(&opti->internal_pool, 0);\n+\tif (!reinitialize)\n+\t\topti->pool = NULL;\n+#endif\n+\n \t/* Clean out callback_data as well. */\n \tFREE_AND_NULL(renames->callback_data);\n \trenames->callback_data_nr = renames->callback_data_alloc = 0;\n@@ -4344,6 +4363,12 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \n \t/* Initialization of various renames fields */\n \trenames = &opt->priv->renames;\n+#if USE_MEMORY_POOL\n+\tmem_pool_init(&opt->priv->internal_pool, 0);\n+\topt->priv->pool = &opt->priv->internal_pool;\n+#else\n+\topt->priv->pool = NULL;\n+#endif\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->dirs_removed[i],\n \t\t\t\t\t NOT_RELEVANT, NULL, 0);\n-- \ngitgitgadget\n\n"},{"id":"431033","messageId":"e30b8c8fea110b2475d00a4b37eb62a4883999c4.1627044897.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.git.1627044897.gitgitgadget@gmail.com","subject":"[PATCH 3/7] merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-23T12:54:53Z","receivedAt":"2021-07-23T12:55:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe need functions which will either call\n xmalloc, xcalloc, xstrndup\nor\n mem_pool_alloc, mem_pool_calloc, mem_pool_strndup\ndepending on whether we have a non-NULL memory pool. Add these\nfunctions; the next commit will make use of these.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 24 ++++++++++++++++++++++++\n 1 file changed, 24 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex cb33c76760f..2bca4b71f2a 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -683,6 +683,30 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n+MAYBE_UNUSED\n+static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n+{\n+\tif (!pool)\n+\t\treturn xcalloc(count, size);\n+\treturn mem_pool_calloc(pool, count, size);\n+}\n+\n+MAYBE_UNUSED\n+static void *pool_alloc(struct mem_pool *pool, size_t size)\n+{\n+\tif (!pool)\n+\t\treturn xmalloc(size);\n+\treturn mem_pool_alloc(pool, size);\n+}\n+\n+MAYBE_UNUSED\n+static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len)\n+{\n+\tif (!pool)\n+\t\treturn xstrndup(str, len);\n+\treturn mem_pool_strndup(pool, str, len);\n+}\n+\n /* add a string to a strbuf, but converting \"/\" to \"_\" */\n static void add_flattened_path(struct strbuf *out, const char *s)\n {\n-- \ngitgitgadget\n\n"},{"id":"431034","messageId":"2db932bc601aa362ee2006817b9e4cdbb290421b.1627044897.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.git.1627044897.gitgitgadget@gmail.com","subject":"[PATCH 5/7] diffcore-rename, merge-ort: add wrapper functions for filepair alloc/dealloc","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-23T12:54:55Z","receivedAt":"2021-07-23T12:55:08Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe want to be able to allocate filespecs and filepairs using a mem_pool.\nHowever, filespec data will still remain outside the pool (perhaps in\nthe future we could plumb the pool through the various diff APIs to\nallocate the filespec data too, but for now we are limiting the scope).\nAdd some extra functions to allocate these appropriately based on the\nnon-NULL-ness of opt->priv->pool, as well as some extra functions to\nhandle correctly deallocating the relevant parts of them. A future\ncommit will make use of these new functions.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 41 +++++++++++++++++++++++++++++++++++++++++\n diffcore.h | 2 ++\n merge-ort.c | 42 ++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 85 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 23b917eca42..09606501cea 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -1328,6 +1328,47 @@ static void handle_early_known_dir_renames(struct dir_rename_info *info,\n \trename_src_nr = new_num_src;\n }\n \n+static void free_filespec_data(struct diff_filespec *spec)\n+{\n+\tif (!--spec->count)\n+\t\tdiff_free_filespec_data(spec);\n+}\n+\n+MAYBE_UNUSED\n+static void pool_free_filespec(struct mem_pool *pool,\n+\t\t\t struct diff_filespec *spec)\n+{\n+\tif (!pool) {\n+\t\tfree_filespec(spec);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Similar to free_filespec(), but only frees the data. The spec\n+\t * itself was allocated in the pool and should not be individually\n+\t * freed.\n+\t */\n+\tfree_filespec_data(spec);\n+}\n+\n+MAYBE_UNUSED\n+void pool_diff_free_filepair(struct mem_pool *pool,\n+\t\t\t struct diff_filepair *p)\n+{\n+\tif (!pool) {\n+\t\tdiff_free_filepair(p);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Similar to diff_free_filepair() but only frees the data from the\n+\t * filespecs; not the filespecs or the filepair which were\n+\t * allocated from the pool.\n+\t */\n+\tfree_filespec_data(p->one);\n+\tfree_filespec_data(p->two);\n+}\n+\n void diffcore_rename_extended(struct diff_options *options,\n \t\t\t struct strintmap *relevant_sources,\n \t\t\t struct strintmap *dirs_removed,\ndiff --git a/diffcore.h b/diffcore.h\nindex 533b30e21e7..b58ee6b1934 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -127,6 +127,8 @@ struct diff_filepair {\n #define DIFF_PAIR_MODE_CHANGED(p) ((p)->one->mode != (p)->two->mode)\n \n void diff_free_filepair(struct diff_filepair *);\n+void pool_diff_free_filepair(struct mem_pool *pool,\n+\t\t\t struct diff_filepair *p);\n \n int diff_unmodified_pair(struct diff_filepair *);\n \ndiff --git a/merge-ort.c b/merge-ort.c\nindex 5fd2a4ccd35..59428e45884 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -690,6 +690,48 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n+MAYBE_UNUSED\n+static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n+\t\t\t\t\t\t const char *path)\n+{\n+\tstruct diff_filespec *spec;\n+\tsize_t len;\n+\n+\tif (!pool)\n+\t\treturn alloc_filespec(path);\n+\n+\t/* Same code as alloc_filespec, except allocate from pool */\n+\tlen = strlen(path);\n+\n+\tspec = mem_pool_calloc(pool, 1, st_add3(sizeof(*spec), len, 1));\n+\tmemcpy(spec+1, path, len);\n+\tspec->path = (void*)(spec+1);\n+\n+\tspec->count = 1;\n+\tspec->is_binary = -1;\n+\treturn spec;\n+}\n+\n+MAYBE_UNUSED\n+static struct diff_filepair *pool_diff_queue(struct mem_pool *pool,\n+\t\t\t\t\t struct diff_queue_struct *queue,\n+\t\t\t\t\t struct diff_filespec *one,\n+\t\t\t\t\t struct diff_filespec *two)\n+{\n+\tstruct diff_filepair *dp;\n+\n+\tif (!pool)\n+\t\treturn diff_queue(queue, one, two);\n+\n+\t/* Same code as diff_queue, except allocate from pool */\n+\tdp = mem_pool_calloc(pool, 1, sizeof(*dp));\n+\tdp->one = one;\n+\tdp->two = two;\n+\tif (queue)\n+\t\tdiff_q(queue, dp);\n+\treturn dp;\n+}\n+\n static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n {\n \tif (!pool)\n-- \ngitgitgadget\n\n"},{"id":"431035","messageId":"8231c8e34cd35cc648f87d31e48b0692dd2fe6cd.1627044897.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.git.1627044897.gitgitgadget@gmail.com","subject":"[PATCH 4/7] merge-ort: switch our strmaps over to using memory pools","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-23T12:54:54Z","receivedAt":"2021-07-23T12:55:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nFor all the strmaps (including strintmaps and strsets) whose memory is\nunconditionally freed as part of clear_or_reinit_internal_opts(), switch\nthem over to using our new memory pool.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 202.5 ms ± 3.2 ms 198.1 ms ± 2.6 ms\n mega-renames: 1.072 s ± 0.012 s 715.8 ms ± 4.0 ms\n just-one-mega: 357.3 ms ± 3.9 ms 276.8 ms ± 4.2 ms\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 125 +++++++++++++++++++++++++++++++---------------------\n 1 file changed, 75 insertions(+), 50 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 2bca4b71f2a..5fd2a4ccd35 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -539,15 +539,19 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \tvoid (*strset_func)(struct strset *) =\n \t\treinitialize ? strset_partial_clear : strset_clear;\n \n-\t/*\n-\t * We marked opti->paths with strdup_strings = 0, so that we\n-\t * wouldn't have to make another copy of the fullpath created by\n-\t * make_traverse_path from setup_path_info(). But, now that we've\n-\t * used it and have no other references to these strings, it is time\n-\t * to deallocate them.\n-\t */\n-\tfree_strmap_strings(&opti->paths);\n-\tstrmap_func(&opti->paths, 1);\n+\tif (opti->pool)\n+\t\tstrmap_func(&opti->paths, 0);\n+\telse {\n+\t\t/*\n+\t\t * We marked opti->paths with strdup_strings = 0, so that\n+\t\t * we wouldn't have to make another copy of the fullpath\n+\t\t * created by make_traverse_path from setup_path_info().\n+\t\t * But, now that we've used it and have no other references\n+\t\t * to these strings, it is time to deallocate them.\n+\t\t */\n+\t\tfree_strmap_strings(&opti->paths);\n+\t\tstrmap_func(&opti->paths, 1);\n+\t}\n \n \t/*\n \t * All keys and values in opti->conflicted are a subset of those in\n@@ -556,16 +560,19 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t */\n \tstrmap_func(&opti->conflicted, 0);\n \n-\t/*\n-\t * opti->paths_to_free is similar to opti->paths; we created it with\n-\t * strdup_strings = 0 to avoid making _another_ copy of the fullpath\n-\t * but now that we've used it and have no other references to these\n-\t * strings, it is time to deallocate them. We do so by temporarily\n-\t * setting strdup_strings to 1.\n-\t */\n-\topti->paths_to_free.strdup_strings = 1;\n-\tstring_list_clear(&opti->paths_to_free, 0);\n-\topti->paths_to_free.strdup_strings = 0;\n+\tif (!opti->pool) {\n+\t\t/*\n+\t\t * opti->paths_to_free is similar to opti->paths; we\n+\t\t * created it with strdup_strings = 0 to avoid making\n+\t\t * _another_ copy of the fullpath but now that we've used\n+\t\t * it and have no other references to these strings, it is\n+\t\t * time to deallocate them. We do so by temporarily\n+\t\t * setting strdup_strings to 1.\n+\t\t */\n+\t\topti->paths_to_free.strdup_strings = 1;\n+\t\tstring_list_clear(&opti->paths_to_free, 0);\n+\t\topti->paths_to_free.strdup_strings = 0;\n+\t}\n \n \tif (opti->attr_index.cache_nr) /* true iff opt->renormalize */\n \t\tdiscard_index(&opti->attr_index);\n@@ -683,7 +690,6 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n-MAYBE_UNUSED\n static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n {\n \tif (!pool)\n@@ -691,7 +697,6 @@ static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n \treturn mem_pool_calloc(pool, count, size);\n }\n \n-MAYBE_UNUSED\n static void *pool_alloc(struct mem_pool *pool, size_t size)\n {\n \tif (!pool)\n@@ -699,7 +704,6 @@ static void *pool_alloc(struct mem_pool *pool, size_t size)\n \treturn mem_pool_alloc(pool, size);\n }\n \n-MAYBE_UNUSED\n static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len)\n {\n \tif (!pool)\n@@ -835,8 +839,9 @@ static void setup_path_info(struct merge_options *opt,\n \tassert(!df_conflict || !resolved); /* df_conflict implies !resolved */\n \tassert(resolved == (merged_version != NULL));\n \n-\tmi = xcalloc(1, resolved ? sizeof(struct merged_info) :\n-\t\t\t\t sizeof(struct conflict_info));\n+\tmi = pool_calloc(opt->priv->pool, 1,\n+\t\t\t resolved ? sizeof(struct merged_info) :\n+\t\t\t\t sizeof(struct conflict_info));\n \tmi->directory_name = current_dir_name;\n \tmi->basename_offset = current_dir_name_len;\n \tmi->clean = !!resolved;\n@@ -1128,7 +1133,7 @@ static int collect_merge_info_callback(int n,\n \tlen = traverse_path_len(info, p->pathlen);\n \n \t/* +1 in both of the following lines to include the NUL byte */\n-\tfullpath = xmalloc(len + 1);\n+\tfullpath = pool_alloc(opt->priv->pool, len + 1);\n \tmake_traverse_path(fullpath, len + 1, info, p->path, p->pathlen);\n \n \t/*\n@@ -1383,7 +1388,7 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\tcopy = renames->deferred[side].possible_trivial_merges;\n \t\tstrintmap_init_with_options(&renames->deferred[side].possible_trivial_merges,\n \t\t\t\t\t 0,\n-\t\t\t\t\t NULL,\n+\t\t\t\t\t opt->priv->pool,\n \t\t\t\t\t 0);\n \t\tstrintmap_for_each_entry(©, &iter, entry) {\n \t\t\tconst char *path = entry->key;\n@@ -2335,12 +2340,21 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \tVERIFY_CI(ci);\n \n \t/* Find parent directories missing from opt->priv->paths */\n-\tcur_path = new_path;\n+\tif (opt->priv->pool) {\n+\t\tcur_path = mem_pool_strdup(opt->priv->pool, new_path);\n+\t\tfree((char*)new_path);\n+\t\tnew_path = (char *)cur_path;\n+\t} else {\n+\t\tcur_path = new_path;\n+\t}\n+\n \twhile (1) {\n \t\t/* Find the parent directory of cur_path */\n \t\tchar *last_slash = strrchr(cur_path, '/');\n \t\tif (last_slash) {\n-\t\t\tparent_name = xstrndup(cur_path, last_slash - cur_path);\n+\t\t\tparent_name = pool_strndup(opt->priv->pool,\n+\t\t\t\t\t\t cur_path,\n+\t\t\t\t\t\t last_slash - cur_path);\n \t\t} else {\n \t\t\tparent_name = opt->priv->toplevel_dir;\n \t\t\tbreak;\n@@ -2349,7 +2363,8 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\t/* Look it up in opt->priv->paths */\n \t\tentry = strmap_get_entry(&opt->priv->paths, parent_name);\n \t\tif (entry) {\n-\t\t\tfree((char*)parent_name);\n+\t\t\tif (!opt->priv->pool)\n+\t\t\t\tfree((char*)parent_name);\n \t\t\tparent_name = entry->key; /* reuse known pointer */\n \t\t\tbreak;\n \t\t}\n@@ -2376,12 +2391,15 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\tparent_name = cur_dir;\n \t}\n \n-\t/*\n-\t * We are removing old_path from opt->priv->paths. old_path also will\n-\t * eventually need to be freed, but it may still be used by e.g.\n-\t * ci->pathnames. So, store it in another string-list for now.\n-\t */\n-\tstring_list_append(&opt->priv->paths_to_free, old_path);\n+\tif (!opt->priv->pool) {\n+\t\t/*\n+\t\t * We are removing old_path from opt->priv->paths.\n+\t\t * old_path also will eventually need to be freed, but it\n+\t\t * may still be used by e.g. ci->pathnames. So, store it\n+\t\t * in another string-list for now.\n+\t\t */\n+\t\tstring_list_append(&opt->priv->paths_to_free, old_path);\n+\t}\n \n \tassert(ci->filemask == 2 || ci->filemask == 4);\n \tassert(ci->dirmask == 0);\n@@ -2416,7 +2434,8 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\tnew_ci->stages[index].mode = ci->stages[index].mode;\n \t\toidcpy(&new_ci->stages[index].oid, &ci->stages[index].oid);\n \n-\t\tfree(ci);\n+\t\tif (!opt->priv->pool)\n+\t\t\tfree(ci);\n \t\tci = new_ci;\n \t}\n \n@@ -3623,7 +3642,8 @@ static void process_entry(struct merge_options *opt,\n \t\t * the directory to remain here, so we need to move this\n \t\t * path to some new location.\n \t\t */\n-\t\tCALLOC_ARRAY(new_ci, 1);\n+\t\tnew_ci = pool_calloc(opt->priv->pool, 1, sizeof(*new_ci));\n+\n \t\t/* We don't really want new_ci->merged.result copied, but it'll\n \t\t * be overwritten below so it doesn't matter. We also don't\n \t\t * want any directory mode/oid values copied, but we'll zero\n@@ -3713,7 +3733,7 @@ static void process_entry(struct merge_options *opt,\n \t\t\tconst char *a_path = NULL, *b_path = NULL;\n \t\t\tint rename_a = 0, rename_b = 0;\n \n-\t\t\tnew_ci = xmalloc(sizeof(*new_ci));\n+\t\t\tnew_ci = pool_alloc(opt->priv->pool, sizeof(*new_ci));\n \n \t\t\tif (S_ISREG(a_mode))\n \t\t\t\trename_a = 1;\n@@ -3777,12 +3797,14 @@ static void process_entry(struct merge_options *opt,\n \t\t\t\tstrmap_remove(&opt->priv->paths, path, 0);\n \t\t\t\t/*\n \t\t\t\t * We removed path from opt->priv->paths. path\n-\t\t\t\t * will also eventually need to be freed, but\n-\t\t\t\t * it may still be used by e.g. ci->pathnames.\n-\t\t\t\t * So, store it in another string-list for now.\n+\t\t\t\t * will also eventually need to be freed if not\n+\t\t\t\t * part of a memory pool...but it may still be\n+\t\t\t\t * used by e.g. ci->pathnames. So, store it in\n+\t\t\t\t * another string-list for now in that case.\n \t\t\t\t */\n-\t\t\t\tstring_list_append(&opt->priv->paths_to_free,\n-\t\t\t\t\t\t path);\n+\t\t\t\tif (!opt->priv->pool)\n+\t\t\t\t\tstring_list_append(&opt->priv->paths_to_free,\n+\t\t\t\t\t\t\t path);\n \t\t\t}\n \n \t\t\t/*\n@@ -4322,6 +4344,7 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n {\n \tstruct rename_info *renames;\n \tint i;\n+\tstruct mem_pool *pool = NULL;\n \n \t/* Sanity checks on opt */\n \ttrace2_region_enter(\"merge\", \"sanity checks\", opt->repo);\n@@ -4393,9 +4416,10 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n #else\n \topt->priv->pool = NULL;\n #endif\n+\tpool = opt->priv->pool;\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->dirs_removed[i],\n-\t\t\t\t\t NOT_RELEVANT, NULL, 0);\n+\t\t\t\t\t NOT_RELEVANT, pool, 0);\n \t\tstrmap_init_with_options(&renames->dir_rename_count[i],\n \t\t\t\t\t NULL, 1);\n \t\tstrmap_init_with_options(&renames->dir_renames[i],\n@@ -4409,7 +4433,7 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t\t */\n \t\tstrintmap_init_with_options(&renames->relevant_sources[i],\n \t\t\t\t\t -1 /* explicitly invalid */,\n-\t\t\t\t\t NULL, 0);\n+\t\t\t\t\t pool, 0);\n \t\tstrmap_init_with_options(&renames->cached_pairs[i],\n \t\t\t\t\t NULL, 1);\n \t\tstrset_init_with_options(&renames->cached_irrelevant[i],\n@@ -4419,9 +4443,9 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t}\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->deferred[i].possible_trivial_merges,\n-\t\t\t\t\t 0, NULL, 0);\n+\t\t\t\t\t 0, pool, 0);\n \t\tstrset_init_with_options(&renames->deferred[i].target_dirs,\n-\t\t\t\t\t NULL, 1);\n+\t\t\t\t\t pool, 1);\n \t\trenames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */\n \t}\n \n@@ -4434,9 +4458,10 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t * In contrast, conflicted just has a subset of keys from paths, so\n \t * we don't want to free those (it'd be a duplicate free).\n \t */\n-\tstrmap_init_with_options(&opt->priv->paths, NULL, 0);\n-\tstrmap_init_with_options(&opt->priv->conflicted, NULL, 0);\n-\tstring_list_init(&opt->priv->paths_to_free, 0);\n+\tstrmap_init_with_options(&opt->priv->paths, pool, 0);\n+\tstrmap_init_with_options(&opt->priv->conflicted, pool, 0);\n+\tif (!opt->priv->pool)\n+\t\tstring_list_init(&opt->priv->paths_to_free, 0);\n \n \t/*\n \t * keys & strbufs in output will sometimes need to outlive \"paths\",\n-- \ngitgitgadget\n\n"},{"id":"431036","messageId":"629d042884a6a846faca411b01b1b15ace99b540.1627044898.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.git.1627044897.gitgitgadget@gmail.com","subject":"[PATCH 6/7] merge-ort: store filepairs and filespecs in our mem_pool","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-23T12:54:56Z","receivedAt":"2021-07-23T12:55:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 198.1 ms ± 2.6 ms 198.5 ms ± 3.4 ms\n mega-renames: 715.8 ms ± 4.0 ms 679.1 ms ± 5.6 ms\n just-one-mega: 276.8 ms ± 4.2 ms 271.9 ms ± 2.8 ms\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 9 ++++-----\n diffcore.h | 1 +\n merge-ort.c | 26 ++++++++++++++------------\n 3 files changed, 19 insertions(+), 17 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 09606501cea..e30e4288d1b 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -1334,7 +1334,6 @@ static void free_filespec_data(struct diff_filespec *spec)\n \t\tdiff_free_filespec_data(spec);\n }\n \n-MAYBE_UNUSED\n static void pool_free_filespec(struct mem_pool *pool,\n \t\t\t struct diff_filespec *spec)\n {\n@@ -1351,7 +1350,6 @@ static void pool_free_filespec(struct mem_pool *pool,\n \tfree_filespec_data(spec);\n }\n \n-MAYBE_UNUSED\n void pool_diff_free_filepair(struct mem_pool *pool,\n \t\t\t struct diff_filepair *p)\n {\n@@ -1370,6 +1368,7 @@ void pool_diff_free_filepair(struct mem_pool *pool,\n }\n \n void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct mem_pool *pool,\n \t\t\t struct strintmap *relevant_sources,\n \t\t\t struct strintmap *dirs_removed,\n \t\t\t struct strmap *dir_rename_count,\n@@ -1683,7 +1682,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \t\t\tpair_to_free = p;\n \n \t\tif (pair_to_free)\n-\t\t\tdiff_free_filepair(pair_to_free);\n+\t\t\tpool_diff_free_filepair(pool, pair_to_free);\n \t}\n \tdiff_debug_queue(\"done copying original\", &outq);\n \n@@ -1693,7 +1692,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \n \tfor (i = 0; i < rename_dst_nr; i++)\n \t\tif (rename_dst[i].filespec_to_free)\n-\t\t\tfree_filespec(rename_dst[i].filespec_to_free);\n+\t\t\tpool_free_filespec(pool, rename_dst[i].filespec_to_free);\n \n \tcleanup_dir_rename_info(&info, dirs_removed, dir_rename_count != NULL);\n \tFREE_AND_NULL(rename_dst);\n@@ -1710,5 +1709,5 @@ void diffcore_rename_extended(struct diff_options *options,\n \n void diffcore_rename(struct diff_options *options)\n {\n-\tdiffcore_rename_extended(options, NULL, NULL, NULL, NULL);\n+\tdiffcore_rename_extended(options, NULL, NULL, NULL, NULL, NULL);\n }\ndiff --git a/diffcore.h b/diffcore.h\nindex b58ee6b1934..badc2261c20 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -181,6 +181,7 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count);\n void diffcore_break(struct repository *, int);\n void diffcore_rename(struct diff_options *);\n void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct mem_pool *pool,\n \t\t\t struct strintmap *relevant_sources,\n \t\t\t struct strintmap *dirs_removed,\n \t\t\t struct strmap *dir_rename_count,\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 59428e45884..d29c7fe8a30 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -690,7 +690,6 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n-MAYBE_UNUSED\n static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n \t\t\t\t\t\t const char *path)\n {\n@@ -712,7 +711,6 @@ static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n \treturn spec;\n }\n \n-MAYBE_UNUSED\n static struct diff_filepair *pool_diff_queue(struct mem_pool *pool,\n \t\t\t\t\t struct diff_queue_struct *queue,\n \t\t\t\t\t struct diff_filespec *one,\n@@ -930,6 +928,7 @@ static void add_pair(struct merge_options *opt,\n \t\t unsigned dir_rename_mask)\n {\n \tstruct diff_filespec *one, *two;\n+\tstruct mem_pool *pool = opt->priv->pool;\n \tstruct rename_info *renames = &opt->priv->renames;\n \tint names_idx = is_add ? side : 0;\n \n@@ -980,11 +979,11 @@ static void add_pair(struct merge_options *opt,\n \t\t\treturn;\n \t}\n \n-\tone = alloc_filespec(pathname);\n-\ttwo = alloc_filespec(pathname);\n+\tone = pool_alloc_filespec(pool, pathname);\n+\ttwo = pool_alloc_filespec(pool, pathname);\n \tfill_filespec(is_add ? two : one,\n \t\t &names[names_idx].oid, 1, names[names_idx].mode);\n-\tdiff_queue(&renames->pairs[side], one, two);\n+\tpool_diff_queue(pool, &renames->pairs[side], one, two);\n }\n \n static void collect_rename_info(struct merge_options *opt,\n@@ -2893,6 +2892,7 @@ static void use_cached_pairs(struct merge_options *opt,\n {\n \tstruct hashmap_iter iter;\n \tstruct strmap_entry *entry;\n+\tstruct mem_pool *pool = opt->priv->pool;\n \n \t/*\n \t * Add to side_pairs all entries from renames->cached_pairs[side_index].\n@@ -2906,9 +2906,9 @@ static void use_cached_pairs(struct merge_options *opt,\n \t\t\tnew_name = old_name;\n \n \t\t/* We don't care about oid/mode, only filenames and status */\n-\t\tone = alloc_filespec(old_name);\n-\t\ttwo = alloc_filespec(new_name);\n-\t\tdiff_queue(pairs, one, two);\n+\t\tone = pool_alloc_filespec(pool, old_name);\n+\t\ttwo = pool_alloc_filespec(pool, new_name);\n+\t\tpool_diff_queue(pool, pairs, one, two);\n \t\tpairs->queue[pairs->nr-1]->status = entry->value ? 'R' : 'D';\n \t}\n }\n@@ -3016,6 +3016,7 @@ static int detect_regular_renames(struct merge_options *opt,\n \tdiff_queued_diff = renames->pairs[side_index];\n \ttrace2_region_enter(\"diff\", \"diffcore_rename\", opt->repo);\n \tdiffcore_rename_extended(&diff_opts,\n+\t\t\t\t opt->priv->pool,\n \t\t\t\t &renames->relevant_sources[side_index],\n \t\t\t\t &renames->dirs_removed[side_index],\n \t\t\t\t &renames->dir_rename_count[side_index],\n@@ -3066,7 +3067,7 @@ static int collect_renames(struct merge_options *opt,\n \n \t\tif (p->status != 'A' && p->status != 'R') {\n \t\t\tpossibly_cache_new_pair(renames, p, side_index, NULL);\n-\t\t\tdiff_free_filepair(p);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n \t\t\tcontinue;\n \t\t}\n \n@@ -3079,7 +3080,7 @@ static int collect_renames(struct merge_options *opt,\n \n \t\tpossibly_cache_new_pair(renames, p, side_index, new_path);\n \t\tif (p->status != 'R' && !new_path) {\n-\t\t\tdiff_free_filepair(p);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n \t\t\tcontinue;\n \t\t}\n \n@@ -3197,7 +3198,7 @@ cleanup:\n \t\tside_pairs = &renames->pairs[s];\n \t\tfor (i = 0; i < side_pairs->nr; ++i) {\n \t\t\tstruct diff_filepair *p = side_pairs->queue[i];\n-\t\t\tdiff_free_filepair(p);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n \t\t}\n \t}\n \n@@ -3210,7 +3211,8 @@ simple_cleanup:\n \tif (combined.nr) {\n \t\tint i;\n \t\tfor (i = 0; i < combined.nr; i++)\n-\t\t\tdiff_free_filepair(combined.queue[i]);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool,\n+\t\t\t\t\t\tcombined.queue[i]);\n \t\tfree(combined.queue);\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"431037","messageId":"17aa0a748494e72df1e0e2303ed650917e59a101.1627044898.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.git.1627044897.gitgitgadget@gmail.com","subject":"[PATCH 7/7] merge-ort: reuse path strings in pool_alloc_filespec","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-23T12:54:57Z","receivedAt":"2021-07-23T12:55:12Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\npool_alloc_filespec() was written so that the code when pool != NULL\nmimicked the code from alloc_filespec(), which including allocating\nenough extra space for the path and then copying it. However, the path\npassed to pool_alloc_filespec() is always going to already be in the\nsame memory pool, so we may as well reuse it instead of copying it.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 198.5 ms ± 3.4 ms 198.3 ms ± 2.9 ms\n mega-renames: 679.1 ms ± 5.6 ms 661.8 ms ± 5.9 ms\n just-one-mega: 271.9 ms ± 2.8 ms 264.6 ms ± 2.5 ms\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 29 ++++++++++++++++++++++-------\n 1 file changed, 22 insertions(+), 7 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex d29c7fe8a30..0fb942692a7 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -694,17 +694,13 @@ static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n \t\t\t\t\t\t const char *path)\n {\n \tstruct diff_filespec *spec;\n-\tsize_t len;\n \n \tif (!pool)\n \t\treturn alloc_filespec(path);\n \n-\t/* Same code as alloc_filespec, except allocate from pool */\n-\tlen = strlen(path);\n-\n-\tspec = mem_pool_calloc(pool, 1, st_add3(sizeof(*spec), len, 1));\n-\tmemcpy(spec+1, path, len);\n-\tspec->path = (void*)(spec+1);\n+\t/* Similar to alloc_filespec, but allocate from pool and reuse path */\n+\tspec = mem_pool_calloc(pool, 1, sizeof(*spec));\n+\tspec->path = (char*)path; /* spec won't modify it */\n \n \tspec->count = 1;\n \tspec->is_binary = -1;\n@@ -2904,6 +2900,25 @@ static void use_cached_pairs(struct merge_options *opt,\n \t\tconst char *new_name = entry->value;\n \t\tif (!new_name)\n \t\t\tnew_name = old_name;\n+\t\tif (pool) {\n+\t\t\t/*\n+\t\t\t * cached_pairs has _copies* of old_name and new_name,\n+\t\t\t * because it has to persist across merges. When\n+\t\t\t * pool != NULL\n+\t\t\t * pool_alloc_filespec() will just re-use the existing\n+\t\t\t * filenames, which will also get re-used by\n+\t\t\t * opt->priv->paths if they become renames, and then\n+\t\t\t * get freed at the end of the merge, leaving the copy\n+\t\t\t * in cached_pairs dangling. Avoid this by making a\n+\t\t\t * copy here.\n+\t\t\t *\n+\t\t\t * When pool == NULL, pool_alloc_filespec() calls\n+\t\t\t * alloc_filespec(), which makes a copy; we don't want\n+\t\t\t * to add another.\n+\t\t\t */\n+\t\t\told_name = mem_pool_strdup(pool, old_name);\n+\t\t\tnew_name = mem_pool_strdup(pool, new_name);\n+\t\t}\n \n \t\t/* We don't care about oid/mode, only filenames and status */\n \t\tone = pool_alloc_filespec(pool, old_name);\n-- \ngitgitgadget\n"},{"id":"431103","messageId":"CAPig+cQK8F2p4JT6FCFxNALxhoj2_opZit-ao=pLNhOHxN+gDw@mail.gmail.com","threadId":"56152","inReplyTo":"9f8ab62b84256be6d7d984d576ff4fda09d88a1d.1627044897.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/7] diffcore-rename: use a mem_pool for exact rename detection's hashmap","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-07-23T21:59:37Z","receivedAt":"2021-07-23T21:59:51Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Jul 23, 2021 at 8:55 AM Elijah Newren via GitGitGadget\n wrote:\n> Exact rename detection, via insert_file_table(), uses a hashmap to store\n> files by oid. Use a mem_pool for the hashmap entries so these can all be\n> allocated and deallocated together.\n> [...]\n> Signed-off-by: Elijah Newren \n> ---\n> diff --git a/diffcore-rename.c b/diffcore-rename.c\n> @@ -355,7 +357,7 @@ static int find_exact_renames(struct diff_options *options)\n> /* Free the hash data structure and entries */\n> - hashmap_clear_and_free(&file_table, struct file_similarity, entry);\n> + hashmap_clear(&file_table);\n\nDoes the in-code comment become a bit out of date with this change?\n(It might make sense to drop the comment altogether -- or, if not,\nexplain that the hashmap entries get thrown away later with the pool?)\n\nNot necessarily worth a re-roll.\n"},{"id":"431106","messageId":"CABPp-BHmY37fNw36E6d8thExA13zWY3sROewoHqT3CBA-Bj2Gg@mail.gmail.com","threadId":"56152","inReplyTo":"CAPig+cQK8F2p4JT6FCFxNALxhoj2_opZit-ao=pLNhOHxN+gDw@mail.gmail.com","subject":"Re: [PATCH 1/7] diffcore-rename: use a mem_pool for exact rename detection's hashmap","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-23T22:03:34Z","receivedAt":"2021-07-23T22:03:47Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Jul 23, 2021 at 2:59 PM Eric Sunshine wrote:\n>\n> On Fri, Jul 23, 2021 at 8:55 AM Elijah Newren via GitGitGadget\n> wrote:\n> > Exact rename detection, via insert_file_table(), uses a hashmap to store\n> > files by oid. Use a mem_pool for the hashmap entries so these can all be\n> > allocated and deallocated together.\n> > [...]\n> > Signed-off-by: Elijah Newren \n> > ---\n> > diff --git a/diffcore-rename.c b/diffcore-rename.c\n> > @@ -355,7 +357,7 @@ static int find_exact_renames(struct diff_options *options)\n> > /* Free the hash data structure and entries */\n> > - hashmap_clear_and_free(&file_table, struct file_similarity, entry);\n> > + hashmap_clear(&file_table);\n>\n> Does the in-code comment become a bit out of date with this change?\n> (It might make sense to drop the comment altogether -- or, if not,\n> explain that the hashmap entries get thrown away later with the pool?)\n\nAh, good catch. Yeah, I should update it or drop the comment.\n\n> Not necessarily worth a re-roll.\n\nI'm sure someone will comment on something else, so I'll just include\nthis among the fixes.\n"},{"id":"431107","messageId":"CAPig+cQD28hPV30B2Oih=LUDmJGWKTFXmZGE65YYhqAeH0FVvQ@mail.gmail.com","threadId":"56152","inReplyTo":"e30b8c8fea110b2475d00a4b37eb62a4883999c4.1627044897.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/7] merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-07-23T22:07:41Z","receivedAt":"2021-07-23T22:07:55Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Jul 23, 2021 at 8:55 AM Elijah Newren via GitGitGadget\n wrote:\n> We need functions which will either call\n> xmalloc, xcalloc, xstrndup\n> or\n> mem_pool_alloc, mem_pool_calloc, mem_pool_strndup\n> depending on whether we have a non-NULL memory pool. Add these\n> functions; the next commit will make use of these.\n>\n> Signed-off-by: Elijah Newren \n\nPatch [2/7] feels somewhat incomplete without the utility functions\nintroduced by this patch (and, indeed, when reading [2/7], I was\nwondering how you were going to deal with the potential NULL pointer).\nFrom a review standpoint, I could easily see [2/7] and [3/7] presented\nas a single patch, but it's not worth a re-roll.\n\n> diff --git a/merge-ort.c b/merge-ort.c\n> @@ -683,6 +683,30 @@ static void path_msg(struct merge_options *opt,\n> +MAYBE_UNUSED\n> +static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n> +{\n> + if (!pool)\n> + return xcalloc(count, size);\n> + return mem_pool_calloc(pool, count, size);\n> +}\n> +\n> +MAYBE_UNUSED\n> +static void *pool_alloc(struct mem_pool *pool, size_t size)\n> +{\n> + if (!pool)\n> + return xmalloc(size);\n> + return mem_pool_alloc(pool, size);\n> +}\n> +\n> +MAYBE_UNUSED\n> +static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len)\n> +{\n> + if (!pool)\n> + return xstrndup(str, len);\n> + return mem_pool_strndup(pool, str, len);\n> +}\n> +\n"},{"id":"431190","messageId":"bc24ea9c-178c-bcc9-2097-2271df93ae76@gmail.com","threadId":"56152","inReplyTo":"e30b8c8fea110b2475d00a4b37eb62a4883999c4.1627044897.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/7] merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-26T14:36:39Z","receivedAt":"2021-07-26T14:36:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/23/2021 8:54 AM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> We need functions which will either call\n> xmalloc, xcalloc, xstrndup\n> or\n> mem_pool_alloc, mem_pool_calloc, mem_pool_strndup\n> depending on whether we have a non-NULL memory pool. Add these\n> functions; the next commit will make use of these.\n\nI briefly considered that this should just be the way the\nmem_pool_* methods work. It does rely on the caller knowing\nto free() the allocated memory when their pool is NULL, so\nperhaps such a universal change might be too much. What do\nyou think?\n\nThanks,\n-Stolee\n"},{"id":"431191","messageId":"884002ab-c59d-143d-0cf0-8d06d5fb5f0f@gmail.com","threadId":"56152","inReplyTo":"pull.990.git.1627044897.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/7] Final optimization batch (#15): use memory pools","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-26T14:44:23Z","receivedAt":"2021-07-26T14:44:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/23/2021 8:54 AM, Elijah Newren via GitGitGadget wrote:\n...\n> === Basic Optimization idea ===\n> \n> In this series, I make use of memory pools to get faster allocations and\n> deallocations for many data structures that tend to all be deallocated at\n> the same time anyway.\n\nMakes sense. This is appropriate for a final optimization, since the gains\ntend to be quite small.\n\n> === Results ===\n> \n> For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n> performance work; instrument with trace2_region_* calls\", 2020-10-28), the\n> changes in just this series improves the performance as follows:\n> \n> Before Series After Series\n> no-renames: 204.2 ms ± 3.0 ms 198.3 ms ± 2.9 ms\n> mega-renames: 1.076 s ± 0.015 s 661.8 ms ± 5.9 ms\n> just-one-mega: 364.1 ms ± 7.0 ms 264.6 ms ± 2.5 ms\n\n\nBut these are larger than I anticipated! Amazing.\n\n> === Overall Results across all optimization work ===\n\nI enjoyed reading this section. I'm excited to make ORT the default in\nthe microsoft/git fork and see how this improves the lives of our users.\n\n> === Caveat ===\n> \n> It may be worth noting, though, that my optimization numbers above for\n> merge-ort use test-tool fast-rebase. git rebase -s ort on the three\n> testcases above is 5-20 times slower (taking 3.835s, 6.798s, and 1.235s,\n> respectively).\n\nThe performance and behavior changes recommended here should definitely\nbe considered. However, the benefits still apply and at the moment users\ndo not expect immediate responses from 'git rebase' so we have some time\nto approach with caution.\n\nI only had one small question that is not even important to the\ncorrectness of this series, so feel free to ignore it. The patches tell\na convincing story.\n\nJust to be careful, have you taken the time to run the merge-ORT tests\nwith --valgrind?\n\nThanks,\n-Stolee\n"},{"id":"431443","messageId":"CABPp-BGjrKWXn2ZzeJ-nsZTwsDE=gbaX=1YKfiNtC70VeN9VVA@mail.gmail.com","threadId":"56152","inReplyTo":"bc24ea9c-178c-bcc9-2097-2271df93ae76@gmail.com","subject":"Re: [PATCH 3/7] merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-28T22:49:18Z","receivedAt":"2021-07-28T22:49:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jul 26, 2021 at 8:36 AM Derrick Stolee wrote:\n>\n> On 7/23/2021 8:54 AM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > We need functions which will either call\n> > xmalloc, xcalloc, xstrndup\n> > or\n> > mem_pool_alloc, mem_pool_calloc, mem_pool_strndup\n> > depending on whether we have a non-NULL memory pool. Add these\n> > functions; the next commit will make use of these.\n>\n> I briefly considered that this should just be the way the\n> mem_pool_* methods work. It does rely on the caller knowing\n> to free() the allocated memory when their pool is NULL, so\n> perhaps such a universal change might be too much. What do\n> you think?\n\nThat's interesting, but I'm worried it might be a bit much. Do others\non the list have an opinion here?\n"},{"id":"431444","messageId":"CABPp-BGF7kMP_KKVQguy6=x4Ex24UNfbKvEDEhd7wHSaDZ3+jg@mail.gmail.com","threadId":"56152","inReplyTo":"884002ab-c59d-143d-0cf0-8d06d5fb5f0f@gmail.com","subject":"Re: [PATCH 0/7] Final optimization batch (#15): use memory pools","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-28T22:52:29Z","receivedAt":"2021-07-28T22:52:42Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jul 26, 2021 at 8:44 AM Derrick Stolee wrote:\n>\n> On 7/23/2021 8:54 AM, Elijah Newren via GitGitGadget wrote:\n> ...\n> > === Basic Optimization idea ===\n> >\n> > In this series, I make use of memory pools to get faster allocations and\n> > deallocations for many data structures that tend to all be deallocated at\n> > the same time anyway.\n>\n> Makes sense. This is appropriate for a final optimization, since the gains\n> tend to be quite small.\n>\n> > === Results ===\n> >\n> > For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n> > performance work; instrument with trace2_region_* calls\", 2020-10-28), the\n> > changes in just this series improves the performance as follows:\n> >\n> > Before Series After Series\n> > no-renames: 204.2 ms ± 3.0 ms 198.3 ms ± 2.9 ms\n> > mega-renames: 1.076 s ± 0.015 s 661.8 ms ± 5.9 ms\n> > just-one-mega: 364.1 ms ± 7.0 ms 264.6 ms ± 2.5 ms\n>\n>\n> But these are larger than I anticipated! Amazing.\n>\n> > === Overall Results across all optimization work ===\n>\n> I enjoyed reading this section. I'm excited to make ORT the default in\n> the microsoft/git fork and see how this improves the lives of our users.\n>\n> > === Caveat ===\n> >\n> > It may be worth noting, though, that my optimization numbers above for\n> > merge-ort use test-tool fast-rebase. git rebase -s ort on the three\n> > testcases above is 5-20 times slower (taking 3.835s, 6.798s, and 1.235s,\n> > respectively).\n>\n> The performance and behavior changes recommended here should definitely\n> be considered. However, the benefits still apply and at the moment users\n> do not expect immediate responses from 'git rebase' so we have some time\n> to approach with caution.\n>\n> I only had one small question that is not even important to the\n> correctness of this series, so feel free to ignore it. The patches tell\n> a convincing story.\n>\n> Just to be careful, have you taken the time to run the merge-ORT tests\n> with --valgrind?\n\nYes. In addition to the testsuite, I also ran the testcases above\nunder valgrind (especially mega-renames) -- and with those testcases I\nhad the leak checker turned on. It was somewhat surprising how much\nslowdown I saw when I introduced some accidental memory leaks while\noptimizing. But it all runs clean with the patches I submitted.\n"},{"id":"431457","messageId":"pull.990.v2.git.1627531121.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.git.1627044897.gitgitgadget@gmail.com","subject":"[PATCH v2 0/7] Final optimization batch (#15): use memory pools","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-29T03:58:34Z","receivedAt":"2021-07-29T03:58:48Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series textually depends on en/ort-perf-batch-14, but the ideas are\northogonal to it and orthogonal to previous series. It can be reviewed\nindependently.\n\nThis series is more about strmaps & memory pools than merge logic. CC'ing\nPeff since he reviewed the strmap work[1], and that work included a number\nof decisions that specifically had this series in mind.\n\nChanges since v1, addressing Eric's feedback:\n\n * Fixed a comment that became out-of-date in patch 1\n * Swapped commits 2 and 3 so that one can better motivate the other.\n\nNote: Stolee also had an interesting question about whether we should tweak\nthe mem_pool_*() API; he and I were both worried it was a bit much, so I've\nleft it out unless others on list chime in with their opinions on that\nchange.\n\n[1]\nhttps://lore.kernel.org/git/20201111200701.GB39046@coredump.intra.peff.net/\n\n=== Basic Optimization idea ===\n\nIn this series, I make use of memory pools to get faster allocations and\ndeallocations for many data structures that tend to all be deallocated at\nthe same time anyway.\n\n=== Results ===\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28), the\nchanges in just this series improves the performance as follows:\n\n Before Series After Series\nno-renames: 204.2 ms ± 3.0 ms 198.3 ms ± 2.9 ms\nmega-renames: 1.076 s ± 0.015 s 661.8 ms ± 5.9 ms\njust-one-mega: 364.1 ms ± 7.0 ms 264.6 ms ± 2.5 ms\n\n\nAs a reminder, before any merge-ort/diffcore-rename performance work, the\nperformance results we started with were:\n\nno-renames-am: 6.940 s ± 0.485 s\nno-renames: 18.912 s ± 0.174 s\nmega-renames: 5964.031 s ± 10.459 s\njust-one-mega: 149.583 s ± 0.751 s\n\n\n=== Overall Results across all optimization work ===\n\nThis is my final prepared optimization series. It might be worth reviewing\nhow my optimizations fared overall, comparing the original merge-recursive\ntimings with three things: how much merge-recursive improved (as a\nside-effect of optimizing merge-ort), how much improvement we would have\ngotten from a hypothetical infinite parallelization of rename detection, and\nwhat I achieved at the end with merge-ort:\n\n Timings\n\n Infinite\n merge- merge- Parallelism\n recursive recursive of rename merge-ort\n v2.30.0 current detection current\n ---------- --------- ----------- ---------\nno-renames: 18.912 s 18.030 s 11.699 s 198.3 ms\nmega-renames: 5964.031 s 361.281 s 203.886 s 661.8 ms\njust-one-mega: 149.583 s 11.009 s 7.553 s 264.6 ms\n\n Speedup factors\n\n Infinite\n merge- merge- Parallelism\n recursive recursive of rename\n v2.30.0 current detection merge-ort\n ---------- --------- ----------- ---------\nno-renames: 1 1.05 1.6 95\nmega-renames: 1 16.5 29 9012\njust-one-mega: 1 13.6 20 565\n\n\nAnd, for partial clone users:\n\n Factor reduction in number of objects needed\n\n Infinite\n merge- merge- Parallelism\n recursive recursive of rename\n v2.30.0 current detection merge-ort\n ---------- --------- ----------- ---------\nmega-renames: 1 1 1 181.3\n\n\n=== Caveat ===\n\nIt may be worth noting, though, that my optimization numbers above for\nmerge-ort use test-tool fast-rebase. git rebase -s ort on the three\ntestcases above is 5-20 times slower (taking 3.835s, 6.798s, and 1.235s,\nrespectively). At this point, any further optimization work should go into\nmaking a faster full-featured rebase by copying the ideas from fast-rebase:\navoid unnecessary process forking, avoid updating the index and working copy\nuntil either the rebase is finished or you hit a conflict (and don't write\nrebase metadata to disk until that point either), get rid of the glacially\nslow revision walking of the upstream side of history (nuke\ncan_fast_forward(), make --reapply-cherry-picks the default) or at least\ndon't revision walk so many times (multiple calls to get_merge_bases in\ncan_fast_forward() plus a is_linear_history() walk, checking for upstream\ncherry-picks, probably more), turn off per-commit hooks that probably should\nhave never been on anyway, etc.\n\nElijah Newren (7):\n diffcore-rename: use a mem_pool for exact rename detection's hashmap\n merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers\n merge-ort: set up a memory pool\n merge-ort: switch our strmaps over to using memory pools\n diffcore-rename, merge-ort: add wrapper functions for filepair\n alloc/dealloc\n merge-ort: store filepairs and filespecs in our mem_pool\n merge-ort: reuse path strings in pool_alloc_filespec\n\n diffcore-rename.c | 68 +++++++++++--\n diffcore.h | 3 +\n merge-ort.c | 247 +++++++++++++++++++++++++++++++++++-----------\n 3 files changed, 252 insertions(+), 66 deletions(-)\n\n\nbase-commit: c9ada8369e6575be488028aae0f654422a9b1410\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-990%2Fnewren%2Fort-perf-batch-15-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-990/newren/ort-perf-batch-15-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/990\n\nRange-diff vs v1:\n\n 1: 9f8ab62b842 ! 1: ea08b34d29b diffcore-rename: use a mem_pool for exact rename detection's hashmap\n @@ diffcore-rename.c: static int find_exact_renames(struct diff_options *options)\n \t\t\t\t rename_src[i].p->one);\n \n @@ diffcore-rename.c: static int find_exact_renames(struct diff_options *options)\n + \tfor (i = 0; i < rename_dst_nr; i++)\n \t\trenames += find_identical_files(&file_table, i, options);\n \n - \t/* Free the hash data structure and entries */\n +-\t/* Free the hash data structure and entries */\n -\thashmap_clear_and_free(&file_table, struct file_similarity, entry);\n ++\t/* Free the hash data structure (entries will be freed with the pool) */\n +\thashmap_clear(&file_table);\n \n \treturn renames;\n 3: e30b8c8fea1 ! 2: fdfc2b93ba4 merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers\n @@ Commit message\n or\n mem_pool_alloc, mem_pool_calloc, mem_pool_strndup\n depending on whether we have a non-NULL memory pool. Add these\n - functions; the next commit will make use of these.\n + functions; a subsequent commit will make use of these.\n \n Signed-off-by: Elijah Newren \n \n 2: 77367a69daa = 3: c7150869107 merge-ort: set up a memory pool\n 4: 8231c8e34cd = 4: dd8839b2843 merge-ort: switch our strmaps over to using memory pools\n 5: 2db932bc601 = 5: 560800a80ef diffcore-rename, merge-ort: add wrapper functions for filepair alloc/dealloc\n 6: 629d042884a = 6: 94d60c8a476 merge-ort: store filepairs and filespecs in our mem_pool\n 7: 17aa0a74849 = 7: fda885dabe6 merge-ort: reuse path strings in pool_alloc_filespec\n\n-- \ngitgitgadget\n"},{"id":"431458","messageId":"fdfc2b93ba4b8b0c8c6c51a28c07932ec8f7f59a.1627531121.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v2.git.1627531121.gitgitgadget@gmail.com","subject":"[PATCH v2 2/7] merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-29T03:58:36Z","receivedAt":"2021-07-29T03:58:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe need functions which will either call\n xmalloc, xcalloc, xstrndup\nor\n mem_pool_alloc, mem_pool_calloc, mem_pool_strndup\ndepending on whether we have a non-NULL memory pool. Add these\nfunctions; a subsequent commit will make use of these.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 24 ++++++++++++++++++++++++\n 1 file changed, 24 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex e361443087a..39ddc9b9f2f 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -664,6 +664,30 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n+MAYBE_UNUSED\n+static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n+{\n+\tif (!pool)\n+\t\treturn xcalloc(count, size);\n+\treturn mem_pool_calloc(pool, count, size);\n+}\n+\n+MAYBE_UNUSED\n+static void *pool_alloc(struct mem_pool *pool, size_t size)\n+{\n+\tif (!pool)\n+\t\treturn xmalloc(size);\n+\treturn mem_pool_alloc(pool, size);\n+}\n+\n+MAYBE_UNUSED\n+static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len)\n+{\n+\tif (!pool)\n+\t\treturn xstrndup(str, len);\n+\treturn mem_pool_strndup(pool, str, len);\n+}\n+\n /* add a string to a strbuf, but converting \"/\" to \"_\" */\n static void add_flattened_path(struct strbuf *out, const char *s)\n {\n-- \ngitgitgadget\n\n"},{"id":"431459","messageId":"ea08b34d29b999c1df54bce8dc668f1a23915471.1627531121.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v2.git.1627531121.gitgitgadget@gmail.com","subject":"[PATCH v2 1/7] diffcore-rename: use a mem_pool for exact rename detection's hashmap","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-29T03:58:35Z","receivedAt":"2021-07-29T03:58:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nExact rename detection, via insert_file_table(), uses a hashmap to store\nfiles by oid. Use a mem_pool for the hashmap entries so these can all be\nallocated and deallocated together.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 204.2 ms ± 3.0 ms 202.5 ms ± 3.2 ms\n mega-renames: 1.076 s ± 0.015 s 1.072 s ± 0.012 s\n just-one-mega: 364.1 ms ± 7.0 ms 357.3 ms ± 3.9 ms\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 22 ++++++++++++++++------\n 1 file changed, 16 insertions(+), 6 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 4ef0459cfb5..73d884099eb 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -317,10 +317,11 @@ static int find_identical_files(struct hashmap *srcs,\n }\n \n static void insert_file_table(struct repository *r,\n+\t\t\t struct mem_pool *pool,\n \t\t\t struct hashmap *table, int index,\n \t\t\t struct diff_filespec *filespec)\n {\n-\tstruct file_similarity *entry = xmalloc(sizeof(*entry));\n+\tstruct file_similarity *entry = mem_pool_alloc(pool, sizeof(*entry));\n \n \tentry->index = index;\n \tentry->filespec = filespec;\n@@ -336,7 +337,8 @@ static void insert_file_table(struct repository *r,\n * and then during the second round we try to match\n * cache-dirty entries as well.\n */\n-static int find_exact_renames(struct diff_options *options)\n+static int find_exact_renames(struct diff_options *options,\n+\t\t\t struct mem_pool *pool)\n {\n \tint i, renames = 0;\n \tstruct hashmap file_table;\n@@ -346,7 +348,7 @@ static int find_exact_renames(struct diff_options *options)\n \t */\n \thashmap_init(&file_table, NULL, NULL, rename_src_nr);\n \tfor (i = rename_src_nr-1; i >= 0; i--)\n-\t\tinsert_file_table(options->repo,\n+\t\tinsert_file_table(options->repo, pool,\n \t\t\t\t &file_table, i,\n \t\t\t\t rename_src[i].p->one);\n \n@@ -354,8 +356,8 @@ static int find_exact_renames(struct diff_options *options)\n \tfor (i = 0; i < rename_dst_nr; i++)\n \t\trenames += find_identical_files(&file_table, i, options);\n \n-\t/* Free the hash data structure and entries */\n-\thashmap_clear_and_free(&file_table, struct file_similarity, entry);\n+\t/* Free the hash data structure (entries will be freed with the pool) */\n+\thashmap_clear(&file_table);\n \n \treturn renames;\n }\n@@ -1341,6 +1343,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \tint num_destinations, dst_cnt;\n \tint num_sources, want_copies;\n \tstruct progress *progress = NULL;\n+\tstruct mem_pool local_pool;\n \tstruct dir_rename_info info;\n \tstruct diff_populate_filespec_options dpf_options = {\n \t\t.check_binary = 0,\n@@ -1409,11 +1412,18 @@ void diffcore_rename_extended(struct diff_options *options,\n \t\tgoto cleanup; /* nothing to do */\n \n \ttrace2_region_enter(\"diff\", \"exact renames\", options->repo);\n+\tmem_pool_init(&local_pool, 32*1024);\n \t/*\n \t * We really want to cull the candidates list early\n \t * with cheap tests in order to avoid doing deltas.\n \t */\n-\trename_count = find_exact_renames(options);\n+\trename_count = find_exact_renames(options, &local_pool);\n+\t/*\n+\t * Discard local_pool immediately instead of at \"cleanup:\" in order\n+\t * to reduce maximum memory usage; inexact rename detection uses up\n+\t * a fair amount of memory, and mem_pools can too.\n+\t */\n+\tmem_pool_discard(&local_pool, 0);\n \ttrace2_region_leave(\"diff\", \"exact renames\", options->repo);\n \n \t/* Did we only want exact renames? */\n-- \ngitgitgadget\n\n"},{"id":"431461","messageId":"dd8839b284330892a3bbcafbc03d71489fc9b01f.1627531121.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v2.git.1627531121.gitgitgadget@gmail.com","subject":"[PATCH v2 4/7] merge-ort: switch our strmaps over to using memory pools","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-29T03:58:38Z","receivedAt":"2021-07-29T03:58:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nFor all the strmaps (including strintmaps and strsets) whose memory is\nunconditionally freed as part of clear_or_reinit_internal_opts(), switch\nthem over to using our new memory pool.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 202.5 ms ± 3.2 ms 198.1 ms ± 2.6 ms\n mega-renames: 1.072 s ± 0.012 s 715.8 ms ± 4.0 ms\n just-one-mega: 357.3 ms ± 3.9 ms 276.8 ms ± 4.2 ms\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 125 +++++++++++++++++++++++++++++++---------------------\n 1 file changed, 75 insertions(+), 50 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 2bca4b71f2a..5fd2a4ccd35 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -539,15 +539,19 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \tvoid (*strset_func)(struct strset *) =\n \t\treinitialize ? strset_partial_clear : strset_clear;\n \n-\t/*\n-\t * We marked opti->paths with strdup_strings = 0, so that we\n-\t * wouldn't have to make another copy of the fullpath created by\n-\t * make_traverse_path from setup_path_info(). But, now that we've\n-\t * used it and have no other references to these strings, it is time\n-\t * to deallocate them.\n-\t */\n-\tfree_strmap_strings(&opti->paths);\n-\tstrmap_func(&opti->paths, 1);\n+\tif (opti->pool)\n+\t\tstrmap_func(&opti->paths, 0);\n+\telse {\n+\t\t/*\n+\t\t * We marked opti->paths with strdup_strings = 0, so that\n+\t\t * we wouldn't have to make another copy of the fullpath\n+\t\t * created by make_traverse_path from setup_path_info().\n+\t\t * But, now that we've used it and have no other references\n+\t\t * to these strings, it is time to deallocate them.\n+\t\t */\n+\t\tfree_strmap_strings(&opti->paths);\n+\t\tstrmap_func(&opti->paths, 1);\n+\t}\n \n \t/*\n \t * All keys and values in opti->conflicted are a subset of those in\n@@ -556,16 +560,19 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t */\n \tstrmap_func(&opti->conflicted, 0);\n \n-\t/*\n-\t * opti->paths_to_free is similar to opti->paths; we created it with\n-\t * strdup_strings = 0 to avoid making _another_ copy of the fullpath\n-\t * but now that we've used it and have no other references to these\n-\t * strings, it is time to deallocate them. We do so by temporarily\n-\t * setting strdup_strings to 1.\n-\t */\n-\topti->paths_to_free.strdup_strings = 1;\n-\tstring_list_clear(&opti->paths_to_free, 0);\n-\topti->paths_to_free.strdup_strings = 0;\n+\tif (!opti->pool) {\n+\t\t/*\n+\t\t * opti->paths_to_free is similar to opti->paths; we\n+\t\t * created it with strdup_strings = 0 to avoid making\n+\t\t * _another_ copy of the fullpath but now that we've used\n+\t\t * it and have no other references to these strings, it is\n+\t\t * time to deallocate them. We do so by temporarily\n+\t\t * setting strdup_strings to 1.\n+\t\t */\n+\t\topti->paths_to_free.strdup_strings = 1;\n+\t\tstring_list_clear(&opti->paths_to_free, 0);\n+\t\topti->paths_to_free.strdup_strings = 0;\n+\t}\n \n \tif (opti->attr_index.cache_nr) /* true iff opt->renormalize */\n \t\tdiscard_index(&opti->attr_index);\n@@ -683,7 +690,6 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n-MAYBE_UNUSED\n static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n {\n \tif (!pool)\n@@ -691,7 +697,6 @@ static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n \treturn mem_pool_calloc(pool, count, size);\n }\n \n-MAYBE_UNUSED\n static void *pool_alloc(struct mem_pool *pool, size_t size)\n {\n \tif (!pool)\n@@ -699,7 +704,6 @@ static void *pool_alloc(struct mem_pool *pool, size_t size)\n \treturn mem_pool_alloc(pool, size);\n }\n \n-MAYBE_UNUSED\n static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len)\n {\n \tif (!pool)\n@@ -835,8 +839,9 @@ static void setup_path_info(struct merge_options *opt,\n \tassert(!df_conflict || !resolved); /* df_conflict implies !resolved */\n \tassert(resolved == (merged_version != NULL));\n \n-\tmi = xcalloc(1, resolved ? sizeof(struct merged_info) :\n-\t\t\t\t sizeof(struct conflict_info));\n+\tmi = pool_calloc(opt->priv->pool, 1,\n+\t\t\t resolved ? sizeof(struct merged_info) :\n+\t\t\t\t sizeof(struct conflict_info));\n \tmi->directory_name = current_dir_name;\n \tmi->basename_offset = current_dir_name_len;\n \tmi->clean = !!resolved;\n@@ -1128,7 +1133,7 @@ static int collect_merge_info_callback(int n,\n \tlen = traverse_path_len(info, p->pathlen);\n \n \t/* +1 in both of the following lines to include the NUL byte */\n-\tfullpath = xmalloc(len + 1);\n+\tfullpath = pool_alloc(opt->priv->pool, len + 1);\n \tmake_traverse_path(fullpath, len + 1, info, p->path, p->pathlen);\n \n \t/*\n@@ -1383,7 +1388,7 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\tcopy = renames->deferred[side].possible_trivial_merges;\n \t\tstrintmap_init_with_options(&renames->deferred[side].possible_trivial_merges,\n \t\t\t\t\t 0,\n-\t\t\t\t\t NULL,\n+\t\t\t\t\t opt->priv->pool,\n \t\t\t\t\t 0);\n \t\tstrintmap_for_each_entry(©, &iter, entry) {\n \t\t\tconst char *path = entry->key;\n@@ -2335,12 +2340,21 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \tVERIFY_CI(ci);\n \n \t/* Find parent directories missing from opt->priv->paths */\n-\tcur_path = new_path;\n+\tif (opt->priv->pool) {\n+\t\tcur_path = mem_pool_strdup(opt->priv->pool, new_path);\n+\t\tfree((char*)new_path);\n+\t\tnew_path = (char *)cur_path;\n+\t} else {\n+\t\tcur_path = new_path;\n+\t}\n+\n \twhile (1) {\n \t\t/* Find the parent directory of cur_path */\n \t\tchar *last_slash = strrchr(cur_path, '/');\n \t\tif (last_slash) {\n-\t\t\tparent_name = xstrndup(cur_path, last_slash - cur_path);\n+\t\t\tparent_name = pool_strndup(opt->priv->pool,\n+\t\t\t\t\t\t cur_path,\n+\t\t\t\t\t\t last_slash - cur_path);\n \t\t} else {\n \t\t\tparent_name = opt->priv->toplevel_dir;\n \t\t\tbreak;\n@@ -2349,7 +2363,8 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\t/* Look it up in opt->priv->paths */\n \t\tentry = strmap_get_entry(&opt->priv->paths, parent_name);\n \t\tif (entry) {\n-\t\t\tfree((char*)parent_name);\n+\t\t\tif (!opt->priv->pool)\n+\t\t\t\tfree((char*)parent_name);\n \t\t\tparent_name = entry->key; /* reuse known pointer */\n \t\t\tbreak;\n \t\t}\n@@ -2376,12 +2391,15 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\tparent_name = cur_dir;\n \t}\n \n-\t/*\n-\t * We are removing old_path from opt->priv->paths. old_path also will\n-\t * eventually need to be freed, but it may still be used by e.g.\n-\t * ci->pathnames. So, store it in another string-list for now.\n-\t */\n-\tstring_list_append(&opt->priv->paths_to_free, old_path);\n+\tif (!opt->priv->pool) {\n+\t\t/*\n+\t\t * We are removing old_path from opt->priv->paths.\n+\t\t * old_path also will eventually need to be freed, but it\n+\t\t * may still be used by e.g. ci->pathnames. So, store it\n+\t\t * in another string-list for now.\n+\t\t */\n+\t\tstring_list_append(&opt->priv->paths_to_free, old_path);\n+\t}\n \n \tassert(ci->filemask == 2 || ci->filemask == 4);\n \tassert(ci->dirmask == 0);\n@@ -2416,7 +2434,8 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\tnew_ci->stages[index].mode = ci->stages[index].mode;\n \t\toidcpy(&new_ci->stages[index].oid, &ci->stages[index].oid);\n \n-\t\tfree(ci);\n+\t\tif (!opt->priv->pool)\n+\t\t\tfree(ci);\n \t\tci = new_ci;\n \t}\n \n@@ -3623,7 +3642,8 @@ static void process_entry(struct merge_options *opt,\n \t\t * the directory to remain here, so we need to move this\n \t\t * path to some new location.\n \t\t */\n-\t\tCALLOC_ARRAY(new_ci, 1);\n+\t\tnew_ci = pool_calloc(opt->priv->pool, 1, sizeof(*new_ci));\n+\n \t\t/* We don't really want new_ci->merged.result copied, but it'll\n \t\t * be overwritten below so it doesn't matter. We also don't\n \t\t * want any directory mode/oid values copied, but we'll zero\n@@ -3713,7 +3733,7 @@ static void process_entry(struct merge_options *opt,\n \t\t\tconst char *a_path = NULL, *b_path = NULL;\n \t\t\tint rename_a = 0, rename_b = 0;\n \n-\t\t\tnew_ci = xmalloc(sizeof(*new_ci));\n+\t\t\tnew_ci = pool_alloc(opt->priv->pool, sizeof(*new_ci));\n \n \t\t\tif (S_ISREG(a_mode))\n \t\t\t\trename_a = 1;\n@@ -3777,12 +3797,14 @@ static void process_entry(struct merge_options *opt,\n \t\t\t\tstrmap_remove(&opt->priv->paths, path, 0);\n \t\t\t\t/*\n \t\t\t\t * We removed path from opt->priv->paths. path\n-\t\t\t\t * will also eventually need to be freed, but\n-\t\t\t\t * it may still be used by e.g. ci->pathnames.\n-\t\t\t\t * So, store it in another string-list for now.\n+\t\t\t\t * will also eventually need to be freed if not\n+\t\t\t\t * part of a memory pool...but it may still be\n+\t\t\t\t * used by e.g. ci->pathnames. So, store it in\n+\t\t\t\t * another string-list for now in that case.\n \t\t\t\t */\n-\t\t\t\tstring_list_append(&opt->priv->paths_to_free,\n-\t\t\t\t\t\t path);\n+\t\t\t\tif (!opt->priv->pool)\n+\t\t\t\t\tstring_list_append(&opt->priv->paths_to_free,\n+\t\t\t\t\t\t\t path);\n \t\t\t}\n \n \t\t\t/*\n@@ -4322,6 +4344,7 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n {\n \tstruct rename_info *renames;\n \tint i;\n+\tstruct mem_pool *pool = NULL;\n \n \t/* Sanity checks on opt */\n \ttrace2_region_enter(\"merge\", \"sanity checks\", opt->repo);\n@@ -4393,9 +4416,10 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n #else\n \topt->priv->pool = NULL;\n #endif\n+\tpool = opt->priv->pool;\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->dirs_removed[i],\n-\t\t\t\t\t NOT_RELEVANT, NULL, 0);\n+\t\t\t\t\t NOT_RELEVANT, pool, 0);\n \t\tstrmap_init_with_options(&renames->dir_rename_count[i],\n \t\t\t\t\t NULL, 1);\n \t\tstrmap_init_with_options(&renames->dir_renames[i],\n@@ -4409,7 +4433,7 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t\t */\n \t\tstrintmap_init_with_options(&renames->relevant_sources[i],\n \t\t\t\t\t -1 /* explicitly invalid */,\n-\t\t\t\t\t NULL, 0);\n+\t\t\t\t\t pool, 0);\n \t\tstrmap_init_with_options(&renames->cached_pairs[i],\n \t\t\t\t\t NULL, 1);\n \t\tstrset_init_with_options(&renames->cached_irrelevant[i],\n@@ -4419,9 +4443,9 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t}\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->deferred[i].possible_trivial_merges,\n-\t\t\t\t\t 0, NULL, 0);\n+\t\t\t\t\t 0, pool, 0);\n \t\tstrset_init_with_options(&renames->deferred[i].target_dirs,\n-\t\t\t\t\t NULL, 1);\n+\t\t\t\t\t pool, 1);\n \t\trenames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */\n \t}\n \n@@ -4434,9 +4458,10 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t * In contrast, conflicted just has a subset of keys from paths, so\n \t * we don't want to free those (it'd be a duplicate free).\n \t */\n-\tstrmap_init_with_options(&opt->priv->paths, NULL, 0);\n-\tstrmap_init_with_options(&opt->priv->conflicted, NULL, 0);\n-\tstring_list_init(&opt->priv->paths_to_free, 0);\n+\tstrmap_init_with_options(&opt->priv->paths, pool, 0);\n+\tstrmap_init_with_options(&opt->priv->conflicted, pool, 0);\n+\tif (!opt->priv->pool)\n+\t\tstring_list_init(&opt->priv->paths_to_free, 0);\n \n \t/*\n \t * keys & strbufs in output will sometimes need to outlive \"paths\",\n-- \ngitgitgadget\n\n"},{"id":"431460","messageId":"c715086910714fc8ac160c96e5827e8f0a820240.1627531121.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v2.git.1627531121.gitgitgadget@gmail.com","subject":"[PATCH v2 3/7] merge-ort: set up a memory pool","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-29T03:58:37Z","receivedAt":"2021-07-29T03:58:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nmerge-ort has a lot of data structures, and they all tend to be freed\ntogether in clear_or_reinit_internal_opts(). Set up a memory pool to\nallow us to make these allocations and deallocations faster. Future\ncommits will adjust various callers to make use of this memory pool.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 39ddc9b9f2f..2bca4b71f2a 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -37,6 +37,8 @@\n #include \"unpack-trees.h\"\n #include \"xdiff-interface.h\"\n \n+#define USE_MEMORY_POOL 1 /* faster, but obscures memory leak hunting */\n+\n /*\n * We have many arrays of size 3. Whenever we have such an array, the\n * indices refer to one of the sides of the three-way merge. This is so\n@@ -339,6 +341,17 @@ struct merge_options_internal {\n \t */\n \tstruct strmap conflicted;\n \n+\t/*\n+\t * pool: memory pool for fast allocation/deallocation\n+\t *\n+\t * We allocate room for lots of filenames and auxiliary data\n+\t * structures in merge_options_internal, and it tends to all be\n+\t * freed together too. Using a memory pool for these provides a\n+\t * nice speedup.\n+\t */\n+\tstruct mem_pool internal_pool;\n+\tstruct mem_pool *pool; /* NULL, or pointer to internal_pool */\n+\n \t/*\n \t * paths_to_free: additional list of strings to free\n \t *\n@@ -603,6 +616,12 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\tstrmap_clear(&opti->output, 0);\n \t}\n \n+#if USE_MEMORY_POOL\n+\tmem_pool_discard(&opti->internal_pool, 0);\n+\tif (!reinitialize)\n+\t\topti->pool = NULL;\n+#endif\n+\n \t/* Clean out callback_data as well. */\n \tFREE_AND_NULL(renames->callback_data);\n \trenames->callback_data_nr = renames->callback_data_alloc = 0;\n@@ -4368,6 +4387,12 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \n \t/* Initialization of various renames fields */\n \trenames = &opt->priv->renames;\n+#if USE_MEMORY_POOL\n+\tmem_pool_init(&opt->priv->internal_pool, 0);\n+\topt->priv->pool = &opt->priv->internal_pool;\n+#else\n+\topt->priv->pool = NULL;\n+#endif\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->dirs_removed[i],\n \t\t\t\t\t NOT_RELEVANT, NULL, 0);\n-- \ngitgitgadget\n\n"},{"id":"431462","messageId":"560800a80efe9506e567c09916102aa999932c50.1627531121.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v2.git.1627531121.gitgitgadget@gmail.com","subject":"[PATCH v2 5/7] diffcore-rename, merge-ort: add wrapper functions for filepair alloc/dealloc","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-29T03:58:39Z","receivedAt":"2021-07-29T03:58:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe want to be able to allocate filespecs and filepairs using a mem_pool.\nHowever, filespec data will still remain outside the pool (perhaps in\nthe future we could plumb the pool through the various diff APIs to\nallocate the filespec data too, but for now we are limiting the scope).\nAdd some extra functions to allocate these appropriately based on the\nnon-NULL-ness of opt->priv->pool, as well as some extra functions to\nhandle correctly deallocating the relevant parts of them. A future\ncommit will make use of these new functions.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 41 +++++++++++++++++++++++++++++++++++++++++\n diffcore.h | 2 ++\n merge-ort.c | 42 ++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 85 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 73d884099eb..5bc559f79e9 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -1328,6 +1328,47 @@ static void handle_early_known_dir_renames(struct dir_rename_info *info,\n \trename_src_nr = new_num_src;\n }\n \n+static void free_filespec_data(struct diff_filespec *spec)\n+{\n+\tif (!--spec->count)\n+\t\tdiff_free_filespec_data(spec);\n+}\n+\n+MAYBE_UNUSED\n+static void pool_free_filespec(struct mem_pool *pool,\n+\t\t\t struct diff_filespec *spec)\n+{\n+\tif (!pool) {\n+\t\tfree_filespec(spec);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Similar to free_filespec(), but only frees the data. The spec\n+\t * itself was allocated in the pool and should not be individually\n+\t * freed.\n+\t */\n+\tfree_filespec_data(spec);\n+}\n+\n+MAYBE_UNUSED\n+void pool_diff_free_filepair(struct mem_pool *pool,\n+\t\t\t struct diff_filepair *p)\n+{\n+\tif (!pool) {\n+\t\tdiff_free_filepair(p);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Similar to diff_free_filepair() but only frees the data from the\n+\t * filespecs; not the filespecs or the filepair which were\n+\t * allocated from the pool.\n+\t */\n+\tfree_filespec_data(p->one);\n+\tfree_filespec_data(p->two);\n+}\n+\n void diffcore_rename_extended(struct diff_options *options,\n \t\t\t struct strintmap *relevant_sources,\n \t\t\t struct strintmap *dirs_removed,\ndiff --git a/diffcore.h b/diffcore.h\nindex 533b30e21e7..b58ee6b1934 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -127,6 +127,8 @@ struct diff_filepair {\n #define DIFF_PAIR_MODE_CHANGED(p) ((p)->one->mode != (p)->two->mode)\n \n void diff_free_filepair(struct diff_filepair *);\n+void pool_diff_free_filepair(struct mem_pool *pool,\n+\t\t\t struct diff_filepair *p);\n \n int diff_unmodified_pair(struct diff_filepair *);\n \ndiff --git a/merge-ort.c b/merge-ort.c\nindex 5fd2a4ccd35..59428e45884 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -690,6 +690,48 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n+MAYBE_UNUSED\n+static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n+\t\t\t\t\t\t const char *path)\n+{\n+\tstruct diff_filespec *spec;\n+\tsize_t len;\n+\n+\tif (!pool)\n+\t\treturn alloc_filespec(path);\n+\n+\t/* Same code as alloc_filespec, except allocate from pool */\n+\tlen = strlen(path);\n+\n+\tspec = mem_pool_calloc(pool, 1, st_add3(sizeof(*spec), len, 1));\n+\tmemcpy(spec+1, path, len);\n+\tspec->path = (void*)(spec+1);\n+\n+\tspec->count = 1;\n+\tspec->is_binary = -1;\n+\treturn spec;\n+}\n+\n+MAYBE_UNUSED\n+static struct diff_filepair *pool_diff_queue(struct mem_pool *pool,\n+\t\t\t\t\t struct diff_queue_struct *queue,\n+\t\t\t\t\t struct diff_filespec *one,\n+\t\t\t\t\t struct diff_filespec *two)\n+{\n+\tstruct diff_filepair *dp;\n+\n+\tif (!pool)\n+\t\treturn diff_queue(queue, one, two);\n+\n+\t/* Same code as diff_queue, except allocate from pool */\n+\tdp = mem_pool_calloc(pool, 1, sizeof(*dp));\n+\tdp->one = one;\n+\tdp->two = two;\n+\tif (queue)\n+\t\tdiff_q(queue, dp);\n+\treturn dp;\n+}\n+\n static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n {\n \tif (!pool)\n-- \ngitgitgadget\n\n"},{"id":"431463","messageId":"94d60c8a47640511f68fb88d6e7f42bad8c0811c.1627531121.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v2.git.1627531121.gitgitgadget@gmail.com","subject":"[PATCH v2 6/7] merge-ort: store filepairs and filespecs in our mem_pool","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-29T03:58:40Z","receivedAt":"2021-07-29T03:58:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 198.1 ms ± 2.6 ms 198.5 ms ± 3.4 ms\n mega-renames: 715.8 ms ± 4.0 ms 679.1 ms ± 5.6 ms\n just-one-mega: 276.8 ms ± 4.2 ms 271.9 ms ± 2.8 ms\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 9 ++++-----\n diffcore.h | 1 +\n merge-ort.c | 26 ++++++++++++++------------\n 3 files changed, 19 insertions(+), 17 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 5bc559f79e9..7e6b3e1b143 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -1334,7 +1334,6 @@ static void free_filespec_data(struct diff_filespec *spec)\n \t\tdiff_free_filespec_data(spec);\n }\n \n-MAYBE_UNUSED\n static void pool_free_filespec(struct mem_pool *pool,\n \t\t\t struct diff_filespec *spec)\n {\n@@ -1351,7 +1350,6 @@ static void pool_free_filespec(struct mem_pool *pool,\n \tfree_filespec_data(spec);\n }\n \n-MAYBE_UNUSED\n void pool_diff_free_filepair(struct mem_pool *pool,\n \t\t\t struct diff_filepair *p)\n {\n@@ -1370,6 +1368,7 @@ void pool_diff_free_filepair(struct mem_pool *pool,\n }\n \n void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct mem_pool *pool,\n \t\t\t struct strintmap *relevant_sources,\n \t\t\t struct strintmap *dirs_removed,\n \t\t\t struct strmap *dir_rename_count,\n@@ -1683,7 +1682,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \t\t\tpair_to_free = p;\n \n \t\tif (pair_to_free)\n-\t\t\tdiff_free_filepair(pair_to_free);\n+\t\t\tpool_diff_free_filepair(pool, pair_to_free);\n \t}\n \tdiff_debug_queue(\"done copying original\", &outq);\n \n@@ -1693,7 +1692,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \n \tfor (i = 0; i < rename_dst_nr; i++)\n \t\tif (rename_dst[i].filespec_to_free)\n-\t\t\tfree_filespec(rename_dst[i].filespec_to_free);\n+\t\t\tpool_free_filespec(pool, rename_dst[i].filespec_to_free);\n \n \tcleanup_dir_rename_info(&info, dirs_removed, dir_rename_count != NULL);\n \tFREE_AND_NULL(rename_dst);\n@@ -1710,5 +1709,5 @@ void diffcore_rename_extended(struct diff_options *options,\n \n void diffcore_rename(struct diff_options *options)\n {\n-\tdiffcore_rename_extended(options, NULL, NULL, NULL, NULL);\n+\tdiffcore_rename_extended(options, NULL, NULL, NULL, NULL, NULL);\n }\ndiff --git a/diffcore.h b/diffcore.h\nindex b58ee6b1934..badc2261c20 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -181,6 +181,7 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count);\n void diffcore_break(struct repository *, int);\n void diffcore_rename(struct diff_options *);\n void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct mem_pool *pool,\n \t\t\t struct strintmap *relevant_sources,\n \t\t\t struct strintmap *dirs_removed,\n \t\t\t struct strmap *dir_rename_count,\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 59428e45884..d29c7fe8a30 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -690,7 +690,6 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n-MAYBE_UNUSED\n static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n \t\t\t\t\t\t const char *path)\n {\n@@ -712,7 +711,6 @@ static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n \treturn spec;\n }\n \n-MAYBE_UNUSED\n static struct diff_filepair *pool_diff_queue(struct mem_pool *pool,\n \t\t\t\t\t struct diff_queue_struct *queue,\n \t\t\t\t\t struct diff_filespec *one,\n@@ -930,6 +928,7 @@ static void add_pair(struct merge_options *opt,\n \t\t unsigned dir_rename_mask)\n {\n \tstruct diff_filespec *one, *two;\n+\tstruct mem_pool *pool = opt->priv->pool;\n \tstruct rename_info *renames = &opt->priv->renames;\n \tint names_idx = is_add ? side : 0;\n \n@@ -980,11 +979,11 @@ static void add_pair(struct merge_options *opt,\n \t\t\treturn;\n \t}\n \n-\tone = alloc_filespec(pathname);\n-\ttwo = alloc_filespec(pathname);\n+\tone = pool_alloc_filespec(pool, pathname);\n+\ttwo = pool_alloc_filespec(pool, pathname);\n \tfill_filespec(is_add ? two : one,\n \t\t &names[names_idx].oid, 1, names[names_idx].mode);\n-\tdiff_queue(&renames->pairs[side], one, two);\n+\tpool_diff_queue(pool, &renames->pairs[side], one, two);\n }\n \n static void collect_rename_info(struct merge_options *opt,\n@@ -2893,6 +2892,7 @@ static void use_cached_pairs(struct merge_options *opt,\n {\n \tstruct hashmap_iter iter;\n \tstruct strmap_entry *entry;\n+\tstruct mem_pool *pool = opt->priv->pool;\n \n \t/*\n \t * Add to side_pairs all entries from renames->cached_pairs[side_index].\n@@ -2906,9 +2906,9 @@ static void use_cached_pairs(struct merge_options *opt,\n \t\t\tnew_name = old_name;\n \n \t\t/* We don't care about oid/mode, only filenames and status */\n-\t\tone = alloc_filespec(old_name);\n-\t\ttwo = alloc_filespec(new_name);\n-\t\tdiff_queue(pairs, one, two);\n+\t\tone = pool_alloc_filespec(pool, old_name);\n+\t\ttwo = pool_alloc_filespec(pool, new_name);\n+\t\tpool_diff_queue(pool, pairs, one, two);\n \t\tpairs->queue[pairs->nr-1]->status = entry->value ? 'R' : 'D';\n \t}\n }\n@@ -3016,6 +3016,7 @@ static int detect_regular_renames(struct merge_options *opt,\n \tdiff_queued_diff = renames->pairs[side_index];\n \ttrace2_region_enter(\"diff\", \"diffcore_rename\", opt->repo);\n \tdiffcore_rename_extended(&diff_opts,\n+\t\t\t\t opt->priv->pool,\n \t\t\t\t &renames->relevant_sources[side_index],\n \t\t\t\t &renames->dirs_removed[side_index],\n \t\t\t\t &renames->dir_rename_count[side_index],\n@@ -3066,7 +3067,7 @@ static int collect_renames(struct merge_options *opt,\n \n \t\tif (p->status != 'A' && p->status != 'R') {\n \t\t\tpossibly_cache_new_pair(renames, p, side_index, NULL);\n-\t\t\tdiff_free_filepair(p);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n \t\t\tcontinue;\n \t\t}\n \n@@ -3079,7 +3080,7 @@ static int collect_renames(struct merge_options *opt,\n \n \t\tpossibly_cache_new_pair(renames, p, side_index, new_path);\n \t\tif (p->status != 'R' && !new_path) {\n-\t\t\tdiff_free_filepair(p);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n \t\t\tcontinue;\n \t\t}\n \n@@ -3197,7 +3198,7 @@ cleanup:\n \t\tside_pairs = &renames->pairs[s];\n \t\tfor (i = 0; i < side_pairs->nr; ++i) {\n \t\t\tstruct diff_filepair *p = side_pairs->queue[i];\n-\t\t\tdiff_free_filepair(p);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n \t\t}\n \t}\n \n@@ -3210,7 +3211,8 @@ simple_cleanup:\n \tif (combined.nr) {\n \t\tint i;\n \t\tfor (i = 0; i < combined.nr; i++)\n-\t\t\tdiff_free_filepair(combined.queue[i]);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool,\n+\t\t\t\t\t\tcombined.queue[i]);\n \t\tfree(combined.queue);\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"431464","messageId":"fda885dabe654b09822c5d7697abceda8b3e65d5.1627531121.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v2.git.1627531121.gitgitgadget@gmail.com","subject":"[PATCH v2 7/7] merge-ort: reuse path strings in pool_alloc_filespec","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-29T03:58:41Z","receivedAt":"2021-07-29T03:59:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\npool_alloc_filespec() was written so that the code when pool != NULL\nmimicked the code from alloc_filespec(), which including allocating\nenough extra space for the path and then copying it. However, the path\npassed to pool_alloc_filespec() is always going to already be in the\nsame memory pool, so we may as well reuse it instead of copying it.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 198.5 ms ± 3.4 ms 198.3 ms ± 2.9 ms\n mega-renames: 679.1 ms ± 5.6 ms 661.8 ms ± 5.9 ms\n just-one-mega: 271.9 ms ± 2.8 ms 264.6 ms ± 2.5 ms\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 29 ++++++++++++++++++++++-------\n 1 file changed, 22 insertions(+), 7 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex d29c7fe8a30..0fb942692a7 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -694,17 +694,13 @@ static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n \t\t\t\t\t\t const char *path)\n {\n \tstruct diff_filespec *spec;\n-\tsize_t len;\n \n \tif (!pool)\n \t\treturn alloc_filespec(path);\n \n-\t/* Same code as alloc_filespec, except allocate from pool */\n-\tlen = strlen(path);\n-\n-\tspec = mem_pool_calloc(pool, 1, st_add3(sizeof(*spec), len, 1));\n-\tmemcpy(spec+1, path, len);\n-\tspec->path = (void*)(spec+1);\n+\t/* Similar to alloc_filespec, but allocate from pool and reuse path */\n+\tspec = mem_pool_calloc(pool, 1, sizeof(*spec));\n+\tspec->path = (char*)path; /* spec won't modify it */\n \n \tspec->count = 1;\n \tspec->is_binary = -1;\n@@ -2904,6 +2900,25 @@ static void use_cached_pairs(struct merge_options *opt,\n \t\tconst char *new_name = entry->value;\n \t\tif (!new_name)\n \t\t\tnew_name = old_name;\n+\t\tif (pool) {\n+\t\t\t/*\n+\t\t\t * cached_pairs has _copies* of old_name and new_name,\n+\t\t\t * because it has to persist across merges. When\n+\t\t\t * pool != NULL\n+\t\t\t * pool_alloc_filespec() will just re-use the existing\n+\t\t\t * filenames, which will also get re-used by\n+\t\t\t * opt->priv->paths if they become renames, and then\n+\t\t\t * get freed at the end of the merge, leaving the copy\n+\t\t\t * in cached_pairs dangling. Avoid this by making a\n+\t\t\t * copy here.\n+\t\t\t *\n+\t\t\t * When pool == NULL, pool_alloc_filespec() calls\n+\t\t\t * alloc_filespec(), which makes a copy; we don't want\n+\t\t\t * to add another.\n+\t\t\t */\n+\t\t\told_name = mem_pool_strdup(pool, old_name);\n+\t\t\tnew_name = mem_pool_strdup(pool, new_name);\n+\t\t}\n \n \t\t/* We don't care about oid/mode, only filenames and status */\n \t\tone = pool_alloc_filespec(pool, old_name);\n-- \ngitgitgadget\n"},{"id":"431492","messageId":"629ee63a-4585-be6f-d8f3-28944bbc7ace@gmail.com","threadId":"56152","inReplyTo":"pull.990.v2.git.1627531121.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/7] Final optimization batch (#15): use memory pools","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-29T14:58:26Z","receivedAt":"2021-07-29T14:58:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/28/2021 11:58 PM, Elijah Newren via GitGitGadget wrote:\n> This series textually depends on en/ort-perf-batch-14, but the ideas are\n> orthogonal to it and orthogonal to previous series. It can be reviewed\n> independently.\n> \n> This series is more about strmaps & memory pools than merge logic. CC'ing\n> Peff since he reviewed the strmap work[1], and that work included a number\n> of decisions that specifically had this series in mind.\n> \n> Changes since v1, addressing Eric's feedback:\n> \n> * Fixed a comment that became out-of-date in patch 1\n> * Swapped commits 2 and 3 so that one can better motivate the other.\n\nChanges look good to me. Thanks!\n \n> Note: Stolee also had an interesting question about whether we should tweak\n> the mem_pool_*() API; he and I were both worried it was a bit much, so I've\n> left it out unless others on list chime in with their opinions on that\n> change.\n\nThis was mostly a thought experiment on my part. There is no need to\ndecide one way or another in this series since it would be easy to\nadapt what you have here to match a change to the mem_pool_*() API\nif we thought that was a good idea. (Still not sure it is.)\n\nThanks,\n-Stolee\n"},{"id":"431493","messageId":"YQLInKdjor6Vs1yE@coredump.intra.peff.net","threadId":"56152","inReplyTo":"CABPp-BGjrKWXn2ZzeJ-nsZTwsDE=gbaX=1YKfiNtC70VeN9VVA@mail.gmail.com","subject":"Re: [PATCH 3/7] merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-29T15:26:20Z","receivedAt":"2021-07-29T15:26:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 28, 2021 at 04:49:18PM -0600, Elijah Newren wrote:\n\n> On Mon, Jul 26, 2021 at 8:36 AM Derrick Stolee wrote:\n> >\n> > On 7/23/2021 8:54 AM, Elijah Newren via GitGitGadget wrote:\n> > > From: Elijah Newren \n> > >\n> > > We need functions which will either call\n> > > xmalloc, xcalloc, xstrndup\n> > > or\n> > > mem_pool_alloc, mem_pool_calloc, mem_pool_strndup\n> > > depending on whether we have a non-NULL memory pool. Add these\n> > > functions; the next commit will make use of these.\n> >\n> > I briefly considered that this should just be the way the\n> > mem_pool_* methods work. It does rely on the caller knowing\n> > to free() the allocated memory when their pool is NULL, so\n> > perhaps such a universal change might be too much. What do\n> > you think?\n> \n> That's interesting, but I'm worried it might be a bit much. Do others\n> on the list have an opinion here?\n\nFWIW, I had the same thought. You can also provide a helper to make the\nfreeing side nicer:\n\n static void mem_pool_free(struct mem_pool *m, void *ptr)\n {\n\tif (m)\n\t\treturn; /* will be freed when pool frees */\n\tfree(ptr);\n }\n\nWe do something similar with unuse_commit_buffer(), where the caller\nisn't aware of we pulled the buffer from cache or allocated it\nespecially for them.\n\n-Peff\n"},{"id":"431494","messageId":"YQLJOsvATnTBd9pB@coredump.intra.peff.net","threadId":"56152","inReplyTo":"dd8839b284330892a3bbcafbc03d71489fc9b01f.1627531121.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/7] merge-ort: switch our strmaps over to using memory pools","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-29T15:28:58Z","receivedAt":"2021-07-29T15:29:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 29, 2021 at 03:58:38AM +0000, Elijah Newren via GitGitGadget wrote:\n\n> diff --git a/merge-ort.c b/merge-ort.c\n> index 2bca4b71f2a..5fd2a4ccd35 100644\n> --- a/merge-ort.c\n> +++ b/merge-ort.c\n> @@ -539,15 +539,19 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n> \tvoid (*strset_func)(struct strset *) =\n> \t\treinitialize ? strset_partial_clear : strset_clear;\n> \n> -\t/*\n> -\t * We marked opti->paths with strdup_strings = 0, so that we\n> -\t * wouldn't have to make another copy of the fullpath created by\n> -\t * make_traverse_path from setup_path_info(). But, now that we've\n> -\t * used it and have no other references to these strings, it is time\n> -\t * to deallocate them.\n> -\t */\n> -\tfree_strmap_strings(&opti->paths);\n> -\tstrmap_func(&opti->paths, 1);\n> +\tif (opti->pool)\n> +\t\tstrmap_func(&opti->paths, 0);\n\nThis isn't new in your patch here, but I did scratch my head a bit over\nwhat \"strmap_func\" is. It's a bit less confusing if you read the whole\nfunction (as opposed to a diff), since then you're more likely to see\nthe definition. But something like \"strmap_clear_func()\" would have been\na lot less confusing.\n\nArguably, the existence of these function indirections is perhaps a sign\nthat the strmap API should provide a version of the clear functions that\ntakes \"partial / not-partial\" as a parameter.\n\n(Again, not really part of this patch series, but I hadn't looked at\nsome of the earlier optimization steps).\n\n-Peff\n"},{"id":"431495","messageId":"YQLVPN9n5E1Yvi/q@coredump.intra.peff.net","threadId":"56152","inReplyTo":"pull.990.v2.git.1627531121.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/7] Final optimization batch (#15): use memory pools","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-29T16:20:12Z","receivedAt":"2021-07-29T16:20:15Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 29, 2021 at 03:58:34AM +0000, Elijah Newren via GitGitGadget wrote:\n\n> This series is more about strmaps & memory pools than merge logic. CC'ing\n> Peff since he reviewed the strmap work[1], and that work included a number\n> of decisions that specifically had this series in mind.\n\nI haven't been following the other optimization threads very closely,\nbut I'll try to give my general impressions.\n\n> === Basic Optimization idea ===\n> \n> In this series, I make use of memory pools to get faster allocations and\n> deallocations for many data structures that tend to all be deallocated at\n> the same time anyway.\n> \n> === Results ===\n> \n> For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n> performance work; instrument with trace2_region_* calls\", 2020-10-28), the\n> changes in just this series improves the performance as follows:\n> \n> Before Series After Series\n> no-renames: 204.2 ms ± 3.0 ms 198.3 ms ± 2.9 ms\n> mega-renames: 1.076 s ± 0.015 s 661.8 ms ± 5.9 ms\n> just-one-mega: 364.1 ms ± 7.0 ms 264.6 ms ± 2.5 ms\n\nPretty good results for the mega-renames case. I do wonder how much this\nmatters in practice. That case is intentionally stressing the system,\nthough I guess it's not too far-fetched (it's mostly a big directory\nrename). However, just \"git checkout\" across the rename already takes\nmore than a second. So on the one hand, 400ms isn't nothing. On the\nother, I doubt anybody is likely to notice in the grand scheme of\nthings.\n\nAnd we're paying a non-trivial cost in code complexity to do it (though\nI do think you've done an admirable job of making that cost as low as\npossible). Dropping the USE_MEMORY_POOL flag and just always using a\npool would make a lot of that complexity go away. I understand how it\nmakes leak hunting harder, but I think simpler code would probably be\nworth the tradeoff (and in a sense, there _aren't_ leaks in an\nalways-pool world; we're holding on to all of the memory through the\nwhole operation).\n\nI assume your tests are just done using the regular glibc allocator. I\nalso wondered how plugging in a better allocator might fare. Here are\ntimings I did of your mega-renames case with three binaries: one built\nwith USE_MEMORY_POOL set to 0, one with it set to 1, and one with it set\nto 0 but adding \"-ltcmalloc\" to EXTLIBS via config.mak.\n\n $ hyperfine \\\n -p 'git checkout hwmon-updates &&\n git reset --hard fd8bdb23b91876ac1e624337bb88dc1dcc21d67e &&\n git checkout 5.4-renames^0' \\\n -L version nopool,pool,tcmalloc \\\n './test-tool.{version} fast-rebase --onto HEAD base hwmon-updates'\n \n Benchmark #1: ./test-tool.nopool fast-rebase --onto HEAD base hwmon-updates\n Time (mean ± σ): 921.1 ms ± 146.0 ms [User: 843.0 ms, System: 77.5 ms]\n Range (min … max): 660.9 ms … 1112.2 ms 10 runs\n \n Benchmark #2: ./test-tool.pool fast-rebase --onto HEAD base hwmon-updates\n Time (mean ± σ): 635.4 ms ± 125.5 ms [User: 563.7 ms, System: 71.3 ms]\n Range (min … max): 496.8 ms … 856.7 ms 10 runs\n \n Benchmark #3: ./test-tool.tcmalloc fast-rebase --onto HEAD base hwmon-updates\n Time (mean ± σ): 727.3 ms ± 139.9 ms [User: 654.1 ms, System: 72.9 ms]\n Range (min … max): 476.3 ms … 900.5 ms 10 runs\n \n Summary\n './test-tool.pool fast-rebase --onto HEAD base hwmon-updates' ran\n 1.14 ± 0.32 times faster than './test-tool.tcmalloc fast-rebase --onto HEAD base hwmon-updates'\n 1.45 ± 0.37 times faster than './test-tool.nopool fast-rebase --onto HEAD base hwmon-updates'\n\nThe pool allocator does come out ahead when comparing means, but the\nimprovement is within the noise (and the fastest run was actually with\ntcmalloc).\n\nI was also curious about peak heap usage. According to massif, the pool\nversion peaks at ~800k extra (out of 82MB), which is negligible. Plus it\nhas fewer overall allocations, so it seems to actually save 4-5MB in\nmalloc overhead (though I would imagine that varies between allocators;\nI'm just going from massif numbers here).\n\nSo...I dunno. It's hard to assess the real-world impact of the speedup,\ncompared to the complexity cost. Ultimately, this is changing code that\nyou wrote and will probably maintain. So I'd leave the final decision\nfor that tradeoff to you. I'm just injecting some thoughts and numbers. :)\n\n-Peff\n"},{"id":"431496","messageId":"YQLV/MTQ+MkxUBSn@coredump.intra.peff.net","threadId":"56152","inReplyTo":"YQLVPN9n5E1Yvi/q@coredump.intra.peff.net","subject":"Re: [PATCH v2 0/7] Final optimization batch (#15): use memory pools","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-29T16:23:24Z","receivedAt":"2021-07-29T16:23:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 29, 2021 at 12:20:13PM -0400, Jeff King wrote:\n\n> I assume your tests are just done using the regular glibc allocator. I\n> also wondered how plugging in a better allocator might fare. Here are\n> timings I did of your mega-renames case with three binaries: one built\n> with USE_MEMORY_POOL set to 0, one with it set to 1, and one with it set\n> to 0 but adding \"-ltcmalloc\" to EXTLIBS via config.mak.\n\nOh, btw, I wasn't able to apply your series from the list on top of\nen/ort-perf-batch-14 (there are some problems in patch 4, and \"am -3\"\nsays my clone of git.git is missing some of the pre-image sha1s). I\nfetched ort-perf-batch-15 from https://github.com/newren/git and timed\nthat, which I imagine is the same. But you may need to tweak the patches\nso that Junio can pick them up.\n\n-Peff\n"},{"id":"431503","messageId":"CABPp-BHzJ365Jed38s-VQb2PqRy7t=58sZwgmeVG=mqKeA2neg@mail.gmail.com","threadId":"56152","inReplyTo":"YQLJOsvATnTBd9pB@coredump.intra.peff.net","subject":"Re: [PATCH v2 4/7] merge-ort: switch our strmaps over to using memory pools","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-29T18:37:52Z","receivedAt":"2021-07-29T18:38:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jul 29, 2021 at 9:29 AM Jeff King wrote:\n>\n> On Thu, Jul 29, 2021 at 03:58:38AM +0000, Elijah Newren via GitGitGadget wrote:\n>\n> > diff --git a/merge-ort.c b/merge-ort.c\n> > index 2bca4b71f2a..5fd2a4ccd35 100644\n> > --- a/merge-ort.c\n> > +++ b/merge-ort.c\n> > @@ -539,15 +539,19 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n> > void (*strset_func)(struct strset *) =\n> > reinitialize ? strset_partial_clear : strset_clear;\n> >\n> > - /*\n> > - * We marked opti->paths with strdup_strings = 0, so that we\n> > - * wouldn't have to make another copy of the fullpath created by\n> > - * make_traverse_path from setup_path_info(). But, now that we've\n> > - * used it and have no other references to these strings, it is time\n> > - * to deallocate them.\n> > - */\n> > - free_strmap_strings(&opti->paths);\n> > - strmap_func(&opti->paths, 1);\n> > + if (opti->pool)\n> > + strmap_func(&opti->paths, 0);\n>\n> This isn't new in your patch here, but I did scratch my head a bit over\n> what \"strmap_func\" is. It's a bit less confusing if you read the whole\n> function (as opposed to a diff), since then you're more likely to see\n> the definition. But something like \"strmap_clear_func()\" would have been\n> a lot less confusing.\n\nMakes sense.\n\n> Arguably, the existence of these function indirections is perhaps a sign\n> that the strmap API should provide a version of the clear functions that\n> takes \"partial / not-partial\" as a parameter.\n\nAre you suggesting a modification of str{map,intmap,set}_clear() to\ntake an extra parameter, or removing the\nstr{map,intmap,set}_partial_clear() functions and introducing new\nfunctions that take a partial/not-partial parameter? I think you're\nsuggesting the latter, and that makes more sense to me...but I'm\ndrawing blanks trying to come up with a reasonable function name.\n\n(If it helps for context -- the only current callers of the\n*_partial_clear() functions are found in diffcore-rename.c and\nmerge-ort.c, so it'd be a pretty easy change to make to those. There\nare additionally some callers of strmap_clear() and strset_clear() in\nbuiltin/shortlog.c and rerere.c, and it'd be nice to avoid exposing\nthose to the complexity of the partial clearing.)\n\n> (Again, not really part of this patch series, but I hadn't looked at\n> some of the earlier optimization steps).\n\nYeah, but this is the kind of reason I wanted you to review this\nseries, because I figured you might have more good comments on the\nstr{map,intmap,set} API calls. :-)\n"},{"id":"431515","messageId":"xmqq1r7g6gyg.fsf@gitster.g","threadId":"56152","inReplyTo":"YQLV/MTQ+MkxUBSn@coredump.intra.peff.net","subject":"Re: [PATCH v2 0/7] Final optimization batch (#15): use memory pools","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-29T19:46:47Z","receivedAt":"2021-07-29T19:46:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Thu, Jul 29, 2021 at 12:20:13PM -0400, Jeff King wrote:\n>\n>> I assume your tests are just done using the regular glibc allocator. I\n>> also wondered how plugging in a better allocator might fare. Here are\n>> timings I did of your mega-renames case with three binaries: one built\n>> with USE_MEMORY_POOL set to 0, one with it set to 1, and one with it set\n>> to 0 but adding \"-ltcmalloc\" to EXTLIBS via config.mak.\n>\n> Oh, btw, I wasn't able to apply your series from the list on top of\n> en/ort-perf-batch-14 (there are some problems in patch 4, and \"am -3\"\n> says my clone of git.git is missing some of the pre-image sha1s). I\n> fetched ort-perf-batch-15 from https://github.com/newren/git and timed\n> that, which I imagine is the same. But you may need to tweak the patches\n> so that Junio can pick them up.\n\nThanks, but the batch #15 has been in 'seen' since 23rd ;-)\n"},{"id":"431519","messageId":"YQMLBW6LAZmQ8nDz@coredump.intra.peff.net","threadId":"56152","inReplyTo":"CABPp-BHzJ365Jed38s-VQb2PqRy7t=58sZwgmeVG=mqKeA2neg@mail.gmail.com","subject":"Re: [PATCH v2 4/7] merge-ort: switch our strmaps over to using memory pools","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-29T20:09:41Z","receivedAt":"2021-07-29T20:09:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 29, 2021 at 12:37:52PM -0600, Elijah Newren wrote:\n\n> > Arguably, the existence of these function indirections is perhaps a sign\n> > that the strmap API should provide a version of the clear functions that\n> > takes \"partial / not-partial\" as a parameter.\n> \n> Are you suggesting a modification of str{map,intmap,set}_clear() to\n> take an extra parameter, or removing the\n> str{map,intmap,set}_partial_clear() functions and introducing new\n> functions that take a partial/not-partial parameter? I think you're\n> suggesting the latter, and that makes more sense to me...but I'm\n> drawing blanks trying to come up with a reasonable function name.\n\nIt does seem a shame to add the \"partial\" parameter to strmap_clear(),\njust because most callers don't need it (so they end up with this\ninscrutable \"0\" parameter).\n\nWhat if there was a flags field? Then it could be combined with the\nfree_values parameter. The result is kind of verbose in two ways:\n\n - now strset_clear(), etc, need a \"flags\" parameter, which they didn't\n before (and is just \"0\" most of the time!)\n\n - now \"strmap_clear(foo, 1)\" becomes \"strmap_clear(foo, STRMAP_FREE_VALUES)\".\n That's a lot longer, though arguably it's easier to understand since\n the boolean is explained.\n\nHaving gone through the exercise, I am not sure it is actually making\nanything more readable (messy patch is below for reference).\n\ndiff --git a/builtin/shortlog.c b/builtin/shortlog.c\nindex 3e7ab1ca82..dfbdba53da 100644\n--- a/builtin/shortlog.c\n+++ b/builtin/shortlog.c\n@@ -242,7 +242,7 @@ void shortlog_add_commit(struct shortlog *log, struct commit *commit)\n \t\tinsert_records_from_trailers(log, &dups, commit, &ctx, oneline_str);\n \t}\n \n-\tstrset_clear(&dups);\n+\tstrset_clear(&dups, 0);\n \tstrbuf_release(&ident);\n \tstrbuf_release(&oneline);\n }\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 7e6b3e1b14..0c960111d1 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -665,9 +665,10 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n \n \tstrmap_for_each_entry(dir_rename_count, &iter, entry) {\n \t\tstruct strintmap *counts = entry->value;\n-\t\tstrintmap_clear(counts);\n+\t\tstrintmap_clear(counts, 0);\n \t}\n-\tstrmap_partial_clear(dir_rename_count, 1);\n+\tstrmap_clear(dir_rename_count,\n+\t\t STRMAP_FREE_VALUES | STRMAP_PARTIAL_CLEAR);\n }\n \n static void cleanup_dir_rename_info(struct dir_rename_info *info,\n@@ -683,15 +684,15 @@ static void cleanup_dir_rename_info(struct dir_rename_info *info,\n \t\treturn;\n \n \t/* idx_map */\n-\tstrintmap_clear(&info->idx_map);\n+\tstrintmap_clear(&info->idx_map, 0);\n \n \t/* dir_rename_guess */\n-\tstrmap_clear(&info->dir_rename_guess, 1);\n+\tstrmap_clear(&info->dir_rename_guess, STRMAP_FREE_VALUES);\n \n \t/* relevant_source_dirs */\n \tif (info->relevant_source_dirs &&\n \t info->relevant_source_dirs != dirs_removed) {\n-\t\tstrintmap_clear(info->relevant_source_dirs);\n+\t\tstrintmap_clear(info->relevant_source_dirs, 0);\n \t\tFREE_AND_NULL(info->relevant_source_dirs);\n \t}\n \n@@ -716,7 +717,7 @@ static void cleanup_dir_rename_info(struct dir_rename_info *info,\n \n \t\tif (!strintmap_get(dirs_removed, source_dir)) {\n \t\t\tstring_list_append(&to_remove, source_dir);\n-\t\t\tstrintmap_clear(counts);\n+\t\t\tstrintmap_clear(counts, 0);\n \t\t\tcontinue;\n \t\t}\n \n@@ -1045,8 +1046,8 @@ static int find_basename_matches(struct diff_options *options,\n \t\t}\n \t}\n \n-\tstrintmap_clear(&sources);\n-\tstrintmap_clear(&dests);\n+\tstrintmap_clear(&sources, 0);\n+\tstrintmap_clear(&dests, 0);\n \n \treturn renames;\n }\n@@ -1700,7 +1701,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \tFREE_AND_NULL(rename_src);\n \trename_src_nr = rename_src_alloc = 0;\n \tif (break_idx) {\n-\t\tstrintmap_clear(break_idx);\n+\t\tstrintmap_clear(break_idx, 0);\n \t\tFREE_AND_NULL(break_idx);\n \t}\n \ttrace2_region_leave(\"diff\", \"write back to queue\", options->repo);\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 0fb942692a..0765e23577 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -532,15 +532,10 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n {\n \tstruct rename_info *renames = &opti->renames;\n \tint i;\n-\tvoid (*strmap_func)(struct strmap *, int) =\n-\t\treinitialize ? strmap_partial_clear : strmap_clear;\n-\tvoid (*strintmap_func)(struct strintmap *) =\n-\t\treinitialize ? strintmap_partial_clear : strintmap_clear;\n-\tvoid (*strset_func)(struct strset *) =\n-\t\treinitialize ? strset_partial_clear : strset_clear;\n+\tunsigned flags = reinitialize ? STRMAP_PARTIAL_CLEAR : 0;\n \n \tif (opti->pool)\n-\t\tstrmap_func(&opti->paths, 0);\n+\t\tstrmap_clear(&opti->paths, flags);\n \telse {\n \t\t/*\n \t\t * We marked opti->paths with strdup_strings = 0, so that\n@@ -550,15 +545,15 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\t * to these strings, it is time to deallocate them.\n \t\t */\n \t\tfree_strmap_strings(&opti->paths);\n-\t\tstrmap_func(&opti->paths, 1);\n+\t\tstrmap_clear(&opti->paths, flags | STRMAP_FREE_VALUES);\n \t}\n \n \t/*\n \t * All keys and values in opti->conflicted are a subset of those in\n \t * opti->paths. We don't want to deallocate anything twice, so we\n \t * don't free the keys and we pass 0 for free_values.\n \t */\n-\tstrmap_func(&opti->conflicted, 0);\n+\tstrmap_clear(&opti->conflicted, flags);\n \n \tif (!opti->pool) {\n \t\t/*\n@@ -579,24 +574,24 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \n \t/* Free memory used by various renames maps */\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n-\t\tstrintmap_func(&renames->dirs_removed[i]);\n-\t\tstrmap_func(&renames->dir_renames[i], 0);\n-\t\tstrintmap_func(&renames->relevant_sources[i]);\n+\t\tstrintmap_clear(&renames->dirs_removed[i], flags);\n+\t\tstrmap_clear(&renames->dir_renames[i], flags);\n+\t\tstrintmap_clear(&renames->relevant_sources[i], flags);\n \t\tif (!reinitialize)\n \t\t\tassert(renames->cached_pairs_valid_side == 0);\n \t\tif (i != renames->cached_pairs_valid_side &&\n \t\t -1 != renames->cached_pairs_valid_side) {\n-\t\t\tstrset_func(&renames->cached_target_names[i]);\n-\t\t\tstrmap_func(&renames->cached_pairs[i], 1);\n-\t\t\tstrset_func(&renames->cached_irrelevant[i]);\n+\t\t\tstrset_clear(&renames->cached_target_names[i], flags);\n+\t\t\tstrmap_clear(&renames->cached_pairs[i], flags | STRMAP_FREE_VALUES);\n+\t\t\tstrset_clear(&renames->cached_irrelevant[i], flags);\n \t\t\tpartial_clear_dir_rename_count(&renames->dir_rename_count[i]);\n \t\t\tif (!reinitialize)\n \t\t\t\tstrmap_clear(&renames->dir_rename_count[i], 1);\n \t\t}\n \t}\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n-\t\tstrintmap_func(&renames->deferred[i].possible_trivial_merges);\n-\t\tstrset_func(&renames->deferred[i].target_dirs);\n+\t\tstrintmap_clear(&renames->deferred[i].possible_trivial_merges, flags);\n+\t\tstrset_clear(&renames->deferred[i].target_dirs, flags);\n \t\trenames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */\n \t}\n \trenames->cached_pairs_valid_side = 0;\n@@ -1482,7 +1477,7 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\t\tif (ret < 0)\n \t\t\t\treturn ret;\n \t\t}\n-\t\tstrintmap_clear(©);\n+\t\tstrintmap_clear(©, 0);\n \t\tstrintmap_for_each_entry(&renames->deferred[side].possible_trivial_merges,\n \t\t\t\t\t &iter, entry) {\n \t\t\tconst char *path = entry->key;\ndiff --git a/strmap.c b/strmap.c\nindex 4fb9f6100e..7343800df5 100644\n--- a/strmap.c\n+++ b/strmap.c\n@@ -37,10 +37,11 @@ void strmap_init_with_options(struct strmap *map,\n \tmap->strdup_strings = strdup_strings;\n }\n \n-static void strmap_free_entries_(struct strmap *map, int free_values)\n+static void strmap_free_entries_(struct strmap *map, unsigned flags)\n {\n \tstruct hashmap_iter iter;\n \tstruct strmap_entry *e;\n+\tint free_values = flags & STRMAP_FREE_VALUES;\n \n \tif (!map)\n \t\treturn;\n@@ -64,16 +65,13 @@ static void strmap_free_entries_(struct strmap *map, int free_values)\n \t}\n }\n \n-void strmap_clear(struct strmap *map, int free_values)\n+void strmap_clear(struct strmap *map, unsigned flags)\n {\n-\tstrmap_free_entries_(map, free_values);\n-\thashmap_clear(&map->map);\n-}\n-\n-void strmap_partial_clear(struct strmap *map, int free_values)\n-{\n-\tstrmap_free_entries_(map, free_values);\n-\thashmap_partial_clear(&map->map);\n+\tstrmap_free_entries_(map, flags);\n+\tif (flags & STRMAP_PARTIAL_CLEAR)\n+\t\thashmap_partial_clear(&map->map);\n+\telse\n+\t\thashmap_clear(&map->map);\n }\n \n static struct strmap_entry *create_entry(struct strmap *map,\ndiff --git a/strmap.h b/strmap.h\nindex 1e152d832d..d03d451654 100644\n--- a/strmap.h\n+++ b/strmap.h\n@@ -46,16 +46,14 @@ void strmap_init_with_options(struct strmap *map,\n \t\t\t struct mem_pool *pool,\n \t\t\t int strdup_strings);\n \n-/*\n- * Remove all entries from the map, releasing any allocated resources.\n- */\n-void strmap_clear(struct strmap *map, int free_values);\n+#define STRMAP_FREE_VALUES 1 /* 1 for historical compat, but we should probably\n+\t\t\t\tupdate callers to use the correct name) */\n+#define STRMAP_PARTIAL_CLEAR 2\n \n /*\n- * Similar to strmap_clear() but leaves map->map->table allocated and\n- * pre-sized so that subsequent uses won't need as many rehashings.\n+ * Remove all entries from the map, releasing any allocated resources.\n */\n-void strmap_partial_clear(struct strmap *map, int free_values);\n+void strmap_clear(struct strmap *map, unsigned flags);\n \n /*\n * Insert \"str\" into the map, pointing to \"data\".\n@@ -148,14 +146,10 @@ static inline void strintmap_init_with_options(struct strintmap *map,\n \tmap->default_value = default_value;\n }\n \n-static inline void strintmap_clear(struct strintmap *map)\n-{\n-\tstrmap_clear(&map->map, 0);\n-}\n-\n-static inline void strintmap_partial_clear(struct strintmap *map)\n+static inline void strintmap_clear(struct strintmap *map, unsigned flags)\n {\n-\tstrmap_partial_clear(&map->map, 0);\n+\t/* maybe clear STRMAP_FREE_VALUES bit for extra protection */\n+\tstrmap_clear(&map->map, flags);\n }\n \n static inline int strintmap_contains(struct strintmap *map, const char *str)\n@@ -232,14 +226,9 @@ static inline void strset_init_with_options(struct strset *set,\n \tstrmap_init_with_options(&set->map, pool, strdup_strings);\n }\n \n-static inline void strset_clear(struct strset *set)\n-{\n-\tstrmap_clear(&set->map, 0);\n-}\n-\n-static inline void strset_partial_clear(struct strset *set)\n+static inline void strset_clear(struct strset *set, unsigned flags)\n {\n-\tstrmap_partial_clear(&set->map, 0);\n+\tstrmap_clear(&set->map, flags);\n }\n \n static inline int strset_contains(struct strset *set, const char *str)\n"},{"id":"431523","messageId":"CABPp-BH2PQ_rHDPNBghdkhO=kaNfQBnU28UFh9c-sPBNUOM4gA@mail.gmail.com","threadId":"56152","inReplyTo":"YQLVPN9n5E1Yvi/q@coredump.intra.peff.net","subject":"Re: [PATCH v2 0/7] Final optimization batch (#15): use memory pools","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-29T20:46:03Z","receivedAt":"2021-07-29T20:46:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jul 29, 2021 at 10:20 AM Jeff King wrote:\n>\n> On Thu, Jul 29, 2021 at 03:58:34AM +0000, Elijah Newren via GitGitGadget wrote:\n>\n> > This series is more about strmaps & memory pools than merge logic. CC'ing\n> > Peff since he reviewed the strmap work[1], and that work included a number\n> > of decisions that specifically had this series in mind.\n>\n> I haven't been following the other optimization threads very closely,\n> but I'll try to give my general impressions.\n>\n> > === Basic Optimization idea ===\n> >\n> > In this series, I make use of memory pools to get faster allocations and\n> > deallocations for many data structures that tend to all be deallocated at\n> > the same time anyway.\n> >\n> > === Results ===\n> >\n> > For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n> > performance work; instrument with trace2_region_* calls\", 2020-10-28), the\n> > changes in just this series improves the performance as follows:\n> >\n> > Before Series After Series\n> > no-renames: 204.2 ms ± 3.0 ms 198.3 ms ± 2.9 ms\n> > mega-renames: 1.076 s ± 0.015 s 661.8 ms ± 5.9 ms\n> > just-one-mega: 364.1 ms ± 7.0 ms 264.6 ms ± 2.5 ms\n>\n> Pretty good results for the mega-renames case. I do wonder how much this\n> matters in practice. That case is intentionally stressing the system,\n> though I guess it's not too far-fetched (it's mostly a big directory\n> rename). However, just \"git checkout\" across the rename already takes\n> more than a second. So on the one hand, 400ms isn't nothing. On the\n> other, I doubt anybody is likely to notice in the grand scheme of\n> things.\n\nThe mega-renames case was spurred by looking at a repository at\n$DAYJOB and trying to generate a similar testcase on a well-known open\nsource repository with approximately the same number of files. The\nlinux kernel was handy for that. Technically, to make it match the\nright number of renames, I would have needed to rename more toplevel\ndirectories, but it was close enough and I liked it being a simple\ntestcase. So, to me, the testcase is more unusual in the number of\npatches being rebased rather than in the number of renames, but I\nchose a long sequence of patches (with lots of different types of\nchanges) because that served as a good correctness case and even ended\nup being good for spurring searches for tweaks to existing\noptimizations. And I added the just-one-mega to see how a simple\ncherry-pick would work with the large number of renames. It too has a\ngood relative speedup.\n\nYou make fair points about the absolute timings for rebase, but the\n\"grand scheme of things\" involves usecases outside of traditional\nrebases. Some of those involve far more merges, making the relative\ntimings more important. They also involve much less overhead -- not\nonly do they get to ignore the \"git checkout\" present in most rebases,\nbut they also get to exclude the index or worktree updating that are\npresent above. Without that extra overhead, the relative improvement\nfrom this patch is even greater. One particular example usecase that\nis coming soon is the introduction of `git log --remerge-diff`; it\nmakes the timing of individual merges critical since it does so many\nof them. And it becomes even more important for users who change the\ndefault from --diff-merges=off to --diff-merges=remerge-diff, because\nthen any `git log -p` will potentially remerge hundreds or thousands\nof merge commits. (I've got lots of users who have -p imply\n--remerge-diff since last November.)\n\n> And we're paying a non-trivial cost in code complexity to do it (though\n> I do think you've done an admirable job of making that cost as low as\n> possible). Dropping the USE_MEMORY_POOL flag and just always using a\n> pool would make a lot of that complexity go away. I understand how it\n> makes leak hunting harder, but I think simpler code would probably be\n> worth the tradeoff (and in a sense, there _aren't_ leaks in an\n> always-pool world; we're holding on to all of the memory through the\n> whole operation).\n\nYeah, I had to keep the USE_MEMORY_POOL flag as I was rebasing my\nsequence of optimization series to make sure I kept the intermediate\nsteps clean while upstreaming all the work. I ended up just leaving\n(a simplified form of) it in when I was all done, but it has probably\nalready served its purpose. I'd be fine with dropping it.\n\n> I assume your tests are just done using the regular glibc allocator. I\n\nYes.\n\n> also wondered how plugging in a better allocator might fare. Here are\n> timings I did of your mega-renames case with three binaries: one built\n> with USE_MEMORY_POOL set to 0, one with it set to 1, and one with it set\n> to 0 but adding \"-ltcmalloc\" to EXTLIBS via config.mak.\n>\n> $ hyperfine \\\n> -p 'git checkout hwmon-updates &&\n> git reset --hard fd8bdb23b91876ac1e624337bb88dc1dcc21d67e &&\n> git checkout 5.4-renames^0' \\\n> -L version nopool,pool,tcmalloc \\\n> './test-tool.{version} fast-rebase --onto HEAD base hwmon-updates'\n\nOoh, I didn't know about -L.\n\n> Benchmark #1: ./test-tool.nopool fast-rebase --onto HEAD base hwmon-updates\n> Time (mean ± σ): 921.1 ms ± 146.0 ms [User: 843.0 ms, System: 77.5 ms]\n> Range (min … max): 660.9 ms … 1112.2 ms 10 runs\n>\n> Benchmark #2: ./test-tool.pool fast-rebase --onto HEAD base hwmon-updates\n> Time (mean ± σ): 635.4 ms ± 125.5 ms [User: 563.7 ms, System: 71.3 ms]\n> Range (min … max): 496.8 ms … 856.7 ms 10 runs\n>\n> Benchmark #3: ./test-tool.tcmalloc fast-rebase --onto HEAD base hwmon-updates\n> Time (mean ± σ): 727.3 ms ± 139.9 ms [User: 654.1 ms, System: 72.9 ms]\n> Range (min … max): 476.3 ms … 900.5 ms 10 runs\n\nThat's some _really_ wide variance on your runs, making me wonder if\nyou are running other things on your (I presume) laptop that are\npotentially muddying the numbers. Would the tcmalloc case actually\nhave the fastest run in general, or was it just lucky to hit a \"quiet\"\nmoment on the laptop?\n\nOr perhaps my pre-warming script helps reduce variance more than I thought...\n\n> Summary\n> './test-tool.pool fast-rebase --onto HEAD base hwmon-updates' ran\n> 1.14 ± 0.32 times faster than './test-tool.tcmalloc fast-rebase --onto HEAD base hwmon-updates'\n> 1.45 ± 0.37 times faster than './test-tool.nopool fast-rebase --onto HEAD base hwmon-updates'\n>\n> The pool allocator does come out ahead when comparing means, but the\n> improvement is within the noise (and the fastest run was actually with\n> tcmalloc).\n>\n> I was also curious about peak heap usage. According to massif, the pool\n> version peaks at ~800k extra (out of 82MB), which is negligible. Plus it\n> has fewer overall allocations, so it seems to actually save 4-5MB in\n> malloc overhead (though I would imagine that varies between allocators;\n> I'm just going from massif numbers here).\n\nI did similar testing a year ago, before I even looked at memory\npools. I was surprised by how big a speedup I saw, and considered\nasking on the list if we could push to use a different allocator by\ndefault. Ultimately, I figured that probably wouldn't fly and\ndistributors might override our choices anyway. It was at that point\nthat I decided to start tweaking mem-pool.[ch] (which ended up getting\nmerged at edab8a8d07 (\"Merge branch 'en/mem-pool'\", 2020-08-27)), and\nthen integrating that into strmap/strset/strintmap -- all in an effort\nto guarantee that we realized the speedups that I knew were possible\ndue to my testing with the special allocators.\n\n> So...I dunno. It's hard to assess the real-world impact of the speedup,\n> compared to the complexity cost. Ultimately, this is changing code that\n> you wrote and will probably maintain. So I'd leave the final decision\n> for that tradeoff to you. I'm just injecting some thoughts and numbers. :)\n\nIt's nice to see others duplicate the results, and I appreciate the\nsanity check on overall usefulness. If it were just optimizing\nrebase, I probably could have quit long ago when I had 15s rebases of\n35 patches. It was part stubbornness on my part wondering why it\ncouldn't be sub-second, and part knowing of other usecases that I\nwanted to make attractive to others. I want --remerge-diff to be\nuseful *and* practical. I want rebasing/cherry-picking un-checked-out\nbranches to be useful and practical. And I'd like to entice server\noperators (GitHub, GitLab, etc.) to use the same merge machinery used\nby end users so that reported merge-ability matches what end users see\nwhen they try it themselves.\n\nI think you make a good suggestion to drop the USE_MEMORY_POOL switch.\nI think I'll do it as an additional patch at the end of the series,\njust so it's easy for me to restore if by change it's ever needed.\n"},{"id":"431524","messageId":"xmqq4kcc4zjv.fsf@gitster.g","threadId":"56152","inReplyTo":"xmqq1r7g6gyg.fsf@gitster.g","subject":"Re: [PATCH v2 0/7] Final optimization batch (#15): use memory pools","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-29T20:48:04Z","receivedAt":"2021-07-29T20:48:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Jeff King writes:\n>\n>> On Thu, Jul 29, 2021 at 12:20:13PM -0400, Jeff King wrote:\n>>\n>>> I assume your tests are just done using the regular glibc allocator. I\n>>> also wondered how plugging in a better allocator might fare. Here are\n>>> timings I did of your mega-renames case with three binaries: one built\n>>> with USE_MEMORY_POOL set to 0, one with it set to 1, and one with it set\n>>> to 0 but adding \"-ltcmalloc\" to EXTLIBS via config.mak.\n>>\n>> Oh, btw, I wasn't able to apply your series from the list on top of\n>> en/ort-perf-batch-14 (there are some problems in patch 4, and \"am -3\"\n>> says my clone of git.git is missing some of the pre-image sha1s). I\n>> fetched ort-perf-batch-15 from https://github.com/newren/git and timed\n>> that, which I imagine is the same. But you may need to tweak the patches\n>> so that Junio can pick them up.\n>\n> Thanks, but the batch #15 has been in 'seen' since 23rd ;-)\n\nOh, that is the previous round. I haven't had the chance to pick up\nthis new round.\n\n"},{"id":"431527","messageId":"CABPp-BFRqPVR42c4oESnWEHHPDky+cW4Re8Pptu1yQxXttQF-g@mail.gmail.com","threadId":"56152","inReplyTo":"xmqq4kcc4zjv.fsf@gitster.g","subject":"Re: [PATCH v2 0/7] Final optimization batch (#15): use memory pools","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-29T21:05:21Z","receivedAt":"2021-07-29T21:05:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jul 29, 2021 at 2:48 PM Junio C Hamano wrote:\n>\n> Junio C Hamano writes:\n>\n> > Jeff King writes:\n> >\n> >> On Thu, Jul 29, 2021 at 12:20:13PM -0400, Jeff King wrote:\n> >>\n> >>> I assume your tests are just done using the regular glibc allocator. I\n> >>> also wondered how plugging in a better allocator might fare. Here are\n> >>> timings I did of your mega-renames case with three binaries: one built\n> >>> with USE_MEMORY_POOL set to 0, one with it set to 1, and one with it set\n> >>> to 0 but adding \"-ltcmalloc\" to EXTLIBS via config.mak.\n> >>\n> >> Oh, btw, I wasn't able to apply your series from the list on top of\n> >> en/ort-perf-batch-14 (there are some problems in patch 4, and \"am -3\"\n> >> says my clone of git.git is missing some of the pre-image sha1s). I\n> >> fetched ort-perf-batch-15 from https://github.com/newren/git and timed\n> >> that, which I imagine is the same. But you may need to tweak the patches\n> >> so that Junio can pick them up.\n> >\n> > Thanks, but the batch #15 has been in 'seen' since 23rd ;-)\n>\n> Oh, that is the previous round. I haven't had the chance to pick up\n> this new round.\n\nOh, interesting. At some point I had noticed that Junio based\nen/ort-perf-batch-14 on top of a version of master that did not\ninclude en/ort-perf-batch-12. While that was fine from a correctness\npoint of view, it made my claims of speedups a bit weird and difficult\nfor others to reproduce as they'd need to merge some other series\nfirst. It looks like the base for en/ort-perf-batch-14 at some point\nadvanced to include a version of master that included\nen/ort-perf-batch-12.\n\nAnyway, I'm happy to re-roll this series and base it on\nen/ort-perf-batch-14. Peff has a few suggested improvements for me to\ninclude in that re-roll.\n"},{"id":"431529","messageId":"YQMaTTjgTB3wD8wY@coredump.intra.peff.net","threadId":"56152","inReplyTo":"CABPp-BH2PQ_rHDPNBghdkhO=kaNfQBnU28UFh9c-sPBNUOM4gA@mail.gmail.com","subject":"Re: [PATCH v2 0/7] Final optimization batch (#15): use memory pools","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-29T21:14:53Z","receivedAt":"2021-07-29T21:14:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 29, 2021 at 02:46:03PM -0600, Elijah Newren wrote:\n\n> You make fair points about the absolute timings for rebase, but the\n> \"grand scheme of things\" involves usecases outside of traditional\n> rebases. Some of those involve far more merges, making the relative\n> timings more important. They also involve much less overhead -- not\n> only do they get to ignore the \"git checkout\" present in most rebases,\n> but they also get to exclude the index or worktree updating that are\n> present above. Without that extra overhead, the relative improvement\n> from this patch is even greater. One particular example usecase that\n> is coming soon is the introduction of `git log --remerge-diff`; it\n> makes the timing of individual merges critical since it does so many\n> of them. And it becomes even more important for users who change the\n> default from --diff-merges=off to --diff-merges=remerge-diff, because\n> then any `git log -p` will potentially remerge hundreds or thousands\n> of merge commits. (I've got lots of users who have -p imply\n> --remerge-diff since last November.)\n\nOoh, I hadn't considered doing a bunch of fast in-memory merges for\n--remerge-diff. That is a very compelling use case, I agree.\n\n> > Benchmark #1: ./test-tool.nopool fast-rebase --onto HEAD base hwmon-updates\n> > Time (mean ± σ): 921.1 ms ± 146.0 ms [User: 843.0 ms, System: 77.5 ms]\n> > Range (min … max): 660.9 ms … 1112.2 ms 10 runs\n> >\n> > Benchmark #2: ./test-tool.pool fast-rebase --onto HEAD base hwmon-updates\n> > Time (mean ± σ): 635.4 ms ± 125.5 ms [User: 563.7 ms, System: 71.3 ms]\n> > Range (min … max): 496.8 ms … 856.7 ms 10 runs\n> >\n> > Benchmark #3: ./test-tool.tcmalloc fast-rebase --onto HEAD base hwmon-updates\n> > Time (mean ± σ): 727.3 ms ± 139.9 ms [User: 654.1 ms, System: 72.9 ms]\n> > Range (min … max): 476.3 ms … 900.5 ms 10 runs\n> \n> That's some _really_ wide variance on your runs, making me wonder if\n> you are running other things on your (I presume) laptop that are\n> potentially muddying the numbers. Would the tcmalloc case actually\n> have the fastest run in general, or was it just lucky to hit a \"quiet\"\n> moment on the laptop?\n\nYeah, I noticed that, too. The system was otherwise unloaded, but there\nI think a big part of it was that my prepare commands flipped back and forth\nbetween the pre-/post-rename states twice. Even though that isn't\nincluded in the timings, I think it was just creating a lot of delayed\nwork for the OS to do when it decided to write back all those inodes.\n\nSwitching it to:\n\n hyperfine \\\n -p 'git checkout 5.4-renames^0 &&\n git branch -f hwmon-updates fd8bdb23b91876ac1e624337bb88dc1dcc21d67e' \\\n -L version nopool,pool,tcmalloc \\\n './test-tool.{version} fast-rebase --onto HEAD base hwmon-updates'\n\nproduces much smoother results:\n\n Benchmark #1: ./test-tool.nopool fast-rebase --onto HEAD base hwmon-updates\n Time (mean ± σ): 649.7 ms ± 5.0 ms [User: 595.1 ms, System: 54.3 ms]\n Range (min … max): 643.7 ms … 661.9 ms 10 runs\n \n Benchmark #2: ./test-tool.pool fast-rebase --onto HEAD base hwmon-updates\n Time (mean ± σ): 405.0 ms ± 3.0 ms [User: 354.9 ms, System: 50.0 ms]\n Range (min … max): 401.0 ms … 411.9 ms 10 runs\n \n Benchmark #3: ./test-tool.tcmalloc fast-rebase --onto HEAD base hwmon-updates\n Time (mean ± σ): 476.7 ms ± 3.9 ms [User: 430.1 ms, System: 46.4 ms]\n Range (min … max): 472.3 ms … 484.1 ms 10 runs\n \n Summary\n './test-tool.pool fast-rebase --onto HEAD base hwmon-updates' ran\n 1.18 ± 0.01 times faster than './test-tool.tcmalloc fast-rebase --onto HEAD base hwmon-updates'\n 1.60 ± 0.02 times faster than './test-tool.nopool fast-rebase --onto HEAD base hwmon-updates'\n\nSo the pool is definitively faster, though we can go a fair ways by\nusing a better allocator. :)\n\n> I did similar testing a year ago, before I even looked at memory\n> pools. I was surprised by how big a speedup I saw, and considered\n> asking on the list if we could push to use a different allocator by\n> default. Ultimately, I figured that probably wouldn't fly and\n> distributors might override our choices anyway. It was at that point\n> that I decided to start tweaking mem-pool.[ch] (which ended up getting\n> merged at edab8a8d07 (\"Merge branch 'en/mem-pool'\", 2020-08-27)), and\n> then integrating that into strmap/strset/strintmap -- all in an effort\n> to guarantee that we realized the speedups that I knew were possible\n> due to my testing with the special allocators.\n\nYeah, I think choice of allocator should be outside the scope of Git. It\nwould be a packaging issue if people want to squeeze out every bit. I do\nagree there's something to be said for Git just handling this itself,\nregardless of the platform or build. I'm just always on the lookout for\nplaces where we can get free speedups without having to pay any\nmaintenance cost for them. :)\n\n> I think you make a good suggestion to drop the USE_MEMORY_POOL switch.\n> I think I'll do it as an additional patch at the end of the series,\n> just so it's easy for me to restore if by change it's ever needed.\n\nYeah, between that cleanup reducing the maintenance cost and your\ncompelling use cases above, I think I'm convinced that this is a good\ndirection.\n\n-Peff\n"},{"id":"431538","messageId":"CABPp-BF+1=yajJ_Z9MHHbeL6XkctfStj63xVXTkguNicx_PcPw@mail.gmail.com","threadId":"56152","inReplyTo":"YQLInKdjor6Vs1yE@coredump.intra.peff.net","subject":"Re: [PATCH 3/7] merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-30T02:27:51Z","receivedAt":"2021-07-30T02:28:05Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jul 29, 2021 at 9:26 AM Jeff King wrote:\n>\n> On Wed, Jul 28, 2021 at 04:49:18PM -0600, Elijah Newren wrote:\n>\n> > On Mon, Jul 26, 2021 at 8:36 AM Derrick Stolee wrote:\n> > >\n> > > On 7/23/2021 8:54 AM, Elijah Newren via GitGitGadget wrote:\n> > > > From: Elijah Newren \n> > > >\n> > > > We need functions which will either call\n> > > > xmalloc, xcalloc, xstrndup\n> > > > or\n> > > > mem_pool_alloc, mem_pool_calloc, mem_pool_strndup\n> > > > depending on whether we have a non-NULL memory pool. Add these\n> > > > functions; the next commit will make use of these.\n> > >\n> > > I briefly considered that this should just be the way the\n> > > mem_pool_* methods work. It does rely on the caller knowing\n> > > to free() the allocated memory when their pool is NULL, so\n> > > perhaps such a universal change might be too much. What do\n> > > you think?\n> >\n> > That's interesting, but I'm worried it might be a bit much. Do others\n> > on the list have an opinion here?\n>\n> FWIW, I had the same thought. You can also provide a helper to make the\n> freeing side nicer:\n>\n> static void mem_pool_free(struct mem_pool *m, void *ptr)\n> {\n> if (m)\n> return; /* will be freed when pool frees */\n> free(ptr);\n> }\n>\n> We do something similar with unuse_commit_buffer(), where the caller\n> isn't aware of we pulled the buffer from cache or allocated it\n> especially for them.\n\nHaving a paired function may help one side, but I worry that the name\n(mem_pool_free) might introduce some confusion of its own -- \"Why is\nthere a mem_pool_free() function, isn't the point of memory pools to\nnot need to individually free things?\" Or, \"Why are they freeing the\npool here and what's the extra parameter?\"\n\nI'm not sure I see the right way to address that, so I think I'm going\nto leave this part out of my series and let someone else add such\nchanges on top if they feel motivated to do so.\n"},{"id":"431539","messageId":"CABPp-BE8OEjcMrJO8uijLFsvvpQH9KA=a+p_6bH71Z6QxXLB6w@mail.gmail.com","threadId":"56152","inReplyTo":"YQMLBW6LAZmQ8nDz@coredump.intra.peff.net","subject":"Re: [PATCH v2 4/7] merge-ort: switch our strmaps over to using memory pools","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-30T02:30:23Z","receivedAt":"2021-07-30T02:30:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jul 29, 2021 at 2:09 PM Jeff King wrote:\n>\n> On Thu, Jul 29, 2021 at 12:37:52PM -0600, Elijah Newren wrote:\n>\n> > > Arguably, the existence of these function indirections is perhaps a sign\n> > > that the strmap API should provide a version of the clear functions that\n> > > takes \"partial / not-partial\" as a parameter.\n> >\n> > Are you suggesting a modification of str{map,intmap,set}_clear() to\n> > take an extra parameter, or removing the\n> > str{map,intmap,set}_partial_clear() functions and introducing new\n> > functions that take a partial/not-partial parameter? I think you're\n> > suggesting the latter, and that makes more sense to me...but I'm\n> > drawing blanks trying to come up with a reasonable function name.\n>\n> It does seem a shame to add the \"partial\" parameter to strmap_clear(),\n> just because most callers don't need it (so they end up with this\n> inscrutable \"0\" parameter).\n>\n> What if there was a flags field? Then it could be combined with the\n> free_values parameter. The result is kind of verbose in two ways:\n>\n> - now strset_clear(), etc, need a \"flags\" parameter, which they didn't\n> before (and is just \"0\" most of the time!)\n>\n> - now \"strmap_clear(foo, 1)\" becomes \"strmap_clear(foo, STRMAP_FREE_VALUES)\".\n> That's a lot longer, though arguably it's easier to understand since\n> the boolean is explained.\n>\n> Having gone through the exercise, I am not sure it is actually making\n> anything more readable (messy patch is below for reference).\n\nThanks for diving in. Since it's not clear if it's helping, I'll just\ntake your earlier suggestion to rename the \"strmap_func\" variable to\n\"strmap_clear_func\" instead.\n\n>\n> diff --git a/builtin/shortlog.c b/builtin/shortlog.c\n> index 3e7ab1ca82..dfbdba53da 100644\n> --- a/builtin/shortlog.c\n> +++ b/builtin/shortlog.c\n> @@ -242,7 +242,7 @@ void shortlog_add_commit(struct shortlog *log, struct commit *commit)\n> insert_records_from_trailers(log, &dups, commit, &ctx, oneline_str);\n> }\n>\n> - strset_clear(&dups);\n> + strset_clear(&dups, 0);\n> strbuf_release(&ident);\n> strbuf_release(&oneline);\n> }\n> diff --git a/diffcore-rename.c b/diffcore-rename.c\n> index 7e6b3e1b14..0c960111d1 100644\n> --- a/diffcore-rename.c\n> +++ b/diffcore-rename.c\n> @@ -665,9 +665,10 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n>\n> strmap_for_each_entry(dir_rename_count, &iter, entry) {\n> struct strintmap *counts = entry->value;\n> - strintmap_clear(counts);\n> + strintmap_clear(counts, 0);\n> }\n> - strmap_partial_clear(dir_rename_count, 1);\n> + strmap_clear(dir_rename_count,\n> + STRMAP_FREE_VALUES | STRMAP_PARTIAL_CLEAR);\n> }\n>\n> static void cleanup_dir_rename_info(struct dir_rename_info *info,\n> @@ -683,15 +684,15 @@ static void cleanup_dir_rename_info(struct dir_rename_info *info,\n> return;\n>\n> /* idx_map */\n> - strintmap_clear(&info->idx_map);\n> + strintmap_clear(&info->idx_map, 0);\n>\n> /* dir_rename_guess */\n> - strmap_clear(&info->dir_rename_guess, 1);\n> + strmap_clear(&info->dir_rename_guess, STRMAP_FREE_VALUES);\n>\n> /* relevant_source_dirs */\n> if (info->relevant_source_dirs &&\n> info->relevant_source_dirs != dirs_removed) {\n> - strintmap_clear(info->relevant_source_dirs);\n> + strintmap_clear(info->relevant_source_dirs, 0);\n> FREE_AND_NULL(info->relevant_source_dirs);\n> }\n>\n> @@ -716,7 +717,7 @@ static void cleanup_dir_rename_info(struct dir_rename_info *info,\n>\n> if (!strintmap_get(dirs_removed, source_dir)) {\n> string_list_append(&to_remove, source_dir);\n> - strintmap_clear(counts);\n> + strintmap_clear(counts, 0);\n> continue;\n> }\n>\n> @@ -1045,8 +1046,8 @@ static int find_basename_matches(struct diff_options *options,\n> }\n> }\n>\n> - strintmap_clear(&sources);\n> - strintmap_clear(&dests);\n> + strintmap_clear(&sources, 0);\n> + strintmap_clear(&dests, 0);\n>\n> return renames;\n> }\n> @@ -1700,7 +1701,7 @@ void diffcore_rename_extended(struct diff_options *options,\n> FREE_AND_NULL(rename_src);\n> rename_src_nr = rename_src_alloc = 0;\n> if (break_idx) {\n> - strintmap_clear(break_idx);\n> + strintmap_clear(break_idx, 0);\n> FREE_AND_NULL(break_idx);\n> }\n> trace2_region_leave(\"diff\", \"write back to queue\", options->repo);\n> diff --git a/merge-ort.c b/merge-ort.c\n> index 0fb942692a..0765e23577 100644\n> --- a/merge-ort.c\n> +++ b/merge-ort.c\n> @@ -532,15 +532,10 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n> {\n> struct rename_info *renames = &opti->renames;\n> int i;\n> - void (*strmap_func)(struct strmap *, int) =\n> - reinitialize ? strmap_partial_clear : strmap_clear;\n> - void (*strintmap_func)(struct strintmap *) =\n> - reinitialize ? strintmap_partial_clear : strintmap_clear;\n> - void (*strset_func)(struct strset *) =\n> - reinitialize ? strset_partial_clear : strset_clear;\n> + unsigned flags = reinitialize ? STRMAP_PARTIAL_CLEAR : 0;\n>\n> if (opti->pool)\n> - strmap_func(&opti->paths, 0);\n> + strmap_clear(&opti->paths, flags);\n> else {\n> /*\n> * We marked opti->paths with strdup_strings = 0, so that\n> @@ -550,15 +545,15 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n> * to these strings, it is time to deallocate them.\n> */\n> free_strmap_strings(&opti->paths);\n> - strmap_func(&opti->paths, 1);\n> + strmap_clear(&opti->paths, flags | STRMAP_FREE_VALUES);\n> }\n>\n> /*\n> * All keys and values in opti->conflicted are a subset of those in\n> * opti->paths. We don't want to deallocate anything twice, so we\n> * don't free the keys and we pass 0 for free_values.\n> */\n> - strmap_func(&opti->conflicted, 0);\n> + strmap_clear(&opti->conflicted, flags);\n>\n> if (!opti->pool) {\n> /*\n> @@ -579,24 +574,24 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n>\n> /* Free memory used by various renames maps */\n> for (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n> - strintmap_func(&renames->dirs_removed[i]);\n> - strmap_func(&renames->dir_renames[i], 0);\n> - strintmap_func(&renames->relevant_sources[i]);\n> + strintmap_clear(&renames->dirs_removed[i], flags);\n> + strmap_clear(&renames->dir_renames[i], flags);\n> + strintmap_clear(&renames->relevant_sources[i], flags);\n> if (!reinitialize)\n> assert(renames->cached_pairs_valid_side == 0);\n> if (i != renames->cached_pairs_valid_side &&\n> -1 != renames->cached_pairs_valid_side) {\n> - strset_func(&renames->cached_target_names[i]);\n> - strmap_func(&renames->cached_pairs[i], 1);\n> - strset_func(&renames->cached_irrelevant[i]);\n> + strset_clear(&renames->cached_target_names[i], flags);\n> + strmap_clear(&renames->cached_pairs[i], flags | STRMAP_FREE_VALUES);\n> + strset_clear(&renames->cached_irrelevant[i], flags);\n> partial_clear_dir_rename_count(&renames->dir_rename_count[i]);\n> if (!reinitialize)\n> strmap_clear(&renames->dir_rename_count[i], 1);\n> }\n> }\n> for (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n> - strintmap_func(&renames->deferred[i].possible_trivial_merges);\n> - strset_func(&renames->deferred[i].target_dirs);\n> + strintmap_clear(&renames->deferred[i].possible_trivial_merges, flags);\n> + strset_clear(&renames->deferred[i].target_dirs, flags);\n> renames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */\n> }\n> renames->cached_pairs_valid_side = 0;\n> @@ -1482,7 +1477,7 @@ static int handle_deferred_entries(struct merge_options *opt,\n> if (ret < 0)\n> return ret;\n> }\n> - strintmap_clear(©);\n> + strintmap_clear(©, 0);\n> strintmap_for_each_entry(&renames->deferred[side].possible_trivial_merges,\n> &iter, entry) {\n> const char *path = entry->key;\n> diff --git a/strmap.c b/strmap.c\n> index 4fb9f6100e..7343800df5 100644\n> --- a/strmap.c\n> +++ b/strmap.c\n> @@ -37,10 +37,11 @@ void strmap_init_with_options(struct strmap *map,\n> map->strdup_strings = strdup_strings;\n> }\n>\n> -static void strmap_free_entries_(struct strmap *map, int free_values)\n> +static void strmap_free_entries_(struct strmap *map, unsigned flags)\n> {\n> struct hashmap_iter iter;\n> struct strmap_entry *e;\n> + int free_values = flags & STRMAP_FREE_VALUES;\n>\n> if (!map)\n> return;\n> @@ -64,16 +65,13 @@ static void strmap_free_entries_(struct strmap *map, int free_values)\n> }\n> }\n>\n> -void strmap_clear(struct strmap *map, int free_values)\n> +void strmap_clear(struct strmap *map, unsigned flags)\n> {\n> - strmap_free_entries_(map, free_values);\n> - hashmap_clear(&map->map);\n> -}\n> -\n> -void strmap_partial_clear(struct strmap *map, int free_values)\n> -{\n> - strmap_free_entries_(map, free_values);\n> - hashmap_partial_clear(&map->map);\n> + strmap_free_entries_(map, flags);\n> + if (flags & STRMAP_PARTIAL_CLEAR)\n> + hashmap_partial_clear(&map->map);\n> + else\n> + hashmap_clear(&map->map);\n> }\n>\n> static struct strmap_entry *create_entry(struct strmap *map,\n> diff --git a/strmap.h b/strmap.h\n> index 1e152d832d..d03d451654 100644\n> --- a/strmap.h\n> +++ b/strmap.h\n> @@ -46,16 +46,14 @@ void strmap_init_with_options(struct strmap *map,\n> struct mem_pool *pool,\n> int strdup_strings);\n>\n> -/*\n> - * Remove all entries from the map, releasing any allocated resources.\n> - */\n> -void strmap_clear(struct strmap *map, int free_values);\n> +#define STRMAP_FREE_VALUES 1 /* 1 for historical compat, but we should probably\n> + update callers to use the correct name) */\n> +#define STRMAP_PARTIAL_CLEAR 2\n>\n> /*\n> - * Similar to strmap_clear() but leaves map->map->table allocated and\n> - * pre-sized so that subsequent uses won't need as many rehashings.\n> + * Remove all entries from the map, releasing any allocated resources.\n> */\n> -void strmap_partial_clear(struct strmap *map, int free_values);\n> +void strmap_clear(struct strmap *map, unsigned flags);\n>\n> /*\n> * Insert \"str\" into the map, pointing to \"data\".\n> @@ -148,14 +146,10 @@ static inline void strintmap_init_with_options(struct strintmap *map,\n> map->default_value = default_value;\n> }\n>\n> -static inline void strintmap_clear(struct strintmap *map)\n> -{\n> - strmap_clear(&map->map, 0);\n> -}\n> -\n> -static inline void strintmap_partial_clear(struct strintmap *map)\n> +static inline void strintmap_clear(struct strintmap *map, unsigned flags)\n> {\n> - strmap_partial_clear(&map->map, 0);\n> + /* maybe clear STRMAP_FREE_VALUES bit for extra protection */\n> + strmap_clear(&map->map, flags);\n> }\n>\n> static inline int strintmap_contains(struct strintmap *map, const char *str)\n> @@ -232,14 +226,9 @@ static inline void strset_init_with_options(struct strset *set,\n> strmap_init_with_options(&set->map, pool, strdup_strings);\n> }\n>\n> -static inline void strset_clear(struct strset *set)\n> -{\n> - strmap_clear(&set->map, 0);\n> -}\n> -\n> -static inline void strset_partial_clear(struct strset *set)\n> +static inline void strset_clear(struct strset *set, unsigned flags)\n> {\n> - strmap_partial_clear(&set->map, 0);\n> + strmap_clear(&set->map, flags);\n> }\n>\n> static inline int strset_contains(struct strset *set, const char *str)\n"},{"id":"431552","messageId":"pull.990.v3.git.1627645664.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v2.git.1627531121.gitgitgadget@gmail.com","subject":"[PATCH v3 0/9] Final optimization batch (#15): use memory pools","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-30T11:47:35Z","receivedAt":"2021-07-30T11:47:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series textually depends on en/ort-perf-batch-14, but the ideas are\northogonal to it and orthogonal to previous series. It can be reviewed\nindependently.\n\nChanges since v1, addressing Eric's feedback:\n\n * Fixed a comment that became out-of-date in patch 1\n * Swapped commits 2 and 3 so that one can better motivate the other.\n\nChanges since v2, addressing Peff's feedback\n\n * Rebased on en/ort-perf-batch-14 (resolving a trivial conflict with the\n new string_list_init_nodup() usage)\n * Added a new preliminary patch renaming str*_func() to str*_clear_func()\n * Added a new final patch that hardcodes that we'll just use memory pools\n\n=== Basic Optimization idea ===\n\nIn this series, I make use of memory pools to get faster allocations and\ndeallocations for many data structures that tend to all be deallocated at\nthe same time anyway.\n\n=== Results ===\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28), the\nchanges in just this series improves the performance as follows:\n\n Before Series After Series\nno-renames: 204.2 ms ± 3.0 ms 198.3 ms ± 2.9 ms\nmega-renames: 1.076 s ± 0.015 s 661.8 ms ± 5.9 ms\njust-one-mega: 364.1 ms ± 7.0 ms 264.6 ms ± 2.5 ms\n\n\nAs a reminder, before any merge-ort/diffcore-rename performance work, the\nperformance results we started with were:\n\nno-renames-am: 6.940 s ± 0.485 s\nno-renames: 18.912 s ± 0.174 s\nmega-renames: 5964.031 s ± 10.459 s\njust-one-mega: 149.583 s ± 0.751 s\n\n\n=== Overall Results across all optimization work ===\n\nThis is my final prepared optimization series. It might be worth reviewing\nhow my optimizations fared overall, comparing the original merge-recursive\ntimings with three things: how much merge-recursive improved (as a\nside-effect of optimizing merge-ort), how much improvement we would have\ngotten from a hypothetical infinite parallelization of rename detection, and\nwhat I achieved at the end with merge-ort:\n\n Timings\n\n Infinite\n merge- merge- Parallelism\n recursive recursive of rename merge-ort\n v2.30.0 current detection current\n ---------- --------- ----------- ---------\nno-renames: 18.912 s 18.030 s 11.699 s 198.3 ms\nmega-renames: 5964.031 s 361.281 s 203.886 s 661.8 ms\njust-one-mega: 149.583 s 11.009 s 7.553 s 264.6 ms\n\n Speedup factors\n\n Infinite\n merge- merge- Parallelism\n recursive recursive of rename\n v2.30.0 current detection merge-ort\n ---------- --------- ----------- ---------\nno-renames: 1 1.05 1.6 95\nmega-renames: 1 16.5 29 9012\njust-one-mega: 1 13.6 20 565\n\n\nAnd, for partial clone users:\n\n Factor reduction in number of objects needed\n\n Infinite\n merge- merge- Parallelism\n recursive recursive of rename\n v2.30.0 current detection merge-ort\n ---------- --------- ----------- ---------\nmega-renames: 1 1 1 181.3\n\n\n=== Caveat ===\n\nIt may be worth noting, though, that my optimization numbers above for\nmerge-ort use test-tool fast-rebase. git rebase -s ort on the three\ntestcases above is 5-20 times slower (taking 3.835s, 6.798s, and 1.235s,\nrespectively). At this point, any further optimization work should go into\nmaking a faster full-featured rebase by copying the ideas from fast-rebase:\navoid unnecessary process forking, avoid updating the index and working copy\nuntil either the rebase is finished or you hit a conflict (and don't write\nrebase metadata to disk until that point either), get rid of the glacially\nslow revision walking of the upstream side of history (nuke\ncan_fast_forward(), make --reapply-cherry-picks the default) or at least\ndon't revision walk so many times (multiple calls to get_merge_bases in\ncan_fast_forward() plus a is_linear_history() walk, checking for upstream\ncherry-picks, probably more), turn off per-commit hooks that probably should\nhave never been on anyway, etc.\n\nElijah Newren (9):\n merge-ort: rename str{map,intmap,set}_func()\n diffcore-rename: use a mem_pool for exact rename detection's hashmap\n merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers\n merge-ort: set up a memory pool\n merge-ort: switch our strmaps over to using memory pools\n diffcore-rename, merge-ort: add wrapper functions for filepair\n alloc/dealloc\n merge-ort: store filepairs and filespecs in our mem_pool\n merge-ort: reuse path strings in pool_alloc_filespec\n merge-ort: remove compile-time ability to turn off usage of memory\n pools\n\n diffcore-rename.c | 68 +++++++++++--\n diffcore.h | 3 +\n merge-ort.c | 238 ++++++++++++++++++++++++++++++++--------------\n 3 files changed, 231 insertions(+), 78 deletions(-)\n\n\nbase-commit: 8b09a900a1f1f00d4deb04f567994ae8f1804b5e\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-990%2Fnewren%2Fort-perf-batch-15-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-990/newren/ort-perf-batch-15-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/990\n\nRange-diff vs v2:\n\n -: ----------- > 1: e075d985f26 merge-ort: rename str{map,intmap,set}_func()\n 1: ea08b34d29b = 2: 8416afa89fb diffcore-rename: use a mem_pool for exact rename detection's hashmap\n 2: fdfc2b93ba4 ! 3: 2c0b90eaba5 merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers\n @@ Metadata\n ## Commit message ##\n merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers\n \n - We need functions which will either call\n + Make the code more flexible so that it can handle both being run with or\n + without a memory pool by adding utility functions which will either call\n xmalloc, xcalloc, xstrndup\n or\n mem_pool_alloc, mem_pool_calloc, mem_pool_strndup\n - depending on whether we have a non-NULL memory pool. Add these\n - functions; a subsequent commit will make use of these.\n + depending on whether we have a non-NULL memory pool. A subsequent\n + commit will make use of these.\n +\n + (We will actually be dropping these functions soon and just assuming we\n + always have a memory pool, but the flexibility was very useful during\n + development of merge-ort so I want to be able to restore it if needed.)\n \n Signed-off-by: Elijah Newren \n \n 3: c7150869107 = 4: 6646f6fd1ca merge-ort: set up a memory pool\n 4: dd8839b2843 ! 5: 7c49aa601d0 merge-ort: switch our strmaps over to using memory pools\n @@ Commit message\n \n ## merge-ort.c ##\n @@ merge-ort.c: static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n - \tvoid (*strset_func)(struct strset *) =\n + \tvoid (*strset_clear_func)(struct strset *) =\n \t\treinitialize ? strset_partial_clear : strset_clear;\n \n -\t/*\n @@ merge-ort.c: static void clear_or_reinit_internal_opts(struct merge_options_inte\n -\t * to deallocate them.\n -\t */\n -\tfree_strmap_strings(&opti->paths);\n --\tstrmap_func(&opti->paths, 1);\n +-\tstrmap_clear_func(&opti->paths, 1);\n +\tif (opti->pool)\n -+\t\tstrmap_func(&opti->paths, 0);\n ++\t\tstrmap_clear_func(&opti->paths, 0);\n +\telse {\n +\t\t/*\n +\t\t * We marked opti->paths with strdup_strings = 0, so that\n @@ merge-ort.c: static void clear_or_reinit_internal_opts(struct merge_options_inte\n +\t\t * to these strings, it is time to deallocate them.\n +\t\t */\n +\t\tfree_strmap_strings(&opti->paths);\n -+\t\tstrmap_func(&opti->paths, 1);\n ++\t\tstrmap_clear_func(&opti->paths, 1);\n +\t}\n \n \t/*\n \t * All keys and values in opti->conflicted are a subset of those in\n @@ merge-ort.c: static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t */\n - \tstrmap_func(&opti->conflicted, 0);\n + \tstrmap_clear_func(&opti->conflicted, 0);\n \n -\t/*\n -\t * opti->paths_to_free is similar to opti->paths; we created it with\n @@ merge-ort.c: static void merge_start(struct merge_options *opt, struct merge_res\n \t */\n -\tstrmap_init_with_options(&opt->priv->paths, NULL, 0);\n -\tstrmap_init_with_options(&opt->priv->conflicted, NULL, 0);\n --\tstring_list_init(&opt->priv->paths_to_free, 0);\n +-\tstring_list_init_nodup(&opt->priv->paths_to_free);\n +\tstrmap_init_with_options(&opt->priv->paths, pool, 0);\n +\tstrmap_init_with_options(&opt->priv->conflicted, pool, 0);\n +\tif (!opt->priv->pool)\n -+\t\tstring_list_init(&opt->priv->paths_to_free, 0);\n ++\t\tstring_list_init_nodup(&opt->priv->paths_to_free);\n \n \t/*\n \t * keys & strbufs in output will sometimes need to outlive \"paths\",\n 5: 560800a80ef = 6: 08cf2498f96 diffcore-rename, merge-ort: add wrapper functions for filepair alloc/dealloc\n 6: 94d60c8a476 = 7: 4ffa5af8b57 merge-ort: store filepairs and filespecs in our mem_pool\n 7: fda885dabe6 = 8: 1556f0443c3 merge-ort: reuse path strings in pool_alloc_filespec\n -: ----------- > 9: de30dbac25e merge-ort: remove compile-time ability to turn off usage of memory pools\n\n-- \ngitgitgadget\n"},{"id":"431553","messageId":"e075d985f269cdbf1fc67c99584b85d07538d603.1627645664.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v3.git.1627645664.gitgitgadget@gmail.com","subject":"[PATCH v3 1/9] merge-ort: rename str{map,intmap,set}_func()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-30T11:47:36Z","receivedAt":"2021-07-30T11:47:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIn order to make it clearer that these three variables holding a\nfunction refer to functions that will clear the strmap/strintmap/strset,\nrename them to str{map,intmap,set}_clear_func().\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 26 +++++++++++++-------------\n 1 file changed, 13 insertions(+), 13 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex e75b524153e..401a40247a3 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -519,11 +519,11 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n {\n \tstruct rename_info *renames = &opti->renames;\n \tint i;\n-\tvoid (*strmap_func)(struct strmap *, int) =\n+\tvoid (*strmap_clear_func)(struct strmap *, int) =\n \t\treinitialize ? strmap_partial_clear : strmap_clear;\n-\tvoid (*strintmap_func)(struct strintmap *) =\n+\tvoid (*strintmap_clear_func)(struct strintmap *) =\n \t\treinitialize ? strintmap_partial_clear : strintmap_clear;\n-\tvoid (*strset_func)(struct strset *) =\n+\tvoid (*strset_clear_func)(struct strset *) =\n \t\treinitialize ? strset_partial_clear : strset_clear;\n \n \t/*\n@@ -534,14 +534,14 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t * to deallocate them.\n \t */\n \tfree_strmap_strings(&opti->paths);\n-\tstrmap_func(&opti->paths, 1);\n+\tstrmap_clear_func(&opti->paths, 1);\n \n \t/*\n \t * All keys and values in opti->conflicted are a subset of those in\n \t * opti->paths. We don't want to deallocate anything twice, so we\n \t * don't free the keys and we pass 0 for free_values.\n \t */\n-\tstrmap_func(&opti->conflicted, 0);\n+\tstrmap_clear_func(&opti->conflicted, 0);\n \n \t/*\n \t * opti->paths_to_free is similar to opti->paths; we created it with\n@@ -559,24 +559,24 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \n \t/* Free memory used by various renames maps */\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n-\t\tstrintmap_func(&renames->dirs_removed[i]);\n-\t\tstrmap_func(&renames->dir_renames[i], 0);\n-\t\tstrintmap_func(&renames->relevant_sources[i]);\n+\t\tstrintmap_clear_func(&renames->dirs_removed[i]);\n+\t\tstrmap_clear_func(&renames->dir_renames[i], 0);\n+\t\tstrintmap_clear_func(&renames->relevant_sources[i]);\n \t\tif (!reinitialize)\n \t\t\tassert(renames->cached_pairs_valid_side == 0);\n \t\tif (i != renames->cached_pairs_valid_side &&\n \t\t -1 != renames->cached_pairs_valid_side) {\n-\t\t\tstrset_func(&renames->cached_target_names[i]);\n-\t\t\tstrmap_func(&renames->cached_pairs[i], 1);\n-\t\t\tstrset_func(&renames->cached_irrelevant[i]);\n+\t\t\tstrset_clear_func(&renames->cached_target_names[i]);\n+\t\t\tstrmap_clear_func(&renames->cached_pairs[i], 1);\n+\t\t\tstrset_clear_func(&renames->cached_irrelevant[i]);\n \t\t\tpartial_clear_dir_rename_count(&renames->dir_rename_count[i]);\n \t\t\tif (!reinitialize)\n \t\t\t\tstrmap_clear(&renames->dir_rename_count[i], 1);\n \t\t}\n \t}\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n-\t\tstrintmap_func(&renames->deferred[i].possible_trivial_merges);\n-\t\tstrset_func(&renames->deferred[i].target_dirs);\n+\t\tstrintmap_clear_func(&renames->deferred[i].possible_trivial_merges);\n+\t\tstrset_clear_func(&renames->deferred[i].target_dirs);\n \t\trenames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */\n \t}\n \trenames->cached_pairs_valid_side = 0;\n-- \ngitgitgadget\n\n"},{"id":"431554","messageId":"8416afa89fb9d1e638bf6f52403f113c82de7424.1627645664.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v3.git.1627645664.gitgitgadget@gmail.com","subject":"[PATCH v3 2/9] diffcore-rename: use a mem_pool for exact rename detection's hashmap","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-30T11:47:37Z","receivedAt":"2021-07-30T11:47:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nExact rename detection, via insert_file_table(), uses a hashmap to store\nfiles by oid. Use a mem_pool for the hashmap entries so these can all be\nallocated and deallocated together.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 204.2 ms ± 3.0 ms 202.5 ms ± 3.2 ms\n mega-renames: 1.076 s ± 0.015 s 1.072 s ± 0.012 s\n just-one-mega: 364.1 ms ± 7.0 ms 357.3 ms ± 3.9 ms\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 22 ++++++++++++++++------\n 1 file changed, 16 insertions(+), 6 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 4ef0459cfb5..73d884099eb 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -317,10 +317,11 @@ static int find_identical_files(struct hashmap *srcs,\n }\n \n static void insert_file_table(struct repository *r,\n+\t\t\t struct mem_pool *pool,\n \t\t\t struct hashmap *table, int index,\n \t\t\t struct diff_filespec *filespec)\n {\n-\tstruct file_similarity *entry = xmalloc(sizeof(*entry));\n+\tstruct file_similarity *entry = mem_pool_alloc(pool, sizeof(*entry));\n \n \tentry->index = index;\n \tentry->filespec = filespec;\n@@ -336,7 +337,8 @@ static void insert_file_table(struct repository *r,\n * and then during the second round we try to match\n * cache-dirty entries as well.\n */\n-static int find_exact_renames(struct diff_options *options)\n+static int find_exact_renames(struct diff_options *options,\n+\t\t\t struct mem_pool *pool)\n {\n \tint i, renames = 0;\n \tstruct hashmap file_table;\n@@ -346,7 +348,7 @@ static int find_exact_renames(struct diff_options *options)\n \t */\n \thashmap_init(&file_table, NULL, NULL, rename_src_nr);\n \tfor (i = rename_src_nr-1; i >= 0; i--)\n-\t\tinsert_file_table(options->repo,\n+\t\tinsert_file_table(options->repo, pool,\n \t\t\t\t &file_table, i,\n \t\t\t\t rename_src[i].p->one);\n \n@@ -354,8 +356,8 @@ static int find_exact_renames(struct diff_options *options)\n \tfor (i = 0; i < rename_dst_nr; i++)\n \t\trenames += find_identical_files(&file_table, i, options);\n \n-\t/* Free the hash data structure and entries */\n-\thashmap_clear_and_free(&file_table, struct file_similarity, entry);\n+\t/* Free the hash data structure (entries will be freed with the pool) */\n+\thashmap_clear(&file_table);\n \n \treturn renames;\n }\n@@ -1341,6 +1343,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \tint num_destinations, dst_cnt;\n \tint num_sources, want_copies;\n \tstruct progress *progress = NULL;\n+\tstruct mem_pool local_pool;\n \tstruct dir_rename_info info;\n \tstruct diff_populate_filespec_options dpf_options = {\n \t\t.check_binary = 0,\n@@ -1409,11 +1412,18 @@ void diffcore_rename_extended(struct diff_options *options,\n \t\tgoto cleanup; /* nothing to do */\n \n \ttrace2_region_enter(\"diff\", \"exact renames\", options->repo);\n+\tmem_pool_init(&local_pool, 32*1024);\n \t/*\n \t * We really want to cull the candidates list early\n \t * with cheap tests in order to avoid doing deltas.\n \t */\n-\trename_count = find_exact_renames(options);\n+\trename_count = find_exact_renames(options, &local_pool);\n+\t/*\n+\t * Discard local_pool immediately instead of at \"cleanup:\" in order\n+\t * to reduce maximum memory usage; inexact rename detection uses up\n+\t * a fair amount of memory, and mem_pools can too.\n+\t */\n+\tmem_pool_discard(&local_pool, 0);\n \ttrace2_region_leave(\"diff\", \"exact renames\", options->repo);\n \n \t/* Did we only want exact renames? */\n-- \ngitgitgadget\n\n"},{"id":"431555","messageId":"2c0b90eaba5bbf03553c8ac2487b57b66a6ed18e.1627645664.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v3.git.1627645664.gitgitgadget@gmail.com","subject":"[PATCH v3 3/9] merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-30T11:47:38Z","receivedAt":"2021-07-30T11:47:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMake the code more flexible so that it can handle both being run with or\nwithout a memory pool by adding utility functions which will either call\n xmalloc, xcalloc, xstrndup\nor\n mem_pool_alloc, mem_pool_calloc, mem_pool_strndup\ndepending on whether we have a non-NULL memory pool. A subsequent\ncommit will make use of these.\n\n(We will actually be dropping these functions soon and just assuming we\nalways have a memory pool, but the flexibility was very useful during\ndevelopment of merge-ort so I want to be able to restore it if needed.)\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 24 ++++++++++++++++++++++++\n 1 file changed, 24 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 401a40247a3..63f67246d3d 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -664,6 +664,30 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n+MAYBE_UNUSED\n+static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n+{\n+\tif (!pool)\n+\t\treturn xcalloc(count, size);\n+\treturn mem_pool_calloc(pool, count, size);\n+}\n+\n+MAYBE_UNUSED\n+static void *pool_alloc(struct mem_pool *pool, size_t size)\n+{\n+\tif (!pool)\n+\t\treturn xmalloc(size);\n+\treturn mem_pool_alloc(pool, size);\n+}\n+\n+MAYBE_UNUSED\n+static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len)\n+{\n+\tif (!pool)\n+\t\treturn xstrndup(str, len);\n+\treturn mem_pool_strndup(pool, str, len);\n+}\n+\n /* add a string to a strbuf, but converting \"/\" to \"_\" */\n static void add_flattened_path(struct strbuf *out, const char *s)\n {\n-- \ngitgitgadget\n\n"},{"id":"431556","messageId":"7c49aa601d0b0880dde559579a336809d88a1c01.1627645664.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v3.git.1627645664.gitgitgadget@gmail.com","subject":"[PATCH v3 5/9] merge-ort: switch our strmaps over to using memory pools","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-30T11:47:40Z","receivedAt":"2021-07-30T11:47:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nFor all the strmaps (including strintmaps and strsets) whose memory is\nunconditionally freed as part of clear_or_reinit_internal_opts(), switch\nthem over to using our new memory pool.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 202.5 ms ± 3.2 ms 198.1 ms ± 2.6 ms\n mega-renames: 1.072 s ± 0.012 s 715.8 ms ± 4.0 ms\n just-one-mega: 357.3 ms ± 3.9 ms 276.8 ms ± 4.2 ms\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 125 +++++++++++++++++++++++++++++++---------------------\n 1 file changed, 75 insertions(+), 50 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 3f425436263..99c75690855 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -539,15 +539,19 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \tvoid (*strset_clear_func)(struct strset *) =\n \t\treinitialize ? strset_partial_clear : strset_clear;\n \n-\t/*\n-\t * We marked opti->paths with strdup_strings = 0, so that we\n-\t * wouldn't have to make another copy of the fullpath created by\n-\t * make_traverse_path from setup_path_info(). But, now that we've\n-\t * used it and have no other references to these strings, it is time\n-\t * to deallocate them.\n-\t */\n-\tfree_strmap_strings(&opti->paths);\n-\tstrmap_clear_func(&opti->paths, 1);\n+\tif (opti->pool)\n+\t\tstrmap_clear_func(&opti->paths, 0);\n+\telse {\n+\t\t/*\n+\t\t * We marked opti->paths with strdup_strings = 0, so that\n+\t\t * we wouldn't have to make another copy of the fullpath\n+\t\t * created by make_traverse_path from setup_path_info().\n+\t\t * But, now that we've used it and have no other references\n+\t\t * to these strings, it is time to deallocate them.\n+\t\t */\n+\t\tfree_strmap_strings(&opti->paths);\n+\t\tstrmap_clear_func(&opti->paths, 1);\n+\t}\n \n \t/*\n \t * All keys and values in opti->conflicted are a subset of those in\n@@ -556,16 +560,19 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t */\n \tstrmap_clear_func(&opti->conflicted, 0);\n \n-\t/*\n-\t * opti->paths_to_free is similar to opti->paths; we created it with\n-\t * strdup_strings = 0 to avoid making _another_ copy of the fullpath\n-\t * but now that we've used it and have no other references to these\n-\t * strings, it is time to deallocate them. We do so by temporarily\n-\t * setting strdup_strings to 1.\n-\t */\n-\topti->paths_to_free.strdup_strings = 1;\n-\tstring_list_clear(&opti->paths_to_free, 0);\n-\topti->paths_to_free.strdup_strings = 0;\n+\tif (!opti->pool) {\n+\t\t/*\n+\t\t * opti->paths_to_free is similar to opti->paths; we\n+\t\t * created it with strdup_strings = 0 to avoid making\n+\t\t * _another_ copy of the fullpath but now that we've used\n+\t\t * it and have no other references to these strings, it is\n+\t\t * time to deallocate them. We do so by temporarily\n+\t\t * setting strdup_strings to 1.\n+\t\t */\n+\t\topti->paths_to_free.strdup_strings = 1;\n+\t\tstring_list_clear(&opti->paths_to_free, 0);\n+\t\topti->paths_to_free.strdup_strings = 0;\n+\t}\n \n \tif (opti->attr_index.cache_nr) /* true iff opt->renormalize */\n \t\tdiscard_index(&opti->attr_index);\n@@ -683,7 +690,6 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n-MAYBE_UNUSED\n static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n {\n \tif (!pool)\n@@ -691,7 +697,6 @@ static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n \treturn mem_pool_calloc(pool, count, size);\n }\n \n-MAYBE_UNUSED\n static void *pool_alloc(struct mem_pool *pool, size_t size)\n {\n \tif (!pool)\n@@ -699,7 +704,6 @@ static void *pool_alloc(struct mem_pool *pool, size_t size)\n \treturn mem_pool_alloc(pool, size);\n }\n \n-MAYBE_UNUSED\n static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len)\n {\n \tif (!pool)\n@@ -835,8 +839,9 @@ static void setup_path_info(struct merge_options *opt,\n \tassert(!df_conflict || !resolved); /* df_conflict implies !resolved */\n \tassert(resolved == (merged_version != NULL));\n \n-\tmi = xcalloc(1, resolved ? sizeof(struct merged_info) :\n-\t\t\t\t sizeof(struct conflict_info));\n+\tmi = pool_calloc(opt->priv->pool, 1,\n+\t\t\t resolved ? sizeof(struct merged_info) :\n+\t\t\t\t sizeof(struct conflict_info));\n \tmi->directory_name = current_dir_name;\n \tmi->basename_offset = current_dir_name_len;\n \tmi->clean = !!resolved;\n@@ -1128,7 +1133,7 @@ static int collect_merge_info_callback(int n,\n \tlen = traverse_path_len(info, p->pathlen);\n \n \t/* +1 in both of the following lines to include the NUL byte */\n-\tfullpath = xmalloc(len + 1);\n+\tfullpath = pool_alloc(opt->priv->pool, len + 1);\n \tmake_traverse_path(fullpath, len + 1, info, p->path, p->pathlen);\n \n \t/*\n@@ -1383,7 +1388,7 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\tcopy = renames->deferred[side].possible_trivial_merges;\n \t\tstrintmap_init_with_options(&renames->deferred[side].possible_trivial_merges,\n \t\t\t\t\t 0,\n-\t\t\t\t\t NULL,\n+\t\t\t\t\t opt->priv->pool,\n \t\t\t\t\t 0);\n \t\tstrintmap_for_each_entry(©, &iter, entry) {\n \t\t\tconst char *path = entry->key;\n@@ -2335,12 +2340,21 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \tVERIFY_CI(ci);\n \n \t/* Find parent directories missing from opt->priv->paths */\n-\tcur_path = new_path;\n+\tif (opt->priv->pool) {\n+\t\tcur_path = mem_pool_strdup(opt->priv->pool, new_path);\n+\t\tfree((char*)new_path);\n+\t\tnew_path = (char *)cur_path;\n+\t} else {\n+\t\tcur_path = new_path;\n+\t}\n+\n \twhile (1) {\n \t\t/* Find the parent directory of cur_path */\n \t\tchar *last_slash = strrchr(cur_path, '/');\n \t\tif (last_slash) {\n-\t\t\tparent_name = xstrndup(cur_path, last_slash - cur_path);\n+\t\t\tparent_name = pool_strndup(opt->priv->pool,\n+\t\t\t\t\t\t cur_path,\n+\t\t\t\t\t\t last_slash - cur_path);\n \t\t} else {\n \t\t\tparent_name = opt->priv->toplevel_dir;\n \t\t\tbreak;\n@@ -2349,7 +2363,8 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\t/* Look it up in opt->priv->paths */\n \t\tentry = strmap_get_entry(&opt->priv->paths, parent_name);\n \t\tif (entry) {\n-\t\t\tfree((char*)parent_name);\n+\t\t\tif (!opt->priv->pool)\n+\t\t\t\tfree((char*)parent_name);\n \t\t\tparent_name = entry->key; /* reuse known pointer */\n \t\t\tbreak;\n \t\t}\n@@ -2376,12 +2391,15 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\tparent_name = cur_dir;\n \t}\n \n-\t/*\n-\t * We are removing old_path from opt->priv->paths. old_path also will\n-\t * eventually need to be freed, but it may still be used by e.g.\n-\t * ci->pathnames. So, store it in another string-list for now.\n-\t */\n-\tstring_list_append(&opt->priv->paths_to_free, old_path);\n+\tif (!opt->priv->pool) {\n+\t\t/*\n+\t\t * We are removing old_path from opt->priv->paths.\n+\t\t * old_path also will eventually need to be freed, but it\n+\t\t * may still be used by e.g. ci->pathnames. So, store it\n+\t\t * in another string-list for now.\n+\t\t */\n+\t\tstring_list_append(&opt->priv->paths_to_free, old_path);\n+\t}\n \n \tassert(ci->filemask == 2 || ci->filemask == 4);\n \tassert(ci->dirmask == 0);\n@@ -2416,7 +2434,8 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\tnew_ci->stages[index].mode = ci->stages[index].mode;\n \t\toidcpy(&new_ci->stages[index].oid, &ci->stages[index].oid);\n \n-\t\tfree(ci);\n+\t\tif (!opt->priv->pool)\n+\t\t\tfree(ci);\n \t\tci = new_ci;\n \t}\n \n@@ -3623,7 +3642,8 @@ static void process_entry(struct merge_options *opt,\n \t\t * the directory to remain here, so we need to move this\n \t\t * path to some new location.\n \t\t */\n-\t\tCALLOC_ARRAY(new_ci, 1);\n+\t\tnew_ci = pool_calloc(opt->priv->pool, 1, sizeof(*new_ci));\n+\n \t\t/* We don't really want new_ci->merged.result copied, but it'll\n \t\t * be overwritten below so it doesn't matter. We also don't\n \t\t * want any directory mode/oid values copied, but we'll zero\n@@ -3715,7 +3735,7 @@ static void process_entry(struct merge_options *opt,\n \t\t\tconst char *a_path = NULL, *b_path = NULL;\n \t\t\tint rename_a = 0, rename_b = 0;\n \n-\t\t\tnew_ci = xmalloc(sizeof(*new_ci));\n+\t\t\tnew_ci = pool_alloc(opt->priv->pool, sizeof(*new_ci));\n \n \t\t\tif (S_ISREG(a_mode))\n \t\t\t\trename_a = 1;\n@@ -3788,12 +3808,14 @@ static void process_entry(struct merge_options *opt,\n \t\t\t\tstrmap_remove(&opt->priv->paths, path, 0);\n \t\t\t\t/*\n \t\t\t\t * We removed path from opt->priv->paths. path\n-\t\t\t\t * will also eventually need to be freed, but\n-\t\t\t\t * it may still be used by e.g. ci->pathnames.\n-\t\t\t\t * So, store it in another string-list for now.\n+\t\t\t\t * will also eventually need to be freed if not\n+\t\t\t\t * part of a memory pool...but it may still be\n+\t\t\t\t * used by e.g. ci->pathnames. So, store it in\n+\t\t\t\t * another string-list for now in that case.\n \t\t\t\t */\n-\t\t\t\tstring_list_append(&opt->priv->paths_to_free,\n-\t\t\t\t\t\t path);\n+\t\t\t\tif (!opt->priv->pool)\n+\t\t\t\t\tstring_list_append(&opt->priv->paths_to_free,\n+\t\t\t\t\t\t\t path);\n \t\t\t}\n \n \t\t\t/*\n@@ -4335,6 +4357,7 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n {\n \tstruct rename_info *renames;\n \tint i;\n+\tstruct mem_pool *pool = NULL;\n \n \t/* Sanity checks on opt */\n \ttrace2_region_enter(\"merge\", \"sanity checks\", opt->repo);\n@@ -4406,9 +4429,10 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n #else\n \topt->priv->pool = NULL;\n #endif\n+\tpool = opt->priv->pool;\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->dirs_removed[i],\n-\t\t\t\t\t NOT_RELEVANT, NULL, 0);\n+\t\t\t\t\t NOT_RELEVANT, pool, 0);\n \t\tstrmap_init_with_options(&renames->dir_rename_count[i],\n \t\t\t\t\t NULL, 1);\n \t\tstrmap_init_with_options(&renames->dir_renames[i],\n@@ -4422,7 +4446,7 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t\t */\n \t\tstrintmap_init_with_options(&renames->relevant_sources[i],\n \t\t\t\t\t -1 /* explicitly invalid */,\n-\t\t\t\t\t NULL, 0);\n+\t\t\t\t\t pool, 0);\n \t\tstrmap_init_with_options(&renames->cached_pairs[i],\n \t\t\t\t\t NULL, 1);\n \t\tstrset_init_with_options(&renames->cached_irrelevant[i],\n@@ -4432,9 +4456,9 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t}\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->deferred[i].possible_trivial_merges,\n-\t\t\t\t\t 0, NULL, 0);\n+\t\t\t\t\t 0, pool, 0);\n \t\tstrset_init_with_options(&renames->deferred[i].target_dirs,\n-\t\t\t\t\t NULL, 1);\n+\t\t\t\t\t pool, 1);\n \t\trenames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */\n \t}\n \n@@ -4447,9 +4471,10 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t * In contrast, conflicted just has a subset of keys from paths, so\n \t * we don't want to free those (it'd be a duplicate free).\n \t */\n-\tstrmap_init_with_options(&opt->priv->paths, NULL, 0);\n-\tstrmap_init_with_options(&opt->priv->conflicted, NULL, 0);\n-\tstring_list_init_nodup(&opt->priv->paths_to_free);\n+\tstrmap_init_with_options(&opt->priv->paths, pool, 0);\n+\tstrmap_init_with_options(&opt->priv->conflicted, pool, 0);\n+\tif (!opt->priv->pool)\n+\t\tstring_list_init_nodup(&opt->priv->paths_to_free);\n \n \t/*\n \t * keys & strbufs in output will sometimes need to outlive \"paths\",\n-- \ngitgitgadget\n\n"},{"id":"431557","messageId":"6646f6fd1cae3b7ebe279317b95405f2b6d71e9e.1627645664.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v3.git.1627645664.gitgitgadget@gmail.com","subject":"[PATCH v3 4/9] merge-ort: set up a memory pool","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-30T11:47:39Z","receivedAt":"2021-07-30T11:48:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nmerge-ort has a lot of data structures, and they all tend to be freed\ntogether in clear_or_reinit_internal_opts(). Set up a memory pool to\nallow us to make these allocations and deallocations faster. Future\ncommits will adjust various callers to make use of this memory pool.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 63f67246d3d..3f425436263 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -37,6 +37,8 @@\n #include \"unpack-trees.h\"\n #include \"xdiff-interface.h\"\n \n+#define USE_MEMORY_POOL 1 /* faster, but obscures memory leak hunting */\n+\n /*\n * We have many arrays of size 3. Whenever we have such an array, the\n * indices refer to one of the sides of the three-way merge. This is so\n@@ -339,6 +341,17 @@ struct merge_options_internal {\n \t */\n \tstruct strmap conflicted;\n \n+\t/*\n+\t * pool: memory pool for fast allocation/deallocation\n+\t *\n+\t * We allocate room for lots of filenames and auxiliary data\n+\t * structures in merge_options_internal, and it tends to all be\n+\t * freed together too. Using a memory pool for these provides a\n+\t * nice speedup.\n+\t */\n+\tstruct mem_pool internal_pool;\n+\tstruct mem_pool *pool; /* NULL, or pointer to internal_pool */\n+\n \t/*\n \t * paths_to_free: additional list of strings to free\n \t *\n@@ -603,6 +616,12 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\tstrmap_clear(&opti->output, 0);\n \t}\n \n+#if USE_MEMORY_POOL\n+\tmem_pool_discard(&opti->internal_pool, 0);\n+\tif (!reinitialize)\n+\t\topti->pool = NULL;\n+#endif\n+\n \t/* Clean out callback_data as well. */\n \tFREE_AND_NULL(renames->callback_data);\n \trenames->callback_data_nr = renames->callback_data_alloc = 0;\n@@ -4381,6 +4400,12 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \n \t/* Initialization of various renames fields */\n \trenames = &opt->priv->renames;\n+#if USE_MEMORY_POOL\n+\tmem_pool_init(&opt->priv->internal_pool, 0);\n+\topt->priv->pool = &opt->priv->internal_pool;\n+#else\n+\topt->priv->pool = NULL;\n+#endif\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->dirs_removed[i],\n \t\t\t\t\t NOT_RELEVANT, NULL, 0);\n-- \ngitgitgadget\n\n"},{"id":"431558","messageId":"4ffa5af8b570772bc4ccc859ce05c04c289646b0.1627645664.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v3.git.1627645664.gitgitgadget@gmail.com","subject":"[PATCH v3 7/9] merge-ort: store filepairs and filespecs in our mem_pool","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-30T11:47:42Z","receivedAt":"2021-07-30T11:48:01Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 198.1 ms ± 2.6 ms 198.5 ms ± 3.4 ms\n mega-renames: 715.8 ms ± 4.0 ms 679.1 ms ± 5.6 ms\n just-one-mega: 276.8 ms ± 4.2 ms 271.9 ms ± 2.8 ms\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 9 ++++-----\n diffcore.h | 1 +\n merge-ort.c | 26 ++++++++++++++------------\n 3 files changed, 19 insertions(+), 17 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 5bc559f79e9..7e6b3e1b143 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -1334,7 +1334,6 @@ static void free_filespec_data(struct diff_filespec *spec)\n \t\tdiff_free_filespec_data(spec);\n }\n \n-MAYBE_UNUSED\n static void pool_free_filespec(struct mem_pool *pool,\n \t\t\t struct diff_filespec *spec)\n {\n@@ -1351,7 +1350,6 @@ static void pool_free_filespec(struct mem_pool *pool,\n \tfree_filespec_data(spec);\n }\n \n-MAYBE_UNUSED\n void pool_diff_free_filepair(struct mem_pool *pool,\n \t\t\t struct diff_filepair *p)\n {\n@@ -1370,6 +1368,7 @@ void pool_diff_free_filepair(struct mem_pool *pool,\n }\n \n void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct mem_pool *pool,\n \t\t\t struct strintmap *relevant_sources,\n \t\t\t struct strintmap *dirs_removed,\n \t\t\t struct strmap *dir_rename_count,\n@@ -1683,7 +1682,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \t\t\tpair_to_free = p;\n \n \t\tif (pair_to_free)\n-\t\t\tdiff_free_filepair(pair_to_free);\n+\t\t\tpool_diff_free_filepair(pool, pair_to_free);\n \t}\n \tdiff_debug_queue(\"done copying original\", &outq);\n \n@@ -1693,7 +1692,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \n \tfor (i = 0; i < rename_dst_nr; i++)\n \t\tif (rename_dst[i].filespec_to_free)\n-\t\t\tfree_filespec(rename_dst[i].filespec_to_free);\n+\t\t\tpool_free_filespec(pool, rename_dst[i].filespec_to_free);\n \n \tcleanup_dir_rename_info(&info, dirs_removed, dir_rename_count != NULL);\n \tFREE_AND_NULL(rename_dst);\n@@ -1710,5 +1709,5 @@ void diffcore_rename_extended(struct diff_options *options,\n \n void diffcore_rename(struct diff_options *options)\n {\n-\tdiffcore_rename_extended(options, NULL, NULL, NULL, NULL);\n+\tdiffcore_rename_extended(options, NULL, NULL, NULL, NULL, NULL);\n }\ndiff --git a/diffcore.h b/diffcore.h\nindex b58ee6b1934..badc2261c20 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -181,6 +181,7 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count);\n void diffcore_break(struct repository *, int);\n void diffcore_rename(struct diff_options *);\n void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct mem_pool *pool,\n \t\t\t struct strintmap *relevant_sources,\n \t\t\t struct strintmap *dirs_removed,\n \t\t\t struct strmap *dir_rename_count,\ndiff --git a/merge-ort.c b/merge-ort.c\nindex e79830f9181..f4f0a3d57f0 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -690,7 +690,6 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n-MAYBE_UNUSED\n static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n \t\t\t\t\t\t const char *path)\n {\n@@ -712,7 +711,6 @@ static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n \treturn spec;\n }\n \n-MAYBE_UNUSED\n static struct diff_filepair *pool_diff_queue(struct mem_pool *pool,\n \t\t\t\t\t struct diff_queue_struct *queue,\n \t\t\t\t\t struct diff_filespec *one,\n@@ -930,6 +928,7 @@ static void add_pair(struct merge_options *opt,\n \t\t unsigned dir_rename_mask)\n {\n \tstruct diff_filespec *one, *two;\n+\tstruct mem_pool *pool = opt->priv->pool;\n \tstruct rename_info *renames = &opt->priv->renames;\n \tint names_idx = is_add ? side : 0;\n \n@@ -980,11 +979,11 @@ static void add_pair(struct merge_options *opt,\n \t\t\treturn;\n \t}\n \n-\tone = alloc_filespec(pathname);\n-\ttwo = alloc_filespec(pathname);\n+\tone = pool_alloc_filespec(pool, pathname);\n+\ttwo = pool_alloc_filespec(pool, pathname);\n \tfill_filespec(is_add ? two : one,\n \t\t &names[names_idx].oid, 1, names[names_idx].mode);\n-\tdiff_queue(&renames->pairs[side], one, two);\n+\tpool_diff_queue(pool, &renames->pairs[side], one, two);\n }\n \n static void collect_rename_info(struct merge_options *opt,\n@@ -2893,6 +2892,7 @@ static void use_cached_pairs(struct merge_options *opt,\n {\n \tstruct hashmap_iter iter;\n \tstruct strmap_entry *entry;\n+\tstruct mem_pool *pool = opt->priv->pool;\n \n \t/*\n \t * Add to side_pairs all entries from renames->cached_pairs[side_index].\n@@ -2906,9 +2906,9 @@ static void use_cached_pairs(struct merge_options *opt,\n \t\t\tnew_name = old_name;\n \n \t\t/* We don't care about oid/mode, only filenames and status */\n-\t\tone = alloc_filespec(old_name);\n-\t\ttwo = alloc_filespec(new_name);\n-\t\tdiff_queue(pairs, one, two);\n+\t\tone = pool_alloc_filespec(pool, old_name);\n+\t\ttwo = pool_alloc_filespec(pool, new_name);\n+\t\tpool_diff_queue(pool, pairs, one, two);\n \t\tpairs->queue[pairs->nr-1]->status = entry->value ? 'R' : 'D';\n \t}\n }\n@@ -3016,6 +3016,7 @@ static int detect_regular_renames(struct merge_options *opt,\n \tdiff_queued_diff = renames->pairs[side_index];\n \ttrace2_region_enter(\"diff\", \"diffcore_rename\", opt->repo);\n \tdiffcore_rename_extended(&diff_opts,\n+\t\t\t\t opt->priv->pool,\n \t\t\t\t &renames->relevant_sources[side_index],\n \t\t\t\t &renames->dirs_removed[side_index],\n \t\t\t\t &renames->dir_rename_count[side_index],\n@@ -3066,7 +3067,7 @@ static int collect_renames(struct merge_options *opt,\n \n \t\tif (p->status != 'A' && p->status != 'R') {\n \t\t\tpossibly_cache_new_pair(renames, p, side_index, NULL);\n-\t\t\tdiff_free_filepair(p);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n \t\t\tcontinue;\n \t\t}\n \n@@ -3079,7 +3080,7 @@ static int collect_renames(struct merge_options *opt,\n \n \t\tpossibly_cache_new_pair(renames, p, side_index, new_path);\n \t\tif (p->status != 'R' && !new_path) {\n-\t\t\tdiff_free_filepair(p);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n \t\t\tcontinue;\n \t\t}\n \n@@ -3197,7 +3198,7 @@ cleanup:\n \t\tside_pairs = &renames->pairs[s];\n \t\tfor (i = 0; i < side_pairs->nr; ++i) {\n \t\t\tstruct diff_filepair *p = side_pairs->queue[i];\n-\t\t\tdiff_free_filepair(p);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n \t\t}\n \t}\n \n@@ -3210,7 +3211,8 @@ simple_cleanup:\n \tif (combined.nr) {\n \t\tint i;\n \t\tfor (i = 0; i < combined.nr; i++)\n-\t\t\tdiff_free_filepair(combined.queue[i]);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool,\n+\t\t\t\t\t\tcombined.queue[i]);\n \t\tfree(combined.queue);\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"431559","messageId":"08cf2498f96759a94543d61b9a05e02280ab19d2.1627645664.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v3.git.1627645664.gitgitgadget@gmail.com","subject":"[PATCH v3 6/9] diffcore-rename, merge-ort: add wrapper functions for filepair alloc/dealloc","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-30T11:47:41Z","receivedAt":"2021-07-30T11:48:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe want to be able to allocate filespecs and filepairs using a mem_pool.\nHowever, filespec data will still remain outside the pool (perhaps in\nthe future we could plumb the pool through the various diff APIs to\nallocate the filespec data too, but for now we are limiting the scope).\nAdd some extra functions to allocate these appropriately based on the\nnon-NULL-ness of opt->priv->pool, as well as some extra functions to\nhandle correctly deallocating the relevant parts of them. A future\ncommit will make use of these new functions.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 41 +++++++++++++++++++++++++++++++++++++++++\n diffcore.h | 2 ++\n merge-ort.c | 42 ++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 85 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 73d884099eb..5bc559f79e9 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -1328,6 +1328,47 @@ static void handle_early_known_dir_renames(struct dir_rename_info *info,\n \trename_src_nr = new_num_src;\n }\n \n+static void free_filespec_data(struct diff_filespec *spec)\n+{\n+\tif (!--spec->count)\n+\t\tdiff_free_filespec_data(spec);\n+}\n+\n+MAYBE_UNUSED\n+static void pool_free_filespec(struct mem_pool *pool,\n+\t\t\t struct diff_filespec *spec)\n+{\n+\tif (!pool) {\n+\t\tfree_filespec(spec);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Similar to free_filespec(), but only frees the data. The spec\n+\t * itself was allocated in the pool and should not be individually\n+\t * freed.\n+\t */\n+\tfree_filespec_data(spec);\n+}\n+\n+MAYBE_UNUSED\n+void pool_diff_free_filepair(struct mem_pool *pool,\n+\t\t\t struct diff_filepair *p)\n+{\n+\tif (!pool) {\n+\t\tdiff_free_filepair(p);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Similar to diff_free_filepair() but only frees the data from the\n+\t * filespecs; not the filespecs or the filepair which were\n+\t * allocated from the pool.\n+\t */\n+\tfree_filespec_data(p->one);\n+\tfree_filespec_data(p->two);\n+}\n+\n void diffcore_rename_extended(struct diff_options *options,\n \t\t\t struct strintmap *relevant_sources,\n \t\t\t struct strintmap *dirs_removed,\ndiff --git a/diffcore.h b/diffcore.h\nindex 533b30e21e7..b58ee6b1934 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -127,6 +127,8 @@ struct diff_filepair {\n #define DIFF_PAIR_MODE_CHANGED(p) ((p)->one->mode != (p)->two->mode)\n \n void diff_free_filepair(struct diff_filepair *);\n+void pool_diff_free_filepair(struct mem_pool *pool,\n+\t\t\t struct diff_filepair *p);\n \n int diff_unmodified_pair(struct diff_filepair *);\n \ndiff --git a/merge-ort.c b/merge-ort.c\nindex 99c75690855..e79830f9181 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -690,6 +690,48 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n+MAYBE_UNUSED\n+static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n+\t\t\t\t\t\t const char *path)\n+{\n+\tstruct diff_filespec *spec;\n+\tsize_t len;\n+\n+\tif (!pool)\n+\t\treturn alloc_filespec(path);\n+\n+\t/* Same code as alloc_filespec, except allocate from pool */\n+\tlen = strlen(path);\n+\n+\tspec = mem_pool_calloc(pool, 1, st_add3(sizeof(*spec), len, 1));\n+\tmemcpy(spec+1, path, len);\n+\tspec->path = (void*)(spec+1);\n+\n+\tspec->count = 1;\n+\tspec->is_binary = -1;\n+\treturn spec;\n+}\n+\n+MAYBE_UNUSED\n+static struct diff_filepair *pool_diff_queue(struct mem_pool *pool,\n+\t\t\t\t\t struct diff_queue_struct *queue,\n+\t\t\t\t\t struct diff_filespec *one,\n+\t\t\t\t\t struct diff_filespec *two)\n+{\n+\tstruct diff_filepair *dp;\n+\n+\tif (!pool)\n+\t\treturn diff_queue(queue, one, two);\n+\n+\t/* Same code as diff_queue, except allocate from pool */\n+\tdp = mem_pool_calloc(pool, 1, sizeof(*dp));\n+\tdp->one = one;\n+\tdp->two = two;\n+\tif (queue)\n+\t\tdiff_q(queue, dp);\n+\treturn dp;\n+}\n+\n static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n {\n \tif (!pool)\n-- \ngitgitgadget\n\n"},{"id":"431560","messageId":"1556f0443c35a95fbdbc1cb1841f81f9c4615f4c.1627645665.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v3.git.1627645664.gitgitgadget@gmail.com","subject":"[PATCH v3 8/9] merge-ort: reuse path strings in pool_alloc_filespec","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-30T11:47:43Z","receivedAt":"2021-07-30T11:48:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\npool_alloc_filespec() was written so that the code when pool != NULL\nmimicked the code from alloc_filespec(), which including allocating\nenough extra space for the path and then copying it. However, the path\npassed to pool_alloc_filespec() is always going to already be in the\nsame memory pool, so we may as well reuse it instead of copying it.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 198.5 ms ± 3.4 ms 198.3 ms ± 2.9 ms\n mega-renames: 679.1 ms ± 5.6 ms 661.8 ms ± 5.9 ms\n just-one-mega: 271.9 ms ± 2.8 ms 264.6 ms ± 2.5 ms\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 29 ++++++++++++++++++++++-------\n 1 file changed, 22 insertions(+), 7 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex f4f0a3d57f0..86ab8f60121 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -694,17 +694,13 @@ static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n \t\t\t\t\t\t const char *path)\n {\n \tstruct diff_filespec *spec;\n-\tsize_t len;\n \n \tif (!pool)\n \t\treturn alloc_filespec(path);\n \n-\t/* Same code as alloc_filespec, except allocate from pool */\n-\tlen = strlen(path);\n-\n-\tspec = mem_pool_calloc(pool, 1, st_add3(sizeof(*spec), len, 1));\n-\tmemcpy(spec+1, path, len);\n-\tspec->path = (void*)(spec+1);\n+\t/* Similar to alloc_filespec, but allocate from pool and reuse path */\n+\tspec = mem_pool_calloc(pool, 1, sizeof(*spec));\n+\tspec->path = (char*)path; /* spec won't modify it */\n \n \tspec->count = 1;\n \tspec->is_binary = -1;\n@@ -2904,6 +2900,25 @@ static void use_cached_pairs(struct merge_options *opt,\n \t\tconst char *new_name = entry->value;\n \t\tif (!new_name)\n \t\t\tnew_name = old_name;\n+\t\tif (pool) {\n+\t\t\t/*\n+\t\t\t * cached_pairs has _copies* of old_name and new_name,\n+\t\t\t * because it has to persist across merges. When\n+\t\t\t * pool != NULL\n+\t\t\t * pool_alloc_filespec() will just re-use the existing\n+\t\t\t * filenames, which will also get re-used by\n+\t\t\t * opt->priv->paths if they become renames, and then\n+\t\t\t * get freed at the end of the merge, leaving the copy\n+\t\t\t * in cached_pairs dangling. Avoid this by making a\n+\t\t\t * copy here.\n+\t\t\t *\n+\t\t\t * When pool == NULL, pool_alloc_filespec() calls\n+\t\t\t * alloc_filespec(), which makes a copy; we don't want\n+\t\t\t * to add another.\n+\t\t\t */\n+\t\t\told_name = mem_pool_strdup(pool, old_name);\n+\t\t\tnew_name = mem_pool_strdup(pool, new_name);\n+\t\t}\n \n \t\t/* We don't care about oid/mode, only filenames and status */\n \t\tone = pool_alloc_filespec(pool, old_name);\n-- \ngitgitgadget\n\n"},{"id":"431561","messageId":"de30dbac25e78e5ad81053b60c3ec2ab6256fde2.1627645665.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v3.git.1627645664.gitgitgadget@gmail.com","subject":"[PATCH v3 9/9] merge-ort: remove compile-time ability to turn off usage of memory pools","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-30T11:47:44Z","receivedAt":"2021-07-30T11:48:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSimplify code maintenance a bit by removing the ability to toggle\nbetween usage of memory pools and direct allocations. This allows us to\nalso remove and simplify some auxiliary functions.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 63 +++++++++++++----------------------------------------\n 1 file changed, 15 insertions(+), 48 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 86ab8f60121..63829f5cace 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -37,8 +37,6 @@\n #include \"unpack-trees.h\"\n #include \"xdiff-interface.h\"\n \n-#define USE_MEMORY_POOL 1 /* faster, but obscures memory leak hunting */\n-\n /*\n * We have many arrays of size 3. Whenever we have such an array, the\n * indices refer to one of the sides of the three-way merge. This is so\n@@ -623,11 +621,9 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\tstrmap_clear(&opti->output, 0);\n \t}\n \n-#if USE_MEMORY_POOL\n \tmem_pool_discard(&opti->internal_pool, 0);\n \tif (!reinitialize)\n \t\topti->pool = NULL;\n-#endif\n \n \t/* Clean out callback_data as well. */\n \tFREE_AND_NULL(renames->callback_data);\n@@ -693,12 +689,10 @@ static void path_msg(struct merge_options *opt,\n static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n \t\t\t\t\t\t const char *path)\n {\n+\t/* Similar to alloc_filespec(), but allocate from pool and reuse path */\n \tstruct diff_filespec *spec;\n \n-\tif (!pool)\n-\t\treturn alloc_filespec(path);\n-\n-\t/* Similar to alloc_filespec, but allocate from pool and reuse path */\n+\tassert(pool != NULL);\n \tspec = mem_pool_calloc(pool, 1, sizeof(*spec));\n \tspec->path = (char*)path; /* spec won't modify it */\n \n@@ -712,12 +706,10 @@ static struct diff_filepair *pool_diff_queue(struct mem_pool *pool,\n \t\t\t\t\t struct diff_filespec *one,\n \t\t\t\t\t struct diff_filespec *two)\n {\n+\t/* Same code as diff_queue(), except allocate from pool */\n \tstruct diff_filepair *dp;\n \n-\tif (!pool)\n-\t\treturn diff_queue(queue, one, two);\n-\n-\t/* Same code as diff_queue, except allocate from pool */\n+\tassert(pool != NULL);\n \tdp = mem_pool_calloc(pool, 1, sizeof(*dp));\n \tdp->one = one;\n \tdp->two = two;\n@@ -726,27 +718,6 @@ static struct diff_filepair *pool_diff_queue(struct mem_pool *pool,\n \treturn dp;\n }\n \n-static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n-{\n-\tif (!pool)\n-\t\treturn xcalloc(count, size);\n-\treturn mem_pool_calloc(pool, count, size);\n-}\n-\n-static void *pool_alloc(struct mem_pool *pool, size_t size)\n-{\n-\tif (!pool)\n-\t\treturn xmalloc(size);\n-\treturn mem_pool_alloc(pool, size);\n-}\n-\n-static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len)\n-{\n-\tif (!pool)\n-\t\treturn xstrndup(str, len);\n-\treturn mem_pool_strndup(pool, str, len);\n-}\n-\n /* add a string to a strbuf, but converting \"/\" to \"_\" */\n static void add_flattened_path(struct strbuf *out, const char *s)\n {\n@@ -875,9 +846,9 @@ static void setup_path_info(struct merge_options *opt,\n \tassert(!df_conflict || !resolved); /* df_conflict implies !resolved */\n \tassert(resolved == (merged_version != NULL));\n \n-\tmi = pool_calloc(opt->priv->pool, 1,\n-\t\t\t resolved ? sizeof(struct merged_info) :\n-\t\t\t\t sizeof(struct conflict_info));\n+\tmi = mem_pool_calloc(opt->priv->pool, 1,\n+\t\t\t resolved ? sizeof(struct merged_info) :\n+\t\t\t\t\tsizeof(struct conflict_info));\n \tmi->directory_name = current_dir_name;\n \tmi->basename_offset = current_dir_name_len;\n \tmi->clean = !!resolved;\n@@ -1170,7 +1141,7 @@ static int collect_merge_info_callback(int n,\n \tlen = traverse_path_len(info, p->pathlen);\n \n \t/* +1 in both of the following lines to include the NUL byte */\n-\tfullpath = pool_alloc(opt->priv->pool, len + 1);\n+\tfullpath = mem_pool_alloc(opt->priv->pool, len + 1);\n \tmake_traverse_path(fullpath, len + 1, info, p->path, p->pathlen);\n \n \t/*\n@@ -2389,9 +2360,9 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\t/* Find the parent directory of cur_path */\n \t\tchar *last_slash = strrchr(cur_path, '/');\n \t\tif (last_slash) {\n-\t\t\tparent_name = pool_strndup(opt->priv->pool,\n-\t\t\t\t\t\t cur_path,\n-\t\t\t\t\t\t last_slash - cur_path);\n+\t\t\tparent_name = mem_pool_strndup(opt->priv->pool,\n+\t\t\t\t\t\t cur_path,\n+\t\t\t\t\t\t last_slash - cur_path);\n \t\t} else {\n \t\t\tparent_name = opt->priv->toplevel_dir;\n \t\t\tbreak;\n@@ -3701,7 +3672,7 @@ static void process_entry(struct merge_options *opt,\n \t\t * the directory to remain here, so we need to move this\n \t\t * path to some new location.\n \t\t */\n-\t\tnew_ci = pool_calloc(opt->priv->pool, 1, sizeof(*new_ci));\n+\t\tnew_ci = mem_pool_calloc(opt->priv->pool, 1, sizeof(*new_ci));\n \n \t\t/* We don't really want new_ci->merged.result copied, but it'll\n \t\t * be overwritten below so it doesn't matter. We also don't\n@@ -3794,7 +3765,8 @@ static void process_entry(struct merge_options *opt,\n \t\t\tconst char *a_path = NULL, *b_path = NULL;\n \t\t\tint rename_a = 0, rename_b = 0;\n \n-\t\t\tnew_ci = pool_alloc(opt->priv->pool, sizeof(*new_ci));\n+\t\t\tnew_ci = mem_pool_alloc(opt->priv->pool,\n+\t\t\t\t\t\tsizeof(*new_ci));\n \n \t\t\tif (S_ISREG(a_mode))\n \t\t\t\trename_a = 1;\n@@ -4482,13 +4454,8 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \n \t/* Initialization of various renames fields */\n \trenames = &opt->priv->renames;\n-#if USE_MEMORY_POOL\n \tmem_pool_init(&opt->priv->internal_pool, 0);\n-\topt->priv->pool = &opt->priv->internal_pool;\n-#else\n-\topt->priv->pool = NULL;\n-#endif\n-\tpool = opt->priv->pool;\n+\tpool = opt->priv->pool = &opt->priv->internal_pool;\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->dirs_removed[i],\n \t\t\t\t\t NOT_RELEVANT, pool, 0);\n-- \ngitgitgadget\n"},{"id":"431565","messageId":"87bl6kq631.fsf@evledraar.gmail.com","threadId":"56152","inReplyTo":"YQMLBW6LAZmQ8nDz@coredump.intra.peff.net","subject":"Re: [PATCH v2 4/7] merge-ort: switch our strmaps over to using memory pools","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-30T13:30:07Z","receivedAt":"2021-07-30T13:33:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 29 2021, Jeff King wrote:\n\n> On Thu, Jul 29, 2021 at 12:37:52PM -0600, Elijah Newren wrote:\n>\n>> > Arguably, the existence of these function indirections is perhaps a sign\n>> > that the strmap API should provide a version of the clear functions that\n>> > takes \"partial / not-partial\" as a parameter.\n>> \n>> Are you suggesting a modification of str{map,intmap,set}_clear() to\n>> take an extra parameter, or removing the\n>> str{map,intmap,set}_partial_clear() functions and introducing new\n>> functions that take a partial/not-partial parameter? I think you're\n>> suggesting the latter, and that makes more sense to me...but I'm\n>> drawing blanks trying to come up with a reasonable function name.\n>\n> It does seem a shame to add the \"partial\" parameter to strmap_clear(),\n> just because most callers don't need it (so they end up with this\n> inscrutable \"0\" parameter).\n>\n> What if there was a flags field? Then it could be combined with the\n> free_values parameter. The result is kind of verbose in two ways:\n>\n> - now strset_clear(), etc, need a \"flags\" parameter, which they didn't\n> before (and is just \"0\" most of the time!)\n>\n> - now \"strmap_clear(foo, 1)\" becomes \"strmap_clear(foo, STRMAP_FREE_VALUES)\".\n> That's a lot longer, though arguably it's easier to understand since\n> the boolean is explained.\n>\n> Having gone through the exercise, I am not sure it is actually making\n> anything more readable (messy patch is below for reference).\n\nI've got some WIP patches for string-list.h and strmap.h to make the API\nnicer, and it's probably applicable to strset.h too.\n\nI.e. I found when using strset.h that it was a weird API to use, because\nunlike string-list.h it didn't pay attention to your \"dup\" field when\nfreeing, you had to do it explicitly.\n\nAnd then in e.g. merge-ort.c there's this \"strdup dance\" pattern where\nwe flip the field back and forth.\n\nThe below diff is exctracted from that WIP work, with the relevant two\nAPI headers and then two changed API users for show (the tree-wide\nchanges are much larger).\n\nI think making the promise I make in the updated docs at \"We guarantee\nthat the `clearfunc`[...]\" in string-list.h makes for particularly nice\nAPI behavior.\n\n builtin/remote.c | 37 ++++++++++++++++++++---------------\n merge-ort.c | 32 +++++++-----------------------\n string-list.h | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n strmap.h | 13 +++++++++++++\n 4 files changed, 98 insertions(+), 43 deletions(-)\n\ndiff --git a/builtin/remote.c b/builtin/remote.c\nindex 7f88e6ce9de..ec1dbd49f71 100644\n--- a/builtin/remote.c\n+++ b/builtin/remote.c\n@@ -340,10 +340,24 @@ static void read_branches(void)\n \n struct ref_states {\n \tstruct remote *remote;\n-\tstruct string_list new_refs, stale, tracked, heads, push;\n+\n+\tstruct string_list new_refs;\n+\tstruct string_list stale;\n+\tstruct string_list tracked;\n+\tstruct string_list heads;\n+\tstruct string_list push;\n+\n \tint queried;\n };\n \n+#define REF_STATES_INIT { \\\n+\t.new_refs = STRING_LIST_INIT_DUP, \\\n+\t.stale = STRING_LIST_INIT_DUP, \\\n+\t.tracked = STRING_LIST_INIT_DUP, \\\n+\t.heads = STRING_LIST_INIT_DUP, \\\n+\t.push = STRING_LIST_INIT_DUP, \\\n+}\n+\n static int get_ref_states(const struct ref *remote_refs, struct ref_states *states)\n {\n \tstruct ref *fetch_map = NULL, **tail = &fetch_map;\n@@ -355,9 +369,6 @@ static int get_ref_states(const struct ref *remote_refs, struct ref_states *stat\n \t\t\tdie(_(\"Could not get fetch map for refspec %s\"),\n \t\t\t\tstates->remote->fetch.raw[i]);\n \n-\tstates->new_refs.strdup_strings = 1;\n-\tstates->tracked.strdup_strings = 1;\n-\tstates->stale.strdup_strings = 1;\n \tfor (ref = fetch_map; ref; ref = ref->next) {\n \t\tif (!ref->peer_ref || !ref_exists(ref->peer_ref->name))\n \t\t\tstring_list_append(&states->new_refs, abbrev_branch(ref->name));\n@@ -406,7 +417,6 @@ static int get_push_ref_states(const struct ref *remote_refs,\n \n \tmatch_push_refs(local_refs, &push_map, &remote->push, MATCH_REFS_NONE);\n \n-\tstates->push.strdup_strings = 1;\n \tfor (ref = push_map; ref; ref = ref->next) {\n \t\tstruct string_list_item *item;\n \t\tstruct push_info *info;\n@@ -449,7 +459,6 @@ static int get_push_ref_states_noquery(struct ref_states *states)\n \tif (remote->mirror)\n \t\treturn 0;\n \n-\tstates->push.strdup_strings = 1;\n \tif (!remote->push.nr) {\n \t\titem = string_list_append(&states->push, _(\"(matching)\"));\n \t\tinfo = item->util = xcalloc(1, sizeof(struct push_info));\n@@ -483,7 +492,6 @@ static int get_head_names(const struct ref *remote_refs, struct ref_states *stat\n \trefspec.force = 0;\n \trefspec.pattern = 1;\n \trefspec.src = refspec.dst = \"refs/heads/*\";\n-\tstates->heads.strdup_strings = 1;\n \tget_fetch_map(remote_refs, &refspec, &fetch_map_tail, 0);\n \tmatches = guess_remote_head(find_ref_by_name(remote_refs, \"HEAD\"),\n \t\t\t\t fetch_map, 1);\n@@ -905,7 +913,7 @@ static void clear_push_info(void *util, const char *string)\n {\n \tstruct push_info *info = util;\n \tfree(info->dest);\n-\tfree(info);\n+\t/* note: fixed memleak here */\n }\n \n static void free_remote_ref_states(struct ref_states *states)\n@@ -1159,7 +1167,7 @@ static int get_one_entry(struct remote *remote, void *priv)\n \t\tstring_list_append(list, remote->name)->util =\n \t\t\t\tstrbuf_detach(&url_buf, NULL);\n \t} else\n-\t\tstring_list_append(list, remote->name)->util = NULL;\n+\t\tstring_list_append(list, remote->name);\n \tif (remote->pushurl_nr) {\n \t\turl = remote->pushurl;\n \t\turl_nr = remote->pushurl_nr;\n@@ -1179,10 +1187,9 @@ static int get_one_entry(struct remote *remote, void *priv)\n \n static int show_all(void)\n {\n-\tstruct string_list list = STRING_LIST_INIT_NODUP;\n+\tstruct string_list list = STRING_LIST_INIT_DUP;\n \tint result;\n \n-\tlist.strdup_strings = 1;\n \tresult = for_each_remote(get_one_entry, &list);\n \n \tif (!result) {\n@@ -1212,7 +1219,7 @@ static int show(int argc, const char **argv)\n \t\tOPT_BOOL('n', NULL, &no_query, N_(\"do not query remotes\")),\n \t\tOPT_END()\n \t};\n-\tstruct ref_states states;\n+\tstruct ref_states states = REF_STATES_INIT;\n \tstruct string_list info_list = STRING_LIST_INIT_NODUP;\n \tstruct show_info info;\n \n@@ -1334,8 +1341,7 @@ static int set_head(int argc, const char **argv)\n \tif (!opt_a && !opt_d && argc == 2) {\n \t\thead_name = xstrdup(argv[1]);\n \t} else if (opt_a && !opt_d && argc == 1) {\n-\t\tstruct ref_states states;\n-\t\tmemset(&states, 0, sizeof(states));\n+\t\tstruct ref_states states = REF_STATES_INIT;\n \t\tget_remote_ref_states(argv[0], &states, GET_HEAD_NAMES);\n \t\tif (!states.heads.nr)\n \t\t\tresult |= error(_(\"Cannot determine remote HEAD\"));\n@@ -1374,14 +1380,13 @@ static int set_head(int argc, const char **argv)\n static int prune_remote(const char *remote, int dry_run)\n {\n \tint result = 0;\n-\tstruct ref_states states;\n+\tstruct ref_states states = REF_STATES_INIT;\n \tstruct string_list refs_to_prune = STRING_LIST_INIT_NODUP;\n \tstruct string_list_item *item;\n \tconst char *dangling_msg = dry_run\n \t\t? _(\" %s will become dangling!\")\n \t\t: _(\" %s has become dangling!\");\n \n-\tmemset(&states, 0, sizeof(states));\n \tget_remote_ref_states(remote, &states, GET_REF_STATES);\n \n \tif (!states.stale.nr) {\ndiff --git a/merge-ort.c b/merge-ort.c\nindex ec0c5904211..53ed78e7a01 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -432,16 +432,6 @@ struct conflict_info {\n \tassert((ci) && !(mi)->clean); \\\n } while (0)\n \n-static void free_strmap_strings(struct strmap *map)\n-{\n-\tstruct hashmap_iter iter;\n-\tstruct strmap_entry *entry;\n-\n-\tstrmap_for_each_entry(map, &iter, entry) {\n-\t\tfree((char*)entry->key);\n-\t}\n-}\n-\n static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\t\t\t\t int reinitialize)\n {\n@@ -455,13 +445,11 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\treinitialize ? strset_partial_clear : strset_clear;\n \n \t/*\n-\t * We marked opti->paths with strdup_strings = 0, so that we\n-\t * wouldn't have to make another copy of the fullpath created by\n-\t * make_traverse_path from setup_path_info(). But, now that we've\n-\t * used it and have no other references to these strings, it is time\n-\t * to deallocate them.\n+\t * We used the the pattern of re-using already allocated\n+\t * strings strmap_clear_strings() in make_traverse_path from\n+\t * setup_path_info(). Deallocate them.\n \t */\n-\tfree_strmap_strings(&opti->paths);\n+\tstrmap_clear_strings(&opti->paths, 0);\n \tstrmap_func(&opti->paths, 1);\n \n \t/*\n@@ -472,15 +460,10 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \tstrmap_func(&opti->conflicted, 0);\n \n \t/*\n-\t * opti->paths_to_free is similar to opti->paths; we created it with\n-\t * strdup_strings = 0 to avoid making _another_ copy of the fullpath\n-\t * but now that we've used it and have no other references to these\n-\t * strings, it is time to deallocate them. We do so by temporarily\n-\t * setting strdup_strings to 1.\n+\t * opti->paths_to_free is similar to opti->paths; it's memory\n+\t * we borrowed and need to free with string_list_clear_strings().\n \t */\n-\topti->paths_to_free.strdup_strings = 1;\n-\tstring_list_clear(&opti->paths_to_free, 0);\n-\topti->paths_to_free.strdup_strings = 0;\n+\tstring_list_clear_strings(&opti->paths_to_free, 0);\n \n \tif (opti->attr_index.cache_nr) /* true iff opt->renormalize */\n \t\tdiscard_index(&opti->attr_index);\n@@ -2664,7 +2647,6 @@ static int collect_renames(struct merge_options *opt,\n \t * and have no other references to these strings, it is time to\n \t * deallocate them.\n \t */\n-\tfree_strmap_strings(&collisions);\n \tstrmap_clear(&collisions, 1);\n \treturn clean;\n }\ndiff --git a/string-list.h b/string-list.h\nindex 0d6b4692396..9eeea996888 100644\n--- a/string-list.h\n+++ b/string-list.h\n@@ -109,6 +109,9 @@ void string_list_init_dup(struct string_list *list);\n */\n void string_list_init(struct string_list *list, int strdup_strings);\n \n+void string_list_cmp_init(struct string_list *list, int strdup_strings,\n+\t\t\t compare_strings_fn cmp);\n+\n /** Callback function type for for_each_string_list */\n typedef int (*string_list_each_func_t)(struct string_list_item *, void *);\n \n@@ -129,14 +132,66 @@ void filter_string_list(struct string_list *list, int free_util,\n */\n void string_list_clear(struct string_list *list, int free_util);\n \n+/**\n+ * Free a string list initialized without `strdup_strings = 1`, but\n+ * where we also want to free() the strings. You usually want to just\n+ * use string_list_clear() after initializing with\n+ * `STRING_LIST_INIT_DUP' instead.\n+ *\n+ * Useful to free e.g. a string list whose strings came from\n+ * strbuf_detach() or other memory that we didn't initially allocate\n+ * on the heap, but which we now manage.\n+ *\n+ * Under the hood this is identical in behavior to temporarily setting\n+ * `strbuf_strings` to `1` for the duration of this function call, but\n+ * without the verbosity of performing that dance yourself.\n+ */\n+void string_list_clear_strings(struct string_list *list, int free_util);\n+\n+/**\n+ * Clear only the `util` pointer, but not the `string`, even if\n+ * `strdup_strings = 1` is set. Useful for the idiom of doing e.g.:\n+ *\n+ * string_list_append(&list, str + offs)->util = str;\n+ *\n+ * Where we add a string at some offset, own the string (so\n+ * effectively `strdup_strings = `), but can't free() the string\n+ * itself at the changed offset, but need to free the original data in\n+ * `util` instead.\n+ */\n+void string_list_clear_util(struct string_list *list);\n+\n /**\n * Callback type for `string_list_clear_func`. The string associated\n * with the util pointer is passed as the second argument\n */\n typedef void (*string_list_clear_func_t)(void *p, const char *str);\n \n-/** Call a custom clear function on each util pointer */\n-void string_list_clear_func(struct string_list *list, string_list_clear_func_t clearfunc);\n+/**\n+ * Like string_list_clear() except that it first calls a custom clear\n+ * function on each util pointer.\n+ *\n+ * We guarantee that the `clearfunc` will be called on all util\n+ * pointers in a list before we proceed to free the first string or\n+ * util pointer, i.e. should you need to it's OK to peek at other util\n+ * items in the list itself, or to otherwise iterate it from within\n+ * the `clearfunc`.\n+ *\n+ * You do not need to free() the passed-in util pointer itself,\n+ * i.e. after calling all `clearfunc` this has the seme behavior as\n+ * string_list_clear() called with with `free_util = 1`.\n+ */\n+void string_list_clear_func(struct string_list *list,\n+\t\t\t string_list_clear_func_t clearfunc);\n+\n+/**\n+ * Like string_list_clear_func() but free the strings too, using the\n+ * same dance as described for string_list_clear_strings()\n+ * above. You'll usually want to initialize with\n+ * `STRING_LIST_INIT_DUP` and use string_list_clear_strings() instead.\n+ */\n+void string_list_clear_strings_func(struct string_list *list,\n+\t\t\t\t string_list_clear_func_t clearfunc);\n \n /**\n * Apply `func` to each item. If `func` returns nonzero, the\ndiff --git a/strmap.h b/strmap.h\nindex 1e152d832d6..337f6278e86 100644\n--- a/strmap.h\n+++ b/strmap.h\n@@ -51,12 +51,25 @@ void strmap_init_with_options(struct strmap *map,\n */\n void strmap_clear(struct strmap *map, int free_values);\n \n+/**\n+ * To strmap_clear() what string_list_clear_strings() is to\n+ * string_list_clear(). I.e. free your keys too, which we used as-is\n+ * without `strdup_strings = 1`.\n+ */\n+void strmap_clear_strings(struct strmap *map, int free_values);\n+\n /*\n * Similar to strmap_clear() but leaves map->map->table allocated and\n * pre-sized so that subsequent uses won't need as many rehashings.\n */\n void strmap_partial_clear(struct strmap *map, int free_values);\n \n+/**\n+ * To strmap_partial_clear() what string_list_clear_strings() is to\n+ * string_list_clear(). See strmap_clear_strings() above.\n+ */\n+void strmap_partial_clear_strings(struct strmap *map, int free_values);\n+\n /*\n * Insert \"str\" into the map, pointing to \"data\".\n *\n"},{"id":"431577","messageId":"CABPp-BHc16hjYz-+073LCA_cxHP22zW2QO6ikKKcXpG8zS1gWQ@mail.gmail.com","threadId":"56152","inReplyTo":"87bl6kq631.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 4/7] merge-ort: switch our strmaps over to using memory pools","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-30T14:36:35Z","receivedAt":"2021-07-30T14:36:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Ævar,\n\nOn Fri, Jul 30, 2021 at 7:33 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n>\n> On Thu, Jul 29 2021, Jeff King wrote:\n>\n> > On Thu, Jul 29, 2021 at 12:37:52PM -0600, Elijah Newren wrote:\n> >\n> >> > Arguably, the existence of these function indirections is perhaps a sign\n> >> > that the strmap API should provide a version of the clear functions that\n> >> > takes \"partial / not-partial\" as a parameter.\n> >>\n> >> Are you suggesting a modification of str{map,intmap,set}_clear() to\n> >> take an extra parameter, or removing the\n> >> str{map,intmap,set}_partial_clear() functions and introducing new\n> >> functions that take a partial/not-partial parameter? I think you're\n> >> suggesting the latter, and that makes more sense to me...but I'm\n> >> drawing blanks trying to come up with a reasonable function name.\n> >\n> > It does seem a shame to add the \"partial\" parameter to strmap_clear(),\n> > just because most callers don't need it (so they end up with this\n> > inscrutable \"0\" parameter).\n> >\n> > What if there was a flags field? Then it could be combined with the\n> > free_values parameter. The result is kind of verbose in two ways:\n> >\n> > - now strset_clear(), etc, need a \"flags\" parameter, which they didn't\n> > before (and is just \"0\" most of the time!)\n> >\n> > - now \"strmap_clear(foo, 1)\" becomes \"strmap_clear(foo, STRMAP_FREE_VALUES)\".\n> > That's a lot longer, though arguably it's easier to understand since\n> > the boolean is explained.\n> >\n> > Having gone through the exercise, I am not sure it is actually making\n> > anything more readable (messy patch is below for reference).\n>\n> I've got some WIP patches for string-list.h and strmap.h to make the API\n> nicer, and it's probably applicable to strset.h too.\n\nThere is no strset.h; strset and strintmap along with strmap are part\nof strmap.h.\n\n> I.e. I found when using strset.h that it was a weird API to use, because\n> unlike string-list.h it didn't pay attention to your \"dup\" field when\n> freeing, you had to do it explicitly.\n\nDo you mean strmap.h instead of strset.h?\n\nIn general, if you are asking strmap/strset/strintmap to dup your keys\nand are explicitly freeing the strings, then you are misusing the API\nand either freeing pointers that were never allocated or getting\ndouble frees. It's wrong to explicitly deallocate them because:\n * When using a pool, we just allocate from the pool. The memory\nwill be freed when the pool is freed.\n * When not using a pool, we use FLEXPTR_ALLOC_STR in order to make\nthe string be part of the allocated strmap_entry. The string's memory\nis deallocated when the strmap_entry is.\n\nThe only reason to explicitly free keys in a strmap/strset/strintmap\nis if you do NOT have strdup_strings set and allocated the strings\nelsewhere and left your strmap as the only thing tracking the strings.\n\n> And then in e.g. merge-ort.c there's this \"strdup dance\" pattern where\n> we flip the field back and forth.\n>\n> The below diff is exctracted from that WIP work, with the relevant two\n> API headers and then two changed API users for show (the tree-wide\n> changes are much larger).\n>\n> I think making the promise I make in the updated docs at \"We guarantee\n> that the `clearfunc`[...]\" in string-list.h makes for particularly nice\n> API behavior.\n>\n> builtin/remote.c | 37 ++++++++++++++++++++---------------\n> merge-ort.c | 32 +++++++-----------------------\n> string-list.h | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n> strmap.h | 13 +++++++++++++\n> 4 files changed, 98 insertions(+), 43 deletions(-)\n>\n> diff --git a/builtin/remote.c b/builtin/remote.c\n> index 7f88e6ce9de..ec1dbd49f71 100644\n> --- a/builtin/remote.c\n> +++ b/builtin/remote.c\n> @@ -340,10 +340,24 @@ static void read_branches(void)\n>\n> struct ref_states {\n> struct remote *remote;\n> - struct string_list new_refs, stale, tracked, heads, push;\n> +\n> + struct string_list new_refs;\n> + struct string_list stale;\n> + struct string_list tracked;\n> + struct string_list heads;\n> + struct string_list push;\n> +\n> int queried;\n> };\n>\n> +#define REF_STATES_INIT { \\\n> + .new_refs = STRING_LIST_INIT_DUP, \\\n> + .stale = STRING_LIST_INIT_DUP, \\\n> + .tracked = STRING_LIST_INIT_DUP, \\\n> + .heads = STRING_LIST_INIT_DUP, \\\n> + .push = STRING_LIST_INIT_DUP, \\\n> +}\n> +\n> static int get_ref_states(const struct ref *remote_refs, struct ref_states *states)\n> {\n> struct ref *fetch_map = NULL, **tail = &fetch_map;\n> @@ -355,9 +369,6 @@ static int get_ref_states(const struct ref *remote_refs, struct ref_states *stat\n> die(_(\"Could not get fetch map for refspec %s\"),\n> states->remote->fetch.raw[i]);\n>\n> - states->new_refs.strdup_strings = 1;\n> - states->tracked.strdup_strings = 1;\n> - states->stale.strdup_strings = 1;\n> for (ref = fetch_map; ref; ref = ref->next) {\n> if (!ref->peer_ref || !ref_exists(ref->peer_ref->name))\n> string_list_append(&states->new_refs, abbrev_branch(ref->name));\n> @@ -406,7 +417,6 @@ static int get_push_ref_states(const struct ref *remote_refs,\n>\n> match_push_refs(local_refs, &push_map, &remote->push, MATCH_REFS_NONE);\n>\n> - states->push.strdup_strings = 1;\n> for (ref = push_map; ref; ref = ref->next) {\n> struct string_list_item *item;\n> struct push_info *info;\n> @@ -449,7 +459,6 @@ static int get_push_ref_states_noquery(struct ref_states *states)\n> if (remote->mirror)\n> return 0;\n>\n> - states->push.strdup_strings = 1;\n> if (!remote->push.nr) {\n> item = string_list_append(&states->push, _(\"(matching)\"));\n> info = item->util = xcalloc(1, sizeof(struct push_info));\n> @@ -483,7 +492,6 @@ static int get_head_names(const struct ref *remote_refs, struct ref_states *stat\n> refspec.force = 0;\n> refspec.pattern = 1;\n> refspec.src = refspec.dst = \"refs/heads/*\";\n> - states->heads.strdup_strings = 1;\n> get_fetch_map(remote_refs, &refspec, &fetch_map_tail, 0);\n> matches = guess_remote_head(find_ref_by_name(remote_refs, \"HEAD\"),\n> fetch_map, 1);\n> @@ -905,7 +913,7 @@ static void clear_push_info(void *util, const char *string)\n> {\n> struct push_info *info = util;\n> free(info->dest);\n> - free(info);\n> + /* note: fixed memleak here */\n> }\n>\n> static void free_remote_ref_states(struct ref_states *states)\n> @@ -1159,7 +1167,7 @@ static int get_one_entry(struct remote *remote, void *priv)\n> string_list_append(list, remote->name)->util =\n> strbuf_detach(&url_buf, NULL);\n> } else\n> - string_list_append(list, remote->name)->util = NULL;\n> + string_list_append(list, remote->name);\n> if (remote->pushurl_nr) {\n> url = remote->pushurl;\n> url_nr = remote->pushurl_nr;\n> @@ -1179,10 +1187,9 @@ static int get_one_entry(struct remote *remote, void *priv)\n>\n> static int show_all(void)\n> {\n> - struct string_list list = STRING_LIST_INIT_NODUP;\n> + struct string_list list = STRING_LIST_INIT_DUP;\n> int result;\n>\n> - list.strdup_strings = 1;\n> result = for_each_remote(get_one_entry, &list);\n>\n> if (!result) {\n> @@ -1212,7 +1219,7 @@ static int show(int argc, const char **argv)\n> OPT_BOOL('n', NULL, &no_query, N_(\"do not query remotes\")),\n> OPT_END()\n> };\n> - struct ref_states states;\n> + struct ref_states states = REF_STATES_INIT;\n> struct string_list info_list = STRING_LIST_INIT_NODUP;\n> struct show_info info;\n>\n> @@ -1334,8 +1341,7 @@ static int set_head(int argc, const char **argv)\n> if (!opt_a && !opt_d && argc == 2) {\n> head_name = xstrdup(argv[1]);\n> } else if (opt_a && !opt_d && argc == 1) {\n> - struct ref_states states;\n> - memset(&states, 0, sizeof(states));\n> + struct ref_states states = REF_STATES_INIT;\n> get_remote_ref_states(argv[0], &states, GET_HEAD_NAMES);\n> if (!states.heads.nr)\n> result |= error(_(\"Cannot determine remote HEAD\"));\n> @@ -1374,14 +1380,13 @@ static int set_head(int argc, const char **argv)\n> static int prune_remote(const char *remote, int dry_run)\n> {\n> int result = 0;\n> - struct ref_states states;\n> + struct ref_states states = REF_STATES_INIT;\n> struct string_list refs_to_prune = STRING_LIST_INIT_NODUP;\n> struct string_list_item *item;\n> const char *dangling_msg = dry_run\n> ? _(\" %s will become dangling!\")\n> : _(\" %s has become dangling!\");\n>\n> - memset(&states, 0, sizeof(states));\n> get_remote_ref_states(remote, &states, GET_REF_STATES);\n>\n> if (!states.stale.nr) {\n\nEverything up to here looks like a very nice cleanup.\n\n> diff --git a/merge-ort.c b/merge-ort.c\n> index ec0c5904211..53ed78e7a01 100644\n> --- a/merge-ort.c\n> +++ b/merge-ort.c\n> @@ -432,16 +432,6 @@ struct conflict_info {\n> assert((ci) && !(mi)->clean); \\\n> } while (0)\n>\n> -static void free_strmap_strings(struct strmap *map)\n> -{\n> - struct hashmap_iter iter;\n> - struct strmap_entry *entry;\n> -\n> - strmap_for_each_entry(map, &iter, entry) {\n> - free((char*)entry->key);\n> - }\n> -}\n> -\n> static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n> int reinitialize)\n> {\n> @@ -455,13 +445,11 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n> reinitialize ? strset_partial_clear : strset_clear;\n>\n> /*\n> - * We marked opti->paths with strdup_strings = 0, so that we\n> - * wouldn't have to make another copy of the fullpath created by\n> - * make_traverse_path from setup_path_info(). But, now that we've\n> - * used it and have no other references to these strings, it is time\n> - * to deallocate them.\n> + * We used the the pattern of re-using already allocated\n> + * strings strmap_clear_strings() in make_traverse_path from\n> + * setup_path_info(). Deallocate them.\n> */\n> - free_strmap_strings(&opti->paths);\n> + strmap_clear_strings(&opti->paths, 0);\n> strmap_func(&opti->paths, 1);\n>\n> /*\n\nIt's not clear to me that strmap should handle the freeing of the keys\nat all; maybe it should and strmap_clear_strings() makes sense to\nintroduce. However, this change is clearly wrong regardless, for two\nreasons: (1) You are double clearing since strmap_func() is also\ncalled afterwards, and (2) you are also ignoring the potential partial\nbit since strmap_func might be strmap_partial_clear() rather than\nstrmap_clear().\n\n> @@ -472,15 +460,10 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n> strmap_func(&opti->conflicted, 0);\n>\n> /*\n> - * opti->paths_to_free is similar to opti->paths; we created it with\n> - * strdup_strings = 0 to avoid making _another_ copy of the fullpath\n> - * but now that we've used it and have no other references to these\n> - * strings, it is time to deallocate them. We do so by temporarily\n> - * setting strdup_strings to 1.\n> + * opti->paths_to_free is similar to opti->paths; it's memory\n> + * we borrowed and need to free with string_list_clear_strings().\n> */\n> - opti->paths_to_free.strdup_strings = 1;\n> - string_list_clear(&opti->paths_to_free, 0);\n> - opti->paths_to_free.strdup_strings = 0;\n> + string_list_clear_strings(&opti->paths_to_free, 0);\n\nThis is very nice. I really like this new function and API.\n\n> if (opti->attr_index.cache_nr) /* true iff opt->renormalize */\n> discard_index(&opti->attr_index);\n> @@ -2664,7 +2647,6 @@ static int collect_renames(struct merge_options *opt,\n> * and have no other references to these strings, it is time to\n> * deallocate them.\n> */\n> - free_strmap_strings(&collisions);\n> strmap_clear(&collisions, 1);\n> return clean;\n> }\n\nThis hunk is wrong.\n\n> diff --git a/string-list.h b/string-list.h\n> index 0d6b4692396..9eeea996888 100644\n> --- a/string-list.h\n> +++ b/string-list.h\n> @@ -109,6 +109,9 @@ void string_list_init_dup(struct string_list *list);\n> */\n> void string_list_init(struct string_list *list, int strdup_strings);\n>\n> +void string_list_cmp_init(struct string_list *list, int strdup_strings,\n> + compare_strings_fn cmp);\n> +\n\nSeems unrelated to what you were trying to highlight?\n\n> /** Callback function type for for_each_string_list */\n> typedef int (*string_list_each_func_t)(struct string_list_item *, void *);\n>\n> @@ -129,14 +132,66 @@ void filter_string_list(struct string_list *list, int free_util,\n> */\n> void string_list_clear(struct string_list *list, int free_util);\n>\n> +/**\n> + * Free a string list initialized without `strdup_strings = 1`, but\n> + * where we also want to free() the strings. You usually want to just\n> + * use string_list_clear() after initializing with\n> + * `STRING_LIST_INIT_DUP' instead.\n> + *\n> + * Useful to free e.g. a string list whose strings came from\n> + * strbuf_detach() or other memory that we didn't initially allocate\n> + * on the heap, but which we now manage.\n> + *\n> + * Under the hood this is identical in behavior to temporarily setting\n> + * `strbuf_strings` to `1` for the duration of this function call, but\n> + * without the verbosity of performing that dance yourself.\n> + */\n> +void string_list_clear_strings(struct string_list *list, int free_util);\n> +\n> +/**\n> + * Clear only the `util` pointer, but not the `string`, even if\n> + * `strdup_strings = 1` is set. Useful for the idiom of doing e.g.:\n> + *\n> + * string_list_append(&list, str + offs)->util = str;\n> + *\n> + * Where we add a string at some offset, own the string (so\n> + * effectively `strdup_strings = `), but can't free() the string\n> + * itself at the changed offset, but need to free the original data in\n> + * `util` instead.\n> + */\n> +void string_list_clear_util(struct string_list *list);\n> +\n> /**\n> * Callback type for `string_list_clear_func`. The string associated\n> * with the util pointer is passed as the second argument\n> */\n> typedef void (*string_list_clear_func_t)(void *p, const char *str);\n>\n> -/** Call a custom clear function on each util pointer */\n> -void string_list_clear_func(struct string_list *list, string_list_clear_func_t clearfunc);\n> +/**\n> + * Like string_list_clear() except that it first calls a custom clear\n> + * function on each util pointer.\n> + *\n> + * We guarantee that the `clearfunc` will be called on all util\n> + * pointers in a list before we proceed to free the first string or\n> + * util pointer, i.e. should you need to it's OK to peek at other util\n> + * items in the list itself, or to otherwise iterate it from within\n> + * the `clearfunc`.\n> + *\n> + * You do not need to free() the passed-in util pointer itself,\n> + * i.e. after calling all `clearfunc` this has the seme behavior as\n> + * string_list_clear() called with with `free_util = 1`.\n> + */\n> +void string_list_clear_func(struct string_list *list,\n> + string_list_clear_func_t clearfunc);\n> +\n> +/**\n> + * Like string_list_clear_func() but free the strings too, using the\n> + * same dance as described for string_list_clear_strings()\n> + * above. You'll usually want to initialize with\n> + * `STRING_LIST_INIT_DUP` and use string_list_clear_strings() instead.\n> + */\n> +void string_list_clear_strings_func(struct string_list *list,\n> + string_list_clear_func_t clearfunc);\n>\n> /**\n> * Apply `func` to each item. If `func` returns nonzero, the\n\nstring_list_clear_strings() looks very nice. The others are probably\ngood too, though I'm curious about the need for double walking the\nlist to free it instead of doing it in a single walk; what callers\nneed to walk the list and check out other values?\n\n> diff --git a/strmap.h b/strmap.h\n> index 1e152d832d6..337f6278e86 100644\n> --- a/strmap.h\n> +++ b/strmap.h\n> @@ -51,12 +51,25 @@ void strmap_init_with_options(struct strmap *map,\n> */\n> void strmap_clear(struct strmap *map, int free_values);\n>\n> +/**\n> + * To strmap_clear() what string_list_clear_strings() is to\n> + * string_list_clear(). I.e. free your keys too, which we used as-is\n> + * without `strdup_strings = 1`.\n> + */\n> +void strmap_clear_strings(struct strmap *map, int free_values);\n\nstrmap.h doesn't depend on string-list.h, so the comment should be\nself-standing. The analogy also doesn't seem to hold since we do NOT\nneed to free the keys when strdup_strings is 1; Peff suggested\nFLEXPTR_ALLOC_STR specifically to avoid that extra allocation in that\ncase.\n\n> +\n> /*\n> * Similar to strmap_clear() but leaves map->map->table allocated and\n> * pre-sized so that subsequent uses won't need as many rehashings.\n> */\n> void strmap_partial_clear(struct strmap *map, int free_values);\n>\n> +/**\n> + * To strmap_partial_clear() what string_list_clear_strings() is to\n> + * string_list_clear(). See strmap_clear_strings() above.\n> + */\n> +void strmap_partial_clear_strings(struct strmap *map, int free_values);\n> +\n\nSame comment as above for strmap_clear_strings() applies here.\n"},{"id":"431580","messageId":"YQQk7Vgf8hBQ9Ko+@coredump.intra.peff.net","threadId":"56152","inReplyTo":"CABPp-BF+1=yajJ_Z9MHHbeL6XkctfStj63xVXTkguNicx_PcPw@mail.gmail.com","subject":"Re: [PATCH 3/7] merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-30T16:12:29Z","receivedAt":"2021-07-30T16:12:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 29, 2021 at 08:27:51PM -0600, Elijah Newren wrote:\n\n> > FWIW, I had the same thought. You can also provide a helper to make the\n> > freeing side nicer:\n> >\n> > static void mem_pool_free(struct mem_pool *m, void *ptr)\n> > {\n> > if (m)\n> > return; /* will be freed when pool frees */\n> > free(ptr);\n> > }\n> >\n> > We do something similar with unuse_commit_buffer(), where the caller\n> > isn't aware of we pulled the buffer from cache or allocated it\n> > especially for them.\n> \n> Having a paired function may help one side, but I worry that the name\n> (mem_pool_free) might introduce some confusion of its own -- \"Why is\n> there a mem_pool_free() function, isn't the point of memory pools to\n> not need to individually free things?\" Or, \"Why are they freeing the\n> pool here and what's the extra parameter?\"\n\nYeah, \"mem_pool_maybe_free\" or something might explain it. But...\n\n> I'm not sure I see the right way to address that, so I think I'm going\n> to leave this part out of my series and let someone else add such\n> changes on top if they feel motivated to do so.\n\nThat's fine, especially as dropping the conditiona USE_MEMORY_POOL flag\nmeans these functions will go away entirely.\n\n-Peff\n"},{"id":"431581","messageId":"YQQlCvqbCaJEHMy3@coredump.intra.peff.net","threadId":"56152","inReplyTo":"CABPp-BE8OEjcMrJO8uijLFsvvpQH9KA=a+p_6bH71Z6QxXLB6w@mail.gmail.com","subject":"Re: [PATCH v2 4/7] merge-ort: switch our strmaps over to using memory pools","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-30T16:12:58Z","receivedAt":"2021-07-30T16:13:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 29, 2021 at 08:30:23PM -0600, Elijah Newren wrote:\n\n> > What if there was a flags field? Then it could be combined with the\n> > free_values parameter. The result is kind of verbose in two ways:\n> >\n> > - now strset_clear(), etc, need a \"flags\" parameter, which they didn't\n> > before (and is just \"0\" most of the time!)\n> >\n> > - now \"strmap_clear(foo, 1)\" becomes \"strmap_clear(foo, STRMAP_FREE_VALUES)\".\n> > That's a lot longer, though arguably it's easier to understand since\n> > the boolean is explained.\n> >\n> > Having gone through the exercise, I am not sure it is actually making\n> > anything more readable (messy patch is below for reference).\n> \n> Thanks for diving in. Since it's not clear if it's helping, I'll just\n> take your earlier suggestion to rename the \"strmap_func\" variable to\n> \"strmap_clear_func\" instead.\n\nThat sounds just fine with me. Thanks for considering my tangent. :)\n\n-Peff\n"},{"id":"431585","messageId":"YQQn1yS/l8j6ymZO@coredump.intra.peff.net","threadId":"56152","inReplyTo":"de30dbac25e78e5ad81053b60c3ec2ab6256fde2.1627645665.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 9/9] merge-ort: remove compile-time ability to turn off usage of memory pools","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-30T16:24:55Z","receivedAt":"2021-07-30T16:24:58Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 30, 2021 at 11:47:44AM +0000, Elijah Newren via GitGitGadget wrote:\n\n> -#if USE_MEMORY_POOL\n> \tmem_pool_init(&opt->priv->internal_pool, 0);\n> -\topt->priv->pool = &opt->priv->internal_pool;\n> -#else\n> -\topt->priv->pool = NULL;\n> -#endif\n> -\tpool = opt->priv->pool;\n> +\tpool = opt->priv->pool = &opt->priv->internal_pool;\n\nSince opt->priv->pool always points at internal_pool now, I think we can\nsimplify much more. Every \"if (!opt->priv->pool)\" can go away, and in\nturn paths_to_free does, too. Those are really the spots I was most\nworried about in terms of complexity.\n\nAn easy way to find these spots is to get rid of internal_pool, and just\nmake \"pool\" the struct. Then the compiler helpfully complains about all\nof the places that check the boolean value of a struct. :)\n\nThe patch below is from a fairly mechanical conversion I did. All of the\nspots were found by the compiler, except the one in use_cached_pairs (it\nassigns to a local \"pool\" pointer, which is always non-NULL, but that's\nnot necessarily obvious to the compiler).\n\nYou might spot further opportunities for cleanup, as somebody who's more\nfamiliar with the allocation patterns (I happened to notice manually\nthat paths_to_free is not needed anymore, but I don't know if there are\nany other subtle bits).\n\n-Peff\n\n merge-ort.c | 151 +++++++++--------------------------\n 1 file changed, 37 insertions(+), 114 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 63829f5cac..441dc4e094 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -303,8 +303,6 @@ struct merge_options_internal {\n \t * * these keys serve to intern all the path strings, which allows\n \t * us to do pointer comparison on directory names instead of\n \t * strcmp; we just have to be careful to use the interned strings.\n-\t * (Technically paths_to_free may track some strings that were\n-\t * removed from froms paths.)\n \t *\n \t * The values of paths:\n \t * * either a pointer to a merged_info, or a conflict_info struct\n@@ -347,18 +345,7 @@ struct merge_options_internal {\n \t * freed together too. Using a memory pool for these provides a\n \t * nice speedup.\n \t */\n-\tstruct mem_pool internal_pool;\n-\tstruct mem_pool *pool; /* NULL, or pointer to internal_pool */\n-\n-\t/*\n-\t * paths_to_free: additional list of strings to free\n-\t *\n-\t * If keys are removed from \"paths\", they are added to paths_to_free\n-\t * to ensure they are later freed. We avoid free'ing immediately since\n-\t * other places (e.g. conflict_info.pathnames[]) may still be\n-\t * referencing these paths.\n-\t */\n-\tstruct string_list paths_to_free;\n+\tstruct mem_pool pool;\n \n \t/*\n \t * output: special messages and conflict notices for various paths\n@@ -537,19 +524,7 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \tvoid (*strset_clear_func)(struct strset *) =\n \t\treinitialize ? strset_partial_clear : strset_clear;\n \n-\tif (opti->pool)\n-\t\tstrmap_clear_func(&opti->paths, 0);\n-\telse {\n-\t\t/*\n-\t\t * We marked opti->paths with strdup_strings = 0, so that\n-\t\t * we wouldn't have to make another copy of the fullpath\n-\t\t * created by make_traverse_path from setup_path_info().\n-\t\t * But, now that we've used it and have no other references\n-\t\t * to these strings, it is time to deallocate them.\n-\t\t */\n-\t\tfree_strmap_strings(&opti->paths);\n-\t\tstrmap_clear_func(&opti->paths, 1);\n-\t}\n+\tstrmap_clear_func(&opti->paths, 0);\n \n \t/*\n \t * All keys and values in opti->conflicted are a subset of those in\n@@ -558,20 +533,6 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t */\n \tstrmap_clear_func(&opti->conflicted, 0);\n \n-\tif (!opti->pool) {\n-\t\t/*\n-\t\t * opti->paths_to_free is similar to opti->paths; we\n-\t\t * created it with strdup_strings = 0 to avoid making\n-\t\t * _another_ copy of the fullpath but now that we've used\n-\t\t * it and have no other references to these strings, it is\n-\t\t * time to deallocate them. We do so by temporarily\n-\t\t * setting strdup_strings to 1.\n-\t\t */\n-\t\topti->paths_to_free.strdup_strings = 1;\n-\t\tstring_list_clear(&opti->paths_to_free, 0);\n-\t\topti->paths_to_free.strdup_strings = 0;\n-\t}\n-\n \tif (opti->attr_index.cache_nr) /* true iff opt->renormalize */\n \t\tdiscard_index(&opti->attr_index);\n \n@@ -621,9 +582,7 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\tstrmap_clear(&opti->output, 0);\n \t}\n \n-\tmem_pool_discard(&opti->internal_pool, 0);\n-\tif (!reinitialize)\n-\t\topti->pool = NULL;\n+\tmem_pool_discard(&opti->pool, 0);\n \n \t/* Clean out callback_data as well. */\n \tFREE_AND_NULL(renames->callback_data);\n@@ -846,7 +805,7 @@ static void setup_path_info(struct merge_options *opt,\n \tassert(!df_conflict || !resolved); /* df_conflict implies !resolved */\n \tassert(resolved == (merged_version != NULL));\n \n-\tmi = mem_pool_calloc(opt->priv->pool, 1,\n+\tmi = mem_pool_calloc(&opt->priv->pool, 1,\n \t\t\t resolved ? sizeof(struct merged_info) :\n \t\t\t\t\tsizeof(struct conflict_info));\n \tmi->directory_name = current_dir_name;\n@@ -895,7 +854,7 @@ static void add_pair(struct merge_options *opt,\n \t\t unsigned dir_rename_mask)\n {\n \tstruct diff_filespec *one, *two;\n-\tstruct mem_pool *pool = opt->priv->pool;\n+\tstruct mem_pool *pool = &opt->priv->pool;\n \tstruct rename_info *renames = &opt->priv->renames;\n \tint names_idx = is_add ? side : 0;\n \n@@ -1141,7 +1100,7 @@ static int collect_merge_info_callback(int n,\n \tlen = traverse_path_len(info, p->pathlen);\n \n \t/* +1 in both of the following lines to include the NUL byte */\n-\tfullpath = mem_pool_alloc(opt->priv->pool, len + 1);\n+\tfullpath = mem_pool_alloc(&opt->priv->pool, len + 1);\n \tmake_traverse_path(fullpath, len + 1, info, p->path, p->pathlen);\n \n \t/*\n@@ -1396,7 +1355,7 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\tcopy = renames->deferred[side].possible_trivial_merges;\n \t\tstrintmap_init_with_options(&renames->deferred[side].possible_trivial_merges,\n \t\t\t\t\t 0,\n-\t\t\t\t\t opt->priv->pool,\n+\t\t\t\t\t &opt->priv->pool,\n \t\t\t\t\t 0);\n \t\tstrintmap_for_each_entry(©, &iter, entry) {\n \t\t\tconst char *path = entry->key;\n@@ -2348,19 +2307,15 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \tVERIFY_CI(ci);\n \n \t/* Find parent directories missing from opt->priv->paths */\n-\tif (opt->priv->pool) {\n-\t\tcur_path = mem_pool_strdup(opt->priv->pool, new_path);\n-\t\tfree((char*)new_path);\n-\t\tnew_path = (char *)cur_path;\n-\t} else {\n-\t\tcur_path = new_path;\n-\t}\n+\tcur_path = mem_pool_strdup(&opt->priv->pool, new_path);\n+\tfree((char*)new_path);\n+\tnew_path = (char *)cur_path;\n \n \twhile (1) {\n \t\t/* Find the parent directory of cur_path */\n \t\tchar *last_slash = strrchr(cur_path, '/');\n \t\tif (last_slash) {\n-\t\t\tparent_name = mem_pool_strndup(opt->priv->pool,\n+\t\t\tparent_name = mem_pool_strndup(&opt->priv->pool,\n \t\t\t\t\t\t cur_path,\n \t\t\t\t\t\t last_slash - cur_path);\n \t\t} else {\n@@ -2371,8 +2326,6 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\t/* Look it up in opt->priv->paths */\n \t\tentry = strmap_get_entry(&opt->priv->paths, parent_name);\n \t\tif (entry) {\n-\t\t\tif (!opt->priv->pool)\n-\t\t\t\tfree((char*)parent_name);\n \t\t\tparent_name = entry->key; /* reuse known pointer */\n \t\t\tbreak;\n \t\t}\n@@ -2399,16 +2352,6 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\tparent_name = cur_dir;\n \t}\n \n-\tif (!opt->priv->pool) {\n-\t\t/*\n-\t\t * We are removing old_path from opt->priv->paths.\n-\t\t * old_path also will eventually need to be freed, but it\n-\t\t * may still be used by e.g. ci->pathnames. So, store it\n-\t\t * in another string-list for now.\n-\t\t */\n-\t\tstring_list_append(&opt->priv->paths_to_free, old_path);\n-\t}\n-\n \tassert(ci->filemask == 2 || ci->filemask == 4);\n \tassert(ci->dirmask == 0);\n \tstrmap_remove(&opt->priv->paths, old_path, 0);\n@@ -2442,8 +2385,6 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\tnew_ci->stages[index].mode = ci->stages[index].mode;\n \t\toidcpy(&new_ci->stages[index].oid, &ci->stages[index].oid);\n \n-\t\tif (!opt->priv->pool)\n-\t\t\tfree(ci);\n \t\tci = new_ci;\n \t}\n \n@@ -2859,7 +2800,7 @@ static void use_cached_pairs(struct merge_options *opt,\n {\n \tstruct hashmap_iter iter;\n \tstruct strmap_entry *entry;\n-\tstruct mem_pool *pool = opt->priv->pool;\n+\tstruct mem_pool *pool = &opt->priv->pool;\n \n \t/*\n \t * Add to side_pairs all entries from renames->cached_pairs[side_index].\n@@ -2871,25 +2812,20 @@ static void use_cached_pairs(struct merge_options *opt,\n \t\tconst char *new_name = entry->value;\n \t\tif (!new_name)\n \t\t\tnew_name = old_name;\n-\t\tif (pool) {\n-\t\t\t/*\n-\t\t\t * cached_pairs has _copies* of old_name and new_name,\n-\t\t\t * because it has to persist across merges. When\n-\t\t\t * pool != NULL\n-\t\t\t * pool_alloc_filespec() will just re-use the existing\n-\t\t\t * filenames, which will also get re-used by\n-\t\t\t * opt->priv->paths if they become renames, and then\n-\t\t\t * get freed at the end of the merge, leaving the copy\n-\t\t\t * in cached_pairs dangling. Avoid this by making a\n-\t\t\t * copy here.\n-\t\t\t *\n-\t\t\t * When pool == NULL, pool_alloc_filespec() calls\n-\t\t\t * alloc_filespec(), which makes a copy; we don't want\n-\t\t\t * to add another.\n-\t\t\t */\n-\t\t\told_name = mem_pool_strdup(pool, old_name);\n-\t\t\tnew_name = mem_pool_strdup(pool, new_name);\n-\t\t}\n+\n+\t\t/*\n+\t\t * cached_pairs has _copies* of old_name and new_name,\n+\t\t * because it has to persist across merges.\n+\t\t *\n+\t\t * pool_alloc_filespec() will just re-use the existing\n+\t\t * filenames, which will also get re-used by\n+\t\t * opt->priv->paths if they become renames, and then\n+\t\t * get freed at the end of the merge, leaving the copy\n+\t\t * in cached_pairs dangling. Avoid this by making a\n+\t\t * copy here.\n+\t\t */\n+\t\told_name = mem_pool_strdup(pool, old_name);\n+\t\tnew_name = mem_pool_strdup(pool, new_name);\n \n \t\t/* We don't care about oid/mode, only filenames and status */\n \t\tone = pool_alloc_filespec(pool, old_name);\n@@ -3002,7 +2938,7 @@ static int detect_regular_renames(struct merge_options *opt,\n \tdiff_queued_diff = renames->pairs[side_index];\n \ttrace2_region_enter(\"diff\", \"diffcore_rename\", opt->repo);\n \tdiffcore_rename_extended(&diff_opts,\n-\t\t\t\t opt->priv->pool,\n+\t\t\t\t &opt->priv->pool,\n \t\t\t\t &renames->relevant_sources[side_index],\n \t\t\t\t &renames->dirs_removed[side_index],\n \t\t\t\t &renames->dir_rename_count[side_index],\n@@ -3053,7 +2989,7 @@ static int collect_renames(struct merge_options *opt,\n \n \t\tif (p->status != 'A' && p->status != 'R') {\n \t\t\tpossibly_cache_new_pair(renames, p, side_index, NULL);\n-\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n+\t\t\tpool_diff_free_filepair(&opt->priv->pool, p);\n \t\t\tcontinue;\n \t\t}\n \n@@ -3066,7 +3002,7 @@ static int collect_renames(struct merge_options *opt,\n \n \t\tpossibly_cache_new_pair(renames, p, side_index, new_path);\n \t\tif (p->status != 'R' && !new_path) {\n-\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n+\t\t\tpool_diff_free_filepair(&opt->priv->pool, p);\n \t\t\tcontinue;\n \t\t}\n \n@@ -3184,7 +3120,7 @@ static int detect_and_process_renames(struct merge_options *opt,\n \t\tside_pairs = &renames->pairs[s];\n \t\tfor (i = 0; i < side_pairs->nr; ++i) {\n \t\t\tstruct diff_filepair *p = side_pairs->queue[i];\n-\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n+\t\t\tpool_diff_free_filepair(&opt->priv->pool, p);\n \t\t}\n \t}\n \n@@ -3197,7 +3133,7 @@ static int detect_and_process_renames(struct merge_options *opt,\n \tif (combined.nr) {\n \t\tint i;\n \t\tfor (i = 0; i < combined.nr; i++)\n-\t\t\tpool_diff_free_filepair(opt->priv->pool,\n+\t\t\tpool_diff_free_filepair(&opt->priv->pool,\n \t\t\t\t\t\tcombined.queue[i]);\n \t\tfree(combined.queue);\n \t}\n@@ -3672,7 +3608,7 @@ static void process_entry(struct merge_options *opt,\n \t\t * the directory to remain here, so we need to move this\n \t\t * path to some new location.\n \t\t */\n-\t\tnew_ci = mem_pool_calloc(opt->priv->pool, 1, sizeof(*new_ci));\n+\t\tnew_ci = mem_pool_calloc(&opt->priv->pool, 1, sizeof(*new_ci));\n \n \t\t/* We don't really want new_ci->merged.result copied, but it'll\n \t\t * be overwritten below so it doesn't matter. We also don't\n@@ -3765,7 +3701,7 @@ static void process_entry(struct merge_options *opt,\n \t\t\tconst char *a_path = NULL, *b_path = NULL;\n \t\t\tint rename_a = 0, rename_b = 0;\n \n-\t\t\tnew_ci = mem_pool_alloc(opt->priv->pool,\n+\t\t\tnew_ci = mem_pool_alloc(&opt->priv->pool,\n \t\t\t\t\t\tsizeof(*new_ci));\n \n \t\t\tif (S_ISREG(a_mode))\n@@ -3835,19 +3771,8 @@ static void process_entry(struct merge_options *opt,\n \t\t\t\tb_path = path;\n \t\t\tstrmap_put(&opt->priv->paths, b_path, new_ci);\n \n-\t\t\tif (rename_a && rename_b) {\n+\t\t\tif (rename_a && rename_b)\n \t\t\t\tstrmap_remove(&opt->priv->paths, path, 0);\n-\t\t\t\t/*\n-\t\t\t\t * We removed path from opt->priv->paths. path\n-\t\t\t\t * will also eventually need to be freed if not\n-\t\t\t\t * part of a memory pool...but it may still be\n-\t\t\t\t * used by e.g. ci->pathnames. So, store it in\n-\t\t\t\t * another string-list for now in that case.\n-\t\t\t\t */\n-\t\t\t\tif (!opt->priv->pool)\n-\t\t\t\t\tstring_list_append(&opt->priv->paths_to_free,\n-\t\t\t\t\t\t\t path);\n-\t\t\t}\n \n \t\t\t/*\n \t\t\t * Do special handling for b_path since process_entry()\n@@ -4454,8 +4379,8 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \n \t/* Initialization of various renames fields */\n \trenames = &opt->priv->renames;\n-\tmem_pool_init(&opt->priv->internal_pool, 0);\n-\tpool = opt->priv->pool = &opt->priv->internal_pool;\n+\tmem_pool_init(&opt->priv->pool, 0);\n+\tpool = &opt->priv->pool;\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->dirs_removed[i],\n \t\t\t\t\t NOT_RELEVANT, pool, 0);\n@@ -4492,15 +4417,13 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t * Although we initialize opt->priv->paths with strdup_strings=0,\n \t * that's just to avoid making yet another copy of an allocated\n \t * string. Putting the entry into paths means we are taking\n-\t * ownership, so we will later free it. paths_to_free is similar.\n+\t * ownership, so we will later free it.\n \t *\n \t * In contrast, conflicted just has a subset of keys from paths, so\n \t * we don't want to free those (it'd be a duplicate free).\n \t */\n \tstrmap_init_with_options(&opt->priv->paths, pool, 0);\n \tstrmap_init_with_options(&opt->priv->conflicted, pool, 0);\n-\tif (!opt->priv->pool)\n-\t\tstring_list_init_nodup(&opt->priv->paths_to_free);\n \n \t/*\n \t * keys & strbufs in output will sometimes need to outlive \"paths\",\n"},{"id":"431586","messageId":"878s1nrbxn.fsf@evledraar.gmail.com","threadId":"56152","inReplyTo":"CABPp-BHc16hjYz-+073LCA_cxHP22zW2QO6ikKKcXpG8zS1gWQ@mail.gmail.com","subject":"Re: [PATCH v2 4/7] merge-ort: switch our strmaps over to using memory pools","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-30T16:23:33Z","receivedAt":"2021-07-30T16:41:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jul 30 2021, Elijah Newren wrote:\n\n> Hi Ævar,\n>\n> On Fri, Jul 30, 2021 at 7:33 AM Ævar Arnfjörð Bjarmason\n> wrote:\n>>\n>>\n>> On Thu, Jul 29 2021, Jeff King wrote:\n>>\n>> > On Thu, Jul 29, 2021 at 12:37:52PM -0600, Elijah Newren wrote:\n>> >\n>> >> > Arguably, the existence of these function indirections is perhaps a sign\n>> >> > that the strmap API should provide a version of the clear functions that\n>> >> > takes \"partial / not-partial\" as a parameter.\n>> >>\n>> >> Are you suggesting a modification of str{map,intmap,set}_clear() to\n>> >> take an extra parameter, or removing the\n>> >> str{map,intmap,set}_partial_clear() functions and introducing new\n>> >> functions that take a partial/not-partial parameter? I think you're\n>> >> suggesting the latter, and that makes more sense to me...but I'm\n>> >> drawing blanks trying to come up with a reasonable function name.\n>> >\n>> > It does seem a shame to add the \"partial\" parameter to strmap_clear(),\n>> > just because most callers don't need it (so they end up with this\n>> > inscrutable \"0\" parameter).\n>> >\n>> > What if there was a flags field? Then it could be combined with the\n>> > free_values parameter. The result is kind of verbose in two ways:\n>> >\n>> > - now strset_clear(), etc, need a \"flags\" parameter, which they didn't\n>> > before (and is just \"0\" most of the time!)\n>> >\n>> > - now \"strmap_clear(foo, 1)\" becomes \"strmap_clear(foo, STRMAP_FREE_VALUES)\".\n>> > That's a lot longer, though arguably it's easier to understand since\n>> > the boolean is explained.\n>> >\n>> > Having gone through the exercise, I am not sure it is actually making\n>> > anything more readable (messy patch is below for reference).\n>>\n>> I've got some WIP patches for string-list.h and strmap.h to make the API\n>> nicer, and it's probably applicable to strset.h too.\n>\n> There is no strset.h; strset and strintmap along with strmap are part\n> of strmap.h.\n>\n>> I.e. I found when using strset.h that it was a weird API to use, because\n>> unlike string-list.h it didn't pay attention to your \"dup\" field when\n>> freeing, you had to do it explicitly.\n>\n> Do you mean strmap.h instead of strset.h?\n\nYes, sorry. Brainfart.\n\n> In general, if you are asking strmap/strset/strintmap to dup your keys\n> and are explicitly freeing the strings, then you are misusing the API\n> and either freeing pointers that were never allocated or getting\n> double frees. It's wrong to explicitly deallocate them because:\n> * When using a pool, we just allocate from the pool. The memory\n> will be freed when the pool is freed.\n> * When not using a pool, we use FLEXPTR_ALLOC_STR in order to make\n> the string be part of the allocated strmap_entry. The string's memory\n> is deallocated when the strmap_entry is.\n>\n> The only reason to explicitly free keys in a strmap/strset/strintmap\n> is if you do NOT have strdup_strings set and allocated the strings\n> elsewhere and left your strmap as the only thing tracking the strings.\n\nYes, sorry. I think I was trying to address that case, i.e. it started\nwith fixing some memory leaks, but it's part of a branch of mine that's\nin a messy WIP state of not passing the tests. Please ignore the rest of\nthe hunks to do with strmap.h.\n\nI might have some worthwhile fixes there, maybe not. It mainly started\nwith carrying things over from similar changes in string-list.h.\n\n>> And then in e.g. merge-ort.c there's this \"strdup dance\" pattern where\n>> we flip the field back and forth.\n>>\n>> The below diff is exctracted from that WIP work, with the relevant two\n>> API headers and then two changed API users for show (the tree-wide\n>> changes are much larger).\n>>\n>> I think making the promise I make in the updated docs at \"We guarantee\n>> that the `clearfunc`[...]\" in string-list.h makes for particularly nice\n>> API behavior.\n>>\n>> builtin/remote.c | 37 ++++++++++++++++++++---------------\n>> merge-ort.c | 32 +++++++-----------------------\n>> string-list.h | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n>> strmap.h | 13 +++++++++++++\n>> 4 files changed, 98 insertions(+), 43 deletions(-)\n>>\n>> diff --git a/builtin/remote.c b/builtin/remote.c\n>> index 7f88e6ce9de..ec1dbd49f71 100644\n>> --- a/builtin/remote.c\n>> +++ b/builtin/remote.c\n>> @@ -340,10 +340,24 @@ static void read_branches(void)\n>>\n>> struct ref_states {\n>> struct remote *remote;\n>> - struct string_list new_refs, stale, tracked, heads, push;\n>> +\n>> + struct string_list new_refs;\n>> + struct string_list stale;\n>> + struct string_list tracked;\n>> + struct string_list heads;\n>> + struct string_list push;\n>> +\n>> int queried;\n>> };\n>>\n>> +#define REF_STATES_INIT { \\\n>> + .new_refs = STRING_LIST_INIT_DUP, \\\n>> + .stale = STRING_LIST_INIT_DUP, \\\n>> + .tracked = STRING_LIST_INIT_DUP, \\\n>> + .heads = STRING_LIST_INIT_DUP, \\\n>> + .push = STRING_LIST_INIT_DUP, \\\n>> +}\n>> +\n>> static int get_ref_states(const struct ref *remote_refs, struct ref_states *states)\n>> {\n>> struct ref *fetch_map = NULL, **tail = &fetch_map;\n>> @@ -355,9 +369,6 @@ static int get_ref_states(const struct ref *remote_refs, struct ref_states *stat\n>> die(_(\"Could not get fetch map for refspec %s\"),\n>> states->remote->fetch.raw[i]);\n>>\n>> - states->new_refs.strdup_strings = 1;\n>> - states->tracked.strdup_strings = 1;\n>> - states->stale.strdup_strings = 1;\n>> for (ref = fetch_map; ref; ref = ref->next) {\n>> if (!ref->peer_ref || !ref_exists(ref->peer_ref->name))\n>> string_list_append(&states->new_refs, abbrev_branch(ref->name));\n>> @@ -406,7 +417,6 @@ static int get_push_ref_states(const struct ref *remote_refs,\n>>\n>> match_push_refs(local_refs, &push_map, &remote->push, MATCH_REFS_NONE);\n>>\n>> - states->push.strdup_strings = 1;\n>> for (ref = push_map; ref; ref = ref->next) {\n>> struct string_list_item *item;\n>> struct push_info *info;\n>> @@ -449,7 +459,6 @@ static int get_push_ref_states_noquery(struct ref_states *states)\n>> if (remote->mirror)\n>> return 0;\n>>\n>> - states->push.strdup_strings = 1;\n>> if (!remote->push.nr) {\n>> item = string_list_append(&states->push, _(\"(matching)\"));\n>> info = item->util = xcalloc(1, sizeof(struct push_info));\n>> @@ -483,7 +492,6 @@ static int get_head_names(const struct ref *remote_refs, struct ref_states *stat\n>> refspec.force = 0;\n>> refspec.pattern = 1;\n>> refspec.src = refspec.dst = \"refs/heads/*\";\n>> - states->heads.strdup_strings = 1;\n>> get_fetch_map(remote_refs, &refspec, &fetch_map_tail, 0);\n>> matches = guess_remote_head(find_ref_by_name(remote_refs, \"HEAD\"),\n>> fetch_map, 1);\n>> @@ -905,7 +913,7 @@ static void clear_push_info(void *util, const char *string)\n>> {\n>> struct push_info *info = util;\n>> free(info->dest);\n>> - free(info);\n>> + /* note: fixed memleak here */\n>> }\n>>\n>> static void free_remote_ref_states(struct ref_states *states)\n>> @@ -1159,7 +1167,7 @@ static int get_one_entry(struct remote *remote, void *priv)\n>> string_list_append(list, remote->name)->util =\n>> strbuf_detach(&url_buf, NULL);\n>> } else\n>> - string_list_append(list, remote->name)->util = NULL;\n>> + string_list_append(list, remote->name);\n>> if (remote->pushurl_nr) {\n>> url = remote->pushurl;\n>> url_nr = remote->pushurl_nr;\n>> @@ -1179,10 +1187,9 @@ static int get_one_entry(struct remote *remote, void *priv)\n>>\n>> static int show_all(void)\n>> {\n>> - struct string_list list = STRING_LIST_INIT_NODUP;\n>> + struct string_list list = STRING_LIST_INIT_DUP;\n>> int result;\n>>\n>> - list.strdup_strings = 1;\n>> result = for_each_remote(get_one_entry, &list);\n>>\n>> if (!result) {\n>> @@ -1212,7 +1219,7 @@ static int show(int argc, const char **argv)\n>> OPT_BOOL('n', NULL, &no_query, N_(\"do not query remotes\")),\n>> OPT_END()\n>> };\n>> - struct ref_states states;\n>> + struct ref_states states = REF_STATES_INIT;\n>> struct string_list info_list = STRING_LIST_INIT_NODUP;\n>> struct show_info info;\n>>\n>> @@ -1334,8 +1341,7 @@ static int set_head(int argc, const char **argv)\n>> if (!opt_a && !opt_d && argc == 2) {\n>> head_name = xstrdup(argv[1]);\n>> } else if (opt_a && !opt_d && argc == 1) {\n>> - struct ref_states states;\n>> - memset(&states, 0, sizeof(states));\n>> + struct ref_states states = REF_STATES_INIT;\n>> get_remote_ref_states(argv[0], &states, GET_HEAD_NAMES);\n>> if (!states.heads.nr)\n>> result |= error(_(\"Cannot determine remote HEAD\"));\n>> @@ -1374,14 +1380,13 @@ static int set_head(int argc, const char **argv)\n>> static int prune_remote(const char *remote, int dry_run)\n>> {\n>> int result = 0;\n>> - struct ref_states states;\n>> + struct ref_states states = REF_STATES_INIT;\n>> struct string_list refs_to_prune = STRING_LIST_INIT_NODUP;\n>> struct string_list_item *item;\n>> const char *dangling_msg = dry_run\n>> ? _(\" %s will become dangling!\")\n>> : _(\" %s has become dangling!\");\n>>\n>> - memset(&states, 0, sizeof(states));\n>> get_remote_ref_states(remote, &states, GET_REF_STATES);\n>>\n>> if (!states.stale.nr) {\n>\n> Everything up to here looks like a very nice cleanup.\n>\n>> diff --git a/merge-ort.c b/merge-ort.c\n>> index ec0c5904211..53ed78e7a01 100644\n>> --- a/merge-ort.c\n>> +++ b/merge-ort.c\n>> @@ -432,16 +432,6 @@ struct conflict_info {\n>> assert((ci) && !(mi)->clean); \\\n>> } while (0)\n>>\n>> -static void free_strmap_strings(struct strmap *map)\n>> -{\n>> - struct hashmap_iter iter;\n>> - struct strmap_entry *entry;\n>> -\n>> - strmap_for_each_entry(map, &iter, entry) {\n>> - free((char*)entry->key);\n>> - }\n>> -}\n>> -\n>> static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n>> int reinitialize)\n>> {\n>> @@ -455,13 +445,11 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n>> reinitialize ? strset_partial_clear : strset_clear;\n>>\n>> /*\n>> - * We marked opti->paths with strdup_strings = 0, so that we\n>> - * wouldn't have to make another copy of the fullpath created by\n>> - * make_traverse_path from setup_path_info(). But, now that we've\n>> - * used it and have no other references to these strings, it is time\n>> - * to deallocate them.\n>> + * We used the the pattern of re-using already allocated\n>> + * strings strmap_clear_strings() in make_traverse_path from\n>> + * setup_path_info(). Deallocate them.\n>> */\n>> - free_strmap_strings(&opti->paths);\n>> + strmap_clear_strings(&opti->paths, 0);\n>> strmap_func(&opti->paths, 1);\n>>\n>> /*\n>\n> It's not clear to me that strmap should handle the freeing of the keys\n> at all; maybe it should and strmap_clear_strings() makes sense to\n> introduce. However, this change is clearly wrong regardless, for two\n> reasons: (1) You are double clearing since strmap_func() is also\n> called afterwards, and (2) you are also ignoring the potential partial\n> bit since strmap_func might be strmap_partial_clear() rather than\n> strmap_clear().\n\n*Nod*, see above.\n\n>> @@ -472,15 +460,10 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n>> strmap_func(&opti->conflicted, 0);\n>>\n>> /*\n>> - * opti->paths_to_free is similar to opti->paths; we created it with\n>> - * strdup_strings = 0 to avoid making _another_ copy of the fullpath\n>> - * but now that we've used it and have no other references to these\n>> - * strings, it is time to deallocate them. We do so by temporarily\n>> - * setting strdup_strings to 1.\n>> + * opti->paths_to_free is similar to opti->paths; it's memory\n>> + * we borrowed and need to free with string_list_clear_strings().\n>> */\n>> - opti->paths_to_free.strdup_strings = 1;\n>> - string_list_clear(&opti->paths_to_free, 0);\n>> - opti->paths_to_free.strdup_strings = 0;\n>> + string_list_clear_strings(&opti->paths_to_free, 0);\n>\n> This is very nice. I really like this new function and API.\n>\n>> if (opti->attr_index.cache_nr) /* true iff opt->renormalize */\n>> discard_index(&opti->attr_index);\n>> @@ -2664,7 +2647,6 @@ static int collect_renames(struct merge_options *opt,\n>> * and have no other references to these strings, it is time to\n>> * deallocate them.\n>> */\n>> - free_strmap_strings(&collisions);\n>> strmap_clear(&collisions, 1);\n>> return clean;\n>> }\n>\n> This hunk is wrong.\n\n*nod*\n\n>> diff --git a/string-list.h b/string-list.h\n>> index 0d6b4692396..9eeea996888 100644\n>> --- a/string-list.h\n>> +++ b/string-list.h\n>> @@ -109,6 +109,9 @@ void string_list_init_dup(struct string_list *list);\n>> */\n>> void string_list_init(struct string_list *list, int strdup_strings);\n>>\n>> +void string_list_cmp_init(struct string_list *list, int strdup_strings,\n>> + compare_strings_fn cmp);\n>> +\n>\n> Seems unrelated to what you were trying to highlight?\n\nYes, sorry. I just extracted all the diff WIP diff I had for\nstring-list.h, this bit was unrelated.\n\nIt's unrelated cleanup of various things that hardcoded all the fields\nduring their \"init\", just because they need strcasecmp or whatever\ninstead of strcmp.\n\n>> /** Callback function type for for_each_string_list */\n>> typedef int (*string_list_each_func_t)(struct string_list_item *, void *);\n>>\n>> @@ -129,14 +132,66 @@ void filter_string_list(struct string_list *list, int free_util,\n>> */\n>> void string_list_clear(struct string_list *list, int free_util);\n>>\n>> +/**\n>> + * Free a string list initialized without `strdup_strings = 1`, but\n>> + * where we also want to free() the strings. You usually want to just\n>> + * use string_list_clear() after initializing with\n>> + * `STRING_LIST_INIT_DUP' instead.\n>> + *\n>> + * Useful to free e.g. a string list whose strings came from\n>> + * strbuf_detach() or other memory that we didn't initially allocate\n>> + * on the heap, but which we now manage.\n>> + *\n>> + * Under the hood this is identical in behavior to temporarily setting\n>> + * `strbuf_strings` to `1` for the duration of this function call, but\n>> + * without the verbosity of performing that dance yourself.\n>> + */\n>> +void string_list_clear_strings(struct string_list *list, int free_util);\n>> +\n>> +/**\n>> + * Clear only the `util` pointer, but not the `string`, even if\n>> + * `strdup_strings = 1` is set. Useful for the idiom of doing e.g.:\n>> + *\n>> + * string_list_append(&list, str + offs)->util = str;\n>> + *\n>> + * Where we add a string at some offset, own the string (so\n>> + * effectively `strdup_strings = `), but can't free() the string\n>> + * itself at the changed offset, but need to free the original data in\n>> + * `util` instead.\n>> + */\n>> +void string_list_clear_util(struct string_list *list);\n>> +\n>> /**\n>> * Callback type for `string_list_clear_func`. The string associated\n>> * with the util pointer is passed as the second argument\n>> */\n>> typedef void (*string_list_clear_func_t)(void *p, const char *str);\n>>\n>> -/** Call a custom clear function on each util pointer */\n>> -void string_list_clear_func(struct string_list *list, string_list_clear_func_t clearfunc);\n>> +/**\n>> + * Like string_list_clear() except that it first calls a custom clear\n>> + * function on each util pointer.\n>> + *\n>> + * We guarantee that the `clearfunc` will be called on all util\n>> + * pointers in a list before we proceed to free the first string or\n>> + * util pointer, i.e. should you need to it's OK to peek at other util\n>> + * items in the list itself, or to otherwise iterate it from within\n>> + * the `clearfunc`.\n>> + *\n>> + * You do not need to free() the passed-in util pointer itself,\n>> + * i.e. after calling all `clearfunc` this has the seme behavior as\n>> + * string_list_clear() called with with `free_util = 1`.\n>> + */\n>> +void string_list_clear_func(struct string_list *list,\n>> + string_list_clear_func_t clearfunc);\n>> +\n>> +/**\n>> + * Like string_list_clear_func() but free the strings too, using the\n>> + * same dance as described for string_list_clear_strings()\n>> + * above. You'll usually want to initialize with\n>> + * `STRING_LIST_INIT_DUP` and use string_list_clear_strings() instead.\n>> + */\n>> +void string_list_clear_strings_func(struct string_list *list,\n>> + string_list_clear_func_t clearfunc);\n>>\n>> /**\n>> * Apply `func` to each item. If `func` returns nonzero, the\n>\n> string_list_clear_strings() looks very nice. The others are probably\n> good too, though I'm curious about the need for double walking the\n> list to free it instead of doing it in a single walk; what callers\n> need to walk the list and check out other values?\n\nI think there were a couple of users that needed that, but maybe I'm\nwrong. I think even if there's not clearly defining the callback freeing\nsemantics makes sense for caller sanity.\n\nWe double-walk now in string_list_clear(), this is mostly documenting\nand extending current behavior to being able to clear any arbitrary\ncombination of string/util with an optional cb, regardless of your\nstrdup_strings state.\n\n>> diff --git a/strmap.h b/strmap.h\n>> index 1e152d832d6..337f6278e86 100644\n>> --- a/strmap.h\n>> +++ b/strmap.h\n>> @@ -51,12 +51,25 @@ void strmap_init_with_options(struct strmap *map,\n>> */\n>> void strmap_clear(struct strmap *map, int free_values);\n>>\n>> +/**\n>> + * To strmap_clear() what string_list_clear_strings() is to\n>> + * string_list_clear(). I.e. free your keys too, which we used as-is\n>> + * without `strdup_strings = 1`.\n>> + */\n>> +void strmap_clear_strings(struct strmap *map, int free_values);\n>\n> strmap.h doesn't depend on string-list.h, so the comment should be\n> self-standing. The analogy also doesn't seem to hold since we do NOT\n> need to free the keys when strdup_strings is 1; Peff suggested\n> FLEXPTR_ALLOC_STR specifically to avoid that extra allocation in that\n> case.\n\n*nod*, see above (i.e. maybe all this strmap.h stuff is wrong). FWIW\nthat comment was meant as a \"if you're familiar with X in API A, this Y\nin B works similarly\".\n\n>> +\n>> /*\n>> * Similar to strmap_clear() but leaves map->map->table allocated and\n>> * pre-sized so that subsequent uses won't need as many rehashings.\n>> */\n>> void strmap_partial_clear(struct strmap *map, int free_values);\n>>\n>> +/**\n>> + * To strmap_partial_clear() what string_list_clear_strings() is to\n>> + * string_list_clear(). See strmap_clear_strings() above.\n>> + */\n>> +void strmap_partial_clear_strings(struct strmap *map, int free_values);\n>> +\n>\n> Same comment as above for strmap_clear_strings() applies here.\n\n*nod*\n"},{"id":"431629","messageId":"8416afa89fb9d1e638bf6f52403f113c82de7424.1627752458.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v4.git.1627752458.gitgitgadget@gmail.com","subject":"[PATCH v4 2/9] diffcore-rename: use a mem_pool for exact rename detection's hashmap","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-31T17:27:31Z","receivedAt":"2021-07-31T17:27:45Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nExact rename detection, via insert_file_table(), uses a hashmap to store\nfiles by oid. Use a mem_pool for the hashmap entries so these can all be\nallocated and deallocated together.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 204.2 ms ± 3.0 ms 202.5 ms ± 3.2 ms\n mega-renames: 1.076 s ± 0.015 s 1.072 s ± 0.012 s\n just-one-mega: 364.1 ms ± 7.0 ms 357.3 ms ± 3.9 ms\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 22 ++++++++++++++++------\n 1 file changed, 16 insertions(+), 6 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 4ef0459cfb5..73d884099eb 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -317,10 +317,11 @@ static int find_identical_files(struct hashmap *srcs,\n }\n \n static void insert_file_table(struct repository *r,\n+\t\t\t struct mem_pool *pool,\n \t\t\t struct hashmap *table, int index,\n \t\t\t struct diff_filespec *filespec)\n {\n-\tstruct file_similarity *entry = xmalloc(sizeof(*entry));\n+\tstruct file_similarity *entry = mem_pool_alloc(pool, sizeof(*entry));\n \n \tentry->index = index;\n \tentry->filespec = filespec;\n@@ -336,7 +337,8 @@ static void insert_file_table(struct repository *r,\n * and then during the second round we try to match\n * cache-dirty entries as well.\n */\n-static int find_exact_renames(struct diff_options *options)\n+static int find_exact_renames(struct diff_options *options,\n+\t\t\t struct mem_pool *pool)\n {\n \tint i, renames = 0;\n \tstruct hashmap file_table;\n@@ -346,7 +348,7 @@ static int find_exact_renames(struct diff_options *options)\n \t */\n \thashmap_init(&file_table, NULL, NULL, rename_src_nr);\n \tfor (i = rename_src_nr-1; i >= 0; i--)\n-\t\tinsert_file_table(options->repo,\n+\t\tinsert_file_table(options->repo, pool,\n \t\t\t\t &file_table, i,\n \t\t\t\t rename_src[i].p->one);\n \n@@ -354,8 +356,8 @@ static int find_exact_renames(struct diff_options *options)\n \tfor (i = 0; i < rename_dst_nr; i++)\n \t\trenames += find_identical_files(&file_table, i, options);\n \n-\t/* Free the hash data structure and entries */\n-\thashmap_clear_and_free(&file_table, struct file_similarity, entry);\n+\t/* Free the hash data structure (entries will be freed with the pool) */\n+\thashmap_clear(&file_table);\n \n \treturn renames;\n }\n@@ -1341,6 +1343,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \tint num_destinations, dst_cnt;\n \tint num_sources, want_copies;\n \tstruct progress *progress = NULL;\n+\tstruct mem_pool local_pool;\n \tstruct dir_rename_info info;\n \tstruct diff_populate_filespec_options dpf_options = {\n \t\t.check_binary = 0,\n@@ -1409,11 +1412,18 @@ void diffcore_rename_extended(struct diff_options *options,\n \t\tgoto cleanup; /* nothing to do */\n \n \ttrace2_region_enter(\"diff\", \"exact renames\", options->repo);\n+\tmem_pool_init(&local_pool, 32*1024);\n \t/*\n \t * We really want to cull the candidates list early\n \t * with cheap tests in order to avoid doing deltas.\n \t */\n-\trename_count = find_exact_renames(options);\n+\trename_count = find_exact_renames(options, &local_pool);\n+\t/*\n+\t * Discard local_pool immediately instead of at \"cleanup:\" in order\n+\t * to reduce maximum memory usage; inexact rename detection uses up\n+\t * a fair amount of memory, and mem_pools can too.\n+\t */\n+\tmem_pool_discard(&local_pool, 0);\n \ttrace2_region_leave(\"diff\", \"exact renames\", options->repo);\n \n \t/* Did we only want exact renames? */\n-- \ngitgitgadget\n\n"},{"id":"431631","messageId":"pull.990.v4.git.1627752458.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v3.git.1627645664.gitgitgadget@gmail.com","subject":"[PATCH v4 0/9] Final optimization batch (#15): use memory pools","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-31T17:27:29Z","receivedAt":"2021-07-31T17:27:46Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series textually depends on en/ort-perf-batch-14, but the ideas are\northogonal to it and orthogonal to previous series. It can be reviewed\nindependently.\n\nChanges since v1, addressing Eric's feedback:\n\n * Fixed a comment that became out-of-date in patch 1\n * Swapped commits 2 and 3 so that one can better motivate the other.\n\nChanges since v2, addressing Peff's feedback:\n\n * Rebased on en/ort-perf-batch-14 (resolving a trivial conflict with the\n new string_list_init_nodup() usage)\n * Added a new preliminary patch renaming str*_func() to str*_clear_func()\n * Added a new final patch that hardcodes that we'll just use memory pools\n\nChanges since v3, as per Peff's feedback:\n\n * Don't only remove the extra complexity from the USE_MEMORY_POOL #define;\n also remove the original bookkeeping complexity needed to track\n individual frees when not using a memory pool.\n\n=== Basic Optimization idea ===\n\nIn this series, I make use of memory pools to get faster allocations and\ndeallocations for many data structures that tend to all be deallocated at\nthe same time anyway.\n\n=== Results ===\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28), the\nchanges in just this series improves the performance as follows:\n\n Before Series After Series\nno-renames: 204.2 ms ± 3.0 ms 198.3 ms ± 2.9 ms\nmega-renames: 1.076 s ± 0.015 s 661.8 ms ± 5.9 ms\njust-one-mega: 364.1 ms ± 7.0 ms 264.6 ms ± 2.5 ms\n\n\nAs a reminder, before any merge-ort/diffcore-rename performance work, the\nperformance results we started with were:\n\nno-renames-am: 6.940 s ± 0.485 s\nno-renames: 18.912 s ± 0.174 s\nmega-renames: 5964.031 s ± 10.459 s\njust-one-mega: 149.583 s ± 0.751 s\n\n\n=== Overall Results across all optimization work ===\n\nThis is my final prepared optimization series. It might be worth reviewing\nhow my optimizations fared overall, comparing the original merge-recursive\ntimings with three things: how much merge-recursive improved (as a\nside-effect of optimizing merge-ort), how much improvement we would have\ngotten from a hypothetical infinite parallelization of rename detection, and\nwhat I achieved at the end with merge-ort:\n\n Timings\n\n Infinite\n merge- merge- Parallelism\n recursive recursive of rename merge-ort\n v2.30.0 current detection current\n ---------- --------- ----------- ---------\nno-renames: 18.912 s 18.030 s 11.699 s 198.3 ms\nmega-renames: 5964.031 s 361.281 s 203.886 s 661.8 ms\njust-one-mega: 149.583 s 11.009 s 7.553 s 264.6 ms\n\n Speedup factors\n\n Infinite\n merge- merge- Parallelism\n recursive recursive of rename\n v2.30.0 current detection merge-ort\n ---------- --------- ----------- ---------\nno-renames: 1 1.05 1.6 95\nmega-renames: 1 16.5 29 9012\njust-one-mega: 1 13.6 20 565\n\n\nAnd, for partial clone users:\n\n Factor reduction in number of objects needed\n\n Infinite\n merge- merge- Parallelism\n recursive recursive of rename\n v2.30.0 current detection merge-ort\n ---------- --------- ----------- ---------\nmega-renames: 1 1 1 181.3\n\n\n=== Caveat ===\n\nIt may be worth noting, though, that my optimization numbers above for\nmerge-ort use test-tool fast-rebase. git rebase -s ort on the three\ntestcases above is 5-20 times slower (taking 3.835s, 6.798s, and 1.235s,\nrespectively). At this point, any further optimization work should go into\nmaking a faster full-featured rebase by copying the ideas from fast-rebase:\navoid unnecessary process forking, avoid updating the index and working copy\nuntil either the rebase is finished or you hit a conflict (and don't write\nrebase metadata to disk until that point either), get rid of the glacially\nslow revision walking of the upstream side of history (nuke\ncan_fast_forward(), make --reapply-cherry-picks the default) or at least\ndon't revision walk so many times (multiple calls to get_merge_bases in\ncan_fast_forward() plus a is_linear_history() walk, checking for upstream\ncherry-picks, probably more), turn off per-commit hooks that probably should\nhave never been on anyway, etc.\n\nElijah Newren (9):\n merge-ort: rename str{map,intmap,set}_func()\n diffcore-rename: use a mem_pool for exact rename detection's hashmap\n merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers\n merge-ort: set up a memory pool\n merge-ort: switch our strmaps over to using memory pools\n diffcore-rename, merge-ort: add wrapper functions for filepair\n alloc/dealloc\n merge-ort: store filepairs and filespecs in our mem_pool\n merge-ort: reuse path strings in pool_alloc_filespec\n merge-ort: remove compile-time ability to turn off usage of memory\n pools\n\n diffcore-rename.c | 68 ++++++++++++++---\n diffcore.h | 3 +\n merge-ort.c | 188 +++++++++++++++++++++++++---------------------\n 3 files changed, 165 insertions(+), 94 deletions(-)\n\n\nbase-commit: 8b09a900a1f1f00d4deb04f567994ae8f1804b5e\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-990%2Fnewren%2Fort-perf-batch-15-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-990/newren/ort-perf-batch-15-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/990\n\nRange-diff vs v3:\n\n 1: e075d985f26 = 1: e075d985f26 merge-ort: rename str{map,intmap,set}_func()\n 2: 8416afa89fb = 2: 8416afa89fb diffcore-rename: use a mem_pool for exact rename detection's hashmap\n 3: 2c0b90eaba5 = 3: 2c0b90eaba5 merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers\n 4: 6646f6fd1ca = 4: 6646f6fd1ca merge-ort: set up a memory pool\n 5: 7c49aa601d0 = 5: 7c49aa601d0 merge-ort: switch our strmaps over to using memory pools\n 6: 08cf2498f96 = 6: 08cf2498f96 diffcore-rename, merge-ort: add wrapper functions for filepair alloc/dealloc\n 7: 4ffa5af8b57 = 7: 4ffa5af8b57 merge-ort: store filepairs and filespecs in our mem_pool\n 8: 1556f0443c3 = 8: 1556f0443c3 merge-ort: reuse path strings in pool_alloc_filespec\n 9: de30dbac25e ! 9: f8cd50794e9 merge-ort: remove compile-time ability to turn off usage of memory pools\n @@ Metadata\n ## Commit message ##\n merge-ort: remove compile-time ability to turn off usage of memory pools\n \n - Simplify code maintenance a bit by removing the ability to toggle\n - between usage of memory pools and direct allocations. This allows us to\n - also remove and simplify some auxiliary functions.\n + Simplify code maintenance by removing the ability to toggle between\n + usage of memory pools and direct allocations. This allows us to also\n + remove paths_to_free since it was solely about bookkeeping to make sure\n + we freed the necessary paths, and allows us to remove some auxiliary\n + functions.\n \n + Suggested-by: Jeff King \n Signed-off-by: Elijah Newren \n \n ## merge-ort.c ##\n @@ merge-ort.c\n /*\n * We have many arrays of size 3. Whenever we have such an array, the\n * indices refer to one of the sides of the three-way merge. This is so\n +@@ merge-ort.c: struct merge_options_internal {\n + \t * * these keys serve to intern all the path strings, which allows\n + \t * us to do pointer comparison on directory names instead of\n + \t * strcmp; we just have to be careful to use the interned strings.\n +-\t * (Technically paths_to_free may track some strings that were\n +-\t * removed from froms paths.)\n + \t *\n + \t * The values of paths:\n + \t * * either a pointer to a merged_info, or a conflict_info struct\n +@@ merge-ort.c: struct merge_options_internal {\n + \t * freed together too. Using a memory pool for these provides a\n + \t * nice speedup.\n + \t */\n +-\tstruct mem_pool internal_pool;\n +-\tstruct mem_pool *pool; /* NULL, or pointer to internal_pool */\n +-\n +-\t/*\n +-\t * paths_to_free: additional list of strings to free\n +-\t *\n +-\t * If keys are removed from \"paths\", they are added to paths_to_free\n +-\t * to ensure they are later freed. We avoid free'ing immediately since\n +-\t * other places (e.g. conflict_info.pathnames[]) may still be\n +-\t * referencing these paths.\n +-\t */\n +-\tstruct string_list paths_to_free;\n ++\tstruct mem_pool pool;\n + \n + \t/*\n + \t * output: special messages and conflict notices for various paths\n +@@ merge-ort.c: static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n + \tvoid (*strset_clear_func)(struct strset *) =\n + \t\treinitialize ? strset_partial_clear : strset_clear;\n + \n +-\tif (opti->pool)\n +-\t\tstrmap_clear_func(&opti->paths, 0);\n +-\telse {\n +-\t\t/*\n +-\t\t * We marked opti->paths with strdup_strings = 0, so that\n +-\t\t * we wouldn't have to make another copy of the fullpath\n +-\t\t * created by make_traverse_path from setup_path_info().\n +-\t\t * But, now that we've used it and have no other references\n +-\t\t * to these strings, it is time to deallocate them.\n +-\t\t */\n +-\t\tfree_strmap_strings(&opti->paths);\n +-\t\tstrmap_clear_func(&opti->paths, 1);\n +-\t}\n ++\tstrmap_clear_func(&opti->paths, 0);\n + \n + \t/*\n + \t * All keys and values in opti->conflicted are a subset of those in\n +@@ merge-ort.c: static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n + \t */\n + \tstrmap_clear_func(&opti->conflicted, 0);\n + \n +-\tif (!opti->pool) {\n +-\t\t/*\n +-\t\t * opti->paths_to_free is similar to opti->paths; we\n +-\t\t * created it with strdup_strings = 0 to avoid making\n +-\t\t * _another_ copy of the fullpath but now that we've used\n +-\t\t * it and have no other references to these strings, it is\n +-\t\t * time to deallocate them. We do so by temporarily\n +-\t\t * setting strdup_strings to 1.\n +-\t\t */\n +-\t\topti->paths_to_free.strdup_strings = 1;\n +-\t\tstring_list_clear(&opti->paths_to_free, 0);\n +-\t\topti->paths_to_free.strdup_strings = 0;\n +-\t}\n +-\n + \tif (opti->attr_index.cache_nr) /* true iff opt->renormalize */\n + \t\tdiscard_index(&opti->attr_index);\n + \n @@ merge-ort.c: static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\tstrmap_clear(&opti->output, 0);\n \t}\n \n -#if USE_MEMORY_POOL\n - \tmem_pool_discard(&opti->internal_pool, 0);\n - \tif (!reinitialize)\n - \t\topti->pool = NULL;\n +-\tmem_pool_discard(&opti->internal_pool, 0);\n +-\tif (!reinitialize)\n +-\t\topti->pool = NULL;\n -#endif\n ++\tmem_pool_discard(&opti->pool, 0);\n \n \t/* Clean out callback_data as well. */\n \tFREE_AND_NULL(renames->callback_data);\n @@ merge-ort.c: static void path_msg(struct merge_options *opt,\n -\t\treturn alloc_filespec(path);\n -\n -\t/* Similar to alloc_filespec, but allocate from pool and reuse path */\n -+\tassert(pool != NULL);\n \tspec = mem_pool_calloc(pool, 1, sizeof(*spec));\n \tspec->path = (char*)path; /* spec won't modify it */\n \n @@ merge-ort.c: static struct diff_filepair *pool_diff_queue(struct mem_pool *pool,\n -\t\treturn diff_queue(queue, one, two);\n -\n -\t/* Same code as diff_queue, except allocate from pool */\n -+\tassert(pool != NULL);\n \tdp = mem_pool_calloc(pool, 1, sizeof(*dp));\n \tdp->one = one;\n \tdp->two = two;\n @@ merge-ort.c: static void setup_path_info(struct merge_options *opt,\n -\tmi = pool_calloc(opt->priv->pool, 1,\n -\t\t\t resolved ? sizeof(struct merged_info) :\n -\t\t\t\t sizeof(struct conflict_info));\n -+\tmi = mem_pool_calloc(opt->priv->pool, 1,\n ++\tmi = mem_pool_calloc(&opt->priv->pool, 1,\n +\t\t\t resolved ? sizeof(struct merged_info) :\n +\t\t\t\t\tsizeof(struct conflict_info));\n \tmi->directory_name = current_dir_name;\n \tmi->basename_offset = current_dir_name_len;\n \tmi->clean = !!resolved;\n +@@ merge-ort.c: static void add_pair(struct merge_options *opt,\n + \t\t unsigned dir_rename_mask)\n + {\n + \tstruct diff_filespec *one, *two;\n +-\tstruct mem_pool *pool = opt->priv->pool;\n + \tstruct rename_info *renames = &opt->priv->renames;\n + \tint names_idx = is_add ? side : 0;\n + \n +@@ merge-ort.c: static void add_pair(struct merge_options *opt,\n + \t\t\treturn;\n + \t}\n + \n +-\tone = pool_alloc_filespec(pool, pathname);\n +-\ttwo = pool_alloc_filespec(pool, pathname);\n ++\tone = pool_alloc_filespec(&opt->priv->pool, pathname);\n ++\ttwo = pool_alloc_filespec(&opt->priv->pool, pathname);\n + \tfill_filespec(is_add ? two : one,\n + \t\t &names[names_idx].oid, 1, names[names_idx].mode);\n +-\tpool_diff_queue(pool, &renames->pairs[side], one, two);\n ++\tpool_diff_queue(&opt->priv->pool, &renames->pairs[side], one, two);\n + }\n + \n + static void collect_rename_info(struct merge_options *opt,\n @@ merge-ort.c: static int collect_merge_info_callback(int n,\n \tlen = traverse_path_len(info, p->pathlen);\n \n \t/* +1 in both of the following lines to include the NUL byte */\n -\tfullpath = pool_alloc(opt->priv->pool, len + 1);\n -+\tfullpath = mem_pool_alloc(opt->priv->pool, len + 1);\n ++\tfullpath = mem_pool_alloc(&opt->priv->pool, len + 1);\n \tmake_traverse_path(fullpath, len + 1, info, p->path, p->pathlen);\n \n \t/*\n +@@ merge-ort.c: static int handle_deferred_entries(struct merge_options *opt,\n + \t\tcopy = renames->deferred[side].possible_trivial_merges;\n + \t\tstrintmap_init_with_options(&renames->deferred[side].possible_trivial_merges,\n + \t\t\t\t\t 0,\n +-\t\t\t\t\t opt->priv->pool,\n ++\t\t\t\t\t &opt->priv->pool,\n + \t\t\t\t\t 0);\n + \t\tstrintmap_for_each_entry(©, &iter, entry) {\n + \t\t\tconst char *path = entry->key;\n @@ merge-ort.c: static void apply_directory_rename_modifications(struct merge_options *opt,\n + \tVERIFY_CI(ci);\n + \n + \t/* Find parent directories missing from opt->priv->paths */\n +-\tif (opt->priv->pool) {\n +-\t\tcur_path = mem_pool_strdup(opt->priv->pool, new_path);\n +-\t\tfree((char*)new_path);\n +-\t\tnew_path = (char *)cur_path;\n +-\t} else {\n +-\t\tcur_path = new_path;\n +-\t}\n ++\tcur_path = mem_pool_strdup(&opt->priv->pool, new_path);\n ++\tfree((char*)new_path);\n ++\tnew_path = (char *)cur_path;\n + \n + \twhile (1) {\n \t\t/* Find the parent directory of cur_path */\n \t\tchar *last_slash = strrchr(cur_path, '/');\n \t\tif (last_slash) {\n -\t\t\tparent_name = pool_strndup(opt->priv->pool,\n -\t\t\t\t\t\t cur_path,\n -\t\t\t\t\t\t last_slash - cur_path);\n -+\t\t\tparent_name = mem_pool_strndup(opt->priv->pool,\n ++\t\t\tparent_name = mem_pool_strndup(&opt->priv->pool,\n +\t\t\t\t\t\t cur_path,\n +\t\t\t\t\t\t last_slash - cur_path);\n \t\t} else {\n \t\t\tparent_name = opt->priv->toplevel_dir;\n \t\t\tbreak;\n +@@ merge-ort.c: static void apply_directory_rename_modifications(struct merge_options *opt,\n + \t\t/* Look it up in opt->priv->paths */\n + \t\tentry = strmap_get_entry(&opt->priv->paths, parent_name);\n + \t\tif (entry) {\n +-\t\t\tif (!opt->priv->pool)\n +-\t\t\t\tfree((char*)parent_name);\n + \t\t\tparent_name = entry->key; /* reuse known pointer */\n + \t\t\tbreak;\n + \t\t}\n +@@ merge-ort.c: static void apply_directory_rename_modifications(struct merge_options *opt,\n + \t\tparent_name = cur_dir;\n + \t}\n + \n +-\tif (!opt->priv->pool) {\n +-\t\t/*\n +-\t\t * We are removing old_path from opt->priv->paths.\n +-\t\t * old_path also will eventually need to be freed, but it\n +-\t\t * may still be used by e.g. ci->pathnames. So, store it\n +-\t\t * in another string-list for now.\n +-\t\t */\n +-\t\tstring_list_append(&opt->priv->paths_to_free, old_path);\n +-\t}\n +-\n + \tassert(ci->filemask == 2 || ci->filemask == 4);\n + \tassert(ci->dirmask == 0);\n + \tstrmap_remove(&opt->priv->paths, old_path, 0);\n +@@ merge-ort.c: static void apply_directory_rename_modifications(struct merge_options *opt,\n + \t\tnew_ci->stages[index].mode = ci->stages[index].mode;\n + \t\toidcpy(&new_ci->stages[index].oid, &ci->stages[index].oid);\n + \n +-\t\tif (!opt->priv->pool)\n +-\t\t\tfree(ci);\n + \t\tci = new_ci;\n + \t}\n + \n +@@ merge-ort.c: static void use_cached_pairs(struct merge_options *opt,\n + {\n + \tstruct hashmap_iter iter;\n + \tstruct strmap_entry *entry;\n +-\tstruct mem_pool *pool = opt->priv->pool;\n + \n + \t/*\n + \t * Add to side_pairs all entries from renames->cached_pairs[side_index].\n +@@ merge-ort.c: static void use_cached_pairs(struct merge_options *opt,\n + \t\tconst char *new_name = entry->value;\n + \t\tif (!new_name)\n + \t\t\tnew_name = old_name;\n +-\t\tif (pool) {\n +-\t\t\t/*\n +-\t\t\t * cached_pairs has _copies* of old_name and new_name,\n +-\t\t\t * because it has to persist across merges. When\n +-\t\t\t * pool != NULL\n +-\t\t\t * pool_alloc_filespec() will just re-use the existing\n +-\t\t\t * filenames, which will also get re-used by\n +-\t\t\t * opt->priv->paths if they become renames, and then\n +-\t\t\t * get freed at the end of the merge, leaving the copy\n +-\t\t\t * in cached_pairs dangling. Avoid this by making a\n +-\t\t\t * copy here.\n +-\t\t\t *\n +-\t\t\t * When pool == NULL, pool_alloc_filespec() calls\n +-\t\t\t * alloc_filespec(), which makes a copy; we don't want\n +-\t\t\t * to add another.\n +-\t\t\t */\n +-\t\t\told_name = mem_pool_strdup(pool, old_name);\n +-\t\t\tnew_name = mem_pool_strdup(pool, new_name);\n +-\t\t}\n ++\n ++\t\t/*\n ++\t\t * cached_pairs has *copies* of old_name and new_name,\n ++\t\t * because it has to persist across merges. Since\n ++\t\t * pool_alloc_filespec() will just re-use the existing\n ++\t\t * filenames, which will also get re-used by\n ++\t\t * opt->priv->paths if they become renames, and then\n ++\t\t * get freed at the end of the merge, that would leave\n ++\t\t * the copy in cached_pairs dangling. Avoid this by\n ++\t\t * making a copy here.\n ++\t\t */\n ++\t\told_name = mem_pool_strdup(&opt->priv->pool, old_name);\n ++\t\tnew_name = mem_pool_strdup(&opt->priv->pool, new_name);\n + \n + \t\t/* We don't care about oid/mode, only filenames and status */\n +-\t\tone = pool_alloc_filespec(pool, old_name);\n +-\t\ttwo = pool_alloc_filespec(pool, new_name);\n +-\t\tpool_diff_queue(pool, pairs, one, two);\n ++\t\tone = pool_alloc_filespec(&opt->priv->pool, old_name);\n ++\t\ttwo = pool_alloc_filespec(&opt->priv->pool, new_name);\n ++\t\tpool_diff_queue(&opt->priv->pool, pairs, one, two);\n + \t\tpairs->queue[pairs->nr-1]->status = entry->value ? 'R' : 'D';\n + \t}\n + }\n +@@ merge-ort.c: static int detect_regular_renames(struct merge_options *opt,\n + \tdiff_queued_diff = renames->pairs[side_index];\n + \ttrace2_region_enter(\"diff\", \"diffcore_rename\", opt->repo);\n + \tdiffcore_rename_extended(&diff_opts,\n +-\t\t\t\t opt->priv->pool,\n ++\t\t\t\t &opt->priv->pool,\n + \t\t\t\t &renames->relevant_sources[side_index],\n + \t\t\t\t &renames->dirs_removed[side_index],\n + \t\t\t\t &renames->dir_rename_count[side_index],\n +@@ merge-ort.c: static int collect_renames(struct merge_options *opt,\n + \n + \t\tif (p->status != 'A' && p->status != 'R') {\n + \t\t\tpossibly_cache_new_pair(renames, p, side_index, NULL);\n +-\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n ++\t\t\tpool_diff_free_filepair(&opt->priv->pool, p);\n + \t\t\tcontinue;\n + \t\t}\n + \n +@@ merge-ort.c: static int collect_renames(struct merge_options *opt,\n + \n + \t\tpossibly_cache_new_pair(renames, p, side_index, new_path);\n + \t\tif (p->status != 'R' && !new_path) {\n +-\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n ++\t\t\tpool_diff_free_filepair(&opt->priv->pool, p);\n + \t\t\tcontinue;\n + \t\t}\n + \n +@@ merge-ort.c: cleanup:\n + \t\tside_pairs = &renames->pairs[s];\n + \t\tfor (i = 0; i < side_pairs->nr; ++i) {\n + \t\t\tstruct diff_filepair *p = side_pairs->queue[i];\n +-\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n ++\t\t\tpool_diff_free_filepair(&opt->priv->pool, p);\n + \t\t}\n + \t}\n + \n +@@ merge-ort.c: simple_cleanup:\n + \tif (combined.nr) {\n + \t\tint i;\n + \t\tfor (i = 0; i < combined.nr; i++)\n +-\t\t\tpool_diff_free_filepair(opt->priv->pool,\n ++\t\t\tpool_diff_free_filepair(&opt->priv->pool,\n + \t\t\t\t\t\tcombined.queue[i]);\n + \t\tfree(combined.queue);\n + \t}\n @@ merge-ort.c: static void process_entry(struct merge_options *opt,\n \t\t * the directory to remain here, so we need to move this\n \t\t * path to some new location.\n \t\t */\n -\t\tnew_ci = pool_calloc(opt->priv->pool, 1, sizeof(*new_ci));\n -+\t\tnew_ci = mem_pool_calloc(opt->priv->pool, 1, sizeof(*new_ci));\n ++\t\tnew_ci = mem_pool_calloc(&opt->priv->pool, 1, sizeof(*new_ci));\n \n \t\t/* We don't really want new_ci->merged.result copied, but it'll\n \t\t * be overwritten below so it doesn't matter. We also don't\n @@ merge-ort.c: static void process_entry(struct merge_options *opt,\n \t\t\tint rename_a = 0, rename_b = 0;\n \n -\t\t\tnew_ci = pool_alloc(opt->priv->pool, sizeof(*new_ci));\n -+\t\t\tnew_ci = mem_pool_alloc(opt->priv->pool,\n ++\t\t\tnew_ci = mem_pool_alloc(&opt->priv->pool,\n +\t\t\t\t\t\tsizeof(*new_ci));\n \n \t\t\tif (S_ISREG(a_mode))\n \t\t\t\trename_a = 1;\n +@@ merge-ort.c: static void process_entry(struct merge_options *opt,\n + \t\t\t\tb_path = path;\n + \t\t\tstrmap_put(&opt->priv->paths, b_path, new_ci);\n + \n +-\t\t\tif (rename_a && rename_b) {\n ++\t\t\tif (rename_a && rename_b)\n + \t\t\t\tstrmap_remove(&opt->priv->paths, path, 0);\n +-\t\t\t\t/*\n +-\t\t\t\t * We removed path from opt->priv->paths. path\n +-\t\t\t\t * will also eventually need to be freed if not\n +-\t\t\t\t * part of a memory pool...but it may still be\n +-\t\t\t\t * used by e.g. ci->pathnames. So, store it in\n +-\t\t\t\t * another string-list for now in that case.\n +-\t\t\t\t */\n +-\t\t\t\tif (!opt->priv->pool)\n +-\t\t\t\t\tstring_list_append(&opt->priv->paths_to_free,\n +-\t\t\t\t\t\t\t path);\n +-\t\t\t}\n + \n + \t\t\t/*\n + \t\t\t * Do special handling for b_path since process_entry()\n @@ merge-ort.c: static void merge_start(struct merge_options *opt, struct merge_result *result)\n \n \t/* Initialization of various renames fields */\n \trenames = &opt->priv->renames;\n -#if USE_MEMORY_POOL\n - \tmem_pool_init(&opt->priv->internal_pool, 0);\n +-\tmem_pool_init(&opt->priv->internal_pool, 0);\n -\topt->priv->pool = &opt->priv->internal_pool;\n -#else\n -\topt->priv->pool = NULL;\n -#endif\n -\tpool = opt->priv->pool;\n -+\tpool = opt->priv->pool = &opt->priv->internal_pool;\n ++\tmem_pool_init(&opt->priv->pool, 0);\n ++\tpool = &opt->priv->pool;\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->dirs_removed[i],\n \t\t\t\t\t NOT_RELEVANT, pool, 0);\n +@@ merge-ort.c: static void merge_start(struct merge_options *opt, struct merge_result *result)\n + \t * Although we initialize opt->priv->paths with strdup_strings=0,\n + \t * that's just to avoid making yet another copy of an allocated\n + \t * string. Putting the entry into paths means we are taking\n +-\t * ownership, so we will later free it. paths_to_free is similar.\n ++\t * ownership, so we will later free it.\n + \t *\n + \t * In contrast, conflicted just has a subset of keys from paths, so\n + \t * we don't want to free those (it'd be a duplicate free).\n + \t */\n + \tstrmap_init_with_options(&opt->priv->paths, pool, 0);\n + \tstrmap_init_with_options(&opt->priv->conflicted, pool, 0);\n +-\tif (!opt->priv->pool)\n +-\t\tstring_list_init_nodup(&opt->priv->paths_to_free);\n + \n + \t/*\n + \t * keys & strbufs in output will sometimes need to outlive \"paths\",\n\n-- \ngitgitgadget\n"},{"id":"431630","messageId":"e075d985f269cdbf1fc67c99584b85d07538d603.1627752458.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v4.git.1627752458.gitgitgadget@gmail.com","subject":"[PATCH v4 1/9] merge-ort: rename str{map,intmap,set}_func()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-31T17:27:30Z","receivedAt":"2021-07-31T17:27:47Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIn order to make it clearer that these three variables holding a\nfunction refer to functions that will clear the strmap/strintmap/strset,\nrename them to str{map,intmap,set}_clear_func().\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 26 +++++++++++++-------------\n 1 file changed, 13 insertions(+), 13 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex e75b524153e..401a40247a3 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -519,11 +519,11 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n {\n \tstruct rename_info *renames = &opti->renames;\n \tint i;\n-\tvoid (*strmap_func)(struct strmap *, int) =\n+\tvoid (*strmap_clear_func)(struct strmap *, int) =\n \t\treinitialize ? strmap_partial_clear : strmap_clear;\n-\tvoid (*strintmap_func)(struct strintmap *) =\n+\tvoid (*strintmap_clear_func)(struct strintmap *) =\n \t\treinitialize ? strintmap_partial_clear : strintmap_clear;\n-\tvoid (*strset_func)(struct strset *) =\n+\tvoid (*strset_clear_func)(struct strset *) =\n \t\treinitialize ? strset_partial_clear : strset_clear;\n \n \t/*\n@@ -534,14 +534,14 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t * to deallocate them.\n \t */\n \tfree_strmap_strings(&opti->paths);\n-\tstrmap_func(&opti->paths, 1);\n+\tstrmap_clear_func(&opti->paths, 1);\n \n \t/*\n \t * All keys and values in opti->conflicted are a subset of those in\n \t * opti->paths. We don't want to deallocate anything twice, so we\n \t * don't free the keys and we pass 0 for free_values.\n \t */\n-\tstrmap_func(&opti->conflicted, 0);\n+\tstrmap_clear_func(&opti->conflicted, 0);\n \n \t/*\n \t * opti->paths_to_free is similar to opti->paths; we created it with\n@@ -559,24 +559,24 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \n \t/* Free memory used by various renames maps */\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n-\t\tstrintmap_func(&renames->dirs_removed[i]);\n-\t\tstrmap_func(&renames->dir_renames[i], 0);\n-\t\tstrintmap_func(&renames->relevant_sources[i]);\n+\t\tstrintmap_clear_func(&renames->dirs_removed[i]);\n+\t\tstrmap_clear_func(&renames->dir_renames[i], 0);\n+\t\tstrintmap_clear_func(&renames->relevant_sources[i]);\n \t\tif (!reinitialize)\n \t\t\tassert(renames->cached_pairs_valid_side == 0);\n \t\tif (i != renames->cached_pairs_valid_side &&\n \t\t -1 != renames->cached_pairs_valid_side) {\n-\t\t\tstrset_func(&renames->cached_target_names[i]);\n-\t\t\tstrmap_func(&renames->cached_pairs[i], 1);\n-\t\t\tstrset_func(&renames->cached_irrelevant[i]);\n+\t\t\tstrset_clear_func(&renames->cached_target_names[i]);\n+\t\t\tstrmap_clear_func(&renames->cached_pairs[i], 1);\n+\t\t\tstrset_clear_func(&renames->cached_irrelevant[i]);\n \t\t\tpartial_clear_dir_rename_count(&renames->dir_rename_count[i]);\n \t\t\tif (!reinitialize)\n \t\t\t\tstrmap_clear(&renames->dir_rename_count[i], 1);\n \t\t}\n \t}\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n-\t\tstrintmap_func(&renames->deferred[i].possible_trivial_merges);\n-\t\tstrset_func(&renames->deferred[i].target_dirs);\n+\t\tstrintmap_clear_func(&renames->deferred[i].possible_trivial_merges);\n+\t\tstrset_clear_func(&renames->deferred[i].target_dirs);\n \t\trenames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */\n \t}\n \trenames->cached_pairs_valid_side = 0;\n-- \ngitgitgadget\n\n"},{"id":"431632","messageId":"2c0b90eaba5bbf03553c8ac2487b57b66a6ed18e.1627752458.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v4.git.1627752458.gitgitgadget@gmail.com","subject":"[PATCH v4 3/9] merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-31T17:27:32Z","receivedAt":"2021-07-31T17:27:50Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMake the code more flexible so that it can handle both being run with or\nwithout a memory pool by adding utility functions which will either call\n xmalloc, xcalloc, xstrndup\nor\n mem_pool_alloc, mem_pool_calloc, mem_pool_strndup\ndepending on whether we have a non-NULL memory pool. A subsequent\ncommit will make use of these.\n\n(We will actually be dropping these functions soon and just assuming we\nalways have a memory pool, but the flexibility was very useful during\ndevelopment of merge-ort so I want to be able to restore it if needed.)\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 24 ++++++++++++++++++++++++\n 1 file changed, 24 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 401a40247a3..63f67246d3d 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -664,6 +664,30 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n+MAYBE_UNUSED\n+static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n+{\n+\tif (!pool)\n+\t\treturn xcalloc(count, size);\n+\treturn mem_pool_calloc(pool, count, size);\n+}\n+\n+MAYBE_UNUSED\n+static void *pool_alloc(struct mem_pool *pool, size_t size)\n+{\n+\tif (!pool)\n+\t\treturn xmalloc(size);\n+\treturn mem_pool_alloc(pool, size);\n+}\n+\n+MAYBE_UNUSED\n+static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len)\n+{\n+\tif (!pool)\n+\t\treturn xstrndup(str, len);\n+\treturn mem_pool_strndup(pool, str, len);\n+}\n+\n /* add a string to a strbuf, but converting \"/\" to \"_\" */\n static void add_flattened_path(struct strbuf *out, const char *s)\n {\n-- \ngitgitgadget\n\n"},{"id":"431633","messageId":"7c49aa601d0b0880dde559579a336809d88a1c01.1627752458.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v4.git.1627752458.gitgitgadget@gmail.com","subject":"[PATCH v4 5/9] merge-ort: switch our strmaps over to using memory pools","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-31T17:27:34Z","receivedAt":"2021-07-31T17:27:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nFor all the strmaps (including strintmaps and strsets) whose memory is\nunconditionally freed as part of clear_or_reinit_internal_opts(), switch\nthem over to using our new memory pool.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 202.5 ms ± 3.2 ms 198.1 ms ± 2.6 ms\n mega-renames: 1.072 s ± 0.012 s 715.8 ms ± 4.0 ms\n just-one-mega: 357.3 ms ± 3.9 ms 276.8 ms ± 4.2 ms\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 125 +++++++++++++++++++++++++++++++---------------------\n 1 file changed, 75 insertions(+), 50 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 3f425436263..99c75690855 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -539,15 +539,19 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \tvoid (*strset_clear_func)(struct strset *) =\n \t\treinitialize ? strset_partial_clear : strset_clear;\n \n-\t/*\n-\t * We marked opti->paths with strdup_strings = 0, so that we\n-\t * wouldn't have to make another copy of the fullpath created by\n-\t * make_traverse_path from setup_path_info(). But, now that we've\n-\t * used it and have no other references to these strings, it is time\n-\t * to deallocate them.\n-\t */\n-\tfree_strmap_strings(&opti->paths);\n-\tstrmap_clear_func(&opti->paths, 1);\n+\tif (opti->pool)\n+\t\tstrmap_clear_func(&opti->paths, 0);\n+\telse {\n+\t\t/*\n+\t\t * We marked opti->paths with strdup_strings = 0, so that\n+\t\t * we wouldn't have to make another copy of the fullpath\n+\t\t * created by make_traverse_path from setup_path_info().\n+\t\t * But, now that we've used it and have no other references\n+\t\t * to these strings, it is time to deallocate them.\n+\t\t */\n+\t\tfree_strmap_strings(&opti->paths);\n+\t\tstrmap_clear_func(&opti->paths, 1);\n+\t}\n \n \t/*\n \t * All keys and values in opti->conflicted are a subset of those in\n@@ -556,16 +560,19 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t */\n \tstrmap_clear_func(&opti->conflicted, 0);\n \n-\t/*\n-\t * opti->paths_to_free is similar to opti->paths; we created it with\n-\t * strdup_strings = 0 to avoid making _another_ copy of the fullpath\n-\t * but now that we've used it and have no other references to these\n-\t * strings, it is time to deallocate them. We do so by temporarily\n-\t * setting strdup_strings to 1.\n-\t */\n-\topti->paths_to_free.strdup_strings = 1;\n-\tstring_list_clear(&opti->paths_to_free, 0);\n-\topti->paths_to_free.strdup_strings = 0;\n+\tif (!opti->pool) {\n+\t\t/*\n+\t\t * opti->paths_to_free is similar to opti->paths; we\n+\t\t * created it with strdup_strings = 0 to avoid making\n+\t\t * _another_ copy of the fullpath but now that we've used\n+\t\t * it and have no other references to these strings, it is\n+\t\t * time to deallocate them. We do so by temporarily\n+\t\t * setting strdup_strings to 1.\n+\t\t */\n+\t\topti->paths_to_free.strdup_strings = 1;\n+\t\tstring_list_clear(&opti->paths_to_free, 0);\n+\t\topti->paths_to_free.strdup_strings = 0;\n+\t}\n \n \tif (opti->attr_index.cache_nr) /* true iff opt->renormalize */\n \t\tdiscard_index(&opti->attr_index);\n@@ -683,7 +690,6 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n-MAYBE_UNUSED\n static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n {\n \tif (!pool)\n@@ -691,7 +697,6 @@ static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n \treturn mem_pool_calloc(pool, count, size);\n }\n \n-MAYBE_UNUSED\n static void *pool_alloc(struct mem_pool *pool, size_t size)\n {\n \tif (!pool)\n@@ -699,7 +704,6 @@ static void *pool_alloc(struct mem_pool *pool, size_t size)\n \treturn mem_pool_alloc(pool, size);\n }\n \n-MAYBE_UNUSED\n static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len)\n {\n \tif (!pool)\n@@ -835,8 +839,9 @@ static void setup_path_info(struct merge_options *opt,\n \tassert(!df_conflict || !resolved); /* df_conflict implies !resolved */\n \tassert(resolved == (merged_version != NULL));\n \n-\tmi = xcalloc(1, resolved ? sizeof(struct merged_info) :\n-\t\t\t\t sizeof(struct conflict_info));\n+\tmi = pool_calloc(opt->priv->pool, 1,\n+\t\t\t resolved ? sizeof(struct merged_info) :\n+\t\t\t\t sizeof(struct conflict_info));\n \tmi->directory_name = current_dir_name;\n \tmi->basename_offset = current_dir_name_len;\n \tmi->clean = !!resolved;\n@@ -1128,7 +1133,7 @@ static int collect_merge_info_callback(int n,\n \tlen = traverse_path_len(info, p->pathlen);\n \n \t/* +1 in both of the following lines to include the NUL byte */\n-\tfullpath = xmalloc(len + 1);\n+\tfullpath = pool_alloc(opt->priv->pool, len + 1);\n \tmake_traverse_path(fullpath, len + 1, info, p->path, p->pathlen);\n \n \t/*\n@@ -1383,7 +1388,7 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\tcopy = renames->deferred[side].possible_trivial_merges;\n \t\tstrintmap_init_with_options(&renames->deferred[side].possible_trivial_merges,\n \t\t\t\t\t 0,\n-\t\t\t\t\t NULL,\n+\t\t\t\t\t opt->priv->pool,\n \t\t\t\t\t 0);\n \t\tstrintmap_for_each_entry(©, &iter, entry) {\n \t\t\tconst char *path = entry->key;\n@@ -2335,12 +2340,21 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \tVERIFY_CI(ci);\n \n \t/* Find parent directories missing from opt->priv->paths */\n-\tcur_path = new_path;\n+\tif (opt->priv->pool) {\n+\t\tcur_path = mem_pool_strdup(opt->priv->pool, new_path);\n+\t\tfree((char*)new_path);\n+\t\tnew_path = (char *)cur_path;\n+\t} else {\n+\t\tcur_path = new_path;\n+\t}\n+\n \twhile (1) {\n \t\t/* Find the parent directory of cur_path */\n \t\tchar *last_slash = strrchr(cur_path, '/');\n \t\tif (last_slash) {\n-\t\t\tparent_name = xstrndup(cur_path, last_slash - cur_path);\n+\t\t\tparent_name = pool_strndup(opt->priv->pool,\n+\t\t\t\t\t\t cur_path,\n+\t\t\t\t\t\t last_slash - cur_path);\n \t\t} else {\n \t\t\tparent_name = opt->priv->toplevel_dir;\n \t\t\tbreak;\n@@ -2349,7 +2363,8 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\t/* Look it up in opt->priv->paths */\n \t\tentry = strmap_get_entry(&opt->priv->paths, parent_name);\n \t\tif (entry) {\n-\t\t\tfree((char*)parent_name);\n+\t\t\tif (!opt->priv->pool)\n+\t\t\t\tfree((char*)parent_name);\n \t\t\tparent_name = entry->key; /* reuse known pointer */\n \t\t\tbreak;\n \t\t}\n@@ -2376,12 +2391,15 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\tparent_name = cur_dir;\n \t}\n \n-\t/*\n-\t * We are removing old_path from opt->priv->paths. old_path also will\n-\t * eventually need to be freed, but it may still be used by e.g.\n-\t * ci->pathnames. So, store it in another string-list for now.\n-\t */\n-\tstring_list_append(&opt->priv->paths_to_free, old_path);\n+\tif (!opt->priv->pool) {\n+\t\t/*\n+\t\t * We are removing old_path from opt->priv->paths.\n+\t\t * old_path also will eventually need to be freed, but it\n+\t\t * may still be used by e.g. ci->pathnames. So, store it\n+\t\t * in another string-list for now.\n+\t\t */\n+\t\tstring_list_append(&opt->priv->paths_to_free, old_path);\n+\t}\n \n \tassert(ci->filemask == 2 || ci->filemask == 4);\n \tassert(ci->dirmask == 0);\n@@ -2416,7 +2434,8 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\tnew_ci->stages[index].mode = ci->stages[index].mode;\n \t\toidcpy(&new_ci->stages[index].oid, &ci->stages[index].oid);\n \n-\t\tfree(ci);\n+\t\tif (!opt->priv->pool)\n+\t\t\tfree(ci);\n \t\tci = new_ci;\n \t}\n \n@@ -3623,7 +3642,8 @@ static void process_entry(struct merge_options *opt,\n \t\t * the directory to remain here, so we need to move this\n \t\t * path to some new location.\n \t\t */\n-\t\tCALLOC_ARRAY(new_ci, 1);\n+\t\tnew_ci = pool_calloc(opt->priv->pool, 1, sizeof(*new_ci));\n+\n \t\t/* We don't really want new_ci->merged.result copied, but it'll\n \t\t * be overwritten below so it doesn't matter. We also don't\n \t\t * want any directory mode/oid values copied, but we'll zero\n@@ -3715,7 +3735,7 @@ static void process_entry(struct merge_options *opt,\n \t\t\tconst char *a_path = NULL, *b_path = NULL;\n \t\t\tint rename_a = 0, rename_b = 0;\n \n-\t\t\tnew_ci = xmalloc(sizeof(*new_ci));\n+\t\t\tnew_ci = pool_alloc(opt->priv->pool, sizeof(*new_ci));\n \n \t\t\tif (S_ISREG(a_mode))\n \t\t\t\trename_a = 1;\n@@ -3788,12 +3808,14 @@ static void process_entry(struct merge_options *opt,\n \t\t\t\tstrmap_remove(&opt->priv->paths, path, 0);\n \t\t\t\t/*\n \t\t\t\t * We removed path from opt->priv->paths. path\n-\t\t\t\t * will also eventually need to be freed, but\n-\t\t\t\t * it may still be used by e.g. ci->pathnames.\n-\t\t\t\t * So, store it in another string-list for now.\n+\t\t\t\t * will also eventually need to be freed if not\n+\t\t\t\t * part of a memory pool...but it may still be\n+\t\t\t\t * used by e.g. ci->pathnames. So, store it in\n+\t\t\t\t * another string-list for now in that case.\n \t\t\t\t */\n-\t\t\t\tstring_list_append(&opt->priv->paths_to_free,\n-\t\t\t\t\t\t path);\n+\t\t\t\tif (!opt->priv->pool)\n+\t\t\t\t\tstring_list_append(&opt->priv->paths_to_free,\n+\t\t\t\t\t\t\t path);\n \t\t\t}\n \n \t\t\t/*\n@@ -4335,6 +4357,7 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n {\n \tstruct rename_info *renames;\n \tint i;\n+\tstruct mem_pool *pool = NULL;\n \n \t/* Sanity checks on opt */\n \ttrace2_region_enter(\"merge\", \"sanity checks\", opt->repo);\n@@ -4406,9 +4429,10 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n #else\n \topt->priv->pool = NULL;\n #endif\n+\tpool = opt->priv->pool;\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->dirs_removed[i],\n-\t\t\t\t\t NOT_RELEVANT, NULL, 0);\n+\t\t\t\t\t NOT_RELEVANT, pool, 0);\n \t\tstrmap_init_with_options(&renames->dir_rename_count[i],\n \t\t\t\t\t NULL, 1);\n \t\tstrmap_init_with_options(&renames->dir_renames[i],\n@@ -4422,7 +4446,7 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t\t */\n \t\tstrintmap_init_with_options(&renames->relevant_sources[i],\n \t\t\t\t\t -1 /* explicitly invalid */,\n-\t\t\t\t\t NULL, 0);\n+\t\t\t\t\t pool, 0);\n \t\tstrmap_init_with_options(&renames->cached_pairs[i],\n \t\t\t\t\t NULL, 1);\n \t\tstrset_init_with_options(&renames->cached_irrelevant[i],\n@@ -4432,9 +4456,9 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t}\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->deferred[i].possible_trivial_merges,\n-\t\t\t\t\t 0, NULL, 0);\n+\t\t\t\t\t 0, pool, 0);\n \t\tstrset_init_with_options(&renames->deferred[i].target_dirs,\n-\t\t\t\t\t NULL, 1);\n+\t\t\t\t\t pool, 1);\n \t\trenames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */\n \t}\n \n@@ -4447,9 +4471,10 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t * In contrast, conflicted just has a subset of keys from paths, so\n \t * we don't want to free those (it'd be a duplicate free).\n \t */\n-\tstrmap_init_with_options(&opt->priv->paths, NULL, 0);\n-\tstrmap_init_with_options(&opt->priv->conflicted, NULL, 0);\n-\tstring_list_init_nodup(&opt->priv->paths_to_free);\n+\tstrmap_init_with_options(&opt->priv->paths, pool, 0);\n+\tstrmap_init_with_options(&opt->priv->conflicted, pool, 0);\n+\tif (!opt->priv->pool)\n+\t\tstring_list_init_nodup(&opt->priv->paths_to_free);\n \n \t/*\n \t * keys & strbufs in output will sometimes need to outlive \"paths\",\n-- \ngitgitgadget\n\n"},{"id":"431634","messageId":"6646f6fd1cae3b7ebe279317b95405f2b6d71e9e.1627752458.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v4.git.1627752458.gitgitgadget@gmail.com","subject":"[PATCH v4 4/9] merge-ort: set up a memory pool","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-31T17:27:33Z","receivedAt":"2021-07-31T17:27:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nmerge-ort has a lot of data structures, and they all tend to be freed\ntogether in clear_or_reinit_internal_opts(). Set up a memory pool to\nallow us to make these allocations and deallocations faster. Future\ncommits will adjust various callers to make use of this memory pool.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 63f67246d3d..3f425436263 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -37,6 +37,8 @@\n #include \"unpack-trees.h\"\n #include \"xdiff-interface.h\"\n \n+#define USE_MEMORY_POOL 1 /* faster, but obscures memory leak hunting */\n+\n /*\n * We have many arrays of size 3. Whenever we have such an array, the\n * indices refer to one of the sides of the three-way merge. This is so\n@@ -339,6 +341,17 @@ struct merge_options_internal {\n \t */\n \tstruct strmap conflicted;\n \n+\t/*\n+\t * pool: memory pool for fast allocation/deallocation\n+\t *\n+\t * We allocate room for lots of filenames and auxiliary data\n+\t * structures in merge_options_internal, and it tends to all be\n+\t * freed together too. Using a memory pool for these provides a\n+\t * nice speedup.\n+\t */\n+\tstruct mem_pool internal_pool;\n+\tstruct mem_pool *pool; /* NULL, or pointer to internal_pool */\n+\n \t/*\n \t * paths_to_free: additional list of strings to free\n \t *\n@@ -603,6 +616,12 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\tstrmap_clear(&opti->output, 0);\n \t}\n \n+#if USE_MEMORY_POOL\n+\tmem_pool_discard(&opti->internal_pool, 0);\n+\tif (!reinitialize)\n+\t\topti->pool = NULL;\n+#endif\n+\n \t/* Clean out callback_data as well. */\n \tFREE_AND_NULL(renames->callback_data);\n \trenames->callback_data_nr = renames->callback_data_alloc = 0;\n@@ -4381,6 +4400,12 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \n \t/* Initialization of various renames fields */\n \trenames = &opt->priv->renames;\n+#if USE_MEMORY_POOL\n+\tmem_pool_init(&opt->priv->internal_pool, 0);\n+\topt->priv->pool = &opt->priv->internal_pool;\n+#else\n+\topt->priv->pool = NULL;\n+#endif\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->dirs_removed[i],\n \t\t\t\t\t NOT_RELEVANT, NULL, 0);\n-- \ngitgitgadget\n\n"},{"id":"431635","messageId":"08cf2498f96759a94543d61b9a05e02280ab19d2.1627752458.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v4.git.1627752458.gitgitgadget@gmail.com","subject":"[PATCH v4 6/9] diffcore-rename, merge-ort: add wrapper functions for filepair alloc/dealloc","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-31T17:27:35Z","receivedAt":"2021-07-31T17:27:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe want to be able to allocate filespecs and filepairs using a mem_pool.\nHowever, filespec data will still remain outside the pool (perhaps in\nthe future we could plumb the pool through the various diff APIs to\nallocate the filespec data too, but for now we are limiting the scope).\nAdd some extra functions to allocate these appropriately based on the\nnon-NULL-ness of opt->priv->pool, as well as some extra functions to\nhandle correctly deallocating the relevant parts of them. A future\ncommit will make use of these new functions.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 41 +++++++++++++++++++++++++++++++++++++++++\n diffcore.h | 2 ++\n merge-ort.c | 42 ++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 85 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 73d884099eb..5bc559f79e9 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -1328,6 +1328,47 @@ static void handle_early_known_dir_renames(struct dir_rename_info *info,\n \trename_src_nr = new_num_src;\n }\n \n+static void free_filespec_data(struct diff_filespec *spec)\n+{\n+\tif (!--spec->count)\n+\t\tdiff_free_filespec_data(spec);\n+}\n+\n+MAYBE_UNUSED\n+static void pool_free_filespec(struct mem_pool *pool,\n+\t\t\t struct diff_filespec *spec)\n+{\n+\tif (!pool) {\n+\t\tfree_filespec(spec);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Similar to free_filespec(), but only frees the data. The spec\n+\t * itself was allocated in the pool and should not be individually\n+\t * freed.\n+\t */\n+\tfree_filespec_data(spec);\n+}\n+\n+MAYBE_UNUSED\n+void pool_diff_free_filepair(struct mem_pool *pool,\n+\t\t\t struct diff_filepair *p)\n+{\n+\tif (!pool) {\n+\t\tdiff_free_filepair(p);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Similar to diff_free_filepair() but only frees the data from the\n+\t * filespecs; not the filespecs or the filepair which were\n+\t * allocated from the pool.\n+\t */\n+\tfree_filespec_data(p->one);\n+\tfree_filespec_data(p->two);\n+}\n+\n void diffcore_rename_extended(struct diff_options *options,\n \t\t\t struct strintmap *relevant_sources,\n \t\t\t struct strintmap *dirs_removed,\ndiff --git a/diffcore.h b/diffcore.h\nindex 533b30e21e7..b58ee6b1934 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -127,6 +127,8 @@ struct diff_filepair {\n #define DIFF_PAIR_MODE_CHANGED(p) ((p)->one->mode != (p)->two->mode)\n \n void diff_free_filepair(struct diff_filepair *);\n+void pool_diff_free_filepair(struct mem_pool *pool,\n+\t\t\t struct diff_filepair *p);\n \n int diff_unmodified_pair(struct diff_filepair *);\n \ndiff --git a/merge-ort.c b/merge-ort.c\nindex 99c75690855..e79830f9181 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -690,6 +690,48 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n+MAYBE_UNUSED\n+static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n+\t\t\t\t\t\t const char *path)\n+{\n+\tstruct diff_filespec *spec;\n+\tsize_t len;\n+\n+\tif (!pool)\n+\t\treturn alloc_filespec(path);\n+\n+\t/* Same code as alloc_filespec, except allocate from pool */\n+\tlen = strlen(path);\n+\n+\tspec = mem_pool_calloc(pool, 1, st_add3(sizeof(*spec), len, 1));\n+\tmemcpy(spec+1, path, len);\n+\tspec->path = (void*)(spec+1);\n+\n+\tspec->count = 1;\n+\tspec->is_binary = -1;\n+\treturn spec;\n+}\n+\n+MAYBE_UNUSED\n+static struct diff_filepair *pool_diff_queue(struct mem_pool *pool,\n+\t\t\t\t\t struct diff_queue_struct *queue,\n+\t\t\t\t\t struct diff_filespec *one,\n+\t\t\t\t\t struct diff_filespec *two)\n+{\n+\tstruct diff_filepair *dp;\n+\n+\tif (!pool)\n+\t\treturn diff_queue(queue, one, two);\n+\n+\t/* Same code as diff_queue, except allocate from pool */\n+\tdp = mem_pool_calloc(pool, 1, sizeof(*dp));\n+\tdp->one = one;\n+\tdp->two = two;\n+\tif (queue)\n+\t\tdiff_q(queue, dp);\n+\treturn dp;\n+}\n+\n static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n {\n \tif (!pool)\n-- \ngitgitgadget\n\n"},{"id":"431636","messageId":"1556f0443c35a95fbdbc1cb1841f81f9c4615f4c.1627752458.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v4.git.1627752458.gitgitgadget@gmail.com","subject":"[PATCH v4 8/9] merge-ort: reuse path strings in pool_alloc_filespec","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-31T17:27:37Z","receivedAt":"2021-07-31T17:27:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\npool_alloc_filespec() was written so that the code when pool != NULL\nmimicked the code from alloc_filespec(), which including allocating\nenough extra space for the path and then copying it. However, the path\npassed to pool_alloc_filespec() is always going to already be in the\nsame memory pool, so we may as well reuse it instead of copying it.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 198.5 ms ± 3.4 ms 198.3 ms ± 2.9 ms\n mega-renames: 679.1 ms ± 5.6 ms 661.8 ms ± 5.9 ms\n just-one-mega: 271.9 ms ± 2.8 ms 264.6 ms ± 2.5 ms\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 29 ++++++++++++++++++++++-------\n 1 file changed, 22 insertions(+), 7 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex f4f0a3d57f0..86ab8f60121 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -694,17 +694,13 @@ static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n \t\t\t\t\t\t const char *path)\n {\n \tstruct diff_filespec *spec;\n-\tsize_t len;\n \n \tif (!pool)\n \t\treturn alloc_filespec(path);\n \n-\t/* Same code as alloc_filespec, except allocate from pool */\n-\tlen = strlen(path);\n-\n-\tspec = mem_pool_calloc(pool, 1, st_add3(sizeof(*spec), len, 1));\n-\tmemcpy(spec+1, path, len);\n-\tspec->path = (void*)(spec+1);\n+\t/* Similar to alloc_filespec, but allocate from pool and reuse path */\n+\tspec = mem_pool_calloc(pool, 1, sizeof(*spec));\n+\tspec->path = (char*)path; /* spec won't modify it */\n \n \tspec->count = 1;\n \tspec->is_binary = -1;\n@@ -2904,6 +2900,25 @@ static void use_cached_pairs(struct merge_options *opt,\n \t\tconst char *new_name = entry->value;\n \t\tif (!new_name)\n \t\t\tnew_name = old_name;\n+\t\tif (pool) {\n+\t\t\t/*\n+\t\t\t * cached_pairs has _copies* of old_name and new_name,\n+\t\t\t * because it has to persist across merges. When\n+\t\t\t * pool != NULL\n+\t\t\t * pool_alloc_filespec() will just re-use the existing\n+\t\t\t * filenames, which will also get re-used by\n+\t\t\t * opt->priv->paths if they become renames, and then\n+\t\t\t * get freed at the end of the merge, leaving the copy\n+\t\t\t * in cached_pairs dangling. Avoid this by making a\n+\t\t\t * copy here.\n+\t\t\t *\n+\t\t\t * When pool == NULL, pool_alloc_filespec() calls\n+\t\t\t * alloc_filespec(), which makes a copy; we don't want\n+\t\t\t * to add another.\n+\t\t\t */\n+\t\t\told_name = mem_pool_strdup(pool, old_name);\n+\t\t\tnew_name = mem_pool_strdup(pool, new_name);\n+\t\t}\n \n \t\t/* We don't care about oid/mode, only filenames and status */\n \t\tone = pool_alloc_filespec(pool, old_name);\n-- \ngitgitgadget\n\n"},{"id":"431637","messageId":"4ffa5af8b570772bc4ccc859ce05c04c289646b0.1627752458.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v4.git.1627752458.gitgitgadget@gmail.com","subject":"[PATCH v4 7/9] merge-ort: store filepairs and filespecs in our mem_pool","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-31T17:27:36Z","receivedAt":"2021-07-31T17:27:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 198.1 ms ± 2.6 ms 198.5 ms ± 3.4 ms\n mega-renames: 715.8 ms ± 4.0 ms 679.1 ms ± 5.6 ms\n just-one-mega: 276.8 ms ± 4.2 ms 271.9 ms ± 2.8 ms\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 9 ++++-----\n diffcore.h | 1 +\n merge-ort.c | 26 ++++++++++++++------------\n 3 files changed, 19 insertions(+), 17 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 5bc559f79e9..7e6b3e1b143 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -1334,7 +1334,6 @@ static void free_filespec_data(struct diff_filespec *spec)\n \t\tdiff_free_filespec_data(spec);\n }\n \n-MAYBE_UNUSED\n static void pool_free_filespec(struct mem_pool *pool,\n \t\t\t struct diff_filespec *spec)\n {\n@@ -1351,7 +1350,6 @@ static void pool_free_filespec(struct mem_pool *pool,\n \tfree_filespec_data(spec);\n }\n \n-MAYBE_UNUSED\n void pool_diff_free_filepair(struct mem_pool *pool,\n \t\t\t struct diff_filepair *p)\n {\n@@ -1370,6 +1368,7 @@ void pool_diff_free_filepair(struct mem_pool *pool,\n }\n \n void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct mem_pool *pool,\n \t\t\t struct strintmap *relevant_sources,\n \t\t\t struct strintmap *dirs_removed,\n \t\t\t struct strmap *dir_rename_count,\n@@ -1683,7 +1682,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \t\t\tpair_to_free = p;\n \n \t\tif (pair_to_free)\n-\t\t\tdiff_free_filepair(pair_to_free);\n+\t\t\tpool_diff_free_filepair(pool, pair_to_free);\n \t}\n \tdiff_debug_queue(\"done copying original\", &outq);\n \n@@ -1693,7 +1692,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \n \tfor (i = 0; i < rename_dst_nr; i++)\n \t\tif (rename_dst[i].filespec_to_free)\n-\t\t\tfree_filespec(rename_dst[i].filespec_to_free);\n+\t\t\tpool_free_filespec(pool, rename_dst[i].filespec_to_free);\n \n \tcleanup_dir_rename_info(&info, dirs_removed, dir_rename_count != NULL);\n \tFREE_AND_NULL(rename_dst);\n@@ -1710,5 +1709,5 @@ void diffcore_rename_extended(struct diff_options *options,\n \n void diffcore_rename(struct diff_options *options)\n {\n-\tdiffcore_rename_extended(options, NULL, NULL, NULL, NULL);\n+\tdiffcore_rename_extended(options, NULL, NULL, NULL, NULL, NULL);\n }\ndiff --git a/diffcore.h b/diffcore.h\nindex b58ee6b1934..badc2261c20 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -181,6 +181,7 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count);\n void diffcore_break(struct repository *, int);\n void diffcore_rename(struct diff_options *);\n void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct mem_pool *pool,\n \t\t\t struct strintmap *relevant_sources,\n \t\t\t struct strintmap *dirs_removed,\n \t\t\t struct strmap *dir_rename_count,\ndiff --git a/merge-ort.c b/merge-ort.c\nindex e79830f9181..f4f0a3d57f0 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -690,7 +690,6 @@ static void path_msg(struct merge_options *opt,\n \tstrbuf_addch(sb, '\\n');\n }\n \n-MAYBE_UNUSED\n static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n \t\t\t\t\t\t const char *path)\n {\n@@ -712,7 +711,6 @@ static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n \treturn spec;\n }\n \n-MAYBE_UNUSED\n static struct diff_filepair *pool_diff_queue(struct mem_pool *pool,\n \t\t\t\t\t struct diff_queue_struct *queue,\n \t\t\t\t\t struct diff_filespec *one,\n@@ -930,6 +928,7 @@ static void add_pair(struct merge_options *opt,\n \t\t unsigned dir_rename_mask)\n {\n \tstruct diff_filespec *one, *two;\n+\tstruct mem_pool *pool = opt->priv->pool;\n \tstruct rename_info *renames = &opt->priv->renames;\n \tint names_idx = is_add ? side : 0;\n \n@@ -980,11 +979,11 @@ static void add_pair(struct merge_options *opt,\n \t\t\treturn;\n \t}\n \n-\tone = alloc_filespec(pathname);\n-\ttwo = alloc_filespec(pathname);\n+\tone = pool_alloc_filespec(pool, pathname);\n+\ttwo = pool_alloc_filespec(pool, pathname);\n \tfill_filespec(is_add ? two : one,\n \t\t &names[names_idx].oid, 1, names[names_idx].mode);\n-\tdiff_queue(&renames->pairs[side], one, two);\n+\tpool_diff_queue(pool, &renames->pairs[side], one, two);\n }\n \n static void collect_rename_info(struct merge_options *opt,\n@@ -2893,6 +2892,7 @@ static void use_cached_pairs(struct merge_options *opt,\n {\n \tstruct hashmap_iter iter;\n \tstruct strmap_entry *entry;\n+\tstruct mem_pool *pool = opt->priv->pool;\n \n \t/*\n \t * Add to side_pairs all entries from renames->cached_pairs[side_index].\n@@ -2906,9 +2906,9 @@ static void use_cached_pairs(struct merge_options *opt,\n \t\t\tnew_name = old_name;\n \n \t\t/* We don't care about oid/mode, only filenames and status */\n-\t\tone = alloc_filespec(old_name);\n-\t\ttwo = alloc_filespec(new_name);\n-\t\tdiff_queue(pairs, one, two);\n+\t\tone = pool_alloc_filespec(pool, old_name);\n+\t\ttwo = pool_alloc_filespec(pool, new_name);\n+\t\tpool_diff_queue(pool, pairs, one, two);\n \t\tpairs->queue[pairs->nr-1]->status = entry->value ? 'R' : 'D';\n \t}\n }\n@@ -3016,6 +3016,7 @@ static int detect_regular_renames(struct merge_options *opt,\n \tdiff_queued_diff = renames->pairs[side_index];\n \ttrace2_region_enter(\"diff\", \"diffcore_rename\", opt->repo);\n \tdiffcore_rename_extended(&diff_opts,\n+\t\t\t\t opt->priv->pool,\n \t\t\t\t &renames->relevant_sources[side_index],\n \t\t\t\t &renames->dirs_removed[side_index],\n \t\t\t\t &renames->dir_rename_count[side_index],\n@@ -3066,7 +3067,7 @@ static int collect_renames(struct merge_options *opt,\n \n \t\tif (p->status != 'A' && p->status != 'R') {\n \t\t\tpossibly_cache_new_pair(renames, p, side_index, NULL);\n-\t\t\tdiff_free_filepair(p);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n \t\t\tcontinue;\n \t\t}\n \n@@ -3079,7 +3080,7 @@ static int collect_renames(struct merge_options *opt,\n \n \t\tpossibly_cache_new_pair(renames, p, side_index, new_path);\n \t\tif (p->status != 'R' && !new_path) {\n-\t\t\tdiff_free_filepair(p);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n \t\t\tcontinue;\n \t\t}\n \n@@ -3197,7 +3198,7 @@ cleanup:\n \t\tside_pairs = &renames->pairs[s];\n \t\tfor (i = 0; i < side_pairs->nr; ++i) {\n \t\t\tstruct diff_filepair *p = side_pairs->queue[i];\n-\t\t\tdiff_free_filepair(p);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n \t\t}\n \t}\n \n@@ -3210,7 +3211,8 @@ simple_cleanup:\n \tif (combined.nr) {\n \t\tint i;\n \t\tfor (i = 0; i < combined.nr; i++)\n-\t\t\tdiff_free_filepair(combined.queue[i]);\n+\t\t\tpool_diff_free_filepair(opt->priv->pool,\n+\t\t\t\t\t\tcombined.queue[i]);\n \t\tfree(combined.queue);\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"431638","messageId":"f8cd50794e9ae8974ef350218a1f205e007f969d.1627752458.git.gitgitgadget@gmail.com","threadId":"56152","inReplyTo":"pull.990.v4.git.1627752458.gitgitgadget@gmail.com","subject":"[PATCH v4 9/9] merge-ort: remove compile-time ability to turn off usage of memory pools","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-31T17:27:38Z","receivedAt":"2021-07-31T17:28:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSimplify code maintenance by removing the ability to toggle between\nusage of memory pools and direct allocations. This allows us to also\nremove paths_to_free since it was solely about bookkeeping to make sure\nwe freed the necessary paths, and allows us to remove some auxiliary\nfunctions.\n\nSuggested-by: Jeff King \nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 209 ++++++++++++----------------------------------------\n 1 file changed, 47 insertions(+), 162 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 86ab8f60121..88ade50f4ed 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -37,8 +37,6 @@\n #include \"unpack-trees.h\"\n #include \"xdiff-interface.h\"\n \n-#define USE_MEMORY_POOL 1 /* faster, but obscures memory leak hunting */\n-\n /*\n * We have many arrays of size 3. Whenever we have such an array, the\n * indices refer to one of the sides of the three-way merge. This is so\n@@ -305,8 +303,6 @@ struct merge_options_internal {\n \t * * these keys serve to intern all the path strings, which allows\n \t * us to do pointer comparison on directory names instead of\n \t * strcmp; we just have to be careful to use the interned strings.\n-\t * (Technically paths_to_free may track some strings that were\n-\t * removed from froms paths.)\n \t *\n \t * The values of paths:\n \t * * either a pointer to a merged_info, or a conflict_info struct\n@@ -349,18 +345,7 @@ struct merge_options_internal {\n \t * freed together too. Using a memory pool for these provides a\n \t * nice speedup.\n \t */\n-\tstruct mem_pool internal_pool;\n-\tstruct mem_pool *pool; /* NULL, or pointer to internal_pool */\n-\n-\t/*\n-\t * paths_to_free: additional list of strings to free\n-\t *\n-\t * If keys are removed from \"paths\", they are added to paths_to_free\n-\t * to ensure they are later freed. We avoid free'ing immediately since\n-\t * other places (e.g. conflict_info.pathnames[]) may still be\n-\t * referencing these paths.\n-\t */\n-\tstruct string_list paths_to_free;\n+\tstruct mem_pool pool;\n \n \t/*\n \t * output: special messages and conflict notices for various paths\n@@ -539,19 +524,7 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \tvoid (*strset_clear_func)(struct strset *) =\n \t\treinitialize ? strset_partial_clear : strset_clear;\n \n-\tif (opti->pool)\n-\t\tstrmap_clear_func(&opti->paths, 0);\n-\telse {\n-\t\t/*\n-\t\t * We marked opti->paths with strdup_strings = 0, so that\n-\t\t * we wouldn't have to make another copy of the fullpath\n-\t\t * created by make_traverse_path from setup_path_info().\n-\t\t * But, now that we've used it and have no other references\n-\t\t * to these strings, it is time to deallocate them.\n-\t\t */\n-\t\tfree_strmap_strings(&opti->paths);\n-\t\tstrmap_clear_func(&opti->paths, 1);\n-\t}\n+\tstrmap_clear_func(&opti->paths, 0);\n \n \t/*\n \t * All keys and values in opti->conflicted are a subset of those in\n@@ -560,20 +533,6 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t */\n \tstrmap_clear_func(&opti->conflicted, 0);\n \n-\tif (!opti->pool) {\n-\t\t/*\n-\t\t * opti->paths_to_free is similar to opti->paths; we\n-\t\t * created it with strdup_strings = 0 to avoid making\n-\t\t * _another_ copy of the fullpath but now that we've used\n-\t\t * it and have no other references to these strings, it is\n-\t\t * time to deallocate them. We do so by temporarily\n-\t\t * setting strdup_strings to 1.\n-\t\t */\n-\t\topti->paths_to_free.strdup_strings = 1;\n-\t\tstring_list_clear(&opti->paths_to_free, 0);\n-\t\topti->paths_to_free.strdup_strings = 0;\n-\t}\n-\n \tif (opti->attr_index.cache_nr) /* true iff opt->renormalize */\n \t\tdiscard_index(&opti->attr_index);\n \n@@ -623,11 +582,7 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\tstrmap_clear(&opti->output, 0);\n \t}\n \n-#if USE_MEMORY_POOL\n-\tmem_pool_discard(&opti->internal_pool, 0);\n-\tif (!reinitialize)\n-\t\topti->pool = NULL;\n-#endif\n+\tmem_pool_discard(&opti->pool, 0);\n \n \t/* Clean out callback_data as well. */\n \tFREE_AND_NULL(renames->callback_data);\n@@ -693,12 +648,9 @@ static void path_msg(struct merge_options *opt,\n static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool,\n \t\t\t\t\t\t const char *path)\n {\n+\t/* Similar to alloc_filespec(), but allocate from pool and reuse path */\n \tstruct diff_filespec *spec;\n \n-\tif (!pool)\n-\t\treturn alloc_filespec(path);\n-\n-\t/* Similar to alloc_filespec, but allocate from pool and reuse path */\n \tspec = mem_pool_calloc(pool, 1, sizeof(*spec));\n \tspec->path = (char*)path; /* spec won't modify it */\n \n@@ -712,12 +664,9 @@ static struct diff_filepair *pool_diff_queue(struct mem_pool *pool,\n \t\t\t\t\t struct diff_filespec *one,\n \t\t\t\t\t struct diff_filespec *two)\n {\n+\t/* Same code as diff_queue(), except allocate from pool */\n \tstruct diff_filepair *dp;\n \n-\tif (!pool)\n-\t\treturn diff_queue(queue, one, two);\n-\n-\t/* Same code as diff_queue, except allocate from pool */\n \tdp = mem_pool_calloc(pool, 1, sizeof(*dp));\n \tdp->one = one;\n \tdp->two = two;\n@@ -726,27 +675,6 @@ static struct diff_filepair *pool_diff_queue(struct mem_pool *pool,\n \treturn dp;\n }\n \n-static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size)\n-{\n-\tif (!pool)\n-\t\treturn xcalloc(count, size);\n-\treturn mem_pool_calloc(pool, count, size);\n-}\n-\n-static void *pool_alloc(struct mem_pool *pool, size_t size)\n-{\n-\tif (!pool)\n-\t\treturn xmalloc(size);\n-\treturn mem_pool_alloc(pool, size);\n-}\n-\n-static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len)\n-{\n-\tif (!pool)\n-\t\treturn xstrndup(str, len);\n-\treturn mem_pool_strndup(pool, str, len);\n-}\n-\n /* add a string to a strbuf, but converting \"/\" to \"_\" */\n static void add_flattened_path(struct strbuf *out, const char *s)\n {\n@@ -875,9 +803,9 @@ static void setup_path_info(struct merge_options *opt,\n \tassert(!df_conflict || !resolved); /* df_conflict implies !resolved */\n \tassert(resolved == (merged_version != NULL));\n \n-\tmi = pool_calloc(opt->priv->pool, 1,\n-\t\t\t resolved ? sizeof(struct merged_info) :\n-\t\t\t\t sizeof(struct conflict_info));\n+\tmi = mem_pool_calloc(&opt->priv->pool, 1,\n+\t\t\t resolved ? sizeof(struct merged_info) :\n+\t\t\t\t\tsizeof(struct conflict_info));\n \tmi->directory_name = current_dir_name;\n \tmi->basename_offset = current_dir_name_len;\n \tmi->clean = !!resolved;\n@@ -924,7 +852,6 @@ static void add_pair(struct merge_options *opt,\n \t\t unsigned dir_rename_mask)\n {\n \tstruct diff_filespec *one, *two;\n-\tstruct mem_pool *pool = opt->priv->pool;\n \tstruct rename_info *renames = &opt->priv->renames;\n \tint names_idx = is_add ? side : 0;\n \n@@ -975,11 +902,11 @@ static void add_pair(struct merge_options *opt,\n \t\t\treturn;\n \t}\n \n-\tone = pool_alloc_filespec(pool, pathname);\n-\ttwo = pool_alloc_filespec(pool, pathname);\n+\tone = pool_alloc_filespec(&opt->priv->pool, pathname);\n+\ttwo = pool_alloc_filespec(&opt->priv->pool, pathname);\n \tfill_filespec(is_add ? two : one,\n \t\t &names[names_idx].oid, 1, names[names_idx].mode);\n-\tpool_diff_queue(pool, &renames->pairs[side], one, two);\n+\tpool_diff_queue(&opt->priv->pool, &renames->pairs[side], one, two);\n }\n \n static void collect_rename_info(struct merge_options *opt,\n@@ -1170,7 +1097,7 @@ static int collect_merge_info_callback(int n,\n \tlen = traverse_path_len(info, p->pathlen);\n \n \t/* +1 in both of the following lines to include the NUL byte */\n-\tfullpath = pool_alloc(opt->priv->pool, len + 1);\n+\tfullpath = mem_pool_alloc(&opt->priv->pool, len + 1);\n \tmake_traverse_path(fullpath, len + 1, info, p->path, p->pathlen);\n \n \t/*\n@@ -1425,7 +1352,7 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\tcopy = renames->deferred[side].possible_trivial_merges;\n \t\tstrintmap_init_with_options(&renames->deferred[side].possible_trivial_merges,\n \t\t\t\t\t 0,\n-\t\t\t\t\t opt->priv->pool,\n+\t\t\t\t\t &opt->priv->pool,\n \t\t\t\t\t 0);\n \t\tstrintmap_for_each_entry(©, &iter, entry) {\n \t\t\tconst char *path = entry->key;\n@@ -2377,21 +2304,17 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \tVERIFY_CI(ci);\n \n \t/* Find parent directories missing from opt->priv->paths */\n-\tif (opt->priv->pool) {\n-\t\tcur_path = mem_pool_strdup(opt->priv->pool, new_path);\n-\t\tfree((char*)new_path);\n-\t\tnew_path = (char *)cur_path;\n-\t} else {\n-\t\tcur_path = new_path;\n-\t}\n+\tcur_path = mem_pool_strdup(&opt->priv->pool, new_path);\n+\tfree((char*)new_path);\n+\tnew_path = (char *)cur_path;\n \n \twhile (1) {\n \t\t/* Find the parent directory of cur_path */\n \t\tchar *last_slash = strrchr(cur_path, '/');\n \t\tif (last_slash) {\n-\t\t\tparent_name = pool_strndup(opt->priv->pool,\n-\t\t\t\t\t\t cur_path,\n-\t\t\t\t\t\t last_slash - cur_path);\n+\t\t\tparent_name = mem_pool_strndup(&opt->priv->pool,\n+\t\t\t\t\t\t cur_path,\n+\t\t\t\t\t\t last_slash - cur_path);\n \t\t} else {\n \t\t\tparent_name = opt->priv->toplevel_dir;\n \t\t\tbreak;\n@@ -2400,8 +2323,6 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\t/* Look it up in opt->priv->paths */\n \t\tentry = strmap_get_entry(&opt->priv->paths, parent_name);\n \t\tif (entry) {\n-\t\t\tif (!opt->priv->pool)\n-\t\t\t\tfree((char*)parent_name);\n \t\t\tparent_name = entry->key; /* reuse known pointer */\n \t\t\tbreak;\n \t\t}\n@@ -2428,16 +2349,6 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\tparent_name = cur_dir;\n \t}\n \n-\tif (!opt->priv->pool) {\n-\t\t/*\n-\t\t * We are removing old_path from opt->priv->paths.\n-\t\t * old_path also will eventually need to be freed, but it\n-\t\t * may still be used by e.g. ci->pathnames. So, store it\n-\t\t * in another string-list for now.\n-\t\t */\n-\t\tstring_list_append(&opt->priv->paths_to_free, old_path);\n-\t}\n-\n \tassert(ci->filemask == 2 || ci->filemask == 4);\n \tassert(ci->dirmask == 0);\n \tstrmap_remove(&opt->priv->paths, old_path, 0);\n@@ -2471,8 +2382,6 @@ static void apply_directory_rename_modifications(struct merge_options *opt,\n \t\tnew_ci->stages[index].mode = ci->stages[index].mode;\n \t\toidcpy(&new_ci->stages[index].oid, &ci->stages[index].oid);\n \n-\t\tif (!opt->priv->pool)\n-\t\t\tfree(ci);\n \t\tci = new_ci;\n \t}\n \n@@ -2888,7 +2797,6 @@ static void use_cached_pairs(struct merge_options *opt,\n {\n \tstruct hashmap_iter iter;\n \tstruct strmap_entry *entry;\n-\tstruct mem_pool *pool = opt->priv->pool;\n \n \t/*\n \t * Add to side_pairs all entries from renames->cached_pairs[side_index].\n@@ -2900,30 +2808,24 @@ static void use_cached_pairs(struct merge_options *opt,\n \t\tconst char *new_name = entry->value;\n \t\tif (!new_name)\n \t\t\tnew_name = old_name;\n-\t\tif (pool) {\n-\t\t\t/*\n-\t\t\t * cached_pairs has _copies* of old_name and new_name,\n-\t\t\t * because it has to persist across merges. When\n-\t\t\t * pool != NULL\n-\t\t\t * pool_alloc_filespec() will just re-use the existing\n-\t\t\t * filenames, which will also get re-used by\n-\t\t\t * opt->priv->paths if they become renames, and then\n-\t\t\t * get freed at the end of the merge, leaving the copy\n-\t\t\t * in cached_pairs dangling. Avoid this by making a\n-\t\t\t * copy here.\n-\t\t\t *\n-\t\t\t * When pool == NULL, pool_alloc_filespec() calls\n-\t\t\t * alloc_filespec(), which makes a copy; we don't want\n-\t\t\t * to add another.\n-\t\t\t */\n-\t\t\told_name = mem_pool_strdup(pool, old_name);\n-\t\t\tnew_name = mem_pool_strdup(pool, new_name);\n-\t\t}\n+\n+\t\t/*\n+\t\t * cached_pairs has *copies* of old_name and new_name,\n+\t\t * because it has to persist across merges. Since\n+\t\t * pool_alloc_filespec() will just re-use the existing\n+\t\t * filenames, which will also get re-used by\n+\t\t * opt->priv->paths if they become renames, and then\n+\t\t * get freed at the end of the merge, that would leave\n+\t\t * the copy in cached_pairs dangling. Avoid this by\n+\t\t * making a copy here.\n+\t\t */\n+\t\told_name = mem_pool_strdup(&opt->priv->pool, old_name);\n+\t\tnew_name = mem_pool_strdup(&opt->priv->pool, new_name);\n \n \t\t/* We don't care about oid/mode, only filenames and status */\n-\t\tone = pool_alloc_filespec(pool, old_name);\n-\t\ttwo = pool_alloc_filespec(pool, new_name);\n-\t\tpool_diff_queue(pool, pairs, one, two);\n+\t\tone = pool_alloc_filespec(&opt->priv->pool, old_name);\n+\t\ttwo = pool_alloc_filespec(&opt->priv->pool, new_name);\n+\t\tpool_diff_queue(&opt->priv->pool, pairs, one, two);\n \t\tpairs->queue[pairs->nr-1]->status = entry->value ? 'R' : 'D';\n \t}\n }\n@@ -3031,7 +2933,7 @@ static int detect_regular_renames(struct merge_options *opt,\n \tdiff_queued_diff = renames->pairs[side_index];\n \ttrace2_region_enter(\"diff\", \"diffcore_rename\", opt->repo);\n \tdiffcore_rename_extended(&diff_opts,\n-\t\t\t\t opt->priv->pool,\n+\t\t\t\t &opt->priv->pool,\n \t\t\t\t &renames->relevant_sources[side_index],\n \t\t\t\t &renames->dirs_removed[side_index],\n \t\t\t\t &renames->dir_rename_count[side_index],\n@@ -3082,7 +2984,7 @@ static int collect_renames(struct merge_options *opt,\n \n \t\tif (p->status != 'A' && p->status != 'R') {\n \t\t\tpossibly_cache_new_pair(renames, p, side_index, NULL);\n-\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n+\t\t\tpool_diff_free_filepair(&opt->priv->pool, p);\n \t\t\tcontinue;\n \t\t}\n \n@@ -3095,7 +2997,7 @@ static int collect_renames(struct merge_options *opt,\n \n \t\tpossibly_cache_new_pair(renames, p, side_index, new_path);\n \t\tif (p->status != 'R' && !new_path) {\n-\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n+\t\t\tpool_diff_free_filepair(&opt->priv->pool, p);\n \t\t\tcontinue;\n \t\t}\n \n@@ -3213,7 +3115,7 @@ cleanup:\n \t\tside_pairs = &renames->pairs[s];\n \t\tfor (i = 0; i < side_pairs->nr; ++i) {\n \t\t\tstruct diff_filepair *p = side_pairs->queue[i];\n-\t\t\tpool_diff_free_filepair(opt->priv->pool, p);\n+\t\t\tpool_diff_free_filepair(&opt->priv->pool, p);\n \t\t}\n \t}\n \n@@ -3226,7 +3128,7 @@ simple_cleanup:\n \tif (combined.nr) {\n \t\tint i;\n \t\tfor (i = 0; i < combined.nr; i++)\n-\t\t\tpool_diff_free_filepair(opt->priv->pool,\n+\t\t\tpool_diff_free_filepair(&opt->priv->pool,\n \t\t\t\t\t\tcombined.queue[i]);\n \t\tfree(combined.queue);\n \t}\n@@ -3701,7 +3603,7 @@ static void process_entry(struct merge_options *opt,\n \t\t * the directory to remain here, so we need to move this\n \t\t * path to some new location.\n \t\t */\n-\t\tnew_ci = pool_calloc(opt->priv->pool, 1, sizeof(*new_ci));\n+\t\tnew_ci = mem_pool_calloc(&opt->priv->pool, 1, sizeof(*new_ci));\n \n \t\t/* We don't really want new_ci->merged.result copied, but it'll\n \t\t * be overwritten below so it doesn't matter. We also don't\n@@ -3794,7 +3696,8 @@ static void process_entry(struct merge_options *opt,\n \t\t\tconst char *a_path = NULL, *b_path = NULL;\n \t\t\tint rename_a = 0, rename_b = 0;\n \n-\t\t\tnew_ci = pool_alloc(opt->priv->pool, sizeof(*new_ci));\n+\t\t\tnew_ci = mem_pool_alloc(&opt->priv->pool,\n+\t\t\t\t\t\tsizeof(*new_ci));\n \n \t\t\tif (S_ISREG(a_mode))\n \t\t\t\trename_a = 1;\n@@ -3863,19 +3766,8 @@ static void process_entry(struct merge_options *opt,\n \t\t\t\tb_path = path;\n \t\t\tstrmap_put(&opt->priv->paths, b_path, new_ci);\n \n-\t\t\tif (rename_a && rename_b) {\n+\t\t\tif (rename_a && rename_b)\n \t\t\t\tstrmap_remove(&opt->priv->paths, path, 0);\n-\t\t\t\t/*\n-\t\t\t\t * We removed path from opt->priv->paths. path\n-\t\t\t\t * will also eventually need to be freed if not\n-\t\t\t\t * part of a memory pool...but it may still be\n-\t\t\t\t * used by e.g. ci->pathnames. So, store it in\n-\t\t\t\t * another string-list for now in that case.\n-\t\t\t\t */\n-\t\t\t\tif (!opt->priv->pool)\n-\t\t\t\t\tstring_list_append(&opt->priv->paths_to_free,\n-\t\t\t\t\t\t\t path);\n-\t\t\t}\n \n \t\t\t/*\n \t\t\t * Do special handling for b_path since process_entry()\n@@ -4482,13 +4374,8 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \n \t/* Initialization of various renames fields */\n \trenames = &opt->priv->renames;\n-#if USE_MEMORY_POOL\n-\tmem_pool_init(&opt->priv->internal_pool, 0);\n-\topt->priv->pool = &opt->priv->internal_pool;\n-#else\n-\topt->priv->pool = NULL;\n-#endif\n-\tpool = opt->priv->pool;\n+\tmem_pool_init(&opt->priv->pool, 0);\n+\tpool = &opt->priv->pool;\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n \t\tstrintmap_init_with_options(&renames->dirs_removed[i],\n \t\t\t\t\t NOT_RELEVANT, pool, 0);\n@@ -4525,15 +4412,13 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t * Although we initialize opt->priv->paths with strdup_strings=0,\n \t * that's just to avoid making yet another copy of an allocated\n \t * string. Putting the entry into paths means we are taking\n-\t * ownership, so we will later free it. paths_to_free is similar.\n+\t * ownership, so we will later free it.\n \t *\n \t * In contrast, conflicted just has a subset of keys from paths, so\n \t * we don't want to free those (it'd be a duplicate free).\n \t */\n \tstrmap_init_with_options(&opt->priv->paths, pool, 0);\n \tstrmap_init_with_options(&opt->priv->conflicted, pool, 0);\n-\tif (!opt->priv->pool)\n-\t\tstring_list_init_nodup(&opt->priv->paths_to_free);\n \n \t/*\n \t * keys & strbufs in output will sometimes need to outlive \"paths\",\n-- \ngitgitgadget\n"},{"id":"431698","messageId":"bc74066d-028a-bf7a-cbf1-fda0053bfe00@gmail.com","threadId":"56152","inReplyTo":"pull.990.v4.git.1627752458.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 0/9] Final optimization batch (#15): use memory pools","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-02T15:27:56Z","receivedAt":"2021-08-02T15:28:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/31/2021 1:27 PM, Elijah Newren via GitGitGadget wrote:\n> This series textually depends on en/ort-perf-batch-14, but the ideas are\n> orthogonal to it and orthogonal to previous series. It can be reviewed\n> independently.\n> \n> Changes since v1, addressing Eric's feedback:\n> \n> * Fixed a comment that became out-of-date in patch 1\n> * Swapped commits 2 and 3 so that one can better motivate the other.\n> \n> Changes since v2, addressing Peff's feedback:\n> \n> * Rebased on en/ort-perf-batch-14 (resolving a trivial conflict with the\n> new string_list_init_nodup() usage)\n> * Added a new preliminary patch renaming str*_func() to str*_clear_func()\n> * Added a new final patch that hardcodes that we'll just use memory pools\n> \n> Changes since v3, as per Peff's feedback:\n> \n> * Don't only remove the extra complexity from the USE_MEMORY_POOL #define;\n> also remove the original bookkeeping complexity needed to track\n> individual frees when not using a memory pool.\nI read the discussion leading to these changes and gave this version another\npass. Looks good to me. Thanks!\n\n-Stolee\n"},{"id":"431835","messageId":"YQlkmLfO+eDWE9SJ@coredump.intra.peff.net","threadId":"56152","inReplyTo":"pull.990.v4.git.1627752458.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 0/9] Final optimization batch (#15): use memory pools","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-03T15:45:28Z","receivedAt":"2021-08-03T15:45:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Jul 31, 2021 at 05:27:29PM +0000, Elijah Newren via GitGitGadget wrote:\n\n> Changes since v3, as per Peff's feedback:\n> \n> * Don't only remove the extra complexity from the USE_MEMORY_POOL #define;\n> also remove the original bookkeeping complexity needed to track\n> individual frees when not using a memory pool.\n\nThanks. I skimmed over the changes in patch 9, under the assumption that\nthey were mostly what I showed before. ;)\n\nThe whole thing looks good to me.\n\n-Peff\n"}]}